We will perform a basic, bare minimum setup of Kafka, only as a message broker (message going in an out of a component), playing a primary role of a Flume sink. We will go ahead with default configurations for now.

1. Download the Kafka binaries with the following command:

wget http://redrockdigimark.com/apachemirror/kafka/0.10.1.1/kafka_2.11-0.10.1.1.tgz

2. Change directory to a user directory, where we will want to extract the contents of the kafka tarball using the following command:

tar -xzvf <DOWNLOAD_DIRECTORY>/kafka_2.11-0.10.1.1.tgz

3. Let us refer to the extracted Kafka folder as ${KAFKA_HOME} and configure the same using the following command and add the same to ~/.bashrc file. Also, as with other installations, you can optionally update $PATH with ${KAFKA_HOME}/bin:

export KAFKA_HOME=${KAFKA_HOME}

4. Change the directory into the extracted Kafka folder, ${KAFKA_HOME} and run the following commands to start the Kafka server:

${KAFKA_HOME}/bin/zookeeper-server-start.sh
${KAFKA_HOME}/config/zookeeper.properties

And then, in a separate bash shell, start the kafka server with the following command

${KAFKA_HOME}/bin/kafka-server-start.sh
${KAFKA_HOME}/config/server.properties

5. On successful start of the Kafka server, you should be able to see the message started (kafka.server.KafkaServer) on the shell console.
6. This Kafka instance is started based on the default server.properties file which is bundled within the Kafka binary. For the purpose of Flume as an acquisition layer, this will be good enough.

In the following sections we will be detailing a number of examples required for completion of SCV scenarios. One such scenario will be to load the data from database while the other will be to load the data from unstructured data source, such as a spool file. We will see how to realize these scenarios with these examples.