In this section, we will develop a configuration file that will contain input and output. Here, input will be twitter and output will be elasticsearch, as we need to store data in Elasticsearch for visualization in Kibana. We will not use a filter as we want to store the tweets in the same way as they are tweeted.

The configuration file will look like this:

input {
  twitter {
    consumer_key =>  "XXXXXXXXXXXXXXXXXXX"
    consumer_secret =>  "XXXXXXXXXXXXXXXXXX"
    oauth_token =>  "XXXXXXXXXXXXXXXXXXXXXXXX"
    oauth_token_secret =>  "XXXXXXXXXXXXXXXXXXXX"
    keywords => ["#DragMeDownDay,"#BePositive"]
    full_tweet => "true"
  }
}
output {
  elasticsearch {
    protocol => "http"
    host => "localhost"
    port => "9200"
    index => "twitter"
    document_type => "realtime"
  }
}

Save this configuration as twitter.conf inside the bin folder of the downloaded Logstash folder.

Let's decode each parameter for better understanding.

Here, input defines an input, which is Twitter. Then, consumer_key, consumer_secret, oauth_token (the access token), and oauth_token_secret (the access secret token) are the credentials needed for the authorization of your application on Twitter, which is done as shown earlier. keywords is used to specify the keywords for which you want to fetch data. In this case, we fetched data about the trending topics on Twitter. All of these parameters are mandatory. The full_tweet parameter, when set to true, specifies that we want to fetch tweets with all fields. Setting it to false would specify Logstash to fetch tweets with limited fields.

Then, we have the following parameters of the output block:

Using the command prompt in Windows or the terminal in Ubuntu, navigate to the bin folder inside the Logstash downloaded folder. To run the Logstash configuration, run the following command:

logstash agent –f twitter.conf

Here, it tells Logstash to start its agent and read the configuration from the twitter.conf file.

Upon successful execution of the Logstash agent, the following message will be displayed: