After understanding the various components of Kibana, let's explore in detail how to use Kibana to analyze and visualize data for real-world scenarios. In this chapter, we will see an end-to-end workflow of how to fetch Twitter data, along with storing data in Elasticsearch. This will be followed by building beautiful visualizations in Kibana to examine various scenarios.

The two possible ways of fetching Twitter data directly into Elasticsearch are by using:

We will use Logstash Twitter input because rivers acting as plugins in Elasticsearch have become deprecated; that is, they will be removed in future versions of Elasticsearch.

Before we move further, let's understand Logstash in brief.

Logstash is an open source tool created by Jordan Sissel. He later joined Elasticsearch, which was renamed Elastic. It is a data collection tool aimed at fetching events for processing. Events are nothing but data containing a timestamp field in it. Logstash is responsible for processing events by connecting with various input sources and storing data in various output sources. It helps combine data from multiple sources and parses it by applying filters to modify the incoming data.

The main purposes of using Logstash are to read event data from different kinds of input sources (these can be a file, HTTP, GitHub, Elasticsearch, and so on), apply filters to transform and process the incoming events (these can be parsing, encoding JSON, aggregation, and so on), and send processed events to the destination source (this can be CSV, a file, CloudWatch, Elasticsearch. and so on).

Logstash can be described in brief as:

Input -------> Filter ---------à Output

The various input plugins available in Logstash are shown here:

The various filter plugins available in Logstash are as follows:

Finally, the various output plugins available in Logstash are shown in the following image:

In this chapter, we are going to take a look at the following topics: