By now, we should have a basic understanding of ELK Stack and what role it plays in log analysis or event analysis systems. In Chapter 2, Building Your First Data Pipeline with ELK, we built analytics over the GOOG stock price data with the ELK Stack configuration, and also understood role of each component of the stack in the pipeline.

In this chapter, we will get into more detail on Logstash, the most important component of the ELK Stack, and see how Logstash helps collect, parse, and transform any format and any type of data to a common format, which can be used to build a wide variety of analytics systems across many applications.

We saw in Chapter 1, Introduction to ELK Stack, the importance of log analysis and problem with log analysis methods. Also, there are a variety of log formats, and date and time formats in logs. Often these logs are customized to each application and require expert knowledge to gather essential information out of them. Now, we will see how Logstash provides us with a variety of plugins that help us overcome all of these problems and build easily-configurable and manageable visualizations on top of it.

In this chapter, while explaining various configurations of Logstash plugins, we assume that you have installed Logstash, as explained in Chapter 1, Introduction to ELK Stack, and have the basic configuration set up and running. If you do not, you should get Logstash installed and run a basic stdin, stdout configuration and then resume this chapter so that you can try out some configuration options based on the explanations.

Configuring Logstash

As we have seen in previous chapters, a general Logstash plugin configuration looks like this:

input {
  
}

filter {
  
}

output {
  
}

A Logstash configuration consists of a series of input, filter, and output plugins and their corresponding properties. Each plugin plays an important role towards parsing, processing, and finally putting the data in the required format. Input plugins generate the event, filters modify them, and output will ship them to other systems.

Logstash plugins