The file plugin is used to stream events and log lines files to Logstash. It automatically detects file rotations, and reads from the point last read by it.

A most basic file configuration looks like this:

input{
file{
path => "/path/to/logfiles"
}

The only required configuration property is the path to the files. Let's look at how we can make use of some of the configuration properties of the file plugin to read different types of files.

add_field => { "input_time" => "%{@timestamp}" }

path =>["/app/packtpub/logs/*"]
exclude => "*.gz"

tags =>["processed"]

filter{
if  "processed" in tags[]{

}
}

input {
file{
path => ["var/log/syslog/*"]
type => "syslog"
}
file{
path => ["var/log/apache/*"]
type => "apache"
}
}

filter {
if [type] == "syslog" {
grok {

}

}
if [type] == "apache" {
grok {

}
}
}

The stdin plugin is used to stream events and log lines from standard input.

A basic configuration for stdin looks like this:

stdin {

}

When we configure stdin like this, whatever we type in the console will go as input to the Logstash event pipeline. This is mostly used as the first level of testing of configuration before plugging in the actual file or event input.

You may need to analyze a Twitter stream based on a topic of interest for various purposes, such as sentiment analysis, trending topics analysis, and so on. The twitter plugin is helpful to read events from the Twitter streaming API. This requires a consumer key, consumer secret, keyword, oauth token, and oauth token secret to work.

These details can be obtained by registering an application on the Twitter developer API page (https://dev.twitter.com/apps/new):

twitter {
    consumer_key => "your consumer key here"
    keywords => "keywords which you want to filter on streams" 
    consumer_secret => "your consumer secret here"
    oauth_token => "your oauth token here"
    oauth_token_secret => "your oauth token secret here"
}

keywords  => ["elk","packtpub"]

The lumberjack plugin is useful to receive events via the lumberjack protocol that is used in Logstash forwarder.

The basic required configuration option for the lumberjack plugin looks like this:

lumberjack {
    port => 
    ssl_certificate => 
    ssl_key => 
}

ssl_certificate => "/etc/ssl/logstash.pub"

    ssl_key => "/etc/ssl/logstash.key"

The redis plugin is used to read events and logs from the redis instance.

The basic configuration of the redis input plugin looks like this:

redis {
}

The csv plugin is used to write a CSV file as output, specifying the fields in csv and the path of the file.

The basic configuration of the csv output plugin looks like this:

csv {
    fields => ["date","open_price","close_price"]
    path => "/path/to/file.csv"
}

csv_options => {"col_sep" => "	" "row_sep" => "
"}

The file output plugin, just like the file input plugin, will be used to write events to a file in the file system.

The basic configuration of the file output plugin looks like this:

file
{
path = > "path/to/file"
}

The email plugin is a very important output plugin as it is very useful to send e-mails for certain events and failure scenarios.

The basic required configuration looks like this:

email {
    to => "[email protected]"
}

The elasticsearch plugin is the most important plugin used in ELK Stack, because it is where you will want to write your output to be stored to analyze later in Kibana.

We will take a look at ElasticSearch in more detail in Chapter 5, Why Do We Need Elasticsearch in ELK?, but let's look at the configuration options for this plugin here:

The basic configuration for the elasticsearch plugin looks like this:

elasticsearch {
}

Ganglia is a monitoring tool that is used to monitor the performance of a cluster of machines in a distributed computing environment. Ganglia makes uses of a daemon called Gmond, which is a small service that is installed on each machine that needs to be monitored.

The ganglia output plugin in Logstash is used to send metrics to the gmond service based on events in logs.

The basic ganglia output plugin configuration looks like this:

ganglia {
    metric => 
    value => 
}

The jira plugin doesn't come by default in Logstash installation but can be easily installed by a plugin install command like this:

bin/plugin install logstash-output-jira

The jira plugin is used to send events to a JIRA instance, which can create JIRA tickets based on certain events in your logs. To use this, the JIRA instance must accept REST API calls, since it internally makes use of JIRA REST API to pass the output events from Logstash to JIRA.

The basic configuration of the jira output plugin looks like this:

jira {
    issuetypeid => 
    password => 
    priority => 
    projectid =>
    summary => 
    username => 
}

As explained on the Hortonworks Kafka page (http://hortonworks.com/hadoop/kafka/):

The kafka output plugin is used to write certain events to a topic on kafka. It uses the Kafka Producer API to write messages to a topic on the broker.

The basic kafka configuration looks like this:

kafka {
    topic_id => 
}

The lumberjack plugin is used to write output to a Logstash forwarder or lumberjack.

The basic configuration for the lumberjack plugin looks like this:

lumberjack {
    hosts => 
    port => 
    ssl_certificate => 
}

The redis plugin is used to send events to a redis instance.

In RabbitMQ, the producer always sends messages to an exchange, and the exchange decides what to do with the messages. There are various exchange types that defines a further course of action for the messages, namely direct, topic, headers and fanout.

The rabbitmq plugin pushes the events from logs to the RabbitMQ exchange.

The basic configuration of the rabbitmq plugin looks like this:

rabbitmq {
    exchange => 
    exchange_type => 
    host => 
} 

The stdout plugin writes the output events to the console. It is used to debug the configuration to test the event output from Logstash before integrating with other systems.

The basic configuration looks like this:

output {
  stdout {}
}

MongoDB is a document-oriented NoSQL database, which stores data as JSON documents.

Like the jira plugin, this is also a community maintained plugin and doesn't ship with Logstash. It can be easily installed using the following plugin install command:

bin/plugin install logstash-output-mongodb

The basic configuration for the mongodb output plugin is:

mongodb {
    collection => 
    database => 
    uri => 
}

The csv filter is used to parse the data from an incoming CSV file and assign values to fields.

In ELK, it is very important to assign the correct timestamp to an event so that it can be analyzed on the time filter in Kibana. The date filter is meant to assign the appropriate timestamp based on fields in logs, or events, and assign a proper format to the timestamp.

If the date filter is not set, Logstash will assign a timestamp as the first time it sees the event or when the file is read.

The basic configuration of the date filter looks like this:

date {
} 

The drop filter is used to drop everything that matches the conditionals for this filter.

Let's take the following instance as an example:

filter {
if [fieldname == "test"] {
drop {
}
}
}

The preceding filter will cause all events having the test fieldname to be dropped. This is very helpful to filter out non useful information out of the incoming events.

The geoip filter is used to add the geographical location of the IP address present in the incoming event. It fetches this information from the Maxmind database.

The basic configuration of the geoip filter looks like this:

geoip {
    source => 
}

The grok option is by far the most popular and most powerful plugin that Logstash has. It can parse any unstructured log event and convert it into a structured set of fields that can be processed further and used in analysis.

It is used to parse any type of logs, whether it be apache logs, mysql logs, custom application logs, or just any unstructured text in events.

Logstash, by default, comes with a set of grok patterns that can be directly used to tag certain types of fields, and custom regular expressions are also supported.

All available grok patterns are available at:

https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

Some examples of the grok patterns are as follows:

HOSTNAME (?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(.?|)
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)
YEAR (?>dd){1,2}
HOUR (?:2[0123]|[01]?[0-9])
MINUTE (?:[0-5][0-9])

The preceding grok patterns can be directly used to tag fields of those types with an operator like this:

%{HOSTNAME:host_name}

Here, host_name is the field name that we want to assign to the part of the log event that represents the hostname like string.

Let's try to look at grok in more detail:

The grok patterns in logs are represented by this general format: %{SYNTAX:SEMANTIC}

Here, SYNTAX is the name of the pattern that matches the text in log, and SEMANTIC is the field name that we want to assign to that pattern.

Let's take the following instance as an example:

Let's say you want to represent the number of bytes transferred in one event:

%{NUMBER:bytes_transferred}

Here, bytes_transferred will refer to the actual value of bytes transferred in the log event.

Let's take a look at how we can represent a line from HTTP logs:

54.3.245.1 GET /index.html 14562  0.056

The grok pattern would be represented as:

%{IP:client_ip} %{WORD: request_method } %{URIPATHPARAM:uri_path} %{NUMBER:bytes_transferred} %{NUMBER:duration}

The basic grok configuration for the preceding event will look like this:

filter{
grok{
match =>  { "message" =>"%{IP:client_ip} %{WORD:request_method} %{URIPATHPARAM:uri_path} %{NUMBER:bytes_transferred} %{NUMBER:duration}"}
}
}

After being processed with this grok filter, we can see the following fields added to the event with the values:

The mutate filter is an important filter plugin that helps rename, remove, replace, and modify fields in an incoming event. It is also specially used to convert the data type of fields, merge two fields, and convert text from lower case to upper case and vice versa.

The basic configuration of the mutate filter looks like this:

filter {
mutate {
}
}

The sleep option is used to put Logstash in sleep mode for the amount of time specified. We can also specify the frequency of sleep intervals based on the number of events.

Let's take the following instance as an example:

If we want to let Logstash sleep for 1 sec for every fifth event processed, we can configure it like this:

filter {
  sleep {
    time => "1"   # Sleep 1 second
    every => 5   # Sleep on every 5th event.
  }
}

An extensive list and the latest documentation on all available Logstash filter plugins is available at https://www.elastic.co/guide/en/logstash/current/filter-plugins.html.

If your input event or output event consists of full JSON documents, then the json codec plugin is helpful. It can be defined as:

input{
stdin{
codec => json{
}
}
}

Or it can be simply defined as:

input{
stdin{
codec => "json"
}
}

The line codec is used to read each line in an input as an event or to decode each outgoing event as a line. It can be defined as:

input{
stdin{
codec => line{
}
}
}

Or it can be simply defined as:

input{
stdin{
codec => "line"
}
}

The multiline codec is very helpful for certain types of events where you like to take more than one line as one event. This is really helpful in cases such as Java Exceptions or stack traces.

For example, the following configuration can take a full stack trace as one event:

input {
  file {
    path => "/var/log/someapp.log"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601} "
      negate => true
      what => previous
    }
  }
}

This will take all lines that don't start with a timestamp as a part of the previous line and consider everything as a single event.

The plain plugin is used to specify that there is no encoding or decoding required for events as it will be taken care of by corresponding input or output plugin types itself. For many plugins, such as redis, mongodb and so on, this is the default codec type.

The rubydebug plugin is used only with output event data, and it prints output event data using the Ruby Awesome Print library.

An extensive list and latest documentation on all available Logstash codec plugins is available at https://www.elastic.co/guide/en/logstash/current/codec-plugins.html.