On the Red Hat family

We install Java via the yum command:

$ sudo yum install java-1.8.0-openjdk

On Debian & Ubuntu

We install Java via the apt-get command:

$ sudo apt-get -y install default-jre

Testing Java is installed

We then test that Java is installed via the java binary:

$ java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

Any Java versioned prefixed 1.8 indicates Java 8.

Installing Logstash via configuration management

You could also install Logstash via a variety of configuration management tools like Puppet or Chef or via Docker or Vagrant.

You can find Chef cookbooks for Logstash at the Chef supermarket.

You can find Puppet modules for Logstash on the Puppet Forge.

You can find an Ansible role for Logstash here.

You can find Docker images for Logstash on the Docker Hub.

You can find a Vagrant configuration for Logstash on GitHub.

Installing Elasticsearch

Elasticsearch's only prerequisite is Java. As we installed a JDK earlier in this chapter we don't need to install anything additional for it. Elasticsearch is currently not well packaged in distributions but it is easy to download packages. The Elasticsearch team provides tarballs, RPMs and DEB packages. You can find the Elasticsearch download page here.

As we're installing onto Ubuntu we can use the DEB packages provided:

First, if we haven't already, we install the Elastic.co package key.

$ wget -O - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Now, again assuming we haven't already down it above, we add the Elastic repository to our Apt configuration.

$ echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

Now we install Elasticsearch.

$ sudo apt-get update
$ sudo apt-get install elasticsearch

Installing the package should also automatically start the Elasticsearch server but if it does not then you can manage it via the service command:

$ sudo service elasticsearch start

And enable it to run at boot.

$ sudo systemctl enable elasticsearch

You can then check is Elasticsearch is running with the status command.

$ sudo service elasticsearch status

If it is NOT running you'll see failed start output. If your failed output looks like this:

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2016-11-03 13:11:32 UTC; 2s ago
     Docs: http://www.elastic.co

. . .

Nov 03 13:11:32 smoker.example.com systemd[1]: Started Elasticsearch.
Nov 03 13:11:32 smoker.example.com elasticsearch[1541]: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x000000008a660000, 1973026816, 0) failed; error='C
Nov 03 13:11:32 smoker.example.com elasticsearch[1541]: #
Nov 03 13:11:32 smoker.example.com elasticsearch[1541]: # There is insufficient memory for the Java Runtime Environment to continue.

. . .

Then Elasticsearch doesn't have enough memory to run! You'll need a host with at least 4Gb of RAM to run Elasticsearch.

Introduction to Elasticsearch

Now we've installed Elasticsearch we should learn a little about how it works. A decent understanding is going to be useful later as we use and scale Elasticsearch. Elasticsearch is a text indexing search engine. The best metaphor is the index of a book. You flip to the back of the book1, look up a word and then find the reference to a page. That means, rather than searching text strings directly, it creates an index from incoming text and performs searches on the index rather than the content. As a result it is fast.

Under the covers Elasticsearch uses Apache Lucene to create this index. Each index is a logical namespace, in Logstash's case the default indexes are named for the day the events are received, for example:

logstash-2012.12.31

Each Logstash event is made up of fields and these fields become a document inside that index. If we were comparing Elasticsearch to a relational database: an index is a table, a document is a table row and a field is a table column. Like a relational database you can define a schema too. Elasticsearch calls these schemas "mappings".

It's important to note that you don't have to specify any mappings for operations, indeed many of the searches you'll use with Logstash don't need mappings, but they often make life much easier. You can see an example of an Elasticsearch mapping here. Since Logstash 1.3.2 a default mapping is applied to your Elasticsearch and you generally no longer need to worry about setting your own mapping.

Like a schema, mapping declares what data and data types fields documents contain, any constraints present, unique and primary keys and how to index and search each field. Unlike a schema you can also specify Elasticsearch settings.

Indexes are stored in Lucene instances called "shards". There are two types of shards: primary and replica. Primary shards are where your documents are stored. Each new index automatically creates five primary shards. This is a default setting and you can increase or decrease the number of primary shards when the index is created but not AFTER it is created. Once you've created the index the number of primary shards cannot be changed.

Replica shards are copies of the primary shards that exist for two purposes:

• To protect your data.
• To make your searches faster.

Each primary shard will have one replica by default but can also have more if required. Unlike primary shards, this can be changed dynamically to scale out or make an index more resilient. Elasticsearch will cleverly distribute these shards across the available nodes and ensure primary and replica shards for an index are not present on the same node.

Shards are stored on Elasticsearch "nodes". Each node is automatically part of an Elasticsearch cluster, even if it's a cluster of one. When new nodes are created they can use unicast or multicast to discover other nodes that share their cluster name and will try to join that cluster. Elasticsearch distributes shards amongst all nodes in the cluster. It can move shards automatically from one node to another in the case of node failure or when new nodes are added.

Configuring our Elasticsearch cluster and node

Next we need to configure our Elasticsearch cluster and node name. Elasticsearch is started with a default cluster name and a random, allegedly amusing, node name, for example "Frank Kafka" or "Spider-Ham". A new random node name is selected each time Elasticsearch is restarted. Remember that new Elasticsearch nodes join any cluster with the same cluster name they have defined. So we want to customize our cluster and node names to ensure we have unique names. To do this we need to edit the /etc/elasticsearch/elasticsearch.yml file. This is Elasticsearch's YAML-based configuration file. Look for the following entries in the file:

# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1

We're going to uncomment and change both the cluster and node name. We're going to choose a cluster name of logstash and a node name matching our central server's host name.

cluster.name: logstash
node.name: "smoker"

We then need to restart Elasticsearch to reconfigure it.

$ sudo service elasticsearch restart

We can now check if Elasticsearch is running and active.

Determining Elasticsearch is running

You can tell if Elasticsearch is running by browsing to port 9200 on your host, for example:

$ curl http://localhost:9200

This should return some status information that looks like:

{
  "name" : "smoker",
  "cluster_name" : "logstash",
  "cluster_uuid" : "RyF1O9JLQqafAb6jzKIZkQ",
  "version" : {
    "number" : "5.0.0",
    "build_hash" : "253032b",
    "build_date" : "2016-10-26T05:11:34.737Z",
    "build_snapshot" : false,
    "lucene_version" : "6.2.0"
  },
  "tagline" : "You Know, for Search"
}

You can also browse to a more detailed statistics page:

http://localhost:9200/_stats?pretty=true

This will return a page that contains a variety of information about the state and status of your Elasticsearch server.

{
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  },
  "_all" : {
    "primaries" : { },
    "total" : { }
  },
  "indices" : { }
}

You can install a wide variety of plugins to Elasticsearch that can help you manage it.

You can list and install a plugin using the elasticsearch-plugin command that ships with Elasticsearch, for example:

$ sudo /usr/share/elasticsearch/bin/elasticsearch-plugin list

Plugins are generally available in the Elasticsearch server via URLs with a specific URL path, _plugins, being reserved for them.

The central.conf input block

For the input block we've specified one plugin: beats, with one option: port. The beats plugin starts a server on our Logstash central host to listen, on port 5044, for incoming events from Beats. On our client nodes we're going to use a Filebeat to send our events to the central server.

The central.conf output block

The contents of central.conf's output block is fairly easy to understand. We've already seen the stdout plugin in Chapter 2. Incoming events will be outputted to STDOUT and therefore to Logstash's own log file. I've done this for debugging purposes so we will be more easily able to see our incoming events. In a production environment you would probably disable this to prevent any excess noise being generated.

We've added another plugin called elasticsearch. This plugin sends events from Logstash to Elasticsearch to be stored and made available for searching. We're not configuring any options, which will mean that Logstash tries to connect to an Elasticsearch cluster located on the localhost on port 9200. In our case this will be the Elasticsearch cluster we installed earlier.

Checking Logstash is running

We can confirm that Logstash is running by a variety of means. First, we can use the init script itself:

$ sudo service logstash status
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2016-11-05 10:45:34 UTC; 2s ago
. . .

Finally, Logstash will send its own log output to log files in the /var/log/logstash/ directory.

Checking Filebeat is running

We can confirm that Filebeat is running by a variety of means. First, we can use the service command:

$ sudo service filebeat status
● filebeat.service - filebeat
   Loaded: loaded (/lib/systemd/system/filebeat.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-11-03 21:36:55 UTC; 5min ago
     Docs: https://www.elastic.co/guide/en/beats/filebeat/current/index.html
. . .

Finally, Filebeat will send its own log output to log files in the /var/log/filebeat/ directory.