As we have seen so far in this chapter, use just the Python tools as we have done would adequately monitor your network with enough scalability for all types of networks, large and small alike. However, I would like to introduce one additional open source, general-purpose, distributed, search and analytics engine called Elasticsearch (https://www.elastic.co/). It is often referred to as the Elastic or ELK stack for combining with the frontend and input tools.

If you look at network monitoring in general, it is really about analyzing network data and making sense out of them. The ELK stack contains Elasticsearch, Logstash, and Kibina as a full stack to ingest information with Logstash, index and analyze data with Elasticsearch, and present the graphics output via Kibana. It is really three projects in one with the flexibility to substitute Logstash with another input, such as Beats. Alternatively, you can use other tools, such as Grafana, instead of Kibana for visualization. The ELK stack by Elastic Co. also provides many add-on tools, referred to as X-Pack, for additional security, alerting, monitoring, and such.

As you can probably tell by the description, ELK (or even Elasticsearch alone) is a deep topic to cover, and there are many books written on the subject. Even covering the basic usage would take up more space than we can spare in this book. I have at times considered leaving the subject out of the book simply for its depth. However, ELK has become a very important tool for many of the projects that I am working on, including network monitoring. I feel leaving it out would be a huge disservice to you.

Therefore, I am going to take a few pages to briefly introduce the tool and a few use cases along with information for you to dig deeper if needed. We will go through the following topics:

• Setting up a hosted ELK service
• The Logstash format
• Python's helper script for Logstash formatting