Deployment architecture

The following diagram depicts the commonly used Elastic Stack deployment architecture:

This diagram depicts three possible architectures:

  • Ship the operation metrics directly to Elasticsearch: As seen in the preceding diagram, you will install various types of Beats, such as Metricbeat, Filebeat, Packetbeat, and so on, on the edge servers from which you would like to ship the operation metrics/logs. If no further processing is required, then the generated events can be shipped directly to the Elasticsearch cluster. Once the data is present in Elasticsearch, it can then be visualized/analyzed using Kibana. In this architecture, the flow of events would be Beats → Elasticsearch → Kibana
  • Ship the operation metrics to Logstash: The operation metrics/logs that are captured by Beats and installed on edge servers is sent to Logstash for further processing, such as parsing the logs or enriching log events. Then, the parsed/enriched events are pushed to Elasticsearch. To increase the processing capacity, you can scale up Logstash instances, for example, by configuring a set of Beats to send data to Logstash instance 1 and configuring another set of Beats to send data to Logstash instance 2, and so on. In this architecture, the flow of events would be Beats → Logstash → Elasticsearch → Kibana
  • Ship the operation metrics to a resilient queue: If the generated events are at a very high rate and if Logstash is unable to cope with the load or to prevent loss of data/events when Logstash is down, you can go for resilient queues such as Apache Kafka so that events are queued. Then, Logstash can process them at its own speed, thus avoiding the loss of operation metrics/logs captured by Beats. In this architecture, the flow of events would be Beats → Kafka → Logstash → Elasticsearch → Kibana
Starting with Logstash 5.x, you can make use of the persistent queue settings of Logstash and make use of it as queue, too. However, it doesn't offer a high degree of resilience like Kafka. 

In the aforementioned architectures, you can easily scale up/scale down instances of Elasticsearch, Logstash, and Kibana based on the use case at hand.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
52.15.231.106