Using Logstash

Logstash is a popular open source data collection engine with real-time pipelining capabilities. Logstash allows us to easily build a pipeline that can help in collecting data from a wide variety of input sources, and parse, enrich, unify, and store it in a wide variety of destinations. Logstash provides a set of plugins known as input filters and output plugins, which are easy to use and are pluggable in nature, thus easing the process of unifying and normalizing huge volumes and varieties of data. Logstash does the work of the ETL engine:

Some of the salient features of logstash are as follows:

  • Pluggable data pipeline architecture: Logstash contains over 200 plugins that have been developed by Elastic and the open source community, which can be used to mix, match, and orchestrate different inputs, filters, and outputs, while building pipelines for data processing.
  • Extensibility: Logstash is written in JRuby and, since it supports the pluggable pipeline architecture, you can easily build/create custom plugins to meet your custom needs.
  • Centralized data processing: Data from disparate sources can be easily pulled using the various input plugins it provides and can be enriched, transformed, and sent to different/multiple destinations. 
  • Variety and volume: Logstash handles all types of logging data, for example, Apache, NGNIX logs, system logs, and window event logs, and also collects metrics from a wide range of application platforms over TCP and UDP. Logstash can transform HTTP requests into events and provides webhooks for applications like Meetup, GitHub, JIRA, and so on. It also supports consuming data from existing relational/NoSQL databases and queues including Kafka, RabbitMQ, and so on. The Logstash data processing pipeline can be easily scaled horizontally, and, since Logstash 5, it supports persistent queues, thus providing the ability to reliably process huge volumes of incoming events/data.
  • Synergy: Logstash has a strong synergy with Elasticsearch, Beats, and Kibana, thus allowing you to build end-to-end log analysis solutions with ease.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.223.124.24