If there is a need to handle all log data in one place, then ELK is being touted as the best-in-class open source log analytics solution. There are an application as well as system logs. Logs are typically errors, warnings, and exceptions. ELK is a combination of three different products, namely Elasticsearch, Logstash, and Kibana (ELK). The macro-level ELK architecture is given as follows:

• Elasticsearch is a search mechanism that is based on the Lucene search to store and retrieve its data. Elasticsearch is, in a way, a NoSQL database. That is, it stores multi-structured data and does not support SQL as the query language. Elasticsearch has a REST API, which uses either PUT or POST to fetch the data. Precisely speaking, Elasticsearch is a distributed RESTful search and analytics engine, and there are many use cases emerging for this excellent and elegant search tool. It can discover the expected as well as uncover the unexpected. Elasticsearch lets you perform and combine many types of searches (structured, unstructured, geo, and metrics). Elasticsearch is really fast and can run on your laptop or on hundreds of servers for searching petabytes of data.
  Elasticsearch uses standard RESTful APIs and JSON. The clients can be coded using many programming languages including Java, Python, .NET, and Groovy. As we all know, Hadoop is the most visible and viable mechanism for big data analytics. But Hadoop does batch processing only. If you want real-time processing of big data, then Elasticsearch is the way forward. It is all about leveraging the real-time search and analytics features of Elasticsearch to work on your big data by using the Elasticsearch-Hadoop (ES-Hadoop) connector. Increasingly, Elasticsearch is being primed for real-time and affordable log analytics.

• Logstash is an open source and server-side data processing pipeline that ingests data from a variety of data sources simultaneously and transforms and sends them to a preferred database. Logstash also handles unstructured data with ease. Logstash has more than 200 plugins built in, and it is easy to come out on our own. Because of its tight integration with Elasticsearch, Logstash is a popular choice for loading data from a multitude of data sources (system and application logs, web and application server logs) into the Elasticsearch database. Logstash offers a number of pre-built filters to readily and rewardingly transform common data types and index them in Elasticsearch.
  Logstash provides plugins for ingesting unstructured and semi-structured logs that are generated by IT servers, business applications, and mobile devices into the Elasticsearch cluster. Elasticsearch indexes the data and readies it for real-time analysis. The prominent use cases include application monitoring and anomaly and fraud detection. Depending on brewing needs, there are a number of alternative solutions to speedily ingest data into the Elasticsearch database. For example, the Amazon Elasticsearch service offers built-in integration with a number of its other services, such as Amazon Kinesis Firehose, Amazon CloudWatch Logs, and AWS IoT to get data seamlessly and to perform the analytics. There are open source solutions such as Apache Kaka and Apache FluentD to build our own data pipeline.
• Kibana is the last module of the famous ELK toolset and is an open source data visualization and exploration tool mainly used for performing log and time-series analytics, application monitoring, and IT operational analytics (ITOA). Kibana is gaining a lot of market and mind shares, as it makes it easy to make histograms, line graphs, pie charts, and heat maps. We can use Kibana to search, view, and interact with data stored in Elasticsearch indices. Furthermore, advanced data analysis and visualization comprising charts, maps, graphs, and tables can be easily accomplished through this unique tool.  

• Logz.io, the commercialized version of the ELK platform, is the world's most popular open source log analysis platform. This is made available as an enterprise-grade service in the cloud. The high availability, unbreakable security, and scalability are innately assured. Logz.io intrinsically applies advanced machine-learning capability to unravel critical and unnoticed errors and exceptions in real-time. Furthermore, it throws actionable and contextual data for faster resolution of hidden issues. Logz.io comes out with a suite of analytics and optimization tools, which helps organizations to sharply reduce the overall logging expenses as the data size grows. Logz.io enables users to start ELK in five minutes, perform and scale with ease. The upgrade and capacity management is being taken care of by the service providers. The enterprise version of the Logz.io platform ensures enterprise-grade security toward data security and privacy. Logz.io goes beyond what ELK achieves to create a comprehensive log analytics platform with a number of powerful features, such as integrated alerts, multiple sub-accounts, and third-party integration. Logz.io inherently applies pre-built and usecase-specific machine-learning across data, and user behavior and community knowledge to precisely identify anomalies. In short, it facilitates the goals of the world of data-driven insights and insights-driven decisions/actions.