Alongside SIEM technology, incident response analysts can also leverage a bundle of applications for log analysis. This bundle, referred to as the ELK Stack, combines three tools together that allows for the analysis of large sets of data. The ELK Stack is comprised of three components. The first of these is Elasticsearch. Elasticsearch is a log searching tool that allows for near real-time searching of log data. This is accomplished through full text searching, powered by Lucene. This allows analysts to perform queries against log files for such elements as user IDs, IP addresses, or log entry numbers. Another key feature of Elasticsearch is the ability for the platform to expand the solution as the enterprise grows larger and with more data sources. This is useful for organizations that may want to test this capability and then add data sources and log files incrementally.
The next component in the ELK Stack is Logstash. Logstash is the mechanism that handles the intake of log files from the sources across the network, process log entries, and finally, allows for their output through a visualization platform. Logstash can be configured and deployed easily. The integration of Logstash with Elasticsearch provides the incident response analyst the ability to conduct fast queries against a large amount of log data.
The final component of the ELK Stack is Kibana. Kibana serves as the visual interface or dashboard of the ELK Stack. This platform allows analysts to gain insight into the data through the use of dashboards. Kibana also allows analysts to drill down into specific key data points for detailed analysis. Incident response analysts can customize the dashboards so that the most critical information, such as intrusion detection logs or connection logs, are immediately available for review.
For example, the Kibana dashboard utilizes a number of pie charts to display log activity. Utilizing these allows for an overview of what information is available to an analyst.
In addition, there is the ability to drill down into greater detail for specific events:
These can also be further expanded, giving analysts depth into the specific event:
The ELK Stack is an open source platform and it can be configured by installing the separate components. This may be time consuming and does not allow for a proper evaluation of the stack. As a result, there are a few virtual machines or scripts that run that allow for evaluation of the toolset. For example, Phil Hagen of SANS has configured a virtual machine that is used in the SANS training classes. The VM and associated configuration files are available at: https://github.com/philhagen/sof-elk.
Another option available is to place the ELK Stack on top of a Security Onion installation. Chapter 3, Network Evidence Collection contained an overview of the Security Onion log management platform. The engineers have created an evaluation installation of the ELK Stack for review. The stack is installed through a script run on an existing Security Onion installation. The files and associated instructions can be found here: http://blog.securityonion.net/2017/03/towards-elk-on-security-onion.html.
The number of features of the ELK Stack and their use are too many to include as they can take up a whole volume themselves. Incident response analysts who are involved in parsing and examining log files would be best served by evaluating, and possibly deploying, the ELK Stack in their environment if there is currently no solution in place that allows for the aggregation and deep mining of log files for incident investigations.