Recipe Difficulty: Medium

Python Version: 2.7

Operating System: Any

Log files can quickly become quite sizable due to the level of detail and time frame preserved. As you may have noticed, the CSV report from the prior recipe can easily become too large for our spreadsheet application to open or browse efficiently. Rather than analyzing this data in a spreadsheet, one alternative would be to load the data into a database.

Splunk is a platform that incorporates a NoSQL database with an ingestion and query engine, making it a powerful analysis tool. Its database operates in a manner like Elasticsearch or MongoDB, permitting the storage of documents or structured records. Because of this, we do not need to provide records with a consistent key-value mapping to store them in the database. This is what makes NoSQL databases so useful for log analysis, as log formats can be variable depending on the event type.

In this recipe, we learn to index the CSV report from the previous recipe into Splunk, allowing us to interact with the data inside the platform. We also design the script to run queries against the dataset and to export the resulting subset of data responsive to the query to a CSV file. These processes are handled in separate stages so we can independently query and export data as needed.