In the previous chapter, we learned about Splunk's new data model and Pivot functionality and how they can be used to further intelligence reporting. In this chapter, we will return to Splunk's SPL, diving deeper and making use of some very powerful search commands to facilitate a better understanding and correlation of event data. You will learn how to create transactions, build subsearches and understand concurrency, leverage field associations, and so on.

Looking at event counts, applying statistics to calculate averages, or finding the top values over time only provide a view of the data limited to one angle. Splunk's SPL contains some very powerful search commands that provide the ability to correlate data from different sources and understand or build relationships between the events. Through the building of relationships between data sets and looking at different angles of the data, you can better understand the impact one event might have on another. Additionally, correlating related values can provide a much more contextual value to teams when reviewing or analyzing a series of data.

Identifying and grouping transactions

Single events can be easily interpreted and understood, but these single events are often part of a series of events, where the event might be influenced by the preceding events or might affect the other events to come. By leveraging Splunk's ability to group associated events into transactions based on field values, the data can be presented in such a way that the reader understands the full context of an event and gets what led up to this point. Building transactions can also be useful when needing to understand the time duration between the start and finish of specific events or calculating values within a given transaction and comparing them to the values of others.

Converging data sources

Context is everything when it comes to building successful Operational Intelligence, and when you are stuck analyzing events from a single data source at a time, you might miss out on rich contextual information that other data sources could provide. With Splunk's ability to converge multiple data sources using the join or append search commands and search across them as if they were a single source, you can easily enrich the single data source and understand events from other sources that occurred at, or around, the same time.

For example, you might notice there are more timeouts than usual on your website, but when you analyze the website access log, everything appears normal. However, when you look at the application log, you notice that there are numerous failed connections to the database. Even so, by looking at each data source individually, it is hard to understand where the actual issue lies. Using Splunk's SPL to converge the data sources will allow for both the web access and application logs to be brought together into one view, to better understand and troubleshoot the sequence of events that might lead to website timeouts.

Identifying relationships between fields

In the Operational Intelligence world, the ability to identify relationships between fields can be a powerful asset. Understanding the values of a field, and how these values might have a relationship with the other field values within the same event, allows you to calculate the degree of certainty the values will provide in future events. By continually sampling events as they come in over time, you can become more accurate at predicting values in events as they occur. When used correctly, this can provide tremendous value in being able to actively predict the values of fields within events, leading to a more proactive incident or issue identification.

Predicting future values

Understanding system, application, and user behavior will always prove to be extremely valuable when building any intelligence program; however, the ability to predict future values can provide values more immense than simple modeling actions. The addition of predictive capabilities to an Operational Intelligence program enables the ability to become more proactive to issue identification, forecast system behavior, and plan and optimize thresholds more effectively.

Imagine being able to predict the number of sessions on your website, the number of purchases of a specific product, the response times during peak periods, or general tuning alerting thresholds to values that are substantiated rather than taking an educated guess. All of this is possible with predictive analytics; by looking back over the past events, you can better understand what the future will hold.

Discovering anomalous values

With the volume of data ever increasing, looking for events that are outliers is becoming more difficult and requires different techniques for their detection. The value of identifying these values is that it can lead to the identification of a resource issue, highlight malicious activities hidden within high volumes of events, or simply detect the users attempting to interact with the application in a way they were not designed to. Capitalize on these opportunities to capture the abnormalities and triage them accordingly.