In this chapter, we will learn about the alerting capabilities within Splunk. You will learn about:
- Alerting on abnormal web page response times
- Alerting on errors during checkout in real time
- Alerting on abnormal user behavior
- Alerting on failure and triggering a scripted response
- Alerting when predicted sales exceed inventory
Throughout the previous chapters in this book, you created a great deal of Splunk searches, including historic searches that look back over a period of time and real-time searches. In this chapter, you will learn about alerting—arguably, one of Splunk's most powerful features.
A key part of gaining complete operational intelligence is the ability to be proactive rather than reactive. Periodic, ad hoc searching of the data for certain conditions might provide some operational insight, but a better approach would be to continually monitor the data and know immediately when certain conditions are met. For example, instead of reacting to a network outage after it has occurred, it would be better to proactively look for the factors that could lead to a network outage and prevent it from occurring in the first place. It is this type of proactive approach that Splunk's alerting functionality allows for.
In this chapter, we will continue to build our Operational Intelligence application and incorporate alerting for a number of different scenarios. You will learn how to implement the different types of alerts and leverage a number of different alert actions.
About Splunk alerts
As with many features of Splunk, alerts are powered-off underlying searches. These underlying searches can either run on a schedule against historically indexed data or run against real-time data as it flows into Splunk. Alerts can then be triggered every time a search runs or when certain conditions are met as a result of the search.
Additionally, all alerting in Splunk can be throttled such that alerts do not continuously fire if similar conditions are met repeatedly, and this will be covered later in the chapter.
Note
Splunk has a dedicated manual for alerting, which can be found at http://docs.splunk.com/Documentation/Splunk/latest/Alert/Aboutalerts.
Types of alerts
There are three types of alerts and these are detailed in the following table:
- An example of a scheduled alert would be to trigger an alert whenever the number of web server errors exceeds 200 in any 30-minute interval.
- An example of a per-result alert would be to trigger an alert whenever a web server error occurs on a specified host. To avoid a flood of alerts, this can optionally be throttled to alert only once per time period, such as once per 10 minutes.
- An example of a rolling-window alert would be to trigger an alert whenever there are five consecutive errors of the same type for a specific session ID within a 10-minute window, but they are optionally throttled such that only one alert for a particular session ID is sent within a 30-minute time frame.
In this chapter, you will gain experience in creating all three types of alerts and apply them to real-world operational intelligence examples.
Trigger conditions
Alerts are triggered when the results of the search meet specific conditions. For example, you might have a condition that specifies to only alert when the count of results is greater than X. Triggering conditions are set when you set up the alert, and the following table lists the various conditions that are available:
|
Condition |
When is it triggered? |
|---|---|
|
Per-result |
This triggers whenever a search returns a result. It is only available for real-time alerts and leveraged by the per-result alert type. |
|
Number of results |
This triggers based on the number of search results. The options include greater than, less than, equal to, and not equal to. |
|
Number of hosts |
This triggers based on the number of hosts seen. The options include greater than, less than, equal to, and not equal to. |
|
Number of sources |
This triggers based on the number of sources seen. The options include greater than, less than, equal to, and not equal to. |
|
Custom |
This triggers based on a custom search condition. Think of this as sticking a custom search at the end of the base search; for example, |
Alert actions
So, what happens when an alert fires in Splunk? Well, that is up to you, as Splunk offers a number of possible actions out of the box, and these are detailed in the following table:
Multiple alert actions can be selected for a given alert. For example, you might wish to send an e-mail and also execute a script when a particular alert is triggered.
Tip
There are commands for Splunk that allow you to craft a search and send an e-mail directly from the search itself. These can be used in a fashion similar to alerting if the search that contains the commands is scheduled. One of these commands is sendemail, which is bundled with Splunk and allows search results to be sent to specified e-mail addresses. Another command is sendresults, which is developed by Discovered Intelligence and is available for free in the Splunk app store. The sendresults command allows you to dynamically evaluate where to send the search results, based on the search results themselves.