In this recipe, you will write a relatively simple real-time per-result type of alert to look for abnormal user behavior. The abnormal behavior you will be looking for is successful payments that did not go through the checkout process.
To step through this recipe, you will need a running Splunk Enterprise server, with the sample data loaded from Chapter 1, Play Time – Getting Data In. You should be familiar with navigating the Splunk user interface. You should also have configured the e-mail settings on your Splunk server to enable the delivery of e-mail alerts.
Follow the steps in this recipe to create an alert when abnormal user behavior occurs:
- Log in to your Splunk server.
- Select the Operational Intelligence application.
- In the Search bar, enter the following search over Last 24 hours:
index=main sourcetype=log4j requestType=checkout (numberOfItems>10 OR total>3000) | table ipAddress, numberOfItems, total, invoice, customerId, paymentId, orderId
- The search will run but might not produce any results if there are no results to display, and this is okay. As in the previous recipes, click on the Save As dropdown and select Alert.
- A pop-up box will appear to provide the alert configuration options. Perform the following steps on this screen:
- From the Severity dropdown, select Medium.
- Click on Add Actions. Select Send email from the Add Actions dropdown.
- Enter a valid e-mail address in the To box; this is where the alert will go to.
- In the Include section, check Link to Alert, Link to Results, Inline Table, and Trigger Time. These settings will ensure that there are links in the e-mail to both the alert and the results in Splunk. The Inline Table option will tabulate the results in the e-mail body.
- In the Type section, select HTML & Plain Text.
- Verify that the given details are entered correctly, and then click on Save.
- Click on View Alert. A summary screen should be displayed. The alert is now configured and running in real time.
- When the alert triggers, you should receive an e-mail similar to the following screenshot. Note that the key fields from the data are included in the e-mail body as an inline table, as specified when configuring the alert.
This recipe revolved around a fairly simplistic search that looked for purchase events that included more than 10 items or where the total value of the purchase was greater than $3,000. This might be considered abnormal in an environment where typical purchases involve two items and the total value is less than $1,000. Simplicity aside, it served to illustrate how a per-result type of alert functions. Essentially, as soon as a matching result is detected, the alert is triggered. The search runs over All time, in real time, just waiting and watching for a matching event to come in. There was no throttling enabled, so if five matching events were to come in, then the alert would be triggered five times. A per-result type of alert would not have been suitable for the previous recipe, as the previous recipe relied on a number of events over time being transacted together.
There are many different aspects of abnormal user behavior that you might wish to alert on, and this recipe touched on a rather obvious abnormality. For example, a more discrete user behavior might be where a successful order is made but there is no checkout event. This might indicate unauthorized access to the backend database, where an order has been made without actually paying for it.
In order to detect purchases where no checkout event exists, you will use a similar search as you did in the previous recipe (Alerting on errors during checkout in real time). A transactional search is required to group the entire thread together; once this has been performed, you can look for the threads that do not include the requestType checkout. The search will be as follows:
index=main sourcetype=log4j | transaction threadId maxspan=5m | search paymentReceived="Y" result="success" NOT requestType="checkout" | stats count by threadId, sessionId, orderId, invoice, paymentId, result
You cannot use a per-result alert type for this alert as it is transactional in nature, grouping events together over a time period. Instead, a rolling-window alert type should be used.