The Splunk dashboard can be configured to implement custom alerting actions. Splunk can be integrated with other enterprise applications to automate the workflow and thus increase efficiency.
Splunk is a big data tool with lots of data streaming in every second from numerous different sources, and there can be multiple dashboards and use case scenarios implemented on Splunk. It would be difficult to keep monitoring all the dashboards for any issue when it arises in real time. Hence, whenever any specific condition is met, an alert can be sent in the form of an e-mail, SMS, or chat notification informing the admin that the defined condition has been found on the data.
Thus, the feature of alerting in Splunk helps to avoid monitoring of the dashboard live and 24 x 7 for issues and thus reduces manpower. Since the process is automated, there will not be a single instance when the specific issue is caused and not reported. Thus, the alerting feature increases the efficiency of data monitoring in real time.
The alerting feature can be useful in various business domains, such as security, compliance, fraud, IT operations, IOT, M2M, business analytics, and so on. The alerting system can be integrated with the issue tracking and ticketing system, messaging, e-mails and chatting applications, running custom scripts to switch on/off devices, managing and monitoring IOT devices, and so on. Thus, the alerting feature of Splunk, when integrated with the enterprise application's real-time alerting and workflow, helps us obtain automation.
The following are some of the alert action examples that can be easily integrated with Splunk:
- Issue/incident tracking/ticketing: Most widely used incident ticketing systems such as Jira and ServiceNow can be directly integrated with Splunk to automate the creation of tickets from Splunk itself.
- Security, compliance, and fraud: Security and networking devices, such as firewalls and gateways, can be informed to take the necessary action. The fraud and compliance team can be informed to take the necessary action in case of any fraud detected by Splunk.
- Alert notification: Whenever any specified condition is met in Splunk, various alert notifications can be sent either via e-mails, SMSes, or via instant messaging clients, such as Slack and HipChat.
- IT monitoring: Incident notifications can be sent to various IT incident monitoring tools, such as BigPanda, xMatters, and so on, directly from Splunk whenever an issue arises.
- IOT/M2M actions: IOT/M2M devices can be switched off/on and alarm generation or any custom action by the device can be configured via scripts. Custom actions can also be performed using webhook (sending HTTP POST actions to URLs).
- Custom actions: Custom scripts or integration with any enterprise applications can help the alerting feature of Splunk to trigger any custom actions as per the requirement.
Splunk already had an alerting feature in the previous versions, which were capable of sending e-mail notifications whenever the specified condition was met. In Splunk 6.3 Enterprise, the custom alert action feature has been introduced, which makes Splunk alerts trigger an automate workflow in enterprise applications.
The following are some of the features of the custom alert action introduced in Splunk 6.3:
- Various third-party applications/add-ons directly integrate with enterprise applications
- UI-based configuration of settings and administration options can be set directly from the Splunk Web console
- Various enterprise-applications-based extensions can be developed and distributed to Splunk users for integration with Splunk
- Enterprise application add-ons/extensions are already available for the famous, most widely used alerting, incident management tools and thus can be directly integrated and used
Splunk supports the following custom alert framework to use and manage custom alert actions:
- Custom alert actions will be made available as add-ons or extensions that can be directly used by configuring on Splunk.
- Configurations can either be done by a UI from the Splunk Web console or via the respective conf files. Features such as access control and add-on management can also be enabled for user configuration.
- We can explicitly invoke scripts (Python scripts and bash scripts) and can even pass information via tokens to the scripts.
- The configuration framework supports encrypting confidential information with the access method to read/write from the alert script.
- Splunk provides lots of enterprise application support for custom alert actions, such as BigPanda, ServiceNow, xMatters, HipChat, Slack, Hue Bulbs, Alert Manager, Insteon Home Automation Control, Jira, Yammer, and so on.
- The inbuilt plugins can be configured from the Splunk Web console by navigating to Settings | Alert Actions.
The Alert Actions page will be visible with a list of installed alert actions along with the option to install add-ons/extensions from the Splunk app store. The following figure shows the Alert Actions page of the Splunk Web console:
The Custom Alert page accessed via the Splunk Web console can be used to configure and manage alert actions that are already installed or new alert actions can be installed from the Splunk app store. The alert action panel also provides access to usage statistics and log events.
Let's see how to create and use a custom alert action on the Splunk Web console. Follow these steps to do so:
- Write the search query on whose output the custom
Alertaction is to be defined on the Splunk search. Click on the Save As button and then on Alert:
- The Save As Alert screen pops up, similar to what is shown in the following figure, where various information needs to be configured. The first segment of Save As Alert is Settings, as shown in the following figure:
- The title and description of the alert can be specified as per the user to identify the alert
- The access control option describes whether the alert is private or shared in an app and can be configured from here
- The alert schedule describes whether a schedule is to be run every day, every hour, every month, or in real time, and can be configured from this section
- The next section is Trigger Conditions. Here, when the preceding alert runs and the output is generated, then the condition can be specified in the trigger condition to do custom alert actions:
The alert condition can be configured by choosing one of the inbuilt options or a custom option can be selected where a user can specify the condition. Once this condition is satisfied, the configured custom alert action will be invoked.
There is also an option to throttle the trigger, which means that if a trigger condition is met, then for the specified time, the alert trigger will be suppressed to avoid duplicate triggers. The throttle condition can be enabled/disabled as per the requirement from the preceding settings.
- Since we have configured the alert search, the trigger condition will configure the custom alert action. The alert action is the action that will be performed when the alert triggered condition is met. The last section of Save As Alert is Trigger Actions:
When we click on the Add Actions button from the Trigger Actions section, another pop up appears listing all the custom alert action add-ons/plugins installed in the Splunk instance:
- The list of custom alert actions is available when we click on Add Actions, and then the user can choose one or more than one alert actions from the list. Each alert action selected can be configured from the same screen, and click on Save to create a custom alert action. The following screenshot shows three alert actions configured for an alert. Similarly, as per the required custom alert, actions can be chosen and configured:
Let's understand the use of the custom alert action with the help of a real-world example. Let's suppose that we have data from various payment gateways for credit cards on Splunk. Every second, millions of transactions take place from various sources, such as e-commerce portals, mobile devices, ATM/PoS transactions, and the log information from these sources is made available on Splunk.
Taking advantage of Splunk's custom alert action, the following are some of the use cases that can be implemented on the said data:
- Fraud detection: Fraud detection can be configured either on the machine-learning logic or correlation or outlier detection. Once the fraud detection search is ready, a custom alert can be defined whenever any such condition is met. The alert can be used to send an instant SMS or e-mail to the respective people. Run a custom script to inform the payment gateway in order to abort the transaction, and so on. Thus, the necessary action can be taken as required to avoid any loss to the business.
- Automatic incident tracking: Whenever the transaction fails due to any reason, such as an incorrect password, failure due to any hardware, network failure, and so on, the automatic incident will be logged in the incident tracking system to automate the fixing of the concerned issue.
Similarly, depending on the requirement and use case, a custom alert action can be defined, and the automation of workflow is achieved.