Anomaly Detection Overview

Anomaly detection is the process of identifying data points that are unusual or unexpected. Regular events of the CPU, memory, swap memory, disk, etc., are normal to operations, but if there is any “application down” or firewall event, then it represents an unusual scenario. The goal of anomaly detection is to identify such unusual scenarios (what we call outliers) within a dataset that differ significantly from other observations. Though the task of detecting an anomaly can be performed by all three types of machine learning algorithms, its implementation is done extensively on unlabeled data by clustering them into various groups. In IT operations, the execution of a service improvement plan is a regular exercise, and operations do not have a specific target to predict. Rather, they need to analyze lots of data and then try to observe similarities and club together them to form different groups to understand anomalies and formulate recommendations. We briefly discussed different clustering algorithms available in unsupervised machine learning in Chapter 5 for anomaly detection, and K-means clustering is one of the simplest and most popular unsupervised machine learning algorithms that we will be discovering in this chapter.

K-Means Algorithms

As discussed in Chapter 5, K-means is a centroid-based clustering algorithm that is extensively used in data mining to detect patterns and group together similar data points into a cluster. Mathematically, if there is a dataset {x1, . . . , xn} consisting of N data points, as shown in Figure 8-1, then our goal is to partition the dataset into some number K of clusters.

A scatterplot of distinct clusters that can be created around centroids. There are ways to determine the centroids from the dataset. In the scatterplot horizontal access values are 0, 200, 400, 600, 800 and 1000, and the vertical access values are 0, 5000, 10000, 15000, 20000 and 25000.

Here, K represent the number of distinct clusters that can be created around centroids. There are ways to determine the centroids from the dataset, and this section will use the Elbow method in our implementation to determine the appropriate number of clusters. There are three centroids detected in the dataset that are marked as red dots in Figure 8-2.

The K-means algorithm tries to determine the similarity of data points with centroids and accordingly cluster the data points around these centroids, as shown in Figure 8-2.

A scatterplot determine the similarity of data points with centroids and accordingly cluster the data points around these centroids. The scatterplot horizontal access values are 0, 200, 400, 600, 800 and 1000, and the vertical access values are 0, 5000, 10000, 15000, 20000 and 25000.

From an AIOps perspective, it is probably the simplest clustering algorithm that can detect anomalies in a wide range of scenarios. In this algorithm, first a centroid is chosen for every cluster, and then data points are grouped into the cluster based on their distance from centroids. There are multiple methods to calculate distance such as Minkowski, Manhattan, Euclidean, Hamming, etc.

Clustering is an iterative process to optimize the position of centroids with which the distance is getting calculated. For example, the majority of organizations receive thousands of events related to performance or availability, but rarely do they receive security alerts related to DDoS/DoS attacks. Not only the frequency but the alarm text message will differ. In this case, you can apply K-means clustering on event messages that group related messages over a time scale, leaving aside security alerts as an anomaly.

Let’s consider an implementation scenario where there are multiple alerts in the system and you want to analyze alert messages to segregate frequently occurring events as well as anomalies and then try to determine the root cause of events. For this implementation, you will work with text data; hence, you have to use the subdomain of AI called natural language processing (NLP). You will be using NLP to analyze the text by tokenizing it and getting the relevance of tokens and their similarities in different events to create clusters.

As shown in Figure 8-3, we have 1,050 events (the first row is a header), each with six columns.

This is K-mean algorithms dataset. In that image the headers are Column, Non-Null Count and Data type. Where at the top class pandas dot core dot frame dot data Frame, Range Index 1051 entries, 0 to 1050. There are 6 columns in total, Source, Alert Time, Alert Description, Alert Class, Alert Type and Alert Manager.

As shown in Figure 8-4, the input dataset contains six different event classes that represent performance and availability issues related to the operating system, network, applications, and configuration changes. These cover most common types of events that get generated in any operations.

This is K-mean algorithms unique values. The values are ,array parenthesis box bracket operating system, Network Performance, Transaction Monitoring, Network Availability, MERAKI, Configuration, Box Bracket, Datatype is equal to object parenthesis.

Next, start applying the NLP algorithm. First, we need to download the English words dictionary from nltk. You need to add domain-specific words as that may be missing from the English words dictionary like app, HTTP, CPU, etc. These domain-specific words are important for efficiently creating clusters.

For each event sentence, we have the relevant tokens for analysis. Now we need to map these text tokens from the vocabulary to a corresponding vector of real numbers so that we can apply statistical algorithms. This process is called vectorization.

Readers can get more details on TF-IDF at https://en.wikipedia.org/wiki/Tf–idf.

Now we have the final list of tokens on which we can apply the K-means algorithm.

To apply the K-means algorithm, first we need to determine what will be the ideal value of K. For this we will be using the Elbow method, which is a heuristic used in determining the number of clusters in a dataset.

From the graph in Figure 8-5 we know that total seven clusters are possible in our dataset. So, let’s set the value of K as 7 and execute the K-means algorithm.

A elbow graph of score versus number of clusters. The Numbers of Clusters from 3 to 11, and the scores from negative 250 to 0. The line goes up and to the right from (3, negative 250) to (7, negative 50), then goes to the right till (11, negative 25). All values are approximated.

Now you can see that all the top keywords representing issues are clustered in different clusters.

Figure 8-6 shows that all types of alerts as features get clustered in cluster 0 and their significance or weightage in it. As observed, this cluster primarily consists of CPU-related alerts or issues that occurred with other less significant alerts or issues. As observed from this cluster, CPU utilization alerts are not related to any application-related issues or alerts, which indicates a strong probability that CPU alerts generate a lot of noise, and you may need to tune the CPU utilization parameter.

A horizontal bar graph represents C P U issues. The vertical axis denotes features and the horizontal axis denotes scores. The features are C P U, critic, util, state, w w w, failur, emp t s m, employee Im, emp record and environ. The values of the respective bars across the features are 0.62, 0.59, 0.36, 0.36, 0, 0, 0, 0, 0, and 0 approximate.

Cluster 1, as shown in Figure 8-7, represents issues that are primarily related to the application Global HR. Also, this cluster indicates a potential relationship between the Global HR application issue and a memory issue and will help the IT operations teams in the troubleshooting and remediation process. This is quite an important insight that gets automatically detected by ML algorithms without writing any rule.

A horizontal bar graph represents global H R application issues. The horizontal axis denotes scores and vertical axis denotes features. The features are global h r, warn, memori, util, state, w w w, failur, em, emp t s m and employee I m. The value of the respective bars across the features are 0.7, 0.45, 0.45, 0.25, and 0.25 approximately.

Similar to cluster 1, Figure 8-8 represents issues that are primarily related to the other application, Payroll, which also gets impacted due to memory-related issues. Alerts in this cluster should be used by the application team to understand how to improve its performance and reduce application alerts.

A horizontal bar graph represents payroll application issues. The vertical axis denotes features and the horizontal axis denotes scores. The features are pyrol, warn, memori, util, state, www, failur, em, emp t s m, employee I m. The values of the respective bars across the features are 0.71, 0.42, 0.42, 0.25, 0.25, 0, 0, 0, 0, and 0 approximately.

Cluster 3 in Figure 8-9 represents alerts related to the application Employee Record, which needs to be analyzed and fixed. This is also an important input to the problem management team because they need to provide a long-term resolution for this issue.

A horizontal bar graph of employee record application issues. The vertical axis denotes features and the horizontal axis denotes scores. The features are emprecord, warn, memori, util, state, www, failur, em, emp t s m and employee I m. The values of the respective bars across the features are 0.71, 0.42, 0.42, 0.25, and 0.25 approximately.

Cluster 4 in Figure 8-10 represents issues related to network interface utilization that needs to be analyzed by the network team. AIOps can use automation to create a single ticket for the network team against all the alerts that are clustered here. This single ticket will avoid overloading the ticketing system with duplicate incidents.

A horizontal bar graph of network interface issues. The vertical axis denotes features and the horizontal axis denotes score. The features are Interfac, warm, util, state, w w w, failur, em, emp t s m, employee I m and emprecord. The values of the bars across the features are 0.81, 0.42, 0.25, 0.25, 0, 0, 0, 0, 0, and 0 approximately.

Cluster 5 in Figure 8-11 represents memory-related issues, and we have also observed in previous clusters that memory-related alerts are clustered along with application alerts. This indicates that multiple applications might be using common infrastructure such as a common virtualized infrastructure, which is nearing capacity. Since different clusters are capturing different issues automatically, analyzing them together significantly improves the resolution time.

A horizontal bar graph of memory utilization issues. The vertical axis denotes features and the horizontal axis denotes score. The features are memori, critic, util, state, www, font, emp tsm, employee Im, emprecord and environ. The values of the bar across the vertical axis are 0.61, 0.58, 0.35, 0.35, 0, 0, 0, 0, 0, and 0 approximately.

Cluster 6 in Figure 8-12 is most interesting as it detected a potential outage. As shown in Figure 8-10, cluster 6 consists of various simulations, transactions, and application-related alerts that indicate some issue at the infrastructure level and not necessarily at the application level. With traditional IT ops methods, finding the root cause in such data would have been difficult, but the ML model created this cluster, which detected a potential relationship and pointed toward the root cause.

A horizontal bar graph of a potential outage. The vertical axis denotes features and the horizontal axis denotes scores. The features are simul, session, client, page, transact, affect, web, payroll sap g t, applic and app. The values of the bars across the vertical axis are 0.26, 0.19, 0.19, 0.9, 0.9, 0.9, 0.9, 0.8, 0.6 and 0.6 approximately.

As shown in Figure 8-13, there are 43 events that got clustered together in cluster 6.

This is K-mean algorithms dataset. In that image the headers are column, Non-Null count and data type. Where at the top class pandas dot core dot frame dot Data Frame. Int 64 Index 43 entries, from 31 to 1050. There are 7 columns in total. Source, Alert Time, Alert Description, Alert Class, Alert Type, Alert Manager and Cluster.

On closely reviewing the events in cluster 6, you can see there are five events from different transaction failures that arrived on July 8 almost at the same time, about 19:30, indicating an outage. As shown in Figure 8-14, one out of five events belongs to a network interface being down, indicating it as a probable cause because immediately after this event, there are events for failed simulated transactions for the application Global HR.

A table has four columns and seven rows, columns heading are Alert Time, Alert Class, Alert Type and Alert Description. In this tables column five rows are marked by a red box, which are 706, 707, 708, 709 and 710.

With the help of ML and NLP capabilities, the algorithm discovered and clustered useful information from 1,000+ events, and that was without writing any static rules or using any topology-related details.

Let’s also analyze these clusters together over a time scale to uncover more insights.

From Figure 8-15, there are few important observations you can make here:

• There is a consistent lot of noise due to CPU utilization events that are represented in cluster 0. This is feedback to tune the thresholds of the CPU monitoring parameter. Or you can submit a recommendation to the capacity planning team to increase the CPU capacity.

• The algorithm automatically created three clusters, one for each application, which primarily contains alerts related to that specific application only. These clusters can be analyzed for incident or problem management–related tasks. These clusters provide a lot of visibility to the applications teams and are immensely helpful in executing service improvement programs.

This is K-mean algorithms timeline analysis. In that image horizontal access denotes Alert time and vertical access denotes Cluster numbers. The cluster number which are from 0 to 6, and the alert times from 2021-06-01 to 2021-08-01.

This AIOps use case of anomaly detection also makes it extremely helpful and efficient to detect potentially critical events, such as security events, from the enormous pile of events and initiate automated actions, such as invocating a kill switch to minimize the impact. You have seen how K-means clustering detected noise and can generate recommendations for problem and capacity management processes.

Along with anomaly detection, another common use case of AIOps is correlation and association, which will be discussed next.

Correlation and Association

Correlation is a statistical term that means identifying any relationship or association between two or more entities. From an AIOps perspective, correlation is used to determine any dependency or relation between entities and cluster them together for efficient analysis. We covered multiple regression algorithms that determine the relationship between multiple entities to establish correlation in Chapter 5.

However, establishing correlation at the event layer is a bit difficult as multiple events from varied sources come at a varied time interval. A time-based sequence or pattern of event is quite rare at the event layer. One of the algorithms that works efficiently at the event layer to determine correlation and association is DBSCAN, and it is particularly useful in scenarios where data points are close to each other (dense) along with the presence of a considerable amount of noise or outliers.

Topology-Based Correlation

The accuracy of clustering is one of the most important KPIs of an event management system that drives the efficiency of the engine. To improve the accuracy of clustering and the determination of the root cause, you need to provide some topology context. But the CMDB and discovery are challenges in themselves, and having a 100 percent accurate CMDB with 100 percent discovery and modeling of infrastructure is a tough task.

Considering the various challenges with the CMDB and discovery, topology correlation can still be implemented for network topology correlation as well as for application topology correlation. Consider the sample topology shown in Figure 8-16. Connections marked in black represent the network topology, whereas connections marked in red represent the application topology. We will use this sample topology to discuss correlation for network topology and application topology.

A chart has one main branch and ten sub branches. The main branch is network switch -1, In that image the red arrow denotes application Topology and the black arrow denote Network Topology.

Summary

This chapter covered the important use cases in AIOps around anomaly detection using K-means clustering. You also used NLP techniques such as stopword removal and TF-IDF to make sense of textual data. You learned about other techniques such as application dependency mapping and the CMDB and how they are relevant in AIOps. In the next chapter, we will cover how to set up AIOps as a practice in an organization.