Linear Regression

This algorithm is applicable where the dataset shows a linear relationship between input and output variables and the output variable has continuous values, such as percentage utilization of CPU, etc.

In linear regression, input variables, referred to as independent variables or features, from the dataset get chosen to predict the values of the output variable, referred to as the dependent variable or target value. Mathematically, the linear regression model establishes a relationship between the dependent variable (Y) and one or more independent variables (Xi) using a best-fit straight line (called the regression line) and is represented by the equation Y = a + bXi + e, where a is the intercept, b is the slope of the line, and e is the error in prediction from the actual value. When there are multiple independent variables, it’s called multiple linear regression.

Let’s first understand simple linear regression that has only one independent variable.

In Figure 7-2’s graph, there are a set of actual data points that exhibit a linear relationship, and an imaginary straight line runs somewhere in the middle of these data points.

A graph of dependent variable y versus independent variable X. Eight vertical lines, regression line with a dot on tip denotes actual data, joins at the top and bottom of the inclined line between y-intercept a, and y equals a plus bX plus epsilon. Two horizontal and vertical lines, delta X and delta y make a triangle with the regression line.

This imaginary straight line is the regression line, which consists of predicted data points. Now the distance between each one of the actual data points, and the line is called an error. Multiple lines can be drawn that pass through these data points, so the algorithm finds the best line that minimizes this error by calculating distances with all points. Whichever lines gives the smallest error becomes the regression line and provides the best prediction. From an AIOps perspective, linear regression provides a predictive capability for correlation, but it does not show causation.

Now let’s understand the implementation of linear regression to forecast the database response time (in seconds) based on the memory utilization (MB) and CPU utilization (percent) of the server. Here we have two independent variables, namely, CPU and memory, so we will be using multiple linear regression models to predict the database response time based on the specific utilization value of CPU and memory.

You can download the code from https://github.com/dryice-devops/AIOps/blob/main/Ch-7_AutomatedBaselinig-Regression.ipynb.

Figure 7-3 shows the sample values for CPU and memory utilization and the corresponding database response times.

A table for sample values for C P U and memory has three columns with headers D B response time, C P U usage, and memory utilization M B.

As shown in Figure 7-4, there are a total of 24 rows in the dataset providing values for each of the three independent and dependent variables.

Out open box brackets 3 close square bracket colon open parentheses 24 comma 3 close parentheses.

As per the graph in Figure 7-5, there exists a linear relationship between CPU Utilization and DB Response Time.

A graph of C P U usage versus D B response time, where a line with scatter plots is graphed. The scatter plot has a tight pattern with an upward slope on either side of the line between (2.2, 6.8) and (6.5, 11) approximately. The vertical and horizontal bars with a curve line are graphed on the top and right windows, respectively.

Figure 7-6 also shows that a linear relationship exists between DB Response Time and Memory Usage, which satisfies the condition of applying the linear regression algorithm.

A graph of memory utilization M B versus D B response time, where a line with scatter plots is graphed. The scatter plot has a tight pattern with an upward slope on either side of the line between (2.2, 980) and (6.5, 1800) approximately. The vertical and horizontal bars with a curve line are graphed on the top and right windows, respectively.

Based on the CPU and Memory Utilization values present in the test data, our model has forecasted DB Response Time Values, as shown in the output in Figure 7-7.

Out open box bracket 8 close box bracket colon array open parenthesis open box bracket 3.60063185, 5.02933512, 2.49967556, 5.33846418, 3.92287079 close box bracket close parenthesis.

As observed in the output in Figure 7-8, predicted values are very close to the actual observed value. This indicates that the model is performing well.

A double bar graph of predicted versus actual, where the vertical axis represents D B response time, and the horizontal axis represents the test dataset. The values across the test data set are as follows. Predicted: 3.6, 5, 2.3, 5.2 and 3.9. Actual: 3.3, 4.9, 2, 5.4, and 4. All the values are approximate.

As per the output, the r2 score value is 0.9679, which means that the model is predicting the database response time with around 97 percent accuracy for a given scenario of CPU and memory utilization, which is quite good. Based on the complexity and requirements, this model can be further expanded to include other independent variables such as the number of transactions, user connections, available disk space, etc., which makes this model production usable to perform capacity planning, upgrades, or migration-related tasks to improve DB response time, which will eventually improve the application performance and user experience.

There are a few limitations with linear regression model where it performs correlation between the dependent variable with one or more independent variables to perform predictions. The first big challenge is that correlations between variables are independent of any time series associated with them. Linear regression cannot predict the value of the dependent variable at a “specific time” with given values of independent variables. Second, prediction is entirely based on the independent variables without considering values of the dependent variable that were already predicted in the recent past. To overcome such challenges, you can use the time-series model, which we will discuss in the next section.

Time-Series Data

Every time-series data will have a date or time column along with one or more data columns. If there is only one data column, then it’s called a univariate time series, and if there are multiple data columns, then it’s called a multivariate time series. Figure 7-9 is the example of a univariate time series with the daily average CPU utilization of a server.

A table of C P U utilization time series has two columns titled time stamp and C P U utilization Y.

For the purpose of analysis, a univariate time series may need to be converted into multivariate by adding some additional supporting columns. With the help of TimeStamp, we have added another column, Day of the week, and from Figure 7-10 we can observe that there is an unusual spike in CPU utilization on weekends as compared to weekdays.

A table of time series analysis of C P U utilization has three columns titled time stamp, C P U utilization, and day of the week. The fourth and fifth rows are highlighted.

Stationary Time Series

One of the most important statistical properties of a time series is stationarity, where the mean and the variance of the time series do not change over time, as shown in Figure 7-11. In simple terms, a stationary time series can have different values with different timestamps, but the underlying logic or method to generate those values remains constant and does not change.

A graph of stationary time series has a high-frequency wave graphed on a horizontal line between two parallel dashed lines.

Lag Variable

A lag variable is a dependent variable that is lagged in over time. Lag-1 (Y_t-1) represents the value of dependent variable Y_t at previous one unit of time, Lag 2 (Y_t-2) will have value of Y_t at previous two units of time, and so on.

Consider the earlier example of time series on CPU utilization at specific timestamps. In this example, Lag-1 (Y_t-1) at 7/3/20 2:00 PM will be the value of Y_t at 7/2/20 2:00 PM, which is 8.82, and Lag 2 (Y_t-2) will have value of Y_t at 7/1/20 2:00 PM, which is 8.33, as shown in Figure 7-12.

A table of lag variables has four columns titled time stamp, C P U utilization Y subscript t, Lag 1 Y subscript t minus 1, and Lag 2 Y subscript t minus 2.

Mathematically if there exists a relationship or correlation between Y_t and its lagged values Y_t-1, Y_t-2, etc., then a model can be created to detect the pattern and predict future values of the same series using its lagged values. For example, assume that CPU utilization increases around 100 percent on every Saturday; then the value at Lag-7 (Yt-7) can be considered for prediction.

The next step is to determine the strength of the relationship between Y_t and its lags and how many lags have statistically significant relationship with Y_t that we can use in the prediction model. That’s where ACF and PACF plots help, and we will be discussing these next.

ACF and PACF

Both ACF and PACF are statistical methods used by researchers to understand the temporal dynamics of an individual time series, which basically means detecting correlation over time, often called autocorrelation. A detailed discussion on ACF and PACF is beyond scope of this book, so we will be limiting the discussion to the basics of ACF and PACF to understand how to use them in the AIOps model development.

ACF is a statistical term that represents the autocorrelation function and is used to measure the correlation of time-series values Y_t with its own lags Y_t-1, Y_t-2, Y_t-3, etc. This correlation of various lags can be visualized using an ACF plot on a scale of 1 to -1, where 1 represents a perfect positive relationship and -1 represents a perfect negative relationship.

As shown in Figure 7-13, the y-axis represents the correlation value, and the x-axis represents the lags. The first value of the series will always be 100 percent correlated with itself, and that’s why in the ACF plot the Lag 0 correlation value is going until 1. The ACF plot is useful in detecting the pattern in the correlated data.

An A C F plot where the y-axis represents the correlation value and the x-axis represents the lags from 0 to 7. The dashed lines on either side of the x-axis indicate significance. The first value of the series starts from 1 and decreases gradually with vales 8, 7, 5, negative 4, negative 5, negative 7, and negative 8, approximately.

There is a red line shown on both sides in the ACF plot called the significance line, which is creating a range or band referred to as the error band. The lags that cross this line have a statistically significant relationship with variable Y. In a sample ACF plot, three lags are crossing this line, so up to Lag 3, there is a statistically significant relationship, and anything within this band is not statistically significant.

PACF is another statistical term that represents the partial autocorrelation function, which is similar to an ACF plot except that it considers the strength of the relationship between the variable and lag at a specific time after removing the effects of all intermediate lags.

For example, in the sample ACF plot in Figure 7-14, it is visible that there is a significant correlation up to three lags, but the PACF plot tells how much the strength of correlation of Y_t-3 alone with Y_t, is after removing any impact of Y_t-1 and Y_t-2. PACF can be considered as more of a filtered value of ACF.

The P A C F plot where the y-axis represents the correlation value and the x-axis represents the lags from 0 to 7. The dashed lines on either side of the x-axis indicate significance. The first value of the series starts from 1 and decreases gradually with vales 6, 3, 2, negative 2, negative 2, negative 4.5, and negative 8, approximately.

Now we are all set to implement two most commonly used time-series models, ARIMA and SARIMA, for noise reduction by predicting the appropriate monitoring threshold of parameters. This is one of the most common use cases in an AIOps implementation.

ARIMA

ARIMA is one of the foundational time-series forecasting techniques; it is popular and has been extensively used for quite a while now. This concept was first introduced by George Box and Gwilym Jenkins in 1976 in a book titled Time Series Analysis: Forecasting and Control. They defined a systematic method called the Box: Jenkins Analysis to identify, fit, and use an autoregressive (AR) technique integrated (I) with a moving average (MA) technique resulting in an ARIMA time-series model.

Model Development

The ARIMA model has three components.

Autoregression or AR (p)

This component, known as autoregression, shows that the dependent variable Y_t can be calculated as a function of its own lags Y_t-1, Y_t-2, and so on.

Y subscript t equals alpha plus beta subscript 1 Y subscript t minus 1 plus beta subscript 2 Y subscript t minus 2 plus second derivative plus beta subscript p Y subscript t minus p.Here,

α is the constant term (intercept).

β is the coefficient of lag with a subscript indicating its position.

ε is the error or white noise.

We already discussed that the PACF plot shows the partial autocorrelation between the series and its lag after excluding the effect from the intermediate lags. We can use the PACF plot to determine the last lag p, which cuts off the significant line and can be considered as a good estimate value of AR. This value is represented by the model variable p as an order of the AR model.

Based on the sample PACF plot in Figure 7-15, we can observe that lag 1 seems to be the last lag crossing a significant line. So, we can set p as 1 and represent it as AR(1).

The P A C F plot where the y-axis represents the correlation value and the x-axis represents the lags from 0 to 7. The dashed lines on either side of the x-axis indicate significance. The first value of the series starts from 1 and decreases gradually with vales 6, 4, 3.5, negative 2, negative 3, negative 4, and negative 8, approximately.

Moving Average or MA (q)

This component, known as moving average, shows that the dependent variable Y_t can be calculated as a function of the weighted average of the forecast errors in previous lags.Y subscript t equals alpha plus epsilon subscript t plus phi subscript 1 epsilon subscript t minus 1 plus phi subscript 2 epsilon subscript t minus 2 plus second derivative plus phi subscript q epsilon subscript t minus q.

α is usually the mean of the series.

ε is the error from the autoregressive models at respective lags.

MA can detect trends and patterns in time-series data. You can use the ACF plots to determine the value of model variable q, which represents how many moving averages are required to remove any autocorrelation in a stationary series. From the ACF plot in Figure 7-16, we can observe that there is cut-off after lags 2 and 6. So, we can try ACF with q =2 , MA (2) or q=6, MA(6).

An A C F plot where the y-axis represents the correlation value and the x-axis represents the lags from 0 to 7. The dashed lines on either side of the x-axis indicate significance. The first value of the series starts from 1 and decreases gradually with vales 8, 7, 5, negative 4, negative 5, negative 7, and negative 8, approximately.

Both AR and MA equations needs to be integrated (I or d) to create the ARIMA model, which is represented as ARIMA (p,d,q).

ARIMA acts as a foundation for various other autoregressive time-series models as follows:

• SARIMA: If there is a repetitive pattern at a regular time interval, then it’s referred to as seasonality, and the SARIMA model is used where S stands for seasonality. We will explore this more in the next section.

• ARIMAX: If the series depends on external factors, then the ARIMAX model is used, where X stands for external factors.

• SARIMAX: If along with seasonality there is a strong influence of external factors, then the SARIMAX model is used.

One of the big limitations of ARIMA is the inability to consider the seasonality in time-series data, which is common in real-world scenarios like when there is exceptionally high sales figures during the holiday period of Thanksgiving and Christmas every year, high expenditure during the first week of every month, high footprints during the weekend in shopping malls, and so on. Such seasonality gets considered in another algorithm, called SARIMA, which is a variant of ARIMA model.

SARIMA

A cyclical pattern at a regular time interval is referred as seasonality and should be considered in a prediction model to improve the accuracy of prediction. That’s how the ARIMA model is modified to develop SARIMA, which considers seasonality in time-series data and supports univariate time-series data with a seasonal component.

The final notation for an SARIMA model is specified as SARIMA (p,d,q)(P,D,Q)m.

Implementation of ARIMA and SARIMA

Let’s perform a sample implementation of the ARIMA and SARIMA models on CPU utilization data to predict its thresholds.

As shown in Figure 7-17, there are two columns, date_time and cpu_utilization, and a total of 396 data points (first row is the header) are available for analysis.

A model description data file with header Open chevron class pandas dot core dot frame dot data frame close chevron, where range index is 397 entries, 0 to 396, and data columns has two columns titles non null count and dtype. Dtypes has float64 1, and object 1, and memory usage of 6.3 plus K B.

Set the data type of column date_time to DateTime, set it as an index to sort the data points in a data frame, and plot the CPU utilization on a time scale, as shown in Figure 7-18.

total_records = cpu_util_raw_data.count()['cpu_utilization']

cpu_util_raw_data['date_time'] = pd.to_datetime(cpu_util_raw_data['date_time'])

cpu_util_raw_data.set_index('date_time',inplace=True)

cpu_util_raw_data = cpu_util_raw_data.sort_values(by="date_time")

fig = plt.figure(figsize =(10, 5))

cpu_util_raw_data['cpu_utilization'].plot()

A graph of C P U utilization implementation, where the vertical axis is marked as data points and the horizontal axis is marked as date underscore time. A high-frequency wave is graphed from point 4 and ends at 55 approximately. The high-frequency wave has the highest point at 88 and the lowest point at 0.

Based on Figure 7-19, there are no outliers and no “zero” values, so let’s continue with the given dataset.

A box plot is graphed for outer line analysis. The vertical axis has points, and the horizontal axis denotes C P U utilization. The interquartile range is between points 19 to 59, where the median is at point 40. The minimum outlier is at point 1, and the maximum outlier is at point 93.

The A D F result implementation has two series. Series 1, A D F statistic: negative 2.5335679, p-value: 0.107493, A D F test result: the time series is non-stationary. Series 2, A D F statistic: negative 15.396157, p-value: 0.000000, A D F test result: the time series is stationary. A remark is below, differencing order to make data stationery: 1.

The ADF result on differenced time series shows that the p-value is 0, which confirms that data series is now stationary. This can be visualized by plotting both the original series and a differenced series on a time scale graph, as shown in Figure 7-20.

fig, ax = plt.subplots(figsize=(10,8), dpi=100)

#  Differencing

ax.plot(cpu_util_raw_data.cpu_utilization[:], label='Original Series')

ax.plot(cpu_util_raw_data.cpu_utilization.diff(1), label='1st Order Differencing')

ax.set_title('1st Order Differencing')

ax.legend(loc='upper left', fontsize=10)

plt.legend(loc='upper left', fontsize=10)

plt.suptitle('CPU Usage - Time Series Dataset', fontsize=16)

plt.show()

A graph of time-series dataset of C P U usage, where the vertical axis is marked as data points and the horizontal axis is marked as date underscore time. Two high-frequency waves of original series and 1st order differencing are graphed from point 0 in January 2020 and end at points 50 in January 2021 and 0 in January 2021, respectively.

As we have done differencing only one time, we can set the value of the differencing (d) component of ARIMA as d=1.

In the PACF graph in Figure 7-21, it can be observed that until lag 6 it is crossing the significant line, so the AR component, p=6, can be initially set for modeling.

A P A C F graph correlation value versus lags from 0 to 25. The shaded bar on the horizontal axis indicates significance. The values of the series that cross the significance bar are 1, negative 0.35, negative 0.22, negative 0.30, negative 0.20, negative 0.13, negative 0.23, negative 0.12, negative 0.11, and negative 0.14 approximately.

In the ACF graph in Figure 7-22, there is sharp cut-off on the lag 2 and lag 7 crossing significance line, so we can try with the MA component as q=2 or q =7 for modeling.

An A C F graph correlation value versus lags from 0 to 25. The shaded bar on the horizontal axis indicates significance. The series values that cross the significance bar are 1, negative 0.38, negative 0.15, 0.20, negative 0.11, 0.11, and negative 0.15 approximately.

Next split the data into training and test data to be used for the ARIMA model (6,1,2). Out of 396 data points, let’s use the first 300 (~75 percent) data points for training and the rest of the data points for testing purposes and plot them on a time scale, as shown in Figure 7-23.

# Plot train and test data

train_data = cpu_util_raw_data['cpu_utilization'][:300]

test_data = cpu_util_raw_data['cpu_utilization'][300:]

fig, ax = plt.subplots(figsize=(10,5), dpi=100)

ax.plot(train_data, label='Train Data Points')

ax.plot(test_data, label='Test Data Points')

ax.set_title('CPU Usage Timeline')

ax.legend(loc='upper left', fontsize=10)

plt.legend(loc='upper left', fontsize=10)

plt.suptitle('Test-Train Data Splits', fontsize=16)

plt.show()

A graph of test train data splits of C P U usage, where the vertical axis is marked as data points and the horizontal axis is marked as date underscore time. Two high-frequency waves of train data points and test data points start from 4 in January 2020 and November 2020 and end at points 70 in November 2020 and 50 in January 2021, respectively.

Figure 7-24 shows the output from the ARIMA model on the train data.

A report of SARIMAX results has several outputs with headers, depanneur variable, model, date, time, sample, covariance type, number of observations, log-likelihood, AIC, BIC, and HQIC in the top. A table is in the middle and in the bottom, there are more outputs with headers, Ljung box, prob, heteroskedasticity, prob, and Jaraque-Bera.

Based on PACF and ACF charts, multiple values can be tried to lower the AIC and BIC values. For now let’s proceed with ARIMA (6,1,2).

As observed in Figure 7-25, the model predictions are quite close to actual values, indicating that it gets trained quite well.

A graph of actual versus predicted C P U utilization on train data, where the vertical axis denotes data points, and the horizontal axis denotes date underscore time. Two high-frequency waves of prediction and training data are graphed. Both waves start at point 70 in January 2020 and end at point 60 in January 2021.

Figure 7-26 shows that the ARIMA (6,1,2) model performance on the test dataset is not good and its predictions are deteriorating over time.

A graph of prediction analysis, data points versus date underscore time. Three high-frequency waves of training data, actual data, and forecast data start from 70 in January 2020, 55 in November 2020 and 70 in November 2020 and end at points 70 in November 2020, 55 in January 2021, and 55 in January 2021, respectively.

Out open box bracket 16 close box bracket colon open braces Apostrophe M A P E colon Apostrophe colon 24.071440955749733, Apostrophe M A E colon Apostrophe colon 11.286266474840719, Apostrophe R M S E colon Apostrophe colon 13.201605001517883, Apostrophe R M S L E colon Apostrophe colon 0.2593143147566868 close braces.

Based on the performance metrics, the model predictions are about 76 percent (MAPE = ~24 percent) accurate.

In practical scenarios, multiple values of ARIMA components need to be used to arrive at a model with maximum accuracy in predictions. Manually performing this task requires a lot of time and expertise in statistical techniques. Also, it becomes practically impossible to create models for a large number of entities such as servers, parameters, stocks, etc.

There are ML approaches called Auto ARIMA to learn optimal values of the components p,d,q. In Python there is a library called pdarima that provides the function auto_arima to try various values of ARIMA/SARIMA components in model creation and search for most optimal model with the lowest AIC and BIC values.

As per the auto ARIMA result shown in Figure 7-27, the best model is identified as ARIMA (6,1,0) with marginal improvement in the AIC and BIC values as compared to the previous ARIMA (6,1,2) model.

A prediction analysis depicts stepwise performance to minimize A I C. SARIMAX results are given below with different output headers depanneur variable, model, date, time, sample, covariance type, number observations, log-likelihood, A I C, B I C, H Q I C.

As observed in Figure 7-28 and as the performance metrics are obtained, there is even further degradation in the accuracy of the ARIMA (1,1,1) model.

A graph of C P U utilization versus the index. A frequency wave and a curve line are graphed. The frequency wave denotes actual starts from (0, 70) and ends at (40, 95). The curve line denotes ARIMA prediction starts from (0, 65) and ends at (95, 68). The points are approximate.

Based on the graph in Figure 7-29, we can observe seasonality in the data, which implies we should use the SARIMA model.

Four graphs. Graph 1, C P U utilization versus date-time where a high-frequency wave is graphed. Graph 2, trend versus date-time where a low-frequency wave is graphed. Graph 3, seasonal versus date-time where a high-frequency wave with the same height is graphed. Graph 4, resid versus date-time where a scatter plot is graphed.

As shown in Figure 7-30, auto ARIMA tried various combination and has detected SARIMA (3,0,2)(0,1,1)[7] as a best-fit model with considerable improvement in the AIC values as compared to previous ARIMA models.

An auto Arima model execution result on train data depicts stepwise performance to minimize A I C. SARIMAX results are given below with different output headers depanneur variable, model, date, time, sample, covariance type, number observations, log-likelihood, A I C, B I C, H Q I C.

After considering the seasonality, the prediction’s accuracy improved to about 62 percent as MAPE reduced to about 38 percent. This improvement in accuracy can also be observed in Figure 7-31 by plotting predictions of the ARIMA and SARIMA models against the actual test data observed.

A graph of C P U utilization versus the index. Two frequency waves and a curve line are graphed. The frequency waves denote SARIMA and actual. The actual starts around (0, 70) and ends at (40, 95) and SARIMA starts around (0, 65) and ends at (45, 95). The curve line denotes ARIMA prediction starts around (0, 65) and ends at (95, 68).

This model can be used to generate future predictions to determine an appropriate baseline dynamically rather than going with a static global baseline. This approach drastically reduces the noise in large environments.

This model can also be used for one of the most common use cases around the autoscaling of infrastructure, especially when there is seasonality in the utilization. The SARIMA (or SARIMAX) model can be used to analyze historical time-series data along with seasonality and any external factor to predict the utilization and accordingly scale up or down infrastructure for a specific duration and provide potential cost savings.