Regression models

• •How the trends and patterns identified in the data can be used to predict the future business outcome(s)?
• •How can we identify appropriate prediction models?
• •How the models be used in making prediction about how things will turn out in the future—what will happen in the future?
• •How can we predict the future trends of the key performance indicators using the past data and models and make predictions?

• •Regression models: (a) simple regression models; (b) multiple regression models; (c) nonlinear regression models, including the quadratic or second-order models, and polynomial regression models; (d) regression models with indicator or qualitative independent variables; (e) regression models with interaction terms or interaction models; and (f) logistic regression models.

Forecasting models

How to predict the future behavior of the key business outcomes or variables using different forecasting techniques suited to predict future business phenomena?

How different forecasting models using both the qualitative and quantitative forecasting techniques can be applied to predict a number of future business phenomena?

How some of the key variables, including the sales, revenue, number of customers, demand, inventory, customer behavior, number of visits to the business website, and many others, can be predicted using a number of proven techniques?

The prediction and forecasting methods use a number of time series models as well as data mining techniques.

How the forecast can be used in short-term and long-term business planning?

Forecasting techniques: Widely used predictive models involve a class of time series analysis and forecasting models. The commonly used forecasting models fall into the following categories:

• •Techniques using average: simple moving average, weighted moving average, exponential smoothing
• •Techniques for trend: linear trend equation (similar to simple regression), double moving average or moving average with trend, exponential smoothing with trend or trend-adjusted exponential smoothing
• •Techniques for seasonality: forecasting data with seasonal pattern
• •Associative forecasting techniques: simple regression, multiple regression analysis, nonlinear regression, regression involving categorical or indicator variables, and other regression models
• •Regression-based models that use regression analysis to forecast future trends. Other time series forecasting models are simple moving average, moving average with trend, exponential smoothing, exponential smoothing with trend, and forecasting seasonal data.

ANOVA (analysis of variance)

ANOVA in its simplest form is a way to study multiple means. Single-factor, two- and multiple factor ANOVA along with design of experiment (DOE) techniques are powerful tools used in data analysis to study and identify key variables and build prediction equations. These models are used in modeling and predictive analytics to predict future outcomes.

ANOVA and DOE techniques include single-factor ANOVA, two-factor ANOVA, and multiple factor ANOVA. Factorial designs and DOE tools are used to create models involving multiple factors.

Data mining

Determines meaningful patterns and deriving insights from large data sets. It is closely related to analytics. Data mining uses statistics, machine learning, and artificial intelligence techniques to derive meaningful patterns and make predictions.

Data mining techniques are used to extract useful information from huge amounts of data using predictive analytics, computer algorithms, software, mathematical, and statistical tools.

Other tools of predictive analytics:

Machine learning, artificial intelligence, neural networks, and deep learning

Machine learning is a method used to design systems that can learn, adjust, and improve based on the data fed to them. Machine-learning works based on predictive and statistical algorithms that are provided to these machines. The algorithms are designed to learn and improve as more data flow through the system.

Machine learning, artificial intelligence, neural networks, and deep learning have been used successfully in fraud detection, e-mail spam, GPS systems, medicine, medical diagnosis, and predicting and treating a number of medical conditions. There are other applications of machine learning.

Simple regression model

Background: Regression analysis is used to investigate the relationship between two or more variables.

Often we are interested in predicting a variable using one or more independent variables.

In general, we have one dependent or response variable y and one or more independent variables x1, x2,…,xk.

The independent variables are also called predictors. If there is only one independent variable x that we are trying to relate to the dependent variable y, then this is a case of simple regression. On the other hand, if we have two or more independent variables that are related to a single response or dependent variable, we have a case of multiple regression.

The purpose of simple regression analysis is to develop a statistical model that can be used to predict the value of a response or dependent variable using an independent variable.

For example, we might be interested in predicting the profit using the number of customers or we might be interested in predicting the time required to produce certain number of products in a production situation. In these cases, the variable profit or the variable time that we are trying to predict is known as the dependent or the response variable, and the other variable, sales or the number of products, is referred to as the independent variable or predictor.

In a simple linear regression method, we study the linear relationship between two variables, the dependent or the response variable (y) and the independent variable or predictor (x). The following is an example relating the advertising expenditure and sales of a company. The relationship is linear and the objective is to predict sales—response variable (y) using advertisement—the independent variable or predictor (x). A scatter plot as shown in Figure 6.2 is one of the first steps in studying the relationship (linear or nonlinear) between the variables.

Figure 6.2 Scatter plot of sales versus advertising

Multiple regression models

In regression analysis, we have one dependent or response variable y and one or more independent variables, x1, x2,…,xk. The independent variables are also called predictors. If there is only one independent variable x that we are trying to relate to the dependent variable y, then this is a case of simple regression. On the other hand, if we have two or more independent variables that are related to a single response or dependent variable, then we have a case of multiple regression.

The relationship between the dependent and independent variable or variables are described by a mathematical model known as a regression equation. The regression model is described in the form of a regression equation that is obtained using the least squares method. In case of a multiple linear regression the equation is of the form: y = b0 + b1x1 + b2x2 + b3x3 + … + bnxn, where b0, b1, b2, …, bn are the regression coefficients and x1, x2,…,xk are the independent variables.

A pharmaceutical company is concerned about declining sales of one of its drugs. The drug was introduced in the market approximately two-and-a half years ago. In the recent few months the sales of this product is in constant decline and the company is concerned about losing its market share as it is one of the major drugs the company markets. The head of the sales and marketing department wants to investigate the possible causes and evaluate some strategies to boost the sales. He would like to build a regression model of the sales volume and several independent variables believed to be strongly related to the sales. A multiple regression model will help the company to determine the important variables and also predict the future sales volume. The marketing director believes that the sales volume is directly related to three major factors: dollars spent on advertisement, commission paid to the salespersons, and the number of salespersons deployed for marketing this drug. A multiple regression model can be built to study this problem.

In a multiple regression, the least squares method determines the best fitting plane or the hyperplane through the data points that ensures that the sum of the squares of the vertical distances or deviations from the given points and the plane are a minimum.

Figure 6.3 below shows a multiple regression model with two independent variables. The response y with two independent variables x1 and x2 forms a regression plane.

Figure 6.3 A multiple regression model

Nonlinear regression (quadratic and polynomial) models

The above models—simple and multiple regression—are based on the assumption of linearity, that is, the relationship between the independent variable(s) and the response variable can be well approximated by a linear model. However, in certain situations the relationship between the variables is not linear but may be described by quadratic or second-order model. Sometimes the relationships can be described using a polynomial model.

A nonlinear (second-order) regression model is described here:

The life of an electronic component is believed to be related to the temperature in the operating environment. A scatter plot shown below was created to study the relationship. The scatter plot in Figure 6.4 shows the life of the components (in hours) and the corresponding operating temperature (in ° F). From the scatter plot, it is clear that the relationship between the variables is not linear. An appropriate model in this case would be a second-order or quadratic model that will predict the life of the component. In this case, the life of the component is the dependent variable (y) and the operating temperature is the independent variable (x).

Figure 6.4 Scatter plot of life (y) versus operating temp. (x)

Figure 6.5 shows a second-order model with the regression equation that can be used to predict the life of the components using temperature.

Figure 6.5 A second-order regression model

Multiple regression using dummy or indicator variables

In regression we often encounter qualitative or indicator variables that need to be included as one of the independent variables in the model.

To include such qualitative variables in the model, we use a dummy or indicator variable. The use of dummy or indicator variable in a regression model allows us to include qualitative variables in the model. For example, to include the sex of employees in a regression model as an independent variable, we define this variable as

In the above formulation, a “1” indicates that the employee is a male and a “0” means the employee is a female. Which one of the male or female is assigned the value of 1 is arbitrary. In general, the number of dummy or indicator variable needed is one less than the total number of indicator variables to be included in the model.

Application of a regression model with a dummy variable:

We would like to write a model relating the mean profit of a grocery chain. It is believed that the profit to a large extent depends on the location of the stores. Suppose that the management is interested in three specific locations where the stores are located. We will call these locations A, B, and C. In this case, the store location is a single qualitative variable, which is at three levels corresponding to the three locations A, B, and C. The prediction equation relating the mean profit () and the three locations can be written as:

The variables x1 and x2 are known as the dummy variables that make the model function.

All subset and stepwise regression

Finding the best set of predictor variables to be included in the model

Some other regression models:

Reciprocal transformation of x variable

This transformation can produce a linear relationship and is of the form

This model is appropriate when x and y have an inverse relationship. Note that the inverse relationship is not linear.

Log transformation of x variable

Log transformation of x and y variables

The logarithmic transformation is of the form

This is a useful curvilinear form where ln(x) is the natural logarithm of x and x > 0.

The purpose of this transformation is to achieve a linear relationship. The model is valid for positive values of x and y. This transformation is more involved and is difficult to compare it with other models with y as the dependent variable.

Logistic regression

This model is used when the response variable is categorical. In all the regression models we developed in this book, response variable was a quantitative variable. In cases, where the response is categorical or qualitative, the simple and multiple least-squares regression model violates the normality assumption. The correct model in this case is logistic regression.

Forecasting models

A forecast is a statement about the future value of a variable of interest such as demand.

Forecasting is used to make informed decisions and may be long-range or short-range.

Forecasts affect decisions and activities throughout an organization. Produce-to-order companies depend on demand forecast to plan their production. Inventory planning and decisions are affected by forecast. Following are some of the areas where forecasting is used.

Forecasting methods are classified as qualitative or quantitative.

Qualitative forecasting methods use expert judgment to develop forecasts. These methods are used when historical data on the variable being forecast are usually not available. The method is also known as judgmental as they use subjective inputs.

Usually the first step in forecasting is to plot the historical data. This is critical in identifying the pattern in the time series and applying the correct forecasting method. If the data are plotted over time, such plots are known as time series plots. This plot involves plotting the time on the horizontal axis and the variable of interest on the vertical axis. The time series plot is a graphical representation of data over time where the data may be weekly, monthly, quarterly, or annually. Some of the common time series patterns are discussed.

Figure 6.6 below shows that the demand data are fluctuating around an average. The averaging techniques such as simple moving average or simple exponential smoothing can be used to forecast such patterns. The actual data and the forecast are shown in Figure 6.7.

Figure 6.6 Plot of demand over time

These forecasts may be based on consumer surveys, opinions of sales and marketing, market sensing, and Delphi method that uses opinions of managers or consensus.

The objective of forecasting is to predict the future outcome based on the past pattern or data. When the historical data are not available, qualitative methods are used. These methods are used in absence of past data or in cases when a new product is to be launched for which information is not available. Qualitative methods forecast the future outcome based on opinion, judgement, or experience.

Figure 6.7 Demand and forecast

Figures 6.8 and 6.9 show the sales data for a company over a period of 65 weeks. Clearly, the data are fluctuating around an average and showing an increasing trend. Forecasting techniques such as double moving average or exponential smoothing with a trend can be used to forecast such patterns. The plot in Figure 6.10 shows the sales and forecast for the data.

The forecast may be based on the consumer/customer surveys, executive opinions, sales force opinions, surveys of similar competitive products, Delphi method, expert knowledge, and opinions of managers, achieving a consensus on the forecast.

Quantitative forecasting is based on historical data. The most common methods are time series and associative forecasting methods. These are discussed in detail in the subsequent sections. The forecasting methods and models can be divided into following categories:

Techniques using average

Simple moving average

Weighted moving average

Exponential smoothing

Techniques for trend

Linear trend equation (similar to simple regression)

Figure 6.8 Sales over time

Double moving average or moving average with trend, exponential smoothing with trend or trend-adjusted exponential smoothing

Techniques for seasonality

Forecasting data with seasonal pattern

Associative forecasting techniques

Simple regression

Multiple regression analysis

Nonlinear regression

Regression involving categorical or indicator variables, and

Other regression models

Figure 6.9 Sales and forecast for the data in Figure 6.8

The other forecasting techniques involve a number of regression models, and forecasting seasonal patterns. Figure 6.10 shows a seasonal pattern and forecast.Figure 6.10 A seasonal pattern

Figure 6.10 A seasonal pattern

ANOVA (analysis of variance)

A single-factor completely randomized design is the simplest experimental design. This design involves one factor at different levels that can be dealt with a single-factor factorial experiment. The analysis method for such problems is known as ANOVA.

In ANOVA, the procedure uses variances to determine whether the means of multiple groups are different. The process works by comparing the variance between group vs. the variance within groups and determines whether the groups are all part of a single population or separate populations.

Design of experiment (DOE) is a powerful tool. Many variations of design involve two-factor factorial design using a two-factor ANOVA. More than two factors can be studied using specially designed experiments.

Consider an example in which the marketing manager of a franchise wants to know whether there is a difference in the average profit among four of their stores. He randomly selected four stores and recorded the profit for these stores. The data would look like Table 6.1. In this case, the single factor of interest is store. Since there are four stores, store 1, store 2, store 3, and store 4, we have four levels of the same factor. Recall that the levels of a factor are also known as treatments or groups; therefore, we can say that there are four treatments or groups. The manager wants to study the profit for the selected stores; therefore, profit is the response variable. The response variable is the variable that is measured in an experiment. This is an example of a one-factor ANOVA where the single-factor store is at four levels and the response variable is profit.

Table 6.1

The null and alternate hypotheses for a one-factor ANOVA involving k treatments or groups tests whether the k treatment means are equal.

Data mining

Data mining involves exploring new patterns and relationships from the collected data—a part of predictive analytics that involves processing and analyzing huge amounts of data to extract useful information and patterns hidden in the data. The overall goal of data mining is knowledge discovery from the data. Data mining techniques are used to (i) extract previously unknown and potential useful knowledge or patterns from massive amount of data collected and stored and (ii) exploring and analyzing these large quantities of data to discover meaningful pattern, and transforming data into an understandable structure for further use. The field of data mining is rapidly growing and statistics plays a major role in it. Data mining is also known as knowledge discovery in databases (KDD), pattern analysis, information harvesting, business intelligence, analytics, etc. Besides statistics, data mining uses artificial intelligence, machine learning, database systems, advanced statistical tools, and pattern recognition.

Data mining is one of the major tools of predictive analytics. In business, data mining is used to analyze business data. Business transaction data along with other customer and product related data are continuously stored in the databases. The data mining software are used to analyze the vast amount of customer data to reveal hidden patterns, trends, and other customer behavior. Businesses use data mining to perform market analysis to identify and develop new products, analyze their supply chain, find the root cause of manufacturing problems, study the customer behavior for product promotion, improve sales by understanding the needs and requirements of their customer, prevent customer attrition, and acquire new customers. For example, Wal-Mart collects and processes over 20 million point-of-sale transactions every day. These data are stored in a centralized database and are analyzed using data mining software to understand and determine customer behavior, needs, and requirements. The data are analyzed to determine sales trends and forecasts, develop marketing strategies, and predict customer-buying habits http://www.laits.utexas.edu/~anorman/BUS.FOR/course.mat/Alex

The success with data mining and predictive modeling has encouraged many businesses to invest in data mining to achieve a competitive advantage. Data mining has been successfully applied in several areas of business and industry, including customer service, banking, credit card fraud detection, risk management, sales and advertising, sales forecast, customer segmentation, and manufacturing.

Data mining is “the process of uncovering hidden trends and patterns that lead to predictive modeling using a combination of explicit knowledge base, sophisticated analytical skills and academic domain knowledge (Luan, Jing, 2002).” Data mining has been used successfully in science, engineering, business, and finance to extract previously unknown patterns in the databases containing massive amount of data and to make predictions that are critical in decision making and improving the overall system performance.

In recent years, data mining combined with machine learning/artificial intelligence is finding larger and wider applications in analyzing business data, thereby predicting future business outcomes. The reason for this is the growing interest in knowledge management and in moving from data to information and finally to knowledge discovery.

In this age of technology, companies collect massive amount of data automatically using different means. A large quantity of data is also collected using remote sensors and satellites. With the huge quantities of data collected today—usually referred to as big data, traditional techniques of data analysis are infeasible for processing the raw data. The data in its raw form have no meaning unless processed and analyzed. Among several tools and techniques available and currently emerging with the advancement of technology and computers, it is now possible to analyze big data using data mining, machine learning, and artificial intelligence (AI) techniques.

Machine learning

Machine learning methods use complex models and algorithms that are used to make predictions. These models allow the analysts to make predictions by learning from the trends, patterns, and relationships in the historical data. The algorithms are designed to learn iteratively from data without being programmed. In a way, machine learning automates model building.

Machine-learning algorithms have extensive applications in data-driven predictions and are a major decision-making tool. Some applications where machine learning has been used are e-mail filtering, cyber security, signal processing, fraud detection, and others. Machine learning is employed in a range of computing tasks. Although machine-learning models are being used in a number of applications, it has limitations in designing and programming explicit algorithms that are reproducible and have repeatability with good performance. With current research and the use of newer technology, the field of machine learning and artificial intelligence are becoming more promising.