Least Squares Regression: Not Just a Clever Name

For a moment, let's just focus on our output variable—lemonade sales. If we wanted to figure out how much lemonade you might sell in the future, a reasonable prediction could be the mean of past sales (12 + 13 + 15 + 14 + 17 + 16 + 19) / 7 = $15.14. In short, we have an easy linear model, where Sales = $15.14.

Notice, this is still a linear equation, just without temperature. That means, no matter what the temperature is, we're going to predict sales of $15.14. We know it's naive, but it does fit our definition of a model that maps inputs to outputs, even if every output happens to be the same.

So how well does our simple model do? To measure the model's performance, let's calculate how far away the predicted sales value is from each of the actual sales. When the temperature was 86, sales were $19; the model predicted $15.14. When the temperature was 81, sales were $12; the model again predicted $15.14. The first instance represents an underprediction of about $4; the second, an overprediction of about $3. So far, we see that our model isn't perfect. And to really understand its ability to predict well, we'll need to understand the difference between what our model predicted and what actually happened. These differences are called errors—and they represent how far off our prediction is from what we know to be true.

So how could we measure these errors to understand how well our model is doing? Well, we could take every actual sales price and subtract it from the mean, which we predicted to be $15.14. But if you do this, you'll see that every error, when summed up, always totals to zero. That's because the mean, which we're using as our predictor, represents the arithmetic center of these points. Subtracting the difference of all these points to the center makes the result zero.

But we do need a way to aggregate these errors because, clearly, the model isn't perfect. The most common approach is called the least squares method. And, in this approach, we instead square the differences from our prediction to make everything positive.⁵ When we add up these numbers, the result won't be zero (unless, in fact, there is no error in our data). We call the result the sum of squares.

Let's take a look at Figure 9.3(a). You'll see the original scatter plot with x = Temperature and y = Sales. Notice, we've applied our naive model: temperature has no effect on the prediction. That is, the prediction is always 15.14, which gives a horizontal line at Sales = 15.14. In other words: the data point with temperature = 86 and actual sales = 19 has predicted sales of 15.14. The solid vertical line shows the difference between the actual value and predicted value for this point (and all other points). In regression, you square these lengths to create a square with area 14.9.

When sales were $15, our model still predicted $15.14, thus the corresponding square has an area of (15.14 – 15)² = 0.02. Add up the areas (sum the squares) from all points to represent the total error of your simple model. The squares you see on the right plot of Figure 9.3(a) are a visual representation of the squared error calculations. The larger the sum of squares, the worse the model fits the data. The smaller the sum of squares, the better the fit.

The question then becomes, can we arrive at a slope and intercept that would optimize (that is, reduce) the sum of squared errors as much as possible? Right now, our naive model has no slope, but it does have an intercept of $15.14.

Clearly 9.3(a) doesn't do the best job. To get even closer to a good fit, let's add a slope m, which brings temperature into the equation. In Figure 9.3(b), we speculate a reasonable slope and intercept could be 0.6 and –34.91, respectively. Notice that adding this relationship makes our flat-lined model from 9.3(a) into a sloped line that appears to capture some upward trend. You can also immediately see a reduction in the overall area of the squares.

Visually, the error has decreased substantially now that your model includes temperature. Your prediction for the point with temperature = 86 went from 15.14 in the simple model to Sales = 0.6(86) – 34.91= 16.69, which reduced this observation's contribution to the sum of squares from 14.9 to (16.69 – 19)² = 5.34.

You could do the guesswork yourself—that is, plugging numbers into the slope and intercept until you might get the one combination that would reduce the sum of squares to its mathematical minimum. But linear regression does this for you mathematically. Figure 9.3(c) shows, quite literally, the least amount of squared errors you can see with this data. The slightest derivation from these slope and intercept numbers would make those squares bigger.

You can use the information to assess how well the final model fits the data. Note Figure 9.3(c), which contains the linear regression result, is still imperfect. But you can clearly see it's better than Figure 9.3(a) where you predicted $15.14 each time.

How much better? Well, you started with an area (sum of squares) of 34.86, and the final model reduced the area to 7.4. That means we've reduced the area of the control model by (34.86 – 7.4) = 27.46, which is a 27.46/34.86 = 78.8% percent reduction in total area. You'll often hear people say the model has “explained,” “described,” or “predicted” 78.8% (or 0.788) of the variation in the data. This number is called “R-squared” or R².

If the model fits the data perfectly, R² = 1. But don't count on seeing such models with a high R² in your work.⁶ If you do, someone probably made a mistake—and you should ask to review the data collection processes. Recall from Chapter 3 that variation is in everything, and there's always variation we can't fully explain. That's just how the universe is.

Extending to Many Features

Your business, we assume, is a little more complicated than a lemonade stand. Your sales would not only be a function of temperature (if a seasonal business), but many other features or inputs. Fortunately, the simple linear regression model you've learned about can be extended to include many features.⁷ Regression with one input is called simple linear regression; with multiple inputs, it's called multiple linear regression.

To show you an example, we fit a quick multiple linear regression line to the housing data we explored in Chapter 5. This data had 1234 houses and 81 inputs, but to simplify the example, we're going to look at 6 inputs. (We could have also used PCA to reduce the dimensionality, but we didn't want to overcomplicate this example.)

Let's build a model to predict the sales price of a home (output) based on lot area, year built, 1st floor square footage, 2nd floor square footage, basement square footage, and number of full bathrooms. The linear regression algorithm will work its magic, learn from the data, and output the best coefficients to complete the model, resulting in the intercept and coefficients in Table 9.2.

A core tenet of a multiple regression model is to isolate the effect of one variable while controlling for the others. We can say, for example, if all other inputs are held constant, a home built one year sooner adds (on average) $818.38 to the sales price. The coefficients of each feature are showing the magnitude and direction the input has on price. Make sure to consider the units. Adding 1 unit to square footage is different than adding 1 unit to the number of bathrooms. A statistician can scale these appropriately if you need an apples-to-apples comparison between coefficients.

Each coefficient also undergoes an associated statistical test to let us know if it's statistically different than zero. If it is not, we can safely remove it from the model because it doesn't add information or change the output.

Omitted Variables

Supervised learning models cannot learn the relationship between an input variable and an output variable if the input variable was omitted from the model. Consider our simple model that predicted lemonade sales based on average past sales but without regard to temperature.

Data Heads like yourself, now aware of this issue, might suggest informative, relevant features to include in models. But don't just leave feature selection up to your data workers. Subject matter expertise and getting the right data into a supervised learning model is key to having a successful model.

For example, the housing model in the previous section has an R² value of 0.75. This means we've explained 75% of the variation in sales price with our model. Now think about features not in this model that would help one predict a home's price—perhaps things like economic conditions, interest rates, elementary school ratings, etc. These omitted variables not only impact predictions of the model but may lead to dubious interpretations. Did you notice in Table 9.2 the coefficient for the number of bathrooms was negative? That doesn't make sense.

Here's another example. Consider a linear regression model that “learned” shoe size has a large positive coefficient in a linear regression model for the number of words a person can read per minute. Age is clearly a missing input variable here: including it would remove the need for “shoe size” as an input. Certainly, examples in your work may not be as easy to tease out as this example but believe us when we say omitted variables can and will cause headaches and misinterpretations. Further, many things are correlated with time, a common omitted variable.

We hope the phrase “correlation does not mean causation” comes to mind once again while reading this section. Having one variable labeled as an input to a model and another as an output does not mean the input causes the output.

Multicollinearity

If your goal with linear regression is interpretability—to be able to study the impact of the input variables on the output by studying the coefficients—then you must be aware of multicollinearity. This means several variables are correlated, and it creates a direct challenge to your model being interpretable.

Recall, a goal of multiple regression is to isolate the effect of one input's contribution while holding every other input constant. But this is only possible if the underlying data is uncorrelated.

To offer a quick example, suppose the lemonade stand data we plotted earlier had the temperature in both Celsius and Fahrenheit. Obviously, the two are perfectly correlated because one is a function of the other, but for this example, suppose each temperature reading was taken with a different instrument to introduce some variation.⁸ The model would go from:

1. Sales = 1.03(Temperature) – 71.07 to
2. Sales = –0.2(Temperature in F) + 2.1(Temperature in C) – 30.8.

Now it looks like adding a degree in Fahrenheit has a negative impact! Yet, we know the inputs are comingled—in fact, redundant—and linear regression cannot break apart the relationship. Multicollinearity is present in most observational datasets, so consider this a fair warning. Data collected experimentally, however, is specifically designed to prevent the inputs from being multicollinear to the extent possible.⁹

Data Leakage

Imagine building a model to predict the sales price of a home (like we did earlier in the chapter). But this time, the training data not only includes features about the home (size, number of bedrooms, etc.), but also the initial offer made on the house. A snapshot of the dataset is shown in Table 9.3.

If you run the model on the data, you might notice that the initial offer is very predictive of the sales price. Great!—you think. You can rely on that to help you predict home prices for your company.

The model then goes into production. You attempt to run the model only to find that you don't have access to the initial offer on the houses whose sales prices you are trying to predict—they haven't sold yet! This is an example of data leakage. Data leakage¹⁰ happens when a concurrent output variable masquerades as an input variable.

The problem with using the initial offer is one of timing. Consider for a moment that you could only learn what the starting offer was after the home was already sold.

As we are immersed in data, data leakage is easy to overlook. Sadly, many textbooks don't cover it because the datasets inside are pristine for learning, but real-world datasets always have the possibility for leakage. As a Data Head, you must be on the lookout to ensure your input and output data do not contain overlapping information.

We will explore data leakage again in later chapters.

Extrapolation Failures

Extrapolation means predicting beyond the range of the input data you used to build your model. If the temperature was 0⁰F, the lemonade stand model would predict sales of –$71.07. If a house had no square footage and no bathrooms (practically speaking, the house wouldn't exist), the model would predict a sales price of –$1,614,841.60. Both cases are nonsense.

The models are predicting outside the range of data they “learned” from, and unlike humans, equations don't have common sense to know it's wrong. Math equations cannot think. If you give it numbers as inputs, it will spit back a number as an output. It's up to you, the Data Head, to know if extrapolation is taking place.

It must be stressed that model results are always predicted on the given data. Which is to say, you should not make predictions with data that “fits” the range of the training data but not the context in which the dataset was collected. The model has no context for changes taking place in the world.

If you built a model that predicted housing prices in 2007, your model would have performed terribly in 2008 after the housing market crashed. Using the model in 2008 would have extrapolated the market conditions of 2007 data to be vastly different in 2008. Industries are facing this at the time of writing, 2021, because of the COVID-19 pandemic. Models trained on pre-COVID data no longer reflect many of the relationships identified, and the models are no longer valid.

Many Relationships Aren't Linear

Linear regression would not do well modeling the performance of the stock market. Historically, it has grown exponentially, not linearly. The Statistics department at Procter & Gamble would offer this advice: “Don't fit a straight line through a banana.”

Statisticians have tools in their arsenal to transform some non-linear data into linear data. Even so, sometimes, you must accept that linear regression is not the right tool for the job.

Are You Explaining or Predicting?

Throughout this chapter, we've been discussing two possible goals with regression models: explain relationships and make predictions. Linear regression models can seemingly do both. A linear regression model's coefficients (under the right conditions) offer interpretability. Many industries focus their attention on interpretability—like in clinical trials where researchers need to know the precise magnitude and direction of the input “dosage of a medicine” to the output “blood pressure.” In this case, great care must be taken to avoid multicollinearity and omitted variables to ensure model explanations are possible.

In other fields, like machine learning, accurate prediction is the goal.¹¹ The presence of multicollinearity, for example, may not be a concern if the model can predict future outputs well. When the purpose of a model is to predict new outputs, you absolutely must be careful to avoid overfitting the data.

Recall that models are simplified versions of reality. A good model approximates the relationships between inputs and outputs well. Indeed, it's capturing some underlying phenomenon. The data itself is merely an expression of this underlying phenomenon.

An overfit model, however, does not capture that relationship we speculate exists. Rather it captures the interaction of the training data, including the noise and variation in the data itself. Thus, the predictions it makes are not what's being modeled, but rather just the data points we have.

Models that overfit effectively memorize the sample training data. They don't generalize well to new observations. Check out Figure 9.4. On the left, you see the lemonade data with the linear regression model. On the right, a complex regression model that perfectly predicts some of the points. Which would you want to use to make a prediction?

The way to prevent overfitting is to split a dataset into two parts: a training set you use to build a model and a test set to see how it performs. The performance on your test set, data your model did not learn from, will be the ultimate judge of how well your model predicts.

Regression Performance

Whenever you come across a regression model at work, whether it's a multiple linear regression model or something more sophisticated that was just invented last week, the best way to judge how well it fits your data is with an “actual by predicted” plot. In our experience, some people assume we can't visualize regression performance when we have too many inputs, but remember what the model has done. It converted inputs (either one or several) into an output.

So, for each row in your dataset, you have the actual value and its associated predicted value. Plot them in a scatter plot! These should be well correlated. This eye-ball test will let you know quickly how well your model has done. Your data workers can provide several associated metrics (R-squared being one), but never look at those numbers alone. Always, always demand to see the actual vs. predicted plot. An example is shown in Figure 9.5 using the model we built on the housing data.