Overview

A linear regression models the relationship between a dependent variable, Y, and a set of independent variables, {X₀, …, X_k}, under the assumption of linearity in the coefficients. Linearity requires that the relationship between each X_j and Y can be modeled as a constant slope, represented by a scalar coefficient, β_j. Equation 3-1 provides the general form for a linear model with k independent variables.

Equation 3-1. A linear model.

In many cases, we will adopt the notation given in Equation 3-2, which explicitly specifies an index for each observation. Y_i, for instance, denotes the value of variable Y for entity i.

Equation 3-2. A linear model with entity indices.

In addition to entity indices, we will often use time indices in economic problems. In such cases, we will typically use a t subscript to indicate the time period in which the variable is observed, as we have done in Equation 3-3.

Equation 3-3. A linear model with entity and time indices.

In a linear regression, the model parameters, {α, β₁, …, β_k}, do not vary with time or by entity and, thus, are not indexed by either. Additionally, non-linear transformations of the parameters are not permitted. A dense neural network layer, for instance, has a similar functional form, but applies a non-linear transformation to the sum of coefficient-variable products, as shown in Equation 3-4, where σ represents the sigmoid function.

Equation 3-4. A dense layer of a neural network with a sigmoid activation function.

While linearity may appear to be a severe functional form restriction, it does not prevent us from applying transformations – including non-linear transformations – to the independent variables. We can, for instance, re-define X₀ as its natural logarithm and include it as an independent variable. Linear regressions also permit interactions between two variables, such as X₀ ∗ X₁, or indicator variables, such as . Additionally, in time series and panel settings, we can include lags of variables, such as X_t − 1j and X_t − 2j.

Transforming and re-defining variables makes linear regression a flexible method that can be used to approximate non-linear functions to an arbitrarily high degree of precision. For instance, consider the case where the true relationship between X and Y is given by the exponential function in Equation 3-5.

Equation 3-5. An exponential model.

If we take the natural logarithm of Y_i, we can perform the linear regression in Equation 3-6 to recover the model parameters, {α, β}.

Equation 3-6. A transformed exponential model.

In most settings, we won’t know the underlying data generating process (DGP). Furthermore, there will not be a deterministic relationship between the dependent variable and independent variables. Rather, there will be some noise, ϵ_i, associated with each observation, which could arise as the result of unobserved, random differences across entities or measurement error.

As an example of this, let’s say that we have data drawn from a process that is known to be non-linear, but its exact functional form is unknown. Figure 3-1 shows a scatterplot of the data, along with plots of two linear regression models. The first is trained under the assumption that the relationship between X and Y is well-approximated over the [0, 10] interval using a single line, as in Equation 3-7. The second is trained under the assumption that five line segments are needed, as in Equation 3-8.

Equation 3-7. A linear approximation to a non-linear model.

Equatio 3-8. A linear approximation to a non-linear relationship.

Figure 3-1 suggests that using a linear regression model with a single slope and intercept was insufficient; however, using multiple line segments in the form of a piecewise polynomial spline was sufficient to approximate the non-linear function, even though we worked entirely within the framework of linear regression.

Ordinary Least Squares (OLS)

Linear regression , as we have seen, is a versatile method that can be used to model the relationship between a dependent variable and set of independent variables. Even when that relationship is non-linear, we saw that it was possible to approximate it in a linear model using indicator functions, variable interactions, or variable transformations. In some cases, we were even able to capture it exactly through a variable transformation.

In this section, we’ll discuss how to implement a linear regression in TensorFlow. The way in which we do this will depend on our choice of loss function. In economics, the most common loss function is the sum or mean of the squared errors, which we will consider first. For the purpose of this example, we will stack all of the independent variables in an n x k matrix, X, where n is the number of observations and k is the number of independent variables, including the constant (bias) term.

We will let denote the vector of estimated coefficients on the independent variables, which we distinguish from the true parameter values, β. The “error” term that we will use to construct our loss function is given in Equation 3-9. It will often be referred to by different names, such as error, residual, or disturbance term.

Equation 3-9. The disturbance term from a linear regression.

Note that ϵ is an n-element column vector. This means that we can square and sum each element by pre-multiplying by its transpose, as in Equation 3-10, which gives us the sum of squared errors.

Equation 3-10. The sum of squared errors.

One of the benefits of using the sum of squared errors as a loss function  – also called performing “ordinary least squares” (OLS) – is that it permits an analytical solution, as derived in Equation 3-11, which means that we do not need to use time-consuming and error-prone optimization algorithms. We obtain this solution by choosing to minimize the sum of squared errors.

Equation 3-11. Minimizing the sum of squared errors.

The only thing left to check is whether is a minimum or a maximum. It will be a minimum whenever X has “full rank.” This will hold if no column of X is a linear combination of one or more other columns of X. Listing 3-1 provides a demonstration of how we can perform ordinary least squares (OLS) in TensorFlow for a toy problem .

import tensorflow as tf

# Define the data as constants.

X = tf.constant([[1, 0], [1, 2]], tf.float32)

Y = tf.constant([[2], [4]], tf.float32)

# Compute vector of parameters.

XT = tf.transpose(X)

XTX = tf.matmul(XT,X)

beta = tf.matmul(tf.matmul(tf.linalg.inv(XTX),XT),Y)

Listing 3-1

Implementation of OLS in TensorFlow 2

For convenience, we have defined the transpose of X as XT. We have also defined XTX as XT post-multiplied by X. We can compute by inverting XTX, post-multiplying by XT, and then post-multiplying by Y again.

The parameter vector we’ve computed, , minimizes the sum of squared errors. While computing was simple, it might be unclear why we would want to use TensorFlow for such a task. If we had instead used MATLAB, the syntax for writing the linear algebra operations would have been compact and readable. Alternatively, if we had used Stata or any statistics module in Python or R, we’d be able to automatically compute standard errors and confidence intervals for the vector of parameters, as well as measures of fit for the regression.

TensorFlow does, of course, have natural advantages if a task requires parallel or distributed computing; however, the need for this is likely to be minor when performing OLS analytically. The value of TensorFlow will become apparent when we want to minimize a loss function that doesn’t have an analytical solution or when we cannot hold all of the data in memory.

Least Absolute Deviations (LAD)

While OLS is the most commonly used form of linear regression in economics and has many attractive properties, we will sometimes want to use an alternative loss function. We may, for instance, want to minimize the sum of the absolute values of the errors, rather than the sum of the squares. This form of linear regression is referred to as Least Absolute Deviations (LAD) or Least Absolute Errors (LAE).

For all models, including OLS and LAD, the sensitivity of parameter estimates to outliers is driven by the loss function. Since OLS minimizes the squares of errors, it places a high emphasis on setting parameter values to explain outliers. That is, OLS will place a greater emphasis on eliminating a single large error than it will on two errors half of its size. To the contrary, LAD would place equal weight on the large error and the two smaller errors.

Another difference between OLS and LAD is that we cannot express the solution to a LAD regression analytically, since the absolute value prevents us from obtaining a closed-form algebraic expression. This means we must search for the minimum by “training” or “estimating” the model.

While TensorFlow wasn’t particularly useful for solving OLS, it has clear advantages when performing a LAD regression or training another type of model that has no analytical solution. We’ll see how to do this in TensorFlow and also evaluate how accurately TensorFlow identifies the true parameter values at the same time. More specifically, we’ll perform a Monte Carlo experiment, where we randomly generate data under certain assumed parameter values. We’ll then use the data to estimate the model, allowing us to compare the true and estimated parameters.

Listing 3-2 shows how the data is generated. We start by defining the number of observations and number of samples. Since we want to evaluate TensorFlow’s performance, we’ll train the model parameters on 100 separate samples. We’ll also use 10,000 observations to ensure that there is sufficient data to train the model.

Next, we define the true values of the model parameters, alpha and beta, which correspond to the constant (bias) term and the slope. We set the constant term to 1.0 and the slope to 3.0. Since these are the true values of the parameters and do not need to be trained, we will use tf.constant() to define them.

We now draw X and epsilon from normal distributions. For X, we use a standard normal distribution, which has a mean of 0 and a standard deviation of 1. These are the default parameter values for tf.random.normal(), so we do not need to specify anything beyond the number of samples and observations. For epsilon, we use a standard deviation of 0.25, which we specify using the stddev parameter . Finally, we compute the dependent variable, Y.

Listing 3-3 provides the code for the first step in the model training process in TensorFlow. We first draw values from a normal distribution with a mean of 0 and a standard deviation of 5.0 and then use them to initialize alphaHat and betaHat . The choice of 5.0 is arbitrary, but is intended to emulate a problem in which we have limited prior knowledge about the true parameter values. We use the suffix “Hat” to indicate that these are not the true values, but estimates. Since we want to train the parameters to minimize the loss function, we will define them using tf.Variable() , rather than tf.constant().

The next step is to define a function to compute the loss. A LAD regression minimizes the sum of absolute errors, which is equivalent to minimizing the mean absolute error. We will minimize the mean absolute error, since this has better numerical properties.¹

Furthermore, alphaHat and betaHat do not appear to adjust any further after they converge on their true parameter values. This suggests that the training process was stable and the stochastic gradient descent algorithm, which we will discuss in detail later in the chapter, was able to identify a clear local minimum, which turned out to be the global minimum in this case.²

Now that we’ve tested the solution method for one sample, we’ll repeat the process 100 times with different initial parameter values and different samples. We’ll then evaluate the performance of our solution method to determine whether it is sensitive to the choice of initial values or the data sample drawn. Figure 3-3 shows a histogram of the parameter value estimates at the 1000th epoch for each sample. Most estimates appear to be tightly clustered around the true parameter values; however, there are some deviations, due to either the initial values or the sample drawn. If we were planning to use LAD on a dataset with attributes similar to what we’ve generated in the Monte Carlo experiment, we might want to consider using a higher number of epochs to increase the probability that we converge to the true parameter values.

Beyond changing the number of epochs, we may also want to consider adjusting the optimization algorithm’s hyperparameters, rather than using the default options. Alternatively, we might consider using a different optimization algorithm altogether. As we will discuss later in the chapter, this is relatively simple to do in TensorFlow.

Other Loss Functions

As we discussed, OLS has an analytical solution, but LAD does not. Since most machine learning models do not permit an analytical solution, LAD can provide an instructive example. The same process we used to construct a model, define a loss function, and perform minimization for LAD will be repeated throughout the chapter and book. Indeed, the steps used to perform LAD can be applied to any form of linear regression by simply modifying the loss function.

There are, of course, reasons to favor OLS beyond the fact that it has a closed-form solution. For instance, if the conditions for the Gauss-Markov Theorem are satisfied, then the OLS estimator has the lowest variance among all linear and unbiased estimators.³ There is also a large econometric literature which builds on OLS and its variants, making it a natural choice for related work.

In many machine learning applications within economics and finance, however, the objective will often be to perform prediction, rather than hypothesis testing . In those cases, it may make sense to use a different form of linear regression; and using TensorFlow will make this task easier.

Discrete Dependent Variables

The submodule tf.losses offers two loss functions for discrete dependent variables in regression settings: tf.binary_crossentropy(), tf.categorical_crossentropy(), and tf.sparse_categorical_crossentropy(). We have previously covered the binary cross-entropy function, which is used in logistic regression. This provides us with a measure of loss when we have a binary dependent variable, such as an indicator for whether the economy is a recession, and a continuous prediction, such as a probability of being in a recession. For convenience, we repeat the formula for binary cross-entropy in Equation 3-17.

Equation 3-17. Binary cross-entropy loss function .

The categorical cross-entropy loss is simply the extension of the binary cross-entropy loss to cases where the dependent variable has more than two categories. Such models are commonly used in discrete choice problems, such as a model of the decision to commute by subway, bicycle, car, or foot. Within machine learning, categorical cross-entropy is the standard loss function for classification problems with more than two classes and is commonly used in neural networks that perform image and text classification. The equation for categorical cross-entropy is given in Equation 3-18. Note that (Y_i==k) is a binary variable equal to 1 if Y_i is class k and 0 otherwise. Additionally, p_k(X_i) is the probability that the model assigns to X_i being class k.

Equation 3-18. Categorical cross-entropy loss function.

Finally, if we have a problem with a dependent variable that may belong to multiple categories – that is, a “multi-label” problem – we’ll use the sparse categorical cross-entropy loss function, rather than categorical cross-entropy. Notice that the normal cross-entropy loss function assumes that the dependent variable can have only one class.

Continuous Dependent Variables

For continuous dependent variables, the most common loss functions are the mean absolute error (MAE) and mean squared error (MSE). MAE is used in LAD and MSE in OLS. Equation 3-19 defines the MAE loss function, and Equation 3-20 defines the MSE loss. Recall that is the model’s predicted value for observation i.

Equation 3-19. Mean absolute error loss .

Equation 3-20. Mean squared error loss .

Note that we can compute the losses using tf.losses.mae() and tf.losses.mse().

Other common loss functions for linear regression include the mean absolute percentage error (MAPE), the mean squared logarithmic error (MSLE) , and the Huber error, which are defined in Equations 3-21, 3-22, and 3-23. Respectively, these are available as tf.losses.MAPE(), tf.losses.MSLE(), and tf.losses.Huber().

Equation 3-21. Mean absolute percentage error.

Equation 3-22. Mean squared logarithmic error.

Equation 3-23. Huber error.

Figure 3-8 provides a comparison of selected loss functions. For each loss function, the loss value is plotted against the error value. Notice that the MAE loss scales linearly in the error. To the contrary, the MSE loss increases slowly near zero, but grows much faster far away from zero, leading to the application of a substantial penalty on outliers. Finally, the Huber loss is similar to the MSE loss near zero, but similar to the MAE loss as the error increases in size.

Stochastic Gradient Descent (SGD)

Stochastic gradient descent (SGD) is a minimization algorithm that updates parameter values through the use of the gradient. In this case, the gradient is a tensor of partial derivatives of the loss function with respect to each of the parameters.

The parameter update process is given in Equation 3-24. To ensure compatibility with the equivalent TensorFlow operation, we use the definition provided in the documentation. Note that θ_t is a vector of parameter values at iteration t, lr is the learning rate, and g_t is the gradient computed in iteration i.

Equation 3-24. Stochastic gradient descent in TensorFlow.

You might wonder in what sense SGD is “stochastic.” The stochasticity arises from the sampling process used to update the parameters. This differs from gradient descent, where the entire sample is used at each iteration. The benefits of the stochastic version of gradient descent are that it increases iteration speed and alleviates memory constraints.

Let’s take a look at a single SGD step for a linear regression with an intercept term and a single variable, where θ_t = [α_t, β_t]. We’ll start at iteration 0 and assume we’ve computed the gradient, g₀, for the batch of data as [−0.25, 0.33]. Additionally, we’ll set the learning rate, lr, to 0.01. What does this imply for θ₁? Using Equation 3-24, we can see that θ₁ = [α₀ + 0.025, β₀ − 0.033]. That is, we decrease α₀ by 0.025 and increase β₀ by 0.033.

Why do we increase a parameter value when the partial derivative is negative and decrease it when it is positive? Because the partial derivatives tell us how the loss function changes in response to a change in a given parameter. If the loss function is increasing, we’re moving further away from the minimum, so we want to change direction; however, if the loss function is decreasing, we’re moving toward a minimum, so we want to continue on the same direction. Furthermore, if the loss function is neither increasing nor decreasing, this means we’re at a minimum and the algorithm will naturally terminate.

Figure 3-9 illustrates this for the partial derivative of the loss function with respect to the intercept term. We focus on a narrow window around the true value of the intercept and plot both the loss function and its derivative. We can see that the derivative is initially negative, but increases to 0 at the true value of the intercept. It then becomes positive and increasing thereafter.

Returning to Equation 3-24, notice that the selection of the learning rate can also be quite consequential. If we select a high learning rate, we’ll take larger steps with each iteration, which could bring us closer to a minimum faster. However, taking larger steps could also lead us to skip over the minimum , missing it entirely. The selection of the learning rate should take this trade-off into consideration.

Finally, it is worth mentioning that the “minima” we’re identifying are local and, thus, may be higher than the global minimum. That is, SGD makes no distinction between the lowest point in an area and the lowest value of the loss function. Consequently, it may be worthwhile to re-run the algorithm for several different sets of initial parameter values to see if we always converge to the same minimum.

Modern Optimizers

While SGD is easy to understand, it is rarely used in machine learning applications in its original form. This is because modern extensions typically offer more flexibility and robustness and perform better on benchmark tasks. The most common extensions of SGD are root mean square propagation (RMSProp), adaptive moment estimation (Adam), and adaptive gradient methods (Adagrad and Adadelta).

There are several advantages to using modern extensions of SGD. First, starting with RMSProp, which is the oldest, they allow for the application of separate learning rates to each parameter. In many optimization problems, there will be orders of magnitude differences between partial derivatives in the gradient. Consequently, applying a learning rate of 0.001, for instance, may be sensible for one parameter, but not for another. RMSProp allows us to overcome this problem. It also allows for the use of “momentum,” where the gradients accumulate over mini-batches, making it possible for the algorithm to break out of local minima.

Adagrad, Adadelta, and Adam all offer variants on the use of momentum and adaptive updates for each individual parameter. Adam tends to work well for many optimization problems with its default parameters. Adagrad is centered around the accumulation of gradients and the adaptation of learning rates for individual parameters. And Adadelta modifies Adagrad by introducing a window over which accumulated gradients are retained.⁸

In all cases, the use of optimizers will follow a familiar two-step process. We’ll first instantiate an optimizer and will set its parameter values in the process using the tf.optimizer submodule. And second, we’ll iteratively apply the minimize function and pass the loss function to it as a lambda function.

For SGD, we set the learning rate and the momentum. If we’re concerned that there are many local minima, we can increase momentum to a higher value. For RMSProp, we not only set a momentum parameter but also set rho, which is the rate at which information about the gradient decays. The Adadelta parameter, which retains gradients for a period of time, also has the same decay parameter, rho. For Adagrad, we set an initial accumulator value, related to the intensity with which gradients are accumulated over time. Finally, for the Adam optimizer, we set decay rates for the accumulation of information about the mean and variance of the gradients. In this case, we have used the default values for the Adam optimizer, which generally perform well in large optimization problems.

We’ve now introduced the main optimizers we will use throughout the book. We will return to them again in detail when we apply them to train models. The modern variants of SGD will be particularly useful when we train large models with thousands of parameters.