Chapter 3: Regression Analysis with Autocorrelated Errors

Introduction

Correction of Standard Errors with PROC AUTOREG

Adjustment of Standard Deviations by the Newey-West Method

Cochrane-Orcutt Estimation Using PROC AUTOREG

Simultaneous Estimation Using PROC AUTOREG

Conclusion

Introduction

In this chapter, the example from Chapter 2 is continued, using PROC AUTOREG, which is specially designed for regression that uses time series data. The conclusion from the analysis in Chapter 2 remains, however, that the results rely on unrealistic assumptions. The very idea of forming a regression model that takes no account of the dynamics of the time series is wrong.

Correction of Standard Errors with PROC AUTOREG

Program 3.1 presents the code to redo the analysis of Program 2.2, using PROC AUTOREG instead of PROC REG. You can easily switch between these procedures because the syntax is almost identical.

Program 3.1: Regression Using Time Series Data and PROC AUTOREG

Among the plots given by the option PLOTS=ALL to the PROC statement is the autocorrelation function, which is denoted as ACF in Figure 3.1. The autocorrelation function is a graph of all autocorrelations defined for lag 0 up to some limit, which is 25 by default. The autocorrelation at lag 0 is 1 by definition. In the code, all plots are unpacked with the UNPACK suboption to the PLOT option. This means that the plots are presented one by one, while the default is to present them in panels of up to six plots.

Figure 3.1 clearly shows the autocorrelation problem as seen in the previous chapter. The form of the autocorrelations is a monotone declining sequence with no oscillations. This picture can be interpreted as a first-order autoregressive model for the residuals, εt. However, the ideal form of the autocorrelations for a first-order autoregressive model should be an exponential declining behavior, not linear as is probably the case here. A linear form of the autocorrelation can be taken as an indication of a unit root. This topic is addressed by further testing and model fitting in Chapters 4 and 5.

Figure 3.1: Residual Autocorrelation Function for Ordinary Least Squares Estimation

Adjustment of Standard Deviations by the Newey-West Method

An attractive possibility is to adjust the testing procedure for the presence of autocorrelation without trying to formulate a more specific time series model for the dynamics of the two time series. This is possible because of the COWEST=NEWEYWEST option in the MODEL statement (Program 3.2). With this option, the standard errors for the estimated regression parameters are adjusted according to the time series properties of the residual process.

In practice, the standard deviation is calculated by a formula that includes the observed autocorrelations for the residual series, εt. Also, in this way, eventual heteroscedasticity, which is a time-varying variance of the residuals, εt, is accounted for. These standard deviations are often denoted as heteroscedasticity and autocorrelation consistent (HAC). Many econometricians apply these corrected standard deviations as standard practice because, in the simple case of no autocorrelation, the method, at least in theory, does no harm. For precise details of how these standard deviations are derived, see the online Help. Also see the online Help for the many suboptions that PROC AUTOREG offers for the precise calculation of these values.

Program 3.2: Adjustment of Standard Deviations by the Newey-West Method

The resulting output is seen in Output 3.1. The estimated parameter values are the same as in Output 2.2, but the standard deviations are different. The seasonal dummy for the second quarter is significant. It says that milk production is significantly larger in the second quarter than in the fourth quarter. (The fourth quarter is chosen as the basic quarter when only dummy variables for the three first quarters are included in the model.) Also, the simultaneous test for all three seasonal dummy variables is significant, as seen in Output 3.2.

Output 3.1: Parameter Estimates with HAC Standard Deviations

Output 3.2: Simultaneous Test for Seasonality Using HAC Standard Deviations

It has to be stressed that this method in no way properly fits a model for the time series dynamics of the problem. Only the problem of misleading standard errors of ordinary least squares estimation is taken into account. The correction for autocorrelations relies on the assumption that the true statistical model has the form of a linear regression with autocorrelated residuals.

Cochrane-Orcutt Estimation Using PROC AUTOREG

The first simple extension of this ordinary least squares analysis as it was shown in Chapter 2 is repeated with the use of PROC AUTOREG in the next section of this chapter. PROC AUTOREG includes the Cochrane-Orcutt estimation method to cover the special situation of only lag one autocorrelation. PROC AUTOREG is designed for univariate regression models for time series data with autocorrelated errors.

Program 3.3 presents the code to redo the analysis of Program 2.3, using PROC AUTOREG instead of PROC REG. The syntax of these procedures is almost identical.

Program 3.3: Cochrane-Orcutt by PROC AUTOREG

The resulting coefficient of the number of cows, 7.8623, in Output 3.3, is almost identical to the estimated 7.9559, presented by Output 2.4 as the coefficient to the independent variable, which is the transformed number of cows. This difference is to some extent a consequence of rounding errors in the estimated autocorrelation r1 = .936 as printed in Output 2.1, which is used in the manual coding in Program 2.3. If the more correct value r1 = .935982 is used instead, then the results become identical to the fourth decimal place.

Output 3.3: Estimated Parameters Using Cochrane-Orcutt by PROC AUTOREG

The estimates in Output 3.3 are calculated with use of the first-order autocorrelation for the estimated residuals. These residuals are found from the first regression.This regression is estimated by ordinary least squares (for example, Program 2.3). This value (here φ1 = .936) is then assumed to be fixed, which means that its estimation error is ignored in the next step in the Cochrane-Orcutt algorithm and in the final reported standard errors of the regression coefficients. This numerical algorithm works well as an approximation. But the method is inefficient, and the analysis is not telling the true story, even if the method in most cases ends up giving useful results.

Another problem is that the estimated first-order autocorrelation is not the maximum likelihood estimator for the autoregressive parameter, even if it is close to being so. The formula for the autocorrelation includes only T − 1 lags in the summation for the T observation. In practice, this is usually of no importance, and this estimate (also called the Yule Walker estimate) is often applied. But this estimator can be biased for parameter values close to the boundary value, |φ1| = 1.

A better solution is to consider the estimation of the autoregressive parameter and the regression coefficients as a simultaneous estimation problem. Estimating all parameters by the method of maximum likelihood is an optimal method. Nowadays, efficient algorithms exist, and all the calculated standard deviations and test statistics are valid under the assumption of a full model of a regression with first-order autoregressive errors. The next section includes an application of full maximum likelihood estimation using PROC AUTOREG.

Simultaneous Estimation Using PROC AUTOREG

The code in Program 3.4 allows for a first-order autoregressive model for the residuals, which is estimated together with the parameters in the linear regression model. The option NLAG=1 specifies that an autoregressive model of order 1 is fitted to the series of residuals. This model has the following form:

${\text{ε}}_t={\text{φ}}_1{\text{ε}}_{t-1}+{\text{ζ}}_t$

where remainder terms, ζt, are assumed uncorrelated. In this way, the observed autocorrelation in the original remainder terms, εt, is included in the estimation procedure. The autoregressive parameter is estimated in the same optimization as the regression coefficients by the option METHOD=ML. This maximum likelihood estimation is more efficient than the various approximation methods in Chapter 2, which were previously more commonly applied because of computer constraints.

Program 3.4: Inclusion of a Lagged Value of the Dependent Variable

This option changes the estimates. They are now estimated in a different model and therefore have a different interpretation. The parameter estimates are presented in Output 3.4, and the test for seasonal dummies is presented in Output 3.5. Note that the sign of the estimated autoregressive parameter in Output 3.4 is negative because PROC AUTOREG uses a parameterization that is different from the other procedures that are applied in this book. The parameterization used by PROC AUTOREG is as follows:

${\text{ε}}_t={\text{φ}}_1{\text{ε}}_{t-1}+{\text{ζ}}_t$

Output 3.4: Estimates Using Maximum Likelihood in a Model with AR(1) Errors

The estimated value of the autoregressive parameter φ₁ is numerically closer to the boundary value 1 than was the case in the previous estimations. This is because the full maximum likelihood estimation is less biased toward zero than the two-stage estimation algorithm in the Cochrane-Orcutt algorithm.

Output 3.5: Simultaneous Test for Seasonality in a Model with AR(1) Errors

The model fit, according to the autocorrelation function of the error terms (Figure 3.2), is now more acceptable than it would be without using an autoregressive model for the error process in the model. However, the significant autocorrelation at lag 2 indicates that more lags should be included in the autoregressive model for the error process.

Figure 3.2: The Residual Autocorrelation Function for the Model with AR(1) Errors

Conclusion

In many practical applications, the purpose of the analysis is just a variant of the following formulation: Estimate a linear relation between the number of cows and the production of milk, adjusting for eventual seasonal factors. For this simple case, the regression coefficient, β, using this ordinary least squares estimation, equals 17.99, with a standard error of 4.83 (see Output 2.2). The standard deviation is reduced to 4.41 when calculated by the Newey-West method, which adjusts the covariance matrix estimator to account for autocorrelation as in Output 3.1. In this setup, we have R² = .25.

In a model that includes a time series structure of the error process in the form of a first-order autoregressive model, the estimate is β = 7.05, with a standard error of 1.96, which is a completely different result. In this case, it is R² = .98, which is much larger than in the model estimated by ordinary least squares. This large value of R² is found by comparing the variance of the dependent variable to the variance of the second-order remainder term, ζt, which is much smaller than the variance of  the residuals, εt, of the first ordinary least squares regression. According to this example, considering the possible autocorrelation in error terms might significantly improve the fitting for time series data.

An important point for consideration is whether such conclusions are valid. According to the setup, both results at first glance seem valid, but how valid are the underlying assumptions? Is the data-generating process really an ordinary regression with a residual process in the form of an autoregressive process? Or is it too simple to model these two time series by an ordinary regression having some type of autoregressive error? In most cases, the answer is no to both possibilities.

One “red flag” tells you that something is wrong: This is the result of the estimated autoregressive parameter in the maximum likelihood estimation in Program 3.4. This estimated value, φ1 = .996, is very close to the boundary value φ1 = 1, which corresponds to a unit root that will be considered in more detail in the following chapters.