12.2.1 Fitting a Linear Model

The main problem with a simple linear regression model is the estimation of unknown vector of regression coefficients and the variance of the error term . If we have estimates of the parameters, we can use the regression model for prediction purposes. For an intuitive discussion about the choice of the regression coefficients, the reader may refer to Chapter 6 of Tattar (2013).

For many statistical reasons, the parameters are estimated using the least squares method. The least-squares criterion is to find those values of and which will ensure that the sum of the squares for the difference between the actual values and over all the observations is minimized:

12.2

Differentiating the above equation with respect to and equating them to zero gives us the least-squares normal equations, and solving them further gives us the estimators for :

12.3

12.4

where

12.5

12.6

Of course, the term is known as the sum-of-squares term and as sum-of-cross-products term. Towards finding an estimate of the variance , we proceed along the following lines. We first define the model fitted values as the regressand value predicted by the fitted model . Next, we define residuals as the difference between the observed value and the corresponding model fitted value , that is, for :

12.7

Define the residual or error sum of squares, denoted by , by:

12.8

Since the residuals are based on observations, and the parameters and are estimated from it, the degrees of freedom associated with is . An unbiased estimator of is given by

12.9

It is thus meaningful that an estimator of the variance is the residual mean square. Using the estimator of variance, we can carry out statistical tests for the parameters of the regression line. Mathematically, the expressions for the variance of and are respectively:

Using the above expressions and the estimator of the variance of the error term, we estimate the standard error of and using:

12.10

12.11

Thus, if we are interested in investigating that the covariate has an effect of magnitude, say , we need to test the hypothesis against the hypothesis that it does not have an impact of magnitude , a useful test statistic is

12.12

which is distributed as a -distribution with degrees of freedom. Here we have used the notation to indicate that the hypothesis testing problem is related to the regression coefficient . In general, when we wish to test the hypothesis that the covariate has no effect on the regressand, the hypothesis testing problem becomes against the hypothesis . Thus, an -level test would be to reject the hypothesis if

where is the upper percentile point of the distribution.

Similarly, for the test statistic for the regression coefficient , or the intercept term, the hypotheses problem is against the hypothesis is

12.13

which is again distributed as an RV with a -distribution with degrees of freedom, and the test procedure parallels the testing of .

12.2.2 Confidence Intervals

Using the null distributions of and , equivalently and , the % confidence intervals of the slope and intercept are given by

12.14

and

12.15

Finally, we state that a percent confidence interval for is

12.16

We will illustrate all the above concepts for the Euphorbiaceae data.

12.2.3 The Analysis of Variance (ANOVA)

In the previous sub-section, we investigated whether the covariate has an explanatory power for the regressand. However, we would like to query whether the simple linear regression model 12.1 overall explains the variation in the actual data. An answer to this question is provided by the statistical technique Analysis of Variance, ANOVA. This tool is nearly a century old and Gelman (2005) provides a comprehensive review of the same and explains why it is still relevant and a very useful tool today. Here, ANOVA is used to test the significance of the regression model.

We will first define two quantities, total sum of squares, denoted by , and regression sum of squares, , similar to the residual sum of squares 12.8, next:

12.17

12.18

A straightforward algebraic manipulation step will give us the result that the total sum of squares is equal to the sum of regression sum of squares and residual sum of squares :

that is,

12.19

The degrees of freedom (df) for the three quantities are related as

The explicit values of df are obtained as

that is, .

Under the hypothesis , has a distribution. Since the distributions of is a distribution, and is independent of , we can use the -test for testing significance of the covariate. That is

12.20

is distributed as an -distribution with 1 and degrees of freedom. The test procedure is to reject the hypothesis if , where is the percentile of an -distribution with 1 and degrees of freedom. We summarize the ANOVA procedure in the form of an ANOVA Table 12.1.

Source of Variation	Sum of Squares	Degrees of Freedom	Mean Square	-Statistic
Regression		1
Residual
Total

The computations and illustration of ANOVA will continue with the euphorbiaceae example.

12.2.4 The Coefficient of Determination

We have thus seen two methods of investigating the significance of the covariate in the regression model. An important question, in the case of the covariate being significant, is how useful the covariate is towards explaining the variation of the regressand. Recall that the total sum of squares is given by and that the regression sum of squares is given in . Thus, the ratio gives an explanation of the total variation explained by the fitted regression model. This important measure is called the coefficient of determination, or the , and is defined as

12.21

A natural extension of the is quickly seen by choosing and in place of and , and this measure is more popularly known as Adjusted-:

12.22

There are some disadvantages of the measure , and this will be seen later in Section 12.4. We will first obtain these two measures for the euphorbiaceae tree.

12.2.5 The “lm” Function from R

The linear regression model discussed thus far has been put through a lot of rudimentary codes. The lm function helps to create the linear regression model in R through the powerful object of class formula. The tilde operator ∼ in the formula object helps to set up a linear regression model by allowing the user to specify the regressand on the left-hand side of the tilde ∼ and the covariates on its right-hand side. This operator has also been used earlier for various graphical methods, statistical methods, etc. In conjunction with the lm function, we will now set up many useful linear regression models.

The main idea of working with the rudimentary codes exercise is to give a programming guide through R, and also in parallel an understanding of the underlying theory with computations. We now see how the lm command can be used for regression analysis.

12.2.6 Residuals for Validation of the Model Assumptions

We have earlier defined residuals, which is the difference between the original values and the model fitted values. In the beginning of the section, we considered the model , and we also defined the residual for the fitted model as . We will have a preliminary look at some properties of the residuals.

Properties of the Residuals. The mean of the residuals is 0, that is,

The variance of the residuals is the mean residual sum of squares , or as seen earlier:

It can be proved that is an unbiased estimator for the variance of the linear regression model . Finally, we need to record that the residuals are not independent. These are some of the important properties of the residuals.

Semi-Studentized Residuals. We know that the residuals are zero-mean, non-independent random variables. It has been noted by researchers and practitioners that non-independence is not a major problem and we can continue to treat them as independent variables. Thus, the Studentization method is known here as the semi-Studentization method. The semi-Studentized residuals are then defined as

12.23

The residuals, including the semi-Studentized residuals, can be used to study departures in the simple linear regression models, adapted from Kutner, et al. (2005), on the following lines:

1. 1. The regression function is not linear.
2. 2. The error terms do not have constant variance, that is, for some .
3. 3. The error terms are not independent.
4. 4. The model fits all but one or a few outlier observations.
5. 5. The error terms are not normally distributed, .
6. 6. One or several important predictor variables have been omitted from the model.

As further stated in Kutner, et al. (2005), the diagnostics for the above problems may be visualized in the plots of residuals (or semi-Studentized residuals):

1. 1. Plot of residuals against predictor variable. If the linear regression model is appropriate, the residuals are expected to fall in a horizontal band around 0. The plot should be fairly random in the positive and negative residual range across the predictor variable .
2. 2. Plot of absolute or squared residuals against predictor variable. If the residuals vary in a systematic manner in the positive and negative residual values, such curvilinear behavior will be captured in the absolute or squared residuals plot against the predictor variable.
3. 3. Plot of residuals against fitted values. The purpose of residuals against fitted values is similar to the previous two plots. These plots also help to determine whether the model error has constant variance against the range of predictor variables.
4. 4. Plot of residuals against time. The plot of residuals against time, observation numbers in most cases, is useful to check for randomness of the errors. That is, this plot should resemble a random walk and not exhibit any kind of systematic pattern.
5. 5. Plots of residuals against omitted predictor variables. This plot will be more appropriate for the multiple linear regression model, Section 12.4.
6. 6. Box plot of residuals. In the presence of outliers, the box plot of residuals will reflect such observations beyond the whiskers.
7. 7. Normal probability plot of residuals. The assumption of normality for the errors is appropriately validated with the normal probability plot of the residuals.

From the graphical methods developed earlier in this book, we are equipped to handle the first six plots mentioned here. For the example of labor hours required for the lot size of a Toluca Company, we will illustrate these six plots.

The Normal Probability Plot. We need to explain the normal probability plot of residuals. In the normal probability plot, the ranked residual values are plotted against their expected value under the normality assumption. The normal probability plot is obtained in the following steps:

• Find the rank of each residual.

• The expected value of the smallest residual is given by

  12.24

  where denotes the cumulative distribution of a standard normal variate. The subscript is a standard notation in the subject of Order Statistics and denotes the smallest observation of .

• Plot the residuals against .

If the plot of ranked residuals against the expected rank residuals is a straight line, the theoretical assumption of normal distribution for the error term is a valid assumption. In the event of this plot not reflecting a straight line, we conclude that the normality assumption is not a tenable assumption. The normal probability plot will be illustrated for the Toluca company dataset.

12.2.7 Prediction for the Simple Regression Model

Most often the purpose of fitting regression models is prediction. That is, given some values of the predictor variables, we need to predict the output values. This problem is known as the prediction problem.

Suppose that is the value of the predictor variable of interest. A natural estimator of the true response using the least-squares fitted model is

To develop a prediction interval for the observation , we define

Using the linearity property of expectations and distribution theory, we can easily see that the random variable is normally distributed with mean 0 and variance

To develop the prediction confidence interval, we will again use as an estimator of the variance . Thus, the prediction interval is given by

12.25

Fine, we will simply use the predict inbuilt function of R for computations related to the prediction interval. For example, if we want to predict the number of labor hours required based on a lot size 85, we need to use the predict function which returns the fitted values and the 95% prediction interval:

> predict(tclm,newdata=data.frame(Lot_Size=85),interval ="prediction")
      fit      lwr      upr
1 365.833 262.2730 469.3931

12.2.8 Regression through the Origin

Practical considerations may require that the regression line passes through the origin, that is, we may have information that the intercept term is . In such cases, the regression model is

12.26

In this case, the least-squares criteria becomes and the normal equation leads to the estimator:

12.27

which in return gives us the fitted model . An unbiased estimator of the variance would be the following:

12.28

It is left as an exercise for the reader to obtain the necessary expressions related to the confidence interval, prediction interval, etc. In the next example, we will illustrate this concept through a dataset.

The residual plots play a central role in validating the assumptions of the linear regression model. We have explored the relevant R functions for obtaining them.

12.4.1 Scatter Plots: A First Look

For the multiple regression data, let us use the R graphics method called pairs. The reader is recommended to try out the strength of this graphical utility with example(pairs). In a matrix of the scatter plot, a scatter plot is provided for each variable against the other variables. Thus, the output will be a symmetric matrix of scatter plots.

12.4.2 Other Useful Graphical Methods

The matrix of a scatter plot is a two-dimensional plot. However, we can also visualize scatter plots in three dimensions. Particularly, if we have two covariates and a regressand, the three-dimensional scatter plot comes in very handy for visualization purposes. The R package scatterplot3d may be used to visualize three-dimensional plots. We will consider the three-dimensional visualization of scatter plots in the next example. The discussion here is slightly varied in the sense that we do not consider a real dataset for obtaining the three-dimensional plots.

Contour plots are again useful techniques for understanding multivariate data through slices of the dataset. In continuation of the previous regression equations, from Example 13.4.3, let us view them in terms of a contour plot.

12.4.3 Fitting a Multiple Linear Regression Model

Given observations, the multiple linear regression model can be written in a matrix form as

12.34

where , , and , and the covariate matrix ¹ is

The least-squares normal equations for the multiple linear regression model is given by

which leads to the least-squares estimator

12.35

The model fitted values are obtained as

12.36

where we define

12.37

The matrix is called the hat matrix, which has many useful properties.

Properties of the Hat Matrix. The hat matrix is symmetric and idempotent, that is,

1. 1. , symmetric property.
2. 2. , idempotent property.
3. 3. Furthermore, is also symmetric and idempotent.

The hat matrix plays a vital role in determining the residual values, which in turn is very useful in model adequacy, as will be seen in the Section 12.5. Chatterjee and Hadi (1988) refer to the hat matrix as the prediction matrix. The importance of the hat matrix is that it serves as a bridge between the observed values and the fitted values. By definition

12.38

We had seen earlier, in Section 12.2, the role of the residuals in estimation of the variance . The results further extend here in the following manner:

12.39

Thus, the residual mean square is

12.40

As in Section 12.2, is an unbiased estimator of , that is,

12.41

12.4.4 Testing Hypotheses and Confidence Intervals

12.4.4.1 Testing for Significance of the Regression Model

We need to first assert if the linear relationship between the regressand and the predictor variables is significant or not. In the terms of this hypothesis, this translates into testing for against the hypothesis for at least one . Towards the problem of testing the hypotheses against , we define the following:

12.42

12.43

12.44

In Equation 12.42, we have . Computationally, we recognize that the regression sum of squares is obtained by exploiting the constraint , that is by . Under the hypothesis , follows a distribution, and follows a distribution. Furthermore, it is noted that and are independent. Thus, the test statistic for the hypothesis is the ratio of the mean of regression sum of squares to the mean of residual sum of squares, which is an -statistic. In summary, the test statistic is

12.45

which is an distribution. Now, to test , compute the test statistic and reject it if

The ANOVA table is hence consolidated in Table 12.3.

Source of Variation	Sum of Squares	Degrees of Freedom	Mean Square	-Statistic
Regression
Residual
Total

12.4.4.2 The Role of C Matrix for Testing the Regression Coefficients

The hypothesis tested above is a global test of model adequacy. If we reject the previous hypothesis, we may be interested in the specific predictors which are significant. Thus, there is a need to find such predictors.

The least-squares estimator is an unbiased estimator of , and further

12.46

Define . Then the variance of the estimator of the regression coefficient is , where is the diagonal element of the matrix , and the covariance between and is .

We are now specifically interested in testing the hypothesis against the hypothesis , . The test statistic for the hypothesis is

12.47

Under the hypothesis , the test statistic follows a -distribution with degrees of freedom. Thus the test procedure is to reject the hypothesis if .

Confidence Intervals

An important result, see Montgomery, et al. (2003), is the following:

12.48

This property allows us to construct the joint confidence region for the vector of regression coefficients as follows:

The confidence intervals, region actually, for the regression coefficients is given by

In the -dimensional space, the above inequality is a region of elliptical shape. As the next natural step, if we are interested in some specific predictor variable, the confidence interval for its regression coefficient is given by

12.49

Finally, consider a new observation point . A natural estimate of the future observation is . A prediction interval is given as

12.50

The theoretical developments for multiple linear regression model will now be taken to an R session.

The model validation task for the multiple linear regression model will be undertaken next.

12.5.1 Residuals

Standardized Residuals. Scaling the residuals by dividing them by their estimated standard deviation, which is the square root of the mean residual sum of squares, we obtain the standardized residuals as

12.51

Note that the mean of the residuals is zero and in that sense it is present in the above expression. The standardized residuals are approximately standard normal variates, and any of their absolute values greater than 3 is an indicator of the presence of an outlier.

Semi-Studentized Residuals. Recollect the definition of the hat matrix from Sub-section 13.4.3 and its usage to define the residuals as

12.52

Substituting , and on further evaluation, we arrive at

12.53

Thus, the covariance matrix of the residuals is

12.54

The last step follows, since is an idempotent matrix. As the matrix is not necessarily a diagonal matrix, the residuals have different variances and are also correlated. Particularly, it can be shown that the variance of the residual is

12.55

where is the diagonal element of the hat matrix . Furthermore, the covariance between the residuals and is

12.56

where is the diagonal element of the hat matrix. The semi-Studentized residuals are then defined as

12.57

PRESS Residuals. The PRESS residual, also called the predicted residual, denoted by , is based on a fit to data which does not include the observation. Define as the least-squares estimate of the regression coefficients obtained by excluding the observation from the dataset. Then, the predicted residual is defined by

The picture looks a bit scary so that we may need to fit models to obtain the predicted residuals. Even if we were willing to do this, for large values, it may not be feasible to run the computer for so many hours, particularly if we can avoid it. There is an interesting relationship between the predicted residuals and the residuals, which circumvents the troublesome route of fitting models. Equation 2.2.23 of Cook and Weisberg (1982) shows that the predicted residuals and the residuals are related by

12.58

The variance of the predicted residual is

12.59

The standardized prediction residual, denoted by , is obtained by

12.60

R-Student Residuals. Montgomery, et al. (2003) mention that the Studentized residuals are useful for outlier diagnostics and that the estimate of is an internal scaling of the residual, since it is an internally generated estimate of . Similar to the approach of prediction residual, we can construct an estimator of by removing the observation from the dataset. The estimator of by removing the observation, denoted by , is computed by

Using for scaling purposes, the R-Student residual is defined by

12.61

The utility of these residuals is studied through the next example.

12.5.2 Influence and Leverage Diagnostics

In the previous subsection, we saw the power of leave-one-out residuals. The residual plots help in understanding the model adequacy. We will now take measures to explain the effect of each covariate on the model fit and also of each output value on the same.

A large value of residual may arise on account of either a large value of the covariate or a large value of the output. If some observations are disparagingly distinct from the overall dataset and if the reason for this is the covariate value, then such points are called the leverage points. The leverage points do not affect the estimates of the regression coefficients, although they are known to drastically change the values. For the fourth quartet of the Anscombe dataset, refer to the scatter plot in Figure 12.8, any value of the regressand should correspond to the covariate value of 8. Hence, we may say that the eighth observation is a levarage point.

Another source of disparaging observations, for reasonable levels of the covariate values, may be from the output values. Such data points are called influence points. The influence points have a significant affect on the estimated values of the regression coefficients, and in particular tilt the model relationship towards them. For the third quarter of the Anscombe dataset, refer to the scatter plot in Figure 12.8, the outliers are clearly influential.

Leverage Points

We will recollect the definition of the hat matrix:

The elements of the hat matrix may be interpreted as the amount of leverage exerted by the observation on the fitted value . Note that the diagonal elements may be easily obtained as

12.62

This hat matrix diagonal is a standardized measure of the distance of the observation from the center of the -space. The average size of a hat diagonal is . Any observation for which the value exceeds twice the expected leverage of is considered as a leverage point.

Cook's Distance for the Influence Points

The Cook's distance for identifying the influence points is based on the leave-one-out approach as seen earlier. Let denote the fitted value, and denote the predicted value when the observation is removed from the model building, that is,

12.63

Then the Cook's distance for the observation is given by

12.64

The distance can be interpreted as the squared Euclidean distance that the vector of fitted values moves when the observation is deleted. The magnitude of the distance is assessed by comparing with . As a rule of thumb, any value of the distance may be called an influential observation.

DFFITS and DFBETAS Measures for the Influence Points

To understand the influence of the observation on the regression coefficient , the measure proposed by Belsley, Kuh and Welsh (1980) is given as

12.65

where is the diagonal element of the matrix, and is the estimate of the regression coefficient with the deletion of the observation. Belsley, Kuh, and Welsh suggest a cut-off value as .

Similarly, a measure for understanding the affect of an observation on the fitted value is given by . This measure is defined as

12.66

The relevant rule of thumb is a cut-off value of .

For a detailed list of R functions in the context of regression analysis, refer to the pdf file at http://cran.r-project.org/doc/contrib/Ricci-refcard-regression.pdf. The leverage points answer the problem of an outlier with respect to the covariate value. There may be at times linear dependencies among the covariates themselves, which lead to unstable estimates of the regression coefficients and this problem is known as the multicollinearity problem, which will be dealt with in the next section.

12.6.1 Variance Inflation Factor

The variance inflation factor for the variable is given by

12.67

where is the square of the multiple correlation coefficient from the regression of the explanatory variable on the remaining explanatory variables. The variance inflation factor of an explanatory variable indicates the strength of the linear relationship between the variable and the remaining explanatory variables. A rough rule of thumb is that variance inflation factors greater than ten give cause for concern.

The eigen system analysis provides another approach to identify the presence of multicollinearity and will be considered in the next subsection.

12.6.2 Eigen System Analysis

Eigenvalues were introduced in Section 2.4. The eigenvalues of the matrix help to identify the multicollinearity effect in the data. Here, we need to standardize the matrix in the sense that each column has a mean zero and standard deviation at 1. Suppose are eigenvalues of the matrix . Let and . Define the condition number and condition indices of respectively by

12.68

12.69

The multicollinearity problem is not an issue for the condition number less than 100, and moderate multicollinearity exists for the values of condition numbers in the range of 100 to 1000. If the condition number exceeds the large number of 1000, the problem of linear dependence among the covariates severely exists.Similarly, any condition index in excess of 1000 is an indicator of the almost linear-dependence of the covariates, and in general if the condition indices are greater than 1000, we have number of linear dependencies.

The decomposition of the matrix through the eigenvalues gives us the eigen system analysis in the sense that the matrix is decomposed with

12.70

where is a diagonal matrix with diagonal elements consisting of the eigenvalues of and is a orthogonal matrix whose columns consists of the eigenvectors of . If any of the eigenvalues is closer to zero, the corresponding eigenvector gives away the associated linear dependency. We will now continue to use these techniques on the US crime dataset.

12.7.1 Linearization

As seen in the introduction, linearity is one of our basic assumptions. In many cases, it is possible to achieve linearity by an application of transformation. Many models which have non-linear terms may be transformed to a linear model by a suitable choice of transformation. See Table 6.1 of Chatterjee and Hadi (2006), or Table 5.1 of Montgomery, et al. (2003). For example, if we have the model , by using a log transformation, we obtain . The transformed model is linear in terms of its parameters. Figure 12.10 displays the behavior of for various choices of and (before the transformation).

> x <- seq(0,5,0.1)
> alpha <- 1
> par(mfrow=c(1,2))
> plot(x,y=alpha*xˆ{beta=1},xlab="x",ylab="y","l",lwd=1)
> points(x,y=alpha*xˆ{beta=0.5},"l",lwd=2)
> points(x,y=alpha*xˆ{beta=1.5},"l",lwd=3)
> points(x,y=alpha*xˆ{beta= -0.5},"b",lwd=2)
> points(x,y=alpha*xˆ{beta= -1},"b",lwd=3)
> points(x,y=alpha*xˆ{beta= -2},"b",lwd=2)

We will next consider an example where a transformation achieves linearization.

12.7.2 Variance Stabilization

In general, the transformation stabilizes the variance of the error term. If the error variance is not constant for different values, we say that the error is heteroscedastic. The problem of heteroscedasticity is usually detected by the residual plots.

Technically, we had built the model . The choice of transformation is not obvious. Moreover, it is difficult to say whether the power of is appropriate or some other positive number. Also, it is not always easy to choose between the power transformation and logarithmic transformation. A more generic approach will be considered in the next subsection.

12.7.3 Power Transformation

In the previous two subsections, we were helped by the logarithmic and square-root transformations. Box and Cox (1962) proposed a general transformation, called the power transformation. This method is useful when we do not have theoretical or empirical guidelines for an appropriate transformation. The Box-Cox transformation is given by

12.71

where . For details, refer to Section 5.4.1 of Montgomery, et al. (2003). The linear regression model that will be built is the following:

12.72

For the choice of away from 0, we achieve variance stabilization, whereas for a value closer to 0, we obtain an approximate logarithmic transformation. The exact inference procedure for the choice of cannot be taken up here, and we simply note that the MLE technique is used for this purpose.

12.8.1 Backward Elimination

The backward elimination method is the simplest of all variable selection procedures and can be easily implemented without a special function/package. In situations where there is a complex hierarchy, backward elimination can be run manually while taking account of what variables are eligible for removal. This method starts with a model containing all the explanatory variables and eliminates the variables one at a time, at each stage choosing the variable for exclusion as the one leading to the smallest decrease in the regression sum of squares. An -type statistic is used to judge when further exclusions would represent a significant deterioration in the model. The algorithm of backward selection is as follows:

1. 1. Start with all the predictors in the model.
2. 2. Remove the predictor with highest -value greater than critical.
3. 3. Refit the model and go to 2.
4. 4. Stop when all -values are less than alpha critical.

The alpha critical is sometimes called the p-to-remove and does not have to be 5%. If prediction performance is the goal, then a 15 to 20% cut-off may work best, although methods designed more directly for optimal prediction should be preferred.

The above example is really taxing and there is a need to refine this before obtaining the final result. A customized R function which will achieve the same result is given next.

# The Backward Selection Methodology
pvalueslm <- function(lm) {summary(lm)$coefficients[,4]}
backwardlm <- function(lm,criticalalpha) {
 lm2 <- lm
 while(max(pvalueslm(lm2))>criticalalpha) {
 lm2 <- update(lm2,paste(".∼.-",attr(lm2$terms,"term.labels") [(which(pvalueslm(lm2)
              ==max(pvalueslm(lm2))))-1],sep=""))
 }
 return(lm2)
 }

12.8.2 Forward and Stepwise Selection

The forward selection method reverses the backward method. This method starts with a model containing none of the explanatory variables and then considers variables one by one for inclusion. At each step, the variable added is the one that results in the biggest increase in the regression sum of squares. An -type statistic is used to judge when further additions would not represent a significant improvement in the model.

1. 1. Start with no variables in the model.
2. 2. For all predictors not in the model, check their -value if they are added to the model. Choose the one with lowest -value less than alpha critical.
3. 3. Continue until no new predictors can be added.

For a function similar to backwardlm, refer to Chapter 6 of Tattar (2013) for an implementation of the forward selection algorithm. Stepwise regression is a combination of forward selection and backward elimination. This addresses the situation where variables are added or removed early in the process and we want to change our mind about them later. Starting with no variables in the model, variables are added as with the forward selection method. Here, however, with each addition of a variable, a backward elimination process is considered to assess whether variables entered earlier might now be removed, because they no longer contribute significantly to the model.

We will now look at another criteria for model selection: Akaike Information Criteria, abbreviated as AIC. Let the log-likelihood function for a fitted regression model with covariates be denoted by . The total number of estimated parameters is denoted by . The AIC for the regression model is then given by

12.73

where . The term is referred to as the penalty term. The model which has the least AIC value is considered the best model. For more details, refer to Sheather (2009). The use of AIC for forward and stepwise selection will be illustrated in the following.

12.9.1 Early Classics

Draper and Smith (1966–98) is a treatise on applied regression analysis. Chatterjee and Hadi (1977–2006) and Chatterjee and Hadi (1988) are two excellent companions for regression analysis. Bapat (2000) builds linear models using linear algebra and the book gives the reader an indepth knowledge of the necessary theory. Christensen (2011) develops a projective approach for linear models. For firm foundations in linear models, the reader may also use Rao, et al. (2008). Searle (1971) is a classic book, which is still preferred by some readers.

12.9.2 Industrial Applications

Montgomery, et al. (2003) and Kutner, et al. (2005) provide comprehensive coverages of linear models with dedicated emphasis on industrial applications.

12.9.3 Regression Details

Belsley, Kuh and Welsh (1980), Fox (1991), Cook (1998), Cook and Weisberg (1982), Cook and Weisberg (1994), and Cook and Weisberg (1999) are the monographs which have details about regression diagnostics. For a robust regression model, the reader will find a very useful source in Rousseeuw and Leroy (1987).

12.9.4 Modern Regression Texts

Andersen and Skovgaard (2010) consider many variants of regression models which have linear predictors. Gelman and Hill (2007) develop hierarchical regression models. Freedman (2009) is an instant classic which lays more emphasis on matrix algebra. Sengupta and Rao (2003), Clarke (2008), Seber and Lee (2003), and Rencher and Schaalje (2008) are also useful accounts of the linear models.

12.9.5 R for Regression

Fox (2002) is an early text on the use of R software for regression analysis. Faraway (2002) is an open source book which has detailed R programs for linear models.