Theory and logic: By far the most important test for either endogeneity or feedback is your careful consideration of what theory says about how the variables work (i.e. previously published theories of causes, effects and dynamics surrounding your main variables). If you are interested in customer loyalty, you should read the best sources (such as the Journal of Marketing in this case) on what the best thinkers in the field believe causes loyalty, what loyalty in turn affects, why, and how the inter-dynamics of these variables operate. If you don’t apply theory and logic, you are really just guessing when you throw variables together in a regression. You can also use your own logic in considering the relationships between variables. A further consideration is what previous studies have actually statistically found to be the case. If either theory, logic or overwhelming prior findings suggest either endogeneity or feedback, consider the alternatives discussed previously (structural equation modeling is probably best).

Correlations as a back-up diagnostic only: Another simple way to assess endogeneity (independent variables causing each other) is to look for moderate to strong correlations between independent variables. For instance, if you run the correlations between the major variables in the chapter case study, as seen in Chapter 8, you will note correlations between a few predictor variables, such as trust and enquiries (r = .57) and small and trust (r = -.50) which are moderate but not massive. Such correlations do not necessarily mean there is justification to infer inter-causation; you would only make that assumption in conjunction with theoretical justification. You can implement two-stage least squares in SAS through PROC SYSLIN. However, I would go for structural equation modeling first. Note that it’s harder to assess feedback loops via correlations.

When implementing the more complex techniques mentioned above, you can make formal statistical comparisons between the basic regression model and these more advanced models. Structural equation modeling is especially good for this. Such comparisons are assessments of whether endogeneity, feedback loops, and other complex models apply.

Remove one of the unimportant collinear predictors: One solution is to remove one of the offending variables altogether if it is not a core predictor variable – for instance, if organizational and job tenure are too collinear, perhaps use only one and remove the other (i.e. re-run the regression without one of them).

Combine similar predictors into aggregate variables: If two or more variables are collinear and they are very similar in construct, then perhaps consider aggregating such variables into a single construct if that option seems logical (see Chapter 4 and Chapter 9 on multi-item scales).

Adjustment: There are also other solutions, such as ridge regression, which you can read about in more advanced texts.

Non-linearity: When the true relational shape between a predictor and the dependent variables is a shape other than a straight line.

Heteroscedasticity: When the relationship could be linear but the residuals from the regression are not regular.

Nonlinearity is a fundamentally interesting finding, which suggests that your initial linear regression is not quite right as a model. Usually, you need to change the fundamental mathematical shape against which you are comparing the data, and compare the data against the new model.

Heteroscedasticity, on the other hand, is usually a less serious issue, because the correct underlying shape is a straight line. Usually, the sizes of the slopes – the crucial issue in regression – are not overly affected. The fact that the residuals are not even across the length of the regression line does, however, sometimes affect confidence intervals and p-values, therefore accuracy assessments.

Transformations: Using mathematical transformations of data can help in the case of some heteroscedasticity. For instance, in many cases researchers find that logging variables helps with certain types of heteroscedasticity. For an example, open, look at, and run “Code13b Regression logged” where I have logged all the continuous variables. There are many types of transformations that may help, and a lot more to learn on this topic – the interested reader should follow up in more advanced texts.

Rely on bootstrapped confidence intervals: Since heteroscedasticity only really affects the confidence intervals and p-values, using the bootstrap procedure described in Chapter 12 to get more accurate confidence intervals often mitigates the problem.

Weighted regression: There are various weighting procedures that can down-weight larger residuals. These procedures are often not quite as good as the right transformation and bootstrapping but can be useful.

Examination of residual graphs

Examination of outlier diagnostic statistics

The Cook’s D (“Cooks”) is a measure of influence. Cook’s D scores much larger than those below may suggest a large impact of an outlier on regression slopes, therefore that the observation may be influential. The diagnostic data set is automatically sorted by Cooks D values; therefore, look at those in the first few rows. A “spike” in the Cooks D value, compared to those a bit below it, possibly indicates that the observation is an influential point and may need to be dealt with (see below). For example, in Figure 13.22 Example of outlier diagnostics in SAS datasheet we see the Cooks D for the top observation has a quite substantially larger value than the one below. This might be an influential point.

The Studentized Residual (“Std_Resid”) is the standardized residual (extent to which the regression line misses the data). On this standardized scale, a studentized residuals score > +-2 possibly means that this observation is an influential point. In Figure 13.22 Example of outlier diagnostics in SAS datasheet we see the top two Cooks D values also have high studentized residuals of more than 3 in absolute value.

Leverage scores (also sometimes called “Hats”) give an overall idea of the extent to which the observation is an outlier on the independent variables only (the Hats scores do not take into account the dependent variable). There is also a formal suggested cut-off for possibly big hat scores[4]. Therefore, a very high Hat score may indicate that the outlier effect is somewhere in the independent variables, not the dependent variable. However, you may not always see strange scores in the actual predictor columns just because there is a big Hat score. Instead, a big Hat score may occur because of an abnormal combination of independent variable scores, not necessarily that any one is abnormal. Again, in Figure 13.22 Example of outlier diagnostics in SAS datasheet we see the top observation having what may be a high Hat score compared to others below it, although we would need to see the whole range.

Actual variable scores of the high-influence observations: For those variables at the top of the sorted dataset with the highest influence, examine their actual variable scores. Do they have unusual scores on one or a few variables? Make sure there are no data capturing mistakes (like a “0” for age). Is there a natural but very high score on one variable in particular or throughout? In Figure 13.22 Example of outlier diagnostics in SAS datasheet we see the top observation by Cooks D not having impossible data, but perhaps having some unusual combinations (it is a small firm with very high sales and enquiries but low satisfaction).

Robust regression: This is an adjustment protocol offered by some programs for outliers. Robust regression estimates which observations are outliers, and in some way down-weights influential outliers. Such down-weighting ranges from entirely deleting the observation to giving it a lesser influence on the final regression results. Luckily SAS has a great robust regression option (PROC ROBUSTREG), which can do further assessments of the influence of outliers and also give you new slopes that are adjusted for outliers, based on various approaches. Robust regression is popular and powerful, but like all regressions they have their own rules and diagnostics that help you decide whether to accept the final outcome. This book cannot cover all these considerations, but you should read the SAS helpfiles on ROBUSTREG if you want to use it in the future[5], and analyze the main robust regression output in the program.

Manual outlier weighting or deletion: You could just delete bad influential outliers (i.e. excluding the row of data that produces an outlier); however I would not do this normally unless you are forced to, and especially not if the observation is important (e.g. if you are studying IT companies you would NOT necessarily delete a huge player like Cisco just because it was an outlier!) Certain robust regression techniques actually delete certain observations anyway. If you delete observations yourself, compare the regression with and without deletions to see their effects. If interesting or important outliers need to be deleted to make the regression work, then analyze these critical data points separately as stand-alone case-studies.

Bootstrapping: The bootstrap procedure can also help ameliorate outliers to some extent, although only in their effect on accuracy estimations like confidence intervals. I still suggest a robust solution or judicious deletion.

The raw Durbin-Watson statistic: A rudimentary indication is that this statistic should be in the range of 2 to 4. If the statistic tends toward zero, it might indicate that autocorrelation exists. In Figure 13.24 The SAS Durbin-Watson section for autocorrelation it is about 2; therefore, it is alright. However, this test is not necessarily definitive; the p-values are better.

Durbin-Watson p-values: SAS gives a formal p-value test of the Durbin-Watson, where significant values indicate a potential autocorrelation issue – in Figure 13.24 The SAS Durbin-Watson section for autocorrelation neither p-value is significant, suggesting no issue with autocorrelation.

Residual patterns: Waves in the residual plot can indicate autocorrelation.

Basic dataset analysis: Chapter 4 and Chapter 9 have already discussed diagnosis of missing data in a dataset without any reference to multivariate analysis like regression. See the Textbook program “Code09a Gregs missing data analysis” designed especially for your use in this diagnosis.

Total loss of observations: The first table in the SAS PROC REG analysis also identifies the overall loss of observations, so this can act as an immediate diagnosis after you run an initial regression.

Comparison of original and remedial regressions: Identifying the extent of missing data does not actually help establish the extent to which it changes or affects the regression. This question can be a complex one, and there is much reading to be done for the interested practitioner. The next section discusses some remedial regression options, the results of which can be compared to an original ordinary least squares regression including the missing data. Major discrepancies between the findings may indicate that missing data is a big issue.

Missing data in observations (rows): As discussed in Chapter 9, you may consider getting rid of observations that are missing all data for the dependent variable and deleting observations that are missing lots of data overall, before you try other options.

Missing data in variables (columns): Once you are left with only the observations that have data on the dependent variable and most of the data for the rest of the variables, assess variables for missing data. Before implementing other options, consider excluding or substituting variables that have a significant amount of missing data, unless the variables are strictly required for the analysis.

Simple imputation: Some researchers replace missing data themselves. For instance, if the variable is continuous, the average might be substituted for the missing data. This approach is not very desirable, since it may amplify any bias in the missing data. (For instance, if many more men answer a certain question than women, replacing the female-dominant missing cells with the average will impute the qualities of men to the missing women – but what if the genders fundamentally differ in the population?)

More complex solutions: There are two more complex and effective solutions[7] that are leading solutions for missing data in regression: