C h a p t e r   9

Effects of a Quantitative on a Quantitative Variable

The Situation at INCU

Correlation Analysis

The Bivariate Platform and the Density Ellipse

Correlation Coefficient

Regression Analysis

Assumptions of Linear Regression

Regression Analysis in the Bivariate Platform

Summary

Chapter Glossary

Questions and Problems

Notes

In this chapter, we will examine the third component of the Fit Y by X platform, the Bivariate platform. This platform examines the effects of a quantitative variable on another quantitative variable. As the name implies, this platform examines two variables (X and Y) simultaneously. In statistical terms, this chapter will introduce correlation analysis and regression analysis. The distinction between correlation and regression analysis, although subtle, is important and is amplified in more advanced multivariate statistical methods. The material in this chapter will provide a foundation for study of the more advanced techniques, some of which will be covered in Chapter 11.

John Riggins, a recent MBA graduate, was just hired at INCU, and Hank knows that John has had several courses in statistics and financial analytics. He asks John to try to come up with some data that they can use to develop a way of predicting deposits based on member characteristics. John gathered information from a number of sources within INCU and came up with a set of variables that he thought might be useful in predicting deposits.

After a few weeks of determined work with databases, John tied together a sample of 50 members who had recently applied for loans at INCU and for which he could find information on income and debt. He also gathered information on age, education, how long they had been at their current job, and how long they had lived at their current address. Part of the data table that John built (Member Deposits.jmp) is shown in Figure 9.1.

Also, in correlation analysis, we are usually just interested in whether or not there is a relationship between the two variables with the correlation being a measure of the strength of that relationship. In regression, we are usually interested in making predictions about the Response (Y) based on the value of the Factor (X). The regression equation defines the relationship between the two variables in terms of an equation. Despite these differences, as we will see, in actual practice there is a good deal of overlap between regression and correlation, and they can both be accessed via the same platform in JMP.

Correlation analysis begins with a scatter plot. A scatter plot is simply a plot of the coordinates of the two values X and Y. We introduced scatter plots in Chapter 3 as a means of looking at the relationship between two quantitative variables. Figure 9.2 shows some different types of relationship patterns that might appear in a scatter plot. Plot (a) indicates a perfect positive relationship. The relationship is perfect because if we know the value of X, we can predict the value of Y exactly. Panel (b) indicates a strong positive relationship in that as X increases Y also tends to increase. Panel (c) indicates a strong negative relationship because as X increases, Y tends to decrease. Finally, panel (d) indicates no relationship since there is no discernable pattern in Y as X changes.

As you can see from the figure, it is not quite as easy to discern patterns with real data as it was with our artificial data. However, it does appear that there is a somewhat positive relationship between Deposits and Age.

To further explore the relationship between these two variables, we need to use some of the Bivariate options. Figure 9.4 shows the options available in the Bivariate platform. Most of the options relate to fitting lines to the points. We will return to these options shortly. The option we want now is the Density Ellipse option. This option fits an ellipse around the data that encompasses a certain percentage of the points. The percentage included in the ellipse depends on your choice of options shown in Figure 9.4. The most common selection is .95 to include 95% of the data that lies within the ellipse. Figure 9.5 shows the results after John selects the .95 option for the density ellipse. John knows from his use of JMP in his MBA program that an ellipse that is round like a circle indicates no relationship between the two variables. Otherwise, the direction of tilt in the ellipse will indicate whether the relationship is positive or negative. The ellipse in Figure 9.5 indicates a positive relationship between Age and Deposits. However, John knows that there is a numerical measure of the strength of relationship between two variables called a correlation that will give him a specific value to describe the relationship.

The correlation (r) is a measure of the strength of the relationship between two variables, and the value of r varies between -1.0 and +1.0. Negative values of r indicate a negative relationship, and positive values of r indicate a positive relationship. A value of 0 indicates no relationship between X and Y. Figure 9.5 shows that the correlation between Age and Deposits is about .34. To understand what this value means, it is instructive to look at how the correlation is calculated. The equation for the sample correlation is shown in Equation 9.1.

  (9.1)

The numerator of the correlation is called the covariance. The covariance measures the extent to which X and Y vary together or “co-vary.” Figure 9.6 may help explain what a covariance is. The area of the scatter plot is divided into four quadrants by lines at the mean of Y and the mean of X. The quadrants are labeled I, II, III, and IV.

In quadrant I, the Y values are above their mean, and the X values are below their mean so the product in the covariance will be negative (-). In quadrant II, the Y values are above their mean, and the X values are above their mean so that the product will be positive (+). In quadrant III, the Y values are below their mean, and the X values are also below their mean so that the product will be positive (+). In quadrant IV, the Y values are below their mean, and the X values are above their mean so the product will be negative. As you can see, in a positive relationship most of the points fall into quadrants II and III so that the sum of the products, the covariance, will be positive. Just the opposite will be true for a negative relationship. Therefore, it is clear that the covariance term determines the sign of the correlation since the denominator of the ratio can never be negative.

The covariance also determines the size of the correlation. If there is no correlation between X and Y, then points will fall randomly in all four quadrants, and the positives and negatives will cancel out leaving a sum of about zero. We will see the importance of the covariance again later when we discuss regression analysis.

It can be shown that this null hypothesis is tested with the familiar Student’s t distribution. The test statistic is

  (9.2)

where ρ is the hypothesized population correlation (usually 0), and the standard error is

  (9.3)

In figure 9.5, we saw that the correlation coefficient was .341. Next to the correlation, in the results window is a value labeled Signif. Prob, which is the p-value for this t-test of the correlation. In Figure 9.5, the p-value is .0153. Since this is less than .05, we can reject the null hypothesis and conclude that the population correlation between age and the size of the deposits at INCU is not zero. In statistical terms, we say that there is a (statistically) significant correlation between age and deposits.

1. A significant correlation does not mean that X causes Y. Correlations tell us that there is a relationship between the two variables, but the mere existence of a relationship does not tell us that one variable causes another. We cannot infer causality from simple correlations.
2. In some ways, the correlation is misleading as to the magnitude of the relationship. In the past two chapters, we discussed a measure called r², which was a measure of the percentage of variability captured by the model. We will discuss a similar measure in correlation and regression, which is the square of the correlation. This value, called the coefficient of determination, is a much better measure of the strength of the relationship, and it is always less than the correlation unless the correlation is 0 or 1. For example, consider a correlation of .7. This seems to indicate a strong relationship between X and Y, and in some cases this may well be the case. However, the relationship between X and Y still accounts for less than 50% of the variability in the Y values. In our example, the correlation of .341 means that the relationship between age and deposits accounts for approximately 12% of the variability in deposits. Later in this chapter, we will explain more about the calculation of r² and what it means.
3. Care should be taken in examining the data for outliers since the value of the correlation can be greatly influenced by unusual data points. We will return to this point in our discussion of regression analysis.

Regression analysis involves fitting a line to the points in the scatter plot so that the line is as “close” as possible to all of the points. There are two important concepts implicit in this statement.

1. What do we mean by line? There are a variety of lines that we could draw through the set of points. There are straight lines, parabolas, curved lines, etc. What kind of line are we going to draw? In this chapter, we will discuss only straight lines or linear regression.
2. What do we mean by close? In order to fit a line close to all of the points, we have to define what we mean by “close.” It is common in statistics to use the principle of least squares to determine the fit of a line to a set of points. If we let Y be the value of the Response column and Y′ be the point on the line, we can define distance of the point to the line as (Y – Y′)². The term (Y – Y′) is a very important concept and is called a residual. The principle of least squares states that we want to determine a line that minimizes the sum of the squared residuals.

1. In the population, X and Y are related by a linear equation with error

   

   where β₀ is the Y intercept or constant term, and β₁ is the slope of the line. The ε_i term indicates that there is error in the relationship for a given value of X_i. In other words, we cannot perfectly predict the value of Y for a given value of X_i, even if we have the entire population.³ The remaining assumptions relate primarily to the error term ε_i.
2. For each value of X_i, the mean of the error terms is zero. This means that, in the population, we can perfectly predict the average value of Y for a given value of X_i.

   

   where μ_Y|X is read as the mean of Y given X.
3. The standard deviation of the error terms (ε_i) is the same for all values of X.
4. For each value of X, the error terms (ε_i) form a normal distribution in the population.

Putting these assumptions together, we get a population like that illustrated in Figure 9.7 where for each value of X there is a normal distribution of values for Y centered on the regression line with a standard deviation σ, which is the same for all values of X.

The line of best fit shown in Figure 9.8 represents the sample regression equation

Y′ = b₀ + b₁X

where b₀ and b₁ are estimates of the corresponding population values β₀ and β₁. The Y intercept b₀ is the predicted value of Y when X is equal to zero. The slope b₁ is the change in Y for a unit change in X. The numerical values for these estimates are shown in the section Linear Fit. As shown in Figure 9.8, the equation is

Deposit = 6542.4375 + 179.43343*Age

We could then use this equation for making predictions by substituting a value for Age in the equation. For example, for a member age 40 we would predict

Deposit = 6542.4375 + 179.43343*(40) = $13,719.77

Prediction is often a primary goal of regression analysis, but you must be careful about predicting outside the range of values observed in the data. This is because you have not observed the relationship outside this range and therefore do not know what that relationships looks like for other values of the Factor. This makes prediction for values very far outside the range of observed values dangerous. Often the intercept is outside the range of observed values. This is why, in many cases, the intercept or constant term simply does not make sense. In our example, the literal interpretation would be that a member 0 years old would be predicted to have deposits of $6,542.44.

The slope, however, is always meaningful. The slope of 179.43343 in our example implies that each additional year in age would increase deposits by $179.43 on average.

The next term in the Summary of Fit section (Root Mean Square Error) is sometimes called the standard error of the estimate. However, root mean square error is a better term because this value is not a standard error in the traditional sense. Recall that we defined the term standard error in Chapter 5 as the standard deviation of a sampling distribution. The term being discussed here is a standard deviation, but it is not the standard deviation of any sampling distribution. The root mean square error can be calculated as

  (9.4)

Essentially, the root mean square error is the square root of the sum of the squared residuals divided by the degrees of freedom (n-2). This is a standard deviation, but the standard deviation of what? One of our regression assumptions was that in the population the standard deviation around the regression line was the same for all values of X. The root mean square error is an estimate of this common standard deviation. It is an estimate of the standard deviation of the Y values around the regression line in the population.

The final two values in the summary are self explanatory, the mean of the response column and the number of paired observations in the data.

SST = SSR + SSE   (9.5)

The analysis of variance table in Figure 9.8 shows the total variability of Deposit (929920480) broken down into SSR (108340099) plus SSE (821580380). If you look at the ratio of SSR to SST, you will find that 108340099/929920480 = .116505, which is the value of r². This is what we mean when we say that r² is the percentage of variability in the Response column that our regression equation predicts.

As usual, when we divide sum of squares by the corresponding degrees of freedom, we obtain mean squares. When we divide the mean square for regression by mean square error, we obtain an F value with a corresponding p-value. In this case, the p-value is .0153, which means the results are statistically significant. Notice that this p-value is exactly the same as the p-value we obtained for the t-test of the correlation earlier. We will return to this point a little later.

But what hypothesis is being tested here? This is basically a test of the fit of the overall model. Technically, the hypotheses being tested here are

H₀: ρ² = 0

H₁: ρ² ≠ 0

where ρ² is the population coefficient of determination. As with the correlation, the null hypothesis essentially states that there is no relationship between X and Y, and the alternative says that there is a relationship. Since we reject the null hypothesis, we can conclude that there is a statistically significant relationship between age and deposits.

This part of the output shows the values of the sample estimates b₀ and b₁ along with their standard errors. Dividing the estimates by their standard error terms gives a t ratio that can be tested using the usual Student’s t distribution. In most cases, we are not interested in testing hypotheses about the population Y intercept (β₀). One of the main reasons for this is that usually this value is well outside the range of values that we have observed. As discussed earlier, it is dangerous to speculate on values outside the range of values observed in our data.

The slope is our primary interest in regression analysis. The slope is closely related to the correlation coefficient examined in correlation analysis. The equation for the sample slope is shown in Equation 9.6.

  (9.6)

Comparing this equation with Equation 9.1 for the correlation, you can see that the numerator is exactly the same and is the covariance of X and Y. Since the covariance gives both the correlation and the slope their sign (positive or negative), the correlation and slope must have the same sign for any given set of data. Moreover, as we will see shortly, hypothesis tests for the two statistics are also very closely related. The usual form of the hypothesis tests for the population slope (β₁) is

H₀: β₁ = 0

H₁: β₁ ≠ 0

This test is conducted using a t-test

  (9.7)

where the normal hypothesized value of ß₁ is 0. Figure 9.10 shows a t ratio of 2.52 with a p-value of .0153. Notice that this is the same p-value as that of the F ratio in the analysis of variance table. You may also note that if you square the t value of 2.52, you will get the 6.3504, which is very close to the F ratio in the ANOVA table.⁴

The value of the sample slope, in this case 179.43, is also usually of interest. The interpretation of the slope is the change in the Response for a unit change in the Factor. In this case, the slope tells us that, for each additional year in age, deposits should increase by approximately $179.43.

You may wonder why we need three equivalent tests all testing basically the same hypothesis. The reasons will become clearer in Chapter 11 when we discuss multiple regression. In the more general case with multiple factors, the three tests are no longer equivalent.

The results for our example are shown in Figure 9.11. As you can see, the test is an F test and is reported in an ANOVA table. Note that the Total sum of squares here is the sum of squares error from the original ANOVA table. This is broken down into pure error and lack of fit error. The lack of fit error reflects variability that may be due to an incorrect model, in this case the linear model. Rejecting the null hypothesis here means that the linear model is not appropriate for our data and that we should use a different model. In this case, the p-value is .5413, which is much larger than .05. This indicates that the linear model is likely appropriate for our data. In our example, there are quite a few repeated values for the Factor (Age). This provides enough degrees of freedom for a good test of the lack of fit. In some situations, there may be very few repeat observations and, therefore, very few degrees of freedom for the lack of fit test. In these situations, one should not place too much reliance on this test.

Using the Fit Y by X platform to predict Y from X yields the results shown in Figure 9.13. The results show a negative slope in the regression equation, but the slope is not statistically significant, and the R² value is only about 3% (correlation of -.167). Looking at the scatter plot in Figure 9.13, we find that the outlier data point can be easily identified in the lower right corner; it is circled in the figure.

Now suppose we delete the outlier observation (observation 8 in the data table) and rerun the analysis. The results are shown in Figure 9.14. Here we can see that the slope of the regression line is positive and now predicts approximately 26% of the variability in the Response column (the correlation is .510). Notice that the slope and the correlation changed signs when the outlier was deleted and both increased greatly in value. This illustrates the impact that outliers can have on correlation and regression. We will return to this issue of outliers in Chapter 11 when we discuss regression diagnostics. Suffice it to state here that one should always examine the scatter plot produced by the Fit Y by X platform to see if there are any potential problems with outliers.

In regression analysis, the primary emphasis is on prediction of the Response using the Factor. Therefore, in regression, which variable is the Response or dependent variable and which is the Factor or independent variable is a critical distinction. Prediction is in terms of a linear equation whose sample form is Y′ = b₀ + b₁X. The parameter b₀ is called the Y intercept or constant, and b₁ is called the slope.

Regression analysis derives the numerical values by fitting a straight line to the points in the scatter plot using the principle of least squares. The principle of least squares fits the line so as to minimize the squared distance of the points from the line, the squared residuals. The summary fit measure for regression analysis is the coefficient of determination or r². The statistical significance of the regression equation can be tested using analysis of variance and the F ratio to test a hypothesis about the population coefficient of determination or with a Student’s t-test for the population slope. These two tests are equivalent and are both equivalents to the t-test for the population correlation in correlation analysis. If there are repeat values of the Factor column, then the regression analysis also produces a report that tests the fit of the model form, i.e., the linear form of the relationship.

2.  Is there a significant correlation between age and income? How would this relate to the first problem?

3.  Develop a regression equation to predict deposits from debt. Explain the slope of the regression line. What are the predicted deposits for someone with a debt of $11,000?

Bob Reed in Human Resources wonders if he can use correlation or regression to get a better handle on which factors drive salaries at INCU. Use the data table Employee Data.jmp to answer the following questions for Bob.

4.  Correlate age and seniority with salary. Which seems to be most strongly related to salary?

5.  Develop a regression equation predicting salary from age. How much of the variability in salary can your regression equation explain? What salary would you predict for someone 42 years old?

6.  Develop a regression equation predicting salary from seniority. Is the relationship statistically significant? What percentage of the variability in salaries can your equation explain? What would be the predicted salary for someone with 15 years of seniority?