Chapter 9 Multivariate Linear Models

9.1 Introduction

9.2 A One-Way Multivariate Analysis of Variance

9.3 Hotelling’s T² Test

9.4 A Two-Factor Factorial

9.5 Multivariate Analysis of Covariance

9.6 Contrasts in Multivariate Analyses

9.7 Statistical Background

9.1 Introduction

Although the methods presented in earlier chapters are sufficient for studying one dependent variable at a time, there are many situations in which several dependent variables are studied simultaneously. For example, in monitoring the growth of animals, a researcher might measure the length, weight, and girth of animals receiving different treatments. The goal of the experiment would be to see if the treatments had an effect on the growth of the animals. One way to determine this would be to use analysis of variance or regression methods to analyze the effects of the treatment on the length, weight, and girth of the animals. There are two problems with this approach. First, trying to interpret results produced by separate univariate (one variable at a time) analyses of each of the variables can be unwieldy. In the example with three dependent variables, this may not seem like a problem. But if you have ten or twenty dependent variables, the task could be substantial. Second, and more importantly, when many variables are studied simultaneously, they are almost always correlated—that is, the value of each variable may be related to the values of others. This is true for measurements such as height and weight, or responses to similar questions on a questionnaire. In cases like these, considering the univariate analyses separately would not take into account information contained in the data due to the correlation. Moreover, this approach could mislead a naïve researcher into believing that a factor has a very significant effect, when in fact it does not. On the other hand, a significant effect that only becomes apparent when all the dependent variables are studied simultaneously may not be discovered from the univariate analyses alone. In most multivariate data applications, you should usually examine the results of the multivariate tests first, then examine the univariate analyses cautiously if significant results do not appear in the multivariate analysis.

Although some of the details of a multivariate analysis differ from those of a univariate analysis, the two are similar in many ways. Experimental factors of interest are related to the dependent variables by a linear model, and functions of sums of squares are computed to test hypotheses about these factors. In general, if you have designed an experiment with only one dependent variable, the extension of the analysis to the multivariate case can be carried out in a very straightforward manner.

We present examples of several types of multivariate analyses. Basic theory for the methods is presented in Section 9.7, “Statistical Background.”

9.2 A One-Way Multivariate Analysis of Variance

Test scores from two exams taken by students with three different teachers are analyzed in Output 9.1.

Output 9.1 Two Exam Scores for Students in Three Teachers’ Classes

We first perform an analysis of variance to compare the teacher means for each variable, SCORE1 and SCORE2. Run the SAS statements for these analyses:

proc glm;
   class teach;
   model score1 score2=teach;

Results are shown in Output 9.2. There is no difference among teachers when considering a univariate analysis of either SCORE1 or SCORE2.

Output 9.2 Results of Univariate One-Way Analysis

 

The objective now is to see if a model using both scores shows a difference among teachers. The following SAS statements are used for this analysis:

  proc glm;
        class teach;
        model score1 score2=teach;
        manova h=teach / printh printe;

The first three statements produce the usual univariate analyses of the two scores as shown in Output 9.2. The MANOVA statement produces results in Output 9.3.

Output 9.3 Results of One-Way Multivariate Analysis: The MANOVA Statement

The PRINTH and PRINTE options cause the printing of the hypothesis and error matrices, respectively. In addition, the PRINTE option produces a matrix of partial correlation coefficients derived from the error SSCP matrix. This correlation matrix represents the correlations of the dependent variables corrected for all the independent factors in the MODEL statement.

The results in Output 9.3 are described below. The callout numbers have been added to the output to key the following descriptions:

➊ The elements of the error matrix. The diagonal elements of this matrix represent the error sums of squares from the corresponding univariate analyses (see Output 9.2).

➋ The associated partial correlation matrix. In this example it appears that SCORE1 and SCORE2 are highly correlated (r=0.945640).

➌ The elements of the hypothesis matrix, H. Again, the diagonal elements correspond to the hypothesis sums of squares from the corresponding univariate analysis.

➍ The characteristic roots and vectors of E^-1H. The elements of the characteristic vector describe a linear combination of the analysis variables that produces the largest possible univariate F-ratio.

➎ The four test statistics previously discussed. The values of S, M, and N, printed above the table of statistics, provide information that is used in constructing the F-approximations for the criteria. (For more information, see Morrison (1976).) All four tests give similar results, although this is not always the case. Note that the p-values for the “Hypothesis of No Overall TEACH Effect” are much lower for the multivariate tests than any of the univariate tests would indicate. This is an example of how viewing a set of variables together can help you detect differences that you would not detect by looking at the individual variables.

9.3 Hotelling’s T² Test

Consider a common situation in multivariate analysis. You have several different measurements taken on each of several subjects, and you want to know if the means of the different variables are all the same. For example, you may have used different recording devices to measure the same phenomenon, or you may have observed subjects under a variety of conditions and administered a test under each of the conditions. If you had only two means to compare, you could use the familiar t-test, but it is important to use an analysis that takes into account the correlations among the dependent variables, just as in the previous examples, even though there are no independent factors in the model—that is, no terms on the right side of the MODEL statement. In this situation you could use Hotelling’s T² test. As an example, consider the following data taken from Morrison (1976). Weight gains in rats given a special diet were measured at one (GAIN1), two (GAIN2), three (GAIN3), and four (GAIN4) weeks after administration of the diet. The question of interest is whether the rats’ weight gains stayed constant over the course of the experiment; in other words, were the mean weight gains of the rats the same at each of the four weeks? Output 9.4 shows the data.

Output 9.4 Data for Hotelling’s T² Test

Note that the following MODEL statement fits a model with only an intercept:

model gain1 gain2 gain3 gain4 = ;

Following this statement with a MANOVA statement to test the intercept tests the hypothesis that the means of the four weight gains are all 0. It does not test the hypothesis of interest—that the four means are equal. To test the hypothesis that the four means are equal, the dependent variables must be transformed in such a way that their transformed means being 0 will imply that the original means are equal for all four variables. It turns out that there are many different transformations that have this effect. (This problem is discussed in detail in Chapter 8, “Repeated-Measures Analysis.”) One simple transformation that achieves this goal is subtracting one of the variables from each of the other variables; in this example, the first gain could be subtracted from each of the other gains. You should understand that the only way all the means could be equal to each other is if each of these differences is 0 and that if all the means are equal, then all the differences must be 0. In this way, Hotelling’s T² test for equality of means can be performed using the MANOVA statement.

One way of producing the transformed variables necessary to perform Hotelling’s T² test is to produce new variables in a DATA step and to perform an analysis on the new variables. A quicker and more efficient way, however, is to use the M= option in the MANOVA statement. Using the M= option, you can perform an analysis on a set of variables that is a linear transformation of the original variables as listed on the left side of the equal sign in the MODEL statement.

The following SAS statements perform the appropriate analysis:

  proc glm data=wtgain;
        model gain1 gain2 gain3 gain4 = / nouni;
        manova h=intercept
               m=gain2-gain1, gain3-gain1, gain4-gain1
               mnames=diff2 diff3 diff4 / summary;

The NOUNI option in the MODEL statement suppresses the individual analyses of the gain variables. This is done because the multivariate hypothesis of equality of the four means is the hypothesis of interest. The SUMMARY option in the MANOVA statement produces analysis-oF-variance tables of the transformed variables. The MNAMES= option provides labels for these transformed variables; if omitted, the procedure uses the names MVAR1, MVAR2, and so on. The results appear in Output 9.5.

Output 9.5 Analysis of Transformed Variables: Hotelling's T2 Test

Although PROC ANOVA does not produce the Hotelling T² statistic, it produces the correct F-test and probability level. Note that all the multivariate test statistics result in the same F-values and that they are labeled as “Exact F Statistics.” These tests are always exact when the hypothesis being tested has only 1 degree of freedom, such as the hypothesis of no INTERCEPT effect in this example. In order to calculate the actual value of the T² statistic, the following formula can be used:

T² = (nobs − 1)((1 / λ) − 1)

In the data set, nobs is the number of observations and λ is the value of Wilks’s Criterion that is printed by PROC ANOVA. (See Section 9.7, “Statistical Background.”) In this case, the calculation leads to T² = (10 – 1)(1/0.2517 – 1) = 26.757.

9.4 A Two-Factor Factorial

The total weight of a mature cotton boll can be divided into three parts: the weight of the seeds, the weight of the lint, and the weight of the bract. Lint and seed constitute the economic yield of cotton.

In the following data, the differences in the three components of the cotton bolls due to two varieties (VARIETY) and two plant spacings (SPACING) are studied. Five plants are chosen at random from each of the four treatment combinations. Two bolls are picked from each plant, and the weights of the seeds, lint, and bract are recorded. The most appropriate error term for testing VARIETY, SPACING, and the interaction of the two is the variation among plants. In univariate analyses, the TEST statement is used to specify this alternative error term; for multivariate analyses, the error term is specified in the MANOVA statement or statements. (Each MANOVA statement can have, at most, one error term specified.)

The following SAS statements are used:

proc glm;
   class variety spacing plant;
   model seed lint bract=variety spacing variety*spacing
                         plant(variety spacing)/ss3;
   test h=variety|spacing e=plant(variety spacing);
   means variety|spacing;
   manova h=variety|spacing e=plant(variety spacing);

The data used in this analysis appear in Output 9.6.

Output 9.6 Data for Two-Way Multivariate Analysis

Note that the TEST statement is used to obtain the appropriate error terms. The results of the three univariate analyses appear in Output 9.7.

Output 9.7 Results of Two-Way Multivariate Analysis: Univariate Analyses

 

VARIETY has a statistically significant effect on SEED and LINT; no effects are statistically significant for BRACT.

The means for all levels and combinations of levels of VARIETY and SPACING produced by the MEANS statement appear in Output 9.8.

Output 9.8 Results of Two-Way Multivariate Analysis: The MEANS Statement

The results of the multivariate analyses appear in Output 9.9.

Output 9.9 Results of Two-Way Multivariate Analysis

 

The only highly significant effect is VARIETY, and all four multivariate statistics are the same because there is only one hypothesis degree of freedom.

9.5 Multivariate Analysis of Covariance

This section illustrates multivariate analysis of covariance using the data on orange sales presented in Output 7.6. Sales of two types of oranges are related to experimentally determined prices (PRICE) as well as stores (STORE) and days of the week (DAY). The analysis is expanded here to consider the simultaneous multivariate relationship of the price to both types of oranges.

The following SAS statements are used for the analysis:

proc glm;
   class store day;
   model q1 q2=store day p1 p2 / nouni;
   manova h=store day p1 p2 / printh printe;

Note that PROC ANOVA is not appropriate in this situation because of the presence of the covariates. The NOUNI option in the MODEL statement suppresses printing of the univariate analyses that are already shown in Chapter 7, “Analysis of Covariance.” Results of the multivariate analysis appear in Output 9.10.

Output 9.10 Results of Multivariate Analysis of Covariance

 

 

 

For the STORE effect, none of the statistics produce significant results. This is not totally consistent with univariate results where sales of the first type of oranges are nearly significant at the 5% level.

The DAY effect is quite significant according to all statistics, although there is a considerable difference in the level of significance (Pr > F) among the statistics. Also, there is one dominant eigenvalue, indicating that the trend in sales over days is roughly parallel for the two types of oranges. This can be verified by printing and plotting the least-squares means.

The results for the PRICE effects are relatively straightforward. Even though both P1 and P2 are significant for only one dependent variable in the univariate analyses, the multivariate analysis indicates that their effects are substantial enough to be significant overall.

9.6 Contrasts in Multivariate Analyses

Chapter 3, “Analysis of Variance for Balanced Data,” discusses how specialized questions concerning certain levels of factors in univariate analyses can be answered by using the CONTRAST statement to define a hypothesis to be tested. This same technique can be useful in multivariate analyses of variance. The GLM procedure prints output for CONTRAST statements as part of its multivariate analysis. As an example, consider the study described in Section 9.5, “Multivariate Analysis of Covariance.” Assume you want to know if Saturday sales differ from weekday sales, averaged across the two types of oranges. Because the levels for DAY are coded as 1 through 6 corresponding to Monday through Saturday, you need to construct a contrast that compares the average of the first five levels of DAY to the sixth. The following SAS statement is required:

contrast 'SAT. vs. WEEKDAYS' day .2 .2 .2 .2 .2 -1;

The label SAT. vs. WEEKDAYS appears in the output to identify the contrast. If this CONTRAST statement is appended to the program of the previous section, preceding the MANOVA statement, then Output 9.11 is produced. This statement must precede the MANOVA statement so that PROC GLM will know that the multivariate test for the contrast is wanted.

Output 9.11 Results of Multivariate Analysis: The CONTRAST Statement

The results of the multivariate tests indicate no significant overall difference between the Saturday and weekday sales for the two types of oranges (Pr > F=0.7322).

9.7 Statistical Background

The multivariate linear model can be written as

Y = XB + U

where

Y	is an nXk matrix of observed values of k dependent variables or responses. Each column corresponds to a specific dependent variable and each row to an observation.
X	is an nXm matrix of n observations on the m independent variables (which may contain dummy variables).
B	is an mXk matrix of regression coefficients or parameters. Each column of B is a vector of coefficients corresponding to each of k dependent variables, and each row contains the coefficients associated with each of m independent variables.
U	is the nXk matrix of the n random errors, with columns corresponding to the dependent variables.

The matrix of estimated coefficients is

$\widehat{\mathbf{B}}={\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\prime }\mathbf{Y}$

Each column of $\widehat{\mathbf{B}}$ is the vector of estimated coefficients that would be obtained by estimating coefficients for each response variable separately.

The partitioning of sums of squares is parallel to that developed in previous chapters except that the partitions consist of kXk matrices of sums of squares and crossproducts:

Sums of Squares	SSCP Matrix
TOTAL	ÝY
MODEL	${\widehat{\mathbf{B}}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{Y}$
ERROR	${\mathbf{Y}}^{\prime }\mathbf{Y}-{\widehat{\mathbf{B}}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{Y}$

In the univariate analysis of variance, the F-statistic is the statistic of choice in most cases for testing hypotheses about the factors being considered. Recall that this statistic is derived by taking the ratio of two sums of squares, one derived from the hypothesis being tested and the other derived from an appropriate error term. In multivariate linear models, these sums of squares are replaced by matrices of sums of squares and crossproducts. These matrices are represented by H for the hypothesis, corresponding to the numerator sum of squares, and E for the error matrix, corresponding to the denominator sum of squares. Since division of matrices is not possible, E^-1H is the matrix that is the basis for test statistics for multivariate hypotheses. Four different functions of this matrix are used as test statistics and are available in the GLM, ANOVA, and other multivariate procedures in SAS. Each of these statistics is a function of the characteristic roots (also known as eigenvalues) of the matrix E^-1H. In the formulas below, represents the characteristic roots.

Corresponding to each characteristic root is a characteristic vector, or eigenvector, that represents a linear combination of the dependent variables being analyzed. A function of the characteristic root, λ/(1 + λ), is the value of R² that would be obtained if the linear combination of dependent variables represented by the corresponding characteristic vector were used as the dependent variable in a univariate analysis of the same model. For this reason, that function of the characteristic root is sometimes called the canonical correlation. In the formulas below, $r_i^2$ represents the canonical correlations.

Hotelling-Lawley Trace		Pillai's Trace
= Tr(E-1H)		= Tr(H(H + E)-1)
= $\sum {{\displaystyle r}}_i^2/(1-{{\displaystyle r}}_i^2)$		= $\sum {{\displaystyle r}}_i^2$
= $\sum {\displaystyle \lambda }i$		= ∑ λi / (1 + λi
Wilks' Lambda		Roy's Greatest Root
= |E| / |H + E|		= max λi
= ${\displaystyle \prod (1-{{\displaystyle r}}_i^2})$
= ${\displaystyle \prod (1/(1+{{\displaystyle \lambda }}_i}))$

Not one of these criteria has been identified as being universally superior to the others, although there are hypothesized situations where one criterion may outperform the others. Because we generally do not know the exact form of the alternative hypotheses being studied, the decision of which test criterion to use often becomes a matter of personal choice. Wilks’ criterion is derived from a likelihood-ratio approach and appeals to some statisticians on those grounds.