Example with a Significant Interaction
Remember that your initial hypothesis was that the investment scores would increase more for the experimental group than for the control group and that, to be supported, this hypothesis required a significant TIME × GROUP interaction. This section presents the results of an analysis of a different set of fictitious data: participant responses that will provide the desired interaction. You will see that it is necessary to perform several follow-up analyses when you obtain a significant interaction in a mixed-design ANOVA. The dataset for this appears below. Otherwise, the SAS program is the same as the previous example and need not be repeated here.
7 DATALINES; 8 01 1 10 09 10 9 02 1 12 13 12 10 03 1 10 10 10 11 04 1 09 09 09 12 05 1 08 08 08 13 06 1 12 13 12 14 07 1 10 09 10 15 08 1 08 09 10 16 09 1 11 12 11 17 10 1 12 11 12 18 11 2 10 14 13 19 12 2 07 12 11 20 13 2 08 09 11 21 14 2 13 14 14 22 15 2 11 11 12 23 16 2 07 08 08 24 17 2 09 08 10 25 18 2 08 13 14 26 19 2 11 12 12 27 20 2 06 09 10 28 ; 29 RUN;
Output 13.3 presents the results of an analysis in which the TIME × GROUP interaction is significant. The following section shows how to interpret the output and perform the necessary tests for simple effects.
Output 13.3. Results of Two-Way Mixed Design ANOVA with a Significant Interaction
Steps in Interpreting the Output
1. Determine Whether the Interaction Term Is Statistically Significant
After you scan the results in the usual manner to verify that there are no obvious errors in the program or dataset, check the appropriate statistics to see whether the interaction between the between-subjects factor and the repeated-measures factor is significant. The univariate results appear on page 4 of Output 13.3, in the section headed “Source: TIME*GROUP.” The F value for this interaction term is 7.76. With 2 and 36 degrees of freedom, the unadjusted p value for this F is less than .01 as is the adjusted p value (Greenhouse-Geisser adjustment). (See “Modified Univariate Tests,” in Chapter 12 for a discussion of when to report Greenhouse-Geisser adjusted p values.) The interaction is therefore significant.
This finding is consistent with the hypothesis that the relationship between the TIME variable and investment scores is different for the two experimental groups. To determine whether the experimental group shows greater increases in investments than the control group (as hypothesized), you must plot the interaction.
2. Plot the Interaction
To plot the interaction, you once again need to obtain mean investment scores for each of the two groups at each level of TIME. You can obtain these means by using PROC MEANS with a BY GROUP statement, as follows:
1 PROC SORT DATA=MIXED; 2 BY GROUP; 3 RUN; 4 5 PROC MEANS DATA=MIXED; 6 BY GROUP; 7 RUN;
Chapter 10 shows how you can use the results from this MEANS procedure to plot an interaction. The means from the present dataset appear on page 6 of Output 13.3, and are plotted in Figure 13.9.
Figure 13.9. Significant TIME × GROUP Interaction Obtained in the Analysis of the Investment Model Data
Notice that the line for the experimental group in Figure 13.9 (the solid line) is not perfectly parallel to the line for the control group (the dashed line). This is what you would expect with a significant TIME × GROUP interaction. You can see that the line for the control group is relatively flat. There does not appear to be much of an increase in perceived investment from Time 1 to Time 2 to Time 3. In contrast, notice the angle of the line for the experimental group. There is an apparent increase in investment size from Time 1 to Time 2, and another slight increase from Time 2 to Time 3.
These findings are consistent with the hypothesis that there would be a larger increase in investment scores in the experimental group than in the control group. To have confidence in this conclusion, you must test for simple effects.
3. Test for Simple Effects
When you test for simple effects, you determine whether there is a significant relationship between one predictor variable and the criterion for participants at one level of the second predictor variable. (The concept of simple effects first appeared in Chapter 10.)
For example, in the present study you might want to determine whether there is a significant relationship between TIME and the investment variable among those participants in the experimental group. If there is, you could state that there is a simple effect for TIME at the experimental group level of GROUP.
To understand the meaning of this, consider just the solid line in Figure 13.9 (i.e., the line that represents the experimental group). This line plots investment scores for the experimental group at three points in time. If there is a simple effect for TIME in the experimental group, it means that investment scores obtained at one point in time are significantly different from investment scores obtained at at least one other point in time. If the marriage encounter program really is effective, you would expect to see a simple effect for TIME in the experimental group (i.e., investment scores should improve over time in this group). You would not expect to see a simple effect for TIME in the control group since they did not experience the program.
Testing for simple effects following a mixed-design ANOVA is relatively straightforward. To test for the simple effects of the repeated-measures factor (TIME, in this case), it is necessary to do the following:
divide the participants into subgroups based on their classification under the group variable (GROUP, in this case);
perform a one-way ANOVA with one repeated-measures factor on just those participants in the first treatment group;
repeat this repeated-measures ANOVA for each of the remaining subgroups.
For the present study, this means that you first divide participants into experimental and control groups. Next, using data from just the experimental group, you perform a one-way repeated measures ANOVA in which investment scores are the criterion and TIME is the predictor variable. You then repeat this analysis using data from the control group. You would compute and interpret these ANOVAs according to the procedures described in Chapter 12.
The first step in this process is to divide participants into the experimental and control groups. You do this with the following statements:
1 DATA CONTROL; 2 SET MIXED; 3 IF GROUP=1 THEN OUTPUT; 4 RUN; 5 6 DATA EXP; 7 SET MIXED; 8 IF GROUP=2 THEN OUTPUT; 9 RUN;
Lines 1 through 4 of the preceding program create a new dataset called CONTROL. It begins by setting the dataset equal to the MIXED dataset but retains only those participants whose score on the GROUP variable is equal to 1 (see line 3). Therefore, the dataset called CONTROL contains only data from the control group. In the same way, you can see that lines 6 through 9 create a new dataset called EXP that contain only data from participants in the experimental group (i.e., those with a score on GROUP equal to 2).
Later in the same program, you will add the lines to test the simple effect for TIME in the control group. The following lines accomplish this:
1 PROC GLM DATA=CONTROL; 2 MODEL PRE POST FOLLOWUP = / NOUNI; 3 REPEATED TIME 3 CONTRAST (1) / SUMMARY; 4 RUN;
In line 1 of the preceding program, the DATA=CONTROL option requests that the analysis be performed using the dataset that contains only responses from the control group. To test this simple effect in the experimental group, it is necessary only to change the dataset that is named in this line:
1 PROC GLM DATA=EXP; 2 MODEL PRE POST FOLLOWUP = / NOUNI; 3 REPEATED TIME 3 CONTRAST (1) / SUMMARY; 4 RUN;
Therefore, the complete program (minus data input) that creates the two new datasets and performs the two tests for simple effects is as follows:
1 DATA CONTROL; 2 SET MIXED; 3 IF GROUP=1 THEN OUTPUT; 4 RUN; 5 6 DATA EXP; 7 SET MIXED; 8 IF GROUP=2 THEN OUTPUT; 9 RUN; 10 11 PROC GLM DATA=CONTROL; 12 MODEL PRE POST FOLLOWUP = / NOUNI; 13 REPEATED TIME 3 CONTRAST (1) / SUMMARY; 14 RUN; 15 16 PROC GLM DATA=EXP; 17 MODEL PRE POST FOLLOWUP = / NOUNI; 18 REPEATED TIME 3 CONTRAST (1) / SUMMARY; 19 RUN;
The results of these two ANOVAs would constitute tests of the simple effects of TIME for each level of GROUP. See Chapter 12 for guidelines in interpreting the results. The results of this analysis for the current study are presented here as Output 13.4.
Output 13.4. Results of Tests for Simple Effects of TIME among Participants in the Control Group (Output Pages 1 to 4) and Participants in the Experimental Group (Output Pages 5 to 8)
The test of the simple effect of TIME at the control group level of GROUP appears on pages 1 through 4 of Output 13.4. The results of the univariate test appear on output page 3, which shows an F value of 0.76. With 2 and 18 degrees of freedom, the simple effect for TIME in the control group is nonsignificant (p = .48).
The test of the simple effect of TIME at the experimental group level of GROUP appears on pages 5 through 8 of Output 13.4. The results for the univariate test appear on output page 7 which shows an F value of 11.96. With 2 and 18 degrees of freedom, the simple effect for TIME in the experimental group is significant (p < .01).
Since the simple effect for TIME is significant in the experimental group, you can now proceed to interpret the results of the planned contrasts that appear on page 8 of Output 13.4. These contrasts show that there is a significant difference between scores obtained at Time 1 versus those obtained at Time 2, F(1, 9) = 9.00, p < .05, and that there was also a significant difference between scores obtained at Time 1 versus Time 3, F(1, 9) = 19.74, p < .01.
Although the preceding seems fairly straightforward, there are actually two complications in the interpretation of these results. First, you are now performing multiple tests (one test for each level under the group factor), and this means that the significance level you have initially chosen will have to be adjusted so that your experiment-wise probability of making a Type I error does not get out of hand. Looney and Stanley (1989) have recommended dividing the initial alpha (.05 in the present study) by the number of tests to be conducted, and using the result as the required significance level for each test (also referred to as a Bonferroni correction). For example, assume that in this investigation your alpha is initially set at .05 (as per convention). You are performing two tests for simple effects; so, this initial alpha must be divided by 2, resulting in a revised alpha of .025. When you conduct your tests for simple effects, you conclude that an effect is significant only if the p value that appears in the output is less than .025. This approach to adjusting alpha is viewed as a rigorous adjustment; you might want to consider alternative approaches for protecting the experiment-wise error rate.
The second complication involves the error term used in the analyses. With the preceding approach, the error term used in computing the F ratio for a simple effect is the mean square error from the one-way ANOVA based on data from just one group. An alternative approach is to use the mean square error from the “omnibus” test that includes the between-subjects factor, the repeated-measures factor, and the interaction term (from Output 13.3). This second approach has the advantage of using the same yardstick (the same mean square error) in the computation of both simple effects. Some authors recommend this procedure when the mean squares for the various groups are homogeneous. On the other hand, the approach just presented in which different error terms are used in the different tests has the advantage (or disadvantage, depending upon your perspective) of providing a slightly more conservative F test since it involves fewer degrees of freedom for the denominator. For a discussion of alternative approaches for testing simple effects, see Keppel (1982, pp. 428–431), and Winer (1971, pp. 527–528).
So far, this section has discussed simple effects for only the TIME factor. It is also possible to perform tests for simple effects for the GROUP factor. You can test three simple effects for GROUP in the present study. Specifically, you can determine whether there is a significant difference between the experimental group and the control group with respect to the following:
investment scores obtained at Time 1;
investment scores obtained at Time 2;
investment scores obtained at Time 3.
To test a simple effect for GROUP, you simply perform a one-way ANOVA with one between-subjects factor. In this analysis, the criterion variable is the variable that includes investment scores at a specific point in time (such as Time 1). The predictor variable is GROUP.
For example, assume that you want to test the simple effect for GROUP with respect to investment scores taken at Time 1. The investment scores obtained at Time 1 are contained in a variable named PRE (as discussed earlier). The following program tests this simple effect:
1 PROC GLM DATA=MIXED; 2 CLASS GROUP; 3 MODEL PRE = GROUP ; 4 RUN;
To test the simple effect for GROUP with respect to investment scores observed at Time 2, it is only necessary to change the MODEL statement in the preceding program so that the variable PRE is replaced with the variable POST (Time 2 scores). Similarly, you can test the simple effect at Time 3 by using FOLLOWUP as the criterion variable. These lines perform these analyses:
5 PROC GLM DATA=MIXED; 6 CLASS GROUP; 7 MODEL POST = GROUP ; 8 RUN; 9 10 PROC GLM DATA=MIXED; 11 CLASS GROUP; 12 MODEL FOLLOWUP = GROUP ; 13 RUN;
The results produced by the preceding lines appear here as Output 13.5.
Output 13.5. Results of Tests for Simple Effects of GROUP for Data Gathered at Time 1 (Output pages 8 and 9), Time 2 (Output pages 10 and 11), and Time 3 (Output pages 12 and 13)
You can assess the simple effects for GROUP at the Time 1 level of TIME by reviewing pages 8 and 9 of Output 13.5. These pages show the results of the analysis in which PRE is the criterion variable. In the section for the Type III sum of squares at the bottom of output page 9, you can see that the F for this simple effect is only 1.98 which, with 1 and 18 degrees of freedom, is nonsignificant (p = .18). In other words, the simple effect for GROUP at the Time 1 level of TIME is nonsignificant.
By the same token, you can see that the simple effect for GROUP is also nonsignificant at Time 2 (F = 0.41, p = .53 from output page 11), and at Time 3 (F = 2.00, p= .17 from output page 13).
To step back and get the big picture concerning GROUP × TIME interaction discovered in this analysis, it becomes clear that the interaction is due to a simple effect for TIME at the experimental group level of GROUP. On the other hand, there is no evidence of a simple effect for GROUP at any level of TIME.
4. Prepare Your Own Version of the ANOVA Summary Table
Table 13.2 presents the ANOVA summary table from the analysis.
| Source | df | SS | MS | F | |
|---|---|---|---|---|---|
| Between Subjects | 19 | 157.40 | |||
| Group (A) | 1 | 0.27 | 0.27 | 0.86 | |
| Residual between | 18 | 157.13 | 8.73 | ||
| Within Subjects | 40 | 67.33 | |||
| Time (B) | 2 | 21.73 | 10.87 | 12.28[*] | |
| A × B Interaction | 2 | 13.73 | 6.87 | 7.76[*] | |
| Residual within | 36 | 31.87 | 0.86 | ||
| Total | 59 | 224.73 | |||
| Note: N = 20. | |||||
[*] p < .01
Formal Description of Results for a Paper
Results were analyzed using a two-way ANOVA with repeated measures on one factor. The Group × Time interaction was significant, F(2,36) = 7.76, p < .01. Tests for simple effects showed that the mean investment scores for the control group displayed no significant differences across time, F(2,18) = 0.76, ns. However, the experimental group did display significant increases in perceived investment across time, F(2,18) = 11.96, p < .01. Post-hoc contrasts showed that the experimental group has significantly higher scores at both post-test, F(1,9) = 9.00; p < .05 and follow-up, F(1,9) = 19.74; p < .01 compared to baseline.