Example with a Nonsignificant Interaction

The fictitious example presented here is a follow-up to the pilot study described in Chapter 12. The results of that pilot study suggest that scores on a measure of perceived investment significantly increase following a marriage encounter weekend. However (as described earlier), you could argue that those investment scores merely increased as a function of time, and not because of the experimental manipulation. In other words, the investment scores could have increased simply because the couples spent time together and this would have occurred with any activity, not just a marriage encounter experience.

To address concerns about this possible confound (as well as some others inherent in the one-group design), replicate the study as a two-group design. An experimental group again takes part in the marriage encounter program, and a control group does not. Repeated measures on perceived investment are obtained from both groups. (The design of this study is illustrated in Figure 13.2.) Remember that this two-group design is generally considered to be superior to the design described in Chapter 12; you would use it in place of that design in most instances.

Your primary hypothesis is that the experimental group will show a greater increase in investment scores at post-treatment and follow-up than the control group. This result would be confirmed by a significant GROUP × TIME interaction in the ANOVA. You also hypothesize (based on the results obtained from the pilot project described in Chapter 12) that the increased investment scores in the treatment group will still be found at follow-up. To determine whether the results are consistent with these hypotheses, it is necessary to check group means, review the results from the “omnibus” test of the interaction, and consult a number of post-hoc tests to be described later.

Writing the SAS Program

The criterion variable in this study is perceived investment size: the amount of time and effort that participants believe they have invested in their relationships (their marriages). Perceived investment will be assessed with a survey, and cumulative investment scores will be calculated so that higher scores indicate greater levels of investment. Assume that these scores are assessed on an interval scale and that responses to the scale have been shown to demonstrate acceptable psychometric properties (i.e., valid and reliable scale responses).

Creating the SAS Dataset

Because you will obtain investment scores at three points in time (Time 1, Time 2, and Time 3), you must create three SAS variables to contain these scores. Use the following approach:

• A SAS variable named PRE contains investment scores obtained at Time 1 (scores from the baseline survey).

• A SAS variable named POST contains investment scores obtained at Time 2 (from the post-treatment survey).

• A SAS variable named FOLLOWUP contains investment scores obtained at Time 3 (from the follow-up survey).

You need a number of additional variables to complete the analysis. Create the variable named PARTICIPANT to denote identification numbers (values will range from 01 through n, where n = the number of participants). In addition, create a variable named GROUP to designate membership in either the experimental or control groups. Code this variable so that a value of 1 indicates that the participant is assigned to the control group and a value of 2 indicates assignment to the experimental group.

You can use the following program to create a SAS dataset called MIXED that contains fictitious data from the mixed-design study:

 1     DATA MIXED;
 2     INPUT #1 @1    PARTICIPANT   2.
 3              @5    GROUP         1.
 4              @10   PRE           2.
 5              @15   POST          2.
 6              @20   FOLLOWUP      2. ;
 7     DATALINES;
 8     01  1    08   10    10
 9     02  1    10   13    12
10     03  1    07   10    12
11     04  1    06   09    10
12     05  1    07   08    09
13     06  1    11   15    14
14     07  1    08   10    09
15     08  1    05   08    08
16     09  1    12   11    12
17     10  1    09   12    12
18     11  2    10   14    13
19     12  2    07   12    11
20     13  2    08   08    09
21     14  2    13   14    14
22     15  2    11   11    12
23     16  2    07   08    07
24     17  2    09   08    10
25     18  2    08   13    14
26     19  2    10   12    12
27     20  2    06   09    10
28     ;
29     RUN;

Lines 8 through 27 of the preceding program include data from the study. The first column includes the participant number variable labeled PARTICIPANT. You can see that 20 participants provided data.

The second column of data (in column 5) indicates the condition or GROUP to which participants are assigned. You can see that data from participants in the control group appear on lines 8 through 17 (coded as 1), while the data from the participants in the experimental group appear in lines 18 through 27 (identified by 2).

The third column of data (in columns 10 and 11) lists reported investment scores obtained at Time 1. These scores are given the SAS variable name PRE. The fourth column of data (columns 15 and 16) lists values for the SAS variable POST (investment scores obtained at Time 2). Finally, the last column of data (columns 20 and 21) lists values for the SAS variable FOLLOWUP (investment scores obtained at Time 3).

Obtaining Descriptive Statistics with PROC MEANS

After entering the data, perform a PROC MEANS to obtain descriptive statistics from all variables. This serves two important purposes. First, scanning the n, minimum value, and maximum value for each variable provides an opportunity to check for obvious data entry errors. Second, you need the means and standard deviations for the variables to interpret any significant effects observed in the ANOVA results to be reviewed later. Using PROC MEANS is particularly important when analyzing data from a mixed-design study because the means for within-subjects variables are not routinely included in the output of PROC GLM when it is used to perform a mixed-design ANOVA.

You need means and other descriptive statistics for all three of your investment score variables: PRE; POST; and FOLLOWUP. You need the overall means (based on the complete sample) as well as the means by GROUP (i.e., you need the means on these three variables for the control group as well as the means for the experimental group). You can obtain all of this information by adding the following lines to the preceding SAS program:

1     PROC MEANS  DATA=MIXED;
2     RUN;
3
4     PROC SORT   DATA=MIXED;
5        BY GROUP;
6     RUN;
7
8     PROC MEANS  DATA=MIXED;
9        BY GROUP;
10    RUN;

Lines 1 and 2 of the preceding program request that PROC MEANS be performed on all variables for the complete sample. Lines 4 through 6 sort the dataset by the variable GROUP, and lines 8 through 10 request that the MEANS procedure be performed twice: once for the control group; and once for the experimental group.

The output produced by the previous program appears below as Output 13.1. Results of the first PROC MEANS (performed on the combined sample) appear on page 1 of this output. It is instructive to review the means from this page of the output to get a sense for any general trends in the data. Remember that the variables PRE, POST, and FOLLOWUP contain investment scores obtained at Times 1, 2, and 3, respectively. The PRE variable displays a mean score of 8.60, meaning that the average investment score was 8.60 just before the marriage encounter weekend. The mean score on POST shows that the average investment score increased to 10.75 immediately after the marriage encounter weekend, and the mean score on FOLLOWUP shows that investment scores averaged 11.00 two weeks following the program. These means seem to display a fairly large increase in investment scores from Time 1 to Time 2 suggesting that you might observe a significant effect for TIME when you review ANOVA results (presented later).

Output 13.1. Results of PROC MEANS

[View full size image]

[View full size image]

It is important to remember that, in order for your hypotheses to be supported, it is not adequate to merely observe a significant effect for TIME; instead, it is necessary that you obtain a significant TIME × GROUP interaction. Specifically, you must find that any increase in investment scores over time is greater among participants in the experimental group than among those in the control group. To see whether such an interaction has occurred, it is necessary to prepare a figure that plots data for the two groups separately and consult the appropriate statistical analyses. You can prepare the necessary figure by referring to the group means that appear on page 2 of Output 13.1. (A later section presents the appropriate statistical analyses.)

In Figure 13.8, the dashed line illustrates mean investment scores from the control group. These mean scores were obtained from the PRE, POST, and FOLLOWUP variables that appeared on the part of Output 13.1 labeled “GROUP=1.” The solid line in the figure illustrates mean scores for the experimental group. These scores are obtained from the section of Output 13.1 labeled “GROUP=2.” (For a review of how to prepare figures such as Figure 13.8 from a table of means, see Chapter 10.)

Figure 13.8. Mean Investment Scores from Output 13.1

[View full size image]

The general pattern of means plotted in Figure 13.8 does not suggest an interaction between TIME and GROUP. When two variables are involved in an interaction, the lines that represent the various groups tend not to be parallel to one another. So far, the lines for the control group and experimental group of Figure 13.8 do appear to be parallel. This might mean that the interaction is nonsignificant. However, the only way to be sure is to analyze the data and compute the appropriate statistical test. The next section shows how to do this.

Testing for Significant Effects with PROC GLM

The general form for the SAS program to perform a factorial ANOVA with one repeated-measures factor and one between-subjects factor is as follows:

PROC GLM  DATA=filename;
   CLASS  group-variable-name;
   MODEL  trial1  trial2  trial3... trialn = group-variable-name
          / NOUNI;
   REPEATED  trial-variable-name  #levels  CONTRAST (level#) /
SUMMARY;
RUN;

The actual SAS program needed to analyze this dataset is as follows:

1     PROC GLM  DATA=MIXED;
2        CLASS GROUP;
3        MODEL PRE POST FOLLOWUP = GROUP / NOUNI;
4        REPEATED TIME 3 CONTRAST (1) / SUMMARY;
5     RUN;

Notes Regarding the SAS Program

The analysis begins with the PROC GLM statement on line 1. The CLASS statement on line 2 identifies the variable that codes the between-subjects factor (i.e., the variable that codes the experimental group versus the control group). In this study, the between-subjects factor has the SAS variable name GROUP.

In the MODEL statement on line 3, the variables that contain the criterion variable scores are left of the equal sign. The number of variables equals the number of levels of the repeated-measures variable. These levels are represented as “trial1 trial2... trialn” in the general form. In this study, the repeated-measures variable has three levels (Time 1, Time 2, and Time 3), so the variables that represent these levels (PRE, POST, and FOLLOWUP) appear to the left of the equal sign.

Also in the MODEL statement, the variable that codes the between-subjects factor should be right of the equal sign. (This will be the same variable name listed in the CLASS statement.) In the present study, this between-subjects factor is GROUP.

The last entries in the MODEL statement are a slash (which indicates that options are to follow) and the NOUNI option. The NOUNI option suppresses the printing of certain univariate statistics that are of no interest in this analysis.

The REPEATED statement appears next in the program; in this statement, you must list a “trial-variable-name.” That is, you must create a new variable name to represent your repeated-measures factor. In this study, the levels of the repeated-measures factor were Time 1, Time 2, and Time 3 (represented as PRE, POST, and FOLLOWUP in the MODEL statement). It therefore follows that an appropriate name for the repeated-measures variable in this study might be TIME. Note that variable name TIME appears as the first entry on the REPEATED statement on line 4 of the preceding program.

The general form for the program shows that the next entry in the REPEATED statement should be “#levels,” a number that indicates how many levels are coded under the repeated-measures factor. In the present study, the repeated-measures variable has three levels (Time 1, Time 2, and Time 3), so the number 3 appears to the right of TIME in the REPEATED statement.

The next entry in the REPEATED statement is the “CONTRAST (level#)” option. In a repeated-measures analysis, contrasts are planned comparisons between different levels of the repeated-measures variable. The CONTRAST option allows you to choose the types of contrasts that will be made. The number that you specify in the place of “level#” identifies the specific level of the repeated-measures factor against which the other levels will be compared. The preceding program specifies “CONTRAST (1).” A “1” appears in the parentheses with this option, and this means that level 1 under the repeated-measures factor will be contrasted with level 2 and with level 3. In more concrete terms, this means that investment scores obtained at Time 1 will be contrasted with those obtained at Time 2, and with those obtained at Time 3. Unless otherwise instructed, PROC GLM automatically performs tests that contrast the last (nth) level of the repeated-measures variable with each of the preceding levels.

The preceding CONTRAST command requests that both the post-treatment and follow-up scores be compared to the baseline score. However, remember that you should interpret these tests only if the interaction effect is not significant and the TIME effect is significant. If the interaction effect is significant, then other post-hoc procedures are necessary.

Finally, the REPEATED statement ends with a slash and the SUMMARY option. This SUMMARY option asks for the statistics from the contrasts to appear in the output.

Results from the SAS Output

With LINESIZE=80 and PAGESIZE=60 in the OPTIONS statement (first line), the preceding program would produce five pages of output. This output appears here as Output 13.2. The information that appears on each page is summarized here:

• Page 1 provides level information for the between-subjects factor (GROUP, in this case).

• Page 2 provides the following:

  • the level of information for the repeated-measures factor (TIME, in this case);

  • the results of the multivariate significance test for the main effect of TIME;

  • the results of the multivariate significance test for the TIME × GROUP interaction.

• Page 3 provides the results of the significance test for the main effect of GROUP.

• Page 4 provides the following:

  • the results of the univariate significance tests for the main effect for TIME and for the TIME × GROUP interaction;

  • information related to the error term in the univariate significance tests that involve the repeated-measures factor (the error degrees of freedom, the Type III error sum of squares, and the error mean square);

  • two estimates of the epsilon statistic.

• Page 5 provides the results of the planned comparisons requested with the CONTRAST option.

Output 13.2. Results of the Two-Way Mixed-Design ANOVA with a Nonsignificant Interaction

[View full size image]

[View full size image]

[View full size image]

[View full size image]

[View full size image]

Steps in Interpreting the Output

1. Make Sure That Everything Looks Right

As always, review the output for possible signs of problems before interpreting the results. Most of these steps with a mixed-design ANOVA are similar to those used with between-subjects designs (e.g., check the number of observations listed on page 1 to make certain data from all participants were used in the analysis, and check the number of levels that appear under each predictor variable). Because most of these steps were already discussed in earlier chapters, they are not reviewed here.

2. Determine Whether the Interaction Term Is Statistically Significant

As discussed in Chapter 12, when an analysis included a repeated-measures factor, the SAS output includes results of univariate, modified univariate, and multivariate ANOVAs. The “Further Notes on Repeated-Measures Analyses” section from that chapter reviews some of the basic issues to consider when choosing between univariate versus multivariate statistics.

Interpretation of the current study is somewhat more complicated than the interpretation of the one-factor design described in Chapter 12. This is because the current mixed-design requires that you interpret the effects of both a between-subjects factor and an interaction in addition to the effect of the repeated-measures factor. To simplify matters, this chapter follows the same general procedure recommended in Chapter 10 in which you first check for interactions before proceeding to test for main effects and post-hoc analyses.

As discussed in Chapter 10, the first step in interpreting a two-factor ANOVA is to check the interaction effect. If the interaction effect is not statistically significant, then you can proceed with interpretation of main effects. You should interpret the univariate test of the interaction effect first; this univariate test appears on page 4 of Output 13.2. On the left side of page 4, find the heading “Source: TIME*GROUP”; the information relevant to the interaction appears in this section. Under the heading “F Value,” you can see that the univariate F for the TIME × GROUP interaction is 0.10. With 2 and 36 degrees of freedom, this F has an associated p value of .91, and is clearly nonsignificant.

The next step is to check the multivariate test for the interaction. In most analyses, the univariate and multivariate tests yield tests with similar p values. The results of these two approaches are likely to vary only if the assumptions for the univariate test are grossly violated, or possibly if the Time variables are highly intercorrelated. The results of the multivariate test for the interaction appear on page 2 of Output 13.2. Find the section on this page headed “MANOVA Test Criteria and Exact F Statistics for the Hypothesis of no TIME*GROUP Effect.” To the right of the heading “Wilks’ Lambda,” you can see that the MANOVA yielded an F value of 0.07 and a corresponding p value of approximately .94. Once again, you fail to reject the null hypothesis of a TIME*GROUP interaction.

The primary hypothesis for this study required a significant interaction but you now know that the present data do not support such an interaction. If you were actually conducting this research project, your analyses might terminate at this point. However, in order to illustrate additional steps in the analysis of data from a mixed-design, this section proceeds with the tests for main effects.

3. Determine If the Group Effect Is Statistically Significant

In this chapter, the term group effect is used to refer to the effect of the between-subjects factor. In the present study, the variable named GROUP represents this effect.

The group effect is of no real interest in the present investment-model investigation since support for the study’s central hypothesis required a significant interaction. Nonetheless, it is still useful to plot group means and review the statistic for the group effect in order to validate the methodology used to assign participants to groups. For example, if the effect for GROUP proves to be significant and if the scores for one of the treatment groups are consistently higher than the corresponding scores for the other (particularly at Time 1), it could indicate that the two groups were not equivalent at the beginning of the study. This might suggest that there was some type of bias in the selection and assignment processes. Such a finding could invalidate any other results from the study.

The significance test for the group effect appears on page 3 of Output 13.2. You can see that the obtained F value for the GROUP effect is only 0.23 which, with 1 and 18 degrees of freedom, is nonsignificant (the p value for this F is quite large at approximately 0.64). This indicates that there was not an overall difference between experimental and control groups with respect to their mean investment scores. This finding is also illustrated by Figure 13.8, which shows that there is very little separation between the line for the experimental group and the line for the control group.

4. Determine If the Time Effect Is Statistically Significant

In this chapter, the term time effect is used to refer to the effect of the repeated-measures factor. Remember that the variable TIME was used to code this factor. A main effect for TIME would suggest that there was a significant difference between investment scores obtained at one time and the investment scores obtained at least one other time during the study (e.g., that the scores obtained at Time 2 are significantly higher than those obtained at Time 1).

The univariate test for the TIME effect appears on page 4 of Output 13.2, under the section headed “Source: TIME.” Under the heading “F Value,” you can see that the univariate F for the TIME effect is 28.84. With 2 and 36 degrees of freedom, this F is significant at p < .01.

The multivariate test for the TIME effect appears on page 2 of Output 13.2, under the heading “MANOVA Test Criteria and Exact F Statistics for the Hypothesis of no TIME Effect.” To the right of “Wilks’ Lambda,” you can see that the multivariate F for the TIME effect is 19.35 which, with 2 and 17 degrees of freedom, is also significant at p < .01. Clearly, there is a significant effect for the TIME variable.

The group means plotted in Figure 13.8 are helpful in interpreting this TIME effect. The trend displayed by the means suggests that Time 2 scores can be significantly higher than Time 1 scores, and Time 3 scores can also be significantly higher than Time 1 scores. A later section shows how to interpret the results requested by the CONTRAST option to see if these differences are statistically significant.

5. Prepare Your Own Version of the ANOVA Summary Table

Table 13.1 summarizes the preceding analysis of variance:

^[*] p < .01

6. Review the Results of the Contrasts

The program presented earlier included the following REPEATED statement:

4        REPEATED TIME 3 CONTRAST (1) / SUMMARY;

The keyword CONTRAST in this statement is followed by a “1” in parentheses. This requests that level 1 under the TIME variable be contrasted with the other levels under TIME. In other words, it requests that investment scores observed at Time 1 be contrasted with investment scores at Time 2, and that Time 1 also be contrasted with Time 3.

The resulting contrast analyses appear on page 5 of Output 13.2. The analysis that compares Time 1 investment scores with Time 2 scores appears under the heading “Contrast Variable: TIME_2.” The information of interest appears to the right of the heading “MEAN.” You can see that this comparison results in an F ratio of 27.69, which is significant at p < .01. You can therefore reject the null hypothesis that there is no difference between Time 1 investment scores and Time 2 investment scores in the population. The nature of the means displayed in Figure 13.8 shows that Time 2 scores are significantly higher than Time 1 scores.

The analysis that compares Time 1 to Time 3 appears under the heading “Contrast Variable: TIME_3.” With an F ratio of 40.98, it is clear that investment scores at Time 3 are also significantly higher than Time 1 scores (p < .01).

Summarizing the Results of the Analysis

The results of this mixed-design analysis could be summarized using the standard statistical interpretation format presented earlier in this text:

This summary could be performed three times: once for the null hypothesis of no interaction in the population; once for the null hypothesis of no main effect for GROUP; and once for the null hypothesis of no main effect for TIME. As this format has been previously described, it does not appear again here.

Formal Description of Results for a Paper

Below is one way that you could summarize the present results for a research paper. Notice that the names of the independent variables (time and group) are not capitalized, although the first letter of each variable’s name is capitalized in the expression “Group × Time interaction.”

Results were analyzed using a two-way ANOVA with repeated measures on one factor. The Group × Time interaction was not significant, F(2,36) = 0.10, ns nor was the main effect for group significant, F(1,18) = 0.23, ns. However, this analysis did reveal a significant effect for time, F(2,36) = 28.84, p < .01. Post-hoc contrasts found that investment scores at post-treatment (F[1,18] = 27.69, p < .01) and follow-up (F[1,18] = 40.98, p < .01) were significantly higher than the scores observed at baseline or Time 1.