Example: Significant Differences in Investment Size across Time

To demonstrate the use of the repeated measures ANOVA, this chapter uses a new fictitious experiment that examines a different aspect of the investment model (Rusbult, 1980). Remember from earlier chapters that the investment model is a theory of interpersonal attraction that describes the variables that determine commitment to romantic relationships and other interpersonal associations.

Designing the Study

Some of the earlier chapters have described fictitious investigations of the investment model that involved the use of fictitious partners: written descriptions of potential romantic partners that the participants responded to as if they were real people. Assume that critics of the previous studies are very skeptical about the use of this analogue methodology and contend that investigations using fictitious partners do not generalize to how real individuals actually behave in the real world. To address these criticisms, this chapter presents a different study that could be used to evaluate aspects of the investment model while using actual couples.

This study focuses on the investment size construct from the investment model. Investment size refers to the amount of time, effort, and personal resources that an individual has put into his or her relationship with a romantic partner. People report heavy investments in a relationship when they have spent a good deal of time with their romantic partner, when they have a lot of shared activities or friends that they would lose if the relationship were to end, and so forth.

Assume that it is generally desirable for couples to believe that they have invested a good deal of time and effort in their relationships. Assume that this is desirable because research has shown that couples are more likely to stay together when they feel they have invested a great deal in their relationships.

The “Marriage Encounter” Intervention

Given that higher levels of perceived investment size are generally a good thing, assume that you are interested in finding interventions that are likely to increase perceived investments in a marriage. Specifically, you have read research indicating that a program called “marriage encounter” is likely to increase perceived investments. In a marriage encounter program, couples spend a weekend together under the guidance of counselors, sharing their feelings, learning to communicate, and engaging in exercises intended to strengthen their relationship.

Based on what you have read, you hypothesize that couples’ perceived investments in their relationships will increase immediately after participation in a marriage encounter program. In other words, you hypothesize that, if couples are asked to rate how much they have invested in a relationship both before and immediately after the marriage encounter weekend, the “post” ratings will be significantly higher than the “pre” ratings. This is the primary hypothesis for your study.

However, being something of a skeptic, assume further that you do not expect these increased investment perceptions to endure. Specifically, you believe that if couples rate their perceived investments at a “follow-up” point two weeks after the marriage encounter weekend, these ratings will have declined to their initial “pre” levels that were observed just before the weekend. In other words, you hypothesize that there will not be a significant difference between investment ratings obtained before the weekend and those obtained three weeks after. This is the secondary hypothesis for your study.

To test these hypotheses, you conduct a study that uses a single-group experimental design with repeated measures. The design of this study is illustrated in Figure 12.1:

Figure 12.1. Single-Group Experimental Design with Repeated Measures

[View full size image]

Specifically, you recruit 20 couples, all of whom are about to go through a marriage encounter program. The criterion variable in this study is perceived investment, and you measure this variable with a multiple-item questionnaire. Higher scores on this scale reflect higher levels of perceived investment.

You obtain investment ratings from each couple at three points in time. These points are illustrated by the three circles in Figure 12.1. (The “D.V.” in each circle stands for “Dependent Variable.”) Specifically:

• A “baseline survey” obtains investment scores at Time 1, just before the marriage encounter weekend.

• A “post-treatment survey” obtains investment scores at Time 2, immediately after the encounter weekend.

• A “follow-up survey” obtains investment scores at Time 3, three weeks after the encounter weekend.

Notice that, in Figure 12.1, an “Experimental Treatment” appears between Time 1 and Time 2. This treatment is the marriage encounter program.

Problems with Single-Group Studies

Before proceeding, be warned that the study described here uses a relatively weak research design. To understand why, remember the main hypothesis that you would like to test: the hypothesis that investment scores would increase significantly from Time 1 to Time 2 because of the marriage encounter program. Imagine for a moment that you obtain just these results when you analyze your data, that Time 2 investment scores are significantly higher than Time 1 scores. Does this provide strong evidence that the marriage encounter manipulation caused the increase in investment scores?

Not really. It would be very easy for someone to provide alternative explanations for the increase. It could be argued that investment scores naturally increase over time among married couples regardless of participation in a marriage encounter weekend. Perhaps an increase in scores was due to some television program that most of the couples saw and was not due to the marriage encounter. There is a long list of alternative explanations that could potentially be offered.

The point is simply that repeated-measures studies must be designed very carefully to avoid confounds and other problems, and this study was not designed very carefully. It is used here merely for illustration. A later section in this chapter (“Sequence Effects”) discusses some of the problems associated with repeated-measures studies and reviews strategies for dealing with them. In addition, Chapter 13, “Factorial ANOVA with Repeated-Measures Factors and Between-Subjects Factors,” shows how you can make the present single-group design much stronger through the addition of a control group.

Predicted Results

Remember that your primary hypothesis is that couples’ ratings of investment in their relationships increases immediately following the encounter weekend. The secondary hypothesis is that this increase is a transient, or temporary, effect rather than a permanent change. These hypothesized results appear graphically in Figure 12.2. Notice that, in the figure, mean investment scores increase from Time 1 to Time 2, and then decrease again at Time 3.

You test the primary hypothesis by comparing the investment scores obtained at post-treatment (Time 2) to the baseline scores (Time 1). By also comparing the follow-up scores (Time 3) to the baseline scores (Time 1), you learn whether any change observed at post-treatment is maintained. You can use this comparison to test the secondary hypothesis.

Assume that this study is exploratory because studies of this type have not previously been undertaken and you therefore are uncertain of what methodological problems might be encountered or the magnitude of possible changes. This example might serve as a pilot study to assist in the design of a more definitive study and the determination of an appropriate sample size. (A follow-up study based on the pilot data from this project is presented in Chapter 13.)

Writing the SAS Program

Twenty couples participated in this study. To keep things simple, imagine that only the investment ratings made by the wife in each couple were actually analyzed. This means that your sample actually consists of data from the 20 women who participated in the marriage encounter program.^[1]

^[1] One assumption of this statistical procedure is independence of observations; thus, only one spouse per couple can be considered with use of one-way ANOVA with one repeated-measures factor (Kashy & Snyder, 1995). See “Assumptions Underlying the One-Way ANOVA with One Repeated-Measures Factor” at the end of this chapter.

The criterion variable in this study is the size of the investments that participants (the wives) believe they have made in their relationships. Investment size was measured by a scale. Assume that these scores are measured on an interval scale (see Chapter 1, “Basic Concepts in Research and Data Analysis”).

Creating the SAS Dataset

Enter the investment scores obtained at the three different points in time as three separate variables in the SAS dataset. You create a new variable named PRE to include investment scores obtained at Time 1, a variable named POST to include investment scores obtained at Time 2, and a variable named FOLLOWUP to include investment scores obtained at Time 3. A variable name ID (for “identification number”) denotes the participant number. The following program could be used to create a dataset named REP that contains data from the fictitious study:

 1     DATA REP;
 2        INPUT #1  @1    ID          2.
 3                  @5    PRE         2.
 4                  @9    POST        2.
 5                  @13   FOLLOWUP    2. ;
 6     DATALINES;
 7     01  08  10   10
 8     02  10  13   12
 9     03  07  10   12
10     04  06  09   10
11     05  07  08   09
12     06  11  15   14
13     07  08  10   09
14     08  05  08   08
15     09  12  11   12
16     10  09  12   12
17     11  10  14   13
18     12  07  12   11
19     13  08  08   09
20     14  13  14   14
21     15  11  11   12
22     16  07  08   07
23     17  09  08   10
24     18  08  13   14
25     19  10  12   12
26     20  06  09   10
27     ;
28     RUN;

The actual data from the study appear in lines 7 through 26 of the preceding program. The first column of data (in columns 1 and 2) includes each participant’s identification number. The next column of data (in columns 5 and 6) includes each participant’s score on PRE (the investment ratings observed at Time 1). The next column of data (in columns 9 and 10) includes each participant’s score on POST (the investment ratings observed at Time 2). Finally, the last column of data (in columns 13 and 14) includes each participant’s score on FOLLOWUP (investment ratings from Time 3).

Obtaining Descriptive Statistics with PROC MEANS

After inputting the dataset, you should perform a PROC MEANS to obtain descriptive statistics for the three investment variables. This serves two important purposes. First, scanning the sample size, minimum value, and maximum value for each variable provides an opportunity to check for obvious data entry errors. Secondly, you need the means and standard deviations for the variables to interpret significant differences found in the ANOVA. Performing this separate PROC MEANS is necessary because the means for within-subjects variables are not routinely included in the output of the PROC GLM program that performs the repeated-measures ANOVA.

Here are the lines that you can add to the preceding program to obtain simple descriptive statistics for the study’s variables:

29     PROC MEANS  DATA=REP;
30     RUN;

The output from this MEANS procedure appears in Output 12.1. After you review the columns headed “N,” “Minimum,” and “Maximum” to verify that there are no obvious errors in data entry, you should review the average investment model scores that appear under the heading “Mean.” Where the row headed “PRE” intersects with the column headed “Mean,” you can see that the mean investment score observed at Time 1 was 8.60. In the row headed “POST,” you can see that this average investment score had increased to 10.75 by Time 2. Finally, the row headed “FOLLOWUP” shows that the mean score had increased to 11.00 by Time 3. Figure 12.3 plots these means graphically:

Output 12.1. Results of PROC MEANS, Investment Model Study

Variable    N             Mean         Std Dev         Minimum          Maximum
-------------------------------------------------------------------------------
ID          20      10.5000000       5.9160798       1.0000000       20.0000000
PRE         20       8.6000000       2.1373865       5.0000000       13.0000000
POST        20      10.7500000       2.2912878       8.0000000       15.0000000
FOLLOWUP    20      11.0000000       2.0261449       7.0000000       14.0000000
-------------------------------------------------------------------------------

A review of the means presented in Figure 12.3 suggests that you might receive support for your study’s primary hypothesis, but apparently will not receive support for the study’s secondary hypothesis. Notice that mean investment scores seem to increase from the baseline survey (Time 1) to the post-treatment survey (Time 2). If subsequent analyses show that this increase is statistically significant, this finding would be consistent with your primary hypothesis that perceived investment would increase immediately following the marriage encounter weekend.

However, notice that the mean investment scores remain at a relatively high level at the follow-up survey, three weeks following the program (at Time 3). This trend is inconsistent with your secondary hypothesis that the increase in perceived investment would be short-lived.

At this point, of course, you are only “eyeballing” the data, and as yet it is not clear whether any of the differences that appear in Figure 12.3 are statistically significant. To determine this, you must analyze the data using a repeated-measures ANOVA. The next section shows how to do this.

Testing for Significant Effects with PROC GLM

The general form for the SAS program to perform a one-way repeated-measures ANOVA is as follows:

PROC GLM  DATA=filename;
   MODEL  trial1  trial2  trial3... trialn =  / NOUNI;
   REPEATED  trial-variable-name  #levels  CONTRAST (level#) /
SUMMARY;
RUN;

The actual SAS program that you need to analyze the dataset above is as follows:

1     PROC GLM  DATA=REP;
2        MODEL PRE POST FOLLOWUP =  / NOUNI;
3        REPEATED TIME 3 CONTRAST (1) / SUMMARY;
4     RUN;

Notes regarding the SAS Program

The repeated-measures ANOVA is similar to the between-subjects ANOVA in that both procedures can be performed with PROC GLM. However, notice that you write the MODEL statement differently in a repeated-measures analysis. In the repeated-measures ANOVA, the names of the variables that contain scores on the criterion variable appear to the left of the equals sign in the MODEL statement. In the general form of the program provided here, these variables are represented as “trial1 trial2 trial3... trialn”. In the present study, the variable PRE contains investment scores obtained at Time 1, POST contains investment scores obtained at Time 2, and FOLLOWUP contains investment scores obtained at Time 3. Therefore, PRE, POST, and FOLLOWUP appear to the left of the equals sign in the MODEL statement.

It should be clear that you need a different criterion variable to include scores obtained under each level of the independent variable. This means that the number of variables appearing to the left of the equal sign in the MODEL statement should equal the number of levels under your independent variable. The present study had three levels under the repeated-measures independent variable (Time 1, Time 2, and Time 3), so three variables appear to the left of the equal sign (PRE, POST, and FOLLOWUP).

In a between-subjects study, the name of the predictor variable would normally appear to the right of the equals sign in the MODEL statement. Since there is no between-subjects factor in the present study, no variable name appears to the right of the equals sign in the preceding program. Yet, the MODEL statement does include a slash (which signals that options are being requested) followed by the NOUNI option. The NOUNI option suppresses the printing of output relevant to certain univariate analyses that are of no interest with this design. Without the NOUNI option, the SAS program computes and prints a univariate F test for each of the three levels of the trial variable. These tests are not of interest when analyzing within-subjects effects.

The REPEATED statement appears in line 3 of the preceding program. The general form of the program indicates that the first entry in the REPEATED statement should be the “trial-variable-name.” This is a name that you supply to refer to your repeated-measures factor. In the present study, the three levels of your repeated-measures factor were “Time 1,” “Time 2,” and “Time 3.” It therefore makes sense to give this repeated-measures factor the name TIME. (Notice where the name TIME appears on line 3 of the preceding program.) Obviously, you could have used any variable name of your choosing, provided that it complies with the usual naming rules for SAS variables.

The general form of the program shows that, to the right of the trial variable name, you are to provide the “#levels.” This means that you should specify a number that represents the number of levels that appear under your repeated-measures factor. In the present study, the repeated-measures factor included three levels (Time 1, Time 2, and Time 3) and so the number “3” was entered next to TIME in the REPEATED statement.

The next entry in the REPEATED statement is the “CONTRAST (level#)” option. In a repeated-measures analysis, contrasts are planned comparisons between different levels of the repeated-measures variable. The CONTRAST option allows you to choose the types of contrasts that will be made. The number that you specify in the place of “level#” identifies the specific level of the repeated-measures factor against which the other levels will be compared. The preceding program specifies “CONTRAST (1).” A “1” appears in the parentheses with this option and this means that level 1 under the repeated-measures factor will be contrasted with level 2 and with level 3. In concrete terms, this requests that the mean investment scores obtained at Time 1 will be contrasted with those obtained at Time 2, and with those obtained at Time 3. Unless otherwise instructed, PROC GLM automatically performs tests that contrast the last (nth) level of the repeated-measures variable with each of the preceding levels.

The preceding CONTRAST command requests that both the post-treatment (Time 2) and follow-up scores (Time 3) be compared to the baseline score (Time 1). These are the appropriate contrasts for this analysis because these contrasts directly evaluate the two hypotheses of this study. However, remember that you should interpret these contrast tests only if the analysis of variance shows that the effect for TIME (the repeated-measures factor) is significant.

In addition to the planned contrasts previously described, there are a number of post hoc multiple-comparison tests that are available using SAS/STAT software such as the Tukey test and the Scheffe test. Many of these tests control for the experiment-wise probability of making a Type I error (i.e., incorrectly rejecting a correct null hypothesis). In Chapter 9, you learned how to use these tests to determine which pairs of groups are significantly different on the criterion variable. Technically, these tests can also be used in a repeated-measures ANOVA. However, these multiple comparison procedures cannot be applied to the repeated-measures variable when you use the REPEATED statement. Use of these procedures is limited to variables that appear in the MODEL statement. If using one of these tests is essential, then you can run the repeated-measures ANOVA according to another method in which you do not use the REPEATED statement. See the section “Use of Other Post Hoc Tests with the Repeated-Measures Variable” in Chapter 13 for a description of this method.

Finally, the REPEATED statement ends with a slash and the SUMMARY option. This SUMMARY option requests the statistics from the contrasts to appear in the output.

In summary, the complete program (minus the DATA step) that provides means for the criterion variable and performs the repeated-measures ANOVA is as follows:

1     PROC MEANS  DATA=REP;
2     RUN;
3
4     PROC GLM  DATA=REP;
5        MODEL PRE POST FOLLOWUP =  / NOUNI;
6        REPEATED TIME 3 CONTRAST (1) / SUMMARY;
7     RUN;

Results from the SAS Output

With LINESIZE=80 and PAGESIZE=60 in the OPTIONS statement (first line), the preceding program (including both PROC MEANS and PROC GLM) produces five pages of output. The information that appears on each page is summarized here:

• Page 1 (not shown) provides means and other descriptive statistics for the criterion variables (reproduced earlier as Output 12.1).

• Page 2 displays the number of observations included in the analyses performed by PROC GLM.

• Page 3 provides the multivariate significance test for the repeated-measures factor (TIME, in this case).

• Page 4 provides the univariate significance test for TIME along with information related to the error term used in this F test, and two estimates of the epsilon statistic (discussed later).

• Page 5 provides the results of the planned comparisons requested with the CONTRAST option.

Output 12.2 provides the results of the GLM procedure requested by the preceding program. (The results of PROC MEANS appeared earlier as Output 12.1 and are not reproduced here.)

Output 12.2. Results of PROC GLM, Investment Model Study

[View full size image]

[View full size image]

[View full size image]

[View full size image]

Steps in Interpreting the Output

1. Make Sure That Everything Looks Right

First, check the number of observations listed on page 2 of the output to make certain that data from all participants were included in the analysis. If any data are missing, then all data for that participant will automatically be dropped from the analysis. Next, check the number of levels assigned to your criterion variable on page 3 of the output. (The criterion variable is called “Dependent Variable” in the output.)

2. Review the Appropriate F Statistic and Its Associated Probability Value

The first step in actually interpreting the results of the analysis is to review the F value and associated statistics on page 4 (“Univariate Tests of Hypotheses for Within Subject Effects”) of the printout. The F value for the trial variable, as it was named in the REPEATED statement in the SAS program, appears here. This F value tells you whether to reject the null hypothesis. In the study, the null hypothesis is that, in the population, there is no difference in investment scores obtained at the three points in time. This hypothesis can be represented symbolically in this way:

H₀: T1 = T2 = T3

where T1 is the mean investment score obtained at baseline, T2 is the mean score following a marriage encounter weekend, and T3 is the mean score two weeks after the weekend.

To find the F ratio that tests this null hypothesis, look on page 4 of Output 12.2 in the section headed “Source: TIME.” Under the heading “F Value,” you can see that the F statistic for the TIME effect is 30.28. There are 2 and 38 degrees of freedom for this F test. (Find these degrees of freedom on the left side of the output page under the heading “DF.”)

Notice that there are actually three p values associated with this F test. The unadjusted p value appears under the heading “Pr > F.” Two adjusted p values appear under the heading “Adj Pr > F.” The use of these adjusted p values is explained in “Modified Univariate Tests,” later in this chapter.

For the sake of simplicity, consider only the unadjusted p value at this point. You can see that this unadjusted p value (under “Pr > F”) is very small at less than .01. Because this p value is less than .05, you can reject your null hypothesis of no differences in mean levels of commitment in the population. In other words, you conclude that there is an effect for your repeated-measures variable, TIME.

You might have noticed that page 3 of Output 12.2 provides a “MANOVA” test for the TIME effect. For the moment, ignore these results and focus only on the univariate test for the TIME effect. The multivariate (MANOVA) approach to conducting a repeated-measures analysis is discussed later in the section “Further Notes on Repeated Measures Analyses.”

3. Prepare Your Own Version of the ANOVA Summary Table

As with other ANOVAs, the next step is to formulate the summary table. This is somewhat easier with this analysis as most of the values that you need for the table appear on page 4 of Output 12.2. Table 12.1 illustrates the ANOVA summary table for the analysis just completed:

^[*] p < .01

4. Review the Results of the Contrasts

The significant effect for TIME revealed on page 4 of the output simply tells you that investment scores obtained at one point in time are significantly different from scores obtained at some other point in time. You still do not know which scores significantly differ. To determine this, you must consult the planned comparisons requested by the CONTRAST option. These appear on page 5 of Output 12.2.

Remember that you specified (in the SAS program with the CONTRAST option) that the mean from trial 1 (baseline) would be compared to each of the other two trials. Results of the analysis comparing trial 1 to trial 2 appear on the upper part of output page 5, below the heading “Contrast Variable: TIME_2.” It is this comparison that directly evaluates your primary hypothesis that investment increases significantly at post-treatment (Time 2) compared to baseline (Time 1). The F ratio for this contrast appears where the “MEAN” row intersects the column headed “F Value.” You can see that the obtained F value is 29.01 which, with 1 and 19 degrees of freedom, is significant at p <.01. This tells you that there was, in fact, a significant increase in investment scores immediately following the marriage encounter program. This is what you might have expected, after you reviewed the mean scores plotted earlier in Figure 12.3. This finding supports your primary hypothesis that investment scores would display a significant increase from Time 1 to Time 2.

The other contrast appears on the bottom half of page 5 of Output 12.2, beneath the heading “Contrast Variable: TIME_3.” This contrast compares baseline scores to the follow-up scores (taken three weeks after the program). The F value for this contrast is also statistically significant (F[1, 19] = 43.09, p < .01), indicating that the two treatment values are different. Inspection of the means in Figure 12.3 shows that the follow-up mean is greater than the baseline mean. Your second hypothesis, that the increase in investment would not be sustained, is therefore not supported. The increase in perceived investment size is maintained two weeks after the treatment.

Summarizing the Results of the Analysis

The standard statistical interpretation format used for between-subjects ANOVA (as described in Chapter 9) is also appropriate for the repeated-measures design. The outline of this format appears again here:

Since most sections of this format appeared in Chapter 9 or in this chapter (e.g., the ANOVA summary table), they are not repeated here; instead, the formal description of the results follows.

Formal Description of Results for a Paper

Mean investment size scores across the three trials are displayed in Figure 12.3. Results were analyzed using one-way analysis of variance (ANOVA), repeated-measures design revealing a significant effect for the treatment, F(2,38) = 30.28, p <.01. Contrasts showed that the baseline measure was significantly lower than the post-treatment trial, F(1,19) = 29.01; p <.01, and the follow-up trial, F(1,19) = 43.09, p <.01.