Example with Significant Differences between Experimental Conditions

To illustrate one-way ANOVA, imagine that you replicate the study that examined the effect of rewards on commitment in romantic relationships. (This study was described in Chapter 8.) To give the study an added twist, however, in this investigation you use three experimental conditions instead of the two conditions described in the last chapter.

Recall that in the preceding chapter it was hypothesized that the rewards people experience in a romantic relationship will have a causal effect on their commitment to those relationships. You tested this prediction by conducting an experiment with 20 participants. All 20 participants were asked to read the descriptions of 10 potential romantic partners. For each partner, participants imagined what it would be like to date this person and rated how committed they would be to a relationship with that person. For the first nine partners, every participant saw exactly the same description. There were some important differences with respect to partner 10, however:

• Half of the participants had been assigned to a “high-reward” condition and these participants were told that partner 10 “...enjoys the same recreational activities that you enjoy, and is very good looking.”

• The other half were assigned to the “low-reward” condition and were told that partner 10 “...does not enjoy the same recreational activities that you enjoy, and is not very good looking.”

You are now going to repeat this experiment using essentially the same procedure as before, but this time you will add a third experimental condition called the “mixed-reward” condition. Here is the description of partner 10 that will be read by those assigned to this group:

PARTNER 10:  Imagine that you have been dating partner 10 for about
1 year, and you have put a great deal of time and effort into this
relationship.  There are not very many potential partners where you
live, so it would be difficult to replace this person with someone
else.  Partner 10 lives in the same neighborhood as you, so it is
easy to see him/her as often as you like.  Sometimes this person
seems to enjoy the same recreational activities that you enjoy and
sometimes s/he does not.  Sometimes partner 10 seems to be very good
looking and sometimes s/he does not.

Notice that the first three sentences of the preceding description are identical to the descriptions read by participants in the low-reward and high-reward conditions in the experiment described in the preceding chapter. These three sentences deal with investment size, alternative value, and costs, respectively. However, the last two sentences deal with rewards, and they are different from the sentences dealing with rewards in the other two conditions. In this mixed-reward condition, partner 10 is portrayed as somewhat of a chameleon: sometimes s/he is rewarding to be around and sometimes s/he is not.

The purpose of the present study is to determine how this new mixed-reward version of partner 10 will be rated. Will this version be rated more positively than the version provided in the low-reward condition? Will it be rated more negatively than the version provided in the high-reward condition?

To conduct this study, you begin with a total of 18 participants. You randomly assign six to the high-reward condition, and they read the high-reward version of partner 10 as it was described in Chapter 8. You randomly assign six participants to the low-reward condition, and they read the low-reward version of partner 10. Finally, you randomly assign six to the mixed-reward condition, and they read the mixed-reward version just presented. As was the case in the previous investigation, you analyze participants’ ratings of how committed they would be to a relationship with partner 10 and ignore their ratings of the other nine partners.

You can see that this study is appropriate for analysis with a one-way ANOVA, between groups design because:

• it involves a single predictor variable assessed on a nominal scale (type of rewards);

• it involves a single criterion variable assessed on an interval scale (rated commitment);

• it involves three treatment conditions (low-reward, mixed-reward, and high-reward), making it inappropriate for analysis with an independent-groups t test.

The following section shows how to write a program in which PROC GLM analyzes some fictitious data for this study using a one-way ANOVA with one between-subjects factor.

Writing the SAS Program

There was one predictor variable in this study: “type of rewards.” For purposes of writing the program, you will assign participants a value of “3” if they were in the high-reward condition, a value of “2” if they were in the mixed-reward condition, and a value of “1” if they were in the low-reward condition. You need a SAS variable name for this variable, so you call it REWARDGROUP.

There is one criterion variable in this study, “commitment”: the participants’ rating of how committed they would be to a relationship with partner 10. This variable is actually based on the sum of responses to four questionnaire items described in the last chapter. This variable could assume values from 4 through 36 and was measured on an interval scale. You give it a SAS variable name of COMMITMENT.

There are a number of ways to perform a one-way ANOVA with SAS. In fact, there is even a procedure called PROC ANOVA that is designed specifically for analysis of variance. However, this manual presents the use of PROC GLM instead. PROC GLM is more appropriate when some of your experimental groups have more participants than other experimental groups. Statistics textbooks sometimes refer to these situations as studies with unequal cells or studies with an unbalanced design. PROC GLM is appropriate for either balanced or unbalanced designs. At the end of this chapter, PROC GLM with unequal cells is discussed further.

Earlier, you learned that you would use a multiple comparison procedure when your ANOVA reveals significant results and you want to determine which pairs of groups are significantly different. The program provided here includes a statement to request Tukey’s HSD test so that, if the overall ANOVA is significant, the multiple comparison tests are included in the output, ready for interpretation. However, remember that you only interpret a multiple comparison test if the F for the overall ANOVA is significant.

This is the syntax for the SAS program that performs a one-way ANOVA followed by Tukey’s HSD Test:

PROC GLM DATA=filename;
   CLASS  predictor-variable;
   MODEL  criterion-variable = predictor-variable;
   MEANS  predictor-variable / TUKEY;
   MEANS  predictor-variable;
RUN;

Here is the entire program—including the DATA step—to analyze fictitious data from the preceding study:

 1        DATA D1;
 2           INPUT  #1  @1  REWARDGROUP  1.
 3                      @3  COMMITMENT  2.  ;
 4        DATALINES;
 5        1 13
 6        1 06
 7        1 12
 8        1 04
 9        1 09
10        1 08
11        2 33
12        2 29
13        2 35
14        2 34
15        2 28
16        2 27
17        3 36
18        3 32
19        3 31
20        3 32
21        3 35
22        3 34
23        ;
24        RUN;
25
26        PROC GLM DATA=D1;
27           CLASS REWARDGROUP;
28           MODEL COMMITMENT = REWARDGROUP;
29           MEANS REWARDGROUP / TUKEY;
30           MEANS REWARDGROUP;
31        RUN;

Notes Regarding the SAS Program

The data in the preceding program were entered so that there is one line of data for each participant. They were entered according to the following format:

You can see that it was necessary to create just two columns of variables to perform the ANOVA. The first column of data (in column 1) included each participant’s score on REWARDGROUP, the variable that coded group membership. REWARDGROUP was coded using 1s, 2s and 3s, but you could have used any numbers (or even letters such as Ls, Ms, and Hs).

The second column of data (in columns 3 to 4) included participant scores on the commitment variable, COMMITMENT. COMMITMENT is a two-digit variable because it could take on a value as high as 36 (i.e., a two-digit number). All values on COMMITMENT were entered as two-digit numbers, putting a zero in front of single-digit numbers where appropriate (e.g., 06, 09). Using zeros in this fashion can make it easier to keep your place when entering multiple-digit variables.

In the program, data for the six low-reward participants were entered in lines 5 to 10, data for the six mixed-reward participants were entered on lines 11 through 16, and data for the six high-reward participants were entered on lines 17 through 22. It is not necessary to separate data for the three groups in this way, however; you could have entered the data in a random sequence if desired.

The PROC GLM step that requests the analysis of variance appears on lines 26 through 31 of the program. The CLASS statement on line 27 specifies the name of the nominal-scale predictor variable for the analysis. In this analysis, REWARDGROUP is the predictor.

The MODEL statement appears on line 28 of the program. With the MODEL statement, the name of the interval-level or ratio-level criterion variable should appear to the left of the “=” sign, and the name of the nominal-level predictor should appear to the right. If you want, you can list more than one criterion variable to the left of the “=” sign, and separate ANOVAs will be performed for each. Do not list more than one predictor variable to the right of the “=” sign, however, or the analysis performed will not be a one-way ANOVA.

The MEANS statement on line 29 requests that the Tukey HSD test be performed. The second MEANS statement (on line 30) merely requests that the mean commitment scores for the three treatment groups be printed.

Results from the SAS Output

With LINESIZE=80 and PAGESIZE=60 in the OPTIONS statement (first line), the preceding program would produce the following output. The contents of each page are summarized here:

• Page 1 provides class level information and the number of observations in the dataset.

• Page 2 provides the analysis of variance table.

• Page 3 provides the results of the Tukey HSD test.

• Page 4 provides the means and standard deviations requested by the second MEANS statement.

The output created by the program is reproduced here as Output 9.1.

Output 9.1. Results of One-Way ANOVA: Significant Differences Observed

[View full size image]

[View full size image]

[View full size image]

[View full size image]

Steps in Interpreting the Output

1. Make Sure That Everything Looks Right

Page 1 of the PROC GLM output provides class level information for the analysis. Verify that the name of your predictor variable appears under the heading “Class.” Under the heading “Levels,” the output should indicate how many groups were included in your study (in this case, 3). Under the heading “Values,” the output should indicate the specific numbers or letters that you used to code this predictor variable (in this case, the values were 1, 2 and 3). Finally, the last line indicates the number of observations in the dataset. The present example used three groups with six participants each for a total of N = 18. If any of this does not look right, you probably made an error in either entering your data or writing your INPUT statement.

Page 2 of the output provides the analysis of variance table. Near the top of page 2 on the left side, the name of the criterion variable should appear to the right of the heading “Dependent Variable.” In this case, the dependent variable is COMMITMENT. Below this heading is the ANOVA summary table.

The first column of the ANOVA Summary table is headed “Source” and below the heading “Source” are three subheadings: “Model,” “Error,” and “Corrected Total.” Look to the right of the heading “Corrected Total” and under the column headed “DF.” In this case, you see the number “17.” This number represents the corrected total degrees of freedom. This number should always be equal to N – 1, where N represents the total number of participants for whom you have a complete set of data. In this study, N is 18 and 18 – 1 = 17, so it looks like everything is correct so far. (A complete set of data means the number of participants for whom you have scores on both the predictor and criterion variables.)

The means for the three experimental groups are found on page 3 of the output. Toward the bottom of the page, the third column from the right is headed “Mean.” Below this heading are the group means. Notice that, in this case, they range from 8.67 through 33.33. Remember that your participants used a scale that could range from 4 through 36, so these group means appear reasonable. If these means had taken on values such as 0.21 or 88.33, it would probably mean that you had made an error in either entering the data, or writing the INPUT statements.

2. Review the Appropriate F Statistic and Its Associated p Value

Once you verify that there are no obvious errors in the output, you can review your results. First you should review the F statistic from the ANOVA to see if you can reject the null hypothesis. You can state the null hypothesis for the present study as follows: In the population, there is no difference between the high-reward, the mixed-reward, and the low-reward groups with respect to mean commitment scores. Symbolically, you can represent the null hypothesis this way:

H₀: M₁ = M₂ = M₃

where M₁ is the mean commitment score for the low-reward condition, M₂ is the mean commitment score for the mixed-reward condition, and M₃ is the mean commitment score the high-reward condition.

You will generally use the degrees of freedom, sum of squares, F value, and p value associated with the Type III sum of squares. This is an important point because when you actually run a program, the output provides two ANOVA summary tables at the bottom of page 2. For this example, the Type I and Type III sums of squares are identical because this is a balanced design (i.e., equal numbers of participants in each of the three conditions). Use of the Type III sum of squares is particularly important with unbalanced designs.

The first summary table provides the results using something called the “Type I sum of squares,” and has the following headings:

DF     Type I SS       Mean Square     F Value     Pr > F

Below this, you see results for a second ANOVA table that uses the “Type III sum of squares” and has the following headings:

DF     Type III SS     Mean Square     F Value     Pr > F

The F statistic appropriate to test your null hypothesis appears toward the bottom of page 2 in the section that provides the Type III sum of squares. Notice that there is a heading called “Source” that refers to “source of variation.” Below this heading you will find the name of your predictor variable, which in this case is REWARDGROUP. To the right, you see an ANOVA summary table using the Type III sum of squares.

In this ANOVA summary table at the bottom of the page, look to the right of the word REWARDGROUP. Under the heading “DF,” you see that there are 2 degrees of freedom associated with the predictor variable. The degrees of freedom associated with a predictor are equal to k – 1, where k represents the number of groups. Under the heading “Type III SS,” you find the sum of squares associated with your predictor variable, REWARDGROUP, which in this case is 2225.33. Under the heading “F Value,” you find the F statistic appropriate for the test of your null hypothesis. In this case, F = 122.12, which is very large. The last heading is “Pr > F,” which gives you the probability of obtaining an F this large or larger if the null hypothesis were true. In the present case, this p value is very small, less than .01. When a p value is less than .05, you can reject the null hypothesis. In this case, the null hypothesis of no population differences is rejected. In other words, you conclude that there is a significant effect for the type of rewards independent variable.

3. Prepare Your Own Version of the ANOVA Summary Table

Before moving on to interpret the sample means for the three experimental groups, you should prepare your own version of the ANOVA summary table. All of the information that you need either is presented on page 2 of the output or you can easily calculate it by hand.

Table 9.1 provides the completed ANOVA summary table for the current analysis:

^[*] p < . 01

In completing this table, you should first summarize the information associated with the predictor variable, type of rewards. On the bottom of page 2 of Output 9.1, look to the right of the name of the predictor variable (REWARDGROUP, in this case). Here you find the results associated with the Type III sum of squares. Remember that all of the information related to “type of rewards” in Table 9.1 must come from this section of the output that deals with the Type III sum of squares. Under “DF,” find the between-subjects degrees of freedom, two in this case. Note where these degrees of freedom are placed in Table 9.1: they appear in the column headed df, to the right of “Type of rewards.”

On the output, next find the sum of squares associated with REWARDGROUP under the heading “Type III SS.” This value (2225.33) goes under SS in Table 9.1.

Next in Table 9.1, under the heading MS, you find the mean square associated with REWARDGROUP. This mean square is also known as the “mean square between groups.” It is calculated by PROC GLM using this simple formula:

In the above formula, Type III SS represents the Type III sum of squares associated with REWARDGROUP and DF represents the degrees of freedom associated with REWARDGROUP. This mean square has already been calculated by PROC GLM and appears in Output 9.1 under the heading “Mean Square.” It should appear in Table 9.1 under MS.

Next, find the F statistic appropriate for the test of your null hypothesis. This is also on page 2 of Output 9.1 under the heading “F Value” and to the right of “REWARDGROUP.” This statistic is 122.12 and is placed under the heading F in Table 9.1.

Under the heading “Pr > F” from Output 9.1, find the p value associated with this F. However, you do not have a separate column in Table 9.1 for this p value. Instead, you can mark the F value with an asterisk (*) to indicate that the F is statistically significant. At the bottom of the table, you place a note that indicates the level of significance attained by this F value. In this case, the output indicated that p < 0.01 so add a note at the bottom of the table that indicates that the effect for type of reward was significant at the p < . 01 level:

* p < .01

If the F value had been significant at the .05 level, your note would have looked like this:

* p < .05

If the F value had not been statistically significant, you would not have put an asterisk next to it or placed a note at the bottom of the table.

Finally, you need to calculate the R² for this predictor variable. As mentioned before, the R² statistic indicates the percent of variance in the criterion (COMMITMENT) that is accounted for by the predictor (type of reward). The formula for R² is also straightforward:

In the above formula, Type III SS represents the Type III sum of squares associated with REWARDGROUP and Corrected Total Sum of Squares represents the total sum of squares from the analysis. This total sum of squares appears on page 2 of Output 9.1 to the right of the heading, “Corrected Total,” and under the heading, “Sum of Squares.” Here, the appropriate figures from page 2 of Output 9.1 are inserted in the formula:

With R² = .94, this indicates that 94% of the variance in commitment is accounted for by the “type of rewards” independent variable. This is a very high value of R², much higher than you are likely to obtain in an actual investigation. (Remember that the present data are fictitious.)

If you look closely at Output 9.1, you will notice that the R² value is already presented on page 2, just below the ANOVA summary table. Nevertheless, this chapter still advises that you calculate R² values by hand because, depending on the type of analysis that you do, the output of PROC GLM will not always provide the R² value in which you are interested. This is particularly true when you have more than one predictor variable. Therefore, it is good practice to ignore the R² provided in the output and, instead, calculate it manually.

Now that you have filled in the line headed “Type of rewards” in Table 9.1, you fill in the line headed “Within groups.” This line deals with the error variance in your sample and everything you need appears on page 2 of Output 9.1 to the right of the heading “Error.” Under “DF” on the output, you can see that there are 15 degrees of freedom associated with this estimate of error variance. Note where this goes in Table 9.1: where the column headed df intersects with the row headed “Within groups.”

Under the heading “Sum of Squares” on page 2 of Output 9.1, you can see that the sum of squares associated with the error term is 136.67. This term is transferred to its corresponding spot in Table 9.1 in the column headed SS.

Finally, under the heading “Mean Square” on the output, you can see that the mean square error (also called the mean square within groups) is 9.11. This term is transferred to Table 9.1 under the column headed MS.

The last line in Table 9.1 deals with the total variability in the dataset. You will find all of the information that you need on page 2 of Output 9.1 to the right of the heading “Corrected Total.” Under the heading “DF,” you can see that 17 degrees of freedom are associated with the total variance estimate. This goes in Table 9.1 under the heading df, to the right of the heading, “Total.” Under the “Sum of Squares” heading on the output, you find that the total sum of squares is 2362.00. Note that this appears in Table 9.1 in the column headed SS.

Place a note at the bottom of the table to indicate the size of the total sample and your own version of the ANOVA summary table is now complete.

4. Review the Sample Means and the Results of the Multiple Comparison Procedure

Because the F statistic is significant, you reject the null hypothesis of no differences in population means and tentatively accept the alternative hypothesis that at least one of the means is different from at least one of the others. However, since you have three experimental conditions, you now have a new problem: which of these groups is significantly different from the others? You have performed a multiple comparison procedure called Tukey’s HSD test to answer this question. The results of this analysis appear on page 3 of Output 9.1.

About halfway down this table, it indicates that “Alpha = 0.05.” This means that if any of the groups are significantly different from the others, they will be different with a significance level of p < .05.

On the bottom half of the table, there are four columns of figures. The last column always represents your predictor variable (which, in this case, is REWARDGROUP). Below the REWARDGROUP heading, you find the values used to code the experimental conditions. In this case, you used the value of 3, 2, and 1, which coded the high-reward, mixed-reward, and low-reward conditions, respectively. To the left of the REWARDGROUP column is a column headed “N.” This column indicates how many participants were in each condition. In the present case, there were six participants in each. To the left of the “N” column is a column headed “Mean,” which provides the mean scores for the three groups on the criterion variable (the commitment variable in this study). You can see that the high-reward group had a mean commitment score of 33.33, the mixed-reward group had a mean score of 31.00, and the low-reward group had a mean score of 8.67.

Finally, to the left of the “Mean” column is the column headed “Tukey Grouping.” It is this column that tells you which groups are significantly different from which. The rules to use here are simple:

• Two groups that are identified by the same letter are not significantly different from each other.

• Two groups that are identified by different letters are significantly different from each other.

For example, notice that group 3 is identified with the letter “A” in the grouping column while group 1 is identified with “B.” This means that groups 3 and 1 are significantly different from each other on commitment. (Notice how different their means are.) Similarly, groups 2 and 1 are also identified by different letters, so they are also significantly different. However, groups 3 and 2 are both identified by “A.” Therefore, groups 3 and 2 are not significantly different.

Remember that you review the results of a multiple comparison procedure only if the F value from the preceding ANOVA summary table is statistically significant. If the F for the predictor variable is nonsignificant, then ignore the results of this Tukey HSD test.

5. Summarize the Analysis Results

When performing one between-subjects factor ANOVA, you can use the following format to summarize the results of your analysis:

An example summary follows:

A) Statement of the problem: The purpose of this study was to determine whether there was a difference among people in a high-reward relationship, people in a mixed-reward relationship, and people in a low-reward relationship with respect to their commitment to the relationship.

B) Nature of the variables: This analysis involved two variables. The predictor variable was type of rewards, which was measured on a nominal scale and could assume one of three values: a low-reward condition (coded as 1); a mixed-reward condition (coded as 2); and a high-reward condition (coded as 3). The criterion variable represented participants’ commitment, which was measured on an interval/ratio scale.

C) Statistical test: One-way ANOVA, between-subjects design.

D) Null hypothesis (H₀): M₁ = M₂ = M₃. In the population, there is no difference between those in a high-reward relationship, those in a mixed-reward relationship, and those in a low-reward relationship with respect to their mean commitment scores.

E) Alternative hypothesis (H₁): In the population, there is a difference between at least two of the following three groups with respect to their mean commitment scores: those in a high-reward relationship; those in a mixed-reward relationship; and those in a low-reward relationship.

F) Obtained statistic: F (2,15) = 122.12.

G) Obtained probability (p) value: p < . 01.

H) Conclusion regarding the null hypothesis: Reject the null hypothesis.

I) Magnitude of treatment effect: R² = .94 (large treatment effect).

J) ANOVA Summary Table: See Table 9.1.

K) Figure representing the results:

Formal Description of Results for a Paper

You could use the following approach to summarize this analysis for a research paper:

Results were analyzed using a one-way ANOVA, between-subjects design. This analysis revealed a significant effect for type of rewards, F(2,15) = 122.12; p < .01. The magnitude of the treatment effect was computed as R² = .94 (large treatment effect). The sample means are displayed in Figure 9.1. Tukey's HSD test showed that participants in the high- reward and mixed-reward conditions scored significantly higher on commitment than did participants in the low-reward condition (p < .05). There were no significant differences between participants in the high-reward condition and participants in the mixed-reward condition.