The most usual comparison of groups is that between two independent groups (samples), i.e. where (in terms of our taxonomy of choices):

In the case example, we might focus on whether average sales of services differ significantly between types of license (Freeware vs. Premium).

As stated in the first section, you can see the literal difference in average in simple descriptive statistics, so what value does the t-test add? The answer is that the t-test goes a little further to answer the question, ”is the difference in means statistically significant?” Any two datasets of this type will have a difference in their averages, even if it is a small difference arising from random data differences. The question is whether the difference is too big to be due to random chance, which would make it a statistically significant difference.

Because the focus of the t-test is on statistical significance, the test culminates in the analysis of confidence intervals for the difference, as explained in the next section.

The aim of the t-test is to statistically analyze the difference in averages (means) between two groups. We see in Figure 14.3 The t-test confidence interval that the difference in the means is $80,483 for freeware customers less $103,825 for premium customers, i.e. a difference in averages of -$23,342.

You probably don’t need a complex statistical program to calculate these averages and analyze the size of the difference for yourself. However, t-tests and the other comparison of means tests described below are concerned with whether a difference in means is large enough to be different from zero.

The problem faced by t-tests and other comparison of means tests is that sheer randomness in data usually guarantees that there will be some difference between different groups in data. However, if the only reason two groups differ is randomness, the difference should be close to zero. If there is a true and substantial difference between the groups, the difference should be at least large enough to be bigger than what might occur randomly. Comparison of means tests this distinction.

To test whether a difference in the means of two groups is statistically different from zero, t-tests provide a significance test (through confidence intervals and p-values). Figure 14.3 The t-test confidence interval illustrates the analysis of t-test confidence intervals. Your aim is typically to assess whether the difference in means is effectively zero, in which case the two groups do not differ significantly on the dependent variable. The confidence interval for the difference then tells us whether or not to reject this proposition: an interval lying entirely above or below zero suggests the means are significantly different from each other.

In addition to confidence intervals, t-tests also provide p-values as in Figure 14.4 The t-test p-values. However, I suggest ignoring the p-values and sticking with the more informative confidence intervals.

Before getting to the t-test confidence intervals and p-values you should first assess and deal with data assumptions. The next section discusses the steps to go through in a t-test.

Comparison of two means in this context is done in SAS through PROC TTest. To see the licenses versus sales example, open and run the file “Code14a Independent Samples TTest.” Having run the analysis, your next job is, as usual, to assess the tests.

As in regression, there are certain data assumptions incorporated in the classic t-test:

You need to either satisfy yourself that the data meets these assumptions, in which case you can look at the standard t-test confidence intervals, or find alternatives that bypass data problems.

I suggest the following initial tests after you have run the t-test program:

If either equality of variances or normality is violated significantly, you may wish to implement alternatives. In the example, the actual standard deviations are relatively similar, the formal equality of variances test does not seem to indicate violations of equal assumptions, and the regression-like normality plots seem to indicate rough normality: there seems to be no major violations.

As discussed in Data Assumptions and Alternatives when Comparing Categories above, the standard, traditional, parametric t-test is your first port of call for deciding if there is a statistical basis to claim that the means of your groups differ. You should really only use the traditional parametric test if there are no assumption violations at all. However, no data is ever perfect, so I suggest that you use the remedial versions or at least compare the traditional t-test to the remedial versions.

In addition to the traditional test there are four alternatives for serious assumption violations. I start with the remedial measures involving traditional (parametric) t-tests tests. The fourth option is nonparametric tests, which I cover in the next section. As mentioned already in Data Assumptions and Alternatives when Comparing Categories you have the following parametric options:

As introduced in Nonparametric Statistics, nonparametric tests usually eliminate the assumptions problem, often (but not always) through the following simple method:

To implement the nonparametric version of the 2-sample t-test, run or adapt the code “Code14b Nonparametric TTest.”

The nonparametric analogue of the independent samples t-test is called the “Wilcoxon-Mann-Whitney” test or just the Wilcoxon test, but there is also a nonparametric confidence interval for the differences called the Hodges-Lehmann test, as seen in Figure 14.6 Nonparametric T-Tests.

You can therefore compare the results of the nonparametric and parametric t-tests. As seen in Figure 14.6 Nonparametric T-Tests, the diagnosis of the parametric options was confirmed: the confidence interval excludes zero and the p-values are small, suggesting significant differences in sales of services between freeware and premium customers.

An alternate set of tests are designed for cases where the categories or samples are related instead of independent. This usually occurs in the following two major cases.

These two cases are discussed in the following sections.

In the first example, there is one sample contributing data to more than one category. The aim is to compare scores between the categories. There are two major sub-types here.

Related data case # 1: Repeated measures

The first example is where the same measurement is made at different time periods on the same observations. Earlier, I mentioned the example of a brand awareness measure taken before and after an advertisement. The sample consumer set is asked to provide data twice, once in each time period, and the time period is the category. Figure 14.7 Related data case #1 (same sample, repeated measures) below shows this ”repeated measures” design.

Related data case # 2: Same sample rating different categories

Another case is where each observation provides data on different categories. For instance, you could ask consumers to rate the price and the practicality of your new car model. Your focus is to see if consumers rate price different to practicality, or some similar analysis need. Figure 14.8 Related data case #2 (same sample rating different categories) shows this design.

The third type of related data is where the same measurement is made on two samples, but each observation in each sample is matched to an observation in another sample.

For instance, we may have a pharmaceutical trial in which one group of participants get the true drug, and for comparison, another sample group get a fake drug (a placebo). These samples would be independent without further methodological design, however, researchers sometimes pair each observation in the one sample with an observation in another sample, based on demographics. (For instance, you may pair a 25 year old Caucasian woman in one sample with a roughly similarly aged Caucasian woman in another sample, and so on.)

Figure 14.9 Related data case # 3: Paired samples below shows this comparison.

When you are dealing with only two samples or categories, the essential steps for related data analysis are similar to those for independent samples shown earlier, with just a few elaborations. Simply set up the data in two columns. In the repeated measures example, each column represents the measure at a time period. In the one-sample two-category example, each column is a rating on a category. In the paired samples test, column 1 is the first sample and column 2 the second sample, where each row represents a pair of matched observations. Then, you run a ”paired sample” t-test on the data.

Take, for example, the data in “Dataset09_Related_TTests” in the “Textbook Materials” folder. Here, we see a comparison between two time periods on a single sample and single measure. If you open and run “Code14c Paired T Test,“ you will see how to set up the code and you will be able to easily analyze the results using the same process as for independent samples.