9.2 Designs with One Categorical Factor

Experimentation is to create a specific condition and investigate its effect on a specific outcome. We may be interested in how a surface treatment affects the properties of a material, how a solvent affects the yield of a synthesis or how a drug treatment affects a medical condition. The input variables are then called categorical factors, since they describe states that can be labeled but not ordered with respect to each other: we either use one treatment or another, or possibly none at all. In such cases, the hypothesis tests introduced in the last chapter can be used to analyze the results.

If the natural variation in the measured variable is large and the samples are small, it may be difficult to discern the effect of the treatment from the noise in the data. In such cases it is often necessary to design the experiment with noise and background factors in mind. This need is common in medical studies, for example, since patients are affected by a wealth of social, environmental and other background factors that are not under the experimenter's control.

The most common method for avoiding background effects is to make controlled experiments. This means that two groups are compared, where one is exposed to the experimental treatment and the other does not receive any treatment at all. If the experimental group shows an effect and the control group fails to do so, this indicates that the effect is due to the treatment. A background factor that may still be important in such studies is that the mere expectation of an improvement may produce a response in the experimental group. If the control group is aware that it does not receive treatment this effect will be absent among them. For this reason they should be given a placebo, which is a simulated treatment that has no effect in itself. The study should be blinded, meaning that the control group is to be kept unaware of the fact that they receive a placebo. In some cases it may even be expected that the experimenters’ expectation of an effect will cause them to act differently with the two groups and this, too, could induce a false response. It is then better if both the experimenters and the subjects are kept unaware of which group receives the placebo, and this is called a double-blind study. To minimize the risk that other background factors bias the outcome, the subjects should be randomly allocated to the different treatment groups. Such experiments are collectively referred to as randomized controlled experiments. The data are analyzed using a two-sample t-test or, if the procedure is generalized to more than one treatment, using ANOVA. It is important to use sufficient sample sizes to decrease the risk of Type II errors. As discussed in the last chapter, an appropriate sample size is found by power analysis.

When discussing the t-test in the last chapter we differed between dependent and independent samples. With independent samples we simply compared the sample means to each other. In dependent samples, however, one individual is associated with one observation in each sample. By investigating the difference between these paired observations we remove the variation between individuals. This increases the precision in the data compared to independent samples. The analysis is made using the paired t-test.

Randomized controlled experiments are often considered to be one of the most rigorous test procedures in medical and similar studies but, since they are based on independent samples, the precision can be improved. Variation between the individuals may obscure the effect of a treatment. If we instead design the experiment to study the difference induced by the treatment in single individuals, we may analyze the results using the paired t-test. If you need to review why this procedure increases the precision in the data, you could have another look at the discussion about the dietary data in Table 8.5.

This procedure is used in an experimental design called a crossover experiment. The subjects are then exposed to a sequence of at least two different treatments, one of which may be a placebo. As described in Figure 9.1, the study has two arms and the subjects should be randomly allocated to each arm. The design is most often balanced so that all subjects are exposed to the same number of treatments and treatment periods. We may, for example, be interested in finding out if a certain substance decreases the LDL cholesterol level in the blood. Before treatment we measure the baseline cholesterol level in each individual. The subjects in one arm of the study then receive the substance for a period of, say, 15 days. At this point the cholesterol level is measured again. The next step is to cross this group over to the other treatment, which could be a placebo. A “washout” period is usually employed to avoid carry-over effects that may arise if the response remains in the subjects for some time after the treatment. After receiving the placebo for 15 days the cholesterol level is measured anew. Subjects in the other arm of the study are treated in exactly the same way but they receive the placebo first and the substance of interest last.

Figure 9.1 Schematic description of a crossover experiment.

So, compared to a randomized controlled experiment, what have we gained by this more complex procedure? The most obvious advantage is that each individual is his or her own control. Comparing the cholesterol level before and after treatment in each person we remove the baseline variation between the individuals, exactly analogous to the dietary experiment in Chapter 8. This is appropriate because we are of course interested in the change induced by the substance and not in how the level varies between individuals before the treatment. If we were to exchange the placebo for a second type of treatment, this design also allows us to analyze the potential effect of the order of the treatments.

In crossover designs, the variation from the baseline can be analyzed using a paired t-test. In our case this test both gives the effect of the treatment and the placebo effect. If we are interested in the effect of the order in which two treatments are given, a two-way ANOVA can be used with the order and the treatment as the two factors. Again, randomization is often necessary to avoid bias in the data; a power analysis should be made to determine an appropriate sample size.