8.2 The Importance of Randomization

If the lady should lack the ability to distinguish between the treatments, the experiment must be governed by chance for the significance test to be useful. As we mentioned in the last chapter, there are always numerous factors in experiments that are more or less outside our control, which may still affect the outcome. In this example, the quantity of milk might differ between the cups, the strength of the tea and the tasting temperature may change over time, and so forth. All of these factors could cause variations in taste that are greater than that caused by the experimental treatment. If, for instance, different persons were to prepare the cups belonging to the different treatments, systematic variations in taste could be introduced that were not due to the treatments. We could go to considerable lengths trying to eliminate such disturbances, inventing ever more accurate methods of pouring the milk and brewing the tea, but it would be a futile exercise to try to do away with them completely. It simply cannot be done.

It is important to keep in mind that experiments are always conducted using limited resources. When planning an experiment we must consider how to best utilize these resources. Which causes of disturbance should we care about and which ought to be deliberately ignored? How much effort should be put into diminishing the magnitude of those disturbances that we choose not to ignore? Most importantly, since it is impossible to eliminate the disturbances completely, it is important to randomize the experiment as in this example, where we present the cups to the lady in random order. Randomization minimizes the risk of systematic errors due to drift in the conditions. It maintains the integrity of the frequency distribution, which is the basis for evaluating the results. In short, randomization is what makes the significance test valid [1].

It should be said that it is sometimes difficult or even impossible to randomize an experiment. Luckily, there are techniques for dealing with such situations. Repeated replication of one or more measurement points over time is one way of detecting drift in the conditions. It is also possible to run the experiments in such an order (the run order) that makes it possible to separate experimental effects from time trends. If you find yourself unable to randomize your experiment you are recommended to consult a statistics book or, better still, a statistician. In most cases, however, randomization should be standard procedure.