9.1 Statistics and the Scientific Method

We saw in Chapter 6 that many experiments do not involve statistical tests at all. The interesting effect may be so strong that it is not necessary to use statistics to separate it from the noise. In many cases, however, statistical tools are indispensable components of the experimental strategy, especially when effects are weak or when background factors are important. The combination of statistical and experimental method is called design of experiments. Fisher [1] originally used the term for planning an experiment with a particular significance test in mind. The teatime experiment in the last chapter is a good example of this, borrowed from his book The Design of Experiments. It shows how an experiment can be designed to discern the potential weak effect of a single factor by applying replication and randomization. Today, as we shall see, the term “design of experiments” has been extended to include orthogonal multifactor experiments where the analysis is often based on linear models of the data rather than pure significance tests.

When significance tests are used, it is important not to confuse the statistical hypothesis with the scientific one. Scientific hypotheses go beyond particular sets of data and make general statements about the world. As explained in Chapter 6, they may even be a basis for explanatory theory if they involve a mechanism. Statistical hypotheses are not explanatory; they only make simple statements about mathematical relationships. They may suggest that the mean of one sample is greater than that of another but make no statements about the reason behind this difference. While statistical hypotheses can be important in providing support for a more complex scientific hypothesis, they say very little in themselves about the workings of the world. We should also remember that individual tests rarely provide sufficient information to support a final conclusion about the truth of a scientific hypothesis. To have generality, results should be consistent under a wide variety of circumstances. The lack of statistical significance in one experiment is not proof of the lack of an effect.

In summary, statistical tests are tools that may help researchers draw conclusions that would otherwise be difficult to support but they rarely answer research questions directly. The statistical question has to do with the potential difference between specific samples, whereas the research question concerns general patterns and regularities. Statistics is a study of numbers and science is the study of the world. The reason that they often occur together is that the world can often be studied with the help of numbers.