6.2 Questions, Answers and Experiments
The reason why it is important to distinguish between experimentation and passive observation is that they provide different kinds of evidence. An observational study reveals correlations between variables whereas a well conducted experiment can provide evidence for causation. This is because active and structured manipulation of a system can isolate the effect that one variable has on another. In an observational study we do not take control over the variables, which makes it difficult to work out “what does what to what” in the system.
In Chapter 2 we compared the inductive and hypothetico-deductive approaches to research and concluded that they provide different sorts of knowledge. We said that the inductive approach could only give us descriptive theories, whereas an hypothesis could be a basis for explanatory theory. These two approaches have nothing to do with our definition of experimentation. Both experiments and observational studies can test hypotheses but neither of them has to be hypothesis-driven. If we test hypotheses or not is a matter of how we formulate our research questions.
By contrast, if we experiment or not has to do with how we go about answering the questions. As explained in Figure 6.3, these are two completely independent dimensions of scientific enquiry. The two types of study (observational and experimental) and the two types of research question (non-hypothetical and hypothetical) give us four possible combinations. I have chosen not to use the term inductive in this figure, since philosophical purists may argue that an inductive study is an impossibility, and the crucial point is really if the research question is based on an hypothesis or not. If it is not, the knowledge obtained can only be descriptive. If it is, it is at least possible to obtain explanatory knowledge from the study. Whether we do or not depends on the nature of the hypothesis.
Figure 6.3 The research question and how it is answered (type of study) are two independent dimensions of scientific enquiry. Different types of questions lead to different types of knowledge, and different types of studies give evidence for different types of relationship.
Recall from Chapter 5 that phenomenological theories describe relationships between variables, whereas explanatory theories describe the mechanisms behind phenomena. Hypotheses can be of both these kinds. For instance, an hypothesis stating that a certain nutrient decreases the blood pressure can only result in phenomenological understanding, because it only makes an assertion about the relation between two variables without explaining why there is a relationship. If we, on the other hand, state that the nutrient decreases the blood pressure because a substance in it interacts in a specific way with the cells in the blood vessels, then testing this hypothesis will give us insight into whether the explanation is true or not. In this way, an hypothesis involving a mechanism provides explanatory knowledge. As previously mentioned, this type of knowledge is the long-term goal of science, though descriptive knowledge can be an important intermediate goal.
It is perhaps easiest to clarify the combinations in Figure 6.3 by some simple examples. Say that you discover, during your travels at the far end of the world, that people in two villages show very different incidences of hay fever. You hear that the villagers of the community where hay fever is uncommon regularly eat a specific herb. As this herb is not eaten in the other village, you suspect that it causes the difference. This discovery was not preceded by the formulation of an hypothesis, nor did it result from an experiment. You just made an accidental discovery of a potential relationship between two variables, and thereby find yourself in the upper left box in Figure 6.3. As long as you have not conducted a structured study your case is very weak. It is based on anecdotal evidence for the effect of the herb. The knowledge obtained, if we can call it that, is purely descriptive: a village where people eat the herb has few cases of hay fever. We do not know if one causes the other and much less why it would.
The other example in the upper left box is data mining, which is a more reliable method than accidental discovery. It uses statistical methods to look for relationships between variables in large data sets. Since the method explores a very large number of potential relationships, spurious relationships are sometimes found. These are accidental correlations between unrelated variables. The fact thereby remains that observational studies do not give good evidence for causation and, if there is no hypothesis, they cannot provide explanatory knowledge.
Now, your discovery in the two villages could inspire you to make a larger study. Say that you formulate an hypothesis stating that eating the herb somehow reduces the risk for contracting hay fever. To build a case you start collecting data about a large number of people in an epidemiologic study. Other possibilities must be excluded, such as social, hereditary and environmental factors that also could influence the risk for hay fever. Your data set must, therefore, go far beyond whether people eat the herb and whether they have hay fever. This is now a hypothesis-driven study. Since your hypothesis does not suggest an explanation of why the herb is effective, it can only result in descriptive, phenomenological knowledge. To build an explanatory theory you must formulate an hypothesis that includes a mechanism, which may be possible at a later stage. As you are not actively administering the herb, however, this is still an observational study and you find yourself in the lower left box in Figure 6.3. (Incidentally, it should be noted that not all epidemiological studies are confined to this box.) Observational studies are improved by using larger amounts of data as this decreases the uncertainty in the conclusions. Although this study is much larger and more structured than the first one, it still does not provide good evidence for causation. To show that variation in one variable is the actual cause of a response in another, you must actively manipulate that variable in a structured way.
To demonstrate causation you must, in other words, conduct an experiment. You could, for example, devise a blinded experiment to test the hypothesis that the herb decreases the risk for hay fever. Such a study involves a control group that receives a placebo, whereas the experimental group receives the herb. It is conducted in a standardized fashion so that the substances are taken in the same quantity at the same intervals under the same conditions. The results will now provide good evidence of causation, since the experiment isolates the potential effect of the herb. The control group excludes the possibility that an effect is due to background variables that are outside the experimenter's control. The knowledge obtained is, however, only descriptive. To explain the effect the hypothesis must involve a mechanism. You could, for instance, formulate an hypothesis about how a specific substance in the herb interacts with the immune system and test it in a cleverly designed laboratory experiment. In that case your study could both give good evidence for causation and explanatory knowledge. Whichever hypothesis you use, you find yourself in the lower right box of Figure 6.3.
It may seem confusing that a study in the lower left box could be the basis of explanatory theory but not give evidence for causation. There is, however, no contradiction in this. Data can provide support for an explanatory hypothesis but never prove it. Evidence for causation is stronger than a mere correlation but it is still not proof for causation. There is always a degree of uncertainty in our conclusions. To make the strongest possible case for our conclusions we should strive to be as low and as far to the right in Figure 6.3 as possible.
The only box we have not visited yet is the upper right one. How can an experiment be conducted without an hypothesis? Well, it is fully possible to expose a system to different treatments just to find out what happens. This is probably how Brewster managed to induce birefringence in materials under stress, as mentioned in Chapter 2. We could call this type of study “black box testing”, as it is more concerned with how something behaves than how it works. It exposes a system to various treatments to produce reactions, without preconceived ideas about its internal workings. This is actually a common method for testing software. Due to the active manipulation of the system the results can give good evidence that the output is caused by the input, but it says nothing about why.
Now that we have sorted out some fundamentals, it is time to ask ourselves how experimenters, in practice, go about answering research questions. The best way to do this is perhaps to look at a number of real world examples.