11.1 Generating Understanding from Data
Every day we hear claims that are based on observations. Some of these observations provide hard data that can be expressed exactly in numbers, while others provide soft data that are not easily quantified. For example, saying that the weather is cold could be based on hard data from an outside thermometer, but saying that the neighbor looks happy today is certainly based on soft data.
The famous physicist Lord Kelvin said that it is important for scientists to be able to measure what they are speaking of because “when you cannot express it in numbers, your knowledge is of a meagre kind”. This statement is often criticized because it plays down the value of soft data. It must be acknowledged that soft data are very useful in many situations. It is also true that quantitative data are not the same thing as meaningful knowledge. To realize this we only have to recall the methodical gardener who counted his apples in Chapter 2. However unique the information, knowing that there are 1493 apples in his garden is not terribly interesting, unless this is stated in the light of a meaningful problem. Data must answer a well-posed question in an unambiguous way to be useful; formulating that question was the central topic of the last chapter.
The point of Kelvin's statement is that, to provide an unambiguous answer to a research question, observations must be quantified at some level. Research requires a high degree of inter-observer correlation, meaning that other researchers must be able to repeat your experiment and come to the same result. Scientific conclusions cannot be based on subjective opinions. If you are unable to quantify responses in an experiment your data cannot be used to support any statement about effects. This, indeed, is a meagre kind of knowledge. When quantitative data are not meaningful, the problem always lies with the question and not with the data.
Experimental equipment often has a number of “control knobs”. This is especially common in applied research. Machines, instruments and industrial processes have actuators and other components that are controlled using these knobs. As they are intentionally easy to manipulate, it is often tempting for an experimenter to use these control knobs as factors in an experiment. On the other hand, it is important to think carefully about how the experimental factors are connected to your research problem. To be able to explain your observations as well as possible you should investigate root causes; the “control knob variables” seldom belong to this category. Although a chemical plant, for example, is controlled using valves, valve settings say little about what is actually happening in the chemical process. Such control variables may provide descriptive knowledge about how a process may be tuned but offer little in the way of explanations.
As repeatedly stated in this book, science aims to explain rather than just describe, and scientists therefore need to focus on explanatory variables. In the case of a chemical plant, it is probably more interesting to know something about concentrations of substances or local temperatures than about the settings of valves. An engine developer might be interested in the throttle position of an engine, but an engine researcher is likely to be more interested in the corresponding air mass-flow into the engine, or the level of turbulence that it produces in the cylinder. A developer of drugs may be interested in the concentration of a substance in a pill, whereas a medical scientist is more interested in how the same substance acts in the body.
Understanding how something works requires more direct measures than those needed to control it. To understand and explain you must connect the output from your experiment to cause variables. These are often more difficult to identify and significantly more difficult to manipulate than the control knobs on the apparatus in your laboratory. If you find that your research question involves control variables rather than cause variables, consider revisiting the planning phase of the last chapter before proceeding with data collection.
As described in Figure 11.1, data collection is the research phase that takes place between the planning and the analysis of an experiment. The smaller black arrow indicates that it is sometimes necessary to return to the planning stage before finishing the data collection. This could be due to unexpected results that change the focus of the study, or to practical problems. In the beginning many Ph.D. students consider data collection to be the most important activity in research. Data are necessary to draw conclusions but the actual collection of data often makes up a relatively minor part of a research project. In many respects, it is the most technical part of the scientific process. In the following sections we will discuss the central steps in the data collection phase. They treat the development of measurement systems, how to assess the accuracy and precision of such systems, and how improve them. Firstly, however, we will discuss an inherent property of all measurements – uncertainty.
Figure 11.1 The three phases of research. During the data collection phase it may be necessary to revisit the planning phase, due to practical problems or unexpected results.
