10.8 Planning Checklist

Now that you have identified promising hypotheses by discarding the least promising ones using cause-and-effect tables and thought experiments, it is time to devise the actual experiment. In this final section of the chapter we will go through a rough checklist for this part of your investigation.

Firstly, all research begins with a question. You should be able to state clearly what you want to learn from the experiment. If you do not have a clear question, you will not obtain a clear answer. If the aim of the experiment is to test an hypothesis you must find a good method for testing it. Which variables are connected with the hypothesis? Which observations could support or disprove it? The answers to these questions will provide clues for a fruitful approach. You may have to build a special apparatus or a dedicated setup of equipment to create the conditions needed for the experiment, or to be able to observe the effect that you are interested in. Each experiment is unique but all involve a question and a method for answering it.

You must choose response variables that are relevant for your hypothesis. Sometimes, the effect of interest can be coupled to several different response variables. Be sure to collect all the information needed to analyze the experiment and understand the root causes behind the effect. It would be unfortunate to find that relevant information is missing during the analysis.

The response variables are coupled to input variables, or factors. Engineers and scientists often take different attitudes to the choice of variables. It is often sufficient for engineers to optimize a system without understanding its fundamentals, so the input variables may be any “control knobs” that are practically useful. Scientists aim to understand and explain. Their input variables must often be more carefully chosen because they should be coupled to root causes. This is discussed further in the next chapter.

The next step is to choose an experimental design. It could be one of the standard designs described in the last chapter but, as established in Chapter 6, you may need to devise a unique measurement strategy for your specific problem. Approaches generally vary according to the nature of the factors. If they are categorical, the experiment tends to focus on the comparison of two or more conditions. Randomized controlled experiments, crossover and factorial experiments are common examples. With continuous, numerical variables, experimenters tend to sweep variables or use response surface designs. The design must be aligned both with the objective of the experiment and the resources at hand, because experiments can be expensive. The number of experimental treatments or measurements may be limited by the supply of raw materials, time constraints or other resources. If so, you should strive to obtain the data that is most pertinent to the research question. When designing your experiment it is also important to consider the need for a control case to provide a reference level for the effects of your treatments. Finally, you need to consider noise and background variables. Noise can be due to variations in experimental settings, environmental factors, the detection system and so forth. They are attended to by replication and randomization. Background variables are less random in nature and can cause drift or other false patterns in the response. Such effects are often avoided by replication and blocking. As previously mentioned, there may be other methods for handling noise and background factors that are more suited to your needs.

Once you have devised an experiment with responses and factors, it is time to decide on the levels that the factors should be set to in the experiment. These define the ranges of the input variables. It is often useful to vary the factors in wide ranges, as this makes their effects easier to detect. Be careful, however, not to extend the range to the point that the system under study fails to function normally. If the range is too small, on the other hand, you may not be able to detect an effect, even if one exists. Appropriate factor levels can usually be identified in an initial screening phase through variation around a known baseline setting. It is always dangerous to give general guidelines but, as a rough rule of thumb, if your response varies by a factor of two compared to the baseline condition, the effect is readily detectable.

Before collecting data you also need to plan how to collect the data. Do you, for example, know the capability of your measurement system? If not, you should make a measurement system analysis, which will be described further in the next chapter. We also learned in this chapter that it is important to determine the scope of your study. It may be difficult to answer every aspect of your research question in a single experiment. If so, you should break the problem down into several parts and attend to them individually. If you are using design of experiments, it is useful to make sequential experiments: starting with a screening design and then adding treatments as you proceed, maybe to obtain a response surface design. Never use the majority of your resources on your initial tests; save them to the later stages when you have found and solved the bugs in your setup.

The above activities can be summarized in the following checklist:

• Clearly state the question
• Select appropriate responses
• Select useful factors
• Design the experiment
• Determine factor ranges
• Plan the data collection.

Experiments often fail because items on this checklist have been overlooked. A factor may be varied in too narrow or too wide a range, the measurement system may be too poor, or the experiment may not have been randomized. Furthermore, as a general rule, things can and will go wrong. It is, therefore, useful to formulate “Plan B” at an early stage. As Dwight D. Eisenhower put it, “I have always found that plans are useless, but planning is indispensable.”

Last, but not least, when planning and conducting an experiment it is vitally important to continuously document your thoughts and observations. It may be weeks or months before you analyze the data and your notebook will become a valuable asset when your memory fails to recall details of discussions and events in the laboratory.