There are two very different paradigms for examining patterns in or between constructs, namely, modeling a preconceived idea that you have about the data versus data mining.

Modeling preconceived ideas about data is where you look for patterns that you have predetermined might or should exist, so it is idea-based analysis. Here the researcher thinks in advance of analysis about what patterns he or she expects to find in the data. Optimally the researcher builds a strong theory regarding the analysis at hand, including whatever he or she expects to find in the focal variables. If you have predictor variables, you typically have predetermined ideas about how they explain or predict the focal variables, and whether they interact with each other. Such theories are often based on the thinking expressed in books and articles written by experts and thinkers in your field, who deeply consider the knowledge in an area and suggest theories. If you want to do something in the area, you may read these theories and then test to see if they’re right. You may merely use your own rational logic to think in advance about what you may find.

For example, say that you are doing research on employee turnover in your company, a focal variable you would love to explain and predict better. You are especially interested in whether turnover differs by employee performance, since losing top performers is a real concern. You read up on turnover literature and find out that theorists such as Jackofsky (1984) and others suggest that low and high performers are more likely to leave and mid-level performers are the least likely to leave. You may then set out to look for this pattern in your data.

Data mining is essentially looking for any patterns that data can show you. Data mining is quite different, because here the researcher does not have a predetermined theory about what patterns might be found in the data. Rather, data mining usually involves using computer algorithms to search out patterns, which are then analyzed by the researchers after they have been found. Data mining is more often used when there is a lot of data – notably that gathered by companies – and where there is no background theory.

For instance, financial services companies keep lots of data about their customers and are interested in focal variables such as defaults on loan repayments and value of transactions. They routinely data mine this for patterns, not necessarily knowing in advance what they will find. Say that you find out that people who pay tithes to churches are far less likely to default on loans. If a consistent pattern, this may be used as part of the decision process behind whether to offer loans.

As stated above, looking for patterns by visual examination of data is often too difficult. More often, we compare the distribution and relationships in data versus various specific definite relationships as defined by mathematical models.

The mathematical shape that we compare the actual data against is essentially a precise expectation of a given pattern – a perfect pattern. Take for example Figure 11.4 Example: comparing data against mathematical shapes for fit, which is again tenure data for a company. Perhaps we wish to test whether or not this data follows the precise mathematical shape seen in the black line in Figure 11.4 Example: comparing data against mathematical shapes for fit, which represents a normal distribution. (The left tail of the distribution has been cut off by SAS here to make the graph area work).

The question in Figure 11.4 Example: comparing data against mathematical shapes for fit is not whether the actual data will exactly fit a perfect shape: data is always at least a bit random so it never fits any exact shape perfectly. However, the question is whether the data fit sufficiently tightly to the exact mathematical ideal to be deemed close enough, so that we can have reasonable certainty that we have found the pattern.

In statistics this correspondence between an exact mathematical ideal shape and actual data is called ”statistical fit” – usually, this is how we really confirm or disconfirm the existence of patterns in data. Therefore, to test fit we usually do the following:

The next section gives a somewhat tongue-in-cheek metaphor for the process of fitting models to data.

I like the metaphor of star constellations as a picture for data fitting. Think of the stars as data. Nothing in the stars is an exact pattern. But, sometime in history, some (possibly stoned?) Greek turned around to his friend and went “Dude! Those stars over there look like a lion!” Figure 11.5 The stars (data) of the constellation Leo shows stars in the constellation Leo – you might think of these as actual data.

Figure 11.6 Overlaying a hypothetical picture of a Lion over the stars of Leo shows a hypothetical picture of a lion laid over the stars – you may think of the picture as an exact perfect model of a lion. The stars do not follow the lion picture exactly, but perhaps they come close enough to give agreement that a lion is a good approximation of the pattern.

Your task would be to determine how closely the data (or stars) really fit to the picture of the lion. Plots and certain fit statistics would typically be available for this. If the data does not fit a lion, perhaps there is no shape, or perhaps some other shape will fit well (a dog? a car?) The final shape that fits may or may not be helpful to you – that would be another question, and is discussed in the next major step on interpretation.

In a more typical mode, the next sections give three statistical examples of formally testing for fit of a pattern in data[1].

Correlation and regression-type statistics mostly look for straight line (linear) relationships in data. Chapter 13 deals with such relationships in far more depth, but let us get a more general idea here. Say you have only two variables, sales and age, and you think age might affect levels of sales. (Sales is therefore your focus variable, and age is a predictor variable.) You might also have some other control or environmental variables in the background. Figure 11.7 Fitting a straight line to data shows an example of age versus sales data (each dot represents a salesperson situated above his or her age and next to his or her sales averages). The dotted line represents an exact straight line.

Focus on the dotted line that is the hypothetical, perfect linear relationship between age and sales. If you think back to your school mathematics, you probably learned that the exact, mathematical equation for a straight line is Y = mX + c or something similar. In this example, Y would be the dependent variable Sales, and X would be the predictor variable Age. There are also two numbers m and c that are condensed descriptions of the shape of the line (we call these ”parameters” or ”coefficients”): c indicates the intercept of the line and m indicates how steep the line is (its slope). So, the dotted line is represented by some version of the equation Sales = m(Age) + c where m and c are parameters that must take on exact values. For instance, in the example here, the dotted line might be defined by the equation Sales = 7,200(Age) + 24,500. Here, we care most for the slope (steepness) of the line. The size of the slope parameter (which is $7,200 here) is the key thing.

Your first question – the fit one – is “does most of the data lie close to this exact straight line?” If so, you can go on to interpret what the straight line tells us. What does the $7,200 slope tell us? Chapter 13 on regression goes on to discuss these straight-line equation concepts in far more detail.

Recall from Appendix A of Chapter 7 that one basic example of testing a dataset for a pattern is comparing the distribution of a variable against normality. If the variable is normal it will follow a normal-like curve.

In that Appendix, I discuss that you can visually check the graph for differences between the perfect mathematical model of the black normality line and the actual data. However, you would probably want more confirmation, by using statistical measures. The shape of a normal curve can be condensed into just two basic statistical measures, namely ”skewness” and ”kurtosis.”

In summary we undertook the following process as discussed earlier:

Every year, when the Nobel prizes are awarded, a group of jokers in the scientific community also award the “IgNobel” awards for research that should never have happened or should never be repeated. In 2002 the IgNobel award in mathematics went to K.P. Sreekumar and G. Nirmalanof for coming up with an equation for estimating the total surface area of Indian elephants. You laugh? Well, true, if you tried to test almost any actual dataset against this equation, it would not fit … unless your data was in fact measurements from the surface areas of Indian elephants! In that case, you would in fact probably have the right equation, and you would be able to test whether the equation does a good job of fitting the data.

In summary, we often test for patterns in data by comparing the data to a hypothetical perfect shape. Note, again, that although I’ve used pictures above as one way of assessing fit to a shape, you cannot always do so, and formal statistical fit tests with given rules are more often used for this purpose. You will learn about several such fit statistics through the course of this book.

It is very important that you don’t misinterpret the fact that data fits to a pattern for importance or usefulness of the pattern. Step 3: Interpret the Pattern discusses the very different step of interpreting the pattern.

So far, I have said that when looking for patterns in data you get a certain class of statistical tests or numbers – often called fit statistics – that help you to do this.

However, the point that has been made above is that assessing whether your data has a certain pattern or not is only a step along the path. The characteristics of the pattern – if there is one – are what finally count, not the fact that there is a pattern. The characteristics of the pattern are also described by statistical numbers, and it is these final statistical parameters and coefficients that we care about.

Readers often get confused between the intermediate fit statistics that help identify patterns and the final statistical coefficients that actually describe the pattern. In any advanced statistical techniques there are pages of numbers. Much of the output is often the intermediate fit assessment just telling you if there is an appropriate pattern in the data or not. But don’t forget that ”holy grail” of the final few statistics that describe the pattern itself, if there is a pattern, and therefore tell you what you wanted to know about the world.

In case this still seems a little confusing, let me give a metaphor for the difference between intermediate fit statistics and final statistical parameters. Say that you want to buy a car. You ultimately care about the combinations of price, age, type, safety rating, etc. It is the statistics on these final characteristics (price etc.) that you care about. Being an individual, you may have preferences on the optimal combination of price, age, and the like.

Now, there are hundreds of thousands of cars for sale out there. Say that you download some database of cars for sale. You could look through it one record at a time, but this might be tough. Instead, you write a computer algorithm that identifies all the cars that are close to your preferences (this is a bit like data mining).

The algorithm might tell you that a certain Subaru is 88% close to your specifications. This is an intermediate fit statistic that helps you understand fit of data to your preferred pattern. The 88% does not actually tell you what the price is exactly, or if the high score has to do with safety more than age, for example. However, the 88% fit statistic has helped you along the way. Now you can look at the actual price, safety and other characteristics that are the final statistical coefficients of interest.

Researchers too often want to see or force patterns that are not there, usually because they had a previous opinion about what was ”right” or desirable to see. (Actually this is a bigger risk for theory-based research than data mining.) My favorite expression for this is “torturing the data until it confesses!”

Here’s one possible example of torturing the data until it confesses. In a fascinating New Yorker review, Lehrer (2010) discusses research findings about things like the effectiveness of second-generation antipsychotic drugs, which are big business for pharmaceutical companies. He points out that the findings of drug trials were initially seemingly strong, leading to the drugs’ approval and high sales. However, the more researchers studied the drugs, the more that the effects seemed to dwindle. This has happened in other scientific areas as well. One explanation may be that the scientific community wants to find results when it both does studies and when it chooses what to publish. So, researchers may “cherry pick” only strong results from arrays of weak to strong tests, or only results that corroborate their theories. Then, only (possibly fluke) strong results get published early on. Other studies may have found weak effects but these were not interesting enough to get published. Ultimately the published studies seem to show support for the drug, whereas possibly, the effect is found only sometimes at high dosages of the drug, and in fact maybe not very often.

It is especially common that researchers want to see linear relationships of the positive or negative correlation type, as we will see later in the section on regression type. Here are a few examples of this type of bias. ”Surely when satisfaction of employees goes up, turnover will go down?“ ”Surely when we spend more on advertising, we will see an increase in sales?”

Of course, the world is not necessarily as simple as we think or want it to be:

Next I discuss what to do if you do not find an expected pattern in the data.

When we don’t see the patterns we expected – or the patterns are expressed far more weakly than we expected – we have choices, and we must face these bravely.

First, if we were doing theory-based research, we might need to discount the theory. Don’t rush to this conclusion, but don’t be afraid of it. If we had a basic theory in the first place but we find that the data doesn’t support it, we can discount this theory, i.e. decide that it’s wrong. If we do so, we should come up with an alternative set of explanations about why what we thought would happen has not in fact happened. It is usually up to other researchers or further research done by yourself to corroborate your lack of the expected finding and then to deliberately test and confirm alternative explanations.

Here’s an example of not finding a pattern and looking for alternative theory. The on-the-job training theory of Gary Becker (Nobel prize-winning economist) predicts that when training is general (the skills can be shared across multiple or all firms) that the employee will pay for such training (Becker, 1975) because employees can either use the skills to move to a better job or negotiate current wages up. Much research went into this: basically Becker was at least partially wrong since firms are found on average to pay for substantial amounts of general training. Many researchers such as Acemoglu and Pischke (1999) and Boom (2005) have offered alternate explanations and theories. The research process went from theory (Becker’s theory on general training and who pays for it) to constructs (payment towards training, predicted by type of training, possibly controlling for environmental variables) to data (actual measures of the constructs) to patterns in the data (whoops, not as expected because general training is not even nearly paid for completely by employees) to a model (suggestions for an alternative model since the data is not supportive) to implications and further research.

If you are data mining without theory, perhaps you should discount the exercise. If you are data mining without a pre-set expectation of the findings and you cannot find patterns of discernible value, you are probably less prone to trying to find something in particular that is not there (because in data mining you didn’t expect to find anything in particular). However, at some stage you should terminate the data mining exercise or the current dataset! Although you could add complexity to the algorithms being used in the data mining to try to find patterns (you may indeed find patterns not discovered before), at some stage this approach becomes prohibitive.

Perhaps you should gather new data. One other option is that the data you used was badly measured or in some other way inappropriate, and that the theory or data mining exercise still holds. The researcher may revisit the way that the old data was put together or gather new data and continue to test or mine it for the same study or algorithms.

Remember what we said at the beginning, finding a pattern is not always enough. Ultimately, the patterns need to mean something. That’s why there is an extra step – that of interpreting the pattern. This is discussed in the next section.