Getting started with data analysis can be so easy. We just plug numbers into a function or library and retrieve the results. But sometimes, it's easy to forget that we have to pay attention to how the data and experiments are constructed and how the questions are framed. Much of the reliability of statistics comes from following good practices and developed processes for framing and executing the tests and experiments.
Of course, there's a lot to setting up statistical experiments and following best practices in gathering data and applying statistical tests. We won't be able to do more than cursorily glance at this topic. Hopefully, either it will serve as a reminder of things you already know or it will outline what you need to know and point you in the right direction to learn more.
Over the course of this chapter, we'll move back and forth between looking at the problem we're tackling and seeing what null hypothesis testing is, how it can help us, and how we can apply it.
In this chapter, we will cover the following topics:
- Introducing confirmatory data analysis
- Understanding null hypothesis testing
- Understanding crime
- Getting the data
- Transforming the data
- Conducting the experiment
- Interpreting the results
So without any further delay, let's learn about the techniques and the problems we'll address with these methods in this chapter.
Oftentimes, data analysis seems like a menu of analyses applied to problems, but lacking an overall structure. Of course, this isn't the case, but it seems that way to programmers without a strong background in statistics.
Frameworks such as confirmatory data analysis and null hypothesis testing provide the structure that may be missing. Generally, when you begin working with data, you start by generating some summary statistics that highlight some of the basic characteristics of the data. Afterwards, you probably generate some graphs that further elucidate the essential qualities of the data. This all falls into the realm of exploratory data analysis.
However, as the exploration wraps up, you'll probably start to think of some theories about the data that you'd like to test. You'll generate some hypotheses, and you'll need to test whether they're true or not. And based on those tests, you'll further refine your knowledge of the data, what's in it, and what it means.
This more formal stage of data analysis represents confirmatory data analysis. At this stage, you're concerned with using reliable tests that match your data, and you're trying to determine how representative your sample is. You are minimizing error and trying to get a pvalue—the probability that a result so extreme could have happened by chance—that means that the results are statistically significant.
But what does all this mean, exactly? How do we go about conceptualizing, planning, and executing these tests?