Part IV

Density Estimation and Smoothing

There are three concepts that connect the remaining chapters in this book. First, the methods are generally nonparametric. The lack of a formal statistical model introduces computing tasks beyond straightforward parameter estimation.

Second, the methods are generally intended for description rather than formal inference. We may wish to describe the probability distribution of a random variable or estimate the relationship between several random variables.

The most interesting questions in statistics ask how one thing depends on another. The paragon of all statistical strategies for addressing this question is the concept of regression (with all its forms, generalizations, and analogs), which describes how the conditional distribution of some variables depends on the value of other variables.

The standard regression approach is parametric: One assumes an explicit, parameterized functional relationship between variables and then estimates the parameters using the data. This philosophy embraces the rigid assumption of a prespecified form for the regression function and in exchange enjoys the potential benefits of simplicity. Typically, all the data contribute to parameter estimation and hence to the global fit. The opposite trade-off is possible, however. We can reject the parametric assumptions in order to express the relationship more flexibly, but the estimated relationship can be more complex.

Generally, we will call these approaches smoothing, and this brings us to the third theme of the chapters ahead: The methods are usually based on the concept of local averaging. Within small neighborhoods of predictor space a summary statistic (e.g., the mean) of the values of the response variable(s) within that neighborhood is used to describe the relationship. We will see that the local averaging concept is also implicit in our chapter on density estimation that begins this part of the book. Nonparametric density estimation is useful because for most real-world problems an appropriate parametric form for the density is either unknown or doesn't exist.

Thus, the primary focus of the remaining chapters is on nonparametric methods to describe and estimate densities or relationships using the philosophy of local averaging. Along the way, we detour into some related topics that extend these concepts in interesting alternative directions.