1.3. Costs and Benefits of Fixed Effects Methods

As already noted, the major attraction of fixed effects methods in nonexperimental research is the ability to control for all stable characteristics of the individuals in the study, thereby eliminating potentially large sources of bias. Within-subject comparisons have also been popular in certain kinds of designed experiments known as changeover or crossover designs (Senn 1993). In these designs, subjects receive different treatments at different times, and a response variable is measured for each treatment. Ideally, the order in which the treatments are received is randomized. The objective of the crossover design is not primarily to reduce bias, but to reduce sampling variability and hence produce more powerful tests of hypotheses.

The rationale is that by differencing out the individual variability across subjects, one can eliminate much of the error variance that is present with conventional experimental designs in which each subject receives only one treatment.

By contrast, when fixed effects methods are applied to nonexperimental data, there is often an increase in sampling variability relative to alternative methods of analysis. The reason is that in the typical observational study, the independent variables of interest vary both within and between subjects. Suppose, for example, that one of the independent variables is personal income, measured annually for five successive years. While there might be considerable within-person variation in income over time, the bulk of the variation is likely to be between persons.

Fixed effects methods completely ignore the between-person variation and focus only on the within-person variation. Unfortunately, discarding the between-person variation can yield standard errors that are considerably higher than those produced by methods that utilize both within- and between-person variation. So why do it? The answer is that the between-person variation is very likely to be contaminated by unmeasured personal characteristics that are correlated with income. By restricting ourselves to the within-person variation, we eliminate that contamination and are much more likely to get unbiased estimates.

So what we're dealing with is a trade-off between bias and sampling variability. For nonexperimental data, fixed effects methods tend to reduce bias at the expense of greater sampling variability. Given the many reasons for expecting bias in observational studies, I think this is usually an attractive bargain. Nevertheless, one crucial limitation to fixed effects methods arises when the ratio of within- to between-person variance declines to 0: fixed effects methods cannot estimate coefficients for variables that have no within-subject variation. Hence, a fixed effects method will not give you coefficients for race, sex, or region of birth. Among adults, it won't be very helpful in estimating effects of height or years of schooling (although there may be a little within-person variation on the latter). Keep in mind, however, that all these stable variables are controlled in a fixed effects regression, even if there are no measurements of them. In fact, the control is likely to be much more effective than in conventional regression. And as we'll see later, you can include interactions between stable variables such as sex and variables that vary over time. But for most observational studies, fixed effects methods are primarily useful for investigating the effects of variables that vary within subject.

For experimental data, the situation with respect to bias and sampling variability is exactly reversed. Bias is eliminated by giving the same set of treatments to all subjects and by randomizing the order in which the treatments are presented. The result is approximately zero correlation between treatment and stable characteristics of the subjects, which means that there is no need for fixed effects to reduce bias. On the other hand, by design, all the variation on the independent variables (the treatments) is within subjects. So no information is lost by restricting attention to the within-subject variation. Indeed, standard errors can be greatly reduced by fixed effects methods because the error term has smaller variance.