4.6. Overdispersion
When estimating logit models with grouped data, it often happens that the model doesn’t fit—the deviance and Pearson chi-square are large, relative to the degrees of freedom. Lack of fit is sometimes described as overdispersion. Overdispersion has two possible causes:
An incorrectly specified model: more interactions and/or nonlinearities are needed in the model.
Lack of independence of the observations: this can arise from unobserved heterogeneity that operates at the level of groups rather than individuals.
We’ve already seen examples of the first cause. Now let’s look at an example where overdispersion may arise from dependence among the observations. The sample consists of 40 U.S. biochemistry departments in the late 1950s and early 1960s (McGinnis, Allison and Long 1982). Three variables are measured:
| NIH | Total NIH obligations to the university in 1964, in millions of dollars |
| DOCS | Number of biochemistry doctorates awarded during the period |
| PDOC | Number of doctorates who got postdoctoral training |
The aim is to estimate a logit model predicting the probability that a doctorate will receive postdoctoral training.
Here’s the SAS code to read in the data:
DATA my.nihdoc; INPUT nih docs pdoc; DATALINES; .5 8 1 .5 9 3 .835 16 1 .998 13 6 1.027 8 2 2.036 9 2 2.106 29 10 2.329 5 2 2.523 7 5 2.524 8 4 2.874 7 4 3.898 7 5 4.118 10 4 4.130 5 1 4.145 6 3 4.242 7 2 4.280 9 4 4.524 6 1 4.858 5 2 4.893 7 2 4.944 5 4 5.279 5 1 5.548 6 3 5.974 5 4 6.733 6 5 7 12 5 9.115 6 2 9.684 5 3 12.154 8 5 13.059 5 3 13.111 10 8 13.197 7 4 13.433 86 33 13.749 12 7 14.367 29 21 14.698 19 5 15.440 10 6 17.417 10 8 18.635 14 9 21.524 18 16 ;
We then specify a logit model by using GENMOD with the event/trials syntax:
PROC GENMOD DATA=my.nihdoc; MODEL pdoc/docs=nih / D=B; RUN;
In Output 4.14, we see that there is a highly significant effect of NIH obligations. Each one-million dollar increase is associated with a 100[1–exp(.0729)])=7.6% increase in the odds that a graduate will pursue postdoctoral training. But the model doesn’t fit well. The deviance is nearly 70% larger than the degrees of freedom, with a p-value less than .01.
Output 4.14. GENMOD Results for Doctorate Data
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 38 64.1642 1.6885
Scaled Deviance 38 64.1642 1.6885
Pearson Chi-Square 38 61.0211 1.6058
Scaled Pearson X2 38 61.0211 1.6058
Log Likelihood . -306.2974 .
Analysis Of Parameter Estimates
Parameter DF Estimate Std Err ChiSquare Pr>Chi
INTERCEPT 1 -0.7871 0.1748 20.2746 0.0001
NIH 1 0.0729 0.0158 21.2405 0.0001 |
What can we do about this? Well, we can get a saturated model by deleting NIH and putting in dummy variables for 39 of the 40 universities. That wouldn’t be very informative, however—we might just as well look at the percentages receiving postdoctoral training across the 40 universities. Because there’s only one independent variable, we don’t have the option of including interactions, but we can allow for nonlinearities by including powers of NIH in the model. I tried a squared term, but that didn’t help at all. The addition of a cubed term got the p-value up to .032, but further powers didn’t bring any improvement (I tried up to the 7th power). In short, any polynomial model with enough terms to fit would be so complicated that it would have little appeal over the saturated model.
However, it’s quite possible that the lack of fit is due not to departures from linearity in the effect NIH funding but to a lack of independence in the observations. There are many characteristics of biochemistry departments besides NIH funding that may have some impact on whether their graduates seek and get postdoctoral training. Possibilities include the prestige of the department, whether the department is in an agricultural school or a medical school, and the age of the department. Omitting these variables from the model could induce a residual correlation among the observations: people from the same department tend to have the same outcome because they share a common environment. This lack of independence will produce what is called extra-binomial variation—the variance of the dependent variable will be greater than what is expected under the assumption of a binomial distribution. Besides producing a large deviance, extra-binomial variation can result in underestimates of the standard errors and overestimates of the chi-square statistics.
One approach to this problem is to adjust the chi-squares and test statistics, leaving the coefficient estimates unchanged. The adjustment is quite simple: Take the ratio of the goodness-of-fit chi-square to its degrees of freedom, and then divide all the individual chi-squares by that ratio. Equivalently, take the square root of the ratio and multiply all the standard errors by that number. This adjustment can be based either on the Pearson chi-square or the deviance, but most authorities prefer the Pearson statistic.
The adjustment is easily implemented in GENMOD. In the MODEL statement, the option DSCALE makes the adjustment with the deviance chi-square and the option PSCALE uses the Pearson chi-square. In LOGISTIC, the corresponding options are SCALE=D and SCALE=P. You must also include the AGGREGATE option or LOGISTIC won’t know how to compute the goodness-of-fit statistics.
Output 4.15 shows the results for GENMOD with the PSCALE option. For the first time, we see something besides 1.0 for the SCALE parameter. As noted at the bottom of the output, this is just the square root of the Pearson’s chi-square divided by the degrees of freedom. The standard errors were obtained by multiplying the original standard errors by this scale factor. The chi-squares were obtained by dividing the chi-squares in Output 4.14 by 1.6058, resulting in a decline from 21.2 to 13.2 for NIH. Because the deviance and the Pearson chi-square are pretty close for this data, switching to the DSCALE option wouldn’t make much difference.
Output 4.15. Doctorate Model with Overdispersion Adjustment
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 38 64.1642 1.6885
Scaled Deviance 38 39.9573 1.0515
Pearson Chi-Square 38 61.0211 1.6058
Scaled Pearson X2 38 38.0000 1.0000
Log Likelihood . -190.7424 .
Analysis Of Parameter Estimates
Parameter DF Estimate Std Err ChiSquare Pr>Chi
INTERCEPT 1 -0.7871 0.2215 12.6257 0.0004
NIH 1 0.0729 0.0200 13.2272 0.0003
SCALE 0 1.2672 0.0000 . .
NOTE: The scale parameter was estimated by the square root of
Pearson's Chi-Squared/DOF. |
Note that the coefficients in Output 4.15 with the PSCALE option are exactly the same as those in Output 4.14 without the overdispersion correction. PROC LOGISTIC offers an additional overdispersion correction proposed by Williams (1982) that modifies the coefficients as well as the standard errors. Based on the method of quasi-likelihood, these coefficients may be more statistically efficient than the conventional estimates. The SAS code for implemeting this correction is
PROC LOGISTIC DATA=my.nihdoc; MODEL pdoc/docs=nih / D=B AGGREGATE SCALE=WILLIAMS; RUN;
Results are shown in Output 4.16. Williams’ method uses iterative reweighting of the observations, and this is reflected in much of the reported output. In the preliminary output, we see the weight function used at the final iteration, along with the sum of the weights. This sum can be thought of as the effective sample size after correction for overdispersion. Although both Pearson and deviance chi-squares are reported, there is a warning that they are not to be used to assess the fit of the model. Both the coefficient and the chi-square for NIH are a little larger than they were under the simpler PSCALE correction in Output 4.15. (For other approaches to analyzing this data see Allison (1987)). Note that when the group size variable (DOCS in this example) has the same value for all groups, Williams’ method does not alter the conventional coefficient estimates, and the standard error correction is the same as the simpler adjustment using Pearson’s chi-square.
Output 4.16. Doctorate Model with Williams’ Adjustment for Overdispersion
Weight Variable: 1 / ( 1 + 0.042183 * (DOCS - 1) )
Sum of Weights: 276.89669094
Link Function: Logit
Response Profile
Ordered Binary Total
Value Outcome Count Weight
1 EVENT 216 133.24419
2 NO EVENT 243 143.65250
Deviance and Pearson Goodness-of-Fit Statistics
Pr >
Criterion DF Value Value/DF Chi-Square
Deviance 38 39.8872 1.0497 0.3862
Pearson 38 38.0000 1.0000 0.4695
Number of events/trials observations: 40
WARNING: Because the Williams method was used to accommodate
overdispersion, the Pearson chi-squared statistic and the
deviance can no longer be used to assess the goodness of fit
of the model.
Model Fitting Information and Testing Global Null Hypothesis BETA=0
Intercept
Intercept and
Criterion Only Covariates Chi-Square for Covariates
AIC 385.469 370.396 .
SC 389.598 378.654 .
-2 LOG L 383.469 366.396 17.073 with 1 DF (p=0.0001)
Score . . 16.788 with 1 DF (p=0.0001)
Analysis of Maximum Likelihood Estimates
Parameter Standard Wald Pr > Standardized Odds
Variable DF Estimate Error Chi-Square Chi-Square Estimate Ratio
INTERCPT 1 -0.7522 0.2084 13.0333 0.0003 . .
NIH 1 0.0829 0.0206 16.1428 0.0001 0.218879 1.086 |
What makes the doctorate example different from the earlier examples in this chapter is that the individuals are grouped into naturally occurring clusters, in this case, university departments. It’s reasonable to suppose that individuals in the same department are not independent—not only can they influence each other, but they are also exposed to many common factors that may produce the same outcome. In most contingency table analyses, on the other hand, individuals are grouped together merely because they have the same values on some discrete variable. In such cases, there is usually no reason to think that the observations within groups are anything other than independent. When independence is presumed, the correct strategy for dealing with overdispersion is to elaborate the model until you find a version that does fit the data, not to casually invoke the overdispersion options. If, after diligent investigation, you cannot come up with a reasonably parsimonious model with an acceptable fit to the data, then there may be some value in correcting the standard errors and test statistics for any remaining overdispersion. But be aware that the correction for overdispersion always produces chi-square tests that are lower than they would be without the correction. Note also that overdispersion does not arise from heterogeneity within groups. The problem stems from differences between groups, which are not fully described by the measured variables. In Chapter 8, we’ll see how to analyze individual-level data when there is clustering and a lack of independence. The problem is the same—standard errors that are biased downward—but the solution is rather different.
It’s also worth noting that the “badness of fit” for the NIH data comes primarily from a small number of observations. When I used the OBSTATS option in the MODEL statement of GENMOD, I found that two departments had deviance residuals (see Section 2.8) greater than 2.5. In particular, the largest department—with 86 doctorates and 33 postdocs—had a predicted value of 47 postdocs. The department with 19 doctorates had a predicted value of 11 postdocs, but only 5 observed postdocs. When the model is fit after deleting these two observations, the deviance falls to 37.9 with 36 degreees of freedom, for a p-value of .38. While this result is not inconsistent with the hypothesis of overdispersion, it does suggest that special attention be paid to these two departments to determine how they differ from others.
One way they differ is size. Both departments graduated many more doctorates than the median number of 8. Perhaps we can improve the fit by including DOCS as an independent variable in the model:
PROC GENMOD DATA=my.nihdoc; MODEL pdoc/docs=nih docs/ D=B; RUN;
As shown in Output 4.17, this change does produce a much better fitting model, with a p-value of .07. Departmental size has a highly significant, negative effect on the probability that a new doctorate will get a postdoctoral fellowship. In general, the most desirable way to deal with overdispersion is to incorporate covariates that account for differences among the groups. But that may not always be possible.
Output 4.17. Doctorate Model with DOCS as a Covariate
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 37 50.3161 1.3599
Scaled Deviance 37 50.3161 1.3599
Pearson Chi-Square 37 47.8917 1.2944
Scaled Pearson X2 37 47.8917 1.2944
Log Likelihood . -299.3734 .
Analysis Of Parameter Estimates
Parameter DF Estimate Std Err ChiSquare Pr>Chi
INTERCEPT 1 -0.6662 0.1786 13.9102 0.0002
NIH 1 0.0988 0.0178 30.6586 0.0001
DOCS 1 -0.0131 0.0036 13.3971 0.0003 |