6.1.1. Types of Residuals in Linear Regression Models

Let the predicted mean response for subject i at time point j be ${\widehat{\mu }}_{\mathrm{ij}}=X_{ij}^{\prime }\widehat{\beta }$. An n_i-dimensional vector of residuals for each subject can then be derived from a linear mixed model, given by

$r_{\text{m}i}=Y_i-X_i^{\prime }\widehat{\beta }\text{,}$

(6.1)

where $r_{\text{m}i}$, briefly discussed in Chapter 4, is referred to as the vector of the marginal residuals because $X_{ij}^{\prime }\widehat{\beta }$ is a marginal mean vector in linear mixed models. If the random-effects component is considered, residuals for each subject become conditional on the random effects, written as

$r_{\text{c}i}=Y_i-X_i^{\prime }\widehat{\beta }-Z_i{\widehat{b}}_i=r_{\text{m}i}-Z_i{\widehat{b}}_i\text{,}$

(6.2)

where $r_{\text{c}i}$ represents the conditional residuals.

$\mathrm{var}({\widehat{r}}_{\text{m}i})={\widehat{V}}_i-X_i{(X_i^{\prime }{\widehat{V}}_i^{-1}{X}_i)}^{-1}{X}^{\prime }\text{,}$

(6.3)

where ${\widehat{V}}_i$ is the estimated total variance of Y_i, from either maximum likelihood estimate or the restricted maximum likelihood (REML) estimator. The variance–covariance matrix of conditional residuals can be specified according to Gregoire et al. (1995) as follows:

$\mathrm{var}\left({\widehat{r}}_{\text{c}i}\right)=\left({\text{I}}_{n_i}-Z_i\widehat{G}{Z}_i^{\prime }{{\widehat{V}}_i}^{-1}\right)\mathrm{var}\left({\widehat{r}}_{\text{m}i}\right){\left({\text{I}}_{n_i}-Z_i\widehat{G}{Z}_i^{\prime }{{\widehat{V}}_i}^{-1}\right)}^{\prime }\text{.}$

(6.4)

In linear regression models, the distributions of residuals at different data points may not necessarily follow an expected pattern, even if random errors from the true function value are identically distributed. For example, in the conditional residuals, the random effect estimate ${\widehat{b}}_i$, obtained from the empirical Bayes approximation, heavily depends on the normality assumption and is also influenced by the assumed covariance structure ${\widehat{V}}_i$. Therefore, the elements in ${\widehat{b}}_i$ are the empirical best linear unbiased predictors (BLUPs), thereby being shrunk toward the population fixed effects β. As a result, the distribution of the BLUPs does not accurately represent the distribution of the true random effects. In the application of linear mixed models, residuals often need to be adjusted by the expected variability for comparative purposes at different time points. A popular approach for such an adjustment is to standardize the raw residuals by using the estimated residual variance, referred to as studentizing in statistical analysis, given by

$r_{\text{m}i}^{\text{student}}=\frac{r_{\text{m}i}}{\sqrt{\mathrm{var}\left(r_{\text{m}i}\right)}}\text{,}$

(6.5a)

$r_{\text{c}i}^{\text{student}}=\frac{r_{\text{cm}i}}{\sqrt{\mathrm{var}\left(r_{\text{c}i}\right)}}\text{,}$

(6.5b)

where $r_{\text{m}i}^{\text{student}}$ is the vector of studentized marginal residuals for subject i and $r_{\text{c}i}^{\text{student}}$ is the corresponding vector of studentized conditional residuals. Standardization of residuals is particularly important in the longitudinal data with distinctive outliers. Some scientists suggest external studentization of residuals in which a common residual variance estimate is used for all subjects.

$r_{\text{m}i}^{\text{pearson}}=\frac{r_{\text{m}i}}{\sqrt{\mathrm{var}\left(y_i\right)}}\text{,}$

(6.6a)

$r_{\text{c}i}^{\text{pearson}}=\frac{r_{\text{cm}i}}{\sqrt{\mathrm{var}\left(y_i\right)}}\text{.}$

(6.6b)

The application of the Cholesky decomposition on the variance–covariance matrix starts with construction of a lower triangular matrix for each subject, denoted by ${\stackrel{⌢}{C}}_i$, which satisfies the condition

${\widehat{V}}_i={\stackrel{⌢}{C}}_i{\stackrel{⌢}{C}}_i^{\prime }\text{,}$

where ${\stackrel{⌢}{C}}_i$ represents the Cholesky root of ${\widehat{V}}_i$, with ${{\stackrel{⌢}{C}}^{\prime }}_i^{-1}{Y}_i$ having constant variance and zero correlation.

Given the attached properties, the ${\stackrel{⌢}{C}}_i$ matrix can be used to transform the correlated residuals to correlation-free transformed residuals. For the marginal distribution of longitudinal data, the scaled residuals, denoted by $R_{\text{m}i}$, are defined as

$R_{\text{m}i}={\stackrel{⌢}{C}}_i^{-1}{r}_{\text{m}i}={\stackrel{⌢}{C}}_i^{-1}\left(Y_i-X_i^{\prime }\widehat{\beta }\right)\text{,}$

(6.7)

For practical purposes, the scaled residuals can be plotted against the transformed predictions of Y, denoted by ${\widehat{Y}}_i^{*}$ and defined as

${\widehat{Y}}_i^{*}={\stackrel{⌢}{C}}_i^{-1}{X}_i^{\prime }\widehat{\beta }\text{.}$

If a linear mixed model is correctly specified, the plot of the transformed residuals against ${\widehat{Y}}_i^{*}$ should be scattered randomly around zero with a constant range of variation. In contrast, if this scatter-plot displays a systematic trend, residuals remain correlated even with the specified random effects and/or a selected residual covariance structure. In the latter case, more random effect terms need to be considered in the specification of a linear mixed model and in statistical inference. Sometimes, the researcher might want to fit a smooth curve to the scatter-plot. If the linear mixed model is adequately assumed, the fitted line should not display distinctive systematic departures from a horizontal line centered at about 0.8 (Fitzmaurice et al., 2004). The transformed residuals can also be used to identify skewness, detect potential outliers, and assess the normal distribution hypothesis.

6.1.2. Semivariogram in Random Intercept Linear Models

Let $Y_i={\left(Y_{i1}\mathrm{...},Y_{i{n}_i}\right)}^{\prime }$ be an n_i-dimensional vector of repeated measurements for subject i and $e_i={\left(e_{i1},\mathrm{...},e_{i{n}_i}\right)}^{\prime }$ be an n_i × 1 column vector of assumed errors. A generalized multivariate linear regression model is then written as

$Y_i=X_i^{\prime }\beta +e_i\text{,}$

(6.8)

where X_i is a known n_i × M matrix of covariates and β is an M × 1 vector of unknown population parameters. Given the correlation between serial observations for the same subject, e_i is assumed to follow a multivariate normal distribution with mean 0 and n_i × n_i covariance matrix V_i. Correspondingly, Y_i can be expressed in terms of $Y_i\sim \text{MVN}\left(X_i^{\prime }\beta \text{,}{V}_i\right)$. As summarized by Diggle (1988) and Diggle et al. (2002), an appropriate V_i should at least accommodate three different sources of random variations. First, average responses usually vary randomly between subjects, with some subjects being intrinsically high and some being low. Second, a subject’s observed measurement profile may be a response to time-varying stochastic processes. Third, as the individual measurements involve some kind of subsampling within subjects, the measurement process adds a component of variation to the data. This perspective on the breakdown of variability is briefly mentioned in Chapter 1.

$Y_{ij}=X_{ij}^{\prime }\beta +b_i+{\tilde{W}}_i\left(T_{ij}\right)+{\varepsilon }_{ij}\text{,}$

(6.9)

where $X_{ij}^{\prime }\beta$ represents the model-based mean response for subject i at time point j, b_i indicates the i.i.d. variation in average response between subjects with mean 0 and variance ${\sigma }_b^2$, and ɛ_ij represents the subsampling variation within the subject with mean 0 and variance ${\sigma }_{\varepsilon }^2$. The term ${\tilde{W}}_i\left(T_{ij}\right)$ reflects independent stationary Gaussian processes with zero expectation and covariance function ${\sigma }^2\rho \left(\left|T_{j^{\prime }}-T_j\right|\right)$. Clearly, the terms b_i, ${\tilde{W}}_i\left(T_{ij}\right)$, and ɛ_ij correspond to the random intercept, autoregressive correlation, and within-subject random error, respectively, as described previously.

$V_i={\sigma }_b^{2}J+{\sigma }^{2}R\left(T_i\right)+{\sigma }_{\varepsilon }^2\text{I}\text{,}$

(6.10)

where J is a square matrix with all its elements being unity, $T_i={\left(T_{i1},\mathrm{...},T_{i{n}_i}\right)}^{\prime }$ is the time vector, $R\left(T_i\right)$ is a symmetric matrix with element $\rho \left(\left|T_{j^{\prime }}-T_j\right|\right)$, and I is the identity matrix. As indicated in Chapter 5, the form of $\rho \left(\left|T_{j^{\prime }}-T_j\right|\right)$ can be parameterized in different situations. When ${\sigma }^2=0$, the above-generalized multivariate linear model reduces to the uniformed correlation model (Diggle, 1988). If ${\sigma }_b^2$ is also equal to zero, the model reduces further to a typical general linear model with independent random errors. Such cases, however, are empirically very rare in longitudinal data.

Let $T\in T$ be a continuous time variable. Then the semivariogram of a random process $\left\{Y\left(T\right)\right\}:T\in T$ is defined as the function

$g\left(u\right)=\frac{1}{2}\left[\text{E}{\left\{Y\left(T\right)-Y\left(T-u\right)\right\}}^2\right]:u\ge 0\text{,}$

(6.11)

$d_{j{j}^{\prime }}^2=\frac{1}{2}{\left[y\left(T_j\right)-y\left(T_{j^{\prime }}\right)\right]}^2\text{,}$

Empirically, the marginal residuals, denoted by r_mi, can be used to generate the empirical semivariogram. With $r_{ij}=Y_{ij}-X_{ij}^{\prime }\widehat{\beta }$, the empirical variogram can be written as

$\frac{1}{2}\left[\text{E}{\left(r_{ij}-r_{i{j}^{\prime }}\right)}^2\right]={\sigma }_{\varepsilon }^2+{\sigma }^2\left[1-\rho \left(\left|T_{ij}-T_{i{j}^{\prime }}\right|\right)\right]\text{.}$

(6.12)

6.1.3. Semivariogram in the Linear Random Coefficient Model

$Y_i=X_i^{\prime }\beta +Z_i^{\prime }{b}_i+e_i\text{,}$

(6.13)

where e_i is assumed to follow a multivariate normal distribution with mean 0 and n_i × n_i covariance matrix R_i. As indicated earlier, in the standard linear random coefficient model, $\mathrm{cov}\left(e_i\right)={\sigma }^2\text{I}$ is usually assumed given the specified covariates and the random effects. In performing residual diagnostics, however, specifying $\mathrm{cov}\left(e_i\right)=R_i$ as multivariate is considered essential to check whether residuals remain correlated with the specified random coefficients. Chi and Reinsel (1989) propose a score test to examine the random coefficient model with $\mathrm{cov}\left(e_i\right)={\sigma }^2\text{I}$ against the random coefficient multivariate model with the AR(1) errors for e_i. This approach provides a simple check for possible autocorrelation in residuals, which, in linear mixed models, are generally assumed to be conditionally independent. It is found from the score test that the specification of the random effects generally accounts for the serial correlation among repeated measurements, and therefore, the random coefficient model adding a serial correlation term overparameterizes the covariance structure.

$e_i=Z_i^{\prime }{b}_i+{\varepsilon }_{1i}+{\varepsilon }_{2i}\text{,}$

(6.14)

where ɛ_1i is the n_i × 1 residual vector to indicate time-varying stochastic processes operating within the subject (serial correlation), assumed to be normally distributed with mean zero and covariance matrix ${\tilde{H}}_i$. The covariance matrix ${\tilde{H}}_i$ has element ${\tilde{h}}_{ij{j}^{\prime }}$ taking the form ${\tilde{\tau }}^{2}g\left(\left|T_{ij}-T_{i{j}^{\prime }}\right|\right)$, where ${\tilde{\tau }}^2$ is the profiled variance for all elements of ɛ_1i and g(.) is the unknown positive decreasing function. The vector ɛ_2i represents random errors assumed to be independent and identically distributed with zero expectation. The q-dimensional vector b_i, the subject-specific random effects, are assumed to follow a multivariate normal distribution with mean zero and covariance matrix G, as regularly specified.

Given Equation (6.14), the response vector y_i marginally follows a normal distribution with mean vector $X_i^{\prime }\beta$ and covariance matrix

$V_i=Z_{i}G{Z}_i^{\prime }+{\tilde{H}}_i+{\sigma }_{\varepsilon }^2{\text{I}}_{n_i}\text{,}$

(6.15)

where ${\text{I}}_{n_i}$ is the n_i-dimensional identity matrix. This marginal random coefficient multivariate model can be estimated by using either the maximum likelihood (ML) or the REML approach. As discussed by Verbeke et al. (1998), this regression model can be used to check whether a classical linear mixed model sufficiently accounts for intraindividual correlation without considering an additional residual covariance structure.

With the inclusion of the random coefficients, the semivariogram based on the random intercept model is extended to the random coefficient perspective. As the covariance structure specified in Equation (6.15) has been found to be dominated by its first component $Z_{i}G{Z}_i^{\prime }$, it is necessary to remove all variability explained by the random effects b_i before proceeding with the serial correlation check. The scaled residuals, denoted by $R_{\text{m}i}$ and described in Section 6.1.1, can be used for this removal. These standardized residuals are independent of any distributional assumptions on b_i, and therefore, their computation does not require an estimate of the random-effects covariance matrix G (Verbeke et al., 1998). Because the scaled residuals $R_{\text{m}i}$ have unit variance and zero correlation, the semivariogram based on the linear random coefficient model can be written as

$\begin{array}{l}\frac{1}{2}\left[\text{E}{\left(R_{ij}-R_{j{j}^{\prime }}\right)}^2\right]=\frac{1}{2}\mathrm{var}\left(R_{ij}\right)+\frac{1}{2}\mathrm{var}\left(R_{i{j}^{\prime }}\right)-\mathrm{cov}\left(R_{ij},R_{j{j}^{\prime }}\right)\\ =\frac{1}{2}+\frac{1}{2}-0=1.\end{array}$

(6.16)

Equation (6.16) indicates that if a random coefficient linear model is correctly specified, the plot of the semivariogram of the transformed residuals against time should be scattered randomly around unity without displaying any systematic time trend. With this property, the semivariogram can be applied to check whether the specified random effects explain all serial correlation between repeated measurements. In performing this residual diagnostic method, the conditional residuals, defined as $Y_i-X_i^{\prime }\widehat{\beta }-Z_i\widehat{b}$, are not recommended for use because the BLUP ${\widehat{b}}_i$ from the empirical Bayes approximation depends heavily on the normality assumption and is also influenced by the assumed covariance structure ${\widehat{V}}_i$.

6.2.1. Cook’s D and Related Influence Diagnostics

Let $\widehat{\beta }$ be the estimate of β that maximizes the log-likelihood or the log restricted likelihood function and ${\widehat{\beta }}_{\left(-i\right)}$ be the same estimate of β when subject i is eliminated from the estimating process. For a single covariate X_m, the distance in the estimated regression coefficient after removing the subject, denoted ${\stackrel{⌢}{d}}_{mi}$, is written as

${\stackrel{⌢}{d}}_{mi}={\widehat{\beta }}_m-{\widehat{\beta }}_{m_{\left(-i\right)}}.$

(6.17)

Equation (6.17) provides an exact measurement for the absolute influence of deleting subject i from the estimating process on the regression coefficient estimate, referred to as Cook’s distance statistic. This statistic can be expressed in terms of the entire vector of the estimated regression coefficients, given by ${\stackrel{⌢}{d}}_i=\widehat{\beta }-{\widehat{\beta }}_{\left(-i\right)}$. A greater value of ${\stackrel{⌢}{d}}_i$ suggests subject i to have a stronger influence on the estimate of β; likewise, a lower value indicates that subject is impact on the model fit is limited.

${\overline{d}}_i=\frac{\left[\widehat{\beta }-{\widehat{\beta }}_{\left(-i\right)}\right]\prime \left(X\prime X\right)\left[\widehat{\beta }-{\widehat{\beta }}_{\left(-i\right)}\right]}{\left[1+\text{rank}\left(X\right)\right]s2}\text{,}$

(6.18)

where ${\overline{d}}_i$ is the scaled Cook’s distance, or simply Cook’s D, and s² is the mean square error of the data. In the literature of influence diagnostics, the two Cook’s distance statistics, ${\stackrel{⌢}{d}}_i$ and ${\overline{d}}_i$, are also referred to as DFBETA_i and DFBETAS_i, respectively (Belsley et al., 1980; Fox, 1991).

Like the original statistic ${\stackrel{⌢}{d}}_i$, a large value in ${\overline{d}}_i$ indicates that the parameter estimates are sensitive to removal of the ith subject. With standardization, ${\overline{d}}_i$ approximately follows an F-distribution with the numerator degrees of freedom being rank (X) and the denominator degrees of freedom being N-rank (X). Given the F-distribution, the significance of Cook’s D can be statistically tested. Specifically, assuming X to have full rank, the F statistic can be used to test the null hypothesis with threshold F_{M, N − M, α}.

${\overline{d}}_i=\frac{{\left[\widehat{\beta }-{\widehat{\beta }}_{\left(-i\right)}\right]}^{\prime }\mathrm{var}{\left(\widehat{\beta }\right)}^{-1}\left[\widehat{\beta }-{\widehat{\beta }}_{\left(-i\right)}\right]}{\text{rank}\left(X\right)}\text{,}$

(6.19)

where $\mathrm{var}{\left(\widehat{\beta }\right)}^{-1}$ is the matrix from sweeping

$\left[X\prime V\left(\widehat{G},\widehat{R}\right)-1X\right]-1\text{.}$

If V is known, Cook’s D can be evaluated according to a chi-square distribution with the degrees of freedom being the rank of X (Christensen et al., 1992). If V is unknown, an estimate of V needs to be obtained from the approach described in Chapters 3 and 4, and the statistical evaluation of ${\overline{d}}_i$ should be based on the F_{M, N − M, α} distribution. For large samples, however, checking the statistical significance of ${\overline{d}}_i$ for each subject in sequence is unrealistic. In these situations, plotting the scaled statistics graphically is a more practical approach for a quick diagnostic check. An analytic approach to checking the statistical significance of ${\overline{d}}_i$ without removing subjects in sequence will be described in Section 6.2.4.

$\text{PRESS}=\sum _{\stackrel{⌢}{i}\ne i}{\widehat{r}}_{\stackrel{⌢}{i}\left(-i\right)}^2\text{,}$

(6.20)

${\widehat{r}}_{\stackrel{⌢}{i}\left(-i\right)}=y_i-X_i^{\prime }{\widehat{\beta }}_{\left(-i\right)}\text{.}$

6.2.2. Leverage

The basic measurement of leverage is the so-called hat-value, denoted by h_i. In general linear models, the hat-value is specified as a weight variable in the expression of the fitted value of the predicted response ${\widehat{y}}_j$, given by

${\widehat{y}}_{\tilde{j}}=\sum _{i=1}^N{h}_{i\tilde{j}}{y}_i,$

(6.21)

where $h_{i\tilde{j}}$ is the weight of subject i in predicting the outcome Y at data point $\tilde{j}$ ($\tilde{j}=\mathrm{1,2,...},N$), and ${\widehat{y}}_{\tilde{j}}$ is specified as a weighted average of N observed values. Therefore, the weight variable $h_{i\tilde{j}}$ displays the influence of y_i on ${\widehat{y}}_{\tilde{j}}$, with a higher score indicating a greater impact on the fitted value. Let

$h_i=h_{ii}=\sum _{\tilde{j}=1}^N{h}_{i\tilde{j}}^2.$

(6.22)

According to Equation (6.22), the hat-value h_i, with property $0\le h_i\le 1$, is the leverage score of y_i on all fitted values.

$H=X{\left(X^{\prime }X\right)}^{-1}{X}^{\prime }.$

(6.23)

$\widehat{y}=\mathrm{Hy}\text{.}$

(6.24)

$\mathrm{var}\left(\widehat{y}\right)={\sigma }^{2}H\text{,}$

(6.25a)

$\mathrm{var}\left(r\right)={\sigma }^2\left(1-H\right)\text{.}$

(6.25b)

$H=X{\left[X^{\prime }V{\left(\widehat{\Theta }\right)}^{-1}X\right]}^{-}{X}^{\prime }V{\left(\widehat{\Theta }\right)}^{-1}.$

(6.26)

6.2.3. DFFITS, MDFFITS, COVTRACE, and COVRATIO Statistics

$\text{DFFIT}{\text{S}}_i=\frac{{\widehat{y}}_i-{\widehat{y}}_{i\left(-i\right)}}{s_{\left(-i\right)}\sqrt{h_i}}\text{,}$

(6.27)

where ${\widehat{y}}_i$ is the predicted value of Y for subject i from the linear regression including i, ${\widehat{y}}_{i\left(-i\right)}$ is the same prediction after removing data point i in fitting the regression model, $s_{\left(-i\right)}$ is the standard error estimate without i, and h_i is the corresponding leverage score. Given studentizing, the DFFITS statistic follows a Student t distribution multiplied by a leverage factor, given by

$\text{DFFIT}{\text{S}}_i=t_i\sqrt{\frac{h_i}{1-h_i}}\text{.}$

(6.28)

$\text{DFFIT}{\text{S}}_i=\frac{{\widehat{y}}_i-{\widehat{y}}_{i\left(-i\right)}}{\text{ese}\left({\widehat{y}}_i\right)}\text{,}$

(6.29)

where $\text{ese}\left({\widehat{y}}_i\right)$ is the asymptotic standard error estimate for ${\widehat{y}}_i$. This statistic can be approximated by

$\text{ese}\left({\widehat{y}}_i\right)=\sqrt{{x^{\prime }}_i{\left[X^{\prime }V{\left({\widehat{\Theta }}_{\left(-i\right)}\right)}^{-}X\right]}^{-1}}{x}_i,$

(6.30)

$\text{MDFFITS}\left[{\beta }_{\left(-i\right)}\right]=\frac{{\left[\widehat{\beta }-{\widehat{\beta }}_{\left(-i\right)}\right]}^{\prime }\mathrm{var}{\left[{\widehat{\beta }}_{\left(-i\right)}\right]}^{-}\left[\widehat{\beta }-{\widehat{\beta }}_{\left(-i\right)}\right]}{\text{rank}\left(X\right)}\text{.}$

(6.31)

There is a striking similarity between the above MDFFITS[β_(−i)] formulation and Cook’s D, specified by Equation (6.19). Both statistics measure the influence of data points on a vector of the fixed effects, with the only difference being the specification of $\mathrm{var}\left[{\widehat{\beta }}_{\left(-i\right)}\right]$ in Equation (6.31).

If the covariance parameters are assumed to be fixed, the MDFFITS score for each subject can be estimated by a noniterative procedure to check only the fixed effects and the residual variance. The MDFFITS score, however, can be underestimated if a subject’s impact on the covariance parameters is overlooked. Therefore, the iterative approach is preferred to assess the overall impact of a subject, including the influence on the covariance parameter estimates. For the iterative influence analysis, $\mathrm{var}\left[{\widehat{\beta }}_{\left(-i\right)}\right]$ is evaluated at ${\widehat{\Theta }}_{\left(-i\right)}$, and therefore, an MDFFITS score can be computed specifically for the covariance parameters, written as

$\text{MDFFITS}\left[{\Theta }_{\left(-i\right)}\right]={\left[\widehat{\Theta }-{\widehat{\Theta }}_{\left(-i\right)}\right]}^{\prime }\mathrm{var}{\left[{\widehat{\Theta }}_{\left(-i\right)}\right]}^{-1}\left[\widehat{\Theta }-{\widehat{\Theta }}_{\left(-i\right)}\right]\text{.}$

(6.32)

$\text{COVTRACE}\left[{\beta }_{\left(-i\right)}\right]=\left|\text{trace}\left\{\mathrm{var}{\left(\widehat{\beta }\right)}^{-}\mathrm{var}\left[{\widehat{\beta }}_{\left(-i\right)}\right]\right\}-\text{rank}\left(X\right)\right|\text{,}$

(6.33)

$\text{COVRATIO}\left[{\beta }_{\left(-i\right)}\right]=\frac{\det \left\{\mathrm{var}\left[{\widehat{\beta }}_{\left(-i\right)}\right]\right\}}{\det \left[\mathrm{var}\left(\widehat{\beta }\right)\right]}\text{,}$

(6.34)

where $\det \left\{\mathrm{var}\left[{\widehat{\beta }}_{\left(-i\right)}\right]\right\}$ indicates the determinant of the nonsingular part of the $\mathrm{var}\left[{\widehat{\beta }}_{\left(-i\right)}\right]$ matrix. The COVRATIO statistic can be used to assess the precision of the fixed effects with the following criteria: if COVRATIO > 1, inclusion of subject i in the regression improves the precision of the parameter estimates; if COVRATIO < 1, the incorporation of the subject in the estimating process reduces the precision of the estimation, so that subject i may be deleted in the model fit.

$\text{COVTRACE}\left[{\Theta }_{\left(-i\right)}\right]=\left|\text{trace}\left\{\mathrm{var}{\left(\widehat{\Theta }\right)}^{-}\mathrm{var}\left[{\widehat{\Theta }}_{\left(-i\right)}\right]\right\}-\text{rank}\left[\text{var}\left(\widehat{\Theta }\right)\right]\right|\text{,}$

(6.35)

$\text{COVRATIO}\left[\left({\Theta }_{\left(-i\right)}\right)\right]=\frac{\det \left\{\mathrm{var}\left[{\widehat{\Theta }}_{\left(-i\right)}\right]\right\}}{\det \left[\mathrm{var}\left(\widehat{\Theta }\right)\right]}\text{.}$

(6.36)

6.2.4. Likelihood Displacement Statistic Approximation

$\text{L}{\text{D}}_i=2l\left(\widehat{\beta }\right)-2l\left[{\widehat{\beta }}_{\left(-i\right)}\right]\text{,}$

(6.37a)

$\text{RL}{\text{D}}_i=2l_R\left(\widehat{\beta }\right)-2l_R\left[{\widehat{\beta }}_{\left(-i\right)}\right]\text{,}$

(6.37b)

where $l\left(\widehat{\beta }\right)$ is the log-likelihood function and $l_R\left(\widehat{\beta }\right)$ is the log restricted likelihood function with respect to the estimated regression coefficients $\widehat{\beta }$. In simple linear regression models, the LD statistic is distributed as chi-square with one degree of freedom under the null hypothesis that ${\widehat{\beta }}_{\left(-i\right)}=\widehat{\beta }$. With this desirable property, the observations having a strong impact on $\widehat{\beta }$ can be statistically tested as well as graphically identified. Consequently, this statistic provides sufficient information on whether an identified influential case should be removed from the regression.

In longitudinal data analysis, the use of this exact statistic is often not realistic. When the sample size is large, this case-deletion process becomes extremely tedious and time-consuming. Consider, for example, a sample of 2000 subjects: the analyst needs to create 2001 linear mixed models to identify which case or cases have exceptionally strong impact on the parameter estimates. If the distance score, denoted by $\widehat{\theta }-{\widehat{\theta }}_{\left(-i\right)}$, can be statistically approximated by a scalar measurement in linear mixed models, influential observations can be identified without removing each case in a sequence from the estimating process.

Cook (1986) developed a method to approximate Equation (6.37) by introducing weights into the likelihood function, with different individuals allowed different weights. In this method, an N-dimensional vector $w={\left(w_1,w_2,\mathrm{...},w_N\right)}^{\prime }$ is created, with element w_i (i = 1, 2, …, N) being the weight for subject i. The w vector in the classical log-likelihoods can be denoted by $w_0={\left(\mathrm{1,1,...,1}\right)}^{\prime }$, in which each subject has weight one. This approach can be readily extended to linear mixed models. Let θ be a parameter matrix consisting of β, G, and R. The log-likelihood and the log restricted likelihood functions with weights, referred to as perturbed log-likelihoods (Verbeke and Molenberghs, 2000), are then given by

$l\left(\theta \left|w\right.\right)=\sum _{i=1}^N{w}_i{l}_i\left(\theta \right)\text{,}$

(6.38a)

$l_{\text{R}}\left(\theta \left|w\right.\right)=\sum _{i=1}^N{w}_i{l}_{Ri}\left(\theta \right)\text{,}$

(6.38b)

where $l\left(\theta \left|w\right.\right)$ is the perturbed log-likelihood function given the inclusion of the weight vector w, and $l_R\left(\theta \left|w\right.\right)$ is the corresponding perturbed log restricted likelihood function. In longitudinal data analysis, w_i is set at zero if the entire set of observations for subject i is not considered in linear mixed models.

Given the specification of weights, the influential cases can be identified by measuring the distance between ${\widehat{\theta }}_w$ and $\widehat{\theta }$ in the LD formulation. The approximated LD and RLD scores for all subjects, denoted by $\text{LD}\left(w\right)$ or $\text{RLD}\left(w\right)$, are given by

$\text{LD}\left(w\right)=2l\left(\widehat{\theta }\right)-2l\left[{\widehat{\theta }}_w\right]\text{,}$

(6.39a)

$\text{RLD}\left(w\right)=2l_R\left(\widehat{\theta }\right)-2l_R\left[{\widehat{\theta }}_w\right]\text{.}$

(6.39b)

Obviously, it is still unrealistic to evaluate $\text{LD}\left(w\right)$ or $\text{RLD}\left(w\right)$ for all elements in w. Cook (1986) suggests studying the local behavior of $\text{LD}\left(w\right)$ around w₀ because it describes the sensitivity of $\text{LD}\left(w\right)$ with respect to slight perturbations of the weights. Accordingly, the $\text{LD}\left(w\right)$ at w₀ can be approximated by the classical likelihood expressions without directly including w. In the context of linear mixed models, the LD score for subject i can then be approximated by

$\text{L}{\text{D}}_i\left(w_i\right)\approx {\tilde{U}}_i^{\prime }{I}^{-1}\left(\widehat{\theta }\right){\tilde{U}}_i\text{,}$

(6.40)

where ${\tilde{U}}_i$ is the score function for subject I, mathematically defined as the first partial derivative of the log-likelihood function with respect to θ, and $I^{-1}\left(\widehat{\theta }\right)$ is the inverse of the observed Fisher information matrix, mathematically defined as the second partial derivative of the log-likelihood function with respect to θ. Both matrices are evaluated at $\theta =\widehat{\theta }$. As both ${\tilde{U}}_i$ and $I\left(\widehat{\theta }\right)$ can be obtained from the standard linear mixed modeling, this approximation does not include additional statistical inference, thereby evading the specification of w_i. Given Equation (4.16), the $\text{RL}{\text{D}}_i\left(w_i\right)$ statistic can be obtained in the same fashion with minor modifications (Lesaffre and Verbeke, 1998).

As discussed in Lesaffre and Verbeke (1998) and Verbeke and Molenberghs (2000), the change in the log-likelihoods between ${\widehat{\theta }}_w$ and $\widehat{\theta }$ reflects variability of $\widehat{\theta }$. If the $\text{LD}\left(w\right)$ or the $\text{RLD}\left(w\right)$ score is large, $l\left(\theta \right)$ or $l_R\left(\theta \right)$ is strongly curved at $\widehat{\theta }$, thereby suggesting θ to be estimated with high precision. If the statistic is small, θ is estimated with high variability. As a result, a graph of $\text{LD}\left(w\right)$ or $\text{RLD}\left(w\right)$ approximates against the w vector can provide important information on the total local influence of each subject to identify influential cases. For statistical details concerning this graphical method, the reader is referred to Gruttola et al. (1987), Lesaffre and Verbeke (1998), and Verbeke and Molenberghs (2000, Chapter 11).

6.2.5. LMAX Statistic for Identification of Influential Observations

$B\approx {\tilde{U}}^{\prime }{I}^{-1}\left(\widehat{\theta }\right)\tilde{U}\text{,}$

(6.41)

where $\tilde{U}$ is the matrix with rows containing the score residual vector ${\widehat{\tilde{U}}}_i$.

With B defined, Cook considers the direction of the N × 1 vector $\tilde{l}$ that maximizes ${\tilde{l}}^{\prime }B\tilde{l}$, and $\tilde{l}$ is standardized to have unit length. Because the M × M matrix $\widehat{I}{\left(\widehat{\theta }\right)}^{-1}$ is positive definite, the N × N symmetric matrix B is positive semidefinite, with rank no more than M. The statistic ${\tilde{l}}_{\text{max}}$ corresponds to the unit length eigenvector of B, which has the largest eigenvalue ${\tilde{\gamma }}_{\max }$. Therefore, ${\tilde{l}}_{\text{max}}^{\prime }B{\tilde{l}}_{\text{max}}$ maximizes ${\tilde{l}}^{\prime }B\tilde{l}$ and satisfies the equation

$B{\tilde{l}}_{\text{max}}={\ddot{\lambda }}_{\max }{\tilde{l}}_{\text{max}}\text{and}{{\tilde{l}}_{\text{max}}}^{\prime }{\tilde{l}}_{\text{max}}=\text{1,}$

where ${\ddot{\lambda }}_{\max }$ is the largest eigenvalue of B, and ${\tilde{l}}_{\text{max}}$ is the eigenvector associated with ${\ddot{\lambda }}_{\max }$. The elements of ${\tilde{l}}_{\text{max}}$, standardized to unit length, measure sensitivity of the model fit to each observation. The absolute value of ${\tilde{l}}_i$, the ith element in ${\tilde{l}}_{\text{max}}$, is used as the LMAX score for subject i. Given the unit length, the expected value of the squared LMAX statistic for each case is 1/N where N is sample size, and correspondingly, a value significantly greater than this expected value indicates a strong influence on the overall fit of a regression model thereby being identified. If M = 1, the LMAX statistic is proportional to ${\tilde{U}}^{\prime }$ and ${\ddot{\lambda }}_{\max }=I{\left(\widehat{\theta }\right)}^{\text{-1}}∥\tilde{U}∥\text{.}$ When M > 1, an advantage of examining elements of the LMAX statistic is that each case has a single summary measurement of influence.

As the LMAX score is a standardized statistic, it is useful to plot the elements of the LMAX scores against time points and/or the values of other covariates. The standardization of ${\tilde{l}}_{\text{max}}$ to unit length means that the squares of the elements in ${\tilde{l}}_{\text{max}}$ sum up to unity, and therefore, the signs of the elements of ${\tilde{l}}_{\text{max}}$ are not of concern. Therefore, for ${\tilde{l}}_{\text{i}}$, only the absolute value needs to be plotted. Observations that have the most unduly influences on parameter estimates and the model fit can be readily identified by examining the relative influence of the elements in ${\tilde{l}}_{\text{max}}$. If none of the subjects has an undue impact on the model fit, the plot of the LMAX scores should approximate a horizontal line.

There is a lack of analytic expressions for ${\tilde{l}}_{\text{max}}$. Given high proportional contributions of the graphically influential cases to the overall fit of a linear mixed model, it is necessary to check the exact LD for them. The researcher can perform the significance test to further analyze those influential cases with two steps (Liu, 2012). First, graphically identify the most influential observations by using the LMAX approximation (there are usually only a few distinctive outliers in linear models). Next, analytically examine the exact changes in the estimates of the fixed effects and the covariance parameters after deleting each of those few cases, using the classical LD criterion. Following this strategy, a decision can be made regarding whether those influential cases should be removed in fitting a linear mixed model.

6.3.1. Influence Checks on Linear Mixed Model Concerning Effectiveness of Acupuncture Treatment on PCL Score

Table 6.1 displays that each of the ten subjects has multiple observations except subject 271, among whom eight have complete information on the PCL score. All subjects have relatively small values of Cook’s D and the MDFFITS statistics on the fixed effects. As they differ only in the specification of the covariance matrix for the fixed effects, the values of Cook’s D and the MDFFITS are very close, with the MDEFITS value slightly smaller given the use of $\mathrm{var}\left[{\widehat{\beta }}_{\left(-i\right)}\right]$ rather than $\mathrm{var}\left[\widehat{\beta }\right]$ in computation. The COVRATIO and the LD statistics are also fairly stable, thereby suggesting that none of these ten subjects has a strong impact on the fit of the linear mixed model on PCL_SUM. From the results of other influence diagnostics and for other subjects, no distinctively influential cases can be found.

In Table 6.2, the first linear mixed model uses full data, with results previously reported. The second model is fitted after removing subject 791 with four observations. The exact LD statistic, from the formula $2l\left(\widehat{\theta }\right)-2l\left[{\widehat{\theta }}_{\left(-i\right)}\right]$, is statistically significant with four degrees of freedom and at α = 0.05 (LD = 40.0, p < 0.01), thereby indicating that this subject makes a very strong statistical impact on the overall fit of the first linear mixed model. Likewise, the third model is fitted after deleting subject 814, with the exact LD statistic also statistically significant at the same criterion (LD = 42.7, p < 0.01). Furthermore, in both the second and the third models, the parameter estimates, including the regression coefficients and the standard errors, vary notably after removing each of those two cases. Obviously, those two cases make a genuine impact in fitting the linear mixed model. Given the considerable changes in the estimated regression coefficients as well as the exceptionally strong statistical contributions, I would recommend that the two influential cases should be eliminated from the regression. The lack of statistical stability may also be linked to the small sample size for the data. For large samples, a strong relative influence of a few particular cases usually can be washed out by the effects of the vast majority of the normal observations in the estimating process. This argument will be further discussed in the next example where a large sample is analyzed.

6.3.2. Influence Diagnostics on Linear Mixed Model Concerning Marital Status and Disability Severity Among Older Americans

In Table 6.3, the first linear model uses full data, while the second model is fitted after deleting the most influential case (subject ID: 200520020). The exact LD statistic, from the formula $2l\left(\widehat{\theta }\right)-2l\left[{\widehat{\theta }}_{\left(-i\right)}\right]$, is statistically significant with two degrees of freedom (this subject has two observations) and at α = 0.05 (LD = 15.8, p < 0.01), indicating that the most influential case makes a very strong statistical impact on the overall fit of the fixed effects. Likewise, the third mixed model is fitted after deleting the second most influential case (subject ID: 207669010), with the exact LD statistic also statistically significant with six degrees of freedom at the same criterion (LD = 31.3, p < 0.01). In both the second and the third models, however, the fixed estimates, including the regression coefficients and the standard errors, do not vary at all after removing each of those two cases. The fixed effects of the three control variables, not presented in Table 6.3, are also strikingly consistent after removing each of those two cases. The three sets of the estimated regression coefficients are almost identical. Obviously, deleting those two cases makes no genuine impact on the fixed-effects estimates, albeit strong statistical influences.