8.2.1. Basic specifications of generalized linear mixed models

Technically, GLMMs are a straightforward extension of the classical generalized linear perspective from univariate data to clustered measurements. Let $y_i={\left\{y_{i1}\text{,}\mathrm{...}\text{,}{y}_{i{n}_i}\right\}}^{\prime }$ be a realization of a random variable Y_i with mean μ_i and variance–covariance V_i. Suppose that g(y_i, X_i) is a nonlinear function linking y_i to the covariate vector $X_i={\left\{X_{i1}\text{,}\mathrm{...}\text{,}{X}_{i{n}_i}\right\}}^{\prime }$ and a q × 1 vector of unknown individual random effects $b_i={\left\{b_{i1}\text{,}\mathrm{...}\text{,}{b}_{iq}\right\}}^{\prime }$. In the construct of GLMMs, the function g(y_i, X_i, b_i) is assumed to be associated linearly with X_i and b_i where g is a link function. The mean of the response variable for subject i at n_i time points, namely μ_i, is then a modification of Equation (8.1) given the addition of the random effect vector b_i, written as

$\text{E}\left(y_i\right)={\mu }_i=\text{E}\left[g^{-1}\left(X_i^{\prime }\beta +Z_i^{\prime }{b}_i\right)\right]\text{,}$

(8.15)

where E(y_i) is a set of expected values of the conditional distribution of the response given the random effects. With this specification, GLMMs are also referred to as the conditional model in contrast to the marginal GEE, which will be described in Chapter 9. Analogous to the specification in GLMs, β is the M × 1 vector of unknown regression parameters to be estimated. The term Z_i is a design matrix, and b_i represents the between-subjects random effects assumed to be distributed as $\text{N}(\text{0,}G)$. As in linear mixed models, Z_i often contains time or one or more other covariates whose association with the response is assumed to vary between subjects. Given these specifications, the components of Y_i are assumed to be independent of each other conditionally on covariate X_i and the random effect vector b_i.

$\begin{array}{l}\text{var}\left(y_i\right)=\mathrm{var}\left({\mu }_i\right)+\text{E}\left[\phi \upsilon \left({\mu }_i\right)\right]\\ =\mathrm{var}\left[g^{-1}\left(X\prime \beta +Z_i^{\prime }{b}_i\right)\right]+\text{E}\left\{\phi \upsilon \left[g^{-1}\left(X\prime \beta +Z_i^{\prime }{b}_i\right)\right]\right\},\end{array}$

(8.16)

${\eta }_i\left[\text{E}\left(y_i\left|b_i\right.\right)\right]=X_i^{\prime }\beta +Z_i^{\prime }{b}_i\text{.}$

(8.17)

${\eta }_i\left[\text{E}\left(y_i\left|b_i\right.\right)\right]=X_i^{\prime }\beta +Z_i^{\prime }{b}_i+{\varepsilon }_i\text{,}$

(8.18)

where ${\varepsilon }_i={\left\{{\varepsilon }_{i1},\mathrm{...},{\varepsilon }_{i{n}_i}\right\}}^{\prime }$ is the vector of within-subject random errors representing uncertainty in the response. The vector of within-subject random errors ɛ_i, conditionally on b_i, is often assumed to have the property $\text{E}\left({\varepsilon }_i\left|b_i\right.\right)=0$. The specification of such random errors, however, is much more complex than in linear mixed models, with variances and covariances depending on a specific link function belonging to the exponential distribution family.

Let $\tilde{\alpha }$ be a correlation parameter, and $R\left(\tilde{a}\right)$ be the n_i × n_i matrix of $\tilde{\alpha }$ describing the correlation pattern within subject i. In situations where both intraindividual correlation and heterogeneity of within-subject error variance are evident, the common within-subject variance–covariance structure can be expressed as

$\mathrm{var}\left({\varepsilon }_i\left|b_i\right.\right)=\phi A_i^{1/2}\left({\mu }_i\right)R_i\left(\tilde{a}\right)A_i^{1/2}\left({\mu }_i\right)\text{,}$

(8.19)

where $A_i\left({\mu }_i\right)$ is an n_i × n_i diagonal within-subject variance matrix containing elements ν, evaluated at μ_i in the formulation of a specific GLMM, and R_i is unknown.

For analytic simplicity without loss of generality, the matrices included in Equation (8.19) are often assumed to be common to all subjects. Compared to the variance–covariance matrix specified for linear mixed models that depends on i through its dimension, Equation (8.19) allows dependence on i through the subject-specific information and the individual response, given $\beta$ and b_i. Such generalization provides a flexible structure for accommodating complex patterns of within-subject variability associated with nonlinear longitudinal data. In empirical studies, the convenient and commonly used assumption of normality is often made about the conditional distribution of ɛ_i given β and b_i, given by

${\varepsilon }_i\left|\beta ,\right.b_i=\text{N}\left\{0,\phi A_i^{1/2}\left({\mu }_i\right)R_i\left(\tilde{a}\right)A_i^{1/2}\left({\mu }_i\right)\right\}\text{.}$

(8.20)

$V_i=\phi A_i^{1/2}\left({\mu }_i\right)R_i\left(\tilde{a}\right)A_i^{1/2}\left({\mu }_i\right)+Z_i^{\prime }\mathrm{GZ}\text{.}$

(8.21)

8.2.2. Statistical inference and likelihood functions

Let y_i be an n_i-dimensional vector of repeated measurements of the response for subject i and $b_i\sim \text{N}(0,G)$ be the q-dimensional vector of the random effects. By extending the probability distribution of GLMs to the GLMM context, the p.d.f. of any distribution in GLMMs can then be expressed as

$f\left(y_{ij}\left|\beta ,b_i,\phi \right.\right)=\exp \left[\frac{y_{ij}{\theta }_{ij}-\stackrel{⌢}{a}\left({\theta }_{ij}\right)}{\phi }+\tilde{b}\left(y_{ij};\phi \right)\right]\text{,}$

(8.22)

where y_ij is the response measure for subject i at time point j, and θ_ij is the parameter related to the mean of the distribution. In the longitudinal setting, θ_ij can be specified by vector ${\theta }_i={\eta }_i=X_i^{\prime }\beta +Z_i^{\prime }b$. The functions $\stackrel{⌢}{a}\left(\cdot \right)$ and $\tilde{b}\left(\cdot \right)$ as well as the scale factor φ are the same as defined earlier, and the inverse of the mean vector for a known link function is specified by Equation (8.18).

$f_i\left(y_i\left|\beta ,G,\phi \right.\right)=\int \prod _{j=1}^{n_i}{f}_{ij}\left(y_{ij}\left|\beta ,b_i,\phi \right.\right)f\left(b_i\left|G\right.\right)\text{d}{b}_i\text{,}$

(8.23)

$\begin{array}{l}L\left(\beta ,G,\phi \right)=\prod _{i=1}^N{f}_i\left(y_i\left|\beta ,G,\phi \right.\right)\\ =\prod _{i=1}^N\int \prod _{j=1}^{n_i}{f}_{ij}\left(y_{ij}\left|\beta ,b_i,\phi \right.\right)f\left(b_i\left|G\right.\right)\text{d}{b}_i\text{.}\end{array}$

(8.24)

$\begin{array}{l}\log L\left(\beta ,G,\phi \right)=\log \left[\prod _{i=1}^N\int \prod _{j=1}^{n_i}{f}_{ij}\left(y_{ij}\left|\beta ,b_i,\phi \right.\right)f\left(b_i\left|G\right.\right)\text{d}{b}_i\right]\\ =\sum _{i,j}\left[\frac{y_{ij}{\theta }_{ij}-\stackrel{⌢}{a}\left({\theta }_{ij}\right)}{\phi }+b\left(y_{ij};\phi \right)\right]+\sum _i\log {\int }_{-\infty }^{\infty }f\left(b_i\left|G\right.\right)\text{d}b\text{.}\end{array}$

(8.25)

The fact that the GLMMs’ formalization is conditional on the unobservable random effects sometimes calls for the application of the Bayes methodology to derive the model fit. In this approach, prior distributions are specified for β, φ, and G, usually assuming prior independence, with the corresponding densities written as $f(\beta )$, f ($\phi$), and f (G), respectively. There are different ways to select prior distributions, and most recommendations and preferences for the selection are based on the empirical data evidence given a specific link function of the response.

$\begin{array}{l}f\left(\beta ,G,\phi ,b_1,\mathrm{...},b_N\left|Y_1,\mathrm{...},\right.Y_N\right)\\ \propto \prod _{i=1}^N\prod _{j=1}^{n_i}{f}_i\left(y_{ij}\left|\beta ,b_i,\phi \right.\right)\prod _{i=1}^{N}f\left(b_i\left|G\right.\right)f\left(G\right)f\left(\beta \right)f\left(\phi \right)\text{.}\end{array}$

(8.26)

With the complexity of data structures in GLMMs, the Bayes approach is usually applied to estimate the between-subject random effects b_i, the fixed effects β, and the variance components φ and G within an integrated process. In this process, the estimator ${\widehat{b}}_i$ is the value of b_i that maximizes the density $f_i\left(b_i\left|y_i,\beta ,G\right.,\phi \right)$, where the unknown parameters are replaced by the estimates from the ML estimation. The resulting estimates are the empirical Bayes estimates. In the application of GLMMs, nonlinear predictions of the response outcomes are often required to aid in the interpretation of analytic results, and in such predictions, the random effect approximates are included to predict both the conditional and the marginal longitudinal trajectory of individuals.

8.2.3. Procedures of maximization and hypothesis testing on fixed effects

$\begin{array}{l}\frac{\partial }{\partial \beta }l\left(\beta ,G,\phi \right)=\frac{\partial }{\partial \beta }\int f\left(y\left|b\right.\right)f\left(b\left|G\right.\right)\text{d}b\text{/}f\left(y\right)\\ =\int \left[\frac{\partial }{\partial \beta }f\left(y\left|b\right.\right)\right]f\left(b\left|G\right.\right)\text{d}b\text{/}f\left(y\right)\\ =\int \left[\frac{\partial }{\partial \beta }\log f\left(y\left|b\right.\right)\right]f\left(b\left|y\right.\right)\text{d}b\text{.}\end{array}$

(8.27)

$\begin{array}{l}\frac{\partial }{\partial b}l\left(\beta ,G,\phi \right)=\int \frac{\partial \log f\left(\beta \right)}{\partial b}f\left(b\left|y\right.\right)\text{d}b\\ =\text{E}\left[\int \frac{\partial \log f\left(\beta \right)}{\partial b}\left|y\right.\right]\text{.}\end{array}$

(8.28)

Given various estimating procedures for GLMMs, $\widehat{\beta }$ is asymptotically normal, given the large-sample approximation to the joint distribution of parameters, with mean 0 and variance–covariance matrix $I{\left(\widehat{\beta }\right)}^{-1}$. This asymptotic distribution facilitates testing of hypotheses on $\widehat{\beta }$ by applying the standard approaches. For example, given H₀: $\widehat{\beta }=\beta$, the Wald statistic may be used to test $\widehat{\beta }$, given by

${\left(\widehat{\beta }-\beta \right)}^{\prime }I\left(\widehat{\beta }\right)\left(\widehat{\beta }-\beta \right)\sim {\chi }_M^2\text{,}$

(8.29)

where $I\left(\widehat{\beta }\right)$ is the asymptotically consistent estimator for the expected information matrix $I\left(\beta \right)$, analogous to the specification in linear mixed models.

Similarly, the standard likelihood ratio statistic, which compares the likelihoods between two models with different parameter spaces, can be used to test the null hypothesis either on H₀: $\beta =\widehat{\beta }$ for all components in $\widehat{\beta }$ or on H₀: θ_m = ${\beta }_m={\widehat{\beta }}_m$ for a single component in $\widehat{\beta }$. The likelihood ratio statistic is given by

$\Lambda =2l\left(\widehat{\beta }\right)-2l\left(\beta \right)\text{,}$

(8.30)

where l(β) is the log-likelihood function for the model without one or more parameters, and $l\left(\widehat{\beta }\right)$ is the log-likelihood function containing all parameters. The likelihood ratio statistic, Λ, is also asymptotically distributed as ${\chi }_{\left(M\right)}^2$. Consequently, statistical testing on the fixed effects can be conducted with larger values of Λ associated with smaller p-values, thereby providing more evidence against H₀. Specifically, if Λ is associated with a p-value smaller than α, the null hypothesis about $\widehat{\beta }$ should be rejected; if it is associated with a p-value equal to or greater than α, the null hypothesis about $\widehat{\beta }$ is accepted.

8.2.4. Hypothesis testing on variance components

When the significance test is performed on the variance–covariance components of the random effects, some mixture distributions need to be used to test the null and the alternative hypotheses that ${\text{H}}_0:{\sigma }_{b_{\tilde{i}}}^2=0$ and ${\text{H}}_1:{\sigma }_{b_{\tilde{i}}}^2>0$, where ${\sigma }_{b_{\tilde{i}}}^2$ is the $\tilde{i}\text{th}$ element in G. As indicated in Chapter 4, the parameter space for a variance–covariance component of G has open interval (0, ∞), and therefore, the hypotheses are tested on the boundary of the parameter space (Verbeke and Molenberghs, 2003). As a result, the conventional tests, such as the likelihood ratio and the Wald statistics, cannot be directly applied to test the significance of the elements in G. As the G matrix in GLMMs is generally specified to follow multivariate normality, the procedures described in Section 4.5 can be borrowed to approximate a mixture of distributions for the significance test of elements of G in GLMMs.

8.3.1. Penalized quasi-likelihood method

PQL derives the pseudo-likelihood estimates of model parameters based on the assumption that the regression parameters β and b are known and are equal to the current estimates $\tilde{\beta }$ and $\tilde{b}$. Given this hypothesis, the parameters are estimated by the ML approach given multivariate normality of the pseudo data. By implementing this pseudo-likelihood approach, β and b are estimated from the linear mixed model equations, and the unknown parameters in G and R can be estimated either by the ML or by the restricted maximum likelihood (REML) estimator (Wolfinger and O’Connell, 1993). Practically, the procedure is to estimate the fixed and the random parameter components separately and then iterate until the convergence criterion is met.

$\text{E}\left(y\left|b\right.\right)=\mu =g^{-1}\left(X\prime \beta +Z\prime b+\varepsilon \right)=g^{-1}\left(\eta \right)\text{,}$

(8.31)

where $b\sim \text{N}(0,G)$. The variance of y conditionally on b is given by

$\mathrm{var}\left(y\left|b\right.\right)=\phi A^{1/2}R\left(\tilde{\alpha }\right)A^{1/2}\text{.}$

(8.32)

According to Wolfinger and O’Connell (1993), the first-order Taylor series expansion of μ about $\tilde{\beta }$ and $\tilde{b}$ yields approximation of mean

${\text{g}}^{-\text{1}}\left(\eta \right)={\text{g}}^{-\text{1}}\left(\tilde{\eta }\right)+\widetilde{∆}{X}^{\prime }\left(\beta -\tilde{\beta }\right)+\widetilde{∆}{Z}^{\prime }\left(b-\tilde{b}\right)+\epsilon \text{,}$

(8.33)

where $\widetilde{∆}$ is the diagonal matrix of the derivatives of the conditional mean evaluated at the expansion, mathematically defined by

$\widetilde{∆}={\left[\frac{\partial g^{-1}\left(\eta \right)}{\partial \eta }\right]}_{\tilde{\beta },\tilde{b}}\text{.}$

${\widetilde{∆}}^{-1}\left[\mu -{\text{g}}^{-1}\left(\tilde{\eta }\right)\right]+X\prime \tilde{\beta }+Z\prime \tilde{b}=X\prime \beta +Z\prime b\text{.}$

(8.34)

The left side of Equation (8.34) can be regarded as the expected value of the pseudo-response variable conditionally on b, denoted by $\tilde{y}$, given by

${\widetilde{∆}}^{-1}\left[y-{\text{g}}^{-1}\left(\tilde{\eta }\right)\right]+X\prime \tilde{\beta }+Z\prime \tilde{b}\equiv \tilde{y}\text{,}$

(8.35)

$\mathrm{var}\left(\tilde{y}\left|b\right.\right)={\widetilde{∆}}^{-1}{A}^{1/2}R\left(\tilde{\alpha }\right)A^{1/2}{\widetilde{∆}}^{-1}\text{.}$

(8.36)

$\tilde{y}=\dot{X}\beta +Z^{\prime }b+\tilde{\epsilon }\text{,}$

(8.37)

$\tilde{\varepsilon }={\widetilde{∆}}^{-1}\left[\tilde{y}-{\text{g}}^{-1}\left(\tilde{\eta }\right)\right]\text{,}$

and this pseudo-error term $\tilde{\varepsilon }$ is assumed to follow a normal distribution with zero expectation. Therefore, we can specify

$\tilde{y}\left|\beta ,b\right.\sim \text{N}\left(X\prime \beta +Z\prime b,{\widetilde{∆}}^{-1}{A}^{1/2}R\left(\tilde{\alpha }\right)A^{1/2}{\widetilde{∆}}^{-1}\right)\text{.}$

(8.38)

Equation (8.37) takes the form of a linear mixed model with the pseudo-response variable $\tilde{y}$, the fixed effects β, the random effects b, and $\mathrm{var}\left(\tilde{\varepsilon }\right)=\mathrm{var}\left(\tilde{y}\left|b\right.\right)$. Given such specifications, the estimation of parameters can be performed by following the standard procedures for fitting a linear mixed model. Given starting values of β, G, and φ in the marginal likelihood, the empirical Bayes estimates can be computed for b, thereby generating the pseudo-data $\tilde{y}$. Consequently, the approximate linear mixed model can be fitted, yielding updated estimates for β, G, and φ. The iterative process continues until a convergence criterion is reached.

$V\left(\tilde{\theta }\right)=\mathrm{ZG}Z\prime +{\widetilde{∆}}^{-1}{A}^{1/2}R\left(\tilde{\alpha }\right)A^{1/2}{\widetilde{∆}}^{-1}\text{,}$

(8.39)

where $\tilde{\theta }$ contains the unknown parameters in G and R.

Given a penalty term on the random effects, the estimates from optimizing a quasi-likelihood function on $\tilde{y}$ are referred to as PQL estimates by Breslow and Clayton (1993). Correspondingly, the approach for deriving such estimates is called PQL. The application of PQL can be based either on the maximum log pseudo-likelihood or on the restricted log pseudo-likelihood function for the pseudo-response variable $\tilde{y}$. Both estimators, ML or REML, can be performed in SAS by the PROC GLIMMIX procedure, as associated with a specific link function. The reader interested in learning more details about PQL is referred to Breslow and Clayton (1993) and Wolfinger and O’Connell (1993).

8.3.2. Marginal quasi-likelihood method

$\text{E}(y)=\mu =g^{-1}\left(X\prime \beta +\varepsilon \right)\text{.}$

(8.40)

$y\approx g^{-1}\left(X^{\prime }\beta \right)+\widetilde{∆}{Z}^{\prime }b+\epsilon \text{,}$

(8.41)

$\mathrm{var}(y)\approx V_0+{\widetilde{∆}}^{-1}Z\prime \mathrm{GZ}{\widetilde{∆}}^{-1}\text{,}$

(8.42)

$V_0=\phi A^{1/2}R\left(X\prime \beta \right)A^{1/2}\text{.}$

(8.43)

Let $\mu ={\left({\mu }_1,\mathrm{...},{\mu }_N\right)}^{\prime }$ be a marginal mean vector and $\tilde{\theta }$ be an unknown vector of all variance–covariance components. Using the quasi-likelihood equations appropriate for the response variable, the regression coefficients β in the marginal model can then be estimated by solving the following estimating equation

$\tilde{U}\left(\beta ,\tilde{\theta }\right)=\frac{\partial \mu }{\partial \beta }{V}^{-1}\left(y\right)\left(y-\mu \right)=\text{0,}$

where $\tilde{U}\left(\beta ,\tilde{\theta }\right)$ denotes the score statistic vector, mathematically defined as the first partial derivative of the log-likelihood function with respect to β and $\tilde{\theta }$. The inference of the quasi-likelihood functions is provided in Appendix C.

$X^{\prime }{\left(\widetilde{∆}{V}_0\widetilde{∆}+Z\prime GZ\prime \right)}^{-1}\widetilde{∆}\left(y-\mu \right)=0\text{.}$

(8.44)

Given the above specifications, this model specifies the linear predictor $\eta ={\left({\eta }_1,\mathrm{...},{\eta }_N\right)}^{\prime }$ as the vector of linear predictors ${\eta }_i=X_i^{\prime }\beta$, and Fisher scoring is thought to result in the regression $y=\eta +\widetilde{∆}\left(y-\mu \right)$ on X with the weight matrix

$V^{-1}={\left(\widetilde{∆}{V}_0\widetilde{∆}+Z\prime GZ\prime \right)}^{-1}\text{.}$

(8.45)

Consequently, by optimizing a quasi-likelihood function that only includes the first- and the second-order moments, the MQL estimates can be derived. In other words, the information of the random effects is thought to be reflected in the variance–covariance matrix of the marginal mean. Therefore, without the specification of the random effects, this marginal approach is considered not to affect the estimation of regression coefficients. Given the marginal features, the model is referred to as the MQL. According to Goldstein (1991), the estimation of β is the same either with $\varepsilon =\text{N}\left(0,\widetilde{∆}{V}_0\widetilde{∆}\right)$ or with $b=\text{N}\left(0,G\right)$. Given such properties, this quasi-likelihood approximation technique based on linearization only concerns the mean of the random effects, with the estimates evaluated in the marginal linear predictor $X_{ij}^{\prime }\widehat{\beta }$ rather than in the conditional linear predictor $X_{ij}^{\prime }\widehat{\beta }+Z_{ij}^{\prime }{\widehat{b}}_i$. In SAS, the MQL method can also be performed by the PROC GLIMMIX procedure.

8.3.3. The Laplace method

${\int }_{\stackrel{⌢}{a}}^{\stackrel{⌢}{b}}\exp \left[Nf\left(z\right)\right]\text{d}z\text{,}$

(8.46)

$∆f\left(z\right)=\left[\frac{\partial f\left(z\right)}{\partial z_{\stackrel{⌢}{a}}},\frac{\partial f\left(z\right)}{\partial z_{\stackrel{⌢}{a}+1}},\mathrm{...},\frac{\partial f\left(z\right)}{\partial z_{\stackrel{⌢}{b}}}\right]=0\text{.}$

${H^{\prime }}_0=\left[\frac{{\partial }^{2}f\left(z\right)}{\partial z_i\partial z_j}\right]{|}_{z=z_0},$

$f\left(z\right)=f\left(z_0\right)+f^{\prime }\left(z_0\right)\left(z-z_0\right)+\frac{1}{2}{f^{\prime }}^{\prime }\left(z_0\right){\left(z-z_0\right)}^2+O\left[{\left(z-z_0\right)}^3\right]\text{,}$

(8.47)

where $O\left[{\left(z-z_0\right)}^3\right]\stackrel{p}{}0$.

Suppose that z₀ is not an endpoint and ${f^{\prime }}^{\prime }\left(z_0\right)<0$. As $f^{\prime }\left(z_0\right)=0$ given the above-specified conditions, f(z) can be approximated to the quadratic form

$f\left(z\right)=f\left(z_0\right)-\frac{1}{2}{f^{\prime }}^{\prime }\left(z_0\right){\left(z-z_0\right)}^2\text{.}$

(8.48)

$\begin{array}{l}p\left(y\right)=\prod _{i=1}^{N}p\left(y_i\right)=\prod _{i=1}^N\int p\left(y_i\left|b_i\right.\right)p\left(b_i\right)\text{d}{b}_i\\ =\prod _{i=1}^N\int \exp \left\{\log \left[p\left(y_i\left|b_i\right.\right)p\left(b_i\right)\right]\right\}\text{d}{b}_i\\ =\prod _{i=1}^N\int \exp \left\{\sum _{j=1}^{n_i}\log \left[p\left(y_{ij}\left|b_i\right.\right)\right]+n_i\log \left[p\left(b_i\right)\right]\right\}\text{d}{b}_i\text{.}\end{array}$

(8.49)

$\begin{array}{c}l\left(\beta ,\tilde{\theta };\widehat{b},y\right)=\sum _{i=1}^N\left[\sum _{j=1}^{n_i}\log \left[p\left(y_{ij}\left|b_i\right.\right)\right]+n_i\log \left[p\left(b_i\right)\right]\right.\\ +\frac{1}{2}{n}_{b_i}\log \left(2\pi \right)\left.-\log \left|-\frac{1}{2}{n}_i{f^{\prime }}^{\prime }\left(\beta ,\tilde{\theta };{\widehat{b}}_i\right)\right|\right]\text{,}\end{array}$

(8.50)

where $n_{b_i}$ is the common dimension of the random effect b_i.

Maximizing the above log-likelihood function with respect to β, $\tilde{\theta }$, and b yields the MLE of the Laplace method parameters. There are different approaches in the optimization process for finding the appropriate estimates in the application of this integral approximation method. The reader interested in learning more details of those approaches is referred to Pinheiro and Bates (1995) and Wolfinger (1993).

8.3.4. Gaussian quadrature and adaptive Gaussian quadrature methods

${\int }_{-\infty }^{\infty }f\left(z\right)p\left(z\right)\text{d}z\approx \sum _{q=1}^Q{w}_{q}f\left(z_q\right)\text{,}$

(8.51)

${\stackrel{⌢}{b}}_i=G^{-1/2}{b}_i\text{,}$

(8.52)

where $\stackrel{⌢}{b}\sim \text{N}(\text{0}\text{,}\text{I})$ is the standardized matrix of the random effects. Correspondingly, the variance component of the between-subjects random effects can be specified in the linear predictor, given by

${\eta }_i=X_i^{\prime }\beta +Z_i^{\prime }{G}^{1/2}{b}_i\text{.}$

(8.53)

$\begin{array}{l}{f}_i\left(y_i\left|\beta ,G,\phi \right.\right)=\int \prod _{j=1}^{n_i}{f}_{ij}\left(y_{ij}\left|\beta ,b_i,\phi \right.\right)f\left(b_i\left|G\right.\right)\text{d}{b}_i\\ =\int \prod _{j=1}^{n_i}{f}_{ij}\left(y_{ij}\left|\beta ,{\stackrel{⌢}{b}}_i,G,\phi \right.\right)f\left({\stackrel{⌢}{b}}_i\right)\text{d}{\stackrel{⌢}{b}}_i\text{.}\end{array}$

(8.54)

${\int }_{-\infty }^{\infty }\exp \left(-z^2\right)f\left(z\right)\text{d}z\approx \sum _{q=1}^Q{\tilde{w}}_{q}f\left(z_q\right)\text{,}$

(8.55)

${\tilde{w}}_q=\frac{2^{Q-1}Q!\sqrt{\pi }}{Q^2{\left[H_{Q-1}\left(z_q\right)\right]}^2}\text{.}$

$-\log \left[f\left(y_i\left|x_i,b_i,\phi \right.\right)p\left(b_i\left|G\right.\right)\right]$

$\begin{array}{l}\int f\left(y_i\left|X_i,b_i,\phi \right.\right)p\left(b_i\left|G\right.\right)\text{d}{b}_i\approx \\ 2^{n_b/2}\left|\tilde{\Gamma }\left(X_i,\tilde{\theta }\right)\right|{}^{-1/2}\sum _{q_1=1}^Q\mathrm{...}\sum _{q_{n_b}=1}^Q\left[f\left(y_i\left|X_i,c_{q\mathrm{1,...},q{n}_b},\phi \right.\right)f\left(c_{q\mathrm{1,...},q{n}_b}\left|G\right.\right)\prod _{r=1}^{n_b}{w}_{qr}\exp \left(z_{qr}^2\right)\right],\end{array}$

(8.56)

where n_b is the dimension of b_i, $\tilde{\Gamma }\left(X_i,\tilde{\theta }\right)$ is the Hessian matrix from the empirical Bayes minimization, and

$c_{q\mathrm{1,...},qR}={\stackrel{⌢}{b}}_i+2^{-1/2}{\left|\tilde{\Gamma }\left(X_i,\tilde{\theta }\right)\right|}^{-1/2}{z}_{q\mathrm{1,...},qR}\text{.}$

8.3.5. Markov Chain Monte Carlo methods

${\int }_a^b\tilde{h}\left(Y\right)\text{d}Y={\int }_a^{b}f\left(Y\right)p\left(Y\right)\text{d}Y=E_{p\left(Y\right)}\left[f\left(Y\right)\right]\cong \frac{1}{n}\sum _{i=1}^{n}f\left(Y_i\right)\text{,}$

(8.57)

where ${\int }_{-\infty }^{\infty }\tilde{h}\left(Y\right)\text{d}Y$ is a complex integral. Let $\widehat{\tilde{H}}\left(Y\right)$ be the approximated integral. Then, the estimated Monte Carlo standard error can be written as

$\text{se}\left[\widehat{\tilde{H}}\left(Y\right)\right]=\sqrt{\frac{1}{n}\left\{\frac{1}{n-1}\sum _{i=1}^n{\left[f\left(Y_i\right)-\widehat{\tilde{H}}\left(Y\right)\right]}^2\right\}}\text{.}$

(8.58)

In the analysis of nonlinear longitudinal data with the specified random effects, the Monte Carlo integration, as a Bayes-type technique, can be used to approximate marginal posterior distributions given the assumption of a Markov Chain process. Both the Metropolis–Hastings algorithm and the Gibbs sampler are typical applications of the Markov Chain techniques. In those approaches, attempts are made to draw samples from some distributions. In terms of the Metropolis–Hastings algorithm, the goal is to draw samples from a probability distribution to approximate the distribution of interest and then accept or reject the drawn value with a specified probability (Metropolis et al., 1953; Hastings, 1970). The original Metropolis algorithm starts with some initial value θ₀ that satisfies $f\left({\theta }_0\right)>0$. Using the current θ value, a candidate point θ^* is then sampled from a jumping distribution $\tilde{q}\left({\theta }_1,{\theta }_2\right)$, which is the probability of returning a value of θ₂ given a previous value of θ₁. In the Metropolis algorithm, the only restriction on the jumping density is that it is symmetric, and therefore, $\tilde{q}\left({\theta }_1,{\theta }_2\right)=\tilde{q}\left({\theta }_2,{\theta }_1\right)$. Given the candidate point θ^*, one can compute the ratio of the density at the candidate $\left({\theta }^{*}\right)$ and current $\left({\theta }_{j-1}\right)$ points, denoted $\stackrel{⌢}{\alpha }$ and given by

$\stackrel{⌢}{\alpha }=\frac{p\left({\theta }^{*}\right)}{p\left({\theta }_{j-1}\right)}=\frac{f\left({\theta }^{*}\right)}{f\left({\theta }_{j-1}\right)}\text{.}$

The measurement $\stackrel{⌢}{\alpha }$ provides a standard for accepting or rejecting the candidate point. If the jump increases the density $\left(\stackrel{⌢}{\alpha }>1\right)$, accept the candidate point $\left({\theta }_j={\theta }^{*}\right)$ and return to the step of selection. If the jump decreases the density $\left(\stackrel{⌢}{\alpha }<1\right)$, with probability $\stackrel{⌢}{\alpha }$ accept the candidate point, otherwise reject it and return to the step of selection. Therefore, in the application of the Metropolis sampling, one should first compute the value of $\stackrel{⌢}{\alpha }$ given by

$\stackrel{⌢}{\alpha }=\min \left[\frac{f\left({\theta }^{*}\right)}{f\left({\theta }_{j-1}\right)},1\right]\text{,}$

(8.59)

and then accept a candidate point with the probability of a move ($\stackrel{⌢}{\alpha }$).

Hastings (1970) generalizes the above algorithm by using an arbitrary transition probability function $\tilde{q}\left({\theta }_1,{\theta }_2\right)=\mathrm{Pr}\left({\theta }_1\to {\theta }_2\right)$ and setting the acceptance probability for a candidate point as

$\stackrel{⌢}{\alpha }=\min \left[\frac{f\left({\theta }^{*}\right)\tilde{q}\left({\theta }^{*},{\theta }_{j-1}\right)}{f\left({\theta }_{j-1}\right)\tilde{q}\left({\theta }_{j-1},{\theta }^{*}\right)},1\right]\text{.}$

(8.60)

Equation (8.60) is referred to as the Metropolis–Hastings algorithm. The use of this algorithm is based on the scenario that if $\stackrel{⌢}{\alpha }>1$, the value of the candidate point θ^* is accepted and the equation ${\theta }_j={\theta }^{*}$ is set; if $\stackrel{⌢}{\alpha }<1$, the value of θ^* is randomly accepted as the next iterate θ_j with probability $\stackrel{⌢}{\alpha }$, and otherwise, keep the current value ${\theta }_j={\theta }_{j-1}$.

The Gibbs sampler, perhaps the most popular MCMC method, was originally developed by mathematical physicists in image processing (Geman and Geman, 1984) and later introduced and formalized into the realm of statistics by Gelfand and Smith (1990). This approach can be regarded as a special case of the Metropolis–Hastings algorithm where the random value θ^* is always accepted ($\stackrel{⌢}{\alpha }=1$). Specifically, the Gibbs sampler draws samples from the conditional distribution of each component of a multivariate random variable given the other components in a cyclic fashion. The primary feature of this method is such that only univariate conditional distributions are considered. Compared to the Metropolis–Hastings algorithm, the specification of univariate conditional distributions is far easier to simulate than that of complex joint distributions with simple forms. In practice, one only needs to simulate n random variables sequentially from n univariate conditions rather than generate a single n-dimensional vector by specifying a full joint distribution.

For practical purposes, the procedure of performing the Gibbs sampler is described below by borrowing the specification from Gelfand and Smith (1990). Suppose that three variables, X, Y, and Z, are considered and the conditional distribution of each is denoted by $\left(X\left|Y,Z\right.\right)$, $\left(Y\left|X,Z\right.\right)$, and $\left(Z\left|X,Y\right.\right)$, respectively. The joint distribution is denoted by $\left(X,Y,Z\right)$, assumed to be positive over its entire domain for ensuring the full determination of the joint distribution by the three conditions. The Gibbs sampler considers a sequence of conditional distributions to generate a random variate $\left(X,Y,Z\right)$. With arbitrary starting values $X^{\left(0\right)}$, $Y^{\left(0\right)}$, $Z^{\left(0\right)}$, $X^{\left(1\right)}$ can be drawn from $\left(X\left|Y^{\left(0\right)},Z^{\left(0\right)}\right.\right)$, $Y^{\left(1\right)}$ from $\left(Y\left|X^{\left(0\right)},Z^{\left(0\right)}\right.\right)$, and $Z^{\left(1\right)}$ from $\left(Z\left|X^{\left(0\right)},Y^{\left(0\right)}\right.\right)$, respectively. After a large number $\tilde{n}$ of iterations, $\left(X,Y,Z\right)$ is approximated by $\left(X^{\left(\tilde{n}\right)},Y^{\left(\tilde{n}\right)},Z^{\left(\tilde{n}\right)}\right)$. As $\tilde{n}\to \infty$, the joint distribution of $\left(X^{\left(\tilde{n}\right)},Y^{\left(\tilde{n}\right)},Z^{\left(\tilde{n}\right)}\right)$ converges in probability at an exponential rate to $\left(X,Y,Z\right)$ (Geman and Geman, 1984). Consequently, given the joint distribution uniquely defined, the Gibbs sampler can extract the marginal distributions from the full conditional distributions.

In GLMMs, the Gibbs sampler can be applied to approximate the joint distribution $\left(\beta ,G,b\left|y\right.\right)$ from the marginals $\left(\beta ,G\left|y\right.\right)$ and $\left(b_i\left|y\right.\right)$. By extending the above specifications, the joint distribution can be obtained from sampling the conditional distributions with a predetermined value of $\tilde{n}$. In SAS, the PROC MCMC procedure provides a flexible, simulation-based procedure for applying the Gibbs sampler given the specification of a likelihood function for the data and a prior distribution for parameters. The Gibbs sampler, together with other MCMC methods, however, is not considered to be as statistically efficient and mathematically precise as the Gaussian quadrature and the Laplace techniques. For more details concerning various MCMC methods, the reader is referred to Gelfand and Smith (1990), Geman and Geman (1984), Hastings (1970), Metropolis et al. (1953), Schafer (1997), Tanner and Wong (1987), and Zeger and Karim (1991).