CHAPTER 10
A NONPARAMETRIC APPROACH

10.1 PREVIEW

Here we present a nonparametric approach for analysis of continuous method comparison data. Attention is restricted to the evaluation of similarity and agreement. The methodology makes no assumption about either the shape of the data distribution or how the observed measurements are related to the underlying true values. It is an alternative to the normality-based parametric approaches of previous chapters, and is especially attractive for data with marked deviations from normality. It works for unreplicated as well as unlinked repeated measurements data. It takes a statistical functional approach that treats the population quantities, including the measures of similarity and agreement, as features of a population distribution and estimates them using the same features of an empirical distribution. Under certain assumptions, the resulting estimators are approximately normal for large samples. This result is used to develop nonparametric analogs of the procedures in Chapter 7 involving multiple methods. Its application is illustrated using two case studies.

10.2 INTRODUCTION

Consider the setup of Chapter 7. We have measurements from J ( ≥ 2) methods on n subjects. It is assumed that n is large. The measurements may be unreplicated or repeated. The latter are assumed to be unlinked. The unreplicated measurements are denoted by Y_ij, j = 1,..., J, i = 1,...,n. The repeated measurements are denoted by Y_ijk, k = 1,..., m_ij, j = 1 ..., J, i = 1,...,n. In the balanced case when all m_ij are equal, m denotes the common value. To present a unified framework encompassing both data types, we treat unreplicated data as a special case of repeated measurements data with all m_ij = 1. Thus, Y_ij is essentially an alias for Y_ij1, and m = 1 refers to the unreplicated case. As before, (Y₁,...,Y_J) denotes the population vector of single measurements from the J methods on the same subject. Let F (y₁,...,y_J) be its joint cdf. A cdf is also used to refer to the distribution it characterizes.

The methods used in previous chapters are parametric in that the data are assumed to follow a probability distribution of a known form, specifically a normal distribution, characterized in terms of model parameters. The parameters are estimated by the ML method. Under certain assumptions, the ML estimators are efficient, essentially meaning that, if the assumed model is correct and n is large, one cannot find more accurate estimators than them. This optimality property breaks down if the assumed model is wrong, in which case the ML estimators may be a poor choice. In contrast, the methodology of this chapter is nonparametric or distribution-free in that no particular form is assumed for the data distribution, thereby allowing the methodology to be broadly applicable. However, there is a trade-off. Although a nonparametric estimator may be more accurate than the ML estimator if the assumed model is wrong, the converse is true if the assumed model is correct because of the latter’s efficiency. This motivates the common practice of initially attempting a parametric analysis wherein a normality-based model is first fit, followed by model diagnostics to verify the normality assumption. If no serious violations are seen, one proceeds with the parametric analysis, otherwise a nonparametric analysis is pursued. The violations include skewness and outliers in the data. Often, however, the two analyses produce similar inferences regardless of how serious the violations are.

The nonparametric approach also has limitations brought upon by its assumptions or lack thereof, including the following two that are relevant here. First, it does not decompose the observed values into true values and errors. As a result, the usual characteristics of measurement methods such as biases and precisions considered in Section 1.7, and hence the associated measures of similarity are left undefined. Thus, there is a need for alternative measures. Second, the approach does not generalize well to data with features such as heteroscedasticity and covariate effects that call for explicit modeling. It is easier to analyze such data within a parametric framework.

Notwithstanding its limitations, a nonparametric approach remains useful for analysis of basic types of method comparison data. To describe it, we let

and assume that the distribution of (Y₁,...,Y_J) is continuous; the measurements on different subjects are independent; and for each i = 1,...,n, the c_i possible J -tuples formed by selecting one measurement per method from the available measurements on the ith subject, that is,

are identically distributed as (Y₁,...,Y_J).

For unreplicated data, there is only one J-tuple per subject. The assumptions imply they are i.i.d. draws from the distribution of (Y₁,...,Y_J). For repeated measurements data, there are c_i J-tuples from the ith subject, resulting in a total of tuples. These tuples are also identically distributed draws from the distribution of (Y₁,...,Y_J), but independence holds only for the tuples from different subjects. Those from the same subject are dependent. This also means that a subject’s multiple tuples are treated the same way in that they carry the same amount of information.

The population moments and functions thereof, viz., means, variances, and correlations, play a key role in the analysis of method comparison data. These quantities are fine for normal data, but one may question their value for non-normal data. Their usage may be considered particularly problematic if the data have outliers because the nonparametric estimators of population moments are sample moments, which are not robust to outliers. However, a discussion of robust alternatives to moments and measures based on them is beyond the scope of this book (see Bibliographic Note for a reference). Instead, if outliers are seen in the data, then as in a parametric approach, we perform the analysis twice—with and without the outliers—and compare the conclusions. The analysis also involves inference on percentiles. But their nonparametric estimators are relatively more robust to outliers than those of moments.

10.3 THE STATISTICAL FUNCTIONAL APPROACH

This approach is a general method for constructing nonparametric estimators. It treats the population quantities as statistical functionals, that is, functions of the form h(F), where h is a known function and F is the population cdf. Writing a population quantity in this way highlights it as a feature h of the population distribution F. The cdf F is estimated nonparametrically by the empirical cdf . Thereafter, F in h(F) is replaced by to get the plug-in estimator h()—the same feature h of the sample distribution . This estimator may be considered a “natural” estimator for h(F). Parameters such as population moments and percentiles and functions thereof are readily written as statistical functionals (Exercise 10.1), and their sample analogs can be thought of as plug-in estimators (Exercise 10.2).

For a concrete example, consider the population mean µ_j of the jth measurement method. The feature h here is the mean of the jth component of the distribution F of (Y₁,...,Y_J). From unreplicated data, F can be estimated by the empirical cdf,

(10.1)

where I(A) is the indicator function of an event A, defined as

It is readily seen that the mean of the jth component of the distribution is the sample mean (Exercise 10.2). Thus, we have the anticipated result that the sample mean is a plug-in estimator of the population mean.

10.3.1 A Weighted Empirical CDF

It is clear that we need a nonparametric estimator of F to construct plug-in estimators of population quantities. For unreplicated data, the empirical cdf (10.1) is a natural choice. But no such candidate is apparent for repeated measurements data in general. Here we focus on a weighted generalization of (10.1) of the form:

(10.2)

where w is a weight function; m_i = (m_i1,...,m_iJ)^T; and {(Y_i1_k1,...,Y_iJkJ), k_j = 1,..., m_ij, j = 1,..., J} are the c_i J-tuples formed by the measurements on the ith subject. The J-tuples from a subject receive the same weight. The weight depends on the number of replications on the subject and not on (Y₁,...,Y_J). The weight function is non-negative and is assumed to satisfy an unbiasedness condition,

(10.3)

so that is an unbiased estimator of F.

When the design is balanced with all m_ij = m, the weight function in (10.2) does not depend on the subject index. In fact, in this case c_i is a constant and the unique weight function satisfying (10.3) is the constant function

(10.4)

It gives equal weight to all J-tuples in the data. Due to its uniqueness, the resulting may be considered a “standard” empirical cdf for balanced designs. Besides, when m = 1, it reduces to the cdf (10.1) for unreplicated data. Therefore, hereafter we use (10.4) as the weight function for unreplicated as well as balanced repeated measurements data.

For unbalanced data, we focus on two candidate weight functions. They are

(10.5)

The first gives equal weight to each subject i in the data and distributes that weight equally over all its c_i J-tuples. The second gives equal weight to all the J-tuples in the entire dataset. These two are extreme but useful special cases of an optimal weight function (Exercise 10.6). All these weight functions reduce to (10.4) for balanced designs. Thus, choosing between w₁ and w₂ is an issue only for unbalanced designs. In this case, we suggest analyzing data using both of them and comparing the conclusions because neither is a uniformly better choice than the other (see Bibliographic Note).

10.3.2 Distributions Induced by Empirical CDF

The cdf in (10.2) induces a discrete multivariate empirical distribution for (Y₁,...,Y_J), whose possible values are the observed J-tuples in the dataset. Despite the fact that the underlying population distribution is continuous, there may be ties among a method’s measurements and hence among the J-tuples. Allowing for the possibility of ties, the joint probability mass function (pmf) of (Y₁,...,Y_J) under is

(10.6)

The notation p emphasizes that the pmf is associated with the distribution . The inner sum on the right in (10.6) essentially counts the frequency of (y₁,...,y_J) among each subject’s tuples. A weighted sum of these frequencies is the probability of observing (y₁,...,y_J). Upon summing (10.6) over the remaining components, we get the joint pmf of (Y_j, Y_l), j ≠ l as

(10.7)

and the marginal pmf of Y_j as

(10.8)

The expressions for these pmfs are verified in Exercise 10.3. Their special cases when there are no within-method ties are derived in Exercise 10.4 for unbalanced designs and in Exercise 10.5 for balanced designs.

Now that we have the pmfs (10.7) and (10.8) under , we can compute certain moments and percentiles involving (Y_j, Y_l) by simply invoking their definitions. They serve as plug-in estimators of their population analogs under F. In particular, the estimators of first-and second-order moments are

(10.9)

As before, the notation E emphasizes that the expectation is associated with the distribution . It follows that the corresponding estimators of means and variances of Y_j and Y_l and their correlation are

(10.10)

These estimators can also be used in the usual manner to get the corresponding estimators of mean and variance of the difference D_jl = Y_l − Y_j.

Next, let G_jl be the cdf of | D_jl |. It can be written as an expectation as

Hence its plug-in estimator is

(10.11)

Exercise 10.8 gives alternative expressions for the estimators in (10.9) and (10.11); those may be simpler to evaluate. The 100 γth percentile of | D_jl | is defined as the inverse cdf wher

Its sample counterpart is also similarly defined, that is,

(10.12)

10.4 EVALUATION OF SIMILARITY AND AGREEMENT

The nonparametric approach of this chapter does not make any assumptions about: (a) the shape of the data distribution, and (b) how the observed measurements are related to the true values. Although this does not pose any difficulty with measures of agreement such as CCC and TDI, the limits of agreement discussed earlier are not meaningful anymore because their definition is tied to normality of data. In addition, (b) precludes us from defining measures of similarity such as the difference in fixed biases and ratio of precisions. A similar situation occurred in Chapter 4 for paired measurements data when we switched from the mixed-effects model (4.1) to the bivariate normal model (4.6). However, as in that chapter, the similarity of two methods can be examined by directly comparing their means and variances using mean difference and variance ratio, respectively.

Just like Chapter 7, suppose there are Q specified pairwise comparisons of interest involving the J methods in the study. The population vector of their measurements is (Y₁,...,Y_J). Its cdf F is estimated by the weighted empirical cdf given in (10.2) with weights given by (10.4) for balanced designs and by (10.5) for unbalanced designs. For a method pair ( j, l), j < l = 1,..., J of interest, the measures of similarity are the mean difference µ_l − µ_j and the variance ratio σ²_l /σ²_j. Moreover, the measures of agreement are CCC_jl and TDI_jl(p). These measures are statistical functionals of the distribution F (Exercise 10.7). The plug-in estimators of the similarity measures are and those of the agreement measures are

(10.13)

where the quantities involved are given by (10.9) and (10.12).

Thus, we have nonparametric estimators of measures of similarity and agreement for each of the Q pairs of interest. Under certain assumptions, these plug-in estimators are approximately normal from the large-sample theory of statistical functionals, and their SEs can be approximated using estimated moments of influence functions (see Section 10.7). As in Chapter 7, we examine Q two-sided simultaneous confidence intervals for a similarity measure, and Q one-sided simultaneous confidence bounds for an agreement measure. These are of the usual form

(10.14)

possibly on a transformed scale. Specifically, a log transformation is applied for the variance ratios and a Fisher’s z-transformation is applied for the CCCs. One exception to (10.14) is TDI for which upper confidence bounds of the following from is suggested (Section 10.7):

(10.15)

These bounds avoid the estimation of the pdf of | D_jl | in the tails, which is needed for the bounds in (10.14). Such density estimates are generally not stable unless n is quite large. The critical points are N1(0, 1) percentiles when Q = 1. For Q > 1, they are computed numerically as described in Section 3.3 (see also Section 10.7).

Two observations are now in order about the nonparametric estimators of means, variances, and correlation of Y_j and Y_l given by (10.10). First, in the case of unreplicated data with J = 2, these estimators are identical to the ML estimators given by (4.8) under the bivariate normal model (4.6) (Exercise 10.2). Second, in the case of balanced repeated measurements data, the estimators in (10.10) often tend to be close to the ML estimators under the mixed-effects model (7.7). Consequently, the nonparametric and the ML estimators of the moment-based measures, and CCC_jl, coincide in the first case and tend to be close in the second case.

10.5 CASE STUDIES

In this section, we illustrate the nonparametric methodology by using it to analyze two datasets. Both consist of measurements of systolic blood pressure (mm Hg), but the first has unreplicated observations from two methods, whereas the second has replicated observations from three methods. For convenience, we refer to the first as “unreplicated blood pressure data” and the second as “replicated blood pressure data.”

10.5.1 Unreplicated Blood Pressure Data

This dataset has paired blood pressure measurements of 200 subjects taken by a standard method using arm pressure (method 1) and a test method using finger pressure (method 2). There is a total of 200 × 2 = 400 observations in the data. They range between 60 and 228 mm Hg. Figure 10.1 shows their trellis plot. The measurements from the two methods do not have much overlap. The finger’s measurements exceed their arm counterparts for most subjects. The former also exhibit somewhat larger variability than the latter. A handful of subjects have relatively large differences, suggesting that the differences have a skewed distribution. However, there are no clear outliers. There is considerable within-subject variation in the measurements, but this variation is small in relation to the between-subject variation. The data can be considered homoscedastic. The scatterplot in Figure 10.2 shows modest correlation in the methods. The Bland-Altman plot in the same figure shows a mild upward trend, corroborating the somewhat unequal variances of the methods. This trend may also be indicative of unequal scales of the methods but we cannot be sure due to lack of replications. The boxplots in Figure 10.3 confirm the difference in their marginal distributions. They also show right-skewness in the data. The non-normality is confirmed by model diagnostics in Exercise 10.14.

To perform the nonparametric analysis, we first need the empirical cdf . Since these are unreplicated data, is the bivariate cdf that gives 1/n weight to each observed pair.

Under this , the sample means, variances (with divisor n), and correlation of (Y₁, Y₂) are plug-in estimators of their population counterparts (Exercise 10.2). The estimates are

They are consistent with what we expect from the exploratory analysis. They also lead to an estimated mean of 4.3 and standard deviation of 14.5 for the difference D = Y₂ − Y₁. The standard deviation is rather large mainly due to the modest correlation between the methods.

Table 10.1 presents plug-in estimates of the similarity measures µ₂ − µ₁ and σ₂²/σ²₁, and and the agreement measures CCC and TDI(0.90). Also presented are two-sided 95% intervals for the similarity measures and one-sided 95% bounds for the agreement measures. These use percentiles of a standard normal distribution as critical points because only two methods are compared. The critical points are 1.96 for the mean difference and log of variance ratio, 1.645 for the Fisher’s z-transformation of CCC, and −1.645 for the TDI bound, which is computed using (10.15). The interval for µ₂ − µ₁ ranges from 2.3 to 6.3, implying a larger mean for the finger method compared to the arm method. The interval for σ₂²/σ₁² is barely to the right of one, providing borderline evidence for higher variability of the finger method. Thus, on the whole, the methods do not have similar characteristics.

The CCC lower bound of 0.75 suggests a weak agreement in the methods. The TDI upper bound of 28 also suggests this conclusion by clearly showing that the methods cannot be considered to have acceptable agreement because doing so would amount to treating a 28 mm Hg difference in blood pressure as clinically acceptable. It is also apparent from the similarity evaluation that no simple recalibration can bring the two methods into acceptable agreement. Although subtracting 4.3 from the finger’s measurements makes its mean match that of the arm’s, the resulting improvement in CCC and TDI bounds, now at 0.77 and 26.3, respectively, is not large enough for the conclusion to change. For further improvement, the correlation between the methods needs to increase, for example, by increasing the precisions of the methods, thereby reducing the size of the differences.

Even though the normality assumption does not hold for these data, it is of interest to analyze them anyway using a normality-based approach and compare conclusions. The relevant methodology is the one in Chapter 4 that assumes the bivariate normal model (4.6) for the data. Table 10.1 also presents the resulting parametric inferences. We see that the two estimates of every moment-based measure are identical. This is expected because the ML estimators of the first-and second-order moments under (4.6) are also the nonparametric plug-in estimators (Exercise 10.2). But even the confidence intervals and bounds for these measures are identical to the number of decimal places reported. In addition, although the two estimates and bounds for TDI are not identical, there is little practical difference between them. Overall, this seems to suggest that the skewness of these data is not severe enough to invalidate the parametric analysis. Note also that the number of subjects here (n = 200) is relatively large.

10.5.2 Replicated Blood Pressure Data

This dataset, introduced in Exercise 7.12, has three replicate measurements of systolic blood pressure (mm Hg) taken in quick succession on 85 subjects by each of two experienced observers named “J” (method 1) and “R” (method 2) using a sphygmomanometer, and by a semi-automatic blood pressure monitor (method 3). We treat these as unlinked repeated measurements data. The design is balanced with a total of 85 × 3 × 3 = 765 observations ranging from 74 to 228 mm Hg. We are interested in all-pairwise comparisons involving the three methods.

A trellis plot of the data is displayed in Figure 10.4. The measurements of the two observers largely overlap, but those of the monitor tend be larger than theirs. It also has a higher within-subject variation than the observers. This variation does not seem to depend on the measurement’s magnitude in a consistent manner. Also, as in the previous dataset, some subjects have large differences between monitor’s and observers’ measurements, suggesting skewness in the distributions of their differences. But none of the observations is a clear outlier. Figure 10.5 displays a matrix of scatterplots and the corresponding Bland-Altman plots for a subset of data consisting of one measurement per method selected randomly for every subject. The scatterplots show a very high correlation between the observers but only modest correlations between the observers and the monitor. The Bland-Altman plots do not have any trend, implying a common scale for the methods. These plots also corroborate the monitor’s tendency to produce larger measurements than the observers’. The side-by-side boxplots of all measurements from the three methods are displayed in Figure 10.6. Besides confirming the foregoing observation, they also show that the two observers have remarkably similar marginal distributions. In addition, there is clear right-skewness in all three marginal distributions, invalidating the normality assumption for the data (Exercise 7.12).

To get the empirical cdf , 3³ = 27 triplets (or 3-tuples) are formed for each subject using the repeated measurements of the three methods. This results in a total of 85 × 27 = 2295 triplets in the data. Therefore, from (10.4), is a trivariate cdf that gives a weight of 1 /2295 to each triplet. From Exercise 10.15, the plug-in estimates of (mean, standard deviation) of measurements from methods 1, 2, and 3 are (127.4, 31.0), (127.3, 30.7), and (143.0, 32.5), respectively. In addition, the estimated correlation between the methods in the pairs (1, 2), (1, 3), and (2, 3) are 0.97, 0.79, and 0.79, respectively. Thus, the two observers have practically the same estimated means and variances, and their correlation is very high. On the other hand, the monitor produces higher readings than the observers by about 16 mm Hg on average and correlates less with their measurements. It also has a somewhat higher variability than the observers. The standard deviation of the difference between two observers is about 7 and it is three times as much for the difference between the monitor and an observer.

For similarity evaluation, Table 10.2 presents estimates and 95% simultaneous confidence intervals for the three all-pairwise mean differences and variance ratios. The critical point for the former is 2.24 and it is 2.25 for the latter on the log scale. The first interval for mean difference is tight around zero, implying no difference in the means of the two observers. The other two intervals are essentially identical and confirm that the monitor has a higher mean than the observers. The first interval for variance ratio, although barely covering one, is quite tight around 1.02. The value of 1 is deep inside the remaining two intervals, which are also essentially identical. This suggests that all methods can be taken to have the same variance. Overall, the two observers have very similar characteristics, but the same is not true for the monitor-observer pairs due to the larger mean of the monitor.

For agreement evaluation, Table 10.3 presents estimates and 95% simultaneous one-sided confidence bounds for the three all-pairwise CCCs and TDIs with p = 0.90. The critical point is 1.92 for CCC after applying the Fisher’s z-transformation. The TDI critical point, for use in (10.15), is 1.99. The TDI bounds imply that for 90% of subjects the measurement differences between method pairs (1, 2), (1, 3), and (2, 3) are estimated to lie between ±14, ±54, and ±53, respectively. The value of 14 is about 11% of the average value for the observers. This extent of agreement can be considered acceptable. However, this is clearly not the case for the two monitor-observer pairs that have quite similar extent of agreement. The same conclusion is reached on the basis of CCC bounds. It is also evident that the monitor’s unacceptable agreement with the observers is caused by a large mean difference and a relatively low correlation that itself is a consequence of the monitor’s relatively high within-subject variation (see the trellis plot in Figure 10.4). The first cause can be fixed by subtracting 15.6 from the monitor’s measurements. But this recalibration helps only to a limited extent as the new TDI bounds for (1, 3) and (2, 3) pairs are still high at 40 and 38, respectively, and both the new CCC bounds are still relatively low at 0.78. Further improvement requires reducing the monitor’s error variation, which cannot be achieved by a simple recalibration.