Confidence interval around the difference

$\overline{D}\pm t_{\text{a}}\frac{s_{\text{D}}}{\sqrt{n}}$

where $\overline{D}$ is the mean of the difference scores (as was used in computing the test statistic),

$29.5\pm 2.06\frac{14.125}{\sqrt{26}}$

$29.5\pm 5.705$

Practical significance

The difference is statistically significant, but is it practically significant? To answer this question depends on how we interpret the lowest and highest plausible differences. Even the lowest estimate of the difference of 23.8 points puts Product A at 45% higher than Product B. It also helps to know something about SUS scores. A difference of 23.8 points crosses a substantial range of products and places Product A’s perceived usability much higher than Product B’s relative to hundreds of other product scores (Sauro, 2011a; also see Chapter 8, Tables 8.5 and 8.7). Given this information it seems reasonable to conclude that users would notice the difference in the usability and it suggests that the difference is both statistically and practically meaningful.

Technical Note 2: For the confidence interval formula we use the convention that t_a represents a two-sided confidence level. Many, but not all, statistics books use the convention $t_{\left(1-\frac{\alpha }{2}\right)}$ which is based on a table of values that is one-sided. We find this approach more confusing because in most cases you’ll be working with two-sided rather than one-sided confidence intervals. It is also inconsistent with the Excel TINV function, which is a very convenient way to get desired values of t when computing confidence intervals.

Normality assumption of the paired t-test

Confidence interval around the difference

$({\widehat{x}}_1-{\widehat{x}}_2)\pm t_{\text{a}}\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}$

where ${\widehat{x}}_1\text{and}{\widehat{x}}_2$ are means from samples 1 and 2,

$s_1\text{and}{s}_2$ are standard deviations from samples 1 and 2,

$n_1\text{and}{n}_2$ are the sample size from samples 1 and 2, and

$51.6-49.6\pm 2.086\sqrt{\frac{{4.07}^2}{11}+\frac{{4.63}^2}{12}}$

$2.0\pm 3.8$

Example 2: Comparing two task times

Twenty users were asked to add a contact to a CRM application. Eleven users completed the task on the existing version and nine different users completed the same task on the new enhanced version. Is there compelling evidence to conclude that there has been a reduction in the mean time to complete the task? The raw values (in seconds) appear in Table 5.4.

Table 5.4

Data for Comparison of Task Times from Independent Groups

Old	New
18	12
44	35
35	21
78	9
38	2
18	10
16	5
22	38
40	30
77
20

The mean task time for the 11 users of the old version was 37 s with a standard deviation of 22.4 s. The mean task time for the nine users of the new version was 18 s with a standard deviation of 13.4 s. Plugging in the values we get

$t=\frac{{\widehat{x}}_1-{\widehat{x}}_2}{\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}}$

$t=\frac{37-18}{\sqrt{\frac{{22.4}^2}{11}+\frac{{13.4}^2}{9}}}$

$t=2.33$

The observed difference in mean times generates a test statistic (t) equal to 2.33. To determine whether this is significant we need to find the p-value using a t-table, the Excel function =TDIST, or the calculator available at http://www.usablestats.com/calcs/tdist.

We have 16 degrees of freedom (see the sidebar on “Degrees of Freedom for the Two-sample t-test”) and want the two-sided area, so the p-value is =TDIST(2.33,16,2) = 0.033. Because this value is rather small (less than 0.05) there is reasonable evidence that the two task times are different. We can conclude users take less time with the new design. From this sample we can estimate the likely range of the difference between mean times by generating a confidence interval. For a 95% confidence interval with 16 degrees of freedom, the critical value of t is 2.12. See http://www.usablestats.com/calcs/tdist for obtaining critical values from the t-distribution.

Plugging the values into the formula we get

$({\widehat{x}}_1-{\widehat{x}}_2)\pm t_{\text{a}}\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}$

$37-18\pm 2.12\sqrt{\frac{{22.4}^2}{11}+\frac{{13.4}^2}{9}}$

$19\pm 17.2$

We can be 95% confident the difference in mean times is between about 2 and 36 s.

Normality

Equality of variances

Don’t worry too much about violating assumptions (except representativeness)

Chi-square test of independence

${\chi }^2=\frac{{(ad-bc)}^{2}N}{mnrs}$

Degrees of freedom for chi-square tests

For a 2 × 2 table, it’s always 1

The general formula for calculating the degrees of freedom for a chi-square test of independence is to multiply one less than the number of rows by one less than the number of columns.

$df=(r-1)(c-1)$

In a 2 × 2 table, there are two rows and two columns, so all chi-square tests conducted on these types of tables have 1 degree of freedom.

${\chi }^2=\frac{{(40\times 20-20\times 15)}^2\times 95}{60\times 35\times 55\times 40}$

${\chi }^2=5.1406$

Small sample sizes

$\frac{(r\times m)}{N}=\frac{(55\times 60)}{95}=34.74$

$\frac{(s\times m)}{N}=\frac{(40\times 60)}{95}=25.26$

$\frac{(r\times n)}{N}=\frac{(55\times 35)}{95}=20.26$

$\frac{(s\times n)}{N}=\frac{(40\times 35)}{95}=14.74$

${\chi }^2=\frac{{(ad-bc)}^{2}N}{mnrs}$

${\chi }^2=\frac{{(11 \times 5-1 \times 5)}^2 \times 22}{12 \times 10 \times 16 \times 6}$

${\chi }^2=4.7743$

$\frac{(r\times m)}{N}=\frac{(16\times 12)}{22}=8.73$

$\frac{(s\times m)}{N}=\frac{(6\times 12)}{22}=3.27$

$\frac{(r\times n)}{N}=\frac{(16\times 10)}{22}=7.27$

$\frac{(s\times n)}{N}=\frac{(6\times 10)}{22}=2.73$

Two-proportion test

$z=\frac{({\widehat{p}}_1-{\widehat{p}}_2)}{\sqrt{PQ\times \left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}$

Fisher exact test

$p=\frac{m!n!r!s!}{a!b!c!d!N!}$

Yates correction

${\chi }_{\text{yates}}^2=\frac{{\left(\left|ad-bc\right|-\frac{N}{2}\right)}^{2}N}{mnrs}$

${\chi }_{\text{yates}}^2=\frac{{\left(\left|11\times 5-1\times 5\right|-\frac{22}{2}\right)}^2\times 22}{12\times 10\times 16\times 6}$

${\chi }_{\text{yates}}^2=2.905$

N−1 Chi-square test

${\chi }^2=\frac{{(ad-bc)}^2(N-1)}{mnrs}$

${\chi }^2=\frac{{(11\times 5-1\times 5)}^2\times 21}{12\times 10\times 16\times 6}$

${\chi }^2=4.557$

N−1 Two-proportion test

The N−1 two-proportion test uses the standard large sample two-proportion formula (as shown in the previous section) except that it is adjusted by a factor of $\sqrt{\frac{N-1}{N}}$. This adjustment is algebraically equivalent to the N−1 chi-square adjustment. The resulting formula is

$z=\frac{({\hat{p}}_1-{\hat{p}}_2)\sqrt{\frac{N-1}{N}}}{\sqrt{PQ\times \left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}$

${\widehat{p}}_1 \text{and} {\widehat{p}}_2$ are the sample proportions

$P=\left(\frac{x_1+x_2}{n_1+n_2}\right)$, where x₁ and x₂ are the numbers completing or converting, and n₁ and n₂ are the numbers attempting

$P=\left(\frac{11+5}{12+10}\right)=0.727\text{and}Q=1-0.727=0.273$

$z=\frac{(0.917-0.50)\sqrt{\frac{22-1}{22}}}{\sqrt{0.727\times 0.273\times \left(\frac{1}{12}+\frac{1}{10}\right)}}$

$z=2.135$

Confidence interval for the difference between proportions

${\widehat{p}}_{\text{adj}}=\frac{x+\frac{z^2}{4}}{n+\frac{z^2}{2}}=\frac{x+\frac{{1.96}^2}{4}}{n+\frac{{1.96}^2}{2}}=\frac{x+.9604}{n+1.92}\approx \frac{x+1}{n+2}$

$({\widehat{p}}_{\text{adj}1}-{\widehat{p}}_{\text{adj}2})\pm {\text{z}}_{\alpha }\sqrt{\frac{{\widehat{p}}_{\text{adj1}}(1-{\widehat{p}}_{\text{adj}1})}{n_{\text{adj}1}}+\frac{{\widehat{p}}_{\text{adj}2}(1-{\widehat{p}}_{\text{adj}2})}{n_{\text{adj}2}}}$

${\widehat{p}}_{\text{adj}1}=\frac{x+\frac{z^2}{4}}{n+\frac{z^2}{2}}=\frac{11+\frac{{1.96}^2}{4}}{12+\frac{{1.96}^2}{2}}=\frac{11+0.9604}{12+1.92}=\frac{11.96}{13.92}=0.859$

${\widehat{p}}_{\text{adj}2}=\frac{x+\frac{z^2}{4}}{n+\frac{z^2}{2}}=\frac{5+\frac{{1.96}^2}{4}}{10+\frac{{1.96}^2}{2}}=\frac{5+0.9604}{10+1.92}=\frac{5.96}{11.92}=0.50$

$(0.859-0.50)\pm 1.96\sqrt{\frac{0.859(1-0.859)}{13.92}+\frac{0.50(1-0.50)}{11.92}}$

$0.359\pm 0.338$

Example 1: Comparing two completion rates

A new version of a CRM software application was created to improve the process of adding contacts to a distribution list. Four out of nine users (44.4%) completed the task on the old version and 11 out of 12 (91.7%) completed it on the new version. Is there enough evidence to conclude the new design improves completion rates? We will use the N−1 two-proportion test.

$z=\frac{({\hat{p}}_1-{\hat{p}}_2)\sqrt{\frac{N-1}{N}}}{\sqrt{PQ\times \left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}$

$P=\left(\frac{x_1+x_2}{n_1+n_2}\right)$

Filling in the values we get

$P=\left(\frac{4+11}{9+12}\right)=0.714\text{and}Q=1-0.714=0.286$

$z=\frac{(0.917-0.444)\sqrt{\frac{21-1}{21}}}{\sqrt{0.714\times 0.286\times \left(\frac{1}{9}+\frac{1}{12}\right)}}$

$z=2.313$

We can use a normal (z) table to look up the two-sided p-value or the Excel function NORMSDIST for the test statistic of 2.313. To use NORMSDIST, you need to copy the formula =(1-NORMSDIST(2.313))*2, which generates a p-value of 0.0207. Because this value is low, we have reasonable evidence to conclude the completion rate on the new CRM design has improved. To estimate the actual improvement in the completion rate for the entire user population, we now generate a 95% confidence interval around the difference in proportions using the adjusted-Wald procedure.

${\widehat{p}}_{\text{adj}1}=\frac{x+\frac{z^2}{4}}{n+\frac{z^2}{2}}=\frac{4+\frac{{1.96}^2}{4}}{9+\frac{{1.96}^2}{2}}=\frac{4+0.96}{9+1.92}=\frac{4.96}{10.92}=0.454$

${\widehat{p}}_{\text{adj}2}=\frac{x+\frac{z^2}{4}}{n+\frac{z^2}{2}}=\frac{11+\frac{{1.96}^2}{4}}{12+\frac{{1.96}^2}{2}}=\frac{11+0.96}{12+1.92}=\frac{11.96}{13.92}=0.859$

The critical value of z for a 95% confidence level is 1.96.

$(0.859-0.454)\pm 1.96\sqrt{\frac{0.859(1-0.859)}{13.92}+\frac{0.454(1-0.454)}{10.92}}$

$0.405\pm 0.347$

The 95% confidence interval is 0.058 to 0.752, that is, we can be 95% confident the actual improvement in completion rates on the new task design is between 6% and 75%.

Example 2: A/B testing

An A/B test was conducted live on an e-commerce website for two weeks to determine which product page converted more users to purchase a product. Concept A was presented to 455 users and 37 (8.13%) purchased the product. Concept B was presented to 438 users and 22 (5.02%) purchased the product. Is there evidence that one concept is statistically better than the other? Using the N−1 two-proportion test we get

$P=\left(\frac{37+22}{455+438}\right)=0.066\text{and}Q=1-0.066=0.934$

$z=\frac{(0.0813-0.0502)\sqrt{\frac{893-1}{893}}}{\sqrt{0.066\times 0.934\times \left(\frac{1}{455}+\frac{1}{438}\right)}}$

$Z=1.87$

Looking up the test statistic 1.87 in a normal table, we get a two-sided p-value of 0.06. The probability the two concepts have the same conversion rate is around 6%. That is, there is about a 94% probability the completion rates are different. The 90% confidence interval around the difference in conversion rates (which uses the critical value of 1.64) is

${\widehat{p}}_{\text{adj}1}=\frac{x+\frac{z^2}{4}}{n+\frac{z^2}{2}}=\frac{37+\frac{{1.64}^2}{4}}{455+\frac{{1.64}^2}{2}}=\frac{37+0.68}{455+1.35}=\frac{37.68}{456.35}=0.083$

${\widehat{p}}_{\text{adj}1}=\frac{x+\frac{z^2}{4}}{n+\frac{z^2}{2}}=\frac{22+\frac{{1.64}^2}{4}}{438+\frac{{1.64}^2}{2}}=\frac{22+0.68}{438+1.35}=\frac{22.68}{439.35}=0.052$

$(0.083-0.052)\pm 1.64\sqrt{\frac{0.083(1-0.083)}{466.35}+\frac{0.052(1-0.052)}{439.35}}$

$0.031\pm 0.027$

The 90% confidence interval around the observed difference of 0.031 ranges from 0.004 to 0.058. That is, if Concept A was used on all users (assuming the two week period was representative) we could expect it to convert between 0.4% and 6% more users than Concept B. As with any confidence interval, the actual long-term conversion rate is more likely to be closer to the middle value of 3.1% than to either of the extreme end points. For many large-volume e-commerce websites, however, even the small lower limit estimated advantage of 0.4% for Concept A could translate into a lot more revenue.

McNemar exact test

$p(x)=\frac{n!}{x!(n-x)!}{p}^x{(1-p)}^{(n-x)}$

Note: The term n! is pronounced “n factorial” and is $n\times (n-\text{1})\times (n-\text{2})\times \cdots \times \text{2}\times 1$.

Discordant pairs

• Six users completed on Design A but failed on Design B (cell b)

• One user failed on Design A and passed on Design B (cell c)

Table 5.12 shows the discordant users along with a sign (positive or negative) to indicate whether they performed better (plus sign) or worse (negative sign) on Design B. By the way, this is where this procedure gets its name the “sign test”—we’re testing whether the proportion of pluses to minuses is significantly different from 0.50.

Table 5.12

Discordant Performance from Example 1

User	Relative Performance on B
1	−
4	−
5	−
8	+
9	−
13	−
15	−

In total, there were seven discordant pairs (cell b + cell c). Most users who performed differently performed better on Design A (six of seven). We will use the smaller of the discordant cells to simplify the computation, which is the one person in cell c who failed on Design A and passed on Design B. (Note that you will get the same result if you used the larger of the discordant cells, but it would be more work.) Plugging these values in the formula, we get

$p(0)=\frac{7!}{0!(7-0)!}{0.5}^0{(1-0.5)}^{(7-0)}=0.0078$

$p(1)=\frac{7!}{1!(7-1)!}{0.5}^1{(1-0.5)}^{(7-1)}=0.0547$

The one-tailed exact-p value is these two probabilities added together, 0.0078 + 0.0547 = 0.0625, so the two-tailed probability is double this (0.125). The mid-probability is equal to half the exact probability for the value observed plus the cumulative probability of all values less than the one observed. In this case, the probability of all values less than the one observed is just the probability of 0 discordant pairs, which is 0.0078:

$\text{Mid}p=\frac{1}{2}0.0547+0.0078$

$\text{Mid}p=0.0352$

The one-tailed mid-p value is 0.0352, so the two-tailed mid-p value is double this (0.0704). Thus, the probability of seeing one out of seven users perform better on Design A than B if there really was no difference is 0.0704. Put another way, we can be about 93% sure Design A has a better completion rate than Design B.

The computations for this two-sided mid-p value are rather tedious to do by hand, but are fairly easy to get using the Excel function =2*(BINOMDIST(0,7,0.5,FALSE) + 0.5*BINOMDIST(1,7,0.5,FALSE)).

If you need to guarantee that the reported p-value is greater than or equal to the actual long-term probability, then you should use the exact-p values instead of the mid-p values. This is similar to the recommendation we gave when comparing the completion rate to a benchmark and when computing binomial confidence intervals (see Chapter 4). For most applications in user research, the mid-p value will work better (lead to more correct decisions) over the long run (Agresti and Coull, 1998).

Alternate approaches

Chi-square statistic

The most common recommendation in statistics text books for large sample within-subject comparisons is to use the chi-square statistic. It is typically called the McNemar chi-square test (McNemar, 1969), as opposed to the McNemar exact test which we presented in the earlier section. It uses the following formula:

${\chi }^2=\frac{{(c-b)}^2}{c+b}$

You will notice that the formula only uses the discordant cells (b and c). You can look up the test statistic in a chi-square table with 1 degree of freedom to generate the p-value, or use the Excel CHIDIST function. Using the data from Example 1 with seven discordant pairs we get a test statistic of

${\chi }^2=\frac{{(1-6)}^2}{7}=3.571$

Using the Excel function =CHIDIST(3.571, 1), we get the p-value of 0.0587, which, for this example, is reasonably close to our mid-p value of 0.0704.

However, to use this approach, the sample size needs to be reasonably large to have accurate results. As a general guide, it is a large enough sample if the number of discordant pairs (b + c) is greater than 30 (Agresti and Franklin, 2007).

You can equivalently use the z-statistic and corresponding normal table of values to generate a p-value instead of the chi-square statistic, by simply taking the square root of the entire equation.

$Z=\frac{c-b}{\sqrt{c+b}}$

$Z=\frac{6-1}{\sqrt{6+1}}=\frac{5}{\sqrt{7}}=1.89$

Using the Excel NORMSDIST function (=2*NORMSDIST(1.89)), we get p = 0.0587, demonstrating the mathematical equivalence of the methods.

Yates correction to the chi-square statistic

To further complicate matters, some texts recommend using a Yates corrected chi-square for all sample sizes (Bland, 2000). As shown in the following, the Yates correction is

${\chi }^2=\frac{{(|c-b|-1)}^2}{b+c}$

Using the data from Example 1 with seven discordant pairs we get

${\chi }^2=\frac{{(|1-6|-1)}^2}{7}=2.29$

We look up this value in a chi-square table of values with 1 degree of freedom or use the Excel function =CHIDIST(2.29, 1) to get the p-value of 0.1306. For this example, this value is even higher than the exact-p value from the sign test, which we expect to overstate the magnitude of p. A major criticism of the Yates correction is that it will likely exceed the p-value from the sign test. Recall that this overcorrection also occurs with the Yates correction of the between-subjects chi-square test. For this reason, we do not recommend the use of the Yates correction.

Table 5.13 provides a summary of the p-values generated from the different approaches and our recommendations.

Table 5.13

Summary of p-values Generated from Sample Data for McNemar Tests

Method	P-value	Notes
McNemar exact test using mid-probabilities	0.0704	Recommended: For all sample sizes will provide best average long-term probability, but some individual tests may understate actual probability
McNemar exact test using exact probabilities	0.125	Recommended: For all sample sizes when you need to guarantee the long-term probability is greater than or equal to the p-value (a conservative approach)
McNemar chi-square test/z test	0.0587	Not Recommended: Understates true probability for sample sizes and is unclear about what constitutes a large sample size
McNemar chi-square test with Yates correction	0.1306	Not Recommended: Overstates true probability for all sample sizes

Confidence interval around the difference for matched pairs

Using the same notation from the 2 × 2 table with the “adj” meaning to add $\frac{z_{\alpha }^2}{8}$ to each value, we have the formula:

$({\widehat{p}}_{2\text{adj}}-{\widehat{p}}_{1\text{adj}})\pm {\text{z}}_{\alpha }\sqrt{\frac{({\widehat{p}}_{12\text{adj}}+{\widehat{p}}_{21\text{adj}})-{({\widehat{p}}_{21\text{adj}}-{\widehat{p}}_{12\text{adj}})}^2}{N_{\text{adj}}}}$

${\widehat{p}}_{1\text{adj}}=\frac{m_{\text{adj}}}{N_{\text{adj}}}$

${\widehat{p}}_{2\text{adj}}=\frac{r_{\text{adj}}}{N_{\text{adj}}}$

${\widehat{p}}_{12\text{adj}}=\frac{b_{\text{adj}}}{N_{\text{adj}}}$

${\widehat{p}}_{21\text{adj}}=\frac{c_{\text{adj}}}{N_{\text{adj}}}$

$\frac{z_{\alpha }^2}{8}$ = The adjustment added to each cell (e.g., for a 95% confidence level this is $\frac{{1.96}^2}{8}=0.48$)

${\widehat{p}}_{1\text{adj}}=\frac{14}{17}=0.825$

${\widehat{p}}_{2\text{adj}}=\frac{9}{17}=0.529$

${\widehat{p}}_{12\text{adj}}=\frac{6.5}{17}=0.383$

${\widehat{p}}_{21\text{adj}}=\frac{1.5}{17}=0.087$

$({\widehat{p}}_{2\text{adj}}-{\widehat{p}}_{1\text{adj}})\pm z_{\alpha }\sqrt{\frac{({\widehat{p}}_{12\text{adj}}+{\widehat{p}}_{21\text{adj}})-{({\widehat{p}}_{21\text{adj}}-{\widehat{p}}_{12\text{adj}})}^2}{N_{\text{adj}}}}$

$(0.529-0.825)\pm 1.96\sqrt{\frac{(0.383+0.087)-{(0.087-0.383)}^2}{17}}$

$-0.296\pm 0.295$

Example 2: Completion rates

In a comparative usability test, 14 users attempted to rent the same type of car in the same city on two different websites (Avis.com and Enterprise.com). All 14 users completed the task on Avis.com but only 10 of 14 completed it on Enterprise.com. The users and their task results appear in Table 5.17 and Table 5.18. Is there sufficient evidence that more users could complete the task on Avis.com than on Enterprise.com (as designed at the time of this study)?

Table 5.17

Completion Data from CUE-8 Task

User	Avis.com	Enterprise.com
1	1	1
2	1	1
3	1	0
4	1	0
5	1	1
6	1	1
7	1	1
8	1	0
9	1	1
10	1	1
11	1	1
12	1	0
13	1	1
14	1	1
Comp rate	100%	71%

Table 5.18

Organization of Concordant and Discordant Pairs from CUE-8 Task

	Enterprise.com Pass	Enterprise.com Fail Pass	Total
Avis.com Pass	10 (a)	4 (b)	14 (m)
Avis.com Fail	0 (c)	0 (d)	0 (n)
Total	10 (r)	4 (s)	14 (N)

In total there were four discordant users (cell b + cell c), all of which performed better on Avis.com. Table 5.19 shows the improvement performance difference for the four users on Enterprise.com.

Table 5.19

Discordant Performance from CUE-8 Task

User	Relative Performance on Enterprise.com
3	—
4	—
8	—
12	—

Plugging the appropriate values in the formula we get

$p(x)=\frac{n!}{x!(n-x)!}{p}^x{(1-p)}^{(n-x)}$

$p(0)=\frac{4!}{0!(4-0)!}{0.5}^0{(1-0.5)}^{(4-0)}=0.0625$

The one-tailed exact-p value is 0.0625, so the two-tailed probability is double this (0.125). The mid-probability is equal to half the exact probability for the value observed plus the cumulative probability of all values less than the one observed. Because there are no values less than 0, the one-tailed mid-probability is equal to half of 0.0625:

$\text{Mid}\hspace{0.17em}\text{-}p=\frac{1}{2}(0.0625)$

$\text{Mid-}\hspace{0.17em}p=0.0313$

The one-tailed mid-p value is 0.0313, so the two-tailed mid-p value is double this (0.0625). Thus, the probability of seeing zero out of four users perform worse on Enterprise.com if there really was no difference is 0.0625. Put another way, we can be around 94% sure Avis.com had a better completion rate than Enterprise.com on this rental car task at the time of this study.

The 95% confidence interval around the difference is found by first adjusting the values in each interior cell of the 2 × 2 table by 0.5 $\left(\frac{{1.96}^2}{8}=0.48\approx 0.5\right)$, as shown in Table 5.20.

$({\widehat{p}}_{2\text{adj}}-{\widehat{p}}_{1\text{adj}})\pm z_{\alpha }\sqrt{\frac{({\widehat{p}}_{12\text{adj}}+{\widehat{p}}_{21\text{adj}})-{({\widehat{p}}_{21\text{adj}}-{\widehat{p}}_{12\text{adj}})}^2}{N_{\text{adj}}}}$

${\widehat{p}}_{1\text{adj}}=\frac{m_{\text{adj}}}{N_{\text{adj}}}=\frac{15}{16}=0.938$

${\widehat{p}}_{2\text{adj}}=\frac{r_{\text{adj}}}{N_{\text{adj}}}=\frac{11}{16}=0.688$

${\widehat{p}}_{12\text{adj}}=\frac{b_{\text{adj}}}{N_{\text{adj}}}=\frac{4.5}{16}=0.281$

${\widehat{p}}_{21\text{adj}}=\frac{c_{\text{adj}}}{N_{\text{adj}}}=\frac{0.5}{16}=0.03$

$(0.938-0.688)\pm 1.96\sqrt{\frac{(0.281+0.03)-{(0.03-0.281)}^2}{16}}$

$0.250\pm 0.245$