3.2.1 Tests on an Expectation

We know simple and composite hypotheses.

In a simple hypothesis the value of θ = μ is fixed at a real number μ₀.

A simple null hypothesis could be H₀ : μ = μ₀. In the pair H₀ : μ = μ₀; H_A : μ = μ₁ both hypotheses are simple.

Examples for composite null hypotheses are:

1. H₀ : μ = μ₀, σ² arbitrary
2. H₀ : μ = μ₀  or  μ = μ₁
3. H₀ : μ < μ₀,
4. H₀ : μ ≠ μ₀.

3.2.1.1 Testing the Hypothesis on the Expectation of a Normal Distribution with Known Variance

We first assume that any component of Y = (y₁, y₂, … , y_n)^T is independently distributed as y, namely normally with unknown expectation μ and known variance σ².

We like to test the null hypothesis

1. H₀ : μ = μ₀ against
   1. (a) H_A : μ = μ₁ > μ₀ (one‐sided alternative)
   2. (b) H_A : μ = μ₁ < μ₀ (one‐sided alternative)
   3. (c) H_A : μ = μ₁ ≠ μ₀ (two‐sided alternative)

with a significance level α.

All hypotheses are simple hypotheses. In this case, usually the test statistic

3.1

is applied where is the mean taken from the random sample Y of size n.

Starting with the (random) sample mean first the value μ₀ of the null hypothesis is subtracted and then this difference is divided by the standard error of .

We know that is normally distributed with variance 1 and, under the null hypothesis, the expectation of is 0. Under the alternative hypothesis the expectation of is . The value

3.2

is called the non‐centrality parameter (ncp) or, in applications, it is also called the relative effect size and δ = μ₁ − μ₀ is called the effect size.

We calculate from a realised sample, i.e. from observations Y = (y₁, y₂, … , y_n)^T, the sample mean following Problem 2.1 and from this the realised test statistic

3.3

Now we reject the null hypothesis for the alternative hypotheses (a), (b), and (c) with z from (3.3) and a fixed first kind risk or significance level α if:

1. (a) z > Z(1 − α)
2. (b) z < Z(α)
3. (c) or .

Above Z(P), 0 < P < 1, is the P‐quantile of z, which has a standard normal distribution N(0,1).

3.2.1.2 Testing the Hypothesis on the Expectation of a Normal Distribution with Unknown Variance

We now assume that the components of a random sample Y = (y₁, y₂, … , y_n)^T are distributed normally with unknown expectation μ and unknown variance σ².

We like to test the null hypothesis:

1. H₀ : μ = μ₀, σ² arbitrary against
   1. (a) H_A : μ = μ₁ > μ₀, σ² arbitrary (one‐sided alternative)
   2. (b) H_A : μ = μ₁ < μ₀, σ² arbitrary (one‐sided alternative)
   3. (c) H_A : μ = μ₁ ≠ μ₀, σ² arbitrary (two‐sided alternative).

All hypotheses are composite hypotheses because σ² is arbitrary. In this case, usually the test statistic (where s is the sample standard deviation):

3.6

is used which is non‐centrally t‐distributed with n − 1 degrees of freedom and ncp

Under the null hypothesis, the distribution is central t with n − 1 degrees of freedom.

If the type I error probability is α, H₀ will be rejected if:

1. in case (a), t > t(n − 1;1 − α),
2. in case (b), t < −t(n − 1;1 − α),
3. in case (c),|t| > t(n − 1;1 − α/2).

Our precision requirement is given by α and the risk of the second kind β if μ − μ₀ = δ.

From this we get

where t(n − 1;λ;β) is the β‐quantile of the non‐central t‐distribution with n − 1 degrees of freedom and ncp

For example, assuming a power of 0.9 the relative effect can be read on the abscissa; it is approximately 1.5 for n = 7.

From the precision requirement above the minimum sample size is iteratively determined as the integer solution of

3.7

(Rasch et al. 2011b) with P = 1 − α in the one‐sided case and P = 1 − α/2 in the two‐sided case. In Problem 3.3 it is shown what to do when σ is unknown.

The R‐implementation reads as follows:

 > size=function(p,rdelta,beta)
  {f=function(n,p,rdelta,beta)
  {qt(p,n-1,0)-qt(beta,n-1,rdelta*sqrt(n))}
  k=uniroot(f,c(2,1000),p=p,rdelta=rdelta,beta=beta)$root
  k0=ceiling(k)
  print(paste("optimum sample number: n =",k0), quote=F)}

In the command rdelta is equal to and p equals 1 − α. As an example we calculate

 > size(p=0.9,rdelta=0.5,beta=0.1)
[1] optimum sample number: n = 28.

The requirement (3.7) is illustrated in Figure 3.2 for the one‐sided case.

3.2.2 Test on the Median

Sometimes the population median m should be preferred to the expectation of a random variable y. For any probability distribution of y on the real line R¹with distribution function F(y), regardless of whether it is a continuous distribution or a discrete one, a median is by definition any real number m that satisfies the inequalities

If m is unique, the population median is the point where the distribution function equals 0.5.

It is better to use the median instead of the expectation if we expect serious outliers or if the distribution has an extremely skewness, so that in a sample the median better characterises the data than the mean. Tests on the population median are so‐called non‐parametric tests or distribution‐free tests. These are tests without a specific assumption about the distribution.

It is notable that such an approach is not intrinsically justified by a suspected deviation from the normal distribution of the character in question. Particularly as Rasch and Guiard (2004) showed that all tests based on the t‐distribution are extraordinarily robust against violations from the normal distribution. That is, even if the normal random variable is not a good model for the given character, the type I risk actually holds almost accurately. Therefore, if a test statistic results, on some occasions, in significance, we can be quite sure that an erroneous decision concerning the rejection of the null hypothesis has no larger risk than in the case of a normal distribution. We describe Wilcoxon's signed‐ranks test useful for continuous distributions.

We test the hypothesis

1. H₀: m = m₀ against a one‐ or two‐sided alternative:
   • H_A: m < m₀
   • H_A: m > m₀
   • H_A: m ≠ m₀.

First, we calculate from each observation y_i of a sample Y=(y₁, y₂, … , y_n)^T the differences d_i = y_i − med(Y) between the observations and the median =med(Y) . If the value of d_i is equal to zero, we do not use these values. From the remaining data subset of size n* ≤ n the rank of the absolute values |d_i| is calculated. The sum,

3.9

with is the test statistic. Wilcoxon's signed‐ranks test is based on the distribution of the corresponding random variable, V.

Planning a study is generally difficult for non‐parametric tests, because it is hard to formulate a non‐centrality parameter (however see, for instance, Munzel and Brunner 2002).

3.2.3 Test on the Variance of a Normal Distribution

Equation (2.21) shows an unbiased estimator s² of the variance σ² of a normal distribution.

We would like to test the null hypothesis

• against one of the alternatives below:
  • 
  • 
  • 

with a first kind risk α.

We know that

3.10

is centrally χ² distributed with n − 1 degrees of freedom.

We calculate from a sample of size n the realisation of (3.10) and compare the result with the P‐quantile CS(P, n − 1) of the central χ² distribution as follows:

3.11

3.2.4 Test on a Probability

We already discussed the analysis problem in Problem 3.5. Here we show how to determine the sample size approximately.

Let p be the probability of some event. To test one of the pairs of hypotheses below:

1. (a) H₀ : p ≤ p₀, H_A : p > p₀
2. (b) H₀ : p ≥ p₀, H_A : p < p₀
3. (c) H₀ : p = p₀, H_A : p ≠ p₀

with a risk of the first kind α, and if we want the risk of the second kind to be not larger than β as long as

1. (a) p₁ > p_0, and p₁ − p₀ ≥ δ
2. (b) p₁ < p₀ and p₀ − p₁ ≥ δ
3. (c) p₀ ≠ p₁ and |p₀ − p₁| ≥ δ.

Determine the minimum sample size from

3.12

The same size is needed for the other one‐sided test. In the two‐sided case:

we replace α by α/2 and calculate (3.12) twice for p₁ = p₀ − δ > 0 and for p₁ = p₀ + δ < 1 if δ is the difference from p₀, which should not be overlooked with a probability of at least 1 − β. From the two n‐values, we then take the maximum.

3.2.5 Paired Comparisons

If we measure two traits x and y from some individuals, we could say that we have a two‐sample problem. However, from a mathematical point of view we can say that the two measurements are realisations of a two‐dimensional random variable and from the corresponding universe one sample is drawn with measurements . We then speak about paired observations or statistical twins. In place of discussing the question of whether x_i and y_i have the same distribution we can ask whether the expectation of d_i = x_i − y_i is zero. The problem is reduced to that considered in Section 3.2.1 and must not be discussed again.

If the random variable d is extremely non‐normal (skew for instance) with the median m, then based on n realisations d_i, i = 1, … , n of the d_i null hypothesis, for the test below all d_i = 0 are excluded from the sample; the reduced sample size is denoted by n^*.

H₀ : m = m₀ against H_A : m ≠ m₀ can be tested with Wilcoxon's signed rank test. For this the new differences are calculated as well as their absolute values after deleting the values 0. The remaining data set has a size n* ≤ n. Finally, these values are to be ordered (ascending) and ranks R_i are to be assigned accordingly.

Ties (identical values) receive a rank equal to the average of the ranks they span. The sum of the so‐called signed ranks is the test statistic of Wilcoxon's signed rank test. The corresponding random variable V has expectation E(V) = n*(n* + 1)/4 and variance . The null hypothesis H₀ : m = m₀ is rejected if V > V(n^*, 1 − α), where α is the risk of the first kind. The critical values V(n*, 1 − α) are calculated by the R program in the following example.

3.2.6 Sequential Tests

The technique of sequential testing offers the advantage that, given many studies, on average fewer research units need to be sampled as compared to the ‘classic’ approach of hypothesis testing in Section 3.2.1 with sample size fixed beforehand.

Nevertheless, we need also the precision requirements α, β, and δ, moreover a sequential test cannot be applied without this, which means that we cannot start observations without fixing β. Until now a sample of fixed size n was given; Stein (1945) proposed a method of realising a two‐stage experiment. In the first stage a sample of size n₀ > 1 is drawn to estimate σ² using the variance of this sample and to calculate the sample size n of the method using (3.7). In the second stage n−n_o further measurements are taken. Following the original method of Stein in the second stage at least one further measurement is necessary from a theoretical point of view. In this section we simplify this method by introducing the condition that no further measurements are to be taken for n − n_o ≤0. Nevertheless, this yields an α‐test of acceptable power.

Since both parts of the experiment are carried out one after the other, such experiments are called sequential. Sometimes it is even tenable to make all measurements step by step, where each measurement is followed by calculating a new test statistic. A sequential testing of this kind can be used if the observations of a random variable in an experiment take place successively in time. Typical examples are series of single experiments in a laboratory, psychological diagnostics in single sessions, medical treatments of patients in hospitals, consultations of clients of certain institutions, and certain procedures of statistical quality control, where the sequential approach was used the first time (Dodge and Romig 1929). The basic idea is to utilise the observations already made before the next are at hand.

For example, testing the hypothesis H₀: μ = μ₀ against H_A: μ > μ₀ (where the sample size can be determined by (3.7)) there are three possibilities in each step of evaluation, namely:

• accept H₀
• reject H₀
• continue the investigation.

The, up to now unsurpassed, textbook of Wald (1947) has since been reprinted and is therefore generally available (Wald 2004), and new results can be found in the books of Ghosh and Sen (1991) and DeGroot (2005). We do not recommend the application of this general theory, but we recommend closed plans, which end after a finite number of steps with certainty (and not only with probability 1). We first give a general definition useful in other sections independent of our one‐sample problem.

3.3.1 Tests on Two Expectations

We assume that the character of interest is in each population modelled by a normally distributed random variable. That is to say, independent random samples of size n₁ and n₂, respectively, are drawn from populations 1 and 2. The observations of the random variables on the one hand and on the other hand will be and . We call the underlying parameters μ₁ and μ₂, and and , respectively. The unbiased estimators are then used for the expectations and , respectively, and for the variances and , respectively (according to Section 3.2).

We test the null hypothesis

1. H₀: μ₁ = μ₂ = μ; against one of the alternative hypotheses
   1. (a) H_A: μ₁ < μ₂
   2. (b) H_A: μ₁ > μ₂
   3. (c) H_A: μ₁ ≠ μ₂.

According to recent findings from simulation studies (Rasch et al. 2011a), the traditional approach is usually unsatisfactory because it is not theoretically regulated for all possibilities.

Nevertheless, we start with an introduction to the traditional approach because the current literature on applications of statistics refers to it, and methods for more complex issues are built on it. Further, there are situations in which it is certain that both populations have equal variances, especially if standardisation takes place, as in intelligence tests.

3.3.1.1 The Two‐Sample t‐Test

The traditional approach bases upon the test statistic in Formula (3.15). It examines the given null hypothesis for the case that the variances and , respectively, of the relevant variables in the two populations, are not known; this is the usual case. However, it assumes further that these variances are equal in both populations. The test statistic

3.15

is (central) t‐distributed with n₁ + n₂ − 2 degrees of freedom. This means, that we can decide in favour of, or against, the null hypothesis, analogous to Section 3.2, by using the (1 − α)‐quantile of the (central) t‐distribution t(n₁ + n₂ − 2, 1 − α) and t(n₁ + n₂ − 2, 1 − ), respectively, depending on whether it is a one‐ or two‐sided question.

H₀: μ₁ = μ₂ = μ; will be rejected if:

1. (a) t < t(n₁ + n₂ − 2, α)
2. (b) t > t(n₁ + n₂ − 2, 1 − α)
3. (c)

If the null hypothesis is rejected, we say that the two expectations differ significantly.

Thus, the minimum sample size for the one-sided case is 106, for the two-sided case we would need at least 121 observations each.

3.3.1.2 The Welch Test

We now discuss the case that either it is known that the variances and of the two populations are unequal or that we are not sure that they are equal – this is often the case. Therefore, we recommend always using the proposed Welch‐test below in place of the two‐sample t‐test. The recommendation to use the Welch test instead of the t‐test cannot be found in most textbooks of applied statistics and may therefore surprise experienced users. This recommendation is instead based on the results of current research from Rasch et al. (2011a). There it is also explained that pre‐testing first for equality of variances and if the equality is accepted to continue with the two‐sample t‐test makes no sense. We show in Table 3.5 a part of table 2 from Rasch et al. (2011a).

δ = 0
With pre‐tests			Without pre‐tests
√(σ21/σ22)	n1	n2	t	Welch (Levene)	W (KS)	t	Welch	W
1	10	10	4.95	4.55 (4.84)	0.00 (0.02)	4.96	4.82	5.21
30	30	4.97	4.82 (4.98)	11.11 (0.09)	4.96	4.95	4.92
10	30	5.00	4.87 (4.31)	0.00 (0.06)	5.01	5.05	4.87
30	10	4.97	5.09 (4.32)	0.00 (0.06)	4.96	5.15	5.00
30	100	4.86	5.00 (4.80)	9.09 (0.11)	4.84	4.91	4.82
2	10	10	6.08	3.33 (32.99)	0.00 (0.03)	5.38	4.93	6.06
30	30	7.37	4.78 (91.89)	10.00 (0.10)	5.15	4.98	5.88
10	30	1.02	5.90 (30.18)	0.00 (0.07)	0.90	5.00	2.19
30	10	19.38	2.98 (73.73)	16.67 (0.06)	15.51	5.19	10.09
30	100	1.33	4.99 (98.38)	0.00 (0.11)	0.63	4.96	1.89

The entries give the percentage of rejecting the (final) hypothesis H₀: μ₁ = μ₂. t is for student's t‐test and W for the Wilcoxon–Mann–Whitney‐test both with and without pre‐testing. Levene means the Levene test and KS the Kolmogorov–Smirnov test on normality.

Pre‐tests of the assumptions are worthless, as shown by our simulation experiments. First testing the normal assumption per sample, testing then the null hypothesis for equality of variances, with the Levene test from Section 3.3.4, and finally, testing the null hypothesis for equality of means leads to terrible risks, i.e. the risk estimated using simulation is only in the case of the Welch test near the chosen value α = 0.05 – it is exceeded by more than 20% in only a few cases. By contrast, the t‐test rejects the null hypothesis largely. Moreover, the Wilcoxon W‐test (see Section 3.3.1.3) does not maintain the type‐I risk in any way; it is too large and it has to be carried out with a newly collected pair of samples.

The Welch test is based on the test statistic

The distribution of the test statistic was derived by Welch (1947) and is presented in theorem 3.18 in Mathematical Statistics (Rasch and Schott, 2018).

If the pair of hypotheses

is to be tested, often the approximate test statistic

is used. H₀ is rejected if |t*| is larger than the corresponding quantile of the central t‐distribution with approximate degrees of freedom f, the so‐called Satterthwaite's f,

degrees of freedom.

Referring to the t‐test, the desired relative effect size is, for equal sample sizes

For the Welch test, the desired relative effect size then reads:

Planning a study for the Welch test is carried out in a rather similar manner to that for two‐sample t‐test. A problem arises, however: in most cases it is not known whether the two variances in the relevant population are equal or not, and if not, to what extent they are unequal. If one knew this, then it would be possible to calculate the necessary sample size exactly. In the other case, one can use the largest appropriate size, which results from equal, realistically maximum expected variances.

3.3.1.3 The Wilcoxon Rank Sum Test

Wilcoxon (1945) proposed for equal sample sizes, and later Mann and Whitney (1947) extended for unequal sample sizes, a two‐sample distribution‐free test based on the ranks of the observations. This test is not based on the normal assumption, in its exact form only two continuous distributions with all moments existing are assumed; we call it the Wilcoxon test. As is seen from the title of Mann and Whitney (1947), this test is testing whether one of the underlying random variables is stochastically larger than the other. It can be used for testing the equality of medians under additional assumptions: if the continuous distributions are symmetric, the medians are equal to the expectations. The null hypothesis tested by the Wilcoxon test corresponds to the hypothesis of equality of the medians m₁ and m₂, H₀ : m₁ = m₂ = m if and only if all higher moments of the two populations exist and are equal. Otherwise a rejection of the Wilcoxon hypothesis says little about the rejection of H₀ : m₁ = m₂ = m. The test runs as follows based on the observations and of the random variables and respectively. We assume the sample size n₁ ≤ n₂.

Calculate:

3.16

and

3.17

We denote the P‐quantiles of W based on n₁ ≤ n₂ observations by W(n₁, n₂, P).

Reject H₀:

• in case (a), if W > W(n₁, n₂, 1 − α),
• in case (b), if W < W(n₁, n₂, α),
• in case (c), if either or ,

and accept it otherwise (the three cases are those given in problem 3.13).

The expectation of W if H₀ is valid is E(W) = n₁ (n₁ + n₂)/2.

The variance of W if H₀ is valid is var(W) = n₁ n₂(n₁ + n₂ + 1)/12.

Mann–Whitney used the test statistic U, which is the number of times that an x observation is larger than a y observation. The relation between U and W is: U = W − n₁(n₁ + 1)/2. Hence the expectation of U is E(U) = E(W) − n₁(n₁ + 1)/2 and the variance of U is var(U) = var(W − n₁(n₁ + 1)/2)

The expressions for var(W) and var(U) must be corrected if there are ties (equal observations) in the data.

If there are ties of size t_i then var(W) must be subtracted by n₁n₂[Σ(t_i³ − t_i)]/[12(N² − N)] and with ties we have var(W) = var(U) = {n₁ n₂ (N³ − N − Σ(t_i³ − t_i)}/[12(N² − N)].

The expectations of E(W) and E(U) remain the same in the case of ties.

Be aware that the quantiles of the random test statistic U are in R denoted by W and are given by the R‐command

 > qwilcox(P,n1,n2) 

as an example take

 > qwilcox(0.05,15,20)
[1] 101
>  qwilcox(0.95,15,20)
[1] 199 

3.3.2 Test on Two Medians

To test H₀ whether two populations have the same median, against the alternative hypothesis H_A that the medians are different, a random sample of size n_i (i = 1,2) is drawn from each population. Be aware that the scale of the continuous measurement is at least ordinal, or else the term median is without meaning. A 2 × 2 contingency table is constructed, with rows ‘>median’ and ‘≤median’ and the columns for the sample i = 1, 2 of the two populations. The two entries in the ith column are the numbers of observations in the ith sample, which are above and below the combined sample grand median ; this is the median of all observations combined.

The median test has three appealing properties from the practical standpoint. In the first place it is primarily sensitive to differences in location between cells and not to the shapes of the cell distributions. Thus, if the observations of some cells were symmetrically distributed while in other cells they were positively skewed, the rank test would be inclined to reject the null hypothesis even though all population medians were the same. The median test would not be much affected by such differences in the shapes of the cell distributions. In the second place the computations associated with the median test are quite simple and the test itself is nothing more than the familiar contingency table test. In the third place when we come to consider more complex experiments it will be found that the median tests are not much affected by differing cell sizes. Bradley (1968) gives the following rationale and example for the Westenberg–Mood median test (Westenberg 1948; Mood 1950, 1954).

3.3.3 Test on Two Probabilities

Let y be distributed as B(n, p). Knowing one observation y = Y we want to test H₀: p = p₀ against H_A: p ≠ p₀, p₀ ɛ (0, 1). The natural parameter is η =, and y is sufficient with respect to the family of binomial distributions. Therefore the uniformly most powerful unbiased (UMPU)‐α‐test is given by (3.27) in Rasch and Schott (2018), where the c_iα and γ′_iα (i = 1, 2) have to be determined from (3.28) and (3.29) in Rasch and Schott (2018). With

Equation (3.28) in Rasch and Schott (2018) has the form

3.20

Regarding

the relation (3.29) in Rasch and Schott (2018) leads to

3.21

The solution of these two simultaneous equations can be obtained by statistical software, e.g. by R. Further results can be found in the book of Fleiss et al. (2003).

3.3.4 Tests on Two Variances

We consider two independent random samples , where the components y_ij are supposed to be distributed as N(μ_i ). We intend to test the null hypothesis

against the alternative

Under H₀ we have = l, and the random variable

does not depend on μ₁  and μ₂, and . Since the random variable F is centrally distributed as F(n₁ − 1, n₂ − 1) under H₀, the function

defines a UMPU‐α‐test, where F(n₁ − 1, n₂ − 1| P) is the P‐quantile of the F‐distribution with n₁ − 1 and n₂ − 1 degrees of freedom. This test is very sensitive to deviations from the normal distribution. Therefore the following Levene test should be used instead of it in the applications. Therefore we present no example or problem.

Instead we propose the Levene test. Box (1953) has already mentioned the extreme non‐robustness of the F‐test comparing two variances. Rasch and Guiard (2004) have reported on extensive simulation experiments devoted to this problem. The results of Box show that non‐robustness has to be expected already for relatively small deviations from the normal distribution. Hence, we generally suggest applying the test of Levene (1960) which is described now.

For j = 1, 2 we put

and

where .

The null hypothesis H₀ is rejected if

3.22

In R the Levene test is not available in the base package, but it is quite easy to program it yourself or it can be taken from the package car v3.0.0 by John Fox, see https://cran.r‐project.org/web/packages/car/car.pdf.