4.1 INTRODUCTION

In many experiments an observation is expressible, not as a single numerical quantity, but as a family of several separate numerical quantities. Thus, for example, if a pair of distinguishable dice is tossed, the outcome is a pair (x, y), where x denotes the face value on the first die, and y, the face value on the second die. Similarly, to record the height and weight of every person in a certain community we need a pair (x, y), where the components represent, respectively, the height and weight of a particular individual. To be able to describe such experiments mathematically we must study the multidimensional random variables.

In Section 4.2 we introduce the basic notations involved and study joint, marginal, and conditional distributions. In Section 4.3 we examine independent random variables and investigate some consequences of independence. Section 4.4 deals with functions of several random variables and their induced distributions. Section 4.5 considers moments, covariance, and correlation, and in Section 4.6 we study conditional expectation. The last section deals with ordered observations.

4.2 MULTIPLE RANDOM VARIABLES

In this section we study multidimensional RVs. Let (Ω, , P) be a fixed but otherwise arbitrary probability space.

From now on we will restrict attention mainly to two-dimensional random variables. The discussion for the n-dimensional case is similar except when indicated. The development follows closely the one-dimensional case.

Then F satisfies both (i) and (ii) above. However, F is not a DF since

Let and . We have

for all pairs (x₁,y₁), (x₂,y₂) with (see Fig. 2).

The “if” part of the theorem has already been established. The “only if” part will not be proved here (see Tucker [114, p. 26]).

Theorem 2 can be generalized to the n-dimensional case in the following manner.

We restrict ourselves here to two-dimensional RVs of the discrete or the continuous type, which we now define.

Let (X, Y) be a two-dimensional RV with PMF

Then

(6)

and

(7)

Let us write

(8)

Then and , and , and {p_i·}, {p_j·} represent PMFs.

If (X, Y) is an RV of the continuous type with PDF f, then

(9)

and

(10)

satisfy , and , . It follows that f₁(x) and f₂(y) are PDFs.

In general, given a DF F(x₁, x₂, …, x_n) of an n-dimensional RV (X₁, X₂, …, X_n), one can obtain any k-dimensional marginal DF from it. Thus the marginal DF of (X_i1, X_i2, …, X_ik), where , is given by

We now consider the concept of conditional distributions. Let (X, Y) be an RV of the discrete type with PMF . The marginal PMFs are Recall that, if and , the conditional probability of A, given B, is defined by

Take and , and assume that Then and

For fixed j, the function and Thus , for fixed j, defines a PMF.

Next suppose that (X, Y) is an RV of the continuous type with joint PDF f. Since for any x,y, the probability or is not defined. Let , and suppose that . For every x and every interval , consider the conditional probability of the event , given that . We have

For any fixed interval , the above expression defines the conditional DF of X, given that , provided that . We shall be interested in the case where the limit

exists.

Suppose that (X, Y) is an RV of the continuous type with PDF f. At every point (x, y) where f is continuous and the marginal PDF and is continuous, we have

Dividing numerator and denominator by 2ε and passing to the limit as , we have

It follows that there exists a conditional PDF of X, given , that is expressed by

We have thus proved the following theorem.

It is clear that similar definitions may be made for the conditional DF and conditional PDF of the RV Y, given X = x, and an analog of Theorem 6 holds.

In the general case, let (X₁, X₂, …, X_n) be an n-dimensional RV of the continuous type with PDF . Also, let {i₁ < i₂ < < i_k, j₁ < j₂ < < j_l} be a subset of {1, 2, …, n}. Then

(17)

provided that the denominator exceeds 0. Here is the joint marginal PDF of . The conditional densities are obtained in a similar manner.

The case in which (X₁, X₂, … , X_n) is of the discrete type is similarly treated.

We conclude this section with a discussion of a technique called truncation. We consider two types of truncation each with a different objective. In probabilistic modeling we use truncated distributions when sampling from an incomplete population.

If X is a discrete RV with PMF , the truncated distribution of X is given by

(18)

If X is of the continuous type with PDF f, then

(19)

The PDF of the truncated distribution is given by

(20)

Here T is not necessarily a bounded set of real numbers. If we write Y for the RV with distribution function , then Y has support T.

The second type of truncation is very useful in probability limit theory specially when the DF F in question does not have a finite mean. Let be finite real numbers. Define RV X* by

This method produces an RV for which so that X* has moments of all orders. The special case when and is quite useful in probability limit theory when we wish to approximate X through bounded rvs. We say that X^c is X truncated at c if for . Then . Moreover,

so that c can be selected sufficiently large to make arbitrarily small. For example, if then

and given , we can choose c such that .

The distribution of X^c is no longer the truncated distribution . In fact,

where F is the DF of X and F^c, that of X^c.

A third type of truncation, sometimes called Winsorization, sets

This method also produces an RV for which , moments of all orders for X^* exist but its DF is given by

PROBLEMS 4.2

1. Let if , if . Does F define a DF in the plane?
2. Let T be a closed triangle in the plane with vertices , and . Let F(x,y) denote the elementary area of the intersection of T with . Show that F defines a DF in the plane, and find its marginal DFs.
3. Let (X, Y) have the joint PDF f defined by inside the square with corners at the points (1,0), (0,1), (−1,0), and (0, −1) in the (x,y)-plane, and = 0 otherwise. Find the marginal PDFs of X and Y and the two conditional PDFs.
4. Let otherwise, be the joint PDF of (X, Y, Z). Compute and P{X = Y < Z}.
5. Let (X, Y) have the joint PDF , and = 0 otherwise. Find .
6. For DFs F, F₁, F₂,…,F_n show that
   

   for all real numbers x₁,x₂,…,x_n if and only if F_i,’s are marginal DFs of F.

7. For the bivariate negative binomial distribution
   

   where is an integer, , find the marginal PMFs of X and Y and the conditional distributions.

   In Problems 8−10 the bivariate distributions considered are not unique generalizations of the corresponding univariate distributions.

8. For the bivariate Cauchy RV (X, Y) with PDF
   

   find the marginal PDFs of X and Y. Find the conditional PDF of Y given .

   

9. For the bivariate beta RV (X, Y) with PDF
   

   where p₁ , p₂, p₃ are positive real numbers, find the marginal PDFs of X and Y and the conditional PDFs. Find also the conditional PDF of , given X = x.

10. For the bivariate gamma RV (X, Y) with PDF
    

    find the marginal PDFs of X and Y and the conditional PDFs. Also, find the conditional PDF of , given , and the conditional distribution of X/Y, given .

11. For the bivariate hypergeometric RV (X, Y) with PMF
    

    where , N,n integers with , and so that , find the marginal PMFs of X and Y and the conditional PMFs.

12. Let X be an RV with PDF if , and = 0 otherwise. Let . Find the PDF of the truncated distribution of X, its means, and its variance.
13. Let X be an RV with PMF
    

    Suppose that the value cannot be observed. Find the PMF of the truncated RV, its mean, and its variance.

14. Is the function
    

    a joint density function? If so, find , where (X, Y, Z, U) is a random variable with density f.

15. Show that the function defined by
    

    and 0 elsewhere is a joint density function.

    1. Find .
    2. Find .
16. Let (X, Y) have joint density function f and joint distribution function F. Suppose that
    

    holds for and . Show that

    
17. Suppose (X, Y, Z) are jointly distributed with density
    

    Find . Hence find the probability that . (Here g is density function on .)

4.3 INDEPENDENT RANDOM VARIABLES

We recall that the joint distribution of a multiple RV uniquely determines the marginal distributions of the component random variables, but, in general, knowledge of marginal distributions is not enough to determine the joint distribution. Indeed, it is quite possible to have an infinite collection of joint densities f_α with given marginal densities.

In this section we deal with a very special class of distributions in which the marginal distributions uniquely determine the joint distribution of a multiple RV. First we consider the bivariate case.

Let F(x, y) and F₁(x), F₂(y), respectively, be the joint DF of (X, Y) and the marginal DFs of X and Y.

Note that Φ(X²) and ψ(Y²) are independent where Φ and ψ are Borel–measurable functions. But X is not a Borel-measurable function of X².

It is clear that an analog of Theorem 1 holds, but we leave the reader to construct it.

The following result is easy to prove.

Remark 1. It is quite possible for RVs X₁, X₂,…X_n to be pairwise independent without being mutually independent. Let (X, Y, Z) have the joint PMF defined by

Clearly, X, Y, Z are not independent (why?). We have

It follows that X and Y, Y and Z, and X and Z are pairwise independent.

Similarly, one can speak of an independent family of RVs.

According to Definition 4, X and Y are identically distributed if and only if they have the same distribution. It does not follow that with probability 1 (see Problem 7). If , we say that X and Y are equivalent RVs. All Definition 4 says is that X and Y are identically distributed if and only if

Nothing is said about the equality of events and .

Of course, the independence of (X₁, X₂,… ,X_m) and (Y₁, Y₂,… ,Y_n) does not imply the independence of components X₁, X₂,… ,X_m of X or components Y₁, Y₂,… ,Y_n of Y.

Remark 2. It is possible that an RV X may be independent of Y and also of Z, but X may not be independent of the random vector (Y,Z). See the example in Remark 1.

Let X₁, X₂,… ,X_n be independent and identically distributed RVs with common DF F. Then the joint DF G of (X₁,X₂,… ,X_n) is given by

We note that for any of the n! permutations of (x₁, x₂,…,x_n)

so that G is a symmetric function of x₁, x₂,… ,x_n. Thus , where means that X and Y are identically distributed RVs.

Clearly if X₁, X₂,…, X_n are exchangeable, then X_i are identically distributed but not necessarily independent.

In view of Theorem 6, X^s is symmetric about 0 so that

If ,then , and EX^s = 0.

The technique of symmetrization is an important tool in the study of probability limit theorems. We will need the following result later. The proof is left to the reader.

PROBLEMS 4.3

1. Let A be a set of k numbers, and Ω be the set of all ordered samples of size n from A with replacement. Also, let be the set of all subsets of Ω, and P be a probability defined on . Let X₁, X₂,…, X_n be RVs defined on (Ω, , P) by setting
   

   Show that X₁, X₂,… ,X_n are independent if and only if each sample point is equally likely.

2. Let X₁, X₂ be iid RVs with common PMF
   

   Write . Show that X₁, X₂, X₃ are pairwise independent but not independent.

3. Let (X₁, X₂, X₃) be an RV with joint PMF
   

   where

   

   Are X₁,X₂,X₃ independent? Are X₁,X₂,X₃ pairwise independent? Are and X₃ independent?

   No; Yes; No.

4. Let X and Y be independent RVs such that XY is degenerate at . That is, . Show that X and Y are also degenerate.
5. Let (Ω, , P) be a probability space and A, B ∈ . Define X and Y so that
   

   Show that X and Y are independent if and only if A and B are independent.

6. Let X₁,X₂,… ,X_n be a set of exchangeable RVs. Then
   
7. Let X and Y be identically distributed. Construct an example to show that X and Y need not be equal, that is, need not equal 1.
8. Prove Lemma 1.
9. Let X₁,X₂,… ,X_n be RVs with joint PDF f, and let f_j be the marginal PDF of . Show that X₁, X₂,… ,X_n are independent if and only if
   
10. Suppose two buses, A and B, operate on a route. A person arrives at a certain bus stop on this route at time 0. Let X and Y be the arrival times of buses A and B, respectively, at this bus stop. Suppose X and Y are independent and have density functions given, respectively, by
    

    What is the probability that bus A will arrive before bus B?

11. Consider two batteries, one of Brand A and the other of Brand B. Brand A batteries have a length of life with density function
    

    whereas Brand B batteries have a length of life with density function given by

    

    Brand A and Brand B batteries operate independently and are put to a test. What is the probability that Brand B battery will outlast Brand A? In particular, what is the probability if ?

12. 1. Let (X, Y) have joint density f. Show that X and Y are independent if and only if for some constant and nonnegative functions f₁ and f₂
       
       for all .
    2. Let , and f_X, f_Y are marginal densities of X and Y, respectively. Show that if X and Y are independent then .
13. If Φ is the CF of X, show that the CF of X^s is real and even.
14. Let X, Y be jointly distributed with PDF otherwise. Show that and has a symmetric distribution.

4.4 FUNCTIONS OF SEVERAL RANDOM VARIABLES

Let X₁, X₂,… ,X_n be RVs defined on a probability space (Ω, , P). In practice we deal with functions of X₁, X₂,… , X_n such as , min (X₁,… ,X_n), and so on. Are these also RVs? If so, how do we compute their distribution given the joint distribution of X₁, X₂ …, X_n?

What functions of (X₁, X₂,… ,X_n) are RVs?

In particular, if g: is a continuous function, then g(X₁,X₂,… ,X_n) is an RV.

How do we compute the distribution of g(X₁,X₂,… ,X_n)? There are several ways to go about it. We first consider the method of distribution functions. Suppose that is real-valued, and let . Then

where in the continuous case f is the joint PDF of (X₁,…, X_n).

In the continuous case we can obtain the PDF of by differentiating the DF with respect to y provided that Y is also of the continuous type. In the discrete case it is easier to compute .

We take a few examples,

There are two cases to consider according to whether or (Fig. 1a and 1b). In the former case,

and in the latter case,

Hence the density function of Y is given by

The method of distribution functions can also be used in the case when g takes values in _m, 1 ≤ m ≤ n, but the integration becomes more involved.

Let and . Then the joint distribution of (Y₁, Y₂) is given by

where Clearly, so that the set A is as shown in Fig. 2. It follows that

Hence the joint density of Y₁, Y₂ is given by

The marginal densities of Y₁, Y₂ are easily obtained as

We next consider the method of transformations. Let (X₁,…, X_n) be jointly distributed with continuous PDF f(x₁, x₂,…,x_n), and let , where

be a mapping of _n to R_n. Then

where . Let us choose B to be the n-dimensional interval

Then the joint DF of Y is given by

and (if G_Y is absolutely continuous) the PDF of Y is given by

at every continuity point y of w. Under certain conditions it is possible to write w in terms of f by making a change of variable in the multiple integral.

Then (Y₁, Y₂,… ,Y_n) has a joint absolutely continuous DF with PDF given by

(1)

Proof. For (y_1, y₂,…,y_n) ∈ _n, let

Then

and

Result (1) now follows on differentiation of DF G_Y.

Remark 1. In actual applications we will not know the mapping from x₁, x₂,…,x_n to y₁, y₂,…,y_n completely, but one or more of the functions g_i will be known. If only , of the g_i’s are known, we introduce arbitrarily functions such that the conditions of the theorem are satisfied. To find the joint marginal density of these k variables we simply integrate the w function over all the variables that were arbitrarily introduced.

Remark 2. An analog of Theorem 2.5.4 holds, which we state without proof.

Let be an RV of the continuous type with joint PDF f, and let , be a mapping of _n into itself. Suppose that for each y the transformation g has a finite number of inverses. Suppose further that _n can be partitioned into k disjoint sets A₁, A₂,… ,A_k, such that the transformation g from into _n is one-to-one with inverse transformation

Suppose that the first partial derivatives are continuous and that each Jacobian

is different from 0 in the range of the transformation. Then the joint PDF of Y is given by

In Example 6 the transformation used is orthogonal and is known as Helmert’s transformation. In fact, we will show in Section 6.5 that under orthogonal transformations iid RVs with PDF f defined above are transformed into iid RVs with the same PDF.

It is easily verified that

We have therefore proved that is independent of This is a very important result in mathematical statistics, and we will return to it in Section 7.7.

An important application of the result in Remark 2 will appear in Theorem 4.7.2.

Finally, we consider a technique based on MGF or CF which can be used in certain situations to determine the distribution of a function g(X₁, X₂,… ,X_n) of X₁, X₂,… ,X_n.

Let (X₁, X₂,… ,X_n) be an n-variate RV, and g be a Borel-measurable function from _n to ₁.

Let , and let h(y) be its PDF. If then

An analog of Theorem 3.2.1 holds. That is,

in the sense that if either integral exists so does the other and the two are equal. The result also holds in the discrete case.

Some special functions of interest are , where k₁, k₂,… ,k_n are non-negative integers, , where t₁,t₂,… ,t_n are real numbers, and , where .

We will mostly deal with MGF even though the condition that it exist for restricts its application considerably. The multivariate MGF (CF) has properties similar to the univariate MGF discussed earlier. We state some of these without proof. For notational convenience we restrict ourselves to the bivariate case.

A formal definition of moments in the multivariate case will be given in Section 4.5.

The MGF technique uses the uniqueness property of Theorem 4. In order to find the distribution (DF, PDF, or PMF) of we compute the MGF of Y using definition. If this MGF is one of the known kind then Y must have this kind of distribution. Although the technique applies to the case when Y is an m-dimensional RV, , we will mostly use it for the case.

The following result has many applications as we will see. Example 9 is a special case.

From these examples it is clear that to use this technique effectively one must be able to recognize the MGF of the function under consideration. In Chapter 5 we will study a number of commonly occurring probability distributions and derive their MGFs (whenever they exist). We will have occasion to use Theorem 7 quite frequently.

For integer-valued RVs one can sometimes use PGFs to compute the distribution of certain functions of a multiple RV.

We emphasize the fact that a CF always exists and analogs of Theorems 4–7 can be stated in terms of CFs.

PROBLEMS 4.4

1. Let F be a DF and ε be a positive real number. Show that
   

   and

   

   are also distribution functions.

2. Let X, Y be iid RVs with common PDF
   
   1. Find the PDF of RVs , X – Y, XY, X/Y, min{X, Y}, max{X, Y}, min{X, Y}/max{X, Y}, and
   2. Let and . Find the conditional PDF of V, given , for some fixed .
   3. Show that U and are independent.
3. Let X and Y be independent RVs defined on the space (Ω, , P). Let X be uniformly distributed on (–a, a), , and Y be an RV of the continuous type with density f, where f is continuous and positive on . Let F be the DF of Y. If u₀ ∈ (–a, a) is a fixed number, show that
   

   where is the conditional density function of Y, given. .

4. Let X and Y be iid RVs with common PDF
   

   Find the PDFs of RVs XY, X/Y, min {X, Y}, max {X, Y}, min {X, Y}/max {X, Y}.

5. Let X₁, X₂, X₃ be iid RVs with common density function
   

   Show that the PDF of is given by

   

   An extension to the n-variate case holds.

6. Let X and Y be independent RVs with common geometric PMF
   

   Also, let . Find the joint distribution of M and X, the marginal distribution of M, and the conditional distribution of X, given M.

7. Let X be a nonnegative RV of the continuous type. The integral part, Y, of X is distributed with PMF ; and the fractional part, Z, of X has PDF, if , and = 0 otherwise. Find the PDF of X, assuming that Y and Z are independent.
8. Let X and Y be independent RVs. If at least one of X and Y is of the continuous type, show that is also continuous. What if X and Y are not independent?
9. Let X and Y be independent integral RVs. Show that
   

   where P, P_X, and P_Y, respectively, are the PGFs of , X, and Y.

10. Let X and Y be independent nonnegative RVs of the continuous type with PDFs f and g, respectively. Let , and if and let g be arbitrary. Show that the MGF M (t) of Y, which is assumed to exist, has the property that the DF of X/Y is .
11. Let X, Y, Z have the joint PDF
    

    Find the PDF of .

    .

12. Let X and Y be iid RVs with common PDF
    

    Find the PDF of .

13. Let X and Y be iid RVs with common PDF f defined in Example 8. Find the joint PDF of U and V in the following cases:
    1. , ,
    2. ,
14. Construct an example to show that even when the MGF of can be written as a product of the MGF of X and the MGF of Y, X and Y need not be independent.
15. Let X₁, X₂,…, X_n be iid with common PDF
    

    Using the distribution function technique show that

    1. The joint PDF of , and is given by
       
       and = 0 otherwise.
    2. The PDF of X_(n) is given by
       
       and that of X₍₁₎ by
       
16. Let X₁, X₂ be iid with common Poisson PMF
    

    where is a constant. Let and . Find the PMF of X₍₂₎.

17. Let X have the binomial PMF
    

    Let Y be independent of X and . Find PMF OF and .

4.5 COVARIANCE, CORRELATION AND MOMENTS

Let X and Y be jointly distributed on (Ω, , P). In Section 4.4 we defined Eg (X, Y) for Borel functions g on ₂. Functions of the form where j and k are nonnegative integers are of interest in probability and statistics.

Recall (Theorem 3.2.8) that is minimized when we choose so that EY may be interpreted as the best constant predictor of Y. If instead, we choose to predict Y by a linear function of X, say , and measure the error in this prediction by , then we should choose a and b to minimize this so-called mean square error. Clearly, is minimized, for any a, by choosing . With this choice of b, we find a such that

is minimum. An easy computation shows that the minimum occurs if we choose

(7)

Provided . Moreover,

(8)

Let us write

(9)

Then (8) shows that predicting Y by a linear function of X reduces the prediction error from to . We may therefore think of ρ as a measure of the linear dependence between RVs X and Y.

If X and Y are independent, then from (5) , and X and Y are uncorrelated. If, however, then X and Y may not necessarily be independent.

Let us now study some properties of the correlation coefficient. From the definition we see that ρ (and also cov (X, Y)) is symmetric in X and Y.

Equality in (11) holds if and only if , or equivalently, holds. This implies and is implied by . Here a ≠ 0.

Remark 1. From (7) and (9) we note that the signs of a and ρ are the same so if then where a > 0, and if then .

The existence of ES follows easily by replacing each a_j by |a_j| and each x_ij by |x_ij| and remembering that . The case of continuous type (X₁, X₂,…,X_n) is similarly treated.

Let X and Y be independent, and g₁ (∙) and g₂ (∙) be Borel-measurable functions. Then we know (Theorem 4.3.3) that g₁ (X) and g₂ (Y) are independent. If E{g₁(X)}, E{g₂(Y)}, and E{g₁ (X)g₂(Y)} exist, it follows from Theorem 4 that

(14)

Conversely, if for any Borel sets A₁ and A₂ we take if X ∈ A₁, and = 0 otherwise, and if , and = 0 otherwise, then

and , Relation (14) implies that for any Borel sets A₁ and A₂ of real numbers

It follows that X and Y are independent if (14) holds. We have thus proved the following theorem.

Note that the result holds if we replace independence by the condition that X_i’s are exchangeable and uncorrelated.

We conclude this section with some important moment inequalities. We begin with the simple inequality

(20)

where for and for r > 1. For and , (20) is trivially true.

First note that it is sufficient to prove (20) when . Let , and write . Then

Writing , we see that

where . It follows that^. , and < 0 if r < 1. Thus

while

Note that is trivially true since

An immediate application of (20) is the following result.

Corollary. Taking , we obtain the Cauchy–Schwarz inequality,

The final result of this section is an inequality due to Minkowski.

PROBLEMS 4.5

1. Suppose that the RV (X, Y) is uniformly distributed over the region Find the covariance between X and Y.
2. Let (X, Y) have the joint PDF given by
   

   Find all moments of order 2.

3. Let (X, Y) be distributed with joint density
   

   Find the MGF of (X, Y). Are X, Y independent? If not, find the covariance between X and Y.

   dependent.

4. For a positive RV X with finite first moment show that (1) and (2) .
5. If X is a nondegenerate RV with finite expectation and such that , then
   

   (Kruskal [56])

6. Show that for
   

   and hence that

   
7. Given a PDF f that is nondecreasing in the interval , show that for any s>0
   

   with the inequality reversed if f is nonincreasing.

8. Derive the Lyapunov inequality (Theorem 3.4.3)
   

   from Hölder’s inequality (22).

9. Let X be an RV with for . Show that the function log E|X|^r is a convex function of r.
10. Show with the help of an example that Theorem 9 is not true for
11. Show that the converse of Theorem 8 also holds for independent RVs, that is, if for some and X and Y are independent, then .

    [Hint: Without loss of generality assume that the median of both X and Y is 0. Show that, for any , . Now use the remarks preceding Lemma 3.2.2 to conclude that .]

12. Let (Ω, , P) be a probability space, and A₁, A₂,…,A_n be events in such that . Show that
    

    (Chung and Erdös [14])

    [Hint: Let X_k be the indicator function of A_k, . Use the Cauchy–Schwarz inequality.]

13. Let (Ω, , P) be a probability space, and A,B, ∈ with , . Define ρ(A, B) by ρ(A, B) = correlation coefficient between RVs I_A, and I_B, where I_A, I_B, are the indicator functions of A and B, respectively. Express ρ(A, B) in terms of PA, PB, and P(AB) and conclude that if and only if A and B are independent. What happens if or if ?
    1. Show that
       
       and
       
    2. Show that
       
14. Let X₁, X₂,…,X_n be iid RVs and define
    

    Suppose that the common distribution is symmetric. Assuming the existence of moments of appropriate order, show that .

15. Let X,Y be iid RVs with common standard normal density
    

    Let and . Find the MGF of the rendom variable (U, V). Also, find the correlation coefficient between U and V. Are U and V independent?

16. Let X and Y be two discrete RVs:
    

    and

    

    Show that X and Y are independent if and only if the correlation coefficient between X and Y is 0.

17. Let X and Y be dependent RVs with common means 0, variance 1, and correlation coefficient ρ. Show that
    
18. Let X_1, X₂ be independent normal RVs with density functions
    

    Also let

    

    Find the correlation coefficient between Z and W and show that

    

    where ρ denotes the correlation coefficient between Z and W.

19. Let (X₁, X₂,…,X_n) be an RV such that the correlation coefficient between each pair X_i, X_j, , is ρ. Show that .
20. Let X₁, X₂,…,X_m+n be iid RVs with finite second moment. Let . Find the correlation coefficient between S_n and , where .
21. Let f be the PDF of a positive RV, and write
    

    Show that g is a density function in the plane. If the mth moment of f exists for some positive integer m, find EX^m. Compute the means and variances of X and Y and the correlation coefficient between X and Y in terms of moments of f. (Adapted from Feller [26, p. 100].)

    If U has PDF f, then for ; .

22. A die is thrown times. After each throw a + sign is recorded for 4, 5, or 6, and a – sign for 1, 2, or 3, the signs forming an ordered sequence. Each sign, except the first and the last, is attached to a characteristic RV that assumes the value 1 if both the neighboring signs differ from the one between them and 0 otherwise. Let X₁,X₂,…,X_n be these characteristic RVs, where X_i corresponds to the st sign (i = 1, 2,…,n) in the sequence. Show that
    
23. Let (X, Y) be jointly distributed with PDF f defined by inside the square with corners at the points (0, 1), (1, 0), (–1, 0), (0,–1) in the (x, y)-plane, and otherwise. Are X, Y independent? Are they uncorrelated?

4.6 CONDITIONAL EXPECTATION

In Section 4.2 we defined the conditional distribution of an RV X, given Y. We showed that, if (X, Y) is of the discrete type, the conditional PMF of X, given , where , is a PMF when considered as a function of the x_i’s (for fixed y_j). Similarly, if (X, Y) is an RV of the continuous type with PDF f (x,y) and marginal densitiesf₁ and f₂, respectively, then, at every point (x, y) at which f is continuous and at which and is continuous, a conditional density function of X, given Y, exists and may be defined by

We also showed that , for fixed y, when considered as a function of x is a PDF in its own right. Therefore, we can (and do) consider the moments of this conditional distribution.

Needless to say, a similar definition may be given for the conditional expectation .

It is immediate that satisfies the usual properties of an expectation provided we remember that is not a constant but an RV. The following results are easy to prove. We assume existence of indicated expectations.

(2)

(3)

for any Borel functions g₁, g₂.

(4)

(5)

The statements in (3), (4), and (5) should be understood to hold with probability 1.

(6)

for independent RVs X and Y.

If ϕ(X, Y) is a function of X and Y, then

(7)

(8)

for any Borel functions ψ and Φ.

Again (8) should be understood as holding with probability 1. Relation (7) is useful as a computational device. See Example 3 below.

The moments of a conditional distribution are defined in the usual manner. Thus, for , defines the rth moment of the conditional distribution. We can define the central moments of the conditional distribution and, in particular, the variance. There is no difficulty in generalizing these concepts for n-dimensional distributions when . We leave the reader to furnish the details.

Theorem 1 is quite useful in computation of Eh(X) in many applications.

Equation (11) follows immediately from (10). The equality in (11) holds if and only if

which holds if and only if with probability 1

(12)

PROBLEMS 4.6

1. Let X be and RV with PDF given by
   

   Find , where a and b are constants.

   where Φ is the standard normal DF.

2. 1. Let (X, Y) be jointly distributed with density
      
      Find E{Y | X}.
   2. Do the same for the joint density
      
3. Let (X, Y) be jointly distributed with bivariate normal density
   

   Find and . (Here, , and .)

4. Find .
5. Show that is minimized by choosing .
6. Let X have PMF
   

   and suppose that λ is a realization of a RV Λ with PDF

   

   Find .

7. Find E(XY) by conditioning on X or Y for the following cases:
   1. .
   2. .
8. Suppose X has uniform PDF , and 0 otherwise Let Y be chosen from interval (0, X] according to PDF
   

   Find and EY^k for any fixed constant .

4.7 ORDER STATISTICS AND THEIR DISTRIBUTIONS

Let (X₁, X₂,… ,X_n) be an n-dimensional random variable and (x₁, x₂,… ,x_n) be an n-tuple assumed by (X₁, X₂,… ,X_n). Arrange (x₁, x₂,… ,x_n) in increasing order of magnitude so that

where , x₍₂₎ is the second smallest value in x₁, x₂,… ,x_n, and so on, . If any two x_i, x_j are equal, their order does not matter.

Statistical considerations such as sufficiency, completeness, invariance, and ancillarity (Chapter 8) lead to the consideration of order statistics in problems of statistical inference. Order statistics are particularly useful in nonparametric statistics (Chapter 13) where, for example, many test procedures are based on ranks of observations. Many of these methods require the distribution of the ordered observations which we now study.

In the following we assume that X₁, X₂,… , X_n are iid RVs. In the discrete case there is no magic formula to compute the distribution of any X_(j) or any of the joint distributions. A direct computation is the best course of action.

In the following we assume that X₁, X₂ ,… ,X_n are iid RVs of the continuous type with PDF f. Let {X₍₁₎, X₍₂₎,… ,X_(n)} be the set of order statistics for X₁, X₂ ,… ,X_n. Since the X_i are all continuous type RVs, it follows with probability 1 that

It follows (see Remark 2) that

The procedure for computing the marginal PDF of X_(r), the rth-order statistic of X₁, X₂,… ,X_n is similar. The following theorem summarizes the result.

We now compute the joint PDF of X(j) and X(k) .

In a similar manner we can show that the joint PDF of , is given by

for y₁ < y₂ < <y_k, and = 0 otherwise.

The joint PDF of X₍₁₎ and X_(n) is given by

and that of the range by

Finally, we consider the moments, namely, the means, variances, and covariances of order statistics. Suppose X₁, X₂,…X_n are iid RVs with common DF F. Let g be a Borel function on such that , where X has DF F. Then for

and we write

for . The converse also holds. Suppose for . Then,

for and hence

Moreover, it also follows that

As a consequence of the above remarks we note that if for some r, , then and conversely, if for some r, .

PROBLEMS 4.7

1. Let X₍₁₎, X₍₂₎,…X_(n) be the set of order statistics of independent RVs X₁, X₂,…,X_n with common PDF
   
   1. Show that X_(r) and X_(s) – X_(r) are independent for any .
   2. Find the PDF of .
   3. Let . Show that (Z₁, Z₂,…,Z_n) and (X₁, X₂,…,X_n) are identically distributed.
2. Let X₁, X₂,…X_n be iid from PMF
   

   Find the marginal distributions of X₍₁₎, X_(n), and their joint PMF.

3. Let X₁, X₂,…,X_n be iid with a DF
   

   Show that X_(i)/X_(n), , and X_(n) are independent.

4. Let X₁, X₂,…,X_n be iid RVs with common Pareto DF , where , Show that
   1. X₍₁₎, (X₍₂₎/X₍₁₎, …, X_(n)/X₍₁₎) are independent,
   2. X₍₁₎ has Pareto (σ, nα) distribution, and
   3. has PDF
      
5. Let X₁, X₂,…,X_n be iid nonnegative RVs of the continuous type. If , show that . Write . Show that
   

   Find EM_n in each of the following cases:

   1. X_i have the common DF
      
   2. X_i have the common DF
      
6. Let X₁, X₍₂₎…,X_(n) be that order statistics of n independent RVs X₁, X₂,…,X_n with common PDF if , and = 0 otherwise. Show that , and are independent. Find the PDf of Y₁, Y₂,…,Y_n.
7. For the PDF in Problem 4 find EX_(r).
8. An urn contains N identical marbles numbered 1 through N. From the urn n marbles are drawn and let X_(n) be the largest number drawn. Show that , and .