5.1 INTRODUCTION

In preceding chapters we studied probability distributions in general. In this chapter we will study some commonly occurring probability distributions and investigate their basic properties. The results of this chapter will be of considerable use in theoretical as well as practical applications. We begin with some discrete distributions in Section 5.2 and follow with some continuous models in Section 5.3. Section 5.4 deals with bivariate and multivariate normal distributions and in Section 5.5 we discuss the exponential family of distributions.

5.2 SOME DISCRETE DISTRIBUTIONS

In this section we study some well-known univariate and multivariate discrete distributions and describe their important properties.

5.2.1 Degenerate Distribution

The simplest distribution is that of an RV X degenerate at point k, that is, and = 0 elsewhere. If we define

(1)

the DF of the RV X is . Clearly, , , and . In particular, . This property characterizes a degenerate RV. As we shall see, the degenerate RV plays an important role in the study of limit theorems.

5.2.2 Two-Point Distribution

We say that an RV X has a two-point distribution if it takes two values, x ₁ and x ₂, with probabilities

We may write

(2)

where I_A is the indicator function of A. The DF of X is given by

(3)

Also

(4)

(5)

In particular,

(6)

and

(7)

If , , we get the important Bernoulli RV:

(8)

For a Bernoulli RV X with parameter p, we write X ~ b(1, p) and have

(9)

Bernoulli RVs occur in practice, for example, in-coin-tossing experiments. Suppose that , , and . Define RV X so that and . Then and . Each repetition of the experiment will be called a trial. More generally, any nontrivial experiment can be dichotomized to yield a Bernoulli model. Let (Ω, , P) be the sample space of an experiment, and let with . Then . Each performance of the experiment is a Bernoulli trial. It will be convenient to call the occurrence of event A a success and the occurrence of A^c, a failure.

5.2.3 Uniform Distribution on n Points

X is said to have a uniform distribution on n points {x ₁, x ₂, … , x_n } if its PMF is of the form

(10)

Thus we may write

(11)

(12)

and

(13)

if we write . Also,

(14)

If, in particular, ,

(15)

(16)

5.2.4 Binomial Distribution

We say that X has a binomial distribution with parameter p if its PMF is given by

(17)

Since , the p_k ’s indeed define a PMF. If X has PMF (17), we will write X ~ b(n, p). This is consistent with the notation for a Bernoulli RV. We have

In Example 3.2.5 we showed that

(18)

(19)

and

(20)

where . Also

(21)

The PGF of is given by .

Binomial distribution can also be considered as the distribution of the sum of n independent, identically distributed b (1, p) random variables. If we toss a coin, with constant probability p of heads and 1 − p of tails, n times, the distribution of the number of heads is given by (17). Alternatively, if we write

the number of heads in n trials is the sum . Also

Thus

and

5.2.5 Negative Binomial Distribution (Pascal or Waiting Time Distribution)

Let (Ω, , P) be a probability space of a given statistical experiment, and let with . On any performance of the experiment, if A happens we call it a success, otherwise a failure. Consider a succession of trials of this experiment, and let us compute the probability of observing exactly r successes, where is a fixed integer. If X denotes the number of failures that precede the rth success, is the total number of replications needed to produce r successes. This will happen if and only if the last trial results in a success and among the previous trials there are exactly X failures. It follows by independence that

(22)

Rewriting (22) in the form

(23)

we see that

(24)

It follows that

Let X be a b(n, p) RV, and let Y be the RV defined in (28). If there are r or more successes in the first n trials, at most n trials were required to obtain the first r of these successes.

We have

(31)

and also

(32)

In the special case when , the distribution of X is given by

(33)

An RV X with PMF (33) is said to have a geometric distribution. Clearly, for the geometric distribution, we have

(34)

5.2.6 Hypergeometric Distribution

A box contains N marbles. Of these, M are drawn at random, marked, and returned to the box. The contents of the box are then thoroughly mixed. Next, n marbles are drawn at random from the box, and the marked marbles are counted. If X denotes the number of marked marbles, then

(40)

Since x cannot exceed M or n, we must have

(41)

Also and , so that

(42)

Note that

for arbitrary numbers a, b and positive integer n. It follows that

5.2.7 Negative Hypergeometric Distribution

Consider the model of Section 5.2.6. A box contains N marbles, M of these are marked (or say defective) and are unmarked. A sample of size n is taken and let X denote the number of defective marbles in the sample. If the sample is drawn without replacement we saw that X has a hypergeometric distribution with PMF (40). If, on the other hand, the sample is drawn with replacement then where .

Let Y denote the number of draws needed to draw the rth defective marble. If the draws are made with replacement then Y has the negative binomial distribution given in (22) with . What if the draws are made without replacement? In that case in order that the kth draw () be the rth defective marble drawn, the kth draw must produce a defective marble, whereas the previous draws must produce defectives. It follows that

for . Rewriting we see that

(50)

An RV Y with PMF (50) is said to have a negative hypergeometric distribution.

It is easy to see that

and

Also, if , and as , then

which is (22).

5.2.9 Multinomial Distribution

The binomial distribution is generalized in the following natural fashion. Suppose that an experiment is repeated n times. Each replication of the experiment terminates in one of k mutually exclusive and exhaustive events A₁, A₂, …, A_k. Let p_j be the probability that the experiment terminates in A_j, , and suppose that p_j () remains constant for all n replications. We assume that the n replications are independent.

Let x₁, x₂, …, x_k−1 be nonnegative integers such that . Then theprobability that exactly x_i trials terminate in A_i, and hence that trials terminate in A_k is clearly

If (X₁, X_2,…, X_k) is a random vector such that X_j = x_j means that event A_j has occurred x_j times, x_j = 0,1, 2, …, n, the joint PMF of (X₁, X₂, …, X_k) is given by

(55)

From the MGF of (X₁, X₂, … , X_k−1) or directly from the marginal PMFs we can compute the moments. Thus

(59)

and for , and ,

(60)

It follows that the correlation coefficient between X_i and X_j is given by

(61)

Finally, we note that, if and are two independent multinomial RVs with common parameter (p₁, p₂, … , p_k), then is also a multinomial RV with probabilities (p₁, p₂, … , p_k). This follows easily if one employs the MGF technique, using (57). Actually this property characterizes the multinomial distribution. If X and Y are k-dimensional, nonnegative, independent random vectors, and if is a multinomial random vector with parameter (p ₁ , p ₂, … , p_k ), then X and Y also have multinomial distribution with the same parameter. This result is due to Shanbhag and Basawa [103] and will not be proved here.

5.2.10 Multivariate Hypergeometric Distribution

Consider an urn containing N items divided into k categories containing n ₁, n ₂, … , n_k items, where . A random sample, without replacement, of size n is taken from the urn. Let X_i = number of items in sample of type i. Then

(65)

where and

We say that (X ₁, X ₂ , … , X _k−1) has multivariate hypergeometric distribution if its joint PMF is given by (65). It is clear that each X_j has a marginal hypergeometric distribution. Moreover, the conditional distributions are also hypergeometric. Thus

and

and so on. It is therefore easy to write down the marginal and conditional means and variances. We leave the reader to show that

and

5.2.11 Multivariate Negative Binomial Distribution

Consider the setup of Section 5.2.9 where each replication of an experiment terminates in one of k mutually exclusive and exhaustive events A ₁, A ₂, … , A_k .Let , j = 1, 2, … , k. Suppose the experiment is repeated until event A_k is observed for the rth time, . Then

(66)

for , , , , and .

We say that (X ₁, X ₂, … , X _k−1) has a multivariate negative binomial (or negative multinomial) distribution if its joint PMF is given by (66).

It is easy to see the marginal PMF of any subset of {X ₁ , X ₂ , … , X _k–1} is negative multinomial. In particular, each X_j has a negative binomial distribution.

We will leave the reader to show that

(67)

and

(68)

PROBLEMS 5.2

1. 1. Let us write
      

      Show that, as k goes 0 to n, b(k; n, p) first increases monotonically and then decreases monotonically. The greatest value is assumed when , where m is an integer such that

      

      except that when .

   2. If , then
      

      and if , then

      
2. Generalize the result in Theorem 10 to n independent Poisson RVs, that is, if X₁, X₂, … , X_n are independent RVs with , the conditional distribution of X ₁, X ₂,…X_n given , is multinomial with parameters t, .
3. Let X ₁, X ₂ be independent RVs with . What is the PMF of ?
4. A box contains N identical balls numbered 1 through N. Of these balls, n are drawn at a time. Let X ₁, X ₂, … , X_n denote the numbers on the n balls drawn. Let . Find var(S_n ).
5. From a box containing N identical ball marked 1 through N, M balls are drawn one after another without replacement. Let X_i denote the number on the ith ball drawn, , . Let . Find the DF and the the PMF of Y. Also find the conditional distribution of X₁, X₂, … , X_M given Y = y. Find EY and var(Y).
6. Let f(x; r, p), , denote the PMF of an NB(r; p) RV. Show that the terms f(x; r, p) first increase monotonically and then decrease monotonically. When is the greatest value assumed?
7. Show that the terms
   

   of the Poisson PMF reach their maximum when k is the largest integer ≤ λ and at (λ – 1) and λ if λ is an integer.

8. Show that
   

   as n → ∞ and p → 0, so that np = λ remains constant.

   [Hint: Use Stirling’s approximation, namely, as n → ∞.]

9. A biased coin is tossed indefinitely. Let p(0 < p < 1) be the probability of success (heads). Let Y ₁ denote the length of the first run, and Y ₂, the length of the second run. Find the PMFs of Y ₁ and Y ₂ and show that EY ₁ = q/p + p/q, EY ₂ = 2. If Y_n denotes the length of the nth run, n ≥ 1, what is the PMF of Y_n ? Find EY_n .
10. Show that
    

    as N → ∞.

11. Show that
    

    as p → 1 and r → ∞ in such a way that r(1 – p) = λ remains fixed.

12. Let X and Y be independent geometric RVs. Show that min (X, Y) and X – Y are independent.
13. Let X and Y be independent RVs with PMFs , 1, 2, … , where p_k , q_k > 0 and . Let
    

    Then α_t = α for all t, and

    

    where , and - θ > 0 is arbitrary.

    (Chatterji [13])

14. Generalize the result of Example 10 to the case of k urns, k ≥ 3.
15. Let (X ₁, X ₂, … , X _k–1) have a multinomial distribution with parameters n, p ₁, p ₂, … , p _k−1. Write
    

    where p_k = 1 – p ₁ – – p _k–1, and . Find EY and var(Y).

16. Let X ₁, X ₂ be iid RVs with common DF F, having positive mass at 0, 1, 2,… Also, let U = max(X ₁, X ₂) and . Then
    

    For all j if and only if F is a geometric distribution.

    (Srivastava [109])

17. Let X and Y be mutually independent RVs, taking nonnegative integer values. Then
    

    Holds for n = 0, 1, 2,… and some α > 0 if and only if

    

    [Hint: Use Problem 3.3.8.]

    (Puri [83])

18. Let X₁, X₂,… be a sequence of independent b(1, p) RVs with . Also, let , where N is a P(λ) RV which is independent of the X_i ’s. Show that Z_N and N – Z_N are independent.
19. Prove Theorems 5, 7, 8 and 11.
20. In Example 2 show that
    1. 
    2. 

5.3 SOME CONTINUOUS DISTRIBUTIONS

In this section we study some most frequently used absolutely continuous distributions and describe their important properties. Before we introduce specific distributions it should be remarked that associated with each PDF f there is an index or a parameter θ (may be multidimensional) which takes values in an index set Θ. For any particular choice of θ ∈ Θ we obtain a specific PDF f_θ from the family of PDFs {f_θ , θ ∈ Θ}.

Let X be an RV with PDF f_θ (x), where θ is a real-valued parameter. We say that θ is a location parameter and {f_θ } is a location family if X – θ has PDF f(x) which does not depend on θ. The parameter θ is said to be a scale parameter and {f_θ } is a scale family of PDFs if X/θ has PDF f(x) which is free of θ. If is two-dimensional, we say that θ is a location-scale parameter if the PDF of (X–μ)/σ is free of μ and σ. In that case {f_θ } is known as a location-scale family.

It is easily seen that θ is a location parameter if and only if , a scale parameter if and only , and a location-scale parameter if , for some PDF f. The density f is called the standard PDF for the family {f_θ, θ ∈ Θ}.

A location parameter simply relocates or shifts the graph of PDF f without changing its shape. A scale parameter stretches (if ) or contracts (if ) the graph of f. A location-scale parameter, on the other hand, stretches or contracts the graph of f with the scale parameter and then shifts the graph to locate at μ. (see Fig. 1.)

Some PDFs also have a shape parameter. Changing its value alters the shape of the graph. For the Poisson distribution λ is a shape parameter.

For the following PDF

and = 0 otherwise, μ is a location, β, a scale, and α, a shape parameter. The standard density for this location-scale family is

and = 0 otherwise. For the standard PDF f, α is a shape parameter.

5.3.1 Uniform Distribution (Rectangular Distribution)

Indeed, implies, that, for every , . Since is arbitrary and F is continuous on the right, we let ε → 0 and conclude that . Since implies by definition (7), it follows that (8) holds generally. Thus

Theorem 2 is quite useful in generating samples with the help of the uniform distribution.

To complete the proof we consider the case where x is a positive irrational number. Then we can find a decreasing of positive rational x ₁, x ₂,… such that x_n → x. Since f is right continuous,

Now, for ,

Since , we must have , so that

This complete the proof.

5.3.2 Gamma Distribution

The integral

(9)

converges or diverges according as or ≤ 0. For the integral in (9) is called the gamma function. In particular, if , . If , integration by parts yields

(10)

If is a positive integer, then

(11)

Also writing in we see that

Now consider the integral . We have

and changing to polar coordinates we get

It follows that

Let us write , , in the integral in (9). Then

(12)

so that

(13)

Since the integrand in (13) is positive for , it follows that the function

(14)

defines a PDF for ,

The special case when leads to the exponential distribution with parameter β. The PDF of an exponentially distributed RV is therefore

(21)

Note that we can speak of the exponential distribution on (−∞, 0). The PDF of such an RV is

(22)

Clearly, if , we have

(23)

(24)

(25)

Another special case of importance is when , (an integer), and .

If X ~ χ²(n), then

(27)

(28)

and

(29)

5.3.3 Beta Distribution

The integral

(37)

converges for and and is called a beta function. For or the integral in (37) diverges. It is easy to see that for and

and

(38)

(39)

and

(40)

It follows that

(41)

defines a pdf.

Let X₁,X₂,…,X_n be iid RVs with the uniform distribution on [0, 1]. Let X_(k) be the kth-order statistic.

5.3.4 Cauchy Distribution

We will write for a Cauchy RV with density (49)

Figure 4 gives graph of a Cauchy PDF.

We first check that (49) in fact defines a PDF. Substituting , we get

The DF of a (1, 0) RV is given by

(50)

It follows from Theorem 17 that the MGF of a Cauchy RV does not exist. This creates some manipulative problems. We note, however, that the CF of is given by

(51)

In particular, if X ₁,X₂, … , X_n are iid (1,0) RVs, n^—1S_n is also a (1,0) RV. This is a remarkable result, the importance of which will become clear in Chapter 7. Actually, this property uniquely characterizes the Cauchy distribution. If F is a nondegenerate DF with the property that n^-1S_n also has DF F, then F must be a Cauchy distribution (see Thompson [113, p. 112]).

The proof of the following result is simple.

We emphasize that if X and 1 /X have the same PDF on (−∞,∞), it does not follow* that X is (1,0), for let X be an RV with PDF

Then X and 1 /X have the same PDF, as can be easily checked.

5.3.5 Normal Distribution (the Gaussian Law)

One of the most important distributions in the study of probability and mathematical statistics is the normal distribution, which we will examine presently.

If X is a normally distributed RV with parameters μ and σ, we will write . In this notation φ defined by (53) is the PDF of an (0,1) RV. The DF of an (0, 1) RV will be denoted Φ (x), where

(54)

Clearly, if , then . Z is called a standard normal RV. For the MGF of an RV, we have

(55)

for all real values of t. Moments of all order exist and may be computed from the MGF. Thus

(56)

and

(57)

Thus

(58)

Clearly, the central moments of odd order are all 0. The central moments of even order are as follows:

(59)

As for the absolute moment of order α, for a standard normal RV Z we have

(60)

As remarked earlier, the normal distribution is one of the most important distributions in probability and statistics, and for this reason the standard normal distribution is available in tabular form. Table ST2 at the end of the book gives the probability for various values of in the tail of an RV. In this book we will write z_a for the value of Z that satisfies .

We remark that if X₁, X₂, … , X_n are iid RVs with such that also has the same distribution for each , that distribution can only be (0,1). This characterization of the normal distribution will become clear when we study the central limit theorem in Chapter 7.

If X and Y are independent normal RVs, is normal by Theorem 22. The converse is due to Cramér [16] and will not be proved here.

In Chapter 6 we will prove the necessity part of this result, which is basic to the theory of t-tests in statistics (Chapter 10; see also Example 4.4.6). The sufficiency part was proved by Lukacs [67] , and we will not prove it here.

We remark that the converse of this result does not hold; that is, if is the quotient of two iid RVs and Z has a (1, 0) distribution, it does not follow that X and Y are normal, for take X and Y to be iid with PDF

We leave the reader to verify that is (1, 0).

5.3.6 Some Other Continuous Distributions

Several other distributions which are related to distributions studied earlier also arise in practice. We record briefly some of these and their important characteristics. We will use these distributions infrequently. We say that X has a lognormal distribution if X has a normal distribution. The PDF of X is then

(65)

and for , where . In fact for

Where Φ is the DF of a (0, 1) RV which leads to (65). It is easily seen that for n≥0

(66)

The MGF of X does not exist.

We say that the RV X has a Pareto distribution with parameters and if its PDF is given by

(67)

and 0 otherwise. Here θ is scale parameter and α is a shape parameter. It is easy to check that

(68)

for α > 2. The MGF of X does not exist since all moments of X do not.

Suppose X has a Pareto distribution with parameters θ and α. Wright we see that Y has PDF

(69)

and DF

The PDF in (69) is known as a logistic distribution. We introduce location and scale parameters μ and σ by writing taking and then the PDF of Z is easily seen

to be

(70)

for all real z. This is the PDF of a logistic RV with location-scale parameters μ and σ. We leave the reader to check that

(71)

Pareto distribution is also related to an exponential distribution. Let X have Pareto PDF of the form

(72)

and 0 otherwise. A simple transformation leads to PDF (72) from (67). Then it is easily seen that has an exponential distribution with mean 1/α. Thus some properties of exponential distribution which are preserved under monotone transformations can be derived for Pareto PDF (72) by using the logarithmic transformation.

Some other distributions are related to the gamma distribution. Suppose .

Let . Then Y has PDF

(73)

and 0 otherwise. The RV Y is said to have a Weibull distribution. We leave the reader to show that

(74)

The MGF of Y exists only for but for it does not have a form useful in applications. The special case and is known as a Rayleigh distribution.

Suppose X has a Weibull distribution with PDF (73). Let . Then Y has DF

Setting and we get

(75)

with PDF

(76)

for and . An RV with PDF (76) is called an extreme value distribution with location-scale parameters θ and σ. It can be shown that

(77)

where is the Euler constant.

The final distribution we consider is also related to a G(1, β) RV. Let f ₁ be the PDF of G(1, β) and f ₂ the PDF

Clearly f ₂ is also an exponential PDF defined on . Consider the mixture PDF

(78)

Clearly,

(79)

and the PDF f defined in (79) is called a Laplace or double exponential pdf. It is convenient to introduce a location parameter μ and consider instead the PDF

(80)

where . It is easy to see that for RV X with PDF (80) we have

(81)

for .

For completeness let us define a mixture PDF (PMF). Let be a PDF and let h(θ) be a mixing PDF. Then the PDF

(82)

is called a mixture density function. In case h is a PMF with support set {θ1, θ2, …, θ _κ }, then (82) reduces to a finite mixture density function

(83)

The quantities h(θ _i ) are called mixing proportions. The PDF (78) is an example with , and .