Chapter One

Probability Space

1.1 Introduction/Purpose of the Chapter

The most important object when working with probability is the proper definition of the space studied. Typically, one wants to obtain answers about real-life phenomena which do not have a predetermined outcome. For example, when playing a complex game a person may be wondering: What are my chances to win this game? Or, am I paying too much to play this game, and is there perhaps a different game I should rather play? A certain civil engineer wants to know what is the probability that a particular construction material will fail under a lot of stress. To be able to answer these and other questions, we need to make the transition from reality to a space describing what may happen and to create consistent laws on that space. This framework allows the creation of a mathematical model of the random phenomena. This model, should it be created in the proper (consistent) way, will allow the modeler to provide approximate answers to the relevant questions asked. Thus, the first and the most important step in creating consistent models is to define a probability space which is capable of answering the interesting questions that may be asked.

We denote with Ω the set that contains all the possible outcomes of a random experiment. The set Ω is often called sample space or universal sample space. For example, if one rolls a die, Ω = {1, 2, 3, 4, 5, 6}. The space Ω does not necessarily contain numbers but rather some representation of the outcomes of the real phenomena. For example, if one looks at the types of bricks which may be used to build a house, a picture of each possible brick is a possible representation of each element of Ω.

A generic element of Ω will be denoted by ω. Any collection of outcomes (elements in Ω) is called an event. That is, an event is any subset of the sample space Ω. We will use capital letters from the beginning of the alphabet (A, B, C, etc.) to denote events.

In probability, one needs to measure the size of these events. Since an event is just a subset, we need to define those subsets of 3 Ω that can be measured. The concept of sigma algebra allows us to define a collection of subsets of the sample space on which a measure can be defined. In this chapter we introduce the notion of algebra and sigma algebra and we discuss their basic properties.

1.2 Vignette/Historical Notes

The first recorded notions of Probability Theory appear in 1654 in an exchange of letters between the famous mathematicians Blaise Pascal and Pierre de Fermat. The correspondence was prompted by a simple observation by Antoine Gombaud Chevalier de Méré, a French nobleman with an interest in gambling and who was puzzled by an apparent contradiction concerning a popular dice game. The game consisted of throwing a pair of dice 24 times; the problem was to decide whether or not to bet even money on the occurrence of at least one “double six” during the 24 throws. A seemingly well-established gambling rule led de Méré to believe that betting on a double six in 24 throws would be profitable (based on the payoff of the game). However, his own calculations based on many repetitions of the 24 throws indicated just the opposite. If we translate de Méré problem in todays language, he was trying to establish if the event has probability greater than 0.5. Today the confusion is easy to pinpoint to the proper definition of the probability space. For example, the convention at the time was that rolling a three (two and one showing on the dice) would be the same as rolling a two (a double one). Puzzled by this and other similar gambling problems, de Méré wrote to Pascal. The further correspondence between Fermat and Pascal is the first known documentation of the fundamental principles of the theory of probability.

The first formal treatment of probability theory was provided by Pierre-Simon, marquis de Laplace (1749–1827) in his Théorie Analytique des Probabilités published in 1812. (Laplace, 1886, republished). In 1933 the monograph Grundbegriffe der Wahrscheinlichkeitsrechnung by the Russian preeminent mathematician Andrey Nikolaevich Kolmogorov (1903–1987) outlined the axiomatic approach that forms the basis of the modern probability theory as we know it today (Kolmogoroff, 1973, republished).

1.3 Notations and Definitions

The following notations will be used throughout the book for set (event) operations. In the following, ω is any element and A, B are any sets in the sample space Ω.

• We describe a set or a collection of elements using a notation of the form

• ø is a notation for the set that does not contain any element. This set is called the empty set.
• ω A denotes that the element ω is in the set A; we say “ω belongs to A.” Obviously, ω ∉ A means that the element is not in the set.
• A ⊆ B denotes that A is a subset of B; that is, every element in A is also in B. The set A may actually be equal to B. In contrast, the notation A ⊂ B means that A is a proper subset of B; that is, A is strictly included in B. Mathematically, A ⊆ B is equivalent to the following statement: Any x A implies x B.
• Union of sets:

• Intersection of sets:

• Complement of a set:

• Difference of two sets:

• Symmetric difference:

• Two sets A, B such that A∩ B = ø are called disjoint or mutually exclusive sets.
• A collection of sets A₁, A₂, …, A_n such that A₁ ∪ A₂ ∪ … ∪ A_n = Ω and A_i∩ A_j = ø for any i ≠ j is called a partition of the space Ω.

Every set operation may be expressed in terms of basic operations. For example,

There is a distributive law for intersection over union. If A, B, C are included in Ω, then

and

Furthermore, Ω^c =ø and ø^c = Ω, and for all A ⊂ Ω we have

The De Morgan laws are also very important:

(1.1)

All of these rules may be extended to any finite number of sets in an obvious way. More details and further references about set operations may be found in Billingsley (1995) or Chung (2000).

1.4 Theory and Applications

1.4.1 Algebras

We introduce the notion of σ-algebra (or σ-field) to introduce a collection of sets which we may measure. In other words, we introduce a proper domain of definition for the (soon to be introduced) probability function. First let us denote

that is, is the collection of all possible subsets of Ω, a set containing all possible sets in Ω. This collection is called the parts of Ω.

An algebra on Ω is a collection of such sets in (including Ω) which is closed under complementarity and finite union.

Definition 1.1 (Algebra on Ω) Let Ω be a nonempty space. A collection of events in Ω is called an algebra (or field) on Ω if and only if:

i. Ω .

ii. If A , then A^c .

iii. If A, B , then A ∪ B .

---

Let us list some immediate property of an algebra.

Proposition 1.2 Let be an algebra on Ω. Then:

1. ø .

2. If A, B , then A ∩ B .

3. For any n natural number in , if A₁, …, A_n , then

---

Proof: Note that these are properties of the collection of sets. The collection must contain these. Specifically, ø = Ω^c and since Ω the second point in Definition 1.1 says that ø . Since

by the DeMorgan laws (1.1), points ii and iii in Definition 1.1 imply that (A ∩ B)^c is in and so A ∩ B is in . The proof for the third part is a simple induction on n ≥ 2, with the verification step already provided in iii of Definition 1.1.

1.4.2 Sigma Algebras

A σ-algebra on Ω is a generalization of an algebra on Ω.

Definition 1.3 (σ-Algebra on Ω) A σ-algebra on Ω is an algebra on Ω and in addition it is closed under countable unions. That is, the first two points in Definition 1.1 remain the same, and the third property is replaced with:

iii. If is a natural number and for all n, then

---

Definition 1.3 implies other properties of a σ-algebra. For example, if a sigma algebra contains two elements A and B, it also contains A ∩ B and A ∪ B. The fact that the intersection must be in the σ-algebra is an easy consequence of point ii in the above definitions. For the σ-algebra case the intersection can be extended to a countable number of events as the next result shows.

Proposition 1.4 Let be a σ-algebra on Ω. Then:

1. .

2. If are elements of the σ-algebra , then

---

Proof: Since ø = Ω^c and , it follows that

For the second part, we note that

and since for every , from the second part of the definition, their countable union is , and thus its complement is .

The σ-algebra is a very nice collection of sets. Using simple the operations union, intersection, complementarity, and difference applied to any sets in the σ-algebra, we end up with a set still in the σ-algebra. In fact, we may apply these simple operations in any combination a countably infinite number of times and we still end up with a set in the σ-algebra.

However, a noncountable intersection or union of the elements of a σ-algebra does not necessarily belongs to it (although it may). This can only happen when the index n is in some continuous set—for example, n (0, 1). Such situations are advanced: however, they become relevant when talking about stochastic processes. Extra care has to be observed in these cases. In the context of stochastic processes, we will introduce filtrations, which are simply increasing σ-algebras.

Because of these nice properties, a σ-algebra provides a suitable domain of definition for the probability function (and thus defines probabilities of random events). However, a σ-algebra is a very abstract concept which, in general, is hard to work with. To simplify notions, we introduce the next definition. It will be much easier to work with the generators of a σ-algebra.

This will be a recurring theme in probability; in order to show a property for a big class, we show the property for a small generating subset of the class and then use standard arguments to extend the property to the entire class.

Definition 1.5 (σ-algebra generated by a class of sets in Ω) Let be any collection of sets in Ω. Then we define σ() as the smallest σ-algebra on Ω that contains . That is, σ() satisfies the following properties:

1. It contains all the sets in : ⊆ σ().

2. σ() is a σ-algebra.

3. It is the smallest σ-algebra with this property: If ⊆ and is another σ-field, then σ() ⊆ .

---

The idea, as mentioned earlier, is to verify some statement on the set . Then, due to the properties that would be presented later, the said statement will be extended and be valid for all the sets in σ().

Proposition 1.6 Let . Then

---

Proof: The proof is left as an exercise (see Exercise 1.12).

Again, these properties are easy to derive directly from the definition and are left as exercise.

In general, listing all the elements of a σ-algebra explicitly is hard. Only in simple cases is this is even possible. This explains why we prefer to work with the generating collection instead of directly with the σ-algebra σ().

1.4.3 Measurable Spaces

After introducing σ-algebras, we are now able to give the notion of a space on which we can introduce probability measure.

Definition 1.8 (Measurable space) A pair , where Ω is a nonempty set and is a σ-algebra on Ω is called a measurable space.

---

On this type of space we shall introduce the probability measure.

1.4.4 Examples

Example 1.1 The Measurable Space Generated by a Set

Suppose a set A ⊂ Ω. Let us calculate σ(A). Clearly, by definition, Ω is in σ(A). Using the complementarity property, we clearly see that A^c and ø are also in σ(A). We only need to take unions of these sets and see that there are no more new sets. Thus,

---

Example 1.2 The Space Modeling the Rolls of a Six Sided Die

Roll a six-sided die. Then use Ω = {1, 2, 3, 4, 5, 6} to denote the space of outcomes of rolling the die. Let A = {1, 2}. Then

---

Example 1.3 The Measurable Set Generated by two Sets

Suppose that = {A, B}, where A and B are two sets in Ω such that A ⊂ B. List all the sets in σ().

---

Solution: A common mistake made by students who learn these notions is the following argument:

A ⊂ B; therefore, using the fourth property in the proposition above, we obtain σ(A) ⊆ σ(B) and therefore the σ-algebra asked is σ(A, B) = σ(B) = {Ω, ø, B, B^c}.

This argument is wrong on several levels. Firstly, the quoted property refers to collections of sets and not to the sets themselves. While it is true that A ⊂ B, it is not true that {A} ⊂ {B} as collections of sets. Instead, {A} ⊂ {A, B} and indeed this implies σ(A) ⊆ σ(A, B) (and similarly for σ(B)). But this just means that the result should contain all the sets in σ(A) (the sets in the previous example).

Second, as this example shows and as the following proposition says, it is not true that σ(A) ∪ σ(B) = σ(A, B), so we can't just simply list all the elements in σ(A) and σ(B) together. The only way to solve this problem is the hard way—that is, actually calculating the sets.

Clearly, σ(A, B) should contain the basic sets and their complements; thus

The σ-algebra should also contain all their unions according to the definition. Therefore, it must contain

where the equalities are obtained using that A ⊂ B. So the only new set to be added is A ∪ B^c and thus its complement as well: (A ∪ B^c)^c = A^c ∩ B. Now we need to verify that by taking unions we do not obtain any new sets, a task left for the reader. In conclusion, when A ⊂ B we have

Proposition 1.10 (Intersection and union of σ-algebras) Suppose that and are two σ-algebras on Ω. Then:

1. is a σ-algebra.

2. is not a σ-algebra. The smallest σ-algebra that contains both of them is and is denoted

---

Proof: For part 2 there is nothing to show. A counterexample is immediate from the previous exercise. Take and . Example 1.3 calculates σ(A, B), and it is simple to see that needs more sets to become a σ-algebra. It is worth mentioning that in some trivial cases the union may be a σ-algebra, such as for example when and they both contain a countable number of elements.

For part 1 we need to verify the three properties in the definition of the sigma algebra. For example, take A in . So A belongs to both collections of sets. Since is a σ-algebra by definition, we have . Similarly, . Therefore, . The rest of the definition is verified in a similar manner.

Remark 1.11 Part 1 of the proposition above may be generalized to any countable intersection of σ-algebras. That is, if is any sequence of σ-fields and I is countable, then

is a σ-field.

---

1.4.5 The Borel σ-Algebra

Let Ω be a topological space (think geometry is defined in this space, and this assures us that the open subsets exist in this space).

Definition 1.12 We define:

(1.2)

---

In the special case when , we denote , the Borel sets of . This is the most important σ-algebra. The reason is that most experiments on abstract spaces Ω may be made equivalent with experiments on (as we shall see later when we talk about random variables). Thus, if we define a probability measure on , we have a way to calculate probabilities for most experiments.

There is nothing special about using the open sets as generators, except for the fact that the open sets can be defined in any topological space. The Borel sets over may be generated by various other classes of intervals; in the end the result is the same σ-algebra. Please see problem 2.10 for these classes of generators.

Pretty much, any subset of you can think about is in . However, it is possible (though very difficult) to explicitly construct a subset of which is not in . See (Billingsley, 1995, page 45) for such a construction in the case Ω = (0, 1]. These sets are generally constructed using a Cantor set argument.

Example 1.4

Let Ω6 =ø and let A₁, …, A_n be a partition of Ω with a finite number of sets. We denote

We will show that is the σ-algebra generated by the sets in the partition—that is in fact the same as σ(A₁, …, A_n).

---

Solution: We start by proving that is a σ-algebra. First,

which implies that .

Suppose that is an element of . Then its complement is

This is easy to see since A_i form a partition of Ω. Thus and the intersection of these gives exactly the sets not indexed by J. Therefore, A^c .

Moreover, for any integer .

which implies that is closed under finite number of unions, and is an algebra. Furthermore, since the total index set {1, 2, …, n} is finite, any countable union of subsets J_p is also included in {1, 2, …, n} and is closed under countable unions, thus it is in fact a σ-algebra.

Since is a σ-algebra which contains the events A_i, it must also contain the sigma algebra generated by A₁, …, A_n. This means that ⊇ σ(A₁, …, A_n).

Conversely, σ(A₁, …, A_n) is a σ-algebra, thus by definition it must contain any countable (in particular finite) union of A_i's. But the set of finite unions is precisely , which shows that ⊆ σ(A₁, …, A_n).

The double inclusion above then shows that is the σ-algebra generated by A₁, …, A_n.

Example 1.5 Examples of Algebras and σ-Algebras on

Let be the σ-algebra generated by the finite subsets of (i.e., all subsets in with a finite number of elements). Denote by the set of subsets such that A is countable or A^c is countable.

1. Show that is a σ-algebra.

2. Show that and coincide.

3. Compare and the Borel σ-algebra .

---

Solution: 1. We need to verify the three properties of a σ-algebra. Clearly ø , and by the definition of we have A^c if A .

Let A_n be a sequence of events included in . We have two possibilities: Either all A_n's in the sequence are countable, or there exist an event in the sequence (A_n)_n which is not countable.

If every A_n is countable, then clearly

is countable (a countable union of countable sets is countable) and consequently _nA_n is in .

Suppose that there exists at least some set, say A_p, that is not countable. Since A_p , then is countable. By DeMorgan laws (1.1) we have

where the last inclusion holds because is countable. Thus, the complement of the set is at most countable and therefore it belongs to .

2. Clearly ⊆ . Recall that any finite set is also countable and so is a σ-algebra that contains all the finite parts of . Since is the smallest σ-algebra that contains these finite parts, the inclusion follows.

Conversely, we will prove that contains every set such that A or A^c is finite or countable. Indeed, if A is countable, then A can be written as

Since every singleton {x} is in and is stable under countable union, we deduce that A . If A^c is countable, the same argument applied to it will imply that A^c . But is a σ-algebra and so A . In either case we deduce that any set A in belongs to . Therefore ⊆ , and by double inclusion the two are the same.

3. The Borel σ-algebra contains all the finite subsets of . Therefore, we have ⊂ . This inclusion is strict. Indeed, the interval (0, 1) is in , but it is not in because neither (0, 1) nor its complement is countable. Since is the same as , it is a proper subset (included strictly) of .

1.5 Summary

In this chapter we introduce the space on which probability is defined. This beginning chapter introduces the notion of σ-algebra, which is going to be the domain of definition for the probability measure. We need to know what we can measure and what we cannot. The sets that can be measured and given a probability will always be in the σ-algebra. Furthermore, this chapter introduces the basic notions of the set operations such as inclusion, intersection, and so on.

Exercises

1.1 Show that if A, B are two event in Ω, then

1.2 If (A_i)_iI is any collection of events in Ω, then show that

and

1.3 Roll a six-sided die. Use Ω = {1, 2, 3, 4, 5, 6} to denote the possible outcomes. An example of an event in this space Ω is A = {Roll an even number} = {2, 4, 6}. Find the cardinality (total number of elements) of .

1.4 Suppose two events A and B are in some space Ω. List the elements of the generated σ algebra σ(A, B) in the following cases:

1.5 An algebra which is not a σ-algebra

Let ₀ be the collection of sets of the form (a₁, a₁'] ∪ (a₂, a₂'] ∪ … ∪ (a_m, a_m'], for any and all a₁ < a₁' < a₂ < a₂' < … < a_m < a_m' in Ω = (0, 1]. Verify that ₀ is an algebra. Show that ₀ is not a σ-algebra.

1.6 Let .

a. Show that is an algebra.

b. Show that if Ω is finite, then is a σ-algebra.

c. Show that if Ω is infinite, then is not a σ-algebra.

1.7 A σ-algebra does not necessarily contain all the events in Ω Let . Show that is a σ-algebra.

Note that if Ω is uncountable, this implies that it contains a set A such that both A and A^c are uncountable, thus .

1.8 Show that the Borel sets of .

Hint: Show that the generating set is the same; that is, show that any set of the form (− ∞, x] can be written as countable union (or intersection) of open intervals and vice versa that any open interval in can be written as countable union (or intersection) of sets of the form (− ∞, x].

1.9 Show that the following classes all generate the Borel σ-algebra; or, expressed differently, show the equality of the following collections of sets:

where is the set of all rational numbers.

1.10 Let A, B, C be three events in a probability space. Express the following events in terms of elements in A, B, C using union, intersection, complementarity, etc.

1.11 Let , be collections of subsets of Ω.

a. If , prove that

b. If , show that

1.12 Give a proof for Proposition 1.6.

Hint. Use exercise 1.11 for one of the two inclusions (the other one is obvious).

1.13 This problem gives the structure of the Borel σ-algebra on .

a. Let be the set of open rectangles in with extremities in —that is, the set of the form (p, q) × (r, s), where . Show that is countable.

b. Let U be an open set in . Prove that

that is, the open sets may be written as countable unions of elements in .

c. Finally, conclude that the σ-algebra generated by the open rectangles in coincides with the Borel σ-algebra on .

1.14 Prove the claim stated in Remark 1.11.