3.2.1 Matrices, Vectors, and Arrays

To avoid confusion, we need to clarify some concepts and the terms we use to reference these concepts. In linear algebra, the term scalar refers to a nondimensional quantity, whereas values commonly refers to vectors, matrices, or arrays. Informally, the terms matrix, vector, and array are sometimes used interchangeably, but more formally, an array is a set of numbers organized by a finite number of fixed sized dimensions. Within MATLAB, the term array can also denote a data structure, a set of numeric values. However, in this tutorial, we will use “array” in its mathematical sense.

A matrix is a two-dimensional array of numbers or variables. Matrices are usually depicted as a rectangular group of numbers, with rows and columns corresponding to the two dimensions. If there are more than two dimensions, we would call it a “tensor,” but let’s not get into that at this point. The sizes of the two dimensions of a matrix are often written as m×n, where m indicates the number of rows and n indicates the number of columns. Here is an example of a 2×3 matrix, A:

In MATLAB, we use square brackets for defining a matrix. The following MATLAB code defines a matrix A with the above values. The matrix name is usually capitalized.

The entire content of the matrix is contained by the square brackets, and a semicolon is used to separate the rows.

In contrast to a matrix, a vector is a one-dimensional array of numbers or variables. An individual row or column of a matrix can be identified as a row vector or a column vector. Here is an example row vector B from matrix A, above:

Just like matrices, vectors are also entered into MATLAB with square brackets.

Note that no semicolons appeared within the square brackets for the definition of B. This was because B has only a single row.

You can refer to a particular element in a matrix by its row and column placement. So, for the matrix A, the element in the first row and third column is the number 8. These two values identifying an element within the matrix are called indices. Likewise, an element of a vector can be identified with a single index.

In MATLAB, indices can be specified using parentheses to select elements from matrices or vectors. For example,

A vector would need only one index.

Some matrices are special and can be categorized further. We will refer to these definitions in the following sections.

Square matrices are those matrices where both dimensions are equal. Square matrices in which only the values along the main diagonal are non-zero are called diagonal matrices. Here is an example:

Finally, diagonal matrices where all the non-zero values are 1 are termed identity matrices. The capital letter I is usually reserved in linear algebra for representing identity matrices. Here is an example of a 4×4 identity matrix:

Often in linear algebra, an identity matrix is referred to as “the identity matrix,” with dimension inferred from context, and subsequent sections will adhere to this convention. MATLAB has a special function for creating an identity matrix of any desired size: eye(n), where n is the dimension desired.

There is a reason MATLAB has its own function for the identity matrix. It plays a central role in linear algebra, as will become clear in the rest of this tutorial.

3.2.2 Transposition

One common operation on matrices is transposition. Transposition flips rows and columns; each row of the original matrix becomes the corresponding column of the new matrix. In mathematical notation, transposition is usually indicated with the superscript t. Here is how we would write the transposition of the matrix A defined in the previous section.

You carry out this operation in MATLAB by using the punctuation ’ after the matrix name: In this case, A’.

These preliminaries might seem excessive, but a precise nomenclature of operations matters a lot in linear algebra. It will soon become obvious why.

3.2.3 Addition

Addition is an operation that is defined for two matrices or two vectors of the same dimensionality. Adding matrices algebraically is adding corresponding components to form a new matrix. Thus, each of the two matrices or two vectors being added must contain only elements that correspond with those in the other. Because addition is defined only for cases where the two values being added have the same dimensionality, cases where the dimensionality differ, such as adding A from the previous section with its transpose, would be termed undefined or meaningless. But you don’t actually have to worry about this. MATLAB will literally not let you add matrices with differing dimensionalities; it will complain that there has been an error and that “Matrix dimensions must agree,” all red and bothered (unless you changed the color preferences).

Here is an example of matrix addition. We define the matrix F as

3.2.4 Scalar Multiplication

If this is starting to look to you like we are retracing our steps from elementary school, you would be right. All operations you learned in first grade for actual numbers have their corresponding operation for arrays in linear algebra (except for transpose; that wouldn’t make any sense for scalars, as each scalar is its own transpose, so we mercifully skipped that in elementary school; now you know). Note that if you add A to itself, as in Exercise 3.1, the resulting matrix has values double to those of the corresponding values in A. This suggests a simple definition for the scalar multiplication of a matrix. Indeed, when a matrix is multiplied by a scalar value, each element of the matrix is simply multiplied by that number.

In MATLAB, a scalar multiplication is performed with an asterisk, if one of the multiplicants is a scalar number.

3.2.5 Matrix Multiplication

So far, so simple. But this is the precise point where things get hairy and the majority of students get lost with linear algebra. This is because matrix multiplication is the first point where the analogy to elementary school math starts to break down. As you already learned elementary school math, this highly practiced cognitive template will start to interfere with learning this crucial step. We urge you to pay extreme attention to matrix multiplication and practice it as much as you can to override your strong cognitive priors. As most of linear algebra crucially hinges on matrix multiplication, this dire warning is not overstated. This is the point where you most likely will get lost, if you get lost. So proceed with the utmost care.

Multiplication can also be defined for two matrices or for two vectors. When you multiply two matrices together, AB, each element of the resulting matrix, C, is the sum of the corresponding row elements of A times the corresponding column elements of B. In other words, all elements of C may be obtained by using the following simple but perhaps counterintuitive rule (we are not big fans of rote memorization, but it pays to memorize this one by heart; otherwise, it will haunt you forever):

Here is an example with two square matrices C and D.

This definition constrains the dimensionality of the two matrices or vectors in a matrix multiplication. For two matrices A and B, the number of columns in A must match the number of rows in B for the product AB to be defined. Also, the dimensions of the product are m×n, where m is the number of rows in A and n is the number of columns in B. If you try to multiply “incompatible” matrices (in terms of dimensionality), MATLAB won’t let you do it and will inform you of this fact.

Matrix multiplication can occur between a vector and a matrix, provided that both meet the dimensionality constraints.

Observe that AB is not the same as BA, throwing off elementary school intuitions. Unlike scalar multiplication, matrix multiplication is not commutative; in general, matrices do not commute under multiplication. In the mathematical sense, commuting has nothing to do with traveling to your place of business. It simply means, as stated above, that AB is not the same as BA. It is extremely important to keep this property in mind when manipulating non-scalar values in algebraic equations.

In MATLAB, matrix multiplication also uses the asterisk like scalar multiplication, but both multiplicants are now non-scalars:

Much like in non-matrix multiplication where every number N has a reciprocal such that , matrix multiplication defines the concept of an inverse. However, unlike with scalar multiplication, only some matrices have inverses. We were not kidding when we mentioned that matrix multiplication is where the vanilla world of elementary school scalar multiplication is shattered.

The inverse of a matrix D, D⁻¹, is the matrix that, when multiplied with the original matrix, equals the identity matrix:

Note that this definition requires that the matrix D be square. This falls out from the constraints of matrix multiplication and the definition of the identity matrix as a square matrix.

So, for example, if we define D as

then its inverse D⁻¹ is

For your convenience, MATLAB provides a function inv(A), which calculates the inverse of a matrix.

As mentioned above, only square matrices have a defined inverse. Even among square matrices, not all have inverses. The MATLAB function inv() returns Inf in such cases. For instance, the matrix X below looks completely innocent, but alas, it does not have an inverse.

As MATLAB’s warning implies, such square matrices without inverses are termed singular. We will discuss criteria for assessing when a matrix has a defined inverse when we discuss determinants, in Section 3.2.6.

3.2.6 Geometrical Interpretation of Matrix Multiplication

In addition to linear algebra, there is also a corresponding geometrical interpretation of matrix-vector multiplication that can be extremely useful. First, see what happens when a vector is multiplied by a scalar. Suppose that

You can plot the vector B on the Cartesian plane if you assume that the x-component of the vector is the element in the first row and the y-component of the vector is the element in the second row. In such cases, we can define unit vectors and . Therefore, the vector B can be written in terms of simple and elementary unit-length component vectors as . This can be readily seen by substituting the definitions for and into the equation for B:

This decomposition in terms can be demonstrated in MATLAB as well.

This results in the graph shown in Figure 3.1.

Next, you can multiply the vector B by a scalar, 2, to get:

If you plot this new vector alongside B, then you get the graph shown in Figure 3.2.

Notice that multiplying a vector by a scalar changes only its length. It does not change the direction of the vector. Now see what happens when a vector is multiplied by a matrix.

Since the matrix A is square, the product of A and B has the same dimensions as the vector B (in this case, both are 2×1). Therefore, you can plot the vectors A * B and B on the same graph to obtain the result shown in Figure 3.3.

Here, you can see that multiplication of vector B by the matrix A has resulted in rotating B counterclockwise and stretching it out. Now, try another example, where A is the same, but:

If you plot B and AB on the same graph, then you get the result shown in Figure 3.4.

In this case, multiplication of the vector B by the matrix A is equivalent to multiplication of B by a scalar—in this case 3. It turns out that this scenario is a general one. For many square matrices A, there exist corresponding vectors B such that

where is a scalar constant. (So, in the previous example, .)

Geometrically, this means that for a given matrix A, there is a vector B that does not rotate when multiplied by A. The scalar is called an eigenvalue of the matrix A. The invariant vector B is called an eigenvector of the matrix A, and each eigenvector B is associated with a particular eigenvalue .

While we mentioned this concept in passing, it pays to do a full stop and truly appreciate the concept. It is even more fundamental to linear algebra than matrix multiplication itself. There are quite a few people who spend their days calculating eigenvalues of systems (represented by matrices). Even worse, there are plenty of intellectual posers who fail this basic academic shibboleth; they tellingly refer to them as “Igon values” instead (Gladwell, 2009). You don’t want to be that guy. Seriously, throwing buzzwords around is all fun and games, but we want you to actually understand them. Only if you understand these concepts can you meaningfully work with them, and we assure you that you will be dealing with eigenvalues and eigenvectors as long as you do linear algebra. And you will probably be doing linear algebra as long as you are doing science. As for how long you want to do science, that’s up to you.

Back to eigenvectors, there is actually a relatively simple visual interpretation. Imagine rotating a globe around its axis (or imagine the actual planet earth spinning around its axis on a daily basis). The values on this axis are rotation invariant: They do not change when the system is rotated. You can imagine that these are special values and it is important to know them, as they characterize in a way what the system as a whole (the spinning earth) is doing. If the system was doing something else, the values would be different.

We do believe that in addition to this visual you get the best appreciation for eigenvectors and eigenvalues not by reading or writing about them, but by working with them, which is exactly what we will do in the next few sections, where we will discuss how to determine eigenvectors/eigenvalue pairs for square matrices and their applications. However, before we can cover eigenvectors and eigenvalues, we need to discuss the determinants of matrices.

3.2.7 The Determinant

As discussed earlier, only some square matrices actually have defined inverses. The determinant is a value (defined only for square matrices) that aids in determining whether a matrix has a defined inverse or not. In addition, the determinant aids in identifying whether a matrix has eigenvectors as well.

The definition of the determinant for larger matrices is complex, and, for completeness, we refer the reader to a suitable reference. For 2×2 matrices, however, the determinant is a relatively simple expression. Defining the matrix A as

the determinant is defined as ad−bc, multiplying then subtracting the values on the two diagonals of the matrix. The determinant is written in linear algebra as det() around a matrix or as vertical bars. The following equation shows the two notations and the value of the determinant for 2×2 square matrices.

MATLAB provides a function det() that calculates the determinant of a matrix for you.

Now you know how to calculate it. But why would you want to? The value of the determinant can be used to determine (hence the name: It actually determines a range of other matrix properties as well) whether a square matrix has a defined inverse. A square matrix has a defined inverse if and only if the determinant of that matrix is nonzero (As we saw earlier that such matrices with zero valued determinants and no inverse are called singular.) This can be seen by attempting to determine the value of a matrix inverse analytically.

Let the matrices A and A⁻¹ be defined as

For A⁻¹ to be the inverse of A, AA⁻¹ must equal the identity matrix. This generates a set of equations in the elements of both matrices:

Ideally, we would want to represent the elements of A⁻¹ in terms of a,b,c,d; the elements of A. Starting with ce+gd=0, the value of e is

Substituting this value for e into the equation results in an equation using only one term from A⁻¹, g:

This yields an expression for the element g of A⁻¹ solely in terms of elements of the original matrix A, and this process can be repeated for the other three elements of A⁻¹. However, in rearranging terms, the equation was divided by the term , implying that this term must not be zero (as no one can divide by zero). You may recognize this term as the negative of the 2×2 determinant defined above, bc−ad. Thus, if the determinant is zero, the system of equations identified by the inverse has no solution. QED.

3.2.8 Eigenvalues and Eigenvectors

Recall that finding the eigenvalues and corresponding eigenvectors of a square matrix A is equivalent to solving for scalar and vector B such that

One valid but obviously degenerate solution to this equation is the zero vector,

regardless of the matrix A, as long as A is a 2×2 matrix. The zero-vector solution is called the trivial solution and will not be of interest here (it is rather of interest to philosophical discussions of mathematical conceptualizations of death). Thus, to limit solutions to the non-trivial solutions, we will require that any solutions for B not be zero vectors.

The eigenvector equation is , where I is the identity matrix. This can be rearranged as:

If the matrix has an inverse, then multiplying through this equation by the inverse gives:

Because a matrix multiplied by its inverse is the identity, this would imply that

which is exactly the trivial solution that you do NOT want, which means that must not have an inverse if nontrivial solutions for B exist. Recall from the previous section that a matrix has no inverse if its determinant equals zero. So, for , nontrivial solutions for B exist only if . QED, yet again. This equation is called the characteristic equation of the matrix A. It is the only equation you need to calculate the eigenvalues and eigenvectors of a matrix.

Through the characteristic equation, we can solve for the eigenvalues of A, , and then we can use the values of to determine the corresponding eigenvectors of A.

We use this now in an example calculation of eigenvectors and eigenvalues for the matrix

You can solve the quadratic equation for to get . These are the eigenvalues of the matrix A! You can solve for the corresponding eigenvectors as follows. For , the equation becomes

Substitute A into the preceding equation and let:

The preceding equation becomes:

Solving the system of equations gives y=2x. Thus, any B such that

is an eigenvector of the matrix A corresponding to the eigenvalue . Note that this demonstrates that any vector with the same orientation will be invariant to changes in orientation imposed by multiplication by A (recall the spinning globe).

In MATLAB, the command [V,D]=eig(A) will return two matrices: D and V. The elements of the diagonal matrix D are the eigenvalues of the square matrix A. The columns of the matrix V are the corresponding eigenvectors.

3.2.9 Applications of Eigenvectors: Eigendecomposition

Here we will describe a powerful theorem called the eigendecomposition theorem. This theorem states:

For any n×n matrix A with distinct eigenvalues you can write:

where V is the square matrix whose columns are the eigenvectors of A, and D is the square diagonal matrix formed by placing the eigenvalues of A along the primary diagonal of D.

Powerful stuff indeed. Note that the matrices V and D are exactly those matrices returned by the MATLAB function eig()!

This theorem allows any matrix A with distinct eigenvalues to be decomposed into a diagonal matrix. This decomposition is especially useful in cases where a matrix needs to be raised to a power:

Note here that each pair VV⁻¹ between diagonal matrices D is equivalent to the identity matrix and thus drops out of the equation.

Only the diagonal matrix D is raised to the power N. Recall that a diagonal matrix raised to a power N is exceptionally easily calculated (raise each element of the diagonal to the power N). Thus, raising a matrix with distinct eigenvalues to a large power becomes a far simpler calculation.

3.2.10 Applications of Eigenvectors: PCA

The eigendecomposition theorem can be used in many remarkable ways. In this section, we will explore one application, principal component analysis (which we will revisit with practical examples in Chapter 17 of this book). Principal component analysis provides a means of identifying the independent axes responsible for major sources of variability in a multivariate sample. Once these axes are identified, the axes can be used for classification and simplification of the data. For instance, if two dimensions capture all the variability inherent in 200 dimensions, the data can be simplified to the “loading” of the data on those two dimensions. This will become clearer later on.

Let X be a set of data represented as an m×n matrix X, where m is the number of data points and n is the number of dimensions in the data set. For this data, we can calculate an n×n covariance matrix . According to the eigendecomposition theorem, we can represent as . Under this reformulation, the eigenvectors form a new set of axes that indicate independent directions of variance. One can see this by a rearrangement of the equation:

Thus, through the rotation and scaling of the eigenvector matrix, the original covariance matrix can be transformed into a diagonal covariance matrix, eliminating covariance altogether.

Equally significantly, if the eigenvectors are normalized, then the eigenvalues indicate the relative contribution of each of the eigenvectors to the covariance matrix. For large datasets, the relative weights provided by the eigenvalues can be used to reduce the dimensions of the data.

We will use an example.

Load the data file data.mat.

You’ll notice that the included matrix is a 50×3 matrix of data, corresponding to 50 samples of a three-valued vector quantity. Because this data has more than two dimensions, visualizing this data is fairly difficult. We will use PCA to remap the data to new axes that better represent the variance of the data.

First, we can use the MATLAB cov() function to generate the covariance matrix:

Next, we will calculate the eigenvectors of C:

The value of D shows the three eigenvalues. Note that one of the eigenvalues is far smaller than the other two (1.601e−03). This indicates that the corresponding eigenvector (column 2 of V) only weakly contributes to the covariance matrix. As a demonstration of this, we will apply the eigenvector matrix V to the original data and examine the data:

Note that these variances of the modified data set match the eigenvalues of the original covariance matrix. Just as important, the variance corresponding to the second axis is substantially smaller than the other two. Because of this discrepancy, we can omit this axis in the rotated data set while still preserving the variance of the original data.

3.3.1 Introduction

The intent of this section is a brief, rapid introduction to probability and statistics and their use in MATLAB. This primer cannot hope to replace a good elementary statistics sequence. That said, those readers with a less extensive background in statistics may find this section useful. This primer expects a basic understanding of calculus and a passing familiarity with MATLAB, such as what might be expected by having gone through the introductory chapters of this very book.

3.3.2 Random Variables

Much of probability is built upon the concept of a random variable. A random variable is a variable that can take any one of a number of defined values and whose actual value is determined solely by chance. As a simple example, we will define a random variable X to represent the outcome of a flip of a coin, where the value 1 denotes an outcome of “heads” and a value of 0 signifies the outcome “tails.” With a fair coin, the probability of heads or tails is equal.

Usually, we will represent the probability of an outcome as a rational number, often a fraction. As a fraction, the numerator represents the number of outcomes that yield the event, and the denominator represents the total number of outcomes in the system. So, in the case of random variable X, the probability of a tails event is . There are two possible outcomes, and the event of getting “tails” is the result of only one. Similarly, the probability of a heads event is also . Together, the probability of a heads event or a tails event occurring is or 1. This should make sense, as flipping an idealized coin should yield one or the other.

This property of probabilities, summing to one, is a general one, and in a formal treatment of probability is usually defined axiomatically. This usually includes three axioms:

These three are also known as the “Kolmogorov axioms” and form the traditional axiomatic foundation of probability theory.

We can generalize the coin flip example by allowing the probabilities of the two outcomes to differ from . Under such a generalization, a random variable having two possible outcomes is called a Bernoulli random variable. Unlike the case of X, where we modeled a coin flip, a Bernoulli random variable does not necessarily have equal probabilities for the two outcomes. The probabilities for both must, however, sum to one.

Given a Bernoulli random variable Y, with probability p of outcome 1 and probability (1−p) of outcome 0, we denote the probability of an outcome 1 of Y as Pr(Y=1). Here, Pr(Y=1) would be equal to p. We can define a function f(y)=Pr(Y=y) such that the value of f(y) is the probability of value y of random value Y. In other words,

This function is termed the probability mass function (PMF) of random variable Y. It literally outlines where the mass of the probability of the variable lies.

Thus far, our example has focused on a single Bernoulli variable representing a single binary outcome. As a more complex example, we can flip a coin multiple times and count the number of heads. This can be represented as a sum of Bernoulli random variables. We can also define a new type of random variable, a binomial random variable, to represent this scenario.

Formally, given a series of n Bernoulli random variables , all with equal probability p of outcome 1 , a binomial random variable Y represents the total number of positive (i.e., 1 valued) outcomes. We say here that n represents the number of trials. Since each trial must have either a positive or negative (i.e., zero-valued) outcome, the total number of positive and negative outcomes must equal the number of trials, and the number of negative (i.e., zero-valued) outcomes is n−Y.

Much like with Bernoulli random variables, we can define a probability mass function. However, because a binomial random variable has more outcomes, this case is more complex. Take a series of three coin tosses and a random variable Y representing the total number of heads the series of flips (see Table 3.1).

There are now eight possible outcomes. This can be calculated quickly from 2³=8. (Each flip occurs independently of the others and doubles the number of outcomes in the series. Thus, with 3 flips in the series, the total number of outcomes is 2×2×2=8.) Of these 8, only one outcome involves 0 or 3 heads. 1 or 2 heads both involve 3 outcomes. With this information and from this table, we can construct a probability mass function for Y.

Values below 0 or above 3 were omitted as they are necessarily 0 (these already sum up to 1). In general, the probability mass function for a binomial random variable can be calculated from the formula

Here, n is the number of trials, p is the probability of a positive outcome on any one trial, and k is the number of successes or nonzero-valued results. The notation might look scary due to its unfamiliarity, but it is simply the number of combinations, also called the binomial coefficient, and it can be calculated from . The MATLAB function C=nchoosek(n, k) will calculate the number of combinations automatically for you:

For more complex distributions, a bar graph provides an excellent tool for visualizing a probability mass function. Figure 3.5 depicts the probability mass function for the probability of the total number of heads for eight coin flips. With the probabilities in the vector p, the command

produced the plot shown in Figure 3.5. (The expression 0:8 generated the markers for each bar at the bottom of the plot, denoting the number of successes.)

As n increases, you may notice that the probability mass function acquires a bulging shape, where success counts near the middle of the possible range have much higher probabilities relative to the probabilities of all heads or all tails. Descriptive statistics provides a number of standard terms that we can use to characterize the distribution.

We can describe the central tendency of the distribution. We will discuss three common ways of describing the central tendency of a distribution. The first is the mode. The mode is defined as the most probable outcome in the distribution. From a bar graph depicting a probability mass function, the mode is the outcome with the highest probability. The second central tendency is the median. The median is defined as the outcome corresponding to the point where the probability masses above or below the outcome are equal. This can also be stated as the outcome for which the cumulative probability is equal to or exceeds 0.5.

The final central tendency that we will discuss is the mean. Occasionally, the term expected value is also used to indicate the mean expected value. This term suggests an interpretation for the mean, given a binomial random variable Y drawn from a known distribution, what value should you expect? We can define the expected value of a function f(x) relative to a distribution for a random variable X as

or, the expected value of a function f(x) is the value of the function at x multiplied by the probability of x, summed over all values x for the random variable X. The mean is defined as the expected value of the function f(x)=x. So, the mean of the three coin toss example discussed previously is

Clearly, the mean is not a valid number of successes (we cannot have a fraction of a coin flip). The mean is not guaranteed to be a valid count for the variable in question. Nonetheless, the empirical average of many trials or random variables will grow ever closer to the true mean of distribution, assuming that all trials originate from the same underlying distribution and are statistically independent. This property, that the empirical average over many trials will approach the expected value, is called the Central Limit Theorem.

We can easily demonstrate that the mean for any Bernoulli random variable with probability p of outcome 1 is p:

As mentioned earlier, the mean provides a measure of the central tendency. What the mean doesn’t provide is a measure of the dispersion of the data. One group of data may be widely dispersed, and another may be tightly clustered, but both may have very similar means.

Unfortunately, we cannot simply use the sum of the differences between the random variable and the mean as a measure of dispersion, as the positive and negative deflections around the mean tend to cancel each other out, tending toward an expected value of zero:

Instead, we can calculate the expected value of square of this difference (as they don’t cancel out), called the variance:

Thus, the variance is the difference between the expected value of the random variable squared and the square of the mean. Unlike the expectation of the difference between the random variable and the mean, the variance is rarely zero. Because the variance is in units equivalent to the square of the random variable, we will often use the standard deviation instead, which is defined as the square root of the variance. The Greek letter sigma is often used to represent standard deviation:

By definition, sigma squared represents the variance of the random variable.

3.3.3 The Poisson Distribution

The Poisson distribution is used to describe phenomena that are comparatively rare. In other words, a Poisson random variable will relatively accurately describe a phenomenon if there are few “successes” (positive outcomes) over many trials.

The Poisson distribution has a single parameter, . For a Poisson distribution modeling a binomial phenomenon, can be taken as an approximation of np.

Aside from use as an approximation for the binomial distribution, the Poisson distribution has another common interpretation. For an infrequently occurring event, the parameter lambda can be viewed as the mean rate, or , where n is the mean events per unit time, and T is the number of time units. In such a case, a Poisson distribution with the appropriate parameter will approximate the distribution of events over time or the number of events in an interval.

Events whose occurrence follows a Poisson distribution have another interesting property. Given a series of Poisson distributed independent random variables and their corresponding arrival times , we can calculate the distribution of the corresponding inter-event intervals.

Let N(t) equal the number of events at some time t, where P(k=N(t)) follows a Poisson process with parameter . Then, the probability of the nth event occurring at

Likewise, we can calculate a distribution for the inter-event intervals. Here, the probability of an interval between two successive events T_k and T_k−1 being larger than some time t is the same as the probability of exactly k−1 events occurring within the interval 0 to . This can be expressed as follows:

The probability of k−1 events during the interval is equivalent to the probability of no events during the interval 0 to t.

By definition, k−1 events occurred during the interval 0 to T_k−1. Thus, if any events occur during the interval T_k−1 to , this would mean that the number of events in the interval would not be k−1 but greater than k−1.

This implies that the intervals are distributed in a “memoryless” fashion. In other words, for an ongoing process following a Poisson distribution of events, the distribution of waiting time to the next event does not change over time. Put differently, knowing when the last event occurred does not give you any information about when to expect the next one.

The distribution of intervals that we have derived here is called the exponential distribution. This is our first example of a continuous distribution. Unlike with discrete distributions, calculating the exact probability of a single value in a continuous distribution is not feasible, as it is always zero. To understand why, one can use the previous example of the exponential distribution of time intervals for a Poisson process. The probability of a specific value, say 3 seconds, would correspond to the probability of the time interval being exactly equal to 3 seconds. Since this would exclude any interval even infinitesimally close to 3, this will be vanishingly small regardless of the number chosen. Therefore, when working with continuous probabilities, we compute the probability of a variable falling within a range of values.

Because of this fundamental difference, continuous distributions do not have a probability mass function like discrete distribution. Continuous distributions are defined in terms of cumulative distribution functions. The derivation of the exponential distribution above provides an excellent example. Above, the probability of an inter-event interval T being greater than some value t was found to be equal to . Usually, the cumulative distribution function F is defined as the probability of a random variable being less than a given value. Following those conventions, the cumulative distribution function F for the exponential distribution can be defined as

(This falls out of the requirement that the sum of probability be equal to one. If .)

To determine the probability of a random variable falling within a specific range, we can subtract ranges. For example, the probability of a random variable T falling between t₁ and t₂ can be expressed in terms of each value alone:

This holds true because the probability of the random variable T being less than t₂ includes all cases where the random variable is less than t₁. So, the probability mass corresponding to must be subtracted out.

This can be more clearly understood by introducing the probability density function, a function f(x) such that the cumulative distribution function . For a probability expression with a density function f(x), the probability . Likewise, the range .

Continuous distributions also have expectations like discrete distributions. Instead of summing over all probabilities, the expectation is defined in terms of an integral over the probability density function. For a probability density function f(x), the expectation of the function g(x) is defined as

With this, we can calculate the mean and variance for the exponential distribution:

3.3.4 Normal Distribution

The last classical distribution that we will discuss here is the normal distribution. There are many more, some of which will be visited in later chapters for more specialized purposes. Because its cumulative distribution function is not solvable analytically, the normal distribution is usually defined in terms of its probability density function,

Here, the parameters μ and σ define the mean and standard deviation of the distribution. A normal distribution with μ=0 and σ=1 is called a standard distribution.

The cumulative distribution function of the normal distribution is the integral over all values x:

The cumulative distribution function of the standard distribution is often denoted as . This cumulative distribution function is often defined in terms of another special function whose form is very similar to the integral over the probability density. This function is commonly called the error function . So, in terms of the error function, the cumulative distribution function of the standard distribution is .

MATLAB defines both a cumulative distribution function, normcdf(x, mu, sigma), and the error function, erf(x), for the normal distribution.

The normal distribution approximates how many phenomena vary. In particular, the normal distribution is useful in understanding error.

3.3.5 Confidence Values

The normal distribution is particularly useful because of the central limit theorem. Given N independent, identically distributed random variables with mean and variance , the central limit theorem asserts that the distribution of the mean of the random variables will converge to a normal distribution with mean and variance . The dependency of the variance on the number of variables (in this case, samples) is particularly important. As we will see, this result is especially relevant to estimating distributions from samples.

The term standard error of the mean (or just standard error) is defined as

where s is the estimate of the standard deviation and n is the number of samples in the estimate.

Often, the error will be expressed as confidence intervals around the mean. A confidence interval is expressed as the interval surrounding the mean within which an estimate of the mean should fall with a certain probability (often 90% or 95%). For a 95% confidence interval, this is approximately 1.96 times the standard error on either side of the mean estimate.

For example, let’s assume we have a normally distributed population whose actual mean is 25 and whose variance is 5. We can collect a sample of 10 as follows.

In this case, the 95% confidence interval around the mean estimate 25.5606 would be 24.1473 to 26.9739.

For values other than 95%, we can calculate the factor of the standard error directly using the MATLAB function erfinv(). erfinv() calculates the inverse of the error function discussed earlier. To determine the factor to replace the 1.96, you will need to calculate , where p is the confidence interval probability. It is important to note that this assumes normally distributed values.

3.3.6 Significance Testing

Hand in hand with the idea of a confidence interval is significance testing. Take a known distribution: a normal distribution with a mean of 15 and a standard deviation of 3. A sample of five values has a mean of 11. Is this sample likely to be drawn from the same population? How about a sample of 100 values with the same mean? There is always a chance that the sample was drawn from the distribution. The question is, with which probability? Significance testing provides systematic methods for answering such questions.

Returning to the original question posed about the estimated sample mean of 11, we can describe this in a probabilistic way; in fact, we can rephrase this probabilistically in at least two ways. First, we can ask how probable a mean of 11 or lower might be, or

Secondly, we can ask how probable a mean at least as extreme as 6 might be, or

Classical statistics distinguishes these two refinements of our original questions as a hypothesis test: we use the properties of the distribution to test a null hypothesis (often written as H₀) that the five item sample is drawn from the known distribution. An extremely low probability of such an extreme result would argue against the null hypothesis or, alternatively, for rejecting the null hypothesis.

Typically, a maximum threshold for the probability is chosen, called the significance level. Common values are 1%, 5%, and occasionally 10%. Outcomes with probabilities below the significance level are termed statistically significant at the corresponding level, and usually strongly argue for rejecting the null hypothesis. It is important to keep in mind that hypothesis testing is only evidence for or against rejecting the null hypothesis. For example, a significance level of 5% indicates that only one out of every twenty repetitions would produce a result as extreme. If an experiment is repeated 20 times, on average, the outcome would be statistically significant once. All that the p value gives you is the probability that such data so extreme (or more extreme) could happen by chance, assuming that the null hypothesis is true. By this logic, if the significance level is not met, it does not mean that the null hypothesis is true, just that we failed to reject it at this significance level.

Often, a significance level is selected prior to analysis or even the collection of data, and a significance of 5% is especially common. The selection of a significance level requires a tradeoff between two types of error. Choosing a less stringent significance level increases the risk of interpreting a result as indicating that the null hypothesis should be rejected when it’s not actually false (this is classically called a Type I error and refers to spurious findings). Alternatively, a more stringent significance level enforces a more severe threshold for the rejection of the null hypothesis, but setting too low a significance level can miss rejection when the null hypothesis is actually false (a classical Type II error: missing differences that are really there). How one should pick the significance level depends on the relative value of the outcomes in a given practical case: how serious is it to miss real effects versus how serious it is to claim the existence of effects that are not really there. More than the brief treatment of Type I/II errors here is beyond the scope of this primer, and the reader is referred to a more detailed reference for an in-depth discussion.

Significance tests can be classified as one-tailed or two-tailed hypothesis tests. The origin of these names can be easily understood from an illustration of the expected distribution of sample means. From the central limit theorem, as discussed in the last section, we know that sample means from the known distribution should vary with a normal distribution whose mean matches that of the underlying distribution and whose standard deviation is .

Figure 3.7 shows the expected PDF (remember, probability distribution function) for sample means for samples with five elements. Shaded is the probability of the sample mean having a value at least as extreme as the collection value. Figure 3.8 shows the shaded portion of the PDF in greater detail. (Noting that may help in understanding Figure 3.8.)

Figure 3.9 shows the same PDF with the probability of the sample mean being less than 11 shaded. In this case, the shaded probability covers only one of the two ends of the PDF. This is a one-tailed test. Likewise, the previous case covering both tails of the PDF is called a two-tailed test.

Using significance testing correctly requires determining whether the question at hand involves a one-tailed or two-tailed test. Here we are interested in ascertaining whether the measured sample comes from the known distribution. Understanding the extreme nature of the sample mean is what we’re interested in, so a two-tailed test is most appropriate.

Since the sample mean should be distributed according to a known normal distribution, we can calculate the two-tailed probability using the MATLAB function normcdf. Recall that normcdf(x, mu, sigma) returns the cumulative distribution function at x for a normal distribution with =mu and =sigma.

First, ,

Next, we can calculate . normcdf() can be used for this as well, but if the same procedure is followed with 19 substituted for 11, we will calculate :

Note that the value here is substantially larger than the probability of the first tail. To calculate , we can use the equality :

This matches the probability mass of the first tail, as one might expect from Figure 3.9. The sum of the two is the probability that the sample mean is at least as extreme as the estimated mean here:

So, roughly 3% of the time, the empirical mean of five samples from the known distribution would be at least as extreme as 11. At the 1% significance level, this would not provide sufficient support for rejecting the null hypothesis, but it would at the 5% level.

3.3.7 Linear Regression

Assume two variables x and y, with an expected linear relationship between them such that . Under this relationship, we will call the y the dependent variable and x the independent variable. Let’s say that we have collected sample pairs and want to estimate the parameters (constant offset) and (slope) so that we can express y as a function of x.

This estimation procedure is called linear regression (think “prediction”). Here, we are regressing x onto y. Under traditional terminology, as the independent variable, x is the regressor (plural: regressors). We can approach the estimation problem as a search for values of (we will call these estimates ) that minimize the distance between the predicted value and the actual measured value . The distance for each measured pair can be represented as

Then, the estimates can be calculated from the partial derivatives of S:

The estimate can be calculated in the same manner:

This has yielded a pair of equations in . The equation for the estimate of can be substituted in and a closed form solution for obtained:

This method of linear regression is called least squares optimization, because the estimates originate from optimizing (here, minimizing) the sum of the squared distance.

Once an estimate is calculated, one should look at the residuals, the difference between the values predicted by the estimated parameters in the regression equation and the measured values. Assuming that the estimates for are stored within MATLAB variables a and b respectively, the residuals for a single variable can be calculated with

After any linear regression, it is important to review the residuals. In a perfectly linear relationship between dependent variables and regressors, the residuals will be randomly distributed (and ideally small). This implies that the difference between the prediction and measured value is primarily the result of error and not an additional nonlinear relationship.

After fitting, we can also look at the coefficient of determination, r².

Here the numerator is the sum of the squares of the residuals, and the denominator is the sum of the squares of the difference between the dependent variable and its mean, essentially N times the var(y). Thus, given estimated parameters in MATLAB variables a, b, we can calculate r² with

The coefficient of determination ranges from 0 to 1, with values near 1 indicating a better fit to the data. One interpretation of r² is the proportion of variance explained by the model. Thus, a value closer to 1 indicates that most of the variance in the dependent variables originate in the variation of the regressors, as propagated through the model. A lower r² implies that the dependent variables have some variance unaccounted for by the model.

Instead of calculating the estimates and coefficient of determination by hand, we can use the MATLAB function regress() in the Statistics Toolbox.

regress can perform multivariate linear regression, so for a single variable, x should be an N×2 matrix, where N is the number of observations. The first column data are the single variate observations, and the second column data consists of ones to indicate the constant offset. regress() returns a vector b with the estimates for the coefficients, . regress() also returns 95% confidence intervals for the coefficients in bint. regress() will also return the residuals if a third return parameter, r, is included.

3.3.8 Introduction to Bayesian Reasoning

Briefly put, Bayes’ theorem allows you to invert conditional probabilities if the base rates (or priors) are known. Illustrating the perils of eponyms, Thomas Bayes, a Presbyterian minister who first worked out the basic idea (but never published it), would probably be surprised by the attribution, in light of his other work.

When would one want to invert conditional probabilities? Surprisingly often. As a matter of fact, an awareness of Bayes’ theorem is likely much more useful in contributing to everyday wisdom in decision making than, say, calculus (which doesn’t stop the professionals from making up contrived examples anyway). This is due to the fact that it is often much easier to measure the conditional probability of one, but not another (yet related) event. If this appears too abstract, it is. A plastic example might help.

A country that shall remain unnamed has suffered a recent spate of vicious and unprovoked terrorist attacks on civilians. You have been hired by the government of this country to advise on a rational response to this unacceptable barbarism. Specifically, the question is whether the government should implement profiling measures against the known characteristics of the perpetrators, and if so, to what degree.

Here are the known facts:

And that’s all she wrote. In reality, having solid numbers on this is probably a better position than most governments can manage, so you are in a strong position.

What you really would like to know is the probability that someone is a terrorist given that he is bearded, but you only have the probability that someone is bearded if you already know he is a terrorist. This doesn’t help. If you already know someone is a terrorist, they might either already have committed their heinous act, or are in hiding. Language can be misleading here. Even though the probability that someone is bearded is 1, given that they are a terrorist, being bearded and being a terrorist are not synonymous in this case. Intuitively, if they were, there would have to be many more terrorists around (more than 1 in a million), given that 1 in 5 people is bearded. But how much is the risk of being a terrorist increased, given that someone is bearded?

Here, Bayes comes to the rescue.

In other words, the probability is 5 millionth. To put this in perspective, the absolute probability that someone is a terrorist if he is bearded is extremely low, while the relative probability is 5 times higher. This makes sense, as we use strong diagnostic information to link the two (as p (Bearded|Terrorist) is 1) and the base rate of bearded people in the country is 1 in 5.

You can now either advise the government that each bearded person should be scrutinized 5 times as much as a non-bearded person (to match the increased risk), or forgo profiling altogether, as the absolute risk is still so negligibly small. Both responses are rational.

What if the probability of being bearded given that someone is a terrorist was only half as strong—0.5? In other words, there are other kinds of terrorists around, they are not all bearded.

Plugging in the numbers yields

This makes sense. As the strength of the diagnostic information declined, the value of the criterion of beardedness to indicate terroristic tendencies declined in kind.

What if beardedness was much rarer; say only one in a hundred thousand people is bearded (and the strength of the link was back to 1)?

Now there is a 10% absolute chance that the person is a terrorist, just by virtue of being bearded, and the a priori chance is increased a hundred thousand-fold. Now, this seems like a rational case for strong profiling measures.

This admittedly somewhat contrived example was used to provide an intuitive feel for Bayesian statistics, using a striking and emotional case study.

While it is unlikely that you will be hired by a government in this position, it is far from unlikely that you will encounter Bayesian reasoning in your study of the neural and cognitive sciences. As a matter of fact, it has been suggested that the entire sensory apparatus of the brain works as one giant Bayesian machine. It would make sense that it might, as Bayes’ theorem allows making use of previous experience (establishing base rates) in a rational fashion. It has been shown that people do use prior experience to gauge what to expect from future interactions with members of a given class.

Moreover, the value of Bayesian reasoning is obvious in everyday life. It helps to know it. “Experts” might give misleading answers. For instance, if you go to the doctor for an AIDS test or a mammogram, you are not interested in the probability that the test is positive if you have the disease. You are interested in the probability that you have the disease if the test is positive! It has been conclusively shown that doctors frequently confuse these two probabilities, and give you wildly inaccurate odds (Gigerenzer and Hoffrage, 1995; Hoffrage and Gigerenzer, 1998). This concludes our introduction to Bayesian Reasoning—as well as our Mathematics and Statistics Tutorial for the purposes of this book. If you are interested in specific applications of Bayesian Reasoning in neuroscience, see chapter 22. For a more extensive treatment of Bayes’ theorem and Bayesian inference, see MacKay (2003).

3.3.9 Outlook

There is an almost infinite number of concepts we could add at this point. We could discuss other distributions, such as the Chi-squared distribution. We could take Bayes one step further and talk about likelihood modeling. However, this will suffice as a conceptual introduction to mathematical and statistical fundamentals. Some of these issues, e.g., maximum likelihood estimations, will be revisited in later chapters where appropriate. For now, we feel that enough groundwork has been laid, and that—if you have been working with us through this material—it is solid enough to carry you through the next chapters, which is really all anyone can ask for.

eye

inv

det

eig

cov

nchoosek

mean

var

std

normrnd

normcdf

erf

scatter

erfinv

fcdf

regress