Linear algebra has been described as the mathematics of computer science, and this chapter will be a bit different from prior chapters. Prior chapters discussed topics such as regression and statistical significance tests, and techniques that can directly be applied to a dataset to produce a solution of interest. A single linear algebra technique in isolation rarely provides a solution of interest to a substantive researcher. However, many numerical analysis techniques rely on linear algebra and matrix operations, making them an important part of scientific computing.
In this chapter, we will discuss the following topics:
- Matrix properties
- Mathematical operations on matrices
- Matrix inversion
- Solving linear systems
- Eigenvalues and eigenvectors
- LU decomposition
- Singular value decomposition of a matrix
- Choleski decomposition of a matrix
- Outer products
- Applications of linear algebra using R
In mathematics, matrices are simply an organized table of numbers. The reason they are used is because scientists and mathematicians have found that some numerical problems (generally systems of equations) can be solved algorithmically when the data placed into the problem is organized this way. In mathematics, a row or column of a matrix is termed a "vector", though R has a vector data structure that is not the same thing as the row or column of a matrix. The numbers characterizing a vector in mathematics are thought of as the endpoint of that vector with vector's origin at the origin of the coordinate system (this is what we term a "vector" mathematically, but it is really a point).
While the term "matrix" is often used to refer to a rectangular arrangement for string data (that is, a table), we must be much more cautious with our use of the term in numeric computation since R has more than one way to store composite variables.
While data analysts often load a dataset from an external file into a data frame, which looks like a matrix at first glance, a matrix has a number of defining features, mentioned as follows:
- Matrices can store the data of only a single data type (for example, we can't mix floating point numbers and categorical data in a single matrix)
- Matrices are more memory-efficient in R
- Data frames allow us to call values by the name of the variable, but they are a special instance of the list rather than a matrix
- Matrix operations tend to require a matrix as input and do not allow data frames
- R's matrix operations are often one hundred times faster than data frame operations
To convert a data frame into a matrix, we use the as.matrix function, which will require that all data elements be of the same data type.
A vector data structure in R means something slightly different than a vector data structure in mathematics. In mathematics, a vector can either be a column vector or a row vector, but in R, a vector has no dimensions. Let's take a look at the following examples:
> a <- c(1,2,3,4)
> b <- matrix(a, nrow = 1)
> a
[1] 1 2 3 4
> b
[,1] [,2] [,3] [,4]
[1,] 1 2 3 4 As we can see, both a and b store the same values, but R recognizes columns and rows for b, something that it does not do for a. However, if we were to apply a in a linear algebraic operation, R would treat it as a column vector.
We typically refer to a matrix as an m x n data structure with m rows and n columns. There a few different kinds of matrices that we will define, as follows:
- Rectangular: Arguably, all matrices are some type of a rectangular matrix. The matrix command defines a square matrix with the number of specified rows and columns. To create a two-row matrix filled with the numbers 1 to 24, we will use the following code:
matrix(c(1:24), nrow = 2)
Alternatively, we can also specify the number of columns as follows:
matrix(c(1:24), ncol = 12)
- Square: This is a rectangular matrix in which m is equal to n. Many commonly used matrix operations can only be done on a square matrix.
- Diagonal: This is a square matrix in which all elements are zero, except for the elements along the diagonal. R has a special function to create diagonal matrices, in which we specify the values to the diagonal, and the number of rows and columns. R will apply this command even if column and row dimensions do not match, as follows:
diag(c(1:3), 3, 3)
- Triangular: The elements either above the diagonal or below the diagonal (but not necessarily including the diagonal) are all equal to zero. There are upper and lower triangular matrices that refer to whether the non-zero side is the upper or lower side, respectively. We will cover triangular matrices in detail in the Triangular matrices section in this chapter, shortly.
- Symmetric: This is a matrix in which each (m, n) element is equal to the corresponding (n, m) element (that is, reversing the rows and columns of the matrix will yield the same matrix).
- Identity: This is a diagonal matrix in which the diagonal is filled with 1s. Any matrix multiplied by its identity matrix will be equal to the original matrix (that is, multiplying a matrix by its identity matrix is like multiplying a number by 1). An identity matrix can be easily created in R, using the
diag()function and passing only the dimensions to it as follows:diag(5)
- Vector: Mathematically speaking, this is a matrix with either a width of one or a height of one, termed column matrix and row matrix, respectively. In R, a vector can be created either as a one-dimensional matrix or using the R's
c()function, which will coerce all elements to be of the same data type. - Sparse matrix: This term refers to a matrix that is mostly composed of zeros. Alternatively, a dense matrix is mostly filled with values. These matrices are commonly used as design matrices for mathematical functions, which will be explained in further detail in the following sections.