The previous chapter explored linear algebra and matrix operations. This chapter can best be characterized as concerned with the linear algebra of covariance and correlation matrices. Principal component analysis (PCA) and factor analysis (FA) are two classic methods of identifying structures in the correlations of datasets. Despite the fact that they are concerned with covariances and correlations, many statisticians have very limited experience with these methods, because they make heavy use not only of statistics, but also of linear algebra; therefore, they straddle the realms of both statistics and engineering.
In this chapter, the following topics will be discussed:
- A primer on correlation and covariance structures
- Principle component analysis
- Basic exploratory factor analysis
- Advanced exploratory factor analysis
In the case of both PCA and FA, it is worth taking a moment to review some basic mathematical properties of covariances and correlations. Covariance is a measure of the linear codependence of two variables. Correlation is the covariance divided by the product of the standard deviation of the two variables. Thus, the correlation is a scaled covariance. In this chapter, we will be placing these covariances or correlations into matrices, called covariance matrices or correlation matrices.
The correlation matrix is a square matrix that has as many rows (and as many columns) as there are variables. Each element of the matrix represents the correlation between two variables. For example, in a dataset with three variables, A, B, and C, the correlation matrix would be as follows:
The diagonals are all 1 because the correlation of a variable with itself is 1. In a covariance matrix, the variances of the variables fall along the diagonal.