3.3. Least Squares Estimation

A natural criterion to obtain some meaningful estimators of B is to minimize with respect to the matrix B = (β₁:β₂:...:β_p). This is merely the sum of the squared deviations from the corresponding means over all observations and over all responses or dependent variables. This criterion is the same as that of minimizing tr(Y − XB)′(Y − XB), the trace (the sum of the diagonal elements) of the p by p matrix (Y − XB)′(Y − XB), resulting in the system of normal (matrix) equations

and yielding

as the least squares estimator of matrix B. It means that β_i, the i^th column of B, which estimates β_i, the i^th column of B, is given by

It is easy to demonstrate that B given by Equation 3.4 is unbiased for B, that is, E() = B. Further, covβ_i,β_j = σ_ij(X′X)⁻¹ for all i, j = 1,..., p. In addition, under the assumption of the model in Equation 3.2, is the Best Linear Unbiased Estimator (BLUE) of B in the sense that it has the smallest total variance among all linear unbiased estimators. By total variance we mean the sum of the variances of all elements of the matrix used as the estimator.

When the matrix X is not of full rank, a least squares solution to the system of normal equations, Equation 3.3, is given by

and

The matrix defined in Equation 3.5

• is not unique,

• depends on the particular choice of the generalized inverse used, and

• merely represents one of the many solutions to a singular (that is where at least one linear equation is redundant and is implied by the others) system of normal equations.

In this sense, is really not an estimator and one or more components of may be biased for their counterparts in B. However, as indicated in Searle (1971), certain linear functions of B can still be uniquely estimated. Such functions are called the estimable functions. Specifically, as shown by Bose (1951), and Searle (1971), a linear function c′B, where c ≠ 0 is a nonrandom (k+1) by 1 vector, is estimable if and only if

Accordingly, a linear hypothesis on the regression parameters will be a "testable hypothesis" if and only if it involves only the estimable functions of B.