We can add to the structure of a vector space by defining a scalar or inner product. Such a product is not a true vector multiplication, since to every pair of vectors it associates a scalar rather than a third vector. For example, in ${ℝ}^2$, we can define the scalar product of two vectors x and y to be ${\mathbf{x}}^T\mathbf{y}$. We can think of vectors in ${ℝ}^2$ as directed line segments beginning at the origin. It is not difficult to show that the angle between two line segments will be a right angle if and only if the scalar product of the corresponding vectors is zero. In general, if V is a vector space with a scalar product, then two vectors in V are said to be orthogonal if their scalar product is zero.

We can think of orthogonality as a generalization of the concept of perpendicularity to any vector space with an inner product. To see the significance of this, consider the following problem: Let l be a line passing through the origin, and let Q be a point not on l. Find the point P on l that is closest to Q. The solution P to this problem is characterized by the condition that QP is perpendicular to OP (see Figure 5.0.1). If we think of the line l as corresponding to a subspace of ${ℝ}^2$ and $\mathbf{v}=OQ$ as a vector in ${ℝ}^2$, then the problem is to find a vector in the subspace that is “closest” to v. The solution p will then be characterized by the property that p is orthogonal to $\mathbf{v}-\mathbf{p}$ (see Figure 5.0.1). In the setting of a vector space with an inner product, we are able to consider general least squares problems. In these problems, we are given a vector v in V and a subspace W. We wish to find a vector in W that is “closest” to v. A solution p must be orthogonal to $\mathbf{v}-\mathbf{p}$. This orthogonality condition provides the key to solving the least squares problem. Least squares problems occur in many statistical applications involving data fitting.

Figure 5.0.1.