Chapter 10
Partial least squares
This chapter provides a detailed analysis of PLS and its maximum redundancy formulation. The data models including the underlying assumptions for obtaining a PLS and a MRPLS model are outlined in Sections 2.2 and 2.3, respectively.
Section 10.1 presents preliminaries of projecting the recorded samples of the input variables, 
onto an n-dimensional subspace, n ≤ nx, and show how a sequence of rank-one matrices extract variation from the sets of input and output variables x0 and 
, respectively. Section 10.2 then develops a PLS algorithm and Section 10.3 summarizes the basic steps of this algorithm.
Section 10.4 then analyzes the statistical and geometric properties of PLS and finally, Section 10.5 discusses the properties of MRPLS. Further material covering the development and analysis of PLS may be found in de Jong (1993); Geladi and Kowalski (1986); Höskuldsson (1988, 1996); ter Braak and de Jong (1998); Wold et al. (1984) and Young (1994).
10.1 Preliminaries
In a similar fashion to PCA, PLS extracts information from the input and output data matrices, 
and 
by defining a series of rank-one matrices
10.1b 
The data matrices store mean-centered observations of the input and output variable sets, that is 
and 
with 
and 
being mean vectors. In the above equation, 
and 
are the rank-one matrices for the input and output matrices, respectively, the n vectors 
are t-score vectors which are estimated from the input matrix, 
and 
are estimated loading vectors for the input and output matrices, respectively, and 
and 
are residual matrices of the input and output matrices, respectively. It should be noted that the residual matrices have a negligible or no contribution to the prediction of the output data matrix.
To establish (10.1a), the PLS algorithm determines a sequence of parallel projections, one sequence for the observations stored in the input matrix and a second sequence for the observations stored in the output matrix. Reformulating (10.1a)
10.2a 
10.2b 
and defining
10.3a 
10.3b 
allows determining the sequence of projections for the input and output variables
10.4b 
Here, 
is the u-score vector of the output matrix, and 
and 
are the weight vectors for the input and output variable sets, respectively. Finally, according to (10.4a), the score variables, tk and uk, are given by
The set of weight and loading vectors, wj, qj, pj and 
, as well as the set of score variables, tj and uj, make up the jth latent variable set.
10.2 The core algorithm
PLS determines the score variables tj and uj such that they maximize an objective function describing their covariance, which is subject to the constraint that the projection vectors wj and qj are of unit length
where 
. The optimal solution for the objective function in (10.6) is given by
10.7 
which have to be solved simultaneously. This yields
Note that 
. Equation (10.8) also confirms that the two Lagrangian multipliers are identical, since
Hence, 
. Combining the two expressions in (10.8) gives rise to
10.10b 
The weight vectors wj and qj are therefore the dominant eigenvectors of the matrix expressions 
and 
, respectively. The score vectors for E(j) and F(j) can now be computed using (10.4a).
After determining the weight and score vectors, the next step involves the calculation of a regression coefficient between the score variables tj and uj. It is important to note, however, that the determination of this regression coefficient can be omitted, as this step can be incorporated into the calculation of the 
-loading vector, which is proven in Section 10.4. For a better understanding of the geometry of the PLS algorithm, however, the introduction of the PLS algorithm here includes this step. Equation (10.11) shows the least squares solution for determining the regression parameter
where 
. The final step to complete the determination of the jth set of latent variables requires formulation of objective functions for computing the loading vectors
The solutions to (10.12) are
10.13 
Working out the relationships yields
Before computing the (j + 1)th set of LVs, (10.5) highlights that the contribution of the jth set of latent variables must be subtracted from e(j) and f(j)
10.15b 
It should be noted that substituting (10.14) into (10.15a) gives rise to
10.16 
which, however, requires the 
-loading vector to be determined as follows
It should also be noted that the deflation procedure can be applied directly to the covariance matrix 
and the cross-covariance matrix 
10.18 
The above relationship relies on (10.14), (10.15a) and (10.17).
The steps of the PLS algorithm can be carried out using the NIPALS algorithm (Geladi and Kowalski 1986), the SIMPLS algorithm (de Jong 1993) or the computationally more efficient Kernel algorithms (Dayal and MacGregor 1997a; Lindgren et al. 1993; Rännar et al. 1994). Each of these algorithms are iterative in nature, that is, one pair of latent variables are obtained and the contribution of the t-score vector is deflated from the input and output matrices in one iteration step.
10.3 Summary of the PLS algorithm
The preceding analysis showed that PLS extracts covariance information from the input and output variables, x0 and y0, by defining a sequence of score variables which are extracted from the input variable set (10.5). The contribution of each score variable is maximized by the determination of loading vectors, such that the original variable sets are defined by 
and 
(10.15).
The calculation of the n score variables, t1, t2, …, tn relies on an objective function that maximizes a covariance criterion between ti and a score variable that is extracted from the output variable set ui, E{tiui} (10.6). In other words, there are a total of n score variables computed from the input variable set and n score variables calculated from the output variable set. These score variables are obtained in pairs, ti and ui and are given by a projection of the input and output variable set onto the weight vectors wi and qi (10.5). The solution for the pairs of weight vectors gives rise to the determination of dominant eigenvectors of symmetric and positive semi-definite matrices.
Unlike the PCA algorithm, the sets of latent variables can only be determined sequentially using the power method (Geladi and Kowalski 1986). This is an iterative method for determining the dominant eigenvector of a symmetric positive semi-definite matrix (Golub and van Loan 1996). Using the basic steps, developed in the previous subsection, Table 10.1 presents a PLS algorithm for determining the weight and loading vectors from the covariance matrix 
and the cross-covariance matrix 
. The next subsection presents a detailed statistical and geometric analysis of the PLS algorithm and introduces a computationally more efficient algorithm to that described in Table 10.1.
Table 10.1 PLS algorithm developed from the steps in Section 10.2.
| Step | Description | Equation |
| 1 | Initiate iteration | j = 1 |
| 2 | Obtain covariance matrix |  |
| 3 | Determine cross-covariance matrix |  |
| 4 | Set-up initial q-weight vector |  |
| 5 | Calculate w-weight vector |  |
| 6 | Scale w-weight vector to unit length |  |
| 7 | Compute q-weight vector |  |
| 8 | Scale q-weight vector to unit length |  |
| If ||1qj − 0qj|| > ϵ, set | ||
| 9 | Check for convergence | 0qj = 1qj and go to Step 5; else go to Step 10 |
| 10 | Determine p-loading vector |  |
| 11 | Calculate  -loading vector |
 |
| 12 | Deflate cross-covariance matrix |  |
| Check whether there is significant | If so, go to Step 14 | |
| 13 | variation left in the cross- | if not, terminate modeling |
| covariance matrix | procedure | |
| 14 | Deflate covariance matrix |  |
| If j < nx, set j = j + 1 | ||
| 15 | Check for dimension | and go to Step 4 |
| if not, terminate modeling | ||
| procedure |
This algorithm is similar to the Kernel PLS algorithm by Lindgren et al. (1993).
10.4 Properties of PLS
The PLS algorithm, developed and summarized in the last two subsections, has the statistical and geometrical properties listed below. For a detailed discussion of these properties, it is important to note that the preceding discussion has assumed the availability of the covariance matrix 
and the cross-covariance matrix 
. This has been for the convenience and simplicity of the presentation. Unless stated otherwise, the analysis that follows, however, removes this assumption and relies on the available data matrices X0 and Y0, whilst acknowledging that the covariance and cross-covariance matrices can be estimated from these data matrices. Hence, the weight, score and loading vectors become estimates.
.
and 
are equivalent.
for all i < j.
-loading vector.
-loading vector point in the same direction.
.
has full rank nx and the maximum number of latent variable sets have been computed, the PLS regression matrix between the input and output variables, 
, is equivalent to that calculated by the ordinary least squares solution, 
.
.The above properties are now formulated mathematically and proven.
.
is available, there exists the following relationship between the jth pair of weight vectors and the maximum of the objective function for determining these vectors.
, respectively (Kaspar and Ray 1993).
and 
, respectively. Moreover, the largest eigenvalue of both matrices is 
. On the other hand, a singular value decomposition of a matrix 
of arbitrary dimension is equal to 
, where the column vectors of 
, that is, the left singular vectors, are the eigenvectors of 
, the column vectors of 
, that is, the right singular vectors, are the eigenvectors of 
, and the elements of the diagonal matrix 
are the square root of the eigenvalues of 
or 
(Golub and van Loan 1996). Note that the eigenvectors of 
or 
are scaled to unit length. Now, replacing 
with 
, it follows that the first column vector of 
is wj, the first column vector of 
is qj and square root of the eigenvalue of 
or 
is λj, is the first diagonal element of 
, that is, the largest singular value of 
. This largest singular value, however, is equal to the maximum of the objective function in (10.6), which concludes this proof.
and 
has the following geometric property.
and 
, i ≠ j, are mutually orthogonal, that is 
.
10.20 
10.21 
10.24 
10.25 
and 
.
and 
.
and 
are equivalent
10.26 
.
if i < j.
for all i < j and 
for all i ≥ j.
10.27 
yields10.28 
. Equation (10.19) shows that 
forms part of the calculation of the p-loading vector
for all i < j.
, the scalar product 
becomes10.30 
10.31 
and 
, respectively. The deflation of these matrix products leads to
. To prove the general case, the deflation procedure allows computing E(j) from the output data matrix X0 using the score vectors 
, … , 
.10.33 
for all i < j, outlined in Lemma 10.4.4. Applying the same steps yields
10.36 
, 1 ≤ j ≤ n.
10.38 
needs to be deflated, given that the r-weight vectors allow the computation of the t-score vectors directly, which concludes the proof of Theorem 10.4.6. Moreover, it is also important to note the following.
.Corollary 10.4.8 follows from the fact that 
, discussed in (10.34). Whilst this does not require the introduction of the r-weight vectors in Lemma 10.4.7, it requires the deflation of two matrix products, that is, 
and 
, for computing the pairs of weight and loading vectors. It is, however, computationally more expensive to deflate both matrix products. The following rank-one modification presents a numerically expedient way to deflate the matrix product 
It should be noted that the scalar product 
is required for the calculation of the loading vectors and hence available for the deflation of 
. The relationship of (10.39) relies on (10.19), (10.29) and (10.32).
and 
, which has the following property.
and 
, i ≠ j, are mutually orthonormal, that is 
.
can be rewritten as10.40 
reveals that it is equal to zero10.41 
.
is given by10.42 
10.43 
for i = j follows from the computation of the p-loading vector10.44 
.
-loading vector
is equal to the norm of the 
-loading vector 
, that is 
.
yields
is the dominant eigenvector of 
, the expression 
is equal to the largest eigenvalue. According to (10.6) and (10.10a) this eigenvalue is equal to the square of the Lagrangian multiplier of the objective function for computing the jth pair of weight vectors. Moreover, the eigenvalue of 
is (K − 1)2 times the eigenvalue of 
, 
, and hence, equal to 
. On the other hand, 
is the estimate for K − 1 times the covariance between the t- and u-score variables. Consequently, (10.45) becomes10.46 
.
-loading vectors.
-loading vectors exists.
and the 
-loading vector 
have the same direction and the scaling factor between these vectors is the regression coefficient 
.
10.47 
-loading vector is given by
10.49 
. Equation (10.48) therefore becomes10.50 
, where the scaling factor between both vectors is the regression coefficient 
.
10.51 
10.53 
and e by pj, 
and g, respectively, (10.52) shows the same formulation as that for computing the t-score variables using PCA. Consequently, the proof of Theorem 9.3.4 is also applicable to the proof of Theorem 10.4.13.
.
using the sets of LVs computed by the PLS algorithm.
has full rank nx, the nx sets of LVs allow a complete reconstruction of the cross-covariance matrix 
using the nx p-loading vectors, the nx value of the objective functions function and the nx q-weight vectors.
follows from10.54 
;- the t-score vectors are mutually orthogonal, which Theorem 10.4.2 outlines;
, which Theorem 10.4.11 confirms;
which follows from (10.11); and
which follows from (10.9).
is nx the PLS regression model is identical to that obtained by an ordinary least squares solution, that is 
.
10.55 
10.56 
and 
as well as between 
and 
concludes this proof, since10.57 
10.59 
.
, 
.
has full rank nx, the nx sets of LVs allow one computation of an estimate of the parameter matrix 
without requiring the inversion of any squared matrix.
and R, however, can be computed iteratively10.61 
. Subsection 6.2.2 presents an excellent example to demonstrate the benefit of the iterative PLS procedure over OLS.
and 
are nx × nx and ny × ny, respectively. Given that there is the following linear relationship between the weight vectors10.62 
reduces to a vector of dimension nx. In this case, the w-weight vector is proportional to 
. It should be noted that the algorithm in Table 10.2 assumes the availability of the covariance and cross-covariance matrices 
and 
. As they are not available in most practical cases, they need to be estimated from the recorded samples stored in X0 and Y0 and the computed weight and loading vectors become, accordingly, estimates. It should also be noted that the PLS algorithm in Table 10.2 is similar to that reported in Dayal and MacGregor (1997a).Table 10.2 Computationally efficient PLS algorithm
| Step | Description | Equation |
| 1 | Initiate iteration | j = 1,  |
 |
||
| 2 | Set up matrix product | if nx < ny else |
 |
||
| 3 | Check dimension of y0 | if ny = 1,  and go |
| to Step 9, if not go to Step 4 | ||
 |
||
| 4 | Initiate power method | if nx < ny else |
 |
||
| 1wj = Mw(0wj) | ||
| 5 | Compute matrix-vector product | if nx < ny else |
| 1qj = Mq(0qk) | ||
 |
||
| 6 | Scale weight vector | if nx < ny else |
 |
||
| if ||1wj − 0wj|| > ϵ | ||
| or ||1qj − 0qj|| > ϵ | ||
| 7 | Check for convergence | 0wj = 1wj or 0qj = 1qj and |
| go to Step 5 else set wj = 0wj | ||
| or qj = 0qj and go to Step 8 | ||
 |
||
| 8 | Calculate 2nd weight vector | if nx < ny else |
 |
||
| 9 | Compute r-weight vector |  |
| 10 | Determine scalar |  |
| 11 | Calculate p-loading vector |  |
| 12 | Obtain  -loading vector |
 |
| 13 | Deflate cross-covariance matrix |  |
| 14 | Check whether there is significant | If so, go to Step 15 |
variation left in  |
if not go to Step 16 | |
| 15 | Check for dimension | If j < nx, set j = j + 1 and go to |
| Step 2, if not go to Step 16 | ||
| 16 | Compute regression matrix |  |
10.5 Properties of maximum redundancy PLS
Section 2.3 introduces MRPLS as a required extension to PLS to model the data structure in 2.51. This section offers a detailed examination of MRPLS in term of its geometric properties and develops a numerically more efficient algorithm. Readers who are predominantly interested in the application of the methods discussed in this book can note the computationally efficient MRPLS algorithm in Table 10.3 or the batch algorithm for simultaneously computing the n q-loading and w-weight vectors in Table 10.4.
Table 10.3 Computationally efficient MRPLS algorithm
| Step | Description | Equation |
| 1 | Initiate iteration | n = 1, j = 1 |
 , |
||
| 2 | Set up matrix products |  and |
 |
||
| 3 | Set up initial q-weight vector |  |
| 4 | Compute auxiliary weight vector |  |
| if j = n | ||
| 5 | Calculate w-weight vector |  |
else :  |
||
| 6 | Determine q-weight vector |  |
if  |
||
| 7 | Check for convergence | set  and go to Step 4 |
else set  and go to Step 8 |
||
if j = n :  |
||
| 8 | Compute r-weight vector | else : |
 |
||
if j = n :  |
||
| 9 | Determine p-loading vector | else compute  and |
 |
||
if j = n  |
||
| 10 | Determine  -loading vector |
else |
 |
||
if j = n  |
||
| 11 | Deflate cross-product matrix | else |
 |
||
| Check whether there is | if so j = j + 1, n = n + 1 | |
| 12 | still significant variation | and go to Step 4 |
remaining in  |
if not j = j + 1, go to Step 13 | |
| 13 | Check whether j = nx | if so then terminate else go to Step 3 |
Table 10.4 Simultaneous MRPLS algorithm for LV sets.
| Step | Description | Equation |
| 1 | Form matrix products |  and  |
| 2 | Compute SVD of  |
 |
| 3 | Form matrix product |  |
| 4 | Form matrix product |  |
| 5 | Calculate SVD of S |  |
| 6 | Determine w-weight matrix |  |
| 7 | Compute q-weight matrix |  |
| 8 | Calculate w-loading matrix |  |
| 9 | Obtain  -loading matrix |
 |
The analysis of the properties of the MRPLS algorithm concentrates on the geometric properties of score, loading and weight vectors first. The results enable a further analysis regarding the deflation procedure and contribute to the numerically and computationally efficient algorithm that is summarized in Table 10.3.
It should be noted that the proposed MRPLS algorithm in Table 2.3 incorporates the fact that only one of the matrices needs to be deflated and that the length-constraint for the w-weight vectors 
is equal to 
. This is also proven in this section as part of the analysis of the deflation procedure.
The properties of the MRPLS algorithm are as follows.
and 
are equal to 
and 
, respectively.
-loading vectors.
is equal to 
.
is equal to 
.
.
.
and 
, respectively.
.
10.63 
yields10.64 
10.65 
, which implies that 
if i ≠ j and 1 if i = j.
commences with
Assuming that i < j and applying the deflation procedure for E(j) a total of j − i − 1 times gives rise to
10.67 
, which yields10.68 
10.71 
. Hence, 
.
and 
and 
.
10.72 
. Similarly, the deflation of F(j) is given by10.73 
10.74 
10.75 
.
and 
if i ≠ j, and the vectors 
and 
are equal up to a scaling factor.
, which follows from10.76 
, as this includes the case 
for all i ≠ j. Given that 
, 
can be written as
-loading vectors.
-loading vectors are mutually orthogonal, i.e. 
for all i ≠ j.
, into 
under the assumption that i < j, gives rise to
Given that 
and 
, for all i ≠ j, (10.78) reduces to 
. Next, that 
for all i ≠ j follows from
10.80 
10.81 
.
, which is discussed in Lemma 10.5.6.
is equal to 
.
is equal to 
. Next, incorporating the fact that the w-weight and the p-loading vectors are mutually orthonormal (Theorem 10.5.4) gives rise to10.82 
gives rise to
, and the jth auxiliary weight vector 
.
, respectively.
is the left eigenvector of 
, associated with the largest eigenvalue, can be confirmed by solving the relationships of the MRPLS objective function
, is initially determined as the matrix-vector product 
. By substituting Steps 4 to 8 into Step 3 yields10.87 
is the dominant right eigenvector and 
is the dominant left eigenvector of 
.
-loading vectors are also eigenvectors of a specific matrix product.
-loading vector, 
and 
, are the dominant eigenvectors of the matrix product 
.
10.88 
. Finally, this section concludes with the development of a batch algorithm for simultaneously computing the n q- and w-weight vectors.Simultaneous computation of weight vectors
Recall that the derivation of the MRPLS algorithm in Subsection 2.3.3 consists of two steps. The first step involves the computation of the q-loading vectors by solving 2.63. The second step subsequently determines the w-weight vector by solving 2.66. It has then been shown that both steps can be combined. More precisely, the solution of the combined objective function in 2.68 is equivalent to the individual solutions of 2.63 and 2.66.
Coming back to the facts (i) that the q-weight vectors can be determined independently from the w-weight vectors and (ii) that the jth q-weight vector is the dominant eigenvector of the matrix 
gives rise to the following theorem.
, which is a symmetric and positive semi-definite matrix of rank n ≤ ny.
is of rank n ≤ ny. With regards to the data structure in 2.51, the covariance and cross-covariance matrices 
and 
are equal to10.90 
10.91 
and 
are equal up to the scaling factor K − 1. As the rank of the true cross-covariance matrix is n, given that 
, 
and 
, the rank of 
is asymptotically n, under the assumption that Sss has full rank rank n, that is, the source signals are not linearly dependent. The jth q-weight vector is an eigenvector associated with the eigenvalue 
, which suggests a different deflation for 
. Abbreviating this matrix expression by 
and defining 
, the first q-weight vector satisfies10.92 
is symmetric and positive definite, the deflation procedure for determining the second q-weight vector that is orthogonal to the first one is as follows10.93 
10.94 
is n, a total of n eigenpairs can be determined by the iterative power method with deflation. On the other hand, the n eigenvectors and eigenvalues can also be determined simultaneously in a batch mode, for example discussed in Chapter 8 in Golub and van Loan (1996). Once the n q-weight vectors are available, the w-weight vectors can be computed. Before demonstrating that this can also be done in a batch mode, Lemma 10.5.11 shows how to compute the u-scores directly from the output data matrix.
.
. However, the fact that the q-weight vectors are mutually orthonormal yields10.95 
, which (10.79) highlights.Next, incorporating 
into the objective function for determining the w-weight vector in 2.66 gives rise to
Taking advantage of the fact that 
and 
for all i ≠ j, (10.96) can be expanded upon
where 
. Equation (10.97) can also be written as
10.98 
which follows from Theorem 10.5.1 and yields in this batch formulation the same solution as those obtained individually. More precisely, defining the maximum of the combined objective by 
, it can only be a maximum if each of the sum elements are a maximum, according to the Bellman principle of optimality (Bellman 1957). As shown in (10.97), storing the n w-weight vectors to form the matrix 
, the solution of the objective function in (10.97) is given by
Working out the partial relationships, (10.99) becomes
10.100 
where 
and 
.
To simultaneously calculate the n w-weight vectors in batch form, the solution to the objective function in (10.97) is therefore
10.101 
Equations (10.86) and (10.89) outline that the n diagonal elements of the matrix 
are equal to the square root of the eigenvalues of 
and 
. The conclusion of this proof requires to show that
To start with, Theorem 10.5.5 shows that 
, i.e. the column vectors or 
are mutually orthonormal. Next, (10.83) and (10.84) highlight that
10.103 
which confirms (10.102).
In summary, Theorem 10.5.10 outlined that the q-weight vectors can be simultaneously computed as eigenvectors of the matrix product 
, which is positive semi-definite, whose rank asymptotically converges to n ≤ ny. On the other hand, it is also possible to simultaneously compute the w-weight vectors as eigenvectors of the matrix product 
, which is a positive semi-definite matrix with an asymptotic rank of n ≤ ny. Furthermore, the computation of the w-weight vectors is independent of the determination of the q-weight vectors. Equations (10.105) and (10.106) finally summarize the eigendecomposition of these matrices
10.104 
and
For canonical correlation analysis (CCA), a multivariate statistical method developed by Harold Hotelling in the 1930s (Hotelling 1935, 1936) that determines weight vectors to produce score variables that have a maximum correlation, a batch algorithm has been proposed to simultaneously determine Q and W. This solution for simultaneously computing the n q- and w-weight vectors can be taken advantage of by simultaneously computing the weight vectors of the MRPLS algorithm. Prior to that, the next paragraph discusses the similarities between the objective functions for CCA and maximum redundancy.
As outlined in Stewart and Love (1968), ten Berge (1985) and van den Wollenberg (1977), the CCA objective function does not consider the predictability of the output variables in the same way as maximum redundancy does. More precisely, the CCA objective function is given by
10.106 
Similar to the maximum redundancy formulation, the Lagrangian multipliers are identical and the solution of the CCA objective function for the jth pair of weight vectors, or canonical variates, is given by (Anderson 2003)
10.107 
and
10.108 
It is interesting to note that the difference in the objective function between the CCA and the maximum redundancy is the presence of the matrix product 
, which results from the different constraint for the q-weight vector. For the simultaneous computation of the n weight vectors, a batch algorithm that relies on a series of singular value decompositions (SVDs) has been developed (Anderson 2003). Table 10.4 summarizes the steps of the resultant batch algorithm for simultaneously determining the n q- and w-weight vectors.