We have briefly covered the most basic operation between two matrices; the matrix product. For any other kind of product, we resort to the basic utilities in the NumPy libraries, as: dot product for arrays or vectors (dot, vdot), inner and outer products of two arrays (inner, outer), tensor dot product along specified axes (tensordot), or the Kronecker product of two arrays (kron).

Let's see an example of creating an orthonormal basis.

Create an orthonormal basis in the nine-dimensional real space from an orthonormal basis of the three-dimensional real space.

Let's choose, for example, the orthonormal basis formed by the vectors as shown in following diagram:

We compute the desired basis by collecting these vectors in a matrix and using a Kronecker product, as follows:

>>> import numpy
>>> import scipy.linalg
>>> mu = 1/numpy.sqrt(2)
>>> A = numpy.matrix([[mu,0,mu],[0,1,0],[mu,0,-mu]])
>>> B = scipy.linalg.kron(A,A)

The columns of matrix B shown previously, give us an orthonormal basis directly. For instance, the vectors with odd indices would be the columns of the following submatrix:

>>> print (B[:,0:-1:2])

The output is shown as follows:

[[ 0.5  0.5  0.   0.5]
 [ 0.   0.   0.   0. ]
 [ 0.5 -0.5  0.   0.5]
 [ 0.   0.   0.   0. ]
 [ 0.   0.   1.   0. ]
 [ 0.  -0.   0.   0. ]
 [ 0.5  0.5  0.  -0.5]
 [ 0.   0.   0.  -0. ]
 [ 0.5 -0.5  0.  -0.5]]

The scipy.linalg module offers a useful set of functions on matrices. The basic two commands on square matrices are inv (for the inverse of a matrix) and det (for the determinant). The power of a square matrix is given by the standard exponentiation; that is, if A is a square matrix, then A**2 indicates the matrix product A*A, which is shown in the following code snippet:

>>> A=numpy.matrix("1,1j;21,3")
>>> A; A*A; A**2

The output is shown as follows:

matrix([[  1.+0.j,   0.+1.j],
        [ 21.+0.j,   3.+0.j]])
matrix([[  1.+21.j,   0. +4.j],
        [ 84. +0.j,   9.+21.j]])
matrix([[  1.+21.j,   0. +4.j],
        [ 84. +0.j,   9.+21.j]])

It should be pointed out that as a type array, the product of A*A (or A**2) is calculated by squaring each element of the array:

>>> numpy.asarray(A); numpy.asarray(A)*numpy.asarray(A); numpy.asarray(A)**2

The output is shown as follows:

array([[  1.+0.j,   0.+1.j],
       [ 21.+0.j,   3.+0.j]])
array([[   1.+0.j,   -1.+0.j],
       [ 441.+0.j,    9.+0.j]])
array([[   1.+0.j,   -1.+0.j],
       [ 441.+0.j,    9.+0.j]])

More advanced commands compute matrix functions that rely on the power series representation of expressions involving matrix powers, such as the matrix exponential (for which there are three possibilities—expm, expm2, and expm3), the matrix logarithm (logm), matrix trigonometric functions (cosm, sinm, tanm), matrix hyperbolic trigonometric functions (coshm, sinhm, tanhm), the matrix sign function (signm), or the matrix square root (sqrtm).

Notice the difference between the application of the normal exponential function on a matrix, and the result of a matrix exponential function.

In the former case, we obtain the application of numpy.exp to each entry of the matrix; in the latter, we actually compute the exponential of the matrix following the power series representation:

The preceding formula is illustrated in this code snippet:

>>> import numpy
>>> import scipy.linalg
>>> a=numpy.arange(0,2*numpy.pi,1.6)
>>> A = scipy.linalg.toeplitz(a)
>>> print (A)

The output is shown as follows:

[[ 0.   1.6  3.2  4.8]
 [ 1.6  0.   1.6  3.2]
 [ 3.2  1.6  0.   1.6]
 [ 4.8  3.2  1.6  0. ]]

Let's perform the exp() operation on A:

>>> print (numpy.exp(A))

The output is shown as follows:

[[   1.            4.95303242   24.5325302   121.51041752]
 [   4.95303242    1.            4.95303242   24.5325302 ]
 [  24.5325302     4.95303242    1.            4.95303242]
 [ 121.51041752   24.5325302     4.95303242    1.        ]]

Let's perform the expm() operation on A:

>>> print (scipy.linalg.expm(A))

The output is shown as follows:

[[ 1271.76972856   916.49316549   916.63015271  1271.70874469]
 [  916.49316549   660.86560972   660.5306514    916.63015271]
 [  916.63015271   660.5306514    660.86560972   916.49316549]
 [ 1271.70874469   916.63015271   916.49316549  1271.76972856]]

For sparse square matrices, we have an optimized inverse function, as well as a matrix exponential—scipy.sparse.linalg.inv, scipy.sparse.linalg.expm.

For general matrices, we have the basic norm function (norm), as well as two versions of the Moore-Penrose pseudoinverse (pinv and pinv2).

Once again, we need to emphasize how important it is to rely on these functions, rather than coding their equivalent expressions manually. For instance, note the norm computation of vectors or matrices, scipy.linalg.norm. Let's show you, by example, the 2-norm of a two-dimensional vector v=numpy.matrix([x,y]), where at least one of the x and y values is extremely large—large enough so that x*x overflows:

>>> import numpy
>>> import scipy.linalg
>>> x=10**100; y=9; v=numpy.matrix([x,y])
>>> scipy.linalg.norm(v,2)

The output is shown as follows:

1e+100

Now, let's perform the sqrt() operation:

>>> numpy.sqrt(x*x+y*y)

The output is an error which is shown as follows:

Traceback (most recent call last)
  File "<stdin>", line 1, in <module>
AttributeError: 'long' object has no attribute 'sqrt'

Another set of operations heavily used on matrices is to compute and handle eigenvalues and eigenvectors of square matrices. These two problems rank among the most complex operations that we can perform on square matrices, and extensive research has been put in place to obtain good algorithms with low complexity and optimal usage of memory resources. SciPy has state-of-the-art code to implement these ideas.

For the computation of eigenvalues, the scipy.linalg module provides three routines: eigvals (for any ordinary or general eigenvalue problem), eigvalsh (if the matrix is symmetric of complex Hermitian), and eigvals_banded (if the matrix is banded). To compute the eigenvectors, we similarly have three corresponding choices: eig, eigh, and eigh_banded.

The syntax used in all cases is very similar. For example, for the general case of eigenvalues, we use the following line of code where matrix A must be square:

eigvals(A, B=None, overwrite_a=False)

This should be the only parameter passed to the routine if we wish to solve an ordinary eigenvalue problem. If we wish to generalize this, we may provide an extra square matrix (of the same dimensions as matrix A). This is passed in the B parameter.

The module also offers an extensive collection of functions that compute different decompositions of matrices, as follows:

At this point, we have an interesting application—image compression, which makes use of some of the routines explained so far.

This is a very simple application where a square image A of size n x n, and stored as ndarray is regarded as a matrix, and where a singular value decomposition (SVD) is performed on it. This operation is visible in the following diagram:

From all the singular values of s we choose a fraction, together with their corresponding left and right singular vectors u, v. We compute a new matrix by collecting them according to the formula given in the following diagram:

Note, for example, the similarity between the original (512 singular values) and an approximation using only 32 singular values:

>>> import numpy
>>> import scipy.misc
>>> from scipy.linalg import svd
>>> import matplotlib.pyplot as plt
>>> img=scipy.misc.lena()
>>> U,s,Vh=svd(img)      # Singular Value Decomposition
>>> A = numpy.dot( U[:,0:32],  # use only 32 singular values
        numpy.dot( numpy.diag(s[0:32]),
                   Vh[0:32,:]))
>>> plt.subplot(121,aspect='equal'), plt.imshow(img); plt.gray()
>>> plt.subplot(122,aspect='equal'), plt.imshow(A)
>>> plt.show()

This produces the following images, of which the picture to the left is the original image and the picture to the right, the approximation using 32 singular values:

Using the svd approximation we managed to compress the original image of 262,144 coefficients (512 * 512)to only 32,800 coefficients ((2 * 32 * 512) + 32), or to one-eighth of the original information.

One of the fundamental applications of linear algebra is to solve large systems of linear equations. For the basic systems of the form Ax=b, for any square matrix A and general matrix b (with as many rows as columns in A), we have two generic methods to find x (solve for dense matrices and spsolve for sparse matrices), using the following syntax:

solve(A, b, sym_pos=False, lower=False, overwrite_a=False, overwrite_b=False, debug=False)
spsolve(A, b[, permc_spec, use_umfpack])

There are solvers that are even more sophisticated in SciPy, with enhanced performance for situations in which the structure of the matrix A is known. For dense matrices we have three commands in the scipy.linalg module: solve_banded (for banded matrices), solveh_banded (if besides banded, A is Hermitian), and solve_triangular (for triangular matrices).

When a solution is not possible (for example, if A is a singular matrix), it is still possible to obtain a matrix x that minimizes the norm of b-Ax in a least-squares sense. We can compute such a matrix with the lstsq command, which has the following syntax:

lstsq(A, b, cond=None, overwrite_a=False, overwrite_b=False)

The output of this function is a tuple that contains the following:

Let's illustrate this routine with a simple example, to solve the following system:

The following is the code snippet:

>>> import numpy
>>> import scipy.linalg
>>> A=numpy.mat(numpy.eye(3,k=1))
>>> print(A)

The output is shown as follows:

[[ 0.  1.  0.]
 [ 0.  0.  1.]
 [ 0.  0.  0.]]

Let's move further into the code and perform the following operations on b:

>>> b=numpy.mat(numpy.arange(3) + 1).T
>>> print(b)

The output is shown as follows:

[[1]
 [2]
 [3]]

Further, let's perform the lstsq operation:

>>> xinfo=scipy.linalg.lstsq(A,b)
>>> print (xinfo[0].T)      # output the solution

The output is shown as follows:

[[ 0.  1.  2.]]

The overwrite_ options are designed to enhance performance of the algorithms, and should be used carefully, since they destroy the original data.

The truly fastest solvers in SciPy are based upon decomposition of matrices. Reducing the system into something simpler easily solves huge and really complicated systems of linear equations. We may accomplish this using decomposition techniques presented in the Eigenvalue problems and matrix decompositions and Image compression via the singular value decomposition subsections under the Matrix methods section of this chapter, but of course, the SciPy philosophy is to help us deal with all nuisances of memory and resources internally. To this end, the module also has the lu_solve (for solutions based on LU decompositions), and cho_solve, cho_solve_banded (for solutions based on Cholesky decompositions).

Finally, you will also find solvers for very complex matrix equations—the Sylvester equation (solve_sylvester), both the continuous and discrete algebraic Riccati equations (solve_continuous_are, solve_discrete_are) and both the continuous and discrete Lyapunov equations (solve_discrete_lyapunov, solve_lyapunov).

Most of the matrix decompositions and solutions to eigenvalue problems are contemplated for sparse matrices in the scipy.sparse.linalg module with a similar naming convention, but with much more robust use of computer resources and error control.