10.2.1 Probabilistic PCA

Euclidean principal component analysis (PCA) is traditionally defined as a fit of best approximating linear subspaces of a given dimension to data, either by maximizing variance

$\hat{W}={\mathrm{argmax}}_{W\in O({\mathbb{R}}^r,{\mathbb{R}}^d)}\sum _{i=1}^N{\Vert W{W}^T{y}_i\Vert }^2$

of the centered data $y_1,\dots ,y_N$ projected to r-dimensional subspaces of ${\mathbb{R}}^d$ represented here by orthonormal matrices $W\in O({\mathbb{R}}^r,{\mathbb{R}}^d)$ of rank r or by minimizing residual errors

$\hat{W}={\mathrm{argmin}}_{W\in O({\mathbb{R}}^r,{\mathbb{R}}^d)}\sum _{i=1}^N{\Vert y_i-W{W}^T{y}_i\Vert }^2$

between the observations and their projections to the subspace. We see that fundamental for this construction is the notion of linear subspace, projections to linear subspaces, and squared distances. The dimension r of the fitted subspace determines the number of principal components.

PCA can however also be defined from a probabilistic viewpoint [37,29]. The approach is here to fit the latent variable model (10.3) with W of fixed rank r. The conditional distribution of the data given the latent variable $x\in {\mathbb{R}}^r$ is normal

$y|x\sim N(m+Wx,{\sigma }^{2}I).$

With x normally distributed $\mathcal{N}(0,{\mathrm{Id}}_r)$ and noise $ϵ\sim \mathcal{N}(0,{\sigma }^2{\mathrm{Id}}_d)$, the marginal distribution of y is $y\sim N(m,\mathrm{\Sigma })$ with $\mathrm{\Sigma }=W{W}^T+{\sigma }^2{\mathrm{Id}}_d$.

The Euclidean principal components of the data are here interpreted as the conditional distribution $x|y_i$ of x given the data $y_i$. From the data conditional distribution, a single quantity representing $y_i$ can be obtained by taking expectation $x_i:=\mathbb{E}[x|y_i]={(W^{T}W+{\sigma }^{2}I)}^{-1}{W}^T(y_i-m)$. The parameters of the model m, W, σ can be found by maximizing the likelihood

$\mathcal{L}(W,\sigma ,m;y)=|2\pi \mathrm{\Sigma }{|}^{-\frac{1}{2}}{e}^{-\frac{1}{2}{(y-m)}^T{\mathrm{\Sigma }}^{-1}(y-m))}.$

Up to rotation, the ML fit of W is given by ${\hat{W}}_{\mathrm{ML}}={\hat{U}}_r{(\hat{\mathrm{\Lambda }}-{\sigma }^2{\mathrm{Id}}_d)}^{1/2}$, where $\hat{\mathrm{\Lambda }}=\mathrm{diag}({\hat{\lambda }}_1,\dots ,{\hat{\lambda }}_r)$, ${\hat{U}}_r$ contains the first r principal eigenvectors of the sample covariance matrix of $y_i$ in the columns, and ${\hat{\lambda }}_1,\dots ,{\hat{\lambda }}_r$ are the corresponding eigenvalues.

10.2.2 Riemannian normal distribution and probabilistic PGA

We saw in chapter 2 the Normal law or Riemannian normal distribution defined via its density

$p(y;m,{\sigma }^2)=C{(m,{\sigma }^2)}^{-1}{e}^{-\frac{\mathrm{dist}{(m,y)}^2}{2{\sigma }^2}}$

with normalization constant $C(m,{\sigma }^2)$ and the parameter ${\sigma }^2$ controlling the dispersion of the distribution. The density is given with respect to the volume measure ${\mathrm{d}\mathcal{V}}_g$ on $\mathcal{M}$, so that the actual distribution is $p(\cdot ;m,{\sigma }^2){\mathrm{d}\mathcal{V}}_g$. Because of the use of the Riemannian distance function, the distribution is at first sight related to a normal distribution $N(0,{\sigma }^2{\mathrm{Id}}_d)$ in $T_m\mathcal{M}$; however, its definition with respect to the measure ${\mathrm{d}\mathcal{V}}_g$ implies that it differs from the density of the normal distribution at each point of $T_m\mathcal{M}$ by the square root determinant of the metric $|g{|}^{\frac{1}{2}}$. The isotropic precision/concentration matrix ${\sigma }^{-2}{\mathrm{Id}}_d$ can be exchanged with a more general concentration matrix in $T_m\mathcal{M}$. The distribution maximizes the entropy for fixed parameters $(m,\mathrm{\Sigma })$ [26].

This distribution is used in [39] to generalize Euclidean PPCA. Here the distribution of the latent variable x is normal in $T_m\mathcal{M}$, x is mapped to $\mathcal{M}$ using ${\mathrm{Exp}}_m$, and the conditional distribution $y|x$ of the observed data y given x is Riemannian normal $p(y;{\mathrm{Exp}}_{m}x,{\sigma }^2){\mathrm{d}\mathcal{V}}_g$. The matrix W models the square root covariance $\mathrm{\Sigma }=W{W}^T$ of the latent variable x in $T_m\mathcal{M}$. The model is called probabilistic principal geodesic analysis (PPGA).

10.2.3 Transition distributions and stochastic differential equations

Instead of mapping latent variables from $T_m\mathcal{M}$ to $\mathcal{M}$ using the exponential map, we can take an infinitesimal approach and only map infinitesimal displacements to the manifold, thereby avoiding the use of ${\mathrm{Exp}}_m$ and the implicit linearization coming from the use of a single tangent space. The idea is to create probability distributions as solutions to stochastic differential equations, SDEs. In Euclidean space, SDEs are usually written on the form

$dy(t)=b(t,y(t))dt+a(t,y(t))dx(t),$

where $a:\mathbb{R}\times {\mathbb{R}}^d\to {\mathbb{R}}^{d\times d}$ is the diffusion field modeling the local diffusion of the process, and $b:\mathbb{R}\times {\mathbb{R}}^d\to {\mathbb{R}}^d$ models the deterministic drift. The process $x(t)$ of which we multiply the infinitesimal increments $dx(t)$ on a is a semimartingale. For our purposes, we can assume that it is a standard Brownian motion, often written $W(t)$ or $B(t)$. Solutions to (10.9) are defined by an integral equation that discretized in time takes the form

$y(t_i)=y(0)+\sum _{j=1}^{i-1}b(t_j,y(t_j))(t_{j+1}-t_j)+a(t_j,y(t_j))(x(t_{j+1})-x(t_j)).$

This is called an Itô equation. Alternatively, we can use the Fisk–Stratonovich solution

$y(t_i)=y(0)+\sum _{j=1}^{i-1}b(t_j^{⁎},y(t_j^{⁎}))(t_{j+1}-t_j)+a(t_j^{⁎},y(t_j^{⁎}))(x(t_{j+1})-x(t_j))$

where $t_j^{⁎}=(t_{j+1}-t_j)/2$, that is, the integrand is evaluated at the midpoint. Notationally, Fisk–Stratonovich SDEs, often just called Stratonovich SDEs, are distinguished from Itô SDEs by adding ∘ in the diffusion term $a(t,y(t))\circ dx(t)$ in (10.9). The main purpose here of using Stratonovich SDEs is that solutions obey the ordinary chain rule of differentiation and therefore map naturally between manifolds.

A solution $y(t)$ to an SDE is a t-indexed family of probability distributions. If we fix a time $T>0$, then the transition distribution $y(T)$ denotes the distribution of endpoints of sample paths $y(\omega )(t)$, where ω is a particular random event. We can thus generate distributions in this way and set $\mu (\theta )=y(T)$, where the parameters θ now control the dynamics of the process via the SDE, particularly the drift b, the diffusion field a, and the starting point $y_0$ of the process.

The use of SDEs fits the differential structure of manifolds well because SDEs are defined infinitesimally. However, because we generally do not have global coordinate systems to write up an SDE as in (10.9), defining SDEs on manifolds takes some work. We will see several examples of this in the sections below.

Particularly, we will define an SDE that reformulates (10.3) as a time-sequence of random steps, where the latent variable x will be replaced by a latent process $x(t)$, where the covariance W will be parallel transported over $\mathcal{M}$. This process will again have parameters $(m,W,\sigma )$. We define the distribution $\mu (m,W,\sigma )$ by setting $\mu (m,W,\sigma )=y(T)$, and we then assume that the observed data $y_1,\dots ,y_N$ have marginal distribution $y_i\sim \mu (m,W,\sigma )$. Note that $y(T)$ is a distribution, whereas $y_i$, $i=1,\dots ,N$, denote the data.

Let $p(y_i;m,W,\sigma )$ denote the density of the distribution $\mu (m,W,\sigma )$ with respect to a fixed measure. As in the PPCA situation, we then have a likelihood for the model

$\mathcal{L}(m,W,\sigma ;y_i)=p(y_i;m,W,\sigma ),$

and we can optimize for the ML estimate $\hat{\theta }=(\hat{m},\hat{W},\hat{\sigma })$. Again, similarly to the PPCA construction, we get the generalization of the principal components by conditioning the latent process on the data: $x_{i,t}:=x(t)|y(T)=y_i$. The picture here is that among all sample paths $y(\omega )(t)$, we single out those hitting $y_i$ at time T and consider the corresponding realizations of the latent process $x(\omega )(t)$ a representation of the data.

Fig. 10.1 displays the result of pursuing this construction compared to tangent space PCA. Because the anisotropic covariance is now transported with the process instead of being tied to a single tangent space, the curvature of the sphere is in a sense incorporated into the model, and the linear view of the data $x_{i,t}$, particularly the endpoints $x_i:=x_{i,T}$, provide an improved picture of the data variation on the manifold.

Below, we will make the construction of the underlying stochastic process precise and present other examples of geometrically natural processes that allow for generating geometrically natural families of probability distributions $\mu (\theta )$.

10.3.1 Brownian motion on Riemannian manifolds

A Riemannian metric g defines the Laplace–Beltrami operator ${\mathrm{\Delta }}_g$ that generalizes the usual Euclidean Laplace operator used in (10.13). The operator is defined on real-valued functions by ${\mathrm{\Delta }}_{g}f=\mathrm{div}{}_g\mathrm{grad}{}_{g}f$. When $e_1,\dots ,e_d$ is an orthonormal basis for $T_{y}M$, it has the expression ${\mathrm{\Delta }}_{g}f(y)={{\sum }_{i=1}}^d{\mathrm{\nabla }}_y^{2}f(e_i,e_i)$ when evaluated at y similarly to the Euclidean Laplacian. The expression ${\mathrm{\nabla }}_y^{2}f(e_i,e_i)$ denotes the Hessian ${\mathrm{\nabla }}_y^2$ evaluated at the pair of vectors $(e_i,e_i)$. The heat equation on $\mathcal{M}$ is the partial differential equation defined from the Laplace–Beltrami operator by

${\partial }_{t}p(t,y)=\frac{1}{2}{\mathrm{\Delta }}_{g}p(t,y),y\in \mathcal{M}.$

With initial condition $p(0,\cdot )$ at $t=0$ being the indicator function ${\delta }_m(y)$, the solution is called the heat kernel and written $p(t,m,y)$ when evaluated at $y\in M$. The heat equation again models point sourced heat flows starting at m and diffusing through the medium with the Laplace–Beltrami operator now ensuring that the flow is adapted to the nonlinear geometry. The heat kernel is symmetric in that $p(t,m,y)=p(t,y,m)$ and satisfies the semigroup property

$p(t+s,m,y)={\int }_{\mathcal{M}}p(t,m,z)p(s,z,y){\mathrm{d}\mathcal{V}}_g(z).$

Similarly to the Euclidean situation, we can recover the heat kernel from a diffusion process on $\mathcal{M}$, the Brownian motion. The Brownian motion on Riemannian manifolds and Lie groups with a Riemannian metric can be constructed in several ways: Using charts, by embedding in a Euclidean space, or using left/right invariance as we pursue in this section. A particular important construction here is the Eells–Elworthy–Malliavin construction of Brownian motion that uses a fiber bundle of the manifold to define an SDE for the Brownian motion. We will use this construction in section 10.4 and through the rest of the chapter.

The heat kernel $p(t,m,y)$ is related to a Brownian motion $x(t)$ on $\mathcal{M}$ by its transition density, that is,

${\mathbb{P}}_m(x(t)\in C)={\int }_{C}p(t,m,y){\mathrm{d}\mathcal{V}}_g(y)$

for subsets $C\subset \mathcal{M}$. If $\mathcal{M}$ is assumed compact, it can be shown that it is stochastically complete, which implies that the Brownian motion exists for all time and that ${\int }_{\mathcal{M}}p(t,m,y){\mathrm{d}\mathcal{V}}_g(y)=1$ for all $t>0$. If $\mathcal{M}$ is not compact, the long time existence can be ensured by, for example, bounding the Ricci curvature of $\mathcal{M}$ from below; see, for example, [7]. In coordinates, a solution $y(t)$ to the Itô SDE

$dy{(t)}^i=b(y(t))dt+{\sqrt{g{(y(t))}^{-1}}}^{i}dB(t)$

is a Brownian motion on $\mathcal{M}$ [13]. Here $B(t)$ is a Euclidean ${\mathbb{R}}^d$-valued Brownian motion, the diffusion field $\sqrt{g{(y(t))}^{-1}}$ is a square root of the cometric tensor $g{(y(t))}^{ij}$, and the drift $b(y(t))$ is the contraction $-\frac{1}{2}g{(y(t))}^{kl}\mathrm{\Gamma }{(y(t))}_{kl}$ of the metric and the Christoffel symbols $\mathrm{\Gamma }_{kl}{}^i$. Fig. 10.2 shows sample paths from a Brownian motion on the sphere ${\mathbb{S}}^2$.

10.3.2 Lie groups

With a left-invariant metric on a Lie group G (see chapter 1), the Laplace–Beltrami operator takes the form $\mathrm{\Delta }f(x)=\mathrm{\Delta }(L_{y}f)(y^{-1}x)$ for all $x,y\in G$. By left-translating to the identity the operator thus needs only be computed at $x=e$, that is, at the Lie algebra $\mathfrak{g}$. Like the Laplace–Beltrami operator, the heat kernel is left-invariant [21] when the metric is left-invariant. Similar invariance happens in the right-invariant case.

Let $e_1,\dots ,e_d$ be an orthonormal basis for $\mathfrak{g}$, so that $X_i(y)={(L_y)}_{⁎}(e_i)$ is an orthonormal set of vector fields on G. Let $C^i{}_{jk}$ denote the structure coefficients given by

$[X_j,X_k]=C^i{}_{jk}{X}_i,$

and let $B(t)$ be a standard Brownian motion on ${\mathbb{R}}^d$. Then the solution $y(t)$ of the Stratonovich differential equation

$dy(t)=-\frac{1}{2}\sum _{j,i}C^j{}_{ij}{X}_i(y(t))dt+X_i(y(t))\circ dB{(t)}^i$

is a Brownian motion on G. Fig. 10.3 visualizes a sample path of $B(t)$ and the corresponding sample of $y(t)$ on the group $\mathrm{SO}(3)$. When the metric on $\mathfrak{g}$ is in addition Ad-invariant, the drift term vanishes leaving only the multiplication of the Brownian motion increments on the basis.

The left-invariant fields $X_i(y)$ here provide a basis for the tangent space at y that in (10.17) is used to map increments of the Euclidean Brownian motion $B(t)$ to $T_{y}G$. The fact that $X_i$ are defined globally allows this construction to specify the evolution of the process at all points of G without referring to charts as in (10.15). We will later on explore a different approach to obtain a structure much like the Lie group fields $X_i$ but on general manifolds, where we do not have globally defined continuous and nonzero vector fields. This allows us to write the Brownian motion globally as in the Lie group case.

10.4.1 The frame bundle

A fiber bundle over a manifold M is a manifold E with a map $\pi :E\to M$, called the projection, such that for sufficiently small neighborhoods $U\subset M$, the preimage ${\pi }^{-1}(U)$ can be written as a product ${\pi }^{-1}\simeq U\times F$ between U and a manifold F, the fiber. When the fibers are vector spaces, fiber bundles are called vector bundles. The most commonly occurring vector bundle is the tangent bundle TM. Recall that a tangent vector always lives in a tangent space at a point in M, that is, $v\in T_{y}M$. The map $\pi (v)=y$ is the projection, and the fiber ${\pi }^{-1}(y)$ of the point y is the vector space $T_{y}M$, which is isomorphic to ${\mathbb{R}}^d$.

Consider now basis vectors $W_1,\dots ,W_d$ for $T_{y}M$. As an ordered set $(W_1,\dots ,W_d)$, the vectors are in combination called a frame. The frame bundle $F\mathcal{M}$ is a fiber bundle over M such that the fibers ${\pi }^{-1}(y)$ are sets of frames. Therefore a point $u\in F\mathcal{M}$ consists of a collection of basis vectors $(W_1,\dots ,W_d)$ and the base point $y\in M$ of which $W_1,\dots ,W_d$ make up a basis for $T_{y}M$. We can use the local product structure of frame bundles to locally write $u=(y,W)$ where $y\in M$ as $W_1,\dots ,W_d$ are the basis vectors. Often, we denote the basis vectors in u just $u_1,\dots ,u_d$. The frame bundle has interesting geometric properties, which we will use through the chapter. The frame bundle of ${\mathbb{S}}^2$ is illustrated in Fig. 10.4.

10.4.2 Horizontality

The frame bundle, being a manifold, itself has a tangent bundle $TF\mathcal{M}$ with derivatives $\dot{u}(t)$ of paths $u(t)\in F\mathcal{M}$ being vectors in $T_{u(t)}F\mathcal{M}$. We can use the fiber bundle structure to split $TF\mathcal{M}$ and thereby define two different types of infinitesimal movements in $F\mathcal{M}$. First, a path $u(t)$ can vary solely in the fiber direction meaning that for some $y\in M$, $\pi (u(t))=y$ for all t. Such a path is called vertical. At a point $u\in F\mathcal{M}$ the derivative of the path lies in the linear subspace $V_{u}F\mathcal{M}$ of $T_{u}F\mathcal{M}$ called the vertical subspace. For each y, $V_{u}F\mathcal{M}$ is a $d^2$-dimensional manifold. It corresponds to changes of the frame, the basis vectors for $T_{y}M$, while the base point y is kept fixed. $F\mathcal{M}$ is a $(d+d^2)$-dimensional manifold, and the subspace containing the remaining d dimensions of $T_{u}F\mathcal{M}$ is in a particular sense separate from the vertical subspace. It is therefore called the horizontal subspace $H_{u}F\mathcal{M}$. Just as tangent vectors in $V_{u}F\mathcal{M}$ model changes only in the frame keeping y fixed, the horizontal subspace models changes of y keeping, in a sense, the frame fixed. However, frames are tied to tangent spaces, so we need to define what is meant by keeping the frame fixed. When $\mathcal{M}$ is equipped with a connection ∇, being constant along paths is per definition having zero acceleration as measured by the connection. Here, for each basis vector $u_i$, we need ${\mathrm{\nabla }}_{\dot{y}(t)}{u}_i(t)=0$ when $u(t)$ is the path in the frame bundle and $y(t)=\pi (u(t))$ is the path of base points. This condition is exactly satisfied when the frame vectors $u_i(t)$ are each parallel transported along $y(t)$. The derivatives $\dot{u}(t)$ of paths satisfying this condition make up the horizontal subspace of $T_{y(t)}M$. In other words, the horizontal subspace of $TF\mathcal{M}$ contains derivatives of paths where the base point $y(t)$ changes, but the frame is kept as constant as possible as sensed by the connection.

The frame bundle has a special set of horizontal vector fields $H_1,\dots ,H_d$ that make up a global basis for $HF\mathcal{M}$. This set is in a way a solution to defining the SDE (10.18) on manifolds: Although we cannot in the general situation find a set of globally defined vectors fields as we used in the Euclidean and Lie group situation to drive the Brownian motion (10.17), we can lift the problem to the frame bundle where such a set of vector fields exists. This will enable us to drive the SDE in the frame bundle and then subsequently project its solution to the manifold using π. To define $H_i$, take the ith frame vector $u_i\in T_{y}M$, move y infinitesimally in the direction of the frame vector $u_i$, and parallel transport each frame vector $u_j$, $j=1,\dots ,d$, along the infinitesimal curve. The result is an infinitesimal displacement in $TF\mathcal{M}$, a tangent vector to $F\mathcal{M}$, which by construction is an element of $HF\mathcal{M}$. This can be done for any $u\in F\mathcal{M}$ and any $i=1,\dots ,d$. Thus we get the global set of horizontal vector fields $H_i$ on $F\mathcal{M}$. Together, the fields $H_i$ are linearly independent because they model displacement in the direction of the linearly independent vectors $u_i$. In combination the fields make up a basis for the d-dimensional horizontal spaces $H_{\pi (u)}F\mathcal{M}$ for each $u\in F\mathcal{M}$.

For each $y\in \mathcal{M}$, $T_y\mathcal{M}$ has dimension d, and with $u\in F\mathcal{M}$, we have a basis for $T_y\mathcal{M}$. Using this basis, we can map a vector $v\in {\mathbb{R}}^d$ to a vector $uv\in T_y\mathcal{M}$ by setting $uv:=u_i{v}^i$ using the Einstein summation convention. This mapping is invertible, and we can therefore consider the $F\mathcal{M}$ element u a map in $\mathrm{GL}({\mathbb{R}}^d,T\mathcal{M})$. Similarly, we can map v to an element of $H_{u}F\mathcal{M}$ using the horizontal vector fields $H_i(u)v^i$, a mapping that is again invertible. Combining this, we can map vectors from $T_y\mathcal{M}$ to ${\mathbb{R}}^d$ and then to $H_{u}F\mathcal{M}$. This map is called the horizontal lift $h_u:T_{\pi (u)}\mathcal{M}\to H_{u}F\mathcal{M}$. The inverse of $h_u$ is just the push-forward ${\pi }_{⁎}:H_{u}F\mathcal{M}\to T_{\pi (u)}\mathcal{M}$ of the projection π. Note the u dependence of the horizontal lift: $h_u$ is a linear isomorphism between $T_{\pi (u)}\mathcal{M}$ and $H_{u}F\mathcal{M}$, but the mapping will change with different u, and it is not an isomorphism between the bundles $T\mathcal{M}$ and $HF\mathcal{M}$ as can be seen from the dimensions 2d and $2d+d^2$, respectively.

10.4.3 Development and stochastic development

We now use the horizontal fields $H_1,\dots ,H_d$ to construct paths and SDEs on $F\mathcal{M}$ that can be mapped to $\mathcal{M}$. Keep in mind the Lie group SDE (10.17) for Brownian motion where increments of a Euclidean Brownian motion $B(t)$ or $x(t)$ are multiplied on an orthonormal basis. We now use the horizontal fields $H_i$ for the same purpose. We start deterministically. Let $x(t)$ be a $C^1$ curve on ${\mathbb{R}}^d$ and define the ODE

$\dot{u}(t)=H_i(u(t)){\dot{x}}^i(t)$

on $F\mathcal{M}$ started with a frame bundle element $u_0=u$. By mapping the derivative of $x(t)$ in ${\mathbb{R}}^d$ to $TF\mathcal{M}$ using the horizontal fields $H_i(u(t))$ we thus obtain a curve in $F\mathcal{M}$. Such a curve is called the development of $x(t)$. See Fig. 10.5 for a schematic illustration. We can then directly obtain a curve $y(t)$ in $\mathcal{M}$ by setting $y(t)=\pi (u(t))$, that is, removing the frame from the generated path. The development procedure is often visualized as rolling the manifold $\mathcal{M}$ along the path of $x(t)$ in the manifold ${\mathbb{R}}^d$. For this reason, it is denoted “rolling without slipping”. We will use the letter x for the curve $x(t)$ in ${\mathbb{R}}^d$, u for its development $u(t)$ in $F\mathcal{M}$, and y for the resulting curve $y(t)$ on $\mathcal{M}$.

The development procedure has a stochastic counterpart: Let now $x(t)$ be an ${\mathbb{R}}^d$-valued Euclidean semimartingale. For our purposes, $x(t)$ will be a Brownian motion on ${\mathbb{R}}^d$. The stochastic development SDE is then

$du(t)=H_i(u(t))\circ d{x}^i(t)$

using Stratonovich integration. In the stochastic setting, $x(t)$ is sometimes called the driving process for $y(t)$. Observe that the development procedure above, which was based on mapping differentiable curves, here works for processes that are almost surely nowhere differentiable. It is not immediate that this works, and arguing rigorously for the well-posedness of the stochastic development employs nontrivial stochastic calculus; see, for example, [13].

The stochastic development has several interesting properties: (1) It is a mapping from the space of stochastic paths on ${\mathbb{R}}^d$ to M, that is, each sample path $x(\omega )(t)$ gets mapped to a path $y(\omega )(t)$ on $\mathcal{M}$. It is in this respect different from the tangent space linearizations, where vectors, not paths, in $T_{m}M$ are mapped to points in M. (2) It depends on the initial frame $u_0$. In particular, if $\mathcal{M}$ is Riemannian and $u_0$ orthonormal, then the process $y(t)$ is a Riemannian Brownian motion when $x(t)$ is a Euclidean Brownian motion. (3) It is defined using the connection of the manifold. From (10.20) and the definition of the horizontal vector fields we can see that a Riemannian metric is not used. However, a Riemannian metric can be used to define the connection, and a Riemannian metric can be used to state that $u_0$ is, for example, orthonormal. If $\mathcal{M}$ is Riemannian, stochastically complete and $u_0$ orthonormal, we can write the density of the distribution $y(t)$ with respect to the Riemannian volume, that is, $y(t)=p(t;u_0){\mathrm{d}\mathcal{V}}_g$. If $\pi (u_0)=m$, then the density $p(t;u_0)$ will then equal the heat kernel $p(t,m,\cdot )$.

10.5.1 Infinitesimal covariance

Recall the definition of covariance of a multivariate Euclidean stochastic variable X: $\mathrm{cov}(X^i,X^j)=\mathbb{E}[(X^i-{\overline{X}}^i)(X^j-{\overline{X}}^j)]$, where $\overline{X}=\mathbb{E}[X]$ is the mean value. This definition relies by construction on the coordinate system used to extract the components $X^i$ and $X^j$. Therefore it cannot be transferred to manifolds directly. Instead, other similar notions of covariance have been treated in the literature, for example,

${\mathrm{cov}}_m(X^i,X^j)=\mathbb{E}[{\mathrm{Log}}_m{(X)}^i{\mathrm{Log}}_m{(X)}^j]$

defined in [26]. In the form expressed here, a basis for $T_m\mathcal{M}$ is used to extract components of the vectors ${\mathrm{Log}}_m(X)$. Here we take a different approach and define a notion of infinitesimal covariance in the case where the distribution y is generated by a driving stochastic process. This will allow us to extend the transition distribution of the Brownian motion, which is isotropic and has trivial covariance, to the case of anisotropic distributions with nontrivial infinitesimal covariance.

Recall that when $\sigma =0$, the marginal distribution of y in (10.3) is normal $N(m,\mathrm{\Sigma })$ with covariance $\mathrm{\Sigma }=W{W}^T$. The same distribution appears when we take the stochastic process view and use W in (10.18). We now take this to the manifold situation by starting the process (10.20) at a point $u=(m,W)$ in the frame bundle. This is a direct generalization of (10.18). When W is an orthonormal basis, the generated distribution is the transition distribution of a Riemannian Brownian motion. However, when W is not orthonormal, the generated distribution becomes anisotropic. Fig. 10.6 shows density plots of the Riemannian normal distribution and a Brownian motion both with $W=0.5{\mathrm{Id}}_d$, and an anisotropic distribution with $W\propto ̸{\mathrm{Id}}_d$.

We can write up the likelihood of observing a point $y\in M$ at time $t=T$ under the model,

$\mathcal{L}(m,W;y)=p(T,y;m,W),$

where $p(t,y;m,W)$ is the time t-density at the point $y\in \mathcal{M}$ of the generated anisotropic distribution $y(t)$. Without loss of generality, the observation time can be set to $T=1$ and skipped from the notation. The density can only be written with respect to a base measure, here denoted ${\mu }_0$, such that $y(T)=p(T;m,W){\mu }_0$. If $\mathcal{M}$ is Riemannian, then we can set ${\mu }_0={\mathrm{d}\mathcal{V}}_g$, but this is not a requirement: The construction only needs a connection and a fixed base measure with respect to which we define the likelihood.

The parameters of the model, m and W, are represented by one element u of the frame bundle $F\mathcal{M}$, that is, the starting point of the process $u(t)$ in $F\mathcal{M}$. Writing θ for the parameters combined, we have $\theta =u=(m,W)$. These parameters correspond to the mean m and covariance $\mathrm{\Sigma }=W{W}^T$ for the Euclidean normal distribution $N(m,\mathrm{\Sigma })$. We can take a step further and define the mean for such a distribution to be x as we pursue below. Similarly, we can define the notion of infinitesimal square root covariance of $y(T)$ to be W.

10.5.2 Isotropic noise

The linear model (10.3) includes both the matrix W and isotropic noise $ϵ\sim N(0,{\sigma }^{2}I)$. We now discuss how this additive structure can be modeled, including the case where W is not of full rank d.

We have so far considered distributions resulting from a Brownian motion analogues of isotropic normal distributions and seen that they can be represented by the frame bundle SDE (10.20). The fundamental structure is that $u_0$ being orthonormal spreads the infinitesimal variation equally in all directions as seen by the Riemannian metric. There exists a subbundle of $F\mathcal{M}$ called the orthonormal frame bundle $O\mathcal{M}$ that consists of only such orthonormal frames. Solutions to (10.20) will always stay in $O\mathcal{M}$ if $u_0\in O\mathcal{M}$. We here use the symbol R for elements of $O\mathcal{M}$ to emphasize their pure rotation, not the scaling effect. We can model the added isotropic noise by modifying the SDE (10.20) to

$dW(t)=H_i(W(t))\circ dx{(t)}^i+H_i(R(t))\circ dϵ{(t)}^i,dR(t)=h_{R(t)}({\pi }_{⁎}(dW)),$

where the flow now has both the base point and covariance component $W(t)$ and a pure rotation $R(t)$ component serving as basis for the noise process $ϵ(t)$. As before, we let the generated distribution on $\mathcal{M}$ be $y(t)=\pi (W(t))$, that is, $W(t)$ takes the place of $u(t)$.

Elements of $O\mathcal{M}$ differ only by a rotation, and since $ϵ(t)$ is a Brownian motion scaled by σ, we can exchange $R(t)$ in the right-hand side of $dW(t)$ by any other element of $O\mathcal{M}$ without changing the distribution. Computationally, we can therefore skip $R(t)$ from the integration and instead find an arbitrary element of $O\mathcal{M}$ at each time step of a numerical integration. This is particularly important when the dimension d of the manifold is large because $R(t)$ has $d^2$ components.

We can explore this even further by letting W be a $d\times r$ matrix with $r\ll d$, thus reducing the rank of W similar to the PPCA situation (10.6). Without addition of the isotropic noise, this would in general result in the density $p(\cdot ;m,W)$ being degenerate, just as the Euclidean normal density function requires full rank covariance matrix. However, with the addition of the isotropic noise, $W+\sigma R$ can still be of full rank even though W has zero eigenvalues. This has further computational advantages: If we instead of using the frame bundle $F\mathcal{M}$, let W be an element of the bundle of rank r linear maps ${\mathbb{R}}^r\to TM$ so that $W_1,\dots ,W_r$ are r linearly independent basis vectors in $T_{\pi (W)}\mathcal{M}$, and if we remove $R(t)$ from the flow (10.22) as described before, then the flow now lives in a $(d+rd)$-dimensional fiber bundle compared to the $d+d^2$ dimensions of the full frame bundle. For low r, this can imply a substantial reduction in computation time.

10.5.3 Euclideanization

Tangent space linearizations using the ${\mathrm{Exp}}_m$ and ${\mathrm{Log}}_m$ maps provide a linear view of the data $y_i$ on $\mathcal{M}$. When the data are concentrated close to a mean m, this view gives a good picture of the data variation. However, as data spread grows larger, curvature starts having an influence, and the linear view can provide a progressively distorted picture of the data. Whereas linear views of a curved geometry will never give truly faithful picture of the data, we can use a generalization of (10.3) to provide a linearization that integrates the effect of curvature at points far from m. The standard PCA dimension-reduced view of the data is writing $W=U\mathrm{\Lambda }$, where Λ is the diagonal matrix with the eigenvalues ${\lambda }_1,\dots ,{\lambda }_r$ of W in the diagonal. In PPCA this is used to provide a low-dimensional representation of the data from the data conditional expectation $x|y_i$. This can be further reduced to a single data descriptor $x_i:=\mathbb{E}[x|y_i]$ by taking expectation, and then we obtain an equivalent of the standard PCA view by displaying $\mathrm{\Lambda }{x}_i$.

In the current probabilistic model, we can likewise condition the latent variables on the data to get a Euclidean entity describing the data. Since the latent variable is now a time-dependent path, the result of the conditioning is a process $x(t)|y(T)=y_i$ where the conditioning is on the response process hitting the data at time T. This results in a quite different view of the data as illustrated in Fig. 10.7 and exemplified in Fig. 10.1: as in PPCA, taking expectation, we get

$\overline{x}{(t)}_i=\mathbb{E}[x(t)|y(T)=y_i].$

To get a single data descriptor, we can integrate $d\overline{x}{(t)}_i$ in time to get the endpoint $x_i:={\int }_0^{T}d\overline{x}{(t)}_{i}dt=\overline{x}{(T)}_i$. From the example in Fig. 10.1 we see that this Euclideanization of the data can be quite different compared to tangent space linearization.

10.6.1 Normal distributions and maximum likelihood

Considering the transition distribution $\mu (\theta )=y(T)$ of solutions $u(t)$ to (10.20) projected to the manifold $y(t)=\pi (u(t))$ started at $\theta =u=(m,W)$ a normal distribution with infinitesimal covariance $\mathrm{\Sigma }=W{W}^T$, we can now define the sample maximum likelihood mean ${\hat{m}}_{\mathrm{ML}}$ by

${\hat{m}}_{\mathrm{ML}}={\mathrm{argmax}}_m\prod _{i=1}^N\mathcal{L}(m;y_i)$

from samples $y_1,\dots ,y_N\in \mathcal{M}$. Here, we implicitly assume that W is orthonormal with respect to a Riemannian metric. Alternatively, we can set

${\hat{m}}_{\mathrm{ML}}={\mathrm{argmax}}_m\underset{W}{\max }\prod _{i=1}^N\mathcal{L}(m,W;y_i),$

where we simultaneously optimize to find the most likely infinitesimal covariance. The former definition defines ${\hat{m}}_{\mathrm{ML}}$ as the starting point of the Brownian motion with transition density making the observations most likely. The latter includes the effect of the covariance, the anisotropy, and because of this it will in general give different results. In practice the likelihood is evaluated by Monte Carlo sampling. Parameter estimation procedures with parameters $\theta =(m)$ or $\theta =(m,W)$ and sampling methods are the topic of section 10.7.

We can use the likelihood (10.21) to get ML estimates for both the mean x and the infinitesimal covariance W by modifying (10.25) to

$({\hat{m}}_{ML},{\hat{W}}_{ML})={\hat{u}}_{ML}={\mathrm{argmax}}_u\prod _{i=1}^N\mathcal{L}(u;y_i).$

Note the nonuniqueness of the result when estimating the square root W instead of the covariance $\mathrm{\Sigma }=W{W}^T$. We discuss this point from a more geometric view in section 10.8.4.

10.6.2 Infinitesimal PPCA

The latent model (10.22) is used as the basis for the infinitesimal version of manifold PPCA [34], which we discussed in general terms in section 10.2.3. As in Euclidean PPCA, r denotes the number of principal eigenvectors to be estimated. With a fixed base measure ${\mu }_0$, we write the density of the distribution generated from the low-rank plus noise system (10.22) as $\mu (m,W,\sigma )=p(T;m,W,\sigma ){\mathrm{d}\mathcal{V}}_g$ and use this to define the likelihood $\mathcal{L}(m,W,\sigma ;y)$ in (10.12). The major difference in relation to (10.26) is now that the noise parameter σ is estimated from the data and that W is of rank $r\le d$.

The Euclideanization approach of section 10.5.3 gives equivalents of Euclidean principal components by conditioning the latent process on the data $x_i:=x(t)|y(T)=y_i$. By taking expectation this can be reduced to a single path $\overline{x}{(t)}_i:=\mathbb{E}[x(t)|y(T)=y_i]$ or a single vector $x_i:=\overline{x}{(T)}_i$.

The model is quite different from constructions of manifold PCA [11,14,31,5,27] that seek subspaces of the manifold having properties related to Euclidean linear subspaces. The probabilistic model and the horizontal frame bundle flows in general imply that no subspace is constructed in the present model. Instead, we can extract the parameters of the generated distribution and the information present in the conditioned latent process. As we discuss in section 10.8.1, the fact that the model does not generate subspaces is fundamentally linked to curvature, the curvature tensor, and nonintegrability of the horizontal distribution in $F\mathcal{M}$.

10.6.3 Regression

The generalized latent model (10.3) is used in [17] to define a related regression model. Here we assume observations $(x_i,y_i)$, $i=1,\dots ,N$, with $x_i\in {\mathbb{R}}^d$ and $y_i\in \mathcal{M}$. As in the previous models, the unknown is the point $m\in \mathcal{M}$, which takes the role of the intercept in multivariate regression, the coefficient matrix W, and the noise variance ${\sigma }^2$. Whereas the infinitesimal nature of the model that relies on the latent variable being a semimartingale makes it geometrically natural, the fact that the latent variable is a process implies that its values in the interval $(0,T)$ are unobserved if $x_i$ is the observation at time T. This turns the construction into a missing data problem, and the values of $x(t)$ in the unobserved interval $(0,T)$ needs to be integrated out. This can be pursued by combining bridge sampling as described below with matching of the sample moments of data with moments of the response variable y defined by the model [18].

10.7.1 Anisotropic least squares

The Fréchet mean (see Chapter 2) is defined from the least-squares principle. Here we aim to derive a similar least-squares condition for the variables $\theta =m$, $\theta =(m,W)$, or $\theta =(m,W,\sigma )$. With this approach, the inferred parameters $\hat{\theta }$ will only approximate the actual maximum likelihood estimates in a certain limit. Although only providing an approximation, the least-squares approach is different from Riemannian least-squares, and it is thereby both of geometric interest and gives perspective on the bridge sampling described later.

Until now we have assumed the observation time T to be strictly positive or simply $T=1$. If instead we let $T\to 0$, then we can explore the short-time asymptotic limit of the generated density. Starting with the Brownian motion, the limit has been extensively studied in the literature. For the Euclidean normal density, we know that $p_{N(m,T\mathrm{Id})}(y)={(2\pi T)}^{-\frac{d}{2}}\exp (-\frac{{\Vert y-m\Vert }^2}{2T})$. In particular, the density obeys the limit ${\lim }_{T\to 0}T\log p_{N(m,T\mathrm{Id})}(y)=-\frac{1}{2}{\Vert y-m\Vert }^2$. The same limit occurs on complete Riemannian manifolds with $\mathrm{dist}{(m,y)}^2$ instead of the norm ${\Vert y-m\Vert }^2$ and when y is outside the cut locus $C(m)$; see, for example, [13]. Thus, minimizing the squared distance to data can be seen as equivalent to maximizing the density, and hence the likelihood, for short running times of the Brownian motion specified by small T.

It is shown in [35] that there exists a function $d_Q:F\mathcal{M}\times \mathcal{M}\to \mathbb{R}$ that, for each $u\in F\mathcal{M}$, incorporates the anisotropy modeled in u in a measurement of the closeness $d_Q(u,y)$ of $m=\pi (u)$ and y. Like the Riemannian distance, which is defined as the minimal length or energy between curves linking two points in $\mathcal{M}$, $d_Q$ is defined using curves in $F\mathcal{M}$ from u to the fiber ${\pi }^{-1}(y)$ over y but now with energy weighted by a matrix ${\mathrm{\Sigma }}^{-1}$: