8.1.3.1 Sample-Affinity Matrix (SAM)

We introduce SAM to measure the similarity among pairs of video samples in multiple views. Suppose that we are given training videos of V views, ${\{X^v,{\mathit{y}}^v\}}_{v=1}^V$. The data of the vth view $X^v$ consist of N action videos, $X^v=[{\mathit{x}}_1^v,\dots ,{\mathit{x}}_N^v]\in {\mathbb{R}}^{d\times N}$ with corresponding labels ${\mathit{y}}^v=[y_1^v,\dots ,y_N^v]$. SAM $Z\in {\mathbb{R}}^{VN\times VN}$ is interpreted as a block diagonal matrix

$Z=\text{diag}(Z_1,\dots ,Z_N),Z_i=\left(\begin{array}{cccc}\hfill 0\hfill & \hfill z_i^{12}\hfill & \hfill \dots \hfill & \hfill z_i^{1V}\hfill \\ \hfill z_i^{21}\hfill & \hfill 0\hfill & \hfill \dots \hfill & \hfill z_i^{2V}\hfill \\ \hfill ⋮\hfill & \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill z_i^{V1}\hfill & \hfill z_i^{V2}\hfill & \hfill \dots \hfill & \hfill 0\hfill \end{array}\right),$

where $\text{diag}(\cdot )$ creates a diagonal matrix, and $z_i^{uv}$ is the distance between two views in the ith sample computed by $z_i^{uv}=\exp ({\Vert {\mathit{x}}_i^v-{\mathit{x}}_i^u\Vert }^2/2c)$ parameterized by c.

Essentially, SAM Z captures within-class between-view information and between-class within-view information. Block $Z_i$ in Z characterizes appearance variations in different views within one class. This explains how an action varies if view changes. Such information makes it possible to transfer information among views and build robust cross-view features. Additionally, since the off-diagonal blocks in SAM Z are zeros, this restricts information sharing among classes in the same view. Consequently, the features from different classes but in the same view are encouraged to be distinct. This enables us to differentiate various action categories if they appear similarly in some views.

8.1.3.2 Preliminaries on Autoencoders

Our approach is based on a popular deep learning approach, called autoencoder (AE) [29,10,30]. AE maps the raw inputs x to hidden units h using an “encoder” ${\mathit{f}}_1(\cdot )$, $\mathit{h}={\mathit{f}}_1(\mathit{x})$, and then maps the hidden units to outputs using a “decoder” ${\mathit{f}}_2(\cdot )$, $\mathit{o}={\mathit{f}}_2(\mathit{h})$. The objective of learning AE is to encourage similar or identical input–output pairs, where the reconstruction loss is minimized after decoding, $\min {\sum }_{i=1}^N{\Vert {\mathit{x}}_i-{\mathit{f}}_2({\mathit{f}}_1({\mathit{x}}_i))\Vert }^2$. Here, N is the number of training samples. In this way, the neurons in the hidden layer are good representations for the inputs as the reconstruction process captures the intrinsic structure of the input data.

As opposed to the two-level encoding and decoding in AE, marginalized stacked denoising autoencoder [10] (mSDA) reconstructs the corrupted inputs with a single mapping W, $\min {\sum }_{i=1}^N{\Vert {\mathit{x}}_i-W{\tilde{\mathit{x}}}_i\Vert }^2$, where ${\tilde{\mathit{x}}}_i$ is the corrupted version of ${\mathit{x}}_i$ obtained by setting each feature to 0 with a probability p. mSDA performs m passes over the training set, each time with different corruptions. This essentially performs a dropout regularization on the mSDA [31]. By setting $m\to \infty$, mSDA effectively computes the transformation matrix W that is robust to noise using infinitely many copies of noisy data. mSDA is stackable and can be calculated in closed-form.

8.1.3.3 Single-Layer Feature Learning

The single-layer feature learner described in this subsection builds on mSDA. We attempt to learn both discriminative shared features between multiple views and private features particularly owned by one view for cross-view action classification. Considering large motion variations in different views, we incorporate SAM Z in learning shared features to balance information transfer between views so as to build more robust features.

We use the following objective function to learn shared features and private features:

$\begin{array}{cc}\hfill \underset{W,\{G^v\}}{\min }\mathcal{Q}& ,\mathcal{Q}={\Vert W\tilde{X}-XZ\Vert }_F^2+\sum _v[\alpha {\Vert G^v{\tilde{X}}^v-X^v\Vert }_F^2\hfill \\ \hfill & +\beta {\Vert W^{\mathrm{T}}{G}^v\Vert }_F^2+\gamma \text{Tr}(P^v{X}^{v}L{X}^{v\mathrm{T}}{P}^{v\mathrm{T}})],\hfill \end{array}$

where W is the mapping matrix for learning shared features, ${\{G^v\}}_{v=1}^V$ is a group of mapping matrices for learning private features particularly for each view, and $P^v=(W;G^v)$. The above objective function contains 4 terms: $\psi ={\Vert W\tilde{X}-XZ\Vert }_F^2$ learns shared features between views, which essentially reconstructs an action data from one view with the data from all views; ${\varphi }_v={\Vert G^v{\tilde{X}}^v-X^v\Vert }_F^2$ learns view-specific private features that are complementary to the shared features; $r_{1v}={\Vert W^{\mathrm{T}}{G}^v\Vert }_F^2$ and $r_{2v}=\text{Tr}(P^v{X}^{v}L{X}^{v\mathrm{T}}{P}^{v\mathrm{T}})$ are model regularizers. Here, $r_{1v}$ reduces redundancy between two mapping matrices, and $r_{2v}$ encourages the shared and private features of the same class and the same view to be similar, while $\alpha ,\beta ,\gamma$ are parameters balancing the importance of these components. Further details about these terms are discussed in the following.

Note that in cross-view action recognition, data from all the views are available in training to learn shared and private features. Data from some views are unavailable only in testing.

Shared Features. Humans can perceive an action from one view and envision what the action will look like if we observe from other views. This is possibly because we have studied similar actions before from multiple views. This inspires us to reconstruct an action data from one view (target view) using the action data from all the views (source view). In this way, information shared between views can be summarized and transferred to the target view.

We define the discrepancy between the data of the vth target view and the data of all the V source views as

$\psi =\sum _{i=1}^N\sum _{v=1}^V{\Vert W{\tilde{\mathit{x}}}_i^v-\sum _u{\mathit{x}}_i^u{z}_i^{uv}\Vert }^2={\Vert W\tilde{X}-XZ\Vert }_F^2,$

where $z_i^{uv}$ is a weight measuring the contributions of the uth view action in the reconstruction of the sample ${\mathit{x}}_i^v$ of the vth view, $W\in {\mathbb{R}}^{d\times d}$ is a single linear mapping for the corrupted input ${\tilde{\mathit{x}}}_i^v$ of all the views, $Z\in {\mathbb{R}}^{VN\times VN}$ is a sample-affinity matrix encoding all the weights $\{z_i^{uv}\}$. Matrices $X,\tilde{X}\in {\mathbb{R}}^{d\times VN}$ denote the input training matrix and the corresponding corrupted version of X, respectively [10]. The corruption essentially performs a dropout regularization on the model [31].

The SAM Z here allows us to precisely balance information transfer among views and assists learn more discriminative shared features. Instead of using equal weights [7,8], we reconstruct the ith training sample of the vth view based on the samples from all V views with different contributions. As shown in Fig. 8.3, a sample of side view (source 1) will be more identical to the one also from side view (target view) than the one from top view (source 2). Thus, more weight should be given to source 1 in order to learn more descriptive shared features for the target view. Note that SAM Z limits information sharing across samples (off-diagonal blocks are zeros) since it cannot capture view-invariant information for cross-view action recognition.

Private Features. Besides the information shared across views, there is still some remaining discriminative information that exclusively exists in each view. In order to utilize such information and make it robust to viewpoint variations, we adopt the robust feature learning in [10], and learn view-specific private features for the samples in the vth view using a mapping matrix $G^v\in {\mathbb{R}}^{d\times d}$,

${\varphi }_v=\sum _{i=1}^N{\Vert G^v{\tilde{\mathit{x}}}_i^v-{\mathit{x}}_i^v\Vert }^2={\Vert G^v{\tilde{X}}^v-X^v\Vert }_F^2.$

Here, ${\tilde{X}}^v$ is the corrupted version of the feature matrix $X^v$ of the vth view. We will learn V mapping matrices ${\{G^v\}}_{v=1}^V$ given corresponding inputs of different views.

It should be noted that using Eq. (8.3) may also capture some redundant shared information from the vth view. In this work, we reduce such redundancy by encouraging the incoherence between the view-shared mapping matrix W and view-specific mapping matrix $G^v$,

$r_{1v}={\Vert W^{\mathrm{T}}{G}^v\Vert }_F^2.$

The incoherence between W and $\{G^v\}$ enables our approach to independently exploit the discriminative information included in the view-specific features and view-shared features.

Label Information. Large motion and posture variations may appear in action data captured from various views. Therefore, the shared and private features extracted using Eqs. (8.2) and (8.3) may not be discriminative enough for actions classification with large variations. We enforce the shared and private features of the same class and same view to be similar to address the issue. A within-class within-view variance is defined to regularize the learning of the view-shared mapping matrix W and view-specific mapping matrix $G^v$ as

$\begin{array}{cc}\hfill & r_{2v}=\sum _{i=1}^N\sum _{j=1}^N[{\Vert W{\mathit{x}}_i^v-W{\mathit{x}}_j^v\Vert }^2+{\Vert G^v{\mathit{x}}_i^v-G^v{\mathit{x}}_j^v\Vert }^2]\hfill \\ \hfill & =\text{Tr}(W{X}^{v}L{X}^{v\mathrm{T}}{W}^{\mathrm{T}})+\text{Tr}(G^v{X}^{v}L{X}^{v\mathrm{T}}{G}^{v\mathrm{T}})\hfill \\ \hfill & =\text{Tr}(P^v{X}^{v}L{X}^{v\mathrm{T}}{P}^{v\mathrm{T}}).\hfill \end{array}$

Here, $L\in {\mathbb{R}}^{N\times N}$ is the label-view Laplacian matrix, $L=D-A$, D is the diagonal degree matrix with $D_{(i,i)}={\sum }_{j=1}^N{a}_{(i,j)}$, A is the adjacent matrix that represents the label relationships of training videos. The $(i,j)$th element $a_{(i,j)}$ in A is 1 if $y_i=y_j$ and 0 otherwise.

Note that since we have implicitly used this idea in Eq. (8.2), we do not need features from different views in the same class to be similar. In learning the shared feature, features of the same class from multiple views will be mapped to a new space using the mapping matrix W. Consequently, we can better represent the projected features of one sample by the features from multiple views of the same sample. Therefore, the discrepancy among views is minimized, and thus the within-class cross-view variance in Eq. (8.5) is not necessary.

Discussion. Using label information in Eq. (8.5) contributes to a supervised approach. We can also replace this term with an unsupervised one by making $\gamma =0$. We refer to the unsupervised approach as Ours-1 and the supervised approach as Ours-2 in the following discussions.

8.1.3.4 Learning

We develop a coordinate descent algorithm to solve the optimization problem in Eq. (8.1) and optimize parameters W and ${\{G^v\}}_{v=1}^V$. More specifically, in each step, one parameter matrix is updated by fixing the others, and computing the derivative of $\mathcal{Q}$ w.r.t. to the parameter and setting it to 0.

Update W. Parameters ${\{G^v\}}_{v=1}^V$ are fixed in updating W, which can be updated by setting the derivative $\frac{\partial \mathcal{Q}}{\partial W}=0$, yielding

$\begin{array}{cc}\hfill W=[\sum _v& {(\beta G^v{G}^{v\mathrm{T}}+\gamma X^{v}L{X}^{v\mathrm{T}}+I)]}^{-1}\hfill \\ \hfill & \cdot (XZ{\tilde{X}}^{\mathrm{T}}){[\tilde{X}{\tilde{X}}^{\mathrm{T}}+I]}^{-1}.\hfill \end{array}$

It should be noted that $XZ{\tilde{X}}^{\mathrm{T}}$ and $\tilde{X}{\tilde{X}}^{\mathrm{T}}$ are computed by repeating the corruption $m\to \infty$ times. By the weak law of large numbers [10], $XZ{\tilde{X}}^{\mathrm{T}}$ and $\tilde{X}{\tilde{X}}^{\mathrm{T}}$ can be computed by their expectations $E_p(XZ{\tilde{X}}^{\mathrm{T}})$ and $E_p(\tilde{X}{\tilde{X}}^{\mathrm{T}})$ with the corruption probability p, respectively.

Update $G^v$. Fixing W and ${\{G^u\}}_{u=1,u\ne v}^V$, parameter $G^v$ is updated by setting the derivative $\frac{\partial \mathcal{Q}}{\partial G^v}=0$, giving

$\begin{array}{cc}\hfill G^v=(\beta & {W{W}^{\mathrm{T}}+\gamma X^{v}L{X}^{v\mathrm{T}}+I)}^{-1}\hfill \\ \hfill & \cdot (\alpha X^v{\tilde{X}}^{v\mathrm{T}}){[\alpha {\tilde{X}}^v{\tilde{X}}^{v\mathrm{T}}+I]}^{-1}.\hfill \end{array}$

Similar to the procedure of updating W, $X^v{\tilde{X}}^{v\mathrm{T}}$ and ${\tilde{X}}^v{\tilde{X}}^{v\mathrm{T}}$ are computed by their expectations with corruption probability p.

Convergence. Our learning algorithm iteratively updates W and ${\{G^v\}}_{v=1}^V$. The problem in Eq. (8.1) can be divided into $V+1$ subproblems, each of which is a convex problem with respect to one variable. Therefore, by solving the subproblems alternatively, the learning algorithm is guaranteed to find an optimal solution to each subproblem. Finally, the algorithm will converge to a local solution.

8.1.3.5 Deep Architecture

Inspired by the deep architecture in [10,32], we also design a deep model by stacking multiple layers of feature learners described in Sect. 8.1.3.3. A nonlinear feature mapping is performed layer by layer. More specifically, a nonlinear squashing function $\sigma (\cdot )$ is applied on the output of one layer, $H_w=\sigma (WX)$ and $H_g^v=\sigma (G^v{X}^v)$, resulting in a series of hidden feature matrices.

A layer-wise training scheme is used in this work to train the networks ${\{W_k\}}_{k=1}^K$, ${\{G_k^v\}}_{k=1,v=1}^{K,V}$ with K layers. Specifically, the outputs of the fth layer $H_{kw}$ and $H_{kg}^v$ are used as the input to the $(k+1)$th layer. The mapping matrices $W_{k+1}$ and ${\{G_{k+1}^v\}}_{v=1}^V$ are then trained using these inputs. For the first layer, the inputs $H_{0w}$ and $H_{0g}^v$ are the raw features X and $X^v$, respectively. More details are shown in Algorithm 8.1.

8.1.4.1 IXMAS Dataset

Dense trajectory and histogram of oriented optical flow [35] are extracted from videos. A dictionary of size 2000 is built for each type of features using k-means. We use the bag-of-words model to encode these features, and represent each video as a feature vector.

We adopt the same leave-one-action-class-out training scheme in [25,7,8] for fair comparison. At each time, one action class is used for testing. In order to evaluate the effectiveness of the information transfer in our approaches, all the videos in this action are excluded from the feature learning procedure including k-means and our approaches. Note that these videos can be seen in training the action classifiers. We evaluate both the unsupervised approach (Ours-1) and the supervised approach (Ours-2).

8.1.4.2 Daily and Sports Activities Data Set

8.2.3.1 Architecture Overview

Our method takes an RGB-D video v as input and outputs the corresponding action label. Our HCRNN is employed to find a transform h that maps the RGB-D video into a feature vector x, $x=h(v)$. Feature vector x is then fed into a classifier, and the action label y is obtained. We treat an RGB-D video as multichannel data, and extract gray, gradient, optical flow and depth data from RGB and depth modalities. HCRNN is applied to each of these channels.

3D-CNN. The lower part of our HCRNN is a 3D-CNN model that extracts features hierarchically. The 3D-CNN (Fig. 8.6) in this work has five stages: 3D convolution (Sect. 8.2.3.2), absolute rectification, local normalization, average pooling and subsampling. We sample N 3D patches of size $(s_r,s_c,s_t)$ (height, width, frame) from each channel with stride size $s_p$. The 3D-CNN g takes these patches as the input, and outputs a K-dimensional vector u for each patch, $g:{\mathbb{R}}^S\to {\mathbb{R}}^K$. Here, K is the number of learned filters and $S=s_r\times s_c\times s_t$.

Rich motion and geometry change information is captured using the 3D convolution in 3D-CNN. Each video of size (height, width, frame) $d_I$ is convolved with K filters of size $d_P$, resulting in K filter responses of dimensionality N (N is the number of patches extracted from one channel of a video). Then, absolute rectification is performed which applies absolute value function to all the components of the filter responses. This step is followed by local contrast normalization (LCN). LCN module performs local subtractive and divisive normalization, enforcing a sort of local competition between adjacent features in a feature map, and between features at the same spatiotemporal location in different feature maps. To improve the robustness of features to small distortions, we add average pooling and subsampling modules to 3D-CNN. Patch features whose locations are within a small spatiotemporal neighborhood are averaged and pooled to generate one parent feature of dimensionality K.

In this chapter, we augment gray and depth feature maps (channels) with gradient-x, gradient-y, optflow-x, and optflow-y as in [59]. The gradient-x and gradient-y feature maps are computed by computing the gradient along the horizontal and vertical directions, respectively. The optflow-x and optflow-y feature maps are computed by running the optical flow algorithm and separating the horizontal and vertical flow field data.

3D-RNN. The 3D-RNN model is to hierarchically learn compositional features. The graphical illustration of 3D-RNN is shown in Fig. 8.7. It merges a spatiotemporal block of patch feature vectors and generate a parent feature vector. The input for 3D-RNN is a $K\times M$ matrix, where M is the number of feature vectors generated by 3D-CNN ($M\ne N$ since we apply subsampling in 3D-CNN). The output of 3D-RNN is a K-dimensional vector, which is the feature for one channel of the video. We adopt a tree-structured 3D-RNN with J layers. At each layer, child vectors, whose spatiotemporal locations are within a 3D block, are merged into one parent vector. This procedure continues to generate one parent vector in the top layer. We concatenate feature vectors of all the channels generated by 3D-RNN, and derive a KC-dimensional vector as the action feature given C channels of data.

8.2.3.2 3D Convolutional Neural Networks

We use 3D-CNN to extract features from RGB-D action videos. 2D-CNNs have been successfully used in 2D images, such as object recognition [52–54] and scene understanding [57,58]. In these methods, 2D convolution are adopted to extract features from local neighborhood on the feature map. Then an addictive bias is added and a sigmoid function is used for feature mapping. However, in action recognition, a 3D convolution is desired as successive frames in videos encode rich motion information. In this work, we develop a new 3D convolution operation for RGB-D videos.

The 3D convolution in the 3D-CNN is achieved by convolving a filter on 3D patches extracted from RGB-D videos. Consequently, local spatiotemporal motion and structure information can be well captured in the convolution layer. It also captures motion and structure relationships of body parts in a local neighborhood, e.g., arm–hand and leg–foot. Suppose p is a 3D spatiotemporal patch randomly extracted from a video. We apply a nonlinear mapping to map p onto the feature map in the next layer,

$g_k(p)=\max ({\overline{d}}_k-{\Vert p-z_k\Vert }_2,0),$

where ${\overline{d}}_k=\frac{1}{K}{\sum }_k{\Vert x-z_k\Vert }_2$ is the averaged distance of the sample p to all the filters, $z_k$ is the kth filter, and K is the number of the learned filters. Note that the convolution in Eq. (8.8) is different from [59,62], but philosophically similar; ${\overline{d}}_k$ can be considered as the bias, ${\Vert p-z_k\Vert }_2$ is similar to the convolution operation, which is a similarity measure between the filter $z_k$ and the patch p. The $\max (\cdot )$ function is a nonlinear mapping function as the sigmoid function. The filters in Eq. (8.8) are easily trained in unsupervised fashion (Sect. 8.2.3.5) by running k-means [63].

After 3D convolution, rectification, local contrast normalization and averaged down sampling are applied as in object recognition but they are performed on 3D patches.

3D-CNN generates a list of K-dimensional vectors $u_i^{rct}\in U$ ($i=1,\dots ,M$), where r, c, and t are the locations of the vector in row, column and temporal dimensions, respectively, U is the set of vectors generated by 3D-CNN. Each vector in U is the feature for a patch. All these patch features are then given as inputs to 3D-RNN to compose high-order features.

Our 3D-CNN extracts discriminative motion and geometry changes information from RGB-D data. It also captures relationships of human body parts in local neighborhood, and allows body part to be deformable in actions. Therefore, the learned features are robust to pose variations in RGB-D data.

8.2.3.3 3D Recursive Neural Networks

The idea of recursive neural networks is to learn hierarchical features by applying the same neural network recursively in a tree structure. In our case, 3D-RNN can be regarded as combining convolution and pooling over 3D patches into one efficient, hierarchical operation.

We use balanced fixed-tree structure of 3D-RNN. Compared with previous RNN approaches, the tree structure offers high speed operation, and making use of parallelization. In our tree-structure 3D-RNN, each leaf node is a K-dimensional vector, which is an output of the 3D-CNN. At each layer, the 3D-RNN merges adjacent vectors into one vector. As this process repeats, high-order relationships and long-range dependencies of body parts can be encoded in the learned feature.

3D-RNN takes a list of K-dimensional vectors $u_i^{rct}\in U$ generated by 3D-CNN ($i=1,\dots ,M$) as inputs, and recursively merges a block of vectors into one parent vector $q\in {\mathbb{R}}^K$. We define a 3D block of size $b_r\times b_c\times b_t$ as a list of adjacent vectors to be merged. For example, if $b_r=b_c=b_t=2$, then $B=8$ vectors will be merged. We define the merging function as

$q=f\left(W\left[\begin{array}{c}\hfill u_1\hfill \\ \hfill ⋮\hfill \\ \hfill u_B\hfill \end{array}\right]\right).$

Here, W is the parameter of size $K\times BK$ ($B=b_r\times b_c\times b_t$), $f(\cdot )$ is a nonlinear function (e.g., $\tanh (\cdot )$). Bias term is omitted here as it does not affect performance.

3D-RNN is a tree with multiple layers, where the jth layer composes high-order features over the $(j-1)$th layer. In the jth layer, all the blocks of vectors in the $(j-1)$th are merged into one parent vector using the same weight W in Eq. (8.9). This process is repeated until only one parent vector x remains. Fig. 8.7 shows an example of a pooled CNN output of size $K\times 2\times 2\times 2$ and an RNN tree structure with blocks of 8 children, $u_1,\dots ,u_8$.

We apply 3D-RNN to C channel data separately, and obtain C parent vectors from 3D-RNN $x_c$, $c=1,\dots ,C$. Each parent vector $x_c$ in 3D-RNN is a K-dimensional vector, computed from one channel data of an RGB-D video. The vectors from all the channels are then concatenated into a long vector to encode rich motion and structure information for the RGB-D video. Finally, this feature is fed into the softmax classifier for action classification.

The feature learned by 3D-RNN captures high-order relationships of body parts and encodes long-range dependencies of body parts. Therefore, human actions can be well represented and can be classified accurately.

8.2.3.4 Multiple 3D-RNNs

3D-RNN abstracts high-order features using the same weight W in a recursive way. The weight W, randomly learned, expresses the knowledge of which vector is more important in the parent vector for the classification task. It may not be accurate due to the randomness of W.

This problem can be solved by using multiple 3D-RNNs. Similar to [54], we use multiple 3D-RNNs with different random weights. Consequently, different importance of adjacent vectors can be well captured by different weights, and high quality feature vectors can then be produced. We concatenate vectors generated by multiple 3D-RNNs to feed into the softmax classifier.

8.2.3.5 Model Learning

Unsupervised Learning of 3D-CNN Filters. CNN models can be learned using supervised or unsupervised approaches [64,59]. Since convolution operates on millions of 3D patches, using back propagation and fine tuning the entire networks may not be practical or efficient. Instead, we train our 3D-CNN model using an unsupervised approach.

Inspired by [63], we learn 3D-CNN filters in an unsupervised way by clustering random 3D patches. We treat multichannel data (gray, gradient, optical flow, and depth) as separated feature maps, and randomly generate spatiotemporal 3D patches from each channel. The extracted 3D patches are then normalized and whitened. Finally, these 3D patches are clustered to build K cluster centers $z_k$, $k=1,\dots ,K$, which are used in the 3D convolution (Eq. (8.8)).

Random Weights for 3D-RNN. Recent work [54] shows that RNNs with random weights can also generate features with high discriminability. We follow [54] to learn 3D-RNNs with random weights W. We show that learning RNNs with random weights provides an efficient, yet powerful model for action recognition from depth camera.

8.2.3.6 Classification

As described in Sect. 8.2.3.4, features generated by multiple 3D-RNNs will be concatenated to produce the feature vector x for the depth video. We train a multiclass softmax classifier to classify the depth action x,

$f(x,y)=\frac{\exp ({\mathit{\theta }}_y^{\mathrm{T}}{x}_i)}{{\sum }_{l\in \mathcal{Y}}\exp ({\mathit{\theta }}_l^{\mathrm{T}}{x}_i)},$

where ${\mathit{\theta }}_y$ is the parameter for class y. The prediction is performed by taking the $\arg \max$ of the vector whose lth element is $f(x,l)$, $y^{⁎}=\arg {\max }_{l}f(x,l)$. The multiclass cross entropy loss function is defined in learning the model parameter θ for all the classes. The model parameter θ is learned using the limited-memory variable-metric gradient ascent (BFGS) method.

8.2.4.1 MSR-Gesture3D Dataset

MSR-Gesture3D dataset is a hand gesture dataset containing 336 depth sequences captured by a depth camera. There are 12 categories of hand gestures in the dataset: “bathroom”, “blue”, “finish”, “green”, “hungry”, “milk”, “past”, “pig”, “store”, “where”, “j”, and “z”. This is a challenging dataset due to the self-occlusion issue and visually similarity. Our HCRNN takes an input video of size $80\times 80\times 18$. The number of filters in 3D-CNN is set to 256 and the number of 3D-RNNs is set to 16. The kernel size (filter size) in 3D-CNN is $6\times 6\times 4$, and the receptive filed size in 3D-RNN is $2\times 2\times 2$. As in [65], only depth frames are used in the experiments. The leave-one-out cross validation is employed to in evaluation.

Fig. 8.9 shows the confusion matrix of HCRNN on the MSR-Gesture3D dataset. Our method achieves 93.75% accuracy in classifying hand gestures. Our method misses some of the examples in “ASL Past” and “ASL Store”, “ASL Finish” and “ASL Past”, and “ASL Blue” and “ASL J” due to their visual similarities. As shown in Fig. 8.9, our method can recognize visually similar hand gestures as the HCRNN discovers discriminative features and abstracts expressive high-order features for the task. HCRNN confuses some of examples shown in Fig. 8.9 due to the self-occlusion and intra-class motion variations.

We compare our HCRNN with [44,66,65,67] on the MSR-Gesture3D dataset. Methods in [44,66,65] are particularly designed for depth sequences and [67] proposed HoG3D descriptor which was originally designed for color sequences. Compared with these methods that are based on hand-crafted features, the HCRNN learns features from data. Results in Table 8.5 indicate that our method outperforms all these comparison methods. Our method learns features from data, which better represent intra-class variations and inter-class similarities, and thus achieves better performance. In addition, HCRNN encodes high-order context information of body part, and allows deformable body parts. These two benefits help improve the expressiveness of the learned features.

8.2.4.2 MSR-Action3D Dataset

MSR-Action3D dataset [40] consists of 20 classes of human actions: “bend”, “draw circle”, “draw tick”, “draw x”, “forward kick”, “forward punch”, “golf swing”, “hammer”, “hand catch”, “hand clap”, “high arm wave”, “high throw”, “horizontal arm wave”, “jogging”, “pick up & throw”, “side boxing”, “side kick”, “tennis serve”, “tennis swing”, and “two hand wave”. A total of 567 depth videos are contained in the dataset which are captured using a depth camera.

In this dataset, the background is preprocessed to remove the discontinuities caused by the undefined depth regions. However, it is still challenging since many actions are visually very similar. The same training/testing splits in [44] is adopted in this experiment, i.e., the videos of the first five subjects (295 videos) are used for training and the remaining 272 videos are for testing. Our HCRNN takes an input video of size $120\times 160\times 30$. The number of filters in 3D-CNN is set to 256, and the number of 3D-RNNs is set to 32. The kernel size (filter size) in 3D-CNN is $6\times 6\times 4$, and the receptive filed size in 3D-RNN is $2\times 2\times 2$.

The confusion matrix of HCRNN is displayed in Fig. 8.10A. Our method achieves 90.07% recognition accuracy on the MSR-Action3D dataset. Confusions mostly occur between visually similar actions, e.g., “horizontal hand wave” and “clap”, “hammer” and “tennis serve”, and “draw x” and “draw tick”. The learned filters used in the experiment are also illustrated in Fig. 8.10B. Our filter learning method discovers various representative patterns, which can be used to accurately describe local 3D patches.

We compare with methods particularly designed for depth sequences [40,65,66,68], as well as conventional action recognition methods that use spatiotemporal interest point detectors [34,49,67]. Results in Table 8.6A show that our method outperforms all the comparison methods. Our method achieves 90.07% recognition accuracy, which is higher than that of the state-of-the-art methods [42,14]. It should be noted that our method does not utilize the skeleton tracker, and yet outperforms the skeleton-based method [40]. Table 8.6B shows the recognition of HCRFF with different number of 3D-RNNs. The HCRNN achieves the best performance at $n_r=32$. HCRNN obtains the worse performance with $n_r=1$. With more 3D-RNNs, HCRNN achieves higher accuracy until 32 3D-RNNs are used. Then, with more 3D-RNNs, its performance degrades due to the overfitting problem.