5.2.1 KLMS Algorithm

Consider the learning of a nonlinear function $y=f(\mathit{x})$ based on the sample sequence $\{{\mathit{x}}_i\in {\mathbb{R}}^d,y_i\in \mathbb{R}\}$, $(i=1,2,...)$. First, we transform the input data into a feature space $\mathbb{F}$ through a nonlinear mapping $\phi (.)$ induced by a Mercer kernel $\kappa (.,.)$. Then applying the well-known LMS algorithm [12] on the transformed sample sequence $\{\phi ({\mathit{x}}_i),y_i\}$, we have

$\begin{array}{c}{\mathbf{\Omega }}_i={\mathbf{\Omega }}_{i-1}-\mu \frac{\partial }{\partial {\mathbf{\Omega }}_{i-1}}{e}_i^2\hfill \\ ={\mathbf{\Omega }}_{i-1}+\eta e_i\phi ({\mathit{x}}_i),\hfill \end{array}$

where ${\mathbf{\Omega }}_i$ denotes the weight vector in feature space at iteration i, $\eta =2\mu$ is the step size parameter and $e_i$ stands for the prediction error, that is,

$\begin{array}{c}{e}_i=y_i-{\mathbf{\Omega }}_{i-1}^T\phi ({\mathit{x}}_i).\hfill \end{array}$

Identifying $\phi (\mathit{x})=\kappa (.,\mathit{x})$ and ${\mathbf{\Omega }}_i=f_i$, where $f_i$ denotes the estimated function at iteration i, and setting the initial function $f_0=0$, we can obtain the following update equations for KLMS [28]:

$\begin{array}{c}{f}_0=0,\hfill \\ e_i=y_i-f_{i-1}({\mathit{x}}_i),\hfill \\ f_i=f_{i-1}+\eta e_i\kappa ({\mathit{x}}_i,.).\hfill \end{array}$

Note that, for the reproducing kernel $\kappa (.,.)$, we have ${\langle \kappa (.,\mathit{x}),f_i\rangle }_{\mathbb{F}}=f_i(\mathit{x})$. With a radial basis kernel (e.g., Gaussian kernel), the KLMS creates a growing RBF-like network [10], with the input vectors as the centers and prediction errors (scaled by η) as the output weights, as shown in Fig. 5.1.

5.2.2 KRLS Algorithm

Next, we present the KRLS algorithm [29]. Consider the following optimization cost function:

$\ell \left(\mathbf{\Omega }\right)=\sum _{j=1}^i{\zeta }^{i-j}{e}_i^2+{\zeta }^i\gamma {\Vert \mathbf{\Omega }\Vert }^2,$

where ζ is the forgetting factor and γ is the regularization factor. Setting $\frac{\partial }{\partial \mathbf{\Omega }}\ell \left(\mathbf{\Omega }\right)=0$, one can derive

$\mathbf{\Omega }={({\mathbf{\Phi }}_i{\mathbf{B}}_{\mathbf{i}}{\mathbf{\Phi }}_i^T+\gamma {\zeta }^i\mathit{I})}^{-1}{\mathbf{\Phi }}_i{\mathbf{B}}_{\mathbf{i}}{\mathit{y}}_i,$

where ${\mathbf{\Phi }}_i=[{\mathit{\phi }}_1,\cdots ,{\mathit{\phi }}_i]$ with ${\mathit{\phi }}_i$ being ${\mathit{\phi }}_i=\phi ({\mathit{x}}_i)$, ${\mathit{y}}_i=[y_1,\cdots ,y_i]$, I is an identity matrix and ${\mathbf{B}}_{\mathbf{i}}=diag\{{\zeta }^{i-1},{\zeta }^{i-2},\cdots ,1\}$ is a diagonal matrix. Using the matrix inversion lemma [10], we have

$\mathbf{\Omega }={\mathbf{\Phi }}_i{({\mathbf{\Phi }}_i^T{\mathbf{\Phi }}_i+\gamma {\zeta }^i{\mathbf{B}}_{\mathbf{i}}^{-\mathbf{1}})}^{-1}{\mathit{y}}_i.$

Let ${\mathbf{a}}_i={({\mathbf{\Phi }}_i^T{\mathbf{\Phi }}_i+\gamma {\zeta }^i{\mathbf{B}}_{\mathbf{i}}^{-\mathbf{1}})}^{-1}{\mathit{y}}_i$ be a coefficient vector. Then the weight vector Ω can be expressed explicitly as a linear combination of the transformed data, that is, $\mathbf{\Omega }=\sum _{j=1}^i{\left({\mathbf{a}}_i\right)}_j{\mathit{\phi }}_j$, where ${\left({\mathbf{a}}_i\right)}_j$ denotes the jth component of ${\mathbf{a}}_i$. Denote ${\mathit{Q}}_i={({\mathbf{\Phi }}_i^T{\mathbf{\Phi }}_i+\gamma {\zeta }^i{\mathbf{B}}_{\mathbf{i}}^{-\mathbf{1}})}^{-1}$. We have

${{\mathit{Q}}_i}^{-1}=\left[\begin{array}{cc}\hfill {{\mathit{Q}}_{i-1}}^{-1}\hfill & \hfill {\mathbf{h}}_i\hfill \\ \hfill {{\mathbf{h}}_i}^T\hfill & \hfill {{\mathit{\phi }}_i}^T{\mathit{\phi }}_i+\gamma {\zeta }^i\hfill \end{array}\right],$

where ${\mathbf{h}}_i={{\mathbf{\Phi }}_{i-1}}^T{\mathit{\phi }}_i$. Using the block matrix inversion identity [10], we have the following update rule for ${\mathit{Q}}_i$:

${\mathit{Q}}_i={r_i}^{-1}\left[\begin{array}{cc}\hfill {\mathit{Q}}_{i-1}{r}_i+{\mathbf{z}}_i{{\mathbf{z}}_i}^T\hfill & \hfill -{\mathbf{z}}_i\hfill \\ \hfill -{{\mathbf{z}}_i}^T\hfill & \hfill 1\hfill \end{array}\right],$

where ${\mathbf{z}}_i={\mathit{Q}}_{i-1}{\mathbf{h}}_i$ and $r_i=\gamma {\zeta }^i+{{\mathit{\phi }}_i}^T{\mathit{\phi }}_i-{{\mathbf{z}}_i}^T{\mathbf{h}}_i$. Therefore, the coefficient vector ${\mathbf{a}}_i$ can be updated as

${\mathbf{a}}_i=\left[\begin{array}{c}\hfill {\mathbf{a}}_{i-1}-{\mathbf{z}}_i{r_i}^{-1}{e}_i\hfill \\ \hfill {r_i}^{-1}{e}_i\hfill \end{array}\right].$

From the above derivation, the KRLS algorithm can be briefly described in Algorithm 1.

Similar to the KLMS, the KRLS also produces a growing RBF-like network. In order to reduce the computational cost and memory requirements of KAFs, several approaches have been proposed to curb the network growth [10]. Particularly, a simple online quantization approach was proposed in [31]. The basic idea of the quantization approach is to represent the input data with a smaller data set and quantize the original centers to the nearest code vector in the quantization code book (or dictionary).

5.3.1 Correntropy

First, we give some background on information theoretic learning (ITL) [14,15]. There is a close relationship between kernel methods and ITL. Most ITL cost functions, when estimated using the Parzen kernel estimator, can be expressed in terms of inner products in kernel space $\mathbb{F}$. For example, the Parzen estimator of the quadratic information potential (QIP) from samples $x_1,x_2,\cdots ,x_N\in \mathbb{R}$ can be obtained as [14]

$QIP(X)=\frac{1}{N^2\sqrt{2\pi }\sigma }\sum _{i=1}^N\sum _{j=1}^N{\kappa }_{\sigma }(x_i-x_j),$

where ${\kappa }_{\sigma }(.)$ denotes the translation-invariant Gaussian kernel with bandwidth σ, given by

${\kappa }_{\sigma }(x-y)=\exp (-\frac{{(x-y)}^2}{2{\sigma }^2}).$

Since ${\kappa }_{\sigma }(x,y)={\langle \phi (x),\phi (y)\rangle }_{\mathbb{F}}$, we have

$QIP(X)=\frac{1}{\sqrt{2\pi }\sigma }{\Vert \frac{1}{N}\sum _{i=1}^N\phi (x_i)\Vert }_{\mathbb{F}}^2=\frac{1}{\sqrt{2\pi }\sigma }{\langle \frac{1}{N}\sum _{i=1}^N\phi (x_i),\frac{1}{N}\sum _{j=1}^N\phi (x_j)\rangle }_{\mathbb{F}},$

where ${\Vert .\Vert }_{\mathbb{F}}$ stands for the norm in $\mathbb{F}$. Thus, the estimated QIP represents the squared mean of the transformed data in kernel space.

The intrinsic link between ITL and kernel methods enlightens researchers to define new similarity measures in kernel space. As a local similarity measure in ITL, the correntropy between two random variables X and Y is defined [18] by

$V(X,Y)=\mathbf{E}[{\kappa }_{\sigma }(X-Y)]=\int {\kappa }_{\sigma }(x-y)d{F}_{XY}(x,y),$

where the expectation value is over the joint distribution $F_{XY}(x,y)$. Correntropy can easily be estimated from data as

$V(X,Y)=\frac{1}{N}\sum _{i=1}^N{\kappa }_{\sigma }(x_i-y_i).$

Correntropy can be expressed in terms of an inner product as

$V(X,Y)=\mathbf{E}\left[{\langle \phi (X),\phi (Y)\rangle }_{\mathbb{F}}\right],$

which is a generalized correlation measure in kernel space and itself is also positive definite, i.e., it defines a new kernel space for inference. In addition, by a simple Taylor series expansion on the kernel, one can see that correntropy provides a number that is the sum of all the statistical moments expressed by the kernel. In many applications this sum may be sufficient to quantify better than correlation the relationships of interest and it is much simpler to estimate than the higher-order statistical moments. Therefore, it can be considered as a new type of statistical descriptor. To date correntropy has been successfully applied to develop new adaptive filters for non-Gaussian noises [21–24], a correntropy MACE filter for pattern matching [48], robust PCA methods [19], new ICA algorithms [49], pitch detection [50] and nonlinear Granger causality analysis [51] methods with very good results. It is worth noting that many other similarity measures can be defined in kernel space such as the centered correntropy [14], correntropy coefficient [14], kernel risk sensitive loss [52] and kernel mean p-power error [53].

Based on the correntropy, one can define the correntropic loss (C-Loss) [52] as

$CL(X,Y)=1-\mathbf{E}[{\kappa }_{\sigma }(X-Y)],$

which essentially is the MSE in kernel space since we have

$CL(X,Y)=\frac{1}{2}\mathbf{E}\left[{\Vert \phi (X)-\phi (Y)\Vert }_{\mathbb{F}}^2\right].$

The empirical C-Loss can be calculated by

$\widehat{CL}(X,Y)=1-\frac{1}{N}\sum _{i=1}^N{\kappa }_{\sigma }(x_i-y_i),$

which is a function of the sample vectors $\mathit{X}={[x_1,x_2,\cdots ,x_N]}^T$ and $\mathit{Y}={[y_1,y_2,\cdots ,y_N]}^T$. If no confusion can arise, we denote the empirical C-Loss by $\widehat{CL}(\mathit{X},\mathit{Y})$. The square root of $\widehat{CL}(\mathit{X},\mathit{Y})$ is also called the Correntropy Induced Metric (CIM) between $\mathit{X}$ and $\mathit{Y}$ since it defines a metric in the sample space [18].

Fig. 5.2 shows the contours of distance from $\mathit{e}=\mathit{X}-\mathit{Y}$ to the origin in a two-dimensional space. The interesting observation from the figure is as follows. When two points are close, CIM behaves like an $L_2$ norm (which is clear from the Taylor expansion of the kernel function) and we call this area the Euclidean zone; outside of the Euclidean zone CIM then behaves like an $L_1$, which is named the Transition zone; eventually in the Rectification zone as two points are further apart, the metric is much less sensitive to the change of the distance and saturates. Specifically, note that the CIM can be used in at least two respects for robust signal processing: (1) the relative insensitivity to very large values and the $L_1$ equivalence in the transition region enables its use as an outlier-robust error measure and (2) its concave contours for large-norm vectors in the rectification regime can be used to further impose sparsity in adaptive system coefficients, which is typically done using an $L_p(p\le 1)$ norm penalization of the weight-vector norm. In both cases, the CIM has the useful property that for small-signal/weight situations the traditional quadratic measures are approximated, while for large-signal/weight conditions, it makes a smooth but sharp transition to desirable and robust $L_p(p\le 1)$ penalization. The kernel size controls the speed of transition and can be optimally determined to best fit the data through the kernel density estimation connection. It is worth noting that there is a single parameter in correntropy, the kernel bandwidth, which controls the behavior of the CIM and MCC. Intuitively, if the data size is large and the problem dimension is low, a small kernel size can be used so that MCC searches with high precision (small bias) for the maximum position of the error PDF. However, if the data size is small and/or the problem dimension large, the kernel size has to be chosen so that the bulk of the data are inside the bandwidth to ensure efficient data usage while the outliers stay outside to enable an effective attenuation (small variance).

The kernel function in correntropy can easily be extended to the generalized Gaussian function and the new measure is called the generalized correntropy [23]. Specifically, the generalized correntropy between random variables X and Y is defined by

$V_{\alpha ,\beta }(X,Y)=\mathbf{E}[G_{\alpha ,\beta }(X-Y)],$

with $G_{\alpha ,\beta }(.)$ being the generalized Gaussian density (GGD) function [23]:

$G_{\alpha ,\beta }(x)=\frac{\alpha }{2\beta \mathrm{\Gamma }(1/\alpha )}\exp (-{\left|\frac{x}{\beta }\right|}^{\alpha })={\gamma }_{\alpha ,\beta }\exp (-\lambda {\left|x\right|}^{\alpha }),$

where $\mathrm{\Gamma }(.)$ denotes the gamma function, $\alpha >0$ is the shape parameter, $\beta >0$ is the scale parameter, $\lambda =1/{\beta }^{\alpha }$ is the kernel parameter and ${\gamma }_{\alpha ,\beta }=\alpha /(2\beta \mathrm{\Gamma }(1/\alpha ))$ is the normalization constant. Obviously, when $\alpha =2$, the generalized correntropy is equivalent to the original correntropy. Note that the kernel function in the generalized correntropy is a Mercer kernel only when $0<\alpha \le 2$.

Similarly, one can define the generalized C-Loss (GC-Loss) and generalized CIM (GCIM). The GCIM also behaves like different norms (from $L_{\alpha }$ to $L_0$) of the data in different regions (see [23] for more details).

5.3.2 Maximum Correntropy Criterion

The correntropy and generalized correntropy can be used as an optimization cost in estimation-related problems. Given two random variables $X\in \mathbb{R}$, an unknown real-valued parameter to be estimated, and $Y\in {\mathbb{R}}^m$, a vector of observations (measurements), an estimator of X can be defined as a function of Y:$\hat{X}=g(Y)$, where g is solved by optimizing a certain cost function. Under the MCC, the estimator g can be solved by maximizing the correntropy (or minimizing the C-Loss) between X and $\hat{X}$, that is [45],

$g_{MCC}=\underset{g\in \mathbf{G}}{\arg \max }V(X,\hat{X})=\underset{g\in \mathbf{G}}{\arg \max }\mathbf{E}[{\kappa }_{\sigma }(X-g(Y))],$

where G stands for the collection of all measurable functions of Y. If $\forall y\in {\mathbb{R}}^m$, X has a conditional PDF $p_{X|Y}(x|y)$, then the estimation error $e=X-\hat{X}$ has PDF

$p_e(\epsilon )={\int }_{{\mathbb{R}}^m}{p}_{X|Y}(\epsilon +g(y)|y)d{F}_Y(y),$

where $F_Y(y)$ is the distribution function of Y. It follows that

$g_{MCC}=\underset{g\in G}{\arg \max }{\int }_{\mathbb{R}}{\kappa }_{\sigma }(\epsilon ){\int }_{{\mathbb{R}}^m}{p}_{X|Y}(\epsilon +g(y)|y)d{F}_Y(y)d\epsilon .$

In [45], it has been proven that the MCC estimate is a smoothed maximum a posteriori probability (MAP) estimate, which equals the mode (at which the PDF attains its maximum value). Similar results hold for the generalized MCC (GMCC) estimation.

5.4.1 Generalized Kernel Maximum Correntropy

In a way very similar to the derivation of KLMS, applying the GMCC on the transformed sample sequence $\{\phi ({\mathit{x}}_i),y_i\}$, one can derive

$\begin{array}{c}{\mathbf{\Omega }}_i={\mathbf{\Omega }}_{i-1}+\mu \frac{\partial }{\partial {\mathbf{\Omega }}_{i-1}}\exp (-\lambda {\left|e_i\right|}^{\alpha })\hfill \\ ={\mathbf{\Omega }}_{i-1}+\mu \lambda \alpha \exp (-\lambda {\left|e_i\right|}^{\alpha }){\left|e_i\right|}^{\alpha -1}sign(e_i)\phi ({\mathit{x}}_i)\hfill \\ ={\mathbf{\Omega }}_{i-1}+\eta \exp (-\lambda {\left|e_i\right|}^{\alpha }){\left|e_i\right|}^{\alpha -1}sign(e_i)\phi ({\mathit{x}}_i),\hfill \end{array}$

where the step size $\eta =\mu \lambda \alpha$. The above algorithm is called the generalized kernel maximum correntropy (GKMC) [54]. From (5.32), we have the following observations:

1) When $\alpha =2$, the GKMC becomes

${\mathbf{\Omega }}_i={\mathbf{\Omega }}_{i-1}+\eta \exp (-\lambda e_i^2)e_i\phi ({\mathit{x}}_i),$

which is the kernel maximum correntropy (KMC) algorithm developed in [21].

2) When $\lambda \to 0+$, the GKMC becomes

${\mathbf{\Omega }}_i={\mathbf{\Omega }}_{i-1}+\eta {\left|e_i\right|}^{\alpha -1}sign(e_i)\phi ({\mathit{x}}_i),$

which is the kernel LMP (KLMP) algorithm developed in [42].

3) When $\alpha =2$ and $\lambda \to 0+$, the GKMC will reduce to the KLMS algorithm

${\mathbf{\Omega }}_i={\mathbf{\Omega }}_{i-1}+\eta e_i\phi ({\mathit{x}}_i).$

In addition, one can rewrite (5.32) as

${\mathbf{\Omega }}_i={\mathbf{\Omega }}_{i-1}+{\eta }_i{e}_i\phi ({\mathit{x}}_i),$

where ${\eta }_i=\eta \exp (-\lambda {\left|e_i\right|}^{\alpha }){\left|e_i\right|}^{\alpha -2}$ is a variable step size (VSS). Thus, the GKMC can be viewed as a KLMS with VSS ${\eta }_i$. Since ${\eta }_i\to 0$ as $\left|e_i\right|\to \infty$, the learning rate of GKMC will approach zero when a large error occurs. This property ensures the robustness of the GKMC with respect to large outliers.

In summary, the GKMC algorithm is summarized in Algorithm 2, where ${\mathbf{a}}_i$ denotes the coefficient vector at iteration i, ${\left({\mathbf{a}}_i\right)}_j$ is the jth component of ${\mathbf{a}}_i$ and ${\mathit{C}}_i$ is the center set (or dictionary).

The network size of the GKMC grows linearly with the number of input data. One can use a simple quantization approach to curb the network growth [31]. In this way, the weight update Eq. (5.32) becomes

${\mathbf{\Omega }}_i={\mathbf{\Omega }}_{i-1}+\eta \exp (-\lambda {\left|e_i\right|}^{\alpha }){\left|e_i\right|}^{\alpha -1}sign(e_i)\phi (Q[{\mathit{x}}_i]),$

where $Q[.]$ denotes the quantization operator [31]:

$Q[{\mathit{x}}_i]=\left\{\begin{array}{cc}\hfill {\mathit{C}}_{j^{⁎}}(i-1)\hfill & \hfill dis({\mathit{x}}_i,{\mathit{C}}_{i-1})\le \epsilon \hfill \\ \hfill {\mathit{x}}_i\hfill & \hfill dis({\mathit{x}}_i,{\mathit{C}}_{i-1})>\epsilon \hfill \end{array}\right\},$

with ${\mathit{C}}_{i-1}$ being the codebook at iteration $i-1$, ε the quantization size and $dis({\mathit{x}}_i,{\mathit{C}}_{i-1})$ the distance between ${\mathit{x}}_i$ and ${\mathit{C}}_{i-1}$. We have

$dis({\mathit{x}}_i,{\mathit{C}}_{i-1})=\underset{1\le j\le size({\mathit{C}}_{i-1})}{\min }\Vert {\mathit{x}}_i-{({\mathit{C}}_{i-1})}_j\Vert ,$

where ${({\mathit{C}}_{i-1})}_j$ denotes the jth element (code vector) of the codebook ${\mathit{C}}_{i-1}$, $j^{⁎}$ is the index of the closest center

$j^{⁎}=\underset{1\le j\le size(\mathit{C}(i-1))}{\arg \min }\Vert {\mathit{x}}_i-{({\mathit{C}}_{i-1})}_j\Vert .$

The quantized GKMC (QGKMC) algorithm is summarized in Algorithm 3.

5.4.2 Generalized Kernel Recursive Maximum Correntropy

Next, we derive the generalized kernel recursive maximum correntropy (GKRMC) algorithm [55]. A weighted and regularized cost function under GMCC is proposed as

$\ell (\mathbf{\Omega })=\sum _{j=1}^i{\zeta }^{i-j}{G}_{\alpha ,\beta }(y_j-{\mathbf{\Omega }}^{\mathrm{T}}{\mathit{\phi }}_j)-\frac{1}{2}{\zeta }^i\gamma {\Vert \mathbf{\Omega }\Vert }^2,$

where ζ is the forgetting factor, satisfying $0\le \zeta \le 1$, γ is the regularization factor and $G_{\alpha ,\beta }(y_j-{\mathbf{\Omega }}^{\mathrm{T}}{\mathit{\phi }}_j)=\frac{\alpha }{2\beta \mathrm{\Gamma }\left(1/\alpha \right)}\exp (-{\left|\frac{y_j-{\mathbf{\Omega }}^T{\mathit{\phi }}_j}{\beta }\right|}^{\alpha })$. Setting $\frac{\partial }{\partial \mathbf{\Omega }}\ell \left(\mathbf{\Omega }\right)=0$, one can obtain the solution

${\mathrm{\Omega }}_i={({\mathrm{\Phi }}_i{\mathbf{B}}_i{{\mathrm{\Phi }}_i}^{\mathrm{T}}+\gamma {\zeta }^i{\beta }^{\alpha }\mathrm{I})}^{-1}{\mathrm{\Phi }}_i{\mathbf{B}}_i{y}_i,$

where

$\begin{array}{c}{\mathbf{B}}_i=diag\{b_1,b_2,\dots ,b_i\}\hfill \end{array}$

with

$\begin{array}{c}{b}_j={\zeta }^{i-j}\times {(y_j-{\mathrm{\Omega }}^{\mathrm{T}}{\phi }_j)}^{\alpha -2}\times \left(\frac{{\alpha }^2}{2\beta \mathrm{\Gamma }\left(1/\alpha \right)}\right)\times \exp (-{\left|\frac{y_j-{\mathrm{\Omega }}^T{\phi }_j}{\beta }\right|}^{\alpha }),j=1,2,\dots i.\hfill \end{array}$

With the matrix inversion lemma, we have

${({\mathrm{\Phi }}_i{\mathbf{B}}_i{{\mathrm{\Phi }}_i}^{\mathrm{T}}+\gamma {\zeta }^i{\beta }^{\alpha }\mathrm{I})}^{-1}{\mathrm{\Phi }}_i{\mathbf{B}}_i={\mathrm{\Phi }}_i{({{\mathrm{\Phi }}_i}^{\mathrm{T}}{\mathrm{\Phi }}_i+\gamma {\zeta }^i{\beta }^{\alpha }{{\mathbf{B}}_i}^{-1})}^{-1}.$

Thus, (5.42) can be rewritten as

${\mathrm{\Omega }}_i={\mathrm{\Phi }}_i{({{\mathrm{\Phi }}_i}^{\mathrm{T}}{\mathrm{\Phi }}_i+\gamma {\zeta }^i{\beta }^{\alpha }{{\mathbf{B}}_i}^{-1})}^{-1}{y}_i.$

The above weight vector can be expressed as a linear combination of the transformed feature vectors, where the combination coefficients can be calculated recursively. We denote

${\mathrm{\Omega }}_i={\mathrm{\Phi }}_i{\mathbf{a}}_i$ with ${\mathbf{a}}_i={({{\mathrm{\Phi }}_i}^{\mathrm{T}}{\mathrm{\Phi }}_i+\gamma {\zeta }^i{\beta }^{\alpha }{{\mathbf{B}}_i}^{-1})}^{-1}{y}_i$ being the coefficient vector. In addition, let ${\mathit{Q}}_i={({{\mathbf{\Phi }}_i}^{\mathrm{T}}{\mathbf{\Phi }}_i+\gamma {\zeta }^i{\beta }^{\alpha }\times {{\mathbf{B}}_i}^{-1})}^{-1}$. Then we have

${\mathit{Q}}_i={\left[\begin{array}{cc}\hfill {{\mathbf{\Phi }}_{i-1}}^{\mathrm{T}}{\mathbf{\Phi }}_{i-1}+\gamma {\zeta }^i{\beta }^{\alpha }{{\mathbf{B}}_{i-1}}^{-1}\hfill & \hfill {{\mathbf{\Phi }}_{i-1}}^{\mathrm{T}}{\mathit{\phi }}_i\hfill \\ \hfill {{\mathit{\phi }}_i}^{\mathrm{T}}{\mathbf{\Phi }}_{i-1}\hfill & \hfill {{\mathit{\phi }}_i}^T{\mathit{\phi }}_i+\gamma {\zeta }^i{\beta }^{\alpha }{\theta }_i\hfill \end{array}\right]}^{-1},$

where ${\theta }_i={(y_i-{\mathrm{\Omega }}^{\mathrm{T}}{\mathit{\phi }}_i)}^{\alpha -2}\times \left(\frac{{\alpha }^2}{2\beta \mathrm{\Gamma }\left(1/\alpha \right)}\right)\times \exp (-{\left|\frac{y_i-{\mathrm{\Omega }}^T{\mathit{\phi }}_i}{\beta }\right|}^{\alpha })$. The above result can be converted to

${{\mathit{Q}}_i}^{-1}=\left[\begin{array}{cc}\hfill {{\mathit{Q}}_{i-1}}^{-1}\hfill & \hfill {\mathbf{h}}_i\hfill \\ \hfill {{\mathbf{h}}_i}^{\mathrm{T}}\hfill & \hfill {{\mathit{\phi }}_i}^T{\mathit{\phi }}_i+\gamma {\zeta }^i{\beta }^{\alpha }{\theta }_i\hfill \end{array}\right],$

where ${\mathbf{h}}_i={{\mathrm{\Phi }}_{i-1}}^{\mathrm{T}}{\mathit{\phi }}_i$. Based on the following block matrix inversion identity,

${\left[\begin{array}{cc}\hfill \mathbf{A}\hfill & \hfill \mathbf{B}\hfill \\ \hfill \mathbf{C}\hfill & \hfill \mathbf{D}\hfill \end{array}\right]}^{-1}=\left[\begin{array}{cc}\hfill (\mathbf{A}-\mathbf{B}{\mathbf{D}}^{-\mathbf{1}}{\mathbf{C})}^{-\mathbf{1}}\hfill & \hfill -{\mathbf{A}}^{-\mathbf{1}}\mathbf{B}{\left(\mathbf{D}-\mathbf{C}{\mathbf{A}}^{-\mathbf{1}}\mathbf{B}\right)}^{-\mathbf{1}}\hfill \\ \hfill -{\mathbf{D}}^{-\mathbf{1}}\mathbf{C}{\left(\mathbf{A}-\mathbf{B}{\mathbf{D}}^{-\mathbf{1}}\mathbf{C}\right)}^{-\mathbf{1}}\hfill & \hfill {\left(\mathbf{D}-\mathbf{C}{\mathbf{A}}^{-\mathbf{1}}\mathbf{B}\right)}^{-\mathbf{1}}\hfill \end{array}\right],$

one can derive

${\mathit{Q}}_i={r_i}^{-1}\left[\begin{array}{cc}\hfill {\mathit{Q}}_{i-1}{r}_i+{\mathbf{z}}_i{{\mathbf{z}}_i}^{\mathrm{T}}\hfill & \hfill -{\mathbf{z}}_i\hfill \\ \hfill -{{\mathbf{z}}_i}^{\mathrm{T}}\hfill & \hfill 1\hfill \end{array}\right],$

where $r_i=\gamma {\zeta }^i{\beta }^{\alpha }{\mathit{\theta }}_i+{{\mathit{\phi }}_i}^T{\mathit{\phi }}_i-{{\mathbf{z}}_i}^{\mathrm{T}}{\mathbf{h}}_i$ and ${\mathbf{z}}_i={\mathit{Q}}_{i-1}{\mathbf{h}}_i$. Therefore, the expansion coefficients are

$\begin{array}{c}{\mathbf{a}}_i={\mathit{Q}}_i{y}_i\hfill \\ ={r_i}^{-1}\left[\begin{array}{cc}\hfill {\mathit{Q}}_{i-1}{r}_i+{\mathbf{z}}_i{{\mathbf{z}}_i}^{\mathrm{T}}\hfill & \hfill -{\mathbf{z}}_i\hfill \\ \hfill -{{\mathbf{z}}_i}^{\mathrm{T}}\hfill & \hfill 1\hfill \end{array}\right]\left[\begin{array}{c}\hfill y_{i-1}\hfill \\ \hfill y_i\hfill \end{array}\right]\hfill \\ =\left[\begin{array}{c}\hfill {\mathbf{a}}_{i-1}-{\mathbf{z}}_i{r_i}^{-1}{e}_i\hfill \\ \hfill {r_i}^{-1}{e}_i\hfill \end{array}\right],\hfill \end{array}$

where $e_i=y_i-{\mathrm{\Omega }}^{\mathrm{T}}{\mathit{\phi }}_i$. Then we have obtained the GKRMC algorithm, whose pseudocode is given in Algorithm 4.

5.5.1 Frequency Doubling Problem

First, consider the frequency-doubling problem, where the input signal is a sine wave with 20 samples per period. The desired signal is another sine wave with double frequency, disturbed by an additive noise $v_i$. With a time embedding dimension of 5, the input vector is ${\mathit{x}}_i=[x_{i-4},x_{i-3},...,x_i]$. In this example, we consider a mixture noise model which combines two independent noise processes, given by

$v_i=(1-{\delta }_i)A_i+{\delta }_i{B}_i,$

where ${\delta }_i$ is binary distributed over $\{0,1\}$, with probability $p({\delta }_i=1)=c$, $p({\delta }_i=0)=1-c$, where $0\le c\le 1$ controls the occurrence probability of the noise processes $A_i$ and $B_i$. In general, $A_i$ represents the “background noise” with small variance, while $B_i$ stands for large outliers that occur occasionally with larger variance. In the simulations below, without explicit mention we set $c=0.05$. $B_i$ is assumed to be a white Gaussian process with zero mean and variance 15. The Gaussian kernel is adopted as the Mercer kernel in kernel adaptive filters and the bandwidth is set to $\sigma =2.0$. The step sizes are chosen such that all the algorithms achieve almost the same initial convergence speed.

We choose 4000 noisy samples for training and 200 clean samples for testing (the filters are fixed during the testing). The average learning curves of the KLMS and GKMC over 100 Monte Carlo runs in terms of the testing MSEs with different values of α ($\lambda =2.0$) and distributions of $A_i$ (Gaussian and uniform) are illustrated in Fig. 5.3, and the corresponding testing MSEs at the final iteration are summarized in Table 5.1. From the simulation results we observe: (i) the GKMC can outperform the KLMS significantly, showing much stronger robustness with respect to the impulsive noises; (ii) with different values of α, the GKMC can achieve difference performances, and the best performance may be obtained when $\alpha \ne 2.0$. Note that the GKMC will reduce to the KMC algorithm when $\alpha =2.0$.

Further, we compare the performance of the GKMC and QGKMC. We set $\epsilon =0.03$, $\alpha =4$, $\lambda =2$ and $\eta =0.3$ and assume that $A_i$ is of Gaussian distribution with variance 0.04. The average learning curves over 100 runs are shown in Fig. 5.4, and the network size growth curve of the QGKMC is presented in Fig. 5.5. One can see that both algorithms achieve almost the same convergence performance but the network size of QGKMC has been significantly curbed (only slightly larger than 30 at steady state).

5.5.2 Online Nonlinear System Identification

Next, we show the performance of the GKMC and GKRMC algorithms in online nonlinear system identification. Suppose the nonlinear system is of the form [41]

$\begin{array}{c}{y}_i=(0.8-0.5exp(-y_{i-1}^2))y_{i-1}\hfill \\ -(0.3+0.9\exp (-y_{i-1}^2))y_{i-2}\hfill \\ +0.1\sin (3.1415926y_{i-1})+v_i,\hfill \end{array}$

where $v_i$ is a disturbance noise described by (5.50). The average learning curves in terms of testing MSEs over 100 Monte Carlo runs with different values of α and distributions of $A_i$ (uniform and alpha-stable) are shown in Fig. 5.6, and the corresponding testing MSEs at final iteration are listed in Table 5.2. In the simulation, 1000 noisy samples are used for training and 100 noise-free samples are used for testing (the filters are fixed during the testing). The parameters of the GKMC and GKRMC are experimentally chosen such that both achieve a desirable performance. It can be clearly seen that the GKRMC can outperform the GKMC, with faster convergence speed and lower testing MSE. However, we should note that the GKRMC is computationally more expensive than the GKMC.

5.5.3 Noise Cancellation

In this part, we consider a noise cancellation problem in signal processing. The basic structure of a noise cancellation system is shown in Fig. 5.7. The primary signal is $s_i$, whose noisy measurement $d_i$ serves as the desired signal of the system. The term ${\rho }_i$ represents an unknown white noise process and $x_i$ is its reference measurement. The signal $x_i$ can be used as an input of an adaptive filter such that the filter output is an estimate of the noise ${\rho }_i$. Clearly, the signal-to-noise ratio (SNR) can be improved by subtracting the estimated noise from $d_i$. In the simulation, the noise source is assumed to be uniformly distributed over $[-0.5,0.5]$. The interference distortion function is

$x_i={\rho }_i-0.2x_{i-1}-x_{i-1}{\rho }_{i-1}+0.1{\rho }_{i-1}+0.4x_{i-2},$

which is nonlinear with an infinite impulsive response. It is impossible to recover ${\rho }_i$ from a finite time delay embedding of $x_i$. Actually, one can rewrite the distortion function as

${\rho }_i=x_i+0.2x_{i-1}-0.4x_{i-2}+(x_{i-1}-0.1){\rho }_{i-1}.$

Obviously, the present value of the noise source ${\rho }_i$ not only depends on the reference noise measurements $[x_i,x_{i-1},x_{i-2}]$, but also on the past value ${\rho }_{i-1}$. However, the filter output $y_{i-1}$ can be used as a surrogate signal. In this case, the input vector of the adaptive filter is $[x_i,x_{i-1},x_{i-2},y_{i-1}]$. We assume the primary signal $s_i=0$ during the training phase. The system tries to reconstruct the noise source from the reference measurements. We use three nonlinear filters trained with KLMS, KRLS and GKRMC, respectively. In the simulation, the step size and the Gaussian kernel bandwidth for KLMS are 2.0 and 0.15. The regularization parameter for KRLS and GKRMC is 0.1. The Gaussian kernel bandwidth for KRLS and GKRMC are 0.17 and 0.1, respectively. The parameter β for GKRMC is set to 0.5. The ensemble learning curves over 100 Monte Carlo runs for different algorithms are illustrated in Fig. 5.8. In addition, the noise reduction factor (NR) defined by $10{\log }_{10}\{\mathbf{E}\left[{\rho }_i^2\right]/\mathbf{E}{[{\rho }_i-y_i]}^2\}$ is given in Table 5.3. In this example, the GKRMC achieves the best performance when $\alpha =6$.