3.1.1 Notation

In this chapter matrices are represented by boldface capital letters, i.e., $\mathbf{A}\in {\mathbb{R}}^{M\times N}$, while $\mathrm{Tr}(\cdot )$ denotes the trace of a matrix. All vectors are column vectors, denoted by boldface lowercase letters, like $\mathbf{w}\in {\mathbb{R}}^{M\times 1}={[w[0],w[1],\dots ,w[M-1]]}^T$, where $w[i]$ denotes the ith individual entry of w. In the recursive algorithm definition, a discrete-time subscript index n is added. For example, the weight vector, calculated according to some law, is written as ${\mathbf{w}}_{n+1}={\mathbf{w}}_n+\mathrm{\Delta }{\mathbf{w}}_n$. In the case of signal regression, vectors are indicated as ${\mathbf{x}}_n\in {\mathbb{R}}^{M\times 1}={[x[n],x[n-1],\dots ,x[n-M+1]]}^T$. Moreover, regarding the interpolated nonlinearity, the LUT control points are denoted ${\mathbf{q}}_n\in {\mathbb{R}}^{Q\times 1}={[q_1,q_2,\dots ,q_Q]}^T$, where $q_j$ is the jth control point, while the ith span is denoted by ${\mathbf{q}}_{i,n}\in {\mathbb{R}}^{4\times 1}={[q_i,q_{i+1},q_{i+2},q_{i+3}]}^T$. Finally, the Euclidean norm and the transpose of a vector are denoted by $\Vert \cdot \Vert$ and ${(\cdot )}^T$, respectively.

3.2.1 Uniform Cubic Spline

The recursive Eq. (3.3) can be entirely pre-calculated with considerable computational saving, by imposing a uniform distribution knot-vector, i.e., constant sampling step of the abscissa $\mathrm{\Delta }x=q_{x,i}-q_{x,i+1}$ [39]. This implies that the local polynomial ${\phi }_i(u)$ in (3.4) can be written in a very simple form. A similar approach was successfully used in multilayer feed-forward neural networks [53].

Regarding the choice of the optimal degree P of a spline basis, a good compromise between flexibility and approximation behavior is represented by uniform cubic spline nonlinearities [54], as they provide good approximation capabilities avoiding the problem of over-fitting.

In the case of our interest where $P=3$, the precalculation of the recursion (3.3) provides [39,54,53,40]

$N_i^3(u)=\{\begin{array}{cc}\frac{1}{6}[-{(u-q_i)}^3+3{(u-q_i)}^2-3(u-q_i)+1],\hfill & q_{x,i}⩽u<q_{x,i+1},\hfill \\ \frac{1}{6}[3{(u-q_{i+1})}^3-6{(u-q_{i+1})}^2+4,],\hfill & q_{x,i+1}⩽u<q_{x,i+2},\hfill \\ \frac{1}{6}[-3{(u-q_{i+2})}^3+3{(u-q_{i+2})}^2+3(u-q_{i+2})+1],\hfill & q_{x,i+2}⩽u<q_{x,i+3},\hfill \\ \frac{1}{6}{(u-q_{i+3})}^3,\hfill & q_{x,i+3}⩽u<q_{x,i+4},\hfill \end{array}$

which can be reformulated in a matrix form as

${\phi }_i(u)=[\begin{array}{cccc}\hfill u^3\hfill & \hfill u^2\hfill & \hfill u\hfill & \hfill 1\hfill \end{array}]\mathbf{C}\left[\begin{array}{c}\hfill q_i\hfill \\ \hfill q_{i+1}\hfill \\ \hfill q_{i+2}\hfill \\ \hfill q_{i+3}\hfill \end{array}\right],$

for $i=0,1,\dots ,Q$ and $u\in [0,1)$ and where the basis matrix C is defined as

$\mathbf{C}=\frac{1}{6}\left[\begin{array}{cccc}\hfill -1& \hfill 3& \hfill -3& \hfill 1\hfill \\ \hfill 3& \hfill -6& \hfill 3& \hfill 0\hfill \\ \hfill -3& \hfill 0& \hfill 3& \hfill 0\hfill \\ \hfill 1& \hfill 4& \hfill 1& \hfill 0\hfill \end{array}\right].$

Imposing different constraints to the approximation relationship, several spline bases with different properties can be evaluated in a similar manner. An example of such a basis is the Catmull–Rom (CR) spline [55], very important in many applications. The interpolation scheme is the same of (3.5), while the CR-spline basis matrix C has the form

$\mathbf{C}=\frac{1}{2}\left[\begin{array}{cccc}\hfill -1& \hfill 3& \hfill -3& \hfill 1\\ \hfill 2& \hfill -5& \hfill 4& \hfill -1\\ \hfill -1& \hfill 0& \hfill 1& \hfill 0\\ \hfill 0& \hfill 2& \hfill 0& \hfill 0\end{array}\right].$

Note that CR-splines, initially developed for computer graphics purposes, represent a local interpolating scheme. The CR-spline, in fact, specifies a curve that passes through all of the control points, a feature which is not necessarily true for other spline methodologies. Overall, the CR-spline results in a more local approximation with respect to the B-spline.

For both the CR- and B-basis the curve has a continuous first derivative. For the B-spline the second derivative is also a continuous function, while for the CR spline the second derivative is not continuous at the knots. However, there is a regularization property common to most of the polynomial spline basis set (CR- and B-spline included) called variation diminishing property [39], which ensures the absence of unwanted oscillations of the curve between two consecutive control points, as well as the exact representation of linear segments.

Note also that the entries of the matrix C satisfy an important general property: the sum of all the entries is equal to one. This property is a direct consequence of the fact that the spline basis functions, derived from Bézier curves, are a generalization of the Bernstein polynomials and hence they represent a partition of unit (i.e., the sum of all basis function is one) [39,51].

In the next section, we present the main ideas on the SAFs and the derivation of the basic SAF models.

3.3.1 Wiener SAF

The Wiener SAF (WSAF) architecture consists of the cascade of a linear filter, represented by the filter coefficient weight vector ${\mathbf{w}}_n$, and a memoryless nonlinear function implemented by a spline interpolation scheme whose knots are denoted by the vector ${\mathbf{q}}_n$. The Wiener architecture is shown in Fig. 3.1, where $s[n]$ denotes the output of the linear filter and it is the input to the spline function.

With reference to Fig. 3.1, let us define the output error $e[n]$ as

$e[n]=d[n]-y[n]=d[n]-{\phi }_i(u_n),$

where $d[n]$ is the reference signal, i.e., the output of the model to be identified. The online learning algorithm can be derived by the minimization of (3.11). We proceed by applying the stochastic gradient adaptation [40], obtaining the LMS iterative algorithm

${\mathbf{w}}_{n+1}={\mathbf{w}}_n+{\mu }_w[n]e[n]{\phi }_i^{\prime }(u){\mathbf{x}}_n,$

for the weights of the linear filter and

${\mathbf{q}}_{i,n+1}={\mathbf{q}}_{i,n}+{\mu }_q[n]e[n]{\mathbf{C}}^T{\mathbf{u}}_n,$

for the spline control points. The parameters ${\mu }_w[n]$ and ${\mu }_q[n]$ represent the learning rates for the weights and for the control points, respectively, and, for simplicity, incorporate the other constant values.

3.3.2 Hammerstein SAF

The Hammerstein SAF (HSAF) architecture consists of the cascade of a memoryless nonlinear function implemented by a spline interpolation scheme whose knots are denoted by the vector ${\mathbf{q}}_n$ and a linear filter, represented by the filter coefficient weight vector ${\mathbf{w}}_n$. The Hammerstein architecture is shown in Fig. 3.2, where $s[n]$ denotes the output of the nonlinear function.

With reference to Fig. 3.2, we can define the output error $e[n]$ as

$e[n]=d[n]-y[n]=d[n]-{\mathbf{w}}_n^T\cdot {\mathbf{s}}_n,$

where $d[n]$ is the reference signal, ${\mathbf{s}}_n={[s[n],s[n-1],\dots ,s[n-M+1]]}^T$, and each sample $s[n-k]$ is evaluated by using (3.9).

As in the Wiener case, the learning algorithm is derived by minimizing the CF (3.11). We proceed by applying the stochastic gradient adaptation as done in [41], obtaining the LMS iterative algorithm

${\mathbf{w}}_{n+1}={\mathbf{w}}_n+{\mu }_w[n]e[n]{\mathbf{s}}_n$

for the weights of the linear filter and

${\mathbf{q}}_{i,n+1}={\mathbf{q}}_{i,n}+{\mu }_q[n]e[n]{\mathbf{C}}^T{\mathbf{U}}_{i,n}{\mathbf{w}}_n$

for the spline control points. In (3.17), ${\mathbf{U}}_{i,n}\in {\mathbb{R}}^{4\times M}=[{\mathbf{u}}_{i,n},{\mathbf{u}}_{i,n-1},\dots ,{\mathbf{u}}_{i,n-M+1}]$ is a matrix which collects M past vectors ${\mathbf{u}}_{i,n-k}$, where each vector assumes the value of ${\mathbf{u}}_{n-k}$ if evaluated in the same span i of the current sample input $x[n]$ or in an overlapped span; otherwise it is a zero vector.

As in the Wiener case, the learning rates ${\mu }_w[n]$ and ${\mu }_q[n]$ incorporate the other constant values.

3.3.3 Sandwich 1 SAF

The Sandwich 1 (S1SAF) architecture [43] consists of the cascade of a memoryless nonlinear function implemented by a spline interpolation scheme whose knots are denoted by the vector ${\mathbf{q}}_n^{(1)}$, a linear filter, represented by the filter coefficient weight vector ${\mathbf{w}}_n$, and a second memoryless nonlinear function implemented by a spline interpolation scheme whose knots are denoted by the vector ${\mathbf{q}}_n^{(2)}$. The S1 architecture is shown in Fig. 3.3, where $s[n]$ denotes the output of the first nonlinear function and $r[n]$ is the output of the linear filter.

With reference to Fig. 3.3, we can define the output error $e[n]$ as

$e[n]=d[n]-y[n]=d[n]-{\phi }^{(2)}({\mathbf{w}}_n^T\cdot {\mathbf{s}}_n),$

where $d[n]$ is the reference signal. As in the previous cases, the learning algorithm is derived by minimizing the CF (3.11). We proceed by applying the stochastic gradient adaptation as done in [43], obtaining the LMS iterative algorithm

${\mathbf{q}}_{i,n+1}^{(2)}={\mathbf{q}}_{i,n}^{(2)}+{\mu }_q^{(2)}[n]e[n]{\mathbf{C}}^T{\mathbf{u}}_n^{(2)}$

for the control points of the second nonlinearity,

${\mathbf{w}}_{n+1}={\mathbf{w}}_n+{\mu }_w[n]e[n]{\phi }_i^{\prime }(u_n^{(2)}){\mathbf{s}}_n$

for the weights of the linear filter and

${\mathbf{q}}_{i,n+1}^{(1)}={\mathbf{q}}_{i,n}^{(1)}+{\mu }_q^{(1)}[n]e[n]{\phi }_i^{\prime }(u_n^{(2)}){\mathbf{C}}^T{\mathbf{U}}_{i,n}^{(1)}{\mathbf{w}}_n$

for the control points of the first nonlinearity.

As in the previous cases, the learning rates ${\mu }_w[n]$, ${\mu }_q^{(1)}[n]$ and ${\mu }_q^{(2)}[n]$ incorporate the other constant values.

3.3.4 Sandwich 2 SAF

The Sandwich 2 (S2SAF) architecture [43] consists of the cascade of a linear filter, represented by the filter coefficient weight vector ${\mathbf{w}}_n^{(1)}$, a memoryless nonlinear function implemented by a spline interpolation scheme whose knots are denoted by the vector ${\mathbf{q}}_n$ and a second linear filter, represented by the filter coefficient weight vector ${\mathbf{w}}_n^{(2)}$. The S2 architecture is shown in Fig. 3.4, where $s[n]$ denotes the output of the first linear filter and $r[n]$ is the output of the nonlinear function.

With reference to Fig. 3.4, we can define the output error $e[n]$ as

$e[n]=d[n]-y[n]=d[n]-{{\mathbf{w}}_n^{(2)}}^T\cdot \phi ({{\mathbf{w}}_n^{(1)}}^T\cdot {\mathbf{x}}_n),$

where again $d[n]$ is the reference signal. As in the previous cases, the learning algorithm is derived by minimizing the CF (3.11). We proceed by applying the stochastic gradient adaptation as done in [43], obtaining the LMS iterative algorithm

${\mathbf{w}}_{n+1}^{(2)}={\mathbf{w}}_n^{(2)}+{\mu }_w^{(2)}[n]e[n]{\mathbf{r}}_n$

for the weights of the second linear filter,

${\mathbf{q}}_{i,n+1}={\mathbf{q}}_{i,n}+{\mu }_q[n]e[n]{\mathbf{C}}^T{\mathbf{U}}_{i,n}{\mathbf{w}}_n^{(2)}$

for the control points of the nonlinearity and

${\mathbf{w}}_{n+1}^{(1)}={\mathbf{w}}_n^{(1)}+{\mu }_w^{(1)}[n]e[n]{\phi }_i^{\prime }(u_n){\mathbf{X}}_n{\mathbf{w}}_n^{(2)}$

for the weights of the first linear filter.

As in the previous cases, the learning rates ${\mu }_w^{(1)}[n]$, ${\mu }_w^{(2)}[n]$ and ${\mu }_q[n]$ incorporate the other constant values.

3.3.5 Other Architectures

It is possible to derive other kinds of block-oriented architectures by suitable combinations of the basic architectures introduced in the previous subsections, thus obtaining particular cascade and/or parallel systems that can be successfully used to identify specific types of nonlinear systems or by using an IIR linear filter instead of a FIR one, like done in [42]. A more general situation of particular interest is the case of a convex (and affine) combination of the former architectures, like the ones proposed in [56] for another important class of nonlinear adaptive filters. In this scenario, the whole combined filter is able to automatically detect the subsystem that can provide better results. The extension of SAF to a combined scenario will be postponed to future works.

In a similar fashion, the work [46] introduces a hybrid nonlinear spline filter, which is designed as a cascade of an adaptive spline function and a single-layer adaptive nonlinear network, devoted to the problem of removing noise in the acoustic domain. Another work worthy to consider is [57], where the HSAF architecture is generalized to a Functional Link network. The proposed work shows very interesting results also in terms of classification ability.

Another type of generalization of SAFs is based on a suitable choice of the learning schemes. In most works, SAFs are adapted by using a general LMS strategy. However, this situation cannot be the best one. An extension to the more general normalized LMS (NLMS) is provided in [48], while the extension to the correntropy criterion can be found in [49]. The latter strategy is particularly useful when data are corrupted by impulsive noise or contain large outliers.

Finally, [50] derives a diffused version of the WSAF (D-SAF) by applying ideas from the Diffusion Adaptation (DA) framework. DA algorithms allow a network of agents to collectively estimate a parameter vector, by jointly minimizing the sum of their local cost functions. Experimental evaluations show that the D-SAF is able to robustly learn the underlying nonlinear model, with a significant gain compared with a noncooperative solution [50]. A detailed description of D-SAFs can be found in Chapter 9.

3.3.6 Computational Cost

From a computational point of view each SAF architecture requires the evaluation of two or three processing blocks. For each block the computational cost of the forward phase has to be evaluated (i.e., the evaluation of the system output) and also that of the backward phase (i.e., the adaptation of the adaptive system). It is well known from the literature that the computational cost of the LMS algorithm is $2M+1$ multiplications plus 2M additions for each input sample [1], if M denotes the filter length.

In addition the cost of the spline evaluation must be considered. For each iteration only the ith span of the curve is modified by calculating the quantities $u_n$, i and the expressions ${\mathbf{u}}_n^T{\mathbf{Cq}}_{i,n}$, ${\dot{\mathbf{u}}}_n^T{\mathbf{Cq}}_{i,n}$ and ${\mathbf{C}}^T{\mathbf{u}}_n$. Note that the calculation of the quantity ${\mathbf{Cq}}_{i,n}$ is executed during the output computation, as well as in the backward phase (the spline derivatives). However, most of the computations can be done through the reuse of past calculations (e.g., by explicating the polynomial and by use of the well-known Horner rule [58]). The cost for the spline output computation and its adaptation is $4K_M$ multiplications plus $4K_A$ additions, where $K_M$ and $K_A$ are constants (less than 16), depending on the implementation structure (see for example [53]). In any case, for high deep memory SAF, where the filter length is $M>>4$, the computational overhead, for the nonlinear function computation and its adaptation, can be neglected with respect to a simple linear filter. It is straightforward to note that the computational cost can be further reduced by using some smart trick, like choosing the sampling step Δx or the learning rates ${\mu }_j[n]$ as a power of two. In fact, these constraints transform the multiplications into shift operations.

The total computational cost, inclusive of the forward and backward computation, for the presented four SAF architectures is summarized in Table 3.1. In this table M denotes the linear filter length. In the case of S2SAF architecture, $M_1$ and $M_2$ are the length of the first and second linear filter, respectively.

3.4.1 Bounds on the Learning Rates

A necessary but not sufficient condition for the algorithm convergence is the identification of a bound on the learning rate choice. These bounds on the learning rates can be derived by the Taylor series expansion of the error $e[n+1]$ around the time index n, stopped at the first order. This series expansion involves the derivatives of the error with respect to the free parameters of the considered architecture.

By performing these evaluations for the four SAF architectures introduced in Section 3.3 and by using the previous notation, we can easily obtain

$0<{\mu }_w[n]{\phi }_i^{\prime 2}(u){\Vert {\mathbf{x}}_n\Vert }^2+{\mu }_q[n]{\Vert {\mathbf{C}}^T\mathbf{u}\Vert }^2<2,$

$0<{\mu }_w[n]{\Vert {\mathbf{s}}_n\Vert }^2+{\mu }_q[n]{\Vert {\mathbf{C}}^T{\mathbf{U}}_{i,n}{\mathbf{w}}_n\Vert }^2<2,$

$0<{\mu }_q^{(1)}[n]\frac{{\phi }_i^{\prime 2}(u_n^{(2)})}{\mathrm{\Delta }{x}^{(2)}}{\Vert {\mathbf{C}}^T{\mathbf{U}}_{i,n}^{(1)}{\mathbf{w}}_n\Vert }^2+{\mu }_q^{(2)}[n]{\Vert {\mathbf{C}}^T{\mathbf{u}}_n^{(2)}\Vert }^2+{\mu }_w[n]\frac{{\phi }_i^{\prime 2}(u_n^{(2)})}{\mathrm{\Delta }{x}^{(2)}}{\Vert {\mathbf{s}}_n\Vert }^2<2,$

$0<{\mu }_w^{(1)}[n]\frac{{\phi }_i^{\prime 2}(u_n)}{\mathrm{\Delta }x}{\Vert {\mathbf{X}}_n{\mathbf{w}}_n^{(2)}\Vert }^2+{\mu }_w^{(2)}[n]{\Vert {\mathbf{r}}_n\Vert }^2+{\mu }_q[n]{\Vert {\mathbf{C}}^T{\mathbf{U}}_{i,n}{\mathbf{w}}_n^{(2)}\Vert }^2<2.$

The full derivation of these bounds can be found in [40,41,43].

3.4.2 Steady-State MSE

The aim of this paragraph is to evaluate the mean square performance of some of the proposed architectures at steady state. In particular, we are interested in the derivation of the EMSE of the nonlinear adaptive filter [2]. For simplicity we limit our analysis to the case of the Wiener and Hammerstein systems. The steady-state analysis is performed in a separate manner, by considering first the case of the linear filter alone and then the nonlinear spline function alone. Note that, since we are interested only in the steady state, the fixed subsystem in each substep can be considered adapted. The analysis is conducted by using a model of the same type of the used one, as shown in Fig. 3.5 for the particular case of the WSAF. Similar schemes are valid for the other kinds of architectures.

In the following we denote with $\epsilon [n]$ the a priori error of the whole system, with ${\epsilon }_w[n]$ the a priori error when only the linear filter is adapted while the spline control points are fixed and similarly with ${\epsilon }_q[n]$ the a priori error when only the control points are adapted while the linear filter is fixed. In order to make tractable the mathematical derivation, some assumption must be introduced:

A1. The noise sequence $v[n]$ is independent and identically distributed with variance ${\sigma }_v^2$ and zero mean.

A2. The noise sequence $v[n]$ is independent of ${\mathbf{x}}_n$, ${\mathbf{s}}_n$, $\epsilon [n]$, ${\epsilon }_w[n]$ and ${\epsilon }_q[n]$.

A3. The input sequence $x[n]$ is zero mean with variance ${\sigma }_x^2$ and at steady state it is independent of ${\mathbf{w}}_n$.

A4. At steady state, the error sequence $e[n]$ and the a priori error sequence $\epsilon [n]$ are independent of ${\phi }_i^{\prime }(u_n)$, ${\phi }_i^{\prime 2}(u_n)$, $\Vert {\mathbf{x}}_n\Vert$, $\Vert {\mathbf{s}}_n\Vert$, $\Vert {\mathbf{C}}^T{\mathbf{u}}_n\Vert$ and $\Vert {\mathbf{C}}^T{\mathbf{U}}_{i,n}{\mathbf{w}}_n\Vert$.

The EMSE ζ is defined at steady state as

$\zeta \equiv \underset{n\to \infty }{\lim }E\{{\epsilon }^2[n]\}.$

Similar definitions hold for the EMSE ${\zeta }_w$ due to the linear filter component ${\mathbf{w}}_n$ only and the EMSE ${\zeta }_q$ due to the spline nonlinearity ${\mathbf{q}}_n$ only.

The MSE analysis has been performed by the energy conservation method [2]. Following the derivation presented in [41] and [44] we obtain an interesting property.

The values of ${\zeta }_w$ and ${\zeta }_q$ are dependent on the particular SAF architecture considered. Specifically, for the Wiener case, we have [44]

${\zeta }_w=\frac{{\mu }_w[n]{\sigma }_v^{2}E\{{\phi }_i^{\prime 2}(u_n){\Vert {\mathbf{x}}_n\Vert }^2\}}{2\mathrm{\Delta }{x}^{2}E\left\{\frac{{\phi }_i^{\prime }(u_n)}{{\mathbf{c}}_3{\mathbf{q}}_{i,n}}\right\}-{\mu }_w[n]E\{{\phi }_i^{\prime 2}(u_n){\Vert {\mathbf{x}}_n\Vert }^2\}},$

where ${\mathbf{c}}_3$ is the third row of the C matrix and

${\zeta }_q=\frac{{\mu }_q[n]{\sigma }_v^{2}E\left\{{\Vert {\mathbf{C}}^{\mathrm{T}}{\mathbf{u}}_n\Vert }^2\right\}}{2-{\mu }_q[n]E\left\{{\Vert {\mathbf{C}}^{\mathrm{T}}{\mathbf{u}}_n\Vert }^2\right\}}.$

Other useful properties on the steady-state performances of the WSAF can be found in [44]. For the Hammerstein case, we have [41]

${\zeta }_w=\frac{{\mu }_w[n]{\sigma }_v^{2}E\left\{{\Vert {\mathbf{s}}_n\Vert }^2\right\}}{2-{\mu }_w[n]E\left\{{\Vert {\mathbf{s}}_n\Vert }^2\right\}}$

and

${\zeta }_q=\frac{{\mu }_q[n]{\sigma }_v^{2}E\left\{{\Vert {\mathbf{C}}^T{\mathbf{U}}_{i,n}{\mathbf{w}}_n\Vert }^2\right\}}{2-{\mu }_q[n]E\left\{{\Vert {\mathbf{C}}^T{\mathbf{U}}_{i,n}{\mathbf{w}}_n\Vert }^2\right\}}.$

3.5.1 Results From Simulated Scenarios

Regarding the experiments on nonlinear systems taken from the literature, we compare results obtained by SAFs with those of a full third-order Volterra adaptive filter [59] and the approach proposed by Jeraj and Mathews in [60] where the nonlinearity is implemented as a third-order polynomial. In addition, Sandwich architectures are also compared with the LNL approach proposed by Hegde et al. in [61,62]. In all experiments the input signal $x[n]$ consists of 50 000 samples of the signal generated by the following relationship:

$x[n]=ax[n-1]+\sqrt{1-a^2}\xi [n],$

where $\xi [n]$ is a zero mean white Gaussian noise with unitary variance and a ($0⩽a<1$) is a parameter that determines the level of correlation between adjacent samples. Experiments were conducted with a set to 0.1 and 0.95. In addition an additive white noise $v[n]$ with variance ${\sigma }_v^2$ is considered.

In the following we propose four tests in order to verify the Wiener, Hammerstein, Sandwich 1 and Sandwich 2 architectures described in Section 3.3. All the SAF models use a CR-spline basis in (3.7) with $\mathrm{\Delta }x=0.2$.

WSAF—The Wiener architecture to be identified is composed of the Back and Tsoi NARMA model reported in [63]. This model consists of a cascade of the third-order IIR filter

$H(z)=\frac{0.0154+0.0462z^{-1}+0.0462z^{-2}+0.0154z^{-3}}{1-1.99z^{-1}+1.572z^{-2}-0.4583z^{-3}}$

and the following nonlinearity:

$y[n]=\sin (x[n]).$

The input signal $x[n]$ is the colored signal obtained with (3.36), choosing $a=0.95$. The learning rates are set to ${\mu }_w={\mu }_q=0.1$ and the filter length is $M=15$ taps. The third-order Volterra filter has $M_v=10$ filter taps and the learning rate is set to ${\mu }_v=0.1$, while the Jeraj and Mathews architecture is characterized by $M_a=1$, $M_b=15$, a nonlinearity of order $N=3$ and learning rate ${\mu }_L={\mu }_{NL}={10}^{-5}$. The MSE, averaged over 100 trials, is reported in Fig. 3.6A.

HSAF—The Hammerstein architecture to be identified, drawn from [64], is composed of the following nonlinearity:

$s[n]=\frac{1}{8}+\lfloor 2x[n]-1\rfloor ,$

where $\lfloor •\rfloor$ is the floor operator, and the FIR filter $\mathbf{h}=\frac{1}{2}[1,0.75,0.5,0.25,0,-0.25]$. The learning rates are set to ${\mu }_w=0.01$ and ${\mu }_q=0.02$, while the parameters of Volterra and Jeraj and Mathews architectures are the same as in the Wiener case. The MSE, averaged over 100 trials, is reported in Fig. 3.6B.

S1SAF—The identification of a Sandwich 1 system is drawn from [65]. The first and last blocks are the following two static nonlinear functions:

$s[n]=\frac{x[n]}{\sqrt{0.1+0.9x^2[n]}}$

and

$y[n]=r[n]-0.5r^2[n]+0.2r^3[n],$

while the second block is the following FIR filter:

$H(z)=1+0.5z-0.25z^{-1}+0.05z^{-2}+0.001z^{-3}.$

The input signal $x[n]$ is taken from (3.36) with $a=0.1$. Also $\mathrm{SNR}=30$ dB is considered. The learning rates are set to ${\mu }_w={\mu }_q^{(1)}={\mu }_q^{(2)}=0.05$ and the length is set to $M=15$. The proposed methods were compared with a standard full third-order Volterra filter, the approach of Jeraj and Mathews with the same parameters of the Wiener case and the LNL approach proposed by Hegde et al. with $M_1=M_2=5$, ${\mu }_A={\mu }_B={10}^{-4}$ and ${\mu }_C={10}^{-3}$. The MSE, averaged over 100 trials, is reported in Fig. 3.7A.

S2SAF—Finally, a Sandwich 2 to be identified is drawn from [66]. The first and last blocks are two fourth-order IIR filters, Butterworth and Chebyshev, respectively, with transfer functions

$\begin{array}{cc}\hfill H_B(z)=& \frac{(0.2851+0.5704z^{-1}+0.2851z^{-2})}{(1-0.1024z^{-1}+0.4475z^{-2})}\hfill \\ \hfill & \cdot \frac{(0.2851+0.5701z^{-1}+0.2851z^{-2})}{(1-0.0736z^{-1}+0.0408z^{-2})}\hfill \end{array}$

and

$\begin{array}{cc}\hfill H_C(z)=& \frac{(\mathrm{0}.\mathrm{2025}+0.2880z^{-1}+0.2025z^{-2})}{(1-1.01z^{-1}+0.5861z^{-2})}\hfill \\ \hfill & \cdot \frac{(\mathrm{0}.\mathrm{2025}+0.0034z^{-1}+\mathrm{0}.\mathrm{2025}{z}^{-2})}{(1-0.6591z^{-1}+0.1498z^{-2})},\hfill \end{array}$

while the second block is the following nonlinearity:

$y[n]=\frac{2x[n]}{1+|x[n]{|}^2}.$

This system is similar to radio frequency amplifiers used in satellite communications (high-power amplifiers), in which the linear filters model the dispersive transmission paths, while the nonlinearity models the amplifier saturation. The input signal $x[n]$ is as in (3.36) with $a=0.1$. The learning rates are set to ${\mu }_w^{(1)}=0.005$, ${\mu }_w^{(2)}=0.009$ and ${\mu }_q=0.04$ and the lengths are set to $M_1=M_2=5$. Comparisons with Volterra, Jeraj and Mathews and Hegde et al. have been performed with the previous parameters. The MSE, averaged over 100 trials, is reported in Fig. 3.7B.

3.5.2 Results From Real-World Scenarios

In evaluating nonlinear architectures, it is important to test the performance of the proposed algorithm on completely unknown systems. For this purpose, we test our architecture from data sets collected from laboratory processes that can be described by block-oriented dynamic models. In particular, a number of interesting databases for system identification is represented by DaISy.³

From the DaISy database we have collected the following interesting and challenge data sets:

All results have been compared with other state-of-the-art approaches used in previous experiments, namely a full third-order Volterra architecture [59] (with $M_v=15$), the LNL approach proposed by Hegde et al. in [61,62] (with $M_1=M_2=5$), where the nonlinearity is implemented as a third-order polynomial and the approach proposed by Jeraj and Mathews in [60] (with $M_a=1$, $M_b=5$ and $N=3$). In order to provide a fair comparison of the performance of such algorithms, Table 3.2 shows their asymptotic computational complexity.

Table 3.2

Comparison of the asymptotic computational complexity of the algorithms used in experimental comparisons

Algorithm	Asymptotic complexity
WSAF [40]	$\mathcal{O}(M)$
HSAF [41]	$\mathcal{O}(M)$
S1 SAF [43]	$\mathcal{O}(M)$
S2 SAF [43]	$\mathcal{O}(M_1{M}_2)$
Third-order Volterra [59]	$\mathcal{O}(M_v^3)$
Hegde et al.[61]	$\mathcal{O}(M_1{M}_2)$
Jeraj and Mathews [60]	$\mathcal{O}({(M_b{M}_{a}N)}^2)$

Moreover, we have chosen two additional measured data from coupled electric drives that consist of two electric motors that drive a pulley using a flexible belt and two measured data from a fluid level control system consisting of two cascaded tanks with free outlets fed by a pump. Additional details as regards these systems and data sets can be found in [67].

A last data set is taken from [68,69] and consist of the identification of a nonlinear electric circuit composed of the cascade of a third-order Chebyshev linear filter, a nonlinearity built by using a diode circuit and a second linear filter designed as a third-order inverse Chebyshev filter with a transmission of zero in the frequency band of interest.

Results in terms of MSEs for all data sets are summarized in Table 3.3. It can be seen that the proposed SAF approaches provide the best performance in almost all cases.

3.5.3 Applicative Scenarios

In the previous subsections, we provided some numerical results on the general problem of nonlinear system identification. However, SAFs have been used in applicative scenarios providing interesting and very good results.

A first applicative scenario is the nonlinear acoustic echo cancellation (NAEC) problem. The SAF approach has been successfully applied to NAEC in [30], where it is shown that the proposed solutions outperform other linear and nonlinear approaches.

Another challenging scenario is the nonlinear adaptive noise cancellation (NANC) problem. In particular, in [45] authors show the effectiveness of SAF approach to other state-of-the-art algorithms. Moreover, the proposed solution has been extended with additional refinements in [46] and [47].

Finally, as underlined in Section 3.3.5, the SAF approach has been recently extended to the diffusion case [50], involving a distributed (wireless) sensors network. The authors of this work derive a D-SAF by applying ideas from the DA framework, as explained in Chapter 9. Experimental evaluations on the D-SAF approach show that it is able to robustly learn the underlying nonlinear model, with a significant gain compared with a noncooperative solution.