14.2.1 HJI Equation for $H_{\infty }$ Optimal Tracking

In this section, it is first shown that the problem of solving the $H_{\infty }$ tracking problem can be transformed into a min–max optimization problem subject to an augmented system composed of the tracking error dynamics and the command generator dynamics. A tracking HJI equation is then developed which gives the solution to the min–max optimization problem. The stability and $L_2$-gain boundedness of the tracking HJI control solution are discussed.

Define the augmented system state

$\mathbf{X}(t)={[{\mathbf{e}}_{\mathbf{d}}{(t)}^T\mathbf{r}{(t)}^T]}^T\in {\mathbb{R}}^{2n},$

where ${\mathbf{e}}_{\mathbf{d}}(t)$ is the tracking error defined in (14.3) and $\mathbf{r}(t)$ is the reference trajectory.

Using (14.2) and (14.4), define the augmented system

$\dot{\mathbf{X}}(t)=\mathbf{F}(\mathbf{X}(t))+\mathbf{G}(\mathbf{X}(t))\mathbf{u}(t)+\mathbf{K}(\mathbf{X}(t))\mathbf{w}(t),$

where $\mathbf{u}(t)=\mathbf{u}(\mathbf{X}(t))$ and

$\mathbf{F}(\mathbf{X})=\left[\begin{array}{c}\hfill \mathbf{f}({\mathbf{e}}_{\mathbf{d}}+\mathbf{r})-{\mathbf{h}}_{\mathbf{d}}(\mathbf{r})\hfill \\ \hfill {\mathbf{h}}_{\mathbf{d}}(\mathbf{r})\hfill \end{array}\right],\mathbf{G}(\mathbf{X})=\left[\begin{array}{c}\hfill \mathbf{g}({\mathbf{e}}_{\mathbf{d}}+\mathbf{r})\hfill \\ \hfill \mathbf{0}\hfill \end{array}\right],\mathbf{K}(\mathbf{X})=\left[\begin{array}{c}\hfill \mathbf{k}({\mathbf{e}}_{\mathbf{d}}+\mathbf{r})\hfill \\ \hfill \mathbf{0}\hfill \end{array}\right].$

The disturbance attenuation condition (14.7) using the augmented state becomes

${\int }_t^{\infty }{e}^{-\alpha (\tau -t)}({\mathbf{X}}^T{\mathbf{Q}}_{\mathbf{T}}\mathbf{X}+{\mathbf{u}}^T\mathbf{R}\mathbf{u})d\tau ⩽{\gamma }^2{\int }_t^{\infty }{e}^{-\alpha (\tau -t)}({\mathbf{w}}^T\mathbf{w})d\tau ,$

where

${\mathbf{Q}}_{\mathbf{T}}=\left[\begin{array}{cc}\hfill \mathbf{Q}\hfill & \hfill \mathbf{0}\hfill \\ \hfill \mathbf{0}\hfill & \hfill \mathbf{0}\hfill \end{array}\right].$

Based on (14.9), define the performance function

$J(\mathbf{u},\mathbf{w})={\int }_t^{\infty }{e}^{-\alpha (\tau -t)}({\mathbf{X}}^T{\mathbf{Q}}_{\mathbf{T}}\mathbf{X}+{\mathbf{u}}^T\mathbf{R}\mathbf{u}-{\gamma }^2{\mathbf{w}}^T\mathbf{w})d\tau .$

Solvability of the $H_{\infty }$ control problem is equivalent to solvability of the following zero-sum game [42]:

$V^{\star }(\mathbf{X}(t))=J({\mathbf{u}}^{\star },{\mathbf{w}}^{\star })=\underset{\mathbf{u}}{\min }\underset{\mathbf{d}}{\max }J(\mathbf{u},\mathbf{w}),$

where J is defined in (14.10) and $V^{\star }(\mathbf{X}(t))$ is defined as the optimal value function. This two-player zero-sum game control problem has a unique solution if a game theoretic saddle point exists, i.e., if the following Nash condition holds:

$V^{\star }(\mathbf{X}(t))=\underset{\mathbf{u}}{\min }\underset{\mathbf{d}}{\max }J(\mathbf{u},\mathbf{w})=\underset{\mathbf{d}}{\max }\underset{\mathbf{u}}{\min }J(\mathbf{u},\mathbf{w}).$

Differentiating (14.10), note that $V(\mathbf{X}(t))=J(\mathbf{u}(t),\mathbf{w}(t))$ gives the following Bellman equation:

$H(V,\mathbf{u},\mathbf{w})\stackrel{\mathrm{\Delta }}{=}{\mathbf{X}}^T{\mathbf{Q}}_{\mathbf{T}}\mathbf{X}+{\mathbf{u}}^T\mathbf{R}\mathbf{u}-{\gamma }^2{\mathbf{w}}^T\mathbf{w}-\alpha V+{V_{\mathbf{X}}}^T(\mathbf{F}+\mathbf{G\hspace{0.17em}u}+\mathbf{K\hspace{0.17em}w})=0,$

where $\mathbf{F}\triangleq \mathbf{F}(\mathbf{X})$, $\mathbf{G}\triangleq \mathbf{G}(\mathbf{X})$, $\mathbf{K}\triangleq \mathbf{K}(\mathbf{X})$ and $V_{\mathbf{X}}=\partial V/\partial \mathbf{X}$.

Applying stationarity conditions $\partial H(V^{\star },\mathbf{u},\mathbf{w})/\partial \mathbf{u}=0,\partial H(V^{\star },\mathbf{u},\mathbf{w})/\partial \mathbf{w}=0$ [43] gives the optimal control and disturbance inputs as

${\mathbf{u}}^{\star }=-\frac{1}{2}{\mathbf{R}}^{-1}{\mathbf{G}}^T{V_{\mathbf{X}}}^{\star },$

${\mathbf{w}}^{\star }=\frac{1}{2{\gamma }^2}{\mathbf{K}}^T{V_{\mathbf{X}}}^{\star },$

where $V^{\star }$ is the optimal value function defined in (14.11). Substituting the control input (14.13) and the disturbance (14.14) into (14.12), the following tracking HJI equation is obtained:

$\begin{array}{cc}\hfill & H(V^{\star },{\mathbf{u}}^{\star },{\mathbf{w}}^{\star })\triangleq {\mathbf{X}}^T{\mathbf{Q}}_{\mathbf{T}}\mathbf{X}+{V_{\mathbf{X}}}^{\star T}\mathbf{F}-\alpha V_{\mathbf{X}}\hfill \\ \hfill & -\frac{1}{4}{V_{\mathbf{X}}}^{\star T}{\mathbf{G}}^T{\mathbf{R}}^{-1}\mathbf{G}{V_{\mathbf{X}}}^{\star }+\frac{1}{4{\gamma }^2}{V_{\mathbf{X}}}^{\star T}\mathbf{K\hspace{0.17em}}{\mathbf{K}}^{\mathbf{T}}{V_{\mathbf{X}}}^{\star }=0.\hfill \end{array}$

It is shown in [44] that the control solution (14.13)–(14.15) satisfies the disturbance attenuation condition (14.9) (part (i) of Definition 2) and that it guarantees the stability of the tracking error dynamics (14.4) without the disturbance (part (ii) of Definition 2), if the discount factor is less than an upper bound.

14.2.2 Off-Policy IRL for Learning the Tracking HJI Equation

In this section, an off-policy RL algorithm is first given to learn this control solution online and without requiring any knowledge of the system dynamics.

The Bellman equation (14.12) is linear in the cost function V, while the HJI equation (14.15) is nonlinear in the value function $V^{\star }$. Therefore, solving the Bellman equation for V is easier than solving the HJI for $V^{\star }$. Instead of directly solving for $V^{\star }$, a policy iteration (PI) algorithm iterates on both control and disturbance players to break the HJI equation into a sequence of differential equations linear in the cost. An offline PI algorithm for solving the $H_{\infty }$ optimal tracking problem is given as follows.

Algorithm 1 extends the results of the simultaneous RL algorithm in [27] to the tracking problem. The convergence of this algorithm to the minimal nonnegative solution of the HJI equation was shown in [27]. In fact, similar to [27], the convergence of Algorithm 1 can be established by proving that iteration on (14.16) is essentially a Newton iterative sequence which converges to the unique solution of the HJI equation (14.15).

Algorithm 1 requires complete knowledge of the system dynamics. In the following, the off-policy IRL algorithm, which was presented in [14,15] for solving the $H_2$ optimal regulation problem, is extended here to solve the $H_{\infty }$ optimal tracking for systems with completely unknown dynamics. To this end, the system dynamics (14.8) is first written as

$\dot{\mathbf{X}}=\mathbf{F}+\mathbf{G}{\mathbf{u}}^j+\mathbf{K}{\mathbf{w}}^j+\mathbf{G}(\mathbf{u}-{\mathbf{u}}^j)+\mathbf{K}(\mathbf{w}-{\mathbf{w}}^j),$

where ${\mathbf{u}}^j\in {\mathbb{R}}^m$ and ${\mathbf{w}}^j\in {\mathbb{R}}^q$ are policies to be updated. In this equation, the control input u is the behavior policy which is applied to the system to generate data for learning, while ${\mathbf{u}}^j$ is the target policy which is evaluated and updated using data generated by the behavior policy. The fixed control policy u should be a stable and exploring control policy. Moreover, the disturbance input w is the actual external disturbance that comes from an external source and is not under our control. However, the disturbance ${\mathbf{w}}^j$ is the disturbance that is evaluated and updated. One advantage of this off-policy IRL Bellman equation is that, in contrast to on-policy RL-based methods, the disturbance input that is applied to the system does not require to be adjustable.

Differentiating $V^j(\mathbf{X})$ along with the system dynamics (14.19) and using (14.16)–(14.18) gives

$\begin{array}{cc}\hfill & {\dot{V}}^j={({V_{\mathbf{X}}}^j)}^T(\mathbf{F}+\mathbf{G}{\mathbf{u}}^j+\mathbf{K}{\mathbf{w}}^j)+{({V_{\mathbf{X}}}^j)}^T\mathbf{G}(\mathbf{u}-{\mathbf{u}}^j)+{({V_{\mathbf{X}}}^j)}^T\mathbf{K}(\mathbf{w}-{\mathbf{w}}^j)\hfill \\ \hfill & =\alpha V^j-{\mathbf{X}}^T{\mathbf{Q}}_{\mathbf{T}}\mathbf{X}-{({\mathbf{u}}^j)}^T\mathbf{R}{\mathbf{u}}^j+{\gamma }^2{({\mathbf{w}}^j)}^T{\mathbf{w}}^j-\hfill \\ \hfill & 2{({\mathbf{u}}^{j+1})}^T\mathbf{R}(\mathbf{u}-{\mathbf{u}}^j)+2{\gamma }^2{({\mathbf{w}}^{j+1})}^T(\mathbf{w}-{\mathbf{w}}^j).\hfill \end{array}$

Multiplying both sides of (14.20) by $e^{-\alpha (\tau -t)}$ and integrating from both sides yields the following off-policy IRL Bellman equation:

$\begin{array}{cc}\hfill & e^{-\alpha T}{V}^j(\mathbf{X}(t+T))-V^j(\mathbf{X}(t))=\hfill \\ \hfill & {\int }_t^{t+T}{e}^{-\alpha (\tau -t)}(-{\mathbf{X}}^T{\mathbf{Q}}_{\mathbf{T}}\mathbf{X}-{({\mathbf{u}}^j)}^T\mathbf{R}{\mathbf{u}}^j+{\gamma }^2{({\mathbf{w}}^j)}^T{\mathbf{w}}^j)d\tau \hfill \\ \hfill & +{\int }_t^{t+T}{e}^{-\alpha (\tau -t)}(-2{({\mathbf{u}}^{j+1})}^T\mathbf{R}(\mathbf{u}-{\mathbf{u}}^j)+2{\gamma }^2{({\mathbf{w}}^{j+1})}^T(\mathbf{w}-{\mathbf{w}}^j))d\tau .\hfill \end{array}$

Note that, for a fixed control policy u (the policy that is applied to the system) and a given disturbance w (the actual disturbance that is applied to the system), Eq. (14.21) can be solved for both the value function $V^j$ and the updated policies ${\mathbf{u}}^{j+1}$ and ${\mathbf{w}}^{j+1}$ simultaneously.

The following algorithm uses the off-policy tracking Bellman equation (14.21) to iteratively solve the HJI equation (14.15) without requiring any knowledge of the system dynamics. The implementation of this algorithm is discussed in the next subsection. It is shown how the data collected from a fixed control policy u are reused to evaluate many updated control policies ${\mathbf{u}}_i$ sequentially until convergence to the optimal solution is achieved.

Inspired by the off-policy algorithm in [14], Algorithm 2 has two separate phases. First, a fixed initial exploratory control policy u is applied and the system information is recorded over the time interval T. Second, without requiring any knowledge of the system dynamics, the information collected in phase 1 is repeatedly used to find a sequence of updated policies ${\mathbf{u}}^j$ and ${\mathbf{w}}^j$ converging to ${\mathbf{u}}^{\star }$ and ${\mathbf{w}}^{\star }$. Note that Eq. (14.23) is a scalar equation and can be solved in a least square sense after collecting enough data samples from the system. It is shown in the following section how to collect required information in phase 1 and reuse it in phase 2 in a least square sense to solve (14.23) for $V^j$, ${\mathbf{u}}^{j+1}$ and ${\mathbf{w}}^{j+1}$ simultaneously. After the learning is done and the optimal control policy ${\mathbf{u}}^{\star }$ is found, it can be applied to the system.

14.2.3 Implementing Algorithm 2 Using Neural Networks

In order to implement the off-policy RL Algorithm 2, it is required to reuse the collected information found by applying a fixed control policy u to the system to solve Eq. (14.23) for $V^j$, ${\mathbf{u}}^{j+1}$ and ${\mathbf{w}}^{j+1}$ iteratively. Three neural networks (NNs), i.e., the actor NN, the critic NN and the disturber NN, are used here to approximate the value function and the updated control and disturbance policies in the Bellman equation (14.23). That is, the solution $V^j$, ${\mathbf{u}}^{j+1}$ and ${\mathbf{w}}^{j+1}$ of the Bellman equation (14.23) is approximated by three NNs as

${\hat{V}}^j(\mathbf{X})={{\hat{\mathbf{W}}}_{\mathbf{1}}}^T\mathit{\sigma }(\mathbf{X}),$

${\hat{\mathbf{u}}}^{j+1}(\mathbf{X})={{\hat{\mathbf{W}}}_{\mathbf{2}}}^T\mathit{\varphi }(\mathbf{X}),$

${\hat{\mathbf{w}}}^{j+1}(\mathbf{X})={{\hat{\mathbf{W}}}_{\mathbf{3}}}^T\mathit{\phi }(\mathbf{X}),$

where $\mathit{\sigma }=[{\sigma }_1,...,{\sigma }_{l_1}]\in {\mathbb{R}}^{l_1}$, $\mathit{\varphi }=[{\varphi }_1,...,{\varphi }_{l_2}]\in {\mathbb{R}}^{l_2}$ and $\mathit{\phi }=[{\phi }_1,...,{\phi }_{l_3}]\in {\mathbb{R}}^{l_3}$ provide suitable basis function vectors, $\widehat{{\mathbf{W}}_{\mathbf{1}}}\in {\mathbb{R}}^{l_1}$, $\widehat{{\mathbf{W}}_{\mathbf{2}}}\in {\mathbb{R}}^{m\times l_2}$ and $\widehat{{\mathbf{W}}_{\mathbf{3}}}\in {\mathbb{R}}^{q\times l_3}$ are constant weight vectors and $l_1$, $l_2$ and $l_3$ are the number of neurons. Define ${\mathbf{v}}^1={[v_1^1,...,v_1^m]}^T=\mathbf{u}-{\mathbf{u}}^j$, ${\mathbf{v}}^2={[v_1^2,...,v_q^2]}^T=\mathbf{w}-{\mathbf{w}}^j$ and assume $\mathbf{R}=diag(r,...,r_m)$. Then, substituting (14.24)–(14.26) in (14.23) yields

$\begin{array}{cc}\hfill & e(t)={{\hat{\mathbf{W}}}_1}^T(e^{-\alpha T}\mathit{\sigma }(\mathbf{X}(t+T))-\mathit{\sigma }(\mathbf{X}(t)))\hfill \\ \hfill & -{\int }_t^{t+T}{e}^{-\alpha (\tau -t)}(-{\mathbf{X}}^T{\mathbf{Q}}_{\mathbf{T}}\mathbf{X}-{({\mathbf{u}}^j)}^T\mathbf{R}{\mathbf{u}}^j+{\gamma }^2{({\mathbf{w}}^j)}^T{\mathbf{w}}^j)d\tau \hfill \\ \hfill & +2\sum _{l=1}^m{r}_l{\int }_t^{t+T}{e}^{-\alpha (\tau -t)}{{\hat{\mathbf{W}}}_{2,l}}^T\mathit{\varphi }(\mathbf{X}(t))v_l^{1}d\tau \hfill \\ \hfill & -2{\gamma }^2\sum _{k=1}^q{\int }_t^{t+T}{e}^{-\alpha (\tau -t)}{{\hat{\mathbf{W}}}_{3,k}}^T\mathit{\phi }(\mathbf{X}(t))v_k^{2}d\tau ,\hfill \end{array}$

where $e(t)$ is the Bellman approximation error, ${\hat{\mathbf{W}}}_{\mathbf{2},\mathbf{l}}$ is the lth column of ${\hat{\mathbf{W}}}_{\mathbf{2}}$ and ${\hat{\mathbf{W}}}_{\mathbf{3},\mathbf{k}}$ is the kth column of ${\hat{\mathbf{W}}}_{\mathbf{3}}$. The Bellman approximation error is the continuous-time counterpart of the temporal difference (TD) [10]. In order to bring the TD error to its minimum value, the least squares method is used. To this end, rewrite Eq. (14.27) as

$y(t)+e(t)={\hat{\mathbf{W}}}^{\mathbf{T}}\mathbf{h}(\mathbf{t}),$

where

$\begin{array}{c}\hfill \hat{\mathbf{W}}={[{{\hat{\mathbf{W}}}_{\mathbf{1}}}^T,{{\hat{\mathbf{W}}}_{\mathbf{2},\mathbf{l}}}^T,...,{{\hat{\mathbf{W}}}_{\mathbf{2},\mathbf{m}}}^T,{{\hat{\mathbf{W}}}_{\mathbf{3},\mathbf{1}}}^T,...,{{\hat{\mathbf{W}}}_{\mathbf{3},\mathbf{q}}}^T]}^T\in {\mathbb{R}}^{l_1+m\times l_2+q\times l_3},\hfill \\ \hfill \mathbf{h}(\mathbf{t})=\left[\begin{array}{c}\hfill e^{-\alpha T}\mathit{\sigma }(\mathbf{X}(t+T))-\mathit{\sigma }(\mathbf{X}(t)))\hfill \\ \hfill 2r_1{\int }_t^{t+T}{e}^{-\alpha (\tau -t)}\mathit{\varphi }(\mathbf{X}(t))v_1^{1}d\tau \hfill \\ \hfill ⋮\hfill \\ \hfill 2r_m{\int }_t^{t+T}{e}^{-\alpha (\tau -t)}\mathit{\varphi }(\mathbf{X}(t))v_m^{1}d\tau \hfill \\ \hfill -2{\gamma }^2{\int }_t^{t+T}{e}^{-\alpha (\tau -t)}\mathit{\phi }(\mathbf{X}(t))v_1^{2}d\tau \hfill \\ \hfill ⋮\hfill \\ \hfill -2{\gamma }^2{\int }_t^{t+T}{e}^{-\alpha (\tau -t)}\mathit{\phi }(\mathbf{X}(t))v_q^{2}d\tau \hfill \end{array}\right],\hfill \end{array}$

$y(t)={\int }_t^{t+T}{e}^{-\alpha (\tau -t)}(-{\mathbf{X}}^T{\mathbf{Q}}_{\mathbf{T}}\mathbf{X}-{({\mathbf{u}}^j)}^T\mathbf{R}{\mathbf{u}}^j+{\gamma }^2{({\mathbf{w}}^j)}^T{\mathbf{w}}^j)d\tau .$

The parameter vector $\hat{\mathbf{W}}$, which gives the approximated value function, actor and disturbance (14.24)–(14.26), is found by minimizing, in the least squares sense, the Bellman error. Assume that the systems state, input and disturbance information are collected at $N⩾l_1+m\times l_2+q\times l_3$ (the number of independent elements in $\hat{\mathbf{W}}$) points $t_1$ to $t_N$ in the state space, over the same time interval T in phase 1. Then, for a given ${\mathbf{u}}^j$ and ${\mathbf{w}}^j$, one can use this information to evaluate (14.29) and (14.30) at N points to form

$\begin{array}{c}\hfill \mathbf{H}=[\mathbf{h}(t_1),....,\mathbf{h}(t_N)],\hfill \\ \hfill \mathbf{Y}={[y(t_1),....,y(t_N)]}^T.\hfill \end{array}$

The least squares solution to (14.28) is then equal to

$\hat{\mathbf{W}}={(\mathbf{H}{\mathbf{H}}^T)}^{-1}\mathbf{H}\mathbf{Y},$

which gives $V^j$, ${\mathbf{u}}^{j+1}$ and ${\mathbf{w}}^{j+1}$. Note that although $\mathbf{X}(t+T)$ appears in Eq. (14.27), this equation is solved in a least square sense after observing N samples $\mathbf{X}(t)$, $\mathbf{X}(t+T)$, …, $\mathbf{X}(t+NT)$. Therefore, the knowledge of the system is not required to predict the future state $\mathbf{X}(t+T)$ at time t to solve (14.27).

14.3.1 A Class of Nonaffine Dynamical Systems

A special class of nonaffine systems can be described as

$\dot{\mathbf{X}}(t)=\mathbf{f}(\mathbf{X}(t))+\mathbf{g}(\mathbf{X}(t))\mathbf{L}(\mathbf{u})+\mathbf{D}\mathbf{w}(t),$

where $\mathbf{X}(t)\in {\mathbb{R}}^n$, $\mathbf{u}(t)\in {\mathbb{R}}^m$ and $\mathbf{w}(t)\in {\mathbb{R}}^p$ are the state of the system, the control input and the external disturbance input, respectively. The functions $\mathbf{f}(\mathbf{X}(t))$ and $\mathbf{g}(\mathbf{X}(t))$ are Lipschitz functions. This system is affine in a nonlinear function $\mathbf{L}(.)$ of the control input $\mathbf{u}(t)$. This class of nonaffine systems allows the definition of a new performance function for the optimal $H_{\infty }$ problem such that the existence of the constrained optimal control is assured (if any exists).

The following example shows that the UAV as a real-world application can be presented in the form of (14.31).

14.3.2 Performance Function and $H_{\infty }$ Control Tracking for Nonaffine Systems

It is shown in [46] that the existence of an admissible optimal control solution for nonaffine systems depends on how the utility function $r(\mathbf{X},\mathbf{u})$ is defined. Moreover, to deal with the input constraints, a nonquadratic performance index needs to be defined as follows.

Let the reference trajectory be generated by the command generator dynamics (14.2). The performance or control output $\mathbf{z}(t)$ is defined such that it satisfies

${\Vert \mathbf{z}(t)\Vert }^2={(\mathbf{X}-\mathbf{r})}^{\text{T}}\mathbf{Q}(\mathbf{X}-\mathbf{r})+W(\mathbf{L}(\mathbf{u})),$

where $\mathbf{Q}⪰0$ and $W(\mathbf{L}(\mathbf{u}))$ is a positive definite nonquadratic function of $L(\mathbf{u})$ which penalizes the control effort and is chosen as follows to assure the constrained control effort:

$W(\mathbf{L}(\mathbf{u}))={\int }_0^{\mathbf{L}(\mathbf{u})}\mathbf{w}(s)ds=\sum _{j=1}^l({\int }_0^{{\mathbf{L}}_j(\mathbf{u})}{w}_j(s_j)d{s}_j),$

where $\mathbf{w}(s)={\tanh }^{-1}({\overline{\mathbf{L}}}^{-1}s)={\left[\begin{array}{ccc}\hfill w_1(s_1)\hfill & \hfill \cdots \hfill & \hfill w_l(s_l)\hfill \end{array}\right]}^{\text{T}}$ and $\overline{\mathbf{L}}$ is the constant diagonal matrix given by $\overline{\mathbf{L}}=diag({\overline{L}}_1,...,{\overline{L}}_l)$, which determines the bounds on $\mathbf{L}(\mathbf{u})$. Note that the bounds are originally given for the control input $\mathbf{u}(t)$ itself. However, one can transform these bounds to bounds on $L(\mathbf{u})$.

The $H_{\infty }$ control is to develop a control input such that (1) the system (1) with $\mathbf{w}=\mathbf{0}$ is asymptotically stable and (2) the $L_2$ gain condition (14.6) with $\mathbf{z}(t)$ defined in (14.36) is satisfied in the presence of $\mathbf{w}\in L_2[0,\infty )$.

The disturbance attenuation condition is satisfied if the following cost function is nonpositive:

$J(\mathbf{X})={\int }_t^{\infty }{e}^{-\alpha (\tau -t)}[{(\mathbf{X}-\mathbf{r})}^{\text{T}}\mathbf{Q}(\mathbf{X}-\mathbf{r})+W(\mathbf{L}(\mathbf{u}))-{\gamma }^2{w}^{\text{T}}w]d\tau .$

14.3.3 Solution of the $H_{\infty }$ Control Tracking Problem of Nonaffine Systems

Define the tracking error as (14.3). Then, using (14.2) and (14.31), the tracking error dynamics becomes

${\dot{\mathbf{e}}}_{\mathbf{d}}(t)=\dot{\mathbf{X}}(t)-\dot{\mathbf{r}}(t)=\mathbf{f}(\mathbf{X}(t))+\mathbf{g}(\mathbf{X}(t))\mathbf{L}(\mathbf{u})+\mathbf{D}\mathbf{w}(t)-{\mathbf{h}}_{\mathbf{d}}(\mathbf{r})(t).$

Based on (14.2) and (14.38), an augmented system can be constructed in terms of the tracking error $\mathbf{e}(t)$ and the reference trajectory $\mathbf{r}(t)$ as

$\begin{array}{cc}\hfill \dot{\mathbf{Z}}(t)& =\left[\begin{array}{c}\hfill \dot{\mathbf{e}}(\mathbf{t})\hfill \\ \hfill \dot{\mathbf{r}}(\mathbf{t})\hfill \end{array}\right]=\left[\begin{array}{c}\hfill \mathbf{f}(\mathbf{e}(t)+\mathbf{r}(t))-{\mathbf{h}}_{\mathbf{d}}(\mathbf{r}(t))\hfill \\ \hfill {\mathbf{h}}_{\mathbf{d}}(\mathbf{r}(t))\hfill \end{array}\right]+\left[\begin{array}{c}\hfill \mathbf{g}(\mathbf{e}(t)+\mathbf{r}(t))\hfill \\ \hfill \mathbf{0}\hfill \end{array}\right]\mathbf{L}(\mathbf{u})+\left[\begin{array}{c}\hfill \mathbf{D}\hfill \\ \hfill \mathbf{0}\hfill \end{array}\right]\mathbf{w}(t)\hfill \\ \hfill & \equiv \mathbf{F}(\mathbf{Z}(t))+\mathbf{G}(\mathbf{Z}(t))\mathbf{L}(\mathbf{u})+\mathbf{Kw}(t),\hfill \end{array}$

where the augmented state is

$\mathbf{Z}(t)=\left[\begin{array}{c}\hfill \mathbf{e}(t)\hfill \\ \hfill \mathbf{r}(t)\hfill \end{array}\right].$

The performance index (14.37) can be rewritten as

$J(\mathbf{L}(\mathbf{u}),\mathbf{w})={\int }_t^{\infty }{e}^{-\alpha (\tau -t)}({\mathbf{Z}}^T(\tau ){\mathbf{Q}}_{\mathbf{1}}\mathbf{Z}(\tau )+W(\mathbf{L}(\mathbf{u}))-{\gamma }^2{\mathbf{w}}^{\text{T}}\mathbf{w})d\tau ,$

with ${\mathbf{Q}}_{\mathbf{1}}=\left[\begin{array}{cc}\hfill \mathbf{Q}\hfill & \hfill \mathbf{0}\hfill \\ \hfill \mathbf{0}\hfill & \hfill \mathbf{0}\hfill \end{array}\right]$.

The $H_{\infty }$ control problem can be expressed as a two-player zero-sum differential game in which the control effort policy player $\mathbf{L}(\mathbf{u})$ seeks to minimize the value function, while the disturbance policy player $\mathbf{w}(t)$ desires to maximize it. The goal is to find the feedback saddle point $({\mathbf{L}}^{\star }(\mathbf{u}),{\mathbf{w}}^{\star })$ such that [42]

$V^{\star }(\mathbf{Z}(t))=\underset{\mathbf{L}(\mathbf{u})}{\min }\underset{\mathbf{w}}{\max }J(\mathbf{L}(\mathbf{u}),\mathbf{w}).$

On the basis of (14.40) and noting that $V(\mathbf{Z}(t))=J(\mathbf{L}(\mathbf{u}),\mathbf{w})$, the $H_{\infty }$ tracking Bellman equation is

${\mathbf{Z}}^{\text{T}}{\mathbf{Q}}_{\mathbf{1}}\mathbf{Z}+W(\mathbf{L}(\mathbf{u}))-{\gamma }^2{\mathbf{w}}^{\text{T}}\mathbf{w}-\alpha V(\mathbf{Z})+\dot{V}(\mathbf{Z})=0$

and the Hamiltonian is given by

$H(\mathbf{Z},L(\mathbf{u}),\mathbf{w},V_{\mathbf{Z}})={\mathbf{Z}}^{\text{T}}{\mathbf{Q}}_{\mathbf{1}}\mathbf{Z}+W(\mathbf{L}(\mathbf{u}))-{\gamma }^2{\mathbf{w}}^{\text{T}}\mathbf{w}-\alpha V(\mathbf{Z})+V_{\mathbf{Z}}^{\text{T}}(\mathbf{F}(\mathbf{Z})+\mathbf{G}(\mathbf{Z})\mathbf{L}(\mathbf{u})+\mathbf{Kw}).$

Then the optimal control effort $L(\mathbf{u})$ and disturbance input $w(t)$ for the given problem are obtained by employing the stationarity condition

$\begin{array}{c}\hfill {\mathbf{L}}^{\star }(\mathbf{u})=\underset{\mathbf{L}(\mathbf{u})}{\arg \min }H(\mathbf{Z},\mathbf{L}(\mathbf{u}),\mathbf{w},V^{⁎})\triangleq \frac{d[{\mathbf{Z}}^{\text{T}}{\mathbf{Q}}_{\mathbf{1}}\mathbf{Z}+W(\mathbf{L}(\mathbf{u}))-{\gamma }^2{\mathbf{w}}^{\text{T}}\mathbf{w}-\alpha V^{\star }+{(V_{\mathbf{Z}}^{\star })}^{\text{T}}\dot{\mathbf{Z}}]}{d\mathbf{L}(\mathbf{u})},\hfill \\ \hfill {\mathbf{w}}^{\star }=\underset{\mathbf{w}}{\arg \max }H(\mathbf{Z},\mathbf{L}(\mathbf{u}),\mathbf{w},V^{\star })\triangleq \frac{d[{\mathbf{Z}}^{\text{T}}{\mathbf{Q}}_{\mathbf{1}}\mathbf{Z}+W(\mathbf{L}(\mathbf{u}))-{\gamma }^2{\mathbf{w}}^{\text{T}}\mathbf{w}-\alpha V^{\star }+{(V_{\mathbf{Z}}^{\star })}^{\text{T}}\dot{\mathbf{Z}}]}{d\mathbf{w}},\hfill \end{array}$

which give

${\mathbf{L}}^{\star }(\mathbf{u})=-\overline{\mathbf{L}}{\tanh }^{\text{T}}({\mathbf{v}}^{\star }),$

${\mathbf{w}}^{\star }=\frac{1}{2}{\gamma }^{-2}{(V_{\mathbf{Z}}^{⁎})}^{\text{T}}\mathbf{K},$

where

${\mathbf{v}}^{\star }={(V_{\mathbf{Z}}^{\star })}^{\text{T}}\mathbf{G}.$

Substituting (14.43) and (14.44) in Bellman equation (14.42) yields the HJI equation

${\mathbf{Z}}^{\text{T}}{\mathbf{Q}}_{\mathbf{1}}\mathbf{Z}+W({\mathbf{L}}^{\star }(\mathbf{u}))-{\gamma }^2{({\mathbf{w}}^{\star })}^{\text{T}}{\mathbf{w}}^{\star }-\alpha V^{\star }(\mathbf{Z})+{\dot{V}}^{\star }(Z)=0.$