3.2. STANDARD BACKPROPAGATION ALGORITHM 27
e neuron has a bias b, which is summed with the weighted inputs to form the net input
n, which can be expressed by
n D
R
X
j D1
w
j
p
j
C b D W
p
C b: (3.1)
en the net input n passes through an active function f , which generates the neuron
output a:
a D f .n/: (3.2)
In this study, the log-sigmoid activation function is adopted. It can be given by the fol-
lowing expression:
f .x/ D
1
1 C e
x
: (3.3)
us, the multi-input FFNN in Fig. 3.2 implements the following equation:
a
2
D f
2
0
@
S
X
iD1
w
2
1;i
f
1
0
@
R
X
j D1
w
1
i;j
p
j
C b
1
i
1
A
C b
2
1
A
; (3.4)
where
a
2
denotes the output of the overall networks.
R
is the number of inputs,
S
is the number
of neurons in the hidden layer, and p
j
indicates the j th input. f
1
and f
2
are the activation
functions of the hidden layer and output layer, respectively. b
1
i
represents the bias of the ith
neuron in the hidden layer, and b
2
is the bias of the neuron in the output layer. w
1
i;j
represents
the weight connecting the j th input and the ith neuron of the hidden layer, and w
2
1;i
represents
the weight connecting the i th source of the hidden layer to the output layer neuron.
3.2 STANDARD BACKPROPAGATION ALGORITHM
In order to train the established FFNN, the backpropagation algorithm can be utilized [67].
Considering a multilayer FFNN, such as the one with three-layer shown in Fig. 3.2, its operation
can be described using the following equation:
a
mC1
D f
mC1
W
mC1
a
m
C b
mC1
; (3.5)
where a
m
and a
mC1
are the outputs of the mth and (m C 1)th layers of the networks, respectively.
b
mC1
is the bias vector of (m C 1)th layers of the networks. m D 0; 1; : : : ; M 1, where M is
the number of layers of the neural network. e neurons of the first layer obtain inputs:
a
0
D p: (3.6)
Equation (3.6) provides the initial condition for Equation (3.5). e outputs of the neurons in
the last layer can be seen as the overall networks’ outputs:
a D a
M
: (3.7)
28 3. STATE ESTIMATION OF CYBER-PHYSICAL VEHICLE SYSTEMS
e task is to train the network with associations between a specified set of input-output
pairs f.p
1
; t
1
/; .p
2
; t
2
/; : : : ; .p
Q
; t
Q
/g, where p
q
is an input to the network, and t
q
is the corre-
sponding target output. As each input is applied to the network, the network output is compared
to the target.
e backpropagation algorithm uses mean square error as the performance index, which
is to be minimized by adjusting the network parameters, as shown in Equation (3.8):
F .x/ D E
e
T
e
D E
.t a/
T
.t a/
; (3.8)
where x is the vector matrix of network weights and biases. Using the approximate steepest
descent rule, the performance index F .x/ can be approximated by
O
F .x/ D .t.k/ a.k//
T
.t.k/ a.k// D e
T
.k/e.k/; (3.9)
where the expectation of the squared error in Equation (3.8) has been replaced by the squared
error at iteration step k.
e steepest descent algorithm for the approximate mean square error is
w
m
i;j
.k C 1/ D w
m
i;j
.k/ ˛
@
O
F
@w
m
i;j
(3.10)
b
m
i
.k C 1/ D b
m
i
.k/ ˛
@
O
F
@b
m
i
; (3.11)
where ˛ is the learning rate.
Based on the chain rule, the derivatives in Equations (3.10) and (3.11) can be calculated
as:
@
O
F
@w
m
i;j
D
@
O
F
@n
m
i
@n
m
i
@w
m
i;j
;
@
O
F
@b
m
i
D
@
O
F
@n
m
i
@n
m
i
@b
m
i
: (3.12)
We now define s
m
i
as the sensitivity of
O
F to changes in the ith element of the net input
at layer m:
s
m
i
@
O
F
@n
m
i
: (3.13)
Using the defined sensitivity, then the derivatives in Equation (3.12) can be simplified as
@
O
F
@w
m
i;j
D s
m
i
a
m1
j
(3.14)
@
O
F
@b
m
i
D s
m
i
: (3.15)
3.2. STANDARD BACKPROPAGATION ALGORITHM 29
en the approximate steepest descent algorithm can be rewritten in matrix form as:
W
m
.k C 1/ D W
m
.k/ ˛s
m
a
m1
T
(3.16)
b
m
.k C 1/ D b
m
.k/ ˛s
m
; (3.17)
where
s
m
@
O
F
@n
m
D
"
@
O
F
@n
m
1
;
@
O
F
@n
m
2
; : : : ;
@
O
F
@n
m
S
m
#
T
: (3.18)
To derive the recurrence relationship for the sensitivities, the following Jacobian matrix is
utilized:
@n
mC1
@n
m
2
6
6
6
6
6
6
6
6
6
6
6
4
@n
mC1
1
@n
m
1
@n
mC1
1
@n
m
2
@n
mC1
1
@n
m
S
m
@n
mC1
2
@n
m
1
@n
mC1
2
@n
m
2
@n
mC1
2
@n
m
S
m
:
:
:
:
:
:
:
:
:
@n
mC1
S
mC1
@n
m
1
@n
mC1
S
mC1
@n
m
2
@n
mC1
S
mC1
@n
m
S
m
3
7
7
7
7
7
7
7
7
7
7
7
5
: (3.19)
Consider the i , j element in the matrix:
@n
mC1
1
@n
m
1
D w
mC1
i;j
@a
m
j
@n
m
j
D w
mC1
i;j
P
f
m
n
m
j
: (3.20)
us, the Jacobian matrix can be rewritten as
@n
mC1
@n
m
D W
mC1
P
F
m
.
n
m
/
; (3.21)
where
P
F
m
.
n
m
/
D
2
6
6
6
6
6
4
P
f
m
n
m
1
0 : : : 0
0
P
f
m
n
m
2
0
:
:
:
:
:
:
:
:
:
0 0 : : :
P
f
m
n
m
S
m
3
7
7
7
7
7
5
: (3.22)
en the recurrence relation for the sensitivity can be obtained by using the chain rule:
s
m
D
@
O
F
@n
m
D
@n
mC1
@n
m
T
@
O
F
@n
mC1
D
P
F
m
.
n
m
/
W
mC1
T
s
mC1
:
(3.23)
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.