1.1.1 Artificial Neural Network

Artificial intelligence (AI) also coined as machine intelligence is the concept and development of intelligent machines that work and react like humans. In the twentieth century, AI has a great impact starting from human life to various industries. Researchers from various parts of the world are developing new methodologies based on AI so as to make human life more comfortable. AI has different research areas such as artificial neural network (ANN), genetic algorithm (GA), support vector machine (SVM), fuzzy concept, etc.

In 1943, McCulloh and Pitts [1] designed a computational model for neural network based on mathematical concepts. The concept of perceptron has been proposed by Rosenblatt [2]. In the year 1975, Werbos [3] introduced the well‐known concept of backpropagation algorithm. After the creation of backpropagation algorithm, different researchers explored the various aspects of neural network.

The concept of ANN inspires from the biological neural systems. Neural network consists of a number of identical units known as artificial neurons. Mainly, neural network comprises three layers, viz., input, hidden, and output. Each neuron in the nth layer has been interconnected with the neurons of the (n + 1)th layer by some signal. Each connection has been assigned with weight. The output has been calculated after multiplying each input with its corresponding weight. Further, the output passes through an activation function to get the desired output.

ANN has various applications in the field of engineering and science such as image processing, weather forecasting, function approximation, voice recognition, medical diagnosis, robotics, signal processing, etc.

1.1.2 Types of Neural Networks

There are various types of neural network models, but commonly used models have been discussed further.

1. Feedforward neural network: In feedforward network, the connection between the neurons does not form a cycle. In this case, the information passes through in one direction.
   1. Single‐layer perceptron: It is a simple form of feedforward network without having hidden layers, viz., function link neural network (FLNN).
   2. Multilayer perceptron: Multilayer perceptron is feedforward network, which comprises of three layers, viz., input, hidden, and output.
   3. Convolutional neural network (CNN): CNN is a class of deep neural networks that is mostly used in analyzing visual imagery.
2. Feedback or recurrent neural network: In feedback neural network, the different connections between the neurons form a directed cycle. In this case, the information flow is bidirectional.

The general model of an ANN has been depicted in Figure 1.1.

1.1.3 Learning in Neural Network

There are mainly three types of learning, viz., supervised or learning with a teacher, unsupervised or learning without a teacher, and reinforcement learning. In the supervised case, the desired output of the ANN model is known, but on the other hand, the desired output is unknown in the case of unsupervised learning. In reinforcement learning, the ANN model learns its behavior based on the feedback from the environment. Various learning rules have been developed to train or update the weights. Some of them are mentioned below.

1. (a) Hebbian learning rule
2. (b) Perceptron learning rule
3. (c) Delta learning rule or backpropagation learning rule
4. (d) Out star learning rule
5. (e) Correlation learning rule

1.1.4 Activation Function

An output node receives a number of input signals, depending on the intensity of input signals, we summed these input signals and pass them through some function known as activation function. Depending on the input signals, it generates an output and accordingly further actions have been performed. There are different types of activation functions from which some of them are listed below.

1. Identity function
2. Bipolar step function
3. Binary step function
4. Sigmoidal function
   • Unipolar sigmoidal function
   • Bipolar sigmoidal function
5. Tangent hyperbolic function
6. Rectified linear unit (ReLU)

Although we have mentioned a number of activation functions, but for the sake of completeness, we have just discussed about sigmoidal function below.

1.1.5 Advantages of Neural Network

1. (a) The ANN model works with incomplete or partial information.
2. (b) Information processing has been done in a parallel manner.
3. (c) An ANN architecture learns and need not to be programmed.
4. (d) It can handle large amount of data.
5. (e) It is less sensitive to noise.

1.1.6 Functional Link Artificial Neural Network (FLANN)

Functional Link Artificial Neural Network (FLANN) is a class of higher order single‐layer ANN models. FLANN model is a flat network without an existence of a hidden layer, which makes the training algorithm of the network less complicated. The single‐layer FLANN model is developed by Pao [4,5] for function approximation and pattern recognition with less computational cost and faster convergence rate. In FLANN, the hidden layer is replaced by a functional expansion block for artificial enhancement of the input patterns using linearly independent functions such as orthogonal polynomials, trigonometric polynomials, exponential polynomials, etc. Being a single‐layer model, its number of parameters is less than multilayer ANN. Thus, FLNN model has less computational complexity and higher convergence speed. In recent years, many researchers have commonly used various types of orthogonal polynomials, viz., Chebyshev, Legendre, Hermite, etc., as a basis of FLANN.

As such, Chebyshev neural network model is widely used for handling various complicated problems, viz., function approximation [6], channel equalization [7], digital communication [8], solving nonlinear differential equations (DEs) [9,10], system identification [11,12], etc. Subsequently, Legendre neural network has been used for solving DEs [13], system identification [14], and for prediction of machinery noise [15]. Also, Hermite neural network is used for solving Van der Pol Duffing oscillator equations [16,17].

Figure 1.3 shows the structure of FLANN model, which consists of an input layer, a set of basis functions, a functional expansion block, and an output layer. The input layer contains T = [t₁, t₂, t₃, …, t_m] input vectors, a set of m basis functions {ψ₁, ψ₂, …, ψ_m} ∈ B_M, and weight vectors W = [w₁, w₂, …, w_m] from input to output layer associated with functional expansion block. The output of FLANN with the following input–output relationship may be written as

here, T ∈ R^m and denote the input and output vectors of FLANN, respectively, ψ_j(T) is the expanded input data with respect to basis functions, and w_ji stands for the weight associated with the jth output. The nonlinear differentiable functions, viz., sigmoid, tan hyperbolic, etc., are considered as an activation function λ(·) of single‐layer FLANN model.

1.1.7 Differential Equations (DEs)

Mathematical representation of scientific and engineering problem(s) is called a mathematical model. The corresponding equation for the system is generally given by differential equation (ordinary and partial), and it depends on the particular physical problem. As such, there exist various methods, namely, exact solution when the differential equations are simple, and numerical methods, viz., Runge–Kutta, predictor–corrector, finite difference, finite element when the differential equations are complex. Although these numerical methods provide good approximations to the solution, they require a discretization of the domain (via meshing), which may be challenging in two or higher dimensional problems.

In recent decade, many researchers have used ANN techniques for solving various types of ordinary differential equations (ODEs) [18–21] and partial differential equations (PDEs) [22–25]. ANN‐based approximate solutions of DEs are more advantageous than other traditional numerical methods. ANN method is general and can be applied to solve linear and nonlinear singular initial value problems. Computational complexity does not increase considerably with the increase in number of sampling points and dimension of the problem. Also, we may use ANN model as a black box to get numerical results at any arbitrary point in the domain after training of the model.

1.1.8 Integral Equation

Integral equations are defined as the equations where an unknown function appears under an integral sign. There are mainly two types of integral equations, viz., Fredholm and Volterra. ANN is a novel technique for handling integral equations. As such, some articles related to integral equations have been discussed. Golbabai and Seifollahi [26] proposed a novel methodology based on radial basis function network for solving linear second‐order integral equations of Fredholm and Volterra types. An radial basis function (RBF) network has been presented by Golbabai et al. [27] to find the solution of a system of nonlinear integral equations. Jafarian and Nia [28] introduced ANN architecture for solving second kind linear Fredholm integral systems. A novel ANN‐based model has been developed by Jafarian et al. [29] for solving a second kind linear Volterra integral equation system. S. Effati and Buzhabadi [30] proposed a feedforward neural network‐based methodology for solving Fredholm integral equations of second kind. Bernstein polynomial expansion method has been used by Jafarian et al. [31] for solving linear second kind Fredholm and Volterra integral equation systems. Kitamura and Qing [32] found the numerical solution of Fredholm integral equations of first kind using ANN. Asady et al. [33] constructed a feedforward ANN architecture to solve two‐dimensional Fredholm integral equations. Radial basis functions have been chosen to find an iterative solution for linear integral equations of second kind by Golbabai and Seifollahi [34]. Cichocki and Unbehauen [35] gave a neural network‐based method for solving a linear system of equations. A multilayer ANN architecture has been proposed by Jeswal and Chakraverty [36] for solving transcendental equation.

1.1.8.1 Fredholm Integral Equation of First Kind

The Fredholm equation of first kind can be defined as

where K is called kernel and φ and f are the unknown and known functions, respectively. The limits of integration are constants.

1.1.8.2 Fredholm Integral Equation of Second Kind

The Fredholm equation of second kind is

here, λ is an unknown parameter.

1.1.8.3 Volterra Integral Equation of First Kind

The Volterra integral equation of first kind is defined as

1.1.8.4 Volterra Integral Equation of Second Kind

The Volterra Integral equation of second kind may be written as

1.1.8.5 Linear Fredholm Integral Equation System of Second Kind

A system of linear Fredholm integral equations of second kind may be formulated as [28]

here, u, v ∈ [a, b] and L(u), K(v, u) are real functions. G(u) = [G₁(u), G₂(u), …, G_n(u)] is the solution set to be found.

1.2.1 FLANN‐Based General Formulation of Differential Equations

In this section, general formulation of differential equations using FLANN has been described. In particular, the formulations of ODEs are discussed here.

In general, differential equations (ordinary as well as partial) can be written as [18]

(1.1)

where F defines the structure of DEs, ∇ stands for differential operator, and y(t) is the solution of DEs. It may be noted that for ODEs and for PDEs . Let y_Lg(t, p) denotes the trial solution of FLANN (Laguerre neural network, LgNN) with adjustable parameters p and then the above general differential equation changes to the form

(1.2)

The trial solution y_Lg(t, p) satisfies the initial and boundary conditions and may be formulated as

(1.3)

where K(t) satisfies the boundary/initial conditions and contains no weight parameters. N(t, p) denotes the single output of FLANN with parameters p and input t. H(t, N(t, p)) makes no contribution to the initial or boundary conditions, but it is used in FLANN where weights are adjusted to minimize the error function.

The general form of corresponding error function for the ODE can be expressed as

(1.4)

We take some particular cases in Sections 1.2.1.1 and 1.2.1.2.

1.2.1.1 Second‐Order Initial Value Problem

Second‐order initial value problem may be written as

(1.5)

with initial conditions y(a) = C, .

The LgNN trial solution is formulated as

(1.6)

The error function is computed as follows

(1.7)

1.2.1.2 Second‐Order Boundary Value Problem

Next, let us consider a second‐order boundary value problem as

(1.8)

with boundary conditions y(a) = A, y(b) = B.

The LgNN trial solution for the above boundary value problem is

(1.9)

In this case, the error function is the same as Eq. (1.7).

1.2.2 Proposed Laguerre Neural Network (LgNN) for Differential Equations

Here, we have included architecture of LgNN, general formulation for DEs, and its learning algorithm with gradient computation of the ANN parameters with respect to its inputs.

1.2.2.1 Architecture of Single‐Layer LgNN Model

Structure of single‐layer LgNN model has been displayed in Figure 1.4. LgNN is one type of FLANN model based on Laguerre polynomials. LgNN consists of single‐input node, single‐output node, and functional expansion block of Laguerre orthogonal polynomials. The hidden layer is removed by transforming the input data to a higher dimensional space using Laguerre polynomials. Laguerre polynomials are the set of orthogonal polynomials obtained by solving Laguerre differential equation

(1.10)

These are the first few Laguerre polynomials

(1.11)

The higher order Laguerre polynomials may be generated by recursive formula

(1.12)

Here, we have considered that one input node has h number of data, i.e. t = (t₁, t₂, …, t_h)^T. Then, the functional expansion block of input data is determined by using Laguerre polynomials as

(1.13)

In view of the above discussion, main advantages of the single‐layer LgNN model for solving differential equations may be given as follows:

1. The total number of network parameters is less than that of traditional multilayer ANN.
2. It is capable of fast learning and is computationally efficient.
3. It is simple to implement.
4. There are no hidden layers in between the input and output layers.
5. The backpropagation algorithm is unsupervised.
6. No other optimization technique is to be used.

1.2.2.2 Training Algorithm of Laguerre Neural Network (LgNN)

Backpropagation (BP) algorithm with unsupervised training has been applied here for handling differential equations. The idea of BP algorithm is to minimize the error function until the FLANN learned the training data. The gradient descent algorithm has been used for training and the network parameters (weights) are updated by taking negative gradient at each step (iteration). The training of the single‐layer LgNN model is to start with random weight vectors from input to output layer.

We have considered the nonlinear tangent hyperbolic (tanh) function, viz., , as the activation function.

The output of LgNN may be formulated as

(1.14)

where t is the input data, p stands for weights of LgNN, and s is the weighted sum of expanded input data. It is formulated as

(1.15)

where Lg_k − 1(t) and w_k with k = {1, 2, 3, …, m} are the expanded input and weight vector, respectively, of the LgNN model.

The weights are modified by taking negative gradient at each step of iteration

(1.16)

here, ρ is the learning parameter, T is the step of iteration, and E(t, p) denotes the error function of LgNN model.

1.2.2.3 Gradient Computation of LgNN

Derivatives of the network output with respect to the corresponding inputs are required for error computation. As such, derivative of N(t, p) with respect to t is

(1.17)

Similarly, the second derivative of N(t, p) can be computed as

(1.18)

After simplifying Eq. (1.18), we get

(1.19)

where w_k denotes the weights of LgNN and are the first and second derivatives of Laguerre polynomials, respectively.

Differentiate Eq. (1.6) with respect to t

(1.20)

Finally, the converged LgNN results may be used in Eqs. (1.6) and (1.9) to obtain the approximate results.

1.3.1 Algorithm

• Step 1: Choose the precision tolerance “tol,” learning rate η, random weights, q = 0 and E = 0;
• Step 2: Calculate , e_i = m_i − z_i;
• Step 3: The error term is given by
  

  and weight updating formula is

  

  where

  

  If E < tol, then go to step 2, else go to step 4.

• Step 4: Print y.

1.4.1 Differential Equations and Applications

In this section, a boundary value problem, a nonlinear Lane‐Emden equation, and an application problem of Duffing oscillator equation have been investigated to show the efficiency and powerfulness of the proposed method. In all cases, the feedforward neural network and unsupervised BP algorithm have been used for training. The initial weights from the input layer to the output layer are taken randomly.

1.4.2 Integral Equations

In this section, we have investigated three example problems of a system of Fredholm integral equations of second kind. The system of integral equations may be converted to a linear system using the trapezoidal rule. As such, a feedforward neural network model has been proposed for solving linear system.