We will now switch our focus to recurrent neural networks and study a simple recurrent neural network model. An Elman neural network is a simple recurrent ANN with a single input, output, and hidden layer. There is also an extra context layer of neural nodes. Elman neural networks are used to simulate short-term memory in supervised and unsupervised machine learning problems. Enclog does include support for Elman neural networks, and we will demonstrate how we can build an Elman neural network using the Enclog library.

The context layer of an Elman neural network receives unweighted inputs from the hidden layer of the ANN. In this way, the ANN can remember the previous values that we generated using the hidden layer and use these values to affect the predicted value. Thus, the context layer serves as a type of short-term memory for the ANN. An Elman neural network can be illustrated by the following diagram:

The structure of an Elman network, as depicted by the preceding diagram, resembles that of a feed-forward multilayer perceptron ANN. An Elman network adds an extra context layer of neural nodes to the ANN. The Elman network illustrated in the preceding diagram takes two inputs and produces two outputs. The input and hidden layers of the Elman network add an extra bias input, similar to a multilayer perceptron. The activations of the hidden layers' neurons are fed directly to the two context nodes and . The values stored in these context nodes are then used later by the nodes in the hidden layer of the ANN to recollect the previous activations to determine the new activation values.

We can create an Elman network that specifies the :elman keyword to the neural-pattern function from the Enclog library as follows:

(def elman-network (network (neural-pattern :elman)
                             :activation :sigmoid
                             :input      2
                             :output     1
                             :hidden     [3]))

To train the Elman network, we can use the resilient propagation algorithm (for more information, refer to Empirical Evaluation of the Improved Rprop Learning Algorithm). This algorithm can also be used to train other recurrent networks supported by Enclog. Interestingly, the resilient propagation algorithm can be used to train feed-forward networks as well. This algorithm also performs significantly better than the backpropagation learning algorithm. Although a complete description of this algorithm is beyond the scope of this book, the reader is encouraged to learn more about this learning algorithm. The resilient propagation algorithm is specified as the :resilient-prop keyword to the train-network function, which we had defined earlier. We can train the Elman neural network using the train-network function and the dataset variable as follows:

user> (def EN (train-network elman-network dataset 
                             :resilient-prop))
Iteration # 1 Error: 26.461526% Target-Error: 1.000000%
Iteration # 2 Error: 25.198031% Target-Error: 1.000000%
Iteration # 3 Error: 25.122343% Target-Error: 1.000000%
Iteration # 4 Error: 25.179218% Target-Error: 1.000000%
...
...
Iteration # 99 Error: 0.979165% Target-Error: 1.000000%
#'user/EN

As shown in the preceding code, the resilient propagation algorithm requires a relatively smaller number of iterations in comparison to the backpropagation algorithm. We can now use this trained ANN to simulate an XOR gate just like we did in the previous example.

In summary, recurrent neural network models and training algorithms are the other useful models that can be used to model classification or regression problems using ANNs.