What Exactly Is a Recurrent Neural Network (RNN)?

A subcategory of neural networks known as recurrent neural networks, or RNNs, are advantageous for the modeling of sequence data . RNNs are derived from feed-forward networks and display behavior that is analogous to the way in which human brains operate. Put another way, recurrent neural networks are capable of producing predicted outcomes with sequential data, while other algorithms are unable to do so.

How RNNs Work

In order to have a complete comprehension of RNNs, you will need to have a functional understanding of “regular” feed-forward neural networks as well as sequential data.

The most fundamental definition of sequential data is simply ordered data in which related items follow one another in time. Examples of this include the DNA sequence and financial data. The most common kind of sequential data is probably time series data, which is nothing more than a string of data points that are presented in the appropriate chronological sequence.

The manner in which information is transmitted is what gives RNNs and feed-forward neural networks their respective titles.

A feed-forward network is a single-pass network, which means there is no recurrence in it. Input is passed via various layers, and the output of the network is compared with actual output, and error correction is introduced via the backpropagation (BP) mechanism. This is only helpful in handling one input at a time.

Because it can just handle one input at a time, a feed-forward neural network is terrible at forecasting what will happen next because it has no memory to store the information that it takes in. A feed-forward network does not have any concept of chronological order because it simply takes into account the most recent input. It simply has no memory of anything that occurred in the past, other than the training that it received.

In an RNN, the information is repeated endlessly within a loop. When it comes time to make a choice, it takes into account the most recent input in addition to the lessons it has picked up from the inputs it has previously been given.

Figure 1-1 presents this difference in a visual form.

A representation for the difference between two types of neural networks one is a recurrent neural network and the other is a feed-forward neural network.

Another effective method of illuminating the idea of the memory of a recurrent neural network is to describe it by way of an illustration.

Imagine you have a typical feed-forward neural network and you feed it the word “machine” as an input. You then watch as the network processes the word, one character at a time. Because by the time it reaches the character “h,” it has already forgotten about “m,” “a,” and “c,” it is nearly difficult for this form of neural network to anticipate which character will come next because it has already forgotten about those characters.

However, due to the fact that it possesses its own internal memory, a recurrent neural network is able to remember those characters. It generates output, replicates that output, and then feeds both of those versions back into the network.

To put it another way, recurrent neural networks incorporate recent history into the analysis of the present.

As a result, an RNN takes into consideration both the present and the recent past as its inputs. This is significant because the sequence of data provides essential information about what is to come after it. This is one of the reasons an RNN is able to accomplish tasks that other algorithms are unable to.

The output is first produced by a feed-forward neural network , which, like all other deep learning algorithms, begins by assigning a weight matrix to the inputs of the network. Take note that RNNs assign weights not only to the most recent input but also to the inputs that came before it. In addition, a recurrent neural network will adjust the weights over the course of time using gradient descent as well as backpropagation (BP) .

As things evolved, we discovered that there are challenges with RNNs and their counterparts in terms of processing time and capturing long-term dependencies between words in a sentence. This led us to the evolution of language models, which we describe in the next section.

What Advantages Does Using a Language Model Give Us?

The abstract comprehension of natural language , which is required in order to deduce word probabilities based on context, can be put to use in a variety of contexts and activities.

We are able to execute extractive or abstractive summarization of texts if we have an accurate language model. If we have models for a variety of languages, it will be much simpler to develop a system for automatic translation. Among the more complicated applications is the process of question answering (with or without context). Language models today are being put to very interesting tasks like software code generation, text-to-image generation like DALL-E 2 from OpenAI, as well as other text generation mechanisms like GPT3 and so on.

As explained in the “n-grams ” section, calculating the probabilities of n-grams results in the construction of a straightforward probabilistic language model (a) (an n-gram being an n-word sequence, n being an integer greater than 0). The likelihood of an n-gram can be defined as the conditional probability that the n-gram’s final word follows a specific n-1 gram (leaving out the last word). In everyday terms, it refers to the frequency of occurrence of the final word that comes after the n-1 gram that does not include the last word. Given the n-1 gram, which represents the present, the probabilities of the n-gram, which represents the future, do not depend on the n-2, n-3, etc. grams, which represent the past. This is an assumption made by Markov.

There are obvious disadvantages associated with taking this method. The probability distribution of the following word is only influenced by the n words that came before it. This is the most essential point. Texts that are difficult to understand contain rich contexts, which can have a significant impact on the choice of the next word. Therefore, the identity of the following word might not be discernible from the n words that came before it, even if n is as high as 50.

On top of that, it is obvious that this method does not scale well: the number of possible permutations skyrockets as the size of the dataset (n) increases, despite the fact that the majority of the variants never appear in the actual text. In addition, each and every occurring probability (or each and every n-gram count) needs to be computed and saved! Additionally, non-occurring n-grams cause a sparsity problem because the granularity of the probability distribution might be relatively low when there are so few of them (word probabilities have few different values ; therefore, most of the words have the same probability).

Neural Network–Based Language Models

The manner in which neural network–based language models encode inputs makes it easier to deal with the sparsity problem. Embedding layers produce a vector of arbitrary size for each word, where the semantic links between the words is taken into account. These continuous vectors generate the granularity that is so desperately required in the probability distribution of the following word. In addition, the language model is basically a function (as are all neural networks, which include a great deal of matrix calculations), which means that it is not necessary to store all of the n-gram counts in order to construct the probability distribution of the next word.

The sparsity problem can be solved using neural networks, but there is still a difficulty with the context. First, the process of developing language models consisted of finding ways to solve the context problem in an ever-increasingly effective manner. This was accomplished by bringing in ever-more context words in order to impact the probability distribution in an ever-more effective manner. Second, the objective was to design an architecture that would endow the model with the capability of discovering which phrases in a given context are more significant than others.

Utilizing recurrent neural networks (described in the previous section) is a step in the right direction with regard to the topic of handling context. When selecting the following word, it takes into account all of the words that came before it because it is either an LSTM or a GRU cell-based network.

The fact that RNN-based designs are sequential is the primary downside associated with using these kinds of models. Due to the absence of any opportunity for parallel processing, the amount of time required for training skyrockets when dealing with lengthy sequences. The transformer architecture is the answer to this predicament that you’re having. It is recommended that you read the original document that was produced by Google Brain.

Additionally, OpenAI’s GPT models and Google’s BERT are utilizing the transformer design (which we will discuss in upcoming chapters). In addition to this, these models make use of a technique known as attention , which allows the model to discover which inputs, in particular circumstances, merit more attention than others.

In terms of the architecture of models, the most significant quantum leaps were firstly made by RNNs (specifically LSTM and GRU), which solved the sparsity problem and allowed language models to use a great deal less disk space, and then, more recently, by the transformer architecture, which made parallelization possible and created attention mechanisms. Both of these developments were important. However, architecture is not the only domain in which a language model can demonstrate its prowess.

OpenAI released few language models based on the transformer architecture. The first model was GPT1, and the latest one is GPT3. When contrasted with the architecture of GPT1, GPT3 contains almost no innovative features. But it is massive. It was trained using the largest corpus a model has ever been trained on, which is known as the Common Crawl , and it has 175 billion different parameters. This is made possible in part by the semi-supervised training method of a language model, which allows for the use of a text as a training example while simultaneously omitting some words. The remarkable power of GPT3 stems from the fact that it read virtually all of the text that has been published anywhere on the Internet over the course of the last few years and possesses the capacity to reflect the majority of the complexity that natural language possesses.

In conclusion, I would like to present the T5 model that Google offers (Figure 1-2). In the past, language models were utilized for conventional natural language processing tasks, such as part-of-speech (POS) tagging or machine translation, albeit with a few tweaks here and there. Because of its abstract capability to understand the fundamental structure of natural language, BERT, for instance, may be retrained to function as a POS tagger with only a little bit of additional instruction.

An illustration represents a language model which is representing a Google T5 model which is a text-to-text model this model is translating English to German.

Figure 1-2 shows a representation of the T5 language model .

When using T5 , there is no requirement for any sort of modification in order to complete NLP jobs. If it receives a text that contains certain M tokens, it recognizes that these tokens represent blanks that need to be filled in with the proper words. It is also capable of providing answers. If it is provided with some context after the questions , it will seek that context for the answer; otherwise, it will answer based on the information it already possesses. It is an interesting fact that its designers have been bested by it in a trivia contest. Additional examples of possible applications can be seen in the image (Figure 1-2) on the left.