Now we will create the RNN model which takes a few input parameters such as:

• size_layer: The number of units in the RNN cell
• num_layers: Number of hidden layers
• embedded_size: The size of the embedding
• dict_size: The vocabulary size
• dimension_output: Number of classes we need to classify
• learning_rate: The learning rate of the optimization algorithm

The architecture of our RNN model consists of the following parts:

1. Two placeholders one to feed sequence data into the model and second placeholder for the output.
2. A variable to store the embedding lookup from the dictionary.
3. Then adding RNN layer with multiple basic RNN cells.
4. Create weight and bias variables
5. Compute logits 
6. Compute loss
7. Add Adam optimizer
8. Calculate prediction and accuracy.

This model is similar to what we created in the previous chapter for CNN's except the RNN cell part:

class Model:
    def __init__(self, size_layer, num_layers, embedded_size,
                 dict_size, dimension_output, learning_rate):
        
        def cells(reuse=False):
            return tf.nn.rnn_cell.BasicRNNCell(size_layer,reuse=reuse)
        
        self.X = tf.placeholder(tf.int32, [None, None])
        self.Y = tf.placeholder(tf.float32, [None, dimension_output])

        encoder_embeddings = tf.Variable(tf.random_uniform([dict_size, embedded_size], -1, 1))
        encoder_embedded = tf.nn.embedding_lookup(encoder_embeddings, self.X)

        rnn_cells = tf.nn.rnn_cell.MultiRNNCell([cells() for _ in range(num_layers)])
        outputs, _ = tf.nn.dynamic_rnn(rnn_cells, encoder_embedded, dtype = tf.float32)

        W = tf.get_variable('w',shape=(size_layer, dimension_output),initializer=tf.orthogonal_initializer())
        b = tf.get_variable('b',shape=(dimension_output),initializer=tf.zeros_initializer())

        self.logits = tf.matmul(outputs[:, -1], W) + b
        self.cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = self.logits, labels = self.Y))
        self.optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(self.cost)

        correct_pred = tf.equal(tf.argmax(self.logits, 1), tf.argmax(self.Y, 1))
        self.accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32))

In this model, the data flows from the variables which we created in step 1. Then it moves to the embedding layer defined in step 2, further we have RNN layer, which performs the computation in 2 hidden layers of RNN cells. Later logits are computed by performing matrix multiplication of weight and the output from the RNN layer and addition of bias.  The last step is that we define the cost function we will be using the softmax cross entropy function.

This is what the complete model looks like after computation.

The image below represents the structure of the RNN block from the above image. In this architecture, we have two RNN cells which are incorporated in the hidden layers.