Now to the part that all Deep Learning Engineers love: designing the model architecture!  We will be using  four distinctive types of layers in our model architecture:

• LSTM a type of an RNN layer.
• Dropout a technique for regularisation. This helps prevent the model from overfitting by randomly dropping some nodes.
• Dense is a fully connected layer where every input node is connected to every output node.
• Activation determines the activation function to be used to produce the node's output.

We will again employ the Keras APIs to make the implementation quick:

model = Sequential()
model.add(LSTM(
    256,
    input_shape=(network_input.shape[1], network_input.shape[2]),
    return_sequences=True
))
model.add(Dropout(0.5))
model.add(LSTM(512, return_sequences=True))
model.add(Dropout(0.3))
model.add(LSTM(256))
model.add(Dense(256))
model.add(Dropout(0.3))
model.add(Dense(n_vocab))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', 
                  optimizer='rmsprop', 
                  metrics=['accuracy'])

The generative model architecture we designed has three LSTM layers, three Dropout layers, two Dense layers and one activation layer as shown in the figure below. Categorical cross entropy will be used to calculate the loss for each iteration of the training.  We will again use the Adam optimizer in this network. Now that we have our Deep Learning model architecture configured, it's time to train the model. We have decided to train the model for 200 epochs, each with 25 batches using model.fit(). We also want to track the reduction in loss over each epoch and will use checkpoints for this purpose.

This figure helps us visualize our deep learning model:

filepath = "weights-{epoch:02d}-{loss:.4f}.hdf5"
checkpoint = ModelCheckpoint(
    filepath,
    monitor='loss',
    verbose=0,
    save_best_only=True,
    mode='min'
)
callbacks_list = [checkpoint]

history = model.fit(network_input, network_output, epochs=200, batch_size=64, callbacks=callbacks_list)

The performance of the model can be seen as follows:

Now that the training process is completed, we will load the trained models and generate our own music.