Musical Notation

We’ll be using the Python library music21 to load the MIDI files into Python for processing. Example 7-1 shows how to load a MIDI file and visualize it (Figure 7-1), both as a score and as structured data.

from music21 import converter

dataset_name = 'cello'
filename = 'cs1-2all'
file = "./data/{}/{}.mid".format(dataset_name, filename)

original_score = converter.parse(file).chordify()

Figure 7-1. Musical notation

We use the chordify method to squash all simultaneously played notes into chords within a single part, rather than them being split between many parts. Since this piece is performed by one instrument (a cello), we are justified in doing this, though sometimes we may wish to keep the parts separate to generate music that is polyphonic in nature. This presents additional challenges that we shall tackle later on in this chapter.

The code in Example 7-2 loops through the score and extracts the pitch and duration for each note (and rest) in the piece into two lists. Individual notes in chords are separated by a dot, so that the whole chord can be stored as a single string. The number after each note name indicates the octave that the note is in—since the note names (A to G) repeat, this is needed to uniquely identify the pitch of the note. For example, G2 is an octave below G3.

notes = []
durations = []

for element in original_score.flat:

    if isinstance(element, chord.Chord):
        notes.append('.'.join(n.nameWithOctave for n in element.pitches))
        durations.append(element.duration.quarterLength)

    if isinstance(element, note.Note):
        if element.isRest:
            notes.append(str(element.name))
            durations.append(element.duration.quarterLength)
        else:
            notes.append(str(element.nameWithOctave))
            durations.append(element.duration.quarterLength)

The output from this process is shown in Table 7-1.

The resulting dataset now looks a lot more like the text data that we have dealt with previously. The words are the pitches, and we should try to build a model that predicts the next pitch, given a sequence of previous pitches. The same idea can also be applied to the list of durations. Keras gives us the flexibility to be able to build a model that can handle the pitch and duration prediction simultaneously.

Attention

The attention mechanism was originally applied to text translation problems—in particular, translating English sentences into French.

In the previous chapter, we saw how encoder–decoder networks can solve this kind of problem, by first passing the input sequence through an encoder to generate a context vector, then passing this vector through the decoder network to output the translated text. One observed problem with this approach is that the context vector can become a bottleneck. Information from the start of the source sentence can become diluted by the time it reaches the context vector, especially for long sentences. Therefore these kinds of encoder–decoder networks sometimes struggle to retain all the required information for the decoder to accurately translate the source.

As an example, suppose we want the model to translate the following sentence into German: I scored a penalty in the football match against England.

Clearly, the meaning of the entire sentence would be changed by replacing the word scored with missed. However, the final hidden state of the encoder may not be able to sufficiently retain this information, as the word scored appears early in the sentence.

The correct translation of the sentence is: Ich habe im Fußballspiel gegen England einen Elfmeter erzielt.

If we look at the correct German translation, we can see that the word for scored (erzielt) actually appears right at the end of the sentence! So not only would the model have to retain the fact that the penalty was scored rather than missed through the encoder, but also all the way through the decoder as well.

Exactly the same principle is true in music. To understand what note or sequence of notes is likely to follow from a particular given passage of music, it may be crucial to use information from far back in the sequence, not just the most recent information. For example, take the opening passage of the Prelude to Bach’s Cello Suite No. 1 (Figure 7-4).

Figure 7-4. The opening of Bach’s Cello Suite No. 1 (Prelude)

What note do you think comes next? Even if you have no musical training you may still be able to guess. If you said G (the same as the very first note of the piece), then you’d be correct. How did you know this? You may have been able to see that every bar and half bar starts with the same note and used this information to inform your decision. We want our model to be able to perform the same trick—in particular, we want it to not only care about the hidden state of the network now, but also to pay particular attention to the hidden state of the network eight notes ago, when the previous low G was registered.

The attention mechanism was proposed to solve this problem. Rather than only using the final hidden state of the encoder RNN as the context vector, the attention mechanism allows the model to create the context vector as a weighted sum of the hidden states of the encoder RNN at each previous timestep. The attention mechanism is just a set of layers that converts the previous encoder hidden states and current decoder hidden state into the summation weights that generate the context vector.

If this sounds confusing, don’t worry! We’ll start by seeing how to apply an attention mechanism after a simple recurrent layer (i.e., to solve the problem of predicting the next note of Bach’s Cello Suite No. 1), before we see how this extends to encoder–decoder networks, where we want to predict a whole sequence of subsequent notes, rather than just one.

Building an Attention Mechanism in Keras

First, let’s remind ourselves of how a standard recurrent layer can be used to predict the next note given a sequence of previous notes. Figure 7-5 shows how the input sequence (x₁,…,x_n) is fed to the layer one step at a time, continually updating the hidden state of the layer. The input sequence could be the note embeddings, or the hidden state sequence from a previous recurrent layer. The output from the recurrent layer is the final hidden state, a vector with the same length as the number of units. This can then be fed to a Dense layer with softmax output to predict a distribution for the next note in the sequence.

Figure 7-5. A recurrent layer for predicting the next note in a sequence, without attention

Figure 7-6 shows the same network, but this time with an attention mechanism applied to the hidden states of the recurrent layer.

Figure 7-6. A recurrent layer for predicting the next note in a sequence, with attention

Let’s walk through this process step by step:

1. First, each hidden state h_j (a vector of length equal to the number of units in the recurrent layer) is passed through an alignment function a to generate a scalar, e_j. In this example, this function is simply a densely connected layer with one output unit and a tanh activation function.

2. Next, the softmax function is applied to the vector e₁,…,e_n to produce the vector of weights α₁,…,α_n.

3. Lastly, each hidden state vector h_j is multiplied by its respective weight α_j, and the results are then summed to give the context vector c (thus c has the same length as a hidden state vector).

The context vector can then be passed to a Dense layer with softmax output as usual, to output a distribution for the potential next note.

This network can be built in Keras as shown in Example 7-3.

notes_in = Input(shape = (None,)) 
durations_in = Input(shape = (None,))

x1 = Embedding(n_notes, embed_size)(notes_in) 
x2 = Embedding(n_durations, embed_size)(durations_in)

x = Concatenate()([x1,x2]) 

x = LSTM(rnn_units, return_sequences=True)(x) 
x = LSTM(rnn_units, return_sequences=True)(x)

e = Dense(1, activation='tanh')(x) 
e = Reshape([-1])(e)

alpha = Activation('softmax')(e) 

c = Permute([2, 1])(RepeatVector(rnn_units)(alpha)) 
c = Multiply()([x, c])
c = Lambda(lambda xin: K.sum(xin, axis=1), output_shape=(rnn_units,))(c)

notes_out = Dense(n_notes, activation = 'softmax', name = 'pitch')(c) 
durations_out = Dense(n_durations, activation = 'softmax', name = 'duration')(c)

model = Model([notes_in, durations_in], [notes_out, durations_out]) 

att_model = Model([notes_in, durations_in], alpha) 

opti = RMSprop(lr = 0.001)
model.compile(loss=['categorical_crossentropy', 'categorical_crossentropy'],
              optimizer=opti) 

There are two inputs to the network: the sequence of previous note names and duration values. Notice how the sequence length isn’t specified—the attention mechanism does not require a fixed-length input, so we can leave this as variable.

The Embedding layers convert the integer values of the note names and durations into vectors.

The vectors are concatenated to form one long vector that will be used as input into the recurrent layers.

Two stacked LSTM layers are used as the recurrent part of the network. Notice how we set return_sequences to True to make each layer pass the full sequence of hidden states to the next layer, rather than just the final hidden state.

The alignment function is just a Dense layer with one output unit and tanh activation. We can use a Reshape layer to squash the output to a single vector, of length equal to the length of the input sequence (seq_length).

The weights are calculated through applying a softmax activation to the alignment values.

To get the weighted sum of the hidden states, we need to use a RepeatVector layer to copy the weights rnn_units times to form a matrix of shape [rnn_units, seq_length], then transpose this matrix using a Permute layer to get a matrix of shape [seq_length, rnn_units]. We can then multiply this matrix pointwise with the hidden states from the final LSTM layer, which also has shape [seq_length, rnn_units]. Finally, we use a Lambda layer to perform the summation along the seq_length axis, to give the context vector of length rnn_units.

The network has a double-headed output, one for the next note name and one for the next note length.

The final model accepts the previous note names and note durations as input and outputs a distribution for the next note name and next note duration.

We also create a model that outputs the alpha layer vector, so that we will be able to understand how the network is attributing weights to previous hidden states.

The model is compiled using categorical_crossentropy for both the note name and note duration output heads, as this is a multiclass classification problem.

A diagram of the full model built in Keras is shown in Figure 7-7.

Figure 7-7. The LSTM model with attention for predicting the next note in a sequence

You can train this LSTM with attention by running the notebook called 07_02_lstm_compose_train.ipynb in the book repository.

Analysis of the RNN with Attention

The following analysis can be produced by running the notebook 07_03_lstm_compose_analysis.ipynb from the book repository, once you have trained your network.

We’ll start by generating some music from scratch, by seeding the network with only a sequence of <START> tokens (i.e., we are telling the model to assume it is starting from the beginning of the piece). Then we can generate a musical passage using the same iterative technique we used in Chapter 6 for generating text sequences, as follows:

1. Given the current sequence (of note names and note durations), the model predicts two distributions, for the next note name and duration.

2. We sample from both of these distributions, using a temperature parameter to control how much variation we would like in the sampling process.

3. The chosen note is stored and its name and duration are appended to the respective sequences.

4. If the length of the sequence is now greater than the sequence length that the model was trained on, we remove one element from the start of the sequence.

5. The process repeats with the new sequence, and so on, for as many notes as we wish to generate.

Figure 7-8 shows examples of music generated from scratch by the model at various epochs of the training process.

Most of our analysis in this section will focus on the note pitch predictions, rather than rhythms, as for Bach’s Cello Suites the harmonic intricacies are more difficult to capture and therefore more worthy of investigation. However, you can also apply the same analysis to the rhythmic predictions of the model, which may be particularly relevant for other styles of music that you could use to train this model (such as a drum track).

There are several points to note about the generated passages in Figure 7-8. First, see how the music is becoming more sophisticated as training progresses. To begin with, the model plays it safe by sticking to the same group of notes and rhythms. By epoch 10, the model has begun to generate small runs of notes, and by epoch 20 it is producing interesting rhythms and is firmly established in a set key (E-flat major).

Figure 7-8. Some examples of passages generated by the model when seeded only with a sequence of <START> tokens; here we use a temperature of 0.5 for the note names and durations

Second, we can analyze the distribution of note pitches over time by plotting the predicted distribution at each timestep as a heatmap. Figure 7-9 shows this heatmap for the example from epoch 20 in Figure 7-8.

Figure 7-9. The distribution of possible next notes over time (at epoch 20): the darker the square, the more certain the model is that the next note is at this pitch

An interesting point to note here is that the model has clearly learned which notes belong to particular keys, as there are gaps in the distribution at notes that do not belong to the key. For example, there is a gray gap along the row for note 54 (corresponding to Gb/F#). This note is highly unlikely to appear in a piece of music in the key of E-flat major. Early on in the generation process (the lefthand side of the diagram) the key is not yet firmly established and therefore there is more uncertainty in how to choose the next note. As the piece progresses, the model settles on a key and certain notes become almost certain not to appear. What is remarkable is that the model hasn’t explicitly decided to set the music in a certain key at the beginning, but instead is literally making it up as it goes along, trying to choose the note that best fits with those it has chosen previously.

It is also worth pointing out that the model has learned Bach’s characteristic style of dropping to a low note on the cello to end a phrase and bouncing back up again to start the next. See how around note 20, the phrase ends on a low E-flat—it is common in the Bach Cello Suites to then return to a higher, more sonorous range of the instrument for the start of next phrase, which is exactly what the model predicts. There is a large gray gap between the low E-flat (pitch number 39) and the next note, which is predicted to be around pitch number 50, rather than continuing to rumble around the depths of the instrument.

Lastly, we should check to see if our attention mechanism is working as expected. Figure 7-10 shows the values of the alpha vector elements calculated by the network at each point in the generated sequence. The horizontal axis shows the generated sequence of notes; the vertical axis shows where the attention of the network was aimed when predicting each note along the horizontal axis (i.e., the alpha vector). The darker the square, the greater the attention placed on the hidden state corresponding to this point in the sequence.

Figure 7-10. Each square in the matrix indicates the amount of attention given to the hidden state of the network corresponding to the note on the vertical axis, at the point of predicting the note on the horizontal axis; the more red the square, the more attention was given

We can see that for the second note of the piece (B-3 = B-flat), the network chose to place almost all of its attention on the fact that the first note of the piece was also B-3. This makes sense; if you know that the first note is a B-flat, you will probably use this information to inform your decision about the next note.

As we move through the next few notes, the network spreads its attention roughly equally among previous notes—however, it rarely places any weight on notes more than six notes ago. Again, this makes sense; there is probably enough information contained in the previous six hidden states to understand how the phrase should continue.

There are also examples of where the network has chosen to ignore a certain note nearby, as it doesn’t add any additional information to its understanding of the phrase. For example, take a look inside the white box marked in the center of the diagram, and note how there is a strip of boxes in the middle that cuts through the usual pattern of looking back at the previous four to six notes. Why would the network willingly choose to ignore this note when deciding how to continue the phrase?

If you look across to see which note this corresponds to, you can see that it is the first of three E-3 (E-flat) notes. The model has chosen to ignore this because the note prior to this is also an E-flat, an octave lower (E-2). The hidden state of the network at this point will provide ample information for the model to understand that E-flat is an important note in this passage, and therefore the model does not need to pay attention to the subsequent higher E-flat, as it doesn’t add any extra information.

Additional evidence that the model has started to understand the concept of an octave can be seen inside the green box below and to the right. Here the model has chosen to ignore the low G (G2) because the note prior to this was also a G (G3), an octave higher. Remember we haven’t told the model anything about which notes are related through octaves—it has worked this out for itself just by studying the music of J.S. Bach, which is remarkable.

Attention in Encoder–Decoder Networks

The attention mechanism is a powerful tool that helps the network decide which previous states of the recurrent layer are important for predicting the continuation of a sequence. So far, we have seen this for one-note-ahead predictions. However, we may also wish to build attention into encoder–decoder networks, where we predict a sequence of future notes by using an RNN decoder, rather than building up sequences one note at a time.

To recap, Figure 7-11 shows how a standard encoder–decoder model for music generation might look, without attention—the kind that we introduced in Chapter 6.

Figure 7-12 shows the same network, but with an attention mechanism between the encoder and the decoder.

Figure 7-11. The standard encoder–decoder model

Figure 7-12. An encoder–decoder model with attention

The attention mechanism works in exactly the same way as we have seen previously, with one alteration: the hidden state of the decoder is also rolled into the mechanism so that the model is able to decide where to focus its attention not only through the previous encoder hidden states, but also from the current decoder hidden state. Figure 7-13 shows the inner workings of an attention module within an encoder–decoder framework.

Figure 7-13. An attention mechanism within the context of an encoder-decoder network, connected to decoder cell i

While there are many copies of the attention mechanism within the encoder–decoder network, they all share the same weights, so there is no extra overhead in the number of parameters to be learned. The only change is that now, the decoder hidden state is rolled into the attention calculations (the red lines in the diagram). This slightly changes the equations to incorporate an extra index (i) to specify the step of the decoder.

Also notice how in Figure 7-11 we use the final state of the encoder to initialize the hidden state of the decoder. In an encoder–decoder with attention, we instead initialize the decoder using the built-in standard initializers for a recurrent layer. The context vector c_i is concatenated with the incoming data y_i–1 to form an extended vector of data into each cell of the decoder. Thus, we treat the context vectors as additional data to be fed into the decoder.

Generating Polyphonic Music

The RNN with attention mechanism framework that we have explored in this section works well for single-line (monophonic) music, but could it be adapted to multiline (polyphonic) music?

The RNN framework is certainly flexible enough to conceive of an architecture whereby multiple lines of music are generated simultaneously, through a recurrent mechanism. But as it stands, our current dataset isn’t well set up for this, as we are storing chords as single entities rather than parts that consist of multiple individual notes. There is no way for our current RNN to know, for example, that a C-major chord (C, E, and G) is actually very close to an A-minor chord (A, C, and E)—only one note would need to change, the G to an A. Instead, it treats both as two distinct elements to be predicted independently.

Ideally, we would like to design a network that can accept multiple channels of music as individual streams and learn how these streams should interact with each other to generate beautiful-sounding music, rather than disharmonious noise.

Doesn’t this sound a bit like generating images? For image generation we have three channels (red, green, and blue), and we want the network to learn how to combine these channels to generate beautiful-looking images, rather than random pixelated noise.

In fact, as we shall see in the next section, we can treat music generation directly as an image generation problem. This means that instead of using recurrent networks we can apply the same convolutional-based techniques that worked so well for image generation problems to music—in particular, GANs.

Before we explore this new architecture, there is just enough time to visit the concert hall, where a performance is about to begin…

Chords, Style, Melody, and Groove

Let’s now take a closer look at the four different inputs that feed the generator.

def conv_t(self, x, f, k, s, a, p, bn):
    x = Conv2DTranspose(
                filters = f
                , kernel_size = k
                , padding = p
                , strides = s
                , kernel_initializer = self.weight_init
                )(x)

    if bn:
        x = BatchNormalization(momentum = 0.9)(x)

    if a == 'relu':
        x = Activation(a)(x)
    elif a == 'lrelu':
        x = LeakyReLU()(x)

    return x


def TemporalNetwork(self):

    input_layer = Input(shape=(self.z_dim,), name='temporal_input') 

    x = Reshape([1,1,self.z_dim])(input_layer) 
    x = self.conv_t(x, f=1024, k=(2,1), s=(1,1), a= 'relu',
                    p = 'valid', bn = True)
    x = self.conv_t(x, f=self.z_dim, k=(self.n_bars - 1,1)
      , s=(1,1), a= 'relu', p = 'valid', bn = True)  

    output_layer = Reshape([self.n_bars, self.z_dim])(x) 

    return Model(input_layer, output_layer)

The Bar Generator

The bar generator converts a vector of length 4 * z_dim to a single bar for a single track—i.e., a tensor of shape [1, n_steps_per_bar, n_pitches, 1]. The input vector is created through the concatenation of the four relevant chord, style, melody, and groove vectors, each of length z_dim.

The bar generator is a neural network that uses convolutional transpose layers to expand the time and pitch dimensions. We will be creating one bar generator for every track, and weights are not shared. The Keras code to build a bar generator is given in Example 7-6.

def BarGenerator(self):

    input_layer = Input(shape=(self.z_dim * 4,), name='bar_generator_input') 

    x = Dense(1024)(input_layer)
    x = BatchNormalization(momentum = 0.9)(x)
    x = Activation('relu')(x)

    x = Reshape([2,1,512])(x) 
    x = self.conv_t(x, f=512, k=(2,1), s=(2,1), a= 'relu',  p = 'same', bn = True) 
    x = self.conv_t(x, f=256, k=(2,1), s=(2,1), a= 'relu', p = 'same', bn = True)
    x = self.conv_t(x, f=256, k=(2,1), s=(2,1), a= 'relu', p = 'same', bn = True)
    x = self.conv_t(x, f=256, k=(1,7), s=(1,7), a= 'relu', p = 'same',bn = True) 
    x = self.conv_t(x, f=1, k=(1,12), s=(1,12), a= 'tanh', p = 'same', bn = False) 

    output_layer = Reshape([1, self.n_steps_per_bar , self.n_pitches ,1])(x) 

    return Model(input_layer, output_layer)

The input to the bar generator is a vector of length 4 * z_dim.

After passing through a Dense layer, we reshape the tensor to prepare it for the convolutional transpose operations.

First we expand the tensor along the timestep axis…

…then along the pitch axis.

The final layer has a tanh activation applied, as we will be using a WGAN-GP (which requires tanh output activation) to train the network.

The tensor is reshaped to add two extra dimensions of size 1, to prepare it for concatenation with other bars and tracks.

Putting It All Together

Ultimately the MuseGAN has one single generator that incorporates all of the temporal networks and bar generators. This network takes the four input tensors and converts them into a multitrack, multibar score. The Keras code to build the overall generator is provided in Example 7-7.

chords_input = Input(shape=(self.z_dim,), name='chords_input') 
style_input = Input(shape=(self.z_dim,), name='style_input')
melody_input = Input(shape=(self.n_tracks, self.z_dim), name='melody_input')
groove_input = Input(shape=(self.n_tracks, self.z_dim), name='groove_input')

# CHORDS -> TEMPORAL NETWORK 
self.chords_tempNetwork = self.TemporalNetwork()
self.chords_tempNetwork.name = 'temporal_network'
chords_over_time = self.chords_tempNetwork(chords_input) # [n_bars, z_dim]

# MELODY -> TEMPORAL NETWORK 
melody_over_time = [None] * self.n_tracks # list of n_tracks [n_bars, z_dim] tensors
self.melody_tempNetwork = [None] * self.n_tracks
for track in range(self.n_tracks):
    self.melody_tempNetwork[track] = self.TemporalNetwork()
    melody_track = Lambda(lambda x: x[:,track,:])(melody_input)
    melody_over_time[track] = self.melody_tempNetwork[track](melody_track)

# CREATE BAR GENERATOR FOR EACH TRACK 
self.barGen = [None] * self.n_tracks
for track in range(self.n_tracks):
    self.barGen[track] = self.BarGenerator()

# CREATE OUTPUT FOR EVERY TRACK AND BAR 
bars_output = [None] * self.n_bars
for bar in range(self.n_bars):
    track_output = [None] * self.n_tracks

    c = Lambda(lambda x: x[:,bar,:]
	, name = 'chords_input_bar_' + str(bar))(chords_over_time)
    s = style_input

    for track in range(self.n_tracks):

        m = Lambda(lambda x: x[:,bar,:])(melody_over_time[track])
        g = Lambda(lambda x: x[:,track,:])(groove_input)

        z_input = Concatenate(axis = 1
	, name = 'total_input_bar_{}_track_{}'.format(bar, track)
	)([c,s,m,g])

        track_output[track] = self.barGen[track](z_input)

    bars_output[bar] = Concatenate(axis = -1)(track_output)

generator_output = Concatenate(axis = 1, name = 'concat_bars')(bars_output) 

self.generator = Model([chords_input, style_input, melody_input, groove_input]
	, generator_output) 

The inputs to the generator are defined.

Pass the chords input through the temporal network.

Pass the melody input through the temporal network.

Create an independent bar generator network for every track.

Loop over the tracks and bars, creating a generated bar for each combination.

Concatenate everything together to form a single output tensor.

The MuseGAN model takes four distinct noise tensors as input and outputs a generated multitrack, multibar score.