Using out-of-vocabulary tokens

One tool to use to handle these situations is an out-of-vocabulary (OOV) token. This can help your neural network to understand the context of the data containing previously unseen text. For example, given the previous small example corpus, suppose you want to process sentences like these:

  test_data = [
    'Today is a snowy day',
    'Will it be rainy tomorrow?'
]

Remember that you’re not adding this input to the corpus of existing text (which you can think of as your training data), but considering how a pretrained network might view this text. If you tokenize it with the words that you’ve already used and your existing tokenizer, like this:

  test_sequences = tokenizer.texts_to_sequences(test_data)
print(word_index)
print(test_sequences)

Your results will look like this:

  {'today': 1, 'is': 2, 'a': 3, 'sunny': 4, 'day': 5, 'rainy': 6, 'it': 7}
[[1, 2, 3, 5], [7, 6]]

So the new sentences, swapping back tokens for words, would be “today is a day” and “it rainy”.

As you can see, you’ve pretty much lost all context and meaning. An out-of-vocabulary token might help here, and you can specify it in the tokenizer. You do this by adding a parameter called oov_token, as shown here—you can assign it any string you like, but make sure it’s not one that appears elsewhere in your corpus:

  tokenizer = Tokenizer(num_words = 100, oov_token="<OOV>")
tokenizer.fit_on_texts(sentences)
word_index = tokenizer.word_index

sequences = tokenizer.texts_to_sequences(sentences)

test_sequences = tokenizer.texts_to_sequences(test_data)
print(word_index)
print(test_sequences)

You’ll see the output has improved a bit:

  {'<OOV>': 1, 'today': 2, 'is': 3, 'a': 4, 'sunny': 5, 'day': 6, 'rainy': 7, 
 'it': 8}

[[2, 3, 4, 1, 6], [1, 8, 1, 7, 1]]

Your tokens list has a new item, “<OOV>”, and your test sentences maintain their length. Reverse-encoding them will now give “today is a <OOV> day” and “<OOV> it <OOV> rainy <OOV>”.

The former is much closer to the original meaning. The latter, because most of its words aren’t in the corpus, still lacks a lot of context, but it’s a step in the right direction.

Understanding padding

When training neural networks you typically need all your data to be in the same shape. Recall from earlier chapters that when training with images, you reformatted the images to be the same width and height. With text you face the same issue—once you’ve tokenized your words and converted your sentences into sequences, they can all be different lengths. To get them to be the same size and shape, you can use padding.

To explore padding, let’s add another, much longer, sentence to the corpus:

  sentences = [
    'Today is a sunny day',
    'Today is a rainy day',
    'Is it sunny today?',
    'I really enjoyed walking in the snow today'
]

When you sequence that, you’ll see that your lists of numbers have different lengths:

  [
  [2, 3, 4, 5, 6], 
  [2, 3, 4, 7, 6], 
  [3, 8, 5, 2], 
  [9, 10, 11, 12, 13, 14, 15, 2]
]

(When you print the sequences they’ll all be on a single line, but I’ve broken them into separate lines here for clarity.)

If you want to make these the same length, you can use the pad_sequences API. First, you’ll need to import it:

  from tensorflow.keras.preprocessing.sequence import pad_sequences

Using the API is very straightforward. To convert your (unpadded) sequences into a padded set, you simply call pad_sequences like this:

  padded = pad_sequences(sequences)

print(padded)

You’ll get a nicely formatted set of sequences. They’ll also be on separate lines, like this:

  [[ 0  0  0  2  3  4  5  6]
 [ 0  0  0  2  3  4  7  6]
 [ 0  0  0  0  3  8  5  2]
 [ 9 10 11 12 13 14 15  2]]

The sequences get padded with 0, which isn’t a token in our word list. If you had wondered why the token list began at 1 when typically programmers count from 0, now you know!

You now have something that’s regularly shaped that you can use for training. But before going there, let’s explore this API a little, because it gives you many options that you can use to improve your data.

First, you might have noticed that in the case of the shorter sentences, to get them to be the same shape as the longest one, the requisite number of zeros were added at the beginning. This is called prepadding, and it’s the default behavior. You can change this using the padding parameter. For example, if you want your sequences to be padded with zeros at the end, you can use:

  padded = pad_sequences(sequences, padding='post')

The output from this will be:

  [[ 2  3  4  5  6  0  0  0]
 [ 2  3  4  7  6  0  0  0]
 [ 3  8  5  2  0  0  0  0]
 [ 9 10 11 12 13 14 15  2]]

You can see that now the words are at the beginning of the padded sequences, and the 0 characters are at the end.

The next default behavior you may have observed is that the sentences were all made to be the same length as the longest one. It’s a sensible default because it means you don’t lose any data. The tradeoff is you get a lot of padding. But what if you don’t want this, perhaps because you have one crazy long sentence that means you’d have too much padding in the padded sequences? To fix that you can use the maxlen parameter, specifying the desired maximum length, when calling pad_sequences, like this:

  padded = pad_sequences(sequences, padding='post', maxlen=6)

The output from this will be:

  [[ 2  3  4  5  6  0]
 [ 2  3  4  7  6  0]
 [ 3  8  5  2  0  0]
 [11 12 13 14 15  2]]

Now your padded sequences are all the same length, and there isn’t too much padding. You have lost some words from your longest sentence, though, and they’ve been truncated from the beginning. What if you don’t want to lose the words from the beginning and instead want them truncated from the end of the sentence? You can override the default behavior with the truncating parameter, as follows:

  padded = pad_sequences(sequences, padding='post', maxlen=6, truncating='post')

The result of this will show that the longest sentence is now truncated at the end instead of the beginning:

  [[ 2  3  4  5  6  0]
 [ 2  3  4  7  6  0]
 [ 3  8  5  2  0  0]
 [ 9 10 11 12 13 14]]

Using the IMDb subwords datasets

TFDS also contains a couple of preprocessed IMDb datasets using subwords. Here, instead of you needing to break up the sentences by word, they have already been split up into subwords for you. Using subwords is a happy medium between splitting the corpus into individual letters (relatively few tokens with low semantic meaning) and individual words (many tokens with high semantic meaning), and this approach can often be used very effectively to train a classifier for language. These datasets also include the encoders and decoders used to split and encode the corpus.

To access them, you can call tfds.load and pass it imdb_reviews/subwords8k or imdb_reviews/subwords32k like this:

  (train_data, test_data), info = tfds.load(
    'imdb_reviews/subwords8k', 
    split = (tfds.Split.TRAIN, tfds.Split.TEST),
    as_supervised=True,
    with_info=True
)

You can access the encoder on the info object like this. This will help you see the vocab_size:

  encoder = info.features['text'].encoder
print ('Vocabulary size: {}'.format(encoder.vocab_size))

This will output 8185 because the vocabulary in this instance is made up of 8,185 tokens. If you want to see the list of subwords, you can get it with the encoder.subwords property:

  print(encoder.subwords)

['the_', ', ', '. ', 'a_', 'and_', 'of_', 'to_', 's_', 'is_', 'br', 'in_', 'I_', 
 'that_',...]

Some things you might notice here are that stopwords, punctuation, and grammar are all in the corpus, as are HTML tags like <br>. Spaces are represented by underscores, so the first token is the word “the”.

Should you want to encode a string, you can do so with the encoder like this:

  sample_string = 'Today is a sunny day'

encoded_string = encoder.encode(sample_string)
print ('Encoded string is {}'.format(encoded_string))

The output of this will be a list of tokens:

  Encoded string is [6427, 4869, 9, 4, 2365, 1361, 606]

So, your five words are encoded into seven tokens. To see the tokens you can use the subwords property on the encoder, which returns an array. It’s zero-based, so whereas “Tod” in “Today” was encoded as 6427, it’s the 6,426th item in the array:

  print(encoder.subwords[6426])
Tod

If you want to decode, you can use the decode method of the encoder:

  encoded_string = encoder.encode(sample_string)

original_string = encoder.decode(encoded_string)
test_string = encoder.decode([6427, 4869, 9, 4, 2365, 1361, 606])

The latter lines will have an identical result because encoded_string, despite its name, is a list of tokens just like the one that is hardcoded on the next line.

Creating training and test subsets

Now that the text corpus has been read into a list of sentences, you’ll need to split it into training and test subsets for training a model. For example, if you want to use 28,000 sentences for training with the rest held back for testing, you can use code like this:

  training_size = 28000

training_sentences = sentences[0:training_size]
testing_sentences = sentences[training_size:]
training_labels = labels[0:training_size]
testing_labels = labels[training_size:]

Now that you have a training set, you need to create the word index from it. Here is the code to use the tokenizer to create a vocabulary with up to 20,000 words. We’ll set the maximum length of a sentence at 10 words, truncate longer ones by cutting off the end, pad shorter ones at the end, and use “<OOV>” as an out-of-vocabulary token:

  vocab_size = 20000
max_length = 10
trunc_type='post'
padding_type='post'
oov_tok = "<OOV>"

tokenizer = Tokenizer(num_words=vocab_size, oov_token=oov_tok)
tokenizer.fit_on_texts(training_sentences)

word_index = tokenizer.word_index

training_sequences = tokenizer.texts_to_sequences(training_sentences)

training_padded = pad_sequences(training_sequences, maxlen=max_length, 
                                padding=padding_type, 
                                truncating=trunc_type)

You can inspect the results by looking at training_sequences and training_padded. For example, here we print the first item in the training sequence, and you can see how it’s padded to a max length of 10:

  print(training_sequences[0])
print(training_padded[0])

[18, 3257, 47, 4770, 613, 508, 951, 423]
[  18 3257   47 4770  613  508  951  423    0    0]

You can also inspect the word index by printing it:

  {'<OOV>': 1, 'just': 2, 'not': 3, 'now': 4, 'day': 5, 'get': 6, 'no': 7, 
 'good': 8, 'like': 9, 'go': 10, 'dont': 11, ...}

There are many words here you might want to consider getting rid of as stopwords, like “like” and “dont”. It’s always useful to inspect the word index.

Reading JSON files

Python’s json library makes reading JSON files simple. Given that JSON uses name/value pairs, you can index the content based on the name. So, for example, for the Sarcasm dataset you can create a file handle to the JSON file, open it with the json library, and then have an iterable go through and read each field line by line and get the data item using the name of the field.

Here’s the code:

  import json
with open("/tmp/sarcasm.json", 'r') as f:
    datastore = json.load(f)
    for item in datastore:
        sentence = item['headline'].lower()
        label= item['is_sarcastic']
        link = item['article_link']

This makes it simple to create lists of sentences and labels as you’ve done throughout this chapter, and then tokenize the sentences. You can also do preprocessing on the fly as you read a sentence, removing stopwords, HTML tags, punctuation, and more. Here’s the complete code to create lists of sentences, labels, and URLs, while having the sentences cleaned of unwanted words and characters:

  with open("/tmp/sarcasm.json", 'r') as f:
    datastore = json.load(f)

sentences = [] 
labels = []
urls = []
for item in datastore:
    sentence = item['headline'].lower()
    sentence = sentence.replace(",", " , ")
    sentence = sentence.replace(".", " . ")
    sentence = sentence.replace("-", " - ")
    sentence = sentence.replace("/", " / ")
    soup = BeautifulSoup(sentence)
    sentence = soup.get_text()
    words = sentence.split()
    filtered_sentence = ""
    for word in words:
        word = word.translate(table)
        if word not in stopwords:
            filtered_sentence = filtered_sentence + word + " "
    sentences.append(filtered_sentence)
    labels.append(item['is_sarcastic'])
    urls.append(item['article_link'])

As before, these can be split into training and test sets. If you want to use 23,000 of the 26,000 items in the dataset for training, you can do the following:

  training_size = 23000

training_sentences = sentences[0:training_size]
testing_sentences = sentences[training_size:]
training_labels = labels[0:training_size]
testing_labels = labels[training_size:]

To tokenize the data and get it ready for training, you can follow the same approach as earlier. Here, we again specify a vocab size of 20,000 words, a maximum sequence length of 10 with truncation and padding at the end, and an OOV token of “<OOV>”:

  vocab_size = 20000
max_length = 10
trunc_type='post'
padding_type='post'
oov_tok = "<OOV>"

tokenizer = Tokenizer(num_words=vocab_size, oov_token=oov_tok)
tokenizer.fit_on_texts(training_sentences)

word_index = tokenizer.word_index

training_sequences = tokenizer.texts_to_sequences(training_sentences)
padded = pad_sequences(training_sequences, padding='post')
print(word_index)

The output will be the whole index, in order of word frequency:

  {'<OOV>': 1, 'new': 2, 'trump': 3, 'man': 4, 'not': 5, 'just': 6, 'will': 7,  
 'one': 8, 'year': 9, 'report': 10, 'area': 11, 'donald': 12, ... }

Hopefully the similar-looking code will help you see the pattern that you can follow when preparing text for neural networks to classify or generate. In the next chapter you’ll see how to build a classifier for text using embeddings, and in Chapter 7 you’ll take that a step further, exploring recurrent neural networks. Then, in Chapter 8, you’ll see how to further enhance the sequence data to create a neural network that can generate new text!