Tokenization

When we said “convert the text into a list of words,” we left out a lot of details. For instance, what do we do with punctuation? How do we deal with a word like “don’t”? Is it one word or two? What about long medical or chemical words? Should they be split into their separate pieces of meaning? How about hyphenated words? What about languages like German and Polish, which can create really long words from many, many pieces? What about languages like Japanese and Chinese that don’t use bases at all, and don’t really have a well-defined idea of word?

Because there is no one correct answer to these questions, there is no one approach to tokenization. There are three main approaches:

Word-based

Split a sentence on spaces, as well as applying language-specific rules to try to separate parts of meaning even when there are no spaces (such as turning “don’t” into “do n’t”). Generally, punctuation marks are also split into separate tokens.

Subword based

Split words into smaller parts, based on the most commonly occurring substrings. For instance, “occasion” might be tokenized as “o c ca sion”.

Character-based

Split a sentence into its individual characters.

We’ll look at word and subword tokenization here, and we’ll leave character-based tokenization for you to implement in the questionnaire at the end of this chapter.

Word Tokenization with fastai

Rather than providing its own tokenizers, fastai provides a consistent interface to a range of tokenizers in external libraries. Tokenization is an active field of research, and new and improved tokenizers are coming out all the time, so the defaults that fastai uses change too. However, the API and options shouldn’t change too much, since fastai tries to maintain a consistent API even as the underlying technology changes.

Let’s try it out with the IMDb dataset that we used in Chapter 1:

from fastai.text.all import *
path = untar_data(URLs.IMDB)

We’ll need to grab the text files in order to try out a tokenizer. Just as get_image_files (which we’ve used many times already), gets all the image files in a path, get_text_files gets all the text files in a path. We can also optionally pass folders to restrict the search to a particular list of subfolders:

files = get_text_files(path, folders = ['train', 'test', 'unsup'])

Here’s a review that we’ll tokenize (we’ll print just the start of it here to save space):

txt = files[0].open().read(); txt[:75]

'This movie, which I just discovered at the video store, has apparently sit '

As we write this book, the default English word tokenizer for fastai uses a library called spaCy. It has a sophisticated rules engine with special rules for URLs, individual special English words, and much more. Rather than directly using SpacyTokenizer, however, we’ll use WordTokenizer, since that will always point to fastai’s current default word tokenizer (which may not necessarily be spaCy, depending when you’re reading this).

Let’s try it out. We’ll use fastai’s coll_repr(collection,n) function to display the results. This displays the first n items of collection, along with the full size—it’s what L uses by default. Note that fastai’s tokenizers take a collection of documents to tokenize, so we have to wrap txt in a list:

spacy = WordTokenizer()
toks = first(spacy([txt]))
print(coll_repr(toks, 30))

(#201) ['This','movie',',','which','I','just','discovered','at','the','video','s
 > tore',',','has','apparently','sit','around','for','a','couple','of','years','
 > without','a','distributor','.','It',"'s",'easy','to','see'...]

As you see, spaCy has mainly just separated out the words and punctuation. But it does something else here too: it has split “it’s” into “it” and “’s”. That makes intuitive sense; these are separate words, really. Tokenization is a surprisingly subtle task, when you think about all the little details that have to be handled. Fortunately, spaCy handles these pretty well for us—for instance, here we see that “.” is separated when it terminates a sentence, but not in an acronym or number:

first(spacy(['The U.S. dollar $1 is $1.00.']))

(#9) ['The','U.S.','dollar','$','1','is','$','1.00','.']

fastai then adds some additional functionality to the tokenization process with the Tokenizer class:

tkn = Tokenizer(spacy)
print(coll_repr(tkn(txt), 31))

(#228) ['xxbos','xxmaj','this','movie',',','which','i','just','discovered','at',
 > 'the','video','store',',','has','apparently','sit','around','for','a','couple
 > ','of','years','without','a','distributor','.','xxmaj','it',"'s",'easy'...]

Notice that there are now some tokens that start with the characters “xx”, which is not a common word prefix in English. These are special tokens.

For example, the first item in the list, xxbos, is a special token that indicates the start of a new text (“BOS” is a standard NLP acronym that means “beginning of stream”). By recognizing this start token, the model will be able to learn it needs to “forget” what was said previously and focus on upcoming words.

These special tokens don’t come from spaCy directly. They are there because fastai adds them by default, by applying a number of rules when processing text. These rules are designed to make it easier for a model to recognize the important parts of a sentence. In a sense, we are translating the original English language sequence into a simplified tokenized language—a language that is designed to be easy for a model to learn.

For instance, the rules will replace a sequence of four exclamation points with a single exclamation point, followed by a special repeated character token and then the number four. In this way, the model’s embedding matrix can encode information about general concepts such as repeated punctuation rather than requiring a separate token for every number of repetitions of every punctuation mark. Similarly, a capitalized word will be replaced with a special capitalization token, followed by the lowercase version of the word. This way, the embedding matrix needs only the lowercase versions of the words, saving compute and memory resources, but can still learn the concept of capitalization.

Here are some of the main special tokens you’ll see:

xxbos

Indicates the beginning of a text (here, a review)

xxmaj

Indicates the next word begins with a capital (since we lowercased everything)

xxunk

Indicates the next word is unknown

To see the rules that were used, you can check the default rules:

defaults.text_proc_rules

[<function fastai.text.core.fix_html(x)>,
 <function fastai.text.core.replace_rep(t)>,
 <function fastai.text.core.replace_wrep(t)>,
 <function fastai.text.core.spec_add_spaces(t)>,
 <function fastai.text.core.rm_useless_spaces(t)>,
 <function fastai.text.core.replace_all_caps(t)>,
 <function fastai.text.core.replace_maj(t)>,
 <function fastai.text.core.lowercase(t, add_bos=True, add_eos=False)>]

As always, you can look at the source code for each of them in a notebook by typing the following:

??replace_rep

Here is a brief summary of what each does:

fix_html

Replaces special HTML characters with a readable version (IMDb reviews have quite a few of these)

replace_rep

Replaces any character repeated three times or more with a special token for repetition (xxrep), the number of times it’s repeated, then the character

replace_wrep

Replaces any word repeated three times or more with a special token for word repetition (xxwrep), the number of times it’s repeated, then the word

spec_add_spaces

Adds spaces around / and #

rm_useless_spaces

Removes all repetitions of the space character

replace_all_caps

Lowercases a word written in all caps and adds a special token for all caps (xxcap) in front of it

replace_maj

Lowercases a capitalized word and adds a special token for capitalized (xxmaj) in front of it

lowercase

Lowercases all text and adds a special token at the beginning (xxbos) and/or the end (xxeos)

Let’s take a look at a few of them in action:

coll_repr(tkn('©   Fast.ai www.fast.ai/INDEX'), 31)

"(#11) ['xxbos','©','xxmaj','fast.ai','xxrep','3','w','.fast.ai','/','xxup','ind
 > ex'...]"

Now let’s take a look at how subword tokenization would work.

Subword Tokenization

In addition to the word tokenization approach seen in the preceding section, another popular tokenization method is subword tokenization. Word tokenization relies on an assumption that spaces provide a useful separation of components of meaning in a sentence. However, this assumption is not always appropriate. For instance, consider this sentence: 我的名字是郝杰瑞 (“My name is Jeremy Howard” in Chinese). That’s not going to work very well with a word tokenizer, because there are no spaces in it! Languages like Chinese and Japanese don’t use spaces, and in fact they don’t even have a well-defined concept of a “word.” Other languages, like Turkish and Hungarian, can add many subwords together without spaces, creating very long words that include a lot of separate pieces of information.

To handle these cases, it’s generally best to use subword tokenization. This proceeds in two steps:

1. Analyze a corpus of documents to find the most commonly occurring groups of letters. These become the vocab.

2. Tokenize the corpus using this vocab of subword units.

Let’s look at an example. For our corpus, we’ll use the first 2,000 movie reviews:

txts = L(o.open().read() for o in files[:2000])

We instantiate our tokenizer, passing in the size of the vocab we want to create, and then we need to “train” it. That is, we need to have it read our documents and find the common sequences of characters to create the vocab. This is done with setup. As we’ll see shortly, setup is a special fastai method that is called automatically in our usual data processing pipelines. Since we’re doing everything manually at the moment, however, we have to call it ourselves. Here’s a function that does these steps for a given vocab size and shows an example output:

def subword(sz):
    sp = SubwordTokenizer(vocab_sz=sz)
    sp.setup(txts)
    return ' '.join(first(sp([txt]))[:40])

Let’s try it out:

subword(1000)

'▁This ▁movie , ▁which ▁I ▁just ▁dis c over ed ▁at ▁the ▁video ▁st or e , ▁has
 > ▁a p par ent ly ▁s it ▁around ▁for ▁a ▁couple ▁of ▁years ▁without ▁a ▁dis t
 > ri but or . ▁It'

When using fastai’s subword tokenizer, the special character ▁ represents a space character in the original text.

If we use a smaller vocab, each token will represent fewer characters, and it will take more tokens to represent a sentence:

subword(200)

'▁ T h i s ▁movie , ▁w h i ch ▁I ▁ j us t ▁ d i s c o ver ed ▁a t ▁the ▁ v id e
 > o ▁ st or e , ▁h a s'

On the other hand, if we use a larger vocab, most common English words will end up in the vocab themselves, and we will not need as many to represent a sentence:

subword(10000)

"▁This ▁movie , ▁which ▁I ▁just ▁discover ed ▁at ▁the ▁video ▁store , ▁has
 > ▁apparently ▁sit ▁around ▁for ▁a ▁couple ▁of ▁years ▁without ▁a ▁distributor
 > . ▁It ' s ▁easy ▁to ▁see ▁why . ▁The ▁story ▁of ▁two ▁friends ▁living"

Picking a subword vocab size represents a compromise: a larger vocab means fewer tokens per sentence, which means faster training, less memory, and less state for the model to remember; but on the downside, it means larger embedding matrices, which require more data to learn.

Overall, subword tokenization provides a way to easily scale between character tokenization (i.e., using a small subword vocab) and word tokenization (i.e., using a large subword vocab), and handles every human language without needing language-specific algorithms to be developed. It can even handle other “languages” such as genomic sequences or MIDI music notation! For this reason, in the last year its popularity has soared, and it seems likely to become the most common tokenization approach (it may well already be, by the time you read this!).

Once our texts have been split into tokens, we need to convert them to numbers. We’ll look at that next.

Numericalization with fastai

Numericalization is the process of mapping tokens to integers. The steps are basically identical to those necessary to create a Category variable, such as the dependent variable of digits in MNIST:

1. Make a list of all possible levels of that categorical variable (the vocab).

2. Replace each level with its index in the vocab.

Let’s take a look at this in action on the word-tokenized text we saw earlier:

toks = tkn(txt)
print(coll_repr(tkn(txt), 31))

(#228) ['xxbos','xxmaj','this','movie',',','which','i','just','discovered','at',
 > 'the','video','store',',','has','apparently','sit','around','for','a','couple
 > ','of','years','without','a','distributor','.','xxmaj','it',"'s",'easy'...]

Just as with SubwordTokenizer, we need to call setup on Numericalize; this is how we create the vocab. That means we’ll need our tokenized corpus first. Since tokenization takes a while, it’s done in parallel by fastai; but for this manual walk-through, we’ll use a small subset:

toks200 = txts[:200].map(tkn)
toks200[0]

(#228)
 > ['xxbos','xxmaj','this','movie',',','which','i','just','discovered','at'...]

We can pass this to setup to create our vocab:

num = Numericalize()
num.setup(toks200)
coll_repr(num.vocab,20)

"(#2000) ['xxunk','xxpad','xxbos','xxeos','xxfld','xxrep','xxwrep','xxup','xxmaj
 > ','the','.',',','a','and','of','to','is','in','i','it'...]"

Our special rules tokens appear first, and then every word appears once, in frequency order. The defaults to Numericalize are min_freq=3 and max_vocab=60000. max_vocab=60000 results in fastai replacing all words other than the most common 60,000 with a special unknown word token, xxunk. This is useful to avoid having an overly large embedding matrix, since that can slow down training and use up too much memory, and can also mean that there isn’t enough data to train useful representations for rare words. However, this last issue is better handled by setting min_freq; the default min_freq=3 means that any word appearing fewer than three times is replaced with xxunk.

fastai can also numericalize your dataset using a vocab that you provide, by passing a list of words as the vocab parameter.

Once we’ve created our Numericalize object, we can use it as if it were a function:

nums = num(toks)[:20]; nums

tensor([  2,   8,  21,  28,  11,  90,  18,  59,   0,  45,   9, 351, 499,  11,
 > 72, 533, 584, 146,  29,  12])

This time, our tokens have been converted to a tensor of integers that our model can receive. We can check that they map back to the original text:

' '.join(num.vocab[o] for o in nums)

'xxbos xxmaj this movie , which i just xxunk at the video store , has apparently
 > sit around for a'

Now that we have numbers, we need to put them in batches for our model.

Putting Our Texts into Batches for a Language Model

When dealing with images, we needed to resize them all to the same height and width before grouping them together in a mini-batch so they could stack together efficiently in a single tensor. Here it’s going to be a little different, because one cannot simply resize text to a desired length. Also, we want our language model to read text in order, so that it can efficiently predict what the next word is. This means each new batch should begin precisely where the previous one left off.

Suppose we have the following text:

In this chapter, we will go back over the example of classifying movie reviews we studied in chapter 1 and dig deeper under the surface. First we will look at the processing steps necessary to convert text into numbers and how to customize it. By doing this, we’ll have another example of the PreProcessor used in the data block API.

Then we will study how we build a language model and train it for a while.

The tokenization process will add special tokens and deal with punctuation to return this text:

xxbos xxmaj in this chapter , we will go back over the example of classifying movie reviews we studied in chapter 1 and dig deeper under the surface . xxmaj first we will look at the processing steps necessary to convert text into numbers and how to customize it . xxmaj by doing this , we ‘ll have another example of the preprocessor used in the data block xxup api . xxmaj then we will study how we build a language model and train it for a while .

We now have 90 tokens, separated by spaces. Let’s say we want a batch size of 6. We need to break this text into 6 contiguous parts of length 15:

xxbos

xxmaj

in

this

chapter

,

we

will

go

back

over

the

example

of

classifying

movie

reviews

we

studied

in

chapter

1

and

dig

deeper

under

the

surface

.

xxmaj

first

we

will

look

at

the

processing

steps

necessary

to

convert

text

into

numbers

and

how

to

customize

it

.

xxmaj

by

doing

this

,

we

‘ll

have

another

example

of

the

preprocessor

used

in

the

data

block

xxup

api

.

xxmaj

then

we

will

study

how

we

build

a

language

model

and

train

it

for

a

while

.

In a perfect world, we could then give this one batch to our model. But that approach doesn’t scale, because outside this toy example, it’s unlikely that a single batch containing all the tokens would fit in our GPU memory (here we have 90 tokens, but all the IMDb reviews together give several million).

So, we need to divide this array more finely into subarrays of a fixed sequence length. It is important to maintain order within and across these subarrays, because we will use a model that maintains a state so that it remembers what it read previously when predicting what comes next.

Going back to our previous example with 6 batches of length 15, if we chose a sequence length of 5, that would mean we first feed the following array:

Then, this one:

And finally:

Going back to our movie reviews dataset, the first step is to transform the individual texts into a stream by concatenating them together. As with images, it’s best to randomize the order of the inputs, so at the beginning of each epoch we will shuffle the entries to make a new stream (we shuffle the order of the documents, not the order of the words inside them, or the texts would not make sense anymore!).

We then cut this stream into a certain number of batches (which is our batch size). For instance, if the stream has 50,000 tokens and we set a batch size of 10, this will give us 10 mini-streams of 5,000 tokens. What is important is that we preserve the order of the tokens (so from 1 to 5,000 for the first mini-stream, then from 5,001 to 10,000…), because we want the model to read continuous rows of text (as in the preceding example). An xxbos token is added at the start of each text during preprocessing, so that the model knows when it reads the stream when a new entry is beginning.

So to recap, at every epoch we shuffle our collection of documents and concatenate them into a stream of tokens. We then cut that stream into a batch of fixed-size consecutive mini-streams. Our model will then read the mini-streams in order, and thanks to an inner state, it will produce the same activation, whatever sequence length we picked.

This is all done behind the scenes by the fastai library when we create an LMDataLoader. We do this by first applying our Numericalize object to the tokenized texts

nums200 = toks200.map(num)

and then passing that to LMDataLoader:

dl = LMDataLoader(nums200)

Let’s confirm that this gives the expected results, by grabbing the first batch

x,y = first(dl)
x.shape,y.shape

(torch.Size([64, 72]), torch.Size([64, 72]))

and then looking at the first row of the independent variable, which should be the start of the first text:

' '.join(num.vocab[o] for o in x[0][:20])

'xxbos xxmaj this movie , which i just xxunk at the video store , has apparently
 > sit around for a'

The dependent variable is the same thing offset by one token:

' '.join(num.vocab[o] for o in y[0][:20])

'xxmaj this movie , which i just xxunk at the video store , has apparently sit
 > around for a couple'

This concludes all the preprocessing steps we need to apply to our data. We are now ready to train our text classifier.

Language Model Using DataBlock

fastai handles tokenization and numericalization automatically when TextBlock is passed to DataBlock. All of the arguments that can be passed to Tokenizer and Numericalize can also be passed to TextBlock. In the next chapter, we’ll discuss the easiest ways to run each of these steps separately, to ease debugging, but you can always just debug by running them manually on a subset of your data as shown in the previous sections. And don’t forget about DataBlock’s handy summary method, which is very useful for debugging data issues.

Here’s how we use TextBlock to create a language model, using fastai’s defaults:

get_imdb = partial(get_text_files, folders=['train', 'test', 'unsup'])

dls_lm = DataBlock(
    blocks=TextBlock.from_folder(path, is_lm=True),
    get_items=get_imdb, splitter=RandomSplitter(0.1)
).dataloaders(path, path=path, bs=128, seq_len=80)

One thing that’s different from previous types we’ve used in DataBlock is that we’re not just using the class directly (i.e., TextBlock(...), but instead are calling a class method. A class method is a Python method that, as the name suggests, belongs to a class rather than an object. (Be sure to search online for more information about class methods if you’re not familiar with them, since they’re commonly used in many Python libraries and applications; we’ve used them a few times previously in the book, but haven’t called attention to them.) The reason that TextBlock is special is that setting up the numericalizer’s vocab can take a long time (we have to read and tokenize every document to get the vocab).

To be as efficient as possible, fastai performs a few optimizations:

• It saves the tokenized documents in a temporary folder, so it doesn’t have to tokenize them more than once.

• It runs multiple tokenization processes in parallel, to take advantage of your computer’s CPUs.

We need to tell TextBlock how to access the texts, so that it can do this initial preprocessing—that’s what from_folder does.

show_batch then works in the usual way:

dls_lm.show_batch(max_n=2)

Now that our data is ready, we can fine-tune the pretrained language model.

Fine-Tuning the Language Model

To convert the integer word indices into activations that we can use for our neural network, we will use embeddings, just as we did for collaborative filtering and tabular modeling. Then we’ll feed those embeddings into a recurrent neural network (RNN), using an architecture called AWD-LSTM (we will show you how to write such a model from scratch in Chapter 12). As we discussed earlier, the embeddings in the pretrained model are merged with random embeddings added for words that weren’t in the pretraining vocabulary. This is handled automatically inside language_model_learner:

learn = language_model_learner(
    dls_lm, AWD_LSTM, drop_mult=0.3,
    metrics=[accuracy, Perplexity()]).to_fp16()

The loss function used by default is cross-entropy loss, since we essentially have a classification problem (the different categories being the words in our vocab). The perplexity metric used here is often used in NLP for language models: it is the exponential of the loss (i.e., torch.exp(cross_entropy)). We also include the accuracy metric to see how many times our model is right when trying to predict the next word, since cross entropy (as we’ve seen) is both hard to interpret and tells us more about the model’s confidence than its accuracy.

Let’s go back to the process diagram from the beginning of this chapter. The first arrow has been completed for us and made available as a pretrained model in fastai, and we’ve just built the DataLoaders and Learner for the second stage. Now we’re ready to fine-tune our language model!

It takes quite a while to train each epoch, so we’ll be saving the intermediate model results during the training process. Since fine_tune doesn’t do that for us, we’ll use fit_one_cycle. Just like cnn_learner, language_model_learner automatically calls freeze when using a pretrained model (which is the default), so this will train only the embeddings (the only part of the model that contains randomly initialized weights—i.e., embeddings for words that are in our IMDb vocab, but aren’t in the pretrained model vocab):

learn.fit_one_cycle(1, 2e-2)

This model takes a while to train, so it’s a good opportunity to talk about saving intermediary results.

Saving and Loading Models

You can easily save the state of your model like so:

learn.save('1epoch')

This will create a file in learn.path/models/ named 1epoch.pth. If you want to load your model in another machine after creating your Learner the same way, or resume training later, you can load the content of this file as follows:

learn = learn.load('1epoch')

Once the initial training has completed, we can continue fine-tuning the model after unfreezing:

learn.unfreeze()
learn.fit_one_cycle(10, 2e-3)

Once this is done, we save all of our model except the final layer that converts activations to probabilities of picking each token in our vocabulary. The model not including the final layer is called the encoder. We can save it with save_encoder:

learn.save_encoder('finetuned')

This completes the second stage of the text classification process: fine-tuning the language model. We can now use it to fine-tune a classifier using the IMDb sentiment labels. Before we move on to fine-tuning the classifier, however, let’s quickly try something different: using our model to generate random reviews.

Text Generation

Because our model is trained to guess the next word of the sentence, we can use it to write new reviews:

TEXT = "I liked this movie because"
N_WORDS = 40
N_SENTENCES = 2
preds = [learn.predict(TEXT, N_WORDS, temperature=0.75)
         for _ in range(N_SENTENCES)]

print("
".join(preds))

i liked this movie because of its story and characters . The story line was very
 > strong , very good for a sci - fi film . The main character , Alucard , was
 > very well developed and brought the whole story
i liked this movie because i like the idea of the premise of the movie , the (
 > very ) convenient virus ( which , when you have to kill a few people , the "
 > evil " machine has to be used to protect

As you can see, we add some randomness (we pick a random word based on the probabilities returned by the model) so we don’t get exactly the same review twice. Our model doesn’t have any programmed knowledge of the structure of a sentence or grammar rules, yet it has clearly learned a lot about English sentences: we can see it capitalizes properly (I is transformed to i because our rules require two characters or more to consider a word as capitalized, so it’s normal to see it lowercased) and is using consistent tense. The general review makes sense at first glance, and it’s only if you read carefully that you can notice something is a bit off. Not bad for a model trained in a couple of hours!

But our end goal wasn’t to train a model to generate reviews, but to classify them…so let’s use this model to do just that.

Creating the Classifier DataLoaders

We’re now moving from language model fine-tuning to classifier fine-tuning. To re-cap, a language model predicts the next word of a document, so it doesn’t need any external labels. A classifier, however, predicts an external label—in the case of IMDb, it’s the sentiment of a document.

This means that the structure of our DataBlock for NLP classification will look very familiar. It’s nearly the same as we’ve seen for the many image classification datasets we’ve worked with:

dls_clas = DataBlock(
    blocks=(TextBlock.from_folder(path, vocab=dls_lm.vocab),CategoryBlock),
    get_y = parent_label,
    get_items=partial(get_text_files, folders=['train', 'test']),
    splitter=GrandparentSplitter(valid_name='test')
).dataloaders(path, path=path, bs=128, seq_len=72)

Just as with image classification, show_batch shows the dependent variable (sentiment, in this case) with each independent variable (movie review text):

dls_clas.show_batch(max_n=3)

Looking at the DataBlock definition, every piece is familiar from previous data blocks we’ve built, with two important exceptions:

• TextBlock.from_folder no longer has the is_lm=True parameter.

• We pass the vocab we created for the language model fine-tuning.

The reason that we pass the vocab of the language model is to make sure we use the same correspondence of token to index. Otherwise, the embeddings we learned in our fine-tuned language model won’t make any sense to this model, and the fine-tuning step won’t be of any use.

By passing is_lm=False (or not passing is_lm at all, since it defaults to False), we tell TextBlock that we have regular labeled data, rather than using the next tokens as labels. There is one challenge we have to deal with, however, which has to do with collating multiple documents into a mini-batch. Let’s see with an example, by trying to create a mini-batch containing the first 10 documents. First we’ll numericalize them:

nums_samp = toks200[:10].map(num)

Let’s now look at how many tokens each of these 10 movie reviews has:

nums_samp.map(len)

(#10) [228,238,121,290,196,194,533,124,581,155]

Remember, PyTorch DataLoaders need to collate all the items in a batch into a single tensor, and a single tensor has a fixed shape (i.e., it has a particular length on every axis, and all items must be consistent). This should sound familiar: we had the same issue with images. In that case, we used cropping, padding, and/or squishing to make all the inputs the same size. Cropping might not be a good idea for documents, because it seems likely we’d remove some key information (having said that, the same issue is true for images, and we use cropping there; data augmentation hasn’t been well explored for NLP yet, so perhaps there are actually opportunities to use cropping in NLP too!). You can’t really “squish” a document. So that leaves padding!

We will expand the shortest texts to make them all the same size. To do this, we use a special padding token that will be ignored by our model. Additionally, to avoid memory issues and improve performance, we will batch together texts that are roughly the same lengths (with some shuffling for the training set). We do this by (approximately, for the training set) sorting the documents by length prior to each epoch. The result is that the documents collated into a single batch will tend of be of similar lengths. We won’t pad every batch to the same size, but will instead use the size of the largest document in each batch as the target size.

The sorting and padding are automatically done by the data block API for us when using a TextBlock with is_lm=False. (We don’t have this same issue for language model data, since we concatenate all the documents together first and then split them into equally sized sections.)

We can now create a model to classify our texts:

learn = text_classifier_learner(dls_clas, AWD_LSTM, drop_mult=0.5,
                                metrics=accuracy).to_fp16()

The final step prior to training the classifier is to load the encoder from our fine-tuned language model. We use load_encoder instead of load because we have only pretrained weights available for the encoder; load by default raises an exception if an incomplete model is loaded:

learn = learn.load_encoder('finetuned')

Fine-Tuning the Classifier

The last step is to train with discriminative learning rates and gradual unfreezing. In computer vision, we often unfreeze the model all at once, but for NLP classifiers, we find that unfreezing a few layers at a time makes a real difference:

learn.fit_one_cycle(1, 2e-2)

In just one epoch, we get the same result as our training in Chapter 1—not too bad! We can pass -2 to freeze_to to freeze all except the last two parameter groups:

learn.freeze_to(-2)
learn.fit_one_cycle(1, slice(1e-2/(2.6**4),1e-2))

Then we can unfreeze a bit more and continue training:

learn.freeze_to(-3)
learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3))

And finally, the whole model!

learn.unfreeze()
learn.fit_one_cycle(2, slice(1e-3/(2.6**4),1e-3))

We reached 94.3% accuracy, which was state-of-the-art performance just three years ago. By training another model on all the texts read backward and averaging the predictions of those two models, we can even get to 95.1% accuracy, which was the state of the art introduced by the ULMFiT paper. It was beaten only a few months ago, by fine-tuning a much bigger model and using expensive data augmentation techniques (translating sentences in another language and back, using another model for translation).

Using a pretrained model let us build a fine-tuned language model that is pretty powerful, to either generate fake reviews or help classify them. This is exciting stuff, but it’s good to remember that this technology can also be used for malign purposes.

Further Research

1. See what you can learn about language models and disinformation. What are the best language models today? Take a look at some of their outputs. Do you find them convincing? How could a bad actor best use such a model to create conflict and uncertainty?

2. Given the limitation that models are unlikely to be able to consistently recognize machine-generated texts, what other approaches may be needed to handle large-scale disinformation campaigns that leverage deep learning?