A simple classifier without this pipeline

When we talk about the above pipeline, we are referring to a supervised machine learning scenario. However, it is possible to build a simple classifier without machine learning, and without this pipeline. Consider the following problem statement: we are given a corpus of tweets where each tweet is labeled with its corresponding sentiment - negative or positive. For example, a tweet saying: “The new James Bond movie is great!” is clearly expressing a positive sentiment, whereas a tweet saying: “I would never visit this restaurant again, horrible place!!” has a negative sentiment. We wish to build a classification system that will predict the sentiment of an unseen tweet using only the text of the tweet. A simple solution could be to create lists of positive and negative words in English, compare the usage of positive vs negative words in the tweet and make a prediction based on this information. Further enhancements could involve creating more sophisticated dictionaries with degrees of positive/negative/neutral sentiment of words or formulating specific heuristics (e.g., usage of certain smileys indicate positive sentiment) and using them to make predictions.

Clearly, this does not involve any “learning” of text classification i.e., it is based on a set of heuristics or rules and custom built resources such as dictionaries. While this approach may seem too simple to handle many real world scenarios, it may enable you to quickly deploy a Minimum Viable Product (MVP), which is nothing but an initial working approach which can be tested with customers and obtain feedback for further development. Most importantly, this simple model can lead to better understanding of the problem and give you a simple baseline for your evaluation metric and speed. From our experience, it is always good to start with such simpler models when you tackle a new NLP problem with data. However, eventually, we will need learning methods which can infer more insights from large collections of text data.

Using existing text classification APIs

Another scenario where you may not have to “learn” a classifier or follow this pipeline is when your task is more generic in nature, such as, identifying a general category of a text (e.g., whether it is about technology or music). In such cases, existing APIs such as Google Cloud Natural Language³ can provide you with off the shelf content classification models which can identify close to 700 different categories of text. Another such popular classification task is sentiment analysis. All major service providers (e.g., Google, Microsoft) serve sentiment analysis APIs, with varying payment structures. If you are tasked with building a sentiment classifier, you may not have to build your own system anymore, if an existing API addresses your business need. However, many classification tasks could be specific to your organization’s business needs. For the rest of this chapter, we will address that task of building our own classifier, by considering the pipeline described earlier in this section.

Naive Bayes Classifier

Naive Bayes is a probabilistic classifier which uses Bayes’ theorem to classify texts, based on the evidence seen in training data. It estimates the conditional probability of each feature of a given text for each class based on the occurrence of that feature in that class, and multiplies these probabilities of all features of a given text to get final probability of classification for each class. Finally, it chooses the class with maximum probability. A detailed step by step explanation of the classifier is beyond the scope of this book. However, for an interested reader, Naive Bayes, specifically in the context of text classification, is explained in detail in Chapter 4 of Jurafsky & Martin (2018)⁷. Although simple, naive bayes is commonly used as a baseline algorithm in classification experiments.

Let us walk through the pipeline described earlier for our dataset, using Naive Bayes implementation in sklearn. Since we already have access to the dataset, our next step is to split the data into train and test data, as shown in the code snippet below:

          #Step 1: train-test split
          X = our_data.text #the column text contains textual data to extract features from
          y = our_data.relevance #this is the column we are learning to predict.
          # split X and y into training and testing sets. By default, it splits 75% training and 25% test
          #random_state=1 for reproducibility
          X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)
          

The next step now is to pre-process the texts and convert them into feature vectors. While there are many different ways to do the pre-processing, let us say we want to do the following: lowercasing, removal of punctuation, digits and any custom strings, and stopwords. The below code snippet shows this pre-processing and converting the train/test data into feature vectors using CountVectorizer in sklearn, which is the implementation of bag of words approach we discussed in Chapter 3.

          #Step 2-3: Preprocess and Vectorize train and test data
          vect = CountVectorizer(preprocessor=clean) 
          #clean is a function we defined for pre-processing, seen in the notebook.
          X_train_dtm = vect.fit_transform(X_train)
          X_test_dtm = vect.transform(X_test)
          print(X_train_dtm.shape, X_test_dtm.shape)
          

Once you run this in the notebook, you will see that we ended up having a feature vector with over 45K features! We now have the data in a format we want i.e., feature vectors. So, the next step is to train and evaluate a classifier. The code snippet below shows how to do the training and evaluation of a naive bayes classifier with the features we extracted above.

          nb = MultinomialNB() #instantiate a Multinomial Naive Bayes classifier
          nb.fit(X_train_dtm, y_train)#train the mode 
          y_pred_class = nb.predict(X_test_dtm)#make class predictions for test data
          

Figure 2-5. Confusion Matrix for Naive Bayes classifier

As we can see here, the classifier is doing fairly well with identifying the non-relevant articles correctly, only making errors 14% of the time. However, it does not perform well in comparison on the second category i.e., relevance. The category is identified correctly only 42% of the time. Since the dataset is already given, and we cannot change it or collect additional data, we can think of a few possible reasons for this performance and ways to improve this classifier. These are summarized in Table 1.

Table 2-1. Reasons for poor classifier performance

Reason 1	Since we extracted all possible features, we ended up in a large, sparse feature vector, where most features are too rare and end up being noise.
Reason 2	There are very few examples of relevant articles (~20%) compared to the non-relevant articles (~80%) in the dataset. This makes the learning process skewed towards the non-relevant articles category as there are very few examples of “relevant” articles.
Reason 3	Perhaps we need a better learning algorithm
Reason 4	Perhaps we need a better pre-processing and feature extraction mechanism addressed in [Sec. 4.5]
Reason 5	Perhaps we should look for tuning the classifier’s parameters and hyper-parameters [partially addressed in Sec 4.6]

Let us see how to improve our classification performance by addressing some of these possible reasons. One way to approach Reason 1 above is to reduce noise in the feature vectors. The approach in [Link to Come].5 had close to 40,000 features (refer to the jupyter notebook for details). Let us see what happens if we restrict this to 5000 and re-run the training and evaluation process. This requires us to change the CountVectorizer instantiation in the process as shown in the code snippet below and repeating all the steps.

          vect = CountVectorizer(preprocessor=clean, max_features=5000)
          

Figure 2-6. Improved classification performance with Naive Bayes and feature selection

Now, clearly, while the average performance seems lower than before, the identification of relevant articles correctly increased by over 20%. At that point, one may wonder whether this is what we want. The answer to that question depends on the problem we are trying to solve. If we care about doing reasonably well with non-relevant article identification, and doing as best as possible with relevant article identification, or do equally on both, we could conclude that reducing the feature vector size was useful for this data set, with Naive Bayes classifier.

Reason 2 in our list was the problem of a skew towards majority class. There are several ways to address this. Two typical approaches are oversampling the instances belonging to minority classes or undersampling the majority class to create a balanced dataset. Imbalanced-Learn⁸ is a python library that incorporates some of the sampling methods to address this issue. While we will not delve in to the details of this library here, classifiers also have a built in mechanism to address such imbalanced datasets. We will see how to use that in the next subsection, by taking another classifier - Logistic Regression.

Logistic Regression

When we described Naive Bayes classifier, we mentioned that it learns the probability of a text for each class, and chooses the one with maximum probability. Such a classifier is called a “generative classifier”. In contrast, there are “discriminative classifier” which aims to learn the probability distribution over all classes. Logistic regression is an example of such a classifier, and is commonly used in text classification, as a baseline in research and as an MVP in real-world industry scenarios.

Unlike naive bayes, which estimates probabilities based on feature occurrence in classes, logistic regression “learns” the weights for individual features based on how important they are to make a classification decision. The goal of logistic regression is to learn a linear separator between classes in the training data with the aim of maximizing the probability of the data. This “learning” of feature weights and probability distribution over all classes is done through a function called “logistic” function, and hence the name, logistic regression. For a detailed mathematical description of logistic regression, refer to Chapter 5 in Jurafsky & Martin (2018).

Let us take the 5000 dimensional feature vector from the last step of the naive bayes example and train a logistic regression classifier instead of naive bayes. The code snippet below shows how to use Logisitic Regression for this task.

          from sklearn.linear_model import LogisticRegression 
          logreg = LogisticRegression(class_weight="balanced")
          logreg.fit(X_train_dtm, y_train) 
          y_pred_class = logreg.predict(X_test_dtm)
          print("Accuracy: ", metrics.accuracy_score(y_test, y_pred_class))
          

This results in a classifier with an accuracy of 73.7%. [Link to Come].7 shows the confusion matrix with this approach.

Figure 2-7. Classification performance with Logistic Regression

Our Logistic Regression classifier instantiation has an argument: class_weight, which is given a value “balanced”. This tells the classifier to boost the weights for classes in the inverse proportion to the number of samples for that class i.e., we expect to see better performance for the less represented classes. You can experiment with this code by removing that argument and re-training the classifier, to witness a fall (by approximately 5%) in the bottom right cell in the confusion matrix. However, Logistic Regression clearly seems to perform worse than Naive Bayes for this dataset.

Reason 3 in our list was: “perhaps we need a better learning algorithm”. This will give raise to the question - “What is a better learning algorithm?”. A general rule of thumb in working with machine learning approaches is: there is no algorithm that learns well on all datasets. A commonly followed approach is to experiment with multiple algorithms and compare them. Let us see if this idea helps us, by replacing logistic regression with a well-known classification algorithm that was shown to be useful for several text classification tasks, called the “Support Vector Machine.”

Support Vector Machine (SVM)

We described logistic regression as a discriminative classifier that learns the weights for individual features, and predicts a probability distribution over the classes. Support Vector Machine, first invented in early 1960s, is a discriminative classifier like logistic regression. However, unlike logistic regression, it does not aim to maximize the dataset’s probability. Instead, it aims to look for an optimal hyperplane in a higher dimensional space, which can separate the classes in the data by a maximum margin possible. Further, SVMs are capable of learning even non-linear separations between classes, unlike logistic regression. However, they may also take longer to train.

SVMs come in different variants in sklearn. Let us see the use of one of them by keeping everything else the same, and altering maximum features to 1000 instead of 5000 in the previous example. The code snippet below shows how to do this and [Link to Come].8 shows the resultant confusion matrix.

          from sklearn.svm import LinearSVC
          vect = CountVectorizer(preprocessor=clean, max_features=1000) #Step-1
          X_train_dtm = vect.fit_transform(X_train)#combined step 2 and 3
          X_test_dtm = vect.transform(X_test)
          classifier = LinearSVC(class_weight='balanced') #notice the “balanced” option
          classifier.fit(X_train_dtm, y_train) #fit the model with training data
          y_pred_class = classifier.predict(X_test_dtm)
          print("Accuracy: ", metrics.accuracy_score(y_test, y_pred_class))
          

Figure 2-8. Classification with SVM

When compared to logistic regression, SVMs seem to have done better with “relevant” articles category, although, among this small set of experiments we did, naive bayes, with the smaller set of features, seems to be the best classifier for this dataset.

All the examples in this section showed how changes in different steps affected the classification performance, and how to interpret the results. Clearly, we excluded many other possibilities such as: exploring different other classifiers, changing the different parameters of all classifiers, coming up with better pre-processing methods etc. We leave them as further exercises for the reader, using the notebook as a playground. A real world text classification project involves exploring multiple options, starting with the simplest approach in terms of modeling, deployment and scale, and gradually increasing the complexity. Our eventual goal is to build the classifier that best meets our business need given all the other constraints.

Let us now consider a part of Reason 4 in the Table 1: better feature representation. So far, in this chapter, we have used bag-of-words features. Let us see how we can use other feature representations we saw in Chapter 3 for text classification.

Using Neural Embeddings in Text Classification

In the later half of Chapter 3, we discussed advanced feature engineering techniques using neural networks, such as word-embeddings, character-embeddings, and document embeddings. The advantage of using embedding based features is that they create a dense, low-dimensional feature representation instead of the sparse, high-dimensional structure of bag of words/TFIDF and other such features. There are different ways of designing and using features based on such neural embeddings. In this section, let us see some ways of using such embedding representations for text classification.

Word Embeddings

Words and ngrams have been primarily used as features in text classification for a long time. Different ways of vectorizing words have been proposed, and we used one such representation in the past section, using CountVectorizer. In the past 5, 6 years, neural network based architectures became popular for “learning” word representations, which are known as “word embeddings”. We briefly surveyed some of the intuitions behind this in Chapter 3. Let us now take a look at how to use word embeddings as features for text classification. We will use the sentiment labelled sentences dataset from the UCI repository, consisting of dataset consists of 1500 positive, and 1500 negative sentiment sentences from Amazon, Yelp, and IMDB. All the steps are detailed in the notebook (Notebook_2.ipynb). Let us walk through the important steps, and where this approach differs from the previous section’s procedures.

Loading and pre-processing the text data remains a common step. However, instead of vectorizing the texts using bag of words based features, we will now rely on neural embedding models. As mentioned earlier, we will use a pre-trained embedding model. Word2Vec is a popular algorithm we briefly discussed in Chapter 3, for training word embedding models. There are several downloadable pre-trained word2vec models trained on large corpora. Let us use the one from Google. The following code snippet shows how to load this model into python using gensim.

          data_path= "/your/folder/path"
          path_to_model = os.path.join(data_path,'GoogleNews-vectors-negative300.bin')
          training_data_path = os.path.join(data_path, "sentiment_sentences.txt")
          #Load W2V model. This will take some time.
          w2v_model = KeyedVectors.load_word2vec_format(path_to_model, binary=True)
          print('done loading Word2Vec')
          

This is a large model which can be seen as a dictionary where the keys are words in the vocabulary, and the values are their learnt embedding representations. How do we use this model to extract features? As we discussed in Chapter 3, there are multiple ways of doing this. A simple approach is just to average the embeddings for individual words in text. The code snippet below shows a simple function to do this.

          # Creating a feature vector by averaging all embeddings for all sentences
          def embedding_feats(list_of_lists):
           DIMENSION = 300
           zero_vector = np.zeros(DIMENSION)
           feats = []
           for tokens in list_of_lists:
           feat_for_this = np.zeros(DIMENSION)
           count_for_this = 0
           for token in tokens:
           if token in w2v_model:
           feat_for_this += w2v_model[token]
           count_for_this +=1
           feats.append(feat_for_this/count_for_this) 
           return feats
          train_vectors = embedding_feats(texts_processed)
          print(len(train_vectors))
          

Once the feature extraction is done, the final step is similar to what we did in the previous section i.e., use this feature set and train a classifier. We leave that as an exercise to you (you can refer to the notebook for the code). When trained with a Logistic Regression classifier, these features gave a classification accuracy of 81% on our dataset (see the notebook for more details). Considering that we just used an existing word embeddings model, and followed only basic pre-processing steps, this is a great model to have as a baseline! We saw in Chapter 3 that there are other pre-trained embedding approaches, which can all be experimented with for this approach. Gensim, which we used in this example, also supports training our own word embeddings if necessary. If we are working on a custom domain, whose vocabulary is remarkably different from that of the pre-trained news embeddings we used here, it would make sense to train our own embeddings to extract features. An important factor to consider when deploying models with embedding based feature extraction approaches is that the learnt or pre-trained embedding models have to be stored and loaded into memory while using these approaches. If the model itself is bulky (e.g., the pre-trained model we used is 3.6 GB), we need to factor this into our deployment needs.

Subword Embeddings and fastText

Word embeddings, as the name indicates, are about word representations. Even off the shelf embeddings seem to work well on a classification task, as we saw earlier. However, if a word in your dataset was not seen in the pre-trained model’s vocabulary, how will we get a representation for this word? In our previous example, we just ignored such words from feature extraction. Is there a better way?

We discussed fastText embeddings in Chapter 3. They are based on the idea of enriching word embeddings with sub-word level information. Thus, the embedding representation for each word is represented as a sum of the representations of individual character n-grams. While this may seem like a longer process compared to just estimating word level embeddings, this has two advantages:

• This approach can handle words that did not appear in training data.

• The implementation facilitates extremely fast learning on even very large corpora.

While fastText is a general purpose library to learn the embeddings, it also supports off the shelf text classification by providing end-to-end classifier training/testing. i.e., we don’t have to handle feature extraction separately.

The remaining part of this subsection talks about using fastText for text classification We will work with DBPedia dataset⁹. It is a balanced dataset consisting of 14 classes, with 40,000 training and 5000 testing examples per class. Thus, the total size of the dataset is 560,000 training and 70,000 testing data points. The step by step process is detailed in the associated Jupyter notebook (Notebook_3.ipynb). Let us get started!

The training and test sets are provided as csv files. So, the first step involves reading these files into your python environment, and cleaning the text to remove extraneous characters, similar to what we did in the pre-processing steps for the other classifier examples we saw so far. Once this is done, the process to use fastText is quite simple. The code snippet shows a simple fastText model.

          ## Using fastText for feature extraction and training
          from fasttext import supervised
          """fastText expects and training file (csv), a model name as input arguments.
          label_prefix refers to the prefix before label string in the dataset.
          default is __label__. In our dataset, it is __class__.
          There are several other parameters which can be seen in:
          https://pypi.org/project/fasttext/
          """
          model = supervised(train_file, 'temp', label_prefix="__class__")
          results = model.test(test_file)
          print(results.nexamples, results.precision, results.recall)
          

If you run this code in the notebook, you will notice that despite the fact that this is a huge dataset, and we gave the classifier raw text and not the feature vector, the training took only a few seconds, and we got an close to 98% precision and recall! As an exercise, try to build a classifier using the same dataset, but with either BOW or word embedding features, and using, say, logistic regression. Notice how long will it take for the individual steps of feature extraction and classification learning!

When you have a large dataset, and when learning seems infeasible with the approaches described so far, fastText is definitely a good option to use to set up a strong working baseline! However, there is one concern to keep in mind when using fastText, as it was the case with word2vec embeddings. It uses pretrained character ngram embeddings. Thus, when you save the trained model, it carries the entire character ngram embeddings dictionary with it. This results in a bulky model. For example, the model stored with the name “temp” in the above code sample has the size of close to 450 MB. However, fastText implementation also comes with options to reduce the memory footprint of its classification models with minimal reduction in classification performance. They do this by doing vocabulary pruning and using compression algorithms. Exploring these possibilities could be a good option in cases where large model sizes are a constraint.

We hope this discussion gives a good overview of the usefulness of fastText for text classification. What we showed here is a default classification model without any tuning of the hyper-parameters. fastText’s documentation contains more information on the different options to tune your classifier, and to train custom embedding representations for a dataset you want. However, both the embedding representations we saw so far learn a representation of words and characters, and collect them together to form a text representation. Let us see how to directly learn the representation for a document, using the doc2vec approach we discussed earlier in Chapter 3.

Document Embeddings

In doc2vec embedding scheme, we learn a direct representation for the entire document rather than each word. Just as we used word and character embeddings as features for performing text classification, we can also use doc2vec as a feature representation mechanism. Since there are no existing pre-trained models that work with the latest version doc2vec¹⁰, let us see how to build our own doc2vec model and use it for text classification.

We will use a dataset called “sentiment analysis: emotion in text” from figure-eight.com, which contains 40,000 tweets labeled with 13 labels signifying different emotions. Let us take the three most frequent labels in this dataset (neutral, worry, happiness) and build a text classifier for classifying new tweets into one of these three classes! The notebook for this subsection (Notebook_4.ipynb) walks you through the steps involved in using doc2vec for text classification and the dataset is also provided with the notebook.

After loading the dataset and taking a subset of the three most frequent labels, an important step to consider here is pre-processing of the data. What is different here, compared to previous examples? Why can’t we just follow the same procedure as before? There are a few things that are different about tweets, compared to news articles or other such text, as we also briefly discussed in Chapter 2 when we talked about text pre-processing. Firstly, they are very short. Second - our traditional tokenizers may not work well with tweets, splitting smileys, hashtags, twitter handles etc into multiple tokens. Such specialized needs prompted a lot of research in the recent past into NLP for Twitter, which resulted in several pre-processing options for tweets. One such solution is a TweetTokenizer, implemented in NLTK¹¹ library in Python. Let us use that instead of what we did in the past few examples. The code snippet below shows how to use it.

          tweeter = TweetTokenizer(strip_handles=True,preserve_case=False)
          mystopwords = set(stopwords.words("english"))
          #Function to pre-process and tokenize tweets
          def preprocess_corpus(texts):
           def remove_stops_digits(tokens):
           #Nested function that removes stopwords and digits from a list of tokens
           return [token for token in tokens if token not in mystopwords and not token.isdigit()]
           return [remove_stops_digits(tweeter.tokenize(content)) for content in texts]
          mydata = preprocess_corpus(df_subset['content'])
          mycats = df_subset['sentiment']
          

The next step in this process is to train a doc2vec model to learn tweet representations. Ideally, any large dataset of tweets will work for this step. However, since we don’t have such a ready made corpus, we will split our dataset into train-test, and use the training data for learning the doc2vec representations. First part of this process involves converting the data into a format readable by the doc2vec implementation, which can be done using TaggedDocument class. It is used to represent a document as a list of tokens, followed by a “tag”, which, in its simplest form, can just be the file name or id of the document. However, doc2vec by itself can also be used as a nearest neighbor classifier for both multi-class and multi-label classification problems, using TaggedDocument. We will leave this as an exploratory exercise for the reader. Let us now see how to use this class to use the text of the tweets in our training data to train a doc2vec mode, through the code snippet below.

          train_data, test_data, train_cats, test_cats = train_test_split(mydata,mycats,random_state=1234)
          #prepare training data in doc2vec format:
          train_doc2vec = [TaggedDocument((d), tags=[str(i)]) for i, d in enumerate(train_data)]
          #Train a doc2vec model to learn tweet representations. Use only training data!!
          model = Doc2Vec(vector_size=50, alpha=0.025, min_count=10, dm =1, epochs=100)
          model.build_vocab(train_doc2vec)
          model.train(train_doc2vec, total_examples=model.corpus_count, epochs=model.epochs)
          model.save("d2v.model")
          print("Model Saved")
          

Training for doc2vec involves making several choices regarding parameters, as seen in the model definition in the code snippet above. Vector_size refers to the dimensionality of the learned embeddings, alpha is the learning rate, min_count is the minimum frequency of words that remain in vocabulary, dm, which stands for distributed memory is one of the representation learners implemented in doc2vec (the other is dbow, distributed bag of words), epochs are the number of training iterations. There are a few other parameters which can be customized. While there are some guidelines on choosing optimal parameters for training doc2vec models¹², these are not exhaustively validated, and we don’t know if the guidelines work for tweets.

The best way to address this issue is to explore a range of values for the ones that matter to you (e.g., dm vs dbow, vector sizes, learning rate) and compare multiple models. How do we compare these models, as they only learn the text representation? One way to do that is to start using these learned representations in a downstream task, i.e., text classification in this case. Doc2vec’s infer_vector function can be used to infer the vector representation for a given text, using a pre-trained model. Since there is some amount of randomness due to negative sampling and other parameters, the inferred vectors differ each time we extract them. For this reason, to get a stable representation, we run it multiple times (called steps) and aggregate the vectors. Let us use the learned model, to infer features for our data, and train a logistic regression classifier.

          #Infer the feature representation for training and test data using the trained model
          model= Doc2Vec.load("d2v.model")
          #infer in multiple steps to get a stable representation.
          train_vectors = [model.infer_vector(list_of_tokens, steps=50) for list_of_tokens in train_data]
          test_vectors = [model.infer_vector(list_of_tokens, steps=50) for list_of_tokens in test_data]
          myclass = LogisticRegression(class_weight="balanced") #because classes are not balanced.
          myclass.fit(train_vectors, train_cats)
          preds = myclass.predict(test_vectors)
          print(classification_report(test_cats, preds))
          

Now, the performance of this model seems rather poor, achieving an F1 score of 0.51 on a reasonably large corpus, and with only three classes. There can be a couple of interpretations for this poor result. Firstly, unlike full news articles, or even well-formed sentences, tweets contain very little data per instance. Further, people write with a wide variety in spelling and syntax when they tweet. There are a lot of emoticons, in different forms. Our feature representation should be able to capture such aspects. While tuning the algorithm’s by searching a large parameter space for the best model may help, an alternative in such situations could be to explore problem specific feature representations, as we discussed in Chapter 3.

An important concern to keep in mind when using doc2vec is the same as for fastText. If we have to use doc2vec for feature representation, we have to store the model that learnt the representation. Why it is not typically as bulky as fastText, it is also not as fast to train. Such trade-offs need to be considered and compared against, before taking a deployment decision.

So far, we saw a range of feature representations and how they can be useful for text classification using machine learning algorithms. Let us now turn to a family of algorithms that became popular in the past few years, known as “deep learning.”

CNN for Text Classification

Let us now look at how to define, train, and evaluate a CNN model for text classification. CNNs typically consist of a series of Convolution and Pooling layers as the hidden layers. In the context of text classification, CNNs can be thought of as learning the most useful bag-of-words/ngrams features, instead of taking the entire collection of words/ngrams as features as we did in earlier in this chapter. Since our dataset has only two classes - positive and negative, the output layer has two outputs, with the softmax activation function. We will define a CNN with 3 convolution-pooling layers using the Sequential model class in Keras, which allows us to specify deep learning models as a sequential stack of layers - one after another. Once the layers and their activation functions are specified, the next task is to define other important parameters such as the optimizer, loss function and the evaluation metric to tune the hyperparameters of the model. Once all this is done, the next step is to train and evaluate the model. The following code snippet shows one way of specifying a CNN architecture for this task using the Python library Keras, and the results with IMDB dataset for this model.

          print('Define a 1D CNN model.')
          cnnmodel = Sequential()
          cnnmodel.add(embedding_layer)
          cnnmodel.add(Conv1D(128, 5, activation='relu'))
          cnnmodel.add(MaxPooling1D(5))
          cnnmodel.add(Conv1D(128, 5, activation='relu'))
          cnnmodel.add(MaxPooling1D(5))
          cnnmodel.add(Conv1D(128, 5, activation='relu'))
          cnnmodel.add(GlobalMaxPooling1D())
          cnnmodel.add(Dense(128, activation='relu'))
          cnnmodel.add(Dense(len(labels_index), activation='softmax'))
          cnnmodel.compile(loss='categorical_crossentropy',
           optimizer='rmsprop',
           metrics=['acc'])
          cnnmodel.fit(x_train, y_train,
           batch_size=128,
           epochs=1, validation_data=(x_val, y_val))
          score, acc = cnnmodel.evaluate(test_data, test_labels)
          print('Test accuracy with CNN:', acc)
          

As you can see in this code snippet, we made a lot of choices in specifying the model such as: activation functions, hidden layers, layer sizes, loss function, optimizer, metrics, epochs and batch size. While there are some commonly recommended options for these, there is no consensus on one combination that works best for all data sets and problems. A good approach while building your models is to experiment with different settings (called hyper parameters). Keep in mind that all these decisions come with some associated cost. For example, in practice, we have the number of epochs as 10 or above. But it also increases the amount of time it takes to train the model. Another thing to note is: if you want to train a embedding layer instead of using pre-trained embeddings in this model, the only thing that changes is the line: cnnmodel.add(embedding_layer). Instead of that, we can specify a new embedding layer, for example, as: cnnmodel.add(Embedding(Param1, Param2)). The Figure below shows the code snippet and model performance for the same.

          print("Defining and training a CNN model, training embedding layer on the fly instead of using pre-trained embeddings")
          cnnmodel = Sequential()
          cnnmodel.add(Embedding(MAX_NUM_WORDS, 128))
          …
          ...
          cnnmodel.fit(x_train, y_train,
           batch_size=128,
           epochs=1, validation_data=(x_val, y_val))
          score, acc = cnnmodel.evaluate(test_data, test_labels)
          print('Test accuracy with CNN:', acc)
          

If you run this code in the notebook, we notice that, in this case, training the embedding layer on our own dataset seems to result in better classification on test data. However, if the training data were substantially small, sticking to the pre-trained embeddings, or using the domain adaptation techniques we will discuss later in this chapter would be a better choice.

LSTMs for text classification

LSTMs, and other variants of RNNs in general, have become the goto way of doing neural language modeling in the past few years. This is primarily because language is sequential in nature and RNNs are specialized in working with sequential data. Current word in the sentence depends on its context - words before and after. However, when we model text using CNNs, this crucial fact is not taken into account. RNNs work on the principle of using this context while learning the language representation or a model of language. Hence, they are known to work well for NLP tasks. There are also CNN variants that can take such context into account and CNNs versus RNNs is still an open area of debate. In this section, we will see an example of using RNNs for text classification. Now that we already saw one neural network in action, it is relatively easy to train another! Just replace the convolutional/pooling parts with an LSTM in the above two code examples! The following code snippet shows how to train an LSTM model using the same IMDB dataset for text classification.

          print("Defining and training an LSTM model, training embedding layer on the fly")
          rnnmodel = Sequential()
          rnnmodel.add(Embedding(MAX_NUM_WORDS, 128))
          rnnmodel.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
          rnnmodel.add(Dense(2, activation='sigmoid'))
          rnnmodel.compile(loss='binary_crossentropy',
           optimizer='adam',
           metrics=['accuracy'])
          print('Training the RNN')
          rnnmodel.fit(x_train, y_train,
           batch_size=32,
           epochs=1,
           validation_data=(x_val, y_val))
          score, acc = rnnmodel.evaluate(test_data, test_labels,
           batch_size=32)
          print('Test accuracy with RNN:', acc)
          

You will notice that this code took much longer to run than the CNN example. One needs to note that while LSTMs are more powerful in utilizing he sequential nature of text, they are much more data hungry as compared to CNNs. Thus, the relative lower performance of the LSTM on a dataset need not necessarily be interpreted as a shortcoming of the model itself. It is possible that the amount of data we have is not sufficient to utilize the full potential of an LSTM. As with the case of CNN, several parameters, and hyper parameters play a very important role in the model performance, and it is always a good practice to explore multiple options and compare different models before finalizing on one.

In this section, we introduced the idea of using deep learning for text classification, using two neural network architectures - CNN and LSTM. There are several variants to these architectures, and new models are being proposed everyday by NLP researchers. However, in our experience as industry practitioners, several NLP tasks, especially text classification, still widely use several of the non-deep learning approaches we described earlier in the chapter. Two primary reasons for this could be: lack of large amounts of task specific training data that neural networks demand, and issues related to computing and deployment costs. We will end this section reiterating what we mentioned earlier when we discussed about a text classification pipeline- in a practical scenario, it always makes sense to start with a simpler, easy to deploy approach as your MVP and incrementally go from there, taking customer needs and feasibility into account. Let us now look at how to address the first reason - training data.

No Training Data

Let us say you were asked to design a customer complaint classifier for your ecommerce company. The classifier is expected to automatically route customer complaint emails into a set of categories, say, billing, delivery, and others. If you are fortunate, you may discover a source of large amounts of annotated data for this task within the organization, in the form of a historical database of customer requests and their categories. If such a database does not exist, where should we start to build our classifier?

The first step in such a case is the creation of an annotated dataset, where customer complaints are mapped to a set of categories mentioned above. One way to approach this is to ask customer service agents to manually label some of the requests, and use that as the training data for your machine learning model. Another approach is called “bootstrapping” or “weak supervision”. There can be certain patterns of information in different categories of customer requests. Perhaps, billing related requests mention variants of the word bill, amounts in a currency etc. Delivery related requests talk about shipping, delays etc. One can get started with compiling some such patterns, and using their presence in a customer request to label it, thereby creating a small (perhaps noisy) annotated dataset for this classification task. From here, one can build a classifier to annotate a larger collection of data. Snorkel¹⁷, a recent software tool developed by Stanford University, is a useful tool to deploy weak supervision for various learning tasks, including classification. Snorkel was used deploy weak supervision based text classification models at industrial scale, at Google¹⁸. They showed that weak supervision could create classifiers comparable in quality to those trained on tens of thousands of hand labeled examples!

In some other scenarios, where large scale collection of data is necessary and feasible, crowdsourcing can be seen as an option. Websites such as Amazon Mechanical Turk (https://www.mturk.com/) and figure-eight.com provide platforms to make use of human intelligence to create high quality training data for machine learning tasks. A popular example of using crowd wisdom to create a classification dataset is the Captcha test Google uses to ask if a set of images contain an object (e.g., Select all images that contain a street sign.).

Less Training Data: Active Learning and Domain Adaptation

In the scenario described earlier when you collected small amounts of data using human annotations or bootstrapping, it may sometimes turn out that the amount of data was too small to build a good classification model. It is also possible that most of the requests we collected belonged to billing, and very few belonged to the other categories - which will result in a very imbalanced dataset. Asking the agents to spend many hours doing manual annotation is not realistic either. What should we do in such scenarios?

One approach to address such problems is “active learning”, which is primarily about choosing the examples to use as training data. The first step in this involves training the classifier with the available amount of data, and start using it to make predictions on new data. Wherever the classifier is very unsure of its predictions, send them to human annotators for correct classification. Then, re-train the model using the previous training data and the newly compiled human annotations. This process is repeated until a satisfactory model performance is reached. Tools such as Prodi.gy¹⁹ have active learning solutions implemented for text classification, and support the efficient usage of active learning to create annotated data and text classification models quickly.

Imagine a scenario for your customer complaint classifier, where you have a lot of historical data for a range of products. However, you are now asked to tune it to work on a newer products. What is potentially challenging in this situation? Typical text classification approaches rely on the vocabulary of the training data. Hence, they are inherently biased towards the kind of language seen in the training data. So, if the new products are of a very different nature (e.g., model is trained on a suite of electronic products, and we are using it with complaints on cosmetic products), the pre-trained classifiers trained on some other source data may not do well. However, it is also not realistic to train a new model on each product or product suite, as we will again run into the problem of insufficient training data. Domain adaptation is a method to address such scenarios, using which we can “transfer” what we learnt with one domain (source) with large amounts of data to another domain (target), with lesser amount of labeled, but large amounts of unlabeled data.

A typical pipeline for domain adaptation in text classification works as follows:

1. Start with a large, pre-trained language model of the source domain (e.g., Wikipedia data).

2. Fine-tune this model using the target language’s unlabeled data

3. Train a classifier on the labeled target domain data, by extracting feature representations from the fine-tuned language model from Step 2.

ULMFit²⁰ is a popular domain adaptation approach for text classification. In research experiments, it was shown that this approach matches the performance of training from scratch with 10-20 times more training examples with only 100 labeled examples in text classification tasks. When unlabeled data was used to fine tune the pre-trained language model, it matched the performance of using 50-100 times more labeled examples when trained from scratch, on the same text classification tasks. Domain adaptation and transfer learning methods are currently an active area of research in NLP. While their use for text classification did not show dramatic improvements yet on standard datasets and they are not so commonly used in industry setups yet, we can expect to see this getting better in near future.

A Case Study

Let us consider a real-world scenario and how we can use some of these concepts we discussed in this section for that. Imagine you are asked to build a corporate ticketing system for your organization. This system will track all the tickets or issues people face in the organization and route it in internally or to externally. [Link to Come].9 shows a representative screenshot for such a system:

Figure 2-9. A corporate ticketing system

Now let us say your company has recently hired a medical counsel and partnered with a hospital. So your system should also be able to pinpoint any medical related issue and pinpoint it to relevant people and teams. But while you have some past tickets, none of them are labeled. How will you go about building such a system?

Let us explore a couple of options:

1. Existing APIs or Libraries: One option is to start with a public API or a library and map its classes to what is relevant to you. For instance, if we look at the Google APIs we mentioned earlier in the chapter, it can classify content into over 700 categories. There are 82 categories that are associated with medical or health issues. These include categories like /Health/Health Conditions/Pain Management, /Health/Medical Facilities & Services/Doctors’ Offices, /Finance/Insurance/Health Insurance etc.

2. While not all categories are relevant to your organisation, some could be and you can map these accordingly. For instance, let us say your company does not consider substance abuse and obesity issues as relevant for medical counsel. So, you can ignore /Health/Substance Abuse and /Health/Health Conditions/Obesity in this API. Similarly whether insurance should be a part of HR or referred outside, can be handled with these categories.

3. Public Datasets: You can also adopt public datasets for your needs. For example, 20 Newsgroups is a popular text classification dataset, which is also a part of sklearn library. It has a range of topics including sci.med. We can also use it to train a basic classifier, classifying all other topics in one category and sci.med in another.

4. Weak Supervision: We have a history of past tickets, but they are not labelled. So, we can consider bootstrapping a dataset out of it using the approaches described earlier in this section. For example, consider having a rule - “if the past ticket contains words like Fever, Diarrhea, Headache or Nausea you put them in the medical counsel category”. This rule can create a small amount of data, which we can use as a starting point for our classifier.

5. Active Learning: We can use tools like Prodigy to conduct a data collection experiments, where we ask someone working in the customer service desk to look at ticket descriptions and tag them with a preset list of categories. [Link to Come] shows an example of using prodigy for this purpose.

   

6. [Link to Come] Active Learning with Prodigy

7. Learning from Implicit and Explicit Feedback: Throughout the process of building, iterating and deploying this solution, you are getting feedback that you can use to improve your system. Explicit feedback can be when the medical counsel or the hospital explicitly says that the ticket was not relevant. Implicit feedback can be extracted from other dependent variables like ticket response times and ticket response rates. All of these could be factored to improve your model, using active learning techniques.

A sample pipeline incorporating these ideas can look as shown in [Link to Come].11:

Figure 2-10. A pipeline for building a classifier when there is no training data

In this section, we started looking at a practical scenario of not having enough training data for building our own text classifier, for our custom problem. We discussed several possible solutions to address this issue. We expect that this prepared you to anticipate some of the scenarios related to data collection/creation in your future projects related to text classification.