Applying the labels to the datasets

The process of getting the labeling functions to “vote” on each record is called “applying the labeling functions”. The family of objects that applies the labeling functions is called appliers. There are several appliers:

• DaskLFApplier- Applies the defined labeling functions to the Dask DataFrames (Dask is a DataFrame composed of smaller Pandas DataFrames that are accessed and operated upon in parallel).

• PandasLFApplier - Applies the array of labeling functions to a Pandas data frame.

• PandasParallelLFApplier - This applier operates by parallelizing the Pandas data frame into a Dask data frame and uses the DaskLFApplier to work on the partitions in parallel, being, therefore, faster than the PandasLFApplier.

• SparkLFApplier - Applies the labeling functions over a Spark Resilient Distributed Dataset(RDD).

For our example, since the data is small and the process can easily run on a desktop or laptop, we will be using the PandasLFApplier.

Let’s import the Snorkel PandasLFApplier.

from snorkel.labeling import PandasLFApplier

Next, let’s define an array, lfs where we will declare all the labeling functions, followed by the applier PandasLFApplier. The applier.apply call applies the labeling functions to the data frame df.

lfs = [
        is_odd,
        is_even,
        is_two,
        is_known_prime
      ]

applier = PandasLFApplier(lfs=lfs)
L_train = applier.apply(df=df)

Analyzing the labeling performance

After applying the labeling functions, we can run an analysis on how they are performing. Snorkel comes equipped with LFAnalysis which summarizes the performance of the labeling functions by describing: polarity, coverage, overlaps, and conflicts. To make use of LFAnalysis let us start with importing the package, and then printing the LFAnalysis summary.

from snorkel.labeling import LFAnalysis

LFAnalysis(L=L_train, lfs=lfs).lf_summary()

The output is a pandas data frame summarizing the four metrics above.

LFAnalysis(L=L_train, lfs=lfs).lf_polarities()

[[0], [0], [1], [1]]

df["is_odd"] = df.apply(is_odd, axis=1)
df["is_even"] = df.apply(is_even, axis=1)
df["is_two"] = df.apply(is_two, axis=1)
df["is_known_prime"] = df.apply(is_known_prime, axis=1)

LFAnalysis(L=L_train, lfs=lfs).lf_coverages()

array([0.55, 0.55, 0.05, 0.2 ])

LFAnalysis(L=L_train, lfs=lfs).lf_overlaps()

array([0.55, 0.55, 0.05, 0.05])

LFAnalysis(L=L_train, lfs=lfs).lf_conflicts()

array([0.05, 0.05, 0.05, 0.05])

Using a validation set

Let us assume that for a small portion of the data we have ground truth labels. This data can serve as a validation set. Even when we do not have a validation set, it often is possible to label a subset of the data using subject-matter experts (SMEs). Having this validation dataset (10% - 20% the size of the training data, depending on how representative of the entire dataset you think that portion is) can help evaluate further the labelers. On this set, we can calculate the additional statistics: “Correct”, “Incorrect” and “Empirical Accuracy”.

To make use of the validation set, we would start by applying the labeling functions to it.

# define a validation set, and create a DataFrame
validation = [22, 11, 7, 2, 32]
df_val = pd.DataFrame(validation, columns=["Number"])

# gather the ground truth labels
true_labels = np.array([0, 1, 1, 1, 0])

# apply the labels
L_valid = applier.apply(df_val)

# analyze the labelers and get the summary df
LFAnalysis(L_valid, lfs).lf_summary(true_labels)

The resulting data frame will have the additional three-column statistics, from the initial one presented in Table 2-9: Correct, Incorrect, Empirical Accuracy.

The “Correct” and “Incorrect” are the actual count of samples this labeling function has correctly or incorrectly labeled. It is easier to interpret these numbers by looking at the output of the labeling functions. Doing this, of course, is just for this illustration, as it does not scale in practice.

df_val = pd.DataFrame(validation, columns=["Number"])
df_val["is_odd"] = df_val.apply(is_odd, axis=1)
df_val["is_even"] = df_val.apply(is_even, axis=1)
df_val["is_two"] = df_val.apply(is_two, axis=1)
df_val["is_known_prime"] = df_val.apply(is_known_prime, axis=1)
df_val

“Empirical Accuracy” is the percentage of samples that have been labelled correctly, ignoring samples on which the labeling function abstained. For is_known_prime, we can see that it has returned PRIME for three samples, and they have all been correct, therefore its Empirical Accuracy is 1.0. The is_even function, on the other hand, has been correct for 2/3 non-ABSTAIN returns, therefore its accuracy is 67%.

Intuition behind LabelModel

The agreement and disagreement of the labeling functions serve as the source of information to calculate the accuracies of the labeling functions. The label model combines these agreements and disagreements with a type of independence—conditional independence, to obtain certain relationships that enable solving for the accuracies. The intuition behind why this is possible relates to the fact that if we have enough labeling functions, even just examining the most popular vote will provide a noisy, but higher-quality, estimate of the true label. Once this has been obtained, we can examine how each labeling function performs against this label to get an estimate of its accuracy. The actual functionality of the label model avoids this two-step approach and instead uses an application of the law of total probability and Bayes’ rule. Given conditional independence of the labeling functions S1, S2, it holds that

P(S1,S2) = P(S1, S2,Y=-1) + P(S1, S2,Y=+1)
= P(S1, S2 | Y=-1)P(Y=-1) + P(S1, S2 | Y=+1)P(Y=+1)
= P(S1|Y=-1)P(S2|Y=-1)P(Y=-1) + P(S1|Y=+1)P(S2 | Y=+1)P(Y=+1)

The left-hand side contains the agreement/disagreement rate between the two labeling functions, while the last equation contains the sum of products of accuracies (and the priors P(Y)).

Now, we can obtain at least two more such equations by adding additional labeling functions. We can then solve for the accuracy terms such as P(S1|Y=-1), either via gradient descent, or by solving a system of equations.

LabelModel parameter estimation

Above we provided some of the intuition behind how Snorkel obtains the accuracy parameters from agreements and disagreements among a few labeling functions. This can be generalized into an algorithm to obtain all of the accuracies. There are several such algorithms; the one used in Snorkel is described in more depth in Ratner et al. (Ratner et al. 2018).

As we saw, the key requirement was having conditional independence between labeling functions. A useful tool to represent the presence of such independences is a graphical model. These are graphical representations of dependence relationships between the variables, which can be used to obtain valid equations with the form described above. Remarkably, the graph structure also relates to the inverse of the covariance matrix between the labeling functions and the latent label node, and this relationship enables a matrix-based approach to solving for the accuracy parameters.

Specifically, it is possible to complete the inverse of the covariance matrix just based on the agreement and disagreement rates between the labeling functions. Having done this, we have access to the accuracies - the off-diagonal blocks of this covariance matrix - and can then better synthesize the labeling function votes into probabilistic labels using the graphical model formulation.

To illustrate the intuition about learning the probability of the accuracy of each labeling function with an example based on agreement/disagreement, let us image that we want to solve the following classification problem: separating a set of images into indoor and outdoor categories. Let us also assume that we have three labeling functions recognizing features related to the sky, clouds, and wooden floor. The labeling function describing the sky features, and the labeling function describing the clouds will probably agree in most pictures on whether the picture is indoor or outdoor, and have the opposite prediction with a labeling function judging whether there is a wooden floor in the picture. For datapoints we will have a data frame like the one represented in the table below:

To train a LabelModel on this DataFrame, we would do:

L = np.array(df[["has_sky", "has_clouds", "has_wooden_floor"]])
label_model = LabelModel()
label_model.fit(L)
np.round(label_model.get_weights(), 2)

When you examine the weights of the label_model for each classification source (labeling function), you will notice how the first labeling function has_sky is weighted less than the second one, has_clouds due to the mistake it makes on data point “outdoors5”:

array([0.83, 0.99, 0.01])

The calculated conditional probabilities for each source and each class (outdoor, indoor, abstain) are retrieved via:

np.round(label_model.get_conditional_probs(), 2)

And the actual conditional probability values placed in a matrix with dimensions [number of labeling function, number of labels + 1(for abstain), number of classes], rounded are:

array([[[0.  , 0.  ],
        [0.66, 0.01],
        [0.34, 0.99]],

       [[0.  , 0.  ],
        [0.99, 0.01],
        [0.01, 0.99]],

       [[0.  , 0.  ],
        [0.01, 0.99],
        [0.99, 0.01]]])

Data augmentation through word removal

To put into action one of the EDA techniques, let’s try building a transformation function that will remove all the adverbs from the text. The meaning of the sentence should still be preserved since the adverbs enhance the meaning of the sentence, but their absence still conveys most of the information, and leaves the sentence still sounding grammatically correct. We will make use of the nltk package to tokenize the text and tag the parts of speech (POS), so we can identify and then remove the adverbs.

The nltk package annotates the adverbs with RB, RBR and RBS, as shown in the Table 2-14 table. The remove_adverbs function does just what we described above: tokenizes the sentence, extracts the (word, POS tag) array of tuples, and filters out the elements where the POS is in the tags_to_remove list before rejoining the remaining words into a sentence. To leave the sentence grammatically correct, at the end we correct the spacing that might be introduced in front of the period by joining. Decorating the function with transformation_function, converts this Python function into a Snorkel transformation_function that can be easily applied to all elements in the dataset using Snorkel tooling.

import nltk
nltk.download("averaged_perceptron_tagger")

tags_to_remove = ["RB", "RBR", "RBS"]
@transformation_function()
def remove_adverbs(x):
    tokens = nltk.word_tokenize(x["review_text"])
    pos_tags = nltk.pos_tag(tokens)
    new_text = " ".join([x[0] for x in pos_tags if x[1]
        not in tags_to_remove]).replace(" .", ".")
    if(len(new_text) != len(tokens)):
        x["review_text"] = new_text
    else:
        x["review_text"] = "DUPLICATE"
    return x

The last several lines of the remove_adverbs function checks whether the generated text is any different from the original. By setting the review_text to DUPLICATE in this case, we can filter those records out later.

Let’s now inspect the outcome of applying this transformation_function to one of the reviews.

df[df.product_name == "Honest Illusions: Books: Nora Roberts"].iloc[0]["review_text"]

The original review is as follows:

“I simply adore this book and it’s what made me go out to buy every other Nora Roberts novel. It just amazes me about the chemistry between Luke and Roxanne. Everytime I read it I can’t get over the excitement and the smile it brings to me as they barb back and forth. It’s just an amazing story over the course of growing up together and how it all develops. Roxanne is smart and sassy and Luke is just too cool. This romantic duo shines and no one else compares to them yet for me. My very favorite romance.”

And the version obtained from the transformation_function is:

“I adore this book and it ’s what made me go out to buy every other Nora Roberts novel. It amazes me the chemistry between Luke and Roxanne. Everytime I read it I ca get over the excitement and the smile it brings to me as they barb and forth. It ’s an amazing story over the course of growing up and how it all develops. Roxanne is smart and sassy and Luke is cool. This romantic duo shines and no one compares to them for me. My favorite romance”

Note that the adverbs: “simply”, “just”, “very”, “too”, “yet” ,"everytime” are missing from the output of the transformation_function.

Snorkel Preprocessors

Snorkel has a dedicated module for preprocessing data before it is passed to a labeling_function or transformation_function. The results of this preprocessing are added to the data points as additional columns that can be accessed from the body of both labeling_functions and transformation_functions.

One of the built-in Snorkel preprocessors is the SpacyPreprocessor. The SpacyPreprocessor makes use of the SpaCy nlp package to tokenize the words of the text, and attach information about entities, part of speech tags found on the specified text column of a data point.

This is offered for convenience, so the package users don’t have to code from scratch (as we did above for adverbs), tokenizing the text inside the body of the transformation_function, as we did before, but can make use of the output of SpaCy by just specifying the pre parameter in the transformation_function annotation. Using the SpacyPreprocessor the SpaCy doc is present and ready for inspection.

Let’s first import the `SpacyPreprocessor module, and instantiate it:

from Snorkel.preprocess.nlp import SpacyPreprocessor

spacy = SpacyPreprocessor(text_field="review_text", doc_field="doc", language="en")

The name of the dataset column containing the text to be processed gets passed to the text_field argument, the name of the column where the preprocessing output will be placed is defined in the doc_field entry. The third argument is the language of the text. Our transformation_function that makes use of the SpacyPreprocessor would then be as follows:

@transformation_function(pre=[spacy])
def spacy_remove_adverbs(x):
    words_no_adverbs = [token for i, token in enumerate(x.doc)
                        if token.pos_ != "ADV"]
    new_sentence = " ".join([x.text for x in words_no_adverbs])
    if(len(words_no_adverbs) != len(x["review_text"])):
        x["review_text"] = new_sentence.replace(" .  ", ". ")
    else:
        x["review_text"]= "DUPLICATE"
    return x

The list of Token objects is already present in the doc field. Like the remove_adverbs function, the spacy_remove_adverbs inspects the part-of-speech tags and removes the ones marked as “AVD"(adverbs), but with much less work!

Data augmentation through GPT-2 prediction

In “Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations”, Kobayashi (Kobayashi, 2018) proposes text data augmentation by replacing words in the text using a bi-directional language model. As an example of this, we can leverage Snorkel to generate new records of data using predictive language models like GPT-2 (at the time of writing, GPT-2 is the best open-source available model for text prediction). There are several versions of the GPT-2 models that can be downloaded from the OpenAI Azure storage account: 124M, 355M, 774M and 1558M. The models are named after the number of parameters they use.

Let’s get started by installing the required packages and downloading one of the models, the 355M one (so we can strike a balance between accuracy and ease of use) following the instructions in Developers.md

# Create a Python 3.7 environment
conda create -n GPT2 Python=3.7
# Activate the environment
conda activate GPT2
# Download GPT2
git clone https://github.com/openai/gpt-2.git

pip install -r requirements.txt
pip install tensorflow-gpu==1.12.0 fire regex
# Download one of the pretrained models.
Python download_model.py 355M

Now that we have the GPT-2 model, inference scripts, and utilities to encode the text, we can use them to write a Snorkel transformation_function that takes as arguments 10 words from the review text and predicts another few words based on it. The core of the Snorkel transformer function is simply invoking a slightly modified version of the interact_model method, from the gpt-2 repository.

import fire
import json
import os
import numpy as np
import tensorflow as tf
import model
import sample
import encoder
from Snorkel.augmentation import transformation_function

def interact_model(
    raw_text,
    model_name="355M",
    seed=None,
    nsamples=1,
    batch_size=1,
    length= None,
    temperature=1,
    top_k=40,
    top_p=1,
    models_dir=r"C:gpt-2models",
):
    enc = encoder.get_encoder(model_name, models_dir)
    hparams = model.default_hparams()
    with open(os.path.join(models_dir, model_name, "hparams.json")) as f:
        hparams.override_from_dict(json.load(f))

    if length is None:
        length = hparams.n_ctx // 2

    with tf.Session(graph=tf.Graph()) as sess:
        context = tf.placeholder(tf.int32, [batch_size, None])
        np.random.seed(seed)
        tf.set_random_seed(seed)
        output = sample.sample_sequence(
            hparams=hparams, length=length,
            context=context,
            batch_size=batch_size,
            temperature=temperature, top_k=top_k, top_p=top_p
        )

        saver = tf.train.Saver()
        ckpt = tf.train.latest_checkpoint(os.path.join(models_dir, model_name))
        saver.restore(sess, ckpt)

        context_tokens = enc.encode(raw_text)
        all_text = []
        for _ in range(nsamples // batch_size):
            out = sess.run(output, feed_dict={
                context: [context_tokens for _ in range(batch_size)]
            })[:, len(context_tokens):]
            for i in range(batch_size):
                text = enc.decode(out[i])
                all_text.append(text)
        return ''.join(all_text)

@transformation_function()
def predict_next(x):
    review = x["review_text"]
    # extract the first sentence
    period_index = review.find('.')
    first_sentence = review[:period_index+1]
    #predict and get full sentences only.
    predicted = interact_model(review, length=50)
    last_period = predicted.rfind('.')
    sentence = first_sentence+" "+predicted[:last_period+1]
    x["review_text"] = sentence
    return x

The interact_model function takes as arguments the initial text, in the raw_text argument, as well as a model name and a series of parameters to tune the prediction. It encodes the text, creates a TensorFlow session, and starts generating the sample text.

Checking one of the examples of prediction using the following record with the review text as illustrated below.

df[df.product_name == "Honest Illusions: Books: Nora Roberts"]["review_text"]
.to_list()[0]

“I simply adore this book and it’s what made me go out to buy every other Nora Roberts novel. It just amazes me the chemistry between Luke and Roxanne. Every time I read it I can’t get over the excitement and the smile it brings to me as they barb back and forth. It’s just an amazing story over the course of growing up together and how it all develops. Roxanne is smart and sassy and Luke is just too cool. This romantic duo shines and no one else compares to them yet for me. My very favorite romance.”

Trying the transformation function for this particular entry generates the following text (on different runs the results will be different. If you need reproducible results, set the seed parameter in interact_model).

predict_next(df[df.product_name == "Honest Illusions: Books: Nora Roberts"].iloc[0])

The output is:

“I simply adore this book and it’s what made me go out to pick up books on it and give them a try. It’s a beautiful blend of romance and science, and it’s both very simple and complex at the same time. It’s something I’d recommend to anyone.”

The example above is generated setting the prediction length to 50, for illustration. For most practical text augmentation tasks, to follow the recommendation of Kobayashi and only generate a word, this parameter should be set to a much smaller value.

 predicted = interact_model(review, length=5)

Data Augmentation through translation

As we mentioned in the introduction to this augmentation section, translating text from one language to another, and back to the original language, is an effective augmentation technique, introduced by Yu et al. (Yu et al., 2018).

Let’s get started writing the next transformation_function using the Azure Translator to translate text from English to French and back to English. At the moment of writing, you can deploy a free SKU of the Translator, and translate up to 2 million characters per month.

To deploy the Translator, you can look for the product in Azure Marketplace, as shown in Figure 2-2.

Figure 2-2. Azure Translator

After the deployment, you will need the resource key, the endpoint, and the region of the resource, to authenticate the calls to the service. You can find them in the Keys and Endpoint view as shown in Figure 2-3.

Figure 2-3. Azure Translator keys, endpoint, region

The transformation function request to translate the text is adapted from the GitHub Translator examples.

import os, requests, uuid, json
import spacy
nlp = spacy.load("en")

resource_key = "<RESOURCE_SUBSCRIPTION_KEY>"
endpoint = "<ENDPOINT>"
region = "<REGION>"

def translate(text, language):
    headers = {
    "Ocp-Apim-Subscription-Key": resource_key,
    "Ocp-Apim-Subscription-Region" : region,
    "Content-type": "application/json",
    "X-ClientTraceId": str(uuid.uuid4())
    }

    body = [{"text" : text}]
    body = [{"text" : text}]
    response = requests.post(
        endpoint+"/translate?api-version=3.0&to="+language,
        headers=headers, json=body)
    translation = response.json()[0]["translations"][0]["text"]
    print(translation)
    return translation


@transformation_function()
def augment_by_translation(x):
    text = x["review_text"]
    french = translate(text, "fr")
    english = translate(french, "en")
    score = nlp(text).similarity(nlp(english))
    if(score < 1.0):
        x["review_text"] = english
    else:
        x["review_text"] = "DUPLICATE"
    return x

The translate function takes the text to translate and the language as arguments, and returns the translated text. The augment_by_translation transformation_function invokes the translate function twice: to translate into French then back to English. The similarity of the original text and the translation is calculated through SpaCy’s similarity routine, which measures the cosine similarity between context-sensitive blocks of text (for the small models). Perfect duplicates would generate a score of 1.0.

Reusing the same example, the review for “Honest Illusions”, we can compare the initial English text, with the text obtained after re-translating the French output back to English.

English: “I simply adore this book and it’s what made me go out to buy every other Nora Roberts novel. It just amazes me the chemistry between Luke and Roxanne. Every time I read it I can’t get over the excitement and the smile it brings to me as they barb back and forth. It’s just an amazing story over the course of growing up together and how it all develops. Roxanne is smart and sassy and Luke is just too cool. This romantic duo shines and no one else compares to them yet for me. My very favorite romance.”

French: “J’adore ce livre et c’est ce qui m’a fait sortir pour acheter tous les autres romans de Nora Roberts. Ça m’étonne la chimie entre Luke et Roxanne. Chaque fois que je le lis, je ca obtenir sur l’excitation et le sourire qu’il m’apporte comme ils barbe et en avant. C’est une histoire incroyable au cours de sa croissance et comment tout se développe. Roxanne est intelligente et impersy et Luke est cool. Ce duo romantique brille et personne ne se compare à eux pour moi. Ma romance préférée.”

Back to English: “I love this book and that’s what made me come out to buy all the other novels of Nora Roberts. I’m surprised at the chemistry between Luke and Roxanne. Every time I read it, I can get on the excitement and smile it brings me as they beard and forward. It’s an incredible story during its growth and how everything develops. Roxanne is smart and impersy and Luke is cool. This romantic duo shines and no one compares to them for me. My favorite romance.”

Applying the transformation functions to the dataset

We now have the transformation functions. To be able to use them, we also need to decide upon a policy to use for applying each transformation function to the dataset in order to get a diverse, but realistic output. The Snorkel augmentation package contains several policies, for this purpose:

• ApplyAllPolicy - This policy will take each data point, “book review” in our case, and apply all the transformation functions available to it, one by one, in the same order the transformation_functions are listed. The resulting text will be the product of having removed the adverbs, translated, and then supplied to gpt-2 for the next sentence prediction.

from Snorkel.augmentation import *

tfs = [predict_next, augment_by_translation, spacy_remove_adverbs]
policy = ApplyAllPolicy(len(tfs), n_per_original=1, keep_original=True)

tf_applier = PandasTFApplier(tfs, policy)
policy.generate_for_example()

The policy.generate_for_example object contains the list of the indices of the transformation functions selected to be applied to one particular data point, selected at random. The number of new data points to generate for each sample is defined by the n_per_original argument. For our case that would be all of them in order:

[[], [0, 1, 2]]

This policy will add one more data point per sample, doubling the dataset.

• ApplyEachPolicy - unlike ApplyAllPolicy, this policy will apply each transformation_function to each data point, separately, without combining them. keep_original indicates whether to keep the original data points in the augmented dataset.

policy =ApplyEachPolicy(len(tfs), keep_original=True)
policy.generate_for_example()

[[], [0], [1], [2]]

After applying this policy, for each data point, we will have 3 separate, new data points, each generated from one of the transformation functions.

• ApplyOnePolicy - applies a single policy, the first one, to the data.

policy = ApplyOnePolicy(4, keep_original=True)
policy.generate_for_example()

[[], [0], [0], [0], [0]]

After applying this policy, for each dataset record, we will have an additional 4 records, generated using the first transformation functon from the list (the transformation function at index 0).

• RandomPolicy - creates a list of transformation_functions by sampling the list of tfs at random. The length of this list is determined by the sequence_length argument.

policy =RandomPolicy(len(tfs), sequence_length=5, n_per_original = 2, keep_original=True)
policy.generate_for_example()

[[], [1, 2, 1, 2, 2], [2, 1, 2, 0, 1]]

• MeanFieldPolicy - this policy is probably the one that will have more uses because it allows the frequency of transformation function application to be defined by a user-supplied probability distribution. With this policy, a sequence of transformations will be applied to each data point, and the order/composition of that sequence will be determined by sampling from the list of transformation functions according to the supplied probability distribution.

policy = MeanFieldPolicy(
    len(tfs),
    sequence_length=1,
    n_per_original=1,
    keep_original=True,
    p=[0.3, 0.3, 0.4],
)
policy.generate_for_example()

[[], [0]]

After selecting a policy, for our case we kept the MeanFieldPolicy we go ahead and apply the policy to the dataset and get the synthetic data points.

df_train_augmented = tf_applier.apply(df2)

A preview of some of the records of the augmented dataset is shown in table Table 2-15.

We can use this dataset now, to train supervised models.