The Power of Pretraining

The “pre” in “pretraining” indicates that it refers to a process that occurs before the formal training of the model on the dataset (Figure 2-3).

Given that we want pretraining to benefit training, pretraining should orient the model in a way that makes training “easier.” With the pretraining step, the model’s performance should be better than without pretraining. If pretraining makes model performance worse, you should reevaluate your methods of pretraining.

In pretraining, a model must be trained to perform a task different from its ultimate task of predicting the label based on the input data. This pretraining task ideally presents important “context” and skills for the ultimate task such that the model can attain better performance on that ultimate task.

While these are two dominant attributes of pretraining, they are the result of an underlying phenomenon: conceptually, the process of pretraining brings the optimizer “closer” to the true solution in the loss landscape.

Consider the following loss landscape: it is the objective of the neural network optimizer to find the set of parameters such that the corresponding loss is minimized. This landscape has several features common in most modern loss landscapes: it has several local minima but only one global minimum and sloping, jagged fluctuations (Figure 2-4).

A model without pretraining may initialize and follow the following path. Since the first minima it encounters is more or less shallow, say the optimization algorithm overcomes it and discovers the next minima. Since this minima resides in a deeper pit and the optimizer has travelled a long distance, it will likely judge this to be the global optimum and converge at that location (Figure 2-5).

With pretraining, however, we’re able to get the model “closer” to the true global minimum of the loss landscape such that it converges to a more optimal solution faster, because it already begins from a place “close” to the global optimum. Correspondingly, we can use a less risky or erratically behaving optimizer because it begins from a convenient position (Figure 2-6).

Keep in mind that pretraining does not necessarily bring the model “closer” to the optimum in the literal sense of distance but rather that it becomes more convenient or “easier” for the model to find the optimum than if it hadn’t undergone pretraining. An optimizer very close to the global optimum but with several very deep local minima and very high local maxima would find it more difficult to arrive at the global optimum than an optimizer that was farther but whose path was smoother and easier to descend. What “closer” exactly means is dependent on the shape of the loss landscape, the optimizer’s behavior, and a host of other factors. For simplicity, though, thinking of “closer” in terms of “how easy it would be to navigate to the global optimum” suffices.

Note that the actual loss landscape for the pretraining task is different from that of the loss landscape of the ultimate task (as is displayed in Figure 2-6). It is not that pretraining operates within the loss landscape of the ultimate task and moves the model to a convenient position – that is just training on the task dataset, not pretraining.

Rather, in pretraining, we rely upon a similarity between the loss landscapes of the pretraining task and the ultimate task the model aims to perform. Therefore, a model that succeeds at the pretraining task should be at a generally successful location in the loss landscape for the ultimate task, from which it can further improve. This also means that pretraining may not be helpful if you believe the loss landscapes of the pretraining dataset and the task dataset are very “different.” Because the two often cannot be compared quantitatively for technical reasons, it is up to you to decide the necessity and performance boost of pretraining in a particular context.

Next, we’ll build upon this conceptual model to discuss the intuition behind two pretraining methods: transfer learning and self-supervised learning.

Transfer Learning Intuition

Transfer learning is premised upon the idea that knowledge gained from solving one problem can be used, or transferred, in solving another problem. Usually, the knowledge derived from a more general problem is used to aid a model’s ability to address a more specific problem.

Consider two datasets: a general dataset and a task dataset. The general dataset is designated for pretraining, whereas it is the model’s ultimate purpose to perform well on the task dataset (Figure 2-7).

In transfer learning, an initialized model is first trained on the general dataset to provide context and skills required to succeed on the task dataset. After the model is pretrained on the general dataset, it is then trained on the task dataset. This training structure is visualized in Figure 2-8. Because the weights are retained from the end of pretraining to the beginning of pretraining, the model has already acquired important skills and representations for pretraining that it can use when training on the general dataset.

Usually, the pretraining component of transfer learning is not done in house (by yourself). Selecting an appropriate general dataset and conceptualizing, implementing, and training a model to perform well on that general dataset is a significant amount of work, and it’s not necessary. There is a repository of pretrained models available in most deep learning frameworks (Figure 2-9), like Keras/TensorFlow and PyTorch and other sources, like pypi libraries, online code forums, and hosting sites. Each of these pretrained models is trained on some general dataset and is available for you to use.

You can then choose one of the already pretrained models and begin training them on your specific task dataset (Figure 2-10).

The number of problems with which this set of pretrained models could be used vastly outnumbers the number of pretrained models in the repository. This is fine, though, because we can expect each pretrained model to be capable of being applied to a wide array of problem types, since each model is expected to possess a form of “general knowledge.”

For instance, let’s say that we want to train a model to classify images of dogs and cats (Figure 2-11). We decide we want to use transfer learning and choose a general dataset of all sorts of real-life items and objects, like a parrot, an airplane, a car, a tomato, a fish, etc. For the purposes of this example, let’s say that images of dogs and cats are not present in the general dataset.

A model that succeeds on the general dataset must have already adapted to and accounted for these features of the general dataset. In a sense, it “understands” how to “interpret” edges, the dynamics of light, texture, and other important qualities. All that is necessary afterward is to adapt those learned skills and representations toward a similar but more specific task.

You may ask, “why not approach task dataset directly? Why develop these auxiliary skills when the model could have developed those skills more specifically and directly for the task dataset?” Indeed, in some cases transfer learning is not a suitable choice. However, in most sufficiently difficult tasks, pretraining via transfer learning helps to set up the foundation for learning that would be difficult for the model to build by itself.

The latter is more likely to be a successful strategy, because the average length of the jump from knowing nothing about addition to adding one-digit numbers to adding two-digit numbers is much smaller than the average length of the jump from knowing nothing about addition to adding two-digit numbers. Transfer learning should operate by this same intuition of teaching the model a general task such that a more specific, ultimate task becomes easier to approach.

Self-Supervised Learning Intuition

Self-supervised learning follows the logic of pretraining – some sort of pretraining task is performed before training the model on the task dataset to orient it toward better attaining representations and skills needed to perform well on the ultimate task.

The difference between self-supervised learning and transfer learning is that in transfer learning, the pretraining dataset is different from the task dataset, whereas in self-supervised learning, the pretraining dataset is constructed from the input data of the task dataset. Thus, while you would need two datasets to build the complete transfer learning training structure, only one is technically needed to build the complete self-supervised learning training structure (Figure 2-12).

It’s important to note that the altered task dataset is generally derived only from the input data of the task dataset. For instance, if a model’s task is to identify whether an image is of a dog or a cat, the altered task dataset could only be built upon the images (the input data of the task dataset), not the labels (the labels of the task dataset).

In each of these examples, none of the task dataset’s labels are needed – only the input data, or the x, of the task dataset is used in constructing these supervised datasets. You, as the deep learning engineer, can make some change to the task dataset input data and construct labels from that change.

Once the altered task dataset has been derived from the input data of the task dataset, it can be used in the pretraining training structure (Figure 2-13).

Note that in self-supervised learning (and pretraining in general), attaining a high performance metric is not the ultimate goal. In fact, if a model performs too well on the pretraining task, it may be that the pretraining task was too easy and hence did not support the growth of valuable skills and representations of data for formal training. On the other hand, if the model performs very poorly, the pretraining task may be too difficult.

It should be noted that self-supervised learning is technically unsupervised learning because we are extracting insights from the input data without any knowledge of the labels. However, self-supervised learning is a more commonly used and appropriate term: there is undoubtedly a supervised character to this sort of operation that distinguishes it from traditional unsupervised machine learning algorithms like K-means in that while the data generation (constructing altered task dataset) is technically unsupervised, the learning procedure (gradient updates, etc.) is supervised.

Self-supervised learning is also valuable because it allows us to capture valuable information without needing labels. Datamation estimates that the amount of unstructured data is increasing by 55% to 65% every year,² and the International Data Corporation projects that 80% of data will be unstructured by 2025.³ Unstructured data is defined as data that cannot be easily fit into a standard database model. An auxiliary characteristic of unstructured data, thus, is that there are seldom corresponding labels to train massive deep learning models with. However, the logic of self-supervised learning allows us to find hidden structures in unstructured data without labels and to exploit that knowledge to solve supervised problems.

For this reason, self-supervised learning is especially valuable for training on small datasets, in which there are few labels. Pretraining via self-supervised learning can allow the model to derive important insights it may not have obtained with only the traditional training structure.

Unlike transfer learning, there is no repository of self-supervised pretrained models, because in self-supervised learning, the pretraining task must be designed based on the task dataset. However, altering task datasets for self-supervised learning is generally not difficult if you have a good grasp of programming and data flows (see Chapter 1).

Of course, this is not to suggest that transfer learning does not develop world-representing knowledge or that self-supervised learning does not develop important skills. Rather, these are the root-level “spirits” or “characters” of transfer learning and self-supervised learning that can be used to guide intuition and what to expect from either method.

You may notice that we have been exploring these concepts through relatively fluid and artistic descriptors and ideas – “extracting insights,” “developing representations,” “world-representing.” While a textbook may attempt to formulate these ideas in equations and mathematical relationships, the truth is that even the most modern deep learning knowledge cannot fully explain and understand the depth of neural network behavior. Having an intuitive grasp of how neural networks function and “learn” – even if it is not mathematically rigorous – is valuable toward successful and efficient design.

Next, we will explore practical concepts in pretraining to understand how to manipulate neural network architectures for the implementation of transfer learning and self-supervised learning.

Transfer Learning Models and Model Structure

There are several important pretrained models and datasets to know. In this section, we will primarily be exploring image-based models, although you can apply much of the logic to other contexts. Image-based pretrained models generally follow a standard structure, consisting of two key steps: feature extraction and feature interpretation (Figure 2-14).

Feature extraction serves the purpose of assembling information into meaningful observations, like identifying and amplifying (or reducing) the presence of certain edges, shapes, or color patterns. A well-tuned feature extraction component is able to identify and amplify characteristics that are relevant to the problem. For instance, if the problem is to classify various grayscale images of shapes, the feature extraction component must be highly capable at detecting and amplifying the shape of edges.

Feature interpretation interprets the compiled extracted features to make a final judgment about the image. It takes in extracted information regarding the features of the image and can perform comparisons and other complex analyses across various regions of the image. A well-tuned feature interpretation component is able to effectively aggregate and make sense of extracted features in relation to the target output.

In practice, the roles of “feature extraction” and “feature interpretation” are played by convolutional and fully connected components (Figure 2-15). The convolutional component generally consists of layers like convolutions, pooling, and other image-based feature extraction layers. The fully connected component generally consists of fully connected or dense (in Keras terminology) layers.

We’ll talk more about certain architectures in depth in later chapters, but here we’ll do a very quick overview of important models.

In Keras, pretrained models are arranged into modules from which that model and related functions can be found within keras.applications.module_name. For instance, if you wanted to find the pretrained model and processing functions relating to the InceptionV3 model, you would find it in keras.applications.inception_v3. In addition to the pretrained model object, these modules often contain processing functions to apply to your data or the model output. We will see how these processing functions work in relation to the model soon.

The ImageNet Dataset

ImageNet is one of the most important datasets in image recognition. Professor Fei-Fei Li began working on ImageNet at Princeton in early 2007. Throughout the development of neural network applications in the image domain, ImageNet has been a core dataset upon which new methods and designs were conceived, tested, and reinvented.

The ImageNet dataset is structurally organized according to the WordNet hierarchy, in which objects are arranged in hierarchical relation to each other (Figure 2-16).

Each “meaningful concept” in ImageNet is referred to as a “synonym set” or a “synset.” Each “synset” can be described by multiple words or series of words. ImageNet contains over 100,000 synsets; each synset contains an average of 1000 images. Across 22,000 objects, ImageNet contains over 14 million images.

The ImageNet project was inspired by two important needs in computer vision research. The first was the need to establish a clear North Star problem in computer vision… Second, there was a critical need for more data to enable more generalizable machine learning methods. Ever since the birth of the digital era and the availability of web-scale data exchanges, researchers in these fields have been working hard to design more and more sophisticated algorithms to index, retrieve, organize and annotate multimedia data. But good research requires good resources…The convergence of these two intellectual reasons motivated us to build ImageNet.

—ImageNet Research Team⁵

The ImageNet Large Scale Visual Recognition Challenge ran from 2010 to 2017 and was instrumental to the development of image-processing deep learning work. Hence, many of the pretrained models available in Keras/TensorFlow and other deep learning frameworks are pretrained on the ImageNet dataset. Models pretrained on the ImageNet dataset have developed skills to recognize edges, textures, and other attributes of images of three-dimensional spaces and objects .

Changing Pretrained Model Architectures

To change pretrained model architectures , usually only the pretrained model’s convolutional component is transferred. Recall that transfer learning is primarily concerned with transferring the skills learned from one general dataset for application in a specific task dataset. Because the task dataset is different from the general dataset, while a model operating on the task dataset would still benefit from the feature-extracting skills transferred via transfer learning, it would need to develop its own interpretations.

Thus, while the skills for extracting features are valuable and can be transferred almost universally across most generally similar problems, often the interpretations of said features vary widely enough that the most successful strategy is to build a custom fully connected component, instead of transferring it (Figure 2-17).

However, in some occasions the problem set may be similar enough that the fully connected layer weights may also be transferred (Figure 2-18). This method requires no architecture manipulation of the pretrained model, since all the weights from the pretrained model are being transferred. To use this method, simply instantiate the pretrained model and train it on the task dataset without any architecture manipulation.

Another alternative is to keep the pretrained model’s fully connected component and to add more custom layers afterward (Figure 2-19). Think back to the art student analogy: you’re starting from the interpretative perspective of the Impressionist style that you were trained in, but additionally developing new interpretations of your interpretations to account for the new problem type. However, this method is generally less often used than only transferring weights in the convolutional component, since to be effective the neural network must be significantly deepened (more layers added), which can pose problems for updating weights, time, and model performance. Under certain conditions, though, it can be the optimal strategy.

Although we will only explore the first method through examples, all can be implemented through tools covered in Chapter 1, like the Functional API. Models can be treated like functional layers, expressed as functions of an input. See Chapter 3 for more complex exploration into neural network architecture manipulation.

Neural Network “Top” Inclusivity

All Keras/TensorFlow pretrained models have an important parameter, named include_top, which allows us to easily implement the first method of changing pretrained model architectures for training.

Recall that the fully connected component of an image-based neural network architecture is often referred to as its “top.” Thus, if the include_top parameter is set to True, the fully connected component will be retained in the instantiated pretrained model object. On the other hand, if the include_top parameter is set to False, the fully connected component will not be retained and the instantiated pretrained model object will contain only the input and convolutional components.

It’s important to realize that when there is no top, the input image shape is more or less arbitrary (see Note) because convolutional layers do not rely upon an absolute shape. A convolutional layer can act on an image of almost any shape (see Note for the qualifier “almost”), since it simply “slides” filters over an image. On the other hand, a dense layer in the fully connected component can only act upon inputs of a certain shape. Thus, if the include_top parameter is set to True, you are also limited in the shape of your input data to whatever the shape of the data the model was pretrained on.

Layer Freezing

Layer freezing is a useful tool for the implementation of transfer learning. When a layer is “frozen,” its weights are fixed and it cannot be trained. The purpose of transfer learning is to utilize important skills the pretrained model has developed. Thus, it is common to freeze the convolutional component of the pretrained model and to train only the custom fully connected component (Figure 2-20). This way, the feature-extracting skills learned and stored in the weights of the convolutional component are kept as is, but the interpretative fully connected component is trained to best interpret the extracted features. After this, sometimes the entire network is trained for fine-tuning. This allows the convolutional component to be updated in accordance with the developed interpretations in the fully connected component. Be aware that too much fine-tuning can lead to overfitting.

Think of layer freezing in relation to the art student analogy. Initially, you want to utilize the feature-extracting skills you developed in your art training and to construct new interpretations of contemporary art corresponding to the extracted features. However, after you have constructed new interpretations for the particular problem, you may find it helpful to go back and slightly update your feature-extracting skills to better service your new interpretative capabilities.

On a practical level, layer freezing is immensely helpful for practical success. Although it can be easy to build massive neural networks and call .fit() with increasing computational power, it remains true that it is not easy to optimize hundreds of millions (or even billions) of parameters by brute-force and achieve good performance. Layer freezing allows the neural network to focus on optimizing one segment of its architectures at a time and allows you to better use the weights gained from pretraining by restricting how much the neural network can deviate from the learned pretrained weights .

General Implementation Structure: A Template

It should be also noted that although in this context we are discussing image-based pretrained models, as transfer learning began its extensive development in the domain of images, the logic of a feature-extracting segment and an interpretative segment can be applied to other problem domains as well.

No Architecture or Weight Changes

If we make no changes to the architecture, we cannot adjust the input or the output of the pretrained model. Therefore, if we want to make no changes to the architecture, we must use the pretrained model for its original purpose.

The InceptionV3 model was trained on the ImageNet dataset. Thus, it takes an image of size (299,299,3) and returns an output vector with the probability the image belongs to one of 1000 classes.

Let’s begin by loading an example image to demonstrate how InceptionV3 would function (Listing 2-1). We will use requests to read a URL, PIL to read the corresponding image (Figure 2-21), and numpy to convert the image into an array.

import PIL, requests, numpy as np

from PIL import Image

url = 'https://cdn.pixabay.com/photo/2018/04/13/21/24/lion-3317670_1280.jpg'

im = np.array(Image.open(requests.get(url, stream=True).raw))

Listing 2-1

Loading image of a lion

The output of preprocess_layer (the result of the resized, preprocessed input data) is passed as the input to the InceptionV3 model. Note that we are making no changes to the model architecture or weights, so we set inlucde_top=True and weights='imagenet'. We can treat the model as a layer that takes in the output of a previous layer and can be passed as an input to the following layer: Inceptionv3 = InceptionV3(include_top=True, weights="imagenet")(preprocess_layer).

Note that you can set weights to None to just use the model architectures, without the pretrained weights. Although this doesn’t quite count as transfer learning, you may want to use this when the task dataset is so different from the pretraining dataset that any pretrained weights wouldn’t be of much benefit, but want to take advantage of some architecture’s characteristics, like efficiency or power.

We can create a model out of this set of layers using keras.models.Model (note that the pretrained model is treated like a layer and thus is considered an outputs layer): model = keras.models.Model(inputs=input_layer, outputs=Inceptionv3).

Note that the model expects four-dimensional data – for example, data with shape (100,299,299,3), indicating that it consists of 100 299x299 pixel RGB colored images. Even though we are submitting an individual image for prediction, it still needs to be four-dimensional. An easy way to accomplish this is to wrap it as an element in another array with np.array([im]).

A similar process can be applied to other pretrained models available via Keras. Most Keras pretrained models have associated preprocess_input and decode_predictions functions.

Transfer Learning Without Layer Freezing

If we want to adapt the pretrained model for our own task dataset, we need to make some minimal architecture changes such that the input and the output can accommodate our dataset. This follows a very similar structure to the previous application of no architecture or weight changes.

Note that we are keeping the ImageNet weights, but we do not include the top. While we want to keep the feature extraction skills in the convolutional layers, because we are appropriating the pretrained model for our own purposes, we don’t need the weights for the interpretative fully connected component.

Visualizing the model shows the shape of the data over time (Figure 2-22) with keras.utils.plot_model(model, show_shapes=True). The visualization also demonstrates that the InceptionV3 model is treated very much like a standard layer.

The model can be correspondingly compiled and trained. It should be noted that usually the fully connected component is much longer and complex to account for the complex nature of modern deep learning datasets.

Transfer Learning with Layer Freezing

In order to freeze a layer, set layer_obj.trainable=False.

Calling model.summary() is a good way to double-check if layer freezing worked or not. When several forms and series of neural network manipulations are being performed, it can be difficult to keep track and separate different forms of functions and instantiated objects.

You can unfreeze with inception_model.trainable = True and continue fitting again for fine-tuning.

You can also freeze individual layers with a similar method or by aggregating layers together into models and unfreezing groups of layers.

Accessing PyTorch Models

Keras/TensorFlow offers a large selection of pretrained models, and platforms like the model zoo (https://modelzoo.co/) or pypi libraries offer a wide range of pretrained models and/or architectures in Keras/TensorFlow.

PyTorch , a major alternative framework to Keras/TensorFlow, offers a larger selection of pretrained models. If a PyTorch pretrained model is not already implemented in Keras/TensorFlow, there’s a simple method to convert PyTorch models into Keras model objects so you can work with the model using familiar methods and steps.

PyTorch and Keras/TensorFlow, as well as most other frameworks, are built upon the Open Neural Network Exchange (ONNX), an AI ecosystem that establishes standards for representing various AI algorithms. ONNX was formulated with the purpose of encouraging innovation via conversion of algorithm representations across frameworks. Correspondingly, to convert a model from PyTorch to Keras/TensorFlow, the PyTorch model is first converted into ONNX format and then from ONNX format to the Keras/TensorFlow framework (Figure 2-23).

Let’s start by installing PyTorch with pip install torch. It can be imported as import torch.

You can find a list of PyTorch image-based pretrained models at https://pytorch.org/vision/stable/models.html (note that you can access other non-image-based pretrained models as well). These image-based pretrained models can be accessed at torchvision.models.model_name. You can install TorchVision via pip with pip install torchvision.

Now that we have instantiated the PyTorch model, we can convert it into a Keras model using the pytorch2keras model, which can be installed via pip with pip install pytorch2keras.

The input_np variable defines the shape of the input in channels-first format, where the “depth” of the image is listed as the first element of the shape. The torch.FloatTensor and torch.autograd.Variable functions convert the shape of the numpy array into valid tensor format.

The resulting Keras model can be visualized to double-check that the architecture has been transferred correctly and then compiled and fitted.

Note that not all PyTorch architectures can be converted into Keras/TensorFlow if some layer or operation is not supported for export by ONNX. See a list of transferable layers on the pytorch2keras GitHub: https://github.com/gmalivenko/pytorch2keras .

Transfer Learning Case Study: Adversarial Exploitation of Transfer Learning

As deep learning models are increasingly deployed in common production, it is becoming more important to incorporate the findings from the rising field of adversarial learning into deep learning designs. Adversarial learning is concerned with how to exploit weaknesses in deep learning models, often by making small changes that are imperceptible to the human eye but that completely change the model’s output. Vulnerabilities in deep learning can lead to dangerous outcomes – imagine, for instance, if malicious or accidental alterations to a traffic sign cause a self-driving car to suddenly accelerate or brake.

In a 2020 paper entitled “A Target-Agnostic Attack on Deep Models: Exploiting Security Vulnerabilities of Transfer Learning,” Shahbaz Rezaei and Xin Liu⁶ explore how adversarial learning can be applied to transfer learning. Because many modern applications of deep learning use transfer learning, a malicious hacker could analyze the architecture and weights of the pretrained model to exploit weaknesses in the deployed model.

Rezaei and Liu show that a hacker, with knowledge only of the pretrained model used in the application model, can develop adversarial inputs leading to any desired outcome. Because the hacker does not need to know the weights or architecture of the application model or the data it was retrained on, the method of attack is target agonistic, which allows it to exploit wide swaths of deployed deep learning models that utilize publicly available pretrained models.

Consider the following model structure, which uses a simple form of transfer learning (Figure 2-25). The input to the model is passed through the pretrained model (both convolutional and FC components). The outputs of the pretrained model (a three-dimensional vector) are passed through a custom (newly added) FC layer, which yields the output (a two-dimensional vector). The weights of the FC layer are labeled – for instance, the connection between the top node of the pretrained model’s output and the bottom node of the custom FC layer is 4. For the purposes of simplicity, ignore the bias term in each node.

The hacker has access only to the pretrained model, not the custom FC layer and any operations that process the output of the pretrained model.

Say that the hacker wanted to construct an input that would trigger a certain output neuron, like the bottom neuron in the custom FC layer. The hacker can construct an input such that the top neuron output of the pretrained model is some very large number and all the other output neurons are 0. When multiplied by the corresponding weights, the outputs of the complete model yield 10 and 40; passing through the softmax activation function magnifies this difference, and the bottom output neuron of the complete model is the final decision of the model, as desired, as it has the largest associated value/probability (Figure 2-26).

Alternatively, if the hacker wanted to activate the top output neuron of the complete model, it could design an input such that the output neurons of the pretrained model were all zero except for the bottom one, which would be an arbitrarily large number (Figure 2-27).

However, because the hacker does not know the weights outside of the pretrained model in the custom FC layer, they can perform a brute-force search to attempt to force the model to output a certain result. In this search, the hacker aims to construct an input such that the output of the pretrained model is [a, 0, …, 0, 0] the first time, [0, a, …, 0, 0] the second time, and so on until the last time, [0, 0, …, 0, a], where a is some arbitrarily large number. For each desired output, some node in the complete network’s output will be activated. The hacker simply records which one of the complete network’s output nodes is activated for each designed output of the pretrained model.

It should be noted that for this strategy to be capable of activating every output node of the complete model, for each output node of the complete model, there must exist a connection from an output node of the pretrained model whose weight is larger than the connection from the output node of that pretrained model to any other output node in the complete model. This may not be true in more complex datasets, especially heavy multiclass problems, but in most networks, this strategy works to “force” the network to predict almost any desired result.

Because the problem has now been boiled down to finding the input to the model that will lead to a certain output of the pretrained component, the hacker can use methods like gradient descent to solve the following optimization problem: adjust the values of the input image such that the output of the pretrained model can as closely match the desired output (minimize error between the output of the pretrained model and the desired output). In this case, the desired output is some case where all the output nodes except for one are zero, and the one nonzero node is some very large number. The optimization algorithm begins with a standard input image and makes small changes to the image such that the output of the pretrained model is the desired output, constructing an adversarial image. Passing this adversarial image through a model that uses that pretrained model via transfer learning fools it, even though to the human eye no significant changes were made (Figure 2-28).

Making small, specialized distortions to the original input image results in the model making different classifications between the first and second images. Moreover, this could be done without any knowledge of the complete networks’ weights or architectures – an impressive feat.

As the number of target classes increases, the attack needs to make more queries to break all classes, and its effectiveness decreases.

Nevertheless, the effectiveness is high, even as the number of target classes and new layers increases. This study is an interesting investigation into the implications of transfer learning for the deployment of models in real-world applications. It’s important to keep an eye toward deployment and the model’s role in the real world rather than in an airtight experimentation laboratory environment when designing deep learning approaches.

This overview was a simplification of the study; to read more, you can find the paper at https://openreview.net/pdf?id=BylVcTNtDS.

Self-Supervised Learning Case Study: Predicting Rotations

In “Unsupervised Representation Learning by Predicting Image Rotations,” Spyros Gidaris, Praveer Singh, and Nikos Komodakis⁷ propose a simple self-supervised learning method that develops strong representations of the data and yields effective results.

With no access to the original, unrotated image, given an image rotated θ degrees (chosen from θ = 0°, 90°, 180°, 270°), the network must maximize the probability that the image is rotated θ degrees (Figure 2-29).

To visually see the difference in the learned representations for self-supervised learning as opposed to standard supervised learning (only training the model on the task dataset without constructing a pretraining task), Gidaris et al. visualize the attention maps from a model trained with a standard supervised procedure and another trained with a self-supervised procedure (Figure 2-30).

The attention map of a convolutional layer can be found by computing the feature maps for that layer and aggregating them. You can visually see that the attention maps from the self-supervised model focus more on the high-level objects, like the eyes, nose, tails, and heads – in accordance with important features required to perform object recognition.

We’ll use the EfficientNetB4 architecture (not using its weights) to predict which one of four rotations was applied (Listing 2-21). Note that the output of the EfficientNetB4 architecture is 100-dimensional, which is projected via the final layer out into the four desired outputs. A more efficient approach, at first glance, may seem to simply define the number of output classes in the base EfficientNetB4 model to be 4. We will see soon why this design instead is required for this sort of self-supervised learning operation.

After fitting on the altered task dataset, though, we run into a problem: the current model outputs only four values, since the altered task dataset had four unique labels, but the task dataset (i.e., the formal CIFAR-10) dataset has ten classes. This requires an architectural modification.

If we instead had trained a base model with output four classes and attached another output layer with ten classes afterward, which is another technically available option (in that the code runs), performance would likely be poor due to the bottleneck imposed by the four-class output. In this case, think of the EfficientNet model not as outputting 100 classes but as outputting 100 features that are compiled and used to make decisions in the output layer. The base model still contains all the weights learned from self-supervised learning. The new model can then be compiled and trained on the task dataset.

This self-supervised learning approach drastically improves the state-of-the-art results on unsupervised feature learning for a variety of object detection tasks, including ImageNet, PASCAL, and CIFAR-10.

The work of Gidaris et al. points to the existence of simple yet effective approaches to apply self-supervised learning to boost deep learning model performance.

Self-Supervised Learning Case Study: Learning Image Context and Designing Nontrivial Pretraining Tasks

Earlier , we discussed relatively simpler self-supervised learning examples, like adding noise to the task dataset and pretraining the model to predict the degree of noise. Other examples were simple conceptually (although perhaps difficult to implement or accomplish), like pretraining the model to colorize grayscale images. While sometimes simpler self-supervised strategies like predicting the degree of rotation are effective, in other occasions for a bigger performance boost, a more complex pretraining task needs to be used.

Carl Doersch, Abhinav Gupta, and Alexei A. Efros propose a complex, creative, and effective self-supervised learning pretraining task in their 2016 paper, “Unsupervised Visual Representation Learning by Context Prediction.”⁸ Their design is a great case study of the implications and possibilities that need to be considered to design a successful self-supervised pretraining strategy.

In the context prediction pretraining task, the model must predict the relative location, or the context, of one patch to another (Figure 2-31). Patches are small, square-shaped regions of the image. The model is fed a base patch (outlined in blue) and a neighboring patch selected from the eight regions around the base patch (outlined in dotted red) and trained to predict the location of the neighboring patch in relation to the base patch. In the example, the label would be “top left.”

Each general direction that could describe the location of a neighboring patch relative to the base patch is associated with a number. Thus, the model takes in two images (the base patch and the neighboring patch) and outputs a vector indicating which region relative to the base patch the neighboring patch is most likely to be located (Figure 2-32).

To perform this pretraining task, Doersch et al. constructed a multi-input network that takes in two inputs, processes the images through a series of image-based layers in separate “branches,” merges the two branches, processes the merged results, and outputs the location of one input relative to the other (Figure 2-33). Dotted lines in Figure 2-33 indicate shared weights, which means that the weights for the two layers are the same. We’ll discuss multi-input networks, branching, and weight sharing in later chapters.

This pretraining task, at least in theory, requires a deep understanding of the core features of certain items and the high-level relationships between attributes of objects. This makes it difficult, even for humans. Try answering these two visual questions (Figure 2-34).

However, Doersch et al. found that the neural network was learning “trivial solutions” – that is, it was finding shortcuts and easy approaches to the pretraining task that were not related to the concepts Doersch et al. had hoped it would learn.

For instance, low-level cues like boundary patterns or the continuation of textures across patches serve as trivial shortcuts that the neural network can exploit without learning the high-level relationships it would ideally learn. To address this, Doersch et al. added both a gap between patches about half the length of the patch width and a random “jitter” for each patch location by up to 7 pixels. (These are shown visually in Figure 2-32.) With these modifications, the network could not reliably exploit trivial shortcuts like the continuation of textures and lines.

However, another unexpected problem arose: a phenomenon called chromatic aberration, in which light rays pass through a lens focus at different points depending on their wavelength. Thus, in camera lenses that are affected by chromatic aberration, one color – usually green – is “shrunk” toward the center of the image relative to other color channels (red and blue). The network can thus parse the absolute location of patches by detecting the shape and position of the separation between green and magenta (the color comprising the non-green color channels, red and blue). Finding the relative position given knowledge about the absolute location of two patches thus becomes a trivial task.

To address this, Doersch et al. randomly dropped two of three color channels from each patch and replaced the dropped channels with Gaussian noise. This method prevents the neural network from exploiting information in the color channels and instead to focus on the general shapes and edges – the higher-level features – in its pretraining task.

By making modifications to the architecture, like only using one “branch” and adding more convolutional layers, Doersch et al. converted the two-image-to-vector architecture into an image-to-vector architecture. The authors tested this pretrained architecture on the PASCAL Visual Object Classes (VOC) Challenge and found that this self-supervised learning strategy was the best result without using labels not included in the dataset (to their knowledge, and at the time of paper release). The authors found in other datasets that this context prediction task yielded a significant boost to model performance.

This paper is both an example of a well-thought-out self-supervised learning pretraining task and of the process of modifying the pretraining task to accomplish the ultimate objective by discouraging trivial solutions. Self-supervised learning offers us a clever way to find, extract, and utilize rich insights and information without needing to search for costly labels and wandering outside of the dataset.