Human-Level Performance and Bayes Error

In most of the datasets that we use for supervised learning, someone has labelled the observations. Take for example a dataset where we have images that are classified. If we ask people to classify all images (imagine this being possible, regardless of the number of images), the accuracy obtained will never be 100%. Some images maybe too blurry to be classified correctly, and people make mistakes. If, for example, 5% of the images are not classified correctly, due for example to how blurry they are, we must expect that the maximum accuracy people can reach will always be less than 95%.

Let’s consider a problem of classification. First, let’s define what we mean by the word error. We associate the word “error” with the following quantity, indicated with ϵ:

For example, if we reach an accuracy of 95%, we will have ϵ = 1 − 0.95 = 0.05 or expressed as percent ϵ = 5%.

A useful concept to understand is the human-level performance, which can be defined as “the lowest value for the error ϵ that can be reached by a person performing the classification task.” We will indicate it with ϵ_hlp.

Let’s look at a concrete example. Let’s suppose we have a set of 100 images. Now let’s suppose we ask three people to classify the 100 images. Let’s imagine that they obtain 85%, 83% and 84% accuracy. In this case, human-level performance accuracy will be ϵ_hlp = 5%. Note that someone else maybe much better at this task, and therefore it’s always important to consider that the value of ϵ_hlp we get is an estimate and should only serve as a guideline.

Now let’s complicate things a bit. Suppose we are working on a problem in which doctors classify MRI scans in two classes: ones with signs of cancer and ones without. Now let’s suppose we calculate ϵ_hlp from the results of untrained students obtaining 15%, from doctors with a few years of experience obtaining 8%, from experienced doctors obtaining 2%, and from experienced groups of doctors obtaining 0.5%. What is ϵ_hlp in this case? You should always choose the lowest value you can get, for reasons we will discuss later.

We can now expand the definition of ϵ_hlp to a second definition:

Human-level performance is “the lowest value for the error ϵ that can be reached by people or groups of people performing the classification task.”

Now let’s talk a bit about the why we must choose the lowest value we can get for ϵ_hlp. The lowest error that that can be reached by any classifier is called the Bayes error and it’s a very important quantity. We will indicate it with ϵ_Bayes. Usually, ϵ_hlp is very close to ϵ_Bayes, at least in tasks where humans excel, like image recognition. It is commonly said that that human-level performance error is a proxy for the Bayes error. Normally it’s impossible to evaluate ϵ_Bayes and therefore practictioners use ϵ_hlp, assuming the two are close since the latter is easier (relatively) to estimate.

Now keep in mind that it makes sense to compare the two values and assume that ϵ_hlp is a proxy for ϵ_Bayes only if people (or groups of people) perform classification in the same way as the classifier. For example, it’s okay if both use the same images to do classification. But, in the cancer example, if the doctors use additional scans and analysis to make a diagnosis, the comparison is not fair anymore since human-level performance is not a proxy for Bayes error anymore. Doctors, having more data at their disposal, will clearly be better than the model, which has as the input only the images.

Something else that you will notice when working on models is that with relatively low effort you can reach a low error, and often (almost) reach ϵ_hlp. After passing human-level performance (and in several cases that is possible), progress tends to be very slow, as shown in Figure 10-1.

As soon as your algorithm gets better than human-level performance you cannot rely on those techniques anymore. It’s important to get an idea of those numbers to be able to decide what to do to get better results. In our example of MRI scans, we could get better labels by relying on other sources that are not related to humans, such as checking the diagnosis a few years after the MRI time point, when it’s more clear if the patient developed cancer or not. In the case of image classification, you might decide to label a few thousands of images of specific classes. This is not usually possible, but you must be aware that you can get labels using means other than asking humans to perform the same kind of task that your algorithm is performing.

A Short Story About Human-Level Performance

It is instructive to know about the work that Andrej Karpathy has done, while trying to estimate human-level performance in a specific case. You can read the entire story on his blog post (a long post, but worth reading) [1]. What he did is extremely informative about human-level performance. Karpathy was involved in the ILSVRC contest: ImageNet Large Scale Visual Recognition Challenge in 2014 [2]. The task was made up of 1.2 million images (training set) classified in 1000 categories and included objects like animals, abstract objects like a spiral, scenes, and many others. Results were evaluated on a dev dataset. GoogleLeNet (a model developed by Google) reached an astounding 6.7% error. Karpathy asked himself how do humans compare?

The question is a lot more complicated that it may seem at first. Since the images were all classified by humans, shouldn’t ϵ_hlp = 0%? Well, not really. In fact, the images were first obtained with a web search, then filtered and labelled by asking people binary questions: is this a hook or not (for example).

The images were collected, as Karpathy mentioned in his blog post, in a binary way. People were not asked to assign to each image a class by choosing from the 1000 available as the algorithms were doing. You may think that this is a technicality, but the difference in how the labelling occurs makes the correct evaluation of ϵ_hlp quite a complicated matter. So Karpathy set to work and developed a web interface that consisted of an image on the left and the 1000 classes with examples on the right. You can see an example of the interface in Figure 10-2.

You can try the interface [3] to realize how complicated is such a task. People kept missing classes and making mistakes. The best error that was reached was around 15%. So, he did what every scientist at some point in his career have to do: he bored himself out of his mind and did a careful annotation, sometimes needing 20 minutes for a single image. As he formulated in his blog post, he did it only #forscience. He was able to reach a stunning ϵ_hlp = 5.1%. 1.7% better than the best algorithm at the time. He listed sources of errors that GoogLeNet is more susceptible to than humans, such as problems with multiple objects in an image, and sources of errors that humans are more susceptible to than GoogLeNet, such as problems with classes with a huge granularity (dogs are classified in 120 different subclasses).

If you have a few hours to spare, try it. You will get a whole new appreciation of the difficulties of evaluating human-level performance. Defining and evaluating human-level performance is a very tricky task. It is important to understand that ϵ_hlp is dependent on how humans approach the classification task, and is dependent on the time invested, on the patience of the people, and on many factors that are difficult to quantify. The main reason for it being so important, apart from the philosophical aspect of knowing when a machine becomes better than humans, is that it is often taken as a proxy for the Bayes error.

Human-Level Performance on MNIST

Before moving on to the next subject, let’s look at another example of human-level performance on a dataset we have analyzed together: the MNIST dataset. Human-level performance has been widely analyzed and has been found to be ϵ_hlp = 0.2% [4]. Now you may wonder why a human cannot reach a 100% accuracy on simple digits. Look at Figure 10-3 and see if you can say which digits are in the image. You may understand why ϵ_hlp = 0% is not possible, and why a person cannot reach 100% accuracy. Other reasons may be related to which culture people are coming from. In some countries, the digit 7 is written in a very similar way to 1s are for example, and in some cases, mistakes can be made. In other countries, the digit 7 has a small dash along the vertical bar, making it easier to distinguish from a 1.

Bias

Now let’s start with metric analysis: a set of procedures that will give you information on how your model is doing—how good or bad your data is—by looking at your optimized metric evaluated on different datasets.

To start, we need to define a third error: the one evaluated on the training dataset, indicated with ϵ_train.

The first question we want to answer is if our model is not as flexible or complex as needed to reach human-level performance. Or, in other words, we want to know if our model has a high bias with respect to human-level performance.

Let’s define the following quantity

Now there is something else you need to understand. Knowing ϵ_hlp and reducing the bias to reach it are two very different things. Suppose you know the ϵ_hlp for your problem. This does not mean that you need to reach it. It may well be that you are using the wrong architecture, but you may not have the required skills to be able to develop a network sophisticated enough. It may even be that the effort required to reach that error level is prohibitive (in terms of hardware or infrastructure). Always keep in mind what your problem’s requirements are. Always try to understand what is good enough. For an application that recognizes cancer, you may want to invest as much as possible to reach the highest accuracy possible. You do not want to send someone home and discover later that they really do have cancer. On the other hand, if you build a system to recognize cats from web images, you may decide that a higher error than ϵ_hlp is completely acceptable.

Metric Analysis Diagram

This chapter looks at different problems you will encounter when developing your models and how to spot them. We looked at the first one: bias, sometimes also called avoidable bias. We saw how this can be spotted by calculating Δϵ_Bias. At the end of this chapter, you will have a few of those quantities that you can calculate to spot problems.

To make understanding them easier, let’s build a Metric Analysis Diagram (MAD). It is simply a bar diagram, where each bar represents a problem. Let’s start building one with (for the moment) the only quantity we have discussed: bias. You can see it in Figure 10-4. At the moment it is a pretty dumb diagram, but you will see how useful it is to keep things under control when you have several problems at the same time.

Training Set Overfitting

Another problem we have discussed at length in the previous chapters is overfitting the training data . You will remember that while doing regression you can have overfitting if MSE_train ≫ MSE_dev. The same applies in classification problems. Let’s indicate, using ϵ_train, the error our model has on our training dataset and use ϵ_dev for the one on the dev dataset. We can then say we are overfitting the training set if ϵ_train ≫ ϵ_dev. Let’s define a new quantity

With this quantity, we can say we are overfitting the training dataset if Δϵ_{overfitting train} is bigger than a few percent.

Now let’s suppose our model has bias and is slightly overfitting the training dataset, meaning we have Δϵ_Bias = 6% and Δϵ_{overfitting train} = 4%. Our MAD now becomes what is depicted in Figure 10-5.

In Figure 10-5, you can see the relative gravity of the problems we have, and you may decide which one you want to address first.

As usual, there are no fixed rules, and you must determine which techniques work best on your problem by testing .

Test Set

Now let’s quickly mention another problem you may find. Let’s recall how you choose the best model in a machine learning project (this is not specific to deep learning by the way). Let’s suppose you are working on a classification problem. First you decide which optimizing metric you want, and suppose you decide to use accuracy. Then you build an initial system, feed it with training data, and see how it is doing on the dev dataset to see if you are overfitting your training data. You will remember that in previous chapters we talked often about hyper-parameters: parameters that are not influenced by the learning process. Examples of hyper-parameters are the learning rate, regularization parameter, etc. We have seen many of them in the previous chapters.

Let’s say you are working with a specific neural network architecture; you need to search the best values for the hyper-parameters to see how good your model can get. To do that, you train several models with different values of the hyper-parameters and check their performance on the dev dataset. What can happen is that your models work well on the dev dataset but do not generalize at all, since you select only the best values using the dev dataset. You incur in the risk of overfitting the dev dataset with choosing specific values of your hyper-parameters. To see if this is the case, you create a third dataset, called the test dataset, and cut a portion of the observations from your starting dataset. Then you use that test dataset to check the performance of your models.

We must define a new quantity

Where ϵ_test is the error evaluated on the test set. We can add it to the MAD diagram, as shown in Figure 10-6.

Note that if you are not doing a hyper-parameter search, you will not need a test dataset. It is only useful when you are doing extensive searches. Otherwise, in most cases, it’s useless and takes away observations that you may use for training. What we discussed so far assumes that your dev and test sets observations have the same characteristics. If use high-resolution images from a smartphone for training and the dev dataset, and you use images from the web in low resolution for your test dataset, you may see a big |ϵ_dev − ϵ_test|. But that will probably be due to the differences in the images and not because of an overfitting problem. Later in the chapter, we discuss what can happen when different sets come from different distributions (another way of saying that the observations have different characteristics).

How to Split Your Dataset

Now let’s briefly discuss how to split the data.

What exactly does split mean? Well, as we discussed in the previous section, you will need a set of observations to make the model learn, which you call your training set. You will also need a set of observations that will make your dev set and then a final set that is called the test set. Data is typically split so that 60% for the training set, 20% for the dev set, and 20% for test set. Usually, the kind of split is indicated in this form: 60/20/20, where the first number (60) refers to the percentage of the entire dataset that is in the training set, the second (20) to the percentage of the entire dataset that is in the dev set, and the last (20) to the percentage that is in the test set. You may find in books, blogs, or articles, sentences like this “We split our dataset 80/10/10” for example. This is what it means.

In the deep learning field you will deal with big datasets. For example, if we have m = 10⁶, we could use a split like 98/1/1. Keep in mind that 1% of 10⁶ is 10⁴, so still a big number! Remember that the dev/test set must be big enough to give high confidence of the performance of the model, but not unnecessarily big. Additionally, you want to save as many observations as possible for your training set.

You can compare these results to the one from the entire dataset. You will notice that they are very close—not the same, but close enough. In this case, we can proceed without worries. But if this is not the case, be sure you have a similar distribution in every dataset you’re going to use.

If the training, dev, and test datasets don’t have the same distribution, this can be quite dangerous when checking how the model is doing. Your model may end up learning from a unbalanced class distribution.

If you have a dataset with three classes, for example, where you have 1000 observations in each class, then the dataset has a perfectly balanced class distribution, but if you have only 100 observations in class 1, 10,000 observations in class 2, and 5,000 in class 3, then this is an unbalanced class distribution. This is not a rare occurrence. Suppose you need to build a model that recognizes fraudulent credit card transactions. It is safe to assume that those transactions are a very small percentage of the entire number of transactions that you have at your disposal!

Details on how to deal with unbalanced datasets are beyond the scope of this book, but is important to understand what kind of consequences this can have. In the next section, you learn what can happen if you feed an unbalanced dataset to a neural network. At the end of that section, we give you a few hints on what to do in such a case.

Unbalanced Class Distribution: What Can Happen

Since we are talking about how to split our dataset to perform metric analysis , it’s important to grasp the concept of unbalanced class distribution and know how to deal with it. In deep learning, you will find yourself very often splitting datasets and you should be aware of the problems you may encounter if you do it the wrong way. Let’s look at a concrete example of how bad things can go if you do it wrong.

We will use the MNIST dataset [5], and we will do basic logistic regression with a single neuron. The MNIST database is a large database of handwritten digits that we can use to train our model. The MNIST database contains 70,000 images.

“The original black and white (bilevel) images from MNIST were normalized to fit in a 20x20 pixel box while preserving their aspect ratio. The resulting images contain gray levels as a result of the anti-aliasing technique used by the normalization algorithm. The images were centered in a 28x28 image by computing the center of mass of the pixels and translating the image so as to position this point at the center of the 28x28 field” [5].

Our features will be the gray value for each pixel, so we will have 28x28 = 784 features whose values go from 0 to 255 (gray values). The dataset contains all ten digits, from 0 to 9. With the following code, you can prepare the data to use in the following sections. As usual, let’s first import the necessary library.

# general libraries

import pandas as pd

import numpy as np

import matplotlib

import matplotlib.pyplot as plt

import matplotlib.font_manager as fm

# sklearn libraries

from sklearn.metrics import confusion_matrix

# tensorflow libraries

import tensorflow as tf

from tensorflow import keras

from tensorflow.keras import layers

import tensorflow_docs as tfdocs

import tensorflow_docs.modeling

First, we load the data

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()

It is useful to define a function to visualize the digits, to get an idea of how they look.

def plot_digit(some_digit):

  plt.imshow(some_digit, cmap = matplotlib.cm.binary, interpolation = "nearest")

  plt.axis("off")

  plt.show()

For example , we can plot one randomly (see Figure 10-7).

plot_digit(x_train[36003])

Now that we have inspected the dataset a bit, here comes the important part. We create a new label. We assign to all observations for the digit 1 the label 0, and to all other digits (0, 2, 3, 4, 5, 6, 7, 8, and 9) the label 1 with this code

y_train_unbalanced = np.zeros_like(y_train)

y_train_unbalanced[np.any([y_train == 1], axis = 0)] = 0

y_train_unbalanced[np.any([y_train != 1], axis = 0)] = 1

y_test_unbalanced = np.zeros_like(y_test)

y_test_unbalanced[np.any([y_test == 1], axis = 0)] = 0

y_test_unbalanced[np.any([y_test != 1], axis = 0)] = 1

The y_train_unbalanced and y_test_unbalanced arrays will contain the new labels. Note that the dataset is now heavily unbalanced. Label 0 appears roughly 10% of the time, while label 1 appears 90% of the time.

We then reshape (to provide the data in the right dimension required by the neural network) and normalize the training and the test data

x_train_reshaped = x_train.reshape(60000, 784)

x_test_reshaped = x_test.reshape(10000, 784)

x_train_normalised = x_train_reshaped/255.0

x_test_normalised = x_test_reshaped/255.0

Then we build our network with a single neuron

def build_model():

  # one unit as network's output

  # sigmoid function as activation function

  # sequential groups a linear stack of layers into a

  # tf.keras.Model

  # activation parameter: if you don't specify anything, no

  # activation

  # is applied (i.e. "linear" activation: a(x) = x).

  model = keras.Sequential([

    layers.Dense(1,

                 input_shape = [len(x_train_normalised[0])],

                 activation = 'sigmoid')

  ])

  # optimizer that implements the Gradient Descent algorithm

  optimizer = tf.keras.optimizers.SGD(momentum = 0.0,

                                      learning_rate = 0.0001)

  # the compile() method takes a metrics argument, which can be a list of metrics

  # loss = cross-entropy, metrics = accuracy,

  model.compile(loss = 'binary_crossentropy',

                optimizer = optimizer,

                metrics = ['binary_crossentropy',

                           'binary_accuracy'])

  return model

At this point in the book, you should understand this simple model easily, as you have seen it several times. Now we define the function to run the model

model = build_model()

Let’s run the model with the code

EPOCHS = 50

history = model.fit(

  x_train_normalised, y_train_unbalanced,

  epochs = EPOCHS, verbose = 0, batch_size = 1000,

  callbacks = [tfdocs.modeling.EpochDots()])

And check the accuracy with the code

hist = pd.DataFrame(history.history)

hist['epoch'] = history.epoch

hist.tail()

      loss        binary_crossentropy    binary_accuracy    epoch

45    0.292092    0.292092               0.887617           45

46    0.290531    0.290531               0.887633           46

47    0.289016    0.289016               0.887633           47

48    0.287542    0.287542               0.887633           48

49    0.286109    0.286109               0.887633           49

You get a very good 88.1% accuracy . Not bad right? But are you sure that the result is that good? Now let’s check the confusion matrix¹ for our labels with the code

train_predictions = model.predict(x_train_normalised).flatten()

confusion_matrix(y_train_unbalanced, train_predictions > 0.5)

When you run the code, you get the following result

array([[    0,  6742],

       [    0, 53258]])

Slightly better formatted and with some explanatory information, the matrix looks like Table 10-1.

Table 10-1

Confusion Matrix for the Model Described in the Text

	Predicted Class 0	Predicated Class 1
Real class 0	0	6742
Real class 1	0	53258

How should we read this table? In the “Predicted Class 0” column, you will see the number of observations that our model predicts of being of class 0 for each real class. 0 is the number of observations our model predicts of being of class 0 and are really in class 0. 0 is the number of observations that our model predicts in class 0 but are really in class 1.

It should be easy to see that our model effectively predicts all observations to be in class 1 (a total of 6742+53258 = 60000). Since we have a total of 60,000 observations in our training set, we get an accuracy of 53258/60000 = 0.887, as our Keras code told us. But not because our model is good, simply because it has effectively classified all observations in class 1. We don’t need a neural network in this case to reach this accuracy. What happens is that our model sees observations belonging to class 0 so rarely that they almost don’t influence the learning, which is dominated by the observations in class 1.

What at the beginning seemed a nice result , turns out is a really bad one. This is an example of how badly things can go if you don’t pay attention to the distributions of your classes. This of course applies not only when splitting your dataset, but in general when you approach a classification problem, regardless of the classifier you want to train (it does not apply only to neural networks).

Tip

When splitting your dataset into complex problems, you need to pay close attention not only to the number of observations you have in your datasets, but also to what observations you choose and to the distribution of the classes.

To conclude this section, let me give you a few hints about how to deal with unbalanced datasets:

• Change your metric: In our example, you may want to use something else instead of accuracy, since it can be misleading. You could try using the confusion matrix for example, or other metrics like precision, recall, or F1 (review them if you don’t know them, as they are very important to know). Another important way of checking how your model is doing, and one that I suggest you learn, is the ROC Curve, which will help you tremendously.

• Work with an undersampled dataset: If you have for example 1000 observations in class 1 and 100 in class 2, you may create a new dataset with 100 random observations in class 1 and the 100 you have in class 2. The problem with this method is that you will have a lot less data to feed to your model to train it.

• Work with an oversampled dataset: You may try to do the opposite. You may take the 100 observations in class 2 and replicate them ten times to end up with 1000 observations in class 2 (sometimes called sampling with replacement).

• Get more data in the class with less observations: This is not always possible. In the case of fraudulent credit card transactions, you cannot go around and generate new data, unless you want to go to jail .

Datasets with Different Distributions

Now I want to discuss another terminology issue, which will lead us to understanding a common problem in the deep learning world. Very often you will hear sentences like: “the sets come from different distributions.” This sentence is not always easy to understand. Take for example two datasets formed by images taken with a professional DSLR, and a second one created by images taken with an old smartphone. In the deep learning world, we say that those two sets come from different distributions. But what is the meaning of the sentence? The two datasets differ for various reasons: resolution of images, blurriness due to different quality of lens, amount of colors, how much is in focus, and possibly more. All those differences are what is usually meant by different “distributions.”

Let’s look at another example. We could consider two datasets: one made of images of white cats and one made of images of black cats. Also, in this case we talk about different distributions. This becomes a problem when you train a model on one set and want to apply it to the other. For example, if you train a model on a set of images of white cats, you probably are not going to do very good on the dataset of black cats, since your model has never seen black cats during training.

Since data is precious, people often try to create the different datasets (train, dev, etc.) from different sources. For example, you may decide to train your model on a set made of images taken from the web and see how good it is on a set made of images you took with your smartphone. This may seem a nice idea to be able to use as much data as possible, but it may give you many headaches. Let’s see what happens in a real case so that you see the consequences of doing something like this.

We have 12,700 observations in the training dataset and 2167 in the test dataset (we will call it dev dataset from now on in this chapter).

To make the shift easy, we first reshaped the images in a 28x28 matrix, then simply shifted the columns with tmp_shifted[:,10:28] = tmp[:,0:18]. Then we simply reshape the images in a one-dimensional array of 784 elements. The labels remain the same. Figure 10-8 shows a random image from the dev dataset on the left and its shifted version on the right.

What has happened is that the model has learned from a dataset where all the images are centered in the box and therefore could not generalize images shifted and not centered.

When training a model on a dataset, you will get good results usually on observations that are like the ones in the training set. But how can you find out if you have such a problem? There is a relatively easy way of doing that, by expanding the MAD diagram. Let’s see how to do it.

Suppose you have a training dataset and a dev dataset where the observations have different characteristics (they are coming from different distributions). You create a small subset from the training set, called the train-dev dataset, and end up with three datasets—a training and a train-dev coming from the same distribution (the observations have the same characteristics) and a dev set, where the observations are somehow different, as we discussed previously. Then you train your model on your training set, and evaluate your error ϵ on the three datasets: ϵ_train, ϵ_dev and ϵ_{train − dev}. If your train and dev sets are coming from the same distributions, so is the train-dev set. In this case you should expect ϵ_dev ≈ ϵ_{train − dev}. If we define

we should expect Δϵ_{train − dev} ≈ 0. If the train (and train-dev) and the dev set are coming from different distributions (the observations have different characteristics), we should expect Δϵ_{train − dev} to be big. If we consider the MNIST example, we have in fact Δϵ_{train − dev} = 0.46 or 46%, which is a huge difference.

Figure 10-9 is an example of the MAD diagram with the problem we just discussed. Don’t look at the numbers; they are just for illustrative purposes (read: I just put them there).

Note that you don’t need to create the bar plot as we have done here. Technically, you just need the four numbers to draw the same conclusions.

As usual, there are no fixed rules. Just be aware of the problem and think about the following: your model will learn the characteristics from your training data, so when it’s applied to completely different data, it won’t usually do well. Always get training data that reflects the data you want your model to work on, not vice versa.

k-fold Cross Validation

A typical value for k is 10, but that depends on the size of your dataset and on the characteristic of your problem.

Remember that the discussion about how to split a dataset also applies here.

Although this may seem a very attractive technique to deal generally with datasets with less than optimal size, it may be quite complex to implement. But, as you will see shortly, checking your metric on the different folds will give you important information on a possible overfitting of your training dataset.

Let’s try this on a real dataset and see how to implement it. Note that you can implement k-fold cross validation easily in sklearn, but I will develop it from scratch, to show you what is happening in the background. Everyone (well, almost) can copy code from the web to implement k-fold cross validation in sklearn, but not many can explain how it works and understand it, and therefore be able to choose the right sklearn method or parameters.

As a dataset we will use the reduced MNIST dataset containing only digits 1 and 2. We will do a simple logistic regression with one neuron to make the code easy to understand and to let you concentrate on the cross-validation part and not on other implementation details that are not relevant here. The goal of this section is to show you how k-fold cross validation works and why it’s useful, not to implement it with the smallest amount of code possible.

As before, import the MNIST dataset. Remember that the dataset has 70,000 observations and is made of grayscale images, each 28x28 pixel in size. Then select only digit 1 and 2 and rescale the labels to make sure that digit 1 has label 0 and digit 2 has label 1.

Now we need a small trick. To keep the code simple, we will consider only the training dataset and we will create folds starting from it.

You could normalize the data in one shot, but we like to make it evident that we are dealing with folds to make it clear for the reader.

Notice that you will get slightly different accuracy values for each fold. It is very instructive to study how the accuracy values are distributed. Since we have ten folds, we have ten values to study. In Figure 10-10, you can see the distribution of the values for the train set (left plot) and for the dev set (right plot).

The image is quite instructive. You can see that the accuracy values for the training set are concentrated around the average, while the ones evaluated on the dev set are much more spread out! This shows how the model on new data behaves less well than on the data it has trained on. The standard deviation for the training data is 3.5 · 10⁻⁴ and for the dev set it’s 3.5 · 10⁻³, ten times bigger than the value on the train set. In this way you also get an estimate of the variance of your metric when applied to new data, and to how it generalizes.

If you are interested in learning how to do this quickly with sklearn, check out the official documentation for the KFold method [6]. When you are dealing with datasets with many classes (remember our discussion about how to split your sets?), you must pay attention and do what is called stratified sampling. Sklearn provides a method to do that too: stratifiedKFold [7].

You can now easily find averages and standard deviations. For the training set, we have an average accuracy of 98.5% and a standard deviation of 0.035%, while for the dev set we have an average accuracy of 98.4% with a standard deviation of 0.35%. Now you can even provide an estimate of the variance of your metric. Pretty cool!

Manual Metric Analysis: An Example

We mentioned earlier that sometimes it’s useful to do a manual analysis of your data, to check if the results (or the errors) you are getting are plausible. This section has a basic example to shows you a concrete idea of what is meant by this and how complicated it can be. Let’s consider the following question: our very simple model (remember we are using only one neuron) can get 98% of accuracy! Is the problem of recognizing digits that easy? Let’s try to see if that is the case. First of all, note that our training set does not even have the two-dimensional information about the images. If you remember, each image is converted into a one-dimensional array of values: the gray values of each pixel, starting on the top-left and going row by row from top to bottom. Are the 1s and the 2s easy to recognize? Let’s see how the real input for our model looks and start analyzing the digit 1. Let’s take an example from fold 0. In Figure 10-11, you can see the image on the left and a bar plot of the gray values of the 784 pixels as they are seen from our model. Remember that as observations, we have a one-dimensional array of the 784 gray values of the pixels in the image.

Remember that we reshape our 28x28 image into a one-dimensional array, so when reshaping the digit 1 in Figure 10-11, we find black points roughly each 28 pixels, since the 1 is almost a vertical column of black points. In Figure 10-12, you can see other 1s, and you will notice how, when reshaped as one dimensional, they look all the same: several bars roughly equally spaced. Now that you know what to look for, we can easily say that all the images in Figure 10-12 are of the digit 1.

Now let’s look at the digit 2. In Figure 10-13, you can see an example, similar to what we had in Figure 10-11.

Now things look different. We have two regions where the bars are much more dense in the plot on the right in Figure 10-13. This is between pixels 100 and 200 and especially after pixel 500. Why? Well, the two areas correspond to the two horizontal parts of the image. Figure 10-14 highlights how different parts look when they’re reshaped as a one-dimensional array.

Horizontal parts (A) and (B) are clearly different than part (C) when reshaped as a one-dimensional array. The vertical part (C) looks like the digit 1, with many equally spaced bars, as you can check in the lower-right bar plot labelled (C). While the more horizontal parts appear as many bars clustered together in groups, as can be seen in the upper-right and lower-left bar plots labelled (A) and (B). So, when reshaped, if you find those cluster of bars, that means you are looking at a 2. If you see only equally spaced small groups of bars, as in the plot (C) in Figure 10-14, you are looking at a 1. You don’t even need to see the two-dimensional image if you know what to look for. Note that this pattern is constant. Figure 10-15 shows four examples of the digit 2, and you can clearly see the wider clusters of bars.

As you can imagine, this is an easy pattern to spot for an algorithm, and so it’s to be expected that this model works so good. Even a human can spot the images, even when reshaped, without any effort. Such a detailed analysis would not be necessary in a real-life project, but it’s really instructive to see what you can learn from your data. Understanding the characteristics of your data may help you design your model or understand why is not working. Advanced architectures, like convolutional networks, will be able to learn exactly those two-dimensional features in a very efficient way.

Let’s also check how the network learned to recognize digits. You will remember that the output of our neuron is

Where σ is the sigmoid function, x_i for i = 1, …, 784 are the gray values of the pixel of the image, w_i for i = 1, …, 784 are the weights and b is the bias. Remember that when we classify the image in class 1 (so digit 2), and if we classify the image in class 0 (so digit 1). Now if you remember our discussion in Chapter 2 of the sigmoid function you will remember that σ(z) ≥ 0.5 when z ≥ 0 and σ(z) < 0.5 for z < 0. That means that our network should learn the weights in such a way that for all the 1s, we have z < 0 and for all the 2s, z ≥ 0. Let’s see if that is really the case.

In Figure 10-16, you can see a plot for a digit 1 where you can find the weights w_i (as a solid line) of our trained network after 600 epochs (and after reaching an accuracy of 98%) and the gray value of the pixel x_i rescaled to have a maximum of 0.5 (as a dashed line). Look how each time x_i is big, then w_i is negative. And when w_i > 0 the x_i are almost zero. Clearly the result will be negative, and therefore σ(z) < 0.5 and the network will identify the image as a 1. Figure 10-16 is zoomed in to make this behavior more evident.

In Figure 10-17, you can see the same plot for a digit 2. You will remember from our previous discussion that, for a 2, we can see many bars clustered together in groups up to pixel 250 (roughly). Let’s look at the weights in that region. Where the pixel gray values are big, the weights are positive, giving then a positive value of z and therefore σ(z) ≥ 0.5. That means the image would be classified as a 2.

As an additional check, Figure 10-18 plots w_i · x_i for all values of i for a digit 1. You can see how almost all the points lie below zero. Note that b =  − 0.16 in this case as well.

As you can see, in very easy cases it is possible to understand how a network learns and therefore it would be much easier to debug strange behaviors. But don’t expect this to be possible when dealing with more complex cases. The analysis we did here would not be so easy, for example, if you tried to do the same with digits 3 and 8 instead of 1 and 2.

Exercises

References