Human-Level Performance and Bayes Error

In most of the datasets that we use for supervised learning, someone must have labeled the observations. Take, for example, a dataset in which we have images that are classified. If we ask people to classify all images (imagine this being possible, regardless of the number of images), the accuracy obtained will never be 100%. Some images may be too blurry to be classified correctly, and people make mistakes. If, for example, 5% of the images are not classifiable correctly, owing, for example, to how blurry they are, we must expect that the maximum accuracy people can reach will always be less than 95%.

Let’s consider a classification problem. First, let’s define what we mean by the word error. In this chapter, the word error will be used to indicate the following quantity, represented by ϵ:

For example, if, with a model, we achieve an accuracy of 95%, we will have ϵ = 1 − 0.95 = 0.05 or, expressed as a percent, ϵ = 5%.

A useful concept to understand is human-level performance, which can be defined as follows:

Human-level performance ( definition 1) : The lowest value for the error ϵ that can be achieved by a person performing the classification task. We will indicate it with ϵ_hlp.

Let’s devise a concrete example. Suppose we have a set of 100 images. Now let’s suppose we ask three people to classify the 100 images. Imagine that they obtain 95%, 93%, and 94% accuracy. In this case, human-level performance accuracy will be ϵ_hlp = 5%. Note that someone else may be much better at this task, and, therefore, it is always important to consider that the value of ϵ_hlp we get is always an estimate and should only serve as a guideline.

Now let’s complicate things a bit. Suppose we are working on a problem in which doctors classify MRI scans in two classes: with signs of cancer and without. Now let’s suppose we calculate ϵ_hlp from the results of untrained students obtaining 15%, from doctors with a few years of experience obtaining 8%, from experienced doctors obtaining 2%, and from experienced groups of doctors obtaining 0.5%. What then is ϵ_hlp? You should always choose the lowest value you can get, for reasons I will discuss later.

We can now expand the definition of ϵ_hlp with a second definition.

Human level performance ( definition 2) : The lowest value for the error ϵ that can be achieved by people or groups of people performing the classification task

Now I’ll talk a bit about why we must choose the lowest value we can get for ϵ_hlp. Suppose that of the 100 images, 9 are too blurry to be correctly classified. This means that the lowest error any classifier will be able to reach is 9%. The lowest error that can be reached by any classifier is called the Bayes error . We will indicate this with ϵ_Bayes. In this example, ϵ_Bayes = 9%. Usually, ϵ_hlp is very close to ϵ_Bayes, at least in tasks at which humans excel, such as image recognition. It is commonly said that human-level performance error is a proxy for the Bayes error. Normally it is impossible or very hard to know ϵ_Bayes, and, therefore, practitioners use ϵ_hlp assuming the two are close, because the latter is easier (relatively) to estimate.

Keep in mind that it makes sense to compare the two values and assume that ϵ_hlp is a proxy for ϵ_Bayes only if persons (or groups of persons) perform classification in the same way as the classifier. For example, it is OK if both use the same images to do classification. But, in our cancer example, if the doctors use additional scans and analysis to diagnose cancer, the comparison is no longer fair, because human-level performance will not be a proxy for a Bayes error anymore. Doctors, having more data at their disposal, clearly will be more accurate than the model that has as input only the images at its disposal.

Something else that you will notice when working on models is that with relatively little effort, you can achieve a quite low rate of error and often (almost) reach ϵ_hlp. After passing human-level performance (and, in several cases, that is possible), progress tends to be very, very slow, as is illustrated in Figure 6-1.

As soon as your algorithm exceeds human-level performance, you cannot rely on those techniques anymore. So, it is important to get an idea of those numbers, to decide what to do to obtain better results. Taking our example of MRI scans, we could get better labels by relying on sources that are not related to humans, for example, checking diagnoses a few years after the date of the MRI, when it is usually clear whether a patient has developed cancer. Or, for example, in the case of image classification, you may decide yourself to take a few thousands of images of specific classes. This is not usually possible, but I wanted to make the concept clear: you can get labels by means other than by asking humans to perform the same kind of task that your algorithm is performing.

A Short Story About Human-Level Performance

I want to tell you a story about the work that Andrej Karpathy has done while trying to estimate human-level performance in a specific case. You can read the entire story on his blog post (a long post, but one that I suggest you read) at https://goo.gl/iqCbC0 . Let me summarize what he did, since it is extremely instructive concerning what human-level performance really is. Karpathy was involved in the ILSVRC (ImageNet Large Scale Visual Recognition Challenge) in 2014 ( https://goo.gl/PCHWMJ ). The task was made up of 1.2 million images (training set) classified in 1000 categories, including such objects as animals, abstract objects such as a spiral, scenes, and many more. Results were evaluated on a dev dataset. GoogleLeNet (a model developed by Google) reached an astounding 6.7% error. Karpathy wondered how humans would compare.

The question is a lot more complicated than it may seem at first sight. Because the images were all classified by humans, shouldn’t ϵ_hlp = 0%? Well, actually, no. In fact, the images were first obtained with a web search, then they were filtered and labeled by asking people binary questions, for example, Is this a hook or not? The images were collected, as Karpathy mentions in his blog post, in a binary way. People were not asked to assign to each image a class, choosing from the 1000 available, as the algorithms were doing. You may think that this is a technicality, but the difference in how the labeling occurs makes the correct evaluation of a ϵ_hlp quite a complicated matter. So, Karpathy set to work and developed a web interface that consisted of an image on the left, and the 1000 classes with examples on the right. You can see an example of the interface in Figure 6-2. You can try the interface (and I suggest you do so) at https://goo.gl/Rh8S6g , to understand how complicated such a task is. People trying the interface kept missing classes and making mistakes. The best error that was reached was about 15%. So, Karpathy did what every scientist at some point in his/her career must do: he bored himself to death and did a careful annotation himself, sometimes requiring 20 minutes for a single image. As he states in his blog post, he did it only #forscience. He was able to reach a stunning ϵ_hlp = 5.1%, 1.7% better than the best algorithm at the time. He listed sources of errors to which GoogLeNet is more susceptible than humans, such as problems with multiple objects in an image, and sources of errors to which humans are more susceptible than GoogLeNet, such as problems with classes with a huge granularity (dogs are classified in 120 different subclasses, for example).

If you have a few hours to spare, I suggest you try. You will gain a whole new appreciation of the difficulties of evaluating human-level performance. Defining and evaluating human-level performance is a very tricky task. It is important to understand that ϵ_hlp is dependent on how humans approach the classification task, which is dependent on the time invested, the patience of the persons performing the task, and on many factors that are difficult to quantify. The main reason for it being so important, apart from the philosophical aspect of knowing when a machine becomes better than humans, is that it is often taken as a proxy for the Bayes error, which gives a lower limit of our possibilities.

Human-Level Performance on MNIST

Before moving on to the next subject, I would like to give you another example of human-level performance on a dataset we have analyzed together: the MNIST dataset. Human-level performance has been widely analyzed, and it has been found that ϵ_hlp = 0.2%. (You can read a good review on the subject by Dan Cireşan: “Multi-column Deep Neural Networks for Image Classification,” Technical Report No. IDSIA-04-12, Dalle Molle Institute for Artificial Intelligence, https://goo.gl/pEHZVB .) Now you may wonder why a human cannot achieve 100% accuracy classifying simple digits, but see Figure 6-3, and attempt to identify which digits are in the images. I surely cannot. You may, therefore, better understand why ϵ_hlp = 0% is not possible, and why a person cannot achieve 100% accuracy. Other reasons may be related to which culture people come from. In some countries, the digit representing seven is written in a very similar way to that for ones, for example, and in some cases, mistakes can be made. In other countries, the digit seven has a small dash along the vertical bar, making it easier to distinguish from a one.

Bias

Now let’s start with a metric analysis: a set of procedures that will give you information on how your model is doing and how good or bad your data is, by evaluating your optimizing metric on different datasets.

To start, we must first define a third error: the one evaluated on the training dataset, indicated with ϵ_train.

The first question we want to answer is if our model is not as flexible or complex as needed to reach human-level performance. Or, in other words, we want to know if our model has a high bias, with respect to human-level performance.

To answer the previous question, we can do the following: calculate the error from our model from our training dataset ϵ_train and then calculate |ϵ_train − ϵ_hlp|. If the number is not small (bigger than a few percent), we are in the presence of bias (sometimes called avoidable bias), that is, our model is too simple to capture the real subtleties of our data.

Let’s define the following quantity

There is something else you need to understand. Knowing ϵ_hlp and reducing the bias to reach it are two very different things. Suppose you know the ϵ_hlp for your problem. This does not mean that you have to reach it. It may well be that you are using the wrong architecture, but you may not have the skills required to develop a more sophisticated network. It may even be that the effort required to achieve the desired error level would be prohibitive (in terms of hardware or infrastructure). Always keep in mind what your problem requirements are. Always try to understand what is good enough. For an application that recognizes cancer, you may want to invest as much as possible to achieve the highest accuracy possible: you don’t want to send someone home only to discover the presence of cancer months later. On the other hand, if you build a system to recognize cats from web images, you may find a higher error than ϵ_hlp completely acceptable.

Metric Analysis Diagram

In this chapter, we look at different problems that you will encounter when developing your models and how to spot them. We have looked at the first one: bias, sometimes also called avoidable bias. We have seen how this can be spotted by calculating Δϵ_Bias. At the end of this chapter, you will have a few of those quantities that you can calculate to spot problems. To make understanding them easier, I use what I like to call the metric analysis diagram (MAD). It is simply a bar diagram, in which each bar represents a problem. Let’s start to build one with (for the moment) the only quantity we have discussed: bias. You can see it in Figure 6-4. At the moment, it is a pretty dumb diagram, but you will see how useful it is to keep things under control when you have several problems at the same time.

Training Set Overfitting

Another problem we have discussed at length in the previous chapters is overfitting of training data. You will remember in Chapter 5, while executing regression, we saw an extreme case of overfitting, in which MSE_train ≪ MSE_dev. The same applies in classification problems. Let’s indicate with ϵ_train the error our model has on our training dataset and with ϵ_dev the one on the dev dataset. We can then say we are overfitting the training set if ϵ_train ≪ ϵ_dev. Let’s define a new quantity

With this quantity, we can say that we are overfitting the training dataset if Δϵ_{overfitting train} is bigger than a few percent.

Now let’s suppose our model has bias and is slightly overfitting the training dataset, meaning we have Δϵ_Bias = 6% and Δϵ_{overfitting train} = 4%. Our MAD now becomes what is depicted in Figure 6-5.

As you can see in Figure 6-5, you can have a quick overview of the relative gravity of the problems we have, and you may decide which one you want to address first.

As usual, there are no fixed rules, and you must test which techniques work best on your problem.

Test Set

I would like to quickly mention another problem you may encounter. We will look at it in detail in Chapter 7, because it is related to hyperparameter search. Recall how you choose the best model in a machine-learning project (this is not specific to deep learning, by the way)? Let’s suppose we are working on a classification problem. First, we decide which optimizing metric we want, let’s suppose we decide to use accuracy. Then we build an initial system, feed it with training data, and see how it is doing on the dev dataset, to check if we are overfitting our training data. You will remember that in previous chapters, we have talked often about hyperparameters—parameters that are not influenced by the learning process. Examples of hyperparameters are the learning rate, regularization parameter, etc. We have seen many of them in the previous chapters. Let’s say you are working with a specific neural network architecture. You need to search the best values for the hyperparameters, to see how good your model can get. To do this, you train several models with different values of the hyperparameters and check their performance on the dev dataset. What can happen is that your models work well on the dev dataset but don’t generalize at all, because you select the best values using only the dev dataset. You incur the risk of overfitting the dev dataset by choosing specific values for your hyperparameters. To check if this is the case, you create a third dataset, called the test dataset, cutting a portion of the observations from your starting dataset, which you use to check the performance of your models.

We must define a new quantity

where ϵ_test is the error evaluated on the test set. We can add it to our MAD diagram (Figure 6-6).

Note that if you are not doing any hyperparameter search, you will not need a test dataset. It is only useful when you are doing extensive searches; otherwise, in most cases, it is useless and takes away observations that you may use for training. What we discussed so far assumes that your dev and test set observations have the same characteristics. For example, if you are working on an image recognition problem and you decide to use images from a smartphone with high resolution for training and the dev dataset, and images from the Web in low resolution for your test dataset, you may see a big |ϵ_dev − ϵ_test|, but that will probably be owing to the differences in the images and not to an overfitting problem. I will discuss later in the chapter what can happen when different sets come from different distributions (another way of saying that the observations have different characteristics), what exactly this means, and what you can do about it.

How to Split Your Dataset

Now I would like to discuss briefly how to split your data in both a general and deep-learning context.

But what exactly does “split” mean? Well, as discussed in the previous section, you will require a set of observations to make the model learn, which you call your training set. You also will need a set of observations that will constitute your dev set, and a final set called the test set. Typically, you would see splits such as 60% of observations for the training set, 20% of observations for the dev set, and 20% of observations for the test set. Usually, these kinds of splits are indicated in the following form: 60/20/20, where the first number (60) refers to the percentage of the entire dataset that makes up the training set, the second (20) to the percentage of the entire dataset that makes up the dev set, and the last (20) to the percentage that makes up the test set. In books, blogs, or articles, you may encounter sentences such as “We will split our dataset 80/10/10.” You now have an explanation of what this means.

Usually, in the deep-learning field, you will deal with big datasets. For example, if we have m = 10⁶, we could use a split such as 98/1/1. Keep in mind that 1% of 10⁶ is 10⁴—a big number! Remember that the dev/test set must be big enough to give high confidence to the performance of the model, but not unnecessarily big. Additionally, you will want to save as many observations as possible for your training set.

Do you notice anything different? The biggest difference is that now, class 9 is only appearing in 2.8% of the cases. Before, it was appearing in 9.9% of the cases. Apparently, our hypothesis that the classes are distributed according to a random uniform distribution was not right. This can be quite dangerous when checking how the model is doing or because your model may end up learning from a so-called unbalanced class distribution .

If you have a dataset with three classes, for example, where you have 1000 observations in each class, then the dataset has a perfectly balanced class distribution, but if you have in class 1 only 100 observations, in class 2 10,000 observations, and in class 3 5000, then we talk about an unbalanced class distribution. You should not think that this is a rare occurrence. Suppose you have to build a model that recognizes fraudulent credit card transactions. It is safe to assume that those transactions are a very small percent of the entire amount of transactions that you will have at your disposal.

To go into details on how to deal with unbalanced datasets would be beyond the scope of this book, but it is important to understand what kind of consequences they may have. In the next section, I will show you what can happen if you feed an unbalanced dataset to a neural network, so that you gain a concrete understanding of the possibility. At the end of the section, I will offer a few hints on what to do in such a case.

Unbalanced Class Distribution: What Can Happen

Because we are talking about how to split our dataset to perform metric analysis, it is important to grasp the concept of unbalanced class distribution and how to deal with it. In deep learning, you will find yourself very often splitting datasets, and you should be aware of the problems you may encounter if you do this in the wrong way. Let me give you a concrete example of how bad things can go if you do it wrongly.

How should we read the table? In the column “Predicted Class 0,” you will see the number of observations that our model predicts as being of class 0 for each real class. 659 is the number of observations our model predicts as being of class 0 that are really in class 0. 6 is the number of observations that our model predicts in class 0 that are really in class 1.

It should be easy to see now that our model predicts effectively almost all observations to be in class 1 (a total of 4888 + 50,503 = 55,391). The number of correct classified observations is 659 (for class 0) and 50,503 (for class 1), for a total of 51,162 observations. Because we have a total of 56,056 observations in our training set, we get an accuracy of 51162/56056 = 0.912, as our TensorFlow code above told us. This is not because our model is good; it is simply because it has classified effectively all observations in class 1. In this case, we don’t need a neural network to achieve this accuracy. What happens is that our model sees observations belonging to class 0 so rarely that it almost doesn’t influence the learning, which is dominated by the observations in class 1.

What at the beginning seemed a nice result turns out to be a really bad one. This is an example of how bad things can go, if you don’t pay attention to the distributions of your classes. This, of course, applies not only when splitting your dataset, but in general, when you approach a classification problem, regardless of the classifier you want to train (it does not apply only to neural networks).

Precision, Recall, and F1 Metrics

Let’s look at some other metrics that are very useful when dealing with unbalanced datasets. Consider the following example. Suppose we are doing some tests to determine if a subject has a certain disease or not. Imagine that we have 250 test results. Consider the following confusion matrix (you should know what that is from our previous discussion):

This translates visually to the following:

Let’s also indicate with ty the number of patients who really have a disease, in this example, ty = 10 + 150 = 160, and with tno, the number of patients who don’t have a disease, in this example, tno = 75 + 15 = 90. In the examples we would have

All those quantities can be used as metrics, depending on your problem. Let’s create an example. Suppose your test should predict if a person has cancer or not. In this case, what you want is the highest sensitivity possible, because it is important to detect the disease. But at the same time, you also want the specificity to be high, because there is nothing worse than sending someone home without treatment when it is needed.

Let’s look a bit closer at precision and recall. Having a high precision means that when you say someone is sick, you are right. But you don’t know how many people really have the sickness, since the quantity is defined only by the result of your test. Precision is a measure of how your test is doing. Having a high recall means that you can identify all the sick people in your sample. Let me give you another example, to make the point even clearer. Suppose we have 1000 people. Only 10 are sick and 990 are healthy. Let’s suppose we want to identify healthy people (this is important), and we build a test that returns yes if someone is healthy and always predicts that people are healthy. The confusion matrix would look like this:

We would have

This looks good, right? If want to find healthy people, this test would be great. The only problem is that it is a lot more important to identify sick people! Let’s recalculate the preceding quantities, but this time, considering that a positive result is when someone is sick. In this case, the confusion matrix would like this:

because this time, a yes result means that someone is sick and not, as before, that someone is healthy. Let’s calculate the quantities above again.

Note how the accuracy remains the same. If you look only at that, you would not be able to understand how your model is doing. We have simply changed what we want to predict and use only accuracy. We cannot say anything about the performance of our model. But look at how recall and precision changed. See the matrix below for a comparison.

Now we have something that changes that can give us enough information, depending on the question we pose. Note that changing what we want to predict will change how the confusion matrix will look. We can immediately say, looking at the preceding matrix, that our model that predicts that everyone is healthy works very well when predicting healthy people (not very useful) but fails miserably when trying to predict sick people.

There is another metric that is important to know, and that is the F1 score. It is defined as

An intuitive understanding is difficult to get, but it is basically the harmonic average of precision and recall. The example we created was a bit extreme and, having 0% for recall or precision , would not allow us to calculate F1. Let’s suppose that our model is bad at predicting sick people, but not that bad. Let’s suppose we have the following confusion matrix:

In this case, we would have (I leave the calculation to you)

(Precision:)

(Recall:)

We would have

This quantity will give you information keeping precision (the portion of tests predicting correctly the subject having the disease with respect to all positive results obtained) and recall (how often the test really predicts yes when the subjects have the disease) in consideration. For some problems, you want to maximize precision, and for others, you want to maximize recall. If that is the case, simply choose the right metric. The F1 score will be the same for two cases in which you have Precision = 32% and Recall = 45% and one with Precision = 45% and Recall = 32%. Be aware of this fact. Use F1 score, if you want to find a balance between Precision and Recall.

If we calculate F1 when predicting healthy people, as we did at first, we would have

This tells us that the model is quite good at predicting healthy people.

The F1 score is usually useful, because, normally, as a metric, you want one single number, and in this way, you don’t have to decide between precision or recall, as both are useful. Remember that the values of the metrics discussed will always be dependent on the question you are asking (what is yes and no for you). Be aware that an interpretation is always dependent on the question you want to answer.

Datasets with Different Distributions

Now I would like to discuss another terminology issue, which will lead you to understand a common problem in the deep-learning world. Very often, you will hear sentences such as “The sets come from different distributions.” This sentence is not always easy to understand. Take, for example, two datasets formed by images taken with a professional DSLR, and a second one made up of images taken with a dodgy smartphone. In the deep-learning world, we would characterize those two sets as coming from different distributions. But what is the real meaning of the sentence? The two datasets differ for various reasons: resolution of images, blurriness resulting from different quality lenses, amount of colors, quality of the focus, and possibly more. All these differences are what is usually meant by distributions. Let’s look at another example. We could consider two datasets: one made of images of white cats and one made of images of black cats. Also, in this case, we are talking about different distributions. This becomes a problem when you train a model on one set and want to apply it to the other. For example, if you train a model on a set of images of white cats, you probably are not going to do very well on the dataset of black cats, because your model has never seen black cats during training.

Because data is so precious, people often try to create different datasets (train, dev, etc.) from different sources. For example, you may decide to train your model on a set made of images taken from the Web and check how good it is with a set made of images you’ve taken with your smartphone. It may seem like a good idea to be able to use as much data as possible, but this may give you many headaches. Let’s see what happens in a real case, so that you may get a feeling of the consequences of doing something similar.

To make the shift easy, I first reshaped the images in a 28 × 28 matrix, then simply shifted the columns with tmp_shifted[:,10:28] = tmp[:,0:18], and then I simply reshaped the images in a one-dimensional array of 784 elements. The labels remain the same. In Figure 6-7, you can see a random image from the dev dataset on the left and its shifted version on the right.

What has happened is that the model has learned from a dataset where all images are centered in the box and, therefore, could not generalize well to images shifted and no longer centered.

When training a model on a dataset, usually you will get good results for observations that are like the ones in the training set. But how can you find out if you have such a problem? There is a relatively easy way of doing that: expanding our MAD diagram . Let’s see how to do it.

Suppose you have a training dataset and a dev dataset in which the observations have different characteristics (come from different distributions). What you do is create a small subset from the training set, called the train-dev dataset, ending up with three datasets: a training and a train-dev from the same distribution (the observations have the same characteristics) and a dev set, for which the observations are somehow different, as I have discussed previously. What you do now is train your model on your training set and then evaluate your error ϵ on the three datasets: ϵ_train, ϵ_dev, and ϵ_{train − dev}. If your train and dev sets come from the same distributions, so does the train-dev set. In this case, you should expect ϵ_dev ≈ ϵ_{train − dev}. If we define

In Figure 6-8, you can see an example of the MAD diagram with the added problem just discussed. Don’t look at the numbers; they are there only for illustrative purposes (read: I just put them there).

Note that you don’t need to create the bar plot, as I have done here. Technically, you require only the four numbers, to draw the same conclusions.

As usual, there are no fixed rules. Just be aware of the problem and think about the following: your model will learn the characteristics from your training data, so when applied to completely different data, it (usually) won’t do well. Always get training data that reflect the data you want your model to work on, not vice versa.

K-Fold Cross-Validation

A typical value for k is 10, but that depends on the size of your dataset and the characteristic of your problem.

Remember that the discussion we did on how to split a dataset applies here also.

Although this may seem a very attractive technique to deal generally with datasets with less than optimal size, it may be quite complex to implement. But, as you will see shortly, checking your metric on the different folds will give you important information on possible overfitting of your training dataset.

Let’s try it on a real dataset and see how to implement it. Note that you can implement k-fold cross-validation easily in with the sklearn library, but I will develop it from scratch, to show you what is happening in the background. Everyone (well, almost) can copy code from the Web to implement k-fold cross-validation in sklearn, but not many can explain how it works or understand it, therefore being able to choose the right sklearn method or parameters. As a dataset, we will use the same we used in Chapter 2: the reduced MNIST dataset containing only digits 1 and 2. We will perform a simple logistic regression with one neuron, to make the code easy to understand and to let us concentrate on the cross-validation part and not on other implementation details that are not relevant here. The goal of this section is to let you understand how k-fold cross-validation works and why it is useful, not on how to implement it with the smallest number of lines of code possible.

You will notice that you will get for each fold slightly different accuracy values. It is very instructive to study how the accuracy values are distributed. Because we have 10 folds, we have 10 values to study. In Figure 6-9, you can see the distribution of the values for the train set (left) and for the dev set (right).

The image is quite instructive. You can see that the accuracy values for the training set are quite concentrated around the average, while the ones evaluated on the dev set are much more spread. This shows how the model on new data behaves less well than on the data it has trained on. The standard deviation for the training data is 5.4 · 10⁻⁴ and for the dev set 2.4 · 10⁻³, 4.5 times larger than the value on the train set. In this way, you also get an estimate of the variance of your metric when applied on new data, and on how it generalizes. If you are interested in learning how to do this quickly with sklearn, you can check the official documentation for the KFold method here: https://goo.gl/Gq1Ce4 . When you are dealing with datasets with many classes (remember the discussion on how to split your sets?), you must pay attention and do what is called stratified sampling.² sklearn provides a method to do that too: stratifiedKFold, which can be accessed here: https://goo.gl/ZBKrdt .

You can now easily find averages and standard deviations. For the training set, we have an average accuracy of 98.7% and a standard deviation of 0.054%, while for the dev set, we have an average of 98.6% with a standard deviation of 0.24%. So, now you can even give an estimate of the variance of your metric. Pretty cool!

Manual Metric Analysis: An Example

I mentioned earlier that sometimes it is useful to do a manual analysis of your data, to check if the results (or the errors) you are getting are plausible. I would like to give you a basic example here, to give you a concrete idea of what is involved and how complicated it can be. Let’s consider the following: our very simple model (remember, we are using only one neuron) can get 98% of accuracy. Is the problem of recognizing digits that easy? Let’s try to see if that is the case. First, note that our training set does not even have the two-dimensional information of the images. If you remember, each image is converted in a one-dimensional array of values: the gray values of each pixel, starting on the top left and going row by row from top to bottom. Are the ones and the twos so easy to recognize? Let’s check how the real input for our model might look. Let’s start analyzing the digit 1. Let’s take an example from fold 0. In Figure 6-10, you can see the image on the left and a bar plot of the gray values of 784 pixels, as they are seen from our model. Remember that as observations, we have a one-dimensional array of the 784 gray values of the pixels of the image.

Remember that we reshape our 28 × 28 pixels image in a one-dimensional array , so when reshaping the digit 1 in Figure 6-10, we will find black points roughly each 28 pixels, because the 1 is almost a vertical column of black points. In Figure 6-11, you can see other ones, and you will notice how, when reshaped as one dimensional, they all look the same: several bars roughly equally spaced. Now that you know what to look for, you can easily say that all the images in Figure 6-11 are all of digit 1.

Now let’s look at the digit 2. In Figure 6-12, you can see an example, similar to what we had in Figure 6-10.

Now things look different. We have two regions in which the bars are much denser, seen in the plot on the right in Figure 6-12. This is the case between pixels 100 and 200 and especially after pixel 500. Why? Well, the two areas correspond to the two horizontal parts of the image. In Figure 6-13, I have highlighted how different parts look when reshaped as one-dimensional arrays.

Horizontal parts (A) and (B) are clearly different from part (C) when reshaped as a one-dimensional array . The vertical part (C) looks like the digit 1, with many equally spaced bars, as you can see in the lower right bar plot labeled (C), while the more horizontal parts appear as many bars clustered in groups, as can be seen in the upper right and lower left bar plots labeled (A) and (B). So, when reshaped, if you find those clusters of bars, you are looking at a 2. If you see only equally spaced small groups of bars, as in the plot (C) in Figure 6-13, you are looking at a 1. You don’t even have to see the two-dimensional image, if you know what to look for. Note that this pattern is very constant. In Figure 6-14, you can see four examples of the digit 2, and you can clearly see the wider clusters of bars.

As you can imagine, this is an easy pattern to spot for an algorithm, and so it is to be expected that our model works well. Even a human can spot the images, even when reshaped, without any effort. Such a detailed analysis would not be necessary in a real-life project, but it is instructive to see what you can learn from your data. Understanding the characteristics of your data may help you in designing your model or understanding why it is not working. Advanced architectures, such as convolutional networks, will be able to learn those two-dimensional features exactly in a very efficient way.

Let’s also check how the network learned to recognize digits. You will remember that the output of our neuron is

where σ is the sigmoid function, x_i for i = 1, …, 784 are the gray values of the pixel of the image, w_i for i = 1, …, 784 are the weights, and b is the bias. Remember that when , we classify the image in class 1 (so, digit 2), and if , we classify the image in class 0 (so, digit 1). Now from the discussion of the sigmoid function in Chapter 2, you will remember that σ(z) ≥ 0.5 when z ≥ 0 and σ(z) < 0.5 for z < 0. This means that our network should learn the weights in such a way that for all the ones, we have z < 0, and for all the twos, z ≥ 0. Let’s see if that is really the case. In Figure 6-15, you can see a plot for a digit 1, from which you can find the weights w_i (solid line) of our trained network after 600 epochs (and after reaching an accuracy of 98%) and the gray value of the pixel x_i rescaled to have a maximum of 0.5 (dashed line). Note how each time x_i is big, w_i is negative. And when w_i > 0, the x_i are almost zero. Clearly, the result will be negative, and, therefore, σ(z) < 0.5, and the network will identify the image as a 1. In the image, I zoomed in to make this behavior more evident.

In Figure 6-16 , you can see the same plot for a digit 2. You will remember from the previous discussion that for a 2, we can see many bars clustered together in groups up to pixel 250 (roughly). Let’s check how the weights in that region are. Now you will see that where the pixel gray values are big, the weights are positive, giving, then, a positive value of z and, therefore, σ(z) ≥ 0.5, and so the image would be classified as a 2.

As an additional check, I plotted w_i · x_i for all values of i for a digit 1, shown in Figure 6-17. You can see how almost all points lie below zero. Note also that b =  − 0.16, in this case.

As you can see, in very easy cases, it is possible to understand how a network learns and, therefore, it is much easier to debug strange behaviors. But don’t expect this to be possible when dealing with much more complex cases. The analysis we have done would not be so easy, for example, if you tried to do the same with digits 3 and 8, instead of 1 and 2.