Network Architecture

Neural network architecture is quite easy to understand. It consists of an input layer (the inputs ), several layers (called hidden, because they are sandwiched between the input and the output layers , so they are “invisible” from the outside, so to speak), and an output layer. In each layer, you may have one to several neurons. The main property of such a network is that each neuron receives input from each neuron in the preceding layer and feeds its output to every neuron in the next layer. In Figure 3-1, you can see a graphical representation of such a network.

In a network such as the one in Figure 3-1, we will indicate the total number of neurons with N_neurons, which can be written as

where, by convention, we defined n₀ = n_x. Each connection between two neurons will have its own weight. Let’s indicate the weight between neuron i in layer l and neuron j in layer l − 1 with . In Figure 3-2, I have drawn only the first two layers (input and layer 1) of our generic network of Figure 3-1, with the weights between the first neuron in the input layer and all the others in layer 1. All other neurons are grayed out for clarity.

The weights between the input layer and layer 1 can be written as a matrix, as follows:

This means that our matrix W^[1] has dimensions n₁ × n_x. Of course, this can be generalized between any two layers l and l − 1, meaning that the weight matrix between two adjacent layers l and l − 1, indicated by W^[l], will have dimensions n_l × n_l − 1. By convention, n₀ = n_x is the number of input features (not the number of observations that we indicate with m).

The bias (indicated by b in Chapter 2) will be a matrix this time. Remember that each neuron that receives inputs will have its own bias, so when considering our two layers, l and l − 1, we will require n_l different values of b. We will indicate this matrix with b^[l], and it will have dimensions n_l × 1.

Output of Neurons

Now let’s start considering the output of our neurons. To begin, we will consider the i^th neuron of the first layer (remember that our input layer is by definition layer 0). Let’s indicate its output with and assume that all neurons in layer l use the same activation function, which we will indicate by g^[l]. Then we will have

where we have indicated, as you will remember from Chapter 2, z_i as

As you can imagine, we want to have a matrix for all the output of layer 1, so we will use the notation

where Z^[1] will have dimensions n₁ × 1, and where with X, we have indicated our matrix with all our observations (rows for the features, and columns for observations), as I have already discussed in Chapter 2. We assume here that all neurons in layer l will use the same activation function that we will indicate with g^[l].

We can easily generalize the previous equation for a layer l

because layer l will get its input from layer l − 1. We just need to substitute X with Z^[l − 1]. Z^[l] will have dimensions n_l × 1. Our output in matrix form will then be

where the activation function acts, as usual, element by element.

Example: Equations for a Network with Three Layers

To make this discussion a bit more concrete, let’s consider an example of a network with three layers (so L = 3), with n₁ = 3, n₂ = 2, and n₃ = 1, as depicted in Figure 3-3.

In this case, we will have to calculate the following quantities:

• , where W^[1] has dimensions 3 × n_x, b has dimensions 3 × 1, and X has dimensions n_x × m

• , where W^[2] has dimensions 2 × 3, b has dimensions 2 × 1, and Z^[1] has dimensions 3 × m

• , where W^[3] has dimensions 1 × 2, b has dimensions 1 × 1, and Z^[2] has dimensions 2 × m

and your network output, , will have, as expected, dimensions 1 × m.

All this may seem rather abstract (and, in fact, it is). You will see later in the chapter how easy it is to implement in TensorFlow, simply by building the right computational graph, based on the steps just discussed.

sof tmax Function for Multiclass Classification

You will still have to suffer a bit more theory before getting to some TensorFlow code. The kinds of networks described in this chapter start to be complex enough to be able to perform some multiclass classification with reasonable results. To do this, we must first introduce the softmax function.

Mathematically speaking, the softmax function S is one that transforms a k dimensional vector into another k dimensional vector of real values, each between 0 and 1, and that sum up to 1. Given k real values z_i for i = 1, …, k, we define the vector z = (z₁, …, z_k), and we define the softmax vector function S(z) = (S(z)₁ S(z)₂… S(z)_k) as

Because the denominator is always bigger than the nominator, S(z)_i < 1. Additionally, we have

So, S(z)_i behaves like a probability, because its sum over i is 1, and its elements are all less than 1. We will consider S(z)_i as a probability distribution over k possible outcomes. For us, S(z)_i will simply be the probability of our input observation of being of class i. Let’s suppose we are trying to classify an observation into three classes. We may get the following output: S(z)₁ = 0.1, S(z)₂ = 0.6, and S(z)₃ = 0.3. That means that our observation has a 10% probability of being of class 1, a 60% probability of being of class 2, and 30% probability of being of class 3. Normally, one chooses to classify the input observation into the class with the higher probability, in this example, class 2, with 60% probability.

To be able to use the softmax function for classification, we will have to use a specific output layer. We will have to use ten neurons, each of which will give z_i as its output, and then one neuron that will output S(z). This neuron will have the softmax function as activation function and will have as inputs the 10 outputs, z_i, of the last layer with 10 neurons. In TensorFlow, you use the tf.nn.softmax function applied to the last layer with 10 neurons. Remember that this tensorflow function will act element by element. Later in the chapter, you will find a concrete example showing how to implement this from start to finish.

A Brief Digression: Overfitting

One of the most common problems that you will encounter when training deep neural networks will be overfitting. What can happen is that your network may, owing to its flexibility, learn patterns that are due to noise, errors, or simply wrong data. It is very important to understand what overfitting is, so I will give you a practical example of what can happen, to give you an intuitive understanding of it. To make it easier to visualize, I will work with a simple two-dimensional dataset, which I will create for the purpose. I hope that at the end of the next section, you will have a clear idea of what overfitting is.

A Practical Example of Overfitting

Networks described in the previous sections are rather complex and can easily lead to overfitting of the dataset. Let me explain briefly the concept of overfitting. To understand it, consider the following problem: find the best polynomial that approximates a given dataset. Given a set of two-dimensional points (x⁽ⁱ⁾, y⁽ⁱ⁾), we want to find the best polynomial of degree K in the form

that minimizes the mean square error

where, as usual, m indicates the number of data points we have. I don’t want only to determine all the parameters a_j, but also the value of K that best approximates our data. K, in this case, measures our model complexity. For example, for K = 0, we simply have f (x⁽ⁱ⁾) = a₀ (a constant), the simplest polynomial we can think of. For higher K, we have higher order polynomials, meaning that our function is more complex, having more parameters available for training.

Here is an example of our function for K = 3:

where we have four parameters that can be tuned during our training models. Let’s generate some data, starting from a second order polynomial (K = 2)

and adding some random error (this will make overfitting visible). Let’s first import our standard libraries with the addition of the curve_fit function , which will minimize automatically the standard error and find the best parameters. Don’t worry too much about this function. The goal here is to show you what can happen when you use a model that is too complex.

import numpy as np

import matplotlib.pyplot as plt

from scipy.optimize import curve_fit

Let’s define a function for a second-degree polynomial

def func_2(p, a, b, c):

    return a+b*p + c*p**2

then let’s generate our dataset

x = np.arange(-5.0, 5.0, 0.05, dtype = np.float64)

y = func_2(x, 1,2,3)+18.0*np.random.normal(0, 1, size=len(x))

To add some random noise to the function, we have used the function np.random.normal(0, 1, size=len(x)), which generates a numpy array of random values from a normal distribution of length len(x), with average 0 and standard deviation 1.

In Figure 3-4, you can see what the data looks like for a = 1, b = 2, and c = 3.

Now let’s consider a model that is too simple to capture the feature of the data, meaning that we will see what a model with high bias¹ can do. Let’s consider a linear model (K = 1). The code will be

def func_1(p, a, b):

    return a+b*p

popt, pcov = curve_fit(func_1, x, y)

That will give the best values for a and b that minimize the standard error. In Figure 3-5, it is clear how this model completely misses the main feature of the data, being too simple.

Let’s try to fit a two-degree polynomial (K = 2). The results appear in Figure 3-6.

That is better. This model seems to capture the main features of the model, ignoring the random noise. Now let’s try a very complex model—a 21-degree polynomial (K = 21). The results are shown in Figure 3-7.

Now, this model shows features that we know are wrong, because we created our data. These features are not present, but the model is so flexible that it captures the random variability that we have introduced with noise. Here, I am referring to the oscillations that have appeared using this high-ordered polynomial.

In this case, we talk about overfitting, meaning we start capturing with our model features owing, for example, to random error. It is easy to understand that this generalizes quite badly. If we applied this 21-degree polynomial model to new data, it would not work well, because the random noise would be different in new data, and so the oscillations we see in Figure 3-7 would make no sense on new data. In Figure 3-8, I have plotted the best 21-degree polynomial models obtained by fitting data generated with 10 different random noise values added. You can clearly see how much it varies. It is not stable and is strongly dependent on the random noise present. The oscillations are always different! In this case, we are talking about high variance.

Now let’s create the same plot with our linear model, while varying our random noise, as we did in Figure 3-8. You can check the results in Figure 3-9.

You can see that our model is much more stable. Our linear model does not capture any feature that is dependent on our noise, but it misses the main features of our data (the concave nature). We are talking here of high bias.

Figure 3-10 should help you to gain an intuitive understanding of bias and variance. Bias is a measure of how close our measurements are to the true values (the center of the figure), and variance is a measure of how spread the measurements are around the average (not necessarily the true value, as you can see on the right).

In the case of neural networks, we have many hyperparameters (number of layers, number of neurons in each layer, activation function, and so on), and it is very difficult to know in which regime we are. How can we tell if our model has a high variance or a high bias, for example? I will dedicate an entire chapter to this subject, but the first step in performing this error analysis is to split our dataset into two different ones. Let’s see what this means and why we do it.

Note

The essence of overfitting is to have unknowingly extracted some of the residual variation (i.e., the noise) as if that variation represented the underlying model structure (see Burnham, K. P.; Anderson, D. R., Model Selection and Multimodel Inference, 2nd ed., New York; Springer-Verlag, 2002). The opposite is called underfitting—when the model cannot capture the structure of the data.

The problem with overfitting and deep neural networks is that there is no way of visualizing easily the results, and, therefore, we require a different approach to determine if our model is overfitting, underfitting, or is just right. This can be achieved by splitting our dataset into different parts and evaluating and comparing the metrics on all of them. Let’s explore the basic idea in the next section.

The Zalando Dataset

Zalando SE is a German e-commerce company based in Berlin. The company maintains a cross-platform store that sells shoes, clothing, and other fashion items.³ For a kaggle competition (if you don’t know what this is, check the website www.kaggle.com , from which you can participate in many competitions that have the goal of solving problems with data science), Zalando prepared a MNIST-similar dataset of images of its clothing, for which they provided 60,000 training images and 10,000 test images. As in MNIST, each image was 28 × 28 pixels in grayscale. Zalando grouped all images in ten different classes and provided the labels for each image. The dataset has 785 columns. The first column is the class label (an integer going from 0 to 9), and the remaining 784 contain the pixel gray value of the image (you can calculate that as 28 × 28 = 784), exactly as we have seen in Chapter 2, in the discussion related to the MNIST dataset of handwritten digits.

In Figure 3-11, you can see an example of each class, chosen randomly from the dataset.

The dataset has been provided under the MIT License.⁴ The data file can be downloaded from kaggle ( www.kaggle.com/zalando-research/fashionmnist/data ) or directly from GitHub ( https://github.com/zalandoresearch/fashion-mnist ). If you choose the second option. you will have to prepare the data a bit. (You can convert it to CSV with the script located at https://pjreddie.com/projects/mnist-in-csv/ .) If you download it from kaggle, the data will already be in the correct format. You will find two CSV files zipped on the kaggle web site. When unzipped, you will have fashion-mnist_train.csv, with 60,000 images (roughly 130MB), and fashion-mnist_test.csv, with 10,000 (roughly 21MB). Let’s fire up a Jupyter notebook and start coding!

You can also read the file with standard NumPy functions (such as loadtxt()), but using read_csv() from pandas gives you a lot of flexibility in slicing and analyzing your data. Additionally, it is a lot faster. Reading the file (that is, roughly 130MB) with pandas takes about 10 seconds, while with NumPy, it takes 1 minute, 20 seconds on my laptop. So, if you are dealing with big datasets, keep this in mind. It is common practice to use pandas to read and prepare the data. If you aren’t familiar with pandas, don’t worry. All you need to understand will be explained in detail.

you can see the first five lines of your dataset, as shown in Figure 3-12.

giving the column name in square brackets.

Now the tensor labels have the dimension (1, 60000), as we want.

Building a Model with tensorflow

Now it is time to expand what we did with TensorFlow in Chapter 2 with one neuron to networks with many layers and neurons. Let’s first discuss the network architecture and what kind of output layer we need, and then let’s build our model with TensorFlow.

Network Architecture

We will start with a network with just one hidden layer. We will have an input layer with 784 features, then a hidden layer (in which we will vary the number of neurons), then an output layer of ten neurons that will feed their output into a neuron that will have as an activation function the softmax function . See first Figure 3-13, for a graphical representation of the network, and then I will spend some time explaining the various parts, especially the output layers.

Let me explain why there is this strange output layer with ten neurons and why there is a need for an additional neuron for the softmax function . Remember that for each image, we want to be able to determine to which class it belongs. To do this, as explained when discussing the softmax function, we will have to get ten outputs for each observation: each being the probability of the image of being of each of the classes. So, given an input x⁽ⁱ⁾, we will need the ten values: P( y⁽ⁱ⁾ = 1| x⁽ⁱ⁾), P( y⁽ⁱ⁾ = 2| x⁽ⁱ⁾), …., P( y⁽ⁱ⁾ = 10| x⁽ⁱ⁾) (probability of the observation class y⁽ⁱ⁾ being one of the ten possibilities given the input x⁽ⁱ⁾), or, in other words, our output should be a tensor of dimensions 1 × 10 in the form

And because the observation must be of one single class the condition

must be satisfied. This can be understood as follows: the observation has a 100% probability of being of one of the ten classes, or, in other words, all the probabilities must add to 1. We solve this problem in two steps:

• We create an output layer with ten neurons. In this way, we will have our ten values as output.

• Then we feed the ten values to a new neuron (let’s call it “softmax” neuron) that will take the ten inputs and give as output ten values that are all less than 1 and that add up to 1.

Figure 3-14 shows our “softmax” neuron in detail.

Calling z_i the output of the i^th neuron in the last layer (with i going from 1 to 10), we will have

That is exactly what the tensorflow function tf.nn.softmax() does. It takes a tensor as input and returns a tensor with the same dimensions as the input but “normalized,” as discussed previously. In other words, if we feed z = (z₁  z₂ …  z₁₀) to the function, it will return a tensor with the same dimensions as z, meaning 1 × 10, where each element is the last equation.

Modifying Labels for the softmax Function—One-Hot Encoding

Before developing our network, first we must solve another problem. You will remember from Chapter 2 that in classification, we will use the following cost function:

cost = - tf.reduce_mean(Y * tf.log(y_)+(1-Y) * tf.log(1-y_))

where Y contains our labels and y_ is the result of our network. So, the two tensors must have the same dimensions. In our case, I explained to you that our network will give as output a vector with ten elements, while a label in our dataset is simply a scalar. Therefore, we have y_ that has dimensions (10,1) and Y that has dimensions (1,1). This will not work if we don’t do something smart. We must transform our labels in a tensor that has dimensions (10,1). A vector with a value for each class is also required, but what value should we use?

We must perform what is known as one-hot encoding .⁵ This means that we will transform our labels (integers from 0 to 9) to tensors with dimensions (1,10) with the following algorithm: our one-hot encoded vector will have all zeros, except at the index of the label. For example, for a label 2, our 1 × 10 tensor will have all zeros, except at the position of index 2, or, in other words, it will be (0,0,1,0,0,0,0,0,0,0). Try some other examples (see Table 3-2), and the concept will become clear immediately.

Table 3-2

Examples of How One-Hot Encoding Works (Remember that labels go from 0 to 9 as indexes.)

Label	One-Hot Encoded Label
0	(1,0,0,0,0,0,0,0,0,0)
2	(0,0,1,0,0,0,0,0,0,0)
5	(0,0,0,0,0,1,0,0,0,0)
7	(0,0,0,0,0,0,0,1,0,0)

In Figure 3-15, you can see a graphical representation of the process of one-hot encoding a label. In the figure, two labels (2 and 5) are one-hot encoded in two tensors. The grayed element of the tensor (in this case, a one-dimensional vector) is the one that becomes one, while the white ones remain zero.

Sklearn has several ways of doing this automatically (check, for example, the function OneHotEncoder()), but I think it is instructive to undertake the process manually, to really see how it is done. Once you understand why you need it, and in which format you need it, you can use the function you like most. The Python code to do this is very simple (the last line just converts the pandas data frame into a NumPy array):

labels_ = np.zeros((60000, 10))

labels_[np.arange(60000), labels] = 1

labels_ = labels_.transpose()

labels_ = np.array(labels_)

First, you create a new array with the right dimensions: (60000,10), then you fill it with zeros with the NumPy function np.zeros((60000,10)). Next, you set to 1 only the columns related to the label itself, using pandas functionalities to slice data frames with the line labels_[np.arange(60000), labels] = 1. Then you transpose it, to have the dimensions we want at the end: (10, 60000), where each column indicates a different observation.

Now in our code, we can finally compare Y and y_, because both now have the dimensions (10,1) for one observation, or when considering the entire training dataset of (10, 60000). Each row in y_ will now represent the probability of our observation as being of a specific class. At the very end, when calculating the accuracy of our model, we will assign the class with the highest probability to each observation.

Note

Our network will give us the ten probabilities for the observation as being of each of the ten classes. At the end, we will assign to the observation the class that has the highest probability.

Gradient Descent Variations

In Chapter 2, I described the very basic gradient descent algorithm (also called batch gradient descent). This is not the smartest way of finding the cost function minimum. Let’s have a look at the variations that you need to know, and let’s compare how efficient they are, using the Zalando dataset.

Comparison of the Variations

Following is a summary of the findings for our three variations of gradient descent for 100 epochs (Table 3-3).

Table 3-3

Summary of the Findings for Three Variations of Gradient Descent for 100 Epochs

Gradient Descent Variation	Running Time	Final Value of Cost Function	Accuracy
Batch gradient descent	2.5 min	0.323	16%
Mini-batch gradient descent	2.5 min	0.14	66%
Stochastic gradient descent (SGD)	35 min	0.094	80%

Now you can see that SGD is the algorithm that achieves the lowest value of cost function with the same number of epochs, although it is by far the slowest. For the mini-batch gradient descent to reach a value of 0.094 for the cost function, it takes 450 epochs and roughly 11 minutes. Still, this is a huge improvement over SGD—31% of the time for the same results.

In Figure 3-16, you can see the difference in how the cost function decreases with different mini-batch sizes. It is clear how, with respect to the number of epochs, the smaller the mini-batch size, the faster the decrease (though not in time). The learning rate used for this figure was γ=0.001. Note that the time required in each case is not the same, and the smaller the mini-batch size, the more time is required for the algorithm.

Note

The best compromise between running time and convergence speed (with respect to number of epochs) is achieved by mini-batch gradient descent. The optimal size of the mini-batches is dependent on your problem, but, usually, small numbers, such as 30 or 50, are a good option. You will find a compromise between running time and convergence speed.

To give you an idea of how the running time depends on the value the cost function can reach after 100 epochs, see Figure 3-17 . Each point is labeled with the size of the mini-batch used in that run. Note that the points are single runs, and the plot is only indicative of the dependency. Running time and cost function value have a small variance when evaluated over several runs. This variance is not shown in the plot. You can see that decreasing the mini-batch size from 300 quickly decreases the value of J after 100 epochs, without increasing the running time significantly, until you arrive at a value for the mini-batch size that is about 50. At that point, the time starts to increase quickly, and the value for J after 100 epochs no longer decreases as quickly and flattens out. Intuitively, the best compromise is to choose a value for the mini-batch size when the curve is closer to zero (small running time and small cost function value), and that is at a mini-batch size value between 50 and 30. This is why those are the values chosen most frequently. After that point, the increase in running time becomes very quick and is no longer worth decreasing the mini-batch size. Note that for other datasets, the optimal value may be very different. So, it is worth trying different values, to see which one works best. In very big datasets, you may want to try bigger values, such as 200, 300, or 500. In our case, we have 60,000 observations and a mini-batch size of 50, which gives 1200 batches. If you have much more data, for example 1e-6 observations, a mini-batch size of 50 would give 20,000 batches. Keep that in mind and try different values, to see which works best.

It is good programming practice to write a function that runs your evaluations. In this way, you can tune your hyperparameters (such as the mini-batch size) without copying and pasting the same chunk of code over and over. The following function is one you can use to train our model:

def model(minibatch_size, training_epochs, features, classes, logging_step = 100, learning_r = 0.001):

    sess = tf.Session()

    sess.run(tf.global_variables_initializer())

    cost_history = []

    for epoch in range(training_epochs+1):

        for i in range(0, features.shape[1], minibatch_size):

            X_train_mini = features[:,i:i + minibatch_size]

            y_train_mini = classes[:,i:i + minibatch_size]

            sess.run(optimizer, feed_dict = {X: X_train_mini,

                                             Y: y_train_mini,

                                             learning_rate: learning_r})

        cost_ = sess.run(cost, feed_dict={ X:features, Y: classes, learning_rate: learning_r})

        cost_history = np.append(cost_history, cost_)

        if (epoch % logging_step == 0):

                print("Reached epoch",epoch,"cost J =", cost_)

    return sess, cost_history

The model() function will accept the following parameters:

• minibatch_size: The number of observations we want in each batch. Note that if we choose for this hyperparameter a number q that is not a divisor of m (number of observations), or, in other words, m/q is not an integer, we will have the last mini-batch with a different number of observations than all the others. But this will not be an issue for the training. For example, suppose we have a hypothetical dataset with m=100, and you decide to use mini-batch sizes of 32 observations. Then, with m=100, you will have 3complete mini-batches with 32 observations and 1 with just 4, since 100 = 3*32+4. Now you may wonder what will happen with a line such as

  X_train_mini = features[:,i:i + 32]

  when i=96 and features has only 100 elements. Are we not going over the limits of the array? Fortunately, Python is nice to programmers and takes care of this. Consider the following code:

  l = np.arange(0,100)

  for i in range (0, 100, 32):

  print (l[i:i+32])

  The result is

  [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31]

  [32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63]

  [64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95]

  [96 97 98 99]

  And as you see, the last batch has only four elements, and we don’t get any error. So, you should not worry about this, and you can choose any mini-batch size that works better for your problem.

• training_epochs: The number of epochs we want

• features: The tensor that contains our features

• classes: The tensor that contains our labels

• logging_step: This tells the function to print the value of the cost function every logging_step epoch

• learning_r: The learning rate we want to use

Note

Writing a function with the hyperparameters as inputs is common practice. This allows you to test different models with different values for the hyperparameters and check which one is better.

Examples of Wrong Predictions

Running the model with batch gradient descent, one hidden layer with 5 neurons for 1000 epochs, and learning rate of 0.001 will give us an accuracy on the training set of 82.3%. You can increase the accuracy by using more neurons in your hidden layer. For example, using 50 neurons, using 1000 epochs and a learning rate of 0.001, will allow you to reach 86.4% on the training set and 86.1% on the test set. It is interesting to check a few examples of wrongly classified images, to see if we can understand something from the errors. Figure 3-18 shows an example of wrongly classified images for each class. Over each image, the True class (labeled as “True:”) and the predicted (labeled as “Pred:”) class are reported. The model used here has one hidden layer with five neurons and has been run for 1000 epochs with a learning rate of 0.001.

Some errors are understandable, such as, for example, that at the lower left of the figure. A shirt has been wrongly classified as a coat. It is also difficult to determine which item is which, and I could easily have made the same mistake. The wrongly classified bag is, on the other hand, easy for a human to sort out.

Weight Initialization

But why choose a standard deviation of 0.1?

You will surely have wondered why. In the previous sections, I wanted you to focus on understanding how such a network works, without the distraction of additional information, but it is now time to look at this problem a bit more closely, because it plays a fundamental role with many layers. Basically, we initialize the weights with a small standard deviation to prevent the gradient descent algorithm from exploding and starting to return nans. For example, in our first layer for the i^th neuron, we will have to calculate the ReLU activation function of the quantity (refer to the beginning of the chapter for an explanation, if you’ve forgotten why), as follows:

Normally in a deep network, the number of weights is quite big, so you can easily imagine that if the are big, the quantity z_i, too, can be quite big, and the ReLU activation function can return a nan value, because the argument is too big for Python to calculate it properly. So, you want the z_i to be small enough to avoid an explosion of the output of the neurons and big enough to prevent the outputs from dying out and, therefore, making the convergence a very slow process.

In a layer l, the number of inputs will be the number of neurons of the preceding layer l − 1, and the number of outputs will be the number of neurons in the layer coming next: l + 1. So, we will have

and

Very often, deep networks such as the one discussed before will have several layers, all with the same number of neurons. Therefore, you will have, for most of the layers, n_l − 1 = n_l + 1, and, therefore, you will have the following for Xavier initialization:

For the ReLU activation functions, the He initialization will be

Typically, to make evaluation and construction of networks easier, the most typical initialization form used is for ReLU activation function

and

Using this initialization can speed up training considerably and is the standard way in which many libraries initialize weights (for example, the Caffe library).

Adding Many Layers Efficiently

Now the code is much easier to understand, and you can use it to create networks as big as you wish.

With the preceding functions, it is very easy to run several models and compare them, as I have done in Figure 3-19, which illustrates five different tested models.

• One layer and ten neurons each layer

• Two layers and ten neurons each layer

• Three layers and ten neurons each layer

• Four layers and ten neurons each layer

• 4 layers and 100 neurons each layer

In case you are wondering , the model with four layers, each with 100 neurons, which seems much better than the others, is starting to go in the overfitting regime, with a train set accuracy of 94% and of 88% on the dev set (after only 200 epochs).

Advantages of Additional Hidden Layers

I suggest you play with the models. Try varying the number of layers, number of neurons, how to initialize the weights, and so on. If you invest some time, you can achieve an accuracy of more than 90% in a few minutes of running time, but that requires some work. If you try several models, you may realize that in this case, using several layers does not seem to accrue benefits vs. a network with just one. This is often the case.

Theoretically speaking, a one-layer network can approximate every function you can imagine, but the number of neurons needed may be very large, and, therefore, the model becomes much less useful. The catch is that the ability to approximate a function does not mean that the network is able to learn to do it, owing, for example, to the sheer number of neurons involved or the time required.

Empirically, it has been shown that networks with more layers require much smaller numbers of neurons to reach the same results and usually generalize better to unknown data.

In addition, having more layers may allow your network to learn different aspects of your inputs. For example, one layer may learn to recognize vertical edges of an image, and another, horizontal ones. Remember that in this chapter, I have discussed networks in which each layer is identical (up to the number of neurons) to all the others. You will see later, in Chapter 4, how you can build networks in which each layer performs very different tasks and is also structured very differently from another, making this kind of network much more powerful for certain tasks that have been discussed previously in this chapter.

You may remember that in Chapter 2, we tried to predict the selling prices of houses in the Boston area. In that case, a network with several layers might reveal more information about how the features relate to the price. For example, the first layer might reveal basic relationships, such as bigger houses equal higher prices. But the second layer might reveal more complex relationships, such as big houses with a smaller numbers of bathrooms equal low selling prices.

Comparing Different Networks

Now you should know how to build neural networks with a huge number of layers or neurons. But it is relatively easy to lose yourself in a forest of possible models without knowing which are worth trying. Suppose you start with a network (as I have done in the previous sections) with one hidden layer with five neurons, one layer with ten neurons (for our “softmax” function) and our “softmax” neuron. Suppose you have reached some accuracy and would like to try different models. At first, you should try increasing the number of neurons in your hidden layers, to see what you can achieve. In Figure 3-20, I have plotted the cost function as it decreases for different numbers of neurons. The calculations have been performed with a mini-batch gradient descent with a batch size of 50, one hidden layer with respectively 1, 5, 15, and 30 neurons, and a learning rate of 0.05. You can see how moving from one neuron to five immediately makes the convergence faster. But further increasing the number of neurons doesn’t result in much improvement. For example, increasing the neurons from 15 to 30 adds almost no improvement.

Let’s first try to find a way of comparing these networks. Comparing only the number of neurons can be very misleading, as I will show you shortly. Remember that your algorithm is trying to find the best combinations of weights and biases to minimize your cost function. But how many learnable parameters do we have in our model? We have the weights and the biases. You will remember from our theoretical discussion that we can associate a certain number of weights to each layer, and the number of learnable parameters in our layer l that we will indicate with Q^[l] is given by the total number of elements in the matrix W^[l], that is, n_ln_l − 1 (where we have n₀ = n_x by definition), plus the number of biases we have (in each layer we will have n_l biases). The number Q^[l] can then be written as

so that the total number of learnable parameters in our network (indicated here with Q) can be written as

I would like to draw your attention to networks A and B. Both have 25 neurons, but the parameter Q_A is much bigger (more than a factor of ten) than Q_B. You can easily imagine how network A will be much more flexible in learning than network B, even if the number of neurons is the same.

From Figure 3-20, let’s suppose we choose the model with 15 neurons as our candidate as our best model. Now let’s suppose we want to try a model with 3 layers, all with the same number of neurons, that should compete (and possibly be better) than our (for the moment) candidate model with 1 layer and 15 neurons. What should we choose as a starting point for the number of neurons in the three layers? Let’s indicate as model A the one with 1 layer with 15 neurons and as B a model with 3 layers with an (as of yet) unknown number of neurons in each layer, indicated with n_B. We can easily calculate the parameter Q for both networks

and

What value for n_B will give Q_B ≈ Q_A? We can easily solve the equation.

You should be able to solve a quadratic equation, so I will only give the solution here (hint: try to solve it). This equation is solved for a value of n_B = 14.4, but because we cannot have 14.4 neurons, we will have to use the closest integer, which would be n_B = 14. For n_B = 14, we will have Q_B = 11560, a value very close to 11935.

Our model with 3 layers with each 14 neurons could, however, be a good starting point for further testing.

Let’s discuss another point that is important when dealing with a complex dataset. Consider our first layer. Suppose we consider the Zalando dataset and we create a network with two layers: the first with one neuron and the second with many. All the complex features that your dataset has may well be lost in your single first neuron, because it will combine all features in one single value and pass the same exact value to all other neurons of the second layer.

Tips for Choosing the Right Network

I hear you crying, “You’ve discussed a lot of cases, given us a lot of formulas, but how can we decide how to design our network?”