Neural Networks

For complex and unstructured data, we build deeper models that use a combination of smaller individual learning units to form a bigger network. This network of learning units can learn complex patterns from a large number of features. This is called a neural network.

A common analogy that is used to represent this is the human brain, which contains a network of biological cells called neurons that are connected via axons and dendrites. If you recall from your biology textbooks, we have signals flowing into neurons through dendrites and processed outputs flowing to other neurons or muscles through axons. In fact, neural networks are heavily inspired by the structure of the human brain. Figure 4.1 shows a simple representation.

Similar to the human brain, these artificial neural networks contain processing units called neurons and connections between them. These networks are structured into layers, with each layer extracting valuable information from the data that is fed to it. These are Deep Learning networks and they have many layers of learning. They try to map the input space to a set of possible outcomes or classes. Let's look at a very simple neural network, shown in Figure 4.2. Let's understand some basics and then we will get into building more complex networks.

This neural network has three inputs in the first input layer. The second layer of neurons is the hidden layer, with three neurons again. The final layer is the output layer, with one neuron.

Each neuron is a learning unit that may receive inputs from other neurons, perform some calculations, and send output to other neurons. The flow of information is shown by the arrows in Figure 4.2. Now let's look closer at what happens at the neuron stage.

Let's revisit our diagram from the logistic regression function in Chapter 2. We first calculated a weighted sum of inputs (Z1) and then applied a function to return a value A1 between 0 and 1. This result can also be called an activation. This is exactly what is happening at each neuron in the neural network. You have inputs coming into each neuron, activations being calculated using weights, and an activation function—and these activations feed into the next neuron in the network. This is how neural networks work. See Figure 4.3.

This representation is often called a computational graph or dataflow. You have the nodes as circles where inputs are entered or some computation is done. The edges of this graph represent weights. You can look at this as a flow of data through the edges between nodes, with each node adding some processing to the data. Let's get back to our simple neural network in Figure 4.2.

A neural network has many neurons or learning units organized into layers. Inputs flow into each network layer and calculated outputs or activations move to the next layer. If we have a small number of layers, typically two or three in a network, we call them shallow networks. These take less processing time and can quickly calculate results. However, they cannot learn complex patterns, especially with unstructured data. The basic Machine Learning models we saw in Chapter 2 typically have two layers—one input and one output layer like linear and logistic regressions. These are the shallow learning models. We can find out what's going on inside them and they can be quickly trained—in a few milliseconds.

When we have to capture complex and non‐linear patterns in the data that would not be possible by simpler shallow learning models, we need models with many layers called Deep Learning models. Deep Learning models learn in stages, or layers, with each layer extracting some pattern that is fed into the next layer.

For example, if you are learning to detect faces in an image, your deep network will take as input the pixel array of the image. Then, during the first stage, it may learn to detect lines and curves. Next, it will combine these to form figures like rectangles and circles. Finally, it will combine these to recognize any pattern of faces. This is the power that deep neural networks give us. They learn complex patterns in the data and capture non‐linear relations extremely well.

The last layer of the neural network is the output layer, and the number of neurons there correspond to the number of outputs we want to learn. If we just want to make a prediction as to buy/don't buy based on housing variables, then our network will have a single neuron in the output layer with its value determining the buy decision. Now if we want to also predict another variable, we can add that as a neuron to the output layer. This new output should be considered in the training data we provide to the network while training. That's it—there is no special consideration needed and we can use the same network to predict two outputs instead of one.

The key difference between deep neural networks and the other ML algorithms we saw in Chapter 2 is that deep networks learn important features of the data on their own. We configure the inputs and outputs we seek, decide on the numbers of layers and neurons in each, and build a good training dataset. The network learns all the complex patterns in the data and establishes correlations between inputs and outputs. Basically, it maps the input space (Xs) to the output space (Ys). Hence, neural networks are often referred to as blackboxes, because they don't really tell us how they find these relations; they only predict outputs by internally capturing these relationships.

Because these networks are complex, we often analyze them by considering individual layers. Let's look at the neural network shown in Figure 4.2 again. We have an input layer with three neurons representing input Xs. There is one hidden layer with three neurons and an output layer with one neuron—the Ys.

Now this was just one example of a neural network and a pretty simple one at that. We only have layers where every neuron in the layer is connected to every other neuron in the subsequent layer. Such a layer of neurons is called a fully‐connected or dense layer. In a dense layer, every neuron learns features by considering the output generated from every neuron from the previous layer. Hence, these layers tend to be memory consuming. In practice you will find these layers at the end of a deep network to learn from features extracted in earlier layers and make predictions. The earlier layers in a network may have more local connections to extract features. We look at some of these feature‐extracting layers in Chapter 5, which discusses advanced Deep Learning.

The hidden and output layer neurons do exactly what we discussed earlier for a logistic regression unit. They get a weighted sum of inputs and apply an activation function. At each neuron these calculations are done and the results are fed forward to the next layer of neurons. This architecture, with all neurons feeding their output forward, is called a feed‐forward architecture. At each layer, many calculations occur that are handled in parallel using multi‐dimensional arrays. We will not go into detail about the equations, but it will help to get an understanding of weights in layers. Let's populate a few weights into our diagram, as shown in Figure 4.4.

You will see different conventions used in books and articles. Let's say we have the weight W1‐32 for layer 1 and it connects neuron 3 from one layer to neuron 2 to the next layer. We can see that between the first hidden layer and input layer, we have (2 × 3), which is six weights. Then, between the output and hidden layers, we have (3 × 1), which is three weights. For this simple network, we have 6 + 3, which is nine weights. These are the weights that need to be “learned” during the training process.

Now let's add a special type of neuron to this network called a bias neuron. We saw the importance of bias in Chapter 2 in the linear regression equation. Bias helps the network learn certain assumptions about the data so that it does not depend only on the variables for generating results. All the inputs (X1, X2, X3) in our network are zero, which means both outputs (Y1, Y2) will always be zero, regardless of the weights. The network has no bias that influences its values in the absence of inputs.

All right, so let's add bias neurons to this network. Bias neurons don't do any calculations. We can just add a constant value of +1 and, just like other neurons, they have weights associated with them. See Figure 4.5. We will use the letter B to associate these weights.

The convention used here is B2‐1, which is the weight associated with a bias neuron from the first layer to neuron 1 from the next layer. We have added bias neurons to the input and hidden layers. Adding to the output does not make sense since we don't calculate anything from this layer.

So, the total number of bias weights will be 3 + 1, which is four. Adding this to the previous nine weights, we have a total of 13 weights that this network has to learn. Now let's talk about the Activation functions.

Activation functions take the weighted sum of inputs at each neuron and apply some non‐linear function to them. Popular functions used are Tanh, Sigmoid, and ReLU (Rectified Linear Units). Each function helps in thresholding the output value based on the weighted sum of inputs. Usually you apply the same Activation function to a particular layer. So, every neuron in that layer uses the same Activation function. The “References” section at the end of the book includes a reference to material, with details on each Activation function. But as a rule of thumb, Tanh and ReLU are used for hidden layers and ReLU is more common. Sigmoid is mainly used for output layer neurons. Sigmoid produces a value between 0 and 1 so it can be used for classification problems.

Back‐Propagation and Gradient Descent

Let see how neural networks are trained. I will avoid fancy formulas so you can understand the concepts clearly and then we will see a code example.

In Chapter 2, we saw an overview of the gradient descent method to calculate gradients and optimize weights as part of the learning process. We follow the same process for training a neural network, but at a network level. The most popular algorithm used for training neural networks is called back‐propagation. Back‐propagation is essentially a smart way for calculating the partial derivatives (gradients) of the Cost function with respective to different weights.

The idea is to have a Cost function similar to mean squared error (MSE) that we saw during regression model training. Then we adjust the weights using Gradient Descent so as to minimize this Cost function. To do this, we calculate the partial derivative of the Cost function with respect to each weight value. Then, based on the error term, we use this partial derivative to find the magnitude and direction of the change in weight and apply the change. After every iteration, the Cost function is calculated and the weights are updated. See Figure 4.6.

Before starting the training, we must establish a cleansed and normalized training dataset. This should contain data points with all our input features (Xs) and corresponding expected outputs (Ys). This will be treated as something referred to in the ML community as the ground truth. The model we train will try to learn patterns so it can be good enough to generate results as good as the ground truth. In other words, the ground truth is the standard that our ML model will aim to achieve.

Establishing the ground truth and defining a good training and testing set is a general starting point in the Machine Learning project lifecycle. We cover the ML lifecycle in detail in Chapter 9. For now, you can assume that we have the data for the Xs and Ys properly cleansed and available and we can use it as‐is for training.

Here are the general steps involved in back‐propagation training of neural networks:

1. Initialize the weight values to zero or random numbers. Run all the data points in consideration through the network and predict Ys of each X in the dataset.
2. Compare each Y value with the expected outcomes or the ground truth. Find the difference in the values. Based on the difference in values, calculate the error for each output term.
3. Establish a Cost function, which is basically a function of all the weights in the network—include weights at every layer of the network including bias weights. The Cost function helps us define a metric to how far our model predictions are from the ground truth. The choice of Cost function is very important in training a good ML model.
4. The Cost function may be the same as the mean absolute error (MAE) or mean square error (MSE) that we used earlier. This Cost function is ideal when we are predicting a value and we can directly see how far away our predictions are from real values using the MAE or MSE. If we are predicting a class then the preferred Cost function is cross‐entropy. Cross‐entropy tries to minimize the information loss by doing a wrong classification and hence it helps capture the classification loss better.

   The objective of having a Cost function is to establish a relationship between the weights and the error in the prediction. So, as we tune the weights, the error reduces and we get an accurate model. (I provide a link to an article explaining different cost functions in detail in the “References” section at end of the book.)

5. As the name suggests, we back‐propagate the error calculated through the network. As we go back from the outputs to the inputs, we update the weights using gradients of the Cost function with respect to the corresponding weight. This is the same gradient descent algorithm we saw in Figure 4.5, but we are now applying it to the whole network.

   We establish an overall Cost function of the network and, using partial derivatives, we calculate the gradient of this Cost function with respect to each weight. Now we start from the last layer and calculate the gradients from the errors between predicted values and ground truth. We back‐propagate this error from last to first layer and thus calculate gradients for each neuron in every layer. The gradients are then used to adjust the weight values at each neuron connection in a layer. The details of this algorithm are excellently explained by Andrew Ng in his video from the ML class. I include a link to it in the “References” section at end of the book.

6. With all the weights adjusted, we run all the data points again and find the Error term. We iterate through this process until so many iterations are completed or until we achieve an acceptable Error value.

We will see this process in action with an example in Keras shortly.

Batch vs. Stochastic Gradient Descent

Gradient Descent applied to neural networks may be of the batch or stochastic type. We run back‐propagation using training data points and adjust the weights. When all the training data points have passed through the entire network it is known as an epoch. In Batch Gradient Descent, we wait for a whole epoch to pass so our network has seen all the training data and then adjust weights. This usually takes a long time and needs to store variables in memory. This approach takes a long time, but it helps us find the optimum value for weights in one go. Usually, if we have a limited training dataset and lots of memory, we follow this approach.

The other type of gradient descent is stochastic gradient descent (SGD), where we adjust weights after the pass of every data point through the network. Here, we don't store much data in memory and rapidly update weights. This is a very fast method, but we see fluctuations in the training because we tend to overshoot the local minima. This approach works when we have limited memory and a large training dataset to work with. We learn at every data point and update our model. The problem with this approach is that it's possible to get “lost” and move away from the minima, especially when we have some bad data points, which may cause major errors. Hence, in practice, we use a compromise of these two approaches.

That compromise is called mini‐batch gradient descent. Here, we divide our data into smaller batches and update weights after each pass of each batch through the network. This is found to be a better approach to training neural networks. At each iteration of training, a smaller subset of the training data is loaded in memory to calculate errors and back‐propagate that error to get gradients. Hence, the memory (RAM) used by the algorithm is less, as compared to loading the whole training dataset in memory. Also, we don't need to wait until all the training data is processed to see results. We usually see a faster convergence to the minima using this approach and reduce training time by a few magnitudes.

Neural Network Architectures

This particular neural network in Figure 4.4, where we have all layers as dense or fully‐connected, is known as a multi‐layered perceptron (MLP). It is very good at learning patterns and has several applications especially around finding non‐linear patterns in structured data. If you can have your data as a single‐dimensional vector, then the MLP can quickly learn patterns due to its fully connected nature and make predictions. It works really well for structured data. We can apply MLP to unstructured data like images, but with some modification. Since the MLP handles data in single‐dimensional layers, we have to convert the three‐dimensional image data into a large flattened vector and feed it to the MLP. Hence, all the valuable spatial information that is stored in the three dimensions is lost and we have one large vector. There are some other deep architectures that extend the idea of MLP and are more suitable for specific types of unstructured data, which we discuss in Chapter 5. Before getting into their details, let's first build an example of an MLP neural network.