Deep Learning Function Approximation

Critical to understanding what deep learning does is understanding that a network is just a good function approximator . In fact, simply put, that is all deep learning systems do. They just approximate known or unknown functions well. Now this can be a bit confusing when we think about how neural networks classify, so let’s look at an example.

Consider Figure 3-1, which classifies images of cats and dogs into two groups using a function called logistic regression . In the figure we can see the boundary line representing the function output. On one side of this function the images of cats will fall and on the other the images of dogs. This would be an example of a deep learning network that learns an unknown function. We know there is a function that can define the classification between cats and dogs visually and mathematically; we just don't know what it is.

When a network learns to classify images, it learns the logistics or set of logistic regression functions that break things into classes. While we know there is a set of functions that define the boundary to those classes, in most cases we don’t worry about the exact function. Instead, through training and learning, our deep learning system learns or approximates the function on its own.

To demonstrate how networks do function approximation, we are going to jump into another code example. Exercise 3-1 borrows from a previous regression example, and instead of using data, we can define a known function. Using a known function will demonstrate with certainty how our networks learn.

EXERCISE 3-1. DEEP LEARNING FUNCTION APPROXIMATION

1. 1.

   Open the GEN_3_function_approx.ipynb notebook from the GitHub project site. If you are unsure how, then consult Appendix B.

    
2. 2.
   Run the whole notebook by selecting Runtime ➤ Run all. Then look at the top cells, which define a simple function that we will approximate here:

   def function(X):

     return X * X + 5.

   X = np.array([[1.],[2.],[3.],[4.],[5.],[6.],[7.],[8.],[9.],[10.]])

   y = function(X)

   inputs = X.shape[1]

   y = y.reshape(-1, 1)

   plt.plot(X, y, 'o', color='black')
    
3. 3.

   The function function defines a parabolic equation we will train our network on. In the next cell, you can see how we set up hard-coded X inputs 1–10 that we use to define our learned outputs, y. Notice how we can simply feed the set of inputs X into the function to output our labels or learned values, which are labeled y.

    
4. 4.

   Figure 3-2 shows the plotted output of the equation using a sample set of inputs (1–10). This is the equation we want our network to learn how to approximate.

    

1. 5.
   We have already reviewed most of the remaining code in previous chapters. Our focus now will be on identifying slight differences. Notice how we still break our input data into training and testing splits with the following:

   X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

   num_train = X_train.shape[0]

   X_train[:2], y_train[:2]

   num_train
    
2. 6.

   With only 10 input points, it may seem like a silly step to break the data up into 8/2 points, respectively. However, even when tuning to a known function, it can help validate our results, so we still like to do validation/test splits.

    
3. 7.
   The only other change to this notebook is another step of validation we do at the end with the following code:

   X_a = torch.rand(25,1).clone() * 9

   y_a = net(X_a)

   y_a = y_a.detach().numpy()

   plt.plot(X_a, y_a, 'o', color='black')
    
4. 8.

   This section of code creates 25 random values using torch.rand. Since the values are from 0–1, we multiply the output by 9 to scale the values from 0–9. We then run the values X_a through the network net and plot the output shown in Figure 3-3.

    

1. 9.

   Notice in Figure 3-3 how the network is excellent at approximating the middle values of our known function. However, you can see that the network struggles at the bottom (close to X of 0) and the top end. You can see that the network has greater difficulty approximating the boundaries. Yet it still does very well matching the function’s middle values.

    

What we just demonstrated is how a simple network can learn to approximate a function. However, we did notice how the network still struggled to approximate the boundaries, and we need to understand why that is.

Part of the problem is the limited amount of data we used to train the network. After all, we used only 10 points. However, the function approximation through the center is very good compared to what happens at the ends. So, why is that?

Simply put, a deep learning system can approximate well to a known or unknown equation within the bounds of the known data or limits of the equation. In our last example, we used data values from 0–10, with bounds of 0 and 10. Yet we see from Figure 3-3 that the function approximation falls down around an input of X=7 or so. As well, on the lower bounds we can see an issue when X<1.

The reason for these inconsistencies is not about the data or indeed the limits to the data but rather the calculus itself. Remember our discussion about auto-differentiation with calculus in Chapter 2? Now the problem isn’t the auto-part of differentiation but rather the differentiation itself.

Calculus helps us to understand the rate of change in an equation or function. This is useful for everything from launching spaceships to building better buildings and of course deep learning. It is also important for us to understand why calculus is used to approximate the functions in deep learning, which is covered in the next section.

The Limitations of Calculus

If you have studied calculus at all, you have learned the limitations of calculus. One of them is that all functions we differentiate with calculus need to be continuous. Figure 3-4 shows example functions with areas called discontinuities. These areas are examples of functions that would prevent a deep learning system from learning or being able to approximate the function.

In generative modeling, you can think of a discontinuity as a gap or vacuum in the data. Now don’t confuse a gap in this context with space or distance. We use the term gap here to refer to a break where nothing in the function has any sample values. The term space or latent space or hidden space is used to define the unknown area in a continuous set of data.

Generally, when we see black regions in generated output, that represents a problem with limits or discontinuities in the input data.

Understanding the difference between continuous and discontinuous data will take some time, and sometimes it may not be so obvious. The key takeaway here is to understand that function approximation has limits, and those limits are often defined by the data itself, sometimes requiring a network to approximate a continuous function over discontinuous data.

Now, getting back to limits, let’s understand how limits can affect how a network learns or approximates to data. In our exercise example, our network had limits of 1 and 10. If we refer to Figure 3-3, we can see that our function flattens at X<1, which makes sense considering the limit of 1. This means our network just limits the function approximation to y=5 when X is less than 0.

What is not so obvious, however, is why our function approximation fails on values greater than 7 when the limit was 10. To understand why this issue arises, we need to understand how deep learning uses calculus to approximate functions, which is covered in the next section.

Deep Learning Hill Climbing

At the heart of any deep mathematical concept there is always a grounded intuition that explains the why. Unfortunately, most classic mathematics courses often miss explaining the why. Instead, they rely on the student to uncover the intuition on their own through performing proofs or regurgitating equations. This is something that works well for budding mathematicians but tends to overwhelm everyone else. For that reason, we will always focus more on the why of the math rather than the specific how.

Calculus is a tool we use in deep learning systems to balance the weights/parameters of a network. Recall that calculus works by understanding the rate of change of a system. In deep learning, calculus allows us to determine how much change and to where we need to adjust the weights in a network.

Calculus and gradient optimization or gradient descent is not the only tool we can use to find the weights in a network. It is currently the most efficient, but deep learning systems have and continue to use other machine learning methods. If you are interested in learning more, Google deep learning PSO or deep learning GA.

Figure 3-4 shows a one-dimensional function with a ball at position 1. We can think of the goal of a deep learning system as guiding that ball over the function. Further yet, we want the ball to be able to follow that function as close as possible. The better that ball is able to follow the function path, the better our networks will predict or generate content.

Notice that in Figure 3-4 we are moving the ball an amount calculated through automatic differentiation and our optimization method multiplied by the learning rate (alpha). This is labeled “Gradient X alpha” in the figure. The amount of gradient can be altered by selecting different optimization methods and/or tuning the hyperparameters. Tuning the optimizer, be it stochastic gradient descent or Adam, is something we will cover later.

That leaves us with alpha, the learning rate or learning parameter. This hyperparameter allows us to control how much tuning a network uses to map that ball to a function. In Figure 3-5 and in our previous example, we are applying a high learning rate. As you can see, this is demonstrated with the movement of the ball to position 2 and then 3. Notice how the ball moves well above the function line in the third position. This is what causes our problem in the previous example.

In Figure 3-6, we can see how reducing the learning rate allows the network to better approximate the function path. You can see with the addition of the boundaries it also makes it critical that our ball stays with bounds. When the ball bounces outside the maximum bounds, the value becomes maxed. This is the reason for the maximum values not mapping well in Figure 3-3.

The primary takeaway from this is that the learning rate is often our most sensitive hyperparameter when mapping to functions. Learning rate, or alpha, is almost always the first to tune and understand for your network. Alpha is also the key to tuning how quickly a network will train. When alpha is too large, the network will overshoot the function. Conversely, keeping alpha very small allows the ball to learn the function well. However, this can come at great cost of computing performance.

Consider a learning rate of .001 compared to a value of .00001. Say we can train a network with the smaller rate in, say, 10 minutes. The smaller rate, being 100 times smaller, would likely take 100 times longer due to the increase in training iterations, increasing training times to 10 × 100 = 1000 minutes, or more than 16 hours. This is all from a smaller but unfortunately less efficient learning rate.

To demonstrate this key concept further, let’s look at another example. Exercise 3-2 shows how we will tune the alpha (learning rate) to better approximate the function in our previous exercise.

EXERCISE 3-2. TUNING ALPHA FOR EFFICIENT LEARNING

1. 1.

   Open the GEN_3_learning_rate.ipynb notebook from the GitHub project site. If you are unsure how, then consult Appendix B.

    
2. 2.
   Run the whole notebook by selecting Runtime ➤ Run all. Notice that where we define the optimizer, as shown in the following code, is where we set the learning rate lr.

   loss_fn = nn.MSELoss()

   optimizer = torch.optim.Adam(net.parameters(), lr = .001)
    
3. 3.

   In our previous example of GEN_3_function_approx.ipynb, our learning rate was .025, a value that was on the cusp of being able to correctly map the function.

    
4. 4.

   Our goal here is to find the learning rate that will allow the network to map 100 percent to the function or close to it.

    
5. 5.

   Change the learning rate lr in the previous block of code and rerun the entire notebook by selecting Runtime ➤ Run all.

    
6. 6.

   See if you can tune the lr value to match or approximate Figure 3-7. What happens if you make the learning rate really small?

    
7. 7.
   If your network completes training before finding the function, this is easily fixed by increasing the number of epochs. Change num_epochs to something larger and see what results this has.

   num_epochs = 8000

   y_train_t = torch.from_numpy(y_train).clone().reshape(-1, 1)
    
8. 8.

   Attempt to find the smallest number of epochs and the greatest learning rate that will train the network to match the function x2 + 5.

    
9. 9.

   If you are feeling more advanced, try altering the number of neurons in each layer or altering the network itself.

    

As you worked through the previous exercise, it may have occurred to you that there may be limitations to the network itself and the data. In fact, that is exactly the case in the exercise. For this particular example, the network attempts to overfit the data. We will look at how to resolve both over- and underfitting in the next section.

Over- and Underfitting

Oftentimes when newcomers design networks, their first assumption is that having more neurons means better results. Unfortunately, this is rarely the case for a number of reasons that we will understand in later chapters. For now, we want to focus on how the network size itself can overfit or underfit to the function.

You may often hear deep learners mention their network is over- or underfitting to the data. What they really mean is that the network is over- or underfitting to the function that matches or categorizes that data.

Figure 3-8 shows our previous example plotted against the function x2 + 5 to see how well we were actually matching to the results. As you can see, we are not matching well—in fact not really well at all considering the size of our network. However, the size of our network isn’t the entire problem. Network size and the amount of data go almost hand in hand.

Think back to our previous exercise and our attempts to approximate to the function x2 + 5. Our first couple of attempts used a set number of data points from 1 to 10. We fed those data points into the function to generate y outputs. This generated our set of trainable data, which we then broke up into training and testing splits.

Figure 3-9 shows the plot of the 10 data points we generated to train and approximate the function. Of those points, we are removing 20 percent or 2 for test. This decreases the number of points in the dataset and in essence the size of space in between our training data.

That means when our model goes to approximate the function x2 + 5, it has large areas between data. We call these areas the latent or hidden space. Again, this is not to be confused with a gap, chasm, or vacuum in the data. This hidden space in the data represents an area of the function missing any actual training data. That means the network must approximate values between these data points.

As it turns out, a big part of generative modeling is understanding how to extract results from this latent space, as we will see throughout this book. It’s not unlike how we tried to extract the results of our previous function approximation exercise. Our previous assumption may have been to reduce the number of training points, but as we can see now, this increases the size of the latent space, causing our model to overfit to the function.

Overfitting is caused when a model approximates to a function but wrongly fills or approximates the latent space. In our previous function approximation examples, this was clearly the case, which was caused by two problems. The first was the lack of data, which caused the network to estimate large areas of latent space. The second was the size of our network, which had too much power to estimate the space in data. So, it essentially made up values, which can be a problem.

When a network is large, it has a lot of power, in this case too much power. Large networks can memorize data of all forms including images or video. This may be useful for networks scoped to specific data, but in generative modeling as well as most other machine learning disciplines, our goal is always to generalize.

Generalizing is always a goal we will undertake when building models. That means the data we feed into the model needs to be well distributed. Well-distributed data provides us with even latent space between data. In our previous example, our data was distributed into 10 bins, with values from 1–10. However, as we can see, our data was not distributed well enough.

Generalizing also means that our network should be only the size it needs, no more or less. If our network is too large, it will tend to overfit or fill in those latent space with incorrect values. Likewise, networks that are too small will underfit. Underfitting is often a consequence of too large of gaps or latent space in a model.

Figure 3-10 shows the classic example of underfitting to the x2 + 5 function by using a linear model or approximating a line. Underfitting in this example would be the result of the model not being complex enough. However, it can also occur when a network is not being fed enough data or that data is poorly distributed.

To understand this in a tangible manner, our next exercise will focus on how we create models that over- and underfit to a set of function data. We will continue to use the base of our previous exercise, but this time we will modify some details to understand the problem better; see Exercise 3-3.

The intention of the previous exercise was to demonstrate how our networks can over- and underfit the function that maps to our data. In this simple example, we could see quick and explicit feedback for what each of our hyperparameters did, allowing us to understand how a network can over- or underfit.

As we progress through this book and use more complex forms of data, like images, over- and underfitting often becomes less obvious. Fortunately, there are several clues that can help us identify these types of problems that we explore as we progress to those exercises.

One key element we didn’t cover and can cause a lot of over/underfitting issues is data distribution. In the examples, the data was uniformly distributed across a range of values. This is rarely something we see in the real world.

For example, assume you had a dataset of 30,000 animal images (10,000 cats, 15,000 of dogs, and 5,000 birds) that you wanted to classify into cats, dogs, and birds. The problem is your data is not evenly distributed. You have more pictures of dogs, and hence your model will overestimate or memorize dogs. Instead, what we should do is reduce the number of cat and dog images to 5,000 to match the number of bird images. You could still argue that the cat and dog are more visually similar than a bird, thus potentially influencing your results.

We use and understand how data is distributed in several ways in generative modeling. It is in fact a foundation of many of the methods. In the next section, we will understand why data distributions are important to the variational autoencoder.