Defining deep networks

To further provide color to our definition of deep learning, here we define the four major architectures of deep networks:

• Unsupervised Pretrained Networks
• Convolutional Neural Networks
• Recurrent Neural Networks
• Recursive Neural Networks

There is continuous research in the domain of neural networks, but for the purposes of this lesson, we’ll focus on these four architectures. These architectures have evolved over the past 20 years. Let’s take a quick look at some of the highlights, with a history lesson on the history of feed-forward multilayer neural networks.

Evolutionary progress and resurgence

When we last left off in Chapter 1, neural networks had entered a “winter period” in the mid-1980s when the promise of AI fell short of what it could deliver. As happens many times when promising technology falls into the Trough of Disillusionment (Figure 2-1), there were many researchers still doing important work in the realm of neural networks.

Figure 2-1. Trough of Disillusionment (source: https://en.wikipedia.org/wiki/Hype_cycle)

One important development in neural networks was Yann LeCun’s work at AT&T Bell Labs on optical character recognition.³ His lab was focused on check image recognition for the financial services sector. Through this work, LeCun and his team developed the concept of the biologically inspired model of image recognition we know today as CNN. This eventually led to the creation of the MNIST handwriting benchmark (we cover this more later in the chapter) and a progressive number of record accuracy marks achieved by deep learning.

Advances in modeling sequential data with recurrent neural networks appeared in the late-1980s and early 1990s by researchers such as Sepp Hochreiter. As time went on, the research community created better artificial neuron variants (e.g., Long Short-Term Memory [LSTM] Memory Cell and Memory Cell with Forget Gate) over the course of the late 1990s.​ The stage for a neural network resurgence was being set quietly in research labs around the world.

During the 2000s, researchers and industry applications began to progressively apply these advances in products such as the following:

• Self-driving cars
• Google Translate
• Amazon Echo
• AlphaGo

Self-driving cars in the 2006 Darpa Grand Challenge used many techniques beyond just deep learning. The top teams (Stanford and Carnegie Mellon University) were able to take advantage of the big improvements of image processing.

Better image analysis allowed the planning systems in the cars to better choose paths through uncertain terrain and avoid obstacles more safely. Other advances in deep learning allowed models to more accurately translate and recognize audio data, driving value in the Google Translate and Amazon Echo line of products. Most recently, we’ve seen another complex game fall at the master level when the AlphaGo system beat the 9-dan professional Go player Lee Sedol.

Big advances in what machine learning can accomplish are not always easy to see. Public recognition for these advances many times is the culmination of many different lines of work that is exhibited in high-profile demonstrations such as The Darpa Grand Challenge or Watson beating Ken Jennings in Jeopardy. However, behind the scenes, the underpinnings for these advances change slowly but constantly. Just like the changing of the seasons, we don’t always notice these changes in our daily lives until they’ve crossed some threshold.

In the near future, we’ll continue to see deep learning being applied in unique and innovative ways. This application will be more of the latent intelligence variety (e.g., recommendations or voice recognition) coupled with pragmatic engineering to make them useful in everyday aspects of our lives. What we’re unlikely to see (in the near term, at least) are out of control malevolent artificial agents blowing us out of airlocks at inopportune times (think HAL 9000 from 2001: A Space Odyssey).

Deep learning continues to push the field forward in many domains and on many core machine learning problems. Here are just a few of the benchmark records deep learning has achieved in the last few years:

• Text-to-speech synthesis (Fan et al., Microsoft, Interspeech 2014)
• Language identification (Gonzalez-Dominguez et al., Google, Interspeech 2014)
• Large vocabulary speech recognition (Sak et al., Google, Interspeech 2014)
• Prosody contour prediction (Fernandez et al., IBM, Interspeech 2014)
• Medium vocabulary speech recognition (Geiger et al., Interspeech 2014)
• English-to-French translation (Sutskever et al., Google, NIPS 2014)
• Audio onset detection (Marchi et al., ICASSP 2014)
• Social signal classification (Brueckner & Schulter, ICASSP 2014)
• Arabic handwriting recognition (Bluche et al., DAS 2014)
• TIMIT phoneme recognition (Graves et al., ICASSP 2013)
• Optical character recognition (Breuel et al., ICDAR 2013)
• Image caption generation (Vinyals et al., Google, 2014)
• Video-to-textual description (Donahue et al., 2014)
• Syntactic parsing for natural language processing (Vinyals et al., Google, 2014)
• Photo-real talking heads (Soong and Wang, Microsoft, 2014)

Based on these accomplishments, we can easily project deep learning to impact many applications over the next decade. Some of the more impressive demonstrations of applied deep learning include the following:

• Automated image sharpening
• Automating image upscaling
• WaveNet: generating human speech that can imitate anyone’s voice
• WaveNet: generating believable classical music
• Speech reconstruction from silent video
• Generating fonts
• Image autofill for missing regions
• Automated image captioning (see also: https://github.com/karpathy/neuraltalk2)
• Turning hand-drawn doodles into stylized artwork

We probably won’t realize all of the major commercial applications until they’re right in front of our faces. Understanding the advances in deep network architecture is important in order to understand application ideas going forward.

Advances in network architecture

As research pressed the state of the art forward from multilayer feed-forward networks toward newer architectures like CNNs and Recurrent Neural Networks, the discipline saw changes in how layers were set up, how neurons were constructed, and how we connected layers. Network architectures evolved to take advantage of specific types of input data.

From feature engineering to automated feature learning

Although deep networks might have innovated with new units and layers for their internals, they are still fundamentally capped with a discriminatory classifier at the end with the constructed features as input. Automating feature extraction is a common theme among the various architectures. Each architecture does feature construction differently and is specialized such that it is better at certain types of input than others. Yann LeCun hit on this theme describing deep learning when he said it was: “machines that learn to represent the world.”

Geoffrey Hinton talks about this theme in DBNs when he explains how RBMs are used to decompose the data into higher-order features.⁵

Staying with the image classification theme, we can use the example of face detection. Raw image data of faces as input have issues with how the face is oriented, lighting of the photo, and position of the key features of the face. The key features we’d normally associate with a face are things like the edge of the face, edges of specific features like eyes and nose, and then subtle features we don’t consistently see like dimples.

Generative modeling

Generative modeling is not a new concept, but the level to which deep networks have taken it has begun to rival human creativity. From generating art to generating music to even writing beer reviews, we see deep learning applied in creative ways every day. Recent variants of generative modeling to note include the following:

• Inceptionism
• Modeling artistic style
• Generative Adversarial Networks
• Recurrent Neural Networks

Let’s quickly review each of these.

Modeling artistic style

Variants of convolutional networks have shown to learn the style of specific painters and then generate a new image in this style of arbitrary photographs. Figure 2-2 shows the amazing results. Imagine having your family photo painted by Vincent van Gogh. (By the time this lesson is published, this will probably be a Snapchat filter, so you won’t have to wait that long.)

Figure 2-2. Stylized images by Gatys et al., 2015

In 2015, Gatys et al. published a paper titled “A Neural Algorithm of Artistic Style”⁶ in which they separate the style and the content of a painting. The CNN extracts the artist’s style into the network’s parameters, which can later be applied to arbitrary images to be rendered in the same style.

The Tao of deep learning

There is a lot of marketing noise and hype in the realm of deep learning today, some of it justifiably so. However, deep learning is still trying to answer the same fundamental machine learning questions like: “Is this image a face?” The difference is that deep learning has taken the previous generation’s neural network techniques and added advanced automated feature construction to make computationally difficult questions on complex data easier to answer.

When you use deep learning as a practitioner, the best way to take advantage of this power is to match the input data to the appropriate deep network architecture. If you do this, you can apply deep learning successfully in new and interesting ways. If you don’t, you won’t add any new modeling power beyond basic techniques like logistic regression. The remainder of this lesson is dedicated to giving you, as practitioner, the skills and context necessary to make these decisions and use deep learning well.

Activation functions for general architecture

Depending on the activation function you pick, you will find that some objective functions are more appropriate for different kinds of data (e.g., dense versus sparse). We group these design decisions for network architecture into two main areas across all architectures:

• Hidden layers
• Output layers

Hidden layers are concerned with extracting progressively higher-order features from the raw data. Depending on the architecture we’re working with, we tend to use certain subsets of layer activation functions. As we work through this chapter, we’ll illustrate these patterns more across DBNs, CNNs, and Recurrent Neural Networks. In Chapter 3, we take a deeper look at the impacts of different activation functions on different network architectures in the context of tuning deep networks.

Reconstruction cross-entropy

With the reconstruction entropy loss function, we first apply “Gaussian noise” (a kind of statistical white noise), and then the loss function punishes the network for any result that is less similar to the original input data. This feedback drives the network to learn different features in an attempt to reconstruct the input more effectively and minimize error. In deep learning, reconstruction entropy loss is used for feature engineering in the pretrain phase that involves RBMs.

First-order methods

The Jacobian, as mentioned, is a matrix of partial derivatives of the loss function with respect to the parameters in the network. In practice, we calculate it at a specific point—the current values of the parameters.

If we think about taking one step at a time to reach an objective, first-order methods calculate a gradient (Jacobian) at each step to determine which direction to go in next. This means that at each iteration, or step, we are trying to find the next best possible direction to go, as defined by our objective function. This is why we consider optimization algorithms to be a “search.” They are finding a path toward minimal error.

Gradient descent is a member of this path-finding class of algorithms. Variations of gradient descent exist, but at its core, it finds the next step in the right direction with respect to an objective at each iteration. Those steps move us toward a global minimum error or maximum likelihood.

Stochastic gradient descent (SGD) is machine learning’s workhorse optimization algorithm. SGD trains several orders of magnitude faster than methods such as batch gradient decent, with no loss of model accuracy.

The strengths of SGD are easy implementation and the quick processing of large datasets. You can adjust SGD by adapting the learning rate (e.g., with methods such as Adagrad, discussed shortly) or using second-order information (i.e., the Hessian), as we’ll see next. SGD is also a popular algorithm for training neural networks due to its robustness in the face of noisy updates. That is, it helps you build models that generalize well.

Second-order methods

All second-order methods calculate or approximate the Hessian. As described earlier, we can think of the Hessian as the derivative of the Jacobian. That is, it is a matrix of second-order partial derivatives, analogous to “tracking acceleration rather than speed.” The Hessian’s job is to describe the curvature of each point of the Jacobian. Second-order methods include:

• Limited-memory BFGS (L-BFGS)⁹
• Conjugate gradient¹⁰
• Hessian-free¹¹

Think of these optimization algorithms as a black-box search algorithm that determines the best way to minimize error, given an objective and a defined gradient relative to each layer.

Layer size

Layer size is defined by the number of neurons in a given layer. Input and output layers are relatively easy to figure out because they correspond directly to how our modeling problem handles input and ouput. For the input layer, this will match up to the number of features in the input vector. For the output layer, this will either be a single output neuron or a number of neurons matching the number of classes we are trying to predict.

Deciding on neuron counts for each hidden layer is where hyperparameter tuning becomes a challenge. We can use an arbitrary number of neurons to define a layer and there are no rules about how big or small this number can be. However, how complex of a problem we can model is directly correlated to how many neurons are in the hidden layers of our networks. This might push you to begin with a large number of neurons from the start but these neurons come with a cost.

Depending on the deep network architecture, the connection schema between layers can vary. However, the weights on the connections are the parameters we must train. As we include more parameters in our model, we increase the amount of effort needed to train the network. Large parameter counts can lead to long training times and models that struggle to find convergence.

Magnitude hyperparameters

Hyperparameters in the magnitude group involve the gradient, step size, and momentum.

Regularization

Let’s dig deeper into the idea of regularization that we touched on in Chapter 1. Regularization is a measure taken against overfitting. Overfitting occurs when a model describes the training set but cannot generalize well over new inputs. Overfitted models have no predictive capacity for data that they haven’t seen. Geoffery Hinton described the best way to build a neural network model:

Cause it to overfit, and then regularize it to death.

Regularization for hyperparameters helps modify the gradient so that it doesn’t step in directions that lead it to overfit. Regularization includes the following:

• Dropout
• DropConnect
• L1 penalty
• L2 penalty

Dropout and DropConnect mute parts of the input to each layer, such that the neural network learns other portions. Zeroing-out parts of the data causes a neural network to learn more general representations. Regularization works by adding an extra term to the normal gradient computed.

Mini-batching

With mini-batching,¹⁶ we send more than one input vector (a group or batch of vectors) to be trained in the learning system. This allows us to use hardware and resources more efficiently at the computer-architecture level. This method also allows us to compute certain linear algebra operations (specifically matrix-to-matrix multiplications) in a vectorized fashion. In this scenario we also have the option of sending the vectorized computations to GPUs if they are present.

Network layout

There are five main parts of a basic RBM:

• Visible units
• Hidden units
• Weights
• Visible bias units
• Hidden bias units

A standard RBM has a visible layer and a hidden layer, as shown in Figure 2-3. We can also see a graph of weights (connections) between the hidden and visible units in the figure. Think of these weights in the same way you think of weights in the classical neural network sense.

Figure 2-3. RBM network

With RBMs, every visible unit is connected to every hidden unit, yet no units from the same layer are connected. Each layer of an RBM can be imagined as a row of nodes. The nodes of the visible and hidden layers are connected by connections with associated weights.

Training

The technique known as pretraining using RBMs means teaching it to reconstruct the original data from a limited sample of that data. That is, given a chin, a trained network could approximate (or “reconstruct”) a face. RBMs learn to reconstruct the input dataset. We’ll review the concept of reconstruction in the next section.

Reconstruction

Deep neural networks with unsupervised pretraining (RBMs, autoencoders) perform feature engineering from unlabeled data through reconstruction. In pretraining, the weights learned through unsupervised pretrain learning are used for weight initialization in networks such as Deep Belief Networks.

Figure 2-4 presents a visual explanation of the network involved in reconstruction in RBMs.

Figure 2-4. Reconstruction in RBMs

We can visually explain reconstruction in RBMs by looking at the MNIST dataset. MNIST stands for the mixed National Institute of Standards and Technology dataset that contains the images. The MNIST dataset is a collection of images representing the handwritten numerals 0 through 9. Figure 2-5 depicts a sample of some of the handwritten digits in MNIST.

Figure 2-5. Sample of MNIST digits

The training dataset in MNIST has 60,000 records and the test dataset has 10,000 records. If we use a RBM to learn the MNIST dataset, we can sample from the trained network to see¹⁹ how well it can reconstruct the digits. Figure 2-6 shows renderings of MNIST digits being progressively reconstructed with a RBM.

Figure 2-6. Reconstructing MNIST digits with RBMs

If the training data has a normal distribution, most of them cluster around a central mean, or average, and become scarcer the further you stray from that average. It looks like a bell curve. If we know the mean and the variance, or sigma, of normal data, we can reconstruct that curve. But suppose that we don’t know the mean and variance. Those are parameters we then need to guess. Picking them randomly and contrasting the curve they produce with the original data can operate similarly to a loss function. We measure the difference between two probability distributions much like we measure erroneous classifications, adjust our parameters, and try again.

Other uses of RBMs

Here are some other places we see RBMs used:

• Dimensionality reduction
• Classification
• Regression
• Collaborative filtering
• Topic modeling

Similarities to multilayer perceptrons

Autoencoders share a strong resemblance with multilayer perceptron neural networks in that they have an input layer, hidden layers of neurons, and then an output layer. The key difference to note between a multilayer perceptron network diagram (from earlier chapters) and an autoencoder diagram is the output layer in an autoencoder has the same number of units as the input layer does.

Figure 2-7 presents an example of an autoencoder network.

Figure 2-7. Autoencoder network architecture

Beyond the output layer, there a few other differences, which we outline in the next section.

Defining features of autoencoders

Autoencoders differ from multilayer perceptrons in a couple of ways:

• They use unlabeled data in unsupervised learning.
• They build a compressed representation of the input data.

Training autoencoders

Autoencoders rely on backpropagation to update their weights. The main difference between RBMs and the more general class of autoencoders is in how they calculate the gradients.

Common variants of autoencoders

Two important variants of autoencoders to note are compression autoencoders and denoising autoencoders.

Applications of autoencoders

Building a model to represent the input dataset might not sound useful on the surface. However, we’re less interested in the output itself and more interested in the difference between the input and output representations. If we can train a neural network to learn data it commonly “sees,” then this network can also let us know when it’s “seeing” data that is unusual, or anomalous.