The history of Deep Learning is intimately tied to the limitations of earlier attempts at using neural networks in machine learning and AI, and how these limitations were overcome with newer techniques, technological improvements, and the availability of vast amounts of data.

The perceptron is the basic neural network. Multi-layer networks are used in supervised learning and are built by connecting several hidden layers of neurons to propagate activations forward and using backpropagation to reduce the training error. Several activation functions are used, most commonly, the sigmoid and tanh functions.

The problems of neural networks are vanishing or exploding gradients, slow training, and the trap of local minima.

Deep learning successfully addresses these problems with the help of several effective techniques that can be used for unsupervised as well as supervised learning.

Among the building blocks of deep learning networks are Restricted Boltzmann Machines (RBM), Autoencoders, and Denoising Autoencoders. RBMs are two-layered undirected networks that are able to extract high-level features from their input. Contrastive divergence is used to speed up the training. Autoencoders are also deep learning networks used in unsupervised learning—they attempt to replicate the input by first encoding learned features in the encoding layer and then reconstructing the input via a set of decoding layers. Denoising Autoencoders address some limitations of Autoencoders, which can sometimes cause them to trivially learn the identity function.

Deep learning networks are often pretrained in an unsupervised fashion and then their parameters are fine-tuned via supervised fine-tuning. Stacked RBMs or Autoencoders are used in the pretraining phase and the fine-tuning is typically accomplished with a softmax activation in the output layer in the case of classification.

Deep Autoencoders are good at learning complex latent structures in data and are used in unsupervised learning by employing pre-training and fine-tuning with Autoencoder building blocks. Deep Belief Networks (DBN) are generative models that can be used to create more samples. It is constructed using a directed Bayesian network with an undirected RBM layer on top. Overfitting in deep learning networks can be addressed by learning with dropouts, where some nodes in the network are randomly "turned off".

Convolutional Neural Networks (CNNs) have a number of applications in computer vision. CNNs can learn patterns in the data translation-invariant and robust to linear scaling in the data. They reduce the dimensionality of the data using convolution filters and pooling layers and can achieve very effective results in classification tasks. A use case involving the classification of digital images is presented.

When the data arrives as sequences and there are temporal relationships among data, Recurrent Neural Networks (RNN) are used for modeling. RNNs use feedback from previous layers and emit output continually. The problem of vanishing and exploding gradients recurs in RNNs, and are addressed by several modifications to the architecture, such as Long Short Term Memory (LSTM) and Gated Recurrent Networks (GRU).

In this chapter's case study, we present the experiments done with various deep learning networks to learn from MNIST handwritten digit image datasets. Results using MLP, ConvNet, Variational Autoencoder, and Stacked RBM are presented.

We think that deep neural networks are able to approximate a significant and representative sub-set of key structures that the underlying data is based on. In addition, the hierarchic structures of the data can be easily captured with the help of different hidden layers. Finally, the invariance against rotation, translation, and the scale of images, for instance, is the last key elements of the performance of deep neural networks. The invariance allows us to reduce the number of possible states to be captured by the neural network (References [19]).