Autoencoders, also known as Diabolo networks or autoassociators, was initially proposed in the 1980s by Hinton and the PDP group [1]. They are feedforward networks, without any feedback, and they learn via unsupervised learning. Like multiplayer perceptrons of Chapter 3, Neural Networks-Perceptrons, they use the backpropagation algorithm to learn, but with a major difference--the target is the same as the input. 

We can think of an autoencoder as consisting of two cascaded networks--the first network is an encoder, it takes the input x, and encodes it using a transformation h to encoded signal y:

The second network uses the encoded signal y as its input and performs another transformation f to get a reconstructed signal r:

We define error e as the difference between the original input x and the reconstructed signal r, e = x - r. The network then learns by reducing the mean squared error (MSE), and the error is propagated back to the hidden layers as in the case of MLPs. The following figure shows an autoencoder with encoder and decoder highlighted separately. Autoencoders can have weight sharing, that is, weights of decoder and encoder are simply a transpose of each other, which can help the network learn faster as the number of training parameters is less, but at the same time reduces the degrees of freedom of the network. They are very similar to RBMs of Chapter 7, Unsupervised Learning, but with one big difference--the state of the neurons in autoencoders is deterministic, while in RBMs, the state of the neurons is probabilistic:

Depending on the size of the hidden layer, autoencoders are classified as Undercomplete (hidden layer has fewer neurons than the input layer) or Overcomplete (hidden layer has more neurons than the input layer). Depending on the restriction/constraints imposed on the loss, we have various types of autoencoders: Sparse Autoencoders, Denoising Autoencoders, and Convolution Autoencoders. In this chapter, you will learn about these variations in autoencoders and implement them using TensorFlow.

One of the obvious applications of autoencoders is in the field of dimensionality reduction [2]; the results show that autoencoders yield better results as compared to PCA. Autoencoders can also be used for feature extraction [3], document retrieval [2], classification, and anomaly detection.