Based on the restrictions imposed on the loss, AEs can be grouped into the following types:

• Plain Vanilla AEs: This is the simplest AE architecture possible, with a fully-connected neural layer as the encoder and decoder.
• Sparse AEs: Sparse AEs are an alternative method for introducing an information bottleneck, without requiring a reduction in the number of nodes in our hidden layers. Rather than preferring an undercomplete AE, the loss function is constructed in a way that it penalizes the activations within a layer. For any given observation, the network is encouraged to learn encoding and decoding, which only relies on activating a small number of neurons.
• Denoising AEs: This is a type of overcomplete AE that experiences the risk of learning the identity function or null function. Essentially, the AE learns the output that is equal to the input, therefore making the AE useless. Denoising AEs avoid this problem of learning the identity function by randomly initializing some of the inputs to 0. During the computation of the loss function, the noise-induced input is not considered; therefore, the network still learns the correct weights without the risk of learning the identity function. At the same time, the AE is trained to learn to reconstruct the output, even from the corrupted input.

The following figure is a example of denoising AEs on sample images from the MNIST dataset:

• Convolutional AEs: When dealing with images as inputs, one can use convolutional layers as part of the encoder and decoder networks. Such kinds of AEs that use convolutional layers are termed convolutional AEs. The following figure is an illustration showing the use of convolutions in AEs:

• Stacked AEs: Stacked AEs are ones that have multiple layers in the encoder as well as the decoder. You can refer to the general architecture of an AE as an example illustration of a stacked AE architecture, with the encoder and decoder having two layers (excluding the input and output layers).
  • Variational AEs: A variational AE (VAE), rather than building an encoder that outputs a singl
• e value to describe each latent state attribute, describes a probability distribution for each latent attribute. This makes it possible to design complex generative models of data and also generate fictional celebrity images and digital artwork. The following figure is an illustration depicting the representation of data in VAEs:

In a VAE, the encoder model is sometimes referred to as the recognition model, whereas the decoder model is sometimes referred to as the generative model. The encoder outputs a range of statistical distributions for the latent features. These features are randomly sampled and used by the decoder to reconstruct the input. For any sampling of the latent distributions, the decoder is expected to be able to accurately reconstruct the input. Thus, values that are nearby to one another in latent space should correspond with very similar reconstructions.