The compact representation of the inputs is stored in weights; let's visualize the weights learned by the network. Here are the weights of the encoder layer for the standard and sparse autoencoder respectively. We can see that, in the standard autoencoder, many hidden units have their very large weights, suggesting they are overworked: