The reason for adding more layers to the network comes from our understanding of how the visual cortex works for humans. This is a dedicated area in the rear part of your brain that is used for recognizing objects/patterns/numbers, and so on, and is composed of complex layers of neurons that work to encode visual information and classify them accordingly based on prior knowledge.

Not surprisingly, there is no set rule on how many layers a network needs in order to produce good results and experimentation is highly recommended!