Likely, by now, you must have many questions running through your mind—why three convolutional layers rather than two or four? Why a stride of one? Why a patch size of five? Why end up with fully connected layers rather than start with them?

There is some method to the madness here. At the core, CNN's are built around image processing and patches are built around the features being sought. Why some configurations work well while others do not is not fully understood, though general rules do follow intuition. The exact network architectures are discovered, honed, and increasingly inch toward perfection through thousands of trials and many errors. It continues to be a research-grade task.

The practitioner's general approach is often to find a well working, existing architecture (for example, AlexNet, GoogLeNet, ResNet) and tweak them for use with a specific dataset. That is what we did; we started with AlexNet and tweaked it. Perhaps, that is not fulfilling, but it works and remains the state of practice in 2016.