Here's a visual representation of a neural network:
As you can see from the figure, there are three distinct layers in a neural network: Input layer, Hidden layer (or middle), and Output layer.
There can be more than one hidden layer; however, one hidden layer would be enough to resolve the majority of real-life problems.
How do we determine the network's topology, and how many neurons to create for each layer? Let's make this determination layer by layer.
The input layer defines the number of inputs into the network. For example, let's say you want to create an ANN, which will help you determine what animal you're looking at given a description of its attributes. Let's fix these attributes to weight, length, and teeth. That's a set of three attributes; our network will need to contain three input nodes.
The output layer is equal to the number of classes we identified. Continuing with the preceding example of an animal classification network, we will arbitrarily set the output layer to 4, because we know we're going to deal with the following animals: dog, condor, dolphin, and dragon. If we feed in data for an animal that is not in one of these categories, the network will return the class most likely to resemble this unclassified animal.
The hidden layer contains perceptrons. As mentioned, the vast majority of problems only require one hidden layer; mathematically speaking, there is no verified reason to have more than two hidden layers. We will, therefore, stick to one hidden layer and work with that.
There are a number of rules of thumb to determine the number of neurons contained in the hidden layer, but there is no hard-and-fast rule. The empirical way is your friend in this particular circumstance: test your network with different settings, and choose the one that fits best.
These are some of the most common rules used when building an ANN:
- The number of hidden neurons should be between the size of the input layer and the size of the output layer. If the difference between the input layer size and the output layer is large, it is my experience that a hidden layer size much closer to the output layer is preferable.
- For relatively small input layers, the number of hidden neurons is two-thirds the size of the input layer, plus the size of the output layer, or less than twice the size of the input layer.
One very important factor to keep in mind is overfitting. Overfitting occurs when there's such an inordinate amount of information contained in the hidden layer (for example, a disproportionate amount of neurons in the layer) compared to the information provided by the training data that classification is not very meaningful.
The larger the hidden layer, the more training information is required for the network to be trained properly. And, needless to say, this is going to lengthen the time required by the network to properly train.
So, following the second rules of thumb illustrated earlier, our network will have a hidden layer of size 8, just because after a few runs of the network, I found it to yield the best results. As a side note, the empirical approach is very much encouraged in the world of ANNs. The best network topology is related to the type of data fed to the network, so don't refrain from testing ANNs in a trial-and-error fashion.
In summary, our network has the following sizes:
- Input:
3 - Hidden:
8 - Output:
4
There are a number of learning algorithms used by ANNs, but we can identify three major ones:
- Supervised learning: With this algorithm, we want to obtain a function from the ANN, which describes the data we labeled. We know, a priori, the nature of this data, and we delegate to the ANN the process of finding a function that describes the data.
- Unsupervised learning: This algorithm differs from supervised learning; in this, the data is unlabeled. This implies that we don't have to select and label data, but it also means the ANN has a lot more work to do. The classification of the data is usually obtained through techniques such as (but not only) clustering, which we explored in Chapter 7, Detecting and Recognizing Objects.
- Reinforcement learning: Reinforcement learning is a little more complex. A system receives an input; a decision-making mechanism determines an action, which is performed and scored (success/failure and grades in between); and finally the input and the action are paired with their score, so the system learns to repeat or change the action to be performed for a certain input or state.
Now that we have a general idea of what ANNs are, let's see how OpenCV implements them, and how to put them to good use. Finally, we'll work our way up to a full blown application, in which we will attempt to recognize handwritten digits.