The building block of a neural network is the abstraction of a biological neuron, a quite simplistic but powerful computational unit that was proposed for the first time by F. Rosenblatt in 1957, to make up the simplest neural architecture, called a perceptron, that we are going to analyze in the next section. Contrary to Hebbian Learning, which is more biologically plausible but has some strong limitations, the artificial neuron has been designed with a pragmatic viewpoint and, of course, only its structure is based on a few elements characterizing a biological cell. However, recent deep learning research activities have unveiled the enormous power of this kind of architecture. Even if there are more complex and specialized computational cells, the basic artificial neuron can be summarized as the conjunction of two blocks, which are clearly shown in the following diagram:
The input of a neuron is a real-valued vector x ∈ ℜn, while the output is a scalar y ∈ ℜ. The first operation is linear:
The vector w ∈ ℜn is called weight-vector (or synaptic weight vector, because, analogously to a biological neuron, it reweights the input values), while the scalar term b ∈ ℜ is a constant called bias. In many cases, it's easier to consider only the weight vector. It's possible to get rid of the bias by adding an extra input feature equal to 1 and a corresponding weight:
In this way, the only element that must be learned is the weight vector. The following block is called an activation function, and it's responsible for remapping the input into a different subset. If the function is fa(z) = z, the neuron is called linear and the transformation can be omitted. The first experiments were based on linear neurons that are much less powerful than non-linear ones, and this was a reason that led many researchers to consider the perceptron as a failure, but, at the same time, this limitation opened the door for a new architecture that, instead, showed its excellent abilities. Let's now start this analysis with the first neural network ever proposed.