Using our biological analogy, we can construct a model of a computational neuron, and this model is known as the McCulloch-Pitts model of a neuron:

This computational neuron is the simplest example of a neural network. We can construct the output function, y, of our neural network directly from following our diagram:

The function g() in our neural network is the activation function. Here, the specific activation function that is chosen is the step function:

When the linear weighted sum of inputs exceeds zero, the step function outputs 1, and when it does not, the function outputs -1. It is customary to create a dummy input feature x₀ which is always taken to be 1, in order to merge the bias or threshold w₀ into the main sum as follows:

Using our experience with logistic regression, it should be very easy to conclude that we could construct a simple classifier using this setup for the binary classification problem. The only difference is that in logistic regression, we would choose the logistic function as the activation function. In fact, in 1957, Frank Rosenblatt proposed a supervised learning algorithm for training the McCulloch-Pitts model of neurons to perform binary classification, and this algorithm along with the learning model produced is known as the Rosenblatt perceptron.

We've thus far presented linear and logistic regression as models that can solve supervised learning problems and showed the criteria that are used to train them without actually going into the optimization details of the training algorithms involved. This was done intentionally to allow us to focus our attention on understanding the models themselves, and how to apply them in R.

Now that we have built up some experience with classification and regression, this chapter is going to be different, in that we will look at some of the details of how predictive models are trained, as this too is an important process that adds to our overall understanding of a model. In addition, neural networks differ substantially from previous models we have seen so far, in that training a neural network is often more time consuming and involves adjusting a number of parameters, many of which arise from the optimization procedure itself. Thus, it helps to understand the role these parameters play during training and how they can affect the final model.

Before we present a training algorithm for the perceptron, we'll first have to learn one of the most fundamental techniques used in solving optimization problems.