Synapses

Synapses are the connecting junction between axon and dendrites. The majority of synapses send signals from the axon of a neuron to the dendrite of another neuron. The exceptions for this case are when a neuron might lack dendrites, or a neuron lacks an axon, or a synapse, which connects an axon to another axon.

Dendrites

Dendrites have fibers branching out from the soma in a bushy network around the nerve cell. Dendrites allow the cell to receive signals from connected neighboring neurons and each dendrite is able to perform multiplication by that dendrite’s weight value. Here multiplication means an increase or decrease in the ratio of synaptic neurotransmitters to signal chemicals introduced into the dendrite.

Axons

Axons are the single, long fibers extending from the main soma. They stretch out longer distances than dendrites and measure generally 1 centimeter in length (100 times the diameter of the soma). Eventually, the axon will branch and connect to other dendrites. Neurons are able to send electrochemical pulses through cross-membrane voltage changes generating action potential. This signal travels along the cell’s axon and activates synaptic connections with other neurons.

Information flow across the biological neuron

Synapses that increase the potential are considered excitatory, and those that decrease the potential are considered inhibitory. Plasticity refers the long-term changes in strength of connections in response to input stimulus. Neurons also have been shown to form new connections over time and even migrate. These combined mechanics of connection change drive the learning process in the biological brain.​​

From biological to artificial

The animal brain has been shown to be responsible for the fundamental components of the mind. We can study the basic components of the brain and understand them. Research has shown ways to map out functionality of the brain and track signals as they move through neurons.

However, we still do not completely understand how this collection of decentralized functional units provides the foundation for thought and the seat of consciousness.

Now that we’ve established the basics of how a biological neuron works, let’s take a look at the first attempts at modeling the neuron with the advent of the perceptron.

History of the perceptron

The perceptron was invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt. It was funded by the US Office of Naval Research and was covered by the New York Times:

[T]he embryo of an electronic computer that [the Navy] expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence.

Obviously these predictions were a bit premature—as we’ve seen many times with the promise of machine learning and AI. Early versions were intended to be implemented as a physical machine rather than a software program. The first software implementation was for the IBM 704, and then later it was implemented in the Mark I Perceptron machine.

It also should be noted that McCulloch and Pitts introduced the basic concept of analyzing neural activity in 1943¹ based on thresholds and weighted sums. These concepts were key in developing a model for later variations like the perceptron.

Definition of the perceptron

The perceptron is a linear-model binary classifier with a simple input–output relationship as depicted in Figure 1-3, which shows we’re summing n number of inputs times their associated weights and then sending this “net input” to a step function with a defined threshold. Typically with perceptrons, this is a Heaviside step function with a threshold value of 0.5. This function will output a real-valued single binary value (0 or a 1), depending on the input.

Figure 1-3. Single-layer perceptron

We can model the decision boundary and the classification output in the Heaviside step function equation, as follows:

To produce the net input to the activation function (here, the Heaviside step function) we take the dot product of the input and the connection weights. We see this summation in the left half of Figure 1-3 as the input to the summation function. Table 1-1 provides an explanation of how the summation function is performed as well as notes about the parameters involved in the summation function.

The output of the step function (activation function) is the output for the perceptron and gives us a classification of the input values. If the bias value is negative, it forces the learned weights sum to be a much greater value to get a 1 classification output. The bias term in this capacity moves the decision boundary around for the model. Input values do not affect the bias term, but the bias term is learned through the perceptron learning algorithm.​

As a basic linear classifier we consider the single-layer perceptron to be the simplest form of the family of feed-forward neural networks.

The perceptron learning algorithm

The perceptron learning algorithm changes the weights in the perceptron model until all input records are all correctly classified. The algorithm will not terminate if the learning input is not linearly separable. A linearly separable dataset is one for which we can find the values of a hyperplane that will cleanly divide the two classes of the dataset.

The perceptron learning algorithm initializes the weight vector with small random values or 0.0s at the beginning of training. The perceptron learning algorithm takes each input record, as we can see in Figure 1-3, and computes the output classification to check against the actual classification label. To produce the classification, the columns (features) are matched up to weights where n is the number of dimensions in both our input and weights. The first input value is the bias input, which is always 1.0 because we don’t affect the bias input. The first weight is our bias term in this diagram. The dot product of the input vector and the weight vector gives us the input to our activation function, as we’ve previously discussed.

If the classification is correct, no weight changes are made. If the classification is incorrect, the weights are adjusted accordingly. Weights are updated between individual training examples in an “online learning” fashion. This loop continues until all of the input examples are correctly classified. If the dataset is not linearly separable, the training algorithm will not terminate. Figure 1-4 demonstrates a dataset that is not linearly separable, the XOR logic function.

Figure 1-4. The XOR function

A basic perceptron (single-layer variant) cannot solve the XOR logic modeling problem, illustrating an early limitation of the perceptron model.

Limitations of the early perceptron

After initial promise, the perceptron was found to be limited in the types of patterns it could recognize. The initial inability to solve nonlinear (e.g., datasets that are not linearly separable) problems was seen as a failure for the field of neural networks. The 1969 book Perceptrons by Minsky and Papert illustrated the limitations of the single-layer perceptron. However, what the general industry did not widely realize was that a multilayer perceptron could indeed solve the XOR problem, among many other nonlinear problems.

Evolution of the artificial neuron

The artificial neuron of the multilayer perceptron is similar to its predecessor, the perceptron, but it adds flexibility in the type of activation layer we can use. Figure 1-5 shows an updated diagram for the artificial neuron that is based on the perceptron.

Figure 1-5. Artificial neuron for a multilayer perceptron

This diagram is similar to Figure 1-3 for the single-layer perceptron, yet we notice a more generalized activation function. We’ll develop this diagram further in a detailed look at the artificial neuron as we proceed.

The net input to the activation function is still the dot product of the weights and input features yet the flexible activation function allows us to create different types out of output values. This is a major contrast to the earlier perceptron design that used a piecewise linear Heaviside step function as this improvement now allowed the artificial neuron because express more complex activation output.

Artificial neuron input

The artificial neuron (see Figure 1-6) takes input that, based on the weights on the connections, can be ignored (by a 0.0 weight on an input connection) or passed on to the activation function. The activation function also has the ability to filter out data if it does not provide a non-zero activation value as output.

Figure 1-6. Details of an artificial neuron in a multilayer perceptron neural network

We express the net input to a neuron as the weights on connections multiplied by activation incoming on connection, as shown in Figure 1-6. For the input layer, we’re just taking the feature at that specific index, and the activation function is linear (it passes on the feature value). For hidden layers, the input is the activation from other neurons. Mathematically, we can express the net input (total weighted input) of the artificial neuron as

• input_sum_i = W_i · A_i

where W_i is the vector of all weights leading into neuron i and A_i is the vector of activation values for the inputs to neuron i. Let’s build on this equation by accounting for the bias term that is added per layer (explained further below):

• input_sum_i = W_i · A_i + b

To produce output from the neuron, we’d then wrap this net input with an activation function g, as demonstrated in the following equation:

$a_i=g(input\_su{m}_i)$

We can then expand this function with the definition of $inpu{t}_i$:

$a_i=g({\mathbf{W}}_i·{\mathbf{A}}_i+b)$

This activation value for neuron i is the output value passed on to the next layer through connections to other artificial neurons (multiplied by weights on connections) as an input value.

Different Notations

An alternate notation for expressing the output of an artificial neuron we often see in research papers is as follows:

• h_w,b(x) = g(w · x + b)

This notation is slightly different from the previous equation above. We show this alternate notation to illustrate how some papers will use slightly different notations to explain these concepts, and we want you to be able to recognize the variants.

If our activation function is the sigmoid function, we’ll have this:

$g\left(z\right)=\frac{1}{1+e^{-z}}$

This output will have the range [ 0, 1 ], which is the same output as the logistic regression function.

Sigmoid Activation Functions in Practice

We list the sigmoid activation function here for historical demonstration. As we’ll see later on in this lesson, the sigmoid activation function has largely fallen out of favor.

The inputs are the data from which you want to produce information, and the connection weights and biases are the quantities that govern the activity, activating it or not. As with the perceptron, there is a learning algorithm to change the weights and bias value for each artificial neuron. During the training phase, the weights and biases change as the network learns. We’ll cover the learning algorithm for neural networks later in the chapter.

Just as biological neurons don’t pass on every electro-chemical impulse they receive, artificial neurons are not just wires or diodes passing on a signal. They are designed to be selective. They filter the data they receive, and aggregate, convert, and transmit only certain information to the next neuron(s) in the network. As these filters and transformations work on data, they convert raw input data to useful information in the context of the larger multilayer perceptron neural network. We illustrate this effect more in the next section.

Artificial neurons can be defined by the kind of input they are able to receive (binary or continuous) and the kind of transform (activation function) they use to produce output. In DL4J, all neurons in a layer have the same activation function.

Comparing the biological neuron and the artificial neuron

If we loop back for a moment and think about the biological neuron on which the artificial neuron is based, we can ask, “How close does the artificial variant match up to the biological version?” The concepts match up with the input connection functionality being performed by dendrites in the biological neuron and the summation functionality being provided by the soma. Finally, we see the activation function being performed by the axon in the biological neuron.

Feed-forward neural network architecture

Now that we understand the differences between the artificial neuron and the perceptron, we can better understand the structure of the full multilayer feed-forward neural network. With multilayer feed-forward neural networks, we have artificial neurons arranged into groups called layers. Building on the layer concept, we see that the multilayer neural network has the following:

• A single input layer
• One or many hidden layers, fully connected
• A single output layer

As Figure 1-7 depicts, the neurons in each layer (represented by the circles) are all fully connected to all neurons in all adjacent layers.

The neurons in each layer all use the same type of activation function (most of the time). For the input layer, the input is the raw vector input. The input to neurons of the other layers is the output (activation) of the previous layer’s neurons. As data moves through the network in a feed-forward fashion, it is influenced by the connection weights and the activation function type. Let’s now take a look at the specifics of each layer type.

Figure 1-7. Fully connected multilayer feed-forward neural network topology

Algorithm intuition

Backpropagation learning is similar to the perceptron learning algorithm. We want to compute the input example’s output with a forward pass through the network. If the output matches the label, we don’t do anything. If the output does not match the label, we need to adjust the weights on the connections in the neural network.​

To further illustrate general neural network learning, let’s take a look at the pseudocode for the algorithm, as shown in Example 1-1.

function neural-network-learning( training-records ) returns network
    network <- initialize weights (randomly)
    start loop
        for each example in training-records do
            network-output = neural-network-output( network, example )
            actual-output = observed outcome associated with example
            update weights in network based on
                { example, network-output, actual-output }
        end for
    end loop when all examples correctly predicted or hit stopping conditions
    return network

The key is to distribute the blame for the error and divide it between the contributing weights. With the perceptron learning algorithm, it’s easy because there is only one weight per input to influence the output value. With feed-forward multilayer networks learning algorithms have a bigger challenge. There are many weights connecting each input to the output, so it becomes more difficult. Each weight contributes to more than one output, so our learning algorithm must be more clever.

Backpropagation is a pragmatic approach to dividing the contribution of error for each weight. It is similar to the perceptron learning algorithm. With backpropagation, we’re trying to minimize the error between the label (or “actual”) output associated with the training input and the value generated from the network output. In the next section, we take a look at the mathematical notation that you will see in most literature on neural networks for backpropagation of feed-forward neural networks.

A closer look at backpropagation

For most of this lesson, we don’t intend to throw a lot of math at you. However, for the topic of backpropagation and to better understand a core fundamental concept that much of the lesson is based on, we feel we should provide a section that illustrates these concepts down to the notation level.

Let’s zoom in on the previous diagram to focus on the input layer and the first hidden layer. Figure 1-8 provides the view.

Figure 1-8. Multilayer perceptron network zoomed-in with component labels

In Figure 1-8 we see our previous multilayer neural network diagram zoomed-in and updated with some notation. We’ll use this notation for the rest of this section’s explanation of backpropagation. Table 1-2 lists the notations in the figure.

To further set the stage for explaining this algorithm, let’s take a look at the pseudocode of the backpropagation learning algorithm, as shown in Example 1-2.

function backpropagation-algorithm
    ( network, training-records, learning-rate ) returns network
    network <- initialize weights (randomly)
    start loop
        for each example in training-records do

            // compute the output for this input example
            network-output <- neural-network-output( network, example )

            // compute the error and the [delta] for neurons in the output layer
            example_err <- target-output - network-output

            // update the weights leading to the output layer     
            $W_{j,i}\leftarrow W_{j,i}+\alpha \times a_j\times Er{r}_i\times g^{\text{'}}(input\_su{m}_i)$

            for each subsequent-layer in network do

                // compute the error at each node
                ${\Delta }_j\leftarrow g^{\text{'}}(input\_su{m}_j){\Sigma }_i{W}_{j,i}{\Delta }_i$    

                // update the weights leading into the layer
                $W_{k,j}\leftarrow W_{k,j}+\alpha \times a_k\times {\Delta }_j$

            end for

        end for
    end loop when network has converged
    return network

Understanding backpropagation pseudocode

Example 1-2 has the following inputs:

• Network: a multilayer feed-forward neural network
• Training records: a set of training vectors with associated outputs
• Learning rate: the learning rate (sometimes denoted by the Greek alpha symbol)

To start the algorithm, we initialize our neural network and begin looping through the input examples (until we encounter a terminating condition or a maximum number of epochs). First, we compute the output of the current network for the current input example. We compare this output to the actual output associated with the input and compute the error (example_err).

Now we’re ready to compute the weight updates leading to the output layer.

Updating the output layer weights

The output layer weights update is calculated by using the following:

$W_{j,i}\leftarrow W_{j,i}+\alpha \times a_j\times Er{r}_i\times g^{\text{'}}(input\_su{m}_i)$

This is the weight update rule for the connection between neuron j and neuron i for all connections in the neural network. We can see these connections leading into the output layer highlighted in Figure 1-9.

Figure 1-9. Updating connections into the output layer

Breaking this diagram down further, we can see we are working with the neuron j weight from the previous hidden layer connecting to the current neuron i. We’re multiplying the learning rate α (discussed later in the chapter) by the incoming activation from the j neuron. This input is calculated by getting the net input to the j neuron and then computing the activation of neuron j.

Computing the total weight input to the activation function for neuron i, we compute the dot product of the incoming weight vector W_j and the activation vector A_j and then add in the bias term value:

input_sum_i = W_i · A_i + b

Here is the equation for computing the activation value of neuron j:

$a_j=g(input\_su{m}_j)$

The error term for example e at neuron i is denoted as Err_i. We denote the derivative of the activation function as g'( x ), which is applied to the net input of neuron i with the term:

$g^{\text{'}}(input\_su{m}_i)$

This update rule is similar to how we’d update a perceptron except we’re using the activations of the previous layers as opposed to their raw input values. This rule also contains a term for the derivative of the activation function to get the gradient of the activation function.

Updating the hidden layers

With the backpropagation algorithm, we walk back across the hidden layers, updating the connection between each one until we reach the input layer. Figure 1-10 illustrates a connections between the two hidden layers highlighted.

To update these connections we take the input from the fractional error value computed previously and multiply it by the activation (input) from the connection from the previous layer and the learning rate.

Figure 1-10. Highlighting connections between hidden layers

We then add this to the previous value for the weight, and this is our updated value:

$W_{k,j}\leftarrow W_{k,j}+\alpha \times a_k\times {\Delta }_j$

Weights and biases that have been assigned the blame for error are reduced to limit their signal. Weights and biases that pass along signals supporting correct answers are strengthened. The adjustment of weights to an optimal state is an iterative process, taken one step at a time.

The length of these learning steps, or the amount the weights that are changed with each iteration, is known as the learning rate. The learning rate is a parameter we define (as opposed to being a measurement of the network’s performance). We discuss learning rate later in this chapter when we talk more about hyperparameters in general.

AI Winter II: Early 1990s

In the late 1980s and early 1990s, we saw an overpromotion of technologies such as expert systems and LISP machines, both of which failed to live up to expectations. The Strategic Computing Initiative cancelled new spending at the end of this cycle. The fifth-generation computer failed its goals.

Leaky ReLU

Leaky ReLUs are a strategy to mitigate the “dying ReLU” issue.² As opposed to having the function being zero when x < 0, the leaky ReLU will instead have a small negative slope (e.g., “around 0.01”). Some success has been seen in practice with this ReLU variation but results are not always consistent. The equation is given here:

Figure 1-15.  

Softplus

This activation function is considered to be the “smooth version of the ReLU,” as is illustrated in Figure 1-16. Compare this plot to the ReLU in Figure 1-14.

Figure 1-16 shows that the softplus activation function (f(x) = ln[ 1 + exp(x) ]) has a similar shape to the ReLU. We also notice the differentiability and nonzero derivative of the softplus everywhere on the graph, in contrast to the ReLU.

Figure 1-16. Visualizing the ReLU and softplus activation functions

Mean squared error loss

When working on a regression model that requires a real valued output, we use the squared loss function, much like the case of ordinary least squares in linear regression. Consider the case in which we have to predict only one output feature (M = 1). The error in a prediction is squared and is averaged over the number of data points, plain and simple, as we see in the following equation for MSE loss:

What if M is greater than one and we are looking to predict multiple output features for a given set of input features? In this case, the desired and predicted entities, Y and $\widehat{Y}$, respectively, are an ordered list of numbers, or, in other words, vectors.

Let’s now look at another variation of the MSE loss function:

If you’re familiar with linear algebra, you’ll recognize the inner sigma in the preceding equation as the square of the Euclidean distance. In fact, the MSE is sometimes referred to by these terms. Note that N, the size of your dataset, and M, the number of features the network has to predict, are constants. So, consider these as simple scaling factors that you can account for in other ways (like by scaling the learning rate). In a lot of use cases (including DL4J), the M is dropped and a division by two is added for mathematical convenience (which will become clearer in the context of its gradient in backpropagation). In the following equation, we see the version of MSE that the DL4J library uses for regression:

Other loss functions for regression

Although the MSE is used widely, it is quite sensitive to outliers, and this is something that you should consider when picking a loss function. When picking a stock to invest in, we want to take the outliers into account. But perhaps when buying a house we don’t. In this case, what is of most interest is what most people would pay for it. In which case, we are more interested in the median and less so in the mean.

Regression loss function discussion

These are all valid choices, and there certainly is no single loss function that will outperform all other loss functions for every scenario. The MSE is very widely used and is a safe bet in most cases. So is the MAE. The MSLE and the MAPE are worth taking into consideration if our network is predicting outputs that vary largely in range. Suppose that a network is to predict two output variables: one in the range of [0, 10] and the other in the range of [0, 100]. In this case, the MAE and the MSE will penalize the error in the second output more significantly than the first. The MAPE makes it a relative error and therefore doesn’t discriminate based on the range. The MSLE squishes the range of all the outputs down, simply how 10 and 100 translate to 1 and 2 (in log base 10).

Hinge loss

Hinge loss is the most commonly used loss function when the network must be optimized for a hard classification. For example, 0 = no fraud and 1 = fraud, which by convention is called a 0-1 classifier. The 0,1 choice is somewhat arbitrary and –1, 1 is also seen in lieu of 0–1. Hinge loss is also seen in a class of models called maximum-margin classification models (e.g., support vector machines, a somewhat distant cousin to neural networks).

Following is the equation for hinge loss when data points must be categorized as –1 or 1:

The hinge loss is mostly used for binary classifications. There are extensions for multiclass classification (e.g., “one versus all,” “one versus one”) for the hinge loss that are not covered here.

Logistic loss

Logistic loss functions are used when probabilities are of greater interest than hard classifications. Great examples of these would be flagging potential fraud, with a human-in-the-loop solution or predicting the “probability of someone clicking on an ad,” which can then be linked to a currency number.

Predicting valid probabilities means generating numbers between 0 and 1. Predicting valid probabilities also means making sure the probability of mutually exclusive outcomes should sum to one. For this reason, it is essential that the very last layer of a neural network used in classification is a softmax. Note that the sigmoid activation function also will give valid values between 0 and 1. However, you cannot use it for scenarios in which the outputs are mutually exclusive, because it does not model the dependencies between the output values.

Now that we have made sure our neural network will produce valid probabilities for the classes we have, we can dive headlong into the loss function and into the idea of what we should be optimizing here. We want to optimize for what is formally called the “maximum likelihood.” In other words, we want to maximize the probability we predict for the correct class AND we want to do so for every single sample we have.

Let us consider the case in which our network predicts a probability for two classes, like the fraud and not fraud 0–1 classifier. Based on the notation described earlier, we can express our output for a given input $X_i$ as $h\left(X_i\right)$ and $1-h\left(X_i\right)$, given a set of weights and biases, W and b, expressing the probability of 1 and 0, respectively, as shown here:

We can combine these equations and express them as follows:

The word “AND” from our colloquial definition of maximum likelihood in the context of probability should immediately ring a bell. The “AND” across all available samples translates to the product of the probabilities in question, as shown here:

Let’s now move on and take a look at negative log likelihood.