Rectified Linear Unit (ReLU) is the most popular activation function in use. We will be using it as the primary activation function in a number of advanced architectures in later chapters.

It can be described as follows:

func relu(x){
   return Max(0,x)
}

If we were to chart it out, it looks something like the following diagram:

As you can see, it is extremely similar to a linear function, except that it goes to zero (therefore indicating that the neuron is not activated).

ReLU also has many useful properties, as follows:

• It is nonlinear: Therefore, stacking several layers of these will not necessarily result in being the same as one layer
• It is differentiable: Therefore, it works with backpropagation
• It is quick: It calculates quickly, which is important when we are running this calculation numerous times across layers or training passes of our network

ReLU goes to zero if the input is negative. This can be useful, since this results in fewer neurons being activated, and, therefore, this can potentially speed up our calculations. However, since it can result in 0, this can very quickly cause a neuron to die and never activate again, given certain inputs.