An activation function formulates how the inputs to a particular neuron will be processed to generate an output.

As shown in the following diagram, each of the neurons in a neural network has an activation function that determines how inputs will be processed:

In the preceding diagram, we can see that the results generated by an activation function are passed on to the output. The activation function sets the criteria that how the values of the inputs are supposed to be interpreted to generate an output.

For exactly the same input values, different activation functions will produce different outputs. Understanding how to select the right activation function is important when using neural networks to solve problems.

Let's now look into these activation functions one by one.