Activation functions are used to introduce nonlinearity in neural networks. We apply the activation function to the input which is multiplied by weights and added to the bias, that is, f(z), where z = (input * weights) + bias. There are different types of activation functions as follows:

• Sigmoid function: The sigmoid function is one of the most commonly used activation functions. It scales the value between 0 and 1. The sigmoid function can be defined as . When we apply this function to z, the values will be scaled in the range of 0 to 1. This is also called a logistic function. It is s-shaped, as shown in the following diagram: 

• Hyperbolic tangent function: Unlike the sigmoid function, the hyperbolic tangent function scales the value between -1 and +1. The hyperbolic tangent function can be defined as . When we apply this function to z, the values will be scaled in the range of -1 to +1. It is also s-shaped but zero centered, as shown in the following diagram:

• ReLU function: ReLU is also known as a rectified linear unit. It is one of the most widely used activation functions. The ReLU function can be defined as , that is, f(z) is 0 when z is less than 0 and f(z) is equal to z when z is greater than or equal to 0:

• Softmax function: The softmax function is actually the generalization of the sigmoid function. It is usually applied on the final layer of the network and while performing multi-class classification tasks. It gives the probabilities of each class being an output and thus the sum of softmax values will always equal to 1. It can be defined as .