We're going to be talking about why nonlinearity matters, and then we'll look at some visualizations of the two most commonly used nonlinear functions: sigmoid and relu.

So, nonlinearity may sound like a complicated mathematical concept, but all you basically need to know is that it doesn't go in a straight line. This allows neural networks to learn more complex shapes, and this learning of complex shapes inside of the structure of the network is what lets neural networks and deep learning actually learn.

So, let's take a look at the sigmoid function:

It's kind of an S-curve that ranges from zero to one. It's actually built out of e to an exponent and a ratio. Now, the good news is that you'll never actually have to code the math that you see here, because when we want to use sigmoid in Keras, we simply reference it by the name sigmoid.

Now, let's look at relu. The relu nonlinear function is kind of only technically a nonlinear function because when it's less than zero, it's a straight line:

When it's greater than zero, it's also a straight line. But the combination of the two, the flat part before zero and the angle after zero together, does not form a straight line:

ReLu nonlinear function—greater than zero.

Because it's a very constant function, this is mathematically efficient when carried out inside of the computer, so you'll see relu used in many production neural network models simply because it computes faster. But the trick with relu functions, as we learned when we talked about normalization in the previous chapter, is that they can generate values larger than one, so various tricks and techniques in building your neural network, including normalizations and creating further layers, are often required to get relu functions to perform well.

A lot of what's going on in machine learning involves computing the inputs to these relu and sigmoid functions repeatedly.

There's a lot of math going on under the covers, so the interaction of a large number of nonlinearities allows a machine learner to conceptually draw a high-dimensional mathematical shape around the answers.