Having written a fairly straightforward feed forward function, let's now look at how to make the neural network learn.

Recall that we said earlier that a neural network learns when you tell it that it's made a mistake? More technically, we ask the question: what kind of cost function can we use so that it is able to convey to the neural network accurately about what the true value is.

The cost function we want to use for this project is the sum of squared errors. What is an error? Well, an error is simply the difference between the real value and the predicted value. Does this mean that if the real value is 7, and the neural network predicted 2, the cost would just be 7-2 ? No. This is because we should not treat the labels as numbers. They are labels.

So what do we subtract? Recall the one-hot vector that we created earlier? If we peek inside the Predict function, we can see that pred, the result of the final activation is a slice of ten float64s. That's what we're going to subtract. Because both are slices of ten float64s, we would have to subtract them element-wise.

Merely subtracting the slices would not be useful; the results may be negative. Imagine if you were tasked to find the lowest possible costs for a product. If someone came up to you and told you that their product costs negative amounts and that they would pay you to use it, would you not use it? So to prevent that, we take the square of the errors.

To calculate the sum of squared errors, we simply square the result. Because we're training the neural network one image at a time, the sum is simply the squared errors of that one image.