Gradient descent

Gradient descent is a general-purpose optimization algorithm that will find stationary points of smooth functions. The solution will be a global optimum if the objective function is convex. Variations of gradient descent are widely used in the training of complex neural networks, but also to compute solutions for MLE problems.

The algorithm uses the gradient of the objective function that contains its partial derivatives with respect to the parameters. These derivatives indicate how much the objective changes for infinitesimal steps in the direction of the corresponding parameters. It turns out that the maximal change of the function value results from a step in the direction of the gradient itself.

Hence, when minimizing a function that describes, for example, the cost of a prediction error, the algorithm computes the gradient for the current parameter values using the training data and modifies each parameter according to the negative value of its corresponding gradient component. As a result, the objective function will assume a lower value and move the parameters move closer to the solution. The optimization stops when the gradient becomes small, and the parameter values change very little.

The size of these steps is the learning rate, which is a critical parameter that may require tuning; many implementations include the option for this learning rate to increase with the number of iterations gradually. Depending on the size of the data, the algorithm may iterate many times over the entire dataset. Each such iteration is called an epoch. The number of epochs and the tolerance used to stop further iterations are hyperparameters you can tune.

Stochastic gradient descent randomly selects a data point and computes the gradient for this data point as opposed to an average over a larger sample to achieve a speedup. There are also batch versions that use a certain number of data points for each step.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
44.193.80.126