Gradient descent

We have a loss function — the error as a function of the weights. We want to reach the bottom. Gradient descent: work out the slope (the gradient) where we are standing, take a small step against the slope, repeat.

w ← w − η · L'(w)

η (eta) is the learning rate: the step length. Too small and it takes forever. Too large and you jump over the valley and can end up higher than you started.

L(w) = (w − 3)², L'(w) = 2(w − 3). Starting at w = 0, η = 0.25:

step	w	L'(w)	new w
1	0	−6	0 − 0.25·(−6) = 1.5
2	1.5	−3	2.25
3	2.25	−1.5	2.625

It is approaching 3. In Python:

w, eta = 0.0, 0.25
for step in range(20):
    grad = 2 * (w - 3)
    w = w - eta * grad
print(w)   # ≈ 3.0

With several weights the gradient is a vector of partial derivatives — the same rule, component by component. The stochastic variant of gradient descent (SGD) computes the gradient on a small batch of examples at a time, which is cheaper and often better.