Partial derivatives and the gradient

A function of several variables, f(x, y), is a surface. The slope depends on which way you walk.

A partial derivative ∂f/∂x: the slope if you change only x and hold y fixed. You differentiate as usual and treat y as a constant.

f(x, y) = x² + 3xy + y²

• ∂f/∂x = 2x + 3y (y constant)
• ∂f/∂y = 3x + 2y (x constant)

The gradient ∇f = (∂f/∂x, ∂f/∂y) is a vector pointing in the direction where the function grows fastest. Its length is how steep that is.

Which is why gradient descent goes minus the gradient: the steepest way down.

$\nabla f(x,y) = \left(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}\right)$. At the point $(1, 2)$ for $f = x^2 + 3xy + y^2$: $\nabla f = (2\cdot1 + 3\cdot2,\; 3\cdot1 + 2\cdot2) = (8, 7)$.

The directional derivative in a unit direction $u$ is $D_u f = \nabla f \cdot u$ — largest when $u$ points the same way as $\nabla f$ (the dot product is maximised), zero perpendicular to the gradient (you are then walking along a level curve).

At a minimum $\nabla f = 0$ (every partial derivative is zero) — necessary but not sufficient: it could also be a maximum or a saddle point.

For a model with millions of parameters the gradient is a vector with millions of components — one per parameter. Backpropagation is the method of computing all of them in one pass, instead of one at a time.

import numpy as np

def f(x, y):
    return x**2 + 3*x*y + y**2

def grad(x, y):
    return np.array([2*x + 3*y, 3*x + 2*y])

print(grad(1, 2))          # [8 7]

# a numerical check (always worth doing)
eps = 1e-6
num = np.array([(f(1+eps, 2) - f(1-eps, 2)) / (2*eps),
                (f(1, 2+eps) - f(1, 2-eps)) / (2*eps)])
print(np.round(num, 4))    # [8. 7.]

The numerical gradient check is the standard trick when you have implemented backprop yourself: if the analytic and numerical gradients agree, the implementation is probably right.