Sigmoid and softmax as functions

A neural network spits out arbitrary numbers — «logits» — which can be −13.7 or 42. Probabilities have to lie between 0 and 1 and sum to 1. Something has to translate.

Sigmoid translates one number into one probability:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

S-shaped, symmetric about $(0, 0.5)$, flat at both ends.

Softmax translates several numbers into one distribution:

$\mathrm{softmax}(z)_i = \frac{e^{z_i}}{\sum_j e^{z_j}}$

All of them become positive (thanks to $e^x > 0$) and the sum becomes exactly 1 (thanks to the division by the sum).

A simple rule: a yes/no question → sigmoid. A question with several mutually exclusive alternatives → softmax. Several labels that can hold at the same time → sigmoid per label.

Sigmoid from the odds. The odds of an event are $\frac{p}{1-p}$. The log odds (the logit) are $z = \ln\frac{p}{1-p}$. Solve for $p$:

$e^z = \frac{p}{1-p} \;\Rightarrow\; e^z - p e^z = p \;\Rightarrow\; p = \frac{e^z}{1+e^z} = \frac{1}{1+e^{-z}}$

Sigmoid is therefore the inverse of the logit function. It is not an arbitrary S-curve that somebody made up — it is the function that turns log odds into a probability, which is precisely what logistic regression needs.

The derivative is unusually elegant:

$\sigma'(z) = \sigma(z)\,(1 - \sigma(z))$

The derivation: $\sigma(z) = (1+e^{-z})^{-1}$, so $\sigma'(z) = -(1+e^{-z})^{-2}\cdot(-e^{-z}) = \dfrac{e^{-z}}{(1+e^{-z})^2} = \sigma(z)\cdot\dfrac{e^{-z}}{1+e^{-z}} = \sigma(z)(1-\sigma(z))$.

The derivative can therefore be computed from the function value — no need to keep $z$ around. That is one of the reasons sigmoid was popular in early networks.

But: the maximum is $\sigma'(0) = 0.25$, and at $|z| > 5$ the derivative is near zero. That is vanishing gradients in their purest form, and the reason ReLU took over in deep networks.

Softmax properties:

1. Translation invariance. $\mathrm{softmax}(z + c) = \mathrm{softmax}(z)$ for every constant $c$ — the factor $e^c$ cancels. It is used for numerical stability ($c = -\max z$) and means that only the differences between the logits matter.
2. Sigmoid is the two-class special case: $\mathrm{softmax}([z, 0])_0 = \frac{e^z}{e^z + 1} = \sigma(z)$.
3. Temperature. $\mathrm{softmax}(z/T)$: a low $T$ gives a sharp distribution, a high $T$ an even one. It is the same parameter that controls the randomness in text generation.
4. The Jacobian is $\frac{\partial p_i}{\partial z_j} = p_i(\delta_{ij} - p_j)$ — a matrix, not a simple element-wise derivative, since every output depends on all the logits.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

def softmax(z):
    z = np.asarray(z, float)
    e = np.exp(z - z.max())          # translation invariance → numerical stability
    return e / e.sum()

for z in (-4, -1, 0, 1, 4):
    s = sigmoid(z)
    print(f"z={z:3d}  σ={s:.4f}  σ'={s * (1 - s):.4f}")
# z= -4  σ=0.0180  σ'=0.0177
# z=  0  σ=0.5000  σ'=0.2500   ← the maximal derivative
# z=  4  σ=0.9820  σ'=0.0177   ← almost flat: a vanishing gradient

print(softmax([2.0, 1.0, 0.1]))               # [0.6590 0.2424 0.0986]
print(softmax([2.0, 1.0, 0.1]) - softmax([102.0, 101.0, 100.1]))   # ~0 — invariant

# Sigmoid is softmax with two classes
z = 1.3
print(round(float(sigmoid(z)), 6), round(float(softmax([z, 0.0])[0]), 6))   # 0.785835 0.785835

# The temperature controls the sharpness
for T in (0.5, 1.0, 2.0, 5.0):
    print(f"T={T}: {np.round(softmax(np.array([2.0, 1.0, 0.1]) / T), 3)}")
# T=0.5: [0.864 0.117 0.019]   sharp
# T=1.0: [0.659 0.242 0.099]
# T=2.0: [0.502 0.304 0.194]
# T=5.0: [0.4   0.327 0.273]   even

# The derivative can be computed from the function value — handy in the backward pass
s = sigmoid(np.array([-2.0, 0.0, 2.0]))
print(np.round(s * (1 - s), 4))               # [0.105 0.25  0.105]