RoPE in detail

Attention builds on the dot product $q\cdot k$. The problem is that it does not know where in the sequence the token stands.

RoPE's idea: rotate $q$ and $k$ by an angle that depends on the position. Then the dot product automatically becomes dependent on the difference between the positions.

Think of a pair of dimensions as a point in the plane. Rotate the point by $m\theta$ radians if it stands at position $m$. When you then take the dot product between a vector rotated by $m\theta$ and one rotated by $n\theta$, the result depends only on $(m-n)\theta$ — the rotations partly cancel.

$\langle R_m q,\, R_n k\rangle = \langle q,\, R_{n-m} k\rangle$

That is the whole point: the absolute position is encoded, but only the relative distance affects the attention. No extra table, no learnt position vectors, and no positions the model has never seen during training.

The construction. Split the vector into pairs of dimensions. For pair $i$ (of $d/2$) the angular frequency

$\theta_i = \text{base}^{-2i/d}, \qquad \text{base} = 10\,000 \text{ (usually)}$

is used. At position $m$, pair $i$ is rotated by the angle $m\theta_i$:

$\begin{pmatrix} x'_{2i} \\ x'_{2i+1}\end{pmatrix} = \begin{pmatrix}\cos m\theta_i & -\sin m\theta_i \\ \sin m\theta_i & \cos m\theta_i\end{pmatrix}\begin{pmatrix} x_{2i} \\ x_{2i+1}\end{pmatrix}$

Why it becomes relative. A rotation matrix satisfies $R_m^\top R_n = R_{n-m}$. So:

$(R_m q)^\top (R_n k) = q^\top R_m^\top R_n k = q^\top R_{n-m} k$

The result depends only on $n - m$. That holds exactly, not approximately.

The frequency spectrum. Low $i$ gives $\theta_i$ near 1 — a fast rotation, which tells neighbours apart. High $i$ gives $\theta_i$ near $10^{-4}$ — a slow rotation, which tells distant positions apart. Together they cover the whole scale, rather like the digits in a positional number system.

Extending the context. RoPE extrapolates better than learnt positional embeddings, but not without limit: the fastest frequencies have time to go round several times beyond the training length and the model has never seen those angles in that setting.

Three methods for stretching the context:

Method	The idea	The cost
Positional interpolation (PI)	divide the position by a factor $s$ so that the new length is mapped into the trained range	a short fine-tuning; a little resolution lost at short distances
NTK-aware scaling	change the base instead of the position — it scales the high-frequency dimensions less	often works without fine-tuning
YaRN	combines them: interpolates the slow frequencies, extrapolates the fast ones, plus a temperature adjustment	the best results, requires some fine-tuning

Simply raising the base (from 10 000 to 500 000, say) is the simplest variant and is used in several modern models — it makes all the rotations slower and thereby stretches the whole scale out.

A practical detail: RoPE is applied to $q$ and $k$ but not to $v$. The values should not be rotated — only how much attention they get should depend on the position.

import torch, math

def rope_frequencies(d, max_pos, base=10_000.0, scale=1.0):
    """Returns the cosine and the sine for every position and dimension pair."""
    i = torch.arange(0, d, 2, dtype=torch.float32)
    theta = base ** (-i / d)                      # (d/2,)
    pos = torch.arange(max_pos, dtype=torch.float32) / scale   # scale > 1 = interpolation
    angle = pos[:, None] * theta[None, :]         # (max_pos, d/2)
    return angle.cos(), angle.sin()

def apply_rope(x, cos, sin):
    """x: (..., T, d). Rotates pairs of dimensions."""
    x1, x2 = x[..., 0::2], x[..., 1::2]
    return torch.stack([x1 * cos - x2 * sin,
                        x1 * sin + x2 * cos], dim=-1).flatten(-2)

# Check the relative property NUMERICALLY
d, T = 64, 128
torch.manual_seed(0)
cos, sin = rope_frequencies(d, T)
q = torch.randn(d)
k = torch.randn(d)

def dot(m, n):
    qm = apply_rope(q[None], cos[m:m+1], sin[m:m+1])[0]
    kn = apply_rope(k[None], cos[n:n+1], sin[n:n+1])[0]
    return float(qm @ kn)

# The same relative distance → the same dot product, whatever the absolute position
for (m, n) in [(0, 5), (10, 15), (50, 55), (100, 105)]:
    print(f"pos {m:>3} → {n:>3} (distance {n-m}): {dot(m, n):+.6f}")
# pos   0 →   5 (distance 5): -1.234567
# pos  10 →  15 (distance 5): -1.234567
# pos  50 →  55 (distance 5): -1.234567     ← identical

# The frequency spectrum: from fast to slow rotation
theta = 10_000.0 ** (-torch.arange(0, d, 2).float() / d)
print("the fastest pair: turns per position", float(theta[0] / (2 * math.pi)))
print("the slowest:      turns per position", float(theta[-1] / (2 * math.pi)))
print("the slowest period:", round(2 * math.pi / float(theta[-1])), "positions")

# Extending the context: interpolation against a raised base
trained_length, new_length = 4096, 16384
print("\npositional interpolation: scale =", new_length / trained_length)
cos_pi, sin_pi = rope_frequencies(d, new_length, scale=new_length / trained_length)

print("a raised base:", 10_000 * (new_length / trained_length) ** (d / (d - 2)))
cos_base, sin_base = rope_frequencies(d, new_length, base=500_000.0)

The printout in the middle is the whole proof in numerical form: the same relative distance gives an identical dot product wherever in the sequence the token stands.