The memory hierarchy and bandwidth

Computation is cheap. Moving data is expensive. That is the single most important insight for performance on modern hardware.

The memory hierarchy on a GPU, approximately:

Level	Size	Bandwidth	Relative cost
Registers	~kB per core	extremely high	1
SRAM (shared memory)	~200 kB per SM	~19 TB/s	~10
HBM (VRAM)	40–80 GB	1–3 TB/s	~100
Host RAM over PCIe	hundreds of GB	~30 GB/s	~10 000
NVMe	TB	~7 GB/s	~100 000

The difference between SRAM and HBM is roughly a factor of ten in bandwidth — and that is the difference flash attention exploits.

The consequence: an operation that reads a lot of data and computes little is limited by the memory, not by the GPU's compute capacity. Buying a faster card then does not help.

Arithmetic intensity is the ratio

$I = \frac{\text{FLOPs}}{\text{bytes moved}}$

Every piece of hardware has a break-even point: an A100 manages about 312 TFLOP/s in bf16 and has about 2.0 TB/s of HBM bandwidth, which gives a break-even point around 156 FLOP/byte. If your operation is below that it is bandwidth-bound.

Operation	Arithmetic intensity	Bound by
Element-wise (ReLU, add)	~0.25	memory
Layer/BatchNorm	~1	memory
Attention (naive)	~1–10	memory
Matrix multiplication, large	~100–1 000	computation
Generation, one token	~1	memory

The last row explains why token generation is slow: every token requires all the model weights to be read from HBM, but only one vector–matrix product is computed. The throughput is therefore limited by

$\text{tokens/s} \lesssim \frac{\text{bandwidth}}{\text{model size in bytes}}$

The roofline model puts it together: the performance is $\min(\text{peak FLOPs},\ I \times \text{bandwidth})$. Plotting your operation in that diagram says directly what to optimise.

Three optimisations that follow from this:

Optimisation	The idea
Kernel fusion	merge several element-wise operations so that the data is read once instead of three times
Tiling	split into blocks that fit in SRAM and reuse them
Batching	read the weights once, use them for many examples

Batching is the most dramatic one in generation: the weights are read once whether the batch is 1 or 64, so the throughput grows nearly linearly until the computation becomes the bottleneck.

This also explains why quantisation helps so much in inference. Going from bf16 to 4 bits makes the model four times smaller, and since the speed is bandwidth-bound the generation becomes roughly four times faster — not because the computation is faster, but because there is less to read.

import torch, time

def arithmetic_intensity(flops, bytes_moved):
    return flops / bytes_moved

# An element-wise operation on a large tensor
n = 100_000_000
flops = n                                   # one operation per element
byts = n * 2 * 2                            # read + write, bf16
print(f"ReLU: I = {arithmetic_intensity(flops, byts):.2f} FLOP/byte  → memory-bound")

# A matrix multiplication N×N
for N in (128, 1024, 8192):
    flops = 2 * N ** 3
    byts = 3 * N ** 2 * 2                   # two inputs plus one output, bf16
    print(f"matmul {N:>5}: I = {arithmetic_intensity(flops, byts):>8.1f} FLOP/byte")
# matmul   128: I =     42.7 FLOP/byte   → memory-bound
# matmul  1024: I =    341.3 FLOP/byte   → compute-bound
# matmul  8192: I =   2730.7 FLOP/byte   → clearly compute-bound

# The break-even point for a given piece of hardware
def break_even(tflops, tb_per_s):
    return tflops * 1e12 / (tb_per_s * 1e12)

for name, tf, bw in (("A100 bf16", 312, 2.0), ("H100 bf16", 990, 3.35), ("laptop GPU", 20, 0.3)):
    print(f"{name:<11} break-even {break_even(tf, bw):>6.0f} FLOP/byte")
# A100 bf16   break-even    156 FLOP/byte
# H100 bf16   break-even    296 FLOP/byte
# laptop GPU  break-even     67 FLOP/byte

# Generation: the ceiling comes from the bandwidth, not from the computation
def generation_ceiling(model_gb, bandwidth_gb_s):
    return bandwidth_gb_s / model_gb

for name, gb, bw in (("7B bf16", 14, 2000), ("7B int4", 3.5, 2000), ("70B int4", 35, 2000)):
    print(f"{name:<9} ceiling ~{generation_ceiling(gb, bw):>6.0f} tokens/s")
# 7B bf16   ceiling ~   143 tokens/s
# 7B int4   ceiling ~   571 tokens/s     ← a 4× smaller model, 4× faster
# 70B int4  ceiling ~    57 tokens/s

# Kernel fusion: three operations, one read instead of three
x = torch.randn(10_000_000, device="cuda" if torch.cuda.is_available() else "cpu")

def unfused(x):
    return torch.relu(x * 2.0 + 1.0)

fused = torch.compile(unfused)              # torch.compile fuses automatically

The 7B int4 row is the most practical consequence: quantisation gives a fourfold speed-up in generation, not because the computation is faster but because there is a quarter as much to read.