Improvement or noise? A release decision step by step
The team has a new version of a support bot. On the same 50 test cases, version 1 passes 42 (84%) and version 2 passes 44 (88%). Four percentage points better. May it ship? This is how to reason — the same four steps the course's final project tests in code.
1. A number without uncertainty says little
88% on 50 cases is a sample, not the truth. A 95% confidence interval for the share correct (the Wilson interval, which holds even near 0 and 1) is roughly 76–94% for version 2 and 71–92% for version 1. The intervals overlap almost completely.
With 500 cases instead of 50, the interval for 88% shrinks to roughly 85–91%. That is why large eval sets matter: not because the number gets higher, but because it gets more certain.
2. Compare paired — only the cases where the versions differ count
Both versions ran on the same cases. Comparing two independent proportions is then wrong. Look instead at the cases where the answers differ: version 2 solves 6 cases version 1 missed, but loses 4 that version 1 handled. The other 40 cases say nothing about the difference.
Is 6 against 4 more than chance? McNemar's test asks exactly that: if the versions were equally good, each differing case would be a coin flip. With 10 differing cases split 6–4, p ≈ 0.75. That is noise. For a difference on 10 cases to be significant at the 5% level you need about 1–9; with 12 differing cases, 2–10 is enough (p ≈ 0.04).
The conclusion is not that version 2 is worse, but that the measurement cannot tell them apart. If you need the answer, the test set has to grow — especially with cases where the versions actually behave differently.
3. Regression tests: some cases must never break
One of the four cases version 2 lost is that the bot must refuse to disclose personal identity numbers. That is a critical case. A release gate blocks on critical cases regardless of the total — higher accuracy cannot buy back a data-protection failure.
A common trap: cases missing from the new run, for example because they crashed or were filtered out. If they are not counted at all, accuracy rises by itself. A missing case must count as a failure.
- List regressions: cases that went from right to wrong.
- Mark critical cases and block if any of them fails.
- Set a minimum accuracy and count missing cases as failures.
- Store results per case, so the next release is compared against the same baseline.
4. LLM as judge: consistent is not the same as correct
When answers are free text, a language model is often used as a judge comparing two answers pairwise. Judges have known biases: they often prefer the first option (position bias), the longest answer and their own style.
So run every comparison in both orders. If the judge switches sides when the order is swapped, the verdict is a tie. But that is not enough: a judge that always picks the longest answer is perfectly consistent and still wrong. Also measure how often the judge agrees with human raters on a sample before letting it decide releases.
The decision
Version 2 should not ship. The improvement cannot be told apart from noise, and a critical case has broken. The right next step is to fix the identity-number case, grow the test set and measure again — not to look for a metric that looks better. Testing many metrics or subsets until something becomes significant is the multiple comparisons problem.