Regression tests for models

An overall metric that goes from 0.87 to 0.86 looks harmless — but it can hide that twenty cases became wrong while nineteen others became right. Regression tests look at the cases, not the average.

Three levels:

The level	What	On failure
Golden cases	~30 cases that must work (safety, core flows)	blocks the release
A threshold	the overall metric ≥ a limit	blocks the release
A regression list	cases that went from right to wrong	requires review, not an automatic stop

The third is the most useful day to day — and the one that requires you to save the result per case, not just the sum.

import json, pathlib

def run_and_compare(model, cases, baseline_file="evals/baseline.json", golden=frozenset(), threshold=0.85):
    results = {c["id"]: bool(check(model(c["input"]), c)) for c in cases}
    acc = sum(results.values()) / len(results)

    base = json.loads(pathlib.Path(baseline_file).read_text()) if pathlib.Path(baseline_file).exists() else {}
    regressions = sorted(k for k, v in base.items() if v and not results.get(k, False))
    improvements = sorted(k for k, v in results.items() if v and not base.get(k, True))
    golden_failures = sorted(k for k in golden if not results.get(k, False))

    report = {"accuracy": round(acc, 4), "regressions": regressions,
              "improvements": improvements, "golden_failures": golden_failures,
              "blocks": bool(golden_failures) or acc < threshold}
    print(json.dumps(report, ensure_ascii=False, indent=2))
    return report, results

# In CI:
# report, results = run_and_compare(...)
# assert not report["blocks"], report
# on an approved release: rewrite baseline.json with the new result

Maintaining the set of cases is what decides whether the suite is worth anything in a year:

• Every production bug becomes a new case. Always.
• Every new feature comes with its cases.
• Cases every version has passed for a year can be moved to a slow suite.
• The baseline is updated only at a deliberate release, never automatically — otherwise degradations are silently normalised.