Skip to content
AI-grafen
Sign in
☰ Menu
FAI engineeringProject· about 75 min· server sandbox

Final project: ship a model version you can defend

Decide whether version 2 of an assistant may be released. Report accuracy with uncertainty, decide whether the difference from version 1 is real or noise, block the release when a critical case breaks and check whether the LLM judge can be trusted.

Teaches: Statistical significance in evals

Requires: Build an eval harnessConfidence intervalsRegression tests for modelsAn LLM as a judge

Theory

An eval number without uncertainty says little. Compare versions on the same cases: only the cases where they differ carry information (McNemar). A release is blocked by critical cases even when the total rises, and a case missing from the run counts as a failure. Run an LLM judge in both orders; being consistent does not mean it agrees with humans.

Sub-tasks

  1. Accuracy with uncertainty — accuracy_ci(outcomes) returns {accuracy, low, high, n} where low/high is a 95% Wilson interval. An empty list raises ValueError.
  2. Improvement or noise? — paired_comparison(a, b) compares two versions on the same cases. Return {diff, a_only, b_only, p, verdict} where p comes from an exact two-sided McNemar test and verdict is IMPROVED, WORSE or NOISE (p ≥ alpha). Different lengths raise ValueError.
  3. Release gate — release_gate(baseline, candidate, critical, min_accuracy) returns {accuracy, regressions, critical_failures, block, reasons}. A case missing from candidate counts as a failure. Block if a critical case fails or accuracy is below the threshold.
  4. Audit the LLM judge — judge_audit(judge, pairs, human) runs each pair in both orders. The verdict is A, B, or TIE if it flips with the order. Return {verdicts, consistency, agreement} where agreement is measured against the human choices.

Passes when: checks >= 1

The starter code

runs in an isolated sandbox on the server
"""Evals-slutprov. Ändra funktionerna nedan; testerna bedömer beteende på nya resultat."""

IMPROVED = "förbättring"
WORSE = "försämring"
NOISE = "brus"
TIE = "LIKA"


def accuracy_ci(outcomes, z=1.96):
    """outcomes: [bool] per testfall -> {accuracy, low, high, n} med Wilsonintervall."""
    # TODO: räkna andelen rätt och ett 95 %-intervall som håller även nära 0 och 1.
    raise NotImplementedError


def paired_comparison(a, b, alpha=0.05):
    """a, b: [bool] för samma fall i samma ordning -> {diff, a_only, b_only, p, verdict}."""
    # TODO: räkna fall där bara en version har rätt och gör ett exakt, tvåsidigt McNemar-test.
    # verdict är IMPROVED eller WORSE om p < alpha, annars NOISE.
    raise NotImplementedError


def release_gate(baseline, candidate, critical=(), min_accuracy=0.0):
    """baseline, candidate: {fall_id: bool} -> {accuracy, regressions, critical_failures, block, reasons}."""
    # TODO: ett fall som saknas i candidate räknas som fel. Blockera vid kritiskt fel eller för låg träffsäkerhet.
    raise NotImplementedError


def judge_audit(judge, pairs, human):
    """judge(question, first, second) -> "1" | "2" | annat. pairs: [{id, question, a, b}], human: {id: "A"|"B"}.
    Returnerar {verdicts, consistency, agreement}."""
    # TODO: kör varje par i båda ordningarna; ett domslut som byter sida med ordningen blir TIE.
    raise NotImplementedError

You write the code; tests you cannot see decide whether it holds up. Create a free account to run the lab.

Try the diagnosticCreate a free account

Expected results

All tests and all four release checks pass. The server also runs hidden cases with other numbers, missing cases and new judges. The course is complete only after a verified full run of this project and all knowledge steps.

Common mistakes

A normal approximation instead of Wilson gives intervals outside 0–1; McNemar counting cases where both versions are right; a removed case dropping out of the denominator and raising accuracy; running the judge in one order so position bias stays hidden; running only one subtask instead of All.