Skip to content
AI-grafen
DAI developerScientific method· about 45 min· fundamentals that rarely change· verified 2026-10-22· EN

Hypotheses and falsifiability

Be able to formulate a hypothesis that can be disproved and design a test.

See the full experiment: from hypothesis to decision

Intuition

A hypothesis is a claim that can be wrong — and you say in advance what would show that.

  • «Our model is good» — not falsifiable.
  • «Our model reaches ≥ 85 % on test set X» — falsifiable: run it and see.
  • «Dropout 0.5 reduces the train–test gap by at least 5 percentage points compared with 0, over three seeds» — falsifiable and precise.

The trap: explaining after the fact. The result came out at 83 %? «Well, the test set was hard.» If every outcome can be explained the hypothesis has no content. Write the prediction down before you run.

Interactive

Rewrite these as falsifiable hypotheses:

VagueFalsifiable
More data helpsDoubling the training data raises the validation accuracy by ≥ 2 pp (3 seeds, the mean)
The model understands sarcasmOn 200 sarcastic examples ≥ 70 % right, against 50 % for chance
The prompt is betterOn the same 100 cases, prompt B wins ≥ 60 % of the pairwise comparisons

For each: what is the outcome that disproves it? If you cannot write that down the hypothesis is not testable. Then write it in the lab journal before the run — with a date.

Formal

A complete decision, not just a hypothesis: A school wants to test whether a new AI tutor helps more students finish a lesson within seven days. Before the experiment starts, the team records:

  1. Population and assignment: 400 students are randomized, 200 to the old version and 200 to the new one.
  2. Primary outcome: the number finishing within seven days divided by all assigned students in each group.
  3. Decision rule: switch only if a 95% interval for the difference (new minus old) is entirely above zero and the share of incorrect advice does not rise.

The result is 80/200 = 40% versus 96/200 = 48%. The difference is +8 percentage points, but a simple 95% interval for two independent proportions is roughly −2 to +18 percentage points. The result does not meet the decision rule. It does not prove that the versions are equivalent: the interval allows both a small deterioration and an improvement that would matter.

If the team chooses “ninth-grade students” afterwards because that subgroup happened to look best, that is a new, exploratory hypothesis. Test it in a new, planned experiment. Record changes to the analysis plan instead of rewriting the original prediction.

Mastery means

  • Formulates a hypothesis that can be disproved
  • Designs a test with a clear outcome that would falsify it
  • Distinguishes a prediction from an explanation after the fact

Prerequisites

Sign in to do the exercises and build your mastery up.

Sources

All the sources and licences