Hypotheses and falsifiability
Be able to formulate a hypothesis that can be disproved and design a test.
Intuition
A hypothesis is a claim that can be wrong — and you say in advance what would show that.
- «Our model is good» — not falsifiable.
- «Our model reaches ≥ 85 % on test set X» — falsifiable: run it and see.
- «Dropout 0.5 reduces the train–test gap by at least 5 percentage points compared with 0, over three seeds» — falsifiable and precise.
The trap: explaining after the fact. The result came out at 83 %? «Well, the test set was hard.» If every outcome can be explained the hypothesis has no content. Write the prediction down before you run.
Interactive
Rewrite these as falsifiable hypotheses:
| Vague | Falsifiable |
|---|---|
| More data helps | Doubling the training data raises the validation accuracy by ≥ 2 pp (3 seeds, the mean) |
| The model understands sarcasm | On 200 sarcastic examples ≥ 70 % right, against 50 % for chance |
| The prompt is better | On the same 100 cases, prompt B wins ≥ 60 % of the pairwise comparisons |
For each: what is the outcome that disproves it? If you cannot write that down the hypothesis is not testable. Then write it in the lab journal before the run — with a date.
Formal
A complete decision, not just a hypothesis: A school wants to test whether a new AI tutor helps more students finish a lesson within seven days. Before the experiment starts, the team records:
- Population and assignment: 400 students are randomized, 200 to the old version and 200 to the new one.
- Primary outcome: the number finishing within seven days divided by all assigned students in each group.
- Decision rule: switch only if a 95% interval for the difference (new minus old) is entirely above zero and the share of incorrect advice does not rise.
The result is 80/200 = 40% versus 96/200 = 48%. The difference is +8 percentage points, but a simple 95% interval for two independent proportions is roughly −2 to +18 percentage points. The result does not meet the decision rule. It does not prove that the versions are equivalent: the interval allows both a small deterioration and an improvement that would matter.
If the team chooses “ninth-grade students” afterwards because that subgroup happened to look best, that is a new, exploratory hypothesis. Test it in a new, planned experiment. Record changes to the analysis plan instead of rewriting the original prediction.
Mastery means
- Formulates a hypothesis that can be disproved
- Designs a test with a clear outcome that would falsify it
- Distinguishes a prediction from an explanation after the fact
Prerequisites
Sign in to do the exercises and build your mastery up.