Scientific method in AI

ML research has a reproducibility problem: many reported improvements disappear when somebody reruns them with the same seeds, the same baseline tuning and more datasets. The method that protects you:

1. Question → hypothesis that can be falsified, written before the experiment.
2. Control: only one thing changes. The baseline gets as much hyperparameter tuning as the new method.
3. Variation: at least 3–5 seeds; report the mean ± std or a CI. One seed is an anecdote.
4. Several datasets: an improvement on one dataset is often noise.
5. Ablation: remove each component — which one is doing the work?
6. Report negative results and everything you tried, not just what worked.

Common faults: the test set used for choosing, an undertuned baseline, cherry-picked seeds, the metric switched afterwards, «significant» with no interval.

Scrutinise a claim: «Our new optimiser gives 1.2 pp better on CIFAR-10 than Adam.»

The questions you ask:

• How many seeds? What is the std between seeds for Adam? (Often ~0.3–0.5 pp → 1.2 could be 2σ, or it could be noise.)
• Did Adam get the same lr search?
• How were the epochs and early stopping chosen — on the test set?
• Does it work on ImageNet, on text?
• Is there code, are there seeds and an exact configuration?

If the answers are missing the claim is not wrong — it is untested. Apply the same scrutiny to your own results before you show them to anyone.