Did the AI tutor improve? Follow an experiment through to a decision
A school tries two versions of an AI tutor. Out of 200 randomly assigned students per version, 80 finish the lesson with the old version and 96 with the new one within seven days. That is an eight percentage point difference. Is it enough to switch?
1. Decide what counts before the test
Write down the primary outcome in advance: the share of assigned students who finish the lesson within seven days. Keep everyone assigned to a version in the denominator, including students who never opened it. Counting only active users can make a version look better because of who chose to start.
Also define a safety outcome, such as the share of incorrect answers that students accepted. A higher completion rate is no improvement if the tutor teaches more mistakes.
2. Randomize the assignment
If students choose their own version, the more motivated students may gather in the new group. Motivation, rather than the tutor, could then explain the difference. Assign a version at random at the start and keep each student on that version throughout the experiment. Run both groups over the same calendar period.
A difference after randomization can still be sampling noise. Randomization makes the groups comparable on average; it does not make an individual result certain.
3. Calculate the difference and its uncertainty
Old version: 80/200 = 40%. New version: 96/200 = 48%. The observed difference is +8 percentage points. For two independent groups, a simple approximate 95% interval for the difference is −2 to +18 percentage points. It includes both no effect and a meaningful improvement.
The interval is an approximation for these sample sizes and proportions. It is not a promise about the effect at another school. Do not compare the two groups' separate intervals alone; calculate uncertainty for the difference itself.
4. Make the decision using the rule chosen in advance
Suppose the plan was to switch only when the whole interval for the difference is above zero and the safety outcome has not worsened. These results do not meet that rule. They do not prove that the versions are equivalent either. Plan a new experiment with the same outcomes and a larger sample size chosen in advance.
Picking the grade or metric that happened to look best afterwards gives you a new hypothesis to test, not confirmation from the original experiment.