Skip to content
AI-grafen

The goal E University

Statistics for experiments

Conditional probability, simulation, A/B testing and variance between runs — so that you can tell a real improvement from chance.

Knowledge nodes
69
From zero
about 44 h
Labs
6
See what you already know — no account

The diagnostic suggests a starting point. The exercises show what you can actually do.

What you can do afterwards

Did the AI tutor improve? Follow an experiment through to a decision

A school tries two versions of an AI tutor. Out of 200 randomly assigned students per version, 80 finish the lesson with the old version and 96 with the new one within seven days. That is an eight percentage point difference. Is it enough to switch?

1. Decide what counts before the test

Write down the primary outcome in advance: the share of assigned students who finish the lesson within seven days. Keep everyone assigned to a version in the denominator, including students who never opened it. Counting only active users can make a version look better because of who chose to start.

Also define a safety outcome, such as the share of incorrect answers that students accepted. A higher completion rate is no improvement if the tutor teaches more mistakes.

2. Randomize the assignment

If students choose their own version, the more motivated students may gather in the new group. Motivation, rather than the tutor, could then explain the difference. Assign a version at random at the start and keep each student on that version throughout the experiment. Run both groups over the same calendar period.

A difference after randomization can still be sampling noise. Randomization makes the groups comparable on average; it does not make an individual result certain.

3. Calculate the difference and its uncertainty

Old version: 80/200 = 40%. New version: 96/200 = 48%. The observed difference is +8 percentage points. For two independent groups, a simple approximate 95% interval for the difference is −2 to +18 percentage points. It includes both no effect and a meaningful improvement.

The interval is an approximation for these sample sizes and proportions. It is not a promise about the effect at another school. Do not compare the two groups' separate intervals alone; calculate uncertainty for the difference itself.

4. Make the decision using the rule chosen in advance

Suppose the plan was to switch only when the whole interval for the difference is above zero and the safety outcome has not worsened. These results do not meet that rule. They do not prove that the versions are equivalent either. Plan a new experiment with the same outcomes and a larger sample size chosen in advance.

Picking the grade or metric that happened to look best afterwards gives you a new hypothesis to test, not confirmation from the original experiment.

Labs along the way

You write the code. Tests you cannot see decide whether it holds up.

The whole path

Everything the goal builds on, grouped by level and in the order it builds on itself. Show on the map

AExplorer5 knowledge nodes
  1. Patterns and categories
  2. Collect data with a survey
  3. Probability and risk in everyday life
  4. Sequences and precise instructions
  5. Integer arithmetic and order of operations
BInvestigator13 knowledge nodes
  1. Data in everyday life
  2. Charts and tables
  3. Algorithmic thinking
  4. Rule-based systems and machine learning
  5. Randomness and probability: basic calculations
  6. Is the chance equal for everyone?
  7. Fractions, decimals, and percentages
  8. Negative numbers in everyday life and AI
  9. Training data, features and labels
  10. Classification: how a model sorts information
  11. Generative AI: how it creates content
  12. Language models and probabilities
  13. AI's invented answers
CBuilder17 knowledge nodes
  1. Functions and coordinate systems
  2. Programming logic — variables, conditions, loops
  3. Python — the basics
  4. Python — lists, loops and dictionaries
  5. Relative frequency and the law of large numbers
  6. Statistics — mean, median and spread
  7. Probability — the basics
  8. Combinatorics
  9. Probability in several steps
  10. Linear regression: fitting a straight line to data
  11. Neural networks — the intuition
  12. Asking effective questions to AI
  13. Compare two AI answers
  14. Evaluate AI performance with your own questions
  15. Variables and algebraic expressions
  16. Powers and roots
  17. Misleading charts
DAI developer23 knowledge nodes
  1. Derivatives and optimisation
  2. Conditional probability
  3. Bayes' theorem
  4. Probability distributions
  5. Loss functions
  6. The normal distribution and standardisation
  7. Samples and uncertainty
  8. Tokenisation
  9. Gradient descent
  10. Better or just different?
  11. Logarithms
  12. Vectors
  13. Matrices and matrix multiplication
  14. Linear regression with several features
  15. Neural networks — the forward pass with matrices
  16. Backpropagation
  17. Embeddings — words as vectors
  18. Attention
  19. Overfitting and generalisation
  20. PyTorch — tensors and autograd
  21. Training, validation and test
  22. Train a neural network in PyTorch
  23. Transformers — the architecture
EUniversity11 knowledge nodes
  1. Monte Carlo methods
  2. Confidence intervals
  3. The bootstrap and resampling
  4. Hypothesis testing and p-values
  5. A/B tests and experiment design
  6. Maximum likelihood
  7. Language models — training and generation
  8. Model evaluation
  9. Scientific method in AI
  10. Reproducibility
  11. Random seeds and the variance between runs