GFrontier LabLab· about 90 min· server sandbox

Lab: ablation — which component does the work?

Build an ablation tool that generates configurations (leave-one-out and add-one), runs a pipeline with several seeds, reports mean ± std and Δ against the full configuration, and decides with a pre-registered rule which components are shown to matter.

Teaches: Ablation studies

Requires: ReproducibilityEvals for language models and agents

Theory

An ablation isolates a component's contribution by changing one thing at a time with everything else equal. Differences smaller than ~2 standard deviations (across seeds) are not shown. Add-one variants reveal interactions: two components can be worthless on their own but decisive together. The pipeline here is a text classifier with three components: bigrams, a stop-word filter and class weighting — and the data is constructed so that only some components matter.

Sub-tasks

configurations — configurations(components) → dict name→config with "full", "baslinje" (baseline), "utan_<k>" (without k) for each k and "bara_<k>" (only k) for each k.
running and statistics — run_ablation(run_fn, components, seeds) → list of rows {name, mean, std, delta} where delta = mean − mean(full); std with ddof=1; rows in the same order as configurations().
interpretation — important(rows, sigma_mult=2.0) → sorted list of components k where utan_k has delta < −sigma_mult·std(full); interactions(rows, components, sigma_mult) → pairs (a, b) where bara_a and bara_b ≈ baseline but full ≫ baseline.

Passes when: correct >= 1

The starter code

runs in an isolated sandbox on the server

"""Ablationsverktyg. Fyll i funktionerna — pipeline.run(config, seed) är given."""
import numpy as np


def configurations(components):
    """namn -> config (dict komponent -> bool). Ordning: full, baslinje, utan_<k>…, bara_<k>…"""
    # TODO
    raise NotImplementedError


def run_ablation(run_fn, components, seeds):
    """Kör varje konfiguration med varje frö. Returnerar lista av dict {name, mean, std, delta} i configurations()-ordning."""
    # TODO: std med ddof=1; delta = mean - mean(full)
    raise NotImplementedError


def important(rows, sigma_mult=2.0):
    """Komponenter k där raden 'utan_k' har delta < -sigma_mult * std(full). Sorterad lista."""
    # TODO
    raise NotImplementedError


def interactions(rows, components, sigma_mult=2.0):
    """Par (a, b) (a < b) där bara_a och bara_b båda ligger inom sigma_mult*std(baslinje) från baslinjen,
    men full ligger mer än sigma_mult*std(baslinje) över baslinjen."""
    # TODO
    raise NotImplementedError

You write the code; tests you cannot see decide whether it holds up. Create a free account to run the lab.

Try the diagnostic Create a free account

Expected results

The ablation of the built-in pipeline (3 seeds) should single out exactly {bigram, klassvikt} as important and the stop-word filter as not shown to matter (Δ within the noise). Also look at the add-one rows: bara_bigram and bara_klassvikt are both close to the baseline while full is far above — an interaction. Eval: correct = 1 when the set is right.

Common mistakes

std with ddof=0 (population variance) instead of ddof=1.
Compares delta against the variant's std instead of the std of full.
Forgets that rows must come in configurations() order.
Runs different seeds for different variants — then you measure noise, not the component.