Annotation and labelling of data

The quality of a labelled dataset is set by the instructions, not by the annotators' diligence. A good guideline has: a definition per class, positive and negative examples, explicit edge cases with a decision, and a rule for «don't know».

Agreement is measured with Cohen's kappa (κ): the share agreed, corrected for chance. κ < 0.4 = the guideline is unclear; 0.6–0.8 = usable; > 0.8 = strong. Per cent agreement is misleading under imbalance (two annotators who always say «neutral» agree 95 % of the time).

The process: a pilot on 50 examples with 2–3 annotators → κ → discuss the disagreements → revise the guideline → pilot again → only then the whole set, with 10 % overlap for ongoing checking. Disagreements are settled by a third person (adjudication) and become examples in the guideline.

import numpy as np

def cohens_kappa(a, b):
    a, b = np.asarray(a), np.asarray(b); labels = np.unique(np.concatenate([a, b]))
    po = np.mean(a == b)
    pe = sum(np.mean(a == l) * np.mean(b == l) for l in labels)
    return (po - pe) / (1 - pe) if pe < 1 else 1.0

a = ["pos", "neg", "neu", "pos", "neu", "neu", "neg", "pos", "neu", "neu"]
b = ["pos", "neg", "neu", "neu", "neu", "neu", "neg", "pos", "pos", "neu"]
print(np.mean(np.array(a) == np.array(b)), round(cohens_kappa(a, b), 2))   # 0.8  0.68

# imbalance: 95 % agreement can be κ ≈ 0
x = ["neu"] * 95 + ["pos"] * 5; y = ["neu"] * 95 + ["neg"] * 5
print(np.mean(np.array(x) == np.array(y)), round(cohens_kappa(x, y), 2))   # 0.95  -0.05

For more than two annotators: Fleiss' kappa or Krippendorff's alpha (the krippendorff package).