Baselines and controls

«Our model gets 91 %.» Is that good? If 90 % of the examples belong to one class, «always guess the majority» gets 90 %. Then 91 % is nearly nothing.

Baselines in ascending order:

1. Chance / the majority — the floor.
2. A simple rule — a keyword, a threshold, «the same as yesterday». Often surprisingly strong.
3. A simple model — logistic regression on TF-IDF, k-NN on embeddings. Fast to train, hard to beat with little data.
4. The standard method — the pre-trained model everybody uses, properly tuned.
5. A human being — what does a trained person get on the same test set?

Your method should be reported alongside all of these. And the baseline should have been given the same love — the same time on hyperparameters, the same data — otherwise you are comparing your best day with its worst.

import numpy as np
from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score

baselines = {
  "majority": DummyClassifier(strategy="most_frequent"),
  "chance": DummyClassifier(strategy="stratified", random_state=0),
  "tfidf+lr": make_pipeline(TfidfVectorizer(ngram_range=(1, 2), min_df=2), LogisticRegression(max_iter=1000, C=3.0)),
}
for name, m in baselines.items():
    s = cross_val_score(m, texts, y, cv=5, scoring="f1_macro")
    print(f"{name:10s} {s.mean():.3f} ± {s.std():.3f}")
# majority   0.31 ± 0.00
# chance     0.33 ± 0.02
# tfidf+lr   0.78 ± 0.03   ← your transformer has to beat this clearly, with the same cv