Manual data labelling

A model learns from labelled examples: an image with the label “cat”, a customer review with the label “positive”. Someone must set these labels manually. It sounds simple until you do it yourself.

“Customer service was not bad.” — Positive? “Good delivery, but the product was broken.” — ?

Every ambiguous case is a choice. Two people can label the same text differently. That choice is encoded into the model. That is why we write instructions (guidelines) and measure how often the labelers agree with each other.

Do this: Write 30 short messages (or extract them from a customer chat you have access to). Label each message as “friendly”, “neutral”, or “rude”. Have a colleague label the same 30 messages without seeing your answers.

Calculate: How many times did you agree on the label? 24 out of 30 = 80% agreement. Review the 6 where you disagreed. Write a rule that resolves them (e.g. “Irony counts as rude if …”). Relabel. Did the agreement increase?

This is exactly how professional datasets are built.