data-labeling

Overview

Data labeling turns raw examples into training signal: class labels, preference rankings, bounding boxes, span annotations, or structured judgments. The uncomfortable truth of the field is that model quality is bounded by label quality — a state-of-the-art architecture trained on noisy labels underperforms a simpler model trained on clean ones. Professional labeling is not "pay people to click"; it's a measurement operation with guidelines as the spec, agreement metrics as the QC, and adjudication as the error correction.

The pipeline has four parts: task design (what exactly is being labeled and what the edge cases are), guidelines (the written contract between you and annotators), the labeling pass itself (with redundancy for measurement), and quality control (agreement metrics, gold questions, adjudication). Whether annotators are in-house experts, crowd workers, or AI models doing the labeling, the same structure applies — only the cost of iteration changes.

Labeling is the foundation every model stands on. Cracks here propagate silently into every downstream metric, which is why the discipline matters more than the tooling.

When to use

• Building any supervised dataset: classification, NER, preference pairs, safety labels, evaluation sets.
• Diagnosing a model that "should work" but doesn't — label noise is a prime suspect.
• Creating gold-standard eval sets where label correctness is the whole point.
• Scaling annotation with AI labelers while keeping a human-verified quality bar.
• Regulated or high-stakes domains where label provenance and agreement must be auditable.
• Before any "the model is bad" conclusion — rule out "the labels are bad" first.

Core concepts

• Annotation guidelines: the single highest-leverage artifact. Good guidelines define each label with positive and negative examples, adjudicate the 20 most common edge cases explicitly, and are versioned like code. If two careful readers disagree after reading the guidelines, the guidelines are broken, not the readers.
• Inter-annotator agreement: measured with Cohen's kappa or Krippendorff's alpha (not raw percent agreement, which inflates with class imbalance). Track per-label agreement — aggregate scores hide the labels annotators actually confuse.
• Redundancy: labeling each item 2–3 times lets you measure agreement and resolve by majority or adjudication. The cost is real; spend it on ambiguous slices, not on trivially easy items.
• Gold questions: items with known answers seeded into the queue to measure individual annotator accuracy continuously. Annotators falling below threshold get retrained or removed.
• Adjudication: an expert resolves disagreements. The adjudicated set doubles as the hardest, most informative training data — and as material for guideline v2.
• AI-assisted labeling: models pre-label, humans correct. Faster, but watch for automation bias — humans rubber-stamping model outputs. Mitigate with blind review samples and by measuring correction rates.
• Label schema design: the set of labels and their definitions. Smaller schemas are more reliable; every added label multiplies ambiguity. Design the schema from the pilot data, not from a priori taxonomies.
• Provenance: who labeled what, when, under which guideline version. Audit trails turn "the data says" into a checkable claim.

Practical workflow

1. Define the label schema. Enumerate labels, write one-paragraph definitions with examples. Keep the schema as small as the task allows — every added label multiplies ambiguity.
2. Pilot on 100–200 items. Two annotators, full guidelines. Measure agreement per label. Anything below ~0.7 kappa (for trained annotators) means the guidelines or schema need work — fix before scaling.
3. Write guidelines v1 from the pilot. Every disagreement in the pilot becomes an explicit rule or example in the guidelines. This is the step most teams skip and most regret skipping.
4. Qualify annotators. A paid trial batch with gold questions before production work. Annotator quality varies enormously — selection beats training.
5. Scale with QC. Deploy gold questions (5–10% of queue), track per-annotator accuracy, and double-label a sample for ongoing agreement measurement.
6. Adjudicate disagreements. Expert review of conflicting labels; feed resolutions back into guideline updates. Version the guidelines and re-annotate affected slices when definitions change.
7. Audit the final dataset. Sample 2–5% for independent expert review. Report: agreement metrics, adjudication rate, annotator accuracy distribution, guideline version. Ship these stats with the dataset.

Checklist before training on a labeled dataset:

• Agreement measured on a representative sample, reported per label.
• Guideline version recorded; edge-case rules documented.
• Gold-question accuracy thresholds enforced during collection.
• Annotator qualification process documented.
• A final audit sample reviewed by someone who didn't write the guidelines.

Common pitfalls

• Guidelines written once, never updated. The pilot disagreements are the guidelines trying to tell you what's ambiguous. Listen.
• Percent agreement instead of kappa. With 95% negative class, "always label negative" scores 95% agreement and teaches nothing. Use chance-corrected metrics.
• No redundancy anywhere. Without overlap you can't measure agreement, and without measurement you're guessing about quality.
• Annotator pool mismatch. Crowd workers for expert tasks (medical, legal) without qualification screening produces confident, wrong labels.
• Automation bias in AI-assisted labeling. If humans correct less than ~5% of model pre-labels, they're probably not really reviewing. Insert blind human-only samples to check.
• Label schema churn mid-project. Changing definitions halfway invalidates earlier labels. Version guidelines and re-annotate affected data, or quarantine by version.
• Skipping annotator qualification. Assuming all annotators are equivalent. The quality spread is huge — measure individuals and act on it.
• No provenance. Labels without records of who, when, and under which guidelines. When problems surface, you can't trace or fix them.