Find the annotators whose labels are noise
Firing the bottom five annotators fires the wrong one
Eval datasets get labelled by a pool of annotators, and the pool always has a few who are careless or random. Ranking by raw agreement and cutting the bottom five punishes the diligent person who labelled thirty items and can miss the careless one who labelled eight hundred. Consensus tests each annotator against the pool and tells you whose labels to re-check.
The problem
Raw agreement is a coin flip at thirty items and a verdict at nine hundred
A diligent annotator with thirty items and 90% accuracy will post an agreement rate anywhere between 80% and 100% by luck alone. Put them on a leaderboard next to someone with nine hundred items and the low-volume annotator lands at the bottom or the top for no reason at all. The cut-the-bottom-five rule then removes a good annotator, keeps every bad one that happened to sit sixth, and nobody re-checks the labels the bad ones already wrote. The dataset carries that noise into every eval that uses it.
The insight
It is a small-sample ranking problem, and one with a real test attached
Agreement with consensus is a rate, and a rate from thirty trials deserves less belief than one from nine hundred. Empirical Bayes shrinkage fits the pool's own spread and pulls each annotator toward the pool in proportion to how little data they have. Then each annotator is a hypothesis: is this person below the pool, or just small? A one-sided binomial test answers it, and because thirty annotators are thirty tests, Benjamini-Hochberg decides how many can survive at once. The bar is stated with the chance-agreement rate next to it, so 88% agreement is read against the 52% a coin would get, not against 100%.
Per-annotator agreement with three-way majority consensus as a binomial rate, beta-binomial empirical Bayes shrinkage toward the pooled rate, a one-sided exact binomial test per annotator against the pooled rate, and Benjamini-Hochberg across the pool at q = 0.05, with chance agreement reported from the consensus base rate.
How it works
Four steps, no data science team
Item, annotator, label. Consensus computes the majority per item and each annotator's agreement with it.
The pool's prior is fitted from the data. Low-volume annotators move toward it, high-volume ones barely move.
Each annotator is tested for sitting below the pool. BH across all of them decides who survives at a stated false-discovery rate.
The annotators to re-check, the items their labels touch, and the estimated noise rate of the dataset as it stands.
Who it is for
Whoever owns the eval dataset
Teams that maintain an eval or training dataset labelled by more than a handful of people, in-house or through a vendor, and that currently manage annotator quality with a spreadsheet.
Pricing
- –Shrunk agreement per annotator
- –FDR-controlled flags
- –Chance-agreement bar
- –Live audit on ingest
- –Per-annotator drift
- –Vendor report export
- –Re-label queue integration
- –Multi-class and ordinal labels
- –Item difficulty estimates
- –SSO and audit log
- –API access
Competition
What exists, and what it does not do
| Who | What they do | The gap |
|---|---|---|
| Scale, Labelbox consensus QA | Track per-annotator agreement inside the labelling platform. | Raw agreement rates ranked in a table. No shrinkage, no test, no correction across the pool, and the vendor is grading its own workforce. |
| Krippendorff's alpha scripts | Report a single reliability coefficient for the dataset. | One number for the whole pool. It says the dataset is noisy and cannot say who is responsible or what to re-check. |
| A spreadsheet of agreement rates | Export, sort, look at the bottom. | Sorting is the entire problem. A thirty-item annotator sorts to the bottom by luck and a careless eight-hundred-item annotator sorts to the middle by volume. |
| Fire the bottom five | What most teams do today, on whatever schedule the vendor review falls. | Removes a fixed count regardless of how many are actually bad, and never triggers a re-check of the labels already written. |
The labelling platforms are the obvious builders and they own the data. A shrinkage step and a BH pass are a week of work for them; the reason they have not is that grading their own annotators harshly costs them money. The method assumes labels are independent given the item. If annotators see each other's labels or copy a model suggestion, agreement is inflated for everyone and the pool prior is wrong; the demo shows the chance-agreement bar and the prior strength as the diagnostic for that. It also assumes consensus is close to truth, which fails when two careless annotators land on the same item, so the noise rate reported is the disagreement rate with consensus and is stated as such. This fails as a company if teams keep buying labels by volume and never look at who wrote them.
Market
Priced against the labelling spend it protects and the re-labelling it makes targeted instead of wholesale
Every team with a labelled eval set and more than ten annotators is a candidate, and a vendor review or a benchmark dispute is the forcing event. Two thousand teams at the Team tier is $9.6M ARR; the Scale tier is priced against one wholesale re-label of a dataset.