Everyone talks about model quality. In practice, the ceiling on training data is annotation quality, and that ceiling shows up long before your loss curve flatlines. We learned this running a VQA pipeline at roughly 215,000 image–question–answer pairs per month. Throughput was never the hard part. The hard part was discovering, three weeks into a batch, that a slice of “completed” tasks had been labeled under an ambiguous rubric — same headcount, same SLA, radically different usable yield. You do not catch that with spot checks. You catch it when agreement across raters collapses on tasks that looked fine in isolation.
Headcount is a capacity metric. Inter-annotator agreement (IAA) is a quality metric. They correlate weakly.
Consider two batches, each with three raters per item. Batch A: on a traffic-scene relevance question, all three pick “relevant” on 94% of items and disagree only on borderline cases. Batch B: same rubric, same throughput, but raters split 40/35/25 on items where the policy is unambiguous. Same annotator count. Batch A is shippable; Batch B will poison a preference model. Completion rate measures effort, not alignment.
We use three coefficients, selected by rater count and data shape — not by whichever number looks best in a slide.
Cohen’s Kappa is for exactly two raters and categorical labels. It compares observed agreement to chance agreement given each rater’s marginal distribution. It is fast and the right tool for pilot studies or pairwise adjudication. Its limitation is structural: it expects paired ratings and ignores missing data. If one rater skipped half the batch, you drop those items or impute. In a live workbench with partial completion, that is not an edge case.
Fleiss’ Kappa extends the idea to three or more raters on the same items. It works when every item has the same number of ratings and categories are nominal. It breaks down when raters per item vary, when labels are ordinal but treated as nominal, or when missingness is uneven — the normal state of a production queue. We use Fleiss on clean audit subsets, not as the default gate on live traffic.
Krippendorff’s Alpha is our default for production with three or more raters. It accepts missing values, supports nominal through ratio scales, and handles variable ratings per unit. Alpha near one means alignment beyond chance; near zero means you might as well sample labels at random. When four of nine raters are active and two are still in queue, alpha uses every pairable judgment instead of discarding the task. That is why it drives our auto-router on multi-rater work.
We treat α ≥ 0.70 as the minimum for production on multi-rater tasks. Krippendorff’s guidance treats 0.667–0.80 as tentative; in our runs, batches below 0.70 correlated with relabel rates that erased throughput gains. Above 0.80, review cost drops; below 0.60, you are often re-annotating from scratch.
Below threshold, failures are not random noise. Raters agree with each other but against policy, or they disagree on every ambiguous frame while agreeing on easy negatives — usually a rubric that was never internalized. We have seen α ≈ 0.45 where raw percent agreement looked high because everyone picked the majority class. The coefficient catches that; a simple agreement rate does not. For two-rater paths we use Cohen’s κ with a 0.60 gate, expecting a third rater before merge.
Every task is scored, compared to threshold, and routed without manual triage. If agreement passes, it is approved. If not, the path depends on how many raters have weighed in.
ADDITIONAL_RATER fires when fewer than three ratings exist and agreement is low. We do not yet have enough signal — send another pass. This fixes many failures from one confused rater or an incomplete trio.
EXPERT_QUEUE fires when three or more raters are in and alpha is still below threshold. That reads as rubric ambiguity or bad task design, not a missing vote. A fourth generalist rarely helps; you need a binding policy call. The distinction matters because the costs and SLAs differ. Treating every failure as “add another annotator” burns budget; sending two-way splits to experts too early burns capacity on tasks that only needed one more vote.
Our spatial and driving-scene rubrics are calibrated against Stanford Phantom, Cornell RHyME, and RoadscapesQA (arXiv:2602.12877), so IAA measures agreement on tasks comparable to what AV and robotics actually evaluate.
We run a nine-annotator workbench on high-volume VQA: overlapping ratings, partial completion, automatic reroute on every batch close. In one push we onboarded roughly 1,500 evaluators in seven days without lowering the alpha gate. Early cohorts spread wide — many batches at 0.55–0.65 while rubrics settled — then stabilized: most traffic clustered 0.72–0.88, with a tail held for additional rating or expert review instead of slipping into training mixes.
IAA is not a monthly report. It is the control loop that decides whether a batch is data or debt. Measuring agreement correctly, and acting on it without triage, was the difference between shipping 215K pairs that improved a model and shipping 215K pairs that had to be forgotten.