Demo · Label to eval loop
An eval score is downstream of a human decision you never see
the metric definition nobody re-reads, quietly capping what the agent can do
Start here · the everyday version
Say your friend Theo sells lemonade, and he thinks he has found a better recipe. He cannot be the judge himself, he will always root for his own new batch.
So he runs a taste test. A group of friends each drink a cup of the old and a cup of the new, and mark which one they like better.
Theo never watches them taste. He only sees the tally at the end. The new one won by twelve.
Now, that twelve is standing on something you never checked. It is only as good as the tasting you did not see. If the tasters paid attention and really compared the two cups, twelve means something. If half of them were distracted, or marked a cup without tasting, or guessed when they could not tell, then twelve is noise wearing a number.
And the damage is worse than a wrong score. When the tasting is sloppy the marks scatter. Some say the new one won big, some say the old one won, and when you add it up you get better by twelve, give or take twenty. Give or take twenty means it could just as easily be zero. You ran the whole test and you still cannot say the new recipe beat the old one. Not because the recipe is bad. Because the judging was.
So when the number comes back shaky, you check the tasting before you blame the lemonade. Careful tasting, and even a small win is real. Sloppy tasting, and even a big win might be nothing.
That is the whole idea on this page. The score you see is downstream of judging you never watched, and when that judging is careless you do not just lose points, you lose the right to trust the number at all. Everything below is that same idea, with the lemonade swapped for a metrics library.
In a metrics library the labels are the metric definitions people author. Their quality propagates through retrieval and retraining into the model eval score, and into the confidence you can put on the measured gain.
Users query an agent against the catalog. Ambiguous or colliding definitions get flagged, humans review and label them, governance certifies, retrieval re-indexes, and the eval score moves. Drag the one control below, label quality, and watch the eval score and its confidence band respond.
Governance postures · click one to play it through
Reviewed and certified loaded. Drag the control to make it yours.
One control. Everything downstream is a consequence of it.
The loop · live at this label quality
Model eval score, change vs no loop
+24.0 pts
Confidence band plus or minus 3.7 points, the measured interval is +20.2 to +27.7.
A real gain, but below the anchored loop. The band is tight enough to trust the direction.
Rater agreement
87.0%
chance corrected kappa 0.74. Assumes two independent labels on a balanced binary task.
Gold accuracy
93.0%
label quality measured on the seeded gold definitions.
Dataset label quality
93.0%
probability a certified definition is correct, the input to the eval curve.
The input side lesson
The consumption side asks whether the trust grade is honest. The input side decides whether there was anything honest to grade.
The same trust loop discipline runs on the authoring side of a metrics library. Human label quality on definitions is the causal upstream of the eval score. I ran this loop for real and it lifted a model's response accuracy 24 percent, validated by A/B test. This is that mechanism, translated to the input side, with its assumptions labeled on screen.
The consumption side siblings