The bench said yes. Real traffic said no.

Our action-bench watches every task type a client's assistant serves and continuously asks: is there a cheaper model that passes this job? On 2026-08-16, for the DLR back-office assistant, it had a confident answer: move nine actions from claude-sonnet-4-6 to gpt-oss-120b and save 43–80% per action. The evidence was as clean as synthetic evidence gets: 100% pass on every one of the nine suites, judge scores at or near a perfect 5.0.

Then the second layer ran. Before any swap applies, the candidate is scored against the live model's own recent production turns — real questions from real usage, graded by the same reference judge against the answer that actually served. Across 409 real turns, the candidate scored between 1.0 and 2.0 out of 5. The incumbent scored 5.0 on the same turns. Every recommendation was held. Zero were confirmed.

Read one row out loud to feel the trap: open-invoices promised −79.8% with a flawless synthetic record, and delivered a 1.0 out of 5 on the 27 real turns it would have answered. The dashboard would have stayed green — cost down, tests passing — while the product quietly broke.

The synthetic suite isn't bad. It's made of real task shapes, graded by a judge we calibrate separately, and the candidate genuinely passed it. The gap is structural: a suite is a sampleof a workload, and production traffic is the workload — including the ambiguity, the follow-ups, the messy phrasing, and the long tail no suite author thought to include. A model can be good at the sample and bad at the distribution. The only way to catch that is to score it on the distribution — which is exactly what the veto layer does, using the incumbent's served turns as the exam.

This is also why the veto is not a reason to skip synthetic benching. The suite finds the candidates worth testing and catches regressions cheaply every day; the veto is the last gate before real money and real users are involved. Cheap screen, expensive confirmation — the same architecture as any good testing pipeline.

Run-an-eval-on-your-workload is becoming a product category: OpenRouter's Ori Eval scores models against your codebase for $10–30 a run, Supabase published an open benchmark of coding agents on real Supabase tasks, and every vendor newsletter now tells you public leaderboards don't predict your use case. They're right, and we've been saying the same thing — it's the founding premise of this site.

But every one of the nine rows above would have passeda one-shot workload eval. The candidate did pass — 100%, on task shapes drawn from the very workload it was being hired for. A ranking produced once, however customized, is a photograph; traffic is a movie. The protection came from the parts a one-shot product doesn't have: the benchmark ran continuously, the candidate was scored on the live turnsit would inherit, and the swap could not apply while the two disagreed. Measurement that can't say “no” to a saving isn't measurement — it's a purchase justification.

Three portable rules fall out of the table above. First, a synthetic pass is a hypothesis, not a verdict — treat any eval, ours included, as the reason to run a confirmation, never as the confirmation. Second, score candidates on served traffic— the incumbent's real turns are the one exam that actually matches the job, and replaying them costs a few dollars against a regression's real price. Third, wire the veto into the switch — a recommendation a human has to remember to distrust will eventually ship; ours is held by the same system that would apply it, so the contradiction blocks the swap mechanically.

Method notes: judge scores are 1–5 against the served answer's ground truth, same reference judge on both arms; real-turn counts per action are in the table (409 total). The snapshot this page renders from is committed alongside it — when a later analyst run changes the picture, the numbers here change with it, or the page is wrong and we fix the page.