Judge bake-off: can a cheaper model do the job?

Token value is our first principle — so we don’t assume a smaller model is good enough, we measure it. Here we take a real job our own system does with a premier model and ask whether cheaper, faster models reach the same verdicts. Same task, same inputs, same prompt — the only variable is the model.

The job: bot-coverage gap judge (did the assistant answer from site data, or punt?) Our site assistant is audited by asking it questions the site can answer; a “judge” model reads each answer and decides answered or punted. We normally use a premier judge — this test asks which cheaper model could take its place.Reference: Opus 4.8 (ref) (claude-opus-4-8) · 16 probes · run October 8, 2026 · total test cost $0.1377

Best value judge

Haiku 5.5

100% agreement with Opus 4.8

Cost

~44× cheaper

$0.0011 vs $0.0471 for the reference

Speed

~1.6× faster

1078ms vs 1712ms per verdict

Read this run narrowly: the reference flagged a gap on all 16 probes, so the set held no answered probe. It shows which judges catch a gap the way the reference does. It does not test whether a judge raises false alarms on a good answer, so 100% here does not mean “as accurate as the reference”.

Every candidate, scored against the reference

Judge model	Agreement	Gaps flagged	Latency	Cost
Opus 4.8referenceanthropic/claude-opus-4-8	—	16	1712ms	$0.0471
Haiku 4.5anthropic/claude-haiku-4-5-20251001	100%	16	1134ms	$0.0072
Haiku 5.5best valueanthropic/claude-haiku-5-5	100%	16	1078ms	$0.0011
Llama-3.3-70Bgroq/llama-3.3-70b-versatile	not measured	0 · 16 err	144ms	$0.0000
Kimi K2 (OpenR)openrouter/moonshotai/kimi-k2-0905	100%	16	1801ms	$0.0035
Kimi K3 (OpenR)openrouter/moonshotai/kimi-k3	100%	15 · 1 err	4832ms	$0.0413
Muse Spark 1.1openrouter/meta/muse-spark-1.1	100%	15 · 1 err	4967ms	$0.0358
Gemini 3.1 Flashgemini/gemini-3.1-flash-lite	93.8%	15	1039ms	$0.0017

“Agreement” = share of probes where the model reached the same answered/punted verdict as the reference, counted over the probes it returned a readable verdict for; unreadable ones show as “err”. A judge with no readable verdict at all is shown as not measured: that is a failed call or parse, not a 0% judge. A judge that flags far more gaps than the reference is over-strict — false alarms, the costly direction for an audit.

Where the models disagreed

The honest part: 1 of 16 probes split the judges. = “answered”, = “flagged a gap”.

Probe	Opus 4.8	Haiku 4.5	Haiku 5.5	Llama-3.3-70B	Kimi K2	Kimi K3	Muse Spark 1.1	Gemini 3.1 Flash
what is the highest rated model for classification?

How this was measured

• The site assistant was asked a fixed set of probes once; its answers were frozen.
• Every candidate model then judged that same frozen set with the identical prompt and parser — the only variable is the model.
• “Agreement” is measured against the premier reference, not against ground truth — the reference is our chosen best judgment, not an oracle. A single run has real variance; the value is in re-running it as models change.
• This is a manual, reproducible test (not a daily cron) — re-run when the models or the site change enough to re-check.

Other bake-offs

This is one of several. Router comparison asks whether our own routing beats OpenRouter's automatic one, and the takeoff bake-off asks which models can be trusted with a construction quantity. See them all, with what each found.

All bake-offs The full model leaderboard Why token value comes first

Measured test