Judge bake-off: can a cheaper model do the job?

Token value is our first principle — so we don’t assume a smaller model is good enough, we measure it. Here we take a real job our own system does with a premier model and ask whether cheaper, faster models reach the same verdicts. Same task, same inputs, same prompt — the only variable is the model.

The job: bot-coverage gap judge (did the assistant answer from site data, or punt?) Our site assistantis audited by asking it questions the site can answer; a “judge” model reads each answer and decides answered or punted. We normally use a premier judge — this test asks which cheaper model could take its place.Reference: Opus 4.8 (ref) (claude-opus-4-8) · 23 probes · run July 14, 2026 · total test cost $0.0864

Best value judge

Llama-3.3-70B

95.7% agreement with Opus 4.8

Cost

~18× cheaper

$0.0037 vs $0.0660 for the reference

Speed

~5.7× faster

417ms vs 2363ms per verdict

Every candidate, scored against the reference

Judge model	Agreement	Gaps flagged	Latency	Cost
Opus 4.8referenceanthropic/claude-opus-4-8	—	6	2363ms	$0.0660
Haiku 4.5anthropic/claude-haiku-4-5-20251001	82.6%	8	2111ms	$0.0114
Llama-3.3-70Bbest valuegroq/llama-3.3-70b-versatile	95.7%	5	417ms	$0.0037
Kimi K2 (OpenR)openrouter/moonshotai/kimi-k2-0905	65.2%	12	2205ms	$0.0053
Gemini 3.1 Flashgemini/gemini-3.1-flash-lite	73.9%	12	786ms	$0.0000

“Agreement” = share of probes where the model reached the same answered/punted verdict as the reference. A judge that flags far more gaps than the reference is over-strict — false alarms, the costly direction for an audit.

Where the models disagreed

The honest part: 14 of 23 probes split the judges.= “answered”,= “flagged a gap”.

Probe	Opus	Haiku	Llama-3.3-70B	Kimi	Gemini
what is the highest rated model for classification?
what is the highest rated model for coding?
what is the highest rated model for extraction?
what is the highest rated model for format compliance?
what is the highest rated model for JSON / structured output?
what is the highest rated model for long-context recall?
what is the highest rated model for multi-step logic?
what is the highest rated model for reasoning?
what is the highest rated model for speed?
what is the highest rated model for tool use?
what governance or policy coverage do you have?
do you have a page about data centers?
which providers do you benchmark?
do you host chatbots for clients?

How this was measured

• The site assistant was asked a fixed set of probes once; its answers were frozen.
• Every candidate model then judged that same frozen set with the identical prompt and parser — the only variable is the model.
• “Agreement” is measured against the premier reference, not against ground truth — the reference is our chosen best judgment, not an oracle. A single run has real variance; the value is in re-running it as models change.
• This is a manual, reproducible test (not a daily cron) — re-run when the models or the site change enough to re-check.

The full model leaderboard Why token value comes first

Measured test