Loading benchmark data…
EyesInAI·Loading live benchmark data
A bake-off is a real, dated, reproducible run where the only thing that changes is the model. Same task, same inputs, same prompt. We run one when we need to answer a specific question with money attached — is the cheap model good enough for this job, is our routing actually worth it, can this model be trusted with a number.
Each entry below opens to show what the run found and — the part that matters more — what it does not prove. Every figure is read from the same data file its detail page renders, so nothing here can drift from the run it describes.
Coding agents load a short summary of a product charter (a ledger of settled decisions) at every session start. Can a model write it without dropping a condition an agent needs: by rewriting it, by only ranking decisions that are then pasted verbatim, or with a second model reviewing that ranking?
3 real charter ledgers, frozen once; the same prompt for every model, one attempt; a blind Claude judge audited each summary against its full ledger. All runs on flat-rate plans.
One run per model per charter (3 per model); round 3 is one run per charter, so its change is suggestive, not proven. The judge is a Claude model grading Claude summaries. Round 2 had a larger budget than round 1. Charter contents are internal, so only scores are published.
Several parts of this site, and our clients' bots, call a model for a fixed job. When a newer or cheaper model appears, from any provider, can it do that exact job as well as the model in use, for less?
For each task, every model gets the same real inputs and prompt as the model in use. Judges from different providers grade each answer blind; an input passes when more than half of its gradings pass, and invented facts are counted separately.
Each test covers one job with a small set of real inputs (14 in the latest). A result holds for that job and that prompt. It does not rank models in general, and the judges can be wrong.
Switchyard can send each type of request to a different Claude model, which saves money only if the map from request type to model is right. The first map was written by hand. Which model does each request type actually need?
5 hand-written problems per request type, 3 runs each, every model on the same problems. Claude Opus 5.5 graded each answer against a written expected answer. Costs are answer tokens at list price, with the judge excluded.
A test tenant with a fictional company: this proves the method, not a client's bar. 15 graded answers per cell, so one miss falls below the 95% bar. The deep-analysis problems were well-defined calculations, which says nothing about open-ended analysis.
The Claude 5.5 models think by default. We send each its lowest-thinking setting to save tokens and time. Does that lose answers?
The same problems as the Claude tier bake-off, every model at both settings in one run, graded by Claude Opus 5.5 against written expected answers. Answer cost includes thinking tokens, at list price.
One fictional test tenant with short, well-defined requests, 10–20 answers per cell. Tasks that need a long chain of reasoning could gain more from thinking.
Claude Haiku 5.5 costs 5× more per token above 100K input tokens. Under and over that line, does it still find what it is asked for, and what does a long prompt cost on each tier?
Real Python source from our own codebase as the prompt, exact-answer questions at several depths, graded by string match. Every model at its lowest-thinking setting, with prompt caching.
Five questions per size and model, one run, simple by design. Costs are computed from token counts at list price, not read from a bill.
Claude Code lets you choose the model and the effort. On the same coding tasks as the coding-CLI bake-off, which combination of Haiku, Sonnet or Opus 5.5 and low, medium or high effort passes the hidden tests?
The coding-CLI bake-off's 24 tasks and hidden tests, each combination as a Claude Code agent with a 25-minute limit. Costs are Claude Code's list-price figures, not a bill.
One run per task and setting. Effort is passed to Claude Code but not reported back, so it is not verified. The agents could run plain pytest only.
Our system runs several small, frequent jobs on Claude Haiku: picking a model, grading against a rubric, scoring answers, screening bot replies and labelling intent. Before moving them to Haiku 5.5, does it do each job at least as well as Haiku 4.5?
Each job ran with its production prompt and parser on the same frozen inputs for both models, thinking off, each run under its own spending cap. Only aggregate scores are published; the raw log holds real requests and client answers.
Small samples and mostly one run per job. The intent row measures agreement with a Sonnet 5 reference, and that reference matched production's logged labels on only 36% of rows, so it is not accuracy. No cost is measured on this page.
Four coding agents, the same 24 bug-fix and feature tasks, graded by hidden tests the agent never saw. Who passes, how fast, and how often does an agent say DONE when the tests fail?
Each agent ran headless and sandboxed, one after another, with the same prompt and grader. A pass needs the hidden tests and the existing suite green on a clean copy of the work, inside a 25-minute limit.
One run per task with no repeats, on 24 mostly-Python tasks, with agent versions as of 2026-10-03. A one-task gap between two agents is inside the noise.
Answer a visitor from a website's own pages, or refuse. 6 of the 15 questions cannot be answered from the source, so a confident, factually-true answer on those rows is scored as a failure — the model cannot have grounded it.
The grounding was frozen once — same pages, same order, same budget for every model — then only the model varied. Two independent confirmation runs; scores are mechanical and every apparent regression was re-read by hand.
That these models are interchangeable in general. The set covers one corpus at one prompt, and six of eight scoring perfectly means it has hit its ceiling — it cannot rank the top of the field, only show that this particular job is easier than assumed. Nothing here tests prompt injection or conflicting sources.
Fill a 27-line construction budget from real drawings and a soil study, where inventing a number is worse than leaving the cell blank. Which models can be trusted with a figure that costs money if it is wrong?
A deterministic harness fills the budget from the same drawings for each model, then two independent checks run over the result: a citation check for invented quantities and a scope auditor for figures applied to the wrong extent. The engine validates the arithmetic separately.
Nothing here shows a model can fill correctly when the data is good. The reference run itself could only defensibly fill 1 of 27 lines on most runs, because the drawings lack zone-level areas. Every agreement figure is capped by that ceiling.
For real chatbot turns on live client sites, does picking the model ourselves from measured benchmarks beat handing the choice to OpenRouter’s automatic router — on answer quality and on cost?
Each turn is run through both routers with identical context, then both answers are scored blind by an LLM judge on a 1–5 scale. Cost is the real billed cost of the model each router actually chose.
A sample this size (57 turns) is directional. It does not prove either router is permanently cheaper — the last two runs disagreed on exactly that — and with 50 of 57 diverged turns scoring level, nobody gets to claim a quality edge either. The durable finding is the method: re-run, publish the number you get, and route on measurement rather than on either month's screenshot.
We use an LLM to judge whether an assistant answered from site data or punted. That judge is an instrument, and running it on a premier model costs real money. Can a cheaper model do the grading without changing the verdict?
Every candidate grades the same probe set with the same rubric and prompt, with only the model varying. Agreement is measured against a fixed premier-model reference, and unparseable responses are counted rather than discarded.
16 probes on one task. The reference flagged a gap on every probe, so false alarms on a good answer were not tested. This says which judges track our reference on THIS grading job — it does not transfer to a different rubric, and it does not establish that the reference itself is correct. The judge is an instrument; this run calibrates it, it does not validate it.
The useful part of a bake-off is rarely the winner. It is the failure mode you didn't know to look for — a model that fills every cell and gets the arithmetic wrong, a judge that returns text nothing can parse, a router that picks differently every time and lands in the same place. Those are the findings that change what we ship.
So each run above is published with its ceiling attached. A bake-off on one task, with one reference, at one point in time, tells you about that task. We say so rather than letting a narrow result read as a general claim — and when a run is superseded by a fix, the old numbers stay visible, marked, rather than quietly disappearing.