A support chatbot, routed to the right model for each task — 32.5% cheaper, same accuracy.

We don’t trust a single run or a leaderboard score. For each of the 11 actions the chatbot performs, we wrote tasks containing the real data the model would see and the correct answer it should produce. Then we ran every candidate model against every task 10 times — 1,250 responses in all — and had a strong independent model grade each one against the ground truth.

• Repetition over single runs. One good answer can be luck. Ten runs per task expose the models that are consistently right.
• Graded against ground truth.A model “passes” a task only when an independent judge scores it ≥4 of 5 against the known-correct answer — not on vibes.
• Cost tracked per task.Every response’s real token cost is recorded, so we can price each routing decision exactly.

Accuracy across all 11actions, ranked. Note the strongest model isn’t far ahead of a much cheaper open-weights one on routine work:

Judge: claude-sonnet-4-6. Pass = score ≥ 4/5 vs ground truth, averaged across all 11 actions.

The deliverable isn’t “use this model.” It’s a per-action map of the cheapest model that still passes. Most actions go to a cheap, fast model; only the genuinely hard reasoning tasks need the expensive one.

• The strongest model is not the most cost-effective everywhere — only 2 of 11 actions actually need the top model.
• An open-weights model (gpt-oss-120b) matched the frontier model on 9 of 11 routine actions at a fraction of the cost.
• Two tasks exposed ambiguous ground truth that all models struggled with — caught by the suite-quality guard, not blamed on the models.
• Per-action routing cut blended cost 32.5% versus running everything on the strongest model, with no measurable accuracy loss.

The 32.5% figure on this page is computed from a real benchmark run — 5 models × 25 tasks × 10 runs, 1,250graded responses. We deliberately don’t name the client or show their data; the methodology and the numbers are real, the customer specifics stay private. When tasks are too ambiguous for any model to answer reliably, our process flags them as a ground-truth problem to fix — not as a model to blame.