We ran our router against OpenRouter's for 65 real turns. Here is what actually happened.

Across 65 turns on three chatbots, our routing cost $0.03310 against auto-beta's $0.08000 — 2.4× cheaper in aggregate. On quality, of 65 turns where the two routers picked different models, 56 were graded a tie, we scored higher on 8, and auto-beta scored higher on 1. The two routers disagreed on the model for every single turn.

The aggregate hides the most interesting thing, so look at the last column: the cost gap runs from 6.0× down to 1.0×. On one bot the advantage nearly disappears.

On the DLR back-office assistant the two arms cost within a few percent of each other. Auto-beta happened to route those turns to genuinely cheap models that were also good enough — which is exactly what a router is supposed to do. There was no cost story to tell. What remained was a quality gap (4 divergent turns to us, 0 to them) and a latency gap: we answered roughly three times faster.

We could have left that bot out. The aggregate looks better without it, and nobody would have known. But a comparison you only publish when it flatters you isn't a measurement, it's an advertisement — and the whole argument for measured routing is that we publish the number even when the number is inconvenient. The honest read: cost is the headline only when popularity happens to favour an expensive model.When it doesn't, our edge is quality and speed, which is a narrower claim and a truer one.

The version of this comparison we published on 22 July claimed 7.4× cheaper. That number was real, but it was measured against the wrong opponent. OpenRouter ships two routers: openrouter/auto, a general meta-router that has existed since 2023, and openrouter/auto-beta, the task-aware one that launched in July 2026. Our harness called auto while every comment, log line and published page described the auto-beta mechanism. They are different model ids that route differently — on an identical prompt, one picked a model costing about twenty times what the other picked.

Re-running against the router we had been describing cut our advantage from 7.4× to 2.4×. That is the correction, and it is the kind of thing that only surfaces if you check your own claim before someone else does. The failure was ours: the harness was written before auto-beta existed, the constant was never revisited, and the label on the chart drifted away from the thing being measured. It now records which rival each run used, so a future run can't quietly compare against a different opponent.

Spend-share routing answers “what is everyone else buying for prompts shaped like this?” It is a good question and a genuinely useful default — it needs no setup, no benchmark, and no knowledge of your workload. But popularity is a proxy. It tells you what the crowd bought, not what passes yourtasks. Measured routing answers a narrower question: on a benchmark of this client's own work, which is the least expensive model that actually scored a pass? When those two answers coincide, the routers tie — which is what 56 of 65 divergent turns look like. When they diverge, measurement wins on cost, on quality, or both.

That is also why the advantage is uneven rather than uniform. We are not claiming a better model; both arms draw from the same pool. We are claiming a better selection signal— and a selection signal only pays when the crowd's default is wrong for your workload. Sometimes it isn't.

65 turns across three bots is a small, dated sample. It is directional, not a guarantee, and a single run carries real variance — the reference judge is our chosen best judgment, not an oracle. The honest summary is three sentences: the two routers almost never pick the same model; quality is close to a wash with a consistent edge to us (8 wins to 1 across 65 divergent turns); and the cost advantage is real but varies by an order of magnitude depending on the bot.

Every figure on this page is read from the same published file that drives the full comparison, including the per-turn rows — so this article cannot drift from the measurement behind it. Last run 2026-07-28. For what the two routing philosophies are actually arguing about, see spend share vs. measured quality.