The routing Pareto trap: the bill drops, the product breaks

Writing in Towards Data Science, Pratik Rupareliya describes a customer-support AI agent for a SaaS product with ~4M monthly users, running every query on one capable reasoning model and a six-figure monthly bill. The team built the consensus playbook: a light classifier in front, labeling each query “simple” or “complex,” routing the simple 65% to a model a quarter of the cost and keeping the complex 35% on the capable one.

It was textbook. The classifier hit 94% equivalent quality on a 5,000-query holdout; rollout was gradual; every metric held green; the bill fell to ~40% of its prior level. The CFO sent a thank-you note. And then, over the next three months, customer satisfaction dropped, churn ticked up, and human support volume climbed — until the inferred cost of the quality loss came in at a conservative 4–5× the savings. The team had not won. They had moved the cost somewhere they weren't measuring.

The most important part of the story is why nobody saw it. The measurement architecture had been built for a single model, and the routing layer quietly broke every assumption underneath it:

• Aggregate review hid the gap.Human review wasn't split by tier, so 65% cheap-model samples (equivalent on the easy center) diluted the failures on the hard edge to invisibility.
• The regression suite was static. Curated six months earlier with no notion of routing, it tested an idealized distribution — the cheap model passed the suite and degraded on the live edge.
• The feedback widget was too noisy. Thumbs-down at ~3 per 1,000, skewed toward already-frustrated users — unable to detect anything short of a major regression.

None of these were caused by routing; they were latent, and routing exposed them. With one model there was one quality distribution to measure. Routing created two, and the architecture could not see them separately. The drift began in week three, was misread as “provider model-version drift” in week six, and only hit business metrics by week thirteen.

The reason this is structural, not situational, is geometry. Query difficulty follows a power law: a big easy center, and a long tail of ambiguous, context-dependent queries. Frontier models are over-provisioned for the center — which is exactly why the savings are real — but the classifier cannot separate the easy center from the hidden long tail at decision time, because it only sees the surface form.

Three mechanisms compound. Surface form is a poor proxy for depthexactly where model choice matters most — the classifier is well- calibrated where it doesn't need to be and poorly calibrated where it does. Small models fail confidently— a complete, plausible, wrong answer is harder to flag than a frontier model's hedge or clarification. And distributions drift — a classifier trained on six months of history misroutes a growing share as new products and cohorts arrive, while the savings stay flat and the quality cost grows silently.

The cost of the failure, crucially, lands in a different cost center — the human support team, retention, customer experience — none owned by the team that did the optimization. Each team optimizes its own budget; the combined optimization is negative. That asymmetry is the Pareto trap. (Rupareliya found the same shape in two more audits — a mid-market SaaS and, most sharply, a fintech case where “informational” queries carried regulatory weight and a wrong answer is a violation, not an inconvenience.)

The article's conclusion is one we'll happily put our name next to: the measurement architecture matters more than the routing decision. A team with good per-tier observability can route aggressively because it will catch the drift; a team without it cannot safely run any routing layer at scale. The three additions it prescribes are exactly the discipline EyesInAI exists to provide:

• Per-tier quality monitoring — every quality signal split by routing tier, never read as an aggregate. The aggregate number is structurally blind to tier-specific drift.
• Long-tail oversampling — deliberately measure the queries the classifier was least confident about, because the failure is invisible in the average.
• Routing-confidence drift— track the classifier's confidence distribution against training; it shifts weeks before the quality signal does, which is the lead time to course-correct.

This is precisely why we argue a router that just moves the requestisn't enough — it has to be wired to a measured quality signal. (See prompt routing and how we test.)

Rupareliya's recommended pattern is an uncertainty- routed cascade: instead of a classifier deciding about a query before any model sees it, every query starts at the cheap model, which returns a calibrated confidence; low-confidence answers escalate to the capable model. The cheap model decides for itself rather than being decided about — so the hard queries it would have answered wrongly-and- confidently surface as low-confidence and escalate instead. He pairs it with shadow scoring (run the capable model on a small % of traffic in parallel to catch drift in real conditions) and quality-weighted routing (feed observed satisfaction back into the threshold over time).

The honest tradeoffs are real and we'll name them too: a cascade adds latency on escalated queries (cheap + capable in series), cost is harder to predict in advance (it depends on the confidence distribution), and calibrating the cheap model's confidence is non-trivial. Those are tradeoffs against a quality floor the pre-routing approach simply doesn't maintain — and in any deployment where the long tail carries real customer cost, that floor is the point. The cheap optimization quietly breaks the product; the honest one survives the quarter.