Baseten's Nemotron-3-Ultra is the only Pareto-optimal model here, with 21/21 passing at about $0.0004 per call.
Baseten's nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B is the only Pareto-frontier model, passing 100% of 21 tests at avg $0.00038 and 1553ms avg latency. No regressions were detected. The most surprising finding is openrouter's qwen/qwen3.7-plus:batch, with the lowest median latency at 129ms, though its recorded pass rates are 0% on all six tests. Several providers (deepinfra, fireworks, gemini) have no cached results, and gemini-2.5-flash-era models are only represented via openrouter history.
Cost-vs-Quality Frontier
1 optimal models
baseten
baseten nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B is the sole Pareto-frontier model, at 100% pass rate, $0.00038 average cost and 1553ms average latency across 21 tests. No other model in the cached dataset is cost-optimal at a comparable pass rate.
Per-Task Recommendations
For ping, reasoning, json, code, context and tool_use, route to baseten nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B, which has the lowest cost and passes every test in each category. For speed and long generation, the same model leads on throughput at 298 tokens/sec. For fallback traffic, the openrouter/nvidia/nemotron-3.5-lightning profile is not usable as a recommendation since its cost data is null.
openrouter qwen/qwen3.7-plus:batch posted the lowest median latency across all tests at 129ms, yet had a 0% pass rate on all six tests it ran, so the latency figure should not drive routing.
·baseten nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B: 21/21 tests passed at 1553ms avg, the only model on the Pareto frontier.
·openrouter qwen/qwen3.7-plus:batch: 0% pass rate on all six tests despite 129ms median latency, flagged as the fastest-overall surprising finding.
·nvidia: all cached models (meta/llama-4-maverick, meta/llama-3.3-70b, meta/llama-3.1-405b, deepseek-r1 distills) show 0% pass rate across all tests, consistent with a provider-level failure rather than model regressions.
·gemini: gemma-3-27b-it and gemma-3-12b-it show 0% pass rate on every test, a pattern to monitor rather than a confirmed retirement.
·openai o3-mini-2025-01-31 and gpt-4.1-nano-2025-04-14: 0% pass rate across all tests.
·minimax MiniMax-M3.1-Flash speed test failed validation with reasoning_only output, the only failing test in its 15-test run.
·openai gpt-6-luna speed test returned an empty response (validation_failed:empty_response) at 9052ms.
Recommendation
Route ping, reasoning, json, code, context, and tool_use traffic to baseten nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B, the only model passing all tests at a Pareto-optimal cost, and migrate openrouter google/gemini-2.5-flash, anthropic/claude-haiku-4.5 and nvidia meta/llama-4-maverick traffic to it.