AI Benchmark Intelligence

EyesInAI·Loading leaderboard…

Leaderboard

Live benchmark results across all tested AI models. Sortable by cost, latency, throughput.

Models tested

201

across 6 providers

Tasks completed

1033

of 1352 attempts

Avg cost / 1k runs

$0.561

across passing tasks

Cheapest passing

<$0.01

ling-2.6-flash

Leaderboard Charts Compare AI Analysis

Today's HeadlineJun 2, 09:00 PM

Gemini 2.0 Flash retired; Claude Haiku is the new cost-efficiency leader at $0.17/1M with 100% core test pass rate.

Claude Haiku (anthropic) is the cheapest passing model at $0.17 per 1M tokens with 83% pass rate. Gemini 2.0 Flash has been retired upstream (404 errors across all tests). No regressions detected in active models. OpenRouter's free tier models (Nemotron 3-nano, GPT-OSS) offer strong cost-efficiency but with lower pass rates; Claude Sonnet 4.6 achieves 100% pass rate at $2.50/1M tokens.

Cost-vs-Quality Frontier

3 optimal models

openrouter

The Pareto frontier spans three models: nvidia/nemotron-3-nano-30b-a3b (67% pass, free), anthropic/claude-3-haiku (83% pass, $0.17/1M), and anthropic/claude-sonnet-4.6 (100% pass, $2.50/1M). Free NVIDIA models offer zero-cost baseline but lower accuracy; Haiku balances cost and reliability; Sonnet dominates for mission-critical deployments.

Per-Task Recommendations

For ping/preprocessing, route to groq's openai/gpt-oss-20b (263ms, free). For speed (throughput 500-token generations), use same model (618.8 tps). For json/code/reasoning (structured output), use anthropic/claude-haiku-4-5-20251001 or openrouter/claude-3-haiku (both ~1s latency, 100% pass). For context retrieval (3-fact recall from 20 options), anthropic claude-haiku passes 100%; for tool_use (get_weather calls), claude-haiku also passes 100%. High-value recommendation: anthropic/claude-sonnet-4.6 for any mission-critical task (100% pass, <2s latency, $2.50/1M cost).

Speed

cheapest free

openroutergpt-oss-20b

fastest 385ms

openrouternemotron-3-nano-30b-a3b

throughput 1299 tok/s

openrouternemotron-3-nano-30b-a3b

Ping

cheapest free

openroutergpt-oss-20b

Surprising Findings

allenai/olmo-3-32b-think exhibits the lowest latency across all benchmarks (avg 166ms) but fails every single test (0% pass rate on all 6 core tests). Despite sub-200ms response times, the model is non-functional for all measured tasks — likely a downstream API issue or fundamentally broken model variant.

openrouterolmo-3-32b-think

lowest median latency across all tests at 166ms

Migration Paths

retired or failing models — consider switching

fromopenroutergemini-flash-latesttry

openrouter

claude-sonnet-4.6

100% · $2.50

openrouter

claude-opus-4.6-fast

100% · $23.43

openrouter

claude-3-haiku

83% · $0.172

fromopenrouterglm-5try

openrouter

Category Winners

Ping

groq263ms

gpt-oss-20b

Reasoning

groq315ms

gpt-oss-20b

JSON

anthropic100%

claude-haiku-4-5-20251001

Code

anthropic2061ms

claude-haiku-4-5-20251001

Speed

groq

Provider Health

anthropic

Pass rate100%

Avg latency1488ms

groq

Pass rate57%

Avg latency510ms

gemini

Pass rate0%

Avg latency344ms

Notable Changes

·gemini-2.0-flash (gemini provider) retired upstream — all 7 tests return 404 NOT_FOUND; users must migrate immediately
·groq free-tier openai/gpt-oss-20b passes ping and speed tests but fails reasoning, json, code, and context; 57% overall pass rate despite zero cost
·anthropic claude-haiku-4-5-20251001 achieves 100% pass rate (13/13 tests) including all advanced tasks (extract, instruction_follow, long_context_needle); lowest-cost premium option
·openrouter openai/gpt-oss-20b fastest on ping (263ms) and speed (618.8 tps) but fails 4 of 7 core tests; recommend for latency-sensitive pipelines only
·allenai/olmo-3-32b-think shows 0% pass rate across all 6 tests with suspiciously fast latencies (106–124ms); likely model unavailable or fundamentally broken

Recommendation

Migrate gemini-2.0-flash users to anthropic/claude-sonnet-4.6 (100% pass rate, $2.50/1M) or to anthropic/claude-3-haiku via OpenRouter ($0.17/1M, 83% pass rate). For latency-critical workloads, use groq's openai/gpt-oss-20b (263ms ping, free) only for ping and speed; route structured output tasks (json, code, reasoning) to Claude Haiku.