Loading benchmark data…
EyesInAI·Loading live benchmark data
Claude Code lets you pick the model and how much effort it spends. This run puts every combination of Haiku, Sonnet and Opus 5.5 with low, medium and high effort on the same 24 coding tasks from the coding-CLI bake-off, graded by hidden tests the agent never saw.
Data as of October 15, 2026 · 216 agent runs over 24 tasks · $43.75 at list price for the compared runs (run on a subscription)
Tasks passed out of the 24 that every combination finished, by model (rows) and effort (columns).
| low | medium | high | |
|---|---|---|---|
| Claude Haiku 5.5 | 20/24 | 21/24 | 20/24 |
| Claude Sonnet 5.5 | 21/24 | 19/24 | 21/24 |
| Claude Opus 5.5 | 20/24 | 21/24 | 23/24 |
| Model and effort | Passed | Easy | Real-world | Hard | False DONE | Timed out | Median time | Median thinking tokens | List-price cost |
|---|---|---|---|---|---|---|---|---|---|
| Opus 5.5 · high | 23/24 | 6/6 | 10/10 | 7/8 | 1 | 0 | 54s | 935 | $13.04 |
| Haiku 5.5 · medium | 21/24 | 6/6 | 8/10 | 7/8 | 3 | 0 | 48s | 1,827 | $0.93 |
| Sonnet 5.5 · low | 21/24 | 6/6 | 8/10 | 7/8 | 3 | 0 | 32s | 141 | $3.23 |
| Sonnet 5.5 · high | 21/24 | 6/6 | 8/10 | 7/8 | 3 | 0 | 42s | 359 | $4.74 |
| Opus 5.5 · medium | 21/24 | 6/6 | 8/10 | 7/8 | 3 | 0 | 54s | 480 | $9.81 |
| Haiku 5.5 · low | 20/24 | 6/6 | 7/10 | 7/8 | 4 | 0 | 32s | 1,128 | $0.57 |
| Haiku 5.5 · high | 20/24 | 6/6 | 7/10 | 7/8 | 4 | 0 | 70s | 6,560 | $1.85 |
| Opus 5.5 · low | 20/24 | 6/6 | 8/10 | 6/8 | 4 | 0 | 37s | 134 | $6.43 |
| Sonnet 5.5 · medium | 19/24 | 6/6 | 7/10 | 6/8 | 5 | 0 | 32s | 224 | $3.15 |
A pass means every hidden test passed. False DONE: the agent said it was done, but hidden tests failed. Runs were stopped at 25 minutes. – means the figure could not be read for every run.
passed failed, claimed DONE failed, said not done hit the 25-minute limit harness error◇ failed only a stricter check (see below)? grading record missing
| Task | Haiku low | Haiku medium | Haiku high | Sonnet low | Sonnet medium | Sonnet high | Opus low | Opus medium | Opus high |
|---|---|---|---|---|---|---|---|---|---|
| Easy pilot. Four public upstream bugs and two synthetic tasks. | |||||||||
| more-itertools | |||||||||
| humanize | |||||||||
| boltons | |||||||||
| click | |||||||||
| synthetic root-cause bug | |||||||||
| synthetic feature + CLI | |||||||||
| Rebuilt real-world. Standalone code we wrote to reproduce bugs found in past code reviews. Published as numbered items only. | |||||||||
| Real-world 1 | ◇ | ◇ | ◇ | ◇ | |||||
| Real-world 2 | |||||||||
| Real-world 3 | |||||||||
| Real-world 4 | |||||||||
| Real-world 5 | |||||||||
| Real-world 6 | ◇ | ◇ | ◇ | ||||||
| Real-world 7 | |||||||||
| Real-world 8 | |||||||||
| Real-world 9 | |||||||||
| Real-world 10 | |||||||||
| Hard public bugs. Real bugs from networkx, pyparsing, sqlglot, lark and flask. | |||||||||
| networkx: connectivity | |||||||||
| networkx: ISMAGS | |||||||||
| networkx: weak views | |||||||||
| sqlglot: BD literal | |||||||||
| pyparsing: match previous | |||||||||
| pyparsing: bounded repeat | |||||||||
| lark: interactive copy | |||||||||
| flask: IPv6 | |||||||||
Two hidden tests check more than the issue text asks for. We kept them, because a careful fix should pass them, and a run that fails one still counts as a fail. They are marked ◇ so you can discount them.
Other coding agents on the same tasks: coding-CLI bake-off. Thinking settings on support requests: thinking bake-off. Every run we publish: all bake-offs.