Loading benchmark data…
EyesInAI·Loading live benchmark data
The Claude 5.5 models think before they answer unless told not to. Every route we run sends each model its lowest-thinking setting instead, because thinking costs tokens and time. This run checks that choice: the same requests from the Claude tier bake-off, answered at the lowest setting and at the model's own default, graded the same way.
Run October 14, 2026 · 540 graded answers · judge Claude Opus 5.5 · total spend $6.46
| Model | Lowest thinking (what we send) | Model default |
|---|---|---|
| Claude Haiku 5.5 | thinking disabled | adaptive thinking, effort medium (default) |
| Claude Sonnet 5.5 | thinking between_tools | adaptive thinking, effort high (default) |
| Claude Opus 5.5 | thinking adaptive, effort low | adaptive thinking, effort medium (default) |
Compared only on answers graded at both settings (2 runs of each problem): passed out of those pairs, with their cost and latency. A pass is a judge score of 4 or 5 out of 5 against a written expected answer.
Passed · Answer cost per 1,000 · Median latency
Passed out of 10–20 per cell, lowest setting first, then the default.
| Request type | Haiku 5.5 | Sonnet 5.5 | Opus 5.5 |
|---|---|---|---|
| Order status | 8/10 · 9/10 | 10/10 · 10/10 | 10/10 · 10/10 |
| Refund request | 10/10 · 10/10 | 10/10 · 10/10 | 10/10 · 10/10 |
| Product FAQ | 10/10 · 10/10 | 10/10 · 10/10 | 10/10 · 10/10 |
| Quick question | 10/10 · 10/10 | 10/10 · 10/10 | 10/10 · 10/10 |
| Writing | 10/10 · 9/10 | 10/10 · 10/10 | 9/10 · 10/10 |
| Summarize a ticket | 10/10 · 9/10 | 10/10 · 10/10 | 10/10 · 10/10 |
| Code | 10/10 · 10/10 | 10/10 · 10/10 | 10/10 · 10/10 |
| Deep analysis | 17/20 · 19/20 | 20/20 · 20/20 | 20/20 · 20/20 |
One fictional test company and its support requests, so this proves the method, not a client's bar. Only 10–20 answers per cell: a difference of one or two passes may be chance. The problems are short and well defined; a task that needs a long chain of reasoning could gain more from thinking than these did.
45 hand-written problems across eight request types, 2 runs each, all three models at both settings in one run. Same prompt and answer budget; the default setting got extra room for thinking tokens. Claude Opus 5.5 graded the answers against an expected answer. Costs are each answer's input and output tokens (thinking included) at list price, judge excluded. 0 answers went ungraded. Run id thinking_20261014T123152Z.
Which model each request type needs: Claude tier bake-off. Every run we publish: all bake-offs.