Claude Haiku 5.5 is the cheapest current Claude model, but its price per token rises 5× once a prompt passes 100K tokens, and a coding agent that reads a real repository passes that line easily. So we built two prompts from our own code, one under the line and one over it, hid questions with exact answers at different depths, and asked all three 5.5 models.
Run October 14, 2026 · 30 calls · total spend $2.97
4 lookups (find one value) and 1 two-step question (find two values and add them), graded by the number in the reply. The first call writes the prompt cache; the rest read it.
| Model | Lookups right | Two-step | First call | Average cached call | Median latency |
|---|---|---|---|---|---|
| Claude Haiku 5.5 | 4/4 | cut off | $0.01 | $0.0008 | 2.1s |
| Claude Sonnet 5.5 | 4/4 | right | $0.20 | $0.02 | 1.2s |
| Claude Opus 5.5 | 4/4 | right | $0.40 | $0.03 | 1.9s |
| Model | Lookups right | Two-step | First call | Average cached call | Median latency |
|---|---|---|---|---|---|
| Claude Haiku 5.5 | 4/4 | wrong | $0.13 | $0.01 | 2.9s |
| Claude Sonnet 5.5 | 4/4 | right | $0.50 | $0.04 | 1.9s |
| Claude Opus 5.5 | 4/4 | right | $1.00 | $0.08 | 2.4s |
Five questions per size and model, one run: one wrong answer is visible, but a rate is not. The questions are simple by design; a long agentic task asks far more of long context than a lookup does. Costs are computed from the returned token counts at list price, not read from a bill, and the 5× Haiku rate is the published price.
The prompt is real Python source from our own codebase, one header per file, sized with the token-counting API. The questions name values that appear exactly once. Each model ran at the lowest-thinking setting every route of ours sends (claude-haiku-5-5: thinking disabled; claude-sonnet-5-5: thinking between_tools; claude-opus-5-5: thinking adaptive, effort low). Grading checked that the expected number appears in the reply; a reply that also names other numbers could pass, and we have since made it stricter for future runs. 0 calls failed. Run id context_20261014T123448Z.
Thinking at its lowest setting vs the model default: thinking bake-off. Every run we publish: all bake-offs.