Measured test

Haiku 4.5 vs Haiku 5.5 on our own small jobs

Our system runs several small, frequent jobs on Claude Haiku. Before moving them from Haiku 4.5 to Haiku 5.5 we ran both on the same frozen inputs with the same prompt and parser each job uses in production. The only variable is the model. Each job is scored against something other than 4.5’s own past answers, because those would favour 4.5 by construction.

Run October 9, 2026 · 3 runs · total cost $0.60 · thinking off for both

What it found

Interpreter: Haiku 5.5 picked the rule-correct model on 88.3% of calls and Haiku 4.5 on 45%. The same pick on both runs: 96.7% of cases for 5.5, 70% for 4.5.
On the three grading jobs (rubric, bot screen, answer pass/fail), using the latest rubric run, the two were within 1.7 points of each other. In no rubric run did either model let a failing response pass.
On rubric run 1, Haiku 5.5 was 8.3 points behind 4.5, with 6 of its calls erroring.
Exact 1–5 answer scores matching the reference: Haiku 4.5 75.9%, Haiku 5.5 62.1%. 5.5 was always within a point of the reference. On average 5.5 scored 0.17 points lower than the reference.
Intent labels: Haiku 5.5 matched the Sonnet 5 reference on 52% of calls and 4.5 on 41%. Against production’s logged labels it was 82% for 4.5 and 56% for 5.5, but 4.5 wrote those labels itself, so that row cannot pick a winner.
Rows where 4.5 came out ahead: Correct verdicts (run 1 of 2); Calls that errored (run 1 of 2); Same real verdict every time, per pair (run 1 of 2); Same real verdict every time, per pair (run 2 of 2); Exactly the reference score.

Each job, measure by measure

“n” is the number of inputs. Each input was run one to three times per model (see the method below), and percentages and counts are over those calls; “same on both runs” is per input. A call with no readable answer counts as wrong in the rubric and intent rows and is left out of the answer-scorer rows.

Interpreter

Picks which model should serve a request, under cost, latency and quality limits. Scored against the rule-correct pick and a Sonnet 5 reference.

Measure	Haiku 4.5	Haiku 5.5	n	Ahead
Picked the rule-correct model	45%	88.3%	30	Haiku 5.5
Same pick as the reference	51.7%	85%	30	Haiku 5.5
Same pick on both runs (per case)	70%	96.7%	30	Haiku 5.5
Picks that broke a stated limit	4	2	30	Haiku 5.5

Rubric grader

Grades a response pass or fail against a written rule. Scored against hand-written pairs with a known answer. Ran more than once; every run is shown.

Measure	Haiku 4.5	Haiku 5.5	n	Ahead
Correct verdicts(run 1 of 2)	95%	86.7%	20	Haiku 4.5
False passes(run 1 of 2)	0	0	20	tie
Calls that errored(run 1 of 2)	0	6	20	Haiku 4.5
Same real verdict every time, per pair(run 1 of 2)	100%	85%	20	Haiku 4.5
Correct verdicts(run 2 of 2)	95%	96.7%	20	Haiku 5.5
False passes(run 2 of 2)	0	0	20	tie
Calls that errored(run 2 of 2)	0	0	20	tie
Same real verdict every time, per pair(run 2 of 2)	100%	95%	20	Haiku 4.5

Answer scorer

Scores stored assistant answers 1–5. Scored against an Opus 5 reference; some answers had a value deliberately corrupted.

Measure	Haiku 4.5	Haiku 5.5	n	Ahead
Same pass/fail as the reference	96.6%	96.6%	29	tie
Within 1 point of the reference	96.6%	100%	29	Haiku 5.5
Exactly the reference score	75.9%	62.1%	29	Haiku 4.5
Average gap to the reference score (points)	0	-0.17	29	not a score
Corrupted answers caught	10 of 11	11 of 11	11	Haiku 5.5

Bot screen

Decides whether our site assistant answered or punted. Live answers checked against Opus 5, plus controls with a known label.

Measure	Haiku 4.5	Haiku 5.5	n	Ahead
Correct answered/punted calls	95.8%	95.8%	24	tie
Same call on both runs (per item)	100%	100%	24	tie

Intent labeller

Labels what a router request is asking for. Scored against a Sonnet 5 reference.

Measure	Haiku 4.5	Haiku 5.5	n	Ahead
Same label as the reference	41%	52%	50	Haiku 5.5
Same label as production’s log (written by 4.5)	82%	56%	50	not a score
Same label on both runs (per row)	84%	92%	50	Haiku 5.5

What this does not prove

Small samples throughout, and one run per job, apart from the rubric (2 runs). A few calls either way would move most rows.
Every rubric run is in the table. The grading finding above uses the latest, in which neither model errored.
The intent reference agreed with production’s logged labels on only 36% of rows. That row measures agreement with another model, not accuracy, and neither candidate matched the reference on more than 52% of calls.
The interpreter’s rule answer is mechanical, and the Sonnet 5 reference matched it on 86.7% of cases, so the rule itself may not be what a person would pick.
Rubric run 1: the run’s own log puts Haiku 5.5’s steadiness at 95%, because it counts a pair that errored every time as steady. The table counts only pairs that returned the same real verdict every time: 85%.
No call errored in the answer-scorer or intent jobs.
This page does not measure cost per model. The judge bake-off does, on a different job.

How this was measured

Each job used its production prompt and parser; inputs were frozen once and given to both models.
Rubric pairs were graded 3 times each; interpreter cases 2 times; bot-screen items and intent rows twice; stored answers once. The repeats are what “same on both runs” measures.
Each run had its own spending cap, checked after every priced call: $0.26 of $2.00, $0.02 of $0.10, $0.31 of $2.00. None reached its cap. The raw log keeps real requests and client answers, so only the totals are published here.

Haiku 5.5 is also in the judge bake-off, which measures its cost. See every bake-off, with what each found.

All bake-offs

EyesInAI·Loading live benchmark data