Measured

Model vs model

Each entry asks one question: can a different model do a job we already run, as well or better, for less? For each task, every model gets the same real inputs and the same prompt as the model in use, so the model is the only thing that changes. Newest test first.

How a test is scored

Judges from different providers grade every answer blind, with the model names hidden. An answer passes a grading when it scores at least the bar, is in the right format, and the judge found nothing invented. An input passes when more than half of its gradings pass. Fabrication flags are counted separately, because a cheap model that makes things up is not a saving.

Meta's Muse Spark Open on eyesinai's four AI tasks

Run October 16, 2026 · 14 inputs · 3 models · 56 gradings by gemini-3.1-pro-preview and gpt-5.6-terra · spent $0.32 of a $2.97 cap

The question

Meta released Muse Spark Open, an open-weight model, and Together AI offered it free for a few days. Could it replace the Claude models that run this site's four AI tasks today?

Best value among models tested on every input

meta-models/Muse-Spark-Open-816B-A42B-promo (Meta (via Together AI)) passed 11 of 14 inputs with 2 fabrication flags, at $1.99 per 1,000 calls.

Model	Provider	Passed	Gradings passed	Fabrication flags	Cost per 1k calls	Median time
meta-models/Muse-Spark-Open-816B-A42B-promo	Meta (via Together AI)	11/14	22/28	2	$1.99	2.8 s
claude-haiku-5-5in use	Anthropic	10/12	21/24	1	$0.25	2.3 s
claude-sonnet-5-5in use	Anthropic	1/2	3/4	0	$8.48	3.2 s

By task

Model	Public model summary	Article analysis (internal)	Editor chat (internal)	Routing config (internal)
meta-models/Muse-Spark-Open-816B-A42B-promo	6/6	1/3	2/3	2/2
claude-haiku-5-5	4/6in use	3/3in use	3/3in use	—
claude-sonnet-5-5	—	—	—	1/2in use

What we did with it

No change. Muse Spark Open scored better than Claude Haiku 5.5 on the public model summary, but most of Haiku's misses there were blank lines in a reply that asks for exactly three lines, and the site already drops blank lines when it shows the summary. Muse fell behind on article analysis, where it broke the requested shape on one article, and on editor chat. At its paid price it also costs more than Haiku on those three tasks. It did well on the routing config, but that task had only two inputs, and the plan for it is still to re-test once the gateway that carries non-Anthropic calls is live. The decision from this test is not to add Muse to any recurring job.

What this does not prove

A handful of inputs per task, one run each. Read this as a direction, not a ranking to two decimal places.
The model was free on Together AI during this test. Its costs here, and the spend figure above, use Together's paid list price for the same model, which is what it would cost after the promotion. Its answers went through Together's free promotional endpoint.
Muse Spark Open is a reasoning model. It ran with reasoning turned off, close to how this site runs the Claude models on these tasks. With reasoning on it might score differently, and it could cost more.
The analysis and chat inputs were rebuilt from the site's news on the day of the test, so those two tasks do not use the same inputs as the earlier test of these four tasks.
The summary answers were graded twice. The first grading used one set of rules for both summary formats, which led a judge to fault a plain-prose answer for lacking the three-line format. The table uses the second grading, with separate rules for each format. Between the two gradings the same summary answers also lost three invented-detail flags, two of them on Muse's answers, so single grades are noisy.
On one analysis article, which allowed outside context, both judges flagged Muse's answer for unsupported claims. One judge's stated reason was wrong: it called real, released models non-existent. That answer's low grades are in the table as recorded.
Two judges from OpenAI and Google graded every answer, so neither contender's company graded its own model. Both judges' grades count toward the totals, and they disagree on some answers.

Counting exactly half the gradings as a pass would change claude-haiku-5-5 from 10/12 to 11/12, claude-sonnet-5-5 from 1/2 to 2/2.

Judge consistency: each judge graded each answer once, so whether a judge repeats its own verdict was not measured. The judges agreed with each other 26 of 28 times in round 1.

Pass bar: a score of 4 out of 5. 2 judges graded the answers in up to 1 round; the gradings column shows how many gradings each model actually received. Cost is the measured list-price cost of each answer, averaged per call.

Chatbot fallback models, Claude 4.x vs 5.5

Run October 16, 2026 · 39 inputs · 4 models · 312 gradings by gemini-3.1-pro-preview and gpt-5.6-terra · spent $1.07 of a $8.00 cap

The question

The chatbots in this test answer with an open-weight model first and fall back to a Claude model when it fails or is skipped. Should those fallbacks move from Claude Haiku 4.5 and Claude Sonnet 4.6 to Claude Haiku 5.5 and Claude Sonnet 5.5? Each model answered the same real visitor questions, with the same bot instructions and the same retrieved site content.

Best value among models tested on every input

claude-sonnet-5-5 (Anthropic) passed 22 of 39 inputs with 6 fabrication flags, at $13.60 per 1,000 calls.

Model	Provider	Passed	Gradings passed	Fabrication flags	Cost per 1k calls	Median time
claude-sonnet-5-5	Anthropic	22/39	57/78	6	$13.60	4.0 s
claude-sonnet-4-6	Anthropic	17/39	46/78	10	$14.47	5.2 s
claude-haiku-5-5	Anthropic	16/39	43/78	11	$0.64	2.3 s
claude-haiku-4-5-20251001in use	Anthropic	10/39	34/78	22	$3.84	3.5 s

By task

Model	Client bot: education	This site's assistant	Demo bot (sample business data)	Client bot: workspace rental	Client bot: construction, in Spanish
claude-sonnet-5-5	5/7	0/8	6/8	6/8	5/8
claude-sonnet-4-6	3/7	1/8	4/8	6/8	3/8
claude-haiku-5-5	2/7	0/8	6/8	6/8	2/8
claude-haiku-4-5-20251001	3/7in use	2/8in use	2/8in use	2/8in use	1/8

What we did with it

On 2026-10-16 the chatbots' Claude fallbacks moved to the 5.5 generation: Claude Haiku 4.5 to Claude Haiku 5.5, and Claude Sonnet 4.6 to Claude Sonnet 5.5. Bots that had both kept both, in the same order. That decision was made on an earlier blind side-by-side comparison of these same answers by a single judge. This entry re-grades them under this page's rules. Across all the questions together, each 5.5 model passes more answers than the model it replaced, but not on every bot: Claude Haiku 5.5 passed fewer than Claude Haiku 4.5 on this site's assistant and on the education bot, and Claude Sonnet 5.5 passed fewer than Claude Sonnet 4.6 on this site's assistant. The per-bot table shows each. Claude Sonnet 5.5 also passes more answers than Claude Haiku 5.5, at a much higher price per answer. The construction bot uses Claude Sonnet 5 as its fallback, which was not tested here, so it did not change. The same move also covered two bots that were not in this test. One of them uses a Claude model as its main model, and that model was not changed, because it has no test of its own yet.

What this does not prove

These are fallback models: a Claude model answers only when the bot's open-weight model fails or is skipped. Here every model was made to answer every question, with no earlier conversation, so this is not a sample of the conversations that actually reach a fallback.
'In use' marks the model each bot used when the answers were recorded, before the move. This site's assistant and the demo bot also had Claude Sonnet 4.6 as a second fallback, and the table can mark only one model per bot.
The answers were recorded once, before a fix that keeps every bot prompt under the AI gateway's 32,000-character limit. Five prompts were longer, so the models saw those prompts cut while the judges saw them whole. On those questions a grade can reflect the cut rather than the model, in either direction.
This site's assistant is told to keep its summaries to one or two sentences, and every model regularly wrote more. The judges also marked down links and unsupported details there, so that bot fails most answers for all four models.
Both judges are from outside Anthropic, because every model tested is a Claude model. They graded once each and often disagree, as the agreement line shows. Fabrication flags are the judges' calls under a strict rule (any claim the provided content does not support), not checked by hand. Read this as a direction, not a ranking to two decimal places.
The construction bot has no model in use marked: its fallback is Claude Sonnet 5, which is not in this test. It was included for its long Spanish prompts.
Costs are each answer's cost as reported by our gateway at list price on the day. The spend shown is the re-grading only: recording the answers cost $1.27 more, and the earlier single-judge comparison about $0.52.

Counting exactly half the gradings as a pass would change claude-sonnet-5-5 from 22/39 to 35/39, claude-sonnet-4-6 from 17/39 to 29/39, claude-haiku-5-5 from 16/39 to 27/39, claude-haiku-4-5-20251001 from 10/39 to 24/39.

Judge consistency: each judge graded each answer once, so whether a judge repeats its own verdict was not measured. The judges agreed with each other 106 of 156 times in round 1.

eyesinai's own four AI tasks

Run October 14, 2026 · 14 inputs · 8 models · 364 gradings by claude-opus-5-5 and gemini-3.1-pro-preview · spent $2.25 of a $4.50 cap

The question

Four parts of this site call a model. Should they stay on the Claude models they used, move to newer Claude models, or move to OpenAI or Google?

Best value among models tested on every input

gpt-5.6-luna (OpenAI) passed 14 of 14 inputs with 0 fabrication flags, at $0.32 per 1,000 calls.

Model	Provider	Passed	Gradings passed	Fabrication flags	Cost per 1k calls	Median time
gpt-5.6-luna	OpenAI	14/14	54/56	0	$0.32	2.3 s
gpt-5.6-terra	OpenAI	14/14	55/56	1	$3.36	2.3 s
claude-haiku-5-5	Anthropic	12/14	48/56	0	$0.28	2.1 s
gemini-3.5-flash-lite	Google	10/14	41/56	4	$0.61	1.3 s
claude-sonnet-5-5	Anthropic	9/14	19/28	0	$6.48	3.4 s
gemini-3.8-flash	Google	9/14	41/56	7	$1.34	3.6 s
claude-haiku-4-5-20251001in use	Anthropic	5/12	24/48	9	$1.77	2.5 s
claude-sonnet-4-6in use	Anthropic	0/2	0/8	0	$13.52	9.5 s

By task

Model	Public model summary	Article analysis (internal)	Editor chat (internal)	Routing config (internal)
gpt-5.6-luna	6/6	3/3	3/3	2/2
gpt-5.6-terra	6/6	3/3	3/3	2/2
claude-haiku-5-5	5/6	2/3	3/3	2/2
gemini-3.5-flash-lite	6/6	3/3	1/3	0/2
claude-sonnet-5-5	2/6	3/3	2/3	2/2
gemini-3.8-flash	5/6	1/3	2/3	1/2
claude-haiku-4-5-20251001	2/6in use	0/3in use	3/3in use	—
claude-sonnet-4-6	—	—	—	0/2in use

What we did with it

The same day, the three Haiku 4.5 tasks moved to Claude Haiku 5.5 and the routing config moved to Claude Sonnet 5.5. That was an interim step that stays on Anthropic, because the gateway that will carry OpenAI and Google calls is not live yet. When it is, the plan is to re-run this test and then move the summary and analysis to gpt-5.6-luna and the routing config to gpt-5.6-terra. One more decision came from this run: the bot shadow check no longer falls back to a Gemini Flash model. gemini-3.8-flash, the obvious successor to the retired gemini-2.5-flash, is in the table above, and its pass count and fabrication flags rule it out.

What this does not prove

A handful of inputs per task. Read this as a direction, not a ranking to two decimal places.
The models in use were tested only on the tasks they run: Claude Haiku 4.5 on three tasks, Claude Sonnet 4.6 on the routing config.
Claude Sonnet 5.5's first answers failed on a harness setting (it needs a different thinking setting). It was re-run and graded in the second round only, so it has half as many gradings as the other candidates.
Costs are list prices. OpenAI rates were taken from our bench price table, not re-read from OpenAI's page. gemini-3.8-flash is on an introductory rate that doubles on 2027-01-01.
The summary answers were graded again on 2026-10-16. The first grading used one set of rules for both summary formats, which led a judge to fault plain-prose answers for lacking the three-line format, two grades per model. The table uses the second grading, with separate rules for each format, in both rounds and from both judges. The answers themselves were not re-run. The spend and cap above include this regrade and its own separate cap.

Counting exactly half the gradings as a pass would change claude-haiku-5-5 from 12/14 to 13/14, gemini-3.5-flash-lite from 10/14 to 11/14, claude-sonnet-5-5 from 9/14 to 10/14, gemini-3.8-flash from 9/14 to 12/14, claude-haiku-4-5-20251001 from 5/12 to 6/12.

Judge consistency: across rounds, a judge gave the same pass or fail to the same answer 157 times out of 168. The judges agreed with each other 71 of 84 times in round 1 and 87 of 98 times in round 2.

Pass bar: a score of 4 out of 5. 2 judges graded the answers in up to 2 separate rounds; the gradings column shows how many gradings each model actually received. Cost is the measured list-price cost of each answer, averaged per call.

All bake-offs

EyesInAI·Loading live benchmark data