Loading benchmark data…
EyesInAI·Loading live benchmark data
Every night our Mac mini runs Qwen3.6-35B-A3B on its own hardware and answers the same graded tests as 4 paid cloud models (Qwen3.6-35B-A3B (OpenRouter), gpt-oss-120b (Together), Haiku 5.5, Sonnet 5.5). Each answer is checked against explicit rules by code, not by another model. This page is rebuilt each morning from the latest night, so it is a running record, not a one-off. See also serving local models.
Latest run: November 2, 2026 (UTC) · 1 night published · cloud cost of the run $0.19, under a $5 nightly cap
Qwen3.6-35B-A3B (Q4 quantization) on a Mac mini with an Apple M5 Pro and 64 GB of memory. Loaded, the model holds about 22 GB. The median answer took 19.1 s, generating about 64 tokens a second (its own server’s clock).
13 tests, 3 attempts each. An error, or an attempt skipped after repeated errors, counts as a fail.
| Model | Runs on | Passed | Errors | Cost of the run | List price per 1M tokens (in / out) |
|---|---|---|---|---|---|
| Qwen3.6-35B-A3B (hub, Q4) | Our Mac mini | 35/39 (90%) | 3 | $0 API fees | — |
| Qwen3.6-35B-A3B (OpenRouter) | Cloud API | 39/39 (100%) | 0 | $0.06 | $0.1 / $0.9 |
| gpt-oss-120b (Together) | Cloud API | 39/39 (100%) | 0 | $0.01 | $0.15 / $0.6 |
| Haiku 5.5 | Cloud API | 38/39 (97%) | 0 | $0.0066 | $0.1 / $0.5 |
| Sonnet 5.5 | Cloud API | 38/39 (97%) | 0 | $0.11 | $2 / $10 |
Cloud costs are at list price. A call whose cost the host did not report is counted at its worst-case reserve, so these are upper bounds, not invoices.
Cloud response times are not shown. Our timer for cloud calls has been stopping when the response headers arrived instead of the full answer, which made one host look about 15× faster than it is, so those times are wrong. The Mac mini times above come from its own server and are not affected.
| Kind | Qwen3.6-35B-A3B (hub, Q4) | Qwen3.6-35B-A3B (OpenRouter) | gpt-oss-120b (Together) | Haiku 5.5 | Sonnet 5.5 |
|---|---|---|---|---|---|
| Reasoning under pressure (5) | 15/15 | 15/15 | 15/15 | 15/15 | 14/15 |
| Format fidelity (4) | 11/12 | 12/12 | 12/12 | 11/12 | 12/12 |
| Long context (4) | 9/12 | 12/12 | 12/12 | 12/12 | 12/12 |
| Test | Qwen3.6-35B-A3B (hub, Q4) | Qwen3.6-35B-A3B (OpenRouter) | gpt-oss-120b (Together) | Haiku 5.5 | Sonnet 5.5 |
|---|---|---|---|---|---|
| reasoning_date_arithmetic | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| reasoning_conditional_probability | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 |
| reasoning_mixed_units | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| reasoning_billing_trap | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| reasoning_knights_knaves | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| format_json_escaping | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| format_constrained_writing | 2/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| format_csv_rfc4180 | 3/3 | 3/3 | 3/3 | 2/3 | 3/3 |
| format_number_rounding | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| longctx_multi_needle | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| longctx_distractor_needles | 0/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| longctx_scattered_facts | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
| longctx_contradiction_override | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 |
Passed out of attempts, newest first. * marks a night that did not finish every planned attempt.
| Ran (UTC) | Qwen3.6-35B-A3B (hub, Q4) | Qwen3.6-35B-A3B (OpenRouter) | gpt-oss-120b (Together) | Haiku 5.5 | Sonnet 5.5 |
|---|---|---|---|---|---|
| November 2, 2026 | 35/39 | 39/39 | 39/39 | 38/39 | 38/39 |
A separate experiment: Qwen3.6-35B-A3B on the Mac mini with its reasoning turned on and off, 16 tests, 3 attempts each. Run November 2, 2026 (UTC).
| Setting | Passed | Median time | Ran out of tokens |
|---|---|---|---|
| Thinking on | 47/48 | 16.4 s | 1 |
| Thinking off | 35/48 | 0.2 s | 0 |
Turning thinking off lowered the pass rate from 98% to 73%; the median answer took 0.2 s instead of 16.4 s.
13 short tests, 3 attempts each: a few points either way can be chance. "$0 API fees" leaves out the price of the machine and the electricity it draws while it answers. The local model is a Q4 quantized copy; the cloud copy of the same model id may not be. Tool use and open-ended writing are not tested here.
13 tests from our Deep Suite that code can grade (reasoning, format and long-context recall), 3 attempts per model every night from 01:10 Mac mini time, every model with the same output budget and its host's default sampling, under a $5 nightly cap on cloud spend. A night counts if it ran the full suite, or is marked partial if it stopped early (for example at the cap); the figures are rebuilt the next morning.