We are testing Qwen on a Mac mini

We run Qwen3.6-35B-A3B (Q4_K_XL, served by llama-server) on the Mac mini that already runs our benchmarks. The question is not how much it saves. It is what the model can actually do, first on short tests and then on the real inputs of jobs our system runs today. Every run happens on the Mac mini, costs no API fees, and changes nothing in production. We measure thinking on and thinking off separately. The same model also runs our nightly local-vs-cloud test.

Results so far, last updated November 3, 2026. One run per setting unless a row says otherwise.

So far

• Calling a tool, on 4 easy one-call tests run 5 times each: 20/20 with thinking off and 19/20 with thinking on. Harder, multi-step tool use is not tested yet.
• Summarising 12 real batches of alerts, ranked blind by one judge: both Qwen settings had a worse mean rank than both cloud models (3.25 and 3.33, against 1.08 and 2.33). Some of its errors called warnings normal.
• Triaging our review queue (197 real items in 17 batches): with thinking on, Qwen agreed with the verdicts Sonnet gave and the job applied on 168/197 (85%). Sonnet is the reference, not the truth, but our hand check mostly sided with it, so the job stays with Sonnet.

1. Calling a tool

The tool tests from our Deep Suite: 4 short tests where the model must call the right tool with the right arguments, 5 runs each. These are test prompts, not inputs from a real job.

• With thinking on, there was one miss; it ran out of output tokens while still thinking.
• These are easy, one-call tests. Tool use over several steps, or inside a real job, is not measured yet.
• The frontier row is an approximate figure from a smaller grid (3 runs per test), so the rows are not a like-for-like race.

2. Summarising alerts

12 real batches of our own Telegram alerts, summarised by each model and ranked blind by one judge (Claude Opus). Each setting ran once per batch.

• Qwen with thinking off had the lowest median time of the four (2.0 s), and Qwen with thinking on the highest (29.8 s). Turning thinking on did not improve Qwen's mean rank (3.25 off, 3.33 on).
• Its errors were the kind that matter in alerts: it called warning heartbeats "normal" and "stable", and it said a session's "context was nearly full" when nothing said so.
• One judge, from the same model family as Claude Haiku, so it may favour Haiku. 12 batches is a small sample.

3. Triaging a review queue

Every logged batch of our daily article and email review (17 batches, 197 real items, 2026-10-18 to 2026-11-03), replayed on Qwen. We compared its verdicts with the ones Claude Sonnet gave and the job applied. Qwen's verdicts were never applied.

• With thinking off, 2 of 17 batches came back in a form the job refuses (one looped on its tools, one dropped an item). Its 117/191 (61%) still counts the items it returned in those batches; we have not scored the usable batches on their own.
• We checked the 29 thinking-on disagreements by hand against the job's written rules: Sonnet was right in about 20, Qwen in 1, and about 8 were arguable.
• Sonnet's verdicts are the reference, not the truth. Thinking on is slow here: a median of 98 s a batch.
• Verdict: this job stays with Sonnet.

4. What a labelling test taught us about our labels

Our chatbot labels each question with a type before it answers. We replayed the 100 most recent logged questions (2026-10-27 to 2026-11-02) through the live labelling prompt on Qwen, and compared each label with the one Claude Haiku 5.5 gave at the time. Our hourly shadow job had already put the same 100 logged questions to paid models.

• This is not a score for Qwen. Claude Haiku 4.5 and gpt-oss-120b also matched the live label only 36–39% of the time. Haiku 5.5, asked again, matched its own earlier label 82% of the time.
• The questions repeat: all 568 logged questions hold only 30 different ones, and 407 of them are "what is the highest rated model for X?". The 100 sampled rows hold 24 different questions; of the 24 asked more than once, live Haiku gave 7 more than one label.
• Qwen returned a valid label in 100% of its calls, and its two thinking-off runs gave the same label 87% of the time.
• What we take from it: the label set and the traffic need fixing before any model can be fairly scored on this job.

The same model on short graded tests every night, next to paid cloud models: local model vs the cloud.