Construction quantity takeoff — fill a 27-line Mexican budget from DXF drawings and a soil study, leaving a cell blank rather than inventing a figure. Most benchmarks reward a model for producing an answer. This one doesn’t: a blank cell is frequently correct, because the drawings genuinely don’t support a figure. What we measure is whether the numbers a model does give can be trusted.
Results generated 2026-07-26 from the run log. This page reads the same JSON the harness writes, so it cannot drift from what was actually measured.
a quantity with NO cited source — a fabricated number
a REAL, correctly-measured figure applied to the WRONG EXTENT (e.g. the site boundary used for a zone-limited concept). More dangerous than an invented value: the citation is genuine, so every other check passes.
the engine's own validator refused the answer (e.g. amount != quantity x unit_price). DISQUALIFYING and invisible in every other field: such a run can show the highest `filled` with invented=0 and scope_violations=0.
any invented value, scope violation, or validator rejection disqualifies, regardless of cost or fill count
Do not rank by `filled`. Disqualify first on invented / scope_violations / validator_rejected, and only then compare cost and latency among survivors. A candidate with filled=0 has not passed — it abstained, which is untested rather than safe. Agreement figures are only as strong as the reference run: check the matching entry in `reference_runs` for how many lines the reference itself filled before trusting a high agreement score.
We stayed on our existing stack. Of the candidates, one was rejected by the validator, one committed a real scope violation, and one abstained entirely. Gemini 3.6 Flash matched the reference exactly at roughly a tenth of the cost and is the one we are watching — but nothing here certifies it. It agreed with a reference that could only fill 1 of 27 lines, and it produced fewer findings than that reference did. No clean multi-filled reference exists yet, so this run cannot show that any model fills correctly when the drawings do support a figure.
| Model | Filled | Invented | Scope | Rejected | Findings | Out | Latency |
|---|---|---|---|---|---|---|---|
gemini-3.6-flash gemini | 1/27 | 0 | 0 | — | 6 | 4.2k | 46.7s |
gpt-5.4-2026-03-05 openai | 5/27 | 0 | 0 | yes | 27 | 5.6k | 41.0s answer refused |
llama-3.3-70b-versatile groq | 0/27 | 0 | 0 | — | 3 | 2.9k | 5.7s filled nothing |
kimi-for-coding kimi-cli · subscription | 4/27 | 0 | 3 | — | 13 | null | 110.7s |
Kept for the record. This run predates a fix to the scope auditor, which was flagging one concept that the rules genuinely permit to use the site boundary — so scope violations shown here may not be real. Read the current run above.
| Model | Filled | Invented | Scope | Rejected | Findings | Out | Latency |
|---|---|---|---|---|---|---|---|
gemini-3.6-flash gemini | 1/27 | 0 | 0 | — | 23 | 6.0k | 51.5s |
gpt-5.4-2026-03-05 openai | 4/27 | 0 | 3 | — | 19 | 4.5k | 31.0s |
llama-3.3-70b-versatile groq | 0/27 | 0 | 0 | — | 29 | 4.5k | 8.4s filled nothing |
Kept for the record. This run predates a fix to the scope auditor, which was flagging one concept that the rules genuinely permit to use the site boundary — so scope violations shown here may not be real. Read the current run above.
| Model | Filled | Invented | Scope | Rejected | Findings | Out | Latency |
|---|---|---|---|---|---|---|---|
kimi-for-coding kimi-cli · subscription | 4/27 | 0 | 3 | — | 12 | null | 101.5s |
The baseline the candidates are measured against — the same harness run on our own stack. Included because a bake-off without a control is just a list of numbers.
These reference runs predate the scope-auditor fix and have not been re-run. One shows three scope violations that may be the same false positive corrected in the current bake-off — we have not verified it either way, so it is left as measured. Note also the spread: most reference runs could defensibly fill only 1 of 27 lines, which is the ceiling every agreement figure on this page is measured against.
| Model | Filled | Invented | Scope | Rejected | Findings | Out | Latency |
|---|---|---|---|---|---|---|---|
claude-opus-5 agent-sdk · subscription | 1/27 | 0 | 0 | — | 32 | 10.4k | — |
claude-opus-5 agent-sdk · subscription | 1/27 | 0 | 0 | — | 12 | 8.1k | — |
claude-opus-5 agent-sdk · subscription | 1/27 | 0 | 0 | — | 35 | 13.4k | — |
claude-opus-5 agent-sdk · subscription | 1/27 | 0 | 0 | — | 15 | 10.6k | — |
claude-opus-5 agent-sdk · subscription | 5/27 | 0 | 3 | — | 14 | 11.4k | — |
claude-opus-5 agent-sdk · subscription | 1/27 | 0 | 0 | — | 15 | 8.3k | — |
Every figure on this page comes from one JSON file, served openly so you can check it or chart it yourself.
This is one of three. The judge bake-off asks whether a cheaper model can grade as reliably as a premier one, and the router comparison asks whether our routing beats OpenRouter's automatic one. See all three, with what each found.