Our leaderboard called gpt-4.1 OpenAI's best model for ten weeks. A seven-word list was why.

Our bench discovers models rather than hardcoding them: it asks each provider's API what exists, then filters out the things that aren't chat models — embeddings, audio, moderation, image endpoints. For most providers that filter is a skip list: drop what matches, keep the rest. For OpenAI it was a keep list:

OPENAI_KEEP_PREFIXES = ['gpt-4', 'gpt-3.5-turbo', 'o1', 'o3', 'o4', 'o3-mini', 'o1-pro']

A model survived only if its id started with one of those. gpt-5.6-sol starts with none of them. Neither does gpt-5.4, or gpt-5-mini, or any of the thirty-nine 5.x ids OpenAI was serving. They were fetched from the API, checked against the list, and silently discarded — every scheduled run, for 69 days, from the bench's first commit on 2026-05-24 until we found it on 2026-08-01.

The leaderboard was not lying about anything it measured. Every number on it was real. It was simply answering a question nobody asked: of the OpenAI models we happen to test, which is best? The answer was gpt-4.1, and it was correct, and it was useless.

This is the part worth stealing. We already run a cron whose entire job is to notice when a vendor ships a new model — it diffs the discovered model list run over run and sends a digest of launches, removals and price moves. GPT-5 launched. GPT-5.4 launched. The 5.6 family launched. It never said a word.

It couldn't. The watcher reads the list of models the bench discovered, and discovery filtered the 5.x family out before the watcher ever saw it. The detector was downstream of the defect. From where it sat, OpenAI had shipped nothing in ten weeks — which is indistinguishable from OpenAI genuinely shipping nothing.

A monitor that reads filtered data inherits the filter's blind spots. Ours was built to answer “did anything new appear?” and it answered honestly about a list that had already been censored. When a detector goes quiet, the first question is not “is the world quiet?” but “what feeds this thing, and can it still see?”

The structural lesson is about which way a filter fails. Our other providers use skip-lists — drop what matches embed, whisper, tts, keep everything else. When a vendor ships something new, a skip-list lets it through. It might arrive unpriced or mislabelled, and you will notice, because a strange row shows up on a page you look at.

A keep-list fails the other way. Something new arrives and is dropped, and the failure produces no row, no error, no log line — an absence, which looks exactly like the vendor not having shipped. Every other provider's filter failed open and loud. The one keep-list failed closed and silent, and that is the one that broke.

Neither is right in the abstract. A keep-list is the safer default when the cost of admitting something wrong is high, which is why allowlists are the correct shape for agent permissions. The mistake is using one where the cost of missing something is high and nobody is watching for silence. Discovery is exactly that.

Adding 'gpt-5' to the keep list took one line and immediately created a second problem. The prefix admits the whole family, including four superseded generations — 5.0 through 5.3, twenty-four ids — that we had no pricing for. A model with no price in our tables reports null, which renders as $0on a public cost-efficiency page. The fix would have replaced a wrong “best model” with two dozen free-looking frontier models.

So the change skips the generations we don't price, and there is one more trap inside that. The skip rules are substring matches, and bare gpt-5 is a substring of every model in the family — skipping it that way would have re-hidden everything we had just un-hidden. It needed a separate exact-match rule for that one id.

The rule we wrote down afterwards: price a generation before you un-skip it, and treat any hardcoded list of model prefixes as a thing that expires. Auto-discovery is only as current as its filter.

Ten weeks of a wrong public answer to “which OpenAI model is best?” — and, because the leaderboard is the input to our own routing recommendations, ten weeks in which no GPT-5 model could be recommended by us for anything. We have since priced and benched the 5.4, 5.5 and 5.6 families; they now run on every scheduled pass.

One honest limit on the scope: GPT-5 first appeared in OpenAI's API on 2025-08-01, nearly a year before we found this — but our bench did not exist until 2026-05-24, so the defect's life is 69 days, not a year. It is the smaller and more accurate number, and it is the one we quote.

We found this by hand while adding the 5.6 family, not because any check caught it — so the real fix was never the one-line filter change. It was the test that did not exist. That is now written: ten offline checks over the discovery filters, asserting that every model we bench survives its own provider's filter, that every benched model has a price, and that the context and tool-support maps know about each family. It is negative-tested — reintroducing the original one-word bug fails five of the ten, each naming the cause. A test that cannot fail on the bug it documents is decoration.

Writing it immediately surfaced a second instance of the same shape, in a different provider — a case that turned out to be deliberate, so the test now allows it explicitly rather than being loosened. And it has a limit worth naming: these checks are offline, so they cannot notice a generation nobody has told them about. A human still adds the new family to a list; the test then enforces the four other places it has to be wired. That is a smaller gap than the one we started with, not the absence of one.

We publish our corrections for the same reason we publish a router bake-off that flattered us less than the previous one. A leaderboard is an instrument, and an instrument nobody audits is a claim wearing a lab coat. This one was wrong in a way that took ten weeks and a hand-audit to notice — worth saying out loud, so the next person to build one checks their filter on day one.