We measure what AI models actually do, not what vendors claim.

Every AI provider publishes benchmark scores. Almost all of them show their own model winning. That's because the model and the benchmark are designed together — the benchmark optimizes for what the model is good at, not what developers actually need to build with.

Developers picking a model for a real workload don't care about MMLU. They care about whether the model writes valid JSON when asked, whether it runs the function tool with the right arguments, whether the cost-per-task pencils out at scale, and whether the latency stays under their SLO during a traffic spike.

EyesInAI runs 18deterministic tests — tests where the answer is either correct or not, with no human or LLM judging the output — across every major model nightly. Then we publish everything: the cost of a passing run, the regression history, what's on the cost-vs-quality Pareto frontier, what's newly broken.

We run benchmark calls against each provider's official API — Anthropic, OpenAI, Groq, Google (Gemini), OpenRouter, NVIDIA — using our own API keys and a Mac Mini test rig. The rig sits behind a Cloudflare tunnel; the public site reads from it via a hardened proxy.

Each night a rotation cron divides the full model catalog into 7 groups (one per weekday) and benchmarks 1/7th of it, so every model gets fresh data weekly. Costs are computed per-call from the provider's current pricing. Results land in Supabase and become visible on the site within minutes.

The AI Analysis tab is a different beast: each morning Claude Haiku summarizes the previous day's data into structured insights (Pareto frontier, task routings, regressions, surprising findings). The narrative wraps around facts we computed in Python — we never let the LLM make up numbers.