How to Evaluate AI Models for Your Use Case When Every Model Wins the Benchmark
Public leaderboard scores have converged into noise. The only signal left worth reading is cost per correct task on a test set you keep private.
Markets price information, not effort. So when a run of frontier releases lands within days of each other and every one posts a near-identical top score on the same public tests, the market has just been handed a number that no longer separates anything. That is the awkward position procurement teams now sit in, and it is why how to evaluate AI models for your use case has quietly become a purchasing problem rather than a leaderboard-reading one.
The leaderboard was genuinely useful once, when models were far apart and a public test could tell a strong one from a weak one at a glance. That window is closing. Once a benchmark becomes the thing vendors are graded on, it stops measuring capability and starts measuring how hard everyone optimised toward the same fixed target. The scores rise, the gaps between them shrink, and the ranking turns into a marketing artefact. A one-point lead at the top of a saturated board is noise wearing a rosette.
Why have published AI benchmarks stopped telling buyers anything?
Two reasons, and neither requires assuming bad faith. First, a public test set is a finite, static target, and anything static gets learned. Whether through direct contamination or the slower drift of the whole field tuning against the same questions, published tests lose their power to discriminate the longer they stay published. Treat the claim that labs deliberately train on test data as an argument to keep in mind, not a proven fact. You do not need the conspiracy for the maths to bite. Second, convergence itself is the tell. When five models finish within a point of each other, the honest reading is not that they are equally good at your work. It is that the test has run out of headroom.
Look at how much thinner the signal gets on close inspection. When independent testers ran the ARC-AGI puzzle set, a heavily-hyped preview model scored around 75.7% on the semi-private split, an impressive figure that a headline can quote and a buyer can do very little with. Accuracy alone tells you a model can reach an answer. It says nothing about what reaching that answer cost, and cost is where the real spread between today's models hides.
That is the metric vendors cannot game away: cost per correct task at a fixed capability level. It is economically real. A model that matches a rival's accuracy at a fraction of the spend is the meaningful result, not a decimal-place lead on a saturated board. Whether you can afford to run a model a million times is the only question that survives contact with deployment. Two models can post the same reasoning score while their cost per solved task differs by an order of magnitude, and that gap is invisible on the leaderboard that gets the press.
What does cost per correct task actually measure?
Define it precisely, because the phrase does more work than it looks. Cost per correct task is the total spend to run your task set divided by the number of tasks the model actually got right: in the single-attempt case, the per-task token cost divided by the pass rate. Nothing exotic. But it only carries weight under three constraints. You need a grader you trust, an automated check or a human, that can call each answer right or wrong on your held-out set. You need to measure token usage on your own prompts rather than a vendor's tidy assumption, because verbose reasoning traces move the number more than the headline rate does. And you have to decide what a wrong answer costs: where a plausible-but-wrong output is expensive to catch, cost per correct task flatters the cheap model, so weight failures accordingly.
Put real numbers through it. Take two widely deployed models at their published list prices: a budget model at $0.15 per million input tokens and $0.60 per million output, and a premium model from the same vendor at $2.50 and $10 for the same volumes. Say each task in your set runs to roughly 2,000 input tokens and 500 output tokens. The budget model costs about $0.0006 per attempt; the premium one about $0.01, some sixteen times more. Now bring in accuracy, the half the price list ignores. Suppose that on your held-out set the budget model solves 55% of tasks and the premium model 75%. Divide cost by pass rate and the real figure appears: roughly $0.0011 per correct answer for the budget model against about $0.013 for the premium one.
Read that back slowly. The premium model is twenty accuracy points ahead and still costs about twelve times more per problem it actually solves. To buy 5,500 correct answers the budget model charges you around $6; the premium one charges roughly $73 for the identical 5,500. Widen the field and the spread only sharpens: published list prices run from budget tiers near $0.15 and $0.60 per million tokens, through mid options around $0.80 and $4, up to frontier tiers at $3 and $15 and higher for the largest reasoning modes, a range of two orders of magnitude sitting under benchmark scores separated by a point or two. The leaderboard shows you the accuracy gap and none of the price one. Whether the trade is worth it is a real decision, not a default: if a wrong answer is cheap to detect and retry, the budget model wins outright; where a silent error is expensive, you pay for accuracy and you are right to.
How do you evaluate AI models for your use case?
You build a private harness. Assemble a held-out set of tasks drawn from your own work, ones no vendor has seen and none can tune toward, and score every candidate model on cost-adjusted accuracy over that set. Keep it private and refreshed, and let it be the number you actually buy on. A public score answers a question you were never asking; your own eval answers the one you were. This is the same discipline we argue for in getting AI-ready before you build, and it is why model selection belongs inside your technical strategy rather than downstream of a press release. The harness is cheap next to the cost of standardising an entire company on the wrong vendor because a marketing number looked like a capability signal.
The second-order risks are the ones a leaderboard will never surface. Spatial and physical reasoning remain a structural weakness: in our own testing, models that ace a general reasoning board still stumble when a task turns on depth, relative position or physical layout inferred from an image. If your deployment reads depth or position from a single camera or purely visual input, a general reasoning score is worse than useless, because it will read high while the specific ability you depend on reads low. That is exactly the kind of failure mode a private eval catches and a public one launders, and it is a large part of why we keep a human in the loop on anything consequential.
What would change this position? A public benchmark that held its discriminating power a year after release, with scores that stayed spread rather than bunching at the ceiling, would be strong evidence that leaderboards still carry signal. So would a market where cost per correct task tracked headline accuracy closely enough that the cheap proxy stopped lying. Neither looks likely while the incentive is to optimise the visible number. Until it changes, the asymmetry nobody has priced in is this: the buyers quietly running their own held-out evals are making better decisions than the ones reading the same leaderboard as their competitors, and paying a fraction of the price to do it.
Questions people ask
What should an AI model evaluation framework actually measure?
Cost-adjusted accuracy on tasks that mirror your real workload, held out and kept private, rather than raw scores on public tests. Measure whether a model reaches the right answer and what each right answer costs, because the second number is where models that look identical on a leaderboard pull apart.
Are public AI benchmarks completely useless for choosing a model?
No, but their value decays fast. A fresh benchmark on genuinely novel tasks can be informative. A saturated one where frontier models finish within a point of each other cannot separate them for you, so treat leaderboard rank as a weak prior and let your own held-out evaluation carry the decision.
Why does spatial reasoning matter when comparing AI models?
Because it is a documented structural weakness that general reasoning scores hide. If your use case infers depth, position or physical layout from visual input, a high overall benchmark can mask a low ability at the exact skill you need, which is why a task-specific private eval beats any published aggregate.
Related
- AI Deskilling Is the Workforce Risk Your Board Isn't Pricing
- AI Therapy's Business Case Is a Floor, Not a Ceiling
- The Memory Oligopoly Behind AI's Cost Inflation
- AI & Automation
Written by an AI editorial persona of Abyshire's proprietary editorial system and reviewed by our team.