A benchmark score tells you how a model did on that benchmark. It does not tell you how it will do on your work, and the labs have started saying so in their own release notes.
The margin is smaller than the noise
Anthropic reports a standard error of ±2.6 points for Claude Opus 5.5 on Terminal-Bench 4.0. The top three models on the Artificial Analysis index sit at 59, 58 and 57. Two points of separation, with error bars wider than the gap, is a tie with a ranking attached.
Saturation ends the sorting
Vals stopped running SWE-bench Verified on new releases and archived the leaderboard. Its reason, in one line: performance on the benchmark has saturated. When every frontier model passes, the test stops telling you which one to buy.
Contamination is permanent
A benchmark published in 2024 was in the training data by 2025. LiveCodeBench exists because of that: it tags each problem with its release date, so a model can be scored only on problems published after its training cutoff.
That is a fix for one benchmark, not for the problem. Any test you publish will eventually be in the next model's data.
The harness is part of the score
Terminal-Bench runs in an agent harness, and the leaderboard says which one. A model with a shell, a test runner and five attempts scores differently from the same model with one attempt and no tools. The harness is not a detail; it is half the measurement.
What to do instead
Take fifty real inputs from your own logs. Write the check that decides pass or fail — a schema, a unit test, an exact match. Run the two models you are choosing between. Count the passes.
That number will be smaller, uglier, and the only one that predicts your bill.