SWE-bench asks whether a patch makes real tests pass. LiveCodeBench asks whether a model can solve a problem published after it was trained. They measure different things, and neither measures the work you are hiring a model to do.
| Benchmark | What it measures | What it cannot tell you | Status |
|---|---|---|---|
| SWE-bench Verified | Resolving real GitHub issues, graded by the project's tests | Whether an agent can work across ten turns | Saturated; Vals archived it on 1 September 2026 |
| SWE-bench Pro | The same, on harder and newer issues | Anything about cost or speed | Opus 5.5 reported at 89.9% |
| LiveCodeBench | Competitive problems, tagged by release date | Repository work, tools, or editing | Running, contamination-resistant |
| Terminal-Bench 4.0 | Multi-step command-line tasks inside a harness | Which harness you would use | Running, ±2.6 points of standard error |
SWE-bench measures the patch
One issue, one patch, one test run. It is a clean measurement of a narrow thing: given a bug report and a repository, does the change make the failing test pass without breaking the others.
The tests are the grader, which is why the benchmark lasted as long as it did. It is also why it saturated: the tasks are public, the tests are public, and the training data caught up.
LiveCodeBench measures the unseen
Problems are tagged with the date they were published, so a model can be scored on problems released after it was trained. That makes contamination visible instead of assumed.
What it does not do is test a repository. A competitive problem is a self-contained function with a stated input and output. Most real work is a change to a codebase you have to read first.
Terminal-Bench measures the loop
Terminal-Bench runs an agent in a shell for many steps. That is closer to real work, and it is also why the score depends on the harness, the effort setting and the number of attempts. Anthropic's footnote lists all three.
Pick the one that matches your task
If you write single functions, LiveCodeBench is the closest. If you fix bugs in a repository, SWE-bench Pro. If you want an agent to run commands, Terminal-Bench — and then check which harness produced the number.