Stanford authors say about 30% of Terminal-Bench 2.0 tasks had issues, and release 2.1
A paper submitted to arXiv on November 5, 2026 proposes continuous validation for AI benchmarks. The authors say they found issues in roughly 30% of Terminal-Bench 2.0 tasks.
What changed
The authors released repairs as Terminal-Bench 2.1, which they say raise accuracy by over 10 points and move leaderboard positions by up to three ranks. They report the ranking stays highly correlated, with a Spearman rho of 0.958.
Why it matters
Coding agent leaderboards are widely used, so flawed tasks and time budgets can change how models appear to compare.
What remains uncertain
This is a preprint and has not been peer reviewed. The findings come from the authors’ own analysis of their benchmark.
What to watch
Watch for leaderboards switching to Terminal-Bench 2.1 and other benchmarks adopting similar checks.
Source
Why this ranks here
Relevant to how agents are measured, but a single preprint.
- Impact5
- Reach5
- Novelty6
- Commitment5
- Coverage quality6
- Future potential6