Terminal-Bench 4.0 Leaderboard
The definitive test of autonomous agent capability in real terminal environments. 66 professional computer-work tasks across software engineering, machine learning, systems, and operations evaluated over 5 trials per task with an 8-hour agent timeout.
| Model & Provider | Effort | Resolution Rate | Agent Harness | Eval Run Cost | Tokens | API Price (Out) | Source / Verify |
|---|
What Changed in Terminal-Bench 4.0?
Terminal-Bench (curated by the Laude Institute, Stanford researchers, and open-source contributors) evaluates whether autonomous agents can complete complex terminal tasks end-to-end. Unlike SWE-bench, which isolates git patches for individual Python issues, Terminal-Bench tasks require operating shells, installing toolchains, compiling multi-language dependencies, executing benchmarks, and resolving system faults.
Reading the table: The public board moved on two fronts since our last check. Anthropic's Opus 5.5 and Sonnet 5.5 — previously reported only through Anthropic's own runs — are now public rows at 64.8% ±3.1 and 61.8% ±2.9, both at max effort, which puts Claude models at #1 and #2 for the first time. OpenAI's GPT-6.1 Sol landed at 58.2% ±3.1, a statistical tie with GPT-6 Astra at 58.2% ±2.8, for a published run cost of $0.6k against Astra's $3.3k. Provider and third-party runs still diverge sharply from the board: Anthropic reports 70.6% and 66.4% for the two Claude models, Artificial Analysis measures 63.6% and 59.6%, and Vals AI reports 64.1% and 65.2%. Only public-board rows are like-for-like.
1. Task Set Recalibration (66 Tasks)
Version 4.0 removes 8 tasks that became saturated, refusal-prone, publicly solved, or flawed, and fixes 19 more with revised environments and tighter verifiers. Those breaking changes required fresh trials, so 4.0 scores are not comparable with 3.0 or 2.x.
2. 8-Hour Compute Budget & Multi-Trial Rigor
The public protocol uses 5 independent trials per task with an 8-hour agent timeout, and each lab brings its own agent harness. Artificial Analysis instead runs 3 repeats per task inside mini-SWE-agent, and Vals AI runs its own harness — which is exactly why the same model can score 70.6% in one table and 63.6% in another.
3. The Efficiency Divide: Same Score, a Fifth of the Bill
GPT-6 Astra and GPT-6.1 Sol both land at 58.2% on 1.5B tokens, but the published run costs are $3.3k and $0.6k. Sonnet 5.5 buys 3.6 more points with 19.4B tokens and a $7.3k run, and Opus 5.5 tops the board at 64.8% for $4.7k across 8.0B tokens.
4. Open-Weight Breakthrough: GLM-5.3
Z.AI's GLM-5.3 is still the best open-weight row on the public board at 41.8% ±3.2 inside Claude Code — ahead of GPT-5.6 Sol (37.3%) and Opus 4.8 (23.6%) — at $1.40 per 1M input and $4.40 per 1M output tokens.
Official Verification Sources & Links
- Public Terminal-Bench 4.0 Leaderboard: tbench.ai/leaderboard/terminal-bench/4.0
- Harbor Hub Dataset & Leaderboard: hub.harborframework.com/datasets/terminal-bench
- Terminal-Bench Official Portal: tbench.ai/news/terminal-bench-4-0
- Artificial Analysis Terminal-Bench 4.0 (independent, mini-SWE-agent): artificialanalysis.ai/evaluations/terminalbench-4-0
- Vals AI Terminal-Bench 4.0 Runs: vals.ai
- Anthropic Claude Sonnet 5.5 Launch Table: anthropic.com/claude-sonnet-5-5
- Claude Opus 5.5: anthropic.com/claude-opus-5-5
- Fable 5.1 & Mythos 5.1 System Card: Anthropic system card (PDF)
- Gemini 4 Argon Model Evaluation: Google DeepMind evaluation table (PDF)
- GitHub Repository: github.com/harbor-framework/terminal-bench
Test Frontier CLI & Terminal Agents on Real Workloads
Compare Claude Opus 5.5, Claude Sonnet 5.5, GPT-6.1 Sol and GLM-5.3 in isolated sandbox environments on CodingFleet.
⚡ Launch CodingFleet Sandbox →