Terminal-Bench 4.0 and the Frontier Saturation Problem: GPT-6 Astra, Claude Fable 5.1 Within 0.3 Points

Terminal-Bench measures whether a model can complete difficult, multi step computer work, 66 professional tasks in the current 4.0 revision, with each model given up to eight hours per task and scored across five trials. The top of the current leaderboard is GPT-6 Astra at 58.18 percent, Claude Fable 5.1 at 57.88 percent, and GPT-6 Astra again, this time running with xhigh reasoning effort, tied with Fable at 57.88 percent. The gap between first and third place on this leaderboard is 0.30 percentage points. That is the actual headline, not which model happens to sit in first.

The current leaderboard

Rank	Model	Score	Note
1	GPT-6 Astra	58.18%	Standard reasoning configuration
2	Claude Fable 5.1	57.88%
3 (tied)	GPT-6 Astra, xhigh reasoning	57.88%	Higher reasoning effort setting did not improve on the model's own standard result

Why a 0.30 point gap should change how you read leaderboards

A benchmark score with a 0.30 percentage point spread between rank one and rank three is, in any statistically grounded sense, a tie. Five trials per task across 66 tasks is not a large enough sample to distinguish between models that close to each other with confidence, and Terminal-Bench's own framing acknowledges this: the page describes the top three results as clustered, which is the maintainers' own way of saying do not read ranking order as a meaningful signal at this margin. Compare this to Terminal-Bench 2.1's leaderboard, covered here in a prior post, where the spread between leading models was wide enough to represent a real capability difference. The 4.0 revision's top tier no longer has that separation, at least not on this particular set of 66 tasks.

The detail that matters more than the ranking: xhigh reasoning did not help

GPT-6 Astra's xhigh reasoning configuration scored identically to Claude Fable 5.1, and notably did not outscore GPT-6 Astra's own standard configuration. That is a more interesting result than the top line ranking, because it means throwing additional reasoning effort at Terminal-Bench's task set produced no measurable benefit for the model that already leads the benchmark. Terminal-Bench tasks are long horizon, multi step computer operations, the kind of work where more deliberate reasoning per step plausibly helps, and the fact that it did not move the needle here suggests the bottleneck on these particular tasks is not reasoning depth. It is more likely tool use reliability, environment state tracking across long sessions, or recovery from errors mid task, none of which scale simply with more reasoning tokens per decision.

What frontier saturation on one benchmark does and does not tell you

Saturation on Terminal-Bench 4.0 specifically means these particular 66 tasks no longer differentiate the top models well. It does not mean these models are equivalent on your workload. A benchmark is a fixed, finite task set, and a model's aggregate score across it is an average that can hide meaningful task by task variance. Two models tied at 58 percent aggregate can have very different failure patterns, one might be strong on long running shell sessions and weak on environment setup, the other the reverse, and that variance is exactly the kind of thing an aggregate leaderboard number cannot show you. The practical implication is that once a benchmark reaches this level of top tier clustering, it stops being a useful tool for choosing between the top two or three models and becomes more useful as a floor check, confirming a model clears a baseline capability level, with the actual selection decision needing task specific evaluation against your own workload instead.

How this compares to the SWE-bench Pro situation

This is a different problem from the Scale versus vendor reported disagreement on SWE-bench Pro, where the issue is inconsistent measurement methodology producing very different scores for the same models. Terminal-Bench 4.0's issue is the opposite: the measurement is consistent, and the models are simply performing close enough to each other that the resulting scores cluster. Both problems point toward the same conclusion from different directions, that a single leaderboard number is an increasingly unreliable basis for a model selection decision at the frontier, whether because the methodology itself is unstable or because the models being measured have genuinely converged on this particular task distribution.

What to actually do with a saturated benchmark

Use Terminal-Bench 4.0 as a gate, not a ranking. If a candidate model clears roughly 55 to 58 percent, it is in the range of the current frontier on this task type, and the aggregate score should stop being the deciding factor between it and its closest competitors. From there, the decision should shift to task specific evaluation: run your own representative long horizon computer use tasks against the top two or three candidates and compare failure modes directly, rather than trusting a 0.30 point aggregate gap to represent a real difference. For teams building agentic coding or computer use tooling, this is also a signal to weight evaluation criteria beyond raw completion rate, latency per task, cost per completed task, and recoverability from mid task failures are all dimensions a tied aggregate score does not capture.

When this applies to your stack

If your team is choosing a model for agentic coding or long horizon computer use tasks based on a Terminal-Bench 4.0 leaderboard position, stop treating the ranking as decisive once you are inside this clustering. The actual differentiator at this margin is how a model fails, not its aggregate pass rate, and that requires testing against your own tasks.

Contra Collective builds and evaluates production agentic coding infrastructure for enterprise clients, including the task specific benchmarking work that goes beyond leaderboard position once frontier models cluster this tightly. If your team is choosing between top tier models based on a 0.30 point leaderboard gap, that decision deserves a closer look before it gets made.

FAQ

Does GPT-6 Astra's first place ranking on Terminal-Bench 4.0 mean it is the best coding agent model overall? Not reliably. A 0.30 percentage point gap between the top three results is within the range where trial to trial variance across five runs on 66 tasks could plausibly explain the ordering. Treat the top three as tied rather than ranked.

Why didn't xhigh reasoning improve GPT-6 Astra's score? The benchmark's own results show no improvement from the higher reasoning effort configuration, which suggests Terminal-Bench 4.0's task set is bottlenecked on something other than reasoning depth, most plausibly tool use reliability or long session state tracking rather than decision quality per step.

How many tasks and trials does Terminal-Bench 4.0 use? 66 professional tasks, with each model allowed up to eight hours per task and scored across five trials per task.

Should we stop using Terminal-Bench 4.0 to evaluate models if the top results are clustered? No, but its role should shift. Use it to confirm a candidate model clears the current frontier floor, then evaluate top candidates against your own representative tasks to make the actual selection decision, since the aggregate leaderboard no longer differentiates the top tier meaningfully.