Terminal-Bench 2.1 & the bounded protocol
The tasks are terminal-native and pass/fail. Routine scoring runs a bounded, stratified subset — not all 89 tasks every time — for cost. Here's exactly how the pool is split.
What a task looks like
Each task drops the agent into a Docker container with a real problem — fix a broken git repo, stand up a gRPC server, recover a corrupted database write-ahead log, modernize a COBOL program — and an oracle check that verifies the final system state. Scoring is binary: the oracle passes or it doesn't. No partial credit.
The four stages of the pool
| Stage | Tasks | Trials | Purpose |
|---|---|---|---|
| Canary preflight | 4 | 1 | One-eighth-scale paid check, easy→hard |
| Smoke gate | 16 | 3 (median) | Cheap pass/cost signal; gates the full run |
| Ranked (“full”) run | 32 = 16 smoke + 16 stratified | 1 | The scored run behind the leaderboard |
| Audit pool | 84 (modality-neutral) | — | Periodic integrity audits, not routine scoring |
Terminal-Bench 2.1 dataset (~89 tasks)
│
├─▶ 84-task modality-neutral pool ──────────────▶ periodic AUDITS (catch overfitting)
│
└─▶ bounded scoring protocol
┌────────────┐ gate ┌─────────────────────────────┐
│ smoke: 16 │ ───────▶ │ ranked: 32 (16 smoke + 16) │ ──▶ billed $ = score
└────────────┘ └─────────────────────────────┘
median-of-3 one trial
Why a subset: a difficulty-stratified slice reproduces the full-benchmark ranking at a fraction of the cost. Absolute pass-rates shift on a subset, but who-beats-whom is stable — which is all a leaderboard needs. The 84-task pool stays in reserve so anyone who overfits the fixed 32 gets caught on audit.
The smoke set (16)
fix-gitgit-leak-recoverypypi-serverkv-store-grpcwrite-compressornginx-request-loggingconfigure-git-webservermodernize-scientific-stacksanitize-git-repopassword-recoveryquery-optimizecustom-memory-heap-crashdb-wal-recoverymodel-extraction-relu-logitsregex-logpytorch-model-cli
The 16 stratified additions (→ 32 ranked)
cobol-modernizationcaffe-cifar-10constraints-schedulingextract-elfheadless-terminallargest-eigenvalmulti-source-data-mergeropenssl-selfsigned-certpytorch-model-recoveryqemu-startupcancel-async-tasksfix-code-vulnerabilityllm-inference-batching-schedulermake-mips-interpreterprotein-assemblysparql-university