AutoRouter Arena
Overview

How Harness Arena works

Everyone runs the same coding agent — a vanilla pi CLI on OpenRouter. You compete by tuning the harness around it, not the model, to solve terminal tasks for the fewest billed dollars. These pages document the whole system, one topic at a time.

The premise

Most benchmarks compare models. Harness Arena fixes the model layer and compares the harness — the skills, agent profiles, instructions, and config wrapped around a fixed agent binary. The base agent is @earendil-works/pi-coding-agent v0.80.9, installed from a hash-pinned tarball, so every entrant runs byte-identical agent code. The only thing that differs between entries is the harness and the model choice.

Why fix the agent and score the harness? Because in controlled studies the harness is the dominant lever: holding the model constant and swapping the harness accounts for roughly a 7.8× larger share of performance variance than swapping models, and most pairwise rankings flip when the harness changes. A competition that isolates the harness is measuring the thing that actually moves outcomes — and it does so on a cost axis, because the research is equally clear that simple agents routinely match complex ones at a fraction of the spend. Accuracy without a cost budget just rewards burning tokens.

The one-line rule

score = billed dollars from OpenRouter's generation ledger
match or beat the baseline pass countthen cheapest winsTerminal-Bench 2.x

You must first qualify — solve at least as many tasks as the baseline harness. Among everyone who qualifies, the cheapest billed run wins. Cost is real spend pulled from the provider's ledger, not self-reported.

The loop


   ┌─────────────┐   tune skills/     ┌──────────────┐  ha run --local ┌─────────────┐
   │  your       │   profiles/config  │  local       │   (16 tasks,     │  pass/cost  │
   │  harness    │ ─────────────────▶ │  iteration   │ ───your key────▶ │  in seconds │
   └─────────────┘                    └──────────────┘                  └──────┬──────┘
          ▲                                                                    │ good?
          │ refine                                                            ▼
          │                          ┌──────────────┐   judge + queue   ┌─────────────┐
          └───────────────────────── │ ha submit │ ◀──────────────── │   submit    │
                                     └──────┬───────┘                   └─────────────┘
                                            │ accepted, EC2 worker runs it
                                            ▼
                                     ┌──────────────┐   billed $ from    ┌─────────────┐
                                     │  scored run  │ ──the ledger────▶  │ LEADERBOARD │
                                     └──────────────┘   (from the DB)    └─────────────┘
  1. Iterate locally with ha run --local against a small task subset on your own key — fast, cheap feedback. See Iterating locally.
  2. Submit with ha submit; the API runs a tripwire scan and an LLM judge, then queues it. See Submission pipeline.
  3. An EC2 worker scores it on org-funded infra with capped keys, and writes the billed cost to the database behind the leaderboard. See Scoring.

Where to go next

  • The rules — exactly what you may and may not change.
  • The benchmark — Terminal-Bench 2.1 and the bounded scoring protocol.
  • Scoring — how a dollar figure becomes a rank, with a worked example.
  • Reading the site — how to interpret the leaderboard, frontier, and heatmap.
Next →The rules