Scoring

How billed dollars become a rank

Your score is real spend from OpenRouter's ledger — caching included, never self-reported. Qualify on pass count, then the cheapest run wins.

The decision, step by step


   your run ──▶ pass_count ≥ eligibility bar ?
                     │                    │
                    NO                   YES
                     ▼                    ▼
              listed "below the      eligible → rank by billed_usd (ascending)
              bar" (not ranked)             │
                                            ▼
                              cheapest qualifying run = #1
                              tie-break / efficiency read: $ / passed

Step	Rule
Eligibility	Match or beat the baseline's pass count (the bar, frozen by a baseline probe)
Ranking	Among eligible entries, lowest `billed_usd` wins
Efficiency read	`$/passed` = billed cost ÷ tasks solved
Stability	The smoke gate uses the median of 3 trials

Worked example

Say the baseline passes 61 and costs $9.81, so the bar is 61. Four entries come in:

Entry	Passed	Billed	$/passed	Result
pjhyett	62	$2.80	$0.045	✅ #1 — qualifies, cheapest
mona	64	$5.50	$0.086	✅ #2 — qualifies, pricier
defunkt	68	$8.10	$0.119	✅ #3 — most passes, but costliest
octocat	57	$1.90	$0.033	❌ below the bar — cheapest, but under 61

Note octocat is the cheapest run andbest $/passed, but doesn't qualify — you can't win by solving fewer tasks cheaply. And defunkt solves the most but ranks last of the qualifiers because the metric is cost, not accuracy.

Why cost, not accuracy

Without a cost budget, an agent can chase a fractional pass-rate gain with unbounded API calls. The literature shows simple agents matching complex ones at a small fraction of the spend, so the dollar axis is the honest signal of an efficient harness. Accuracy is the gate; cost is the score.