Integrity & anti-abuse
Why the numbers can be trusted
A leaderboard with money attached is only worth reading if the scores can't be gamed. Here is each way someone might try, and what stops them.
Threat model
| Attack | Mitigation |
|---|---|
| Self-reported / fudged cost | Billed straight from OpenRouter's generation ledger on org-funded infra |
| Runaway or padded spend | Per-run capped keys — $1.50 smoke, $10 full |
| Baked-in task answers | Tripwire scan + claude-sonnet-4-6 judge |
| Overfitting the fixed 32 | 84-task audit pool held in reserve for periodic re-scoring |
| Spam / brute forcing configs | Per-account submission limits; 1 MB & 200-file submission caps |
| Impersonation / multi-entry | Identity = verified GitHub login, one entry per person |
| Free-variant cost dodging | Free model variants banned outright |
The trust chain
Every number on the board traces back to a source nobody in the competition controls:
pinned pi binary (sha256) ┐
fixed task set + oracle checks ├─▶ org-funded run ─▶ OpenRouter ledger ─▶ committed score
capped provider key ┘ │ │ │
can't alter agent can't fake spend public + auditable
The submitter never touches the key, the infra, or the billing record — so the cost on the board is the cost the provider actually charged.
Transparency as a defense
Nothing is hidden. Every entry's harness files, per-task pass/fail, and per-task model/cost are public on its run page. That's deliberate: open submissions let the community spot anything fishy, and let everyone learn from what works.