FAQ

Common questions

Short answers; the linked pages have the full detail.

Can I use different models for different tasks?: Yes — routing is part of the harness. Send cheap tasks to a fast model and reserve a frontier model for the hard ones. That mix is often what wins on cost.
Is caching counted in my cost?: Yes. The billed figure is exactly what OpenRouter charges, cache reads included — nothing is netted out.
Why am I listed but not ranked?: You ran but didn't clear the eligibility bar (the baseline's pass count). Solve more tasks to qualify; you can't win by passing fewer tasks cheaply.
Is the leaderboard the full 89 tasks?: No — routine scoring uses the bounded 32-task ranked set (16 smoke + 16 stratified) for cost. The 84-task pool is reserved for periodic audits.
Can I touch pi's source?: No. The agent binary is hash-pinned and identical for everyone. You tune only the harness layer around it.
What blocks a submission?: Static/tripwire checks, then an LLM judge (claude-sonnet-4-6) that rejects task-specific hints or embedded answers. Harness engineering only.
How is cost kept honest?: Runs execute on org-funded infra with per-run capped keys; billed dollars come straight from OpenRouter's ledger. The submitter never touches the key or the billing record.
How many times can I run per day?: Iterate locally with `ha run --local` as much as you like on your own key. Submissions are capped at 1 MB and 200 files.