The Verifiable Arena

Every published benchmark number is, structurally, an assertion: somebody says they ran something and got a figure. Nothing in the artifact separates a measured score from a typed one, which is why benchmark gaming is cheap and catching it is not.

The Arena removes the assertion. Each task of each run is a real metered inference through the ordinary chat path, so it is held, capped, guardrailed, metered and — the part that matters — receipted: an individually signed payload naming the model, the engine that actually served, the token counts, the cost and the time. The run stores those receipt ids and marks the receipts shared, so a stranger can pull every one of them, verify each signature, and confirm the row was produced by work that actually happened.

A score that was typed has no receipts to show.

The Arena is built and tested. It is not on the production deployment yet, so the endpoints below describe a surface you cannot reach at api.buildsable.com today, and no board is being published.

Rerun it yourself

The other half is the declaration. A suite's tasks are published verbatim at GET /v1/arena/suites/{slug} and hashed canonically. That hash rides on every leaderboard row, and a run stores the hash it ran against rather than pointing at whatever the suite says today.

So the tasks can be fetched, re-hashed, and re-run by anyone — against the same models, through their own account — and a suite quietly edited under a stable name shows up as a hash that no longer matches the runs beneath it.

tasks_sha256 = sha256(utf8(canonical_tasks))

where canonical_tasks is the task array re-serialized with exactly the fields {prompt, assertions: [{type, value}]}, in declared order, no whitespace. The per-suite response returns canonical_tasks alongside the hash, so you hash the bytes we showed you rather than reconstructing them — and returns tasks_sha256_recomputed and tasks_sha256_matches, which is Sable checking itself: if a row's stored hash ever disagreed with its own tasks, the response says so rather than serving the stored number as if it had been checked.

Two rules that keep the numbers honest

A run scores only if every task completed. If any task's call fails — upstream down, key out of credit, guardrail block — the whole run is recorded failed with a fixed error class and no score is published. Scoring an unreachable model 0% would be a false claim about the model rather than a true one about the gateway. Failed runs are still served by GET /v1/arena/runs/{id}, with the error class and no score: an outage is part of the record, not something to omit.

Scoring has one implementation. Assertions are evaluated by the same function the session-authed Evals product and the drift monitors use. There is no second scorer to drift.

Ranking

Stated in the response rather than left to be inferred:

Within each suite: score_bps descending (tasks passed, in basis points), then ran_at descending, then model ascending. Ties are ties — no hidden tiebreak, no weighting, no aggregate across suites. Only runs whose every task completed are listed.

The board is grouped by suite because comparing a score from one task list against a score from another is the exact error the whole thing exists to make visible.

Endpoints

All public. A benchmark that can only be read by the party that ran it is a claim, not a proof.

curl -s https://api.buildsable.com/v1/arena/leaderboard

# {
#   "suites": [{
#     "suite": "sable-core-v1", "name": "Sable Core v1",
#     "tasks_sha256": "4c81…", "suite_url": "/v1/arena/suites/sable-core-v1",
#     "standings": [{
#       "model": "sable-llama-3.3-70b", "score_bps": 8333,
#       "passed": 5, "total": 6, "ran_at": "2026-09-13T02:00:00Z",
#       "run_id": "…", "receipt_count": 6, "cost_micro_usd": 812,
#       "run_url": "/v1/arena/runs/…"
#     }]
#   }],
#   "ordering": "within each suite: score_bps descending …",
#   "trust_model": "attested: every task of every run is a real metered inference …"
# }

How a row is checked

1. GET /v1/arena/suites/{slug} → recompute sha256 over the canonical task serialization and compare with tasks_sha256 on the row.
2. GET /v1/arena/runs/{run_id} → for each receipt id, GET /v1/receipts/shared/{request_id}, then POST /v1/receipts/verify with {receipt, signature}.

Every receipt of a run shares one chain id, published as run_chain_id, so the run also has a single head hash. The chain endpoints themselves are owner-authed; the id is published because it is what ties the receipts together.

Suites

Suites are seeded by the operator and cannot be submitted by a caller — there is no create endpoint. The first one, sable-core-v1, is six short, deterministically-checkable tasks: instruction following, arithmetic, format compliance, refusal of an ungrounded claim, extraction, and negation.

Small on purpose. The Arena's claim is that scores are receipted, not that the tasks are hard.

Cost and configuration

The Arena spends real money on a schedule, so the whole feature is off unless an operator names a house account, and every pass is bounded before it starts.

Ceilings are computed from the catalogue's published prices at the worst case — every task using its full max_tokens of prompt and completion — with the deployment's margin applied. That is the price the house account will actually be billed, not the upstream cost.

Privacy posture

Arena tasks are Sable's own declared public prompts. They are published verbatim precisely so a third party can rerun them, so storing them in the clear stores nothing private. That is true only because these tasks are public by construction, and it generalises to nothing else.

No user content ever enters these tables: a caller cannot submit a suite, and there is deliberately no column anywhere for a model's output. A run stores counts, costs and receipt ids, and the receipts are metadata-only by construction. See the privacy contract.

Related

• Verifiable receipts — what a score is chased down to.
• Evals — the same assertion engine, on your own cases and your own key.
• Models — the catalogue a roster is drawn from.