Portal
Documentation: all sections

Evals

Evals turn a model into something you can regression-test. You define a suite of cases, each with assertions about the output, run the suite against a model, and get a pass rate back. Run it again later and Sable flags a regression when the pass rate dropped versus the previous run.

Runs are real inference: each case is a genuine metered call on your prepaid balance, so the pass rate reflects the model as it actually behaves. What is stored is only the score, the pass/fail of each case, and which assertions failed. The model output itself is never stored, in keeping with the privacy contract.

Evals are session-authed: they run behind your Sign-In With Ethereum dashboard session, and there is a portal page for them at Portal → Evals.

Assertions

Each case carries one or more assertions. An assertion has a type and a value:

TypePasses when the output...
containscontains value as a substring (case-sensitive).
icontainscontains value, case-insensitively.
not_containsdoes not contain value.
equalsequals value exactly.
regexmatches the regular expression value.
min_lengthis at least value characters long.

A case passes only when all of its assertions pass.

Quickstart

# 1. Create a suite
curl https://api.buildsable.com/v1/evals \
-H "authorization: Bearer $SABLE_SESSION_TOKEN" \
-H 'content-type: application/json' \
-d '{
  "name": "support-tone",
  "model": "sable",
  "cases": [
    {
      "input": "A customer asks for a refund. Reply in one sentence.",
      "assertions": [
        {"type": "icontains", "value": "refund"},
        {"type": "min_length", "value": 20}
      ]
    }
  ]
}'

# 2. Run it (real, metered inference)
curl -X POST https://api.buildsable.com/v1/evals/$EVAL_ID/run \
-H "authorization: Bearer $SABLE_SESSION_TOKEN"

A run reports the pass rate, whether it regressed, the cost, and the per-case outcome, including which assertions failed:

{
  "run_id": "run_5f2a…",
  "total": 1,
  "passed": 1,
  "pass_rate": 1.0,
  "regression": false,
  "cost_micro_usd": 240,
  "results": [
    { "index": 0, "passed": true, "failed_assertions": [] }
  ]
}

Tracking regressions over time

GET /v1/evals/{id}/runs returns the run history with each run's pass rate, so you can watch a suite over time and see exactly when a model change moved the number. The regression flag on a run is set whenever its pass rate is lower than the run before it, which is the signal to look at what changed.

Endpoints

All session-authed (Authorization: Bearer sess_…).

MethodPathWhat it does
POST/v1/evalsCreate a suite. Body {name, model, cases}.
GET/v1/evalsList your suites.
DELETE/v1/evals/{id}Delete a suite.
POST/v1/evals/{id}/runRun the suite against its model. Returns pass rate, regression, cost, and per-case results.
GET/v1/evals/{id}/runsRun history with pass rate over time.

Privacy and limits