modelrig 0.11.0 → 0.11.1

package/docs/agent-skill/SKILL.md

@@ -65,14 +65,14 @@ MODELRIG_API_KEY=rig_sk_…
 <!-- /BLOCK:KEY -->
 
 <!-- BLOCK:CONFIRM_ORG -->
-**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in, once your app has started with the key. It asks GET /v1/whoami which organization the key belongs to and prints the org's name and, in parentheses, its slug; the name must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
 
 ```bash
 npx modelrig doctor
-# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+# [modelrig] key rig_sk_…AbCd → org «Your Org» (your-org) ✓  (ingest https://api.modelrig.ai, …)
 
-# No CLI in that environment? Ask the API directly:
-curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+# No CLI in that environment? Ask the API directly (same host the SDK uses):
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" "${MODELRIG_INGEST_URL:-https://api.modelrig.ai}/v1/whoami"
 ```
 <!-- /BLOCK:CONFIRM_ORG -->
 
@@ -112,7 +112,103 @@ await rig.close(); // flushes the exporter — call once at shutdown
 **What ModelRig does not see unless you record it.** ModelRig records the calls your code makes through the SDK (`rig.run`, `rig.runBundle`, `rig.runRaw`, `rig.runRawStream`) or the gateway, plus any call your code makes itself and reports with `rig.recordCall`. Nothing else is seen. Five things stay invisible or unattached unless you record them: (1) direct provider SDK calls, with or without a JSON schema; (2) streams you read from a provider client yourself; (3) a fallback your code makes after the raw lane refuses a `responseSchema` it cannot enforce (deepseek, deepinfra, fireworks and zai fail closed with `invariant_violation`); (4) calls made outside `run.scope` / `run.start`, which land on /inferences but on no run; (5) embeddings. A recorded call shows as "recorded · unverified": ModelRig neither dispatched nor validated it, and its tokens are the numbers you reported. Details and the console tells: https://modelrig.dev/not-captured.html.
 <!-- /BLOCK:NOT_CAPTURED -->
 
-A provider call you can't move onto a lane (an unsupported client, a stream you read yourself, a fallback): record it with `rig.recordCall({ provider, model, usage, servedTier, latencyMs, stepKey })` right after it returns, never leave it invisible (modelrig ≥ 0.11.0; https://modelrig.dev/not-captured.html).
+A provider call you can't move onto a lane (an unsupported client, a stream you read yourself, a fallback, an embedding): record it right after it returns, never leave it invisible (https://modelrig.dev/not-captured.html):
+
+<!-- BLOCK:RECORD_CALL -->
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "recorded · unverified", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+
+```ts
+const started = Date.now();
+const resp = await genai.models.generateContent({ model: "gemini-3.1-pro-preview", contents });
+const u = resp.usageMetadata;
+const finishReason = resp.candidates?.[0]?.finishReason;
+
+const rec = rig.recordCall({
+  provider: "gemini",
+  model: "gemini-3.1-pro-preview",
+  usage: {
+    tokensIn: u?.promptTokenCount ?? 0, // total input, cached included
+    tokensOut: (u?.candidatesTokenCount ?? 0) + (u?.thoughtsTokenCount ?? 0),
+    tokensCached: u?.cachedContentTokenCount ?? 0,
+    // optional fields: spread only when the provider reported them
+    ...(u?.thoughtsTokenCount !== undefined ? { reasoningTokens: u.thoughtsTokenCount } : {}),
+  },
+  servedTier: "standard", // the tier that SERVED the call: pass what the provider reports (flex, priority); it is priced at that tier
+  latencyMs: Date.now() - started,
+  stepKey: "summarize", // the step on /runs (inside run.scope / run.start)
+  ...(finishReason !== undefined ? { finishReason } : {}),
+});
+if (rec.inferenceId !== null) rig.grade({ kind: "inference", id: rec.inferenceId }, { score: 1, kind: "deterministic" });
+```
+<!-- /BLOCK:RECORD_CALL -->
+
+No routes yet (observe, or a `runRaw` seam)? Build the rig raw-only:
+
+<!-- BLOCK:RAW_RIG -->
+**No routes yet? Build the rig raw-only.** Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with `routesDir: null`: `rig.runRaw`, `rig.runRawStream` and `rig.recordCall` work at once, and `rig.run` / `rig.runBundle` refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.
+
+```ts
+import { createRig, loadConfigFromEnv } from "modelrig";
+
+// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
+// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
+const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
+```
+<!-- /BLOCK:RAW_RIG -->
+
+Grade from a signal you already have, and prove a cheaper model on your own traffic:
+
+<!-- BLOCK:GRADE_RUN -->
+**Attach a grade to a call or a whole run.** A grade is your judgment: a 0–1 score with an optional comment, from a validator, a downstream accept/reject, a human or a model. Grade one call by its inference id (`result.meta.inferenceId`, or the `inferenceId` rig.recordCall returns). Grade a whole run by its run id, `run.runId` on the handle `run.start` returns: that grade shows on the run page and on /projects. A run grade whose id is the run's `episodeKey` also shows, on the run page only. `run_id` is a reserved tag the SDK stamps, not a variable your code has. A grade whose id matches no call or run attaches to nothing. The write is synchronous and local and never throws into your pipeline.
+
+```ts
+const run = rig.artifacts.run.start({ pipeline: "support", episodeKey: ticketId, environment: "production" });
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },
+  tags: { subject: customer },
+  stepKey: "summarize",
+});
+run.end("succeeded");
+
+// one call: the inference id rides on every result
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "deterministic", grader: "schema-check" });
+// the whole run: its id is run.runId from run.start (shows on the run page and on /projects)
+rig.grade({ kind: "run", id: run.runId }, { score: 0, kind: "human", comment: "missed the refund request" });
+```
+<!-- /BLOCK:GRADE_RUN -->
+
+<!-- BLOCK:BAKEOFF -->
+**Prove a cheaper model on your own traffic.** A bake-off replays your own inputs through the variants a route declares and reports the effective cost of conformance for each. `--variants` names entries in the route's `variants:` block (`default` is the route as declared), never model ids, and a variant can only narrow the route's candidates. Set `capture: true` first: replay works on traffic captured after that, not before. With no captured traffic (a run-based pipeline), promote a real /runs output to a golden (the console's make golden, or `promoteToEval`) and use `--from-eval-cases <task>`; its variants come from the route bundle named after the task. A bake-off measures and never switches anything.
+
+```bash
+# 1. in modelrig/routes/example.support_summarize.yaml: capture traffic, and declare
+#    a cheaper variant (narrowing only: its candidates must be in the route's list)
+capture: true
+variants:
+  - name: cheap
+    candidates:
+      - provider: gemini
+        model: gemini-3.1-flash-lite
+
+# 2. after traffic accumulates, compare the route as declared (default) with cheap
+npx modelrig bakeoff --route example.support_summarize --variants default,cheap --replay-last 50
+
+# no captured route traffic? promote a real /runs output to a golden, then
+npx modelrig bakeoff --from-eval-cases example.support_summarize --variants default,cheap
+```
+<!-- /BLOCK:BAKEOFF -->
+
+On a hosted worker, pin the org a deployment must export to:
+
+<!-- BLOCK:EXPECT_ORG -->
+**Pin the org a deployment must export to.** `--expect-org` and `expectOrg` take the org's slug, not its display name. `modelrig doctor` prints both, as `→ org «Your Org» (your-org) ✓`: the slug is the part in parentheses. In the console, hover the org badge in the header to see it. With `--expect-org <slug>`, doctor exits non-zero when the key belongs to another org, so a deploy check fails loudly instead of exporting somewhere else.
+
+```ts
+// The same check in code: the exporter asks whoami on its first flush and logs
+// ORG MISMATCH on every flush while the key belongs to another org.
+const rig = createRig({ ...loadConfigFromEnv(), expectOrg: "your-org" });
+```
+<!-- /BLOCK:EXPECT_ORG -->
 
 Which lane fits a codebase (observe, the `runRaw` seam, or named routes):
 <https://modelrig.dev/choosing-a-lane.html>. Hosted workers:
@@ -122,11 +218,12 @@ Which lane fits a codebase (observe, the `runRaw` seam, or named routes):
 
 Before migrating anything, you can just **watch** an existing pipeline. ModelRig
 records each run as a run → step → artifact chain you click through in the
-console — no routes moved, no model calls rerouted, and with the flag off the
+console — no routes moved, no model calls rerouted, and with no key set the
 pipeline behaves byte-for-byte as before. Scaffold it:
 `npx modelrig observe [path] --scope <one pipeline>` writes a self-contained
-`record-step` seam and an `OBSERVE.md` whose wiring is emitted as diffs you
-apply. The instrumentation is a code-level SDK seam (no zero-code observe), but
+`record-step` seam (a raw-only rig, `routesDir: null`, that reports each call
+with `rig.recordCall` so steps carry a cost) and an `OBSERVE.md` whose wiring,
+including `environment` on the run, is emitted as diffs you apply. The instrumentation is a code-level SDK seam (no zero-code observe), but
 you no longer hand-write it — see <https://modelrig.dev/instrumentation.html>.
 The namespace is **on by default once a control plane is configured** (a
 `rig_sk_` API key, or the self-host Supabase pair); a pure local-only rig stays
@@ -150,8 +247,8 @@ behaviour-affecting, so the agent *proposes* it, proves it with
 never the skill's job.
 
 **Runs are the standard path, not an appendix.** Every pipeline execution should
-be a **run**: wrap it in `rig.artifacts.run.start` / `run.scope`, name one step
-per model-call family (pass `stepKey`), `artifact.save` the work products, and tag
+be a **run**: wrap it in `rig.artifacts.run.start` / `run.scope`, name each
+model call's step with `stepKey`, `artifact.save` the work products, and tag
 `subject` / `feature`. Inside a run context the SDK auto-stamps `tags.run_id` and
 `tags.step` for you, so the run fills `/runs` **and** `/projects`; acceptance is
 the run appearing on `/runs` with its steps. `run_id` is reserved: never set it

package/docs/how-it-fits.md

@@ -12,7 +12,7 @@ What is retained beyond that is your choice, per route, and there are two lanes.
 
 **Pure router** (capture off). Nothing retained. Real, supported, selectable, and the honest answer for a ZDR-strict or data-residency-bound workload.
 
-**Optimization on** (recommended). Your traffic is what makes the routing smarter and the bill smaller. **What is kept:** per-inference telemetry rows, and bounded review samples for the bake-offs you create. **Where:** your organization, row-scoped by the database, not by our application code. **How you leave:** see it all in the console, delete it, export it, take your routes and go — they were always YAML in your git. **What you get back:** community priors on your very first bake-off and RigIndex-grade evidence on your own routes — at the same flat 2% either way; the corpus earns its place by being useful, not by being a discount.
+**Optimization on** (recommended). Your traffic is what makes the routing smarter and the bill smaller. **What is kept:** per-inference telemetry rows, and bounded review samples for the bake-offs you create. **Where:** your organization, row-scoped by the database, not by our application code. **How you leave:** see it all in the console and export it; never a silent swap — promote a change in ModelRig (console or MCP; versioned, reversible) or merge it as a PR if you keep routes in git. **What you get back:** community priors on your very first bake-off and RigIndex-grade evidence on your own routes — at the same flat 2% either way; the corpus earns its place by being useful, not by being a discount.
 
 Captures power **bake-offs**: replay your own recent traffic through route variants — a cheaper model, a different prompt — and get the effective cost of conformance for each, with confidence intervals. A winning variant is a proposal for you to apply in your YAML; nothing swaps models automatically.
 

package/docs/instrumentation.md

@@ -37,8 +37,6 @@ path — never blurred with the telemetry axis here.
 # set_org_settings MCP tool), not this env var.
 export MODELRIG_ARTIFACTS=1                  # local-only namespace toggle: force ON
                                              #   with =1 on a rig with no control plane
-export MODELRIG_ARTIFACT_STEPS=step_a,step_b   # optional: instrument a subset
-                                            # (unset/empty = every instrumented step)
 ```
 
 Rows land in the local telemetry buffer and export to the control plane through
@@ -54,38 +52,60 @@ agent-driven setup should plan that one handoff rather than stall on it.
 The namespace hangs off `rig.artifacts`. Two moving parts: a **run context**
 around the whole pipeline, and a **per-step save** of the work products.
 
+No routes yet? Observing needs none. Build the rig raw-only:
+
+<!-- BLOCK:RAW_RIG -->
+**No routes yet? Build the rig raw-only.** Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with `routesDir: null`: `rig.runRaw`, `rig.runRawStream` and `rig.recordCall` work at once, and `rig.run` / `rig.runBundle` refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.
+
 ```ts
 import { createRig, loadConfigFromEnv } from "modelrig";
 
-const rig = createRig(loadConfigFromEnv());
+// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
+// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
+const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
+```
+<!-- /BLOCK:RAW_RIG -->
 
+Then wrap the pipeline. Here each step's model call stays your own direct call,
+reported with `rig.recordCall` so the step carries tokens and a cost:
+
+```ts
 // 1. Open a run context around the pipeline. episodeKey bridges grades;
 //    it is a grouping key, not a unique id — a natural choice is your own
 //    per-run attempt id. environment fills the Env column on /runs.
 const run = rig.artifacts.run.start({
   pipeline: "my-pipeline",
   episodeKey: runAttemptId,
-  environment: process.env.NODE_ENV,   // e.g. "production", "staging", "test"
+  environment: process.env.NODE_ENV ?? "development", // e.g. "production", "staging", "test"
 });
 try {
   for (const step of steps) {
     const prompt = assemble(step);
-    const result = await callModel(prompt);          // your own model call
+    const started = Date.now();
+    const result = await callModel(prompt);          // your own direct model call
+
+    // 2. Report the call: tokens → cost on the step and on /inferences.
+    rig.recordCall({
+      provider: "openai",
+      model: "gpt-5.4-mini",
+      usage: { tokensIn: result.usage.inputTokens, tokensOut: result.usage.outputTokens },
+      latencyMs: Date.now() - started,
+      stepKey: step.key,
+    });
 
-    // 2. Save the step's artifacts. stepKey groups them in the console's
+    // 3. Save the step's artifacts. stepKey groups them in the console's
     //    chain view; task carries the semantic task name.
-    const p = rig.artifacts.artifact.save(prompt, { name: `${step}.prompt`, type: "prompt", stepKey: step });
-    const raw = rig.artifacts.artifact.save(result.rawText, { name: `${step}.raw`, type: "raw_response", stepKey: step });
+    const p = rig.artifacts.artifact.save(prompt, { name: `${step.key}.prompt`, type: "prompt", stepKey: step.key });
+    const raw = rig.artifacts.artifact.save(result.rawText, { name: `${step.key}.raw`, type: "raw_response", stepKey: step.key });
     const parsed = rig.artifacts.artifact.save(result.json, {
-      name: `${step}.output`, type: "step_output", task: step, stepKey: step, schema: step.schema,
+      name: `${step.key}.output`, type: "step_output", task: step.key, stepKey: step.key, schema: step.schema,
     });
 
-    // 3. Lineage: the parsed output derives from the prompt + raw; this step's
-    //    prompt consumed the previous step's output.
+    // 4. Lineage: the parsed output derives from the prompt + raw.
     if (p && parsed) rig.artifacts.artifact.link(parsed, p, "derived_from");
     if (raw && parsed) rig.artifacts.artifact.link(parsed, raw, "derived_from");
 
-    // 4. Grades (optional, deterministic): flag conformance, degraded
+    // 5. Grades (optional, deterministic): flag conformance, degraded
     //    provenance, anything you can decide without an LLM judge.
     if (parsed) rig.grade({ kind: "artifact", id: parsed.id }, {
       score: conforms(result.json) ? 1 : 0, kind: "deterministic", grader: "schema-check",
@@ -96,8 +116,13 @@ try {
   run.end("failed");
   throw err;
 }
+await rig.close(); // once, at shutdown
 ```
 
+Here `step` is your own `{ key, schema }` record, and `callModel` returns the
+provider's text, parsed JSON and usage. A call routed through `rig.run` or
+`rig.runRaw` records its own step, so it needs no `recordCall`.
+
 For **concurrent** runs in one process, use `rig.artifacts.run.scope(opts, fn)`
 instead of `start`/`end` — each scope keeps its own ambient context so
 overlapping runs never cross-attribute. `run.start` uses `enterWith`, which is
@@ -111,9 +136,10 @@ right for one run at a time on a request/async path.
 - A **step** is an execution row. `rig.run()` auto-creates a **ground-truth**
   step (its inference is real, so cost and latency are measured) when it runs
   inside a run context. A stage whose model call does *not* go through the rig
-  produces artifacts with `step_key` set and **no step row** — which is correct
-  and honest. Never route a stage through the rig just to mint a step row; the
-  console renders "cost n/a" for un-routed steps rather than inventing a number.
+  and is not reported with `rig.recordCall` produces artifacts with `step_key`
+  set and **no step row**, which is correct and honest: the console renders
+  "cost n/a" for that step rather than inventing a number. Report the call
+  (below) and the step gets its tokens and cost.
 
 ### Record a call your code made itself (`rig.recordCall`)
 
@@ -132,6 +158,34 @@ you reported. It is synchronous and never throws (bad input warns and returns
 Full contract, usage convention and cost notes:
 [Not captured → record them](not-captured.md#calls-your-code-makes-itself-record-them).
 
+<!-- BLOCK:RECORD_CALL -->
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "recorded · unverified", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+
+```ts
+const started = Date.now();
+const resp = await genai.models.generateContent({ model: "gemini-3.1-pro-preview", contents });
+const u = resp.usageMetadata;
+const finishReason = resp.candidates?.[0]?.finishReason;
+
+const rec = rig.recordCall({
+  provider: "gemini",
+  model: "gemini-3.1-pro-preview",
+  usage: {
+    tokensIn: u?.promptTokenCount ?? 0, // total input, cached included
+    tokensOut: (u?.candidatesTokenCount ?? 0) + (u?.thoughtsTokenCount ?? 0),
+    tokensCached: u?.cachedContentTokenCount ?? 0,
+    // optional fields: spread only when the provider reported them
+    ...(u?.thoughtsTokenCount !== undefined ? { reasoningTokens: u.thoughtsTokenCount } : {}),
+  },
+  servedTier: "standard", // the tier that SERVED the call: pass what the provider reports (flex, priority); it is priced at that tier
+  latencyMs: Date.now() - started,
+  stepKey: "summarize", // the step on /runs (inside run.scope / run.start)
+  ...(finishReason !== undefined ? { finishReason } : {}),
+});
+if (rec.inferenceId !== null) rig.grade({ kind: "inference", id: rec.inferenceId }, { score: 1, kind: "deterministic" });
+```
+<!-- /BLOCK:RECORD_CALL -->
+
 ## Tags, dimensions, and the graph
 
 ModelRig has **two** grouping mechanisms, and they compose:
@@ -237,7 +291,9 @@ const { bundle: runnable, arm } = resolveCallConfigForRun(bundle, resolved, { in
 
 // Thread the arm onto the run so it lands on inferences.meta.arm.
 await rig.artifacts.run.scope({ pipeline: "report", episodeKey: runId }, async () => {
-  const out = await rig.runBundle(runnable, { input, tags: { subject, feature }, arm, stepKey });
+  // arm is string | null; RunOptions.arm is optional, so pass it only when drawn
+  // (a bare `arm` fails under exactOptionalPropertyTypes).
+  const out = await rig.runBundle(runnable, { input, tags: { subject, feature }, stepKey, ...(arm !== null ? { arm } : {}) });
   rig.artifacts.artifact.save(out.output, { name: "section.output", type: "step_output" });
 });
 ```
@@ -294,14 +350,14 @@ orgs and the console picker: [Orgs and keys](https://modelrig.dev/orgs-and-keys.
 Check first that the console is showing the org your key exports to:
 
 <!-- BLOCK:CONFIRM_ORG -->
-**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in, once your app has started with the key. It asks GET /v1/whoami which organization the key belongs to and prints the org's name and, in parentheses, its slug; the name must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
 
 ```bash
 npx modelrig doctor
-# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+# [modelrig] key rig_sk_…AbCd → org «Your Org» (your-org) ✓  (ingest https://api.modelrig.ai, …)
 
-# No CLI in that environment? Ask the API directly:
-curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+# No CLI in that environment? Ask the API directly (same host the SDK uses):
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" "${MODELRIG_INGEST_URL:-https://api.modelrig.ai}/v1/whoami"
 ```
 <!-- /BLOCK:CONFIRM_ORG -->
 
@@ -319,22 +375,33 @@ an AI — attaches to a subject: one model call (an **inference**), a whole run
 [optimization coverage](grade-protocol.md) is built from — the evidence level
 that decides how much confidence a swap proposal carries.
 
+<!-- BLOCK:GRADE_RUN -->
+**Attach a grade to a call or a whole run.** A grade is your judgment: a 0–1 score with an optional comment, from a validator, a downstream accept/reject, a human or a model. Grade one call by its inference id (`result.meta.inferenceId`, or the `inferenceId` rig.recordCall returns). Grade a whole run by its run id, `run.runId` on the handle `run.start` returns: that grade shows on the run page and on /projects. A run grade whose id is the run's `episodeKey` also shows, on the run page only. `run_id` is a reserved tag the SDK stamps, not a variable your code has. A grade whose id matches no call or run attaches to nothing. The write is synchronous and local and never throws into your pipeline.
+
 ```ts
-// One call — the inference id rides on every RunResult:
-const result = await rig.run("example.support_summarize", { input, tags: { subject } });
-rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" });
+const run = rig.artifacts.run.start({ pipeline: "support", episodeKey: ticketId, environment: "production" });
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },
+  tags: { subject: customer },
+  stepKey: "summarize",
+});
+run.end("succeeded");
 
-// A whole run: the subject id is the run's run_id (its episodeKey, or its id).
-rig.grade({ kind: "run", id: runAttemptId }, { score: 0.2, comment: "CAGR off by 10x", kind: "human" });
+// one call: the inference id rides on every result
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "deterministic", grader: "schema-check" });
+// the whole run: its id is run.runId from run.start (shows on the run page and on /projects)
+rig.grade({ kind: "run", id: run.runId }, { score: 0, kind: "human", comment: "missed the refund request" });
 ```
-
-- **The subject** is `{ kind: "inference", id }` (`result.meta.inferenceId`),
-  `{ kind: "run", id }` (the run's `run_id` — its `episodeKey`, or its id), or
-  `{ kind: "artifact", id }`. **`opts`** is `{ score, scale?, comment?, kind,
-  source? }` — the `score` in `[0, 1]` is the primitive (up = 1, down = 0), and
-  the pass/fail verdict is derived from it against the route threshold. A subject
-  naming no known inference, run, or artifact attaches to nothing — attribution
-  is never guessed.
+<!-- /BLOCK:GRADE_RUN -->
+
+- **The subject** is `{ kind: "inference", id }` (`result.meta.inferenceId`, or
+  the `inferenceId` `rig.recordCall` returns), `{ kind: "run", id }` (the run's
+  id, `run.runId` from `run.start`; a grade carrying the run's `episodeKey`
+  also shows, on the run page only), or `{ kind: "artifact", id }`. **`opts`**
+  is `{ score, scale?, comment?, kind, grader?, source? }` — the `score` in
+  `[0, 1]` is the primitive (up = 1, down = 0), and the pass/fail verdict is
+  derived from it against the route threshold. A subject naming no known
+  inference, run, or artifact attaches to nothing — attribution is never guessed.
 - **Emit it from a signal you already have.** Most pipelines already decide
   pass/fail somewhere — a schema validator, a downstream accept/reject, a human
   QA gate. Emit `rig.grade` from that (score 1 on a clean pass, 0 on a rejection)
@@ -368,15 +435,40 @@ const caseId = rig.artifacts.artifact.promoteToEval(handle, { task: "report.sect
 
 ```bash
 # Replay the promoted eval cases through variants — the raw/run-based lane.
+# The variants (default, cheap) are read from the route bundle named report.section.
 modelrig bakeoff --from-eval-cases report.section --variants default,cheap
 ```
 
 - `promoteToEval(ref, { task?, corrected?, origin? })` binds the case to a route by
   `task` (from `opts`, else the task the artifact was saved with this session), and
   is also reachable from the console's one-click **"make golden"**.
-- `--route` needs a **declared route**; `--from-eval-cases <task>` is the path for a
-  `runRaw` seam that owns no route. Either way a bake-off **measures** — it never
-  switches what serves a route; a swap is a human-approved PR.
+- `--route` replays **captured route traffic**; `--from-eval-cases <task>` replays
+  the promoted cases instead, so it is the path for a `runRaw` seam with no
+  captured traffic. Both read the variants from a route bundle (`--route`, or the
+  bundle named `<task>`): selecting the cases needs no routes directory, but a
+  case that replays needs that bundle and its `variants:` block. Either way a
+  bake-off **measures** — it never switches what serves a route; never a silent swap — promote it in ModelRig (console or MCP; versioned, reversible) or merge it as a PR if you keep routes in git.
+
+<!-- BLOCK:BAKEOFF -->
+**Prove a cheaper model on your own traffic.** A bake-off replays your own inputs through the variants a route declares and reports the effective cost of conformance for each. `--variants` names entries in the route's `variants:` block (`default` is the route as declared), never model ids, and a variant can only narrow the route's candidates. Set `capture: true` first: replay works on traffic captured after that, not before. With no captured traffic (a run-based pipeline), promote a real /runs output to a golden (the console's make golden, or `promoteToEval`) and use `--from-eval-cases <task>`; its variants come from the route bundle named after the task. A bake-off measures and never switches anything.
+
+```bash
+# 1. in modelrig/routes/example.support_summarize.yaml: capture traffic, and declare
+#    a cheaper variant (narrowing only: its candidates must be in the route's list)
+capture: true
+variants:
+  - name: cheap
+    candidates:
+      - provider: gemini
+        model: gemini-3.1-flash-lite
+
+# 2. after traffic accumulates, compare the route as declared (default) with cheap
+npx modelrig bakeoff --route example.support_summarize --variants default,cheap --replay-last 50
+
+# no captured route traffic? promote a real /runs output to a golden, then
+npx modelrig bakeoff --from-eval-cases example.support_summarize --variants default,cheap
+```
+<!-- /BLOCK:BAKEOFF -->
 
 ## A reusable per-step seam
 
@@ -388,15 +480,18 @@ as diffs to apply — it never edits your source. The seam is equally yours to
 write by hand; here it is, self-contained: 
 
 ```ts
-import { createRig, loadConfigFromEnv, type Rig } from "modelrig";
+import { createRig, loadConfigFromEnv, type RecordCallInput, type Rig } from "modelrig";
 
-const rig: Rig = createRig(loadConfigFromEnv());
+// routesDir: null — the seam needs no route files, so importing it never throws.
+const rig: Rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
 
-/** Save prompt + raw + parsed for one step, link lineage, flag conformance. */
+/** Report the call (tokens → cost), save prompt + raw + parsed, link lineage,
+ * flag conformance. */
 function recordStep(
   stepKey: string,
-  work: { prompt: string; rawText: string; json: unknown; conforms: boolean },
+  work: { prompt: string; rawText: string; json: unknown; conforms: boolean; call?: Omit<RecordCallInput, "stepKey"> },
 ): void {
+  if (work.call !== undefined) rig.recordCall({ ...work.call, stepKey });
   const p = rig.artifacts.artifact.save(work.prompt, { name: `${stepKey}.prompt`, type: "prompt", stepKey });
   const raw = rig.artifacts.artifact.save(work.rawText, { name: `${stepKey}.raw`, type: "raw_response", stepKey });
   const parsed = rig.artifacts.artifact.save(work.json, { name: `${stepKey}.output`, type: "step_output", task: stepKey, stepKey });

package/docs/llms.txt

@@ -4,7 +4,7 @@
 > grounding, and caching measured empirically, with a capability-aware
 > gateway that routes by them. Registry data CC BY 4.0; code Apache-2.0.
 
-_This reference is as of modelrig v0.11.0 (the published npm package). Any surface below that a newer workspace adds is marked "unreleased — workspace only" until its release; nothing is so marked as of this version._
+_This reference is as of modelrig v0.11.1 (the published npm package). Any surface below that a newer workspace adds is marked "unreleased — workspace only" until its release; nothing is so marked as of this version._
 
 ## Docs
 
@@ -60,14 +60,14 @@ _The default ingest host is https://api.modelrig.ai (override with `MODELRIG_ING
 MODELRIG_API_KEY=rig_sk_…
 ```
 
-**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in, once your app has started with the key. It asks GET /v1/whoami which organization the key belongs to and prints the org's name and, in parentheses, its slug; the name must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
 
 ```bash
 npx modelrig doctor
-# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+# [modelrig] key rig_sk_…AbCd → org «Your Org» (your-org) ✓  (ingest https://api.modelrig.ai, …)
 
-# No CLI in that environment? Ask the API directly:
-curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+# No CLI in that environment? Ask the API directly (same host the SDK uses):
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" "${MODELRIG_INGEST_URL:-https://api.modelrig.ai}/v1/whoami"
 ```
 
 **Wrap one pipeline execution in a run.** Every pipeline execution is a run. Wrap each execution in `run.scope` (or `run.start` / `run.end`), name one step per model call with `stepKey`, `artifact.save` the work products, and pass `environment` so the Env column on /runs is filled. Inside the run the SDK stamps `run_id` and `step` on every call. `run_id` is reserved: don't set it yourself.
@@ -103,17 +103,99 @@ await rig.close(); // flushes the exporter — call once at shutdown
 ```text
 1. npx modelrig doctor, from the environment your app runs in.
    no export sink configured → this process does not see MODELRIG_API_KEY: set it, redeploy, open a new shell.
+   malformed key             → the value is truncated, quoted or padded: paste the whole rig_sk_… secret again.
    AUTH FAILED (401 / 403)   → the key is mistyped, revoked or not accepted: mint a key, set it, redeploy, open a new shell.
+   whoami unreachable        → no network path to the ingest host (firewall, proxy, wrong MODELRIG_INGEST_URL): fix the route out, retry.
 2. The org doctor prints is not the org in the badge → wrong org: switch the console, or mint a key in the org you meant.
 3. /inferences has rows but /runs is empty → the calls are not inside run.scope / run.start.
 4. A run is missing a step or a cost → that call is not on a ModelRig lane (see not-captured).
 5. A doc describes something your install lacks → compare `npm ls modelrig` with the version stamp in modelrig.dev/llms.txt.
 ```
 
-**After you change the key on a hosted worker.** A hosted process reads the environment it was started with. After you set or change MODELRIG_API_KEY, wait for the redeploy to finish, then open a new shell: a shell attached before the deploy still has the old environment. Confirm from inside it that the key resolves to the org you expect.
+**After you change the key on a hosted worker.** A hosted process reads the environment it was started with. After you set or change MODELRIG_API_KEY, wait for the redeploy to finish, then open a new shell: a shell attached before the deploy still has the old environment. Confirm from inside it that the key resolves to the org you expect: `--expect-org` takes the org's slug, the part in parentheses in doctor's output.
 
 ```bash
-npx modelrig doctor --expect-org your-org-slug   # exits non-zero if the key belongs to another org
+npx modelrig doctor --expect-org your-org   # exits non-zero if the key belongs to another org
+```
+
+**No routes yet? Build the rig raw-only.** Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with `routesDir: null`: `rig.runRaw`, `rig.runRawStream` and `rig.recordCall` work at once, and `rig.run` / `rig.runBundle` refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.
+
+```ts
+import { createRig, loadConfigFromEnv } from "modelrig";
+
+// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
+// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
+const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
+```
+
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "recorded · unverified", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+
+```ts
+const started = Date.now();
+const resp = await genai.models.generateContent({ model: "gemini-3.1-pro-preview", contents });
+const u = resp.usageMetadata;
+const finishReason = resp.candidates?.[0]?.finishReason;
+
+const rec = rig.recordCall({
+  provider: "gemini",
+  model: "gemini-3.1-pro-preview",
+  usage: {
+    tokensIn: u?.promptTokenCount ?? 0, // total input, cached included
+    tokensOut: (u?.candidatesTokenCount ?? 0) + (u?.thoughtsTokenCount ?? 0),
+    tokensCached: u?.cachedContentTokenCount ?? 0,
+    // optional fields: spread only when the provider reported them
+    ...(u?.thoughtsTokenCount !== undefined ? { reasoningTokens: u.thoughtsTokenCount } : {}),
+  },
+  servedTier: "standard", // the tier that SERVED the call: pass what the provider reports (flex, priority); it is priced at that tier
+  latencyMs: Date.now() - started,
+  stepKey: "summarize", // the step on /runs (inside run.scope / run.start)
+  ...(finishReason !== undefined ? { finishReason } : {}),
+});
+if (rec.inferenceId !== null) rig.grade({ kind: "inference", id: rec.inferenceId }, { score: 1, kind: "deterministic" });
+```
+
+**Attach a grade to a call or a whole run.** A grade is your judgment: a 0–1 score with an optional comment, from a validator, a downstream accept/reject, a human or a model. Grade one call by its inference id (`result.meta.inferenceId`, or the `inferenceId` rig.recordCall returns). Grade a whole run by its run id, `run.runId` on the handle `run.start` returns: that grade shows on the run page and on /projects. A run grade whose id is the run's `episodeKey` also shows, on the run page only. `run_id` is a reserved tag the SDK stamps, not a variable your code has. A grade whose id matches no call or run attaches to nothing. The write is synchronous and local and never throws into your pipeline.
+
+```ts
+const run = rig.artifacts.run.start({ pipeline: "support", episodeKey: ticketId, environment: "production" });
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },
+  tags: { subject: customer },
+  stepKey: "summarize",
+});
+run.end("succeeded");
+
+// one call: the inference id rides on every result
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "deterministic", grader: "schema-check" });
+// the whole run: its id is run.runId from run.start (shows on the run page and on /projects)
+rig.grade({ kind: "run", id: run.runId }, { score: 0, kind: "human", comment: "missed the refund request" });
+```
+
+**Prove a cheaper model on your own traffic.** A bake-off replays your own inputs through the variants a route declares and reports the effective cost of conformance for each. `--variants` names entries in the route's `variants:` block (`default` is the route as declared), never model ids, and a variant can only narrow the route's candidates. Set `capture: true` first: replay works on traffic captured after that, not before. With no captured traffic (a run-based pipeline), promote a real /runs output to a golden (the console's make golden, or `promoteToEval`) and use `--from-eval-cases <task>`; its variants come from the route bundle named after the task. A bake-off measures and never switches anything.
+
+```bash
+# 1. in modelrig/routes/example.support_summarize.yaml: capture traffic, and declare
+#    a cheaper variant (narrowing only: its candidates must be in the route's list)
+capture: true
+variants:
+  - name: cheap
+    candidates:
+      - provider: gemini
+        model: gemini-3.1-flash-lite
+
+# 2. after traffic accumulates, compare the route as declared (default) with cheap
+npx modelrig bakeoff --route example.support_summarize --variants default,cheap --replay-last 50
+
+# no captured route traffic? promote a real /runs output to a golden, then
+npx modelrig bakeoff --from-eval-cases example.support_summarize --variants default,cheap
+```
+
+**Pin the org a deployment must export to.** `--expect-org` and `expectOrg` take the org's slug, not its display name. `modelrig doctor` prints both, as `→ org «Your Org» (your-org) ✓`: the slug is the part in parentheses. In the console, hover the org badge in the header to see it. With `--expect-org <slug>`, doctor exits non-zero when the key belongs to another org, so a deploy check fails loudly instead of exporting somewhere else.
+
+```ts
+// The same check in code: the exporter asks whoami on its first flush and logs
+// ORG MISMATCH on every flush while the key belongs to another org.
+const rig = createRig({ ...loadConfigFromEnv(), expectOrg: "your-org" });
 ```
 
 ## Capabilities (the full surface)
@@ -135,7 +217,7 @@ npx modelrig doctor --expect-org your-org-slug   # exits non-zero if the key bel
 - Rates below are sampled statistics with confidence intervals in the source data — do not quote them without their n and date.
 - For PRICING, do not recall list prices — read them: the leaderboard (https://modelrig.dev/leaderboard.html) ranks models by effective cost per 1,000 schema-conformant outputs (list price ≠ what conformance costs), and each model's per-token rates ride the registry JSON with as_of stamps. Quote the effective cost, not the sticker rate.
 - Attach a GRADE to what you run — capture records what ran, a grade records whether it was good: `rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" })` (or `{ kind: "run", id: run_id }` for a whole run) from a signal you already have (a validator, a downstream accept/reject, a human QA gate). A grade is a 0–1 score with an optional comment; up = 1, down = 0. It is a synchronous local write, exporter-mirrored, and never blocks or throws. A route with ten attached grades reaches Rung 1 on /coverage; a route with no schema to conform to (a prose output) gains proposal confidence only this way. From a non-SDK surface, `POST /v1/grade` with a grade-scoped `rig_sk_` key. See https://modelrig.dev/grade-protocol.html.
-- Prove a cheaper model on YOUR own traffic on the lane that fits: a DECLARED route uses `modelrig bakeoff --route <route> --replay-last N`; a raw/run-based pipeline (a `rig.runRaw` seam, no routes) promotes a real /runs output to a golden (`promoteToEval`, or the console's one-click "make golden") and runs `modelrig bakeoff --from-eval-cases <task> --variants default,<challenger>`. A bake-off measures; it never switches what serves a route — a swap is a human-approved PR. See https://modelrig.dev/bakeoffs.html.
+- Prove a cheaper model on YOUR own traffic on the lane that fits: a DECLARED route uses `modelrig bakeoff --route <route> --replay-last N`; a raw/run-based pipeline (a `rig.runRaw` seam, no routes) promotes a real /runs output to a golden (`promoteToEval`, or the console's one-click "make golden") and runs `modelrig bakeoff --from-eval-cases <task> --variants default,<challenger>`. A bake-off measures; it never switches what serves a route — never a silent swap — promote it in ModelRig (console or MCP; versioned, reversible) or merge it as a PR if you keep routes in git. See https://modelrig.dev/bakeoffs.html.
 
 ## Ask this data directly (MCP)