modelrig 0.10.0 → 0.11.1

package/docs/how-it-fits.md

@@ -12,7 +12,7 @@ What is retained beyond that is your choice, per route, and there are two lanes.
 
 **Pure router** (capture off). Nothing retained. Real, supported, selectable, and the honest answer for a ZDR-strict or data-residency-bound workload.
 
-**Optimization on** (recommended). Your traffic is what makes the routing smarter and the bill smaller. **What is kept:** per-inference telemetry rows, and bounded review samples for the bake-offs you create. **Where:** your organization, row-scoped by the database, not by our application code. **How you leave:** see it all in the console, delete it, export it, take your routes and go — they were always YAML in your git. **What you get back:** community priors on your very first bake-off and RigIndex-grade evidence on your own routes — at the same flat 2% either way; the corpus earns its place by being useful, not by being a discount.
+**Optimization on** (recommended). Your traffic is what makes the routing smarter and the bill smaller. **What is kept:** per-inference telemetry rows, and bounded review samples for the bake-offs you create. **Where:** your organization, row-scoped by the database, not by our application code. **How you leave:** see it all in the console and export it; never a silent swap — promote a change in ModelRig (console or MCP; versioned, reversible) or merge it as a PR if you keep routes in git. **What you get back:** community priors on your very first bake-off and RigIndex-grade evidence on your own routes — at the same flat 2% either way; the corpus earns its place by being useful, not by being a discount.
 
 Captures power **bake-offs**: replay your own recent traffic through route variants — a cheaper model, a different prompt — and get the effective cost of conformance for each, with confidence intervals. A winning variant is a proposal for you to apply in your YAML; nothing swaps models automatically.
 

package/docs/instrumentation.md

@@ -37,8 +37,6 @@ path — never blurred with the telemetry axis here.
 # set_org_settings MCP tool), not this env var.
 export MODELRIG_ARTIFACTS=1                  # local-only namespace toggle: force ON
                                              #   with =1 on a rig with no control plane
-export MODELRIG_ARTIFACT_STEPS=step_a,step_b   # optional: instrument a subset
-                                            # (unset/empty = every instrumented step)
 ```
 
 Rows land in the local telemetry buffer and export to the control plane through
@@ -54,38 +52,60 @@ agent-driven setup should plan that one handoff rather than stall on it.
 The namespace hangs off `rig.artifacts`. Two moving parts: a **run context**
 around the whole pipeline, and a **per-step save** of the work products.
 
+No routes yet? Observing needs none. Build the rig raw-only:
+
+<!-- BLOCK:RAW_RIG -->
+**No routes yet? Build the rig raw-only.** Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with `routesDir: null`: `rig.runRaw`, `rig.runRawStream` and `rig.recordCall` work at once, and `rig.run` / `rig.runBundle` refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.
+
 ```ts
 import { createRig, loadConfigFromEnv } from "modelrig";
 
-const rig = createRig(loadConfigFromEnv());
+// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
+// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
+const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
+```
+<!-- /BLOCK:RAW_RIG -->
 
+Then wrap the pipeline. Here each step's model call stays your own direct call,
+reported with `rig.recordCall` so the step carries tokens and a cost:
+
+```ts
 // 1. Open a run context around the pipeline. episodeKey bridges grades;
 //    it is a grouping key, not a unique id — a natural choice is your own
 //    per-run attempt id. environment fills the Env column on /runs.
 const run = rig.artifacts.run.start({
   pipeline: "my-pipeline",
   episodeKey: runAttemptId,
-  environment: process.env.NODE_ENV,   // e.g. "production", "staging", "test"
+  environment: process.env.NODE_ENV ?? "development", // e.g. "production", "staging", "test"
 });
 try {
   for (const step of steps) {
     const prompt = assemble(step);
-    const result = await callModel(prompt);          // your own model call
+    const started = Date.now();
+    const result = await callModel(prompt);          // your own direct model call
+
+    // 2. Report the call: tokens → cost on the step and on /inferences.
+    rig.recordCall({
+      provider: "openai",
+      model: "gpt-5.4-mini",
+      usage: { tokensIn: result.usage.inputTokens, tokensOut: result.usage.outputTokens },
+      latencyMs: Date.now() - started,
+      stepKey: step.key,
+    });
 
-    // 2. Save the step's artifacts. stepKey groups them in the console's
+    // 3. Save the step's artifacts. stepKey groups them in the console's
     //    chain view; task carries the semantic task name.
-    const p = rig.artifacts.artifact.save(prompt, { name: `${step}.prompt`, type: "prompt", stepKey: step });
-    const raw = rig.artifacts.artifact.save(result.rawText, { name: `${step}.raw`, type: "raw_response", stepKey: step });
+    const p = rig.artifacts.artifact.save(prompt, { name: `${step.key}.prompt`, type: "prompt", stepKey: step.key });
+    const raw = rig.artifacts.artifact.save(result.rawText, { name: `${step.key}.raw`, type: "raw_response", stepKey: step.key });
     const parsed = rig.artifacts.artifact.save(result.json, {
-      name: `${step}.output`, type: "step_output", task: step, stepKey: step, schema: step.schema,
+      name: `${step.key}.output`, type: "step_output", task: step.key, stepKey: step.key, schema: step.schema,
     });
 
-    // 3. Lineage: the parsed output derives from the prompt + raw; this step's
-    //    prompt consumed the previous step's output.
+    // 4. Lineage: the parsed output derives from the prompt + raw.
     if (p && parsed) rig.artifacts.artifact.link(parsed, p, "derived_from");
     if (raw && parsed) rig.artifacts.artifact.link(parsed, raw, "derived_from");
 
-    // 4. Grades (optional, deterministic): flag conformance, degraded
+    // 5. Grades (optional, deterministic): flag conformance, degraded
     //    provenance, anything you can decide without an LLM judge.
     if (parsed) rig.grade({ kind: "artifact", id: parsed.id }, {
       score: conforms(result.json) ? 1 : 0, kind: "deterministic", grader: "schema-check",
@@ -96,8 +116,13 @@ try {
   run.end("failed");
   throw err;
 }
+await rig.close(); // once, at shutdown
 ```
 
+Here `step` is your own `{ key, schema }` record, and `callModel` returns the
+provider's text, parsed JSON and usage. A call routed through `rig.run` or
+`rig.runRaw` records its own step, so it needs no `recordCall`.
+
 For **concurrent** runs in one process, use `rig.artifacts.run.scope(opts, fn)`
 instead of `start`/`end` — each scope keeps its own ambient context so
 overlapping runs never cross-attribute. `run.start` uses `enterWith`, which is
@@ -111,9 +136,55 @@ right for one run at a time on a request/async path.
 - A **step** is an execution row. `rig.run()` auto-creates a **ground-truth**
   step (its inference is real, so cost and latency are measured) when it runs
   inside a run context. A stage whose model call does *not* go through the rig
-  produces artifacts with `step_key` set and **no step row** — which is correct
-  and honest. Never route a stage through the rig just to mint a step row; the
-  console renders "cost n/a" for un-routed steps rather than inventing a number.
+  and is not reported with `rig.recordCall` produces artifacts with `step_key`
+  set and **no step row**, which is correct and honest: the console renders
+  "cost n/a" for that step rather than inventing a number. Report the call
+  (below) and the step gets its tokens and cost.
+
+### Record a call your code made itself (`rig.recordCall`)
+
+_Needs `modelrig` 0.11.0 or newer._
+
+If a stage's model call has to stay a direct provider call, report it instead:
+`rig.recordCall({ provider, model, usage: { tokensIn, tokensOut, tokensCached }, servedTier, latencyMs, stepKey })`
+right after the call returns. Inside a run context it records a step row for
+that `stepKey` (stored as `ground-truth`, the only joined-inference value),
+joined to the recorded inference, so the step gets a cost instead of "cost n/a".
+The console shows that step and its call as **recorded · unverified**, never as
+measured:
+ModelRig neither dispatched nor validated the call, and the tokens are the ones
+you reported. It is synchronous and never throws (bad input warns and returns
+`inferenceId: null`); the returned `inferenceId` is gradeable with `rig.grade`.
+Full contract, usage convention and cost notes:
+[Not captured → record them](not-captured.md#calls-your-code-makes-itself-record-them).
+
+<!-- BLOCK:RECORD_CALL -->
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "recorded · unverified", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+
+```ts
+const started = Date.now();
+const resp = await genai.models.generateContent({ model: "gemini-3.1-pro-preview", contents });
+const u = resp.usageMetadata;
+const finishReason = resp.candidates?.[0]?.finishReason;
+
+const rec = rig.recordCall({
+  provider: "gemini",
+  model: "gemini-3.1-pro-preview",
+  usage: {
+    tokensIn: u?.promptTokenCount ?? 0, // total input, cached included
+    tokensOut: (u?.candidatesTokenCount ?? 0) + (u?.thoughtsTokenCount ?? 0),
+    tokensCached: u?.cachedContentTokenCount ?? 0,
+    // optional fields: spread only when the provider reported them
+    ...(u?.thoughtsTokenCount !== undefined ? { reasoningTokens: u.thoughtsTokenCount } : {}),
+  },
+  servedTier: "standard", // the tier that SERVED the call: pass what the provider reports (flex, priority); it is priced at that tier
+  latencyMs: Date.now() - started,
+  stepKey: "summarize", // the step on /runs (inside run.scope / run.start)
+  ...(finishReason !== undefined ? { finishReason } : {}),
+});
+if (rec.inferenceId !== null) rig.grade({ kind: "inference", id: rec.inferenceId }, { score: 1, kind: "deterministic" });
+```
+<!-- /BLOCK:RECORD_CALL -->
 
 ## Tags, dimensions, and the graph
 
@@ -220,7 +291,9 @@ const { bundle: runnable, arm } = resolveCallConfigForRun(bundle, resolved, { in
 
 // Thread the arm onto the run so it lands on inferences.meta.arm.
 await rig.artifacts.run.scope({ pipeline: "report", episodeKey: runId }, async () => {
-  const out = await rig.runBundle(runnable, { input, tags: { subject, feature }, arm, stepKey });
+  // arm is string | null; RunOptions.arm is optional, so pass it only when drawn
+  // (a bare `arm` fails under exactOptionalPropertyTypes).
+  const out = await rig.runBundle(runnable, { input, tags: { subject, feature }, stepKey, ...(arm !== null ? { arm } : {}) });
   rig.artifacts.artifact.save(out.output, { name: "section.output", type: "step_output" });
 });
 ```
@@ -277,14 +350,14 @@ orgs and the console picker: [Orgs and keys](https://modelrig.dev/orgs-and-keys.
 Check first that the console is showing the org your key exports to:
 
 <!-- BLOCK:CONFIRM_ORG -->
-**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in, once your app has started with the key. It asks GET /v1/whoami which organization the key belongs to and prints the org's name and, in parentheses, its slug; the name must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
 
 ```bash
 npx modelrig doctor
-# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+# [modelrig] key rig_sk_…AbCd → org «Your Org» (your-org) ✓  (ingest https://api.modelrig.ai, …)
 
-# No CLI in that environment? Ask the API directly:
-curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+# No CLI in that environment? Ask the API directly (same host the SDK uses):
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" "${MODELRIG_INGEST_URL:-https://api.modelrig.ai}/v1/whoami"
 ```
 <!-- /BLOCK:CONFIRM_ORG -->
 
@@ -302,22 +375,33 @@ an AI — attaches to a subject: one model call (an **inference**), a whole run
 [optimization coverage](grade-protocol.md) is built from — the evidence level
 that decides how much confidence a swap proposal carries.
 
+<!-- BLOCK:GRADE_RUN -->
+**Attach a grade to a call or a whole run.** A grade is your judgment: a 0–1 score with an optional comment, from a validator, a downstream accept/reject, a human or a model. Grade one call by its inference id (`result.meta.inferenceId`, or the `inferenceId` rig.recordCall returns). Grade a whole run by its run id, `run.runId` on the handle `run.start` returns: that grade shows on the run page and on /projects. A run grade whose id is the run's `episodeKey` also shows, on the run page only. `run_id` is a reserved tag the SDK stamps, not a variable your code has. A grade whose id matches no call or run attaches to nothing. The write is synchronous and local and never throws into your pipeline.
+
 ```ts
-// One call — the inference id rides on every RunResult:
-const result = await rig.run("example.support_summarize", { input, tags: { subject } });
-rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" });
+const run = rig.artifacts.run.start({ pipeline: "support", episodeKey: ticketId, environment: "production" });
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },
+  tags: { subject: customer },
+  stepKey: "summarize",
+});
+run.end("succeeded");
 
-// A whole run: the subject id is the run's run_id (its episodeKey, or its id).
-rig.grade({ kind: "run", id: runAttemptId }, { score: 0.2, comment: "CAGR off by 10x", kind: "human" });
+// one call: the inference id rides on every result
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "deterministic", grader: "schema-check" });
+// the whole run: its id is run.runId from run.start (shows on the run page and on /projects)
+rig.grade({ kind: "run", id: run.runId }, { score: 0, kind: "human", comment: "missed the refund request" });
 ```
-
-- **The subject** is `{ kind: "inference", id }` (`result.meta.inferenceId`),
-  `{ kind: "run", id }` (the run's `run_id` — its `episodeKey`, or its id), or
-  `{ kind: "artifact", id }`. **`opts`** is `{ score, scale?, comment?, kind,
-  source? }` — the `score` in `[0, 1]` is the primitive (up = 1, down = 0), and
-  the pass/fail verdict is derived from it against the route threshold. A subject
-  naming no known inference, run, or artifact attaches to nothing — attribution
-  is never guessed.
+<!-- /BLOCK:GRADE_RUN -->
+
+- **The subject** is `{ kind: "inference", id }` (`result.meta.inferenceId`, or
+  the `inferenceId` `rig.recordCall` returns), `{ kind: "run", id }` (the run's
+  id, `run.runId` from `run.start`; a grade carrying the run's `episodeKey`
+  also shows, on the run page only), or `{ kind: "artifact", id }`. **`opts`**
+  is `{ score, scale?, comment?, kind, grader?, source? }` — the `score` in
+  `[0, 1]` is the primitive (up = 1, down = 0), and the pass/fail verdict is
+  derived from it against the route threshold. A subject naming no known
+  inference, run, or artifact attaches to nothing — attribution is never guessed.
 - **Emit it from a signal you already have.** Most pipelines already decide
   pass/fail somewhere — a schema validator, a downstream accept/reject, a human
   QA gate. Emit `rig.grade` from that (score 1 on a clean pass, 0 on a rejection)
@@ -351,15 +435,40 @@ const caseId = rig.artifacts.artifact.promoteToEval(handle, { task: "report.sect
 
 ```bash
 # Replay the promoted eval cases through variants — the raw/run-based lane.
+# The variants (default, cheap) are read from the route bundle named report.section.
 modelrig bakeoff --from-eval-cases report.section --variants default,cheap
 ```
 
 - `promoteToEval(ref, { task?, corrected?, origin? })` binds the case to a route by
   `task` (from `opts`, else the task the artifact was saved with this session), and
   is also reachable from the console's one-click **"make golden"**.
-- `--route` needs a **declared route**; `--from-eval-cases <task>` is the path for a
-  `runRaw` seam that owns no route. Either way a bake-off **measures** — it never
-  switches what serves a route; a swap is a human-approved PR.
+- `--route` replays **captured route traffic**; `--from-eval-cases <task>` replays
+  the promoted cases instead, so it is the path for a `runRaw` seam with no
+  captured traffic. Both read the variants from a route bundle (`--route`, or the
+  bundle named `<task>`): selecting the cases needs no routes directory, but a
+  case that replays needs that bundle and its `variants:` block. Either way a
+  bake-off **measures** — it never switches what serves a route; never a silent swap — promote it in ModelRig (console or MCP; versioned, reversible) or merge it as a PR if you keep routes in git.
+
+<!-- BLOCK:BAKEOFF -->
+**Prove a cheaper model on your own traffic.** A bake-off replays your own inputs through the variants a route declares and reports the effective cost of conformance for each. `--variants` names entries in the route's `variants:` block (`default` is the route as declared), never model ids, and a variant can only narrow the route's candidates. Set `capture: true` first: replay works on traffic captured after that, not before. With no captured traffic (a run-based pipeline), promote a real /runs output to a golden (the console's make golden, or `promoteToEval`) and use `--from-eval-cases <task>`; its variants come from the route bundle named after the task. A bake-off measures and never switches anything.
+
+```bash
+# 1. in modelrig/routes/example.support_summarize.yaml: capture traffic, and declare
+#    a cheaper variant (narrowing only: its candidates must be in the route's list)
+capture: true
+variants:
+  - name: cheap
+    candidates:
+      - provider: gemini
+        model: gemini-3.1-flash-lite
+
+# 2. after traffic accumulates, compare the route as declared (default) with cheap
+npx modelrig bakeoff --route example.support_summarize --variants default,cheap --replay-last 50
+
+# no captured route traffic? promote a real /runs output to a golden, then
+npx modelrig bakeoff --from-eval-cases example.support_summarize --variants default,cheap
+```
+<!-- /BLOCK:BAKEOFF -->
 
 ## A reusable per-step seam
 
@@ -371,15 +480,18 @@ as diffs to apply — it never edits your source. The seam is equally yours to
 write by hand; here it is, self-contained: 
 
 ```ts
-import { createRig, loadConfigFromEnv, type Rig } from "modelrig";
+import { createRig, loadConfigFromEnv, type RecordCallInput, type Rig } from "modelrig";
 
-const rig: Rig = createRig(loadConfigFromEnv());
+// routesDir: null — the seam needs no route files, so importing it never throws.
+const rig: Rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
 
-/** Save prompt + raw + parsed for one step, link lineage, flag conformance. */
+/** Report the call (tokens → cost), save prompt + raw + parsed, link lineage,
+ * flag conformance. */
 function recordStep(
   stepKey: string,
-  work: { prompt: string; rawText: string; json: unknown; conforms: boolean },
+  work: { prompt: string; rawText: string; json: unknown; conforms: boolean; call?: Omit<RecordCallInput, "stepKey"> },
 ): void {
+  if (work.call !== undefined) rig.recordCall({ ...work.call, stepKey });
   const p = rig.artifacts.artifact.save(work.prompt, { name: `${stepKey}.prompt`, type: "prompt", stepKey });
   const raw = rig.artifacts.artifact.save(work.rawText, { name: `${stepKey}.raw`, type: "raw_response", stepKey });
   const parsed = rig.artifacts.artifact.save(work.json, { name: `${stepKey}.output`, type: "step_output", task: stepKey, stepKey });

package/docs/llms.txt

@@ -4,7 +4,7 @@
 > grounding, and caching measured empirically, with a capability-aware
 > gateway that routes by them. Registry data CC BY 4.0; code Apache-2.0.
 
-_This reference is as of modelrig v0.10.0 (the published npm package). Any surface below that a newer workspace adds is marked "unreleased — workspace only" until its release; nothing is so marked as of this version._
+_This reference is as of modelrig v0.11.1 (the published npm package). Any surface below that a newer workspace adds is marked "unreleased — workspace only" until its release; nothing is so marked as of this version._
 
 ## Docs
 
@@ -60,14 +60,14 @@ _The default ingest host is https://api.modelrig.ai (override with `MODELRIG_ING
 MODELRIG_API_KEY=rig_sk_…
 ```
 
-**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in, once your app has started with the key. It asks GET /v1/whoami which organization the key belongs to and prints the org's name and, in parentheses, its slug; the name must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
 
 ```bash
 npx modelrig doctor
-# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+# [modelrig] key rig_sk_…AbCd → org «Your Org» (your-org) ✓  (ingest https://api.modelrig.ai, …)
 
-# No CLI in that environment? Ask the API directly:
-curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+# No CLI in that environment? Ask the API directly (same host the SDK uses):
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" "${MODELRIG_INGEST_URL:-https://api.modelrig.ai}/v1/whoami"
 ```
 
 **Wrap one pipeline execution in a run.** Every pipeline execution is a run. Wrap each execution in `run.scope` (or `run.start` / `run.end`), name one step per model call with `stepKey`, `artifact.save` the work products, and pass `environment` so the Env column on /runs is filled. Inside the run the SDK stamps `run_id` and `step` on every call. `run_id` is reserved: don't set it yourself.
@@ -96,29 +96,112 @@ await rig.artifacts.run.scope(
 await rig.close(); // flushes the exporter — call once at shutdown
 ```
 
-**What ModelRig does not see.** ModelRig records the calls your code makes through the SDK (`rig.run`, `rig.runBundle`, `rig.runRaw`, `rig.runRawStream`) or the gateway, and nothing else. Five things stay invisible or unattached: (1) direct provider SDK calls, with or without a JSON schema; (2) streams you read from a provider client yourself; (3) a fallback your code makes after the raw lane refuses a `responseSchema` it cannot enforce (deepseek, deepinfra, fireworks and zai fail closed with `invariant_violation`); (4) calls made outside `run.scope` / `run.start`, which land on /inferences but on no run; (5) embeddings. Details and the console tells: https://modelrig.dev/not-captured.html.
+**What ModelRig does not see unless you record it.** ModelRig records the calls your code makes through the SDK (`rig.run`, `rig.runBundle`, `rig.runRaw`, `rig.runRawStream`) or the gateway, plus any call your code makes itself and reports with `rig.recordCall`. Nothing else is seen. Five things stay invisible or unattached unless you record them: (1) direct provider SDK calls, with or without a JSON schema; (2) streams you read from a provider client yourself; (3) a fallback your code makes after the raw lane refuses a `responseSchema` it cannot enforce (deepseek, deepinfra, fireworks and zai fail closed with `invariant_violation`); (4) calls made outside `run.scope` / `run.start`, which land on /inferences but on no run; (5) embeddings. A recorded call shows as "recorded · unverified": ModelRig neither dispatched nor validated it, and its tokens are the numbers you reported. Details and the console tells: https://modelrig.dev/not-captured.html.
 
 **The console is empty: five checks, in order.** If /runs (or /inferences) is empty for the org in the header badge, work these checks in order. The full tree is https://modelrig.dev/troubleshooting.html.
 
 ```text
 1. npx modelrig doctor, from the environment your app runs in.
    no export sink configured → this process does not see MODELRIG_API_KEY: set it, redeploy, open a new shell.
+   malformed key             → the value is truncated, quoted or padded: paste the whole rig_sk_… secret again.
    AUTH FAILED (401 / 403)   → the key is mistyped, revoked or not accepted: mint a key, set it, redeploy, open a new shell.
+   whoami unreachable        → no network path to the ingest host (firewall, proxy, wrong MODELRIG_INGEST_URL): fix the route out, retry.
 2. The org doctor prints is not the org in the badge → wrong org: switch the console, or mint a key in the org you meant.
 3. /inferences has rows but /runs is empty → the calls are not inside run.scope / run.start.
 4. A run is missing a step or a cost → that call is not on a ModelRig lane (see not-captured).
 5. A doc describes something your install lacks → compare `npm ls modelrig` with the version stamp in modelrig.dev/llms.txt.
 ```
 
-**After you change the key on a hosted worker.** A hosted process reads the environment it was started with. After you set or change MODELRIG_API_KEY, wait for the redeploy to finish, then open a new shell: a shell attached before the deploy still has the old environment. Confirm from inside it that the key resolves to the org you expect.
+**After you change the key on a hosted worker.** A hosted process reads the environment it was started with. After you set or change MODELRIG_API_KEY, wait for the redeploy to finish, then open a new shell: a shell attached before the deploy still has the old environment. Confirm from inside it that the key resolves to the org you expect: `--expect-org` takes the org's slug, the part in parentheses in doctor's output.
 
 ```bash
-npx modelrig doctor --expect-org your-org-slug   # exits non-zero if the key belongs to another org
+npx modelrig doctor --expect-org your-org   # exits non-zero if the key belongs to another org
+```
+
+**No routes yet? Build the rig raw-only.** Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with `routesDir: null`: `rig.runRaw`, `rig.runRawStream` and `rig.recordCall` work at once, and `rig.run` / `rig.runBundle` refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.
+
+```ts
+import { createRig, loadConfigFromEnv } from "modelrig";
+
+// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
+// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
+const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
+```
+
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "recorded · unverified", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+
+```ts
+const started = Date.now();
+const resp = await genai.models.generateContent({ model: "gemini-3.1-pro-preview", contents });
+const u = resp.usageMetadata;
+const finishReason = resp.candidates?.[0]?.finishReason;
+
+const rec = rig.recordCall({
+  provider: "gemini",
+  model: "gemini-3.1-pro-preview",
+  usage: {
+    tokensIn: u?.promptTokenCount ?? 0, // total input, cached included
+    tokensOut: (u?.candidatesTokenCount ?? 0) + (u?.thoughtsTokenCount ?? 0),
+    tokensCached: u?.cachedContentTokenCount ?? 0,
+    // optional fields: spread only when the provider reported them
+    ...(u?.thoughtsTokenCount !== undefined ? { reasoningTokens: u.thoughtsTokenCount } : {}),
+  },
+  servedTier: "standard", // the tier that SERVED the call: pass what the provider reports (flex, priority); it is priced at that tier
+  latencyMs: Date.now() - started,
+  stepKey: "summarize", // the step on /runs (inside run.scope / run.start)
+  ...(finishReason !== undefined ? { finishReason } : {}),
+});
+if (rec.inferenceId !== null) rig.grade({ kind: "inference", id: rec.inferenceId }, { score: 1, kind: "deterministic" });
+```
+
+**Attach a grade to a call or a whole run.** A grade is your judgment: a 0–1 score with an optional comment, from a validator, a downstream accept/reject, a human or a model. Grade one call by its inference id (`result.meta.inferenceId`, or the `inferenceId` rig.recordCall returns). Grade a whole run by its run id, `run.runId` on the handle `run.start` returns: that grade shows on the run page and on /projects. A run grade whose id is the run's `episodeKey` also shows, on the run page only. `run_id` is a reserved tag the SDK stamps, not a variable your code has. A grade whose id matches no call or run attaches to nothing. The write is synchronous and local and never throws into your pipeline.
+
+```ts
+const run = rig.artifacts.run.start({ pipeline: "support", episodeKey: ticketId, environment: "production" });
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },
+  tags: { subject: customer },
+  stepKey: "summarize",
+});
+run.end("succeeded");
+
+// one call: the inference id rides on every result
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "deterministic", grader: "schema-check" });
+// the whole run: its id is run.runId from run.start (shows on the run page and on /projects)
+rig.grade({ kind: "run", id: run.runId }, { score: 0, kind: "human", comment: "missed the refund request" });
+```
+
+**Prove a cheaper model on your own traffic.** A bake-off replays your own inputs through the variants a route declares and reports the effective cost of conformance for each. `--variants` names entries in the route's `variants:` block (`default` is the route as declared), never model ids, and a variant can only narrow the route's candidates. Set `capture: true` first: replay works on traffic captured after that, not before. With no captured traffic (a run-based pipeline), promote a real /runs output to a golden (the console's make golden, or `promoteToEval`) and use `--from-eval-cases <task>`; its variants come from the route bundle named after the task. A bake-off measures and never switches anything.
+
+```bash
+# 1. in modelrig/routes/example.support_summarize.yaml: capture traffic, and declare
+#    a cheaper variant (narrowing only: its candidates must be in the route's list)
+capture: true
+variants:
+  - name: cheap
+    candidates:
+      - provider: gemini
+        model: gemini-3.1-flash-lite
+
+# 2. after traffic accumulates, compare the route as declared (default) with cheap
+npx modelrig bakeoff --route example.support_summarize --variants default,cheap --replay-last 50
+
+# no captured route traffic? promote a real /runs output to a golden, then
+npx modelrig bakeoff --from-eval-cases example.support_summarize --variants default,cheap
+```
+
+**Pin the org a deployment must export to.** `--expect-org` and `expectOrg` take the org's slug, not its display name. `modelrig doctor` prints both, as `→ org «Your Org» (your-org) ✓`: the slug is the part in parentheses. In the console, hover the org badge in the header to see it. With `--expect-org <slug>`, doctor exits non-zero when the key belongs to another org, so a deploy check fails loudly instead of exporting somewhere else.
+
+```ts
+// The same check in code: the exporter asks whoami on its first flush and logs
+// ORG MISMATCH on every flush while the key belongs to another org.
+const rig = createRig({ ...loadConfigFromEnv(), expectOrg: "your-org" });
 ```
 
 ## Capabilities (the full surface)
 
 - **Runs & projects** — the run → step → artifact graph at /runs, grouped by tag at /projects (the standard path above): https://modelrig.dev/instrumentation.html.
+- **Recorded calls (`rig.recordCall`, modelrig ≥ 0.11.0)** — a provider call your code makes itself (no lane fits) is reported after it returns: one row, priced at list for the served tier from the tokens you report, joined to the run/step, gradeable, badged "recorded · unverified" (ModelRig neither dispatched nor validated it), never in routed/conformance math: https://modelrig.dev/not-captured.html.
 - **Tenants & statements** — a `subject` tag or project-scoped key becomes a per-customer tenant with a derived statement (attempted, verified, billable, true cost; never content or another org's data): https://modelrig.dev/tenants.html.
 - **Published receipts** — a signed, login-free receipt of a bake-off or route at /r/<id>, verifiable by anyone at /v1/verify/<id>; internal names aliased, content never leaves: https://modelrig.dev/receipts.html.
 - **Analytics query API** — a self-describing, read-only query surface over your own telemetry (`GET /v1/analytics/meta`, `POST /v1/analytics/query`): an allowlisted SELECT-only grammar that returns metadata only (enums, numbers, hashes), never content: https://modelrig.dev/optimization-loop.html.
@@ -134,7 +217,7 @@ npx modelrig doctor --expect-org your-org-slug   # exits non-zero if the key bel
 - Rates below are sampled statistics with confidence intervals in the source data — do not quote them without their n and date.
 - For PRICING, do not recall list prices — read them: the leaderboard (https://modelrig.dev/leaderboard.html) ranks models by effective cost per 1,000 schema-conformant outputs (list price ≠ what conformance costs), and each model's per-token rates ride the registry JSON with as_of stamps. Quote the effective cost, not the sticker rate.
 - Attach a GRADE to what you run — capture records what ran, a grade records whether it was good: `rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" })` (or `{ kind: "run", id: run_id }` for a whole run) from a signal you already have (a validator, a downstream accept/reject, a human QA gate). A grade is a 0–1 score with an optional comment; up = 1, down = 0. It is a synchronous local write, exporter-mirrored, and never blocks or throws. A route with ten attached grades reaches Rung 1 on /coverage; a route with no schema to conform to (a prose output) gains proposal confidence only this way. From a non-SDK surface, `POST /v1/grade` with a grade-scoped `rig_sk_` key. See https://modelrig.dev/grade-protocol.html.
-- Prove a cheaper model on YOUR own traffic on the lane that fits: a DECLARED route uses `modelrig bakeoff --route <route> --replay-last N`; a raw/run-based pipeline (a `rig.runRaw` seam, no routes) promotes a real /runs output to a golden (`promoteToEval`, or the console's one-click "make golden") and runs `modelrig bakeoff --from-eval-cases <task> --variants default,<challenger>`. A bake-off measures; it never switches what serves a route — a swap is a human-approved PR. See https://modelrig.dev/bakeoffs.html.
+- Prove a cheaper model on YOUR own traffic on the lane that fits: a DECLARED route uses `modelrig bakeoff --route <route> --replay-last N`; a raw/run-based pipeline (a `rig.runRaw` seam, no routes) promotes a real /runs output to a golden (`promoteToEval`, or the console's one-click "make golden") and runs `modelrig bakeoff --from-eval-cases <task> --variants default,<challenger>`. A bake-off measures; it never switches what serves a route — never a silent swap — promote it in ModelRig (console or MCP; versioned, reversible) or merge it as a PR if you keep routes in git. See https://modelrig.dev/bakeoffs.html.
 
 ## Ask this data directly (MCP)