modelrig 0.11.0 → 0.12.0

package/docs/instrumentation.md

@@ -37,8 +37,6 @@ path — never blurred with the telemetry axis here.
 # set_org_settings MCP tool), not this env var.
 export MODELRIG_ARTIFACTS=1                  # local-only namespace toggle: force ON
                                              #   with =1 on a rig with no control plane
-export MODELRIG_ARTIFACT_STEPS=step_a,step_b   # optional: instrument a subset
-                                            # (unset/empty = every instrumented step)
 ```
 
 Rows land in the local telemetry buffer and export to the control plane through
@@ -54,38 +52,60 @@ agent-driven setup should plan that one handoff rather than stall on it.
 The namespace hangs off `rig.artifacts`. Two moving parts: a **run context**
 around the whole pipeline, and a **per-step save** of the work products.
 
+No routes yet? Observing needs none. Build the rig raw-only:
+
+<!-- BLOCK:RAW_RIG -->
+**No routes yet? Build the rig raw-only.** Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with `routesDir: null`: `rig.runRaw`, `rig.runRawStream` and `rig.recordCall` work at once, and `rig.run` / `rig.runBundle` refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.
+
 ```ts
 import { createRig, loadConfigFromEnv } from "modelrig";
 
-const rig = createRig(loadConfigFromEnv());
+// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
+// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
+const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
+```
+<!-- /BLOCK:RAW_RIG -->
+
+Then wrap the pipeline. Here each step's model call stays your own direct call,
+reported with `rig.recordCall` so the step carries tokens and a cost:
 
+```ts
 // 1. Open a run context around the pipeline. episodeKey bridges grades;
 //    it is a grouping key, not a unique id — a natural choice is your own
 //    per-run attempt id. environment fills the Env column on /runs.
 const run = rig.artifacts.run.start({
   pipeline: "my-pipeline",
   episodeKey: runAttemptId,
-  environment: process.env.NODE_ENV,   // e.g. "production", "staging", "test"
+  environment: process.env.NODE_ENV ?? "development", // e.g. "production", "staging", "test"
 });
 try {
   for (const step of steps) {
     const prompt = assemble(step);
-    const result = await callModel(prompt);          // your own model call
+    const started = Date.now();
+    const result = await callModel(prompt);          // your own direct model call
+
+    // 2. Report the call: tokens → cost on the step and on /inferences.
+    rig.recordCall({
+      provider: "openai",
+      model: "gpt-5.4-mini",
+      usage: { tokensIn: result.usage.inputTokens, tokensOut: result.usage.outputTokens },
+      latencyMs: Date.now() - started,
+      stepKey: step.key,
+    });
 
-    // 2. Save the step's artifacts. stepKey groups them in the console's
+    // 3. Save the step's artifacts. stepKey groups them in the console's
     //    chain view; task carries the semantic task name.
-    const p = rig.artifacts.artifact.save(prompt, { name: `${step}.prompt`, type: "prompt", stepKey: step });
-    const raw = rig.artifacts.artifact.save(result.rawText, { name: `${step}.raw`, type: "raw_response", stepKey: step });
+    const p = rig.artifacts.artifact.save(prompt, { name: `${step.key}.prompt`, type: "prompt", stepKey: step.key });
+    const raw = rig.artifacts.artifact.save(result.rawText, { name: `${step.key}.raw`, type: "raw_response", stepKey: step.key });
     const parsed = rig.artifacts.artifact.save(result.json, {
-      name: `${step}.output`, type: "step_output", task: step, stepKey: step, schema: step.schema,
+      name: `${step.key}.output`, type: "step_output", task: step.key, stepKey: step.key, schema: step.schema,
     });
 
-    // 3. Lineage: the parsed output derives from the prompt + raw; this step's
-    //    prompt consumed the previous step's output.
+    // 4. Lineage: the parsed output derives from the prompt + raw.
     if (p && parsed) rig.artifacts.artifact.link(parsed, p, "derived_from");
     if (raw && parsed) rig.artifacts.artifact.link(parsed, raw, "derived_from");
 
-    // 4. Grades (optional, deterministic): flag conformance, degraded
+    // 5. Grades (optional, deterministic): flag conformance, degraded
     //    provenance, anything you can decide without an LLM judge.
     if (parsed) rig.grade({ kind: "artifact", id: parsed.id }, {
       score: conforms(result.json) ? 1 : 0, kind: "deterministic", grader: "schema-check",
@@ -96,13 +116,25 @@ try {
   run.end("failed");
   throw err;
 }
+await rig.close(); // once, at shutdown
 ```
 
+Here `step` is your own `{ key, schema }` record, and `callModel` returns the
+provider's text, parsed JSON and usage. A call routed through `rig.run` or
+`rig.runRaw` records its own step, so it needs no `recordCall`.
+
 For **concurrent** runs in one process, use `rig.artifacts.run.scope(opts, fn)`
 instead of `start`/`end` — each scope keeps its own ambient context so
 overlapping runs never cross-attribute. `run.start` uses `enterWith`, which is
 right for one run at a time on a request/async path.
 
+**Why a run failed.** *(needs `modelrig` 0.12.0 or newer)* End a failed run with its
+reason — `run.end("failed", { failureReason: "eps_integrity_failure" })` — or,
+inside `run.scope`, call `rig.artifacts.run.setFailureReason("…")` before the
+callback throws. The reason must be an identifier (≤128 characters, no
+whitespace). It lands in `runs.meta.failure_reason`, and the run-outcome verdict
+on `/runs` uses it as the failed run's label instead of a step's failure class.
+
 ## Steps vs. artifacts
 
 - An **artifact** is a work product — a durable thing your pipeline made. It
@@ -111,27 +143,58 @@ right for one run at a time on a request/async path.
 - A **step** is an execution row. `rig.run()` auto-creates a **ground-truth**
   step (its inference is real, so cost and latency are measured) when it runs
   inside a run context. A stage whose model call does *not* go through the rig
-  produces artifacts with `step_key` set and **no step row** — which is correct
-  and honest. Never route a stage through the rig just to mint a step row; the
-  console renders "cost n/a" for un-routed steps rather than inventing a number.
+  and is not reported with `rig.recordCall` produces artifacts with `step_key`
+  set and **no step row**, which is correct and honest: the console renders
+  "—" for that step's cost (shown as "cost n/a" with Developer details on)
+  rather than inventing a number. Report the call
+  (below) and the step gets its tokens and cost.
 
 ### Record a call your code made itself (`rig.recordCall`)
 
-_Needs `modelrig` 0.11.0 or newer._
+*Needs `modelrig` 0.11.0 or newer.*
 
 If a stage's model call has to stay a direct provider call, report it instead:
 `rig.recordCall({ provider, model, usage: { tokensIn, tokensOut, tokensCached }, servedTier, latencyMs, stepKey })`
 right after the call returns. Inside a run context it records a step row for
 that `stepKey` (stored as `ground-truth`, the only joined-inference value),
-joined to the recorded inference, so the step gets a cost instead of "cost n/a".
-The console shows that step and its call as **recorded · unverified**, never as
-measured:
+joined to the recorded inference, so the step gets a cost instead of "—" (shown as
+"cost n/a" with Developer details on). The console shows that step and its call as
+**Reported by your code** (**recorded · unverified** with Developer details on),
+never as measured:
 ModelRig neither dispatched nor validated the call, and the tokens are the ones
 you reported. It is synchronous and never throws (bad input warns and returns
 `inferenceId: null`); the returned `inferenceId` is gradeable with `rig.grade`.
 Full contract, usage convention and cost notes:
 [Not captured → record them](not-captured.md#calls-your-code-makes-itself-record-them).
 
+<!-- BLOCK:RECORD_CALL -->
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "Reported by your code", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+
+```ts
+const started = Date.now();
+const resp = await genai.models.generateContent({ model: "gemini-3.1-pro-preview", contents });
+const u = resp.usageMetadata;
+const finishReason = resp.candidates?.[0]?.finishReason;
+
+const rec = rig.recordCall({
+  provider: "gemini",
+  model: "gemini-3.1-pro-preview",
+  usage: {
+    tokensIn: u?.promptTokenCount ?? 0, // total input, cached included
+    tokensOut: (u?.candidatesTokenCount ?? 0) + (u?.thoughtsTokenCount ?? 0),
+    tokensCached: u?.cachedContentTokenCount ?? 0,
+    // optional fields: spread only when the provider reported them
+    ...(u?.thoughtsTokenCount !== undefined ? { reasoningTokens: u.thoughtsTokenCount } : {}),
+  },
+  servedTier: "standard", // the tier that SERVED the call: pass what the provider reports (flex, priority); it is priced at that tier
+  latencyMs: Date.now() - started,
+  stepKey: "summarize", // the step on /runs (inside run.scope / run.start)
+  ...(finishReason !== undefined ? { finishReason } : {}),
+});
+if (rec.inferenceId !== null) rig.grade({ kind: "inference", id: rec.inferenceId }, { score: 1, kind: "deterministic" });
+```
+<!-- /BLOCK:RECORD_CALL -->
+
 ## Tags, dimensions, and the graph
 
 ModelRig has **two** grouping mechanisms, and they compose:
@@ -237,7 +300,9 @@ const { bundle: runnable, arm } = resolveCallConfigForRun(bundle, resolved, { in
 
 // Thread the arm onto the run so it lands on inferences.meta.arm.
 await rig.artifacts.run.scope({ pipeline: "report", episodeKey: runId }, async () => {
-  const out = await rig.runBundle(runnable, { input, tags: { subject, feature }, arm, stepKey });
+  // arm is string | null; RunOptions.arm is optional, so pass it only when drawn
+  // (a bare `arm` fails under exactOptionalPropertyTypes).
+  const out = await rig.runBundle(runnable, { input, tags: { subject, feature }, stepKey, ...(arm !== null ? { arm } : {}) });
   rig.artifacts.artifact.save(out.output, { name: "section.output", type: "step_output" });
 });
 ```
@@ -294,14 +359,14 @@ orgs and the console picker: [Orgs and keys](https://modelrig.dev/orgs-and-keys.
 Check first that the console is showing the org your key exports to:
 
 <!-- BLOCK:CONFIRM_ORG -->
-**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in, once your app has started with the key. It asks GET /v1/whoami which organization the key belongs to and prints the org's name and, in parentheses, its slug; the name must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
 
 ```bash
 npx modelrig doctor
-# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+# [modelrig] key rig_sk_…AbCd → org «Your Org» (your-org) ✓  (ingest https://api.modelrig.ai, …)
 
-# No CLI in that environment? Ask the API directly:
-curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+# No CLI in that environment? Ask the API directly (same host the SDK uses):
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" "${MODELRIG_INGEST_URL:-https://api.modelrig.ai}/v1/whoami"
 ```
 <!-- /BLOCK:CONFIRM_ORG -->
 
@@ -319,22 +384,33 @@ an AI — attaches to a subject: one model call (an **inference**), a whole run
 [optimization coverage](grade-protocol.md) is built from — the evidence level
 that decides how much confidence a swap proposal carries.
 
+<!-- BLOCK:GRADE_RUN -->
+**Attach a grade to a call or a whole run.** A grade is your judgment: a 0–1 score with an optional comment, from a validator, a downstream accept/reject, a human or a model. Grade one call by its inference id (`result.meta.inferenceId`, or the `inferenceId` rig.recordCall returns). Grade a whole run by its run id, `run.runId` on the handle `run.start` returns: that grade shows on the run page and on /projects. A run grade whose id is the run's `episodeKey` also shows, on the run page only. `run_id` is a reserved tag the SDK stamps, not a variable your code has. A grade whose id matches no call or run attaches to nothing. The write is synchronous and local and never throws into your pipeline.
+
 ```ts
-// One call — the inference id rides on every RunResult:
-const result = await rig.run("example.support_summarize", { input, tags: { subject } });
-rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" });
+const run = rig.artifacts.run.start({ pipeline: "support", episodeKey: ticketId, environment: "production" });
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },
+  tags: { subject: customer },
+  stepKey: "summarize",
+});
+run.end("succeeded");
 
-// A whole run: the subject id is the run's run_id (its episodeKey, or its id).
-rig.grade({ kind: "run", id: runAttemptId }, { score: 0.2, comment: "CAGR off by 10x", kind: "human" });
+// one call: the inference id rides on every result
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "deterministic", grader: "schema-check" });
+// the whole run: its id is run.runId from run.start (shows on the run page and on /projects)
+rig.grade({ kind: "run", id: run.runId }, { score: 0, kind: "human", comment: "missed the refund request" });
 ```
-
-- **The subject** is `{ kind: "inference", id }` (`result.meta.inferenceId`),
-  `{ kind: "run", id }` (the run's `run_id` — its `episodeKey`, or its id), or
-  `{ kind: "artifact", id }`. **`opts`** is `{ score, scale?, comment?, kind,
-  source? }` — the `score` in `[0, 1]` is the primitive (up = 1, down = 0), and
-  the pass/fail verdict is derived from it against the route threshold. A subject
-  naming no known inference, run, or artifact attaches to nothing — attribution
-  is never guessed.
+<!-- /BLOCK:GRADE_RUN -->
+
+- **The subject** is `{ kind: "inference", id }` (`result.meta.inferenceId`, or
+  the `inferenceId` `rig.recordCall` returns), `{ kind: "run", id }` (the run's
+  id, `run.runId` from `run.start`; a grade carrying the run's `episodeKey`
+  also shows, on the run page only), or `{ kind: "artifact", id }`. **`opts`**
+  is `{ score, scale?, comment?, kind, grader?, source? }` — the `score` in
+  `[0, 1]` is the primitive (up = 1, down = 0), and the pass/fail verdict is
+  derived from it against the route threshold. A subject naming no known
+  inference, run, or artifact attaches to nothing — attribution is never guessed.
 - **Emit it from a signal you already have.** Most pipelines already decide
   pass/fail somewhere — a schema validator, a downstream accept/reject, a human
   QA gate. Emit `rig.grade` from that (score 1 on a clean pass, 0 on a rejection)
@@ -368,15 +444,40 @@ const caseId = rig.artifacts.artifact.promoteToEval(handle, { task: "report.sect
 
 ```bash
 # Replay the promoted eval cases through variants — the raw/run-based lane.
+# The variants (default, cheap) are read from the route bundle named report.section.
 modelrig bakeoff --from-eval-cases report.section --variants default,cheap
 ```
 
 - `promoteToEval(ref, { task?, corrected?, origin? })` binds the case to a route by
   `task` (from `opts`, else the task the artifact was saved with this session), and
   is also reachable from the console's one-click **"make golden"**.
-- `--route` needs a **declared route**; `--from-eval-cases <task>` is the path for a
-  `runRaw` seam that owns no route. Either way a bake-off **measures** — it never
-  switches what serves a route; a swap is a human-approved PR.
+- `--route` replays **captured route traffic**; `--from-eval-cases <task>` replays
+  the promoted cases instead, so it is the path for a `runRaw` seam with no
+  captured traffic. Both read the variants from a route bundle (`--route`, or the
+  bundle named `<task>`): selecting the cases needs no routes directory, but a
+  case that replays needs that bundle and its `variants:` block. Either way a
+  bake-off **measures** — it never switches what serves a route; never a silent swap — promote it in ModelRig (console or MCP; versioned, reversible) or merge it as a PR if you keep routes in git.
+
+<!-- BLOCK:BAKEOFF -->
+**Prove a cheaper model on your own traffic.** A bake-off replays your own inputs through the variants a route declares and reports the effective cost of conformance for each. `--variants` names entries in the route's `variants:` block (`default` is the route as declared), never model ids, and a variant can only narrow the route's candidates. Set `capture: true` first: replay works on traffic captured after that, not before. With no captured traffic (a run-based pipeline), promote a real /runs output to a golden (the console's make golden, or `promoteToEval`) and use `--from-eval-cases <task>`; its variants come from the route bundle named after the task. A bake-off measures and never switches anything.
+
+```bash
+# 1. in modelrig/routes/example.support_summarize.yaml: capture traffic, and declare
+#    a cheaper variant (narrowing only: its candidates must be in the route's list)
+capture: true
+variants:
+  - name: cheap
+    candidates:
+      - provider: gemini
+        model: gemini-3.1-flash-lite
+
+# 2. after traffic accumulates, compare the route as declared (default) with cheap
+npx modelrig bakeoff --route example.support_summarize --variants default,cheap --replay-last 50
+
+# no captured route traffic? promote a real /runs output to a golden, then
+npx modelrig bakeoff --from-eval-cases example.support_summarize --variants default,cheap
+```
+<!-- /BLOCK:BAKEOFF -->
 
 ## A reusable per-step seam
 
@@ -388,15 +489,18 @@ as diffs to apply — it never edits your source. The seam is equally yours to
 write by hand; here it is, self-contained: 
 
 ```ts
-import { createRig, loadConfigFromEnv, type Rig } from "modelrig";
+import { createRig, loadConfigFromEnv, type RecordCallInput, type Rig } from "modelrig";
 
-const rig: Rig = createRig(loadConfigFromEnv());
+// routesDir: null — the seam needs no route files, so importing it never throws.
+const rig: Rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
 
-/** Save prompt + raw + parsed for one step, link lineage, flag conformance. */
+/** Report the call (tokens → cost), save prompt + raw + parsed, link lineage,
+ * flag conformance. */
 function recordStep(
   stepKey: string,
-  work: { prompt: string; rawText: string; json: unknown; conforms: boolean },
+  work: { prompt: string; rawText: string; json: unknown; conforms: boolean; call?: Omit<RecordCallInput, "stepKey"> },
 ): void {
+  if (work.call !== undefined) rig.recordCall({ ...work.call, stepKey });
   const p = rig.artifacts.artifact.save(work.prompt, { name: `${stepKey}.prompt`, type: "prompt", stepKey });
   const raw = rig.artifacts.artifact.save(work.rawText, { name: `${stepKey}.raw`, type: "raw_response", stepKey });
   const parsed = rig.artifacts.artifact.save(work.json, { name: `${stepKey}.output`, type: "step_output", task: stepKey, stepKey });

package/docs/llms.txt

@@ -4,7 +4,7 @@
 > grounding, and caching measured empirically, with a capability-aware
 > gateway that routes by them. Registry data CC BY 4.0; code Apache-2.0.
 
-_This reference is as of modelrig v0.11.0 (the published npm package). Any surface below that a newer workspace adds is marked "unreleased — workspace only" until its release; nothing is so marked as of this version._
+_This reference is as of modelrig v0.12.0 (the published npm package). Any surface below that a newer workspace adds is marked "unreleased — workspace only" until its release; nothing is so marked as of this version._
 
 ## Docs
 
@@ -39,9 +39,18 @@ _This reference is as of modelrig v0.11.0 (the published npm package). Any surfa
 - [Managed serving](https://modelrig.dev/managed-serving.html): Serve a route's model on ModelRig's own credentials — destination: modelrig-served — paid from your prepaid balance at provider list cost plus the flat fee: declare the destination, enable managedServing in code, mint a serve-scoped key, opt the organization in, fund the balance; what each refusal teaches; the destination tag on the row; and the honest direct default that nothing here changes.
 - [Orgs and keys](https://modelrig.dev/orgs-and-keys.html): One key = one org: how to see which organization a key exports to (`modelrig doctor`, `GET /v1/whoami`, the console badge), how the console picks the org it shows (the picker, `409 org_required`, `401 session_expired`), why a valid key in the wrong org looks like an empty console, and `doctor --expect-org` / `createRig({ expectOrg })`.
 - [Choosing a lane](https://modelrig.dev/choosing-a-lane.html): Observe (wrap what you have in a run), the `rig.runRaw` seam (one adapter for a router you already own) or named routes (`rig.run`): when each fits, what you get, what you give up, what it costs — and why to start with observe.
-- [Not captured](https://modelrig.dev/not-captured.html): What ModelRig does not see — direct provider SDK calls, streams read outside the SDK, fallbacks after a fail-closed schema, calls outside a run, embeddings — and the console tells: `cost n/a` steps, a missing `inferenceId`, a run total below your provider bill.
+- [Not captured](https://modelrig.dev/not-captured.html): What ModelRig does not see — direct provider SDK calls, streams read outside the SDK, fallbacks after a fail-closed schema, calls outside a run, embeddings — and the console tells: steps with no cost (`—`; `cost n/a` with Developer details on), a missing `inferenceId`, a run total below your provider bill.
 - [Deploying on hosted workers](https://modelrig.dev/deploying.html): Set MODELRIG_API_KEY on every process; Render's env-group vs service-override precedence; a change needs a finished redeploy and a new shell; rotate keys mint → set → redeploy → doctor → revoke; verify after every deploy with `doctor --expect-org`, then one run on /runs; notes for Vercel, Fly and Docker.
 - [Troubleshooting an empty console](https://modelrig.dev/troubleshooting.html): The "console is empty" decision tree: is the process exporting (doctor: AUTH FAILED / no export sink / org ✓), is it the org you are viewing, are the calls inside a run, is the missing call on a ModelRig lane, and does your install match the docs.
+- [Conversations, outcomes and limits](https://modelrig.dev/conversations-outcomes-limits.html): Wrap a chatbot conversation as one run with turns, post outcomes (resolved, booked, CTR) from a webhook, and stop runaway agent loops in-process with run limits.
+- [Template: support conversation](https://modelrig.dev/template-support-conversation.html): A starting point for grading a whole support conversation: resolved for real, policy adherence, correct tool use, tone, escalated when needed. No accuracy claims.
+- [Template: creative approval](https://modelrig.dev/template-creative-approval.html): A starting point for approving generated images, video and audio: on brief, brand safe, legible text, no generation defects, usable as is — human verdicts that feed the approval rate per dollar. Generated media is stored unscreened. No accuracy claims.
+- [Recipe: Vercel AI SDK](https://modelrig.dev/recipe-vercel-ai-sdk.html): Send the Vercel AI SDK's experimental_telemetry traces to ModelRig over OTLP, and set gen_ai.conversation.id so a chat becomes a conversation with turns. Content is not read; attested steps are not priced.
+- [Recipe: OpenAI Agents SDK](https://modelrig.dev/recipe-openai-agents-sdk.html): Forward the OpenAI Agents SDK's traces to OpenTelemetry with a trace processor and export them to ModelRig over OTLP; open an invoke_agent span per user message to number turns.
+- [Recipe: LangGraph / LangChain JS](https://modelrig.dev/recipe-langgraph-js.html): Turn LangChain JS callbacks into OpenTelemetry spans and export them to ModelRig over OTLP; wrap each user message in an invoke_agent span to number turns.
+- [Recipe: Mastra](https://modelrig.dev/recipe-mastra.html): Point Mastra's OTLP telemetry export at ModelRig; each agent request becomes a run, and a conversation when it carries gen_ai.conversation.id and invoke_agent spans.
+- [Recipe: LiveKit Agents](https://modelrig.dev/recipe-livekit-agents.html): Export a LiveKit Agents voice session's OpenTelemetry traces to ModelRig with your conversation id as span metadata — post-call visibility, not duplex voice.
+- [Set up a grader](https://modelrig.dev/set-up-a-grader.html): Teach an AI grader for one step or route: how a task registers (route sync, a runRaw/recordCall step's tags.step, or POST /v1/tasks), a copy-paste prompt your coding agent runs (MCP needs modelrig-oracle 0.3.0+, HTTP otherwise), the console path, TEACHING to CALIBRATED, which grades count, and what is metered.
 - [Leaderboard](https://modelrig.dev/leaderboard.html): Models ranked by effective cost per 1,000 schema-conformant outputs, with declared-vs-probed discrepancy flags.
 
 ## Source
@@ -60,14 +69,14 @@ _The default ingest host is https://api.modelrig.ai (override with `MODELRIG_ING
 MODELRIG_API_KEY=rig_sk_…
 ```
 
-**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in, once your app has started with the key. It asks GET /v1/whoami which organization the key belongs to and prints the org's name and, in parentheses, its slug; the name must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
 
 ```bash
 npx modelrig doctor
-# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+# [modelrig] key rig_sk_…AbCd → org «Your Org» (your-org) ✓  (ingest https://api.modelrig.ai, …)
 
-# No CLI in that environment? Ask the API directly:
-curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+# No CLI in that environment? Ask the API directly (same host the SDK uses):
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" "${MODELRIG_INGEST_URL:-https://api.modelrig.ai}/v1/whoami"
 ```
 
 **Wrap one pipeline execution in a run.** Every pipeline execution is a run. Wrap each execution in `run.scope` (or `run.start` / `run.end`), name one step per model call with `stepKey`, `artifact.save` the work products, and pass `environment` so the Env column on /runs is filled. Inside the run the SDK stamps `run_id` and `step` on every call. `run_id` is reserved: don't set it yourself.
@@ -96,30 +105,146 @@ await rig.artifacts.run.scope(
 await rig.close(); // flushes the exporter — call once at shutdown
 ```
 
-**What ModelRig does not see unless you record it.** ModelRig records the calls your code makes through the SDK (`rig.run`, `rig.runBundle`, `rig.runRaw`, `rig.runRawStream`) or the gateway, plus any call your code makes itself and reports with `rig.recordCall`. Nothing else is seen. Five things stay invisible or unattached unless you record them: (1) direct provider SDK calls, with or without a JSON schema; (2) streams you read from a provider client yourself; (3) a fallback your code makes after the raw lane refuses a `responseSchema` it cannot enforce (deepseek, deepinfra, fireworks and zai fail closed with `invariant_violation`); (4) calls made outside `run.scope` / `run.start`, which land on /inferences but on no run; (5) embeddings. A recorded call shows as "recorded · unverified": ModelRig neither dispatched nor validated it, and its tokens are the numbers you reported. Details and the console tells: https://modelrig.dev/not-captured.html.
+**What ModelRig does not see unless you record it.** ModelRig records the calls your code makes through the SDK (`rig.run`, `rig.runBundle`, `rig.runRaw`, `rig.runRawStream`) or the gateway, plus any call your code makes itself and reports with `rig.recordCall`. Nothing else is seen. Five things stay invisible or unattached unless you record them: (1) direct provider SDK calls, with or without a JSON schema; (2) streams you read from a provider client yourself; (3) a fallback your code makes after the raw lane refuses a `responseSchema` it cannot enforce (deepseek, deepinfra, fireworks and zai fail closed with `invariant_violation`); (4) calls made outside `run.scope` / `run.start`, which land on /inferences but on no run; (5) embeddings. A recorded call shows as "Reported by your code": ModelRig neither dispatched nor validated it, and its tokens are the numbers you reported. Details and the console tells: https://modelrig.dev/not-captured.html.
 
 **The console is empty: five checks, in order.** If /runs (or /inferences) is empty for the org in the header badge, work these checks in order. The full tree is https://modelrig.dev/troubleshooting.html.
 
 ```text
 1. npx modelrig doctor, from the environment your app runs in.
    no export sink configured → this process does not see MODELRIG_API_KEY: set it, redeploy, open a new shell.
+   malformed key             → the value is truncated, quoted or padded: paste the whole rig_sk_… secret again.
    AUTH FAILED (401 / 403)   → the key is mistyped, revoked or not accepted: mint a key, set it, redeploy, open a new shell.
+   whoami unreachable        → no network path to the ingest host (firewall, proxy, wrong MODELRIG_INGEST_URL): fix the route out, retry.
 2. The org doctor prints is not the org in the badge → wrong org: switch the console, or mint a key in the org you meant.
 3. /inferences has rows but /runs is empty → the calls are not inside run.scope / run.start.
 4. A run is missing a step or a cost → that call is not on a ModelRig lane (see not-captured).
 5. A doc describes something your install lacks → compare `npm ls modelrig` with the version stamp in modelrig.dev/llms.txt.
 ```
 
-**After you change the key on a hosted worker.** A hosted process reads the environment it was started with. After you set or change MODELRIG_API_KEY, wait for the redeploy to finish, then open a new shell: a shell attached before the deploy still has the old environment. Confirm from inside it that the key resolves to the org you expect.
+**After you change the key on a hosted worker.** A hosted process reads the environment it was started with. After you set or change MODELRIG_API_KEY, wait for the redeploy to finish, then open a new shell: a shell attached before the deploy still has the old environment. Confirm from inside it that the key resolves to the org you expect: `--expect-org` takes the org's slug, the part in parentheses in doctor's output.
 
 ```bash
-npx modelrig doctor --expect-org your-org-slug   # exits non-zero if the key belongs to another org
+npx modelrig doctor --expect-org your-org   # exits non-zero if the key belongs to another org
+```
+
+**No routes yet? Build the rig raw-only.** Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with `routesDir: null`: `rig.runRaw`, `rig.runRawStream` and `rig.recordCall` work at once, and `rig.run` / `rig.runBundle` refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.
+
+```ts
+import { createRig, loadConfigFromEnv } from "modelrig";
+
+// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
+// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
+const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
+```
+
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "Reported by your code", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+
+```ts
+const started = Date.now();
+const resp = await genai.models.generateContent({ model: "gemini-3.1-pro-preview", contents });
+const u = resp.usageMetadata;
+const finishReason = resp.candidates?.[0]?.finishReason;
+
+const rec = rig.recordCall({
+  provider: "gemini",
+  model: "gemini-3.1-pro-preview",
+  usage: {
+    tokensIn: u?.promptTokenCount ?? 0, // total input, cached included
+    tokensOut: (u?.candidatesTokenCount ?? 0) + (u?.thoughtsTokenCount ?? 0),
+    tokensCached: u?.cachedContentTokenCount ?? 0,
+    // optional fields: spread only when the provider reported them
+    ...(u?.thoughtsTokenCount !== undefined ? { reasoningTokens: u.thoughtsTokenCount } : {}),
+  },
+  servedTier: "standard", // the tier that SERVED the call: pass what the provider reports (flex, priority); it is priced at that tier
+  latencyMs: Date.now() - started,
+  stepKey: "summarize", // the step on /runs (inside run.scope / run.start)
+  ...(finishReason !== undefined ? { finishReason } : {}),
+});
+if (rec.inferenceId !== null) rig.grade({ kind: "inference", id: rec.inferenceId }, { score: 1, kind: "deterministic" });
+```
+
+**Attach a grade to a call or a whole run.** A grade is your judgment: a 0–1 score with an optional comment, from a validator, a downstream accept/reject, a human or a model. Grade one call by its inference id (`result.meta.inferenceId`, or the `inferenceId` rig.recordCall returns). Grade a whole run by its run id, `run.runId` on the handle `run.start` returns: that grade shows on the run page and on /projects. A run grade whose id is the run's `episodeKey` also shows, on the run page only. `run_id` is a reserved tag the SDK stamps, not a variable your code has. A grade whose id matches no call or run attaches to nothing. The write is synchronous and local and never throws into your pipeline.
+
+```ts
+const run = rig.artifacts.run.start({ pipeline: "support", episodeKey: ticketId, environment: "production" });
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },
+  tags: { subject: customer },
+  stepKey: "summarize",
+});
+run.end("succeeded");
+
+// one call: the inference id rides on every result
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "deterministic", grader: "schema-check" });
+// the whole run: its id is run.runId from run.start (shows on the run page and on /projects)
+rig.grade({ kind: "run", id: run.runId }, { score: 0, kind: "human", comment: "missed the refund request" });
+```
+
+**Prove a cheaper model on your own traffic.** A bake-off replays your own inputs through the variants a route declares and reports the effective cost of conformance for each. `--variants` names entries in the route's `variants:` block (`default` is the route as declared), never model ids, and a variant can only narrow the route's candidates. Set `capture: true` first: replay works on traffic captured after that, not before. With no captured traffic (a run-based pipeline), promote a real /runs output to a golden (the console's make golden, or `promoteToEval`) and use `--from-eval-cases <task>`; its variants come from the route bundle named after the task. A bake-off measures and never switches anything.
+
+```bash
+# 1. in modelrig/routes/example.support_summarize.yaml: capture traffic, and declare
+#    a cheaper variant (narrowing only: its candidates must be in the route's list)
+capture: true
+variants:
+  - name: cheap
+    candidates:
+      - provider: gemini
+        model: gemini-3.1-flash-lite
+
+# 2. after traffic accumulates, compare the route as declared (default) with cheap
+npx modelrig bakeoff --route example.support_summarize --variants default,cheap --replay-last 50
+
+# no captured route traffic? promote a real /runs output to a golden, then
+npx modelrig bakeoff --from-eval-cases example.support_summarize --variants default,cheap
+```
+
+**Pin the org a deployment must export to.** `--expect-org` and `expectOrg` take the org's slug, not its display name. `modelrig doctor` prints both, as `→ org «Your Org» (your-org) ✓`: the slug is the part in parentheses. In the console, hover the org badge in the header to see it. With `--expect-org <slug>`, doctor exits non-zero when the key belongs to another org, so a deploy check fails loudly instead of exporting somewhere else.
+
+```ts
+// The same check in code: the exporter asks whoami on its first flush and logs
+// ORG MISMATCH on every flush while the key belongs to another org.
+const rig = createRig({ ...loadConfigFromEnv(), expectOrg: "your-org" });
+```
+
+**Point your OTLP exporter at ModelRig.** ModelRig accepts OpenTelemetry traces at POST /v1/otlp/v1/traces on the ingest host (https://api.modelrig.ai unless you were given another), over OTLP/HTTP as protobuf or JSON, optionally gzipped. Authenticate with a `rig_sk_` key that carries the `ingest` scope; every run and step lands in that key's organization. Each trace becomes one run with one attested step per span: model, provider, operation, timing, status and token usage. Prompts, completions and tool arguments are not read.
+
+```bash
+# Standard OpenTelemetry exporter variables — most OTel SDKs read them as-is.
+OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.modelrig.ai/v1/otlp/v1/traces
+OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer%20rig_sk_…
+OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
+```
+
+**Set gen_ai.conversation.id to your conversation id.** A trace becomes a conversation when one of its spans carries `gen_ai.conversation.id` and it has `invoke_agent` spans. Each top-level `invoke_agent` span is a turn, in start order, and every model call under it becomes a step `turn:<n>.<k>`; the run is marked as a conversation and its episode key is your conversation id. Turns are numbered within a trace: when each user message arrives as its own trace, each shows as its own one-turn conversation run, and the shared conversation id groups them. If your framework does not emit an `invoke_agent` span itself, wrap each user message in one.
+
+```ts
+import { trace } from "@opentelemetry/api";
+
+const tracer = trace.getTracer("support-bot");
+
+// One span per user message. The model calls your framework makes inside it
+// become this conversation's turn steps; the conversation id joins the runs.
+export function turn<T>(conversationId: string, fn: () => Promise<T>): Promise<T> {
+  return tracer.startActiveSpan(
+    "invoke_agent support-bot",
+    { attributes: { "gen_ai.operation.name": "invoke_agent", "gen_ai.conversation.id": conversationId } },
+    async (span) => {
+      try {
+        return await fn();
+      } finally {
+        span.end();
+      }
+    },
+  );
+}
 ```
 
 ## Capabilities (the full surface)
 
 - **Runs & projects** — the run → step → artifact graph at /runs, grouped by tag at /projects (the standard path above): https://modelrig.dev/instrumentation.html.
-- **Recorded calls (`rig.recordCall`, modelrig ≥ 0.11.0)** — a provider call your code makes itself (no lane fits) is reported after it returns: one row, priced at list for the served tier from the tokens you report, joined to the run/step, gradeable, badged "recorded · unverified" (ModelRig neither dispatched nor validated it), never in routed/conformance math: https://modelrig.dev/not-captured.html.
+- **Recorded calls (`rig.recordCall`, modelrig ≥ 0.11.0)** — a provider call your code makes itself (no lane fits) is reported after it returns: one row, priced at list for the served tier from the tokens you report, joined to the run/step, gradeable, badged "Reported by your code" ("recorded · unverified" with Developer details on; ModelRig neither dispatched nor validated it), never in routed/conformance math: https://modelrig.dev/not-captured.html.
+- **Diagnosable export drops (`SinkSendError.reasonCode`, modelrig ≥ 0.12.0)** — a batch the server rejects deterministically (`422`) names the database error code in the body's `reason_code` (e.g. `PGRST102`, `23514`); the SDK carries it on the thrown error and the exporter's `POISON-SKIP` line: https://modelrig.dev/troubleshooting.html.
 - **Tenants & statements** — a `subject` tag or project-scoped key becomes a per-customer tenant with a derived statement (attempted, verified, billable, true cost; never content or another org's data): https://modelrig.dev/tenants.html.
 - **Published receipts** — a signed, login-free receipt of a bake-off or route at /r/<id>, verifiable by anyone at /v1/verify/<id>; internal names aliased, content never leaves: https://modelrig.dev/receipts.html.
 - **Analytics query API** — a self-describing, read-only query surface over your own telemetry (`GET /v1/analytics/meta`, `POST /v1/analytics/query`): an allowlisted SELECT-only grammar that returns metadata only (enums, numbers, hashes), never content: https://modelrig.dev/optimization-loop.html.
@@ -135,7 +260,7 @@ npx modelrig doctor --expect-org your-org-slug   # exits non-zero if the key bel
 - Rates below are sampled statistics with confidence intervals in the source data — do not quote them without their n and date.
 - For PRICING, do not recall list prices — read them: the leaderboard (https://modelrig.dev/leaderboard.html) ranks models by effective cost per 1,000 schema-conformant outputs (list price ≠ what conformance costs), and each model's per-token rates ride the registry JSON with as_of stamps. Quote the effective cost, not the sticker rate.
 - Attach a GRADE to what you run — capture records what ran, a grade records whether it was good: `rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" })` (or `{ kind: "run", id: run_id }` for a whole run) from a signal you already have (a validator, a downstream accept/reject, a human QA gate). A grade is a 0–1 score with an optional comment; up = 1, down = 0. It is a synchronous local write, exporter-mirrored, and never blocks or throws. A route with ten attached grades reaches Rung 1 on /coverage; a route with no schema to conform to (a prose output) gains proposal confidence only this way. From a non-SDK surface, `POST /v1/grade` with a grade-scoped `rig_sk_` key. See https://modelrig.dev/grade-protocol.html.
-- Prove a cheaper model on YOUR own traffic on the lane that fits: a DECLARED route uses `modelrig bakeoff --route <route> --replay-last N`; a raw/run-based pipeline (a `rig.runRaw` seam, no routes) promotes a real /runs output to a golden (`promoteToEval`, or the console's one-click "make golden") and runs `modelrig bakeoff --from-eval-cases <task> --variants default,<challenger>`. A bake-off measures; it never switches what serves a route — a swap is a human-approved PR. See https://modelrig.dev/bakeoffs.html.
+- Prove a cheaper model on YOUR own traffic on the lane that fits: a DECLARED route uses `modelrig bakeoff --route <route> --replay-last N`; a raw/run-based pipeline (a `rig.runRaw` seam, no routes) promotes a real /runs output to a golden (`promoteToEval`, or the console's one-click "make golden") and runs `modelrig bakeoff --from-eval-cases <task> --variants default,<challenger>`. A bake-off measures; it never switches what serves a route — never a silent swap — promote it in ModelRig (console or MCP; versioned, reversible) or merge it as a PR if you keep routes in git. See https://modelrig.dev/bakeoffs.html.
 
 ## Ask this data directly (MCP)
 
@@ -148,6 +273,11 @@ npx modelrig doctor --expect-org your-org-slug   # exits non-zero if the key bel
 - Register it in Claude Code: add `{"mcpServers":{"modelrig":{"command":"npx","args":["modelrig-oracle"]}}}` to your MCP config.
 - Every answer carries `scope`, `probe_data_as_of`, and a staleness flag. A model absent from an answer is UNPROBED, not incapable — say that rather than implying a limit.
 
+## Framework recipes (OTLP) and generated media
+
+- FRAMEWORK RECIPES: send a framework's OpenTelemetry traces to `POST /v1/otlp/v1/traces` with a `rig_sk_` ingest key — Vercel AI SDK (https://modelrig.dev/recipe-vercel-ai-sdk.html), OpenAI Agents SDK (https://modelrig.dev/recipe-openai-agents-sdk.html), LangGraph / LangChain JS (https://modelrig.dev/recipe-langgraph-js.html), Mastra (https://modelrig.dev/recipe-mastra.html), LiveKit Agents (https://modelrig.dev/recipe-livekit-agents.html). Each trace is a run of attested steps (model, provider, operation, timing, status, token usage). Set `gen_ai.conversation.id` on a span and emit `invoke_agent` spans and the trace becomes a conversation: generations numbered `turn:<n>.<k>`, tool calls counted per turn. NOT shown: content (prompts/completions are not read — OTLP content extraction is behind a server gate that is OFF) and cost (attested steps are not priced). The recipe configs are marked untested — verify against your framework version.
+- GENERATION CALLS (modelrig ≥ 0.12.0): `rig.recordCall({ provider, model, usage: { tokensIn: 0, tokensOut: 0 }, units: { images?, seconds?, characters?, steps? }, latencyMs })` records an image/video/TTS/STT call priced per unit — from the registry's declared generation entry (sourced rate + as_of), else the vendored pricing map, else `pricingMissing` (never from token rates). Query them with `query_registry({ kind: "image_gen" | "video_gen" | "tts" | "stt" })`: declared-only, unprobed, never ranked. Save generated assets as `image/*` / `video/*` / `audio/*` artifacts (stored unscreened); approve or reject them in the Grade queue's Media approval tab; Costs shows approval rate per $ by model from ten approvals. Starter guide: https://modelrig.dev/template-creative-approval.html.
+
 ## Live routing & call-config (change what serves a route, no redeploy)
 
 - A route's CALL-CONFIG — which model/provider serves it + its sampling — can change WHILE THE APP RUNS, no YAML edit, no redeploy. Two modes: `pin` (one model serves now) and `experiment` (a weighted live A/B whose drawn arm records on `inferences.meta.arm`). Set it in the console's call-config card, or over MCP (below).
@@ -290,7 +420,7 @@ What `rig.runRaw` sends when you omit a knob. `timeoutMs` is tier-aware since 0.
 
 ## Grades (judgment on inferences, runs & artifacts)
 
-- A GRADE is a 0–1 score (with an optional comment) attached to a SUBJECT — an inference, a run (episode), or an artifact — from your code, a human, or an AI. The pass/fail verdict is DERIVED, never stored: `pass = score >= the route's threshold` (default 0.5). It attaches from the SDK (`rig.grade(subject, { score, kind })`) or posts over HTTP from a non-SDK surface: `POST /v1/grade` on modelrig-server, authenticated with a `rig_sk_` key carrying the `grade` scope, body `{inferenceId | episode, score, scale?, comment?, kind?, source?}`. The org is resolved from the key, never the request body; there is NO public/unauthenticated grade endpoint.
+- A GRADE is a 0–1 score (with an optional comment) attached to a SUBJECT — an inference, a run (episode), or an artifact — from your code, a human, or an AI. The pass/fail verdict is DERIVED, never stored: `pass = score >= the route's threshold` (default 0.5). It attaches from the SDK (`rig.grade(subject, { score, kind })`) or posts over HTTP from a non-SDK surface: `POST /v1/grade` on modelrig-server, authenticated with a `rig_sk_` key carrying the `grade` scope, body `{inferenceId | episode | artifactId, score, scale?, comment?, kind?, source?}` (`artifactId` grades an artifact). The org is resolved from the key, never the request body; there is NO public/unauthenticated grade endpoint.
 - Every run that ends gets one deterministic `run-outcome@v1` grade AUTOMATICALLY, with no model call: it scores 1.0 (a pass) iff the run succeeded AND no step failed AND every deterministic grade passed; otherwise 0.0 with a dominant-failure label.
 - An explicit, bounded model judge (`run-judge@v1`) can ADDITIONALLY score a run's outcome-achievement 0–1 as a `model-judge` grade — invoked only on demand (console button or `modelrig judge <runId>`), capped per-org per-day, and metered as a routed gateway call. There is NO automatic or scheduled model-judging.
 - SDK callers attach their own run- or artifact-level grade the same way: `rig.grade({ kind: "run", id: run_id }, { score, kind, comment? })` — the grader `kind` is one of human | model-judge | deterministic.
@@ -309,7 +439,7 @@ What `rig.runRaw` sends when you omit a knob. `timeoutMs` is tier-aware since 0.
 ## Use-case templates (start a task with a schema + guide)
 
 - A USE-CASE TEMPLATE is a starting point for putting one task on the loop: a starter output schema (with TODO enum slots you fill with your own taxonomy — never our invented values), a starter grading guide in the exact jsonb shape the settings path validates, a golden checklist of what ~20 goldens should cover, and an escalation starter. It carries NO number about how well any model does the task — it is a shape to fill in, and what you fill in is what your goldens and grades then measure. Overview: https://modelrig.dev/use-case-templates.html.
-- Nine ship today, one per task the loop can serve: **ticket triage** (https://modelrig.dev/template-ticket-triage.html), **document extraction** (https://modelrig.dev/template-document-extraction.html), **CS next action** (https://modelrig.dev/template-cs-next-action.html), **lead qualification** (https://modelrig.dev/template-lead-qualification.html), **compliance review** (https://modelrig.dev/template-compliance-review.html), **catalog cleansing** (https://modelrig.dev/template-catalog-cleansing.html), **call disposition QA** (https://modelrig.dev/template-call-disposition-qa.html), **financial classification** (https://modelrig.dev/template-financial-classification.html), and **medical classification** (https://modelrig.dev/template-medical-classification.html). Each machine-readable file also ships in the npm package at `node_modules/modelrig/templates/<use-case>.json`.
+- Eleven templates, one per task the loop can serve (all eleven ship in the npm package): **ticket triage** (https://modelrig.dev/template-ticket-triage.html), **document extraction** (https://modelrig.dev/template-document-extraction.html), **CS next action** (https://modelrig.dev/template-cs-next-action.html), **lead qualification** (https://modelrig.dev/template-lead-qualification.html), **compliance review** (https://modelrig.dev/template-compliance-review.html), **catalog cleansing** (https://modelrig.dev/template-catalog-cleansing.html), **call disposition QA** (https://modelrig.dev/template-call-disposition-qa.html), **financial classification** (https://modelrig.dev/template-financial-classification.html), **medical classification** (https://modelrig.dev/template-medical-classification.html), **support conversation** (https://modelrig.dev/template-support-conversation.html), and **creative approval** (https://modelrig.dev/template-creative-approval.html). Each machine-readable file also ships in the npm package at `node_modules/modelrig/templates/<use-case>.json`.
 - REGULATED ROWS: financial classification and medical classification handle regulated data. Their fit notes quote the published content custody terms verbatim — do not send protected health information or other regulated data through ModelRig until a BAA or DPA is in place — and name the today-path where those terms are not yet in place: a zero-retention route (or content custody off), so ModelRig holds no content and grading is limited to the scores and metadata your own systems send. Terms: https://modelrig.ai/terms/content-custody. No other compliance claim is made.
 - CHECK FIT FIRST (the overview applies these to each task): a task suits the loop when its distribution is stable (goldens stay valid), a rubric is worth writing (the grading guide), expert examples are on hand to grade, and volume makes calibration pay back. A task that fails one structurally is honest to leave at the teaching stage, not force through the loop.
 - TWO CONSUMPTION PATHS, one file: a human copies the schema and guide from the docs page (they run as-is; TODOs mark where to tighten); or a coding agent feeds the template through the tools that ALREADY EXIST — `create_route` drafts the route bundle from the starter schema (files only, never your source), then `set_task_settings` (the `task-settings` scope) authors the grading guide — the starter criteria run as written; sharpen them into your team's own words as you grade. No new tool, no server change. The starter guide has NO `pass_threshold` (that lives in the route's `grade:` block — the guide teaches the grader, the threshold derives the verdict).