modelrig 0.11.1 → 0.12.0

package/docs/llms.txt

@@ -4,7 +4,7 @@
 > grounding, and caching measured empirically, with a capability-aware
 > gateway that routes by them. Registry data CC BY 4.0; code Apache-2.0.
 
-_This reference is as of modelrig v0.11.1 (the published npm package). Any surface below that a newer workspace adds is marked "unreleased — workspace only" until its release; nothing is so marked as of this version._
+_This reference is as of modelrig v0.12.0 (the published npm package). Any surface below that a newer workspace adds is marked "unreleased — workspace only" until its release; nothing is so marked as of this version._
 
 ## Docs
 
@@ -39,9 +39,18 @@ _This reference is as of modelrig v0.11.1 (the published npm package). Any surfa
 - [Managed serving](https://modelrig.dev/managed-serving.html): Serve a route's model on ModelRig's own credentials — destination: modelrig-served — paid from your prepaid balance at provider list cost plus the flat fee: declare the destination, enable managedServing in code, mint a serve-scoped key, opt the organization in, fund the balance; what each refusal teaches; the destination tag on the row; and the honest direct default that nothing here changes.
 - [Orgs and keys](https://modelrig.dev/orgs-and-keys.html): One key = one org: how to see which organization a key exports to (`modelrig doctor`, `GET /v1/whoami`, the console badge), how the console picks the org it shows (the picker, `409 org_required`, `401 session_expired`), why a valid key in the wrong org looks like an empty console, and `doctor --expect-org` / `createRig({ expectOrg })`.
 - [Choosing a lane](https://modelrig.dev/choosing-a-lane.html): Observe (wrap what you have in a run), the `rig.runRaw` seam (one adapter for a router you already own) or named routes (`rig.run`): when each fits, what you get, what you give up, what it costs — and why to start with observe.
-- [Not captured](https://modelrig.dev/not-captured.html): What ModelRig does not see — direct provider SDK calls, streams read outside the SDK, fallbacks after a fail-closed schema, calls outside a run, embeddings — and the console tells: `cost n/a` steps, a missing `inferenceId`, a run total below your provider bill.
+- [Not captured](https://modelrig.dev/not-captured.html): What ModelRig does not see — direct provider SDK calls, streams read outside the SDK, fallbacks after a fail-closed schema, calls outside a run, embeddings — and the console tells: steps with no cost (`—`; `cost n/a` with Developer details on), a missing `inferenceId`, a run total below your provider bill.
 - [Deploying on hosted workers](https://modelrig.dev/deploying.html): Set MODELRIG_API_KEY on every process; Render's env-group vs service-override precedence; a change needs a finished redeploy and a new shell; rotate keys mint → set → redeploy → doctor → revoke; verify after every deploy with `doctor --expect-org`, then one run on /runs; notes for Vercel, Fly and Docker.
 - [Troubleshooting an empty console](https://modelrig.dev/troubleshooting.html): The "console is empty" decision tree: is the process exporting (doctor: AUTH FAILED / no export sink / org ✓), is it the org you are viewing, are the calls inside a run, is the missing call on a ModelRig lane, and does your install match the docs.
+- [Conversations, outcomes and limits](https://modelrig.dev/conversations-outcomes-limits.html): Wrap a chatbot conversation as one run with turns, post outcomes (resolved, booked, CTR) from a webhook, and stop runaway agent loops in-process with run limits.
+- [Template: support conversation](https://modelrig.dev/template-support-conversation.html): A starting point for grading a whole support conversation: resolved for real, policy adherence, correct tool use, tone, escalated when needed. No accuracy claims.
+- [Template: creative approval](https://modelrig.dev/template-creative-approval.html): A starting point for approving generated images, video and audio: on brief, brand safe, legible text, no generation defects, usable as is — human verdicts that feed the approval rate per dollar. Generated media is stored unscreened. No accuracy claims.
+- [Recipe: Vercel AI SDK](https://modelrig.dev/recipe-vercel-ai-sdk.html): Send the Vercel AI SDK's experimental_telemetry traces to ModelRig over OTLP, and set gen_ai.conversation.id so a chat becomes a conversation with turns. Content is not read; attested steps are not priced.
+- [Recipe: OpenAI Agents SDK](https://modelrig.dev/recipe-openai-agents-sdk.html): Forward the OpenAI Agents SDK's traces to OpenTelemetry with a trace processor and export them to ModelRig over OTLP; open an invoke_agent span per user message to number turns.
+- [Recipe: LangGraph / LangChain JS](https://modelrig.dev/recipe-langgraph-js.html): Turn LangChain JS callbacks into OpenTelemetry spans and export them to ModelRig over OTLP; wrap each user message in an invoke_agent span to number turns.
+- [Recipe: Mastra](https://modelrig.dev/recipe-mastra.html): Point Mastra's OTLP telemetry export at ModelRig; each agent request becomes a run, and a conversation when it carries gen_ai.conversation.id and invoke_agent spans.
+- [Recipe: LiveKit Agents](https://modelrig.dev/recipe-livekit-agents.html): Export a LiveKit Agents voice session's OpenTelemetry traces to ModelRig with your conversation id as span metadata — post-call visibility, not duplex voice.
+- [Set up a grader](https://modelrig.dev/set-up-a-grader.html): Teach an AI grader for one step or route: how a task registers (route sync, a runRaw/recordCall step's tags.step, or POST /v1/tasks), a copy-paste prompt your coding agent runs (MCP needs modelrig-oracle 0.3.0+, HTTP otherwise), the console path, TEACHING to CALIBRATED, which grades count, and what is metered.
 - [Leaderboard](https://modelrig.dev/leaderboard.html): Models ranked by effective cost per 1,000 schema-conformant outputs, with declared-vs-probed discrepancy flags.
 
 ## Source
@@ -96,7 +105,7 @@ await rig.artifacts.run.scope(
 await rig.close(); // flushes the exporter — call once at shutdown
 ```
 
-**What ModelRig does not see unless you record it.** ModelRig records the calls your code makes through the SDK (`rig.run`, `rig.runBundle`, `rig.runRaw`, `rig.runRawStream`) or the gateway, plus any call your code makes itself and reports with `rig.recordCall`. Nothing else is seen. Five things stay invisible or unattached unless you record them: (1) direct provider SDK calls, with or without a JSON schema; (2) streams you read from a provider client yourself; (3) a fallback your code makes after the raw lane refuses a `responseSchema` it cannot enforce (deepseek, deepinfra, fireworks and zai fail closed with `invariant_violation`); (4) calls made outside `run.scope` / `run.start`, which land on /inferences but on no run; (5) embeddings. A recorded call shows as "recorded · unverified": ModelRig neither dispatched nor validated it, and its tokens are the numbers you reported. Details and the console tells: https://modelrig.dev/not-captured.html.
+**What ModelRig does not see unless you record it.** ModelRig records the calls your code makes through the SDK (`rig.run`, `rig.runBundle`, `rig.runRaw`, `rig.runRawStream`) or the gateway, plus any call your code makes itself and reports with `rig.recordCall`. Nothing else is seen. Five things stay invisible or unattached unless you record them: (1) direct provider SDK calls, with or without a JSON schema; (2) streams you read from a provider client yourself; (3) a fallback your code makes after the raw lane refuses a `responseSchema` it cannot enforce (deepseek, deepinfra, fireworks and zai fail closed with `invariant_violation`); (4) calls made outside `run.scope` / `run.start`, which land on /inferences but on no run; (5) embeddings. A recorded call shows as "Reported by your code": ModelRig neither dispatched nor validated it, and its tokens are the numbers you reported. Details and the console tells: https://modelrig.dev/not-captured.html.
 
 **The console is empty: five checks, in order.** If /runs (or /inferences) is empty for the org in the header badge, work these checks in order. The full tree is https://modelrig.dev/troubleshooting.html.
 
@@ -128,7 +137,7 @@ import { createRig, loadConfigFromEnv } from "modelrig";
 const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
 ```
 
-**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "recorded · unverified", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "Reported by your code", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
 
 ```ts
 const started = Date.now();
@@ -198,10 +207,44 @@ npx modelrig bakeoff --from-eval-cases example.support_summarize --variants defa
 const rig = createRig({ ...loadConfigFromEnv(), expectOrg: "your-org" });
 ```
 
+**Point your OTLP exporter at ModelRig.** ModelRig accepts OpenTelemetry traces at POST /v1/otlp/v1/traces on the ingest host (https://api.modelrig.ai unless you were given another), over OTLP/HTTP as protobuf or JSON, optionally gzipped. Authenticate with a `rig_sk_` key that carries the `ingest` scope; every run and step lands in that key's organization. Each trace becomes one run with one attested step per span: model, provider, operation, timing, status and token usage. Prompts, completions and tool arguments are not read.
+
+```bash
+# Standard OpenTelemetry exporter variables — most OTel SDKs read them as-is.
+OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://api.modelrig.ai/v1/otlp/v1/traces
+OTEL_EXPORTER_OTLP_TRACES_HEADERS=Authorization=Bearer%20rig_sk_…
+OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
+```
+
+**Set gen_ai.conversation.id to your conversation id.** A trace becomes a conversation when one of its spans carries `gen_ai.conversation.id` and it has `invoke_agent` spans. Each top-level `invoke_agent` span is a turn, in start order, and every model call under it becomes a step `turn:<n>.<k>`; the run is marked as a conversation and its episode key is your conversation id. Turns are numbered within a trace: when each user message arrives as its own trace, each shows as its own one-turn conversation run, and the shared conversation id groups them. If your framework does not emit an `invoke_agent` span itself, wrap each user message in one.
+
+```ts
+import { trace } from "@opentelemetry/api";
+
+const tracer = trace.getTracer("support-bot");
+
+// One span per user message. The model calls your framework makes inside it
+// become this conversation's turn steps; the conversation id joins the runs.
+export function turn<T>(conversationId: string, fn: () => Promise<T>): Promise<T> {
+  return tracer.startActiveSpan(
+    "invoke_agent support-bot",
+    { attributes: { "gen_ai.operation.name": "invoke_agent", "gen_ai.conversation.id": conversationId } },
+    async (span) => {
+      try {
+        return await fn();
+      } finally {
+        span.end();
+      }
+    },
+  );
+}
+```
+
 ## Capabilities (the full surface)
 
 - **Runs & projects** — the run → step → artifact graph at /runs, grouped by tag at /projects (the standard path above): https://modelrig.dev/instrumentation.html.
-- **Recorded calls (`rig.recordCall`, modelrig ≥ 0.11.0)** — a provider call your code makes itself (no lane fits) is reported after it returns: one row, priced at list for the served tier from the tokens you report, joined to the run/step, gradeable, badged "recorded · unverified" (ModelRig neither dispatched nor validated it), never in routed/conformance math: https://modelrig.dev/not-captured.html.
+- **Recorded calls (`rig.recordCall`, modelrig ≥ 0.11.0)** — a provider call your code makes itself (no lane fits) is reported after it returns: one row, priced at list for the served tier from the tokens you report, joined to the run/step, gradeable, badged "Reported by your code" ("recorded · unverified" with Developer details on; ModelRig neither dispatched nor validated it), never in routed/conformance math: https://modelrig.dev/not-captured.html.
+- **Diagnosable export drops (`SinkSendError.reasonCode`, modelrig ≥ 0.12.0)** — a batch the server rejects deterministically (`422`) names the database error code in the body's `reason_code` (e.g. `PGRST102`, `23514`); the SDK carries it on the thrown error and the exporter's `POISON-SKIP` line: https://modelrig.dev/troubleshooting.html.
 - **Tenants & statements** — a `subject` tag or project-scoped key becomes a per-customer tenant with a derived statement (attempted, verified, billable, true cost; never content or another org's data): https://modelrig.dev/tenants.html.
 - **Published receipts** — a signed, login-free receipt of a bake-off or route at /r/<id>, verifiable by anyone at /v1/verify/<id>; internal names aliased, content never leaves: https://modelrig.dev/receipts.html.
 - **Analytics query API** — a self-describing, read-only query surface over your own telemetry (`GET /v1/analytics/meta`, `POST /v1/analytics/query`): an allowlisted SELECT-only grammar that returns metadata only (enums, numbers, hashes), never content: https://modelrig.dev/optimization-loop.html.
@@ -230,6 +273,11 @@ const rig = createRig({ ...loadConfigFromEnv(), expectOrg: "your-org" });
 - Register it in Claude Code: add `{"mcpServers":{"modelrig":{"command":"npx","args":["modelrig-oracle"]}}}` to your MCP config.
 - Every answer carries `scope`, `probe_data_as_of`, and a staleness flag. A model absent from an answer is UNPROBED, not incapable — say that rather than implying a limit.
 
+## Framework recipes (OTLP) and generated media
+
+- FRAMEWORK RECIPES: send a framework's OpenTelemetry traces to `POST /v1/otlp/v1/traces` with a `rig_sk_` ingest key — Vercel AI SDK (https://modelrig.dev/recipe-vercel-ai-sdk.html), OpenAI Agents SDK (https://modelrig.dev/recipe-openai-agents-sdk.html), LangGraph / LangChain JS (https://modelrig.dev/recipe-langgraph-js.html), Mastra (https://modelrig.dev/recipe-mastra.html), LiveKit Agents (https://modelrig.dev/recipe-livekit-agents.html). Each trace is a run of attested steps (model, provider, operation, timing, status, token usage). Set `gen_ai.conversation.id` on a span and emit `invoke_agent` spans and the trace becomes a conversation: generations numbered `turn:<n>.<k>`, tool calls counted per turn. NOT shown: content (prompts/completions are not read — OTLP content extraction is behind a server gate that is OFF) and cost (attested steps are not priced). The recipe configs are marked untested — verify against your framework version.
+- GENERATION CALLS (modelrig ≥ 0.12.0): `rig.recordCall({ provider, model, usage: { tokensIn: 0, tokensOut: 0 }, units: { images?, seconds?, characters?, steps? }, latencyMs })` records an image/video/TTS/STT call priced per unit — from the registry's declared generation entry (sourced rate + as_of), else the vendored pricing map, else `pricingMissing` (never from token rates). Query them with `query_registry({ kind: "image_gen" | "video_gen" | "tts" | "stt" })`: declared-only, unprobed, never ranked. Save generated assets as `image/*` / `video/*` / `audio/*` artifacts (stored unscreened); approve or reject them in the Grade queue's Media approval tab; Costs shows approval rate per $ by model from ten approvals. Starter guide: https://modelrig.dev/template-creative-approval.html.
+
 ## Live routing & call-config (change what serves a route, no redeploy)
 
 - A route's CALL-CONFIG — which model/provider serves it + its sampling — can change WHILE THE APP RUNS, no YAML edit, no redeploy. Two modes: `pin` (one model serves now) and `experiment` (a weighted live A/B whose drawn arm records on `inferences.meta.arm`). Set it in the console's call-config card, or over MCP (below).
@@ -372,7 +420,7 @@ What `rig.runRaw` sends when you omit a knob. `timeoutMs` is tier-aware since 0.
 
 ## Grades (judgment on inferences, runs & artifacts)
 
-- A GRADE is a 0–1 score (with an optional comment) attached to a SUBJECT — an inference, a run (episode), or an artifact — from your code, a human, or an AI. The pass/fail verdict is DERIVED, never stored: `pass = score >= the route's threshold` (default 0.5). It attaches from the SDK (`rig.grade(subject, { score, kind })`) or posts over HTTP from a non-SDK surface: `POST /v1/grade` on modelrig-server, authenticated with a `rig_sk_` key carrying the `grade` scope, body `{inferenceId | episode, score, scale?, comment?, kind?, source?}`. The org is resolved from the key, never the request body; there is NO public/unauthenticated grade endpoint.
+- A GRADE is a 0–1 score (with an optional comment) attached to a SUBJECT — an inference, a run (episode), or an artifact — from your code, a human, or an AI. The pass/fail verdict is DERIVED, never stored: `pass = score >= the route's threshold` (default 0.5). It attaches from the SDK (`rig.grade(subject, { score, kind })`) or posts over HTTP from a non-SDK surface: `POST /v1/grade` on modelrig-server, authenticated with a `rig_sk_` key carrying the `grade` scope, body `{inferenceId | episode | artifactId, score, scale?, comment?, kind?, source?}` (`artifactId` grades an artifact). The org is resolved from the key, never the request body; there is NO public/unauthenticated grade endpoint.
 - Every run that ends gets one deterministic `run-outcome@v1` grade AUTOMATICALLY, with no model call: it scores 1.0 (a pass) iff the run succeeded AND no step failed AND every deterministic grade passed; otherwise 0.0 with a dominant-failure label.
 - An explicit, bounded model judge (`run-judge@v1`) can ADDITIONALLY score a run's outcome-achievement 0–1 as a `model-judge` grade — invoked only on demand (console button or `modelrig judge <runId>`), capped per-org per-day, and metered as a routed gateway call. There is NO automatic or scheduled model-judging.
 - SDK callers attach their own run- or artifact-level grade the same way: `rig.grade({ kind: "run", id: run_id }, { score, kind, comment? })` — the grader `kind` is one of human | model-judge | deterministic.
@@ -391,7 +439,7 @@ What `rig.runRaw` sends when you omit a knob. `timeoutMs` is tier-aware since 0.
 ## Use-case templates (start a task with a schema + guide)
 
 - A USE-CASE TEMPLATE is a starting point for putting one task on the loop: a starter output schema (with TODO enum slots you fill with your own taxonomy — never our invented values), a starter grading guide in the exact jsonb shape the settings path validates, a golden checklist of what ~20 goldens should cover, and an escalation starter. It carries NO number about how well any model does the task — it is a shape to fill in, and what you fill in is what your goldens and grades then measure. Overview: https://modelrig.dev/use-case-templates.html.
-- Nine ship today, one per task the loop can serve: **ticket triage** (https://modelrig.dev/template-ticket-triage.html), **document extraction** (https://modelrig.dev/template-document-extraction.html), **CS next action** (https://modelrig.dev/template-cs-next-action.html), **lead qualification** (https://modelrig.dev/template-lead-qualification.html), **compliance review** (https://modelrig.dev/template-compliance-review.html), **catalog cleansing** (https://modelrig.dev/template-catalog-cleansing.html), **call disposition QA** (https://modelrig.dev/template-call-disposition-qa.html), **financial classification** (https://modelrig.dev/template-financial-classification.html), and **medical classification** (https://modelrig.dev/template-medical-classification.html). Each machine-readable file also ships in the npm package at `node_modules/modelrig/templates/<use-case>.json`.
+- Eleven templates, one per task the loop can serve (all eleven ship in the npm package): **ticket triage** (https://modelrig.dev/template-ticket-triage.html), **document extraction** (https://modelrig.dev/template-document-extraction.html), **CS next action** (https://modelrig.dev/template-cs-next-action.html), **lead qualification** (https://modelrig.dev/template-lead-qualification.html), **compliance review** (https://modelrig.dev/template-compliance-review.html), **catalog cleansing** (https://modelrig.dev/template-catalog-cleansing.html), **call disposition QA** (https://modelrig.dev/template-call-disposition-qa.html), **financial classification** (https://modelrig.dev/template-financial-classification.html), **medical classification** (https://modelrig.dev/template-medical-classification.html), **support conversation** (https://modelrig.dev/template-support-conversation.html), and **creative approval** (https://modelrig.dev/template-creative-approval.html). Each machine-readable file also ships in the npm package at `node_modules/modelrig/templates/<use-case>.json`.
 - REGULATED ROWS: financial classification and medical classification handle regulated data. Their fit notes quote the published content custody terms verbatim — do not send protected health information or other regulated data through ModelRig until a BAA or DPA is in place — and name the today-path where those terms are not yet in place: a zero-retention route (or content custody off), so ModelRig holds no content and grading is limited to the scores and metadata your own systems send. Terms: https://modelrig.ai/terms/content-custody. No other compliance claim is made.
 - CHECK FIT FIRST (the overview applies these to each task): a task suits the loop when its distribution is stable (goldens stay valid), a rubric is worth writing (the grading guide), expert examples are on hand to grade, and volume makes calibration pay back. A task that fails one structurally is honest to leave at the teaching stage, not force through the loop.
 - TWO CONSUMPTION PATHS, one file: a human copies the schema and guide from the docs page (they run as-is; TODOs mark where to tighten); or a coding agent feeds the template through the tools that ALREADY EXIST — `create_route` drafts the route bundle from the starter schema (files only, never your source), then `set_task_settings` (the `task-settings` scope) authors the grading guide — the starter criteria run as written; sharpen them into your team's own words as you grade. No new tool, no server change. The starter guide has NO `pass_threshold` (that lives in the route's `grade:` block — the guide teaches the grader, the threshold derives the verdict).

package/docs/mcp-oracle.md

@@ -1,8 +1,17 @@
 # The MCP oracle
 
-> `modelrig-oracle` is published on npm (0.1.0, 2026-08-21) — the
-> `npx modelrig-oracle` commands on this page work everywhere. Nothing else on
+> `modelrig-oracle` is published on npm (0.4.0, 2026-10-25) — the
+> `npx modelrig-oracle` commands on this page work everywhere. The write tools
+> below (grading guides, `record_grade`, `get_call_config`, the read resources)
+> need **0.3.0 or later**; `record_outcome`, `query_conversations` and
+> `query_registry`'s `kind` filter need **0.4.0 or later**. If you ran an earlier
+> version, refresh once with `npx -y modelrig-oracle@latest`. Nothing else on
 > this site depends on it: llms.txt carries the same probed facts as plain text.
+>
+> **Grading guides on an older oracle.** 0.2.0 (2026-08-22) does not know
+> `grading_guide` and refuses a guide-only `set_task_settings` call — there,
+> write the guide with `PATCH /v1/tasks/<task>/settings` instead
+> ([Set up a grader](set-up-a-grader.md)).
 
 **What it is:** an MCP server that lets a coding agent ask what models can
 actually do — measured, dated, and priced by what conformance really costs —
@@ -31,7 +40,7 @@ anonymous record of the question's *shape* (see [Demand signal](#demand-signal))
 
 | Tool | Answers |
 |---|---|
-| `query_registry` | "Which models can serve strict JSON under $1/M?" — capability predicates plus a price bound, ranked by effective cost of conformance. |
+| `query_registry` | "Which models can serve strict JSON under $1/M?" — capability predicates plus a price bound, ranked by effective cost of conformance. `kind: image_gen` / `video_gen` / `tts` / `stt` lists the declared-only generation models instead — unprobed, never ranked, with the unit price ModelRig could source. |
 | `get_leaderboard` | Standings, optionally for one capability. Cheap-but-unreliable ranks below pricier-but-correct, because the ranking is cost per *conformant* output. |
 | `explain_pricing` | List price for one model, the effective cost of conformance beside it, and why they differ. |
 | `get_call_notes` | Per-model call-shaping quirks: parameters it rejects, shapes it needs, defaults that surprise. Plus any measured disagreement with the provider's claims. |
@@ -65,7 +74,9 @@ on purpose.
 | `decide_proposal` | Records a human's approve/reject on a proposal. Needs the `decision` scope. | Execute or apply the swap. |
 | `execute_swap` | Records an approved swap and **returns** the route-YAML diff to apply via a PR. Needs the `decision` scope. | Write any file; commit to your repo. |
 | `record_grade` | Attaches a 0–1 grade (with an optional comment) to one inference or a whole run — the signal the next cycle learns from. Needs the `grade` scope on the HTTP path. | Spend a billed call; guess attribution for an unknown subject. |
-| `get_task_settings` / `set_task_settings` | Reads / writes a task's governance settings and its **grading guide** (the rubric the task's grader is taught from). `set` needs the `task-settings` scope. | Change route config (the pass threshold lives in the route, and is refused here). |
+| `record_outcome` | Posts a delayed business outcome (resolved, booked, approved, CTR) as a `kind: outcome` grade for a conversation id, run, inference or artifact (`POST /v1/outcomes`); `idempotency_key` makes a retry a no-op. Needs the `ingest` scope. | Calibrate a grader on its own — outcome agreement is a receipt. |
+| `query_conversations` | Lists your agent's conversations (turns, cost, latest grade, outcomes, run-limit stops), or one conversation's turns with `run_id`. Metadata only. | Return message content. |
+| `get_task_settings` / `set_task_settings` | Reads / writes a task's governance settings and its **grading guide** (the rubric the task's grader is taught from). `set` needs the `task-settings` scope. The task must exist: a route (registered on route sync), a `runRaw` / `recordCall` step whose call set `tags.step`, or one created with `POST /v1/tasks` (HTTP only — there is no MCP tool to list or create tasks yet; `GET /v1/tasks` lists them). | Change route config (the pass threshold lives in the route, and is refused here). |
 | `get_call_config` | Reads a route's **current live** call-config — its version, mode, the model(s) it serves, the `effective` model + why (pin / YAML default), and any `drift`. Read-only. | Change anything. |
 | `set_call_config` | **Sets what serves a route right now** — pins a model or starts a live A/B — on the human's behalf. A real, versioned control-plane change. | Bypass the eligibility gate or the `route-config` scope; imply it is a draft. |
 | `rollback_call_config` | Restores a prior call-config version (re-appended as a new version — history is never deleted). | Delete history; escape the scope check. |

package/docs/quickstart.md

@@ -14,9 +14,10 @@ and three things happen at once:
 - **Grade** — you get schema-validated output, served only by a model you
   authorized, you attach grades from signals you already have, and you can replay that traffic against cheaper candidates in a
   bake-off before you change anything.
-- **Improve** — a winning candidate is a proposal you review, and nothing swaps
-  a model on its own. Never a silent swap: promote it in ModelRig (console or
-  MCP; versioned, reversible) or merge it as a PR if you keep routes in git.
+- **Improve** — a winning candidate is a proposal you review. Never a silent
+  swap: automated changes (opt-in) are proven before they apply, and a change
+  you promote in ModelRig (console or MCP) is versioned and reversible — or
+  merge it as a PR if you keep routes in git.
 
 The path is the same whoever does the work: **mint a key → install → confirm the
 org → define a route → wrap one pipeline execution in a run → see it on `/runs`.**
@@ -277,7 +278,7 @@ await rig.close(); // flushes the exporter — call once at shutdown
   embedding) is invisible until you record it:
 
 <!-- BLOCK:RECORD_CALL -->
-**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "recorded · unverified", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "Reported by your code", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
 
 ```ts
 const started = Date.now();
@@ -622,4 +623,4 @@ loader's own words, is in [route-bundles.md](./route-bundles.md#when-it-doesnt-w
   registry entry shows probed `structured_native`. Check the probed registry,
   not memory.
 
-Still stuck? Open an issue at <https://github.com/modelrig/modelrig/issues>.
+Still stuck? See the [Troubleshooting guide](https://modelrig.dev/troubleshooting.html).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelrig",
-  "version": "0.11.1",
+  "version": "0.12.0",
   "private": false,
   "description": "ModelRig — capability-aware AI gateway: routes-as-code, provider adapters, typed failures, budget envelopes, and local-first telemetry.",
   "license": "Apache-2.0",
@@ -63,7 +63,9 @@
     "templates/catalog-cleansing.json",
     "templates/call-disposition-qa.json",
     "templates/financial-classification.json",
-    "templates/medical-classification.json"
+    "templates/medical-classification.json",
+    "templates/support-conversation.json",
+    "templates/creative-approval.json"
   ],
   "scripts": {
     "build": "tsc",