modelrig 0.11.0 → 0.12.0

package/docs/mcp-oracle.md

@@ -1,8 +1,17 @@
 # The MCP oracle
 
-> `modelrig-oracle` is published on npm (0.1.0, 2026-08-21) — the
-> `npx modelrig-oracle` commands on this page work everywhere. Nothing else on
+> `modelrig-oracle` is published on npm (0.4.0, 2026-10-25) — the
+> `npx modelrig-oracle` commands on this page work everywhere. The write tools
+> below (grading guides, `record_grade`, `get_call_config`, the read resources)
+> need **0.3.0 or later**; `record_outcome`, `query_conversations` and
+> `query_registry`'s `kind` filter need **0.4.0 or later**. If you ran an earlier
+> version, refresh once with `npx -y modelrig-oracle@latest`. Nothing else on
 > this site depends on it: llms.txt carries the same probed facts as plain text.
+>
+> **Grading guides on an older oracle.** 0.2.0 (2026-08-22) does not know
+> `grading_guide` and refuses a guide-only `set_task_settings` call — there,
+> write the guide with `PATCH /v1/tasks/<task>/settings` instead
+> ([Set up a grader](set-up-a-grader.md)).
 
 **What it is:** an MCP server that lets a coding agent ask what models can
 actually do — measured, dated, and priced by what conformance really costs —
@@ -31,7 +40,7 @@ anonymous record of the question's *shape* (see [Demand signal](#demand-signal))
 
 | Tool | Answers |
 |---|---|
-| `query_registry` | "Which models can serve strict JSON under $1/M?" — capability predicates plus a price bound, ranked by effective cost of conformance. |
+| `query_registry` | "Which models can serve strict JSON under $1/M?" — capability predicates plus a price bound, ranked by effective cost of conformance. `kind: image_gen` / `video_gen` / `tts` / `stt` lists the declared-only generation models instead — unprobed, never ranked, with the unit price ModelRig could source. |
 | `get_leaderboard` | Standings, optionally for one capability. Cheap-but-unreliable ranks below pricier-but-correct, because the ranking is cost per *conformant* output. |
 | `explain_pricing` | List price for one model, the effective cost of conformance beside it, and why they differ. |
 | `get_call_notes` | Per-model call-shaping quirks: parameters it rejects, shapes it needs, defaults that surprise. Plus any measured disagreement with the provider's claims. |
@@ -65,7 +74,9 @@ on purpose.
 | `decide_proposal` | Records a human's approve/reject on a proposal. Needs the `decision` scope. | Execute or apply the swap. |
 | `execute_swap` | Records an approved swap and **returns** the route-YAML diff to apply via a PR. Needs the `decision` scope. | Write any file; commit to your repo. |
 | `record_grade` | Attaches a 0–1 grade (with an optional comment) to one inference or a whole run — the signal the next cycle learns from. Needs the `grade` scope on the HTTP path. | Spend a billed call; guess attribution for an unknown subject. |
-| `get_task_settings` / `set_task_settings` | Reads / writes a task's governance settings and its **grading guide** (the rubric the task's grader is taught from). `set` needs the `task-settings` scope. | Change route config (the pass threshold lives in the route, and is refused here). |
+| `record_outcome` | Posts a delayed business outcome (resolved, booked, approved, CTR) as a `kind: outcome` grade for a conversation id, run, inference or artifact (`POST /v1/outcomes`); `idempotency_key` makes a retry a no-op. Needs the `ingest` scope. | Calibrate a grader on its own — outcome agreement is a receipt. |
+| `query_conversations` | Lists your agent's conversations (turns, cost, latest grade, outcomes, run-limit stops), or one conversation's turns with `run_id`. Metadata only. | Return message content. |
+| `get_task_settings` / `set_task_settings` | Reads / writes a task's governance settings and its **grading guide** (the rubric the task's grader is taught from). `set` needs the `task-settings` scope. The task must exist: a route (registered on route sync), a `runRaw` / `recordCall` step whose call set `tags.step`, or one created with `POST /v1/tasks` (HTTP only — there is no MCP tool to list or create tasks yet; `GET /v1/tasks` lists them). | Change route config (the pass threshold lives in the route, and is refused here). |
 | `get_call_config` | Reads a route's **current live** call-config — its version, mode, the model(s) it serves, the `effective` model + why (pin / YAML default), and any `drift`. Read-only. | Change anything. |
 | `set_call_config` | **Sets what serves a route right now** — pins a model or starts a live A/B — on the human's behalf. A real, versioned control-plane change. | Bypass the eligibility gate or the `route-config` scope; imply it is a draft. |
 | `rollback_call_config` | Restores a prior call-config version (re-appended as a new version — history is never deleted). | Delete history; escape the scope check. |

package/docs/quickstart.md

@@ -1,4 +1,4 @@
-# ModelRig quickstart — see, prove, and save on every model call
+# ModelRig quickstart — See. Grade. Improve.
 
 **The question this page answers: how do I get from nothing to my first run on `/runs`, with its steps and artifacts, in about ten minutes?**
 
@@ -11,11 +11,13 @@ and three things happen at once:
   artifacts, lineaged and integrity-hashed; see the
   [instrumentation guide](instrumentation.md)), with cost/tokens/tier and a
   grade recorded against your tags.
-- **Prove** — you get schema-validated output, served only by a model you
-  authorized, and you can replay that traffic against cheaper candidates in a
+- **Grade** — you get schema-validated output, served only by a model you
+  authorized, you attach grades from signals you already have, and you can replay that traffic against cheaper candidates in a
   bake-off before you change anything.
-- **Save** — a winning candidate is a change to your YAML you review; nothing
-  swaps a model on its own, and your routes are always YAML in your git.
+- **Improve** — a winning candidate is a proposal you review. Never a silent
+  swap: automated changes (opt-in) are proven before they apply, and a change
+  you promote in ModelRig (console or MCP) is versioned and reversible — or
+  merge it as a PR if you keep routes in git.
 
 The path is the same whoever does the work: **mint a key → install → confirm the
 org → define a route → wrap one pipeline execution in a run → see it on `/runs`.**
@@ -102,9 +104,16 @@ import { createRig, loadConfigFromEnv } from "modelrig";
 Environment (read once, in ModelRig's config layer only):
 
 ```bash
-GEMINI_API_KEY=…        # only providers you configure get adapters
+# Provider keys: only the providers you configure get adapters. Set the ones
+# your routes (or runRaw calls) use; these are the names the SDK reads.
+GEMINI_API_KEY=…
 OPENAI_API_KEY=…
 DEEPSEEK_API_KEY=…
+ANTHROPIC_API_KEY=…
+XAI_API_KEY=…                     # grok
+ZAI_API_KEY=…                     # Z.ai / Zhipu
+MODELRIG_DEEPINFRA_API_KEY=…
+MODELRIG_FIREWORKS_API_KEY=…
 
 # Where your telemetry goes — pick ONE lane:
 MODELRIG_API_KEY=rig_sk_…                      # hosted: rows go to ModelRig,
@@ -119,14 +128,14 @@ MODELRIG_ENVELOPE_BUDGET_USD=25                # default envelope budget
 ### 2. Confirm the org
 
 <!-- BLOCK:CONFIRM_ORG -->
-**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in, once your app has started with the key. It asks GET /v1/whoami which organization the key belongs to and prints the org's name and, in parentheses, its slug; the name must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
 
 ```bash
 npx modelrig doctor
-# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+# [modelrig] key rig_sk_…AbCd → org «Your Org» (your-org) ✓  (ingest https://api.modelrig.ai, …)
 
-# No CLI in that environment? Ask the API directly:
-curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+# No CLI in that environment? Ask the API directly (same host the SDK uses):
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" "${MODELRIG_INGEST_URL:-https://api.modelrig.ai}/v1/whoami"
 ```
 <!-- /BLOCK:CONFIRM_ORG -->
 
@@ -176,6 +185,14 @@ CRITICAL: respond with a single JSON object, no markdown fences.
 Variables must be declared in `prompt.variables`; referencing an undeclared
 variable fails at load time, not at run time.
 
+The route references two files next to it, and both must exist or the route
+fails to load (`schema file not found` / `prompt.system file not found`).
+`npx modelrig init` writes this starter route's two files,
+`modelrig/routes/schemas/support_summarize.schema.json` and
+`modelrig/routes/prompts/support_summarize.system.md`, without overwriting
+files you already have. Then `npx modelrig validate` checks the bundle with no
+provider key.
+
 The `schema` file is a JSON Schema. ModelRig validates output under **JSON
 Schema 2020-12** as the primary dialect and **also accepts draft-07** (a schema
 declaring either `$schema` compiles). Validation runs non-strict with union
@@ -185,6 +202,23 @@ unless you have a draft-07 schema already; both work.
 
 ### 4. Call it
 
+No route yet, because you are observing first or calling providers through a
+`rig.runRaw` seam? You don't need one:
+
+<!-- BLOCK:RAW_RIG -->
+**No routes yet? Build the rig raw-only.** Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with `routesDir: null`: `rig.runRaw`, `rig.runRawStream` and `rig.recordCall` work at once, and `rig.run` / `rig.runBundle` refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.
+
+```ts
+import { createRig, loadConfigFromEnv } from "modelrig";
+
+// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
+// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
+const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
+```
+<!-- /BLOCK:RAW_RIG -->
+
+With the route above in place:
+
 ```ts
 const rig = createRig(loadConfigFromEnv());
 
@@ -200,7 +234,7 @@ result.meta.costEstimateUsd;   // priced from the pinned LiteLLM snapshot
 result.meta.servedTier;        // vs meta.requestedTier — downgrades visible
 result.meta.attemptsByClass;   // retries consumed, per failure class
 
-rig.close();
+await rig.close();             // flushes telemetry; await it once at shutdown (R6)
 ```
 
 ### 5. Wrap it in a run — the standard path
@@ -240,6 +274,37 @@ await rig.close(); // flushes the exporter — call once at shutdown
 
 - **Raw lane:** a `rig.runRaw` call inside a run context records a ground-truth
   step too (since 0.5.0), so grounded/cached pipelines fill the same graph.
+- **A call your code makes itself** (a direct provider client, a stream, an
+  embedding) is invisible until you record it:
+
+<!-- BLOCK:RECORD_CALL -->
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "Reported by your code", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+
+```ts
+const started = Date.now();
+const resp = await genai.models.generateContent({ model: "gemini-3.1-pro-preview", contents });
+const u = resp.usageMetadata;
+const finishReason = resp.candidates?.[0]?.finishReason;
+
+const rec = rig.recordCall({
+  provider: "gemini",
+  model: "gemini-3.1-pro-preview",
+  usage: {
+    tokensIn: u?.promptTokenCount ?? 0, // total input, cached included
+    tokensOut: (u?.candidatesTokenCount ?? 0) + (u?.thoughtsTokenCount ?? 0),
+    tokensCached: u?.cachedContentTokenCount ?? 0,
+    // optional fields: spread only when the provider reported them
+    ...(u?.thoughtsTokenCount !== undefined ? { reasoningTokens: u.thoughtsTokenCount } : {}),
+  },
+  servedTier: "standard", // the tier that SERVED the call: pass what the provider reports (flex, priority); it is priced at that tier
+  latencyMs: Date.now() - started,
+  stepKey: "summarize", // the step on /runs (inside run.scope / run.start)
+  ...(finishReason !== undefined ? { finishReason } : {}),
+});
+if (rec.inferenceId !== null) rig.grade({ kind: "inference", id: rec.inferenceId }, { score: 1, kind: "deterministic" });
+```
+<!-- /BLOCK:RECORD_CALL -->
+
 - **Reserved tags:** `run_id` and `step` are stamped by the SDK; the full list of
   reserved keys is in the
   [instrumentation guide](instrumentation.md#reserved-tag-keys-the-one-list).
@@ -260,7 +325,9 @@ warns if a rig records calls but never starts a run. If the page is empty:
 ```text
 1. npx modelrig doctor, from the environment your app runs in.
    no export sink configured → this process does not see MODELRIG_API_KEY: set it, redeploy, open a new shell.
+   malformed key             → the value is truncated, quoted or padded: paste the whole rig_sk_… secret again.
    AUTH FAILED (401 / 403)   → the key is mistyped, revoked or not accepted: mint a key, set it, redeploy, open a new shell.
+   whoami unreachable        → no network path to the ingest host (firewall, proxy, wrong MODELRIG_INGEST_URL): fix the route out, retry.
 2. The org doctor prints is not the org in the badge → wrong org: switch the console, or mint a key in the org you meant.
 3. /inferences has rows but /runs is empty → the calls are not inside run.scope / run.start.
 4. A run is missing a step or a cost → that call is not on a ModelRig lane (see not-captured).
@@ -268,6 +335,48 @@ warns if a rig records calls but never starts a run. If the page is empty:
 ```
 <!-- /BLOCK:EMPTY_CONSOLE -->
 
+### 7. Grade it, then prove a cheaper model
+
+<!-- BLOCK:GRADE_RUN -->
+**Attach a grade to a call or a whole run.** A grade is your judgment: a 0–1 score with an optional comment, from a validator, a downstream accept/reject, a human or a model. Grade one call by its inference id (`result.meta.inferenceId`, or the `inferenceId` rig.recordCall returns). Grade a whole run by its run id, `run.runId` on the handle `run.start` returns: that grade shows on the run page and on /projects. A run grade whose id is the run's `episodeKey` also shows, on the run page only. `run_id` is a reserved tag the SDK stamps, not a variable your code has. A grade whose id matches no call or run attaches to nothing. The write is synchronous and local and never throws into your pipeline.
+
+```ts
+const run = rig.artifacts.run.start({ pipeline: "support", episodeKey: ticketId, environment: "production" });
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },
+  tags: { subject: customer },
+  stepKey: "summarize",
+});
+run.end("succeeded");
+
+// one call: the inference id rides on every result
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "deterministic", grader: "schema-check" });
+// the whole run: its id is run.runId from run.start (shows on the run page and on /projects)
+rig.grade({ kind: "run", id: run.runId }, { score: 0, kind: "human", comment: "missed the refund request" });
+```
+<!-- /BLOCK:GRADE_RUN -->
+
+<!-- BLOCK:BAKEOFF -->
+**Prove a cheaper model on your own traffic.** A bake-off replays your own inputs through the variants a route declares and reports the effective cost of conformance for each. `--variants` names entries in the route's `variants:` block (`default` is the route as declared), never model ids, and a variant can only narrow the route's candidates. Set `capture: true` first: replay works on traffic captured after that, not before. With no captured traffic (a run-based pipeline), promote a real /runs output to a golden (the console's make golden, or `promoteToEval`) and use `--from-eval-cases <task>`; its variants come from the route bundle named after the task. A bake-off measures and never switches anything.
+
+```bash
+# 1. in modelrig/routes/example.support_summarize.yaml: capture traffic, and declare
+#    a cheaper variant (narrowing only: its candidates must be in the route's list)
+capture: true
+variants:
+  - name: cheap
+    candidates:
+      - provider: gemini
+        model: gemini-3.1-flash-lite
+
+# 2. after traffic accumulates, compare the route as declared (default) with cheap
+npx modelrig bakeoff --route example.support_summarize --variants default,cheap --replay-last 50
+
+# no captured route traffic? promote a real /runs output to a golden, then
+npx modelrig bakeoff --from-eval-cases example.support_summarize --variants default,cheap
+```
+<!-- /BLOCK:BAKEOFF -->
+
 ## Change your route's model live (no redeploy)
 
 A route's **call-config** — which model/provider serves it, and its sampling — can be
@@ -375,7 +484,9 @@ const { bundle: runnable, arm } = resolveCallConfigForRun(bundle, resolved, {
   input,                         // stickyKey (if set) reads its unit from here…
   // unitKey: userId,            // …or pass the unit key explicitly (sticky assignment)
 });
-const result = await rig.runBundle(runnable, { input, tags, arm });
+// arm is string | null and RunOptions.arm is optional: pass it only when one was
+// drawn (a bare `arm` fails to compile under exactOptionalPropertyTypes).
+const result = await rig.runBundle(runnable, { input, tags, ...(arm !== null ? { arm } : {}) });
 // arm is null for a pin / no-override / non-experiment call ⇒ no meta.arm (byte-identical).
 ```
 
@@ -419,7 +530,7 @@ key never burns backoff time. Failures that still billed tokens (refusals,
 truncations) carry their token usage, so envelopes and telemetry account the
 real spend.
 
-Each class — what it means, whether it retries, and how it backs off — is on [Routing & reliability](routing-reliability.md#5-failure-classes); how the whole candidate ladder runs (ordering, validate → repair → fall-through, and what is not built) is on that page too.
+Each class — what it means, whether it retries, and how it backs off — is on [Routing & reliability](routing-reliability.md#5-failure-classes); that page also covers how the whole candidate ladder runs: the ordering, the validate → repair → fall-through sequence, and the parts that are not built yet.
 
 ## Where the telemetry goes
 
@@ -427,7 +538,9 @@ Every attempt (not just every run) writes one row to package-owned SQLite,
 tagged with your `tags` — always, with no key. Setting `MODELRIG_API_KEY` turns
 on **api export mode**: a background exporter ships batches to the ingest API
 (`api.modelrig.ai`, override with `MODELRIG_INGEST_URL`), which the console
-reads — fire-and-forget, queue-on-failure; the inference path never blocks on it.
+reads — fire-and-forget: a transient failure (network, 5xx) is retried, a
+rejected key (401/403) drops the rows and logs `AUTH FAILED`; the inference path
+never blocks on it.
 Inspect live rows in your local SQLite file, or in the console once the key is
 set — never a database URL of your own.
 
@@ -438,7 +551,7 @@ only the web tier silently loses every row a worker or cron produced.
 
 Turning **Optimization on** for a route (`capture: true`) is the **recommended
 default** — your traffic is what makes the routing smarter and the bill smaller,
-and everything it keeps you can see, export and delete in the console. Keeping the
+and everything it keeps you can see and export in the console. Keeping the
 prompts, outputs and evidence too is [content custody](content-custody.md), which
 is **on by default** — every new org is created managed, having accepted the
 published terms at signup — and captures into your managed store, scrubbed for the
@@ -510,4 +623,4 @@ loader's own words, is in [route-bundles.md](./route-bundles.md#when-it-doesnt-w
   registry entry shows probed `structured_native`. Check the probed registry,
   not memory.
 
-Still stuck? Open an issue at <https://github.com/modelrig/modelrig/issues>.
+Still stuck? See the [Troubleshooting guide](https://modelrig.dev/troubleshooting.html).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelrig",
-  "version": "0.11.0",
+  "version": "0.12.0",
   "private": false,
   "description": "ModelRig — capability-aware AI gateway: routes-as-code, provider adapters, typed failures, budget envelopes, and local-first telemetry.",
   "license": "Apache-2.0",
@@ -63,7 +63,9 @@
     "templates/catalog-cleansing.json",
     "templates/call-disposition-qa.json",
     "templates/financial-classification.json",
-    "templates/medical-classification.json"
+    "templates/medical-classification.json",
+    "templates/support-conversation.json",
+    "templates/creative-approval.json"
   ],
   "scripts": {
     "build": "tsc",