modelrig 0.8.1 → 0.10.0

package/docs/how-it-fits.md

@@ -6,7 +6,7 @@ ModelRig sits between your code and the model providers. Your repo declares **ro
 
 Every call writes one telemetry row to a local SQLite file first — the run never blocks on the cloud. An async exporter mirrors rows to the console you are reading. Provider keys are the one thing that deliberately never leaves your machine; the exporter has no code path to them.
 
-Routing is half of it. ModelRig also records the **execution graph** of your workstream — the run, its steps, and the artifacts each step produced — versioned, hashed, and lineaged, into an inspectable store you can open: when someone asks why the pipeline did that, you pull up the run and show them. This is the **standard path**, not an add-on: every pipeline execution is a run — wrap it in `run.start` / `run.scope`, name a step per model-call family (name it with `stepKey`), save the work products, and inside that context the SDK auto-stamps `run_id` for you, so the run appears on `/runs` and `/projects` with no hand-written id (the SDK auto-stamps the `step` key inside a run context too; the `stepKey` you pass names the step, and always wins). The namespace is **on by default once a control plane is configured** (a ModelRig API key, or the self-host Supabase pair); a pure local-only rig stays off, and `MODELRIG_ARTIFACTS=0` opts out anywhere. Artifact *metadata and lineage* ride the same exporter; artifact *content* moves only with per-route consent, off by default. A failing artifact can be promoted to a regression case and replayed as a bake-off — see → prove → save is the loop, and the store is where seeing happens.
+Routing is half of it. ModelRig also records the **execution graph** of your workstream — the run, its steps, and the artifacts each step produced — versioned, hashed, and lineaged, into an inspectable store you can open: when someone asks why the pipeline did that, you pull up the run and show them. This is the **standard path**, not an add-on: every pipeline execution is a run — wrap it in `run.start` / `run.scope`, name a step per model-call family (name it with `stepKey`), save the work products, and inside that context the SDK auto-stamps `run_id` for you, so the run appears on `/runs` and `/projects` with no hand-written id (the SDK auto-stamps the `step` key inside a run context too; the `stepKey` you pass names the step, and always wins). The namespace is **on by default once a control plane is configured** (a ModelRig API key, or the self-host Supabase pair); a pure local-only rig stays off; to opt out of capture, set posture `off` in account settings (the console or the `set_org_settings` MCP tool) — the account posture is the control, not a deployment env var. Artifact *metadata and lineage* ride the same exporter; artifact *content* is a separate axis — the **recommended, encouraged** path, since content custody is what turns your captured prompts, outputs and evidence into analysis, but it stays consent-gated: the signup default is `managed` — every new org is created with content custody on, having accepted the published terms at signup — and opting out is one account setting (in the console or the `set_org_settings` MCP tool, set posture `off` or `metadata`, or use a `zero_retention` route). A failing artifact can be promoted to a regression case and replayed as a bake-off — see → prove → save is the loop, and the store is where seeing happens.
 
 What is retained beyond that is your choice, per route, and there are two lanes.
 
@@ -29,6 +29,7 @@ The five-step get-started is the front door; these are the shipped surfaces it d
 - **Published receipts** — publish a signed, login-free receipt of a bake-off or route at `/r/<id>` — counts, scores and hashes only, verifiable at `/v1/verify/<id>`; names aliased, content never leaves. See [Published receipts](receipts.md).
 - **Analytics query API** — a self-describing, read-only query surface over your own telemetry (`GET /v1/analytics/meta`, `POST /v1/analytics/query`): an allowlisted SELECT grammar that returns metadata only, never content. See [Optimization loop](optimization-loop.md).
 - **Provenance & trust** — provenance-first prompt assembly keeps third-party content out of the most-trusted slot; annotate a variable `x-trust: untrusted` to spotlight it. See [Provenance & trust](screening.md).
-- **Bake-offs — pairwise, disagreement & regret** — replay your own traffic, judge candidates pairwise with disagreement shown, and propose the lowest-**regret** arm under a quality gate (near-optimal, not merely cheapest). See [Bake-offs & replay](bakeoffs.md) and [Run verdicts](run-verdicts.md).
+- **Bake-offs — pairwise, disagreement & regret** — replay your own traffic, judge candidates pairwise with disagreement shown, and propose the lowest-**regret** arm under a quality gate (near-optimal, not merely cheapest). See [Bake-offs & replay](bakeoffs.md) and the [Grade protocol](grade-protocol.md).
+- **Live routing (call-config)** — change which model serves a route *live*, no redeploy: pin a model, or run a weighted A/B (the chosen arm records on `meta.arm`). The SDK reads the route's config just-before-call (fail-open; no override ⇒ byte-identical) and overlays it onto a bundle you serve through `rig.runBundle`. Set it in the console's call-config card, or have your coding agent set it over MCP (`set_call_config`, the `route-config` scope — explicit and reversible). See [Change your route's model live](quickstart.md#change-your-routes-model-live-no-redeploy) and [The MCP oracle](mcp-oracle.md#setting-a-routes-live-config).
 
 New claims are held to the same bar as the five steps: content custody stays early access (metadata and hashes), and pricing stays the launch-terms sentence.

package/docs/instrumentation.md

@@ -1,31 +1,42 @@
 # Artifact instrumentation (runs, steps, artifacts)
 
+**The question this page answers: how do I record what my pipeline actually did, run by run, step by step, without changing what it does?**
+
 ModelRig can record the **execution graph** of any pipeline your code runs — the
 run, its steps, and the artifacts each step produced (the prompt, the raw model
 text, the parsed output, the evidence it grounded on) — with lineage,
-evaluations, and an integrity hash for each. The console then renders that run as
+grades, and an integrity hash for each. The console then renders that run as
 a semantic chain you click through: *what your pipeline actually did*.
 
-It is **on by default when a control plane is configured** (an api-mode
-`rig_sk_` key, or the direct-mode Supabase pair) — metadata + hashes only. A
-**pure local-only rig** (no sink to ship to) stays off. Either way it is
-additive; set `MODELRIG_ARTIFACTS=0` to opt out, and with the namespace off it
-is inert: `run.start` is a no-op, `artifact.save` returns null, no rows are
-written, and your pipeline behaves byte-for-byte as before. `modelrig status`
-prints the current posture (on/off + why).
-
-> **Metadata + hashes only, this release.** The serialized value you save is
-> hashed (sha256) and the bytes are **discarded** — nothing but the metadata row
-> is persisted. Content custody (the blob path) ships in a later release; the
-> hash proves integrity today. A `zeroRetention` run refuses every artifact
-> fail-closed at the SDK gate.
+Recording your workstream is the **recommended, normal path** — it is what the
+run → step → artifact graph and every downstream analysis are built on — so it is
+**on by default when a control plane is configured** (an api-mode `rig_sk_` key,
+or the direct-mode Supabase pair) — metadata + hashes only. A **pure local-only
+rig** (no sink to ship to) stays off. Either way it is additive, and the **opt-out
+is one account setting**: set posture `off` in account settings (the console or the
+`set_org_settings` MCP tool) — the account posture is the customer control, not a
+deployment env var. (A pure local-only rig with no control plane is inert either
+way — `run.start` a no-op, `artifact.save` null, no rows written, your pipeline
+byte-for-byte as before.) `modelrig status` prints the current posture (on/off + why). Storing the artifact *content bytes* on top of this metadata is the
+second axis — the recommended, consent-gated [content custody](content-custody.md)
+path — never blurred with the telemetry axis here.
+
+> **What `artifact.save` keeps.** This page is the metadata axis: every saved
+> artifact records a metadata row with a sha256 integrity hash, whatever your
+> content setting. Whether the serialized *bytes* are kept as well is your
+> organization's [content custody](content-custody.md) posture: new
+> organizations are created `managed` (bytes kept in your managed store); posture
+> `metadata` keeps hashes only and `off` captures nothing. A `zeroRetention` run
+> refuses every artifact fail-closed at the SDK gate.
 
 ## The namespace toggle
 
 ```bash
-# On by default once a control plane is configured — nothing to set.
-export MODELRIG_ARTIFACTS=0                  # opt OUT (or force ON with =1 on a
-                                             #   local-only rig)
+# On by default once a control plane is configured — nothing to set. The CUSTOMER
+# opt-out is the account posture (set posture off in account settings / the
+# set_org_settings MCP tool), not this env var.
+export MODELRIG_ARTIFACTS=1                  # local-only namespace toggle: force ON
+                                             #   with =1 on a rig with no control plane
 export MODELRIG_ARTIFACT_STEPS=step_a,step_b   # optional: instrument a subset
                                             # (unset/empty = every instrumented step)
 ```
@@ -48,13 +59,13 @@ import { createRig, loadConfigFromEnv } from "modelrig";
 
 const rig = createRig(loadConfigFromEnv());
 
-// 1. Open a run context around the pipeline. episodeKey bridges feedback;
+// 1. Open a run context around the pipeline. episodeKey bridges grades;
 //    it is a grouping key, not a unique id — a natural choice is your own
-//    per-run attempt id.
+//    per-run attempt id. environment fills the Env column on /runs.
 const run = rig.artifacts.run.start({
   pipeline: "my-pipeline",
   episodeKey: runAttemptId,
-  environment: process.env.NODE_ENV,
+  environment: process.env.NODE_ENV,   // e.g. "production", "staging", "test"
 });
 try {
   for (const step of steps) {
@@ -74,10 +85,10 @@ try {
     if (p && parsed) rig.artifacts.artifact.link(parsed, p, "derived_from");
     if (raw && parsed) rig.artifacts.artifact.link(parsed, raw, "derived_from");
 
-    // 4. Evaluations (optional, deterministic): flag conformance, degraded
+    // 4. Grades (optional, deterministic): flag conformance, degraded
     //    provenance, anything you can decide without an LLM judge.
-    if (parsed) rig.artifacts.artifact.evaluate(parsed, {
-      evaluator: "schema-check", kind: "deterministic", passed: conforms(result.json),
+    if (parsed) rig.grade({ kind: "artifact", id: parsed.id }, {
+      score: conforms(result.json) ? 1 : 0, kind: "deterministic", grader: "schema-check",
     });
   }
   run.end("succeeded");
@@ -130,19 +141,29 @@ process** (`required dimension "project" missing from run tags — costs … wil
 under-attributed on /costs`) — on the routed lane and the raw lane alike. It is a
 hygiene warning, never an error: the run proceeds.
 
-**Conventional tag keys** (documented, not enforced): `subject` (what a run is
-*about* — a ticker, a client alias), `feature`, `project` (flat grouping),
-`step` (names the step this call belongs to), and `run_id` (bridges a call to a
-run's `episodeKey`). **Inside a `run.start` / `run.scope` context the SDK stamps
-both `run_id` and `step` for you** — `run_id` = the run's
-`episodeKey`, or its id when you gave none; `step` = the current step key (your
-explicit `stepKey`, else `tags.step`, else the auto `route#n` / `provider/model#n`
-key) — on every attempt row of both lanes, unless you passed your own value (yours
-always wins). Outside a run context nothing is stamped, so the row is
-byte-identical. This is why runs are the standard path: a `run.start` alone fills
-both `/runs` (the graph) and `/projects` (the tag tree), with no hand-written id.
-ModelRig owns `lane`, `zdr_enforced`, `rung`, and — on the hosted gateway —
-`key_ownership`; you cannot override those.
+### Reserved tag keys (the one list)
+
+Any other key is yours. These are reserved:
+
+| Key | Who sets it | What it means |
+|---|---|---|
+| `subject` | you | Your end-customer's **opaque** id. It powers the per-customer tenant statements on `/tenants`; hash it if it looks like PII ([tenants.md](./tenants.md)). A tag named `client` or `tenant` is only a cost dimension. |
+| `feature` | you | The product feature the call serves. |
+| `user` | you | The end user, as an opaque id. |
+| `session` | you | The conversation or session the call belongs to. |
+| `cost_center` | you | The team or budget to attribute spend to. |
+| `run_id` | **the SDK**, inside a run | The run's `episodeKey`, or its id when you gave none. **Don't set it yourself.** A value you pass overrides the stamp, so the call no longer joins its run on `/runs` and the console's Run ID group-by disagrees with `/runs`. Put your own attempt id in a key of your own, such as `pipeline_attempt_id`. |
+| `step` | **the SDK**, inside a run | The current step key: your explicit `stepKey`, else `tags.step`, else an auto `route#n` / `provider/model#n` key. Name steps with `stepKey`. |
+| `lane`, `zdr_enforced`, `rung`, `key_ownership` | ModelRig | Set by the SDK or the hosted gateway. You can't override them. |
+| `replay`, `replay_capture`, `variant`, `cutover` | ModelRig tooling | Set by bake-offs and replays. |
+
+The five keys you set are first-class group-bys in the cost and coverage views, so use them verbatim; `modelrig validate` warns about near-misses such as `costcenter` or `team`. `project` and `environment` are common conventions, not reserved.
+
+**Inside a `run.start` / `run.scope` context the SDK stamps `run_id` and `step`
+on every attempt row of both lanes.** Outside a run context nothing is stamped,
+so the row is byte-identical. This is why runs are the standard path: a
+`run.start` alone fills both `/runs` (the graph) and `/projects` (the tag tree),
+with no hand-written id.
 
 **How they compose.** Inside a `run.start` context, both `rig.run` and
 `rig.runRaw` record a **ground-truth step** and carry the run's `run_id` tag, so
@@ -168,18 +189,177 @@ run.end("succeeded");
 > that wrapped raw calls in `run.start` got an **empty tree**. (Raw steps are
 > metadata only — no artifact *content* is written on the raw lane.)
 
+## Live call-config and A/B arms
+
+Instrumentation records *what your pipeline did*; **call-config** changes *which model
+does it*, live, without a redeploy — and the two meet on the telemetry row. When a route
+runs a live A/B (`mode='experiment'`), the arm that served each call is recorded on
+`inferences.meta.arm`, so the run → step → artifact graph and the console's A/B readout can
+group outcomes by arm. This section is how that field gets populated; the call-config API
+itself (the resolver, the fail-open contract, the console card, and the MCP write path) is
+in the quickstart's [Change your route's model live](quickstart.md#change-your-routes-model-live-no-redeploy).
+
+**Overlay live call-config onto a bundle.** A route's live config is read
+**just-before-call** through a resolver and merged over the bundle you serve through
+`rig.runBundle()` (never `rig.runRaw` — the raw lane owns no bundle to overlay). The read
+is fail-open and, with no override, byte-identical to the deploy-time bundle:
+
+```ts
+import {
+  createApiCallConfigFetcher, createCallConfigResolver, resolveCallConfigForRun,
+} from "modelrig";
+
+const resolver = createCallConfigResolver({
+  fetcher: createApiCallConfigFetcher({ ingestUrl, apiKey: process.env.MODELRIG_API_KEY! }),
+  ttlMs: 30_000,   // cached; ETag-revalidated — no per-call network cost when fresh
+});
+
+// Per call: resolve the config, draw an arm locally (pin/no-override ⇒ arm is null).
+const resolved = await resolver.getCallConfig(route);
+const { bundle: runnable, arm } = resolveCallConfigForRun(bundle, resolved, { input });
+
+// Thread the arm onto the run so it lands on inferences.meta.arm.
+await rig.artifacts.run.scope({ pipeline: "report", episodeKey: runId }, async () => {
+  const out = await rig.runBundle(runnable, { input, tags: { subject, feature }, arm, stepKey });
+  rig.artifacts.artifact.save(out.output, { name: "section.output", type: "step_output" });
+});
+```
+
+**How `meta.arm` gets populated.** `resolveCallConfigForRun(bundle, resolved, { input, unitKey })`
+returns `{ bundle, arm }`: for an experiment it draws one weighted arm and returns its stable
+name; for a pin, a null/`live`/malformed resolution, or no override it returns the bundle
+unchanged and `arm: null`. You pass that `arm` on `RunOptions.arm`, and the attempt-meta
+producer stamps it onto **every recorded row's `meta.arm`** for that call (a non-meta-safe
+value is dropped). **Absent `arm` ⇒ no `meta.arm`** — byte-identical to a non-experiment
+run. Drawing is **sticky**: pass `unitKey` (or let `stickyKey` read a field from `input`) and
+the same unit always draws the same arm, so the A/B is measured honestly across calls.
+
+The console reads per-arm outcomes at `GET /v1/console/ab-readout?route=<route>` (samples,
+conformance, and cost grouped by arm; non-experiment rows, which carry no arm, are ignored).
+`meta.arm` sits in the recorded-metadata inventory alongside the rest of the attempt
+envelope — see [content custody](content-custody.md#attempt-envelope-inferencesmeta-attemptmetav1).
+
 ## Never throws into your pipeline
 
 Every entry point is fail-open: a telemetry-buffer write that fails is logged and
 dropped, never thrown into your code. Instrumentation that runs *after* an
 expensive model call must never be the thing that kills the run.
 
+## Which org am I exporting to?
+
+A valid key minted in the wrong org fails silently: every ingest call succeeds,
+and the runs land in someone else's console. So the first time a process flushes
+telemetry, the SDK asks `GET /v1/whoami` once and logs where it is exporting:
+
+```text
+[modelrig] exporting to org «Acme Analytics» (acme-analytics) as key «production exporter» rig_sk_…ab12
+```
+
+The call runs in the background. It never delays or drops a batch, and if it
+fails it says nothing (a rejected key already gets its own loud `AUTH FAILED`
+line). To have the SDK check the org for you, pass the slug you expect in code:
+
+| Config option | Type | Default | What it does |
+|---|---|---|---|
+| `expectOrg` | `string` (org slug) | absent | When the key resolves to a different org, logs `[modelrig] ORG MISMATCH: key resolves to «a», expectOrg is «b» — telemetry is landing in «a»` loudly on every flush once the org is known. It never throws and never drops telemetry: the rows still land in the key's org. |
+
+```ts
+const rig = createRig({ ...loadConfigFromEnv(), expectOrg: "acme-analytics" });
+```
+
+It is a code option, not an environment variable. To gate a deploy or CI step
+on the same check, run `modelrig doctor --expect-org acme-analytics`. It exits 1 and
+prints both slugs when the key belongs to a different org. More on keys,
+orgs and the console picker: [Orgs and keys](https://modelrig.dev/orgs-and-keys.html).
+
 ## See it
 
+Check first that the console is showing the org your key exports to:
+
+<!-- BLOCK:CONFIRM_ORG -->
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+
+```bash
+npx modelrig doctor
+# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+
+# No CLI in that environment? Ask the API directly:
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+```
+<!-- /BLOCK:CONFIRM_ORG -->
+
 The console's **Runs** tab lists every instrumented run; `/runs/[id]` is the
-semantic chain (artifacts grouped by step, with hashes, costs, and evaluation
+semantic chain (artifacts grouped by step, with hashes, costs, and grade
 badges) and `/artifacts/[id]` shows one artifact's lineage, versions, and
-evaluations. Enabling the flag also lights up the `/setup` artifacts line.
+grades. Enabling the flag also lights up the `/setup` artifacts line.
+
+## Grades (`rig.grade`)
+
+Capture records **what** ran; a **grade** records whether it was **good**.
+A grade is a 0–1 **score** (with an optional comment) your code — or a human, or
+an AI — attaches to a subject: one model call (an **inference**), a whole run
+(an **episode**), or an artifact. It is the signal per-route
+[optimization coverage](grade-protocol.md) is built from — the evidence level
+that decides how much confidence a swap proposal carries.
+
+```ts
+// One call — the inference id rides on every RunResult:
+const result = await rig.run("example.support_summarize", { input, tags: { subject } });
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "human" });
+
+// A whole run: the subject id is the run's run_id (its episodeKey, or its id).
+rig.grade({ kind: "run", id: runAttemptId }, { score: 0.2, comment: "CAGR off by 10x", kind: "human" });
+```
+
+- **The subject** is `{ kind: "inference", id }` (`result.meta.inferenceId`),
+  `{ kind: "run", id }` (the run's `run_id` — its `episodeKey`, or its id), or
+  `{ kind: "artifact", id }`. **`opts`** is `{ score, scale?, comment?, kind,
+  source? }` — the `score` in `[0, 1]` is the primitive (up = 1, down = 0), and
+  the pass/fail verdict is derived from it against the route threshold. A subject
+  naming no known inference, run, or artifact attaches to nothing — attribution
+  is never guessed.
+- **Emit it from a signal you already have.** Most pipelines already decide
+  pass/fail somewhere — a schema validator, a downstream accept/reject, a human
+  QA gate. Emit `rig.grade` from that (score 1 on a clean pass, 0 on a rejection)
+  rather than adding a new review step.
+- **Never on your hot path.** The write is synchronous and local (SQLite), never a
+  network call and never a failure your code can see; the cloud mirror rides the
+  telemetry exporter's normal cadence.
+- **Why it matters.** A route with **≥10 attached grades** reaches **Rung 1** on
+  `/coverage`; a route with no schema to conform to (an `output: text` prose
+  route, or a schema-less JSON route) gains proposal confidence only this way. Grades are also what let `/insights`
+  surface a task that is degrading.
+- **From a non-SDK surface** (an end user clicking thumbs-down in your web app),
+  post the same fields to `POST /v1/grade` with a `rig_sk_` key carrying the
+  `grade` scope — the org is resolved from the key, never the body. See the
+  [grade protocol](grade-protocol.md) for the full row shape and the ladder.
+
+## Promote a golden, then bake it off (`promoteToEval` → `--from-eval-cases`)
+
+A run's artifacts are the raw material for a **durable eval case**. Promote a real
+output into the org's eval suite and it becomes replayable evidence — the path a
+**raw/run-based pipeline** (a `rig.runRaw` seam with no declared route) uses to
+prove a cheaper model, alongside the route-scoped `bakeoff --route … --replay-last
+N`.
+
+```ts
+// Promote an artifact you saved this run into the org's eval suite.
+const caseId = rig.artifacts.artifact.promoteToEval(handle, { task: "report.section" });
+// `corrected` set ⇒ a golden-corrected pair; absent ⇒ a failure case.
+// Unresolvable task ⇒ a warned no-op (returns null, never throws).
+```
+
+```bash
+# Replay the promoted eval cases through variants — the raw/run-based lane.
+modelrig bakeoff --from-eval-cases report.section --variants default,cheap
+```
+
+- `promoteToEval(ref, { task?, corrected?, origin? })` binds the case to a route by
+  `task` (from `opts`, else the task the artifact was saved with this session), and
+  is also reachable from the console's one-click **"make golden"**.
+- `--route` needs a **declared route**; `--from-eval-cases <task>` is the path for a
+  `runRaw` seam that owns no route. Either way a bake-off **measures** — it never
+  switches what serves a route; a swap is a human-approved PR.
 
 ## A reusable per-step seam
 
@@ -205,17 +385,10 @@ function recordStep(
   const parsed = rig.artifacts.artifact.save(work.json, { name: `${stepKey}.output`, type: "step_output", task: stepKey, stepKey });
   if (p && parsed) rig.artifacts.artifact.link(parsed, p, "derived_from");
   if (raw && parsed) rig.artifacts.artifact.link(parsed, raw, "derived_from");
-  if (parsed) rig.artifacts.artifact.evaluate(parsed, { evaluator: "schema-check", kind: "deterministic", passed: work.conforms });
+  if (parsed) rig.grade({ kind: "artifact", id: parsed.id }, { score: work.conforms ? 1 : 0, kind: "deterministic", grader: "schema-check" });
 }
 ```
 
 Call `recordStep(step, { … })` once per stage inside the `run.start` / `run.end`
 context above, and every entry point stays fail-open — a save that fails is
 logged and dropped, never thrown into your pipeline.
-
-> **In-repo example (reconstructed via onboarding-test run-2).** ModelRig's own
-> customer-zero pipeline (the InferWealth v3 report generator) was stripped back
-> to a clean slate to be re-onboarded through this exact doc. Its live seam is
-> reconstructed as part of onboarding-test run-2; until then, the snippet above
-> is the reference. Do not resurrect a deleted seam from git history — writing a
-> fresh one is the point.