modelrig 0.10.0 → 0.11.1

package/docs/quickstart.md

@@ -1,4 +1,4 @@
-# ModelRig quickstart — see, prove, and save on every model call
+# ModelRig quickstart — See. Grade. Improve.
 
 **The question this page answers: how do I get from nothing to my first run on `/runs`, with its steps and artifacts, in about ten minutes?**
 
@@ -11,11 +11,12 @@ and three things happen at once:
   artifacts, lineaged and integrity-hashed; see the
   [instrumentation guide](instrumentation.md)), with cost/tokens/tier and a
   grade recorded against your tags.
-- **Prove** — you get schema-validated output, served only by a model you
-  authorized, and you can replay that traffic against cheaper candidates in a
+- **Grade** — you get schema-validated output, served only by a model you
+  authorized, you attach grades from signals you already have, and you can replay that traffic against cheaper candidates in a
   bake-off before you change anything.
-- **Save** — a winning candidate is a change to your YAML you review; nothing
-  swaps a model on its own, and your routes are always YAML in your git.
+- **Improve** — a winning candidate is a proposal you review, and nothing swaps
+  a model on its own. Never a silent swap: promote it in ModelRig (console or
+  MCP; versioned, reversible) or merge it as a PR if you keep routes in git.
 
 The path is the same whoever does the work: **mint a key → install → confirm the
 org → define a route → wrap one pipeline execution in a run → see it on `/runs`.**
@@ -102,9 +103,16 @@ import { createRig, loadConfigFromEnv } from "modelrig";
 Environment (read once, in ModelRig's config layer only):
 
 ```bash
-GEMINI_API_KEY=…        # only providers you configure get adapters
+# Provider keys: only the providers you configure get adapters. Set the ones
+# your routes (or runRaw calls) use; these are the names the SDK reads.
+GEMINI_API_KEY=…
 OPENAI_API_KEY=…
 DEEPSEEK_API_KEY=…
+ANTHROPIC_API_KEY=…
+XAI_API_KEY=…                     # grok
+ZAI_API_KEY=…                     # Z.ai / Zhipu
+MODELRIG_DEEPINFRA_API_KEY=…
+MODELRIG_FIREWORKS_API_KEY=…
 
 # Where your telemetry goes — pick ONE lane:
 MODELRIG_API_KEY=rig_sk_…                      # hosted: rows go to ModelRig,
@@ -119,14 +127,14 @@ MODELRIG_ENVELOPE_BUDGET_USD=25                # default envelope budget
 ### 2. Confirm the org
 
 <!-- BLOCK:CONFIRM_ORG -->
-**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in. It asks GET /v1/whoami which organization the key belongs to, and the name it prints must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
+**Confirm the org before you trust an empty console.** Run `modelrig doctor` from the same environment your app runs in, once your app has started with the key. It asks GET /v1/whoami which organization the key belongs to and prints the org's name and, in parentheses, its slug; the name must match the org badge in the console header. If it names a different organization, your data is landing there: switch the console to that organization, or mint a key in the one you meant (https://modelrig.dev/orgs-and-keys.html).
 
 ```bash
 npx modelrig doctor
-# [modelrig] key rig_sk_…AbCd → org «Your Org» ✓  (ingest https://api.modelrig.ai, …)
+# [modelrig] key rig_sk_…AbCd → org «Your Org» (your-org) ✓  (ingest https://api.modelrig.ai, …)
 
-# No CLI in that environment? Ask the API directly:
-curl -H "Authorization: Bearer $MODELRIG_API_KEY" https://api.modelrig.ai/v1/whoami
+# No CLI in that environment? Ask the API directly (same host the SDK uses):
+curl -H "Authorization: Bearer $MODELRIG_API_KEY" "${MODELRIG_INGEST_URL:-https://api.modelrig.ai}/v1/whoami"
 ```
 <!-- /BLOCK:CONFIRM_ORG -->
 
@@ -176,6 +184,14 @@ CRITICAL: respond with a single JSON object, no markdown fences.
 Variables must be declared in `prompt.variables`; referencing an undeclared
 variable fails at load time, not at run time.
 
+The route references two files next to it, and both must exist or the route
+fails to load (`schema file not found` / `prompt.system file not found`).
+`npx modelrig init` writes this starter route's two files,
+`modelrig/routes/schemas/support_summarize.schema.json` and
+`modelrig/routes/prompts/support_summarize.system.md`, without overwriting
+files you already have. Then `npx modelrig validate` checks the bundle with no
+provider key.
+
 The `schema` file is a JSON Schema. ModelRig validates output under **JSON
 Schema 2020-12** as the primary dialect and **also accepts draft-07** (a schema
 declaring either `$schema` compiles). Validation runs non-strict with union
@@ -185,6 +201,23 @@ unless you have a draft-07 schema already; both work.
 
 ### 4. Call it
 
+No route yet, because you are observing first or calling providers through a
+`rig.runRaw` seam? You don't need one:
+
+<!-- BLOCK:RAW_RIG -->
+**No routes yet? Build the rig raw-only.** Observing a pipeline, or recording calls your code makes itself, needs no route files. Construct the rig with `routesDir: null`: `rig.runRaw`, `rig.runRawStream` and `rig.recordCall` work at once, and `rig.run` / `rig.runBundle` refuse with a teaching error until you add routes. Without it, createRig loads ./modelrig/routes: a directory you configured that is missing throws, and a missing default directory falls back to raw-only with a one-line warning.
+
+```ts
+import { createRig, loadConfigFromEnv } from "modelrig";
+
+// Raw-only on purpose: no route files needed. rig.runRaw, rig.runRawStream and
+// rig.recordCall work; rig.run needs routes (add them later, drop routesDir: null).
+const rig = createRig({ ...loadConfigFromEnv(), routesDir: null });
+```
+<!-- /BLOCK:RAW_RIG -->
+
+With the route above in place:
+
 ```ts
 const rig = createRig(loadConfigFromEnv());
 
@@ -200,7 +233,7 @@ result.meta.costEstimateUsd;   // priced from the pinned LiteLLM snapshot
 result.meta.servedTier;        // vs meta.requestedTier — downgrades visible
 result.meta.attemptsByClass;   // retries consumed, per failure class
 
-rig.close();
+await rig.close();             // flushes telemetry; await it once at shutdown (R6)
 ```
 
 ### 5. Wrap it in a run — the standard path
@@ -240,6 +273,37 @@ await rig.close(); // flushes the exporter — call once at shutdown
 
 - **Raw lane:** a `rig.runRaw` call inside a run context records a ground-truth
   step too (since 0.5.0), so grounded/cached pipelines fill the same graph.
+- **A call your code makes itself** (a direct provider client, a stream, an
+  embedding) is invisible until you record it:
+
+<!-- BLOCK:RECORD_CALL -->
+**Record a call your code makes itself.** A provider call your code makes directly (a client ModelRig has no lane for, a stream you read yourself, an embedding, a fallback) is invisible until you report it. Call `rig.recordCall` right after it returns, inside the run, with the tokens the provider reported: the call shows on /inferences as "recorded · unverified", joins its run and step, and counts toward cost. It is synchronous and never throws; invalid input warns once and records nothing (`inferenceId: null`). Spread optional fields only when the provider reported them, so the call type-checks under `exactOptionalPropertyTypes`.
+
+```ts
+const started = Date.now();
+const resp = await genai.models.generateContent({ model: "gemini-3.1-pro-preview", contents });
+const u = resp.usageMetadata;
+const finishReason = resp.candidates?.[0]?.finishReason;
+
+const rec = rig.recordCall({
+  provider: "gemini",
+  model: "gemini-3.1-pro-preview",
+  usage: {
+    tokensIn: u?.promptTokenCount ?? 0, // total input, cached included
+    tokensOut: (u?.candidatesTokenCount ?? 0) + (u?.thoughtsTokenCount ?? 0),
+    tokensCached: u?.cachedContentTokenCount ?? 0,
+    // optional fields: spread only when the provider reported them
+    ...(u?.thoughtsTokenCount !== undefined ? { reasoningTokens: u.thoughtsTokenCount } : {}),
+  },
+  servedTier: "standard", // the tier that SERVED the call: pass what the provider reports (flex, priority); it is priced at that tier
+  latencyMs: Date.now() - started,
+  stepKey: "summarize", // the step on /runs (inside run.scope / run.start)
+  ...(finishReason !== undefined ? { finishReason } : {}),
+});
+if (rec.inferenceId !== null) rig.grade({ kind: "inference", id: rec.inferenceId }, { score: 1, kind: "deterministic" });
+```
+<!-- /BLOCK:RECORD_CALL -->
+
 - **Reserved tags:** `run_id` and `step` are stamped by the SDK; the full list of
   reserved keys is in the
   [instrumentation guide](instrumentation.md#reserved-tag-keys-the-one-list).
@@ -260,7 +324,9 @@ warns if a rig records calls but never starts a run. If the page is empty:
 ```text
 1. npx modelrig doctor, from the environment your app runs in.
    no export sink configured → this process does not see MODELRIG_API_KEY: set it, redeploy, open a new shell.
+   malformed key             → the value is truncated, quoted or padded: paste the whole rig_sk_… secret again.
    AUTH FAILED (401 / 403)   → the key is mistyped, revoked or not accepted: mint a key, set it, redeploy, open a new shell.
+   whoami unreachable        → no network path to the ingest host (firewall, proxy, wrong MODELRIG_INGEST_URL): fix the route out, retry.
 2. The org doctor prints is not the org in the badge → wrong org: switch the console, or mint a key in the org you meant.
 3. /inferences has rows but /runs is empty → the calls are not inside run.scope / run.start.
 4. A run is missing a step or a cost → that call is not on a ModelRig lane (see not-captured).
@@ -268,6 +334,48 @@ warns if a rig records calls but never starts a run. If the page is empty:
 ```
 <!-- /BLOCK:EMPTY_CONSOLE -->
 
+### 7. Grade it, then prove a cheaper model
+
+<!-- BLOCK:GRADE_RUN -->
+**Attach a grade to a call or a whole run.** A grade is your judgment: a 0–1 score with an optional comment, from a validator, a downstream accept/reject, a human or a model. Grade one call by its inference id (`result.meta.inferenceId`, or the `inferenceId` rig.recordCall returns). Grade a whole run by its run id, `run.runId` on the handle `run.start` returns: that grade shows on the run page and on /projects. A run grade whose id is the run's `episodeKey` also shows, on the run page only. `run_id` is a reserved tag the SDK stamps, not a variable your code has. A grade whose id matches no call or run attaches to nothing. The write is synchronous and local and never throws into your pipeline.
+
+```ts
+const run = rig.artifacts.run.start({ pipeline: "support", episodeKey: ticketId, environment: "production" });
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },
+  tags: { subject: customer },
+  stepKey: "summarize",
+});
+run.end("succeeded");
+
+// one call: the inference id rides on every result
+rig.grade({ kind: "inference", id: result.meta.inferenceId }, { score: 1, kind: "deterministic", grader: "schema-check" });
+// the whole run: its id is run.runId from run.start (shows on the run page and on /projects)
+rig.grade({ kind: "run", id: run.runId }, { score: 0, kind: "human", comment: "missed the refund request" });
+```
+<!-- /BLOCK:GRADE_RUN -->
+
+<!-- BLOCK:BAKEOFF -->
+**Prove a cheaper model on your own traffic.** A bake-off replays your own inputs through the variants a route declares and reports the effective cost of conformance for each. `--variants` names entries in the route's `variants:` block (`default` is the route as declared), never model ids, and a variant can only narrow the route's candidates. Set `capture: true` first: replay works on traffic captured after that, not before. With no captured traffic (a run-based pipeline), promote a real /runs output to a golden (the console's make golden, or `promoteToEval`) and use `--from-eval-cases <task>`; its variants come from the route bundle named after the task. A bake-off measures and never switches anything.
+
+```bash
+# 1. in modelrig/routes/example.support_summarize.yaml: capture traffic, and declare
+#    a cheaper variant (narrowing only: its candidates must be in the route's list)
+capture: true
+variants:
+  - name: cheap
+    candidates:
+      - provider: gemini
+        model: gemini-3.1-flash-lite
+
+# 2. after traffic accumulates, compare the route as declared (default) with cheap
+npx modelrig bakeoff --route example.support_summarize --variants default,cheap --replay-last 50
+
+# no captured route traffic? promote a real /runs output to a golden, then
+npx modelrig bakeoff --from-eval-cases example.support_summarize --variants default,cheap
+```
+<!-- /BLOCK:BAKEOFF -->
+
 ## Change your route's model live (no redeploy)
 
 A route's **call-config** — which model/provider serves it, and its sampling — can be
@@ -375,7 +483,9 @@ const { bundle: runnable, arm } = resolveCallConfigForRun(bundle, resolved, {
   input,                         // stickyKey (if set) reads its unit from here…
   // unitKey: userId,            // …or pass the unit key explicitly (sticky assignment)
 });
-const result = await rig.runBundle(runnable, { input, tags, arm });
+// arm is string | null and RunOptions.arm is optional: pass it only when one was
+// drawn (a bare `arm` fails to compile under exactOptionalPropertyTypes).
+const result = await rig.runBundle(runnable, { input, tags, ...(arm !== null ? { arm } : {}) });
 // arm is null for a pin / no-override / non-experiment call ⇒ no meta.arm (byte-identical).
 ```
 
@@ -419,7 +529,7 @@ key never burns backoff time. Failures that still billed tokens (refusals,
 truncations) carry their token usage, so envelopes and telemetry account the
 real spend.
 
-Each class — what it means, whether it retries, and how it backs off — is on [Routing & reliability](routing-reliability.md#5-failure-classes); how the whole candidate ladder runs (ordering, validate → repair → fall-through, and what is not built) is on that page too.
+Each class — what it means, whether it retries, and how it backs off — is on [Routing & reliability](routing-reliability.md#5-failure-classes); that page also covers how the whole candidate ladder runs: the ordering, the validate → repair → fall-through sequence, and the parts that are not built yet.
 
 ## Where the telemetry goes
 
@@ -427,7 +537,9 @@ Every attempt (not just every run) writes one row to package-owned SQLite,
 tagged with your `tags` — always, with no key. Setting `MODELRIG_API_KEY` turns
 on **api export mode**: a background exporter ships batches to the ingest API
 (`api.modelrig.ai`, override with `MODELRIG_INGEST_URL`), which the console
-reads — fire-and-forget, queue-on-failure; the inference path never blocks on it.
+reads — fire-and-forget: a transient failure (network, 5xx) is retried, a
+rejected key (401/403) drops the rows and logs `AUTH FAILED`; the inference path
+never blocks on it.
 Inspect live rows in your local SQLite file, or in the console once the key is
 set — never a database URL of your own.
 
@@ -438,7 +550,7 @@ only the web tier silently loses every row a worker or cron produced.
 
 Turning **Optimization on** for a route (`capture: true`) is the **recommended
 default** — your traffic is what makes the routing smarter and the bill smaller,
-and everything it keeps you can see, export and delete in the console. Keeping the
+and everything it keeps you can see and export in the console. Keeping the
 prompts, outputs and evidence too is [content custody](content-custody.md), which
 is **on by default** — every new org is created managed, having accepted the
 published terms at signup — and captures into your managed store, scrubbed for the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelrig",
-  "version": "0.10.0",
+  "version": "0.11.1",
   "private": false,
   "description": "ModelRig — capability-aware AI gateway: routes-as-code, provider adapters, typed failures, budget envelopes, and local-first telemetry.",
   "license": "Apache-2.0",