modelrig 0.7.0 → 0.8.0

package/docs/quickstart.md

@@ -0,0 +1,333 @@
+# ModelRig quickstart — see, prove, and save on every model call
+
+ModelRig is the operating layer for your AI workstream. Each LLM task becomes a
+**versioned route file** (prompt template + JSON schema + candidate models +
+policy) that you call with one line — `rig.run("task.name", {input, tags})` —
+and three things happen at once:
+
+- **See** — the run lands in an **inspectable artifact store** (runs → steps →
+  artifacts, lineaged and integrity-hashed; see the
+  [instrumentation guide](instrumentation.md)), with cost/tokens/tier and a
+  verdict recorded against your tags.
+- **Prove** — you get schema-validated output, served only by a model you
+  authorized, and you can replay that traffic against cheaper candidates in a
+  bake-off before you change anything.
+- **Save** — a winning candidate is a change to your YAML you review; nothing
+  swaps a model on its own, and your routes are always YAML in your git.
+
+The rest of this page is Step 0 (two minutes, once), then the two ways to
+instrument your project — point your coding agent at it, or do it by hand.
+
+## Step 0 — create your org and key (two minutes, human-only)
+
+Account setup stays human: your coding agent never creates accounts, signs in,
+or holds credentials. Once, up front:
+
+1. Sign up and create your organization at
+   [app.modelrig.ai](https://app.modelrig.ai) — email and an org name is the
+   whole form.
+2. Issue an API key in the console (the `rig_sk_…` value is shown once — copy it).
+3. Export it beside your provider keys: `MODELRIG_API_KEY=rig_sk_…`.
+
+That key is the only ModelRig credential your app ever holds — never a database
+URL, never a service-role key. Provider keys never leave your process.
+
+**No account needed for the open lane:** with only your provider keys set,
+routes, probes and telemetry run entirely on your machine — the hosted console
+and synced telemetry are what the key unlocks. You can add it later.
+
+There are two ways your calls are keyed and billed, and you pick per route:
+
+- **Bring your own keys (BYOK).** Your provider keys stay in your environment;
+  ModelRig routes to the provider directly. The first 1,000,000 requests each
+  month are free, then a flat 2% of list price. On the raw lane (`rig.runRaw`)
+  the `timeoutMs` default is tier-aware — **flex ⇒ set `timeoutMs` in minutes**
+  (default 900 s since 0.5.0); see the
+  [raw-lane defaults table](route-bundles.md#raw-lane-defaults-rigrunraw).
+- **Managed keys.** ModelRig fronts the provider's inference cost, so managed
+  calls are provider cost + 2% from the first request — no key for you to hold.
+
+Same flat 2% margin either way; your negotiated provider discounts stay yours.
+
+## Point your coding agent at it
+
+The fastest migration is the one your agent performs. From inside your project,
+paste this into Claude Code, Codex, or Cursor:
+
+```
+Migrate this project to ModelRig routes (https://modelrig.dev).
+
+0. If MODELRIG_API_KEY is not set and I want hosted telemetry, STOP and ask me
+   to create an org and issue a rig_sk_ key at https://app.modelrig.ai. Never
+   create accounts, sign in, or handle credentials yourself. Without the key,
+   proceed in local-only mode — that is fully supported.
+1. Read https://modelrig.dev/quickstart.html and
+   https://modelrig.dev/route-bundles.html.
+2. Find every place this codebase calls an LLM API directly.
+3. For each call site, define a route bundle YAML (schema, candidate models,
+   policy, prompt template) and replace the direct call with
+   rig.run("<route.name>"). Keep prompt and schema semantics unchanged.
+4. Before naming any model capability, check the probed registry — declared
+   flags and probed behavior differ; trust the probed layer.
+5. Run the project's existing tests and report every file you changed.
+```
+
+Your agent reads the docs, finds your LLM call sites, and turns them into
+routes. It **won't** create accounts, sign in, or touch your API keys (those
+stay in the environment you set in Step 0), and it swaps call sites to
+`rig.run()` without changing prompt or schema semantics — same model, human-
+reviewed. The full autonomy ladder and same-model guarantees are in the
+[migration playbook](migration-playbook.md).
+
+## Or set it up by hand
+
+### 1. Install
+
+```bash
+pnpm add modelrig      # or npm i modelrig — Node >= 20
+```
+
+It builds a native module (`better-sqlite3`) on install, so the machine needs a
+toolchain.
+
+```ts
+import { createRig, loadConfigFromEnv } from "modelrig";
+```
+
+Environment (read once, in ModelRig's config layer only):
+
+```bash
+GEMINI_API_KEY=…        # only providers you configure get adapters
+OPENAI_API_KEY=…
+DEEPSEEK_API_KEY=…
+
+# Where your telemetry goes — pick ONE lane:
+MODELRIG_API_KEY=rig_sk_…                      # hosted: rows go to ModelRig,
+                                               # scoped to the org that key names
+                                               # SET IT ON EVERY service/worker/cron
+# (omit it and everything below still works, locally)
+MODELRIG_ROUTES_DIR=./modelrig/routes          # default
+MODELRIG_TELEMETRY_DB=./.modelrig/telemetry.db # default
+MODELRIG_ENVELOPE_BUDGET_USD=25                # default envelope budget
+```
+
+Provider keys never leave your process on either lane. A `rig_sk_` key is issued
+in the console at [app.modelrig.ai](https://app.modelrig.ai) and is the only
+ModelRig credential your app ever holds — never a database URL, never a
+service-role key.
+
+### 2. Define a route bundle
+
+One YAML file per route under `modelrig/routes/`:
+
+```yaml
+# modelrig/routes/example.support_summarize.yaml
+route: example.support_summarize   # the task handle you pass to rig.run
+version: 1                         # bump on ANY change
+schema: ./schemas/support_summarize.schema.json  # null for unstructured routes
+candidates:                        # THE candidate set — the invariant boundary.
+  - provider: openai               # Nothing outside this list can ever serve
+    model: gpt-5.4-mini            # the route: enforced by a branded type at
+  - provider: gemini               # compile time and a runtime guard.
+    model: gemini-3.1-flash-lite
+require: [schema_conformant]       # hard constraints on candidate eligibility
+prefer: [cost]                     # advisory ordering (cheapest first)
+prompt:
+  system: ./prompts/support_summarize.system.md
+  variables: [ticket, product, priorContext]
+policy:
+  retries: { content_invalid: 2, network: 4, capacity_shed: 3 }
+  timeout_ms: 60000
+  tier: flex                       # requested tier; served tier is recorded
+  json: native                     # native strict schema | json_mode (rung 2)
+  sampling: { temperature: 0.1, max_output_tokens: 8192 }
+  #                                  ^ preserve the original call's sampling;
+  #                                    absent = the adapter's own defaults
+```
+
+Prompt templates support `{{variable}}` substitution and capability-conditional
+blocks resolved against the **serving** candidate's flags:
+
+```
+Summarise this ticket for the next agent: {{ticket}}
+
+{{#if capability.grounded_native}}
+Use web search to verify recent events.
+{{/if}}
+{{#unless capability.structured_native}}
+CRITICAL: respond with a single JSON object, no markdown fences.
+{{/unless}}
+```
+
+Variables must be declared in `prompt.variables`; referencing an undeclared
+variable fails at load time, not at run time.
+
+The `schema` file is a JSON Schema. ModelRig validates output under **JSON
+Schema 2020-12** as the primary dialect and **also accepts draft-07** (a schema
+declaring either `$schema` compiles). Validation runs non-strict with union
+types allowed, and `format` is annotation-only — it is not asserted (an
+`email`/`date-time` format is documentation, not a constraint). Author 2020-12
+unless you have a draft-07 schema already; both work.
+
+### 3. Call it
+
+```ts
+const rig = createRig(loadConfigFromEnv());
+
+const result = await rig.run("example.support_summarize", {
+  input: { ticket: rawTicket, product: "Acme Cloud", priorContext: history },
+  tags: { run_id: "run-123", step: "summarize", customer: "acme" },
+  budget: { envelope: "run-123" },   // hard-stop cost envelope for the run
+});
+
+result.output;                 // JSON, already validated against your schema
+result.meta.provider;          // which candidate actually served
+result.meta.costEstimateUsd;   // priced from the pinned LiteLLM snapshot
+result.meta.servedTier;        // vs meta.requestedTier — downgrades visible
+result.meta.attemptsByClass;   // retries consumed, per failure class
+
+rig.close();
+```
+
+### 4. Wrap it in a run — the standard path
+
+**Every pipeline execution is a run.** A run is the standard way ModelRig records
+what your workstream did — not an appendix. Wrap the execution in a run context,
+name one **step** per model-call family, and **save the work products** as
+artifacts; then the console's **Runs** tab shows the run → step → artifact chain
+you can click through, and `/projects` groups it by tag.
+
+```ts
+const rig = createRig(loadConfigFromEnv());
+
+// One run per pipeline execution. run.scope isolates concurrent runs; use
+// run.start()/run.end() for one run at a time on a request path.
+await rig.artifacts.run.scope(
+  { pipeline: "support", episodeKey: ticketId },
+  async () => {
+    const result = await rig.run("example.support_summarize", {
+      input: { ticket, product, priorContext },
+      tags: { subject: customer, feature: "summarize" }, // run_id auto-stamped
+      stepKey: "summarize",                              // names this step on /runs
+    });
+    // Save the work product — prompt/raw/parsed all save the same way.
+    rig.artifacts.artifact.save(result.output, { name: "summary", type: "step_output" });
+  },
+);
+rig.close();
+```
+
+- **Tags:** carry `subject` (what the run is *about*) and `feature`. You no longer
+  hand-write `run_id` — **inside a run context the SDK stamps `tags.run_id`** (the
+  run's `episodeKey`, else its id) on every attempt row of both lanes; a value you
+  pass always wins, and outside a run context nothing is stamped. It also stamps
+  `tags.step` (the current step key) the same way; a value you pass always wins,
+  so name the step yourself with `stepKey` (or `tags.step`) when you want a
+  specific label.
+- **Raw lane:** a `rig.runRaw` call inside a run context records a ground-truth
+  step too (since 0.5.0), so grounded/cached pipelines fill the same graph.
+- **See it:** open `/runs` — the run, its steps, its artifacts. `modelrig status`
+  prints `runs recorded: N (last: …)` as the local mirror, and `modelrig validate`
+  warns if a rig records calls but never starts a run.
+
+The full artifact API — lineage (`link`), evaluations (`evaluate`), the reusable
+per-step seam, and `run.scope` vs `run.start` — is the
+[instrumentation guide](instrumentation.md); this section is the standard path
+that links to it.
+
+## When it fails, it fails typed
+
+```ts
+import { RigFailureError } from "modelrig";
+
+try {
+  await rig.run("example.support_summarize", opts);
+} catch (err) {
+  if (err instanceof RigFailureError) {
+    err.failure.class;   // content_invalid | capacity_shed | network | refusal
+                         // | cache_invalid | timeout | config_auth
+                         // | budget_exhausted | invariant_violation
+                         // | quality_rejected
+    err.failure.fixHint; // machine-readable remediation when available
+  }
+}
+```
+
+Retry budgets are **per class and non-fungible** — exhausting `network`
+retries never consumes the `content_invalid` budget. Terminal classes
+(`budget_exhausted`, `invariant_violation`) are never retried.
+`config_auth` (bad or missing credentials) is **non-retryable by contract**:
+it cannot appear in a bundle's retry budgets, is never retried on the same
+candidate, and the run loop advances straight to the next candidate — a dead
+key never burns backoff time. Failures that still billed tokens (refusals,
+truncations) carry their token usage, so envelopes and telemetry account the
+real spend.
+
+Each class — what it means, whether it retries, and how it backs off — is on [Routing & reliability](routing-reliability.md#5-failure-classes); how the whole candidate ladder runs (ordering, validate → repair → fall-through, and what is not built) is on that page too.
+
+## Where the telemetry goes
+
+Every attempt (not just every run) writes one row to package-owned SQLite,
+tagged with your `tags` — always, with no key. Setting `MODELRIG_API_KEY` turns
+on **api export mode**: a background exporter ships batches to the ingest API
+(`api.modelrig.ai`, override with `MODELRIG_INGEST_URL`), which the console
+reads — fire-and-forget, queue-on-failure; the inference path never blocks on it.
+Inspect live rows in your local SQLite file, or in the console once the key is
+set — never a database URL of your own.
+
+`MODELRIG_API_KEY` gates the exporter **per process**. Set it on EVERY service,
+worker, and cron that constructs a rig — a process without it buffers to its own
+local SQLite and ships nothing. A multi-service deployment that sets the key on
+only the web tier silently loses every row a worker or cron produced.
+
+Telemetry says what happened; **verdicts** say whether it was good. Attach a
+thumbs up/down to a call or a whole run with `rig.feedback(...)` — from the SDK
+or over HTTP for a non-SDK surface (see the
+[feedback protocol](feedback-protocol.md)) — and every run gets an automatic
+`run-outcome@v1` verdict, with an explicit, bounded model judge available when
+you want one (see [run verdicts](run-verdicts.md)).
+
+## Runtime requirements
+
+ModelRig assumes a **long-lived Node process with a writable disk**, and a few
+things follow from that. Check them before you deploy:
+
+- **Node ≥ 20**, and a native build toolchain at install time — it compiles
+  `better-sqlite3` (a C/C++ addon) for the local telemetry buffer. On a slim
+  container add the build essentials (`python3`, `make`, a C++ compiler).
+- **A writable disk.** Telemetry and captures go to a local SQLite file
+  (`MODELRIG_TELEMETRY_DB`, default `./.modelrig/telemetry.db`); the background
+  exporter batches from it. A read-only filesystem breaks this.
+- **Serverless (Vercel functions, AWS Lambda, Google Cloud Functions):**
+  supported, with two required adjustments. (1) Point the telemetry DB at the
+  only writable location — `MODELRIG_TELEMETRY_DB=/tmp/modelrig/telemetry.db`.
+  (2) `await rig.close()` **once per invocation** before the handler returns:
+  the function may freeze or be killed the moment you respond, and `close()` is
+  what drains the final rows (H1) — skip it and you silently lose the last
+  attempts of every invocation. Expect a native-module cold-start cost on the
+  first call after a scale-up.
+- **Edge runtimes (Vercel Edge, Cloudflare Workers) are unsupported.** They have
+  no native-addon support and no filesystem, so `better-sqlite3` cannot load at
+  all. Run ModelRig in a Node runtime; if an edge function must call a route,
+  put ModelRig behind a Node service it calls over HTTP.
+
+## When it doesn't work
+
+The first three failures below stop the bundle loading; the fourth lets it load
+and then misbehave, so it is the one to watch for. The full list, in the
+loader's own words, is in [route-bundles.md](./route-bundles.md#when-it-doesnt-work).
+
+- **The bundle won't load, naming a schema or prompt path.** The file isn't at
+  the path the bundle names — paths resolve relative to the bundle. Create it
+  there, or set `schema: null` for a free-form task.
+- **The bundle won't load, naming an undeclared variable.** A `{{name}}` in the
+  template isn't in `prompt.variables`. Add it (or fix the typo).
+- **The first call succeeds locally but nothing appears in the console.**
+  `MODELRIG_API_KEY` isn't set in that environment. Everything runs without it;
+  it only decides where telemetry lands.
+- **The call runs but the output isn't what the old code produced.** Usually a
+  wrong `json: native` — a claim about the model, valid only for one whose
+  registry entry shows probed `structured_native`. Check the probed registry,
+  not memory.
+
+Still stuck? Open an issue at <https://github.com/modelrig/modelrig/issues>.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "modelrig",
-  "version": "0.7.0",
+  "version": "0.8.0",
   "private": false,
   "description": "ModelRig — capability-aware AI gateway: routes-as-code, provider adapters, typed failures, budget envelopes, and local-first telemetry.",
   "license": "Apache-2.0",
@@ -24,17 +24,33 @@
   },
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
+  "//exports": "ADDITIVE sub-path map (RP-1). `.` preserves the historical bare entry; `./receipts` (Receipt@v1 shape + canonicalize/hash/classification) is exposed ONLY here, never re-exported from index.ts (keeps the public surface — and index.ts's LOC — untouched). `./data/*` + `./registry/*` keep the runtime data/registry files reachable now that an exports map turns resolution into an allowlist. The root `receipts.d.ts` re-export shim exists ALONGSIDE this so classic-`Node` moduleResolution consumers (e.g. modelrig-server) type-resolve the sub-path too.",
+  "exports": {
+    ".": { "types": "./dist/index.d.ts", "default": "./dist/index.js" },
+    "./receipts": { "types": "./dist/receipts/types.d.ts", "default": "./dist/receipts/types.js" },
+    "./package.json": "./package.json",
+    "./data/*": "./data/*",
+    "./registry/*": "./registry/*"
+  },
   "//files": "ALLOWLIST, not a denylist, and the difference is not academic. Without this field npm falls back to .npmignore, then to THIS DIRECTORY's .gitignore — never the repo root's. The root ignores `.env.*`, which is why git has never committed .env.local; the package-level .gitignore says nothing about it, so `npm pack` happily included it. A publish would have shipped 13 live secrets, the Supabase service-role key among them, into an immutable public tarball. Anything added here must be something we would hand a stranger. See __tests__/publish-safety.test.ts.",
+  "//files-docs": "GS-3: the customer-facing runs/onboarding docs ship so `npm i modelrig` + read-the-package teaches run→step→artifact OFFLINE (the C1-WALK newbie walkthrough surfaced that the tarball taught nothing about runs). These are the SAME files publish-oss already mirrors PUBLIC via the docs/** allowlist (oss-publish-policy §3) — shipping them in the tarball is consistent, not a new disclosure. Curated named set (NOT docs/**) so internal phase-*/verification reports stay out. docs/llms.txt is GENERATED by scripts/build-docs-site.ts (it also writes the committed copy here) and tripwire-tested against llmsTxt() so the shipped copy can never go stale. publish-safety.test.ts asserts each ships AND carries no leak pattern.",
   "//files-registry": "registry/{registry,leaderboard}.json are fail-open runtime reads (facts.ts/rows.ts degrade to the static baseline when absent) — shipped by R-1 F2 RULING so a clean install routes on the PROBED capability profile the docs and the DVP report describe, not the static fallback. Enumerated file entries, NOT the registry/ dir (it also holds build scripts + tests). Public CC BY 4.0.",
   "//files-data": "`data/` is a RUNTIME dependency, not a build input (R-1 clean-room audit, A1): getPricing() reads data/pricing-map.json from the package root at first cost lookup, with NO existsSync guard — so an absent data/ throws, not degrades. 0.1.0 shipped dist-only and broke the flagship 'effective cost' story on a clean `npm i`. It is public CC BY 4.0 pricing data (data/PROVENANCE.md) — safe to hand a stranger. The tarball clean-room test (__tests__/tarball-cleanroom.test.ts) installs the packed artifact and asserts getPricing() returns a real rate, so this can never silently regress again.",
   "files": [
     "dist",
     "data",
+    "receipts.d.ts",
     "registry/registry.json",
     "registry/leaderboard.json",
     "README.md",
     "CHANGELOG.md",
-    "LICENSE"
+    "LICENSE",
+    "docs/quickstart.md",
+    "docs/instrumentation.md",
+    "docs/how-it-fits.md",
+    "docs/migration-playbook.md",
+    "docs/agent-skill/SKILL.md",
+    "docs/llms.txt"
   ],
   "scripts": {
     "build": "tsc",

package/receipts.d.ts

@@ -0,0 +1,12 @@
+/**
+ * Root type shim for the `modelrig/receipts` sub-path export (RP-1).
+ *
+ * The package.json `exports` map points `modelrig/receipts` at
+ * `dist/receipts/types` for runtime (Node honours `exports`) and for modern
+ * (`Node16`/`NodeNext`/`Bundler`) TypeScript consumers via its `types`
+ * condition. Classic `moduleResolution: "Node"` consumers — including this
+ * monorepo's `modelrig-server` — ignore `exports` and resolve a bare subpath
+ * against the package root, so this file is what they type-resolve. Both paths
+ * land on the same declarations; there is no second source of truth.
+ */
+export * from "./dist/receipts/types";