modelrig 0.1.0 → 0.3.0

package/CHANGELOG.md

@@ -0,0 +1,158 @@
+# Changelog
+
+All notable changes to the `modelrig` npm package are recorded here. The public
+surface is one thing with two halves — this package and the docs/registry site
+at [modelrig.dev](https://modelrig.dev). They ship together under one release
+ceremony (see `docs/ModelRig/oss-publish-policy.md`, §6). The generated docs
+carry an "as of modelrig vX.Y.Z" stamp so a reader can see when the docs and the
+published package diverge.
+
+This project adheres to [Semantic Versioning](https://semver.org/). Dates are
+UTC.
+
+## [0.3.0] — 2026-08-21
+
+The `modelrig-oracle` MCP server ships to npm alongside this release (its own
+package, 0.1.0) — `npx modelrig-oracle` now works as the docs describe.
+
+CE-2 raw-lane parity — three new provider knobs on `rig.runRaw` /
+`rig.runRawStream` (and the OpenAI-compatible `/v1/chat/completions` raw lane),
+so the Lane-B report-pipeline seam (`callProvider` → `rig.runRaw`) can swap
+behaviour-identically. Every knob is ADDITIVE: absent ⇒ byte-identical to the
+0.2.0 request shape.
+
+### Added — `RawRunInput` (all additive, absent ⇒ byte-identical)
+
+- **`grounding?: { mode: "native" }`** — native provider web grounding on the
+  raw lane (F1). Maps to the adapter's native-grounding directive (gemini →
+  the `googleSearch` tool). GEMINI-ONLY in v1: declared on any other provider
+  throws `RigFailureError` with class `invariant_violation` PRE-DISPATCH (no
+  meter, no provider call, no telemetry row). Interplay with `cache.key` needs
+  no code — the gemini `cachedContent` path excludes tools by design (the
+  `googleSearch` tool is expected to be baked into the customer's cache resource
+  at creation).
+- **`serviceTier?: "standard" | "flex" | "priority"`** — service tier on the raw
+  lane (F2). Forwarded to the adapter's tier path; `RunMeta.requestedTier` now
+  reports the DECLARED tier (previously hardcoded `"standard"` — an honesty fix).
+  ModelRig does NOT degrade flex→standard on the raw lane; a capacity rejection
+  surfaces classified (`capacity_shed`) for the customer's own retry ladder.
+- **`reasoning?: { level: "minimal" | "low" | "medium" | "high" }`** —
+  provider-neutral reasoning/thinking effort (F3). Gemini maps `level` to
+  `thinkingConfig.thinkingLevel` (replacing the hardcoded `"high"`). GEMINI-ONLY
+  in v1: declared on another provider throws `invariant_violation` pre-dispatch;
+  declared on a non-thinking-capable gemini model is rejected classified by the
+  adapter (never a silent drop, never an unguarded provider 400).
+
+### Added — `AdapterRequest`
+
+- **`reasoning?: ReasoningParams`** (+ exported `ReasoningParams` /
+  `ReasoningLevel` types) — the provider-neutral reasoning shape adapters map to
+  their native field.
+
+### Added — HTTP gateway (`/v1/chat/completions`, raw lane only)
+
+- Top-level OpenAI-standard `temperature` / `top_p` / `max_tokens` /
+  `service_tier` are now forwarded onto the raw lane (the open CE-6 co-item);
+  `modelrig.grounding` / `modelrig.reasoning` ride in the `modelrig` body object.
+  A malformed `grounding.mode` / `reasoning.level` is a teaching 4xx. The SLUG
+  lane still ignores all generation params (the verified route owns them).
+
+### Added — route loader
+
+- Load-time WARN when a route's output schema has an ARRAY root — OpenAI strict
+  structured output requires a top-level object, so an array-root schema cannot
+  use strict enforcement on OpenAI candidates (the A5 co-item). The route still
+  loads and serves as declared.
+
+### Confirmed (already true on 0.2.0, now pinned by regression tests)
+
+- **F5** — an invalid/evicted customer `cache.key` on the raw lane surfaces as a
+  thrown `RigFailureError` with `.failure.class === "cache_invalid"`; the raw
+  lane is single-dispatch, so a silent un-cached retry is structurally
+  impossible.
+- **F6** — success `RunMeta` carries `servedTier` / `tokensCached` / `latencyMs`
+  (and provider/model/cost); a per-attempt failure's class + provider/model
+  travel on the thrown error's `.failure`.
+
+## [0.2.0] — unreleased (staged for the R-A ceremony)
+
+- Ships the probed capability registry and leaderboard (`registry/registry.json`, `registry/leaderboard.json`) so a clean install routes on the probed profile instead of the static fallback (R-1 F2 ruling). The `modelrig-oracle` MCP server remains unreleased and is flagged as such in the docs.
+
+`0.1.0` was frozen on 2026-08-07 and shipped **dist-only** (`files: [dist,
+README, LICENSE]`). The SDK grew a large public surface across the following six
+weeks while the published artifact stood still, and one runtime dependency —
+`data/` — was never in the tarball at all. This release makes the published
+package match the workspace and self-tests that it does.
+
+### Fixed — packaging (the reason for this release)
+
+- **`data/` now ships.** `getPricing()` reads `data/pricing-map.json` (and the
+  optional `data/pricing-overlay.json`) from the package root at runtime, with
+  no `existsSync` guard — so on `0.1.0` the very first cost lookup threw
+  `ENOENT` on a clean `npm i modelrig`, breaking the flagship "effective cost"
+  story out of the box. `data` is now in the `files` allowlist. It is public
+  pricing data (CC BY 4.0; see `data/PROVENANCE.md`), safe to distribute.
+- **Regression pinned forever.** A new hermetic test (`npm pack` → install the
+  tarball into a temp dir → load the *installed* package) asserts
+  `getPricing()` returns a real rate and the `modelrig` bin runs. Packaging
+  breaks are now caught before publish, not by a customer.
+- **`modelrig --help` / `-h` / `help` exit 0** with usage on stdout (was: fell
+  through to the unknown-command path and exited 1).
+
+### Added — public API surface accumulated since 0.1.0
+
+None of the below existed in the published `0.1.0`; instructions written for the
+current docs failed against it with missing exports. They are all in this tag:
+
+- **Inspectable artifact store** — the `artifacts` namespace (runs → steps →
+  artifacts, lineaged and integrity-hashed), `resolveArtifactsConfig`,
+  per-artifact and per-run evaluation (`artifact.evaluate` / `run.evaluate`).
+- **Run verdicts** — the automatic deterministic `run-outcome@v1` on every run,
+  the explicit bounded model judge `run-judge@v1` (`modelrig judge <runId>`),
+  and the feedback protocol (`rig.feedback`, `POST /v1/feedback`,
+  `computeCoverage`).
+- **Cache accounting & routing** — cache-aware `estimateCostUsd` (cache-read
+  and cache-write buckets), `effectiveInputRate` / `classifyCacheVerdict`, the
+  `cache: auto` route directive with the miss-streak alert, and the customer
+  cache-handle pass-through `rig.run(task, { input, cache: { key, provider } })`
+  / `rig.runRaw({ …, cache: { key } })` (CE-1) — the customer owns the TTL
+  heartbeat; ModelRig never creates, refreshes, or deletes the resource.
+- **Observe / init (T0–T2)** — `modelrig observe` (scaffold the record-step seam
+  behind `MODELRIG_ARTIFACTS=1`, no route migration), `modelrig init --scope`
+  (draft a route bundle per call site), `modelrig validate` (keyless structural
+  check of `rig.yaml` + every bundle), and `modelrig verify-swap`
+  (shadow-equivalence report for a same-model swap).
+- **Route sampling params (CE-6)** — `policy.sampling: { temperature?, top_p?,
+  max_output_tokens? }` on a route, validated at load and sent to the provider
+  as declared; a sub-1.0 temperature on a Gemini-3 candidate loads with a
+  load-time advisory. **This is in the tag.**
+- **OTLP ingest** — bring your own traces (`normalizeOtlpTraces`,
+  `startOtlpEmitterIfConfigured`, the OTel semantic-convention map).
+- **Raw lane** (`runRaw` / `runRawStream`), streaming (`runStream`), the
+  grounding-inject rung, the `openai-compat` adapter host, and the Anthropic /
+  Grok adapters.
+- **Bake-offs & the optimization loop** — `runBakeoff`, `replayVariant`,
+  `computeProposal`, the Wilson-interval stats, the swap ledger + verified-
+  savings odometer, and the scheduled Rig Analyst Insights surface.
+
+### Fixed — behaviour
+
+- **H1 close-drain.** `rig.close()` now resolves only after the exporter's final
+  flush drains and after the artifacts streams drain (stores close last), so a
+  one-shot CLI or a serverless per-invocation `close()` can no longer lose the
+  last rows. One-shot callers **must** `await rig.close()`.
+
+### Notes
+
+- Requires **Node ≥ 20**. Installs a native module (`better-sqlite3`), so the
+  machine needs a build toolchain. Assumes a long-lived process with a writable
+  disk; serverless and edge callers: see the Runtime requirements section of the
+  [quickstart](https://modelrig.dev/quickstart.html).
+- Route output schemas validate under **JSON Schema 2020-12** (primary), with
+  **draft-07** also accepted. See the quickstart / route-bundles reference.
+
+## [0.1.0] — 2026-08-07
+
+Initial public release: capability-aware gateway, routes-as-code, provider
+adapters (Gemini / OpenAI / DeepSeek), typed failures, budget envelopes, and
+local-first SQLite telemetry.

package/README.md

@@ -32,25 +32,28 @@ This runs with **only a provider key** — no account, no signup.
 export GEMINI_API_KEY=...       # or OPENAI_API_KEY, ANTHROPIC_API_KEY, …
 ```
 
-**2. Define a route** at `modelrig/routes/research.growth_scenarios.yaml`:
+**2. Define a route** at `modelrig/routes/example.support_summarize.yaml` (the
+example the package ships, in `modelrig/routes/`):
 
 ```yaml
-route: research.growth_scenarios
+route: example.support_summarize
 version: 1
-schema: ./schemas/growth_scenarios.schema.json   # JSON Schema for the output
+schema: ./schemas/support_summarize.schema.json  # JSON Schema (2020-12 recommended; draft-07 still accepted)
 candidates:                                      # THE candidate set — nothing else can serve
-  - provider: gemini
-    model: gemini-3.1-pro-preview
   - provider: openai
     model: gpt-5.4-mini
+  - provider: gemini
+    model: gemini-3.1-flash-lite
+  - provider: deepseek
+    model: deepseek-chat
 require: [schema_conformant]
 prefer: [cost]
 prompt:
-  system: ./prompts/growth_scenarios.system.md   # {{var}} + {{#if capability.X}} blocks
-  variables: [ticker, financials, priorSteps]
+  system: ./prompts/support_summarize.system.md  # {{var}} + {{#if capability.X}} blocks
+  variables: [ticket, product, priorContext]
 policy:
   retries: { content_invalid: 2, network: 4, capacity_shed: 3 }
-  timeout_ms: 300000
+  timeout_ms: 60000
   tier: flex
   json: native
 ```
@@ -61,9 +64,9 @@ policy:
 import { createRig, loadConfigFromEnv } from "modelrig";
 
 const rig = createRig(loadConfigFromEnv());        // keys from GEMINI_API_KEY etc.
-const result = await rig.run("research.growth_scenarios", {
-  input: { ticker, financials, priorSteps },       // template variables
-  tags: { run_id, step: "growth_scenarios", ticker },
+const result = await rig.run("example.support_summarize", {
+  input: { ticket, product, priorContext },        // template variables
+  tags: { run_id, step: "summarize" },
   budget: { envelope: run_id },                    // hard-stop cost envelope
 });
 // result.output  — schema-validated JSON (ajv), regardless of serving model
@@ -72,6 +75,45 @@ const result = await rig.run("research.growth_scenarios", {
 rig.close();
 ```
 
+**Data-governance routing (opt-in).** Add `zero_retention` to a route's
+`require`, or pass `zeroRetention: true` on a single `rig.run(...)` call, to
+route only to zero-retention–designated endpoints. It is a hard, opt-in filter
+that fails closed: if no candidate is designated zero-retention, the run raises
+the no-eligible-candidate error rather than dispatching to a retaining endpoint.
+(Capability gating is on by default here — unlike OpenRouter's opt-in
+`require_parameters`.)
+
+**Request-time routing controls (opt-in).** A single `rig.run(...)` call can
+narrow the candidate set: `maxPriceUsdPerMTok` drops any model over a price
+ceiling (unknown price + a ceiling → dropped, fail-closed), and
+`providers: { only, ignore }` restricts or excludes providers. `maxPrice` is a
+candidate *filter* (which models are eligible) — distinct from `budget.envelope`,
+which is a *spend cap* on the run's total cost; both compose, both fail closed on
+unpriced models.
+
+**Customer cache handle (opt-in).** If your code already engineered provider
+caching, carry it through: `rig.run(task, { input, cache: { key, provider } })`
+passes a customer-owned cache identifier (a Gemini `cachedContents/<id>`, or a
+`prompt_cache_key` value) verbatim to the matching-provider candidate. It wins
+over the route's `cache: auto` stamp, and a non-matching fallback dispatches
+without it. ModelRig carries the handle and prices the hits — it never creates,
+refreshes, or deletes the resource, so a long job needs your own TTL heartbeat
+(see the [caching lifecycle](https://modelrig.dev/caching-lifecycle.html) doc).
+The Lane-B `rig.runRaw({ provider, model, …, cache: { key } })` seam takes the
+same field.
+
+`rig.runRaw` also accepts
+three provider knobs, each additive — absent ⇒ byte-identical dispatch:
+`grounding: { mode: "native" }` (native provider web grounding, gemini-only in
+v1), `serviceTier: "standard" | "flex" | "priority"`, and
+`reasoning: { level: "minimal" | "low" | "medium" | "high" }` (gemini-only in
+v1, mapped to the thinking level). A declared directive a provider cannot honor
+FAILS CLOSED — a thrown `RigFailureError` with class `invariant_violation`,
+pre-dispatch (no meter, no provider call) — never a silent drop. A grounded step
+that also passes a `cache.key` keeps its cached-token ratio: bake the
+`googleSearch` tool into the cache resource at creation (ModelRig sends
+`cachedContent` without tools, which the provider requires).
+
 Every attempt is recorded in a local SQLite file (`.modelrig/telemetry.db`)
 tagged with your `tags`. Failures are typed —
 `RigFailureError.failure.class` is one of `content_invalid`, `capacity_shed`,

package/data/PROVENANCE.md

@@ -0,0 +1,19 @@
+# data/pricing-map.json — provenance
+
+Copied manually from `vendor/litellm-upstream/model_prices_and_context_window.json`
+at the pinned SHA recorded in `upstream/litellm.pin.json` (`de706a35a6`, pinned
+2026-07-30). 2,986 model entries at time of copy.
+
+Source: LiteLLM (https://github.com/BerriAI/litellm),
+Copyright (c) 2023 Berri AI, MIT License.
+
+NOTE (Phase 0 gap): the automated extraction script (`scripts/sync-upstream.ts`
+with provenance header + diff output) was not built before Phase 1 started, so
+this snapshot was copied by hand per implementation-plan §9. Replace this manual
+copy with the sync script's output when Phase 0 tooling lands; until then, any
+pin bump must re-copy this file and update this note.
+
+Key naming (verified against the snapshot):
+- Gemini API models: `gemini/<model>` (bare ids resolve to vertex_ai pricing — do not use)
+- OpenAI models: bare `<model>`
+- DeepSeek models: `deepseek/<model>` (bare ids also present)

package/data/pricing-manual.json

@@ -0,0 +1,22 @@
+{
+  "_provenance": {
+    "role": "Website-sourced per-token prices for hosts that do not expose pricing via an API our key can read (today: Fireworks — its /models listing rejects an inference-scoped key). Hand-read from the vendor's public model page, each entry carrying its source URL + read date. Merged into pricing-overlay.json by scripts/pricing-sweep.ts alongside the API-swept prices; both are gap-fillers beneath the pinned LiteLLM snapshot (snapshot wins on shared keys). Replace an entry here with an API/scraper adapter when one exists.",
+    "note": "Flat prices (input_cost_per_token / output / cache_read) are the BASE serverless standard rate. Serving-tier multipliers (Fireworks: +25% priority, +50% fast, +10% US-only) are now carried in the additive `pricing_v2` block (pricing-first-class spec) as MULTIPLIERS on base (1.25 / 1.5 / 1.1). Context-length/batch/modality bands are still uncaptured for this model and are absent (not guessed). The flat fields stay for backward compatibility."
+  },
+  "prices": {
+    "fireworks_ai/accounts/fireworks/models/kimi-k3": {
+      "input_cost_per_token": 3.0e-6,
+      "output_cost_per_token": 1.5e-5,
+      "cache_read_input_token_cost": 3.0e-7,
+      "source": "fireworks-website:https://fireworks.ai/models/fireworks/kimi-k3 (base serverless standard; +25% priority / +50% fast / +10% US-only)",
+      "as_of": "2026-08-09",
+      "pricing_v2": {
+        "base": { "input": 3.0e-6, "output": 1.5e-5, "cache_read": 3.0e-7 },
+        "tiers": { "priority": 1.25, "fast": 1.5, "region_us": 1.1 },
+        "source": "fireworks-website:https://fireworks.ai/models/fireworks/kimi-k3",
+        "as_of": "2026-08-09",
+        "basis": "base serverless · standard tier · text"
+      }
+    }
+  }
+}