modelrig 0.7.0 → 0.8.0

package/CHANGELOG.md

@@ -10,6 +10,113 @@ published package diverge.
 This project adheres to [Semantic Versioning](https://semver.org/). Dates are
 UTC.
 
+## [0.8.0] — 2026-08-24
+
+Packaging (GS-3) — the tarball now ships the customer-facing runs docs
+(`docs/quickstart.md`, `docs/instrumentation.md`, `docs/how-it-fits.md`,
+`docs/migration-playbook.md`, `docs/agent-skill/SKILL.md`, and the generated
+`docs/llms.txt`) plus a self-contained **Runs — the standard path** section in
+`README.md`, so `npm i modelrig` + read-the-package teaches run → step → artifact
+offline; and every doc now states the `MODELRIG_ARTIFACTS` default consistently
+(on by default once a control plane is configured; a local-only rig stays off;
+`=0` opts out, `=1` forces on). Docs/packaging only — no SDK behaviour change.
+
+Get-started v5 — runs are the standard path (GS-1). The front-door instructions
+(www `/get-started`, the console setup prompt, `quickstart.md`, `how-it-fits.md`,
+the agent `SKILL.md`) now teach **run → step → artifact** as the standard way to
+instrument a pipeline, not an appendix; `instrumentation.md` stays the reference
+they link to. SDK: inside a run context the SDK now also stamps **`tags.step`**
+(the current step key: your explicit `stepKey`, else `tags.step`, else the auto
+`route#n` / `provider/model#n` key) on every attempt row of both lanes when the
+caller set none — the mirror of RC-1's `run_id` stamp; a caller value always wins,
+and outside a run context rows are byte-identical. `modelrig status` prints
+**`runs recorded: N (last: <ts>)`** from a best-effort readonly read of the local
+buffer, and `modelrig validate` **warns when a rig has recorded calls but never
+started a run** (a teaching hint — never a failure). ZERO DDL, no new deps.
+
+Provenance-first assembly (screening B3) — additive, opt-in SDK seams for keeping
+customer- and web-derived text out of the trusted system slot. **Default
+behaviour is byte-identical**: a route with no `grounding.assembly` and no
+`x-trust` takes the legacy path unchanged. A route may set
+`grounding.assembly: spotlight` to route search results through the assembler as
+an untrusted user block (search results are delivered as data, not instructions);
+a template variable may be annotated `x-trust: untrusted` to move it out of the
+system slot into that spotlighted block. Model-output citations are allowlisted to
+the injected sources, with a `RunMeta.grounding.citations_dropped` count. `modelrig
+validate` warns on legacy grounding and on an untrusted-shaped variable left
+unannotated in the system slot. `EquivalenceReport.subject` gains
+`"screen-assembly"`. `screen/provenance.ts` + `screen/types.ts` now ship in
+`dist/`. ZERO DDL, no new deps.
+
+Near-optimal pairwise + disagreement (N5/N6) — bake-offs gain a second read on
+regret and a fixture-mining output, both additive and **byte-identical for
+routes that don't opt in**. N5 scores each served arm's **pairwise standing vs
+the reference arm** on the paired replay set — a `pairwise_win_rate`
+(win=1/tie=0.5/loss=0) with a seeded-bootstrap CI and n — surfaced as
+`VariantMetrics.pairwise` and a `pairwise-win` column in the report. It is
+DETERMINISTIC-FIRST, exactly like the RB-1 regret tiers: when both an arm and the
+reference carry a per-sample value-accuracy score the verdict is free (the sign
+of their difference); only the samples nothing scored fall to an OPT-IN, CAPPED,
+METERED pairwise judge, declared `bakeoff.pairwise.judge` (judge model ∈ declared
+candidates, #223 F3; every call a metered inference tagged `bakeoff_pairwise`,
+capped per arm). A new `objective.pairwiseAware` (default off) adds one
+eligibility clause to the `quality-first`/`balanced` gate — a proposed arm must
+not be pairwise-worse than the reference within CI — and never touches the
+cost-first gate. N6 mines the same replay for **eval-case candidates**: ranked
+POINTERS (by `captureId`) to the samples where the bake-off's own signals
+disagree (conformance vs value score, the Q1 gate vs conformance, served arm vs
+reference, a pairwise upset), surfaced on `BakeoffArtifact.disagreement` and a
+report summary line. Candidates are **metadata only** (captureId + numbers /
+booleans, asserted meta-safe — never input or output text) and are **never
+auto-promoted**; the full pre-cap `found` count rides the truncated list so a
+capped surface is never mistaken for all of them. No new root exports — additive
+route YAML plus report/artifact fields. ZERO DDL, no new deps. See
+`docs/bakeoffs.md`.
+
+Published receipts (RP-1 / RP-1c) — a signed, login-free **Receipt@v1** an org
+can publish from its own measured rows. **This is two halves; only the shape is
+in the npm package.** The `modelrig` package exposes the receipt CONTRACT via the
+new `modelrig/receipts` sub-path export ONLY (never re-exported from `index.ts`):
+the `ReceiptV1` shape, `canonicalizeReceipt` / `hashReceipt` (JCS-style
+canonical body, self-describing `sha256:` hash), and the pure `isReceiptSafe`
+classification gate — `measured` carries numbers / booleans / hashes /
+identifiers only, no content, no tenant ids (fail-closed). The additive
+`exports` map (`.`, `./receipts`, `./data/*`, `./registry/*`) plus a
+`receipts.d.ts` shim keep the historical bare surface and the runtime data /
+registry files reachable now that a sub-path map turns resolution into an
+allowlist; `dist/receipts/**` now ships. **Everything else lives in
+`modelrig-server` (NOT the npm tag):** Ed25519 signing with a `key_id` (node
+crypto, no new dep), authed publish / revoke, the store-only public family —
+`GET /v1/r/:id`, `…/badge.svg`, `GET /v1/verify/:id` (re-canonicalize → re-hash →
+verify signature), `GET /.well-known/modelrig-receipts.json` (key rotation) — a
+handler constructed with NO database client, an org alias that never leaks
+`org_id`, `recipient` visibility (a login-free capability URL never listed
+publicly), and a revoke tombstone that answers 410. RP-1c adds the `run` and
+`fact-sheet` producers + source readers and the `source_ref` lookup (X9 — the
+PR-body "verify this" link). **Honest limit:** publish / revoke / signing degrade
+to **501** until `MODELRIG_RECEIPT_SIGNING_KEY` + `MODELRIG_RECEIPT_KEY_ID` and
+the content store are provisioned (a half-configured signer is refused, not
+half-enabled); the public read / verify surface answers 404 until an object is
+published. ZERO DDL, no new deps. See `docs/receipts.md`.
+
+Tenant ledger (TB-1) — lazy per-customer identity and derived statements for a
+platform that resells routed inference. **Server / console / docs only — the
+`modelrig` npm surface is unchanged by TB-1.** TB-1a adds tenant identity as a
+single column: migration `00059` puts a `kind` discriminator
+(`'project'` | `'tenant'`) on `projects`, with a reserved per-org `__tenants__`
+parent and tenant leaves slugged `t_<sha256(org_id:subject)[:24]>`, materialized
+service-role at ingest from the principal's org (never the payload) — **zero new
+tables**. TB-1b derives a per-tenant statement (`TenantStatementV1`) on read,
+**never stored**: counts / cents / hashes only, where `verified` is DETERMINISTIC
+conformance only (run-outcome@v1 / schema-conformant artifact; model judge and
+human feedback are `quality_context` and never gate billable), unpriced models
+fall to a separate "priced at list" line rather than into the total, and a new
+Tenants console renders it. Rate cards and real price / margin are **TB-3, NOT in
+this tag** — every TB-1 statement is "priced at list" with `billable = verified`
+and null price / margin. The migration ships in the repo but is applied by a
+Supabase-enabled session, not this build. ZERO DDL added by this release; no new
+deps. See `docs/tenants.md`.
+
 ## [0.7.0] — 2026-08-23
 
 Released with the docs/registry mirror in one R-A ceremony. Headline: wrap a

package/README.md

@@ -229,6 +229,69 @@ lines up with the console's built-ins:
 `customer`, `tenant`, `user_id`, `team`, … is nudged toward the reserved key it
 resembles. They stay ordinary tags; nothing is required.
 
+## Runs — the standard path
+
+Every pipeline execution is a **run** — the standard way ModelRig records what
+your workstream did, not an appendix. Wrap the execution in a run context, name
+one **step** per model-call family, and **save the work products** as artifacts;
+the console's **Runs** tab then shows the run → step → artifact chain you click
+through, and `/projects` groups it by your tags.
+
+```ts
+import { createRig, loadConfigFromEnv } from "modelrig";
+
+const rig = createRig(loadConfigFromEnv());
+
+// One run per pipeline execution. run.scope isolates concurrent runs; use
+// run.start()/run.end() for one run at a time on a request path.
+await rig.artifacts.run.scope(
+  { pipeline: "support", episodeKey: ticketId },
+  async () => {
+    const result = await rig.run("example.support_summarize", {
+      input: { ticket, product, priorContext },
+      tags: { subject: customer, feature: "summarize" }, // run_id auto-stamped
+      stepKey: "summarize",                              // names this step on /runs
+    });
+    // Save the work product — prompt / raw / parsed all save the same way.
+    rig.artifacts.artifact.save(result.output, { name: "summary", type: "step_output" });
+  },
+);
+rig.close();
+```
+
+- **Tags:** carry `subject` (what the run is *about* — an opaque id) and
+  `feature`. Inside a run context the SDK stamps `tags.run_id` (the run's
+  `episodeKey`, else its id) and `tags.step` (the current step key) on every
+  attempt row of both lanes — a value you pass always wins, and outside a run
+  context nothing is stamped, so the row is byte-identical.
+- **Raw lane too:** a `rig.runRaw` call inside a run context records a
+  ground-truth step (since 0.5.0), so grounded/cached BYOK pipelines fill the
+  same graph.
+- **Acceptance:** the run appears on `/runs` with its steps in order and the
+  artifacts you saved attached. `modelrig status` prints `runs recorded: N` as
+  the local mirror; `modelrig validate` warns if a rig records calls but never
+  starts a run.
+
+**On by default under a control plane.** The run/artifact namespace is **on by
+default once a control plane is configured** — set `MODELRIG_API_KEY=rig_sk_…`
+(or the self-host Supabase pair) and runs record automatically, metadata and
+hashes only. A **pure local-only rig** (no sink to ship to) stays off; force it
+on there with `MODELRIG_ARTIFACTS=1`, or opt out anywhere with
+`MODELRIG_ARTIFACTS=0`. With the namespace off it is inert — `run.start` is a
+no-op, `artifact.save` returns null, zero rows are written, and your pipeline
+behaves byte-for-byte as before. `modelrig status` prints the current posture.
+
+> **Metadata and hashes only, this release.** Content custody (holding the
+> serialized bytes) is early access; today the value you save is hashed (sha256)
+> and the bytes are discarded — the metadata row and its integrity hash are what
+> persist. A `zeroRetention` run refuses every artifact fail-closed at the SDK
+> gate.
+
+The full artifact API — lineage (`link`), evaluations (`evaluate`), the reusable
+per-step seam, and `run.scope` vs `run.start` — is the
+[instrumentation guide](./docs/instrumentation.md), shipped in this package so
+you can read it offline.
+
 ## CLI
 
 The package ships a `modelrig` binary:

package/dist/bakeoff/disagreement.d.ts

@@ -0,0 +1,93 @@
+/**
+ * N6 — disagreement mining into eval-case candidates (near-optimal brief §3.2 N6).
+ *
+ * The discriminating-fixtures gap (brief §2): a route's fixtures separate *valid*
+ * from *invalid* but rarely *good* from *great*. The richest place to look for the
+ * fixtures that WOULD discriminate is where the bake-off's own signals DISAGREE on
+ * a replayed sample:
+ *   - deterministic conformance says one thing, the value score another
+ *     (schema-valid-but-wrong, or invalid-but-scored-right);
+ *   - the Q1 quality gate's verdict contradicts conformance;
+ *   - the SERVED arm and the REFERENCE arm diverge on the same capture (the
+ *     top-of-distribution signal the brief names);
+ *   - the pairwise judge upsets the expected order (arm beats / loses to the ref).
+ *
+ * Each such sample becomes an eval-case CANDIDATE — a ranked POINTER (captureId)
+ * an operator may promote through the existing E4 path. NEVER auto-promoted.
+ *
+ * CONTENT CUSTODY (P6 / spec): a candidate carries ONLY already-captured
+ * metadata — the captureId (an identifier), booleans, and numeric scores. No
+ * output text, no input text. Every string field is asserted meta-safe
+ * (`isMetaSafeString`) before it is surfaced, the same predicate the WI-1
+ * envelope uses. This module is PURE and fixture-tested.
+ */
+import type { ReplaySampleResult } from "./replay";
+import type { PairwiseOutcome } from "./pairwise";
+export type DisagreementReason = "conform_vs_value" | "conform_vs_gate" | "arm_vs_reference" | "pairwise_upset";
+/** A per-sample verdict divergence, surfaced as a ranked eval-case candidate. All
+ * fields are already-captured metadata — a POINTER to promote, never content. */
+export interface DisagreementCandidate {
+    /** The captured input's inference id — the promote path rehydrates from it. */
+    readonly captureId: string;
+    /** Which served arm this divergence is on (never the reference arm). */
+    readonly arm: string;
+    /** 0..1 magnitude of the disagreement — the ranking key (desc). */
+    readonly score: number;
+    /** Every signal pair that fired (ranked candidates usually have one; the
+     * top ones several). */
+    readonly reasons: readonly DisagreementReason[];
+    /** The already-captured numbers/booleans behind the score (no content). */
+    readonly signals: {
+        readonly armConform: boolean;
+        readonly armValueAccuracy: number | null;
+        readonly armQualityPassed: boolean | null;
+        readonly refConform?: boolean;
+        readonly refValueAccuracy?: number | null;
+        readonly pairwise?: PairwiseOutcome;
+    };
+}
+export interface DisagreementReport {
+    /** Paired/served samples examined across all arms. */
+    readonly examined: number;
+    /** Ranked candidates (score desc, then captureId asc), truncated to the cap. */
+    readonly candidates: readonly DisagreementCandidate[];
+    /** How many candidates existed BEFORE the cap — surfaced so a truncated list is
+     * never mistaken for "all of them" (no silent caps). */
+    readonly found: number;
+    /** The cap applied when `found > candidates.length`, else null. */
+    readonly cappedAt: number | null;
+    /** Count of candidates each reason contributed to (a candidate with two
+     * reasons counts once under each). */
+    readonly counts: Readonly<Record<DisagreementReason, number>>;
+}
+/** Default cap on the surfaced candidate list — enough to act on, bounded so a
+ * noisy replay set does not flood the surface. The full `found` count rides
+ * alongside so the truncation is legible. */
+export declare const DEFAULT_DISAGREEMENT_CAP = 50;
+export interface MineInput {
+    /** One entry per SERVED arm (never the reference): its per-sample results. */
+    readonly arms: readonly {
+        readonly arm: string;
+        readonly samples: readonly ReplaySampleResult[];
+    }[];
+    /** The reference arm's per-sample results, keyed for the arm-vs-reference and
+     * pairwise signals. Absent ⇒ those two reasons never fire (only the
+     * within-arm signals do). */
+    readonly reference?: readonly ReplaySampleResult[];
+    /** Pairwise verdicts keyed `${arm}\0${captureId}` → outcome, when a pairwise
+     * signal was computed. */
+    readonly pairwise?: ReadonlyMap<string, PairwiseOutcome>;
+    /** Surfaced-list cap (default {@link DEFAULT_DISAGREEMENT_CAP}). */
+    readonly maxCandidates?: number;
+}
+/** The pairwise map key — arm + captureId, NUL-joined so neither can spoof the
+ * other's boundary. */
+export declare function pairwiseKey(arm: string, captureId: string): string;
+/**
+ * Mine a replayed bake-off's per-sample results for verdict disagreements and
+ * rank them as eval-case candidates (N6). Pure over its inputs; deterministic
+ * ordering (score desc, then captureId asc). A produced sample with NO
+ * divergence past {@link SIGNAL_EPS} yields no candidate.
+ */
+export declare function mineDisagreements(input: MineInput): DisagreementReport;
+//# sourceMappingURL=disagreement.d.ts.map

package/dist/bakeoff/disagreement.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"disagreement.d.ts","sourceRoot":"","sources":["../../src/bakeoff/disagreement.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;GAsBG;AAGH,OAAO,KAAK,EAAE,kBAAkB,EAAE,MAAM,UAAU,CAAC;AACnD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAElD,MAAM,MAAM,kBAAkB,GAC1B,kBAAkB,GAClB,iBAAiB,GACjB,kBAAkB,GAClB,gBAAgB,CAAC;AAErB;iFACiF;AACjF,MAAM,WAAW,qBAAqB;IACpC,+EAA+E;IAC/E,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,wEAAwE;IACxE,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAC;IACrB,mEAAmE;IACnE,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB;4BACwB;IACxB,QAAQ,CAAC,OAAO,EAAE,SAAS,kBAAkB,EAAE,CAAC;IAChD,2EAA2E;IAC3E,QAAQ,CAAC,OAAO,EAAE;QAChB,QAAQ,CAAC,UAAU,EAAE,OAAO,CAAC;QAC7B,QAAQ,CAAC,gBAAgB,EAAE,MAAM,GAAG,IAAI,CAAC;QACzC,QAAQ,CAAC,gBAAgB,EAAE,OAAO,GAAG,IAAI,CAAC;QAC1C,QAAQ,CAAC,UAAU,CAAC,EAAE,OAAO,CAAC;QAC9B,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;QAC1C,QAAQ,CAAC,QAAQ,CAAC,EAAE,eAAe,CAAC;KACrC,CAAC;CACH;AAED,MAAM,WAAW,kBAAkB;IACjC,sDAAsD;IACtD,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,gFAAgF;IAChF,QAAQ,CAAC,UAAU,EAAE,SAAS,qBAAqB,EAAE,CAAC;IACtD;4DACwD;IACxD,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,mEAAmE;IACnE,QAAQ,CAAC,QAAQ,EAAE,MAAM,GAAG,IAAI,CAAC;IACjC;0CACsC;IACtC,QAAQ,CAAC,MAAM,EAAE,QAAQ,CAAC,MAAM,CAAC,kBAAkB,EAAE,MAAM,CAAC,CAAC,CAAC;CAC/D;AAED;;6CAE6C;AAC7C,eAAO,MAAM,wBAAwB,KAAK,CAAC;AAY3C,MAAM,WAAW,SAAS;IACxB,8EAA8E;IAC9E,QAAQ,CAAC,IAAI,EAAE,SAAS;QAAE,QAAQ,CAAC,GAAG,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,OAAO,EAAE,SAAS,kBAAkB,EAAE,CAAA;KAAE,EAAE,CAAC;IACpG;;iCAE6B;IAC7B,QAAQ,CAAC,SAAS,CAAC,EAAE,SAAS,kBAAkB,EAAE,CAAC;IACnD;8BAC0B;IAC1B,QAAQ,CAAC,QAAQ,CAAC,EAAE,WAAW,CAAC,MAAM,EAAE,eAAe,CAAC,CAAC;IACzD,oEAAoE;IACpE,QAAQ,CAAC,aAAa,CAAC,EAAE,MAAM,CAAC;CACjC;AAED;uBACuB;AACvB,wBAAgB,WAAW,CAAC,GAAG,EAAE,MAAM,EAAE,SAAS,EAAE,MAAM,GAAG,MAAM,CAElE;AAED;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,SAAS,GAAG,kBAAkB,CA0FtE"}

package/dist/bakeoff/disagreement.js

@@ -0,0 +1,140 @@
+"use strict";
+/**
+ * N6 — disagreement mining into eval-case candidates (near-optimal brief §3.2 N6).
+ *
+ * The discriminating-fixtures gap (brief §2): a route's fixtures separate *valid*
+ * from *invalid* but rarely *good* from *great*. The richest place to look for the
+ * fixtures that WOULD discriminate is where the bake-off's own signals DISAGREE on
+ * a replayed sample:
+ *   - deterministic conformance says one thing, the value score another
+ *     (schema-valid-but-wrong, or invalid-but-scored-right);
+ *   - the Q1 quality gate's verdict contradicts conformance;
+ *   - the SERVED arm and the REFERENCE arm diverge on the same capture (the
+ *     top-of-distribution signal the brief names);
+ *   - the pairwise judge upsets the expected order (arm beats / loses to the ref).
+ *
+ * Each such sample becomes an eval-case CANDIDATE — a ranked POINTER (captureId)
+ * an operator may promote through the existing E4 path. NEVER auto-promoted.
+ *
+ * CONTENT CUSTODY (P6 / spec): a candidate carries ONLY already-captured
+ * metadata — the captureId (an identifier), booleans, and numeric scores. No
+ * output text, no input text. Every string field is asserted meta-safe
+ * (`isMetaSafeString`) before it is surfaced, the same predicate the WI-1
+ * envelope uses. This module is PURE and fixture-tested.
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.DEFAULT_DISAGREEMENT_CAP = void 0;
+exports.pairwiseKey = pairwiseKey;
+exports.mineDisagreements = mineDisagreements;
+const attempt_meta_1 = require("../telemetry/attempt-meta");
+/** Default cap on the surfaced candidate list — enough to act on, bounded so a
+ * noisy replay set does not flood the surface. The full `found` count rides
+ * alongside so the truncation is legible. */
+exports.DEFAULT_DISAGREEMENT_CAP = 50;
+/** Per-sample quality on the SAME 0..1 scale regret uses: the value-accuracy
+ * score when present, else 1/0 conformance. */
+function quality01(s) {
+    return s.valueAccuracy ?? (s.conform ? 1 : 0);
+}
+/** A signal is "meaningfully divergent" past this magnitude (below it is noise —
+ * e.g. a 0.98-vs-1.0 field-accuracy hair-split is not a fixture worth promoting). */
+const SIGNAL_EPS = 0.05;
+/** The pairwise map key — arm + captureId, NUL-joined so neither can spoof the
+ * other's boundary. */
+function pairwiseKey(arm, captureId) {
+    return `${arm}\0${captureId}`;
+}
+/**
+ * Mine a replayed bake-off's per-sample results for verdict disagreements and
+ * rank them as eval-case candidates (N6). Pure over its inputs; deterministic
+ * ordering (score desc, then captureId asc). A produced sample with NO
+ * divergence past {@link SIGNAL_EPS} yields no candidate.
+ */
+function mineDisagreements(input) {
+    const cap = input.maxCandidates ?? exports.DEFAULT_DISAGREEMENT_CAP;
+    const refByCapture = new Map((input.reference ?? []).map((s) => [s.captureId, s]));
+    const counts = {
+        conform_vs_value: 0,
+        conform_vs_gate: 0,
+        arm_vs_reference: 0,
+        pairwise_upset: 0,
+    };
+    const candidates = [];
+    let examined = 0;
+    for (const { arm, samples } of input.arms) {
+        for (const s of samples) {
+            // Only PRODUCED samples carry verdicts to disagree about; a failed sample
+            // (no output) has nothing to mine. Guard the captureId is meta-safe before
+            // it is surfaced (captureIds are identifiers, always safe — asserted, never
+            // assumed, per the content-custody rule).
+            if (s.failureClass !== null)
+                continue;
+            if (!(0, attempt_meta_1.isMetaSafeString)(s.captureId))
+                continue;
+            examined += 1;
+            const reasons = [];
+            let score = 0;
+            const conform01 = s.conform ? 1 : 0;
+            // conform_vs_value: schema conformance disagrees with the value score.
+            if (s.valueAccuracy !== null) {
+                const gap = Math.abs(conform01 - s.valueAccuracy);
+                if (gap > SIGNAL_EPS) {
+                    reasons.push("conform_vs_value");
+                    score = Math.max(score, gap);
+                }
+            }
+            // conform_vs_gate: the Q1 boolean verdict contradicts conformance.
+            if (s.qualityPassed !== null && s.qualityPassed !== s.conform) {
+                reasons.push("conform_vs_gate");
+                score = Math.max(score, 1);
+            }
+            const ref = refByCapture.get(s.captureId);
+            const signals = {
+                armConform: s.conform,
+                armValueAccuracy: s.valueAccuracy,
+                armQualityPassed: s.qualityPassed,
+                ...(ref !== undefined ? { refConform: ref.conform, refValueAccuracy: ref.valueAccuracy } : {}),
+            };
+            // arm_vs_reference: the served arm and the reference diverge on this capture
+            // (the top-of-distribution signal — where the fixture corpus learns to tell
+            // good from great).
+            if (ref !== undefined) {
+                const gap = Math.abs(quality01(s) - quality01(ref));
+                if (gap > SIGNAL_EPS) {
+                    reasons.push("arm_vs_reference");
+                    score = Math.max(score, gap);
+                }
+            }
+            // pairwise_upset: a non-tie pairwise verdict is itself a divergence worth a
+            // fixture (an arm beating OR losing to the reference by preference).
+            const pw = input.pairwise?.get(pairwiseKey(arm, s.captureId));
+            if (pw !== undefined && pw !== "tie") {
+                reasons.push("pairwise_upset");
+                score = Math.max(score, 1);
+            }
+            if (reasons.length === 0)
+                continue;
+            for (const r of reasons)
+                counts[r] += 1;
+            candidates.push({
+                captureId: s.captureId,
+                arm,
+                score,
+                reasons,
+                signals: pw !== undefined ? { ...signals, pairwise: pw } : signals,
+            });
+        }
+    }
+    // Deterministic ranking: strongest disagreement first, captureId as a stable
+    // tiebreak so the same replay set always ranks identically.
+    candidates.sort((a, b) => b.score - a.score || a.captureId.localeCompare(b.captureId));
+    const found = candidates.length;
+    const surfaced = candidates.slice(0, cap);
+    return {
+        examined,
+        candidates: surfaced,
+        found,
+        cappedAt: found > surfaced.length ? cap : null,
+        counts,
+    };
+}

package/dist/bakeoff/engine.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"engine.d.ts","sourceRoot":"","sources":["../../src/bakeoff/engine.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAGH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,sBAAsB,CAAC;AACzD,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAG1C,OAAO,KAAK,EAAE,aAAa,EAAE,aAAa,EAAE,WAAW,EAAgB,MAAM,gBAAgB,CAAC;AAM9F,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAEpD,OAAO,KAAK,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC7D,OAAO,KAAK,EAAE,eAAe,EAAE,gBAAgB,EAAmC,cAAc,EAAE,MAAM,SAAS,CAAC;AAElH;;+BAE+B;AAC/B,eAAO,MAAM,aAAa,kBAAkB,CAAC;AAE7C;;;gFAGgF;AAChF,MAAM,MAAM,kBAAkB,GAAG,CAAC,KAAK,EAAE;IACvC,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,IAAI,EAAE,SAAS,UAAU,EAAE,CAAC;CACtC,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;AAgBpB,MAAM,WAAW,cAAc;IAC7B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,qEAAqE;IACrE,QAAQ,CAAC,QAAQ,EAAE,SAAS,MAAM,EAAE,CAAC;IACrC,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,2EAA2E;IAC3E,QAAQ,CAAC,qBAAqB,EAAE,MAAM,CAAC;IACvC;;wEAEoE;IACpE,QAAQ,CAAC,WAAW,CAAC,EAAE,eAAe,CAAC;IACvC;iEAC6D;IAC7D,QAAQ,CAAC,aAAa,CAAC,EAAE,aAAa,CAAC;IACvC;;wEAEoE;IACpE,QAAQ,CAAC,oBAAoB,CAAC,EAAE,CAAC,KAAK,EAAE,WAAW,KAAK,aAAa,GAAG,IAAI,CAAC;IAC7E;+BAC2B;IAC3B,QAAQ,CAAC,QAAQ,CAAC,EAAE,WAAW,CAAC,MAAM,EAAE;QAAE,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAA;KAAE,CAAC,CAAC;IACrE;;;;;+EAK2E;IAC3E,QAAQ,CAAC,WAAW,CAAC,EAAE,WAAW,CAAC;CACpC;AAsBD,wBAAsB,UAAU,CAC9B,IAAI,EAAE,OAAO,EACb,QAAQ,EAAE,YAAY,EACtB,GAAG,EAAE,cAAc,EACnB,WAAW,CAAC,EAAE,kBAAkB,GAC/B,OAAO,CAAC,eAAe,CAAC,CAiJ1B;AAsCD;iEACiE;AACjE,MAAM,WAAW,eAAe;IAC9B,QAAQ,CAAC,SAAS,CAAC,EAAE,gBAAgB,CAAC;IACtC,mEAAmE;IACnE,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;CACpC;AAoBD;;;;GAIG;AACH,wBAAgB,eAAe,CAC7B,SAAS,EAAE,SAAS,MAAM,EAAE,EAC5B,OAAO,EAAE,SAAS,cAAc,EAAE,EAClC,GAAG,CAAC,EAAE,eAAe,GACpB,eAAe,CAAC,UAAU,CAAC,CA4G7B;AAED,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,QAAQ,CAAC,cAAc,EAAE,MAAM,CAAC;IAChC,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,OAAO,CAAC,EAAE,OAAO,KAAK,CAAC;CACjC;AAED;;uDAEuD;AACvD,wBAAsB,aAAa,CACjC,QAAQ,EAAE,eAAe,EACzB,MAAM,EAAE,mBAAmB,GAC1B,OAAO,CAAC,IAAI,CAAC,CA6Bf;AAED;;;;;GAKG;AACH,wBAAgB,eAAe,CAAC,QAAQ,EAAE,eAAe,EAAE,QAAQ,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CA2BpG"}
+{"version":3,"file":"engine.d.ts","sourceRoot":"","sources":["../../src/bakeoff/engine.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAGH,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,sBAAsB,CAAC;AACzD,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,YAAY,CAAC;AAG1C,OAAO,KAAK,EAAE,aAAa,EAAE,aAAa,EAAE,WAAW,EAAgB,MAAM,gBAAgB,CAAC;AAS9F,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,kBAAkB,CAAC;AAEpD,OAAO,KAAK,EAAE,UAAU,EAAE,eAAe,EAAE,MAAM,WAAW,CAAC;AAC7D,OAAO,KAAK,EAAE,eAAe,EAAE,gBAAgB,EAAmC,cAAc,EAAE,MAAM,SAAS,CAAC;AAElH;;+BAE+B;AAC/B,eAAO,MAAM,aAAa,kBAAkB,CAAC;AAE7C;;;gFAGgF;AAChF,MAAM,MAAM,kBAAkB,GAAG,CAAC,KAAK,EAAE;IACvC,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,IAAI,EAAE,SAAS,UAAU,EAAE,CAAC;CACtC,KAAK,OAAO,CAAC,IAAI,CAAC,CAAC;AAgBpB,MAAM,WAAW,cAAc;IAC7B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,qEAAqE;IACrE,QAAQ,CAAC,QAAQ,EAAE,SAAS,MAAM,EAAE,CAAC;IACrC,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,2EAA2E;IAC3E,QAAQ,CAAC,qBAAqB,EAAE,MAAM,CAAC;IACvC;;wEAEoE;IACpE,QAAQ,CAAC,WAAW,CAAC,EAAE,eAAe,CAAC;IACvC;iEAC6D;IAC7D,QAAQ,CAAC,aAAa,CAAC,EAAE,aAAa,CAAC;IACvC;;wEAEoE;IACpE,QAAQ,CAAC,oBAAoB,CAAC,EAAE,CAAC,KAAK,EAAE,WAAW,KAAK,aAAa,GAAG,IAAI,CAAC;IAC7E;+BAC2B;IAC3B,QAAQ,CAAC,QAAQ,CAAC,EAAE,WAAW,CAAC,MAAM,EAAE;QAAE,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAA;KAAE,CAAC,CAAC;IACrE;;;;;+EAK2E;IAC3E,QAAQ,CAAC,WAAW,CAAC,EAAE,WAAW,CAAC;CACpC;AAsBD,wBAAsB,UAAU,CAC9B,IAAI,EAAE,OAAO,EACb,QAAQ,EAAE,YAAY,EACtB,GAAG,EAAE,cAAc,EACnB,WAAW,CAAC,EAAE,kBAAkB,GAC/B,OAAO,CAAC,eAAe,CAAC,CA8K1B;AAuCD;iEACiE;AACjE,MAAM,WAAW,eAAe;IAC9B,QAAQ,CAAC,SAAS,CAAC,EAAE,gBAAgB,CAAC;IACtC,mEAAmE;IACnE,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;CACpC;AA8BD;;;;GAIG;AACH,wBAAgB,eAAe,CAC7B,SAAS,EAAE,SAAS,MAAM,EAAE,EAC5B,OAAO,EAAE,SAAS,cAAc,EAAE,EAClC,GAAG,CAAC,EAAE,eAAe,GACpB,eAAe,CAAC,UAAU,CAAC,CA8G7B;AAED,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,QAAQ,CAAC,cAAc,EAAE,MAAM,CAAC;IAChC,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;IAC1B,QAAQ,CAAC,OAAO,CAAC,EAAE,OAAO,KAAK,CAAC;CACjC;AAED;;uDAEuD;AACvD,wBAAsB,aAAa,CACjC,QAAQ,EAAE,eAAe,EACzB,MAAM,EAAE,mBAAmB,GAC1B,OAAO,CAAC,IAAI,CAAC,CA6Bf;AAED;;;;;GAKG;AACH,wBAAgB,eAAe,CAAC,QAAQ,EAAE,eAAe,EAAE,QAAQ,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAuCpG"}

package/dist/bakeoff/engine.js

@@ -22,6 +22,8 @@ const loader_1 = require("../routes/loader");
 const replay_1 = require("./replay");
 const stats_1 = require("./stats");
 const judge_1 = require("./judge");
+const pairwise_1 = require("./pairwise");
+const disagreement_1 = require("./disagreement");
 const reference_1 = require("./reference");
 const samples_1 = require("./samples");
 /** The arm name carrying the reference (ceiling) metrics. Reserved — a user
@@ -75,6 +77,10 @@ async function runBakeoff(deps, captures, req, samplesSink) {
     const keepOutputs = req.keepOutputs !== undefined && samplesSink !== undefined;
     const bakeoffId = (0, node_crypto_1.randomUUID)();
     const judgeCfg = route.bakeoff?.judge;
+    // N5: a declared pairwise judge needs each sample's output retained (arm-vs-
+    // reference comparison in the engine) — WITHOUT the keepOutputs sink persist.
+    const pairwiseJudgeCfg = route.bakeoff?.pairwise?.judge;
+    const retainForPairwise = pairwiseJudgeCfg !== undefined;
     // Replay every requested arm. Raw samples are kept ALWAYS (paired regret is
     // computed from per-sample quality); the keepOutputs sink is separate. A fresh
     // judge scorer per arm keeps `max_samples` per-arm (spec §4.3).
@@ -88,6 +94,7 @@ async function runBakeoff(deps, captures, req, samplesSink) {
             lastN: req.replayLastN,
             envelopeUsd: req.envelopeUsdPerVariant,
             ...(keepOutputs ? { keepOutputs: true } : {}),
+            ...(retainForPairwise ? { retainOutputForPairwise: true } : {}),
             ...(req.expected !== undefined ? { expected: req.expected } : {}),
             ...(judgeCfg !== undefined ? { judge: (0, judge_1.makeJudgeScorer)(deps, route, judgeCfg) } : {}),
             ...(req.qualityGate !== undefined ? { qualityGate: req.qualityGate } : {}),
@@ -101,6 +108,10 @@ async function runBakeoff(deps, captures, req, samplesSink) {
     // arm's paired regret vs it. Partial-result honesty is preserved.
     const referenceInfo = resolveReference(route, req);
     let reference = null;
+    // N6: pairwise verdicts keyed `${arm} ${captureId}` (a mining input) + the
+    // reference samples, hoisted so disagreement mining sees them after the block.
+    const pairwiseOutcomes = new Map();
+    let refSamplesForMining = [];
     if (referenceInfo !== null) {
         const { deps: refDeps } = withReferenceArm(deps, route, referenceInfo.candidate);
         const refEnvelope = route.bakeoff?.referenceEnvelopeUsd ?? req.envelopeUsdPerVariant;
@@ -109,16 +120,26 @@ async function runBakeoff(deps, captures, req, samplesSink) {
             variant: exports.REFERENCE_ARM,
             lastN: req.replayLastN,
             envelopeUsd: refEnvelope,
+            ...(retainForPairwise ? { retainOutputForPairwise: true } : {}),
             ...(req.expected !== undefined ? { expected: req.expected } : {}),
             ...(judgeCfg !== undefined ? { judge: (0, judge_1.makeJudgeScorer)(deps, route, judgeCfg) } : {}),
             ...(req.qualityGate !== undefined ? { qualityGate: req.qualityGate } : {}),
         });
         const partial = refSamples.length < replayN || refSamples.some((s) => s.failureClass === "budget_exhausted");
+        refSamplesForMining = refSamples;
         for (let i = 0; i < metrics.length; i += 1) {
             const raw = rawByArm.get(metrics[i].variant) ?? [];
+            // N5: pairwise standing vs the reference — deterministic where both sides
+            // carry value accuracy, else the opt-in capped judge (fresh per arm, so
+            // maxSamples is per-arm). A REPORTED signal beside regret.
+            const pwJudge = pairwiseJudgeCfg !== undefined ? (0, pairwise_1.makePairwiseJudge)(deps, route, pairwiseJudgeCfg) : null;
+            const { stats: pairwise, outcomes } = await (0, pairwise_1.scoreArmPairwise)(raw, refSamples, pwJudge);
+            for (const [captureId, outcome] of outcomes)
+                pairwiseOutcomes.set((0, disagreement_1.pairwiseKey)(metrics[i].variant, captureId), outcome);
             metrics[i] = {
                 ...metrics[i],
                 regret: (0, stats_1.pairedRegret)(raw, refSamples),
+                ...(pairwise !== null ? { pairwise } : {}),
                 ...(partial ? { referencePartial: true } : {}),
             };
         }
@@ -181,6 +202,15 @@ async function runBakeoff(deps, captures, req, samplesSink) {
                 "re-run with --keep-outputs to retry the samples.");
         }
     }
+    // N6: mine the replay/verdict set for disagreements → ranked eval-case
+    // candidates (never auto-promoted). Pure over the per-arm raw samples + the
+    // reference + pairwise verdicts. Rides the LOCAL artifact only — captureIds are
+    // pointers to already-captured inputs and are never mirrored.
+    const disagreement = (0, disagreement_1.mineDisagreements)({
+        arms: req.variants.map((variant) => ({ arm: variant, samples: rawByArm.get(variant) ?? [] })),
+        ...(refSamplesForMining.length > 0 ? { reference: refSamplesForMining } : {}),
+        ...(pairwiseOutcomes.size > 0 ? { pairwise: pairwiseOutcomes } : {}),
+    });
     return {
         id: bakeoffId,
         route: route.route,
@@ -192,6 +222,7 @@ async function runBakeoff(deps, captures, req, samplesSink) {
         objective,
         reference,
         qualityMetric,
+        disagreement,
     };
 }
 function isDeclared(route, candidate) {
@@ -224,6 +255,7 @@ function resolveObjective(route, req) {
         maxRegret: route.objective?.maxRegret ?? null,
         priceCeiling: route.objective?.priceCeiling ?? null,
         lambda: route.objective?.lambda ?? null,
+        ...(route.objective?.pairwiseAware === true ? { pairwiseAware: true } : {}),
     };
 }
 const EPS = 1e-9;
@@ -233,6 +265,16 @@ function savingsPct(incumbent, winner) {
         ? 0
         : ((inc - winner.effectiveUsdPer1kConformant) / inc) * 100;
 }
+/** N5 pairwise-aware gate clause (default off). When `objective.pairwiseAware` is
+ * set, an eligible arm must NOT be pairwise-worse than the reference within CI —
+ * its pairwise win-rate lower bound ≥ 0.5. Fail-closed: an arm with NO pairwise
+ * stats cannot be confirmed not-worse, so it is ineligible under the knob. When
+ * the knob is off this returns true for every arm ⇒ RB-1 behaviour byte-identical. */
+function pairwiseEligible(m, objective) {
+    if (objective.pairwiseAware !== true)
+        return true;
+    return m.pairwise != null && m.pairwise.winRateCi95[0] >= 0.5 - EPS;
+}
 /** A price ceiling test: usd_per_1k on the effective-$/1k axis, usd_per_run on
  * mean cost. Null ceiling ⇒ no cost constraint (regret alone gates). */
 function withinCeiling(m, ceiling) {
@@ -298,7 +340,8 @@ function computeProposal(requested, metrics, ctx) {
         const eligible = pool.filter((m) => m.regret != null &&
             m.regret.nPaired >= 10 &&
             (objective.maxRegret === null || m.regret.regretCi95[1] <= objective.maxRegret + EPS) &&
-            withinCeiling(m, objective.priceCeiling));
+            withinCeiling(m, objective.priceCeiling) &&
+            pairwiseEligible(m, objective));
         if (eligible.length === 0)
             return null;
         // Lowest regret, tie → lowest cost.
@@ -333,7 +376,8 @@ function computeProposal(requested, metrics, ctx) {
     const eligible = pool.filter((m) => m.regret != null &&
         m.regret.nPaired >= 10 &&
         (objective.maxRegret === null || m.regret.regretCi95[1] <= objective.maxRegret + EPS) &&
-        withinCeiling(m, objective.priceCeiling));
+        withinCeiling(m, objective.priceCeiling) &&
+        pairwiseEligible(m, objective));
     if (eligible.length === 0)
         return null;
     const winner = [...eligible].sort((a, b) => blended(a) - blended(b))[0];
@@ -395,6 +439,17 @@ function bakeoffEvidence(artifact, spentUsd) {
         reference: artifact.reference ?? null,
         regret: artifact.proposal?.regret ?? null,
         quality_metric: artifact.qualityMetric ?? null,
+        // N6: disagreement COUNTS only (numbers — meta-safe) ride the evidence; the
+        // captureId candidate list stays on the LOCAL artifact and is never sent to
+        // the control plane (content custody).
+        disagreement: artifact.disagreement === undefined
+            ? null
+            : {
+                examined: artifact.disagreement.examined,
+                found: artifact.disagreement.found,
+                capped_at: artifact.disagreement.cappedAt,
+                counts: artifact.disagreement.counts,
+            },
         matrix: artifact.variants.map((m) => ({
             variant: m.variant,
             samples: m.samples,
@@ -408,6 +463,7 @@ function bakeoffEvidence(artifact, spentUsd) {
             quality_mean: m.qualityMean,
             quality_metric: m.qualityMetric,
             regret: m.regret ?? null,
+            pairwise: m.pairwise ?? null,
         })),
     };
 }