pi-smart-router 0.9.1 → 0.9.3

package/README.md

@@ -568,8 +568,57 @@ At runtime, `ClusterMatcher` prefers `routing_centroids` from the calibration bu
 
 Below these thresholds the train path returns bootstrap centroids unchanged. The verify script reports `oats_refinement` metadata when refinement ran.
 
+These guards match `DEFAULT_OATS_MIN_POSITIVE_SAMPLES` / `DEFAULT_OATS_MIN_NEGATIVE_SAMPLES` in `scripts/lib/oats-centroid-refinement.ts` and `MINIMUM_TRAINING_SAMPLES.routing_centroids` (10) in `scripts/calibration-aggregate.ts`. The same global floors appear under `minimum_training_samples` in [`config/routing-calibration.json.example`](config/routing-calibration.json.example) (`hydra_projection` 100, `triage_thresholds` 50, `p_success_weights` / `isotonic_calibrator` 30, `routing_centroids` 10).
+
 See [routing-roadmap.md](docs/routing-roadmap.md) §2 P2 OATS and GitHub [#77](https://github.com/beettlle/pi-smart-router/issues/77).
 
+### Privacy-safe label packs + calibration dry-run (SP-189–SP-191 / #102)
+
+Offline **label packs** are feature-vector + binary-outcome JSONL (never prompt/message text). Schema: `scripts/lib/label-pack-schema.ts`. Provenance, pins, and field maps: [`tests/eval/corpus/label-packs/PROVENANCE.md`](tests/eval/corpus/label-packs/PROVENANCE.md).
+
+**Regenerate packs (offline / no network for CI fixtures):**
+
+```bash
+# SWE-Gym verifier-style → pack
+npm run routing:ingest-swe-gym -- \
+  --input tests/eval/corpus/label-packs/swe-gym/ci-fixture.jsonl \
+  --output /tmp/swe-gym-pack.jsonl
+
+# FC-RewardBench preference / flat → pack
+npm run routing:ingest-fc-rewardbench -- \
+  --input tests/eval/corpus/label-packs/fc-rewardbench/ci-fixture.jsonl \
+  --output /tmp/fc-rewardbench-pack.jsonl
+
+# Optional weak TwinRouterBench tier proxy (exclude from holdout ECE)
+npm run routing:ingest-twinrouterbench-weak -- \
+  --input tests/eval/corpus/label-packs/twinrouterbench-weak/ci-fixture.jsonl \
+  --output /tmp/trb-weak-pack.jsonl
+```
+
+**Calibration dry-run (holdout ECE):**
+
+```bash
+# CI fixtures (ingests packs in-memory; sample-starved → report-only)
+npm run routing:calibration-dry-run
+
+# Operator packs (schema-valid JSONL)
+npm run routing:calibration-dry-run -- --packs /tmp/swe-gym-pack.jsonl /tmp/fc-rewardbench-pack.jsonl
+
+# Soft ECE advisory fail (threshold 0.25 calibrated ECE; not a release-gate absolute)
+npm run routing:calibration-dry-run -- --packs /tmp/swe-gym-pack.jsonl --enforce-soft-ece
+```
+
+Dry-run behavior:
+
+| Condition | Result |
+|-----------|--------|
+| < 30 ECE-eligible rows | `SAMPLE_STARVED` report-only (exit 0); no soft pass/fail |
+| ≥ 30 ECE-eligible rows | Fit logistic + isotonic; report `holdout_ece_raw` / `holdout_ece_calibrated` |
+| Rows with `exclude_from_holdout_ece` | Counted separately; **never** enter holdout ECE metrics (weak TwinRouterBench) |
+| Soft threshold | Advisory `0.25` calibrated ECE — **does not** change `config/release-gates.json` |
+
+**#96 / `modernbert_k4` advisory:** when deciding whether to enable ModernBERT K=4 heads, use **pack holdout ECE / Top-1 error on verifier-grade packs** (SWE-Gym + FC-RewardBench), not fixture-only QR. Weak TwinRouterBench rows are warm-start only. This task does **not** flip `modernbert_k4` defaults.
+
 ### Operator tuning (frugality slider)
 
 The multi-objective scoring weights control the cost-vs-quality tradeoff:
@@ -793,6 +842,10 @@ Contributors must run `npm run build` before publishing or consuming the library
 | `npm run routing:train-calibration` | Train routing calibration artifact bundle |
 | `npm run routing:train-p-success` | Train standalone `config/p-success-weights.json` (synthetic fixture by default) |
 | `npm run routing:verify-calibration` | Verify calibration bundle against benchmark prompts |
+| `npm run routing:calibration-dry-run` | Pack-fed isotonic dry-run: holdout ECE on label packs (CI fixtures by default) |
+| `npm run routing:ingest-swe-gym` | Convert SWE-Gym verifier-style JSONL → privacy-safe label pack |
+| `npm run routing:ingest-fc-rewardbench` | Convert FC-RewardBench JSONL → privacy-safe label pack |
+| `npm run routing:ingest-twinrouterbench-weak` | Convert TwinRouterBench weak tier labels → pack (exclude from ECE) |
 | `npm run routing:ingest-benchmarks` | Regenerate `config/benchmark-profiles.json` from leaderboard fixtures |
 | `npm run routing:verify-benchmark-profiles` | CI smoke: assert checked-in profiles match fixture ingest |
 | `npm run routing:eval-replay` | Counterfactual replay on eval trace fixtures |
@@ -801,6 +854,9 @@ Contributors must run `npm run build` before publishing or consuming the library
 | `npm run routing:eval-harness:corpus-smoke` | Harness summary on TwinRouterBench CI corpus subset (`tests/eval/corpus/twinrouterbench`) |
 | `npm run routing:assert-release-gates:corpus-report` | Soft-feed: assert corpus vs absolute gates with `--report-only` (exit 0; does not gate releases) |
 | `npm run routing:ingest-twinrouterbench` | Convert TwinRouterBench `question_bank.jsonl` → CI subset / full corpus JSON |
+| `npm run routing:ingest-llmrouterbench` | Convert LLMRouterBench BaselineRecord JSONL → static-track subset JSON |
+| `npm run routing:llmrouterbench-regret` | Offline regret / CS report on vendored LLMRouterBench subset (optional; not PR CI) |
+| `npm run routing:community-bench` | Privacy-safe community bench report (Track A TwinRouterBench + optional Track B/C) |
 | `npm run benchmark:encoder` | Compare MiniLM vs Granite encoder latency on held-out agent turns |
 
 ### Offline eval harness (agent-native routing)
@@ -847,6 +903,57 @@ npm run routing:eval-replay
 
 **Deferred:** RouterBench classic (outcome-matrix) smoke is out of scope for SP-188; prefer TwinRouterBench static track + dogfood for #95.
 
+#### LLMRouterBench offline regret (SP-192 / SP-193)
+
+Optional local / nightly report on the **pinned code/tool subset** — not part of PR CI (no full HF corpus download).
+
+| Item | Location / command |
+|------|--------------------|
+| **Pinned upstream** | HF `NPULH/LLMRouterBench` `@0e5af1b84bf73437a01a1849c0f1d2468baa93fc` + git schema `@c77cb0506949d8f959e97967d2fefca0e8ff1b05` (MIT) |
+| **CI subset** | `tests/eval/corpus/llmrouterbench/ci-subset.json` (≤20 synthetic offline records) |
+| **Provenance / refresh** | `tests/eval/corpus/llmrouterbench/PROVENANCE.md` (quarterly pin refresh; re-run report after catalog/subset changes) |
+| **Regenerate subset** | `npm run routing:ingest-llmrouterbench -- --input <jsonl> --output tests/eval/corpus/llmrouterbench/ci-subset.json --limit 20 --prefer-code-tool` |
+| **Regret / CS report** | `npm run routing:llmrouterbench-regret` |
+| **Community Track C** | `npm run routing:community-bench -- --llmrouterbench` (same vendored subset; optional) |
+
+PR CI continues to smoke TwinRouterBench only (`routing:eval-harness:corpus-smoke`). Absolute `config/release-gates.json` thresholds are unchanged. See [Contribute a community bench report](#contribute-a-community-bench-report) for Track A + optional Track C sharing.
+
+#### Contribute a community bench report
+
+Share a privacy-safe setup fingerprint + Track A (TwinRouterBench) gate result with maintainers. No SMTP auto-send and no upload server — you copy artifacts yourself.
+
+**Maintainer contact** (must match the CLI footer constant `COMMUNITY_BENCH_MAINTAINER_CONTACT`):
+
+`https://github.com/beettlle/pi-smart-router/issues/new?labels=community-bench`
+
+```bash
+# Track A offline smoke (vendored TwinRouterBench corpus — no network)
+npm run routing:community-bench -- \
+  --output /tmp/community-bench-report.json \
+  --email-file /tmp/community-bench-report.txt
+
+# Optional Track C: offline LLMRouterBench regret/CS on the vendored subset (no full HF download)
+npm run routing:community-bench -- \
+  --llmrouterbench \
+  --output /tmp/community-bench-report.json \
+  --email-file /tmp/community-bench-report.txt
+```
+
+**How to send:**
+
+1. **Email `.txt`** — open `/tmp/community-bench-report.txt` (or your `--email-file` path). It includes a `Subject:` line, privacy blurb, fingerprint, Track A PASS/FAIL, and optional Track C metrics. Paste into your mail client; attach `community-bench-report.json` if useful. Do **not** expect the CLI to send mail.
+2. **GitHub issue** — open the [maintainer contact](https://github.com/beettlle/pi-smart-router/issues/new?labels=community-bench) URL, or run with `--print-issue-body` and paste stdout into a new issue. Issues list: https://github.com/beettlle/pi-smart-router/issues
+
+**Tracks:**
+
+| Track | Corpus | When |
+|-------|--------|------|
+| **A (required)** | [TwinRouterBench CI corpus](#twinrouterbench-ci-corpus-sp-186--sp-187--sp-188) (`tests/eval/corpus/twinrouterbench`) | Always |
+| **B (optional)** | Dogfood export (`--dogfood-export PATH`) | Skips with an explicit reason until [#95](https://github.com/beettlle/pi-smart-router/issues/95) adapter lands — never invents labels |
+| **C (optional)** | [LLMRouterBench offline subset](#llmrouterbench-offline-regret-sp-192--sp-193) (`tests/eval/corpus/llmrouterbench`) | `--llmrouterbench` or `--full`; offline only |
+
+PR CI does **not** download full TwinRouterBench / LLMRouterBench corpora. Absolute gate thresholds in `config/release-gates.json` are unchanged by this CLI.
+
 Sample fixtures under `tests/eval/fixtures/twinrouterbench/` remain the small adapter unit-test inputs and are unchanged by corpus ingest.
 
 Capability scores in `config/benchmark-profiles.json` are grounded from public leaderboard snapshots under `tests/fixtures/benchmark-leaderboards/` (and optional **recorded** live snapshots under `tests/fixtures/benchmark-leaderboards/recorded/`). Each artifact records provenance (`source_urls`, `scrape_date`, `catalog_freeze_date`) in its header.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-smart-router",
-  "version": "0.9.1",
+  "version": "0.9.3",
   "description": "Auto-model router middleware for the pi.dev coding agent",
   "type": "module",
   "license": "MIT",
@@ -67,12 +67,19 @@
     "routing:ingest-benchmarks": "tsx scripts/ingest-benchmark-profiles.ts",
     "routing:verify-benchmark-profiles": "vitest run tests/unit/ingest-benchmark-profiles.test.ts -t \"checked-in artifact matches fixture ingest\"",
     "routing:verify-calibration": "bash -c 'if [[ \" $* \" == *\" --skip-embed \"* ]]; then vitest run tests/unit/train-routing-calibration.test.ts -t \"verifyRoutingCalibration passes|evaluates benchmark triage gates\"; else npm run build && tsx scripts/verify-routing-calibration.ts \"$@\"; fi' --",
+    "routing:calibration-dry-run": "tsx scripts/verify-routing-calibration.ts --dry-run-packs",
     "routing:eval-replay": "tsx scripts/eval/counterfactual-replay.ts",
     "routing:eval-harness": "tsx scripts/eval/run-harness.ts",
     "routing:eval-harness:smoke": "tsx scripts/eval/run-harness.ts --summary-only",
     "routing:eval-harness:corpus-smoke": "tsx scripts/eval/run-harness.ts --fixtures tests/eval/corpus/twinrouterbench --summary-only",
     "routing:assert-release-gates:corpus-report": "tsx scripts/eval/assert-release-gates.ts --fixtures tests/eval/corpus/twinrouterbench --report-only",
+    "routing:community-bench": "tsx scripts/eval/community-bench.ts",
     "routing:ingest-twinrouterbench": "tsx scripts/eval/ingest-twinrouterbench-corpus.ts",
+    "routing:ingest-llmrouterbench": "tsx scripts/eval/ingest-llmrouterbench-subset.ts",
+    "routing:llmrouterbench-regret": "tsx scripts/eval/llmrouterbench-regret-report.ts",
+    "routing:ingest-swe-gym": "tsx scripts/ingest-swe-gym-labels.ts",
+    "routing:ingest-fc-rewardbench": "tsx scripts/ingest-fc-rewardbench-labels.ts",
+    "routing:ingest-twinrouterbench-weak": "tsx scripts/ingest-twinrouterbench-weak-labels.ts",
     "routing:capture-baseline": "tsx scripts/eval/capture-baseline.ts",
     "routing:test-projection": "npm run build && node --experimental-strip-types scripts/test-hydra-projection.ts",
     "benchmark:encoder": "tsx scripts/benchmark-encoder-latency.ts",