pi-smart-router 0.8.0 → 0.9.1

package/README.md

@@ -798,6 +798,9 @@ Contributors must run `npm run build` before publishing or consuming the library
 | `npm run routing:eval-replay` | Counterfactual replay on eval trace fixtures |
 | `npm run routing:eval-harness` | Three-track eval harness (capability, cost, continuity) on fixture traces |
 | `npm run routing:eval-harness:smoke` | Harness summary JSON only (CI smoke; no network) |
+| `npm run routing:eval-harness:corpus-smoke` | Harness summary on TwinRouterBench CI corpus subset (`tests/eval/corpus/twinrouterbench`) |
+| `npm run routing:assert-release-gates:corpus-report` | Soft-feed: assert corpus vs absolute gates with `--report-only` (exit 0; does not gate releases) |
+| `npm run routing:ingest-twinrouterbench` | Convert TwinRouterBench `question_bank.jsonl` → CI subset / full corpus JSON |
 | `npm run benchmark:encoder` | Compare MiniLM vs Granite encoder latency on held-out agent turns |
 
 ### Offline eval harness (agent-native routing)
@@ -812,9 +815,12 @@ Run locally:
 # Full metrics JSON (per-fixture + aggregate track summaries)
 npm run routing:eval-harness
 
-# CI-style summary only
+# CI-style summary only (default fixtures under tests/eval/fixtures)
 npm run routing:eval-harness:smoke
 
+# TwinRouterBench CI corpus subset (≤50 code/tool records; offline)
+npm run routing:eval-harness:corpus-smoke
+
 # Custom fixture directory (includes TwinRouterBench static track subdirs)
 npm run routing:eval-harness -- --fixtures tests/eval/fixtures
 
@@ -822,13 +828,28 @@ npm run routing:eval-harness -- --fixtures tests/eval/fixtures
 npm run routing:eval-replay
 ```
 
-**CI smoke:** `.github/workflows/eval-harness-smoke.yml` runs on PRs that touch eval scripts, fixtures, or the workflow. It executes `routing:eval-harness:smoke` and eval unit tests — fast, offline, no provider network calls.
+**CI smoke:** `.github/workflows/eval-harness-smoke.yml` runs on PRs that touch eval scripts, fixtures, or the workflow. It executes `routing:eval-harness:smoke`, `routing:eval-harness:corpus-smoke`, and eval unit tests — fast, offline, no provider network calls. Job timeout stays at 10 minutes.
 
 **TwinRouterBench static track:** import step-level router-visible prefixes with execution-verified target tiers (`track: "static"`). The adapter in `scripts/eval/twinrouterbench-adapter.ts` converts static track records into native eval fixtures for the three-track harness. See `docs/gemini-research.md` §9 for methodology context.
 
-### Benchmark profile refresh
+#### TwinRouterBench CI corpus (SP-186 / SP-187 / SP-188)
+
+| Item | Location / command |
+|------|--------------------|
+| **Pinned upstream** | CommonstackAI/TwinRouterBench `@430acecac71141de77afd8e5e13690d236d58e93` (Apache-2.0) |
+| **CI subset** | `tests/eval/corpus/twinrouterbench/ci-subset.json` (≤50 code/tool records) |
+| **Provenance** | `tests/eval/corpus/twinrouterbench/PROVENANCE.md` |
+| **Regenerate** | `npm run routing:ingest-twinrouterbench -- --input <question_bank.jsonl> --output tests/eval/corpus/twinrouterbench/ci-subset.json --limit 50 --prefer-code-tool` |
+| **Harness smoke** | `npm run routing:eval-harness:corpus-smoke` |
+| **Gate soft-feed** | `npm run routing:assert-release-gates:corpus-report` |
+
+**Absolute release gates stay on default fixtures.** `npm run release:functional-smoke` continues to assert `tests/eval/fixtures` against `config/release-gates.json` — do not point it at the corpus without operator review. Today the corpus subset fails `mean_over_routing_rate_max` (≈0.85 vs absolute max 0.15); that gap is intentional soft signal for the [#95](https://github.com/beettlle/pi-smart-router/issues/95) public static-track acceptance criteria alongside live dogfood traces. Use `--fixtures tests/eval/corpus/twinrouterbench` (or the corpus-report script) for #95 public-track scoring; keep absolute threshold edits out of band until operators approve.
 
-Capability scores in `config/benchmark-profiles.json` are grounded from public leaderboard snapshots under `tests/fixtures/benchmark-leaderboards/`. Each artifact records provenance (`source_urls`, `scrape_date`, `catalog_freeze_date`) in its header.
+**Deferred:** RouterBench classic (outcome-matrix) smoke is out of scope for SP-188; prefer TwinRouterBench static track + dogfood for #95.
+
+Sample fixtures under `tests/eval/fixtures/twinrouterbench/` remain the small adapter unit-test inputs and are unchanged by corpus ingest.
+
+Capability scores in `config/benchmark-profiles.json` are grounded from public leaderboard snapshots under `tests/fixtures/benchmark-leaderboards/` (and optional **recorded** live snapshots under `tests/fixtures/benchmark-leaderboards/recorded/`). Each artifact records provenance (`source_urls`, `scrape_date`, `catalog_freeze_date`) in its header.
 
 **Fleet ID aliases (SP-174):** live pi/Cursor scoped-fleet model IDs often differ from leaderboard `model_id` strings. The artifact’s optional `aliases` map sends those fleet IDs to an existing grounded row (never invents scores). `mapPiModelToProfile` sets `capability_source` to `benchmark` when a direct row or alias hits, otherwise `pattern_default`. Operators can also call `getCapabilitySource(modelId)` / `resolveBenchmarkModelId(modelId)`.
 
@@ -840,16 +861,61 @@ Capability scores in `config/benchmark-profiles.json` are grounded from public l
 4. Re-run ingest anytime — the CLI **preserves** existing `aliases` from the output file. Seed defaults live in `DEFAULT_FLEET_BENCHMARK_ALIASES` when no prior artifact exists.
 5. Confirm with `npm run routing:verify-benchmark-profiles` and a mapper unit test that `capability_source === 'benchmark'` for the fleet id.
 
+**Operator refresh command (SP-179 / SP-180):**
+
+| Mode | Command | Network? | When to use |
+|------|---------|----------|-------------|
+| **Fixtures (default)** | `npm run routing:ingest-benchmarks` | No | Local edits, CI, PR smoke |
+| **Recorded replay** | `npm run routing:ingest-benchmarks -- --recorded` | No | Replay last successful live snapshots offline |
+| **Live + record** | `npm run routing:ingest-benchmarks -- --live` | Yes | Operator refresh; writes `tests/fixtures/benchmark-leaderboards/recorded/` then regenerates profiles |
+
+Optional flags: `--catalog-freeze-date YYYY-MM-DD`, `--scrape-date YYYY-MM-DD`, `--record-dir DIR`, `--live-url BENCHMARK=URL`, `--output PATH`. See `npm run routing:ingest-benchmarks -- --help`.
+
+**Live sources (per benchmark):** each `--live` run resolves independently — live adapter → recorded → checked-in fixtures. One failing source never invents scores or blocks siblings. Logs: `ingest-benchmark-profiles: <id> source=live|recorded|fixture (…)`.
+
+| Benchmark | Default live fetch | Score field | Fallback |
+|-----------|-------------------|-------------|----------|
+| `swebench_verified` | Native: [SWE-bench `leaderboards.json`](https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json) (Verified board) | `resolved` → `score` (0–100) | recorded → fixtures |
+| `livecodebench` | Native: [LCB `performances_generation.json`](https://raw.githubusercontent.com/LiveCodeBench/livecodebench.github.io/main/src/mocks/performances_generation.json) | aggregate `pass@1` | recorded → fixtures |
+| `bfcl` | Native: [Gorilla `data_overall.csv`](https://raw.githubusercontent.com/ShishirPatil/gorilla/gh-pages/data_overall.csv) | `Overall Acc` | recorded → fixtures |
+| `terminal_bench` | **No free stable JSON** (tbench.ai is HTML; HF leaderboard is submissions-only; paid Parse API is **not** the default). Pass `--live-url terminal_bench=URL` at a fixture-shaped mirror | `score` (0–100) | recorded → fixtures |
+
+**Terminal-Bench operator mirror schema** (SP-185 / #104):
+
+```json
+{
+  "benchmark": "terminal_bench",
+  "source_url": "https://www.tbench.ai/leaderboard",
+  "scrape_date": "YYYY-MM-DD",
+  "entries": [{ "model_id": "claude-opus-4-5", "score": 72.5 }]
+}
+```
+
+Example: `npm run routing:ingest-benchmarks -- --live --live-url terminal_bench=https://example.com/tb-mirror.json`. HTML bodies fail fast; without `--live-url`, TB uses recorded/fixtures. `release:refresh-benchmarks` uses the same `--live` path (fixture fallback on total failure).
+
+**Cadence (release-tied, not calendar):**
+
+| Trigger | When | Behavior |
+|---------|------|----------|
+| **Pre-tag release gate** | `npm run release:check` → `release:refresh-benchmarks` | Attempt **live** ingest; on failure fall back to fixtures; **fail if** `config/benchmark-profiles.json` or recorded snapshots are dirty — commit on `main`, re-run, then `npm version` / tag |
+| **Manual dispatch** | Actions → *Benchmark Profile Refresh* → `workflow_dispatch` (`use_live` default `true`) | Same live-or-fixture path; opens a bot PR when scores change; set `use_live=false` for fixtures-only |
+| **PR smoke** | PRs touching fixtures / ingest / artifact / workflow | **Fixtures only** — `npm run routing:verify-benchmark-profiles` (offline, no network) |
+
+There is **no monthly cron**. Refresh runs when you ship so each release packages the latest grounded scores. Tag-triggered Release publish uses `--ignore-scripts` and does not re-fetch (profiles are already frozen in the tag). Offline skip: `SMART_ROUTER_SKIP_LIVE_BENCHMARK_REFRESH=1 npm run release:check`.
+
 **Operator policy:**
 
-1. **PR smoke** — `.github/workflows/benchmark-profile-refresh.yml` runs on PRs that touch fixtures, ingest, or the checked-in artifact. It executes `npm run routing:verify-benchmark-profiles` so fixture edits cannot drift from `config/benchmark-profiles.json`.
-2. **Monthly refresh** — the same workflow runs on the 1st of each month (06:00 UTC) and via `workflow_dispatch`. It re-ingests fixtures, updates `catalog_freeze_date` to the run date, and opens a PR when model scores change.
-3. **Manual updates** — after editing fixture snapshots, run `npm run routing:ingest-benchmarks` (optionally `--catalog-freeze-date YYYY-MM-DD`) and commit the regenerated `config/benchmark-profiles.json` with the PR.
+1. **PR smoke** — fixture-only verify so PRs never require live network.
+2. **Every release** — live with fixture fallback via `release:check`; commit any profile/recorded diffs on `main` before tagging.
+3. **Ad-hoc** — Actions dispatch or local `--live` when refreshing between releases.
+4. **Manual local updates** — prefer fixtures or `--recorded` for offline work; use `--live` when refreshing from public leaderboard JSON endpoints.
 
-Regenerate locally:
+Verify after any regenerate:
 
 ```bash
 npm run routing:ingest-benchmarks
+# or: npm run routing:ingest-benchmarks -- --live
+# or: npm run routing:ingest-benchmarks -- --recorded
 npm run routing:verify-benchmark-profiles
 ```
 
@@ -863,7 +929,9 @@ Tag-triggered publish via GitHub Actions (requires `NPMSECRET` repository secret
 2. `routing:verify-benchmark-profiles` — checked-in capability profiles match fixture ingest
 3. `assert-release-gates --fixtures tests/eval/fixtures --baseline-version 0.6.0` — eval harness aggregate metrics vs `config/release-gates.json` and semver baseline regression vs `tests/eval/baselines/v0.6.0.json`
 
-`release:check` runs the full pre-release path: `verify:ci`, consumer pack verify, then Tier 0 functional smoke.
+The TwinRouterBench CI corpus is **not** part of Tier 0: use `routing:eval-harness:corpus-smoke` / `routing:assert-release-gates:corpus-report` for offline public-track soft-feed ([#95](https://github.com/beettlle/pi-smart-router/issues/95)). Absolute thresholds in `config/release-gates.json` stay fixture-backed until operators approve a change.
+
+`release:check` runs the full pre-release path: **live benchmark profile refresh** (fixture fallback; dirty-tree fail), then `verify:ci`, consumer pack verify, then Tier 0 functional smoke.
 
 **Baseline re-capture (post-tag):** after shipping a new semver (e.g. v0.7.0), freeze harness metrics for the next regression reference:
 
@@ -876,7 +944,8 @@ npm run routing:capture-baseline -- --version 0.7.0
 
 Commit the new baseline JSON and update `baseline_regression.reference_version` in `config/release-gates.json` plus the `--baseline-version` flag in `release:functional-smoke`. Re-run `npm run release:check` before tagging the next release.
 
-1. `npm run release:check` (CI parity + consumer pack + Tier 0 functional smoke)
+1. `npm run release:check` (live benchmark refresh → CI parity + consumer pack + Tier 0 functional smoke)
+   - If refresh rewrites profiles/recorded snapshots, **commit them on `main`** and re-run until clean
 2. `npm version 0.1.1` (creates commit + `v0.1.1` tag)
 3. `git push && git push --tags`
 4. Actions → **Release** runs pack smoke, consumer pack verify, Tier 0 functional smoke, `npm publish`, and creates a GitHub Release

package/config/benchmark-profiles.json

@@ -7,8 +7,8 @@
       "livecodebench": "https://livecodebench.github.io/leaderboard.html",
       "bfcl": "https://gorilla.cs.berkeley.edu/leaderboard.html"
     },
-    "scrape_date": "2026-07-09",
-    "catalog_freeze_date": "2026-07-09"
+    "scrape_date": "2026-07-11",
+    "catalog_freeze_date": "2026-07-11"
   },
   "aliases": {
     "claude-3-5-sonnet": "claude-sonnet-4-6",
@@ -34,50 +34,24 @@
   },
   "models": [
     {
-      "model_id": "claude-3.5-haiku",
+      "model_id": "claude-opus-4-5",
       "capabilities": {
-        "reasoning": 0.5685,
-        "code_gen": 0.587,
-        "tool_use": 0.6095
+        "reasoning": 0.7585,
+        "code_gen": 0.708,
+        "tool_use": 0.7498
       },
       "benchmark_sources": {
         "bfcl": {
-          "raw_score": 70.6,
-          "normalized": 0.706
+          "raw_score": 77.47,
+          "normalized": 0.7746999999999999
         },
         "livecodebench": {
-          "raw_score": 55,
-          "normalized": 0.55
-        },
-        "swebench_verified": {
           "raw_score": 62.4,
           "normalized": 0.624
         },
-        "terminal_bench": {
-          "raw_score": 51.3,
-          "normalized": 0.513
-        }
-      }
-    },
-    {
-      "model_id": "claude-opus-4-5",
-      "capabilities": {
-        "reasoning": 0.767,
-        "code_gen": 0.7915,
-        "tool_use": 0.8035
-      },
-      "benchmark_sources": {
-        "bfcl": {
-          "raw_score": 88.2,
-          "normalized": 0.882
-        },
-        "livecodebench": {
-          "raw_score": 77.4,
-          "normalized": 0.774
-        },
         "swebench_verified": {
-          "raw_score": 80.9,
-          "normalized": 0.809
+          "raw_score": 79.2,
+          "normalized": 0.792
         },
         "terminal_bench": {
           "raw_score": 72.5,
@@ -88,22 +62,22 @@
     {
       "model_id": "claude-sonnet-4-6",
       "capabilities": {
-        "reasoning": 0.738,
-        "code_gen": 0.7635,
-        "tool_use": 0.7625
+        "reasoning": 0.724,
+        "code_gen": 0.681,
+        "tool_use": 0.7062
       },
       "benchmark_sources": {
         "bfcl": {
-          "raw_score": 84.5,
-          "normalized": 0.845
+          "raw_score": 73.24,
+          "normalized": 0.7323999999999999
         },
         "livecodebench": {
-          "raw_score": 73.1,
-          "normalized": 0.731
+          "raw_score": 59.4,
+          "normalized": 0.594
         },
         "swebench_verified": {
-          "raw_score": 79.6,
-          "normalized": 0.7959999999999999
+          "raw_score": 76.8,
+          "normalized": 0.768
         },
         "terminal_bench": {
           "raw_score": 68,
@@ -114,22 +88,22 @@
     {
       "model_id": "gemini-2.5-flash",
       "capabilities": {
-        "reasoning": 0.5295,
-        "code_gen": 0.5525,
-        "tool_use": 0.5735
+        "reasoning": 0.3827,
+        "code_gen": 0.5192,
+        "tool_use": 0.5202
       },
       "benchmark_sources": {
         "bfcl": {
-          "raw_score": 66.9,
-          "normalized": 0.669
+          "raw_score": 56.24,
+          "normalized": 0.5624
         },
         "livecodebench": {
-          "raw_score": 52.4,
-          "normalized": 0.524
+          "raw_score": 75.1,
+          "normalized": 0.7509999999999999
         },
         "swebench_verified": {
-          "raw_score": 58.1,
-          "normalized": 0.581
+          "raw_score": 28.73,
+          "normalized": 0.2873
         },
         "terminal_bench": {
           "raw_score": 47.8,
@@ -140,22 +114,14 @@
     {
       "model_id": "gpt-5.3-codex",
       "capabilities": {
-        "reasoning": 0.796,
-        "code_gen": 0.833,
-        "tool_use": 0.8215
+        "reasoning": 0.743,
+        "code_gen": 0.744,
+        "tool_use": 0.742
       },
       "benchmark_sources": {
-        "bfcl": {
-          "raw_score": 90.1,
-          "normalized": 0.9009999999999999
-        },
-        "livecodebench": {
-          "raw_score": 81.6,
-          "normalized": 0.816
-        },
         "swebench_verified": {
-          "raw_score": 85,
-          "normalized": 0.85
+          "raw_score": 74.4,
+          "normalized": 0.7440000000000001
         },
         "terminal_bench": {
           "raw_score": 74.2,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-smart-router",
-  "version": "0.8.0",
+  "version": "0.9.1",
   "description": "Auto-model router middleware for the pi.dev coding agent",
   "type": "module",
   "license": "MIT",
@@ -51,7 +51,8 @@
   "scripts": {
     "build": "tsc --project tsconfig.build.json",
     "prepublishOnly": "npm run release:check",
-    "release:check": "npm run verify:ci && npm run release:consumer-pack && npm run release:functional-smoke",
+    "release:check": "npm run release:refresh-benchmarks && npm run verify:ci && npm run release:consumer-pack && npm run release:functional-smoke",
+    "release:refresh-benchmarks": "tsx scripts/release-refresh-benchmark-profiles.ts",
     "release:consumer-pack": "bash scripts/verify-consumer-pack.sh",
     "release:functional-smoke": "npm run routing:verify-calibration -- --skip-embed && npm run routing:verify-benchmark-profiles && tsx scripts/eval/assert-release-gates.ts --fixtures tests/eval/fixtures --baseline-version 0.6.0",
     "typecheck": "tsc --noEmit",
@@ -69,6 +70,9 @@
     "routing:eval-replay": "tsx scripts/eval/counterfactual-replay.ts",
     "routing:eval-harness": "tsx scripts/eval/run-harness.ts",
     "routing:eval-harness:smoke": "tsx scripts/eval/run-harness.ts --summary-only",
+    "routing:eval-harness:corpus-smoke": "tsx scripts/eval/run-harness.ts --fixtures tests/eval/corpus/twinrouterbench --summary-only",
+    "routing:assert-release-gates:corpus-report": "tsx scripts/eval/assert-release-gates.ts --fixtures tests/eval/corpus/twinrouterbench --report-only",
+    "routing:ingest-twinrouterbench": "tsx scripts/eval/ingest-twinrouterbench-corpus.ts",
     "routing:capture-baseline": "tsx scripts/eval/capture-baseline.ts",
     "routing:test-projection": "npm run build && node --experimental-strip-types scripts/test-hydra-projection.ts",
     "benchmark:encoder": "tsx scripts/benchmark-encoder-latency.ts",