pi-smart-router 0.8.0 → 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +53 -7
- package/config/benchmark-profiles.json +33 -67
- package/package.json +3 -2
package/README.md
CHANGED
|
@@ -828,7 +828,7 @@ npm run routing:eval-replay
|
|
|
828
828
|
|
|
829
829
|
### Benchmark profile refresh
|
|
830
830
|
|
|
831
|
-
Capability scores in `config/benchmark-profiles.json` are grounded from public leaderboard snapshots under `tests/fixtures/benchmark-leaderboards
|
|
831
|
+
Capability scores in `config/benchmark-profiles.json` are grounded from public leaderboard snapshots under `tests/fixtures/benchmark-leaderboards/` (and optional **recorded** live snapshots under `tests/fixtures/benchmark-leaderboards/recorded/`). Each artifact records provenance (`source_urls`, `scrape_date`, `catalog_freeze_date`) in its header.
|
|
832
832
|
|
|
833
833
|
**Fleet ID aliases (SP-174):** live pi/Cursor scoped-fleet model IDs often differ from leaderboard `model_id` strings. The artifact’s optional `aliases` map sends those fleet IDs to an existing grounded row (never invents scores). `mapPiModelToProfile` sets `capability_source` to `benchmark` when a direct row or alias hits, otherwise `pattern_default`. Operators can also call `getCapabilitySource(modelId)` / `resolveBenchmarkModelId(modelId)`.
|
|
834
834
|
|
|
@@ -840,16 +840,61 @@ Capability scores in `config/benchmark-profiles.json` are grounded from public l
|
|
|
840
840
|
4. Re-run ingest anytime — the CLI **preserves** existing `aliases` from the output file. Seed defaults live in `DEFAULT_FLEET_BENCHMARK_ALIASES` when no prior artifact exists.
|
|
841
841
|
5. Confirm with `npm run routing:verify-benchmark-profiles` and a mapper unit test that `capability_source === 'benchmark'` for the fleet id.
|
|
842
842
|
|
|
843
|
+
**Operator refresh command (SP-179 / SP-180):**
|
|
844
|
+
|
|
845
|
+
| Mode | Command | Network? | When to use |
|
|
846
|
+
|------|---------|----------|-------------|
|
|
847
|
+
| **Fixtures (default)** | `npm run routing:ingest-benchmarks` | No | Local edits, CI, PR smoke |
|
|
848
|
+
| **Recorded replay** | `npm run routing:ingest-benchmarks -- --recorded` | No | Replay last successful live snapshots offline |
|
|
849
|
+
| **Live + record** | `npm run routing:ingest-benchmarks -- --live` | Yes | Operator refresh; writes `tests/fixtures/benchmark-leaderboards/recorded/` then regenerates profiles |
|
|
850
|
+
|
|
851
|
+
Optional flags: `--catalog-freeze-date YYYY-MM-DD`, `--scrape-date YYYY-MM-DD`, `--record-dir DIR`, `--live-url BENCHMARK=URL`, `--output PATH`. See `npm run routing:ingest-benchmarks -- --help`.
|
|
852
|
+
|
|
853
|
+
**Live sources (per benchmark):** each `--live` run resolves independently — live adapter → recorded → checked-in fixtures. One failing source never invents scores or blocks siblings. Logs: `ingest-benchmark-profiles: <id> source=live|recorded|fixture (…)`.
|
|
854
|
+
|
|
855
|
+
| Benchmark | Default live fetch | Score field | Fallback |
|
|
856
|
+
|-----------|-------------------|-------------|----------|
|
|
857
|
+
| `swebench_verified` | Native: [SWE-bench `leaderboards.json`](https://raw.githubusercontent.com/SWE-bench/swe-bench.github.io/master/data/leaderboards.json) (Verified board) | `resolved` → `score` (0–100) | recorded → fixtures |
|
|
858
|
+
| `livecodebench` | Native: [LCB `performances_generation.json`](https://raw.githubusercontent.com/LiveCodeBench/livecodebench.github.io/main/src/mocks/performances_generation.json) | aggregate `pass@1` | recorded → fixtures |
|
|
859
|
+
| `bfcl` | Native: [Gorilla `data_overall.csv`](https://raw.githubusercontent.com/ShishirPatil/gorilla/gh-pages/data_overall.csv) | `Overall Acc` | recorded → fixtures |
|
|
860
|
+
| `terminal_bench` | **No free stable JSON** (tbench.ai is HTML; HF leaderboard is submissions-only; paid Parse API is **not** the default). Pass `--live-url terminal_bench=URL` at a fixture-shaped mirror | `score` (0–100) | recorded → fixtures |
|
|
861
|
+
|
|
862
|
+
**Terminal-Bench operator mirror schema** (SP-185 / #104):
|
|
863
|
+
|
|
864
|
+
```json
|
|
865
|
+
{
|
|
866
|
+
"benchmark": "terminal_bench",
|
|
867
|
+
"source_url": "https://www.tbench.ai/leaderboard",
|
|
868
|
+
"scrape_date": "YYYY-MM-DD",
|
|
869
|
+
"entries": [{ "model_id": "claude-opus-4-5", "score": 72.5 }]
|
|
870
|
+
}
|
|
871
|
+
```
|
|
872
|
+
|
|
873
|
+
Example: `npm run routing:ingest-benchmarks -- --live --live-url terminal_bench=https://example.com/tb-mirror.json`. HTML bodies fail fast; without `--live-url`, TB uses recorded/fixtures. `release:refresh-benchmarks` uses the same `--live` path (fixture fallback on total failure).
|
|
874
|
+
|
|
875
|
+
**Cadence (release-tied, not calendar):**
|
|
876
|
+
|
|
877
|
+
| Trigger | When | Behavior |
|
|
878
|
+
|---------|------|----------|
|
|
879
|
+
| **Pre-tag release gate** | `npm run release:check` → `release:refresh-benchmarks` | Attempt **live** ingest; on failure fall back to fixtures; **fail if** `config/benchmark-profiles.json` or recorded snapshots are dirty — commit on `main`, re-run, then `npm version` / tag |
|
|
880
|
+
| **Manual dispatch** | Actions → *Benchmark Profile Refresh* → `workflow_dispatch` (`use_live` default `true`) | Same live-or-fixture path; opens a bot PR when scores change; set `use_live=false` for fixtures-only |
|
|
881
|
+
| **PR smoke** | PRs touching fixtures / ingest / artifact / workflow | **Fixtures only** — `npm run routing:verify-benchmark-profiles` (offline, no network) |
|
|
882
|
+
|
|
883
|
+
There is **no monthly cron**. Refresh runs when you ship so each release packages the latest grounded scores. Tag-triggered Release publish uses `--ignore-scripts` and does not re-fetch (profiles are already frozen in the tag). Offline skip: `SMART_ROUTER_SKIP_LIVE_BENCHMARK_REFRESH=1 npm run release:check`.
|
|
884
|
+
|
|
843
885
|
**Operator policy:**
|
|
844
886
|
|
|
845
|
-
1. **PR smoke** —
|
|
846
|
-
2. **
|
|
847
|
-
3. **
|
|
887
|
+
1. **PR smoke** — fixture-only verify so PRs never require live network.
|
|
888
|
+
2. **Every release** — live with fixture fallback via `release:check`; commit any profile/recorded diffs on `main` before tagging.
|
|
889
|
+
3. **Ad-hoc** — Actions dispatch or local `--live` when refreshing between releases.
|
|
890
|
+
4. **Manual local updates** — prefer fixtures or `--recorded` for offline work; use `--live` when refreshing from public leaderboard JSON endpoints.
|
|
848
891
|
|
|
849
|
-
|
|
892
|
+
Verify after any regenerate:
|
|
850
893
|
|
|
851
894
|
```bash
|
|
852
895
|
npm run routing:ingest-benchmarks
|
|
896
|
+
# or: npm run routing:ingest-benchmarks -- --live
|
|
897
|
+
# or: npm run routing:ingest-benchmarks -- --recorded
|
|
853
898
|
npm run routing:verify-benchmark-profiles
|
|
854
899
|
```
|
|
855
900
|
|
|
@@ -863,7 +908,7 @@ Tag-triggered publish via GitHub Actions (requires `NPMSECRET` repository secret
|
|
|
863
908
|
2. `routing:verify-benchmark-profiles` — checked-in capability profiles match fixture ingest
|
|
864
909
|
3. `assert-release-gates --fixtures tests/eval/fixtures --baseline-version 0.6.0` — eval harness aggregate metrics vs `config/release-gates.json` and semver baseline regression vs `tests/eval/baselines/v0.6.0.json`
|
|
865
910
|
|
|
866
|
-
`release:check` runs the full pre-release path: `verify:ci`, consumer pack verify, then Tier 0 functional smoke.
|
|
911
|
+
`release:check` runs the full pre-release path: **live benchmark profile refresh** (fixture fallback; dirty-tree fail), then `verify:ci`, consumer pack verify, then Tier 0 functional smoke.
|
|
867
912
|
|
|
868
913
|
**Baseline re-capture (post-tag):** after shipping a new semver (e.g. v0.7.0), freeze harness metrics for the next regression reference:
|
|
869
914
|
|
|
@@ -876,7 +921,8 @@ npm run routing:capture-baseline -- --version 0.7.0
|
|
|
876
921
|
|
|
877
922
|
Commit the new baseline JSON and update `baseline_regression.reference_version` in `config/release-gates.json` plus the `--baseline-version` flag in `release:functional-smoke`. Re-run `npm run release:check` before tagging the next release.
|
|
878
923
|
|
|
879
|
-
1. `npm run release:check` (CI parity + consumer pack + Tier 0 functional smoke)
|
|
924
|
+
1. `npm run release:check` (live benchmark refresh → CI parity + consumer pack + Tier 0 functional smoke)
|
|
925
|
+
- If refresh rewrites profiles/recorded snapshots, **commit them on `main`** and re-run until clean
|
|
880
926
|
2. `npm version 0.1.1` (creates commit + `v0.1.1` tag)
|
|
881
927
|
3. `git push && git push --tags`
|
|
882
928
|
4. Actions → **Release** runs pack smoke, consumer pack verify, Tier 0 functional smoke, `npm publish`, and creates a GitHub Release
|
|
@@ -7,8 +7,8 @@
|
|
|
7
7
|
"livecodebench": "https://livecodebench.github.io/leaderboard.html",
|
|
8
8
|
"bfcl": "https://gorilla.cs.berkeley.edu/leaderboard.html"
|
|
9
9
|
},
|
|
10
|
-
"scrape_date": "2026-07-
|
|
11
|
-
"catalog_freeze_date": "2026-07-
|
|
10
|
+
"scrape_date": "2026-07-11",
|
|
11
|
+
"catalog_freeze_date": "2026-07-11"
|
|
12
12
|
},
|
|
13
13
|
"aliases": {
|
|
14
14
|
"claude-3-5-sonnet": "claude-sonnet-4-6",
|
|
@@ -34,50 +34,24 @@
|
|
|
34
34
|
},
|
|
35
35
|
"models": [
|
|
36
36
|
{
|
|
37
|
-
"model_id": "claude-
|
|
37
|
+
"model_id": "claude-opus-4-5",
|
|
38
38
|
"capabilities": {
|
|
39
|
-
"reasoning": 0.
|
|
40
|
-
"code_gen": 0.
|
|
41
|
-
"tool_use": 0.
|
|
39
|
+
"reasoning": 0.7585,
|
|
40
|
+
"code_gen": 0.708,
|
|
41
|
+
"tool_use": 0.7498
|
|
42
42
|
},
|
|
43
43
|
"benchmark_sources": {
|
|
44
44
|
"bfcl": {
|
|
45
|
-
"raw_score":
|
|
46
|
-
"normalized": 0.
|
|
45
|
+
"raw_score": 77.47,
|
|
46
|
+
"normalized": 0.7746999999999999
|
|
47
47
|
},
|
|
48
48
|
"livecodebench": {
|
|
49
|
-
"raw_score": 55,
|
|
50
|
-
"normalized": 0.55
|
|
51
|
-
},
|
|
52
|
-
"swebench_verified": {
|
|
53
49
|
"raw_score": 62.4,
|
|
54
50
|
"normalized": 0.624
|
|
55
51
|
},
|
|
56
|
-
"terminal_bench": {
|
|
57
|
-
"raw_score": 51.3,
|
|
58
|
-
"normalized": 0.513
|
|
59
|
-
}
|
|
60
|
-
}
|
|
61
|
-
},
|
|
62
|
-
{
|
|
63
|
-
"model_id": "claude-opus-4-5",
|
|
64
|
-
"capabilities": {
|
|
65
|
-
"reasoning": 0.767,
|
|
66
|
-
"code_gen": 0.7915,
|
|
67
|
-
"tool_use": 0.8035
|
|
68
|
-
},
|
|
69
|
-
"benchmark_sources": {
|
|
70
|
-
"bfcl": {
|
|
71
|
-
"raw_score": 88.2,
|
|
72
|
-
"normalized": 0.882
|
|
73
|
-
},
|
|
74
|
-
"livecodebench": {
|
|
75
|
-
"raw_score": 77.4,
|
|
76
|
-
"normalized": 0.774
|
|
77
|
-
},
|
|
78
52
|
"swebench_verified": {
|
|
79
|
-
"raw_score":
|
|
80
|
-
"normalized": 0.
|
|
53
|
+
"raw_score": 79.2,
|
|
54
|
+
"normalized": 0.792
|
|
81
55
|
},
|
|
82
56
|
"terminal_bench": {
|
|
83
57
|
"raw_score": 72.5,
|
|
@@ -88,22 +62,22 @@
|
|
|
88
62
|
{
|
|
89
63
|
"model_id": "claude-sonnet-4-6",
|
|
90
64
|
"capabilities": {
|
|
91
|
-
"reasoning": 0.
|
|
92
|
-
"code_gen": 0.
|
|
93
|
-
"tool_use": 0.
|
|
65
|
+
"reasoning": 0.724,
|
|
66
|
+
"code_gen": 0.681,
|
|
67
|
+
"tool_use": 0.7062
|
|
94
68
|
},
|
|
95
69
|
"benchmark_sources": {
|
|
96
70
|
"bfcl": {
|
|
97
|
-
"raw_score":
|
|
98
|
-
"normalized": 0.
|
|
71
|
+
"raw_score": 73.24,
|
|
72
|
+
"normalized": 0.7323999999999999
|
|
99
73
|
},
|
|
100
74
|
"livecodebench": {
|
|
101
|
-
"raw_score":
|
|
102
|
-
"normalized": 0.
|
|
75
|
+
"raw_score": 59.4,
|
|
76
|
+
"normalized": 0.594
|
|
103
77
|
},
|
|
104
78
|
"swebench_verified": {
|
|
105
|
-
"raw_score":
|
|
106
|
-
"normalized": 0.
|
|
79
|
+
"raw_score": 76.8,
|
|
80
|
+
"normalized": 0.768
|
|
107
81
|
},
|
|
108
82
|
"terminal_bench": {
|
|
109
83
|
"raw_score": 68,
|
|
@@ -114,22 +88,22 @@
|
|
|
114
88
|
{
|
|
115
89
|
"model_id": "gemini-2.5-flash",
|
|
116
90
|
"capabilities": {
|
|
117
|
-
"reasoning": 0.
|
|
118
|
-
"code_gen": 0.
|
|
119
|
-
"tool_use": 0.
|
|
91
|
+
"reasoning": 0.3827,
|
|
92
|
+
"code_gen": 0.5192,
|
|
93
|
+
"tool_use": 0.5202
|
|
120
94
|
},
|
|
121
95
|
"benchmark_sources": {
|
|
122
96
|
"bfcl": {
|
|
123
|
-
"raw_score":
|
|
124
|
-
"normalized": 0.
|
|
97
|
+
"raw_score": 56.24,
|
|
98
|
+
"normalized": 0.5624
|
|
125
99
|
},
|
|
126
100
|
"livecodebench": {
|
|
127
|
-
"raw_score":
|
|
128
|
-
"normalized": 0.
|
|
101
|
+
"raw_score": 75.1,
|
|
102
|
+
"normalized": 0.7509999999999999
|
|
129
103
|
},
|
|
130
104
|
"swebench_verified": {
|
|
131
|
-
"raw_score":
|
|
132
|
-
"normalized": 0.
|
|
105
|
+
"raw_score": 28.73,
|
|
106
|
+
"normalized": 0.2873
|
|
133
107
|
},
|
|
134
108
|
"terminal_bench": {
|
|
135
109
|
"raw_score": 47.8,
|
|
@@ -140,22 +114,14 @@
|
|
|
140
114
|
{
|
|
141
115
|
"model_id": "gpt-5.3-codex",
|
|
142
116
|
"capabilities": {
|
|
143
|
-
"reasoning": 0.
|
|
144
|
-
"code_gen": 0.
|
|
145
|
-
"tool_use": 0.
|
|
117
|
+
"reasoning": 0.743,
|
|
118
|
+
"code_gen": 0.744,
|
|
119
|
+
"tool_use": 0.742
|
|
146
120
|
},
|
|
147
121
|
"benchmark_sources": {
|
|
148
|
-
"bfcl": {
|
|
149
|
-
"raw_score": 90.1,
|
|
150
|
-
"normalized": 0.9009999999999999
|
|
151
|
-
},
|
|
152
|
-
"livecodebench": {
|
|
153
|
-
"raw_score": 81.6,
|
|
154
|
-
"normalized": 0.816
|
|
155
|
-
},
|
|
156
122
|
"swebench_verified": {
|
|
157
|
-
"raw_score":
|
|
158
|
-
"normalized": 0.
|
|
123
|
+
"raw_score": 74.4,
|
|
124
|
+
"normalized": 0.7440000000000001
|
|
159
125
|
},
|
|
160
126
|
"terminal_bench": {
|
|
161
127
|
"raw_score": 74.2,
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-smart-router",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.9.0",
|
|
4
4
|
"description": "Auto-model router middleware for the pi.dev coding agent",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -51,7 +51,8 @@
|
|
|
51
51
|
"scripts": {
|
|
52
52
|
"build": "tsc --project tsconfig.build.json",
|
|
53
53
|
"prepublishOnly": "npm run release:check",
|
|
54
|
-
"release:check": "npm run verify:ci && npm run release:consumer-pack && npm run release:functional-smoke",
|
|
54
|
+
"release:check": "npm run release:refresh-benchmarks && npm run verify:ci && npm run release:consumer-pack && npm run release:functional-smoke",
|
|
55
|
+
"release:refresh-benchmarks": "tsx scripts/release-refresh-benchmark-profiles.ts",
|
|
55
56
|
"release:consumer-pack": "bash scripts/verify-consumer-pack.sh",
|
|
56
57
|
"release:functional-smoke": "npm run routing:verify-calibration -- --skip-embed && npm run routing:verify-benchmark-profiles && tsx scripts/eval/assert-release-gates.ts --fixtures tests/eval/fixtures --baseline-version 0.6.0",
|
|
57
58
|
"typecheck": "tsc --noEmit",
|