auto-model-router 0.1.4 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. package/.omp-plugin/marketplace.json +2 -2
  2. package/README.md +43 -55
  3. package/bun.lock +606 -0
  4. package/package.json +2 -1
  5. package/research/analyze-ledger.ts +173 -0
  6. package/research/apply-cost-tuning.ts +73 -0
  7. package/research/cost-analysis.ts +150 -0
  8. package/research/feed-check.ts +64 -0
  9. package/research/model-recommendations.ts +86 -0
  10. package/research/project-yield.ts +96 -0
  11. package/research/run-eval.ts +133 -0
  12. package/research/status.ts +55 -0
  13. package/research/tier-fill.ts +109 -0
  14. package/research/tier-map.ts +123 -0
  15. package/src/catalog/benchmark-feeds.ts +397 -0
  16. package/src/catalog/openrouter-catalog.ts +30 -0
  17. package/src/config/defaults.ts +30 -0
  18. package/src/config/load.ts +2 -0
  19. package/src/config/schema.ts +34 -0
  20. package/src/config/types.ts +106 -0
  21. package/src/cost/ledger.ts +23 -2
  22. package/src/cost/types.ts +24 -0
  23. package/src/eval/calibrate.ts +131 -0
  24. package/src/eval/grade.ts +115 -0
  25. package/src/eval/judge.ts +71 -0
  26. package/src/eval/run.ts +126 -0
  27. package/src/eval/tasks.ts +272 -0
  28. package/src/router/candidates.ts +13 -6
  29. package/src/router/explore.ts +59 -0
  30. package/src/router/select.ts +54 -4
  31. package/src/router/tier-plan.ts +57 -1
  32. package/src/router/types.ts +13 -0
  33. package/src/server/turn.ts +9 -2
  34. package/src/util/sqlite.ts +68 -1
  35. package/test/benchmark-feeds.test.ts +222 -0
  36. package/test/eval.test.ts +184 -0
  37. package/test/exploration.test.ts +251 -0
  38. package/test/failover.test.ts +4 -0
  39. package/test/hold-exploration.test.ts +124 -0
  40. package/test/tier-plan.test.ts +55 -1
  41. package/test/tokens.test.ts +7 -0
  42. package/test/trust-attribution.test.ts +96 -2
  43. package/test/turn.test.ts +45 -0
  44. package/tools/smoke.ts +2 -0
  45. package/tools/sync-marketplace-version.ts +60 -0
@@ -7,14 +7,14 @@
7
7
  },
8
8
  "metadata": {
9
9
  "description": "auto-model-router: a local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter",
10
- "version": "0.1.0",
10
+ "version": "0.2.0",
11
11
  "pluginRoot": "."
12
12
  },
13
13
  "plugins": [
14
14
  {
15
15
  "name": "auto-model-router",
16
16
  "description": "Local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter. Runs in-process, routes per turn by price and task complexity, with budget caps, mid-stream escalation, and cache-aware hysteresis.",
17
- "version": "0.1.0",
17
+ "version": "0.2.0",
18
18
  "author": {
19
19
  "name": "drewappling",
20
20
  "email": "drewappling@gmail.com"
package/README.md CHANGED
@@ -74,9 +74,10 @@ without touching routing.
74
74
  - **Reported** — `usage.cost` from OpenRouter, authoritative after the fact.
75
75
  Drives the ledger, `stats`, and prediction-error calibration.
76
76
 
77
- it. No separate Bun install is needed for the embedded path. The standalone
78
- `serve` binary (`npm install -g auto-model-router`) bundles Bun.
79
- it. No separate Bun install is needed for the embedded path.
77
+ ## Installing
78
+
79
+ No separate Bun install is needed for the embedded path. The standalone
80
+ `serve` binary (`npm install -g auto-model-router`) bundles Bun.
80
81
 
81
82
  Two ways to get the router into omp. The **npm package** is the modern path —
82
83
  it installs the `auto-model-router` binary and wires the omp extensions; the
@@ -137,45 +138,6 @@ or install it from the marketplace (see below). The plugin declares all three
137
138
  extensions (`router-embed`, `router-toast`, `router-configure`), so installing
138
139
  it wires the router in without editing `config.yml` by hand.
139
140
 
140
- There is nothing to install system-wide. Run the cross-platform installer
141
- (Windows, macOS, Linux) from the repo:
142
-
143
- ```bash
144
- bun tools/install.ts
145
- ```
146
-
147
- It wires the auto-model-router extensions into omp's `~/.omp/agent/config.yml`
148
- (`$PI_CODING_AGENT_DIR/config.yml` when that env var relocates the agent dir),
149
- backing up the previous file first. It is idempotent — re-running is a no-op.
150
-
151
- Options:
152
-
153
- ```bash
154
- bun tools/install.ts --no-toast --no-configure # only the required embed extension
155
- ```
156
-
157
- The installer adds:
158
-
159
- - `router-embed.ts` — **required**; runs the router in-process.
160
- - `router-toast.ts` — optional; chosen-model toasts.
161
- - `router-configure.ts` — optional; the `/router` command.
162
-
163
- Or add the paths by hand to omp's `~/.omp/agent/config.yml`:
164
-
165
- ```yaml
166
- # ~/.omp/agent/config.yml
167
- extensions:
168
- - /path/to/auto-model-router/omp-extension/router-embed.ts
169
- - /path/to/auto-model-router/omp-extension/router-toast.ts # optional: chosen-model toasts
170
- - /path/to/auto-model-router/omp-extension/router-configure.ts # optional: /router command
171
- ```
172
-
173
- Then restart the omp session (extensions load at session start).
174
-
175
- or install it from the marketplace (see below). The plugin declares all three
176
- extensions (`router-embed`, `router-toast`, `router-configure`), so installing
177
- it wires the router in without editing `config.yml` by hand.
178
-
179
141
  ### Install from the marketplace
180
142
 
181
143
  This repo doubles as its own marketplace: it ships a catalog at
@@ -194,6 +156,9 @@ or in the TUI:
194
156
  /marketplace install auto-model-router@auto-model-router
195
157
  ```
196
158
 
159
+ After installing, restart the omp session (extensions load at session start),
160
+ then `/model` and pick `auto-model-router/auto`.
161
+
197
162
  ### Install from the Pi package marketplace
198
163
 
199
164
  The repo is also a Pi package (see the `pi` manifest and `pi-package` keyword
@@ -210,12 +175,23 @@ or from git:
210
175
  pi install git:github.com/drewappling/auto-model-router
211
176
  ```
212
177
 
213
- To publish to npm (which auto-indexes on pi.dev/packages):
178
+ #### Releasing
179
+
180
+ Cut releases with `npm version` (or `bun run release <patch|minor|major>`), not a
181
+ bare `npm publish`:
214
182
 
215
183
  ```bash
216
- npm publish
184
+ npm version patch && git push --follow-tags # or: bun run release patch
217
185
  ```
218
186
 
187
+ `npm version` runs the `version` lifecycle script
188
+ (`tools/sync-marketplace-version.ts`), which rewrites the Git-marketplace
189
+ catalog (`.omp-plugin/marketplace.json`) to the new version and stages it into
190
+ the version commit — so the npm package and the marketplace catalog can never
191
+ drift. Pushing the `vX.Y.Z` tag triggers the release workflow (npm publish,
192
+ which auto-indexes on pi.dev/packages, plus a GitHub Release). A bare
193
+ `npm publish` skips both the catalog sync and the tag, so avoid it.
194
+
219
195
  ### Hermes
220
196
 
221
197
  Install the router globally (puts the `serve` binary on PATH) and
@@ -279,17 +255,29 @@ providers:
279
255
 
280
256
  ### The OpenRouter key
281
257
 
282
- There should be exactly one OpenRouter key on the machine, and omp already owns
283
- a credential store. Resolution order:
284
-
285
- 1. `openrouter.apiKey` in `$AUTO_MODEL_ROUTER_HOME/config.yml`
286
- 2. `OPENROUTER_API_KEY` (including any `.env` omp loaded into the environment)
287
- 3. **omp's own auth store** — `~/.omp/agent/agent.db`, provider `openrouter`
288
-
289
- So `/login openrouter` inside omp is sufficient setup; nothing needs copying.
290
- The store is opened read-only and never written: omp owns it, including OAuth
291
- refresh. An expired OAuth access token is rejected rather than sent, because
292
- refreshing is omp's job and a stale bearer just burns a turn on a 401. Under
258
+ **omp does not need to be authenticated to OpenRouter.** On a routed turn omp
259
+ never calls OpenRouter directly: the embed extension registers the
260
+ `auto-model-router` provider with a placeholder bearer (`embedded`) pointing at
261
+ the in-process router, and the router holds the real OpenRouter key and makes
262
+ the upstream call. omp only needs to see that the provider "has credentials",
263
+ which the placeholder satisfies.
264
+
265
+ There should be exactly one OpenRouter key on the machine. The router resolves
266
+ it in this order:
267
+
268
+ 1. `openrouter.apiKey` in `$AUTO_MODEL_ROUTER_HOME/config.yml` router-owned,
269
+ never enters omp's environment. Set it with `auto-model-router config` or by
270
+ hand.
271
+ 2. `OPENROUTER_API_KEY` in the environment omp launches from (including any
272
+ `.env` omp loaded).
273
+ 3. **omp's own auth store** — `~/.omp/agent/agent.db`, provider `openrouter`, so
274
+ `/login openrouter` inside omp is sufficient and nothing needs copying.
275
+
276
+ Options 1–2 give the router its own key with omp left unauthenticated; option 3
277
+ is a zero-config convenience for when you *have* logged omp in. The store is
278
+ opened read-only and never written: omp owns it, including OAuth refresh. An
279
+ expired OAuth access token is rejected rather than sent, because refreshing is
280
+ omp's job and a stale bearer just burns a turn on a 401. Under
293
281
  `OMP_AUTH_BROKER_URL` the local store is not consulted at all, since a broker
294
282
  replaces it.
295
283