auto-model-router 0.2.32 → 0.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.omp-plugin/marketplace.json +2 -2
- package/README.md +225 -29
- package/docs/review-2026-09-05.md +267 -0
- package/omp-extension/configure-logic.ts +71 -15
- package/omp-extension/pi-coding-agent.d.ts +79 -2
- package/omp-extension/report-hub.ts +376 -0
- package/omp-extension/report-logic.ts +117 -0
- package/omp-extension/router-configure.ts +203 -51
- package/omp-extension/router-url.ts +52 -0
- package/omp-extension/toast-logic.ts +14 -2
- package/package.json +1 -1
- package/src/catalog/composite.ts +97 -0
- package/src/catalog/ollama-catalog.ts +309 -0
- package/src/catalog/ollama-prices.ts +85 -0
- package/src/catalog/openrouter-catalog.ts +39 -1
- package/src/catalog/types.ts +31 -1
- package/src/cli/args.ts +1 -0
- package/src/cli/config-wizard.ts +190 -28
- package/src/cli/explain.ts +2 -4
- package/src/cli/models.ts +2 -4
- package/src/cli/report.ts +37 -0
- package/src/config/defaults.ts +46 -2
- package/src/config/load.ts +25 -1
- package/src/config/omp-credentials.ts +31 -7
- package/src/config/schema.ts +28 -0
- package/src/config/types.ts +120 -2
- package/src/cost/cache-estimate.ts +52 -0
- package/src/cost/ledger.ts +73 -4
- package/src/cost/report.ts +351 -0
- package/src/cost/types.ts +39 -1
- package/src/index.ts +5 -8
- package/src/router/candidates.ts +52 -4
- package/src/router/classify.ts +33 -6
- package/src/router/features.ts +13 -1
- package/src/router/select.ts +55 -8
- package/src/router/state.ts +6 -2
- package/src/router/tier-plan.ts +49 -11
- package/src/router/types.ts +10 -0
- package/src/server/http.ts +50 -6
- package/src/server/providers.ts +54 -0
- package/src/server/turn.ts +138 -34
- package/src/tokens/estimate.ts +16 -0
- package/src/upstream/multi.ts +26 -0
- package/src/upstream/ollama-usage.ts +163 -0
- package/src/upstream/ollama.ts +275 -0
- package/src/upstream/openrouter.ts +19 -1
- package/src/upstream/types.ts +2 -0
- package/src/util/sqlite.ts +25 -1
- package/test/cache-estimate.test.ts +48 -0
- package/test/catalog.test.ts +44 -0
- package/test/classify.test.ts +41 -5
- package/test/compaction.test.ts +1 -0
- package/test/config-wizard.test.ts +77 -1
- package/test/configure-logic.test.ts +129 -33
- package/test/embed-lifecycle.test.ts +1 -0
- package/test/failover.test.ts +148 -3
- package/test/features.test.ts +35 -0
- package/test/http-resilience.test.ts +24 -0
- package/test/ollama.test.ts +521 -0
- package/test/omp-credentials.test.ts +43 -1
- package/test/report-hub.test.ts +343 -0
- package/test/report-logic.test.ts +93 -0
- package/test/report.test.ts +233 -0
- package/test/select.test.ts +151 -1
- package/test/tier-plan.test.ts +159 -1
- package/test/toast-logic.test.ts +11 -2
- package/test/tokens.test.ts +71 -1
- package/test/trust-attribution.test.ts +2 -2
- package/test/turn.test.ts +173 -7
- package/tools/recompute-ollama-cache.ts +129 -0
|
@@ -7,14 +7,14 @@
|
|
|
7
7
|
},
|
|
8
8
|
"metadata": {
|
|
9
9
|
"description": "auto-model-router: a local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter",
|
|
10
|
-
"version": "0.
|
|
10
|
+
"version": "0.3.1",
|
|
11
11
|
"pluginRoot": "."
|
|
12
12
|
},
|
|
13
13
|
"plugins": [
|
|
14
14
|
{
|
|
15
15
|
"name": "auto-model-router",
|
|
16
16
|
"description": "Local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter. Runs in-process, routes per turn by price and task complexity, with budget caps, mid-stream escalation, and cache-aware hysteresis.",
|
|
17
|
-
"version": "0.
|
|
17
|
+
"version": "0.3.1",
|
|
18
18
|
"author": {
|
|
19
19
|
"name": "drewappling",
|
|
20
20
|
"email": "drewappling@gmail.com"
|
package/README.md
CHANGED
|
@@ -3,10 +3,13 @@
|
|
|
3
3
|
**[Website & benchmarks →](https://drewappling.github.io/auto-model-router/)**
|
|
4
4
|
|
|
5
5
|
A local model router for [Oh My Pi](https://github.com/oh-my-pi). It presents
|
|
6
|
-
itself as one keyless OpenAI-compatible provider, then picks a concrete
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
6
|
+
itself as one keyless OpenAI-compatible provider, then picks a concrete model
|
|
7
|
+
**per turn** — from OpenRouter's catalog, and from
|
|
8
|
+
[Ollama Cloud](#ollama-cloud) when that is enabled too — based on measured
|
|
9
|
+
price and estimated task complexity, including mid-conversation, when a
|
|
10
|
+
session shifts from mechanical tool-loop churn to genuine reasoning work.
|
|
11
|
+
With both providers on, every turn ranks the candidates of both together and
|
|
12
|
+
fails over across them.
|
|
10
13
|
|
|
11
14
|
auto-model-router runs **embedded inside the omp process** (as an omp extension) — no
|
|
12
15
|
separate server, no orphaned process. It binds a free OS-assigned port and
|
|
@@ -145,12 +148,17 @@ Harness, tasks and raw per-turn data:
|
|
|
145
148
|
graph LR
|
|
146
149
|
omp[omp process] -->|OpenAI chat completions| wire[wire/openai]
|
|
147
150
|
wire -->|NormRequest| router[router]
|
|
148
|
-
|
|
151
|
+
orcat[OpenRouter /models] --> catalog[catalog<br/>one merged snapshot]
|
|
152
|
+
olcat[Ollama /api/tags + prices<br/>optional] --> catalog
|
|
153
|
+
catalog --> router
|
|
149
154
|
cost[cost<br/>forecast + ledger] --> router
|
|
150
155
|
router -->|Decision| guard[escalation guard]
|
|
151
|
-
guard -->|rendered body| up[upstream/
|
|
152
|
-
up
|
|
153
|
-
|
|
156
|
+
guard -->|rendered body| up[upstream/multi<br/>by slug prefix]
|
|
157
|
+
up --> or[openrouter]
|
|
158
|
+
up --> ol[ollama<br/>ollama/… slugs]
|
|
159
|
+
or -->|UpstreamChunk| guard
|
|
160
|
+
ol -->|UpstreamChunk| guard
|
|
161
|
+
guard -->|commit, fail over, or retry upward| wire
|
|
154
162
|
guard -->|usage + reported cost| cost
|
|
155
163
|
```
|
|
156
164
|
|
|
@@ -163,15 +171,15 @@ without touching routing.
|
|
|
163
171
|
|
|
164
172
|
| Path | Responsibility |
|
|
165
173
|
| --- | --- |
|
|
166
|
-
| `src/catalog/` | Fetch and normalize OpenRouter `/api/v1/models`: pricing, capability flags, Artificial Analysis quality indices. SQLite-cached with TTL. |
|
|
167
|
-
| `src/cost/` | Cost forecasting per candidate; reconciliation against OpenRouter's authoritative `usage.cost`; the spend ledger; per-model trust; rolling blended rate. |
|
|
174
|
+
| `src/catalog/` | Fetch and normalize OpenRouter `/api/v1/models`: pricing, capability flags, Artificial Analysis quality indices. SQLite-cached with TTL. `ollama-catalog.ts` builds Ollama Cloud models from `/api/tags`, a shipped price table and OpenRouter twins; `composite.ts` merges the two into one snapshot. |
|
|
175
|
+
| `src/cost/` | Cost forecasting per candidate; reconciliation against OpenRouter's authoritative `usage.cost`; the spend ledger; per-model trust; rolling blended rate; `report.ts` usage analytics. |
|
|
168
176
|
| `src/tokens/` | Token estimation with no tokenizer dependency, self-calibrating from observed `prompt_tokens` per tokenizer family. |
|
|
169
177
|
| `src/wire/` | Protocol boundary. `wire/openai/` implements chat completions in and SSE out. |
|
|
170
178
|
| `src/router/` | Feature extraction, complexity classification, candidate filtering and scoring, hysteresis, cache-breakpoint placement, budget guard, probe planning. |
|
|
171
|
-
| `src/upstream/` | OpenRouter
|
|
179
|
+
| `src/upstream/` | Transports: OpenRouter (streaming dispatch, `session_id` stickiness, error classification, fallback arrays) and Ollama Cloud (body rewrite for its compatibility layer, quota/rate-limit breaker); `multi.ts` dispatches by slug prefix. |
|
|
172
180
|
| `src/config/` | Configuration loading, schema validation, and the built-in defaults. |
|
|
173
|
-
| `src/cli/` | `serve`, `stats`, `models`, `explain`, `config` commands. |
|
|
174
|
-
| `omp-extension/` | The omp extensions: `router-embed.ts`, `router-toast.ts`, `router-configure.ts
|
|
181
|
+
| `src/cli/` | `serve`, `stats`, `report`, `models`, `explain`, `config` commands. |
|
|
182
|
+
| `omp-extension/` | The omp extensions: `router-embed.ts`, `router-toast.ts`, `router-configure.ts` (`/router` config, report, status). |
|
|
175
183
|
|
|
176
184
|
### Two cost numbers, never conflated
|
|
177
185
|
|
|
@@ -204,7 +212,7 @@ Then add the shipped extensions to omp's `~/.omp/agent/config.yml`
|
|
|
204
212
|
extensions:
|
|
205
213
|
- auto-model-router/omp-extension/router-embed.ts
|
|
206
214
|
- auto-model-router/omp-extension/router-toast.ts # optional: chosen-model toasts
|
|
207
|
-
- auto-model-router/omp-extension/router-configure.ts # optional: /router
|
|
215
|
+
- auto-model-router/omp-extension/router-configure.ts # optional: /router config, report, status
|
|
208
216
|
```
|
|
209
217
|
|
|
210
218
|
### From the repo (cross-platform installer)
|
|
@@ -227,7 +235,7 @@ The installer adds:
|
|
|
227
235
|
|
|
228
236
|
- `router-embed.ts` — **required**; runs the router in-process.
|
|
229
237
|
- `router-toast.ts` — optional; chosen-model toasts.
|
|
230
|
-
- `router-configure.ts` — optional; the `/router` command.
|
|
238
|
+
- `router-configure.ts` — optional; the `/router` command (configure, usage reports, status).
|
|
231
239
|
|
|
232
240
|
Or add the paths by hand to omp's `~/.omp/agent/config.yml`:
|
|
233
241
|
|
|
@@ -236,7 +244,7 @@ Or add the paths by hand to omp's `~/.omp/agent/config.yml`:
|
|
|
236
244
|
extensions:
|
|
237
245
|
- /path/to/auto-model-router/omp-extension/router-embed.ts
|
|
238
246
|
- /path/to/auto-model-router/omp-extension/router-toast.ts # optional: chosen-model toasts
|
|
239
|
-
- /path/to/auto-model-router/omp-extension/router-configure.ts # optional: /router
|
|
247
|
+
- /path/to/auto-model-router/omp-extension/router-configure.ts # optional: /router config, report, status
|
|
240
248
|
```
|
|
241
249
|
|
|
242
250
|
Then restart the omp session (extensions load at session start).
|
|
@@ -405,7 +413,11 @@ Your OpenRouter guardrails — model and provider allowlists, budget limits,
|
|
|
405
413
|
Zero-Data-Retention and privacy rules — are therefore the router's outer
|
|
406
414
|
boundary: a model your key cannot reach is never a routing candidate. The
|
|
407
415
|
catalog is refetched in the background every `catalogRefreshMs` (default 5 min),
|
|
408
|
-
so tightening or relaxing a guardrail is picked up without a restart.
|
|
416
|
+
so tightening or relaxing a guardrail is picked up without a restart. A refresh
|
|
417
|
+
that keeps fewer than half the previous models is adopted (your guardrails are
|
|
418
|
+
authoritative) but logged at `warn` and reported as `catalog.shrink` on
|
|
419
|
+
`GET /health` until the catalog recovers, because a sharp shrink reroutes every
|
|
420
|
+
turn onto whatever survived. If a
|
|
409
421
|
guardrail narrows the eligible set below a tier's quality floor,
|
|
410
422
|
`adaptiveTierFloors` (on by default) relaxes that tier to the best available
|
|
411
423
|
models rather than leaving it empty — see [Adaptive tier floors](#adaptive-tier-floors)
|
|
@@ -465,6 +477,42 @@ virtual profile it picked. Every routed response carries
|
|
|
465
477
|
|
|
466
478
|
---
|
|
467
479
|
|
|
480
|
+
## Usage reports
|
|
481
|
+
|
|
482
|
+
The ledger records every dispatch: model decided and served, tier, provider,
|
|
483
|
+
tokens (including cached), reported cost, time to first token, total latency,
|
|
484
|
+
escalation signal, error. Three views aggregate it, all from the same
|
|
485
|
+
`buildUsageReport` in `src/cost/report.ts`:
|
|
486
|
+
|
|
487
|
+
- `/router report` in omp — a fullscreen hub with the `/models` look: views
|
|
488
|
+
for overview, providers, models, tiers, by day and status in a sidebar, plus
|
|
489
|
+
a Window selector (24h / 7d / 30d / 90d) and, when `OMP_HARNESS_ID` is set,
|
|
490
|
+
a scope toggle between this harness and all harnesses. ↑/↓ move, Enter
|
|
491
|
+
applies a window or scope, ←/→ also cycle the window, PgUp/PgDn scroll, r
|
|
492
|
+
reloads, Esc closes. Headless sessions get the same report as text in the
|
|
493
|
+
transcript. Falls back to reading the ledger directly if the router is
|
|
494
|
+
unreachable.
|
|
495
|
+
- `auto-model-router report --days 7 [--harness <id>] [--json]` on the terminal.
|
|
496
|
+
- `GET /v1/router/report?days=7&harness=<id>` for dashboards.
|
|
497
|
+
|
|
498
|
+
What it shows, for the window:
|
|
499
|
+
|
|
500
|
+
| Block | Columns |
|
|
501
|
+
| --- | --- |
|
|
502
|
+
| totals | spend, dispatches, conversations, $/dispatch, prompt and completion tokens, cache hit rate, model switches, escalations, failovers, errors (aborted separately) |
|
|
503
|
+
| providers | per upstream (`openrouter`, `ollama`): dispatches, spend, share, cache hit, mean TTFT, tokens/s, escalations, errors |
|
|
504
|
+
| models | per served slug (top 12 by spend): the same plus the tier mix it was routed for |
|
|
505
|
+
| tiers | per tier: dispatches, spend, share, cache hit, mean prompt tokens, escalations |
|
|
506
|
+
| by day | UTC calendar days: dispatches, spend, cache hit |
|
|
507
|
+
|
|
508
|
+
Spend follows the ledger's rule — the provider's reported cost when it gave
|
|
509
|
+
one, else the usage-priced figure the router computed, else the forecast.
|
|
510
|
+
Speed uses only clean streamed rows (TTFT recorded, no error); tokens/s is
|
|
511
|
+
completion tokens over time after first token. Ollama Cloud caches prompt
|
|
512
|
+
prefixes and bills them at its cached rate but reports no count, so the
|
|
513
|
+
router estimates it (see [Ollama Cloud](#ollama-cloud)); cache rates that
|
|
514
|
+
include such rows are shown with a `~`.
|
|
515
|
+
|
|
468
516
|
## Configuring the router
|
|
469
517
|
|
|
470
518
|
The router's own config lives at `$AUTO_MODEL_ROUTER_HOME/config.yml` (default
|
|
@@ -474,12 +522,23 @@ built-in defaults below. There are two ways to edit it:
|
|
|
474
522
|
### Via `/router` (in-omp, native UI)
|
|
475
523
|
|
|
476
524
|
Install the `router-configure` extension, restart omp, then run `/router` in
|
|
477
|
-
the session prompt.
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
525
|
+
the session prompt. With no arguments it shows a menu (Configure, Report,
|
|
526
|
+
Status); the subcommands go straight there:
|
|
527
|
+
|
|
528
|
+
| Command | What it does |
|
|
529
|
+
| --- | --- |
|
|
530
|
+
| `/router config` | Section picker over **every** config key: Server, OpenRouter, Ollama Cloud, Benchmarks, Tiers, Tasks, Filters, Classifier, Escalation, Hysteresis, Exploration, Cache, Compaction, Context (agentdox), Budget, Ledger, Logging, Profiles. Only `ollama.prices` and `ollama.twins` (maps) stay YAML-only. |
|
|
531
|
+
| `/router report` | Usage analytics in a fullscreen hub styled like `/models`: pick a view in the sidebar, set the window (24h / 7d / 30d / 90d) and the harness scope there too. `/router report 30d --all` presets them. See [Usage reports](#usage-reports). |
|
|
532
|
+
| `/router status` | The router's `/health`: key sources, catalog size and age, Ollama availability, plan usage and cost bias, agentdox bridge. |
|
|
533
|
+
|
|
534
|
+
Picking a section lists its fields with their current values (pending edits
|
|
535
|
+
marked), so you see the settings before choosing one to change. Each field
|
|
536
|
+
dialog names the current value in its title, marks it in pickers and uses it
|
|
537
|
+
as the placeholder — empty input keeps it, `-` clears an optional field,
|
|
538
|
+
credentials show as `set`/`unset` and are never echoed. `Save and exit` writes the merged config
|
|
539
|
+
(schema-checked and backed up first). Tier, task, filter, classifier,
|
|
540
|
+
hysteresis, exploration, compaction, cache and budget changes hot-reload;
|
|
541
|
+
restart omp for `server`, `openrouter`, `ollama`, `context` and `ledger`.
|
|
483
542
|
|
|
484
543
|
### Via `auto-model-router config` (text wizard / CLI)
|
|
485
544
|
|
|
@@ -541,6 +600,31 @@ what each one does. All values are optional; omit a key to use its default.
|
|
|
541
600
|
| `catalogTtlMs` | `21600000` (6 h) | How long the model catalog is cached before a forced refetch. |
|
|
542
601
|
| `catalogRefreshMs` | `300000` (5 min) | Background catalog refetch interval; `0` disables it. |
|
|
543
602
|
|
|
603
|
+
### `ollama` — Ollama Cloud as a second upstream
|
|
604
|
+
|
|
605
|
+
Off by default. When enabled, Ollama Cloud models join the same catalog as
|
|
606
|
+
OpenRouter's under `ollama/<id>` slugs and are ranked on the same economics:
|
|
607
|
+
a turn picks whichever provider's model is cheapest above the tier's floor,
|
|
608
|
+
and same-tier failover crosses providers (a 402 or 429 from Ollama retries on
|
|
609
|
+
an OpenRouter sibling). See [Ollama Cloud](#ollama-cloud) below.
|
|
610
|
+
|
|
611
|
+
| Key | Default | Meaning |
|
|
612
|
+
| --- | --- | --- |
|
|
613
|
+
| `enabled` | `false` | Master switch. |
|
|
614
|
+
| `baseUrl` | `http://127.0.0.1:11434/v1` | A local daemon (proxies `:cloud` models under its sign-in) or `https://ollama.com/v1`. |
|
|
615
|
+
| `apiKey` | unset | Bearer for ollama.com. Resolved from config, then `OLLAMA_API_KEY`, then omp's own auth store (`/login ollama-cloud` in omp) — the same borrowing as the OpenRouter key. The daemon needs none. |
|
|
616
|
+
| `timeoutMs` | `600000` | Per-request timeout. |
|
|
617
|
+
| `catalogTtlMs` | `300000` | Re-list models when the last listing is older than this. |
|
|
618
|
+
| `includeLocal` | `false` | Also expose the daemon's local models (only those named in `prices`). |
|
|
619
|
+
| `prices` | `{}` | USD per million tokens by bare cloud name (`{input, cachedInput?, output}`); overrides or extends the shipped snapshot. |
|
|
620
|
+
| `twins` | `{}` | Bare cloud name → OpenRouter slug, to pin a quality-score twin the name match misses. |
|
|
621
|
+
| `costBias` | `1` | Multiplier on Ollama models' effective cost in ranking; below 1 prefers Ollama. The ledger still records list price. |
|
|
622
|
+
| `biasUntilUsage` | `0.9` | Share of the plan's included monthly credits at which `costBias` switches off and Ollama ranks at list price. Read live from ollama.com's `/api/usage`, which reports usage relative to the plan, so the same value is right on Pro, Max or Team. `1` keeps the bias regardless. |
|
|
623
|
+
| `usagePollMs` | `600000` (10 min) | How often plan usage is re-read. `0` disables it (static bias). Needs the API key; the daemon path without one keeps a static bias. |
|
|
624
|
+
| `quotaCooldownMs` | `900000` | Route around Ollama this long after a 402 (credits exhausted). |
|
|
625
|
+
| `rateLimitCooldownMs` | `60000` | Route around Ollama this long after a 429 (concurrency cap). |
|
|
626
|
+
| `planCreditsUsd` | `0` | Dollar value of the plan's included monthly credits (Pro 60, Max 300). Lets `/health` and `/router status` show ollama.com's plan reading as dollars next to the ledger's figure. `0` shows the share only. |
|
|
627
|
+
|
|
544
628
|
### `tiers` — per-tier economic envelope
|
|
545
629
|
|
|
546
630
|
Each tier (`trivial`, `simple`, `moderate`, `hard`) is a `tierConfig`:
|
|
@@ -580,6 +664,7 @@ Each task (`coding`, `vision`, `documentation`, `data`, `chat`) is a
|
|
|
580
664
|
| `latencyWeight` | `0` | How hard to penalise slow models in scoring (soft multiplier on effective cost). `0` disables it. |
|
|
581
665
|
| `latencyMinSamples` | `20` | Streamed samples before latency is judged against a model. |
|
|
582
666
|
| `maxExpectedWaitMs` | unset | Absolute expected-wait ceiling (ms): a hard drop for models *proven* slower (≥ `latencyMinSamples`), regardless of price. The soft penalty is multiplicative and capped, so it cannot demote a slow-but-cheap model — this can. New models keep their cold-start turns; relaxed with trust in tier rescue. Undefined ⇒ off. |
|
|
667
|
+
| `escalationCostWeight` | `0` | Price a model's measured escalation rate at what an escalated retry actually bills (the ledger's $/prompt-token of `attempt > 0` rows), 0–1. The trust divisor reads a 4% escalation rate as a 4% surcharge; the real cost is a whole re-dispatch on the next tier's model. `0` disables the term. |
|
|
583
668
|
|
|
584
669
|
### `classifier` — complexity adjudication
|
|
585
670
|
|
|
@@ -594,6 +679,7 @@ Each task (`coding`, `vision`, `documentation`, `data`, `chat`) is a
|
|
|
594
679
|
| `toolAxis` | `coding` | Quality axis for tool-heavy turns. |
|
|
595
680
|
| `chatAxis` | `intelligence` | Quality axis for chat turns. |
|
|
596
681
|
| `agenticLoopDepth` | `3` | Tool-loop depth at which a turn is treated as agentic. |
|
|
682
|
+
| `mechanicalRetryFactor` | `0.2` | Fraction of the failed-tool and circular-call weights kept on a tool-result continuation; `1` disables the damping. |
|
|
597
683
|
|
|
598
684
|
### `escalation` — mid-stream retry upward
|
|
599
685
|
|
|
@@ -607,6 +693,14 @@ Each task (`coding`, `vision`, `documentation`, `data`, `chat`) is a
|
|
|
607
693
|
| `triggers` | 5 signals | `malformed_tool_args`, `refusal`, `empty_completion`, `repeat_tool_call`, `missing_expected_tool_call`. |
|
|
608
694
|
| `escalateOnLengthStop` | `true` | Escalate on a `length` finish that truncated tool-call args. |
|
|
609
695
|
|
|
696
|
+
The model that produced the rejected output never serves the retry, at this
|
|
697
|
+
tier or the next. Signals that indict the *provider* rather than the tier —
|
|
698
|
+
`empty_completion`, `refusal`, and an error finish — first try a different
|
|
699
|
+
model in the **same** tier (bounded, like a 5xx failover) and only then step
|
|
700
|
+
up; structural signals (`malformed_tool_args`, `repeat_tool_call`, a truncated
|
|
701
|
+
tool call) escalate a tier directly. A client that hangs up after the finish
|
|
702
|
+
event has already arrived is treated as a completed turn, not an error.
|
|
703
|
+
|
|
610
704
|
### `hysteresis` — cache-aware model stickiness
|
|
611
705
|
|
|
612
706
|
| Key | Default | Meaning |
|
|
@@ -614,8 +708,30 @@ Each task (`coding`, `vision`, `documentation`, `data`, `chat`) is a
|
|
|
614
708
|
| `holdTurns` | `2` | Hold a chosen model this many turns before it can downgrade. |
|
|
615
709
|
| `holdTurnsAfterEscalation` | `4` | Hold longer after an escalation. |
|
|
616
710
|
| `switchMargin` | `1.3` | Switching must beat the warm-cache discount by this factor. Lower = switch away from a warm model more readily. |
|
|
711
|
+
| `switchHorizonTurns` | `1` | Turns the stay/switch comparison is amortised over: `H × stayWarm` vs `switchCold + (H − 1) × newWarm`. `1` is the one-turn comparison, which can keep a dear model warm indefinitely when the cheaper winner is itself dear cold; a small `H` lets a switch that pays for itself within a few turns go ahead. |
|
|
617
712
|
| `cacheWarmTtlMs` | `300000` (5 min) | How long a model's prompt cache is considered warm. |
|
|
618
713
|
| `maxDowngradePerTurn` | `1` | Max tiers a turn may drop in one step (avoids quality cliffs). |
|
|
714
|
+
| `breakHoldOnMechanical` | `false` | Let a tool-result continuation that classifies *below* the held tier escape the hold (still bounded by `maxDowngradePerTurn`). Worth enabling when the held tier is expensive. |
|
|
715
|
+
|
|
716
|
+
### `compaction` — shrink stale tool output before dispatch
|
|
717
|
+
|
|
718
|
+
Off by default; see `docs/context-optimization.md`. Every edit shrinks one
|
|
719
|
+
tool-result's content in place behind a re-run breadcrumb, never removes or
|
|
720
|
+
reorders a message, and the plan is persisted per conversation so already
|
|
721
|
+
shrunk results stay shrunk (rewriting them would break the prompt cache).
|
|
722
|
+
|
|
723
|
+
| Key | Default | Meaning |
|
|
724
|
+
| --- | --- | --- |
|
|
725
|
+
| `enabled` | `false` | Master switch. |
|
|
726
|
+
| `budgetTokens` | `40000` | Compact when the (already compacted) prompt exceeds this many tokens. |
|
|
727
|
+
| `floorRatio` | `1` | Once compaction fires, compact down to this fraction of the budget so the plan holds for several turns. |
|
|
728
|
+
| `replanGrowthRatio` | `1` | Above 1, only extend an existing plan once the compacted prompt has grown by this factor since the plan was made. Rations plan churn when the budget is unreachable (every turn over budget); fit-to-window is never rationed. |
|
|
729
|
+
| `fitToWindow` | `true` | Also compact when the prompt would overflow the profile's context window. |
|
|
730
|
+
| `protectRecentTurns` | `4` | Never touch the last N user/assistant turns or the volatile tail. |
|
|
731
|
+
| `maxToolResultBytes` | `4096` | Tool results larger than this (outside the protected window) are truncated. |
|
|
732
|
+
| `keepHeadBytes` / `keepTailBytes` | `512` / `512` | Bytes kept around the elision breadcrumb. |
|
|
733
|
+
| `elideSupersededReads` | `true` | Stub an older result when a newer call to the same resource supersedes it. |
|
|
734
|
+
| `collapseDuplicateResults` | `true` | Collapse byte-identical repeated results to a single copy. |
|
|
619
735
|
|
|
620
736
|
### `cache` — prompt-cache breakpoints
|
|
621
737
|
|
|
@@ -661,9 +777,76 @@ Each profile is a complete entry (arrays replace wholesale):
|
|
|
661
777
|
|
|
662
778
|
| Key | Default | Meaning |
|
|
663
779
|
| --- | --- | --- |
|
|
664
|
-
| `adaptiveTierFloors` | `true` |
|
|
780
|
+
| `adaptiveTierFloors` | `true` | Relax a tier's quality floor to a catalog-derived band when fewer than three available models meet the configured floor (never raising it). A floor that three or more models meet stands as written. |
|
|
665
781
|
| `logLevel` | `info` | `silent`/`error`/`warn`/`info`/`debug`. |
|
|
666
782
|
|
|
783
|
+
## Ollama Cloud
|
|
784
|
+
|
|
785
|
+
[Ollama Cloud](https://ollama.com/cloud) hosts open models behind Ollama's own
|
|
786
|
+
OpenAI-compatible endpoint and bills them per token against a plan's monthly
|
|
787
|
+
credits. The router can treat it as a second upstream next to OpenRouter:
|
|
788
|
+
|
|
789
|
+
```yaml
|
|
790
|
+
ollama:
|
|
791
|
+
enabled: true
|
|
792
|
+
# default: the local daemon, which proxies `:cloud` models under whatever
|
|
793
|
+
# account `ollama signin` used. For ollama.com directly:
|
|
794
|
+
# baseUrl: https://ollama.com/v1
|
|
795
|
+
# apiKey: <from https://ollama.com/settings/keys, or OLLAMA_API_KEY, or
|
|
796
|
+
# borrowed from omp after `/login ollama-cloud` — no copy needed>
|
|
797
|
+
```
|
|
798
|
+
|
|
799
|
+
What happens once it is on:
|
|
800
|
+
|
|
801
|
+
- **One catalog.** Every cloud model Ollama lists becomes `ollama/<id>` (for
|
|
802
|
+
example `ollama/glm-5.3-flash:cloud` through the daemon, `ollama/glm-5.3-flash`
|
|
803
|
+
on ollama.com) with the context length and capabilities Ollama publishes
|
|
804
|
+
(`/api/tags` on the daemon, `/api/show` on ollama.com).
|
|
805
|
+
- **Prices come from a shipped table**, because no Ollama endpoint publishes
|
|
806
|
+
them: the rates on [ollama.com/pricing](https://ollama.com/pricing) as of
|
|
807
|
+
2026-09-05 (`src/catalog/ollama-prices.ts`). `ollama.prices` overrides or
|
|
808
|
+
extends it; a model with no rate from either is left out, on the same rule
|
|
809
|
+
that drops unpriced OpenRouter models.
|
|
810
|
+
- **Quality scores come from the OpenRouter twin.** Ollama publishes none, so
|
|
811
|
+
`glm-5.3-flash` inherits `z-ai/glm-5.3-flash`'s indices by name match, which
|
|
812
|
+
is what lets it serve `simple` and above. `ollama.twins` pins a match the
|
|
813
|
+
name normaliser cannot make; an unmatched model is unscored and serves only
|
|
814
|
+
`trivial`.
|
|
815
|
+
- **Cached prefixes are estimated, not reported.** ollama.com caches prompt
|
|
816
|
+
prefixes automatically and bills them at the published cached-input rate,
|
|
817
|
+
but neither its OpenAI-compatible usage nor the native API carries a cached
|
|
818
|
+
token count. Measured 2026-09-07: twelve identical 162k-token requests to
|
|
819
|
+
`glm-5.3-flash` moved the plan meter by $0.06 against $0.29 at the full
|
|
820
|
+
input rate, and repeats answered in ~1.5 s. Pricing every token fresh had
|
|
821
|
+
overstated a week of Ollama spend 3.7x ($23.01 booked, $6.24 metered). The
|
|
822
|
+
router now applies its own warm-cache rule to Ollama turns: when the same
|
|
823
|
+
model served the previous turn within `hysteresis.cacheWarmTtlMs`, the
|
|
824
|
+
previous prompt is taken as the cached prefix and priced at the cached
|
|
825
|
+
rate; a first turn, a switch, or a longer gap is priced cold. The ledger
|
|
826
|
+
flags these rows (`usage.cachedEstimated`) and reports show their cache
|
|
827
|
+
rate as `~N%`. Set `planCreditsUsd` (Pro 60, Max 300) to see ollama.com's
|
|
828
|
+
own dollar reading in `/router status` as the cross-check.
|
|
829
|
+
- **Same economics, same failover.** Candidates from both providers are ranked
|
|
830
|
+
together; `costBias` tilts the comparison while a plan's included credits
|
|
831
|
+
would otherwise go unused. **Credit-aware by default:** the router reads the
|
|
832
|
+
plan's usage from ollama.com (`/api/usage`, the same figure the dashboard
|
|
833
|
+
shows, as a share of the plan's included credits) every `usagePollMs`, and
|
|
834
|
+
once it passes `biasUntilUsage` (90%) Ollama ranks at list price for the rest
|
|
835
|
+
of the billing month. Because the figure is relative to the plan, nothing
|
|
836
|
+
about Pro, Max or Team needs configuring; `/health` shows the raw reading
|
|
837
|
+
and the multiplier in force. A 402 (credits exhausted) or 429 (concurrency cap) from
|
|
838
|
+
Ollama fails the attempt over to an OpenRouter sibling in the same tier and
|
|
839
|
+
opens a breaker, so following turns route straight to OpenRouter without
|
|
840
|
+
paying a doomed dispatch first; `/health` shows `ollama.available` and the
|
|
841
|
+
cooldown.
|
|
842
|
+
- **Ollama reports no cost per response**, so the ledger records the
|
|
843
|
+
predicted figure at list price for those rows.
|
|
844
|
+
|
|
845
|
+
Ollama's compatibility layer differs from OpenRouter's in a few ways the
|
|
846
|
+
router handles for you: no `models[]` fallback cascade, no `tool_choice`,
|
|
847
|
+
`reasoning_effort` instead of the `reasoning` object, and no `cache_control`
|
|
848
|
+
markers (they are stripped before dispatch).
|
|
849
|
+
|
|
667
850
|
## Multiple coding harnesses, one router
|
|
668
851
|
|
|
669
852
|
A single embedded router can serve several omp sessions without them stepping
|
|
@@ -785,8 +968,10 @@ come from a small omp extension that polls the router's in-process ledger:
|
|
|
785
968
|
```
|
|
786
969
|
|
|
787
970
|
It raises a TUI toast (`ctx.ui.notify`) like
|
|
788
|
-
`meta/muse-glimmer-30b [trivial] · $0.00001`
|
|
789
|
-
|
|
971
|
+
`openrouter · meta/muse-glimmer-30b [trivial] · $0.00001` or
|
|
972
|
+
`ollama · glm-5.3-flash [moderate] · $0.00070` whenever a new model is chosen —
|
|
973
|
+
provider first, so a mixed catalog is legible at a glance. Install it by adding
|
|
974
|
+
the file's absolute path to omp's `extensions:` list.
|
|
790
975
|
|
|
791
976
|
Because the embedded router binds a random port, the toast resolves the router
|
|
792
977
|
base URL on every poll in this order: the embedded router's port file
|
|
@@ -863,13 +1048,20 @@ absolute floor admits nothing and the router is trapped in the lowest tier.
|
|
|
863
1048
|
|
|
864
1049
|
With `adaptiveTierFloors: true` (the default), every catalog refresh ranks the
|
|
865
1050
|
**available** scored models and splits them into four quantile bands, taking
|
|
866
|
-
each band's lower bound as that tier's adaptive floor. The
|
|
867
|
-
|
|
1051
|
+
each band's lower bound as that tier's adaptive floor. The band applies only
|
|
1052
|
+
when the configured floor leaves the tier **thin** — fewer than three available
|
|
1053
|
+
models meet it — in which case the floor enforced is `min(configured, adaptive)`:
|
|
868
1054
|
|
|
869
|
-
- a
|
|
1055
|
+
- a floor that three or more models meet stands exactly as configured — no
|
|
1056
|
+
behaviour change on a healthy catalog;
|
|
870
1057
|
- a narrowed catalog falls back to the adaptive floor, so `hard` still gets the
|
|
871
1058
|
best quartile of what is available instead of nothing.
|
|
872
1059
|
|
|
1060
|
+
The thinness gate matters: a wide catalog carries a long tail of weak scored
|
|
1061
|
+
models, so its quantile bands sit *below* the configured floors (measured on a
|
|
1062
|
+
347-model key-admitted catalog: coding p50 = 45.8 against `moderate`'s 60), and
|
|
1063
|
+
an unconditional `min` would quietly relax every tier.
|
|
1064
|
+
|
|
873
1065
|
Relaxation is one-directional by design: an adaptive floor may only **lower** a
|
|
874
1066
|
tier floor, never raise one. Two things are deliberately exempt:
|
|
875
1067
|
|
|
@@ -934,6 +1126,10 @@ Known gaps:
|
|
|
934
1126
|
|
|
935
1127
|
- The `pi-native` front end is designed for but not implemented; only the
|
|
936
1128
|
OpenAI-compatible wire exists today.
|
|
1129
|
+
- `escalation.maxHoldMs` is only enforced when a chunk arrives: a stream that
|
|
1130
|
+
emits nothing at all holds the client until the upstream ends. Left as is on
|
|
1131
|
+
purpose — a timer would escalate every turn whose first token lands after
|
|
1132
|
+
8s, which on live traffic is most of glm-5.3-flash's.
|
|
937
1133
|
- Blended `cost` figures in `models.yml` are refreshed by re-running
|
|
938
1134
|
`auto-model-router config --write`, not automatically.
|
|
939
1135
|
|