auto-model-router 0.2.31 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.omp-plugin/marketplace.json +2 -2
- package/README.md +211 -29
- package/docs/review-2026-09-05.md +267 -0
- package/omp-extension/configure-logic.ts +71 -15
- package/omp-extension/pi-coding-agent.d.ts +79 -2
- package/omp-extension/report-hub.ts +376 -0
- package/omp-extension/report-logic.ts +115 -0
- package/omp-extension/router-configure.ts +203 -51
- package/omp-extension/router-url.ts +52 -0
- package/omp-extension/toast-logic.ts +14 -2
- package/package.json +1 -1
- package/src/catalog/composite.ts +97 -0
- package/src/catalog/ollama-catalog.ts +309 -0
- package/src/catalog/ollama-prices.ts +85 -0
- package/src/catalog/openrouter-catalog.ts +39 -1
- package/src/catalog/types.ts +31 -1
- package/src/cli/args.ts +1 -0
- package/src/cli/config-wizard.ts +189 -28
- package/src/cli/explain.ts +2 -4
- package/src/cli/models.ts +2 -4
- package/src/cli/report.ts +37 -0
- package/src/config/defaults.ts +43 -2
- package/src/config/load.ts +25 -1
- package/src/config/omp-credentials.ts +31 -7
- package/src/config/schema.ts +28 -0
- package/src/config/types.ts +127 -2
- package/src/cost/ledger.ts +73 -4
- package/src/cost/report.ts +340 -0
- package/src/cost/types.ts +33 -1
- package/src/index.ts +5 -8
- package/src/router/candidates.ts +90 -11
- package/src/router/classify.ts +33 -6
- package/src/router/features.ts +13 -1
- package/src/router/select.ts +55 -8
- package/src/router/state.ts +6 -2
- package/src/router/tier-plan.ts +49 -11
- package/src/router/types.ts +11 -0
- package/src/server/http.ts +47 -6
- package/src/server/providers.ts +54 -0
- package/src/server/turn.ts +122 -34
- package/src/tokens/estimate.ts +16 -0
- package/src/upstream/multi.ts +26 -0
- package/src/upstream/ollama-usage.ts +157 -0
- package/src/upstream/ollama.ts +275 -0
- package/src/upstream/openrouter.ts +19 -1
- package/src/upstream/types.ts +2 -0
- package/src/util/sqlite.ts +25 -1
- package/test/catalog.test.ts +44 -0
- package/test/classify.test.ts +41 -5
- package/test/compaction.test.ts +1 -0
- package/test/config-wizard.test.ts +77 -1
- package/test/configure-logic.test.ts +129 -33
- package/test/embed-lifecycle.test.ts +1 -0
- package/test/failover.test.ts +148 -3
- package/test/features.test.ts +35 -0
- package/test/http-resilience.test.ts +24 -0
- package/test/ollama.test.ts +506 -0
- package/test/omp-credentials.test.ts +43 -1
- package/test/report-hub.test.ts +341 -0
- package/test/report-logic.test.ts +92 -0
- package/test/report.test.ts +217 -0
- package/test/select.test.ts +176 -1
- package/test/tier-plan.test.ts +159 -1
- package/test/toast-logic.test.ts +11 -2
- package/test/tokens.test.ts +71 -1
- package/test/trust-attribution.test.ts +2 -2
- package/test/turn.test.ts +124 -7
- package/tools/build-site.ts +1 -0
|
@@ -7,14 +7,14 @@
|
|
|
7
7
|
},
|
|
8
8
|
"metadata": {
|
|
9
9
|
"description": "auto-model-router: a local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter",
|
|
10
|
-
"version": "0.
|
|
10
|
+
"version": "0.3.0",
|
|
11
11
|
"pluginRoot": "."
|
|
12
12
|
},
|
|
13
13
|
"plugins": [
|
|
14
14
|
{
|
|
15
15
|
"name": "auto-model-router",
|
|
16
16
|
"description": "Local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter. Runs in-process, routes per turn by price and task complexity, with budget caps, mid-stream escalation, and cache-aware hysteresis.",
|
|
17
|
-
"version": "0.
|
|
17
|
+
"version": "0.3.0",
|
|
18
18
|
"author": {
|
|
19
19
|
"name": "drewappling",
|
|
20
20
|
"email": "drewappling@gmail.com"
|
package/README.md
CHANGED
|
@@ -3,10 +3,13 @@
|
|
|
3
3
|
**[Website & benchmarks →](https://drewappling.github.io/auto-model-router/)**
|
|
4
4
|
|
|
5
5
|
A local model router for [Oh My Pi](https://github.com/oh-my-pi). It presents
|
|
6
|
-
itself as one keyless OpenAI-compatible provider, then picks a concrete
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
6
|
+
itself as one keyless OpenAI-compatible provider, then picks a concrete model
|
|
7
|
+
**per turn** — from OpenRouter's catalog, and from
|
|
8
|
+
[Ollama Cloud](#ollama-cloud) when that is enabled too — based on measured
|
|
9
|
+
price and estimated task complexity, including mid-conversation, when a
|
|
10
|
+
session shifts from mechanical tool-loop churn to genuine reasoning work.
|
|
11
|
+
With both providers on, every turn ranks the candidates of both together and
|
|
12
|
+
fails over across them.
|
|
10
13
|
|
|
11
14
|
auto-model-router runs **embedded inside the omp process** (as an omp extension) — no
|
|
12
15
|
separate server, no orphaned process. It binds a free OS-assigned port and
|
|
@@ -145,12 +148,17 @@ Harness, tasks and raw per-turn data:
|
|
|
145
148
|
graph LR
|
|
146
149
|
omp[omp process] -->|OpenAI chat completions| wire[wire/openai]
|
|
147
150
|
wire -->|NormRequest| router[router]
|
|
148
|
-
|
|
151
|
+
orcat[OpenRouter /models] --> catalog[catalog<br/>one merged snapshot]
|
|
152
|
+
olcat[Ollama /api/tags + prices<br/>optional] --> catalog
|
|
153
|
+
catalog --> router
|
|
149
154
|
cost[cost<br/>forecast + ledger] --> router
|
|
150
155
|
router -->|Decision| guard[escalation guard]
|
|
151
|
-
guard -->|rendered body| up[upstream/
|
|
152
|
-
up
|
|
153
|
-
|
|
156
|
+
guard -->|rendered body| up[upstream/multi<br/>by slug prefix]
|
|
157
|
+
up --> or[openrouter]
|
|
158
|
+
up --> ol[ollama<br/>ollama/… slugs]
|
|
159
|
+
or -->|UpstreamChunk| guard
|
|
160
|
+
ol -->|UpstreamChunk| guard
|
|
161
|
+
guard -->|commit, fail over, or retry upward| wire
|
|
154
162
|
guard -->|usage + reported cost| cost
|
|
155
163
|
```
|
|
156
164
|
|
|
@@ -163,15 +171,15 @@ without touching routing.
|
|
|
163
171
|
|
|
164
172
|
| Path | Responsibility |
|
|
165
173
|
| --- | --- |
|
|
166
|
-
| `src/catalog/` | Fetch and normalize OpenRouter `/api/v1/models`: pricing, capability flags, Artificial Analysis quality indices. SQLite-cached with TTL. |
|
|
167
|
-
| `src/cost/` | Cost forecasting per candidate; reconciliation against OpenRouter's authoritative `usage.cost`; the spend ledger; per-model trust; rolling blended rate. |
|
|
174
|
+
| `src/catalog/` | Fetch and normalize OpenRouter `/api/v1/models`: pricing, capability flags, Artificial Analysis quality indices. SQLite-cached with TTL. `ollama-catalog.ts` builds Ollama Cloud models from `/api/tags`, a shipped price table and OpenRouter twins; `composite.ts` merges the two into one snapshot. |
|
|
175
|
+
| `src/cost/` | Cost forecasting per candidate; reconciliation against OpenRouter's authoritative `usage.cost`; the spend ledger; per-model trust; rolling blended rate; `report.ts` usage analytics. |
|
|
168
176
|
| `src/tokens/` | Token estimation with no tokenizer dependency, self-calibrating from observed `prompt_tokens` per tokenizer family. |
|
|
169
177
|
| `src/wire/` | Protocol boundary. `wire/openai/` implements chat completions in and SSE out. |
|
|
170
178
|
| `src/router/` | Feature extraction, complexity classification, candidate filtering and scoring, hysteresis, cache-breakpoint placement, budget guard, probe planning. |
|
|
171
|
-
| `src/upstream/` | OpenRouter
|
|
179
|
+
| `src/upstream/` | Transports: OpenRouter (streaming dispatch, `session_id` stickiness, error classification, fallback arrays) and Ollama Cloud (body rewrite for its compatibility layer, quota/rate-limit breaker); `multi.ts` dispatches by slug prefix. |
|
|
172
180
|
| `src/config/` | Configuration loading, schema validation, and the built-in defaults. |
|
|
173
|
-
| `src/cli/` | `serve`, `stats`, `models`, `explain`, `config` commands. |
|
|
174
|
-
| `omp-extension/` | The omp extensions: `router-embed.ts`, `router-toast.ts`, `router-configure.ts
|
|
181
|
+
| `src/cli/` | `serve`, `stats`, `report`, `models`, `explain`, `config` commands. |
|
|
182
|
+
| `omp-extension/` | The omp extensions: `router-embed.ts`, `router-toast.ts`, `router-configure.ts` (`/router` config, report, status). |
|
|
175
183
|
|
|
176
184
|
### Two cost numbers, never conflated
|
|
177
185
|
|
|
@@ -204,7 +212,7 @@ Then add the shipped extensions to omp's `~/.omp/agent/config.yml`
|
|
|
204
212
|
extensions:
|
|
205
213
|
- auto-model-router/omp-extension/router-embed.ts
|
|
206
214
|
- auto-model-router/omp-extension/router-toast.ts # optional: chosen-model toasts
|
|
207
|
-
- auto-model-router/omp-extension/router-configure.ts # optional: /router
|
|
215
|
+
- auto-model-router/omp-extension/router-configure.ts # optional: /router config, report, status
|
|
208
216
|
```
|
|
209
217
|
|
|
210
218
|
### From the repo (cross-platform installer)
|
|
@@ -227,7 +235,7 @@ The installer adds:
|
|
|
227
235
|
|
|
228
236
|
- `router-embed.ts` — **required**; runs the router in-process.
|
|
229
237
|
- `router-toast.ts` — optional; chosen-model toasts.
|
|
230
|
-
- `router-configure.ts` — optional; the `/router` command.
|
|
238
|
+
- `router-configure.ts` — optional; the `/router` command (configure, usage reports, status).
|
|
231
239
|
|
|
232
240
|
Or add the paths by hand to omp's `~/.omp/agent/config.yml`:
|
|
233
241
|
|
|
@@ -236,7 +244,7 @@ Or add the paths by hand to omp's `~/.omp/agent/config.yml`:
|
|
|
236
244
|
extensions:
|
|
237
245
|
- /path/to/auto-model-router/omp-extension/router-embed.ts
|
|
238
246
|
- /path/to/auto-model-router/omp-extension/router-toast.ts # optional: chosen-model toasts
|
|
239
|
-
- /path/to/auto-model-router/omp-extension/router-configure.ts # optional: /router
|
|
247
|
+
- /path/to/auto-model-router/omp-extension/router-configure.ts # optional: /router config, report, status
|
|
240
248
|
```
|
|
241
249
|
|
|
242
250
|
Then restart the omp session (extensions load at session start).
|
|
@@ -405,7 +413,11 @@ Your OpenRouter guardrails — model and provider allowlists, budget limits,
|
|
|
405
413
|
Zero-Data-Retention and privacy rules — are therefore the router's outer
|
|
406
414
|
boundary: a model your key cannot reach is never a routing candidate. The
|
|
407
415
|
catalog is refetched in the background every `catalogRefreshMs` (default 5 min),
|
|
408
|
-
so tightening or relaxing a guardrail is picked up without a restart.
|
|
416
|
+
so tightening or relaxing a guardrail is picked up without a restart. A refresh
|
|
417
|
+
that keeps fewer than half the previous models is adopted (your guardrails are
|
|
418
|
+
authoritative) but logged at `warn` and reported as `catalog.shrink` on
|
|
419
|
+
`GET /health` until the catalog recovers, because a sharp shrink reroutes every
|
|
420
|
+
turn onto whatever survived. If a
|
|
409
421
|
guardrail narrows the eligible set below a tier's quality floor,
|
|
410
422
|
`adaptiveTierFloors` (on by default) relaxes that tier to the best available
|
|
411
423
|
models rather than leaving it empty — see [Adaptive tier floors](#adaptive-tier-floors)
|
|
@@ -465,6 +477,40 @@ virtual profile it picked. Every routed response carries
|
|
|
465
477
|
|
|
466
478
|
---
|
|
467
479
|
|
|
480
|
+
## Usage reports
|
|
481
|
+
|
|
482
|
+
The ledger records every dispatch: model decided and served, tier, provider,
|
|
483
|
+
tokens (including cached), reported cost, time to first token, total latency,
|
|
484
|
+
escalation signal, error. Three views aggregate it, all from the same
|
|
485
|
+
`buildUsageReport` in `src/cost/report.ts`:
|
|
486
|
+
|
|
487
|
+
- `/router report` in omp — a fullscreen hub with the `/models` look: views
|
|
488
|
+
for overview, providers, models, tiers, by day and status in a sidebar, plus
|
|
489
|
+
a Window selector (24h / 7d / 30d / 90d) and, when `OMP_HARNESS_ID` is set,
|
|
490
|
+
a scope toggle between this harness and all harnesses. ↑/↓ move, Enter
|
|
491
|
+
applies a window or scope, ←/→ also cycle the window, PgUp/PgDn scroll, r
|
|
492
|
+
reloads, Esc closes. Headless sessions get the same report as text in the
|
|
493
|
+
transcript. Falls back to reading the ledger directly if the router is
|
|
494
|
+
unreachable.
|
|
495
|
+
- `auto-model-router report --days 7 [--harness <id>] [--json]` on the terminal.
|
|
496
|
+
- `GET /v1/router/report?days=7&harness=<id>` for dashboards.
|
|
497
|
+
|
|
498
|
+
What it shows, for the window:
|
|
499
|
+
|
|
500
|
+
| Block | Columns |
|
|
501
|
+
| --- | --- |
|
|
502
|
+
| totals | spend, dispatches, conversations, $/dispatch, prompt and completion tokens, cache hit rate, model switches, escalations, failovers, errors (aborted separately) |
|
|
503
|
+
| providers | per upstream (`openrouter`, `ollama`): dispatches, spend, share, cache hit, mean TTFT, tokens/s, escalations, errors |
|
|
504
|
+
| models | per served slug (top 12 by spend): the same plus the tier mix it was routed for |
|
|
505
|
+
| tiers | per tier: dispatches, spend, share, cache hit, mean prompt tokens, escalations |
|
|
506
|
+
| by day | UTC calendar days: dispatches, spend, cache hit |
|
|
507
|
+
|
|
508
|
+
Spend follows the ledger's rule — the provider's reported cost when it gave
|
|
509
|
+
one, else the usage-priced figure the router computed, else the forecast.
|
|
510
|
+
Speed uses only clean streamed rows (TTFT recorded, no error); tokens/s is
|
|
511
|
+
completion tokens over time after first token. Ollama Cloud does not report
|
|
512
|
+
cached tokens, so its cache column reads 0% by construction.
|
|
513
|
+
|
|
468
514
|
## Configuring the router
|
|
469
515
|
|
|
470
516
|
The router's own config lives at `$AUTO_MODEL_ROUTER_HOME/config.yml` (default
|
|
@@ -474,12 +520,23 @@ built-in defaults below. There are two ways to edit it:
|
|
|
474
520
|
### Via `/router` (in-omp, native UI)
|
|
475
521
|
|
|
476
522
|
Install the `router-configure` extension, restart omp, then run `/router` in
|
|
477
|
-
the session prompt.
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
523
|
+
the session prompt. With no arguments it shows a menu (Configure, Report,
|
|
524
|
+
Status); the subcommands go straight there:
|
|
525
|
+
|
|
526
|
+
| Command | What it does |
|
|
527
|
+
| --- | --- |
|
|
528
|
+
| `/router config` | Section picker over **every** config key: Server, OpenRouter, Ollama Cloud, Benchmarks, Tiers, Tasks, Filters, Classifier, Escalation, Hysteresis, Exploration, Cache, Compaction, Context (agentdox), Budget, Ledger, Logging, Profiles. Only `ollama.prices` and `ollama.twins` (maps) stay YAML-only. |
|
|
529
|
+
| `/router report` | Usage analytics in a fullscreen hub styled like `/models`: pick a view in the sidebar, set the window (24h / 7d / 30d / 90d) and the harness scope there too. `/router report 30d --all` presets them. See [Usage reports](#usage-reports). |
|
|
530
|
+
| `/router status` | The router's `/health`: key sources, catalog size and age, Ollama availability, plan usage and cost bias, agentdox bridge. |
|
|
531
|
+
|
|
532
|
+
Picking a section lists its fields with their current values (pending edits
|
|
533
|
+
marked), so you see the settings before choosing one to change. Each field
|
|
534
|
+
dialog names the current value in its title, marks it in pickers and uses it
|
|
535
|
+
as the placeholder — empty input keeps it, `-` clears an optional field,
|
|
536
|
+
credentials show as `set`/`unset` and are never echoed. `Save and exit` writes the merged config
|
|
537
|
+
(schema-checked and backed up first). Tier, task, filter, classifier,
|
|
538
|
+
hysteresis, exploration, compaction, cache and budget changes hot-reload;
|
|
539
|
+
restart omp for `server`, `openrouter`, `ollama`, `context` and `ledger`.
|
|
483
540
|
|
|
484
541
|
### Via `auto-model-router config` (text wizard / CLI)
|
|
485
542
|
|
|
@@ -541,6 +598,30 @@ what each one does. All values are optional; omit a key to use its default.
|
|
|
541
598
|
| `catalogTtlMs` | `21600000` (6 h) | How long the model catalog is cached before a forced refetch. |
|
|
542
599
|
| `catalogRefreshMs` | `300000` (5 min) | Background catalog refetch interval; `0` disables it. |
|
|
543
600
|
|
|
601
|
+
### `ollama` — Ollama Cloud as a second upstream
|
|
602
|
+
|
|
603
|
+
Off by default. When enabled, Ollama Cloud models join the same catalog as
|
|
604
|
+
OpenRouter's under `ollama/<id>` slugs and are ranked on the same economics:
|
|
605
|
+
a turn picks whichever provider's model is cheapest above the tier's floor,
|
|
606
|
+
and same-tier failover crosses providers (a 402 or 429 from Ollama retries on
|
|
607
|
+
an OpenRouter sibling). See [Ollama Cloud](#ollama-cloud) below.
|
|
608
|
+
|
|
609
|
+
| Key | Default | Meaning |
|
|
610
|
+
| --- | --- | --- |
|
|
611
|
+
| `enabled` | `false` | Master switch. |
|
|
612
|
+
| `baseUrl` | `http://127.0.0.1:11434/v1` | A local daemon (proxies `:cloud` models under its sign-in) or `https://ollama.com/v1`. |
|
|
613
|
+
| `apiKey` | unset | Bearer for ollama.com. Resolved from config, then `OLLAMA_API_KEY`, then omp's own auth store (`/login ollama-cloud` in omp) — the same borrowing as the OpenRouter key. The daemon needs none. |
|
|
614
|
+
| `timeoutMs` | `600000` | Per-request timeout. |
|
|
615
|
+
| `catalogTtlMs` | `300000` | Re-list models when the last listing is older than this. |
|
|
616
|
+
| `includeLocal` | `false` | Also expose the daemon's local models (only those named in `prices`). |
|
|
617
|
+
| `prices` | `{}` | USD per million tokens by bare cloud name (`{input, cachedInput?, output}`); overrides or extends the shipped snapshot. |
|
|
618
|
+
| `twins` | `{}` | Bare cloud name → OpenRouter slug, to pin a quality-score twin the name match misses. |
|
|
619
|
+
| `costBias` | `1` | Multiplier on Ollama models' effective cost in ranking; below 1 prefers Ollama. The ledger still records list price. |
|
|
620
|
+
| `biasUntilUsage` | `0.9` | Share of the plan's included monthly credits at which `costBias` switches off and Ollama ranks at list price. Read live from ollama.com's `/api/usage`, which reports usage relative to the plan, so the same value is right on Pro, Max or Team. `1` keeps the bias regardless. |
|
|
621
|
+
| `usagePollMs` | `600000` (10 min) | How often plan usage is re-read. `0` disables it (static bias). Needs the API key; the daemon path without one keeps a static bias. |
|
|
622
|
+
| `quotaCooldownMs` | `900000` | Route around Ollama this long after a 402 (credits exhausted). |
|
|
623
|
+
| `rateLimitCooldownMs` | `60000` | Route around Ollama this long after a 429 (concurrency cap). |
|
|
624
|
+
|
|
544
625
|
### `tiers` — per-tier economic envelope
|
|
545
626
|
|
|
546
627
|
Each tier (`trivial`, `simple`, `moderate`, `hard`) is a `tierConfig`:
|
|
@@ -577,6 +658,10 @@ Each task (`coding`, `vision`, `documentation`, `data`, `chat`) is a
|
|
|
577
658
|
| `minTrustSamples` | `12` | Attempts before trust is enforced. |
|
|
578
659
|
| `trustScopedByHarness` | `false` | `true` = each harness reads only its own trust rows. |
|
|
579
660
|
| `contextHeadroom` | `1.25` | Fraction of context kept free (a model must fit prompt × this). |
|
|
661
|
+
| `latencyWeight` | `0` | How hard to penalise slow models in scoring (soft multiplier on effective cost). `0` disables it. |
|
|
662
|
+
| `latencyMinSamples` | `20` | Streamed samples before latency is judged against a model. |
|
|
663
|
+
| `maxExpectedWaitMs` | unset | Absolute expected-wait ceiling (ms): a hard drop for models *proven* slower (≥ `latencyMinSamples`), regardless of price. The soft penalty is multiplicative and capped, so it cannot demote a slow-but-cheap model — this can. New models keep their cold-start turns; relaxed with trust in tier rescue. Undefined ⇒ off. |
|
|
664
|
+
| `escalationCostWeight` | `0` | Price a model's measured escalation rate at what an escalated retry actually bills (the ledger's $/prompt-token of `attempt > 0` rows), 0–1. The trust divisor reads a 4% escalation rate as a 4% surcharge; the real cost is a whole re-dispatch on the next tier's model. `0` disables the term. |
|
|
580
665
|
|
|
581
666
|
### `classifier` — complexity adjudication
|
|
582
667
|
|
|
@@ -591,6 +676,7 @@ Each task (`coding`, `vision`, `documentation`, `data`, `chat`) is a
|
|
|
591
676
|
| `toolAxis` | `coding` | Quality axis for tool-heavy turns. |
|
|
592
677
|
| `chatAxis` | `intelligence` | Quality axis for chat turns. |
|
|
593
678
|
| `agenticLoopDepth` | `3` | Tool-loop depth at which a turn is treated as agentic. |
|
|
679
|
+
| `mechanicalRetryFactor` | `0.2` | Fraction of the failed-tool and circular-call weights kept on a tool-result continuation; `1` disables the damping. |
|
|
594
680
|
|
|
595
681
|
### `escalation` — mid-stream retry upward
|
|
596
682
|
|
|
@@ -604,6 +690,14 @@ Each task (`coding`, `vision`, `documentation`, `data`, `chat`) is a
|
|
|
604
690
|
| `triggers` | 5 signals | `malformed_tool_args`, `refusal`, `empty_completion`, `repeat_tool_call`, `missing_expected_tool_call`. |
|
|
605
691
|
| `escalateOnLengthStop` | `true` | Escalate on a `length` finish that truncated tool-call args. |
|
|
606
692
|
|
|
693
|
+
The model that produced the rejected output never serves the retry, at this
|
|
694
|
+
tier or the next. Signals that indict the *provider* rather than the tier —
|
|
695
|
+
`empty_completion`, `refusal`, and an error finish — first try a different
|
|
696
|
+
model in the **same** tier (bounded, like a 5xx failover) and only then step
|
|
697
|
+
up; structural signals (`malformed_tool_args`, `repeat_tool_call`, a truncated
|
|
698
|
+
tool call) escalate a tier directly. A client that hangs up after the finish
|
|
699
|
+
event has already arrived is treated as a completed turn, not an error.
|
|
700
|
+
|
|
607
701
|
### `hysteresis` — cache-aware model stickiness
|
|
608
702
|
|
|
609
703
|
| Key | Default | Meaning |
|
|
@@ -611,8 +705,30 @@ Each task (`coding`, `vision`, `documentation`, `data`, `chat`) is a
|
|
|
611
705
|
| `holdTurns` | `2` | Hold a chosen model this many turns before it can downgrade. |
|
|
612
706
|
| `holdTurnsAfterEscalation` | `4` | Hold longer after an escalation. |
|
|
613
707
|
| `switchMargin` | `1.3` | Switching must beat the warm-cache discount by this factor. Lower = switch away from a warm model more readily. |
|
|
708
|
+
| `switchHorizonTurns` | `1` | Turns the stay/switch comparison is amortised over: `H × stayWarm` vs `switchCold + (H − 1) × newWarm`. `1` is the one-turn comparison, which can keep a dear model warm indefinitely when the cheaper winner is itself dear cold; a small `H` lets a switch that pays for itself within a few turns go ahead. |
|
|
614
709
|
| `cacheWarmTtlMs` | `300000` (5 min) | How long a model's prompt cache is considered warm. |
|
|
615
710
|
| `maxDowngradePerTurn` | `1` | Max tiers a turn may drop in one step (avoids quality cliffs). |
|
|
711
|
+
| `breakHoldOnMechanical` | `false` | Let a tool-result continuation that classifies *below* the held tier escape the hold (still bounded by `maxDowngradePerTurn`). Worth enabling when the held tier is expensive. |
|
|
712
|
+
|
|
713
|
+
### `compaction` — shrink stale tool output before dispatch
|
|
714
|
+
|
|
715
|
+
Off by default; see `docs/context-optimization.md`. Every edit shrinks one
|
|
716
|
+
tool-result's content in place behind a re-run breadcrumb, never removes or
|
|
717
|
+
reorders a message, and the plan is persisted per conversation so already
|
|
718
|
+
shrunk results stay shrunk (rewriting them would break the prompt cache).
|
|
719
|
+
|
|
720
|
+
| Key | Default | Meaning |
|
|
721
|
+
| --- | --- | --- |
|
|
722
|
+
| `enabled` | `false` | Master switch. |
|
|
723
|
+
| `budgetTokens` | `40000` | Compact when the (already compacted) prompt exceeds this many tokens. |
|
|
724
|
+
| `floorRatio` | `1` | Once compaction fires, compact down to this fraction of the budget so the plan holds for several turns. |
|
|
725
|
+
| `replanGrowthRatio` | `1` | Above 1, only extend an existing plan once the compacted prompt has grown by this factor since the plan was made. Rations plan churn when the budget is unreachable (every turn over budget); fit-to-window is never rationed. |
|
|
726
|
+
| `fitToWindow` | `true` | Also compact when the prompt would overflow the profile's context window. |
|
|
727
|
+
| `protectRecentTurns` | `4` | Never touch the last N user/assistant turns or the volatile tail. |
|
|
728
|
+
| `maxToolResultBytes` | `4096` | Tool results larger than this (outside the protected window) are truncated. |
|
|
729
|
+
| `keepHeadBytes` / `keepTailBytes` | `512` / `512` | Bytes kept around the elision breadcrumb. |
|
|
730
|
+
| `elideSupersededReads` | `true` | Stub an older result when a newer call to the same resource supersedes it. |
|
|
731
|
+
| `collapseDuplicateResults` | `true` | Collapse byte-identical repeated results to a single copy. |
|
|
616
732
|
|
|
617
733
|
### `cache` — prompt-cache breakpoints
|
|
618
734
|
|
|
@@ -658,9 +774,62 @@ Each profile is a complete entry (arrays replace wholesale):
|
|
|
658
774
|
|
|
659
775
|
| Key | Default | Meaning |
|
|
660
776
|
| --- | --- | --- |
|
|
661
|
-
| `adaptiveTierFloors` | `true` |
|
|
777
|
+
| `adaptiveTierFloors` | `true` | Relax a tier's quality floor to a catalog-derived band when fewer than three available models meet the configured floor (never raising it). A floor that three or more models meet stands as written. |
|
|
662
778
|
| `logLevel` | `info` | `silent`/`error`/`warn`/`info`/`debug`. |
|
|
663
779
|
|
|
780
|
+
## Ollama Cloud
|
|
781
|
+
|
|
782
|
+
[Ollama Cloud](https://ollama.com/cloud) hosts open models behind Ollama's own
|
|
783
|
+
OpenAI-compatible endpoint and bills them per token against a plan's monthly
|
|
784
|
+
credits. The router can treat it as a second upstream next to OpenRouter:
|
|
785
|
+
|
|
786
|
+
```yaml
|
|
787
|
+
ollama:
|
|
788
|
+
enabled: true
|
|
789
|
+
# default: the local daemon, which proxies `:cloud` models under whatever
|
|
790
|
+
# account `ollama signin` used. For ollama.com directly:
|
|
791
|
+
# baseUrl: https://ollama.com/v1
|
|
792
|
+
# apiKey: <from https://ollama.com/settings/keys, or OLLAMA_API_KEY, or
|
|
793
|
+
# borrowed from omp after `/login ollama-cloud` — no copy needed>
|
|
794
|
+
```
|
|
795
|
+
|
|
796
|
+
What happens once it is on:
|
|
797
|
+
|
|
798
|
+
- **One catalog.** Every cloud model Ollama lists becomes `ollama/<id>` (for
|
|
799
|
+
example `ollama/glm-5.3-flash:cloud` through the daemon, `ollama/glm-5.3-flash`
|
|
800
|
+
on ollama.com) with the context length and capabilities Ollama publishes
|
|
801
|
+
(`/api/tags` on the daemon, `/api/show` on ollama.com).
|
|
802
|
+
- **Prices come from a shipped table**, because no Ollama endpoint publishes
|
|
803
|
+
them: the rates on [ollama.com/pricing](https://ollama.com/pricing) as of
|
|
804
|
+
2026-09-05 (`src/catalog/ollama-prices.ts`). `ollama.prices` overrides or
|
|
805
|
+
extends it; a model with no rate from either is left out, on the same rule
|
|
806
|
+
that drops unpriced OpenRouter models.
|
|
807
|
+
- **Quality scores come from the OpenRouter twin.** Ollama publishes none, so
|
|
808
|
+
`glm-5.3-flash` inherits `z-ai/glm-5.3-flash`'s indices by name match, which
|
|
809
|
+
is what lets it serve `simple` and above. `ollama.twins` pins a match the
|
|
810
|
+
name normaliser cannot make; an unmatched model is unscored and serves only
|
|
811
|
+
`trivial`.
|
|
812
|
+
- **Same economics, same failover.** Candidates from both providers are ranked
|
|
813
|
+
together; `costBias` tilts the comparison while a plan's included credits
|
|
814
|
+
would otherwise go unused. **Credit-aware by default:** the router reads the
|
|
815
|
+
plan's usage from ollama.com (`/api/usage`, the same figure the dashboard
|
|
816
|
+
shows, as a share of the plan's included credits) every `usagePollMs`, and
|
|
817
|
+
once it passes `biasUntilUsage` (90%) Ollama ranks at list price for the rest
|
|
818
|
+
of the billing month. Because the figure is relative to the plan, nothing
|
|
819
|
+
about Pro, Max or Team needs configuring; `/health` shows the raw reading
|
|
820
|
+
and the multiplier in force. A 402 (credits exhausted) or 429 (concurrency cap) from
|
|
821
|
+
Ollama fails the attempt over to an OpenRouter sibling in the same tier and
|
|
822
|
+
opens a breaker, so following turns route straight to OpenRouter without
|
|
823
|
+
paying a doomed dispatch first; `/health` shows `ollama.available` and the
|
|
824
|
+
cooldown.
|
|
825
|
+
- **Ollama reports no cost per response**, so the ledger records the
|
|
826
|
+
predicted figure at list price for those rows.
|
|
827
|
+
|
|
828
|
+
Ollama's compatibility layer differs from OpenRouter's in a few ways the
|
|
829
|
+
router handles for you: no `models[]` fallback cascade, no `tool_choice`,
|
|
830
|
+
`reasoning_effort` instead of the `reasoning` object, and no `cache_control`
|
|
831
|
+
markers (they are stripped before dispatch).
|
|
832
|
+
|
|
664
833
|
## Multiple coding harnesses, one router
|
|
665
834
|
|
|
666
835
|
A single embedded router can serve several omp sessions without them stepping
|
|
@@ -782,8 +951,10 @@ come from a small omp extension that polls the router's in-process ledger:
|
|
|
782
951
|
```
|
|
783
952
|
|
|
784
953
|
It raises a TUI toast (`ctx.ui.notify`) like
|
|
785
|
-
`meta/muse-glimmer-30b [trivial] · $0.00001`
|
|
786
|
-
|
|
954
|
+
`openrouter · meta/muse-glimmer-30b [trivial] · $0.00001` or
|
|
955
|
+
`ollama · glm-5.3-flash [moderate] · $0.00070` whenever a new model is chosen —
|
|
956
|
+
provider first, so a mixed catalog is legible at a glance. Install it by adding
|
|
957
|
+
the file's absolute path to omp's `extensions:` list.
|
|
787
958
|
|
|
788
959
|
Because the embedded router binds a random port, the toast resolves the router
|
|
789
960
|
base URL on every poll in this order: the embedded router's port file
|
|
@@ -860,13 +1031,20 @@ absolute floor admits nothing and the router is trapped in the lowest tier.
|
|
|
860
1031
|
|
|
861
1032
|
With `adaptiveTierFloors: true` (the default), every catalog refresh ranks the
|
|
862
1033
|
**available** scored models and splits them into four quantile bands, taking
|
|
863
|
-
each band's lower bound as that tier's adaptive floor. The
|
|
864
|
-
|
|
1034
|
+
each band's lower bound as that tier's adaptive floor. The band applies only
|
|
1035
|
+
when the configured floor leaves the tier **thin** — fewer than three available
|
|
1036
|
+
models meet it — in which case the floor enforced is `min(configured, adaptive)`:
|
|
865
1037
|
|
|
866
|
-
- a
|
|
1038
|
+
- a floor that three or more models meet stands exactly as configured — no
|
|
1039
|
+
behaviour change on a healthy catalog;
|
|
867
1040
|
- a narrowed catalog falls back to the adaptive floor, so `hard` still gets the
|
|
868
1041
|
best quartile of what is available instead of nothing.
|
|
869
1042
|
|
|
1043
|
+
The thinness gate matters: a wide catalog carries a long tail of weak scored
|
|
1044
|
+
models, so its quantile bands sit *below* the configured floors (measured on a
|
|
1045
|
+
347-model key-admitted catalog: coding p50 = 45.8 against `moderate`'s 60), and
|
|
1046
|
+
an unconditional `min` would quietly relax every tier.
|
|
1047
|
+
|
|
870
1048
|
Relaxation is one-directional by design: an adaptive floor may only **lower** a
|
|
871
1049
|
tier floor, never raise one. Two things are deliberately exempt:
|
|
872
1050
|
|
|
@@ -931,6 +1109,10 @@ Known gaps:
|
|
|
931
1109
|
|
|
932
1110
|
- The `pi-native` front end is designed for but not implemented; only the
|
|
933
1111
|
OpenAI-compatible wire exists today.
|
|
1112
|
+
- `escalation.maxHoldMs` is only enforced when a chunk arrives: a stream that
|
|
1113
|
+
emits nothing at all holds the client until the upstream ends. Left as is on
|
|
1114
|
+
purpose — a timer would escalate every turn whose first token lands after
|
|
1115
|
+
8s, which on live traffic is most of glm-5.3-flash's.
|
|
934
1116
|
- Blended `cost` figures in `models.yml` are refreshed by re-running
|
|
935
1117
|
`auto-model-router config --write`, not automatically.
|
|
936
1118
|
|