auto-model-router 0.17.0 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/agentdox/SKILL.md +13 -3
- package/.mcp.json +1 -1
- package/.omp-plugin/marketplace.json +2 -2
- package/CLAUDE.md +2 -2
- package/README.md +35 -6
- package/package.json +1 -1
- package/src/catalog/composite.ts +12 -1
- package/src/cost/ledger.ts +9 -2
- package/src/cost/types.ts +7 -0
- package/src/cost/views.ts +27 -10
- package/src/lib.ts +1 -0
- package/src/router/candidates.ts +31 -16
- package/src/server/catalog-view.ts +130 -0
- package/src/server/http.ts +41 -1
- package/src/server/providers.ts +2 -2
- package/src/server/turn.ts +4 -0
- package/src/util/sqlite.ts +12 -1
- package/test/catalog-view.test.ts +249 -0
- package/test/migrations.test.ts +10 -2
- package/test/trust-attribution.test.ts +2 -2
- package/test/turn.test.ts +23 -0
- package/test/views.test.ts +39 -13
|
@@ -5,7 +5,7 @@ description: "Use agentdox — the shared memory, docs, and context server — t
|
|
|
5
5
|
|
|
6
6
|
# agentdox — the standard interaction protocol
|
|
7
7
|
|
|
8
|
-
agentdox is a shared memory + docs + conversation store at `http://localhost:
|
|
8
|
+
agentdox is a shared memory + docs + conversation store at `http://localhost:8790/agentdox`. It is what
|
|
9
9
|
stops you rediscovering the same project facts every session.
|
|
10
10
|
|
|
11
11
|
**Follow this protocol identically every time.** Consistency is the point: the value of the
|
|
@@ -55,7 +55,7 @@ Deterministic, so the same folder always resolves to the same slug:
|
|
|
55
55
|
repo root:
|
|
56
56
|
|
|
57
57
|
```ini
|
|
58
|
-
AGENTDOX_URL=http://localhost:
|
|
58
|
+
AGENTDOX_URL=http://localhost:8790/agentdox
|
|
59
59
|
AGENTDOX_SCOPE=<slug>
|
|
60
60
|
AGENTDOX_TOKEN=<the global PAT>
|
|
61
61
|
```
|
|
@@ -104,6 +104,16 @@ determine the scope, ask — do not guess, and do not fall back to a default.
|
|
|
104
104
|
|
|
105
105
|
## 1. Pick your transport — MCP or REST
|
|
106
106
|
|
|
107
|
+
> **Where the store lives (since 2026-09-10).** The store is the team edition's embedded agentdox,
|
|
108
|
+
> reached through the team: REST at `http://localhost:8790/agentdox/...` and MCP at
|
|
109
|
+
> `http://localhost:8790/mcp`, with a **team member key** (`amrt_…`) as `AGENTDOX_TOKEN`. The team maps
|
|
110
|
+
> the key to the member's grants (write on every project of their groups, read on the group's
|
|
111
|
+
> context), so a 401 means the key and a 403 means the project belongs to another group — not a PAT.
|
|
112
|
+
> agentdox's own `/auth` routes are not reachable through the team; projects are created on the
|
|
113
|
+
> team's Groups page or appear on a workspace's first turn. The old dev container on :3003 is retired.
|
|
114
|
+
> A member key updates entries (`PATCH /memory/:id`) but cannot delete them (agentdox wants admin on the
|
|
115
|
+
> scope for that): to retire a fact, lower its importance or leave it to the team's retention pass.
|
|
116
|
+
|
|
107
117
|
Both hit the same live store with the same RBAC. **Check which you have, then use it:**
|
|
108
118
|
|
|
109
119
|
- **MCP tools present** — use them. Claude Code and Cursor mount them as `memory_add`,
|
|
@@ -129,7 +139,7 @@ inline JSON and avoids quoting pain in `curl`:
|
|
|
129
139
|
```ts
|
|
130
140
|
const tok = /AGENTDOX_TOKEN=(.+)/.exec(await Bun.file(".env.agentdox").text())?.[1]?.trim() ?? "";
|
|
131
141
|
const H = { Authorization: `Bearer ${tok}`, "content-type": "application/json" };
|
|
132
|
-
await fetch("http://localhost:
|
|
142
|
+
await fetch("http://localhost:8790/agentdox/memory", { method: "POST", headers: H,
|
|
133
143
|
body: JSON.stringify({ content: "…", category: "<scope>", importance: 0.9 }) });
|
|
134
144
|
```
|
|
135
145
|
|
package/.mcp.json
CHANGED
|
@@ -7,14 +7,14 @@
|
|
|
7
7
|
},
|
|
8
8
|
"metadata": {
|
|
9
9
|
"description": "auto-model-router: a local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter",
|
|
10
|
-
"version": "0.
|
|
10
|
+
"version": "0.19.0",
|
|
11
11
|
"pluginRoot": "."
|
|
12
12
|
},
|
|
13
13
|
"plugins": [
|
|
14
14
|
{
|
|
15
15
|
"name": "auto-model-router",
|
|
16
16
|
"description": "Local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter. Runs in-process, routes per turn by price and task complexity, with budget caps, mid-stream escalation, and cache-aware hysteresis.",
|
|
17
|
-
"version": "0.
|
|
17
|
+
"version": "0.19.0",
|
|
18
18
|
"author": {
|
|
19
19
|
"name": "drewappling",
|
|
20
20
|
"email": "drewappling@gmail.com"
|
package/CLAUDE.md
CHANGED
|
@@ -24,7 +24,7 @@ tips). Read it before using agentdox; do not improvise from this summary.
|
|
|
24
24
|
| --- | --- |
|
|
25
25
|
| Token, URL, scope | `.env.agentdox` in this repo root (gitignored — never commit) |
|
|
26
26
|
| What `.mcp.json` reads | the `AGENTDOX_TOKEN` **environment variable** (Windows *User* scope; shells opened before it was set lack it) |
|
|
27
|
-
| Server | `http://localhost:
|
|
27
|
+
| Server | `http://localhost:8790/agentdox` — the team edition's embedded agentdox (the `agentdox-server` container is retired); endpoints in `E:/projects/agentdox/packages/server/src/index.ts` |
|
|
28
28
|
| Admin token (re-mint) | `E:/projects/agentdox/deploy/.env` |
|
|
29
29
|
|
|
30
30
|
Two rules that cause silent mistakes: **memory calls take `category`, everything else takes
|
|
@@ -46,7 +46,7 @@ behind them are recorded there as they are made.
|
|
|
46
46
|
Turning the bridge on for the router itself (distinct from the MCP wiring above):
|
|
47
47
|
|
|
48
48
|
```bash
|
|
49
|
-
export AGENTDOX_URL=http://localhost:
|
|
49
|
+
export AGENTDOX_URL=http://localhost:8790/agentdox
|
|
50
50
|
export AGENTDOX_TOKEN=<the global PAT, same value .mcp.json uses>
|
|
51
51
|
export AGENTDOX_SCOPE=omp-router
|
|
52
52
|
```
|
package/README.md
CHANGED
|
@@ -694,7 +694,9 @@ virtual profile it picked. Every routed response carries
|
|
|
694
694
|
|
|
695
695
|
The ledger records every dispatch: model decided and served, tier, provider,
|
|
696
696
|
tokens (including cached), reported cost, time to first token, total latency,
|
|
697
|
-
escalation signal, error
|
|
697
|
+
escalation signal, error, and the agentdox context scope the turn carried (the
|
|
698
|
+
project it belongs to; NULL for a turn that carried none, and for every row
|
|
699
|
+
written before v0.19.0). Three views aggregate it, all from the same
|
|
698
700
|
`buildUsageReport` in `src/cost/report.ts`:
|
|
699
701
|
|
|
700
702
|
- `/router report` in omp — a fullscreen hub with the `/models` look: views
|
|
@@ -706,16 +708,28 @@ escalation signal, error. Three views aggregate it, all from the same
|
|
|
706
708
|
transcript. Falls back to reading the ledger directly if the router is
|
|
707
709
|
unreachable.
|
|
708
710
|
- `auto-model-router report --days 7 [--harness <id>] [--json]` on the terminal.
|
|
709
|
-
- `auto-model-router export --days 30 [--harness a,b] [--json]`: one row per day, harness
|
|
710
|
-
and
|
|
711
|
-
`
|
|
712
|
-
|
|
711
|
+
- `auto-model-router export --days 30 [--harness a,b] [--json]`: one row per day, harness,
|
|
712
|
+
model and context scope (dispatches, tokens, spend, escalations, errors) as CSV — the
|
|
713
|
+
`scope` column is last, and is empty for a turn that carried none. Also
|
|
714
|
+
`GET /v1/router/export?days=&harness=[&format=json]`; `GET /v1/router/spend?sinceMs=&harness=[&scope=]`
|
|
715
|
+
gives spend over a harness set since an instant, narrowed to one context scope when `scope`
|
|
716
|
+
is given (a project's own bill), and `GET /v1/router/feedback?days=&harness=`
|
|
713
717
|
lists verdicts by model and the recent ones with the harness that gave them, and
|
|
714
718
|
`GET /v1/router/decisions?harness=&days=|since=&slug=&tier=&limit=` is the decision trail
|
|
715
719
|
itself, newest first, each turn with its reasons, the classifier's view, forecast against
|
|
716
720
|
bill, escalation signal and verdicts (`?session=` narrows to one omp session, as `/router
|
|
717
721
|
why` does). These are what a front door such as the team edition reads instead of the
|
|
718
722
|
ledger file.
|
|
723
|
+
- `GET /v1/router/catalog[?policy=<X-Omp-Policy JSON>]` — the model catalog as data: every
|
|
724
|
+
model the router knows, sorted by slug, with provider (`openrouter`, `ollama`, a named
|
|
725
|
+
upstream's id), vendor, context, capabilities, prices in USD per million tokens and quality
|
|
726
|
+
scores; a model whose upstream cannot serve right now (no key, disabled, in cooldown) is
|
|
727
|
+
listed too. With `policy` every model also carries `admitted` and, when out, `reason`
|
|
728
|
+
(`not in the allow list`, `denied by <glob>`, `pinned to <slug>`, the router's own filters,
|
|
729
|
+
or the upstream's state), judged by the same matcher a turn uses, so the view never drifts
|
|
730
|
+
from what a turn gets; tiers are per turn and do not take part. A malformed policy is a
|
|
731
|
+
400; before the first fetch it answers `{ "fetchedAtMs": 0, "models": [] }` rather than
|
|
732
|
+
waiting. A team front door reads it for its model governance views.
|
|
719
733
|
- `GET /v1/router/report?days=7&harness=<id>` for dashboards (`harness` may be
|
|
720
734
|
a comma-separated set of ids, for a group).
|
|
721
735
|
- `GET /v1/router/summary?harness=<id>` — the daily summary as JSON (`auto=1`
|
|
@@ -1287,7 +1301,8 @@ request allow list replaces the configured one, a deny list adds to it.
|
|
|
1287
1301
|
widen it. `pin` forces one model the way `/router pin` does, unless a session
|
|
1288
1302
|
override already pinned one. Every field is optional; a malformed header is
|
|
1289
1303
|
ignored rather than failing the turn. The decision trail records what the
|
|
1290
|
-
policy changed (`policy: …`)
|
|
1304
|
+
policy changed (`policy: …`), and `GET /v1/router/catalog?policy=…` shows what a
|
|
1305
|
+
policy admits, model by model, without routing a turn.
|
|
1291
1306
|
|
|
1292
1307
|
## Using a remote router
|
|
1293
1308
|
|
|
@@ -1544,6 +1559,20 @@ a team posts exactly what it always did, so its block is byte-identical; an olde
|
|
|
1544
1559
|
agentdox ignores the keys it does not know. `context.layers: false` is the kill
|
|
1545
1560
|
switch — the headers are still parsed but nothing new goes to agentdox.
|
|
1546
1561
|
|
|
1562
|
+
### Charging spend back to a project
|
|
1563
|
+
|
|
1564
|
+
Every ledger row records the scope the bridge resolved for that turn — the
|
|
1565
|
+
request's `X-Agentdox-Scope`, or `context.defaultScope` behind it — so the money
|
|
1566
|
+
and the project are the same row. That is what a front door bills from: it names
|
|
1567
|
+
a project's scope on the turns it forwards, then reads its share back with
|
|
1568
|
+
`GET /v1/router/spend?sinceMs=&scope=<scope>` (composable with `harness=`, so one
|
|
1569
|
+
member's spend on one project is one call), or takes the whole split from
|
|
1570
|
+
`GET /v1/router/export`, whose rows now group by day, harness, model **and**
|
|
1571
|
+
scope and carry the scope as their last CSV column. `decisionEntries` (and
|
|
1572
|
+
`GET /v1/router/decisions`) carry it on each turn too. Rows written before
|
|
1573
|
+
v0.19.0, and turns that carried no scope at all, store NULL and export as `""` —
|
|
1574
|
+
old ledgers open and gain the column, they just have nothing to charge.
|
|
1575
|
+
|
|
1547
1576
|
### The origin fingerprint
|
|
1548
1577
|
|
|
1549
1578
|
The scope is the folder's name, and folder names collide: two unrelated repositories
|
package/package.json
CHANGED
package/src/catalog/composite.ts
CHANGED
|
@@ -46,7 +46,7 @@ export function createCompositeCatalog(
|
|
|
46
46
|
ollama: OllamaCatalogSource,
|
|
47
47
|
availability: OllamaAvailability,
|
|
48
48
|
bias: CompositeBias = { costBias: 1, biasUntilUsage: 1, usage: NO_USAGE },
|
|
49
|
-
): CatalogSource & { ollamaModels(): CatalogModel[]; ollamaBias(): number } {
|
|
49
|
+
): CatalogSource & { ollamaModels(): CatalogModel[]; ollamaBias(): number; peekAll(): CatalogSnapshot | null } {
|
|
50
50
|
let lastBase: CatalogSnapshot | null = null;
|
|
51
51
|
let lastOllama: readonly CatalogModel[] = [];
|
|
52
52
|
let lastAvailable = true;
|
|
@@ -118,6 +118,17 @@ export function createCompositeCatalog(
|
|
|
118
118
|
return combine(base, ollama.peek());
|
|
119
119
|
},
|
|
120
120
|
find,
|
|
121
|
+
/**
|
|
122
|
+
* Every model the catalog knows, whether or not its upstream can take a
|
|
123
|
+
* turn now: OpenRouter's without a key, Ollama's in cooldown, a named
|
|
124
|
+
* upstream's in cooldown. `peek()` is what routes; this is what a front
|
|
125
|
+
* door lists so it can say why a model is out. No network.
|
|
126
|
+
*/
|
|
127
|
+
peekAll(): CatalogSnapshot | null {
|
|
128
|
+
const base = openrouter.peek();
|
|
129
|
+
if (base === null) return null;
|
|
130
|
+
return { ...base, models: [...base.models, ...ollama.peek(), ...(bias.named?.models(base.models) ?? NO_NAMED)] };
|
|
131
|
+
},
|
|
121
132
|
lastShrink(): CatalogShrink | null {
|
|
122
133
|
return openrouter.lastShrink?.() ?? null;
|
|
123
134
|
},
|
package/src/cost/ledger.ts
CHANGED
|
@@ -99,6 +99,7 @@ export interface LedgerRow {
|
|
|
99
99
|
upstream_generation_id: string | null;
|
|
100
100
|
error: string | null;
|
|
101
101
|
prompt_tokens_saved: number | null;
|
|
102
|
+
scope: string | null;
|
|
102
103
|
}
|
|
103
104
|
|
|
104
105
|
interface TrustRow {
|
|
@@ -274,6 +275,9 @@ export function toEntry(row: LedgerRow): LedgerEntry {
|
|
|
274
275
|
upstreamGenerationId: row.upstream_generation_id,
|
|
275
276
|
error: row.error,
|
|
276
277
|
promptTokensSaved: row.prompt_tokens_saved ?? 0,
|
|
278
|
+
// Optional under exactOptionalPropertyTypes: an old row (or a scopeless
|
|
279
|
+
// turn) simply has no `scope`, rather than an explicit undefined.
|
|
280
|
+
...(row.scope === null || row.scope === undefined ? {} : { scope: row.scope }),
|
|
277
281
|
};
|
|
278
282
|
}
|
|
279
283
|
|
|
@@ -319,8 +323,8 @@ export function createLedger(db: Database, cfg: RouterConfig): Ledger {
|
|
|
319
323
|
id, created_at_ms, conversation_key, session_id, turn, requested_model, harness_id, omp_session_id, slug, served_slug,
|
|
320
324
|
tier, classification_source, reasons, predicted_usd, reported_usd, usage, cost_breakdown,
|
|
321
325
|
attempt, escalation_signal, latency_ms, ttft_ms, finish_reason, wasted, upstream_generation_id, error,
|
|
322
|
-
error_kind, features, score, confidence, task, classifier_reasons, explored_from, hold_arm, prompt_tokens_saved
|
|
323
|
-
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
|
326
|
+
error_kind, features, score, confidence, task, classifier_reasons, explored_from, hold_arm, prompt_tokens_saved, scope
|
|
327
|
+
) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
|
|
324
328
|
);
|
|
325
329
|
const calibrationStmt = db.query(
|
|
326
330
|
`INSERT INTO token_calibration (tokenizer, est_bytes, actual_tokens, samples) VALUES (?, ?, ?, 1)
|
|
@@ -473,6 +477,9 @@ export function createLedger(db: Database, cfg: RouterConfig): Ledger {
|
|
|
473
477
|
entry.exploredFrom,
|
|
474
478
|
entry.holdArm,
|
|
475
479
|
entry.promptTokensSaved,
|
|
480
|
+
// A turn that carried no scope stores NULL, exactly as every row
|
|
481
|
+
// written before v18 did; "" and absent are the same fact.
|
|
482
|
+
entry.scope === undefined || entry.scope === "" ? null : entry.scope,
|
|
476
483
|
);
|
|
477
484
|
// Always consume the pending estimate, even when the turn failed, so a
|
|
478
485
|
// dead turn's bytes can never pair with a later turn's tokens. Only
|
package/src/cost/types.ts
CHANGED
|
@@ -146,6 +146,13 @@ export interface LedgerEntry {
|
|
|
146
146
|
error: string | null;
|
|
147
147
|
/** Prompt tokens removed by compaction before dispatch. 0 when none. NULL before v12. */
|
|
148
148
|
promptTokensSaved: number;
|
|
149
|
+
/**
|
|
150
|
+
* The agentdox context scope this turn carried — what the bridge resolved
|
|
151
|
+
* for it (the request's `X-Agentdox-Scope`, or `context.defaultScope`).
|
|
152
|
+
* Absent, or empty, when the turn carried none, and stored as NULL; a front
|
|
153
|
+
* door charges the row's spend back to that project with it. NULL before v18.
|
|
154
|
+
*/
|
|
155
|
+
scope?: string;
|
|
149
156
|
/**
|
|
150
157
|
* The catalog model that served, for the cost split. The ledger can price
|
|
151
158
|
* OpenRouter slugs from its own cached catalog payload; a model from another
|
package/src/cost/views.ts
CHANGED
|
@@ -44,6 +44,8 @@ export interface ExportRow {
|
|
|
44
44
|
day: string;
|
|
45
45
|
harnessId: string;
|
|
46
46
|
slug: string;
|
|
47
|
+
/** The agentdox context scope the turns carried; "" for rows that carried none (every row before v18). */
|
|
48
|
+
scope: string;
|
|
47
49
|
provider: string;
|
|
48
50
|
dispatches: number;
|
|
49
51
|
promptTokens: number;
|
|
@@ -125,12 +127,21 @@ export function decisionEntries(db: Database, filter: DecisionFilter): DecisionE
|
|
|
125
127
|
return entries.map((e) => ({ ...e, feedback: verdicts.get(e.id) ?? [] }));
|
|
126
128
|
}
|
|
127
129
|
|
|
128
|
-
/**
|
|
129
|
-
|
|
130
|
+
/**
|
|
131
|
+
* Spend (reported where present, predicted otherwise) since `sinceMs`, digest calls included
|
|
132
|
+
* as the ledger counts them. `contextScope`, when given, narrows to the turns that carried
|
|
133
|
+
* exactly that agentdox scope — what a front door charges back to one project.
|
|
134
|
+
*/
|
|
135
|
+
export function spendUsdSince(db: Database, sinceMs: number, harness: HarnessScope, contextScope?: string): number {
|
|
130
136
|
const s = scope(harness, "harness_id");
|
|
131
137
|
if (s === null) return 0;
|
|
132
|
-
const where = ["created_at_ms >= $since", ...s.sql]
|
|
133
|
-
const
|
|
138
|
+
const where = ["created_at_ms >= $since", ...s.sql];
|
|
139
|
+
const bind: Record<string, string | number> = { $since: sinceMs, ...s.bind };
|
|
140
|
+
if (contextScope !== undefined && contextScope !== "") {
|
|
141
|
+
where.push("scope = $scope");
|
|
142
|
+
bind.$scope = contextScope;
|
|
143
|
+
}
|
|
144
|
+
const row = db.query(`SELECT COALESCE(SUM(${USD}), 0) AS usd FROM ledger WHERE ${where.join(" AND ")}`).get(bind) as { usd: number };
|
|
134
145
|
return row.usd;
|
|
135
146
|
}
|
|
136
147
|
|
|
@@ -163,14 +174,19 @@ export function feedbackView(db: Database, sinceMs: number, harness: HarnessScop
|
|
|
163
174
|
return { byModel, recent };
|
|
164
175
|
}
|
|
165
176
|
|
|
166
|
-
/**
|
|
177
|
+
/**
|
|
178
|
+
* One row per UTC day, harness, served model and context scope since `sinceMs`; digest calls
|
|
179
|
+
* are excluded as in the report. The scope splits a harness's day by project, so a front door
|
|
180
|
+
* can charge each project its own share; rows from before v18 (and turns that carried no
|
|
181
|
+
* scope) group under "".
|
|
182
|
+
*/
|
|
167
183
|
export function exportRows(db: Database, sinceMs: number, harness: HarnessScope): ExportRow[] {
|
|
168
184
|
const s = scope(harness, "harness_id");
|
|
169
185
|
if (s === null) return [];
|
|
170
186
|
const where = ["created_at_ms >= $since", "requested_model <> 'digest'", ...s.sql].join(" AND ");
|
|
171
187
|
const rows = db
|
|
172
188
|
.query(
|
|
173
|
-
`SELECT strftime('%Y-%m-%d', created_at_ms / 1000, 'unixepoch') AS day, harness_id, COALESCE(served_slug, slug) AS slug,
|
|
189
|
+
`SELECT strftime('%Y-%m-%d', created_at_ms / 1000, 'unixepoch') AS day, harness_id, COALESCE(served_slug, slug) AS slug, COALESCE(scope, '') AS scope,
|
|
174
190
|
COUNT(*) AS dispatches,
|
|
175
191
|
COALESCE(SUM(json_extract(usage, '$.promptTokens')), 0) AS prompt_tokens,
|
|
176
192
|
COALESCE(SUM(json_extract(usage, '$.cachedTokens')), 0) AS cached_tokens,
|
|
@@ -178,13 +194,14 @@ export function exportRows(db: Database, sinceMs: number, harness: HarnessScope)
|
|
|
178
194
|
COALESCE(SUM(${USD}), 0) AS spend,
|
|
179
195
|
SUM(CASE WHEN escalation_signal IS NOT NULL THEN 1 ELSE 0 END) AS escalations,
|
|
180
196
|
SUM(CASE WHEN error IS NOT NULL THEN 1 ELSE 0 END) AS errors
|
|
181
|
-
FROM ledger WHERE ${where} GROUP BY day, harness_id, slug ORDER BY day ASC, harness_id ASC, spend DESC`,
|
|
197
|
+
FROM ledger WHERE ${where} GROUP BY day, harness_id, slug, scope ORDER BY day ASC, harness_id ASC, spend DESC`,
|
|
182
198
|
)
|
|
183
|
-
.all({ $since: sinceMs, ...s.bind }) as { day: string; harness_id: string; slug: string; dispatches: number; prompt_tokens: number; cached_tokens: number; completion_tokens: number; spend: number; escalations: number; errors: number }[];
|
|
199
|
+
.all({ $since: sinceMs, ...s.bind }) as { day: string; harness_id: string; slug: string; scope: string; dispatches: number; prompt_tokens: number; cached_tokens: number; completion_tokens: number; spend: number; escalations: number; errors: number }[];
|
|
184
200
|
return rows.map((r) => ({
|
|
185
201
|
day: r.day,
|
|
186
202
|
harnessId: r.harness_id,
|
|
187
203
|
slug: r.slug,
|
|
204
|
+
scope: r.scope,
|
|
188
205
|
provider: providerOfSlug(r.slug),
|
|
189
206
|
dispatches: r.dispatches,
|
|
190
207
|
promptTokens: r.prompt_tokens,
|
|
@@ -196,7 +213,7 @@ export function exportRows(db: Database, sinceMs: number, harness: HarnessScope)
|
|
|
196
213
|
}));
|
|
197
214
|
}
|
|
198
215
|
|
|
199
|
-
export const EXPORT_COLUMNS = ["day", "harness", "model", "provider", "dispatches", "prompt_tokens", "cached_tokens", "completion_tokens", "spend_usd", "escalations", "errors"] as const;
|
|
216
|
+
export const EXPORT_COLUMNS = ["day", "harness", "model", "provider", "dispatches", "prompt_tokens", "cached_tokens", "completion_tokens", "spend_usd", "escalations", "errors", "scope"] as const;
|
|
200
217
|
|
|
201
218
|
export function csvCell(v: string | number): string {
|
|
202
219
|
const s = String(v);
|
|
@@ -206,7 +223,7 @@ export function csvCell(v: string | number): string {
|
|
|
206
223
|
/** CSV of export rows; spend to 6 decimals so sub-cent rows survive. */
|
|
207
224
|
export function exportCsv(rows: readonly ExportRow[]): string {
|
|
208
225
|
const lines = [EXPORT_COLUMNS.join(",")];
|
|
209
|
-
for (const r of rows) lines.push([r.day, r.harnessId, r.slug, r.provider, r.dispatches, r.promptTokens, r.cachedTokens, r.completionTokens, r.spendUsd.toFixed(6), r.escalations, r.errors].map(csvCell).join(","));
|
|
226
|
+
for (const r of rows) lines.push([r.day, r.harnessId, r.slug, r.provider, r.dispatches, r.promptTokens, r.cachedTokens, r.completionTokens, r.spendUsd.toFixed(6), r.escalations, r.errors, r.scope].map(csvCell).join(","));
|
|
210
227
|
return `${lines.join("\n")}\n`;
|
|
211
228
|
}
|
|
212
229
|
|
package/src/lib.ts
CHANGED
|
@@ -28,4 +28,5 @@ export { createFeedbackStore, type FeedbackStore, type FeedbackRecord } from "./
|
|
|
28
28
|
export { buildExecutable, collectPackageFiles, executableFileName, hostTarget, isExecutableTarget, EXECUTABLE_TARGETS, type ExecutableTarget, type BuildExecutableResult } from "./cli/build-executable.ts";
|
|
29
29
|
export { parseSkillsBundle, type SkillsBundle } from "./cli/skills.ts";
|
|
30
30
|
export type { RequestPolicy } from "./wire/types.ts";
|
|
31
|
+
export type { CatalogView, CatalogViewModel } from "./server/catalog-view.ts";
|
|
31
32
|
export type { Ledger, LedgerEntry } from "./cost/types.ts";
|
package/src/router/candidates.ts
CHANGED
|
@@ -48,12 +48,37 @@ export interface BuildCandidatesArgs {
|
|
|
48
48
|
excludeSlugs?: readonly string[];
|
|
49
49
|
}
|
|
50
50
|
|
|
51
|
-
/**
|
|
52
|
-
|
|
51
|
+
/**
|
|
52
|
+
* Tiny glob: `*` matches any run of characters; everything else is literal.
|
|
53
|
+
* The one matcher behind `filters.allow`/`filters.deny` and a request
|
|
54
|
+
* policy's lists; the catalog view (`GET /v1/router/catalog`) judges with it
|
|
55
|
+
* too, so what a front door shows never drifts from what a turn gets.
|
|
56
|
+
*/
|
|
57
|
+
export function globToRe(glob: string): RegExp {
|
|
53
58
|
const escaped = glob.replace(/[.*+?^${}()|[\]\\]/g, (ch) => (ch === "*" ? ".*" : `\\${ch}`));
|
|
54
59
|
return new RegExp(`^${escaped}$`);
|
|
55
60
|
}
|
|
56
61
|
|
|
62
|
+
/**
|
|
63
|
+
* The denials that precede any user configuration, as the reason text, or
|
|
64
|
+
* null when the model may be a candidate. These slugs can never serve an
|
|
65
|
+
* interactive turn:
|
|
66
|
+
* - "~vendor/model-latest": floating aliases whose identity changes
|
|
67
|
+
* underneath us, poisoning ledger trust statistics.
|
|
68
|
+
* - ":batch": asynchronous batch endpoints, unusable for streaming.
|
|
69
|
+
* - "stealth/": cloaked models with no stable identity.
|
|
70
|
+
* - "openrouter/": their meta-routers do our job at unknown cost.
|
|
71
|
+
* - a negative price: OpenRouter's unknown/dynamic sentinel (-1), never a discount.
|
|
72
|
+
*/
|
|
73
|
+
export function builtInDenial(model: CatalogModel): string | null {
|
|
74
|
+
const slug = model.slug;
|
|
75
|
+
if (slug.startsWith("~") || slug.endsWith(":batch") || slug.startsWith("stealth/") || model.author === "openrouter") {
|
|
76
|
+
return "built-in deny: floating alias, batch endpoint, stealth, or meta-router";
|
|
77
|
+
}
|
|
78
|
+
if (model.price.prompt < 0 || model.price.completion < 0) return "dynamic pricing sentinel";
|
|
79
|
+
return null;
|
|
80
|
+
}
|
|
81
|
+
|
|
57
82
|
/**
|
|
58
83
|
* Quality fallback chain: score the preferred axis first, then the general
|
|
59
84
|
* intelligence index (the most widely published), then the remaining axis.
|
|
@@ -182,20 +207,10 @@ export function buildCandidates(args: BuildCandidatesArgs): { candidates: Candid
|
|
|
182
207
|
continue;
|
|
183
208
|
}
|
|
184
209
|
|
|
185
|
-
// Hard-coded denials, before any user configuration.
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
// - ":batch": asynchronous batch endpoints, unusable for streaming.
|
|
190
|
-
// - "stealth/": cloaked models with no stable identity.
|
|
191
|
-
// - "openrouter/": their meta-routers do our job at unknown cost.
|
|
192
|
-
if (slug.startsWith("~") || slug.endsWith(":batch") || slug.startsWith("stealth/") || model.author === "openrouter") {
|
|
193
|
-
rejected.push({ slug, reason: "denylisted", detail: "built-in deny: floating alias, batch endpoint, stealth, or meta-router" });
|
|
194
|
-
continue;
|
|
195
|
-
}
|
|
196
|
-
// A negative price is OpenRouter's unknown/dynamic sentinel (-1), never a discount.
|
|
197
|
-
if (model.price.prompt < 0 || model.price.completion < 0) {
|
|
198
|
-
rejected.push({ slug, reason: "denylisted", detail: "dynamic pricing sentinel" });
|
|
210
|
+
// Hard-coded denials, before any user configuration.
|
|
211
|
+
const builtIn = builtInDenial(model);
|
|
212
|
+
if (builtIn !== null) {
|
|
213
|
+
rejected.push({ slug, reason: "denylisted", detail: builtIn });
|
|
199
214
|
continue;
|
|
200
215
|
}
|
|
201
216
|
if (allowRes.length > 0 && !allowRes.some((re) => re.test(slug))) {
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The router's model catalog as data, with a policy's verdict per model: what
|
|
3
|
+
* `GET /v1/router/catalog` answers. A team front door renders it for its
|
|
4
|
+
* governance views ("which models can this group reach, and why not the
|
|
5
|
+
* rest"), so admission is decided by the SAME matcher and in the SAME order
|
|
6
|
+
* as `buildCandidates` — the built-in denials, then allow, deny, free and
|
|
7
|
+
* tool support — over the filters `applyRequestPolicy` produced, and a pin
|
|
8
|
+
* takes effect only when the pinned model itself survives them, exactly as
|
|
9
|
+
* `select` treats a forced slug. Tiers are per turn and take no part.
|
|
10
|
+
*/
|
|
11
|
+
|
|
12
|
+
import type { CatalogModel, Modality, QualityScores } from "../catalog/types.ts";
|
|
13
|
+
import type { FilterConfig } from "../config/types.ts";
|
|
14
|
+
import { builtInDenial, globToRe } from "../router/candidates.ts";
|
|
15
|
+
|
|
16
|
+
export interface CatalogViewModel {
|
|
17
|
+
slug: string;
|
|
18
|
+
canonicalSlug: string;
|
|
19
|
+
name: string;
|
|
20
|
+
/** The upstream that serves it: `openrouter`, `ollama`, or a named upstream's id. */
|
|
21
|
+
provider: string;
|
|
22
|
+
/**
|
|
23
|
+
* The slug's namespace before the first `/` (`anthropic`). For a named
|
|
24
|
+
* upstream's `<id>/<model>` the vendor is the model id's own namespace when
|
|
25
|
+
* it carries one (`vllm/meta-llama/x` ⇒ `meta-llama`), else the upstream id.
|
|
26
|
+
*/
|
|
27
|
+
vendor: string;
|
|
28
|
+
contextLength: number;
|
|
29
|
+
maxCompletionTokens?: number;
|
|
30
|
+
supportsTools: boolean;
|
|
31
|
+
supportsReasoning: boolean;
|
|
32
|
+
reasoningMandatory: boolean;
|
|
33
|
+
inputModalities: Modality[];
|
|
34
|
+
/** USD per MILLION tokens, the catalog's own units. */
|
|
35
|
+
price: { prompt: number; completion: number; cacheRead?: number; cacheWrite?: number };
|
|
36
|
+
quality: QualityScores;
|
|
37
|
+
isFree: boolean;
|
|
38
|
+
/** Present only when a policy was asked about. */
|
|
39
|
+
admitted?: boolean;
|
|
40
|
+
/** Present only when not admitted: which filter kept the model out. */
|
|
41
|
+
reason?: string;
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
export interface CatalogView {
|
|
45
|
+
/** When the catalog was last fetched; 0 before the first fetch. */
|
|
46
|
+
fetchedAtMs: number;
|
|
47
|
+
/** Sorted by slug. */
|
|
48
|
+
models: CatalogViewModel[];
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
/** The filters a turn routes under: the configured ones, or those `applyRequestPolicy` merged a policy into. */
|
|
52
|
+
export type AdmissionFilters = Pick<FilterConfig, "allow" | "deny" | "includeFree" | "requireToolSupport">;
|
|
53
|
+
|
|
54
|
+
export interface CatalogViewArgs {
|
|
55
|
+
models: readonly CatalogModel[];
|
|
56
|
+
fetchedAtMs: number;
|
|
57
|
+
/** When given, every model carries `admitted` and, if out, `reason`. */
|
|
58
|
+
verdict?: { filters: AdmissionFilters; pin?: string };
|
|
59
|
+
/** Why a provider cannot take a turn now, or null when it can. */
|
|
60
|
+
unserved: (provider: string) => string | null;
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
/** Per-token catalog prices as USD per million, rounded so `0.22` does not come out as `0.22000000000000003`. */
|
|
64
|
+
function perMillion(usdPerToken: number): number {
|
|
65
|
+
return Math.round(usdPerToken * 1e6 * 1e6) / 1e6;
|
|
66
|
+
}
|
|
67
|
+
|
|
68
|
+
export function vendorOf(model: Pick<CatalogModel, "slug" | "provider">): string {
|
|
69
|
+
const slash = model.slug.indexOf("/");
|
|
70
|
+
const head = slash === -1 ? model.slug : model.slug.slice(0, slash);
|
|
71
|
+
if (model.provider === "openrouter" || model.provider === "ollama" || head !== model.provider) return head;
|
|
72
|
+
const rest = model.slug.slice(slash + 1);
|
|
73
|
+
const inner = rest.indexOf("/");
|
|
74
|
+
return inner === -1 ? model.provider : rest.slice(0, inner);
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
/**
|
|
78
|
+
* Why the filters keep a model out, or null when it passes. The order is
|
|
79
|
+
* `buildCandidates`' so the first reason is the one a turn would record.
|
|
80
|
+
*/
|
|
81
|
+
function filterReason(model: CatalogModel, filters: AdmissionFilters, allowRes: readonly RegExp[], denyRes: readonly RegExp[]): string | null {
|
|
82
|
+
const builtIn = builtInDenial(model);
|
|
83
|
+
if (builtIn !== null) return builtIn;
|
|
84
|
+
if (allowRes.length > 0 && !allowRes.some((re) => re.test(model.slug))) return "not in the allow list";
|
|
85
|
+
const denied = denyRes.findIndex((re) => re.test(model.slug));
|
|
86
|
+
if (denied !== -1) return `denied by ${filters.deny[denied]}`;
|
|
87
|
+
if (model.isFree && !filters.includeFree) return "free models excluded (filters.includeFree)";
|
|
88
|
+
if (filters.requireToolSupport && !model.supportsTools) return "no tool support (filters.requireToolSupport)";
|
|
89
|
+
return null;
|
|
90
|
+
}
|
|
91
|
+
|
|
92
|
+
export function catalogView(args: CatalogViewArgs): CatalogView {
|
|
93
|
+
const sorted = [...args.models].sort((a, b) => (a.slug < b.slug ? -1 : a.slug > b.slug ? 1 : 0));
|
|
94
|
+
const verdict = args.verdict;
|
|
95
|
+
const allowRes = verdict === undefined ? [] : verdict.filters.allow.map(globToRe);
|
|
96
|
+
const denyRes = verdict === undefined ? [] : verdict.filters.deny.map(globToRe);
|
|
97
|
+
// One reason per slug before the pin is considered: the pin only bites when
|
|
98
|
+
// the pinned model itself is in, as `select` ignores a pin the filters drop.
|
|
99
|
+
const reasons = new Map<string, string | null>();
|
|
100
|
+
if (verdict !== undefined) for (const m of sorted) reasons.set(m.slug, args.unserved(m.provider) ?? filterReason(m, verdict.filters, allowRes, denyRes));
|
|
101
|
+
const pin = verdict?.pin !== undefined && reasons.get(verdict.pin) === null ? verdict.pin : undefined;
|
|
102
|
+
const models = sorted.map((m): CatalogViewModel => {
|
|
103
|
+
const price: CatalogViewModel["price"] = { prompt: perMillion(m.price.prompt), completion: perMillion(m.price.completion) };
|
|
104
|
+
if (m.price.cacheRead !== undefined) price.cacheRead = perMillion(m.price.cacheRead);
|
|
105
|
+
if (m.price.cacheWrite !== undefined) price.cacheWrite = perMillion(m.price.cacheWrite);
|
|
106
|
+
const out: CatalogViewModel = {
|
|
107
|
+
slug: m.slug,
|
|
108
|
+
canonicalSlug: m.canonicalSlug,
|
|
109
|
+
name: m.name,
|
|
110
|
+
provider: m.provider,
|
|
111
|
+
vendor: vendorOf(m),
|
|
112
|
+
contextLength: m.contextLength,
|
|
113
|
+
...(m.maxCompletionTokens === undefined ? {} : { maxCompletionTokens: m.maxCompletionTokens }),
|
|
114
|
+
supportsTools: m.supportsTools,
|
|
115
|
+
supportsReasoning: m.supportsReasoning,
|
|
116
|
+
reasoningMandatory: m.reasoningMandatory,
|
|
117
|
+
inputModalities: [...m.inputModalities],
|
|
118
|
+
price,
|
|
119
|
+
quality: { ...m.quality },
|
|
120
|
+
isFree: m.isFree,
|
|
121
|
+
};
|
|
122
|
+
if (verdict !== undefined) {
|
|
123
|
+
const reason = reasons.get(m.slug) ?? (pin !== undefined && m.slug !== pin ? `pinned to ${pin}` : null);
|
|
124
|
+
out.admitted = reason === null;
|
|
125
|
+
if (reason !== null) out.reason = reason;
|
|
126
|
+
}
|
|
127
|
+
return out;
|
|
128
|
+
});
|
|
129
|
+
return { fetchedAtMs: args.fetchedAtMs, models };
|
|
130
|
+
}
|
package/src/server/http.ts
CHANGED
|
@@ -6,6 +6,10 @@ import { createBridgeFromConfig } from "../context/index.ts";
|
|
|
6
6
|
import { createFeedbackStore, type Verdict } from "../cost/feedback.ts";
|
|
7
7
|
import { createLedger } from "../cost/ledger.ts";
|
|
8
8
|
import { createSessionOverrides } from "./overrides.ts";
|
|
9
|
+
import { catalogView } from "./catalog-view.ts";
|
|
10
|
+
import { buildUpstreamModels } from "../catalog/static-catalog.ts";
|
|
11
|
+
import { applyRequestPolicy, resolveProfile } from "../router/index.ts";
|
|
12
|
+
import { parsePolicyHeader } from "../wire/openai/request.ts";
|
|
9
13
|
import { createDigester } from "./digest.ts";
|
|
10
14
|
import { advise } from "./advise.ts";
|
|
11
15
|
import { TIER_ORDER, type Tier } from "../router/types.ts";
|
|
@@ -484,12 +488,48 @@ export function startServer(cfg: RouterConfig): StartedServer {
|
|
|
484
488
|
if (req.method === "GET" && url.pathname === "/v1/router/stats") {
|
|
485
489
|
return json(computeStats(ledger));
|
|
486
490
|
}
|
|
491
|
+
if (req.method === "GET" && url.pathname === "/v1/router/catalog") {
|
|
492
|
+
// The catalog as data, judged under `?policy=` (the X-Omp-Policy
|
|
493
|
+
// JSON) when one is given: a front door's governance view. Never
|
|
494
|
+
// blocks on a fetch; before the first one it is empty.
|
|
495
|
+
const rawPolicy = url.searchParams.get("policy");
|
|
496
|
+
let verdict: { filters: RouterConfig["filters"]; pin?: string } | undefined;
|
|
497
|
+
if (rawPolicy !== null) {
|
|
498
|
+
// The header parser forgives a malformed value (a turn must not
|
|
499
|
+
// fail on it); a view asked about one must say so instead.
|
|
500
|
+
let parsed: unknown;
|
|
501
|
+
try {
|
|
502
|
+
parsed = JSON.parse(rawPolicy);
|
|
503
|
+
} catch {
|
|
504
|
+
parsed = undefined;
|
|
505
|
+
}
|
|
506
|
+
if (parsed === undefined || parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) {
|
|
507
|
+
return wireErrorResponse({ status: 400, code: "invalid_request_error", message: "policy must be a JSON object (the X-Omp-Policy shape)" });
|
|
508
|
+
}
|
|
509
|
+
const policed = applyRequestPolicy(resolveProfile(cfg, "auto"), cfg, parsePolicyHeader(rawPolicy), undefined);
|
|
510
|
+
verdict = { filters: policed.cfg.filters, ...(policed.forceSlug === undefined ? {} : { pin: policed.forceSlug }) };
|
|
511
|
+
}
|
|
512
|
+
const snap = catalog.peekAll?.() ?? catalog.peek();
|
|
513
|
+
// A disabled named upstream's models are not built for routing; list them too, so "why not" has an answer.
|
|
514
|
+
const disabled = snap === null ? [] : cfg.upstreams.filter((u) => !u.enabled).flatMap((u) => buildUpstreamModels(u, snap.models));
|
|
515
|
+
const unserved = (provider: string): string | null => {
|
|
516
|
+
if (provider === "openrouter") return cfg.openrouter.apiKey === "" ? "upstream openrouter has no API key" : null;
|
|
517
|
+
if (provider === "ollama") return !cfg.ollama.enabled ? "upstream ollama is disabled" : ollama.available() ? null : "upstream ollama is in cooldown";
|
|
518
|
+
const entry = cfg.upstreams.find((u) => u.id === provider);
|
|
519
|
+
if (entry === undefined) return `upstream ${provider} is not configured`;
|
|
520
|
+
if (!entry.enabled) return `upstream ${provider} is disabled`;
|
|
521
|
+
return providers.named(provider)?.available() ?? true ? null : `upstream ${provider} is in cooldown`;
|
|
522
|
+
};
|
|
523
|
+
return json(catalogView({ models: snap === null ? [] : [...snap.models, ...disabled], fetchedAtMs: snap?.fetchedAtMs ?? 0, ...(verdict === undefined ? {} : { verdict }), unserved }));
|
|
524
|
+
}
|
|
487
525
|
if (req.method === "GET" && url.pathname === "/v1/router/spend") {
|
|
488
526
|
// Spend since an instant over a harness set: what a front door's
|
|
489
527
|
// budget check needs when it cannot read the ledger file.
|
|
490
528
|
const since = Number.parseInt(url.searchParams.get("sinceMs") ?? "", 10);
|
|
491
529
|
if (!Number.isFinite(since)) return wireErrorResponse({ status: 400, code: "invalid_request_error", message: "sinceMs required" });
|
|
492
|
-
|
|
530
|
+
// `scope` narrows to one agentdox context scope: a project's own spend.
|
|
531
|
+
const contextScope = url.searchParams.get("scope") ?? "";
|
|
532
|
+
return json({ sinceMs: since, usd: spendUsdSince(db, since, harnessScopeParam(url.searchParams.get("harness")), contextScope), ...(contextScope === "" ? {} : { scope: contextScope }) });
|
|
493
533
|
}
|
|
494
534
|
if (req.method === "GET" && url.pathname === "/v1/router/feedback") {
|
|
495
535
|
const days = clampDays(url.searchParams.get("days"), 30);
|
package/src/server/providers.ts
CHANGED
|
@@ -11,7 +11,7 @@ import { createAnthropicClient } from "../upstream/anthropic.ts";
|
|
|
11
11
|
import { createCompatClient, type NamedUpstreamClient } from "../upstream/compat.ts";
|
|
12
12
|
import { createOllamaCatalog } from "../catalog/ollama-catalog.ts";
|
|
13
13
|
import { createCatalog } from "../catalog/openrouter-catalog.ts";
|
|
14
|
-
import type { CatalogSource } from "../catalog/types.ts";
|
|
14
|
+
import type { CatalogSnapshot, CatalogSource } from "../catalog/types.ts";
|
|
15
15
|
import type { RouterConfig } from "../config/types.ts";
|
|
16
16
|
import { createMultiUpstream } from "../upstream/multi.ts";
|
|
17
17
|
import { createOllamaClient, type OllamaClient } from "../upstream/ollama.ts";
|
|
@@ -24,7 +24,7 @@ import { createLogger, type Logger } from "../util/log.ts";
|
|
|
24
24
|
|
|
25
25
|
export interface Providers {
|
|
26
26
|
upstream: UpstreamClient;
|
|
27
|
-
catalog: CatalogSource & { ollamaModels?(): unknown[]; ollamaBias?(): number };
|
|
27
|
+
catalog: CatalogSource & { ollamaModels?(): unknown[]; ollamaBias?(): number; peekAll?(): CatalogSnapshot | null };
|
|
28
28
|
/** Always present: it carries the circuit breaker. Whether it SERVES follows `cfg.ollama.enabled`. */
|
|
29
29
|
ollama: OllamaClient;
|
|
30
30
|
/** True while Ollama Cloud is enabled and out of cooldown, read live. */
|
package/src/server/turn.ts
CHANGED
|
@@ -304,6 +304,10 @@ export async function runTurn(
|
|
|
304
304
|
turn: turnNumber,
|
|
305
305
|
requestedModel: req.requestedModel,
|
|
306
306
|
harnessId: req.harnessId,
|
|
307
|
+
// The context scope this turn carried, so a front door can charge the
|
|
308
|
+
// row back to a project. The resolved one the bridge used, header or
|
|
309
|
+
// configured default; "" stores as NULL.
|
|
310
|
+
scope: doxScope,
|
|
307
311
|
ompSessionId: req.ompSessionId,
|
|
308
312
|
slug: decision.slug,
|
|
309
313
|
servedSlug,
|
package/src/util/sqlite.ts
CHANGED
|
@@ -18,7 +18,7 @@ import { mkdirSync } from "node:fs";
|
|
|
18
18
|
import { dirname } from "node:path";
|
|
19
19
|
|
|
20
20
|
/** Bump when a migration is added; guarded below so reopening never regresses it. */
|
|
21
|
-
const USER_VERSION =
|
|
21
|
+
const USER_VERSION = 18;
|
|
22
22
|
|
|
23
23
|
const MIGRATIONS = `
|
|
24
24
|
CREATE TABLE IF NOT EXISTS catalog_cache (
|
|
@@ -286,6 +286,16 @@ const MIGRATE_V17 = `
|
|
|
286
286
|
ALTER TABLE conversations ADD COLUMN upgrade_deferred_tier TEXT;
|
|
287
287
|
`;
|
|
288
288
|
|
|
289
|
+
// v18: ledger records the agentdox context scope the turn carried — the scope
|
|
290
|
+
// the bridge resolved for it (the request header, or the configured default).
|
|
291
|
+
// A front door charges spend back to a project with it: the team edition names
|
|
292
|
+
// a project's scope on every turn, so `scope` is the only column that says
|
|
293
|
+
// which project a row belongs to. NULL on every row written before this, and
|
|
294
|
+
// on any turn that carried no scope at all; there is nothing to backfill from.
|
|
295
|
+
const MIGRATE_V18 = `
|
|
296
|
+
ALTER TABLE ledger ADD COLUMN scope TEXT;
|
|
297
|
+
`;
|
|
298
|
+
|
|
289
299
|
// v9: benchmark_cache holds the external benchmark feeds (Artificial Analysis,
|
|
290
300
|
// BenchLM) that backfill quality scores OpenRouter leaves unpublished. It is a
|
|
291
301
|
// whole new table, created idempotently by the MIGRATIONS block above, so there
|
|
@@ -327,6 +337,7 @@ export function openDb(path: string): Database {
|
|
|
327
337
|
if (!ledgerCols.some((c) => c.name === "explored_from")) db.exec(MIGRATE_V7);
|
|
328
338
|
if (!ledgerCols.some((c) => c.name === "hold_arm")) db.exec(MIGRATE_V8);
|
|
329
339
|
if (!ledgerCols.some((c) => c.name === "prompt_tokens_saved")) db.exec(MIGRATE_V12);
|
|
340
|
+
if (!ledgerCols.some((c) => c.name === "scope")) db.exec(MIGRATE_V18);
|
|
330
341
|
const convCols = db.query("PRAGMA table_info(conversations)").all() as { name: string }[];
|
|
331
342
|
if (!convCols.some((c) => c.name === "context_version")) db.exec(MIGRATE_V11);
|
|
332
343
|
if (!convCols.some((c) => c.name === "compaction_plan")) db.exec(MIGRATE_V13);
|
|
@@ -0,0 +1,249 @@
|
|
|
1
|
+
import { afterAll, beforeAll, describe, expect, test } from "bun:test";
|
|
2
|
+
import { mkdtempSync, rmSync } from "node:fs";
|
|
3
|
+
import { tmpdir } from "node:os";
|
|
4
|
+
import { join } from "node:path";
|
|
5
|
+
|
|
6
|
+
import { normalizeCatalogModel } from "../src/catalog/openrouter-catalog.ts";
|
|
7
|
+
import { buildUpstreamModels } from "../src/catalog/static-catalog.ts";
|
|
8
|
+
import type { CatalogModel } from "../src/catalog/types.ts";
|
|
9
|
+
import { DEFAULT_CONFIG } from "../src/config/defaults.ts";
|
|
10
|
+
import { completeUpstreamEntry } from "../src/config/upstreams.ts";
|
|
11
|
+
import { catalogView, vendorOf, type CatalogView, type CatalogViewModel } from "../src/server/catalog-view.ts";
|
|
12
|
+
import { startServer, type StartedServer } from "../src/server/http.ts";
|
|
13
|
+
import { openDb } from "../src/util/sqlite.ts";
|
|
14
|
+
|
|
15
|
+
/**
|
|
16
|
+
* The catalog as data for a front door's governance views: every model the
|
|
17
|
+
* router knows, and under a policy whether a turn could reach it and why not.
|
|
18
|
+
* Pinned over the pure view and over `GET /v1/router/catalog`, which judges
|
|
19
|
+
* with the router's own matcher — the same globs, the same order as
|
|
20
|
+
* `buildCandidates`, a pin that only bites when the pinned model is in.
|
|
21
|
+
*/
|
|
22
|
+
|
|
23
|
+
const FIXTURE = (await Bun.file("test/fixtures/openrouter-models.json").json()) as { data: unknown[] };
|
|
24
|
+
|
|
25
|
+
function model(over: Partial<CatalogModel> & { slug: string }): CatalogModel {
|
|
26
|
+
return {
|
|
27
|
+
provider: "openrouter",
|
|
28
|
+
canonicalSlug: `${over.slug}-20260101`,
|
|
29
|
+
name: over.slug,
|
|
30
|
+
contextLength: 200_000,
|
|
31
|
+
supportsTools: true,
|
|
32
|
+
supportsReasoning: false,
|
|
33
|
+
reasoningMandatory: false,
|
|
34
|
+
supportsToolChoice: true,
|
|
35
|
+
inputModalities: ["text"],
|
|
36
|
+
price: { prompt: 0.000003, completion: 0.000015 },
|
|
37
|
+
priceTiers: [],
|
|
38
|
+
quality: { intelligence: 60 },
|
|
39
|
+
tokenizer: "Claude",
|
|
40
|
+
isFree: false,
|
|
41
|
+
createdAtMs: 0,
|
|
42
|
+
author: over.slug.split("/")[0] ?? "",
|
|
43
|
+
...over,
|
|
44
|
+
};
|
|
45
|
+
}
|
|
46
|
+
|
|
47
|
+
const vllm = completeUpstreamEntry({ id: "vllm", kind: "openai", baseUrl: "http://vllm:8000/v1", apiKey: "", models: [{ id: "meta-llama/Llama-3", input: 0, output: 0, contextLength: 8_000 }] });
|
|
48
|
+
const azure = completeUpstreamEntry({ id: "azure-eu", kind: "azure", enabled: false, baseUrl: "https://r.openai.azure.com", apiKey: "k", models: [{ id: "gpt-4o-deploy", input: 2.5, output: 10, cachedInput: 1.25 }] });
|
|
49
|
+
|
|
50
|
+
const MODELS: CatalogModel[] = [
|
|
51
|
+
model({ slug: "openai/gpt-5", quality: { intelligence: 70, coding: 72 }, maxCompletionTokens: 128_000, price: { prompt: 0.00000125, completion: 0.00001, cacheRead: 0.000000125 } }),
|
|
52
|
+
model({ slug: "anthropic/claude-sonnet-5", supportsReasoning: true }),
|
|
53
|
+
model({ slug: "anthropic/claude-haiku-5" }),
|
|
54
|
+
model({ slug: "anthropic/claude-opus-5:batch" }),
|
|
55
|
+
model({ slug: "liquid/lfm:free", isFree: true, price: { prompt: 0, completion: 0 } }),
|
|
56
|
+
model({ slug: "tencent/translator", supportsTools: false }),
|
|
57
|
+
model({ slug: "ollama/glm-5.3-flash", provider: "ollama", author: "ollama" }),
|
|
58
|
+
...buildUpstreamModels(vllm, []),
|
|
59
|
+
...buildUpstreamModels(azure, []),
|
|
60
|
+
];
|
|
61
|
+
|
|
62
|
+
const served = () => null;
|
|
63
|
+
const bySlug = (view: CatalogView): Map<string, CatalogViewModel> => new Map(view.models.map((m) => [m.slug, m]));
|
|
64
|
+
|
|
65
|
+
describe("catalogView", () => {
|
|
66
|
+
test("sorted by slug, prices per million, vendor per slug, and no verdict without a policy", () => {
|
|
67
|
+
const view = catalogView({ models: MODELS, fetchedAtMs: 123, unserved: served });
|
|
68
|
+
expect(view.fetchedAtMs).toBe(123);
|
|
69
|
+
expect(view.models.map((m) => m.slug)).toEqual([...MODELS.map((m) => m.slug)].sort());
|
|
70
|
+
const gpt = bySlug(view).get("openai/gpt-5")!;
|
|
71
|
+
expect(gpt).toEqual({
|
|
72
|
+
slug: "openai/gpt-5",
|
|
73
|
+
canonicalSlug: "openai/gpt-5-20260101",
|
|
74
|
+
name: "openai/gpt-5",
|
|
75
|
+
provider: "openrouter",
|
|
76
|
+
vendor: "openai",
|
|
77
|
+
contextLength: 200_000,
|
|
78
|
+
maxCompletionTokens: 128_000,
|
|
79
|
+
supportsTools: true,
|
|
80
|
+
supportsReasoning: false,
|
|
81
|
+
reasoningMandatory: false,
|
|
82
|
+
inputModalities: ["text"],
|
|
83
|
+
price: { prompt: 1.25, completion: 10, cacheRead: 0.125 },
|
|
84
|
+
quality: { intelligence: 70, coding: 72 },
|
|
85
|
+
isFree: false,
|
|
86
|
+
});
|
|
87
|
+
expect("admitted" in gpt).toBe(false);
|
|
88
|
+
expect("reason" in gpt).toBe(false);
|
|
89
|
+
expect("maxCompletionTokens" in bySlug(view).get("anthropic/claude-haiku-5")!).toBe(false);
|
|
90
|
+
// A named upstream's prices are per million already; they round-trip.
|
|
91
|
+
expect(bySlug(view).get("azure-eu/gpt-4o-deploy")!.price).toEqual({ prompt: 2.5, completion: 10, cacheRead: 1.25 });
|
|
92
|
+
});
|
|
93
|
+
|
|
94
|
+
test("vendor: the namespace before the first slash; a named upstream's model id may carry its own", () => {
|
|
95
|
+
expect(vendorOf({ slug: "anthropic/claude-sonnet-5", provider: "openrouter" })).toBe("anthropic");
|
|
96
|
+
expect(vendorOf({ slug: "ollama/glm-5.3-flash", provider: "ollama" })).toBe("ollama");
|
|
97
|
+
expect(vendorOf({ slug: "vllm/meta-llama/Llama-3", provider: "vllm" })).toBe("meta-llama");
|
|
98
|
+
expect(vendorOf({ slug: "azure-eu/gpt-4o-deploy", provider: "azure-eu" })).toBe("azure-eu");
|
|
99
|
+
const view = bySlug(catalogView({ models: MODELS, fetchedAtMs: 0, unserved: served }));
|
|
100
|
+
expect(view.get("vllm/meta-llama/Llama-3")!.vendor).toBe("meta-llama");
|
|
101
|
+
expect(view.get("azure-eu/gpt-4o-deploy")!.vendor).toBe("azure-eu");
|
|
102
|
+
});
|
|
103
|
+
|
|
104
|
+
test("an empty policy still judges: the router's own filters and the upstream's state", () => {
|
|
105
|
+
const unserved = (p: string) => (p === "azure-eu" ? "upstream azure-eu is disabled" : null);
|
|
106
|
+
const view = bySlug(catalogView({ models: MODELS, fetchedAtMs: 0, verdict: { filters: DEFAULT_CONFIG.filters }, unserved }));
|
|
107
|
+
expect(view.get("openai/gpt-5")).toMatchObject({ admitted: true });
|
|
108
|
+
expect("reason" in view.get("openai/gpt-5")!).toBe(false);
|
|
109
|
+
expect(view.get("anthropic/claude-opus-5:batch")).toMatchObject({ admitted: false, reason: "built-in deny: floating alias, batch endpoint, stealth, or meta-router" });
|
|
110
|
+
expect(view.get("liquid/lfm:free")).toMatchObject({ admitted: false, reason: "free models excluded (filters.includeFree)" });
|
|
111
|
+
expect(view.get("tencent/translator")).toMatchObject({ admitted: false, reason: "no tool support (filters.requireToolSupport)" });
|
|
112
|
+
expect(view.get("azure-eu/gpt-4o-deploy")).toMatchObject({ admitted: false, reason: "upstream azure-eu is disabled" });
|
|
113
|
+
// A $0 named-upstream model is self-hosted, never "free".
|
|
114
|
+
expect(view.get("vllm/meta-llama/Llama-3")).toMatchObject({ admitted: true });
|
|
115
|
+
// Filters relaxed: the same models come in.
|
|
116
|
+
const open = bySlug(catalogView({ models: MODELS, fetchedAtMs: 0, verdict: { filters: { ...DEFAULT_CONFIG.filters, includeFree: true, requireToolSupport: false } }, unserved: served }));
|
|
117
|
+
expect(open.get("liquid/lfm:free")!.admitted).toBe(true);
|
|
118
|
+
expect(open.get("tencent/translator")!.admitted).toBe(true);
|
|
119
|
+
});
|
|
120
|
+
|
|
121
|
+
test("allow list, deny glob and pin, in the order a turn applies them", () => {
|
|
122
|
+
const filters = { ...DEFAULT_CONFIG.filters, allow: ["anthropic/*", "vllm/*"], deny: ["*haiku*"] };
|
|
123
|
+
const view = bySlug(catalogView({ models: MODELS, fetchedAtMs: 0, verdict: { filters }, unserved: served }));
|
|
124
|
+
expect(view.get("openai/gpt-5")).toMatchObject({ admitted: false, reason: "not in the allow list" });
|
|
125
|
+
expect(view.get("anthropic/claude-haiku-5")).toMatchObject({ admitted: false, reason: "denied by *haiku*" });
|
|
126
|
+
expect(view.get("anthropic/claude-sonnet-5")).toMatchObject({ admitted: true });
|
|
127
|
+
expect(view.get("vllm/meta-llama/Llama-3")).toMatchObject({ admitted: true });
|
|
128
|
+
// A pin keeps every other model out.
|
|
129
|
+
const pinned = bySlug(catalogView({ models: MODELS, fetchedAtMs: 0, verdict: { filters, pin: "anthropic/claude-sonnet-5" }, unserved: served }));
|
|
130
|
+
expect(pinned.get("anthropic/claude-sonnet-5")).toMatchObject({ admitted: true });
|
|
131
|
+
expect(pinned.get("vllm/meta-llama/Llama-3")).toMatchObject({ admitted: false, reason: "pinned to anthropic/claude-sonnet-5" });
|
|
132
|
+
expect(pinned.get("openai/gpt-5")).toMatchObject({ admitted: false, reason: "not in the allow list" }); // the earlier reason stands
|
|
133
|
+
// A pin the filters drop is ignored, as select ignores it: nothing else is pinned out.
|
|
134
|
+
const dropped = bySlug(catalogView({ models: MODELS, fetchedAtMs: 0, verdict: { filters, pin: "anthropic/claude-haiku-5" }, unserved: served }));
|
|
135
|
+
expect(dropped.get("anthropic/claude-haiku-5")).toMatchObject({ admitted: false, reason: "denied by *haiku*" });
|
|
136
|
+
expect(dropped.get("anthropic/claude-sonnet-5")).toMatchObject({ admitted: true });
|
|
137
|
+
// So is a pin naming no model.
|
|
138
|
+
expect(bySlug(catalogView({ models: MODELS, fetchedAtMs: 0, verdict: { filters, pin: "nobody/here" }, unserved: served })).get("anthropic/claude-sonnet-5")!.admitted).toBe(true);
|
|
139
|
+
});
|
|
140
|
+
});
|
|
141
|
+
|
|
142
|
+
describe("GET /v1/router/catalog", () => {
|
|
143
|
+
let handle: StartedServer;
|
|
144
|
+
let empty: StartedServer;
|
|
145
|
+
const dir = mkdtempSync(join(tmpdir(), "amr-catalog-"));
|
|
146
|
+
const FETCHED = Date.now() - 60_000;
|
|
147
|
+
beforeAll(() => {
|
|
148
|
+
const cfg = structuredClone(DEFAULT_CONFIG);
|
|
149
|
+
cfg.server.host = "127.0.0.1";
|
|
150
|
+
cfg.server.port = 0;
|
|
151
|
+
cfg.server.apiKey = "k";
|
|
152
|
+
cfg.logLevel = "silent";
|
|
153
|
+
// Nothing reaches the network: the catalog is served from its on-disk
|
|
154
|
+
// cache (within TTL), the periodic refresh is off, and the base URL is dead.
|
|
155
|
+
cfg.openrouter.apiKey = "sk-test";
|
|
156
|
+
cfg.openrouter.baseUrl = "http://127.0.0.1:9/api/v1";
|
|
157
|
+
cfg.openrouter.catalogRefreshMs = 0;
|
|
158
|
+
cfg.benchmarks.enabled = false;
|
|
159
|
+
cfg.upstreams = [vllm, azure];
|
|
160
|
+
cfg.ledger.path = join(dir, "router.db");
|
|
161
|
+
const db = openDb(cfg.ledger.path);
|
|
162
|
+
db.run("INSERT INTO catalog_cache (id, payload, fetched_at_ms, etag, key_scoped) VALUES (1, ?, ?, NULL, 0)", [JSON.stringify(FIXTURE.data), FETCHED]);
|
|
163
|
+
db.close();
|
|
164
|
+
handle = startServer(cfg);
|
|
165
|
+
const bare = structuredClone(cfg);
|
|
166
|
+
bare.upstreams = [];
|
|
167
|
+
bare.ledger.path = join(dir, "empty.db");
|
|
168
|
+
empty = startServer(bare);
|
|
169
|
+
});
|
|
170
|
+
afterAll(async () => {
|
|
171
|
+
await handle.stop();
|
|
172
|
+
await empty.stop();
|
|
173
|
+
try {
|
|
174
|
+
rmSync(dir, { recursive: true, force: true });
|
|
175
|
+
} catch {
|
|
176
|
+
/* Windows may hold the WAL briefly */
|
|
177
|
+
}
|
|
178
|
+
});
|
|
179
|
+
const get = (path: string, port = handle.server.port) => fetch(`http://127.0.0.1:${port}${path}`, { headers: { authorization: "Bearer k" } });
|
|
180
|
+
const view = async (path: string): Promise<Map<string, CatalogViewModel>> => bySlug((await (await get(path)).json()) as CatalogView);
|
|
181
|
+
|
|
182
|
+
test("the catalog as data: authenticated, sorted, every upstream's models, no verdict without a policy", async () => {
|
|
183
|
+
expect((await fetch(`http://127.0.0.1:${handle.server.port}/v1/router/catalog`)).status).toBe(401);
|
|
184
|
+
const res = await get("/v1/router/catalog");
|
|
185
|
+
expect(res.status).toBe(200);
|
|
186
|
+
const body = (await res.json()) as CatalogView;
|
|
187
|
+
expect(body.fetchedAtMs).toBe(FETCHED);
|
|
188
|
+
const slugs = body.models.map((m) => m.slug);
|
|
189
|
+
expect(slugs).toEqual([...slugs].sort());
|
|
190
|
+
// Every OpenRouter model the fixture normalises to, plus both named upstreams' — the disabled one too.
|
|
191
|
+
expect(slugs).toHaveLength(FIXTURE.data.filter((m) => normalizeCatalogModel(m) !== null).length + 2);
|
|
192
|
+
const models = bySlug(body);
|
|
193
|
+
const opus = models.get("anthropic/claude-opus-5")!;
|
|
194
|
+
const raw = normalizeCatalogModel(FIXTURE.data.find((m) => (m as { id: string }).id === "anthropic/claude-opus-5"))!;
|
|
195
|
+
expect(opus).toEqual({
|
|
196
|
+
slug: "anthropic/claude-opus-5",
|
|
197
|
+
canonicalSlug: raw.canonicalSlug,
|
|
198
|
+
name: raw.name,
|
|
199
|
+
provider: "openrouter",
|
|
200
|
+
vendor: "anthropic",
|
|
201
|
+
contextLength: raw.contextLength,
|
|
202
|
+
maxCompletionTokens: 128_000,
|
|
203
|
+
supportsTools: true,
|
|
204
|
+
supportsReasoning: raw.supportsReasoning,
|
|
205
|
+
reasoningMandatory: raw.reasoningMandatory,
|
|
206
|
+
inputModalities: raw.inputModalities,
|
|
207
|
+
price: { prompt: 5, completion: raw.price.completion * 1e6, cacheRead: 0.5, ...(raw.price.cacheWrite === undefined ? {} : { cacheWrite: raw.price.cacheWrite * 1e6 }) },
|
|
208
|
+
quality: raw.quality,
|
|
209
|
+
isFree: false,
|
|
210
|
+
});
|
|
211
|
+
expect(models.get("vllm/meta-llama/Llama-3")).toMatchObject({ provider: "vllm", vendor: "meta-llama", contextLength: 8_000 });
|
|
212
|
+
expect(models.get("azure-eu/gpt-4o-deploy")).toMatchObject({ provider: "azure-eu", vendor: "azure-eu", price: { prompt: 2.5, completion: 10, cacheRead: 1.25 } });
|
|
213
|
+
expect(body.models.some((m) => "admitted" in m || "reason" in m)).toBe(false);
|
|
214
|
+
});
|
|
215
|
+
|
|
216
|
+
test("judged under a policy: allow, deny, pin, the router's filters and the upstream's state", async () => {
|
|
217
|
+
const allowed = await view(`/v1/router/catalog?policy=${encodeURIComponent(JSON.stringify({ allow: ["anthropic/*", "azure-eu/*"], deny: ["anthropic/claude-opus-5"], maxTier: "simple" }))}`);
|
|
218
|
+
expect(allowed.get("anthropic/claude-sonnet-4.5")).toMatchObject({ admitted: true });
|
|
219
|
+
expect(allowed.get("anthropic/claude-opus-5")).toMatchObject({ admitted: false, reason: "denied by anthropic/claude-opus-5" });
|
|
220
|
+
expect(allowed.get("anthropic/claude-opus-5:batch")).toMatchObject({ admitted: false, reason: "built-in deny: floating alias, batch endpoint, stealth, or meta-router" });
|
|
221
|
+
expect(allowed.get("openai/gpt-5.6-luna")).toMatchObject({ admitted: false, reason: "not in the allow list" });
|
|
222
|
+
expect(allowed.get("azure-eu/gpt-4o-deploy")).toMatchObject({ admitted: false, reason: "upstream azure-eu is disabled" });
|
|
223
|
+
// The configured deny list stays in force under a policy's, as applyRequestPolicy adds rather than replaces.
|
|
224
|
+
const pinned = await view(`/v1/router/catalog?policy=${encodeURIComponent(JSON.stringify({ pin: "vllm/meta-llama/Llama-3" }))}`);
|
|
225
|
+
expect(pinned.get("vllm/meta-llama/Llama-3")).toMatchObject({ admitted: true });
|
|
226
|
+
expect(pinned.get("anthropic/claude-sonnet-4.5")).toMatchObject({ admitted: false, reason: "pinned to vllm/meta-llama/Llama-3" });
|
|
227
|
+
expect(pinned.get("tencent/hy-mt2-1.8b")).toMatchObject({ admitted: false, reason: "no tool support (filters.requireToolSupport)" });
|
|
228
|
+
expect(pinned.get("liquid/lfm-2.5-2.6b:free")).toMatchObject({ admitted: false, reason: "free models excluded (filters.includeFree)" });
|
|
229
|
+
// `{}` is a policy too: every model carries a verdict.
|
|
230
|
+
const plain = await view("/v1/router/catalog?policy=%7B%7D");
|
|
231
|
+
expect(plain.get("anthropic/claude-sonnet-4.5")).toMatchObject({ admitted: true });
|
|
232
|
+
expect([...plain.values()].every((m) => typeof m.admitted === "boolean")).toBe(true);
|
|
233
|
+
});
|
|
234
|
+
|
|
235
|
+
test("a malformed policy is a 400 in the wire error shape", async () => {
|
|
236
|
+
for (const bad of ["%7B", "not-json", "%5B%5D", "null", "1"]) {
|
|
237
|
+
const res = await get(`/v1/router/catalog?policy=${bad}`);
|
|
238
|
+
expect(res.status).toBe(400);
|
|
239
|
+
const body = (await res.json()) as { error: { message: string; type: string; code: string } };
|
|
240
|
+
expect(body.error.code).toBe("invalid_request_error");
|
|
241
|
+
expect(body.error.message).toContain("policy");
|
|
242
|
+
}
|
|
243
|
+
});
|
|
244
|
+
|
|
245
|
+
test("before the first fetch: empty, never a wait", async () => {
|
|
246
|
+
expect((await (await get("/v1/router/catalog", empty.server.port)).json()) as CatalogView).toEqual({ fetchedAtMs: 0, models: [] });
|
|
247
|
+
expect((await (await get("/v1/router/catalog?policy=%7B%7D", empty.server.port)).json()) as CatalogView).toEqual({ fetchedAtMs: 0, models: [] });
|
|
248
|
+
});
|
|
249
|
+
});
|
package/test/migrations.test.ts
CHANGED
|
@@ -8,6 +8,7 @@ import { createFeedbackStore } from "../src/cost/feedback.ts";
|
|
|
8
8
|
import { createLedger } from "../src/cost/ledger.ts";
|
|
9
9
|
import { buildUsageReport } from "../src/cost/report.ts";
|
|
10
10
|
import { buildDailySummary, createKv } from "../src/cost/summary.ts";
|
|
11
|
+
import { exportRows, spendUsdSince } from "../src/cost/views.ts";
|
|
11
12
|
import { createConversationStore } from "../src/router/state.ts";
|
|
12
13
|
import { openDb } from "../src/util/sqlite.ts";
|
|
13
14
|
|
|
@@ -23,7 +24,7 @@ import { openDb } from "../src/util/sqlite.ts";
|
|
|
23
24
|
|
|
24
25
|
const FIXTURES = join(import.meta.dir, "fixtures", "migrations");
|
|
25
26
|
const files = readdirSync(FIXTURES).filter((f) => /^router-v\d+\.db$/.test(f)).sort((a, b) => Number(/\d+/.exec(a)![0]) - Number(/\d+/.exec(b)![0]));
|
|
26
|
-
const CURRENT_VERSION =
|
|
27
|
+
const CURRENT_VERSION = 18;
|
|
27
28
|
|
|
28
29
|
describe("schema migrations from every shipped version", () => {
|
|
29
30
|
test("fixtures exist for the versions that shipped", () => {
|
|
@@ -43,7 +44,7 @@ describe("schema migrations from every shipped version", () => {
|
|
|
43
44
|
expect((db.query("PRAGMA user_version").get() as { user_version: number }).user_version).toBe(CURRENT_VERSION);
|
|
44
45
|
// Every column the current code writes exists after migration.
|
|
45
46
|
const ledgerCols = new Set((db.query("PRAGMA table_info(ledger)").all() as { name: string }[]).map((c) => c.name));
|
|
46
|
-
for (const c of ["harness_id", "error_kind", "omp_session_id", "features", "explored_from", "hold_arm", "prompt_tokens_saved"]) expect(ledgerCols.has(c)).toBe(true);
|
|
47
|
+
for (const c of ["harness_id", "error_kind", "omp_session_id", "features", "explored_from", "hold_arm", "prompt_tokens_saved", "scope"]) expect(ledgerCols.has(c)).toBe(true);
|
|
47
48
|
const convCols = new Set((db.query("PRAGMA table_info(conversations)").all() as { name: string }[]).map((c) => c.name));
|
|
48
49
|
for (const c of ["context_version", "compaction_plan", "compaction_plan_tokens", "upgrade_deferred_tier"]) expect(convCols.has(c)).toBe(true);
|
|
49
50
|
// The fixture's ledger row survived the ALTERs with its values.
|
|
@@ -61,6 +62,11 @@ describe("schema migrations from every shipped version", () => {
|
|
|
61
62
|
expect(conversations.load("fixture-key").key).toBe("fixture-key");
|
|
62
63
|
expect(buildUsageReport(db, { windowDays: 3650 }).totals.dispatches).toBe(1);
|
|
63
64
|
expect(buildDailySummary(db, {}).current.dispatches).toBe(0);
|
|
65
|
+
// v18: the fixture's row predates `scope`, so it exports under "" and no
|
|
66
|
+
// context scope claims its spend.
|
|
67
|
+
expect(exportRows(db, 0, null).map((r) => r.scope)).toEqual([""]);
|
|
68
|
+
expect(spendUsdSince(db, 0, null, "acme.api")).toBe(0);
|
|
69
|
+
expect(spendUsdSince(db, 0, null)).toBeGreaterThanOrEqual(0);
|
|
64
70
|
expect(ledger.prune?.(0)).toBe(0);
|
|
65
71
|
} finally {
|
|
66
72
|
db.close();
|
|
@@ -77,6 +83,8 @@ describe("schema migrations from every shipped version", () => {
|
|
|
77
83
|
const db = openDb(":memory:");
|
|
78
84
|
try {
|
|
79
85
|
expect((db.query("PRAGMA user_version").get() as { user_version: number }).user_version).toBe(CURRENT_VERSION);
|
|
86
|
+
// A fresh ledger has the scope column the bootstrap never spells out in CREATE TABLE.
|
|
87
|
+
expect((db.query("PRAGMA table_info(ledger)").all() as { name: string }[]).some((c) => c.name === "scope")).toBe(true);
|
|
80
88
|
} finally {
|
|
81
89
|
db.close();
|
|
82
90
|
}
|
|
@@ -259,11 +259,11 @@ describe("v4 migration", () => {
|
|
|
259
259
|
db.close();
|
|
260
260
|
});
|
|
261
261
|
|
|
262
|
-
test("schema is at user_version
|
|
262
|
+
test("schema is at user_version 18", () => {
|
|
263
263
|
const db = openDb(":memory:");
|
|
264
264
|
try {
|
|
265
265
|
const row = db.query("PRAGMA user_version").get() as { user_version: number };
|
|
266
|
-
expect(row.user_version).toBe(
|
|
266
|
+
expect(row.user_version).toBe(18);
|
|
267
267
|
} finally {
|
|
268
268
|
db.close();
|
|
269
269
|
}
|
package/test/turn.test.ts
CHANGED
|
@@ -545,6 +545,29 @@ describe("runTurn", () => {
|
|
|
545
545
|
});
|
|
546
546
|
});
|
|
547
547
|
|
|
548
|
+
describe("the context scope reaches the ledger", () => {
|
|
549
|
+
const run = async (req: NormRequest, config: RouterConfig) => {
|
|
550
|
+
const { router } = mkRouter([mkDecision("trivial", "cheap/model", { escalateTo: "simple" })]);
|
|
551
|
+
const { upstream } = mkUpstream([{ kind: "chunks", chunks: [startChunk("cheap/model"), textChunk("hi"), finishChunk("stop"), usageChunk({ promptTokens: 10, completionTokens: 2 }, 0.0001)] }]);
|
|
552
|
+
const { ledger, entries } = mkLedger();
|
|
553
|
+
const { store } = mkConversations();
|
|
554
|
+
const { sink } = mkSink();
|
|
555
|
+
await runTurn(req, sink, { config, router, upstream, ledger, conversations: store, catalog, context: createDisabledBridge() }, new AbortController().signal);
|
|
556
|
+
return entries;
|
|
557
|
+
};
|
|
558
|
+
|
|
559
|
+
test("a turn's scope is recorded, header first and the configured default behind it", async () => {
|
|
560
|
+
// The header the team front door sets: the row can be charged to that project.
|
|
561
|
+
expect((await run({ ...mkReq(), agentdoxScope: "acme.api" }, mkConfig()))[0]!.scope).toBe("acme.api");
|
|
562
|
+
// No header: the resolved scope is the configured default, which is what the bridge would have used.
|
|
563
|
+
const base = mkConfig();
|
|
564
|
+
const withDefault: RouterConfig = { ...base, context: { ...base.context, defaultScope: "solo" } };
|
|
565
|
+
expect((await run(mkReq(), withDefault))[0]!.scope).toBe("solo");
|
|
566
|
+
// Neither: no scope at all, which the ledger stores as NULL.
|
|
567
|
+
expect((await run(mkReq(), mkConfig()))[0]!.scope).toBe("");
|
|
568
|
+
});
|
|
569
|
+
});
|
|
570
|
+
|
|
548
571
|
describe("exploration reaches the ledger", () => {
|
|
549
572
|
test("an explored turn records the tier it was dropped from", async () => {
|
|
550
573
|
const explored = { ...mkDecision("simple", "cheap/model"), explored: { from: "moderate" as Tier, to: "simple" as Tier } };
|
package/test/views.test.ts
CHANGED
|
@@ -64,9 +64,9 @@ function seeded() {
|
|
|
64
64
|
const db = openDb(":memory:");
|
|
65
65
|
const ledger = createLedger(db, cfg);
|
|
66
66
|
const feedback = createFeedbackStore(db);
|
|
67
|
-
ledger.record(entry({ id: "l1", harnessId: "u_ada", slug: "anthropic/claude-sonnet-5", servedSlug: "anthropic/claude-sonnet-5", predictedUsd: 0.01, reportedUsd: 0.012 }));
|
|
68
|
-
ledger.record(entry({ id: "l2", harnessId: "u_ada", slug: "anthropic/claude-sonnet-5", servedSlug: null, predictedUsd: 0.01, reportedUsd: null, escalationSignal: "circular" }));
|
|
69
|
-
ledger.record(entry({ id: "l3", harnessId: "u_bob", slug: "ollama/glm-5.3-flash", servedSlug: "ollama/glm-5.3-flash", predictedUsd: 0.001, reportedUsd: 0.001, error: "boom" }));
|
|
67
|
+
ledger.record(entry({ id: "l1", harnessId: "u_ada", scope: "acme.api", slug: "anthropic/claude-sonnet-5", servedSlug: "anthropic/claude-sonnet-5", predictedUsd: 0.01, reportedUsd: 0.012 }));
|
|
68
|
+
ledger.record(entry({ id: "l2", harnessId: "u_ada", scope: "acme.web", slug: "anthropic/claude-sonnet-5", servedSlug: null, predictedUsd: 0.01, reportedUsd: null, escalationSignal: "circular" }));
|
|
69
|
+
ledger.record(entry({ id: "l3", harnessId: "u_bob", scope: "acme.api", slug: "ollama/glm-5.3-flash", servedSlug: "ollama/glm-5.3-flash", predictedUsd: 0.001, reportedUsd: 0.001, error: "boom" }));
|
|
70
70
|
ledger.record(entry({ id: "l4", harnessId: "u_bob", requestedModel: "digest", slug: "ollama/glm-5.3-flash", servedSlug: "ollama/glm-5.3-flash", predictedUsd: 0.5, reportedUsd: 0.5 }));
|
|
71
71
|
ledger.record(entry({ id: "l5", harnessId: "u_bob", createdAtMs: NOW - 40 * DAY, slug: "ollama/glm-5.3-flash", predictedUsd: 5, reportedUsd: 5 }));
|
|
72
72
|
feedback.record({ ledgerId: "l1", ompSessionId: "s", slug: "anthropic/claude-sonnet-5", tier: "simple", verdict: "good", note: "" }, NOW - 1000);
|
|
@@ -87,6 +87,17 @@ describe("ledger views", () => {
|
|
|
87
87
|
expect(spendUsdSince(db, since, [])).toBe(0);
|
|
88
88
|
});
|
|
89
89
|
|
|
90
|
+
test("spend narrowed to one context scope, so a front door charges a project", () => {
|
|
91
|
+
// l1 (0.012, u_ada) and l3 (0.001, u_bob) carried acme.api; l2 (0.01 predicted) carried acme.web.
|
|
92
|
+
expect(spendUsdSince(db, since, null, "acme.api")).toBeCloseTo(0.013, 6);
|
|
93
|
+
expect(spendUsdSince(db, since, null, "acme.web")).toBeCloseTo(0.01, 6);
|
|
94
|
+
expect(spendUsdSince(db, since, ["u_ada"], "acme.api")).toBeCloseTo(0.012, 6); // harness and scope compose
|
|
95
|
+
expect(spendUsdSince(db, since, ["u_bob"], "acme.web")).toBe(0);
|
|
96
|
+
expect(spendUsdSince(db, since, null, "nope")).toBe(0);
|
|
97
|
+
expect(spendUsdSince(db, since, null, "")).toBeCloseTo(0.523, 6); // no scope given: every turn, scoped or not
|
|
98
|
+
expect(spendUsdSince(db, since, [], "acme.api")).toBe(0);
|
|
99
|
+
});
|
|
100
|
+
|
|
90
101
|
test("feedback by model with distinct judges, scoped by harness", () => {
|
|
91
102
|
const all = feedbackView(db, since, null);
|
|
92
103
|
expect(all.byModel).toEqual([
|
|
@@ -102,17 +113,22 @@ describe("ledger views", () => {
|
|
|
102
113
|
expect(feedbackView(db, since, []).recent).toEqual([]);
|
|
103
114
|
});
|
|
104
115
|
|
|
105
|
-
test("export rows by day, harness
|
|
116
|
+
test("export rows by day, harness, served model and scope; digest and old rows out; CSV quoting", () => {
|
|
106
117
|
const rows = exportRows(db, since, null);
|
|
107
|
-
|
|
108
|
-
expect(rows
|
|
109
|
-
expect(rows[0]
|
|
110
|
-
expect(rows[
|
|
118
|
+
// u_ada's two turns are one model on one day but two projects, so they no longer share a row.
|
|
119
|
+
expect(rows).toHaveLength(3);
|
|
120
|
+
expect(rows[0]).toMatchObject({ day: "2026-09-07", harnessId: "u_ada", slug: "anthropic/claude-sonnet-5", scope: "acme.api", provider: "openrouter", dispatches: 1, promptTokens: 1000, cachedTokens: 400, completionTokens: 50, escalations: 0, errors: 0 });
|
|
121
|
+
expect(rows[0]!.spendUsd).toBeCloseTo(0.012, 6);
|
|
122
|
+
expect(rows[1]).toMatchObject({ harnessId: "u_ada", scope: "acme.web", dispatches: 1, escalations: 1 });
|
|
123
|
+
expect(rows[1]!.spendUsd).toBeCloseTo(0.01, 6);
|
|
124
|
+
expect(rows[2]).toMatchObject({ harnessId: "u_bob", scope: "acme.api", provider: "ollama", dispatches: 1, errors: 1 });
|
|
111
125
|
expect(exportRows(db, since, ["u_bob"])).toHaveLength(1);
|
|
112
126
|
expect(exportRows(db, since, [])).toEqual([]);
|
|
127
|
+
// A turn that carried no scope groups under "": what every row written before v18 does.
|
|
128
|
+
expect(exportRows(db, NOW - 60 * DAY, ["u_bob"]).map((r) => r.scope).sort()).toEqual(["", "acme.api"]);
|
|
113
129
|
const csv = exportCsv([{ ...rows[0]!, harnessId: 'ada, "L"' }]);
|
|
114
|
-
expect(csv.split("\n")[0]).toBe("day,harness,model,provider,dispatches,prompt_tokens,cached_tokens,completion_tokens,spend_usd,escalations,errors");
|
|
115
|
-
expect(csv.split("\n")[1]).toBe('2026-09-07,"ada, ""L""",anthropic/claude-sonnet-5,openrouter,
|
|
130
|
+
expect(csv.split("\n")[0]).toBe("day,harness,model,provider,dispatches,prompt_tokens,cached_tokens,completion_tokens,spend_usd,escalations,errors,scope");
|
|
131
|
+
expect(csv.split("\n")[1]).toBe('2026-09-07,"ada, ""L""",anthropic/claude-sonnet-5,openrouter,1,1000,400,50,0.012000,0,0,acme.api');
|
|
116
132
|
expect(harnessScopeParam(null)).toBeNull();
|
|
117
133
|
expect(harnessScopeParam(" , ")).toBeNull();
|
|
118
134
|
expect(harnessScopeParam("a, b")).toEqual(["a", "b"]);
|
|
@@ -130,7 +146,7 @@ describe("view routes", () => {
|
|
|
130
146
|
cfg.ledger.path = join(dir, "router.db");
|
|
131
147
|
// Seed through the ledger on the same file before the server opens it.
|
|
132
148
|
const db = openDb(cfg.ledger.path);
|
|
133
|
-
createLedger(db, cfg).record(entry({ id: "r1", createdAtMs: Date.now() - 1000, harnessId: "u_x", predictedUsd: 0.2, reportedUsd: 0.25 }));
|
|
149
|
+
createLedger(db, cfg).record(entry({ id: "r1", createdAtMs: Date.now() - 1000, harnessId: "u_x", scope: "acme.api", predictedUsd: 0.2, reportedUsd: 0.25 }));
|
|
134
150
|
db.close();
|
|
135
151
|
handle = startServer(cfg);
|
|
136
152
|
});
|
|
@@ -155,14 +171,21 @@ describe("view routes", () => {
|
|
|
155
171
|
expect((((await (await get("/v1/router/decisions?harness=u_other&days=1")).json()) as { entries: unknown[] }).entries)).toEqual([]);
|
|
156
172
|
expect((((await (await get("/v1/router/decisions?limit=1")).json()) as { entries: { id: string }[] }).entries.map((e) => e.id))).toEqual(["r1"]); // no filter: everything, as before
|
|
157
173
|
expect(((await (await get(`/v1/router/spend?sinceMs=${Date.now() - DAY}&harness=u_other`)).json()) as { usd: number }).usd).toBe(0);
|
|
174
|
+
// ?scope= charges one project: the row carried acme.api, so acme.web sees nothing.
|
|
175
|
+
const scoped = (await (await get(`/v1/router/spend?sinceMs=${Date.now() - DAY}&scope=acme.api`)).json()) as { usd: number; scope: string };
|
|
176
|
+
expect(scoped.usd).toBeCloseTo(0.25, 6);
|
|
177
|
+
expect(scoped.scope).toBe("acme.api");
|
|
178
|
+
expect(((await (await get(`/v1/router/spend?sinceMs=${Date.now() - DAY}&scope=acme.web`)).json()) as { usd: number }).usd).toBe(0);
|
|
179
|
+
expect(((await (await get(`/v1/router/spend?sinceMs=${Date.now() - DAY}&harness=u_x&scope=acme.api`)).json()) as { usd: number }).usd).toBeCloseTo(0.25, 6);
|
|
158
180
|
const fb = (await (await get("/v1/router/feedback?days=7")).json()) as { days: number; byModel: unknown[]; recent: unknown[] };
|
|
159
181
|
expect(fb).toEqual({ days: 7, byModel: [], recent: [] });
|
|
160
182
|
const csv = await get("/v1/router/export?days=1");
|
|
161
183
|
expect(csv.headers.get("content-type")).toContain("text/csv");
|
|
162
|
-
expect((await csv.text()).split("\n")[1]).toContain("u_x,vendor/model,openrouter,1,1000,400,50,0.250000,0,0");
|
|
163
|
-
const js = (await (await get("/v1/router/export?days=1&format=json&harness=u_x")).json()) as { days: number; rows: { harnessId: string }[] };
|
|
184
|
+
expect((await csv.text()).split("\n")[1]).toContain("u_x,vendor/model,openrouter,1,1000,400,50,0.250000,0,0,acme.api");
|
|
185
|
+
const js = (await (await get("/v1/router/export?days=1&format=json&harness=u_x")).json()) as { days: number; rows: { harnessId: string; scope: string }[] };
|
|
164
186
|
expect(js.days).toBe(1);
|
|
165
187
|
expect(js.rows[0]?.harnessId).toBe("u_x");
|
|
188
|
+
expect(js.rows[0]?.scope).toBe("acme.api");
|
|
166
189
|
});
|
|
167
190
|
});
|
|
168
191
|
|
|
@@ -176,6 +199,9 @@ describe("decision entries", () => {
|
|
|
176
199
|
expect(ada.map((e) => e.id)).toEqual(["l1", "l2"]); // same instant in the fixture; insertion order within it is stable
|
|
177
200
|
expect(ada.find((e) => e.id === "l1")?.feedback).toEqual([{ verdict: "good", note: "", createdAtMs: NOW - 1000 }]);
|
|
178
201
|
expect(ada.find((e) => e.id === "l2")?.escalationSignal).toBe("circular");
|
|
202
|
+
// The context scope is a ledger column now, so it rides along on every entry.
|
|
203
|
+
expect(ada.find((e) => e.id === "l1")?.scope).toBe("acme.api");
|
|
204
|
+
expect(ada.find((e) => e.id === "l2")?.scope).toBe("acme.web");
|
|
179
205
|
// Everyone, within the window: the 40-day-old row stays out; the digest row is a turn like any other.
|
|
180
206
|
expect(decisionEntries(db, { sinceMs: since, harness: null }).map((e) => e.id).sort()).toEqual(["l1", "l2", "l3", "l4"]);
|
|
181
207
|
expect(decisionEntries(db, { sinceMs: 0, harness: null }).length).toBe(5);
|