auto-model-router 0.2.10 → 0.2.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.omp-plugin/marketplace.json +2 -2
- package/README.md +9 -1
- package/docs/AGENTDOX-BRIDGE.md +50 -10
- package/omp-extension/embed-logic.ts +13 -3
- package/package.json +1 -1
- package/src/router/state.ts +29 -6
- package/src/router/types.ts +14 -0
- package/src/server/turn.ts +25 -4
- package/test/embed-logic.test.ts +10 -3
- package/test/failover.test.ts +1 -0
- package/test/state.test.ts +90 -0
- package/test/turn.test.ts +94 -4
|
@@ -7,14 +7,14 @@
|
|
|
7
7
|
},
|
|
8
8
|
"metadata": {
|
|
9
9
|
"description": "auto-model-router: a local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter",
|
|
10
|
-
"version": "0.2.
|
|
10
|
+
"version": "0.2.12",
|
|
11
11
|
"pluginRoot": "."
|
|
12
12
|
},
|
|
13
13
|
"plugins": [
|
|
14
14
|
{
|
|
15
15
|
"name": "auto-model-router",
|
|
16
16
|
"description": "Local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter. Runs in-process, routes per turn by price and task complexity, with budget caps, mid-stream escalation, and cache-aware hysteresis.",
|
|
17
|
-
"version": "0.2.
|
|
17
|
+
"version": "0.2.12",
|
|
18
18
|
"author": {
|
|
19
19
|
"name": "drewappling",
|
|
20
20
|
"email": "drewappling@gmail.com"
|
package/README.md
CHANGED
|
@@ -580,11 +580,19 @@ and brief:
|
|
|
580
580
|
```bash
|
|
581
581
|
export AGENTDOX_URL=http://localhost:3003
|
|
582
582
|
export AGENTDOX_TOKEN=<pat with read+write on the scope>
|
|
583
|
-
export AGENTDOX_SCOPE=ashlands #
|
|
583
|
+
export AGENTDOX_SCOPE=ashlands # fallback only; see below
|
|
584
584
|
```
|
|
585
585
|
|
|
586
586
|
Setting a URL and a token is enough to turn it on.
|
|
587
587
|
|
|
588
|
+
The scope is **derived per workspace** from the directory basename
|
|
589
|
+
(`E:/projects/ashlands` → `ashlands`), and that derivation wins. `AGENTDOX_SCOPE` /
|
|
590
|
+
`context.defaultScope` is only a fallback for workspaces it cannot resolve, because one router
|
|
591
|
+
install serves every project on the machine — a slug pinned there would be sent for all of
|
|
592
|
+
them, injecting one project's context into another's work. A single configured token also
|
|
593
|
+
grants only the scopes it was minted for; for any other project the bridge degrades to inert
|
|
594
|
+
rather than writing somewhere wrong.
|
|
595
|
+
|
|
588
596
|
### It does not cost you a cache miss per turn
|
|
589
597
|
|
|
590
598
|
The context block sits at the front of the prompt, so re-fetching it every turn
|
package/docs/AGENTDOX-BRIDGE.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# agentdox bridge — handoff
|
|
2
2
|
|
|
3
|
-
**Status:** implemented, typechecks clean,
|
|
4
|
-
through omp. The write-back
|
|
3
|
+
**Status:** implemented, typechecks clean, 439 tests pass, injection verified end-to-end
|
|
4
|
+
through omp. The write-back faults in §5 and §6 are **fixed**; `context.recordTurns` is on.
|
|
5
5
|
|
|
6
6
|
Design rationale (why it is built this way):
|
|
7
7
|
`E:/projects/agentdox/docs/architecture/router-context-bridge.md`.
|
|
@@ -110,8 +110,8 @@ Each tool round-trip is its own dispatch, finishing with `tool_calls` and emitti
|
|
|
110
110
|
record of a fragment rather than a truncated answer. Only the final `stop` dispatch carries
|
|
111
111
|
the synthesis. `recordTurn` fired on all ~13, and the last writer won.
|
|
112
112
|
|
|
113
|
-
The same root cause explains the §
|
|
114
|
-
message, which does **not** move while a tool loop runs, so the identical user text was
|
|
113
|
+
The same root cause explains half the duplication in §7: `lastUserText` walks back to the last
|
|
114
|
+
`user` message, which does **not** move while a tool loop runs, so the identical user text was
|
|
115
115
|
appended once per round-trip too.
|
|
116
116
|
|
|
117
117
|
**Fix.** `TurnRecord` gained `turnEnded` (`finishReason !== "tool_calls"`, set in
|
|
@@ -128,13 +128,53 @@ nothing; interleaved conversations buffer independently) and `test/turn.test.ts`
|
|
|
128
128
|
behavior. `tools/agentdox-e2e.ts` step 6 proves it against a live server: four dispatches →
|
|
129
129
|
exactly one user and one assistant message.
|
|
130
130
|
|
|
131
|
-
## 6.
|
|
131
|
+
## 6. FIXED — harness utility calls, and the scope that leaked across projects
|
|
132
132
|
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
133
|
+
Two further faults surfaced the moment `recordTurns` was first switched on, both found by
|
|
134
|
+
reading what actually landed in agentdox.
|
|
135
|
+
|
|
136
|
+
**Utility calls were recorded as turns.** omp drives more than the agent through this
|
|
137
|
+
provider: it asks for a conversation title and a complexity rating, with `model: auto`, over
|
|
138
|
+
the same embedded router. Those answer *about* a conversation rather than participating in
|
|
139
|
+
one, and they finish with `stop`, so `turnEnded` alone does not exclude them. Three junk
|
|
140
|
+
sessions appeared immediately:
|
|
141
|
+
|
|
142
|
+
| recorded assistant text | what it really was |
|
|
143
|
+
| --- | --- |
|
|
144
|
+
| `high` | omp's complexity rating — **this is the original `" high"`** |
|
|
145
|
+
| `<title>Read memory and resume work</title>` | omp's title generation |
|
|
146
|
+
| `<title>Resume settlement 2D slice 3 streaming</title>` | omp's title generation |
|
|
147
|
+
|
|
148
|
+
The discriminator is the tool array: an agent always ships its tool schemas (`toolCount` 12,
|
|
149
|
+
prompts of 60k–90k), while utility calls ship none (`toolCount` 0, prompts of 222–841,
|
|
150
|
+
`task=chat`). `src/server/turn.ts` therefore records only when `req.tools.length > 0`. A
|
|
151
|
+
deliberately tool-less session is not transcribed — silence beats garbage, because every junk
|
|
152
|
+
record is re-injected into every later turn.
|
|
153
|
+
|
|
154
|
+
**`defaultScope` leaked one project's slug to all of them.** `omp-extension/embed-logic.ts`
|
|
155
|
+
resolved the header as `defaultScope !== "" ? defaultScope : derive(cwd)`, so a *scope-agnostic
|
|
156
|
+
global* overrode the *per-workspace* derivation. One router install serves every workspace, so
|
|
157
|
+
with `defaultScope: omp-router` set, an **ashlands** session shipped
|
|
158
|
+
`X-Agentdox-Scope: omp-router`: it injected omp-router's context into ashlands work and filed
|
|
159
|
+
ashlands turns under omp-router. The server always treated the field as a fallback ("the
|
|
160
|
+
configured default covers harnesses that send none"), so the two sides disagreed about the same
|
|
161
|
+
field. Now the workspace derivation wins and `defaultScope` is its fallback, matching the name
|
|
162
|
+
and the server. Same failure class as the `.mcp.json` lesson: a scope-specific value must never
|
|
163
|
+
live in a scope-agnostic file.
|
|
164
|
+
|
|
165
|
+
One consequence worth knowing: `context.token` is a single PAT, but a machine-wide router
|
|
166
|
+
serves N scopes. The omp-router PAT gets `403 no read access to scope "ashlands"`, so with the
|
|
167
|
+
scope now correct the bridge degrades to **inert** for other projects. Correct and safe, but it
|
|
168
|
+
means the bridge only helps projects the configured token actually grants. A multi-scope token
|
|
169
|
+
would fix that, at the cost of one credential reaching every project.
|
|
170
|
+
|
|
171
|
+
## 7. Also worth doing
|
|
172
|
+
|
|
173
|
+
- **Context pollution from test turns.** `context_assemble` includes recent session messages,
|
|
174
|
+
so router test turns feed back into the next block. The omp-router scope had accumulated 19
|
|
175
|
+
sessions of which 18 were noise (`hi`, `say hello`, `Reply with exactly the word: PONG`,
|
|
176
|
+
injection probes, `bridge e2e …`); they were deleted, and `tools/agentdox-e2e.ts` writes two
|
|
177
|
+
more every run. Consider a `sessionLimit` override for the bridge, or excluding
|
|
138
178
|
router-authored sessions.
|
|
139
179
|
- **`context.timeoutMs` is 3000ms** and failures degrade silently at `debug` level by design.
|
|
140
180
|
If agentdox is cold this can no-op invisibly. Consider logging the first failure at `warn`.
|
|
@@ -65,8 +65,9 @@ export interface EmbedConfig {
|
|
|
65
65
|
*
|
|
66
66
|
* The workspace basename is the one identifier that is already stable, already
|
|
67
67
|
* per-project, and requires no configuration — the same convention agentdox's
|
|
68
|
-
* own `project_ensure` slugs follow.
|
|
69
|
-
*
|
|
68
|
+
* own `project_ensure` slugs follow. It WINS over `context.defaultScope`,
|
|
69
|
+
* which is a fallback for workspaces it cannot resolve; see
|
|
70
|
+
* `buildProviderConfig`.
|
|
70
71
|
*/
|
|
71
72
|
export function deriveAgentdoxScope(cwd: string): string {
|
|
72
73
|
// Both separators: omp reports a Windows cwd with backslashes.
|
|
@@ -165,7 +166,16 @@ export function buildProviderConfig(
|
|
|
165
166
|
out.harnessId = cfg.server.harnessId;
|
|
166
167
|
}
|
|
167
168
|
if (cfg.context?.enabled === true) {
|
|
168
|
-
|
|
169
|
+
// The WORKSPACE wins. `defaultScope` is a scope-agnostic global — one
|
|
170
|
+
// router install serves every project on the machine — so letting it
|
|
171
|
+
// override the per-workspace derivation sends one project's slug for all
|
|
172
|
+
// of them: an ashlands session shipped `X-Agentdox-Scope: omp-router`,
|
|
173
|
+
// which both injected the wrong project's context and filed its turns
|
|
174
|
+
// under the wrong scope. The server treats this field as a fallback too
|
|
175
|
+
// ("the configured default covers harnesses that send none"), so the two
|
|
176
|
+
// sides now agree: most specific signal first.
|
|
177
|
+
const derived = deriveAgentdoxScope(cwd ?? "");
|
|
178
|
+
const scope = derived !== "" ? derived : cfg.context.defaultScope;
|
|
169
179
|
if (scope !== "") out.agentdoxScope = scope;
|
|
170
180
|
}
|
|
171
181
|
return out;
|
package/package.json
CHANGED
package/src/router/state.ts
CHANGED
|
@@ -57,13 +57,17 @@ export function createConversationStore(db: Database): ConversationStore {
|
|
|
57
57
|
const insertOne: Statement<unknown, [string, string, number]> = db.query(
|
|
58
58
|
"INSERT INTO conversations (key, session_id, updated_at_ms) VALUES (?, ?, ?)",
|
|
59
59
|
);
|
|
60
|
+
// `spent_usd` and `escalations` are ABSENT from this statement on purpose.
|
|
61
|
+
// They accumulate through `accrueOne` below, so writing a turn-start snapshot
|
|
62
|
+
// back here would erase whatever a billed-but-uncommitted dispatch added.
|
|
63
|
+
// The schema defaults both to 0, so the INSERT arm still works.
|
|
60
64
|
const upsert = db.query(`
|
|
61
65
|
INSERT INTO conversations (
|
|
62
66
|
key, session_id, turn, current_slug, current_tier, sticky_until_turn,
|
|
63
|
-
|
|
67
|
+
last_prompt_tokens, cache_warm_slug, cache_warm_at_ms,
|
|
64
68
|
context_version, context_fetched_at_ms, updated_at_ms
|
|
65
69
|
) VALUES ($key, $sessionId, $turn, $currentSlug, $currentTier, $stickyUntilTurn,
|
|
66
|
-
$
|
|
70
|
+
$lastPromptTokens, $cacheWarmSlug, $cacheWarmAtMs,
|
|
67
71
|
$contextVersion, $contextFetchedAtMs, $updatedAtMs)
|
|
68
72
|
ON CONFLICT(key) DO UPDATE SET
|
|
69
73
|
session_id = excluded.session_id,
|
|
@@ -71,8 +75,6 @@ export function createConversationStore(db: Database): ConversationStore {
|
|
|
71
75
|
current_slug = excluded.current_slug,
|
|
72
76
|
current_tier = excluded.current_tier,
|
|
73
77
|
sticky_until_turn = excluded.sticky_until_turn,
|
|
74
|
-
escalations = excluded.escalations,
|
|
75
|
-
spent_usd = excluded.spent_usd,
|
|
76
78
|
last_prompt_tokens = excluded.last_prompt_tokens,
|
|
77
79
|
cache_warm_slug = excluded.cache_warm_slug,
|
|
78
80
|
cache_warm_at_ms = excluded.cache_warm_at_ms,
|
|
@@ -80,6 +82,19 @@ export function createConversationStore(db: Database): ConversationStore {
|
|
|
80
82
|
context_fetched_at_ms = excluded.context_fetched_at_ms,
|
|
81
83
|
updated_at_ms = excluded.updated_at_ms
|
|
82
84
|
`);
|
|
85
|
+
// Read-modify-write in JS lost money: an aborted or failed dispatch is still
|
|
86
|
+
// billed by the upstream, but it returns before the commit path, so the next
|
|
87
|
+
// dispatch loaded a stale total and overwrote it. Measured on live data:
|
|
88
|
+
// 152 aborted dispatches billing $0.9985 — 30% of all spend — never reached
|
|
89
|
+
// `spent_usd`, leaving the per-conversation budget guard blind to it.
|
|
90
|
+
// Accumulating in SQL is correct regardless of who raced whom.
|
|
91
|
+
const accrueOne = db.query(`
|
|
92
|
+
UPDATE conversations
|
|
93
|
+
SET spent_usd = spent_usd + $spentUsd,
|
|
94
|
+
escalations = escalations + $escalations,
|
|
95
|
+
updated_at_ms = $updatedAtMs
|
|
96
|
+
WHERE key = $key
|
|
97
|
+
`);
|
|
83
98
|
const deleteStale: Statement<unknown, [number]> = db.query("DELETE FROM conversations WHERE updated_at_ms < ?");
|
|
84
99
|
|
|
85
100
|
return {
|
|
@@ -103,6 +118,7 @@ export function createConversationStore(db: Database): ConversationStore {
|
|
|
103
118
|
save(state) {
|
|
104
119
|
// bun:sqlite matches named parameters by their literal `$name` key;
|
|
105
120
|
// bare keys bind nothing at all and every column silently lands NULL.
|
|
121
|
+
// No $spentUsd / $escalations here — see the statement above.
|
|
106
122
|
upsert.run({
|
|
107
123
|
$key: state.key,
|
|
108
124
|
$sessionId: state.sessionId,
|
|
@@ -110,8 +126,6 @@ export function createConversationStore(db: Database): ConversationStore {
|
|
|
110
126
|
$currentSlug: state.currentSlug,
|
|
111
127
|
$currentTier: state.currentTier,
|
|
112
128
|
$stickyUntilTurn: state.stickyUntilTurn,
|
|
113
|
-
$escalations: state.escalations,
|
|
114
|
-
$spentUsd: state.spentUsd,
|
|
115
129
|
$lastPromptTokens: state.lastPromptTokens,
|
|
116
130
|
$cacheWarmSlug: state.cacheWarmSlug,
|
|
117
131
|
$cacheWarmAtMs: state.cacheWarmAtMs,
|
|
@@ -121,6 +135,15 @@ export function createConversationStore(db: Database): ConversationStore {
|
|
|
121
135
|
});
|
|
122
136
|
},
|
|
123
137
|
|
|
138
|
+
accrue(key, delta) {
|
|
139
|
+
const spentUsd = delta.spentUsd ?? 0;
|
|
140
|
+
const escalations = delta.escalations ?? 0;
|
|
141
|
+
// Nothing to add: skip the write rather than bump updated_at_ms and
|
|
142
|
+
// keep a dead conversation alive against `prune`.
|
|
143
|
+
if (spentUsd === 0 && escalations === 0) return;
|
|
144
|
+
accrueOne.run({ $key: key, $spentUsd: spentUsd, $escalations: escalations, $updatedAtMs: Date.now() });
|
|
145
|
+
},
|
|
146
|
+
|
|
124
147
|
prune(maxAgeMs) {
|
|
125
148
|
return deleteStale.run(Date.now() - maxAgeMs).changes;
|
|
126
149
|
},
|
package/src/router/types.ts
CHANGED
|
@@ -179,7 +179,21 @@ export interface ConversationStore {
|
|
|
179
179
|
get(key: string): ConversationState | null;
|
|
180
180
|
/** Loads existing state or creates a fresh record. */
|
|
181
181
|
load(key: string): ConversationState;
|
|
182
|
+
/**
|
|
183
|
+
* Persists the latest-wins fields. Deliberately does NOT write `spentUsd` or
|
|
184
|
+
* `escalations` — those accumulate via `accrue`, and writing back a snapshot
|
|
185
|
+
* here would clobber what a concurrent or already-billed dispatch added.
|
|
186
|
+
*/
|
|
182
187
|
save(state: ConversationState): void;
|
|
188
|
+
/**
|
|
189
|
+
* Adds to the persisted counters, atomically in SQL.
|
|
190
|
+
*
|
|
191
|
+
* Separate from `save` because a dispatch that never reaches the commit path
|
|
192
|
+
* — a client abort, an upstream error — was still BILLED, and its cost must
|
|
193
|
+
* reach the per-conversation budget guard anyway. Requires `load` to have
|
|
194
|
+
* created the row.
|
|
195
|
+
*/
|
|
196
|
+
accrue(key: string, delta: { spentUsd?: number; escalations?: number }): void;
|
|
183
197
|
/** Drops records untouched for longer than `maxAgeMs`. */
|
|
184
198
|
prune(maxAgeMs: number): number;
|
|
185
199
|
}
|
package/src/server/turn.ts
CHANGED
|
@@ -230,6 +230,13 @@ export async function runTurn(
|
|
|
230
230
|
error: fields.error,
|
|
231
231
|
promptTokensSaved: decision.promptTokensSaved,
|
|
232
232
|
});
|
|
233
|
+
// Book the money HERE, beside the ledger row, so the two can never
|
|
234
|
+
// disagree. Every dispatch that reaches this point was billed —
|
|
235
|
+
// committed, wasted by an escalation, or aborted mid-stream — but only
|
|
236
|
+
// the committed path used to reach the state update below, so aborted
|
|
237
|
+
// dispatches (30% of real spend on live data) stayed invisible to the
|
|
238
|
+
// per-conversation budget guard.
|
|
239
|
+
conversations.accrue(req.conversationKey, { spentUsd: reportedUsd ?? decision.forecast.expectedUsd });
|
|
233
240
|
};
|
|
234
241
|
|
|
235
242
|
// "retry" re-enters the attempt loop; "done" means the turn is settled
|
|
@@ -422,7 +429,9 @@ export async function runTurn(
|
|
|
422
429
|
// hysteresis re-arm below can tell whether this turn changed tier.
|
|
423
430
|
const prevTier = state.currentTier;
|
|
424
431
|
state.currentTier = decision.tier;
|
|
425
|
-
|
|
432
|
+
// Escalations accumulate in SQL for the same reason spend does: `save`
|
|
433
|
+
// below no longer writes this column, so a snapshot cannot clobber it.
|
|
434
|
+
conversations.accrue(req.conversationKey, { escalations });
|
|
426
435
|
// Hysteresis window. Only re-arm when the served tier actually changed
|
|
427
436
|
// (or this turn escalated). Re-arming on EVERY turn — even a trivial one
|
|
428
437
|
// served by a held hard model — extends the lock forever: the classifier
|
|
@@ -434,9 +443,11 @@ export async function runTurn(
|
|
|
434
443
|
if (tierChanged || escalations > 0) {
|
|
435
444
|
state.stickyUntilTurn = turnNumber + resolveHoldTurns(config, req.conversationKey, escalations > 0).turns;
|
|
436
445
|
}
|
|
437
|
-
//
|
|
438
|
-
//
|
|
446
|
+
// Spend is already booked in `writeEntry`, beside the ledger row, so it is
|
|
447
|
+
// deliberately NOT accumulated here — doing both would double-count.
|
|
448
|
+
// Keep the in-memory copy coherent for anything reading `state` later.
|
|
439
449
|
state.spentUsd += reportedUsd ?? decision.forecast.expectedUsd;
|
|
450
|
+
state.escalations += escalations;
|
|
440
451
|
state.lastPromptTokens = usage.promptTokens;
|
|
441
452
|
if (usage.cachedTokens > 0 || usage.cacheWriteTokens > 0) {
|
|
442
453
|
// Non-zero cache traffic is direct evidence the upstream cache exists.
|
|
@@ -451,7 +462,17 @@ export async function runTurn(
|
|
|
451
462
|
// artifact of the turn, not a precondition for finishing it. A
|
|
452
463
|
// `tool_calls` finish means the assistant is still working, so the bridge
|
|
453
464
|
// buffers the fragment rather than writing a near-empty turn.
|
|
454
|
-
|
|
465
|
+
//
|
|
466
|
+
// Only the agent's WORKING conversation is transcribed. A harness also
|
|
467
|
+
// drives utility calls through this same provider with `model: auto` —
|
|
468
|
+
// omp asks for a conversation title and a complexity rating — and those
|
|
469
|
+
// answer ABOUT a conversation instead of participating in one, which is
|
|
470
|
+
// where the junk records (`high`, `<title>…</title>`) came from. They are
|
|
471
|
+
// single-shot and carry NO tool schemas, while an agent always ships its
|
|
472
|
+
// tools, so the tool array is the discriminator. A deliberately
|
|
473
|
+
// tool-less session is therefore not transcribed: silence beats garbage,
|
|
474
|
+
// because every junk record is re-injected into every later turn.
|
|
475
|
+
if (doxActive && req.tools.length > 0) {
|
|
455
476
|
const userText = lastUserText(req);
|
|
456
477
|
const turnEnded = finishReason !== "tool_calls";
|
|
457
478
|
log.debug("agentdox record turn", {
|
package/test/embed-logic.test.ts
CHANGED
|
@@ -115,7 +115,11 @@ describe("agentdox scope", () => {
|
|
|
115
115
|
expect(deriveAgentdoxScope("")).toBe("");
|
|
116
116
|
});
|
|
117
117
|
|
|
118
|
-
test("
|
|
118
|
+
test("the workspace derivation wins over the scope-agnostic defaultScope", () => {
|
|
119
|
+
// Regression: one router install serves every project on the machine, so a
|
|
120
|
+
// global `defaultScope` overriding the derivation made an ashlands session
|
|
121
|
+
// ship `X-Agentdox-Scope: omp-router` — wrong context injected, turns
|
|
122
|
+
// filed under the wrong project.
|
|
119
123
|
const base = {
|
|
120
124
|
server: { host: "127.0.0.1" },
|
|
121
125
|
profiles: [],
|
|
@@ -123,8 +127,11 @@ describe("agentdox scope", () => {
|
|
|
123
127
|
};
|
|
124
128
|
const derived = buildProviderConfig(1234, { ...base, context: { enabled: true, defaultScope: "" } }, "/x/ashlands");
|
|
125
129
|
expect(derived.agentdoxScope).toBe("ashlands");
|
|
126
|
-
const
|
|
127
|
-
expect(
|
|
130
|
+
const both = buildProviderConfig(1234, { ...base, context: { enabled: true, defaultScope: "omp-router" } }, "/x/ashlands");
|
|
131
|
+
expect(both.agentdoxScope).toBe("ashlands");
|
|
132
|
+
// The default only applies when the workspace yields nothing.
|
|
133
|
+
const fallback = buildProviderConfig(1234, { ...base, context: { enabled: true, defaultScope: "pinned" } }, "");
|
|
134
|
+
expect(fallback.agentdoxScope).toBe("pinned");
|
|
128
135
|
});
|
|
129
136
|
|
|
130
137
|
test("no scope header when the bridge is off", () => {
|
package/test/failover.test.ts
CHANGED
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
import { describe, expect, test } from "bun:test";
|
|
2
|
+
|
|
3
|
+
import { createConversationStore } from "../src/router/state.ts";
|
|
4
|
+
import { openDb } from "../src/util/sqlite.ts";
|
|
5
|
+
|
|
6
|
+
function mkStore() {
|
|
7
|
+
const db = openDb(":memory:");
|
|
8
|
+
return { db, store: createConversationStore(db) };
|
|
9
|
+
}
|
|
10
|
+
|
|
11
|
+
describe("conversation spend accounting", () => {
|
|
12
|
+
test("accrue accumulates instead of overwriting", () => {
|
|
13
|
+
const { db, store } = mkStore();
|
|
14
|
+
try {
|
|
15
|
+
store.load("k1");
|
|
16
|
+
store.accrue("k1", { spentUsd: 0.25 });
|
|
17
|
+
store.accrue("k1", { spentUsd: 0.5 });
|
|
18
|
+
store.accrue("k1", { escalations: 1 });
|
|
19
|
+
store.accrue("k1", { escalations: 2 });
|
|
20
|
+
|
|
21
|
+
const state = store.get("k1");
|
|
22
|
+
expect(state?.spentUsd).toBeCloseTo(0.75, 10);
|
|
23
|
+
expect(state?.escalations).toBe(3);
|
|
24
|
+
} finally {
|
|
25
|
+
db.close();
|
|
26
|
+
}
|
|
27
|
+
});
|
|
28
|
+
|
|
29
|
+
test("save cannot clobber spend booked by a dispatch that never committed", () => {
|
|
30
|
+
// The live bug: a dispatch is billed by the upstream, aborts mid-stream,
|
|
31
|
+
// and returns before the commit path. The NEXT dispatch had already loaded
|
|
32
|
+
// a turn-start snapshot, and `save` wrote that snapshot's stale total back
|
|
33
|
+
// over the aborted dispatch's cost. 30% of real spend vanished this way.
|
|
34
|
+
const { db, store } = mkStore();
|
|
35
|
+
try {
|
|
36
|
+
const snapshot = store.load("k1");
|
|
37
|
+
expect(snapshot.spentUsd).toBe(0);
|
|
38
|
+
|
|
39
|
+
// An aborted dispatch books its cost while `snapshot` is still in hand.
|
|
40
|
+
store.accrue("k1", { spentUsd: 0.4, escalations: 1 });
|
|
41
|
+
|
|
42
|
+
// The in-flight turn now commits using the state it loaded earlier.
|
|
43
|
+
snapshot.turn = 1;
|
|
44
|
+
snapshot.currentSlug = "cheap/model";
|
|
45
|
+
store.save(snapshot);
|
|
46
|
+
|
|
47
|
+
const after = store.get("k1");
|
|
48
|
+
expect(after?.spentUsd).toBeCloseTo(0.4, 10);
|
|
49
|
+
expect(after?.escalations).toBe(1);
|
|
50
|
+
// The latest-wins fields still persist normally.
|
|
51
|
+
expect(after?.turn).toBe(1);
|
|
52
|
+
expect(after?.currentSlug).toBe("cheap/model");
|
|
53
|
+
} finally {
|
|
54
|
+
db.close();
|
|
55
|
+
}
|
|
56
|
+
});
|
|
57
|
+
|
|
58
|
+
test("interleaved dispatches both keep their money", () => {
|
|
59
|
+
const { db, store } = mkStore();
|
|
60
|
+
try {
|
|
61
|
+
const a = store.load("k1");
|
|
62
|
+
const b = store.get("k1");
|
|
63
|
+
expect(b).not.toBeNull();
|
|
64
|
+
|
|
65
|
+
store.accrue("k1", { spentUsd: 0.1 });
|
|
66
|
+
store.save(a);
|
|
67
|
+
store.accrue("k1", { spentUsd: 0.2 });
|
|
68
|
+
if (b !== null) store.save(b);
|
|
69
|
+
|
|
70
|
+
expect(store.get("k1")?.spentUsd).toBeCloseTo(0.3, 10);
|
|
71
|
+
} finally {
|
|
72
|
+
db.close();
|
|
73
|
+
}
|
|
74
|
+
});
|
|
75
|
+
|
|
76
|
+
test("a zero delta does not touch the row", () => {
|
|
77
|
+
// Bumping updated_at_ms for a no-op write would keep a dead conversation
|
|
78
|
+
// alive against `prune`, which reaps on that timestamp.
|
|
79
|
+
const { db, store } = mkStore();
|
|
80
|
+
try {
|
|
81
|
+
store.load("k1");
|
|
82
|
+
const before = store.get("k1")?.updatedAtMs ?? 0;
|
|
83
|
+
expect(before).toBeGreaterThan(0);
|
|
84
|
+
store.accrue("k1", { spentUsd: 0, escalations: 0 });
|
|
85
|
+
expect(store.get("k1")?.updatedAtMs).toBe(before);
|
|
86
|
+
} finally {
|
|
87
|
+
db.close();
|
|
88
|
+
}
|
|
89
|
+
});
|
|
90
|
+
});
|
package/test/turn.test.ts
CHANGED
|
@@ -241,8 +241,14 @@ function mkLedger(): { ledger: Ledger; entries: LedgerEntry[] } {
|
|
|
241
241
|
return { ledger, entries };
|
|
242
242
|
}
|
|
243
243
|
|
|
244
|
-
function mkConversations(): {
|
|
244
|
+
function mkConversations(): {
|
|
245
|
+
store: ConversationStore;
|
|
246
|
+
map: Map<string, ConversationState>;
|
|
247
|
+
accrued: Map<string, { spentUsd: number; escalations: number }>;
|
|
248
|
+
} {
|
|
245
249
|
const map = new Map<string, ConversationState>();
|
|
250
|
+
// Mirrors the real store: money accumulates here, NOT through `save`.
|
|
251
|
+
const accrued = new Map<string, { spentUsd: number; escalations: number }>();
|
|
246
252
|
const store: ConversationStore = {
|
|
247
253
|
get: (k) => map.get(k) ?? null,
|
|
248
254
|
load: (k) => {
|
|
@@ -270,9 +276,15 @@ function mkConversations(): { store: ConversationStore; map: Map<string, Convers
|
|
|
270
276
|
save: (s) => {
|
|
271
277
|
map.set(s.key, s);
|
|
272
278
|
},
|
|
279
|
+
accrue: (k, d) => {
|
|
280
|
+
const cur = accrued.get(k) ?? { spentUsd: 0, escalations: 0 };
|
|
281
|
+
cur.spentUsd += d.spentUsd ?? 0;
|
|
282
|
+
cur.escalations += d.escalations ?? 0;
|
|
283
|
+
accrued.set(k, cur);
|
|
284
|
+
},
|
|
273
285
|
prune: () => 0,
|
|
274
286
|
};
|
|
275
|
-
return { store, map };
|
|
287
|
+
return { store, map, accrued };
|
|
276
288
|
}
|
|
277
289
|
|
|
278
290
|
function mkSink(): { sink: ResponseSink; chunks: UpstreamChunk[]; errors: WireError[]; finishes: TurnSummary[] } {
|
|
@@ -556,6 +568,9 @@ describe("exploration reaches the ledger", () => {
|
|
|
556
568
|
});
|
|
557
569
|
|
|
558
570
|
describe("agentdox write-back sees the shape of the turn", () => {
|
|
571
|
+
/** An agent ships its tool schemas; a harness utility call does not. */
|
|
572
|
+
const AGENT_TOOL = { name: "read", description: "read a file", schemaBytes: 128 };
|
|
573
|
+
|
|
559
574
|
function mkRecordingBridge(): { bridge: ContextBridge; records: TurnRecord[] } {
|
|
560
575
|
const records: TurnRecord[] = [];
|
|
561
576
|
return {
|
|
@@ -589,8 +604,9 @@ describe("agentdox write-back sees the shape of the turn", () => {
|
|
|
589
604
|
const { store } = mkConversations();
|
|
590
605
|
const { sink, errors } = mkSink();
|
|
591
606
|
const { bridge, records } = mkRecordingBridge();
|
|
592
|
-
// doxActive needs a scope; the request header supplies it.
|
|
593
|
-
|
|
607
|
+
// doxActive needs a scope; the request header supplies it. The tool
|
|
608
|
+
// schemas mark this as the agent's working conversation.
|
|
609
|
+
const req: NormRequest = { ...mkReq(), agentdoxScope: "proj", tools: [AGENT_TOOL] };
|
|
594
610
|
const deps = { config: mkConfig({ enabled: false }), router, upstream, ledger, conversations: store, catalog, context: bridge };
|
|
595
611
|
|
|
596
612
|
await runTurn(req, sink, deps, new AbortController().signal);
|
|
@@ -603,4 +619,78 @@ describe("agentdox write-back sees the shape of the turn", () => {
|
|
|
603
619
|
expect(records[1]?.turnEnded).toBe(true);
|
|
604
620
|
expect(records[1]?.assistantText).toBe("all done");
|
|
605
621
|
});
|
|
622
|
+
|
|
623
|
+
test("a harness utility call is never transcribed", async () => {
|
|
624
|
+
// omp drives title generation and complexity rating through this same
|
|
625
|
+
// provider with `model: auto`. They answer ABOUT the conversation
|
|
626
|
+
// ("high", "<title>…</title>") and carry NO tool schemas. Recording them
|
|
627
|
+
// created junk agentdox sessions that then fed back into every later
|
|
628
|
+
// context block.
|
|
629
|
+
const { router } = mkRouter([mkDecision("trivial", "cheap/model", { escalateTo: null })]);
|
|
630
|
+
const { upstream } = mkUpstream([
|
|
631
|
+
{ kind: "chunks", chunks: [startChunk("cheap/model"), textChunk("high"), finishChunk("stop"), usageChunk({}, 0.0001)] },
|
|
632
|
+
]);
|
|
633
|
+
const { ledger } = mkLedger();
|
|
634
|
+
const { store } = mkConversations();
|
|
635
|
+
const { sink, errors } = mkSink();
|
|
636
|
+
const { bridge, records } = mkRecordingBridge();
|
|
637
|
+
// Same scope, same provider — only the absent tool array differs.
|
|
638
|
+
const req: NormRequest = { ...mkReq(), agentdoxScope: "proj", tools: [] };
|
|
639
|
+
|
|
640
|
+
await runTurn(req, sink, { config: mkConfig({ enabled: false }), router, upstream, ledger, conversations: store, catalog, context: bridge }, new AbortController().signal);
|
|
641
|
+
|
|
642
|
+
expect(errors).toHaveLength(0);
|
|
643
|
+
expect(records).toHaveLength(0);
|
|
644
|
+
});
|
|
645
|
+
});
|
|
646
|
+
|
|
647
|
+
describe("spend reaches the conversation total however the dispatch ends", () => {
|
|
648
|
+
test("a dispatch that dies mid-stream still books what it was billed", async () => {
|
|
649
|
+
// Live data: 152 aborted dispatches billed $0.9985 — 30% of all spend —
|
|
650
|
+
// and none of it reached the conversation's running total, because an abort
|
|
651
|
+
// returns before the commit path. The ledger row and the per-conversation
|
|
652
|
+
// budget guard must never disagree about money. Probe maxTokens 1 commits
|
|
653
|
+
// on the first token, so the retryable error below cannot re-enter the
|
|
654
|
+
// attempt loop and confuse the accounting.
|
|
655
|
+
const { router } = mkRouter([mkDecision("simple", "cheap/model", { maxTokens: 1, escalateTo: null })]);
|
|
656
|
+
const { upstream } = mkUpstream([
|
|
657
|
+
{
|
|
658
|
+
kind: "die",
|
|
659
|
+
chunks: [
|
|
660
|
+
startChunk("cheap/model"),
|
|
661
|
+
textChunk("plenty of text here, enough to commit on"),
|
|
662
|
+
usageChunk({ promptTokens: 47_700, completionTokens: 154 }, 0.0071),
|
|
663
|
+
],
|
|
664
|
+
error: new UpstreamError("network", 0, "request aborted", true),
|
|
665
|
+
},
|
|
666
|
+
]);
|
|
667
|
+
const { ledger, entries } = mkLedger();
|
|
668
|
+
const { store, accrued } = mkConversations();
|
|
669
|
+
const { sink } = mkSink();
|
|
670
|
+
|
|
671
|
+
await runTurn(mkReq(), sink, { config: mkConfig(), router, upstream, ledger, conversations: store, catalog, context: createDisabledBridge() }, new AbortController().signal);
|
|
672
|
+
|
|
673
|
+
// The turn never committed: the ledger row carries the error.
|
|
674
|
+
expect(entries).toHaveLength(1);
|
|
675
|
+
expect(entries[0]?.error).not.toBeNull();
|
|
676
|
+
// ...but the money was still booked.
|
|
677
|
+
expect(accrued.get("conv-test")?.spentUsd).toBeCloseTo(0.0071, 10);
|
|
678
|
+
});
|
|
679
|
+
|
|
680
|
+
test("a committed turn books its cost exactly once", async () => {
|
|
681
|
+
const { router } = mkRouter([mkDecision("simple", "cheap/model", { escalateTo: null })]);
|
|
682
|
+
const { upstream } = mkUpstream([
|
|
683
|
+
{ kind: "chunks", chunks: [startChunk("cheap/model"), textChunk("done"), finishChunk("stop"), usageChunk({}, 0.002)] },
|
|
684
|
+
]);
|
|
685
|
+
const { ledger, entries } = mkLedger();
|
|
686
|
+
const { store, accrued } = mkConversations();
|
|
687
|
+
const { sink, errors } = mkSink();
|
|
688
|
+
|
|
689
|
+
await runTurn(mkReq(), sink, { config: mkConfig(), router, upstream, ledger, conversations: store, catalog, context: createDisabledBridge() }, new AbortController().signal);
|
|
690
|
+
|
|
691
|
+
expect(errors).toHaveLength(0);
|
|
692
|
+
expect(entries).toHaveLength(1);
|
|
693
|
+
// Booked in writeEntry only — the commit path must not add it again.
|
|
694
|
+
expect(accrued.get("conv-test")?.spentUsd).toBeCloseTo(0.002, 10);
|
|
695
|
+
});
|
|
606
696
|
});
|