auto-model-router 0.2.8 → 0.2.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.omp-plugin/marketplace.json +2 -2
- package/README.md +7 -0
- package/docs/AGENTDOX-BRIDGE.md +48 -36
- package/docs/context-optimization.md +32 -10
- package/package.json +1 -1
- package/src/config/defaults.ts +4 -0
- package/src/config/schema.ts +1 -0
- package/src/config/types.ts +7 -0
- package/src/context/bridge.ts +57 -2
- package/src/context/types.ts +13 -0
- package/src/router/cache-control.ts +53 -22
- package/src/router/compaction.ts +25 -5
- package/src/router/select.ts +3 -2
- package/src/server/turn.ts +11 -5
- package/test/cache-control.test.ts +111 -0
- package/test/compaction.test.ts +37 -0
- package/test/context-bridge.test.ts +94 -1
- package/test/failover.test.ts +1 -1
- package/test/turn.test.ts +52 -1
- package/tools/agentdox-e2e.ts +47 -0
|
@@ -7,14 +7,14 @@
|
|
|
7
7
|
},
|
|
8
8
|
"metadata": {
|
|
9
9
|
"description": "auto-model-router: a local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter",
|
|
10
|
-
"version": "0.2.
|
|
10
|
+
"version": "0.2.10",
|
|
11
11
|
"pluginRoot": "."
|
|
12
12
|
},
|
|
13
13
|
"plugins": [
|
|
14
14
|
{
|
|
15
15
|
"name": "auto-model-router",
|
|
16
16
|
"description": "Local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter. Runs in-process, routes per turn by price and task complexity, with budget caps, mid-stream escalation, and cache-aware hysteresis.",
|
|
17
|
-
"version": "0.2.
|
|
17
|
+
"version": "0.2.10",
|
|
18
18
|
"author": {
|
|
19
19
|
"name": "drewappling",
|
|
20
20
|
"email": "drewappling@gmail.com"
|
package/README.md
CHANGED
|
@@ -620,6 +620,13 @@ shows which model produced which turn. Those messages feed back into the next
|
|
|
620
620
|
`context_assemble`, so the model you switch *to* inherits what the model you
|
|
621
621
|
switched *from* actually did.
|
|
622
622
|
|
|
623
|
+
A recorded turn is the whole **user-visible** turn, not one record per upstream
|
|
624
|
+
request. An agentic turn is a loop of dispatches — each tool round-trip finishes
|
|
625
|
+
with `tool_calls` and emits almost no text, and the last user message does not
|
|
626
|
+
move while the loop runs. So the router buffers the assistant's narration across
|
|
627
|
+
the loop and writes it once, together with the closing synthesis, when the
|
|
628
|
+
assistant actually yields back to the user.
|
|
629
|
+
|
|
623
630
|
Write-backs are queued, bounded, and never awaited: agentdox is an enrichment,
|
|
624
631
|
not a dependency. If it is unreachable the turn routes and dispatches normally,
|
|
625
632
|
and a pinned block keeps being served.
|
package/docs/AGENTDOX-BRIDGE.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# agentdox bridge — handoff
|
|
2
2
|
|
|
3
|
-
**Status:** implemented, typechecks clean,
|
|
4
|
-
through omp.
|
|
3
|
+
**Status:** implemented, typechecks clean, 438 tests pass, injection verified end-to-end
|
|
4
|
+
through omp. The write-back bug in §5 is **fixed**; `context.recordTurns` is safe to enable.
|
|
5
5
|
|
|
6
6
|
Design rationale (why it is built this way):
|
|
7
7
|
`E:/projects/agentdox/docs/architecture/router-context-bridge.md`.
|
|
@@ -86,44 +86,56 @@ Two more traps hit during this work:
|
|
|
86
86
|
- Long-running interactive omp sessions hold their own embedded routers from whenever they
|
|
87
87
|
started. Check `Get-Process omp` before trusting a result.
|
|
88
88
|
|
|
89
|
-
## 5.
|
|
90
|
-
|
|
91
|
-
**
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
89
|
+
## 5. FIXED — one record per dispatch, not per turn
|
|
90
|
+
|
|
91
|
+
**Symptom:** through omp the recorded assistant turn was near-empty —
|
|
92
|
+
`assistantChars=4` (literally `" high"`) while omp displayed several paragraphs. Session and
|
|
93
|
+
model attribution (`refs: ["model:…", "tier:…"]`) were always correct; only the assistant
|
|
94
|
+
*content* was wrong.
|
|
95
|
+
|
|
96
|
+
**Root cause — none of the three leads originally listed here.** The text was not
|
|
97
|
+
under-captured; the *wrong requests* were being recorded. A user-visible turn is not one
|
|
98
|
+
upstream request, it is a whole tool loop of them. Live ledger proof, one conversation key,
|
|
99
|
+
`wasted=0` and `attempt=0` on every row:
|
|
100
|
+
|
|
101
|
+
| dispatch | `finish_reason` | `toolLoopDepth` | completion tokens |
|
|
102
|
+
| --- | --- | --- | --- |
|
|
103
|
+
| 1 | `tool_calls` | 0 | 339 |
|
|
104
|
+
| 2–6 | `tool_calls` | 2, 4, 6, 8, 10 | 91, 68, 198, 78, 44 |
|
|
105
|
+
| 7 | **`stop`** | 12 | **596** |
|
|
106
|
+
| 8 | `tool_calls` | 0 *(next turn)* | 167 |
|
|
107
|
+
|
|
108
|
+
Each tool round-trip is its own dispatch, finishing with `tool_calls` and emitting almost no
|
|
109
|
+
`text` — the payload is tool calls. `" high"` was a stray word of preamble, a *complete*
|
|
110
|
+
record of a fragment rather than a truncated answer. Only the final `stop` dispatch carries
|
|
111
|
+
the synthesis. `recordTurn` fired on all ~13, and the last writer won.
|
|
112
|
+
|
|
113
|
+
The same root cause explains the §6 pollution: `lastUserText` walks back to the last `user`
|
|
114
|
+
message, which does **not** move while a tool loop runs, so the identical user text was
|
|
115
|
+
appended once per round-trip too.
|
|
116
|
+
|
|
117
|
+
**Fix.** `TurnRecord` gained `turnEnded` (`finishReason !== "tool_calls"`, set in
|
|
118
|
+
`src/server/turn.ts`). The bridge buffers assistant fragments per conversation in a
|
|
119
|
+
process-local map and flushes **once**, when the assistant yields back to the user, writing
|
|
120
|
+
the loop's narration plus the closing synthesis as one message. Bounded by
|
|
121
|
+
`MAX_PENDING_CHARS` / `MAX_PENDING_CONVERSATIONS`, since a turn that dies without a terminal
|
|
122
|
+
dispatch never flushes. The terminal dispatch is appended past the char cap, so the model's
|
|
123
|
+
actual answer is never what gets dropped.
|
|
124
|
+
|
|
125
|
+
Covered by `test/context-bridge.test.ts` (loop records one turn; a running loop writes
|
|
126
|
+
nothing; interleaved conversations buffer independently) and `test/turn.test.ts` (the
|
|
127
|
+
`tool_calls` → `turnEnded=false` wiring). All four were verified to FAIL against the old
|
|
128
|
+
behavior. `tools/agentdox-e2e.ts` step 6 proves it against a live server: four dispatches →
|
|
129
|
+
exactly one user and one assistant message.
|
|
120
130
|
|
|
121
131
|
## 6. Also worth doing
|
|
122
132
|
|
|
123
133
|
- **Context pollution.** `context_assemble` includes recent session messages, so recorded
|
|
124
|
-
test turns feed back into the next block (
|
|
125
|
-
|
|
126
|
-
|
|
134
|
+
test turns feed back into the next block (observed: a block containing `assistant:: high`
|
|
135
|
+
from a prior run). The §5 fix removes the ~13×-per-turn duplication that made this acute,
|
|
136
|
+
but noisy *test* turns still compound — `tools/agentdox-e2e.ts` writes real sessions into
|
|
137
|
+
the scope every run. Consider a `sessionLimit` override for the bridge, or excluding
|
|
138
|
+
router-authored sessions.
|
|
127
139
|
- **`context.timeoutMs` is 3000ms** and failures degrade silently at `debug` level by design.
|
|
128
140
|
If agentdox is cold this can no-op invisibly. Consider logging the first failure at `warn`.
|
|
129
141
|
- **Four copies of this project exist** on this machine: this repo, the research checkout,
|
|
@@ -55,10 +55,12 @@ These are hard; the design is shaped by them.
|
|
|
55
55
|
result without its call, nor an assistant tool_call without its result, nor
|
|
56
56
|
reorder the pair.
|
|
57
57
|
- **Never touch:** system/developer messages (they carry the system prompt and the
|
|
58
|
-
agentdox-injected block), the **
|
|
59
|
-
trailing tool-result run of the current loop — the same window
|
|
60
|
-
|
|
61
|
-
|
|
58
|
+
agentdox-injected block), the **protected tail** (newest user-authored run, or the
|
|
59
|
+
trailing tool-result run of the current loop — the same window `features.ts`
|
|
60
|
+
treats as fresh), and image parts (they are capability-relevant; see Edge cases).
|
|
61
|
+
Note that `cache-control.ts` deliberately does **not** treat that tail as fresh:
|
|
62
|
+
a conversation is append-only, so the tail is exactly what the *next* turn will
|
|
63
|
+
read back out of the cache, and it gets a breakpoint (see Prompt cache below).
|
|
62
64
|
- **Determinism.** `auto-model-router explain` replays a past decision offline.
|
|
63
65
|
Phase 1 is a pure function of `(messages, target budget, model window)` →
|
|
64
66
|
replayable. Phase 2 breaks this unless the summary is **pinned and persisted**
|
|
@@ -67,10 +69,19 @@ These are hard; the design is shaped by them.
|
|
|
67
69
|
The governing law is already stated for the agentdox bridge
|
|
68
70
|
(`src/context/bridge.ts`): *the same bytes are re-injected verbatim and the cache
|
|
69
71
|
survives; a refresh rides on a cache miss that was happening anyway.* Compaction
|
|
70
|
-
obeys it
|
|
71
|
-
turn)
|
|
72
|
-
|
|
73
|
-
only when the cache is
|
|
72
|
+
obeys it in two ways: rules are **stable** (identical input → identical elision
|
|
73
|
+
each turn) *and* the truncation set is **monotone** (rule 3 below: oldest-first,
|
|
74
|
+
so it only extends forward and never rewrites an already-cached prefix). Phase 2
|
|
75
|
+
summaries are pinned per conversation and refreshed only when the cache is
|
|
76
|
+
already cold.
|
|
77
|
+
- **Breakpoints must be reproducible.** A `cache_control` breakpoint only pays off
|
|
78
|
+
when a later turn asks to read the exact same byte prefix, so
|
|
79
|
+
`planCacheBreakpoints` places them where the next turn will place them again:
|
|
80
|
+
the system prefix, byte **milestones** at fixed multiples of
|
|
81
|
+
`cache.milestoneTokens` (default 20k) measured over *post-compaction* sizes, and
|
|
82
|
+
the **tail**. A boundary at "roughly 75% of history" — the pre-v0.2.9 behaviour —
|
|
83
|
+
drifts by one index per appended message, so every turn wrote a fresh cache entry
|
|
84
|
+
and read none of them.
|
|
74
85
|
|
|
75
86
|
## Precedent to reuse
|
|
76
87
|
|
|
@@ -184,8 +195,19 @@ A pure function `compact(messages, target, protectFrom) → { messages, saved, n
|
|
|
184
195
|
fetch is authoritative.
|
|
185
196
|
3. **Truncate large stale results** (`maxToolResultBytes`): remaining tool results
|
|
186
197
|
outside the protected window whose content exceeds `maxToolResultBytes` are
|
|
187
|
-
reduced to `keepHeadBytes` + breadcrumb + `keepTailBytes`. Applied
|
|
188
|
-
|
|
198
|
+
reduced to `keepHeadBytes` + breadcrumb + `keepTailBytes`. Applied **oldest
|
|
199
|
+
first** until under `target` or exhausted.
|
|
200
|
+
|
|
201
|
+
Oldest-first is a **prompt-cache requirement**, not an aesthetic. The selected
|
|
202
|
+
set is then always an index-ordered prefix of the eligible results, so across
|
|
203
|
+
turns it only ever extends forward: an existing edit keeps its index and keep
|
|
204
|
+
bytes, and new edits land after every previous one — leaving the cached prefix
|
|
205
|
+
byte-identical. Selecting largest-first instead inserts new edits at arbitrary
|
|
206
|
+
early indices on later turns, rewriting history the upstream had already
|
|
207
|
+
cached. Measured under largest-first: 61% cache read (bimodal 44%/90%, with
|
|
208
|
+
`cachedTokens` pinned at the system prefix on half the requests) against
|
|
209
|
+
76–82% on comparable pre-compaction sessions. `test/compaction.test.ts`
|
|
210
|
+
("the edit set only ever extends forward") pins the property.
|
|
189
211
|
4. **Age-drop assistant reasoning:** reasoning fields on assistant messages outside
|
|
190
212
|
the protected window are dropped. (Largely redundant with the model-driven
|
|
191
213
|
`stripAssistantReasoning`; matters only for reasoning-replay authors.)
|
package/package.json
CHANGED
package/src/config/defaults.ts
CHANGED
|
@@ -144,6 +144,10 @@ export const DEFAULT_CONFIG: RouterConfig = {
|
|
|
144
144
|
// Anthropic allows 4 breakpoints; OpenRouter translates for other vendors.
|
|
145
145
|
maxBreakpoints: 4,
|
|
146
146
|
minPromptTokens: 2_048,
|
|
147
|
+
// One slot for the system prefix, one for the tail, leaving two milestones
|
|
148
|
+
// live at 4 breakpoints. 20k spacing keeps them coarse enough that a
|
|
149
|
+
// milestone survives many turns of appended tool output.
|
|
150
|
+
milestoneTokens: 20_000,
|
|
147
151
|
},
|
|
148
152
|
context: {
|
|
149
153
|
// Off until an agentdox URL + token are configured. Enabling this changes
|
package/src/config/schema.ts
CHANGED
|
@@ -127,6 +127,7 @@ const cache = z.strictObject({
|
|
|
127
127
|
injectBreakpoints: z.boolean().optional(),
|
|
128
128
|
maxBreakpoints: z.number().int().positive().optional(),
|
|
129
129
|
minPromptTokens: z.number().int().nonnegative().optional(),
|
|
130
|
+
milestoneTokens: z.number().int().positive().optional(),
|
|
130
131
|
});
|
|
131
132
|
|
|
132
133
|
const context = z.strictObject({
|
package/src/config/types.ts
CHANGED
|
@@ -286,6 +286,13 @@ export interface CacheConfig {
|
|
|
286
286
|
maxBreakpoints: number;
|
|
287
287
|
/** Skip injection below this prompt-token estimate; small prompts cannot cache. */
|
|
288
288
|
minPromptTokens: number;
|
|
289
|
+
/**
|
|
290
|
+
* Spacing of the stable mid-history breakpoints, in prompt tokens. Boundaries
|
|
291
|
+
* land at fixed multiples of this size, so the same prefix recurs turn after
|
|
292
|
+
* turn and each turn reads what the last one wrote. Smaller = finer recovery
|
|
293
|
+
* after a history rewrite, at the cost of more breakpoint slots.
|
|
294
|
+
*/
|
|
295
|
+
milestoneTokens: number;
|
|
289
296
|
}
|
|
290
297
|
|
|
291
298
|
export interface BudgetConfig {
|
package/src/context/bridge.ts
CHANGED
|
@@ -49,6 +49,29 @@ function renderBlock(raw: string, maxChars: number): string {
|
|
|
49
49
|
].join("\n");
|
|
50
50
|
}
|
|
51
51
|
|
|
52
|
+
/**
|
|
53
|
+
* Cap on assistant text buffered for one in-flight turn, chars. A memory guard
|
|
54
|
+
* only, not a quality knob: a 200-round-trip loop must not buffer without
|
|
55
|
+
* limit. The dispatch that ENDS the turn is appended past this cap, so the
|
|
56
|
+
* model's actual answer is never the thing that gets dropped.
|
|
57
|
+
*/
|
|
58
|
+
const MAX_PENDING_CHARS = 64_000;
|
|
59
|
+
|
|
60
|
+
/**
|
|
61
|
+
* Cap on conversations buffering fragments at once. A turn that dies without a
|
|
62
|
+
* terminal dispatch (client disconnect, upstream error) leaves its buffer
|
|
63
|
+
* behind, so this map is bounded rather than trusted to drain.
|
|
64
|
+
*/
|
|
65
|
+
const MAX_PENDING_CONVERSATIONS = 64;
|
|
66
|
+
|
|
67
|
+
/** Appends a mid-loop fragment, bounded. Blank-line joined: separate thoughts. */
|
|
68
|
+
function appendFragment(prior: string, next: string): string {
|
|
69
|
+
if (next === "") return prior;
|
|
70
|
+
if (prior === "") return next.slice(0, MAX_PENDING_CHARS);
|
|
71
|
+
if (prior.length >= MAX_PENDING_CHARS) return prior;
|
|
72
|
+
return `${prior}\n\n${next}`.slice(0, MAX_PENDING_CHARS);
|
|
73
|
+
}
|
|
74
|
+
|
|
52
75
|
export function createContextBridge(opts: BridgeOptions): ContextBridge {
|
|
53
76
|
const { client, store, log, maxStalenessMs, maxBlockChars, recordTurns, maxQueue } = opts;
|
|
54
77
|
|
|
@@ -57,6 +80,10 @@ export function createContextBridge(opts: BridgeOptions): ContextBridge {
|
|
|
57
80
|
let queue: Promise<void> = Promise.resolve();
|
|
58
81
|
let queued = 0;
|
|
59
82
|
let closed = false;
|
|
83
|
+
// Assistant text buffered across an in-flight tool loop, keyed by
|
|
84
|
+
// conversation. Process-local by design: a turn never spans a restart, and
|
|
85
|
+
// losing a buffer whose turn already died costs nothing.
|
|
86
|
+
const pending = new Map<string, string>();
|
|
60
87
|
|
|
61
88
|
const shouldRefresh = (input: ContextResolveInput, pin: ContextPin | null): boolean => {
|
|
62
89
|
if (pin === null) return true;
|
|
@@ -115,7 +142,34 @@ export function createContextBridge(opts: BridgeOptions): ContextBridge {
|
|
|
115
142
|
|
|
116
143
|
recordTurn(rec: TurnRecord) {
|
|
117
144
|
if (!recordTurns || closed || rec.scope === "") return;
|
|
118
|
-
|
|
145
|
+
|
|
146
|
+
// Mid-loop dispatch: keep the fragment and wait for the turn to end.
|
|
147
|
+
// Writing here is what produced ~13 near-empty assistant messages per
|
|
148
|
+
// turn plus ~13 copies of an unchanged user message, which both lost
|
|
149
|
+
// the real answer and poisoned later context assembly.
|
|
150
|
+
if (!rec.turnEnded) {
|
|
151
|
+
if (rec.assistantText === "") return;
|
|
152
|
+
const prior = pending.get(rec.conversationKey);
|
|
153
|
+
if (prior === undefined && pending.size >= MAX_PENDING_CONVERSATIONS) {
|
|
154
|
+
log.debug("agentdox pending transcript budget full; dropping fragment", { conversations: pending.size });
|
|
155
|
+
return;
|
|
156
|
+
}
|
|
157
|
+
pending.set(rec.conversationKey, appendFragment(prior ?? "", rec.assistantText));
|
|
158
|
+
return;
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
// Turn over. Flush the whole loop's narration plus this dispatch's
|
|
162
|
+
// synthesis as ONE assistant message, attributed to the served model.
|
|
163
|
+
const buffered = pending.get(rec.conversationKey) ?? "";
|
|
164
|
+
pending.delete(rec.conversationKey);
|
|
165
|
+
const assistantText =
|
|
166
|
+
buffered === ""
|
|
167
|
+
? rec.assistantText
|
|
168
|
+
: rec.assistantText === ""
|
|
169
|
+
? buffered
|
|
170
|
+
: `${buffered}\n\n${rec.assistantText}`;
|
|
171
|
+
|
|
172
|
+
if (rec.userText === "" && assistantText === "") return;
|
|
119
173
|
if (queued >= maxQueue) {
|
|
120
174
|
log.debug("agentdox write-back queue full; dropping turn record", { queued });
|
|
121
175
|
return;
|
|
@@ -134,7 +188,7 @@ export function createContextBridge(opts: BridgeOptions): ContextBridge {
|
|
|
134
188
|
// every turn shows WHICH model produced it.
|
|
135
189
|
const refs = [`model:${rec.slug}`, `tier:${rec.tier}`];
|
|
136
190
|
if (rec.userText !== "") await client.append(sessionId, "user", rec.userText, []);
|
|
137
|
-
if (
|
|
191
|
+
if (assistantText !== "") await client.append(sessionId, "assistant", assistantText, refs);
|
|
138
192
|
})
|
|
139
193
|
.catch((err: unknown) => {
|
|
140
194
|
log.debug("agentdox write-back failed", { error: err instanceof Error ? err.message : String(err) });
|
|
@@ -150,6 +204,7 @@ export function createContextBridge(opts: BridgeOptions): ContextBridge {
|
|
|
150
204
|
|
|
151
205
|
close() {
|
|
152
206
|
closed = true;
|
|
207
|
+
pending.clear();
|
|
153
208
|
},
|
|
154
209
|
};
|
|
155
210
|
}
|
package/src/context/types.ts
CHANGED
|
@@ -49,10 +49,23 @@ export interface TurnRecord {
|
|
|
49
49
|
/** Title used if this is the first turn and a session must be created. */
|
|
50
50
|
title: string;
|
|
51
51
|
userText: string;
|
|
52
|
+
/** Text THIS dispatch produced. Fragments are joined across a tool loop. */
|
|
52
53
|
assistantText: string;
|
|
53
54
|
/** The slug that actually served the turn — the model attribution. */
|
|
54
55
|
slug: string;
|
|
55
56
|
tier: string;
|
|
57
|
+
/**
|
|
58
|
+
* Whether the assistant yielded control back to the user — i.e. the upstream
|
|
59
|
+
* finish reason was NOT `tool_calls`.
|
|
60
|
+
*
|
|
61
|
+
* A user-visible turn is many dispatches: every tool round-trip is its own
|
|
62
|
+
* request, and only the last carries the model's synthesis. The intermediate
|
|
63
|
+
* ones are almost pure tool calls with a few stray words of text, and the
|
|
64
|
+
* last *user* message does not move while the loop runs. False therefore
|
|
65
|
+
* means "buffer this fragment, the turn is still running" — recording it as
|
|
66
|
+
* a turn would write a near-empty answer and re-append the same user text.
|
|
67
|
+
*/
|
|
68
|
+
turnEnded: boolean;
|
|
56
69
|
}
|
|
57
70
|
|
|
58
71
|
export interface ContextBridge {
|
|
@@ -2,15 +2,39 @@
|
|
|
2
2
|
* Cache-breakpoint placement (Anthropic-style `cache_control: ephemeral`;
|
|
3
3
|
* OpenRouter translates these to OpenAI/Google cache primitives, so one
|
|
4
4
|
* mechanism covers every target). Returns message indices to mark.
|
|
5
|
+
*
|
|
6
|
+
* Placement is chosen for REUSE ACROSS TURNS, not for a single request. A
|
|
7
|
+
* breakpoint only pays off when a LATER turn asks to read the exact same byte
|
|
8
|
+
* prefix, so every boundary here must be one the next turn will reproduce:
|
|
9
|
+
*
|
|
10
|
+
* - the system prefix, which never moves;
|
|
11
|
+
* - byte MILESTONES at fixed multiples of `cache.milestoneTokens`, which land
|
|
12
|
+
* on the same message every turn for as long as the prefix is unchanged
|
|
13
|
+
* (a boundary at "roughly 75% of history" drifts with every appended
|
|
14
|
+
* message, so it writes a fresh entry each turn and never reads one);
|
|
15
|
+
* - the tail, so the whole of this turn's prompt becomes the entry the NEXT
|
|
16
|
+
* turn reads. A conversation is append-only: nothing already in the array
|
|
17
|
+
* can change later, so there is no "volatile tail" to keep out of the
|
|
18
|
+
* cache. Walking back over an agent loop's trailing tool run instead left
|
|
19
|
+
* everything the loop had accumulated permanently uncached.
|
|
20
|
+
*
|
|
21
|
+
* Milestones are measured over POST-compaction sizes, so the boundaries match
|
|
22
|
+
* the bytes actually dispatched.
|
|
5
23
|
*/
|
|
6
24
|
|
|
7
25
|
import type { CatalogModel } from "../catalog/types.ts";
|
|
8
26
|
import type { RouterConfig } from "../config/types.ts";
|
|
9
27
|
import { priceAt } from "../cost/forecast.ts";
|
|
10
28
|
import { estimateTokens } from "../tokens/estimate.ts";
|
|
11
|
-
import type {
|
|
29
|
+
import type { CompactionEdit, NormRequest } from "../wire/types.ts";
|
|
30
|
+
import { compactedBytes } from "./compaction.ts";
|
|
12
31
|
|
|
13
|
-
export function planCacheBreakpoints(
|
|
32
|
+
export function planCacheBreakpoints(
|
|
33
|
+
req: NormRequest,
|
|
34
|
+
model: CatalogModel,
|
|
35
|
+
cfg: RouterConfig,
|
|
36
|
+
compactionPlan: readonly CompactionEdit[] = [],
|
|
37
|
+
): number[] {
|
|
14
38
|
if (!cfg.cache.injectBreakpoints) return [];
|
|
15
39
|
const promptTokens = estimateTokens(req.promptBytes, model.tokenizer, null);
|
|
16
40
|
// Small prompts cannot amortize cache-write cost.
|
|
@@ -19,6 +43,8 @@ export function planCacheBreakpoints(req: NormRequest, model: CatalogModel, cfg:
|
|
|
19
43
|
if (priceAt(model, Math.max(1, promptTokens)).cacheRead === undefined) return [];
|
|
20
44
|
|
|
21
45
|
const messages = req.messages;
|
|
46
|
+
if (messages.length === 0) return [];
|
|
47
|
+
|
|
22
48
|
const picks: number[] = [];
|
|
23
49
|
|
|
24
50
|
// 1. End of the last system message: the most stable, usually largest prefix.
|
|
@@ -30,28 +56,33 @@ export function planCacheBreakpoints(req: NormRequest, model: CatalogModel, cfg:
|
|
|
30
56
|
}
|
|
31
57
|
}
|
|
32
58
|
|
|
33
|
-
// 2.
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
//
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
59
|
+
// 2. The tail: everything this turn sent, cached for the next turn to read.
|
|
60
|
+
picks.push(messages.length - 1);
|
|
61
|
+
|
|
62
|
+
// 3. Stable byte milestones through the history, newest first so the slots
|
|
63
|
+
// left over by 1 and 2 cover the largest readable prefixes.
|
|
64
|
+
const editByIndex = new Map<number, CompactionEdit>();
|
|
65
|
+
for (const e of compactionPlan) editByIndex.set(e.index, e);
|
|
66
|
+
const bytesPerToken = req.promptBytes / Math.max(1, promptTokens);
|
|
67
|
+
const milestoneBytes = Math.max(1, Math.floor(cfg.cache.milestoneTokens * bytesPerToken));
|
|
68
|
+
const milestones: number[] = [];
|
|
69
|
+
let cumulative = 0;
|
|
70
|
+
let nextMilestone = milestoneBytes;
|
|
71
|
+
for (let i = 0; i < messages.length - 1; i++) {
|
|
72
|
+
const m = messages[i];
|
|
73
|
+
if (m === undefined) continue;
|
|
74
|
+
cumulative += compactedBytes(m.textBytes, editByIndex.get(i));
|
|
75
|
+
if (cumulative >= nextMilestone) {
|
|
76
|
+
milestones.push(i);
|
|
77
|
+
// Skip past every milestone this message already crossed, so one huge
|
|
78
|
+
// message cannot claim a run of adjacent boundaries.
|
|
79
|
+
while (cumulative >= nextMilestone) nextMilestone += milestoneBytes;
|
|
49
80
|
}
|
|
50
|
-
if (i >= 0) picks.push(i);
|
|
51
81
|
}
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
82
|
+
for (let i = milestones.length - 1; i >= 0; i--) {
|
|
83
|
+
const idx = milestones[i];
|
|
84
|
+
if (idx !== undefined) picks.push(idx);
|
|
85
|
+
}
|
|
55
86
|
|
|
56
87
|
// Dedupe preserving priority order, cap, return ascending indices.
|
|
57
88
|
const seen = new Set<number>();
|
package/src/router/compaction.ts
CHANGED
|
@@ -26,6 +26,17 @@ const EMPTY: CompactionResult = { edits: [], savedBytes: 0 };
|
|
|
26
26
|
/** Approximate byte cost of an elision breadcrumb; savings are net of it. */
|
|
27
27
|
const BREADCRUMB_BYTES = 120;
|
|
28
28
|
|
|
29
|
+
/**
|
|
30
|
+
* Byte size a message ends up with once `edit` is applied — the size that
|
|
31
|
+
* actually reaches the upstream. Cache-breakpoint placement walks these rather
|
|
32
|
+
* than the raw `textBytes`, so its boundaries match the dispatched bytes.
|
|
33
|
+
*/
|
|
34
|
+
export function compactedBytes(originalBytes: number, edit: CompactionEdit | undefined): number {
|
|
35
|
+
if (edit === undefined) return originalBytes;
|
|
36
|
+
const kept = edit.mode === "stub" ? BREADCRUMB_BYTES : edit.keepHead + edit.keepTail + BREADCRUMB_BYTES;
|
|
37
|
+
return Math.min(originalBytes, kept);
|
|
38
|
+
}
|
|
39
|
+
|
|
29
40
|
/**
|
|
30
41
|
* First string value in a tool call's argument JSON — a schema-agnostic proxy
|
|
31
42
|
* for the resource a call operates on (a `path`, `id`, `query`, ...). Used to
|
|
@@ -73,9 +84,18 @@ interface ToolResult {
|
|
|
73
84
|
/**
|
|
74
85
|
* Plans compaction for a turn's messages toward `targetBytes` of total prompt.
|
|
75
86
|
* Duplicate and superseded elisions (pure stale-data wins) are always applied;
|
|
76
|
-
* large-result truncation (more lossy) runs
|
|
87
|
+
* large-result truncation (more lossy) runs OLDEST-first only until the target
|
|
77
88
|
* is met. `promptBytes` is the whole prompt (messages + system + tool schemas),
|
|
78
89
|
* so the target is compared against the real dispatched size.
|
|
90
|
+
*
|
|
91
|
+
* Oldest-first is a prompt-cache requirement, not a preference. The truncated
|
|
92
|
+
* set is then always an index-ordered PREFIX of the eligible results, so as a
|
|
93
|
+
* conversation grows and the target tightens the set only ever EXTENDS FORWARD:
|
|
94
|
+
* an edit already made keeps the same index and the same keep bytes, and a new
|
|
95
|
+
* edit lands after every previous one. Selecting largest-first instead inserts
|
|
96
|
+
* fresh edits at arbitrarily early indices on later turns, rewriting history
|
|
97
|
+
* the upstream had already cached and collapsing cache reads to the system
|
|
98
|
+
* prefix (measured: 61% cache read, bimodal, vs 76-82% before compaction).
|
|
79
99
|
*/
|
|
80
100
|
export function planCompaction(
|
|
81
101
|
messages: readonly NormMessage[],
|
|
@@ -145,11 +165,11 @@ export function planCompaction(
|
|
|
145
165
|
}
|
|
146
166
|
}
|
|
147
167
|
|
|
148
|
-
// Rule 3: truncate large stale results,
|
|
168
|
+
// Rule 3: truncate large stale results, OLDEST first, until under target.
|
|
169
|
+
// `tools` is already in message order, so the filter alone yields that order
|
|
170
|
+
// and the selected set stays an extend-forward prefix across turns.
|
|
149
171
|
const keepBudget = cfg.keepHeadBytes + cfg.keepTailBytes + BREADCRUMB_BYTES;
|
|
150
|
-
const truncatable = tools
|
|
151
|
-
.filter((t) => !done.has(t.index) && t.bytes > cfg.maxToolResultBytes && t.bytes > keepBudget)
|
|
152
|
-
.sort((a, b) => b.bytes - a.bytes || a.index - b.index);
|
|
172
|
+
const truncatable = tools.filter((t) => !done.has(t.index) && t.bytes > cfg.maxToolResultBytes && t.bytes > keepBudget);
|
|
153
173
|
for (const t of truncatable) {
|
|
154
174
|
if (promptBytes - saved <= targetBytes) break;
|
|
155
175
|
edits.push({ index: t.index, mode: "truncate", keepHead: cfg.keepHeadBytes, keepTail: cfg.keepTailBytes, note: `large ${t.name || "tool"} result` });
|
package/src/router/select.ts
CHANGED
|
@@ -390,8 +390,9 @@ export function select(args: SelectArgs): Decision {
|
|
|
390
390
|
if (fallbacks.length >= 2) break;
|
|
391
391
|
}
|
|
392
392
|
|
|
393
|
-
// 7. Cache breakpoints
|
|
394
|
-
|
|
393
|
+
// 7. Cache breakpoints, measured over post-compaction sizes so the
|
|
394
|
+
// boundaries match the bytes that actually get dispatched.
|
|
395
|
+
const cacheBreakpointMessageIndices = planCacheBreakpoints(req, chosen.model, cfg, compactionPlan);
|
|
395
396
|
|
|
396
397
|
// 8. Guarded probe: only tiers configured for probing, and only when a
|
|
397
398
|
// strictly higher tier exists inside the profile envelope to escalate into.
|
package/src/server/turn.ts
CHANGED
|
@@ -448,22 +448,28 @@ export async function runTurn(
|
|
|
448
448
|
|
|
449
449
|
// Record the settled turn into agentdox, attributed to the model that
|
|
450
450
|
// actually served it. Queued and never awaited: the transcript is an
|
|
451
|
-
// artifact of the turn, not a precondition for finishing it.
|
|
451
|
+
// artifact of the turn, not a precondition for finishing it. A
|
|
452
|
+
// `tool_calls` finish means the assistant is still working, so the bridge
|
|
453
|
+
// buffers the fragment rather than writing a near-empty turn.
|
|
452
454
|
if (doxActive) {
|
|
455
|
+
const userText = lastUserText(req);
|
|
456
|
+
const turnEnded = finishReason !== "tool_calls";
|
|
453
457
|
log.debug("agentdox record turn", {
|
|
454
|
-
|
|
458
|
+
conversationKey: req.conversationKey.slice(0, 8),
|
|
459
|
+
userChars: userText.length,
|
|
455
460
|
assistantChars: assistantText.length,
|
|
456
|
-
|
|
457
|
-
|
|
461
|
+
finishReason,
|
|
462
|
+
turnEnded,
|
|
458
463
|
});
|
|
459
464
|
bridge.recordTurn({
|
|
460
465
|
scope: doxScope,
|
|
461
466
|
conversationKey: req.conversationKey,
|
|
462
467
|
title: sessionTitle(req),
|
|
463
|
-
userText
|
|
468
|
+
userText,
|
|
464
469
|
assistantText,
|
|
465
470
|
slug: servedSlug ?? decision.slug,
|
|
466
471
|
tier: decision.tier,
|
|
472
|
+
turnEnded,
|
|
467
473
|
});
|
|
468
474
|
}
|
|
469
475
|
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
import { describe, expect, test } from "bun:test";
|
|
2
|
+
|
|
3
|
+
import { normalizeCatalogModel } from "../src/catalog/openrouter-catalog.ts";
|
|
4
|
+
import type { CatalogModel } from "../src/catalog/types.ts";
|
|
5
|
+
import { loadConfig } from "../src/config/load.ts";
|
|
6
|
+
import type { RouterConfig } from "../src/config/types.ts";
|
|
7
|
+
import { priceAt } from "../src/cost/forecast.ts";
|
|
8
|
+
import { planCacheBreakpoints } from "../src/router/cache-control.ts";
|
|
9
|
+
import { planCompaction } from "../src/router/compaction.ts";
|
|
10
|
+
import { parseChatRequest } from "../src/wire/openai/request.ts";
|
|
11
|
+
import type { NormRequest } from "../src/wire/types.ts";
|
|
12
|
+
|
|
13
|
+
const FIXTURE = (await Bun.file("test/fixtures/openrouter-models.json").json()) as { data: unknown[] };
|
|
14
|
+
const MODELS: CatalogModel[] = FIXTURE.data.map(normalizeCatalogModel).filter((m): m is CatalogModel => m !== null);
|
|
15
|
+
// Breakpoints are only planned for models that publish a cache-read price.
|
|
16
|
+
const CACHING = MODELS.find((m) => priceAt(m, 100_000).cacheRead !== undefined);
|
|
17
|
+
if (CACHING === undefined) throw new Error("fixture has no model with a published cache-read price");
|
|
18
|
+
const MODEL: CatalogModel = CACHING;
|
|
19
|
+
|
|
20
|
+
const BASE = loadConfig({});
|
|
21
|
+
function cfg(over: Partial<RouterConfig["cache"]> = {}): RouterConfig {
|
|
22
|
+
return { ...BASE, cache: { ...BASE.cache, ...over } };
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
const RESULT_BYTES = 8_000;
|
|
26
|
+
const result = (turn: number): string => `result ${turn}:${"x".repeat(RESULT_BYTES)}`;
|
|
27
|
+
|
|
28
|
+
/** A tool-loop conversation: system, one user ask, then `turns` call/result pairs. */
|
|
29
|
+
function loop(turns: number): NormRequest {
|
|
30
|
+
const messages: Record<string, unknown>[] = [
|
|
31
|
+
{ role: "system", content: `You are a coding agent.${"!".repeat(4_000)}` },
|
|
32
|
+
{ role: "user", content: "find the bug" },
|
|
33
|
+
];
|
|
34
|
+
for (let t = 0; t < turns; t++) {
|
|
35
|
+
messages.push({
|
|
36
|
+
role: "assistant",
|
|
37
|
+
content: null,
|
|
38
|
+
tool_calls: [{ id: `c${t}`, type: "function", function: { name: "read", arguments: `{"path":"f${t}.ts"}` } }],
|
|
39
|
+
});
|
|
40
|
+
messages.push({ role: "tool", tool_call_id: `c${t}`, content: result(t) });
|
|
41
|
+
}
|
|
42
|
+
return parseChatRequest({ model: "auto", messages }, new Headers());
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
describe("planCacheBreakpoints", () => {
|
|
46
|
+
test("marks the tail so the next turn can read this turn's whole prompt", () => {
|
|
47
|
+
const req = loop(12);
|
|
48
|
+
const picks = planCacheBreakpoints(req, MODEL, cfg());
|
|
49
|
+
expect(picks).toContain(req.messages.length - 1);
|
|
50
|
+
});
|
|
51
|
+
|
|
52
|
+
test("marks the system prefix", () => {
|
|
53
|
+
const req = loop(12);
|
|
54
|
+
const picks = planCacheBreakpoints(req, MODEL, cfg());
|
|
55
|
+
expect(picks).toContain(0);
|
|
56
|
+
});
|
|
57
|
+
|
|
58
|
+
test("mid-history boundaries are stable as the conversation grows", () => {
|
|
59
|
+
// Uncapped so the comparison is about placement, not slot eviction.
|
|
60
|
+
const uncapped = cfg({ maxBreakpoints: 64, milestoneTokens: 4_000 });
|
|
61
|
+
const mid = (turns: number): number[] => {
|
|
62
|
+
const req = loop(turns);
|
|
63
|
+
const tail = req.messages.length - 1;
|
|
64
|
+
return planCacheBreakpoints(req, MODEL, uncapped).filter((i) => i !== 0 && i !== tail);
|
|
65
|
+
};
|
|
66
|
+
const early = mid(10);
|
|
67
|
+
expect(early.length).toBeGreaterThan(1);
|
|
68
|
+
for (const turns of [11, 12, 13, 20]) {
|
|
69
|
+
// Every boundary the earlier turn wrote is still a boundary later, so
|
|
70
|
+
// the later turn reads what the earlier one paid to write.
|
|
71
|
+
expect(mid(turns)).toEqual(expect.arrayContaining(early));
|
|
72
|
+
}
|
|
73
|
+
});
|
|
74
|
+
|
|
75
|
+
test("boundaries are spaced by the milestone size, not by message position", () => {
|
|
76
|
+
const req = loop(30);
|
|
77
|
+
const tail = req.messages.length - 1;
|
|
78
|
+
const coarse = planCacheBreakpoints(req, MODEL, cfg({ maxBreakpoints: 64, milestoneTokens: 20_000 })).filter(
|
|
79
|
+
(i) => i !== 0 && i !== tail,
|
|
80
|
+
);
|
|
81
|
+
const fine = planCacheBreakpoints(req, MODEL, cfg({ maxBreakpoints: 64, milestoneTokens: 4_000 })).filter(
|
|
82
|
+
(i) => i !== 0 && i !== tail,
|
|
83
|
+
);
|
|
84
|
+
expect(fine.length).toBeGreaterThan(coarse.length);
|
|
85
|
+
});
|
|
86
|
+
|
|
87
|
+
test("keeps the system prefix and the tail when slots are scarce", () => {
|
|
88
|
+
const req = loop(30);
|
|
89
|
+
const picks = planCacheBreakpoints(req, MODEL, cfg({ maxBreakpoints: 2, milestoneTokens: 4_000 }));
|
|
90
|
+
expect(picks).toEqual([0, req.messages.length - 1]);
|
|
91
|
+
});
|
|
92
|
+
|
|
93
|
+
test("milestones follow post-compaction sizes", () => {
|
|
94
|
+
const req = loop(30);
|
|
95
|
+
const tail = req.messages.length - 1;
|
|
96
|
+
const plan = planCompaction(req.messages, BASE.compaction, req.promptBytes * 0.3, req.promptBytes);
|
|
97
|
+
expect(plan.edits.length).toBeGreaterThan(0);
|
|
98
|
+
const options = cfg({ maxBreakpoints: 64, milestoneTokens: 4_000 });
|
|
99
|
+
const raw = planCacheBreakpoints(req, MODEL, options).filter((i) => i !== 0 && i !== tail);
|
|
100
|
+
const compacted = planCacheBreakpoints(req, MODEL, options, plan.edits).filter((i) => i !== 0 && i !== tail);
|
|
101
|
+
// Shrinking early results pushes each byte milestone later in the history.
|
|
102
|
+
expect(compacted.length).toBeLessThan(raw.length);
|
|
103
|
+
expect(Math.min(...compacted)).toBeGreaterThan(Math.min(...raw));
|
|
104
|
+
});
|
|
105
|
+
|
|
106
|
+
test("injects nothing below the minimum prompt size, or when disabled", () => {
|
|
107
|
+
const small = parseChatRequest({ model: "auto", messages: [{ role: "user", content: "hi" }] }, new Headers());
|
|
108
|
+
expect(planCacheBreakpoints(small, MODEL, cfg())).toEqual([]);
|
|
109
|
+
expect(planCacheBreakpoints(loop(12), MODEL, cfg({ injectBreakpoints: false }))).toEqual([]);
|
|
110
|
+
});
|
|
111
|
+
});
|
package/test/compaction.test.ts
CHANGED
|
@@ -99,6 +99,43 @@ describe("planCompaction", () => {
|
|
|
99
99
|
const b = planCompaction(msgs, CFG, 1, 10_000);
|
|
100
100
|
expect(a).toEqual(b);
|
|
101
101
|
});
|
|
102
|
+
|
|
103
|
+
// The prompt-cache contract: an edit, once made, keeps its index and its keep
|
|
104
|
+
// bytes for the rest of the conversation, and every later edit lands AFTER
|
|
105
|
+
// it. Anything else rewrites already-cached history and forces a full
|
|
106
|
+
// re-read of the prefix on the next turn.
|
|
107
|
+
test("the edit set only ever extends forward as the conversation grows", () => {
|
|
108
|
+
// Sizes GROW with age-descending order (newest results are the biggest), so
|
|
109
|
+
// a size-ordered planner selects newest-first and its later additions move
|
|
110
|
+
// BACKWARD into already-cached history. Equal-sized results would make
|
|
111
|
+
// every ordering identical and the assertions vacuous.
|
|
112
|
+
const loop = (pairs: number): NormMessage[] => {
|
|
113
|
+
const msgs: NormMessage[] = [user("go")];
|
|
114
|
+
for (let i = 0; i < pairs; i++) {
|
|
115
|
+
const content = `R${i}:${"x".repeat(200 + i * 40)}`;
|
|
116
|
+
msgs.push(asst(`c${i}`, "read", `{"path":"f${i}.ts"}`), toolMsg(`c${i}`, "read", content));
|
|
117
|
+
}
|
|
118
|
+
return msgs;
|
|
119
|
+
};
|
|
120
|
+
let previous: number[] = [];
|
|
121
|
+
for (let pairs = 4; pairs <= 24; pairs++) {
|
|
122
|
+
const msgs = loop(pairs);
|
|
123
|
+
const promptBytes = msgs.reduce((n, m) => n + m.textBytes, 0);
|
|
124
|
+
// A target the plan can hit with a handful of edits: this is where the
|
|
125
|
+
// selection ORDER decides which results get truncated. A saturating
|
|
126
|
+
// target would truncate everything and hide the difference.
|
|
127
|
+
const indices = planCompaction(msgs, CFG, Math.floor(promptBytes * 0.8), promptBytes).edits.map((e) => e.index);
|
|
128
|
+
// Nothing already compacted may be dropped...
|
|
129
|
+
expect(indices).toEqual(expect.arrayContaining(previous));
|
|
130
|
+
// ...and anything new lands after every existing edit.
|
|
131
|
+
const added = indices.filter((i) => !previous.includes(i));
|
|
132
|
+
if (previous.length > 0 && added.length > 0) {
|
|
133
|
+
expect(Math.min(...added)).toBeGreaterThan(Math.max(...previous));
|
|
134
|
+
}
|
|
135
|
+
previous = indices;
|
|
136
|
+
}
|
|
137
|
+
expect(previous.length).toBeGreaterThan(4);
|
|
138
|
+
});
|
|
102
139
|
});
|
|
103
140
|
|
|
104
141
|
describe("renderUpstreamBody applies compaction", () => {
|
|
@@ -3,7 +3,7 @@ import { describe, expect, test } from "bun:test";
|
|
|
3
3
|
import type { AgentDoxClient } from "../src/context/agentdox.ts";
|
|
4
4
|
import { createContextBridge } from "../src/context/bridge.ts";
|
|
5
5
|
import { createContextStore } from "../src/context/store.ts";
|
|
6
|
-
import type { ContextResolveInput } from "../src/context/types.ts";
|
|
6
|
+
import type { ContextResolveInput, TurnRecord } from "../src/context/types.ts";
|
|
7
7
|
import { createLogger } from "../src/util/log.ts";
|
|
8
8
|
import { openDb } from "../src/util/sqlite.ts";
|
|
9
9
|
import { injectForTest } from "./helpers/inject.ts";
|
|
@@ -246,6 +246,7 @@ describe("context bridge write-back", () => {
|
|
|
246
246
|
assistantText: "done",
|
|
247
247
|
slug: "anthropic/claude-haiku-4.5",
|
|
248
248
|
tier: "simple",
|
|
249
|
+
turnEnded: true,
|
|
249
250
|
});
|
|
250
251
|
bridge.recordTurn({
|
|
251
252
|
scope: "ashlands",
|
|
@@ -255,6 +256,7 @@ describe("context bridge write-back", () => {
|
|
|
255
256
|
assistantText: "ok",
|
|
256
257
|
slug: "anthropic/claude-opus-4.5",
|
|
257
258
|
tier: "hard",
|
|
259
|
+
turnEnded: true,
|
|
258
260
|
});
|
|
259
261
|
await bridge.flush();
|
|
260
262
|
|
|
@@ -280,6 +282,7 @@ describe("context bridge write-back", () => {
|
|
|
280
282
|
assistantText: "a",
|
|
281
283
|
slug: "x",
|
|
282
284
|
tier: "simple",
|
|
285
|
+
turnEnded: true,
|
|
283
286
|
});
|
|
284
287
|
await bridge.flush();
|
|
285
288
|
expect(client.appended).toHaveLength(0);
|
|
@@ -287,6 +290,96 @@ describe("context bridge write-back", () => {
|
|
|
287
290
|
db.close();
|
|
288
291
|
}
|
|
289
292
|
});
|
|
293
|
+
|
|
294
|
+
/** One dispatch of a turn; `turnEnded` marks the one that yields to the user. */
|
|
295
|
+
function mkRecord(over: Partial<TurnRecord> & { turnEnded: boolean }): TurnRecord {
|
|
296
|
+
return {
|
|
297
|
+
scope: "ashlands",
|
|
298
|
+
conversationKey: "k1",
|
|
299
|
+
title: "movement fix",
|
|
300
|
+
userText: "fix movement",
|
|
301
|
+
assistantText: "",
|
|
302
|
+
slug: "z-ai/glm-5.3-flash",
|
|
303
|
+
tier: "simple",
|
|
304
|
+
...over,
|
|
305
|
+
};
|
|
306
|
+
}
|
|
307
|
+
|
|
308
|
+
test("a tool loop records one turn, not one record per dispatch", async () => {
|
|
309
|
+
const client = mkClient();
|
|
310
|
+
const { bridge, db } = mkBridge(client);
|
|
311
|
+
try {
|
|
312
|
+
// One user-visible turn: five tool round-trips, then the synthesis.
|
|
313
|
+
// Every dispatch carries the SAME unchanged user text — recording per
|
|
314
|
+
// dispatch appended it once per round-trip and buried the real answer
|
|
315
|
+
// under near-empty assistant messages.
|
|
316
|
+
for (const assistantText of ["let me look", "", "checking the ledger", "", "almost there"]) {
|
|
317
|
+
bridge.recordTurn(mkRecord({ assistantText, turnEnded: false }));
|
|
318
|
+
}
|
|
319
|
+
bridge.recordTurn(mkRecord({ assistantText: "fixed: the damping was inverted.", turnEnded: true }));
|
|
320
|
+
await bridge.flush();
|
|
321
|
+
|
|
322
|
+
expect(client.sessionsCreated).toBe(1);
|
|
323
|
+
const users = client.appended.filter((m) => m.role === "user");
|
|
324
|
+
const assistants = client.appended.filter((m) => m.role === "assistant");
|
|
325
|
+
expect(users).toHaveLength(1);
|
|
326
|
+
expect(assistants).toHaveLength(1);
|
|
327
|
+
// The loop's narration AND the closing synthesis survive, in order.
|
|
328
|
+
expect(assistants[0]?.content).toBe(
|
|
329
|
+
"let me look\n\nchecking the ledger\n\nalmost there\n\nfixed: the damping was inverted.",
|
|
330
|
+
);
|
|
331
|
+
expect(assistants[0]?.refs).toEqual(["model:z-ai/glm-5.3-flash", "tier:simple"]);
|
|
332
|
+
} finally {
|
|
333
|
+
db.close();
|
|
334
|
+
}
|
|
335
|
+
});
|
|
336
|
+
|
|
337
|
+
test("a tool loop still running writes nothing", async () => {
|
|
338
|
+
const client = mkClient();
|
|
339
|
+
const { bridge, db } = mkBridge(client);
|
|
340
|
+
try {
|
|
341
|
+
bridge.recordTurn(mkRecord({ assistantText: "let me look", turnEnded: false }));
|
|
342
|
+
await bridge.flush();
|
|
343
|
+
// The assistant has not answered yet. Writing here is what produced the
|
|
344
|
+
// 4-char transcripts, so mid-loop must stay silent.
|
|
345
|
+
expect(client.appended).toHaveLength(0);
|
|
346
|
+
expect(client.sessionsCreated).toBe(0);
|
|
347
|
+
} finally {
|
|
348
|
+
db.close();
|
|
349
|
+
}
|
|
350
|
+
});
|
|
351
|
+
|
|
352
|
+
test("interleaved conversations buffer independently", async () => {
|
|
353
|
+
const client = mkClient();
|
|
354
|
+
const { bridge, db } = mkBridge(client);
|
|
355
|
+
try {
|
|
356
|
+
bridge.recordTurn(mkRecord({ conversationKey: "k1", assistantText: "k1 narration", turnEnded: false }));
|
|
357
|
+
bridge.recordTurn(mkRecord({ conversationKey: "k2", assistantText: "k2 narration", turnEnded: false }));
|
|
358
|
+
bridge.recordTurn(mkRecord({ conversationKey: "k2", assistantText: "k2 answer", turnEnded: true }));
|
|
359
|
+
bridge.recordTurn(mkRecord({ conversationKey: "k1", assistantText: "k1 answer", turnEnded: true }));
|
|
360
|
+
await bridge.flush();
|
|
361
|
+
|
|
362
|
+
const assistants = client.appended.filter((m) => m.role === "assistant");
|
|
363
|
+
expect(assistants).toHaveLength(2);
|
|
364
|
+
expect(assistants[0]?.content).toBe("k2 narration\n\nk2 answer");
|
|
365
|
+
expect(assistants[1]?.content).toBe("k1 narration\n\nk1 answer");
|
|
366
|
+
} finally {
|
|
367
|
+
db.close();
|
|
368
|
+
}
|
|
369
|
+
});
|
|
370
|
+
|
|
371
|
+
test("a silent turn still records the user message", async () => {
|
|
372
|
+
const client = mkClient();
|
|
373
|
+
const { bridge, db } = mkBridge(client);
|
|
374
|
+
try {
|
|
375
|
+
bridge.recordTurn(mkRecord({ assistantText: "", turnEnded: true }));
|
|
376
|
+
await bridge.flush();
|
|
377
|
+
expect(client.appended.filter((m) => m.role === "user")).toHaveLength(1);
|
|
378
|
+
expect(client.appended.filter((m) => m.role === "assistant")).toHaveLength(0);
|
|
379
|
+
} finally {
|
|
380
|
+
db.close();
|
|
381
|
+
}
|
|
382
|
+
});
|
|
290
383
|
});
|
|
291
384
|
|
|
292
385
|
describe("context injection into the wire body", () => {
|
package/test/failover.test.ts
CHANGED
|
@@ -68,7 +68,7 @@ function mkConfig(escalation: Partial<EscalationConfig> = {}): RouterConfig {
|
|
|
68
68
|
},
|
|
69
69
|
hysteresis: { holdTurns: 2, holdTurnsAfterEscalation: 4, switchMargin: 1.5, cacheWarmTtlMs: 600_000, maxDowngradePerTurn: 1 },
|
|
70
70
|
exploration: { enabled: false, rates: {}, stickyPolicy: "never", holdTurns: { enabled: false, values: [2, 3, 4] } },
|
|
71
|
-
cache: { injectBreakpoints: true, maxBreakpoints: 4, minPromptTokens: 1024 },
|
|
71
|
+
cache: { injectBreakpoints: true, maxBreakpoints: 4, minPromptTokens: 1024, milestoneTokens: 20_000 },
|
|
72
72
|
context: { enabled: false, baseUrl: "", token: "", defaultScope: "", timeoutMs: 3_000, maxStalenessMs: 900_000, maxBlockChars: 24_000, recordTurns: false, maxQueue: 64 },
|
|
73
73
|
compaction: { enabled: false, budgetTokens: 40_000, fitToWindow: true, protectRecentTurns: 4, maxToolResultBytes: 4_096, keepHeadBytes: 512, keepTailBytes: 512, elideSupersededReads: true, collapseDuplicateResults: true },
|
|
74
74
|
budget: { onExceeded: "downgrade" },
|
package/test/turn.test.ts
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { describe, expect, test } from "bun:test";
|
|
2
2
|
import { createDisabledBridge } from "../src/context/bridge.ts";
|
|
3
|
+
import type { ContextBridge, TurnRecord } from "../src/context/types.ts";
|
|
3
4
|
import type { CatalogSource } from "../src/catalog/types.ts";
|
|
4
5
|
import type { EscalationConfig, RouterConfig } from "../src/config/types.ts";
|
|
5
6
|
import { EMPTY_USAGE, type Ledger, type LedgerEntry, type UsageCounts } from "../src/cost/types.ts";
|
|
@@ -68,7 +69,7 @@ function mkConfig(escalation: Partial<EscalationConfig> = {}): RouterConfig {
|
|
|
68
69
|
},
|
|
69
70
|
hysteresis: { holdTurns: 2, holdTurnsAfterEscalation: 4, switchMargin: 1.5, cacheWarmTtlMs: 600_000, maxDowngradePerTurn: 1 },
|
|
70
71
|
exploration: { enabled: false, rates: {}, stickyPolicy: "never", holdTurns: { enabled: false, values: [2, 3, 4] } },
|
|
71
|
-
cache: { injectBreakpoints: true, maxBreakpoints: 4, minPromptTokens: 1024 },
|
|
72
|
+
cache: { injectBreakpoints: true, maxBreakpoints: 4, minPromptTokens: 1024, milestoneTokens: 20_000 },
|
|
72
73
|
context: { enabled: false, baseUrl: "", token: "", defaultScope: "", timeoutMs: 3_000, maxStalenessMs: 900_000, maxBlockChars: 24_000, recordTurns: false, maxQueue: 64 },
|
|
73
74
|
compaction: { enabled: false, budgetTokens: 40_000, fitToWindow: true, protectRecentTurns: 4, maxToolResultBytes: 4_096, keepHeadBytes: 512, keepTailBytes: 512, elideSupersededReads: true, collapseDuplicateResults: true },
|
|
74
75
|
budget: { onExceeded: "downgrade" },
|
|
@@ -553,3 +554,53 @@ describe("exploration reaches the ledger", () => {
|
|
|
553
554
|
expect(entries[0]?.exploredFrom).toBeNull();
|
|
554
555
|
});
|
|
555
556
|
});
|
|
557
|
+
|
|
558
|
+
describe("agentdox write-back sees the shape of the turn", () => {
|
|
559
|
+
function mkRecordingBridge(): { bridge: ContextBridge; records: TurnRecord[] } {
|
|
560
|
+
const records: TurnRecord[] = [];
|
|
561
|
+
return {
|
|
562
|
+
records,
|
|
563
|
+
bridge: {
|
|
564
|
+
enabled: true,
|
|
565
|
+
resolve: () => Promise.resolve(null),
|
|
566
|
+
recordTurn: (rec) => {
|
|
567
|
+
records.push(rec);
|
|
568
|
+
},
|
|
569
|
+
flush: () => Promise.resolve(),
|
|
570
|
+
close: () => {},
|
|
571
|
+
},
|
|
572
|
+
};
|
|
573
|
+
}
|
|
574
|
+
|
|
575
|
+
test("a tool_calls finish is a fragment; only a yielding finish ends the turn", async () => {
|
|
576
|
+
// A user-visible turn is many dispatches. The orchestrator must tell the
|
|
577
|
+
// bridge which one actually handed control back, or the transcript records
|
|
578
|
+
// a near-empty answer per tool round-trip and re-appends the same user
|
|
579
|
+
// text every time.
|
|
580
|
+
const { router } = mkRouter([
|
|
581
|
+
mkDecision("simple", "cheap/model", { escalateTo: null }),
|
|
582
|
+
mkDecision("simple", "cheap/model", { escalateTo: null }),
|
|
583
|
+
]);
|
|
584
|
+
const { upstream } = mkUpstream([
|
|
585
|
+
{ kind: "chunks", chunks: [startChunk("cheap/model"), textChunk("let me look"), finishChunk("tool_calls"), usageChunk({}, 0.0001)] },
|
|
586
|
+
{ kind: "chunks", chunks: [startChunk("cheap/model"), textChunk("all done"), finishChunk("stop"), usageChunk({}, 0.0001)] },
|
|
587
|
+
]);
|
|
588
|
+
const { ledger } = mkLedger();
|
|
589
|
+
const { store } = mkConversations();
|
|
590
|
+
const { sink, errors } = mkSink();
|
|
591
|
+
const { bridge, records } = mkRecordingBridge();
|
|
592
|
+
// doxActive needs a scope; the request header supplies it.
|
|
593
|
+
const req: NormRequest = { ...mkReq(), agentdoxScope: "proj" };
|
|
594
|
+
const deps = { config: mkConfig({ enabled: false }), router, upstream, ledger, conversations: store, catalog, context: bridge };
|
|
595
|
+
|
|
596
|
+
await runTurn(req, sink, deps, new AbortController().signal);
|
|
597
|
+
await runTurn(req, sink, deps, new AbortController().signal);
|
|
598
|
+
|
|
599
|
+
expect(errors).toHaveLength(0);
|
|
600
|
+
expect(records).toHaveLength(2);
|
|
601
|
+
expect(records[0]?.turnEnded).toBe(false);
|
|
602
|
+
expect(records[0]?.assistantText).toBe("let me look");
|
|
603
|
+
expect(records[1]?.turnEnded).toBe(true);
|
|
604
|
+
expect(records[1]?.assistantText).toBe("all done");
|
|
605
|
+
});
|
|
606
|
+
});
|
package/tools/agentdox-e2e.ts
CHANGED
|
@@ -93,6 +93,7 @@ bridge.recordTurn({
|
|
|
93
93
|
assistantText: "yes - refs carry model: and tier:.",
|
|
94
94
|
slug: "anthropic/claude-haiku-4.5",
|
|
95
95
|
tier: "simple",
|
|
96
|
+
turnEnded: true,
|
|
96
97
|
});
|
|
97
98
|
await bridge.flush();
|
|
98
99
|
|
|
@@ -118,6 +119,52 @@ if (mine !== undefined) {
|
|
|
118
119
|
);
|
|
119
120
|
}
|
|
120
121
|
|
|
122
|
+
// 6. A tool loop must record ONE turn, not one record per dispatch. This is the
|
|
123
|
+
// regression that made transcripts useless: every tool round-trip is its own
|
|
124
|
+
// dispatch, finishing with `tool_calls` and carrying an UNCHANGED last user
|
|
125
|
+
// message, so recording per dispatch wrote a near-empty assistant message
|
|
126
|
+
// and a duplicate user message per round-trip.
|
|
127
|
+
const loopKey = `${conversationKey}-loop`;
|
|
128
|
+
const loopTitle = `bridge e2e loop ${loopKey}`;
|
|
129
|
+
function loopDispatch(assistantText: string, turnEnded: boolean): void {
|
|
130
|
+
bridge.recordTurn({
|
|
131
|
+
scope,
|
|
132
|
+
conversationKey: loopKey,
|
|
133
|
+
title: loopTitle,
|
|
134
|
+
userText: "why did cache read fall?",
|
|
135
|
+
assistantText,
|
|
136
|
+
slug: "z-ai/glm-5.3-flash",
|
|
137
|
+
tier: "simple",
|
|
138
|
+
turnEnded,
|
|
139
|
+
});
|
|
140
|
+
}
|
|
141
|
+
for (const fragment of ["reading the ledger", "", "checking the cache column"]) loopDispatch(fragment, false);
|
|
142
|
+
loopDispatch("the breakpoint index drifted every turn.", true);
|
|
143
|
+
await bridge.flush();
|
|
144
|
+
|
|
145
|
+
const loopRes = await fetch(`${baseUrl}/sessions?scope=${encodeURIComponent(scope)}`, {
|
|
146
|
+
headers: { authorization: `Bearer ${token}` },
|
|
147
|
+
});
|
|
148
|
+
const loopSessions = (await loopRes.json()) as { id: string; title: string }[];
|
|
149
|
+
const loopSession = loopSessions.find((s) => s.title === loopTitle);
|
|
150
|
+
check("tool loop created a session", loopSession !== undefined, loopSession?.id ?? "not found");
|
|
151
|
+
|
|
152
|
+
if (loopSession !== undefined) {
|
|
153
|
+
const full = await fetch(`${baseUrl}/sessions/${loopSession.id}`, {
|
|
154
|
+
headers: { authorization: `Bearer ${token}` },
|
|
155
|
+
});
|
|
156
|
+
const session = (await full.json()) as { messages: { role: string; content: string }[] };
|
|
157
|
+
const users = session.messages.filter((m) => m.role === "user");
|
|
158
|
+
const assistants = session.messages.filter((m) => m.role === "assistant");
|
|
159
|
+
check("four dispatches wrote exactly one user message", users.length === 1, `${users.length} user messages`);
|
|
160
|
+
check("four dispatches wrote exactly one assistant message", assistants.length === 1, `${assistants.length} assistant messages`);
|
|
161
|
+
check(
|
|
162
|
+
"the loop's narration and the closing synthesis both survive",
|
|
163
|
+
assistants[0]?.content === "reading the ledger\n\nchecking the cache column\n\nthe breakpoint index drifted every turn.",
|
|
164
|
+
JSON.stringify(assistants[0]?.content ?? ""),
|
|
165
|
+
);
|
|
166
|
+
}
|
|
167
|
+
|
|
121
168
|
db.close();
|
|
122
169
|
console.log(failures === 0 ? "\nAll bridge e2e checks passed." : `\n${failures} check(s) failed.`);
|
|
123
170
|
process.exit(failures === 0 ? 0 : 1);
|