@ssheleg/agent-stack 0.18.2 → 0.19.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +60 -0
- package/package.json +1 -1
- package/plugins/agent-stack/.claude-plugin/plugin.json +1 -1
- package/plugins/agent-stack/skills/agent-harness/references/system-prompt.md +14 -0
- package/plugins/agent-stack/skills/agent-interop/references/mcp-scale.md +7 -0
- package/plugins/agent-stack/skills/agent-orchestrator/SKILL.md +17 -45
- package/plugins/agent-stack/skills/agent-orchestrator/references/context-engineering.md +13 -7
- package/plugins/agent-stack/skills/agent-orchestrator/references/kv-cache.md +186 -0
- package/plugins/agent-stack/skills/agent-orchestrator/references/patterns.md +49 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,63 @@
|
|
|
1
|
+
## v0.19.0 — the prompt cache is an architectural constraint, and the gate that never watched the body
|
|
2
|
+
|
|
3
|
+
Fourteen findings from nine independent bundles of the 2026-08-31 harvest said the same
|
|
4
|
+
thing, and this pack had **zero** coverage of it: `grep -riE 'kv.cache|prefix cache|prompt
|
|
5
|
+
cach|cache hit|cacheable'` over all four skills returned nothing.
|
|
6
|
+
|
|
7
|
+
**`agent-orchestrator/references/kv-cache.md`** is the missing layer. Its spine is one
|
|
8
|
+
directional rule — *the earlier a change lands, the more cache it invalidates* — from which
|
|
9
|
+
the append-only discipline stops being a style preference and becomes arithmetic:
|
|
10
|
+
|
|
11
|
+
- A `Current time: {{now}}` line added to a system prompt at **100,000 conversations/day**
|
|
12
|
+
took TTFT from **0.5 s to 3–5 s** and **nearly doubled the monthly bill**. Nothing else
|
|
13
|
+
changed; the line simply sat early in the prefix.
|
|
14
|
+
- **N binary runtime conditions in front of the cache boundary produce 2^N cache
|
|
15
|
+
populations.** Three innocuous ones — OS × debug × locale — turn one warm cache into eight
|
|
16
|
+
cold ones.
|
|
17
|
+
- The named invalidators, including the two that read as architecture rather than mistakes:
|
|
18
|
+
a **role switch that replaces the system prompt** buys a hard tool boundary and pays the
|
|
19
|
+
prefix for it, where a skill keeps the prefix and gives an instruction the model may
|
|
20
|
+
ignore; and a **hot-loaded plugin** owes its documentation a KV-cache impact, not only a
|
|
21
|
+
behavioural one.
|
|
22
|
+
- Four things that look like optimisations and are not, with the measurement that settles
|
|
23
|
+
each: sorting tools by usage frequency costs the prefix and buys **almost no
|
|
24
|
+
tool-selection accuracy**; per-request few-shot retrieval guarantees a permanent miss;
|
|
25
|
+
`defer_loading` works by keeping definitions out of the *prefix* rather than out of the
|
|
26
|
+
request; and "manual concatenation breaks the cache" is **false** — caching is over bytes,
|
|
27
|
+
and the real cost of flattening is an out-of-distribution format.
|
|
28
|
+
|
|
29
|
+
**The architectural error it corrects.** SKILL.md §10 said the prompt is rebuilt per request
|
|
30
|
+
from live capability flags "so the two can never disagree" — right for correctness, silent
|
|
31
|
+
on cost. It now states that rebuilding is free **only while it is byte-identical**, and that
|
|
32
|
+
a capability change mid-session is appended rather than written into the prefix.
|
|
33
|
+
|
|
34
|
+
**Four of the compaction ladder's five rungs were labelled `free`, and none of them is.**
|
|
35
|
+
`context-engineering.md` gains a **Cache** column: every rung edits history, so every rung
|
|
36
|
+
invalidates from its replacement point. The consequence is a scheduling rule — compact in
|
|
37
|
+
batch at a threshold, never every round — because a per-round compactor pays a full
|
|
38
|
+
re-prefill each round to save tokens it had already paid for once.
|
|
39
|
+
|
|
40
|
+
**Displacement, not deletion.** The body sat at **4749 of the 4750-token working limit** —
|
|
41
|
+
one token of headroom — so the §10 correction was paid for by moving the Data Verification
|
|
42
|
+
Protocol into `agent-harness/references/system-prompt.md`, which is where §10's own text
|
|
43
|
+
says prompt *content* belongs. A protocol sitting in the wiring section had contradicted the
|
|
44
|
+
boundary its section drew.
|
|
45
|
+
|
|
46
|
+
**The ceiling was enforced, but only where nobody was looking.** CI's *House skill audit*
|
|
47
|
+
job runs `make-skill`'s auditor and fails on the headroom GAP — it caught this very release
|
|
48
|
+
and is why the body was displaced rather than shipped over the line. What was missing is the
|
|
49
|
+
**local** half: `npm test` measured nothing, so a file could sit one token from the edge
|
|
50
|
+
through any number of local runs and only find out on a push. `test/validate.py` now fails
|
|
51
|
+
hard at the 5000 platform budget and **reports** at the 4750 working limit, on every run,
|
|
52
|
+
pass or fail — a warning printed only beside a failure is invisible on exactly the runs where
|
|
53
|
+
it can still be acted on. Its estimator is deliberately cruder than the auditor's and errs
|
|
54
|
+
high (~4947 against ~4804 for the same file), so it warns early and never late.
|
|
55
|
+
|
|
56
|
+
**Headroom went up, not just under the bar.** The §2 tool-calling listing moved to
|
|
57
|
+
`references/patterns.md` — the body keeps the six steps and the guard, the reference carries
|
|
58
|
+
the code — taking the body from **4749/4750 before this release to 4609/4750 after**, with
|
|
59
|
+
the cache reference added rather than at its expense.
|
|
60
|
+
|
|
1
61
|
## v0.18.2 — the card lost eleven characters and the check that watched it counted bytes
|
|
2
62
|
|
|
3
63
|
`docs/assets/social-preview.png` — the image every link to this repository renders —
|
package/package.json
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
"name": "agent-stack",
|
|
4
4
|
"displayName": "Agent Stack",
|
|
5
5
|
"description": "Four skills: agent-orchestrator — tool-calling loops, pipelines with checkpoints, provider routing with fallback, memory architecture, plus the wallet side of reselling LLM access; agent-evals — run/trace/thread evals, LLM judges, and fixtures grown from production; agent-interop — MCP servers and clients, A2A agent cards, the MCP Registry, and gateways; agent-harness — system prompts, tool shaping, workflow-vs-agent, and auditing an agent system.",
|
|
6
|
-
"version": "0.
|
|
6
|
+
"version": "0.19.0",
|
|
7
7
|
"author": {
|
|
8
8
|
"name": "ssheleg",
|
|
9
9
|
"url": "https://x.com/sshlg93"
|
|
@@ -68,6 +68,20 @@ current date, environment, tenant, available capabilities, the user's locale —
|
|
|
68
68
|
not assumed. A capability-aware prompt that lists only the tools actually connected beats a
|
|
69
69
|
static prompt describing tools that may be absent.
|
|
70
70
|
|
|
71
|
+
**A worked injection, because the shape is the lesson.** A data-verification protocol added
|
|
72
|
+
only when a database is connected:
|
|
73
|
+
|
|
74
|
+
> - First-time metrics: ask the user *"Do these numbers match expectations?"*
|
|
75
|
+
> - Financial figures: mention units (cents vs dollars) and ask for confirmation
|
|
76
|
+
> - Anomalies: explain proactively and ask the user to verify
|
|
77
|
+
> - Rejected data: investigate the discrepancy, and record the finding as a learning
|
|
78
|
+
|
|
79
|
+
Note what makes it injectable rather than permanent: every line is conditional on a
|
|
80
|
+
capability the agent may or may not have. It arrived here from `agent-orchestrator`'s §10,
|
|
81
|
+
which owns the *wiring* that assembles a prompt and says in its own text that the content
|
|
82
|
+
belongs in this file — so a protocol sitting there contradicted the boundary its section
|
|
83
|
+
had drawn.
|
|
84
|
+
|
|
71
85
|
## Flexible while learning, strict in production
|
|
72
86
|
|
|
73
87
|
The same instruction should not survive the whole lifecycle.
|
|
@@ -84,6 +84,13 @@ Most providers cache the prompt prefix — **including the `tools` array**. Addi
|
|
|
84
84
|
a definition mid-conversation **invalidates that cache**, and the resulting miss can cost more
|
|
85
85
|
tokens than the definitions you so carefully removed.
|
|
86
86
|
|
|
87
|
+
**This is one instance of a general rule**, and the platform now has a cache-safe path the three
|
|
88
|
+
mitigations below were approximating: `defer_loading` keeps deferred definitions out of the
|
|
89
|
+
system-prompt prefix and appends discovered ones inline as `tool_reference` blocks, leaving the
|
|
90
|
+
cached prefix untouched (at least one tool must stay non-deferred — all-deferred is a 400). The
|
|
91
|
+
economics, the arithmetic and the other invalidators are in `agent-orchestrator`'s
|
|
92
|
+
`references/kv-cache.md`.
|
|
93
|
+
|
|
87
94
|
Three mitigations, in the order they are usually right:
|
|
88
95
|
|
|
89
96
|
- **Append** newly discovered definitions after the cache breakpoint rather than re-sorting
|
|
@@ -86,45 +86,20 @@ Typed result subclasses per agent (e.g. `SQLAgentResult` with `query`, `results`
|
|
|
86
86
|
|
|
87
87
|
## 2. Tool-Calling Loop (Simple Path)
|
|
88
88
|
|
|
89
|
-
The
|
|
89
|
+
The loop, in six steps, and every one of them is load-bearing:
|
|
90
|
+
|
|
91
|
+
1. **Manage context pressure first** — trim, and inject a wrap-up instruction if the
|
|
92
|
+
window is nearly full, *before* spending the call.
|
|
93
|
+
2. **Call the model with retry**, tools attached.
|
|
94
|
+
3. **No tool calls means the answer** — that is the loop's only clean exit.
|
|
95
|
+
4. **Append the assistant turn** with its tool calls, unedited.
|
|
96
|
+
5. **Dispatch** — `asyncio.gather` for independent tools, sequentially for stateful ones.
|
|
97
|
+
6. **Append every result** keyed by `tool_call_id`, then iterate.
|
|
98
|
+
|
|
99
|
+
The `else` on the `for` is the guard: on exhaustion, compose a best-effort answer from what
|
|
100
|
+
was gathered rather than returning nothing. The full listing is in
|
|
101
|
+
[`references/patterns.md`](references/patterns.md) → *The tool-calling loop, in full*.
|
|
90
102
|
|
|
91
|
-
```python
|
|
92
|
-
max_iter = settings.max_orchestrator_iterations # e.g. 10
|
|
93
|
-
for iteration in range(max_iter):
|
|
94
|
-
# 1. Context pressure management
|
|
95
|
-
messages, did_trim = trim_loop_messages(messages, context_window)
|
|
96
|
-
if should_wrap_up(messages, context_window):
|
|
97
|
-
messages.append(Message(role="system",
|
|
98
|
-
content="IMPORTANT: Stop making tool calls. Compose final answer now."))
|
|
99
|
-
|
|
100
|
-
# 2. LLM call with retry
|
|
101
|
-
llm_resp = await llm_call_with_retry(messages, tools, provider, model)
|
|
102
|
-
|
|
103
|
-
# 3. No tool calls = final answer
|
|
104
|
-
if not llm_resp.tool_calls:
|
|
105
|
-
final_text = llm_resp.content
|
|
106
|
-
break
|
|
107
|
-
|
|
108
|
-
# 4. Dispatch tool calls
|
|
109
|
-
messages.append(Message(role="assistant", content=llm_resp.content,
|
|
110
|
-
tool_calls=llm_resp.tool_calls))
|
|
111
|
-
|
|
112
|
-
# 5. Parallel execution (except sequential-only tools)
|
|
113
|
-
if len(llm_resp.tool_calls) > 1 and not has_sequential_tool:
|
|
114
|
-
results = await asyncio.gather(
|
|
115
|
-
*(handle_tool(tc, context) for tc in llm_resp.tool_calls),
|
|
116
|
-
return_exceptions=True)
|
|
117
|
-
else:
|
|
118
|
-
results = [await handle_tool(tc, context) for tc in llm_resp.tool_calls]
|
|
119
|
-
|
|
120
|
-
# 6. Append tool results
|
|
121
|
-
for tc, (text, sub_result) in zip(llm_resp.tool_calls, results):
|
|
122
|
-
messages.append(Message(role="tool", content=text,
|
|
123
|
-
tool_call_id=tc.id, name=tc.name))
|
|
124
|
-
else:
|
|
125
|
-
# Max iterations reached — compose partial answer from gathered data
|
|
126
|
-
final_text = "I reached maximum analysis steps. Here is what I found..."
|
|
127
|
-
```
|
|
128
103
|
|
|
129
104
|
**Critical details:**
|
|
130
105
|
- **Parallel tool dispatch**: Use `asyncio.gather` for independent tools, sequential for stateful ones (e.g. data processing that depends on prior query results)
|
|
@@ -280,13 +255,9 @@ What belongs in that text, at what altitude, and how to enumerate the vocabulary
|
|
|
280
255
|
agent stops inventing status values is the **`agent-harness`** skill's
|
|
281
256
|
`agent-harness/references/system-prompt.md` — one home, and it is not this one. What is *this* skill's
|
|
282
257
|
is the wiring: the prompt is rebuilt per request from the same capability flags the tool
|
|
283
|
-
list is built from, so the two can never disagree.
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
- First-time metrics: ask user "Do these numbers match expectations?"
|
|
287
|
-
- Financial figures: mention units (cents vs dollars), ask for confirmation
|
|
288
|
-
- Anomalies: proactively explain and ask user to verify
|
|
289
|
-
- Rejected data: investigate discrepancy, record finding as learning
|
|
258
|
+
list is built from, so the two can never disagree. **Rebuilding is free only while it is
|
|
259
|
+
byte-identical**; when capabilities move mid-session, append the change rather than rewrite
|
|
260
|
+
the prefix — `references/kv-cache.md` for what an edit before the boundary costs.
|
|
290
261
|
|
|
291
262
|
---
|
|
292
263
|
|
|
@@ -383,6 +354,7 @@ there; this table is only an index.
|
|
|
383
354
|
| [`references/memory-lifecycle.md`](references/memory-lifecycle.md) | the memory **write path** — what enters, and what leaves |
|
|
384
355
|
| [`references/memory-landscape.md`](references/memory-landscape.md) | **build or adopt** a memory layer, and what is settled practice |
|
|
385
356
|
| [`references/context-engineering.md`](references/context-engineering.md) | the loop is **running out of window** |
|
|
357
|
+
| [`references/kv-cache.md`](references/kv-cache.md) | you are deciding **what goes where in a request** — the static prefix against the trajectory, what a miss costs, the 2^N cache populations a runtime condition in front of the boundary creates, the named invalidators, and the four common optimisations that pay cache for nothing |
|
|
386
358
|
| [`references/runtime.md`](references/runtime.md) | the agent must **survive a crash, a pause, a second message or a schedule** |
|
|
387
359
|
| [`references/governance.md`](references/governance.md) | the question is **permission, not cost** — what it may do, and how you prove it |
|
|
388
360
|
| [`references/llm-proxy-billing.md`](references/llm-proxy-billing.md) | the product **resells LLM access** |
|
|
@@ -33,13 +33,13 @@ Compaction is not one call. It is an ordered set of strategies, and the expensiv
|
|
|
33
33
|
is last. A loop that summarizes at the first sign of pressure pays a model call and a
|
|
34
34
|
round-trip for what a string operation would have solved.
|
|
35
35
|
|
|
36
|
-
| Rung | What it does | Cost | Loses |
|
|
37
|
-
|
|
38
|
-
| 1. Microcompact | Replace old tool results with a tombstone: `[tool result cleared]`. Keep the N most recent. | free | old observations, kept recent ones |
|
|
39
|
-
| 2. Head/tail collapse | For an oversized text block, keep a head and a tail, elide the middle with a marker | free | the middle of long outputs |
|
|
40
|
-
| 3. Session condensation | Collapse each older message to a one-line summary, capped in total | free | phrasing, keeps the thread of events |
|
|
41
|
-
| 4. LLM compaction | One model call summarizes the transcript into a structured brief | a call + latency | anything the summarizer does not think to keep |
|
|
42
|
-
| 5. Prompt-round truncation | Drop the oldest whole prompt rounds, boundary-aligned | free | the earliest history entirely |
|
|
36
|
+
| Rung | What it does | Cost | Cache | Loses |
|
|
37
|
+
|---|---|---|---|---|
|
|
38
|
+
| 1. Microcompact | Replace old tool results with a tombstone: `[tool result cleared]`. Keep the N most recent. | free | invalidates from the oldest replacement | old observations, kept recent ones |
|
|
39
|
+
| 2. Head/tail collapse | For an oversized text block, keep a head and a tail, elide the middle with a marker | free | invalidates from that block | the middle of long outputs |
|
|
40
|
+
| 3. Session condensation | Collapse each older message to a one-line summary, capped in total | free | invalidates from the first collapsed message | phrasing, keeps the thread of events |
|
|
41
|
+
| 4. LLM compaction | One model call summarizes the transcript into a structured brief | a call + latency | invalidates the whole trajectory | anything the summarizer does not think to keep |
|
|
42
|
+
| 5. Prompt-round truncation | Drop the oldest whole prompt rounds, boundary-aligned | free | invalidates from the first dropped round | the earliest history entirely |
|
|
43
43
|
|
|
44
44
|
Rung 5 exists for one case: **the compaction request itself does not fit.** When rung 4
|
|
45
45
|
fails because the transcript it must summarize is over the limit, summarizing harder is
|
|
@@ -49,6 +49,12 @@ not available — you drop oldest rounds and retry, bounded by a small retry cou
|
|
|
49
49
|
as position: anything past a few thousand characters, and every result from an external
|
|
50
50
|
tool server, whose outputs are the usual window hog and the least likely to be re-read.
|
|
51
51
|
|
|
52
|
+
**"Free" is a compute column, and none of these rungs is free in cache.** Every one edits
|
|
53
|
+
history, and an edit invalidates the prefix from the replacement point onward — so the
|
|
54
|
+
frequency of compaction is itself a cost decision: **compact in batch at a threshold, never
|
|
55
|
+
every round.** A per-round compactor pays a full re-prefill each round to save tokens it had
|
|
56
|
+
already paid for once. `references/kv-cache.md` carries the economics and the arithmetic.
|
|
57
|
+
|
|
52
58
|
## Re-measure between rungs
|
|
53
59
|
|
|
54
60
|
After every rung, measure again and stop if you are under the threshold. Two failure
|
|
@@ -0,0 +1,186 @@
|
|
|
1
|
+
# The prompt cache — an architectural constraint, not an optimisation
|
|
2
|
+
|
|
3
|
+
**Load this before deciding what goes where in a request**: the system prompt's shape,
|
|
4
|
+
whether the tool list may change mid-run, how often to compact, whether to swap a role or
|
|
5
|
+
load a skill, whether few-shot examples may be selected per request.
|
|
6
|
+
|
|
7
|
+
It reads as a cost topic and is not. The cache decides **context layout before semantics
|
|
8
|
+
does**, because every other decision in this skill is cheap or ruinous depending on where
|
|
9
|
+
in the request it lands. A team that treats it as tuning discovers it as an incident.
|
|
10
|
+
|
|
11
|
+
## Contents
|
|
12
|
+
|
|
13
|
+
- [The split, and the one directional rule](#the-split-and-the-one-directional-rule)
|
|
14
|
+
- [What a miss costs, measured](#what-a-miss-costs-measured)
|
|
15
|
+
- [Every condition before the boundary doubles the keys](#every-condition-before-the-boundary-doubles-the-keys)
|
|
16
|
+
- [The named invalidators](#the-named-invalidators)
|
|
17
|
+
- [Three regions, never mixed](#three-regions-never-mixed)
|
|
18
|
+
- [Compaction is a cache decision](#compaction-is-a-cache-decision)
|
|
19
|
+
- [Four things that look like optimisations and are not](#four-things-that-look-like-optimisations-and-are-not)
|
|
20
|
+
- [Where this constrains the rest of the skill](#where-this-constrains-the-rest-of-the-skill)
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## The split, and the one directional rule
|
|
25
|
+
|
|
26
|
+
Every request is two parts:
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
agent context = static prefix + trajectory
|
|
30
|
+
^^^^^^^^^^^^^ ^^^^^^^^^^
|
|
31
|
+
system prompt user messages, assistant messages,
|
|
32
|
+
tool definitions tool results — append-only
|
|
33
|
+
few-shot examples
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Caching operates on the **byte sequence of tokens**. A request hits the cache for as long
|
|
37
|
+
as its bytes are an exact prefix of a previous one, and stops hitting at the first
|
|
38
|
+
differing token.
|
|
39
|
+
|
|
40
|
+
> **The earlier a change lands, the more cache it invalidates.**
|
|
41
|
+
|
|
42
|
+
That single line is the whole discipline. It is why the append-only shape of a trajectory
|
|
43
|
+
is not a stylistic preference: appending changes nothing before the append point, so the
|
|
44
|
+
entire prior context stays a valid prefix. Editing an earlier message discards everything
|
|
45
|
+
after it.
|
|
46
|
+
|
|
47
|
+
**The structural fix follows from the rule and is worth stating as a rule of its own:** a
|
|
48
|
+
configuration change is **appended as a new message**, never applied by editing an earlier
|
|
49
|
+
one. Codex deliberately keeps the old prompt an exact prefix of the new one for this
|
|
50
|
+
reason — the agent loop is quadratic in JSON and linear in sampling *only while cache hits
|
|
51
|
+
hold*.
|
|
52
|
+
|
|
53
|
+
## What a miss costs, measured
|
|
54
|
+
|
|
55
|
+
A customer-service agent at **100,000 conversations/day** was healthy until an engineer
|
|
56
|
+
added `Current time: {{now}}` to the system prompt. Nothing else changed:
|
|
57
|
+
|
|
58
|
+
| | before | after |
|
|
59
|
+
|---|---|---|
|
|
60
|
+
| TTFT | **0.5 s** | **3–5 s** |
|
|
61
|
+
| monthly inference bill | baseline | **nearly doubled** |
|
|
62
|
+
|
|
63
|
+
The line sat near the front of the prefix, so every request differed from that token
|
|
64
|
+
onward and the whole remainder was recomputed and re-billed, every time, for every user.
|
|
65
|
+
|
|
66
|
+
A second measurement from the other direction: on one real user in one day, a harness that
|
|
67
|
+
treats prefix stability as a loop invariant moved **435M input tokens**, at a hit price
|
|
68
|
+
roughly **1/120** of a miss. At that ratio the cache is not a saving — it is the only
|
|
69
|
+
reason the workload is affordable at all.
|
|
70
|
+
|
|
71
|
+
**The general shape:** anything that varies per request and sits early in the prefix is a
|
|
72
|
+
permanent, silent, whole-fleet cost. A timestamp is the classic; a user id, a session
|
|
73
|
+
counter, a randomly ordered list and a freshly retrieved example are the same defect
|
|
74
|
+
wearing different clothes.
|
|
75
|
+
|
|
76
|
+
## Every condition before the boundary doubles the keys
|
|
77
|
+
|
|
78
|
+
Providers let you mark where the cacheable region ends. Content before the boundary can be
|
|
79
|
+
cached **across users and sessions**; content after it is user- or session-specific.
|
|
80
|
+
|
|
81
|
+
The arithmetic is unforgiving: **N binary runtime conditions in front of the boundary
|
|
82
|
+
produce 2^N distinct cache populations**, each of which must be warmed independently.
|
|
83
|
+
|
|
84
|
+
| conditions before the boundary | cache populations |
|
|
85
|
+
|---|---|
|
|
86
|
+
| 1 (e.g. OS) | 2 |
|
|
87
|
+
| 2 (+ debug mode) | 4 |
|
|
88
|
+
| 3 (+ locale) | **8** |
|
|
89
|
+
| 5 | 32 |
|
|
90
|
+
|
|
91
|
+
Three innocuous conditions — macOS/Linux × normal/debug × zh/en — and one warm cache
|
|
92
|
+
becomes eight cold ones. **Push every runtime condition after the boundary**, or accept
|
|
93
|
+
that the prefix is no longer shared.
|
|
94
|
+
|
|
95
|
+
## The named invalidators
|
|
96
|
+
|
|
97
|
+
Each of these changes the prefix and costs everything after it:
|
|
98
|
+
|
|
99
|
+
- **Changing the `tools` array mid-conversation** — adding, removing *or reordering* a
|
|
100
|
+
definition. Tool definitions are hundreds of tokens each and sit in the prefix.
|
|
101
|
+
- **Changing the model.**
|
|
102
|
+
- **Changing sandbox configuration, approval mode, or working directory**, where those are
|
|
103
|
+
rendered into the prompt.
|
|
104
|
+
- **Changing the thinking / reasoning budget.**
|
|
105
|
+
- **Hot-loading a plugin at runtime** — it changes the tools and prompt fragments the model
|
|
106
|
+
sees, so a plugin's documentation owes the reader its impact on the KV cache, not only on
|
|
107
|
+
behaviour.
|
|
108
|
+
- **Replacing the system prompt to switch a role** (`transfer_to_agent`-style). This is a
|
|
109
|
+
real architectural trade, not a mistake: replacing the prompt makes an out-of-scope tool
|
|
110
|
+
*absent* — a hard boundary — where loading a **skill** keeps the static prefix intact and
|
|
111
|
+
gives the model an *instruction* it may ignore. Boundary versus instruction, paid for in
|
|
112
|
+
cache. Decide it deliberately.
|
|
113
|
+
|
|
114
|
+
## Three regions, never mixed
|
|
115
|
+
|
|
116
|
+
The cleanest shape observed in a production harness partitions every request and never
|
|
117
|
+
lets the parts blend:
|
|
118
|
+
|
|
119
|
+
| Region | Holds | Lifetime |
|
|
120
|
+
|---|---|---|
|
|
121
|
+
| **Immutable prefix** | system prompt, tool specs, few-shot examples | computed once per session, pinned |
|
|
122
|
+
| **Append-only log** | `[assistant][tool][assistant]…` | every prior turn stays a prefix of the next |
|
|
123
|
+
| **Volatile scratch** | reasoning traces, transient state | reset each turn, **never sent upstream** |
|
|
124
|
+
|
|
125
|
+
The value is that the invariant becomes checkable: if anything ever writes into region one
|
|
126
|
+
after the session starts, that is a bug with a name, not a mystery in the bill.
|
|
127
|
+
|
|
128
|
+
## Compaction is a cache decision
|
|
129
|
+
|
|
130
|
+
Compression happens **between** API calls, not inside one, and it never touches the static
|
|
131
|
+
prefix — its target is tool results in the history. But every such edit invalidates the
|
|
132
|
+
cache **from the replacement point onward**.
|
|
133
|
+
|
|
134
|
+
So compression frequency is itself a cost decision: **compact in batch when the context
|
|
135
|
+
approaches the threshold, never every round.** A per-round compactor pays a full re-prefill
|
|
136
|
+
every round to save tokens it was already paying for once.
|
|
137
|
+
|
|
138
|
+
`references/context-engineering.md`'s ladder carries this now: four of its five rungs are
|
|
139
|
+
free in *compute* and none is free in *cache*, because each one edits history.
|
|
140
|
+
|
|
141
|
+
**A marker makes the operation idempotent** — a `[COMPRESSED]` tombstone tells the next
|
|
142
|
+
pass what it may skip, so a second compaction does not rewrite the same region and pay the
|
|
143
|
+
invalidation twice.
|
|
144
|
+
|
|
145
|
+
## Four things that look like optimisations and are not
|
|
146
|
+
|
|
147
|
+
- **Sorting tool definitions by usage frequency.** Invalidates from the first moved
|
|
148
|
+
position onward — and the measurement is the useful half: **a fixed order has almost no
|
|
149
|
+
effect on tool-selection accuracy.** The trade is not accuracy against cost. It is cost
|
|
150
|
+
for nothing.
|
|
151
|
+
- **Retrieving "the most relevant few-shot example" per request.** Examples land early in
|
|
152
|
+
the prefix, so once chosen they must be **byte-for-byte identical** across requests.
|
|
153
|
+
Per-request selection is a RAG-flavoured instinct that guarantees a permanent miss.
|
|
154
|
+
- **Deferred tool loading, misread.** `defer_loading` controls *what enters the context
|
|
155
|
+
window, not what you send* — every definition still ships in the `tools` array on every
|
|
156
|
+
request. What it buys is real and specific: deferred tools are excluded from the
|
|
157
|
+
**system-prompt prefix** and discovered ones are appended inline as `tool_reference`
|
|
158
|
+
blocks, so the cached prefix is untouched. That makes it the platform's cache-safe path,
|
|
159
|
+
which the three hand-rolled mitigations in `agent-interop/references/mcp-scale.md` were
|
|
160
|
+
approximating. At least one tool must stay non-deferred — all-deferred is a 400.
|
|
161
|
+
- **"Manual message concatenation breaks the cache."** It does not. Caching is over bytes,
|
|
162
|
+
so a hand-built `"USER: … ASSISTANT: …"` prefix that is byte-stable hits exactly like a
|
|
163
|
+
templated one; it breaks only if the concatenation is unstable. The real cost of
|
|
164
|
+
flattening is different and worse — an **out-of-distribution format**, since the model
|
|
165
|
+
was trained on role-delimited dialogue. Fix the reason, not the myth.
|
|
166
|
+
|
|
167
|
+
**The pinning mechanism, because it is what makes discovery viable at all:** on the turn a
|
|
168
|
+
tool is discovered its schema is appended at the end of the context, and **from then on it
|
|
169
|
+
stays fixed at that position and becomes ordinary history**. It is never moved to the new
|
|
170
|
+
end again — doing so would force a re-prefill every turn and make the whole scheme
|
|
171
|
+
pointless.
|
|
172
|
+
|
|
173
|
+
## Where this constrains the rest of the skill
|
|
174
|
+
|
|
175
|
+
- **§3 (meta-tools) and §10 (dynamic system prompts)** tell you to assemble tools and
|
|
176
|
+
prompt per request from the same capability flags, so the two can never disagree. That is
|
|
177
|
+
right for correctness and silent about cost. The reconciliation: rebuilding is free while
|
|
178
|
+
it is **byte-identical** — which it is whenever the capability set has not moved — and the
|
|
179
|
+
moment capabilities genuinely change mid-session, append the change as a new message
|
|
180
|
+
rather than rewriting the prefix, and take the one miss knowingly.
|
|
181
|
+
- **§2's loop** may not swap the model between iterations of one conversation without
|
|
182
|
+
counting the miss.
|
|
183
|
+
- **`references/context-engineering.md`** owns the ladder; this file owns why its rungs are
|
|
184
|
+
not free.
|
|
185
|
+
- **`agent-interop/references/mcp-scale.md`** owns the tools-array case in an MCP client.
|
|
186
|
+
This file owns the general rule it is an instance of.
|
|
@@ -200,6 +200,55 @@ for attempt in range(1, max_retries + 1):
|
|
|
200
200
|
|
|
201
201
|
---
|
|
202
202
|
|
|
203
|
+
## The tool-calling loop, in full
|
|
204
|
+
|
|
205
|
+
`SKILL.md` §2 states the six steps and the guard; this is the listing they describe. It
|
|
206
|
+
moved here in v0.19.0 to buy the body headroom the §10 cache correction needed — the body
|
|
207
|
+
carries what is read every time, a reference carries what is read once.
|
|
208
|
+
|
|
209
|
+
```python
|
|
210
|
+
max_iter = settings.max_orchestrator_iterations # e.g. 10
|
|
211
|
+
for iteration in range(max_iter):
|
|
212
|
+
# 1. Context pressure management
|
|
213
|
+
messages, did_trim = trim_loop_messages(messages, context_window)
|
|
214
|
+
if should_wrap_up(messages, context_window):
|
|
215
|
+
messages.append(Message(role="system",
|
|
216
|
+
content="IMPORTANT: Stop making tool calls. Compose final answer now."))
|
|
217
|
+
|
|
218
|
+
# 2. LLM call with retry
|
|
219
|
+
llm_resp = await llm_call_with_retry(messages, tools, provider, model)
|
|
220
|
+
|
|
221
|
+
# 3. No tool calls = final answer
|
|
222
|
+
if not llm_resp.tool_calls:
|
|
223
|
+
final_text = llm_resp.content
|
|
224
|
+
break
|
|
225
|
+
|
|
226
|
+
# 4. Dispatch tool calls
|
|
227
|
+
messages.append(Message(role="assistant", content=llm_resp.content,
|
|
228
|
+
tool_calls=llm_resp.tool_calls))
|
|
229
|
+
|
|
230
|
+
# 5. Parallel execution (except sequential-only tools)
|
|
231
|
+
if len(llm_resp.tool_calls) > 1 and not has_sequential_tool:
|
|
232
|
+
results = await asyncio.gather(
|
|
233
|
+
*(handle_tool(tc, context) for tc in llm_resp.tool_calls),
|
|
234
|
+
return_exceptions=True)
|
|
235
|
+
else:
|
|
236
|
+
results = [await handle_tool(tc, context) for tc in llm_resp.tool_calls]
|
|
237
|
+
|
|
238
|
+
# 6. Append tool results
|
|
239
|
+
for tc, (text, sub_result) in zip(llm_resp.tool_calls, results):
|
|
240
|
+
messages.append(Message(role="tool", content=text,
|
|
241
|
+
tool_call_id=tc.id, name=tc.name))
|
|
242
|
+
else:
|
|
243
|
+
# Max iterations reached — compose partial answer from gathered data
|
|
244
|
+
final_text = "I reached maximum analysis steps. Here is what I found..."
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
Read it against §2's critical details: the wrap-up injection at ~70% of the window, in-loop
|
|
248
|
+
trimming at ~80%, compression to 60% and one retry on a token-limit error, and the
|
|
249
|
+
iteration refund that keeps a recoverable provider error from being charged to the guard.
|
|
250
|
+
|
|
251
|
+
|
|
203
252
|
## Context Window Sizes
|
|
204
253
|
|
|
205
254
|
**Do not ship a table of model ids.** This file carried one until 2026-08-15 — nine
|