@qvac/harness 0.2.6 → 0.2.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/gh-stack/SKILL.md +24 -24
- package/.claude/skills/gh-stack/references/commands.md +2 -2
- package/.claude/skills/js-house-style/SKILL.md +89 -68
- package/.claude/skills/kvcache/SKILL.md +7 -7
- package/.claude/skills/pr-create/SKILL.md +58 -15
- package/.claude/skills/pr-review/SKILL.md +83 -14
- package/CLAUDE.md +3 -0
- package/README.md +26 -4
- package/dist/harness.bundle +5 -4
- package/dist-lib/lib/docx/index.d.ts +5 -0
- package/dist-lib/lib/docx/index.js +49 -0
- package/dist-lib/lib/tools/exec/adapters/python/pyodide.js +2 -0
- package/dist-lib/lib/tools/exec/adapters/python/wheels.js +2 -1
- package/dist-lib/lib/tools/exec/adapters/python/worker/client.js +1 -1
- package/docs/decisions/2026-05-29-kvcache-tool-turns.md +75 -0
- package/docs/decisions/2026-06-02-kvcache-tool-turns-reenabled.md +127 -0
- package/docs/decisions/2026-06-23-android-cpu-llm-defaults.md +126 -0
- package/docs/decisions/2026-07-29-assistant-owned-storage-layout.md +88 -0
- package/docs/decisions/2026-08-04-sidecar-orphan-reaping.md +79 -0
- package/docs/decisions/2026-08-11-pyodide-worker-thread.md +68 -0
- package/docs/decisions/2026-08-17-harness-hook-pipeline.md +84 -0
- package/docs/decisions/2026-08-17-harness-outbound-egress-guard.md +176 -0
- package/docs/decisions/2026-08-20-chat-prompt-prefix-is-stable.md +95 -0
- package/docs/decisions/2026-08-20-kvcache-static-placement.md +109 -0
- package/docs/decisions/2026-08-21-vendored-python-wheels.md +85 -0
- package/docs/decisions/2026-08-26-exec-is-not-gated-by-skills.md +64 -0
- package/docs/decisions/2026-08-27-writable-skills-overlay.md +56 -0
- package/docs/decisions/2026-09-01-skill-operation-manifests.md +53 -0
- package/docs/decisions/2026-09-01-usage-breakdown-is-measured.md +78 -0
- package/docs/decisions/2026-09-03-usage-breakdown-reported-by-the-harness.md +136 -0
- package/docs/decisions/INDEX.md +38 -0
- package/docs/decisions/TEMPLATE.md +32 -0
- package/package.json +6 -2
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
---
|
|
2
|
+
id: ADR-2026-08-17-harness-hook-pipeline
|
|
3
|
+
status: active
|
|
4
|
+
introduced_by: PR-000
|
|
5
|
+
supersedes: []
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Harness hooks fold a whole round before any call runs, and fail closed
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
`assistant-harness` exposes four hook phases — `runStart`, `preToolCall`, `postToolCall`,
|
|
13
|
+
`runEnd` — registered in a single process-wide array (`harnessHooks`) and driven by
|
|
14
|
+
`HookPipeline` from inside `completion()`. Hooks receive a deep JSON clone of their payload and
|
|
15
|
+
influence the run only through what they return: new `args` for a pre-hook, a new `result` for a
|
|
16
|
+
post-hook, or a `HookRejection` (`{ error, guidance: true }`).
|
|
17
|
+
|
|
18
|
+
Two ordering rules are durable:
|
|
19
|
+
|
|
20
|
+
- **Pre-hooks fold the whole round before any call in that round executes.** The assistant turn
|
|
21
|
+
written to history replays the folded calls, so history can never disagree with what ran.
|
|
22
|
+
- **Post-hooks fire only after a real `tool.execute()`.** Refused, denied, invalid, guarded,
|
|
23
|
+
capped, and unknown calls never reach a post-hook.
|
|
24
|
+
|
|
25
|
+
A pre-hook may also _repair_ a call: the `args` it returns replace the model's own, and the
|
|
26
|
+
payload carries the tool's declared `parameters` so the repair is schema-driven rather than
|
|
27
|
+
per-tool. `ArgRepairHook` is registered first for that reason — it decodes a structured parameter
|
|
28
|
+
a small model handed over as text (fenced, newline-broken, or missing its outer closer) before
|
|
29
|
+
approval, the activity log, and `execute()` read the call, so all three see the same canonical
|
|
30
|
+
args and the tool itself keeps a strict reader.
|
|
31
|
+
|
|
32
|
+
Failure is closed, not fatal. A throwing tool hook becomes model-facing guidance and the loop
|
|
33
|
+
continues; a throwing `runStart`/`runEnd` hook is logged and ignored, because run authorization
|
|
34
|
+
lives in the policy layer, not here.
|
|
35
|
+
|
|
36
|
+
## Context
|
|
37
|
+
|
|
38
|
+
Tool calls already pass through validation, the trusted-source guard, effect classification, and
|
|
39
|
+
the approval memo. Extending any of those for cross-cutting concerns (redaction, arg rewriting,
|
|
40
|
+
telemetry, per-tool filtering) would have meant editing the gate chain for every new concern.
|
|
41
|
+
|
|
42
|
+
The round-level fold exists because the assistant turn is written to history once, before the
|
|
43
|
+
calls run. Folding per call would let a pre-hook mutate arg N after the turn already recorded the
|
|
44
|
+
model's original wrapper — history and execution would diverge, and the model would be trained by
|
|
45
|
+
its own record to re-emit the un-mutated form.
|
|
46
|
+
|
|
47
|
+
Deep cloning exists because payloads are the run's live objects: model-emitted args, history-bound
|
|
48
|
+
results, the message list. A hook that mutates one in place would leak into the run without
|
|
49
|
+
returning anything, which makes the "influence only through return values" contract unenforceable.
|
|
50
|
+
Payloads are JSON-domain, so a `JSON.parse(JSON.stringify(...))` round-trip severs every reference;
|
|
51
|
+
non-JSON values (`undefined`, functions, `BigInt`, cycles) pass through as-is rather than
|
|
52
|
+
withholding output that already ran.
|
|
53
|
+
|
|
54
|
+
## Alternatives Considered
|
|
55
|
+
|
|
56
|
+
- **Per-call fold (pre(N) after post(N-1)).** Rejected: it desynchronizes the assistant turn from
|
|
57
|
+
what executes. Accepted cost of the round fold is that pre-hooks fire for calls a later gate
|
|
58
|
+
(skill lock, search cap, abort) never runs.
|
|
59
|
+
- **Hooks that throw abort the run.** Rejected: a buggy third-party hook would become a run-level
|
|
60
|
+
denial of service. Failing closed on the _call_ keeps the run recoverable.
|
|
61
|
+
- **In-place mutation of the live payload.** Rejected: no way to distinguish a deliberate change
|
|
62
|
+
from an accidental one, and no way to reject a hook's edit.
|
|
63
|
+
- **Structured-clone / a clone library.** Rejected: payloads are already JSON-domain; a dependency
|
|
64
|
+
buys nothing a `JSON` round-trip does not.
|
|
65
|
+
|
|
66
|
+
## Consequences
|
|
67
|
+
|
|
68
|
+
- A new hook phase must state whether it fires before or after execution, and post-phase hooks
|
|
69
|
+
must never fire for a call that did not execute.
|
|
70
|
+
- Rejection wording must never match the SDK wedge patterns in `models/sdk-faults.ts` — those mark
|
|
71
|
+
a dead native context for eviction, not guidance for the model. A post-hook rejection says the
|
|
72
|
+
tool already executed, so the model does not retry a side effect.
|
|
73
|
+
- With no hook registered a run pays nothing, not even a payload clone. Keep that: `harnessHooks`
|
|
74
|
+
is empty by default.
|
|
75
|
+
- Registration is a process-wide array with no ordering, priority, or per-agent scoping. That is
|
|
76
|
+
deliberate and should be revisited only when a real consumer needs it.
|
|
77
|
+
- Hooks cannot bypass approval, the trusted-source guard, or the denial memo — those gates run on
|
|
78
|
+
the folded args, after the hook.
|
|
79
|
+
|
|
80
|
+
## When Agents Should Read This
|
|
81
|
+
|
|
82
|
+
Read this when changing the paths listed for this ADR in `docs/decisions/INDEX.md` — the tool loop
|
|
83
|
+
in `capabilities/completion.ts`, anything under `lib/hooks/`, or any new gate that must order
|
|
84
|
+
itself against the fold.
|
|
@@ -0,0 +1,176 @@
|
|
|
1
|
+
---
|
|
2
|
+
id: ADR-2026-08-17-harness-outbound-egress-guard
|
|
3
|
+
status: active
|
|
4
|
+
introduced_by: PR-QVAC-23689
|
|
5
|
+
supersedes: []
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Harness tool egress and spawn environments are allowlists, not filters
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
Every outbound URL a harness tool reaches — `http_request` and `mcp_call`, on the
|
|
13
|
+
first request and on each redirect hop — passes `resolveOutboundUrl`
|
|
14
|
+
(`lib/tools/url-guard.ts`) before a socket opens. The guard
|
|
15
|
+
rejects loopback, RFC1918, CGNAT, link-local, cloud-metadata hosts and their IPv6
|
|
16
|
+
equivalents through `lib/tools/private-ip.ts`, and rejects a
|
|
17
|
+
hostname whose resolved addresses land in those ranges. `private-ip.ts`
|
|
18
|
+
default-denies any address it cannot positively classify as global unicast.
|
|
19
|
+
|
|
20
|
+
The check splits along the hook boundary from
|
|
21
|
+
ADR-2026-08-17-harness-hook-pipeline. The model-supplied URL gets a
|
|
22
|
+
synchronous literal-host check from `checkOutboundHost`
|
|
23
|
+
(`lib/tools/url-guard.ts`), run by `EgressGuardHook`
|
|
24
|
+
(`lib/hooks/built-in/egress.ts`), a `preToolCall` hook
|
|
25
|
+
shipped by default in `harnessHooks`
|
|
26
|
+
(`lib/hooks/registry.ts`) so every host gets it without
|
|
27
|
+
registering anything; a rejected call never executes and the rejection
|
|
28
|
+
reaches the model as guidance. The hook does not resolve DNS — it only
|
|
29
|
+
catches a literal blocked host (an IP or bracketed IPv6 literal in the URL
|
|
30
|
+
itself). Resolving the hostname and pinning the connection to the addresses
|
|
31
|
+
the guard approved happens in the tool, which needs those addresses anyway to
|
|
32
|
+
connect: each tool runs `resolveOutboundUrl` itself with a `resolveHost`
|
|
33
|
+
defaulting to `lib/dns-lookup.ts`, so a tool constructed
|
|
34
|
+
directly (bypassing the pipeline) is still guarded. `resolveOutboundUrl`
|
|
35
|
+
returns both the verdict and the resolved address set, which the tool then
|
|
36
|
+
pins the connection to. `http_request` re-runs the
|
|
37
|
+
check on every redirect hop, which only exists mid-execution after the hook
|
|
38
|
+
has fired; `mcp_call` runs it before any credential resolves, so a blocked
|
|
39
|
+
host never gets a token minted for it.
|
|
40
|
+
|
|
41
|
+
On Node, `lib/no-redirect-fetch-node.ts` pins the connection
|
|
42
|
+
to the addresses the guard already approved: it passes them to `http`/`https`
|
|
43
|
+
request's `lookup` callback, which connects only to those addresses and
|
|
44
|
+
rejects private results. On Bare, `lib/no-redirect-fetch-bare.ts`
|
|
45
|
+
does not pin — the fetch agent carries a validating `lookup` hook
|
|
46
|
+
(`lib/validating-lookup.ts`) that re-resolves through the
|
|
47
|
+
same validation, plus a `createConnection` wrapper that re-checks literal-IP
|
|
48
|
+
hosts, so the address it connects to is validated but not necessarily the one
|
|
49
|
+
the guard first approved. That gap is the residual rebinding window on Bare;
|
|
50
|
+
a rebind into a private range is still caught, just by a second resolve
|
|
51
|
+
instead of by reusing the first one.
|
|
52
|
+
|
|
53
|
+
A spawned child receives an environment built by adding to an allowlist, never by
|
|
54
|
+
subtracting from `process.env`. `EXEC_ENV_ALLOWLIST` in
|
|
55
|
+
`lib/helpers/spawn-path.ts` is `PATH`, `HOME`, the temp-dir
|
|
56
|
+
trio, locale, `TZ`, `USER`, `LOGNAME`, the proxy and CA-bundle variables a
|
|
57
|
+
granted binary needs to reach the network at all behind a corporate proxy, plus
|
|
58
|
+
a per-platform set for the Obsidian launchers. Nothing else — no `SSH_*`, no
|
|
59
|
+
`AWS_*`, no host tokens — crosses into a child.
|
|
60
|
+
|
|
61
|
+
The same fail-closed shape covers the two remaining tool inputs that name a host
|
|
62
|
+
resource. `exec`'s shell `workdir` resolves through `resolveKey` against the
|
|
63
|
+
granted `roots`, and with no roots granted it is refused outright rather than
|
|
64
|
+
passed to `spawn`. Skill loading is gated by `SkillCatalogGuardHook`
|
|
65
|
+
(`lib/hooks/built-in/skill-catalog.ts`), shipped by default
|
|
66
|
+
in `harnessHooks` alongside `EgressGuardHook`; a host that loads a catalog
|
|
67
|
+
passes it to the constructor so `name` resolves against the advertised
|
|
68
|
+
catalog, keeping the loadable set equal to the set the model was shown. The
|
|
69
|
+
`skill` tool itself reads through a `Localdrive` rooted at `skillsDir`, which
|
|
70
|
+
refuses traversal and absolute names regardless of the hook.
|
|
71
|
+
|
|
72
|
+
A grant names what a tool may reach; a deny root names what no grant can reach, and
|
|
73
|
+
deny wins. `resolveKey` (`lib/tools/fs/scope.ts`) resolves every
|
|
74
|
+
path against the granted `roots` and then against `deniedRoots`, which the host
|
|
75
|
+
passes and the model cannot influence. Both sides are canonicalized through
|
|
76
|
+
`realpath` before comparison — once per resolution, not once per path segment — so
|
|
77
|
+
a symlink, an uncanonical root, or a `..` walk cannot step around the deny root.
|
|
78
|
+
The walking tools (`file_glob`, `grep`) drop denied paths as they enumerate,
|
|
79
|
+
because a listing must not name what a read may not open. The desktop sidecar and
|
|
80
|
+
the mobile worklet both derive the deny root from the storage layout the host
|
|
81
|
+
already passes (ADR-2026-07-29-assistant-owned-storage-layout), so the app's own
|
|
82
|
+
`credentials.vault`, `vault.key` and plaintext config database sit outside every
|
|
83
|
+
grant without a new host argument.
|
|
84
|
+
|
|
85
|
+
## Context
|
|
86
|
+
|
|
87
|
+
`qvac-app → docs/tool-review-checklist.md` carried SSRF as an unchecked placeholder — "port
|
|
88
|
+
v1 `private-ip.ts`" — while `http_request` and `mcp_call` shipped in desktop and
|
|
89
|
+
mobile builds. Both are model-driven: the URL, the redirect target, the `workdir`
|
|
90
|
+
and the skill name all come from generated arguments, so any host reachable from
|
|
91
|
+
the device was reachable by a prompt. Cloud metadata endpoints and a developer's
|
|
92
|
+
own loopback services are the valuable targets, and both are one plain hostname
|
|
93
|
+
away.
|
|
94
|
+
|
|
95
|
+
An allowlist rather than a denylist for the spawn environment follows from the
|
|
96
|
+
same reasoning applied to a different axis. A denylist has to enumerate every
|
|
97
|
+
secret-bearing variable a host might set, and loses the moment a new tool
|
|
98
|
+
convention appears. The allowlist only has to enumerate what the pinned binaries
|
|
99
|
+
actually need to run, which is short and changes rarely.
|
|
100
|
+
|
|
101
|
+
## Alternatives Considered
|
|
102
|
+
|
|
103
|
+
**Check only the literal URL, and skip DNS resolution.** Cheap, and it stops
|
|
104
|
+
`http://127.0.0.1/`. It does not stop a hostname with an `A` record pointing at
|
|
105
|
+
`127.0.0.1`, which is the standard way this is exploited. Resolving is the
|
|
106
|
+
point of the guard as a whole — the hook's literal check is a fast early
|
|
107
|
+
reject, but the tool must still resolve the hostname and check the resolved
|
|
108
|
+
addresses before it connects, since it is the tool that ends up holding those
|
|
109
|
+
addresses.
|
|
110
|
+
|
|
111
|
+
**Make `resolveHost` a required option.** Rejected: every existing construction
|
|
112
|
+
site would have to pass it, and the one that forgets ships a tool with no guard.
|
|
113
|
+
Defaulting it means the unsafe configuration cannot be expressed by omission.
|
|
114
|
+
|
|
115
|
+
**Put the initial-URL check only in the hook pipeline, not in each tool.**
|
|
116
|
+
Tempting once the pipeline landed: the guard is exactly the cross-cutting
|
|
117
|
+
per-tool gate the pipeline exists for, a rejection there reaches the model as
|
|
118
|
+
guidance without the call executing, and one hook covers every URL-bearing tool
|
|
119
|
+
instead of each tool re-implementing the same preamble. `EgressGuardHook` ships
|
|
120
|
+
by default in `harnessHooks` (`lib/hooks/registry.ts`), so
|
|
121
|
+
every host gets the model-facing guidance without registering anything — but
|
|
122
|
+
the tools still keep their own check, because a tool can be constructed
|
|
123
|
+
directly and called outside the hook pipeline, and that construction site must
|
|
124
|
+
stay guarded on its own. A host that needs to drop the hook builds its own
|
|
125
|
+
hook list instead of using `harnessHooks`; the tool-level check still holds
|
|
126
|
+
either way.
|
|
127
|
+
|
|
128
|
+
**Rely on the redirect-following code to re-enter the guard implicitly.** Rejected
|
|
129
|
+
— redirect handling lives in shared helpers that do not own the resolver, so the
|
|
130
|
+
hop check is written explicitly at each hop instead. A guard that only runs on the
|
|
131
|
+
first URL is bypassed by any open redirect.
|
|
132
|
+
|
|
133
|
+
**Filter `process.env` by pattern (drop anything matching `*TOKEN*`, `*KEY*`).**
|
|
134
|
+
Rejected: it reads as security while depending on naming discipline in software we
|
|
135
|
+
do not control.
|
|
136
|
+
|
|
137
|
+
**Rely on the fs grants alone, and trust hosts not to grant a directory that
|
|
138
|
+
contains the vault.** Rejected: the natural grant on desktop and mobile _is_ the
|
|
139
|
+
user's home or the app sandbox, and both contain the storage root. A boundary that
|
|
140
|
+
holds only while every host picks a narrow grant is not a boundary.
|
|
141
|
+
|
|
142
|
+
## Consequences
|
|
143
|
+
|
|
144
|
+
- A new URL-bearing tool must run `resolveOutboundUrl` itself, on its own target
|
|
145
|
+
and on any URL it derives mid-execution (redirects, discovery), and should be
|
|
146
|
+
added to `EgressGuardHook`'s tool match so the refusal reaches the model
|
|
147
|
+
as guidance. A tool that opens its own socket is outside the boundary and the
|
|
148
|
+
review checklist should catch it.
|
|
149
|
+
- A new host embedding the harness gets `EgressGuardHook` and
|
|
150
|
+
`SkillCatalogGuardHook` by default through `harnessHooks`; registration is
|
|
151
|
+
optional, not required. A host that needs to drop one builds its own hook
|
|
152
|
+
list instead of using `harnessHooks`. Dropping the egress hook costs
|
|
153
|
+
the guidance, not the gate; dropping the catalog hook leaves skill loading
|
|
154
|
+
bounded only by the `skill` tool's `Localdrive` root, so the loadable set is
|
|
155
|
+
the whole skills directory rather than the advertised catalog.
|
|
156
|
+
- New entries in `EXEC_ENV_ALLOWLIST` need a stated reason — the file's comments
|
|
157
|
+
say what each existing entry is load-bearing for, and `PATH` in particular is
|
|
158
|
+
what makes the exec binary allowlist resolvable.
|
|
159
|
+
- The Node fetch path pins each request to the address set returned by the
|
|
160
|
+
egress check. A direct use of the low-level Node fetch adapter without a
|
|
161
|
+
supplied address set performs its own validating lookup before connecting.
|
|
162
|
+
- A new fs-shaped tool takes `deniedRoots` and resolves through `resolveKey`; one
|
|
163
|
+
that walks the tree must also drop denied entries while enumerating, not only
|
|
164
|
+
refuse the eventual read. A new host embedding the harness must pass the deny
|
|
165
|
+
root — omitting it hands the vault to whatever else it grants.
|
|
166
|
+
- `workdir` on a device with no granted directories is an error, not a silent
|
|
167
|
+
fallback to the process cwd. That is intended, and the error text says to use
|
|
168
|
+
absolute paths instead.
|
|
169
|
+
- Revisit if the harness ever needs to reach a private address deliberately — a
|
|
170
|
+
self-hosted MCP server on a LAN, say. That needs an explicit per-device grant,
|
|
171
|
+
not a hole in `private-ip.ts`.
|
|
172
|
+
|
|
173
|
+
## When Agents Should Read This
|
|
174
|
+
|
|
175
|
+
Read this when changing the paths, workflows, runtime boundaries, or platform
|
|
176
|
+
behavior listed for this ADR in `docs/decisions/INDEX.md`.
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
---
|
|
2
|
+
id: ADR-2026-08-20-chat-prompt-prefix-is-stable
|
|
3
|
+
status: active
|
|
4
|
+
introduced_by: PR-1714
|
|
5
|
+
supersedes: []
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# A chat's prompt prefix changes only when the user changes something
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
The system prompt and the tool block a chat sends are byte-identical from one turn to the
|
|
13
|
+
next unless the user changed something: agent config, an installed skill, a device tool
|
|
14
|
+
toggle. The runtime never varies them on its own.
|
|
15
|
+
|
|
16
|
+
`promptPrefixHash` (`lib/capabilities/prompt-prefix.ts`) is the identity of
|
|
17
|
+
that prefix. It rides the harness's per-round `round start` log next to `turns` and `tools`,
|
|
18
|
+
and a harness test asserts consecutive turns of an unchanged agent hash identically while
|
|
19
|
+
enabling a tool moves the hash.
|
|
20
|
+
|
|
21
|
+
The corollary, where a capability is resolved from more than one input: **one predicate
|
|
22
|
+
answers it for every consumer.** Skill activation is resolved by `skillIsActive`
|
|
23
|
+
(`lib/capabilities/completion.ts`), read both by the active catalog that
|
|
24
|
+
fills the skills section and grants tools, and by the skill loader's per-argument allowlist.
|
|
25
|
+
A skill in one and not the other either hides a capability the model was told about or
|
|
26
|
+
advertises one the loader then refuses.
|
|
27
|
+
|
|
28
|
+
## Context
|
|
29
|
+
|
|
30
|
+
The prefix is what the KV cache matches on, so a prefix that moves reprocesses the whole
|
|
31
|
+
chat. That cost is acceptable once, for a change the user made. It is not acceptable per
|
|
32
|
+
turn, and a prefix that moves on its own is usually a capability appearing or disappearing
|
|
33
|
+
rather than a cache problem.
|
|
34
|
+
|
|
35
|
+
The defect that motivated this: `@mention` activation was resolved from the current turn
|
|
36
|
+
only, so a follow-up turn that named no skill lost the skill, the `skill` tool, and every
|
|
37
|
+
tool the skill granted. `@presentations` built a deck; "edit it to make the background
|
|
38
|
+
orange" reached a model with no way to act. The skills section and two tool definitions
|
|
39
|
+
left the prefix on every such turn, which was observable in the cache numbers and observed
|
|
40
|
+
by nobody.
|
|
41
|
+
|
|
42
|
+
The second half is the same failure one level down. Activation had two implementations — the
|
|
43
|
+
catalog filter and the loader's allowlist — and repairing only the first would have listed a
|
|
44
|
+
skill in the prompt that the loader answered with `skill not enabled`.
|
|
45
|
+
|
|
46
|
+
## Alternatives Considered
|
|
47
|
+
|
|
48
|
+
**Sorting the tool block, or otherwise normalizing the prefix, to absorb variation.** It
|
|
49
|
+
would hide exactly the signal this record exists to expose: KV positions are absolute, so an
|
|
50
|
+
inserted or reordered tool invalidates everything after it whatever we hash. The prefix
|
|
51
|
+
should be stable because nothing varies it, not because the hash forgives variation.
|
|
52
|
+
|
|
53
|
+
## Consequences
|
|
54
|
+
|
|
55
|
+
- A change that makes the system prompt or the tool block vary within a chat is a defect,
|
|
56
|
+
whatever motivates it, and the test says so.
|
|
57
|
+
- A mid-chat `CACHE_INIT` or a `cacheTokens` collapse is now attributable: either the logged
|
|
58
|
+
prefix moved, and the config change that moved it can be named, or it did not, and the
|
|
59
|
+
cause is elsewhere.
|
|
60
|
+
- Adding a third consumer of skill activation means extending `SkillActivation`, not
|
|
61
|
+
re-deriving activation locally.
|
|
62
|
+
- **This record does not decide skill-activation semantics.** SKILL-3 has since been
|
|
63
|
+
rewritten: an agent is born with every ready skill and a mention grants nothing, so the
|
|
64
|
+
migration below no longer has a grant to persist. As written at the time,
|
|
65
|
+
[SKILL-3](../product/skills.md#skill-3) required an `@mention` to enable the skill on the
|
|
66
|
+
agent — durably, visibly, and removably — and
|
|
67
|
+
[`qvac-app → docs/plan/tool-safety/p1.md`](../plan/tool-safety/p1.md) §1a-i owns that migration,
|
|
68
|
+
including moving mention detection out of `completion()` to where the user message is
|
|
69
|
+
minted. Until it lands, the harness keeps a chat's already-loaded skills active so a
|
|
70
|
+
follow-up turn does not lose them. That is a runtime repair for the prefix invariant above,
|
|
71
|
+
not the durable mechanism, and it becomes redundant for the mention path once a mention
|
|
72
|
+
persists onto `enabledSkills`.
|
|
73
|
+
- **Accepted risk until §1a-i lands: a mention grant is invisible and cannot be revoked.** The
|
|
74
|
+
only record that the user granted it is the chat transcript, so the Skills UI cannot show it,
|
|
75
|
+
and removing the skill from the agent (`removeSkillFromAgentLists`, which clears both
|
|
76
|
+
`enabledSkills` and `installedSkills`) leaves it active in every chat that already loaded it,
|
|
77
|
+
with the tools it grants, `exec` included. No harness-side signal can fix this: removal clears
|
|
78
|
+
both lists, so a revoked skill is byte-identical to a never-enabled one, and sending
|
|
79
|
+
`installedSkills` to the runtime would not distinguish them either. Uninstalling the skill from
|
|
80
|
+
the _device_ does revoke it everywhere, because the entry then leaves `skillCatalog`.
|
|
81
|
+
- That is a regression in revocability, not just an unbuilt half of SKILL-3. Before, a mention
|
|
82
|
+
grant was revoked by doing nothing, because it evaporated on the next turn, which is the
|
|
83
|
+
defect this record exists to fix. Persisting a mention onto the agent is what makes the grant
|
|
84
|
+
durable _and_ revocable; keeping only the durable half is the interim cost.
|
|
85
|
+
- Not addressed: `historyFromChunks` (`@qvac/core → lib/assistant.ts`) replays only `skill` tool
|
|
86
|
+
results, so a tool-produced file's `attachmentId` never reaches a later turn. Also not
|
|
87
|
+
addressed: the `@mention` preload rewrites the last user turn in memory only, so that turn
|
|
88
|
+
diverges from its persisted form on every later run — a second divergence point downstream
|
|
89
|
+
of the prefix.
|
|
90
|
+
|
|
91
|
+
## When Agents Should Read This
|
|
92
|
+
|
|
93
|
+
Read this before changing what a run puts in the system prompt or the tool list, when adding
|
|
94
|
+
a consumer of skill activation, or when a chat turns out to send a different prefix on
|
|
95
|
+
different turns.
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
---
|
|
2
|
+
id: ADR-2026-08-20-kvcache-static-placement
|
|
3
|
+
status: active
|
|
4
|
+
introduced_by: PR-1770
|
|
5
|
+
supersedes: [ADR-2026-06-02-kvcache-tool-turns-reenabled]
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Chat models load with static tool placement and no sliding window
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
`model-pool.ts` no longer passes `toolsMode` or `n_discarded` in `modelConfig`
|
|
13
|
+
(static placement in PR-1711, the sliding window in PR-1770).
|
|
14
|
+
Models load with `tools: true` only, which selects the SDK's **static**
|
|
15
|
+
placement — the default, and since SDK 0.18 the only one, because that release
|
|
16
|
+
deletes dynamic mode and rejects the key. The per-turn payload is
|
|
17
|
+
`history.slice(savedCount)`, the unsent tail.
|
|
18
|
+
`kvCache: 'chat-<id>'` stays on every completion, unchanged.
|
|
19
|
+
|
|
20
|
+
The two removals ship as separate PRs, deliberately: the dynamic-tools defect
|
|
21
|
+
had to reach a release on its own. Each key removal changes `modelConfig`,
|
|
22
|
+
which re-keys the SDK's identity-by-config `modelId`, so every chat
|
|
23
|
+
cold-primes once per removal — one wave per release rather than one wave for
|
|
24
|
+
both. That cost is accepted in exchange for shipping the fix early.
|
|
25
|
+
|
|
26
|
+
## Context
|
|
27
|
+
|
|
28
|
+
ADR-2026-06-02 chose `toolsMode: 'dynamic'` on the strength of the addon's
|
|
29
|
+
compact-tools anchoring. The 2026-08-18 investigation
|
|
30
|
+
(`docs/kvcache-reference.md`) established that on every shipped model the addon
|
|
31
|
+
half of that feature never ran: the `tools_compact` gate matches architecture
|
|
32
|
+
`qwen3` exactly, and the catalog is `qwen35`/`qwen36`, so the addon logged
|
|
33
|
+
`tools_compact is not supported … ignoring` on every load while the SDK still
|
|
34
|
+
sliced as-if-dynamic. Consequences, measured:
|
|
35
|
+
|
|
36
|
+
- the previous answer was duplicated into the KV every warm turn, and the tool
|
|
37
|
+
block was re-sent with nothing trimming it — three one-line exchanges filled
|
|
38
|
+
a 2048 window;
|
|
39
|
+
- after any cache reset (deleted `.bin`, `configHash`/`modelId` move, new
|
|
40
|
+
device) the dynamic slice still sent only the last exchange, so the model
|
|
41
|
+
silently lost everything older — TTFT unchanged, no error, answer
|
|
42
|
+
confabulated. Under static placement the same reset costs one proportional
|
|
43
|
+
prefill and the fact survives (measured: `promptTokens` 16→65, TTFT
|
|
44
|
+
41→96 ms).
|
|
45
|
+
|
|
46
|
+
The sliding window fails the same way from the other side, and it fails on the
|
|
47
|
+
shipped stack. **A slide is invisible to the SDK**: the cache session records a
|
|
48
|
+
turn's full message count on success and the next turn sends
|
|
49
|
+
`history.slice(savedCount)`, so nothing observes the addon discarding tokens in
|
|
50
|
+
C++ — the discarded span is gone from the live cache and from the saved `.bin`
|
|
51
|
+
while the app still counts it as sent, and the model silently loses the middle
|
|
52
|
+
of the chat. Not sliding also obsoletes the two slide-time addon bugs we
|
|
53
|
+
reported (the reasoning-span wipe, the single-step prefill slide), and 0 is the
|
|
54
|
+
addon's own documented default, so nothing is being opted out of.
|
|
55
|
+
|
|
56
|
+
Upstream has since deleted dynamic mode on both sides (SDK `main` rejects the
|
|
57
|
+
`toolsMode` key at parse time; the addon drops `tools_compact` after 0.39.4),
|
|
58
|
+
and the addon team asked (Slack, 2026-08-19) that `n_discarded != 0` never be
|
|
59
|
+
passed because context shifting is dropped in a future addon. That last one is
|
|
60
|
+
guidance about an unreleased version and carries no weight here: the shipped
|
|
61
|
+
0.39.4 documents the knob as live, default 0, with no deprecation note. The
|
|
62
|
+
decision rests on the defect above, which lands today.
|
|
63
|
+
|
|
64
|
+
## Consequences
|
|
65
|
+
|
|
66
|
+
- A cold cache now falls back to the full history: chats survive cache resets
|
|
67
|
+
at TTFT cost instead of silently forgetting.
|
|
68
|
+
- With no sliding window, a full context surfaces `ContextOverflowError`, and
|
|
69
|
+
the overflow recovery ladder (`lib/context-window.ts`,
|
|
70
|
+
drop-oldest-turns) becomes effective — under dynamic slicing it was inert.
|
|
71
|
+
The planned compact flow (MIGRATION, KV cache follow-ups) is its successor.
|
|
72
|
+
- The tool block is **not** in the primed prefix on the SDK we ship (0.18.2;
|
|
73
|
+
QVAC-23251 landed in 0.17.1) — the prime is the system prompt alone, and the
|
|
74
|
+
block travels with a turn, written once and skipped thereafter. Its names still
|
|
75
|
+
join `configHash`, so a tool-set change (a skill activation, an `enabledTools`
|
|
76
|
+
or `disabledToolsets` edit) re-keys the cache file and costs a cold prime —
|
|
77
|
+
once per change, since the prefix is otherwise stable for the chat
|
|
78
|
+
([2026-08-20-chat-prompt-prefix-is-stable](./2026-08-20-chat-prompt-prefix-is-stable.md)).
|
|
79
|
+
- **Dropping `n_discarded` is what lets that block be skipped at all.** The
|
|
80
|
+
shipped SDK sets `toolBlockEvictable = (modelConfig.n_discarded ?? 0) > 0`,
|
|
81
|
+
because with sliding on the addon's discard window opens exactly where the
|
|
82
|
+
block sits and nothing protects it. So while core-v2 passed the key, the whole
|
|
83
|
+
tool block was re-sent on every turn — measured ~331 tokens for a 4-tool agent.
|
|
84
|
+
This is the second present-day reason for the removal, alongside a slide being
|
|
85
|
+
invisible to the SDK's own `savedCount` accounting.
|
|
86
|
+
- With `n_discarded` at 0 the addon's decode-time slide never engages either: a
|
|
87
|
+
generation that fits at prefill but crosses `ctx_size` mid-stream cannot
|
|
88
|
+
slide, and the overflow ladder cannot recover it — the ladder only retries a
|
|
89
|
+
round that has not yet yielded tokens — so a mid-decode exhaustion ends the
|
|
90
|
+
turn. Rare (bounded by generation length, not chat length); the compact flow's
|
|
91
|
+
proactive trigger is the structural answer. That round still fails with the
|
|
92
|
+
harness's own `CONTEXT_OVERFLOW_MESSAGE` rather than the SDK's text, which
|
|
93
|
+
names the `modelId` hash and blames the prompt — and which the compaction
|
|
94
|
+
trigger would not recognise.
|
|
95
|
+
- **Each removal strands the previous cache file, and nothing sweeps it.** The
|
|
96
|
+
file lives at `{kvCacheDir}/{cacheKey}/{modelId}/{configHash}.bin`, so a
|
|
97
|
+
`modelId` re-key leaves the old `{modelId}/` directory unreferenced — 21–26 MB
|
|
98
|
+
at ctx 2048, more at the desktop default. The SDK's retention sweep cannot
|
|
99
|
+
reach it: eviction plans only keys marked by `markAutoCacheKey`, and the
|
|
100
|
+
custom-key path (`beginCustom`) never marks, so `chat-<id>` is outside
|
|
101
|
+
retention permanently and by design (a config revert may revive the file).
|
|
102
|
+
Pre-existing — every `configHash` move already did this — but two releases in
|
|
103
|
+
a row now do it to every chat at once. The reclaim is the MIGRATION
|
|
104
|
+
"Delete on re-key" item, which deletes the whole key directory (every
|
|
105
|
+
`modelId`/`configHash` under it) on chat deletion and agent re-key; a chat
|
|
106
|
+
that is never deleted and whose agent is never edited keeps its stale
|
|
107
|
+
directories until then.
|
|
108
|
+
- One cold prime per chat on first run after the upgrade (the `modelId`
|
|
109
|
+
re-key) — expected, not a regression.
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
---
|
|
2
|
+
id: ADR-2026-08-21-vendored-python-wheels
|
|
3
|
+
status: active
|
|
4
|
+
introduced_by: PR-1757
|
|
5
|
+
supersedes: []
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# Python packages ship with the app, pinned by hash
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
Every Python distribution a skill declares in `packages` is vendored into the
|
|
13
|
+
app: `wheels/manifest.json` pins each wheel with its sha256,
|
|
14
|
+
`scripts/vendor-python-wheels.mjs` downloads and verifies them at build time,
|
|
15
|
+
and the desktop build copies the tree to `dist/main/python-wheels`. The harness
|
|
16
|
+
receives the directory on argv and installs from it, so a run declaring a pinned
|
|
17
|
+
package makes no network request at all.
|
|
18
|
+
|
|
19
|
+
Two channels need two mechanisms. Lockfile wheels (micropip, pillow, lxml,
|
|
20
|
+
fonttools, typing-extensions, cryptography and its deps) are copied into
|
|
21
|
+
Pyodide's `packageCacheDir` before the interpreter boots, because `loadPackage`
|
|
22
|
+
reads that directory and nothing else. PyPI wheels (fpdf2, pypdf, openpyxl,
|
|
23
|
+
python-docx, python-pptx and their deps) are staged into the interpreter and
|
|
24
|
+
installed by `emfs:` path with `deps=False`, because the manifest already names
|
|
25
|
+
the whole closure and dependency resolution is the step that would reach PyPI.
|
|
26
|
+
|
|
27
|
+
## Context
|
|
28
|
+
|
|
29
|
+
The PDF skill failed on an offline device: `micropip.install` resolves names
|
|
30
|
+
against pypi.org, so `fpdf2` could not be fetched. micropip reports an
|
|
31
|
+
unreachable index as `Can't fetch metadata for 'fpdf2'`, which reads as a wrong
|
|
32
|
+
package name — the model answered it by guessing `fpdf`, `pdf2`, `pdfkit` and
|
|
33
|
+
`reportlab` in turn and never produced a file. Every python-backed skill had the
|
|
34
|
+
same defect, and PyPI wheels were never persisted, so even a device that had
|
|
35
|
+
been online re-downloaded them on the next launch.
|
|
36
|
+
|
|
37
|
+
Pinning was already the skills' convention for reproducibility. Vendoring the
|
|
38
|
+
pinned set turns that into a supply-chain guarantee: the app cannot install a
|
|
39
|
+
version that was not reviewed, and the runtime makes no PyPI request at all.
|
|
40
|
+
|
|
41
|
+
Measured cost for the whole set: 17 wheels, 8,314,167 bytes on disk — 1.8 MB of
|
|
42
|
+
PyPI wheels, 3.9 MB of lockfile wheels, and 2.3 MB for the cryptography closure
|
|
43
|
+
the PDF skill's AES flows need. Two macOS DMGs built from the same tree, one with
|
|
44
|
+
the wheels staged and one without, put the shipped cost at **8,245,247 bytes
|
|
45
|
+
(+3.6%)**: 229,209,450 → 237,454,697. Wheels are zip archives, so the DMG's LZFSE
|
|
46
|
+
pass cannot compress them further and the installer grows by very nearly their
|
|
47
|
+
raw size. Against the 24 MB of Pyodide runtime the app already ships, they are the
|
|
48
|
+
cheap part; lxml, long assumed to be the dominant cost, is 1.6 MB.
|
|
49
|
+
|
|
50
|
+
## Alternatives Considered
|
|
51
|
+
|
|
52
|
+
- **Point Pyodide's `packageCacheDir` at the read-only vendored tree.** Pyodide
|
|
53
|
+
writes downloaded wheels into that directory, so an unvendored package would
|
|
54
|
+
fail on the write rather than on the download, and the online fallback would
|
|
55
|
+
break. Seeding a writable cache keeps both paths working.
|
|
56
|
+
- **Install the vendored wheels and let micropip resolve the rest.** A partial
|
|
57
|
+
install pins versions the resolver then has to work around, and it still needs
|
|
58
|
+
the network for the remainder. The plan is all-or-nothing per request: any
|
|
59
|
+
unvendored package sends the whole install back to PyPI, dependencies included.
|
|
60
|
+
- **Ship the wheels inside `node_modules/pyodide`.** Its `indexURL` is bypassed
|
|
61
|
+
the moment `packageCacheDir` is set, and `bun install` would wipe the tree.
|
|
62
|
+
|
|
63
|
+
## Consequences
|
|
64
|
+
|
|
65
|
+
- A pin change is two edits and a regeneration: the version in `SKILL.md`, the
|
|
66
|
+
version in the manifest, then `node scripts/vendor-python-wheels.mjs --update`.
|
|
67
|
+
The `python-wheels` test suite fails when those disagree, because a silent
|
|
68
|
+
disagreement reverts that skill to online-only.
|
|
69
|
+
- Bumping Pyodide invalidates the lockfile wheels — they carry its ABI tag. The
|
|
70
|
+
vendoring script refuses to run when the manifest and `node_modules/pyodide`
|
|
71
|
+
name different versions.
|
|
72
|
+
- A package no skill declares still installs from PyPI and still needs a
|
|
73
|
+
connection. micropip reports a missing name and an unreachable index
|
|
74
|
+
identically, so `packageFetchFailedHint` names both causes and tells the model
|
|
75
|
+
not to try other spellings, which is the part that ends the loop.
|
|
76
|
+
- Mobile is unchanged: no Pyodide runtime ships there, so no wheels are passed.
|
|
77
|
+
Vendoring for mobile means shipping the interpreter first.
|
|
78
|
+
- The desktop build needs network access at package time to fill the wheel tree.
|
|
79
|
+
A hash mismatch fails the build rather than shipping an unreviewed wheel.
|
|
80
|
+
|
|
81
|
+
## When Agents Should Read This
|
|
82
|
+
|
|
83
|
+
Read this when changing the Python exec adapter's install path, the vendoring
|
|
84
|
+
script or manifest, the `packages` pins in a skill, the Pyodide dependency, or
|
|
85
|
+
the desktop build steps that stage harness assets.
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
---
|
|
2
|
+
id: ADR-2026-08-26-exec-is-not-gated-by-skills
|
|
3
|
+
status: active
|
|
4
|
+
introduced_by: PR-1854
|
|
5
|
+
supersedes: []
|
|
6
|
+
---
|
|
7
|
+
|
|
8
|
+
# A skill no longer decides what `exec` may run; approval does
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
|
|
12
|
+
`exec` is reachable whenever the agent's Tools settings reach it, and it runs whatever
|
|
13
|
+
command it is given. The two gates that made a skill the source of that permission are
|
|
14
|
+
gone:
|
|
15
|
+
|
|
16
|
+
- the call-time lock that refused a granted tool until the granting skill's body had
|
|
17
|
+
been loaded in the same chat, and
|
|
18
|
+
- the per-turn binary allowlist, which built the set of runnable programs out of the
|
|
19
|
+
`exec(...)` scopes of the active skills and refused everything else.
|
|
20
|
+
|
|
21
|
+
What remains in front of a command is the sanitizer — one program per call, no
|
|
22
|
+
chaining, pipes, redirection or shell expansion — and the approval prompt, which every
|
|
23
|
+
`exec` write raises unless the agent is in Skip All Approvals.
|
|
24
|
+
|
|
25
|
+
A skill's `exec(bin)` scope still means something narrower: it names the bin whose
|
|
26
|
+
`cli.schema.json` validates argv and carries the `x-effect: read` annotations that let
|
|
27
|
+
a read run unattended. It no longer decides whether the command may run at all.
|
|
28
|
+
|
|
29
|
+
## Context
|
|
30
|
+
|
|
31
|
+
The agent's Tools tab lists Exec as a capability the user switches on, with the hint
|
|
32
|
+
"Lets QV.AC run commands on the device" ([PERM-10](../product/tool-permissions.md#perm-10)).
|
|
33
|
+
That switch writes the tool's name into the agent's allowlist and nothing else. It
|
|
34
|
+
could not add a binary to the allowlist or load a skill body, so switching Exec on left
|
|
35
|
+
`exec` hidden from the model, and a model that reached it anyway got told to load a
|
|
36
|
+
skill it may not have. The user's switch said yes and the gate said no.
|
|
37
|
+
|
|
38
|
+
Both gates were built when a skill was the only way a tool became available
|
|
39
|
+
(PR-1426 for the lock, and the scope work before it). That premise is what changed.
|
|
40
|
+
|
|
41
|
+
## Consequences
|
|
42
|
+
|
|
43
|
+
- **Approval is the boundary.** With no argv schema for an arbitrary bin, `shellEffect`
|
|
44
|
+
cannot classify a command as a read, so it classifies write and the call prompts. An
|
|
45
|
+
agent on Skip All Approvals runs shell commands unattended; that mode already carried
|
|
46
|
+
that meaning for the granted binaries, and now carries it for all of them.
|
|
47
|
+
- **`exec(python)` no longer gates Pyodide.** `language: "python"` runs without any
|
|
48
|
+
grant. The realm it runs in is unchanged, so
|
|
49
|
+
[PERM-7](../product/tool-permissions.md#perm-7) is exactly as met, or unmet, as it
|
|
50
|
+
was; what changed is that reaching it no longer requires a skill. The mitigation is
|
|
51
|
+
still [tool-safety p3](../plan/tool-safety/p3.md).
|
|
52
|
+
- **The argv schema now reads backwards.** `resolveShellOps` looks the bin up in
|
|
53
|
+
`grantedOps`, which only active skills fill. With apple-notes on,
|
|
54
|
+
`osascript -e 'do shell script "id"'` is refused before approval; with it off, the same
|
|
55
|
+
command sanitizes clean and spawns. Enabling a skill makes `exec` narrower than leaving
|
|
56
|
+
it off. That is kept deliberately: the schema is how a skill states the shape it expects
|
|
57
|
+
its own CLI to be called in, and it bounds nothing that running the command directly
|
|
58
|
+
would not already reach. It stops being defensible the moment anything treats it as a
|
|
59
|
+
security control.
|
|
60
|
+
- **The model is told the truth.** `exec`'s description no longer promises an allowlist,
|
|
61
|
+
and a loaded skill's tool note no longer claims its scopes are the only runnable
|
|
62
|
+
executables.
|
|
63
|
+
- The `skillGated` and `scopeRequired` wire fields survive, reported off, so a build on
|
|
64
|
+
the other side of the sidecar hop reads both gates as open rather than missing.
|