@qvac/harness 0.2.5 → 0.2.7

package/docs/decisions/2026-09-03-usage-breakdown-reported-by-the-harness.md

@@ -0,0 +1,136 @@
+---
+id: ADR-2026-09-03-usage-breakdown-reported-by-the-harness
+status: active
+introduced_by: PR-2067
+supersedes: [ADR-2026-09-01-usage-breakdown-is-measured]
+---
+
+# The device that ran the turn reports the Usage breakdown; the app only renders it
+
+## Decision
+
+The Usage modal's rows come from the run. The harness attributes the prompt it
+just assembled — `systemPrompt`, `agentInstructions`, `activeSkills`, `tools` —
+measured as `.length` on the strings it sent, converts each to tokens at the
+measured density **for that kind of text**, and ships the result inside the
+existing `metrics` JSON of the terminal run status
+(`lib/capabilities/usage-breakdown.ts`).
+
+A prompt row is never a share of the measured total. It is its own characters at
+its own density, so it holds still across every turn of a chat while the
+conversation grows — the property QA asks for. `conversation` is what is left of
+`cacheTokens` once the prompt rows are subtracted, which is also where the costs
+no segment authors end up: the template's per-message wrappers (~28 tokens a
+message on Qwen3.5) and the reply the run just generated, which the cache holds.
+
+`app-runtime` re-derives nothing. `buildUsageSegments` reads the newest reported
+breakdown off the messages and maps it to labels and colors. Conversation is the
+remainder against the live `usedTokens`, never a reported number, so occupancy
+the report cannot know about lands on the turns rather than skewing a prompt row.
+
+Three constants per template, keyed off `detectToolDialect`: prose chars/token
+(the prompt body, the bio, the skills list), schema chars/token (the serialized
+SDK function schemas), and the tool block's token count. Measured on
+Qwen3.5-2B-Q4_K_M as `{ prose: 4.38, schema: 4.08, toolBlockTokens: 189 }` by
+least squares over the addon's own `cacheTokens` across 14 tool/skill/bio
+configurations x 3 turns — 42 samples, leave-one-configuration-out worst error
+5.9%. An unmeasured template borrows these rather than reporting zeros.
+
+Two things the previous design needed are gone: the per-chat chars-per-token fit
+(`usage-calibration.ts`) and the app's copy of the selection gates
+(`usage-segment-system-prompt.ts`, and with it the drift test that policed the
+copy). `TURN_OVERHEAD_TOKENS` is gone too, and so is the 935-char framing
+constant — the tool block is a fixed string, so it is declared in tokens.
+
+The row for attachment filenames is retired. Those labels are composed into the
+user turn by the assistant layer, so they are conversation characters; the app
+was counting raw filenames that never appear in the prompt verbatim.
+
+## Context
+
+Every row except the total was re-derived app-side by re-running
+`permittedTools`, `enabledTools` and `skillIsActive` against a device tool
+report and a skill catalog. The mirror was only ever as current as the last
+person to remember it existed, and because the fit normalizes the known
+characters onto the measured total, an input that goes missing does not read as
+zero in place — it silently reweights every other row.
+
+That failure shipped. After PR-1960 the six provider skills declared tools
+compiled from `operations.json`, so on any host that had not wired
+`compileOperationTools` (the mobile worklet until QVAC-24473; the in-process
+path still) `isToolSupported` marked them unavailable, the estimator dropped
+them, their tool grants went with them, and the System Prompt row absorbed the
+entire window: 2,361 of 2,616 tokens, with Active Skills and Tools at 0.
+
+No tokenizer is exposed — `@qvac/sdk` 0.18.2 ships tokenization only for VLA,
+whose docs tell consumers to tokenize their own input — so `cacheTokens` remains
+the only ground truth and the rows remain shares of it. What changed is where
+the characters are counted: at the point the prompt is built, where being wrong
+is not possible without also sending a different prompt.
+
+## Alternatives Considered
+
+- **Keep the mirror, add a stronger drift test.** Rejected: the drift test runs
+  against a stubbed completion, so it cannot see inputs that arrive over the
+  wire — exactly the class that broke. It also grows a second maintenance
+  surface for every prompt change.
+- **Report chars, fit in the app.** The fit needs the first metrics-bearing turn
+  of the chat, whose prompt the harness can no longer reconstruct. Rejected
+  anyway: per-turn attribution against that turn's own measurement needs no fit,
+  and the prompt rows stay flat regardless (the shares move together).
+- **Report tokens per segment.** Requires a tokenizer we do not have. This is
+  the shape to move to when the SDK grows one: the harness swaps its char counts
+  for tokenize calls and nothing downstream changes.
+- **One density for the whole prompt** (what shipped first, and what QA caught).
+  Rejected on measurement: with rows taken as shares of the total, a chat whose
+  tool schemas are dense JSON and whose turns are prose re-weights every row as
+  the mix changes. Three "hi" turns moved Tools 3,166 -> 3,252 -> 3,268 with an
+  unchanged tool set, and fitting one density across the matrix above is off by
+  up to 46% against six per-kind constants' 1.7%.
+- **Keep the attachment-filenames row, reported by the assistant layer.** Held
+  back: it would put a third layer in the attribution path for a row that reads
+  zero in every chat we have looked at.
+
+## Consequences
+
+- A prompt-shaping change cannot desync the breakdown. There is nothing to keep
+  in sync: the numbers are measured off the prompt that was sent.
+- A delegated chat is attributed by the device that ran it, so the observer no
+  longer needs the host's tool catalog or skill names to price the window.
+  Everything that fed the estimator therefore loses its only reader — the
+  reported tool catalog (`agentToolCatalog`, `selectToolCatalog`,
+  `selectActiveAgentHostToolCatalog`, the `skill.toolCatalog` slice and the
+  `coreV2.tools({})` call that filled it) and the app-side fs-grant mirror
+  (`agent.actor.fs-grants.ts`, its actions, its reducer branches,
+  `state.agent.fsGrants`, its selectors and `AgentFsGrants`/`NO_FS_GRANTS`).
+  PR-2068 deletes them. The real grant plumbing in `@qvac/core → lib/fs-grants.ts`
+  stays either way — that is what gives the harness its roots.
+- The rows still sum exactly to the measured total, and the prompt rows still
+  hold still across turns of one chat. Both are asserted by QA (QA-723).
+- The app parses only the four prompt rows out of the report. The run also sends
+  its own `conversation` and `totalTokens`, which stay useful in a log or a
+  debug view, but requiring them here would turn a harness that trimmed either
+  one into no breakdown at all.
+- A turn recorded before this change, or run by an older peer, carries no
+  breakdown: the modal shows the window on Conversation and holds the System
+  Prompt row back rather than inventing a split. There is no estimator fallback
+  on purpose — a stale mirror is the defect this removes.
+- `promptChars` on the reported tool catalog now has no reader, and neither does
+  the `toolCatalog` field on a device row (still parsed in
+  `device/core-v2-device.ts`, still listed in the Core v2 compatibility matrix).
+  Both stay until one schema bump retires them together.
+- Re-measure the densities by driving `Harness.completion` headlessly with the
+  real SDK (no TUI or pty needed: it takes `model`, `agent` and `messages`
+  directly), reading the char set the harness logs beside the `cacheTokens` the
+  addon reports, and fitting the four terms by least squares. **One fresh SDK
+  home per configuration, one configuration per process**: the SDK persists cache
+  state under its home, and `cacheTokens` counts that whole occupancy, so a reused
+  home reports the same prompt as 152, 219 or 469 tokens. That also means a
+  device's total can include cache another chat left behind — a separate question
+  from this attribution, and the reason `conversation` is the remainder rather
+  than a reported number.
+
+## When Agents Should Read This
+
+Read this when changing the paths, workflows, runtime boundaries, or platform
+behavior listed for this ADR in `docs/decisions/INDEX.md`.

package/docs/decisions/INDEX.md

@@ -0,0 +1,38 @@
+# Harness Architecture Decision Records
+
+Decision records for `@qvac/harness`: the inference engine and the tool runtime
+skills execute in. They answer HOW we chose to build something, when the reason
+is not obvious from the code or the PR that introduced it.
+
+Read this index first. Open a record only when the touched paths, area, or
+keywords match the task at hand.
+
+Paths are relative to this repository. A path prefixed `@qvac/core →`,
+`@qvac/sdk →` or `qvac-app →` lives in another repository and is context, not a
+local file. `node scripts/docs-audit.cjs` checks this directory.
+
+## Active Decisions
+
+| Status | Area                                        | Paths                                                                                                                                                                                                                                                                                                                                                                                                                         | Decision                                                                                                                                     | Keywords                                                                                                                                                                                                                                                                                                                                                                                |
+| ------ | ------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| active | assistant, harness, skills, security        | `lib/tools/operations/manifest.ts`, `lib/tools/operations/compile.ts`, `lib/tools/operations/builders.ts`, `lib/tools/http-request.ts`, `lib/tools/http-request/truncate-response.ts`, `lib/capabilities/tool-selection.ts`, `skills/gmail/operations.json`, `skills/google-calendar/operations.json`                                                                                                                         | [A provider write is a compiled skill operation, not a shaping hook in `http_request`](./2026-09-01-skill-operation-manifests.md)            | `operations.json`, `compileOperationTools`, `parseOperationsManifest`, `OperationSpec`, `BUILDERS`, `gmail_send`, `gmail_draft`, `calendar_create_meet_event`, `calendar_add_meet`, `owningSkill`, `grantedOwners`, `host shaping`, `shapeRequest`, `truncateResponseBody`, `fitJson`, `MAX_RESPONSE_CHARS`                                                                             |
+| active | assistant, harness, skills, security        | `lib/tools/skills/overlay-dir.ts`, `lib/tools/skills/catalog.ts`, `lib/tools/skills/credential-grants.ts`, `lib/tools/skill.ts`                                                                                                                                                                                                                                                                                               | [A user-added skill layers over the bundled base and is trusted only by shadow](./2026-08-27-writable-skills-overlay.md)                     | `skills overlay`, `writable overlay`, `resolveSkillsOverlayDir`, `loadSkillCatalog`, `mergeOverlay`, `origin overlay`, `shadow`, `credentialGrants`, `mcpReadMethods`, `OVERLAY_DEFAULT_PLATFORMS`, `platform gate`, `untrusted skill`, `user-added skill`, `not available on this device`                                                                                              |
+| active | assistant, harness, security, product       | `lib/tools/exec.ts`, `lib/tools/exec/scope.ts`, `lib/capabilities/completion.ts`, `lib/capabilities/tool-selection.ts`, `lib/tools/skill.ts`                                                                                                                                                                                                                                                                                  | [A skill no longer decides what `exec` may run; approval does](./2026-08-26-exec-is-not-gated-by-skills.md)                                  | `exec`, `skillGated`, `scopeRequired`, `grantingSkills`, `grantedScopes`, `toolScopes`, `checkScopedCommand`, `checkPythonScope`, `sanitizeCommand`, `locked until you load the skill`, `binary allowlist`, `RESTRICTED`, `approval is the boundary`, `PERM-7`, `PERM-10`                                                                                                               |
+| active | assistant, harness, runtime, inference      | `lib/capabilities/prompt-prefix.ts`, `lib/capabilities/completion.ts`, `lib/tools/skills/mention.ts`                                                                                                                                                                                                                                                                                                                          | [A chat's prompt prefix changes only when the user changes something](./2026-08-20-chat-prompt-prefix-is-stable.md)                          | `promptPrefixHash`, `prompt prefix`, `round start`, `cacheTokens collapse`, `CACHE_INIT`, `KV cache re-prime`, `skillIsActive`, `SkillActivation`, `activeSkills`, `hasActiveSkills`, `loadedSkillsFrom`, `skillBlocked`, `blockedSkillMessage`, `skills section`, `turn-scoped activation`, `SKILL-3`                                                                                  |
+| active | assistant, harness, runtime                 | `lib/hooks/**`, `lib/hooks/built-in/arg-repair.ts`, `lib/capabilities/completion.ts`                                                                                                                                                                                                                                                                                                                                          | [Harness hooks fold a whole round before any call runs, and fail closed](./2026-08-17-harness-hook-pipeline.md)                              | `HarnessHook`, `HookPipeline`, `harnessHooks`, `runHooks`, `preToolCall`, `postToolCall`, `runStart`, `runEnd`, `HookRejection`, `guidance`, `round fold`, `deep clone`, `fail closed`, `tool executed; output withheld`, `abandoned`, `ArgRepairHook`, `arg repair`, `jsonrepair`, `parameters`                                                                                        |
+| active | assistant, storage, desktop, mobile         | `@qvac/core → lib/storage.ts`, `@qvac/core → lib/image-cache-dir.ts`, `@qvac/core → lib/sdk-storage.ts`, `qvac-app → desktop/electron/main.ts`, `qvac-app → mobile/worker/assistant-worklet.mjs`                                                                                                                                                                                                                              | [Assistant-owned storage separates shared models from private KV](./2026-07-29-assistant-owned-storage-layout.md)                            | `bare-storage`, `models`, `~/.qvac/models`, `SDK shared cache`, `image cache`, `KV cache`, `sdk-home`, `release channel`, `QVAC_MODEL_CACHE_PATH`, `Bare.argv[0]`, `SNAP_USER_COMMON`, `no migration`                                                                                                                                                                                   |
+| active | runtime, inference, harness                 | `lib/models/model-pool.ts`, `lib/context-window.ts`, `@qvac/core → lib/assistant.ts`, `lib/capabilities/completion.ts`                                                                                                                                                                                                                                                                                                        | [Static tool placement, no sliding window](./2026-08-20-kvcache-static-placement.md)                                                         | `kvCache`, `prompt cache`, `static placement`, `toolsMode`, `n_discarded`, `sliding window`, `savedCount`, `cold prime`, `modelId re-key`, `cache reset`, `overflow ladder`                                                                                                                                                                                                             |
+| active | runtime, inference, mobile, harness         | `qvac-app → mobile/lib/android-llm-device.ts`, `qvac-app → mobile/lib/core-v2-harness-config.ts`, `index.ts`, `lib/models/kv-cache.ts`, `lib/models/opencl-cache.ts`                                                                                                                                                                                                                                                          | [Android LLM defaults to CPU except Tensor-era Pixels](./2026-06-23-android-cpu-llm-defaults.md)                                             | `Android`, `Pixel 9`, `Pixel 10`, `Pixel Tablet`, `Pixel Fold`, `Adreno`, `Vulkan`, `OpenCL`, `openclCacheDir`, `kernel cache`, `backendDevice`, `Snapdragon`, `CPU inference`, `GPU inference`, `KV cache`, `q4_0`, `cache-type-k`, `cache-type-v`, `gpu_layers`, `transcriptionDevice`, `parakeet`, `useGPU`, `empty transcript`                                                      |
+| active | runtime, harness, mobile, desktop           | `lib/tools/exec/adapters/python/**`, `lib/tools/exec/scope.ts`                                                                                                                                                                                                                                                                                                                                                                | [Pyodide runs in a worker thread with a shared interrupt buffer](./2026-08-11-pyodide-worker-thread.md)                                      | `Pyodide`, `python worker`, `Bare.Thread`, `bare-channel`, `SharedArrayBuffer`, `setInterruptBuffer`, `KeyboardInterrupt`, `terminate`, `join`, `stowed bundle`, `QVAC_PYODIDE_WORKER_ENTRY`, `QVAC_PYODIDE_TEST_LOADER`, `exec`, `in-process fallback`                                                                                                                                 |
+| active | runtime, harness, desktop, build, security  | `wheels/manifest.json`, `lib/tools/exec/adapters/python/wheels.ts`, `scripts/vendor-python-wheels.mjs`, `qvac-app → desktop/electron.vite.config.ts`                                                                                                                                                                                                                                                                          | [Python packages ship with the app, pinned by hash](./2026-08-21-vendored-python-wheels.md)                                                  | `python wheels`, `micropip`, `packageCacheDir`, `emfs:`, `deps=False`, `vendored wheels`, `offline`, `PyPI`, `sha256`, `supply chain`, `QVAC_PYTHON_WHEELS_DIR`, `fpdf2`, `pypdf`, `openpyxl`, `python-docx`, `python-pptx`, `cryptography`                                                                                                                                             |
+| active | desktop, harness, runtime                   | `qvac-app → desktop/electron/assistant.ts`, `qvac-app → desktop/electron/assistant-sidecar-lifecycle.ts`, `qvac-app → desktop/electron/assistant-sidecar-process.ts`, `worker/harness-sidecar.mjs`                                                                                                                                                                                                                            | [The host reaps the sidecar's process group; the harness cannot reap itself](./2026-08-04-sidecar-orphan-reaping.md)                         | `orphan`, `process group`, `detached`, `SIGKILL`, `reparent to init`, `taskkill`, `Job Object`, `libuv`, `KILL_ON_JOB_CLOSE`, `orphan watchdog`, `os.ppid`, `model memory leak`, `killStowChildren`                                                                                                                                                                                     |
+| active | harness, security, runtime, mobile, desktop | `lib/hooks/built-in/egress.ts`, `lib/hooks/built-in/skill-catalog.ts`, `lib/hooks/registry.ts`, `lib/tools/url-guard.ts`, `lib/tools/private-ip.ts`, `lib/dns-lookup.ts`, `lib/validating-lookup.ts`, `lib/no-redirect-fetch-bare.ts`, `lib/no-redirect-fetch-node.ts`, `lib/helpers/spawn-path.ts`, `lib/tools/http-request.ts`, `lib/tools/mcp-call.ts`, `lib/tools/exec.ts`, `lib/tools/skill.ts`, `lib/tools/fs/scope.ts` | [Harness tool egress and spawn environments are allowlists, not filters](./2026-08-17-harness-outbound-egress-guard.md)                      | `SSRF`, `resolveOutboundUrl`, `private-ip`, `isPrivateOrBlocked`, `DNS rebinding`, `validating lookup`, `resolveHost`, `lookupHost`, `bare-dns`, `metadata endpoint`, `redirect hop`, `EXEC_ENV_ALLOWLIST`, `spawn env allowlist`, `workdir`, `resolveKey`, `granted roots`, `deniedRoots`, `canonicalDenyRoots`, `deny outranks grant`, `storage root`, `skill catalog`, `fail closed` |
+| active | harness, app-runtime, desktop, mobile, chat | `lib/capabilities/usage-breakdown.ts`, `lib/capabilities/completion.ts`, `qvac-app → app-runtime/src/chat/usage-segment.ts`, `qvac-app → app-runtime/src/chat/chunk-to-turn.ts`, `qvac-app → app-runtime/src/agent/**`, `qvac-app → app-runtime/src/skill/skill.actor.catalog.ts`                                                                                                                                             | [The device that ran the turn reports the Usage breakdown; the app only renders it](./2026-09-03-usage-breakdown-reported-by-the-harness.md) | `Usage modal`, `usage segments`, `usageBreakdown`, `promptUsageParts`, `buildUsageSegments`, `cacheTokens`, `charsPerToken`, `FRAMING_CHARS_BY_DIALECT`, `toolBlockFramingChars`, `detectToolDialect`, `chat template framing`, `PROMPT_SEND`, `context used`, `no tokenizer`, `metrics json`                                                                                           |
+
+## Superseded Decisions
+
+| Status     | Area | Paths | Decision                                                                                                               | Keywords |
+| ---------- | ---- | ----- | ---------------------------------------------------------------------------------------------------------------------- | -------- |
+| superseded |      |       | [KV-cache session is disabled for tool-using completions](./2026-05-29-kvcache-tool-turns.md)                          |          |
+| superseded |      |       | [KV-cache session is re-enabled for tool-using completions](./2026-06-02-kvcache-tool-turns-reenabled.md)              |          |
+| superseded |      |       | [The Usage breakdown divides a measured total, it never estimates tokens](./2026-09-01-usage-breakdown-is-measured.md) |          |

package/docs/decisions/TEMPLATE.md

@@ -0,0 +1,32 @@
+---
+id: ADR-YYYY-MM-DD-short-name
+status: active
+introduced_by: PR-000
+supersedes: []
+---
+
+# Short Decision Title
+
+## Decision
+
+State the durable technical choice in one or two short paragraphs.
+
+## Context
+
+Explain the constraints or trade-offs that made this decision necessary.
+
+## Alternatives Considered
+
+Optional. Include only rejected approaches that future maintainers or agents
+are likely to rediscover.
+
+## Consequences
+
+- What future changes must preserve.
+- What this deliberately does not solve.
+- What should trigger revisiting the decision.
+
+## When Agents Should Read This
+
+Read this when changing the paths, workflows, runtime boundaries, or platform
+behavior listed for this ADR in `docs/decisions/INDEX.md`.