cowork-harness 0.28.0 → 0.29.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +22 -12
- package/.claude/skills/cowork-harness/references/ci-recipe.md +4 -4
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +16 -15
- package/.claude/skills/cowork-harness/references/task-recipes.md +1 -1
- package/.claude/skills/cowork-harness/scripts/scenario.py +46 -2
- package/AGENTS.md +5 -4
- package/CHANGELOG.md +138 -0
- package/DESIGN.md +2 -2
- package/README.md +30 -13
- package/SPEC.md +13 -6
- package/baselines/desktop-1.20186.0.json +373 -0
- package/baselines/prompts/cowork-system-prompt-fingerprints.json +16 -0
- package/dist/agent/session.js +112 -18
- package/dist/agent/timeline.js +12 -8
- package/dist/assert.js +90 -25
- package/dist/baseline.js +1 -1
- package/dist/cli.js +26 -6
- package/dist/critique/evaluator.js +147 -25
- package/dist/critique/evidence.js +135 -20
- package/dist/critique/package-evidence.js +79 -22
- package/dist/decide/llm-transport.js +26 -3
- package/dist/decide/semantic-judge.js +15 -5
- package/dist/egress/sidecar.js +6 -7
- package/dist/prompt.js +16 -0
- package/dist/run/artifacts.js +186 -47
- package/dist/run/cassette.js +104 -15
- package/dist/run/chat-result.js +12 -2
- package/dist/run/chat.js +21 -1
- package/dist/run/execute.js +41 -12
- package/dist/run/matrix.js +1 -1
- package/dist/run/pre-run-manifest.js +72 -13
- package/dist/run/renderer.js +9 -1
- package/dist/run/run-index.js +10 -1
- package/dist/run/run-status.js +49 -6
- package/dist/run/run.js +73 -8
- package/dist/run/scenario-tool.js +4 -8
- package/dist/run/trace-view.js +9 -1
- package/dist/run/verdict.js +10 -1
- package/dist/runtime/argv.js +4 -2
- package/dist/runtime/resource-sampler.js +78 -21
- package/dist/sync/cowork-sync.js +214 -19
- package/dist/types.js +16 -10
- package/dist/usage-limit.js +40 -0
- package/docs/boundary.md +5 -0
- package/docs/cassette.md +36 -10
- package/docs/chat.md +8 -0
- package/docs/cowork-spawn-contract-1.12603.1.md +2 -2
- package/docs/invariants.md +4 -0
- package/docs/maintenance.md +11 -0
- package/docs/protocol.md +24 -7
- package/docs/run-status.md +8 -2
- package/docs/scenario.md +16 -14
- package/docs/stats.md +7 -0
- package/examples/replays/README.md +1 -1
- package/examples/replays/example-multiselect-gate.cassette.json +1 -1
- package/examples/replays/example-pdf-skill.cassette.json +1 -1
- package/examples/replays/hostloop-computer-links.cassette.json +1 -1
- package/package.json +1 -1
- package/schema/run-result.json +29 -2
- package/schema/scenario.schema.json +23 -11
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 0.
|
|
7
|
-
tracks-harness: cowork-harness 0.
|
|
6
|
+
version: 0.29.0
|
|
7
|
+
tracks-harness: cowork-harness 0.29.0 (baseline desktop-1.20186.0)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,8 +22,8 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.
|
|
26
|
-
> `desktop-1.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.29.0` (baseline
|
|
26
|
+
> `desktop-1.20186.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
27
27
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
29
29
|
## Preflight — make sure the harness can actually run
|
|
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
|
|
|
38
38
|
Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.
|
|
41
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.29.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.29.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.29.0"`. **Pin `@>=0.29.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.29.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator (passes when the resolved value deep-equals one of the listed members — value ∈ your list, not the reverse), `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, `computer_links_resolve` (new in 0.22.0), and `semantic_matches` (an LLM judge grades a fixed `rubric` of claims against the run's answer — **live-only**, so it is evidence-unavailable / skipped-loud on replay, never a vacuous pass; new in 0.27.0), and **glob-matched** `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` (a pattern like `mcp__workspace__*` matches any tool in the family; exact names match exactly; new in 0.28.0).)
|
|
42
42
|
- **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
|
|
43
43
|
- **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
|
|
44
44
|
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
@@ -63,7 +63,7 @@ reproducible regression (Part II), and **debug** a run that misbehaved or greene
|
|
|
63
63
|
the path, `trace <run-id>` finds it). **Localize the failure post-hoc** from that evidence:
|
|
64
64
|
`cowork-harness trace <run-dir>`'s views + the emitted `result.json` to see what the run actually did,
|
|
65
65
|
then `verify-run` to re-check a suspect assertion — all token-free, no Docker, no re-record. This is
|
|
66
|
-
the loop 0.
|
|
66
|
+
the loop 0.29.0's observability is built for; the *Triage* and *Inspecting a run's observability
|
|
67
67
|
output* sections in **Part III — Debug** are the detail (the fuller human-facing map lives in
|
|
68
68
|
`docs/debugging.md` — repo-only, not shipped with the installed skill).
|
|
69
69
|
- **Multi-turn / interactive reproduction** → `cowork-harness chat` (interactive; gates answered at the
|
|
@@ -206,7 +206,7 @@ them by what you're trying to prove:
|
|
|
206
206
|
| a skill actually **ran** (or must NOT) | `skill_triggered: <regex>`, `no_skill_triggered: <regex>` |
|
|
207
207
|
| a tool ran **inside** a skill's scope | `skill_tool_used: {skill, tool}` |
|
|
208
208
|
| a sub-agent did the work | `subagent_output_contains: {contains}`, `subagent_dispatched: <regex>`, `dispatch_count_max: <N>` |
|
|
209
|
-
| a pre-existing input wasn't mutated | `input_unmodified: [<glob
|
|
209
|
+
| a pre-existing input wasn't mutated (incl. `uploads/**`) | `input_unmodified: <glob>` or `[<glob>, …]` (live/verify-run; not microvm) |
|
|
210
210
|
| a resource ceiling held | `max_peak_rss_bytes: <N>` (**live-only**) |
|
|
211
211
|
| a hook blocked / didn't block a tool | `hook_blocked: <regex>`, `no_hook_blocked: true` (replay needs a `controlOut` cassette) |
|
|
212
212
|
| every MCP round-trip succeeded | `no_mcp_error: true` (**live-only**) |
|
|
@@ -421,13 +421,23 @@ decide which assertions from *Assertions: two orthogonal axes* are worth adding)
|
|
|
421
421
|
`tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`.
|
|
422
422
|
- **`result.json` carries the raw fields** the assertions read: `toolDurations`, `models`, `toolErrors`,
|
|
423
423
|
`redundantToolCalls`, `modelUsage`, `thinking`, `skillActivity`, `subagents[]` (prompt/model/output/
|
|
424
|
-
`attributedSkillId`), `context` (tools/mcpServers/availableSkills), `tasks`,
|
|
425
|
-
`presentedFiles`, `hookEvents`, `mcpErrors`, `contextEvents`, `resources
|
|
426
|
-
|
|
427
|
-
|
|
424
|
+
`attributedSkillId`, `outputTruncated`), `context` (tools/mcpServers/availableSkills), `tasks`,
|
|
425
|
+
`workspaceFiles`, `presentedFiles`, `hookEvents`, `mcpErrors`, `contextEvents`, `resources`
|
|
426
|
+
(`probeFailures` distinguishes a failed sample from a tier that was never sampleable). Provenance/
|
|
427
|
+
evidence-health fields: `command` (`run`/`skill`/`record`/`chat`/`replay` — finer than `mode`),
|
|
428
|
+
`gateProvenance` (per-gate `scripted`/`decided(llm|external)`/`first-option`/`prompt` with a
|
|
429
|
+
`bySource` histogram), `evidenceErrors` (dropped/malformed telemetry lines per stream, incl.
|
|
430
|
+
`egressParse`), `fingerprint.frozen` (replay only — marks the shown staleness fingerprint as the
|
|
431
|
+
cassette's record-time value, not a fresh recompute), and `assertTextTruncated` (companion to
|
|
432
|
+
`outputTruncated` on a matched tool result). (Full per-field semantics: the README's "Observability
|
|
433
|
+
fields" section — repo-only; `schema/run-result.json` is the machine source.)
|
|
428
434
|
- **Opaque failure?** A failed run also records **`errorSource`** (where the failure originated) and
|
|
429
435
|
**`stderrLogPath`** (the captured agent stderr) — read those and `trace <run-dir>` *before* re-running;
|
|
430
|
-
a re-record rarely tells you more than the captured stderr already does.
|
|
436
|
+
a re-record rarely tells you more than the captured stderr already does. Also check
|
|
437
|
+
**`resultErrorKind`** (`"transport" | "agent" | "usage_limit"`) before spending another paid run: a
|
|
438
|
+
`"usage_limit"` failure is a quota exhaustion, not a skill bug — retry after the limit resets rather
|
|
439
|
+
than debugging; `"transport"`/`"agent"` means something actually broke, worth localizing before
|
|
440
|
+
re-running.
|
|
431
441
|
- **Attributing cost to sub-agent work.** `subagents[]` gives the dispatch tree — each sub-agent's
|
|
432
442
|
`model`, `toolsUsed`, `prompt`/`output`, and `attributedSkillId` — but **not** its own token/cost;
|
|
433
443
|
aggregate cost is per-**model** in `modelUsage` (and `trace --view usage`), not per-sub-agent. So a
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.29.0` (baseline `desktop-1.20186.0`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -54,7 +54,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
54
54
|
GitHub-hosted runners, no token/Docker/agent:
|
|
55
55
|
|
|
56
56
|
```yaml
|
|
57
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
57
|
+
- run: npm i -g "cowork-harness@>=0.29.0"
|
|
58
58
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
59
59
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
60
60
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -188,7 +188,7 @@ jobs:
|
|
|
188
188
|
with: { node-version: '20' }
|
|
189
189
|
- uses: actions/setup-python@v5
|
|
190
190
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
191
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
191
|
+
- run: npm i -g "cowork-harness@>=0.29.0"
|
|
192
192
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
193
193
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
194
194
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -217,7 +217,7 @@ jobs:
|
|
|
217
217
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
218
218
|
fi
|
|
219
219
|
- if: steps.guard.outputs.live == 'true'
|
|
220
|
-
run: npm i -g "cowork-harness@>=0.
|
|
220
|
+
run: npm i -g "cowork-harness@>=0.29.0"
|
|
221
221
|
- if: steps.guard.outputs.live == 'true'
|
|
222
222
|
run: cowork-harness run scenarios/ --output-format json
|
|
223
223
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.29.0` (baseline `desktop-1.20186.0`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.
|
|
4
|
-
(baseline `desktop-1.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.29.0`
|
|
4
|
+
(baseline `desktop-1.20186.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
7
7
|
**Minimal scenario** — `prompt` is the only required field:
|
|
@@ -245,18 +245,18 @@ same set live from the schema.
|
|
|
245
245
|
| `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it |
|
|
246
246
|
| `user_visible_artifact: <path>` | exists **and** under a user-visible root (`outputs/` + each connected folder's mount name) — the right primitive for a workspace deliverable when a folder is connected |
|
|
247
247
|
| `no_delete_in_outputs: true` | no delete op touched `mnt/outputs` — **only `true` is valid**; `false` is rejected (omit to allow deletes) |
|
|
248
|
-
| `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); **microvm cannot capture** (use container/hostloop); replay needs `cassette.preRunPaths` (≥0.24 container/hostloop recordings) — cassettes without it **exclude** the key with a loud warning |
|
|
249
|
-
| `input_unmodified: [<glob>, …]` | every **pre-existing** file whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail; **microvm cannot capture**; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree |
|
|
248
|
+
| `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); an **incomplete post-run filesystem walk** (an unreadable subtree — a permission/I-O error) also fails evidence-unavailable rather than reporting "no strays" over a partial tree — distinct from the missing-manifest/microvm case, which fails for a different reason; **microvm cannot capture** (use container/hostloop); replay needs `cassette.preRunPaths` (≥0.24 container/hostloop recordings) — cassettes without it **exclude** the key with a loud warning |
|
|
249
|
+
| `input_unmodified: <glob>` or `[<glob>, …]` | a single glob or a list; every **pre-existing** file (incl. uploaded files under `uploads/**`) whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail; **microvm cannot capture**; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree |
|
|
250
250
|
| `self_heal_ran: <bool>` | a plugin-root self-heal script was (not) invoked |
|
|
251
|
-
| `tool_called: <glob>` | a tool the agent ran matched this **glob** — `*` = any run, `?` = one char, exact when literal, anchored + case-sensitive. Exact name (`Write`) matches only that tool; `mcp__workspace__*` matches any workspace tool. GLOB, not regex (`.` is literal
|
|
252
|
-
| `tool_not_called: <glob>` | NO tool the agent ran matched this glob (`mcp__*` = "no MCP tool ran"). Same glob semantics as `tool_called
|
|
251
|
+
| `tool_called: <glob>` | a tool the agent ran matched this **glob** — `*` = any run, `?` = one char, exact when literal, anchored + case-sensitive. Exact name (`Write`) matches only that tool; `mcp__workspace__*` matches any workspace tool. GLOB, not regex (`.` is literal) — an empty glob, or one containing a regex/brace-expansion metacharacter (`.*`, `.+`, `\|`, `()`, `[]`, `+`, `^`, `$`, `{}`, `\d`/`\w`/`\s`/`\b`), is **rejected at load** (a hard schema error, not a runtime warning) — it would match no real tool name and pass a `_not_`/`_absent` assert vacuously. Applies whether the glob comes from an authored scenario or a recorded cassette's frozen assert. The bundled `scenario.py lint` does NOT perform this check — only the harness enforces it, at actual load (`run`/`skill`/`record`) |
|
|
252
|
+
| `tool_not_called: <glob>` | NO tool the agent ran matched this glob (`mcp__*` = "no MCP tool ran"). Same glob semantics as `tool_called`, including the empty/regex-ish rejection |
|
|
253
253
|
| `tool_result_contains: <str>` | a tool result includes the literal string (content / replay-checkable — substring match) |
|
|
254
254
|
| `tool_result_not_contains: <str>` | no tool result includes the literal string (content / replay-checkable; fails loud when tool results are absent) |
|
|
255
|
-
| `subagent_tool_used: <glob>` | a sub-agent used a tool matching this glob (same `*`/`?`, anchored, case-sensitive semantics as `tool_called
|
|
256
|
-
| `subagent_tool_absent: <glob>` | no sub-agent used a tool matching this glob |
|
|
255
|
+
| `subagent_tool_used: <glob>` | a sub-agent used a tool matching this glob (same `*`/`?`, anchored, case-sensitive semantics as `tool_called`, including the empty/regex-ish rejection) |
|
|
256
|
+
| `subagent_tool_absent: <glob>` | no sub-agent used a tool matching this glob (same rejection) |
|
|
257
257
|
| `subagent_dispatched: <regex>` | a sub-agent whose `agentType` **or dispatch description** matches |
|
|
258
258
|
| `subagent_declared_but_unused: <Tool>` | a sub-agent declared the tool but never used **that** tool (even if it used others) |
|
|
259
|
-
| `subagent_output_contains: {match?, contains}` | a dispatched sub-agent's own output contains the substring `contains` — `match` (optional regex over `agentType`/`description`) narrows to specific dispatch(es); omitted, checks whether ANY dispatch's output contains it (existence check, not "all") |
|
|
259
|
+
| `subagent_output_contains: {match?, contains}` | a dispatched sub-agent's own output contains the substring `contains` — `match` (optional regex over `agentType`/`description`) narrows to specific dispatch(es); omitted, checks whether ANY dispatch's output contains it (existence check, not "all"); a miss against an output that was **truncated at the assert cap** reports evidence-unavailable instead of a proven absence — the substring could lie past the cut |
|
|
260
260
|
| `dispatch_count_max: <N>` | at most N sub-agents dispatched — an author-chosen budget (Cowork imposes no in-conversation Task-dispatch cap; records only, enforces nothing — see gotcha 12) |
|
|
261
261
|
| `skill_triggered: <regex>` | a skill matching the regex (invoked id, e.g. `"plugin:skill"`) was invoked via the `Skill` tool — evidence-unavailable (not a normal fail) if the agent's init tools have no `Skill` tool |
|
|
262
262
|
| `no_skill_triggered: <regex>` | no invoked skill id matched — the negative-control / description-collision catcher; evidence-unavailable (never a vacuous pass) if invocation data is absent or the `Skill` tool is unobservable |
|
|
@@ -268,7 +268,7 @@ same set live from the schema.
|
|
|
268
268
|
`subagent_tool_used`, `subagent_tool_absent` are **glob** (anchored, case-sensitive, `*`/`?`). `tool_available`,
|
|
269
269
|
`skill_triggered`/`no_skill_triggered`, `skill_available`, `connector_available`, `skill_tool_used`,
|
|
270
270
|
`subagent_type` are **regex** (unanchored, case-insensitive). So `tool_called: mcp__workspace__*` (glob) but
|
|
271
|
-
`tool_available: mcp__workspace__.*` (regex) — a `.*` in a `tool_called` glob
|
|
271
|
+
`tool_available: mcp__workspace__.*` (regex) — a `.*` in a `tool_called` glob is a load-time schema error, not silently-matches-nothing.
|
|
272
272
|
| `skill_tool_used: {skill, tool}` | a tool whose name matches `tool` ran inside a skill-activation window whose `skillId` matches `skill` (`RunResult.skillActivity`) — evidence-unavailable if skill-activity telemetry is absent; heuristic for inline skills (a sticky, sequential window matching the agent's `activeSkill` scope, not an exact per-tool boundary) |
|
|
273
273
|
| `max_cost_usd: <N>` | the run's SDK-reported cost is ≤ N USD — evidence-unavailable if cost telemetry is absent. **Replay asserts the frozen recording's cost, not fresh spend** — a real regression needs a live `run` |
|
|
274
274
|
| `max_tokens: <N>` | `usage.input_tokens + usage.output_tokens` ≤ N (cache tokens excluded) — same replay caveat as `max_cost_usd` |
|
|
@@ -281,7 +281,7 @@ same set live from the schema.
|
|
|
281
281
|
| `compaction_occurred: true` | a context-compaction boundary occurred (a `compact_boundary` system event was recorded) — lives in the stdout stream, so meaningfully replay-checkable; evidence-unavailable if context-event telemetry is absent. **Only `true` is valid** — omit to not require it |
|
|
282
282
|
| `all_tasks_completed: true` | every task in `RunResult.tasks[]` reached status `"completed"` — **requires ≥1 task** (a zero-task run fails; assert `task_count_min` for presence); evidence-unavailable if tasks telemetry is absent |
|
|
283
283
|
| `task_count_min: <N>` | at least N tasks were created (`RunResult.tasks.length >= N`) — the presence companion for task assertions |
|
|
284
|
-
| `task_status: {match, status}` | a task whose `subject` OR `id` matches the regex `match` reached `status` — evidence-unavailable if tasks telemetry is absent |
|
|
284
|
+
| `task_status: {match, status}` | a task whose `subject` OR `id` matches the regex `match` reached `status` — evidence-unavailable if tasks telemetry is absent; also fails **malformed** when a TaskCreate result was unparseable (corrupt task telemetry), mirroring the guard `all_tasks_completed`/`task_count_min` already had |
|
|
285
285
|
| `no_scratchpad_leak: true` | every file presented via `present_files` that was in the scratchpad was successfully promoted to `mnt/outputs` (none left behind) — vacuously passes if nothing was presented (pair with a presence check to require a delivery); content-class: both the `present_files` tool_use and its own tool_result live in the ordinary events stream, so `RunResult.presentedFiles` re-derives identically on replay (meaningfully replay-checkable, same as `skill_triggered`); evidence-unavailable if `presentedFiles` telemetry is absent (an older run predating the feature). **`fidelity: container` only** — `present_files` is not served on hostloop/microvm, so a scratchpad-delivered file is neither promoted to `mnt/outputs` nor detected there (a skill that delivers via write-to-cwd→`present_files` will false-red `user_visible_artifact` on those tiers; use `container`, or write directly to `outputs/`). **Only `true` is valid** |
|
|
286
286
|
| `present_files_called: true` | at least one file was actually delivered via the `present_files` tool (`RunResult.presentedFiles` is non-empty) — the presence companion to `no_scratchpad_leak` (which passes vacuously when nothing was presented). Pair them to require a delivery **and** require it not to leak. Content-class (re-derives identically on replay). **`fidelity: container` only** — `present_files` is not served on hostloop/microvm. **Only `true` is valid** |
|
|
287
287
|
| `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
|
|
@@ -300,7 +300,7 @@ same set live from the schema.
|
|
|
300
300
|
| `egress_allowed: <host>` | the host was allowed through |
|
|
301
301
|
| `no_mcp_error: true` | no MCP round-trip failed (`RunResult.mcpErrors` is empty — no unhandled server, no handler throw) — live-only: MCP round-trips are harness-computed, not in the SDK stdout stream, so evidence-unavailable on replay (never a vacuous pass). **Only `true` is valid** |
|
|
302
302
|
| `max_peak_rss_bytes: <N>` | peak sampled RSS of the agent sandbox ≤ N bytes (`RunResult.resources.peakRssBytes`) — live-only: replay never spawns a sandbox to sample, so evidence-unavailable on replay/protocol (never a vacuous pass); also evidence-unavailable when sampling captured no RSS value |
|
|
303
|
-
| `semantic_matches: {rubric: [...], min_pass?, judge_model?}` | a pinned LLM judge grades each fixed `rubric` claim against the run's answer — the **union of the agent's final result text (`RunResult.finalMessage`), the transcript, and the final on-disk content of any files the agent authored during the run** — so a claim about content the skill led the agent to *write to a file* grades as reliably as one about inlined prose (authored-file evidence is unavailable on the **microvm** tier — no pre-run manifest to diff against, so no files are captured; `container`/`hostloop` do capture them). The assert passes iff ≥ `min_pass` claims pass (default: all — avoid for a gating scenario). Results align by claim index and are recorded per-claim in `RunResult.assertions[].semanticClaims` (`[{index, claim, pass}]`, so a consumer can diff the per-claim profile across runs); a rep whose grade can't be parsed (after one retry) is marked `RunResult.assertions[].judgeInvalid` and **never silently dropped** — it is excluded from the pass denominator, and the guard against a misleading score from that exclusion is the gate's minimum-valid-rep floor (`MIN_VALID` ≥ 4) plus this visibility, not a claim that denominator-shrinking inflation is impossible. Within a rep, a grade that's still unparseable after the retry **fails that assert outright** (evidence-unavailable, not a vacuous pass) — a persistently-flaky judge reds the run rather than silently passing. `judge_model` pins the grader (a dated id keeps a before/after comparison reproducible). Live-only: the judge is a live model call, so evidence-unavailable / skipped-loud on replay (never a vacuous pass) |
|
|
303
|
+
| `semantic_matches: {rubric: [...], min_pass?, judge_model?}` | a pinned LLM judge grades each fixed `rubric` claim against the run's answer — the **union of the agent's final result text (`RunResult.finalMessage`), the transcript, and the final on-disk content of any files the agent authored during the run** — so a claim about content the skill led the agent to *write to a file* grades as reliably as one about inlined prose (authored-file evidence is unavailable on the **microvm** tier — no pre-run manifest to diff against, so no files are captured; `container`/`hostloop` do capture them). Beyond the microvm case, when the authored-file evidence backing the judged document is **incomplete** — a file dropped at the capture-size cap, unreadable at read-back, or (on `--resume`) the scratchpad walk skipped — the assert fails evidence-unavailable rather than trusting a judge grade over a partial document; this is separate from the malformed-grade `judgeInvalid` path below. The assert passes iff ≥ `min_pass` claims pass (default: all — avoid for a gating scenario). Results align by claim index and are recorded per-claim in `RunResult.assertions[].semanticClaims` (`[{index, claim, pass}]`, so a consumer can diff the per-claim profile across runs); a rep whose grade can't be parsed (after one retry) is marked `RunResult.assertions[].judgeInvalid` and **never silently dropped** — it is excluded from the pass denominator, and the guard against a misleading score from that exclusion is the gate's minimum-valid-rep floor (`MIN_VALID` ≥ 4) plus this visibility, not a claim that denominator-shrinking inflation is impossible. Within a rep, a grade that's still unparseable after the retry **fails that assert outright** (evidence-unavailable, not a vacuous pass) — a persistently-flaky judge reds the run rather than silently passing. `judge_model` pins the grader (a dated id keeps a before/after comparison reproducible). Live-only: the judge is a live model call, so evidence-unavailable / skipped-loud on replay (never a vacuous pass) |
|
|
304
304
|
| `artifact_json: {artifact, path, …}` | assert a JSON artifact's contents — `equals`/`gt`/`in`/`exists`/`absent`/`is_null` over a dotted `path` (`in` = membership in a list, for a stochastic/LLM value; `absent` ≠ `is_null`; an unresolved intermediate fails loud) |
|
|
305
305
|
| `computer_links_resolve: true` | every `computer://` link in the model-visible transcript resolves to an artifact that exists in the run's collected outputs/mounts — a dangling link fails, naming which target was checked (a live host path, the collected work tree, or the replay manifest). **Requires ≥1 link** (zero links fails — use `computer_links_resolve_if_present` for the presence-free variant). **Only `true` is valid** (`false` is rejected by the schema) |
|
|
306
306
|
| `computer_links_resolve_if_present: true` | like `computer_links_resolve` but passes vacuously when the transcript has zero `computer://` links — the presence-free variant. **Only `true` is valid** |
|
|
@@ -326,6 +326,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
|
326
326
|
|---|---|---|
|
|
327
327
|
| `assertion` | fail | An authored `assert:` item failed |
|
|
328
328
|
| `result_error` | fail | The run's SDK result was `"error"` |
|
|
329
|
+
| `usage_limit` | fail | Usage/quota limit hit (not a skill failure) — retry after the limit resets. Emitted when `RunResult.resultErrorKind === "usage_limit"` |
|
|
329
330
|
| `transport_error` | fail | The connection dropped mid/after-run |
|
|
330
331
|
| `permissive_auto_allow` | fail | A cowork-parity auto-allow real Cowork would block (opt out: `allow_permissive_auto_allow`) |
|
|
331
332
|
| `outputs_delete` | fail | An unauthorized delete touched `mnt/outputs` (opt out: author `no_delete_in_outputs`) |
|
|
@@ -344,7 +345,7 @@ overall run verdict and exit code — `assert result: success` alone won't catch
|
|
|
344
345
|
## Replay class
|
|
345
346
|
|
|
346
347
|
A cassette (`record`/`replay`) has **no filesystem and no network**. `replay` re-evaluates only the
|
|
347
|
-
**content** assertions. The authoritative list is `
|
|
348
|
+
**content** assertions. The authoritative list is `ALWAYS_CONTENT_KEYS`/`QUESTION_GATE_KEYS`/`MANIFEST_KEYS` (composed) in `src/run/cassette.ts`.
|
|
348
349
|
|
|
349
350
|
**Assertion source — frozen by default, on-disk by opt-in.** A plain `replay` evaluates the `assert:` block
|
|
350
351
|
**frozen in the cassette** (byte-deterministic, ignores the working tree); editing `scenarios/<name>.yaml` does
|
|
@@ -458,7 +459,7 @@ reference omits). Neither list is a strict superset of the other — reach for t
|
|
|
458
459
|
are replay-checkable **when the cassette carries an `artifacts` manifest**; without one they are
|
|
459
460
|
skipped too.) *Partial skip:* a mixed `{result, egress_denied}` greens on `result` while
|
|
460
461
|
`egress_denied` is dropped. Both now warn loudly. → put egress/live-only checks on a live gate; one
|
|
461
|
-
concern per item; run the linter. (`
|
|
462
|
+
concern per item; run the linter. (`LIVE_ONLY_KEYS`/`MANIFEST_KEYS` in `src/run/cassette.ts`.)
|
|
462
463
|
|
|
463
464
|
2. **Gate keys need a `controlOut` cassette.** `question_asked`, `questions_count_max`,
|
|
464
465
|
`gate_answers_delivered`, `gate_answer_count_min`, `hook_blocked`, `no_hook_blocked` only evaluate on
|
|
@@ -468,7 +469,7 @@ reference omits). Neither list is a strict superset of the other — reach for t
|
|
|
468
469
|
vacuously when zero gates fired**; use `gate_answer_count_min: 1` to also require a gate to have
|
|
469
470
|
fired. A **header-only gate** (empty `question`, only `header`) can never be keyed and is rejected
|
|
470
471
|
loudly — every gate needs a non-empty `question`.
|
|
471
|
-
(`
|
|
472
|
+
(`QUESTION_GATE_KEYS` in `src/run/cassette.ts`; `src/assert.ts`.)
|
|
472
473
|
|
|
473
474
|
3. **The LLM-decider's two spellings.** Scripted answers + `on_unanswered: fail` is deterministic;
|
|
474
475
|
the stochastic path flags the run `nonDeterministic`. The LLM decider is one mechanism, two
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
4
|
decision path. Every one answers a question a real fleet owner had to work out the hard way. Facts track the harness version in SKILL.md's
|
|
5
|
-
front-matter (currently 0.
|
|
5
|
+
front-matter (currently 0.29.0). Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
|
|
6
6
|
Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
|
|
7
7
|
needed if your CLI meets SKILL.md's version floor.
|
|
8
8
|
|
|
@@ -45,7 +45,7 @@ import sys
|
|
|
45
45
|
from pathlib import Path
|
|
46
46
|
|
|
47
47
|
# --- the replay-class taxonomy ---
|
|
48
|
-
# NB: this is NOT a 1:1 mirror of
|
|
48
|
+
# NB: this is NOT a 1:1 mirror of the ALWAYS_CONTENT_KEYS/QUESTION_GATE_KEYS/MANIFEST_KEYS buckets in src/run/cassette.ts. cassette.ts keeps the verdict
|
|
49
49
|
# modifiers (VERDICT_MODIFIER_KEYS) in its content set so they replay as no-op passes; the linter
|
|
50
50
|
# deliberately keeps them OUT of CONTENT_KEYS so a modifier-only scenario still trips the `replay-noop`
|
|
51
51
|
# warning below (a no-op pass verifies nothing real — exactly what that warning is for).
|
|
@@ -688,6 +688,12 @@ def cmd_lint(args):
|
|
|
688
688
|
# v1 declines to do.
|
|
689
689
|
|
|
690
690
|
_PLUGIN_ROOT_TOKEN = re.compile(r"\$\{?CLAUDE_PLUGIN_ROOT\}?")
|
|
691
|
+
# A runtime SELF-HEAL for a dead ${CLAUDE_PLUGIN_ROOT}: discovering the real mount under /sessions at run
|
|
692
|
+
# time (the prescribed pattern — e.g. `[ -d "$X" ] || X=$(find /sessions ... -name ...)`, or an inline
|
|
693
|
+
# `|| python3 "$(find /sessions ...)"`). When a bash block that uses the token ALSO contains a `find` over
|
|
694
|
+
# /sessions, the token is dead but the block rescues it → downgrade the WARN to INFO (Item 4). Conservative:
|
|
695
|
+
# we do NOT verify the find pattern actually matches the plugin's layout (hence the INFO's "not validated").
|
|
696
|
+
_SELF_HEAL = re.compile(r"\bfind\b[^\n]*/sessions")
|
|
691
697
|
# Opening/closing fence: ``` or ~~~ (>=3), optional info string (language).
|
|
692
698
|
_FENCE = re.compile(r"^\s*(`{3,}|~{3,})\s*([A-Za-z0-9_+-]*)\s*$")
|
|
693
699
|
# A hooks-config command string: `"command": "<value>"` (value may contain escaped quotes).
|
|
@@ -717,6 +723,19 @@ def _finding_plugin_root(path, line, ctx_label):
|
|
|
717
723
|
)
|
|
718
724
|
|
|
719
725
|
|
|
726
|
+
def _finding_plugin_root_guarded(path, line, ctx_label):
|
|
727
|
+
return Finding(
|
|
728
|
+
"INFO",
|
|
729
|
+
"plugin-root-guarded",
|
|
730
|
+
f"`${{CLAUDE_PLUGIN_ROOT}}` used in an in-VM bash context ({ctx_label}), but the same block "
|
|
731
|
+
"self-heals it (a runtime `find` under /sessions) — the dead token is harmless here.",
|
|
732
|
+
"Guard not validated: the linter does not check the `find` pattern actually matches the plugin's "
|
|
733
|
+
"layout. Prefer resolving the mount from the script's own location over a find-fallback.",
|
|
734
|
+
path,
|
|
735
|
+
line,
|
|
736
|
+
)
|
|
737
|
+
|
|
738
|
+
|
|
720
739
|
def _finding_hook_host_write(path, line, what):
|
|
721
740
|
return Finding(
|
|
722
741
|
"WARN",
|
|
@@ -748,6 +767,24 @@ def _lint_skill_text(path, raw_lines, force_json=False):
|
|
|
748
767
|
fence_char = ""
|
|
749
768
|
fence_len = 0
|
|
750
769
|
fence_lang = ""
|
|
770
|
+
# Per-bash-fence buffer (Item 4): plugin-root token hit line numbers + the whole block's text, so a
|
|
771
|
+
# ${CLAUDE_PLUGIN_ROOT} use that is self-healed elsewhere IN THE SAME BLOCK downgrades WARN -> INFO.
|
|
772
|
+
# Emission is deferred to fence close (or EOF) but original 1-based line numbers are preserved.
|
|
773
|
+
bash_token_lines = []
|
|
774
|
+
bash_block_text = []
|
|
775
|
+
|
|
776
|
+
def flush_bash():
|
|
777
|
+
if bash_token_lines:
|
|
778
|
+
healed = _SELF_HEAL.search("\n".join(bash_block_text)) is not None
|
|
779
|
+
for ln in bash_token_lines:
|
|
780
|
+
findings.append(
|
|
781
|
+
_finding_plugin_root_guarded(path, ln, "```bash block")
|
|
782
|
+
if healed
|
|
783
|
+
else _finding_plugin_root(path, ln, "```bash block")
|
|
784
|
+
)
|
|
785
|
+
bash_token_lines.clear()
|
|
786
|
+
bash_block_text.clear()
|
|
787
|
+
|
|
751
788
|
for i, line in enumerate(raw_lines, start=1):
|
|
752
789
|
m = _FENCE.match(line)
|
|
753
790
|
if m:
|
|
@@ -757,6 +794,8 @@ def _lint_skill_text(path, raw_lines, force_json=False):
|
|
|
757
794
|
continue
|
|
758
795
|
# A closing fence uses the same char, is at least as long, and carries no language.
|
|
759
796
|
if marker[0] == fence_char and len(marker) >= fence_len and not lang:
|
|
797
|
+
if fence_lang in _BASH_FENCE_LANGS:
|
|
798
|
+
flush_bash() # decide WARN vs INFO now that the whole block is known
|
|
760
799
|
in_fence = fence_char = ""
|
|
761
800
|
fence_len = 0
|
|
762
801
|
fence_lang = ""
|
|
@@ -775,8 +814,9 @@ def _lint_skill_text(path, raw_lines, force_json=False):
|
|
|
775
814
|
ctx = "prose"
|
|
776
815
|
|
|
777
816
|
if ctx == "bash":
|
|
817
|
+
bash_block_text.append(line) # buffer the block; token hits emit on flush (self-heal aware)
|
|
778
818
|
if _PLUGIN_ROOT_TOKEN.search(line):
|
|
779
|
-
|
|
819
|
+
bash_token_lines.append(i)
|
|
780
820
|
elif ctx == "json":
|
|
781
821
|
for cm in _HOOK_CMD.finditer(line):
|
|
782
822
|
_check_hook_command(path, i, cm.group(1), findings)
|
|
@@ -786,6 +826,10 @@ def _lint_skill_text(path, raw_lines, force_json=False):
|
|
|
786
826
|
for bm in _BASH_DIRECTIVE.finditer(line):
|
|
787
827
|
if _PLUGIN_ROOT_TOKEN.search(bm.group(1)):
|
|
788
828
|
findings.append(_finding_plugin_root(path, i, "Bash() directive"))
|
|
829
|
+
# A bash fence left unclosed at EOF still has buffered token hits — flush them (else a real WARN/INFO
|
|
830
|
+
# would be silently dropped).
|
|
831
|
+
if in_fence and fence_lang in _BASH_FENCE_LANGS:
|
|
832
|
+
flush_bash()
|
|
789
833
|
return findings
|
|
790
834
|
|
|
791
835
|
|
package/AGENTS.md
CHANGED
|
@@ -42,10 +42,11 @@ protocol layer or run-loop bookkeeping in the CLI.
|
|
|
42
42
|
scenario key is retired vocabulary — the alias is gone and it is now rejected as an unknown key — do
|
|
43
43
|
not reintroduce the term.
|
|
44
44
|
- **A new assertion must pick its replay class.** *Content* assertions (read only `ctx.transcript` / the
|
|
45
|
-
record) go in `src/run/cassette.ts`
|
|
46
|
-
|
|
47
|
-
only run on live (non-replay) gates.
|
|
48
|
-
replay checks" prose just describes
|
|
45
|
+
record) go in one of `src/run/cassette.ts`'s exported buckets — `ALWAYS_CONTENT_KEYS`, `QUESTION_GATE_KEYS`,
|
|
46
|
+
or `MANIFEST_KEYS` — so they run on the token-free `replay` PR gate; *filesystem / egress* assertions go
|
|
47
|
+
in `LIVE_ONLY_KEYS` instead, so they only run on live (non-replay) gates. These four exported constants
|
|
48
|
+
are the single source of truth; the README's "what replay checks" prose just describes them. The wrong
|
|
49
|
+
bucket is a **silent no-op in CI**.
|
|
49
50
|
- **`replay` consumes `controlOut` and re-serializes via `serializeDecision` to guard the AskUserQuestion
|
|
50
51
|
answer shape (O7) on the token-free lane. A new decision *kind* must extend BOTH `serializeDecision`
|
|
51
52
|
AND `deserializeDecision` (declared inverses in `src/agent/session.ts`) — they must not drift.**
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,144 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.29.0] — 2026-07-11
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- **Parity baseline for Claude Desktop 1.20186.0.** `baseline: latest` now resolves to `desktop-1.20186.0`
|
|
14
|
+
(VM agent ELF 2.1.202 unchanged; native host app 2.1.205). The 1.20186.0 asar was a minifier re-anchor —
|
|
15
|
+
the value-resolved spawn contract is byte/behaviourally identical; the sync extractor's structural anchors
|
|
16
|
+
were re-anchored to the new bundle shapes (hoisted helpers became namespace-method calls; const spreads
|
|
17
|
+
became export aliases). Verified live across the `protocol`, `container`, and `hostloop` tiers.
|
|
18
|
+
- **`sync` now guards Cowork system-prompt drift.** Each sync fingerprints the prompt (a minifier-independent
|
|
19
|
+
content hash plus a `{{placeholder}}` / `<section>` inventory) and **refuses to write** when the content
|
|
20
|
+
drifts from the committed fingerprint until a new fingerprint entry is added, and **hard-fails** on a
|
|
21
|
+
`{{placeholder}}` the renderer neither substitutes nor explicitly allowlists. This closes a gap where a
|
|
22
|
+
prompt change (e.g. the deployment-gated `{{modelIdentity}}` placeholder added in 1.20186.0, which is
|
|
23
|
+
stripped on first-party so the rendered prompt is unchanged) could slip past the coarse `asarFingerprint`.
|
|
24
|
+
- **Usage/quota-limit runs are now classified as `resultErrorKind: "usage_limit"`.** A session/weekly/model/
|
|
25
|
+
spend/org quota-exhaustion arrives as an `is_error` result with the SDK's misleading `subtype: "success"`
|
|
26
|
+
and HTTP 429 — previously an undifferentiated error, indistinguishable from a real skill regression.
|
|
27
|
+
Detection is conjunctive (`api_error_status === 429` **and** a terminal usage-limit message; a bare 429 is
|
|
28
|
+
a transient overload and is not reclassified). `resultSubtype` is kept verbatim (faithful SDK passthrough).
|
|
29
|
+
Surfaced on `RunResult`, `status.json`, and a distinct verdict signal ("retry after reset") so a batch can
|
|
30
|
+
halt-fast; the `claude -p` decider transport fails fast (non-retryable) on a usage limit instead of
|
|
31
|
+
retrying into a spent quota.
|
|
32
|
+
- **`input_unmodified` can now guard uploaded files (`uploads/**`).** Uploads are captured as a read-only
|
|
33
|
+
input root (hash-only in cassettes — a private upload is never inlined), so a scenario can assert the agent
|
|
34
|
+
didn't mutate an uploaded file; a mutation is attributed to the agent (not excused as an external edit).
|
|
35
|
+
Previously only connected-folder inputs were guardable.
|
|
36
|
+
- **`RunResult.command`** records the exact command that produced a result (`run`/`skill`/`record`/`chat`/
|
|
37
|
+
`replay`) — finer than `mode` (which only distinguishes run vs. chat, so a `skill`/`record` run was
|
|
38
|
+
indistinguishable from a plain `run`). `reindex` now prefers this field, so a rebuilt run index no longer
|
|
39
|
+
relabels a `skill`/`record` run as `run`; it falls back to a prior index row, then to `mode`, on older
|
|
40
|
+
results that predate the field.
|
|
41
|
+
- **`RunResult.evidenceErrors.egressParse`** counts proxy-log lines the egress sidecar dropped as malformed
|
|
42
|
+
(bad JSON, missing host, unknown decision) — previously filtered out silently. Observability only (egress
|
|
43
|
+
assertions are positive, so a dropped line already fails loud), but now visible.
|
|
44
|
+
- **`subagents[].outputTruncated` and `toolResults[].assertTextTruncated`** flag when a captured value was cut
|
|
45
|
+
at the 10 KB assertion cap, so `subagent_output_contains` (and the equivalent substring checks) report
|
|
46
|
+
"evidence unavailable" instead of a false "not found" when the searched text could lie past the cut.
|
|
47
|
+
- **`fingerprint.frozen`** marks, on the replay lane, that the staleness fingerprint is the cassette's
|
|
48
|
+
record-time snapshot rather than a fresh recompute, so a consumer can't mistake one for the other.
|
|
49
|
+
- **Replay results now populate `prompt` and `toolResults`** (the scenario prompt that drove the re-drive and
|
|
50
|
+
the tool-result records already built for evaluation) — both were previously always `undefined` on replay.
|
|
51
|
+
- **`ResourceSummary.probeFailures`** counts resource-sampling probes that actually failed (nonzero exit,
|
|
52
|
+
timeout, parse error), distinct from a tier that was never sampleable (protocol/replay) — "sampling broke"
|
|
53
|
+
is now told apart from "sampling wasn't attempted."
|
|
54
|
+
- **`chat` now samples real resources on the `container`/`hostloop` tiers.** Previously no `ResourceSampler`
|
|
55
|
+
was ever started for a chat session on either tier, so `resources.jsonl` was always empty despite
|
|
56
|
+
`chat-result.ts`'s doc-comment claiming resource parity with `run`; a protocol-tier chat legitimately has
|
|
57
|
+
none (no container/process id to probe against the host `claude` binary).
|
|
58
|
+
|
|
59
|
+
### Changed
|
|
60
|
+
|
|
61
|
+
- **`input_unmodified` accepts a single glob string** as well as an array (`input_unmodified: 'uploads/**'`
|
|
62
|
+
no longer errors "expected array").
|
|
63
|
+
- **`sessionFingerprintDrift` keeps an unresolvable session fingerprint non-failing (informational)** and
|
|
64
|
+
downgrades a mismatch to a non-failing note when the cassette's session was only resolved via a
|
|
65
|
+
name-lookup fallback (the SAME low-confidence signal `scenarioContentDrift` already downgrades on) —
|
|
66
|
+
a relocated cassette whose relative-offset resolution lands on an unrelated same-named session file no
|
|
67
|
+
longer reads as a false "session shape drifted."
|
|
68
|
+
- **`artifact_json`'s replay remedy names an uploaded input distinctly** — a body-less target that was
|
|
69
|
+
captured hash-only because it's an *upload* now says so directly, instead of the prior combined
|
|
70
|
+
readonly-or-over-cap wording that didn't cover the uploads case.
|
|
71
|
+
- **`scenario.py lint-skill`** downgrades a `${CLAUDE_PLUGIN_ROOT}`-in-VM-bash use to a `plugin-root-guarded`
|
|
72
|
+
**INFO** (from WARN) when the same bash block self-heals it at runtime (`[ -d ] || find /sessions …`), so a
|
|
73
|
+
correctly-guarded block no longer shares the alarming WARN class with a genuinely-unguarded use.
|
|
74
|
+
- **`no_unexpected_files` now requires a complete post-run filesystem walk.** An unreadable subtree
|
|
75
|
+
(permission error, race) previously collapsed to "no files found there" and could vacuously pass; the check
|
|
76
|
+
now fails "evidence unavailable" when any part of the tree couldn't be observed.
|
|
77
|
+
- **`semantic_matches` refuses to grade over incomplete authored-file evidence.** When a file the run authored
|
|
78
|
+
was dropped at the capture-size budget or was unreadable at read-back (or, on `--resume`, the scratchpad was
|
|
79
|
+
skipped to avoid misattributing a prior turn's files), the assertion fails "evidence unavailable" rather than
|
|
80
|
+
trusting a judge grade made without that content. The judged document is also size-capped now (per-section
|
|
81
|
+
and total, with an explicit truncation marker) so a long run can't overflow the judge's context; secret
|
|
82
|
+
scrubbing runs before the cap, so a secret can never be exposed by falling on a truncation boundary.
|
|
83
|
+
- **`task_status` now honors corrupt task telemetry** — when a `TaskCreate` result was unparseable it fails
|
|
84
|
+
"malformed" instead of matching against the surviving subset, consistent with `all_tasks_completed` /
|
|
85
|
+
`task_count_min`.
|
|
86
|
+
- **Empty or regex/brace-expansion tool globs are rejected at load.** `tool_called` / `tool_not_called` /
|
|
87
|
+
`subagent_tool_used` / `subagent_tool_absent` values that are empty or contain regex/brace metacharacters
|
|
88
|
+
(`.*`, `|`, `[]`, `{}`, …) match no real tool name and used to pass a `_not_`/`_absent` assertion vacuously;
|
|
89
|
+
they are now a hard schema error for authored scenarios and recorded cassettes alike.
|
|
90
|
+
- **`readRunStatus` validates the shape of `status.json`**, not just that it parses — a truncated or otherwise
|
|
91
|
+
structurally-invalid write (e.g. `{}`) is reported malformed instead of being trusted as a valid status; a
|
|
92
|
+
`--follow` loop resolves only after observing a genuinely valid status.
|
|
93
|
+
- **Answered gates reconcile against actual delivery.** A control-response that was optimistically reported
|
|
94
|
+
delivered (queued) but whose write later failed before reaching the agent is corrected once the run
|
|
95
|
+
completes, so delivery telemetry and `trace --view questions` read the true outcome.
|
|
96
|
+
- **A `TaskCreate` / `WebSearch` / `present_files` call whose result never arrived before the stream ended now
|
|
97
|
+
counts as incomplete evidence** (`evidenceErrors.*`) instead of silently vanishing from the resolved set, so
|
|
98
|
+
the dependent assertion fails "cannot verify" rather than grading a truncated subset as complete.
|
|
99
|
+
- **Bounded hashing/reads.** `classifyWorkspaceFiles` caps per-file hashing (50 MiB, matching the pre-run
|
|
100
|
+
manifest) and reports `hashError: "over-cap"` instead of reading a huge file whole; authored-file capture
|
|
101
|
+
reads only the bytes it retains.
|
|
102
|
+
|
|
103
|
+
### Fixed
|
|
104
|
+
|
|
105
|
+
- **`sync` no longer derives a phantom `nativeStagedPath`.** The native macOS agent app (`claude-code/`) and
|
|
106
|
+
the container/microvm Linux ELF (`claude-code-vm/`) version on independent cadences; `sync` derived the
|
|
107
|
+
native (hostloop) path from the VM `.sdk-version`, producing a path that did not exist whenever the two had
|
|
108
|
+
drifted — and freezing that phantom into the written baseline. It now resolves the native path from the
|
|
109
|
+
newest app actually staged under `claude-code/`, falling back to the version-derived convention only when no
|
|
110
|
+
native app is staged at all.
|
|
111
|
+
- **Two anonymous sub-agent dispatches in different assistant turns could collide on the same synthesized id**
|
|
112
|
+
(the fallback id was derived only from the block's position within its own message). It now derives from the
|
|
113
|
+
message id plus block index — unique across the run and stable across record→replay — fixing dispatch nesting
|
|
114
|
+
in `trace --view dispatches` for skills that dispatch multiple unnamed sub-agents.
|
|
115
|
+
- **The semantic judge's recorded model could name the wrong one.** `assertions[].judgeModel` was stamped from
|
|
116
|
+
the requested alias (e.g. `opus`) before the call; the transport resolves an alias to a concrete model per
|
|
117
|
+
call, so the resolved model is now recorded after the call completes.
|
|
118
|
+
- **`matrix --repeat`'s per-cell pass/fail glyph didn't honor `--allow-budget-stop`**, though the aggregate
|
|
119
|
+
"N/M cells passed" count did — a cell that passed under a permitted budget stop could print as failed next to
|
|
120
|
+
its own row while counting as a pass in the summary.
|
|
121
|
+
- **The resource sampler could miss a short run's only sample.** `stop()` now waits (bounded) for an in-flight
|
|
122
|
+
probe before returning, so a run shorter than one sampling interval reliably records its first sample instead
|
|
123
|
+
of racing teardown.
|
|
124
|
+
- **A corrupt `timeline.jsonl` header was indistinguishable from an absent timeline** — both read as "no
|
|
125
|
+
timeline," so a corrupt read could be baked into a recorded cassette as a clean empty timeline. A corrupt
|
|
126
|
+
header now reads as a distinct "present but corrupt" state, kept out of both live results and cassettes.
|
|
127
|
+
- **A malformed `system/init` frame** (a non-array `tools`/`mcp_servers`/`skills`, or a non-object content
|
|
128
|
+
block) could crash with an uncaught `TypeError`; it is now rejected as a typed protocol error at parse time,
|
|
129
|
+
and the `trace` reconstruction lane skips it instead of aborting.
|
|
130
|
+
- **`verify-cassettes` no longer aborts the whole batch on a nameless cassette.** A lenient cassette with
|
|
131
|
+
no `scenario.name` (and no `scenarioSource`) threw inside the per-file verification, which aborted the
|
|
132
|
+
entire run (`results: []` — a false-green by abort) instead of reporting that one file's error and
|
|
133
|
+
continuing. Each file's verification is now wrapped so a per-file crash becomes that file's error row
|
|
134
|
+
(mirrors `replay`'s existing per-file catch).
|
|
135
|
+
|
|
136
|
+
### Internal
|
|
137
|
+
|
|
138
|
+
- **Eval-gate hardening** (`scripts/eval-gate.ts`, maintainer instrument — not shipped with the skill): a
|
|
139
|
+
missing/edited scenario or discriminating claim now fails the gate (opt-out `--allow-unmatched`); a mixed-model
|
|
140
|
+
capture, a null candidate model against a concrete baseline, and a malformed profile/fraction are refused
|
|
141
|
+
instead of silently collapsing; child runs are bounded by a timeout, output cap, and process-group kill.
|
|
142
|
+
- **Skill-critique hardening** (`scripts/skill-critique.ts`, `src/critique/*`): the reflection turn's exit,
|
|
143
|
+
envelope, and session/output continuity are validated before evaluation; a corrupt or missing turn archive is
|
|
144
|
+
reported rather than substituting turn-2 data; the untrusted self-report is fenced as inert data before the
|
|
145
|
+
evaluator prompt; child turns are bounded and orphaned process groups are killed on interrupt.
|
|
146
|
+
|
|
9
147
|
## [0.28.0] — 2026-07-10
|
|
10
148
|
|
|
11
149
|
### Added
|
package/DESIGN.md
CHANGED
|
@@ -165,7 +165,7 @@ The policy that produces those `allow`/`deny` responses is the **Decider** seam
|
|
|
165
165
|
|
|
166
166
|
> **Machine-readable form:** the five shapes below are schema'd as `schema/protocol.v1.json`, with a golden vector pack at `fixtures/protocol/v1/` — see [docs/protocol.md](./docs/protocol.md) for scope, versioning, and how to conformance-test against them.
|
|
167
167
|
|
|
168
|
-
### Control protocol — VERIFIED end-to-end against the live host CLI (macOS build 2.1.177+; the staged in-VM agent that L1/L2 run is **2.1.202**, baseline **`desktop-1.
|
|
168
|
+
### Control protocol — VERIFIED end-to-end against the live host CLI (macOS build 2.1.177+; the staged in-VM agent that L1/L2 run is **2.1.202**, the native host app that `hostloop` runs is **2.1.205**, baseline **`desktop-1.20186.0`** — a fresh live end-to-end pass across the `protocol`, `container`, and `hostloop` tiers was run against this baseline on 2026-07-11, superseding the prior `1.19367.0` pin; the intervening `1.19367.0`→`1.20186.0` bump was a minifier re-anchor with a byte/behaviourally-identical value-resolved spawn contract, now confirmed live)
|
|
169
169
|
|
|
170
170
|
> The staged agent ELF is unchanged (2.1.181) across the 1.14271.0→1.15200.0 asar bump, and 2.1.187 across the 1.15200.0→1.15962.0 bump. The live scenario suite (`protocol` + `container` tiers) was re-run against the 1.15200.0 baseline; the 1.15962.0 bump was verified via asar analysis (content byte-identical: host-loop generator, system prompt, identity string, gates, and egress domains all unchanged) plus a full local test suite pass. The 1.15962.1→1.17377.1 bump moved the staged agent to **2.1.197** and added `api.claude.ai` to the egress allowlist; re-verified via `sync` (no unknown deltas) plus a manual asar spot-check of the reconstructed prompt content (substantively unchanged — see the Parity entry in CHANGELOG.md) and a full live scenario-suite pass (`protocol` + `container` tiers).
|
|
171
171
|
|
|
@@ -179,7 +179,7 @@ The handshake and shapes below were confirmed empirically with an end-to-end run
|
|
|
179
179
|
|
|
180
180
|
> **Cowork mode is enabled by env, not a flag.** In the staged agent (2.1.197) `--cowork` is a *plugin-scope* flag ("can only be used with user scope") and is rejected by the agent invocation; cowork mode is entered via **`CLAUDE_CODE_IS_COWORK=1`**. (Do **not** also set `CLAUDE_CODE_USE_COWORK_PLUGINS` — Desktop doesn't, and it flips the agent's userSettings filename to `cowork_settings.json` and plugin cache to `cowork_plugins/` via `TSO()` — the minified Desktop helper that derives the cowork settings/cache paths; plugins are delivered via `--plugin-dir`.) The host CLI is a different (macOS) build, so L0 runs plain (control-loop validation only); L1/L2 run the staged **Linux/arm64** binary — bind-mounted from the user's own install.
|
|
181
181
|
|
|
182
|
-
### Spawn contract + host-loop vs VM-loop (binary-verified through asar 1.17377.1; asar analysis since carried through 1.
|
|
182
|
+
### Spawn contract + host-loop vs VM-loop (binary-verified through asar 1.17377.1; asar analysis since carried through 1.20186.0 — the full spawn contract is byte-identical across the 1.18286.0→1.18286.2 bump, and behaviorally identical across the 1.18286.2→1.19367.0→1.20186.0 re-minifications: the value-resolved contract is unchanged and only minified symbol names + bundle layout moved (in 1.20186.0, hoisted helpers became namespace-method calls and const spreads became export aliases), re-verified via the structural anchors and the live-asar spawn-contract tests plus a 1.20186.0-shaped drift-guard fixture — the live end-to-end pass now covers `1.20186.0` too (see the "Control protocol" note above), superseding the prior `1.19367.0` pin)
|
|
183
183
|
|
|
184
184
|
The full Desktop→agent spawn contract (cwd `/sessions/<id>`, `CLAUDE_CONFIG_DIR=mnt/.claude`, the env object, `--tools`/`--allowedTools`/`--plugin-dir`/`--effort`/`--setting-sources`, permission layers, prompt templates) is documented in [docs/cowork-spawn-contract-1.12603.1.md](./docs/cowork-spawn-contract-1.12603.1.md) and encoded in `baseline.spawn`.
|
|
185
185
|
|