cowork-harness 0.24.0 → 0.26.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -3,8 +3,8 @@ name: cowork-harness
3
3
  description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
4
4
  metadata:
5
5
  author: cowork-harness
6
- version: 0.24.0
7
- tracks-harness: cowork-harness 0.24.0 (baseline desktop-1.18286.0)
6
+ version: 0.26.0
7
+ tracks-harness: cowork-harness 0.26.0 (baseline desktop-1.18286.0)
8
8
  ---
9
9
 
10
10
  # cowork-harness
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
22
22
  allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
23
23
  the highest-value part. Read it.
24
24
 
25
- > **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.24.0` (baseline
25
+ > **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.26.0` (baseline
26
26
  > `desktop-1.18286.0`). If your checkout is newer, prefer the live `--help`, `SPEC.md`, and
27
27
  > `docs/*.md` over this snapshot, and re-run the bundled linter.
28
28
 
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
38
38
  Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
39
39
 
40
40
  - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
41
- - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.24.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.24.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.24.0"`. **Pin `@>=0.24.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.24.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
41
+ - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.26.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.26.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.26.0"`. **Pin `@>=0.26.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.26.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
42
42
  - **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
43
43
  - **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
44
44
  - **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
@@ -458,6 +458,27 @@ are the ones that bite hardest.
458
458
  and re-assert is undetected — the notice flags this; re-record if the session changed. `verify-run` reads
459
459
  on-disk `assert:` against a kept *run dir*; `replay --assert-from` is the equivalent for a *cassette*.
460
460
 
461
+ 18. **`questions_count_max` counts sub-questions, not gates.** One `AskUserQuestion` tool call can
462
+ bundle several sub-questions into a single gate; the assertion counts each sub-question, so a
463
+ 3-sub-question bundle counts as 3, not 1. `trace --view questions` shows the same per-gate
464
+ sub-question count and a matching footer total — read that off instead of the tool-call count when
465
+ sizing the budget.
466
+
467
+ 19. **`gate_answers_delivered` passes vacuously when no gate fires — use `gate_answer_count_min: 1` to
468
+ also require a gate.** Whether a gate fires is model-dependent, so `gate_answers_delivered: true`
469
+ alone can't catch "the gate never fired at all"; pair it with `gate_answer_count_min` when
470
+ presence matters, not just delivery.
471
+
472
+ 20. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
473
+ read-only folder's files as path + hash only (`truncated: true`, no `body`) — it's an input the agent
474
+ read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass against it on replay
475
+ (the hash-only entry still materializes a placeholder); `artifact_json` reports a clear
476
+ evidence-unavailable on every lane (live/verify-run/replay agree — no green-record/red-replay). This is
477
+ also why a `mode: r` input never trips the `binary` privacy finding or needs `--allow` — only a
478
+ *committed* body is scanned. `scaffold` won't emit `file_exists` for one either (it's not in
479
+ `RunResult.artifacts`). A `mode: rw`/`rwd` folder's contents are captured with a full body, same as
480
+ `outputs/`.
481
+
461
482
  For the complete gotcha list, the assertion catalog, the YAML schema, the fidelity/answer tables,
462
483
  and the CI recipe, read the files in `references/`.
463
484
 
@@ -1,6 +1,6 @@
1
1
  # CI recipe — replay vs live lanes
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 0.24.0` (baseline `desktop-1.18286.0`).
3
+ Self-contained reference. Tracks `cowork-harness 0.26.0` (baseline `desktop-1.18286.0`).
4
4
 
5
5
  **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
6
6
  job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -53,7 +53,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
53
53
  GitHub-hosted runners, no token/Docker/agent:
54
54
 
55
55
  ```yaml
56
- - run: npm i -g "cowork-harness@>=0.24.0"
56
+ - run: npm i -g "cowork-harness@>=0.26.0"
57
57
  - run: cowork-harness lint scenarios/*.yaml # no silent false-greens
58
58
  - run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
59
59
  - run: cowork-harness replay cassettes/ # token-free content/structure
@@ -187,7 +187,7 @@ jobs:
187
187
  with: { node-version: '20' }
188
188
  - uses: actions/setup-python@v5
189
189
  with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
190
- - run: npm i -g "cowork-harness@>=0.24.0"
190
+ - run: npm i -g "cowork-harness@>=0.26.0"
191
191
  - run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
192
192
  - run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
193
193
  - run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
@@ -216,7 +216,7 @@ jobs:
216
216
  echo "live=true" >> "$GITHUB_OUTPUT"
217
217
  fi
218
218
  - if: steps.guard.outputs.live == 'true'
219
- run: npm i -g "cowork-harness@>=0.24.0"
219
+ run: npm i -g "cowork-harness@>=0.26.0"
220
220
  - if: steps.guard.outputs.live == 'true'
221
221
  run: cowork-harness run scenarios/ --output-format json
222
222
  env:
@@ -1,6 +1,6 @@
1
1
  # Fidelity tiers & answer paths
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 0.24.0` (baseline `desktop-1.18286.0`).
3
+ Self-contained reference. Tracks `cowork-harness 0.26.0` (baseline `desktop-1.18286.0`).
4
4
 
5
5
  ## Fidelity tiers (`fidelity:` in the scenario)
6
6
 
@@ -1,6 +1,6 @@
1
1
  # Scenario & session schema, assertion catalog, web_fetch, full gotchas
2
2
 
3
- Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.24.0`
3
+ Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.26.0`
4
4
  (baseline `desktop-1.18286.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
5
5
  `docs/session.md`, and `SPEC.md`.
6
6
 
@@ -237,9 +237,10 @@ passes only if every key passes. Keep one concern per item unless you mean conju
237
237
  | `tool_calls_max: <N>` | total top-level tool calls (sum of `toolCounts`) ≤ N — meaningfully replay-checkable (re-drive recomputes `toolCounts` deterministically) |
238
238
  | `max_turns: <N>` | the SDK-reported (or fallback-counted) turn count ≤ N — meaningfully replay-checkable (re-drive recounts turns deterministically, same as `tool_calls_max`) |
239
239
  | `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
240
- | `questions_count_max: <N>` | the agent asked at most N questions |
241
- | `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail) |
240
+ | `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
241
+ | `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min` to also require a gate |
242
242
  | `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests |
243
+ | `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass |
243
244
  | `allow_permissive_auto_allow: true` | verdict modifier — suppresses the default-fail when the run recorded a cowork-parity permissive auto-allow; for tests that deliberately assert Cowork's permissive behavior |
244
245
  | `allow_missing_capability: true` | verdict modifier — suppresses the default-fail when the (partial "core") agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables). Assert only when the skill's fallback is genuinely equivalent; otherwise rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check. Live tiers only |
245
246
  | `allow_l0_plugin_divergence: true` | verdict modifier — opt into L0/protocol plugin divergence: suppresses the default-fail when a plugin behaves differently at `protocol` (L0) fidelity than under a sandboxed tier. Live tiers only |
@@ -293,8 +294,10 @@ modifiers `allow_permissive_auto_allow` / `allow_missing_capability` / `allow_l0
293
294
  `allow_stall` are also kept on replay, evaluated as no-op passes.
294
295
 
295
296
  **Gate keys — replay only with a `controlOut` cassette:** `question_asked`, `questions_count_max`,
296
- `gate_answers_delivered`. With `controlOut` present they evaluate; on an old cassette without it, a
297
- **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
297
+ `gate_answers_delivered`, `gate_answer_count_min`. With `controlOut` present they evaluate; on an old
298
+ cassette without it, a **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
299
+ `questions_count_max` counts sub-questions, not gates/tool-calls — see the catalog row above and
300
+ `trace --view questions`.
298
301
 
299
302
  **Filesystem — replay-checkable WITH an artifact manifest:** `file_exists`, `user_visible_artifact`,
300
303
  `artifact_json`, `computer_links_resolve` run on replay when the cassette carries an `artifacts` snapshot
@@ -374,10 +377,12 @@ at the top of this file.
374
377
  concern per item; run the linter. (`contentKeys` in `src/run/cassette.ts`.)
375
378
 
376
379
  2. **Gate keys need a `controlOut` cassette.** `question_asked`, `questions_count_max`,
377
- `gate_answers_delivered` only evaluate on replay with `controlOut`; on an old cassette they warn
378
- and are excluded (not passed). `gate_answers_delivered` **fails on unobserved delivery**
379
- (`delivered: null`) — absence of evidence is failure. A **header-only gate** (empty `question`,
380
- only `header`) can never be keyed and is rejected loudly — every gate needs a non-empty `question`.
380
+ `gate_answers_delivered`, `gate_answer_count_min` only evaluate on replay with `controlOut`; on an
381
+ old cassette they warn and are excluded (not passed). `gate_answers_delivered` **fails on
382
+ unobserved delivery** (`delivered: null`) — absence of evidence is failure — but **passes
383
+ vacuously when zero gates fired**; use `gate_answer_count_min: 1` to also require a gate to have
384
+ fired. A **header-only gate** (empty `question`, only `header`) can never be keyed and is rejected
385
+ loudly — every gate needs a non-empty `question`.
381
386
  (`contentKeys` in `src/run/cassette.ts`; `src/assert.ts`.)
382
387
 
383
388
  3. **The LLM-decider's two spellings.** Scripted answers + `on_unanswered: fail` is deterministic;
@@ -455,3 +460,14 @@ at the top of this file.
455
460
  Consequence: artifact assertions (`artifact_json`) over fields that were redacted will fail at replay
456
461
  — the harness emits `::warning::` at record time when this occurs. (`src/secrets.ts:scrubField`;
457
462
  `src/run/cassette.ts`.)
463
+
464
+ 21. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
465
+ read-only folder's files as `path` + `bytes` + `sha256` only (`truncated: true`, no `body`) — it's an
466
+ input the agent read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass
467
+ against it on replay (the hash-only entry still materializes a 0-byte placeholder); `artifact_json`
468
+ reports a clear evidence-unavailable on every lane (live/verify-run/replay agree). This is also why a
469
+ `mode: r` input never trips the `binary` privacy finding
470
+ or needs `--allow` in `verify-cassettes` — only a *committed* body is scanned. `scaffold` won't emit
471
+ `file_exists` for one either, since it isn't in `RunResult.artifacts`. A `mode: rw`/`rwd` folder's
472
+ contents are captured with a full body, same as `outputs/`. (`src/run/cassette.ts:buildManifest`'s
473
+ `bodyLessPrefixes`; `src/session.ts:readonlyFolderRootsFromPlan`.)
@@ -59,6 +59,7 @@ Top-level fields of a `*.cassette.json` (schema `schema/cassette.v7.json`):
59
59
  | `artifacts` | Output-file manifest (paths + hashes + small inlined bodies) — unlocks `file_exists`/`artifact_json` on replay |
60
60
  | `fingerprint` | Skill/baseline staleness tripwire |
61
61
  | `userVisibleRoots` | The user-visible mount roots captured at record time |
62
+ | `readonlyFolderRoots` | The read-only (`mode: r`) subset of `userVisibleRoots` — captured body-less; lets replay give `artifact_json` the precise remedy for a body-less input |
62
63
  | `preRunPaths` | Pre-run file-path baseline for `no_unexpected_files` (workRoot-relative; co-present with `userVisibleRoots`) |
63
64
  | `scenarioSource` | Relative path to the authored YAML this was recorded from |
64
65
  | `authoring` | Present iff a live decider answered ≥1 gate during recording (`nonDeterministic: true`) |
@@ -83,10 +84,11 @@ record — retrofitting means re-recording:
83
84
  redaction missed.
84
85
  5. **Scoping idioms for scanner findings you've reviewed and accepted:** prefer the class-scoped
85
86
  flags — `--allow-domain 'api\.example\.com'`, `--allow-email '.*@yourco\.com'`,
86
- `--allow-path '/opt/ci-runner/.*'` — over bare `--allow` (which applies to every class and can
87
- let a domain allow swallow an email leak). For a version-controlled allowlist use
88
- `--allow-file <path>`: bare all-class patterns, one regex per line, `#` comments and blank
89
- lines ignored.
87
+ `--allow-path '/opt/ci-runner/.*'` — over bare `--allow <regex>` (a single pattern, which
88
+ applies to every class and can let a domain allow swallow an email leak). For a
89
+ version-controlled allowlist use `--allow-patterns-file <path>`: a FILE of bare all-class
90
+ patterns, one regex per line, `#` comments and blank lines ignored — note the flag name is the
91
+ FILE of patterns, not a path to allow.
90
92
 
91
93
  ## Recipe 4 — Budget assertions without a two-pass record
92
94
 
@@ -11,6 +11,7 @@
11
11
  "egress_allowed",
12
12
  "egress_denied",
13
13
  "file_exists",
14
+ "gate_answer_count_min",
14
15
  "gate_answers_delivered",
15
16
  "max_cost_usd",
16
17
  "max_tokens",
@@ -72,7 +72,7 @@ CONTENT_KEYS = {
72
72
  "max_turns",
73
73
  }
74
74
  # content keys, but only evaluated on replay when the cassette carries controlOut
75
- GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered"}
75
+ GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered", "gate_answer_count_min"}
76
76
  # manifest-backed: replay-checkable when the cassette carries an `artifacts` manifest (record snapshots one);
77
77
  # a manifest-less cassette skips them. Since the 0.3.0 artifact-manifest these are NOT always live-only.
78
78
  # computer_links_resolve joins this bucket (not CONTENT_KEYS): resolving a non-empty link set needs
@@ -716,7 +716,15 @@ def build_scenario(args):
716
716
  if gates:
717
717
  for rx, _ in gates:
718
718
  content_lines.append(f" - question_asked: {_sq(rx)} # gate key: replay only with a controlOut cassette")
719
- content_lines.append(f" - questions_count_max: {len(gates)}")
719
+ # questions_count_max counts SUB-questions at runtime (assert.ts/trace-view.ts), but this
720
+ # scaffold is STATIC — it only knows the number of --gate rules (per-tool-call), never how many
721
+ # sub-questions each gate bundles. Any number emitted here would be a guess: too low false-reds
722
+ # on the first run, too high is a dead tripwire. A budget must come from observation, not
723
+ # fabrication — so emit it COMMENTED OUT with the calibration path, not a made-up value.
724
+ content_lines.append(
725
+ " # - questions_count_max: <N> # BUDGET — calibrate from a real run: `trace --view "
726
+ "questions` prints the SUB-question total (what this asserts); set N to that + headroom."
727
+ )
720
728
  content_lines.append(" - gate_answers_delivered: true # the steered answers actually reached the model")
721
729
 
722
730
  live_lines = []
package/CHANGELOG.md CHANGED
@@ -6,6 +6,117 @@ All notable changes to this project are documented here. The format is based on
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [0.26.0] — 2026-07-05
10
+
11
+ Cassette/run-result **format-freeze** pass before 1.0 — fixes the parts of these schemas that become
12
+ irreversible once 1.0's semver contract freezes them.
13
+
14
+ > **Upgrade notes.**
15
+ > - **Staleness-hash framing changed → `cassetteVersion` 8.** A pre-v8 cassette's staleness fingerprint
16
+ > is non-comparable with v8, so `verify-cassettes`/`rehash` route a pre-v8 cassette to **re-record**
17
+ > with an honest "algorithm changed" message (not a misleading "content changed"). Re-record your
18
+ > committed cassettes. (Cassettes without a `skillHash` are unaffected and keep replaying.)
19
+ > - **`RunResult.unanswered` renamed to `nonReproducibleAnswers`.** Consumers reading
20
+ > `result.unanswered` must switch to `result.nonReproducibleAnswers` (the field never held literally
21
+ > unanswered gates — it holds gates answered by a non-reproducible source).
22
+ > - **Cassette-level `readonlyFolderRoots` removed** in favor of per-entry
23
+ > `ManifestEntry.truncationReason`. No consumer action unless you parsed the cassette directly.
24
+
25
+ ### Fixed
26
+
27
+ - **Two staleness-hash framing collisions (a false-negative — the risk the gate exists to prevent).**
28
+ `skillHash` folded raw file content after a `F:<path>\0` marker, so a file whose content embedded that
29
+ marker could hash identically to a two-file tree; it now folds the fixed-length content SHA
30
+ (self-delimiting, charset disjoint from the `F:`/`L:` prefixes). `contentSig` lacked a type marker and
31
+ used `:` between path and sha, so a file named `a:lnk` could collide with a symlink `a`; entries are now
32
+ type-prefixed (`F:`/`L:`) and NUL-framed, and the `L:` link line NUL-separates path from target. Only
33
+ fixable before the hash algorithm freezes at 1.0.
34
+
35
+ ### Changed
36
+
37
+ - **Cassette format v7 → v8** (`CONTENTSIG_ALGO` 3 → 4). See the framing fix above and the upgrade note.
38
+ - **`RunResult.unanswered` → `nonReproducibleAnswers`** — a self-admitted misnomer in the frozen
39
+ run-result envelope, renamed before the 1.0 freeze.
40
+ - **Body-less manifest entries now carry `ManifestEntry.truncationReason`** (`size` | `readonly` |
41
+ `unreadable`) instead of a cassette-level `readonlyFolderRoots` list — self-describing and
42
+ redaction-immune. Replay reads the reason to give `artifact_json` the precise remedy on every lane
43
+ (assert on a deliverable vs. raise `--max-artifact-bytes`).
44
+ - **`schema/run-result.json` completed** (added `fingerprint`, `userVisibleRoots`, `readonlyFolderRoots`,
45
+ `preRunPaths`, `toolResults`) and now strict-validated by a test, and the `cassette.v8.json`
46
+ `ManifestEntry` shape is fully described — so the schemas frozen at 1.0 match what ships. Corrected the
47
+ git-mode-default documentation (git is the default unless `COWORK_HARNESS_GITSET=0`).
48
+
49
+ ## [0.25.0] — 2026-07-05
50
+
51
+ > **Upgrade notes.**
52
+ > - **`verify-cassettes --allow-file` was renamed to `--allow-patterns-file`** and the old spelling is
53
+ > removed (exits 2). Update any CI step or allowlist wiring that passed `--allow-file`.
54
+ > - **`gate_answers_delivered: true` now passes vacuously when zero gates fired.** If you relied on it
55
+ > as an implicit "a gate must fire" check, add `gate_answer_count_min: 1` alongside it.
56
+ > - **Read-only (`mode: r`) connected-folder inputs are now captured body-less.** Re-record affected
57
+ > cassettes to drop the `--allow` entries they previously required and shrink them; an `artifact_json`
58
+ > assertion pointed at a read-only input now reports evidence-unavailable (assert on a deliverable).
59
+ > This changes committed cassette *contents* (not the staleness hash) — a re-record is cosmetic, not
60
+ > required for correctness.
61
+
62
+ ### Added
63
+
64
+ - **`gate_answer_count_min: <N>` assertion** — the presence companion to `gate_answers_delivered`:
65
+ fails unless at least N `AskUserQuestion` gates fired AND were delivered non-error. Pairs with
66
+ `gate_answers_delivered`'s new zero-gate vacuous-pass (below) the way `transcript_contains` pairs
67
+ with `computer_links_resolve`'s zero-link vacuous-pass. Evaluated on replay only with a `controlOut`
68
+ cassette, like the other gate keys; fails as evidence-unavailable (not vacuous-pass) when
69
+ gate-delivery telemetry is absent from `result.json`.
70
+
71
+ ### Changed
72
+
73
+ - **`gate_answers_delivered: true` now passes vacuously when zero `AskUserQuestion` gates fired**,
74
+ instead of hard-failing. Whether a gate fires is model-dependent, so any skill with optional gating
75
+ could not use the assertion under the old hard-fail. The bad-delivery check (an answered gate whose
76
+ answer wasn't confirmed delivered) is unchanged — it only ever mattered when gates fired. Missing
77
+ gate-delivery telemetry (an old/partial `result.json` on the verify-run lane) still fails loud as
78
+ evidence-unavailable, never vacuous-pass — this is NOT the same as "no gate fired" and preserves the
79
+ bug-#33 false-green fix. Pair with the new `gate_answer_count_min` (below) to also require presence.
80
+ - **`verify-cassettes --allow-file <path>` renamed to `--allow-patterns-file <path>`.** The old name
81
+ read as "allow this file" and was routinely confused with `--allow <regex>` (a pattern matched
82
+ against a finding) — reaching for `--allow-file <artifact-path>` failed with an ENOENT/invalid-regex
83
+ error instead of doing what the user meant. `--allow-patterns-file <path>` is self-documenting: the
84
+ path names a **file of patterns** (one regex per line), not a path to allow. Pre-1.0 rename with **no
85
+ deprecation window** — the old `--allow-file` spelling is hard-removed and now exits 2 with an
86
+ unknown-flag error. The binary-finding recourse message and both `--help` texts now spell out the
87
+ distinction between the two flags.
88
+
89
+ ### Fixed
90
+
91
+ - **Read-only (`mode: r`) connected-folder inputs are captured body-less (path + hash) instead of as
92
+ full binary bodies.** `record` used to sweep a `mode: r` folder's files into the cassette
93
+ `artifacts[]` as full base64 bodies — an input the agent only reads, not a deliverable — causing
94
+ cassette bloat and a hard `binary` privacy finding that forced `--allow`. The manifest entry now
95
+ carries `path` + `bytes` + `sha256` with `truncated: true` and no `body`, reusing the same
96
+ representation the 64-KiB inline cap already produces. The entry still survives in the manifest (it
97
+ is NOT excluded), so `computer_links_resolve`/`file_exists` resolve identically on live and replay.
98
+ `artifact_json` against a body-less target (a read-only input, or any artifact over the body cap)
99
+ reports a clear **evidence-unavailable** on every lane — live, verify-run, and replay all agree, so
100
+ a cassette can't record green and replay red. The cassette persists the read-only-folder set
101
+ (`readonlyFolderRoots`, a subset of `userVisibleRoots`), so replay knows *why* an entry is body-less
102
+ and gives the **precise** remedy — "assert on a deliverable" for a read-only input, "raise
103
+ `--max-artifact-bytes`" for an over-cap artifact — instead of a cryptic JSON parse error or a
104
+ guessed hint. A `mode: rw`/`rwd` folder's contents are unaffected and keep their full body.
105
+ - **`trace --view questions` and the `scaffold` helper disagreed with `questions_count_max` on what
106
+ "a question" counts.** The assertion counts **sub-questions** (one `AskUserQuestion` bundling K
107
+ sub-questions counts as K — the better spam/burden budget, since per-tool-call counting would miss
108
+ a 10-in-1 bundle), but `trace --view questions` collapsed a bundle into one row with no count, and
109
+ the scaffold helper emitted `questions_count_max: <gate count>` — a budget that could start BELOW
110
+ what the assertion computes and false-red on the first real run. No behavior change to the
111
+ assertion itself: `trace --view questions` now annotates each gate row with its sub-question count
112
+ and prints a footer total (`N gate(s), M sub-question(s) total`) that matches what
113
+ `questions_count_max` compares against. The static `scaffold` helper can't know the sub-question
114
+ count offline, so instead of fabricating a value it now emits `questions_count_max` **commented
115
+ out** as a calibration TODO pointing at `trace --view questions` (a budget must come from
116
+ observation, not a guess — a fabricated one either false-reds or is a dead tripwire). The count
117
+ definition is now documented in `docs/scenario.md`, `docs/cassette.md`, `SPEC.md`, and the skill's
118
+ `references/scenario-schema.md` / `SKILL.md` gotcha list.
119
+
9
120
  ## [0.24.0] — 2026-07-05
10
121
 
11
122
  ### Added
package/README.md CHANGED
@@ -84,7 +84,7 @@ Only the committed-cassette `replay` above is fully self-contained. Live `run`/`
84
84
  > - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
85
85
  > - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
86
86
  > - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
87
- > - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx cowork-harness@>=0.24.0`.
87
+ > - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx cowork-harness@>=0.26.0`.
88
88
  >
89
89
  > *(One project, three names: the repo folder is `claude-cowork-headless-emulator`; the npm package, the CLI command, and the GitHub repo are all `cowork-harness`.)*
90
90
 
@@ -104,7 +104,7 @@ This repo ships a **companion skill** (`.claude/skills/cowork-harness/`) that te
104
104
  /plugin install cowork-harness@cowork-harness
105
105
  ```
106
106
 
107
- The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.24.0` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
107
+ The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.26.0` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
108
108
 
109
109
  It also follows the open [Agent Skills](https://github.com/vercel-labs/skills) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) by pointing the `npx skills` CLI at `.claude/skills/cowork-harness` in this repo. (Working *inside* this repo, the skill auto-loads as a project skill — no install needed.)
110
110
 
@@ -119,7 +119,7 @@ A global install is enough for CI `lint`, reading the teaching skill, and replay
119
119
  To `run` the worked examples live or copy them as a starting point, use a source checkout. (The marketplace
120
120
  skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
121
121
  assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
122
- above becomes available once the skill's first command self-bootstraps `npx cowork-harness@>=0.24.0` — see
122
+ above becomes available once the skill's first command self-bootstraps `npx cowork-harness@>=0.26.0` — see
123
123
  [above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
124
124
 
125
125
  ### Prerequisites for anything above `protocol` fidelity
@@ -179,7 +179,7 @@ cowork-harness lint examples/scenarios/*.yaml
179
179
  > `tool_calls_max`/`max_turns`, `result`, the verdict modifiers) always — note `max_cost_usd`/`max_tokens`
180
180
  > assert the *frozen recording's* spend on replay, not fresh spend; a real budget regression is caught by a
181
181
  > live `run` — plus `question_asked`/
182
- > `questions_count_max`/`gate_answers_delivered` when the cassette carries `controlOut`, and
182
+ > `questions_count_max`/`gate_answers_delivered`/`gate_answer_count_min` when the cassette carries `controlOut`, and
183
183
  > **filesystem assertions** (`file_exists`, `user_visible_artifact`, `artifact_json`,
184
184
  > `computer_links_resolve`) when it carries an
185
185
  > `artifacts` manifest. Genuinely live-only keys (`egress_*`, `expect_denied`, `no_delete_in_outputs`,
@@ -307,7 +307,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
307
307
  | `run <scenario.yaml \| dir/>` | Run authored scenarios with `assert:` + a CI-ready exit code; a decider (`--decider-cmd`/`--decider-dir`, or a scenario's `on_unanswered: llm`) can answer unscripted gates. `--repeat N` (2-100) runs each scenario N times and aggregates a variance rollup instead of a single pass/fail — `--min-pass-rate`, `--stop-on-diverge`, `--max-budget-usd` tune the batch verdict/loop. `--matrix <matrix.yaml>` runs ONE scenario across the cross-product of baseline/model/skill_dir axes (worked example: `examples/matrices/csv-metrics-matrix.yaml`) — `--max-cells`/`--concurrency` tune the cap/pool; any cell failing (assertion or infra) fails the run; `--compact`/`--demo` trim output for shareable screenshots/GIFs (shared with `skill`, below) | you want a repeatable, **asserted regression test** — to **measure flakiness** instead of trusting one green — or to **test a compatibility matrix** (multiple baselines/models/skill variants) in one run |
308
308
  | `chat <folder> [prompt]` | Interactive multi-turn REPL against a skill (TTY); optional seed prompt is sent as the first turn. `--upload <file>` / `--folder <dir>` (repeatable) attach files/project folders; `--verbose` shows thinking blocks + tool inputs; `--fidelity protocol\|container\|hostloop` (no `microvm`/`cowork` in the REPL) | debugging a multi-turn flow by hand |
309
309
  | `record` / `replay` | **Record a live run once → replay it token-free, Docker-free thereafter.** Key flags: `--decider-llm`/`--decider-dir` (answer gates live during recording), `--rerecord-stale`/`--concurrency <N>` (batch re-recording), `--assert-from <scenario.yaml>`/`--reassert` (re-check the on-disk `assert:` instead of the frozen one), `--strict`/`--fail-on-skill-drift` (staleness handling on replay), `--no-redact` (skip record-time redaction), `--allow-failing` (relax the post-run verdict gate), `--dry-run` (resolve without recording). Full flag list: `cowork-harness record --help` / `replay --help`. | **token-free, Docker-free CI** from a once-recorded run |
310
- | `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain/path/machine-inventory → exit 1; whole-token allows via `--allow` / class-scoped `--allow-domain` / `--allow-email` / `--allow-path` / `--allow-machine-inventory` / `--allow-file`) + a staleness check (`--skip-privacy` or `--skip-staleness` to run only one half). A dir argument scans `*.cassette.json` in that dir only (**non-recursive**) | gating **committed cassettes** against PII leaks + "edited the skill, forgot to re-record" |
310
+ | `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain/path/machine-inventory → exit 1; whole-token allows via `--allow <regex>` (a pattern) / class-scoped `--allow-domain` / `--allow-email` / `--allow-path` / `--allow-machine-inventory` / `--allow-patterns-file <path>` (a **file** of patterns, one regex per line)) + a staleness check (`--skip-privacy` or `--skip-staleness` to run only one half). A dir argument scans `*.cassette.json` in that dir only (**non-recursive**) | gating **committed cassettes** against PII leaks + "edited the skill, forgot to re-record" |
311
311
  | `verify-run <run-dir> <scenario.yaml>` | Re-evaluate a scenario's `assert:` (and, when the scenario declares `answers:`, whether they still match the run's actual gates) against an already-kept run dir — **no live agent, no tokens, no Docker** (~1s) | iterating on a wrong assertion or a drifted `answer` without a full live re-record |
312
312
  | `trace <run-id>` | Digest a run's `events.jsonl` (`--view tools\|questions\|dispatches`; legacy `--tools` / `--gates` / `--dispatches` aliases still accepted) | "how many sub-agents *actually* dispatched, and which?" |
313
313
  | `inspect <run-id>` | Show what a run **produced**: the artifacts + a shallow field preview of each JSON artifact (`--output-format json` for a digest). Works on a salvaged partial run too | "did it do the job?" — without hand-parsing `…/mnt/outputs/…` |
package/SPEC.md CHANGED
@@ -384,7 +384,7 @@ states in `baseline.provenance.gates`). A skill that ignores these behaves diffe
384
384
  `tool_*`, `subagent_*`, `dispatch_count_max`, `skill_triggered`, `no_skill_triggered`, `max_cost_usd`,
385
385
  `max_tokens`, `tool_calls_max`, `max_turns`, `result`, the verdict modifiers (`allow_permissive_auto_allow`,
386
386
  `allow_missing_capability`, `allow_l0_plugin_divergence`, `allow_stall`), and (when `controlOut` is present)
387
- `question_asked`, `questions_count_max`, `gate_answers_delivered`. `max_cost_usd`/`max_tokens` are
387
+ `question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min`. `max_cost_usd`/`max_tokens` are
388
388
  evaluated against the *frozen recording's* usage/cost on replay, not fresh spend; `tool_calls_max`/
389
389
  `max_turns` are meaningfully replay-checkable (the re-drive recomputes both deterministically).
390
390
  - **Filesystem assertions** (`file_exists`, `user_visible_artifact`, `artifact_json`, `computer_links_resolve`,
@@ -405,10 +405,21 @@ green replay does not imply the recording is still valid. Each finding is surfac
405
405
  `RunResult.staleness[]` for a token-free gate to act on. `replay --strict` fails on any class;
406
406
  `replay --fail-on-skill-drift` fails only on skill-source classes (`skill` / `shared-root` /
407
407
  `unverifiable-skill`). Both realize the gate as failing `assertions[]` entries (so `ok`/exit stay consistent).
408
- - **`question_asked` / `questions_count_max` / `gate_answers_delivered`** additionally require
409
- `controlOut`. Without it, a loud `::warning::` fires and these keys are excluded (not vacuously
410
- passed). The authoritative list is `contentKeys` in `src/run/cassette.ts`; `docs/cassette.md` mirrors
411
- it — consult it for the full table.
408
+ - **`question_asked` / `questions_count_max` / `gate_answers_delivered` / `gate_answer_count_min`**
409
+ additionally require `controlOut`. Without it, a loud `::warning::` fires and these keys are
410
+ excluded (not vacuously passed). The authoritative list is `contentKeys` in `src/run/cassette.ts`;
411
+ `docs/cassette.md` mirrors it — consult it for the full table.
412
+ - **`gate_answers_delivered: true` passes vacuously when zero `AskUserQuestion` gates fired** (gate
413
+ firing is model-dependent). Pair it with `gate_answer_count_min: <N>` to also require that at least
414
+ N gates fired AND were delivered non-error — the presence companion, mirroring
415
+ `computer_links_resolve` (zero-passes) paired with `transcript_contains` (presence). Both keys fail
416
+ evidence-unavailable, never vacuous-pass, when gate-delivery telemetry itself is absent from
417
+ `result.json` (an old/partial run on the verify-run lane).
418
+ - **`questions_count_max` counts sub-questions, not `AskUserQuestion` tool calls/gates.** A bundled
419
+ gate with K sub-questions counts as K (`src/run/run.ts`'s recorder pushes one `rec.questions` entry
420
+ per sub-question; `src/assert.ts` compares against that count). `trace --view questions` shows the
421
+ same per-gate sub-question count and a matching footer total, so the two surfaces agree — see
422
+ `docs/cassette.md` / `docs/scenario.md`'s `questions_count_max` row.
412
423
 
413
424
  **Assertion source (replay):** by default `replay` evaluates the `assert:` block **frozen in the cassette**
414
425
  — byte-deterministic and independent of the working tree. When a sibling scenario resolves and its `assert:`
@@ -493,19 +504,25 @@ and a fail observed) always fails that batch, regardless of the numeric rate.
493
504
  "egress": [{ "host","decision":"allow|deny" }],
494
505
  "assertions":[{ "assertion": <Assertion>, "pass": bool, "message?": "string" }],
495
506
  "subagents": [{ "toolUseId","parentToolUseId?","agentType","declaredTools":[],"toolsUsed":[] }],
496
- "unanswered":[{ "question","chosen","by","rationale?","model?" }],
507
+ "nonReproducibleAnswers?":[{ "question","chosen","by","rationale?","model?" }], // decisions answered by a non-deterministic/non-authoritative source (llm/external/human/first); scripted answers are excluded
497
508
  "usage?": { "turns?": number, /* …SDK usage fields (input_tokens, output_tokens, etc.), pass-through */ },
498
509
  "cost?": { "usd?": number, "raw?": {...} }, // usd = SDK's total_cost_usd for this invocation; raw = the api_metrics event payload (independent source)
499
510
  "durationMs?": number,
511
+ "fingerprint?": { "baseline", "skillHash?", "skillSources?":[], "skillScope?":[], "sharedHash?", "contentSig?", "fileSigs?":[["relpath","contentSha"]], "fileSigsOmitted?": bool, "mode?": "git|raw", "agentScope?": "skill" }, // skill/plugin staleness fingerprint recorded at run time; lets `verify-run` detect a kept run whose gate snapshot predates a skill change
500
512
  "outDir": "string",
501
513
  "workDir?": "string", // the agent's working root (mnt/) inside the run dir
502
514
  "outputsDir?": "string", // the user-visible deliverable mount (mnt/outputs)
515
+ "userVisibleRoots?": ["string"], // user-visible mount roots (relative to mnt/) — `outputs` plus each connected folder's resolved mount name; plugins excluded
516
+ "readonlyFolderRoots?": ["string"], // subset of userVisibleRoots that are read-only (mode:"r") connected-folder mounts — inputs, not deliverables; `artifacts` excludes them
517
+ "artifacts?": [{ "path","bytes" }], // files written under the user-visible roots (paths + sizes only — no content snapshot)
518
+ "preRunPaths?": ["string"], // workRoot-relative paths under the user-visible roots that existed BEFORE the agent ran — the `no_unexpected_files` baseline; absent when the tier didn't capture it (microvm) or the run predates the seam
503
519
  "effectiveFidelity?": "string", // tier actually used (differs from `fidelity` when "cowork" resolved)
504
520
  "nonDeterministic?": bool, // true if any decision came from a non-deterministic source → not reproducible
505
521
  "gateProvenance?": { "total": number, "bySource": {…}, "gates": [{ "question","answeredBy","answer","model?" }] }, // how each AskUserQuestion gate was answered; informational (never fails the verdict); live/partial lane only (absent on replay)
506
522
  "permissiveAutoAllow?": ["string"], // tools auto-allowed by cowork parity that real Cowork BLOCKS → green is NOT faithful
507
523
  "staleness?": [{ "class": "baseline|skill|shared-root|format|unverifiable-baseline|unverifiable-skill|resolved-tier|unverifiable-tier", "message" }], // replay only; cassette-staleness findings, surfaced for a JSON gate. Non-failing by default (a stale but passing replay stays ok:true); `--strict` fails on every class, `--fail-on-skill-drift` on skill/shared-root/unverifiable-skill only. `resolved-tier` = a `fidelity: cowork` cassette's recorded effectiveFidelity no longer matches the tier the scenario's baseline (pinned `baseline:` or `latest`) resolves to today — the recording exercises the wrong tier; `unverifiable-tier` = the tier check couldn't run for a baseline-dependent (`fidelity: cowork`) cassette (no recorded effectiveFidelity, or its pinned baseline failed to load). Tier resolution is baseline-only (the CLAUDE_FORCE_HOST_LOOP env override is suppressed) so verify results can't differ across machines.
508
524
  "skippedAssertions?": { "full": number, "partial": number }, // replay only; count of live-only assertions NOT evaluated (full = whole assertion skipped; partial = content half ran, fs/egress half dropped). The skipped ones are absent from `assertions[]`.
525
+ "toolResults?": [{ "toolUseId?","isError","text","assertText?" }], // tool-result text at assertion-fidelity cap (10 KB); backs tool_result_contains/tool_result_not_contains
509
526
  "skillsInvoked?": ["string"], // Wave 1: skill/plugin ids invoked via the Skill tool_use event, call order, duplicates kept. Backs skill_triggered/no_skill_triggered.
510
527
  "skillToolAvailable?": bool // Wave 1: whether the agent's init tool list included "Skill" — false ⇒ skill_triggered/no_skill_triggered fail as evidence-unavailable (agent-version drift)
511
528
  }
@@ -630,9 +647,10 @@ nothing here is guaranteed; minor versions may break any surface — see [RELEAS
630
647
  `findings` / `staleness` / `notes` / `version` / `error` channels. This is the machine output the
631
648
  CI recipes and the packaged Action steer consumers to parse; renaming or removing a key is
632
649
  breaking, adding one is not.
633
- - **Cassette format** — the current `cassetteVersion` (**7**) and its verdict-modifier assertion keys.
634
- Older-version cassettes (the retained `schema/cassette.v2/v3/v5/v6.json` — no v4 schema was ever
635
- published) stay replayable; dropping a still-emitted version's readability is breaking.
650
+ - **Cassette format** — the current `cassetteVersion` (**8**, `schema/cassette.v8.json`) and its
651
+ verdict-modifier assertion keys. Older-version cassettes (the retained
652
+ `schema/cassette.v2/v3/v5/v6/v7.json` — no v4 schema was ever published) stay replayable; dropping a
653
+ still-emitted version's readability is breaking.
636
654
  - **Control protocol** — `schema/protocol.v1.json` + the golden control-response vectors (§5).
637
655
  - **Environment variables** — the documented `COWORK_HARNESS_*` knobs plus `COWORK_AGENT_BINARY` and
638
656
  `COWORK_AGENT_IMAGE`. Renaming a documented var or changing its meaning is breaking.
package/dist/assert.js CHANGED
@@ -166,7 +166,7 @@ function check(a, ctx) {
166
166
  const results = [];
167
167
  const ok = () => ({ pass: true });
168
168
  const fail = (message) => ({ pass: false, message });
169
- const truncated = ctx.truncatedPaths ?? new Set();
169
+ const truncated = ctx.truncatedPaths ?? new Map();
170
170
  if (a.transcript_contains !== undefined)
171
171
  results.push(ctx.transcriptMissing
172
172
  ? fail(`evidence unavailable: transcript sidecar (run.jsonl) absent — cannot evaluate transcript_contains`)
@@ -471,10 +471,13 @@ function check(a, ctx) {
471
471
  // run/cassette, an unobserved delivery (delivered=null) is NOT neutral — it is absence of the
472
472
  // evidence the assertion requires, so it fails loud ("no silent false-greens"). `delivered:
473
473
  // false` is a real errored tool_result; `null` is "no tool_result observed for this gate".
474
- if (a.gate_answers_delivered) {
475
- if (ctx.gateDeliveries.length === 0) {
476
- results.push(fail("gate_answers_delivered: no gates were recorded during this run — either no gate fired, or gate delivery tracking is not wired. Expected at least one delivered gate."));
477
- }
474
+ // Zero gates fired passes vacuously (whether a gate fires is model-dependent) — pair with
475
+ // gate_answer_count_min to also require presence. Missing telemetry (gateDeliveriesMissing)
476
+ // is NOT the same as zero gates and must fail evidence-unavailable, not vacuous-pass.
477
+ if (ctx.gateDeliveriesMissing) {
478
+ results.push(fail(`evidence unavailable: gate-delivery telemetry absent from result.json — cannot evaluate gate_answers_delivered`));
479
+ }
480
+ else if (a.gate_answers_delivered) {
478
481
  const bad = ctx.gateDeliveries.filter((g) => g.delivered !== true);
479
482
  results.push(bad.length === 0
480
483
  ? ok()
@@ -492,6 +495,17 @@ function check(a, ctx) {
492
495
  results.push(failedConfirmed.length > 0 ? ok() : fail(`expected a confirmed gate-delivery failure but none was observed`));
493
496
  }
494
497
  }
498
+ if (a.gate_answer_count_min !== undefined) {
499
+ if (ctx.gateDeliveriesMissing) {
500
+ results.push(fail(`evidence unavailable: gate-delivery telemetry absent from result.json — cannot evaluate gate_answer_count_min`));
501
+ }
502
+ else {
503
+ const delivered = ctx.gateDeliveries.filter((g) => g.delivered === true).length;
504
+ results.push(delivered >= a.gate_answer_count_min
505
+ ? ok()
506
+ : fail(`only ${delivered} gate answer(s) confirmed delivered, need ≥ ${a.gate_answer_count_min}`));
507
+ }
508
+ }
495
509
  if (a.artifact_json !== undefined) {
496
510
  const aj = a.artifact_json;
497
511
  const file = containedPath(ctx.workRoot, aj.artifact);
@@ -500,12 +514,41 @@ function check(a, ctx) {
500
514
  else {
501
515
  // verify the real path (after symlink resolution) is still under workRoot.
502
516
  const realFile = containedRealPath(ctx.workRoot, file);
517
+ // A body-less manifest entry (a read-only connected-folder input, or an artifact over the body
518
+ // cap) has no content in the cassette — artifact_json cannot be evaluated on replay (the 0-byte
519
+ // placeholder isn't parseable). To keep record/verify-run/replay SYMMETRIC (no green-record →
520
+ // red-replay), treat such a target as evidence-unavailable on EVERY lane: `truncatedPaths` flags
521
+ // it on replay; `readonlyFolderRoots` flags the read-only-input case on the live/verify-run lanes
522
+ // where the real file is still on disk. (Existence keys stay green — existence is provable from
523
+ // the recorded hash — but content is genuinely absent, so this fails loud, never vacuous.)
524
+ const rel = relative(resolve(ctx.workRoot), file);
525
+ // Reason sources by lane: LIVE/verify-run derive read-only from the plan's `readonlyFolderRoots`
526
+ // (no manifest exists at eval time); REPLAY reads the per-entry `truncationReason` off the
527
+ // materialized manifest (`truncated.get(rel)`). Keeping both is complementary, not redundant.
528
+ const liveReadonly = (ctx.readonlyFolderRoots ?? []).some((pre) => rel === pre || rel.startsWith(pre + "/"));
529
+ const replayReason = truncated.get(rel); // undefined if not body-less on replay, or a pre-v8 entry with no reason
530
+ const isReadonlyInput = liveReadonly || replayReason === "readonly";
531
+ const isOverCap = replayReason === "size";
532
+ const bodyLess = truncated.has(rel) || liveReadonly;
503
533
  if (!realFile) {
504
534
  results.push(fail(`unsafe artifact_json path "${aj.artifact}" — symlink target escapes the work root`));
505
535
  }
506
536
  else if (!existsSync(realFile)) {
507
537
  results.push(fail(`artifact_json: file not found: ${aj.artifact} (under ${ctx.workRoot})`));
508
538
  }
539
+ else if (bodyLess) {
540
+ // Precise remedy when the cause is known (read-only ⇒ assert on a deliverable; over-cap ⇒ raise
541
+ // the cap). A pre-v8 entry carries no reason ⇒ name both causes (we can't tell). "unreadable"
542
+ // also falls here — it's a record-time read failure, so the both-causes text is the safe hint.
543
+ const cause = isReadonlyInput
544
+ ? `(read-only connected-folder input — its content is never captured; assert artifact_json on a deliverable instead)`
545
+ : isOverCap
546
+ ? `(larger than the artifact-body cap — raise --max-artifact-bytes to capture it)`
547
+ : `(a read-only connected-folder input, or an artifact larger than the body cap — if an input, assert on a deliverable; if a large deliverable, raise --max-artifact-bytes)`;
548
+ results.push(fail(`evidence unavailable: artifact_json target "${aj.artifact}" was captured body-less ` +
549
+ cause +
550
+ ` — content is not in the cassette, so it cannot be evaluated on replay`));
551
+ }
509
552
  else {
510
553
  let doc;
511
554
  let parsed = true;