cowork-harness 0.24.0 → 0.25.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -3,8 +3,8 @@ name: cowork-harness
3
3
  description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
4
4
  metadata:
5
5
  author: cowork-harness
6
- version: 0.24.0
7
- tracks-harness: cowork-harness 0.24.0 (baseline desktop-1.18286.0)
6
+ version: 0.25.0
7
+ tracks-harness: cowork-harness 0.25.0 (baseline desktop-1.18286.0)
8
8
  ---
9
9
 
10
10
  # cowork-harness
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
22
22
  allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
23
23
  the highest-value part. Read it.
24
24
 
25
- > **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.24.0` (baseline
25
+ > **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.25.0` (baseline
26
26
  > `desktop-1.18286.0`). If your checkout is newer, prefer the live `--help`, `SPEC.md`, and
27
27
  > `docs/*.md` over this snapshot, and re-run the bundled linter.
28
28
 
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
38
38
  Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
39
39
 
40
40
  - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
41
- - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.24.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.24.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.24.0"`. **Pin `@>=0.24.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.24.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
41
+ - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.25.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.25.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.25.0"`. **Pin `@>=0.25.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.25.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
42
42
  - **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
43
43
  - **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
44
44
  - **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
@@ -458,6 +458,27 @@ are the ones that bite hardest.
458
458
  and re-assert is undetected — the notice flags this; re-record if the session changed. `verify-run` reads
459
459
  on-disk `assert:` against a kept *run dir*; `replay --assert-from` is the equivalent for a *cassette*.
460
460
 
461
+ 18. **`questions_count_max` counts sub-questions, not gates.** One `AskUserQuestion` tool call can
462
+ bundle several sub-questions into a single gate; the assertion counts each sub-question, so a
463
+ 3-sub-question bundle counts as 3, not 1. `trace --view questions` shows the same per-gate
464
+ sub-question count and a matching footer total — read that off instead of the tool-call count when
465
+ sizing the budget.
466
+
467
+ 19. **`gate_answers_delivered` passes vacuously when no gate fires — use `gate_answer_count_min: 1` to
468
+ also require a gate.** Whether a gate fires is model-dependent, so `gate_answers_delivered: true`
469
+ alone can't catch "the gate never fired at all"; pair it with `gate_answer_count_min` when
470
+ presence matters, not just delivery.
471
+
472
+ 20. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
473
+ read-only folder's files as path + hash only (`truncated: true`, no `body`) — it's an input the agent
474
+ read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass against it on replay
475
+ (the hash-only entry still materializes a placeholder); `artifact_json` reports a clear
476
+ evidence-unavailable on every lane (live/verify-run/replay agree — no green-record/red-replay). This is
477
+ also why a `mode: r` input never trips the `binary` privacy finding or needs `--allow` — only a
478
+ *committed* body is scanned. `scaffold` won't emit `file_exists` for one either (it's not in
479
+ `RunResult.artifacts`). A `mode: rw`/`rwd` folder's contents are captured with a full body, same as
480
+ `outputs/`.
481
+
461
482
  For the complete gotcha list, the assertion catalog, the YAML schema, the fidelity/answer tables,
462
483
  and the CI recipe, read the files in `references/`.
463
484
 
@@ -1,6 +1,6 @@
1
1
  # CI recipe — replay vs live lanes
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 0.24.0` (baseline `desktop-1.18286.0`).
3
+ Self-contained reference. Tracks `cowork-harness 0.25.0` (baseline `desktop-1.18286.0`).
4
4
 
5
5
  **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
6
6
  job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -53,7 +53,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
53
53
  GitHub-hosted runners, no token/Docker/agent:
54
54
 
55
55
  ```yaml
56
- - run: npm i -g "cowork-harness@>=0.24.0"
56
+ - run: npm i -g "cowork-harness@>=0.25.0"
57
57
  - run: cowork-harness lint scenarios/*.yaml # no silent false-greens
58
58
  - run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
59
59
  - run: cowork-harness replay cassettes/ # token-free content/structure
@@ -187,7 +187,7 @@ jobs:
187
187
  with: { node-version: '20' }
188
188
  - uses: actions/setup-python@v5
189
189
  with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
190
- - run: npm i -g "cowork-harness@>=0.24.0"
190
+ - run: npm i -g "cowork-harness@>=0.25.0"
191
191
  - run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
192
192
  - run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
193
193
  - run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
@@ -216,7 +216,7 @@ jobs:
216
216
  echo "live=true" >> "$GITHUB_OUTPUT"
217
217
  fi
218
218
  - if: steps.guard.outputs.live == 'true'
219
- run: npm i -g "cowork-harness@>=0.24.0"
219
+ run: npm i -g "cowork-harness@>=0.25.0"
220
220
  - if: steps.guard.outputs.live == 'true'
221
221
  run: cowork-harness run scenarios/ --output-format json
222
222
  env:
@@ -1,6 +1,6 @@
1
1
  # Fidelity tiers & answer paths
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 0.24.0` (baseline `desktop-1.18286.0`).
3
+ Self-contained reference. Tracks `cowork-harness 0.25.0` (baseline `desktop-1.18286.0`).
4
4
 
5
5
  ## Fidelity tiers (`fidelity:` in the scenario)
6
6
 
@@ -1,6 +1,6 @@
1
1
  # Scenario & session schema, assertion catalog, web_fetch, full gotchas
2
2
 
3
- Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.24.0`
3
+ Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.25.0`
4
4
  (baseline `desktop-1.18286.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
5
5
  `docs/session.md`, and `SPEC.md`.
6
6
 
@@ -237,9 +237,10 @@ passes only if every key passes. Keep one concern per item unless you mean conju
237
237
  | `tool_calls_max: <N>` | total top-level tool calls (sum of `toolCounts`) ≤ N — meaningfully replay-checkable (re-drive recomputes `toolCounts` deterministically) |
238
238
  | `max_turns: <N>` | the SDK-reported (or fallback-counted) turn count ≤ N — meaningfully replay-checkable (re-drive recounts turns deterministically, same as `tool_calls_max`) |
239
239
  | `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
240
- | `questions_count_max: <N>` | the agent asked at most N questions |
241
- | `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail) |
240
+ | `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
241
+ | `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min` to also require a gate |
242
242
  | `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests |
243
+ | `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass |
243
244
  | `allow_permissive_auto_allow: true` | verdict modifier — suppresses the default-fail when the run recorded a cowork-parity permissive auto-allow; for tests that deliberately assert Cowork's permissive behavior |
244
245
  | `allow_missing_capability: true` | verdict modifier — suppresses the default-fail when the (partial "core") agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables). Assert only when the skill's fallback is genuinely equivalent; otherwise rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check. Live tiers only |
245
246
  | `allow_l0_plugin_divergence: true` | verdict modifier — opt into L0/protocol plugin divergence: suppresses the default-fail when a plugin behaves differently at `protocol` (L0) fidelity than under a sandboxed tier. Live tiers only |
@@ -293,8 +294,10 @@ modifiers `allow_permissive_auto_allow` / `allow_missing_capability` / `allow_l0
293
294
  `allow_stall` are also kept on replay, evaluated as no-op passes.
294
295
 
295
296
  **Gate keys — replay only with a `controlOut` cassette:** `question_asked`, `questions_count_max`,
296
- `gate_answers_delivered`. With `controlOut` present they evaluate; on an old cassette without it, a
297
- **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
297
+ `gate_answers_delivered`, `gate_answer_count_min`. With `controlOut` present they evaluate; on an old
298
+ cassette without it, a **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
299
+ `questions_count_max` counts sub-questions, not gates/tool-calls — see the catalog row above and
300
+ `trace --view questions`.
298
301
 
299
302
  **Filesystem — replay-checkable WITH an artifact manifest:** `file_exists`, `user_visible_artifact`,
300
303
  `artifact_json`, `computer_links_resolve` run on replay when the cassette carries an `artifacts` snapshot
@@ -374,10 +377,12 @@ at the top of this file.
374
377
  concern per item; run the linter. (`contentKeys` in `src/run/cassette.ts`.)
375
378
 
376
379
  2. **Gate keys need a `controlOut` cassette.** `question_asked`, `questions_count_max`,
377
- `gate_answers_delivered` only evaluate on replay with `controlOut`; on an old cassette they warn
378
- and are excluded (not passed). `gate_answers_delivered` **fails on unobserved delivery**
379
- (`delivered: null`) — absence of evidence is failure. A **header-only gate** (empty `question`,
380
- only `header`) can never be keyed and is rejected loudly — every gate needs a non-empty `question`.
380
+ `gate_answers_delivered`, `gate_answer_count_min` only evaluate on replay with `controlOut`; on an
381
+ old cassette they warn and are excluded (not passed). `gate_answers_delivered` **fails on
382
+ unobserved delivery** (`delivered: null`) — absence of evidence is failure — but **passes
383
+ vacuously when zero gates fired**; use `gate_answer_count_min: 1` to also require a gate to have
384
+ fired. A **header-only gate** (empty `question`, only `header`) can never be keyed and is rejected
385
+ loudly — every gate needs a non-empty `question`.
381
386
  (`contentKeys` in `src/run/cassette.ts`; `src/assert.ts`.)
382
387
 
383
388
  3. **The LLM-decider's two spellings.** Scripted answers + `on_unanswered: fail` is deterministic;
@@ -455,3 +460,14 @@ at the top of this file.
455
460
  Consequence: artifact assertions (`artifact_json`) over fields that were redacted will fail at replay
456
461
  — the harness emits `::warning::` at record time when this occurs. (`src/secrets.ts:scrubField`;
457
462
  `src/run/cassette.ts`.)
463
+
464
+ 21. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
465
+ read-only folder's files as `path` + `bytes` + `sha256` only (`truncated: true`, no `body`) — it's an
466
+ input the agent read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass
467
+ against it on replay (the hash-only entry still materializes a 0-byte placeholder); `artifact_json`
468
+ reports a clear evidence-unavailable on every lane (live/verify-run/replay agree). This is also why a
469
+ `mode: r` input never trips the `binary` privacy finding
470
+ or needs `--allow` in `verify-cassettes` — only a *committed* body is scanned. `scaffold` won't emit
471
+ `file_exists` for one either, since it isn't in `RunResult.artifacts`. A `mode: rw`/`rwd` folder's
472
+ contents are captured with a full body, same as `outputs/`. (`src/run/cassette.ts:buildManifest`'s
473
+ `bodyLessPrefixes`; `src/session.ts:readonlyFolderRootsFromPlan`.)
@@ -59,6 +59,7 @@ Top-level fields of a `*.cassette.json` (schema `schema/cassette.v7.json`):
59
59
  | `artifacts` | Output-file manifest (paths + hashes + small inlined bodies) — unlocks `file_exists`/`artifact_json` on replay |
60
60
  | `fingerprint` | Skill/baseline staleness tripwire |
61
61
  | `userVisibleRoots` | The user-visible mount roots captured at record time |
62
+ | `readonlyFolderRoots` | The read-only (`mode: r`) subset of `userVisibleRoots` — captured body-less; lets replay give `artifact_json` the precise remedy for a body-less input |
62
63
  | `preRunPaths` | Pre-run file-path baseline for `no_unexpected_files` (workRoot-relative; co-present with `userVisibleRoots`) |
63
64
  | `scenarioSource` | Relative path to the authored YAML this was recorded from |
64
65
  | `authoring` | Present iff a live decider answered ≥1 gate during recording (`nonDeterministic: true`) |
@@ -83,10 +84,11 @@ record — retrofitting means re-recording:
83
84
  redaction missed.
84
85
  5. **Scoping idioms for scanner findings you've reviewed and accepted:** prefer the class-scoped
85
86
  flags — `--allow-domain 'api\.example\.com'`, `--allow-email '.*@yourco\.com'`,
86
- `--allow-path '/opt/ci-runner/.*'` — over bare `--allow` (which applies to every class and can
87
- let a domain allow swallow an email leak). For a version-controlled allowlist use
88
- `--allow-file <path>`: bare all-class patterns, one regex per line, `#` comments and blank
89
- lines ignored.
87
+ `--allow-path '/opt/ci-runner/.*'` — over bare `--allow <regex>` (a single pattern, which
88
+ applies to every class and can let a domain allow swallow an email leak). For a
89
+ version-controlled allowlist use `--allow-patterns-file <path>`: a FILE of bare all-class
90
+ patterns, one regex per line, `#` comments and blank lines ignored — note the flag name is the
91
+ FILE of patterns, not a path to allow.
90
92
 
91
93
  ## Recipe 4 — Budget assertions without a two-pass record
92
94
 
@@ -11,6 +11,7 @@
11
11
  "egress_allowed",
12
12
  "egress_denied",
13
13
  "file_exists",
14
+ "gate_answer_count_min",
14
15
  "gate_answers_delivered",
15
16
  "max_cost_usd",
16
17
  "max_tokens",
@@ -72,7 +72,7 @@ CONTENT_KEYS = {
72
72
  "max_turns",
73
73
  }
74
74
  # content keys, but only evaluated on replay when the cassette carries controlOut
75
- GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered"}
75
+ GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered", "gate_answer_count_min"}
76
76
  # manifest-backed: replay-checkable when the cassette carries an `artifacts` manifest (record snapshots one);
77
77
  # a manifest-less cassette skips them. Since the 0.3.0 artifact-manifest these are NOT always live-only.
78
78
  # computer_links_resolve joins this bucket (not CONTENT_KEYS): resolving a non-empty link set needs
@@ -716,7 +716,15 @@ def build_scenario(args):
716
716
  if gates:
717
717
  for rx, _ in gates:
718
718
  content_lines.append(f" - question_asked: {_sq(rx)} # gate key: replay only with a controlOut cassette")
719
- content_lines.append(f" - questions_count_max: {len(gates)}")
719
+ # questions_count_max counts SUB-questions at runtime (assert.ts/trace-view.ts), but this
720
+ # scaffold is STATIC — it only knows the number of --gate rules (per-tool-call), never how many
721
+ # sub-questions each gate bundles. Any number emitted here would be a guess: too low false-reds
722
+ # on the first run, too high is a dead tripwire. A budget must come from observation, not
723
+ # fabrication — so emit it COMMENTED OUT with the calibration path, not a made-up value.
724
+ content_lines.append(
725
+ " # - questions_count_max: <N> # BUDGET — calibrate from a real run: `trace --view "
726
+ "questions` prints the SUB-question total (what this asserts); set N to that + headroom."
727
+ )
720
728
  content_lines.append(" - gate_answers_delivered: true # the steered answers actually reached the model")
721
729
 
722
730
  live_lines = []
package/CHANGELOG.md CHANGED
@@ -6,6 +6,77 @@ All notable changes to this project are documented here. The format is based on
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [0.25.0] — 2026-07-05
10
+
11
+ > **Upgrade notes.**
12
+ > - **`verify-cassettes --allow-file` was renamed to `--allow-patterns-file`** and the old spelling is
13
+ > removed (exits 2). Update any CI step or allowlist wiring that passed `--allow-file`.
14
+ > - **`gate_answers_delivered: true` now passes vacuously when zero gates fired.** If you relied on it
15
+ > as an implicit "a gate must fire" check, add `gate_answer_count_min: 1` alongside it.
16
+ > - **Read-only (`mode: r`) connected-folder inputs are now captured body-less.** Re-record affected
17
+ > cassettes to drop the `--allow` entries they previously required and shrink them; an `artifact_json`
18
+ > assertion pointed at a read-only input now reports evidence-unavailable (assert on a deliverable).
19
+ > This changes committed cassette *contents* (not the staleness hash) — a re-record is cosmetic, not
20
+ > required for correctness.
21
+
22
+ ### Added
23
+
24
+ - **`gate_answer_count_min: <N>` assertion** — the presence companion to `gate_answers_delivered`:
25
+ fails unless at least N `AskUserQuestion` gates fired AND were delivered non-error. Pairs with
26
+ `gate_answers_delivered`'s new zero-gate vacuous-pass (below) the way `transcript_contains` pairs
27
+ with `computer_links_resolve`'s zero-link vacuous-pass. Evaluated on replay only with a `controlOut`
28
+ cassette, like the other gate keys; fails as evidence-unavailable (not vacuous-pass) when
29
+ gate-delivery telemetry is absent from `result.json`.
30
+
31
+ ### Changed
32
+
33
+ - **`gate_answers_delivered: true` now passes vacuously when zero `AskUserQuestion` gates fired**,
34
+ instead of hard-failing. Whether a gate fires is model-dependent, so any skill with optional gating
35
+ could not use the assertion under the old hard-fail. The bad-delivery check (an answered gate whose
36
+ answer wasn't confirmed delivered) is unchanged — it only ever mattered when gates fired. Missing
37
+ gate-delivery telemetry (an old/partial `result.json` on the verify-run lane) still fails loud as
38
+ evidence-unavailable, never vacuous-pass — this is NOT the same as "no gate fired" and preserves the
39
+ bug-#33 false-green fix. Pair with the new `gate_answer_count_min` (below) to also require presence.
40
+ - **`verify-cassettes --allow-file <path>` renamed to `--allow-patterns-file <path>`.** The old name
41
+ read as "allow this file" and was routinely confused with `--allow <regex>` (a pattern matched
42
+ against a finding) — reaching for `--allow-file <artifact-path>` failed with an ENOENT/invalid-regex
43
+ error instead of doing what the user meant. `--allow-patterns-file <path>` is self-documenting: the
44
+ path names a **file of patterns** (one regex per line), not a path to allow. Pre-1.0 rename with **no
45
+ deprecation window** — the old `--allow-file` spelling is hard-removed and now exits 2 with an
46
+ unknown-flag error. The binary-finding recourse message and both `--help` texts now spell out the
47
+ distinction between the two flags.
48
+
49
+ ### Fixed
50
+
51
+ - **Read-only (`mode: r`) connected-folder inputs are captured body-less (path + hash) instead of as
52
+ full binary bodies.** `record` used to sweep a `mode: r` folder's files into the cassette
53
+ `artifacts[]` as full base64 bodies — an input the agent only reads, not a deliverable — causing
54
+ cassette bloat and a hard `binary` privacy finding that forced `--allow`. The manifest entry now
55
+ carries `path` + `bytes` + `sha256` with `truncated: true` and no `body`, reusing the same
56
+ representation the 64-KiB inline cap already produces. The entry still survives in the manifest (it
57
+ is NOT excluded), so `computer_links_resolve`/`file_exists` resolve identically on live and replay.
58
+ `artifact_json` against a body-less target (a read-only input, or any artifact over the body cap)
59
+ reports a clear **evidence-unavailable** on every lane — live, verify-run, and replay all agree, so
60
+ a cassette can't record green and replay red. The cassette persists the read-only-folder set
61
+ (`readonlyFolderRoots`, a subset of `userVisibleRoots`), so replay knows *why* an entry is body-less
62
+ and gives the **precise** remedy — "assert on a deliverable" for a read-only input, "raise
63
+ `--max-artifact-bytes`" for an over-cap artifact — instead of a cryptic JSON parse error or a
64
+ guessed hint. A `mode: rw`/`rwd` folder's contents are unaffected and keep their full body.
65
+ - **`trace --view questions` and the `scaffold` helper disagreed with `questions_count_max` on what
66
+ "a question" counts.** The assertion counts **sub-questions** (one `AskUserQuestion` bundling K
67
+ sub-questions counts as K — the better spam/burden budget, since per-tool-call counting would miss
68
+ a 10-in-1 bundle), but `trace --view questions` collapsed a bundle into one row with no count, and
69
+ the scaffold helper emitted `questions_count_max: <gate count>` — a budget that could start BELOW
70
+ what the assertion computes and false-red on the first real run. No behavior change to the
71
+ assertion itself: `trace --view questions` now annotates each gate row with its sub-question count
72
+ and prints a footer total (`N gate(s), M sub-question(s) total`) that matches what
73
+ `questions_count_max` compares against. The static `scaffold` helper can't know the sub-question
74
+ count offline, so instead of fabricating a value it now emits `questions_count_max` **commented
75
+ out** as a calibration TODO pointing at `trace --view questions` (a budget must come from
76
+ observation, not a guess — a fabricated one either false-reds or is a dead tripwire). The count
77
+ definition is now documented in `docs/scenario.md`, `docs/cassette.md`, `SPEC.md`, and the skill's
78
+ `references/scenario-schema.md` / `SKILL.md` gotcha list.
79
+
9
80
  ## [0.24.0] — 2026-07-05
10
81
 
11
82
  ### Added
package/README.md CHANGED
@@ -84,7 +84,7 @@ Only the committed-cassette `replay` above is fully self-contained. Live `run`/`
84
84
  > - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
85
85
  > - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
86
86
  > - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
87
- > - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx cowork-harness@>=0.24.0`.
87
+ > - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx cowork-harness@>=0.25.0`.
88
88
  >
89
89
  > *(One project, three names: the repo folder is `claude-cowork-headless-emulator`; the npm package, the CLI command, and the GitHub repo are all `cowork-harness`.)*
90
90
 
@@ -104,7 +104,7 @@ This repo ships a **companion skill** (`.claude/skills/cowork-harness/`) that te
104
104
  /plugin install cowork-harness@cowork-harness
105
105
  ```
106
106
 
107
- The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.24.0` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
107
+ The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.25.0` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
108
108
 
109
109
  It also follows the open [Agent Skills](https://github.com/vercel-labs/skills) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) by pointing the `npx skills` CLI at `.claude/skills/cowork-harness` in this repo. (Working *inside* this repo, the skill auto-loads as a project skill — no install needed.)
110
110
 
@@ -119,7 +119,7 @@ A global install is enough for CI `lint`, reading the teaching skill, and replay
119
119
  To `run` the worked examples live or copy them as a starting point, use a source checkout. (The marketplace
120
120
  skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
121
121
  assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
122
- above becomes available once the skill's first command self-bootstraps `npx cowork-harness@>=0.24.0` — see
122
+ above becomes available once the skill's first command self-bootstraps `npx cowork-harness@>=0.25.0` — see
123
123
  [above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
124
124
 
125
125
  ### Prerequisites for anything above `protocol` fidelity
@@ -179,7 +179,7 @@ cowork-harness lint examples/scenarios/*.yaml
179
179
  > `tool_calls_max`/`max_turns`, `result`, the verdict modifiers) always — note `max_cost_usd`/`max_tokens`
180
180
  > assert the *frozen recording's* spend on replay, not fresh spend; a real budget regression is caught by a
181
181
  > live `run` — plus `question_asked`/
182
- > `questions_count_max`/`gate_answers_delivered` when the cassette carries `controlOut`, and
182
+ > `questions_count_max`/`gate_answers_delivered`/`gate_answer_count_min` when the cassette carries `controlOut`, and
183
183
  > **filesystem assertions** (`file_exists`, `user_visible_artifact`, `artifact_json`,
184
184
  > `computer_links_resolve`) when it carries an
185
185
  > `artifacts` manifest. Genuinely live-only keys (`egress_*`, `expect_denied`, `no_delete_in_outputs`,
@@ -307,7 +307,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
307
307
  | `run <scenario.yaml \| dir/>` | Run authored scenarios with `assert:` + a CI-ready exit code; a decider (`--decider-cmd`/`--decider-dir`, or a scenario's `on_unanswered: llm`) can answer unscripted gates. `--repeat N` (2-100) runs each scenario N times and aggregates a variance rollup instead of a single pass/fail — `--min-pass-rate`, `--stop-on-diverge`, `--max-budget-usd` tune the batch verdict/loop. `--matrix <matrix.yaml>` runs ONE scenario across the cross-product of baseline/model/skill_dir axes (worked example: `examples/matrices/csv-metrics-matrix.yaml`) — `--max-cells`/`--concurrency` tune the cap/pool; any cell failing (assertion or infra) fails the run; `--compact`/`--demo` trim output for shareable screenshots/GIFs (shared with `skill`, below) | you want a repeatable, **asserted regression test** — to **measure flakiness** instead of trusting one green — or to **test a compatibility matrix** (multiple baselines/models/skill variants) in one run |
308
308
  | `chat <folder> [prompt]` | Interactive multi-turn REPL against a skill (TTY); optional seed prompt is sent as the first turn. `--upload <file>` / `--folder <dir>` (repeatable) attach files/project folders; `--verbose` shows thinking blocks + tool inputs; `--fidelity protocol\|container\|hostloop` (no `microvm`/`cowork` in the REPL) | debugging a multi-turn flow by hand |
309
309
  | `record` / `replay` | **Record a live run once → replay it token-free, Docker-free thereafter.** Key flags: `--decider-llm`/`--decider-dir` (answer gates live during recording), `--rerecord-stale`/`--concurrency <N>` (batch re-recording), `--assert-from <scenario.yaml>`/`--reassert` (re-check the on-disk `assert:` instead of the frozen one), `--strict`/`--fail-on-skill-drift` (staleness handling on replay), `--no-redact` (skip record-time redaction), `--allow-failing` (relax the post-run verdict gate), `--dry-run` (resolve without recording). Full flag list: `cowork-harness record --help` / `replay --help`. | **token-free, Docker-free CI** from a once-recorded run |
310
- | `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain/path/machine-inventory → exit 1; whole-token allows via `--allow` / class-scoped `--allow-domain` / `--allow-email` / `--allow-path` / `--allow-machine-inventory` / `--allow-file`) + a staleness check (`--skip-privacy` or `--skip-staleness` to run only one half). A dir argument scans `*.cassette.json` in that dir only (**non-recursive**) | gating **committed cassettes** against PII leaks + "edited the skill, forgot to re-record" |
310
+ | `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain/path/machine-inventory → exit 1; whole-token allows via `--allow <regex>` (a pattern) / class-scoped `--allow-domain` / `--allow-email` / `--allow-path` / `--allow-machine-inventory` / `--allow-patterns-file <path>` (a **file** of patterns, one regex per line)) + a staleness check (`--skip-privacy` or `--skip-staleness` to run only one half). A dir argument scans `*.cassette.json` in that dir only (**non-recursive**) | gating **committed cassettes** against PII leaks + "edited the skill, forgot to re-record" |
311
311
  | `verify-run <run-dir> <scenario.yaml>` | Re-evaluate a scenario's `assert:` (and, when the scenario declares `answers:`, whether they still match the run's actual gates) against an already-kept run dir — **no live agent, no tokens, no Docker** (~1s) | iterating on a wrong assertion or a drifted `answer` without a full live re-record |
312
312
  | `trace <run-id>` | Digest a run's `events.jsonl` (`--view tools\|questions\|dispatches`; legacy `--tools` / `--gates` / `--dispatches` aliases still accepted) | "how many sub-agents *actually* dispatched, and which?" |
313
313
  | `inspect <run-id>` | Show what a run **produced**: the artifacts + a shallow field preview of each JSON artifact (`--output-format json` for a digest). Works on a salvaged partial run too | "did it do the job?" — without hand-parsing `…/mnt/outputs/…` |
package/SPEC.md CHANGED
@@ -384,7 +384,7 @@ states in `baseline.provenance.gates`). A skill that ignores these behaves diffe
384
384
  `tool_*`, `subagent_*`, `dispatch_count_max`, `skill_triggered`, `no_skill_triggered`, `max_cost_usd`,
385
385
  `max_tokens`, `tool_calls_max`, `max_turns`, `result`, the verdict modifiers (`allow_permissive_auto_allow`,
386
386
  `allow_missing_capability`, `allow_l0_plugin_divergence`, `allow_stall`), and (when `controlOut` is present)
387
- `question_asked`, `questions_count_max`, `gate_answers_delivered`. `max_cost_usd`/`max_tokens` are
387
+ `question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min`. `max_cost_usd`/`max_tokens` are
388
388
  evaluated against the *frozen recording's* usage/cost on replay, not fresh spend; `tool_calls_max`/
389
389
  `max_turns` are meaningfully replay-checkable (the re-drive recomputes both deterministically).
390
390
  - **Filesystem assertions** (`file_exists`, `user_visible_artifact`, `artifact_json`, `computer_links_resolve`,
@@ -405,10 +405,21 @@ green replay does not imply the recording is still valid. Each finding is surfac
405
405
  `RunResult.staleness[]` for a token-free gate to act on. `replay --strict` fails on any class;
406
406
  `replay --fail-on-skill-drift` fails only on skill-source classes (`skill` / `shared-root` /
407
407
  `unverifiable-skill`). Both realize the gate as failing `assertions[]` entries (so `ok`/exit stay consistent).
408
- - **`question_asked` / `questions_count_max` / `gate_answers_delivered`** additionally require
409
- `controlOut`. Without it, a loud `::warning::` fires and these keys are excluded (not vacuously
410
- passed). The authoritative list is `contentKeys` in `src/run/cassette.ts`; `docs/cassette.md` mirrors
411
- it — consult it for the full table.
408
+ - **`question_asked` / `questions_count_max` / `gate_answers_delivered` / `gate_answer_count_min`**
409
+ additionally require `controlOut`. Without it, a loud `::warning::` fires and these keys are
410
+ excluded (not vacuously passed). The authoritative list is `contentKeys` in `src/run/cassette.ts`;
411
+ `docs/cassette.md` mirrors it — consult it for the full table.
412
+ - **`gate_answers_delivered: true` passes vacuously when zero `AskUserQuestion` gates fired** (gate
413
+ firing is model-dependent). Pair it with `gate_answer_count_min: <N>` to also require that at least
414
+ N gates fired AND were delivered non-error — the presence companion, mirroring
415
+ `computer_links_resolve` (zero-passes) paired with `transcript_contains` (presence). Both keys fail
416
+ evidence-unavailable, never vacuous-pass, when gate-delivery telemetry itself is absent from
417
+ `result.json` (an old/partial run on the verify-run lane).
418
+ - **`questions_count_max` counts sub-questions, not `AskUserQuestion` tool calls/gates.** A bundled
419
+ gate with K sub-questions counts as K (`src/run/run.ts`'s recorder pushes one `rec.questions` entry
420
+ per sub-question; `src/assert.ts` compares against that count). `trace --view questions` shows the
421
+ same per-gate sub-question count and a matching footer total, so the two surfaces agree — see
422
+ `docs/cassette.md` / `docs/scenario.md`'s `questions_count_max` row.
412
423
 
413
424
  **Assertion source (replay):** by default `replay` evaluates the `assert:` block **frozen in the cassette**
414
425
  — byte-deterministic and independent of the working tree. When a sibling scenario resolves and its `assert:`
package/dist/assert.js CHANGED
@@ -471,10 +471,13 @@ function check(a, ctx) {
471
471
  // run/cassette, an unobserved delivery (delivered=null) is NOT neutral — it is absence of the
472
472
  // evidence the assertion requires, so it fails loud ("no silent false-greens"). `delivered:
473
473
  // false` is a real errored tool_result; `null` is "no tool_result observed for this gate".
474
- if (a.gate_answers_delivered) {
475
- if (ctx.gateDeliveries.length === 0) {
476
- results.push(fail("gate_answers_delivered: no gates were recorded during this run — either no gate fired, or gate delivery tracking is not wired. Expected at least one delivered gate."));
477
- }
474
+ // Zero gates fired passes vacuously (whether a gate fires is model-dependent) — pair with
475
+ // gate_answer_count_min to also require presence. Missing telemetry (gateDeliveriesMissing)
476
+ // is NOT the same as zero gates and must fail evidence-unavailable, not vacuous-pass.
477
+ if (ctx.gateDeliveriesMissing) {
478
+ results.push(fail(`evidence unavailable: gate-delivery telemetry absent from result.json — cannot evaluate gate_answers_delivered`));
479
+ }
480
+ else if (a.gate_answers_delivered) {
478
481
  const bad = ctx.gateDeliveries.filter((g) => g.delivered !== true);
479
482
  results.push(bad.length === 0
480
483
  ? ok()
@@ -492,6 +495,17 @@ function check(a, ctx) {
492
495
  results.push(failedConfirmed.length > 0 ? ok() : fail(`expected a confirmed gate-delivery failure but none was observed`));
493
496
  }
494
497
  }
498
+ if (a.gate_answer_count_min !== undefined) {
499
+ if (ctx.gateDeliveriesMissing) {
500
+ results.push(fail(`evidence unavailable: gate-delivery telemetry absent from result.json — cannot evaluate gate_answer_count_min`));
501
+ }
502
+ else {
503
+ const delivered = ctx.gateDeliveries.filter((g) => g.delivered === true).length;
504
+ results.push(delivered >= a.gate_answer_count_min
505
+ ? ok()
506
+ : fail(`only ${delivered} gate answer(s) confirmed delivered, need ≥ ${a.gate_answer_count_min}`));
507
+ }
508
+ }
495
509
  if (a.artifact_json !== undefined) {
496
510
  const aj = a.artifact_json;
497
511
  const file = containedPath(ctx.workRoot, aj.artifact);
@@ -500,12 +514,33 @@ function check(a, ctx) {
500
514
  else {
501
515
  // verify the real path (after symlink resolution) is still under workRoot.
502
516
  const realFile = containedRealPath(ctx.workRoot, file);
517
+ // A body-less manifest entry (a read-only connected-folder input, or an artifact over the body
518
+ // cap) has no content in the cassette — artifact_json cannot be evaluated on replay (the 0-byte
519
+ // placeholder isn't parseable). To keep record/verify-run/replay SYMMETRIC (no green-record →
520
+ // red-replay), treat such a target as evidence-unavailable on EVERY lane: `truncatedPaths` flags
521
+ // it on replay; `readonlyFolderRoots` flags the read-only-input case on the live/verify-run lanes
522
+ // where the real file is still on disk. (Existence keys stay green — existence is provable from
523
+ // the recorded hash — but content is genuinely absent, so this fails loud, never vacuous.)
524
+ const rel = relative(resolve(ctx.workRoot), file);
525
+ const isReadonlyInput = (ctx.readonlyFolderRoots ?? []).some((pre) => rel === pre || rel.startsWith(pre + "/"));
526
+ const bodyLess = truncated.has(rel) || isReadonlyInput;
503
527
  if (!realFile) {
504
528
  results.push(fail(`unsafe artifact_json path "${aj.artifact}" — symlink target escapes the work root`));
505
529
  }
506
530
  else if (!existsSync(realFile)) {
507
531
  results.push(fail(`artifact_json: file not found: ${aj.artifact} (under ${ctx.workRoot})`));
508
532
  }
533
+ else if (bodyLess) {
534
+ // Every lane knows WHY the entry is body-less, so the remedy is always precise: live/verify-run
535
+ // derive `readonlyFolderRoots` from the plan; replay reads the set persisted on the cassette
536
+ // (v7). A read-only input can't be captured (assert on a deliverable); an over-cap artifact can
537
+ // (raise the cap). `isReadonlyInput` distinguishes them on all three lanes.
538
+ results.push(fail(`evidence unavailable: artifact_json target "${aj.artifact}" was captured body-less ` +
539
+ (isReadonlyInput
540
+ ? `(read-only connected-folder input — its content is never captured; assert artifact_json on a deliverable instead)`
541
+ : `(larger than the artifact-body cap — raise --max-artifact-bytes to capture it)`) +
542
+ ` — content is not in the cassette, so it cannot be evaluated on replay`));
543
+ }
509
544
  else {
510
545
  let doc;
511
546
  let parsed = true;
package/dist/cli.js CHANGED
@@ -91,7 +91,8 @@ const HELP = `cowork-harness <command> (v${"$VERSION"})
91
91
  [--output-format json]
92
92
  verify-cassettes <file|dir> CI gate (no token): privacy scan + staleness — exit 1 on finding or drift
93
93
  [--skip-privacy|--skip-staleness] skip one check
94
- [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-file <path>]... [--output-format json]
94
+ [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-patterns-file <path>]... [--output-format json]
95
+ --allow <regex> is a PATTERN (matched against a finding); --allow-patterns-file <path> is a FILE of patterns, one regex per line — not a path to allow
95
96
  rehash <dir/> migrate cassette fingerprints to current version when content is provably unchanged (requires contentSig from v3+)
96
97
  init-redact [--force] copy the packaged reference .cowork-redact.json into the cwd (redaction starter
97
98
  for hostloop/protocol recordings; review + tailor the patterns before recording)
@@ -342,7 +343,8 @@ const SUBCOMMAND_USAGE = {
342
343
  replay: "usage: replay <file.cassette.json | dir/> [--strict] [--fail-on-skill-drift] [--assert-from <scenario.yaml> | --reassert] [--output-format text|json]\n" +
343
344
  " by default the assertions FROZEN in the cassette drive the verdict (deterministic); a sibling scenario whose assert: differs only prints a notice.\n" +
344
345
  " --assert-from <file> / --reassert: token-free re-check against the on-disk assert:/expect_denied: — recording-shaping drift (prompt/answers/baseline/skills) and skill staleness HARD-FAIL.",
345
- "verify-cassettes": "usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-file <path>]... [--output-format json]",
346
+ "verify-cassettes": "usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-patterns-file <path>]... [--output-format json]\n" +
347
+ " --allow <regex> is a PATTERN (matched against a finding); --allow-patterns-file <path> is a FILE of patterns, one regex per line — not a path to allow.",
346
348
  trace: "usage: trace <run-id | run-dir | events.jsonl> [--view tools|questions|dispatches] [--translate-paths] [--output-format json]\n --view tools tool call / result rows\n --view questions gate lifecycle (question → answer → delivered)\n --view dispatches sub-agent dispatch tree + dispatch_count_max\n --translate-paths rewrite VM paths to host paths in the tools/default TEXT views only (needs a sibling mounts.json + an effective hostloop run; questions/dispatches views and --output-format json are unaffected)\n (default: all views)\n (for what the run PRODUCED — artifacts — use `inspect`)",
347
349
  assertions: "usage: assertions --list [--output-format json]",
348
350
  scaffold: "usage: scaffold <run-id | run-dir> [--out <file.yaml>] [--output-format text|json]\n Turns a kept run into a starter scenario YAML (gates→answers, artifacts→file_exists).\n Positional <run-id | run-dir> is the canonical form.",
@@ -2788,6 +2790,9 @@ async function cmdVerifyRun(args) {
2788
2790
  // Read the roots persisted at run time (folder mount names are dynamic/gated, not a fixed prefix).
2789
2791
  // Fall back to the legacy prefix for old result.json that predates the field.
2790
2792
  userVisiblePrefixes: result.userVisibleRoots ?? ["outputs", ".projects"],
2793
+ // Read-only folder inputs are captured body-less; keep artifact_json's verdict identical to the
2794
+ // replay lane (evidence-unavailable) instead of parsing the real on-disk input here.
2795
+ readonlyFolderRoots: result.readonlyFolderRoots ?? [],
2791
2796
  // result.json is the single source: every writer populates the field from the run's own
2792
2797
  // pre-run-manifest.json, so a missing field means the baseline genuinely doesn't exist
2793
2798
  // (pre-field run, or the run never captured) — evidence-unavailable, loud.
@@ -2798,6 +2803,7 @@ async function cmdVerifyRun(args) {
2798
2803
  selfHealRan: scan.selfHealRan,
2799
2804
  subagents: result.subagents ?? [],
2800
2805
  gateDeliveries: result.gateDeliveries ?? [],
2806
+ gateDeliveriesMissing: result.gateDeliveries === undefined,
2801
2807
  toolResultTexts: (result.toolResults ?? []).map((r) => r.assertText ?? r.text),
2802
2808
  toolResultsTruncated: (result.toolResults ?? []).map((r) => r.assertText === undefined),
2803
2809
  transcriptMissing: sidecarTranscript === null,
@@ -133,8 +133,14 @@ function isLosslessUtf8(buf) {
133
133
  }
134
134
  /** Snapshot the user-visible artifacts under `workRoot` into manifest entries.
135
135
  * Exported for token-free record→replay round-trip tests. */
136
- export function buildManifest(workRoot, cap, roots = ["outputs", ".projects"]) {
136
+ export function buildManifest(workRoot, cap, roots = ["outputs", ".projects"], bodyLessPrefixes = []) {
137
137
  const limit = cap ?? defaultBodyCap();
138
+ // Read-only connected-folder inputs (`bodyLessPrefixes`) are captured path+bytes+sha256 only, same as
139
+ // an over-cap entry — no body, so no bloat and no `binary` privacy finding (cassette.ts binary scan
140
+ // only fires on a committed base64 body). The manifest entry SURVIVES (unlike full exclusion) so
141
+ // `materializeManifest` writes a 0-byte placeholder and `computer_links_resolve` resolves identically
142
+ // on live and replay (see T3 in the pre-1.0 fix plan).
143
+ const isBodyLess = (path) => bodyLessPrefixes.some((prefix) => path === prefix || path.startsWith(prefix + "/"));
138
144
  return collectArtifacts(workRoot, roots).map(({ path, bytes }) => {
139
145
  // collectArtifacts paths are derived from a directory walk; even though it already skips symlinks,
140
146
  // an entry must not inline content outside the work root. Re-confirm containment before reading the body.
@@ -153,7 +159,7 @@ export function buildManifest(workRoot, cap, roots = ["outputs", ".projects"]) {
153
159
  return { path, bytes, sha256: "", truncated: true };
154
160
  }
155
161
  const sha256 = createHash("sha256").update(buf).digest("hex");
156
- if (buf.length > limit)
162
+ if (buf.length > limit || isBodyLess(path))
157
163
  return { path, bytes, sha256, truncated: true };
158
164
  // store an encoding marker. UTF-8-safe bodies stay text (readable cassettes); binary bodies go
159
165
  // base64 so the record→replay round-trip is byte-exact and the sha256 verify stays valid.
@@ -379,11 +385,12 @@ export function scanCassette(cassette, allow) {
379
385
  // zip/DEFLATE streams. Count a COMMITTED binary body as a real finding (cls "binary", NOT the
380
386
  // benign "unscanned" used for a TRUNCATED/uncommitted entry below) so the gate can't greenlight
381
387
  // raw recoverable PII. Recourse: after reviewing the deliverable, clear it with
382
- // `--allow <path-regex>` (matched on the artifact path above, since the body is unreadable).
388
+ // `--allow <path-regex>` (a PATTERN matched on the artifact path above, since the body is
389
+ // unreadable) — NOT `--allow-patterns-file`, which loads a FILE of patterns, not a path to allow.
383
390
  findings.push({
384
391
  where: `artifact ${a.path}`,
385
392
  cls: "binary",
386
- sample: `(committed binary body — not text-scannable; review and clear with --allow ${a.path})`,
393
+ sample: `(committed binary body — not text-scannable; review and clear with --allow ${a.path} (a pattern matched on this path); note --allow-patterns-file is a FILE of patterns, not this path)`,
387
394
  });
388
395
  }
389
396
  }
@@ -407,6 +414,9 @@ export function scanCassette(cassette, allow) {
407
414
  // cassettes`. Scan them too, prefixed `metadata:` so a reviewer knows redaction here ALSO rewrites
408
415
  // structural paths, distinct from free-text findings in the transcript/deliverable.
409
416
  (cassette.userVisibleRoots ?? []).forEach((r, i) => findings.push(...scanText(r, `metadata:userVisibleRoots[${i}]`, allow, FULL)));
417
+ // readonlyFolderRoots is a subset of userVisibleRoots (same strings), but it's a separately-persisted
418
+ // field — scan it too so a reviewer can't be surprised by an unscanned persisted string.
419
+ (cassette.readonlyFolderRoots ?? []).forEach((r, i) => findings.push(...scanText(r, `metadata:readonlyFolderRoots[${i}]`, allow, FULL)));
410
420
  findings.push(...scanText(cassette.scenario.name ?? "", "metadata:scenario.name", allow, FULL));
411
421
  findings.push(...scanText(cassette.scenario.session ?? "", "metadata:scenario.session", allow, FULL));
412
422
  if (cassette.scenarioSource)
@@ -1014,10 +1024,15 @@ export function redactCassette(cassette, policy) {
1014
1024
  }
1015
1025
  }
1016
1026
  const redactedPreRunPaths = cassette.preRunPaths?.map((p) => redactText(p, policy));
1027
+ // readonlyFolderRoots is a subset of userVisibleRoots; redact it with the same context-free redactText
1028
+ // so the prefix relationship with the (redacted) artifact paths is preserved. The `...cassette` spread
1029
+ // would otherwise carry the UNREDACTED set through.
1030
+ const redactedReadonly = cassette.readonlyFolderRoots?.map((r) => redactText(r, policy));
1017
1031
  return {
1018
1032
  ...cassette,
1019
1033
  scenario,
1020
1034
  userVisibleRoots: redactedRoots,
1035
+ readonlyFolderRoots: redactedReadonly,
1021
1036
  artifacts: redactedArtifacts,
1022
1037
  preRunPaths: redactedPreRunPaths,
1023
1038
  events: cassette.events.map((l) => redactJsonLine(l, policy)),
@@ -1299,14 +1314,18 @@ export function redactionPreflightMessage(items) {
1299
1314
  `COWORK_HARNESS_REDACT_PATTERNS. (The always-on privacy scanner remains the universal net — ` +
1300
1315
  `container-tier recordings can trip it too.)\n`);
1301
1316
  }
1302
- /** Return the `artifact_json.artifact` paths a scenario asserts that ended up TRUNCATED in the manifest
1303
- * (body >cap, hash-only). Such an assertion passes at record (evaluated on the live on-disk file) but FAILS
1304
- * at replay (the materialized body is empty → "not valid JSON"). Paths are normalized through `resolve` so
1305
- * `./outputs/x.json` and `outputs/x.json` join cleanly against the manifest's walk paths. */
1306
- export function artifactJsonTargetsTruncated(scenario, workRoot, artifacts) {
1317
+ /** Return the `artifact_json.artifact` paths a scenario asserts that ended up truncated by SIZE (body
1318
+ * >cap, hash-only) — the genuine green-record/red-replay case whose remedy is "raise the cap". This
1319
+ * EXCLUDES read-only connected-folder inputs: those are body-less by policy (not size), raising the cap
1320
+ * can't capture them, and `artifact_json` against one already fails LOUD-and-symmetrically at record,
1321
+ * verify-run, and replay (evidence-unavailable, see assert.ts) — so they are neither an asymmetry nor a
1322
+ * "too large" problem. Paths are normalized through `resolve` so `./outputs/x.json` and `outputs/x.json`
1323
+ * join cleanly against the manifest's walk paths. */
1324
+ export function artifactJsonTargetsTruncated(scenario, workRoot, artifacts, readonlyFolderRoots = []) {
1325
+ const isReadonly = (p) => readonlyFolderRoots.some((pre) => p === pre || p.startsWith(pre + "/"));
1307
1326
  const truncatedAbs = new Set();
1308
1327
  for (const a of artifacts)
1309
- if (a.truncated)
1328
+ if (a.truncated && !isReadonly(a.path))
1310
1329
  truncatedAbs.add(resolve(workRoot, a.path));
1311
1330
  if (truncatedAbs.size === 0)
1312
1331
  return [];
@@ -1805,7 +1824,10 @@ async function recordScenarioObject(scenario, opts, extraPolicyDirs = []) {
1805
1824
  // Snapshot under the run's REAL user-visible roots (outputs + resolved folder mount names), persisted
1806
1825
  // below as cassette.userVisibleRoots so replay matches — not the legacy hardcoded `.projects/` prefix.
1807
1826
  const recordRoots = result.userVisibleRoots ?? ["outputs", ".projects"];
1808
- const artifacts = (result.workDir ? buildManifest(result.workDir, opts.maxArtifactBytes, recordRoots) : []).map((a) => {
1827
+ // Read-only connected-folder inputs are captured body-less (path + sha256, no body) — see buildManifest's
1828
+ // `bodyLessPrefixes` doc comment. `recordRoots`/`cassette.userVisibleRoots` stay the FULL set; only the
1829
+ // captured bodies under these prefixes are stripped.
1830
+ const artifacts = (result.workDir ? buildManifest(result.workDir, opts.maxArtifactBytes, recordRoots, result.readonlyFolderRoots ?? []) : []).map((a) => {
1809
1831
  if (a.body === undefined)
1810
1832
  return a;
1811
1833
  if (a.encoding === "base64") {
@@ -1846,7 +1868,7 @@ async function recordScenarioObject(scenario, opts, extraPolicyDirs = []) {
1846
1868
  // replay (no committed body). Surface that record→replay asymmetry NOW, at its cause, instead of letting a
1847
1869
  // green record produce a red replay in CI. Honor --allow-failing (warn, don't block) like the verdict gate.
1848
1870
  if (result.workDir) {
1849
- const truncatedAsserted = artifactJsonTargetsTruncated(scenario, result.workDir, artifacts);
1871
+ const truncatedAsserted = artifactJsonTargetsTruncated(scenario, result.workDir, artifacts, result.readonlyFolderRoots ?? []);
1850
1872
  if (truncatedAsserted.length) {
1851
1873
  const cap = opts.maxArtifactBytes ?? defaultBodyCap();
1852
1874
  const msg = `artifact_json asserts artifact(s) too large to commit (>${cap} B, stored hash-only): ${truncatedAsserted.join(", ")} — ` +
@@ -1868,6 +1890,9 @@ async function recordScenarioObject(scenario, opts, extraPolicyDirs = []) {
1868
1890
  effectiveFidelity: result.effectiveFidelity,
1869
1891
  artifacts,
1870
1892
  userVisibleRoots: recordRoots,
1893
+ // Persist the read-only subset ONLY when non-empty (keeps cassettes without connected folders
1894
+ // clean); replay reads it to distinguish a body-less read-only input from an over-cap artifact.
1895
+ ...(result.readonlyFolderRoots?.length ? { readonlyFolderRoots: result.readonlyFolderRoots } : {}),
1871
1896
  // co-present with userVisibleRoots by construction (see the Cassette field comment) — a cassette
1872
1897
  // carrying preRunPaths never hits replay's legacy-roots fallback.
1873
1898
  preRunPaths: result.preRunPaths,
@@ -1956,7 +1981,7 @@ function warnUncheckableOnDiskKeys(cassette, frozen, onDisk) {
1956
1981
  // Reuse the exported classification constant — this local copy drifted once already (it was
1957
1982
  // missing computer_links_resolve), silently suppressing the on-disk warning for that key.
1958
1983
  const manifestKeys = new Set(MANIFEST_KEYS);
1959
- const gateKeys = new Set(["question_asked", "questions_count_max", "gate_answers_delivered"]);
1984
+ const gateKeys = new Set(["question_asked", "questions_count_max", "gate_answers_delivered", "gate_answer_count_min"]);
1960
1985
  const liveOnlyKeys = new Set([
1961
1986
  "egress_denied",
1962
1987
  "egress_allowed",
@@ -2179,9 +2204,9 @@ export function cmdVerifyCassettes(args) {
2179
2204
  p = parseArgs(args, {
2180
2205
  booleans: ["--skip-privacy", "--skip-staleness", "--quiet", "--verbose"],
2181
2206
  values: ["--output-format"],
2182
- repeated: ["--allow", "--allow-domain", "--allow-email", "--allow-path", "--allow-machine-inventory", "--allow-file"],
2207
+ repeated: ["--allow", "--allow-domain", "--allow-email", "--allow-path", "--allow-machine-inventory", "--allow-patterns-file"],
2183
2208
  enums: { "--output-format": ["text", "json"] },
2184
- noDashValue: ["--allow-file"],
2209
+ noDashValue: ["--allow-patterns-file"],
2185
2210
  aliases: { "-q": "--quiet" },
2186
2211
  });
2187
2212
  }
@@ -2198,10 +2223,13 @@ export function cmdVerifyCassettes(args) {
2198
2223
  }
2199
2224
  const doPrivacy = !skipPrivacy;
2200
2225
  const doStaleness = !skipStaleness;
2201
- // Allow model: each entry is whole-token anchored + class-scoped. A bare `--allow` applies to every
2202
- // class (back-compat); `--allow-domain`/`--allow-email`/`--allow-path`/`--allow-machine-inventory`
2203
- // scope to one class so a domain allow can't bleed into the email tripwire. `--allow-file` loads bare
2204
- // (all-class) patterns from a version-controlled file, one per line, `#` comments and blanks ignored.
2226
+ // Allow model: each entry is whole-token anchored + class-scoped. A bare `--allow <regex>` is a single
2227
+ // PATTERN applied to every class (back-compat); `--allow-domain`/`--allow-email`/`--allow-path`/
2228
+ // `--allow-machine-inventory` scope a pattern to one class so a domain allow can't bleed into the
2229
+ // email tripwire. `--allow-patterns-file <path>` is a different thing: it loads bare (all-class)
2230
+ // patterns from a version-controlled FILE of patterns, one regex per line, `#` comments and blanks
2231
+ // ignored — not "allow this file" (the flag does not accept a path to allow, it accepts a path to a
2232
+ // patterns list).
2205
2233
  const allow = [];
2206
2234
  const addAllow = (src, cls, flag) => {
2207
2235
  try {
@@ -2222,24 +2250,25 @@ export function cmdVerifyCassettes(args) {
2222
2250
  addAllow(src, "path", "--allow-path");
2223
2251
  for (const src of p.repeated["--allow-machine-inventory"] ?? [])
2224
2252
  addAllow(src, "machine-inventory", "--allow-machine-inventory");
2225
- for (const file of p.repeated["--allow-file"] ?? []) {
2253
+ for (const file of p.repeated["--allow-patterns-file"] ?? []) {
2226
2254
  let body;
2227
2255
  try {
2228
2256
  body = readFileSync(file, "utf8");
2229
2257
  }
2230
2258
  catch (e) {
2231
- log(`--allow-file: cannot read ${file}: ${e.message}`);
2259
+ log(`--allow-patterns-file: cannot read ${file}: ${e.message}`);
2232
2260
  return process.exit(2);
2233
2261
  }
2234
2262
  for (const raw of body.split("\n")) {
2235
2263
  const line = raw.trim();
2236
2264
  if (line && !line.startsWith("#"))
2237
- addAllow(line, undefined, `--allow-file (${file})`);
2265
+ addAllow(line, undefined, `--allow-patterns-file (${file})`);
2238
2266
  }
2239
2267
  }
2240
2268
  const target = p.positionals[0];
2241
2269
  if (!target) {
2242
- log("usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-file <path>]... [--output-format json]");
2270
+ log("usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-patterns-file <path>]... [--output-format json]");
2271
+ log(" --allow <regex> is a PATTERN (matched against a finding); --allow-patterns-file <path> is a FILE of patterns, one regex per line — not a path to allow.");
2243
2272
  return process.exit(2);
2244
2273
  }
2245
2274
  if (p.positionals.length > 1) {
@@ -2541,7 +2570,12 @@ export const ALWAYS_CONTENT_KEYS = [
2541
2570
  ...VERDICT_MODIFIER_KEYS,
2542
2571
  ];
2543
2572
  /** Assertion keys evaluated on replay only when `controlOut` (full-fidelity) is present. */
2544
- export const QUESTION_GATE_KEYS = ["question_asked", "questions_count_max", "gate_answers_delivered"];
2573
+ export const QUESTION_GATE_KEYS = [
2574
+ "question_asked",
2575
+ "questions_count_max",
2576
+ "gate_answers_delivered",
2577
+ "gate_answer_count_min",
2578
+ ];
2545
2579
  /** Assertion keys evaluated on replay only when the cassette carries an `artifacts` manifest.
2546
2580
  * `computer_links_resolve` joins this bucket (NOT ALWAYS_CONTENT_KEYS): resolving a NON-empty link set
2547
2581
  * needs either a live filesystem (not available on replay) or the cassette's `artifacts` manifest — the
@@ -2746,6 +2780,9 @@ export async function replayCassette(cassette, hooks = [], opts = {}) {
2746
2780
  result: rec.result,
2747
2781
  workRoot: replayWorkRoot,
2748
2782
  userVisiblePrefixes: replayPrefixes,
2783
+ // Persisted on the cassette (v7 optional metadata) so replay tells a body-less read-only input
2784
+ // apart from an over-cap artifact and gives the precise artifact_json remediation.
2785
+ readonlyFolderRoots: cassette.readonlyFolderRoots ?? [],
2749
2786
  preRunPaths: cassette.preRunPaths,
2750
2787
  outputsDeletes: [],
2751
2788
  questions: rec.questions,
@@ -15,7 +15,7 @@ import { writeRunningStatus, startStatusTicker, registerRunForCrashSafety, statu
15
15
  // the cycle is intrinsic — kept runtime-only rather than refactored.
16
16
  import { buildFingerprint } from "./cassette.js";
17
17
  import { loadBaseline } from "../baseline.js";
18
- import { loadSession, resolveSessionPaths, buildLaunchPlan, userVisibleRootsFromPlan } from "../session.js";
18
+ import { loadSession, resolveSessionPaths, buildLaunchPlan, userVisibleRootsFromPlan, readonlyFolderRootsFromPlan } from "../session.js";
19
19
  import { spawnProtocol } from "../runtime/protocol.js";
20
20
  import { spawnContainer } from "../runtime/container.js";
21
21
  import { spawnHostLoop } from "../runtime/hostloop.js";
@@ -588,6 +588,13 @@ export async function executeScenario(scenario, opts = {}) {
588
588
  // (verify reads result.json; replay reads the cassette) match this without rebuilding a LaunchPlan.
589
589
  // Shared with the pre-run baseline walk (userVisibleRootsFromPlan) — pre and post MUST agree.
590
590
  const userVisibleRoots = userVisibleRootsFromPlan(plan);
591
+ // Read-only (`mode: "r"`) connected-folder roots — inputs, not deliverables. Persisted so the cassette
592
+ // recorder strips their captured BODIES (fidelity/no-bloat) and `RunResult.artifacts` excludes them
593
+ // outright (an input is not a `file_exists` target). Does NOT change `userVisibleRoots` above.
594
+ const readonlyFolderRoots = readonlyFolderRootsFromPlan(plan);
595
+ // artifacts (the ENV-MANIFEST) excludes read-only inputs — a deliverable is something the agent wrote,
596
+ // not a pre-existing input it only read.
597
+ const captureRoots = userVisibleRoots.filter((r) => !readonlyFolderRoots.includes(r));
591
598
  // Read the pre-run baseline ONCE: the evaluate ctx and the persisted RunResult must see the same
592
599
  // value — two reads could disagree if the file were touched mid-run.
593
600
  const preRunPaths = readPreRunManifest(outDir);
@@ -605,6 +612,7 @@ export async function executeScenario(scenario, opts = {}) {
605
612
  outDir,
606
613
  workRoot,
607
614
  userVisibleRoots,
615
+ readonlyFolderRoots,
608
616
  effectiveFidelity,
609
617
  egress,
610
618
  durationMs: Date.now() - startedAt,
@@ -633,6 +641,9 @@ export async function executeScenario(scenario, opts = {}) {
633
641
  result: record.result,
634
642
  workRoot,
635
643
  userVisiblePrefixes: userVisibleRoots,
644
+ // Read-only folder inputs are captured body-less; artifact_json must reach the same
645
+ // evidence-unavailable verdict here as on replay (see AssertContext.readonlyFolderRoots).
646
+ readonlyFolderRoots,
636
647
  preRunPaths,
637
648
  outputsDeletes: scan.outputsDeletes,
638
649
  questions: record.questions,
@@ -779,7 +790,8 @@ export async function executeScenario(scenario, opts = {}) {
779
790
  workDir: workRoot,
780
791
  outputsDir: join(workRoot, "outputs"),
781
792
  userVisibleRoots,
782
- artifacts: collectArtifacts(workRoot, userVisibleRoots), // ENV-MANIFEST: observed user-visible files
793
+ readonlyFolderRoots,
794
+ artifacts: collectArtifacts(workRoot, captureRoots), // ENV-MANIFEST: observed user-visible files (excl. read-only inputs)
783
795
  // The pre-spawn baseline no_unexpected_files diffs against (same single read the evaluate ctx got).
784
796
  // undefined = the run didn't capture (key not asserted, microvm, pre-seam) — the assertion then
785
797
  // fails evidence-unavailable, loud.
@@ -946,7 +958,8 @@ export function buildPartialResult(args) {
946
958
  workDir: args.workRoot,
947
959
  outputsDir: join(args.workRoot, "outputs"),
948
960
  userVisibleRoots: args.userVisibleRoots,
949
- artifacts: collectArtifacts(args.workRoot, args.userVisibleRoots),
961
+ readonlyFolderRoots: args.readonlyFolderRoots,
962
+ artifacts: collectArtifacts(args.workRoot, args.userVisibleRoots.filter((r) => !args.readonlyFolderRoots.includes(r))),
950
963
  preRunPaths: readPreRunManifest(args.outDir),
951
964
  effectiveFidelity: args.effectiveFidelity,
952
965
  ...(() => {
@@ -261,6 +261,11 @@ export function buildTrace(file, opts = {}) {
261
261
  * differing keys: the gate's `decision` (events.jsonl) carries both the UUID `request_id` AND the `toolu_`
262
262
  * `toolUseId`; the injected answer lives in `control-out.jsonl` keyed by `request_id`; the delivered result
263
263
  * is a `tool_result` keyed by `toolUseId`.
264
+ *
265
+ * Each row is one gate (one AskUserQuestion tool call), which may bundle several sub-questions —
266
+ * `subQuestionCount` carries that count so a reader can reconcile this view against
267
+ * `questions_count_max`, which counts sub-questions, not gates/tool-calls (see `formatGateTrace`'s
268
+ * footer total and `docs/scenario.md`/`docs/cassette.md`'s `questions_count_max` row).
264
269
  */
265
270
  export function buildGateTrace(file) {
266
271
  const events = eventsOf(file);
@@ -297,6 +302,7 @@ export function buildGateTrace(file) {
297
302
  const tr = req.toolUseId ? results.get(req.toolUseId) : undefined;
298
303
  rows.push({
299
304
  question: req.questions.map((q) => q.question ?? q.header ?? "").join(" / "),
305
+ subQuestionCount: req.questions.length,
300
306
  injectedAnswer: answers.get(req.id),
301
307
  delivered: tr ? (tr.isError ? "error" : "ok") : "unobserved",
302
308
  ...(tr?.isError ? { error: tr.text.split("\n")[0].slice(0, 160) } : {}),
@@ -332,12 +338,16 @@ export function formatGateTrace(rows) {
332
338
  if (!rows.length)
333
339
  return "(no AskUserQuestion gates in this run)";
334
340
  const mark = { ok: "✓", error: "✗", unobserved: "?" };
335
- return rows
336
- .map((r) => {
341
+ const lines = rows.map((r) => {
337
342
  const prov = r.answeredBy ? `\n by: ${labelSource(r.answeredBy)}${r.model ? ` (${r.model})` : ""}` : "";
338
- return `${mark[r.delivered]} gate "${r.question}"\n answered: ${r.injectedAnswer ?? "(none)"}\n delivered: ${r.delivered}${r.error ? ` — ${r.error}` : ""}${prov}`;
339
- })
340
- .join("\n");
343
+ // sub-question count is shown only when the gate bundled more than one — reconciles this row
344
+ // against questions_count_max, which counts sub-questions, not gates.
345
+ const subCount = r.subQuestionCount > 1 ? ` (${r.subQuestionCount} sub-questions)` : "";
346
+ return `${mark[r.delivered]} gate "${r.question}"${subCount}\n answered: ${r.injectedAnswer ?? "(none)"}\n delivered: ${r.delivered}${r.error ? ` — ${r.error}` : ""}${prov}`;
347
+ });
348
+ const totalSubQuestions = rows.reduce((sum, r) => sum + r.subQuestionCount, 0);
349
+ lines.push(`\n${rows.length} gate(s), ${totalSubQuestions} sub-question(s) total — questions_count_max counts sub-questions (assert with \`questions_count_max: ${totalSubQuestions}\`)`);
350
+ return lines.join("\n");
341
351
  }
342
352
  /**
343
353
  * `trace --dispatches` — the sub-agent dispatch tree, so an author can read off the REAL total
package/dist/session.js CHANGED
@@ -116,6 +116,14 @@ export const SessionConfig = z.strictObject({
116
116
  export function userVisibleRootsFromPlan(plan) {
117
117
  return ["outputs", ...plan.mounts.filter((m) => m.kind === "folder").map((m) => m.mountPath)];
118
118
  }
119
+ /** The mount prefixes of read-only (`mode: "r"`) connected folders — inputs, not deliverables. Used to
120
+ * strip captured BODIES from the cassette manifest (fidelity/no-bloat) and to exclude them from
121
+ * `RunResult.artifacts` (an input is not something `scaffold`/`file_exists` should treat as output).
122
+ * Does NOT change `userVisibleRootsFromPlan` — `no_unexpected_files` and `computer_links_resolve`
123
+ * still enumerate these roots; only their captured content changes shape. */
124
+ export function readonlyFolderRootsFromPlan(plan) {
125
+ return plan.mounts.filter((m) => m.kind === "folder" && m.mode === "r").map((m) => m.mountPath);
126
+ }
119
127
  /**
120
128
  * Read a plugin's declared version from its `.claude-plugin/plugin.json`.
121
129
  *
package/dist/types.js CHANGED
@@ -241,7 +241,13 @@ export const Assertion = z.object({
241
241
  gate_answers_delivered: z
242
242
  .boolean()
243
243
  .optional()
244
- .describe("every answered AskUserQuestion gate's tool_result was non-error (the answer reached the model)"),
244
+ .describe("every answered AskUserQuestion gate's tool_result was non-error (the answer reached the model); zero gates fired passes vacuously — pair with gate_answer_count_min to also require a gate"),
245
+ gate_answer_count_min: z
246
+ .number()
247
+ .int()
248
+ .nonnegative()
249
+ .optional()
250
+ .describe("at least N AskUserQuestion gates fired AND were delivered non-error (presence companion to gate_answers_delivered)"),
245
251
  result: z.enum(["success", "error"]).optional().describe("the run's final result was success | error"),
246
252
  allow_permissive_auto_allow: z
247
253
  .boolean()
package/docs/cassette.md CHANGED
@@ -53,9 +53,11 @@ re-record). `expect_denied`/filesystem/egress keys are sourced but stay live-onl
53
53
  "events": [ /* JSON lines from events.jsonl (child→driver stdout) */ ],
54
54
  "controlOut": [ /* JSON lines from control-out.jsonl (driver→child control_responses) */ ],
55
55
  "userVisibleRoots": ["outputs", "myproject"], // visible roots = outputs + each connected folder's mount name (its basename; `.projects` is the pre-1.14271.0 legacy fallback)
56
+ "readonlyFolderRoots": ["carta-folder"], // the mode:r subset of userVisibleRoots (body-less); lets replay give artifact_json the precise remedy. Optional; absent when no read-only folder
56
57
  "artifacts": [ // snapshot of outputs/ + connected folders (optional)
57
58
  { "path": "outputs/x.json", "bytes": 24, "sha256": "…", "body": "{…}" }, // body inlined ≤ 64 KiB
58
- { "path": "outputs/big.bin", "bytes": 9e6, "sha256": "…", "truncated": true } // oversized → hash-only
59
+ { "path": "outputs/big.bin", "bytes": 9e6, "sha256": "…", "truncated": true }, // oversized → hash-only
60
+ { "path": "carta-folder/input.xlsx", "bytes": 4096, "sha256": "…", "truncated": true } // mode:r connected-folder INPUT → body-less (see below), regardless of size
59
61
  ],
60
62
  "fingerprint": { "baseline": "1.15962.1", "skillHash": "…", "mode": "git", "contentSig": "…", "fileSigs": [["skills/x/SKILL.md", "…"]], "skillSources": ["…"] }, // staleness tripwire (v5: fileSigs only; v6: mode + git default; v7: NUL-delimited hash entries)
61
63
  "authoring": { "nonDeterministic": true, "channel": "decider-dir" } // present ONLY when a live decider answered ≥1 gate (see §Answering gates during recording); re-record may drift, replay is still deterministic
@@ -227,17 +229,18 @@ the rules and CI-placement rationale (why each category behaves this way), see
227
229
  | `tool_calls_max` | total top-level tool calls (sub-agent tools excluded) ≤ N — meaningfully replay-checkable; the re-drive recomputes `toolCounts` deterministically |
228
230
  | `max_turns` | SDK-reported (or fallback-counted) turn count ≤ N — replay-checkable, recounted deterministically same as `tool_calls_max` |
229
231
  | `question_asked` | agent asked an AskUserQuestion matching the regex |
230
- | `questions_count_max` | at most N questions asked |
231
- | `gate_answers_delivered` | answered gates' answers reached the model |
232
+ | `questions_count_max` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
233
+ | `gate_answers_delivered` | answered gates' answers reached the model — **zero gates fired passes vacuously** (gate firing is model-dependent); pair with `gate_answer_count_min` to also require a gate |
234
+ | `gate_answer_count_min` | at least N AskUserQuestion gates fired AND were delivered non-error — the presence companion to `gate_answers_delivered`'s vacuous-pass |
232
235
  | `result` | run ended with `success` or `error` |
233
236
  | `allow_permissive_auto_allow` | verdict modifier — kept on replay → no-op pass (the live signal it suppresses is zeroed) |
234
237
  | `allow_missing_capability` | verdict modifier — kept on replay → no-op pass (the live signal it suppresses is zeroed) |
235
238
  | `allow_l0_plugin_divergence` | verdict modifier — kept on replay → no-op pass (the live signal it suppresses is zeroed) |
236
239
  | `allow_stall` | verdict modifier — kept on replay → no-op pass (suppresses the `stalled` default-fail; the stall is re-derived on the replay re-drive) |
237
240
 
238
- **`question_asked`, `questions_count_max`, `gate_answers_delivered` require `controlOut`** (full-fidelity
239
- replay). On an old cassette without `controlOut` these three keys are excluded from evaluation — not
240
- vacuously passed — and a loud warning fires (see §Backward compatibility).
241
+ **`question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min` require
242
+ `controlOut`** (full-fidelity replay). On an old cassette without `controlOut` these keys are excluded
243
+ from evaluation — not vacuously passed — and a loud warning fires (see §Backward compatibility).
241
244
 
242
245
  `file_exists`, `user_visible_artifact`, `artifact_json`, `computer_links_resolve`, and `no_unexpected_files` are **not** in the
243
246
  table above — see the next subsection; they're replay-checkable only when the cassette carries an
@@ -258,7 +261,19 @@ The inline cap is 64 KiB; raise it with `record --max-artifact-bytes <n>` (or
258
261
  fails fast if an `artifact_json` targets an artifact it had to truncate (that would pass at record but fail
259
262
  at replay). `computer_links_resolve` resolves every `computer://` link in the transcript against the same
260
263
  manifest, with host-shaped links normalized via the recorded session folders. Without a manifest (older
261
- cassettes), these are skipped. A green replay re-confirms *record-time* artifacts, **not** that the current
264
+ cassettes), these are skipped.
265
+
266
+ **Read-only (`mode: r`) connected-folder contents are captured body-less.** A folder mounted `mode: r` in
267
+ `session:` holds pre-existing INPUTS the agent only reads, not deliverables — so `record` snapshots them
268
+ the same way it snapshots an over-cap file: path + `bytes` + `sha256`, `truncated: true`, **no `body`**,
269
+ regardless of size. The entry still lands in `artifacts[]` (unlike a fully-excluded path) so
270
+ `materializeManifest` writes a 0-byte placeholder at replay and `computer_links_resolve`/`file_exists`
271
+ resolve identically live and on replay; only `artifact_json` (which needs the inlined body) can't target
272
+ one. Two side benefits: no cassette bloat from a large input file, and no `binary` privacy finding (the
273
+ scanner only flags a *committed* binary body) — so a `mode: r` input never needs `--allow`. A `mode: rw`/`rwd`
274
+ folder's contents are captured with a full body exactly as `outputs/` is.
275
+
276
+ A green replay re-confirms *record-time* artifacts, **not** that the current
262
277
  skill still produces them — `replay --strict` fails the run when the `fingerprint` shows ANY skill/baseline
263
278
  drift, or `replay --fail-on-skill-drift` fails only on skill-source drift (leaving baseline drift a warning).
264
279
  Either way, every replay result also reports the drift in `staleness[]` (class-tagged) for a JSON gate to read.
@@ -303,12 +318,16 @@ On replay, the replay decider (built by `buildReplayDecider()`) indexes `control
303
318
  the decision pipeline instead of consulting a live decider or asking the user. This makes the full
304
319
  `Run.handleDecision` path execute on replay, which populates `rec.questions`, `rec.gateAnswers`, and
305
320
  `rec.gateDeliveries` — exactly as in a live run. Consequence: `question_asked`, `questions_count_max`,
306
- and `gate_answers_delivered` are now genuinely evaluated, not silently skipped or vacuously passed.
321
+ `gate_answers_delivered`, and `gate_answer_count_min` are now genuinely evaluated, not silently skipped
322
+ or vacuously passed.
307
323
 
308
324
  `gate_answers_delivered` accepts a boolean: `: true` asserts the answered gates' answers reached the
309
- model; `: false` is the **inverse** — it asserts a *confirmed delivery failure* (at least one gate whose
310
- `delivered === false`), for scenarios that deliberately exercise a non-delivery path. Unobserved delivery
311
- (`delivered: null`) satisfies neither — absence of evidence is a failure, not a pass.
325
+ model, **passing vacuously when zero gates fired** (gate firing is model-dependent); `: false` is the
326
+ **inverse** — it asserts a *confirmed delivery failure* (at least one gate whose `delivered === false`),
327
+ for scenarios that deliberately exercise a non-delivery path. Unobserved delivery (`delivered: null`)
328
+ satisfies neither — absence of evidence is a failure, not a pass. Pair `gate_answers_delivered: true`
329
+ with `gate_answer_count_min: <N>` when a gate firing at all is part of what you're testing —
330
+ `gate_answer_count_min` fails if fewer than N gates fired AND were delivered non-error.
312
331
 
313
332
  ### The O7 guard — `replay_protocol_fidelity`
314
333
 
@@ -337,8 +356,8 @@ regressing to the prior false-green behavior:
337
356
  ::warning:: [replay] cassette has no controlOut (pre-full-fidelity) — question/gate assertions
338
357
  are NOT checked; re-record to enable them
339
358
  ```
340
- 2. **`question_asked`, `questions_count_max`, `gate_answers_delivered` are excluded** from the evaluated
341
- assertion set for that run — not vacuously passed, absent.
359
+ 2. **`question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min` are
360
+ excluded** from the evaluated assertion set for that run — not vacuously passed, absent.
342
361
  3. All other content assertions (transcript, tool, subagent, result) evaluate normally.
343
362
 
344
363
  This preserves "skipped ≠ false-green." Re-record with a current harness to get the full-fidelity path.
@@ -477,10 +496,12 @@ counts) — committed PII surface. Two layers, distinct from secret-scrub (which
477
496
  live-enumerated app/process inventory sentinel (e.g. a computer-use tool schema's "Available applications on
478
497
  this machine: …") is never legitimate catalog boilerplate either; none of the three share the ambiguity that
479
498
  gets `currency`/`domain` excluded there. `--allow <regex>` suppresses synthetic / public reference names
480
- (e.g. `NVCA`, `Cooley GO`, `Acme`); each allow must match the **whole** finding token (so a bare-domain allow
481
- no longer silently clears an email whose domain it matches), and `--allow-domain` / `--allow-email` /
482
- `--allow-path` / `--allow-machine-inventory` scope an allow to a single finding class, while `--allow-file
483
- <path>` loads allows from a version-controlled file (one regex per line, `#` comments). Multi-word proper
499
+ (e.g. `NVCA`, `Cooley GO`, `Acme`) — each `--allow` value is a **pattern**, matched against a finding, not a
500
+ path to allow; each allow must match the **whole** finding token (so a bare-domain allow no longer silently
501
+ clears an email whose domain it matches), and `--allow-domain` / `--allow-email` / `--allow-path` /
502
+ `--allow-machine-inventory` scope an allow to a single finding class, while `--allow-patterns-file <path>` is a
503
+ different thing — it loads allows from a version-controlled **file of patterns** (one regex per line, `#`
504
+ comments), not a path to allow directly. Multi-word proper
484
505
  names are **not** a default class (too noisy). `verify-cassettes` also runs the **staleness**
485
506
  check (both checks run by default; scope to one with `--skip-privacy` or `--skip-staleness`): a drifted
486
507
  `skillHash` (you edited the skill but didn't re-record) fails the gate.
package/docs/scenario.md CHANGED
@@ -277,9 +277,10 @@ if *every* key passes (don't rely on the first; keep one concern per item unless
277
277
  | `tool_calls_max: <N>` | total top-level tool calls (sum of `toolCounts`, sub-agent tools excluded) ≤ N — unlike the cost/token keys, this **is** meaningfully replay-checkable (the re-drive recomputes `toolCounts` deterministically from the recorded events) |
278
278
  | `max_turns: <N>` | the SDK-reported (or fallback-counted) turn count ≤ N — replay-checkable (the re-drive recounts turns deterministically, same as `tool_calls_max`) |
279
279
  | `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
280
- | `questions_count_max: <N>` | the agent asked at most N questions |
281
- | `gate_answers_delivered: true` | every answered AskUserQuestion gate's answer actually reached the model — requires a positive, observed `tool_result` (an **unobserved** delivery fails too, not only an errored one — no silent false-green) |
280
+ | `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1 (this is a decision-load budget, not a per-tool-call count); `trace --view questions`'s footer total is computed the same way, so it always matches what this key compares against |
281
+ | `gate_answers_delivered: true` | every answered AskUserQuestion gate's answer actually reached the model — requires a positive, observed `tool_result` (an **unobserved** delivery fails too, not only an errored one — no silent false-green); **zero gates fired passes vacuously** (gate firing is model-dependent) — pair with `gate_answer_count_min` to also require a gate |
282
282
  | `gate_answers_delivered: false` | asserts that at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — useful for negative-path tests of delivery failures |
283
+ | `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — the presence companion to `gate_answers_delivered`'s vacuous-pass (mirrors `transcript_contains` pairing with `computer_links_resolve`) |
283
284
  | `allow_permissive_auto_allow: true` | verdict modifier — suppresses the default-fail when the run recorded a cowork-parity permissive auto-allow; use this for tests that **deliberately** assert Cowork's permissive behavior rather than strict scripted coverage |
284
285
  | `allow_missing_capability: true` | verdict modifier (**live tiers only**) — suppresses the default-fail when the lean/`core` agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables); assert only when the skill's fallback is genuinely equivalent, else rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check below. |
285
286
  | `allow_l0_plugin_divergence: true` | verdict modifier — opts into L0/protocol plugin divergence, suppressing the plugin-fidelity default-fail |
@@ -380,10 +381,10 @@ and re-evaluates the **content** assertions. The authoritative list of content k
380
381
  `allow_l0_plugin_divergence` / `allow_stall` (kept on replay as no-op passes). `max_cost_usd`/`max_tokens`
381
382
  assert the *frozen recording's* spend on replay, not fresh spend — see their table entries above.
382
383
 
383
- **`question_asked`, `questions_count_max`, and `gate_answers_delivered`** are also content
384
- assertions, but they require the cassette to carry `controlOut` (full-fidelity replay). When
384
+ **`question_asked`, `questions_count_max`, `gate_answers_delivered`, and `gate_answer_count_min`**
385
+ are also content assertions, but they require the cassette to carry `controlOut` (full-fidelity replay). When
385
386
  `controlOut` is present, the decision pipeline runs on replay and populates `rec.questions` /
386
- `rec.gateDeliveries` — so these three keys are genuinely evaluated.
387
+ `rec.gateDeliveries` — so these keys are genuinely evaluated.
387
388
  When `controlOut` is absent (old cassette), a **loud warning** fires and these keys are **excluded**
388
389
  from evaluation (not vacuously passed). Re-record with a current harness to enable them.
389
390
 
@@ -396,6 +397,15 @@ entry still satisfies `file_exists` but not `artifact_json`. `computer_links_res
396
397
  normalizes to a mount-relative path first (via the recorded connected-folder prefixes + the outputs/uploads
397
398
  mounts), since replay has no live filesystem to probe directly (that direct check only happens on a live
398
399
  `run`/`verify-run`). Without a manifest (older cassettes), all four are **skipped** (loud).
400
+
401
+ A `mode: r` connected folder (see [session.md](./session.md)) holds pre-existing INPUTS, not deliverables —
402
+ `record` captures its contents **body-less** (path + hash, `truncated: true`, no `body`): `file_exists` and
403
+ `computer_links_resolve` still pass against it (the placeholder materializes on replay), while `artifact_json`
404
+ against it reports a clear evidence-unavailable identically on live, verify-run, and replay (so a cassette
405
+ can't record green and replay red). This keeps a read-only input out of the cassette's committed content
406
+ (no bloat, no `binary` privacy finding) while `no_unexpected_files`/`computer_links_resolve` keep enumerating
407
+ the folder as a user-visible root. A `mode: rw`/`rwd` folder's contents are captured with a full body, same
408
+ as `outputs/`.
399
409
  A green `replay` re-confirms *record-time* artifacts, **not** that the current skill still produces them —
400
410
  that needs a live `run` (the cassette's staleness fingerprint warns when the skill/baseline drifted; `replay
401
411
  --strict` fails on any drift, `--fail-on-skill-drift` on skill-source drift only, and every result reports it
@@ -13,7 +13,7 @@ replay path end-to-end, including:
13
13
 
14
14
  Run it with:
15
15
 
16
- > Assumes the `cowork-harness` CLI is available — from a source checkout run `npm ci && npm run build && npm link` first, or `npm i -g cowork-harness@>=0.24.0`. (`replay` itself needs nothing else — no token, no Docker.)
16
+ > Assumes the `cowork-harness` CLI is available — from a source checkout run `npm ci && npm run build && npm link` first, or `npm i -g cowork-harness@>=0.25.0`. (`replay` itself needs nothing else — no token, no Docker.)
17
17
 
18
18
  ```sh
19
19
  cowork-harness replay examples/replays/example-pdf-skill.cassette.json
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "cowork-harness",
3
- "version": "0.24.0",
3
+ "version": "0.25.0",
4
4
  "description": "Scriptable, CI-friendly harness for Claude Cowork's runtime contract for testing skills across scenarios — same agent, mounts, egress allowlist, permission protocol, and sandbox limitations.",
5
5
  "license": "MIT",
6
6
  "type": "module",
@@ -54,6 +54,13 @@
54
54
  },
55
55
  "description": "v4+: the user-visible mount roots captured at record time (`outputs` + each connected folder's resolved mount name). Replay reads THIS instead of a hardcoded prefix. ABSENT on pre-v4 cassettes → replay falls back to the legacy prefix."
56
56
  },
57
+ "readonlyFolderRoots": {
58
+ "type": "array",
59
+ "items": {
60
+ "type": "string"
61
+ },
62
+ "description": "The subset of `userVisibleRoots` that are read-only (`mode: r`) connected folders — captured body-less (path + hash, no body). Persisted so replay distinguishes a body-less read-only input from an over-cap artifact and gives the precise `artifact_json` remediation. Optional metadata (no cassetteVersion bump); ABSENT/empty ⇒ no read-only folder in the run. A subset of `userVisibleRoots` (no new privacy strings)."
63
+ },
57
64
  "preRunPaths": {
58
65
  "type": "array",
59
66
  "items": {
@@ -259,9 +259,15 @@
259
259
  "maximum": 9007199254740991
260
260
  },
261
261
  "gate_answers_delivered": {
262
- "description": "every answered AskUserQuestion gate's tool_result was non-error (the answer reached the model)",
262
+ "description": "every answered AskUserQuestion gate's tool_result was non-error (the answer reached the model); zero gates fired passes vacuously — pair with gate_answer_count_min to also require a gate",
263
263
  "type": "boolean"
264
264
  },
265
+ "gate_answer_count_min": {
266
+ "description": "at least N AskUserQuestion gates fired AND were delivered non-error (presence companion to gate_answers_delivered)",
267
+ "type": "integer",
268
+ "minimum": 0,
269
+ "maximum": 9007199254740991
270
+ },
265
271
  "result": {
266
272
  "description": "the run's final result was success | error",
267
273
  "type": "string",