cowork-harness 0.23.0 → 0.25.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. package/.claude/skills/cowork-harness/SKILL.md +43 -11
  2. package/.claude/skills/cowork-harness/references/ci-recipe.md +25 -5
  3. package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
  4. package/.claude/skills/cowork-harness/references/scenario-schema.md +35 -14
  5. package/.claude/skills/cowork-harness/references/task-recipes.md +110 -0
  6. package/.claude/skills/cowork-harness/scripts/assertion-keys.json +2 -0
  7. package/.claude/skills/cowork-harness/scripts/scenario.py +79 -5
  8. package/.cowork-redact.json +11 -0
  9. package/CHANGELOG.md +199 -0
  10. package/CONTRIBUTING.md +1 -0
  11. package/README.md +6 -5
  12. package/SPEC.md +56 -24
  13. package/baselines/desktop-1.18286.0.json +13 -3
  14. package/baselines/prompts/desktop-1.18286.0/system-prompt-append.md +1 -2
  15. package/dist/assert.js +56 -5
  16. package/dist/cli.js +80 -12
  17. package/dist/glob.js +43 -0
  18. package/dist/run/artifacts.js +109 -0
  19. package/dist/run/cassette.js +300 -44
  20. package/dist/run/execute.js +39 -86
  21. package/dist/run/pre-run-manifest.js +61 -0
  22. package/dist/run/trace-view.js +15 -5
  23. package/dist/runtime/container.js +3 -0
  24. package/dist/runtime/hostloop.js +12 -2
  25. package/dist/runtime/lima.js +3 -3
  26. package/dist/runtime/microvm.js +5 -0
  27. package/dist/runtime/protocol.js +3 -0
  28. package/dist/scan.js +1 -1
  29. package/dist/session.js +14 -0
  30. package/dist/sync/cowork-sync.js +6 -1
  31. package/dist/types.js +12 -2
  32. package/docker/Dockerfile.agent +8 -4
  33. package/docs/cassette.md +55 -22
  34. package/docs/fidelity-gaps.md +62 -10
  35. package/docs/scenario.md +17 -6
  36. package/examples/replays/README.md +1 -1
  37. package/package.json +2 -1
  38. package/python/test_scenario_lint.py +71 -0
  39. package/schema/cassette.v7.json +25 -0
  40. package/schema/run-result.json +32 -0
  41. package/schema/scenario.schema.json +15 -1
  42. package/schema/verify-cassettes.json +70 -0
  43. package/scripts/boot-rootfs-vz.ts +4 -4
  44. package/scripts/build-rootfs-image.ts +1 -1
  45. package/scripts/capture-rootfs-manifest.ts +1 -1
@@ -3,8 +3,8 @@ name: cowork-harness
3
3
  description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
4
4
  metadata:
5
5
  author: cowork-harness
6
- version: 0.23.0
7
- tracks-harness: cowork-harness 0.23.0 (baseline desktop-1.18286.0)
6
+ version: 0.25.0
7
+ tracks-harness: cowork-harness 0.25.0 (baseline desktop-1.18286.0)
8
8
  ---
9
9
 
10
10
  # cowork-harness
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
22
22
  allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
23
23
  the highest-value part. Read it.
24
24
 
25
- > **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.23.0` (baseline
25
+ > **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.25.0` (baseline
26
26
  > `desktop-1.18286.0`). If your checkout is newer, prefer the live `--help`, `SPEC.md`, and
27
27
  > `docs/*.md` over this snapshot, and re-run the bundled linter.
28
28
 
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
38
38
  Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
39
39
 
40
40
  - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
41
- - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.23.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.23.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.23.0"`. **Pin `@>=0.23.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.23.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
41
+ - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.25.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.25.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.25.0"`. **Pin `@>=0.25.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.25.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
42
42
  - **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
43
43
  - **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
44
44
  - **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
@@ -288,10 +288,14 @@ cowork-harness lint scenarios/*.yaml
288
288
  ```
289
289
 
290
290
  `lint` flags: filesystem/egress-only assertions on a `replay` gate (silent no-op), bad regex
291
- quoting, an egress assert on `protocol` fidelity, a `controlOut`-gated key on a non-`controlOut`
292
- replay, mixed-class assertion items, and hallucinated schema (`assertions:` vs `assert:`, unknown
293
- keys). Exit code is non-zero on errors (CI-friendly). `scaffold` auto-upgrades the tier if you ask
294
- for egress on `protocol`, so it never emits a scenario `lint` would reject.
291
+ quoting, an egress assert on `protocol` fidelity, `transcript_no_host_path` on `hostloop`/`protocol`
292
+ (ERROR — fails by design at those tiers; WARN on `fidelity: cowork`, whose tier resolves per the
293
+ baseline's host-loop gate), non-empty `requires_capabilities` on `protocol` without
294
+ `allow_missing_capability` (ERROR — the capability probe can't run there, so the run hard-fails as
295
+ unverifiable), a `controlOut`-gated key on a non-`controlOut` replay, mixed-class assertion items,
296
+ and hallucinated schema (`assertions:` vs `assert:`, unknown keys). Exit code is non-zero on errors
297
+ (CI-friendly). `scaffold` auto-upgrades the tier if you ask for egress on `protocol`, so it never
298
+ emits a scenario `lint` would reject.
295
299
 
296
300
  CI placement: a **token-free `replay` PR gate** (content/structure only) + a **nightly live `run`**
297
301
  (filesystem/egress). Fastest setup: `uses: yaniv-golan/cowork-harness@v1` (a packaged GitHub Action with a
@@ -362,9 +366,12 @@ are the ones that bite hardest.
362
366
  sub-agent used other tools. And `subagent_dispatched` matches by dispatch **description** too —
363
367
  skills often dispatch with no `subagent_type` (`agentType:"unknown"`), so match the description.
364
368
 
365
- 6. **`dispatch_count_max` asserts but does NOT enforce.** The harness records the count; it does not
366
- reproduce Cowork's skip-on-cap (`{perTask:1, global:3}`, deferred). Passing means "happened to
367
- dispatch ≤N this run," not "the harness capped it." Don't read enforcement into the assert.
369
+ 6. **`dispatch_count_max` is an author-chosen budget, not a production cap.** It's a post-hoc count
370
+ assertion: passing means "happened to dispatch ≤N this run," nothing more. Cowork imposes **no**
371
+ in-conversation `Task`-dispatch cap — gate `1648655587`'s `{perTask:1, global:3}` governs the
372
+ separate scheduled/cron-task session scheduler, not the `Task` tool (binary-verified; see SPEC
373
+ §10). So there is no "skip-on-cap" for the harness to reproduce; use this key only to catch a
374
+ fan-out you don't want.
368
375
 
369
376
  7. **`protocol` is rejected (not silently passed) if the scenario asserts egress** — boundary
370
377
  assertions need a sandboxed tier (`container`+). Good: this one fails loud by design.
@@ -451,11 +458,36 @@ are the ones that bite hardest.
451
458
  and re-assert is undetected — the notice flags this; re-record if the session changed. `verify-run` reads
452
459
  on-disk `assert:` against a kept *run dir*; `replay --assert-from` is the equivalent for a *cassette*.
453
460
 
461
+ 18. **`questions_count_max` counts sub-questions, not gates.** One `AskUserQuestion` tool call can
462
+ bundle several sub-questions into a single gate; the assertion counts each sub-question, so a
463
+ 3-sub-question bundle counts as 3, not 1. `trace --view questions` shows the same per-gate
464
+ sub-question count and a matching footer total — read that off instead of the tool-call count when
465
+ sizing the budget.
466
+
467
+ 19. **`gate_answers_delivered` passes vacuously when no gate fires — use `gate_answer_count_min: 1` to
468
+ also require a gate.** Whether a gate fires is model-dependent, so `gate_answers_delivered: true`
469
+ alone can't catch "the gate never fired at all"; pair it with `gate_answer_count_min` when
470
+ presence matters, not just delivery.
471
+
472
+ 20. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
473
+ read-only folder's files as path + hash only (`truncated: true`, no `body`) — it's an input the agent
474
+ read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass against it on replay
475
+ (the hash-only entry still materializes a placeholder); `artifact_json` reports a clear
476
+ evidence-unavailable on every lane (live/verify-run/replay agree — no green-record/red-replay). This is
477
+ also why a `mode: r` input never trips the `binary` privacy finding or needs `--allow` — only a
478
+ *committed* body is scanned. `scaffold` won't emit `file_exists` for one either (it's not in
479
+ `RunResult.artifacts`). A `mode: rw`/`rwd` folder's contents are captured with a full body, same as
480
+ `outputs/`.
481
+
454
482
  For the complete gotcha list, the assertion catalog, the YAML schema, the fidelity/answer tables,
455
483
  and the CI recipe, read the files in `references/`.
456
484
 
457
485
  ## References
458
486
 
487
+ - `references/task-recipes.md` — end-to-end recipes for the four jobs fleet owners actually hit:
488
+ evolve a cassette's `assert:` (usually no re-record), audit a fleet for tier drift, set up
489
+ redaction before the first hostloop/protocol record, derive budget assertions without a
490
+ two-pass record. Start here when the question is "how do I do X", not "what does flag Y mean".
459
491
  - `references/scenario-schema.md` — scenario/session YAML schema, full assertion catalog (with each
460
492
  key's replay class), the web_fetch model, and the complete gotcha list.
461
493
  - `references/fidelity-and-answers.md` — fidelity tiers, answer paths, the determinism contract.
@@ -1,6 +1,6 @@
1
1
  # CI recipe — replay vs live lanes
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 0.23.0` (baseline `desktop-1.18286.0`).
3
+ Self-contained reference. Tracks `cowork-harness 0.25.0` (baseline `desktop-1.18286.0`).
4
4
 
5
5
  **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
6
6
  job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -53,7 +53,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
53
53
  GitHub-hosted runners, no token/Docker/agent:
54
54
 
55
55
  ```yaml
56
- - run: npm i -g "cowork-harness@>=0.23.0"
56
+ - run: npm i -g "cowork-harness@>=0.25.0"
57
57
  - run: cowork-harness lint scenarios/*.yaml # no silent false-greens
58
58
  - run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
59
59
  - run: cowork-harness replay cassettes/ # token-free content/structure
@@ -125,9 +125,17 @@ dollar figures). In a skill repo these cassettes get **committed**. So:
125
125
  customer's cap table.
126
126
  - **Opt-in redaction** rewrites configured PII out of the cassette at record time. Drop a
127
127
  `.cowork-redact.json` next to your scenarios (or set `COWORK_HARNESS_REDACT_PATTERNS` /
128
- `_KEYS`); empty by default. Redaction is **verdict-preserving** — `record` refuses to write if
128
+ `_KEYS`); empty by default. The policy is searched in **cwd → the scenario's dir → the cassette's
129
+ dir** (first file found per dir; env vars merge on top). `cowork-harness init-redact` copies the
130
+ packaged reference template (local-path prefixes + a generic email regex) into the cwd as a starting
131
+ point — review and tailor it. Redaction is **verdict-preserving** — `record` refuses to write if
129
132
  redaction would flip an assertion (a manufactured green). `--no-redact` skips it for known-synthetic
130
133
  inputs.
134
+ - **Pre-spawn preflight**: `record` warns (`::warning::`, before the paid run starts — once per batch
135
+ for `record <dir>` / `--rerecord-stale`) when a scenario about to record at a host-path-bearing tier
136
+ (`hostloop`, `protocol`) has an **empty** redaction policy — that combination commits real host paths
137
+ the `path` scanner then hard-fails at `verify-cassettes` time. The always-on scanner remains the
138
+ universal net (container-tier recordings can trip it too).
131
139
  - **Always-on scan gate** — `verify-cassettes` flags email / currency / bare-domain / local-path /
132
140
  machine-inventory matches it finds in the committed cassettes and **exits non-zero**, so "no leak" is
133
141
  a gate, not discipline.
@@ -179,7 +187,7 @@ jobs:
179
187
  with: { node-version: '20' }
180
188
  - uses: actions/setup-python@v5
181
189
  with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
182
- - run: npm i -g "cowork-harness@>=0.23.0"
190
+ - run: npm i -g "cowork-harness@>=0.25.0"
183
191
  - run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
184
192
  - run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
185
193
  - run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
@@ -208,7 +216,7 @@ jobs:
208
216
  echo "live=true" >> "$GITHUB_OUTPUT"
209
217
  fi
210
218
  - if: steps.guard.outputs.live == 'true'
211
- run: npm i -g "cowork-harness@>=0.23.0"
219
+ run: npm i -g "cowork-harness@>=0.25.0"
212
220
  - if: steps.guard.outputs.live == 'true'
213
221
  run: cowork-harness run scenarios/ --output-format json
214
222
  env:
@@ -235,6 +243,11 @@ scenario = `result === "success" && assertions.every(pass)`. Exit code is non-ze
235
243
  fails or a run errors, so a plain `cowork-harness run scenarios/` is already CI-ready without parsing
236
244
  JSON.
237
245
 
246
+ `verify-cassettes` emits its **own** envelope (`{command, ok, coverage, results[]}` with per-file
247
+ `findings`/`staleness`/`notes`/`version`/`error`), published as `schema/verify-cassettes.json` in the
248
+ npm package. Both envelope schemas are covered 1.0 contract surfaces (SPEC §12) — parse the JSON, not
249
+ the human-readable text (which is explicitly NOT stable).
250
+
238
251
  A run writes to `~/.cowork-harness/runs/<name>/<sessionId>/` by default — outside any working tree. In CI,
239
252
  set `COWORK_HARNESS_RUNS_DIR` (or pass `--run-dir`) to a workspace-relative path (e.g. `runs`) so an
240
253
  artifact-upload step can collect them. Each run dir holds `events.jsonl`, `control-out.jsonl`, `run.jsonl`,
@@ -261,6 +274,13 @@ does **not** imply the recording is still valid. Each replay result carries `sta
261
274
  | `format` | recorded under an older hash format | re-record once |
262
275
  | `unverifiable-baseline` | the latest baseline couldn't be loaded | couldn't verify (env, not skill) |
263
276
  | `unverifiable-skill` | skill dirs unresolvable — skill staleness couldn't be checked | couldn't verify the skill |
277
+ | `resolved-tier` | a `fidelity: cowork` cassette's recorded `effectiveFidelity` no longer matches what the baseline resolves to today (the host-loop gate flipped) | **high** (the recording exercises the wrong tier) |
278
+ | `unverifiable-tier` | tier check couldn't run for a `fidelity: cowork` cassette (no recorded `effectiveFidelity`, or its pinned baseline failed to load) | couldn't verify the tier — re-record |
279
+
280
+ (A pre-`effectiveFidelity` cassette with an **explicit** tier is statically knowable — it passes the tier
281
+ check with a non-failing informational note in the `verify-cassettes` envelope's per-file `notes[]`, a
282
+ `·`-prefixed row in text output. On `verify-cassettes` every staleness *finding* above is a hard fail —
283
+ the gate is class-blind; notes never fail it.)
264
284
 
265
285
  To gate in CI, pick the severity you want:
266
286
 
@@ -1,6 +1,6 @@
1
1
  # Fidelity tiers & answer paths
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 0.23.0` (baseline `desktop-1.18286.0`).
3
+ Self-contained reference. Tracks `cowork-harness 0.25.0` (baseline `desktop-1.18286.0`).
4
4
 
5
5
  ## Fidelity tiers (`fidelity:` in the scenario)
6
6
 
@@ -1,6 +1,6 @@
1
1
  # Scenario & session schema, assertion catalog, web_fetch, full gotchas
2
2
 
3
- Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.23.0`
3
+ Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.25.0`
4
4
  (baseline `desktop-1.18286.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
5
5
  `docs/session.md`, and `SPEC.md`.
6
6
 
@@ -219,6 +219,7 @@ passes only if every key passes. Keep one concern per item unless you mean conju
219
219
  | `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it |
220
220
  | `user_visible_artifact: <path>` | exists **and** under a user-visible root (`outputs/` + each connected folder's mount name) — the right primitive for a workspace deliverable when a folder is connected |
221
221
  | `no_delete_in_outputs: true` | no delete op touched `mnt/outputs` — **only `true` is valid**; `false` is rejected (omit to allow deletes) |
222
+ | `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); **microvm cannot capture** (use container/hostloop); replay needs `cassette.preRunPaths` (≥0.24 container/hostloop recordings) — cassettes without it **exclude** the key with a loud warning |
222
223
  | `self_heal_ran: <bool>` | a plugin-root self-heal script was (not) invoked |
223
224
  | `tool_called: <Tool>` | the agent invoked the tool (actually ran it) |
224
225
  | `tool_not_called: <Tool>` | the agent never invoked it |
@@ -228,7 +229,7 @@ passes only if every key passes. Keep one concern per item unless you mean conju
228
229
  | `subagent_tool_absent: <Tool>` | no sub-agent used the tool |
229
230
  | `subagent_dispatched: <regex>` | a sub-agent whose `agentType` **or dispatch description** matches |
230
231
  | `subagent_declared_but_unused: <Tool>` | a sub-agent declared the tool but never used **that** tool (even if it used others) |
231
- | `dispatch_count_max: <N>` | at most N sub-agents dispatched (records only — does NOT enforce a cap) |
232
+ | `dispatch_count_max: <N>` | at most N sub-agents dispatched — an author-chosen budget (Cowork imposes no in-conversation Task-dispatch cap; records only, enforces nothing — see gotcha 12) |
232
233
  | `skill_triggered: <regex>` | a skill matching the regex (invoked id, e.g. `"plugin:skill"`) was invoked via the `Skill` tool — evidence-unavailable (not a normal fail) if the agent's init tools have no `Skill` tool |
233
234
  | `no_skill_triggered: <regex>` | no invoked skill id matched — the negative-control / description-collision catcher; evidence-unavailable (never a vacuous pass) if invocation data is absent or the `Skill` tool is unobservable |
234
235
  | `max_cost_usd: <N>` | the run's SDK-reported cost is ≤ N USD — evidence-unavailable if cost telemetry is absent. **Replay asserts the frozen recording's cost, not fresh spend** — a real regression needs a live `run` |
@@ -236,9 +237,10 @@ passes only if every key passes. Keep one concern per item unless you mean conju
236
237
  | `tool_calls_max: <N>` | total top-level tool calls (sum of `toolCounts`) ≤ N — meaningfully replay-checkable (re-drive recomputes `toolCounts` deterministically) |
237
238
  | `max_turns: <N>` | the SDK-reported (or fallback-counted) turn count ≤ N — meaningfully replay-checkable (re-drive recounts turns deterministically, same as `tool_calls_max`) |
238
239
  | `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
239
- | `questions_count_max: <N>` | the agent asked at most N questions |
240
- | `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail) |
240
+ | `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
241
+ | `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min` to also require a gate |
241
242
  | `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests |
243
+ | `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass |
242
244
  | `allow_permissive_auto_allow: true` | verdict modifier — suppresses the default-fail when the run recorded a cowork-parity permissive auto-allow; for tests that deliberately assert Cowork's permissive behavior |
243
245
  | `allow_missing_capability: true` | verdict modifier — suppresses the default-fail when the (partial "core") agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables). Assert only when the skill's fallback is genuinely equivalent; otherwise rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check. Live tiers only |
244
246
  | `allow_l0_plugin_divergence: true` | verdict modifier — opt into L0/protocol plugin divergence: suppresses the default-fail when a plugin behaves differently at `protocol` (L0) fidelity than under a sandboxed tier. Live tiers only |
@@ -292,8 +294,10 @@ modifiers `allow_permissive_auto_allow` / `allow_missing_capability` / `allow_l0
292
294
  `allow_stall` are also kept on replay, evaluated as no-op passes.
293
295
 
294
296
  **Gate keys — replay only with a `controlOut` cassette:** `question_asked`, `questions_count_max`,
295
- `gate_answers_delivered`. With `controlOut` present they evaluate; on an old cassette without it, a
296
- **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
297
+ `gate_answers_delivered`, `gate_answer_count_min`. With `controlOut` present they evaluate; on an old
298
+ cassette without it, a **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
299
+ `questions_count_max` counts sub-questions, not gates/tool-calls — see the catalog row above and
300
+ `trace --view questions`.
297
301
 
298
302
  **Filesystem — replay-checkable WITH an artifact manifest:** `file_exists`, `user_visible_artifact`,
299
303
  `artifact_json`, `computer_links_resolve` run on replay when the cassette carries an `artifacts` snapshot
@@ -302,7 +306,9 @@ small-file JSON `body` inlined; a hash-only entry still satisfies `file_exists`.
302
306
  resolves a `/sessions/…/mnt/…`-shaped link directly against the manifest, and a host-shaped (hostloop) link
303
307
  by first normalizing it to a mount-relative path (recorded connected-folder prefixes + the outputs/uploads
304
308
  mounts) — replay has no live filesystem to check a host path against directly (that only happens on a live
305
- `run`/`verify-run`). Without a manifest (older cassettes) all four are skipped. A green replay re-confirms
309
+ `run`/`verify-run`). Without a manifest (older cassettes) all five are skipped; `no_unexpected_files` also
310
+ needs `preRunPaths` (≥0.24 recordings) — without it the key is excluded with a loud warning (live/verify-run
311
+ hard-fails evidence-unavailable instead). A green replay re-confirms
306
312
  *record-time* artifacts, not that the current skill still produces them — `replay --strict` fails when the
307
313
  staleness `fingerprint` shows ANY skill/baseline drift, or `replay --fail-on-skill-drift` only on
308
314
  skill-source drift; every replay result also reports it class-tagged in `staleness[]` for a JSON gate.
@@ -371,10 +377,12 @@ at the top of this file.
371
377
  concern per item; run the linter. (`contentKeys` in `src/run/cassette.ts`.)
372
378
 
373
379
  2. **Gate keys need a `controlOut` cassette.** `question_asked`, `questions_count_max`,
374
- `gate_answers_delivered` only evaluate on replay with `controlOut`; on an old cassette they warn
375
- and are excluded (not passed). `gate_answers_delivered` **fails on unobserved delivery**
376
- (`delivered: null`) — absence of evidence is failure. A **header-only gate** (empty `question`,
377
- only `header`) can never be keyed and is rejected loudly — every gate needs a non-empty `question`.
380
+ `gate_answers_delivered`, `gate_answer_count_min` only evaluate on replay with `controlOut`; on an
381
+ old cassette they warn and are excluded (not passed). `gate_answers_delivered` **fails on
382
+ unobserved delivery** (`delivered: null`) — absence of evidence is failure — but **passes
383
+ vacuously when zero gates fired**; use `gate_answer_count_min: 1` to also require a gate to have
384
+ fired. A **header-only gate** (empty `question`, only `header`) can never be keyed and is rejected
385
+ loudly — every gate needs a non-empty `question`.
378
386
  (`contentKeys` in `src/run/cassette.ts`; `src/assert.ts`.)
379
387
 
380
388
  3. **The LLM-decider's two spellings.** Scripted answers + `on_unanswered: fail` is deterministic;
@@ -410,9 +418,11 @@ at the top of this file.
410
418
  11. **`subagent_declared_but_unused` fires on declared-but-didn't-use-THAT-tool**, even if the
411
419
  sub-agent used other tools.
412
420
 
413
- 12. **`dispatch_count_max` asserts but does NOT enforce.** It records the count; it does not
414
- reproduce Cowork's skip-on-cap (`{perTask:1, global:3}`, deferred). Passing means "dispatched ≤N
415
- this run," not "the harness capped it."
421
+ 12. **`dispatch_count_max` is an author-chosen budget, not a production cap.** It records the count
422
+ and asserts on it; passing means "dispatched ≤N this run," not "the harness capped it." Cowork
423
+ imposes no in-conversation Task-dispatch cap to reproduce — gate `1648655587`'s
424
+ `{perTask:1, global:3}` is the scheduled/cron-task session limiter, a different mechanism
425
+ (binary-verified; SPEC §10).
416
426
 
417
427
  13. **`protocol` is rejected (not silently passed) if the scenario asserts egress** — boundary
418
428
  assertions need `container`+. Fails loud by design.
@@ -450,3 +460,14 @@ at the top of this file.
450
460
  Consequence: artifact assertions (`artifact_json`) over fields that were redacted will fail at replay
451
461
  — the harness emits `::warning::` at record time when this occurs. (`src/secrets.ts:scrubField`;
452
462
  `src/run/cassette.ts`.)
463
+
464
+ 21. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
465
+ read-only folder's files as `path` + `bytes` + `sha256` only (`truncated: true`, no `body`) — it's an
466
+ input the agent read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass
467
+ against it on replay (the hash-only entry still materializes a 0-byte placeholder); `artifact_json`
468
+ reports a clear evidence-unavailable on every lane (live/verify-run/replay agree). This is also why a
469
+ `mode: r` input never trips the `binary` privacy finding
470
+ or needs `--allow` in `verify-cassettes` — only a *committed* body is scanned. `scaffold` won't emit
471
+ `file_exists` for one either, since it isn't in `RunResult.artifacts`. A `mode: rw`/`rwd` folder's
472
+ contents are captured with a full body, same as `outputs/`. (`src/run/cassette.ts:buildManifest`'s
473
+ `bodyLessPrefixes`; `src/session.ts:readonlyFolderRootsFromPlan`.)
@@ -0,0 +1,110 @@
1
+ # Task recipes — end-to-end paths for the four jobs consumers actually reverse-engineer
2
+
3
+ Each recipe composes facts that live scattered across SKILL.md and the other references into one
4
+ decision path. Every one answers a question a real fleet owner had to work out the hard way. Facts track the harness version in SKILL.md's
5
+ front-matter; the `resolved-tier`/`unverifiable-tier` staleness classes (Recipe 2) and
6
+ `init-redact` (Recipe 3) are new in the release AFTER 0.23.0.
7
+
8
+ ## Recipe 1 — Evolve the `assert:` block of an existing cassette (usually NO re-record)
9
+
10
+ You added or changed assertion keys in a scenario whose cassette is already committed. Do you need
11
+ a paid live re-record? Walk this tree — the answer is usually no:
12
+
13
+ 1. **Content / always-replay keys** (`transcript_*`, `tool_called`/`tool_not_called`,
14
+ `tool_result_*`, `subagent_*`, `dispatch_count_max`, `skill_triggered`/`no_skill_triggered`,
15
+ `result`, `max_turns`, `tool_calls_max`, `max_cost_usd`, `max_tokens`) →
16
+ `cowork-harness replay <cassette> --assert-from <scenario.yaml>`. Token-free, no re-record.
17
+ If the recording genuinely lacks the telemetry a key needs (very old cassettes), the key fails
18
+ **loud** as `evidence-unavailable` — that is correct behavior, not a bug; only then re-record.
19
+ 2. **Gate keys** (`question_asked`, `questions_count_max`, `gate_answers_delivered`) on a cassette
20
+ **with `controlOut`** (any modern recording) → same token-free `--assert-from` path.
21
+ 3. **Gate keys** on a **pre-`controlOut`** cassette → one re-record unlocks gate asserts for that
22
+ cassette permanently.
23
+ 4. **Filesystem / egress keys** (`file_exists` without an artifact manifest, `egress_allowed`,
24
+ `egress_denied`, `no_delete_in_outputs`) → live lane **by design**; replay skips them with a
25
+ loud `::warning::`. Keep them in the scenario, run them on the nightly live gate.
26
+
27
+ **The one hard caveat:** `--assert-from` **hard-fails on recording-shaping drift** — if the
28
+ scenario's `prompt:`, `answers:`, baseline, or skill content differ from what shaped the
29
+ recording, re-asserting under them is meaningless and the harness refuses. The trap you'll
30
+ actually hit is `answers:`: sequence your edits — land assert-only changes (token-free), and land
31
+ answer changes separately (those DO force a re-record).
32
+
33
+ ## Recipe 2 — Audit a fleet for tier drift
34
+
35
+ A `fidelity: cowork` scenario records at whatever tier the baseline's host-loop gate resolves to
36
+ at record time. When Cowork flips that gate, your committed cassettes silently stop representing
37
+ what production would do. Two layers of defense:
38
+
39
+ - **Automated (preferred):** `cowork-harness verify-cassettes <dir>` raises a
40
+ **`resolved-tier`** staleness finding when a `fidelity: cowork` cassette's recorded
41
+ `effectiveFidelity` no longer matches what the current baseline resolves to, and
42
+ **`unverifiable-tier`** when a `fidelity: cowork` cassette predates `effectiveFidelity`
43
+ entirely (can't verify → re-record). Explicit-tier cassettes (`hostloop`, `container`, …) are
44
+ statically knowable and only produce a non-failing informational note.
45
+ - **Manual (one-liner):** `grep -h '"effectiveFidelity"' cassettes/*.cassette.json | sort | uniq -c`
46
+ shows the tier distribution of the fleet at a glance.
47
+
48
+ ### Cassette anatomy (what you're looking at when you open one)
49
+
50
+ Top-level fields of a `*.cassette.json` (schema `schema/cassette.v7.json`):
51
+
52
+ | Field | What it is |
53
+ |---|---|
54
+ | `$schema`, `generator`, `cassetteVersion` | Provenance: schema URL, producing tool, format version (current: 7) |
55
+ | `scenario` | The embedded scenario snapshot at record time |
56
+ | `events` | The recorded agent event stream (the replay source) |
57
+ | `controlOut` | Driver→agent control responses — presence unlocks gate asserts on replay |
58
+ | `effectiveFidelity` | The tier the live record actually resolved to (the drift-audit key above) |
59
+ | `artifacts` | Output-file manifest (paths + hashes + small inlined bodies) — unlocks `file_exists`/`artifact_json` on replay |
60
+ | `fingerprint` | Skill/baseline staleness tripwire |
61
+ | `userVisibleRoots` | The user-visible mount roots captured at record time |
62
+ | `readonlyFolderRoots` | The read-only (`mode: r`) subset of `userVisibleRoots` — captured body-less; lets replay give `artifact_json` the precise remedy for a body-less input |
63
+ | `preRunPaths` | Pre-run file-path baseline for `no_unexpected_files` (workRoot-relative; co-present with `userVisibleRoots`) |
64
+ | `scenarioSource` | Relative path to the authored YAML this was recorded from |
65
+ | `authoring` | Present iff a live decider answered ≥1 gate during recording (`nonDeterministic: true`) |
66
+
67
+ ## Recipe 3 — Set up redaction BEFORE your first hostloop/protocol record
68
+
69
+ `hostloop` and `protocol` runs execute on the host, so real host paths (`/Users/you/…`) land in
70
+ the transcript and would be committed inside the cassette. Set the policy up **before** the first
71
+ record — retrofitting means re-recording:
72
+
73
+ 1. `cowork-harness init-redact` — copies the reference `.cowork-redact.json` (host-path +
74
+ email patterns) into the current directory. Version-control it.
75
+ 2. **Search set:** record looks for `.cowork-redact.json` in the **cwd, the scenario's directory,
76
+ and the cassette's output directory** — every distinct file found is MERGED (plus
77
+ `COWORK_HARNESS_REDACT_PATTERNS`/`_KEYS` from the env). Repo root (= cwd in CI) is the
78
+ conventional home.
79
+ 3. **Preflight backstop:** `record` on a host-path-bearing tier with NO effective policy emits a
80
+ loud pre-spawn `::warning::` (once per batch). If you see it, stop and do step 1.
81
+ 4. **Scanner relationship:** redaction (record-time, rewrites content) and the
82
+ `verify-cassettes` privacy scanner (CI-time, read-only tripwire over 4 classes: `domain`,
83
+ `email`, `path`, `machine-inventory`) are independent layers. The scanner catches what
84
+ redaction missed.
85
+ 5. **Scoping idioms for scanner findings you've reviewed and accepted:** prefer the class-scoped
86
+ flags — `--allow-domain 'api\.example\.com'`, `--allow-email '.*@yourco\.com'`,
87
+ `--allow-path '/opt/ci-runner/.*'` — over bare `--allow <regex>` (a single pattern, which
88
+ applies to every class and can let a domain allow swallow an email leak). For a
89
+ version-controlled allowlist use `--allow-patterns-file <path>`: a FILE of bare all-class
90
+ patterns, one regex per line, `#` comments and blank lines ignored — note the flag name is the
91
+ FILE of patterns, not a path to allow.
92
+
93
+ ## Recipe 4 — Budget assertions without a two-pass record
94
+
95
+ `max_turns` / `tool_calls_max` / `questions_count_max` / `dispatch_count_max` need observed
96
+ values, but you don't need a second paid run to get them — derive them from evidence you already
97
+ have:
98
+
99
+ 1. **From the run you already did:** `run` always keeps its run dir. Read observed turns from
100
+ `result.json` (`usage.turns`), tool-call and question counts from
101
+ `cowork-harness trace <run-id> --view tools` / `--view questions`, and the dispatch count from
102
+ `--view dispatches`.
103
+ 2. **From history:** `cowork-harness stats <scenario> --metric turns` gives p50/p95 turns across
104
+ indexed runs — better than a single observation.
105
+ 3. **Set the budget with headroom:** observed × 1.5, rounded up (agentic run lengths vary run to
106
+ run; a budget at the observed value flakes).
107
+ 4. **Replay-class note:** `max_turns` / `tool_calls_max` / `dispatch_count_max` re-evaluate
108
+ token-free on replay; `questions_count_max` needs `controlOut` (Recipe 1's tree applies).
109
+ `max_cost_usd` / `max_tokens` on replay assert the FROZEN recording's spend — near-zero signal
110
+ as a regression gate; if you need a cost gate, put it on the live lane via `stats`.
@@ -11,12 +11,14 @@
11
11
  "egress_allowed",
12
12
  "egress_denied",
13
13
  "file_exists",
14
+ "gate_answer_count_min",
14
15
  "gate_answers_delivered",
15
16
  "max_cost_usd",
16
17
  "max_tokens",
17
18
  "max_turns",
18
19
  "no_delete_in_outputs",
19
20
  "no_skill_triggered",
21
+ "no_unexpected_files",
20
22
  "question_asked",
21
23
  "questions_count_max",
22
24
  "replay_protocol_fidelity",
@@ -16,9 +16,14 @@ Two subcommands:
16
16
 
17
17
  lint flags (see references/scenario-schema.md for the why of each):
18
18
  E egress assertion on `fidelity: protocol` (the harness rejects this run)
19
+ E `transcript_no_host_path` on hostloop/protocol (fails BY DESIGN at those tiers)
20
+ E `requires_capabilities` on `fidelity: protocol` (probe can't run → hard-fails
21
+ unless allow_missing_capability)
19
22
  E on_unanswered: agent / invalid value (schema rejects `agent`)
20
23
  E authored `replay_protocol_fidelity` assertion (replay-synthesized only)
21
24
  E `assertions:` instead of `assert:` (block ignored → every check no-ops)
25
+ W `transcript_no_host_path` on `fidelity: cowork` (tier resolves per baseline gate —
26
+ incompatible if it lands hostloop)
22
27
  W no content assertion → no-op on a replay gate (every assertion is fs/egress)
23
28
  W mixed-class assert item → fs/egress half dropped on replay
24
29
  W unknown top-level / assertion key (typo or hallucinated schema)
@@ -67,12 +72,12 @@ CONTENT_KEYS = {
67
72
  "max_turns",
68
73
  }
69
74
  # content keys, but only evaluated on replay when the cassette carries controlOut
70
- GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered"}
75
+ GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered", "gate_answer_count_min"}
71
76
  # manifest-backed: replay-checkable when the cassette carries an `artifacts` manifest (record snapshots one);
72
77
  # a manifest-less cassette skips them. Since the 0.3.0 artifact-manifest these are NOT always live-only.
73
78
  # computer_links_resolve joins this bucket (not CONTENT_KEYS): resolving a non-empty link set needs
74
79
  # either a live filesystem or the cassette's artifacts manifest — see cassette.ts's manifestKeys comment.
75
- MANIFEST_KEYS = {"file_exists", "user_visible_artifact", "artifact_json", "computer_links_resolve"}
80
+ MANIFEST_KEYS = {"file_exists", "user_visible_artifact", "artifact_json", "computer_links_resolve", "no_unexpected_files"}
76
81
  # live-only: ALWAYS skipped on replay, with a loud warning (no filesystem, no network on the token-free lane)
77
82
  LIVE_ONLY_KEYS = {
78
83
  "egress_denied",
@@ -139,7 +144,7 @@ _EMBEDDED_TOP_LEVEL_KEYS = {
139
144
  "answers",
140
145
  "expect_denied",
141
146
  "assert",
142
- "skills", # F-6: opt-in skill-staleness hash scope
147
+ "skills", # opt-in skill-staleness hash scope
143
148
  "requires_capabilities", # Fix 4b: scenario-level required-capability declaration (pre-flight gate)
144
149
  "allow_host_writes", # hostloop native-split: consent for a writable connected folder (pre-run gate)
145
150
  }
@@ -177,6 +182,13 @@ REGEX_KEYS = {
177
182
  VALID_ON_UNANSWERED = {"fail", "prompt", "first", "llm"}
178
183
  VALID_TIERS = ("protocol", "container", "microvm", "hostloop", "cowork")
179
184
 
185
+ # Gate-id tripwire: the `host-path-assert-cowork` WARN below embeds Cowork's
186
+ # host-loop gate id in offline Python (the linter never reads a baseline). The
187
+ # id is pinned by test/scenario-lint-gate-id.test.ts against the PINNED_GATES
188
+ # entry in src/sync/cowork-sync.ts, so a Desktop gate re-key fails loud there
189
+ # instead of silently rotting this message.
190
+ HOST_LOOP_GATE_ID = "1143815894"
191
+
180
192
 
181
193
  class Finding:
182
194
  __slots__ = ("severity", "rule", "message", "fix", "file", "line")
@@ -300,6 +312,60 @@ def lint_doc(doc, path, raw_lines):
300
312
  )
301
313
  )
302
314
 
315
+ # E/W: transcript_no_host_path is tier-incompatible with hostloop/protocol — the agent
316
+ # legitimately runs on real host paths there, so the assertion fails BY DESIGN (the runtime only
317
+ # warns at run start, after authoring). Lint is deliberately STRICTER than the runtime: the docs
318
+ # declare the combination incompatible, so authoring it is a bug even if a tool-free run could
319
+ # accidentally pass. `cowork` gets a WARN naming the baseline-gate resolution dependency (the
320
+ # linter stays offline — the message carries the gate fact instead of reading a baseline).
321
+ if "transcript_no_host_path" in assert_keys:
322
+ if fidelity in ("hostloop", "protocol"):
323
+ findings.append(
324
+ Finding(
325
+ "ERROR",
326
+ "host-path-assert-tier",
327
+ f"`transcript_no_host_path` on `fidelity: {fidelity}` — the agent runs on real "
328
+ "host paths at this tier, so this assertion FAILS BY DESIGN (it can never be a "
329
+ "meaningful check here).",
330
+ "Run this assertion at fidelity: container (or microvm), or drop it for this tier.",
331
+ path,
332
+ )
333
+ )
334
+ elif fidelity == "cowork":
335
+ findings.append(
336
+ Finding(
337
+ "WARN",
338
+ "host-path-assert-cowork",
339
+ "`transcript_no_host_path` on `fidelity: cowork` — the tier resolves per the "
340
+ f"baseline's host-loop gate ({HOST_LOOP_GATE_ID}); if it resolves to hostloop "
341
+ "this assertion fails by design (and a later gate flip re-stales the cassette).",
342
+ "Pin fidelity: container if the assertion is load-bearing; keep cowork only if "
343
+ "you accept the gate-resolution dependency.",
344
+ path,
345
+ )
346
+ )
347
+
348
+ # E: requires_capabilities on protocol — the capability probe cannot run at protocol tier
349
+ # (clause b of the requires_capabilities contract), so the run HARD-FAILS unless an assert item
350
+ # opts out via allow_missing_capability: true. Offline-detectable fails-by-design, same class as
351
+ # the tier/assert rules above.
352
+ req_caps = doc.get("requires_capabilities")
353
+ if req_caps and fidelity == "protocol":
354
+ opted_out = any(item.get("allow_missing_capability") is True for item in items)
355
+ if not opted_out:
356
+ findings.append(
357
+ Finding(
358
+ "ERROR",
359
+ "capabilities-on-protocol",
360
+ "non-empty `requires_capabilities` on `fidelity: protocol` — the capability "
361
+ "probe cannot run at protocol tier, so the run hard-fails as unverifiable "
362
+ "(fails by design).",
363
+ "Use a sandboxed tier (container/microvm/hostloop), or add "
364
+ "`allow_missing_capability: true` to an assert item to opt out explicitly.",
365
+ path,
366
+ )
367
+ )
368
+
303
369
  # E: retired/invalid on_unanswered
304
370
  ou = doc.get("on_unanswered")
305
371
  if ou is not None and ou not in VALID_ON_UNANSWERED:
@@ -543,7 +609,7 @@ def cmd_lint(args):
543
609
  else:
544
610
  expanded.append(arg)
545
611
  args.files = expanded
546
- # Linter self-check (B3): a valid schema key the replay-class sets don't classify can't be linted
612
+ # Linter self-check: a valid schema key the replay-class sets don't classify can't be linted
547
613
  # correctly — surface it as a hard ERROR so it fails the gate (and --strict) until someone classifies it.
548
614
  if UNCLASSIFIED_KEYS:
549
615
  all_findings.append(
@@ -650,7 +716,15 @@ def build_scenario(args):
650
716
  if gates:
651
717
  for rx, _ in gates:
652
718
  content_lines.append(f" - question_asked: {_sq(rx)} # gate key: replay only with a controlOut cassette")
653
- content_lines.append(f" - questions_count_max: {len(gates)}")
719
+ # questions_count_max counts SUB-questions at runtime (assert.ts/trace-view.ts), but this
720
+ # scaffold is STATIC — it only knows the number of --gate rules (per-tool-call), never how many
721
+ # sub-questions each gate bundles. Any number emitted here would be a guess: too low false-reds
722
+ # on the first run, too high is a dead tripwire. A budget must come from observation, not
723
+ # fabrication — so emit it COMMENTED OUT with the calibration path, not a made-up value.
724
+ content_lines.append(
725
+ " # - questions_count_max: <N> # BUDGET — calibrate from a real run: `trace --view "
726
+ "questions` prints the SUB-question total (what this asserts); set N to that + headroom."
727
+ )
654
728
  content_lines.append(" - gate_answers_delivered: true # the steered answers actually reached the model")
655
729
 
656
730
  live_lines = []
@@ -0,0 +1,11 @@
1
+ {
2
+ "patterns": [
3
+ { "regex": "/Users/[^\\s\"'\\\\)\\]`]+?(?=/mnt/)", "label": "local-path", "flags": "g" },
4
+ { "regex": "/home/[^\\s\"'\\\\)\\]`]+?(?=/mnt/)", "label": "local-path", "flags": "g" },
5
+ { "regex": "/root/[^\\s\"'\\\\)\\]`]+?(?=/mnt/)", "label": "local-path", "flags": "g" },
6
+ { "regex": "/Users/[^\\s\"'\\\\)\\]`]+", "label": "local-path", "flags": "g" },
7
+ { "regex": "/home/[^\\s\"'\\\\)\\]`]+", "label": "local-path", "flags": "g" },
8
+ { "regex": "/root/[^\\s\"'\\\\)\\]`]+", "label": "local-path", "flags": "g" },
9
+ { "regex": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "label": "email", "flags": "g" }
10
+ ]
11
+ }