cowork-harness 0.24.0 → 0.25.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +25 -4
- package/.claude/skills/cowork-harness/references/ci-recipe.md +4 -4
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +25 -9
- package/.claude/skills/cowork-harness/references/task-recipes.md +6 -4
- package/.claude/skills/cowork-harness/scripts/assertion-keys.json +1 -0
- package/.claude/skills/cowork-harness/scripts/scenario.py +10 -2
- package/CHANGELOG.md +71 -0
- package/README.md +5 -5
- package/SPEC.md +16 -5
- package/dist/assert.js +39 -4
- package/dist/cli.js +8 -2
- package/dist/run/cassette.js +61 -24
- package/dist/run/execute.js +16 -3
- package/dist/run/trace-view.js +15 -5
- package/dist/session.js +8 -0
- package/dist/types.js +7 -1
- package/docs/cassette.md +38 -17
- package/docs/scenario.md +15 -5
- package/examples/replays/README.md +1 -1
- package/package.json +1 -1
- package/schema/cassette.v7.json +7 -0
- package/schema/scenario.schema.json +7 -1
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 0.
|
|
7
|
-
tracks-harness: cowork-harness 0.
|
|
6
|
+
version: 0.25.0
|
|
7
|
+
tracks-harness: cowork-harness 0.25.0 (baseline desktop-1.18286.0)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.25.0` (baseline
|
|
26
26
|
> `desktop-1.18286.0`). If your checkout is newer, prefer the live `--help`, `SPEC.md`, and
|
|
27
27
|
> `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
|
|
|
38
38
|
Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.
|
|
41
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.25.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.25.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.25.0"`. **Pin `@>=0.25.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.25.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
|
|
42
42
|
- **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
|
|
43
43
|
- **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
|
|
44
44
|
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
@@ -458,6 +458,27 @@ are the ones that bite hardest.
|
|
|
458
458
|
and re-assert is undetected — the notice flags this; re-record if the session changed. `verify-run` reads
|
|
459
459
|
on-disk `assert:` against a kept *run dir*; `replay --assert-from` is the equivalent for a *cassette*.
|
|
460
460
|
|
|
461
|
+
18. **`questions_count_max` counts sub-questions, not gates.** One `AskUserQuestion` tool call can
|
|
462
|
+
bundle several sub-questions into a single gate; the assertion counts each sub-question, so a
|
|
463
|
+
3-sub-question bundle counts as 3, not 1. `trace --view questions` shows the same per-gate
|
|
464
|
+
sub-question count and a matching footer total — read that off instead of the tool-call count when
|
|
465
|
+
sizing the budget.
|
|
466
|
+
|
|
467
|
+
19. **`gate_answers_delivered` passes vacuously when no gate fires — use `gate_answer_count_min: 1` to
|
|
468
|
+
also require a gate.** Whether a gate fires is model-dependent, so `gate_answers_delivered: true`
|
|
469
|
+
alone can't catch "the gate never fired at all"; pair it with `gate_answer_count_min` when
|
|
470
|
+
presence matters, not just delivery.
|
|
471
|
+
|
|
472
|
+
20. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
|
|
473
|
+
read-only folder's files as path + hash only (`truncated: true`, no `body`) — it's an input the agent
|
|
474
|
+
read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass against it on replay
|
|
475
|
+
(the hash-only entry still materializes a placeholder); `artifact_json` reports a clear
|
|
476
|
+
evidence-unavailable on every lane (live/verify-run/replay agree — no green-record/red-replay). This is
|
|
477
|
+
also why a `mode: r` input never trips the `binary` privacy finding or needs `--allow` — only a
|
|
478
|
+
*committed* body is scanned. `scaffold` won't emit `file_exists` for one either (it's not in
|
|
479
|
+
`RunResult.artifacts`). A `mode: rw`/`rwd` folder's contents are captured with a full body, same as
|
|
480
|
+
`outputs/`.
|
|
481
|
+
|
|
461
482
|
For the complete gotcha list, the assertion catalog, the YAML schema, the fidelity/answer tables,
|
|
462
483
|
and the CI recipe, read the files in `references/`.
|
|
463
484
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.25.0` (baseline `desktop-1.18286.0`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -53,7 +53,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
53
53
|
GitHub-hosted runners, no token/Docker/agent:
|
|
54
54
|
|
|
55
55
|
```yaml
|
|
56
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
56
|
+
- run: npm i -g "cowork-harness@>=0.25.0"
|
|
57
57
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
58
58
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
59
59
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -187,7 +187,7 @@ jobs:
|
|
|
187
187
|
with: { node-version: '20' }
|
|
188
188
|
- uses: actions/setup-python@v5
|
|
189
189
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
190
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
190
|
+
- run: npm i -g "cowork-harness@>=0.25.0"
|
|
191
191
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
192
192
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
193
193
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -216,7 +216,7 @@ jobs:
|
|
|
216
216
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
217
217
|
fi
|
|
218
218
|
- if: steps.guard.outputs.live == 'true'
|
|
219
|
-
run: npm i -g "cowork-harness@>=0.
|
|
219
|
+
run: npm i -g "cowork-harness@>=0.25.0"
|
|
220
220
|
- if: steps.guard.outputs.live == 'true'
|
|
221
221
|
run: cowork-harness run scenarios/ --output-format json
|
|
222
222
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.25.0` (baseline `desktop-1.18286.0`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.25.0`
|
|
4
4
|
(baseline `desktop-1.18286.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
@@ -237,9 +237,10 @@ passes only if every key passes. Keep one concern per item unless you mean conju
|
|
|
237
237
|
| `tool_calls_max: <N>` | total top-level tool calls (sum of `toolCounts`) ≤ N — meaningfully replay-checkable (re-drive recomputes `toolCounts` deterministically) |
|
|
238
238
|
| `max_turns: <N>` | the SDK-reported (or fallback-counted) turn count ≤ N — meaningfully replay-checkable (re-drive recounts turns deterministically, same as `tool_calls_max`) |
|
|
239
239
|
| `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
|
|
240
|
-
| `questions_count_max: <N>` |
|
|
241
|
-
| `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail) |
|
|
240
|
+
| `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
|
|
241
|
+
| `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min` to also require a gate |
|
|
242
242
|
| `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests |
|
|
243
|
+
| `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass |
|
|
243
244
|
| `allow_permissive_auto_allow: true` | verdict modifier — suppresses the default-fail when the run recorded a cowork-parity permissive auto-allow; for tests that deliberately assert Cowork's permissive behavior |
|
|
244
245
|
| `allow_missing_capability: true` | verdict modifier — suppresses the default-fail when the (partial "core") agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables). Assert only when the skill's fallback is genuinely equivalent; otherwise rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check. Live tiers only |
|
|
245
246
|
| `allow_l0_plugin_divergence: true` | verdict modifier — opt into L0/protocol plugin divergence: suppresses the default-fail when a plugin behaves differently at `protocol` (L0) fidelity than under a sandboxed tier. Live tiers only |
|
|
@@ -293,8 +294,10 @@ modifiers `allow_permissive_auto_allow` / `allow_missing_capability` / `allow_l0
|
|
|
293
294
|
`allow_stall` are also kept on replay, evaluated as no-op passes.
|
|
294
295
|
|
|
295
296
|
**Gate keys — replay only with a `controlOut` cassette:** `question_asked`, `questions_count_max`,
|
|
296
|
-
`gate_answers_delivered`. With `controlOut` present they evaluate; on an old
|
|
297
|
-
**loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
|
|
297
|
+
`gate_answers_delivered`, `gate_answer_count_min`. With `controlOut` present they evaluate; on an old
|
|
298
|
+
cassette without it, a **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
|
|
299
|
+
`questions_count_max` counts sub-questions, not gates/tool-calls — see the catalog row above and
|
|
300
|
+
`trace --view questions`.
|
|
298
301
|
|
|
299
302
|
**Filesystem — replay-checkable WITH an artifact manifest:** `file_exists`, `user_visible_artifact`,
|
|
300
303
|
`artifact_json`, `computer_links_resolve` run on replay when the cassette carries an `artifacts` snapshot
|
|
@@ -374,10 +377,12 @@ at the top of this file.
|
|
|
374
377
|
concern per item; run the linter. (`contentKeys` in `src/run/cassette.ts`.)
|
|
375
378
|
|
|
376
379
|
2. **Gate keys need a `controlOut` cassette.** `question_asked`, `questions_count_max`,
|
|
377
|
-
`gate_answers_delivered` only evaluate on replay with `controlOut`; on an
|
|
378
|
-
and are excluded (not passed). `gate_answers_delivered` **fails on
|
|
379
|
-
(`delivered: null`) — absence of evidence is failure
|
|
380
|
-
|
|
380
|
+
`gate_answers_delivered`, `gate_answer_count_min` only evaluate on replay with `controlOut`; on an
|
|
381
|
+
old cassette they warn and are excluded (not passed). `gate_answers_delivered` **fails on
|
|
382
|
+
unobserved delivery** (`delivered: null`) — absence of evidence is failure — but **passes
|
|
383
|
+
vacuously when zero gates fired**; use `gate_answer_count_min: 1` to also require a gate to have
|
|
384
|
+
fired. A **header-only gate** (empty `question`, only `header`) can never be keyed and is rejected
|
|
385
|
+
loudly — every gate needs a non-empty `question`.
|
|
381
386
|
(`contentKeys` in `src/run/cassette.ts`; `src/assert.ts`.)
|
|
382
387
|
|
|
383
388
|
3. **The LLM-decider's two spellings.** Scripted answers + `on_unanswered: fail` is deterministic;
|
|
@@ -455,3 +460,14 @@ at the top of this file.
|
|
|
455
460
|
Consequence: artifact assertions (`artifact_json`) over fields that were redacted will fail at replay
|
|
456
461
|
— the harness emits `::warning::` at record time when this occurs. (`src/secrets.ts:scrubField`;
|
|
457
462
|
`src/run/cassette.ts`.)
|
|
463
|
+
|
|
464
|
+
21. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
|
|
465
|
+
read-only folder's files as `path` + `bytes` + `sha256` only (`truncated: true`, no `body`) — it's an
|
|
466
|
+
input the agent read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass
|
|
467
|
+
against it on replay (the hash-only entry still materializes a 0-byte placeholder); `artifact_json`
|
|
468
|
+
reports a clear evidence-unavailable on every lane (live/verify-run/replay agree). This is also why a
|
|
469
|
+
`mode: r` input never trips the `binary` privacy finding
|
|
470
|
+
or needs `--allow` in `verify-cassettes` — only a *committed* body is scanned. `scaffold` won't emit
|
|
471
|
+
`file_exists` for one either, since it isn't in `RunResult.artifacts`. A `mode: rw`/`rwd` folder's
|
|
472
|
+
contents are captured with a full body, same as `outputs/`. (`src/run/cassette.ts:buildManifest`'s
|
|
473
|
+
`bodyLessPrefixes`; `src/session.ts:readonlyFolderRootsFromPlan`.)
|
|
@@ -59,6 +59,7 @@ Top-level fields of a `*.cassette.json` (schema `schema/cassette.v7.json`):
|
|
|
59
59
|
| `artifacts` | Output-file manifest (paths + hashes + small inlined bodies) — unlocks `file_exists`/`artifact_json` on replay |
|
|
60
60
|
| `fingerprint` | Skill/baseline staleness tripwire |
|
|
61
61
|
| `userVisibleRoots` | The user-visible mount roots captured at record time |
|
|
62
|
+
| `readonlyFolderRoots` | The read-only (`mode: r`) subset of `userVisibleRoots` — captured body-less; lets replay give `artifact_json` the precise remedy for a body-less input |
|
|
62
63
|
| `preRunPaths` | Pre-run file-path baseline for `no_unexpected_files` (workRoot-relative; co-present with `userVisibleRoots`) |
|
|
63
64
|
| `scenarioSource` | Relative path to the authored YAML this was recorded from |
|
|
64
65
|
| `authoring` | Present iff a live decider answered ≥1 gate during recording (`nonDeterministic: true`) |
|
|
@@ -83,10 +84,11 @@ record — retrofitting means re-recording:
|
|
|
83
84
|
redaction missed.
|
|
84
85
|
5. **Scoping idioms for scanner findings you've reviewed and accepted:** prefer the class-scoped
|
|
85
86
|
flags — `--allow-domain 'api\.example\.com'`, `--allow-email '.*@yourco\.com'`,
|
|
86
|
-
`--allow-path '/opt/ci-runner/.*'` — over bare `--allow
|
|
87
|
-
let a domain allow swallow an email leak). For a
|
|
88
|
-
`--allow-file <path>`: bare all-class
|
|
89
|
-
lines ignored
|
|
87
|
+
`--allow-path '/opt/ci-runner/.*'` — over bare `--allow <regex>` (a single pattern, which
|
|
88
|
+
applies to every class and can let a domain allow swallow an email leak). For a
|
|
89
|
+
version-controlled allowlist use `--allow-patterns-file <path>`: a FILE of bare all-class
|
|
90
|
+
patterns, one regex per line, `#` comments and blank lines ignored — note the flag name is the
|
|
91
|
+
FILE of patterns, not a path to allow.
|
|
90
92
|
|
|
91
93
|
## Recipe 4 — Budget assertions without a two-pass record
|
|
92
94
|
|
|
@@ -72,7 +72,7 @@ CONTENT_KEYS = {
|
|
|
72
72
|
"max_turns",
|
|
73
73
|
}
|
|
74
74
|
# content keys, but only evaluated on replay when the cassette carries controlOut
|
|
75
|
-
GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered"}
|
|
75
|
+
GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered", "gate_answer_count_min"}
|
|
76
76
|
# manifest-backed: replay-checkable when the cassette carries an `artifacts` manifest (record snapshots one);
|
|
77
77
|
# a manifest-less cassette skips them. Since the 0.3.0 artifact-manifest these are NOT always live-only.
|
|
78
78
|
# computer_links_resolve joins this bucket (not CONTENT_KEYS): resolving a non-empty link set needs
|
|
@@ -716,7 +716,15 @@ def build_scenario(args):
|
|
|
716
716
|
if gates:
|
|
717
717
|
for rx, _ in gates:
|
|
718
718
|
content_lines.append(f" - question_asked: {_sq(rx)} # gate key: replay only with a controlOut cassette")
|
|
719
|
-
|
|
719
|
+
# questions_count_max counts SUB-questions at runtime (assert.ts/trace-view.ts), but this
|
|
720
|
+
# scaffold is STATIC — it only knows the number of --gate rules (per-tool-call), never how many
|
|
721
|
+
# sub-questions each gate bundles. Any number emitted here would be a guess: too low false-reds
|
|
722
|
+
# on the first run, too high is a dead tripwire. A budget must come from observation, not
|
|
723
|
+
# fabrication — so emit it COMMENTED OUT with the calibration path, not a made-up value.
|
|
724
|
+
content_lines.append(
|
|
725
|
+
" # - questions_count_max: <N> # BUDGET — calibrate from a real run: `trace --view "
|
|
726
|
+
"questions` prints the SUB-question total (what this asserts); set N to that + headroom."
|
|
727
|
+
)
|
|
720
728
|
content_lines.append(" - gate_answers_delivered: true # the steered answers actually reached the model")
|
|
721
729
|
|
|
722
730
|
live_lines = []
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,77 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.25.0] — 2026-07-05
|
|
10
|
+
|
|
11
|
+
> **Upgrade notes.**
|
|
12
|
+
> - **`verify-cassettes --allow-file` was renamed to `--allow-patterns-file`** and the old spelling is
|
|
13
|
+
> removed (exits 2). Update any CI step or allowlist wiring that passed `--allow-file`.
|
|
14
|
+
> - **`gate_answers_delivered: true` now passes vacuously when zero gates fired.** If you relied on it
|
|
15
|
+
> as an implicit "a gate must fire" check, add `gate_answer_count_min: 1` alongside it.
|
|
16
|
+
> - **Read-only (`mode: r`) connected-folder inputs are now captured body-less.** Re-record affected
|
|
17
|
+
> cassettes to drop the `--allow` entries they previously required and shrink them; an `artifact_json`
|
|
18
|
+
> assertion pointed at a read-only input now reports evidence-unavailable (assert on a deliverable).
|
|
19
|
+
> This changes committed cassette *contents* (not the staleness hash) — a re-record is cosmetic, not
|
|
20
|
+
> required for correctness.
|
|
21
|
+
|
|
22
|
+
### Added
|
|
23
|
+
|
|
24
|
+
- **`gate_answer_count_min: <N>` assertion** — the presence companion to `gate_answers_delivered`:
|
|
25
|
+
fails unless at least N `AskUserQuestion` gates fired AND were delivered non-error. Pairs with
|
|
26
|
+
`gate_answers_delivered`'s new zero-gate vacuous-pass (below) the way `transcript_contains` pairs
|
|
27
|
+
with `computer_links_resolve`'s zero-link vacuous-pass. Evaluated on replay only with a `controlOut`
|
|
28
|
+
cassette, like the other gate keys; fails as evidence-unavailable (not vacuous-pass) when
|
|
29
|
+
gate-delivery telemetry is absent from `result.json`.
|
|
30
|
+
|
|
31
|
+
### Changed
|
|
32
|
+
|
|
33
|
+
- **`gate_answers_delivered: true` now passes vacuously when zero `AskUserQuestion` gates fired**,
|
|
34
|
+
instead of hard-failing. Whether a gate fires is model-dependent, so any skill with optional gating
|
|
35
|
+
could not use the assertion under the old hard-fail. The bad-delivery check (an answered gate whose
|
|
36
|
+
answer wasn't confirmed delivered) is unchanged — it only ever mattered when gates fired. Missing
|
|
37
|
+
gate-delivery telemetry (an old/partial `result.json` on the verify-run lane) still fails loud as
|
|
38
|
+
evidence-unavailable, never vacuous-pass — this is NOT the same as "no gate fired" and preserves the
|
|
39
|
+
bug-#33 false-green fix. Pair with the new `gate_answer_count_min` (below) to also require presence.
|
|
40
|
+
- **`verify-cassettes --allow-file <path>` renamed to `--allow-patterns-file <path>`.** The old name
|
|
41
|
+
read as "allow this file" and was routinely confused with `--allow <regex>` (a pattern matched
|
|
42
|
+
against a finding) — reaching for `--allow-file <artifact-path>` failed with an ENOENT/invalid-regex
|
|
43
|
+
error instead of doing what the user meant. `--allow-patterns-file <path>` is self-documenting: the
|
|
44
|
+
path names a **file of patterns** (one regex per line), not a path to allow. Pre-1.0 rename with **no
|
|
45
|
+
deprecation window** — the old `--allow-file` spelling is hard-removed and now exits 2 with an
|
|
46
|
+
unknown-flag error. The binary-finding recourse message and both `--help` texts now spell out the
|
|
47
|
+
distinction between the two flags.
|
|
48
|
+
|
|
49
|
+
### Fixed
|
|
50
|
+
|
|
51
|
+
- **Read-only (`mode: r`) connected-folder inputs are captured body-less (path + hash) instead of as
|
|
52
|
+
full binary bodies.** `record` used to sweep a `mode: r` folder's files into the cassette
|
|
53
|
+
`artifacts[]` as full base64 bodies — an input the agent only reads, not a deliverable — causing
|
|
54
|
+
cassette bloat and a hard `binary` privacy finding that forced `--allow`. The manifest entry now
|
|
55
|
+
carries `path` + `bytes` + `sha256` with `truncated: true` and no `body`, reusing the same
|
|
56
|
+
representation the 64-KiB inline cap already produces. The entry still survives in the manifest (it
|
|
57
|
+
is NOT excluded), so `computer_links_resolve`/`file_exists` resolve identically on live and replay.
|
|
58
|
+
`artifact_json` against a body-less target (a read-only input, or any artifact over the body cap)
|
|
59
|
+
reports a clear **evidence-unavailable** on every lane — live, verify-run, and replay all agree, so
|
|
60
|
+
a cassette can't record green and replay red. The cassette persists the read-only-folder set
|
|
61
|
+
(`readonlyFolderRoots`, a subset of `userVisibleRoots`), so replay knows *why* an entry is body-less
|
|
62
|
+
and gives the **precise** remedy — "assert on a deliverable" for a read-only input, "raise
|
|
63
|
+
`--max-artifact-bytes`" for an over-cap artifact — instead of a cryptic JSON parse error or a
|
|
64
|
+
guessed hint. A `mode: rw`/`rwd` folder's contents are unaffected and keep their full body.
|
|
65
|
+
- **`trace --view questions` and the `scaffold` helper disagreed with `questions_count_max` on what
|
|
66
|
+
"a question" counts.** The assertion counts **sub-questions** (one `AskUserQuestion` bundling K
|
|
67
|
+
sub-questions counts as K — the better spam/burden budget, since per-tool-call counting would miss
|
|
68
|
+
a 10-in-1 bundle), but `trace --view questions` collapsed a bundle into one row with no count, and
|
|
69
|
+
the scaffold helper emitted `questions_count_max: <gate count>` — a budget that could start BELOW
|
|
70
|
+
what the assertion computes and false-red on the first real run. No behavior change to the
|
|
71
|
+
assertion itself: `trace --view questions` now annotates each gate row with its sub-question count
|
|
72
|
+
and prints a footer total (`N gate(s), M sub-question(s) total`) that matches what
|
|
73
|
+
`questions_count_max` compares against. The static `scaffold` helper can't know the sub-question
|
|
74
|
+
count offline, so instead of fabricating a value it now emits `questions_count_max` **commented
|
|
75
|
+
out** as a calibration TODO pointing at `trace --view questions` (a budget must come from
|
|
76
|
+
observation, not a guess — a fabricated one either false-reds or is a dead tripwire). The count
|
|
77
|
+
definition is now documented in `docs/scenario.md`, `docs/cassette.md`, `SPEC.md`, and the skill's
|
|
78
|
+
`references/scenario-schema.md` / `SKILL.md` gotcha list.
|
|
79
|
+
|
|
9
80
|
## [0.24.0] — 2026-07-05
|
|
10
81
|
|
|
11
82
|
### Added
|
package/README.md
CHANGED
|
@@ -84,7 +84,7 @@ Only the committed-cassette `replay` above is fully self-contained. Live `run`/`
|
|
|
84
84
|
> - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
|
|
85
85
|
> - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
|
|
86
86
|
> - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
|
|
87
|
-
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx cowork-harness@>=0.
|
|
87
|
+
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx cowork-harness@>=0.25.0`.
|
|
88
88
|
>
|
|
89
89
|
> *(One project, three names: the repo folder is `claude-cowork-headless-emulator`; the npm package, the CLI command, and the GitHub repo are all `cowork-harness`.)*
|
|
90
90
|
|
|
@@ -104,7 +104,7 @@ This repo ships a **companion skill** (`.claude/skills/cowork-harness/`) that te
|
|
|
104
104
|
/plugin install cowork-harness@cowork-harness
|
|
105
105
|
```
|
|
106
106
|
|
|
107
|
-
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.
|
|
107
|
+
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.25.0` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
|
|
108
108
|
|
|
109
109
|
It also follows the open [Agent Skills](https://github.com/vercel-labs/skills) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) by pointing the `npx skills` CLI at `.claude/skills/cowork-harness` in this repo. (Working *inside* this repo, the skill auto-loads as a project skill — no install needed.)
|
|
110
110
|
|
|
@@ -119,7 +119,7 @@ A global install is enough for CI `lint`, reading the teaching skill, and replay
|
|
|
119
119
|
To `run` the worked examples live or copy them as a starting point, use a source checkout. (The marketplace
|
|
120
120
|
skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
|
|
121
121
|
assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
|
|
122
|
-
above becomes available once the skill's first command self-bootstraps `npx cowork-harness@>=0.
|
|
122
|
+
above becomes available once the skill's first command self-bootstraps `npx cowork-harness@>=0.25.0` — see
|
|
123
123
|
[above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
|
|
124
124
|
|
|
125
125
|
### Prerequisites for anything above `protocol` fidelity
|
|
@@ -179,7 +179,7 @@ cowork-harness lint examples/scenarios/*.yaml
|
|
|
179
179
|
> `tool_calls_max`/`max_turns`, `result`, the verdict modifiers) always — note `max_cost_usd`/`max_tokens`
|
|
180
180
|
> assert the *frozen recording's* spend on replay, not fresh spend; a real budget regression is caught by a
|
|
181
181
|
> live `run` — plus `question_asked`/
|
|
182
|
-
> `questions_count_max`/`gate_answers_delivered` when the cassette carries `controlOut`, and
|
|
182
|
+
> `questions_count_max`/`gate_answers_delivered`/`gate_answer_count_min` when the cassette carries `controlOut`, and
|
|
183
183
|
> **filesystem assertions** (`file_exists`, `user_visible_artifact`, `artifact_json`,
|
|
184
184
|
> `computer_links_resolve`) when it carries an
|
|
185
185
|
> `artifacts` manifest. Genuinely live-only keys (`egress_*`, `expect_denied`, `no_delete_in_outputs`,
|
|
@@ -307,7 +307,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
|
|
|
307
307
|
| `run <scenario.yaml \| dir/>` | Run authored scenarios with `assert:` + a CI-ready exit code; a decider (`--decider-cmd`/`--decider-dir`, or a scenario's `on_unanswered: llm`) can answer unscripted gates. `--repeat N` (2-100) runs each scenario N times and aggregates a variance rollup instead of a single pass/fail — `--min-pass-rate`, `--stop-on-diverge`, `--max-budget-usd` tune the batch verdict/loop. `--matrix <matrix.yaml>` runs ONE scenario across the cross-product of baseline/model/skill_dir axes (worked example: `examples/matrices/csv-metrics-matrix.yaml`) — `--max-cells`/`--concurrency` tune the cap/pool; any cell failing (assertion or infra) fails the run; `--compact`/`--demo` trim output for shareable screenshots/GIFs (shared with `skill`, below) | you want a repeatable, **asserted regression test** — to **measure flakiness** instead of trusting one green — or to **test a compatibility matrix** (multiple baselines/models/skill variants) in one run |
|
|
308
308
|
| `chat <folder> [prompt]` | Interactive multi-turn REPL against a skill (TTY); optional seed prompt is sent as the first turn. `--upload <file>` / `--folder <dir>` (repeatable) attach files/project folders; `--verbose` shows thinking blocks + tool inputs; `--fidelity protocol\|container\|hostloop` (no `microvm`/`cowork` in the REPL) | debugging a multi-turn flow by hand |
|
|
309
309
|
| `record` / `replay` | **Record a live run once → replay it token-free, Docker-free thereafter.** Key flags: `--decider-llm`/`--decider-dir` (answer gates live during recording), `--rerecord-stale`/`--concurrency <N>` (batch re-recording), `--assert-from <scenario.yaml>`/`--reassert` (re-check the on-disk `assert:` instead of the frozen one), `--strict`/`--fail-on-skill-drift` (staleness handling on replay), `--no-redact` (skip record-time redaction), `--allow-failing` (relax the post-run verdict gate), `--dry-run` (resolve without recording). Full flag list: `cowork-harness record --help` / `replay --help`. | **token-free, Docker-free CI** from a once-recorded run |
|
|
310
|
-
| `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain/path/machine-inventory → exit 1; whole-token allows via `--allow
|
|
310
|
+
| `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain/path/machine-inventory → exit 1; whole-token allows via `--allow <regex>` (a pattern) / class-scoped `--allow-domain` / `--allow-email` / `--allow-path` / `--allow-machine-inventory` / `--allow-patterns-file <path>` (a **file** of patterns, one regex per line)) + a staleness check (`--skip-privacy` or `--skip-staleness` to run only one half). A dir argument scans `*.cassette.json` in that dir only (**non-recursive**) | gating **committed cassettes** against PII leaks + "edited the skill, forgot to re-record" |
|
|
311
311
|
| `verify-run <run-dir> <scenario.yaml>` | Re-evaluate a scenario's `assert:` (and, when the scenario declares `answers:`, whether they still match the run's actual gates) against an already-kept run dir — **no live agent, no tokens, no Docker** (~1s) | iterating on a wrong assertion or a drifted `answer` without a full live re-record |
|
|
312
312
|
| `trace <run-id>` | Digest a run's `events.jsonl` (`--view tools\|questions\|dispatches`; legacy `--tools` / `--gates` / `--dispatches` aliases still accepted) | "how many sub-agents *actually* dispatched, and which?" |
|
|
313
313
|
| `inspect <run-id>` | Show what a run **produced**: the artifacts + a shallow field preview of each JSON artifact (`--output-format json` for a digest). Works on a salvaged partial run too | "did it do the job?" — without hand-parsing `…/mnt/outputs/…` |
|
package/SPEC.md
CHANGED
|
@@ -384,7 +384,7 @@ states in `baseline.provenance.gates`). A skill that ignores these behaves diffe
|
|
|
384
384
|
`tool_*`, `subagent_*`, `dispatch_count_max`, `skill_triggered`, `no_skill_triggered`, `max_cost_usd`,
|
|
385
385
|
`max_tokens`, `tool_calls_max`, `max_turns`, `result`, the verdict modifiers (`allow_permissive_auto_allow`,
|
|
386
386
|
`allow_missing_capability`, `allow_l0_plugin_divergence`, `allow_stall`), and (when `controlOut` is present)
|
|
387
|
-
`question_asked`, `questions_count_max`, `gate_answers_delivered`. `max_cost_usd`/`max_tokens` are
|
|
387
|
+
`question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min`. `max_cost_usd`/`max_tokens` are
|
|
388
388
|
evaluated against the *frozen recording's* usage/cost on replay, not fresh spend; `tool_calls_max`/
|
|
389
389
|
`max_turns` are meaningfully replay-checkable (the re-drive recomputes both deterministically).
|
|
390
390
|
- **Filesystem assertions** (`file_exists`, `user_visible_artifact`, `artifact_json`, `computer_links_resolve`,
|
|
@@ -405,10 +405,21 @@ green replay does not imply the recording is still valid. Each finding is surfac
|
|
|
405
405
|
`RunResult.staleness[]` for a token-free gate to act on. `replay --strict` fails on any class;
|
|
406
406
|
`replay --fail-on-skill-drift` fails only on skill-source classes (`skill` / `shared-root` /
|
|
407
407
|
`unverifiable-skill`). Both realize the gate as failing `assertions[]` entries (so `ok`/exit stay consistent).
|
|
408
|
-
- **`question_asked` / `questions_count_max` / `gate_answers_delivered
|
|
409
|
-
`controlOut`. Without it, a loud `::warning::` fires and these keys are
|
|
410
|
-
passed). The authoritative list is `contentKeys` in `src/run/cassette.ts`;
|
|
411
|
-
it — consult it for the full table.
|
|
408
|
+
- **`question_asked` / `questions_count_max` / `gate_answers_delivered` / `gate_answer_count_min`**
|
|
409
|
+
additionally require `controlOut`. Without it, a loud `::warning::` fires and these keys are
|
|
410
|
+
excluded (not vacuously passed). The authoritative list is `contentKeys` in `src/run/cassette.ts`;
|
|
411
|
+
`docs/cassette.md` mirrors it — consult it for the full table.
|
|
412
|
+
- **`gate_answers_delivered: true` passes vacuously when zero `AskUserQuestion` gates fired** (gate
|
|
413
|
+
firing is model-dependent). Pair it with `gate_answer_count_min: <N>` to also require that at least
|
|
414
|
+
N gates fired AND were delivered non-error — the presence companion, mirroring
|
|
415
|
+
`computer_links_resolve` (zero-passes) paired with `transcript_contains` (presence). Both keys fail
|
|
416
|
+
evidence-unavailable, never vacuous-pass, when gate-delivery telemetry itself is absent from
|
|
417
|
+
`result.json` (an old/partial run on the verify-run lane).
|
|
418
|
+
- **`questions_count_max` counts sub-questions, not `AskUserQuestion` tool calls/gates.** A bundled
|
|
419
|
+
gate with K sub-questions counts as K (`src/run/run.ts`'s recorder pushes one `rec.questions` entry
|
|
420
|
+
per sub-question; `src/assert.ts` compares against that count). `trace --view questions` shows the
|
|
421
|
+
same per-gate sub-question count and a matching footer total, so the two surfaces agree — see
|
|
422
|
+
`docs/cassette.md` / `docs/scenario.md`'s `questions_count_max` row.
|
|
412
423
|
|
|
413
424
|
**Assertion source (replay):** by default `replay` evaluates the `assert:` block **frozen in the cassette**
|
|
414
425
|
— byte-deterministic and independent of the working tree. When a sibling scenario resolves and its `assert:`
|
package/dist/assert.js
CHANGED
|
@@ -471,10 +471,13 @@ function check(a, ctx) {
|
|
|
471
471
|
// run/cassette, an unobserved delivery (delivered=null) is NOT neutral — it is absence of the
|
|
472
472
|
// evidence the assertion requires, so it fails loud ("no silent false-greens"). `delivered:
|
|
473
473
|
// false` is a real errored tool_result; `null` is "no tool_result observed for this gate".
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
474
|
+
// Zero gates fired passes vacuously (whether a gate fires is model-dependent) — pair with
|
|
475
|
+
// gate_answer_count_min to also require presence. Missing telemetry (gateDeliveriesMissing)
|
|
476
|
+
// is NOT the same as zero gates and must fail evidence-unavailable, not vacuous-pass.
|
|
477
|
+
if (ctx.gateDeliveriesMissing) {
|
|
478
|
+
results.push(fail(`evidence unavailable: gate-delivery telemetry absent from result.json — cannot evaluate gate_answers_delivered`));
|
|
479
|
+
}
|
|
480
|
+
else if (a.gate_answers_delivered) {
|
|
478
481
|
const bad = ctx.gateDeliveries.filter((g) => g.delivered !== true);
|
|
479
482
|
results.push(bad.length === 0
|
|
480
483
|
? ok()
|
|
@@ -492,6 +495,17 @@ function check(a, ctx) {
|
|
|
492
495
|
results.push(failedConfirmed.length > 0 ? ok() : fail(`expected a confirmed gate-delivery failure but none was observed`));
|
|
493
496
|
}
|
|
494
497
|
}
|
|
498
|
+
if (a.gate_answer_count_min !== undefined) {
|
|
499
|
+
if (ctx.gateDeliveriesMissing) {
|
|
500
|
+
results.push(fail(`evidence unavailable: gate-delivery telemetry absent from result.json — cannot evaluate gate_answer_count_min`));
|
|
501
|
+
}
|
|
502
|
+
else {
|
|
503
|
+
const delivered = ctx.gateDeliveries.filter((g) => g.delivered === true).length;
|
|
504
|
+
results.push(delivered >= a.gate_answer_count_min
|
|
505
|
+
? ok()
|
|
506
|
+
: fail(`only ${delivered} gate answer(s) confirmed delivered, need ≥ ${a.gate_answer_count_min}`));
|
|
507
|
+
}
|
|
508
|
+
}
|
|
495
509
|
if (a.artifact_json !== undefined) {
|
|
496
510
|
const aj = a.artifact_json;
|
|
497
511
|
const file = containedPath(ctx.workRoot, aj.artifact);
|
|
@@ -500,12 +514,33 @@ function check(a, ctx) {
|
|
|
500
514
|
else {
|
|
501
515
|
// verify the real path (after symlink resolution) is still under workRoot.
|
|
502
516
|
const realFile = containedRealPath(ctx.workRoot, file);
|
|
517
|
+
// A body-less manifest entry (a read-only connected-folder input, or an artifact over the body
|
|
518
|
+
// cap) has no content in the cassette — artifact_json cannot be evaluated on replay (the 0-byte
|
|
519
|
+
// placeholder isn't parseable). To keep record/verify-run/replay SYMMETRIC (no green-record →
|
|
520
|
+
// red-replay), treat such a target as evidence-unavailable on EVERY lane: `truncatedPaths` flags
|
|
521
|
+
// it on replay; `readonlyFolderRoots` flags the read-only-input case on the live/verify-run lanes
|
|
522
|
+
// where the real file is still on disk. (Existence keys stay green — existence is provable from
|
|
523
|
+
// the recorded hash — but content is genuinely absent, so this fails loud, never vacuous.)
|
|
524
|
+
const rel = relative(resolve(ctx.workRoot), file);
|
|
525
|
+
const isReadonlyInput = (ctx.readonlyFolderRoots ?? []).some((pre) => rel === pre || rel.startsWith(pre + "/"));
|
|
526
|
+
const bodyLess = truncated.has(rel) || isReadonlyInput;
|
|
503
527
|
if (!realFile) {
|
|
504
528
|
results.push(fail(`unsafe artifact_json path "${aj.artifact}" — symlink target escapes the work root`));
|
|
505
529
|
}
|
|
506
530
|
else if (!existsSync(realFile)) {
|
|
507
531
|
results.push(fail(`artifact_json: file not found: ${aj.artifact} (under ${ctx.workRoot})`));
|
|
508
532
|
}
|
|
533
|
+
else if (bodyLess) {
|
|
534
|
+
// Every lane knows WHY the entry is body-less, so the remedy is always precise: live/verify-run
|
|
535
|
+
// derive `readonlyFolderRoots` from the plan; replay reads the set persisted on the cassette
|
|
536
|
+
// (v7). A read-only input can't be captured (assert on a deliverable); an over-cap artifact can
|
|
537
|
+
// (raise the cap). `isReadonlyInput` distinguishes them on all three lanes.
|
|
538
|
+
results.push(fail(`evidence unavailable: artifact_json target "${aj.artifact}" was captured body-less ` +
|
|
539
|
+
(isReadonlyInput
|
|
540
|
+
? `(read-only connected-folder input — its content is never captured; assert artifact_json on a deliverable instead)`
|
|
541
|
+
: `(larger than the artifact-body cap — raise --max-artifact-bytes to capture it)`) +
|
|
542
|
+
` — content is not in the cassette, so it cannot be evaluated on replay`));
|
|
543
|
+
}
|
|
509
544
|
else {
|
|
510
545
|
let doc;
|
|
511
546
|
let parsed = true;
|
package/dist/cli.js
CHANGED
|
@@ -91,7 +91,8 @@ const HELP = `cowork-harness <command> (v${"$VERSION"})
|
|
|
91
91
|
[--output-format json]
|
|
92
92
|
verify-cassettes <file|dir> CI gate (no token): privacy scan + staleness — exit 1 on finding or drift
|
|
93
93
|
[--skip-privacy|--skip-staleness] skip one check
|
|
94
|
-
[--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-file <path>]... [--output-format json]
|
|
94
|
+
[--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-patterns-file <path>]... [--output-format json]
|
|
95
|
+
--allow <regex> is a PATTERN (matched against a finding); --allow-patterns-file <path> is a FILE of patterns, one regex per line — not a path to allow
|
|
95
96
|
rehash <dir/> migrate cassette fingerprints to current version when content is provably unchanged (requires contentSig from v3+)
|
|
96
97
|
init-redact [--force] copy the packaged reference .cowork-redact.json into the cwd (redaction starter
|
|
97
98
|
for hostloop/protocol recordings; review + tailor the patterns before recording)
|
|
@@ -342,7 +343,8 @@ const SUBCOMMAND_USAGE = {
|
|
|
342
343
|
replay: "usage: replay <file.cassette.json | dir/> [--strict] [--fail-on-skill-drift] [--assert-from <scenario.yaml> | --reassert] [--output-format text|json]\n" +
|
|
343
344
|
" by default the assertions FROZEN in the cassette drive the verdict (deterministic); a sibling scenario whose assert: differs only prints a notice.\n" +
|
|
344
345
|
" --assert-from <file> / --reassert: token-free re-check against the on-disk assert:/expect_denied: — recording-shaping drift (prompt/answers/baseline/skills) and skill staleness HARD-FAIL.",
|
|
345
|
-
"verify-cassettes": "usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-file <path>]... [--output-format json]"
|
|
346
|
+
"verify-cassettes": "usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-patterns-file <path>]... [--output-format json]\n" +
|
|
347
|
+
" --allow <regex> is a PATTERN (matched against a finding); --allow-patterns-file <path> is a FILE of patterns, one regex per line — not a path to allow.",
|
|
346
348
|
trace: "usage: trace <run-id | run-dir | events.jsonl> [--view tools|questions|dispatches] [--translate-paths] [--output-format json]\n --view tools tool call / result rows\n --view questions gate lifecycle (question → answer → delivered)\n --view dispatches sub-agent dispatch tree + dispatch_count_max\n --translate-paths rewrite VM paths to host paths in the tools/default TEXT views only (needs a sibling mounts.json + an effective hostloop run; questions/dispatches views and --output-format json are unaffected)\n (default: all views)\n (for what the run PRODUCED — artifacts — use `inspect`)",
|
|
347
349
|
assertions: "usage: assertions --list [--output-format json]",
|
|
348
350
|
scaffold: "usage: scaffold <run-id | run-dir> [--out <file.yaml>] [--output-format text|json]\n Turns a kept run into a starter scenario YAML (gates→answers, artifacts→file_exists).\n Positional <run-id | run-dir> is the canonical form.",
|
|
@@ -2788,6 +2790,9 @@ async function cmdVerifyRun(args) {
|
|
|
2788
2790
|
// Read the roots persisted at run time (folder mount names are dynamic/gated, not a fixed prefix).
|
|
2789
2791
|
// Fall back to the legacy prefix for old result.json that predates the field.
|
|
2790
2792
|
userVisiblePrefixes: result.userVisibleRoots ?? ["outputs", ".projects"],
|
|
2793
|
+
// Read-only folder inputs are captured body-less; keep artifact_json's verdict identical to the
|
|
2794
|
+
// replay lane (evidence-unavailable) instead of parsing the real on-disk input here.
|
|
2795
|
+
readonlyFolderRoots: result.readonlyFolderRoots ?? [],
|
|
2791
2796
|
// result.json is the single source: every writer populates the field from the run's own
|
|
2792
2797
|
// pre-run-manifest.json, so a missing field means the baseline genuinely doesn't exist
|
|
2793
2798
|
// (pre-field run, or the run never captured) — evidence-unavailable, loud.
|
|
@@ -2798,6 +2803,7 @@ async function cmdVerifyRun(args) {
|
|
|
2798
2803
|
selfHealRan: scan.selfHealRan,
|
|
2799
2804
|
subagents: result.subagents ?? [],
|
|
2800
2805
|
gateDeliveries: result.gateDeliveries ?? [],
|
|
2806
|
+
gateDeliveriesMissing: result.gateDeliveries === undefined,
|
|
2801
2807
|
toolResultTexts: (result.toolResults ?? []).map((r) => r.assertText ?? r.text),
|
|
2802
2808
|
toolResultsTruncated: (result.toolResults ?? []).map((r) => r.assertText === undefined),
|
|
2803
2809
|
transcriptMissing: sidecarTranscript === null,
|
package/dist/run/cassette.js
CHANGED
|
@@ -133,8 +133,14 @@ function isLosslessUtf8(buf) {
|
|
|
133
133
|
}
|
|
134
134
|
/** Snapshot the user-visible artifacts under `workRoot` into manifest entries.
|
|
135
135
|
* Exported for token-free record→replay round-trip tests. */
|
|
136
|
-
export function buildManifest(workRoot, cap, roots = ["outputs", ".projects"]) {
|
|
136
|
+
export function buildManifest(workRoot, cap, roots = ["outputs", ".projects"], bodyLessPrefixes = []) {
|
|
137
137
|
const limit = cap ?? defaultBodyCap();
|
|
138
|
+
// Read-only connected-folder inputs (`bodyLessPrefixes`) are captured path+bytes+sha256 only, same as
|
|
139
|
+
// an over-cap entry — no body, so no bloat and no `binary` privacy finding (cassette.ts binary scan
|
|
140
|
+
// only fires on a committed base64 body). The manifest entry SURVIVES (unlike full exclusion) so
|
|
141
|
+
// `materializeManifest` writes a 0-byte placeholder and `computer_links_resolve` resolves identically
|
|
142
|
+
// on live and replay (see T3 in the pre-1.0 fix plan).
|
|
143
|
+
const isBodyLess = (path) => bodyLessPrefixes.some((prefix) => path === prefix || path.startsWith(prefix + "/"));
|
|
138
144
|
return collectArtifacts(workRoot, roots).map(({ path, bytes }) => {
|
|
139
145
|
// collectArtifacts paths are derived from a directory walk; even though it already skips symlinks,
|
|
140
146
|
// an entry must not inline content outside the work root. Re-confirm containment before reading the body.
|
|
@@ -153,7 +159,7 @@ export function buildManifest(workRoot, cap, roots = ["outputs", ".projects"]) {
|
|
|
153
159
|
return { path, bytes, sha256: "", truncated: true };
|
|
154
160
|
}
|
|
155
161
|
const sha256 = createHash("sha256").update(buf).digest("hex");
|
|
156
|
-
if (buf.length > limit)
|
|
162
|
+
if (buf.length > limit || isBodyLess(path))
|
|
157
163
|
return { path, bytes, sha256, truncated: true };
|
|
158
164
|
// store an encoding marker. UTF-8-safe bodies stay text (readable cassettes); binary bodies go
|
|
159
165
|
// base64 so the record→replay round-trip is byte-exact and the sha256 verify stays valid.
|
|
@@ -379,11 +385,12 @@ export function scanCassette(cassette, allow) {
|
|
|
379
385
|
// zip/DEFLATE streams. Count a COMMITTED binary body as a real finding (cls "binary", NOT the
|
|
380
386
|
// benign "unscanned" used for a TRUNCATED/uncommitted entry below) so the gate can't greenlight
|
|
381
387
|
// raw recoverable PII. Recourse: after reviewing the deliverable, clear it with
|
|
382
|
-
// `--allow <path-regex>` (matched on the artifact path above, since the body is
|
|
388
|
+
// `--allow <path-regex>` (a PATTERN matched on the artifact path above, since the body is
|
|
389
|
+
// unreadable) — NOT `--allow-patterns-file`, which loads a FILE of patterns, not a path to allow.
|
|
383
390
|
findings.push({
|
|
384
391
|
where: `artifact ${a.path}`,
|
|
385
392
|
cls: "binary",
|
|
386
|
-
sample: `(committed binary body — not text-scannable; review and clear with --allow ${a.path})`,
|
|
393
|
+
sample: `(committed binary body — not text-scannable; review and clear with --allow ${a.path} (a pattern matched on this path); note --allow-patterns-file is a FILE of patterns, not this path)`,
|
|
387
394
|
});
|
|
388
395
|
}
|
|
389
396
|
}
|
|
@@ -407,6 +414,9 @@ export function scanCassette(cassette, allow) {
|
|
|
407
414
|
// cassettes`. Scan them too, prefixed `metadata:` so a reviewer knows redaction here ALSO rewrites
|
|
408
415
|
// structural paths, distinct from free-text findings in the transcript/deliverable.
|
|
409
416
|
(cassette.userVisibleRoots ?? []).forEach((r, i) => findings.push(...scanText(r, `metadata:userVisibleRoots[${i}]`, allow, FULL)));
|
|
417
|
+
// readonlyFolderRoots is a subset of userVisibleRoots (same strings), but it's a separately-persisted
|
|
418
|
+
// field — scan it too so a reviewer can't be surprised by an unscanned persisted string.
|
|
419
|
+
(cassette.readonlyFolderRoots ?? []).forEach((r, i) => findings.push(...scanText(r, `metadata:readonlyFolderRoots[${i}]`, allow, FULL)));
|
|
410
420
|
findings.push(...scanText(cassette.scenario.name ?? "", "metadata:scenario.name", allow, FULL));
|
|
411
421
|
findings.push(...scanText(cassette.scenario.session ?? "", "metadata:scenario.session", allow, FULL));
|
|
412
422
|
if (cassette.scenarioSource)
|
|
@@ -1014,10 +1024,15 @@ export function redactCassette(cassette, policy) {
|
|
|
1014
1024
|
}
|
|
1015
1025
|
}
|
|
1016
1026
|
const redactedPreRunPaths = cassette.preRunPaths?.map((p) => redactText(p, policy));
|
|
1027
|
+
// readonlyFolderRoots is a subset of userVisibleRoots; redact it with the same context-free redactText
|
|
1028
|
+
// so the prefix relationship with the (redacted) artifact paths is preserved. The `...cassette` spread
|
|
1029
|
+
// would otherwise carry the UNREDACTED set through.
|
|
1030
|
+
const redactedReadonly = cassette.readonlyFolderRoots?.map((r) => redactText(r, policy));
|
|
1017
1031
|
return {
|
|
1018
1032
|
...cassette,
|
|
1019
1033
|
scenario,
|
|
1020
1034
|
userVisibleRoots: redactedRoots,
|
|
1035
|
+
readonlyFolderRoots: redactedReadonly,
|
|
1021
1036
|
artifacts: redactedArtifacts,
|
|
1022
1037
|
preRunPaths: redactedPreRunPaths,
|
|
1023
1038
|
events: cassette.events.map((l) => redactJsonLine(l, policy)),
|
|
@@ -1299,14 +1314,18 @@ export function redactionPreflightMessage(items) {
|
|
|
1299
1314
|
`COWORK_HARNESS_REDACT_PATTERNS. (The always-on privacy scanner remains the universal net — ` +
|
|
1300
1315
|
`container-tier recordings can trip it too.)\n`);
|
|
1301
1316
|
}
|
|
1302
|
-
/** Return the `artifact_json.artifact` paths a scenario asserts that ended up
|
|
1303
|
-
*
|
|
1304
|
-
*
|
|
1305
|
-
*
|
|
1306
|
-
|
|
1317
|
+
/** Return the `artifact_json.artifact` paths a scenario asserts that ended up truncated by SIZE (body
|
|
1318
|
+
* >cap, hash-only) — the genuine green-record/red-replay case whose remedy is "raise the cap". This
|
|
1319
|
+
* EXCLUDES read-only connected-folder inputs: those are body-less by policy (not size), raising the cap
|
|
1320
|
+
* can't capture them, and `artifact_json` against one already fails LOUD-and-symmetrically at record,
|
|
1321
|
+
* verify-run, and replay (evidence-unavailable, see assert.ts) — so they are neither an asymmetry nor a
|
|
1322
|
+
* "too large" problem. Paths are normalized through `resolve` so `./outputs/x.json` and `outputs/x.json`
|
|
1323
|
+
* join cleanly against the manifest's walk paths. */
|
|
1324
|
+
export function artifactJsonTargetsTruncated(scenario, workRoot, artifacts, readonlyFolderRoots = []) {
|
|
1325
|
+
const isReadonly = (p) => readonlyFolderRoots.some((pre) => p === pre || p.startsWith(pre + "/"));
|
|
1307
1326
|
const truncatedAbs = new Set();
|
|
1308
1327
|
for (const a of artifacts)
|
|
1309
|
-
if (a.truncated)
|
|
1328
|
+
if (a.truncated && !isReadonly(a.path))
|
|
1310
1329
|
truncatedAbs.add(resolve(workRoot, a.path));
|
|
1311
1330
|
if (truncatedAbs.size === 0)
|
|
1312
1331
|
return [];
|
|
@@ -1805,7 +1824,10 @@ async function recordScenarioObject(scenario, opts, extraPolicyDirs = []) {
|
|
|
1805
1824
|
// Snapshot under the run's REAL user-visible roots (outputs + resolved folder mount names), persisted
|
|
1806
1825
|
// below as cassette.userVisibleRoots so replay matches — not the legacy hardcoded `.projects/` prefix.
|
|
1807
1826
|
const recordRoots = result.userVisibleRoots ?? ["outputs", ".projects"];
|
|
1808
|
-
|
|
1827
|
+
// Read-only connected-folder inputs are captured body-less (path + sha256, no body) — see buildManifest's
|
|
1828
|
+
// `bodyLessPrefixes` doc comment. `recordRoots`/`cassette.userVisibleRoots` stay the FULL set; only the
|
|
1829
|
+
// captured bodies under these prefixes are stripped.
|
|
1830
|
+
const artifacts = (result.workDir ? buildManifest(result.workDir, opts.maxArtifactBytes, recordRoots, result.readonlyFolderRoots ?? []) : []).map((a) => {
|
|
1809
1831
|
if (a.body === undefined)
|
|
1810
1832
|
return a;
|
|
1811
1833
|
if (a.encoding === "base64") {
|
|
@@ -1846,7 +1868,7 @@ async function recordScenarioObject(scenario, opts, extraPolicyDirs = []) {
|
|
|
1846
1868
|
// replay (no committed body). Surface that record→replay asymmetry NOW, at its cause, instead of letting a
|
|
1847
1869
|
// green record produce a red replay in CI. Honor --allow-failing (warn, don't block) like the verdict gate.
|
|
1848
1870
|
if (result.workDir) {
|
|
1849
|
-
const truncatedAsserted = artifactJsonTargetsTruncated(scenario, result.workDir, artifacts);
|
|
1871
|
+
const truncatedAsserted = artifactJsonTargetsTruncated(scenario, result.workDir, artifacts, result.readonlyFolderRoots ?? []);
|
|
1850
1872
|
if (truncatedAsserted.length) {
|
|
1851
1873
|
const cap = opts.maxArtifactBytes ?? defaultBodyCap();
|
|
1852
1874
|
const msg = `artifact_json asserts artifact(s) too large to commit (>${cap} B, stored hash-only): ${truncatedAsserted.join(", ")} — ` +
|
|
@@ -1868,6 +1890,9 @@ async function recordScenarioObject(scenario, opts, extraPolicyDirs = []) {
|
|
|
1868
1890
|
effectiveFidelity: result.effectiveFidelity,
|
|
1869
1891
|
artifacts,
|
|
1870
1892
|
userVisibleRoots: recordRoots,
|
|
1893
|
+
// Persist the read-only subset ONLY when non-empty (keeps cassettes without connected folders
|
|
1894
|
+
// clean); replay reads it to distinguish a body-less read-only input from an over-cap artifact.
|
|
1895
|
+
...(result.readonlyFolderRoots?.length ? { readonlyFolderRoots: result.readonlyFolderRoots } : {}),
|
|
1871
1896
|
// co-present with userVisibleRoots by construction (see the Cassette field comment) — a cassette
|
|
1872
1897
|
// carrying preRunPaths never hits replay's legacy-roots fallback.
|
|
1873
1898
|
preRunPaths: result.preRunPaths,
|
|
@@ -1956,7 +1981,7 @@ function warnUncheckableOnDiskKeys(cassette, frozen, onDisk) {
|
|
|
1956
1981
|
// Reuse the exported classification constant — this local copy drifted once already (it was
|
|
1957
1982
|
// missing computer_links_resolve), silently suppressing the on-disk warning for that key.
|
|
1958
1983
|
const manifestKeys = new Set(MANIFEST_KEYS);
|
|
1959
|
-
const gateKeys = new Set(["question_asked", "questions_count_max", "gate_answers_delivered"]);
|
|
1984
|
+
const gateKeys = new Set(["question_asked", "questions_count_max", "gate_answers_delivered", "gate_answer_count_min"]);
|
|
1960
1985
|
const liveOnlyKeys = new Set([
|
|
1961
1986
|
"egress_denied",
|
|
1962
1987
|
"egress_allowed",
|
|
@@ -2179,9 +2204,9 @@ export function cmdVerifyCassettes(args) {
|
|
|
2179
2204
|
p = parseArgs(args, {
|
|
2180
2205
|
booleans: ["--skip-privacy", "--skip-staleness", "--quiet", "--verbose"],
|
|
2181
2206
|
values: ["--output-format"],
|
|
2182
|
-
repeated: ["--allow", "--allow-domain", "--allow-email", "--allow-path", "--allow-machine-inventory", "--allow-file"],
|
|
2207
|
+
repeated: ["--allow", "--allow-domain", "--allow-email", "--allow-path", "--allow-machine-inventory", "--allow-patterns-file"],
|
|
2183
2208
|
enums: { "--output-format": ["text", "json"] },
|
|
2184
|
-
noDashValue: ["--allow-file"],
|
|
2209
|
+
noDashValue: ["--allow-patterns-file"],
|
|
2185
2210
|
aliases: { "-q": "--quiet" },
|
|
2186
2211
|
});
|
|
2187
2212
|
}
|
|
@@ -2198,10 +2223,13 @@ export function cmdVerifyCassettes(args) {
|
|
|
2198
2223
|
}
|
|
2199
2224
|
const doPrivacy = !skipPrivacy;
|
|
2200
2225
|
const doStaleness = !skipStaleness;
|
|
2201
|
-
// Allow model: each entry is whole-token anchored + class-scoped. A bare `--allow
|
|
2202
|
-
// class (back-compat); `--allow-domain`/`--allow-email`/`--allow-path
|
|
2203
|
-
// scope to one class so a domain allow can't bleed into the
|
|
2204
|
-
//
|
|
2226
|
+
// Allow model: each entry is whole-token anchored + class-scoped. A bare `--allow <regex>` is a single
|
|
2227
|
+
// PATTERN applied to every class (back-compat); `--allow-domain`/`--allow-email`/`--allow-path`/
|
|
2228
|
+
// `--allow-machine-inventory` scope a pattern to one class so a domain allow can't bleed into the
|
|
2229
|
+
// email tripwire. `--allow-patterns-file <path>` is a different thing: it loads bare (all-class)
|
|
2230
|
+
// patterns from a version-controlled FILE of patterns, one regex per line, `#` comments and blanks
|
|
2231
|
+
// ignored — not "allow this file" (the flag does not accept a path to allow, it accepts a path to a
|
|
2232
|
+
// patterns list).
|
|
2205
2233
|
const allow = [];
|
|
2206
2234
|
const addAllow = (src, cls, flag) => {
|
|
2207
2235
|
try {
|
|
@@ -2222,24 +2250,25 @@ export function cmdVerifyCassettes(args) {
|
|
|
2222
2250
|
addAllow(src, "path", "--allow-path");
|
|
2223
2251
|
for (const src of p.repeated["--allow-machine-inventory"] ?? [])
|
|
2224
2252
|
addAllow(src, "machine-inventory", "--allow-machine-inventory");
|
|
2225
|
-
for (const file of p.repeated["--allow-file"] ?? []) {
|
|
2253
|
+
for (const file of p.repeated["--allow-patterns-file"] ?? []) {
|
|
2226
2254
|
let body;
|
|
2227
2255
|
try {
|
|
2228
2256
|
body = readFileSync(file, "utf8");
|
|
2229
2257
|
}
|
|
2230
2258
|
catch (e) {
|
|
2231
|
-
log(`--allow-file: cannot read ${file}: ${e.message}`);
|
|
2259
|
+
log(`--allow-patterns-file: cannot read ${file}: ${e.message}`);
|
|
2232
2260
|
return process.exit(2);
|
|
2233
2261
|
}
|
|
2234
2262
|
for (const raw of body.split("\n")) {
|
|
2235
2263
|
const line = raw.trim();
|
|
2236
2264
|
if (line && !line.startsWith("#"))
|
|
2237
|
-
addAllow(line, undefined, `--allow-file (${file})`);
|
|
2265
|
+
addAllow(line, undefined, `--allow-patterns-file (${file})`);
|
|
2238
2266
|
}
|
|
2239
2267
|
}
|
|
2240
2268
|
const target = p.positionals[0];
|
|
2241
2269
|
if (!target) {
|
|
2242
|
-
log("usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-file <path>]... [--output-format json]");
|
|
2270
|
+
log("usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-path <regex>]... [--allow-machine-inventory <regex>]... [--allow-patterns-file <path>]... [--output-format json]");
|
|
2271
|
+
log(" --allow <regex> is a PATTERN (matched against a finding); --allow-patterns-file <path> is a FILE of patterns, one regex per line — not a path to allow.");
|
|
2243
2272
|
return process.exit(2);
|
|
2244
2273
|
}
|
|
2245
2274
|
if (p.positionals.length > 1) {
|
|
@@ -2541,7 +2570,12 @@ export const ALWAYS_CONTENT_KEYS = [
|
|
|
2541
2570
|
...VERDICT_MODIFIER_KEYS,
|
|
2542
2571
|
];
|
|
2543
2572
|
/** Assertion keys evaluated on replay only when `controlOut` (full-fidelity) is present. */
|
|
2544
|
-
export const QUESTION_GATE_KEYS = [
|
|
2573
|
+
export const QUESTION_GATE_KEYS = [
|
|
2574
|
+
"question_asked",
|
|
2575
|
+
"questions_count_max",
|
|
2576
|
+
"gate_answers_delivered",
|
|
2577
|
+
"gate_answer_count_min",
|
|
2578
|
+
];
|
|
2545
2579
|
/** Assertion keys evaluated on replay only when the cassette carries an `artifacts` manifest.
|
|
2546
2580
|
* `computer_links_resolve` joins this bucket (NOT ALWAYS_CONTENT_KEYS): resolving a NON-empty link set
|
|
2547
2581
|
* needs either a live filesystem (not available on replay) or the cassette's `artifacts` manifest — the
|
|
@@ -2746,6 +2780,9 @@ export async function replayCassette(cassette, hooks = [], opts = {}) {
|
|
|
2746
2780
|
result: rec.result,
|
|
2747
2781
|
workRoot: replayWorkRoot,
|
|
2748
2782
|
userVisiblePrefixes: replayPrefixes,
|
|
2783
|
+
// Persisted on the cassette (v7 optional metadata) so replay tells a body-less read-only input
|
|
2784
|
+
// apart from an over-cap artifact and gives the precise artifact_json remediation.
|
|
2785
|
+
readonlyFolderRoots: cassette.readonlyFolderRoots ?? [],
|
|
2749
2786
|
preRunPaths: cassette.preRunPaths,
|
|
2750
2787
|
outputsDeletes: [],
|
|
2751
2788
|
questions: rec.questions,
|
package/dist/run/execute.js
CHANGED
|
@@ -15,7 +15,7 @@ import { writeRunningStatus, startStatusTicker, registerRunForCrashSafety, statu
|
|
|
15
15
|
// the cycle is intrinsic — kept runtime-only rather than refactored.
|
|
16
16
|
import { buildFingerprint } from "./cassette.js";
|
|
17
17
|
import { loadBaseline } from "../baseline.js";
|
|
18
|
-
import { loadSession, resolveSessionPaths, buildLaunchPlan, userVisibleRootsFromPlan } from "../session.js";
|
|
18
|
+
import { loadSession, resolveSessionPaths, buildLaunchPlan, userVisibleRootsFromPlan, readonlyFolderRootsFromPlan } from "../session.js";
|
|
19
19
|
import { spawnProtocol } from "../runtime/protocol.js";
|
|
20
20
|
import { spawnContainer } from "../runtime/container.js";
|
|
21
21
|
import { spawnHostLoop } from "../runtime/hostloop.js";
|
|
@@ -588,6 +588,13 @@ export async function executeScenario(scenario, opts = {}) {
|
|
|
588
588
|
// (verify reads result.json; replay reads the cassette) match this without rebuilding a LaunchPlan.
|
|
589
589
|
// Shared with the pre-run baseline walk (userVisibleRootsFromPlan) — pre and post MUST agree.
|
|
590
590
|
const userVisibleRoots = userVisibleRootsFromPlan(plan);
|
|
591
|
+
// Read-only (`mode: "r"`) connected-folder roots — inputs, not deliverables. Persisted so the cassette
|
|
592
|
+
// recorder strips their captured BODIES (fidelity/no-bloat) and `RunResult.artifacts` excludes them
|
|
593
|
+
// outright (an input is not a `file_exists` target). Does NOT change `userVisibleRoots` above.
|
|
594
|
+
const readonlyFolderRoots = readonlyFolderRootsFromPlan(plan);
|
|
595
|
+
// artifacts (the ENV-MANIFEST) excludes read-only inputs — a deliverable is something the agent wrote,
|
|
596
|
+
// not a pre-existing input it only read.
|
|
597
|
+
const captureRoots = userVisibleRoots.filter((r) => !readonlyFolderRoots.includes(r));
|
|
591
598
|
// Read the pre-run baseline ONCE: the evaluate ctx and the persisted RunResult must see the same
|
|
592
599
|
// value — two reads could disagree if the file were touched mid-run.
|
|
593
600
|
const preRunPaths = readPreRunManifest(outDir);
|
|
@@ -605,6 +612,7 @@ export async function executeScenario(scenario, opts = {}) {
|
|
|
605
612
|
outDir,
|
|
606
613
|
workRoot,
|
|
607
614
|
userVisibleRoots,
|
|
615
|
+
readonlyFolderRoots,
|
|
608
616
|
effectiveFidelity,
|
|
609
617
|
egress,
|
|
610
618
|
durationMs: Date.now() - startedAt,
|
|
@@ -633,6 +641,9 @@ export async function executeScenario(scenario, opts = {}) {
|
|
|
633
641
|
result: record.result,
|
|
634
642
|
workRoot,
|
|
635
643
|
userVisiblePrefixes: userVisibleRoots,
|
|
644
|
+
// Read-only folder inputs are captured body-less; artifact_json must reach the same
|
|
645
|
+
// evidence-unavailable verdict here as on replay (see AssertContext.readonlyFolderRoots).
|
|
646
|
+
readonlyFolderRoots,
|
|
636
647
|
preRunPaths,
|
|
637
648
|
outputsDeletes: scan.outputsDeletes,
|
|
638
649
|
questions: record.questions,
|
|
@@ -779,7 +790,8 @@ export async function executeScenario(scenario, opts = {}) {
|
|
|
779
790
|
workDir: workRoot,
|
|
780
791
|
outputsDir: join(workRoot, "outputs"),
|
|
781
792
|
userVisibleRoots,
|
|
782
|
-
|
|
793
|
+
readonlyFolderRoots,
|
|
794
|
+
artifacts: collectArtifacts(workRoot, captureRoots), // ENV-MANIFEST: observed user-visible files (excl. read-only inputs)
|
|
783
795
|
// The pre-spawn baseline no_unexpected_files diffs against (same single read the evaluate ctx got).
|
|
784
796
|
// undefined = the run didn't capture (key not asserted, microvm, pre-seam) — the assertion then
|
|
785
797
|
// fails evidence-unavailable, loud.
|
|
@@ -946,7 +958,8 @@ export function buildPartialResult(args) {
|
|
|
946
958
|
workDir: args.workRoot,
|
|
947
959
|
outputsDir: join(args.workRoot, "outputs"),
|
|
948
960
|
userVisibleRoots: args.userVisibleRoots,
|
|
949
|
-
|
|
961
|
+
readonlyFolderRoots: args.readonlyFolderRoots,
|
|
962
|
+
artifacts: collectArtifacts(args.workRoot, args.userVisibleRoots.filter((r) => !args.readonlyFolderRoots.includes(r))),
|
|
950
963
|
preRunPaths: readPreRunManifest(args.outDir),
|
|
951
964
|
effectiveFidelity: args.effectiveFidelity,
|
|
952
965
|
...(() => {
|
package/dist/run/trace-view.js
CHANGED
|
@@ -261,6 +261,11 @@ export function buildTrace(file, opts = {}) {
|
|
|
261
261
|
* differing keys: the gate's `decision` (events.jsonl) carries both the UUID `request_id` AND the `toolu_`
|
|
262
262
|
* `toolUseId`; the injected answer lives in `control-out.jsonl` keyed by `request_id`; the delivered result
|
|
263
263
|
* is a `tool_result` keyed by `toolUseId`.
|
|
264
|
+
*
|
|
265
|
+
* Each row is one gate (one AskUserQuestion tool call), which may bundle several sub-questions —
|
|
266
|
+
* `subQuestionCount` carries that count so a reader can reconcile this view against
|
|
267
|
+
* `questions_count_max`, which counts sub-questions, not gates/tool-calls (see `formatGateTrace`'s
|
|
268
|
+
* footer total and `docs/scenario.md`/`docs/cassette.md`'s `questions_count_max` row).
|
|
264
269
|
*/
|
|
265
270
|
export function buildGateTrace(file) {
|
|
266
271
|
const events = eventsOf(file);
|
|
@@ -297,6 +302,7 @@ export function buildGateTrace(file) {
|
|
|
297
302
|
const tr = req.toolUseId ? results.get(req.toolUseId) : undefined;
|
|
298
303
|
rows.push({
|
|
299
304
|
question: req.questions.map((q) => q.question ?? q.header ?? "").join(" / "),
|
|
305
|
+
subQuestionCount: req.questions.length,
|
|
300
306
|
injectedAnswer: answers.get(req.id),
|
|
301
307
|
delivered: tr ? (tr.isError ? "error" : "ok") : "unobserved",
|
|
302
308
|
...(tr?.isError ? { error: tr.text.split("\n")[0].slice(0, 160) } : {}),
|
|
@@ -332,12 +338,16 @@ export function formatGateTrace(rows) {
|
|
|
332
338
|
if (!rows.length)
|
|
333
339
|
return "(no AskUserQuestion gates in this run)";
|
|
334
340
|
const mark = { ok: "✓", error: "✗", unobserved: "?" };
|
|
335
|
-
|
|
336
|
-
.map((r) => {
|
|
341
|
+
const lines = rows.map((r) => {
|
|
337
342
|
const prov = r.answeredBy ? `\n by: ${labelSource(r.answeredBy)}${r.model ? ` (${r.model})` : ""}` : "";
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
.
|
|
343
|
+
// sub-question count is shown only when the gate bundled more than one — reconciles this row
|
|
344
|
+
// against questions_count_max, which counts sub-questions, not gates.
|
|
345
|
+
const subCount = r.subQuestionCount > 1 ? ` (${r.subQuestionCount} sub-questions)` : "";
|
|
346
|
+
return `${mark[r.delivered]} gate "${r.question}"${subCount}\n answered: ${r.injectedAnswer ?? "(none)"}\n delivered: ${r.delivered}${r.error ? ` — ${r.error}` : ""}${prov}`;
|
|
347
|
+
});
|
|
348
|
+
const totalSubQuestions = rows.reduce((sum, r) => sum + r.subQuestionCount, 0);
|
|
349
|
+
lines.push(`\n${rows.length} gate(s), ${totalSubQuestions} sub-question(s) total — questions_count_max counts sub-questions (assert with \`questions_count_max: ${totalSubQuestions}\`)`);
|
|
350
|
+
return lines.join("\n");
|
|
341
351
|
}
|
|
342
352
|
/**
|
|
343
353
|
* `trace --dispatches` — the sub-agent dispatch tree, so an author can read off the REAL total
|
package/dist/session.js
CHANGED
|
@@ -116,6 +116,14 @@ export const SessionConfig = z.strictObject({
|
|
|
116
116
|
export function userVisibleRootsFromPlan(plan) {
|
|
117
117
|
return ["outputs", ...plan.mounts.filter((m) => m.kind === "folder").map((m) => m.mountPath)];
|
|
118
118
|
}
|
|
119
|
+
/** The mount prefixes of read-only (`mode: "r"`) connected folders — inputs, not deliverables. Used to
|
|
120
|
+
* strip captured BODIES from the cassette manifest (fidelity/no-bloat) and to exclude them from
|
|
121
|
+
* `RunResult.artifacts` (an input is not something `scaffold`/`file_exists` should treat as output).
|
|
122
|
+
* Does NOT change `userVisibleRootsFromPlan` — `no_unexpected_files` and `computer_links_resolve`
|
|
123
|
+
* still enumerate these roots; only their captured content changes shape. */
|
|
124
|
+
export function readonlyFolderRootsFromPlan(plan) {
|
|
125
|
+
return plan.mounts.filter((m) => m.kind === "folder" && m.mode === "r").map((m) => m.mountPath);
|
|
126
|
+
}
|
|
119
127
|
/**
|
|
120
128
|
* Read a plugin's declared version from its `.claude-plugin/plugin.json`.
|
|
121
129
|
*
|
package/dist/types.js
CHANGED
|
@@ -241,7 +241,13 @@ export const Assertion = z.object({
|
|
|
241
241
|
gate_answers_delivered: z
|
|
242
242
|
.boolean()
|
|
243
243
|
.optional()
|
|
244
|
-
.describe("every answered AskUserQuestion gate's tool_result was non-error (the answer reached the model)"),
|
|
244
|
+
.describe("every answered AskUserQuestion gate's tool_result was non-error (the answer reached the model); zero gates fired passes vacuously — pair with gate_answer_count_min to also require a gate"),
|
|
245
|
+
gate_answer_count_min: z
|
|
246
|
+
.number()
|
|
247
|
+
.int()
|
|
248
|
+
.nonnegative()
|
|
249
|
+
.optional()
|
|
250
|
+
.describe("at least N AskUserQuestion gates fired AND were delivered non-error (presence companion to gate_answers_delivered)"),
|
|
245
251
|
result: z.enum(["success", "error"]).optional().describe("the run's final result was success | error"),
|
|
246
252
|
allow_permissive_auto_allow: z
|
|
247
253
|
.boolean()
|
package/docs/cassette.md
CHANGED
|
@@ -53,9 +53,11 @@ re-record). `expect_denied`/filesystem/egress keys are sourced but stay live-onl
|
|
|
53
53
|
"events": [ /* JSON lines from events.jsonl (child→driver stdout) */ ],
|
|
54
54
|
"controlOut": [ /* JSON lines from control-out.jsonl (driver→child control_responses) */ ],
|
|
55
55
|
"userVisibleRoots": ["outputs", "myproject"], // visible roots = outputs + each connected folder's mount name (its basename; `.projects` is the pre-1.14271.0 legacy fallback)
|
|
56
|
+
"readonlyFolderRoots": ["carta-folder"], // the mode:r subset of userVisibleRoots (body-less); lets replay give artifact_json the precise remedy. Optional; absent when no read-only folder
|
|
56
57
|
"artifacts": [ // snapshot of outputs/ + connected folders (optional)
|
|
57
58
|
{ "path": "outputs/x.json", "bytes": 24, "sha256": "…", "body": "{…}" }, // body inlined ≤ 64 KiB
|
|
58
|
-
{ "path": "outputs/big.bin", "bytes": 9e6, "sha256": "…", "truncated": true } // oversized → hash-only
|
|
59
|
+
{ "path": "outputs/big.bin", "bytes": 9e6, "sha256": "…", "truncated": true }, // oversized → hash-only
|
|
60
|
+
{ "path": "carta-folder/input.xlsx", "bytes": 4096, "sha256": "…", "truncated": true } // mode:r connected-folder INPUT → body-less (see below), regardless of size
|
|
59
61
|
],
|
|
60
62
|
"fingerprint": { "baseline": "1.15962.1", "skillHash": "…", "mode": "git", "contentSig": "…", "fileSigs": [["skills/x/SKILL.md", "…"]], "skillSources": ["…"] }, // staleness tripwire (v5: fileSigs only; v6: mode + git default; v7: NUL-delimited hash entries)
|
|
61
63
|
"authoring": { "nonDeterministic": true, "channel": "decider-dir" } // present ONLY when a live decider answered ≥1 gate (see §Answering gates during recording); re-record may drift, replay is still deterministic
|
|
@@ -227,17 +229,18 @@ the rules and CI-placement rationale (why each category behaves this way), see
|
|
|
227
229
|
| `tool_calls_max` | total top-level tool calls (sub-agent tools excluded) ≤ N — meaningfully replay-checkable; the re-drive recomputes `toolCounts` deterministically |
|
|
228
230
|
| `max_turns` | SDK-reported (or fallback-counted) turn count ≤ N — replay-checkable, recounted deterministically same as `tool_calls_max` |
|
|
229
231
|
| `question_asked` | agent asked an AskUserQuestion matching the regex |
|
|
230
|
-
| `questions_count_max` | at most N questions asked |
|
|
231
|
-
| `gate_answers_delivered` | answered gates' answers reached the model |
|
|
232
|
+
| `questions_count_max` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
|
|
233
|
+
| `gate_answers_delivered` | answered gates' answers reached the model — **zero gates fired passes vacuously** (gate firing is model-dependent); pair with `gate_answer_count_min` to also require a gate |
|
|
234
|
+
| `gate_answer_count_min` | at least N AskUserQuestion gates fired AND were delivered non-error — the presence companion to `gate_answers_delivered`'s vacuous-pass |
|
|
232
235
|
| `result` | run ended with `success` or `error` |
|
|
233
236
|
| `allow_permissive_auto_allow` | verdict modifier — kept on replay → no-op pass (the live signal it suppresses is zeroed) |
|
|
234
237
|
| `allow_missing_capability` | verdict modifier — kept on replay → no-op pass (the live signal it suppresses is zeroed) |
|
|
235
238
|
| `allow_l0_plugin_divergence` | verdict modifier — kept on replay → no-op pass (the live signal it suppresses is zeroed) |
|
|
236
239
|
| `allow_stall` | verdict modifier — kept on replay → no-op pass (suppresses the `stalled` default-fail; the stall is re-derived on the replay re-drive) |
|
|
237
240
|
|
|
238
|
-
**`question_asked`, `questions_count_max`, `gate_answers_delivered` require
|
|
239
|
-
replay). On an old cassette without `controlOut` these
|
|
240
|
-
vacuously passed — and a loud warning fires (see §Backward compatibility).
|
|
241
|
+
**`question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min` require
|
|
242
|
+
`controlOut`** (full-fidelity replay). On an old cassette without `controlOut` these keys are excluded
|
|
243
|
+
from evaluation — not vacuously passed — and a loud warning fires (see §Backward compatibility).
|
|
241
244
|
|
|
242
245
|
`file_exists`, `user_visible_artifact`, `artifact_json`, `computer_links_resolve`, and `no_unexpected_files` are **not** in the
|
|
243
246
|
table above — see the next subsection; they're replay-checkable only when the cassette carries an
|
|
@@ -258,7 +261,19 @@ The inline cap is 64 KiB; raise it with `record --max-artifact-bytes <n>` (or
|
|
|
258
261
|
fails fast if an `artifact_json` targets an artifact it had to truncate (that would pass at record but fail
|
|
259
262
|
at replay). `computer_links_resolve` resolves every `computer://` link in the transcript against the same
|
|
260
263
|
manifest, with host-shaped links normalized via the recorded session folders. Without a manifest (older
|
|
261
|
-
cassettes), these are skipped.
|
|
264
|
+
cassettes), these are skipped.
|
|
265
|
+
|
|
266
|
+
**Read-only (`mode: r`) connected-folder contents are captured body-less.** A folder mounted `mode: r` in
|
|
267
|
+
`session:` holds pre-existing INPUTS the agent only reads, not deliverables — so `record` snapshots them
|
|
268
|
+
the same way it snapshots an over-cap file: path + `bytes` + `sha256`, `truncated: true`, **no `body`**,
|
|
269
|
+
regardless of size. The entry still lands in `artifacts[]` (unlike a fully-excluded path) so
|
|
270
|
+
`materializeManifest` writes a 0-byte placeholder at replay and `computer_links_resolve`/`file_exists`
|
|
271
|
+
resolve identically live and on replay; only `artifact_json` (which needs the inlined body) can't target
|
|
272
|
+
one. Two side benefits: no cassette bloat from a large input file, and no `binary` privacy finding (the
|
|
273
|
+
scanner only flags a *committed* binary body) — so a `mode: r` input never needs `--allow`. A `mode: rw`/`rwd`
|
|
274
|
+
folder's contents are captured with a full body exactly as `outputs/` is.
|
|
275
|
+
|
|
276
|
+
A green replay re-confirms *record-time* artifacts, **not** that the current
|
|
262
277
|
skill still produces them — `replay --strict` fails the run when the `fingerprint` shows ANY skill/baseline
|
|
263
278
|
drift, or `replay --fail-on-skill-drift` fails only on skill-source drift (leaving baseline drift a warning).
|
|
264
279
|
Either way, every replay result also reports the drift in `staleness[]` (class-tagged) for a JSON gate to read.
|
|
@@ -303,12 +318,16 @@ On replay, the replay decider (built by `buildReplayDecider()`) indexes `control
|
|
|
303
318
|
the decision pipeline instead of consulting a live decider or asking the user. This makes the full
|
|
304
319
|
`Run.handleDecision` path execute on replay, which populates `rec.questions`, `rec.gateAnswers`, and
|
|
305
320
|
`rec.gateDeliveries` — exactly as in a live run. Consequence: `question_asked`, `questions_count_max`,
|
|
306
|
-
and `
|
|
321
|
+
`gate_answers_delivered`, and `gate_answer_count_min` are now genuinely evaluated, not silently skipped
|
|
322
|
+
or vacuously passed.
|
|
307
323
|
|
|
308
324
|
`gate_answers_delivered` accepts a boolean: `: true` asserts the answered gates' answers reached the
|
|
309
|
-
model
|
|
310
|
-
|
|
311
|
-
|
|
325
|
+
model, **passing vacuously when zero gates fired** (gate firing is model-dependent); `: false` is the
|
|
326
|
+
**inverse** — it asserts a *confirmed delivery failure* (at least one gate whose `delivered === false`),
|
|
327
|
+
for scenarios that deliberately exercise a non-delivery path. Unobserved delivery (`delivered: null`)
|
|
328
|
+
satisfies neither — absence of evidence is a failure, not a pass. Pair `gate_answers_delivered: true`
|
|
329
|
+
with `gate_answer_count_min: <N>` when a gate firing at all is part of what you're testing —
|
|
330
|
+
`gate_answer_count_min` fails if fewer than N gates fired AND were delivered non-error.
|
|
312
331
|
|
|
313
332
|
### The O7 guard — `replay_protocol_fidelity`
|
|
314
333
|
|
|
@@ -337,8 +356,8 @@ regressing to the prior false-green behavior:
|
|
|
337
356
|
::warning:: [replay] cassette has no controlOut (pre-full-fidelity) — question/gate assertions
|
|
338
357
|
are NOT checked; re-record to enable them
|
|
339
358
|
```
|
|
340
|
-
2. **`question_asked`, `questions_count_max`, `gate_answers_delivered` are
|
|
341
|
-
assertion set for that run — not vacuously passed, absent.
|
|
359
|
+
2. **`question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min` are
|
|
360
|
+
excluded** from the evaluated assertion set for that run — not vacuously passed, absent.
|
|
342
361
|
3. All other content assertions (transcript, tool, subagent, result) evaluate normally.
|
|
343
362
|
|
|
344
363
|
This preserves "skipped ≠ false-green." Re-record with a current harness to get the full-fidelity path.
|
|
@@ -477,10 +496,12 @@ counts) — committed PII surface. Two layers, distinct from secret-scrub (which
|
|
|
477
496
|
live-enumerated app/process inventory sentinel (e.g. a computer-use tool schema's "Available applications on
|
|
478
497
|
this machine: …") is never legitimate catalog boilerplate either; none of the three share the ambiguity that
|
|
479
498
|
gets `currency`/`domain` excluded there. `--allow <regex>` suppresses synthetic / public reference names
|
|
480
|
-
(e.g. `NVCA`, `Cooley GO`, `Acme`)
|
|
481
|
-
|
|
482
|
-
`--allow-
|
|
483
|
-
|
|
499
|
+
(e.g. `NVCA`, `Cooley GO`, `Acme`) — each `--allow` value is a **pattern**, matched against a finding, not a
|
|
500
|
+
path to allow; each allow must match the **whole** finding token (so a bare-domain allow no longer silently
|
|
501
|
+
clears an email whose domain it matches), and `--allow-domain` / `--allow-email` / `--allow-path` /
|
|
502
|
+
`--allow-machine-inventory` scope an allow to a single finding class, while `--allow-patterns-file <path>` is a
|
|
503
|
+
different thing — it loads allows from a version-controlled **file of patterns** (one regex per line, `#`
|
|
504
|
+
comments), not a path to allow directly. Multi-word proper
|
|
484
505
|
names are **not** a default class (too noisy). `verify-cassettes` also runs the **staleness**
|
|
485
506
|
check (both checks run by default; scope to one with `--skip-privacy` or `--skip-staleness`): a drifted
|
|
486
507
|
`skillHash` (you edited the skill but didn't re-record) fails the gate.
|
package/docs/scenario.md
CHANGED
|
@@ -277,9 +277,10 @@ if *every* key passes (don't rely on the first; keep one concern per item unless
|
|
|
277
277
|
| `tool_calls_max: <N>` | total top-level tool calls (sum of `toolCounts`, sub-agent tools excluded) ≤ N — unlike the cost/token keys, this **is** meaningfully replay-checkable (the re-drive recomputes `toolCounts` deterministically from the recorded events) |
|
|
278
278
|
| `max_turns: <N>` | the SDK-reported (or fallback-counted) turn count ≤ N — replay-checkable (the re-drive recounts turns deterministically, same as `tool_calls_max`) |
|
|
279
279
|
| `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
|
|
280
|
-
| `questions_count_max: <N>` |
|
|
281
|
-
| `gate_answers_delivered: true` | every answered AskUserQuestion gate's answer actually reached the model — requires a positive, observed `tool_result` (an **unobserved** delivery fails too, not only an errored one — no silent false-green) |
|
|
280
|
+
| `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1 (this is a decision-load budget, not a per-tool-call count); `trace --view questions`'s footer total is computed the same way, so it always matches what this key compares against |
|
|
281
|
+
| `gate_answers_delivered: true` | every answered AskUserQuestion gate's answer actually reached the model — requires a positive, observed `tool_result` (an **unobserved** delivery fails too, not only an errored one — no silent false-green); **zero gates fired passes vacuously** (gate firing is model-dependent) — pair with `gate_answer_count_min` to also require a gate |
|
|
282
282
|
| `gate_answers_delivered: false` | asserts that at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — useful for negative-path tests of delivery failures |
|
|
283
|
+
| `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — the presence companion to `gate_answers_delivered`'s vacuous-pass (mirrors `transcript_contains` pairing with `computer_links_resolve`) |
|
|
283
284
|
| `allow_permissive_auto_allow: true` | verdict modifier — suppresses the default-fail when the run recorded a cowork-parity permissive auto-allow; use this for tests that **deliberately** assert Cowork's permissive behavior rather than strict scripted coverage |
|
|
284
285
|
| `allow_missing_capability: true` | verdict modifier (**live tiers only**) — suppresses the default-fail when the lean/`core` agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables); assert only when the skill's fallback is genuinely equivalent, else rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check below. |
|
|
285
286
|
| `allow_l0_plugin_divergence: true` | verdict modifier — opts into L0/protocol plugin divergence, suppressing the plugin-fidelity default-fail |
|
|
@@ -380,10 +381,10 @@ and re-evaluates the **content** assertions. The authoritative list of content k
|
|
|
380
381
|
`allow_l0_plugin_divergence` / `allow_stall` (kept on replay as no-op passes). `max_cost_usd`/`max_tokens`
|
|
381
382
|
assert the *frozen recording's* spend on replay, not fresh spend — see their table entries above.
|
|
382
383
|
|
|
383
|
-
**`question_asked`, `questions_count_max`, and `
|
|
384
|
-
assertions, but they require the cassette to carry `controlOut` (full-fidelity replay). When
|
|
384
|
+
**`question_asked`, `questions_count_max`, `gate_answers_delivered`, and `gate_answer_count_min`**
|
|
385
|
+
are also content assertions, but they require the cassette to carry `controlOut` (full-fidelity replay). When
|
|
385
386
|
`controlOut` is present, the decision pipeline runs on replay and populates `rec.questions` /
|
|
386
|
-
`rec.gateDeliveries` — so these
|
|
387
|
+
`rec.gateDeliveries` — so these keys are genuinely evaluated.
|
|
387
388
|
When `controlOut` is absent (old cassette), a **loud warning** fires and these keys are **excluded**
|
|
388
389
|
from evaluation (not vacuously passed). Re-record with a current harness to enable them.
|
|
389
390
|
|
|
@@ -396,6 +397,15 @@ entry still satisfies `file_exists` but not `artifact_json`. `computer_links_res
|
|
|
396
397
|
normalizes to a mount-relative path first (via the recorded connected-folder prefixes + the outputs/uploads
|
|
397
398
|
mounts), since replay has no live filesystem to probe directly (that direct check only happens on a live
|
|
398
399
|
`run`/`verify-run`). Without a manifest (older cassettes), all four are **skipped** (loud).
|
|
400
|
+
|
|
401
|
+
A `mode: r` connected folder (see [session.md](./session.md)) holds pre-existing INPUTS, not deliverables —
|
|
402
|
+
`record` captures its contents **body-less** (path + hash, `truncated: true`, no `body`): `file_exists` and
|
|
403
|
+
`computer_links_resolve` still pass against it (the placeholder materializes on replay), while `artifact_json`
|
|
404
|
+
against it reports a clear evidence-unavailable identically on live, verify-run, and replay (so a cassette
|
|
405
|
+
can't record green and replay red). This keeps a read-only input out of the cassette's committed content
|
|
406
|
+
(no bloat, no `binary` privacy finding) while `no_unexpected_files`/`computer_links_resolve` keep enumerating
|
|
407
|
+
the folder as a user-visible root. A `mode: rw`/`rwd` folder's contents are captured with a full body, same
|
|
408
|
+
as `outputs/`.
|
|
399
409
|
A green `replay` re-confirms *record-time* artifacts, **not** that the current skill still produces them —
|
|
400
410
|
that needs a live `run` (the cassette's staleness fingerprint warns when the skill/baseline drifted; `replay
|
|
401
411
|
--strict` fails on any drift, `--fail-on-skill-drift` on skill-source drift only, and every result reports it
|
|
@@ -13,7 +13,7 @@ replay path end-to-end, including:
|
|
|
13
13
|
|
|
14
14
|
Run it with:
|
|
15
15
|
|
|
16
|
-
> Assumes the `cowork-harness` CLI is available — from a source checkout run `npm ci && npm run build && npm link` first, or `npm i -g cowork-harness@>=0.
|
|
16
|
+
> Assumes the `cowork-harness` CLI is available — from a source checkout run `npm ci && npm run build && npm link` first, or `npm i -g cowork-harness@>=0.25.0`. (`replay` itself needs nothing else — no token, no Docker.)
|
|
17
17
|
|
|
18
18
|
```sh
|
|
19
19
|
cowork-harness replay examples/replays/example-pdf-skill.cassette.json
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "cowork-harness",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.25.0",
|
|
4
4
|
"description": "Scriptable, CI-friendly harness for Claude Cowork's runtime contract for testing skills across scenarios — same agent, mounts, egress allowlist, permission protocol, and sandbox limitations.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
package/schema/cassette.v7.json
CHANGED
|
@@ -54,6 +54,13 @@
|
|
|
54
54
|
},
|
|
55
55
|
"description": "v4+: the user-visible mount roots captured at record time (`outputs` + each connected folder's resolved mount name). Replay reads THIS instead of a hardcoded prefix. ABSENT on pre-v4 cassettes → replay falls back to the legacy prefix."
|
|
56
56
|
},
|
|
57
|
+
"readonlyFolderRoots": {
|
|
58
|
+
"type": "array",
|
|
59
|
+
"items": {
|
|
60
|
+
"type": "string"
|
|
61
|
+
},
|
|
62
|
+
"description": "The subset of `userVisibleRoots` that are read-only (`mode: r`) connected folders — captured body-less (path + hash, no body). Persisted so replay distinguishes a body-less read-only input from an over-cap artifact and gives the precise `artifact_json` remediation. Optional metadata (no cassetteVersion bump); ABSENT/empty ⇒ no read-only folder in the run. A subset of `userVisibleRoots` (no new privacy strings)."
|
|
63
|
+
},
|
|
57
64
|
"preRunPaths": {
|
|
58
65
|
"type": "array",
|
|
59
66
|
"items": {
|
|
@@ -259,9 +259,15 @@
|
|
|
259
259
|
"maximum": 9007199254740991
|
|
260
260
|
},
|
|
261
261
|
"gate_answers_delivered": {
|
|
262
|
-
"description": "every answered AskUserQuestion gate's tool_result was non-error (the answer reached the model)",
|
|
262
|
+
"description": "every answered AskUserQuestion gate's tool_result was non-error (the answer reached the model); zero gates fired passes vacuously — pair with gate_answer_count_min to also require a gate",
|
|
263
263
|
"type": "boolean"
|
|
264
264
|
},
|
|
265
|
+
"gate_answer_count_min": {
|
|
266
|
+
"description": "at least N AskUserQuestion gates fired AND were delivered non-error (presence companion to gate_answers_delivered)",
|
|
267
|
+
"type": "integer",
|
|
268
|
+
"minimum": 0,
|
|
269
|
+
"maximum": 9007199254740991
|
|
270
|
+
},
|
|
265
271
|
"result": {
|
|
266
272
|
"description": "the run's final result was success | error",
|
|
267
273
|
"type": "string",
|