cowork-harness 0.24.0 → 0.26.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +25 -4
- package/.claude/skills/cowork-harness/references/ci-recipe.md +4 -4
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +25 -9
- package/.claude/skills/cowork-harness/references/task-recipes.md +6 -4
- package/.claude/skills/cowork-harness/scripts/assertion-keys.json +1 -0
- package/.claude/skills/cowork-harness/scripts/scenario.py +10 -2
- package/CHANGELOG.md +111 -0
- package/README.md +5 -5
- package/SPEC.md +27 -9
- package/dist/assert.js +48 -5
- package/dist/cli.js +8 -2
- package/dist/run/cassette.js +75 -34
- package/dist/run/execute.js +18 -5
- package/dist/run/renderer.js +1 -1
- package/dist/run/skill-hash.js +14 -4
- package/dist/run/trace-view.js +15 -5
- package/dist/session.js +8 -0
- package/dist/types.js +7 -1
- package/docs/cassette.md +43 -20
- package/docs/scenario.md +15 -5
- package/examples/replays/README.md +1 -1
- package/examples/replays/example-pdf-skill.cassette.json +30 -24
- package/package.json +1 -1
- package/schema/cassette.v7.json +7 -0
- package/schema/cassette.v8.json +166 -0
- package/schema/run-result.json +55 -1
- package/schema/scenario.schema.json +7 -1
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 0.
|
|
7
|
-
tracks-harness: cowork-harness 0.
|
|
6
|
+
version: 0.26.0
|
|
7
|
+
tracks-harness: cowork-harness 0.26.0 (baseline desktop-1.18286.0)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.26.0` (baseline
|
|
26
26
|
> `desktop-1.18286.0`). If your checkout is newer, prefer the live `--help`, `SPEC.md`, and
|
|
27
27
|
> `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
|
|
|
38
38
|
Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.
|
|
41
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.26.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.26.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.26.0"`. **Pin `@>=0.26.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.26.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
|
|
42
42
|
- **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
|
|
43
43
|
- **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
|
|
44
44
|
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
@@ -458,6 +458,27 @@ are the ones that bite hardest.
|
|
|
458
458
|
and re-assert is undetected — the notice flags this; re-record if the session changed. `verify-run` reads
|
|
459
459
|
on-disk `assert:` against a kept *run dir*; `replay --assert-from` is the equivalent for a *cassette*.
|
|
460
460
|
|
|
461
|
+
18. **`questions_count_max` counts sub-questions, not gates.** One `AskUserQuestion` tool call can
|
|
462
|
+
bundle several sub-questions into a single gate; the assertion counts each sub-question, so a
|
|
463
|
+
3-sub-question bundle counts as 3, not 1. `trace --view questions` shows the same per-gate
|
|
464
|
+
sub-question count and a matching footer total — read that off instead of the tool-call count when
|
|
465
|
+
sizing the budget.
|
|
466
|
+
|
|
467
|
+
19. **`gate_answers_delivered` passes vacuously when no gate fires — use `gate_answer_count_min: 1` to
|
|
468
|
+
also require a gate.** Whether a gate fires is model-dependent, so `gate_answers_delivered: true`
|
|
469
|
+
alone can't catch "the gate never fired at all"; pair it with `gate_answer_count_min` when
|
|
470
|
+
presence matters, not just delivery.
|
|
471
|
+
|
|
472
|
+
20. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
|
|
473
|
+
read-only folder's files as path + hash only (`truncated: true`, no `body`) — it's an input the agent
|
|
474
|
+
read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass against it on replay
|
|
475
|
+
(the hash-only entry still materializes a placeholder); `artifact_json` reports a clear
|
|
476
|
+
evidence-unavailable on every lane (live/verify-run/replay agree — no green-record/red-replay). This is
|
|
477
|
+
also why a `mode: r` input never trips the `binary` privacy finding or needs `--allow` — only a
|
|
478
|
+
*committed* body is scanned. `scaffold` won't emit `file_exists` for one either (it's not in
|
|
479
|
+
`RunResult.artifacts`). A `mode: rw`/`rwd` folder's contents are captured with a full body, same as
|
|
480
|
+
`outputs/`.
|
|
481
|
+
|
|
461
482
|
For the complete gotcha list, the assertion catalog, the YAML schema, the fidelity/answer tables,
|
|
462
483
|
and the CI recipe, read the files in `references/`.
|
|
463
484
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.26.0` (baseline `desktop-1.18286.0`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -53,7 +53,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
53
53
|
GitHub-hosted runners, no token/Docker/agent:
|
|
54
54
|
|
|
55
55
|
```yaml
|
|
56
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
56
|
+
- run: npm i -g "cowork-harness@>=0.26.0"
|
|
57
57
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
58
58
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
59
59
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -187,7 +187,7 @@ jobs:
|
|
|
187
187
|
with: { node-version: '20' }
|
|
188
188
|
- uses: actions/setup-python@v5
|
|
189
189
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
190
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
190
|
+
- run: npm i -g "cowork-harness@>=0.26.0"
|
|
191
191
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
192
192
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
193
193
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -216,7 +216,7 @@ jobs:
|
|
|
216
216
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
217
217
|
fi
|
|
218
218
|
- if: steps.guard.outputs.live == 'true'
|
|
219
|
-
run: npm i -g "cowork-harness@>=0.
|
|
219
|
+
run: npm i -g "cowork-harness@>=0.26.0"
|
|
220
220
|
- if: steps.guard.outputs.live == 'true'
|
|
221
221
|
run: cowork-harness run scenarios/ --output-format json
|
|
222
222
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.26.0` (baseline `desktop-1.18286.0`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.26.0`
|
|
4
4
|
(baseline `desktop-1.18286.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
@@ -237,9 +237,10 @@ passes only if every key passes. Keep one concern per item unless you mean conju
|
|
|
237
237
|
| `tool_calls_max: <N>` | total top-level tool calls (sum of `toolCounts`) ≤ N — meaningfully replay-checkable (re-drive recomputes `toolCounts` deterministically) |
|
|
238
238
|
| `max_turns: <N>` | the SDK-reported (or fallback-counted) turn count ≤ N — meaningfully replay-checkable (re-drive recounts turns deterministically, same as `tool_calls_max`) |
|
|
239
239
|
| `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
|
|
240
|
-
| `questions_count_max: <N>` |
|
|
241
|
-
| `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail) |
|
|
240
|
+
| `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
|
|
241
|
+
| `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min` to also require a gate |
|
|
242
242
|
| `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests |
|
|
243
|
+
| `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass |
|
|
243
244
|
| `allow_permissive_auto_allow: true` | verdict modifier — suppresses the default-fail when the run recorded a cowork-parity permissive auto-allow; for tests that deliberately assert Cowork's permissive behavior |
|
|
244
245
|
| `allow_missing_capability: true` | verdict modifier — suppresses the default-fail when the (partial "core") agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables). Assert only when the skill's fallback is genuinely equivalent; otherwise rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check. Live tiers only |
|
|
245
246
|
| `allow_l0_plugin_divergence: true` | verdict modifier — opt into L0/protocol plugin divergence: suppresses the default-fail when a plugin behaves differently at `protocol` (L0) fidelity than under a sandboxed tier. Live tiers only |
|
|
@@ -293,8 +294,10 @@ modifiers `allow_permissive_auto_allow` / `allow_missing_capability` / `allow_l0
|
|
|
293
294
|
`allow_stall` are also kept on replay, evaluated as no-op passes.
|
|
294
295
|
|
|
295
296
|
**Gate keys — replay only with a `controlOut` cassette:** `question_asked`, `questions_count_max`,
|
|
296
|
-
`gate_answers_delivered`. With `controlOut` present they evaluate; on an old
|
|
297
|
-
**loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
|
|
297
|
+
`gate_answers_delivered`, `gate_answer_count_min`. With `controlOut` present they evaluate; on an old
|
|
298
|
+
cassette without it, a **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
|
|
299
|
+
`questions_count_max` counts sub-questions, not gates/tool-calls — see the catalog row above and
|
|
300
|
+
`trace --view questions`.
|
|
298
301
|
|
|
299
302
|
**Filesystem — replay-checkable WITH an artifact manifest:** `file_exists`, `user_visible_artifact`,
|
|
300
303
|
`artifact_json`, `computer_links_resolve` run on replay when the cassette carries an `artifacts` snapshot
|
|
@@ -374,10 +377,12 @@ at the top of this file.
|
|
|
374
377
|
concern per item; run the linter. (`contentKeys` in `src/run/cassette.ts`.)
|
|
375
378
|
|
|
376
379
|
2. **Gate keys need a `controlOut` cassette.** `question_asked`, `questions_count_max`,
|
|
377
|
-
`gate_answers_delivered` only evaluate on replay with `controlOut`; on an
|
|
378
|
-
and are excluded (not passed). `gate_answers_delivered` **fails on
|
|
379
|
-
(`delivered: null`) — absence of evidence is failure
|
|
380
|
-
|
|
380
|
+
`gate_answers_delivered`, `gate_answer_count_min` only evaluate on replay with `controlOut`; on an
|
|
381
|
+
old cassette they warn and are excluded (not passed). `gate_answers_delivered` **fails on
|
|
382
|
+
unobserved delivery** (`delivered: null`) — absence of evidence is failure — but **passes
|
|
383
|
+
vacuously when zero gates fired**; use `gate_answer_count_min: 1` to also require a gate to have
|
|
384
|
+
fired. A **header-only gate** (empty `question`, only `header`) can never be keyed and is rejected
|
|
385
|
+
loudly — every gate needs a non-empty `question`.
|
|
381
386
|
(`contentKeys` in `src/run/cassette.ts`; `src/assert.ts`.)
|
|
382
387
|
|
|
383
388
|
3. **The LLM-decider's two spellings.** Scripted answers + `on_unanswered: fail` is deterministic;
|
|
@@ -455,3 +460,14 @@ at the top of this file.
|
|
|
455
460
|
Consequence: artifact assertions (`artifact_json`) over fields that were redacted will fail at replay
|
|
456
461
|
— the harness emits `::warning::` at record time when this occurs. (`src/secrets.ts:scrubField`;
|
|
457
462
|
`src/run/cassette.ts`.)
|
|
463
|
+
|
|
464
|
+
21. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
|
|
465
|
+
read-only folder's files as `path` + `bytes` + `sha256` only (`truncated: true`, no `body`) — it's an
|
|
466
|
+
input the agent read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass
|
|
467
|
+
against it on replay (the hash-only entry still materializes a 0-byte placeholder); `artifact_json`
|
|
468
|
+
reports a clear evidence-unavailable on every lane (live/verify-run/replay agree). This is also why a
|
|
469
|
+
`mode: r` input never trips the `binary` privacy finding
|
|
470
|
+
or needs `--allow` in `verify-cassettes` — only a *committed* body is scanned. `scaffold` won't emit
|
|
471
|
+
`file_exists` for one either, since it isn't in `RunResult.artifacts`. A `mode: rw`/`rwd` folder's
|
|
472
|
+
contents are captured with a full body, same as `outputs/`. (`src/run/cassette.ts:buildManifest`'s
|
|
473
|
+
`bodyLessPrefixes`; `src/session.ts:readonlyFolderRootsFromPlan`.)
|
|
@@ -59,6 +59,7 @@ Top-level fields of a `*.cassette.json` (schema `schema/cassette.v7.json`):
|
|
|
59
59
|
| `artifacts` | Output-file manifest (paths + hashes + small inlined bodies) — unlocks `file_exists`/`artifact_json` on replay |
|
|
60
60
|
| `fingerprint` | Skill/baseline staleness tripwire |
|
|
61
61
|
| `userVisibleRoots` | The user-visible mount roots captured at record time |
|
|
62
|
+
| `readonlyFolderRoots` | The read-only (`mode: r`) subset of `userVisibleRoots` — captured body-less; lets replay give `artifact_json` the precise remedy for a body-less input |
|
|
62
63
|
| `preRunPaths` | Pre-run file-path baseline for `no_unexpected_files` (workRoot-relative; co-present with `userVisibleRoots`) |
|
|
63
64
|
| `scenarioSource` | Relative path to the authored YAML this was recorded from |
|
|
64
65
|
| `authoring` | Present iff a live decider answered ≥1 gate during recording (`nonDeterministic: true`) |
|
|
@@ -83,10 +84,11 @@ record — retrofitting means re-recording:
|
|
|
83
84
|
redaction missed.
|
|
84
85
|
5. **Scoping idioms for scanner findings you've reviewed and accepted:** prefer the class-scoped
|
|
85
86
|
flags — `--allow-domain 'api\.example\.com'`, `--allow-email '.*@yourco\.com'`,
|
|
86
|
-
`--allow-path '/opt/ci-runner/.*'` — over bare `--allow
|
|
87
|
-
let a domain allow swallow an email leak). For a
|
|
88
|
-
`--allow-file <path>`: bare all-class
|
|
89
|
-
lines ignored
|
|
87
|
+
`--allow-path '/opt/ci-runner/.*'` — over bare `--allow <regex>` (a single pattern, which
|
|
88
|
+
applies to every class and can let a domain allow swallow an email leak). For a
|
|
89
|
+
version-controlled allowlist use `--allow-patterns-file <path>`: a FILE of bare all-class
|
|
90
|
+
patterns, one regex per line, `#` comments and blank lines ignored — note the flag name is the
|
|
91
|
+
FILE of patterns, not a path to allow.
|
|
90
92
|
|
|
91
93
|
## Recipe 4 — Budget assertions without a two-pass record
|
|
92
94
|
|
|
@@ -72,7 +72,7 @@ CONTENT_KEYS = {
|
|
|
72
72
|
"max_turns",
|
|
73
73
|
}
|
|
74
74
|
# content keys, but only evaluated on replay when the cassette carries controlOut
|
|
75
|
-
GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered"}
|
|
75
|
+
GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered", "gate_answer_count_min"}
|
|
76
76
|
# manifest-backed: replay-checkable when the cassette carries an `artifacts` manifest (record snapshots one);
|
|
77
77
|
# a manifest-less cassette skips them. Since the 0.3.0 artifact-manifest these are NOT always live-only.
|
|
78
78
|
# computer_links_resolve joins this bucket (not CONTENT_KEYS): resolving a non-empty link set needs
|
|
@@ -716,7 +716,15 @@ def build_scenario(args):
|
|
|
716
716
|
if gates:
|
|
717
717
|
for rx, _ in gates:
|
|
718
718
|
content_lines.append(f" - question_asked: {_sq(rx)} # gate key: replay only with a controlOut cassette")
|
|
719
|
-
|
|
719
|
+
# questions_count_max counts SUB-questions at runtime (assert.ts/trace-view.ts), but this
|
|
720
|
+
# scaffold is STATIC — it only knows the number of --gate rules (per-tool-call), never how many
|
|
721
|
+
# sub-questions each gate bundles. Any number emitted here would be a guess: too low false-reds
|
|
722
|
+
# on the first run, too high is a dead tripwire. A budget must come from observation, not
|
|
723
|
+
# fabrication — so emit it COMMENTED OUT with the calibration path, not a made-up value.
|
|
724
|
+
content_lines.append(
|
|
725
|
+
" # - questions_count_max: <N> # BUDGET — calibrate from a real run: `trace --view "
|
|
726
|
+
"questions` prints the SUB-question total (what this asserts); set N to that + headroom."
|
|
727
|
+
)
|
|
720
728
|
content_lines.append(" - gate_answers_delivered: true # the steered answers actually reached the model")
|
|
721
729
|
|
|
722
730
|
live_lines = []
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,117 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.26.0] — 2026-07-05
|
|
10
|
+
|
|
11
|
+
Cassette/run-result **format-freeze** pass before 1.0 — fixes the parts of these schemas that become
|
|
12
|
+
irreversible once 1.0's semver contract freezes them.
|
|
13
|
+
|
|
14
|
+
> **Upgrade notes.**
|
|
15
|
+
> - **Staleness-hash framing changed → `cassetteVersion` 8.** A pre-v8 cassette's staleness fingerprint
|
|
16
|
+
> is non-comparable with v8, so `verify-cassettes`/`rehash` route a pre-v8 cassette to **re-record**
|
|
17
|
+
> with an honest "algorithm changed" message (not a misleading "content changed"). Re-record your
|
|
18
|
+
> committed cassettes. (Cassettes without a `skillHash` are unaffected and keep replaying.)
|
|
19
|
+
> - **`RunResult.unanswered` renamed to `nonReproducibleAnswers`.** Consumers reading
|
|
20
|
+
> `result.unanswered` must switch to `result.nonReproducibleAnswers` (the field never held literally
|
|
21
|
+
> unanswered gates — it holds gates answered by a non-reproducible source).
|
|
22
|
+
> - **Cassette-level `readonlyFolderRoots` removed** in favor of per-entry
|
|
23
|
+
> `ManifestEntry.truncationReason`. No consumer action unless you parsed the cassette directly.
|
|
24
|
+
|
|
25
|
+
### Fixed
|
|
26
|
+
|
|
27
|
+
- **Two staleness-hash framing collisions (a false-negative — the risk the gate exists to prevent).**
|
|
28
|
+
`skillHash` folded raw file content after a `F:<path>\0` marker, so a file whose content embedded that
|
|
29
|
+
marker could hash identically to a two-file tree; it now folds the fixed-length content SHA
|
|
30
|
+
(self-delimiting, charset disjoint from the `F:`/`L:` prefixes). `contentSig` lacked a type marker and
|
|
31
|
+
used `:` between path and sha, so a file named `a:lnk` could collide with a symlink `a`; entries are now
|
|
32
|
+
type-prefixed (`F:`/`L:`) and NUL-framed, and the `L:` link line NUL-separates path from target. Only
|
|
33
|
+
fixable before the hash algorithm freezes at 1.0.
|
|
34
|
+
|
|
35
|
+
### Changed
|
|
36
|
+
|
|
37
|
+
- **Cassette format v7 → v8** (`CONTENTSIG_ALGO` 3 → 4). See the framing fix above and the upgrade note.
|
|
38
|
+
- **`RunResult.unanswered` → `nonReproducibleAnswers`** — a self-admitted misnomer in the frozen
|
|
39
|
+
run-result envelope, renamed before the 1.0 freeze.
|
|
40
|
+
- **Body-less manifest entries now carry `ManifestEntry.truncationReason`** (`size` | `readonly` |
|
|
41
|
+
`unreadable`) instead of a cassette-level `readonlyFolderRoots` list — self-describing and
|
|
42
|
+
redaction-immune. Replay reads the reason to give `artifact_json` the precise remedy on every lane
|
|
43
|
+
(assert on a deliverable vs. raise `--max-artifact-bytes`).
|
|
44
|
+
- **`schema/run-result.json` completed** (added `fingerprint`, `userVisibleRoots`, `readonlyFolderRoots`,
|
|
45
|
+
`preRunPaths`, `toolResults`) and now strict-validated by a test, and the `cassette.v8.json`
|
|
46
|
+
`ManifestEntry` shape is fully described — so the schemas frozen at 1.0 match what ships. Corrected the
|
|
47
|
+
git-mode-default documentation (git is the default unless `COWORK_HARNESS_GITSET=0`).
|
|
48
|
+
|
|
49
|
+
## [0.25.0] — 2026-07-05
|
|
50
|
+
|
|
51
|
+
> **Upgrade notes.**
|
|
52
|
+
> - **`verify-cassettes --allow-file` was renamed to `--allow-patterns-file`** and the old spelling is
|
|
53
|
+
> removed (exits 2). Update any CI step or allowlist wiring that passed `--allow-file`.
|
|
54
|
+
> - **`gate_answers_delivered: true` now passes vacuously when zero gates fired.** If you relied on it
|
|
55
|
+
> as an implicit "a gate must fire" check, add `gate_answer_count_min: 1` alongside it.
|
|
56
|
+
> - **Read-only (`mode: r`) connected-folder inputs are now captured body-less.** Re-record affected
|
|
57
|
+
> cassettes to drop the `--allow` entries they previously required and shrink them; an `artifact_json`
|
|
58
|
+
> assertion pointed at a read-only input now reports evidence-unavailable (assert on a deliverable).
|
|
59
|
+
> This changes committed cassette *contents* (not the staleness hash) — a re-record is cosmetic, not
|
|
60
|
+
> required for correctness.
|
|
61
|
+
|
|
62
|
+
### Added
|
|
63
|
+
|
|
64
|
+
- **`gate_answer_count_min: <N>` assertion** — the presence companion to `gate_answers_delivered`:
|
|
65
|
+
fails unless at least N `AskUserQuestion` gates fired AND were delivered non-error. Pairs with
|
|
66
|
+
`gate_answers_delivered`'s new zero-gate vacuous-pass (below) the way `transcript_contains` pairs
|
|
67
|
+
with `computer_links_resolve`'s zero-link vacuous-pass. Evaluated on replay only with a `controlOut`
|
|
68
|
+
cassette, like the other gate keys; fails as evidence-unavailable (not vacuous-pass) when
|
|
69
|
+
gate-delivery telemetry is absent from `result.json`.
|
|
70
|
+
|
|
71
|
+
### Changed
|
|
72
|
+
|
|
73
|
+
- **`gate_answers_delivered: true` now passes vacuously when zero `AskUserQuestion` gates fired**,
|
|
74
|
+
instead of hard-failing. Whether a gate fires is model-dependent, so any skill with optional gating
|
|
75
|
+
could not use the assertion under the old hard-fail. The bad-delivery check (an answered gate whose
|
|
76
|
+
answer wasn't confirmed delivered) is unchanged — it only ever mattered when gates fired. Missing
|
|
77
|
+
gate-delivery telemetry (an old/partial `result.json` on the verify-run lane) still fails loud as
|
|
78
|
+
evidence-unavailable, never vacuous-pass — this is NOT the same as "no gate fired" and preserves the
|
|
79
|
+
bug-#33 false-green fix. Pair with the new `gate_answer_count_min` (below) to also require presence.
|
|
80
|
+
- **`verify-cassettes --allow-file <path>` renamed to `--allow-patterns-file <path>`.** The old name
|
|
81
|
+
read as "allow this file" and was routinely confused with `--allow <regex>` (a pattern matched
|
|
82
|
+
against a finding) — reaching for `--allow-file <artifact-path>` failed with an ENOENT/invalid-regex
|
|
83
|
+
error instead of doing what the user meant. `--allow-patterns-file <path>` is self-documenting: the
|
|
84
|
+
path names a **file of patterns** (one regex per line), not a path to allow. Pre-1.0 rename with **no
|
|
85
|
+
deprecation window** — the old `--allow-file` spelling is hard-removed and now exits 2 with an
|
|
86
|
+
unknown-flag error. The binary-finding recourse message and both `--help` texts now spell out the
|
|
87
|
+
distinction between the two flags.
|
|
88
|
+
|
|
89
|
+
### Fixed
|
|
90
|
+
|
|
91
|
+
- **Read-only (`mode: r`) connected-folder inputs are captured body-less (path + hash) instead of as
|
|
92
|
+
full binary bodies.** `record` used to sweep a `mode: r` folder's files into the cassette
|
|
93
|
+
`artifacts[]` as full base64 bodies — an input the agent only reads, not a deliverable — causing
|
|
94
|
+
cassette bloat and a hard `binary` privacy finding that forced `--allow`. The manifest entry now
|
|
95
|
+
carries `path` + `bytes` + `sha256` with `truncated: true` and no `body`, reusing the same
|
|
96
|
+
representation the 64-KiB inline cap already produces. The entry still survives in the manifest (it
|
|
97
|
+
is NOT excluded), so `computer_links_resolve`/`file_exists` resolve identically on live and replay.
|
|
98
|
+
`artifact_json` against a body-less target (a read-only input, or any artifact over the body cap)
|
|
99
|
+
reports a clear **evidence-unavailable** on every lane — live, verify-run, and replay all agree, so
|
|
100
|
+
a cassette can't record green and replay red. The cassette persists the read-only-folder set
|
|
101
|
+
(`readonlyFolderRoots`, a subset of `userVisibleRoots`), so replay knows *why* an entry is body-less
|
|
102
|
+
and gives the **precise** remedy — "assert on a deliverable" for a read-only input, "raise
|
|
103
|
+
`--max-artifact-bytes`" for an over-cap artifact — instead of a cryptic JSON parse error or a
|
|
104
|
+
guessed hint. A `mode: rw`/`rwd` folder's contents are unaffected and keep their full body.
|
|
105
|
+
- **`trace --view questions` and the `scaffold` helper disagreed with `questions_count_max` on what
|
|
106
|
+
"a question" counts.** The assertion counts **sub-questions** (one `AskUserQuestion` bundling K
|
|
107
|
+
sub-questions counts as K — the better spam/burden budget, since per-tool-call counting would miss
|
|
108
|
+
a 10-in-1 bundle), but `trace --view questions` collapsed a bundle into one row with no count, and
|
|
109
|
+
the scaffold helper emitted `questions_count_max: <gate count>` — a budget that could start BELOW
|
|
110
|
+
what the assertion computes and false-red on the first real run. No behavior change to the
|
|
111
|
+
assertion itself: `trace --view questions` now annotates each gate row with its sub-question count
|
|
112
|
+
and prints a footer total (`N gate(s), M sub-question(s) total`) that matches what
|
|
113
|
+
`questions_count_max` compares against. The static `scaffold` helper can't know the sub-question
|
|
114
|
+
count offline, so instead of fabricating a value it now emits `questions_count_max` **commented
|
|
115
|
+
out** as a calibration TODO pointing at `trace --view questions` (a budget must come from
|
|
116
|
+
observation, not a guess — a fabricated one either false-reds or is a dead tripwire). The count
|
|
117
|
+
definition is now documented in `docs/scenario.md`, `docs/cassette.md`, `SPEC.md`, and the skill's
|
|
118
|
+
`references/scenario-schema.md` / `SKILL.md` gotcha list.
|
|
119
|
+
|
|
9
120
|
## [0.24.0] — 2026-07-05
|
|
10
121
|
|
|
11
122
|
### Added
|
package/README.md
CHANGED
|
@@ -84,7 +84,7 @@ Only the committed-cassette `replay` above is fully self-contained. Live `run`/`
|
|
|
84
84
|
> - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
|
|
85
85
|
> - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
|
|
86
86
|
> - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
|
|
87
|
-
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx cowork-harness@>=0.
|
|
87
|
+
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx cowork-harness@>=0.26.0`.
|
|
88
88
|
>
|
|
89
89
|
> *(One project, three names: the repo folder is `claude-cowork-headless-emulator`; the npm package, the CLI command, and the GitHub repo are all `cowork-harness`.)*
|
|
90
90
|
|
|
@@ -104,7 +104,7 @@ This repo ships a **companion skill** (`.claude/skills/cowork-harness/`) that te
|
|
|
104
104
|
/plugin install cowork-harness@cowork-harness
|
|
105
105
|
```
|
|
106
106
|
|
|
107
|
-
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.
|
|
107
|
+
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.26.0` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
|
|
108
108
|
|
|
109
109
|
It also follows the open [Agent Skills](https://github.com/vercel-labs/skills) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) by pointing the `npx skills` CLI at `.claude/skills/cowork-harness` in this repo. (Working *inside* this repo, the skill auto-loads as a project skill — no install needed.)
|
|
110
110
|
|
|
@@ -119,7 +119,7 @@ A global install is enough for CI `lint`, reading the teaching skill, and replay
|
|
|
119
119
|
To `run` the worked examples live or copy them as a starting point, use a source checkout. (The marketplace
|
|
120
120
|
skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
|
|
121
121
|
assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
|
|
122
|
-
above becomes available once the skill's first command self-bootstraps `npx cowork-harness@>=0.
|
|
122
|
+
above becomes available once the skill's first command self-bootstraps `npx cowork-harness@>=0.26.0` — see
|
|
123
123
|
[above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
|
|
124
124
|
|
|
125
125
|
### Prerequisites for anything above `protocol` fidelity
|
|
@@ -179,7 +179,7 @@ cowork-harness lint examples/scenarios/*.yaml
|
|
|
179
179
|
> `tool_calls_max`/`max_turns`, `result`, the verdict modifiers) always — note `max_cost_usd`/`max_tokens`
|
|
180
180
|
> assert the *frozen recording's* spend on replay, not fresh spend; a real budget regression is caught by a
|
|
181
181
|
> live `run` — plus `question_asked`/
|
|
182
|
-
> `questions_count_max`/`gate_answers_delivered` when the cassette carries `controlOut`, and
|
|
182
|
+
> `questions_count_max`/`gate_answers_delivered`/`gate_answer_count_min` when the cassette carries `controlOut`, and
|
|
183
183
|
> **filesystem assertions** (`file_exists`, `user_visible_artifact`, `artifact_json`,
|
|
184
184
|
> `computer_links_resolve`) when it carries an
|
|
185
185
|
> `artifacts` manifest. Genuinely live-only keys (`egress_*`, `expect_denied`, `no_delete_in_outputs`,
|
|
@@ -307,7 +307,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
|
|
|
307
307
|
| `run <scenario.yaml \| dir/>` | Run authored scenarios with `assert:` + a CI-ready exit code; a decider (`--decider-cmd`/`--decider-dir`, or a scenario's `on_unanswered: llm`) can answer unscripted gates. `--repeat N` (2-100) runs each scenario N times and aggregates a variance rollup instead of a single pass/fail — `--min-pass-rate`, `--stop-on-diverge`, `--max-budget-usd` tune the batch verdict/loop. `--matrix <matrix.yaml>` runs ONE scenario across the cross-product of baseline/model/skill_dir axes (worked example: `examples/matrices/csv-metrics-matrix.yaml`) — `--max-cells`/`--concurrency` tune the cap/pool; any cell failing (assertion or infra) fails the run; `--compact`/`--demo` trim output for shareable screenshots/GIFs (shared with `skill`, below) | you want a repeatable, **asserted regression test** — to **measure flakiness** instead of trusting one green — or to **test a compatibility matrix** (multiple baselines/models/skill variants) in one run |
|
|
308
308
|
| `chat <folder> [prompt]` | Interactive multi-turn REPL against a skill (TTY); optional seed prompt is sent as the first turn. `--upload <file>` / `--folder <dir>` (repeatable) attach files/project folders; `--verbose` shows thinking blocks + tool inputs; `--fidelity protocol\|container\|hostloop` (no `microvm`/`cowork` in the REPL) | debugging a multi-turn flow by hand |
|
|
309
309
|
| `record` / `replay` | **Record a live run once → replay it token-free, Docker-free thereafter.** Key flags: `--decider-llm`/`--decider-dir` (answer gates live during recording), `--rerecord-stale`/`--concurrency <N>` (batch re-recording), `--assert-from <scenario.yaml>`/`--reassert` (re-check the on-disk `assert:` instead of the frozen one), `--strict`/`--fail-on-skill-drift` (staleness handling on replay), `--no-redact` (skip record-time redaction), `--allow-failing` (relax the post-run verdict gate), `--dry-run` (resolve without recording). Full flag list: `cowork-harness record --help` / `replay --help`. | **token-free, Docker-free CI** from a once-recorded run |
|
|
310
|
-
| `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain/path/machine-inventory → exit 1; whole-token allows via `--allow
|
|
310
|
+
| `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain/path/machine-inventory → exit 1; whole-token allows via `--allow <regex>` (a pattern) / class-scoped `--allow-domain` / `--allow-email` / `--allow-path` / `--allow-machine-inventory` / `--allow-patterns-file <path>` (a **file** of patterns, one regex per line)) + a staleness check (`--skip-privacy` or `--skip-staleness` to run only one half). A dir argument scans `*.cassette.json` in that dir only (**non-recursive**) | gating **committed cassettes** against PII leaks + "edited the skill, forgot to re-record" |
|
|
311
311
|
| `verify-run <run-dir> <scenario.yaml>` | Re-evaluate a scenario's `assert:` (and, when the scenario declares `answers:`, whether they still match the run's actual gates) against an already-kept run dir — **no live agent, no tokens, no Docker** (~1s) | iterating on a wrong assertion or a drifted `answer` without a full live re-record |
|
|
312
312
|
| `trace <run-id>` | Digest a run's `events.jsonl` (`--view tools\|questions\|dispatches`; legacy `--tools` / `--gates` / `--dispatches` aliases still accepted) | "how many sub-agents *actually* dispatched, and which?" |
|
|
313
313
|
| `inspect <run-id>` | Show what a run **produced**: the artifacts + a shallow field preview of each JSON artifact (`--output-format json` for a digest). Works on a salvaged partial run too | "did it do the job?" — without hand-parsing `…/mnt/outputs/…` |
|
package/SPEC.md
CHANGED
|
@@ -384,7 +384,7 @@ states in `baseline.provenance.gates`). A skill that ignores these behaves diffe
|
|
|
384
384
|
`tool_*`, `subagent_*`, `dispatch_count_max`, `skill_triggered`, `no_skill_triggered`, `max_cost_usd`,
|
|
385
385
|
`max_tokens`, `tool_calls_max`, `max_turns`, `result`, the verdict modifiers (`allow_permissive_auto_allow`,
|
|
386
386
|
`allow_missing_capability`, `allow_l0_plugin_divergence`, `allow_stall`), and (when `controlOut` is present)
|
|
387
|
-
`question_asked`, `questions_count_max`, `gate_answers_delivered`. `max_cost_usd`/`max_tokens` are
|
|
387
|
+
`question_asked`, `questions_count_max`, `gate_answers_delivered`, `gate_answer_count_min`. `max_cost_usd`/`max_tokens` are
|
|
388
388
|
evaluated against the *frozen recording's* usage/cost on replay, not fresh spend; `tool_calls_max`/
|
|
389
389
|
`max_turns` are meaningfully replay-checkable (the re-drive recomputes both deterministically).
|
|
390
390
|
- **Filesystem assertions** (`file_exists`, `user_visible_artifact`, `artifact_json`, `computer_links_resolve`,
|
|
@@ -405,10 +405,21 @@ green replay does not imply the recording is still valid. Each finding is surfac
|
|
|
405
405
|
`RunResult.staleness[]` for a token-free gate to act on. `replay --strict` fails on any class;
|
|
406
406
|
`replay --fail-on-skill-drift` fails only on skill-source classes (`skill` / `shared-root` /
|
|
407
407
|
`unverifiable-skill`). Both realize the gate as failing `assertions[]` entries (so `ok`/exit stay consistent).
|
|
408
|
-
- **`question_asked` / `questions_count_max` / `gate_answers_delivered
|
|
409
|
-
`controlOut`. Without it, a loud `::warning::` fires and these keys are
|
|
410
|
-
passed). The authoritative list is `contentKeys` in `src/run/cassette.ts`;
|
|
411
|
-
it — consult it for the full table.
|
|
408
|
+
- **`question_asked` / `questions_count_max` / `gate_answers_delivered` / `gate_answer_count_min`**
|
|
409
|
+
additionally require `controlOut`. Without it, a loud `::warning::` fires and these keys are
|
|
410
|
+
excluded (not vacuously passed). The authoritative list is `contentKeys` in `src/run/cassette.ts`;
|
|
411
|
+
`docs/cassette.md` mirrors it — consult it for the full table.
|
|
412
|
+
- **`gate_answers_delivered: true` passes vacuously when zero `AskUserQuestion` gates fired** (gate
|
|
413
|
+
firing is model-dependent). Pair it with `gate_answer_count_min: <N>` to also require that at least
|
|
414
|
+
N gates fired AND were delivered non-error — the presence companion, mirroring
|
|
415
|
+
`computer_links_resolve` (zero-passes) paired with `transcript_contains` (presence). Both keys fail
|
|
416
|
+
evidence-unavailable, never vacuous-pass, when gate-delivery telemetry itself is absent from
|
|
417
|
+
`result.json` (an old/partial run on the verify-run lane).
|
|
418
|
+
- **`questions_count_max` counts sub-questions, not `AskUserQuestion` tool calls/gates.** A bundled
|
|
419
|
+
gate with K sub-questions counts as K (`src/run/run.ts`'s recorder pushes one `rec.questions` entry
|
|
420
|
+
per sub-question; `src/assert.ts` compares against that count). `trace --view questions` shows the
|
|
421
|
+
same per-gate sub-question count and a matching footer total, so the two surfaces agree — see
|
|
422
|
+
`docs/cassette.md` / `docs/scenario.md`'s `questions_count_max` row.
|
|
412
423
|
|
|
413
424
|
**Assertion source (replay):** by default `replay` evaluates the `assert:` block **frozen in the cassette**
|
|
414
425
|
— byte-deterministic and independent of the working tree. When a sibling scenario resolves and its `assert:`
|
|
@@ -493,19 +504,25 @@ and a fail observed) always fails that batch, regardless of the numeric rate.
|
|
|
493
504
|
"egress": [{ "host","decision":"allow|deny" }],
|
|
494
505
|
"assertions":[{ "assertion": <Assertion>, "pass": bool, "message?": "string" }],
|
|
495
506
|
"subagents": [{ "toolUseId","parentToolUseId?","agentType","declaredTools":[],"toolsUsed":[] }],
|
|
496
|
-
"
|
|
507
|
+
"nonReproducibleAnswers?":[{ "question","chosen","by","rationale?","model?" }], // decisions answered by a non-deterministic/non-authoritative source (llm/external/human/first); scripted answers are excluded
|
|
497
508
|
"usage?": { "turns?": number, /* …SDK usage fields (input_tokens, output_tokens, etc.), pass-through */ },
|
|
498
509
|
"cost?": { "usd?": number, "raw?": {...} }, // usd = SDK's total_cost_usd for this invocation; raw = the api_metrics event payload (independent source)
|
|
499
510
|
"durationMs?": number,
|
|
511
|
+
"fingerprint?": { "baseline", "skillHash?", "skillSources?":[], "skillScope?":[], "sharedHash?", "contentSig?", "fileSigs?":[["relpath","contentSha"]], "fileSigsOmitted?": bool, "mode?": "git|raw", "agentScope?": "skill" }, // skill/plugin staleness fingerprint recorded at run time; lets `verify-run` detect a kept run whose gate snapshot predates a skill change
|
|
500
512
|
"outDir": "string",
|
|
501
513
|
"workDir?": "string", // the agent's working root (mnt/) inside the run dir
|
|
502
514
|
"outputsDir?": "string", // the user-visible deliverable mount (mnt/outputs)
|
|
515
|
+
"userVisibleRoots?": ["string"], // user-visible mount roots (relative to mnt/) — `outputs` plus each connected folder's resolved mount name; plugins excluded
|
|
516
|
+
"readonlyFolderRoots?": ["string"], // subset of userVisibleRoots that are read-only (mode:"r") connected-folder mounts — inputs, not deliverables; `artifacts` excludes them
|
|
517
|
+
"artifacts?": [{ "path","bytes" }], // files written under the user-visible roots (paths + sizes only — no content snapshot)
|
|
518
|
+
"preRunPaths?": ["string"], // workRoot-relative paths under the user-visible roots that existed BEFORE the agent ran — the `no_unexpected_files` baseline; absent when the tier didn't capture it (microvm) or the run predates the seam
|
|
503
519
|
"effectiveFidelity?": "string", // tier actually used (differs from `fidelity` when "cowork" resolved)
|
|
504
520
|
"nonDeterministic?": bool, // true if any decision came from a non-deterministic source → not reproducible
|
|
505
521
|
"gateProvenance?": { "total": number, "bySource": {…}, "gates": [{ "question","answeredBy","answer","model?" }] }, // how each AskUserQuestion gate was answered; informational (never fails the verdict); live/partial lane only (absent on replay)
|
|
506
522
|
"permissiveAutoAllow?": ["string"], // tools auto-allowed by cowork parity that real Cowork BLOCKS → green is NOT faithful
|
|
507
523
|
"staleness?": [{ "class": "baseline|skill|shared-root|format|unverifiable-baseline|unverifiable-skill|resolved-tier|unverifiable-tier", "message" }], // replay only; cassette-staleness findings, surfaced for a JSON gate. Non-failing by default (a stale but passing replay stays ok:true); `--strict` fails on every class, `--fail-on-skill-drift` on skill/shared-root/unverifiable-skill only. `resolved-tier` = a `fidelity: cowork` cassette's recorded effectiveFidelity no longer matches the tier the scenario's baseline (pinned `baseline:` or `latest`) resolves to today — the recording exercises the wrong tier; `unverifiable-tier` = the tier check couldn't run for a baseline-dependent (`fidelity: cowork`) cassette (no recorded effectiveFidelity, or its pinned baseline failed to load). Tier resolution is baseline-only (the CLAUDE_FORCE_HOST_LOOP env override is suppressed) so verify results can't differ across machines.
|
|
508
524
|
"skippedAssertions?": { "full": number, "partial": number }, // replay only; count of live-only assertions NOT evaluated (full = whole assertion skipped; partial = content half ran, fs/egress half dropped). The skipped ones are absent from `assertions[]`.
|
|
525
|
+
"toolResults?": [{ "toolUseId?","isError","text","assertText?" }], // tool-result text at assertion-fidelity cap (10 KB); backs tool_result_contains/tool_result_not_contains
|
|
509
526
|
"skillsInvoked?": ["string"], // Wave 1: skill/plugin ids invoked via the Skill tool_use event, call order, duplicates kept. Backs skill_triggered/no_skill_triggered.
|
|
510
527
|
"skillToolAvailable?": bool // Wave 1: whether the agent's init tool list included "Skill" — false ⇒ skill_triggered/no_skill_triggered fail as evidence-unavailable (agent-version drift)
|
|
511
528
|
}
|
|
@@ -630,9 +647,10 @@ nothing here is guaranteed; minor versions may break any surface — see [RELEAS
|
|
|
630
647
|
`findings` / `staleness` / `notes` / `version` / `error` channels. This is the machine output the
|
|
631
648
|
CI recipes and the packaged Action steer consumers to parse; renaming or removing a key is
|
|
632
649
|
breaking, adding one is not.
|
|
633
|
-
- **Cassette format** — the current `cassetteVersion` (**
|
|
634
|
-
Older-version cassettes (the retained
|
|
635
|
-
published) stay replayable; dropping a
|
|
650
|
+
- **Cassette format** — the current `cassetteVersion` (**8**, `schema/cassette.v8.json`) and its
|
|
651
|
+
verdict-modifier assertion keys. Older-version cassettes (the retained
|
|
652
|
+
`schema/cassette.v2/v3/v5/v6/v7.json` — no v4 schema was ever published) stay replayable; dropping a
|
|
653
|
+
still-emitted version's readability is breaking.
|
|
636
654
|
- **Control protocol** — `schema/protocol.v1.json` + the golden control-response vectors (§5).
|
|
637
655
|
- **Environment variables** — the documented `COWORK_HARNESS_*` knobs plus `COWORK_AGENT_BINARY` and
|
|
638
656
|
`COWORK_AGENT_IMAGE`. Renaming a documented var or changing its meaning is breaking.
|
package/dist/assert.js
CHANGED
|
@@ -166,7 +166,7 @@ function check(a, ctx) {
|
|
|
166
166
|
const results = [];
|
|
167
167
|
const ok = () => ({ pass: true });
|
|
168
168
|
const fail = (message) => ({ pass: false, message });
|
|
169
|
-
const truncated = ctx.truncatedPaths ?? new
|
|
169
|
+
const truncated = ctx.truncatedPaths ?? new Map();
|
|
170
170
|
if (a.transcript_contains !== undefined)
|
|
171
171
|
results.push(ctx.transcriptMissing
|
|
172
172
|
? fail(`evidence unavailable: transcript sidecar (run.jsonl) absent — cannot evaluate transcript_contains`)
|
|
@@ -471,10 +471,13 @@ function check(a, ctx) {
|
|
|
471
471
|
// run/cassette, an unobserved delivery (delivered=null) is NOT neutral — it is absence of the
|
|
472
472
|
// evidence the assertion requires, so it fails loud ("no silent false-greens"). `delivered:
|
|
473
473
|
// false` is a real errored tool_result; `null` is "no tool_result observed for this gate".
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
474
|
+
// Zero gates fired passes vacuously (whether a gate fires is model-dependent) — pair with
|
|
475
|
+
// gate_answer_count_min to also require presence. Missing telemetry (gateDeliveriesMissing)
|
|
476
|
+
// is NOT the same as zero gates and must fail evidence-unavailable, not vacuous-pass.
|
|
477
|
+
if (ctx.gateDeliveriesMissing) {
|
|
478
|
+
results.push(fail(`evidence unavailable: gate-delivery telemetry absent from result.json — cannot evaluate gate_answers_delivered`));
|
|
479
|
+
}
|
|
480
|
+
else if (a.gate_answers_delivered) {
|
|
478
481
|
const bad = ctx.gateDeliveries.filter((g) => g.delivered !== true);
|
|
479
482
|
results.push(bad.length === 0
|
|
480
483
|
? ok()
|
|
@@ -492,6 +495,17 @@ function check(a, ctx) {
|
|
|
492
495
|
results.push(failedConfirmed.length > 0 ? ok() : fail(`expected a confirmed gate-delivery failure but none was observed`));
|
|
493
496
|
}
|
|
494
497
|
}
|
|
498
|
+
if (a.gate_answer_count_min !== undefined) {
|
|
499
|
+
if (ctx.gateDeliveriesMissing) {
|
|
500
|
+
results.push(fail(`evidence unavailable: gate-delivery telemetry absent from result.json — cannot evaluate gate_answer_count_min`));
|
|
501
|
+
}
|
|
502
|
+
else {
|
|
503
|
+
const delivered = ctx.gateDeliveries.filter((g) => g.delivered === true).length;
|
|
504
|
+
results.push(delivered >= a.gate_answer_count_min
|
|
505
|
+
? ok()
|
|
506
|
+
: fail(`only ${delivered} gate answer(s) confirmed delivered, need ≥ ${a.gate_answer_count_min}`));
|
|
507
|
+
}
|
|
508
|
+
}
|
|
495
509
|
if (a.artifact_json !== undefined) {
|
|
496
510
|
const aj = a.artifact_json;
|
|
497
511
|
const file = containedPath(ctx.workRoot, aj.artifact);
|
|
@@ -500,12 +514,41 @@ function check(a, ctx) {
|
|
|
500
514
|
else {
|
|
501
515
|
// verify the real path (after symlink resolution) is still under workRoot.
|
|
502
516
|
const realFile = containedRealPath(ctx.workRoot, file);
|
|
517
|
+
// A body-less manifest entry (a read-only connected-folder input, or an artifact over the body
|
|
518
|
+
// cap) has no content in the cassette — artifact_json cannot be evaluated on replay (the 0-byte
|
|
519
|
+
// placeholder isn't parseable). To keep record/verify-run/replay SYMMETRIC (no green-record →
|
|
520
|
+
// red-replay), treat such a target as evidence-unavailable on EVERY lane: `truncatedPaths` flags
|
|
521
|
+
// it on replay; `readonlyFolderRoots` flags the read-only-input case on the live/verify-run lanes
|
|
522
|
+
// where the real file is still on disk. (Existence keys stay green — existence is provable from
|
|
523
|
+
// the recorded hash — but content is genuinely absent, so this fails loud, never vacuous.)
|
|
524
|
+
const rel = relative(resolve(ctx.workRoot), file);
|
|
525
|
+
// Reason sources by lane: LIVE/verify-run derive read-only from the plan's `readonlyFolderRoots`
|
|
526
|
+
// (no manifest exists at eval time); REPLAY reads the per-entry `truncationReason` off the
|
|
527
|
+
// materialized manifest (`truncated.get(rel)`). Keeping both is complementary, not redundant.
|
|
528
|
+
const liveReadonly = (ctx.readonlyFolderRoots ?? []).some((pre) => rel === pre || rel.startsWith(pre + "/"));
|
|
529
|
+
const replayReason = truncated.get(rel); // undefined if not body-less on replay, or a pre-v8 entry with no reason
|
|
530
|
+
const isReadonlyInput = liveReadonly || replayReason === "readonly";
|
|
531
|
+
const isOverCap = replayReason === "size";
|
|
532
|
+
const bodyLess = truncated.has(rel) || liveReadonly;
|
|
503
533
|
if (!realFile) {
|
|
504
534
|
results.push(fail(`unsafe artifact_json path "${aj.artifact}" — symlink target escapes the work root`));
|
|
505
535
|
}
|
|
506
536
|
else if (!existsSync(realFile)) {
|
|
507
537
|
results.push(fail(`artifact_json: file not found: ${aj.artifact} (under ${ctx.workRoot})`));
|
|
508
538
|
}
|
|
539
|
+
else if (bodyLess) {
|
|
540
|
+
// Precise remedy when the cause is known (read-only ⇒ assert on a deliverable; over-cap ⇒ raise
|
|
541
|
+
// the cap). A pre-v8 entry carries no reason ⇒ name both causes (we can't tell). "unreadable"
|
|
542
|
+
// also falls here — it's a record-time read failure, so the both-causes text is the safe hint.
|
|
543
|
+
const cause = isReadonlyInput
|
|
544
|
+
? `(read-only connected-folder input — its content is never captured; assert artifact_json on a deliverable instead)`
|
|
545
|
+
: isOverCap
|
|
546
|
+
? `(larger than the artifact-body cap — raise --max-artifact-bytes to capture it)`
|
|
547
|
+
: `(a read-only connected-folder input, or an artifact larger than the body cap — if an input, assert on a deliverable; if a large deliverable, raise --max-artifact-bytes)`;
|
|
548
|
+
results.push(fail(`evidence unavailable: artifact_json target "${aj.artifact}" was captured body-less ` +
|
|
549
|
+
cause +
|
|
550
|
+
` — content is not in the cassette, so it cannot be evaluated on replay`));
|
|
551
|
+
}
|
|
509
552
|
else {
|
|
510
553
|
let doc;
|
|
511
554
|
let parsed = true;
|