cowork-harness 0.23.0 → 0.24.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +22 -11
- package/.claude/skills/cowork-harness/references/ci-recipe.md +25 -5
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +11 -6
- package/.claude/skills/cowork-harness/references/task-recipes.md +108 -0
- package/.claude/skills/cowork-harness/scripts/assertion-keys.json +1 -0
- package/.claude/skills/cowork-harness/scripts/scenario.py +69 -3
- package/.cowork-redact.json +11 -0
- package/CHANGELOG.md +128 -0
- package/CONTRIBUTING.md +1 -0
- package/README.md +4 -3
- package/SPEC.md +40 -19
- package/baselines/desktop-1.18286.0.json +13 -3
- package/baselines/prompts/desktop-1.18286.0/system-prompt-append.md +1 -2
- package/dist/assert.js +17 -1
- package/dist/cli.js +72 -10
- package/dist/glob.js +43 -0
- package/dist/run/artifacts.js +109 -0
- package/dist/run/cassette.js +239 -20
- package/dist/run/execute.js +24 -84
- package/dist/run/pre-run-manifest.js +61 -0
- package/dist/runtime/container.js +3 -0
- package/dist/runtime/hostloop.js +12 -2
- package/dist/runtime/lima.js +3 -3
- package/dist/runtime/microvm.js +5 -0
- package/dist/runtime/protocol.js +3 -0
- package/dist/scan.js +1 -1
- package/dist/session.js +6 -0
- package/dist/sync/cowork-sync.js +6 -1
- package/dist/types.js +5 -1
- package/docker/Dockerfile.agent +8 -4
- package/docs/cassette.md +17 -5
- package/docs/fidelity-gaps.md +62 -10
- package/docs/scenario.md +2 -1
- package/examples/replays/README.md +1 -1
- package/package.json +2 -1
- package/python/test_scenario_lint.py +71 -0
- package/schema/cassette.v7.json +18 -0
- package/schema/run-result.json +32 -0
- package/schema/scenario.schema.json +8 -0
- package/schema/verify-cassettes.json +70 -0
- package/scripts/boot-rootfs-vz.ts +4 -4
- package/scripts/build-rootfs-image.ts +1 -1
- package/scripts/capture-rootfs-manifest.ts +1 -1
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 0.
|
|
7
|
-
tracks-harness: cowork-harness 0.
|
|
6
|
+
version: 0.24.0
|
|
7
|
+
tracks-harness: cowork-harness 0.24.0 (baseline desktop-1.18286.0)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.24.0` (baseline
|
|
26
26
|
> `desktop-1.18286.0`). If your checkout is newer, prefer the live `--help`, `SPEC.md`, and
|
|
27
27
|
> `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
|
|
|
38
38
|
Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.
|
|
41
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.24.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.24.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.24.0"`. **Pin `@>=0.24.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.24.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
|
|
42
42
|
- **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
|
|
43
43
|
- **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
|
|
44
44
|
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
@@ -288,10 +288,14 @@ cowork-harness lint scenarios/*.yaml
|
|
|
288
288
|
```
|
|
289
289
|
|
|
290
290
|
`lint` flags: filesystem/egress-only assertions on a `replay` gate (silent no-op), bad regex
|
|
291
|
-
quoting, an egress assert on `protocol` fidelity,
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
291
|
+
quoting, an egress assert on `protocol` fidelity, `transcript_no_host_path` on `hostloop`/`protocol`
|
|
292
|
+
(ERROR — fails by design at those tiers; WARN on `fidelity: cowork`, whose tier resolves per the
|
|
293
|
+
baseline's host-loop gate), non-empty `requires_capabilities` on `protocol` without
|
|
294
|
+
`allow_missing_capability` (ERROR — the capability probe can't run there, so the run hard-fails as
|
|
295
|
+
unverifiable), a `controlOut`-gated key on a non-`controlOut` replay, mixed-class assertion items,
|
|
296
|
+
and hallucinated schema (`assertions:` vs `assert:`, unknown keys). Exit code is non-zero on errors
|
|
297
|
+
(CI-friendly). `scaffold` auto-upgrades the tier if you ask for egress on `protocol`, so it never
|
|
298
|
+
emits a scenario `lint` would reject.
|
|
295
299
|
|
|
296
300
|
CI placement: a **token-free `replay` PR gate** (content/structure only) + a **nightly live `run`**
|
|
297
301
|
(filesystem/egress). Fastest setup: `uses: yaniv-golan/cowork-harness@v1` (a packaged GitHub Action with a
|
|
@@ -362,9 +366,12 @@ are the ones that bite hardest.
|
|
|
362
366
|
sub-agent used other tools. And `subagent_dispatched` matches by dispatch **description** too —
|
|
363
367
|
skills often dispatch with no `subagent_type` (`agentType:"unknown"`), so match the description.
|
|
364
368
|
|
|
365
|
-
6. **`dispatch_count_max`
|
|
366
|
-
|
|
367
|
-
dispatch
|
|
369
|
+
6. **`dispatch_count_max` is an author-chosen budget, not a production cap.** It's a post-hoc count
|
|
370
|
+
assertion: passing means "happened to dispatch ≤N this run," nothing more. Cowork imposes **no**
|
|
371
|
+
in-conversation `Task`-dispatch cap — gate `1648655587`'s `{perTask:1, global:3}` governs the
|
|
372
|
+
separate scheduled/cron-task session scheduler, not the `Task` tool (binary-verified; see SPEC
|
|
373
|
+
§10). So there is no "skip-on-cap" for the harness to reproduce; use this key only to catch a
|
|
374
|
+
fan-out you don't want.
|
|
368
375
|
|
|
369
376
|
7. **`protocol` is rejected (not silently passed) if the scenario asserts egress** — boundary
|
|
370
377
|
assertions need a sandboxed tier (`container`+). Good: this one fails loud by design.
|
|
@@ -456,6 +463,10 @@ and the CI recipe, read the files in `references/`.
|
|
|
456
463
|
|
|
457
464
|
## References
|
|
458
465
|
|
|
466
|
+
- `references/task-recipes.md` — end-to-end recipes for the four jobs fleet owners actually hit:
|
|
467
|
+
evolve a cassette's `assert:` (usually no re-record), audit a fleet for tier drift, set up
|
|
468
|
+
redaction before the first hostloop/protocol record, derive budget assertions without a
|
|
469
|
+
two-pass record. Start here when the question is "how do I do X", not "what does flag Y mean".
|
|
459
470
|
- `references/scenario-schema.md` — scenario/session YAML schema, full assertion catalog (with each
|
|
460
471
|
key's replay class), the web_fetch model, and the complete gotcha list.
|
|
461
472
|
- `references/fidelity-and-answers.md` — fidelity tiers, answer paths, the determinism contract.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.24.0` (baseline `desktop-1.18286.0`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -53,7 +53,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
53
53
|
GitHub-hosted runners, no token/Docker/agent:
|
|
54
54
|
|
|
55
55
|
```yaml
|
|
56
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
56
|
+
- run: npm i -g "cowork-harness@>=0.24.0"
|
|
57
57
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
58
58
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
59
59
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -125,9 +125,17 @@ dollar figures). In a skill repo these cassettes get **committed**. So:
|
|
|
125
125
|
customer's cap table.
|
|
126
126
|
- **Opt-in redaction** rewrites configured PII out of the cassette at record time. Drop a
|
|
127
127
|
`.cowork-redact.json` next to your scenarios (or set `COWORK_HARNESS_REDACT_PATTERNS` /
|
|
128
|
-
`_KEYS`); empty by default.
|
|
128
|
+
`_KEYS`); empty by default. The policy is searched in **cwd → the scenario's dir → the cassette's
|
|
129
|
+
dir** (first file found per dir; env vars merge on top). `cowork-harness init-redact` copies the
|
|
130
|
+
packaged reference template (local-path prefixes + a generic email regex) into the cwd as a starting
|
|
131
|
+
point — review and tailor it. Redaction is **verdict-preserving** — `record` refuses to write if
|
|
129
132
|
redaction would flip an assertion (a manufactured green). `--no-redact` skips it for known-synthetic
|
|
130
133
|
inputs.
|
|
134
|
+
- **Pre-spawn preflight**: `record` warns (`::warning::`, before the paid run starts — once per batch
|
|
135
|
+
for `record <dir>` / `--rerecord-stale`) when a scenario about to record at a host-path-bearing tier
|
|
136
|
+
(`hostloop`, `protocol`) has an **empty** redaction policy — that combination commits real host paths
|
|
137
|
+
the `path` scanner then hard-fails at `verify-cassettes` time. The always-on scanner remains the
|
|
138
|
+
universal net (container-tier recordings can trip it too).
|
|
131
139
|
- **Always-on scan gate** — `verify-cassettes` flags email / currency / bare-domain / local-path /
|
|
132
140
|
machine-inventory matches it finds in the committed cassettes and **exits non-zero**, so "no leak" is
|
|
133
141
|
a gate, not discipline.
|
|
@@ -179,7 +187,7 @@ jobs:
|
|
|
179
187
|
with: { node-version: '20' }
|
|
180
188
|
- uses: actions/setup-python@v5
|
|
181
189
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
182
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
190
|
+
- run: npm i -g "cowork-harness@>=0.24.0"
|
|
183
191
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
184
192
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
185
193
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -208,7 +216,7 @@ jobs:
|
|
|
208
216
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
209
217
|
fi
|
|
210
218
|
- if: steps.guard.outputs.live == 'true'
|
|
211
|
-
run: npm i -g "cowork-harness@>=0.
|
|
219
|
+
run: npm i -g "cowork-harness@>=0.24.0"
|
|
212
220
|
- if: steps.guard.outputs.live == 'true'
|
|
213
221
|
run: cowork-harness run scenarios/ --output-format json
|
|
214
222
|
env:
|
|
@@ -235,6 +243,11 @@ scenario = `result === "success" && assertions.every(pass)`. Exit code is non-ze
|
|
|
235
243
|
fails or a run errors, so a plain `cowork-harness run scenarios/` is already CI-ready without parsing
|
|
236
244
|
JSON.
|
|
237
245
|
|
|
246
|
+
`verify-cassettes` emits its **own** envelope (`{command, ok, coverage, results[]}` with per-file
|
|
247
|
+
`findings`/`staleness`/`notes`/`version`/`error`), published as `schema/verify-cassettes.json` in the
|
|
248
|
+
npm package. Both envelope schemas are covered 1.0 contract surfaces (SPEC §12) — parse the JSON, not
|
|
249
|
+
the human-readable text (which is explicitly NOT stable).
|
|
250
|
+
|
|
238
251
|
A run writes to `~/.cowork-harness/runs/<name>/<sessionId>/` by default — outside any working tree. In CI,
|
|
239
252
|
set `COWORK_HARNESS_RUNS_DIR` (or pass `--run-dir`) to a workspace-relative path (e.g. `runs`) so an
|
|
240
253
|
artifact-upload step can collect them. Each run dir holds `events.jsonl`, `control-out.jsonl`, `run.jsonl`,
|
|
@@ -261,6 +274,13 @@ does **not** imply the recording is still valid. Each replay result carries `sta
|
|
|
261
274
|
| `format` | recorded under an older hash format | re-record once |
|
|
262
275
|
| `unverifiable-baseline` | the latest baseline couldn't be loaded | couldn't verify (env, not skill) |
|
|
263
276
|
| `unverifiable-skill` | skill dirs unresolvable — skill staleness couldn't be checked | couldn't verify the skill |
|
|
277
|
+
| `resolved-tier` | a `fidelity: cowork` cassette's recorded `effectiveFidelity` no longer matches what the baseline resolves to today (the host-loop gate flipped) | **high** (the recording exercises the wrong tier) |
|
|
278
|
+
| `unverifiable-tier` | tier check couldn't run for a `fidelity: cowork` cassette (no recorded `effectiveFidelity`, or its pinned baseline failed to load) | couldn't verify the tier — re-record |
|
|
279
|
+
|
|
280
|
+
(A pre-`effectiveFidelity` cassette with an **explicit** tier is statically knowable — it passes the tier
|
|
281
|
+
check with a non-failing informational note in the `verify-cassettes` envelope's per-file `notes[]`, a
|
|
282
|
+
`·`-prefixed row in text output. On `verify-cassettes` every staleness *finding* above is a hard fail —
|
|
283
|
+
the gate is class-blind; notes never fail it.)
|
|
264
284
|
|
|
265
285
|
To gate in CI, pick the severity you want:
|
|
266
286
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.24.0` (baseline `desktop-1.18286.0`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.24.0`
|
|
4
4
|
(baseline `desktop-1.18286.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
@@ -219,6 +219,7 @@ passes only if every key passes. Keep one concern per item unless you mean conju
|
|
|
219
219
|
| `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it |
|
|
220
220
|
| `user_visible_artifact: <path>` | exists **and** under a user-visible root (`outputs/` + each connected folder's mount name) — the right primitive for a workspace deliverable when a folder is connected |
|
|
221
221
|
| `no_delete_in_outputs: true` | no delete op touched `mnt/outputs` — **only `true` is valid**; `false` is rejected (omit to allow deletes) |
|
|
222
|
+
| `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); **microvm cannot capture** (use container/hostloop); replay needs `cassette.preRunPaths` (≥0.24 container/hostloop recordings) — cassettes without it **exclude** the key with a loud warning |
|
|
222
223
|
| `self_heal_ran: <bool>` | a plugin-root self-heal script was (not) invoked |
|
|
223
224
|
| `tool_called: <Tool>` | the agent invoked the tool (actually ran it) |
|
|
224
225
|
| `tool_not_called: <Tool>` | the agent never invoked it |
|
|
@@ -228,7 +229,7 @@ passes only if every key passes. Keep one concern per item unless you mean conju
|
|
|
228
229
|
| `subagent_tool_absent: <Tool>` | no sub-agent used the tool |
|
|
229
230
|
| `subagent_dispatched: <regex>` | a sub-agent whose `agentType` **or dispatch description** matches |
|
|
230
231
|
| `subagent_declared_but_unused: <Tool>` | a sub-agent declared the tool but never used **that** tool (even if it used others) |
|
|
231
|
-
| `dispatch_count_max: <N>` | at most N sub-agents dispatched (records only
|
|
232
|
+
| `dispatch_count_max: <N>` | at most N sub-agents dispatched — an author-chosen budget (Cowork imposes no in-conversation Task-dispatch cap; records only, enforces nothing — see gotcha 12) |
|
|
232
233
|
| `skill_triggered: <regex>` | a skill matching the regex (invoked id, e.g. `"plugin:skill"`) was invoked via the `Skill` tool — evidence-unavailable (not a normal fail) if the agent's init tools have no `Skill` tool |
|
|
233
234
|
| `no_skill_triggered: <regex>` | no invoked skill id matched — the negative-control / description-collision catcher; evidence-unavailable (never a vacuous pass) if invocation data is absent or the `Skill` tool is unobservable |
|
|
234
235
|
| `max_cost_usd: <N>` | the run's SDK-reported cost is ≤ N USD — evidence-unavailable if cost telemetry is absent. **Replay asserts the frozen recording's cost, not fresh spend** — a real regression needs a live `run` |
|
|
@@ -302,7 +303,9 @@ small-file JSON `body` inlined; a hash-only entry still satisfies `file_exists`.
|
|
|
302
303
|
resolves a `/sessions/…/mnt/…`-shaped link directly against the manifest, and a host-shaped (hostloop) link
|
|
303
304
|
by first normalizing it to a mount-relative path (recorded connected-folder prefixes + the outputs/uploads
|
|
304
305
|
mounts) — replay has no live filesystem to check a host path against directly (that only happens on a live
|
|
305
|
-
`run`/`verify-run`). Without a manifest (older cassettes) all
|
|
306
|
+
`run`/`verify-run`). Without a manifest (older cassettes) all five are skipped; `no_unexpected_files` also
|
|
307
|
+
needs `preRunPaths` (≥0.24 recordings) — without it the key is excluded with a loud warning (live/verify-run
|
|
308
|
+
hard-fails evidence-unavailable instead). A green replay re-confirms
|
|
306
309
|
*record-time* artifacts, not that the current skill still produces them — `replay --strict` fails when the
|
|
307
310
|
staleness `fingerprint` shows ANY skill/baseline drift, or `replay --fail-on-skill-drift` only on
|
|
308
311
|
skill-source drift; every replay result also reports it class-tagged in `staleness[]` for a JSON gate.
|
|
@@ -410,9 +413,11 @@ at the top of this file.
|
|
|
410
413
|
11. **`subagent_declared_but_unused` fires on declared-but-didn't-use-THAT-tool**, even if the
|
|
411
414
|
sub-agent used other tools.
|
|
412
415
|
|
|
413
|
-
12. **`dispatch_count_max`
|
|
414
|
-
|
|
415
|
-
|
|
416
|
+
12. **`dispatch_count_max` is an author-chosen budget, not a production cap.** It records the count
|
|
417
|
+
and asserts on it; passing means "dispatched ≤N this run," not "the harness capped it." Cowork
|
|
418
|
+
imposes no in-conversation Task-dispatch cap to reproduce — gate `1648655587`'s
|
|
419
|
+
`{perTask:1, global:3}` is the scheduled/cron-task session limiter, a different mechanism
|
|
420
|
+
(binary-verified; SPEC §10).
|
|
416
421
|
|
|
417
422
|
13. **`protocol` is rejected (not silently passed) if the scenario asserts egress** — boundary
|
|
418
423
|
assertions need `container`+. Fails loud by design.
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
# Task recipes — end-to-end paths for the four jobs consumers actually reverse-engineer
|
|
2
|
+
|
|
3
|
+
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
|
+
decision path. Every one answers a question a real fleet owner had to work out the hard way. Facts track the harness version in SKILL.md's
|
|
5
|
+
front-matter; the `resolved-tier`/`unverifiable-tier` staleness classes (Recipe 2) and
|
|
6
|
+
`init-redact` (Recipe 3) are new in the release AFTER 0.23.0.
|
|
7
|
+
|
|
8
|
+
## Recipe 1 — Evolve the `assert:` block of an existing cassette (usually NO re-record)
|
|
9
|
+
|
|
10
|
+
You added or changed assertion keys in a scenario whose cassette is already committed. Do you need
|
|
11
|
+
a paid live re-record? Walk this tree — the answer is usually no:
|
|
12
|
+
|
|
13
|
+
1. **Content / always-replay keys** (`transcript_*`, `tool_called`/`tool_not_called`,
|
|
14
|
+
`tool_result_*`, `subagent_*`, `dispatch_count_max`, `skill_triggered`/`no_skill_triggered`,
|
|
15
|
+
`result`, `max_turns`, `tool_calls_max`, `max_cost_usd`, `max_tokens`) →
|
|
16
|
+
`cowork-harness replay <cassette> --assert-from <scenario.yaml>`. Token-free, no re-record.
|
|
17
|
+
If the recording genuinely lacks the telemetry a key needs (very old cassettes), the key fails
|
|
18
|
+
**loud** as `evidence-unavailable` — that is correct behavior, not a bug; only then re-record.
|
|
19
|
+
2. **Gate keys** (`question_asked`, `questions_count_max`, `gate_answers_delivered`) on a cassette
|
|
20
|
+
**with `controlOut`** (any modern recording) → same token-free `--assert-from` path.
|
|
21
|
+
3. **Gate keys** on a **pre-`controlOut`** cassette → one re-record unlocks gate asserts for that
|
|
22
|
+
cassette permanently.
|
|
23
|
+
4. **Filesystem / egress keys** (`file_exists` without an artifact manifest, `egress_allowed`,
|
|
24
|
+
`egress_denied`, `no_delete_in_outputs`) → live lane **by design**; replay skips them with a
|
|
25
|
+
loud `::warning::`. Keep them in the scenario, run them on the nightly live gate.
|
|
26
|
+
|
|
27
|
+
**The one hard caveat:** `--assert-from` **hard-fails on recording-shaping drift** — if the
|
|
28
|
+
scenario's `prompt:`, `answers:`, baseline, or skill content differ from what shaped the
|
|
29
|
+
recording, re-asserting under them is meaningless and the harness refuses. The trap you'll
|
|
30
|
+
actually hit is `answers:`: sequence your edits — land assert-only changes (token-free), and land
|
|
31
|
+
answer changes separately (those DO force a re-record).
|
|
32
|
+
|
|
33
|
+
## Recipe 2 — Audit a fleet for tier drift
|
|
34
|
+
|
|
35
|
+
A `fidelity: cowork` scenario records at whatever tier the baseline's host-loop gate resolves to
|
|
36
|
+
at record time. When Cowork flips that gate, your committed cassettes silently stop representing
|
|
37
|
+
what production would do. Two layers of defense:
|
|
38
|
+
|
|
39
|
+
- **Automated (preferred):** `cowork-harness verify-cassettes <dir>` raises a
|
|
40
|
+
**`resolved-tier`** staleness finding when a `fidelity: cowork` cassette's recorded
|
|
41
|
+
`effectiveFidelity` no longer matches what the current baseline resolves to, and
|
|
42
|
+
**`unverifiable-tier`** when a `fidelity: cowork` cassette predates `effectiveFidelity`
|
|
43
|
+
entirely (can't verify → re-record). Explicit-tier cassettes (`hostloop`, `container`, …) are
|
|
44
|
+
statically knowable and only produce a non-failing informational note.
|
|
45
|
+
- **Manual (one-liner):** `grep -h '"effectiveFidelity"' cassettes/*.cassette.json | sort | uniq -c`
|
|
46
|
+
shows the tier distribution of the fleet at a glance.
|
|
47
|
+
|
|
48
|
+
### Cassette anatomy (what you're looking at when you open one)
|
|
49
|
+
|
|
50
|
+
Top-level fields of a `*.cassette.json` (schema `schema/cassette.v7.json`):
|
|
51
|
+
|
|
52
|
+
| Field | What it is |
|
|
53
|
+
|---|---|
|
|
54
|
+
| `$schema`, `generator`, `cassetteVersion` | Provenance: schema URL, producing tool, format version (current: 7) |
|
|
55
|
+
| `scenario` | The embedded scenario snapshot at record time |
|
|
56
|
+
| `events` | The recorded agent event stream (the replay source) |
|
|
57
|
+
| `controlOut` | Driver→agent control responses — presence unlocks gate asserts on replay |
|
|
58
|
+
| `effectiveFidelity` | The tier the live record actually resolved to (the drift-audit key above) |
|
|
59
|
+
| `artifacts` | Output-file manifest (paths + hashes + small inlined bodies) — unlocks `file_exists`/`artifact_json` on replay |
|
|
60
|
+
| `fingerprint` | Skill/baseline staleness tripwire |
|
|
61
|
+
| `userVisibleRoots` | The user-visible mount roots captured at record time |
|
|
62
|
+
| `preRunPaths` | Pre-run file-path baseline for `no_unexpected_files` (workRoot-relative; co-present with `userVisibleRoots`) |
|
|
63
|
+
| `scenarioSource` | Relative path to the authored YAML this was recorded from |
|
|
64
|
+
| `authoring` | Present iff a live decider answered ≥1 gate during recording (`nonDeterministic: true`) |
|
|
65
|
+
|
|
66
|
+
## Recipe 3 — Set up redaction BEFORE your first hostloop/protocol record
|
|
67
|
+
|
|
68
|
+
`hostloop` and `protocol` runs execute on the host, so real host paths (`/Users/you/…`) land in
|
|
69
|
+
the transcript and would be committed inside the cassette. Set the policy up **before** the first
|
|
70
|
+
record — retrofitting means re-recording:
|
|
71
|
+
|
|
72
|
+
1. `cowork-harness init-redact` — copies the reference `.cowork-redact.json` (host-path +
|
|
73
|
+
email patterns) into the current directory. Version-control it.
|
|
74
|
+
2. **Search set:** record looks for `.cowork-redact.json` in the **cwd, the scenario's directory,
|
|
75
|
+
and the cassette's output directory** — every distinct file found is MERGED (plus
|
|
76
|
+
`COWORK_HARNESS_REDACT_PATTERNS`/`_KEYS` from the env). Repo root (= cwd in CI) is the
|
|
77
|
+
conventional home.
|
|
78
|
+
3. **Preflight backstop:** `record` on a host-path-bearing tier with NO effective policy emits a
|
|
79
|
+
loud pre-spawn `::warning::` (once per batch). If you see it, stop and do step 1.
|
|
80
|
+
4. **Scanner relationship:** redaction (record-time, rewrites content) and the
|
|
81
|
+
`verify-cassettes` privacy scanner (CI-time, read-only tripwire over 4 classes: `domain`,
|
|
82
|
+
`email`, `path`, `machine-inventory`) are independent layers. The scanner catches what
|
|
83
|
+
redaction missed.
|
|
84
|
+
5. **Scoping idioms for scanner findings you've reviewed and accepted:** prefer the class-scoped
|
|
85
|
+
flags — `--allow-domain 'api\.example\.com'`, `--allow-email '.*@yourco\.com'`,
|
|
86
|
+
`--allow-path '/opt/ci-runner/.*'` — over bare `--allow` (which applies to every class and can
|
|
87
|
+
let a domain allow swallow an email leak). For a version-controlled allowlist use
|
|
88
|
+
`--allow-file <path>`: bare all-class patterns, one regex per line, `#` comments and blank
|
|
89
|
+
lines ignored.
|
|
90
|
+
|
|
91
|
+
## Recipe 4 — Budget assertions without a two-pass record
|
|
92
|
+
|
|
93
|
+
`max_turns` / `tool_calls_max` / `questions_count_max` / `dispatch_count_max` need observed
|
|
94
|
+
values, but you don't need a second paid run to get them — derive them from evidence you already
|
|
95
|
+
have:
|
|
96
|
+
|
|
97
|
+
1. **From the run you already did:** `run` always keeps its run dir. Read observed turns from
|
|
98
|
+
`result.json` (`usage.turns`), tool-call and question counts from
|
|
99
|
+
`cowork-harness trace <run-id> --view tools` / `--view questions`, and the dispatch count from
|
|
100
|
+
`--view dispatches`.
|
|
101
|
+
2. **From history:** `cowork-harness stats <scenario> --metric turns` gives p50/p95 turns across
|
|
102
|
+
indexed runs — better than a single observation.
|
|
103
|
+
3. **Set the budget with headroom:** observed × 1.5, rounded up (agentic run lengths vary run to
|
|
104
|
+
run; a budget at the observed value flakes).
|
|
105
|
+
4. **Replay-class note:** `max_turns` / `tool_calls_max` / `dispatch_count_max` re-evaluate
|
|
106
|
+
token-free on replay; `questions_count_max` needs `controlOut` (Recipe 1's tree applies).
|
|
107
|
+
`max_cost_usd` / `max_tokens` on replay assert the FROZEN recording's spend — near-zero signal
|
|
108
|
+
as a regression gate; if you need a cost gate, put it on the live lane via `stats`.
|
|
@@ -16,9 +16,14 @@ Two subcommands:
|
|
|
16
16
|
|
|
17
17
|
lint flags (see references/scenario-schema.md for the why of each):
|
|
18
18
|
E egress assertion on `fidelity: protocol` (the harness rejects this run)
|
|
19
|
+
E `transcript_no_host_path` on hostloop/protocol (fails BY DESIGN at those tiers)
|
|
20
|
+
E `requires_capabilities` on `fidelity: protocol` (probe can't run → hard-fails
|
|
21
|
+
unless allow_missing_capability)
|
|
19
22
|
E on_unanswered: agent / invalid value (schema rejects `agent`)
|
|
20
23
|
E authored `replay_protocol_fidelity` assertion (replay-synthesized only)
|
|
21
24
|
E `assertions:` instead of `assert:` (block ignored → every check no-ops)
|
|
25
|
+
W `transcript_no_host_path` on `fidelity: cowork` (tier resolves per baseline gate —
|
|
26
|
+
incompatible if it lands hostloop)
|
|
22
27
|
W no content assertion → no-op on a replay gate (every assertion is fs/egress)
|
|
23
28
|
W mixed-class assert item → fs/egress half dropped on replay
|
|
24
29
|
W unknown top-level / assertion key (typo or hallucinated schema)
|
|
@@ -72,7 +77,7 @@ GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered"}
|
|
|
72
77
|
# a manifest-less cassette skips them. Since the 0.3.0 artifact-manifest these are NOT always live-only.
|
|
73
78
|
# computer_links_resolve joins this bucket (not CONTENT_KEYS): resolving a non-empty link set needs
|
|
74
79
|
# either a live filesystem or the cassette's artifacts manifest — see cassette.ts's manifestKeys comment.
|
|
75
|
-
MANIFEST_KEYS = {"file_exists", "user_visible_artifact", "artifact_json", "computer_links_resolve"}
|
|
80
|
+
MANIFEST_KEYS = {"file_exists", "user_visible_artifact", "artifact_json", "computer_links_resolve", "no_unexpected_files"}
|
|
76
81
|
# live-only: ALWAYS skipped on replay, with a loud warning (no filesystem, no network on the token-free lane)
|
|
77
82
|
LIVE_ONLY_KEYS = {
|
|
78
83
|
"egress_denied",
|
|
@@ -139,7 +144,7 @@ _EMBEDDED_TOP_LEVEL_KEYS = {
|
|
|
139
144
|
"answers",
|
|
140
145
|
"expect_denied",
|
|
141
146
|
"assert",
|
|
142
|
-
"skills", #
|
|
147
|
+
"skills", # opt-in skill-staleness hash scope
|
|
143
148
|
"requires_capabilities", # Fix 4b: scenario-level required-capability declaration (pre-flight gate)
|
|
144
149
|
"allow_host_writes", # hostloop native-split: consent for a writable connected folder (pre-run gate)
|
|
145
150
|
}
|
|
@@ -177,6 +182,13 @@ REGEX_KEYS = {
|
|
|
177
182
|
VALID_ON_UNANSWERED = {"fail", "prompt", "first", "llm"}
|
|
178
183
|
VALID_TIERS = ("protocol", "container", "microvm", "hostloop", "cowork")
|
|
179
184
|
|
|
185
|
+
# Gate-id tripwire: the `host-path-assert-cowork` WARN below embeds Cowork's
|
|
186
|
+
# host-loop gate id in offline Python (the linter never reads a baseline). The
|
|
187
|
+
# id is pinned by test/scenario-lint-gate-id.test.ts against the PINNED_GATES
|
|
188
|
+
# entry in src/sync/cowork-sync.ts, so a Desktop gate re-key fails loud there
|
|
189
|
+
# instead of silently rotting this message.
|
|
190
|
+
HOST_LOOP_GATE_ID = "1143815894"
|
|
191
|
+
|
|
180
192
|
|
|
181
193
|
class Finding:
|
|
182
194
|
__slots__ = ("severity", "rule", "message", "fix", "file", "line")
|
|
@@ -300,6 +312,60 @@ def lint_doc(doc, path, raw_lines):
|
|
|
300
312
|
)
|
|
301
313
|
)
|
|
302
314
|
|
|
315
|
+
# E/W: transcript_no_host_path is tier-incompatible with hostloop/protocol — the agent
|
|
316
|
+
# legitimately runs on real host paths there, so the assertion fails BY DESIGN (the runtime only
|
|
317
|
+
# warns at run start, after authoring). Lint is deliberately STRICTER than the runtime: the docs
|
|
318
|
+
# declare the combination incompatible, so authoring it is a bug even if a tool-free run could
|
|
319
|
+
# accidentally pass. `cowork` gets a WARN naming the baseline-gate resolution dependency (the
|
|
320
|
+
# linter stays offline — the message carries the gate fact instead of reading a baseline).
|
|
321
|
+
if "transcript_no_host_path" in assert_keys:
|
|
322
|
+
if fidelity in ("hostloop", "protocol"):
|
|
323
|
+
findings.append(
|
|
324
|
+
Finding(
|
|
325
|
+
"ERROR",
|
|
326
|
+
"host-path-assert-tier",
|
|
327
|
+
f"`transcript_no_host_path` on `fidelity: {fidelity}` — the agent runs on real "
|
|
328
|
+
"host paths at this tier, so this assertion FAILS BY DESIGN (it can never be a "
|
|
329
|
+
"meaningful check here).",
|
|
330
|
+
"Run this assertion at fidelity: container (or microvm), or drop it for this tier.",
|
|
331
|
+
path,
|
|
332
|
+
)
|
|
333
|
+
)
|
|
334
|
+
elif fidelity == "cowork":
|
|
335
|
+
findings.append(
|
|
336
|
+
Finding(
|
|
337
|
+
"WARN",
|
|
338
|
+
"host-path-assert-cowork",
|
|
339
|
+
"`transcript_no_host_path` on `fidelity: cowork` — the tier resolves per the "
|
|
340
|
+
f"baseline's host-loop gate ({HOST_LOOP_GATE_ID}); if it resolves to hostloop "
|
|
341
|
+
"this assertion fails by design (and a later gate flip re-stales the cassette).",
|
|
342
|
+
"Pin fidelity: container if the assertion is load-bearing; keep cowork only if "
|
|
343
|
+
"you accept the gate-resolution dependency.",
|
|
344
|
+
path,
|
|
345
|
+
)
|
|
346
|
+
)
|
|
347
|
+
|
|
348
|
+
# E: requires_capabilities on protocol — the capability probe cannot run at protocol tier
|
|
349
|
+
# (clause b of the requires_capabilities contract), so the run HARD-FAILS unless an assert item
|
|
350
|
+
# opts out via allow_missing_capability: true. Offline-detectable fails-by-design, same class as
|
|
351
|
+
# the tier/assert rules above.
|
|
352
|
+
req_caps = doc.get("requires_capabilities")
|
|
353
|
+
if req_caps and fidelity == "protocol":
|
|
354
|
+
opted_out = any(item.get("allow_missing_capability") is True for item in items)
|
|
355
|
+
if not opted_out:
|
|
356
|
+
findings.append(
|
|
357
|
+
Finding(
|
|
358
|
+
"ERROR",
|
|
359
|
+
"capabilities-on-protocol",
|
|
360
|
+
"non-empty `requires_capabilities` on `fidelity: protocol` — the capability "
|
|
361
|
+
"probe cannot run at protocol tier, so the run hard-fails as unverifiable "
|
|
362
|
+
"(fails by design).",
|
|
363
|
+
"Use a sandboxed tier (container/microvm/hostloop), or add "
|
|
364
|
+
"`allow_missing_capability: true` to an assert item to opt out explicitly.",
|
|
365
|
+
path,
|
|
366
|
+
)
|
|
367
|
+
)
|
|
368
|
+
|
|
303
369
|
# E: retired/invalid on_unanswered
|
|
304
370
|
ou = doc.get("on_unanswered")
|
|
305
371
|
if ou is not None and ou not in VALID_ON_UNANSWERED:
|
|
@@ -543,7 +609,7 @@ def cmd_lint(args):
|
|
|
543
609
|
else:
|
|
544
610
|
expanded.append(arg)
|
|
545
611
|
args.files = expanded
|
|
546
|
-
# Linter self-check
|
|
612
|
+
# Linter self-check: a valid schema key the replay-class sets don't classify can't be linted
|
|
547
613
|
# correctly — surface it as a hard ERROR so it fails the gate (and --strict) until someone classifies it.
|
|
548
614
|
if UNCLASSIFIED_KEYS:
|
|
549
615
|
all_findings.append(
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
{
|
|
2
|
+
"patterns": [
|
|
3
|
+
{ "regex": "/Users/[^\\s\"'\\\\)\\]`]+?(?=/mnt/)", "label": "local-path", "flags": "g" },
|
|
4
|
+
{ "regex": "/home/[^\\s\"'\\\\)\\]`]+?(?=/mnt/)", "label": "local-path", "flags": "g" },
|
|
5
|
+
{ "regex": "/root/[^\\s\"'\\\\)\\]`]+?(?=/mnt/)", "label": "local-path", "flags": "g" },
|
|
6
|
+
{ "regex": "/Users/[^\\s\"'\\\\)\\]`]+", "label": "local-path", "flags": "g" },
|
|
7
|
+
{ "regex": "/home/[^\\s\"'\\\\)\\]`]+", "label": "local-path", "flags": "g" },
|
|
8
|
+
{ "regex": "/root/[^\\s\"'\\\\)\\]`]+", "label": "local-path", "flags": "g" },
|
|
9
|
+
{ "regex": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "label": "email", "flags": "g" }
|
|
10
|
+
]
|
|
11
|
+
}
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,134 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.24.0] — 2026-07-05
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- **`no_unexpected_files` assertion** — per-scenario glob allowlist over **newly created** files under the
|
|
14
|
+
user-visible roots (workRoot-relative paths; `**` matches any depth). Post-hoc stray-file / fabricated-
|
|
15
|
+
artifact detection for the founder-skills hand-off contract (consumer ask, 2026-07-04); pairs with their
|
|
16
|
+
script-side `_produced_by` / `UNVALIDATED_ARTIFACT` check for overwrite-in-place bypasses this key cannot
|
|
17
|
+
see. Records a pre-run path baseline (`preRunPaths` on the cassette + `pre-run-manifest.json` on kept runs;
|
|
18
|
+
optional field, no cassetteVersion bump). **Live/verify-run** without a baseline ⇒ evidence-unavailable
|
|
19
|
+
hard-fail; **replay** without `preRunPaths` ⇒ loud exclude (pre-0.24 cassettes). **Microvm** does not
|
|
20
|
+
capture (existing artifact-tree gap) — use container/hostloop. Live runs capture the baseline when the
|
|
21
|
+
scenario asserts the key; recordings always capture it, so a later assert-add replays without re-record.
|
|
22
|
+
|
|
23
|
+
- **Two new documented fidelity gaps: guest runtime identity and session-slug shape.**
|
|
24
|
+
Runtime forensics over a local install's Cowork VM disk images (coworkd's own logs) established
|
|
25
|
+
that production provisions a **dedicated Unix user per session** inside the VM (`useradd`,
|
|
26
|
+
name = session slug, uid=gid sequential ≥1014, `HOME=/sessions/<slug>`) and spawns the agent and
|
|
27
|
+
every bash call as that user via `oneshot-<uuid>` supervisor jobs — where the harness's
|
|
28
|
+
container/microvm tiers run static uid-1000 `ubuntu` with `HOME=/tmp`. Production's in-VM slugs
|
|
29
|
+
are Docker-style name triples (`beautiful-bold-planck`), not the harness's `local_<hrtime36>`;
|
|
30
|
+
the `local_<uuid>` shape exists in production only as Desktop's host-side session-record
|
|
31
|
+
filenames. Both are recorded in `docs/fidelity-gaps.md` with observable consequences (`whoami`,
|
|
32
|
+
`id -u`, `~`-relative writes, cwd shape); the `Dockerfile.agent` uid-1000 comment no longer
|
|
33
|
+
claims runtime parity. Initial lead credit: the founder-skills project's journald forensics,
|
|
34
|
+
independently re-verified here against this machine's own VM images before documenting.
|
|
35
|
+
|
|
36
|
+
- **The `verify-cassettes` JSON envelope is now a published, 1.0-covered contract surface.**
|
|
37
|
+
SPEC §12's covered list previously named the RunResult envelope but was silent on §11.1's
|
|
38
|
+
`verify-cassettes` envelope — even though the CI recipe and the packaged GitHub Action steer
|
|
39
|
+
consumers to parse exactly that output, and an uncovered machine-output shape can rot a consumer's
|
|
40
|
+
`jq` gate silently (a renamed key yields `null`, and the gate stops gating). The envelope is now:
|
|
41
|
+
listed in §12 (rename/removal = MAJOR; additive growth stays MINOR), published as
|
|
42
|
+
`schema/verify-cassettes.json` in the npm package, and pinned by
|
|
43
|
+
`test/verify-envelope-schema.test.ts`, which validates real CLI output against the schema in both
|
|
44
|
+
directions (a required key the CLI stops emitting fails, and — via a strictened copy — a key the
|
|
45
|
+
CLI starts emitting without a schema update also fails). Writing the schema immediately caught one
|
|
46
|
+
doc drift: the per-file `version[]` channel (newer-harness cassette → hard fail) was emitted but
|
|
47
|
+
missing from §11.1's envelope block; the SPEC block, `ok` formula, and exit-code line now include it.
|
|
48
|
+
|
|
49
|
+
- **Skill-effectiveness workstream.** Three deliverables, all targeting the same failure mode
|
|
50
|
+
observed in real skill adoption: the facts existed across the docs, but nothing composed them
|
|
51
|
+
into the task a scenario author actually faces:
|
|
52
|
+
- **`references/task-recipes.md`** — four end-to-end recipes that consumers previously had to
|
|
53
|
+
reverse-engineer from scattered reference docs: evolve a cassette's `assert:` block (usually token-free via
|
|
54
|
+
`replay --assert-from`; the recording-shaping-drift caveat spelled out), audit a fleet for tier
|
|
55
|
+
drift (`effectiveFidelity` + the new `resolved-tier`/`unverifiable-tier` classes + a cassette
|
|
56
|
+
anatomy table), set up redaction before the first hostloop/protocol record (`init-redact`,
|
|
57
|
+
search set, scanner relationship, `--allow-*` scoping idioms), and derive budget assertions
|
|
58
|
+
from an existing run instead of a two-pass record.
|
|
59
|
+
- **Four new eval cases** (`evals/evals.json` 5 → 9, ids 6–9), each a question a real consumer
|
|
60
|
+
hit, passing iff the agent reaches the cheap path: assert evolution without a
|
|
61
|
+
fleet re-record, reading a cassette's recorded tier (with a fixture cassette via `files[]`),
|
|
62
|
+
the exact `--allow-file` format (and its all-class scope trade-off), and batch `record`
|
|
63
|
+
atomicity semantics.
|
|
64
|
+
- **Doc-drift tripwire** (`test/skill-docs-sync.test.ts`), scoped to the kinds of drift that
|
|
65
|
+
actually happened (NOT a naive new-CLI-flag gate, which would have caught neither motivating
|
|
66
|
+
example): extends the `cassette-docs-sync` pattern to pin the skill's assertion catalog against
|
|
67
|
+
`schema/scenario.schema.json` and the skill docs against the current cassette schema's
|
|
68
|
+
top-level field list. Plus a CONTRIBUTING checklist line for consumer-visible workflow changes.
|
|
69
|
+
|
|
70
|
+
### Fixed
|
|
71
|
+
|
|
72
|
+
- **Gate `1648655587` was mislabeled as a Task-dispatch cap; it is the scheduled-task session
|
|
73
|
+
limiter.** A binary-verified forensic pass (asar 1.18286.0, `class L9t "[ScheduledTasks]"`) plus one
|
|
74
|
+
adversarial-review round established that gate `1648655587` (`{perTask:1, global:3}`) governs
|
|
75
|
+
Cowork's **scheduled/recurring (cron) task** session scheduler — ≤1 concurrent session per
|
|
76
|
+
scheduled task, ≤3 concurrent scheduled-task sessions globally — **not** the in-conversation
|
|
77
|
+
`Task` tool. The Desktop imposes **no** cap on `Task`-tool sub-agent fan-out at all (the `Task`
|
|
78
|
+
PreToolUse hook only blocks `run_in_background`). SPEC §10, `docs/fidelity-gaps.md`,
|
|
79
|
+
`docs/scenario.md`, the skill (SKILL.md gotcha 6 + `references/scenario-schema.md`), the
|
|
80
|
+
1.18286.0 baseline gate note, and `dispatch_count_max`'s failure message are all corrected;
|
|
81
|
+
`dispatch_count_max` is now described as an author-chosen budget assertion, not enforcement of a
|
|
82
|
+
(non-existent) production Task cap. The `PINNED_GATES` rename
|
|
83
|
+
(`taskDispatchLimiter → scheduledTaskSessionLimiter`) also landed, regenerated through `sync`
|
|
84
|
+
against the same Desktop 1.18286.0 install (never hand-edited): the 1.18286.0 baseline now
|
|
85
|
+
carries the corrected gate key, older baselines keep the historical label, and the sync
|
|
86
|
+
gate-merge now matches previous entries by gate **id** (not exact `name:id` key) so a rename
|
|
87
|
+
carries the note/state across and can't resurrect the old key as a stale duplicate. This
|
|
88
|
+
supersedes the earlier "faithful Task-dispatch runtime limiter is planned" language — there is
|
|
89
|
+
no production behavior to mirror.
|
|
90
|
+
|
|
91
|
+
- **`schema/cassette.v7.json` was missing `userVisibleRoots` and `scenarioSource`** — both fields
|
|
92
|
+
are written by `record` (since format v4 / 0.8.0 and 0.20.0 respectively); `userVisibleRoots` was
|
|
93
|
+
even named in the schema's prose `description` while absent from `properties`. Found by the new
|
|
94
|
+
skill-docs tripwire while pinning the field list.
|
|
95
|
+
|
|
96
|
+
- **Redaction preflight + `init-redact`.**
|
|
97
|
+
`record` now warns (`::warning::`) **before the agent spawns** when a scenario about to record at a
|
|
98
|
+
host-path-bearing tier (`hostloop`, or `protocol`) has an EMPTY assembled redaction policy — that
|
|
99
|
+
combination commits real host paths, which `verify-cassettes`' `path` scanner then hard-fails; the
|
|
100
|
+
empty-policy discovery used to happen only at the post-run policy load, after the run was paid for.
|
|
101
|
+
Batch `record <dir>` and `--rerecord-stale` preflight ONCE for the whole batch, before the first
|
|
102
|
+
spawn (never N interleaved duplicates). A malformed `.cowork-redact.json` now also fails pre-spawn.
|
|
103
|
+
The reference `.cowork-redact.json` (generic local-path prefixes + email regex) now ships in the npm
|
|
104
|
+
package, and the new `cowork-harness init-redact [--force]` copies it into the cwd — the copy is
|
|
105
|
+
load-bearing, since the policy search set (now documented in `docs/cassette.md` and the skill's
|
|
106
|
+
ci-recipe) is cwd → scenario dir → cassette dir (+ `COWORK_HARNESS_REDACT_*` env), never the package
|
|
107
|
+
dir. The always-on privacy scanner remains the universal net — container-tier recordings can trip it
|
|
108
|
+
too.
|
|
109
|
+
|
|
110
|
+
- **Lint catches fidelity/assert combinations that fail by design.**
|
|
111
|
+
`scenario.py lint` (and the `cowork-harness lint` wrapper) now ERRORs on
|
|
112
|
+
`transcript_no_host_path` + `fidelity: hostloop` or `protocol` (the agent legitimately runs on real
|
|
113
|
+
host paths at those tiers — the assertion can never be a meaningful check there; lint is
|
|
114
|
+
deliberately stricter than the runtime's run-start warning), WARNs on `transcript_no_host_path` +
|
|
115
|
+
`fidelity: cowork` naming the host-loop gate-resolution dependency (the linter stays offline — the
|
|
116
|
+
message carries gate id `1143815894` instead of reading a baseline, pinned against the
|
|
117
|
+
`cowork-sync.ts` gate table by a sync test so a Desktop re-key can't rot it), and ERRORs on
|
|
118
|
+
non-empty `requires_capabilities` + `fidelity: protocol` without `allow_missing_capability` (the
|
|
119
|
+
capability probe can't run at protocol tier, so the run hard-fails as unverifiable).
|
|
120
|
+
|
|
121
|
+
- **Resolved-tier staleness detection.** A
|
|
122
|
+
`fidelity: cowork` cassette records the tier the loop-decision gate resolved to
|
|
123
|
+
(`effectiveFidelity`); `verify-cassettes` / `replay` now detect when the current baseline resolves
|
|
124
|
+
that scenario differently (gate `1143815894` flipped since record) and emit a new `resolved-tier`
|
|
125
|
+
staleness finding — the recording exercises the wrong tier. Resolution is baseline-only (the
|
|
126
|
+
scenario's pinned `baseline:` when present, else `latest`; the `CLAUDE_FORCE_HOST_LOOP` env override
|
|
127
|
+
is suppressed) so verify results can't differ across machines. A `cowork` cassette whose tier can't
|
|
128
|
+
be verified (predates `effectiveFidelity`, or its pinned baseline fails to load) gets a loud
|
|
129
|
+
`unverifiable-tier` finding — never a silent skip, never an aborted sweep. Both classes hard-fail
|
|
130
|
+
`verify-cassettes` (class-blind gate) and warn-by-default on `replay` (`--strict` escalates;
|
|
131
|
+
`--fail-on-skill-drift` ignores them — they are not skill-source drift). A pre-`effectiveFidelity`
|
|
132
|
+
cassette with an *explicit* tier is statically knowable: it passes with a non-failing informational
|
|
133
|
+
note in the new per-file `notes[]` of the `verify-cassettes` envelope (a `·` row in text output).
|
|
134
|
+
`schema/run-result.json` now also declares `staleness` (full class enum) and `skippedAssertions`,
|
|
135
|
+
closing a pre-existing gap vs SPEC §11.
|
|
136
|
+
|
|
9
137
|
## [0.23.0] — 2026-07-04
|
|
10
138
|
|
|
11
139
|
### Added
|
package/CONTRIBUTING.md
CHANGED
|
@@ -66,6 +66,7 @@ Paths inside a scenario/session resolve relative to that file (see [docs/session
|
|
|
66
66
|
- **Don't weaken the boundary.** Changes to `runtime/container.ts` or `docker/compose.yml` must keep the default-deny network + sealed FS. Run `cowork-harness boundary-check` and add/adjust a probe in `src/boundary.ts` if you change the model.
|
|
67
67
|
- **Mark unverified code.** Anything not yet run end-to-end against a live agent gets a `// UNVERIFIED` comment so reviewers know.
|
|
68
68
|
- **Add a test.** New schema fields, `Decider` rules, or egress logic need a unit test in `test/`. Examples must validate (`test/examples.test.ts`).
|
|
69
|
+
- **Consumer-visible workflow changes update the skill.** A change a scenario author would act on — a new assertion key, cassette field, CLI command, or a changed record/replay/verify workflow — must land with a matching update to `.claude/skills/cowork-harness/` (SKILL.md or `references/`). The machine-checkable slices are enforced (`test/skill-docs-sync.test.ts` pins the skill against the assertion-key catalog and the cassette schema's field list; `test/cli-help.test.ts` pins the README command table); prose workflows are on you — this checklist line exists because `effectiveFidelity` shipped consumer-visible and stayed undocumented in the skill until an external consumer flagged it.
|
|
69
70
|
- **Format.** `npm run format:check` must pass (`npx prettier --write "src/**/*.ts" "test/**/*.ts"` to fix).
|
|
70
71
|
|
|
71
72
|
## Extending the sync extractor
|