cowork-harness 4.1.0 → 4.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +4 -4
- package/.claude/skills/cowork-harness/references/assertion-catalog.md +1 -1
- package/.claude/skills/cowork-harness/references/assertions-guide.md +1 -1
- package/.claude/skills/cowork-harness/references/authoring.md +64 -14
- package/.claude/skills/cowork-harness/references/ci-recipe.md +23 -11
- package/.claude/skills/cowork-harness/references/critique.md +1 -1
- package/.claude/skills/cowork-harness/references/debugging.md +9 -3
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/gotchas.md +5 -4
- package/.claude/skills/cowork-harness/references/measurement.md +5 -2
- package/.claude/skills/cowork-harness/references/run-record-replay.md +13 -6
- package/.claude/skills/cowork-harness/references/scenario-schema.md +15 -3
- package/.claude/skills/cowork-harness/references/task-recipes.md +9 -4
- package/CHANGELOG.md +71 -0
- package/README.md +3 -3
- package/SPEC.md +13 -9
- package/dist/baseline.js +32 -1
- package/dist/cli.js +13 -8
- package/dist/errors.js +41 -0
- package/dist/run/cassette.js +62 -28
- package/dist/run/execute.js +86 -19
- package/dist/run/lint-load.js +5 -2
- package/dist/scan.js +20 -0
- package/dist/session.js +5 -0
- package/docs/ci.md +1 -1
- package/docs/cli.md +9 -9
- package/docs/companion-skill.md +2 -2
- package/docs/scenario.md +9 -6
- package/examples/replays/README.md +1 -1
- package/package.json +1 -1
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, asserting artifacts, egress, or sub-agent dispatch, measuring how long each tool call took (toolDurations / trace), or debugging a failed run or verdict from its result.json or transcript. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 4.1.
|
|
7
|
-
tracks-harness: cowork-harness 4.1.
|
|
6
|
+
version: 4.1.1
|
|
7
|
+
tracks-harness: cowork-harness 4.1.1 (baseline desktop-2.9939.4)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -26,7 +26,7 @@ allowlist). This skill exists mostly to keep you out of those traps — the *Inv
|
|
|
26
26
|
full landmine catalog in [`references/gotchas.md`](references/gotchas.md) are the highest-value part.
|
|
27
27
|
Read them.
|
|
28
28
|
|
|
29
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.1.
|
|
29
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.1.1` (baseline
|
|
30
30
|
> `desktop-2.9939.4`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
31
31
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
32
32
|
|
|
@@ -43,7 +43,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
43
43
|
|
|
44
44
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
45
45
|
- **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
|
|
46
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.1.
|
|
46
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.1.1**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@^4.1.1" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@^4.1.1"`. **Pin `@^4.1.1`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
|
|
47
47
|
|
|
48
48
|
This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
|
|
49
49
|
OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.1.
|
|
3
|
+
Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`). Every `assert:` key with its semantics, and the
|
|
4
4
|
verdict-signal table. Which keys survive `replay` is in [`scenario-schema.md`](./scenario-schema.md#replay-class);
|
|
5
5
|
the scenario and session YAML fields are there too.
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertions guide
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.1.
|
|
3
|
+
Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`). Read it when choosing assertion keys: the two orthogonal axes and the goal → key map. The full catalog is `assertion-catalog.md`.
|
|
4
4
|
|
|
5
5
|
### Assertions: two orthogonal axes
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Authoring a scenario
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.1.
|
|
3
|
+
Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`). Read it when composing a `scenarios/*.yaml`: session vs scenario, discovery, the fidelity tier, the answer path, `web_fetch`, and scaffold + lint.
|
|
4
4
|
|
|
5
5
|
## Part I — AUTHOR a scenario
|
|
6
6
|
|
|
@@ -220,25 +220,75 @@ so a file `run`/`record` would refuse fails lint too (running `scenario.py lint`
|
|
|
220
220
|
cowork-harness lint scenarios/*.yaml
|
|
221
221
|
```
|
|
222
222
|
|
|
223
|
-
`lint`
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
`
|
|
228
|
-
|
|
229
|
-
`
|
|
230
|
-
|
|
231
|
-
`
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
223
|
+
`lint` exits non-zero on any ERROR (CI-friendly); `--strict` also fails on WARN. Every rule it reports:
|
|
224
|
+
|
|
225
|
+
| Rule | Severity | Fires on |
|
|
226
|
+
|---|---|---|
|
|
227
|
+
| `assert-contradiction` | ERROR | assert items no single run can satisfy together |
|
|
228
|
+
| `assertions-key` | ERROR | `assertions:` instead of `assert:` — none of the checks would run |
|
|
229
|
+
| `authored-replay-fidelity` | ERROR | an authored `replay_protocol_fidelity` (only the replay lane synthesizes it) |
|
|
230
|
+
| `capabilities-on-protocol` | ERROR | non-empty `requires_capabilities` on `protocol` without `allow_missing_capability` — the probe cannot run there, so the run fails as unverifiable |
|
|
231
|
+
| `cassette-evidence-skipped` | INFO | with `--cassette-dir`: a cassette (or the directory) could not be read, so it cannot quiet replay-evidence advice |
|
|
232
|
+
| `container-only-key-off-container` | ERROR | `no_scratchpad_leak` off `container` — hostloop's `present_files` never promotes, so there is nothing to leak (WARN on `cowork`, whose tier resolves per the baseline gate) |
|
|
233
|
+
| `egress-on-protocol` | ERROR | an egress assertion (`egress_*` / `expect_denied`) on `protocol`, which enforces no egress |
|
|
234
|
+
| `enum-value-invalid` | ERROR | a field value outside its allowed set |
|
|
235
|
+
| `fidelity-missing` | ERROR | no `fidelity:` (required since 4.0.0) |
|
|
236
|
+
| `file-absent-contradiction` | ERROR | one path under both `file_exists` and `file_absent` |
|
|
237
|
+
| `gate-needs-controlout` | INFO | gate assertions, which evaluate on replay only when the cassette has `controlOut` |
|
|
238
|
+
| `host-path-assert-cowork` | WARN | `transcript_no_host_path` on `cowork` — it fails by design if the tier resolves to hostloop |
|
|
239
|
+
| `host-path-assert-tier` | ERROR | `transcript_no_host_path` on `hostloop` / `protocol`, where it fails by design |
|
|
240
|
+
| `lane-remote-incompatible-key` | ERROR | `present_files_called` / `no_scratchpad_leak` / `user_visible_artifact` on `lane: remote` (the runtime rejects them at load, so the tier rules are suppressed there) |
|
|
241
|
+
| `linter-extra-findings-invalid` | ERROR | the loader findings `cowork-harness lint` hands the linter could not be read |
|
|
242
|
+
| `linter-unclassified-key` | ERROR | a valid assertion key this linter cannot classify (the linter is out of date) |
|
|
243
|
+
| `manifest-needs-snapshot` | INFO | manifest-backed keys, which evaluate on replay only when the cassette carries an `artifacts` manifest |
|
|
244
|
+
| `mixed-assert-item` | WARN | one assert item mixing replay-checkable and live-only keys (replay drops the live-only half) |
|
|
245
|
+
| `no-scenarios` | ERROR | a linted directory with no `*.yaml` / `*.yml` |
|
|
246
|
+
| `not-found` | ERROR | a named file that does not exist |
|
|
247
|
+
| `parse` | ERROR | a file that is not YAML, or not a mapping |
|
|
248
|
+
| `positional-choose-order` | INFO | an answer rule with a positional `choose` (first / index), which option re-ordering can move |
|
|
249
|
+
| `present-files-key-off-tier` | ERROR | `present_files_called` on `protocol` / `microvm` (served only at `container` / `hostloop`) |
|
|
250
|
+
| `prompt-slash-not-leading` | WARN | a `prompt:` that names `/<skill>` without starting with it, so it is never expanded |
|
|
251
|
+
| `reference-access-contradiction` | ERROR | one reference under both `reference_read` and `no_observed_reference_access` |
|
|
252
|
+
| `regex-double-quoted` | WARN | a double-quoted regex with an unescaped backslash (YAML strips it) |
|
|
253
|
+
| `replay-noop` | WARN | every assertion is live-only or a verdict modifier, so a replay gate verifies nothing |
|
|
254
|
+
| `tool-called-always-passes` | INFO | `tool_called` with `count: {min: 0}` and no `max` — it asserts nothing |
|
|
255
|
+
| `tool-input-regex-redactable` | WARN | a `tool_not_called` input literal the redaction policy rewrites in the committed cassette (or a policy pattern it cannot check offline) |
|
|
256
|
+
| `tool-input-shell-tier` | INFO | the object form with `tool: Bash` and a `command` on `hostloop` / `cowork`, where shell runs as `mcp__workspace__bash` — list both |
|
|
257
|
+
| `tool-not-called-tier-vacuous` | WARN | `tool_not_called` / `subagent_tool_absent` naming a tool the tier never serves |
|
|
258
|
+
| `transcript-command-shaped` | WARN | a `transcript_*` value shaped like a shell command — those keys read prose only, never a tool call |
|
|
259
|
+
| `unknown-assert-key` | WARN | an assertion key not in the catalog (the loader rejects it) |
|
|
260
|
+
| `unknown-top-key` | WARN | a scenario key not in the schema |
|
|
261
|
+
| `vacuous-gate-assert` | WARN | `gate_answers_delivered` with no presence companion (zero gates passes it), or inert beside `questions_count_max: 0` |
|
|
262
|
+
| `scenario-invalid` | ERROR | the harness's scenario loader refuses the file (via `cowork-harness lint` only — see below) |
|
|
263
|
+
| `baseline-unknown` | ERROR | `baseline:` names no baseline this CLI ships (via `cowork-harness lint` only) |
|
|
264
|
+
| `lint-loader-internal` | ERROR | the wrapper could not run its loader check on a file — a harness bug; it never falls back to a lint that skipped the loader |
|
|
265
|
+
|
|
266
|
+
`scaffold` auto-upgrades the tier if you ask for egress on `protocol`, so it never emits a scenario `lint`
|
|
267
|
+
would reject.
|
|
268
|
+
|
|
269
|
+
**Lint the skill itself: `cowork-harness lint-skill <skill-dir>`.** It checks the skill, not a scenario:
|
|
270
|
+
Cowork host-loop footguns (`${CLAUDE_PLUGIN_ROOT}` in a VM bash step, hook events, a misplaced
|
|
271
|
+
`hooks.json`, an unresolvable `subagent_type`), the evidence corpus a `critique` can package
|
|
272
|
+
(`references/critique.md`), and two size caps. `skill-body-over-reattach-cap` (WARN) fires when the
|
|
273
|
+
`SKILL.md` body, frontmatter excluded, passes 19,000 B — after a compaction the agent re-attaches only the
|
|
274
|
+
start of an invoked skill — and `skill-body-near-reattach-cap` (INFO) from 80% of that;
|
|
275
|
+
`skill-reference-over-read-cap` (WARN) fires on a `references/**.md` over 60,000 B, past which a
|
|
276
|
+
whole-file Read returns a partial view. `--strict` fails on WARN, never on INFO. To accept a reviewed
|
|
277
|
+
judgement-call finding, pass `--ignore-rule <rule>[=<glob>]` (repeatable; the glob matches the finding's
|
|
278
|
+
file) or fence the text in `SKILL.md` with `<!-- lint-skill: ignore-start <rule>[,<rule>…]: <reason> -->`
|
|
279
|
+
… `<!-- lint-skill: ignore-end -->` (outside any code fence). A suppressed finding is still printed; it
|
|
280
|
+
stops gating. A provable rule (an ERROR, a misplaced `hooks.json`, a missing pinned agent) cannot be
|
|
281
|
+
suppressed: naming it, or an unknown rule, in `--ignore-rule` is a usage error (exit 2); in a marker it is
|
|
282
|
+
WARN `lint-skill-ignore-invalid`, as is any other malformed marker. An unclosed marker is WARN
|
|
283
|
+
`lint-skill-ignore-unclosed`, and one that suppresses nothing is INFO `lint-skill-ignore-unused`.
|
|
235
284
|
|
|
236
285
|
**`cowork-harness lint` runs the loader: a file it calls clean is one `run`/`record` will load.** Anything
|
|
237
286
|
the loader refuses — an unknown key, a wrong value type (a scalar `semantic_matches.rubric`), a bad regex,
|
|
238
287
|
a reserved value — is ✗ ERROR `scenario-invalid` (exit 1, with or without `--strict`), and a `baseline:`
|
|
239
288
|
naming no baseline this installed CLI ships is ✗ ERROR `baseline-unknown` (`latest` always resolves). It
|
|
240
289
|
does not check what depends on the machine the run happens on (the session file and its mounts, an
|
|
241
|
-
absolute `baseline:` path
|
|
290
|
+
absolute `baseline:` path that does not exist here — one that exists is checked (4.1.1 and later) —
|
|
291
|
+
environment variables). A session or matrix YAML in a linted directory is not
|
|
242
292
|
a scenario and is reported as one that does not load — keep those out of the linted set. `python3
|
|
243
293
|
scenario.py lint` run directly stays offline and lenient: there an unknown key is only a ⚠ WARN (exit 0).
|
|
244
294
|
`cowork-harness record <file.yaml> --dry-run` also runs the loader and adds the pre-spend refusals (exit 2
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 4.1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -17,7 +17,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
|
|
|
17
17
|
CLI major reaches your workflow the moment it is promoted even though your `uses:` ref never changed — so a
|
|
18
18
|
copy-pasted recipe that omits the input takes a major bump with no say in it. `^4` holds the major, needs no
|
|
19
19
|
patch number to remember, and only wants a human decision at the next major. Pin an exact version
|
|
20
|
-
(e.g. `version: "4.1.
|
|
20
|
+
(e.g. `version: "4.1.1"`) instead when you want byte-reproducible CI.
|
|
21
21
|
|
|
22
22
|
Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
|
|
23
23
|
cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
|
|
@@ -82,7 +82,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
82
82
|
GitHub-hosted runners, no token/Docker/agent:
|
|
83
83
|
|
|
84
84
|
```yaml
|
|
85
|
-
- run: npm i -g "cowork-harness@^4.1.
|
|
85
|
+
- run: npm i -g "cowork-harness@^4.1.1"
|
|
86
86
|
- run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
|
|
87
87
|
# no silent false-greens. WITHOUT --strict this
|
|
88
88
|
# step cannot fail on a WARN-class rule (e.g.
|
|
@@ -241,7 +241,9 @@ cowork-harness replay cassettes/ # replay every *.cassette.jso
|
|
|
241
241
|
Re-record whenever the protocol or your scenario's expected content changes. An old cassette without
|
|
242
242
|
`controlOut` excludes the gate keys (with a loud warning) — re-record to enable them. `record` **refuses
|
|
243
243
|
to freeze a failing live run** into a cassette (pass `--allow-failing` to override) — a committed red
|
|
244
|
-
cassette is a latent false-signal.
|
|
244
|
+
cassette is a latent false-signal. An `--allow-failing` recording of a red run exits 0 with `ok: true`:
|
|
245
|
+
`record`'s `ok` means a cassette was written, and the run's own verdict is `results[0].verdict.pass`
|
|
246
|
+
(`items[].verdict` on `record <dir/>`).
|
|
245
247
|
|
|
246
248
|
## Privacy: cassettes are committed fixtures → record only against SYNTHETIC inputs
|
|
247
249
|
|
|
@@ -256,7 +258,9 @@ dollar figures). In a skill repo these cassettes get **committed**. So:
|
|
|
256
258
|
dir** (first file found per dir; env vars merge on top). `cowork-harness init-redact` copies the
|
|
257
259
|
packaged reference template (local-path prefixes, incl. macOS temp roots and slugged home segments
|
|
258
260
|
like `-Users-<user>-…`, + a generic email regex) into the cwd as a starting point — review and tailor
|
|
259
|
-
it; a copy from an earlier release lacks the temp-root rules, so re-copy with `--force`
|
|
261
|
+
it; a copy from an earlier release lacks the temp-root rules, so re-copy with `--force` (without it
|
|
262
|
+
`init-redact` refuses to overwrite an existing policy; with it your tailoring is replaced — save and
|
|
263
|
+
re-apply it) or add them by hand. Redaction is **verdict-preserving** — `record` refuses to write if
|
|
260
264
|
redaction would flip an assertion (a manufactured green). `--no-redact` skips it for known-synthetic
|
|
261
265
|
inputs.
|
|
262
266
|
- **Pre-spawn preflight**: `record` warns (`::warning::`, before the paid run starts — once per batch
|
|
@@ -342,8 +346,8 @@ A typical skill repo runs four stages, fastest/cheapest first:
|
|
|
342
346
|
This is the shape a CI step wants: **silent on success (no output, exit 0), loud and specific on
|
|
343
347
|
failure** — `--quiet` suppresses the readiness preview but never the `✗ broken:` lines, which name the
|
|
344
348
|
offending file *and* the rejected key, one line per file, and the step still exits 1. It exits 0,
|
|
345
|
-
though, on an input the real record would refuse (a
|
|
346
|
-
tier-vacuous `tool_not_called`): that prints a `⚠ input error:` line and lands in `inputErrors[]`. To
|
|
349
|
+
though, on an input the real record would refuse (a `session:` file that cannot be read — 4.1.1 and
|
|
350
|
+
later — a missing path, an unknown baseline name, a tier-vacuous `tool_not_called`): that prints a `⚠ input error:` line and lands in `inputErrors[]`. To
|
|
347
351
|
gate on those too, use the JSON form (4.1.0 and later):
|
|
348
352
|
|
|
349
353
|
```bash
|
|
@@ -391,7 +395,7 @@ jobs:
|
|
|
391
395
|
with: { node-version: '24' }
|
|
392
396
|
- uses: actions/setup-python@v5
|
|
393
397
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
394
|
-
- run: npm i -g "cowork-harness@^4.1.
|
|
398
|
+
- run: npm i -g "cowork-harness@^4.1.1"
|
|
395
399
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
396
400
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
397
401
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -420,7 +424,7 @@ jobs:
|
|
|
420
424
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
421
425
|
fi
|
|
422
426
|
- if: steps.guard.outputs.live == 'true'
|
|
423
|
-
run: npm i -g "cowork-harness@^4.1.
|
|
427
|
+
run: npm i -g "cowork-harness@^4.1.1"
|
|
424
428
|
- if: steps.guard.outputs.live == 'true'
|
|
425
429
|
run: cowork-harness run scenarios/ --output-format json
|
|
426
430
|
env:
|
|
@@ -443,7 +447,8 @@ sandbox).
|
|
|
443
447
|
|
|
444
448
|
`--output-format json` emits a machine envelope on stdout (human output goes to stderr):
|
|
445
449
|
`{tool, version, command, ok, results[], error}` — one `RunResult` per scenario. **Overall pass for a
|
|
446
|
-
scenario is `verdict.pass`** (envelope-wide: `ok`
|
|
450
|
+
scenario is `verdict.pass`** (envelope-wide: `ok`; on `record`, `ok` only means the command exited 0 —
|
|
451
|
+
see below), and it is strictly stronger than
|
|
447
452
|
`result === "success" && assertions.every(pass)`: the verdict also carries ~20 signal codes that fail a run
|
|
448
453
|
with no failing assertion at all — `stalled`, `outputs_delete`, `mount_delete`, `host_path_leak`,
|
|
449
454
|
`undelivered_deliverables`, `missing_capability`, `permissive_auto_allow`, `ended_with_question`,
|
|
@@ -477,6 +482,11 @@ Keep the `.results[]?` hop and the `?` operators. `.results[0]` silently ignores
|
|
|
477
482
|
the first when you pass a directory, and a bare `.verdict` does not exist at the envelope root at all —
|
|
478
483
|
both read as "no failures" against a run that failed.
|
|
479
484
|
|
|
485
|
+
**`record` is shaped differently.** Its `ok` is the exit code's verdict (a cassette was written), not the
|
|
486
|
+
run's: `record --allow-failing` on a red run is `ok: true`. The verdict is in `results[0].verdict` on
|
|
487
|
+
`record <file>` and in `items[].verdict` on `record <dir/>` and `--rerecord-stale`, which carry no
|
|
488
|
+
`results[]` — so `.results[]?` reads nothing there. Use `[.items[]? | .verdict.failures[]?]` on a batch.
|
|
489
|
+
|
|
480
490
|
> **Do not filter on whether `assertion` is present.** That was the only discriminator before `kind`
|
|
481
491
|
> existed and it never worked in both directions: `coverage` entries carry a key too (an internal
|
|
482
492
|
> `answer_coverage` marker), so they read as authored asserts, while `guard`, `staleness` and
|
|
@@ -486,7 +496,9 @@ both read as "no failures" against a run that failed.
|
|
|
486
496
|
`--output-format json`, `run` / `record` / `replay` / `verify-cassettes` / `status` write their whole
|
|
487
497
|
human rendering — warnings, verdict, `status`'s summary line — to **stderr**, and stdout stays empty.
|
|
488
498
|
A wrapper that captures only stdout gets an empty log and, if it greps that for a state, a silent false
|
|
489
|
-
negative. Capture stderr for the human trail (`2> run.stderr.log`), or ask for JSON and parse stdout
|
|
499
|
+
negative. Capture stderr for the human trail (`2> run.stderr.log`), or ask for JSON and parse stdout —
|
|
500
|
+
`COWORK_HARNESS_OUTPUT_FORMAT=json` makes JSON the default for every command that takes `--output-format`
|
|
501
|
+
(an explicit flag still wins).
|
|
490
502
|
(Commands whose whole job is to print a value — `--version`, `assertions --list`, `scaffold`, `gates`,
|
|
491
503
|
`skill --dry-run` — write it to stdout by design. Under `--output-format json` the value rides inside the
|
|
492
504
|
envelope: `scaffold`'s YAML is `.scenario`, `skill --dry-run`'s preview is the envelope's own fields;
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Critique — the facts a plugin install can't otherwise reach
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.1.
|
|
3
|
+
Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`). This is **not** a trim of the full
|
|
4
4
|
[`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
|
|
5
5
|
flags, cost, reproduction discipline, known limitations all live there). This file covers exactly what a
|
|
6
6
|
plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Debugging a run
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.1.
|
|
3
|
+
Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`). Read it when a run misbehaved or a green looks wrong: triage, the observability output, and `chat`.
|
|
4
4
|
|
|
5
5
|
## Part III — Debug
|
|
6
6
|
|
|
@@ -37,7 +37,11 @@ already does.
|
|
|
37
37
|
**microvm: "control-protocol write failed" with `env: 'claude': No such file or directory` in the agent
|
|
38
38
|
stderr** usually means the VM never finished provisioning (the agent never reached PATH). Check
|
|
39
39
|
`cowork-harness vm status` — a `provisioning` other than `ready` confirms it — and if a run does not
|
|
40
|
-
recover it on its own, `cowork-harness vm delete` and retry.
|
|
40
|
+
recover it on its own, `cowork-harness vm delete` and retry. A run on such a VM can also have cached an
|
|
41
|
+
empty toolchain for it in the capability probe's cache: `vm delete` drops that VM's entry (`vm prune`
|
|
42
|
+
forgets only the orphaned VMs it deletes, never the current one); otherwise delete `capability-cache.json`
|
|
43
|
+
from the runs root (`~/.cowork-harness/runs/` unless
|
|
44
|
+
`COWORK_HARNESS_RUNS_DIR` is set) so the next run probes again.
|
|
41
45
|
|
|
42
46
|
**Is it your skill's bug, or a known harness gap?** Before deep-debugging a wrong behavior, rule out a
|
|
43
47
|
**deliberate fidelity gap** — the harness intentionally does *not* reproduce a few real-Cowork behaviors,
|
|
@@ -80,7 +84,9 @@ decide which assertions from *Assertions: two orthogonal axes* in `assertions-gu
|
|
|
80
84
|
walk observe this run — what distinguishes "nothing was left undelivered" from "cannot tell"), `cost` (`cost.usd` = the SDK's
|
|
81
85
|
`total_cost_usd` for the run — the authoritative single-run spend of the agent session, which leaves out the `semantic_matches` judge and the LLM decider calls; NOT the same source as summing
|
|
82
86
|
`modelUsage[].costUSD`, which is what `trace --view usage` reports, so the two can differ),
|
|
83
|
-
`usage` (`input_tokens`/`output_tokens`/`turns`), `toolDurations` (with `toolDurationsBasis`), `models`,
|
|
87
|
+
`usage` (`input_tokens`/`output_tokens`/`turns`), `toolDurations` (with `toolDurationsBasis`), `models`,
|
|
88
|
+
`toolCalls` (every tool call in stream order: `name`, top-level `input` fields each capped at 10 KB, and
|
|
89
|
+
`origin` `main`/`subagent`/`unknown` — what the object form of `tool_called`/`tool_not_called` reads), `toolErrors`,
|
|
84
90
|
`redundantToolCalls`, `modelUsage`, `thinking`, `skillActivity`, `subagents[]` (prompt/`dispatchModel`/
|
|
85
91
|
`resolvedModel`/output/`attributedSkillId`, `outputTruncated`, `referencesRead`, `reasoning`/`reasoningElided`),
|
|
86
92
|
`context` (tools/mcpServers/availableSkills), `tasks`,
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 4.1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`).
|
|
4
4
|
|
|
5
5
|
> **This page vs. the repo docs.** This is the **offline snapshot** that ships inside the installed
|
|
6
6
|
> plugin — it is self-contained on purpose. The repo carries four other fidelity views, each answering a
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Gotchas
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.1.
|
|
3
|
+
Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`). The full "✓ passed ≠ correct" landmine catalog.
|
|
4
4
|
|
|
5
5
|
## Gotchas — the "✓ passed ≠ correct" landmines
|
|
6
6
|
|
|
@@ -175,9 +175,10 @@ authorable). Reach for this list when debugging a run's behavior, that one while
|
|
|
175
175
|
`--reassert`). That opt-in path is safe by construction for the authored fields — it **hard-fails** if
|
|
176
176
|
`prompt`/`answers`/`baseline`/`fidelity`/`lane`/`skills`/`requires_capabilities` or the skill content (when a
|
|
177
177
|
fingerprint exists) drifted from the recording (re-record then), and `expect_denied`/filesystem/egress keys
|
|
178
|
-
are sourced but stay **live-only** (it warns; they don't move the replay verdict). **Caveat:** the
|
|
179
|
-
|
|
180
|
-
|
|
178
|
+
are sourced but stay **live-only** (it warns; they don't move the replay verdict). **Caveat:** the session's
|
|
179
|
+
`model:` IS in the cassette's `sessionFingerprint` (a `--model`/env model is not; `environment.model`
|
|
180
|
+
records what ran). `verify-cassettes` reports a changed session as staleness (exit 1); `replay` never
|
|
181
|
+
checks it — plain, `--strict` or `--assert-from` — so re-record if the session changed. `verify-run` reads
|
|
181
182
|
on-disk `assert:` against a kept *run dir*; `replay --assert-from` is the equivalent for a *cassette*.
|
|
182
183
|
|
|
183
184
|
18. **`questions_count_max` counts sub-questions, not gates.** One `AskUserQuestion` tool call can
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Measurement
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.1.
|
|
3
|
+
Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`). Read it before comparing runs: `--repeat`, `--ablate-skill`, and the hygiene that keeps a batch valid.
|
|
4
4
|
|
|
5
5
|
### Measure — before/after, with/without (`--repeat`, `--ablate-skill`)
|
|
6
6
|
|
|
@@ -40,7 +40,10 @@ Compare timings between runs of the same tier and model only.
|
|
|
40
40
|
|
|
41
41
|
1. **Pin the model in the session.** A run that resolves no model is refused, but one pinned only by
|
|
42
42
|
`COWORK_HARNESS_MODEL` takes its model from the machine, so two shells can run two models. Set
|
|
43
|
-
`model:` in the session (or pass the same `--model` on every `skill` run).
|
|
43
|
+
`model:` in the session (or pass the same `--model` on every `skill` run). Adding `model:` to a session
|
|
44
|
+
that already has cassettes re-stales them (`verify-cassettes` exits 1 — the model is in the session
|
|
45
|
+
fingerprint); to pin without re-recording now, use `--model` or `COWORK_HARNESS_MODEL` and move the
|
|
46
|
+
pin into the session at the next re-record. Read `result.json`'s `models` back before believing any cross-run comparison — and when
|
|
44
47
|
you do, **ignore any entry wrapped in angle brackets**: `<synthetic>` is the agent marking a turn it
|
|
45
48
|
fabricated locally (no API call), not a model, so two runs of the same pinned model can differ on this
|
|
46
49
|
array purely by whether such a turn occurred.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Run, record and lock
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.1.
|
|
3
|
+
Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`). Read it when running a scenario, recording or placing a cassette, reading verdict signals, checking a background run, or choosing CI lanes.
|
|
4
4
|
|
|
5
5
|
## Part II — RUN, RECORD & LOCK
|
|
6
6
|
|
|
@@ -25,12 +25,14 @@ the discovery/encode/record dance entirely and answer gates **live during the re
|
|
|
25
25
|
refusals the real `record` applies (`on_unanswered: prompt`, and an unsatisfiable assert pairing) **plus the
|
|
26
26
|
cassette-portability pre-flight below**, so it cannot green something a paid run would reject. **That binding
|
|
27
27
|
guarantee is the SINGLE-FILE form only** — it takes the real `--out` and the real flags, so its verdict is the
|
|
28
|
-
one a paid run would give.
|
|
28
|
+
one a paid run would give. Two inputs it reports rather than refuses, at exit 0 under `inputErrors[]` with a
|
|
29
|
+
`⚠ input error:` line: a `session:` file that cannot be read (missing, a directory, not valid YAML — 4.1.1
|
|
30
|
+
and later) and a `tool_not_called` the tier can never violate — `run` and the real `record` refuse both. On a **directory** the path-dependent verdicts (host-inventory, cassette
|
|
29
31
|
portability) are reported as `⚠ would-refuse (advisory)` / `⚠ would-warn (advisory)` notes — the label follows
|
|
30
32
|
the verdict kind, and portability can only ever warn — that do NOT affect the exit code — a dir target
|
|
31
33
|
takes no `--out`, so the destination is a guess — and only the path-independent ones (prompt policy, assert
|
|
32
|
-
contradiction, duplicate cassette target) gate the batch. An input the real record would refuse — a
|
|
33
|
-
path, an unknown baseline name, a `tool_not_called` the tier can never violate — is listed under
|
|
34
|
+
contradiction, duplicate cassette target) gate the batch. An input the real record would refuse — a
|
|
35
|
+
`session:` file that cannot be read (4.1.1 and later), a missing path, an unknown baseline name, a `tool_not_called` the tier can never violate — is listed under
|
|
34
36
|
`inputErrors[]` with a `⚠ input error:` line, also at exit 0. So a directory dry-run CAN exit 0 on a scenario the
|
|
35
37
|
real `record` would refuse; re-run that one file with its real flags for a binding answer, or gate on
|
|
36
38
|
`.ok and (.inputErrors == [])` in the JSON payload (4.1.0 and later). A directory also
|
|
@@ -68,7 +70,10 @@ finished: a failing verdict without `--allow-failing`, an assert on an artifact
|
|
|
68
70
|
quarantined inventory finding, or any other error before the cassette is written. Only the after-the-run
|
|
69
71
|
refusals report the run: under `--output-format json` they carry it in `results[0]` (verdict and
|
|
70
72
|
cost) with `error.category: "runtime"`; every pre-spend refusal, and a run that throws before returning a
|
|
71
|
-
result (an unanswered gate), has `results: []`.
|
|
73
|
+
result (an unanswered gate), has `results: []`. The envelope's `ok` is the exit code (`ok` ⇔ exit 0, a
|
|
74
|
+
cassette was written), not the verdict: an `--allow-failing` recording of a red run is `ok: true` with
|
|
75
|
+
`results[0].verdict.pass: false`. `record <dir/>` reports per scenario under `items[]` (each with `status`,
|
|
76
|
+
and `verdict`/`result` once its run finished), not `results[]`.
|
|
72
77
|
|
|
73
78
|
**Decide WHERE the cassette lives before you record it — a cassette cannot be moved afterwards.**
|
|
74
79
|
Without `--out`, `record` writes `cassettes/<scenario-name-slug>.cassette.json` (gitignored by
|
|
@@ -254,7 +259,9 @@ Recognize these before "fixing" a non-bug:
|
|
|
254
259
|
run that lands on `container` is armed. Author `transcript_no_host_path` to enforce cleanliness where
|
|
255
260
|
it's valid. A host path that came verbatim from the scenario's own uploads, connected folders or prompt
|
|
256
261
|
is not a leak (whole-token match; a token cut short where the path goes on, or one naming a location the
|
|
257
|
-
harness created for the run, is never exempt); `scan.
|
|
262
|
+
harness created for the run, is never exempt); `scan.inputHostPathTokens` counts the host-path tokens the
|
|
263
|
+
inputs carried, `scan.hostPathsFromInputs` the ones exempted, and a non-zero exemption prints a
|
|
264
|
+
`::notice::` so a clean scan that relied on it is never silent.
|
|
258
265
|
- **`exec_infra_error`** (`WARN`, host-loop) — one or more container `exec` calls failed for
|
|
259
266
|
infrastructure reasons (daemon/container-level), so those tool calls returned an error to the agent
|
|
260
267
|
rather than the command's own output. Warn-severity because the run's other evidence is intact — unlike
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, replay class, web_fetch, authoring gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 4.1.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 4.1.1`
|
|
4
4
|
(baseline `desktop-2.9939.4`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
|
|
5
5
|
[`docs/session.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md), and `SPEC.md`.
|
|
6
6
|
|
|
@@ -117,7 +117,8 @@ allow_host_hooks: true # OPTIONAL — required to run `protocol` fi
|
|
|
117
117
|
```
|
|
118
118
|
|
|
119
119
|
Relative paths resolve from the file's own directory, so a scenario + session + referenced files
|
|
120
|
-
form a relocatable bundle. `~` expands to home.
|
|
120
|
+
form a relocatable bundle. `~` expands to home (in a scenario's `session:` itself, 4.1.1 and later; a
|
|
121
|
+
cassette stores such a `session:` as written).
|
|
121
122
|
|
|
122
123
|
> **A recorded cassette is NOT part of that bundle and is NOT relocatable.** It rewrites its own
|
|
123
124
|
> references relative to **its own** directory at record time (`scenario.session` and
|
|
@@ -143,6 +144,10 @@ model: claude-opus-4-8 # a run must resolve a model: --model (or a mat
|
|
|
143
144
|
# fabricated locally, not a model id).
|
|
144
145
|
# On the ad-hoc `skill` lane there is no session file, so `--model <id>`
|
|
145
146
|
# (or COWORK_HARNESS_MODEL) is the ONLY way to pin it.
|
|
147
|
+
# Adding or changing this key re-stales every cassette recorded from
|
|
148
|
+
# the session (it is in the session fingerprint, so verify-cassettes
|
|
149
|
+
# exits 1). To pin without re-recording now, pass --model or set
|
|
150
|
+
# COWORK_HARNESS_MODEL; move it here at the next re-record.
|
|
146
151
|
account_name: my-account # OPTIONAL — display name rendered into {{accountName}} / the prompt's
|
|
147
152
|
# "User name:" line; NOT a credential/identity selector (see src/prompt.ts,
|
|
148
153
|
# https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md)
|
|
@@ -219,7 +224,10 @@ staleness:
|
|
|
219
224
|
The staleness hash uses each skill/plugin source dir's **git-tracked** file set by default (a non-repo dir
|
|
220
225
|
falls back to a raw walk; `COWORK_HARNESS_GITSET=0` opts out). **OS-junk** (`.DS_Store`/`Thumbs.db`/
|
|
221
226
|
`desktop.ini`) is always excluded, so a Finder touch can't re-stale a cassette; run-generated files a skill
|
|
222
|
-
writes into its own dir should be declared in `hash_ignore` / `.cowork-hashignore
|
|
227
|
+
writes into its own dir should be declared in `hash_ignore` / `.cowork-hashignore` — and so should
|
|
228
|
+
scenario or session YAMLs kept inside a mounted plugin dir: they are tracked files, so editing a scenario
|
|
229
|
+
there re-stales every cassette recorded from that plugin (committed `*.cassette.json` files are excluded
|
|
230
|
+
automatically). On a mismatch,
|
|
223
231
|
`verify-cassettes` names the exact changed file; `COWORK_HARNESS_DEBUG_SKILLHASH=1` dumps the full hashed set.
|
|
224
232
|
For a multi-skill plugin, scope a scenario's hash with `skills: [<name>]`; the opt-in
|
|
225
233
|
`COWORK_HARNESS_AGENT_SCOPE=skill` further treats a skill-named `agents/<name>.md` as that skill's private
|
|
@@ -240,6 +248,10 @@ with no session file, the CLI flags `--folder <dir>` and `--upload <file>` are t
|
|
|
240
248
|
(`.local-plugins/marketplaces/local-desktop-app-uploads/<plugin>`). The choice matters to a skill that
|
|
241
249
|
locates its own files from the shell (at host-loop, Cowork's default, the braced `${CLAUDE_PLUGIN_ROOT}` is
|
|
242
250
|
replaced with a HOST path and a bare `$CLAUDE_PLUGIN_ROOT` is empty in the VM shell).
|
|
251
|
+
Do not derive the VM path from the substituted host path by keeping its `/mnt/…` tail: under `hostloop`
|
|
252
|
+
the harness's staged host path happens to end in the same `/mnt/.local-plugins/…` suffix as the VM path,
|
|
253
|
+
so that shortcut passes here and fails in real Cowork, whose host path has no such tail
|
|
254
|
+
([fidelity gap](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/fidelity-gaps.md#hostloop-the-substituted-plugin-path-shares-the-vm-paths-suffix-real-coworks-does-not)).
|
|
243
255
|
Search for the skill's own `SKILL.md`, not for a directory named after the plugin — that finds nothing
|
|
244
256
|
under `.remote-plugins/plugin_<id>` — and set no `-maxdepth` that stops short of the deeper local layout:
|
|
245
257
|
`find /sessions/*/mnt/.local-plugins /sessions/*/mnt/.remote-plugins -path '*/skills/<skill-name>/SKILL.md' 2>/dev/null | head -1`.
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
4
|
decision path. Every one answers a question a real fleet owner had to work out the hard way.
|
|
5
|
-
Tracks `cowork-harness 4.1.
|
|
5
|
+
Tracks `cowork-harness 4.1.1` (baseline `desktop-2.9939.4`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
|
|
6
6
|
Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
|
|
7
7
|
needed if your CLI meets SKILL.md's version floor.
|
|
8
8
|
|
|
@@ -85,7 +85,8 @@ record — retrofitting means re-recording:
|
|
|
85
85
|
|
|
86
86
|
1. `cowork-harness init-redact` — copies the reference `.cowork-redact.json` (host-path +
|
|
87
87
|
email patterns) into the current directory. Version-control it. Re-run with `--force` after an
|
|
88
|
-
upgrade to pick up new rules
|
|
88
|
+
upgrade to pick up new rules: without it `init-redact` refuses to overwrite an existing policy, and with
|
|
89
|
+
it your tailoring is replaced — save it first and merge it back.
|
|
89
90
|
2. **Search set:** record looks for `.cowork-redact.json` in the **cwd, the scenario's directory,
|
|
90
91
|
and the cassette's output directory** — every distinct file found is MERGED (plus
|
|
91
92
|
`COWORK_HARNESS_REDACT_PATTERNS`/`_KEYS` from the env). Repo root (= cwd in CI) is the
|
|
@@ -217,7 +218,9 @@ degrade the advice. It is real work to calibrate; these steps are the traps that
|
|
|
217
218
|
|
|
218
219
|
**Lane note:** `semantic_matches` is **live-only** (the judge is a live model call), so these scenarios
|
|
219
220
|
run on the `run` lane, never token-free `replay` — the linter's "all assertions live-only" warning is
|
|
220
|
-
expected and correct here.
|
|
221
|
+
expected and correct here. It is a WARN (`replay-noop`), so a scenario whose only assertion is
|
|
222
|
+
`semantic_matches` fails `lint --strict`: add a replay-checkable item (`- result: success` is enough), or
|
|
223
|
+
keep these files out of the set a strict lint gate covers.
|
|
221
224
|
|
|
222
225
|
## Recipe 6 — Iterate a skill across fixes (ground findings, don't cross-pair generations)
|
|
223
226
|
|
|
@@ -267,7 +270,9 @@ Hardening a skill is a loop: run → read what it did → fix → run again. Two
|
|
|
267
270
|
separates them afterwards, but a hash whose source was never frozen names a generation that is
|
|
268
271
|
unrecoverable, which makes the comparison uninterpretable rather than merely noisy. And pin the model in the session (`model:`), not
|
|
269
272
|
only through `COWORK_HARNESS_MODEL`: a run that resolves none is refused, but the env var is a property
|
|
270
|
-
of the machine, so a before/after run from two shells can silently straddle two models.
|
|
273
|
+
of the machine, so a before/after run from two shells can silently straddle two models. (Adding `model:`
|
|
274
|
+
to a session that already has cassettes re-stales them in `verify-cassettes`; until the next re-record,
|
|
275
|
+
pin with `--model` instead.) Read `result.json` back to confirm: `modelSource` says whether anything pinned the model at all,
|
|
271
276
|
and `modelPinHonored` whether the pin survived (**absent means unverifiable, not "yes"**). `models`
|
|
272
277
|
lists what served the run — ignore any `<…>`-wrapped entry (`<synthetic>` marks a turn the agent
|
|
273
278
|
fabricated locally, not a model).
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,77 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [4.1.1] — 2026-09-29
|
|
10
|
+
|
|
11
|
+
### Upgrade notes
|
|
12
|
+
|
|
13
|
+
- **Cassettes: no re-record needed.** Nothing under `src/runtime`, `src/hostloop`, `src/staging` or
|
|
14
|
+
`baselines/` changed, `src/session.ts` gained only a `~` path helper, and `CASSETTE_VERSION` is still
|
|
15
|
+
13. A committed cassette's recorded content and fingerprints are unchanged; `verify-cassettes` and
|
|
16
|
+
`replay --strict` pass on the bundled ones. The one new `verify-cassettes` finding (a home directory
|
|
17
|
+
hidden behind `../` in a cassette's path metadata) applies only to a cassette recorded from outside
|
|
18
|
+
the repository — see Fixed.
|
|
19
|
+
|
|
20
|
+
### Fixed
|
|
21
|
+
|
|
22
|
+
- **A scenario whose `session:` file is missing is a usage error, not a harness bug.** `run` answered a
|
|
23
|
+
missing session file with a raw `ENOENT` under error category `internal` (exit `2`); it now refuses it
|
|
24
|
+
as category `usage` (exit `2` unchanged) with `session file not found: <path>` and a hint that
|
|
25
|
+
`session:` resolves relative to the scenario file's directory, before a run directory is created. A
|
|
26
|
+
session path that is a directory, is not readable, is not valid YAML, or cannot be read for any other
|
|
27
|
+
reason gets the same treatment, each with its own one-line message. `run <dir/>` refuses the batch before its first scenario runs, naming
|
|
28
|
+
every such file; `run --matrix` prints the reason once instead of prefixing a second "failed to load
|
|
29
|
+
session". The real `record` already refused it (exit `1`) and now gives the same clean message.
|
|
30
|
+
- **`record --dry-run` reports a session file that cannot be read.** Both the single-file and the
|
|
31
|
+
directory preview list it under `inputErrors[]` with a `⚠ input error:` line that survives `--quiet`;
|
|
32
|
+
the exit code and `ok` are unchanged. Before, the single-file preview exited `0` with a clean preview
|
|
33
|
+
and the directory preview listed the file nowhere.
|
|
34
|
+
- **A `session:` path starting with `~/` now expands to your home directory**, as the session guide
|
|
35
|
+
already said. It was read as a literal `~` directory, so such a scenario could not run. A cassette
|
|
36
|
+
recorded from one stores the `~/…` reference as written — never a path that spells out your home
|
|
37
|
+
directory — and resolves it home-relative, `--rerecord-stale --from-embedded` included. A
|
|
38
|
+
`--session=~/…` override on `replay`, `verify-cassettes`, `rehash` and `boundary-check` expands too.
|
|
39
|
+
A `~<user>/…` path (another user's home) is an input error on every lane, never an internal crash.
|
|
40
|
+
- **A scenario file that cannot be read is a usage error on `run`.** A scenario file that is not valid
|
|
41
|
+
YAML, is not readable, or is a directory matched by `run <dir/>` was reported under category
|
|
42
|
+
`internal` with the YAML parser's multi-line source excerpt; it is now category `usage` (exit `2`
|
|
43
|
+
unchanged) with a one-line message naming the file, the same message `record --dry-run` gives.
|
|
44
|
+
- **A `baseline:` path to a file that does not load is a usage error.** An absolute `baseline:` path
|
|
45
|
+
that is a directory, is not valid JSON, or is not a platform baseline surfaced as a raw error under
|
|
46
|
+
category `internal`; it now fails like an unknown baseline name, with one line saying which (`record`
|
|
47
|
+
keeps its exit `1` for it, on `--dry-run` and the real record alike). `run <dir/>`
|
|
48
|
+
refuses it before the first scenario runs instead of failing on that scenario's turn, and `lint`
|
|
49
|
+
reports it as `baseline-unknown` when the path exists on the machine it runs on (a path that does not
|
|
50
|
+
exist there is still not checked). Committed baseline names are unchanged.
|
|
51
|
+
- **`verify-cassettes` flags a home directory hidden behind `../`.** A cassette stores `scenario.session`
|
|
52
|
+
and `scenarioSource` relative to itself (the fingerprint's skill sources are relative to the session
|
|
53
|
+
file's directory, its file list to each skill directory), so a target under a home directory is written
|
|
54
|
+
`../../Users/<name>/…` and never started with the `/Users/` the path scan looks for. A relative climb
|
|
55
|
+
into `Users`, `home` or `root` in any of those fields is now a `path` finding (exit `1`). The check runs
|
|
56
|
+
at verify time: `record` still writes the relative path it computes, and no redaction rule rewrites these
|
|
57
|
+
fields. To clear it, record from inside the repo that holds the session and scenario, use a `~/…`
|
|
58
|
+
`session:` (stored as written), or accept it with `--allow-path '(\.\./)+Users/<name>.*'` — an
|
|
59
|
+
existing `--allow-path '/Users/<name>.*'` does not match the relative sample.
|
|
60
|
+
- **A YAML syntax error in a `--matrix` or `--answer-policy` file is reported in one line** — the
|
|
61
|
+
parser's first line, which names the problem and its position — instead of its source excerpt.
|
|
62
|
+
|
|
63
|
+
### Documentation
|
|
64
|
+
|
|
65
|
+
- **The companion skill caught up with 4.x.** Corrected: the session's `model:` IS part of a cassette's
|
|
66
|
+
session fingerprint — `verify-cassettes` reports a changed session as stale, while `replay` never checks
|
|
67
|
+
it (plain, `--strict` or `--assert-from`) — so adding `model:` to a session re-stales its cassettes; pin
|
|
68
|
+
with `--model` or `COWORK_HARNESS_MODEL` until the next re-record. Added: `record`'s `ok` is the exit
|
|
69
|
+
code, not the verdict (`--allow-failing` on a red run is `ok: true`; the verdict is in `results[0]`, and
|
|
70
|
+
in `items[]` on `record <dir/>`); a scenario whose only assertion is `semantic_matches` fails
|
|
71
|
+
`lint --strict` on `replay-noop`; the hostloop plugin-path suffix trap; scenario YAMLs inside a mounted
|
|
72
|
+
plugin dir are part of the staleness hash; `scan.inputHostPathTokens` and its notice; the `toolCalls`
|
|
73
|
+
field; `COWORK_HARNESS_OUTPUT_FORMAT`; the microvm capability-cache reset; why `init-redact --force`
|
|
74
|
+
drops your tailoring.
|
|
75
|
+
- **Every `lint` rule is documented.** The skill's `authoring.md` now carries a table of every rule id
|
|
76
|
+
`cowork-harness lint` can report, with its severity, and a section on linting the skill itself
|
|
77
|
+
(`lint-skill`'s size caps, `--ignore-rule`, suppression markers). A test keeps the table in step with
|
|
78
|
+
the rule registries in both directions.
|
|
79
|
+
|
|
9
80
|
## [4.1.0] — 2026-09-29
|
|
10
81
|
|
|
11
82
|
### Upgrade notes
|
package/README.md
CHANGED
|
@@ -36,7 +36,7 @@ npm ci && npm run build
|
|
|
36
36
|
node dist/cli.js replay examples/replays/example-pdf-skill.cassette.json
|
|
37
37
|
```
|
|
38
38
|
|
|
39
|
-
(Installing globally — `npm install -g "cowork-harness@^4.1.
|
|
39
|
+
(Installing globally — `npm install -g "cowork-harness@^4.1.1"` — gives you the `cowork-harness` CLI for your own
|
|
40
40
|
scenarios and cassettes; the bundled example above also replays from a global install — see the `$(npm root -g)` path below.)
|
|
41
41
|
|
|
42
42
|
Full setup → [Quick start](./docs/cli.md#quick-start).
|
|
@@ -49,8 +49,8 @@ Three ways to use this project. Each row is the whole hook — follow the link f
|
|
|
49
49
|
|
|
50
50
|
| I want to… | Start here | Needs |
|
|
51
51
|
|---|---|---|
|
|
52
|
-
| **Run scenarios myself** from a terminal | **[docs/cli.md](./docs/cli.md)**<br><br>`npm i -g "cowork-harness@^4.1.
|
|
53
|
-
| **Have Claude Code drive it** for me | **[docs/companion-skill.md](./docs/companion-skill.md)**<br><br>`/plugin marketplace add yaniv-golan/cowork-harness`<br>`/plugin install cowork-harness@cowork-harness` | Claude Code. The skill self-bootstraps the CLI via `npx "cowork-harness@^4.1.
|
|
52
|
+
| **Run scenarios myself** from a terminal | **[docs/cli.md](./docs/cli.md)**<br><br>`npm i -g "cowork-harness@^4.1.1"`<br>`cowork-harness replay examples/replays/example-pdf-skill.cassette.json` | Node ≥ 22. The replay demo above is token-free and needs nothing else; live tiers above `protocol` need Docker + a staged agent binary |
|
|
53
|
+
| **Have Claude Code drive it** for me | **[docs/companion-skill.md](./docs/companion-skill.md)**<br><br>`/plugin marketplace add yaniv-golan/cowork-harness`<br>`/plugin install cowork-harness@cowork-harness` | Claude Code. The skill self-bootstraps the CLI via `npx "cowork-harness@^4.1.1"` |
|
|
54
54
|
| **Gate my skill in CI** | **[docs/ci.md](./docs/ci.md)**<br><br>`- uses: yaniv-golan/cowork-harness@v4`<br>` with: { command: replay, path: cassettes/ }` | Nothing for the token-free gate; the live lane needs a self-hosted runner with Docker + an agent binary |
|
|
55
55
|
|
|
56
56
|
**In short:** three ways in (the table above). Five `fidelity:` tiers — `protocol`, `container`, `microvm`, `hostloop`,
|