cowork-harness 1.5.0 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,10 +1,10 @@
1
1
  ---
2
2
  name: cowork-harness
3
- description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
3
+ description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
4
4
  metadata:
5
5
  author: cowork-harness
6
- version: 1.5.0
7
- tracks-harness: cowork-harness 1.5.0 (baseline desktop-1.22209.3)
6
+ version: 1.6.0
7
+ tracks-harness: cowork-harness 1.6.0 (baseline desktop-1.22209.3)
8
8
  ---
9
9
 
10
10
  # cowork-harness
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
22
22
  allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
23
23
  the highest-value part. Read it.
24
24
 
25
- > **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.5.0` (baseline
25
+ > **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.6.0` (baseline
26
26
  > `desktop-1.22209.3`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
27
27
  > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
28
28
 
@@ -39,9 +39,21 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
39
39
 
40
40
  - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
41
41
  - **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
42
- - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.5.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.5.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=1.5.0"`. **Pin `@>=1.5.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
43
-
44
- What the ≥ 1.5.0 floor gates, by release:
42
+ - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.6.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.6.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=1.6.0"`. **Pin `@>=1.6.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
43
+
44
+ What the ≥ 1.6.0 floor gates, by release:
45
+
46
+ - **1.6.0 (`critique`'s skill-flag parity):** `critique` accepts most `skill` flags under the same names,
47
+ which is what makes a document-analysis skill (cap table, deck, financial model, transcript)
48
+ critiquable at all: `--upload`/`--folder`/`--plugin`/`--marketplace`/`--enable`/`--model`/
49
+ `--allow-missing-capability` reach BOTH spawned turns; `--label`/`--timeout`/`--answer`/
50
+ `--answer-policy`/`--on-unanswered`/the decider flags reach the graded turn only; `--prompt-file` is
51
+ new. Anything that can't work is REFUSED with a reason rather than silently ignored.
52
+ **Repeating a flag:** `--upload`/`--folder`/`--plugin`/`--marketplace`/`--enable`/`--answer`
53
+ accumulate; every other value-taking flag is single-valued and repeating it is a usage error (exit `2`)
54
+ rather than a silent last-wins — do not assume the CLI-conventional "last one wins". Booleans repeat
55
+ harmlessly. On `1.5.0` these flags do not exist: `critique --upload x.xlsx` fails as "unknown flag", so
56
+ if a user hits that, they are below the floor.
45
57
 
46
58
  - **1.5.0:** `critique <skill-folder> --prompt "<probe>"` (EXPERIMENTAL) — runs a skill, asks the agent
47
59
  what confused it, then grades that self-report against a frozen record of the run: a blinded evaluator
@@ -69,7 +81,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
69
81
  - **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
70
82
  - **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
71
83
  - **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred), or `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
72
- - **`--dotenv` is a GLOBAL flag — put it BEFORE the subcommand.** `cowork-harness --dotenv .env record …`, never `cowork-harness record … --dotenv .env`. Every *other* flag is subcommand-level, so muscle memory fights this one; the harness rejects the misplaced form with an exact-fix error, but placing it first avoids the round-trip.
84
+ - **`--dotenv` is a GLOBAL flag — put it BEFORE the subcommand.** `cowork-harness --dotenv .env record …`, never `cowork-harness record … --dotenv .env`. Every *other* flag is subcommand-level, so muscle memory fights this one; the harness rejects the misplaced form with an exact-fix error, but placing it first avoids the round-trip. **One exception: `critique` also accepts `--dotenv` per-command** (`critique <folder> --prompt "…" --dotenv <path>`) — UNRELEASED, see the floor list; `--run-dir` stays global-only everywhere.
73
85
 
74
86
  ## Orient — the three loops
75
87
 
@@ -1,6 +1,6 @@
1
1
  # CI recipe — replay vs live lanes
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 1.5.0` (baseline `desktop-1.20186.1`).
3
+ Self-contained reference. Tracks `cowork-harness 1.6.0` (baseline `desktop-1.20186.1`).
4
4
 
5
5
  **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
6
6
  job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
13
13
  ```
14
14
 
15
15
  The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
16
- release; pin an exact version (e.g. `version: "1.5.0"`) for reproducible CI.
16
+ release; pin an exact version (e.g. `version: "1.6.0"`) for reproducible CI.
17
17
 
18
18
  Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
19
19
  cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
@@ -57,7 +57,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
57
57
  GitHub-hosted runners, no token/Docker/agent:
58
58
 
59
59
  ```yaml
60
- - run: npm i -g "cowork-harness@>=1.5.0"
60
+ - run: npm i -g "cowork-harness@>=1.6.0"
61
61
  - run: cowork-harness lint scenarios/*.yaml # no silent false-greens
62
62
  - run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
63
63
  - run: cowork-harness replay cassettes/ # token-free content/structure
@@ -197,7 +197,7 @@ jobs:
197
197
  with: { node-version: '20' }
198
198
  - uses: actions/setup-python@v5
199
199
  with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
200
- - run: npm i -g "cowork-harness@>=1.5.0"
200
+ - run: npm i -g "cowork-harness@>=1.6.0"
201
201
  - run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
202
202
  - run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
203
203
  - run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
@@ -226,7 +226,7 @@ jobs:
226
226
  echo "live=true" >> "$GITHUB_OUTPUT"
227
227
  fi
228
228
  - if: steps.guard.outputs.live == 'true'
229
- run: npm i -g "cowork-harness@>=1.5.0"
229
+ run: npm i -g "cowork-harness@>=1.6.0"
230
230
  - if: steps.guard.outputs.live == 'true'
231
231
  run: cowork-harness run scenarios/ --output-format json
232
232
  env:
@@ -1,6 +1,6 @@
1
1
  # Fidelity tiers & answer paths
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 1.5.0` (baseline `desktop-1.20186.1`).
3
+ Self-contained reference. Tracks `cowork-harness 1.6.0` (baseline `desktop-1.20186.1`).
4
4
 
5
5
  ## Fidelity tiers (`fidelity:` in the scenario)
6
6
 
@@ -122,6 +122,7 @@ hands `fn` exactly this dict.
122
122
 
123
123
  - On `skill`: `fail | prompt | first`.
124
124
  - On `run`: `fail | first` (`prompt` rejected — it would break determinism).
125
+ - On `critique` (UNRELEASED): `fail | first` (`prompt` rejected — there is no TTY inside the spawned turn).
125
126
  - **`llm` is NOT an `--on-unanswered` value.** The bare flag `--on-unanswered llm` is rejected; use
126
127
  `--decider-llm` (CLI) or `on_unanswered: llm` (YAML).
127
128
  - `agent` is **retired** — `on_unanswered: agent` is rejected by the schema. The enum is
@@ -1,6 +1,6 @@
1
1
  # Scenario & session schema, assertion catalog, web_fetch, full gotchas
2
2
 
3
- Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.5.0`
3
+ Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.6.0`
4
4
  (baseline `desktop-1.20186.1`). If your checkout is newer, prefer the live `docs/scenario.md`,
5
5
  `docs/session.md`, and `SPEC.md`.
6
6
 
@@ -179,7 +179,11 @@ Hardening a skill is a loop: run → read what it did → fix → run again. Two
179
179
  The full loop (harvest -> reproduce -> fix -> prove freshness -> compare) is written out end-to-end in
180
180
  docs/debugging.md under "The whole loop, end to end". The harness now SHIPS a grader — `cowork-harness critique <skill-folder> --prompt "<probe>"` runs the
181
181
  skill, asks the agent what confused it, and grades that self-report against a frozen record of the run
182
- (blinded evaluator + mechanical citation checking). See docs/critique.md for cost and limits. If you
182
+ (blinded evaluator + mechanical citation checking). **Critiquing a document-analysis skill?** The probe
183
+ attaches nothing on its own — pass `--upload <path>` (repeatable) or `--folder <dir>` exactly as you
184
+ would to `skill`, or the graded run has no file and "there was no file attached" is the correct finding,
185
+ not a skill defect. Source flags reach both spawned turns automatically. UNRELEASED — see SKILL.md's
186
+ floor list. See docs/critique.md for the full flag table, cost and limits. If you
183
187
  prefer to build your own grader, the substrate is still here:
184
188
  - `result.json` → `finalMessage` (the skill's own answer/critique) + `toolResults[]` (tool outputs).
185
189
  - `cowork-harness trace <run-dir> --output-format json` → the tool-call stream. Add `--full-results` so
package/CHANGELOG.md CHANGED
@@ -6,6 +6,48 @@ All notable changes to this project are documented here. The format is based on
6
6
 
7
7
  ## [Unreleased]
8
8
 
9
+ ## [1.6.0] — 2026-07-20
10
+
11
+ ### Added
12
+
13
+ - **`critique` now accepts the `skill` flags a graded run needs — starting with uploads.** A skill whose
14
+ whole job is "here is a document, analyze it" (cap table, pitch deck, financial model, transcript) could
15
+ not be critiqued at all: there was no way to attach the file, so the agent was asked to analyze a
16
+ document that was never there and then asked what confused it — you harvested a finding about the test
17
+ rig. `--upload`, `--folder`, `--plugin`, `--marketplace`/`--enable` (all repeatable), `--model` and
18
+ `--allow-missing-capability` now reach **both** spawned turns; that is forced, not stylistic, because
19
+ those paths are part of the session-origin key the reflection turn's `--resume` recomputes. `--label`,
20
+ `--timeout`, `--answer`/`--answer-policy`, `--on-unanswered` and the decider flags reach the graded turn
21
+ only. `--answer`/`--answer-policy` make **gated** skills critiquable for the first time — the inner spawn
22
+ has no TTY, so an unscripted gate previously killed the task turn before anything could be graded.
23
+ Anything that cannot work is refused **with its reason** rather than silently ignored:
24
+ `--session-id`/`--resume`, the `--repeat` family, `--ablate-skill`, and the rendering/preview flags.
25
+ - **`critique --prompt-file <path>`** — read the probe verbatim from a file, so a probe containing quotes,
26
+ `$` or newlines does not have to survive shell parsing.
27
+ - **"Attached inputs" evidence section** — upload filenames and sizes, plus connected-folder mount names,
28
+ never content. Without it the evaluator could not tell "the agent said there was no file, and correctly
29
+ so" from a confabulation.
30
+ - **One source of truth for the `skill` flag surface** (`src/run/skill-flag-surface.ts`), where each flag's
31
+ critique disposition is a **required** field, plus a parity guard that fails CI when a new `skill` flag
32
+ arrives without one. The old hand-rolled subset drifted silently — this repo's recurring bug shape.
33
+
34
+ ### Changed
35
+
36
+ - **Repeating a single-valued `critique` flag is now a usage error (exit `2`) instead of silent
37
+ last-wins.** `--upload`, `--folder`, `--plugin`, `--marketplace`, `--enable` and `--answer` accumulate —
38
+ repeating them is how you pass several. Everything else is single-valued, and `--prompt a --prompt b`
39
+ silently discarding a probe you typed is the class of no-op this command refuses on principle. Boolean
40
+ flags may still be repeated harmlessly. Documented in `critique --help` and
41
+ [docs/critique.md](./docs/critique.md).
42
+
43
+ ### Fixed
44
+
45
+ - **`critique --dotenv <path>` was documented but unreachable** (shipped that way in 1.5.0). The
46
+ misplaced-global guard rejected the token after any subcommand, and the `--dotenv=x` form that slipped
47
+ past it then hit critique's exact-match parser as "unknown flag". `critique` is now exempt from the guard
48
+ (`--run-dir` still has no per-command meaning and stays rejected everywhere), and a missing file fails
49
+ fast with critique's own error instead of surfacing later as an instrument-failure diagnostic.
50
+
9
51
  ## [1.5.0] — 2026-07-20
10
52
 
11
53
  ### Added
package/README.md CHANGED
@@ -91,7 +91,7 @@ node dist/cli.js replay examples/replays/example-pdf-skill.cassette.json
91
91
 
92
92
  > **Installed globally instead?** Once linked/installed, the same command is `cowork-harness replay
93
93
  > <cassette>` — but the relative path above only resolves from a source checkout's `examples/replays/`.
94
- > From a global install (`npm i -g "cowork-harness@>=1.5.0"`), point at the package root instead:
94
+ > From a global install (`npm i -g "cowork-harness@>=1.6.0"`), point at the package root instead:
95
95
  > `cowork-harness replay "$(npm root -g)/cowork-harness/examples/replays/example-pdf-skill.cassette.json"`
96
96
  > (or copy the cassette into your own project and pass that path).
97
97
 
@@ -101,7 +101,7 @@ Live `run`/`skill` need the prerequisites in the next section — note the `prot
101
101
  > - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
102
102
  > - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
103
103
  > - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
104
- > - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=1.5.0"`.
104
+ > - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=1.6.0"`.
105
105
 
106
106
  Two more worked examples worth knowing about: `examples/scenarios/protocol-smoke.yaml` (zero-Docker smoke
107
107
  test) and `examples/scenarios/skill-loads.yaml` (container-tier acceptance check) — see
@@ -126,7 +126,7 @@ claude plugin marketplace add yaniv-golan/cowork-harness
126
126
  claude plugin install cowork-harness@cowork-harness
127
127
  ```
128
128
 
129
- The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=1.5.0"` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
129
+ The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=1.6.0"` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
130
130
 
131
131
  It also follows the open [Agent Skills](https://agentskills.io) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) via [`npx skills`](https://github.com/vercel-labs/skills) (Vercel Labs' CLI implementation of that spec):
132
132
 
@@ -147,7 +147,7 @@ A global install is enough for CI `lint`, reading the teaching skill, and replay
147
147
  To `run` the worked examples live or copy them as a starting point, use a source checkout. (The marketplace
148
148
  skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
149
149
  assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
150
- above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=1.5.0"` — see
150
+ above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=1.6.0"` — see
151
151
  [above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
152
152
 
153
153
  ### Prerequisites for anything above `protocol` fidelity
@@ -415,7 +415,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
415
415
  | `boundary-check [baseline] [--session <file>]` | Prove the **L1 Docker** sandbox enforces Cowork's limitations (sealed FS + default-deny egress; `container`/`hostloop` share this sandbox — `microvm`'s guest firewall is not probed here); `--session` folds a session's `egress.extra_allow` into the probe allowlist | verifying the harness's own fidelity |
416
416
  | `sync` / `list` | Derive/refresh (`sync [--diff] [--allow-empty\|--force]`) & list platform baselines from the Desktop install | after Claude Desktop updates (baselines ship, so it's optional otherwise) |
417
417
  | `diff <a> <b>` | Compare two baselines, two runs, two cassettes, or a run+cassette — kind auto-detected by content (view/normalization flags below). Token-free, no live Desktop/Docker needed | "what changed between two runs/cassettes/baselines?" |
418
- | `critique <skill-folder>` | **EXPERIMENTAL.** Run a skill, ask the agent what confused it, then grade that self-report against a frozen record of the run — blinded evaluator, mechanical citation checking. Discovery instrument, never a gate (findings always exit 0). Costs four model workloads; see [docs/critique.md](./docs/critique.md) | "what confused the agent about my skill?" |
418
+ | `critique <skill-folder>` | **EXPERIMENTAL.** Run a skill, ask the agent what confused it, then grade that self-report against a frozen record of the run — blinded evaluator, mechanical citation checking. Accepts the `skill` flags a graded run needs — `--upload`/`--folder`/`--plugin` reach both spawned turns, `--answer`/`--decider-*` the graded turn only; anything that can't work is refused with a reason (full table in `critique --help`). Discovery instrument, never a gate (findings always exit 0). Costs four model workloads; see [docs/critique.md](./docs/critique.md) | "what confused the agent about my skill?" — including "does it even see the attached document?" |
419
419
  | `doctor [--tier <t>]` | Read-only prerequisite check, per tier (Docker + agent image for `container`/`hostloop`/`cowork`; **Lima** for `microvm`; plus staged agent, token, baseline); prints the exact `docker build` line if the agent image is missing | "can I run the live tiers — what's missing?" before a first live run |
420
420
  | `prune [<runs-dir>] [--keep-last <n>] [--pinned-older-than <N>d\|h\|m]` | Prune accumulated run dirs (keeps the N most recent per scenario; pinned `--session-id` runs are never pruned unless `--pinned-older-than` opts in to reclaiming stale ones by last-activity age); the optional positional overrides the runs root; `--dry-run` | the machine-global runs root has grown and you want space back |
421
421
  | `rehash <dir/>` | Migrate cassette fingerprints to the current format version when the content is provably unchanged (`--dry-run`); no re-record needed | a cassette-format bump flagged committed fixtures as stale |
@@ -671,7 +671,7 @@ jobs:
671
671
  anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
672
672
  ```
673
673
 
674
- Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` — intentional so recipes track the current release; pin an exact version for reproducible CI. The companion skill's `cowork-harness@>=1.5.0` floor guidance applies to ad-hoc CLI installs, not this input), `strict` (applies to `replay` (staleness findings), `lint`/`lint-skill` (WARN/INFO), and `analyze-skill` (any advisory finding); IGNORED — not forwarded — for `verify-cassettes`/`run`, which don't accept the flag), `fail-on-skill-drift` (**`replay`-only** — never forwarded to the analyzers), `extra-args`, `summary` (default `true`), `anthropic-api-key` (live lane only). See [`action.yml`](./action.yml) for the full input reference.
674
+ Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` — intentional so recipes track the current release; pin an exact version for reproducible CI. The companion skill's `cowork-harness@>=1.6.0` floor guidance applies to ad-hoc CLI installs, not this input), `strict` (applies to `replay` (staleness findings), `lint`/`lint-skill` (WARN/INFO), and `analyze-skill` (any advisory finding); IGNORED — not forwarded — for `verify-cassettes`/`run`, which don't accept the flag), `fail-on-skill-drift` (**`replay`-only** — never forwarded to the analyzers), `extra-args`, `summary` (default `true`), `anthropic-api-key` (live lane only). See [`action.yml`](./action.yml) for the full input reference.
675
675
 
676
676
  The provided [GitHub Actions workflow](.github/workflows/ci.yml) runs a **six-stage pipeline**. The **unit** stage is the token-free gate you can copy into your skill repo; the `action-self-test`, `python`, `boundary`, `scenarios`, and `parity-drift` stages are this repo's own fidelity self-tests and are not directly portable (they build the harness's Docker image and run harness-specific e2e scenarios — see [`ci-recipe.md`](./.claude/skills/cowork-harness/references/ci-recipe.md) for the skill-repo template):
677
677
 
@@ -714,7 +714,7 @@ Most runs need **none** of these — the defaults are correct. They're grouped b
714
714
  - `COWORK_HARNESS_JUDGE_MODEL` — the default model for `semantic_matches`' LLM judge (overridden by a
715
715
  per-assertion `judge_model`; falls back to the pinned default `claude-opus-4-8`). Pin it alongside `judge_model` for a
716
716
  reproducible before/after comparison across re-records.
717
- - `COWORK_HARNESS_RUNS_DIR` (or the `--run-dir <path>` flag — a **global** flag that must precede the subcommand, like `--dotenv`) — override the default run-output root `~/.cowork-harness/runs` (kept out of any working tree so sensitive skill inputs/outputs don't land in a repo). Precedence: `--run-dir` > env > default. The root is flat and machine-global (shared across projects); pinned `--session-id` runs are guarded against cross-project overwrite, and `prune` never prunes them. In CI, set it to a workspace path (e.g. `runs`) so artifact upload can collect the runs. `COWORK_HARNESS_ALLOW_FOREIGN_RESUME=1` overrides the guard that blocks `--resume` onto another project's pinned session.
717
+ - `COWORK_HARNESS_RUNS_DIR` (or the `--run-dir <path>` flag — a **global** flag that must precede the subcommand — `--dotenv` follows the same rule everywhere except `critique`, which also takes it per-command) — override the default run-output root `~/.cowork-harness/runs` (kept out of any working tree so sensitive skill inputs/outputs don't land in a repo). Precedence: `--run-dir` > env > default. The root is flat and machine-global (shared across projects); pinned `--session-id` runs are guarded against cross-project overwrite, and `prune` never prunes them. In CI, set it to a workspace path (e.g. `runs`) so artifact upload can collect the runs. `COWORK_HARNESS_ALLOW_FOREIGN_RESUME=1` overrides the guard that blocks `--resume` onto another project's pinned session.
718
718
  - **Networking / loop:** `COWORK_EGRESS_PROXY` overrides the egress-proxy URL injected into the sandbox; `COWORK_PROXY_IMAGE` overrides the egress proxy Docker image name (default `cowork-egress-proxy:2`); `COWORK_DOCKER_NETWORK` pins the Docker network the agent container joins; `CLAUDE_FORCE_HOST_LOOP=1` forces the host-loop path regardless of the baseline's loop decision (the `cowork` tier's auto-pick). `COWORK_LIMACTL` overrides the `limactl` binary path (default `/opt/homebrew/bin/limactl`).
719
719
  - `COWORK_HARNESS_PRERUN_HASH_CAP` — override the default cap on pre-run file hashing (bytes); raise it if `input_unmodified`/`no_unexpected_files` report evidence unavailable on a large connected folder.
720
720
  - `COWORK_HARNESS_MAX_ARTIFACT_BYTES` — override the inline-artifact-body cap (default 65536 bytes;
package/dist/cli.js CHANGED
@@ -608,7 +608,11 @@ async function main() {
608
608
  // more specific "<flag> requires a value" error; both exit 2, and the input is rare (value omitted AND
609
609
  // literally equal to the token).
610
610
  if (!hasHelp(rest) && COMMANDS.includes(cmd)) {
611
- const misplacedGlobal = rest.find((t) => t === "--dotenv" || t === "--run-dir");
611
+ // `critique` is exempt from the `--dotenv` half: it is a legitimate PER-COMMAND flag there (forwarded
612
+ // to the inner `skill` turns and, since the fix below, validated by critique's own parseArgs), not a
613
+ // misplaced global. `--run-dir` stays rejected for every command, `critique` included — nothing reads
614
+ // a per-command `--run-dir`.
615
+ const misplacedGlobal = rest.find((t) => (t === "--dotenv" && cmd !== "critique") || t === "--run-dir");
612
616
  if (misplacedGlobal)
613
617
  fail(cmd, "usage", `${misplacedGlobal} is a GLOBAL flag and must come BEFORE the subcommand (e.g. \`cowork-harness ${misplacedGlobal} <path> ${cmd} …\`)`, undefined, isJsonOutput(rest));
614
618
  }
@@ -12,7 +12,8 @@
12
12
  // so the tier is pinned rather than left to the caller.
13
13
  import { spawn } from "node:child_process";
14
14
  import { fileURLToPath } from "node:url";
15
- import { existsSync } from "node:fs";
15
+ import { lookupSkillFlag } from "../run/skill-flag-surface.js";
16
+ import { existsSync, readFileSync } from "node:fs";
16
17
  import { randomUUID } from "node:crypto";
17
18
  import { writeSync } from "node:fs";
18
19
  import { basename } from "node:path";
@@ -49,18 +50,49 @@ you while you worked:
49
50
 
50
51
  Answer plainly, in prose. Do not restate the task's final answer.`;
51
52
  function usage() {
52
- return `cowork-harness critique <skill-folder> --prompt "<probe>"
53
+ return `cowork-harness critique <skill-folder> --prompt "<probe>" | --prompt-file <path>
53
54
 
54
55
  EXPERIMENTAL. Runs the skill, asks the agent what confused it, then does NOT believe the answer:
55
56
  a blinded evaluator grades the self-report against a frozen record of what actually happened, and
56
57
  drops any claim whose citation is not verbatim in that evidence. Discovery instrument, not a gate.
57
58
 
58
- --prompt "<probe>" the task to run the skill against (required)
59
- --evaluator-model <id> override the evaluator model (env: COWORK_HARNESS_EVALUATOR_MODEL)
60
- --dotenv <path> load credentials from a dotenv file
61
- --fidelity container container tier only — the resume-continuity this loop depends on is
62
- verified there and nowhere else
63
- --output-format json|text
59
+ Probe (one required):
60
+ --prompt "<probe>" the task to run the skill against
61
+ --prompt-file <path> read the probe verbatim from a file (no shell parsing)
62
+
63
+ Files and sources (forwarded to the graded run — REQUIRED for "analyze this document" skills):
64
+ --upload <path> mount a file at mnt/uploads/<name> (repeatable)
65
+ --folder <dir> connect a folder at mnt/<name> (repeatable)
66
+ --plugin <dir> | --marketplace <dir> --enable <name@mkt> extra skill sources
67
+
68
+ Graded-run tuning (shapes the run being graded):
69
+ --model <id> session model for the agent doing the work AND reflecting
70
+ --timeout <ms> wall-clock budget for the task turn
71
+ --label <tag> generation tag in the run index (pair critiques across fixes)
72
+ --allow-missing-capability don't fail EITHER turn on a lean-image capability gap (both turns)
73
+ --answer "<q-regex>=<choice>" | --answer-policy <yaml> pre-answer the skill's gates (repeatable)
74
+ --on-unanswered fail|first unscripted-gate policy ('prompt' is refused: no TTY inside the spawn)
75
+ --decider-llm [--intent "<line>"] [--decider-model <id>] | --decider-cmd '<helper>' | --decider-dir <dir>
76
+ answer LIVE gates in the graded run (see 'skill --help')
77
+
78
+ Critique's own:
79
+ --evaluator-model <id> the grading model (env: COWORK_HARNESS_EVALUATOR_MODEL)
80
+ --output-format json|text critique's REPORT format (inner turns always speak json internally)
81
+ --fidelity container container tier only
82
+ --keep accepted as a no-op — runs are always kept
83
+ --dotenv <path> credentials
84
+ Global --run-dir <path> must PRECEDE the subcommand
85
+
86
+ Not accepted (each errors with its reason rather than being silently ignored):
87
+ --session-id / --resume critique mints and manages its own session internally
88
+ --repeat + companions fixed two-turn protocol — loop critique itself; pair by fingerprint.skillHash
89
+ --ablate-skill grading a skill you removed is incoherent
90
+ --quiet/--verbose/--compact/--demo/--dry-run inner-turn rendering or preview — no effect on the report
91
+
92
+ Repeating a flag: --upload/--folder/--plugin/--marketplace/--enable/--answer accumulate (that is how you
93
+ pass several). Every other value-taking flag is single-valued and repeating it is a USAGE ERROR rather
94
+ than a silent last-wins — '--prompt a --prompt b' would otherwise discard a probe you typed. Boolean
95
+ flags may be repeated harmlessly.
64
96
 
65
97
  COST AND PREREQUISITES — read before running:
66
98
  * Each critique is FOUR model workloads: two container runs (task + reflection) and two evaluator
@@ -79,6 +111,33 @@ KNOWN LIMITATIONS: container tier only; SKILL.md is capped at 16KB (a larger one
79
111
  the instruction plane from evidence but cannot stop hostile content that merely ARGUES — see
80
112
  docs/critique.md.`;
81
113
  }
114
+ /** Reads a value-flag that may appear as EITHER `--flag value` (space form) OR `--flag=value` (equals
115
+ * form). critique's argv parsing runs in a separate process from cli.ts's `flagValue`/`flagValueEitherForm`
116
+ * helpers (a process-boundary-separated CLI — see the plan's Option (c) rejection), so it gets its own
117
+ * small copy rather than importing those private helpers. Returns `[value, extraTokensConsumed]` — 0 for
118
+ * the equals form (the value is inline in `a`), 1 for the space form (the value is the NEXT token) — so
119
+ * the caller's `i += extraTokensConsumed` advances the loop exactly like the old `argv[++i]` did. */
120
+ /** Returns [value, indexAdvance] and whether the EQUALS form was used (the child's escape hatch for a
121
+ * value starting with `-`, which its spaced-form parser rejects).
122
+ *
123
+ * The empty/missing check lives HERE, not at the call sites: it was previously applied only in the
124
+ * spec-forwarding branch, so `critique … --dotenv` with a forgotten path was ACCEPTED and ran a full
125
+ * four-workload critique without loading env — a silent no-op on the flag this branch made reachable. */
126
+ function flagVal(argv, i, flag) {
127
+ const a = argv[i];
128
+ if (a.startsWith(`${flag}=`)) {
129
+ const value = a.slice(flag.length + 1);
130
+ if (value.trim() === "")
131
+ throw new Error(`${flag} requires a non-empty value\n${usage()}`);
132
+ return { value, adv: 0, equalsForm: true };
133
+ }
134
+ const value = argv[i + 1];
135
+ // trim(), matching the child's own value checks — otherwise `--label " "` passes here and dies
136
+ // one layer later, which is the failure shape this check exists to prevent.
137
+ if (value === undefined || value.trim() === "")
138
+ throw new Error(`${flag} requires a value\n${usage()}`);
139
+ return { value, adv: 1, equalsForm: false };
140
+ }
82
141
  function parseArgs(argv) {
83
142
  const positional = [];
84
143
  let prompt;
@@ -86,32 +145,143 @@ function parseArgs(argv) {
86
145
  let fidelity = "container";
87
146
  let evaluatorModel;
88
147
  let outputFormat = "text";
148
+ let promptFile;
149
+ let taskTimeoutMs;
150
+ const forwardBoth = [];
151
+ const forwardTask = [];
152
+ const seen = new Set();
153
+ /** A repeat of a non-repeatable flag silently discards the earlier value — the exact no-op this
154
+ * command's refusal design exists to prevent. Applied to critique's OWN flags too: an earlier version
155
+ * guarded only the forwarded branch, so `--prompt a --prompt b` quietly dropped a probe the user typed.
156
+ * Arity-0 flags are exempt: there is no value to lose, and the child accepts them idempotently. */
157
+ const once = (flag, arity = 1) => {
158
+ if (arity === 0)
159
+ return;
160
+ if (seen.has(flag))
161
+ throw new Error(`${flag} given more than once (it is not repeatable)\n${usage()}`);
162
+ seen.add(flag);
163
+ };
89
164
  for (let i = 0; i < argv.length; i++) {
90
165
  const a = argv[i];
91
- if (a === "--prompt")
92
- prompt = argv[++i];
93
- else if (a === "--dotenv")
94
- dotenv = argv[++i];
95
- else if (a === "--fidelity")
96
- fidelity = argv[++i];
97
- else if (a === "--evaluator-model")
98
- evaluatorModel = argv[++i];
99
- else if (a === "--output-format")
100
- outputFormat = argv[++i];
101
- else if (a.startsWith("-"))
102
- throw new Error(`unknown flag: ${a}\n${usage()}`);
166
+ if (a === "--prompt" || a.startsWith("--prompt=")) {
167
+ once("--prompt");
168
+ const { value: v, adv } = flagVal(argv, i, "--prompt");
169
+ prompt = v;
170
+ i += adv;
171
+ }
172
+ else if (a === "--dotenv" || a.startsWith("--dotenv=")) {
173
+ once("--dotenv");
174
+ const { value: v, adv } = flagVal(argv, i, "--dotenv");
175
+ dotenv = v;
176
+ i += adv;
177
+ }
178
+ else if (a === "--fidelity" || a.startsWith("--fidelity=")) {
179
+ once("--fidelity");
180
+ const { value: v, adv } = flagVal(argv, i, "--fidelity");
181
+ fidelity = v;
182
+ i += adv;
183
+ }
184
+ else if (a === "--evaluator-model" || a.startsWith("--evaluator-model=")) {
185
+ once("--evaluator-model");
186
+ const { value: v, adv } = flagVal(argv, i, "--evaluator-model");
187
+ evaluatorModel = v;
188
+ i += adv;
189
+ }
190
+ else if (a === "--output-format" || a.startsWith("--output-format=")) {
191
+ once("--output-format");
192
+ const { value: v, adv } = flagVal(argv, i, "--output-format");
193
+ outputFormat = v;
194
+ i += adv;
195
+ }
196
+ else if (a === "--prompt-file" || a.startsWith("--prompt-file=")) {
197
+ once("--prompt-file");
198
+ const { value: v, adv } = flagVal(argv, i, "--prompt-file");
199
+ promptFile = v;
200
+ i += adv;
201
+ }
202
+ else if (a === "--keep" || a.startsWith("--keep=")) {
203
+ // Match the equals form too so it errors as "takes no value" rather than falling through to the
204
+ // owned-flag branch's "unknown flag: --keep=x", which misdescribes the mistake.
205
+ if (a.includes("="))
206
+ throw new Error(`--keep takes no value (got "${a}")\n${usage()}`);
207
+ // accepted no-op: critique always keeps its runs, so the flag's promise already holds. Erroring on
208
+ // an already-satisfied request is hostile; silently ignoring an UNsatisfied one is this repo's
209
+ // anti-pattern — this is the former.
210
+ }
211
+ else if (a.startsWith("-")) {
212
+ // Not critique-owned: consult THE shared spec rather than a hand-mirrored list here. A skill flag
213
+ // with no disposition is impossible — the parity test makes that red CI.
214
+ const name = a.includes("=") ? a.slice(0, a.indexOf("=")) : a;
215
+ const spec = lookupSkillFlag(name);
216
+ if (!spec)
217
+ throw new Error(`unknown flag: ${a}\n${usage()}`);
218
+ // `repeatable` is enforced, not decorative — see `once()`.
219
+ if (!spec.repeatable)
220
+ once(name, spec.arity);
221
+ if (spec.critique.kind === "reject")
222
+ throw new Error(`${name} is not accepted by critique: ${spec.critique.reason}\n${usage()}`);
223
+ if (spec.critique.kind === "owned")
224
+ throw new Error(`unknown flag: ${a}\n${usage()}`); // owned => handled above
225
+ let value;
226
+ let eq = false;
227
+ if (spec.arity === 1) {
228
+ const { value: v, adv, equalsForm } = flagVal(argv, i, name);
229
+ eq = equalsForm;
230
+ value = v;
231
+ i += adv;
232
+ }
233
+ else if (a.includes("=")) {
234
+ // The child rejects `--boolean=x` outright ("takes no value"). Accepting it here and forwarding a
235
+ // BARE flag would silently invert intent — `--allow-missing-capability=false` would enable it.
236
+ throw new Error(`${name} takes no value (got "${a}")\n${usage()}`);
237
+ }
238
+ // --on-unanswered prompt would resolve differently than the caller expects: there is no TTY inside
239
+ // the spawn, so it cannot actually prompt anyone.
240
+ if (name === "--on-unanswered" && value !== "fail" && value !== "first")
241
+ throw new Error(`--on-unanswered must be "fail" or "first" for critique (got "${value}") — there is no TTY inside the spawned turn\n${usage()}`);
242
+ if (name === "--timeout") {
243
+ const n = Number(value);
244
+ if (!Number.isInteger(n) || n <= 0)
245
+ throw new Error(`--timeout requires a positive integer (ms), got "${value}"`);
246
+ taskTimeoutMs = n;
247
+ }
248
+ // Preserve the EQUALS form when that is how it arrived: the child's spaced-form parser rejects a
249
+ // value starting with `-`, so normalising `--intent=-terse` to two argv entries would kill a valid
250
+ // input one layer later with a wrong diagnosis.
251
+ const fragment = spec.arity === 1 ? (eq ? [`${name}=${value}`] : [name, value]) : [name];
252
+ // EXCLUSIVE buckets. `buildTaskTurnArgs` spreads BOTH arrays, so a "both" flag pushed into both
253
+ // would be emitted TWICE on the task turn — which is not a no-op for repeatable flags like
254
+ // --upload (it mounts the file twice). Sources go to forwardBoth ONLY; the task builder picks
255
+ // them up from there.
256
+ if (spec.critique.turns === "both")
257
+ forwardBoth.push(...fragment);
258
+ else
259
+ forwardTask.push(...fragment);
260
+ }
103
261
  else
104
262
  positional.push(a);
105
263
  }
106
264
  if (positional.length !== 1)
107
265
  throw new Error(usage());
266
+ if (prompt !== undefined && promptFile !== undefined)
267
+ throw new Error(`--prompt and --prompt-file are mutually exclusive\n${usage()}`);
268
+ if (promptFile !== undefined) {
269
+ if (!existsSync(promptFile))
270
+ throw new Error(`--prompt-file not found: ${promptFile}`);
271
+ prompt = readFileSync(promptFile, "utf8");
272
+ }
108
273
  if (!prompt || !prompt.trim())
109
- throw new Error(`--prompt "<probe>" is required\n${usage()}`);
274
+ throw new Error(`--prompt "<probe>" or --prompt-file <path> is required\n${usage()}`);
110
275
  if (fidelity !== "container")
111
276
  throw new Error(`skill-critique is container-tier only (the resume-continuity behavior it relies on is only verified there); got --fidelity ${fidelity}`);
112
277
  if (outputFormat !== "json" && outputFormat !== "text")
113
278
  throw new Error(`--output-format must be "text" or "json" (got "${outputFormat}")`);
114
- return { skillFolder: positional[0], prompt, dotenv, fidelity, evaluatorModel, outputFormat };
279
+ // Fail fast with critique's OWN clear error (mirroring cli.ts's global --dotenv existence check,
280
+ // ~line 627) rather than letting an absent file surface later as a generic instrument-failure
281
+ // diagnostic from the child `skill` invocation's own (differently-worded) rejection.
282
+ if (dotenv !== undefined && !existsSync(dotenv))
283
+ throw new Error(`--dotenv file not found: ${dotenv}\n${usage()}`);
284
+ return { skillFolder: positional[0], prompt, dotenv, fidelity, evaluatorModel, outputFormat, forwardBoth, forwardTask, taskTimeoutMs };
115
285
  }
116
286
  // F36: a hung reflection (stuck container, network stall, a `claude` process that never returns) must not
117
287
  // block the whole discovery command forever, and a spewing/looping child must not grow the buffer
@@ -472,6 +642,53 @@ export function buildJsonReport(state) {
472
642
  * invoked, unexpected throw). Distinct from FINDINGS, which never gate. Matches SPEC's exit 2
473
643
  * (usage/runtime) rather than inventing a new code. */
474
644
  const EXIT_INSTRUMENT_FAILURE = 2;
645
+ /** Argv for the GRADED turn. Forwarded fragments come BEFORE critique's pinned flags so a pinned value
646
+ * always wins (value flags are last-wins in the skill lane's parser). Exported for unit tests — the
647
+ * forwarding invariants are the kind that fail every run when wrong, so they are worth testing without
648
+ * paying for a spawn. */
649
+ export function buildTaskTurnArgs(opts, sessionId) {
650
+ const dotenvArgs = opts.dotenv ? ["--dotenv", opts.dotenv] : [];
651
+ return [
652
+ ...dotenvArgs,
653
+ "skill",
654
+ opts.skillFolder,
655
+ opts.prompt,
656
+ ...opts.forwardBoth,
657
+ ...opts.forwardTask,
658
+ "--fidelity",
659
+ "container",
660
+ "--session-id",
661
+ sessionId,
662
+ "--keep",
663
+ "--output-format",
664
+ "json",
665
+ ];
666
+ }
667
+ /** Argv for the REFLECTION turn — a resume of the same session.
668
+ *
669
+ * Only `forwardBoth` is replayed here, and it MUST be: session sources are part of the origin key, so a
670
+ * reflection turn that omits them computes a different identity and the resume throws fail-closed.
671
+ * `forwardTask` is deliberately absent — `--decider-dir` in particular requires a fresh empty dir per run
672
+ * and would break on turn 2, and gates belong to the graded run, not to critique's own protocol turn. */
673
+ export function buildReflectionTurnArgs(opts, sessionId) {
674
+ const dotenvArgs = opts.dotenv ? ["--dotenv", opts.dotenv] : [];
675
+ return [
676
+ ...dotenvArgs,
677
+ "skill",
678
+ opts.skillFolder,
679
+ REFLECTION_PROMPT,
680
+ ...opts.forwardBoth,
681
+ "--session-id",
682
+ sessionId,
683
+ "--resume",
684
+ "--fidelity",
685
+ "container",
686
+ "--on-unanswered",
687
+ "first",
688
+ "--output-format",
689
+ "json",
690
+ ];
691
+ }
475
692
  async function main(argv = process.argv.slice(2)) {
476
693
  if (argv.includes("--help") || argv.includes("-h")) {
477
694
  writeSync(1, usage() + "\n");
@@ -488,23 +705,13 @@ async function main(argv = process.argv.slice(2)) {
488
705
  process.exit(2);
489
706
  return;
490
707
  }
491
- const dotenvArgs = opts.dotenv ? ["--dotenv", opts.dotenv] : [];
492
708
  const sessionId = `crit-${randomUUID()}`;
493
709
  try {
494
710
  // 1. Task turn.
495
- const task = await runSkillTurn([
496
- ...dotenvArgs,
497
- "skill",
498
- opts.skillFolder,
499
- opts.prompt,
500
- "--fidelity",
501
- "container",
502
- "--session-id",
503
- sessionId,
504
- "--keep",
505
- "--output-format",
506
- "json",
507
- ]);
711
+ // Stretch critique's own kill-switch past a forwarded --timeout: otherwise a longer budget would be
712
+ // killed by the INSTRUMENT and misreported as an infra failure rather than a gradeable timeout. The
713
+ // +60s covers staging and container start — not principled, and a cold image pull can exceed it.
714
+ const task = await runSkillTurn(buildTaskTurnArgs(opts, sessionId), opts.taskTimeoutMs ? Math.max(TURN_TIMEOUT_MS, opts.taskTimeoutMs + 60_000) : TURN_TIMEOUT_MS);
508
715
  const outDir = extractOutDir(task);
509
716
  if (!outDir) {
510
717
  // F36: surface WHY there's no envelope/status line when it was the bounded spawn itself that gave up.
@@ -552,21 +759,7 @@ async function main(argv = process.argv.slice(2)) {
552
759
  // 2. Snapshot the turn-1/turn-2 boundary BEFORE the reflection turn touches anything.
553
760
  const boundary = snapshotTurnBoundary(outDir);
554
761
  // 3. Reflection turn: resume the SAME session.
555
- const reflect = await runSkillTurn([
556
- ...dotenvArgs,
557
- "skill",
558
- opts.skillFolder,
559
- REFLECTION_PROMPT,
560
- "--session-id",
561
- sessionId,
562
- "--resume",
563
- "--fidelity",
564
- "container",
565
- "--on-unanswered",
566
- "first",
567
- "--output-format",
568
- "json",
569
- ]);
762
+ const reflect = await runSkillTurn(buildReflectionTurnArgs(opts, sessionId));
570
763
  // F37: validate the reflection turn at the PROTOCOL level — exit code, envelope shape, and
571
764
  // session/turn continuity (turn>1 AND outDir/sessionId match the task turn's) — BEFORE trusting its
572
765
  // `finalMessage` as a self-report or handing anything to the evaluator. A failed reflection (crash, bad
@@ -1,6 +1,7 @@
1
- import { readFileSync, readdirSync, existsSync } from "node:fs";
1
+ import { readFileSync, readdirSync, existsSync, statSync } from "node:fs";
2
2
  import { join } from "node:path";
3
3
  import { readTurn1ResultWithStatus, readTurn1Slice, verifyBoundaryIntegrity } from "./evidence.js";
4
+ import { loadVmPathContext } from "../run/vm-path-ctx-file.js";
4
5
  // Assembles the TURN-1-ONLY evidence document a tool-less, one-shot evaluator model is graded against.
5
6
  // This packager is a from-scratch, load-bearing build (this instrument's design): a tool-less
6
7
  // evaluator can't grep the logs itself, so its recall is bounded by exactly what this function decides to
@@ -50,6 +51,48 @@ function readTurn1Transcript(runDir, boundary) {
50
51
  return { text: "", degraded: true };
51
52
  }
52
53
  }
54
+ /** List of what was ATTACHED to this run — upload filenames + byte sizes, and connected-folder mount names.
55
+ * NEVER file content: the evaluator needs to be able to tell "the agent said there was no file, and
56
+ * correctly so" apart from "the agent confabulated that", but packaging the bytes themselves would blow
57
+ * the package's byte budget and widen the prompt-injection surface the armor exists to contain (see this
58
+ * module's header and `armor.ts`'s header) — names and sizes are enough to answer "was anything attached."
59
+ *
60
+ * Source of truth, with fallbacks, in order:
61
+ * 1. `loadVmPathContext(runDir)` (`run/vm-path-ctx-file.ts`) — the recorded `uploadsHostDir` and
62
+ * `folders` map for THIS run. Already never throws (absent/corrupt `mounts.json` -> `null`).
63
+ * 2. The fixed container layout `<runDir>/work/session/mnt/uploads` (mirrors the derivation in
64
+ * `run/display-translate.ts`'s `vmPathContextFromPlan`) — covers runs where `mounts.json` wasn't
65
+ * written or couldn't be parsed.
66
+ * 3. `(none)` — via `sec()`'s empty-body fallback below. Every filesystem read here is try/catch-guarded
67
+ * so a missing/unreadable uploads dir degrades gracefully rather than throwing and sinking packaging. */
68
+ function listAttachedInputs(runDir) {
69
+ const loaded = loadVmPathContext(runDir);
70
+ const uploadsDir = loaded?.ctx.uploadsHostDir ?? join(runDir, "work", "session", "mnt", "uploads");
71
+ const folderNames = loaded ? Array.from(loaded.ctx.folders.keys()).sort() : [];
72
+ const lines = [];
73
+ try {
74
+ const uploadNames = readdirSync(uploadsDir, { withFileTypes: true })
75
+ .filter((e) => e.isFile())
76
+ .map((e) => e.name)
77
+ .sort();
78
+ for (const name of uploadNames) {
79
+ let sizeNote;
80
+ try {
81
+ sizeNote = `${statSync(join(uploadsDir, name)).size} bytes`;
82
+ }
83
+ catch {
84
+ sizeNote = "size unknown";
85
+ }
86
+ lines.push(`${name} (${sizeNote})`);
87
+ }
88
+ }
89
+ catch {
90
+ /* uploads dir absent/unreadable — no uploads to list, not an error */
91
+ }
92
+ for (const name of folderNames)
93
+ lines.push(`${name} (connected folder)`);
94
+ return lines.join("\n");
95
+ }
53
96
  /** Byte-bound a text section, appending a loud (never silent) truncation marker so the evaluator knows the
54
97
  * section was cut rather than reading a suspiciously short document as "that's everything." Cuts on a
55
98
  * UTF-8-safe boundary (Buffer, not string length, so a truncated multi-byte char can't corrupt the tail). */
@@ -91,6 +134,7 @@ const REFERENCES_READ_CAP = 1 * 1024;
91
134
  const SKILL_MD_CAP = 16 * 1024;
92
135
  const REFERENCE_LIST_CAP = 1 * 1024;
93
136
  const TRANSCRIPT_CAP = 16 * 1024;
137
+ const ATTACHED_INPUTS_CAP = 1 * 1024;
94
138
  /** The overall package hard cap — the whole assembled document is trimmed to this even if every
95
139
  * per-section budget above was individually respected (their sum is deliberately a bit under this, but a
96
140
  * belt-and-suspenders final trim means a future per-section budget change can never silently blow the
@@ -200,6 +244,7 @@ export function packageEvidence(runDir, boundary, skillDir, isResume = false) {
200
244
  turn1ResultDegradedNote, bound(referencesRead.length ? referencesRead.join("\n") : "(none)", REFERENCES_READ_CAP)),
201
245
  sec(skillMdSectionTitle, bound(skillMdSectionBody, SKILL_MD_CAP)),
202
246
  sec("references/ available (filenames only, NOT content — presence, not coverage)", bound(referenceFiles.length ? referenceFiles.join("\n") : "(none)", REFERENCE_LIST_CAP)),
247
+ sec("Attached inputs (mnt/uploads filenames + sizes, and connected-folder mount names — NOT content)", bound(listAttachedInputs(runDir), ATTACHED_INPUTS_CAP)),
203
248
  sec("Transcript (turn 1 only — the reflection turn's own reads/output are excluded by construction)" +
204
249
  (turn1SliceDegraded
205
250
  ? " [DEGRADED: the turn-1/turn-2 boundary for this fallback slice could not be verified — treat gaps as unknown, not as evidence of absence]"
@@ -0,0 +1,117 @@
1
+ // THE source of truth for `cowork-harness skill`'s flag surface, and for what `critique` does with each
2
+ // flag.
3
+ //
4
+ // WHY THIS EXISTS. `critique` runs a skill by spawning two `skill` turns (a task turn, then a `--resume`
5
+ // reflection turn), but its own parser was a hand-rolled SUBSET of `skill`'s flags. That subset drifts
6
+ // silently — a `skill` flag added tomorrow is simply unavailable to `critique`, with nothing failing. This
7
+ // repo has shipped that exact "propagate-minus-one" shape repeatedly, so a design that requires manually
8
+ // mirroring a flag list is wrong even when it is momentarily correct.
9
+ //
10
+ // The fix is structural, in two halves:
11
+ // 1. `critique` field below is REQUIRED, so an entry cannot be added without deciding its disposition.
12
+ // 2. `test/critique-skill-flag-parity.test.ts` extracts the flag literals the `skill` lane actually
13
+ // accepts and fails if any is missing here — so a NEW skill flag is red CI, not a silent gap.
14
+ //
15
+ // Half 2 is the load-bearing half: the required field only gives totality over entries already present.
16
+ const forwardBoth = () => ({ kind: "forward", turns: "both" });
17
+ const forwardTask = () => ({ kind: "forward", turns: "task" });
18
+ export const SKILL_FLAG_SURFACE = [
19
+ // ---- session SOURCES — must reach BOTH turns or the resume throws on an origin-key mismatch ----
20
+ { flag: "--upload", arity: 1, repeatable: true, critique: forwardBoth() },
21
+ { flag: "--folder", arity: 1, repeatable: true, critique: forwardBoth() },
22
+ { flag: "--plugin", arity: 1, repeatable: true, critique: forwardBoth() },
23
+ { flag: "--marketplace", arity: 1, repeatable: true, critique: forwardBoth() },
24
+ { flag: "--enable", arity: 1, repeatable: true, critique: forwardBoth() },
25
+ // ---- session shape that both turns must agree on ----
26
+ // The reflection must run on the SAME model as the task turn: the skill lane rebuilds its inline session
27
+ // per invocation, so an unforwarded resume would reflect using the DEFAULT model — a self-report from a
28
+ // different model than the one that did the work.
29
+ { flag: "--model", arity: 1, critique: forwardBoth() },
30
+ // Without this a lean-image capability gap makes the task turn error, and the critique harvests a fact
31
+ // about the rig instead of the skill — the same failure class as a missing upload.
32
+ { flag: "--allow-missing-capability", arity: 0, critique: forwardBoth() },
33
+ // ---- the GRADED run only ----
34
+ // TASK-only deliberately: forwarding to the reflection turn would inject a near-always-green row into the
35
+ // labelled generation group, inflating passRate and breaking the `runLabel` filter docs/debugging.md
36
+ // documents as the escape hatch for exactly that dilution. Not in the origin key, so nothing forces both.
37
+ { flag: "--label", arity: 1, critique: forwardTask() },
38
+ // Parsed as well as forwarded: critique's own spawn kill-switch is fixed, so a longer --timeout would be
39
+ // killed by the INSTRUMENT and misreported as an infra failure instead of a gradeable timeout.
40
+ { flag: "--timeout", arity: 1, critique: forwardTask() },
41
+ // Makes GATED skills critiquable for the first time: the inner spawn has no TTY, so an unscripted gate
42
+ // otherwise resolves to `fail` and the task turn dies before anything can be graded.
43
+ { flag: "--answer", arity: 1, repeatable: true, critique: forwardTask() },
44
+ { flag: "--answer-policy", arity: 1, critique: forwardTask() },
45
+ // `prompt` is rejected at parse time — no TTY inside the spawn, so it would resolve differently than the
46
+ // caller expects. Gates belong to the graded run; the reflection turn stays pinned deterministic.
47
+ { flag: "--on-unanswered", arity: 1, critique: forwardTask() },
48
+ { flag: "--decider-llm", arity: 0, critique: forwardTask() },
49
+ { flag: "--intent", arity: 1, critique: forwardTask() },
50
+ { flag: "--decider-model", arity: 1, critique: forwardTask() },
51
+ { flag: "--decider-cmd", arity: 1, critique: forwardTask() },
52
+ // HARD invariant: the file channel requires a fresh, empty dir per run. Reaching the reflection turn
53
+ // would break it.
54
+ { flag: "--decider-dir", arity: 1, critique: forwardTask() },
55
+ // ---- refused ----
56
+ {
57
+ flag: "--session-id",
58
+ arity: 1,
59
+ critique: {
60
+ kind: "reject",
61
+ reason: "critique mints and manages its own session (the reflection turn IS a resume of it)",
62
+ },
63
+ },
64
+ {
65
+ flag: "--resume",
66
+ arity: 0,
67
+ critique: {
68
+ kind: "reject",
69
+ reason: "critique mints and manages its own session (the reflection turn IS a resume of it)",
70
+ },
71
+ },
72
+ ...["--repeat", "--min-pass-rate", "--stop-on-diverge", "--max-budget-usd", "--allow-budget-stop"].map((flag) => ({
73
+ flag,
74
+ arity: flag === "--stop-on-diverge" || flag === "--allow-budget-stop" ? 0 : 1,
75
+ critique: {
76
+ kind: "reject",
77
+ reason: "critique is a fixed two-turn protocol — loop `critique` itself and pair generations by fingerprint.skillHash",
78
+ },
79
+ })),
80
+ {
81
+ flag: "--ablate-skill",
82
+ arity: 0,
83
+ critique: { kind: "reject", reason: "grading a skill you removed is incoherent (and ablate+resume is rejected anyway)" },
84
+ },
85
+ ...["--quiet", "-q", "--verbose", "--compact", "--demo"].map((flag) => ({
86
+ flag,
87
+ arity: 0,
88
+ critique: {
89
+ kind: "reject",
90
+ reason: "critique produces its own report; inner-turn rendering flags have no effect on it",
91
+ },
92
+ })),
93
+ {
94
+ flag: "--dry-run",
95
+ arity: 0,
96
+ critique: { kind: "reject", reason: "there is no meaningful two-turn preview — use `skill --dry-run` directly" },
97
+ },
98
+ // ---- critique implements or pins these itself ----
99
+ { flag: "--prompt-file", arity: 1, critique: { kind: "owned", note: "critique's probe prompt, read from a file" } },
100
+ { flag: "--fidelity", arity: 1, critique: { kind: "owned", note: "pinned container — resume continuity is verified only there" } },
101
+ {
102
+ flag: "--output-format",
103
+ arity: 1,
104
+ critique: { kind: "owned", note: "critique's REPORT format; the inner turns always speak json internally" },
105
+ },
106
+ { flag: "--keep", arity: 0, critique: { kind: "owned", note: "accepted as a no-op — critique always keeps its runs" } },
107
+ ];
108
+ /** Flags `critique` defines that the `skill` lane does not have. Kept separate so the parity guard can
109
+ * treat them as legitimately-unknown-to-skill rather than drift. */
110
+ // `--prompt` is here, not in the surface above: the `skill` lane takes its prompt POSITIONALLY
111
+ // (`skill <folder> "<prompt>"`), so there is no `--prompt` flag to have a disposition about. The parity
112
+ // guard caught this modelling error on its first run.
113
+ export const CRITIQUE_ONLY_FLAGS = ["--evaluator-model", "--dotenv", "--prompt"];
114
+ const BY_FLAG = new Map(SKILL_FLAG_SURFACE.map((s) => [s.flag, s]));
115
+ export function lookupSkillFlag(name) {
116
+ return BY_FLAG.get(name);
117
+ }
package/docs/critique.md CHANGED
@@ -63,6 +63,83 @@ evidence going into it trustworthy.
63
63
  **One naming collision to know about:** `hostloop` in this repo is a **fidelity tier** — where the agent
64
64
  process runs. It has nothing to do with loop engineering's "loops".
65
65
 
66
+ ## Flags
67
+
68
+ `critique` accepts the `skill` flags that make sense for a graded run, under the **same names** — what you
69
+ know from `skill` transfers. Anything that cannot work is refused with a reason rather than silently
70
+ ignored.
71
+
72
+ **Probe** (one required)
73
+
74
+ | Flag | |
75
+ |---|---|
76
+ | `--prompt "<text>"` | the task to run the skill against |
77
+ | `--prompt-file <path>` | read the probe verbatim from a file — for probes containing quotes, `$`, or newlines |
78
+
79
+ **Files and sources** — forwarded to the graded run. **Required for "analyze this document" skills.**
80
+
81
+ | Flag | |
82
+ |---|---|
83
+ | `--upload <path>` | mount a file at `mnt/uploads/<name>` (repeatable) |
84
+ | `--folder <dir>` | connect a folder at `mnt/<name>` (repeatable) |
85
+ | `--plugin <dir>`, `--marketplace <dir>` + `--enable <name@mkt>` | extra skill sources |
86
+
87
+ **Session shape** — both turns must agree on these, so they reach the reflection turn too.
88
+
89
+ | Flag | |
90
+ |---|---|
91
+ | `--model <id>` | session model for the agent doing the work *and* reflecting |
92
+ | `--allow-missing-capability` | don't fail either turn when the lean image omits a capability |
93
+
94
+ **Graded-run tuning** — the task turn only; the reflection turn stays pinned deterministic.
95
+
96
+ | Flag | |
97
+ |---|---|
98
+ | `--timeout <ms>` | wall-clock budget for the task turn (critique's own kill-switch stretches to fit) |
99
+ | `--label <tag>` | generation tag in the run index, for pairing critiques across fixes |
100
+ | `--answer "<q-regex>=<choice>"`, `--answer-policy <yaml>` | pre-answer the skill's gates — **this is what makes gated skills critiquable at all** |
101
+ | `--on-unanswered fail\|first` | unscripted-gate policy (`prompt` is refused — there is no TTY inside) |
102
+ | `--decider-llm` / `--intent` / `--decider-model` / `--decider-cmd` / `--decider-dir` | answer live gates in the graded run |
103
+
104
+ **Critique's own**
105
+
106
+ | Flag | |
107
+ |---|---|
108
+ | `--evaluator-model <id>` | the grading model (env: `COWORK_HARNESS_EVALUATOR_MODEL`) |
109
+ | `--output-format json\|text` | critique's *report* format — the inner turns always speak JSON internally |
110
+ | `--fidelity container` | container tier only |
111
+ | `--keep` | accepted as a no-op; runs are always kept |
112
+ | `--dotenv <path>` | credentials — works **before** `critique` (the global form) or **after** it |
113
+ | `--run-dir <path>` | **global, unlike `--dotenv`** — must still PRECEDE the subcommand; a trailing `critique … --run-dir` is rejected |
114
+
115
+ **Refused, and why**
116
+
117
+ | Flag | Reason |
118
+ |---|---|
119
+ | `--session-id` / `--resume` | critique mints and manages its own session — the reflection turn *is* a resume of it |
120
+ | `--repeat` + companions | fixed two-turn protocol; loop `critique` itself and pair by `fingerprint.skillHash` |
121
+ | `--ablate-skill` | grading a skill you removed is incoherent |
122
+ | `--quiet`/`-q` / `--verbose` / `--compact` / `--demo` / `--dry-run` | inner-turn rendering or preview — no effect on the report |
123
+
124
+ **Repeating a flag.** `--upload`, `--folder`, `--plugin`, `--marketplace`, `--enable` and `--answer` accumulate,
125
+ so repeating them is how you pass several. Every other value-taking flag is single-valued and repeating it is
126
+ a **usage error** (exit `2`) rather than a silent last-wins — `--prompt a --prompt b` would otherwise discard
127
+ a probe you typed. Boolean flags may be repeated harmlessly.
128
+
129
+ ### Skills that need an attached file
130
+
131
+ ```bash
132
+ cowork-harness critique ./captable-skill \
133
+ --prompt "Analyze this cap table and flag anything unusual" \
134
+ --upload ./acme-captable.xlsx
135
+ ```
136
+
137
+ Both internal turns receive the source flags — they have to, or the reflection turn's resume computes a
138
+ different session identity and fails. The evidence package records **which files were attached** (names and
139
+ sizes), so the evaluator can tell "the agent said there was no file, and correctly so" from a confabulation.
140
+
141
+ It does **not** record their contents — see Known limitations.
142
+
66
143
  ## Cost and prerequisites
67
144
 
68
145
  - **Four model workloads per critique**: two container runs (task + reflection) and two evaluator passes.
@@ -109,6 +186,13 @@ Changing the evaluator model invalidates that verification.
109
186
  - **SKILL.md is capped at 16KB**; a larger one degrades toward "not adjudicable".
110
187
  - **English-only prompts.**
111
188
  - The evidence package is not persisted; the report is written to stdout.
189
+ - **Attached-file content usually stays out of the evidence — but that is the common case, not a
190
+ guarantee.** "Attached inputs" lists names and sizes only, never bytes, and the primary transcript
191
+ source is assistant prose. But packaging falls back to a raw slice of `events.jsonl` when the archived
192
+ transcript is missing, and that stream carries full tool results — so if the agent read the attached
193
+ file, its content can enter the Transcript section (bounded, still armor-fenced) and a content-level
194
+ citation would resolve. Claims about a document's *contents* are therefore usually NOT ADJUDICABLE, not
195
+ always.
112
196
  - **Citation seams.** Armor inserts a marker line between each section heading and its body. A quote that
113
197
  spans that seam *without* including the marker does not resolve and is DROPPED. Quotes wholly inside one
114
198
  section are unaffected. **Measured:** on a benign package, 9 findings across 5 live pass-1 runs produced
package/docs/debugging.md CHANGED
@@ -135,6 +135,9 @@ built on top of the harness before any of it shipped — the commands now replac
135
135
 
136
136
  ```bash
137
137
  SKILL=./my-skill
138
+ # Skill needs an attached input (cap table, deck, transcript)? Add `--upload <path>` / `--folder <dir>`
139
+ # to BOTH step 1 and step 3 — critique forwards them to its own two turns, and step 3 must reproduce
140
+ # with the same inputs the harvest used. See critique.md → "Skills that need an attached file".
138
141
 
139
142
  # 1. HARVEST — run the skill against a real input and grade what confused the agent.
140
143
  # `critique` runs the task, asks the agent what was unclear, then verifies every claim against a
@@ -16,7 +16,7 @@ DOES exercise a real gate exchange, see `example-multiselect-gate.cassette.json`
16
16
 
17
17
  Run it with:
18
18
 
19
- > Assumes the `cowork-harness` CLI is available — from a source checkout run `npm ci && npm run build && npm link` first, or `npm i -g "cowork-harness@>=1.5.0"`. (`replay` itself needs nothing else — no token, no Docker.)
19
+ > Assumes the `cowork-harness` CLI is available — from a source checkout run `npm ci && npm run build && npm link` first, or `npm i -g "cowork-harness@>=1.6.0"`. (`replay` itself needs nothing else — no token, no Docker.)
20
20
 
21
21
  ```sh
22
22
  cowork-harness replay examples/replays/example-pdf-skill.cassette.json
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "cowork-harness",
3
- "version": "1.5.0",
3
+ "version": "1.6.0",
4
4
  "description": "Scriptable, CI-friendly harness for Claude Cowork's runtime contract for testing skills across scenarios \u2014 same agent, mounts, egress allowlist, permission protocol, and sandbox limitations.",
5
5
  "license": "MIT",
6
6
  "type": "module",