cowork-harness 1.5.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +20 -8
- package/.claude/skills/cowork-harness/references/ci-recipe.md +5 -5
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +2 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +1 -1
- package/.claude/skills/cowork-harness/references/task-recipes.md +5 -1
- package/CHANGELOG.md +42 -0
- package/README.md +7 -7
- package/dist/cli.js +5 -1
- package/dist/critique/command.js +244 -51
- package/dist/critique/package-evidence.js +46 -1
- package/dist/run/skill-flag-surface.js +117 -0
- package/docs/critique.md +84 -0
- package/docs/debugging.md +3 -0
- package/examples/replays/README.md +1 -1
- package/package.json +1 -1
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: cowork-harness
|
|
3
|
-
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
3
|
+
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 1.
|
|
7
|
-
tracks-harness: cowork-harness 1.
|
|
6
|
+
version: 1.6.0
|
|
7
|
+
tracks-harness: cowork-harness 1.6.0 (baseline desktop-1.22209.3)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.6.0` (baseline
|
|
26
26
|
> `desktop-1.22209.3`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
27
27
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
@@ -39,9 +39,21 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
41
|
- **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
|
|
42
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.
|
|
43
|
-
|
|
44
|
-
What the ≥ 1.
|
|
42
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.6.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.6.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=1.6.0"`. **Pin `@>=1.6.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
|
|
43
|
+
|
|
44
|
+
What the ≥ 1.6.0 floor gates, by release:
|
|
45
|
+
|
|
46
|
+
- **1.6.0 (`critique`'s skill-flag parity):** `critique` accepts most `skill` flags under the same names,
|
|
47
|
+
which is what makes a document-analysis skill (cap table, deck, financial model, transcript)
|
|
48
|
+
critiquable at all: `--upload`/`--folder`/`--plugin`/`--marketplace`/`--enable`/`--model`/
|
|
49
|
+
`--allow-missing-capability` reach BOTH spawned turns; `--label`/`--timeout`/`--answer`/
|
|
50
|
+
`--answer-policy`/`--on-unanswered`/the decider flags reach the graded turn only; `--prompt-file` is
|
|
51
|
+
new. Anything that can't work is REFUSED with a reason rather than silently ignored.
|
|
52
|
+
**Repeating a flag:** `--upload`/`--folder`/`--plugin`/`--marketplace`/`--enable`/`--answer`
|
|
53
|
+
accumulate; every other value-taking flag is single-valued and repeating it is a usage error (exit `2`)
|
|
54
|
+
rather than a silent last-wins — do not assume the CLI-conventional "last one wins". Booleans repeat
|
|
55
|
+
harmlessly. On `1.5.0` these flags do not exist: `critique --upload x.xlsx` fails as "unknown flag", so
|
|
56
|
+
if a user hits that, they are below the floor.
|
|
45
57
|
|
|
46
58
|
- **1.5.0:** `critique <skill-folder> --prompt "<probe>"` (EXPERIMENTAL) — runs a skill, asks the agent
|
|
47
59
|
what confused it, then grades that self-report against a frozen record of the run: a blinded evaluator
|
|
@@ -69,7 +81,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
69
81
|
- **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
|
|
70
82
|
- **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
|
|
71
83
|
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred), or `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
72
|
-
- **`--dotenv` is a GLOBAL flag — put it BEFORE the subcommand.** `cowork-harness --dotenv .env record …`, never `cowork-harness record … --dotenv .env`. Every *other* flag is subcommand-level, so muscle memory fights this one; the harness rejects the misplaced form with an exact-fix error, but placing it first avoids the round-trip.
|
|
84
|
+
- **`--dotenv` is a GLOBAL flag — put it BEFORE the subcommand.** `cowork-harness --dotenv .env record …`, never `cowork-harness record … --dotenv .env`. Every *other* flag is subcommand-level, so muscle memory fights this one; the harness rejects the misplaced form with an exact-fix error, but placing it first avoids the round-trip. **One exception: `critique` also accepts `--dotenv` per-command** (`critique <folder> --prompt "…" --dotenv <path>`) — UNRELEASED, see the floor list; `--run-dir` stays global-only everywhere.
|
|
73
85
|
|
|
74
86
|
## Orient — the three loops
|
|
75
87
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 1.6.0` (baseline `desktop-1.20186.1`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
|
|
|
13
13
|
```
|
|
14
14
|
|
|
15
15
|
The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
|
|
16
|
-
release; pin an exact version (e.g. `version: "1.
|
|
16
|
+
release; pin an exact version (e.g. `version: "1.6.0"`) for reproducible CI.
|
|
17
17
|
|
|
18
18
|
Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
|
|
19
19
|
cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
|
|
@@ -57,7 +57,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
57
57
|
GitHub-hosted runners, no token/Docker/agent:
|
|
58
58
|
|
|
59
59
|
```yaml
|
|
60
|
-
- run: npm i -g "cowork-harness@>=1.
|
|
60
|
+
- run: npm i -g "cowork-harness@>=1.6.0"
|
|
61
61
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
62
62
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
63
63
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -197,7 +197,7 @@ jobs:
|
|
|
197
197
|
with: { node-version: '20' }
|
|
198
198
|
- uses: actions/setup-python@v5
|
|
199
199
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
200
|
-
- run: npm i -g "cowork-harness@>=1.
|
|
200
|
+
- run: npm i -g "cowork-harness@>=1.6.0"
|
|
201
201
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
202
202
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
203
203
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -226,7 +226,7 @@ jobs:
|
|
|
226
226
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
227
227
|
fi
|
|
228
228
|
- if: steps.guard.outputs.live == 'true'
|
|
229
|
-
run: npm i -g "cowork-harness@>=1.
|
|
229
|
+
run: npm i -g "cowork-harness@>=1.6.0"
|
|
230
230
|
- if: steps.guard.outputs.live == 'true'
|
|
231
231
|
run: cowork-harness run scenarios/ --output-format json
|
|
232
232
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 1.6.0` (baseline `desktop-1.20186.1`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -122,6 +122,7 @@ hands `fn` exactly this dict.
|
|
|
122
122
|
|
|
123
123
|
- On `skill`: `fail | prompt | first`.
|
|
124
124
|
- On `run`: `fail | first` (`prompt` rejected — it would break determinism).
|
|
125
|
+
- On `critique` (UNRELEASED): `fail | first` (`prompt` rejected — there is no TTY inside the spawned turn).
|
|
125
126
|
- **`llm` is NOT an `--on-unanswered` value.** The bare flag `--on-unanswered llm` is rejected; use
|
|
126
127
|
`--decider-llm` (CLI) or `on_unanswered: llm` (YAML).
|
|
127
128
|
- `agent` is **retired** — `on_unanswered: agent` is rejected by the schema. The enum is
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.6.0`
|
|
4
4
|
(baseline `desktop-1.20186.1`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
@@ -179,7 +179,11 @@ Hardening a skill is a loop: run → read what it did → fix → run again. Two
|
|
|
179
179
|
The full loop (harvest -> reproduce -> fix -> prove freshness -> compare) is written out end-to-end in
|
|
180
180
|
docs/debugging.md under "The whole loop, end to end". The harness now SHIPS a grader — `cowork-harness critique <skill-folder> --prompt "<probe>"` runs the
|
|
181
181
|
skill, asks the agent what confused it, and grades that self-report against a frozen record of the run
|
|
182
|
-
(blinded evaluator + mechanical citation checking).
|
|
182
|
+
(blinded evaluator + mechanical citation checking). **Critiquing a document-analysis skill?** The probe
|
|
183
|
+
attaches nothing on its own — pass `--upload <path>` (repeatable) or `--folder <dir>` exactly as you
|
|
184
|
+
would to `skill`, or the graded run has no file and "there was no file attached" is the correct finding,
|
|
185
|
+
not a skill defect. Source flags reach both spawned turns automatically. UNRELEASED — see SKILL.md's
|
|
186
|
+
floor list. See docs/critique.md for the full flag table, cost and limits. If you
|
|
183
187
|
prefer to build your own grader, the substrate is still here:
|
|
184
188
|
- `result.json` → `finalMessage` (the skill's own answer/critique) + `toolResults[]` (tool outputs).
|
|
185
189
|
- `cowork-harness trace <run-dir> --output-format json` → the tool-call stream. Add `--full-results` so
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,48 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [1.6.0] — 2026-07-20
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- **`critique` now accepts the `skill` flags a graded run needs — starting with uploads.** A skill whose
|
|
14
|
+
whole job is "here is a document, analyze it" (cap table, pitch deck, financial model, transcript) could
|
|
15
|
+
not be critiqued at all: there was no way to attach the file, so the agent was asked to analyze a
|
|
16
|
+
document that was never there and then asked what confused it — you harvested a finding about the test
|
|
17
|
+
rig. `--upload`, `--folder`, `--plugin`, `--marketplace`/`--enable` (all repeatable), `--model` and
|
|
18
|
+
`--allow-missing-capability` now reach **both** spawned turns; that is forced, not stylistic, because
|
|
19
|
+
those paths are part of the session-origin key the reflection turn's `--resume` recomputes. `--label`,
|
|
20
|
+
`--timeout`, `--answer`/`--answer-policy`, `--on-unanswered` and the decider flags reach the graded turn
|
|
21
|
+
only. `--answer`/`--answer-policy` make **gated** skills critiquable for the first time — the inner spawn
|
|
22
|
+
has no TTY, so an unscripted gate previously killed the task turn before anything could be graded.
|
|
23
|
+
Anything that cannot work is refused **with its reason** rather than silently ignored:
|
|
24
|
+
`--session-id`/`--resume`, the `--repeat` family, `--ablate-skill`, and the rendering/preview flags.
|
|
25
|
+
- **`critique --prompt-file <path>`** — read the probe verbatim from a file, so a probe containing quotes,
|
|
26
|
+
`$` or newlines does not have to survive shell parsing.
|
|
27
|
+
- **"Attached inputs" evidence section** — upload filenames and sizes, plus connected-folder mount names,
|
|
28
|
+
never content. Without it the evaluator could not tell "the agent said there was no file, and correctly
|
|
29
|
+
so" from a confabulation.
|
|
30
|
+
- **One source of truth for the `skill` flag surface** (`src/run/skill-flag-surface.ts`), where each flag's
|
|
31
|
+
critique disposition is a **required** field, plus a parity guard that fails CI when a new `skill` flag
|
|
32
|
+
arrives without one. The old hand-rolled subset drifted silently — this repo's recurring bug shape.
|
|
33
|
+
|
|
34
|
+
### Changed
|
|
35
|
+
|
|
36
|
+
- **Repeating a single-valued `critique` flag is now a usage error (exit `2`) instead of silent
|
|
37
|
+
last-wins.** `--upload`, `--folder`, `--plugin`, `--marketplace`, `--enable` and `--answer` accumulate —
|
|
38
|
+
repeating them is how you pass several. Everything else is single-valued, and `--prompt a --prompt b`
|
|
39
|
+
silently discarding a probe you typed is the class of no-op this command refuses on principle. Boolean
|
|
40
|
+
flags may still be repeated harmlessly. Documented in `critique --help` and
|
|
41
|
+
[docs/critique.md](./docs/critique.md).
|
|
42
|
+
|
|
43
|
+
### Fixed
|
|
44
|
+
|
|
45
|
+
- **`critique --dotenv <path>` was documented but unreachable** (shipped that way in 1.5.0). The
|
|
46
|
+
misplaced-global guard rejected the token after any subcommand, and the `--dotenv=x` form that slipped
|
|
47
|
+
past it then hit critique's exact-match parser as "unknown flag". `critique` is now exempt from the guard
|
|
48
|
+
(`--run-dir` still has no per-command meaning and stays rejected everywhere), and a missing file fails
|
|
49
|
+
fast with critique's own error instead of surfacing later as an instrument-failure diagnostic.
|
|
50
|
+
|
|
9
51
|
## [1.5.0] — 2026-07-20
|
|
10
52
|
|
|
11
53
|
### Added
|
package/README.md
CHANGED
|
@@ -91,7 +91,7 @@ node dist/cli.js replay examples/replays/example-pdf-skill.cassette.json
|
|
|
91
91
|
|
|
92
92
|
> **Installed globally instead?** Once linked/installed, the same command is `cowork-harness replay
|
|
93
93
|
> <cassette>` — but the relative path above only resolves from a source checkout's `examples/replays/`.
|
|
94
|
-
> From a global install (`npm i -g "cowork-harness@>=1.
|
|
94
|
+
> From a global install (`npm i -g "cowork-harness@>=1.6.0"`), point at the package root instead:
|
|
95
95
|
> `cowork-harness replay "$(npm root -g)/cowork-harness/examples/replays/example-pdf-skill.cassette.json"`
|
|
96
96
|
> (or copy the cassette into your own project and pass that path).
|
|
97
97
|
|
|
@@ -101,7 +101,7 @@ Live `run`/`skill` need the prerequisites in the next section — note the `prot
|
|
|
101
101
|
> - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
|
|
102
102
|
> - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
|
|
103
103
|
> - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
|
|
104
|
-
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=1.
|
|
104
|
+
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=1.6.0"`.
|
|
105
105
|
|
|
106
106
|
Two more worked examples worth knowing about: `examples/scenarios/protocol-smoke.yaml` (zero-Docker smoke
|
|
107
107
|
test) and `examples/scenarios/skill-loads.yaml` (container-tier acceptance check) — see
|
|
@@ -126,7 +126,7 @@ claude plugin marketplace add yaniv-golan/cowork-harness
|
|
|
126
126
|
claude plugin install cowork-harness@cowork-harness
|
|
127
127
|
```
|
|
128
128
|
|
|
129
|
-
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=1.
|
|
129
|
+
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=1.6.0"` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
|
|
130
130
|
|
|
131
131
|
It also follows the open [Agent Skills](https://agentskills.io) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) via [`npx skills`](https://github.com/vercel-labs/skills) (Vercel Labs' CLI implementation of that spec):
|
|
132
132
|
|
|
@@ -147,7 +147,7 @@ A global install is enough for CI `lint`, reading the teaching skill, and replay
|
|
|
147
147
|
To `run` the worked examples live or copy them as a starting point, use a source checkout. (The marketplace
|
|
148
148
|
skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
|
|
149
149
|
assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
|
|
150
|
-
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=1.
|
|
150
|
+
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=1.6.0"` — see
|
|
151
151
|
[above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
|
|
152
152
|
|
|
153
153
|
### Prerequisites for anything above `protocol` fidelity
|
|
@@ -415,7 +415,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
|
|
|
415
415
|
| `boundary-check [baseline] [--session <file>]` | Prove the **L1 Docker** sandbox enforces Cowork's limitations (sealed FS + default-deny egress; `container`/`hostloop` share this sandbox — `microvm`'s guest firewall is not probed here); `--session` folds a session's `egress.extra_allow` into the probe allowlist | verifying the harness's own fidelity |
|
|
416
416
|
| `sync` / `list` | Derive/refresh (`sync [--diff] [--allow-empty\|--force]`) & list platform baselines from the Desktop install | after Claude Desktop updates (baselines ship, so it's optional otherwise) |
|
|
417
417
|
| `diff <a> <b>` | Compare two baselines, two runs, two cassettes, or a run+cassette — kind auto-detected by content (view/normalization flags below). Token-free, no live Desktop/Docker needed | "what changed between two runs/cassettes/baselines?" |
|
|
418
|
-
| `critique <skill-folder>` | **EXPERIMENTAL.** Run a skill, ask the agent what confused it, then grade that self-report against a frozen record of the run — blinded evaluator, mechanical citation checking. Discovery instrument, never a gate (findings always exit 0). Costs four model workloads; see [docs/critique.md](./docs/critique.md) | "what confused the agent about my skill?" |
|
|
418
|
+
| `critique <skill-folder>` | **EXPERIMENTAL.** Run a skill, ask the agent what confused it, then grade that self-report against a frozen record of the run — blinded evaluator, mechanical citation checking. Accepts the `skill` flags a graded run needs — `--upload`/`--folder`/`--plugin` reach both spawned turns, `--answer`/`--decider-*` the graded turn only; anything that can't work is refused with a reason (full table in `critique --help`). Discovery instrument, never a gate (findings always exit 0). Costs four model workloads; see [docs/critique.md](./docs/critique.md) | "what confused the agent about my skill?" — including "does it even see the attached document?" |
|
|
419
419
|
| `doctor [--tier <t>]` | Read-only prerequisite check, per tier (Docker + agent image for `container`/`hostloop`/`cowork`; **Lima** for `microvm`; plus staged agent, token, baseline); prints the exact `docker build` line if the agent image is missing | "can I run the live tiers — what's missing?" before a first live run |
|
|
420
420
|
| `prune [<runs-dir>] [--keep-last <n>] [--pinned-older-than <N>d\|h\|m]` | Prune accumulated run dirs (keeps the N most recent per scenario; pinned `--session-id` runs are never pruned unless `--pinned-older-than` opts in to reclaiming stale ones by last-activity age); the optional positional overrides the runs root; `--dry-run` | the machine-global runs root has grown and you want space back |
|
|
421
421
|
| `rehash <dir/>` | Migrate cassette fingerprints to the current format version when the content is provably unchanged (`--dry-run`); no re-record needed | a cassette-format bump flagged committed fixtures as stale |
|
|
@@ -671,7 +671,7 @@ jobs:
|
|
|
671
671
|
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
672
672
|
```
|
|
673
673
|
|
|
674
|
-
Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` — intentional so recipes track the current release; pin an exact version for reproducible CI. The companion skill's `cowork-harness@>=1.
|
|
674
|
+
Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` — intentional so recipes track the current release; pin an exact version for reproducible CI. The companion skill's `cowork-harness@>=1.6.0` floor guidance applies to ad-hoc CLI installs, not this input), `strict` (applies to `replay` (staleness findings), `lint`/`lint-skill` (WARN/INFO), and `analyze-skill` (any advisory finding); IGNORED — not forwarded — for `verify-cassettes`/`run`, which don't accept the flag), `fail-on-skill-drift` (**`replay`-only** — never forwarded to the analyzers), `extra-args`, `summary` (default `true`), `anthropic-api-key` (live lane only). See [`action.yml`](./action.yml) for the full input reference.
|
|
675
675
|
|
|
676
676
|
The provided [GitHub Actions workflow](.github/workflows/ci.yml) runs a **six-stage pipeline**. The **unit** stage is the token-free gate you can copy into your skill repo; the `action-self-test`, `python`, `boundary`, `scenarios`, and `parity-drift` stages are this repo's own fidelity self-tests and are not directly portable (they build the harness's Docker image and run harness-specific e2e scenarios — see [`ci-recipe.md`](./.claude/skills/cowork-harness/references/ci-recipe.md) for the skill-repo template):
|
|
677
677
|
|
|
@@ -714,7 +714,7 @@ Most runs need **none** of these — the defaults are correct. They're grouped b
|
|
|
714
714
|
- `COWORK_HARNESS_JUDGE_MODEL` — the default model for `semantic_matches`' LLM judge (overridden by a
|
|
715
715
|
per-assertion `judge_model`; falls back to the pinned default `claude-opus-4-8`). Pin it alongside `judge_model` for a
|
|
716
716
|
reproducible before/after comparison across re-records.
|
|
717
|
-
- `COWORK_HARNESS_RUNS_DIR` (or the `--run-dir <path>` flag — a **global** flag that must precede the subcommand
|
|
717
|
+
- `COWORK_HARNESS_RUNS_DIR` (or the `--run-dir <path>` flag — a **global** flag that must precede the subcommand — `--dotenv` follows the same rule everywhere except `critique`, which also takes it per-command) — override the default run-output root `~/.cowork-harness/runs` (kept out of any working tree so sensitive skill inputs/outputs don't land in a repo). Precedence: `--run-dir` > env > default. The root is flat and machine-global (shared across projects); pinned `--session-id` runs are guarded against cross-project overwrite, and `prune` never prunes them. In CI, set it to a workspace path (e.g. `runs`) so artifact upload can collect the runs. `COWORK_HARNESS_ALLOW_FOREIGN_RESUME=1` overrides the guard that blocks `--resume` onto another project's pinned session.
|
|
718
718
|
- **Networking / loop:** `COWORK_EGRESS_PROXY` overrides the egress-proxy URL injected into the sandbox; `COWORK_PROXY_IMAGE` overrides the egress proxy Docker image name (default `cowork-egress-proxy:2`); `COWORK_DOCKER_NETWORK` pins the Docker network the agent container joins; `CLAUDE_FORCE_HOST_LOOP=1` forces the host-loop path regardless of the baseline's loop decision (the `cowork` tier's auto-pick). `COWORK_LIMACTL` overrides the `limactl` binary path (default `/opt/homebrew/bin/limactl`).
|
|
719
719
|
- `COWORK_HARNESS_PRERUN_HASH_CAP` — override the default cap on pre-run file hashing (bytes); raise it if `input_unmodified`/`no_unexpected_files` report evidence unavailable on a large connected folder.
|
|
720
720
|
- `COWORK_HARNESS_MAX_ARTIFACT_BYTES` — override the inline-artifact-body cap (default 65536 bytes;
|
package/dist/cli.js
CHANGED
|
@@ -608,7 +608,11 @@ async function main() {
|
|
|
608
608
|
// more specific "<flag> requires a value" error; both exit 2, and the input is rare (value omitted AND
|
|
609
609
|
// literally equal to the token).
|
|
610
610
|
if (!hasHelp(rest) && COMMANDS.includes(cmd)) {
|
|
611
|
-
|
|
611
|
+
// `critique` is exempt from the `--dotenv` half: it is a legitimate PER-COMMAND flag there (forwarded
|
|
612
|
+
// to the inner `skill` turns and, since the fix below, validated by critique's own parseArgs), not a
|
|
613
|
+
// misplaced global. `--run-dir` stays rejected for every command, `critique` included — nothing reads
|
|
614
|
+
// a per-command `--run-dir`.
|
|
615
|
+
const misplacedGlobal = rest.find((t) => (t === "--dotenv" && cmd !== "critique") || t === "--run-dir");
|
|
612
616
|
if (misplacedGlobal)
|
|
613
617
|
fail(cmd, "usage", `${misplacedGlobal} is a GLOBAL flag and must come BEFORE the subcommand (e.g. \`cowork-harness ${misplacedGlobal} <path> ${cmd} …\`)`, undefined, isJsonOutput(rest));
|
|
614
618
|
}
|
package/dist/critique/command.js
CHANGED
|
@@ -12,7 +12,8 @@
|
|
|
12
12
|
// so the tier is pinned rather than left to the caller.
|
|
13
13
|
import { spawn } from "node:child_process";
|
|
14
14
|
import { fileURLToPath } from "node:url";
|
|
15
|
-
import {
|
|
15
|
+
import { lookupSkillFlag } from "../run/skill-flag-surface.js";
|
|
16
|
+
import { existsSync, readFileSync } from "node:fs";
|
|
16
17
|
import { randomUUID } from "node:crypto";
|
|
17
18
|
import { writeSync } from "node:fs";
|
|
18
19
|
import { basename } from "node:path";
|
|
@@ -49,18 +50,49 @@ you while you worked:
|
|
|
49
50
|
|
|
50
51
|
Answer plainly, in prose. Do not restate the task's final answer.`;
|
|
51
52
|
function usage() {
|
|
52
|
-
return `cowork-harness critique <skill-folder> --prompt "<probe>"
|
|
53
|
+
return `cowork-harness critique <skill-folder> --prompt "<probe>" | --prompt-file <path>
|
|
53
54
|
|
|
54
55
|
EXPERIMENTAL. Runs the skill, asks the agent what confused it, then does NOT believe the answer:
|
|
55
56
|
a blinded evaluator grades the self-report against a frozen record of what actually happened, and
|
|
56
57
|
drops any claim whose citation is not verbatim in that evidence. Discovery instrument, not a gate.
|
|
57
58
|
|
|
58
|
-
|
|
59
|
-
--
|
|
60
|
-
--
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
--
|
|
59
|
+
Probe (one required):
|
|
60
|
+
--prompt "<probe>" the task to run the skill against
|
|
61
|
+
--prompt-file <path> read the probe verbatim from a file (no shell parsing)
|
|
62
|
+
|
|
63
|
+
Files and sources (forwarded to the graded run — REQUIRED for "analyze this document" skills):
|
|
64
|
+
--upload <path> mount a file at mnt/uploads/<name> (repeatable)
|
|
65
|
+
--folder <dir> connect a folder at mnt/<name> (repeatable)
|
|
66
|
+
--plugin <dir> | --marketplace <dir> --enable <name@mkt> extra skill sources
|
|
67
|
+
|
|
68
|
+
Graded-run tuning (shapes the run being graded):
|
|
69
|
+
--model <id> session model for the agent doing the work AND reflecting
|
|
70
|
+
--timeout <ms> wall-clock budget for the task turn
|
|
71
|
+
--label <tag> generation tag in the run index (pair critiques across fixes)
|
|
72
|
+
--allow-missing-capability don't fail EITHER turn on a lean-image capability gap (both turns)
|
|
73
|
+
--answer "<q-regex>=<choice>" | --answer-policy <yaml> pre-answer the skill's gates (repeatable)
|
|
74
|
+
--on-unanswered fail|first unscripted-gate policy ('prompt' is refused: no TTY inside the spawn)
|
|
75
|
+
--decider-llm [--intent "<line>"] [--decider-model <id>] | --decider-cmd '<helper>' | --decider-dir <dir>
|
|
76
|
+
answer LIVE gates in the graded run (see 'skill --help')
|
|
77
|
+
|
|
78
|
+
Critique's own:
|
|
79
|
+
--evaluator-model <id> the grading model (env: COWORK_HARNESS_EVALUATOR_MODEL)
|
|
80
|
+
--output-format json|text critique's REPORT format (inner turns always speak json internally)
|
|
81
|
+
--fidelity container container tier only
|
|
82
|
+
--keep accepted as a no-op — runs are always kept
|
|
83
|
+
--dotenv <path> credentials
|
|
84
|
+
Global --run-dir <path> must PRECEDE the subcommand
|
|
85
|
+
|
|
86
|
+
Not accepted (each errors with its reason rather than being silently ignored):
|
|
87
|
+
--session-id / --resume critique mints and manages its own session internally
|
|
88
|
+
--repeat + companions fixed two-turn protocol — loop critique itself; pair by fingerprint.skillHash
|
|
89
|
+
--ablate-skill grading a skill you removed is incoherent
|
|
90
|
+
--quiet/--verbose/--compact/--demo/--dry-run inner-turn rendering or preview — no effect on the report
|
|
91
|
+
|
|
92
|
+
Repeating a flag: --upload/--folder/--plugin/--marketplace/--enable/--answer accumulate (that is how you
|
|
93
|
+
pass several). Every other value-taking flag is single-valued and repeating it is a USAGE ERROR rather
|
|
94
|
+
than a silent last-wins — '--prompt a --prompt b' would otherwise discard a probe you typed. Boolean
|
|
95
|
+
flags may be repeated harmlessly.
|
|
64
96
|
|
|
65
97
|
COST AND PREREQUISITES — read before running:
|
|
66
98
|
* Each critique is FOUR model workloads: two container runs (task + reflection) and two evaluator
|
|
@@ -79,6 +111,33 @@ KNOWN LIMITATIONS: container tier only; SKILL.md is capped at 16KB (a larger one
|
|
|
79
111
|
the instruction plane from evidence but cannot stop hostile content that merely ARGUES — see
|
|
80
112
|
docs/critique.md.`;
|
|
81
113
|
}
|
|
114
|
+
/** Reads a value-flag that may appear as EITHER `--flag value` (space form) OR `--flag=value` (equals
|
|
115
|
+
* form). critique's argv parsing runs in a separate process from cli.ts's `flagValue`/`flagValueEitherForm`
|
|
116
|
+
* helpers (a process-boundary-separated CLI — see the plan's Option (c) rejection), so it gets its own
|
|
117
|
+
* small copy rather than importing those private helpers. Returns `[value, extraTokensConsumed]` — 0 for
|
|
118
|
+
* the equals form (the value is inline in `a`), 1 for the space form (the value is the NEXT token) — so
|
|
119
|
+
* the caller's `i += extraTokensConsumed` advances the loop exactly like the old `argv[++i]` did. */
|
|
120
|
+
/** Returns [value, indexAdvance] and whether the EQUALS form was used (the child's escape hatch for a
|
|
121
|
+
* value starting with `-`, which its spaced-form parser rejects).
|
|
122
|
+
*
|
|
123
|
+
* The empty/missing check lives HERE, not at the call sites: it was previously applied only in the
|
|
124
|
+
* spec-forwarding branch, so `critique … --dotenv` with a forgotten path was ACCEPTED and ran a full
|
|
125
|
+
* four-workload critique without loading env — a silent no-op on the flag this branch made reachable. */
|
|
126
|
+
function flagVal(argv, i, flag) {
|
|
127
|
+
const a = argv[i];
|
|
128
|
+
if (a.startsWith(`${flag}=`)) {
|
|
129
|
+
const value = a.slice(flag.length + 1);
|
|
130
|
+
if (value.trim() === "")
|
|
131
|
+
throw new Error(`${flag} requires a non-empty value\n${usage()}`);
|
|
132
|
+
return { value, adv: 0, equalsForm: true };
|
|
133
|
+
}
|
|
134
|
+
const value = argv[i + 1];
|
|
135
|
+
// trim(), matching the child's own value checks — otherwise `--label " "` passes here and dies
|
|
136
|
+
// one layer later, which is the failure shape this check exists to prevent.
|
|
137
|
+
if (value === undefined || value.trim() === "")
|
|
138
|
+
throw new Error(`${flag} requires a value\n${usage()}`);
|
|
139
|
+
return { value, adv: 1, equalsForm: false };
|
|
140
|
+
}
|
|
82
141
|
function parseArgs(argv) {
|
|
83
142
|
const positional = [];
|
|
84
143
|
let prompt;
|
|
@@ -86,32 +145,143 @@ function parseArgs(argv) {
|
|
|
86
145
|
let fidelity = "container";
|
|
87
146
|
let evaluatorModel;
|
|
88
147
|
let outputFormat = "text";
|
|
148
|
+
let promptFile;
|
|
149
|
+
let taskTimeoutMs;
|
|
150
|
+
const forwardBoth = [];
|
|
151
|
+
const forwardTask = [];
|
|
152
|
+
const seen = new Set();
|
|
153
|
+
/** A repeat of a non-repeatable flag silently discards the earlier value — the exact no-op this
|
|
154
|
+
* command's refusal design exists to prevent. Applied to critique's OWN flags too: an earlier version
|
|
155
|
+
* guarded only the forwarded branch, so `--prompt a --prompt b` quietly dropped a probe the user typed.
|
|
156
|
+
* Arity-0 flags are exempt: there is no value to lose, and the child accepts them idempotently. */
|
|
157
|
+
const once = (flag, arity = 1) => {
|
|
158
|
+
if (arity === 0)
|
|
159
|
+
return;
|
|
160
|
+
if (seen.has(flag))
|
|
161
|
+
throw new Error(`${flag} given more than once (it is not repeatable)\n${usage()}`);
|
|
162
|
+
seen.add(flag);
|
|
163
|
+
};
|
|
89
164
|
for (let i = 0; i < argv.length; i++) {
|
|
90
165
|
const a = argv[i];
|
|
91
|
-
if (a === "--prompt")
|
|
92
|
-
prompt
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
else if (a === "--
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
166
|
+
if (a === "--prompt" || a.startsWith("--prompt=")) {
|
|
167
|
+
once("--prompt");
|
|
168
|
+
const { value: v, adv } = flagVal(argv, i, "--prompt");
|
|
169
|
+
prompt = v;
|
|
170
|
+
i += adv;
|
|
171
|
+
}
|
|
172
|
+
else if (a === "--dotenv" || a.startsWith("--dotenv=")) {
|
|
173
|
+
once("--dotenv");
|
|
174
|
+
const { value: v, adv } = flagVal(argv, i, "--dotenv");
|
|
175
|
+
dotenv = v;
|
|
176
|
+
i += adv;
|
|
177
|
+
}
|
|
178
|
+
else if (a === "--fidelity" || a.startsWith("--fidelity=")) {
|
|
179
|
+
once("--fidelity");
|
|
180
|
+
const { value: v, adv } = flagVal(argv, i, "--fidelity");
|
|
181
|
+
fidelity = v;
|
|
182
|
+
i += adv;
|
|
183
|
+
}
|
|
184
|
+
else if (a === "--evaluator-model" || a.startsWith("--evaluator-model=")) {
|
|
185
|
+
once("--evaluator-model");
|
|
186
|
+
const { value: v, adv } = flagVal(argv, i, "--evaluator-model");
|
|
187
|
+
evaluatorModel = v;
|
|
188
|
+
i += adv;
|
|
189
|
+
}
|
|
190
|
+
else if (a === "--output-format" || a.startsWith("--output-format=")) {
|
|
191
|
+
once("--output-format");
|
|
192
|
+
const { value: v, adv } = flagVal(argv, i, "--output-format");
|
|
193
|
+
outputFormat = v;
|
|
194
|
+
i += adv;
|
|
195
|
+
}
|
|
196
|
+
else if (a === "--prompt-file" || a.startsWith("--prompt-file=")) {
|
|
197
|
+
once("--prompt-file");
|
|
198
|
+
const { value: v, adv } = flagVal(argv, i, "--prompt-file");
|
|
199
|
+
promptFile = v;
|
|
200
|
+
i += adv;
|
|
201
|
+
}
|
|
202
|
+
else if (a === "--keep" || a.startsWith("--keep=")) {
|
|
203
|
+
// Match the equals form too so it errors as "takes no value" rather than falling through to the
|
|
204
|
+
// owned-flag branch's "unknown flag: --keep=x", which misdescribes the mistake.
|
|
205
|
+
if (a.includes("="))
|
|
206
|
+
throw new Error(`--keep takes no value (got "${a}")\n${usage()}`);
|
|
207
|
+
// accepted no-op: critique always keeps its runs, so the flag's promise already holds. Erroring on
|
|
208
|
+
// an already-satisfied request is hostile; silently ignoring an UNsatisfied one is this repo's
|
|
209
|
+
// anti-pattern — this is the former.
|
|
210
|
+
}
|
|
211
|
+
else if (a.startsWith("-")) {
|
|
212
|
+
// Not critique-owned: consult THE shared spec rather than a hand-mirrored list here. A skill flag
|
|
213
|
+
// with no disposition is impossible — the parity test makes that red CI.
|
|
214
|
+
const name = a.includes("=") ? a.slice(0, a.indexOf("=")) : a;
|
|
215
|
+
const spec = lookupSkillFlag(name);
|
|
216
|
+
if (!spec)
|
|
217
|
+
throw new Error(`unknown flag: ${a}\n${usage()}`);
|
|
218
|
+
// `repeatable` is enforced, not decorative — see `once()`.
|
|
219
|
+
if (!spec.repeatable)
|
|
220
|
+
once(name, spec.arity);
|
|
221
|
+
if (spec.critique.kind === "reject")
|
|
222
|
+
throw new Error(`${name} is not accepted by critique: ${spec.critique.reason}\n${usage()}`);
|
|
223
|
+
if (spec.critique.kind === "owned")
|
|
224
|
+
throw new Error(`unknown flag: ${a}\n${usage()}`); // owned => handled above
|
|
225
|
+
let value;
|
|
226
|
+
let eq = false;
|
|
227
|
+
if (spec.arity === 1) {
|
|
228
|
+
const { value: v, adv, equalsForm } = flagVal(argv, i, name);
|
|
229
|
+
eq = equalsForm;
|
|
230
|
+
value = v;
|
|
231
|
+
i += adv;
|
|
232
|
+
}
|
|
233
|
+
else if (a.includes("=")) {
|
|
234
|
+
// The child rejects `--boolean=x` outright ("takes no value"). Accepting it here and forwarding a
|
|
235
|
+
// BARE flag would silently invert intent — `--allow-missing-capability=false` would enable it.
|
|
236
|
+
throw new Error(`${name} takes no value (got "${a}")\n${usage()}`);
|
|
237
|
+
}
|
|
238
|
+
// --on-unanswered prompt would resolve differently than the caller expects: there is no TTY inside
|
|
239
|
+
// the spawn, so it cannot actually prompt anyone.
|
|
240
|
+
if (name === "--on-unanswered" && value !== "fail" && value !== "first")
|
|
241
|
+
throw new Error(`--on-unanswered must be "fail" or "first" for critique (got "${value}") — there is no TTY inside the spawned turn\n${usage()}`);
|
|
242
|
+
if (name === "--timeout") {
|
|
243
|
+
const n = Number(value);
|
|
244
|
+
if (!Number.isInteger(n) || n <= 0)
|
|
245
|
+
throw new Error(`--timeout requires a positive integer (ms), got "${value}"`);
|
|
246
|
+
taskTimeoutMs = n;
|
|
247
|
+
}
|
|
248
|
+
// Preserve the EQUALS form when that is how it arrived: the child's spaced-form parser rejects a
|
|
249
|
+
// value starting with `-`, so normalising `--intent=-terse` to two argv entries would kill a valid
|
|
250
|
+
// input one layer later with a wrong diagnosis.
|
|
251
|
+
const fragment = spec.arity === 1 ? (eq ? [`${name}=${value}`] : [name, value]) : [name];
|
|
252
|
+
// EXCLUSIVE buckets. `buildTaskTurnArgs` spreads BOTH arrays, so a "both" flag pushed into both
|
|
253
|
+
// would be emitted TWICE on the task turn — which is not a no-op for repeatable flags like
|
|
254
|
+
// --upload (it mounts the file twice). Sources go to forwardBoth ONLY; the task builder picks
|
|
255
|
+
// them up from there.
|
|
256
|
+
if (spec.critique.turns === "both")
|
|
257
|
+
forwardBoth.push(...fragment);
|
|
258
|
+
else
|
|
259
|
+
forwardTask.push(...fragment);
|
|
260
|
+
}
|
|
103
261
|
else
|
|
104
262
|
positional.push(a);
|
|
105
263
|
}
|
|
106
264
|
if (positional.length !== 1)
|
|
107
265
|
throw new Error(usage());
|
|
266
|
+
if (prompt !== undefined && promptFile !== undefined)
|
|
267
|
+
throw new Error(`--prompt and --prompt-file are mutually exclusive\n${usage()}`);
|
|
268
|
+
if (promptFile !== undefined) {
|
|
269
|
+
if (!existsSync(promptFile))
|
|
270
|
+
throw new Error(`--prompt-file not found: ${promptFile}`);
|
|
271
|
+
prompt = readFileSync(promptFile, "utf8");
|
|
272
|
+
}
|
|
108
273
|
if (!prompt || !prompt.trim())
|
|
109
|
-
throw new Error(`--prompt "<probe>" is required\n${usage()}`);
|
|
274
|
+
throw new Error(`--prompt "<probe>" or --prompt-file <path> is required\n${usage()}`);
|
|
110
275
|
if (fidelity !== "container")
|
|
111
276
|
throw new Error(`skill-critique is container-tier only (the resume-continuity behavior it relies on is only verified there); got --fidelity ${fidelity}`);
|
|
112
277
|
if (outputFormat !== "json" && outputFormat !== "text")
|
|
113
278
|
throw new Error(`--output-format must be "text" or "json" (got "${outputFormat}")`);
|
|
114
|
-
|
|
279
|
+
// Fail fast with critique's OWN clear error (mirroring cli.ts's global --dotenv existence check,
|
|
280
|
+
// ~line 627) rather than letting an absent file surface later as a generic instrument-failure
|
|
281
|
+
// diagnostic from the child `skill` invocation's own (differently-worded) rejection.
|
|
282
|
+
if (dotenv !== undefined && !existsSync(dotenv))
|
|
283
|
+
throw new Error(`--dotenv file not found: ${dotenv}\n${usage()}`);
|
|
284
|
+
return { skillFolder: positional[0], prompt, dotenv, fidelity, evaluatorModel, outputFormat, forwardBoth, forwardTask, taskTimeoutMs };
|
|
115
285
|
}
|
|
116
286
|
// F36: a hung reflection (stuck container, network stall, a `claude` process that never returns) must not
|
|
117
287
|
// block the whole discovery command forever, and a spewing/looping child must not grow the buffer
|
|
@@ -472,6 +642,53 @@ export function buildJsonReport(state) {
|
|
|
472
642
|
* invoked, unexpected throw). Distinct from FINDINGS, which never gate. Matches SPEC's exit 2
|
|
473
643
|
* (usage/runtime) rather than inventing a new code. */
|
|
474
644
|
const EXIT_INSTRUMENT_FAILURE = 2;
|
|
645
|
+
/** Argv for the GRADED turn. Forwarded fragments come BEFORE critique's pinned flags so a pinned value
|
|
646
|
+
* always wins (value flags are last-wins in the skill lane's parser). Exported for unit tests — the
|
|
647
|
+
* forwarding invariants are the kind that fail every run when wrong, so they are worth testing without
|
|
648
|
+
* paying for a spawn. */
|
|
649
|
+
export function buildTaskTurnArgs(opts, sessionId) {
|
|
650
|
+
const dotenvArgs = opts.dotenv ? ["--dotenv", opts.dotenv] : [];
|
|
651
|
+
return [
|
|
652
|
+
...dotenvArgs,
|
|
653
|
+
"skill",
|
|
654
|
+
opts.skillFolder,
|
|
655
|
+
opts.prompt,
|
|
656
|
+
...opts.forwardBoth,
|
|
657
|
+
...opts.forwardTask,
|
|
658
|
+
"--fidelity",
|
|
659
|
+
"container",
|
|
660
|
+
"--session-id",
|
|
661
|
+
sessionId,
|
|
662
|
+
"--keep",
|
|
663
|
+
"--output-format",
|
|
664
|
+
"json",
|
|
665
|
+
];
|
|
666
|
+
}
|
|
667
|
+
/** Argv for the REFLECTION turn — a resume of the same session.
|
|
668
|
+
*
|
|
669
|
+
* Only `forwardBoth` is replayed here, and it MUST be: session sources are part of the origin key, so a
|
|
670
|
+
* reflection turn that omits them computes a different identity and the resume throws fail-closed.
|
|
671
|
+
* `forwardTask` is deliberately absent — `--decider-dir` in particular requires a fresh empty dir per run
|
|
672
|
+
* and would break on turn 2, and gates belong to the graded run, not to critique's own protocol turn. */
|
|
673
|
+
export function buildReflectionTurnArgs(opts, sessionId) {
|
|
674
|
+
const dotenvArgs = opts.dotenv ? ["--dotenv", opts.dotenv] : [];
|
|
675
|
+
return [
|
|
676
|
+
...dotenvArgs,
|
|
677
|
+
"skill",
|
|
678
|
+
opts.skillFolder,
|
|
679
|
+
REFLECTION_PROMPT,
|
|
680
|
+
...opts.forwardBoth,
|
|
681
|
+
"--session-id",
|
|
682
|
+
sessionId,
|
|
683
|
+
"--resume",
|
|
684
|
+
"--fidelity",
|
|
685
|
+
"container",
|
|
686
|
+
"--on-unanswered",
|
|
687
|
+
"first",
|
|
688
|
+
"--output-format",
|
|
689
|
+
"json",
|
|
690
|
+
];
|
|
691
|
+
}
|
|
475
692
|
async function main(argv = process.argv.slice(2)) {
|
|
476
693
|
if (argv.includes("--help") || argv.includes("-h")) {
|
|
477
694
|
writeSync(1, usage() + "\n");
|
|
@@ -488,23 +705,13 @@ async function main(argv = process.argv.slice(2)) {
|
|
|
488
705
|
process.exit(2);
|
|
489
706
|
return;
|
|
490
707
|
}
|
|
491
|
-
const dotenvArgs = opts.dotenv ? ["--dotenv", opts.dotenv] : [];
|
|
492
708
|
const sessionId = `crit-${randomUUID()}`;
|
|
493
709
|
try {
|
|
494
710
|
// 1. Task turn.
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
opts.prompt,
|
|
500
|
-
"--fidelity",
|
|
501
|
-
"container",
|
|
502
|
-
"--session-id",
|
|
503
|
-
sessionId,
|
|
504
|
-
"--keep",
|
|
505
|
-
"--output-format",
|
|
506
|
-
"json",
|
|
507
|
-
]);
|
|
711
|
+
// Stretch critique's own kill-switch past a forwarded --timeout: otherwise a longer budget would be
|
|
712
|
+
// killed by the INSTRUMENT and misreported as an infra failure rather than a gradeable timeout. The
|
|
713
|
+
// +60s covers staging and container start — not principled, and a cold image pull can exceed it.
|
|
714
|
+
const task = await runSkillTurn(buildTaskTurnArgs(opts, sessionId), opts.taskTimeoutMs ? Math.max(TURN_TIMEOUT_MS, opts.taskTimeoutMs + 60_000) : TURN_TIMEOUT_MS);
|
|
508
715
|
const outDir = extractOutDir(task);
|
|
509
716
|
if (!outDir) {
|
|
510
717
|
// F36: surface WHY there's no envelope/status line when it was the bounded spawn itself that gave up.
|
|
@@ -552,21 +759,7 @@ async function main(argv = process.argv.slice(2)) {
|
|
|
552
759
|
// 2. Snapshot the turn-1/turn-2 boundary BEFORE the reflection turn touches anything.
|
|
553
760
|
const boundary = snapshotTurnBoundary(outDir);
|
|
554
761
|
// 3. Reflection turn: resume the SAME session.
|
|
555
|
-
const reflect = await runSkillTurn(
|
|
556
|
-
...dotenvArgs,
|
|
557
|
-
"skill",
|
|
558
|
-
opts.skillFolder,
|
|
559
|
-
REFLECTION_PROMPT,
|
|
560
|
-
"--session-id",
|
|
561
|
-
sessionId,
|
|
562
|
-
"--resume",
|
|
563
|
-
"--fidelity",
|
|
564
|
-
"container",
|
|
565
|
-
"--on-unanswered",
|
|
566
|
-
"first",
|
|
567
|
-
"--output-format",
|
|
568
|
-
"json",
|
|
569
|
-
]);
|
|
762
|
+
const reflect = await runSkillTurn(buildReflectionTurnArgs(opts, sessionId));
|
|
570
763
|
// F37: validate the reflection turn at the PROTOCOL level — exit code, envelope shape, and
|
|
571
764
|
// session/turn continuity (turn>1 AND outDir/sessionId match the task turn's) — BEFORE trusting its
|
|
572
765
|
// `finalMessage` as a self-report or handing anything to the evaluator. A failed reflection (crash, bad
|
|
@@ -1,6 +1,7 @@
|
|
|
1
|
-
import { readFileSync, readdirSync, existsSync } from "node:fs";
|
|
1
|
+
import { readFileSync, readdirSync, existsSync, statSync } from "node:fs";
|
|
2
2
|
import { join } from "node:path";
|
|
3
3
|
import { readTurn1ResultWithStatus, readTurn1Slice, verifyBoundaryIntegrity } from "./evidence.js";
|
|
4
|
+
import { loadVmPathContext } from "../run/vm-path-ctx-file.js";
|
|
4
5
|
// Assembles the TURN-1-ONLY evidence document a tool-less, one-shot evaluator model is graded against.
|
|
5
6
|
// This packager is a from-scratch, load-bearing build (this instrument's design): a tool-less
|
|
6
7
|
// evaluator can't grep the logs itself, so its recall is bounded by exactly what this function decides to
|
|
@@ -50,6 +51,48 @@ function readTurn1Transcript(runDir, boundary) {
|
|
|
50
51
|
return { text: "", degraded: true };
|
|
51
52
|
}
|
|
52
53
|
}
|
|
54
|
+
/** List of what was ATTACHED to this run — upload filenames + byte sizes, and connected-folder mount names.
|
|
55
|
+
* NEVER file content: the evaluator needs to be able to tell "the agent said there was no file, and
|
|
56
|
+
* correctly so" apart from "the agent confabulated that", but packaging the bytes themselves would blow
|
|
57
|
+
* the package's byte budget and widen the prompt-injection surface the armor exists to contain (see this
|
|
58
|
+
* module's header and `armor.ts`'s header) — names and sizes are enough to answer "was anything attached."
|
|
59
|
+
*
|
|
60
|
+
* Source of truth, with fallbacks, in order:
|
|
61
|
+
* 1. `loadVmPathContext(runDir)` (`run/vm-path-ctx-file.ts`) — the recorded `uploadsHostDir` and
|
|
62
|
+
* `folders` map for THIS run. Already never throws (absent/corrupt `mounts.json` -> `null`).
|
|
63
|
+
* 2. The fixed container layout `<runDir>/work/session/mnt/uploads` (mirrors the derivation in
|
|
64
|
+
* `run/display-translate.ts`'s `vmPathContextFromPlan`) — covers runs where `mounts.json` wasn't
|
|
65
|
+
* written or couldn't be parsed.
|
|
66
|
+
* 3. `(none)` — via `sec()`'s empty-body fallback below. Every filesystem read here is try/catch-guarded
|
|
67
|
+
* so a missing/unreadable uploads dir degrades gracefully rather than throwing and sinking packaging. */
|
|
68
|
+
function listAttachedInputs(runDir) {
|
|
69
|
+
const loaded = loadVmPathContext(runDir);
|
|
70
|
+
const uploadsDir = loaded?.ctx.uploadsHostDir ?? join(runDir, "work", "session", "mnt", "uploads");
|
|
71
|
+
const folderNames = loaded ? Array.from(loaded.ctx.folders.keys()).sort() : [];
|
|
72
|
+
const lines = [];
|
|
73
|
+
try {
|
|
74
|
+
const uploadNames = readdirSync(uploadsDir, { withFileTypes: true })
|
|
75
|
+
.filter((e) => e.isFile())
|
|
76
|
+
.map((e) => e.name)
|
|
77
|
+
.sort();
|
|
78
|
+
for (const name of uploadNames) {
|
|
79
|
+
let sizeNote;
|
|
80
|
+
try {
|
|
81
|
+
sizeNote = `${statSync(join(uploadsDir, name)).size} bytes`;
|
|
82
|
+
}
|
|
83
|
+
catch {
|
|
84
|
+
sizeNote = "size unknown";
|
|
85
|
+
}
|
|
86
|
+
lines.push(`${name} (${sizeNote})`);
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
catch {
|
|
90
|
+
/* uploads dir absent/unreadable — no uploads to list, not an error */
|
|
91
|
+
}
|
|
92
|
+
for (const name of folderNames)
|
|
93
|
+
lines.push(`${name} (connected folder)`);
|
|
94
|
+
return lines.join("\n");
|
|
95
|
+
}
|
|
53
96
|
/** Byte-bound a text section, appending a loud (never silent) truncation marker so the evaluator knows the
|
|
54
97
|
* section was cut rather than reading a suspiciously short document as "that's everything." Cuts on a
|
|
55
98
|
* UTF-8-safe boundary (Buffer, not string length, so a truncated multi-byte char can't corrupt the tail). */
|
|
@@ -91,6 +134,7 @@ const REFERENCES_READ_CAP = 1 * 1024;
|
|
|
91
134
|
const SKILL_MD_CAP = 16 * 1024;
|
|
92
135
|
const REFERENCE_LIST_CAP = 1 * 1024;
|
|
93
136
|
const TRANSCRIPT_CAP = 16 * 1024;
|
|
137
|
+
const ATTACHED_INPUTS_CAP = 1 * 1024;
|
|
94
138
|
/** The overall package hard cap — the whole assembled document is trimmed to this even if every
|
|
95
139
|
* per-section budget above was individually respected (their sum is deliberately a bit under this, but a
|
|
96
140
|
* belt-and-suspenders final trim means a future per-section budget change can never silently blow the
|
|
@@ -200,6 +244,7 @@ export function packageEvidence(runDir, boundary, skillDir, isResume = false) {
|
|
|
200
244
|
turn1ResultDegradedNote, bound(referencesRead.length ? referencesRead.join("\n") : "(none)", REFERENCES_READ_CAP)),
|
|
201
245
|
sec(skillMdSectionTitle, bound(skillMdSectionBody, SKILL_MD_CAP)),
|
|
202
246
|
sec("references/ available (filenames only, NOT content — presence, not coverage)", bound(referenceFiles.length ? referenceFiles.join("\n") : "(none)", REFERENCE_LIST_CAP)),
|
|
247
|
+
sec("Attached inputs (mnt/uploads filenames + sizes, and connected-folder mount names — NOT content)", bound(listAttachedInputs(runDir), ATTACHED_INPUTS_CAP)),
|
|
203
248
|
sec("Transcript (turn 1 only — the reflection turn's own reads/output are excluded by construction)" +
|
|
204
249
|
(turn1SliceDegraded
|
|
205
250
|
? " [DEGRADED: the turn-1/turn-2 boundary for this fallback slice could not be verified — treat gaps as unknown, not as evidence of absence]"
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
// THE source of truth for `cowork-harness skill`'s flag surface, and for what `critique` does with each
|
|
2
|
+
// flag.
|
|
3
|
+
//
|
|
4
|
+
// WHY THIS EXISTS. `critique` runs a skill by spawning two `skill` turns (a task turn, then a `--resume`
|
|
5
|
+
// reflection turn), but its own parser was a hand-rolled SUBSET of `skill`'s flags. That subset drifts
|
|
6
|
+
// silently — a `skill` flag added tomorrow is simply unavailable to `critique`, with nothing failing. This
|
|
7
|
+
// repo has shipped that exact "propagate-minus-one" shape repeatedly, so a design that requires manually
|
|
8
|
+
// mirroring a flag list is wrong even when it is momentarily correct.
|
|
9
|
+
//
|
|
10
|
+
// The fix is structural, in two halves:
|
|
11
|
+
// 1. `critique` field below is REQUIRED, so an entry cannot be added without deciding its disposition.
|
|
12
|
+
// 2. `test/critique-skill-flag-parity.test.ts` extracts the flag literals the `skill` lane actually
|
|
13
|
+
// accepts and fails if any is missing here — so a NEW skill flag is red CI, not a silent gap.
|
|
14
|
+
//
|
|
15
|
+
// Half 2 is the load-bearing half: the required field only gives totality over entries already present.
|
|
16
|
+
const forwardBoth = () => ({ kind: "forward", turns: "both" });
|
|
17
|
+
const forwardTask = () => ({ kind: "forward", turns: "task" });
|
|
18
|
+
export const SKILL_FLAG_SURFACE = [
|
|
19
|
+
// ---- session SOURCES — must reach BOTH turns or the resume throws on an origin-key mismatch ----
|
|
20
|
+
{ flag: "--upload", arity: 1, repeatable: true, critique: forwardBoth() },
|
|
21
|
+
{ flag: "--folder", arity: 1, repeatable: true, critique: forwardBoth() },
|
|
22
|
+
{ flag: "--plugin", arity: 1, repeatable: true, critique: forwardBoth() },
|
|
23
|
+
{ flag: "--marketplace", arity: 1, repeatable: true, critique: forwardBoth() },
|
|
24
|
+
{ flag: "--enable", arity: 1, repeatable: true, critique: forwardBoth() },
|
|
25
|
+
// ---- session shape that both turns must agree on ----
|
|
26
|
+
// The reflection must run on the SAME model as the task turn: the skill lane rebuilds its inline session
|
|
27
|
+
// per invocation, so an unforwarded resume would reflect using the DEFAULT model — a self-report from a
|
|
28
|
+
// different model than the one that did the work.
|
|
29
|
+
{ flag: "--model", arity: 1, critique: forwardBoth() },
|
|
30
|
+
// Without this a lean-image capability gap makes the task turn error, and the critique harvests a fact
|
|
31
|
+
// about the rig instead of the skill — the same failure class as a missing upload.
|
|
32
|
+
{ flag: "--allow-missing-capability", arity: 0, critique: forwardBoth() },
|
|
33
|
+
// ---- the GRADED run only ----
|
|
34
|
+
// TASK-only deliberately: forwarding to the reflection turn would inject a near-always-green row into the
|
|
35
|
+
// labelled generation group, inflating passRate and breaking the `runLabel` filter docs/debugging.md
|
|
36
|
+
// documents as the escape hatch for exactly that dilution. Not in the origin key, so nothing forces both.
|
|
37
|
+
{ flag: "--label", arity: 1, critique: forwardTask() },
|
|
38
|
+
// Parsed as well as forwarded: critique's own spawn kill-switch is fixed, so a longer --timeout would be
|
|
39
|
+
// killed by the INSTRUMENT and misreported as an infra failure instead of a gradeable timeout.
|
|
40
|
+
{ flag: "--timeout", arity: 1, critique: forwardTask() },
|
|
41
|
+
// Makes GATED skills critiquable for the first time: the inner spawn has no TTY, so an unscripted gate
|
|
42
|
+
// otherwise resolves to `fail` and the task turn dies before anything can be graded.
|
|
43
|
+
{ flag: "--answer", arity: 1, repeatable: true, critique: forwardTask() },
|
|
44
|
+
{ flag: "--answer-policy", arity: 1, critique: forwardTask() },
|
|
45
|
+
// `prompt` is rejected at parse time — no TTY inside the spawn, so it would resolve differently than the
|
|
46
|
+
// caller expects. Gates belong to the graded run; the reflection turn stays pinned deterministic.
|
|
47
|
+
{ flag: "--on-unanswered", arity: 1, critique: forwardTask() },
|
|
48
|
+
{ flag: "--decider-llm", arity: 0, critique: forwardTask() },
|
|
49
|
+
{ flag: "--intent", arity: 1, critique: forwardTask() },
|
|
50
|
+
{ flag: "--decider-model", arity: 1, critique: forwardTask() },
|
|
51
|
+
{ flag: "--decider-cmd", arity: 1, critique: forwardTask() },
|
|
52
|
+
// HARD invariant: the file channel requires a fresh, empty dir per run. Reaching the reflection turn
|
|
53
|
+
// would break it.
|
|
54
|
+
{ flag: "--decider-dir", arity: 1, critique: forwardTask() },
|
|
55
|
+
// ---- refused ----
|
|
56
|
+
{
|
|
57
|
+
flag: "--session-id",
|
|
58
|
+
arity: 1,
|
|
59
|
+
critique: {
|
|
60
|
+
kind: "reject",
|
|
61
|
+
reason: "critique mints and manages its own session (the reflection turn IS a resume of it)",
|
|
62
|
+
},
|
|
63
|
+
},
|
|
64
|
+
{
|
|
65
|
+
flag: "--resume",
|
|
66
|
+
arity: 0,
|
|
67
|
+
critique: {
|
|
68
|
+
kind: "reject",
|
|
69
|
+
reason: "critique mints and manages its own session (the reflection turn IS a resume of it)",
|
|
70
|
+
},
|
|
71
|
+
},
|
|
72
|
+
...["--repeat", "--min-pass-rate", "--stop-on-diverge", "--max-budget-usd", "--allow-budget-stop"].map((flag) => ({
|
|
73
|
+
flag,
|
|
74
|
+
arity: flag === "--stop-on-diverge" || flag === "--allow-budget-stop" ? 0 : 1,
|
|
75
|
+
critique: {
|
|
76
|
+
kind: "reject",
|
|
77
|
+
reason: "critique is a fixed two-turn protocol — loop `critique` itself and pair generations by fingerprint.skillHash",
|
|
78
|
+
},
|
|
79
|
+
})),
|
|
80
|
+
{
|
|
81
|
+
flag: "--ablate-skill",
|
|
82
|
+
arity: 0,
|
|
83
|
+
critique: { kind: "reject", reason: "grading a skill you removed is incoherent (and ablate+resume is rejected anyway)" },
|
|
84
|
+
},
|
|
85
|
+
...["--quiet", "-q", "--verbose", "--compact", "--demo"].map((flag) => ({
|
|
86
|
+
flag,
|
|
87
|
+
arity: 0,
|
|
88
|
+
critique: {
|
|
89
|
+
kind: "reject",
|
|
90
|
+
reason: "critique produces its own report; inner-turn rendering flags have no effect on it",
|
|
91
|
+
},
|
|
92
|
+
})),
|
|
93
|
+
{
|
|
94
|
+
flag: "--dry-run",
|
|
95
|
+
arity: 0,
|
|
96
|
+
critique: { kind: "reject", reason: "there is no meaningful two-turn preview — use `skill --dry-run` directly" },
|
|
97
|
+
},
|
|
98
|
+
// ---- critique implements or pins these itself ----
|
|
99
|
+
{ flag: "--prompt-file", arity: 1, critique: { kind: "owned", note: "critique's probe prompt, read from a file" } },
|
|
100
|
+
{ flag: "--fidelity", arity: 1, critique: { kind: "owned", note: "pinned container — resume continuity is verified only there" } },
|
|
101
|
+
{
|
|
102
|
+
flag: "--output-format",
|
|
103
|
+
arity: 1,
|
|
104
|
+
critique: { kind: "owned", note: "critique's REPORT format; the inner turns always speak json internally" },
|
|
105
|
+
},
|
|
106
|
+
{ flag: "--keep", arity: 0, critique: { kind: "owned", note: "accepted as a no-op — critique always keeps its runs" } },
|
|
107
|
+
];
|
|
108
|
+
/** Flags `critique` defines that the `skill` lane does not have. Kept separate so the parity guard can
|
|
109
|
+
* treat them as legitimately-unknown-to-skill rather than drift. */
|
|
110
|
+
// `--prompt` is here, not in the surface above: the `skill` lane takes its prompt POSITIONALLY
|
|
111
|
+
// (`skill <folder> "<prompt>"`), so there is no `--prompt` flag to have a disposition about. The parity
|
|
112
|
+
// guard caught this modelling error on its first run.
|
|
113
|
+
export const CRITIQUE_ONLY_FLAGS = ["--evaluator-model", "--dotenv", "--prompt"];
|
|
114
|
+
const BY_FLAG = new Map(SKILL_FLAG_SURFACE.map((s) => [s.flag, s]));
|
|
115
|
+
export function lookupSkillFlag(name) {
|
|
116
|
+
return BY_FLAG.get(name);
|
|
117
|
+
}
|
package/docs/critique.md
CHANGED
|
@@ -63,6 +63,83 @@ evidence going into it trustworthy.
|
|
|
63
63
|
**One naming collision to know about:** `hostloop` in this repo is a **fidelity tier** — where the agent
|
|
64
64
|
process runs. It has nothing to do with loop engineering's "loops".
|
|
65
65
|
|
|
66
|
+
## Flags
|
|
67
|
+
|
|
68
|
+
`critique` accepts the `skill` flags that make sense for a graded run, under the **same names** — what you
|
|
69
|
+
know from `skill` transfers. Anything that cannot work is refused with a reason rather than silently
|
|
70
|
+
ignored.
|
|
71
|
+
|
|
72
|
+
**Probe** (one required)
|
|
73
|
+
|
|
74
|
+
| Flag | |
|
|
75
|
+
|---|---|
|
|
76
|
+
| `--prompt "<text>"` | the task to run the skill against |
|
|
77
|
+
| `--prompt-file <path>` | read the probe verbatim from a file — for probes containing quotes, `$`, or newlines |
|
|
78
|
+
|
|
79
|
+
**Files and sources** — forwarded to the graded run. **Required for "analyze this document" skills.**
|
|
80
|
+
|
|
81
|
+
| Flag | |
|
|
82
|
+
|---|---|
|
|
83
|
+
| `--upload <path>` | mount a file at `mnt/uploads/<name>` (repeatable) |
|
|
84
|
+
| `--folder <dir>` | connect a folder at `mnt/<name>` (repeatable) |
|
|
85
|
+
| `--plugin <dir>`, `--marketplace <dir>` + `--enable <name@mkt>` | extra skill sources |
|
|
86
|
+
|
|
87
|
+
**Session shape** — both turns must agree on these, so they reach the reflection turn too.
|
|
88
|
+
|
|
89
|
+
| Flag | |
|
|
90
|
+
|---|---|
|
|
91
|
+
| `--model <id>` | session model for the agent doing the work *and* reflecting |
|
|
92
|
+
| `--allow-missing-capability` | don't fail either turn when the lean image omits a capability |
|
|
93
|
+
|
|
94
|
+
**Graded-run tuning** — the task turn only; the reflection turn stays pinned deterministic.
|
|
95
|
+
|
|
96
|
+
| Flag | |
|
|
97
|
+
|---|---|
|
|
98
|
+
| `--timeout <ms>` | wall-clock budget for the task turn (critique's own kill-switch stretches to fit) |
|
|
99
|
+
| `--label <tag>` | generation tag in the run index, for pairing critiques across fixes |
|
|
100
|
+
| `--answer "<q-regex>=<choice>"`, `--answer-policy <yaml>` | pre-answer the skill's gates — **this is what makes gated skills critiquable at all** |
|
|
101
|
+
| `--on-unanswered fail\|first` | unscripted-gate policy (`prompt` is refused — there is no TTY inside) |
|
|
102
|
+
| `--decider-llm` / `--intent` / `--decider-model` / `--decider-cmd` / `--decider-dir` | answer live gates in the graded run |
|
|
103
|
+
|
|
104
|
+
**Critique's own**
|
|
105
|
+
|
|
106
|
+
| Flag | |
|
|
107
|
+
|---|---|
|
|
108
|
+
| `--evaluator-model <id>` | the grading model (env: `COWORK_HARNESS_EVALUATOR_MODEL`) |
|
|
109
|
+
| `--output-format json\|text` | critique's *report* format — the inner turns always speak JSON internally |
|
|
110
|
+
| `--fidelity container` | container tier only |
|
|
111
|
+
| `--keep` | accepted as a no-op; runs are always kept |
|
|
112
|
+
| `--dotenv <path>` | credentials — works **before** `critique` (the global form) or **after** it |
|
|
113
|
+
| `--run-dir <path>` | **global, unlike `--dotenv`** — must still PRECEDE the subcommand; a trailing `critique … --run-dir` is rejected |
|
|
114
|
+
|
|
115
|
+
**Refused, and why**
|
|
116
|
+
|
|
117
|
+
| Flag | Reason |
|
|
118
|
+
|---|---|
|
|
119
|
+
| `--session-id` / `--resume` | critique mints and manages its own session — the reflection turn *is* a resume of it |
|
|
120
|
+
| `--repeat` + companions | fixed two-turn protocol; loop `critique` itself and pair by `fingerprint.skillHash` |
|
|
121
|
+
| `--ablate-skill` | grading a skill you removed is incoherent |
|
|
122
|
+
| `--quiet`/`-q` / `--verbose` / `--compact` / `--demo` / `--dry-run` | inner-turn rendering or preview — no effect on the report |
|
|
123
|
+
|
|
124
|
+
**Repeating a flag.** `--upload`, `--folder`, `--plugin`, `--marketplace`, `--enable` and `--answer` accumulate,
|
|
125
|
+
so repeating them is how you pass several. Every other value-taking flag is single-valued and repeating it is
|
|
126
|
+
a **usage error** (exit `2`) rather than a silent last-wins — `--prompt a --prompt b` would otherwise discard
|
|
127
|
+
a probe you typed. Boolean flags may be repeated harmlessly.
|
|
128
|
+
|
|
129
|
+
### Skills that need an attached file
|
|
130
|
+
|
|
131
|
+
```bash
|
|
132
|
+
cowork-harness critique ./captable-skill \
|
|
133
|
+
--prompt "Analyze this cap table and flag anything unusual" \
|
|
134
|
+
--upload ./acme-captable.xlsx
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
Both internal turns receive the source flags — they have to, or the reflection turn's resume computes a
|
|
138
|
+
different session identity and fails. The evidence package records **which files were attached** (names and
|
|
139
|
+
sizes), so the evaluator can tell "the agent said there was no file, and correctly so" from a confabulation.
|
|
140
|
+
|
|
141
|
+
It does **not** record their contents — see Known limitations.
|
|
142
|
+
|
|
66
143
|
## Cost and prerequisites
|
|
67
144
|
|
|
68
145
|
- **Four model workloads per critique**: two container runs (task + reflection) and two evaluator passes.
|
|
@@ -109,6 +186,13 @@ Changing the evaluator model invalidates that verification.
|
|
|
109
186
|
- **SKILL.md is capped at 16KB**; a larger one degrades toward "not adjudicable".
|
|
110
187
|
- **English-only prompts.**
|
|
111
188
|
- The evidence package is not persisted; the report is written to stdout.
|
|
189
|
+
- **Attached-file content usually stays out of the evidence — but that is the common case, not a
|
|
190
|
+
guarantee.** "Attached inputs" lists names and sizes only, never bytes, and the primary transcript
|
|
191
|
+
source is assistant prose. But packaging falls back to a raw slice of `events.jsonl` when the archived
|
|
192
|
+
transcript is missing, and that stream carries full tool results — so if the agent read the attached
|
|
193
|
+
file, its content can enter the Transcript section (bounded, still armor-fenced) and a content-level
|
|
194
|
+
citation would resolve. Claims about a document's *contents* are therefore usually NOT ADJUDICABLE, not
|
|
195
|
+
always.
|
|
112
196
|
- **Citation seams.** Armor inserts a marker line between each section heading and its body. A quote that
|
|
113
197
|
spans that seam *without* including the marker does not resolve and is DROPPED. Quotes wholly inside one
|
|
114
198
|
section are unaffected. **Measured:** on a benign package, 9 findings across 5 live pass-1 runs produced
|
package/docs/debugging.md
CHANGED
|
@@ -135,6 +135,9 @@ built on top of the harness before any of it shipped — the commands now replac
|
|
|
135
135
|
|
|
136
136
|
```bash
|
|
137
137
|
SKILL=./my-skill
|
|
138
|
+
# Skill needs an attached input (cap table, deck, transcript)? Add `--upload <path>` / `--folder <dir>`
|
|
139
|
+
# to BOTH step 1 and step 3 — critique forwards them to its own two turns, and step 3 must reproduce
|
|
140
|
+
# with the same inputs the harvest used. See critique.md → "Skills that need an attached file".
|
|
138
141
|
|
|
139
142
|
# 1. HARVEST — run the skill against a real input and grade what confused the agent.
|
|
140
143
|
# `critique` runs the task, asks the agent what was unclear, then verifies every claim against a
|
|
@@ -16,7 +16,7 @@ DOES exercise a real gate exchange, see `example-multiselect-gate.cassette.json`
|
|
|
16
16
|
|
|
17
17
|
Run it with:
|
|
18
18
|
|
|
19
|
-
> Assumes the `cowork-harness` CLI is available — from a source checkout run `npm ci && npm run build && npm link` first, or `npm i -g "cowork-harness@>=1.
|
|
19
|
+
> Assumes the `cowork-harness` CLI is available — from a source checkout run `npm ci && npm run build && npm link` first, or `npm i -g "cowork-harness@>=1.6.0"`. (`replay` itself needs nothing else — no token, no Docker.)
|
|
20
20
|
|
|
21
21
|
```sh
|
|
22
22
|
cowork-harness replay examples/replays/example-pdf-skill.cassette.json
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "cowork-harness",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.6.0",
|
|
4
4
|
"description": "Scriptable, CI-friendly harness for Claude Cowork's runtime contract for testing skills across scenarios \u2014 same agent, mounts, egress allowlist, permission protocol, and sandbox limitations.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|