cowork-harness 0.12.0 → 0.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -4,6 +4,61 @@ All notable changes to this project are documented here. The format is based on
4
4
  [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The project uses
5
5
  [Semantic Versioning](https://semver.org/); pre-1.0 minor versions may include breaking changes.
6
6
 
7
+ ## [Unreleased]
8
+
9
+ ## [0.13.0] — 2026-06-25
10
+
11
+ ### Added
12
+
13
+ - **`inspect <run-id | run-dir>` — see what a run produced.** Lists the run's artifacts and prints a shallow
14
+ field preview of each JSON artifact (scalars inline, arrays as a count, nested objects collapsed), so
15
+ "did it do the job?" is a first-class check instead of hand-parsing `…/mnt/outputs/...`. `--output-format
16
+ json` emits a structured digest. When the work dir was torn down (a non-`--keep` container/microvm run),
17
+ the artifact manifest still prints from `result.json` and the preview notes it's unavailable.
18
+ - **A whiffed gate no longer discards the paid run.** When a run exits on an unanswered gate
19
+ (`on_unanswered: fail`), the harness now salvages a **PARTIAL** `result.json` (+ `run.jsonl`/`trace.json`)
20
+ with the artifacts the agent wrote before the whiff, then still exits 2. The run dir is printed so you can
21
+ `inspect` it. `partial` and `unansweredGate` are new `RunResult` fields; `verify-run` and `scaffold`
22
+ refuse to treat a partial run's half-finished output as a passing result (scaffold still emits the gates,
23
+ drops the artifact/result asserts, and warns loudly).
24
+ - **Capability pre-flight — fail fast.** A skill that declares `requires_capabilities` against an image that
25
+ provably omits them now **aborts before the (paid) run** (`exit 3`) instead of running ~12 min to a verdict
26
+ that's already known — unless the scenario asserts `allow_missing_capability: true`, which downgrades it to
27
+ a notice and proceeds. `doctor` also surfaces the full-parity remedy on its agent-image line.
28
+
29
+ ### Changed
30
+
31
+ - **LLM decider tolerates a near-miss label.** `--decider-llm` now binds a reply like `Confirmed.` or
32
+ `"Confirmed"` to the label `Confirmed` (trailing sentence punctuation / surrounding quotes trimmed before
33
+ matching) instead of failing loud — common on binary confirm gates. The `:` of the `OTHER:` free-text
34
+ sentinel is never stripped, and fuzzy substring matching stays off, so the change can't mis-bind. The
35
+ tolerance lives in `matchLabel` itself, so the web_fetch-approval path gets it too (a `Deny.` reply now
36
+ binds `Deny`).
37
+ - **`runs gc` → top-level `prune`; the `runs` namespace is dropped.** Pruning accumulated run dirs is now
38
+ `cowork-harness prune [--keep-last <n>] [--dry-run] [<runs-dir>]` (same flags). `runs` was a namespace with
39
+ a single member and collided confusingly with `run` (execute); removing it leaves `run` = execute,
40
+ inspection verbs (`trace`/`inspect`/`verify-run`/`scaffold`) top-level, and `prune` for cleanup.
41
+
42
+ ### Fixed
43
+
44
+ - **Session-protocol over-cap test no longer flakes.** The over-cap control-out test read
45
+ `control-out.jsonl` synchronously, but the stream opens it asynchronously (`createWriteStream`, flags
46
+ `"a"`), so under load the file may not exist yet at the read — a flaky ENOENT. The over-cap frame is
47
+ rejected before any write, so a missing file means nothing was written; the test now treats absence as
48
+ empty.
49
+
50
+ ### Docs
51
+
52
+ - **Documented the LLM-decider free-text path.** `--decider-llm` supplies free text via `OTHER: <value>`
53
+ on an options-bearing gate; a bare out-of-set answer (no matching label, no `OTHER:`) fails loud
54
+ (`UnansweredError` → exit 2), never stalling or guessing, and open-ended (no-option) gates need no
55
+ prefix. Noted in the decider section of `fidelity-and-answers.md` and cross-referenced from the lone
56
+ `OTHER:` mention in `SKILL.md` so the two don't drift.
57
+ - **Documented the live real-document validation workflow.** A new SKILL.md section covers driving a skill
58
+ against real input documents (not recording a cassette): explore with `--decider-llm --intent`, script the
59
+ load-bearing and binary-confirm gates, budget ~1 re-run per file (a whiffed gate now salvages a partial
60
+ run), and `inspect` the outputs to judge correctness.
61
+
7
62
  ## [0.12.0] — 2026-06-24
8
63
 
9
64
  ### Added
package/README.md CHANGED
@@ -97,6 +97,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
97
97
  | `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain → exit 1; whole-token allows via `--allow` / class-scoped `--allow-domain` / `--allow-email` / `--allow-file`) + a staleness check (`--skip-privacy` or `--skip-staleness` to run only one half). A dir argument scans `*.cassette.json` in that dir only (**non-recursive**) | gating **committed cassettes** against PII leaks + "edited the skill, forgot to re-record" |
98
98
  | `verify-run <run-dir> <scenario.yaml>` | Re-evaluate a scenario's `assert:` (and, when the scenario declares `answers:`, whether they still match the run's actual gates) against an already-kept run dir — **no live agent, no tokens, no Docker** (~1s) | iterating on a wrong assertion or a drifted `answer` without a full live re-record |
99
99
  | `trace <run-id>` | Digest a run's `events.jsonl` (`--view tools\|questions\|dispatches`; legacy `--tools` / `--gates` / `--dispatches` aliases still accepted) | "how many sub-agents *actually* dispatched, and which?" |
100
+ | `inspect <run-id>` | Show what a run **produced**: the artifacts + a shallow field preview of each JSON artifact (`--output-format json` for a digest). Works on a salvaged partial run too | "did it do the job?" — without hand-parsing `…/mnt/outputs/…` |
100
101
  | `scaffold <run-id>` | Turn a kept run into a starter scenario YAML (gates→answers, artifacts→`file_exists`); `--from-run <id>` is a deprecated alias for the positional | authoring a scenario from a real run instead of guessing |
101
102
  | `python3 …/scenario.py scaffold --name <name> --skill <dir>` | Generate a starter scenario skeleton from scratch (the `…` is `.claude/skills/cowork-harness/scripts/`) | starting a new scenario when you have no prior run |
102
103
  | `lint <scenario.yaml \| dir/>…` | Check scenarios for silent false-greens — assertions placed on the wrong CI lane, mixed content/live keys, missing `controlOut`-required keys (files or a directory of `*.yaml`/`*.yml`; bundled `scenario.py`; needs python3 — PyYAML is bundled) | before committing a new scenario or after changing assertions |
@@ -106,7 +107,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
106
107
  | `boundary-check [baseline] [--session <file>]` | Prove the sandbox enforces Cowork's limitations; `--session` folds a session's `egress.extra_allow` into the probe allowlist | verifying the harness's own fidelity |
107
108
  | `sync` / `list` | Derive/refresh & list platform baselines from the Desktop install | after Claude Desktop updates (baselines ship, so it's optional otherwise) |
108
109
  | `doctor [--tier <t>]` | Read-only prerequisite check, per tier (Docker + agent image for `container`/`hostloop`; **Lima** for `microvm`; plus staged agent, token, baseline); prints the exact `docker build` line if the agent image is missing | "can I run the live tiers — what's missing?" before a first live run |
109
- | `runs gc [--keep-last <n>]` | Prune accumulated run dirs (keeps the N most recent per scenario; pinned `--session-id` runs are never pruned); `--dry-run` | the machine-global runs root has grown and you want space back |
110
+ | `prune [--keep-last <n>]` | Prune accumulated run dirs (keeps the N most recent per scenario; pinned `--session-id` runs are never pruned); `--dry-run` | the machine-global runs root has grown and you want space back |
110
111
  | `rehash <dir/>` | Migrate cassette fingerprints to the current format version when the content is provably unchanged (`--dry-run`); no re-record needed | a cassette-format bump flagged committed fixtures as stale |
111
112
  | `vm <init\|status\|delete\|prune>` | Manage the L2 Apple-VZ / Lima microVM (`prune` removes orphaned VMs left by config/agent-version changes) | running `--fidelity microvm` |
112
113
 
@@ -207,7 +208,7 @@ This repo ships a **companion skill** (`.claude/skills/cowork-harness/`) that te
207
208
  /plugin install cowork-harness@cowork-harness
208
209
  ```
209
210
 
210
- The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.12.0` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
211
+ The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.13.0` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
211
212
 
212
213
  It also follows the open [Agent Skills](https://github.com/vercel-labs/skills) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) by pointing the `npx skills` CLI at `.claude/skills/cowork-harness` in this repo. (Working *inside* this repo, the skill auto-loads as a project skill — no install needed.)
213
214
 
@@ -473,7 +474,7 @@ Unit tests cover the scripted-answer logic, the egress allowlist matcher, the se
473
474
  - `COWORK_AGENT_IMAGE=<tag>` — override the agent image name (default `cowork-agent-base:2`); `COWORK_AGENT_BINARY=<path>` — override the auto-detected staged agent ELF.
474
475
  - `COWORK_SKIP_CAPABILITY_PROBE=1` — skip the per-run capability probe (the harness otherwise probes the agent image/VM for the document/OCR/Office capabilities the real Cowork rootfs ships and **fails a run that uses one the image omits** — a likely false negative; suppress per-scenario with `allow_missing_capability: true`, or rebuild full parity).
475
476
  - `COWORK_HARNESS_DECIDER_DIR_POLL_MS` / `_TIMEOUT_MS` — tune the `--decider-dir` rendezvous poll/backstop; `COWORK_HARNESS_DECIDER_CMD_TIMEOUT_MS` / `COWORK_HARNESS_LLM_TIMEOUT_MS` — backstop a hung `--decider-cmd` helper / `--decider-llm` model call (default 600 s, fail loud); `COWORK_HARNESS_DIALOG_TIMEOUT_MS` — override the 6 s dialog auto-cancel.
476
- - `COWORK_HARNESS_RUNS_DIR` (or the `--run-dir <path>` flag) — override the default run-output root `~/.cowork-harness/runs` (kept out of any working tree so sensitive skill inputs/outputs don't land in a repo). Precedence: `--run-dir` > env > default. The root is flat and machine-global (shared across projects); pinned `--session-id` runs are guarded against cross-project overwrite, and `runs gc` never prunes them. In CI, set it to a workspace path (e.g. `runs`) so artifact upload can collect the runs. `COWORK_HARNESS_ALLOW_FOREIGN_RESUME=1` overrides the guard that blocks `--resume` onto another project's pinned session.
477
+ - `COWORK_HARNESS_RUNS_DIR` (or the `--run-dir <path>` flag) — override the default run-output root `~/.cowork-harness/runs` (kept out of any working tree so sensitive skill inputs/outputs don't land in a repo). Precedence: `--run-dir` > env > default. The root is flat and machine-global (shared across projects); pinned `--session-id` runs are guarded against cross-project overwrite, and `prune` never prunes them. In CI, set it to a workspace path (e.g. `runs`) so artifact upload can collect the runs. `COWORK_HARNESS_ALLOW_FOREIGN_RESUME=1` overrides the guard that blocks `--resume` onto another project's pinned session.
477
478
  - **Networking / loop:** `COWORK_EGRESS_PROXY` overrides the egress-proxy URL injected into the sandbox; `COWORK_PROXY_IMAGE` overrides the egress proxy Docker image name (default `cowork-egress-proxy:1`); `COWORK_DOCKER_NETWORK` pins the Docker network the agent container joins; `CLAUDE_FORCE_HOST_LOOP=1` forces the host-loop path regardless of the baseline's loop decision (the `cowork` tier's auto-pick). `COWORK_LIMACTL` overrides the `limactl` binary path (default `/opt/homebrew/bin/limactl`).
478
479
  - **Strictness escape hatches** (the harness fails loud by default): `COWORK_HARNESS_SOFT_MISSING=1` downgrades a missing mount source from a hard error to warn-and-exclude; `COWORK_HARNESS_ALLOW_CONFIG_DIR_WRITE=1` permits writing into an existing pinned `plugins.config_dir` (otherwise refused, to avoid clobbering a real Claude config).
479
480
  - **Staleness boundary:** the cassette-staleness skill hash uses the **git-tracked** file set of each skill/plugin source dir by default (a non-repo dir falls back to a raw walk; OS-junk like `.DS_Store` is always excluded). `COWORK_HARNESS_GITSET=0` opts out to the legacy raw walk for every dir. `COWORK_HARNESS_DEBUG_SKILLHASH=1` dumps the exact file set feeding the hash on a staleness mismatch (and flags OS-junk) so a drift source is one line. Declare per-plugin non-runtime paths in a `.cowork-hashignore` file (or the session `staleness.hash_ignore`). `COWORK_HARNESS_AGENT_SCOPE=skill` (opt-in) refines a scenario's `skills:` scope so a **skill-named** `agents/<name>.md` re-stales only that skill's cassettes instead of the whole fleet (generic agents stay shared; stamped into the fingerprint, so flipping it is a one-time re-record like `GITSET`).
package/dist/cli.js CHANGED
@@ -1,6 +1,6 @@
1
1
  #!/usr/bin/env node
2
2
  import { readFileSync, writeFileSync, mkdirSync, readdirSync, writeSync, existsSync } from "node:fs";
3
- import { join, basename, resolve, isAbsolute } from "node:path";
3
+ import { join, basename, resolve, isAbsolute, dirname } from "node:path";
4
4
  import { fileURLToPath } from "node:url";
5
5
  import { parse as parseYaml } from "yaml";
6
6
  import { Scenario, AnswerRule, Assertion } from "./types.js";
@@ -24,6 +24,7 @@ import { loadDotenv } from "./dotenv.js";
24
24
  import { makeRenderer, renderStart, renderFooter, startHeartbeat } from "./run/renderer.js";
25
25
  import { resolveEventsFile, buildTrace, formatTrace, buildGateTrace, formatGateTrace, buildDispatchTree, formatDispatchTree, noteRunsLocation, } from "./run/trace-view.js";
26
26
  import { buildScaffold } from "./run/scaffold.js";
27
+ import { buildInspectView } from "./run/inspect-view.js";
27
28
  import { pkgVersion, jsonEnvelope, jsonError, parseOutputFormat } from "./run/envelope.js";
28
29
  import { computeVerdict } from "./run/verdict.js";
29
30
  import { evaluate, hostMatches } from "./assert.js";
@@ -77,7 +78,7 @@ const HELP = `cowork-harness <command> (v${"$VERSION"})
77
78
  [--skip-privacy|--skip-staleness] skip one check
78
79
  [--allow <regex>]... [--allow-domain/-email <regex>]... [--allow-file <path>]... [--output-format json]
79
80
  rehash <dir/> migrate cassette fingerprints to current version when content is provably unchanged (requires contentSig from v3+)
80
- runs gc [--keep-last <n>] prune accumulated run dirs, keeping N most recent per scenario (default: 5)
81
+ prune [--keep-last <n>] prune accumulated run dirs, keeping N most recent per scenario (default: 5)
81
82
 
82
83
  ── CI lint + assertion reference ──────────────────────────────────────────────
83
84
  lint <scenario.yaml | dir/>… check scenarios for silent false-greens (bundled scenario.py; needs python3 — PyYAML is bundled)
@@ -92,6 +93,8 @@ const HELP = `cowork-harness <command> (v${"$VERSION"})
92
93
  [--output-format json] structured rows
93
94
  verify-run <run-dir> <scenario.yaml> re-evaluate assert: against a kept run dir (no live agent, ~1s)
94
95
  [--output-format json]
96
+ inspect <run-id | run-dir> show what a run produced: artifacts + a shallow field preview of each JSON artifact
97
+ [--output-format json] structured digest
95
98
  scaffold <run-id | run-dir> turn a kept run into a starter scenario YAML (gates→answers, artifacts→file_exists)
96
99
  [--out <file.yaml>] write to a file (default: stdout)
97
100
 
@@ -261,16 +264,17 @@ const SUBCOMMAND_USAGE = {
261
264
  " NOTE: --allow-failing only relaxes the post-run VERDICT gate; it does NOT salvage an unanswered gate (that throws before any cassette is written — use --on-unanswered first / a decider).",
262
265
  replay: "usage: replay <file.cassette.json | dir/> [--strict] [--output-format text|json]",
263
266
  "verify-cassettes": "usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-file <path>]... [--output-format json]",
264
- trace: "usage: trace <run-id | run-dir | events.jsonl> [--view tools|questions|dispatches] [--output-format json]\n --view tools tool call / result rows\n --view questions gate lifecycle (question → answer → delivered)\n --view dispatches sub-agent dispatch tree + dispatch_count_max\n (default: all views)",
267
+ trace: "usage: trace <run-id | run-dir | events.jsonl> [--view tools|questions|dispatches] [--output-format json]\n --view tools tool call / result rows\n --view questions gate lifecycle (question → answer → delivered)\n --view dispatches sub-agent dispatch tree + dispatch_count_max\n (default: all views)\n (for what the run PRODUCED — artifacts — use `inspect`)",
265
268
  assertions: "usage: assertions --list [--output-format json]",
266
269
  scaffold: "usage: scaffold <run-id | run-dir> [--out <file.yaml>]\n Turns a kept run into a starter scenario YAML (gates→answers, artifacts→file_exists).\n Positional <run-id | run-dir> is the canonical form.",
267
270
  decide: 'usage: decide [--question <q>] [--option <o>]... [--decider-cmd <cmd> | --decider-llm] [--answer "<q>=<label>"]... [--answer-policy <p>] [--intent <s>] [--output-format json]',
268
271
  gates: "usage: gates <dir> [--follow] (stream pending in-band gates as JSON lines; pair with --decider-dir)",
269
272
  answer: 'usage: answer <dir> --gate <N> (--choose <label> [--choose <label>…] | --answer "<q>=<label>") (write an in-band gate reply atomically; repeat --choose for a multiSelect gate)',
270
273
  "verify-run": "usage: verify-run <run-dir> <scenario.yaml> [--output-format json] (re-evaluate a scenario's assert: against a kept run dir; no live agent)",
274
+ inspect: "usage: inspect <run-id | run-dir> [--output-format json] (show what a run PRODUCED: artifacts + a shallow preview of each JSON artifact's fields; for what HAPPENED — tools/decisions — use `trace`)",
271
275
  doctor: "usage: doctor [--tier protocol|container|microvm|hostloop|cowork] [--output-format json] (read-only prerequisite check)",
272
276
  rehash: "usage: rehash <dir/> [--dry-run] [--output-format text|json] (migrate cassettes across format bumps using contentSig verification; no re-record needed)",
273
- runs: "usage: runs gc [--keep-last <n>] [--dry-run] [<runs-dir>] (prune accumulated run dirs; default --keep-last 5)",
277
+ prune: "usage: prune [--keep-last <n>] [--dry-run] [<runs-dir>] (prune accumulated run dirs; default --keep-last 5)",
274
278
  };
275
279
  // Known subcommands — used by the global value-flag parsers (`--dotenv`, `--run-dir`) to reject a command
276
280
  // name mistaken as the flag's value (`--dotenv run x.yaml` would otherwise swallow `run`).
@@ -283,6 +287,7 @@ const COMMANDS = [
283
287
  "verify-cassettes",
284
288
  "verify-run",
285
289
  "trace",
290
+ "inspect",
286
291
  "assertions",
287
292
  "scaffold",
288
293
  "decide",
@@ -295,7 +300,7 @@ const COMMANDS = [
295
300
  "lint",
296
301
  "doctor",
297
302
  "rehash",
298
- "runs",
303
+ "prune",
299
304
  ];
300
305
  async function main() {
301
306
  const argv = process.argv.slice(2);
@@ -414,6 +419,8 @@ async function main() {
414
419
  return cmdVerifyRun(rest);
415
420
  case "trace":
416
421
  return cmdTrace(rest);
422
+ case "inspect":
423
+ return cmdInspect(rest);
417
424
  case "assertions":
418
425
  return cmdAssert(rest);
419
426
  case "scaffold":
@@ -424,13 +431,8 @@ async function main() {
424
431
  return cmdGates(rest);
425
432
  case "answer":
426
433
  return cmdAnswer(rest);
427
- case "runs": {
428
- const sub = rest[0];
429
- if (sub === "gc")
430
- return cmdRunsGc(rest.slice(1));
431
- log(`runs: unknown subcommand "${sub ?? ""}" — available: gc`);
432
- return process.exit(2);
433
- }
434
+ case "prune":
435
+ return cmdRunsGc(rest); // top-level: no `gc` token to strip, so pass `rest` whole (NOT rest.slice(1))
434
436
  default:
435
437
  log(`unknown command: ${cmd}\n`);
436
438
  printHelp();
@@ -656,7 +658,7 @@ function resolveExternal(command, flags) {
656
658
  }
657
659
  return flags.deciderCmd != null ? spawnChannel(flags.deciderCmd) : undefined;
658
660
  }
659
- /** The single error exit used by commands + the top-level catch. Every category → exit 2. */
661
+ /** The single error exit used by commands + the top-level catch. boundary → exit 3, every other category → exit 2. */
660
662
  function fail(command, category, message, hint, json) {
661
663
  if (json)
662
664
  out(jsonError(command, category, message, hint));
@@ -1828,6 +1830,14 @@ async function cmdVerifyRun(args) {
1828
1830
  log(`verify-run: cannot read ${resultPath}: ${e.message}`);
1829
1831
  return process.exit(2);
1830
1832
  }
1833
+ // A partial run did NOT complete (it exited on an unanswered gate). Its assertion outcome is empty and its
1834
+ // artifacts are pre-failure, so re-evaluating asserts against it would vouch for a run that never finished.
1835
+ // Refuse rather than false-fail or false-pass.
1836
+ if (result.partial) {
1837
+ log(`verify-run: ${runDir} is a PARTIAL run — it did not complete (exited on an unanswered gate). ` +
1838
+ `Re-run to completion before verifying. (can't verify ⇒ not green)`);
1839
+ return process.exit(2);
1840
+ }
1831
1841
  let scenario;
1832
1842
  try {
1833
1843
  scenario = parseScenarioFile(scenarioFile);
@@ -2086,6 +2096,32 @@ function cmdTrace(args) {
2086
2096
  else
2087
2097
  out(formatTrace(rows));
2088
2098
  }
2099
+ function cmdInspect(args) {
2100
+ if (hasHelp(args))
2101
+ return void log(SUBCOMMAND_USAGE.inspect);
2102
+ ensureOutputFormat("inspect", args);
2103
+ const json = isJsonOutput(args);
2104
+ rejectUnknownFlags("inspect", args, ["--output-format", "--output-format=json", "--output-format=text"], json);
2105
+ const allPositionals = positionals(args, ["--output-format"]);
2106
+ if (allPositionals.length !== 1)
2107
+ return void fail("inspect", "usage", SUBCOMMAND_USAGE.inspect, undefined, json);
2108
+ // Resolve a run-id or run-dir to its dir. A run dir already holding result.json is used directly; otherwise
2109
+ // reuse trace's resolver (run-id → events.jsonl) and take the parent.
2110
+ const target = allPositionals[0];
2111
+ let runDir;
2112
+ try {
2113
+ runDir = existsSync(join(target, "result.json")) ? target : dirname(resolveEventsFile(target));
2114
+ }
2115
+ catch (e) {
2116
+ return void fail("inspect", "usage", String(e.message), undefined, json);
2117
+ }
2118
+ try {
2119
+ out(buildInspectView(runDir, { json }));
2120
+ }
2121
+ catch (e) {
2122
+ return void fail("inspect", "usage", String(e.message), undefined, json);
2123
+ }
2124
+ }
2089
2125
  function diff(a, b, path) {
2090
2126
  const keys = new Set([...Object.keys(a ?? {}), ...Object.keys(b ?? {})]);
2091
2127
  for (const k of keys) {
@@ -256,8 +256,9 @@ export class FirstOptionDecider {
256
256
  return ABSTAIN;
257
257
  }
258
258
  }
259
- /** Strict label match: exact → case-insensitive. NULL on no match (caller fails loud — we deliberately
260
- * do NOT fall back to option 1 like `coerceLabel`, which would silently mis-answer).
259
+ /** Strict label match: exact → case-insensitive → near-miss (surrounding quotes / trailing sentence
260
+ * punctuation trimmed). NULL on no match (caller fails loud — we deliberately do NOT fall back to option 1
261
+ * like `coerceLabel`, which would silently mis-answer).
261
262
  * When `fuzzy` is true, also accept a substring match (ONLY when exactly one label appears in the reply
262
263
  * to avoid ambiguity). Callers that consume human or LLM output should leave `fuzzy` at its default
263
264
  * (false) to require an exact answer — the substring heuristic is opt-in only. */
@@ -266,6 +267,19 @@ export function matchLabel(raw, labels, fuzzy = false) {
266
267
  const exact = labels.find((l) => l === r) ?? labels.find((l) => l.toLowerCase() === r.toLowerCase());
267
268
  if (exact)
268
269
  return exact;
270
+ // Near-miss tier: a model (or human) often replies `Confirmed.` or `"Confirmed"` for the label
271
+ // `Confirmed`, especially on binary confirm gates. Strip surrounding quotes + trailing `.!,;`/whitespace
272
+ // and re-run the exact/ci match. NEVER strip `:` — it is the `OTHER:` free-text sentinel callers rely on,
273
+ // and the trim must not let a near-miss swallow it. Strict (not fuzzy), so it can't mis-bind a substring.
274
+ const norm = r
275
+ .replace(/^["'`]+/, "")
276
+ .replace(/["'`]+$/, "")
277
+ .replace(/[.!,;\s]+$/, "");
278
+ if (norm && norm !== r) {
279
+ const near = labels.find((l) => l === norm) ?? labels.find((l) => l.toLowerCase() === norm.toLowerCase());
280
+ if (near)
281
+ return near;
282
+ }
269
283
  if (!fuzzy)
270
284
  return null;
271
285
  // fuzzy=true: the substring tier fires ONLY when EXACTLY ONE label is contained in the reply. With
@@ -351,8 +365,10 @@ export class LlmDecider {
351
365
  continue;
352
366
  }
353
367
  const raw = await this.complete(this.prompt(text, q.options, ctx), this.model);
354
- // Match a LABEL first (matchLabel is fuzzy=false): keeps the common path unchanged and means a real
355
- // option literally named "OTHER: …" is selected as a label, not hijacked by the free-text branch.
368
+ // Match a LABEL first (matchLabel is fuzzy=false, but tolerates a quoted / trailing-punctuation
369
+ // near-miss): keeps the common path unchanged and means a real option literally named "OTHER: …" is
370
+ // selected as a label (exact match wins; the trim never strips `:`), not hijacked by the free-text
371
+ // branch below.
356
372
  const pick = matchLabel(raw, labels);
357
373
  if (pick) {
358
374
  process.stderr.write(`[llm-decider] "${text}" → "${pick}"${this.intent ? ` (intent: ${this.intent})` : ""}\n`);
@@ -174,7 +174,11 @@ export function runDoctorChecks(tier, probe = realProbe) {
174
174
  id: "image",
175
175
  title: "Agent image",
176
176
  status: present ? "ok" : up ? "fail" : "skip",
177
- detail: present ? `${image} present` : up ? `${image} missing` : `(skipped — ${runtime} not reachable)`,
177
+ detail: present
178
+ ? `${image} present — lean core; OCR / PDF-table skills need the full-parity image (--build-arg COWORK_FULL_PARITY=1)`
179
+ : up
180
+ ? `${image} missing`
181
+ : `(skipped — ${runtime} not reachable)`,
178
182
  remedy: present || !up ? undefined : `build it: ${agentBuildLine(runtime, image)}`,
179
183
  required: up, // only gate on the image once the runtime is actually reachable
180
184
  });
@@ -17,7 +17,7 @@ import { spawnProtocol } from "../runtime/protocol.js";
17
17
  import { spawnContainer } from "../runtime/container.js";
18
18
  import { spawnHostLoop } from "../runtime/hostloop.js";
19
19
  import { spawnMicroVm } from "../runtime/microvm.js";
20
- import { probeImageOmitted, probeMicrovmOmitted, detectCapabilityUse, CAPABILITY_FAMILIES } from "../runtime/image-capabilities.js";
20
+ import { probeImageOmitted, probeMicrovmOmitted, detectCapabilityUse, capabilityPreflightDecision, CAPABILITY_FAMILIES, } from "../runtime/image-capabilities.js";
21
21
  import { instanceName } from "../runtime/lima.js";
22
22
  import { decideLoopFromBaseline, readGateFlag } from "../loop-decision.js";
23
23
  import { startEgressSidecar, registerCleanup } from "../egress/sidecar.js";
@@ -258,6 +258,7 @@ export async function executeScenario(scenario, opts = {}) {
258
258
  let hostProxy;
259
259
  let microvmProxyPort;
260
260
  let record;
261
+ let unansweredErr; // set when a gate whiffs — drives the salvage branch below
261
262
  let child; // hoisted so the finally can reap a crashed/orphaned container (F1)
262
263
  let containerName;
263
264
  let deregisterContainerReap; // 2a: Ctrl-C cleanup for the agent container
@@ -270,6 +271,29 @@ export async function executeScenario(scenario, opts = {}) {
270
271
  const viaApiOn = readGateFlag(baseline, "1978029737", "coworkWebFetchViaApi");
271
272
  const promptGateOn = readGateFlag(baseline, "1978029737", "coworkWebFetchPrompt");
272
273
  const provenanceRef = {};
274
+ // Pre-flight: if the skill DECLARES required capabilities and the image provably omits one, FAIL FAST here
275
+ // — before any paid agent run — instead of burning ~12 min to reach a verdict the post-run guard already
276
+ // knows. The author can opt out with `allow_missing_capability: true` (the fallback is equivalent), which
277
+ // downgrades to a notice. The image probe is digest-cached, so it's shared with the post-run check (no
278
+ // second container spawn), and it spawns a throwaway `--network none` container with no model — zero tokens.
279
+ const declaredCaps = scenario.requires_capabilities ?? [];
280
+ if (declaredCaps.length &&
281
+ (effectiveFidelity === "container" || effectiveFidelity === "hostloop" || effectiveFidelity === "microvm") &&
282
+ process.env.COWORK_SKIP_CAPABILITY_PROBE !== "1") {
283
+ const omitted = effectiveFidelity === "microvm"
284
+ ? probeMicrovmOmitted(instanceName(baseline))
285
+ : probeImageOmitted({
286
+ runtime: process.env.COWORK_CONTAINER_RUNTIME ?? "docker",
287
+ image: process.env.COWORK_AGENT_IMAGE ?? "cowork-agent-base:2",
288
+ tier: effectiveFidelity,
289
+ });
290
+ const allowMissing = scenario.assert.some((a) => a.allow_missing_capability === true);
291
+ const { abort, message } = capabilityPreflightDecision(declaredCaps, omitted, allowMissing);
292
+ if (abort)
293
+ throw new BoundaryError(`[capability] ${message}`);
294
+ if (message)
295
+ warn(`::notice:: [capability] (pre-flight) ${message} (allow_missing_capability asserted — proceeding)\n`);
296
+ }
273
297
  try {
274
298
  // acquire the egress sidecar / host proxy INSIDE the protected try so a throw in resource
275
299
  // acquisition OR in renderPrompts below can't leak a Docker network / a bound proxy port — the `finally`
@@ -365,10 +389,22 @@ export async function executeScenario(scenario, opts = {}) {
365
389
  permissiveMode: plan.permissionMode === "bypassPermissions",
366
390
  };
367
391
  }
368
- record = await run.drive(scenario.prompt, {
369
- subagentAppend: prompts.subagentAppend,
370
- sdkMcp,
371
- });
392
+ try {
393
+ record = await run.drive(scenario.prompt, {
394
+ subagentAppend: prompts.subagentAppend,
395
+ sdkMcp,
396
+ });
397
+ }
398
+ catch (e) {
399
+ // An unanswered gate is recoverable: grab the in-progress record so the work done before the whiff can
400
+ // be salvaged to disk below. Any other error is a genuine fault — keep today's fail-fast behavior.
401
+ if (e instanceof UnansweredError) {
402
+ unansweredErr = e;
403
+ record = run.partial();
404
+ }
405
+ else
406
+ throw e;
407
+ }
372
408
  }
373
409
  finally {
374
410
  // Reap the agent container FIRST (before the sidecar networks), so a crashed/unanswered run can't
@@ -402,6 +438,36 @@ export async function executeScenario(scenario, opts = {}) {
402
438
  // are read-only inputs and are NOT visible roots. Persisted to RunResult so the plan-less lanes
403
439
  // (verify reads result.json; replay reads the cassette) match this without rebuilding a LaunchPlan.
404
440
  const userVisibleRoots = ["outputs", ...plan.mounts.filter((m) => m.kind === "folder").map((m) => m.mountPath)];
441
+ // Salvage path: the run exited on an unanswered gate. Persist a PARTIAL result.json (+ run.jsonl/trace) so
442
+ // the artifacts the agent wrote before the whiff survive for inspection, then re-throw so the CLI still
443
+ // exits 2. Skip assertion eval and the capability probe (a real container spawn) — a partial run has no
444
+ // meaningful assertion or verdict outcome.
445
+ if (unansweredErr) {
446
+ const partialResult = buildPartialResult({
447
+ scenarioName: scenario.name,
448
+ prompt: scenario.prompt,
449
+ fidelity: scenario.fidelity,
450
+ baseline: baseline.appVersion,
451
+ record,
452
+ outDir,
453
+ workRoot,
454
+ userVisibleRoots,
455
+ effectiveFidelity,
456
+ egress,
457
+ durationMs: Date.now() - startedAt,
458
+ unanswered: { message: unansweredErr.message, hint: unansweredErr.hint },
459
+ fingerprint: buildFingerprint(scenario.session, baseline.appVersion, undefined, scenario.skills),
460
+ });
461
+ writeFileSync(join(outDir, "result.json"), scrub(JSON.stringify(partialResult, null, 2), secrets));
462
+ writeRunJsonl(outDir, scenario, effectiveFidelity, record, egress, secrets);
463
+ writeTrace(outDir, record, egress, secrets, partialResult.durationMs);
464
+ scrubFileInPlace(join(outDir, "events.jsonl"), secrets);
465
+ scrubFileInPlace(join(outDir, "control-out.jsonl"), secrets);
466
+ // Loud PARTIAL marker so the populated artifacts are never misread as success (the no-false-green rule).
467
+ warn(`::notice:: [partial] run did NOT complete (unanswered gate) — salvaged the pre-failure work to:\n` +
468
+ ` ${outDir}\n inspect it: cowork-harness inspect ${outDir}\n`);
469
+ throw unansweredErr;
470
+ }
405
471
  const assertions = evaluate(scenario.assert, {
406
472
  transcript: record.transcript,
407
473
  toolsCalled: record.toolsCalled,
@@ -632,6 +698,51 @@ function writeRunJsonl(outDir, scenario, fidelity, rec, egress, secrets) {
632
698
  ];
633
699
  writeFileSync(join(outDir, "run.jsonl"), scrub(lines.map((l) => JSON.stringify(l)).join("\n"), secrets));
634
700
  }
701
+ /** Assemble a RunResult for a run that did NOT complete — it exited on an unanswered gate. The work the
702
+ * agent did before the whiff (artifacts on disk, the partial transcript, decisions/tool counts so far) is
703
+ * salvaged so the paid run is still inspectable instead of vanishing. Deliberately reduced: no assertion
704
+ * outcome (a partial run has none) and no capability/verdict fields (those would need a probe we skip).
705
+ * `partial:true` is the signal that lets consumers (verify-run, scaffold, the footer) refuse to read its
706
+ * populated `artifacts[]` as a passing run. */
707
+ export function buildPartialResult(args) {
708
+ const { record } = args;
709
+ return {
710
+ $schema: RUN_RESULT_SCHEMA_URL,
711
+ generator: "cowork-harness",
712
+ scenario: args.scenarioName,
713
+ prompt: args.prompt,
714
+ fidelity: args.fidelity,
715
+ baseline: args.baseline,
716
+ result: "error",
717
+ partial: true,
718
+ unansweredGate: { message: args.unanswered.message, ...(args.unanswered.hint ? { hint: args.unanswered.hint } : {}) },
719
+ decisions: record.decisions.map((d) => ({
720
+ kind: d.kind,
721
+ name: d.name,
722
+ decision: d.decision,
723
+ by: d.by,
724
+ detail: d.detail,
725
+ rationale: d.rationale,
726
+ })),
727
+ toolCounts: record.toolCounts,
728
+ gateDeliveries: record.gateDeliveries,
729
+ egress: args.egress,
730
+ assertions: [],
731
+ toolResults: record.toolResults,
732
+ subagents: record.subagents,
733
+ unanswered: record.unanswered,
734
+ usage: record.usage,
735
+ cost: record.cost,
736
+ durationMs: args.durationMs,
737
+ outDir: args.outDir,
738
+ workDir: args.workRoot,
739
+ outputsDir: join(args.workRoot, "outputs"),
740
+ userVisibleRoots: args.userVisibleRoots,
741
+ artifacts: collectArtifacts(args.workRoot, args.userVisibleRoots),
742
+ effectiveFidelity: args.effectiveFidelity,
743
+ ...(args.fingerprint ? { fingerprint: args.fingerprint } : {}),
744
+ };
745
+ }
635
746
  /** B3: the structured run trace. */
636
747
  /** ENV-MANIFEST: recursively list files under each user-visible prefix (relative path + byte size).
637
748
  * Paths only — NO content snapshot (that is the cassette manifest, #1).
@@ -0,0 +1,95 @@
1
+ import { readFileSync, existsSync } from "node:fs";
2
+ import { join } from "node:path";
3
+ /** A compact, depth-1 preview of a JSON artifact: scalars kept inline, arrays shown as a count, nested
4
+ * objects collapsed to `{…}` — enough to answer "did it produce the right fields?" without dumping blobs. */
5
+ function shallowPreview(doc) {
6
+ if (Array.isArray(doc))
7
+ return `[${doc.length} items]`;
8
+ if (!doc || typeof doc !== "object")
9
+ return { value: doc };
10
+ const out = {};
11
+ for (const [k, v] of Object.entries(doc)) {
12
+ if (Array.isArray(v))
13
+ out[k] = `[${v.length} items]`;
14
+ else if (v && typeof v === "object")
15
+ out[k] = "{…}";
16
+ else
17
+ out[k] = v;
18
+ }
19
+ return out;
20
+ }
21
+ function humanMs(ms) {
22
+ if (!ms && ms !== 0)
23
+ return "?";
24
+ if (ms < 1000)
25
+ return `${ms}ms`;
26
+ return `${(ms / 1000).toFixed(1)}s`;
27
+ }
28
+ /** Build the digest for one kept run dir: header facts + the artifacts manifest, with a shallow field
29
+ * preview for each JSON artifact (read from the work dir, which result.json records). The manifest
30
+ * (paths + sizes) lives in result.json and always survives; the content preview needs the work dir, which
31
+ * a non-`--keep` container/microvm run tears down. */
32
+ function digestFor(runDir) {
33
+ const resultPath = join(runDir, "result.json");
34
+ if (!existsSync(resultPath))
35
+ throw new Error(`no result.json under ${runDir} (is this a kept run dir? e.g. runs/<scenario>/<sessionId>/)`);
36
+ const result = JSON.parse(readFileSync(resultPath, "utf8"));
37
+ const workDir = result.workDir ?? "";
38
+ const workDirAvailable = !!workDir && existsSync(workDir);
39
+ const artifacts = (result.artifacts ?? []).map((a) => {
40
+ const entry = { path: a.path, bytes: a.bytes };
41
+ if (workDirAvailable && a.path.endsWith(".json")) {
42
+ const abs = join(workDir, a.path);
43
+ if (existsSync(abs)) {
44
+ try {
45
+ entry.preview = shallowPreview(JSON.parse(readFileSync(abs, "utf8")));
46
+ }
47
+ catch {
48
+ /* not valid JSON after all — leave preview off */
49
+ }
50
+ }
51
+ }
52
+ return entry;
53
+ });
54
+ return {
55
+ scenario: result.scenario,
56
+ fidelity: result.fidelity,
57
+ result: result.result,
58
+ ...(result.partial ? { partial: true } : {}),
59
+ ...(result.unansweredGate ? { unansweredGate: result.unansweredGate } : {}),
60
+ durationMs: result.durationMs,
61
+ ...(result.cost ? { cost: result.cost } : {}),
62
+ workDirAvailable,
63
+ artifacts,
64
+ };
65
+ }
66
+ /** Render a run dir for the `inspect` command. `json` mode returns the structured digest; text mode returns
67
+ * a human summary: header, a PARTIAL banner if the run didn't complete, and each artifact with a shallow
68
+ * field preview (or a "work dir torn down" note when content can't be read). */
69
+ export function buildInspectView(runDir, opts = {}) {
70
+ const d = digestFor(runDir);
71
+ if (opts.json)
72
+ return JSON.stringify(d, null, 2);
73
+ const lines = [];
74
+ lines.push(`run: ${d.scenario} (${d.fidelity}) result: ${d.result} ${humanMs(d.durationMs)}`);
75
+ if (d.partial) {
76
+ lines.push(`⚠ PARTIAL — this run did NOT complete (exited on an unanswered gate); artifacts below are pre-failure.`);
77
+ if (d.unansweredGate)
78
+ lines.push(` gate: ${d.unansweredGate.message.split("\n")[0]}`);
79
+ }
80
+ lines.push(`artifacts (${d.artifacts.length}):`);
81
+ for (const a of d.artifacts) {
82
+ lines.push(` ${a.path} (${a.bytes} B)`);
83
+ if (a.preview && typeof a.preview === "object") {
84
+ for (const [k, v] of Object.entries(a.preview))
85
+ lines.push(` ${k}: ${typeof v === "string" ? v : JSON.stringify(v)}`);
86
+ }
87
+ else if (typeof a.preview === "string") {
88
+ lines.push(` ${a.preview}`);
89
+ }
90
+ }
91
+ if (!d.workDirAvailable && d.artifacts.some((a) => a.path.endsWith(".json"))) {
92
+ lines.push(` (work dir torn down — re-run with --keep to preview artifact contents)`);
93
+ }
94
+ return lines.join("\n");
95
+ }
package/dist/run/run.js CHANGED
@@ -73,6 +73,14 @@ export class Run {
73
73
  ctx() {
74
74
  return { task: this.rec.transcript, transcript: () => this.rec.transcript, toolLog: () => this.toolLog, runId: this.rec.runId };
75
75
  }
76
+ /** The in-progress record. When `drive()` throws on an unanswered gate, this holds everything accumulated
77
+ * up to the whiff (transcript, decisions, tool counts, artifacts-on-disk are separate) — the caller uses
78
+ * it to salvage a partial run instead of discarding the work. Fully initialized in the constructor, so
79
+ * it's a usable shell even on a very-early throw. The post-loop gate-delivery pairing has NOT run on the
80
+ * throw path, so `gateDeliveries` is empty here. */
81
+ partial() {
82
+ return this.rec;
83
+ }
76
84
  /** Drive one-shot (string) or multi-turn (async iterable) and return the record. */
77
85
  async drive(turns, startOpts) {
78
86
  const turnIter = typeof turns === "string" ? oneShot(turns) : turns[Symbol.asyncIterator]();
@@ -10,16 +10,16 @@ const DEFAULT_KEEP_LAST = 5;
10
10
  * dir has neither → it is what GC should drop first. `events.jsonl` exists from session start, so an
11
11
  * in-flight run is protected without a wall-clock guard. */
12
12
  const isRealRun = (dir) => existsSync(join(dir, "result.json")) || existsSync(join(dir, "events.jsonl"));
13
- /** `cowork-harness runs gc [--keep-last <n>] [--dry-run] [<runs-dir>]`
13
+ /** `cowork-harness prune [--keep-last <n>] [--dry-run] [<runs-dir>]`
14
14
  *
15
15
  * For each scenario directory under the runs root, ranks EPHEMERAL run dirs by (1) real-run first (has
16
16
  * result.json OR events.jsonl), (2) mtime descending, (3) name — then keeps the N most recent of that order
17
17
  * and removes the rest. So an older COMPLETED run beats a newer empty scaffold dir for a keep slot, but
18
18
  * `--keep-last` stays a HARD CAP (the ranking only decides WHICH N survive — never grows the kept count).
19
- * Do NOT run `runs gc` against an actively-writing runs root.
19
+ * Do NOT run `prune` against an actively-writing runs root.
20
20
  * Pinned `sess-*` dirs (persisted, resumable `--session-id` sessions) are retained unconditionally.
21
21
  * The default root is the flat, machine-global `~/.cowork-harness/runs` (shared across projects), so a
22
- * bare `runs gc` prunes ephemeral runs from ALL projects; pass an explicit <runs-dir> to scope it.
22
+ * bare `prune` prunes ephemeral runs from ALL projects; pass an explicit <runs-dir> to scope it.
23
23
  * Safe by default (dry-run-able). */
24
24
  export function cmdRunsGc(args) {
25
25
  let p;
@@ -34,19 +34,19 @@ export function cmdRunsGc(args) {
34
34
  return process.exit(2);
35
35
  }
36
36
  if (p.positionals.length > 1) {
37
- log(`runs gc takes an optional <runs-dir> (got ${p.positionals.length}: ${p.positionals.join(", ")})`);
37
+ log(`prune takes an optional <runs-dir> (got ${p.positionals.length}: ${p.positionals.join(", ")})`);
38
38
  return process.exit(2);
39
39
  }
40
40
  const rawKeep = p.options["--keep-last"];
41
41
  const keepLast = rawKeep !== undefined ? Number(rawKeep) : DEFAULT_KEEP_LAST;
42
42
  if (!Number.isInteger(keepLast) || keepLast < 1) {
43
- log(`runs gc: --keep-last must be a positive integer (got ${rawKeep})`);
43
+ log(`prune: --keep-last must be a positive integer (got ${rawKeep})`);
44
44
  return process.exit(2);
45
45
  }
46
46
  const dryRun = p.flags["--dry-run"] ?? false;
47
47
  const runsRoot = p.positionals[0] ?? runsWriteRoot();
48
48
  if (!existsSync(runsRoot)) {
49
- log(`✓ runs gc: ${runsRoot} does not exist — nothing to prune`);
49
+ log(`✓ prune: ${runsRoot} does not exist — nothing to prune`);
50
50
  return process.exit(0);
51
51
  }
52
52
  let deleted = 0;
@@ -115,7 +115,7 @@ export function cmdRunsGc(args) {
115
115
  }
116
116
  }
117
117
  log(deleted > 0
118
- ? `✓ runs gc: pruned ${deleted} run dir(s), kept ${kept}${dryRun ? " (dry-run — nothing deleted)" : ""}`
119
- : `✓ runs gc: nothing to prune (${kept} run dir(s) within --keep-last ${keepLast})`);
118
+ ? `✓ prune: pruned ${deleted} run dir(s), kept ${kept}${dryRun ? " (dry-run — nothing deleted)" : ""}`
119
+ : `✓ prune: nothing to prune (${kept} run dir(s) within --keep-last ${keepLast})`);
120
120
  return process.exit(0);
121
121
  }
@@ -46,13 +46,20 @@ export function buildScaffold(eventsFile) {
46
46
  answers.push({ when_question: escapeRx(q), choose: val });
47
47
  }
48
48
  }
49
+ // A partial run did NOT complete (it exited on an unanswered gate). Its artifacts are pre-failure and its
50
+ // result is "error", so neither should become an assertion the author trusts — that would scaffold a
51
+ // scenario asserting the half-finished output. Keep the gate answers (still worth locking) but drop the
52
+ // artifact/result asserts and warn loudly.
53
+ const partial = result.partial === true;
49
54
  // Observed artifacts → file_exists; sub-agent count → dispatch_count_max; final result.
50
55
  const assert = [];
51
- for (const art of result.artifacts ?? [])
52
- assert.push({ file_exists: art.path });
53
- if ((result.subagents ?? []).length)
54
- assert.push({ dispatch_count_max: (result.subagents ?? []).length });
55
- assert.push({ result: result.result ?? "success" });
56
+ if (!partial) {
57
+ for (const art of result.artifacts ?? [])
58
+ assert.push({ file_exists: art.path });
59
+ if ((result.subagents ?? []).length)
60
+ assert.push({ dispatch_count_max: (result.subagents ?? []).length });
61
+ assert.push({ result: result.result ?? "success" });
62
+ }
56
63
  const scenario = {
57
64
  // `result.baseline` is the recorded appVersion; a scenario pins `latest` (or a `desktop-<ver>`) — default
58
65
  // to latest and let the author pin if they need a specific release.
@@ -70,8 +77,13 @@ export function buildScaffold(eventsFile) {
70
77
  // Provenance signature — mirrors the cassette's `generator`/`cassetteVersion` fields. The scenario
71
78
  // schema is `additionalProperties:false`, so it rides as a YAML comment (survives parse + re-serialize)
72
79
  // rather than a structured field. Tool + version let a reader trace a committed scenario to its origin.
80
+ const partialMarker = partial
81
+ ? `# PARTIAL run — this run did NOT complete (it exited on an unanswered gate). The artifact and result\n` +
82
+ `# assertions were omitted (the outputs are pre-failure). Re-run to completion and re-scaffold before trusting.\n`
83
+ : "";
73
84
  return (`# generated by cowork-harness v${pkgVersion()} (scaffold) from ${runDir}\n` +
74
85
  `# REVIEW before committing: tighten the when_question regexes to stable fragments, prune asserts you don't need.\n` +
86
+ partialMarker +
75
87
  multiSelectMarker +
76
88
  stringify(scenario));
77
89
  }
@@ -9,7 +9,7 @@ import { parseMessage } from "../agent/session.js";
9
9
  * convention used by the VM work dir (`lima.ts` `VM_WORK_HOST`). FLAT (shared across all projects on the
10
10
  * machine): ephemeral `local_*` run dirs carry a unique hrtime id so they never path-collide; pinned
11
11
  * `sess-*` dirs are deterministic, so a cross-project collision is guarded in `execute.ts` (the writer
12
- * errors rather than overwriting another project's session) and `runs gc` never prunes them.
12
+ * errors rather than overwriting another project's session) and `prune` never prunes them.
13
13
  */
14
14
  export function defaultRunsHome() {
15
15
  return join(homedir(), ".cowork-harness", "runs");
@@ -210,6 +210,29 @@ function readWorkspaceScript(workRoot, rel) {
210
210
  * error (the module may simply be present, or the script may swallow the failure). Script reads are
211
211
  * read-only, containment-checked, and bounded; missing files are skipped.
212
212
  */
213
+ /** Pre-run check: a skill that DECLARES the capability families its core path needs, run against an image
214
+ * that OMITS one, will hard-fail only after the (paid, ~12-min) run. Surface the build remedy up front so
215
+ * the operator fixes it before paying. Returns null when nothing is declared, all caps are present, or the
216
+ * probe could not run (`omitted` null) — i.e. only warns on a definite, actionable mismatch. */
217
+ export function capabilityPreflightWarning(declared, omitted) {
218
+ if (!declared.length || !omitted)
219
+ return null;
220
+ const missing = declared.filter((c) => omitted.includes(c));
221
+ if (!missing.length)
222
+ return null;
223
+ // State the gap + remedy only; the caller frames the consequence (abort pre-run, or a notice when
224
+ // allow_missing_capability is asserted) — so this string stays accurate in both contexts.
225
+ return (`skill declares requires_capabilities [${missing.join(", ")}] but the agent image omits them — ` +
226
+ `rebuild full parity (--build-arg COWORK_FULL_PARITY=1) and point COWORK_AGENT_IMAGE at it.`);
227
+ }
228
+ /** The pre-run capability decision: a DEFINITE gap (a declared family the image omits) **fails fast** before
229
+ * the paid run, unless the scenario opted out with `allow_missing_capability: true` (then it's a notice and
230
+ * the run proceeds — the author has declared the fallback equivalent). No gap / indefinite probe → no
231
+ * action. Keeping this a pure function makes the fail-fast-by-default policy unit-testable. */
232
+ export function capabilityPreflightDecision(declared, omitted, allowMissing) {
233
+ const message = capabilityPreflightWarning(declared, omitted);
234
+ return { abort: message !== null && !allowMissing, message };
235
+ }
213
236
  export function detectCapabilityUse(eventsFile, omitted, workRoot) {
214
237
  if (!omitted.length)
215
238
  return [];
package/docs/scenario.md CHANGED
@@ -261,6 +261,13 @@ fires even when the skill's fallback masks the gap. The check is computed at run
261
261
  `verify-run`/`replay` honor the recorded outcome — a clean full-parity run records nothing and never
262
262
  false-fails later. Opt out with `allow_missing_capability: true` when the fallback is genuinely equivalent.
263
263
 
264
+ When `requires_capabilities` is declared, the harness probes the image **before** driving and, if a declared
265
+ family is omitted, **fails fast — it aborts the run (exit 3) before spending a single token**, instead of
266
+ running ~12 min to a post-run hard-fail that's already known. Rebuild full parity
267
+ (`--build-arg COWORK_FULL_PARITY=1`) and point `COWORK_AGENT_IMAGE` at it, or assert
268
+ `allow_missing_capability: true` (which downgrades the abort to a notice and proceeds, same as it opts out of
269
+ the post-run check).
270
+
264
271
  ```yaml
265
272
  requires_capabilities: [office_convert, pdf_tables] # fail unless the tier provides (and can verify) these
266
273
  ```
@@ -382,7 +389,8 @@ are) or need a real predicate over a skill's **structured JSON output**:
382
389
  full Python callable with autocomplete and `print(d)`, strictly richer than anything a YAML string can
383
390
  express. **If you're checking structured JSON content and already write Python, prefer the pytest lambda**
384
391
  (a YAML content-predicate would be equal power with worse tooling). Find an artifact's real field paths by
385
- running once with `--keep` and inspecting the JSON under the run's `…/mnt/artifacts/…`.
392
+ running once with `--keep`, then `cowork-harness inspect <run-dir>` (a shallow field preview of each JSON
393
+ artifact) or by reading the JSON under the run's `…/mnt/outputs/…` directly.
386
394
 
387
395
  ## Output
388
396
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "cowork-harness",
3
- "version": "0.12.0",
3
+ "version": "0.13.0",
4
4
  "description": "Scriptable, CI-friendly harness for Claude Cowork's runtime contract for testing skills across scenarios — same agent, mounts, egress allowlist, permission protocol, and sandbox limitations.",
5
5
  "license": "MIT",
6
6
  "type": "module",
@@ -49,6 +49,8 @@
49
49
  "workDir": { "type": "string" },
50
50
  "outputsDir": { "type": "string" },
51
51
  "artifacts": { "type": "array" },
52
+ "partial": { "type": "boolean" },
53
+ "unansweredGate": { "type": "object" },
52
54
  "nonDeterministic": { "type": "boolean" },
53
55
  "nonDeterministicTerminal": { "type": "boolean" },
54
56
  "permissiveAutoAllow": {