cowork-harness 0.12.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +55 -0
- package/README.md +4 -3
- package/dist/cli.js +49 -13
- package/dist/decide/decider.js +20 -4
- package/dist/run/doctor.js +5 -1
- package/dist/run/execute.js +116 -5
- package/dist/run/inspect-view.js +95 -0
- package/dist/run/run.js +8 -0
- package/dist/run/runs-gc.js +8 -8
- package/dist/run/scaffold.js +17 -5
- package/dist/run/trace-view.js +1 -1
- package/dist/runtime/image-capabilities.js +23 -0
- package/docs/scenario.md +9 -1
- package/package.json +1 -1
- package/schema/run-result.json +2 -0
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,61 @@ All notable changes to this project are documented here. The format is based on
|
|
|
4
4
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The project uses
|
|
5
5
|
[Semantic Versioning](https://semver.org/); pre-1.0 minor versions may include breaking changes.
|
|
6
6
|
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.13.0] — 2026-06-25
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- **`inspect <run-id | run-dir>` — see what a run produced.** Lists the run's artifacts and prints a shallow
|
|
14
|
+
field preview of each JSON artifact (scalars inline, arrays as a count, nested objects collapsed), so
|
|
15
|
+
"did it do the job?" is a first-class check instead of hand-parsing `…/mnt/outputs/...`. `--output-format
|
|
16
|
+
json` emits a structured digest. When the work dir was torn down (a non-`--keep` container/microvm run),
|
|
17
|
+
the artifact manifest still prints from `result.json` and the preview notes it's unavailable.
|
|
18
|
+
- **A whiffed gate no longer discards the paid run.** When a run exits on an unanswered gate
|
|
19
|
+
(`on_unanswered: fail`), the harness now salvages a **PARTIAL** `result.json` (+ `run.jsonl`/`trace.json`)
|
|
20
|
+
with the artifacts the agent wrote before the whiff, then still exits 2. The run dir is printed so you can
|
|
21
|
+
`inspect` it. `partial` and `unansweredGate` are new `RunResult` fields; `verify-run` and `scaffold`
|
|
22
|
+
refuse to treat a partial run's half-finished output as a passing result (scaffold still emits the gates,
|
|
23
|
+
drops the artifact/result asserts, and warns loudly).
|
|
24
|
+
- **Capability pre-flight — fail fast.** A skill that declares `requires_capabilities` against an image that
|
|
25
|
+
provably omits them now **aborts before the (paid) run** (`exit 3`) instead of running ~12 min to a verdict
|
|
26
|
+
that's already known — unless the scenario asserts `allow_missing_capability: true`, which downgrades it to
|
|
27
|
+
a notice and proceeds. `doctor` also surfaces the full-parity remedy on its agent-image line.
|
|
28
|
+
|
|
29
|
+
### Changed
|
|
30
|
+
|
|
31
|
+
- **LLM decider tolerates a near-miss label.** `--decider-llm` now binds a reply like `Confirmed.` or
|
|
32
|
+
`"Confirmed"` to the label `Confirmed` (trailing sentence punctuation / surrounding quotes trimmed before
|
|
33
|
+
matching) instead of failing loud — common on binary confirm gates. The `:` of the `OTHER:` free-text
|
|
34
|
+
sentinel is never stripped, and fuzzy substring matching stays off, so the change can't mis-bind. The
|
|
35
|
+
tolerance lives in `matchLabel` itself, so the web_fetch-approval path gets it too (a `Deny.` reply now
|
|
36
|
+
binds `Deny`).
|
|
37
|
+
- **`runs gc` → top-level `prune`; the `runs` namespace is dropped.** Pruning accumulated run dirs is now
|
|
38
|
+
`cowork-harness prune [--keep-last <n>] [--dry-run] [<runs-dir>]` (same flags). `runs` was a namespace with
|
|
39
|
+
a single member and collided confusingly with `run` (execute); removing it leaves `run` = execute,
|
|
40
|
+
inspection verbs (`trace`/`inspect`/`verify-run`/`scaffold`) top-level, and `prune` for cleanup.
|
|
41
|
+
|
|
42
|
+
### Fixed
|
|
43
|
+
|
|
44
|
+
- **Session-protocol over-cap test no longer flakes.** The over-cap control-out test read
|
|
45
|
+
`control-out.jsonl` synchronously, but the stream opens it asynchronously (`createWriteStream`, flags
|
|
46
|
+
`"a"`), so under load the file may not exist yet at the read — a flaky ENOENT. The over-cap frame is
|
|
47
|
+
rejected before any write, so a missing file means nothing was written; the test now treats absence as
|
|
48
|
+
empty.
|
|
49
|
+
|
|
50
|
+
### Docs
|
|
51
|
+
|
|
52
|
+
- **Documented the LLM-decider free-text path.** `--decider-llm` supplies free text via `OTHER: <value>`
|
|
53
|
+
on an options-bearing gate; a bare out-of-set answer (no matching label, no `OTHER:`) fails loud
|
|
54
|
+
(`UnansweredError` → exit 2), never stalling or guessing, and open-ended (no-option) gates need no
|
|
55
|
+
prefix. Noted in the decider section of `fidelity-and-answers.md` and cross-referenced from the lone
|
|
56
|
+
`OTHER:` mention in `SKILL.md` so the two don't drift.
|
|
57
|
+
- **Documented the live real-document validation workflow.** A new SKILL.md section covers driving a skill
|
|
58
|
+
against real input documents (not recording a cassette): explore with `--decider-llm --intent`, script the
|
|
59
|
+
load-bearing and binary-confirm gates, budget ~1 re-run per file (a whiffed gate now salvages a partial
|
|
60
|
+
run), and `inspect` the outputs to judge correctness.
|
|
61
|
+
|
|
7
62
|
## [0.12.0] — 2026-06-24
|
|
8
63
|
|
|
9
64
|
### Added
|
package/README.md
CHANGED
|
@@ -97,6 +97,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
|
|
|
97
97
|
| `verify-cassettes <file\|dir>` | Token-free CI gate over committed cassettes: a privacy scan (email/currency/domain → exit 1; whole-token allows via `--allow` / class-scoped `--allow-domain` / `--allow-email` / `--allow-file`) + a staleness check (`--skip-privacy` or `--skip-staleness` to run only one half). A dir argument scans `*.cassette.json` in that dir only (**non-recursive**) | gating **committed cassettes** against PII leaks + "edited the skill, forgot to re-record" |
|
|
98
98
|
| `verify-run <run-dir> <scenario.yaml>` | Re-evaluate a scenario's `assert:` (and, when the scenario declares `answers:`, whether they still match the run's actual gates) against an already-kept run dir — **no live agent, no tokens, no Docker** (~1s) | iterating on a wrong assertion or a drifted `answer` without a full live re-record |
|
|
99
99
|
| `trace <run-id>` | Digest a run's `events.jsonl` (`--view tools\|questions\|dispatches`; legacy `--tools` / `--gates` / `--dispatches` aliases still accepted) | "how many sub-agents *actually* dispatched, and which?" |
|
|
100
|
+
| `inspect <run-id>` | Show what a run **produced**: the artifacts + a shallow field preview of each JSON artifact (`--output-format json` for a digest). Works on a salvaged partial run too | "did it do the job?" — without hand-parsing `…/mnt/outputs/…` |
|
|
100
101
|
| `scaffold <run-id>` | Turn a kept run into a starter scenario YAML (gates→answers, artifacts→`file_exists`); `--from-run <id>` is a deprecated alias for the positional | authoring a scenario from a real run instead of guessing |
|
|
101
102
|
| `python3 …/scenario.py scaffold --name <name> --skill <dir>` | Generate a starter scenario skeleton from scratch (the `…` is `.claude/skills/cowork-harness/scripts/`) | starting a new scenario when you have no prior run |
|
|
102
103
|
| `lint <scenario.yaml \| dir/>…` | Check scenarios for silent false-greens — assertions placed on the wrong CI lane, mixed content/live keys, missing `controlOut`-required keys (files or a directory of `*.yaml`/`*.yml`; bundled `scenario.py`; needs python3 — PyYAML is bundled) | before committing a new scenario or after changing assertions |
|
|
@@ -106,7 +107,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
|
|
|
106
107
|
| `boundary-check [baseline] [--session <file>]` | Prove the sandbox enforces Cowork's limitations; `--session` folds a session's `egress.extra_allow` into the probe allowlist | verifying the harness's own fidelity |
|
|
107
108
|
| `sync` / `list` | Derive/refresh & list platform baselines from the Desktop install | after Claude Desktop updates (baselines ship, so it's optional otherwise) |
|
|
108
109
|
| `doctor [--tier <t>]` | Read-only prerequisite check, per tier (Docker + agent image for `container`/`hostloop`; **Lima** for `microvm`; plus staged agent, token, baseline); prints the exact `docker build` line if the agent image is missing | "can I run the live tiers — what's missing?" before a first live run |
|
|
109
|
-
| `
|
|
110
|
+
| `prune [--keep-last <n>]` | Prune accumulated run dirs (keeps the N most recent per scenario; pinned `--session-id` runs are never pruned); `--dry-run` | the machine-global runs root has grown and you want space back |
|
|
110
111
|
| `rehash <dir/>` | Migrate cassette fingerprints to the current format version when the content is provably unchanged (`--dry-run`); no re-record needed | a cassette-format bump flagged committed fixtures as stale |
|
|
111
112
|
| `vm <init\|status\|delete\|prune>` | Manage the L2 Apple-VZ / Lima microVM (`prune` removes orphaned VMs left by config/agent-version changes) | running `--fidelity microvm` |
|
|
112
113
|
|
|
@@ -207,7 +208,7 @@ This repo ships a **companion skill** (`.claude/skills/cowork-harness/`) that te
|
|
|
207
208
|
/plugin install cowork-harness@cowork-harness
|
|
208
209
|
```
|
|
209
210
|
|
|
210
|
-
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.
|
|
211
|
+
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx cowork-harness@>=0.13.0` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
|
|
211
212
|
|
|
212
213
|
It also follows the open [Agent Skills](https://github.com/vercel-labs/skills) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) by pointing the `npx skills` CLI at `.claude/skills/cowork-harness` in this repo. (Working *inside* this repo, the skill auto-loads as a project skill — no install needed.)
|
|
213
214
|
|
|
@@ -473,7 +474,7 @@ Unit tests cover the scripted-answer logic, the egress allowlist matcher, the se
|
|
|
473
474
|
- `COWORK_AGENT_IMAGE=<tag>` — override the agent image name (default `cowork-agent-base:2`); `COWORK_AGENT_BINARY=<path>` — override the auto-detected staged agent ELF.
|
|
474
475
|
- `COWORK_SKIP_CAPABILITY_PROBE=1` — skip the per-run capability probe (the harness otherwise probes the agent image/VM for the document/OCR/Office capabilities the real Cowork rootfs ships and **fails a run that uses one the image omits** — a likely false negative; suppress per-scenario with `allow_missing_capability: true`, or rebuild full parity).
|
|
475
476
|
- `COWORK_HARNESS_DECIDER_DIR_POLL_MS` / `_TIMEOUT_MS` — tune the `--decider-dir` rendezvous poll/backstop; `COWORK_HARNESS_DECIDER_CMD_TIMEOUT_MS` / `COWORK_HARNESS_LLM_TIMEOUT_MS` — backstop a hung `--decider-cmd` helper / `--decider-llm` model call (default 600 s, fail loud); `COWORK_HARNESS_DIALOG_TIMEOUT_MS` — override the 6 s dialog auto-cancel.
|
|
476
|
-
- `COWORK_HARNESS_RUNS_DIR` (or the `--run-dir <path>` flag) — override the default run-output root `~/.cowork-harness/runs` (kept out of any working tree so sensitive skill inputs/outputs don't land in a repo). Precedence: `--run-dir` > env > default. The root is flat and machine-global (shared across projects); pinned `--session-id` runs are guarded against cross-project overwrite, and `
|
|
477
|
+
- `COWORK_HARNESS_RUNS_DIR` (or the `--run-dir <path>` flag) — override the default run-output root `~/.cowork-harness/runs` (kept out of any working tree so sensitive skill inputs/outputs don't land in a repo). Precedence: `--run-dir` > env > default. The root is flat and machine-global (shared across projects); pinned `--session-id` runs are guarded against cross-project overwrite, and `prune` never prunes them. In CI, set it to a workspace path (e.g. `runs`) so artifact upload can collect the runs. `COWORK_HARNESS_ALLOW_FOREIGN_RESUME=1` overrides the guard that blocks `--resume` onto another project's pinned session.
|
|
477
478
|
- **Networking / loop:** `COWORK_EGRESS_PROXY` overrides the egress-proxy URL injected into the sandbox; `COWORK_PROXY_IMAGE` overrides the egress proxy Docker image name (default `cowork-egress-proxy:1`); `COWORK_DOCKER_NETWORK` pins the Docker network the agent container joins; `CLAUDE_FORCE_HOST_LOOP=1` forces the host-loop path regardless of the baseline's loop decision (the `cowork` tier's auto-pick). `COWORK_LIMACTL` overrides the `limactl` binary path (default `/opt/homebrew/bin/limactl`).
|
|
478
479
|
- **Strictness escape hatches** (the harness fails loud by default): `COWORK_HARNESS_SOFT_MISSING=1` downgrades a missing mount source from a hard error to warn-and-exclude; `COWORK_HARNESS_ALLOW_CONFIG_DIR_WRITE=1` permits writing into an existing pinned `plugins.config_dir` (otherwise refused, to avoid clobbering a real Claude config).
|
|
479
480
|
- **Staleness boundary:** the cassette-staleness skill hash uses the **git-tracked** file set of each skill/plugin source dir by default (a non-repo dir falls back to a raw walk; OS-junk like `.DS_Store` is always excluded). `COWORK_HARNESS_GITSET=0` opts out to the legacy raw walk for every dir. `COWORK_HARNESS_DEBUG_SKILLHASH=1` dumps the exact file set feeding the hash on a staleness mismatch (and flags OS-junk) so a drift source is one line. Declare per-plugin non-runtime paths in a `.cowork-hashignore` file (or the session `staleness.hash_ignore`). `COWORK_HARNESS_AGENT_SCOPE=skill` (opt-in) refines a scenario's `skills:` scope so a **skill-named** `agents/<name>.md` re-stales only that skill's cassettes instead of the whole fleet (generic agents stay shared; stamped into the fingerprint, so flipping it is a one-time re-record like `GITSET`).
|
package/dist/cli.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import { readFileSync, writeFileSync, mkdirSync, readdirSync, writeSync, existsSync } from "node:fs";
|
|
3
|
-
import { join, basename, resolve, isAbsolute } from "node:path";
|
|
3
|
+
import { join, basename, resolve, isAbsolute, dirname } from "node:path";
|
|
4
4
|
import { fileURLToPath } from "node:url";
|
|
5
5
|
import { parse as parseYaml } from "yaml";
|
|
6
6
|
import { Scenario, AnswerRule, Assertion } from "./types.js";
|
|
@@ -24,6 +24,7 @@ import { loadDotenv } from "./dotenv.js";
|
|
|
24
24
|
import { makeRenderer, renderStart, renderFooter, startHeartbeat } from "./run/renderer.js";
|
|
25
25
|
import { resolveEventsFile, buildTrace, formatTrace, buildGateTrace, formatGateTrace, buildDispatchTree, formatDispatchTree, noteRunsLocation, } from "./run/trace-view.js";
|
|
26
26
|
import { buildScaffold } from "./run/scaffold.js";
|
|
27
|
+
import { buildInspectView } from "./run/inspect-view.js";
|
|
27
28
|
import { pkgVersion, jsonEnvelope, jsonError, parseOutputFormat } from "./run/envelope.js";
|
|
28
29
|
import { computeVerdict } from "./run/verdict.js";
|
|
29
30
|
import { evaluate, hostMatches } from "./assert.js";
|
|
@@ -77,7 +78,7 @@ const HELP = `cowork-harness <command> (v${"$VERSION"})
|
|
|
77
78
|
[--skip-privacy|--skip-staleness] skip one check
|
|
78
79
|
[--allow <regex>]... [--allow-domain/-email <regex>]... [--allow-file <path>]... [--output-format json]
|
|
79
80
|
rehash <dir/> migrate cassette fingerprints to current version when content is provably unchanged (requires contentSig from v3+)
|
|
80
|
-
|
|
81
|
+
prune [--keep-last <n>] prune accumulated run dirs, keeping N most recent per scenario (default: 5)
|
|
81
82
|
|
|
82
83
|
── CI lint + assertion reference ──────────────────────────────────────────────
|
|
83
84
|
lint <scenario.yaml | dir/>… check scenarios for silent false-greens (bundled scenario.py; needs python3 — PyYAML is bundled)
|
|
@@ -92,6 +93,8 @@ const HELP = `cowork-harness <command> (v${"$VERSION"})
|
|
|
92
93
|
[--output-format json] structured rows
|
|
93
94
|
verify-run <run-dir> <scenario.yaml> re-evaluate assert: against a kept run dir (no live agent, ~1s)
|
|
94
95
|
[--output-format json]
|
|
96
|
+
inspect <run-id | run-dir> show what a run produced: artifacts + a shallow field preview of each JSON artifact
|
|
97
|
+
[--output-format json] structured digest
|
|
95
98
|
scaffold <run-id | run-dir> turn a kept run into a starter scenario YAML (gates→answers, artifacts→file_exists)
|
|
96
99
|
[--out <file.yaml>] write to a file (default: stdout)
|
|
97
100
|
|
|
@@ -261,16 +264,17 @@ const SUBCOMMAND_USAGE = {
|
|
|
261
264
|
" NOTE: --allow-failing only relaxes the post-run VERDICT gate; it does NOT salvage an unanswered gate (that throws before any cassette is written — use --on-unanswered first / a decider).",
|
|
262
265
|
replay: "usage: replay <file.cassette.json | dir/> [--strict] [--output-format text|json]",
|
|
263
266
|
"verify-cassettes": "usage: verify-cassettes <file|dir> [--skip-privacy|--skip-staleness] [--allow <regex>]... [--allow-domain <regex>]... [--allow-email <regex>]... [--allow-file <path>]... [--output-format json]",
|
|
264
|
-
trace: "usage: trace <run-id | run-dir | events.jsonl> [--view tools|questions|dispatches] [--output-format json]\n --view tools tool call / result rows\n --view questions gate lifecycle (question → answer → delivered)\n --view dispatches sub-agent dispatch tree + dispatch_count_max\n (default: all views)",
|
|
267
|
+
trace: "usage: trace <run-id | run-dir | events.jsonl> [--view tools|questions|dispatches] [--output-format json]\n --view tools tool call / result rows\n --view questions gate lifecycle (question → answer → delivered)\n --view dispatches sub-agent dispatch tree + dispatch_count_max\n (default: all views)\n (for what the run PRODUCED — artifacts — use `inspect`)",
|
|
265
268
|
assertions: "usage: assertions --list [--output-format json]",
|
|
266
269
|
scaffold: "usage: scaffold <run-id | run-dir> [--out <file.yaml>]\n Turns a kept run into a starter scenario YAML (gates→answers, artifacts→file_exists).\n Positional <run-id | run-dir> is the canonical form.",
|
|
267
270
|
decide: 'usage: decide [--question <q>] [--option <o>]... [--decider-cmd <cmd> | --decider-llm] [--answer "<q>=<label>"]... [--answer-policy <p>] [--intent <s>] [--output-format json]',
|
|
268
271
|
gates: "usage: gates <dir> [--follow] (stream pending in-band gates as JSON lines; pair with --decider-dir)",
|
|
269
272
|
answer: 'usage: answer <dir> --gate <N> (--choose <label> [--choose <label>…] | --answer "<q>=<label>") (write an in-band gate reply atomically; repeat --choose for a multiSelect gate)',
|
|
270
273
|
"verify-run": "usage: verify-run <run-dir> <scenario.yaml> [--output-format json] (re-evaluate a scenario's assert: against a kept run dir; no live agent)",
|
|
274
|
+
inspect: "usage: inspect <run-id | run-dir> [--output-format json] (show what a run PRODUCED: artifacts + a shallow preview of each JSON artifact's fields; for what HAPPENED — tools/decisions — use `trace`)",
|
|
271
275
|
doctor: "usage: doctor [--tier protocol|container|microvm|hostloop|cowork] [--output-format json] (read-only prerequisite check)",
|
|
272
276
|
rehash: "usage: rehash <dir/> [--dry-run] [--output-format text|json] (migrate cassettes across format bumps using contentSig verification; no re-record needed)",
|
|
273
|
-
|
|
277
|
+
prune: "usage: prune [--keep-last <n>] [--dry-run] [<runs-dir>] (prune accumulated run dirs; default --keep-last 5)",
|
|
274
278
|
};
|
|
275
279
|
// Known subcommands — used by the global value-flag parsers (`--dotenv`, `--run-dir`) to reject a command
|
|
276
280
|
// name mistaken as the flag's value (`--dotenv run x.yaml` would otherwise swallow `run`).
|
|
@@ -283,6 +287,7 @@ const COMMANDS = [
|
|
|
283
287
|
"verify-cassettes",
|
|
284
288
|
"verify-run",
|
|
285
289
|
"trace",
|
|
290
|
+
"inspect",
|
|
286
291
|
"assertions",
|
|
287
292
|
"scaffold",
|
|
288
293
|
"decide",
|
|
@@ -295,7 +300,7 @@ const COMMANDS = [
|
|
|
295
300
|
"lint",
|
|
296
301
|
"doctor",
|
|
297
302
|
"rehash",
|
|
298
|
-
"
|
|
303
|
+
"prune",
|
|
299
304
|
];
|
|
300
305
|
async function main() {
|
|
301
306
|
const argv = process.argv.slice(2);
|
|
@@ -414,6 +419,8 @@ async function main() {
|
|
|
414
419
|
return cmdVerifyRun(rest);
|
|
415
420
|
case "trace":
|
|
416
421
|
return cmdTrace(rest);
|
|
422
|
+
case "inspect":
|
|
423
|
+
return cmdInspect(rest);
|
|
417
424
|
case "assertions":
|
|
418
425
|
return cmdAssert(rest);
|
|
419
426
|
case "scaffold":
|
|
@@ -424,13 +431,8 @@ async function main() {
|
|
|
424
431
|
return cmdGates(rest);
|
|
425
432
|
case "answer":
|
|
426
433
|
return cmdAnswer(rest);
|
|
427
|
-
case "
|
|
428
|
-
|
|
429
|
-
if (sub === "gc")
|
|
430
|
-
return cmdRunsGc(rest.slice(1));
|
|
431
|
-
log(`runs: unknown subcommand "${sub ?? ""}" — available: gc`);
|
|
432
|
-
return process.exit(2);
|
|
433
|
-
}
|
|
434
|
+
case "prune":
|
|
435
|
+
return cmdRunsGc(rest); // top-level: no `gc` token to strip, so pass `rest` whole (NOT rest.slice(1))
|
|
434
436
|
default:
|
|
435
437
|
log(`unknown command: ${cmd}\n`);
|
|
436
438
|
printHelp();
|
|
@@ -656,7 +658,7 @@ function resolveExternal(command, flags) {
|
|
|
656
658
|
}
|
|
657
659
|
return flags.deciderCmd != null ? spawnChannel(flags.deciderCmd) : undefined;
|
|
658
660
|
}
|
|
659
|
-
/** The single error exit used by commands + the top-level catch.
|
|
661
|
+
/** The single error exit used by commands + the top-level catch. boundary → exit 3, every other category → exit 2. */
|
|
660
662
|
function fail(command, category, message, hint, json) {
|
|
661
663
|
if (json)
|
|
662
664
|
out(jsonError(command, category, message, hint));
|
|
@@ -1828,6 +1830,14 @@ async function cmdVerifyRun(args) {
|
|
|
1828
1830
|
log(`verify-run: cannot read ${resultPath}: ${e.message}`);
|
|
1829
1831
|
return process.exit(2);
|
|
1830
1832
|
}
|
|
1833
|
+
// A partial run did NOT complete (it exited on an unanswered gate). Its assertion outcome is empty and its
|
|
1834
|
+
// artifacts are pre-failure, so re-evaluating asserts against it would vouch for a run that never finished.
|
|
1835
|
+
// Refuse rather than false-fail or false-pass.
|
|
1836
|
+
if (result.partial) {
|
|
1837
|
+
log(`verify-run: ${runDir} is a PARTIAL run — it did not complete (exited on an unanswered gate). ` +
|
|
1838
|
+
`Re-run to completion before verifying. (can't verify ⇒ not green)`);
|
|
1839
|
+
return process.exit(2);
|
|
1840
|
+
}
|
|
1831
1841
|
let scenario;
|
|
1832
1842
|
try {
|
|
1833
1843
|
scenario = parseScenarioFile(scenarioFile);
|
|
@@ -2086,6 +2096,32 @@ function cmdTrace(args) {
|
|
|
2086
2096
|
else
|
|
2087
2097
|
out(formatTrace(rows));
|
|
2088
2098
|
}
|
|
2099
|
+
function cmdInspect(args) {
|
|
2100
|
+
if (hasHelp(args))
|
|
2101
|
+
return void log(SUBCOMMAND_USAGE.inspect);
|
|
2102
|
+
ensureOutputFormat("inspect", args);
|
|
2103
|
+
const json = isJsonOutput(args);
|
|
2104
|
+
rejectUnknownFlags("inspect", args, ["--output-format", "--output-format=json", "--output-format=text"], json);
|
|
2105
|
+
const allPositionals = positionals(args, ["--output-format"]);
|
|
2106
|
+
if (allPositionals.length !== 1)
|
|
2107
|
+
return void fail("inspect", "usage", SUBCOMMAND_USAGE.inspect, undefined, json);
|
|
2108
|
+
// Resolve a run-id or run-dir to its dir. A run dir already holding result.json is used directly; otherwise
|
|
2109
|
+
// reuse trace's resolver (run-id → events.jsonl) and take the parent.
|
|
2110
|
+
const target = allPositionals[0];
|
|
2111
|
+
let runDir;
|
|
2112
|
+
try {
|
|
2113
|
+
runDir = existsSync(join(target, "result.json")) ? target : dirname(resolveEventsFile(target));
|
|
2114
|
+
}
|
|
2115
|
+
catch (e) {
|
|
2116
|
+
return void fail("inspect", "usage", String(e.message), undefined, json);
|
|
2117
|
+
}
|
|
2118
|
+
try {
|
|
2119
|
+
out(buildInspectView(runDir, { json }));
|
|
2120
|
+
}
|
|
2121
|
+
catch (e) {
|
|
2122
|
+
return void fail("inspect", "usage", String(e.message), undefined, json);
|
|
2123
|
+
}
|
|
2124
|
+
}
|
|
2089
2125
|
function diff(a, b, path) {
|
|
2090
2126
|
const keys = new Set([...Object.keys(a ?? {}), ...Object.keys(b ?? {})]);
|
|
2091
2127
|
for (const k of keys) {
|
package/dist/decide/decider.js
CHANGED
|
@@ -256,8 +256,9 @@ export class FirstOptionDecider {
|
|
|
256
256
|
return ABSTAIN;
|
|
257
257
|
}
|
|
258
258
|
}
|
|
259
|
-
/** Strict label match: exact → case-insensitive
|
|
260
|
-
* do NOT fall back to option 1
|
|
259
|
+
/** Strict label match: exact → case-insensitive → near-miss (surrounding quotes / trailing sentence
|
|
260
|
+
* punctuation trimmed). NULL on no match (caller fails loud — we deliberately do NOT fall back to option 1
|
|
261
|
+
* like `coerceLabel`, which would silently mis-answer).
|
|
261
262
|
* When `fuzzy` is true, also accept a substring match (ONLY when exactly one label appears in the reply
|
|
262
263
|
* to avoid ambiguity). Callers that consume human or LLM output should leave `fuzzy` at its default
|
|
263
264
|
* (false) to require an exact answer — the substring heuristic is opt-in only. */
|
|
@@ -266,6 +267,19 @@ export function matchLabel(raw, labels, fuzzy = false) {
|
|
|
266
267
|
const exact = labels.find((l) => l === r) ?? labels.find((l) => l.toLowerCase() === r.toLowerCase());
|
|
267
268
|
if (exact)
|
|
268
269
|
return exact;
|
|
270
|
+
// Near-miss tier: a model (or human) often replies `Confirmed.` or `"Confirmed"` for the label
|
|
271
|
+
// `Confirmed`, especially on binary confirm gates. Strip surrounding quotes + trailing `.!,;`/whitespace
|
|
272
|
+
// and re-run the exact/ci match. NEVER strip `:` — it is the `OTHER:` free-text sentinel callers rely on,
|
|
273
|
+
// and the trim must not let a near-miss swallow it. Strict (not fuzzy), so it can't mis-bind a substring.
|
|
274
|
+
const norm = r
|
|
275
|
+
.replace(/^["'`]+/, "")
|
|
276
|
+
.replace(/["'`]+$/, "")
|
|
277
|
+
.replace(/[.!,;\s]+$/, "");
|
|
278
|
+
if (norm && norm !== r) {
|
|
279
|
+
const near = labels.find((l) => l === norm) ?? labels.find((l) => l.toLowerCase() === norm.toLowerCase());
|
|
280
|
+
if (near)
|
|
281
|
+
return near;
|
|
282
|
+
}
|
|
269
283
|
if (!fuzzy)
|
|
270
284
|
return null;
|
|
271
285
|
// fuzzy=true: the substring tier fires ONLY when EXACTLY ONE label is contained in the reply. With
|
|
@@ -351,8 +365,10 @@ export class LlmDecider {
|
|
|
351
365
|
continue;
|
|
352
366
|
}
|
|
353
367
|
const raw = await this.complete(this.prompt(text, q.options, ctx), this.model);
|
|
354
|
-
// Match a LABEL first (matchLabel is fuzzy=false
|
|
355
|
-
//
|
|
368
|
+
// Match a LABEL first (matchLabel is fuzzy=false, but tolerates a quoted / trailing-punctuation
|
|
369
|
+
// near-miss): keeps the common path unchanged and means a real option literally named "OTHER: …" is
|
|
370
|
+
// selected as a label (exact match wins; the trim never strips `:`), not hijacked by the free-text
|
|
371
|
+
// branch below.
|
|
356
372
|
const pick = matchLabel(raw, labels);
|
|
357
373
|
if (pick) {
|
|
358
374
|
process.stderr.write(`[llm-decider] "${text}" → "${pick}"${this.intent ? ` (intent: ${this.intent})` : ""}\n`);
|
package/dist/run/doctor.js
CHANGED
|
@@ -174,7 +174,11 @@ export function runDoctorChecks(tier, probe = realProbe) {
|
|
|
174
174
|
id: "image",
|
|
175
175
|
title: "Agent image",
|
|
176
176
|
status: present ? "ok" : up ? "fail" : "skip",
|
|
177
|
-
detail: present
|
|
177
|
+
detail: present
|
|
178
|
+
? `${image} present — lean core; OCR / PDF-table skills need the full-parity image (--build-arg COWORK_FULL_PARITY=1)`
|
|
179
|
+
: up
|
|
180
|
+
? `${image} missing`
|
|
181
|
+
: `(skipped — ${runtime} not reachable)`,
|
|
178
182
|
remedy: present || !up ? undefined : `build it: ${agentBuildLine(runtime, image)}`,
|
|
179
183
|
required: up, // only gate on the image once the runtime is actually reachable
|
|
180
184
|
});
|
package/dist/run/execute.js
CHANGED
|
@@ -17,7 +17,7 @@ import { spawnProtocol } from "../runtime/protocol.js";
|
|
|
17
17
|
import { spawnContainer } from "../runtime/container.js";
|
|
18
18
|
import { spawnHostLoop } from "../runtime/hostloop.js";
|
|
19
19
|
import { spawnMicroVm } from "../runtime/microvm.js";
|
|
20
|
-
import { probeImageOmitted, probeMicrovmOmitted, detectCapabilityUse, CAPABILITY_FAMILIES } from "../runtime/image-capabilities.js";
|
|
20
|
+
import { probeImageOmitted, probeMicrovmOmitted, detectCapabilityUse, capabilityPreflightDecision, CAPABILITY_FAMILIES, } from "../runtime/image-capabilities.js";
|
|
21
21
|
import { instanceName } from "../runtime/lima.js";
|
|
22
22
|
import { decideLoopFromBaseline, readGateFlag } from "../loop-decision.js";
|
|
23
23
|
import { startEgressSidecar, registerCleanup } from "../egress/sidecar.js";
|
|
@@ -258,6 +258,7 @@ export async function executeScenario(scenario, opts = {}) {
|
|
|
258
258
|
let hostProxy;
|
|
259
259
|
let microvmProxyPort;
|
|
260
260
|
let record;
|
|
261
|
+
let unansweredErr; // set when a gate whiffs — drives the salvage branch below
|
|
261
262
|
let child; // hoisted so the finally can reap a crashed/orphaned container (F1)
|
|
262
263
|
let containerName;
|
|
263
264
|
let deregisterContainerReap; // 2a: Ctrl-C cleanup for the agent container
|
|
@@ -270,6 +271,29 @@ export async function executeScenario(scenario, opts = {}) {
|
|
|
270
271
|
const viaApiOn = readGateFlag(baseline, "1978029737", "coworkWebFetchViaApi");
|
|
271
272
|
const promptGateOn = readGateFlag(baseline, "1978029737", "coworkWebFetchPrompt");
|
|
272
273
|
const provenanceRef = {};
|
|
274
|
+
// Pre-flight: if the skill DECLARES required capabilities and the image provably omits one, FAIL FAST here
|
|
275
|
+
// — before any paid agent run — instead of burning ~12 min to reach a verdict the post-run guard already
|
|
276
|
+
// knows. The author can opt out with `allow_missing_capability: true` (the fallback is equivalent), which
|
|
277
|
+
// downgrades to a notice. The image probe is digest-cached, so it's shared with the post-run check (no
|
|
278
|
+
// second container spawn), and it spawns a throwaway `--network none` container with no model — zero tokens.
|
|
279
|
+
const declaredCaps = scenario.requires_capabilities ?? [];
|
|
280
|
+
if (declaredCaps.length &&
|
|
281
|
+
(effectiveFidelity === "container" || effectiveFidelity === "hostloop" || effectiveFidelity === "microvm") &&
|
|
282
|
+
process.env.COWORK_SKIP_CAPABILITY_PROBE !== "1") {
|
|
283
|
+
const omitted = effectiveFidelity === "microvm"
|
|
284
|
+
? probeMicrovmOmitted(instanceName(baseline))
|
|
285
|
+
: probeImageOmitted({
|
|
286
|
+
runtime: process.env.COWORK_CONTAINER_RUNTIME ?? "docker",
|
|
287
|
+
image: process.env.COWORK_AGENT_IMAGE ?? "cowork-agent-base:2",
|
|
288
|
+
tier: effectiveFidelity,
|
|
289
|
+
});
|
|
290
|
+
const allowMissing = scenario.assert.some((a) => a.allow_missing_capability === true);
|
|
291
|
+
const { abort, message } = capabilityPreflightDecision(declaredCaps, omitted, allowMissing);
|
|
292
|
+
if (abort)
|
|
293
|
+
throw new BoundaryError(`[capability] ${message}`);
|
|
294
|
+
if (message)
|
|
295
|
+
warn(`::notice:: [capability] (pre-flight) ${message} (allow_missing_capability asserted — proceeding)\n`);
|
|
296
|
+
}
|
|
273
297
|
try {
|
|
274
298
|
// acquire the egress sidecar / host proxy INSIDE the protected try so a throw in resource
|
|
275
299
|
// acquisition OR in renderPrompts below can't leak a Docker network / a bound proxy port — the `finally`
|
|
@@ -365,10 +389,22 @@ export async function executeScenario(scenario, opts = {}) {
|
|
|
365
389
|
permissiveMode: plan.permissionMode === "bypassPermissions",
|
|
366
390
|
};
|
|
367
391
|
}
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
392
|
+
try {
|
|
393
|
+
record = await run.drive(scenario.prompt, {
|
|
394
|
+
subagentAppend: prompts.subagentAppend,
|
|
395
|
+
sdkMcp,
|
|
396
|
+
});
|
|
397
|
+
}
|
|
398
|
+
catch (e) {
|
|
399
|
+
// An unanswered gate is recoverable: grab the in-progress record so the work done before the whiff can
|
|
400
|
+
// be salvaged to disk below. Any other error is a genuine fault — keep today's fail-fast behavior.
|
|
401
|
+
if (e instanceof UnansweredError) {
|
|
402
|
+
unansweredErr = e;
|
|
403
|
+
record = run.partial();
|
|
404
|
+
}
|
|
405
|
+
else
|
|
406
|
+
throw e;
|
|
407
|
+
}
|
|
372
408
|
}
|
|
373
409
|
finally {
|
|
374
410
|
// Reap the agent container FIRST (before the sidecar networks), so a crashed/unanswered run can't
|
|
@@ -402,6 +438,36 @@ export async function executeScenario(scenario, opts = {}) {
|
|
|
402
438
|
// are read-only inputs and are NOT visible roots. Persisted to RunResult so the plan-less lanes
|
|
403
439
|
// (verify reads result.json; replay reads the cassette) match this without rebuilding a LaunchPlan.
|
|
404
440
|
const userVisibleRoots = ["outputs", ...plan.mounts.filter((m) => m.kind === "folder").map((m) => m.mountPath)];
|
|
441
|
+
// Salvage path: the run exited on an unanswered gate. Persist a PARTIAL result.json (+ run.jsonl/trace) so
|
|
442
|
+
// the artifacts the agent wrote before the whiff survive for inspection, then re-throw so the CLI still
|
|
443
|
+
// exits 2. Skip assertion eval and the capability probe (a real container spawn) — a partial run has no
|
|
444
|
+
// meaningful assertion or verdict outcome.
|
|
445
|
+
if (unansweredErr) {
|
|
446
|
+
const partialResult = buildPartialResult({
|
|
447
|
+
scenarioName: scenario.name,
|
|
448
|
+
prompt: scenario.prompt,
|
|
449
|
+
fidelity: scenario.fidelity,
|
|
450
|
+
baseline: baseline.appVersion,
|
|
451
|
+
record,
|
|
452
|
+
outDir,
|
|
453
|
+
workRoot,
|
|
454
|
+
userVisibleRoots,
|
|
455
|
+
effectiveFidelity,
|
|
456
|
+
egress,
|
|
457
|
+
durationMs: Date.now() - startedAt,
|
|
458
|
+
unanswered: { message: unansweredErr.message, hint: unansweredErr.hint },
|
|
459
|
+
fingerprint: buildFingerprint(scenario.session, baseline.appVersion, undefined, scenario.skills),
|
|
460
|
+
});
|
|
461
|
+
writeFileSync(join(outDir, "result.json"), scrub(JSON.stringify(partialResult, null, 2), secrets));
|
|
462
|
+
writeRunJsonl(outDir, scenario, effectiveFidelity, record, egress, secrets);
|
|
463
|
+
writeTrace(outDir, record, egress, secrets, partialResult.durationMs);
|
|
464
|
+
scrubFileInPlace(join(outDir, "events.jsonl"), secrets);
|
|
465
|
+
scrubFileInPlace(join(outDir, "control-out.jsonl"), secrets);
|
|
466
|
+
// Loud PARTIAL marker so the populated artifacts are never misread as success (the no-false-green rule).
|
|
467
|
+
warn(`::notice:: [partial] run did NOT complete (unanswered gate) — salvaged the pre-failure work to:\n` +
|
|
468
|
+
` ${outDir}\n inspect it: cowork-harness inspect ${outDir}\n`);
|
|
469
|
+
throw unansweredErr;
|
|
470
|
+
}
|
|
405
471
|
const assertions = evaluate(scenario.assert, {
|
|
406
472
|
transcript: record.transcript,
|
|
407
473
|
toolsCalled: record.toolsCalled,
|
|
@@ -632,6 +698,51 @@ function writeRunJsonl(outDir, scenario, fidelity, rec, egress, secrets) {
|
|
|
632
698
|
];
|
|
633
699
|
writeFileSync(join(outDir, "run.jsonl"), scrub(lines.map((l) => JSON.stringify(l)).join("\n"), secrets));
|
|
634
700
|
}
|
|
701
|
+
/** Assemble a RunResult for a run that did NOT complete — it exited on an unanswered gate. The work the
|
|
702
|
+
* agent did before the whiff (artifacts on disk, the partial transcript, decisions/tool counts so far) is
|
|
703
|
+
* salvaged so the paid run is still inspectable instead of vanishing. Deliberately reduced: no assertion
|
|
704
|
+
* outcome (a partial run has none) and no capability/verdict fields (those would need a probe we skip).
|
|
705
|
+
* `partial:true` is the signal that lets consumers (verify-run, scaffold, the footer) refuse to read its
|
|
706
|
+
* populated `artifacts[]` as a passing run. */
|
|
707
|
+
export function buildPartialResult(args) {
|
|
708
|
+
const { record } = args;
|
|
709
|
+
return {
|
|
710
|
+
$schema: RUN_RESULT_SCHEMA_URL,
|
|
711
|
+
generator: "cowork-harness",
|
|
712
|
+
scenario: args.scenarioName,
|
|
713
|
+
prompt: args.prompt,
|
|
714
|
+
fidelity: args.fidelity,
|
|
715
|
+
baseline: args.baseline,
|
|
716
|
+
result: "error",
|
|
717
|
+
partial: true,
|
|
718
|
+
unansweredGate: { message: args.unanswered.message, ...(args.unanswered.hint ? { hint: args.unanswered.hint } : {}) },
|
|
719
|
+
decisions: record.decisions.map((d) => ({
|
|
720
|
+
kind: d.kind,
|
|
721
|
+
name: d.name,
|
|
722
|
+
decision: d.decision,
|
|
723
|
+
by: d.by,
|
|
724
|
+
detail: d.detail,
|
|
725
|
+
rationale: d.rationale,
|
|
726
|
+
})),
|
|
727
|
+
toolCounts: record.toolCounts,
|
|
728
|
+
gateDeliveries: record.gateDeliveries,
|
|
729
|
+
egress: args.egress,
|
|
730
|
+
assertions: [],
|
|
731
|
+
toolResults: record.toolResults,
|
|
732
|
+
subagents: record.subagents,
|
|
733
|
+
unanswered: record.unanswered,
|
|
734
|
+
usage: record.usage,
|
|
735
|
+
cost: record.cost,
|
|
736
|
+
durationMs: args.durationMs,
|
|
737
|
+
outDir: args.outDir,
|
|
738
|
+
workDir: args.workRoot,
|
|
739
|
+
outputsDir: join(args.workRoot, "outputs"),
|
|
740
|
+
userVisibleRoots: args.userVisibleRoots,
|
|
741
|
+
artifacts: collectArtifacts(args.workRoot, args.userVisibleRoots),
|
|
742
|
+
effectiveFidelity: args.effectiveFidelity,
|
|
743
|
+
...(args.fingerprint ? { fingerprint: args.fingerprint } : {}),
|
|
744
|
+
};
|
|
745
|
+
}
|
|
635
746
|
/** B3: the structured run trace. */
|
|
636
747
|
/** ENV-MANIFEST: recursively list files under each user-visible prefix (relative path + byte size).
|
|
637
748
|
* Paths only — NO content snapshot (that is the cassette manifest, #1).
|
|
@@ -0,0 +1,95 @@
|
|
|
1
|
+
import { readFileSync, existsSync } from "node:fs";
|
|
2
|
+
import { join } from "node:path";
|
|
3
|
+
/** A compact, depth-1 preview of a JSON artifact: scalars kept inline, arrays shown as a count, nested
|
|
4
|
+
* objects collapsed to `{…}` — enough to answer "did it produce the right fields?" without dumping blobs. */
|
|
5
|
+
function shallowPreview(doc) {
|
|
6
|
+
if (Array.isArray(doc))
|
|
7
|
+
return `[${doc.length} items]`;
|
|
8
|
+
if (!doc || typeof doc !== "object")
|
|
9
|
+
return { value: doc };
|
|
10
|
+
const out = {};
|
|
11
|
+
for (const [k, v] of Object.entries(doc)) {
|
|
12
|
+
if (Array.isArray(v))
|
|
13
|
+
out[k] = `[${v.length} items]`;
|
|
14
|
+
else if (v && typeof v === "object")
|
|
15
|
+
out[k] = "{…}";
|
|
16
|
+
else
|
|
17
|
+
out[k] = v;
|
|
18
|
+
}
|
|
19
|
+
return out;
|
|
20
|
+
}
|
|
21
|
+
function humanMs(ms) {
|
|
22
|
+
if (!ms && ms !== 0)
|
|
23
|
+
return "?";
|
|
24
|
+
if (ms < 1000)
|
|
25
|
+
return `${ms}ms`;
|
|
26
|
+
return `${(ms / 1000).toFixed(1)}s`;
|
|
27
|
+
}
|
|
28
|
+
/** Build the digest for one kept run dir: header facts + the artifacts manifest, with a shallow field
|
|
29
|
+
* preview for each JSON artifact (read from the work dir, which result.json records). The manifest
|
|
30
|
+
* (paths + sizes) lives in result.json and always survives; the content preview needs the work dir, which
|
|
31
|
+
* a non-`--keep` container/microvm run tears down. */
|
|
32
|
+
function digestFor(runDir) {
|
|
33
|
+
const resultPath = join(runDir, "result.json");
|
|
34
|
+
if (!existsSync(resultPath))
|
|
35
|
+
throw new Error(`no result.json under ${runDir} (is this a kept run dir? e.g. runs/<scenario>/<sessionId>/)`);
|
|
36
|
+
const result = JSON.parse(readFileSync(resultPath, "utf8"));
|
|
37
|
+
const workDir = result.workDir ?? "";
|
|
38
|
+
const workDirAvailable = !!workDir && existsSync(workDir);
|
|
39
|
+
const artifacts = (result.artifacts ?? []).map((a) => {
|
|
40
|
+
const entry = { path: a.path, bytes: a.bytes };
|
|
41
|
+
if (workDirAvailable && a.path.endsWith(".json")) {
|
|
42
|
+
const abs = join(workDir, a.path);
|
|
43
|
+
if (existsSync(abs)) {
|
|
44
|
+
try {
|
|
45
|
+
entry.preview = shallowPreview(JSON.parse(readFileSync(abs, "utf8")));
|
|
46
|
+
}
|
|
47
|
+
catch {
|
|
48
|
+
/* not valid JSON after all — leave preview off */
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
return entry;
|
|
53
|
+
});
|
|
54
|
+
return {
|
|
55
|
+
scenario: result.scenario,
|
|
56
|
+
fidelity: result.fidelity,
|
|
57
|
+
result: result.result,
|
|
58
|
+
...(result.partial ? { partial: true } : {}),
|
|
59
|
+
...(result.unansweredGate ? { unansweredGate: result.unansweredGate } : {}),
|
|
60
|
+
durationMs: result.durationMs,
|
|
61
|
+
...(result.cost ? { cost: result.cost } : {}),
|
|
62
|
+
workDirAvailable,
|
|
63
|
+
artifacts,
|
|
64
|
+
};
|
|
65
|
+
}
|
|
66
|
+
/** Render a run dir for the `inspect` command. `json` mode returns the structured digest; text mode returns
|
|
67
|
+
* a human summary: header, a PARTIAL banner if the run didn't complete, and each artifact with a shallow
|
|
68
|
+
* field preview (or a "work dir torn down" note when content can't be read). */
|
|
69
|
+
export function buildInspectView(runDir, opts = {}) {
|
|
70
|
+
const d = digestFor(runDir);
|
|
71
|
+
if (opts.json)
|
|
72
|
+
return JSON.stringify(d, null, 2);
|
|
73
|
+
const lines = [];
|
|
74
|
+
lines.push(`run: ${d.scenario} (${d.fidelity}) result: ${d.result} ${humanMs(d.durationMs)}`);
|
|
75
|
+
if (d.partial) {
|
|
76
|
+
lines.push(`⚠ PARTIAL — this run did NOT complete (exited on an unanswered gate); artifacts below are pre-failure.`);
|
|
77
|
+
if (d.unansweredGate)
|
|
78
|
+
lines.push(` gate: ${d.unansweredGate.message.split("\n")[0]}`);
|
|
79
|
+
}
|
|
80
|
+
lines.push(`artifacts (${d.artifacts.length}):`);
|
|
81
|
+
for (const a of d.artifacts) {
|
|
82
|
+
lines.push(` ${a.path} (${a.bytes} B)`);
|
|
83
|
+
if (a.preview && typeof a.preview === "object") {
|
|
84
|
+
for (const [k, v] of Object.entries(a.preview))
|
|
85
|
+
lines.push(` ${k}: ${typeof v === "string" ? v : JSON.stringify(v)}`);
|
|
86
|
+
}
|
|
87
|
+
else if (typeof a.preview === "string") {
|
|
88
|
+
lines.push(` ${a.preview}`);
|
|
89
|
+
}
|
|
90
|
+
}
|
|
91
|
+
if (!d.workDirAvailable && d.artifacts.some((a) => a.path.endsWith(".json"))) {
|
|
92
|
+
lines.push(` (work dir torn down — re-run with --keep to preview artifact contents)`);
|
|
93
|
+
}
|
|
94
|
+
return lines.join("\n");
|
|
95
|
+
}
|
package/dist/run/run.js
CHANGED
|
@@ -73,6 +73,14 @@ export class Run {
|
|
|
73
73
|
ctx() {
|
|
74
74
|
return { task: this.rec.transcript, transcript: () => this.rec.transcript, toolLog: () => this.toolLog, runId: this.rec.runId };
|
|
75
75
|
}
|
|
76
|
+
/** The in-progress record. When `drive()` throws on an unanswered gate, this holds everything accumulated
|
|
77
|
+
* up to the whiff (transcript, decisions, tool counts, artifacts-on-disk are separate) — the caller uses
|
|
78
|
+
* it to salvage a partial run instead of discarding the work. Fully initialized in the constructor, so
|
|
79
|
+
* it's a usable shell even on a very-early throw. The post-loop gate-delivery pairing has NOT run on the
|
|
80
|
+
* throw path, so `gateDeliveries` is empty here. */
|
|
81
|
+
partial() {
|
|
82
|
+
return this.rec;
|
|
83
|
+
}
|
|
76
84
|
/** Drive one-shot (string) or multi-turn (async iterable) and return the record. */
|
|
77
85
|
async drive(turns, startOpts) {
|
|
78
86
|
const turnIter = typeof turns === "string" ? oneShot(turns) : turns[Symbol.asyncIterator]();
|
package/dist/run/runs-gc.js
CHANGED
|
@@ -10,16 +10,16 @@ const DEFAULT_KEEP_LAST = 5;
|
|
|
10
10
|
* dir has neither → it is what GC should drop first. `events.jsonl` exists from session start, so an
|
|
11
11
|
* in-flight run is protected without a wall-clock guard. */
|
|
12
12
|
const isRealRun = (dir) => existsSync(join(dir, "result.json")) || existsSync(join(dir, "events.jsonl"));
|
|
13
|
-
/** `cowork-harness
|
|
13
|
+
/** `cowork-harness prune [--keep-last <n>] [--dry-run] [<runs-dir>]`
|
|
14
14
|
*
|
|
15
15
|
* For each scenario directory under the runs root, ranks EPHEMERAL run dirs by (1) real-run first (has
|
|
16
16
|
* result.json OR events.jsonl), (2) mtime descending, (3) name — then keeps the N most recent of that order
|
|
17
17
|
* and removes the rest. So an older COMPLETED run beats a newer empty scaffold dir for a keep slot, but
|
|
18
18
|
* `--keep-last` stays a HARD CAP (the ranking only decides WHICH N survive — never grows the kept count).
|
|
19
|
-
* Do NOT run `
|
|
19
|
+
* Do NOT run `prune` against an actively-writing runs root.
|
|
20
20
|
* Pinned `sess-*` dirs (persisted, resumable `--session-id` sessions) are retained unconditionally.
|
|
21
21
|
* The default root is the flat, machine-global `~/.cowork-harness/runs` (shared across projects), so a
|
|
22
|
-
* bare `
|
|
22
|
+
* bare `prune` prunes ephemeral runs from ALL projects; pass an explicit <runs-dir> to scope it.
|
|
23
23
|
* Safe by default (dry-run-able). */
|
|
24
24
|
export function cmdRunsGc(args) {
|
|
25
25
|
let p;
|
|
@@ -34,19 +34,19 @@ export function cmdRunsGc(args) {
|
|
|
34
34
|
return process.exit(2);
|
|
35
35
|
}
|
|
36
36
|
if (p.positionals.length > 1) {
|
|
37
|
-
log(`
|
|
37
|
+
log(`prune takes an optional <runs-dir> (got ${p.positionals.length}: ${p.positionals.join(", ")})`);
|
|
38
38
|
return process.exit(2);
|
|
39
39
|
}
|
|
40
40
|
const rawKeep = p.options["--keep-last"];
|
|
41
41
|
const keepLast = rawKeep !== undefined ? Number(rawKeep) : DEFAULT_KEEP_LAST;
|
|
42
42
|
if (!Number.isInteger(keepLast) || keepLast < 1) {
|
|
43
|
-
log(`
|
|
43
|
+
log(`prune: --keep-last must be a positive integer (got ${rawKeep})`);
|
|
44
44
|
return process.exit(2);
|
|
45
45
|
}
|
|
46
46
|
const dryRun = p.flags["--dry-run"] ?? false;
|
|
47
47
|
const runsRoot = p.positionals[0] ?? runsWriteRoot();
|
|
48
48
|
if (!existsSync(runsRoot)) {
|
|
49
|
-
log(`✓
|
|
49
|
+
log(`✓ prune: ${runsRoot} does not exist — nothing to prune`);
|
|
50
50
|
return process.exit(0);
|
|
51
51
|
}
|
|
52
52
|
let deleted = 0;
|
|
@@ -115,7 +115,7 @@ export function cmdRunsGc(args) {
|
|
|
115
115
|
}
|
|
116
116
|
}
|
|
117
117
|
log(deleted > 0
|
|
118
|
-
? `✓
|
|
119
|
-
: `✓
|
|
118
|
+
? `✓ prune: pruned ${deleted} run dir(s), kept ${kept}${dryRun ? " (dry-run — nothing deleted)" : ""}`
|
|
119
|
+
: `✓ prune: nothing to prune (${kept} run dir(s) within --keep-last ${keepLast})`);
|
|
120
120
|
return process.exit(0);
|
|
121
121
|
}
|
package/dist/run/scaffold.js
CHANGED
|
@@ -46,13 +46,20 @@ export function buildScaffold(eventsFile) {
|
|
|
46
46
|
answers.push({ when_question: escapeRx(q), choose: val });
|
|
47
47
|
}
|
|
48
48
|
}
|
|
49
|
+
// A partial run did NOT complete (it exited on an unanswered gate). Its artifacts are pre-failure and its
|
|
50
|
+
// result is "error", so neither should become an assertion the author trusts — that would scaffold a
|
|
51
|
+
// scenario asserting the half-finished output. Keep the gate answers (still worth locking) but drop the
|
|
52
|
+
// artifact/result asserts and warn loudly.
|
|
53
|
+
const partial = result.partial === true;
|
|
49
54
|
// Observed artifacts → file_exists; sub-agent count → dispatch_count_max; final result.
|
|
50
55
|
const assert = [];
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
+
if (!partial) {
|
|
57
|
+
for (const art of result.artifacts ?? [])
|
|
58
|
+
assert.push({ file_exists: art.path });
|
|
59
|
+
if ((result.subagents ?? []).length)
|
|
60
|
+
assert.push({ dispatch_count_max: (result.subagents ?? []).length });
|
|
61
|
+
assert.push({ result: result.result ?? "success" });
|
|
62
|
+
}
|
|
56
63
|
const scenario = {
|
|
57
64
|
// `result.baseline` is the recorded appVersion; a scenario pins `latest` (or a `desktop-<ver>`) — default
|
|
58
65
|
// to latest and let the author pin if they need a specific release.
|
|
@@ -70,8 +77,13 @@ export function buildScaffold(eventsFile) {
|
|
|
70
77
|
// Provenance signature — mirrors the cassette's `generator`/`cassetteVersion` fields. The scenario
|
|
71
78
|
// schema is `additionalProperties:false`, so it rides as a YAML comment (survives parse + re-serialize)
|
|
72
79
|
// rather than a structured field. Tool + version let a reader trace a committed scenario to its origin.
|
|
80
|
+
const partialMarker = partial
|
|
81
|
+
? `# PARTIAL run — this run did NOT complete (it exited on an unanswered gate). The artifact and result\n` +
|
|
82
|
+
`# assertions were omitted (the outputs are pre-failure). Re-run to completion and re-scaffold before trusting.\n`
|
|
83
|
+
: "";
|
|
73
84
|
return (`# generated by cowork-harness v${pkgVersion()} (scaffold) from ${runDir}\n` +
|
|
74
85
|
`# REVIEW before committing: tighten the when_question regexes to stable fragments, prune asserts you don't need.\n` +
|
|
86
|
+
partialMarker +
|
|
75
87
|
multiSelectMarker +
|
|
76
88
|
stringify(scenario));
|
|
77
89
|
}
|
package/dist/run/trace-view.js
CHANGED
|
@@ -9,7 +9,7 @@ import { parseMessage } from "../agent/session.js";
|
|
|
9
9
|
* convention used by the VM work dir (`lima.ts` `VM_WORK_HOST`). FLAT (shared across all projects on the
|
|
10
10
|
* machine): ephemeral `local_*` run dirs carry a unique hrtime id so they never path-collide; pinned
|
|
11
11
|
* `sess-*` dirs are deterministic, so a cross-project collision is guarded in `execute.ts` (the writer
|
|
12
|
-
* errors rather than overwriting another project's session) and `
|
|
12
|
+
* errors rather than overwriting another project's session) and `prune` never prunes them.
|
|
13
13
|
*/
|
|
14
14
|
export function defaultRunsHome() {
|
|
15
15
|
return join(homedir(), ".cowork-harness", "runs");
|
|
@@ -210,6 +210,29 @@ function readWorkspaceScript(workRoot, rel) {
|
|
|
210
210
|
* error (the module may simply be present, or the script may swallow the failure). Script reads are
|
|
211
211
|
* read-only, containment-checked, and bounded; missing files are skipped.
|
|
212
212
|
*/
|
|
213
|
+
/** Pre-run check: a skill that DECLARES the capability families its core path needs, run against an image
|
|
214
|
+
* that OMITS one, will hard-fail only after the (paid, ~12-min) run. Surface the build remedy up front so
|
|
215
|
+
* the operator fixes it before paying. Returns null when nothing is declared, all caps are present, or the
|
|
216
|
+
* probe could not run (`omitted` null) — i.e. only warns on a definite, actionable mismatch. */
|
|
217
|
+
export function capabilityPreflightWarning(declared, omitted) {
|
|
218
|
+
if (!declared.length || !omitted)
|
|
219
|
+
return null;
|
|
220
|
+
const missing = declared.filter((c) => omitted.includes(c));
|
|
221
|
+
if (!missing.length)
|
|
222
|
+
return null;
|
|
223
|
+
// State the gap + remedy only; the caller frames the consequence (abort pre-run, or a notice when
|
|
224
|
+
// allow_missing_capability is asserted) — so this string stays accurate in both contexts.
|
|
225
|
+
return (`skill declares requires_capabilities [${missing.join(", ")}] but the agent image omits them — ` +
|
|
226
|
+
`rebuild full parity (--build-arg COWORK_FULL_PARITY=1) and point COWORK_AGENT_IMAGE at it.`);
|
|
227
|
+
}
|
|
228
|
+
/** The pre-run capability decision: a DEFINITE gap (a declared family the image omits) **fails fast** before
|
|
229
|
+
* the paid run, unless the scenario opted out with `allow_missing_capability: true` (then it's a notice and
|
|
230
|
+
* the run proceeds — the author has declared the fallback equivalent). No gap / indefinite probe → no
|
|
231
|
+
* action. Keeping this a pure function makes the fail-fast-by-default policy unit-testable. */
|
|
232
|
+
export function capabilityPreflightDecision(declared, omitted, allowMissing) {
|
|
233
|
+
const message = capabilityPreflightWarning(declared, omitted);
|
|
234
|
+
return { abort: message !== null && !allowMissing, message };
|
|
235
|
+
}
|
|
213
236
|
export function detectCapabilityUse(eventsFile, omitted, workRoot) {
|
|
214
237
|
if (!omitted.length)
|
|
215
238
|
return [];
|
package/docs/scenario.md
CHANGED
|
@@ -261,6 +261,13 @@ fires even when the skill's fallback masks the gap. The check is computed at run
|
|
|
261
261
|
`verify-run`/`replay` honor the recorded outcome — a clean full-parity run records nothing and never
|
|
262
262
|
false-fails later. Opt out with `allow_missing_capability: true` when the fallback is genuinely equivalent.
|
|
263
263
|
|
|
264
|
+
When `requires_capabilities` is declared, the harness probes the image **before** driving and, if a declared
|
|
265
|
+
family is omitted, **fails fast — it aborts the run (exit 3) before spending a single token**, instead of
|
|
266
|
+
running ~12 min to a post-run hard-fail that's already known. Rebuild full parity
|
|
267
|
+
(`--build-arg COWORK_FULL_PARITY=1`) and point `COWORK_AGENT_IMAGE` at it, or assert
|
|
268
|
+
`allow_missing_capability: true` (which downgrades the abort to a notice and proceeds, same as it opts out of
|
|
269
|
+
the post-run check).
|
|
270
|
+
|
|
264
271
|
```yaml
|
|
265
272
|
requires_capabilities: [office_convert, pdf_tables] # fail unless the tier provides (and can verify) these
|
|
266
273
|
```
|
|
@@ -382,7 +389,8 @@ are) or need a real predicate over a skill's **structured JSON output**:
|
|
|
382
389
|
full Python callable with autocomplete and `print(d)`, strictly richer than anything a YAML string can
|
|
383
390
|
express. **If you're checking structured JSON content and already write Python, prefer the pytest lambda**
|
|
384
391
|
(a YAML content-predicate would be equal power with worse tooling). Find an artifact's real field paths by
|
|
385
|
-
running once with `--keep`
|
|
392
|
+
running once with `--keep`, then `cowork-harness inspect <run-dir>` (a shallow field preview of each JSON
|
|
393
|
+
artifact) or by reading the JSON under the run's `…/mnt/outputs/…` directly.
|
|
386
394
|
|
|
387
395
|
## Output
|
|
388
396
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "cowork-harness",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.13.0",
|
|
4
4
|
"description": "Scriptable, CI-friendly harness for Claude Cowork's runtime contract for testing skills across scenarios — same agent, mounts, egress allowlist, permission protocol, and sandbox limitations.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
package/schema/run-result.json
CHANGED
|
@@ -49,6 +49,8 @@
|
|
|
49
49
|
"workDir": { "type": "string" },
|
|
50
50
|
"outputsDir": { "type": "string" },
|
|
51
51
|
"artifacts": { "type": "array" },
|
|
52
|
+
"partial": { "type": "boolean" },
|
|
53
|
+
"unansweredGate": { "type": "object" },
|
|
52
54
|
"nonDeterministic": { "type": "boolean" },
|
|
53
55
|
"nonDeterministicTerminal": { "type": "boolean" },
|
|
54
56
|
"permissiveAutoAllow": {
|