cowork-harness 4.7.0 → 4.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +9 -6
- package/.claude/skills/cowork-harness/references/assertion-catalog-agents-skills-budgets.md +1 -1
- package/.claude/skills/cowork-harness/references/assertion-catalog-gates-hooks-modifiers.md +1 -1
- package/.claude/skills/cowork-harness/references/assertion-catalog-outcome-files-tools.md +5 -5
- package/.claude/skills/cowork-harness/references/assertion-catalog.md +4 -4
- package/.claude/skills/cowork-harness/references/assertions-guide.md +1 -1
- package/.claude/skills/cowork-harness/references/authoring.md +14 -1
- package/.claude/skills/cowork-harness/references/ci-recipe.md +8 -6
- package/.claude/skills/cowork-harness/references/critique.md +63 -8
- package/.claude/skills/cowork-harness/references/debugging.md +10 -4
- package/.claude/skills/cowork-harness/references/eval.md +4 -2
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/gotchas.md +1 -1
- package/.claude/skills/cowork-harness/references/hillclimb-recipe.md +1 -1
- package/.claude/skills/cowork-harness/references/hillclimb.md +21 -1
- package/.claude/skills/cowork-harness/references/measurement.md +1 -1
- package/.claude/skills/cowork-harness/references/run-record-replay.md +6 -5
- package/.claude/skills/cowork-harness/references/scenario-schema.md +10 -4
- package/.claude/skills/cowork-harness/references/semantic-judging.md +1 -1
- package/.claude/skills/cowork-harness/references/task-recipes.md +7 -5
- package/.claude/skills/cowork-harness/scripts/assertion-keys.json +1 -0
- package/.claude/skills/cowork-harness/scripts/scenario.py +67 -3
- package/CHANGELOG.md +202 -0
- package/DESIGN.md +1 -1
- package/README.md +8 -6
- package/SPEC.md +17 -11
- package/dist/answer-channel.js +1 -1
- package/dist/cli.js +3 -1
- package/dist/critique/command.js +322 -25
- package/dist/critique/compare.js +392 -0
- package/dist/critique/corpus-digest.js +55 -0
- package/dist/critique/evaluator.js +22 -1
- package/dist/critique/evidence.js +3 -0
- package/dist/critique/git-target.js +176 -0
- package/dist/critique/package-evidence.js +85 -5
- package/dist/critique/reply-repair.js +61 -0
- package/dist/critique/scrub-artifacts.js +49 -1
- package/dist/critique/summary.js +134 -0
- package/dist/eval/classify.js +5 -3
- package/dist/eval/snapshot.js +22 -21
- package/dist/hostloop/remote-devices-handler.js +115 -0
- package/dist/prompt.js +58 -5
- package/dist/run/cassette.js +81 -13
- package/dist/run/chat-result.js +1 -0
- package/dist/run/execute.js +79 -10
- package/dist/run/input-host-paths.js +5 -1
- package/dist/run/lane-notice.js +9 -0
- package/dist/run/run.js +81 -2
- package/dist/run/tier-vacuous-tools.js +41 -3
- package/dist/run/verdict.js +12 -3
- package/dist/run/verify-context.js +7 -0
- package/dist/runtime/argv.js +1 -1
- package/dist/runtime/container.js +32 -8
- package/dist/runtime/device-container.js +400 -0
- package/dist/runtime/hostloop.js +8 -6
- package/dist/session.js +18 -3
- package/dist/types.js +16 -1
- package/dist/usage-limit.js +28 -0
- package/docs/cassette.md +11 -6
- package/docs/ci.md +3 -2
- package/docs/cli.md +10 -8
- package/docs/companion-skill.md +2 -2
- package/docs/critique.md +187 -37
- package/docs/eval.md +19 -1
- package/docs/fidelity-gaps.md +83 -8
- package/docs/hillclimb.md +20 -0
- package/docs/invariants.md +1 -1
- package/docs/run-status.md +2 -1
- package/docs/scenario.md +39 -12
- package/docs/session.md +3 -2
- package/examples/replays/README.md +1 -1
- package/llms.txt +1 -1
- package/package.json +1 -1
- package/python/test_scenario_lint.py +60 -1
- package/schema/cassette.v17.json +471 -0
- package/schema/critique-compare.json +78 -0
- package/schema/critique-report.json +104 -3
- package/schema/critique-summary.json +122 -0
- package/schema/run-result.json +94 -4
- package/schema/scenario.schema.json +32 -1
- package/schema/session.schema.json +0 -1
- package/scripts/check-versions.ts +2 -1
- package/scripts/gen-schema.ts +13 -0
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, asserting artifacts, egress, or sub-agent dispatch, measuring how long each tool call took (toolDurations / trace), or debugging a failed run or verdict from its result.json or transcript. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. Also for comparing two versions of a skill before merging an edit — did it make answers worse? (`eval`: paired A/B, pinned models) — or improving one round by round (`hillclimb`, the `/claude-api hillclimb` runner). NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold / critique / stats / eval / hillclimb commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 4.
|
|
7
|
-
tracks-harness: cowork-harness 4.
|
|
6
|
+
version: 4.8.0
|
|
7
|
+
tracks-harness: cowork-harness 4.8.0 (baseline desktop-2.31226.1)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -26,7 +26,7 @@ allowlist). This skill exists mostly to keep you out of those traps — the *Inv
|
|
|
26
26
|
full landmine catalog in [`references/gotchas.md`](references/gotchas.md) are the highest-value part.
|
|
27
27
|
Read them.
|
|
28
28
|
|
|
29
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.
|
|
29
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.8.0` (baseline
|
|
30
30
|
> `desktop-2.31226.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
31
31
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
32
32
|
|
|
@@ -43,7 +43,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
43
43
|
|
|
44
44
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
45
45
|
- **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
|
|
46
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.
|
|
46
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.8.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@^4.8.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@^4.8.0"`. **Pin `@^4.8.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
|
|
47
47
|
|
|
48
48
|
This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
|
|
49
49
|
OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
|
|
@@ -90,7 +90,9 @@ CI-grade scenario, and the post-hoc debug loop; the rest are narrower tools that
|
|
|
90
90
|
`report.costUsd.totalUsd`. Reach for it when you want **findings**. **For "what does this skill
|
|
91
91
|
**DO**" — routing, artifact location, narration — use `skill` instead**: no evaluator, a fraction of
|
|
92
92
|
the cost, and it answers that question directly. Report and evidence-package shapes:
|
|
93
|
-
`references/critique.md
|
|
93
|
+
`references/critique.md` — which also covers grading a commit (`git:<ref>:<path>`), the
|
|
94
|
+
"has the skill changed?" hashes, the public `--summary-out` file, and `critique --compare` for
|
|
95
|
+
reading several critiques side by side.
|
|
94
96
|
- **Multi-turn / interactive reproduction** → `cowork-harness chat` (interactive; gates answered at the
|
|
95
97
|
TTY, **not** an asserted test — see *Debugging with `chat`* in `references/debugging.md`).
|
|
96
98
|
**"Interactive" splits two ways — don't take the wrong branch.** Want to answer gates yourself *and*
|
|
@@ -132,7 +134,8 @@ behind each, is [`references/gotchas.md`](references/gotchas.md).
|
|
|
132
134
|
7. **The tier decides what exists.** `protocol` has no sandbox and no egress, tool names differ per tier
|
|
133
135
|
(`container` serves `mcp__workspace__web_fetch`, not `WebFetch`), and every tier models Cowork's
|
|
134
136
|
local lane only; new Pro and Max tasks do not use it from 2026-10-06. Behaviour-shaped results transfer;
|
|
135
|
-
path, mount, delivery and egress results do not.
|
|
137
|
+
path, mount, delivery and egress results do not. `device_bridge: true` on `lane: remote` serves the cloud
|
|
138
|
+
task's `device_bash` (see `references/authoring.md`).
|
|
136
139
|
8. **A WARN signal never blocks a green.** Read the verdict signals after every run
|
|
137
140
|
(`prompt_asset_missing`, `undelivered_deliverables`, `model_fallback`, …).
|
|
138
141
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog: sub-agents, skills and connectors, budgets, tasks, delivery
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Part 2 of 3 of the per-key table. The conventions every key shares (an item with several
|
|
4
4
|
keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
5
5
|
[`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog: gates, hooks, path denial, verdict modifiers, egress, judged keys
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Part 3 of 3 of the per-key table. The conventions every key shares (an item with several
|
|
4
4
|
keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
5
5
|
[`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md).
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog: outcome, transcript, files and artifacts, tools, tool results
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Part 1 of 3 of the per-key table. The conventions every key shares (an item with several
|
|
4
4
|
keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
5
5
|
[`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
|
|
6
6
|
|
|
@@ -11,12 +11,12 @@ keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
|
11
11
|
| `transcript_not_contains: <str>` | it does not **Sees top-level `assistant_text` only — it excludes every `tool_use`/`tool_result`**, so text emitted inside a tool call (a gate question, an option label or description) can never match; use `question_context`/`question_asked` or `tool_result_contains` for those. |
|
|
12
12
|
| `transcript_matches: <regex>` | the transcript matches (case-insensitive) — for stochastic prose **Sees top-level `assistant_text` only — it excludes every `tool_use`/`tool_result`**, so text emitted inside a tool call (a gate question, an option label or description) can never match; use `question_context`/`question_asked` or `tool_result_contains` for those. |
|
|
13
13
|
| `transcript_not_matches: <regex>` | it does not match (e.g. no leaked stack trace) **Sees top-level `assistant_text` only — it excludes every `tool_use`/`tool_result`**, so text emitted inside a tool call (a gate question, an option label or description) can never match; use `question_context`/`question_asked` or `tool_result_contains` for those. |
|
|
14
|
-
| `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it. Object form `{path, authored}`: `authored: true` also requires that THIS run created or rewrote the file (an untouched pre-run file — e.g. from a `workspace_fixture` — fails; no pre-run manifest ⇒ evidence-unavailable); `authored: false` states that inheriting it is fine. On a `workspace_fixture` path one of the two is REQUIRED (refused at load otherwise). On `lane: remote` plain `file_exists` is the proxy for a written path; with `authored: true` it is rejected at load. |
|
|
14
|
+
| `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it. Object form `{path, authored}`: `authored: true` also requires that THIS run created or rewrote the file (an untouched pre-run file — e.g. from a `workspace_fixture` — fails; no pre-run manifest ⇒ evidence-unavailable); `authored: false` states that inheriting it is fine. On a `workspace_fixture` path one of the two is REQUIRED (refused at load otherwise). On `lane: remote` plain `file_exists` is the proxy for a written path; with `authored: true` it is rejected at load. Under `device_bridge`, a path in a connected folder is refused: the folders live in the device container; assert the `device_bash` call instead. |
|
|
15
15
|
| `user_visible_artifact: <path>` | exists **and** under a user-visible root (`outputs/` + each connected folder's mount name) — the right primitive for a workspace deliverable when a folder is connected. Takes the same `{path, authored}` object form as `file_exists` (and `artifact_text`/`artifact_json` take an `authored` field) |
|
|
16
16
|
| `no_delete_in_outputs: true` | no delete op touched `mnt/outputs` — **only `true` is valid**; `false` is rejected. Checks on **every** baseline. Omitting it: on a baseline recording outputs `rw` (Desktop before 2.16120.0) a detected delete still fails via the `outputs_delete` verdict signal, which fires *because* the key was not authored (`allow_outputs_delete: true` accepts an intended one); on `rwd` (2.16120.0+, including `latest`, where Cowork allows deletes in outputs) nothing checks outputs deletes, so author this key to keep the check. Detection is a post-run bash-command scan plus a per-turn filesystem diff of `outputs/`, not mount enforcement, so a green means none was *detected*. Fails when the diff proves the delete, when a delete in command/call position has an `outputs/` path as its own operand, or when the diff could not verify; a hit resting only on the detector's inference (e.g. a Python variable named `rm`) with a clean diff passes, and the `outputs_delete_unconfirmed` warn is still raised in the run output — see that code for the classes of real delete that land there |
|
|
17
|
-
| `no_delete_in_mounts: true` | no delete op touched `outputs` or any `rw` connected folder, except mounts waived by `allow_delete_in`. Covers `outputs` on **every** baseline, including those where Cowork allows an outputs delete (unless outputs is waived, authoring it arms the outputs check, so a filesystem-proven delete fails as the `outputs_delete` signal, which `allow_outputs_delete` waives). Production denies `unlink`/`rmdir` on a `rw` connected folder until approval, so `no_delete_in_outputs` covers only part of the rule. **only `true` is valid** |
|
|
17
|
+
| `no_delete_in_mounts: true` | no delete op touched `outputs` or any `rw` connected folder, except mounts waived by `allow_delete_in`. Covers `outputs` on **every** baseline, including those where Cowork allows an outputs delete (unless outputs is waived, authoring it arms the outputs check, so a filesystem-proven delete fails as the `outputs_delete` signal, which `allow_outputs_delete` waives). Production denies `unlink`/`rmdir` on a `rw` connected folder until approval, so `no_delete_in_outputs` covers only part of the rule. Under `device_bridge` it also reads `device_bash` commands and compares the device copy of each `rw` folder before and after the run. **only `true` is valid** |
|
|
18
18
|
| `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); an **incomplete post-run filesystem walk** (an unreadable subtree — a permission/I-O error) also fails evidence-unavailable rather than reporting "no strays" over a partial tree — distinct from the missing-manifest case (a `--resume` run), which fails for a different reason; captured on every live sandbox tier including microvm (its outputs are snapshotted from the VM into the run dir); replay needs `cassette.preRunPaths` (≥0.24 recordings) — cassettes without it **exclude** the key with a loud warning. ⚠️ **Not a stand-in for "file X must not exist":** there is no negative-existence key, and inverting this one is a different claim — it is new-files-only (a pre-existing file is invisible to it) and needs a pre-run manifest. Assert the content-level consequence instead (`tool_not_called`, or the absent `artifact_json`) rather than widening an allowlist for every incidental lock/temp file. Rejected at load on `lane: remote`. |
|
|
19
|
-
| `input_unmodified: <glob> \| [<glob>, …]` | a single glob or a list; every **pre-existing** file (incl. uploaded files under `uploads/**`) whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a glob that matches **no** pre-run path fails loud (a typo or renamed mount would otherwise verify zero files and pass vacuously); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail (a `--resume` run); captured on every live sandbox tier including microvm; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree. On `lane: remote` it still reads the local stand-in for the inputs, which is right there: an edit to a staged copy inside the cloud container does not reach the device. |
|
|
19
|
+
| `input_unmodified: <glob> \| [<glob>, …]` | a single glob or a list; every **pre-existing** file (incl. uploaded files under `uploads/**`) whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a glob that matches **no** pre-run path fails loud (a typo or renamed mount would otherwise verify zero files and pass vacuously); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail (a `--resume` run); captured on every live sandbox tier including microvm; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree. On `lane: remote` it still reads the local stand-in for the inputs, which is right there: an edit to a staged copy inside the cloud container does not reach the device. Under `device_bridge`, a glob on a connected folder's path is refused (the folders live in the device container). |
|
|
20
20
|
| `self_heal_ran: <bool>` | a bash command the model wrote did (not) name a plugin under `/sessions/<id>/mnt/.local-plugins/` or `/sessions/<id>/mnt/.remote-plugins/` — the plugin-root self-heal path. It reads the command as the model wrote it, so a host plugin path that `hostloop`'s bash tool rewrote to the VM mount does not count |
|
|
21
21
|
| `file_absent: <path>` | the named path does **not** exist under the work root after the run — the direct negative-existence key. Do NOT invert `no_unexpected_files` for this: that is an allowlist over NEW files only, and it needs a pre-run manifest. **LIVE/verify-run only** (a cassette records no walk health, so absence is unprovable on replay); rejected at load on `lane: remote`; evidence-unavailable on `preRunOrigin: remote-unavailable` |
|
|
22
22
|
| `artifact_text: {artifact, contains?: [..], not_contains?: [..], matches?, not_matches?}` | assert over a delivered artifact's TEXT body — `artifact_json`'s companion for non-JSON files, and how you prove an internal name did not leak into a file the user receives. Literal path (no glob), so one entry per delivered surface. Manifest-class; body-less / symlinked / over-cap targets fail evidence-unavailable, and a non-UTF-8 body fails the NEGATIVE matchers rather than passing against bytes it never read. Rejected at load on `lane: remote` (that lane's container filesystem is not locally observable) |
|
|
@@ -26,7 +26,7 @@ keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
|
26
26
|
| `tool_called: {tool: <glob> \| [..], input?, input_any?, result?, scope?, subagent_type?, count?}` | **object form** — asserts what a call CARRIED, where it RAN and what its paired result SAID, which no name glob and no `transcript_*` key can see (`transcript_*` reads top-level prose only, so it passes when the agent merely *says* it ran a command). `tool`: a glob or a list of globs (any-of; list both shells, `[Bash, mcp__workspace__bash]`, for a claim that must hold at hostloop too). `input: {<field>: <regex>}`: every named top-level input field must match (case-insensitive, unanchored; a missing field is no match; a non-string value is matched as JSON). `input_any: <regex>`: some top-level field matches. `result: {matches?, not_matches?, is_error?}`: predicates on the call's paired `tool_result` (by `toolUseId`, 10,240 characters of text — 32,768 for a top-level `Skill` result). `scope`: `main` (default — the main agent, including a `Skill`'s or `Agent(fork)`'s children, the set the string form reads), `subagent` (the parent is a sub-agent dispatch this run recorded, at any depth), or `any` (also calls whose parent is not a recorded dispatch, e.g. a `Skill` invoked inside a sub-agent). `subagent_type` (with `scope: subagent`): regex over the IMMEDIATE parent dispatch's type or description. `count: {min?, max?}` (default `min: 1`). **Fails closed:** an unpaired call never satisfies `result`; a truncated result or input that cannot settle a predicate, or a result.json without `toolCalls`, reports *evidence unavailable*, never a pass or a plain "not called". A red lists the calls it considered and names matches in another scope ("1 matching call in scope subagent — set `scope: any`"). `{tool: X}` alone is exactly `tool_called: X`. A cassette using this form stamps **v13**. |
|
|
27
27
|
| `tool_not_called: {tool: <glob> \| [..], input?, input_any?, result?, scope?, subagent_type?}` | **object form** — no call in scope satisfies every predicate (no `count`). Same fields and fail-closed rules as above: a candidate whose result is unpaired or truncated, or whose input was truncated, is *evidence unavailable*, never a pass. Refused at load when EVERY listed tool is one the tier does not serve. ⚠️ **Redaction hazard — this is the dangerous direction.** A committed cassette is redacted, which rewrites the recorded inputs: an `input` regex naming a literal the policy rewrites (a home path, an email) cannot see its target on replay. On replay, any candidate call whose field (or paired result, for `result.matches`) carries a redaction token is *evidence unavailable*, whatever the regex — the evaluator cannot know what the token replaced — so a negative check never passes over rewritten bytes. `record` warns — naming the redacted call and the ways out (narrow `scope`/`tool`, the string form, or live-only) — and refuses the write (the exact guard), and `lint` flags a regex naming a redactable literal (`tool-input-regex-redactable`) — a heuristic that reads `.cowork-redact.json` from the current and scenario directories only, not the cassette's. Match a part of the input redaction leaves alone (the verb and flags, a workspace-relative path), or keep the check on a live gate. Note also that the default `scope: main` does not see a sub-agent's call — use `scope: any` for "nothing anywhere ran X". |
|
|
28
28
|
| **legacy tool names** | The agent binary canonicalizes legacy spellings (`Task`→`Agent`, `KillShell`/`KillBash`→`TaskStop`, …) while the spawn tool list still declares the LEGACY one — so the init inventory shows `Task` and every call is emitted as `Agent`. `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` match EITHER spelling, globs included. Before this, `tool_called: "Task"` could never pass and `tool_not_called: "Task"` always did. |
|
|
29
|
-
| **tier vacuity — now REFUSED** | `tool_not_called` **and `subagent_tool_absent`** naming a tool the tier does not serve is rejected at scenario load (`Bash`/`WebFetch`/`NotebookEdit` at `hostloop`; `mcp__workspace__bash` at `container`/`microvm`), because it could never be violated. `scenario.py lint` WARNs on the same set before any run. The table is CLOSED to tools the harness itself removes or registers — `--tools` gates only the built-in set while every tier separately passes `--mcp-config`, so a session-MCP tool name is offered without appearing in any tool list and is never refused. Globs are never refused. `subagent_tool_absent` is covered because it reads the tools sub-agents actually USED, not a per-dispatch declared list. Under-approximates on purpose: `REPL` at hostloop is vacuous too and is not caught, which is the right side to err on for a hard refusal. |
|
|
29
|
+
| **tier vacuity — now REFUSED** | `tool_not_called` **and `subagent_tool_absent`** naming a tool the tier does not serve is rejected at scenario load (`Bash`/`WebFetch`/`NotebookEdit` at `hostloop`; `mcp__workspace__bash` at `container`/`microvm`; on those three tiers, `mcp__remote-devices__device_bash`/`get_device_info` without `device_bridge: true`, and every other `mcp__remote-devices__*` tool the harness never serves, bridge or not), because it could never be violated. `scenario.py lint` WARNs on the same set before any run. The table is CLOSED to tools the harness itself removes or registers — `--tools` gates only the built-in set while every tier separately passes `--mcp-config`, so a session-MCP tool name is offered without appearing in any tool list and is never refused. Globs are never refused. `subagent_tool_absent` is covered because it reads the tools sub-agents actually USED, not a per-dispatch declared list. Under-approximates on purpose: `REPL` at hostloop is vacuous too and is not caught, which is the right side to err on for a hard refusal. |
|
|
30
30
|
| `reference_read: <regex>` | a skill `references/`/`scripts/` file whose path matches this **regex** was ACCESSED — main agent or sub-agents, via `Read`, `Grep`/`Glob` (`path` input), or a `Bash`/`mcp__workspace__bash` command naming the path. Regex is **unanchored + case-insensitive** (the shared helper every regex key uses). **Under-approximates by design:** the path must be rooted in the mounted plugin, so a `cd` into the skill dir then a bare `cat references/x.md`, a heredoc body, and a `$VAR`-built path are invisible. Fails **evidence unavailable** when the run recorded no observable tool stream. Replay-capable (cassettes freeze whole tool inputs) |
|
|
31
31
|
| `no_observed_reference_access: <regex>` | no OBSERVED access matched the regex — the progressive-disclosure check: a reference the skill's routing never reaches. Named `observed` because detection under-approximates (see above), so it is **not proof the file went unread** — an agent that `cd`s and `cat`s it passes. Fails **evidence unavailable** rather than passing vacuously when no observable tool stream was recorded |
|
|
32
32
|
| `tool_result_contains: <str>` | a tool result includes the literal string (content / replay-checkable — substring match) |
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Every `assert:` key with its semantics, and the
|
|
4
4
|
verdict-signal table. Which keys survive `replay` is in [`scenario-schema.md`](./scenario-schema.md#replay-class);
|
|
5
5
|
the scenario and session YAML fields are there too.
|
|
6
6
|
|
|
@@ -47,7 +47,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
|
47
47
|
|---|---|---|
|
|
48
48
|
| `assertion` | fail | An authored `assert:` item failed |
|
|
49
49
|
| `result_error` | fail | The run's SDK result was `"error"` |
|
|
50
|
-
| `usage_limit` | fail | Usage/quota limit hit (not a skill failure) — retry after the limit resets. Emitted when `RunResult.resultErrorKind === "usage_limit"` |
|
|
50
|
+
| `usage_limit` | fail | Usage/quota limit hit (not a skill failure) — retry after the limit resets. Emitted when `RunResult.resultErrorKind === "usage_limit"` — including when a sub-agent hit the limit and the main loop ended `success`, where the message names the sub-agent and says to check its model and the account's plan instead of promising a reset (`decisions[]` row `subagent_usage_limit`) |
|
|
51
51
|
| `transport_error` | fail | The connection dropped mid/after-run |
|
|
52
52
|
| `permissive_auto_allow` | fail | A cowork-parity auto-allow real Cowork would block (opt out: `allow_permissive_auto_allow`) |
|
|
53
53
|
| `outputs_delete` | fail | An unauthorized delete touched `mnt/outputs` on a baseline that records outputs `rw` (not raised on `rwd`, Desktop 2.16120.0+, unless `no_delete_in_outputs` or `no_delete_in_mounts` (outputs not waived) is authored — with `no_delete_in_outputs` that assertion fails instead; with `no_delete_in_mounts` this signal itself fires, and its message names that key), confirmed: the per-turn filesystem diff proves it, a delete in command/call position has an `outputs/` path as its own operand, or the diff could not verify the turn. Authoring `no_delete_in_outputs` moves it into that assertion; `allow_outputs_delete` waives it |
|
|
@@ -57,7 +57,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
|
57
57
|
| `host_path_leak` | fail | A host path leaked into model-visible text (opt out: author `transcript_no_host_path`). A path copied verbatim from the scenario's own uploads, connected folders, prompt, or declared plugins' or local skills' files is not a leak |
|
|
58
58
|
| `l0_host_config_contamination` | fail | `protocol` ran against the operator's REAL config dir, so host-installed plugins/skills/MCP servers may have answered instead of the thing under test — the run did not necessarily measure it (opt out: `allow_l0_host_config_contamination`). Name predates the meaning: delivery at L0 works, contamination is what this reports |
|
|
59
59
|
| `missing_capability` | fail | A `requires_capabilities` need was unmet, or the skill used a capability the image omits (opt out: `allow_missing_capability`, or `skill --allow-missing-capability` on an open-ended run) |
|
|
60
|
-
| `infra_error` | fail | A supervising process died mid-run (VM/egress sidecar) — the run's evidence is contaminated, not author-suppressible |
|
|
60
|
+
| `infra_error` | fail | A supervising process died mid-run (the VM/egress sidecar, or the `device_bridge` container) — the run's evidence is contaminated, not author-suppressible |
|
|
61
61
|
| `stalled` | fail | The run ended asking for input with no tool work after the last gate. "Asking" = the final turn's closing sentence ends in `?`; or, once an `AskUserQuestion` gate has fired, it ends in `?` after trailing bold/quotes/`)`/emoji, is a `?` followed only by a `For example: …`/parenthetical aside, or is a request that says the input comes back to the agent (`Please share/provide/upload… so I can…`/`…here`, `Let me know which… you'd like me to use`, `Once you share… I'll…`, `I need X to proceed`, `Once I have the file, I'll…` after a sentence asking for it). Polite closers and hand-offs (`Let me know if…`, `Feel free…`, `with your`/`to your`, `before sending`, `If you…`) and a closing code block or blockquote never count. Hand-offs to a named third party (`with the team`, `to the founders`, `with the CFO`) never count either; `here` is a cue only after share/paste/upload/drop/reply/type/send or as the last word. English-only. Opt out with `allow_stall` (on `run`, `replay` and `eval`); without it, under `eval` a stalled rep is `errored_agent` and fails every row |
|
|
62
62
|
| `non_deterministic` | warn | The run was LLM/external/human-decided — not reproducible |
|
|
63
63
|
| `model_fallback` | warn | The agent fell back off the requested model mid-run (SDK `model_fallback` event); a `model_not_found`/`model_blocked` trigger repeats every run until the pin changes |
|
|
@@ -68,7 +68,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
|
68
68
|
| `delivery_unobservable` | warn | `lane: remote` only — the run produced file(s), but whether any reached the user CANNOT be verified: nothing is delivered by location on that lane and the harness models no remote delivery tool (production uses the agent-native `SendUserFile`). The honest counterpart to `undelivered_deliverables`, which would otherwise fire on every remote run that writes anything — a signal that always fires carries no information. Mutually exclusive with it; quiet when the run produced nothing to deliver. A harness coverage gap, not a skill defect. Opt out: `allow_undelivered_deliverables` |
|
|
69
69
|
| `partly_scripted_gate` | warn | A question batch (one `AskUserQuestion` with several sub-questions) that the scenario's `answers:` matched only PART of. Answers are delivered atomically, so the whole batch went to the `on_unanswered` fallback and the matched answers were NOT delivered — the fallback may contradict them. The message names the matched and unmatched sub-questions and who answered instead; `result.partlyScriptedGates` carries the full lists. Fires only on a partial match within one batch (a single-question gate no rule matched is the ordinary unanswered case). Re-derived on `replay` (from the cassette's frozen answers) and `verify-run` (from the current scenario). Fix: script every sub-question of the batch |
|
|
70
70
|
| `parked_at_question` | warn | The session declares `answer_channel: none` and the run ended on a question nobody can answer: its last message ends in `?`, even after tools ran (a finished answer ending on an offer warns too; a request without `?` does not fire). Takes the place of the `stalled` fail and never changes the verdict: with no answer channel, stopping at the question is the contract. Completion is judged from the file assertions such a scenario must carry (refused at load without one). No opt-out needed |
|
|
71
|
-
| `exec_infra_error` | warn | Host-loop
|
|
71
|
+
| `exec_infra_error` | warn | Host-loop, or a `device_bash` call under `device_bridge`: one or more container `exec` calls failed for infrastructure reasons, so those tool calls returned an error to the agent. Warns rather than fails because the run's other evidence is intact — unlike `infra_error`, where a dead supervisor contaminates everything. Caveat: if *every* exec failed, the agent ran nothing and this still only warns — check `result.infraErrors` |
|
|
72
72
|
|
|
73
73
|
A **fail**-severity signal does not change `result.result` (still `"success"`), but it DOES fail the
|
|
74
74
|
overall run verdict and exit code — `assert result: success` alone won't catch it; check
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertions guide
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it when choosing assertion keys: the two orthogonal axes and the goal → key map. The full catalog starts at `assertion-catalog.md`, which indexes the per-family files that hold every key's row.
|
|
4
4
|
|
|
5
5
|
### Assertions: two orthogonal axes
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Authoring a scenario
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it when composing a `scenarios/*.yaml`: session vs scenario, discovery, the fidelity tier, the answer path, `web_fetch`, and scaffold + lint.
|
|
4
4
|
|
|
5
5
|
## Part I — AUTHOR a scenario
|
|
6
6
|
|
|
@@ -74,6 +74,18 @@ reach the device), and
|
|
|
74
74
|
plain `file_exists` is the proxy for a written path. Delivery itself is the `delivery_unobservable` WARN (see
|
|
75
75
|
`run-record-replay.md`).
|
|
76
76
|
|
|
77
|
+
**`device_bridge: true`** (with `lane: remote` and `fidelity: container`) serves the remote lane's shell on the user's
|
|
78
|
+
computer, `mcp__remote-devices__device_bash`, from a second container holding COPIES of the connected folders at
|
|
79
|
+
`$HOME/mnt/<folder-name>`; the agent's own container gets no folder. Assert on that call, never on the folder's
|
|
80
|
+
files: `tool_called: {tool: mcp__remote-devices__device_bash, result: {matches: <text>}}`, plus `transcript_matches`.
|
|
81
|
+
`file_exists` / `input_unmodified` on a connected folder's path are refused under the bridge (they would read the
|
|
82
|
+
agent's container). `no_delete_in_mounts` covers the folders (the device copy allows a delete production refuses).
|
|
83
|
+
`tool_not_called` on `device_bash`/`get_device_info` without the bridge, or on a device tool the harness never serves
|
|
84
|
+
(with it or without), is refused as vacuous. Egress follows the session allowlist;
|
|
85
|
+
`egress.unrestricted: true` matches production's pass-through. With no `permission_mode` in the session the agent
|
|
86
|
+
runs in `acceptEdits` (a production cloud task's mode under Manual); a mode you set wins. The run's `deviceBridge` lists every known
|
|
87
|
+
difference from production.
|
|
88
|
+
|
|
77
89
|
### Choose an answer path (gates: AskUserQuestion + tool-permission)
|
|
78
90
|
|
|
79
91
|
Default to **deterministic**: scripted `answers:` + `on_unanswered: fail`. Anything that brings a
|
|
@@ -258,6 +270,7 @@ cowork-harness lint scenarios/*.yaml
|
|
|
258
270
|
| `host-path-assert-cowork` | WARN | `transcript_no_host_path` on `cowork` — it fails by design if the tier resolves to hostloop |
|
|
259
271
|
| `host-path-assert-tier` | ERROR | `transcript_no_host_path` on `hostloop` / `protocol`, where it fails by design |
|
|
260
272
|
| `lane-remote-incompatible-key` | ERROR | a key that can never pass on `lane: remote`, which the runtime refuses at load: `present_files_called` / `no_scratchpad_leak` / `user_visible_artifact` (delivery), `artifact_text` / `artifact_json` (file bodies), `file_absent` / `no_unexpected_files` (absence), `computer_links_resolve(_if_present)`, `no_lost_write_back`, `file_exists` with `authored: true`, and `semantic_matches` / `semantic_pairwise` with `evidence_files`. Tier rules are suppressed there |
|
|
273
|
+
| `device-bridge-misplaced` | ERROR | `device_bridge` set anywhere but `true` with `lane: remote` and `fidelity: container`; the loader refuses it |
|
|
261
274
|
| `linter-extra-findings-invalid` | ERROR | the loader findings `cowork-harness lint` hands the linter could not be read |
|
|
262
275
|
| `linter-unclassified-key` | ERROR | a valid assertion key this linter cannot classify (the linter is out of date) |
|
|
263
276
|
| `manifest-needs-snapshot` | INFO | manifest-backed keys, which evaluate on replay only when the cassette carries an `artifacts` manifest (not reported on `lane: remote` for `user_visible_artifact` / `artifact_json` / `artifact_text`, which cannot pass there) |
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 4.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -17,7 +17,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
|
|
|
17
17
|
CLI major reaches your workflow the moment it is promoted even though your `uses:` ref never changed — so a
|
|
18
18
|
copy-pasted recipe that omits the input takes a major bump with no say in it. `^4` holds the major, needs no
|
|
19
19
|
patch number to remember, and only wants a human decision at the next major. Pin an exact version
|
|
20
|
-
(e.g. `version: "4.
|
|
20
|
+
(e.g. `version: "4.8.0"`) instead when you want byte-reproducible CI.
|
|
21
21
|
|
|
22
22
|
Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
|
|
23
23
|
cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
|
|
@@ -82,7 +82,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
82
82
|
GitHub-hosted runners, no token/Docker/agent:
|
|
83
83
|
|
|
84
84
|
```yaml
|
|
85
|
-
- run: npm i -g "cowork-harness@^4.
|
|
85
|
+
- run: npm i -g "cowork-harness@^4.8.0"
|
|
86
86
|
- run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
|
|
87
87
|
# no silent false-greens. WITHOUT --strict this
|
|
88
88
|
# step cannot fail on a WARN-class rule (e.g.
|
|
@@ -380,7 +380,9 @@ A typical skill repo runs four stages, fastest/cheapest first:
|
|
|
380
380
|
```
|
|
381
381
|
|
|
382
382
|
The `jq -e` IS the gate — `--corpus-only` exits 0 on a measurement even over the ceiling. Same
|
|
383
|
-
packager, same git filter as the paid run; the number is a floor (a run-time read can only add).
|
|
383
|
+
packager, same git filter as the paid run; the number is a floor (a run-time read can only add). The same
|
|
384
|
+
payload's `skillTreeHash` (every delivered file of the skill) tells a job whether the skill changed since
|
|
385
|
+
its last critique, at no cost.
|
|
384
386
|
3. **Scenarios (replay)** — `cowork-harness replay cassettes/` on every PR (the committed `*.cassette.json`).
|
|
385
387
|
Token-free; content + structure + gate delivery.
|
|
386
388
|
4. **Parity / live (nightly, self-hosted)** — `cowork-harness run scenarios/` with a token + Docker +
|
|
@@ -409,7 +411,7 @@ jobs:
|
|
|
409
411
|
with: { node-version: '24' }
|
|
410
412
|
- uses: actions/setup-python@v5
|
|
411
413
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
412
|
-
- run: npm i -g "cowork-harness@^4.
|
|
414
|
+
- run: npm i -g "cowork-harness@^4.8.0"
|
|
413
415
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
414
416
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
415
417
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -438,7 +440,7 @@ jobs:
|
|
|
438
440
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
439
441
|
fi
|
|
440
442
|
- if: steps.guard.outputs.live == 'true'
|
|
441
|
-
run: npm i -g "cowork-harness@^4.
|
|
443
|
+
run: npm i -g "cowork-harness@^4.8.0"
|
|
442
444
|
- if: steps.guard.outputs.live == 'true'
|
|
443
445
|
run: cowork-harness run scenarios/ --output-format json
|
|
444
446
|
env:
|
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# Critique — the facts a plugin install can't otherwise reach
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). This is **not** a trim of the full
|
|
4
4
|
[`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
|
|
5
|
-
flags, cost,
|
|
5
|
+
flags, cost, reading several critiques, known limitations all live there). This file covers exactly what a
|
|
6
6
|
plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's
|
|
7
|
-
real field names, and what the evaluator was and was not shown. See **Recipe
|
|
7
|
+
real field names, and what the evaluator was and was not shown. See **Recipe 6** in `task-recipes.md` for
|
|
8
8
|
the harvest → reproduce → fix loop this feeds.
|
|
9
9
|
|
|
10
10
|
## The run-dir artifact a harvester reads
|
|
@@ -49,12 +49,12 @@ carry that; read all three before touching anything.
|
|
|
49
49
|
| `infraFailure` | the reason |
|
|
50
50
|
| `infraFailurePhase` | `task turn` (the graded run) or `reflection turn` (critique's own protocol turn) |
|
|
51
51
|
| `infraFailureKind` | why it failed — a harness `ErrCategory` (error envelope, exit 2/3) **or** a `resultErrorKind` (`usage_limit`/`transport`/`agent`) from a turn that RAN and errored (exit 1, top-level `error: null`). **Absent** = killed, or no envelope |
|
|
52
|
-
| `gradedErrorReason` | on a `taskResult: "error"` run (still gradeable, exit 0): why the GRADED turn errored, so a
|
|
52
|
+
| `gradedErrorReason` | on a `taskResult: "error"` run (still gradeable, exit 0): why the GRADED turn errored, so a dropped connection is not read as a skill defect. A task turn that hit the usage limit (its own result, or a sub-agent's) is not graded: `infraFailurePhase: "task turn"`, `infraFailureKind: "usage_limit"`, exit 2 |
|
|
53
53
|
|
|
54
54
|
**Do NOT read "has a kind" as "the instrument is fine".** The CLI's top-level catch turns every
|
|
55
55
|
unexpected throw into category **`internal`** — Docker down, container start failure, missing staged
|
|
56
56
|
agent, harness bug — and `runtime` carries a refused run dir. Only **`unanswered`, `usage`, `boundary`**
|
|
57
|
-
and, from the result-row taxonomy, **`usage_limit`** (quota
|
|
57
|
+
and, from the result-row taxonomy, **`usage_limit`** (the account's quota — the main loop's or a sub-agent's; on the task turn critique exits 2 with `RUN FAILED (task turn, usage_limit)` and spawns no reflection) and
|
|
58
58
|
**`transport`** (a tail-end drop) are the caller's problem. `agent` is not: for critique's own protocol
|
|
59
59
|
turn that IS the instrument breaking. The header encodes exactly that split and fails closed (an
|
|
60
60
|
unrecognized kind renders as infrastructure):
|
|
@@ -97,15 +97,23 @@ messages, never from the flag. So `graded model(s): unknown` means no assistant
|
|
|
97
97
|
(crash, kill, or a gate before the first reply); passing `--model` does not change that line. Past runs
|
|
98
98
|
can be checked without re-running: the same ids are in each kept run dir's `turns/1/result.json`.
|
|
99
99
|
|
|
100
|
+
## A repaired evaluator reply — `evaluatorRepair`
|
|
101
|
+
|
|
102
|
+
When an evaluator reply was complete except for its final closing `}`, critique appends that one brace and
|
|
103
|
+
parses as usual; the report records `evaluatorRepair: [{pass, appended}]` (summary: `evaluatorRepaired: true`).
|
|
104
|
+
Nothing else is ever repaired — a reply that would also need a `]` (a cut-off list) or was cut mid-item or
|
|
105
|
+
mid-string still fails (exit 2): re-run the critique.
|
|
106
|
+
|
|
100
107
|
## The report's item shape — no `title`, no `summary`
|
|
101
108
|
|
|
102
109
|
Each `items[]` entry's prose fields are **`idea`** and **`recommendedAction`** — there is no `title` field
|
|
103
110
|
and no `summary` field. A consumer that parses for either gets silently blank output instead of an error.
|
|
104
111
|
Full required set: `source` (`evaluator`|`self-report`), `idea`, `classification`, `evidence` (the cited
|
|
105
112
|
excerpt, verbatim-checked against the evidence package), `recommendedAction`. The optional
|
|
106
|
-
`findingFingerprint` hashes `idea`+`classification`+`recommendedAction` —
|
|
107
|
-
match
|
|
108
|
-
|
|
113
|
+
`findingFingerprint` hashes `idea`+`classification`+`recommendedAction` — exact-match on model wording, so a
|
|
114
|
+
LOWER bound: a match shows the same wording came back; across repeats of the same probe it often never recurs,
|
|
115
|
+
and no match shows nothing. Read repeat reports side by side, aligned by `classification`; a shared `evidence`
|
|
116
|
+
excerpt is a cue to compare two items, not a match (unrelated findings often cite the same passage).
|
|
109
117
|
|
|
110
118
|
## What the evaluator was actually shown — `evidenceBudget`
|
|
111
119
|
|
|
@@ -171,6 +179,53 @@ from `dist/` source:
|
|
|
171
179
|
| `trimRecord` | which section the transcript trim shaved, and by how much |
|
|
172
180
|
| `packageTruncated` | `true` if ANY section was cut — check this, not `corpusCuts`, for "was anything trimmed": a transcript-only cut leaves `corpusCuts` empty and would otherwise read as "nothing cut" |
|
|
173
181
|
|
|
182
|
+
## Has the skill changed? — the three hashes
|
|
183
|
+
|
|
184
|
+
Every report and every `--corpus-only` payload carries three `sha256:` hashes, keyed by path inside the mount (so
|
|
185
|
+
two clones of one commit hash alike):
|
|
186
|
+
|
|
187
|
+
| Hash | Covers | Use it for |
|
|
188
|
+
|---|---|---|
|
|
189
|
+
| `skillTreeHash` | every file staging delivered for the skill (`scripts/` included) + resolved agents + linked plugin-root references | "did ANYTHING change since the last critique?" — equal in `--corpus-only` and a graded run |
|
|
190
|
+
| `corpusHash` | the evaluator's static corpus (`SKILL.md`, `references/**`, agents, linked root refs) | a FLOOR: unchanged ≠ unchanged behaviour, since `scripts/` is not in it |
|
|
191
|
+
| `packagedCorpusHash` | what the packager put in the evaluator's corpus, incl. root files the agent READ and the ceiling's cuts | evidence of what was graded — moves with the agent's reading, never a "changed" key |
|
|
192
|
+
|
|
193
|
+
`corpusManifest` lists the files (diff two to see which one moved a hash); `hashBasis` is `git-tracked`
|
|
194
|
+
(working-tree bytes of tracked files), `worktree-all` or `git-commit`; untracked files under the skill are listed
|
|
195
|
+
in `skillTreeUntracked` (first 50; `skillTreeUntrackedCount` = total), never hashed; `.git` is never hashed. `corpusDrift` means a file changed during the run — critique a commit
|
|
196
|
+
instead.
|
|
197
|
+
|
|
198
|
+
## Critique a commit — `git:<ref>:<path>`
|
|
199
|
+
|
|
200
|
+
`critique git:<ref>:<path> …` grades a snapshot of the commit, so a moved HEAD or an edit mid-run changes nothing;
|
|
201
|
+
`source.commit` records the resolved id. A path at `<plugin>/skills/<name>` grades that skill of the plugin (same
|
|
202
|
+
as `<plugin> --skill <name>`). Refused before spend: a git filter such as LFS (`.gitattributes`,
|
|
203
|
+
`.git/info/attributes`, `core.attributesFile`), a symlink whose real path leaves the snapshot, a submodule. Works
|
|
204
|
+
with `--corpus-only` ($0). Kept snapshots: `~/.cowork-harness/critique-snapshots/` (or
|
|
205
|
+
`COWORK_HARNESS_CRITIQUE_SNAPSHOTS_DIR`); nothing prunes them.
|
|
206
|
+
|
|
207
|
+
## A public ledger line — `--summary-out`
|
|
208
|
+
|
|
209
|
+
`--summary-out <file>` writes a summary safe for a PUBLIC repo: ids, hashes, enums, and per finding only
|
|
210
|
+
`findingFingerprint` / `classification` / `source` / `adjudicable` — no finding text, prompt, host path or git
|
|
211
|
+
ref. Values are shape-checked (a failure is `null` + named in `withheld`); a summary a configured secret would
|
|
212
|
+
alter is NOT written (warning, exit unchanged). Cost and the probe hash are opt-in
|
|
213
|
+
(`--summary-include-cost`, `--summary-include-prompt-hash`). With `--summary-out`, a `--label` must match
|
|
214
|
+
`[A-Za-z0-9._:+-]{1,64}` (e.g. `v1.2.0-rc1`, not `fix/x`). Schema: [`schema/critique-summary.json`](https://github.com/yaniv-golan/cowork-harness/blob/main/schema/critique-summary.json).
|
|
215
|
+
|
|
216
|
+
## Several critiques side by side — `critique --compare`
|
|
217
|
+
|
|
218
|
+
`critique --compare <report.json | summary.json …>` ($0, no model) groups by `--label` (one group, or two for
|
|
219
|
+
before/after) and lists every finding per report, aligned by classification. It gives **no verdict**: an exact
|
|
220
|
+
`findingFingerprint` match is `sameWording` k/N — a LOWER bound (it often never recurs across repeats); a
|
|
221
|
+
`sharedExcerpt` is the same cited passage, NOT proven the same finding. Equal `corpusHash` and one shared
|
|
222
|
+
`skillTreeHash` in both groups = `noiseFloorControl` (a scripts/ edit never is). Refused: mixed skills / schemes /
|
|
223
|
+
harness majors, mixed `corpusHash` within a group, the same critique twice, a summary that withheld a key
|
|
224
|
+
field, a pre-4.8 report. Marked, and refused
|
|
225
|
+
by `--strict`: mixed evaluator models / hash bases / skill trees / probes, a pass-1-only member. Marked only:
|
|
226
|
+
`mixedPackagedCorpus`, `probeUnverified` (no prompt hash — the summary default).
|
|
227
|
+
Its output carries finding text — not public-safe. Details: [`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only).
|
|
228
|
+
|
|
174
229
|
## An untracked skill file is not graded
|
|
175
230
|
|
|
176
231
|
Staging delivers **git-tracked files only** (same rule as a marketplace plugin install) — an untracked
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Debugging a run
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it when a run misbehaved or a green looks wrong: triage, the observability output, and `chat`.
|
|
4
4
|
|
|
5
5
|
## Part III — Debug
|
|
6
6
|
|
|
@@ -107,7 +107,9 @@ decide which assertions from *Assertions: two orthogonal axes* in `assertions-gu
|
|
|
107
107
|
`semantic_*` with `evidence_files`);
|
|
108
108
|
`semantic_*` without it is judged on the transcript only. Reach for it
|
|
109
109
|
to check a skill's delivery survives the cloud lane, where new Pro and Max tasks run from 2026-10-06. Orthogonal to `fidelity` — a `lane: remote`
|
|
110
|
-
scenario still runs locally.
|
|
110
|
+
scenario still runs locally. Add `device_bridge: true` (with `fidelity: container`) to serve `device_bash`, the
|
|
111
|
+
cloud task's shell on the user's computer; a red there reads from the `device_bash` tool calls and the run's
|
|
112
|
+
`deviceBridge` record, and a `device-container` / `device-exec` infra error is that second container.
|
|
111
113
|
- **`cowork-harness stats [--metric <m>]`** — aggregate across the run index: `cost`, `duration`,
|
|
112
114
|
`tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`. Filters: `--since`/`--baseline`/`--branch`,
|
|
113
115
|
plus `--skill-hash <prefix>`/`--label <tag>` to narrow to ONE skill generation and
|
|
@@ -148,8 +150,12 @@ decide which assertions from *Assertions: two orthogonal axes* in `assertions-gu
|
|
|
148
150
|
**`stderrLogPath`** (the captured agent stderr) — read those and `trace <run-dir>` *before* re-running;
|
|
149
151
|
a re-record rarely tells you more than the captured stderr already does. Also check
|
|
150
152
|
**`resultErrorKind`** (`"transport" | "agent" | "usage_limit"`) before spending another paid run: a
|
|
151
|
-
`"usage_limit"` failure is
|
|
152
|
-
|
|
153
|
+
`"usage_limit"` failure is the account's quota, not a skill bug — don't debug the skill. It is also set
|
|
154
|
+
when a **sub-agent** hit the limit and the main loop ended `success` (then `errorSource: "agent"`): the
|
|
155
|
+
`usage_limit` verdict message names that sub-agent (dispatch id, type, limit text), and `decisions[]`
|
|
156
|
+
carries a `name: "subagent_usage_limit"` row (`detail: {toolUseId?, subagentType?, taskId?, error,
|
|
157
|
+
prior?}`). Retry once the quota allows; for a sub-agent's limit also check the model it is pinned to and
|
|
158
|
+
the account's plan — a model-scoped or credits limit does not reset on a timer; `"transport"`/`"agent"` means something actually broke, worth localizing before
|
|
153
159
|
re-running.
|
|
154
160
|
- **Attributing cost to sub-agent work.** `subagents[]` gives the dispatch tree — each sub-agent's
|
|
155
161
|
`dispatchModel`/`resolvedModel`, `toolsUsed`, `prompt`/`output`, and `attributedSkillId` — but **not** its own token/cost;
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# `eval` — paired before/after comparison of a skill edit (EXPERIMENTAL)
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). The full guide is
|
|
4
4
|
[docs/eval.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/eval.md); this is the part you
|
|
5
5
|
need while running it.
|
|
6
6
|
|
|
@@ -48,7 +48,9 @@ cowork-harness eval report <eval-dir> # rebuild the report from the eval dir
|
|
|
48
48
|
| `insufficient` | too few valid reps in an arm (4 of 5 needed by default) |
|
|
49
49
|
|
|
50
50
|
A drop is a signal to investigate, not proof: open the run dirs the report links for that row. In order,
|
|
51
|
-
first match wins: an infrastructure failure is excluded and reported — including a
|
|
51
|
+
first match wins: an infrastructure failure is excluded and reported — including a usage limit the run lane
|
|
52
|
+
named (the main loop's or a sub-agent's, rule `kind_usage_limit`, even on a rep that otherwise timed out or hit
|
|
53
|
+
`error_max_turns`), and a rep where no model
|
|
52
54
|
answered (the agent's `Not logged in` / `Authentication required` reply, rule `auth`; a usage or
|
|
53
55
|
spend limit as its final message, even on a nonzero exit after spend, rule `usage_limit`; or only
|
|
54
56
|
`<synthetic>` models at $0, rule `no_model_answered`). An agent-caused failure (timeout, max turns,
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 4.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`).
|
|
4
4
|
|
|
5
5
|
> **This page vs. the repo docs.** This is the **offline snapshot** that ships inside the installed
|
|
6
6
|
> plugin — it is self-contained on purpose. The repo carries four other fidelity views, each answering a
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Gotchas
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). The full "✓ passed ≠ correct" landmine catalog.
|
|
4
4
|
|
|
5
5
|
## Gotchas — the "✓ passed ≠ correct" landmines
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Recipe 7 — Climb a skill with `/claude-api hillclimb` and the harness as its runner
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
|
|
4
4
|
`hillclimb --help` lists `--skill` (help goes to stderr). This page is the loop's procedure, step by step, in the order of the
|
|
5
5
|
`/claude-api hillclimb` guide. Every mechanic (flags, refusals, the gate, `regrade`, `freeze-ref`, exit codes,
|
|
6
6
|
row keys) is in [`hillclimb.md`](hillclimb.md); the setup and the full list of differences from the guide's own
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# `hillclimb` — the runner for a `/claude-api hillclimb` loop
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
|
|
4
4
|
`hillclimb --help` lists `--skill` (help goes to stderr). The command reference is
|
|
5
5
|
[docs/cli.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cli.md); this is the part a loop needs
|
|
6
6
|
while it runs. It covers `run`, `check`, `state-template`, `freeze-ref` and `regrade`.
|
|
@@ -382,6 +382,26 @@ Flags: `--flow DIR`, `--variant all|baseline|v<N>` (default `all`: every variant
|
|
|
382
382
|
row, never by `--allow-doc-drift`); one whose assert has an edit inside a scrubbed literal (only a re-run applies
|
|
383
383
|
it); and an open `judge_invalid` slot.
|
|
384
384
|
|
|
385
|
+
## Cutting the cost per rep
|
|
386
|
+
|
|
387
|
+
Every rep re-runs the whole scenario, so a case that tests the last step of a long pipeline pays for every step
|
|
388
|
+
before it on every rep. Start such a case from a workspace fixture instead: a directory whose files are copied into
|
|
389
|
+
the session's `outputs/` before turn 1, so the prompt asks for the late step alone
|
|
390
|
+
([Starting from a saved workspace](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md#starting-from-a-saved-workspace-workspace_fixture)).
|
|
391
|
+
|
|
392
|
+
- **Make one from a kept run.** `cowork-harness fixture export <run-dir> --out <dir>`, then point the case's
|
|
393
|
+
`workspace_fixture:` at `<dir>` and commit both ([fixture export](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cli.md#exporting-a-runs-outputs-as-a-fixture-fixture-export)).
|
|
394
|
+
Add `--session-paths` when the skill reads back paths it recorded (an outputs-dir probe, a sub-agent's output
|
|
395
|
+
path). `--exclude <path>` leaves out a file the step re-creates, such as a probe the skill re-runs.
|
|
396
|
+
- **Limits.** A fixture with session-path tokens is refused on the `protocol` tier, and `--session-paths` refuses a
|
|
397
|
+
protocol run. Binary files are never rewritten; one that holds a token is refused. Any other host path in a text
|
|
398
|
+
file is still refused (`--allow-host-paths` accepts an ordinary one, never a path into a run dir).
|
|
399
|
+
- **What the case then measures.** Only the steps after the fixture. An untouched fixture file is pre-run, not
|
|
400
|
+
authored, so a judge grades only what this step wrote, and the conversation starts fresh. The earlier steps are no
|
|
401
|
+
longer exercised: keep a full-pipeline case for them.
|
|
402
|
+
- **The fixture is in the harness gate.** Its files are hashed with the scenario, so an edited or re-exported fixture
|
|
403
|
+
changes the sha and needs `--approve-harness` again.
|
|
404
|
+
|
|
385
405
|
## Exit codes
|
|
386
406
|
|
|
387
407
|
- `run`: `0` every attempted (case, rep) was scored; `1` an attempt failed (an `errors.jsonl` row, or a scored
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Measurement
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it before comparing runs: `--repeat`, `--ablate-skill`, and the hygiene that keeps a batch valid.
|
|
4
4
|
|
|
5
5
|
### Measure — before/after, with/without (`--repeat`, `--ablate-skill`)
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Run, record and lock
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.0` (baseline `desktop-2.31226.1`). Read it when running a scenario, recording or placing a cassette, reading verdict signals, checking a background run, or choosing CI lanes.
|
|
4
4
|
|
|
5
5
|
## Part II — RUN, RECORD & LOCK
|
|
6
6
|
|
|
@@ -132,8 +132,9 @@ current skill") rather than vouch against stale labels, but the trace/inspect pa
|
|
|
132
132
|
deliberately. (Same fail-closed family: corrupt gate evidence — unparseable `events.jsonl` lines, or fewer
|
|
133
133
|
gates than `trace.json` recorded questions — a structurally invalid `result.json`, a `command:"replay"`
|
|
134
134
|
result (a replay is a re-check of a recorded cassette, not run evidence — verify the original live run dir),
|
|
135
|
-
|
|
136
|
-
|
|
135
|
+
a `mode:"chat"` result (chat carries no assertions or verdict by contract), and a `device_bridge` run dir (its
|
|
136
|
+
folders lived in the device container; re-run or replay the cassette, in `regrade` and `refs` too) also refuse
|
|
137
|
+
rather than certify.) (A token-free probe of "which gates fire" isn't possible — gates are model-decided per run.)
|
|
137
138
|
|
|
138
139
|
Run artifacts are written to `~/.cowork-harness/runs/…` by default — **outside any working tree**, so a run
|
|
139
140
|
launched from a repo root never drops sensitive skill inputs/outputs into it. Pass `--run-dir <path>` (or set
|
|
@@ -203,7 +204,7 @@ cassette — has its own recipe:
|
|
|
203
204
|
`--label <tag>`); an un-split window spanning more than one generation now warns.
|
|
204
205
|
**Multi-skill plugin caveat (post-1.7.0 CLIs with `--skill`):**
|
|
205
206
|
skillHash keys the whole MOUNTED plugin, so on a multi-skill plugin the hash alone cross-pairs
|
|
206
|
-
critiques of DIFFERENT skills — pair by the report's `(gradedSkillHash, gradedSkill)
|
|
207
|
+
critiques of DIFFERENT skills — pair critiques by the report's per-skill `skillTreeHash` / `corpusHash` (4.8.0+), or by `(gradedSkillHash, gradedSkill)`. **On a pre-1.5.0 CLI the `skill` lane emits no `fingerprint.skillHash` at all**, so a
|
|
207
208
|
pairing step there silently groups on an absent key instead of erroring — check the field is present, or
|
|
208
209
|
require ≥ 1.5.0. See [`docs/debugging.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/debugging.md)
|
|
209
210
|
(repo-only) for the full loop.
|
|
@@ -299,7 +300,7 @@ Recognize these before "fixing" a non-bug:
|
|
|
299
300
|
skill's own files never exempt a path under its host source); `scan.inputHostPathTokens` counts the
|
|
300
301
|
host-path tokens the inputs carried, `scan.hostPathsFromInputs` the ones exempted, and a non-zero exemption
|
|
301
302
|
prints a `::notice::` so a clean scan that relied on it is never silent.
|
|
302
|
-
- **`exec_infra_error`** (`WARN`, host-loop) — one or more container `exec` calls failed for
|
|
303
|
+
- **`exec_infra_error`** (`WARN`, host-loop, or `device_bash` under `device_bridge`) — one or more container `exec` calls failed for
|
|
303
304
|
infrastructure reasons (daemon/container-level), so those tool calls returned an error to the agent
|
|
304
305
|
rather than the command's own output. Warn-severity because the run's other evidence is intact — unlike
|
|
305
306
|
the fail-severity `infra_error`, where a **supervising process** died and contaminated everything. Note
|