cowork-harness 4.7.1 → 4.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +9 -6
- package/.claude/skills/cowork-harness/references/assertion-catalog-agents-skills-budgets.md +1 -1
- package/.claude/skills/cowork-harness/references/assertion-catalog-gates-hooks-modifiers.md +1 -1
- package/.claude/skills/cowork-harness/references/assertion-catalog-outcome-files-tools.md +5 -5
- package/.claude/skills/cowork-harness/references/assertion-catalog.md +3 -3
- package/.claude/skills/cowork-harness/references/assertions-guide.md +1 -1
- package/.claude/skills/cowork-harness/references/authoring.md +14 -1
- package/.claude/skills/cowork-harness/references/ci-recipe.md +8 -6
- package/.claude/skills/cowork-harness/references/critique.md +61 -6
- package/.claude/skills/cowork-harness/references/debugging.md +4 -2
- package/.claude/skills/cowork-harness/references/eval.md +1 -1
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/gotchas.md +1 -1
- package/.claude/skills/cowork-harness/references/hillclimb-recipe.md +1 -1
- package/.claude/skills/cowork-harness/references/hillclimb.md +1 -1
- package/.claude/skills/cowork-harness/references/measurement.md +1 -1
- package/.claude/skills/cowork-harness/references/run-record-replay.md +6 -5
- package/.claude/skills/cowork-harness/references/scenario-schema.md +10 -4
- package/.claude/skills/cowork-harness/references/semantic-judging.md +1 -1
- package/.claude/skills/cowork-harness/references/task-recipes.md +7 -5
- package/.claude/skills/cowork-harness/scripts/assertion-keys.json +1 -0
- package/.claude/skills/cowork-harness/scripts/scenario.py +67 -3
- package/CHANGELOG.md +182 -0
- package/README.md +8 -6
- package/RELEASING.md +4 -1
- package/SPEC.md +15 -9
- package/baselines/desktop-2.31226.1.json +81 -0
- package/dist/answer-channel.js +1 -1
- package/dist/cli.js +29 -8
- package/dist/critique/command.js +288 -15
- package/dist/critique/compare.js +392 -0
- package/dist/critique/corpus-digest.js +55 -0
- package/dist/critique/evaluator.js +22 -1
- package/dist/critique/evidence.js +3 -0
- package/dist/critique/git-target.js +176 -0
- package/dist/critique/package-evidence.js +85 -5
- package/dist/critique/reply-repair.js +61 -0
- package/dist/critique/scrub-artifacts.js +49 -1
- package/dist/critique/summary.js +134 -0
- package/dist/eval/snapshot.js +22 -21
- package/dist/hostloop/remote-devices-handler.js +115 -0
- package/dist/prompt.js +58 -5
- package/dist/run/cassette.js +81 -13
- package/dist/run/chat-result.js +1 -0
- package/dist/run/execute.js +79 -10
- package/dist/run/input-host-paths.js +5 -1
- package/dist/run/lane-notice.js +9 -0
- package/dist/run/run.js +1 -1
- package/dist/run/tier-vacuous-tools.js +41 -3
- package/dist/run/verdict.js +4 -2
- package/dist/run/verify-context.js +7 -0
- package/dist/runtime/argv.js +1 -1
- package/dist/runtime/container.js +32 -8
- package/dist/runtime/device-container.js +400 -0
- package/dist/runtime/hostloop.js +9 -7
- package/dist/session.js +18 -3
- package/dist/sync/baseline-diff.js +92 -4
- package/dist/sync/cowork-sync.js +80 -216
- package/dist/sync/observed-spawn-env.js +350 -0
- package/dist/sync/spawn-env-post-writes.js +371 -0
- package/dist/sync/spawn-env-registry.js +460 -0
- package/dist/types.js +55 -1
- package/docs/cassette.md +11 -6
- package/docs/ci.md +3 -2
- package/docs/cli.md +11 -9
- package/docs/companion-skill.md +2 -2
- package/docs/critique.md +178 -31
- package/docs/fidelity-gaps.md +83 -8
- package/docs/invariants.md +3 -2
- package/docs/maintenance.md +50 -3
- package/docs/scenario.md +39 -12
- package/docs/session.md +3 -2
- package/examples/replays/README.md +1 -1
- package/llms.txt +1 -1
- package/package.json +1 -1
- package/python/test_scenario_lint.py +60 -1
- package/schema/cassette.v17.json +471 -0
- package/schema/critique-compare.json +78 -0
- package/schema/critique-report.json +103 -2
- package/schema/critique-summary.json +122 -0
- package/schema/run-result.json +92 -2
- package/schema/scenario.schema.json +32 -1
- package/schema/session.schema.json +0 -1
- package/scripts/check-versions.ts +2 -1
- package/scripts/gen-schema.ts +13 -0
- package/scripts/release-preflight.ts +39 -0
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, asserting artifacts, egress, or sub-agent dispatch, measuring how long each tool call took (toolDurations / trace), or debugging a failed run or verdict from its result.json or transcript. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. Also for comparing two versions of a skill before merging an edit — did it make answers worse? (`eval`: paired A/B, pinned models) — or improving one round by round (`hillclimb`, the `/claude-api hillclimb` runner). NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold / critique / stats / eval / hillclimb commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 4.
|
|
7
|
-
tracks-harness: cowork-harness 4.
|
|
6
|
+
version: 4.8.1
|
|
7
|
+
tracks-harness: cowork-harness 4.8.1 (baseline desktop-2.31226.1)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -26,7 +26,7 @@ allowlist). This skill exists mostly to keep you out of those traps — the *Inv
|
|
|
26
26
|
full landmine catalog in [`references/gotchas.md`](references/gotchas.md) are the highest-value part.
|
|
27
27
|
Read them.
|
|
28
28
|
|
|
29
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.
|
|
29
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.8.1` (baseline
|
|
30
30
|
> `desktop-2.31226.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
31
31
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
32
32
|
|
|
@@ -43,7 +43,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
43
43
|
|
|
44
44
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
45
45
|
- **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
|
|
46
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.
|
|
46
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.8.1**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@^4.8.1" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@^4.8.1"`. **Pin `@^4.8.1`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
|
|
47
47
|
|
|
48
48
|
This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
|
|
49
49
|
OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
|
|
@@ -90,7 +90,9 @@ CI-grade scenario, and the post-hoc debug loop; the rest are narrower tools that
|
|
|
90
90
|
`report.costUsd.totalUsd`. Reach for it when you want **findings**. **For "what does this skill
|
|
91
91
|
**DO**" — routing, artifact location, narration — use `skill` instead**: no evaluator, a fraction of
|
|
92
92
|
the cost, and it answers that question directly. Report and evidence-package shapes:
|
|
93
|
-
`references/critique.md
|
|
93
|
+
`references/critique.md` — which also covers grading a commit (`git:<ref>:<path>`), the
|
|
94
|
+
"has the skill changed?" hashes, the public `--summary-out` file, and `critique --compare` for
|
|
95
|
+
reading several critiques side by side.
|
|
94
96
|
- **Multi-turn / interactive reproduction** → `cowork-harness chat` (interactive; gates answered at the
|
|
95
97
|
TTY, **not** an asserted test — see *Debugging with `chat`* in `references/debugging.md`).
|
|
96
98
|
**"Interactive" splits two ways — don't take the wrong branch.** Want to answer gates yourself *and*
|
|
@@ -132,7 +134,8 @@ behind each, is [`references/gotchas.md`](references/gotchas.md).
|
|
|
132
134
|
7. **The tier decides what exists.** `protocol` has no sandbox and no egress, tool names differ per tier
|
|
133
135
|
(`container` serves `mcp__workspace__web_fetch`, not `WebFetch`), and every tier models Cowork's
|
|
134
136
|
local lane only; new Pro and Max tasks do not use it from 2026-10-06. Behaviour-shaped results transfer;
|
|
135
|
-
path, mount, delivery and egress results do not.
|
|
137
|
+
path, mount, delivery and egress results do not. `device_bridge: true` on `lane: remote` serves the cloud
|
|
138
|
+
task's `device_bash` (see `references/authoring.md`).
|
|
136
139
|
8. **A WARN signal never blocks a green.** Read the verdict signals after every run
|
|
137
140
|
(`prompt_asset_missing`, `undelivered_deliverables`, `model_fallback`, …).
|
|
138
141
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog: sub-agents, skills and connectors, budgets, tasks, delivery
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). Part 2 of 3 of the per-key table. The conventions every key shares (an item with several
|
|
4
4
|
keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
5
5
|
[`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog: gates, hooks, path denial, verdict modifiers, egress, judged keys
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). Part 3 of 3 of the per-key table. The conventions every key shares (an item with several
|
|
4
4
|
keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
5
5
|
[`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md).
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog: outcome, transcript, files and artifacts, tools, tool results
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). Part 1 of 3 of the per-key table. The conventions every key shares (an item with several
|
|
4
4
|
keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
5
5
|
[`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
|
|
6
6
|
|
|
@@ -11,12 +11,12 @@ keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
|
11
11
|
| `transcript_not_contains: <str>` | it does not **Sees top-level `assistant_text` only — it excludes every `tool_use`/`tool_result`**, so text emitted inside a tool call (a gate question, an option label or description) can never match; use `question_context`/`question_asked` or `tool_result_contains` for those. |
|
|
12
12
|
| `transcript_matches: <regex>` | the transcript matches (case-insensitive) — for stochastic prose **Sees top-level `assistant_text` only — it excludes every `tool_use`/`tool_result`**, so text emitted inside a tool call (a gate question, an option label or description) can never match; use `question_context`/`question_asked` or `tool_result_contains` for those. |
|
|
13
13
|
| `transcript_not_matches: <regex>` | it does not match (e.g. no leaked stack trace) **Sees top-level `assistant_text` only — it excludes every `tool_use`/`tool_result`**, so text emitted inside a tool call (a gate question, an option label or description) can never match; use `question_context`/`question_asked` or `tool_result_contains` for those. |
|
|
14
|
-
| `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it. Object form `{path, authored}`: `authored: true` also requires that THIS run created or rewrote the file (an untouched pre-run file — e.g. from a `workspace_fixture` — fails; no pre-run manifest ⇒ evidence-unavailable); `authored: false` states that inheriting it is fine. On a `workspace_fixture` path one of the two is REQUIRED (refused at load otherwise). On `lane: remote` plain `file_exists` is the proxy for a written path; with `authored: true` it is rejected at load. |
|
|
14
|
+
| `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it. Object form `{path, authored}`: `authored: true` also requires that THIS run created or rewrote the file (an untouched pre-run file — e.g. from a `workspace_fixture` — fails; no pre-run manifest ⇒ evidence-unavailable); `authored: false` states that inheriting it is fine. On a `workspace_fixture` path one of the two is REQUIRED (refused at load otherwise). On `lane: remote` plain `file_exists` is the proxy for a written path; with `authored: true` it is rejected at load. Under `device_bridge`, a path in a connected folder is refused: the folders live in the device container; assert the `device_bash` call instead. |
|
|
15
15
|
| `user_visible_artifact: <path>` | exists **and** under a user-visible root (`outputs/` + each connected folder's mount name) — the right primitive for a workspace deliverable when a folder is connected. Takes the same `{path, authored}` object form as `file_exists` (and `artifact_text`/`artifact_json` take an `authored` field) |
|
|
16
16
|
| `no_delete_in_outputs: true` | no delete op touched `mnt/outputs` — **only `true` is valid**; `false` is rejected. Checks on **every** baseline. Omitting it: on a baseline recording outputs `rw` (Desktop before 2.16120.0) a detected delete still fails via the `outputs_delete` verdict signal, which fires *because* the key was not authored (`allow_outputs_delete: true` accepts an intended one); on `rwd` (2.16120.0+, including `latest`, where Cowork allows deletes in outputs) nothing checks outputs deletes, so author this key to keep the check. Detection is a post-run bash-command scan plus a per-turn filesystem diff of `outputs/`, not mount enforcement, so a green means none was *detected*. Fails when the diff proves the delete, when a delete in command/call position has an `outputs/` path as its own operand, or when the diff could not verify; a hit resting only on the detector's inference (e.g. a Python variable named `rm`) with a clean diff passes, and the `outputs_delete_unconfirmed` warn is still raised in the run output — see that code for the classes of real delete that land there |
|
|
17
|
-
| `no_delete_in_mounts: true` | no delete op touched `outputs` or any `rw` connected folder, except mounts waived by `allow_delete_in`. Covers `outputs` on **every** baseline, including those where Cowork allows an outputs delete (unless outputs is waived, authoring it arms the outputs check, so a filesystem-proven delete fails as the `outputs_delete` signal, which `allow_outputs_delete` waives). Production denies `unlink`/`rmdir` on a `rw` connected folder until approval, so `no_delete_in_outputs` covers only part of the rule. **only `true` is valid** |
|
|
17
|
+
| `no_delete_in_mounts: true` | no delete op touched `outputs` or any `rw` connected folder, except mounts waived by `allow_delete_in`. Covers `outputs` on **every** baseline, including those where Cowork allows an outputs delete (unless outputs is waived, authoring it arms the outputs check, so a filesystem-proven delete fails as the `outputs_delete` signal, which `allow_outputs_delete` waives). Production denies `unlink`/`rmdir` on a `rw` connected folder until approval, so `no_delete_in_outputs` covers only part of the rule. Under `device_bridge` it also reads `device_bash` commands and compares the device copy of each `rw` folder before and after the run. **only `true` is valid** |
|
|
18
18
|
| `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); an **incomplete post-run filesystem walk** (an unreadable subtree — a permission/I-O error) also fails evidence-unavailable rather than reporting "no strays" over a partial tree — distinct from the missing-manifest case (a `--resume` run), which fails for a different reason; captured on every live sandbox tier including microvm (its outputs are snapshotted from the VM into the run dir); replay needs `cassette.preRunPaths` (≥0.24 recordings) — cassettes without it **exclude** the key with a loud warning. ⚠️ **Not a stand-in for "file X must not exist":** there is no negative-existence key, and inverting this one is a different claim — it is new-files-only (a pre-existing file is invisible to it) and needs a pre-run manifest. Assert the content-level consequence instead (`tool_not_called`, or the absent `artifact_json`) rather than widening an allowlist for every incidental lock/temp file. Rejected at load on `lane: remote`. |
|
|
19
|
-
| `input_unmodified: <glob> \| [<glob>, …]` | a single glob or a list; every **pre-existing** file (incl. uploaded files under `uploads/**`) whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a glob that matches **no** pre-run path fails loud (a typo or renamed mount would otherwise verify zero files and pass vacuously); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail (a `--resume` run); captured on every live sandbox tier including microvm; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree. On `lane: remote` it still reads the local stand-in for the inputs, which is right there: an edit to a staged copy inside the cloud container does not reach the device. |
|
|
19
|
+
| `input_unmodified: <glob> \| [<glob>, …]` | a single glob or a list; every **pre-existing** file (incl. uploaded files under `uploads/**`) whose workRoot-relative path matches ≥1 glob keeps an unchanged content hash after the run — the in-place-mutation companion to `no_unexpected_files`'s new-files check (`[]` is rejected by the schema — list at least one glob); a glob that matches **no** pre-run path fails loud (a typo or renamed mount would otherwise verify zero files and pass vacuously); a matched file that was deleted counts as a content change (fails); live/verify-run without a pre-run hash manifest ⇒ evidence-unavailable hard-fail (a `--resume` run); captured on every live sandbox tier including microvm; replay needs `cassette.preRunHashes` — cassettes without it **exclude** the key with a loud warning; on replay it compares against the manifest's recorded `sha256`, never a re-hash of the materialized tree. On `lane: remote` it still reads the local stand-in for the inputs, which is right there: an edit to a staged copy inside the cloud container does not reach the device. Under `device_bridge`, a glob on a connected folder's path is refused (the folders live in the device container). |
|
|
20
20
|
| `self_heal_ran: <bool>` | a bash command the model wrote did (not) name a plugin under `/sessions/<id>/mnt/.local-plugins/` or `/sessions/<id>/mnt/.remote-plugins/` — the plugin-root self-heal path. It reads the command as the model wrote it, so a host plugin path that `hostloop`'s bash tool rewrote to the VM mount does not count |
|
|
21
21
|
| `file_absent: <path>` | the named path does **not** exist under the work root after the run — the direct negative-existence key. Do NOT invert `no_unexpected_files` for this: that is an allowlist over NEW files only, and it needs a pre-run manifest. **LIVE/verify-run only** (a cassette records no walk health, so absence is unprovable on replay); rejected at load on `lane: remote`; evidence-unavailable on `preRunOrigin: remote-unavailable` |
|
|
22
22
|
| `artifact_text: {artifact, contains?: [..], not_contains?: [..], matches?, not_matches?}` | assert over a delivered artifact's TEXT body — `artifact_json`'s companion for non-JSON files, and how you prove an internal name did not leak into a file the user receives. Literal path (no glob), so one entry per delivered surface. Manifest-class; body-less / symlinked / over-cap targets fail evidence-unavailable, and a non-UTF-8 body fails the NEGATIVE matchers rather than passing against bytes it never read. Rejected at load on `lane: remote` (that lane's container filesystem is not locally observable) |
|
|
@@ -26,7 +26,7 @@ keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
|
26
26
|
| `tool_called: {tool: <glob> \| [..], input?, input_any?, result?, scope?, subagent_type?, count?}` | **object form** — asserts what a call CARRIED, where it RAN and what its paired result SAID, which no name glob and no `transcript_*` key can see (`transcript_*` reads top-level prose only, so it passes when the agent merely *says* it ran a command). `tool`: a glob or a list of globs (any-of; list both shells, `[Bash, mcp__workspace__bash]`, for a claim that must hold at hostloop too). `input: {<field>: <regex>}`: every named top-level input field must match (case-insensitive, unanchored; a missing field is no match; a non-string value is matched as JSON). `input_any: <regex>`: some top-level field matches. `result: {matches?, not_matches?, is_error?}`: predicates on the call's paired `tool_result` (by `toolUseId`, 10,240 characters of text — 32,768 for a top-level `Skill` result). `scope`: `main` (default — the main agent, including a `Skill`'s or `Agent(fork)`'s children, the set the string form reads), `subagent` (the parent is a sub-agent dispatch this run recorded, at any depth), or `any` (also calls whose parent is not a recorded dispatch, e.g. a `Skill` invoked inside a sub-agent). `subagent_type` (with `scope: subagent`): regex over the IMMEDIATE parent dispatch's type or description. `count: {min?, max?}` (default `min: 1`). **Fails closed:** an unpaired call never satisfies `result`; a truncated result or input that cannot settle a predicate, or a result.json without `toolCalls`, reports *evidence unavailable*, never a pass or a plain "not called". A red lists the calls it considered and names matches in another scope ("1 matching call in scope subagent — set `scope: any`"). `{tool: X}` alone is exactly `tool_called: X`. A cassette using this form stamps **v13**. |
|
|
27
27
|
| `tool_not_called: {tool: <glob> \| [..], input?, input_any?, result?, scope?, subagent_type?}` | **object form** — no call in scope satisfies every predicate (no `count`). Same fields and fail-closed rules as above: a candidate whose result is unpaired or truncated, or whose input was truncated, is *evidence unavailable*, never a pass. Refused at load when EVERY listed tool is one the tier does not serve. ⚠️ **Redaction hazard — this is the dangerous direction.** A committed cassette is redacted, which rewrites the recorded inputs: an `input` regex naming a literal the policy rewrites (a home path, an email) cannot see its target on replay. On replay, any candidate call whose field (or paired result, for `result.matches`) carries a redaction token is *evidence unavailable*, whatever the regex — the evaluator cannot know what the token replaced — so a negative check never passes over rewritten bytes. `record` warns — naming the redacted call and the ways out (narrow `scope`/`tool`, the string form, or live-only) — and refuses the write (the exact guard), and `lint` flags a regex naming a redactable literal (`tool-input-regex-redactable`) — a heuristic that reads `.cowork-redact.json` from the current and scenario directories only, not the cassette's. Match a part of the input redaction leaves alone (the verb and flags, a workspace-relative path), or keep the check on a live gate. Note also that the default `scope: main` does not see a sub-agent's call — use `scope: any` for "nothing anywhere ran X". |
|
|
28
28
|
| **legacy tool names** | The agent binary canonicalizes legacy spellings (`Task`→`Agent`, `KillShell`/`KillBash`→`TaskStop`, …) while the spawn tool list still declares the LEGACY one — so the init inventory shows `Task` and every call is emitted as `Agent`. `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` match EITHER spelling, globs included. Before this, `tool_called: "Task"` could never pass and `tool_not_called: "Task"` always did. |
|
|
29
|
-
| **tier vacuity — now REFUSED** | `tool_not_called` **and `subagent_tool_absent`** naming a tool the tier does not serve is rejected at scenario load (`Bash`/`WebFetch`/`NotebookEdit` at `hostloop`; `mcp__workspace__bash` at `container`/`microvm`), because it could never be violated. `scenario.py lint` WARNs on the same set before any run. The table is CLOSED to tools the harness itself removes or registers — `--tools` gates only the built-in set while every tier separately passes `--mcp-config`, so a session-MCP tool name is offered without appearing in any tool list and is never refused. Globs are never refused. `subagent_tool_absent` is covered because it reads the tools sub-agents actually USED, not a per-dispatch declared list. Under-approximates on purpose: `REPL` at hostloop is vacuous too and is not caught, which is the right side to err on for a hard refusal. |
|
|
29
|
+
| **tier vacuity — now REFUSED** | `tool_not_called` **and `subagent_tool_absent`** naming a tool the tier does not serve is rejected at scenario load (`Bash`/`WebFetch`/`NotebookEdit` at `hostloop`; `mcp__workspace__bash` at `container`/`microvm`; on those three tiers, `mcp__remote-devices__device_bash`/`get_device_info` without `device_bridge: true`, and every other `mcp__remote-devices__*` tool the harness never serves, bridge or not), because it could never be violated. `scenario.py lint` WARNs on the same set before any run. The table is CLOSED to tools the harness itself removes or registers — `--tools` gates only the built-in set while every tier separately passes `--mcp-config`, so a session-MCP tool name is offered without appearing in any tool list and is never refused. Globs are never refused. `subagent_tool_absent` is covered because it reads the tools sub-agents actually USED, not a per-dispatch declared list. Under-approximates on purpose: `REPL` at hostloop is vacuous too and is not caught, which is the right side to err on for a hard refusal. |
|
|
30
30
|
| `reference_read: <regex>` | a skill `references/`/`scripts/` file whose path matches this **regex** was ACCESSED — main agent or sub-agents, via `Read`, `Grep`/`Glob` (`path` input), or a `Bash`/`mcp__workspace__bash` command naming the path. Regex is **unanchored + case-insensitive** (the shared helper every regex key uses). **Under-approximates by design:** the path must be rooted in the mounted plugin, so a `cd` into the skill dir then a bare `cat references/x.md`, a heredoc body, and a `$VAR`-built path are invisible. Fails **evidence unavailable** when the run recorded no observable tool stream. Replay-capable (cassettes freeze whole tool inputs) |
|
|
31
31
|
| `no_observed_reference_access: <regex>` | no OBSERVED access matched the regex — the progressive-disclosure check: a reference the skill's routing never reaches. Named `observed` because detection under-approximates (see above), so it is **not proof the file went unread** — an agent that `cd`s and `cat`s it passes. Fails **evidence unavailable** rather than passing vacuously when no observable tool stream was recorded |
|
|
32
32
|
| `tool_result_contains: <str>` | a tool result includes the literal string (content / replay-checkable — substring match) |
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). Every `assert:` key with its semantics, and the
|
|
4
4
|
verdict-signal table. Which keys survive `replay` is in [`scenario-schema.md`](./scenario-schema.md#replay-class);
|
|
5
5
|
the scenario and session YAML fields are there too.
|
|
6
6
|
|
|
@@ -57,7 +57,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
|
57
57
|
| `host_path_leak` | fail | A host path leaked into model-visible text (opt out: author `transcript_no_host_path`). A path copied verbatim from the scenario's own uploads, connected folders, prompt, or declared plugins' or local skills' files is not a leak |
|
|
58
58
|
| `l0_host_config_contamination` | fail | `protocol` ran against the operator's REAL config dir, so host-installed plugins/skills/MCP servers may have answered instead of the thing under test — the run did not necessarily measure it (opt out: `allow_l0_host_config_contamination`). Name predates the meaning: delivery at L0 works, contamination is what this reports |
|
|
59
59
|
| `missing_capability` | fail | A `requires_capabilities` need was unmet, or the skill used a capability the image omits (opt out: `allow_missing_capability`, or `skill --allow-missing-capability` on an open-ended run) |
|
|
60
|
-
| `infra_error` | fail | A supervising process died mid-run (VM/egress sidecar) — the run's evidence is contaminated, not author-suppressible |
|
|
60
|
+
| `infra_error` | fail | A supervising process died mid-run (the VM/egress sidecar, or the `device_bridge` container) — the run's evidence is contaminated, not author-suppressible |
|
|
61
61
|
| `stalled` | fail | The run ended asking for input with no tool work after the last gate. "Asking" = the final turn's closing sentence ends in `?`; or, once an `AskUserQuestion` gate has fired, it ends in `?` after trailing bold/quotes/`)`/emoji, is a `?` followed only by a `For example: …`/parenthetical aside, or is a request that says the input comes back to the agent (`Please share/provide/upload… so I can…`/`…here`, `Let me know which… you'd like me to use`, `Once you share… I'll…`, `I need X to proceed`, `Once I have the file, I'll…` after a sentence asking for it). Polite closers and hand-offs (`Let me know if…`, `Feel free…`, `with your`/`to your`, `before sending`, `If you…`) and a closing code block or blockquote never count. Hand-offs to a named third party (`with the team`, `to the founders`, `with the CFO`) never count either; `here` is a cue only after share/paste/upload/drop/reply/type/send or as the last word. English-only. Opt out with `allow_stall` (on `run`, `replay` and `eval`); without it, under `eval` a stalled rep is `errored_agent` and fails every row |
|
|
62
62
|
| `non_deterministic` | warn | The run was LLM/external/human-decided — not reproducible |
|
|
63
63
|
| `model_fallback` | warn | The agent fell back off the requested model mid-run (SDK `model_fallback` event); a `model_not_found`/`model_blocked` trigger repeats every run until the pin changes |
|
|
@@ -68,7 +68,7 @@ codes (`VerdictSignal["code"]` in `src/run/verdict.ts`):
|
|
|
68
68
|
| `delivery_unobservable` | warn | `lane: remote` only — the run produced file(s), but whether any reached the user CANNOT be verified: nothing is delivered by location on that lane and the harness models no remote delivery tool (production uses the agent-native `SendUserFile`). The honest counterpart to `undelivered_deliverables`, which would otherwise fire on every remote run that writes anything — a signal that always fires carries no information. Mutually exclusive with it; quiet when the run produced nothing to deliver. A harness coverage gap, not a skill defect. Opt out: `allow_undelivered_deliverables` |
|
|
69
69
|
| `partly_scripted_gate` | warn | A question batch (one `AskUserQuestion` with several sub-questions) that the scenario's `answers:` matched only PART of. Answers are delivered atomically, so the whole batch went to the `on_unanswered` fallback and the matched answers were NOT delivered — the fallback may contradict them. The message names the matched and unmatched sub-questions and who answered instead; `result.partlyScriptedGates` carries the full lists. Fires only on a partial match within one batch (a single-question gate no rule matched is the ordinary unanswered case). Re-derived on `replay` (from the cassette's frozen answers) and `verify-run` (from the current scenario). Fix: script every sub-question of the batch |
|
|
70
70
|
| `parked_at_question` | warn | The session declares `answer_channel: none` and the run ended on a question nobody can answer: its last message ends in `?`, even after tools ran (a finished answer ending on an offer warns too; a request without `?` does not fire). Takes the place of the `stalled` fail and never changes the verdict: with no answer channel, stopping at the question is the contract. Completion is judged from the file assertions such a scenario must carry (refused at load without one). No opt-out needed |
|
|
71
|
-
| `exec_infra_error` | warn | Host-loop
|
|
71
|
+
| `exec_infra_error` | warn | Host-loop, or a `device_bash` call under `device_bridge`: one or more container `exec` calls failed for infrastructure reasons, so those tool calls returned an error to the agent. Warns rather than fails because the run's other evidence is intact — unlike `infra_error`, where a dead supervisor contaminates everything. Caveat: if *every* exec failed, the agent ran nothing and this still only warns — check `result.infraErrors` |
|
|
72
72
|
|
|
73
73
|
A **fail**-severity signal does not change `result.result` (still `"success"`), but it DOES fail the
|
|
74
74
|
overall run verdict and exit code — `assert result: success` alone won't catch it; check
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertions guide
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). Read it when choosing assertion keys: the two orthogonal axes and the goal → key map. The full catalog starts at `assertion-catalog.md`, which indexes the per-family files that hold every key's row.
|
|
4
4
|
|
|
5
5
|
### Assertions: two orthogonal axes
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Authoring a scenario
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). Read it when composing a `scenarios/*.yaml`: session vs scenario, discovery, the fidelity tier, the answer path, `web_fetch`, and scaffold + lint.
|
|
4
4
|
|
|
5
5
|
## Part I — AUTHOR a scenario
|
|
6
6
|
|
|
@@ -74,6 +74,18 @@ reach the device), and
|
|
|
74
74
|
plain `file_exists` is the proxy for a written path. Delivery itself is the `delivery_unobservable` WARN (see
|
|
75
75
|
`run-record-replay.md`).
|
|
76
76
|
|
|
77
|
+
**`device_bridge: true`** (with `lane: remote` and `fidelity: container`) serves the remote lane's shell on the user's
|
|
78
|
+
computer, `mcp__remote-devices__device_bash`, from a second container holding COPIES of the connected folders at
|
|
79
|
+
`$HOME/mnt/<folder-name>`; the agent's own container gets no folder. Assert on that call, never on the folder's
|
|
80
|
+
files: `tool_called: {tool: mcp__remote-devices__device_bash, result: {matches: <text>}}`, plus `transcript_matches`.
|
|
81
|
+
`file_exists` / `input_unmodified` on a connected folder's path are refused under the bridge (they would read the
|
|
82
|
+
agent's container). `no_delete_in_mounts` covers the folders (the device copy allows a delete production refuses).
|
|
83
|
+
`tool_not_called` on `device_bash`/`get_device_info` without the bridge, or on a device tool the harness never serves
|
|
84
|
+
(with it or without), is refused as vacuous. Egress follows the session allowlist;
|
|
85
|
+
`egress.unrestricted: true` matches production's pass-through. With no `permission_mode` in the session the agent
|
|
86
|
+
runs in `acceptEdits` (a production cloud task's mode under Manual); a mode you set wins. The run's `deviceBridge` lists every known
|
|
87
|
+
difference from production.
|
|
88
|
+
|
|
77
89
|
### Choose an answer path (gates: AskUserQuestion + tool-permission)
|
|
78
90
|
|
|
79
91
|
Default to **deterministic**: scripted `answers:` + `on_unanswered: fail`. Anything that brings a
|
|
@@ -258,6 +270,7 @@ cowork-harness lint scenarios/*.yaml
|
|
|
258
270
|
| `host-path-assert-cowork` | WARN | `transcript_no_host_path` on `cowork` — it fails by design if the tier resolves to hostloop |
|
|
259
271
|
| `host-path-assert-tier` | ERROR | `transcript_no_host_path` on `hostloop` / `protocol`, where it fails by design |
|
|
260
272
|
| `lane-remote-incompatible-key` | ERROR | a key that can never pass on `lane: remote`, which the runtime refuses at load: `present_files_called` / `no_scratchpad_leak` / `user_visible_artifact` (delivery), `artifact_text` / `artifact_json` (file bodies), `file_absent` / `no_unexpected_files` (absence), `computer_links_resolve(_if_present)`, `no_lost_write_back`, `file_exists` with `authored: true`, and `semantic_matches` / `semantic_pairwise` with `evidence_files`. Tier rules are suppressed there |
|
|
273
|
+
| `device-bridge-misplaced` | ERROR | `device_bridge` set anywhere but `true` with `lane: remote` and `fidelity: container`; the loader refuses it |
|
|
261
274
|
| `linter-extra-findings-invalid` | ERROR | the loader findings `cowork-harness lint` hands the linter could not be read |
|
|
262
275
|
| `linter-unclassified-key` | ERROR | a valid assertion key this linter cannot classify (the linter is out of date) |
|
|
263
276
|
| `manifest-needs-snapshot` | INFO | manifest-backed keys, which evaluate on replay only when the cassette carries an `artifacts` manifest (not reported on `lane: remote` for `user_visible_artifact` / `artifact_json` / `artifact_text`, which cannot pass there) |
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 4.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -17,7 +17,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
|
|
|
17
17
|
CLI major reaches your workflow the moment it is promoted even though your `uses:` ref never changed — so a
|
|
18
18
|
copy-pasted recipe that omits the input takes a major bump with no say in it. `^4` holds the major, needs no
|
|
19
19
|
patch number to remember, and only wants a human decision at the next major. Pin an exact version
|
|
20
|
-
(e.g. `version: "4.
|
|
20
|
+
(e.g. `version: "4.8.1"`) instead when you want byte-reproducible CI.
|
|
21
21
|
|
|
22
22
|
Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
|
|
23
23
|
cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
|
|
@@ -82,7 +82,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
82
82
|
GitHub-hosted runners, no token/Docker/agent:
|
|
83
83
|
|
|
84
84
|
```yaml
|
|
85
|
-
- run: npm i -g "cowork-harness@^4.
|
|
85
|
+
- run: npm i -g "cowork-harness@^4.8.1"
|
|
86
86
|
- run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
|
|
87
87
|
# no silent false-greens. WITHOUT --strict this
|
|
88
88
|
# step cannot fail on a WARN-class rule (e.g.
|
|
@@ -380,7 +380,9 @@ A typical skill repo runs four stages, fastest/cheapest first:
|
|
|
380
380
|
```
|
|
381
381
|
|
|
382
382
|
The `jq -e` IS the gate — `--corpus-only` exits 0 on a measurement even over the ceiling. Same
|
|
383
|
-
packager, same git filter as the paid run; the number is a floor (a run-time read can only add).
|
|
383
|
+
packager, same git filter as the paid run; the number is a floor (a run-time read can only add). The same
|
|
384
|
+
payload's `skillTreeHash` (every delivered file of the skill) tells a job whether the skill changed since
|
|
385
|
+
its last critique, at no cost.
|
|
384
386
|
3. **Scenarios (replay)** — `cowork-harness replay cassettes/` on every PR (the committed `*.cassette.json`).
|
|
385
387
|
Token-free; content + structure + gate delivery.
|
|
386
388
|
4. **Parity / live (nightly, self-hosted)** — `cowork-harness run scenarios/` with a token + Docker +
|
|
@@ -409,7 +411,7 @@ jobs:
|
|
|
409
411
|
with: { node-version: '24' }
|
|
410
412
|
- uses: actions/setup-python@v5
|
|
411
413
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
412
|
-
- run: npm i -g "cowork-harness@^4.
|
|
414
|
+
- run: npm i -g "cowork-harness@^4.8.1"
|
|
413
415
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
414
416
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
415
417
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -438,7 +440,7 @@ jobs:
|
|
|
438
440
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
439
441
|
fi
|
|
440
442
|
- if: steps.guard.outputs.live == 'true'
|
|
441
|
-
run: npm i -g "cowork-harness@^4.
|
|
443
|
+
run: npm i -g "cowork-harness@^4.8.1"
|
|
442
444
|
- if: steps.guard.outputs.live == 'true'
|
|
443
445
|
run: cowork-harness run scenarios/ --output-format json
|
|
444
446
|
env:
|
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# Critique — the facts a plugin install can't otherwise reach
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). This is **not** a trim of the full
|
|
4
4
|
[`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
|
|
5
|
-
flags, cost,
|
|
5
|
+
flags, cost, reading several critiques, known limitations all live there). This file covers exactly what a
|
|
6
6
|
plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's
|
|
7
|
-
real field names, and what the evaluator was and was not shown. See **Recipe
|
|
7
|
+
real field names, and what the evaluator was and was not shown. See **Recipe 6** in `task-recipes.md` for
|
|
8
8
|
the harvest → reproduce → fix loop this feeds.
|
|
9
9
|
|
|
10
10
|
## The run-dir artifact a harvester reads
|
|
@@ -97,15 +97,23 @@ messages, never from the flag. So `graded model(s): unknown` means no assistant
|
|
|
97
97
|
(crash, kill, or a gate before the first reply); passing `--model` does not change that line. Past runs
|
|
98
98
|
can be checked without re-running: the same ids are in each kept run dir's `turns/1/result.json`.
|
|
99
99
|
|
|
100
|
+
## A repaired evaluator reply — `evaluatorRepair`
|
|
101
|
+
|
|
102
|
+
When an evaluator reply was complete except for its final closing `}`, critique appends that one brace and
|
|
103
|
+
parses as usual; the report records `evaluatorRepair: [{pass, appended}]` (summary: `evaluatorRepaired: true`).
|
|
104
|
+
Nothing else is ever repaired — a reply that would also need a `]` (a cut-off list) or was cut mid-item or
|
|
105
|
+
mid-string still fails (exit 2): re-run the critique.
|
|
106
|
+
|
|
100
107
|
## The report's item shape — no `title`, no `summary`
|
|
101
108
|
|
|
102
109
|
Each `items[]` entry's prose fields are **`idea`** and **`recommendedAction`** — there is no `title` field
|
|
103
110
|
and no `summary` field. A consumer that parses for either gets silently blank output instead of an error.
|
|
104
111
|
Full required set: `source` (`evaluator`|`self-report`), `idea`, `classification`, `evidence` (the cited
|
|
105
112
|
excerpt, verbatim-checked against the evidence package), `recommendedAction`. The optional
|
|
106
|
-
`findingFingerprint` hashes `idea`+`classification`+`recommendedAction` —
|
|
107
|
-
match
|
|
108
|
-
|
|
113
|
+
`findingFingerprint` hashes `idea`+`classification`+`recommendedAction` — exact-match on model wording, so a
|
|
114
|
+
LOWER bound: a match shows the same wording came back; across repeats of the same probe it often never recurs,
|
|
115
|
+
and no match shows nothing. Read repeat reports side by side, aligned by `classification`; a shared `evidence`
|
|
116
|
+
excerpt is a cue to compare two items, not a match (unrelated findings often cite the same passage).
|
|
109
117
|
|
|
110
118
|
## What the evaluator was actually shown — `evidenceBudget`
|
|
111
119
|
|
|
@@ -171,6 +179,53 @@ from `dist/` source:
|
|
|
171
179
|
| `trimRecord` | which section the transcript trim shaved, and by how much |
|
|
172
180
|
| `packageTruncated` | `true` if ANY section was cut — check this, not `corpusCuts`, for "was anything trimmed": a transcript-only cut leaves `corpusCuts` empty and would otherwise read as "nothing cut" |
|
|
173
181
|
|
|
182
|
+
## Has the skill changed? — the three hashes
|
|
183
|
+
|
|
184
|
+
Every report and every `--corpus-only` payload carries three `sha256:` hashes, keyed by path inside the mount (so
|
|
185
|
+
two clones of one commit hash alike):
|
|
186
|
+
|
|
187
|
+
| Hash | Covers | Use it for |
|
|
188
|
+
|---|---|---|
|
|
189
|
+
| `skillTreeHash` | every file staging delivered for the skill (`scripts/` included) + resolved agents + linked plugin-root references | "did ANYTHING change since the last critique?" — equal in `--corpus-only` and a graded run |
|
|
190
|
+
| `corpusHash` | the evaluator's static corpus (`SKILL.md`, `references/**`, agents, linked root refs) | a FLOOR: unchanged ≠ unchanged behaviour, since `scripts/` is not in it |
|
|
191
|
+
| `packagedCorpusHash` | what the packager put in the evaluator's corpus, incl. root files the agent READ and the ceiling's cuts | evidence of what was graded — moves with the agent's reading, never a "changed" key |
|
|
192
|
+
|
|
193
|
+
`corpusManifest` lists the files (diff two to see which one moved a hash); `hashBasis` is `git-tracked`
|
|
194
|
+
(working-tree bytes of tracked files), `worktree-all` or `git-commit`; untracked files under the skill are listed
|
|
195
|
+
in `skillTreeUntracked` (first 50; `skillTreeUntrackedCount` = total), never hashed; `.git` is never hashed. `corpusDrift` means a file changed during the run — critique a commit
|
|
196
|
+
instead.
|
|
197
|
+
|
|
198
|
+
## Critique a commit — `git:<ref>:<path>`
|
|
199
|
+
|
|
200
|
+
`critique git:<ref>:<path> …` grades a snapshot of the commit, so a moved HEAD or an edit mid-run changes nothing;
|
|
201
|
+
`source.commit` records the resolved id. A path at `<plugin>/skills/<name>` grades that skill of the plugin (same
|
|
202
|
+
as `<plugin> --skill <name>`). Refused before spend: a git filter such as LFS (`.gitattributes`,
|
|
203
|
+
`.git/info/attributes`, `core.attributesFile`), a symlink whose real path leaves the snapshot, a submodule. Works
|
|
204
|
+
with `--corpus-only` ($0). Kept snapshots: `~/.cowork-harness/critique-snapshots/` (or
|
|
205
|
+
`COWORK_HARNESS_CRITIQUE_SNAPSHOTS_DIR`); nothing prunes them.
|
|
206
|
+
|
|
207
|
+
## A public ledger line — `--summary-out`
|
|
208
|
+
|
|
209
|
+
`--summary-out <file>` writes a summary safe for a PUBLIC repo: ids, hashes, enums, and per finding only
|
|
210
|
+
`findingFingerprint` / `classification` / `source` / `adjudicable` — no finding text, prompt, host path or git
|
|
211
|
+
ref. Values are shape-checked (a failure is `null` + named in `withheld`); a summary a configured secret would
|
|
212
|
+
alter is NOT written (warning, exit unchanged). Cost and the probe hash are opt-in
|
|
213
|
+
(`--summary-include-cost`, `--summary-include-prompt-hash`). With `--summary-out`, a `--label` must match
|
|
214
|
+
`[A-Za-z0-9._:+-]{1,64}` (e.g. `v1.2.0-rc1`, not `fix/x`). Schema: [`schema/critique-summary.json`](https://github.com/yaniv-golan/cowork-harness/blob/main/schema/critique-summary.json).
|
|
215
|
+
|
|
216
|
+
## Several critiques side by side — `critique --compare`
|
|
217
|
+
|
|
218
|
+
`critique --compare <report.json | summary.json …>` ($0, no model) groups by `--label` (one group, or two for
|
|
219
|
+
before/after) and lists every finding per report, aligned by classification. It gives **no verdict**: an exact
|
|
220
|
+
`findingFingerprint` match is `sameWording` k/N — a LOWER bound (it often never recurs across repeats); a
|
|
221
|
+
`sharedExcerpt` is the same cited passage, NOT proven the same finding. Equal `corpusHash` and one shared
|
|
222
|
+
`skillTreeHash` in both groups = `noiseFloorControl` (a scripts/ edit never is). Refused: mixed skills / schemes /
|
|
223
|
+
harness majors, mixed `corpusHash` within a group, the same critique twice, a summary that withheld a key
|
|
224
|
+
field, a pre-4.8 report. Marked, and refused
|
|
225
|
+
by `--strict`: mixed evaluator models / hash bases / skill trees / probes, a pass-1-only member. Marked only:
|
|
226
|
+
`mixedPackagedCorpus`, `probeUnverified` (no prompt hash — the summary default).
|
|
227
|
+
Its output carries finding text — not public-safe. Details: [`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only).
|
|
228
|
+
|
|
174
229
|
## An untracked skill file is not graded
|
|
175
230
|
|
|
176
231
|
Staging delivers **git-tracked files only** (same rule as a marketplace plugin install) — an untracked
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Debugging a run
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). Read it when a run misbehaved or a green looks wrong: triage, the observability output, and `chat`.
|
|
4
4
|
|
|
5
5
|
## Part III — Debug
|
|
6
6
|
|
|
@@ -107,7 +107,9 @@ decide which assertions from *Assertions: two orthogonal axes* in `assertions-gu
|
|
|
107
107
|
`semantic_*` with `evidence_files`);
|
|
108
108
|
`semantic_*` without it is judged on the transcript only. Reach for it
|
|
109
109
|
to check a skill's delivery survives the cloud lane, where new Pro and Max tasks run from 2026-10-06. Orthogonal to `fidelity` — a `lane: remote`
|
|
110
|
-
scenario still runs locally.
|
|
110
|
+
scenario still runs locally. Add `device_bridge: true` (with `fidelity: container`) to serve `device_bash`, the
|
|
111
|
+
cloud task's shell on the user's computer; a red there reads from the `device_bash` tool calls and the run's
|
|
112
|
+
`deviceBridge` record, and a `device-container` / `device-exec` infra error is that second container.
|
|
111
113
|
- **`cowork-harness stats [--metric <m>]`** — aggregate across the run index: `cost`, `duration`,
|
|
112
114
|
`tokens`, `cache-tokens`, `model-cost`, `turns`, `pass-rate`. Filters: `--since`/`--baseline`/`--branch`,
|
|
113
115
|
plus `--skill-hash <prefix>`/`--label <tag>` to narrow to ONE skill generation and
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# `eval` — paired before/after comparison of a skill edit (EXPERIMENTAL)
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). The full guide is
|
|
4
4
|
[docs/eval.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/eval.md); this is the part you
|
|
5
5
|
need while running it.
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 4.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`).
|
|
4
4
|
|
|
5
5
|
> **This page vs. the repo docs.** This is the **offline snapshot** that ships inside the installed
|
|
6
6
|
> plugin — it is self-contained on purpose. The repo carries four other fidelity views, each answering a
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Gotchas
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). The full "✓ passed ≠ correct" landmine catalog.
|
|
4
4
|
|
|
5
5
|
## Gotchas — the "✓ passed ≠ correct" landmines
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Recipe 7 — Climb a skill with `/claude-api hillclimb` and the harness as its runner
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
|
|
4
4
|
`hillclimb --help` lists `--skill` (help goes to stderr). This page is the loop's procedure, step by step, in the order of the
|
|
5
5
|
`/claude-api hillclimb` guide. Every mechanic (flags, refusals, the gate, `regrade`, `freeze-ref`, exit codes,
|
|
6
6
|
row keys) is in [`hillclimb.md`](hillclimb.md); the setup and the full list of differences from the guide's own
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# `hillclimb` — the runner for a `/claude-api hillclimb` loop
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). It needs a `cowork-harness` whose
|
|
4
4
|
`hillclimb --help` lists `--skill` (help goes to stderr). The command reference is
|
|
5
5
|
[docs/cli.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cli.md); this is the part a loop needs
|
|
6
6
|
while it runs. It covers `run`, `check`, `state-template`, `freeze-ref` and `regrade`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Measurement
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). Read it before comparing runs: `--repeat`, `--ablate-skill`, and the hygiene that keeps a batch valid.
|
|
4
4
|
|
|
5
5
|
### Measure — before/after, with/without (`--repeat`, `--ablate-skill`)
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Run, record and lock
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). Read it when running a scenario, recording or placing a cassette, reading verdict signals, checking a background run, or choosing CI lanes.
|
|
4
4
|
|
|
5
5
|
## Part II — RUN, RECORD & LOCK
|
|
6
6
|
|
|
@@ -132,8 +132,9 @@ current skill") rather than vouch against stale labels, but the trace/inspect pa
|
|
|
132
132
|
deliberately. (Same fail-closed family: corrupt gate evidence — unparseable `events.jsonl` lines, or fewer
|
|
133
133
|
gates than `trace.json` recorded questions — a structurally invalid `result.json`, a `command:"replay"`
|
|
134
134
|
result (a replay is a re-check of a recorded cassette, not run evidence — verify the original live run dir),
|
|
135
|
-
|
|
136
|
-
|
|
135
|
+
a `mode:"chat"` result (chat carries no assertions or verdict by contract), and a `device_bridge` run dir (its
|
|
136
|
+
folders lived in the device container; re-run or replay the cassette, in `regrade` and `refs` too) also refuse
|
|
137
|
+
rather than certify.) (A token-free probe of "which gates fire" isn't possible — gates are model-decided per run.)
|
|
137
138
|
|
|
138
139
|
Run artifacts are written to `~/.cowork-harness/runs/…` by default — **outside any working tree**, so a run
|
|
139
140
|
launched from a repo root never drops sensitive skill inputs/outputs into it. Pass `--run-dir <path>` (or set
|
|
@@ -203,7 +204,7 @@ cassette — has its own recipe:
|
|
|
203
204
|
`--label <tag>`); an un-split window spanning more than one generation now warns.
|
|
204
205
|
**Multi-skill plugin caveat (post-1.7.0 CLIs with `--skill`):**
|
|
205
206
|
skillHash keys the whole MOUNTED plugin, so on a multi-skill plugin the hash alone cross-pairs
|
|
206
|
-
critiques of DIFFERENT skills — pair by the report's `(gradedSkillHash, gradedSkill)
|
|
207
|
+
critiques of DIFFERENT skills — pair critiques by the report's per-skill `skillTreeHash` / `corpusHash` (4.8.0+), or by `(gradedSkillHash, gradedSkill)`. **On a pre-1.5.0 CLI the `skill` lane emits no `fingerprint.skillHash` at all**, so a
|
|
207
208
|
pairing step there silently groups on an absent key instead of erroring — check the field is present, or
|
|
208
209
|
require ≥ 1.5.0. See [`docs/debugging.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/debugging.md)
|
|
209
210
|
(repo-only) for the full loop.
|
|
@@ -299,7 +300,7 @@ Recognize these before "fixing" a non-bug:
|
|
|
299
300
|
skill's own files never exempt a path under its host source); `scan.inputHostPathTokens` counts the
|
|
300
301
|
host-path tokens the inputs carried, `scan.hostPathsFromInputs` the ones exempted, and a non-zero exemption
|
|
301
302
|
prints a `::notice::` so a clean scan that relied on it is never silent.
|
|
302
|
-
- **`exec_infra_error`** (`WARN`, host-loop) — one or more container `exec` calls failed for
|
|
303
|
+
- **`exec_infra_error`** (`WARN`, host-loop, or `device_bash` under `device_bridge`) — one or more container `exec` calls failed for
|
|
303
304
|
infrastructure reasons (daemon/container-level), so those tool calls returned an error to the agent
|
|
304
305
|
rather than the command's own output. Warn-severity because the run's other evidence is intact — unlike
|
|
305
306
|
the fail-severity `infra_error`, where a **supervising process** died and contaminated everything. Note
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, replay class, web_fetch, authoring gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 4.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 4.8.1`
|
|
4
4
|
(baseline `desktop-2.31226.1`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
|
|
5
5
|
[`docs/session.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md), and `SPEC.md`.
|
|
6
6
|
|
|
@@ -55,8 +55,8 @@ lane: local # OPTIONAL — which Cowork lane's DELIVERY
|
|
|
55
55
|
# only). Assert file_exists + transcript_matches instead; input_unmodified
|
|
56
56
|
# still reads the local stand-in (unconfirmed). Orthogonal to fidelity
|
|
57
57
|
# and execution — a `lane: remote` scenario still runs locally.
|
|
58
|
-
# Delivery semantics only
|
|
59
|
-
#
|
|
58
|
+
# Delivery semantics only, unless device_bridge: true also serves
|
|
59
|
+
# device_bash (see device_bridge below).
|
|
60
60
|
# NEEDS >= 1.14.0: on an older CLI a scenario carrying `lane:` does NOT
|
|
61
61
|
# load (`Unrecognized key: "lane"`, exit 2) — it is NOT reinterpreted as
|
|
62
62
|
# `lane: local`. Adopting the key means raising your floor. That
|
|
@@ -67,6 +67,12 @@ lane: local # OPTIONAL — which Cowork lane's DELIVERY
|
|
|
67
67
|
# by 1.14.0/1.15.0 it is stamped v10 and a pre-`lane` CLI ignores the
|
|
68
68
|
# key (`rehash` re-stamps it). `lane: local` means what an older CLI
|
|
69
69
|
# already does, so it lifts nothing.
|
|
70
|
+
# device_bridge: true # OPTIONAL — lane: remote + fidelity: container only (refused otherwise).
|
|
71
|
+
# Serves mcp__remote-devices__device_bash (and get_device_info) from a
|
|
72
|
+
# second container holding COPIES of the connected folders at
|
|
73
|
+
# $HOME/mnt/<folder-name>; the agent's own container gets no folder.
|
|
74
|
+
# Assert with tool_called. A delete there succeeds where production
|
|
75
|
+
# refuses it: assert no_delete_in_mounts. Stamps cassette v17.
|
|
70
76
|
on_unanswered: fail # policy for unscripted gates: fail | prompt | first | llm — run rejects prompt
|
|
71
77
|
# ("agent" is retired — no longer a valid value)
|
|
72
78
|
|
|
@@ -186,7 +192,7 @@ extended_thinking: true # real Cowork on/off toggle; default true (ON)
|
|
|
186
192
|
# or --thinking disabled when false (no arbitrary budget in real Cowork)
|
|
187
193
|
agent_max_turns: 500 # optional turn ceiling -> agent --max-turns; omit for the agent default
|
|
188
194
|
# (distinct from the max_turns ASSERTION)
|
|
189
|
-
permission_mode: default # default | acceptEdits | plan | bypassPermissions
|
|
195
|
+
permission_mode: default # default | acceptEdits | plan | bypassPermissions; omitted -> default (acceptEdits under device_bridge)
|
|
190
196
|
permission_parity: cowork # cowork (unscripted tool calls allowed) | strict (deny unscripted)
|
|
191
197
|
# answer_channel: none # NOT Cowork: nobody answers the agent (--permission-prompts none; the agent then
|
|
192
198
|
# offers no AskUserQuestion/EnterPlanMode/ExitPlanMode). Needs bypassPermissions +
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# `semantic_matches` — the full semantics
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`). The one-line summary is in
|
|
4
4
|
[assertion-catalog-gates-hooks-modifiers.md](assertion-catalog-gates-hooks-modifiers.md); this is the whole contract, split out so the catalog stays within the
|
|
5
5
|
agent's single-read size.
|
|
6
6
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
4
|
decision path. Every one answers a question a real fleet owner had to work out the hard way.
|
|
5
|
-
Tracks `cowork-harness 4.
|
|
5
|
+
Tracks `cowork-harness 4.8.1` (baseline `desktop-2.31226.1`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
|
|
6
6
|
Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
|
|
7
7
|
needed if your CLI meets SKILL.md's version floor.
|
|
8
8
|
|
|
@@ -50,18 +50,19 @@ what production would do. Two layers of defense:
|
|
|
50
50
|
- **Manual (one-liner):** `grep -h '"effectiveFidelity"' cassettes/*.cassette.json | sort | uniq -c`
|
|
51
51
|
shows the tier distribution of the fleet at a glance.
|
|
52
52
|
- **Do NOT re-record for a `[note]` alone.** A `·`/`[note]` row is informational and the run still exits
|
|
53
|
-
**0** — `session-fingerprint: predates
|
|
53
|
+
**0** — `session-fingerprint: predates … coverage` (`model`, `permission_mode` or `projects[]`) and
|
|
54
|
+
`prompt-assets: cassette predates …` both
|
|
54
55
|
mean "this cassette was recorded before that field existed, and everything the hash *does* cover still
|
|
55
56
|
matches". Re-recording buys the new coverage and nothing else, so on a fleet of heavy cassettes it is a
|
|
56
57
|
real bill for no verdict change. Re-record when a `✗` says to (baseline moved, skill drift, tier moved).
|
|
57
58
|
|
|
58
59
|
### Cassette anatomy (what you're looking at when you open one)
|
|
59
60
|
|
|
60
|
-
Top-level fields of a `*.cassette.json` (schema [`schema/cassette.
|
|
61
|
+
Top-level fields of a `*.cassette.json` (schema [`schema/cassette.v17.json`](https://github.com/yaniv-golan/cowork-harness/blob/main/schema/cassette.v17.json)):
|
|
61
62
|
|
|
62
63
|
| Field | What it is |
|
|
63
64
|
|---|---|
|
|
64
|
-
| `$schema`, `generator`, `cassetteVersion` | Provenance: schema URL, producing tool, format version — the MINIMUM a reader needs for this scenario, not the recorder's version (current max:
|
|
65
|
+
| `$schema`, `generator`, `cassetteVersion` | Provenance: schema URL, producing tool, format version — the MINIMUM a reader needs for this scenario, not the recorder's version (current max: 17 — the hash-format epoch floors every stamp at 12, so a fresh recording stamps 12 unless a key lifts it: 13 when its `assert:` uses the object form of `tool_called` / `tool_not_called`, 14 for a v14 key such as `include_fork_results` on `semantic_matches`, 15 for a v15 one such as `gates_all_scripted`, a glob `artifact_json`, an `artifact_json` carrying `schema`, any `artifact_json` on `lane: remote`, or a recording under `answer_channel: none`, 16 for a recording whose `hookFailureBlocks` lists an event or is unknown, 17 for a scenario with `device_bridge: true`; the full list is in the cassette docs' version table) |
|
|
65
66
|
| `scenario` | The embedded scenario snapshot at record time |
|
|
66
67
|
| `events` | The recorded agent event stream (the replay source) |
|
|
67
68
|
| `controlOut` | Driver→agent control responses — presence unlocks gate asserts on replay |
|
|
@@ -76,7 +77,8 @@ Top-level fields of a `*.cassette.json` (schema [`schema/cassette.v16.json`](htt
|
|
|
76
77
|
| `authoring` | Present iff a live decider answered ≥1 gate during recording (`nonDeterministic: true`) |
|
|
77
78
|
| `answerChannel` | `"none"` when the recording ran under `answer_channel: none` (absent otherwise). Replay re-drives with the channel still absent; such a recording stamps v15 |
|
|
78
79
|
| `hookFailureBlocks` | `{events}`: the events the run may have a hook on that sets `onFailure: "block"` (agent 2.1.295+); or `{unknown, why}` when a hook source could not be read or parsed (`why` says how many). Replay reads a failed or timed-out hook frame of such an event as unreadable, since the agent blocks on it after emitting the frame. A non-empty one stamps v16. Absent on an older recording: read by the agents its init frames report (empty only when every one is 2.1.293 or earlier; any other, or none reported, is unknown) |
|
|
79
|
-
| `
|
|
80
|
+
| `deviceBridge` | Present on a `device_bridge: true` recording: what the run modeled of the remote lane's device bridge, frozen so replay reports it as recorded. `served` and `alwaysLoad` (tool names), `measuredOn` (the Desktop build they were read from), `promptNoteBasis` (`model-report`: the prompt note rests on a model's report of production's instructions, not their text), `promptNote` (`included`, or `omitted` on a maintainer control run), `permissionMode` and `permissionModeSource` (`session`, or `bridge-default`: `acceptEdits` when the session sets no mode), `egress` (`allowlist`, or `unrestricted` under `egress.unrestricted: true`), `folders` (name and mode), and `divergences` from production by id, including `delete-not-refused`: a delete in a connected folder succeeds on the device copy, where production refuses it until the user allows deletion |
|
|
81
|
+
| `sessionFingerprint` | Optional even on v9+ (the minimum readable version): hash of the session's content-relevant SHAPE (model/folders/plugins/skills/mcp/egress/web_fetch, plus projects, agent_env, answer_channel and agent_env.artifacts_root when set, and permission_mode when not default). Under `device_bridge` the hash and the mode the agent runs in part ways: an unset `permission_mode` (the agent runs in `acceptEdits`) and an explicit `default` hash alike, so a switch between them passes `verify-cassettes` clean, while a switch from unset to an explicit `acceptEdits` reports drift although the mode did not change. Checked ONLY by `verify-cassettes`, never the default replay verdict; absent → not checked |
|
|
80
82
|
| `folderPrefixMap` | Optional even on v9+: the record-time connected-folder host-path → mount-name map. Replay's `computer_links_resolve` uses THIS (never the current session file); absent → the link is treated as evidence-unavailable, never reconstructed from the current session |
|
|
81
83
|
| `timeline`, `timelineHeader` | The recorded per-event timeline (harness-observation timestamps for tool_use/tool_result/subagent_dispatch/thinking/decision/result, in total order) plus its header (`startedAtWall`/`startedAtMono` anchors); informational only — never affects the replay verdict. Absent on a cassette recorded before this field existed |
|
|
82
84
|
| `environment` | Recording provenance: `location` (`"local"` on every cassette this harness produces), the resolved `tier`, `agentBinaryFormat`, and `harnessVersion` — the CLI that RECORDED the cassette (≥1.11.0). A harness-code change can shift recorded behaviour at an UNCHANGED baseline, which no staleness class keys off, so this is the only provenance for that class. **Absent** on pre-1.11.0 cassettes, and never backfilled — the absence is itself the signal |
|