cowork-harness 4.11.0 → 4.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +5 -5
- package/.claude/skills/cowork-harness/references/assertion-catalog-agents-skills-budgets.md +1 -1
- package/.claude/skills/cowork-harness/references/assertion-catalog-gates-hooks-modifiers.md +1 -1
- package/.claude/skills/cowork-harness/references/assertion-catalog-outcome-files-tools.md +1 -1
- package/.claude/skills/cowork-harness/references/assertion-catalog.md +1 -1
- package/.claude/skills/cowork-harness/references/assertions-guide.md +1 -1
- package/.claude/skills/cowork-harness/references/authoring.md +3 -2
- package/.claude/skills/cowork-harness/references/ci-recipe.md +6 -6
- package/.claude/skills/cowork-harness/references/critique.md +1 -1
- package/.claude/skills/cowork-harness/references/debugging.md +1 -1
- package/.claude/skills/cowork-harness/references/eval.md +1 -1
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/gotchas.md +2 -2
- package/.claude/skills/cowork-harness/references/hillclimb-recipe.md +1 -1
- package/.claude/skills/cowork-harness/references/hillclimb.md +1 -1
- package/.claude/skills/cowork-harness/references/measurement.md +1 -1
- package/.claude/skills/cowork-harness/references/run-record-replay.md +2 -2
- package/.claude/skills/cowork-harness/references/scenario-schema.md +9 -5
- package/.claude/skills/cowork-harness/references/semantic-judging.md +1 -1
- package/.claude/skills/cowork-harness/references/task-recipes.md +1 -1
- package/CHANGELOG.md +97 -0
- package/DESIGN.md +3 -2
- package/README.md +4 -4
- package/RELEASING.md +22 -0
- package/SPEC.md +1 -0
- package/baselines/desktop-3.6556.2.json +1823 -0
- package/baselines/desktop-4.4212.0.json +1993 -0
- package/baselines/prompts/cowork-system-prompt-fingerprints.json +19 -1
- package/dist/assert.js +7 -1
- package/dist/cli.js +1 -1
- package/dist/hostloop/workspace-handler.js +14 -6
- package/dist/runtime/container.js +1 -0
- package/dist/runtime/hostloop.js +1 -0
- package/dist/session.js +1 -0
- package/dist/sync/baseline-diff.js +10 -2
- package/dist/sync/cowork-sync.js +761 -36
- package/dist/sync/observed-spawn-env.js +4 -2
- package/dist/sync/spawn-env-registry.js +16 -0
- package/dist/types.js +24 -0
- package/docs/ci.md +2 -2
- package/docs/cli.md +5 -5
- package/docs/companion-skill.md +2 -2
- package/docs/fidelity-gaps.md +66 -1
- package/docs/maintenance.md +10 -3
- package/docs/scenario.md +1 -1
- package/docs/session.md +2 -1
- package/examples/replays/README.md +1 -1
- package/examples/replays/example-multiselect-gate.cassette.json +1 -1
- package/examples/replays/example-pdf-skill.cassette.json +1 -1
- package/examples/replays/hostloop-computer-links.cassette.json +1 -1
- package/package.json +1 -1
- package/scripts/release-preflight.ts +5 -0
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, asserting artifacts, egress, or sub-agent dispatch, measuring how long each tool call took (toolDurations / trace), or debugging a failed run or verdict from its result.json or transcript. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. Also for comparing two versions of a skill before merging an edit — did it make answers worse? (`eval`: paired A/B, pinned models) — or improving one round by round (`hillclimb`, the `/claude-api hillclimb` runner). NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold / critique / stats / eval / hillclimb commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 4.
|
|
7
|
-
tracks-harness: cowork-harness 4.
|
|
6
|
+
version: 4.13.0
|
|
7
|
+
tracks-harness: cowork-harness 4.13.0 (baseline desktop-4.4212.0)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -26,8 +26,8 @@ allowlist). This skill exists mostly to keep you out of those traps — the *Inv
|
|
|
26
26
|
full landmine catalog in [`references/gotchas.md`](references/gotchas.md) are the highest-value part.
|
|
27
27
|
Read them.
|
|
28
28
|
|
|
29
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.
|
|
30
|
-
> `desktop-
|
|
29
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 4.13.0` (baseline
|
|
30
|
+
> `desktop-4.4212.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
31
31
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
32
32
|
|
|
33
33
|
## Preflight — make sure the harness can actually run
|
|
@@ -43,7 +43,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
43
43
|
|
|
44
44
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, the baseline and the `device_bridge` image (built, or built on the first bridge run) in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
45
45
|
- **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
|
|
46
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.
|
|
46
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 4.13.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@^4.13.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@^4.13.0"`. **Pin `@^4.13.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
|
|
47
47
|
|
|
48
48
|
This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
|
|
49
49
|
OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog: sub-agents, skills and connectors, budgets, tasks, delivery
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). Part 2 of 3 of the per-key table. The conventions every key shares (an item with several
|
|
4
4
|
keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
5
5
|
[`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog: gates, hooks, path denial, verdict modifiers, egress, judged keys
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). Part 3 of 3 of the per-key table. The conventions every key shares (an item with several
|
|
4
4
|
keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
5
5
|
[`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-outcome-files-tools.md`](./assertion-catalog-outcome-files-tools.md), [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md).
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog: outcome, transcript, files and artifacts, tools, tool results
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). Part 1 of 3 of the per-key table. The conventions every key shares (an item with several
|
|
4
4
|
keys is an AND; glob vs regex name matching) and the verdict-signal table are in
|
|
5
5
|
[`assertion-catalog.md`](./assertion-catalog.md); the other parts are [`assertion-catalog-agents-skills-budgets.md`](./assertion-catalog-agents-skills-budgets.md), [`assertion-catalog-gates-hooks-modifiers.md`](./assertion-catalog-gates-hooks-modifiers.md).
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertion catalog
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). Every `assert:` key with its semantics, and the
|
|
4
4
|
verdict-signal table. Which keys survive `replay` is in [`scenario-schema.md`](./scenario-schema.md#replay-class);
|
|
5
5
|
the scenario and session YAML fields are there too.
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Assertions guide
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). Read it when choosing assertion keys: the two orthogonal axes and the goal → key map. The full catalog starts at `assertion-catalog.md`, which indexes the per-family files that hold every key's row.
|
|
4
4
|
|
|
5
5
|
### Assertions: two orthogonal axes
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Authoring a scenario
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). Read it when composing a `scenarios/*.yaml`: session vs scenario, discovery, the fidelity tier, the answer path, `web_fetch`, and scaffold + lint.
|
|
4
4
|
|
|
5
5
|
## Part I — AUTHOR a scenario
|
|
6
6
|
|
|
@@ -232,7 +232,8 @@ your gate actually has a label called `"first"`.
|
|
|
232
232
|
session
|
|
233
233
|
`web_fetch.approved_domains`, or a live decider.
|
|
234
234
|
|
|
235
|
-
Surprise to remember: adding a host to `egress.extra_allow` is a **no-op** for a provenanced fetch
|
|
235
|
+
Surprise to remember: adding a host to `egress.extra_allow` is a **no-op** for a provenanced fetch, and on Path B it
|
|
236
|
+
cannot admit the baseline's `network.webFetchVendorDeny` hosts (from `desktop-4.4212.0`).
|
|
236
237
|
Full model in `references/scenario-schema.md`.
|
|
237
238
|
|
|
238
239
|
### A skill that parks and resumes in a later session: `legs:`
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 4.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -17,7 +17,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
|
|
|
17
17
|
CLI major reaches your workflow the moment it is promoted even though your `uses:` ref never changed — so a
|
|
18
18
|
copy-pasted recipe that omits the input takes a major bump with no say in it. `^4` holds the major, needs no
|
|
19
19
|
patch number to remember, and only wants a human decision at the next major. Pin an exact version
|
|
20
|
-
(e.g. `version: "4.
|
|
20
|
+
(e.g. `version: "4.13.0"`) instead when you want byte-reproducible CI.
|
|
21
21
|
|
|
22
22
|
Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
|
|
23
23
|
cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
|
|
@@ -36,7 +36,7 @@ jobs:
|
|
|
36
36
|
- uses: actions/checkout@v4
|
|
37
37
|
- name: Stage the agent binary (official channel, sha256-verified against the pinned baseline)
|
|
38
38
|
run: |
|
|
39
|
-
V=2.1.
|
|
39
|
+
V=2.1.309 # match your scenario's pinned baseline's agentVersion
|
|
40
40
|
# The release channel is NOT always the stable one. Desktop also stages release CANDIDATES,
|
|
41
41
|
# served from .../claude-code-releases/rc/<commit>/. For some versions the stable path 404s
|
|
42
42
|
# (2.1.255); for others it returns 200 and serves a DIFFERENT BUILD UNDER THE SAME VERSION
|
|
@@ -82,7 +82,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
82
82
|
GitHub-hosted runners, no token/Docker/agent:
|
|
83
83
|
|
|
84
84
|
```yaml
|
|
85
|
-
- run: npm i -g "cowork-harness@^4.
|
|
85
|
+
- run: npm i -g "cowork-harness@^4.13.0"
|
|
86
86
|
- run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
|
|
87
87
|
# no silent false-greens. WITHOUT --strict this
|
|
88
88
|
# step cannot fail on a WARN-class rule (e.g.
|
|
@@ -411,7 +411,7 @@ jobs:
|
|
|
411
411
|
with: { node-version: '24' }
|
|
412
412
|
- uses: actions/setup-python@v5
|
|
413
413
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
414
|
-
- run: npm i -g "cowork-harness@^4.
|
|
414
|
+
- run: npm i -g "cowork-harness@^4.13.0"
|
|
415
415
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
416
416
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
417
417
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -440,7 +440,7 @@ jobs:
|
|
|
440
440
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
441
441
|
fi
|
|
442
442
|
- if: steps.guard.outputs.live == 'true'
|
|
443
|
-
run: npm i -g "cowork-harness@^4.
|
|
443
|
+
run: npm i -g "cowork-harness@^4.13.0"
|
|
444
444
|
- if: steps.guard.outputs.live == 'true'
|
|
445
445
|
run: cowork-harness run scenarios/ --output-format json
|
|
446
446
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Critique — the facts a plugin install can't otherwise reach
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). This is **not** a trim of the full
|
|
4
4
|
[`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
|
|
5
5
|
flags, cost, reading several critiques, known limitations all live there). This file covers exactly what a
|
|
6
6
|
plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Debugging a run
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). Read it when a run misbehaved or a green looks wrong: triage, the observability output, and `chat`.
|
|
4
4
|
|
|
5
5
|
## Part III — Debug
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# `eval` — paired before/after comparison of a skill edit (EXPERIMENTAL)
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). The full guide is
|
|
4
4
|
[docs/eval.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/eval.md); this is the part you
|
|
5
5
|
need while running it.
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 4.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`).
|
|
4
4
|
|
|
5
5
|
> **This page vs. the repo docs.** This is the **offline snapshot** that ships inside the installed
|
|
6
6
|
> plugin — it is self-contained on purpose. The repo carries four other fidelity views, each answering a
|
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
# Gotchas
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). The full "✓ passed ≠ correct" landmine catalog.
|
|
4
4
|
|
|
5
5
|
## Gotchas — the "✓ passed ≠ correct" landmines
|
|
6
6
|
|
|
7
7
|
Stated as *symptom → why → fix*. This is the **workflow/record/answer-path** view — the broader of the
|
|
8
8
|
two lists, but **not a strict superset**: `references/scenario-schema.md`'s *Authoring gotcha list*
|
|
9
9
|
carries a few assertion-level landmines this one omits (`transcript_no_host_path`'s scan width,
|
|
10
|
-
`egress.extra_allow`'s no-op on the provenanced `web_fetch` path, `replay_protocol_fidelity` not being
|
|
10
|
+
`egress.extra_allow`'s no-op on the provenanced `web_fetch` path and the vendor hosts it cannot admit on Path B, `replay_protocol_fidelity` not being
|
|
11
11
|
authorable). Reach for this list when debugging a run's behavior, that one while authoring `assert:`.
|
|
12
12
|
**The two lists are numbered independently** — a bare "gotcha N" means the list you are reading.
|
|
13
13
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Recipe 7 — Climb a skill with `/claude-api hillclimb` and the harness as its runner
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). It needs a `cowork-harness` whose
|
|
4
4
|
`hillclimb --help` lists `--skill` (help goes to stderr). This page is the loop's procedure, step by step, in the order of the
|
|
5
5
|
`/claude-api hillclimb` guide. Every mechanic (flags, refusals, the gate, `regrade`, `freeze-ref`, exit codes,
|
|
6
6
|
row keys) is in [`hillclimb.md`](hillclimb.md); the setup and the full list of differences from the guide's own
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# `hillclimb` — the runner for a `/claude-api hillclimb` loop
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). It needs a `cowork-harness` whose
|
|
4
4
|
`hillclimb --help` lists `--skill` (help goes to stderr). The command reference is
|
|
5
5
|
[docs/cli.md](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/cli.md); this is the part a loop needs
|
|
6
6
|
while it runs. It covers `run`, `check`, `state-template`, `freeze-ref` and `regrade`.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Measurement
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). Read it before comparing runs: `--repeat`, `--ablate-skill`, and the hygiene that keeps a batch valid.
|
|
4
4
|
|
|
5
5
|
### Measure — before/after, with/without (`--repeat`, `--ablate-skill`)
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Run, record and lock
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). Read it when running a scenario, recording or placing a cassette, reading verdict signals, checking a background run, or choosing CI lanes.
|
|
4
4
|
|
|
5
5
|
## Part II — RUN, RECORD & LOCK
|
|
6
6
|
|
|
@@ -229,7 +229,7 @@ Recognize these before "fixing" a non-bug:
|
|
|
229
229
|
`markitdown`/`magika` (`ml_extract`), `cv2` (`cv`), `camelot`/`tabula` (`pdf_tables`), or `wand`
|
|
230
230
|
(`magick`) can trip this even though real Cowork **ships** those (per the rootfs manifest captured at
|
|
231
231
|
Desktop `2.9939.2` — `baselines/provisioning/rootfs-provisioning.json`, which is the dated evidence
|
|
232
|
-
behind that sentence;
|
|
232
|
+
behind that sentence; 13 baselines have shipped since without a re-capture). The message says so ("likely a FALSE
|
|
233
233
|
NEGATIVE (real Cowork ships them)"). Fix: rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`, point
|
|
234
234
|
`COWORK_AGENT_IMAGE` at it), or — if the skill's fallback is genuinely equivalent — assert
|
|
235
235
|
`allow_missing_capability: true`. (Two sources: a skill *observed using* an omitted family, live lane;
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Scenario & session schema, replay class, web_fetch, authoring gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 4.
|
|
4
|
-
(baseline `desktop-
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 4.13.0`
|
|
4
|
+
(baseline `desktop-4.4212.0`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
|
|
5
5
|
[`docs/session.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md), and `SPEC.md`.
|
|
6
6
|
|
|
7
7
|
**Minimal scenario** — `prompt` and `fidelity` are required:
|
|
@@ -331,7 +331,8 @@ mcp:
|
|
|
331
331
|
|
|
332
332
|
# network (Cowork egress, pre-prompt)
|
|
333
333
|
egress:
|
|
334
|
-
extra_allow: [] # added to the release allowlist (bash / Path-B web_fetch only
|
|
334
|
+
extra_allow: [] # added to the release allowlist (bash / Path-B web_fetch only;
|
|
335
|
+
# never re-admits the baseline's network.webFetchVendorDeny hosts)
|
|
335
336
|
unrestricted: false # true == Cowork "*" (allow all)
|
|
336
337
|
|
|
337
338
|
# web_fetch (TEST CONVENIENCE — not a real Cowork setting)
|
|
@@ -551,7 +552,9 @@ on a manifest-less cassette those are skipped, so the gate can't see the deliver
|
|
|
551
552
|
|
|
552
553
|
**The surprise:** adding a host to `egress.extra_allow` is a **no-op** for a provenanced fetch
|
|
553
554
|
(Path A ignores the allowlist); conversely a provenanced fetch succeeds to a host NOT in
|
|
554
|
-
`extra_allow`. Provenance is the gate, not the allowlist.
|
|
555
|
+
`extra_allow`. Provenance is the gate, not the allowlist. On Path B, from baseline `desktop-4.4212.0`, a few
|
|
556
|
+
vendor hosts (`network.webFetchVendorDeny` in the baseline) are refused even when `extra_allow` or the release
|
|
557
|
+
allowlist names them; only `egress.unrestricted: true` lets web_fetch reach them, as in Cowork.
|
|
555
558
|
|
|
556
559
|
## Scenario YAML vs the pytest lane
|
|
557
560
|
|
|
@@ -652,7 +655,8 @@ debugging a run's behavior. The two are **numbered independently**: a bare "gotc
|
|
|
652
655
|
`.env`.
|
|
653
656
|
|
|
654
657
|
17. **web_fetch: `egress.extra_allow` is a no-op on the provenanced path** — provenance is the gate
|
|
655
|
-
(see the web_fetch section).
|
|
658
|
+
(see the web_fetch section). On Path B it also cannot admit the baseline's `network.webFetchVendorDeny`
|
|
659
|
+
hosts (from `desktop-4.4212.0`); only `egress.unrestricted: true` does. multiSelect gates ARE supported across every answer channel: scripted
|
|
656
660
|
(`choose:` list), in-band `--decider-dir` (repeat `--choose`, or a JSON-array reply), and
|
|
657
661
|
`--decider-cmd` (JSON-array reply) — all deliver the same `", "`-joined wire shape; a member label
|
|
658
662
|
containing a comma warns (the wire join is unescaped — a Cowork limitation).
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# `semantic_matches` — the full semantics
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 4.
|
|
3
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`). The one-line summary is in
|
|
4
4
|
[assertion-catalog-gates-hooks-modifiers.md](assertion-catalog-gates-hooks-modifiers.md); this is the whole contract, split out so the catalog stays within the
|
|
5
5
|
agent's single-read size.
|
|
6
6
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
4
|
decision path. Every one answers a question a real fleet owner had to work out the hard way.
|
|
5
|
-
Tracks `cowork-harness 4.
|
|
5
|
+
Tracks `cowork-harness 4.13.0` (baseline `desktop-4.4212.0`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
|
|
6
6
|
Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
|
|
7
7
|
needed if your CLI meets SKILL.md's version floor.
|
|
8
8
|
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,103 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [4.13.0] - 2026-11-01
|
|
10
|
+
|
|
11
|
+
### Upgrade notes
|
|
12
|
+
|
|
13
|
+
- **Cassettes: re-stamp or re-record one recorded through `baseline: latest`.** `latest` moves to `desktop-4.4212.0`
|
|
14
|
+
(see Changed), so such a cassette reports `[stale] baseline moved 3.6556.2 → 4.4212.0 since record — re-record`.
|
|
15
|
+
Re-stamping `fingerprint.baseline` clears it. The agent moves from 2.1.308 to 2.1.309, so a re-stamped cassette
|
|
16
|
+
recorded on 2.1.308 now gets an `agent-version:` note; re-record it to clear the note. The bundled cassettes are
|
|
17
|
+
re-stamped; they were recorded on 2.1.299 and keep the note they already had.
|
|
18
|
+
- **web_fetch on Path B refuses a few vendor hosts that the allowlist names**, as Cowork now does (see Changed). A
|
|
19
|
+
scenario that fetched one of them through the default allowlist now gets a refusal; `egress.extra_allow` does not
|
|
20
|
+
change that, `egress.unrestricted: true` does.
|
|
21
|
+
|
|
22
|
+
### Changed
|
|
23
|
+
|
|
24
|
+
- **`latest` moves to `desktop-4.4212.0`** (agent **2.1.309**). The Cowork system prompt bundled with Desktop and the
|
|
25
|
+
egress allowlist are unchanged from `desktop-3.6556.2`. The bundled prompt is the one Desktop builds a session from
|
|
26
|
+
when it has no prompt from Anthropic's servers; a signed-in session can receive its prompt from them instead, and
|
|
27
|
+
`sync` cannot see that copy. Observed, as before, from a local scheduled task. Desktop 4.29.0 was not synced
|
|
28
|
+
separately; `sync`'s checks were run against it as well.
|
|
29
|
+
- **web_fetch vendor hosts (modeled):** on the allowlist branch (Path B), Desktop now refuses a short list of vendor
|
|
30
|
+
hosts, and their subdomains, even when the allowlist names them; only an unrestricted allowlist skips the check.
|
|
31
|
+
`sync` records the list in the baseline as `network.webFetchVendorDeny` and checks the order of the gate's steps;
|
|
32
|
+
the harness applies it in the same order. `sync` finds the step by its host list and matcher as well as by its
|
|
33
|
+
wording, so a reworded refusal flags instead of dropping the list from the baseline. A baseline from an older
|
|
34
|
+
Desktop has no list and keeps no refusal. See
|
|
35
|
+
[fidelity gaps](docs/fidelity-gaps.md#web_fetch-refuses-a-few-vendor-hosts-even-when-allowlisted--modeled-from-baseline-442120).
|
|
36
|
+
- **Spawn env (not modeled):** a scheduled task's agent now also gets `CLAUDE_CODE_DISABLE_FILE_MENTIONS`, so a
|
|
37
|
+
file named with `@` in its prompt reaches the model as a path, not as inlined contents. The harness models an
|
|
38
|
+
ordinary task session and does not set it. Like the scheduled-run marker, the key is not reported as a
|
|
39
|
+
default-absent spawn-env key when every session read was a scheduled task. See
|
|
40
|
+
[fidelity gaps](docs/fidelity-gaps.md#-mentioned-files-in-a-scheduled-task--not-modeled).
|
|
41
|
+
- **Typed-artifact tools in chat sessions (not modeled):** from Desktop 4.29.0, Desktop's own chat-typed sessions
|
|
42
|
+
also get the typed-artifact tools of its `cowork` server, and the host loop pre-approves the read-only ones.
|
|
43
|
+
Desktop 4.4212.0 adds `list_typed_artifacts`. The harness serves none of these tools.
|
|
44
|
+
- **Agent 2.1.309, `onFailure: "block"` hooks:** a hook stopped after its own timeout now blocks even when the turn
|
|
45
|
+
was aborted at the same time. The harness already treated every cancelled hook frame on such an event as a
|
|
46
|
+
possible block, so its reading is unchanged.
|
|
47
|
+
- **Cassettes:** see the Upgrade notes.
|
|
48
|
+
- **`sync` reads the Cowork system prompt where Desktop 4.29.0 moved it.** The prompt is now a parameter of the
|
|
49
|
+
function that builds the session options, with its default loaded from a separate chunk; `sync` follows that load
|
|
50
|
+
to the one definition it names instead of matching a constant by name, and refuses when there is not exactly one.
|
|
51
|
+
Its fingerprint is unchanged. `sync` also checks that the code reading the same template for a third-party session
|
|
52
|
+
reads that template and no other.
|
|
53
|
+
- **`sync` checks each extra key of the scheduled-task env object.** Every key beside the scheduled-run marker must
|
|
54
|
+
have a registry row that names it as scheduled-only, and must be built nowhere else in the bundle.
|
|
55
|
+
- **`sync` checks the typed-artifact tool predicate and the host loop's pre-approved list** in builds that date the
|
|
56
|
+
chat-session change: the tools must come from the `cowork` server, be exactly the ones the harness knows (a tool
|
|
57
|
+
added or dropped flags), and reach the pre-approved list only through its read-only export.
|
|
58
|
+
- **`sync --diff` no longer reports the typed-artifact tools as DISAPPEARED when every session read was a scheduled
|
|
59
|
+
task.** Desktop never offers them to one, and offers them to an ordinary session only when the account has
|
|
60
|
+
typed-artifact types, so such a read can neither show nor rule them out; `sync --diff` lists them as not observed,
|
|
61
|
+
with that reason. A tool that appears is still reported as APPEARED. `npm run preflight`'s check of the newest
|
|
62
|
+
baseline's observed tool list names them the same way.
|
|
63
|
+
- **`sync` follows the path hook's pre-pass when it is skipped for launched agents.** In Desktop 4.4212.0 the hook
|
|
64
|
+
that checks file paths delegates to a larger function, and one link of its check is skipped when the agent was
|
|
65
|
+
started through Desktop's CLI launcher. `sync` now reads the delegate's whole body, pairs that skip with the
|
|
66
|
+
pre-pass's matching argument, and flags the launcher if it can do anything beyond being inert.
|
|
67
|
+
- **Release and CI jobs have time limits**, so a hung step fails instead of holding a runner for hours. See
|
|
68
|
+
RELEASING.md for finishing a release whose limit was hit after the npm upload.
|
|
69
|
+
|
|
70
|
+
## [4.12.0] - 2026-10-25
|
|
71
|
+
|
|
72
|
+
### Upgrade notes
|
|
73
|
+
|
|
74
|
+
- **Cassettes: re-stamp or re-record one recorded through `baseline: latest`.** `latest` moves to `desktop-3.6556.2`
|
|
75
|
+
(see Changed), so such a cassette reports `[stale] baseline moved 3.1623.4 → 3.6556.2 since record — re-record`.
|
|
76
|
+
Re-stamping `fingerprint.baseline` clears it. The agent is unchanged (2.1.308): a cassette recorded on it gets no
|
|
77
|
+
`agent-version:` note from the re-stamp, and one recorded on an earlier agent keeps the note it already had. The
|
|
78
|
+
bundled cassettes are re-stamped; they were recorded on 2.1.299 and keep that note.
|
|
79
|
+
|
|
80
|
+
### Changed
|
|
81
|
+
|
|
82
|
+
- **`latest` moves to `desktop-3.6556.2`** (agent **2.1.308**, unchanged). The Cowork system prompt and the egress
|
|
83
|
+
allowlist are unchanged from `desktop-3.1623.4`. Observed, as before, from a local scheduled task. Desktop 3.6556.1
|
|
84
|
+
was not synced separately: no Cowork-related difference between it and 3.6556.2 was found.
|
|
85
|
+
- **Spawn env:** Desktop now sets `CLAUDE_CODE_ATTRIBUTION_PROJECT` when exactly one claude.ai Project is selected
|
|
86
|
+
on a first-party deployment. The session the harness models selects none, so the harness does not set it.
|
|
87
|
+
- **Artifact permission checks (not modeled):** Desktop's Artifact path hook now also covers the tools of the
|
|
88
|
+
`claude-artifacts` MCP server. For that server's file publishing, Desktop records each file's identity before
|
|
89
|
+
the approval card and re-checks it after, including its content hash, and refuses files whose drive cannot
|
|
90
|
+
identify them. `sync` checks how the recorded identity reaches the permission check, the widened matcher, and
|
|
91
|
+
that refusal, and refuses a build whose permission wrapper loses the capture while the hook still covers that
|
|
92
|
+
server's tools; it does not check the re-check itself (the hash, how the file is reopened). In a build with the
|
|
93
|
+
capture, it also requires the existing refusal of an approval whose arguments differ from the request.
|
|
94
|
+
- **MCP tools that need a person (not modeled):** from Desktop 3.6556.1, a call to an MCP tool that marks itself
|
|
95
|
+
as needing a person (`anthropic/requiresUserInteraction`) is refused with no approval card when the turn is
|
|
96
|
+
unattended, such as a scheduled task no person has continued. The harness does not read the mark, so such a tool
|
|
97
|
+
is handled like any other. See [fidelity gaps](docs/fidelity-gaps.md#mcp-tools-that-need-a-person--not-modeled).
|
|
98
|
+
- **Cassettes:** see the Upgrade note.
|
|
99
|
+
- **`sync` requires the permission check's approval-window refusal in every build that has the check.** The
|
|
100
|
+
refusal of a file that changed between the approval card and the answer has been in every such build since Desktop
|
|
101
|
+
1.32352.0, but `sync` did not check for it until now; a build without it is refused.
|
|
102
|
+
- **`sync` checks that the typed-artifact tools stay out of scheduled tasks.** They come from a server the
|
|
103
|
+
init-surface read never records, so the Desktop code that withholds them from a scheduled task is the only place
|
|
104
|
+
that difference can be checked.
|
|
105
|
+
|
|
9
106
|
## [4.11.0] - 2026-10-23
|
|
10
107
|
|
|
11
108
|
### Upgrade notes
|
package/DESIGN.md
CHANGED
|
@@ -55,7 +55,7 @@ in how a file reaches the user, which is what changes skill behaviour: see
|
|
|
55
55
|
[docs/scenario.md](./docs/scenario.md)'s `lane:` key for holding a run to either contract. The local lane:
|
|
56
56
|
|
|
57
57
|
- VM bundle: `~/Library/Application Support/Claude/vm_bundles/claudevm.bundle/` (`rootfs.img`, `sessiondata.img`, `efivars.fd`, `machineIdentifier`, `gvisorMacAddress`, `vmIP`); a warm pool at `vm_bundles/warm/<sha>/`.
|
|
58
|
-
- In-VM agent: `~/Library/Application Support/Claude/claude-code-vm/<ver>/claude` (currently **2.1.
|
|
58
|
+
- In-VM agent: `~/Library/Application Support/Claude/claude-code-vm/<ver>/claude` (currently **2.1.309**, per `baselines/desktop-4.4212.0.json`, an **ELF aarch64** binary), spawned by the host **in cowork mode via the `CLAUDE_CODE_IS_COWORK=1` env var** — *not* a `--cowork` flag (that flag is plugin-scope and the staged agent rejects it; see the control-protocol note below). Each baseline records this ELF's `sha256` (`agentBinary.sha256`/`shaProvenance`), and the resolver integrity-checks the binary it's about to run against that hash by default (opt out `COWORK_HARNESS_VERIFY_AGENT_SHA=0`), so "the same pinned agent" is enforced, not just asserted. Old versions are re-downloadable + verifiable from the official release channel — from the channel that baseline's `agentBinary.releaseBaseUrl` names, which is **not always the stable path** (Desktop stages release candidates too); the exact command is in `docs/maintenance.md`.
|
|
59
59
|
- Network: `vm_network_mode: "gvisor"`, egress through a userspace netstack with a **compiled domain allowlist**; off-list partners rejected (`partner rejected: entry not on compiled allowlist`).
|
|
60
60
|
- Control plane: Electron renderer→main typed IPC on channels named `$eipc_message$_<per-build-UUID>_$_claude.web_$_<Class>_$_<method>`, every handler validating `event.senderFrame.url` against a trusted-origin allowlist. The session manager is `LocalAgentModeSessions` (80 methods: `start`, `sendMessage`, `setDraftSessionFolders`, `onToolPermissionRequest`, `respondToToolPermission`, `getTranscript`, `onEvent`, …), bridged to the renderer as `window.cowork`.
|
|
61
61
|
|
|
@@ -125,6 +125,7 @@ STABLE (rarely changes; lives in code)
|
|
|
125
125
|
VOLATILE (changes per release; lives in baselines/*.json, sync-regenerated)
|
|
126
126
|
- agentVersion
|
|
127
127
|
- network.allowDomains + network.mode + requireFullVmSandbox
|
|
128
|
+
- network.webFetchVendorDeny (Desktop 4.4212.0+; presence read from the asar, value the pinned list)
|
|
128
129
|
- gates
|
|
129
130
|
- asarFingerprint (provenance + "unknown delta" tripwire)
|
|
130
131
|
- cloud (the remote-devices tool list + description fingerprints; data only, never carried forward;
|
|
@@ -208,7 +209,7 @@ The policy that produces those `allow`/`deny` responses is the **Decider** seam
|
|
|
208
209
|
> Cowork system-prompt fingerprint all unchanged from `1.20186.0`, with the staged VM ELF re-synced
|
|
209
210
|
> 2.1.202 → 2.1.205 — and the live pass of that era was deliberately **not** restamped onto it.)
|
|
210
211
|
|
|
211
|
-
> **Scope of that claim.** `2026-10-23 / desktop-3.1623.4` is the baseline carrying the latest live pass, run against agent **2.1.308** (the staged Linux ELF for `container`, and the staged native macOS build `a93ded4b1b47` for `hostloop`, with no override of either); the `protocol` tier runs the HOST `claude`, **2.1.309** for this pass. No baselines have shipped since. **The 4.11.0 pass, 2026-10-23, on `desktop-3.1623.4` (agent 2.1.308; host `claude` 2.1.309 for `protocol`), on the release-prep branch at `a1e3c98d`, with the debugging-reference fix `c1f31ae4` for the steps marked below**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with eight warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`) and wrote a scenario that passes `lint` with no error; the debugging prompt read `debugging.md` but again named a `trace --view gates` that does not exist, so `debugging.md` and `docs/debugging.md` now name the gate view (`questions`), and the re-run at `c1f31ae4` read `debugging.md` and named `--view questions` and `--view dispatches`; the tool-timing prompt read `measurement.md` (and `debugging.md`) and answered correctly, with `trace` flags that exist; (iv) `cowork-harness run examples/scenarios/` at `c1f31ae4`, the first live regression on this baseline: six of eight passed on the first run (`container` ×5, `hostloop-computer-links`). `subagent-manifest-probe` (`hostloop`) failed `no_vm_path_file_op` once: the parent's dispatch prompt named the `/sessions/` path for the shell, the sub-agent tried a `Write` there, the path gate refused it, and the sub-agent then wrote the host path; the re-run passed every assertion. `protocol-smoke` errored with "Invalid API key" because the operator `.env` it loaded also held a stale `ANTHROPIC_API_KEY`, which `protocol` passes to the host `claude` ahead of the OAuth token (the agent reported `apiKeySource: ANTHROPIC_API_KEY`); with only the token loaded it passed. The live suite itself was not re-run, and no cassette was re-recorded (`verify-cassettes` reports none stale). Live spend about **$3.53**. **The 4.10.0 pass, 2026-10-16, on `desktop-2.42304.0` (agent 2.1.299; host `claude` 2.1.301 for `protocol`), on the release-prep branch at `737d4f35`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`) and wrote a scenario that passes `lint` with no error; the debugging prompt read `debugging.md` and gave the right steps, but also named a `trace --view gates` that does not exist (`trace` refuses it, exit 2, and lists the views; neither the skill nor the docs name it); the tool-timing prompt read `measurement.md` (and `debugging.md`) and answered correctly, with `trace` flags that exist; (iv) the frozen search corpus on `hostloop` (`claude-sonnet-5`): a seed run with two real searches built a two-record pool. A run of the same prompt from that pool was served both searches as exact matches (`served 2, missed 0, refused 0`), every live verdict equal to the ranking replayed offline from the built `dist/`, with the label on `verify-run`, `status`, `trace` and the grouped `stats` summary, whose `searchCorpus.sha256` equals the pool's. A run with a shifted prompt was served one nearby round and one miss; `verify-run` then graded both keys on each side of that run's real values: `search_corpus_max_miss_rate` failed at 0.49 and passed at 0.5 on a miss rate of 1/2, and `search_corpus_nearby_citations` failed at 8 and passed at 9 on nine cited URLs served only on nearby rounds. Both keys were refused (exit 2) by `run` without `--search-corpus` and by `verify-run` on a run served without one, and a judged assert on a frozen run carried `search_corpus_semantic_judge`. `record --search-corpus` stays refused. The three cassette re-records of 2026-10-16 on agent 2.1.299 (`example-pdf-skill` and `test/fixtures/tool-call-dispatch/dispatch-shell.cassette.json` on `container`, `hostloop-computer-links` on `hostloop`) carry no `memory_paths`. The full live suite was not re-run. Live spend about **$1.51**, plus **$0.24** on a first attempt stopped when Claude Desktop updated mid-gate. **The 4.9.1 pass, 2026-10-14, on `desktop-2.37337.1` (agent 2.1.295; host `claude` 2.1.299 for `protocol`), on the release-prep branch at `cc753ad5`** (a patch; the commits after it change docs, the CHANGELOG and the `critique` help line only): (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every run `delivered_clean` and every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`) and wrote a scenario that passes `lint` with no error, the debugging prompt read `debugging.md` and answered correctly, and the tool-timing prompt read `measurement.md` (and `debugging.md`) and answered correctly, with `trace` flags that exist. The full live suite was not re-run: the code changes in this release are to help text, `search-pool`, the companion skill's linter, the sync extractors, a hillclimb refactor that leaves every flag unchanged, and maintainer scripts, none of which a live scenario run reaches. Live spend about **$1.06**. **The 4.7.1 pass, 2026-10-10, on `desktop-2.31226.1` (agent 2.1.295; host `claude` 2.1.296 for `protocol`), on the release-prep branch at `3c13a4f8`** (a patch: a smoke on two tiers, not the live suite; the one code change after it widens which task types the usage-limit check reads, on a path neither run reached): (i) `container`, the sub-agent dispatch scenario (`test/fixtures/tool-call-dispatch/dispatch-shell.yaml`, `claude-sonnet-5`): a real sub-agent ran and completed, the stream carried `task_started` (`task_type: local_agent`), a `task_updated` whose patch was `completed` with no error, and a completed `task_notification`; the run passed, `delivered_clean`, with no `subagent_usage_limit` row; (ii) `protocol`, `examples/scenarios/protocol-smoke.yaml`: the first run had no credential in its environment, so it read the operator's real config dir and its verdict failed on `l0_host_config_contamination` alone (`result: success`, every assertion passed, no `subagent_usage_limit` row); the re-run with `COWORK_MANAGED_CONFIG=1` and the credential loaded read a sealed config dir and passed, `delivered_clean`, with no fail signal. A sub-agent that hits the usage limit was not triggered live (it needs a spent quota); that path is covered by tests built from the agent's frame shapes. Live spend about **$0.88**. **The 4.7.0 pass, 2026-10-10, on `desktop-2.31226.1` (agent 2.1.295; host `claude` 2.1.296 for `protocol`), on the release-prep branch at `2dba294f`** (the commits after it change docs only): (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 24 passed, one pre-registered skip; (iii) `artifact_json` `schema:` on `container`: a whole-document schema passed, and a schema of `type: string` on the integer at `count` failed alone, naming the path and "must be string"; a schema using `format` was refused at load (exit 2) before any spend, and `lint` reported it as an error; (iv) `fixture export --session-paths` on `container`: without the flag, export refused the file holding the run's outputs path (`run_path`) and wrote nothing; with it, the path became `__COWORK_HARNESS_VM_SESSION_ROOT__/mnt/outputs`, and a next step staged from that fixture read its own new session's path, with no token left; (v) `record <dir/> --case`: the `--dry-run` budget check priced the named scenario alone and passed where the whole directory was refused, and a real `--case` record wrote one cassette for the named scenario only; it carries `hookFailureBlocks: {"events": []}` and stamps v12, as before; (vi) `onFailure: "block"` on `container`: two runs identical but for that field on a PreToolUse hook that exits 1 on the prompted Bash call. Both recorded the same failed hook frame (exit 1, `outcome: error`). With the field, the agent blocked the call, the run's inventory listed `PreToolUse`, and both forms of `no_hook_event_blocked` failed as evidence-unavailable; without it, the call ran, the inventory was empty, and both passed. Recorded with `--allow-failing`, the first run stamped v16, and its replay reproduced the two evidence-unavailable results; (vii) the companion skill's router check, the authoring prompt on `container`: the first run's first `AskUserQuestion` held a single option and was rejected by the agent's own input validation, and the run ended on a question asked in prose (graded `stalled`), with `authoring.md` read but no scenario written, as in 4.6.1. The release checklist's prompt now names the skill folder and the report path; the re-run read `authoring.md` (and `scenario-schema.md` and an assertion-catalog part) and wrote a session file and a scenario that pass `lint` with no error or warning. Live spend about **$2.07**. **The 4.6.1 pass, 2026-10-09, on `desktop-2.31226.0` (agent 2.1.293; host `claude` 2.1.295 for `protocol`), on the release-prep branch at `e5ca4bd9`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip, and one red in the plugin hook-decision test: the model made a second, unprompted Bash call, the probe's hook denied both, and the harness counted both, so the probe's exact count of one Bash denial failed. The probe now denies only the calls its prompt asks for, and the re-run passed all twelve assertions with exactly one denial each from the Bash, Write and Stop hooks; (iii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iv) Ctrl-C on `hostloop` on macOS, SIGINT to the harness's whole process group while the agent ran `sleep 120` in the sandbox: exit 130 in 1.28 s, and in 1.33 s with twelve concurrent `ps -A -o tty=` loops running, where the same stop took about 8 s before 4.6.1; both times nothing in the process tree captured before the signal survived, the sandbox container was gone and `status.json` ended `error`; (v) a `lane: remote` scenario asserting `artifact_text`, and one asserting `semantic_matches` with `evidence_files`, are refused by `run` at load (exit 2) before any spend, naming the key and a remedy, and `lint` reports them as errors (exit 1); (vi) the companion skill's router check, the authoring prompt on `container`: one run's first `AskUserQuestion` was rejected by the agent's own input validation and the run ended by asking for the skill's path, with no reference read and no scenario written; the re-run read `authoring.md` (and `scenario-schema.md` and an assertion-catalog part) and wrote a scenario that passes `lint` with no error or warning. Live spend about **$2.06**. **The 4.6.0 pass, 2026-10-08, on `desktop-2.26454.2` (agent 2.1.293; host `claude` 2.1.294 for `protocol`), on the release-prep branch at `daf3479d`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 24 passed, one pre-registered skip, including the plugin hook-decision test on `container` (a PreToolUse hook denying by JSON, one exiting 2, and a Stop hook blocking once, graded by `hook_decision` and both forms of `hook_event_blocked`); (iii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iv) the companion skill's router check, three prompts on `container`, against the split assertion catalog: the authoring prompt read `authoring.md` and `scenario-schema.md` and wrote a session file and a scenario that pass `lint` with no error or warning, the debugging prompt read `debugging.md` and asked for the run's `result.json`, and the tool-timing prompt read `measurement.md` and answered with `trace … --view tool-durations`; (v) `gates_all_scripted` on `container`: a scripted multi-select gate passed, and the same scenario with its answer removed and `on_unanswered: first` failed on exactly that assertion, naming the gate as answered by `first`; (vi) `answer_channel: none` on `container`, with a probe skill that records a status file before asking: the run passed with a `parked_at_question` warning and no `stalled` fail, carried the headless label and `answerChannel: "none"`, graded the status file through an `artifact_json` glob, and its init frame listed no `AskUserQuestion`. Live spend about **$2.21**. The 4.4.1, 4.4.0 and 4.3.0 passes below ran on `desktop-2.19675.0` (agent 2.1.286). **The 4.5.0 pass, 2026-10-07, on `desktop-2.26454.0` (agent 2.1.289; host `claude` 2.1.292 for `protocol`), on the release-prep branch at `af26c3c7`** (the commits after it change docs and the companion skill only): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip; (iii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iv) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `assertion-catalog.md`, searching `scenario-schema.md`) but wrote a scenario with an inline `session:` block, which fails `lint`, so it failed; `authoring.md` now says `session:` is a path, and one re-run read `authoring.md` and `scenario-schema.md` and asked for the skill's path instead of writing a scenario, so the corrected sentence is checked by `lint`, not by a live answer. The debugging prompt read no reference and answered with correct clarifying questions; the tool-timing prompt loaded the skill, read no reference and asked which run to time, so `measurement.md` routing was not exercised in this pass; (v) the forced-compaction recipe on `container`: a task turn, a resumed `/compact` turn whose `result.json` records a manual `compact_boundary` (40,266 → 2,896 tokens; the check exits 1 on the task turn), and a resumed turn that still named the command it had described before the compaction. The recipe first said to check this with `verify-run`, which refuses a run dir with more than one turn; it now reads the `/compact` turn's own `result.json`. Live spend about **$2.53**. **The 4.4.1 pass, 2026-10-05, on the release-prep commit `3e6fe443`** (a companion-skill-only release; same baseline and staged agent; host `claude` 2.1.289 for `protocol`): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`, searching `assertion-catalog.md`) and wrote a scenario that passes `lint` with no error or warning, the debugging prompt read `debugging.md` and answered with correct clarifying questions, and the tool-timing prompt read `measurement.md` (and `debugging.md`, which documents `trace`'s views) and answered with `trace … --view tool-durations`. The live suite itself was not re-run: this release changes no code that runs. Live spend about **$1.08**. **The 4.4.0 pass, 2026-10-05, on the release-prep commit `e9c516df`** (same baseline and staged agent; host `claude` 2.1.289 for `protocol`): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip; (iii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each; (iv) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` and wrote a scenario that passes `lint`, the tool-timing prompt read `measurement.md` and answered with `trace … --view tool-durations`, and the debugging prompt read no reference and answered with correct clarifying questions, so `debugging.md` routing was not exercised in this pass; (v) the re-grade scrub-set check: a live run on the default runs root created `scrubset.key` beside the runs directory (mode 0600) and recorded `scrubSet` (`v: 1`) with the configured scrub literal absent from `result.json`; a re-grade of that run with an edited rubric line and a smaller scrub set was refused as `rubric_unverifiable` (exit 2) naming only the edited line, with no judge call and the run directory byte-identical; and a re-grade of a run recorded before 4.4 with an edited rubric line was refused with the `legacy` reason and the one-line remedy, again with no judge call and the run directory byte-identical. Live spend about **$2.83** recorded. **The 4.3.0 pass, on the release commit `8a5e479f`:** (1) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip (the auto-memory prerequisites check, which runs only under `COWORK_LIVE_REQUIRE=1`); before it ran, `vitest list --staticParse=false` listed all 23 with no `SKIPPED` warning, and 6 with four warnings when no token was resolvable, so a silent skip would have shown; (2) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (3) the graders' pinned effort: every judged assert that called the judge in the live hillclimb acceptance runs (45, Opus 4.8: 25 `semantic_matches`, 20 `semantic_pairwise`; host `claude` 2.1.288 and 2.1.289) records `judgeTransport.effort: "high"`; the five pairwise asserts without a `judgeTransport` are the reference variant's own rows, which call no judge. **Earlier in the same release, on the release candidate's tree:** (4) a `microvm` smoke on a freshly booted VM (agent 2.1.286); (5) `boundary-check`, 6/6 constraints enforced; (6) an `eval` A/A at `--reps 4` on `container`, every row `no detectable change`, its report rebuilt byte-identical; (7) the companion skill's router check, three prompts on `container`: routing was correct on all three; one answer was wrong because a reference showed `artifact_text`'s `contains` without its list shape (fixed in the references, with a test that keeps every list-typed field shown as a list), and the same prompt's verdict went red on a false `host_path_leak`, because a reference spelled out the host roots that check looks for and the agent echoed it (fixed in the skill's files and in the check, which no longer counts a literal that comes from the plugin's or a local skill's own staged files); re-run after the fixes, the prompt routed correctly, wrote a scenario that passes `lint` and loads, and carried no `host_path_leak`; (8) the `hostloop` detached-agent kill check: the first pass found the workspace sidecar container, its `docker run` client and its network surviving SIGINT, which was fixed, and the re-run passed on a normal run, SIGINT to the harness process and SIGINT to the whole process group (exit 130, nothing left behind); (9) `vm delete`'s usage refusal; (10) `chat` under a real terminal, on `protocol`: Ctrl-C mid-turn exits 130 in about 2 s with nothing left running and no `result.json`; `/exit` followed by Ctrl-C exits 130 after writing the turn's `result.json`; Ctrl-C at the idle prompt ends the session normally (exit 0, `result.json` written); SIGHUP mid-turn exits 129 with nothing left running; and closing the terminal window mid-turn (run by hand) leaves no harness or agent process and no `result.json`. Total live spend about **$11.20**, including the re-runs. **Scope-out, so this is not read as more than it is.** (a) A live pass verifies observed behaviour, not the whole spawn contract by construction, and these checks are model-dependent, so a single red is evidence of model variance until a re-run says otherwise. (b) **CI does not live-validate anything.** Its "scenario suite (… live inference)" job is skipped as a whole without an `ANTHROPIC_API_KEY` repository secret, and none is set, so it shows as *skipped*. Before 2026-09 it instead ran with every real step skipped and reported *success*, so an older green check there means nothing ran. This note, not CI, is the live evidence. Separately and not a live matter: all four committed cassettes were **re-recorded** on 2026-10-03 against `2.19675.0` with auto-memory off, each on its original model: `example-pdf-skill` and `test/fixtures/tool-call-dispatch/dispatch-shell.cassette.json` (`container`, agent 2.1.286), `hostloop-computer-links` (`hostloop`, agent 2.1.286) and `example-multiselect-gate` (`protocol`, which runs the host CLI, here 2.1.287), about $0.75 in all. Their init frames carry no `memory_paths`. The recorder's own scrub (now applied to every recording) removed, across the four, the account's model menu, a subscription account's `rate_limit_info` (from `example-pdf-skill`, `dispatch-shell` and `hostloop-computer-links`) and the agent's sub-agent hand-back frame (from `dispatch-shell`). `verify-cassettes` exits 0 on all four, with one accepted `unscanned` entry (`example-pdf-skill`'s uploaded artifact body, too large to commit). Re-stamp this paragraph, naming the baseline, whenever a live pass is actually re-run.
|
|
212
|
+
> **Scope of that claim.** `2026-11-01 / desktop-4.4212.0` is the baseline carrying the latest live pass, run against agent **2.1.309** (the staged Linux ELF for `container`, and the staged native macOS build `5af865b9901c` for `hostloop`, with no override of either); the `protocol` tier runs the HOST `claude`, **2.1.316** for this pass. No baselines have shipped since. The `3.6556.2` sync reported two unknown deltas, both classified before the baseline was written: the host-loop permission wrapper's new Artifact file capture (pinned, not modelled) and a project-attribution spawn key set only when one claude.ai Project is selected (allowlisted). The `4.4212.0` sync (read from a scheduled task, the only local session the account runs) classified a second scheduled-run spawn key, a canUseTool link skipped under the host CLI launcher (inert), the frame-tools predicate admitting chat sessions, and the prompt template's move to a lazy chunk, and ported the `web_fetch` vendor-host refusal (Path B). The Cowork system prompt bundled with Desktop and the egress allowlist are unchanged in both; a signed-in session can receive its prompt from Anthropic's servers instead, which no sync sees. **The 4.13.0 pass, 2026-11-01, on `desktop-4.4212.0` (agent 2.1.309; host `claude` 2.1.316 for `protocol`), on the release-prep branch at `c3e2a1a0`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with eight warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`) and wrote a scenario that passes `lint` with no error or warning; the debugging prompt read `debugging.md` and answered correctly; the tool-timing prompt read `measurement.md` (and `debugging.md`) and answered correctly, with `trace` flags that exist; (iv) `cowork-harness run examples/scenarios/`, the first live regression on this baseline: seven of eight passed (`container` ×5, `hostloop-computer-links`, `protocol-smoke`). `subagent-manifest-probe` (`hostloop`) failed twice, both times after the parent's dispatch prompt spelled out a `/sessions/` path for the note: once the sub-agent did all of its work in the shell and never called `Read` or `Write`, and once it called `Write` on that path and the path gate refused it. The total was right both times. The probe passed on a re-run in the 4.11.0 pass and failed in the same way once there; its header says a green shows the sub-agent used the host path, not where it learned it. The live suite itself was not re-run, and no cassette was re-recorded (`verify-cassettes` reports none stale). Live spend about **$3.33**. **The 4.11.0 pass, 2026-10-23, on `desktop-3.1623.4` (agent 2.1.308; host `claude` 2.1.309 for `protocol`), on the release-prep branch at `a1e3c98d`, with the debugging-reference fix `c1f31ae4` for the steps marked below**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with eight warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`) and wrote a scenario that passes `lint` with no error; the debugging prompt read `debugging.md` but again named a `trace --view gates` that does not exist, so `debugging.md` and `docs/debugging.md` now name the gate view (`questions`), and the re-run at `c1f31ae4` read `debugging.md` and named `--view questions` and `--view dispatches`; the tool-timing prompt read `measurement.md` (and `debugging.md`) and answered correctly, with `trace` flags that exist; (iv) `cowork-harness run examples/scenarios/` at `c1f31ae4`, the first live regression on this baseline: six of eight passed on the first run (`container` ×5, `hostloop-computer-links`). `subagent-manifest-probe` (`hostloop`) failed `no_vm_path_file_op` once: the parent's dispatch prompt named the `/sessions/` path for the shell, the sub-agent tried a `Write` there, the path gate refused it, and the sub-agent then wrote the host path; the re-run passed every assertion. `protocol-smoke` errored with "Invalid API key" because the operator `.env` it loaded also held a stale `ANTHROPIC_API_KEY`, which `protocol` passes to the host `claude` ahead of the OAuth token (the agent reported `apiKeySource: ANTHROPIC_API_KEY`); with only the token loaded it passed. The live suite itself was not re-run, and no cassette was re-recorded (`verify-cassettes` reports none stale). Live spend about **$3.53**. **The 4.10.0 pass, 2026-10-16, on `desktop-2.42304.0` (agent 2.1.299; host `claude` 2.1.301 for `protocol`), on the release-prep branch at `737d4f35`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`) and wrote a scenario that passes `lint` with no error; the debugging prompt read `debugging.md` and gave the right steps, but also named a `trace --view gates` that does not exist (`trace` refuses it, exit 2, and lists the views; neither the skill nor the docs name it); the tool-timing prompt read `measurement.md` (and `debugging.md`) and answered correctly, with `trace` flags that exist; (iv) the frozen search corpus on `hostloop` (`claude-sonnet-5`): a seed run with two real searches built a two-record pool. A run of the same prompt from that pool was served both searches as exact matches (`served 2, missed 0, refused 0`), every live verdict equal to the ranking replayed offline from the built `dist/`, with the label on `verify-run`, `status`, `trace` and the grouped `stats` summary, whose `searchCorpus.sha256` equals the pool's. A run with a shifted prompt was served one nearby round and one miss; `verify-run` then graded both keys on each side of that run's real values: `search_corpus_max_miss_rate` failed at 0.49 and passed at 0.5 on a miss rate of 1/2, and `search_corpus_nearby_citations` failed at 8 and passed at 9 on nine cited URLs served only on nearby rounds. Both keys were refused (exit 2) by `run` without `--search-corpus` and by `verify-run` on a run served without one, and a judged assert on a frozen run carried `search_corpus_semantic_judge`. `record --search-corpus` stays refused. The three cassette re-records of 2026-10-16 on agent 2.1.299 (`example-pdf-skill` and `test/fixtures/tool-call-dispatch/dispatch-shell.cassette.json` on `container`, `hostloop-computer-links` on `hostloop`) carry no `memory_paths`. The full live suite was not re-run. Live spend about **$1.51**, plus **$0.24** on a first attempt stopped when Claude Desktop updated mid-gate. **The 4.9.1 pass, 2026-10-14, on `desktop-2.37337.1` (agent 2.1.295; host `claude` 2.1.299 for `protocol`), on the release-prep branch at `cc753ad5`** (a patch; the commits after it change docs, the CHANGELOG and the `critique` help line only): (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every run `delivered_clean` and every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`) and wrote a scenario that passes `lint` with no error, the debugging prompt read `debugging.md` and answered correctly, and the tool-timing prompt read `measurement.md` (and `debugging.md`) and answered correctly, with `trace` flags that exist. The full live suite was not re-run: the code changes in this release are to help text, `search-pool`, the companion skill's linter, the sync extractors, a hillclimb refactor that leaves every flag unchanged, and maintainer scripts, none of which a live scenario run reaches. Live spend about **$1.06**. **The 4.7.1 pass, 2026-10-10, on `desktop-2.31226.1` (agent 2.1.295; host `claude` 2.1.296 for `protocol`), on the release-prep branch at `3c13a4f8`** (a patch: a smoke on two tiers, not the live suite; the one code change after it widens which task types the usage-limit check reads, on a path neither run reached): (i) `container`, the sub-agent dispatch scenario (`test/fixtures/tool-call-dispatch/dispatch-shell.yaml`, `claude-sonnet-5`): a real sub-agent ran and completed, the stream carried `task_started` (`task_type: local_agent`), a `task_updated` whose patch was `completed` with no error, and a completed `task_notification`; the run passed, `delivered_clean`, with no `subagent_usage_limit` row; (ii) `protocol`, `examples/scenarios/protocol-smoke.yaml`: the first run had no credential in its environment, so it read the operator's real config dir and its verdict failed on `l0_host_config_contamination` alone (`result: success`, every assertion passed, no `subagent_usage_limit` row); the re-run with `COWORK_MANAGED_CONFIG=1` and the credential loaded read a sealed config dir and passed, `delivered_clean`, with no fail signal. A sub-agent that hits the usage limit was not triggered live (it needs a spent quota); that path is covered by tests built from the agent's frame shapes. Live spend about **$0.88**. **The 4.7.0 pass, 2026-10-10, on `desktop-2.31226.1` (agent 2.1.295; host `claude` 2.1.296 for `protocol`), on the release-prep branch at `2dba294f`** (the commits after it change docs only): (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 24 passed, one pre-registered skip; (iii) `artifact_json` `schema:` on `container`: a whole-document schema passed, and a schema of `type: string` on the integer at `count` failed alone, naming the path and "must be string"; a schema using `format` was refused at load (exit 2) before any spend, and `lint` reported it as an error; (iv) `fixture export --session-paths` on `container`: without the flag, export refused the file holding the run's outputs path (`run_path`) and wrote nothing; with it, the path became `__COWORK_HARNESS_VM_SESSION_ROOT__/mnt/outputs`, and a next step staged from that fixture read its own new session's path, with no token left; (v) `record <dir/> --case`: the `--dry-run` budget check priced the named scenario alone and passed where the whole directory was refused, and a real `--case` record wrote one cassette for the named scenario only; it carries `hookFailureBlocks: {"events": []}` and stamps v12, as before; (vi) `onFailure: "block"` on `container`: two runs identical but for that field on a PreToolUse hook that exits 1 on the prompted Bash call. Both recorded the same failed hook frame (exit 1, `outcome: error`). With the field, the agent blocked the call, the run's inventory listed `PreToolUse`, and both forms of `no_hook_event_blocked` failed as evidence-unavailable; without it, the call ran, the inventory was empty, and both passed. Recorded with `--allow-failing`, the first run stamped v16, and its replay reproduced the two evidence-unavailable results; (vii) the companion skill's router check, the authoring prompt on `container`: the first run's first `AskUserQuestion` held a single option and was rejected by the agent's own input validation, and the run ended on a question asked in prose (graded `stalled`), with `authoring.md` read but no scenario written, as in 4.6.1. The release checklist's prompt now names the skill folder and the report path; the re-run read `authoring.md` (and `scenario-schema.md` and an assertion-catalog part) and wrote a session file and a scenario that pass `lint` with no error or warning. Live spend about **$2.07**. **The 4.6.1 pass, 2026-10-09, on `desktop-2.31226.0` (agent 2.1.293; host `claude` 2.1.295 for `protocol`), on the release-prep branch at `e5ca4bd9`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip, and one red in the plugin hook-decision test: the model made a second, unprompted Bash call, the probe's hook denied both, and the harness counted both, so the probe's exact count of one Bash denial failed. The probe now denies only the calls its prompt asks for, and the re-run passed all twelve assertions with exactly one denial each from the Bash, Write and Stop hooks; (iii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iv) Ctrl-C on `hostloop` on macOS, SIGINT to the harness's whole process group while the agent ran `sleep 120` in the sandbox: exit 130 in 1.28 s, and in 1.33 s with twelve concurrent `ps -A -o tty=` loops running, where the same stop took about 8 s before 4.6.1; both times nothing in the process tree captured before the signal survived, the sandbox container was gone and `status.json` ended `error`; (v) a `lane: remote` scenario asserting `artifact_text`, and one asserting `semantic_matches` with `evidence_files`, are refused by `run` at load (exit 2) before any spend, naming the key and a remedy, and `lint` reports them as errors (exit 1); (vi) the companion skill's router check, the authoring prompt on `container`: one run's first `AskUserQuestion` was rejected by the agent's own input validation and the run ended by asking for the skill's path, with no reference read and no scenario written; the re-run read `authoring.md` (and `scenario-schema.md` and an assertion-catalog part) and wrote a scenario that passes `lint` with no error or warning. Live spend about **$2.06**. **The 4.6.0 pass, 2026-10-08, on `desktop-2.26454.2` (agent 2.1.293; host `claude` 2.1.294 for `protocol`), on the release-prep branch at `daf3479d`**: (i) `vitest list --staticParse=false` listed all 24 live tests with no `SKIPPED` warning when the token was exported, and 6 with five warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 24 passed, one pre-registered skip, including the plugin hook-decision test on `container` (a PreToolUse hook denying by JSON, one exiting 2, and a Stop hook blocking once, graded by `hook_decision` and both forms of `hook_event_blocked`); (iii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iv) the companion skill's router check, three prompts on `container`, against the split assertion catalog: the authoring prompt read `authoring.md` and `scenario-schema.md` and wrote a session file and a scenario that pass `lint` with no error or warning, the debugging prompt read `debugging.md` and asked for the run's `result.json`, and the tool-timing prompt read `measurement.md` and answered with `trace … --view tool-durations`; (v) `gates_all_scripted` on `container`: a scripted multi-select gate passed, and the same scenario with its answer removed and `on_unanswered: first` failed on exactly that assertion, naming the gate as answered by `first`; (vi) `answer_channel: none` on `container`, with a probe skill that records a status file before asking: the run passed with a `parked_at_question` warning and no `stalled` fail, carried the headless label and `answerChannel: "none"`, graded the status file through an `artifact_json` glob, and its init frame listed no `AskUserQuestion`. Live spend about **$2.21**. The 4.4.1, 4.4.0 and 4.3.0 passes below ran on `desktop-2.19675.0` (agent 2.1.286). **The 4.5.0 pass, 2026-10-07, on `desktop-2.26454.0` (agent 2.1.289; host `claude` 2.1.292 for `protocol`), on the release-prep branch at `af26c3c7`** (the commits after it change docs and the companion skill only): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip; (iii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iv) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `assertion-catalog.md`, searching `scenario-schema.md`) but wrote a scenario with an inline `session:` block, which fails `lint`, so it failed; `authoring.md` now says `session:` is a path, and one re-run read `authoring.md` and `scenario-schema.md` and asked for the skill's path instead of writing a scenario, so the corrected sentence is checked by `lint`, not by a live answer. The debugging prompt read no reference and answered with correct clarifying questions; the tool-timing prompt loaded the skill, read no reference and asked which run to time, so `measurement.md` routing was not exercised in this pass; (v) the forced-compaction recipe on `container`: a task turn, a resumed `/compact` turn whose `result.json` records a manual `compact_boundary` (40,266 → 2,896 tokens; the check exits 1 on the task turn), and a resumed turn that still named the command it had described before the compaction. The recipe first said to check this with `verify-run`, which refuses a run dir with more than one turn; it now reads the `/compact` turn's own `result.json`. Live spend about **$2.53**. **The 4.4.1 pass, 2026-10-05, on the release-prep commit `3e6fe443`** (a companion-skill-only release; same baseline and staged agent; host `claude` 2.1.289 for `protocol`): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (iii) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` (and `scenario-schema.md`, searching `assertion-catalog.md`) and wrote a scenario that passes `lint` with no error or warning, the debugging prompt read `debugging.md` and answered with correct clarifying questions, and the tool-timing prompt read `measurement.md` (and `debugging.md`, which documents `trace`'s views) and answered with `trace … --view tool-durations`. The live suite itself was not re-run: this release changes no code that runs. Live spend about **$1.08**. **The 4.4.0 pass, 2026-10-05, on the release-prep commit `e9c516df`** (same baseline and staged agent; host `claude` 2.1.289 for `protocol`): (i) `vitest list --staticParse=false` listed all 23 live tests with no `SKIPPED` warning when the token was exported, and 6 with four warnings when none was resolvable; (ii) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip; (iii) the auto-memory check on all three tiers, `Tests 2 passed (2)` each; (iv) the companion skill's router check, three prompts on `container`: the authoring prompt read `authoring.md` and wrote a scenario that passes `lint`, the tool-timing prompt read `measurement.md` and answered with `trace … --view tool-durations`, and the debugging prompt read no reference and answered with correct clarifying questions, so `debugging.md` routing was not exercised in this pass; (v) the re-grade scrub-set check: a live run on the default runs root created `scrubset.key` beside the runs directory (mode 0600) and recorded `scrubSet` (`v: 1`) with the configured scrub literal absent from `result.json`; a re-grade of that run with an edited rubric line and a smaller scrub set was refused as `rubric_unverifiable` (exit 2) naming only the edited line, with no judge call and the run directory byte-identical; and a re-grade of a run recorded before 4.4 with an edited rubric line was refused with the `legacy` reason and the one-line remedy, again with no judge call and the run directory byte-identical. Live spend about **$2.83** recorded. **The 4.3.0 pass, on the release commit `8a5e479f`:** (1) the live suite on `protocol`, `container` and `hostloop`: 23 passed, one pre-registered skip (the auto-memory prerequisites check, which runs only under `COWORK_LIVE_REQUIRE=1`); before it ran, `vitest list --staticParse=false` listed all 23 with no `SKIPPED` warning, and 6 with four warnings when no token was resolvable, so a silent skip would have shown; (2) the auto-memory check on all three tiers, `Tests 2 passed (2)` each, every init frame without `memory_paths`; (3) the graders' pinned effort: every judged assert that called the judge in the live hillclimb acceptance runs (45, Opus 4.8: 25 `semantic_matches`, 20 `semantic_pairwise`; host `claude` 2.1.288 and 2.1.289) records `judgeTransport.effort: "high"`; the five pairwise asserts without a `judgeTransport` are the reference variant's own rows, which call no judge. **Earlier in the same release, on the release candidate's tree:** (4) a `microvm` smoke on a freshly booted VM (agent 2.1.286); (5) `boundary-check`, 6/6 constraints enforced; (6) an `eval` A/A at `--reps 4` on `container`, every row `no detectable change`, its report rebuilt byte-identical; (7) the companion skill's router check, three prompts on `container`: routing was correct on all three; one answer was wrong because a reference showed `artifact_text`'s `contains` without its list shape (fixed in the references, with a test that keeps every list-typed field shown as a list), and the same prompt's verdict went red on a false `host_path_leak`, because a reference spelled out the host roots that check looks for and the agent echoed it (fixed in the skill's files and in the check, which no longer counts a literal that comes from the plugin's or a local skill's own staged files); re-run after the fixes, the prompt routed correctly, wrote a scenario that passes `lint` and loads, and carried no `host_path_leak`; (8) the `hostloop` detached-agent kill check: the first pass found the workspace sidecar container, its `docker run` client and its network surviving SIGINT, which was fixed, and the re-run passed on a normal run, SIGINT to the harness process and SIGINT to the whole process group (exit 130, nothing left behind); (9) `vm delete`'s usage refusal; (10) `chat` under a real terminal, on `protocol`: Ctrl-C mid-turn exits 130 in about 2 s with nothing left running and no `result.json`; `/exit` followed by Ctrl-C exits 130 after writing the turn's `result.json`; Ctrl-C at the idle prompt ends the session normally (exit 0, `result.json` written); SIGHUP mid-turn exits 129 with nothing left running; and closing the terminal window mid-turn (run by hand) leaves no harness or agent process and no `result.json`. Total live spend about **$11.20**, including the re-runs. **Scope-out, so this is not read as more than it is.** (a) A live pass verifies observed behaviour, not the whole spawn contract by construction, and these checks are model-dependent, so a single red is evidence of model variance until a re-run says otherwise. (b) **CI does not live-validate anything.** Its "scenario suite (… live inference)" job is skipped as a whole without an `ANTHROPIC_API_KEY` repository secret, and none is set, so it shows as *skipped*. Before 2026-09 it instead ran with every real step skipped and reported *success*, so an older green check there means nothing ran. This note, not CI, is the live evidence. Separately and not a live matter: all four committed cassettes were **re-recorded** on 2026-10-03 against `2.19675.0` with auto-memory off, each on its original model: `example-pdf-skill` and `test/fixtures/tool-call-dispatch/dispatch-shell.cassette.json` (`container`, agent 2.1.286), `hostloop-computer-links` (`hostloop`, agent 2.1.286) and `example-multiselect-gate` (`protocol`, which runs the host CLI, here 2.1.287), about $0.75 in all. Their init frames carry no `memory_paths`. The recorder's own scrub (now applied to every recording) removed, across the four, the account's model menu, a subscription account's `rate_limit_info` (from `example-pdf-skill`, `dispatch-shell` and `hostloop-computer-links`) and the agent's sub-agent hand-back frame (from `dispatch-shell`). `verify-cassettes` exits 0 on all four, with one accepted `unscanned` entry (`example-pdf-skill`'s uploaded artifact body, too large to commit). Re-stamp this paragraph, naming the baseline, whenever a live pass is actually re-run.
|
|
212
213
|
|
|
213
214
|
> The staged agent ELF is unchanged (2.1.181) across the 1.14271.0→1.15200.0 asar bump, and 2.1.187 across the 1.15200.0→1.15962.0 bump. The live scenario suite (`protocol` + `container` tiers) was re-run against the 1.15200.0 baseline; the 1.15962.0 bump was verified via asar analysis (content byte-identical: host-loop generator, system prompt, identity string, gates, and egress domains all unchanged) plus a full local test suite pass. The 1.15962.1→1.17377.1 bump moved the staged agent to **2.1.197** and added `api.claude.ai` to the egress allowlist; re-verified via `sync` (no unknown deltas) plus a manual asar spot-check of the reconstructed prompt content (substantively unchanged — see the Parity entry in CHANGELOG.md) and a full live scenario-suite pass (`protocol` + `container` tiers).
|
|
214
215
|
|
package/README.md
CHANGED
|
@@ -36,7 +36,7 @@ npm ci && npm run build
|
|
|
36
36
|
node dist/cli.js replay examples/replays/example-pdf-skill.cassette.json
|
|
37
37
|
```
|
|
38
38
|
|
|
39
|
-
(Installing globally — `npm install -g "cowork-harness@^4.
|
|
39
|
+
(Installing globally — `npm install -g "cowork-harness@^4.13.0"` — gives you the `cowork-harness` CLI for your own
|
|
40
40
|
scenarios and cassettes; the bundled example above also replays from a global install — see the `$(npm root -g)` path below.)
|
|
41
41
|
|
|
42
42
|
Full setup → [Quick start](./docs/cli.md#quick-start).
|
|
@@ -49,8 +49,8 @@ Three ways to use this project. Each row is the whole hook — follow the link f
|
|
|
49
49
|
|
|
50
50
|
| I want to… | Start here | Needs |
|
|
51
51
|
|---|---|---|
|
|
52
|
-
| **Run scenarios myself** from a terminal | **[docs/cli.md](./docs/cli.md)**<br><br>`npm i -g "cowork-harness@^4.
|
|
53
|
-
| **Have Claude Code drive it** for me | **[docs/companion-skill.md](./docs/companion-skill.md)**<br><br>`/plugin marketplace add yaniv-golan/cowork-harness`<br>`/plugin install cowork-harness@cowork-harness` | Claude Code. The skill self-bootstraps the CLI via `npx "cowork-harness@^4.
|
|
52
|
+
| **Run scenarios myself** from a terminal | **[docs/cli.md](./docs/cli.md)**<br><br>`npm i -g "cowork-harness@^4.13.0"`<br>`cowork-harness replay examples/replays/example-pdf-skill.cassette.json` | Node ≥ 22. The replay demo above is token-free and needs nothing else; live tiers above `protocol` need Docker + a staged agent binary |
|
|
53
|
+
| **Have Claude Code drive it** for me | **[docs/companion-skill.md](./docs/companion-skill.md)**<br><br>`/plugin marketplace add yaniv-golan/cowork-harness`<br>`/plugin install cowork-harness@cowork-harness` | Claude Code. The skill self-bootstraps the CLI via `npx "cowork-harness@^4.13.0"` |
|
|
54
54
|
| **Gate my skill in CI** | **[docs/ci.md](./docs/ci.md)**<br><br>`- uses: yaniv-golan/cowork-harness@v4`<br>` with: { command: replay, path: cassettes/ }` | Nothing for the token-free gate; the live lane needs a self-hosted runner with Docker + an agent binary |
|
|
55
55
|
|
|
56
56
|
Improving a skill round by round with Claude Code's `/claude-api hillclimb` loop? The harness is its runner, with
|
|
@@ -457,6 +457,6 @@ inputs/outputs. Human-readable terminal text is explicitly **not** part of the c
|
|
|
457
457
|
## Status
|
|
458
458
|
|
|
459
459
|
The latest shipped baseline — what `baseline: latest` resolves to, the highest version under `baselines/` — is
|
|
460
|
-
**`desktop-
|
|
460
|
+
**`desktop-4.4212.0`**. Release-by-release verification notes (what was re-verified against
|
|
461
461
|
which live agent/asar) are recorded in [CHANGELOG.md](./CHANGELOG.md); the feature catalogue
|
|
462
462
|
this section would otherwise duplicate lives in the sections above.
|