cowork-harness 1.9.0 → 1.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +8 -97
- package/.claude/skills/cowork-harness/references/ci-recipe.md +5 -5
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +4 -2
- package/.claude/skills/cowork-harness/references/task-recipes.md +1 -1
- package/AGENTS.md +2 -1
- package/CHANGELOG.md +44 -0
- package/CONTRIBUTING.md +4 -2
- package/README.md +14 -7
- package/RELEASING.md +8 -8
- package/dist/agent/session.js +35 -0
- package/dist/assert.js +12 -4
- package/dist/hostloop/plugins-handler.js +208 -0
- package/dist/hostloop/skills-handler.js +197 -0
- package/dist/loop-decision.js +55 -0
- package/dist/run/chat.js +35 -9
- package/dist/run/execute.js +12 -2
- package/dist/run/skill-metadata.js +72 -15
- package/dist/run/verdict.js +5 -3
- package/dist/runtime/container.js +29 -3
- package/dist/runtime/hostloop.js +33 -5
- package/dist/session.js +94 -3
- package/dist/sync/cowork-sync.js +5 -4
- package/docs/README.md +13 -0
- package/docs/boundary.md +15 -2
- package/docs/cassette.md +1 -1
- package/docs/debugging.md +3 -1
- package/docs/fidelity-gaps.md +48 -26
- package/docs/scenario.md +8 -6
- package/docs/session.md +8 -0
- package/examples/README.md +2 -1
- package/examples/replays/README.md +1 -1
- package/package.json +1 -1
- package/python/README.md +4 -2
- package/schema/session.schema.json +8 -0
- package/scripts/check-versions.ts +33 -0
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 1.
|
|
7
|
-
tracks-harness: cowork-harness 1.
|
|
6
|
+
version: 1.10.0
|
|
7
|
+
tracks-harness: cowork-harness 1.10.0 (baseline desktop-1.24012.1)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.10.0` (baseline
|
|
26
26
|
> `desktop-1.24012.1`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
27
27
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
@@ -39,100 +39,11 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
41
|
- **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
|
|
42
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
`schema/critique-report.json` (descriptive, test-pinned — parse the report against it, not prose;
|
|
48
|
-
NOT §12-frozen while critique is EXPERIMENTAL), `gradedSkill` in the report (text header + JSON;
|
|
49
|
-
multi-skill pairing key — pair by `(gradedSkillHash, gradedSkill)`, since the hash alone keys the whole
|
|
50
|
-
plugin and cross-pairs its skills). Critique's text report and stderr diagnostics now **collapse host
|
|
51
|
-
paths to `~`** (a shared report or screenshot no longer leaks the username + filesystem layout; the JSON
|
|
52
|
-
report and persisted-artifact paths stay raw for machine consumers), a **missing or non-directory skill
|
|
53
|
-
folder is refused pre-spawn** (fast exit 2, no stray run dir left behind), and the report header + stderr
|
|
54
|
-
prefixes now read **`critique:`** (were `skill-critique:`).
|
|
55
|
-
|
|
56
|
-
- **1.8.0 (critique hardening + hostloop unpin):** `critique --skill <name>` (multi-skill-plugin
|
|
57
|
-
grading — a plugin root with no `--skill` is refused pre-spend; the evidence package gains the
|
|
58
|
-
invoked skill's `agents/<name>.md` + bounded `references/*.md` CONTENT), run-dir artifacts
|
|
59
|
-
(`critique-report.json` always, `critique-evidence-package.txt` when the evaluator ran,
|
|
60
|
-
`critique-salvage.json` on exit 2) + `--out <path>`, per-critique `costUsd` across all four
|
|
61
|
-
workloads, `findingFingerprint` per item (cross-INPUT clustering; HIGH-precision
|
|
62
|
-
LOW-recall — a match proves reproduction, a mismatch does NOT prove non-reproduction, the same
|
|
63
|
-
finding reworded fingerprints differently; skillHash stays the cross-FIX
|
|
64
|
-
key), gate-answer `--answer` echo in the report, per-item-tolerant evaluator parse
|
|
65
|
-
(`droppedEvaluatorItems`), `skillMdTruncated` (a readable-but-oversized SKILL.md is flagged
|
|
66
|
-
"graded a cut copy" — distinct from missing/unreadable), `subagents[].webSearches` +
|
|
67
|
-
`trace --view subagent-research` (live/record lane), and the hostloop uploads-bullet fix (Read of
|
|
68
|
-
an uploaded file at the advertised path works — no copy-into-outputs workaround needed). Also:
|
|
69
|
-
**`critique --fidelity hostloop`** (the container-tier pin is lifted — resume-continuity proven
|
|
70
|
-
live on the native binary; `microvm`/`protocol`/`cowork` stay refused, each with a tagged reason),
|
|
71
|
-
**`skill --allow-host-writes`** (hostloop writable-folder consent; `critique` forwards it to both
|
|
72
|
-
turns), `verdictProvenance` stamped on every critique report (advisory self-run, not an
|
|
73
|
-
attestation), evidence caps 64KB SKILL.md / 32KB transcript / 144KB package (a flagship-sized
|
|
74
|
-
SKILL.md grades untruncated; ~2–2.5× evaluator cost on large skills), pinned sessions stamp their
|
|
75
|
-
fidelity and a cross-tier `--resume` fails loud pre-spawn, and `doctor`'s advisory
|
|
76
|
-
`image-freshness` check (pulled agent image behind the published GHCR one → warn + re-pull
|
|
77
|
-
remedy).
|
|
78
|
-
|
|
79
|
-
- **1.7.0 (per-turn run-directory layout, single shape):** every run dir — `run`/`skill`/`chat`,
|
|
80
|
-
single-turn or multi-turn (any `--session-id` + `--resume`, and **every `critique`**: task turn +
|
|
81
|
-
reflection turn) — writes each turn's `result.json` / `run.jsonl` / `trace.json` / `resources.jsonl`
|
|
82
|
-
into **`turns/<N>/`**, written once and never renamed. **There is no root compat copy of anything —
|
|
83
|
-
`<run-dir>/result.json` does not exist.** **When a user asks about a run, point them at
|
|
84
|
-
`turns/1/result.json`** (the only completion on a single-turn dir) — or on a `critique` dir, better yet
|
|
85
|
-
at `result.graded.json` / `trace.graded.json`, the role-stable aliases `critique` writes (the graded
|
|
86
|
-
task turn, never the reflection one). `events.jsonl` / `timeline.jsonl` stay cumulative at the run-dir
|
|
87
|
-
root, always. A dir written before this layout existed is refused LOUD, by name, naming the shape found
|
|
88
|
-
— never silently misread as if it were `turns/1/`. **Convert it in place with
|
|
89
|
-
`cowork-harness migrate-run-dir` (dry-run by default); do NOT tell the user to re-run or re-record,
|
|
90
|
-
which throws away history the migrator recovers.** Its `events.jsonl` still fully supports `trace`,
|
|
91
|
-
which is why every refusal points there. `verify-run` REFUSES a multi-turn dir rather than certifying
|
|
92
|
-
the wrong turn, and `trace` shows the latest turn with a `::notice::` when earlier turns exist.
|
|
93
|
-
|
|
94
|
-
- **1.7.0 (limitation provenance):** every `critique` limitation in `critique --help` and
|
|
95
|
-
docs/critique.md is tagged with WHY it exists — `[structural]` (permanent), `[unverified]` (unproven,
|
|
96
|
-
**not** known-impossible), `[deliberate]`, `[not-built]`. **Read the tag before telling a user to
|
|
97
|
-
design around a limitation.** Worked example of why the tag matters: `critique`'s container-tier pin
|
|
98
|
-
was tagged `[unverified]`, not permanent — and 1.8.0 lifted it (hostloop allowed, see above) exactly
|
|
99
|
-
as the tag predicted a proof could.
|
|
100
|
-
|
|
101
|
-
- **1.6.0 (`critique`'s skill-flag parity):** `critique` accepts most `skill` flags under the same names,
|
|
102
|
-
which is what makes a document-analysis skill (cap table, deck, financial model, transcript)
|
|
103
|
-
critiquable at all: `--upload`/`--folder`/`--plugin`/`--marketplace`/`--enable`/`--model`/
|
|
104
|
-
`--allow-missing-capability` reach BOTH spawned turns; `--label`/`--timeout`/`--answer`/
|
|
105
|
-
`--answer-policy`/`--on-unanswered`/the decider flags reach the graded turn only; `--prompt-file` is
|
|
106
|
-
new. Anything that can't work is REFUSED with a reason rather than silently ignored.
|
|
107
|
-
**Repeating a flag:** `--upload`/`--folder`/`--plugin`/`--marketplace`/`--enable`/`--answer`
|
|
108
|
-
accumulate; every other value-taking flag is single-valued and repeating it is a usage error (exit `2`)
|
|
109
|
-
rather than a silent last-wins — do not assume the CLI-conventional "last one wins". Booleans repeat
|
|
110
|
-
harmlessly. On `1.5.0` these flags do not exist: `critique --upload x.xlsx` fails as "unknown flag", so
|
|
111
|
-
if a user hits that, they are below the floor.
|
|
112
|
-
|
|
113
|
-
- **1.5.0:** `critique <skill-folder> --prompt "<probe>"` (EXPERIMENTAL) — runs a skill, asks the agent
|
|
114
|
-
what confused it, then grades that self-report against a frozen record of the run: a blinded evaluator
|
|
115
|
-
plus mechanical citation checking. Findings NEVER gate (exit 0); exit 2 means no critique was produced.
|
|
116
|
-
`skill --repeat N` (2-100) with `--min-pass-rate`/`--stop-on-diverge`/`--max-budget-usd`/
|
|
117
|
-
`--allow-budget-stop` brings the variance rollup to the exploratory lane. `result.json` gains
|
|
118
|
-
`outcome` (`errored`/`no_deliverable`/`delivered_with_verdict_fail`/`delivered_clean`). The
|
|
119
|
-
`skill`/`probe-dispatch` lanes now emit `fingerprint.skillHash`/`skillCommit` at all — before 1.5.0
|
|
120
|
-
they emitted NEITHER, so a generation-pairing step over those lanes silently grouped on an absent key.
|
|
121
|
-
`diff`'s exit code now honours its own documented gateable signal (tools/artifacts/meta; a
|
|
122
|
-
transcript-only difference no longer fails it).
|
|
123
|
-
- **core set (pre-0.21.0 vintage, or mixed):** `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator (passes when the resolved value deep-equals one of the listed members — value ∈ your list, not the reverse), `verify-cassettes` incl. the `--allow-domain`/`--allow-email`/`--allow-patterns-file` allows (`--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, and `/help` in the REPL.
|
|
124
|
-
- **0.21.0:** `verify-cassettes --allow-path` (`path` — local absolute filesystem paths — is the scanner's 4th class), and `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field.
|
|
125
|
-
- **0.22.0:** `computer_links_resolve`.
|
|
126
|
-
- **0.28.0:** `semantic_matches` (an LLM judge grades a fixed `rubric` of claims against the run's answer — **live-only**, so it is evidence-unavailable / skipped-loud on replay, never a vacuous pass), and **glob-matched** `tool_called`/`tool_not_called`/`subagent_tool_used`/`subagent_tool_absent` (a pattern like `mcp__workspace__*` matches any tool in the family; exact names match exactly).
|
|
127
|
-
- **0.30.0:** resolved sub-agent identity on dispatch records (`resolvedAgentType`/`resolvedModel`), the five path-gate assertion keys `no_vm_path_file_op`/`vm_path_denied`/`path_denied`/`no_path_denied`/`subagent_file_write`, and the session-level `agent_env` knob.
|
|
128
|
-
- **0.31.0:** the `lint-skill` (static host-loop footgun + `subagent_type` resolution linter) / `analyze-skill` (advisory `/sessions`-path static scan, `--strict`, `analyze-skill: ignore` marker) / `probe-dispatch` (single-dispatch mechanics probe) commands, `status --latest-for` (resolve a scenario's newest run dir by run time, not directory mtime), the `subagent_dispatch_healthy` composite assertion, and the persisted `result.json` fields `verdict`, `subagents[].referencesRead`/`subagents[].reasoning`, plus the `toolCounts`/`toolErrors`/`toolDurations` shape distinction.
|
|
129
|
-
- **0.32.0:** `analyze-skill`'s directory scan now covering a skill/plugin's full contract surface (recursive `agents/`/`references/`/`commands/`, plugin-root-aware, symlink-following) with line/block-scoped `analyze-skill: ignore-next-line`/`ignore-start`/`ignore-end` markers and multi-path/glob input, and `lint-skill`'s provable in-plugin `subagent_type` typo now a WARN that gates under `--strict`.
|
|
130
|
-
- **0.33.0:** the `redacted` marker on display-omitted reasoning — `subagents[].reasoning` and the top-level `thinking[]` now carry `{text:"", redacted:true}` when the model returns a signed-but-empty thought (so "reasoned, text omitted" is distinct from "no thought"), plus the fenced `debug.thinking_display` escape hatch.
|
|
131
|
-
- **1.0.0:** first stable release — the SPEC §12 compatibility contract takes effect (covered CLI/schema/env/Action surfaces are now stable; breaking changes need a major bump). No new author-facing command; the floor simply tracks the 1.0 release.
|
|
132
|
-
- **1.1.0:** `analyze-skill` now also flags **interactive-artifact write-backs lost under Cowork** — a relative `fetch`/XHR/`sendBeacon`/`<form method=post>` in an emitted `.html` (or its `.py`/`.js` generator) that silently fails when the artifact is served from Cowork's own origin. `artifact-write-back-lost` gates under `--strict`; `artifact-write-back-suspect` is advisory; an unanalyzable candidate is a could-not-verify exit 3. An optional **`analyze-skill --runtime`** drives the artifact in a headless DOM (needs `jsdom`) to *observe* the lost write-back — enrichment only, never changes the exit code. Plus a `lint` check for a container-only assertion key (`no_scratchpad_leak`/`present_files_called`) used off the `container` tier, and the `doctor --output-format json` envelope frozen as a covered SPEC §12 surface (`schema/doctor.json`).
|
|
133
|
-
- **1.2.0:** three new assertion keys — `no_lost_write_back: true` (the write-back detector above wired as a per-scenario gate over the run's authored files, live-only) and the regex siblings `tool_result_matches`/`tool_result_not_matches` (case-insensitive per-result, for an error-signature *family* a literal substring can't express). **`microvm` outputs are now observable** — its session tree is snapshotted from the VM into the run dir, so `file_exists`/`artifact_json`/`user_visible_artifact`/`no_unexpected_files`/`input_unmodified`/`no_lost_write_back`/`semantic_matches` all work there, no longer `container`/`hostloop`-only. `status <dir>` also resolves the newest session under a `--run-dir` root; the completion footer prints a `→ result: …/result.json` pointer; the `on_unanswered=fail` error also points to `on_unanswered: llm`. `analyze-skill` hardening: a phantom `<script>` prose block no longer sinks a real verdict, a delete/remove flow claiming success classifies as lost (error) not just suspect, and write-back detection widened (optional-call `?.` spellings, member-spelled/aliased `fetch`/`sendBeacon`, axios instance/config forms).
|
|
134
|
-
- **1.3.0:** run-identity for the iterate-across-fixes loop — `skill`/`run` take `--label <tag>` (a generation tag surfaced in `result.json` `runLabel`, the run-index row, `inspect`, and `status.json`) alongside an auto-recorded `skillCommit`, layered on the **authoritative** content-exact `fingerprint.skillHash` a harvest step should pair critiques by (`inspect`/index surface a short `skillHash` prefix). `trace --full-results` captures the full input+result of **every** tool call — successful ones too, not just errors — so an external grader can ground a self-critique finding against the call it cites. `verify-run` now **warns** on skillHash drift for answer-less scenarios (was silent unless the scenario declared scripted `answers`). Plus `skill --allow-missing-capability` (the open-ended-run opt-out for a capability FALSE-NEGATIVE on the lean `core` image), a new **warn-severity `ended_with_question`** verdict signal (the agent's final answer contains a question and the run wrote no `outputs/` deliverable — the lenient sibling of the strict `stalled`), and LLM-decider `OTHER:` free-text answers now marked `[via Other free-text]` in gate provenance.
|
|
135
|
-
- **1.4.0:** platform baseline synced to Desktop **1.22209.3** (agent 2.1.215; no prompt/spawn/egress drift), and **`coworkWebFetchDedup` is now enacted** on the hostloop `web_fetch` path — a repeat fetch of the same normalized URL within the TTL (15 min, cap 100) makes **no network request** and returns a marker to re-use the earlier result, with no egress event. Baseline-gated (Desktop ≥ 1.22209.3), keyed under the request + terminal `destination_url`, never caching errors/empty; a hit is assertable via `tool_result_contains: "Already fetched"`.
|
|
42
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.10.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.10.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=1.10.0"`. **Pin `@>=1.10.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
|
|
43
|
+
|
|
44
|
+
This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
|
|
45
|
+
OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
|
|
46
|
+
— upgrade rather than work around it, since this skill's `file:line` pointers and flag names track the floor.
|
|
136
47
|
- **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
|
|
137
48
|
- **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
|
|
138
49
|
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred), or `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 1.10.0` (baseline `desktop-1.24012.1`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
|
|
|
13
13
|
```
|
|
14
14
|
|
|
15
15
|
The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
|
|
16
|
-
release; pin an exact version (e.g. `version: "1.
|
|
16
|
+
release; pin an exact version (e.g. `version: "1.10.0"`) for reproducible CI.
|
|
17
17
|
|
|
18
18
|
Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
|
|
19
19
|
cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
|
|
@@ -57,7 +57,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
57
57
|
GitHub-hosted runners, no token/Docker/agent:
|
|
58
58
|
|
|
59
59
|
```yaml
|
|
60
|
-
- run: npm i -g "cowork-harness@>=1.
|
|
60
|
+
- run: npm i -g "cowork-harness@>=1.10.0"
|
|
61
61
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
62
62
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
63
63
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -197,7 +197,7 @@ jobs:
|
|
|
197
197
|
with: { node-version: '20' }
|
|
198
198
|
- uses: actions/setup-python@v5
|
|
199
199
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
200
|
-
- run: npm i -g "cowork-harness@>=1.
|
|
200
|
+
- run: npm i -g "cowork-harness@>=1.10.0"
|
|
201
201
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
202
202
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
203
203
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -226,7 +226,7 @@ jobs:
|
|
|
226
226
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
227
227
|
fi
|
|
228
228
|
- if: steps.guard.outputs.live == 'true'
|
|
229
|
-
run: npm i -g "cowork-harness@>=1.
|
|
229
|
+
run: npm i -g "cowork-harness@>=1.10.0"
|
|
230
230
|
- if: steps.guard.outputs.live == 'true'
|
|
231
231
|
run: cowork-harness run scenarios/ --output-format json
|
|
232
232
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 1.10.0` (baseline `desktop-1.24012.1`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.10.0`
|
|
4
4
|
(baseline `desktop-1.24012.1`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
@@ -136,6 +136,8 @@ plugins:
|
|
|
136
136
|
remote_plugins: []
|
|
137
137
|
skills:
|
|
138
138
|
local: [] # extra host skill dirs
|
|
139
|
+
suggest_enabled: true # gate 245679952 override — `mcp__skills__suggest_skills` on/off (default true)
|
|
140
|
+
proactive_suggest_enabled: false # gate 1598976391 override — proactive description + `trigger` param (default false)
|
|
139
141
|
mcp:
|
|
140
142
|
config: null # --mcp-config file (standard mcpServers map)
|
|
141
143
|
enabled: []
|
|
@@ -274,7 +276,7 @@ same set live from the schema.
|
|
|
274
276
|
| `no_skill_triggered: <regex>` | no invoked skill id matched — the negative-control / description-collision catcher; evidence-unavailable (never a vacuous pass) if invocation data is absent or the `Skill` tool is unobservable |
|
|
275
277
|
| `skill_available: <regex>` | a staged skill's id matched the regex (offered, not necessarily invoked — see `skill_triggered` for invocation) — content-class: the id list comes from the agent's init `skills` listing, so it replays from the frozen init event (id-only; the `whenToUse` enrichment is live-disk and thus absent on replay, but the id is what's matched); evidence-unavailable only if `RunResult.context.availableSkills` is absent entirely (an older cassette recorded before the available-skills listing was captured) |
|
|
276
278
|
| `connector_available: <regex>` | an MCP server/connector's name matched the regex (available, not necessarily used) — evidence-unavailable if `RunResult.context.mcpServers` is absent |
|
|
277
|
-
| `tool_available: <regex>` | a tool in the init manifest matched the regex (available, not necessarily called — see `tool_called` for invocation) — evidence-unavailable if `RunResult.context.tools` is absent |
|
|
279
|
+
| `tool_available: <regex>` | a tool in the init manifest matched the regex (available, not necessarily called — see `tool_called` for invocation) — evidence-unavailable if `RunResult.context.tools` is absent. The `mcp__skills__*`/`mcp__plugins__*` discovery tools are modeled (as `alwaysLoad`) on `container`/`hostloop`/`cowork` — a miss there is a real absence; `microvm`/`protocol` declare no such server, so a miss on those two tiers means "not modeled at this tier", not "provably unavailable" |
|
|
278
280
|
|
|
279
281
|
**Name-matching styles differ by key (don't mix them up):** `tool_called`, `tool_not_called`,
|
|
280
282
|
`subagent_tool_used`, `subagent_tool_absent` are **glob** (anchored, case-sensitive, `*`/`?`). `tool_available`,
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
4
|
decision path. Every one answers a question a real fleet owner had to work out the hard way.
|
|
5
|
-
Tracks `cowork-harness 1.
|
|
5
|
+
Tracks `cowork-harness 1.10.0` (baseline `desktop-1.24012.1`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
|
|
6
6
|
Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
|
|
7
7
|
needed if your CLI meets SKILL.md's version floor.
|
|
8
8
|
|
package/AGENTS.md
CHANGED
|
@@ -21,7 +21,8 @@ protocol layer or run-loop bookkeeping in the CLI.
|
|
|
21
21
|
- Tests are **token-free & spawn-free** wherever possible (`cli-json` uses usage-errors + cassette replay).
|
|
22
22
|
Don't add a test that needs a live model or Docker to the default suite; that's the `pytest -m cowork` /
|
|
23
23
|
`npm run test:live` lane. Python fast lane (from `python/`): `pytest -m 'not cowork'`.
|
|
24
|
-
- CLI binary `cowork-harness`; env vars `COWORK_HARNESS_*` (+ `COWORK_AGENT_BINARY` / `COWORK_AGENT_IMAGE`)
|
|
24
|
+
- CLI binary `cowork-harness`; env vars `COWORK_HARNESS_*` (+ `COWORK_AGENT_BINARY` / `COWORK_AGENT_IMAGE`) —
|
|
25
|
+
see README's [Reproducibility knobs](./README.md#reproducibility-knobs) for the full env-var list.
|
|
25
26
|
Node ≥ 20.
|
|
26
27
|
- `cowork-harness sync` is **local-only** (needs Desktop + `app.asar`; not on CI). The committed
|
|
27
28
|
`baselines/*.json` are CI's source of truth — never hand-edit release facts into source; they come from
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,50 @@ All notable changes to this project are documented here. The format is based on
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [1.10.0] — 2026-07-25
|
|
10
|
+
|
|
11
|
+
### Changed
|
|
12
|
+
|
|
13
|
+
- **The packaged `cowork-harness` skill no longer carries a per-release version history.** SKILL.md's
|
|
14
|
+
"what the floor gates, by release" list had grown to 19 releases / ~15 KB (about a fifth of the file)
|
|
15
|
+
and was self-defeating: every entry described a feature at or below the floor, so anyone meeting the
|
|
16
|
+
floor already had all of it. Because SKILL.md is loaded into an agent's context on every invocation,
|
|
17
|
+
that was pure token cost for content that cannot change behaviour. The version floor (and *why* to pin
|
|
18
|
+
it rather than use `@latest`) stays; the history now lives where it belongs — this changelog — with a
|
|
19
|
+
pointer for anyone diagnosing an older CLI. SKILL.md: 77 KB → 62 KB.
|
|
20
|
+
|
|
21
|
+
### Added
|
|
22
|
+
|
|
23
|
+
- **Skill/plugin discovery SDK-MCP servers** (`mcp__skills__list_skills`/`suggest_skills`,
|
|
24
|
+
`mcp__plugins__list_plugins`/`search_plugins`/`suggest_plugin_install`) — modeled on `container` and
|
|
25
|
+
`hostloop` (and `cowork`, which resolves to one of those) alongside the existing `cowork`/`workspace`
|
|
26
|
+
servers, via a new `combineSdkMcp` composition helper (`src/agent/session.ts`). Every tool is
|
|
27
|
+
`alwaysLoad` so it surfaces in `system/init.tools` from turn one, matching real Cowork; `list_skills`/
|
|
28
|
+
`list_plugins` are populated deterministically from the session's actually-mounted skills/plugins,
|
|
29
|
+
and `suggest_skills`/`search_plugins`/`suggest_plugin_install` return a deterministic empty-catalog
|
|
30
|
+
advisory result (the real add/install catalog is out of band). Two gates control the `skills` server's
|
|
31
|
+
`suggest_skills` tool — `suggestSkillsEnabled` (default on) and `proactiveSkillSuggestEnabled` (default
|
|
32
|
+
off, adds a `trigger` param) — read from the synced baseline via a new bare-boolean `readGateBool`
|
|
33
|
+
reader, with new session-level overrides `skills.suggest_enabled` / `skills.proactive_suggest_enabled`
|
|
34
|
+
(see [docs/session.md](./docs/session.md)). `tool_available: "mcp__skills__.*"` /
|
|
35
|
+
`"mcp__plugins__.*"` is no longer a false negative on `container`/`hostloop`/`cowork` (still absent on
|
|
36
|
+
`microvm`/`protocol` — see [docs/fidelity-gaps.md](./docs/fidelity-gaps.md)). Fidelity scope: the tool
|
|
37
|
+
inventory, inputSchemas, gating, and the `list_skills`/`list_plugins` envelopes are asar/session-log
|
|
38
|
+
derived; the tool **description strings** and the `search_plugins`/`suggest_plugin_install` envelopes are
|
|
39
|
+
a faithful reconstruction rather than byte-captured — see [docs/fidelity-gaps.md](./docs/fidelity-gaps.md).
|
|
40
|
+
Both catalogs report what the sandbox will actually **receive**: a plugin skill directory that staging
|
|
41
|
+
drops (untracked, under the default git-tracked staging boundary) is omitted from `list_skills` *and*
|
|
42
|
+
`list_plugins`, including on `--resume`, so the two never contradict each other within a run.
|
|
43
|
+
|
|
44
|
+
### Fixed
|
|
45
|
+
|
|
46
|
+
- **`chat --fidelity container` declared SDK-MCP tools it never served.** The container branch built its
|
|
47
|
+
SDK-MCP bundle and then discarded it, so `mcp__cowork__present_files` was advertised on `--tools`/
|
|
48
|
+
`--allowedTools` while its server was never announced in `initialize` — calling it failed, and
|
|
49
|
+
`context.mcpServers` omitted `cowork`. (`run --fidelity container` was unaffected; this was the
|
|
50
|
+
interactive `chat` lane only.) The bundle is now forwarded, which also serves the new
|
|
51
|
+
`skills`/`plugins` discovery servers on that lane.
|
|
52
|
+
|
|
9
53
|
## [1.9.0] — 2026-07-24
|
|
10
54
|
|
|
11
55
|
### Added
|
package/CONTRIBUTING.md
CHANGED
|
@@ -2,7 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
Thanks for helping make Cowork skill-testing reproducible.
|
|
4
4
|
|
|
5
|
-
For the full documentation map, see [docs/README.md](./docs/README.md).
|
|
5
|
+
For the full documentation map, see [docs/README.md](./docs/README.md). Read [AGENTS.md](./AGENTS.md) —
|
|
6
|
+
the canonical agent/architecture contract (architecture seams, invariants) — before changing code.
|
|
6
7
|
|
|
7
8
|
## Development setup
|
|
8
9
|
|
|
@@ -29,7 +30,8 @@ CI Stage 1 (the `build` job in `.github/workflows/ci.yml`) does **not** invoke `
|
|
|
29
30
|
gate steps individually — e.g. `check:versions`, `format:check`, `typecheck`, `build`, a CLI smoke
|
|
30
31
|
(`node dist/cli.js list`), three token-free `replay` gate fixtures, `verify-cassettes`, `lint`, and
|
|
31
32
|
source-guard checks — while the unit suite runs separately as the 4-shard `test` job (see `ci.yml` for
|
|
32
|
-
the authoritative list). Only `release.yml` calls `npm run ci`.
|
|
33
|
+
the authoritative list). Only `release.yml` calls `npm run ci`. For a copy-paste GitHub Actions setup,
|
|
34
|
+
see [ci-recipe.md](./.claude/skills/cowork-harness/references/ci-recipe.md).
|
|
33
35
|
|
|
34
36
|
> **Cutting a release?** See [RELEASING.md](./RELEASING.md) for the branch → PR → tag → publish flow.
|
|
35
37
|
|
package/README.md
CHANGED
|
@@ -91,7 +91,7 @@ node dist/cli.js replay examples/replays/example-pdf-skill.cassette.json
|
|
|
91
91
|
|
|
92
92
|
> **Installed globally instead?** Once linked/installed, the same command is `cowork-harness replay
|
|
93
93
|
> <cassette>` — but the relative path above only resolves from a source checkout's `examples/replays/`.
|
|
94
|
-
> From a global install (`npm i -g "cowork-harness@>=1.
|
|
94
|
+
> From a global install (`npm i -g "cowork-harness@>=1.10.0"`), point at the package root instead:
|
|
95
95
|
> `cowork-harness replay "$(npm root -g)/cowork-harness/examples/replays/example-pdf-skill.cassette.json"`
|
|
96
96
|
> (or copy the cassette into your own project and pass that path).
|
|
97
97
|
|
|
@@ -101,7 +101,7 @@ Live `run`/`skill` need the prerequisites in the next section — note the `prot
|
|
|
101
101
|
> - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
|
|
102
102
|
> - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
|
|
103
103
|
> - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
|
|
104
|
-
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=1.
|
|
104
|
+
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@>=1.10.0"`.
|
|
105
105
|
|
|
106
106
|
Two more worked examples worth knowing about: `examples/scenarios/protocol-smoke.yaml` (zero-Docker smoke
|
|
107
107
|
test) and `examples/scenarios/skill-loads.yaml` (container-tier acceptance check) — see
|
|
@@ -126,7 +126,7 @@ claude plugin marketplace add yaniv-golan/cowork-harness
|
|
|
126
126
|
claude plugin install cowork-harness@cowork-harness
|
|
127
127
|
```
|
|
128
128
|
|
|
129
|
-
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=1.
|
|
129
|
+
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@>=1.10.0"` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 20). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
|
|
130
130
|
|
|
131
131
|
It also follows the open [Agent Skills](https://agentskills.io) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) via [`npx skills`](https://github.com/vercel-labs/skills) (Vercel Labs' CLI implementation of that spec):
|
|
132
132
|
|
|
@@ -147,7 +147,7 @@ A global install is enough for CI `lint`, reading the teaching skill, and replay
|
|
|
147
147
|
To `run` the worked examples live or copy them as a starting point, use a source checkout. (The marketplace
|
|
148
148
|
skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
|
|
149
149
|
assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
|
|
150
|
-
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=1.
|
|
150
|
+
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@>=1.10.0"` — see
|
|
151
151
|
[above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
|
|
152
152
|
|
|
153
153
|
### Prerequisites for anything above `protocol` fidelity
|
|
@@ -363,6 +363,10 @@ L2 microvm parity Optional. Agent inside a real Linux microVM (Lima/Apple-VZ
|
|
|
363
363
|
routes via host curl. A PreToolUse path-containment hook is the security boundary
|
|
364
364
|
for real filesystem access at this tier — see docs/boundary.md.
|
|
365
365
|
|
|
366
|
+
Both container and hostloop also declare the skills/plugins skill/plugin-discovery SDK-MCP servers
|
|
367
|
+
(mcp__skills__list_skills/suggest_skills, mcp__plugins__list_plugins/search_plugins/
|
|
368
|
+
suggest_plugin_install) alongside cowork/workspace — see docs/fidelity-gaps.md.
|
|
369
|
+
|
|
366
370
|
cowork Auto-picks hostloop vs container the way Cowork itself does — decoded from
|
|
367
371
|
GrowthBook gate 1143815894 (Cowork's internal feature-flag system) in the
|
|
368
372
|
synced baseline. "Do what real Cowork does."
|
|
@@ -396,7 +400,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
|
|
|
396
400
|
|
|
397
401
|
| Command | What it does | Reach for it when… |
|
|
398
402
|
|---|---|---|
|
|
399
|
-
| `skill <folder> "<prompt>"` | Run a local skill/plugin folder once against the staged agent; `--compact`/`--demo` trim output for shareable screenshots/GIFs; `--ablate-skill` runs the same prompt with the skill removed (a negative control for skill-lift); `--label <tag>` stamps a generation name for the iterate-across-fixes loop (surfaced in `result.json`/index/`inspect`); `--repeat N` (2-100) runs the same skill+prompt N times and aggregates a variance rollup — "did this finding reproduce, or did it pass once?"; `--allow-missing-capability` stops a capability false-negative on the lean `core` image from failing the verdict (open-ended equivalent of asserting `allow_missing_capability: true`) | ad-hoc "is the skill alive / does it do X?" — the fast inner loop |
|
|
403
|
+
| `skill <folder> "<prompt>"` | Run a local skill/plugin folder once against the staged agent; an unanswered gate's `--on-unanswered` defaults adaptively (`prompt` on a TTY, `fail` in CI/non-TTY); `--compact`/`--demo` trim output for shareable screenshots/GIFs; `--ablate-skill` runs the same prompt with the skill removed (a negative control for skill-lift); `--label <tag>` stamps a generation name for the iterate-across-fixes loop (surfaced in `result.json`/index/`inspect`); `--repeat N` (2-100) runs the same skill+prompt N times and aggregates a variance rollup — "did this finding reproduce, or did it pass once?"; `--allow-missing-capability` stops a capability false-negative on the lean `core` image from failing the verdict (open-ended equivalent of asserting `allow_missing_capability: true`); `--allow-host-writes` consents to a writable `hostloop`-fidelity connected folder (same consent as clicking "connect folder" in Desktop; no effect off `hostloop` or without a writable `--folder`) | ad-hoc "is the skill alive / does it do X?" — the fast inner loop |
|
|
400
404
|
| `run <scenario.yaml \| dir/>` | Run authored scenarios with `assert:` + a CI-ready exit code; a decider can answer unscripted gates; `--repeat`/`--matrix` add variance runs / a compatibility matrix (detail below); `--ablate-skill` runs the same prompt with the skill removed (a negative control for skill-lift) | you want a repeatable, **asserted regression test** — to **measure flakiness** instead of trusting one green — or to **test a compatibility matrix** (multiple baselines/models/skill variants) in one run |
|
|
401
405
|
| `chat <folder> [prompt]` | Interactive multi-turn REPL against a skill (TTY); optional seed prompt is sent as the first turn. `--upload <file>` / `--folder <dir>` (repeatable) attach files/project folders; `--verbose` shows thinking blocks + tool inputs; `--fidelity protocol\|container\|hostloop` (no `microvm`/`cowork` in the REPL); `--allow-host-writes` consents to a writable `hostloop`-fidelity connected folder (same consent as clicking "connect folder" in Desktop); `--raw` skips the control protocol for native `docker run -it` (rejects `--upload`/`--folder`/`--plugin`/`--fidelity`) | debugging a multi-turn flow by hand |
|
|
402
406
|
| `record` / `replay` | **Record a live run once → replay it token-free, Docker-free thereafter** (key flags below; `replay --explain` prints the evidence behind every passing assert) | **token-free, Docker-free CI** from a once-recorded run |
|
|
@@ -429,7 +433,7 @@ Skill testing is the headline use, but the tool is a general harness over the Co
|
|
|
429
433
|
|
|
430
434
|
**Flags worth knowing** (the full list is always `<command> --help`):
|
|
431
435
|
- `run`: a decider (`--decider-cmd <helper>`/`--decider-dir <dir>`, or a scenario's `on_unanswered: llm` — a scenario-YAML key, not a CLI flag; `run` rejects `--on-unanswered llm`) can answer unscripted gates; `--repeat N` (2-100) runs each scenario N times and aggregates a variance rollup instead of a single pass/fail (`--min-pass-rate`, `--stop-on-diverge`, `--max-budget-usd` tune the batch verdict/loop); `--matrix <matrix.yaml>` runs ONE scenario across the cross-product of baseline/model/skill_dir axes (worked example: `examples/matrices/csv-metrics-matrix.yaml`; `--max-cells`/`--concurrency` tune the cap/pool — any cell failing, assertion or infra, fails the run); `--compact`/`--demo` trim `run`/`skill` output for shareable screenshots/GIFs; `--label <tag>` (both `run` and `skill`) stamps a generation name for the iterate-across-fixes loop — surfaced in `result.json` (`runLabel`), the run-index row, `inspect`, and `status.json`, alongside the auto-recorded `skillCommit` (git provenance) and the authoritative `fingerprint.skillHash` a harvest step should pair critiques by.
|
|
432
|
-
- `record`/`replay`: `replay --explain` prints the evidence trail behind every passing assert (the flagship false-green tool — text mode; `--output-format json` already carries `assertions[].evidence`); `--decider-llm`/`--decider-dir` answer gates live during recording; `--rerecord-stale
|
|
436
|
+
- `record`/`replay`: `replay --explain` prints the evidence trail behind every passing assert (the flagship false-green tool — text mode; `--output-format json` already carries `assertions[].evidence`); `--decider-llm`/`--decider-dir` answer gates live during recording; `record <dir/>` is itself a first-class batch input (recording a whole directory of scenarios), and `--rerecord-stale` re-records everything stale in one pass — `--concurrency <N>` bounds the parallelism for either; `--assert-from <scenario.yaml>`/`--reassert` re-check the on-disk `assert:` instead of the frozen one; `--strict`/`--fail-on-skill-drift` control staleness handling on replay; `--no-redact` skips record-time redaction; `--allow-failing` relaxes the post-run verdict gate; `--dry-run` resolves without recording; `--force` overrides the refusal to overwrite a default-path cassette that belongs to a *different* scenario (a slug collision) — not a general overwrite-anything flag; `replay --best-effort-future-cassette` lets a cassette from a newer format version replay anyway (warn instead of the default hard error).
|
|
433
437
|
- `verify-cassettes`: privacy scan (email/currency/domain/path/machine-inventory) + staleness — exit 1 when verification RAN and found a real problem (a finding, a genuine drift, or scenario-prompt drift), exit 3 when it could NOT complete (an unverifiable-class staleness finding, a too-new cassette format, or a read error); whole-token allows via `--allow <regex>` (a pattern) / class-scoped `--allow-domain` / `--allow-email` / `--allow-path` / `--allow-machine-inventory` / `--allow-patterns-file <path>` (a **file** of patterns, one regex per line); `--skip-privacy` or `--skip-staleness` runs only part of the gate; a diverged scenario `prompt` vs. the cassette's frozen prompt is also a hard fail (its own `scenarioDrift` bucket), opt out with `--skip-scenario-drift`; `--margins` adds a per-cassette recorded-vs-budget report for count-bound assertions (a single-sample estimate — diagnostic only, never changes the gate verdict).
|
|
434
438
|
- `stats`: reads `<runsRoot>/index.jsonl`, written automatically at every result; `--since`/`--baseline`/`--branch` filter; `--last <n>` windows per-scenario; `--reindex` rebuilds the index from the physical run-dir tree (the migration path for pre-index runs).
|
|
435
439
|
- `diff`: `--changelog` renders known-field prose for a baseline diff; `--view tools|transcript|artifacts|meta` narrows a run/cassette diff to one section; normalization (default on) masks per-run noise (ids/timestamps/session markers/host paths) so two runs of the same scenario diff as identical — `--no-normalize` compares raw values.
|
|
@@ -690,7 +694,7 @@ jobs:
|
|
|
690
694
|
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
691
695
|
```
|
|
692
696
|
|
|
693
|
-
Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` — intentional so recipes track the current release; pin an exact version for reproducible CI. The companion skill's `cowork-harness@>=1.
|
|
697
|
+
Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` — intentional so recipes track the current release; pin an exact version for reproducible CI. The companion skill's `cowork-harness@>=1.10.0` floor guidance applies to ad-hoc CLI installs, not this input), `strict` (applies to `replay` (staleness findings), `lint`/`lint-skill` (WARN/INFO), and `analyze-skill` (any advisory finding); IGNORED — not forwarded — for `verify-cassettes`/`run`, which don't accept the flag), `fail-on-skill-drift` (**`replay`-only** — never forwarded to the analyzers), `extra-args`, `summary` (default `true`), `anthropic-api-key` (live lane only). See [`action.yml`](./action.yml) for the full input reference.
|
|
694
698
|
|
|
695
699
|
The provided [GitHub Actions workflow](.github/workflows/ci.yml) runs a **seven-stage pipeline**. The **build** + **test** stages are the token-free gate you can copy into your skill repo; the `action-self-test`, `python`, `boundary`, `scenarios`, and `parity-drift` stages are this repo's own fidelity self-tests and are not directly portable (they build the harness's Docker image and run harness-specific e2e scenarios — see [`ci-recipe.md`](./.claude/skills/cowork-harness/references/ci-recipe.md) for the skill-repo template):
|
|
696
700
|
|
|
@@ -734,6 +738,8 @@ Most runs need **none** of these — the defaults are correct. They're grouped b
|
|
|
734
738
|
- `COWORK_HARNESS_JUDGE_MODEL` — the default model for `semantic_matches`' LLM judge (overridden by a
|
|
735
739
|
per-assertion `judge_model`; falls back to the pinned default `claude-opus-4-8`). Pin it alongside `judge_model` for a
|
|
736
740
|
reproducible before/after comparison across re-records.
|
|
741
|
+
- `COWORK_HARNESS_EVALUATOR_MODEL` — the default `critique` grading model (overridden by the `--evaluator-model` flag;
|
|
742
|
+
falls back to the pinned default `claude-opus-4-8`).
|
|
737
743
|
- `COWORK_HARNESS_RUNS_DIR` (or the `--run-dir <path>` flag — a **global** flag that must precede the subcommand — `--dotenv` follows the same rule everywhere except `critique`, which also takes it per-command) — override the default run-output root `~/.cowork-harness/runs` (kept out of any working tree so sensitive skill inputs/outputs don't land in a repo). Precedence: `--run-dir` > env > default. The root is flat and machine-global (shared across projects); pinned `--session-id` runs are guarded against cross-project overwrite, and `prune` never prunes them. In CI, set it to a workspace path (e.g. `runs`) so artifact upload can collect the runs. `COWORK_HARNESS_ALLOW_FOREIGN_RESUME=1` overrides the guard that blocks `--resume` onto another project's pinned session.
|
|
738
744
|
- **Networking / loop:** `COWORK_EGRESS_PROXY` overrides the egress-proxy URL injected into the sandbox; `COWORK_PROXY_IMAGE` overrides the egress proxy Docker image name (default `cowork-egress-proxy:2`); `COWORK_DOCKER_NETWORK` pins the Docker network the agent container joins; `CLAUDE_FORCE_HOST_LOOP=1` forces the host-loop path regardless of the baseline's loop decision (the `cowork` tier's auto-pick). `COWORK_LIMACTL` overrides the `limactl` binary path (default `/opt/homebrew/bin/limactl`).
|
|
739
745
|
- `COWORK_HARNESS_PRERUN_HASH_CAP` — override the default cap on pre-run file hashing (bytes); raise it if `input_unmodified`/`no_unexpected_files` report evidence unavailable on a large connected folder.
|
|
@@ -840,6 +846,7 @@ a global install has them locally too, not just on GitHub.
|
|
|
840
846
|
| [examples/README.md](./examples/README.md) | The worked examples to copy — sessions, scenarios, and skills you can run end-to-end (from a source checkout — a global npm install's only runnable `examples/` subtree is `replays/`; see the "What ships" table). |
|
|
841
847
|
| [SECURITY.md](./SECURITY.md) | Threat model — the sandbox is a fidelity fixture, not a security boundary. |
|
|
842
848
|
| [RELEASING.md](./RELEASING.md) | The release flow — branch → PR → tag → npm publish. |
|
|
849
|
+
| [llms.txt](./llms.txt) | The AI-agent index — a machine-readable map of this repo's docs for an agent bootstrapping context. |
|
|
843
850
|
|
|
844
851
|
## Versioning
|
|
845
852
|
|
package/RELEASING.md
CHANGED
|
@@ -50,14 +50,14 @@ Never push the tag before CI is green for the exact commit you intend to tag. Th
|
|
|
50
50
|
enforces this (`Require ci.yml success for this commit` step), but don't rely on it — tag a green
|
|
51
51
|
SHA.
|
|
52
52
|
|
|
53
|
-
**Tag the MERGE COMMIT (main HEAD after the merge), never the release-branch head — and here's why.**
|
|
54
|
-
The publish gate (`require-ci-success`) queries `ci.yml` runs with `--event push` for the tagged SHA.
|
|
55
|
-
`ci.yml` only triggers `on: push` for **`main`** (plus `pull_request` / `workflow_dispatch`), so a
|
|
56
|
-
release-branch/PR head has *only* a `pull_request` run — which the `--event push` filter ignores.
|
|
57
|
-
Tagging that SHA makes the gate poll ~30 min and then FAIL. Only the merge commit (produced by
|
|
58
|
-
`gh pr merge`, then `git pull`ed onto `main`) has a push-event `ci.yml` run. This is why Phase 3 tags
|
|
59
|
-
`main` HEAD after the merge — do **not** "optimize" by tagging the branch commit whose PR CI you just
|
|
60
|
-
watched go green.
|
|
53
|
+
> **Tag the MERGE COMMIT (main HEAD after the merge), never the release-branch head — and here's why.**
|
|
54
|
+
> The publish gate (`require-ci-success`) queries `ci.yml` runs with `--event push` for the tagged SHA.
|
|
55
|
+
> `ci.yml` only triggers `on: push` for **`main`** (plus `pull_request` / `workflow_dispatch`), so a
|
|
56
|
+
> release-branch/PR head has *only* a `pull_request` run — which the `--event push` filter ignores.
|
|
57
|
+
> Tagging that SHA makes the gate poll ~30 min and then FAIL. Only the merge commit (produced by
|
|
58
|
+
> `gh pr merge`, then `git pull`ed onto `main`) has a push-event `ci.yml` run. This is why Phase 3 tags
|
|
59
|
+
> `main` HEAD after the merge — do **not** "optimize" by tagging the branch commit whose PR CI you just
|
|
60
|
+
> watched go green.
|
|
61
61
|
|
|
62
62
|
When you query runs by SHA, use the **full 40-char SHA** (`git rev-parse HEAD`) —
|
|
63
63
|
`gh run list --commit <short-sha>` silently returns empty. If you mis-tag: `git push origin
|
package/dist/agent/session.js
CHANGED
|
@@ -4,6 +4,41 @@ import { createWriteStream } from "node:fs";
|
|
|
4
4
|
import { join } from "node:path";
|
|
5
5
|
import readline from "node:readline";
|
|
6
6
|
import { TimelineWriter } from "./timeline.js";
|
|
7
|
+
import { BoundaryError } from "../errors.js";
|
|
8
|
+
/**
|
|
9
|
+
* Compose multiple SDK-MCP bundles into one, so a tier can pass several in-process servers through the
|
|
10
|
+
* single `sdkMcp` slot `AgentSession.init/start` accepts (e.g. container: `cowork` + `skills` + `plugins`;
|
|
11
|
+
* hostloop: `workspace` + `skills` + `plugins`). `servers` is the plain concatenation (order preserved,
|
|
12
|
+
* first-bundle-first); `handle` routes each `mcp_message` to the ONE bundle that declared the message's
|
|
13
|
+
* `server_name` — the dispatcher already threads `server` through every `handle(server, jr)` call (see
|
|
14
|
+
* `AgentSession.pump`'s mcp_message branch), so routing by that name is additive, not new plumbing. A
|
|
15
|
+
* `server_name` no bundle declared (shouldn't happen — the agent only addresses a server it was told
|
|
16
|
+
* about via `sdkMcpServers` in `initialize`) surfaces the SAME "no handler configured" shape the
|
|
17
|
+
* no-`sdkMcp`-at-all path already uses, rather than throwing. A bundle's own `handle` may still throw or
|
|
18
|
+
* reject — that propagates UNCHANGED to the caller (which already catches it, see the `mcp_message`
|
|
19
|
+
* branch's try/catch), so no sub-handler error is swallowed or reshaped here.
|
|
20
|
+
*/
|
|
21
|
+
export function combineSdkMcp(...bundles) {
|
|
22
|
+
const servers = bundles.flatMap((b) => b.servers);
|
|
23
|
+
const byServer = new Map();
|
|
24
|
+
for (const b of bundles)
|
|
25
|
+
for (const s of b.servers) {
|
|
26
|
+
// Fail loud on a wiring mistake: two bundles claiming the same server name would silently shadow
|
|
27
|
+
// one handler and double the name in `sdkMcpServers`. Names are distinct today; this guards regressions.
|
|
28
|
+
if (byServer.has(s))
|
|
29
|
+
throw new BoundaryError(`combineSdkMcp: duplicate sdkMcp server name "${s}"`);
|
|
30
|
+
byServer.set(s, b.handle);
|
|
31
|
+
}
|
|
32
|
+
return {
|
|
33
|
+
servers,
|
|
34
|
+
handle: (server, jsonrpc) => {
|
|
35
|
+
const handle = byServer.get(server);
|
|
36
|
+
if (!handle)
|
|
37
|
+
return { error: { code: -32601, message: `no sdkMcp handler configured for server "${server}"` } };
|
|
38
|
+
return handle(server, jsonrpc);
|
|
39
|
+
},
|
|
40
|
+
};
|
|
41
|
+
}
|
|
7
42
|
// ---- Protocol ingress validation (fail-closed) ----
|
|
8
43
|
/** Every control_request that the driver answers carries a non-empty string `request_id` — it is the
|
|
9
44
|
* address the control_response is written back to. A missing / non-string / empty id means the agent sent
|
package/dist/assert.js
CHANGED
|
@@ -1011,10 +1011,18 @@ function check(a, ctx) {
|
|
|
1011
1011
|
results.push(fail(`tool_available: bad regex "${a.tool_available}": ${c.error}`));
|
|
1012
1012
|
else if (ctx.availableTools === undefined)
|
|
1013
1013
|
results.push(fail(`evidence unavailable: availableTools absent from result.json — cannot evaluate tool_available`));
|
|
1014
|
-
else
|
|
1015
|
-
results.push(
|
|
1016
|
-
|
|
1017
|
-
|
|
1014
|
+
else if (ctx.availableTools.some((t) => c.re.test(t)))
|
|
1015
|
+
results.push(ok());
|
|
1016
|
+
else {
|
|
1017
|
+
// The general caveat applies to every miss; the discovery-server paragraph is appended ONLY when
|
|
1018
|
+
// the pattern actually concerns those tools, so an ordinary `tool_available: "Bash"` miss isn't
|
|
1019
|
+
// buried under ~450 characters about a surface it never mentioned.
|
|
1020
|
+
const discoveryRelevant = /mcp__(?:skills|plugins)/.test(a.tool_available);
|
|
1021
|
+
results.push(fail(`no tool in the init manifest matched "${a.tool_available}" — note context.tools is the EAGERLY-LOADED tool set; a factory-deferred tool (surfaced via a system-reminder) can be available yet absent here, so a miss is "not eagerly loaded", not "provably unavailable"` +
|
|
1022
|
+
(discoveryRelevant
|
|
1023
|
+
? `. The mcp__skills__*/mcp__plugins__* discovery tools ARE modeled as alwaysLoad on container/hostloop (and cowork, which resolves to one of those) — a miss there is a real absence, but microvm/protocol still declare no such server, so a miss on those tiers means "not modeled at this tier", not "provably unavailable"`
|
|
1024
|
+
: "")));
|
|
1025
|
+
}
|
|
1018
1026
|
}
|
|
1019
1027
|
if (a.skill_tool_used !== undefined) {
|
|
1020
1028
|
const { skill, tool } = a.skill_tool_used;
|