cowork-harness 0.23.0 → 0.25.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +43 -11
- package/.claude/skills/cowork-harness/references/ci-recipe.md +25 -5
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +35 -14
- package/.claude/skills/cowork-harness/references/task-recipes.md +110 -0
- package/.claude/skills/cowork-harness/scripts/assertion-keys.json +2 -0
- package/.claude/skills/cowork-harness/scripts/scenario.py +79 -5
- package/.cowork-redact.json +11 -0
- package/CHANGELOG.md +199 -0
- package/CONTRIBUTING.md +1 -0
- package/README.md +6 -5
- package/SPEC.md +56 -24
- package/baselines/desktop-1.18286.0.json +13 -3
- package/baselines/prompts/desktop-1.18286.0/system-prompt-append.md +1 -2
- package/dist/assert.js +56 -5
- package/dist/cli.js +80 -12
- package/dist/glob.js +43 -0
- package/dist/run/artifacts.js +109 -0
- package/dist/run/cassette.js +300 -44
- package/dist/run/execute.js +39 -86
- package/dist/run/pre-run-manifest.js +61 -0
- package/dist/run/trace-view.js +15 -5
- package/dist/runtime/container.js +3 -0
- package/dist/runtime/hostloop.js +12 -2
- package/dist/runtime/lima.js +3 -3
- package/dist/runtime/microvm.js +5 -0
- package/dist/runtime/protocol.js +3 -0
- package/dist/scan.js +1 -1
- package/dist/session.js +14 -0
- package/dist/sync/cowork-sync.js +6 -1
- package/dist/types.js +12 -2
- package/docker/Dockerfile.agent +8 -4
- package/docs/cassette.md +55 -22
- package/docs/fidelity-gaps.md +62 -10
- package/docs/scenario.md +17 -6
- package/examples/replays/README.md +1 -1
- package/package.json +2 -1
- package/python/test_scenario_lint.py +71 -0
- package/schema/cassette.v7.json +25 -0
- package/schema/run-result.json +32 -0
- package/schema/scenario.schema.json +15 -1
- package/schema/verify-cassettes.json +70 -0
- package/scripts/boot-rootfs-vz.ts +4 -4
- package/scripts/build-rootfs-image.ts +1 -1
- package/scripts/capture-rootfs-manifest.ts +1 -1
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 0.
|
|
7
|
-
tracks-harness: cowork-harness 0.
|
|
6
|
+
version: 0.25.0
|
|
7
|
+
tracks-harness: cowork-harness 0.25.0 (baseline desktop-1.18286.0)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,7 +22,7 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 0.25.0` (baseline
|
|
26
26
|
> `desktop-1.18286.0`). If your checkout is newer, prefer the live `--help`, `SPEC.md`, and
|
|
27
27
|
> `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
@@ -38,7 +38,7 @@ cowork-harness skill ./my-skill "do X" # run the skill once against the sta
|
|
|
38
38
|
Before the first command, confirm the CLI is reachable and **fail loud** (never fake a pass) when a tier's dependencies are missing:
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.
|
|
41
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 0.25.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=0.25.0" <cmd>` (Node ≥ 20), or install once with `npm i -g "cowork-harness@>=0.25.0"`. **Pin `@>=0.25.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published. (≥ 0.25.0 is what gates the commands/assertions this skill teaches: `assertions --list`, `scaffold <run-id>`, `trace --view dispatches`, `artifact_json` incl. the `in:` operator, `verify-cassettes`, batch `record <dir>`/`--rerecord-stale`, `record --concurrency <N>`, record-time redaction, multiSelect/`answer:`, `verify-run` answer-coverage, `record --max-artifact-bytes`, live record-time deciders, `verify-cassettes --allow-domain`/`--allow-email`/`--allow-path`/`--allow-patterns-file` (`path` — local absolute filesystem paths — is the scanner's 4th class, new in 0.21.0; `--allow-patterns-file <path>` is a FILE of patterns, one regex per line — not a path to allow, unlike `--allow <regex>`), scenario `skills:` staleness scoping with `COWORK_HARNESS_AGENT_SCOPE=skill`, `chat --plugin`, `/help` in the REPL, `hostloop`'s native host/VM process split with its `allow_host_writes:` consent field, and `computer_links_resolve` (new in 0.22.0).)
|
|
42
42
|
- **Agent binary (sandboxed live tiers — `container`/`microvm`/`hostloop`/`cowork`).** The staged Claude Code agent is **bind-mounted** from a local Claude Desktop install, or point `COWORK_AGENT_BINARY` at a `claude-code-vm/<ver>/claude` ELF. Nothing is bundled. `protocol` (L0) and `replay` need no staged agent; for the sandboxed tiers, no agent → no run; report that, don't skip silently.
|
|
43
43
|
- **Docker / Lima.** Only `--fidelity protocol` (L0) runs without them. `container` / `microvm` / `hostloop` / `cowork` need Docker (Lima for L2). If they're absent, drop to `--fidelity protocol` and **say so** — a green that never exercised the sandbox is not a sandbox pass.
|
|
44
44
|
- **Auth.** `CLAUDE_CODE_OAUTH_TOKEN` (preferred) or `ANTHROPIC_API_KEY`, via env or `.env`. Minting an OAuth token needs the **`claude` CLI** (`npm i -g @anthropic-ai/claude-code`, then `claude setup-token`).
|
|
@@ -288,10 +288,14 @@ cowork-harness lint scenarios/*.yaml
|
|
|
288
288
|
```
|
|
289
289
|
|
|
290
290
|
`lint` flags: filesystem/egress-only assertions on a `replay` gate (silent no-op), bad regex
|
|
291
|
-
quoting, an egress assert on `protocol` fidelity,
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
291
|
+
quoting, an egress assert on `protocol` fidelity, `transcript_no_host_path` on `hostloop`/`protocol`
|
|
292
|
+
(ERROR — fails by design at those tiers; WARN on `fidelity: cowork`, whose tier resolves per the
|
|
293
|
+
baseline's host-loop gate), non-empty `requires_capabilities` on `protocol` without
|
|
294
|
+
`allow_missing_capability` (ERROR — the capability probe can't run there, so the run hard-fails as
|
|
295
|
+
unverifiable), a `controlOut`-gated key on a non-`controlOut` replay, mixed-class assertion items,
|
|
296
|
+
and hallucinated schema (`assertions:` vs `assert:`, unknown keys). Exit code is non-zero on errors
|
|
297
|
+
(CI-friendly). `scaffold` auto-upgrades the tier if you ask for egress on `protocol`, so it never
|
|
298
|
+
emits a scenario `lint` would reject.
|
|
295
299
|
|
|
296
300
|
CI placement: a **token-free `replay` PR gate** (content/structure only) + a **nightly live `run`**
|
|
297
301
|
(filesystem/egress). Fastest setup: `uses: yaniv-golan/cowork-harness@v1` (a packaged GitHub Action with a
|
|
@@ -362,9 +366,12 @@ are the ones that bite hardest.
|
|
|
362
366
|
sub-agent used other tools. And `subagent_dispatched` matches by dispatch **description** too —
|
|
363
367
|
skills often dispatch with no `subagent_type` (`agentType:"unknown"`), so match the description.
|
|
364
368
|
|
|
365
|
-
6. **`dispatch_count_max`
|
|
366
|
-
|
|
367
|
-
dispatch
|
|
369
|
+
6. **`dispatch_count_max` is an author-chosen budget, not a production cap.** It's a post-hoc count
|
|
370
|
+
assertion: passing means "happened to dispatch ≤N this run," nothing more. Cowork imposes **no**
|
|
371
|
+
in-conversation `Task`-dispatch cap — gate `1648655587`'s `{perTask:1, global:3}` governs the
|
|
372
|
+
separate scheduled/cron-task session scheduler, not the `Task` tool (binary-verified; see SPEC
|
|
373
|
+
§10). So there is no "skip-on-cap" for the harness to reproduce; use this key only to catch a
|
|
374
|
+
fan-out you don't want.
|
|
368
375
|
|
|
369
376
|
7. **`protocol` is rejected (not silently passed) if the scenario asserts egress** — boundary
|
|
370
377
|
assertions need a sandboxed tier (`container`+). Good: this one fails loud by design.
|
|
@@ -451,11 +458,36 @@ are the ones that bite hardest.
|
|
|
451
458
|
and re-assert is undetected — the notice flags this; re-record if the session changed. `verify-run` reads
|
|
452
459
|
on-disk `assert:` against a kept *run dir*; `replay --assert-from` is the equivalent for a *cassette*.
|
|
453
460
|
|
|
461
|
+
18. **`questions_count_max` counts sub-questions, not gates.** One `AskUserQuestion` tool call can
|
|
462
|
+
bundle several sub-questions into a single gate; the assertion counts each sub-question, so a
|
|
463
|
+
3-sub-question bundle counts as 3, not 1. `trace --view questions` shows the same per-gate
|
|
464
|
+
sub-question count and a matching footer total — read that off instead of the tool-call count when
|
|
465
|
+
sizing the budget.
|
|
466
|
+
|
|
467
|
+
19. **`gate_answers_delivered` passes vacuously when no gate fires — use `gate_answer_count_min: 1` to
|
|
468
|
+
also require a gate.** Whether a gate fires is model-dependent, so `gate_answers_delivered: true`
|
|
469
|
+
alone can't catch "the gate never fired at all"; pair it with `gate_answer_count_min` when
|
|
470
|
+
presence matters, not just delivery.
|
|
471
|
+
|
|
472
|
+
20. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
|
|
473
|
+
read-only folder's files as path + hash only (`truncated: true`, no `body`) — it's an input the agent
|
|
474
|
+
read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass against it on replay
|
|
475
|
+
(the hash-only entry still materializes a placeholder); `artifact_json` reports a clear
|
|
476
|
+
evidence-unavailable on every lane (live/verify-run/replay agree — no green-record/red-replay). This is
|
|
477
|
+
also why a `mode: r` input never trips the `binary` privacy finding or needs `--allow` — only a
|
|
478
|
+
*committed* body is scanned. `scaffold` won't emit `file_exists` for one either (it's not in
|
|
479
|
+
`RunResult.artifacts`). A `mode: rw`/`rwd` folder's contents are captured with a full body, same as
|
|
480
|
+
`outputs/`.
|
|
481
|
+
|
|
454
482
|
For the complete gotcha list, the assertion catalog, the YAML schema, the fidelity/answer tables,
|
|
455
483
|
and the CI recipe, read the files in `references/`.
|
|
456
484
|
|
|
457
485
|
## References
|
|
458
486
|
|
|
487
|
+
- `references/task-recipes.md` — end-to-end recipes for the four jobs fleet owners actually hit:
|
|
488
|
+
evolve a cassette's `assert:` (usually no re-record), audit a fleet for tier drift, set up
|
|
489
|
+
redaction before the first hostloop/protocol record, derive budget assertions without a
|
|
490
|
+
two-pass record. Start here when the question is "how do I do X", not "what does flag Y mean".
|
|
459
491
|
- `references/scenario-schema.md` — scenario/session YAML schema, full assertion catalog (with each
|
|
460
492
|
key's replay class), the web_fetch model, and the complete gotcha list.
|
|
461
493
|
- `references/fidelity-and-answers.md` — fidelity tiers, answer paths, the determinism contract.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.25.0` (baseline `desktop-1.18286.0`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -53,7 +53,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
53
53
|
GitHub-hosted runners, no token/Docker/agent:
|
|
54
54
|
|
|
55
55
|
```yaml
|
|
56
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
56
|
+
- run: npm i -g "cowork-harness@>=0.25.0"
|
|
57
57
|
- run: cowork-harness lint scenarios/*.yaml # no silent false-greens
|
|
58
58
|
- run: cowork-harness verify-cassettes cassettes/ # privacy + staleness
|
|
59
59
|
- run: cowork-harness replay cassettes/ # token-free content/structure
|
|
@@ -125,9 +125,17 @@ dollar figures). In a skill repo these cassettes get **committed**. So:
|
|
|
125
125
|
customer's cap table.
|
|
126
126
|
- **Opt-in redaction** rewrites configured PII out of the cassette at record time. Drop a
|
|
127
127
|
`.cowork-redact.json` next to your scenarios (or set `COWORK_HARNESS_REDACT_PATTERNS` /
|
|
128
|
-
`_KEYS`); empty by default.
|
|
128
|
+
`_KEYS`); empty by default. The policy is searched in **cwd → the scenario's dir → the cassette's
|
|
129
|
+
dir** (first file found per dir; env vars merge on top). `cowork-harness init-redact` copies the
|
|
130
|
+
packaged reference template (local-path prefixes + a generic email regex) into the cwd as a starting
|
|
131
|
+
point — review and tailor it. Redaction is **verdict-preserving** — `record` refuses to write if
|
|
129
132
|
redaction would flip an assertion (a manufactured green). `--no-redact` skips it for known-synthetic
|
|
130
133
|
inputs.
|
|
134
|
+
- **Pre-spawn preflight**: `record` warns (`::warning::`, before the paid run starts — once per batch
|
|
135
|
+
for `record <dir>` / `--rerecord-stale`) when a scenario about to record at a host-path-bearing tier
|
|
136
|
+
(`hostloop`, `protocol`) has an **empty** redaction policy — that combination commits real host paths
|
|
137
|
+
the `path` scanner then hard-fails at `verify-cassettes` time. The always-on scanner remains the
|
|
138
|
+
universal net (container-tier recordings can trip it too).
|
|
131
139
|
- **Always-on scan gate** — `verify-cassettes` flags email / currency / bare-domain / local-path /
|
|
132
140
|
machine-inventory matches it finds in the committed cassettes and **exits non-zero**, so "no leak" is
|
|
133
141
|
a gate, not discipline.
|
|
@@ -179,7 +187,7 @@ jobs:
|
|
|
179
187
|
with: { node-version: '20' }
|
|
180
188
|
- uses: actions/setup-python@v5
|
|
181
189
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
182
|
-
- run: npm i -g "cowork-harness@>=0.
|
|
190
|
+
- run: npm i -g "cowork-harness@>=0.25.0"
|
|
183
191
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
184
192
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
185
193
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -208,7 +216,7 @@ jobs:
|
|
|
208
216
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
209
217
|
fi
|
|
210
218
|
- if: steps.guard.outputs.live == 'true'
|
|
211
|
-
run: npm i -g "cowork-harness@>=0.
|
|
219
|
+
run: npm i -g "cowork-harness@>=0.25.0"
|
|
212
220
|
- if: steps.guard.outputs.live == 'true'
|
|
213
221
|
run: cowork-harness run scenarios/ --output-format json
|
|
214
222
|
env:
|
|
@@ -235,6 +243,11 @@ scenario = `result === "success" && assertions.every(pass)`. Exit code is non-ze
|
|
|
235
243
|
fails or a run errors, so a plain `cowork-harness run scenarios/` is already CI-ready without parsing
|
|
236
244
|
JSON.
|
|
237
245
|
|
|
246
|
+
`verify-cassettes` emits its **own** envelope (`{command, ok, coverage, results[]}` with per-file
|
|
247
|
+
`findings`/`staleness`/`notes`/`version`/`error`), published as `schema/verify-cassettes.json` in the
|
|
248
|
+
npm package. Both envelope schemas are covered 1.0 contract surfaces (SPEC §12) — parse the JSON, not
|
|
249
|
+
the human-readable text (which is explicitly NOT stable).
|
|
250
|
+
|
|
238
251
|
A run writes to `~/.cowork-harness/runs/<name>/<sessionId>/` by default — outside any working tree. In CI,
|
|
239
252
|
set `COWORK_HARNESS_RUNS_DIR` (or pass `--run-dir`) to a workspace-relative path (e.g. `runs`) so an
|
|
240
253
|
artifact-upload step can collect them. Each run dir holds `events.jsonl`, `control-out.jsonl`, `run.jsonl`,
|
|
@@ -261,6 +274,13 @@ does **not** imply the recording is still valid. Each replay result carries `sta
|
|
|
261
274
|
| `format` | recorded under an older hash format | re-record once |
|
|
262
275
|
| `unverifiable-baseline` | the latest baseline couldn't be loaded | couldn't verify (env, not skill) |
|
|
263
276
|
| `unverifiable-skill` | skill dirs unresolvable — skill staleness couldn't be checked | couldn't verify the skill |
|
|
277
|
+
| `resolved-tier` | a `fidelity: cowork` cassette's recorded `effectiveFidelity` no longer matches what the baseline resolves to today (the host-loop gate flipped) | **high** (the recording exercises the wrong tier) |
|
|
278
|
+
| `unverifiable-tier` | tier check couldn't run for a `fidelity: cowork` cassette (no recorded `effectiveFidelity`, or its pinned baseline failed to load) | couldn't verify the tier — re-record |
|
|
279
|
+
|
|
280
|
+
(A pre-`effectiveFidelity` cassette with an **explicit** tier is statically knowable — it passes the tier
|
|
281
|
+
check with a non-failing informational note in the `verify-cassettes` envelope's per-file `notes[]`, a
|
|
282
|
+
`·`-prefixed row in text output. On `verify-cassettes` every staleness *finding* above is a hard fail —
|
|
283
|
+
the gate is class-blind; notes never fail it.)
|
|
264
284
|
|
|
265
285
|
To gate in CI, pick the severity you want:
|
|
266
286
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 0.25.0` (baseline `desktop-1.18286.0`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 0.25.0`
|
|
4
4
|
(baseline `desktop-1.18286.0`). If your checkout is newer, prefer the live `docs/scenario.md`,
|
|
5
5
|
`docs/session.md`, and `SPEC.md`.
|
|
6
6
|
|
|
@@ -219,6 +219,7 @@ passes only if every key passes. Keep one concern per item unless you mean conju
|
|
|
219
219
|
| `file_exists: <path>` | the path exists under the run's `work/` (anchored at `mnt/`, e.g. `outputs/x.md`). For a user-facing deliverable prefer `user_visible_artifact` — with a connected folder the file lands in `mnt/<folder>` (= `{{workspaceFolder}}`), not `mnt/outputs`, so `file_exists: outputs/x.md` misses it |
|
|
220
220
|
| `user_visible_artifact: <path>` | exists **and** under a user-visible root (`outputs/` + each connected folder's mount name) — the right primitive for a workspace deliverable when a folder is connected |
|
|
221
221
|
| `no_delete_in_outputs: true` | no delete op touched `mnt/outputs` — **only `true` is valid**; `false` is rejected (omit to allow deletes) |
|
|
222
|
+
| `no_unexpected_files: [<glob>, …]` | every **newly created** file under a user-visible root matches ≥1 glob (workRoot-relative paths; `**` = whole path segment for any depth — use `outputs/handoff/**` for per-run subdirs); `[]` = no new files; **new-files-only** — overwriting a pre-existing file in place is invisible (use content-level producer stamping); live/verify-run without a pre-run manifest ⇒ evidence-unavailable hard-fail (live runs capture the baseline only when this key is asserted; recordings always capture); **microvm cannot capture** (use container/hostloop); replay needs `cassette.preRunPaths` (≥0.24 container/hostloop recordings) — cassettes without it **exclude** the key with a loud warning |
|
|
222
223
|
| `self_heal_ran: <bool>` | a plugin-root self-heal script was (not) invoked |
|
|
223
224
|
| `tool_called: <Tool>` | the agent invoked the tool (actually ran it) |
|
|
224
225
|
| `tool_not_called: <Tool>` | the agent never invoked it |
|
|
@@ -228,7 +229,7 @@ passes only if every key passes. Keep one concern per item unless you mean conju
|
|
|
228
229
|
| `subagent_tool_absent: <Tool>` | no sub-agent used the tool |
|
|
229
230
|
| `subagent_dispatched: <regex>` | a sub-agent whose `agentType` **or dispatch description** matches |
|
|
230
231
|
| `subagent_declared_but_unused: <Tool>` | a sub-agent declared the tool but never used **that** tool (even if it used others) |
|
|
231
|
-
| `dispatch_count_max: <N>` | at most N sub-agents dispatched (records only
|
|
232
|
+
| `dispatch_count_max: <N>` | at most N sub-agents dispatched — an author-chosen budget (Cowork imposes no in-conversation Task-dispatch cap; records only, enforces nothing — see gotcha 12) |
|
|
232
233
|
| `skill_triggered: <regex>` | a skill matching the regex (invoked id, e.g. `"plugin:skill"`) was invoked via the `Skill` tool — evidence-unavailable (not a normal fail) if the agent's init tools have no `Skill` tool |
|
|
233
234
|
| `no_skill_triggered: <regex>` | no invoked skill id matched — the negative-control / description-collision catcher; evidence-unavailable (never a vacuous pass) if invocation data is absent or the `Skill` tool is unobservable |
|
|
234
235
|
| `max_cost_usd: <N>` | the run's SDK-reported cost is ≤ N USD — evidence-unavailable if cost telemetry is absent. **Replay asserts the frozen recording's cost, not fresh spend** — a real regression needs a live `run` |
|
|
@@ -236,9 +237,10 @@ passes only if every key passes. Keep one concern per item unless you mean conju
|
|
|
236
237
|
| `tool_calls_max: <N>` | total top-level tool calls (sum of `toolCounts`) ≤ N — meaningfully replay-checkable (re-drive recomputes `toolCounts` deterministically) |
|
|
237
238
|
| `max_turns: <N>` | the SDK-reported (or fallback-counted) turn count ≤ N — meaningfully replay-checkable (re-drive recounts turns deterministically, same as `tool_calls_max`) |
|
|
238
239
|
| `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
|
|
239
|
-
| `questions_count_max: <N>` |
|
|
240
|
-
| `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail) |
|
|
240
|
+
| `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
|
|
241
|
+
| `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min` to also require a gate |
|
|
241
242
|
| `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests |
|
|
243
|
+
| `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass |
|
|
242
244
|
| `allow_permissive_auto_allow: true` | verdict modifier — suppresses the default-fail when the run recorded a cowork-parity permissive auto-allow; for tests that deliberately assert Cowork's permissive behavior |
|
|
243
245
|
| `allow_missing_capability: true` | verdict modifier — suppresses the default-fail when the (partial "core") agent image omits a capability the skill used but real Cowork ships (OCR/LibreOffice/markitdown/opencv/PDF-tables). Assert only when the skill's fallback is genuinely equivalent; otherwise rebuild full parity (`--build-arg COWORK_FULL_PARITY=1`). Also opts out of the `requires_capabilities` declared-need check. Live tiers only |
|
|
244
246
|
| `allow_l0_plugin_divergence: true` | verdict modifier — opt into L0/protocol plugin divergence: suppresses the default-fail when a plugin behaves differently at `protocol` (L0) fidelity than under a sandboxed tier. Live tiers only |
|
|
@@ -292,8 +294,10 @@ modifiers `allow_permissive_auto_allow` / `allow_missing_capability` / `allow_l0
|
|
|
292
294
|
`allow_stall` are also kept on replay, evaluated as no-op passes.
|
|
293
295
|
|
|
294
296
|
**Gate keys — replay only with a `controlOut` cassette:** `question_asked`, `questions_count_max`,
|
|
295
|
-
`gate_answers_delivered`. With `controlOut` present they evaluate; on an old
|
|
296
|
-
**loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
|
|
297
|
+
`gate_answers_delivered`, `gate_answer_count_min`. With `controlOut` present they evaluate; on an old
|
|
298
|
+
cassette without it, a **loud warning** fires and they are **excluded** (not vacuously passed). Re-record to enable them.
|
|
299
|
+
`questions_count_max` counts sub-questions, not gates/tool-calls — see the catalog row above and
|
|
300
|
+
`trace --view questions`.
|
|
297
301
|
|
|
298
302
|
**Filesystem — replay-checkable WITH an artifact manifest:** `file_exists`, `user_visible_artifact`,
|
|
299
303
|
`artifact_json`, `computer_links_resolve` run on replay when the cassette carries an `artifacts` snapshot
|
|
@@ -302,7 +306,9 @@ small-file JSON `body` inlined; a hash-only entry still satisfies `file_exists`.
|
|
|
302
306
|
resolves a `/sessions/…/mnt/…`-shaped link directly against the manifest, and a host-shaped (hostloop) link
|
|
303
307
|
by first normalizing it to a mount-relative path (recorded connected-folder prefixes + the outputs/uploads
|
|
304
308
|
mounts) — replay has no live filesystem to check a host path against directly (that only happens on a live
|
|
305
|
-
`run`/`verify-run`). Without a manifest (older cassettes) all
|
|
309
|
+
`run`/`verify-run`). Without a manifest (older cassettes) all five are skipped; `no_unexpected_files` also
|
|
310
|
+
needs `preRunPaths` (≥0.24 recordings) — without it the key is excluded with a loud warning (live/verify-run
|
|
311
|
+
hard-fails evidence-unavailable instead). A green replay re-confirms
|
|
306
312
|
*record-time* artifacts, not that the current skill still produces them — `replay --strict` fails when the
|
|
307
313
|
staleness `fingerprint` shows ANY skill/baseline drift, or `replay --fail-on-skill-drift` only on
|
|
308
314
|
skill-source drift; every replay result also reports it class-tagged in `staleness[]` for a JSON gate.
|
|
@@ -371,10 +377,12 @@ at the top of this file.
|
|
|
371
377
|
concern per item; run the linter. (`contentKeys` in `src/run/cassette.ts`.)
|
|
372
378
|
|
|
373
379
|
2. **Gate keys need a `controlOut` cassette.** `question_asked`, `questions_count_max`,
|
|
374
|
-
`gate_answers_delivered` only evaluate on replay with `controlOut`; on an
|
|
375
|
-
and are excluded (not passed). `gate_answers_delivered` **fails on
|
|
376
|
-
(`delivered: null`) — absence of evidence is failure
|
|
377
|
-
|
|
380
|
+
`gate_answers_delivered`, `gate_answer_count_min` only evaluate on replay with `controlOut`; on an
|
|
381
|
+
old cassette they warn and are excluded (not passed). `gate_answers_delivered` **fails on
|
|
382
|
+
unobserved delivery** (`delivered: null`) — absence of evidence is failure — but **passes
|
|
383
|
+
vacuously when zero gates fired**; use `gate_answer_count_min: 1` to also require a gate to have
|
|
384
|
+
fired. A **header-only gate** (empty `question`, only `header`) can never be keyed and is rejected
|
|
385
|
+
loudly — every gate needs a non-empty `question`.
|
|
378
386
|
(`contentKeys` in `src/run/cassette.ts`; `src/assert.ts`.)
|
|
379
387
|
|
|
380
388
|
3. **The LLM-decider's two spellings.** Scripted answers + `on_unanswered: fail` is deterministic;
|
|
@@ -410,9 +418,11 @@ at the top of this file.
|
|
|
410
418
|
11. **`subagent_declared_but_unused` fires on declared-but-didn't-use-THAT-tool**, even if the
|
|
411
419
|
sub-agent used other tools.
|
|
412
420
|
|
|
413
|
-
12. **`dispatch_count_max`
|
|
414
|
-
|
|
415
|
-
|
|
421
|
+
12. **`dispatch_count_max` is an author-chosen budget, not a production cap.** It records the count
|
|
422
|
+
and asserts on it; passing means "dispatched ≤N this run," not "the harness capped it." Cowork
|
|
423
|
+
imposes no in-conversation Task-dispatch cap to reproduce — gate `1648655587`'s
|
|
424
|
+
`{perTask:1, global:3}` is the scheduled/cron-task session limiter, a different mechanism
|
|
425
|
+
(binary-verified; SPEC §10).
|
|
416
426
|
|
|
417
427
|
13. **`protocol` is rejected (not silently passed) if the scenario asserts egress** — boundary
|
|
418
428
|
assertions need `container`+. Fails loud by design.
|
|
@@ -450,3 +460,14 @@ at the top of this file.
|
|
|
450
460
|
Consequence: artifact assertions (`artifact_json`) over fields that were redacted will fail at replay
|
|
451
461
|
— the harness emits `::warning::` at record time when this occurs. (`src/secrets.ts:scrubField`;
|
|
452
462
|
`src/run/cassette.ts`.)
|
|
463
|
+
|
|
464
|
+
21. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
|
|
465
|
+
read-only folder's files as `path` + `bytes` + `sha256` only (`truncated: true`, no `body`) — it's an
|
|
466
|
+
input the agent read, not a deliverable it wrote. `file_exists`/`computer_links_resolve` still pass
|
|
467
|
+
against it on replay (the hash-only entry still materializes a 0-byte placeholder); `artifact_json`
|
|
468
|
+
reports a clear evidence-unavailable on every lane (live/verify-run/replay agree). This is also why a
|
|
469
|
+
`mode: r` input never trips the `binary` privacy finding
|
|
470
|
+
or needs `--allow` in `verify-cassettes` — only a *committed* body is scanned. `scaffold` won't emit
|
|
471
|
+
`file_exists` for one either, since it isn't in `RunResult.artifacts`. A `mode: rw`/`rwd` folder's
|
|
472
|
+
contents are captured with a full body, same as `outputs/`. (`src/run/cassette.ts:buildManifest`'s
|
|
473
|
+
`bodyLessPrefixes`; `src/session.ts:readonlyFolderRootsFromPlan`.)
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# Task recipes — end-to-end paths for the four jobs consumers actually reverse-engineer
|
|
2
|
+
|
|
3
|
+
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
|
+
decision path. Every one answers a question a real fleet owner had to work out the hard way. Facts track the harness version in SKILL.md's
|
|
5
|
+
front-matter; the `resolved-tier`/`unverifiable-tier` staleness classes (Recipe 2) and
|
|
6
|
+
`init-redact` (Recipe 3) are new in the release AFTER 0.23.0.
|
|
7
|
+
|
|
8
|
+
## Recipe 1 — Evolve the `assert:` block of an existing cassette (usually NO re-record)
|
|
9
|
+
|
|
10
|
+
You added or changed assertion keys in a scenario whose cassette is already committed. Do you need
|
|
11
|
+
a paid live re-record? Walk this tree — the answer is usually no:
|
|
12
|
+
|
|
13
|
+
1. **Content / always-replay keys** (`transcript_*`, `tool_called`/`tool_not_called`,
|
|
14
|
+
`tool_result_*`, `subagent_*`, `dispatch_count_max`, `skill_triggered`/`no_skill_triggered`,
|
|
15
|
+
`result`, `max_turns`, `tool_calls_max`, `max_cost_usd`, `max_tokens`) →
|
|
16
|
+
`cowork-harness replay <cassette> --assert-from <scenario.yaml>`. Token-free, no re-record.
|
|
17
|
+
If the recording genuinely lacks the telemetry a key needs (very old cassettes), the key fails
|
|
18
|
+
**loud** as `evidence-unavailable` — that is correct behavior, not a bug; only then re-record.
|
|
19
|
+
2. **Gate keys** (`question_asked`, `questions_count_max`, `gate_answers_delivered`) on a cassette
|
|
20
|
+
**with `controlOut`** (any modern recording) → same token-free `--assert-from` path.
|
|
21
|
+
3. **Gate keys** on a **pre-`controlOut`** cassette → one re-record unlocks gate asserts for that
|
|
22
|
+
cassette permanently.
|
|
23
|
+
4. **Filesystem / egress keys** (`file_exists` without an artifact manifest, `egress_allowed`,
|
|
24
|
+
`egress_denied`, `no_delete_in_outputs`) → live lane **by design**; replay skips them with a
|
|
25
|
+
loud `::warning::`. Keep them in the scenario, run them on the nightly live gate.
|
|
26
|
+
|
|
27
|
+
**The one hard caveat:** `--assert-from` **hard-fails on recording-shaping drift** — if the
|
|
28
|
+
scenario's `prompt:`, `answers:`, baseline, or skill content differ from what shaped the
|
|
29
|
+
recording, re-asserting under them is meaningless and the harness refuses. The trap you'll
|
|
30
|
+
actually hit is `answers:`: sequence your edits — land assert-only changes (token-free), and land
|
|
31
|
+
answer changes separately (those DO force a re-record).
|
|
32
|
+
|
|
33
|
+
## Recipe 2 — Audit a fleet for tier drift
|
|
34
|
+
|
|
35
|
+
A `fidelity: cowork` scenario records at whatever tier the baseline's host-loop gate resolves to
|
|
36
|
+
at record time. When Cowork flips that gate, your committed cassettes silently stop representing
|
|
37
|
+
what production would do. Two layers of defense:
|
|
38
|
+
|
|
39
|
+
- **Automated (preferred):** `cowork-harness verify-cassettes <dir>` raises a
|
|
40
|
+
**`resolved-tier`** staleness finding when a `fidelity: cowork` cassette's recorded
|
|
41
|
+
`effectiveFidelity` no longer matches what the current baseline resolves to, and
|
|
42
|
+
**`unverifiable-tier`** when a `fidelity: cowork` cassette predates `effectiveFidelity`
|
|
43
|
+
entirely (can't verify → re-record). Explicit-tier cassettes (`hostloop`, `container`, …) are
|
|
44
|
+
statically knowable and only produce a non-failing informational note.
|
|
45
|
+
- **Manual (one-liner):** `grep -h '"effectiveFidelity"' cassettes/*.cassette.json | sort | uniq -c`
|
|
46
|
+
shows the tier distribution of the fleet at a glance.
|
|
47
|
+
|
|
48
|
+
### Cassette anatomy (what you're looking at when you open one)
|
|
49
|
+
|
|
50
|
+
Top-level fields of a `*.cassette.json` (schema `schema/cassette.v7.json`):
|
|
51
|
+
|
|
52
|
+
| Field | What it is |
|
|
53
|
+
|---|---|
|
|
54
|
+
| `$schema`, `generator`, `cassetteVersion` | Provenance: schema URL, producing tool, format version (current: 7) |
|
|
55
|
+
| `scenario` | The embedded scenario snapshot at record time |
|
|
56
|
+
| `events` | The recorded agent event stream (the replay source) |
|
|
57
|
+
| `controlOut` | Driver→agent control responses — presence unlocks gate asserts on replay |
|
|
58
|
+
| `effectiveFidelity` | The tier the live record actually resolved to (the drift-audit key above) |
|
|
59
|
+
| `artifacts` | Output-file manifest (paths + hashes + small inlined bodies) — unlocks `file_exists`/`artifact_json` on replay |
|
|
60
|
+
| `fingerprint` | Skill/baseline staleness tripwire |
|
|
61
|
+
| `userVisibleRoots` | The user-visible mount roots captured at record time |
|
|
62
|
+
| `readonlyFolderRoots` | The read-only (`mode: r`) subset of `userVisibleRoots` — captured body-less; lets replay give `artifact_json` the precise remedy for a body-less input |
|
|
63
|
+
| `preRunPaths` | Pre-run file-path baseline for `no_unexpected_files` (workRoot-relative; co-present with `userVisibleRoots`) |
|
|
64
|
+
| `scenarioSource` | Relative path to the authored YAML this was recorded from |
|
|
65
|
+
| `authoring` | Present iff a live decider answered ≥1 gate during recording (`nonDeterministic: true`) |
|
|
66
|
+
|
|
67
|
+
## Recipe 3 — Set up redaction BEFORE your first hostloop/protocol record
|
|
68
|
+
|
|
69
|
+
`hostloop` and `protocol` runs execute on the host, so real host paths (`/Users/you/…`) land in
|
|
70
|
+
the transcript and would be committed inside the cassette. Set the policy up **before** the first
|
|
71
|
+
record — retrofitting means re-recording:
|
|
72
|
+
|
|
73
|
+
1. `cowork-harness init-redact` — copies the reference `.cowork-redact.json` (host-path +
|
|
74
|
+
email patterns) into the current directory. Version-control it.
|
|
75
|
+
2. **Search set:** record looks for `.cowork-redact.json` in the **cwd, the scenario's directory,
|
|
76
|
+
and the cassette's output directory** — every distinct file found is MERGED (plus
|
|
77
|
+
`COWORK_HARNESS_REDACT_PATTERNS`/`_KEYS` from the env). Repo root (= cwd in CI) is the
|
|
78
|
+
conventional home.
|
|
79
|
+
3. **Preflight backstop:** `record` on a host-path-bearing tier with NO effective policy emits a
|
|
80
|
+
loud pre-spawn `::warning::` (once per batch). If you see it, stop and do step 1.
|
|
81
|
+
4. **Scanner relationship:** redaction (record-time, rewrites content) and the
|
|
82
|
+
`verify-cassettes` privacy scanner (CI-time, read-only tripwire over 4 classes: `domain`,
|
|
83
|
+
`email`, `path`, `machine-inventory`) are independent layers. The scanner catches what
|
|
84
|
+
redaction missed.
|
|
85
|
+
5. **Scoping idioms for scanner findings you've reviewed and accepted:** prefer the class-scoped
|
|
86
|
+
flags — `--allow-domain 'api\.example\.com'`, `--allow-email '.*@yourco\.com'`,
|
|
87
|
+
`--allow-path '/opt/ci-runner/.*'` — over bare `--allow <regex>` (a single pattern, which
|
|
88
|
+
applies to every class and can let a domain allow swallow an email leak). For a
|
|
89
|
+
version-controlled allowlist use `--allow-patterns-file <path>`: a FILE of bare all-class
|
|
90
|
+
patterns, one regex per line, `#` comments and blank lines ignored — note the flag name is the
|
|
91
|
+
FILE of patterns, not a path to allow.
|
|
92
|
+
|
|
93
|
+
## Recipe 4 — Budget assertions without a two-pass record
|
|
94
|
+
|
|
95
|
+
`max_turns` / `tool_calls_max` / `questions_count_max` / `dispatch_count_max` need observed
|
|
96
|
+
values, but you don't need a second paid run to get them — derive them from evidence you already
|
|
97
|
+
have:
|
|
98
|
+
|
|
99
|
+
1. **From the run you already did:** `run` always keeps its run dir. Read observed turns from
|
|
100
|
+
`result.json` (`usage.turns`), tool-call and question counts from
|
|
101
|
+
`cowork-harness trace <run-id> --view tools` / `--view questions`, and the dispatch count from
|
|
102
|
+
`--view dispatches`.
|
|
103
|
+
2. **From history:** `cowork-harness stats <scenario> --metric turns` gives p50/p95 turns across
|
|
104
|
+
indexed runs — better than a single observation.
|
|
105
|
+
3. **Set the budget with headroom:** observed × 1.5, rounded up (agentic run lengths vary run to
|
|
106
|
+
run; a budget at the observed value flakes).
|
|
107
|
+
4. **Replay-class note:** `max_turns` / `tool_calls_max` / `dispatch_count_max` re-evaluate
|
|
108
|
+
token-free on replay; `questions_count_max` needs `controlOut` (Recipe 1's tree applies).
|
|
109
|
+
`max_cost_usd` / `max_tokens` on replay assert the FROZEN recording's spend — near-zero signal
|
|
110
|
+
as a regression gate; if you need a cost gate, put it on the live lane via `stats`.
|
|
@@ -11,12 +11,14 @@
|
|
|
11
11
|
"egress_allowed",
|
|
12
12
|
"egress_denied",
|
|
13
13
|
"file_exists",
|
|
14
|
+
"gate_answer_count_min",
|
|
14
15
|
"gate_answers_delivered",
|
|
15
16
|
"max_cost_usd",
|
|
16
17
|
"max_tokens",
|
|
17
18
|
"max_turns",
|
|
18
19
|
"no_delete_in_outputs",
|
|
19
20
|
"no_skill_triggered",
|
|
21
|
+
"no_unexpected_files",
|
|
20
22
|
"question_asked",
|
|
21
23
|
"questions_count_max",
|
|
22
24
|
"replay_protocol_fidelity",
|
|
@@ -16,9 +16,14 @@ Two subcommands:
|
|
|
16
16
|
|
|
17
17
|
lint flags (see references/scenario-schema.md for the why of each):
|
|
18
18
|
E egress assertion on `fidelity: protocol` (the harness rejects this run)
|
|
19
|
+
E `transcript_no_host_path` on hostloop/protocol (fails BY DESIGN at those tiers)
|
|
20
|
+
E `requires_capabilities` on `fidelity: protocol` (probe can't run → hard-fails
|
|
21
|
+
unless allow_missing_capability)
|
|
19
22
|
E on_unanswered: agent / invalid value (schema rejects `agent`)
|
|
20
23
|
E authored `replay_protocol_fidelity` assertion (replay-synthesized only)
|
|
21
24
|
E `assertions:` instead of `assert:` (block ignored → every check no-ops)
|
|
25
|
+
W `transcript_no_host_path` on `fidelity: cowork` (tier resolves per baseline gate —
|
|
26
|
+
incompatible if it lands hostloop)
|
|
22
27
|
W no content assertion → no-op on a replay gate (every assertion is fs/egress)
|
|
23
28
|
W mixed-class assert item → fs/egress half dropped on replay
|
|
24
29
|
W unknown top-level / assertion key (typo or hallucinated schema)
|
|
@@ -67,12 +72,12 @@ CONTENT_KEYS = {
|
|
|
67
72
|
"max_turns",
|
|
68
73
|
}
|
|
69
74
|
# content keys, but only evaluated on replay when the cassette carries controlOut
|
|
70
|
-
GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered"}
|
|
75
|
+
GATE_KEYS = {"question_asked", "questions_count_max", "gate_answers_delivered", "gate_answer_count_min"}
|
|
71
76
|
# manifest-backed: replay-checkable when the cassette carries an `artifacts` manifest (record snapshots one);
|
|
72
77
|
# a manifest-less cassette skips them. Since the 0.3.0 artifact-manifest these are NOT always live-only.
|
|
73
78
|
# computer_links_resolve joins this bucket (not CONTENT_KEYS): resolving a non-empty link set needs
|
|
74
79
|
# either a live filesystem or the cassette's artifacts manifest — see cassette.ts's manifestKeys comment.
|
|
75
|
-
MANIFEST_KEYS = {"file_exists", "user_visible_artifact", "artifact_json", "computer_links_resolve"}
|
|
80
|
+
MANIFEST_KEYS = {"file_exists", "user_visible_artifact", "artifact_json", "computer_links_resolve", "no_unexpected_files"}
|
|
76
81
|
# live-only: ALWAYS skipped on replay, with a loud warning (no filesystem, no network on the token-free lane)
|
|
77
82
|
LIVE_ONLY_KEYS = {
|
|
78
83
|
"egress_denied",
|
|
@@ -139,7 +144,7 @@ _EMBEDDED_TOP_LEVEL_KEYS = {
|
|
|
139
144
|
"answers",
|
|
140
145
|
"expect_denied",
|
|
141
146
|
"assert",
|
|
142
|
-
"skills", #
|
|
147
|
+
"skills", # opt-in skill-staleness hash scope
|
|
143
148
|
"requires_capabilities", # Fix 4b: scenario-level required-capability declaration (pre-flight gate)
|
|
144
149
|
"allow_host_writes", # hostloop native-split: consent for a writable connected folder (pre-run gate)
|
|
145
150
|
}
|
|
@@ -177,6 +182,13 @@ REGEX_KEYS = {
|
|
|
177
182
|
VALID_ON_UNANSWERED = {"fail", "prompt", "first", "llm"}
|
|
178
183
|
VALID_TIERS = ("protocol", "container", "microvm", "hostloop", "cowork")
|
|
179
184
|
|
|
185
|
+
# Gate-id tripwire: the `host-path-assert-cowork` WARN below embeds Cowork's
|
|
186
|
+
# host-loop gate id in offline Python (the linter never reads a baseline). The
|
|
187
|
+
# id is pinned by test/scenario-lint-gate-id.test.ts against the PINNED_GATES
|
|
188
|
+
# entry in src/sync/cowork-sync.ts, so a Desktop gate re-key fails loud there
|
|
189
|
+
# instead of silently rotting this message.
|
|
190
|
+
HOST_LOOP_GATE_ID = "1143815894"
|
|
191
|
+
|
|
180
192
|
|
|
181
193
|
class Finding:
|
|
182
194
|
__slots__ = ("severity", "rule", "message", "fix", "file", "line")
|
|
@@ -300,6 +312,60 @@ def lint_doc(doc, path, raw_lines):
|
|
|
300
312
|
)
|
|
301
313
|
)
|
|
302
314
|
|
|
315
|
+
# E/W: transcript_no_host_path is tier-incompatible with hostloop/protocol — the agent
|
|
316
|
+
# legitimately runs on real host paths there, so the assertion fails BY DESIGN (the runtime only
|
|
317
|
+
# warns at run start, after authoring). Lint is deliberately STRICTER than the runtime: the docs
|
|
318
|
+
# declare the combination incompatible, so authoring it is a bug even if a tool-free run could
|
|
319
|
+
# accidentally pass. `cowork` gets a WARN naming the baseline-gate resolution dependency (the
|
|
320
|
+
# linter stays offline — the message carries the gate fact instead of reading a baseline).
|
|
321
|
+
if "transcript_no_host_path" in assert_keys:
|
|
322
|
+
if fidelity in ("hostloop", "protocol"):
|
|
323
|
+
findings.append(
|
|
324
|
+
Finding(
|
|
325
|
+
"ERROR",
|
|
326
|
+
"host-path-assert-tier",
|
|
327
|
+
f"`transcript_no_host_path` on `fidelity: {fidelity}` — the agent runs on real "
|
|
328
|
+
"host paths at this tier, so this assertion FAILS BY DESIGN (it can never be a "
|
|
329
|
+
"meaningful check here).",
|
|
330
|
+
"Run this assertion at fidelity: container (or microvm), or drop it for this tier.",
|
|
331
|
+
path,
|
|
332
|
+
)
|
|
333
|
+
)
|
|
334
|
+
elif fidelity == "cowork":
|
|
335
|
+
findings.append(
|
|
336
|
+
Finding(
|
|
337
|
+
"WARN",
|
|
338
|
+
"host-path-assert-cowork",
|
|
339
|
+
"`transcript_no_host_path` on `fidelity: cowork` — the tier resolves per the "
|
|
340
|
+
f"baseline's host-loop gate ({HOST_LOOP_GATE_ID}); if it resolves to hostloop "
|
|
341
|
+
"this assertion fails by design (and a later gate flip re-stales the cassette).",
|
|
342
|
+
"Pin fidelity: container if the assertion is load-bearing; keep cowork only if "
|
|
343
|
+
"you accept the gate-resolution dependency.",
|
|
344
|
+
path,
|
|
345
|
+
)
|
|
346
|
+
)
|
|
347
|
+
|
|
348
|
+
# E: requires_capabilities on protocol — the capability probe cannot run at protocol tier
|
|
349
|
+
# (clause b of the requires_capabilities contract), so the run HARD-FAILS unless an assert item
|
|
350
|
+
# opts out via allow_missing_capability: true. Offline-detectable fails-by-design, same class as
|
|
351
|
+
# the tier/assert rules above.
|
|
352
|
+
req_caps = doc.get("requires_capabilities")
|
|
353
|
+
if req_caps and fidelity == "protocol":
|
|
354
|
+
opted_out = any(item.get("allow_missing_capability") is True for item in items)
|
|
355
|
+
if not opted_out:
|
|
356
|
+
findings.append(
|
|
357
|
+
Finding(
|
|
358
|
+
"ERROR",
|
|
359
|
+
"capabilities-on-protocol",
|
|
360
|
+
"non-empty `requires_capabilities` on `fidelity: protocol` — the capability "
|
|
361
|
+
"probe cannot run at protocol tier, so the run hard-fails as unverifiable "
|
|
362
|
+
"(fails by design).",
|
|
363
|
+
"Use a sandboxed tier (container/microvm/hostloop), or add "
|
|
364
|
+
"`allow_missing_capability: true` to an assert item to opt out explicitly.",
|
|
365
|
+
path,
|
|
366
|
+
)
|
|
367
|
+
)
|
|
368
|
+
|
|
303
369
|
# E: retired/invalid on_unanswered
|
|
304
370
|
ou = doc.get("on_unanswered")
|
|
305
371
|
if ou is not None and ou not in VALID_ON_UNANSWERED:
|
|
@@ -543,7 +609,7 @@ def cmd_lint(args):
|
|
|
543
609
|
else:
|
|
544
610
|
expanded.append(arg)
|
|
545
611
|
args.files = expanded
|
|
546
|
-
# Linter self-check
|
|
612
|
+
# Linter self-check: a valid schema key the replay-class sets don't classify can't be linted
|
|
547
613
|
# correctly — surface it as a hard ERROR so it fails the gate (and --strict) until someone classifies it.
|
|
548
614
|
if UNCLASSIFIED_KEYS:
|
|
549
615
|
all_findings.append(
|
|
@@ -650,7 +716,15 @@ def build_scenario(args):
|
|
|
650
716
|
if gates:
|
|
651
717
|
for rx, _ in gates:
|
|
652
718
|
content_lines.append(f" - question_asked: {_sq(rx)} # gate key: replay only with a controlOut cassette")
|
|
653
|
-
|
|
719
|
+
# questions_count_max counts SUB-questions at runtime (assert.ts/trace-view.ts), but this
|
|
720
|
+
# scaffold is STATIC — it only knows the number of --gate rules (per-tool-call), never how many
|
|
721
|
+
# sub-questions each gate bundles. Any number emitted here would be a guess: too low false-reds
|
|
722
|
+
# on the first run, too high is a dead tripwire. A budget must come from observation, not
|
|
723
|
+
# fabrication — so emit it COMMENTED OUT with the calibration path, not a made-up value.
|
|
724
|
+
content_lines.append(
|
|
725
|
+
" # - questions_count_max: <N> # BUDGET — calibrate from a real run: `trace --view "
|
|
726
|
+
"questions` prints the SUB-question total (what this asserts); set N to that + headroom."
|
|
727
|
+
)
|
|
654
728
|
content_lines.append(" - gate_answers_delivered: true # the steered answers actually reached the model")
|
|
655
729
|
|
|
656
730
|
live_lines = []
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
{
|
|
2
|
+
"patterns": [
|
|
3
|
+
{ "regex": "/Users/[^\\s\"'\\\\)\\]`]+?(?=/mnt/)", "label": "local-path", "flags": "g" },
|
|
4
|
+
{ "regex": "/home/[^\\s\"'\\\\)\\]`]+?(?=/mnt/)", "label": "local-path", "flags": "g" },
|
|
5
|
+
{ "regex": "/root/[^\\s\"'\\\\)\\]`]+?(?=/mnt/)", "label": "local-path", "flags": "g" },
|
|
6
|
+
{ "regex": "/Users/[^\\s\"'\\\\)\\]`]+", "label": "local-path", "flags": "g" },
|
|
7
|
+
{ "regex": "/home/[^\\s\"'\\\\)\\]`]+", "label": "local-path", "flags": "g" },
|
|
8
|
+
{ "regex": "/root/[^\\s\"'\\\\)\\]`]+", "label": "local-path", "flags": "g" },
|
|
9
|
+
{ "regex": "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}", "label": "email", "flags": "g" }
|
|
10
|
+
]
|
|
11
|
+
}
|