cowork-harness 1.18.0 → 1.20.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (50) hide show
  1. package/.claude/skills/cowork-harness/SKILL.md +17 -11
  2. package/.claude/skills/cowork-harness/references/ci-recipe.md +6 -6
  3. package/.claude/skills/cowork-harness/references/critique.md +1 -1
  4. package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
  5. package/.claude/skills/cowork-harness/references/scenario-schema.md +5 -5
  6. package/.claude/skills/cowork-harness/references/task-recipes.md +8 -2
  7. package/.claude/skills/cowork-harness/scripts/scenario.py +202 -26
  8. package/CHANGELOG.md +291 -0
  9. package/CONTRIBUTING.md +2 -2
  10. package/DESIGN.md +2 -2
  11. package/README.md +22 -12
  12. package/RELEASING.md +41 -7
  13. package/SPEC.md +26 -5
  14. package/baselines/desktop-1.26832.0.json +556 -0
  15. package/dist/assert.js +37 -0
  16. package/dist/boundary-paths.js +3 -1
  17. package/dist/boundary.js +62 -4
  18. package/dist/cli.js +5 -9
  19. package/dist/egress/proxy.js +2 -1
  20. package/dist/egress/sidecar.js +2 -1
  21. package/dist/run/budget.js +34 -9
  22. package/dist/run/cassette.js +190 -25
  23. package/dist/run/chat-result.js +1 -0
  24. package/dist/run/chat.js +4 -3
  25. package/dist/run/doctor.js +36 -18
  26. package/dist/run/execute.js +108 -13
  27. package/dist/run/mutate.js +36 -0
  28. package/dist/runtime/agent-image.js +59 -0
  29. package/dist/runtime/argv.js +42 -4
  30. package/dist/runtime/container.js +14 -6
  31. package/dist/runtime/hostloop.js +33 -5
  32. package/dist/scan.js +64 -7
  33. package/dist/sync/cowork-sync.js +5 -0
  34. package/docker/Dockerfile.agent +7 -1
  35. package/docker/agent-image.json +11 -0
  36. package/docs/boundary.md +21 -1
  37. package/docs/cassette.md +53 -4
  38. package/docs/debugging.md +1 -1
  39. package/docs/fidelity-gaps.md +24 -0
  40. package/docs/maintenance.md +31 -1
  41. package/docs/scenario.md +11 -8
  42. package/examples/replays/README.md +1 -1
  43. package/examples/replays/example-multiselect-gate.cassette.json +1 -1
  44. package/examples/replays/example-pdf-skill.cassette.json +1 -1
  45. package/examples/replays/hostloop-computer-links.cassette.json +1 -1
  46. package/llms.txt +1 -1
  47. package/package.json +2 -2
  48. package/python/test_scenario_lint.py +285 -6
  49. package/schema/run-result.json +40 -0
  50. package/scripts/release-preflight.ts +64 -2
@@ -3,8 +3,8 @@ name: cowork-harness
3
3
  description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
4
4
  metadata:
5
5
  author: cowork-harness
6
- version: 1.18.0
7
- tracks-harness: cowork-harness 1.18.0 (baseline desktop-1.25927.0)
6
+ version: 1.20.0
7
+ tracks-harness: cowork-harness 1.20.0 (baseline desktop-1.26832.0)
8
8
  ---
9
9
 
10
10
  # cowork-harness
@@ -22,8 +22,8 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
22
22
  allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
23
23
  the highest-value part. Read it.
24
24
 
25
- > **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.18.0` (baseline
26
- > `desktop-1.25927.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
25
+ > **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.20.0` (baseline
26
+ > `desktop-1.26832.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
27
27
  > `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
28
28
 
29
29
  ## Preflight — make sure the harness can actually run
@@ -39,7 +39,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
39
39
 
40
40
  - **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
41
41
  - **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
42
- - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.18.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.18.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@>=1.18.0"`. **Pin `@>=1.18.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
42
+ - **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.20.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.20.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@>=1.20.0"`. **Pin `@>=1.20.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
43
43
 
44
44
  This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
45
45
  OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
@@ -324,7 +324,10 @@ reworded gate or a `choose:` the run never offered fails here in ~1s instead of
324
324
  the discovery/encode/record dance entirely and answer gates **live during the recording** with
325
325
  `record --decider-dir`/`--decider-llm` (the cassette is flagged non-deterministic but replays deterministically).
326
326
  `run` takes no `--dry-run`: to check that a scenario **loads** without spending, use
327
- `cowork-harness record <file.yaml> --dry-run`; `lint` checks the assertion invariants (both above).
327
+ `cowork-harness record <file.yaml> --dry-run` — it runs the real loader AND the same scenario-level
328
+ refusals the real `record` applies (`on_unanswered: prompt`, and an unsatisfiable assert pairing), so it
329
+ cannot green something a paid run would reject. On a directory it reports every offender and the batch
330
+ cost estimate. `lint` checks the assertion invariants (both above).
328
331
 
329
332
  **Author answers WITHOUT re-paying — the cheap loop.** You don't need a fresh paid record to discover a
330
333
  scenario's gates or their labels: `--keep` ONE run, then `cowork-harness trace <run-dir> --view questions`
@@ -547,7 +550,7 @@ Docker, no re-record.
547
550
  | Situation | Symptom | Reach for (in order) |
548
551
  |---|---|---|
549
552
  | **The skill misbehaved** | wrong output, an unexpected gate, a denied tool, an opaque crash | `inspect` — what did it produce? · `trace <run-dir> --view <view>` — what did it actually do (tools, gates, sub-agent tree)? · `verify-run` — re-assert cheaply when only an assertion is wrong · `diff <old-run> <new-run>` — what changed since it worked · `chat` — reproduce it by hand |
550
- | **A green you don't trust** | an assert that may have tested nothing, a stale cassette, an auto-answered or decided gate | `replay --explain` — the evidence trail behind each *passing* assert · `replay --mutate` — perturbs each recorded JSON artifact value and reports which perturbations NOTHING caught (reporting only; never moves the verdict/exit code) · `lint` — assertions on the wrong CI lane / mixed-class keys · `verify-cassettes` — privacy + staleness over committed cassettes · the Gotchas landmine catalog — how a check passes vacuously · `run --repeat N` / `skill --repeat N` — did it pass, or pass once? · `stats` — flaky or expensive over time |
553
+ | **A green you don't trust** | an assert that may have tested nothing, a stale cassette, an auto-answered or decided gate | `replay --explain` — the evidence trail behind each *passing* assert · `replay --mutate` — perturbs a CAPPED SAMPLE of recorded JSON values (10/file, 50 total) and reports which perturbations NOTHING caught; the report names the sample size, so read it as a sample not a total (reporting only; never moves the verdict/exit code) · `lint` — assertions on the wrong CI lane / mixed-class keys · `verify-cassettes` — privacy + staleness over committed cassettes · the Gotchas landmine catalog — how a check passes vacuously · `run --repeat N` / `skill --repeat N` — did it pass, or pass once? · `stats` — flaky or expensive over time |
551
554
 
552
555
  A failed run also records `errorSource` (where the failure originated) and `stderrLogPath` (the captured
553
556
  agent stderr) — read those before re-running; a re-record rarely tells you more than the captured stderr
@@ -836,10 +839,13 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
836
839
  sub-question count and a matching footer total — read that off instead of the tool-call count when
837
840
  sizing the budget.
838
841
 
839
- 19. **`gate_answers_delivered` passes vacuously when no gate fires — use `gate_answer_count_min: 1` to
840
- also require a gate.** Whether a gate fires is model-dependent, so `gate_answers_delivered: true`
841
- alone can't catch "the gate never fired at all"; pair it with `gate_answer_count_min` when
842
- presence matters, not just delivery.
842
+ 19. **`gate_answers_delivered` passes vacuously when no gate fires — pair it, or drop it.** Whether a
843
+ gate fires is model-dependent, so `gate_answers_delivered: true` alone can't catch "the gate never
844
+ fired at all". If the scenario is meant to gate, pair it with `gate_answer_count_min: 1` (a floor of
845
+ `0` witnesses nothing). If the scenario is gate-clean by design, drop the key — it asserts nothing
846
+ there — and declare `questions_count_max: 0`, which fails loudly if a gate ever appears. Asserting
847
+ `questions_count_max: 0` alongside a gate-presence key is unsatisfiable: `run`/`skill`/`record`
848
+ refuse it before spending.
843
849
 
844
850
  20. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
845
851
  read-only folder's files as path + hash only (`truncated: true`, no `body`) — it's an input the agent
@@ -1,6 +1,6 @@
1
1
  # CI recipe — replay vs live lanes
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 1.18.0` (baseline `desktop-1.25927.0`).
3
+ Self-contained reference. Tracks `cowork-harness 1.20.0` (baseline `desktop-1.26832.0`).
4
4
 
5
5
  **Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
6
6
  job-summary reporter (verdict table, staleness findings, cost/turns when available):
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
13
13
  ```
14
14
 
15
15
  The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
16
- release; pin an exact version (e.g. `version: "1.18.0"`) for reproducible CI.
16
+ release; pin an exact version (e.g. `version: "1.20.0"`) for reproducible CI.
17
17
 
18
18
  Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
19
19
  cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
@@ -32,7 +32,7 @@ jobs:
32
32
  - uses: actions/checkout@v4
33
33
  - name: Stage the agent binary (official channel, sha256-verified — see https://github.com/yaniv-golan/cowork-harness/blob/main/docs/maintenance.md)
34
34
  run: |
35
- V=2.1.221 # match your scenario's pinned baseline's agentVersion
35
+ V=2.1.222 # match your scenario's pinned baseline's agentVersion
36
36
  curl -fSL "https://downloads.claude.ai/claude-code-releases/$V/linux-arm64/claude" -o "$RUNNER_TEMP/claude-$V"
37
37
  chmod +x "$RUNNER_TEMP/claude-$V"
38
38
  # verify against the committed baseline's sha256 (baselines/desktop-*.json → agentBinary.sha256)
@@ -58,7 +58,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
58
58
  GitHub-hosted runners, no token/Docker/agent:
59
59
 
60
60
  ```yaml
61
- - run: npm i -g "cowork-harness@>=1.18.0"
61
+ - run: npm i -g "cowork-harness@>=1.20.0"
62
62
  - run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
63
63
  # no silent false-greens. WITHOUT --strict this
64
64
  # step cannot fail on a WARN-class rule (e.g.
@@ -279,7 +279,7 @@ jobs:
279
279
  with: { node-version: '24' }
280
280
  - uses: actions/setup-python@v5
281
281
  with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
282
- - run: npm i -g "cowork-harness@>=1.18.0"
282
+ - run: npm i -g "cowork-harness@>=1.20.0"
283
283
  - run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
284
284
  - run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
285
285
  - run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
@@ -308,7 +308,7 @@ jobs:
308
308
  echo "live=true" >> "$GITHUB_OUTPUT"
309
309
  fi
310
310
  - if: steps.guard.outputs.live == 'true'
311
- run: npm i -g "cowork-harness@>=1.18.0"
311
+ run: npm i -g "cowork-harness@>=1.20.0"
312
312
  - if: steps.guard.outputs.live == 'true'
313
313
  run: cowork-harness run scenarios/ --output-format json
314
314
  env:
@@ -1,6 +1,6 @@
1
1
  # Critique — the facts a plugin install can't otherwise reach
2
2
 
3
- Tracks `cowork-harness 1.18.0` (baseline `desktop-1.25927.0`). This is **not** a trim of the full
3
+ Tracks `cowork-harness 1.20.0` (baseline `desktop-1.26832.0`). This is **not** a trim of the full
4
4
  [`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
5
5
  flags, cost, reproduction discipline, known limitations all live there). This file covers exactly what a
6
6
  plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's
@@ -1,6 +1,6 @@
1
1
  # Fidelity tiers & answer paths
2
2
 
3
- Self-contained reference. Tracks `cowork-harness 1.18.0` (baseline `desktop-1.25927.0`).
3
+ Self-contained reference. Tracks `cowork-harness 1.20.0` (baseline `desktop-1.26832.0`).
4
4
 
5
5
  ## Fidelity tiers (`fidelity:` in the scenario)
6
6
 
@@ -1,7 +1,7 @@
1
1
  # Scenario & session schema, assertion catalog, web_fetch, full gotchas
2
2
 
3
- Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.18.0`
4
- (baseline `desktop-1.25927.0`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
3
+ Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.20.0`
4
+ (baseline `desktop-1.26832.0`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
5
5
  [`docs/session.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md), and `SPEC.md`.
6
6
 
7
7
  **Minimal scenario** — `prompt` is the only required field:
@@ -329,9 +329,9 @@ same set live from the schema.
329
329
  | `present_files_called: true` | at least one file was actually delivered via the `present_files` tool (`RunResult.presentedFiles` is non-empty) — the presence companion to `no_scratchpad_leak` (which passes vacuously when nothing was presented). Pair them to require a delivery **and** require it not to leak. Content-class (re-derives identically on replay). **`fidelity: container` or `hostloop`** — the harness serves `present_files` at both (hostloop via a handler mirroring production's own host-loop branch: validate the path, pass it through, no promotion). `protocol` and `microvm` report cannot-verify. See the `no_scratchpad_leak` row, which stays container-only for a different reason; and see the lane note there — the tool name differs on remote Cowork. **Only `true` is valid** |
330
330
  | `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
331
331
  | `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
332
- | `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min` to also require a gate |
333
- | `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests |
334
- | `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass |
332
+ | `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min: >= 1` to also require a gate, or drop this key and declare `questions_count_max: 0` in a scenario that expects none |
333
+ | `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests. Requires a gate, so **mutually exclusive** with `questions_count_max: 0` (refused by `run`/`skill`/`record`) |
334
+ | `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass. **`: 0` asserts nothing** and does not satisfy that pairing; `>= 1` is **mutually exclusive** with `questions_count_max: 0` (refused by `run`/`skill`/`record`) |
335
335
  | `hook_blocked: <regex>` | a PreToolUse hook blocked a tool whose name matches the regex (`RunResult.hookEvents`) — evidence-unavailable if hook telemetry is absent. Replay: needs a `controlOut` cassette (a custom hook's decision lives only there, not the recorded stream) |
336
336
  | `no_hook_blocked: true` | no tool was hook-blocked during the run (distinguishes a real tool crash from an intentional hook block) — evidence-unavailable if hook telemetry is absent. Replay: needs a `controlOut` cassette. **Only `true` is valid** |
337
337
  | `vm_path_denied: true` | **`fidelity: hostloop` only** — at least one recorded path denial (`RunResult.pathDenials`, any source) targeted a `/sessions` VM path — evidence-unavailable if path-denial telemetry is absent. Replay: needs a `controlOut` cassette. Any other tier FAILS "cannot verify". **Only `true` is valid** |
@@ -2,7 +2,7 @@
2
2
 
3
3
  Each recipe composes facts that live scattered across SKILL.md and the other references into one
4
4
  decision path. Every one answers a question a real fleet owner had to work out the hard way.
5
- Tracks `cowork-harness 1.18.0` (baseline `desktop-1.25927.0`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
5
+ Tracks `cowork-harness 1.20.0` (baseline `desktop-1.26832.0`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
6
6
  Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
7
7
  needed if your CLI meets SKILL.md's version floor.
8
8
 
@@ -115,7 +115,13 @@ have:
115
115
  indexed runs — better than a single observation.
116
116
  3. **Set the budget with headroom:** observed × 1.5, rounded up (agentic run lengths vary run to
117
117
  run; a budget at the observed value flakes).
118
- 4. **Replay-class note:** `max_turns` / `tool_calls_max` / `dispatch_count_max` re-evaluate
118
+ 4. **`questions_count_max: 0` is a declaration, not a budget.** If the observed value is zero, you are
119
+ writing "this scenario expects no gates" — which is supported and useful, but it is then **mutually
120
+ exclusive** with `gate_answer_count_min: >= 1`, `question_asked` and `gate_answers_delivered: false`
121
+ (a delivered gate records at least one question). `run` / `skill` / `record` refuse that pairing
122
+ before spending, and `lint` reports `assert-contradiction`. Drop `gate_answers_delivered`
123
+ alongside it too — it asserts nothing when no gate fires.
124
+ 5. **Replay-class note:** `max_turns` / `tool_calls_max` / `dispatch_count_max` re-evaluate
119
125
  token-free on replay; `questions_count_max` needs `controlOut` (Recipe 1's tree applies).
120
126
  `max_cost_usd` / `max_tokens` on replay assert the FROZEN recording's spend — near-zero signal
121
127
  as a regression gate; if you need a cost gate, put it on the live lane via `stats` (history) or
@@ -26,6 +26,10 @@ lint flags (see references/scenario-schema.md for the why of each):
26
26
  E on_unanswered: agent / invalid value (schema rejects `agent`)
27
27
  E authored `replay_protocol_fidelity` assertion (replay-synthesized only)
28
28
  E `assertions:` instead of `assert:` (block ignored → every check no-ops)
29
+ E a presence assert + its absence sibling (unsatisfiable; run/skill/record refuse it:
30
+ questions_count_max:0 vs gate presence,
31
+ no_hook_blocked vs hook_blocked,
32
+ no_path_denied vs path_denied/vm_path_denied)
29
33
  W `transcript_no_host_path` on `fidelity: cowork` (tier resolves per baseline gate —
30
34
  incompatible if it lands hostloop)
31
35
  W `no_scratchpad_leak` on `fidelity: cowork` (tier resolves per baseline gate)
@@ -338,6 +342,63 @@ def _all_assert_keys(items):
338
342
  return keys
339
343
 
340
344
 
345
+ def _assert_values(items, key):
346
+ """Every value authored for `key` across the assert items (a key may repeat across entries).
347
+
348
+ `_all_assert_keys` sees only NAMES, which is how this linter shipped two value-blind gate checks:
349
+ one firing on an assertion whose premise it inverted, one exempting a floor of zero."""
350
+ return [item[key] for item in items if isinstance(item, dict) and key in item]
351
+
352
+
353
+ def _numeric(v):
354
+ """The number the HARNESS would see for a YAML scalar, or None if it isn't one.
355
+
356
+ NOT `isinstance(v, int)`. This linter parses with PyYAML (YAML 1.1); the harness loads scenarios
357
+ with the npm `yaml` package (YAML 1.2 core). `1.0` reaches Python as a float and `1e0` as a STRING,
358
+ yet npm yaml resolves both to the integer 1 and `z.number().int()` accepts them -- so both are
359
+ loadable scenarios that an int-only test would mis-read.
360
+
361
+ The dialect cuts the other way too, benignly: PyYAML resolves `1_000` -> 1000 and `1:30` -> 90
362
+ (sexagesimal), which npm yaml leaves as strings that `z.number()` REJECTS at load. Those read as a
363
+ number here and are refused by the loader -- silent lint, loud load. Same shape as `no`/`off`
364
+ resolving to False in 1.1 but staying strings in 1.2."""
365
+ if isinstance(v, bool):
366
+ return None # `True >= 1` is true in Python; a schema-invalid `: true` is not a floor
367
+ if isinstance(v, (int, float)):
368
+ return float(v)
369
+ if isinstance(v, str):
370
+ try:
371
+ return float(v)
372
+ except ValueError:
373
+ return None
374
+ return None
375
+
376
+
377
+ def _tool_glob_matches(pattern, name):
378
+ """Port of `globToRegExp` (src/glob.ts) for a tool name.
379
+
380
+ `tool_called` is a GLOB, not a regex (src/types.ts `toolGlob`, src/assert.ts `toolMatches`):
381
+ anchored, CASE-SENSITIVE, only `*` and `?` special, every other character literal. A value carrying
382
+ a regex metacharacter is rejected at scenario load, so reading this field with `re.search` was wrong
383
+ on three axes at once -- it flagged valid globs, and exempted a wrong-case glob that can never match.
384
+
385
+ Segment-aware on purpose: a whole-segment `**` matches ZERO segments, so `**/AskUserQuestion`
386
+ matches a bare `AskUserQuestion`. That is a property of the pattern's segments, not the subject's,
387
+ which is why "a tool name contains no `/`" does not make a flat per-character loop equivalent."""
388
+ segments = pattern.replace("\\", "/").split("/")
389
+ rx = "^"
390
+ for i, seg in enumerate(segments):
391
+ last = i == len(segments) - 1
392
+ if seg == "**":
393
+ rx += "(?:[^/]+(?:/[^/]+)*)?" if last else "(?:[^/]+/)*"
394
+ continue
395
+ for ch in seg:
396
+ rx += "[^/]*" if ch == "*" else "[^/]" if ch == "?" else re.escape(ch)
397
+ if not last:
398
+ rx += "/"
399
+ return re.match(rx + "$", name) is not None
400
+
401
+
341
402
  def _is_positional_choose(choose):
342
403
  """True if a `choose` value selects by POSITION — `first` or a 1-based index (scalar or in a
343
404
  multiSelect list) — as opposed to an exact label. Positional answers are order-dependent (H1)."""
@@ -600,40 +661,150 @@ def lint_doc(doc, path, raw_lines):
600
661
  # 0 gates sit green for weeks against a scenario asserting exactly this key.
601
662
  #
602
663
  # A companion is anything that FAILS (rather than vacuously passes) on an empty gate set:
603
- # · gate_answer_count_min — the explicit floor
604
- # · question_asked — fails "no question matched" against an empty question list
605
- # · tool_called matching AskUserQuestion — fails "tool not called"
664
+ # · gate_answer_count_min WITH A FLOOR >= 1 — the explicit floor. The value matters: `: 0` is legal
665
+ # (the schema is nonnegative) and `delivered >= 0` always holds, so a zero floor witnesses nothing
666
+ # and must not silence this rule. Reading the key NAME alone made the "looks paired" scenario a
667
+ # silent false-green — worse than the loud one this rule was written for.
668
+ # · question_asked — fails "no question matched" against an empty question list, for ANY pattern
669
+ # (`.some` over an empty list is false, so no value gate is needed here).
670
+ # · tool_called whose GLOB matches AskUserQuestion — fails "tool not called".
606
671
  # `questions_count_max` is deliberately NOT a companion: a MAX passes vacuously at zero, so pairing it
607
672
  # leaves the hole wide open. The harness's own `scaffold` emits `question_asked` alongside this key,
608
673
  # which is why the exemption must include it — otherwise this rule reds the tool's own output.
609
- if "gate_answers_delivered" in assert_keys:
610
- has_companion = bool(assert_keys & {"gate_answer_count_min", "question_asked"})
674
+ #
675
+ # VALUE, not just key. `gate_answers_delivered: false` is the INVERSE assertion — it demands at least
676
+ # one gate whose answer was confirmed NOT delivered, so zero gates FAILS it. It is not the vacuous
677
+ # direction, and `gate_answer_count_min` (which counts delivered === true) is not its companion.
678
+ # Firing there reds a correct negative-path scenario under CI's `--strict --min-severity WARN`, with a
679
+ # message whose premise is inverted. Skip on `is False` specifically, so a schema-invalid non-bool
680
+ # still warns rather than opening a new hole. (`no`/`off` also reach here as False under PyYAML's
681
+ # YAML 1.1 but are rejected by the loader as strings — see _numeric for the same dialect gap.)
682
+ delivered_values = _assert_values(items, "gate_answers_delivered")
683
+ if any(v is not False for v in delivered_values):
684
+ has_companion = "question_asked" in assert_keys or any(
685
+ (n := _numeric(v)) is not None and n >= 1 for v in _assert_values(items, "gate_answer_count_min")
686
+ )
611
687
  if not has_companion:
612
688
  for item in items:
613
689
  tc = item.get("tool_called")
614
- if not isinstance(tc, str):
615
- continue
616
- # `tool_called` is a user regex over observed tool names; ask whether THEIR pattern would
617
- # match the gate tool rather than string-comparing. A bad regex is somebody else's finding.
618
- try:
619
- if re.search(tc, "AskUserQuestion", re.IGNORECASE):
620
- has_companion = True
621
- break
622
- except re.error:
623
- continue
690
+ # A GLOB, not a regex (src/types.ts toolGlob) — ask whether THEIR pattern would match the
691
+ # gate tool under the harness's own matching rules. A non-str value is somebody else's
692
+ # finding; a glob cannot fail to compile, so there is nothing to guard against here.
693
+ if isinstance(tc, str) and _tool_glob_matches(tc, "AskUserQuestion"):
694
+ has_companion = True
695
+ break
624
696
  if not has_companion:
625
- findings.append(
626
- Finding(
627
- "WARN",
628
- "vacuous-gate-assert",
629
- "`gate_answers_delivered` has no presence companion — zero gates fired passes it "
630
- "VACUOUSLY, so this assertion stays green if the skill stops asking altogether "
631
- "(exactly the regression it looks like it is guarding).",
632
- 'Pair it with `gate_answer_count_min: 1` (or a `question_asked: "<rx>"`, or '
633
- '`tool_called: "AskUserQuestion"`) so a gate must actually fire.',
634
- path,
697
+ # The scenario may already have DECLARED that it expects no gates. Then "add a companion" is
698
+ # wrong advice — the key is inert, and the fix is to drop it. Same rule id, different message;
699
+ # it must not go silent, because `questions_count_max: 0` is still not a presence companion.
700
+ declares_zero_gates = any(_numeric(v) == 0 for v in _assert_values(items, "questions_count_max"))
701
+ if declares_zero_gates:
702
+ findings.append(
703
+ Finding(
704
+ "WARN",
705
+ "vacuous-gate-assert",
706
+ "`gate_answers_delivered` is inert here — this scenario already declares it expects "
707
+ "no gates (`questions_count_max: 0`), and zero gates fired passes "
708
+ "`gate_answers_delivered` VACUOUSLY, so the key asserts nothing.",
709
+ "Drop `gate_answers_delivered` — `questions_count_max: 0` already states the intent "
710
+ "and fails loudly if a gate ever appears. If the scenario is meant to gate after all, "
711
+ "drop `questions_count_max: 0` and pair the delivery check with "
712
+ "`gate_answer_count_min: 1`.",
713
+ path,
714
+ )
635
715
  )
716
+ else:
717
+ findings.append(
718
+ Finding(
719
+ "WARN",
720
+ "vacuous-gate-assert",
721
+ "`gate_answers_delivered` has no presence companion — zero gates fired passes it "
722
+ "VACUOUSLY, so this assertion stays green if the skill stops asking altogether "
723
+ "(exactly the regression it looks like it is guarding).",
724
+ "If the scenario is meant to gate, pair it with `gate_answer_count_min: 1` (a floor "
725
+ 'of 0 witnesses nothing) — or a `question_asked: "<rx>"`, or `tool_called: '
726
+ '"AskUserQuestion"` — so a gate must actually fire. If the scenario is gate-clean by '
727
+ "design, DROP `gate_answers_delivered` (it asserts nothing there) and declare the "
728
+ "intent with `questions_count_max: 0` (or `tool_not_called: \"AskUserQuestion\"`), "
729
+ "which fails loudly if a gate ever appears.",
730
+ path,
731
+ )
732
+ )
733
+
734
+ # E: a statically unsatisfiable assert pairing — the scenario can never pass, on any lane.
735
+ #
736
+ # Each group pairs ONE assertion demanding a record NOT exist with the assertions demanding the SAME
737
+ # record does exist, on a single evidence channel:
738
+ # · questions_count_max: 0 vs gate presence — a delivered gate records at least one question (the
739
+ # harness pushes one entry per sub-question BEFORE answering, and a zero-question gate throws).
740
+ # · no_hook_blocked vs hook_blocked — one hook-event list.
741
+ # · no_path_denied vs path_denied / vm_path_denied — one path-denial list.
742
+ # Verified against the assertion implementations, not the schema prose: a scope split would make a
743
+ # pair satisfiable, and there is none. Where the evidence is missing BOTH halves fail
744
+ # evidence-unavailable rather than passing, and the denial keys are hostloop-only so a wrong tier
745
+ # fails both too — no combination yields a both-pass.
746
+ #
747
+ # Deliberately NOT extended to `tool_not_called` as a gate witness: it reads the tool log while the
748
+ # gate keys read the control channel, and a fixture-driven `protocol` run can gate on the control
749
+ # channel alone, so that cross-channel contradiction is not provable from the YAML.
750
+ #
751
+ # ERROR because `run`/`skill`/`record` refuse the scenario outright (assertContradiction in
752
+ # src/run/execute.ts) — the same reason `on_unanswered: agent` is an ERROR. Note the pairing is
753
+ # unsatisfiable, not always-red: on a lane where the evidence is absent it is not evaluated at all
754
+ # (verify-run fails evidence-unavailable; a controlOut-less cassette SKIPS these keys on replay).
755
+ # Either way it guards nothing.
756
+ contradiction_groups = [
757
+ (
758
+ "`questions_count_max: 0`",
759
+ any(_numeric(v) == 0 for v in _assert_values(items, "questions_count_max")),
760
+ [
761
+ (
762
+ "gate_answer_count_min: >= 1",
763
+ any((n := _numeric(v)) is not None and n >= 1 for v in _assert_values(items, "gate_answer_count_min")),
764
+ ),
765
+ ("question_asked", "question_asked" in assert_keys),
766
+ ("gate_answers_delivered: false", any(v is False for v in _assert_values(items, "gate_answers_delivered"))),
767
+ ],
768
+ "a delivered gate records at least one question, so requiring a gate to be present contradicts requiring zero questions",
769
+ ),
770
+ (
771
+ "`no_hook_blocked: true`",
772
+ any(v is True for v in _assert_values(items, "no_hook_blocked")),
773
+ [("hook_blocked", "hook_blocked" in assert_keys)],
774
+ "both read the same hook-event list — the block `hook_blocked` requires is the one `no_hook_blocked` requires not to exist",
775
+ ),
776
+ (
777
+ "`no_path_denied: true`",
778
+ any(v is True for v in _assert_values(items, "no_path_denied")),
779
+ [
780
+ ("path_denied", "path_denied" in assert_keys),
781
+ ("vm_path_denied: true", any(v is True for v in _assert_values(items, "vm_path_denied"))),
782
+ ],
783
+ "both read the same path-denial list — the denial the positive key requires is one `no_path_denied` requires not to exist",
784
+ ),
785
+ ]
786
+ clauses = []
787
+ for absence_label, absent, presences, why in contradiction_groups:
788
+ if not absent:
789
+ continue
790
+ hits = [label for label, present in presences if present]
791
+ # Report EVERY contradictory group, not just the first — a scenario can carry more than one.
792
+ if hits:
793
+ clauses.append(f"{absence_label} alongside {' and '.join(hits)} ({why})")
794
+ if clauses:
795
+ findings.append(
796
+ Finding(
797
+ "ERROR",
798
+ "assert-contradiction",
799
+ # "both" is wrong once more than one group is named -- and a scenario carrying two is
800
+ # exactly the one whose message gets read carefully.
801
+ f"{'; and '.join(clauses)} — "
802
+ + ("no run can satisfy both." if len(clauses) == 1 else "no run can satisfy all of them."),
803
+ "Keep the negative assertion and drop the positive one, or drop the negative if the "
804
+ "scenario really does expect the record.",
805
+ path,
636
806
  )
807
+ )
637
808
 
638
809
  # W: mixed-class assert item → the live-only half is dropped on replay (manifest-backed keys are NOT)
639
810
  for idx, item in enumerate(items):
@@ -1615,8 +1786,13 @@ def build_scenario(args):
1615
1786
  # sub-questions each gate bundles. Any number emitted here would be a guess: too low false-reds
1616
1787
  # on the first run, too high is a dead tripwire. A budget must come from observation, not
1617
1788
  # fabrication — so emit it COMMENTED OUT with the calibration path, not a made-up value.
1789
+ # `<N> >= 1` is not decoration: this block also emits gate-PRESENCE assertions, and
1790
+ # `questions_count_max: 0` alongside one of those is unsatisfiable — refused by run/skill/record
1791
+ # and flagged as `assert-contradiction`. Zero is the natural value to reach for on a
1792
+ # gate-clean scenario, so the hint has to say where it does NOT belong.
1618
1793
  content_lines.append(
1619
- " # - questions_count_max: <N> # BUDGET — calibrate from a real run: `trace --view "
1794
+ " # - questions_count_max: <N> # BUDGET (N >= 1 — `0` contradicts the gate assertions "
1795
+ "below; use it only in a scenario with none) — calibrate from a real run: `trace --view "
1620
1796
  "questions` prints the SUB-question total (what this asserts); set N to that + headroom."
1621
1797
  )
1622
1798
  content_lines.append(" - gate_answers_delivered: true # the steered answers actually reached the model")