cowork-harness 1.18.0 → 1.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +17 -11
- package/.claude/skills/cowork-harness/references/ci-recipe.md +6 -6
- package/.claude/skills/cowork-harness/references/critique.md +1 -1
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +5 -5
- package/.claude/skills/cowork-harness/references/task-recipes.md +8 -2
- package/.claude/skills/cowork-harness/scripts/scenario.py +202 -26
- package/CHANGELOG.md +291 -0
- package/CONTRIBUTING.md +2 -2
- package/DESIGN.md +2 -2
- package/README.md +22 -12
- package/RELEASING.md +41 -7
- package/SPEC.md +26 -5
- package/baselines/desktop-1.26832.0.json +556 -0
- package/dist/assert.js +37 -0
- package/dist/boundary-paths.js +3 -1
- package/dist/boundary.js +62 -4
- package/dist/cli.js +5 -9
- package/dist/egress/proxy.js +2 -1
- package/dist/egress/sidecar.js +2 -1
- package/dist/run/budget.js +34 -9
- package/dist/run/cassette.js +190 -25
- package/dist/run/chat-result.js +1 -0
- package/dist/run/chat.js +4 -3
- package/dist/run/doctor.js +36 -18
- package/dist/run/execute.js +108 -13
- package/dist/run/mutate.js +36 -0
- package/dist/runtime/agent-image.js +59 -0
- package/dist/runtime/argv.js +42 -4
- package/dist/runtime/container.js +14 -6
- package/dist/runtime/hostloop.js +33 -5
- package/dist/scan.js +64 -7
- package/dist/sync/cowork-sync.js +5 -0
- package/docker/Dockerfile.agent +7 -1
- package/docker/agent-image.json +11 -0
- package/docs/boundary.md +21 -1
- package/docs/cassette.md +53 -4
- package/docs/debugging.md +1 -1
- package/docs/fidelity-gaps.md +24 -0
- package/docs/maintenance.md +31 -1
- package/docs/scenario.md +11 -8
- package/examples/replays/README.md +1 -1
- package/examples/replays/example-multiselect-gate.cassette.json +1 -1
- package/examples/replays/example-pdf-skill.cassette.json +1 -1
- package/examples/replays/hostloop-computer-links.cassette.json +1 -1
- package/llms.txt +1 -1
- package/package.json +2 -2
- package/python/test_scenario_lint.py +285 -6
- package/schema/run-result.json +40 -0
- package/scripts/release-preflight.ts +64 -2
|
@@ -3,8 +3,8 @@ name: cowork-harness
|
|
|
3
3
|
description: Test or debug a Claude Code skill/plugin under Claude Cowork's runtime — sandboxed agent, default-deny egress, the can_use_tool permission/question protocol — using the cowork-harness CLI. Use when validating or regression-testing a skill, authoring or debugging a scenario YAML (prompt + scripted answers + assert:), choosing a fidelity tier, scripting AskUserQuestion / tool-permission answers, or asserting artifacts, egress, or sub-agent dispatch. Especially when a harness run no-ops an assertion, fails on an unanswered gate, false-greens, a steered answer never reaches the model, or a web_fetch is unexpectedly denied or gated. Also when iterating or hardening a skill across fixes, or grounding a skill's self-critique against its own run evidence — including a document-analysis skill (cap table, deck, financial model, transcript) that needs an uploaded file attached to be critiqued at all. NOT for generic unit testing (pytest/vitest of your own scripts) or non-Cowork CI. Covers the skill / run / chat / record / replay / trace / decide / assertions / scaffold commands and the session-vs-scenario split.
|
|
4
4
|
metadata:
|
|
5
5
|
author: cowork-harness
|
|
6
|
-
version: 1.
|
|
7
|
-
tracks-harness: cowork-harness 1.
|
|
6
|
+
version: 1.20.0
|
|
7
|
+
tracks-harness: cowork-harness 1.20.0 (baseline desktop-1.26832.0)
|
|
8
8
|
---
|
|
9
9
|
|
|
10
10
|
# cowork-harness
|
|
@@ -22,8 +22,8 @@ flagged with a loud `::warning::`, not silent — auto-answer a gate, observe an
|
|
|
22
22
|
allowlist). This skill exists mostly to keep you out of those traps — the Gotchas section below is
|
|
23
23
|
the highest-value part. Read it.
|
|
24
24
|
|
|
25
|
-
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.
|
|
26
|
-
> `desktop-1.
|
|
25
|
+
> **Version note:** the facts and `file:line` pointers here track `cowork-harness 1.20.0` (baseline
|
|
26
|
+
> `desktop-1.26832.0`). If your checkout is newer, prefer the live `--help` and — in a repo checkout —
|
|
27
27
|
> `SPEC.md` / `docs/*.md` over this snapshot, and re-run the bundled linter.
|
|
28
28
|
|
|
29
29
|
## Preflight — make sure the harness can actually run
|
|
@@ -39,7 +39,7 @@ Before the first command, confirm the CLI is reachable and **fail loud** (never
|
|
|
39
39
|
|
|
40
40
|
- **One-shot check.** Run `cowork-harness doctor [--tier <tier>]` first — a read-only prerequisite check that inspects Docker, the staged agent, the token, and the baseline in one pass. The bullets below explain each thing it checks (and how to fix it).
|
|
41
41
|
- **Replay-only? Skip `doctor`.** Replaying committed cassettes needs no Docker, no staged agent, and no token — and every tier's `doctor` validates the auth token (the live tiers also Docker + the staged agent), so a ✗ there is expected, not a blocker. Go straight to `cowork-harness replay <cassette>`.
|
|
42
|
-
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.
|
|
42
|
+
- **CLI on PATH, recent enough?** Run `cowork-harness --version` — this skill needs **≥ 1.20.0**. If it's missing or older, prefix every command with the version floor `npx "cowork-harness@>=1.20.0" <cmd>` (Node ≥ 22), or install once with `npm i -g "cowork-harness@>=1.20.0"`. **Pin `@>=1.20.0`, never `@latest`** — `@latest` can silently fetch an older CLI and the new commands fail as "unknown command", whereas the floor **fails loud** if no compatible version is published.
|
|
43
43
|
|
|
44
44
|
This skill documents the CURRENT surface, not release history. If `cowork-harness --version` is
|
|
45
45
|
OLDER than the floor, the per-release record of what you are missing is [CHANGELOG.md](https://github.com/yaniv-golan/cowork-harness/blob/main/CHANGELOG.md)
|
|
@@ -324,7 +324,10 @@ reworded gate or a `choose:` the run never offered fails here in ~1s instead of
|
|
|
324
324
|
the discovery/encode/record dance entirely and answer gates **live during the recording** with
|
|
325
325
|
`record --decider-dir`/`--decider-llm` (the cassette is flagged non-deterministic but replays deterministically).
|
|
326
326
|
`run` takes no `--dry-run`: to check that a scenario **loads** without spending, use
|
|
327
|
-
`cowork-harness record <file.yaml> --dry-run
|
|
327
|
+
`cowork-harness record <file.yaml> --dry-run` — it runs the real loader AND the same scenario-level
|
|
328
|
+
refusals the real `record` applies (`on_unanswered: prompt`, and an unsatisfiable assert pairing), so it
|
|
329
|
+
cannot green something a paid run would reject. On a directory it reports every offender and the batch
|
|
330
|
+
cost estimate. `lint` checks the assertion invariants (both above).
|
|
328
331
|
|
|
329
332
|
**Author answers WITHOUT re-paying — the cheap loop.** You don't need a fresh paid record to discover a
|
|
330
333
|
scenario's gates or their labels: `--keep` ONE run, then `cowork-harness trace <run-dir> --view questions`
|
|
@@ -547,7 +550,7 @@ Docker, no re-record.
|
|
|
547
550
|
| Situation | Symptom | Reach for (in order) |
|
|
548
551
|
|---|---|---|
|
|
549
552
|
| **The skill misbehaved** | wrong output, an unexpected gate, a denied tool, an opaque crash | `inspect` — what did it produce? · `trace <run-dir> --view <view>` — what did it actually do (tools, gates, sub-agent tree)? · `verify-run` — re-assert cheaply when only an assertion is wrong · `diff <old-run> <new-run>` — what changed since it worked · `chat` — reproduce it by hand |
|
|
550
|
-
| **A green you don't trust** | an assert that may have tested nothing, a stale cassette, an auto-answered or decided gate | `replay --explain` — the evidence trail behind each *passing* assert · `replay --mutate` — perturbs
|
|
553
|
+
| **A green you don't trust** | an assert that may have tested nothing, a stale cassette, an auto-answered or decided gate | `replay --explain` — the evidence trail behind each *passing* assert · `replay --mutate` — perturbs a CAPPED SAMPLE of recorded JSON values (10/file, 50 total) and reports which perturbations NOTHING caught; the report names the sample size, so read it as a sample not a total (reporting only; never moves the verdict/exit code) · `lint` — assertions on the wrong CI lane / mixed-class keys · `verify-cassettes` — privacy + staleness over committed cassettes · the Gotchas landmine catalog — how a check passes vacuously · `run --repeat N` / `skill --repeat N` — did it pass, or pass once? · `stats` — flaky or expensive over time |
|
|
551
554
|
|
|
552
555
|
A failed run also records `errorSource` (where the failure originated) and `stderrLogPath` (the captured
|
|
553
556
|
agent stderr) — read those before re-running; a re-record rarely tells you more than the captured stderr
|
|
@@ -836,10 +839,13 @@ repeats the assertion/replay-relevant ones alongside the schema (a scoped subset
|
|
|
836
839
|
sub-question count and a matching footer total — read that off instead of the tool-call count when
|
|
837
840
|
sizing the budget.
|
|
838
841
|
|
|
839
|
-
19. **`gate_answers_delivered` passes vacuously when no gate fires —
|
|
840
|
-
|
|
841
|
-
|
|
842
|
-
|
|
842
|
+
19. **`gate_answers_delivered` passes vacuously when no gate fires — pair it, or drop it.** Whether a
|
|
843
|
+
gate fires is model-dependent, so `gate_answers_delivered: true` alone can't catch "the gate never
|
|
844
|
+
fired at all". If the scenario is meant to gate, pair it with `gate_answer_count_min: 1` (a floor of
|
|
845
|
+
`0` witnesses nothing). If the scenario is gate-clean by design, drop the key — it asserts nothing
|
|
846
|
+
there — and declare `questions_count_max: 0`, which fails loudly if a gate ever appears. Asserting
|
|
847
|
+
`questions_count_max: 0` alongside a gate-presence key is unsatisfiable: `run`/`skill`/`record`
|
|
848
|
+
refuse it before spending.
|
|
843
849
|
|
|
844
850
|
20. **A `mode: r` connected folder's contents are recorded body-less, not excluded.** `record` captures a
|
|
845
851
|
read-only folder's files as path + hash only (`truncated: true`, no `body`) — it's an input the agent
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI recipe — replay vs live lanes
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 1.20.0` (baseline `desktop-1.26832.0`).
|
|
4
4
|
|
|
5
5
|
**Fastest path: the packaged Action.** One step gets you `replay`/`lint`/`verify-cassettes` plus a PR
|
|
6
6
|
job-summary reporter (verdict table, staleness findings, cost/turns when available):
|
|
@@ -13,7 +13,7 @@ job-summary reporter (verdict table, staleness findings, cost/turns when availab
|
|
|
13
13
|
```
|
|
14
14
|
|
|
15
15
|
The Action's `version` input defaults to `latest` — intentional so a copy-pasted recipe tracks the current
|
|
16
|
-
release; pin an exact version (e.g. `version: "1.
|
|
16
|
+
release; pin an exact version (e.g. `version: "1.20.0"`) for reproducible CI.
|
|
17
17
|
|
|
18
18
|
Reach for the manual multi-step form below only when you need per-step control the Action's inputs don't
|
|
19
19
|
cover (a custom flag combination, a different runner matrix per step, or `lint`/`verify-cassettes` gated
|
|
@@ -32,7 +32,7 @@ jobs:
|
|
|
32
32
|
- uses: actions/checkout@v4
|
|
33
33
|
- name: Stage the agent binary (official channel, sha256-verified — see https://github.com/yaniv-golan/cowork-harness/blob/main/docs/maintenance.md)
|
|
34
34
|
run: |
|
|
35
|
-
V=2.1.
|
|
35
|
+
V=2.1.222 # match your scenario's pinned baseline's agentVersion
|
|
36
36
|
curl -fSL "https://downloads.claude.ai/claude-code-releases/$V/linux-arm64/claude" -o "$RUNNER_TEMP/claude-$V"
|
|
37
37
|
chmod +x "$RUNNER_TEMP/claude-$V"
|
|
38
38
|
# verify against the committed baseline's sha256 (baselines/desktop-*.json → agentBinary.sha256)
|
|
@@ -58,7 +58,7 @@ sha256-*checked* but not hard-blocking on mismatch — it's advisory for an inte
|
|
|
58
58
|
GitHub-hosted runners, no token/Docker/agent:
|
|
59
59
|
|
|
60
60
|
```yaml
|
|
61
|
-
- run: npm i -g "cowork-harness@>=1.
|
|
61
|
+
- run: npm i -g "cowork-harness@>=1.20.0"
|
|
62
62
|
- run: cowork-harness lint scenarios/*.yaml --strict --min-severity WARN
|
|
63
63
|
# no silent false-greens. WITHOUT --strict this
|
|
64
64
|
# step cannot fail on a WARN-class rule (e.g.
|
|
@@ -279,7 +279,7 @@ jobs:
|
|
|
279
279
|
with: { node-version: '24' }
|
|
280
280
|
- uses: actions/setup-python@v5
|
|
281
281
|
with: { python-version: '3.x' } # python3 only — PyYAML is bundled with the linter
|
|
282
|
-
- run: npm i -g "cowork-harness@>=1.
|
|
282
|
+
- run: npm i -g "cowork-harness@>=1.20.0"
|
|
283
283
|
- run: cowork-harness lint scenarios/*.yaml # no-silent-false-green (needs python3; PyYAML bundled)
|
|
284
284
|
- run: cowork-harness verify-cassettes cassettes/ --output-format json # privacy + staleness gate
|
|
285
285
|
- run: cowork-harness replay cassettes/ --output-format json # token-free content/structure
|
|
@@ -308,7 +308,7 @@ jobs:
|
|
|
308
308
|
echo "live=true" >> "$GITHUB_OUTPUT"
|
|
309
309
|
fi
|
|
310
310
|
- if: steps.guard.outputs.live == 'true'
|
|
311
|
-
run: npm i -g "cowork-harness@>=1.
|
|
311
|
+
run: npm i -g "cowork-harness@>=1.20.0"
|
|
312
312
|
- if: steps.guard.outputs.live == 'true'
|
|
313
313
|
run: cowork-harness run scenarios/ --output-format json
|
|
314
314
|
env:
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Critique — the facts a plugin install can't otherwise reach
|
|
2
2
|
|
|
3
|
-
Tracks `cowork-harness 1.
|
|
3
|
+
Tracks `cowork-harness 1.20.0` (baseline `desktop-1.26832.0`). This is **not** a trim of the full
|
|
4
4
|
[`docs/critique.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/critique.md) (repo-only —
|
|
5
5
|
flags, cost, reproduction discipline, known limitations all live there). This file covers exactly what a
|
|
6
6
|
plugin install cannot otherwise discover: the run-dir artifact a harvester actually reads, the report's
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Fidelity tiers & answer paths
|
|
2
2
|
|
|
3
|
-
Self-contained reference. Tracks `cowork-harness 1.
|
|
3
|
+
Self-contained reference. Tracks `cowork-harness 1.20.0` (baseline `desktop-1.26832.0`).
|
|
4
4
|
|
|
5
5
|
## Fidelity tiers (`fidelity:` in the scenario)
|
|
6
6
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Scenario & session schema, assertion catalog, web_fetch, full gotchas
|
|
2
2
|
|
|
3
|
-
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.
|
|
4
|
-
(baseline `desktop-1.
|
|
3
|
+
Self-contained reference for authoring `cowork-harness` scenarios. Tracks `cowork-harness 1.20.0`
|
|
4
|
+
(baseline `desktop-1.26832.0`). If your checkout is newer, prefer the live [`docs/scenario.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/scenario.md),
|
|
5
5
|
[`docs/session.md`](https://github.com/yaniv-golan/cowork-harness/blob/main/docs/session.md), and `SPEC.md`.
|
|
6
6
|
|
|
7
7
|
**Minimal scenario** — `prompt` is the only required field:
|
|
@@ -329,9 +329,9 @@ same set live from the schema.
|
|
|
329
329
|
| `present_files_called: true` | at least one file was actually delivered via the `present_files` tool (`RunResult.presentedFiles` is non-empty) — the presence companion to `no_scratchpad_leak` (which passes vacuously when nothing was presented). Pair them to require a delivery **and** require it not to leak. Content-class (re-derives identically on replay). **`fidelity: container` or `hostloop`** — the harness serves `present_files` at both (hostloop via a handler mirroring production's own host-loop branch: validate the path, pass it through, no promotion). `protocol` and `microvm` report cannot-verify. See the `no_scratchpad_leak` row, which stays container-only for a different reason; and see the lane note there — the tool name differs on remote Cowork. **Only `true` is valid** |
|
|
330
330
|
| `question_asked: <regex>` | the agent asked an AskUserQuestion whose text matches |
|
|
331
331
|
| `questions_count_max: <N>` | at most N **sub-questions** asked — a bundled `AskUserQuestion` with K sub-questions counts as K, not 1; `trace --view questions`'s footer total uses the same definition |
|
|
332
|
-
| `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min` to also require a gate |
|
|
333
|
-
| `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests |
|
|
334
|
-
| `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass |
|
|
332
|
+
| `gate_answers_delivered: true` | every answered gate's answer reached the model (observed `tool_result`; unobserved = fail); **zero gates fired passes vacuously** — pair with `gate_answer_count_min: >= 1` to also require a gate, or drop this key and declare `questions_count_max: 0` in a scenario that expects none |
|
|
333
|
+
| `gate_answers_delivered: false` | asserts at least one answered gate's answer was **confirmed not delivered** (an observed delivery failure); an unobserved/null delivery does **not** satisfy this — for negative-path delivery tests. Requires a gate, so **mutually exclusive** with `questions_count_max: 0` (refused by `run`/`skill`/`record`) |
|
|
334
|
+
| `gate_answer_count_min: <N>` | at least N AskUserQuestion gates fired AND were delivered non-error — presence companion to `gate_answers_delivered`'s vacuous-pass. **`: 0` asserts nothing** and does not satisfy that pairing; `>= 1` is **mutually exclusive** with `questions_count_max: 0` (refused by `run`/`skill`/`record`) |
|
|
335
335
|
| `hook_blocked: <regex>` | a PreToolUse hook blocked a tool whose name matches the regex (`RunResult.hookEvents`) — evidence-unavailable if hook telemetry is absent. Replay: needs a `controlOut` cassette (a custom hook's decision lives only there, not the recorded stream) |
|
|
336
336
|
| `no_hook_blocked: true` | no tool was hook-blocked during the run (distinguishes a real tool crash from an intentional hook block) — evidence-unavailable if hook telemetry is absent. Replay: needs a `controlOut` cassette. **Only `true` is valid** |
|
|
337
337
|
| `vm_path_denied: true` | **`fidelity: hostloop` only** — at least one recorded path denial (`RunResult.pathDenials`, any source) targeted a `/sessions` VM path — evidence-unavailable if path-denial telemetry is absent. Replay: needs a `controlOut` cassette. Any other tier FAILS "cannot verify". **Only `true` is valid** |
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Each recipe composes facts that live scattered across SKILL.md and the other references into one
|
|
4
4
|
decision path. Every one answers a question a real fleet owner had to work out the hard way.
|
|
5
|
-
Tracks `cowork-harness 1.
|
|
5
|
+
Tracks `cowork-harness 1.20.0` (baseline `desktop-1.26832.0`), same as SKILL.md's front-matter. Recipe 2's `resolved-tier`/`unverifiable-tier` staleness classes and
|
|
6
6
|
Recipe 3's `init-redact` shipped in 0.24.0 and are part of the current feature set — no version gate
|
|
7
7
|
needed if your CLI meets SKILL.md's version floor.
|
|
8
8
|
|
|
@@ -115,7 +115,13 @@ have:
|
|
|
115
115
|
indexed runs — better than a single observation.
|
|
116
116
|
3. **Set the budget with headroom:** observed × 1.5, rounded up (agentic run lengths vary run to
|
|
117
117
|
run; a budget at the observed value flakes).
|
|
118
|
-
4.
|
|
118
|
+
4. **`questions_count_max: 0` is a declaration, not a budget.** If the observed value is zero, you are
|
|
119
|
+
writing "this scenario expects no gates" — which is supported and useful, but it is then **mutually
|
|
120
|
+
exclusive** with `gate_answer_count_min: >= 1`, `question_asked` and `gate_answers_delivered: false`
|
|
121
|
+
(a delivered gate records at least one question). `run` / `skill` / `record` refuse that pairing
|
|
122
|
+
before spending, and `lint` reports `assert-contradiction`. Drop `gate_answers_delivered`
|
|
123
|
+
alongside it too — it asserts nothing when no gate fires.
|
|
124
|
+
5. **Replay-class note:** `max_turns` / `tool_calls_max` / `dispatch_count_max` re-evaluate
|
|
119
125
|
token-free on replay; `questions_count_max` needs `controlOut` (Recipe 1's tree applies).
|
|
120
126
|
`max_cost_usd` / `max_tokens` on replay assert the FROZEN recording's spend — near-zero signal
|
|
121
127
|
as a regression gate; if you need a cost gate, put it on the live lane via `stats` (history) or
|
|
@@ -26,6 +26,10 @@ lint flags (see references/scenario-schema.md for the why of each):
|
|
|
26
26
|
E on_unanswered: agent / invalid value (schema rejects `agent`)
|
|
27
27
|
E authored `replay_protocol_fidelity` assertion (replay-synthesized only)
|
|
28
28
|
E `assertions:` instead of `assert:` (block ignored → every check no-ops)
|
|
29
|
+
E a presence assert + its absence sibling (unsatisfiable; run/skill/record refuse it:
|
|
30
|
+
questions_count_max:0 vs gate presence,
|
|
31
|
+
no_hook_blocked vs hook_blocked,
|
|
32
|
+
no_path_denied vs path_denied/vm_path_denied)
|
|
29
33
|
W `transcript_no_host_path` on `fidelity: cowork` (tier resolves per baseline gate —
|
|
30
34
|
incompatible if it lands hostloop)
|
|
31
35
|
W `no_scratchpad_leak` on `fidelity: cowork` (tier resolves per baseline gate)
|
|
@@ -338,6 +342,63 @@ def _all_assert_keys(items):
|
|
|
338
342
|
return keys
|
|
339
343
|
|
|
340
344
|
|
|
345
|
+
def _assert_values(items, key):
|
|
346
|
+
"""Every value authored for `key` across the assert items (a key may repeat across entries).
|
|
347
|
+
|
|
348
|
+
`_all_assert_keys` sees only NAMES, which is how this linter shipped two value-blind gate checks:
|
|
349
|
+
one firing on an assertion whose premise it inverted, one exempting a floor of zero."""
|
|
350
|
+
return [item[key] for item in items if isinstance(item, dict) and key in item]
|
|
351
|
+
|
|
352
|
+
|
|
353
|
+
def _numeric(v):
|
|
354
|
+
"""The number the HARNESS would see for a YAML scalar, or None if it isn't one.
|
|
355
|
+
|
|
356
|
+
NOT `isinstance(v, int)`. This linter parses with PyYAML (YAML 1.1); the harness loads scenarios
|
|
357
|
+
with the npm `yaml` package (YAML 1.2 core). `1.0` reaches Python as a float and `1e0` as a STRING,
|
|
358
|
+
yet npm yaml resolves both to the integer 1 and `z.number().int()` accepts them -- so both are
|
|
359
|
+
loadable scenarios that an int-only test would mis-read.
|
|
360
|
+
|
|
361
|
+
The dialect cuts the other way too, benignly: PyYAML resolves `1_000` -> 1000 and `1:30` -> 90
|
|
362
|
+
(sexagesimal), which npm yaml leaves as strings that `z.number()` REJECTS at load. Those read as a
|
|
363
|
+
number here and are refused by the loader -- silent lint, loud load. Same shape as `no`/`off`
|
|
364
|
+
resolving to False in 1.1 but staying strings in 1.2."""
|
|
365
|
+
if isinstance(v, bool):
|
|
366
|
+
return None # `True >= 1` is true in Python; a schema-invalid `: true` is not a floor
|
|
367
|
+
if isinstance(v, (int, float)):
|
|
368
|
+
return float(v)
|
|
369
|
+
if isinstance(v, str):
|
|
370
|
+
try:
|
|
371
|
+
return float(v)
|
|
372
|
+
except ValueError:
|
|
373
|
+
return None
|
|
374
|
+
return None
|
|
375
|
+
|
|
376
|
+
|
|
377
|
+
def _tool_glob_matches(pattern, name):
|
|
378
|
+
"""Port of `globToRegExp` (src/glob.ts) for a tool name.
|
|
379
|
+
|
|
380
|
+
`tool_called` is a GLOB, not a regex (src/types.ts `toolGlob`, src/assert.ts `toolMatches`):
|
|
381
|
+
anchored, CASE-SENSITIVE, only `*` and `?` special, every other character literal. A value carrying
|
|
382
|
+
a regex metacharacter is rejected at scenario load, so reading this field with `re.search` was wrong
|
|
383
|
+
on three axes at once -- it flagged valid globs, and exempted a wrong-case glob that can never match.
|
|
384
|
+
|
|
385
|
+
Segment-aware on purpose: a whole-segment `**` matches ZERO segments, so `**/AskUserQuestion`
|
|
386
|
+
matches a bare `AskUserQuestion`. That is a property of the pattern's segments, not the subject's,
|
|
387
|
+
which is why "a tool name contains no `/`" does not make a flat per-character loop equivalent."""
|
|
388
|
+
segments = pattern.replace("\\", "/").split("/")
|
|
389
|
+
rx = "^"
|
|
390
|
+
for i, seg in enumerate(segments):
|
|
391
|
+
last = i == len(segments) - 1
|
|
392
|
+
if seg == "**":
|
|
393
|
+
rx += "(?:[^/]+(?:/[^/]+)*)?" if last else "(?:[^/]+/)*"
|
|
394
|
+
continue
|
|
395
|
+
for ch in seg:
|
|
396
|
+
rx += "[^/]*" if ch == "*" else "[^/]" if ch == "?" else re.escape(ch)
|
|
397
|
+
if not last:
|
|
398
|
+
rx += "/"
|
|
399
|
+
return re.match(rx + "$", name) is not None
|
|
400
|
+
|
|
401
|
+
|
|
341
402
|
def _is_positional_choose(choose):
|
|
342
403
|
"""True if a `choose` value selects by POSITION — `first` or a 1-based index (scalar or in a
|
|
343
404
|
multiSelect list) — as opposed to an exact label. Positional answers are order-dependent (H1)."""
|
|
@@ -600,40 +661,150 @@ def lint_doc(doc, path, raw_lines):
|
|
|
600
661
|
# 0 gates sit green for weeks against a scenario asserting exactly this key.
|
|
601
662
|
#
|
|
602
663
|
# A companion is anything that FAILS (rather than vacuously passes) on an empty gate set:
|
|
603
|
-
# · gate_answer_count_min — the explicit floor
|
|
604
|
-
#
|
|
605
|
-
#
|
|
664
|
+
# · gate_answer_count_min WITH A FLOOR >= 1 — the explicit floor. The value matters: `: 0` is legal
|
|
665
|
+
# (the schema is nonnegative) and `delivered >= 0` always holds, so a zero floor witnesses nothing
|
|
666
|
+
# and must not silence this rule. Reading the key NAME alone made the "looks paired" scenario a
|
|
667
|
+
# silent false-green — worse than the loud one this rule was written for.
|
|
668
|
+
# · question_asked — fails "no question matched" against an empty question list, for ANY pattern
|
|
669
|
+
# (`.some` over an empty list is false, so no value gate is needed here).
|
|
670
|
+
# · tool_called whose GLOB matches AskUserQuestion — fails "tool not called".
|
|
606
671
|
# `questions_count_max` is deliberately NOT a companion: a MAX passes vacuously at zero, so pairing it
|
|
607
672
|
# leaves the hole wide open. The harness's own `scaffold` emits `question_asked` alongside this key,
|
|
608
673
|
# which is why the exemption must include it — otherwise this rule reds the tool's own output.
|
|
609
|
-
|
|
610
|
-
|
|
674
|
+
#
|
|
675
|
+
# VALUE, not just key. `gate_answers_delivered: false` is the INVERSE assertion — it demands at least
|
|
676
|
+
# one gate whose answer was confirmed NOT delivered, so zero gates FAILS it. It is not the vacuous
|
|
677
|
+
# direction, and `gate_answer_count_min` (which counts delivered === true) is not its companion.
|
|
678
|
+
# Firing there reds a correct negative-path scenario under CI's `--strict --min-severity WARN`, with a
|
|
679
|
+
# message whose premise is inverted. Skip on `is False` specifically, so a schema-invalid non-bool
|
|
680
|
+
# still warns rather than opening a new hole. (`no`/`off` also reach here as False under PyYAML's
|
|
681
|
+
# YAML 1.1 but are rejected by the loader as strings — see _numeric for the same dialect gap.)
|
|
682
|
+
delivered_values = _assert_values(items, "gate_answers_delivered")
|
|
683
|
+
if any(v is not False for v in delivered_values):
|
|
684
|
+
has_companion = "question_asked" in assert_keys or any(
|
|
685
|
+
(n := _numeric(v)) is not None and n >= 1 for v in _assert_values(items, "gate_answer_count_min")
|
|
686
|
+
)
|
|
611
687
|
if not has_companion:
|
|
612
688
|
for item in items:
|
|
613
689
|
tc = item.get("tool_called")
|
|
614
|
-
|
|
615
|
-
|
|
616
|
-
#
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
|
|
620
|
-
has_companion = True
|
|
621
|
-
break
|
|
622
|
-
except re.error:
|
|
623
|
-
continue
|
|
690
|
+
# A GLOB, not a regex (src/types.ts toolGlob) — ask whether THEIR pattern would match the
|
|
691
|
+
# gate tool under the harness's own matching rules. A non-str value is somebody else's
|
|
692
|
+
# finding; a glob cannot fail to compile, so there is nothing to guard against here.
|
|
693
|
+
if isinstance(tc, str) and _tool_glob_matches(tc, "AskUserQuestion"):
|
|
694
|
+
has_companion = True
|
|
695
|
+
break
|
|
624
696
|
if not has_companion:
|
|
625
|
-
|
|
626
|
-
|
|
627
|
-
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
|
|
634
|
-
|
|
697
|
+
# The scenario may already have DECLARED that it expects no gates. Then "add a companion" is
|
|
698
|
+
# wrong advice — the key is inert, and the fix is to drop it. Same rule id, different message;
|
|
699
|
+
# it must not go silent, because `questions_count_max: 0` is still not a presence companion.
|
|
700
|
+
declares_zero_gates = any(_numeric(v) == 0 for v in _assert_values(items, "questions_count_max"))
|
|
701
|
+
if declares_zero_gates:
|
|
702
|
+
findings.append(
|
|
703
|
+
Finding(
|
|
704
|
+
"WARN",
|
|
705
|
+
"vacuous-gate-assert",
|
|
706
|
+
"`gate_answers_delivered` is inert here — this scenario already declares it expects "
|
|
707
|
+
"no gates (`questions_count_max: 0`), and zero gates fired passes "
|
|
708
|
+
"`gate_answers_delivered` VACUOUSLY, so the key asserts nothing.",
|
|
709
|
+
"Drop `gate_answers_delivered` — `questions_count_max: 0` already states the intent "
|
|
710
|
+
"and fails loudly if a gate ever appears. If the scenario is meant to gate after all, "
|
|
711
|
+
"drop `questions_count_max: 0` and pair the delivery check with "
|
|
712
|
+
"`gate_answer_count_min: 1`.",
|
|
713
|
+
path,
|
|
714
|
+
)
|
|
635
715
|
)
|
|
716
|
+
else:
|
|
717
|
+
findings.append(
|
|
718
|
+
Finding(
|
|
719
|
+
"WARN",
|
|
720
|
+
"vacuous-gate-assert",
|
|
721
|
+
"`gate_answers_delivered` has no presence companion — zero gates fired passes it "
|
|
722
|
+
"VACUOUSLY, so this assertion stays green if the skill stops asking altogether "
|
|
723
|
+
"(exactly the regression it looks like it is guarding).",
|
|
724
|
+
"If the scenario is meant to gate, pair it with `gate_answer_count_min: 1` (a floor "
|
|
725
|
+
'of 0 witnesses nothing) — or a `question_asked: "<rx>"`, or `tool_called: '
|
|
726
|
+
'"AskUserQuestion"` — so a gate must actually fire. If the scenario is gate-clean by '
|
|
727
|
+
"design, DROP `gate_answers_delivered` (it asserts nothing there) and declare the "
|
|
728
|
+
"intent with `questions_count_max: 0` (or `tool_not_called: \"AskUserQuestion\"`), "
|
|
729
|
+
"which fails loudly if a gate ever appears.",
|
|
730
|
+
path,
|
|
731
|
+
)
|
|
732
|
+
)
|
|
733
|
+
|
|
734
|
+
# E: a statically unsatisfiable assert pairing — the scenario can never pass, on any lane.
|
|
735
|
+
#
|
|
736
|
+
# Each group pairs ONE assertion demanding a record NOT exist with the assertions demanding the SAME
|
|
737
|
+
# record does exist, on a single evidence channel:
|
|
738
|
+
# · questions_count_max: 0 vs gate presence — a delivered gate records at least one question (the
|
|
739
|
+
# harness pushes one entry per sub-question BEFORE answering, and a zero-question gate throws).
|
|
740
|
+
# · no_hook_blocked vs hook_blocked — one hook-event list.
|
|
741
|
+
# · no_path_denied vs path_denied / vm_path_denied — one path-denial list.
|
|
742
|
+
# Verified against the assertion implementations, not the schema prose: a scope split would make a
|
|
743
|
+
# pair satisfiable, and there is none. Where the evidence is missing BOTH halves fail
|
|
744
|
+
# evidence-unavailable rather than passing, and the denial keys are hostloop-only so a wrong tier
|
|
745
|
+
# fails both too — no combination yields a both-pass.
|
|
746
|
+
#
|
|
747
|
+
# Deliberately NOT extended to `tool_not_called` as a gate witness: it reads the tool log while the
|
|
748
|
+
# gate keys read the control channel, and a fixture-driven `protocol` run can gate on the control
|
|
749
|
+
# channel alone, so that cross-channel contradiction is not provable from the YAML.
|
|
750
|
+
#
|
|
751
|
+
# ERROR because `run`/`skill`/`record` refuse the scenario outright (assertContradiction in
|
|
752
|
+
# src/run/execute.ts) — the same reason `on_unanswered: agent` is an ERROR. Note the pairing is
|
|
753
|
+
# unsatisfiable, not always-red: on a lane where the evidence is absent it is not evaluated at all
|
|
754
|
+
# (verify-run fails evidence-unavailable; a controlOut-less cassette SKIPS these keys on replay).
|
|
755
|
+
# Either way it guards nothing.
|
|
756
|
+
contradiction_groups = [
|
|
757
|
+
(
|
|
758
|
+
"`questions_count_max: 0`",
|
|
759
|
+
any(_numeric(v) == 0 for v in _assert_values(items, "questions_count_max")),
|
|
760
|
+
[
|
|
761
|
+
(
|
|
762
|
+
"gate_answer_count_min: >= 1",
|
|
763
|
+
any((n := _numeric(v)) is not None and n >= 1 for v in _assert_values(items, "gate_answer_count_min")),
|
|
764
|
+
),
|
|
765
|
+
("question_asked", "question_asked" in assert_keys),
|
|
766
|
+
("gate_answers_delivered: false", any(v is False for v in _assert_values(items, "gate_answers_delivered"))),
|
|
767
|
+
],
|
|
768
|
+
"a delivered gate records at least one question, so requiring a gate to be present contradicts requiring zero questions",
|
|
769
|
+
),
|
|
770
|
+
(
|
|
771
|
+
"`no_hook_blocked: true`",
|
|
772
|
+
any(v is True for v in _assert_values(items, "no_hook_blocked")),
|
|
773
|
+
[("hook_blocked", "hook_blocked" in assert_keys)],
|
|
774
|
+
"both read the same hook-event list — the block `hook_blocked` requires is the one `no_hook_blocked` requires not to exist",
|
|
775
|
+
),
|
|
776
|
+
(
|
|
777
|
+
"`no_path_denied: true`",
|
|
778
|
+
any(v is True for v in _assert_values(items, "no_path_denied")),
|
|
779
|
+
[
|
|
780
|
+
("path_denied", "path_denied" in assert_keys),
|
|
781
|
+
("vm_path_denied: true", any(v is True for v in _assert_values(items, "vm_path_denied"))),
|
|
782
|
+
],
|
|
783
|
+
"both read the same path-denial list — the denial the positive key requires is one `no_path_denied` requires not to exist",
|
|
784
|
+
),
|
|
785
|
+
]
|
|
786
|
+
clauses = []
|
|
787
|
+
for absence_label, absent, presences, why in contradiction_groups:
|
|
788
|
+
if not absent:
|
|
789
|
+
continue
|
|
790
|
+
hits = [label for label, present in presences if present]
|
|
791
|
+
# Report EVERY contradictory group, not just the first — a scenario can carry more than one.
|
|
792
|
+
if hits:
|
|
793
|
+
clauses.append(f"{absence_label} alongside {' and '.join(hits)} ({why})")
|
|
794
|
+
if clauses:
|
|
795
|
+
findings.append(
|
|
796
|
+
Finding(
|
|
797
|
+
"ERROR",
|
|
798
|
+
"assert-contradiction",
|
|
799
|
+
# "both" is wrong once more than one group is named -- and a scenario carrying two is
|
|
800
|
+
# exactly the one whose message gets read carefully.
|
|
801
|
+
f"{'; and '.join(clauses)} — "
|
|
802
|
+
+ ("no run can satisfy both." if len(clauses) == 1 else "no run can satisfy all of them."),
|
|
803
|
+
"Keep the negative assertion and drop the positive one, or drop the negative if the "
|
|
804
|
+
"scenario really does expect the record.",
|
|
805
|
+
path,
|
|
636
806
|
)
|
|
807
|
+
)
|
|
637
808
|
|
|
638
809
|
# W: mixed-class assert item → the live-only half is dropped on replay (manifest-backed keys are NOT)
|
|
639
810
|
for idx, item in enumerate(items):
|
|
@@ -1615,8 +1786,13 @@ def build_scenario(args):
|
|
|
1615
1786
|
# sub-questions each gate bundles. Any number emitted here would be a guess: too low false-reds
|
|
1616
1787
|
# on the first run, too high is a dead tripwire. A budget must come from observation, not
|
|
1617
1788
|
# fabrication — so emit it COMMENTED OUT with the calibration path, not a made-up value.
|
|
1789
|
+
# `<N> >= 1` is not decoration: this block also emits gate-PRESENCE assertions, and
|
|
1790
|
+
# `questions_count_max: 0` alongside one of those is unsatisfiable — refused by run/skill/record
|
|
1791
|
+
# and flagged as `assert-contradiction`. Zero is the natural value to reach for on a
|
|
1792
|
+
# gate-clean scenario, so the hint has to say where it does NOT belong.
|
|
1618
1793
|
content_lines.append(
|
|
1619
|
-
" # - questions_count_max: <N> # BUDGET
|
|
1794
|
+
" # - questions_count_max: <N> # BUDGET (N >= 1 — `0` contradicts the gate assertions "
|
|
1795
|
+
"below; use it only in a scenario with none) — calibrate from a real run: `trace --view "
|
|
1620
1796
|
"questions` prints the SUB-question total (what this asserts); set N to that + headroom."
|
|
1621
1797
|
)
|
|
1622
1798
|
content_lines.append(" - gate_answers_delivered: true # the steered answers actually reached the model")
|