cowork-harness 2.0.1 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/cowork-harness/SKILL.md +5 -5
- package/.claude/skills/cowork-harness/references/ci-recipe.md +29 -16
- package/.claude/skills/cowork-harness/references/critique.md +1 -1
- package/.claude/skills/cowork-harness/references/fidelity-and-answers.md +1 -1
- package/.claude/skills/cowork-harness/references/scenario-schema.md +3 -3
- package/.claude/skills/cowork-harness/references/task-recipes.md +2 -2
- package/CHANGELOG.md +366 -0
- package/README.md +25 -8
- package/RELEASING.md +7 -2
- package/SPEC.md +24 -4
- package/dist/assert.js +18 -3
- package/dist/baseline.js +37 -15
- package/dist/cli.js +5 -1
- package/dist/prompt.js +5 -2
- package/dist/redact.js +44 -0
- package/dist/run/budget.js +21 -6
- package/dist/run/cassette.js +76 -14
- package/dist/run/chat-result.js +1 -0
- package/dist/run/chat.js +2 -0
- package/dist/run/execute.js +21 -6
- package/dist/run/run.js +19 -0
- package/dist/runtime/argv.js +20 -1
- package/dist/runtime/container.js +11 -3
- package/dist/runtime/hostloop.js +5 -1
- package/dist/runtime/lima.js +7 -1
- package/dist/runtime/microvm.js +22 -4
- package/dist/session.js +4 -0
- package/dist/types.js +1 -1
- package/docs/cassette.md +18 -6
- package/docs/invariants.md +5 -2
- package/docs/maintenance.md +5 -3
- package/docs/protocol.md +23 -5
- package/docs/scenario.md +9 -6
- package/examples/replays/README.md +1 -1
- package/fixtures/protocol/v1/dialog-response.json +10 -0
- package/fixtures/protocol/v1/elicit-response.json +10 -0
- package/fixtures/protocol/v1/elicitation-request.json +17 -0
- package/fixtures/protocol/v1/error-response.json +8 -0
- package/fixtures/protocol/v1/user-dialog-request.json +11 -0
- package/package.json +1 -1
- package/schema/cassette.v12.json +2 -2
- package/schema/protocol.v1.json +447 -79
- package/schema/run-result.json +4 -0
- package/schema/scenario.schema.json +1 -1
- package/scripts/check-versions.ts +134 -1
package/README.md
CHANGED
|
@@ -115,7 +115,7 @@ node dist/cli.js replay examples/replays/example-pdf-skill.cassette.json
|
|
|
115
115
|
|
|
116
116
|
> **Installed globally instead?** Once linked/installed, the same command is `cowork-harness replay
|
|
117
117
|
> <cassette>` — but the relative path above only resolves from a source checkout's `examples/replays/`.
|
|
118
|
-
> From a global install (`npm i -g "cowork-harness@^2.0
|
|
118
|
+
> From a global install (`npm i -g "cowork-harness@^2.2.0"`), point at the package root instead:
|
|
119
119
|
> `cowork-harness replay "$(npm root -g)/cowork-harness/examples/replays/example-pdf-skill.cassette.json"`
|
|
120
120
|
> (or copy the cassette into your own project and pass that path).
|
|
121
121
|
|
|
@@ -125,7 +125,7 @@ Live `run`/`skill` need the prerequisites in the next section — note the `prot
|
|
|
125
125
|
> - **Replay only (zero setup):** `cowork-harness replay <cassette>` — no token, no Docker, no agent. The command above.
|
|
126
126
|
> - **`protocol` (real model, no Docker):** needs only the auth token (item 3 below).
|
|
127
127
|
> - **Live `container` / `microvm` / `hostloop` / `cowork`:** needs Docker (or Lima for `microvm`), a staged agent, and the token — run `cowork-harness doctor` first.
|
|
128
|
-
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@^2.0
|
|
128
|
+
> - **Invocation:** from a source checkout, `node dist/cli.js <cmd>` (or `npm link` to get the `cowork-harness` command); from a global install, `cowork-harness <cmd>`; the companion skill falls back to `npx "cowork-harness@^2.2.0"`.
|
|
129
129
|
|
|
130
130
|
Two more worked examples worth knowing about: `examples/scenarios/protocol-smoke.yaml` (zero-Docker smoke
|
|
131
131
|
test) and `examples/scenarios/skill-loads.yaml` (container-tier acceptance check) — see
|
|
@@ -150,7 +150,7 @@ claude plugin marketplace add yaniv-golan/cowork-harness
|
|
|
150
150
|
claude plugin install cowork-harness@cowork-harness
|
|
151
151
|
```
|
|
152
152
|
|
|
153
|
-
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@^2.0
|
|
153
|
+
The skill **self-bootstraps the CLI**: if `cowork-harness` isn't on your PATH it falls back to `npx "cowork-harness@^2.2.0"` (a version floor that fails loud rather than silently fetching a too-old CLI; Node ≥ 22). Tiers above `protocol` still need Docker/Lima and a Claude Desktop agent binary — see the prerequisites below.
|
|
154
154
|
|
|
155
155
|
It also follows the open [Agent Skills](https://agentskills.io) spec, so it installs cross-editor (Cursor, Codex, OpenCode, …) via [`npx skills`](https://github.com/vercel-labs/skills) (Vercel Labs' CLI implementation of that spec):
|
|
156
156
|
|
|
@@ -175,7 +175,7 @@ global install puts nothing in your working directory. The matrix, answer-policy
|
|
|
175
175
|
ones that still need a source checkout. (The marketplace
|
|
176
176
|
skill install itself only pulls `.claude/skills/cowork-harness/` — SKILL.md + `references/` + `scenario.py`/
|
|
177
177
|
assertion keys, per `.claude-plugin/marketplace.json`'s `source` — not the rest of this table; the full set
|
|
178
|
-
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@^2.0
|
|
178
|
+
above becomes available once the skill's first command self-bootstraps `npx "cowork-harness@^2.2.0"` — see
|
|
179
179
|
[above](#drive-it-from-claude-code-companion-skill) — which pulls the same npm package as the global-install row.)
|
|
180
180
|
|
|
181
181
|
### Prerequisites for anything above `protocol` fidelity
|
|
@@ -358,7 +358,11 @@ Exceptions worth knowing:
|
|
|
358
358
|
|
|
359
359
|
- the `boundary-check` **command**'s own probe failures follow the assertion convention and exit `1` (see SPEC.md §11);
|
|
360
360
|
- `verify-cassettes` reuses exit `3` with a different meaning — "could not verify" (vs. a verified failure's `1`); see the command table;
|
|
361
|
-
- exit `4`
|
|
361
|
+
- `rehash` uses **exit `4` for PARTIAL success** — some cassettes migrated, some could not. That and a total
|
|
362
|
+
failure (`1`, nothing migrated) demand opposite responses, so they are separate codes; `0` means all
|
|
363
|
+
migrated or nothing needed migrating (SPEC §11);
|
|
364
|
+
- exit `4` is otherwise reserved on the `run`/`skill` family, where it is still unused (SPEC §11) — the code
|
|
365
|
+
space is per-command, so `rehash`'s `4` above does not touch that reservation.
|
|
362
366
|
|
|
363
367
|
After a run, the footer **echoes every auto-answered
|
|
364
368
|
question as a copy-pasteable `--answer "<q>=<choice>"` line** — run once exploratorily, then paste them
|
|
@@ -719,11 +723,23 @@ cowork-harness run scenarios/ # your repo's scenarios; runs every *.y
|
|
|
719
723
|
|
|
720
724
|
The fastest path to CI: a composite action wrapping the token-free lane, with a PR job-summary reporter.
|
|
721
725
|
|
|
726
|
+
> **The `uses:` ref pins the Action, not the CLI.** `@main`, `@v1` and a commit SHA all select which
|
|
727
|
+
> *Action* runs; which *CLI* it installs is the separate `version:` input below, which defaults to
|
|
728
|
+
> `latest`. The two move independently — so a workflow whose `uses:` ref has not changed in months still
|
|
729
|
+
> picks up a CLI **major** the moment one is promoted to `latest`. That is not hypothetical: `@v1` points
|
|
730
|
+
> at 1.24.0 and has not moved, yet an `@v1` workflow without a `version:` input installs 2.x today.
|
|
731
|
+
> **To hold a major, pin the input** — `version: ^2` (or `^1`) — not the `uses:` ref.
|
|
732
|
+
>
|
|
733
|
+
> Upgrading across the 1.x → 2.x boundary this way means the hash-format epoch: cassettes recorded before
|
|
734
|
+
> format v12 need `cowork-harness rehash <dir/>` (no re-record). See the 2.0.0 entry in
|
|
735
|
+
> [CHANGELOG.md](./CHANGELOG.md).
|
|
736
|
+
|
|
722
737
|
```yaml
|
|
723
|
-
- uses: yaniv-golan/cowork-harness@
|
|
738
|
+
- uses: yaniv-golan/cowork-harness@v2
|
|
724
739
|
with:
|
|
725
740
|
command: replay # replay | lint | lint-skill | analyze-skill | verify-cassettes | run
|
|
726
741
|
path: cassettes/my-skill.cassette.json
|
|
742
|
+
version: "^2" # hold the CLI major; the input defaults to `latest`
|
|
727
743
|
```
|
|
728
744
|
|
|
729
745
|
| Lane | Commands | Runner requirements | What you get |
|
|
@@ -748,14 +764,15 @@ jobs:
|
|
|
748
764
|
curl -fSL "https://downloads.claude.ai/claude-code-releases/$V/linux-arm64/claude" -o "$RUNNER_TEMP/claude-$V"
|
|
749
765
|
chmod +x "$RUNNER_TEMP/claude-$V"
|
|
750
766
|
echo "COWORK_AGENT_BINARY=$RUNNER_TEMP/claude-$V" >> "$GITHUB_ENV"
|
|
751
|
-
- uses: yaniv-golan/cowork-harness@
|
|
767
|
+
- uses: yaniv-golan/cowork-harness@v2
|
|
752
768
|
with:
|
|
753
769
|
command: run
|
|
754
770
|
path: scenarios/
|
|
771
|
+
version: "^2"
|
|
755
772
|
anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
756
773
|
```
|
|
757
774
|
|
|
758
|
-
Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` —
|
|
775
|
+
Every run writes a Markdown verdict table (scenario, pass/fail, signals, cost/turns when available, staleness findings, and the replay-skipped-assertions honesty line) to the job summary. Inputs: `command`, `path` (required), `version` (npm dist-tag/version, default `latest` — the recipes above pin `^2` instead, because leaving it at `latest` takes a CLI major the moment it is promoted even though your `uses:` ref never changed; pin an exact version for byte-reproducible CI. The companion skill's `cowork-harness@^2.2.0` floor guidance applies to ad-hoc CLI installs, not this input), `strict` (applies to `replay` (staleness findings), `lint`/`lint-skill` (WARN/INFO), and `analyze-skill` (any **`error`**-severity finding — advisory findings are precisely the class that does NOT gate); IGNORED — not forwarded — for `verify-cassettes`/`run`, which don't accept the flag), `fail-on-skill-drift` (**`replay`-only** — never forwarded to the analyzers), `extra-args`, `summary` (default `true`), `anthropic-api-key` (live lane only). Outputs: `ok` (`"true"`/`"false"`, mirrors the exit code), `envelope-path` (path to the raw JSON envelope, for post-processing), `summary-md` (the rendered verdict table, exposed as an output — not just written to `$GITHUB_STEP_SUMMARY` — because that file is scoped to this action's own invocation and a caller's later step gets a fresh, empty one). See [`action.yml`](https://github.com/yaniv-golan/cowork-harness/blob/main/action.yml) for the full input/output reference.
|
|
759
776
|
|
|
760
777
|
The provided [GitHub Actions workflow](https://github.com/yaniv-golan/cowork-harness/blob/main/.github/workflows/ci.yml) runs a **nine-stage pipeline**. The **build** + **test** stages are the token-free gate you can copy into your skill repo; the `floor`, `action-self-test`, `python`, `image-recipe`, `boundary`, `scenarios`, and `parity-drift` stages are this repo's own fidelity self-tests and are not directly portable (they build the harness's Docker image and run harness-specific e2e scenarios — see [`ci-recipe.md`](./.claude/skills/cowork-harness/references/ci-recipe.md) for the skill-repo template):
|
|
761
778
|
|
package/RELEASING.md
CHANGED
|
@@ -223,8 +223,13 @@ tagging `1.0.0`, deliberately review and freeze the surfaces with no machine-rea
|
|
|
223
223
|
gh run watch $(gh run list --workflow=release.yml --limit 1 --json databaseId --jq '.[0].databaseId')
|
|
224
224
|
```
|
|
225
225
|
- [ ] **Clean up**: `git push origin --delete release/X.Y.Z && git branch -d release/X.Y.Z`
|
|
226
|
-
- [
|
|
227
|
-
|
|
226
|
+
- [x] **Move the major/minor tags** — **AUTOMATED** by `release.yml`'s last step, which points `vX` and
|
|
227
|
+
`vX.Y` at the release it just published. It skips a prerelease tag entirely, and never moves an alias
|
|
228
|
+
backwards (re-releasing an older patch on a line moves `vX.Y` but leaves `vX` where it is). Left as a
|
|
229
|
+
checklist item, it lapsed: `v1` sat at 1.24.0 through two releases. Note what these tags do NOT do:
|
|
230
|
+
they select the ACTION, never the CLI. A consumer who leaves the `version:` input at its `latest`
|
|
231
|
+
default already tracks CLI majors regardless of which alias they pin, so moving `vX` neither causes
|
|
232
|
+
nor prevents a cross-major CLI upgrade for them. Manual fallback, if the step ever needs redoing:
|
|
228
233
|
```
|
|
229
234
|
git tag -f vX vX.Y.Z && git tag -f vX.Y vX.Y.Z # e.g. v1 and v1.0 → v1.2.3
|
|
230
235
|
git push -f origin vX vX.Y
|
package/SPEC.md
CHANGED
|
@@ -485,9 +485,11 @@ token-free assertion-iteration loop. That path is safe by construction: it **har
|
|
|
485
485
|
drift (`prompt` / `baseline` / `fidelity` / `answers` / `skills` / `requires_capabilities`) and, when a skill
|
|
486
486
|
fingerprint was recorded, on skill-content staleness (it implies `--fail-on-skill-drift`). `expect_denied` and
|
|
487
487
|
the filesystem/egress keys are sourced from on-disk but remain live-only (sourced ≠ evaluated; replay warns on
|
|
488
|
-
such an edit). The `session`
|
|
489
|
-
|
|
490
|
-
|
|
488
|
+
such an edit). The `session` is **not drift-checked on the replay path**, so a session change between record and
|
|
489
|
+
re-assert does not move the replay verdict — the notice states this; re-record if the session changed.
|
|
490
|
+
It *is* fingerprinted, but only `verify-cassettes` checks that hash (§11.1): `sessionFingerprint`
|
|
491
|
+
covers the session's connected `folders`/`plugins`/`skills`/`mcp`/`egress`/`web_fetch`, plus `projects`
|
|
492
|
+
and `agent_env` when set. `model` is covered by neither, so a model swap is undetected everywhere.
|
|
491
493
|
|
|
492
494
|
**`replay_protocol_fidelity` (O7 guard):** after the run, `replay` re-serializes each decision
|
|
493
495
|
response via `serializeDecision` and compares to the frozen `controlOut` envelope (canonical
|
|
@@ -647,7 +649,8 @@ is never truncated by `process.exit` on a pipe.)
|
|
|
647
649
|
question" outcome (the deferred `on_unanswered: surface` / `needs_input` Track 2). It is currently
|
|
648
650
|
unused — reserving it now keeps a later addition additive rather than a renumbering of the burned
|
|
649
651
|
`0`/`1`/`2`/`3` space. Exit-code space is **per-command**, not global (`status` uses `0`/`1`/`2`/`3`
|
|
650
|
-
with its own meanings); this reservation applies only to the `run`/`skill` family.
|
|
652
|
+
with its own meanings); this reservation applies only to the `run`/`skill` family. `rehash` does use `4`,
|
|
653
|
+
for partial migration success (above) — a different command, so it does not consume this reservation.
|
|
651
654
|
|
|
652
655
|
**Per-command exceptions:** `critique` **never gates on findings** — it exits `0` for any finding of any classification, and even when the task run it graded ERRORED (that is a finding about the skill, not a broken instrument). It exits `2` only for a usage error or an **instrument failure**: the turn was killed, the reflection protocol broke, or the evaluator was never invoked *or threw* — i.e. no critique was produced. Do not gate CI on `critique`; that inverts its design. `lint` exits `127` when `python3` is missing (spawn error); `replay` exits
|
|
653
656
|
`2` on a **whole-cassette operational failure** — anything `readCassette` rejects (unreadable, invalid
|
|
@@ -666,6 +669,14 @@ verification could NOT complete (any `unverifiable-*`-class staleness finding, a
|
|
|
666
669
|
newer harness than this one understands, or a per-file read error/crash — including a
|
|
667
670
|
malformed/unreadable cassette, which is tallied there rather than as a `1` finding). A real finding
|
|
668
671
|
always outranks a could-not-verify signal within the same run, so exit `1` wins if both occur.
|
|
672
|
+
**`rehash` uses its own four-way split:** `0` all migrated (or nothing needed migrating) · **`4` PARTIAL —
|
|
673
|
+
some migrated, some could not** · `1` nothing migrated and at least one could not · `2` usage. The partial
|
|
674
|
+
code is distinct because the two failing shapes demand opposite responses — commit what migrated and budget
|
|
675
|
+
a re-record for the rest, versus nothing here is salvageable — and while both were `1` a shell consumer
|
|
676
|
+
reading only the exit code could not tell them apart (the JSON envelope always carried the split as
|
|
677
|
+
`migrated`/`skipped`/`errors`). `4` rather than `3`: the code space is per-command, and `3`'s "could not
|
|
678
|
+
verify" meaning is load-bearing on `verify-cassettes` — a migration that partly succeeded is not a failed
|
|
679
|
+
verification.
|
|
669
680
|
**Collision warning:** the `run`/`skill` family's exit `3` means *boundary/integrity* (§11 above);
|
|
670
681
|
`verify-cassettes`' exit `3` means *could not verify*. These are unrelated per-command meanings that
|
|
671
682
|
happen to share a number — a CI script that branches on exit code across commands must not conflate
|
|
@@ -839,6 +850,15 @@ Covered-surface changes follow semver as of `1.0.0` — see [RELEASING.md](./REL
|
|
|
839
850
|
future-stamped cassette without refusing, since neither produces a pass/fail verdict. Post-1.0, raising
|
|
840
851
|
the read floor past a still-readable version is breaking.
|
|
841
852
|
- **Control protocol** — `schema/protocol.v1.json` + the golden control-response vectors (§5).
|
|
853
|
+
Removing or narrowing a described frame is breaking; **DESCRIBING A FRAME THE HARNESS ALREADY
|
|
854
|
+
ANSWERS IS NOT** — it can only make a validator that was rejecting real traffic start accepting it,
|
|
855
|
+
so no working consumer changes behaviour. (The same additive latitude the `verify-cassettes` and
|
|
856
|
+
`doctor` envelopes above are granted. Stated explicitly because its absence here read as a
|
|
857
|
+
prohibition: three subtypes the harness had always answered — `request_user_dialog`, `elicitation`
|
|
858
|
+
and `side_question` — plus the fail-closed `subtype:"error"` envelope went undescribed rather than
|
|
859
|
+
added. A schema that rejects frames the code sends is worse than one that grows.) The golden vector
|
|
860
|
+
pack is coupled: every schema definition must be exercised by a vector, so a new frame ships with
|
|
861
|
+
evidence or not at all.
|
|
842
862
|
- **Environment variables** — the documented `COWORK_HARNESS_*` knobs plus `COWORK_AGENT_BINARY` and
|
|
843
863
|
`COWORK_AGENT_IMAGE`. Renaming a documented var or changing its meaning is breaking.
|
|
844
864
|
- **Packaged GitHub Action** — `action.yml` inputs (`command`, `path`, `version`, `strict`,
|
package/dist/assert.js
CHANGED
|
@@ -1141,10 +1141,25 @@ function check(a, ctx) {
|
|
|
1141
1141
|
results.push(fail("present_files_called: present_files is not served on `lane: remote` — a local MCP server cannot reach a remote Cowork session, which delivers via the agent-native SendUserFile instead (not modeled; see docs/fidelity-gaps.md) — cannot verify"));
|
|
1142
1142
|
else if (ctx.effectiveFidelity !== "container" && ctx.effectiveFidelity !== "hostloop")
|
|
1143
1143
|
results.push(fail(`present_files_called: present_files is served only on the container/hostloop tiers (this run: ${ctx.effectiveFidelity ?? "unknown"}) — cannot verify; use fidelity: container or hostloop for present_files-based delivery`));
|
|
1144
|
-
|
|
1145
|
-
|
|
1146
|
-
|
|
1144
|
+
// PRESENCE comes from the invocation count, not from `presentedFiles`. The two answer different
|
|
1145
|
+
// questions, and only one of them survives redaction: `presentedFiles` entries are dropped when a
|
|
1146
|
+
// path can't be classified, which a host-path policy guarantees at hostloop (a real host path
|
|
1147
|
+
// redacts to `[REDACTED:…]/mnt/outputs/f`, and the classifier requires an absolute path). Reading
|
|
1148
|
+
// delivery off classification made a redacted recording report "the tool was never called" about a
|
|
1149
|
+
// run that called it three times — and, because record's redaction self-check replays both
|
|
1150
|
+
// cassettes and compares verdicts, no such cassette could be written at all.
|
|
1151
|
+
// `presentedFiles.length` stays as the fallback for a run recorded before the count existed.
|
|
1152
|
+
else if ((ctx.presentFilesCalls ?? 0) > 0 || (ctx.presentedFiles?.length ?? 0) > 0)
|
|
1147
1153
|
results.push(ok());
|
|
1154
|
+
// Called, but every call's `files` was unusable — the tool WAS invoked, so "never called" would be a
|
|
1155
|
+
// factual claim the harness knows to be false. Mirrors no_scratchpad_leak's malformed branch above.
|
|
1156
|
+
// The count is deliberately NOT interpolated: record's self-check normalizes [REDACTED…] tokens out
|
|
1157
|
+
// of failing messages but not digits, so a count that differs between the base and redacted replays
|
|
1158
|
+
// would trip its message compare and refuse a record that is otherwise fine to write.
|
|
1159
|
+
else if (ctx.evidenceErrors?.presentFilesMalformed)
|
|
1160
|
+
results.push(fail(`present_files_called: present_files WAS called, but no call carried a usable file path (malformed input) — cannot verify delivery`));
|
|
1161
|
+
else
|
|
1162
|
+
results.push(fail(`present_files_called: no file was delivered via present_files (the tool was never called)`));
|
|
1148
1163
|
}
|
|
1149
1164
|
if (a.no_skill_triggered !== undefined) {
|
|
1150
1165
|
const c = compileUserRegex(a.no_skill_triggered);
|
package/dist/baseline.js
CHANGED
|
@@ -319,25 +319,47 @@ function latestBaselineFile() {
|
|
|
319
319
|
files.sort(compareBaselineVersions);
|
|
320
320
|
return join(BASELINES_DIR, files[files.length - 1]);
|
|
321
321
|
}
|
|
322
|
+
/** The one segment the harness's staged session tree always adds under the session root. Every stager
|
|
323
|
+
* and every argv builder composes guest paths with it (`stage.ts`/`hostloop-stage.ts` create
|
|
324
|
+
* `<sessionHost>/mnt`, `dockerRunArgv` nests `:ro` binds at `<sessionRoot>/mnt/<mountPath>`), so a
|
|
325
|
+
* guest mnt root that is anything OTHER than `<sessionRoot>/mnt` cannot be produced. */
|
|
326
|
+
export const GUEST_MNT_SEGMENT = "mnt";
|
|
322
327
|
/**
|
|
323
|
-
* Expand the mount layout for a concrete session id
|
|
324
|
-
*
|
|
325
|
-
*
|
|
328
|
+
* Expand the mount layout for a concrete session id — the layout of the tree THIS HARNESS stages, which
|
|
329
|
+
* is what every guest path must be built from.
|
|
330
|
+
*
|
|
331
|
+
* `mntRoot` is DERIVED as `<sessionRoot>/mnt`, not read from `mountLayout.mntRoot`: the staged tree can
|
|
332
|
+
* only ever be there (see GUEST_MNT_SEGMENT), so honouring a recorded value that says otherwise emits
|
|
333
|
+
* guest paths pointing at directories no stager creates. A baseline recording a different mnt root is a
|
|
334
|
+
* FIDELITY divergence, surfaced by `recordedLayoutDivergence` at spawn, never a path this builds.
|
|
335
|
+
*
|
|
336
|
+
* Guest paths anchor on `sessionRoot` (the bind target), never on `cwd`: production's own working dir is
|
|
337
|
+
* a folder mount or `outputs` rather than the bare session root, so the two are not interchangeable even
|
|
338
|
+
* though every synced baseline currently records them equal. `cwd` is the agent's working directory and
|
|
339
|
+
* nothing else.
|
|
326
340
|
*/
|
|
327
341
|
export function resolveMounts(baseline, sessionId, projectId = "proj1") {
|
|
328
342
|
const subst = (s) => s.replace("{sessionId}", sessionId).replace("{projectId}", projectId);
|
|
329
343
|
const cwd = subst(baseline.mountLayout.cwd);
|
|
330
344
|
const sessionRoot = subst(baseline.mountLayout.sessionRoot);
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
};
|
|
345
|
+
const mntRoot = `${sessionRoot}/${GUEST_MNT_SEGMENT}`;
|
|
346
|
+
return { cwd, sessionRoot, mntRoot };
|
|
347
|
+
}
|
|
348
|
+
/** Does this baseline RECORD a guest layout the harness cannot stage? Reads the recorded fields only
|
|
349
|
+
* (never the derived ones), so it stays a statement about the data: a `mntRoot` that is not
|
|
350
|
+
* `<sessionRoot>/mnt`, or a `sessionRoot` that already ends in the mnt segment — the shape that made
|
|
351
|
+
* `resolveMounts` return a root one level above the staged tree. Returns the divergence for the caller
|
|
352
|
+
* to surface, or undefined when the recording is reproducible. */
|
|
353
|
+
export function recordedLayoutDivergence(baseline) {
|
|
354
|
+
// Read STRUCTURALLY: a partially-constructed baseline (`{ spawn: {} }`) is normal at the argv seam, and
|
|
355
|
+
// this check must never be the thing that throws there. No recorded layout ⇒ nothing to diverge from.
|
|
356
|
+
const { sessionRoot, mntRoot } = baseline.mountLayout ?? {};
|
|
357
|
+
if (typeof sessionRoot !== "string")
|
|
358
|
+
return undefined;
|
|
359
|
+
const staged = `${sessionRoot}/${GUEST_MNT_SEGMENT}`;
|
|
360
|
+
if (mntRoot !== undefined && mntRoot !== staged)
|
|
361
|
+
return { recorded: mntRoot, staged };
|
|
362
|
+
if (sessionRoot.endsWith(`/${GUEST_MNT_SEGMENT}`))
|
|
363
|
+
return { recorded: sessionRoot, staged };
|
|
364
|
+
return undefined;
|
|
343
365
|
}
|
package/dist/cli.js
CHANGED
|
@@ -490,7 +490,10 @@ const SUBCOMMAND_USAGE = {
|
|
|
490
490
|
" match are the algorithm-derived values rewritten. Anything unprovable is refused, never migrated.\n" +
|
|
491
491
|
" --session <file>: for a MOVED cassette, whose recorded session cannot be resolved from its own\n" +
|
|
492
492
|
" directory. One cassette at a time — each may have been recorded against a different tree, so a\n" +
|
|
493
|
-
" directory batch cannot share one override
|
|
493
|
+
" directory batch cannot share one override.\n" +
|
|
494
|
+
" Exit: 0 = all migrated, or nothing needed migrating. 4 = PARTIAL, some migrated and some could not\n" +
|
|
495
|
+
" (commit what migrated, budget a re-record for the rest). 1 = nothing migrated and at least one\n" +
|
|
496
|
+
" could not. 2 = usage. The JSON envelope carries the same split as migrated/skipped/errors.",
|
|
494
497
|
prune: "usage: prune [--keep-last <n>] [--pinned-older-than <N>d|h|m] [--dry-run] [<runs-dir>] (prune accumulated run dirs; default --keep-last 5)",
|
|
495
498
|
"migrate-run-dir": "usage: migrate-run-dir [<runs-dir>] [--scenario <name>] [--write] [--verbose]\n" +
|
|
496
499
|
" convert pre-layout run dirs (artifacts at the run-dir root) to the per-turn `turns/<N>/` layout, in place.\n" +
|
|
@@ -3766,6 +3769,7 @@ async function cmdVerifyRun(args) {
|
|
|
3766
3769
|
fileToolAttempts: result.fileToolAttempts,
|
|
3767
3770
|
pathDenials: result.pathDenials,
|
|
3768
3771
|
presentedFiles: result.presentedFiles,
|
|
3772
|
+
presentFilesCalls: result.presentFilesCalls,
|
|
3769
3773
|
evidenceErrors: result.evidenceErrors,
|
|
3770
3774
|
effectiveFidelity: result.effectiveFidelity,
|
|
3771
3775
|
// verify-run re-checks a kept run dir on the SAME machine that ran it — grouped with the live
|
package/dist/prompt.js
CHANGED
|
@@ -2,6 +2,7 @@ import { warn } from "./io.js";
|
|
|
2
2
|
import { readFileSync, existsSync } from "node:fs";
|
|
3
3
|
import { join } from "node:path";
|
|
4
4
|
import { fileURLToPath } from "node:url";
|
|
5
|
+
import { resolveMounts } from "./baseline.js";
|
|
5
6
|
const BASELINES_DIR = join(fileURLToPath(new URL("..", import.meta.url)), "baselines");
|
|
6
7
|
/** The {{placeholder}} names renderPrompts() substitutes. KEEP IN LOCKSTEP with the `tokens` map below. */
|
|
7
8
|
export const MODELED_PLACEHOLDER_NAMES = new Set([
|
|
@@ -31,8 +32,10 @@ firstFolderMountPath, opts) {
|
|
|
31
32
|
const spawn = baseline.spawn;
|
|
32
33
|
if (!spawn)
|
|
33
34
|
return {};
|
|
34
|
-
|
|
35
|
-
|
|
35
|
+
// Through resolveMounts, NOT a local `/sessions/<id>` literal: the prompt tells the agent where its
|
|
36
|
+
// files are, so it must name the same tree the runtimes stage and bind. A private derivation here was a
|
|
37
|
+
// fourth copy of this rule, and a fourth place for it to drift.
|
|
38
|
+
const { sessionRoot, mntRoot } = resolveMounts(baseline, sessionId);
|
|
36
39
|
const workspaceFolder = firstFolderMountPath ? `${mntRoot}/${firstFolderMountPath}` : `${mntRoot}/outputs`;
|
|
37
40
|
// {{currentDateTime}}/{{currentTimezone}} are deliberately render-time-impure (wall clock, host
|
|
38
41
|
// TZ) — that's what the real Desktop builder substitutes, and the rendered append never enters a
|
package/dist/redact.js
CHANGED
|
@@ -31,6 +31,46 @@ function csv(v) {
|
|
|
31
31
|
.map((s) => s.trim())
|
|
32
32
|
.filter(Boolean);
|
|
33
33
|
}
|
|
34
|
+
/** Pattern ORDER is load-bearing, and nothing used to say so.
|
|
35
|
+
*
|
|
36
|
+
* `redactText` applies patterns sequentially over the ACCUMULATING output, so an earlier pattern can eat
|
|
37
|
+
* the very text a later pattern's lookahead exists to preserve. The shipped policy relies on this: the
|
|
38
|
+
* `(?=/mnt/)`-anchored rules are ordered ahead of the bare catch-alls precisely so a run-dir path redacts
|
|
39
|
+
* to `[REDACTED:…]/mnt/outputs/f.md` and still resolves. Reverse them and the whole path is eaten:
|
|
40
|
+
*
|
|
41
|
+
* as shipped -> [REDACTED:local-path:…]/mnt/outputs/report.md -> normalizes to `outputs/report.md`
|
|
42
|
+
* reordered -> [REDACTED:local-path:…] -> normalizes to `null`
|
|
43
|
+
*
|
|
44
|
+
* i.e. every `computer_links` structural-marker resolution silently stops working — no error, no finding,
|
|
45
|
+
* just links that no longer resolve on replay. This detects the one shape that causes it: a bare pattern
|
|
46
|
+
* ordered ahead of an otherwise-identical lookahead-anchored one.
|
|
47
|
+
*
|
|
48
|
+
* Deliberately CONSERVATIVE — regex subsumption is undecidable in general, so this fires only when a
|
|
49
|
+
* later pattern's source is exactly an earlier one's plus a trailing lookahead (modulo lazy quantifiers).
|
|
50
|
+
* A miss is a missed warning; a false positive would train authors to ignore it. */
|
|
51
|
+
export function findShadowedPatterns(sources) {
|
|
52
|
+
const norm = (r) => r.replace(/\+\?/g, "+").replace(/\*\?/g, "*");
|
|
53
|
+
const out = [];
|
|
54
|
+
for (let j = 0; j < sources.length; j++) {
|
|
55
|
+
const b = norm(sources[j]);
|
|
56
|
+
for (let i = 0; i < j; i++) {
|
|
57
|
+
const a = norm(sources[i]);
|
|
58
|
+
if (!b.startsWith(a))
|
|
59
|
+
continue;
|
|
60
|
+
// The remainder must be exactly a trailing lookahead. Checked by SHAPE, never by parsing the
|
|
61
|
+
// lookahead's body: an earlier version matched `\(\?=[^()]*\)`, and the `[^()]*` silently skipped
|
|
62
|
+
// every lookahead containing a group — so `(?=/mnt(?:/|$|[\s"'\\)\]]))`, the natural way to write
|
|
63
|
+
// "slash, end, or delimiter" and arguably more correct than a bare `(?=/mnt/)`, went unflagged while
|
|
64
|
+
// being just as dangerous. Reported against a real third-party policy. Paren-balancing here would
|
|
65
|
+
// also have to be escape- and char-class-aware (that same policy carries a `\)` inside a class);
|
|
66
|
+
// not looking inside sidesteps both problems.
|
|
67
|
+
const rest = b.slice(a.length);
|
|
68
|
+
if (rest.length > 3 && rest.startsWith("(?=") && rest.endsWith(")"))
|
|
69
|
+
out.push({ shadowed: j, by: i, base: a });
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
return out;
|
|
73
|
+
}
|
|
34
74
|
/** Assemble a redaction policy from `.cowork-redact.json` (searched in `searchDirs`, e.g. cwd then the
|
|
35
75
|
* scenario/cassette dir) merged with `COWORK_HARNESS_REDACT_PATTERNS`/`_KEYS`. No config + no env →
|
|
36
76
|
* an empty policy (the opt-in default; the scanner is the always-on safety net). A malformed regex
|
|
@@ -60,6 +100,10 @@ export function loadRedactionPolicy(searchDirs) {
|
|
|
60
100
|
patterns.push({ re: new RegExp(src, "g"), label: "redacted" });
|
|
61
101
|
for (const k of csv(process.env.COWORK_HARNESS_REDACT_KEYS))
|
|
62
102
|
keyNames.push(k);
|
|
103
|
+
for (const { shadowed, by, base } of findShadowedPatterns(patterns.map((p) => p.re.source)))
|
|
104
|
+
process.stderr.write(`cowork-harness: WARNING .cowork-redact.json pattern [${by}] (\`${base}\`) is ordered ahead of the ` +
|
|
105
|
+
`lookahead-anchored pattern [${shadowed}], so it matches first and eats the text that lookahead ` +
|
|
106
|
+
`preserves. Move [${shadowed}] before [${by}], or computer:// links will stop resolving on replay.\n`);
|
|
63
107
|
return { patterns, keyNames };
|
|
64
108
|
}
|
|
65
109
|
/** Stable, collision-safe token for a matched span. Depends ONLY on the matched text (context-free), so the
|
package/dist/run/budget.js
CHANGED
|
@@ -96,15 +96,21 @@ export function batchBudgetTracker(maxBudgetUsd, enforceRunningTotal, onWarn = l
|
|
|
96
96
|
* that to size a 24-scenario re-record. */
|
|
97
97
|
export function estimateBatchCost(scenarios) {
|
|
98
98
|
let known = 0;
|
|
99
|
+
let pricedRuns = 0;
|
|
100
|
+
let thinnest;
|
|
99
101
|
const unpriced = [];
|
|
100
102
|
for (const s of scenarios) {
|
|
101
103
|
const worst = worstObservedCost(s);
|
|
102
|
-
if (worst === undefined)
|
|
104
|
+
if (worst === undefined) {
|
|
103
105
|
unpriced.push(s);
|
|
104
|
-
|
|
105
|
-
|
|
106
|
+
continue;
|
|
107
|
+
}
|
|
108
|
+
known += worst;
|
|
109
|
+
const n = pricedRunCount(s);
|
|
110
|
+
pricedRuns += n;
|
|
111
|
+
thinnest = thinnest === undefined ? n : Math.min(thinnest, n);
|
|
106
112
|
}
|
|
107
|
-
return { known, unpriced };
|
|
113
|
+
return { known, unpriced, pricedRuns, thinnest };
|
|
108
114
|
}
|
|
109
115
|
/** The one-line estimate, phrased so a partially-unpriced total can never read as authoritative. An
|
|
110
116
|
* unqualified "$0.00" over a corpus that has never run is worse than no number at all. */
|
|
@@ -113,8 +119,17 @@ export function batchCostEstimateLine(scenarios, est) {
|
|
|
113
119
|
const detail = est.unpriced.length
|
|
114
120
|
? ` (${est.unpriced.length}/${scenarios.length} scenario(s) have no priced run history and contribute $0: ` +
|
|
115
121
|
`${est.unpriced.slice(0, 5).join(", ")}${est.unpriced.length > 5 ? `, +${est.unpriced.length - 5} more` : ""})`
|
|
116
|
-
: `
|
|
117
|
-
|
|
122
|
+
: // The complete case used to read `(all N scenario(s) priced from prior runs)`, which is an active
|
|
123
|
+
// claim of authority — and the one case where this line said nothing qualifying. It is still
|
|
124
|
+
// `sum(max(local history))`: a max over whatever THIS machine ran, so at a new baseline or a new
|
|
125
|
+
// agent binary the history describes a materially different configuration and can UNDER-predict.
|
|
126
|
+
// A consumer wrote "that is the ceiling, not the scope" into a plan off this line and had to retract.
|
|
127
|
+
` (all ${scenarios.length} scenario(s) priced)`;
|
|
128
|
+
const basis = est.thinnest === undefined
|
|
129
|
+
? ""
|
|
130
|
+
: ` — basis: ${est.pricedRuns} prior run(s) on THIS machine, thinnest scenario has ${est.thinnest}; ` +
|
|
131
|
+
`a max over that history, NOT a bound`;
|
|
132
|
+
return `estimated batch cost: $${est.known.toFixed(4)}${bound}${detail}${basis}`;
|
|
118
133
|
}
|
|
119
134
|
/**
|
|
120
135
|
* Cumulative pre-flight for a `record` BATCH: refuse before spending anything if the summed worst-case
|
package/dist/run/cassette.js
CHANGED
|
@@ -908,7 +908,10 @@ export function fingerprintSkillDrift(rec, live) {
|
|
|
908
908
|
* undefined ("can't verify", never a false mismatch) for an inline scenario or when the session file
|
|
909
909
|
* can't be read/parsed from `sessionPath` (resolved against `cassetteDir` exactly like
|
|
910
910
|
* `skillSourceDirs`). Arrays are sorted before hashing so authoring order can't spuriously move the hash. */
|
|
911
|
-
export function buildSessionFingerprint(sessionPath, cassetteDir, override
|
|
911
|
+
export function buildSessionFingerprint(sessionPath, cassetteDir, override,
|
|
912
|
+
/** `omitProjects` reproduces the PRE-`projects` shape. Only `sessionFingerprintDrift` passes it, to tell
|
|
913
|
+
* "this recording predates `projects` coverage" apart from "your session changed" — see there. */
|
|
914
|
+
opts) {
|
|
912
915
|
if (sessionPath === "(inline)")
|
|
913
916
|
return undefined;
|
|
914
917
|
// Same resolver as skillSourceDirs: an override that reached only ONE of them would verify skill
|
|
@@ -949,6 +952,15 @@ export function buildSessionFingerprint(sessionPath, cassetteDir, override) {
|
|
|
949
952
|
// affects only hostloop/protocol replay behavior — see agent_env's doc comment) moves the hash so
|
|
950
953
|
// `verify-cassettes` surfaces the drift instead of a cassette quietly replaying stale env behavior.
|
|
951
954
|
...(Object.keys(agentEnv).length ? { agent_env: agentEnv } : {}),
|
|
955
|
+
// Connected PROJECTS, on the same NON-EMPTY-ONLY terms as `agent_env` above: a session with no
|
|
956
|
+
// `projects:` hashes byte-identically to before, so the overwhelming majority of cassettes do not
|
|
957
|
+
// move. Unlike `agent_env` (a brand-new field nobody had), `projects` is an EXISTING field, so a
|
|
958
|
+
// project-bearing session's hash DOES move once — which is why `sessionFingerprintDrift` reports that
|
|
959
|
+
// case as unverifiable rather than as drift. Omitting it was a false green: swapping which directory
|
|
960
|
+
// is mounted at `.projects/<uuid>` changed the run's inputs and `verify-cassettes` said nothing.
|
|
961
|
+
...(!opts?.omitProjects && cfg.projects.length
|
|
962
|
+
? { projects: [...cfg.projects].map((pr) => ({ uuid: pr.uuid, from: pr.from })).sort((a, b) => a.uuid.localeCompare(b.uuid)) }
|
|
963
|
+
: {}),
|
|
952
964
|
};
|
|
953
965
|
return createHash("sha256")
|
|
954
966
|
.update(Buffer.from(JSON.stringify(shape), "utf8"))
|
|
@@ -1729,6 +1741,7 @@ function minimalRec() {
|
|
|
1729
1741
|
fileToolAttempts: [],
|
|
1730
1742
|
pathDenials: [],
|
|
1731
1743
|
presentedFiles: [],
|
|
1744
|
+
presentFilesCalls: 0,
|
|
1732
1745
|
webSearches: [],
|
|
1733
1746
|
infraErrors: [],
|
|
1734
1747
|
evidenceErrors: { taskTracking: 0, webSearchParse: 0, presentFilesMalformed: 0 },
|
|
@@ -3060,6 +3073,9 @@ export async function cmdRecord(args) {
|
|
|
3060
3073
|
refusals,
|
|
3061
3074
|
estimatedCostUsd: estimate.known,
|
|
3062
3075
|
unpricedScenarios: estimate.unpriced,
|
|
3076
|
+
// The basis for `estimatedCostUsd`, so automation need not treat it as a bound. It is
|
|
3077
|
+
// `sum(max(local run history))` — see `batchCostEstimateLine`.
|
|
3078
|
+
estimateBasis: { source: "local-run-index", pricedRuns: estimate.pricedRuns, thinnestScenarioRuns: estimate.thinnest ?? 0 },
|
|
3063
3079
|
token,
|
|
3064
3080
|
agent: agentPayload,
|
|
3065
3081
|
}));
|
|
@@ -3905,6 +3921,7 @@ function replayErrorResult(file) {
|
|
|
3905
3921
|
fileToolAttempts: undefined, // no rec to read from on this early-bail lane
|
|
3906
3922
|
pathDenials: undefined, // no rec to read from on this early-bail lane
|
|
3907
3923
|
presentedFiles: undefined, // no rec to read from on this early-bail lane
|
|
3924
|
+
presentFilesCalls: undefined, // ditto
|
|
3908
3925
|
preRunPaths: undefined,
|
|
3909
3926
|
preRunLinkAware: undefined,
|
|
3910
3927
|
preRunHashes: undefined,
|
|
@@ -4059,6 +4076,23 @@ export function sessionFingerprintDrift(cassette, cassetteDir, sourceVia, sessio
|
|
|
4059
4076
|
};
|
|
4060
4077
|
if (live === cassette.sessionFingerprint)
|
|
4061
4078
|
return { drifted: false };
|
|
4079
|
+
// A recording made BEFORE `projects` was folded into the shape carries a hash that says nothing about
|
|
4080
|
+
// `projects[]`. Report that honestly as UNVERIFIABLE, not as a migration-with-all-clear: the recorded
|
|
4081
|
+
// value cannot distinguish "the field was simply never covered" from "the project mount changed since",
|
|
4082
|
+
// because there is nothing about `projects` in it to compare against. Claiming "nothing changed" here
|
|
4083
|
+
// would reintroduce, in the remedy, exactly the false green this field's coverage exists to close.
|
|
4084
|
+
//
|
|
4085
|
+
// The test is an EXACT match on the pre-`projects` shape, which is also why it can run ahead of the
|
|
4086
|
+
// name-lookup downgrade below: a byte-equal legacy hash is strong evidence the resolved file really is
|
|
4087
|
+
// this cassette's session (an unrelated same-named sibling matching exactly is not a coincidence worth
|
|
4088
|
+
// hedging), so the more specific and more actionable message wins.
|
|
4089
|
+
const legacy = buildSessionFingerprint(cassette.scenario.session, cassetteDir, sessionOverride, { omitProjects: true });
|
|
4090
|
+
if (legacy !== undefined && legacy === cassette.sessionFingerprint)
|
|
4091
|
+
return {
|
|
4092
|
+
drifted: false,
|
|
4093
|
+
unverifiable: true,
|
|
4094
|
+
note: "session-fingerprint: this cassette was recorded before `projects` was part of the session shape, so its hash covers everything EXCEPT `projects[]` — and everything it does cover matches. A change to a project mount since record time is therefore not detectable here. Re-record to gain that coverage.",
|
|
4095
|
+
};
|
|
4062
4096
|
// Only "name-lookup" (a scenario WAS found, but not at its persisted/expected offset — the SAME
|
|
4063
4097
|
// low-confidence signal scenarioContentDrift downgrades on) is grounds to distrust this mismatch.
|
|
4064
4098
|
// "none" means "nothing to compare the layout against" (mirrors scenarioContentDrift's own `!src.path`
|
|
@@ -5009,7 +5043,7 @@ export async function cmdVerifyCassettes(args) {
|
|
|
5009
5043
|
const sourceVia = rr.persistedMissing ? "name-lookup" : "none";
|
|
5010
5044
|
const sfd = sessionFingerprintDrift(rc.cassette, dirname(f), sourceVia, vcSessionOverride);
|
|
5011
5045
|
if (sfd.drifted)
|
|
5012
|
-
staleness.push("session-shape fingerprint differs from the current session file (connected folders/
|
|
5046
|
+
staleness.push("session-shape fingerprint differs from the current session file (connected folders/plugins/skills/mcp/egress/web_fetch config changed since record; projects and agent_env are hashed only when set) — re-record");
|
|
5013
5047
|
if (sfd.note)
|
|
5014
5048
|
notes.push(sfd.note);
|
|
5015
5049
|
}
|
|
@@ -5452,26 +5486,50 @@ export function cmdRehash(args) {
|
|
|
5452
5486
|
const migrated = results.filter((r) => r.action === "migrated").length;
|
|
5453
5487
|
const skipped = results.filter((r) => r.action === "skipped").length;
|
|
5454
5488
|
const errors = results.filter((r) => r.action === "error").length;
|
|
5489
|
+
const code = rehashExitCode({ migrated, errors });
|
|
5455
5490
|
if (asJson) {
|
|
5456
5491
|
out(jsonPayloadEnvelope("rehash", errors === 0, { dryRun, migrated, skipped, errors, results }));
|
|
5457
5492
|
}
|
|
5458
5493
|
else {
|
|
5459
|
-
|
|
5460
|
-
|
|
5461
|
-
|
|
5462
|
-
}
|
|
5494
|
+
// Summary FIRST. It used to print last, after every per-file line had scrolled past — and these two
|
|
5495
|
+
// counts ARE the decision, so a reader had to scroll back up to find them. The per-file lines below
|
|
5496
|
+
// are the detail behind it, not the thing being reported.
|
|
5463
5497
|
if (migrated > 0 || errors > 0) {
|
|
5464
|
-
|
|
5465
|
-
|
|
5466
|
-
|
|
5467
|
-
|
|
5468
|
-
|
|
5498
|
+
const glyph = errors > 0 ? (migrated > 0 ? "!" : "✗") : "✓";
|
|
5499
|
+
const verb = dryRun ? "migratable" : "migrated";
|
|
5500
|
+
// No single target version here — P8 stamps per-scenario (requiredVersionFor), so a batch can
|
|
5501
|
+
// migrate some cassettes to v10 and others to v11; each row's own reason already says which.
|
|
5502
|
+
log(`${glyph} rehash: ${migrated} ${verb}` +
|
|
5503
|
+
(errors > 0 ? `, ${errors} require a re-record` : "") +
|
|
5504
|
+
(skipped > 0 ? `, ${skipped} already current` : "") +
|
|
5505
|
+
(dryRun ? " (dry-run — nothing written)" : ""));
|
|
5469
5506
|
}
|
|
5470
5507
|
else {
|
|
5471
5508
|
log("✓ rehash: nothing to migrate");
|
|
5472
5509
|
}
|
|
5510
|
+
for (const r of results) {
|
|
5511
|
+
const glyph = r.action === "migrated" ? "✓" : r.action === "error" ? "✗" : "·";
|
|
5512
|
+
log(`${glyph} ${r.file}: ${r.reason}`);
|
|
5513
|
+
}
|
|
5473
5514
|
}
|
|
5474
|
-
return process.exit(
|
|
5515
|
+
return process.exit(code);
|
|
5516
|
+
}
|
|
5517
|
+
/** `rehash`'s exit code, documented in SPEC §11 and in its `--help`.
|
|
5518
|
+
*
|
|
5519
|
+
* PARTIAL success gets its own code. `4 migrated, 18 failed` and `0 migrated, 22 failed` demand opposite
|
|
5520
|
+
* responses — commit the four and budget a re-record for the rest, versus nothing here is salvageable —
|
|
5521
|
+
* and while both were `1`, a shell consumer reading only the exit code could not tell them apart. The
|
|
5522
|
+
* JSON envelope always carried the split as `migrated`/`skipped`/`errors`; a bare terminal run did not.
|
|
5523
|
+
*
|
|
5524
|
+
* `4` rather than `3`: the exit-code space is per-command, and `3`'s "could not verify" meaning is
|
|
5525
|
+
* load-bearing on `verify-cassettes` — a migration that partly succeeded is not a failed verification.
|
|
5526
|
+
*
|
|
5527
|
+
* `skipped` is deliberately not an input: an already-current cassette needs no action, so it cannot
|
|
5528
|
+
* turn a clean run dirty or a total failure partial. */
|
|
5529
|
+
export function rehashExitCode(tally) {
|
|
5530
|
+
if (tally.errors === 0)
|
|
5531
|
+
return 0;
|
|
5532
|
+
return tally.migrated > 0 ? 4 : 1;
|
|
5475
5533
|
}
|
|
5476
5534
|
/** Record-time connected-folder host-path -> resolved-mount-name map (Finding 24), persisted onto a
|
|
5477
5535
|
* v9+ cassette's `folderPrefixMap`. Zips `recordRoots` (minus the leading `outputs`) against
|
|
@@ -5564,8 +5622,10 @@ export const ALWAYS_CONTENT_KEYS = [
|
|
|
5564
5622
|
"task_status",
|
|
5565
5623
|
"result",
|
|
5566
5624
|
// content-class, NOT controlOut-gated: both the present_files tool_use and its own tool_result live
|
|
5567
|
-
// in the ordinary events stream, so the re-drive reproduces
|
|
5568
|
-
//
|
|
5625
|
+
// in the ordinary events stream, so the re-drive reproduces the present_files signals like the other
|
|
5626
|
+
// re-derived ones above (skill_triggered, redundantToolCalls, …). `presentFilesCalls` (what
|
|
5627
|
+
// present_files_called reads) reproduces on every tier; `presentedFiles`' promoted/leaked booleans
|
|
5628
|
+
// reproduce at container, where cwd is the session root — no_scratchpad_leak's only tier.
|
|
5569
5629
|
"no_scratchpad_leak",
|
|
5570
5630
|
"present_files_called",
|
|
5571
5631
|
// content-class, NOT controlOut-gated: fileToolAttempts re-derives from frozen tool_use blocks (the
|
|
@@ -6080,6 +6140,7 @@ export async function replayCassette(cassette, hooks = [], opts = {}) {
|
|
|
6080
6140
|
// empty [] (nothing presented) vacuous-passes no_scratchpad_leak instead of reading as
|
|
6081
6141
|
// evidence-unavailable.
|
|
6082
6142
|
presentedFiles: rec.presentedFiles,
|
|
6143
|
+
presentFilesCalls: rec.presentFilesCalls,
|
|
6083
6144
|
evidenceErrors: rec.evidenceErrors,
|
|
6084
6145
|
effectiveFidelity: cassette.effectiveFidelity,
|
|
6085
6146
|
// Replay has no live filesystem — computer_links_resolve normalizes both link shapes against the
|
|
@@ -6437,6 +6498,7 @@ export async function replayCassette(cassette, hooks = [], opts = {}) {
|
|
|
6437
6498
|
// NOT reproduce — this one genuinely re-derives. Uncollapsed (an empty [] is the real "nothing
|
|
6438
6499
|
// presented" signal no_scratchpad_leak's vacuous pass needs, matching live).
|
|
6439
6500
|
presentedFiles: rec.presentedFiles,
|
|
6501
|
+
presentFilesCalls: rec.presentFilesCalls,
|
|
6440
6502
|
preRunPaths: undefined,
|
|
6441
6503
|
// Report the baseline semantics actually used during evaluation above (not undefined) so the returned
|
|
6442
6504
|
// result doesn't misrepresent them. Same source of truth as the evaluate() ctx.
|
package/dist/run/chat-result.js
CHANGED
|
@@ -113,6 +113,7 @@ export function buildChatResult(record, opts) {
|
|
|
113
113
|
fileToolAttempts: record.fileToolAttempts,
|
|
114
114
|
pathDenials: record.pathDenials,
|
|
115
115
|
presentedFiles: record.presentedFiles,
|
|
116
|
+
presentFilesCalls: record.presentFilesCalls,
|
|
116
117
|
egress: opts.egress,
|
|
117
118
|
resources,
|
|
118
119
|
stderrLogPath: join(opts.outDir, "agent.stderr.log"),
|