@surlymochan/evalid 0.3.1 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -3,6 +3,34 @@
3
3
  All notable changes to this project are documented here. Detailed notes live in
4
4
  [RELEASE_NOTES.md](RELEASE_NOTES.md).
5
5
 
6
+ ## [Unreleased]
7
+
8
+ ## [0.5.0] - 2026-11-06
9
+
10
+ ### Added
11
+ - **CR review seal partition (issue #5, P1-2)**: the J6 review proof (`cr/review-0.json`) now splits the changed files into `sealed_unchanged` (bytes still identical to the green TDD proof's sealed hashes) and `seal_drifted`, with a `reread_count`. The yolo skill's verify-consistency guidance uses the partition so the review pass re-reads only drifted files instead of everything the gate already verified. Reporting only — verdicts are unchanged, and without a green seal the partition is `null` and everything is re-read.
12
+ - **Bench Verified set extended to 15 instances (issue #5, P2-2)**: `bench/swebench/instances.json` grows from 12 to 15, adding sympy-15599, sympy-19040 and sympy-24213, and a structural validation of the instance set (`bench/swebench/test_instances.py`) now runs in CI. Gold-patch checks for the three new instances stay pending the benchmark environment.
13
+ - **Bench report aggregates median (IQR) (issue #5, P2-1)**: `bench/swebench/report.py` now aggregates per-arm wall-clock, token and cost results with q1/median/q3 and presents them as median (IQR); reps isolation and the aggregation itself are pinned by `bench/swebench/test_reps_report.py`. Live multi-rep runs stay pending the benchmark environment.
14
+ - **P0-2 acceptance pinned against saved pre-fix patches (issue #5, P0-2)**: regression tests in `bench/swebench/test_diff_filter.py` run the saved pre-fix patches through the evidence filter (`specs/`, `.xflow/`, `.as-xflow/` excluded from solution patches). The saved 9.7 MB evidence-laden patch filters down to 4.3 KB, and every saved patch lands under the 200 KB acceptance line.
15
+
16
+ ## [0.4.1] - 2026-11-06
17
+
18
+ ### Fixed
19
+ - **TDD gate counts skipped/todo tests again on Node 24**: the gate parsed skip/todo counts only from the TAP reporter's `# skipped / # todo` summary lines, but Node 24's built-in runner defaults to the spec reporter when piped, whose summary lines carry a `ℹ` marker — an all-skipped/todo run exited 0, parsed as zero skips, and was waved through as green. The parser now reads the runner's summary block in both reporter formats (anchored to the summary lines, gated on the full node signature), and fails closed: a node summary whose gate counts cannot be fully parsed classifies as `counts_unknown` and is rejected by proof validation (`tests_counts_unknown`) instead of reading as zero skipped.
20
+
21
+ ### Added
22
+ - **Phase cost evidence on every run (issue #5 P1-1)**: the per-phase timing/token report (`src/core/phase-cost.js`, landed in 0.3.0) now also rides on the `workflow_failed` execution-log event — a run that stops mid-way still reports where its time and tokens went — and is persisted as a standalone artifact at `.as-xflow/phase-cost.json` on both completion and failure, so the breakdown is readable without mining the ndjson execution log. Acceptance rerun on the SWE-bench cells stays pending the benchmark environment; the accounting structure (phase sum vs wall clock within 5%) is covered by `test/phase-cost.test.js`.
23
+
24
+ ## [0.4.0] - 2026-11-04
25
+
26
+ ### Changed
27
+ - **Overhead cut 1 (v0.4 F2, issue #5 P1-2/P1-4)**: the lite (yolo) track seals the TDD red proof immediately after `plan` — the issue/branch ceremony no longer sits between planning and red-proof, and the planning pass now authors the failing test, so plan and red share one agent pass. `xflow workflow run --title "..."` also derives the change id from the title when the branch cannot provide one (previously a hard stop on `main` that pushed agents into manual atom-by-atom driving). TDD snapshots are scoped: inside a git work tree I6a seals only the files that differ from HEAD plus the files the test command names, and I6b materializes the rest from the recorded HEAD at replay — on the SWE-bench sympy tree the sealed manifest drops from 1,543 files / 164 KB to the changed files only (2 files / 197 B for a one-file change), and the snapshot copy from 26.9 MB to ~51 KB. Proofs sealed before this change keep verifying (the scope joins the binding only when present).
28
+
29
+ ### Added
30
+ - **CR gate `J6.cr.review` wired into the verify stage (v0.4 F1)**: closes the v0.3.0 known gap where J6 was registered and unit-tested but no workflow phase invoked it. The yolo `verify-consistency` phase and the corps `qa` phase now run J6 on every change — after the spec-delta review, before the stage flips to verify — and declare its verdict record `specs/changes/<change_id>/cr/review-0.json` as a required artifact. A failing verdict (spec misalignment, changed code with no test follow, net assertion decrease, blown loop budget) stops the workflow at verify instead of waving the change through to archive. Orchestration evidence: `test/cr-review-wiring.test.js` drives the real yolo verify phase through `WorkflowExecutor` against a git fixture — an aligned, tested change passes with a verdict record, and an untested change is stopped at the phase.
31
+ - **SWE-bench predictions emitted directly by `run.py` (v0.4 F3)**: the bench harness no longer relies on a one-off converter outside the repo. A stdlib-only `bench/swebench/harness/predictions.py` builds records with exactly `instance_id`, `model_name_or_path`, `model_patch`, re-applies the P0-2 evidence filter (`specs/`, `.xflow/`, `.as-xflow/`) on both paths, and upserts per-arm files so resumable reruns never duplicate lines; `run.py --export-predictions` rebuilds the same files from saved patches with no model calls. Regression: all four pilot arms rebuilt through the new path are byte-identical to the hand-converted pilot files (`cmp` = 0 across 24 instances), and the largest openflow-arm patch stays at 9,650 bytes, well under the 200 KB line.
32
+ - **`xflow init` writes the project `.gitignore` (v0.4 F4)**: init now creates or merges `.gitignore` entries for evalid local state (`.as-xflow/`, `specs/changes/`), appending only the entries that are missing and never modifying existing lines; `.xflow/` stays tracked on purpose (`GOAL.md` is a reviewable project document). Re-running `init --force` tops up projects initialized before v0.4. Closes the v0.3.0 known gap.
33
+
6
34
  ## [0.3.1] - 2026-11-04
7
35
 
8
36
  ### Changed
package/README.md CHANGED
@@ -22,12 +22,12 @@ Underneath, a workflow engine (spec → implement → verify loops with per-phas
22
22
 
23
23
  _Renamed from OpenFlow in v0.3.1 (2026-11-04); old repository and package links redirect._
24
24
 
25
- ## Benchmarks: OpenFlow vs superpowers (2026-10-17)
25
+ ## Benchmarks: evalid vs superpowers (2026-10-17)
26
26
 
27
- _These results were recorded when the project was named OpenFlow; the version labels in this section are kept exactly as measured._
27
+ _These results were measured before the rename, when the project was named OpenFlow; product names in this section have been unified to the current names (evalid, swrm), and all versions and figures are the original measurements._
28
28
 
29
29
  For the synthetic and evidence-check results below, the same model (`grok-4.7-high` via local `cursor-agent`) and same
30
- harness are used for every arm; the SWE-bench block uses DeepSeek via Pi. Pinned: OpenFlow 0.2.1,
30
+ harness are used for every arm; the SWE-bench block uses DeepSeek via Pi. Pinned: evalid 0.2.1,
31
31
  [superpowers](https://github.com/obra/superpowers) v7.0.0 (`bb92a77`). Scripts, raw results and methodology:
32
32
  [`bench/`](bench/README.md) and [`bench/e2e/`](bench/e2e/README.md). Losses are shown as measured.
33
33
 
@@ -40,11 +40,11 @@ DeepSeek-V4.1-Flash (`deepseek-flash`), n=1 per cell, 20-minute cap; details and
40
40
  | --- | --- | --- | --- | --- | --- |
41
41
  | Plain agent (baseline) | 3/6 (50%) | 124 s (110–175) | $0.0234 | $0.16 | 6 |
42
42
  | superpowers v7.0.0 | 2/6 (33%) | 154 s (136–340) | $0.0285 | $0.31 | 6 |
43
- | OpenFlow 0.2.1 | 4/6 (67%) | 230 s (200–266) | $0.0412 | $0.23 | 6 |
44
- | openswarm (Pi executor) | 5/6 (83%) | 228 s (187–254) | $0.0377 | $0.21 | 6 |
43
+ | evalid 0.2.1 | 4/6 (67%) | 230 s (200–266) | $0.0412 | $0.23 | 6 |
44
+ | swrm (Pi executor) | 5/6 (83%) | 228 s (187–254) | $0.0377 | $0.21 | 6 |
45
45
 
46
46
  **Caveats:** n=1 on 6 instances, so one or two resolves are within noise. The two workflow tools took about 1.8x the
47
- baseline's median wall-clock and cost 76% (evalid) / 61% (openswarm) more per task. The first published table undercounted
47
+ baseline's median wall-clock and cost 76% (evalid) / 61% (swrm) more per task. The first published table undercounted
48
48
  pytest because of a shared temp-dir race in grading; it is now fixed and regraded
49
49
  ([regrade notes](bench/swebench/README.md#regrade-2026-10-18)). On the MiniMax-M3.1-Flash-Preview subset, baseline
50
50
  resolved 3/4 and superpowers 2/4; evalid had no clean MiniMax run because the plan's usage cap was hit.
@@ -58,7 +58,7 @@ results: [#5](https://github.com/surlymochan/evalid/issues/5).
58
58
  | --- | --- | --- | --- | --- |
59
59
  | Plain agent (baseline) | 24/24 (100%) | 55 s (46–132) | 105k / 28k | 24 |
60
60
  | superpowers v7.0.0 | 24/24 (100%) | 187 s (121–299) | 312k / 59k | 24 |
61
- | OpenFlow 0.2.1 | 11/12 (92%) | 985 s (920–1195) | 5.29M / 233k ¹ | 12 ² |
61
+ | evalid 0.2.1 | 11/12 (92%) | 985 s (920–1195) | 5.29M / 233k ¹ | 12 ² |
62
62
 
63
63
  On tasks this small the plain agent already succeeds every time, and evalid's full ceremony is the slowest and most
64
64
  token-hungry arm. 3 of 12 evalid runs hit the 20-minute cap while still running the workflow; their code already passed.
@@ -70,10 +70,10 @@ proofs, skipped tests, weak or wrong assertions, contradictory specs, …) and 1
70
70
 
71
71
  | Checker | Faulty caught | Honest wrongly rejected | Cost per check |
72
72
  | --- | --- | --- | --- |
73
- | OpenFlow 0.2.1 gates (deterministic code) | 22/25 (88%): original 16/17, held-out 6/8 | 0/11 | ~0.1–0.2 s, no model tokens |
73
+ | evalid 0.2.1 gates (deterministic code) | 22/25 (88%): original 16/17, held-out 6/8 | 0/11 | ~0.1–0.2 s, no model tokens |
74
74
  | superpowers v7.0.0 skills + LLM judge | 25/25 (100%) | 1/11 | one agent call per check |
75
75
  | LLM judge, no skills | 25/25 (100%) ³ | 0/11 | one agent call per check |
76
- | OpenFlow 0.2.0 (before this benchmark, original 24 only) | 10/17 (59%) | 3/7 | – |
76
+ | evalid 0.2.0 (before this benchmark, original 24 only) | 10/17 (59%) | 3/7 | – |
77
77
 
78
78
  An LLM reviewer that is explicitly asked to audit an episode, and is allowed to rerun the tests, caught more than evalid's static gates did.
79
79
  evalid's gates are deterministic and cost no model calls, but they still miss plausible-but-wrong expected values,
package/RELEASE_NOTES.md CHANGED
@@ -1,5 +1,41 @@
1
1
  # Release Notes
2
2
 
3
+ ## 0.5.0 - Review re-reads only drifted files; the bench set grows to 15
4
+
5
+ Minor release. One product change — the CR review now spends its re-reads only where the seal says bytes moved — plus three bench-hygiene landings from the issue #5 backlog, each pinned by tests rather than by a live run.
6
+
7
+ **Review re-reads only drifted files (#32, issue #5 P1-2).** The J6 review proof (`cr/review-0.json`) now carries a seal partition: every changed file is classified `sealed_unchanged` (bytes still identical to the green TDD proof's sealed hashes) or `seal_drifted`, with a `reread_count`. The yolo verify-consistency guidance consumes the partition, so the review pass re-reads only the drifted files instead of everything the gate already verified. This is reporting only — verdicts are unchanged, and without a green seal the partition is `null` and everything is re-read, exactly as before.
8
+
9
+ **The Verified bench set grows from 12 to 15 instances (#30, issue #5 P2-2).** `bench/swebench/instances.json` adds sympy-15599, sympy-19040 and sympy-24213, and a structural validation of the instance set (`bench/swebench/test_instances.py`) now runs in CI, so a malformed instance entry fails the build instead of a bench run. Gold-patch checks for the three new instances are pending the benchmark environment.
10
+
11
+ **Bench reports now aggregate as median (IQR) (#31, issue #5 P2-1).** `bench/swebench/report.py` aggregates per-arm wall-clock, token and cost results with q1/median/q3 and presents them in median (IQR) form. Reps isolation — repeat runs of the same cell never share grading state — and the aggregation itself are pinned by `bench/swebench/test_reps_report.py`.
12
+
13
+ **P0-2 acceptance is now pinned by regression tests (#29, issue #5 P0-2).** The evidence exclusion itself landed earlier (solution patches drop `specs/`, `.xflow/` and `.as-xflow/`); what was missing was a test that would notice if it regressed. `bench/swebench/test_diff_filter.py` now runs the saved pre-fix patches through the filter: the saved 9.7 MB evidence-laden patch comes out at 4.3 KB, and every saved patch lands under the 200 KB acceptance line.
14
+
15
+ Acceptance debt, stated plainly: the live-run acceptance for P1-2, P2-1 and P2-2 — multi-rep bench runs on the restored environment, including the wall-clock/token lines from issue #5 and gold checks for the new instances — is still pending the benchmark environment. No performance numbers are claimed in this release; what shipped is the reporting, the larger instance set, and the tests that pin their behaviour.
16
+
17
+ ## 0.4.1 - Failed runs report phase costs; Node 24 test counts fail closed
18
+
19
+ Patch release. Two fixes, both about the gate trusting evidence it had not actually read.
20
+
21
+ **Failed runs now report where the time and tokens went (#22).** The per-phase cost report landed in 0.3.0 but only rode on successful runs. It now also rides on the `workflow_failed` execution-log event — a run that stops mid-way still reports its per-phase timing/token breakdown — and is persisted as a standalone artifact at `.as-xflow/phase-cost.json` on both completion and failure, so the breakdown is readable without mining the ndjson execution log. The accounting structure (phase sum vs wall clock within 5%) is covered by `test/phase-cost.test.js`. This lands item P1-1 from issue #5.
22
+
23
+ **TDD gate reads test counts correctly on Node 24, and fails closed when it cannot (#23).** The gate parsed skip/todo counts only from the TAP reporter's summary lines. Node 24's built-in runner defaults to the spec reporter when piped, whose summary lines carry a different marker — an all-skipped/todo run on Node 24 exited 0, parsed as zero skips, and was waved through as green. The parser now reads the runner's summary block in both reporter formats (TAP and spec), requires the full node signature before treating a block as a node summary, and anchors summary lines at line start so test names or program output cannot forge counts. When a detected node summary cannot be fully parsed, the run classifies as `counts_unknown` and proof validation rejects it (`tests_counts_unknown`) instead of reading unknown counts as zero. Measured against pinned captured outputs: the new count-parsing tests pass 9/9 under Node 22 and 9/9 under Node 24; the end-to-end TDD proof suite passes 18/18 under both (it was 17/18 on Node 24 before this fix).
24
+
25
+ With P1-1 landed here, the P0/P1 items tracked on the optimization backlog in #5 stand checked off as verified; the remaining longer-horizon items stay tracked there.
26
+
27
+ ## 0.4.0 - CR gate wired in, overhead cut 1, predictions direct
28
+
29
+ **The CR gate now runs inside the pipeline (F1, #15).** In 0.3.0 `J6.cr.review` was registered and unit-tested but no workflow invoked it — the release notes said so, and this release closes that gap. The yolo `verify-consistency` phase and the corps `qa` phase run J6 on every change, after the spec-delta review and before the stage flips to verify, and the verdict record `cr/review-0.json` is a required artifact: a failing verdict stops the workflow instead of waving the change through to archive. The wiring is proven end-to-end, not just by unit tests: `test/cr-review-wiring.test.js` drives the real yolo verify phase through `WorkflowExecutor` against a git fixture — an aligned, tested change passes with a verdict record on disk, and a code change with no test follow is stopped at the phase with the `cr_code_without_test` finding.
30
+
31
+ **Overhead cut 1 (F2, #17).** The lite (yolo) track seals the TDD red proof immediately after `plan` — the issue/branch ceremony no longer sits between planning and red-proof, and the planning pass authors the failing test, so plan and red share one agent pass. `xflow workflow run --title "..."` derives the change id from the title when the branch cannot provide one. TDD snapshots are scoped: inside a git work tree, I6a seals only the files that differ from HEAD plus the files the test command names, and I6b materializes the rest from the recorded HEAD at replay. Offline measurements on the SWE-bench sympy tree: the sealed manifest drops from 1,543 files / 164 KB to the changed files only (2 files / 197 B for a one-file change), and the snapshot copy from 26.9 MB to ~51 KB. Proofs sealed before this change keep verifying.
32
+
33
+ Acceptance debt, stated plainly: the v0.4 acceptance line for F2 was a live rerun of the same 6 SWE-bench instances — wall-clock ≤ 1.3× the baseline median, total tokens ≤ 2.0M, resolved ≥ 4/6. The bench generation environment is not currently restored, so those three numbers are still owed; they are not claimed here. If the rerun misses the line when it lands, the remaining overhead work continues as v0.4.1 rather than being declared done.
34
+
35
+ **Predictions come straight out of `run.py` (F3, #16).** The official SWE-bench predictions format (`instance_id`, `model_name_or_path`, `model_patch`) is now produced by the bench harness itself, with the P0-2 evidence filter re-applied on both the live and replay paths — the one-off converter outside the repo is retired. Regression evidence: all four pilot arms rebuilt through `run.py --export-predictions` are byte-identical to the hand-converted pilot files (`cmp` = 0, 24 instances).
36
+
37
+ **`xflow init` writes `.gitignore` (F4, #18).** Init now creates or merges the project `.gitignore` for evalid local state (`.as-xflow/`, `specs/changes/`), appending only missing entries and never modifying existing lines — the first line of defense ahead of the diff filtering added in #8. This closes the second v0.3.0 known gap; both are now done.
38
+
3
39
  ## 0.3.1 - Renamed: OpenFlow → evalid
4
40
 
5
41
  Same code as 0.3.0, new name and positioning. The project is now **evalid** — a runtime verification layer for coding agents: it seals each test run and replays it from a clean snapshot, so a claim only ships if the evidence replays green. Repository: `surlymochan/evalid` (old `openflow` links redirect); npm package: `evalid`. The `xflow` CLI name is unchanged in this release.
package/bin/xflow.js CHANGED
@@ -5079,15 +5079,55 @@ async function handleInit(args) {
5079
5079
  mkdirSync(configDir, { recursive: true });
5080
5080
  writeFileSync(configPath, `${JSON.stringify(config, null, 2)}\n`, 'utf8');
5081
5081
 
5082
+ const gitignore = ensureXflowGitignore(projectRoot);
5083
+
5082
5084
  if (parsedArgs.json) {
5083
- console.log(JSON.stringify({ ok: true, config_path: configPath, config }, null, 2));
5085
+ console.log(JSON.stringify({ ok: true, config_path: configPath, config, gitignore }, null, 2));
5084
5086
  return;
5085
5087
  }
5086
5088
 
5087
5089
  console.log(`xflow init: wrote ${configPath}`);
5090
+ if (gitignore.created) {
5091
+ console.log(`xflow init: wrote ${gitignore.path} (ignores ${gitignore.added.join(', ')})`);
5092
+ } else if (gitignore.added.length > 0) {
5093
+ console.log(`xflow init: updated ${gitignore.path} (+ ${gitignore.added.join(', ')})`);
5094
+ } else {
5095
+ console.log(`xflow init: ${gitignore.path} already covers evalid local state`);
5096
+ }
5088
5097
  console.log('Next: xflow doctor --project-root <path>');
5089
5098
  }
5090
5099
 
5100
+ // Local runtime state an evalid project should never commit: the .as-xflow
5101
+ // state dir (config, sqlite state, logs, pending gates/tasks, snapshots) and
5102
+ // per-change spec artifacts (proofs, status files). These mirror the entries
5103
+ // this repository keeps in its own .gitignore.
5104
+ const XFLOW_GITIGNORE_ENTRIES = ['.as-xflow/', 'specs/changes/'];
5105
+ const XFLOW_GITIGNORE_HEADER = '# evalid local state (added by xflow init)';
5106
+
5107
+ function normalizeGitignoreEntry(line) {
5108
+ return line.trim().replace(/^\//, '').replace(/\/$/, '');
5109
+ }
5110
+
5111
+ function ensureXflowGitignore(projectRoot) {
5112
+ const gitignorePath = resolve(projectRoot, '.gitignore');
5113
+
5114
+ if (!existsSync(gitignorePath)) {
5115
+ writeFileSync(gitignorePath, `${XFLOW_GITIGNORE_HEADER}\n${XFLOW_GITIGNORE_ENTRIES.join('\n')}\n`, 'utf8');
5116
+ return { path: gitignorePath, created: true, added: [...XFLOW_GITIGNORE_ENTRIES] };
5117
+ }
5118
+
5119
+ const original = readFileSync(gitignorePath, 'utf8');
5120
+ const present = new Set(original.split('\n').map(normalizeGitignoreEntry));
5121
+ const missing = XFLOW_GITIGNORE_ENTRIES.filter((entry) => !present.has(normalizeGitignoreEntry(entry)));
5122
+ if (missing.length === 0) {
5123
+ return { path: gitignorePath, created: false, added: [] };
5124
+ }
5125
+
5126
+ const separator = original.length === 0 ? '' : original.endsWith('\n') ? '\n' : '\n\n';
5127
+ writeFileSync(gitignorePath, `${original}${separator}${XFLOW_GITIGNORE_HEADER}\n${missing.join('\n')}\n`, 'utf8');
5128
+ return { path: gitignorePath, created: false, added: missing };
5129
+ }
5130
+
5091
5131
  // ─── goal ────────────────────────────────────────────────────────────────────
5092
5132
 
5093
5133
  async function handleGoal(sub, args) {
@@ -5421,7 +5461,17 @@ async function handleWorkflow(sub, args) {
5421
5461
  }
5422
5462
  const isDryRun = args.includes('--dry-run');
5423
5463
  const input = buildWorkflowInput(parsedArgs);
5424
- const changeId = resolveChangeId({ explicit: parsedArgs.change_id, projectRoot });
5464
+ let changeId = resolveChangeId({ explicit: parsedArgs.change_id, projectRoot });
5465
+ if (!changeId && input.title) {
5466
+ // Lite-track turn economy (issue #5 P1-2): the documented invocation
5467
+ // is `workflow run yolo --title "..."`, which dead-ended on branches
5468
+ // like `main` that cannot derive a change id and pushed agents into
5469
+ // manual atom-by-atom driving. Derive the id from the title instead.
5470
+ changeId = normalizeChangeId(input.title);
5471
+ if (changeId) {
5472
+ console.log(`Derived --change-id "${changeId}" from --title (pass --change-id to override).`);
5473
+ }
5474
+ }
5425
5475
  ensureChangeId(changeId, 'workflow run', projectRoot);
5426
5476
  const { load, dryRun } = await import('../src/core/workflow-loader.js');
5427
5477
  if (isDryRun) {
@@ -6667,7 +6717,10 @@ function printInitHelp() {
6667
6717
  Usage: xflow init [--project-root <path>] [--force] [--json]
6668
6718
 
6669
6719
  Create a project-local .as-xflow/config.json with portable defaults for
6670
- workflow paths, skill sync paths, and issue routing.
6720
+ workflow paths, skill sync paths, and issue routing. Also creates or updates
6721
+ the project .gitignore so evalid local state (.as-xflow/, specs/changes/)
6722
+ stays out of version control; existing entries are never modified and
6723
+ re-running init adds nothing twice.
6671
6724
 
6672
6725
  Options:
6673
6726
  --project-root PATH Initialize a specific project root
@@ -1,6 +1,6 @@
1
1
  # evalid Launch Dossier
2
2
 
3
- Generated: 2026-11-03T18:52:08.690Z
3
+ Generated: 2026-11-05T19:40:37.700Z
4
4
  Mode: post_publish
5
5
  Verdict: conditionally ready: engine and adoption evidence are strong, registry publication remains required for a splash launch
6
6
  Launch ready: not yet
@@ -117,7 +117,7 @@ Evidence:
117
117
  - Adoption records: 1; status: pass
118
118
  - Goal audit: pass; goal: Keep evalid credible as an open-source agent delivery runtime: repo-owned workflow contracts, executable verification gates, durable project context, and honest benchmark evidence.
119
119
  - Package evidence: missing; source: registry_json
120
- - Package name/version: @surlymochan/evalid@0.3.1
120
+ - Package name/version: @surlymochan/evalid@0.5.0
121
121
 
122
122
  ## Passing Evidence
123
123
 
@@ -6,6 +6,19 @@ Use this page for the final irreversible public publish step. Everything here
6
6
  is intentionally explicit because `npm publish --access public` changes public
7
7
  distribution state.
8
8
 
9
+ ## Automated publish (trusted publishing)
10
+
11
+ The registry-mutating step normally runs from GitHub Actions, not from a
12
+ local shell: `.github/workflows/npm-publish.yml` publishes via npm trusted
13
+ publishing (OIDC — no stored npm token anywhere) whenever a GitHub Release
14
+ is published, and can also be started manually (workflow_dispatch) to
15
+ publish the version in `package.json` if that version is not on npm yet.
16
+ Before publishing it runs the same `npm run release:pack` gate as the manual
17
+ sequence below, and for release events it refuses to publish when the
18
+ release tag does not match the `package.json` version. The manual sequence
19
+ below remains the fallback for when Actions or the trusted-publisher
20
+ registration on npmjs.com is unavailable.
21
+
9
22
  ## Current Verified State
10
23
 
11
24
  - Package: `evalid@0.2.0`
@@ -130,7 +130,7 @@
130
130
  └──────────────────────────────────────────────────────────────────────────┘
131
131
 
132
132
  ┌──────────────────────────── Verify / Review ────────────────────────────┐
133
- │ 23 verify-consistency impl vs proposal/plan/specs/AHA.md 漂移检查 │
133
+ │ 23 verify-consistency impl vs proposal/plan/specs/AHA.md 漂移检查 + J6 CR 门禁 │
134
134
  │ 24 review 对抗性代码评审 + patch-challenge(corps) │
135
135
  │ 25 qa 行为 + 视觉验证,截图支持(corps) │
136
136
  │ 26 gate_final 镜像 frozen contract,pass/fail/needs-human │
@@ -195,6 +195,12 @@ CHANGE_ID=<id> xflow workflow run workflows/yolo.yaml
195
195
  │ exist
196
196
  ▼
197
197
  ┌──────────────────────┐
198
+ │ 21a tdd-red │ atoms: I6a(red) + I6b(red) + B3 transition
199
+ │ gate: artifact-verify│◄─── red 必须失败(F2 起紧随 plan 封存,与规划同一轮次)
200
+ └──────────┬───────────┘
201
+ │ exist
202
+ ▼
203
+ ┌──────────────────────┐
198
204
  │ 19 openissue │ atoms: A1 issue + A2 branch + B2b status
199
205
  │ gate: deterministic │◄─── E6 post-openissue
200
206
  └──────────┬───────────┘
@@ -207,19 +213,18 @@ CHANGE_ID=<id> xflow workflow run workflows/yolo.yaml
207
213
  │ pass
208
214
  ▼
209
215
  ┌──────────────────────┐
210
- │ 21a tdd-red │ atoms: I6a(red) + I6b(red)
211
- │ 21b tdd-green │ atoms: I6a(green) + I6b(green) + I6c quality
212
- │ gate: artifact-verify│◄─── red 必须失败;green/refactor 必须通过
213
- └──────────┬───────────┘
214
- │ exist
215
- ▼
216
- ┌──────────────────────┐
217
216
  │ 22 execute │ atoms: B3 transition
218
217
  │ gate: human ⏸ │◄─── 等待 xflow gate ack execute
219
218
  └──────────┬───────────┘
220
219
  │ ack
221
220
  ▼
222
221
  ┌──────────────────────┐
222
+ │ 21b tdd-green │ atoms: I6a(green) + I6b(green) + I6c quality
223
+ │ gate: artifact-verify│◄─── green/refactor 必须通过
224
+ └──────────┬───────────┘
225
+ │ exist
226
+ ▼
227
+ ┌──────────────────────┐
223
228
  │ 23 verify-consist. │ atoms: J1 tests.run + J4a structural + J4b delta
224
229
  │ gate: deterministic │◄─── J4a 漂移扫描 + J4b spec delta review
225
230
  └──────────┬───────────┘
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@surlymochan/evalid",
3
- "version": "0.3.1",
3
+ "version": "0.5.0",
4
4
  "description": "Claimed != verified: a runtime verification layer for coding agents. It seals each test run and replays it from a clean snapshot, with a zero-judge gate pipeline (spec clarification, code review, TDD proof) and the xflow CLI.",
5
5
  "keywords": [
6
6
  "ai-agents",
package/specs/workflow.md CHANGED
@@ -79,9 +79,9 @@ No workflow may define a phase not listed here. Phase IDs here are authoritative
79
79
 
80
80
  | # | ID | Source | Description |
81
81
  |---|----|--------|-------------|
82
- | 23 | `verify-consistency` | yolo | Impl vs proposal/plan/specs/AHA.md structural drift check |
82
+ | 23 | `verify-consistency` | yolo | Impl vs proposal/plan/specs/AHA.md structural drift check + deterministic CR gate (`J6.cr.review`: spec alignment, test follow, assertion balance, loop budget) |
83
83
  | 24 | `review` | agentos | Adversarial code review + patch-challenge |
84
- | 25 | `qa` | agentos | Behavior + visual verification, screenshot-backed for strict UI work |
84
+ | 25 | `qa` | agentos | Behavior + visual verification, screenshot-backed for strict UI work; runs the same `J6.cr.review` gate (verdict record `cr/review-0.json`) |
85
85
  | 26 | `gate_final` | agentos | Mirror frozen contract; decide pass / fail / needs-human |
86
86
 
87
87
  ### Archive
@@ -91,9 +91,24 @@ export class WorkflowExecutor {
91
91
  phases: this.runtime.phaseCosts,
92
92
  wallClockMs,
93
93
  });
94
+ this.persistCostReport(this.runtime.costReport);
94
95
  return this.runtime.costReport;
95
96
  }
96
97
 
98
+ // The cost report is run evidence (issue #5 P1-1): besides riding on the
99
+ // terminal execution-log event, persist it as a standalone artifact so a
100
+ // reader never has to mine the ndjson log for the per-phase breakdown.
101
+ // Best-effort by design — accounting must never fail the run itself.
102
+ persistCostReport(report) {
103
+ try {
104
+ const dir = resolve(this.projectRoot, '.as-xflow');
105
+ mkdirSync(dir, { recursive: true });
106
+ writeFileSync(resolve(dir, 'phase-cost.json'), JSON.stringify(report, null, 2));
107
+ } catch {
108
+ // The execution-log event still carries the report.
109
+ }
110
+ }
111
+
97
112
  async run() {
98
113
  this.runtime.workflowStartedAtMs = Date.now();
99
114
  this.runtime.currentPhaseTokens = emptyTokenUsage();
@@ -720,6 +735,7 @@ export class WorkflowExecutor {
720
735
  phase_label: phase?.label || null,
721
736
  reason,
722
737
  detail: detail || null,
738
+ cost: this.buildCostReport(),
723
739
  });
724
740
  } catch {
725
741
  // Failure logging is best-effort and must not hide the real stop reason.
@@ -428,6 +428,7 @@ phases:
428
428
  atoms:
429
429
  - id: J1.tests.run
430
430
  - id: J4b.spec_delta.review
431
+ - id: J6.cr.review
431
432
  - id: H5.visual.review
432
433
  with: { mode: "qa" }
433
434
  - id: E5.artifact.verify
@@ -441,6 +442,7 @@ phases:
441
442
  - id: J5.gate.evidence_collect
442
443
  artifacts:
443
444
  - { path: "specs/changes/${change_id}/qa_acceptance.json", optional: false }
445
+ - { path: "specs/changes/${change_id}/cr/review-0.json", optional: false }
444
446
  gate:
445
447
  type: artifact-verify
446
448
  on_fail: stop
@@ -110,6 +110,28 @@ phases:
110
110
  type: artifact-verify
111
111
  on_fail: stop
112
112
 
113
+ # F2 (issue #5 P1-2): on the lite track the red proof is sealed immediately
114
+ # after planning, in the same agent pass that authored the plan and the
115
+ # failing test. The issue/branch ceremony (openissue, set-in-progress) no
116
+ # longer sits between planning and red-proof, where it forced a separate
117
+ # exploration pass; it still completes before execute.
118
+ - id: tdd-red
119
+ label: "21a · TDD red"
120
+ catalog_ref: 21
121
+ required: true
122
+ atoms:
123
+ - id: I6a.tdd.run
124
+ with: { phase: "red", test_command: "${input.tdd_red_command}", expect: "fail" }
125
+ - id: I6b.tdd.proof_validate
126
+ with: { phase: "red", tdd_mode: "${input.tdd_mode}" }
127
+ - id: B3.status.transition
128
+ with: { to_stage: "tdd" }
129
+ artifacts:
130
+ - { path: "specs/changes/${change_id}/tdd/red-0.json", optional: false }
131
+ gate:
132
+ type: artifact-verify
133
+ on_fail: stop
134
+
113
135
  - id: openissue
114
136
  label: "19 · Open issue + branch"
115
137
  catalog_ref: 19
@@ -145,23 +167,6 @@ phases:
145
167
  with: { phase: "pre-exec" }
146
168
  on_fail: stop
147
169
 
148
- - id: tdd-red
149
- label: "21a · TDD red"
150
- catalog_ref: 21
151
- required: true
152
- atoms:
153
- - id: I6a.tdd.run
154
- with: { phase: "red", test_command: "${input.tdd_red_command}", expect: "fail" }
155
- - id: I6b.tdd.proof_validate
156
- with: { phase: "red", tdd_mode: "${input.tdd_mode}" }
157
- - id: B3.status.transition
158
- with: { to_stage: "tdd" }
159
- artifacts:
160
- - { path: "specs/changes/${change_id}/tdd/red-0.json", optional: false }
161
- gate:
162
- type: artifact-verify
163
- on_fail: stop
164
-
165
170
  - id: execute
166
171
  label: "22 · Execute"
167
172
  catalog_ref: 22
@@ -199,12 +204,15 @@ phases:
199
204
  - id: J1.tests.run
200
205
  - id: J4a.spec_consistency.structural
201
206
  - id: J4b.spec_delta.review
207
+ - id: J6.cr.review
202
208
  - id: B3.status.transition
203
209
  with: { to_stage: "verify" }
204
210
  - id: B2b.status.write
205
211
  with:
206
212
  fields:
207
213
  verification_status: "running"
214
+ artifacts:
215
+ - { path: "specs/changes/${change_id}/cr/review-0.json", optional: false }
208
216
  gate:
209
217
  type: deterministic
210
218
  atom: J4a.spec_consistency.structural
@@ -64,7 +64,9 @@ def main():
64
64
 
65
65
  passed = exit_code == 0
66
66
  combined = f"{stdout}\n{stderr}"
67
- file_hashes = tdd_lib.write_snapshot(project, args.change_id, args.phase, args.attempt)
67
+ file_hashes, snapshot_scope = tdd_lib.write_snapshot(
68
+ project, args.change_id, args.phase, args.attempt, command=args.test_command
69
+ )
68
70
  proof = {
69
71
  'phase': args.phase,
70
72
  'attempt': args.attempt,
@@ -79,6 +81,7 @@ def main():
79
81
  'output_sha256': tdd_lib.output_digest(stdout, stderr),
80
82
  'git_head': tdd_lib.git_head(project),
81
83
  'file_hashes': file_hashes,
84
+ 'snapshot_scope': snapshot_scope,
82
85
  'failure_kind': 'timeout' if timed_out else tdd_lib.classify_output(combined, passed),
83
86
  'recorded_at': datetime.now(timezone.utc).isoformat(),
84
87
  }
@@ -95,8 +95,17 @@ def main():
95
95
  if found is None or found != proof.get('file_hashes'):
96
96
  emit({'ok': False, 'verdict': 'snapshot_mismatch', 'phase': args.phase}, 1)
97
97
 
98
+ scope = proof.get('snapshot_scope')
99
+ scoped = isinstance(scope, dict) and bool(proof.get('git_head'))
98
100
  try:
99
- actual_exit, actual_text = tdd_lib.reexec_snapshot(destination, proof.get('command') or '', timeout=120)
101
+ if scoped:
102
+ actual_exit, actual_text = tdd_lib.reexec_scoped_snapshot(
103
+ project, destination, proof.get('command') or '', proof['git_head'], scope, timeout=120
104
+ )
105
+ else:
106
+ actual_exit, actual_text = tdd_lib.reexec_snapshot(destination, proof.get('command') or '', timeout=120)
107
+ except RuntimeError as exc:
108
+ emit({'ok': False, 'verdict': 'replay_source_missing', 'phase': args.phase, 'error': str(exc)[:300]}, 1)
100
109
  except subprocess.TimeoutExpired:
101
110
  emit({'ok': False, 'verdict': 'reexec_timeout', 'phase': args.phase}, 1)
102
111
 
@@ -127,6 +136,13 @@ def main():
127
136
  path = project / rel
128
137
  if not path.is_file() or tdd_lib.sha256_bytes(path.read_bytes()) != digest:
129
138
  mismatched.append(rel)
139
+ if scoped:
140
+ # Test files outside the sealed scope were pinned to the recorded
141
+ # HEAD at seal time; any that have drifted since are unsealed edits.
142
+ sealed = set(proof.get('file_hashes', {}))
143
+ for rel in tdd_lib.changed_since(project, proof['git_head']):
144
+ if rel not in sealed and tdd_lib.is_test_file(rel):
145
+ mismatched.append(rel)
130
146
  if mismatched:
131
147
  emit({
132
148
  'ok': False,
@@ -138,6 +154,9 @@ def main():
138
154
  if proof.get('failure_kind') == 'skipped' or tdd_lib.output_non_result(actual_text):
139
155
  emit({'ok': False, 'verdict': 'tests_skipped_or_todo', 'phase': args.phase}, 1)
140
156
 
157
+ if proof.get('failure_kind') == 'counts_unknown' or tdd_lib.output_counts_unknown(actual_text):
158
+ emit({'ok': False, 'verdict': 'tests_counts_unknown', 'phase': args.phase}, 1)
159
+
141
160
  expected_passed = expected_result(args.phase)
142
161
  if expected_passed is not None and proof.get('passed') is not expected_passed:
143
162
  emit({
@@ -95,6 +95,15 @@ def proof_skipped(proof: dict) -> bool:
95
95
  return tdd_lib.output_non_result(blob)
96
96
 
97
97
 
98
+ def proof_counts_unknown(proof: dict) -> bool:
99
+ if not proof:
100
+ return False
101
+ if proof.get("failure_kind") == "counts_unknown":
102
+ return True
103
+ blob = f"{proof.get('stdout_tail') or ''}\n{proof.get('stderr_tail') or ''}"
104
+ return tdd_lib.output_counts_unknown(blob)
105
+
106
+
98
107
  def symbols_in(project_root: Path, code_files: list[str]) -> set[str]:
99
108
  symbols = set()
100
109
  for relative in code_files:
@@ -298,6 +307,12 @@ def main() -> None:
298
307
  "code": "tests_skipped_or_todo",
299
308
  "message": "A recorded run skipped or marked todo the tests it claims to have executed.",
300
309
  })
310
+ if proof_counts_unknown(red_proof) or proof_counts_unknown(green_proof):
311
+ findings.append({
312
+ "severity": "fail",
313
+ "code": "tests_counts_unknown",
314
+ "message": "A recorded run's summary counts could not be fully parsed, so its skip/todo counts are unknown and the run cannot be verified.",
315
+ })
301
316
  symbols = symbols_in(project_root, code_files)
302
317
  if symbols and test_files:
303
318
  for relative in test_files:
@@ -26,6 +26,11 @@ Checks (findings are structured JSON, severity fail | warn):
26
26
  (--max-files / --max-lines, or status.json cr_max_files /
27
27
  cr_max_lines). Over budget is a warn with a split-the-spec
28
28
  recommendation; over 2x budget fails.
29
+ 5. Seal partition (reporting only, never a verdict input) - the proof
30
+ splits the changed files into those still byte-identical to the
31
+ green TDD seal (file_hashes in tdd/green-0.json) and those that
32
+ drifted or were never sealed. A review pass re-reads only the
33
+ drifted set; the sealed set is exactly what the TDD gate verified.
29
34
 
30
35
  Self-contained on purpose: this atom does not import tdd_lib so it can
31
36
  run standalone in a reviewed tree. The file/symbol/assertion semantics
@@ -36,6 +41,7 @@ directory is excluded from the reviewed diff.
36
41
  """
37
42
  import argparse
38
43
  import fnmatch
44
+ import hashlib
39
45
  import json
40
46
  import os
41
47
  import re
@@ -200,6 +206,38 @@ def assertion_delta(project_root: Path, diff_ref: str, test_files: list, deleted
200
206
  return added, removed
201
207
 
202
208
 
209
+ def seal_partition(project_root: Path, change_dir: Path, files: list):
210
+ """Split changed files by whether they still match the green seal.
211
+
212
+ The green TDD proof (tdd/green-0.json) records the sha256 of every
213
+ file I6a sealed when the proof was produced (issue #5, P1-2). A
214
+ changed file whose current bytes still hash to the sealed value is
215
+ byte-for-byte what the TDD gate verified, so a review pass does not
216
+ need to re-read it; anything else — drifted after the seal, or never
217
+ sealed — must be re-read. Returns None when no usable green seal
218
+ exists (the review then re-reads everything).
219
+ """
220
+ green = read_json(change_dir / "tdd" / "green-0.json")
221
+ hashes = green.get("file_hashes")
222
+ if not isinstance(hashes, dict) or not hashes:
223
+ return None
224
+ sealed, drifted = [], []
225
+ for relative in files:
226
+ full = project_root / relative
227
+ digest = None
228
+ if full.is_file():
229
+ digest = hashlib.sha256(full.read_bytes()).hexdigest()
230
+ if digest is not None and hashes.get(relative) == digest:
231
+ sealed.append(relative)
232
+ else:
233
+ drifted.append(relative)
234
+ return {
235
+ "sealed_unchanged": sealed,
236
+ "seal_drifted": drifted,
237
+ "reread_count": len(drifted),
238
+ }
239
+
240
+
203
241
  def tokenize(text: str) -> set:
204
242
  tokens = set()
205
243
  for raw in re.findall(r"[A-Za-z_][A-Za-z0-9_]*", text):
@@ -416,6 +454,7 @@ def main() -> None:
416
454
  "changed_files": all_files,
417
455
  "code_files": code_files,
418
456
  "test_files": test_files,
457
+ "seal_partition": seal_partition(project_root, change_dir, all_files),
419
458
  "spec_aligned_files": aligned_files,
420
459
  "assertions": {"added": added_assertions, "removed": removed_assertions},
421
460
  "budget": budget,
@@ -8,6 +8,13 @@ fails unless the snapshot really exits that way and the binding is rebuilt.
8
8
  An agent who can write the tree can still record a genuine failing snapshot
9
9
  and later replace the working tree; the digest is not a trusted third party.
10
10
  Byte-identical stdout is not required, because runners print timings.
11
+
12
+ Scoped sealing (issue #5 P1-4): inside a git work tree the snapshot copies
13
+ only the files that differ from HEAD plus the files the test command names.
14
+ Every other file is pinned to the recorded HEAD: I6b materializes it from
15
+ `git archive` at replay time, so the replayed tree is byte-identical to the
16
+ sealed one while the stored manifest stays proportional to the change.
17
+ Outside a git work tree the legacy full-tree copy is used unchanged.
11
18
  """
12
19
  import hashlib
13
20
  import json
@@ -44,6 +51,7 @@ BINDING_FIELDS = (
44
51
  "stderr_tail",
45
52
  "git_head",
46
53
  "file_hashes",
54
+ "snapshot_scope",
47
55
  "failure_kind",
48
56
  "recorded_at",
49
57
  )
@@ -134,9 +142,16 @@ def sha256_bytes(data: bytes) -> str:
134
142
  return hashlib.sha256(data).hexdigest()
135
143
 
136
144
 
145
+ # Optional binding fields join the payload only when present, so proofs
146
+ # sealed before the field existed keep verifying byte-for-byte.
147
+ OPTIONAL_BINDING_FIELDS = {"snapshot_scope"}
148
+
149
+
137
150
  def binding_payload(proof: dict) -> dict:
138
151
  payload = {}
139
152
  for field in BINDING_FIELDS:
153
+ if field in OPTIONAL_BINDING_FIELDS and proof.get(field) is None:
154
+ continue
140
155
  payload[field] = proof.get(field)
141
156
  return payload
142
157
 
@@ -187,20 +202,128 @@ def iter_project_files(project_root: Path, skip_prefix: str):
187
202
  yield rel, full
188
203
 
189
204
 
190
- def write_snapshot(project_root: Path, change_id: str, phase: str, attempt: int) -> dict:
205
+ def _git_status_paths(project_root: Path):
206
+ """(changed, deleted) path sets from `git status --porcelain -z`.
207
+
208
+ Returns None when the project is not a git work tree. Renames report the
209
+ new path as changed and the old path as deleted.
210
+ """
211
+ result = subprocess.run(
212
+ ["git", "status", "--porcelain=v1", "-z", "--untracked-files=all"],
213
+ cwd=project_root,
214
+ capture_output=True,
215
+ )
216
+ if result.returncode != 0:
217
+ return None
218
+ entries = result.stdout.decode("utf-8", errors="replace").split("\0")
219
+ changed: set[str] = set()
220
+ deleted: set[str] = set()
221
+ index = 0
222
+ while index < len(entries):
223
+ entry = entries[index]
224
+ index += 1
225
+ if len(entry) < 4:
226
+ continue
227
+ code, rel = entry[:2], entry[3:]
228
+ if "R" in code or "C" in code:
229
+ old = entries[index] if index < len(entries) else ""
230
+ index += 1
231
+ if old:
232
+ deleted.add(old)
233
+ if "D" in code:
234
+ deleted.add(rel)
235
+ elif rel:
236
+ changed.add(rel)
237
+ return changed, deleted
238
+
239
+
240
+ def changed_since(project_root: Path, ref: str) -> set[str]:
241
+ """Paths whose working-tree content differs from `ref`, plus untracked."""
242
+ paths: set[str] = set()
243
+ diff = subprocess.run(
244
+ ["git", "diff", "--name-only", "-z", ref],
245
+ cwd=project_root,
246
+ capture_output=True,
247
+ )
248
+ if diff.returncode == 0:
249
+ paths.update(p for p in diff.stdout.decode("utf-8", errors="replace").split("\0") if p)
250
+ others = subprocess.run(
251
+ ["git", "ls-files", "--others", "--exclude-standard", "-z"],
252
+ cwd=project_root,
253
+ capture_output=True,
254
+ )
255
+ if others.returncode == 0:
256
+ paths.update(p for p in others.stdout.decode("utf-8", errors="replace").split("\0") if p)
257
+ return paths
258
+
259
+
260
+ def snapshot_scope(project_root: Path, change_id: str, command: str | None) -> dict | None:
261
+ """Files the proof actually depends on (issue #5 P1-4).
262
+
263
+ Scoped sealing only applies inside a git work tree: every file that
264
+ differs from HEAD is sealed by copy, and every other file is pinned to
265
+ the recorded HEAD and materialized from git at replay time. Returns None
266
+ when no HEAD exists to pin the rest of the tree to (legacy full copy).
267
+ """
268
+ if git_head(project_root) is None:
269
+ return None
270
+ status = _git_status_paths(project_root)
271
+ if status is None:
272
+ return None
273
+ changed, deleted = status
274
+ evidence_prefix = (Path("specs") / "changes" / change_id).as_posix() + "/"
275
+
276
+ def excluded(rel: str) -> bool:
277
+ return rel.startswith(evidence_prefix)
278
+
279
+ def noise(rel: str) -> bool:
280
+ return any(part in SKIP_DIR_NAMES for part in rel.split("/"))
281
+
282
+ files = {rel for rel in changed if not excluded(rel) and not noise(rel)}
283
+ for rel in test_paths(command or ""):
284
+ if not excluded(rel) and (project_root / rel).is_file():
285
+ files.add(rel)
286
+ return {
287
+ "version": 1,
288
+ "files": sorted(files),
289
+ "deleted": sorted(rel for rel in deleted if not excluded(rel) and not noise(rel)),
290
+ }
291
+
292
+
293
+ def write_snapshot(project_root: Path, change_id: str, phase: str, attempt: int,
294
+ command: str | None = None) -> tuple[dict, dict | None]:
295
+ """Seal the tree state a proof ran against.
296
+
297
+ With a scope (git work tree), only the scoped files are copied and
298
+ hashed; the manifest stays proportional to the change, not the tree.
299
+ Without one, the legacy full-tree copy is used.
300
+ """
191
301
  skip_prefix = (Path("specs") / "changes" / change_id / "tdd").as_posix()
192
302
  destination = snapshot_dir(project_root, change_id, phase, attempt)
193
303
  if destination.exists():
194
304
  shutil.rmtree(destination)
195
305
  destination.mkdir(parents=True, exist_ok=True)
306
+ scope = snapshot_scope(project_root, change_id, command)
196
307
  hashes = {}
197
- for rel, full in iter_project_files(project_root, skip_prefix):
308
+ if scope is None:
309
+ for rel, full in iter_project_files(project_root, skip_prefix):
310
+ data = full.read_bytes()
311
+ hashes[rel] = sha256_bytes(data)
312
+ target = destination / rel
313
+ target.parent.mkdir(parents=True, exist_ok=True)
314
+ target.write_bytes(data)
315
+ return hashes, None
316
+ for rel in scope["files"]:
317
+ full = project_root / rel
318
+ if full.is_symlink() or not full.is_file():
319
+ continue
198
320
  data = full.read_bytes()
199
321
  hashes[rel] = sha256_bytes(data)
200
322
  target = destination / rel
201
323
  target.parent.mkdir(parents=True, exist_ok=True)
202
324
  target.write_bytes(data)
203
- return hashes
325
+ scope = {**scope, "files": sorted(hashes)}
326
+ return hashes, scope
204
327
 
205
328
 
206
329
  def snapshot_tree_hashes(destination: Path):
@@ -220,27 +343,68 @@ def output_digest(stdout: str, stderr: str) -> str:
220
343
  return hashlib.sha256(f"{stdout}\0{stderr}".encode("utf-8")).hexdigest()
221
344
 
222
345
 
346
+ # Node's built-in test runner ends a run with a summary block of
347
+ # "<marker> <label> <count>" lines — the runner's own structured tally.
348
+ # The marker depends on the reporter: "#" for the TAP reporter (the piped
349
+ # default up to Node 22) and "ℹ" for the spec reporter (the piped default
350
+ # from Node 24, and selectable on any version). The gate reads only these
351
+ # line-anchored summary lines, never free-floating text, so test names or
352
+ # program output cannot forge a count.
353
+ _NODE_GATE_LABELS = ("pass", "fail", "skipped", "todo")
354
+ # Lines every node:test summary block carries and other TAP emitters do
355
+ # not; their joint presence is what marks a block as a node summary.
356
+ _NODE_SIGNATURE_LABELS = ("tests", "suites", "cancelled")
357
+
358
+
223
359
  def _summary_count(text: str, label: str):
224
- match = re.search(rf"#\s+{label}\s+(\d+)\b", text)
360
+ match = re.search(rf"^[#\u2139]\s*{label}\s+(\d+)\b", text, re.MULTILINE)
225
361
  if not match:
226
362
  return None
227
363
  return int(match.group(1))
228
364
 
229
365
 
366
+ def node_run_counts(text: str):
367
+ """Summary counts from a node:test run, or None when there is no summary.
368
+
369
+ A dict is returned once a block carries the full node signature
370
+ (tests/suites/cancelled). A gate label the block does not carry maps
371
+ to None — unknown, never silently zero: a truncated or reshaped
372
+ block must not read as "zero skipped".
373
+ """
374
+ if not text:
375
+ return None
376
+ if any(_summary_count(text, label) is None for label in _NODE_SIGNATURE_LABELS):
377
+ return None
378
+ labels = _NODE_SIGNATURE_LABELS + _NODE_GATE_LABELS
379
+ return {label: _summary_count(text, label) for label in labels}
380
+
381
+
382
+ def output_counts_unknown(text: str) -> bool:
383
+ """True when a node:test summary is present but a gate count is unreadable."""
384
+ counts = node_run_counts(text)
385
+ if counts is None:
386
+ return False
387
+ return any(counts[label] is None for label in _NODE_GATE_LABELS)
388
+
389
+
230
390
  def output_non_result(text: str) -> bool:
231
391
  """True when the runner reported skip/todo/xfail and no passing test."""
232
392
  if not text:
233
393
  return False
234
- skipped = _summary_count(text, "skipped") or 0
235
- todo = _summary_count(text, "todo") or 0
236
- passed = _summary_count(text, "pass")
237
- failed = _summary_count(text, "fail")
238
- if skipped + todo > 0 and (passed or 0) == 0 and (failed or 0) == 0:
239
- return True
394
+ counts = node_run_counts(text)
395
+ node_failed = None
396
+ if counts is not None:
397
+ skipped = counts["skipped"]
398
+ todo = counts["todo"]
399
+ passed = counts["pass"]
400
+ node_failed = counts["fail"]
401
+ if None not in (skipped, todo, passed, node_failed):
402
+ if skipped + todo > 0 and passed == 0 and node_failed == 0:
403
+ return True
240
404
  py_skipped = sum(int(item) for item in re.findall(r"\b(\d+)\s+skipped\b", text))
241
405
  py_skipped += sum(int(item) for item in re.findall(r"\b(\d+)\s+xfailed\b", text))
242
406
  py_passed = sum(int(item) for item in re.findall(r"\b(\d+)\s+passed\b", text))
243
- if py_skipped > 0 and py_passed == 0 and (failed or 0) == 0:
407
+ if py_skipped > 0 and py_passed == 0 and (node_failed or 0) == 0:
244
408
  return True
245
409
  return False
246
410
 
@@ -249,6 +413,11 @@ def classify_output(text: str, passed: bool) -> str:
249
413
  if output_non_result(text):
250
414
  return "skipped"
251
415
  if passed:
416
+ # Fail closed: a node:test summary that cannot be fully read does
417
+ # not prove a clean pass — the skip/todo counts may sit in the
418
+ # part that failed to parse, so the run is unverifiable, not green.
419
+ if output_counts_unknown(text):
420
+ return "counts_unknown"
252
421
  return "passed"
253
422
  if re.search(r"does not provide an export named|cannot import name\b", text, re.IGNORECASE):
254
423
  return "test_error"
@@ -676,6 +845,46 @@ def reexec_snapshot(destination: Path, command: str, timeout: int = 120):
676
845
  shutil.rmtree(temp, ignore_errors=True)
677
846
 
678
847
 
848
+ def reexec_scoped_snapshot(project_root: Path, destination: Path, command: str,
849
+ git_head_ref: str, scope: dict, timeout: int = 120):
850
+ """Replay a scoped proof: rebuild the sealed tree, then re-run.
851
+
852
+ Unchanged files come from `git archive` at the recorded HEAD (their
853
+ seal-time content is HEAD content by construction of the scope), deleted
854
+ paths stay deleted, and the sealed scoped files are overlaid last so
855
+ they win over the archive.
856
+ """
857
+ temp = Path(tempfile.mkdtemp(prefix="evalid-proof-"))
858
+ try:
859
+ archive = subprocess.run(
860
+ ["git", "archive", git_head_ref],
861
+ cwd=project_root,
862
+ capture_output=True,
863
+ )
864
+ if archive.returncode != 0:
865
+ raise RuntimeError(f"git archive {git_head_ref} failed: {archive.stderr.decode('utf-8', errors='replace')[:300]}")
866
+ extract = subprocess.run(
867
+ ["tar", "-x", "-C", str(temp)],
868
+ input=archive.stdout,
869
+ capture_output=True,
870
+ )
871
+ if extract.returncode != 0:
872
+ raise RuntimeError(f"replay tree extraction failed: {extract.stderr.decode('utf-8', errors='replace')[:300]}")
873
+ for rel in list(scope.get("deleted", [])) + list(scope.get("files", [])):
874
+ target = temp / rel
875
+ if target.is_dir() and not target.is_symlink():
876
+ shutil.rmtree(target)
877
+ elif target.exists() or target.is_symlink():
878
+ target.unlink()
879
+ if any(destination.iterdir()):
880
+ shutil.copytree(destination, temp, dirs_exist_ok=True)
881
+ result = run_shell(command, temp, timeout=timeout)
882
+ text = f"{result.stdout or ''}\n{result.stderr or ''}"
883
+ return result.returncode, text
884
+ finally:
885
+ shutil.rmtree(temp, ignore_errors=True)
886
+
887
+
679
888
  def copy_project(project_root: Path, destination: Path) -> None:
680
889
  def ignore(_directory, names):
681
890
  return [name for name in names if name in SKIP_DIR_NAMES]
@@ -30,6 +30,15 @@ reuse it instead of rewriting the same plan again.
30
30
  If the file is missing or still a scaffold placeholder, create the reusable
31
31
  plan once and leave routing to the execution workflow.
32
32
 
33
+ ## Lite-track merged pass
34
+
35
+ When the change will run the lite (yolo) track, do the red-proof authoring
36
+ in the same pass as the plan: write the failing test and fix the red test
37
+ command in the plan itself. The yolo workflow seals `tdd-red` immediately
38
+ after `plan`, so a separate test-authoring pass would only re-read the
39
+ same files. This does not apply to the corps track, which keeps planning
40
+ and proof authoring apart.
41
+
33
42
  ## Boundary
34
43
 
35
44
  - This skill does not open issues.
@@ -25,7 +25,10 @@ ALLOWED_STAGE_TRANSITIONS: dict[str, set[str]] = {
25
25
  "proposal-freeze": {"proposal-freeze", "proposal-consistency-check", "plan"},
26
26
  "proposal-consistency-check": {"proposal-consistency-check", "plan"},
27
27
  "plan": {"plan", "tdd"},
28
- "openissue": {"openissue", "tdd"},
28
+ # "execute" is reachable from openissue on the lite track: since F2 the
29
+ # red proof is sealed before the issue is filed, so execution may start
30
+ # straight from the openissue stage.
31
+ "openissue": {"openissue", "tdd", "execute"},
29
32
  "tdd": {"tdd", "execute", "verify"},
30
33
  "execute": {"execute", "tdd", "verify"},
31
34
  "verify": {"verify", "tdd", "execute", "archive"},
@@ -22,6 +22,10 @@ For a full run:
22
22
  xflow workflow run workflows/yolo.yaml --title "<中文标题>" --change-type backend
23
23
  ```
24
24
 
25
+ When `--change-id` is omitted and the current branch cannot provide one
26
+ (e.g. `main`), the run derives the change id from `--title` and prints it;
27
+ pass `--change-id` explicitly to override.
28
+
25
29
  ## Plan reuse rule
26
30
 
27
31
  - Read `.xflow/GOAL.md` before execution when present. State whether the current
@@ -45,13 +49,26 @@ xflow workflow run workflows/yolo.yaml --title "<中文标题>" --change-type ba
45
49
  4. `proposal-freeze`
46
50
  5. `proposal-consistency-check`
47
51
  6. `plan`
48
- 7. `openissue`
49
- 8. `set-in-progress`
50
- 9. `tdd`
52
+ 7. `tdd` (red proof — sealed in the same pass as the plan, see below)
53
+ 8. `openissue`
54
+ 9. `set-in-progress`
51
55
  10. `execute`
52
56
  11. `verify-consistency`
53
57
  12. `archive`
54
58
 
59
+ On the lite track the red proof is sealed immediately after planning: the
60
+ pass that writes the plan also writes the failing test, so `tdd-red` only
61
+ runs and seals it — no second exploration pass, and the issue/branch
62
+ ceremony no longer sits between planning and the red proof.
63
+
64
+ In `verify-consistency`, run the deterministic gates first. The J6 proof
65
+ (`specs/changes/${CHANGE_ID}/cr/review-0.json`) carries a `seal_partition`:
66
+ files whose bytes still match the green proof's sealed hashes
67
+ (`sealed_unchanged`) are exactly what the TDD gate verified — the review
68
+ pass does not re-read them. Only `seal_drifted` files (changed after the
69
+ green seal, or never sealed) get a fresh read. When `seal_partition` is
70
+ null there is no green seal to trust, and the review re-reads everything.
71
+
55
72
  The workflow YAML is the canonical execution source for atom order, gates, and
56
73
  artifact checks. This skill only explains the contract.
57
74