@surlymochan/evalid 0.4.1 → 0.5.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/RELEASE_NOTES.md +26 -0
- package/docs/launch-dossier.md +2 -2
- package/package.json +1 -1
- package/xflow/atoms/i6b_tdd_proof_validate.py +32 -10
- package/xflow/atoms/i6c_tdd_quality_review.py +19 -8
- package/xflow/atoms/j6_cr_review.py +141 -5
- package/xflow/atoms/tdd_lib.py +129 -2
- package/xflow/yolo/SKILL.md +8 -0
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,23 @@ All notable changes to this project are documented here. Detailed notes live in
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [0.5.1] - 2026-11-07
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
- **Per-process read cache for the review gates (issue #5, P1-2 follow-up to #34)**: one gate run used to read the same source file several times — I6c for symbols, implementation sources and test review, J6 for spec alignment and then test follow once per changed code file. Both gates now read through a per-process cache keyed by the P1-3 stat anchor (size, mtime_ns, inode): each unchanged file is read from disk once per run and served from memory on later reads, while a changed file is re-read instead of served stale (a mid-read change is retried once, like the seal's `read_with_anchor`). J6 keeps its self-contained copy of the cache and does not import tdd_lib. Both proofs report `read_cache` (`disk_reads` / `cache_hits`) so the skipped re-reads are observable. No verdict logic changed; the token acceptance still needs a benchmark-environment rerun — no performance numbers are claimed here.
|
|
12
|
+
- **Stat-anchored gate verification (issue #5, P1-3)**: the TDD seal now records a stat anchor (size, mtime_ns, inode) per sealed file alongside its sha256 — tree anchors and snapshot-copy anchors in the proof's snapshot scope, bound into the proof like the scope itself. Proof validation (I6b) and the CR review seal partition (J6) stat-check first and re-read + re-hash only files whose anchor moved, instead of re-reading every sealed file on every gate pass; proofs sealed before the anchors existed fall back to the full re-read. Validation reports `reread_files` / `reread_count` and the seal partition reports `stat_checked`, so the skipped re-reads are observable. The token acceptance (median tokens incl. cache ≤ 2.0M) still needs a benchmark-environment rerun — no performance numbers are claimed here.
|
|
13
|
+
|
|
14
|
+
### Changed
|
|
15
|
+
- **Bench harness runs all four SWE-bench arms at high thinking effort**: `run_pi()` in `bench/swebench/harness/arms.py` and `bench/swebench/harness/pi_shim.sh` now set `--effort high` at both pi call sites, so the workflow remains the only variable across arms. The archived DeepSeek tables were recorded with the settings of their time and are not affected.
|
|
16
|
+
|
|
17
|
+
## [0.5.0] - 2026-11-06
|
|
18
|
+
|
|
19
|
+
### Added
|
|
20
|
+
- **CR review seal partition (issue #5, P1-2)**: the J6 review proof (`cr/review-0.json`) now splits the changed files into `sealed_unchanged` (bytes still identical to the green TDD proof's sealed hashes) and `seal_drifted`, with a `reread_count`. The yolo skill's verify-consistency guidance uses the partition so the review pass re-reads only drifted files instead of everything the gate already verified. Reporting only — verdicts are unchanged, and without a green seal the partition is `null` and everything is re-read.
|
|
21
|
+
- **Bench Verified set extended to 15 instances (issue #5, P2-2)**: `bench/swebench/instances.json` grows from 12 to 15, adding sympy-15599, sympy-19040 and sympy-24213, and a structural validation of the instance set (`bench/swebench/test_instances.py`) now runs in CI. Gold-patch checks for the three new instances stay pending the benchmark environment.
|
|
22
|
+
- **Bench report aggregates median (IQR) (issue #5, P2-1)**: `bench/swebench/report.py` now aggregates per-arm wall-clock, token and cost results with q1/median/q3 and presents them as median (IQR); reps isolation and the aggregation itself are pinned by `bench/swebench/test_reps_report.py`. Live multi-rep runs stay pending the benchmark environment.
|
|
23
|
+
- **P0-2 acceptance pinned against saved pre-fix patches (issue #5, P0-2)**: regression tests in `bench/swebench/test_diff_filter.py` run the saved pre-fix patches through the evidence filter (`specs/`, `.xflow/`, `.as-xflow/` excluded from solution patches). The saved 9.7 MB evidence-laden patch filters down to 4.3 KB, and every saved patch lands under the 200 KB acceptance line.
|
|
24
|
+
|
|
8
25
|
## [0.4.1] - 2026-11-06
|
|
9
26
|
|
|
10
27
|
### Fixed
|
package/RELEASE_NOTES.md
CHANGED
|
@@ -1,5 +1,31 @@
|
|
|
1
1
|
# Release Notes
|
|
2
2
|
|
|
3
|
+
## 0.5.1 - Gates stop re-reading unchanged files; bench arms run at high effort
|
|
4
|
+
|
|
5
|
+
Patch release. Two gate cost improvements from the issue #5 backlog, both pinned by tests rather than by a live run, plus a bench-harness change.
|
|
6
|
+
|
|
7
|
+
**Stat-anchored gate verification (#34, issue #5 P1-3).** The TDD seal now records a stat anchor (size, mtime_ns, inode) per sealed file alongside its sha256. Proof validation (I6b) and the CR review seal partition (J6) stat-check first and re-read + re-hash only files whose anchor moved, instead of re-reading every sealed file on every gate pass; proofs sealed before the anchors existed fall back to the full re-read. Validation reports `reread_files` / `reread_count` and the seal partition reports `stat_checked`, so the skipped re-reads are observable. Test evidence (local): `test/tdd-proof.test.js` 22/22 — 4 new, including unchanged tree validating with `reread_count` 0 and anchor-less proofs still verifying by full re-read; `test/cr-review.test.js` 14/14 — 2 new, including anchors accepting unchanged files (`stat_checked` 2) and an edited file re-read and drifted (`stat_checked` 1, `reread_count` 1).
|
|
8
|
+
|
|
9
|
+
**Per-process read cache for the review gates (#35, issue #5 P1-2 follow-up).** Both gates now read through a per-process cache keyed by the P1-3 stat anchor: each unchanged file is read from disk once per run and served from memory on later reads, while a changed file is re-read instead of served stale (a mid-read change is retried once). This removes the repeated re-reads inside one gate run — I6c reading the same source file for symbols, implementation sources and test review, and J6's test follow re-reading every test file once per changed code file. Both proofs report `read_cache` (`disk_reads` / `cache_hits`). No verdict logic changed. Test evidence (local): `test/tdd-proof.test.js` 24/24 — 2 new, including a cache unit test (3 reads → 1 disk read + 2 hits) and the quality-review proof reporting `read_cache` {disk_reads: 2, cache_hits: 2}; `test/cr-review.test.js` 16/16 — 2 new, including one gate run reading each unchanged file from disk once ({disk_reads: 3, cache_hits: 1}) and a test file shared by two code files read from disk once ({disk_reads: 4, cache_hits: 3}).
|
|
10
|
+
|
|
11
|
+
**Bench arms run at high thinking effort (#36).** Both pi call sites in the SWE-bench harness (`harness/arms.py` `run_pi()` and `harness/pi_shim.sh`) now set `--effort high`, so all four arms run MiniMax at the same high thinking effort and the workflow remains the only variable. The archived DeepSeek tables were recorded with the settings of their time and are not affected.
|
|
12
|
+
|
|
13
|
+
Acceptance debt, stated plainly: the live-run acceptance for P1-2 and P1-3 — the wall-clock/token lines from issue #5, including median tokens incl. cache ≤ 2.0M — is still pending the benchmark environment, which is being rebuilt. No bench numbers are claimed in this release; what shipped is the mechanism and the tests that pin its behaviour.
|
|
14
|
+
|
|
15
|
+
## 0.5.0 - Review re-reads only drifted files; the bench set grows to 15
|
|
16
|
+
|
|
17
|
+
Minor release. One product change — the CR review now spends its re-reads only where the seal says bytes moved — plus three bench-hygiene landings from the issue #5 backlog, each pinned by tests rather than by a live run.
|
|
18
|
+
|
|
19
|
+
**Review re-reads only drifted files (#32, issue #5 P1-2).** The J6 review proof (`cr/review-0.json`) now carries a seal partition: every changed file is classified `sealed_unchanged` (bytes still identical to the green TDD proof's sealed hashes) or `seal_drifted`, with a `reread_count`. The yolo verify-consistency guidance consumes the partition, so the review pass re-reads only the drifted files instead of everything the gate already verified. This is reporting only — verdicts are unchanged, and without a green seal the partition is `null` and everything is re-read, exactly as before.
|
|
20
|
+
|
|
21
|
+
**The Verified bench set grows from 12 to 15 instances (#30, issue #5 P2-2).** `bench/swebench/instances.json` adds sympy-15599, sympy-19040 and sympy-24213, and a structural validation of the instance set (`bench/swebench/test_instances.py`) now runs in CI, so a malformed instance entry fails the build instead of a bench run. Gold-patch checks for the three new instances are pending the benchmark environment.
|
|
22
|
+
|
|
23
|
+
**Bench reports now aggregate as median (IQR) (#31, issue #5 P2-1).** `bench/swebench/report.py` aggregates per-arm wall-clock, token and cost results with q1/median/q3 and presents them in median (IQR) form. Reps isolation — repeat runs of the same cell never share grading state — and the aggregation itself are pinned by `bench/swebench/test_reps_report.py`.
|
|
24
|
+
|
|
25
|
+
**P0-2 acceptance is now pinned by regression tests (#29, issue #5 P0-2).** The evidence exclusion itself landed earlier (solution patches drop `specs/`, `.xflow/` and `.as-xflow/`); what was missing was a test that would notice if it regressed. `bench/swebench/test_diff_filter.py` now runs the saved pre-fix patches through the filter: the saved 9.7 MB evidence-laden patch comes out at 4.3 KB, and every saved patch lands under the 200 KB acceptance line.
|
|
26
|
+
|
|
27
|
+
Acceptance debt, stated plainly: the live-run acceptance for P1-2, P2-1 and P2-2 — multi-rep bench runs on the restored environment, including the wall-clock/token lines from issue #5 and gold checks for the new instances — is still pending the benchmark environment. No performance numbers are claimed in this release; what shipped is the reporting, the larger instance set, and the tests that pin their behaviour.
|
|
28
|
+
|
|
3
29
|
## 0.4.1 - Failed runs report phase costs; Node 24 test counts fail closed
|
|
4
30
|
|
|
5
31
|
Patch release. Two fixes, both about the gate trusting evidence it had not actually read.
|
package/docs/launch-dossier.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# evalid Launch Dossier
|
|
2
2
|
|
|
3
|
-
Generated: 2026-11-
|
|
3
|
+
Generated: 2026-11-06T19:58:36.504Z
|
|
4
4
|
Mode: post_publish
|
|
5
5
|
Verdict: conditionally ready: engine and adoption evidence are strong, registry publication remains required for a splash launch
|
|
6
6
|
Launch ready: not yet
|
|
@@ -117,7 +117,7 @@ Evidence:
|
|
|
117
117
|
- Adoption records: 1; status: pass
|
|
118
118
|
- Goal audit: pass; goal: Keep evalid credible as an open-source agent delivery runtime: repo-owned workflow contracts, executable verification gates, durable project context, and honest benchmark evidence.
|
|
119
119
|
- Package evidence: missing; source: registry_json
|
|
120
|
-
- Package name/version: @surlymochan/evalid@0.
|
|
120
|
+
- Package name/version: @surlymochan/evalid@0.5.1
|
|
121
121
|
|
|
122
122
|
## Passing Evidence
|
|
123
123
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@surlymochan/evalid",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.1",
|
|
4
4
|
"description": "Claimed != verified: a runtime verification layer for coding agents. It seals each test run and replays it from a clean snapshot, with a zero-judge gate pipeline (spec clarification, code review, TDD proof) and the xflow CLI.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai-agents",
|
|
@@ -91,11 +91,22 @@ def main():
|
|
|
91
91
|
emit({'ok': False, 'verdict': 'proof_inconsistent', 'phase': args.phase}, 1)
|
|
92
92
|
|
|
93
93
|
destination = tdd_lib.snapshot_dir(project, args.change_id, args.phase, args.attempt)
|
|
94
|
-
found = tdd_lib.snapshot_tree_hashes(destination)
|
|
95
|
-
if found is None or found != proof.get('file_hashes'):
|
|
96
|
-
emit({'ok': False, 'verdict': 'snapshot_mismatch', 'phase': args.phase}, 1)
|
|
97
|
-
|
|
98
94
|
scope = proof.get('snapshot_scope')
|
|
95
|
+
copy_stats = scope.get('copy_stats') if isinstance(scope, dict) else None
|
|
96
|
+
reread_files: set[str] = set()
|
|
97
|
+
if isinstance(copy_stats, dict) and copy_stats:
|
|
98
|
+
# Stat-anchored check (issue #5, P1-3): the seal already hashed
|
|
99
|
+
# these bytes, so only copies whose stat moved are re-read.
|
|
100
|
+
checked = tdd_lib.verify_hashes_by_stat(destination, proof.get('file_hashes'), copy_stats)
|
|
101
|
+
reread_files.update(checked['reread'])
|
|
102
|
+
if checked['mismatched']:
|
|
103
|
+
emit({'ok': False, 'verdict': 'snapshot_mismatch', 'phase': args.phase, 'files': checked['mismatched']}, 1)
|
|
104
|
+
else:
|
|
105
|
+
found = tdd_lib.snapshot_tree_hashes(destination)
|
|
106
|
+
if found is None or found != proof.get('file_hashes'):
|
|
107
|
+
emit({'ok': False, 'verdict': 'snapshot_mismatch', 'phase': args.phase}, 1)
|
|
108
|
+
reread_files.update(proof.get('file_hashes', {}).keys())
|
|
109
|
+
|
|
99
110
|
scoped = isinstance(scope, dict) and bool(proof.get('git_head'))
|
|
100
111
|
try:
|
|
101
112
|
if scoped:
|
|
@@ -130,12 +141,21 @@ def main():
|
|
|
130
141
|
|
|
131
142
|
if args.phase in {'green', 'refactor', 'run'}:
|
|
132
143
|
mismatched = []
|
|
133
|
-
for rel, digest in proof.get('file_hashes', {}).items()
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
144
|
+
test_hashes = {rel: digest for rel, digest in proof.get('file_hashes', {}).items()
|
|
145
|
+
if tdd_lib.is_test_file(rel)}
|
|
146
|
+
file_stats = scope.get('file_stats') if isinstance(scope, dict) else None
|
|
147
|
+
if isinstance(file_stats, dict) and file_stats:
|
|
148
|
+
# Stat-anchored check (issue #5, P1-3): sealed test files whose
|
|
149
|
+
# anchor still matches are accepted without re-reading them.
|
|
150
|
+
checked = tdd_lib.verify_hashes_by_stat(project, test_hashes, file_stats)
|
|
151
|
+
reread_files.update(checked['reread'])
|
|
152
|
+
mismatched = checked['mismatched']
|
|
153
|
+
else:
|
|
154
|
+
for rel, digest in test_hashes.items():
|
|
155
|
+
path = project / rel
|
|
156
|
+
if not path.is_file() or tdd_lib.sha256_bytes(path.read_bytes()) != digest:
|
|
157
|
+
mismatched.append(rel)
|
|
158
|
+
reread_files.update(test_hashes.keys())
|
|
139
159
|
if scoped:
|
|
140
160
|
# Test files outside the sealed scope were pinned to the recorded
|
|
141
161
|
# HEAD at seal time; any that have drifted since are unsealed edits.
|
|
@@ -178,6 +198,8 @@ def main():
|
|
|
178
198
|
'exit_code': proof['exit_code'],
|
|
179
199
|
'phase': args.phase,
|
|
180
200
|
'tdd_mode': mode,
|
|
201
|
+
'reread_files': sorted(reread_files),
|
|
202
|
+
'reread_count': len(reread_files),
|
|
181
203
|
}, 0)
|
|
182
204
|
|
|
183
205
|
if __name__ == '__main__':
|
|
@@ -11,7 +11,10 @@ tdd_mode red-green (default) requires a failing red proof and a passing
|
|
|
11
11
|
green proof. tdd_mode refactor instead requires the existing tests that
|
|
12
12
|
touch the changed code to pass on the pre-change tree and the current tree,
|
|
13
13
|
with no assertion lines removed. tdd_mode docs requires the diff to stay on
|
|
14
|
-
non-code paths.
|
|
14
|
+
non-code paths. File reads go through a per-process cache keyed by the
|
|
15
|
+
P1-3 stat anchor (issue #5, P1-2 follow-up): the same unchanged file is
|
|
16
|
+
read from disk once per run and served from memory on later reads; a
|
|
17
|
+
changed file is re-read. A semantically wrong but plausible expected value is not
|
|
15
18
|
decidable from the diff in general; gate v2 still catches three decidable
|
|
16
19
|
shapes of it: expected values taken from the implementation's own constants
|
|
17
20
|
(the expectation moves when the constant is edited), assertions inside
|
|
@@ -66,17 +69,21 @@ def read_json(path: Path) -> dict:
|
|
|
66
69
|
|
|
67
70
|
|
|
68
71
|
def read_text(path: Path) -> str:
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
72
|
+
return _read_cache.read_text(path)
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
# One cache per gate run (reset in main): the same source file is read
|
|
76
|
+
# several times in one run — symbols, implementation sources, test
|
|
77
|
+
# review — and the cache serves the repeats from memory while the stat
|
|
78
|
+
# anchor still matches (issue #5, P1-2 follow-up to #34). See
|
|
79
|
+
# tdd_lib.FileReadCache for the invalidation contract.
|
|
80
|
+
_read_cache = tdd_lib.FileReadCache()
|
|
73
81
|
|
|
74
82
|
|
|
75
83
|
def review_test_file(project_root: Path, relative_path: str, implementation_sources: list[str] | None = None) -> list[dict]:
|
|
76
84
|
path = project_root / relative_path
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
except OSError:
|
|
85
|
+
content = _read_cache.get(path)
|
|
86
|
+
if content is None:
|
|
80
87
|
return [{"severity": "fail", "code": "test_unreadable", "path": relative_path}]
|
|
81
88
|
findings = []
|
|
82
89
|
for finding in tdd_lib.analyze_test(content, implementation_sources=implementation_sources):
|
|
@@ -226,6 +233,8 @@ def refactor_findings(project_root: Path, diff_ref: str, code_files: list[str],
|
|
|
226
233
|
|
|
227
234
|
|
|
228
235
|
def main() -> None:
|
|
236
|
+
global _read_cache
|
|
237
|
+
_read_cache = tdd_lib.FileReadCache()
|
|
229
238
|
parser = argparse.ArgumentParser()
|
|
230
239
|
parser.add_argument("--project-root", default=".")
|
|
231
240
|
parser.add_argument("--change-id", required=True)
|
|
@@ -344,6 +353,7 @@ def main() -> None:
|
|
|
344
353
|
"changed_files": files,
|
|
345
354
|
"code_files": code_files,
|
|
346
355
|
"test_files": test_files,
|
|
356
|
+
"read_cache": _read_cache.stats(),
|
|
347
357
|
"findings": findings,
|
|
348
358
|
"recorded_at": datetime.now(timezone.utc).isoformat(),
|
|
349
359
|
}
|
|
@@ -357,6 +367,7 @@ def main() -> None:
|
|
|
357
367
|
"verdict": verdict,
|
|
358
368
|
"tdd_mode": mode,
|
|
359
369
|
"proof_file": str(proof_file),
|
|
370
|
+
"read_cache": _read_cache.stats(),
|
|
360
371
|
"failures": sum(1 for finding in findings if finding.get("severity") == "fail"),
|
|
361
372
|
"warnings": sum(1 for finding in findings if finding.get("severity") == "warn"),
|
|
362
373
|
"failed_codes": [
|
|
@@ -26,16 +26,25 @@ Checks (findings are structured JSON, severity fail | warn):
|
|
|
26
26
|
(--max-files / --max-lines, or status.json cr_max_files /
|
|
27
27
|
cr_max_lines). Over budget is a warn with a split-the-spec
|
|
28
28
|
recommendation; over 2x budget fails.
|
|
29
|
+
5. Seal partition (reporting only, never a verdict input) - the proof
|
|
30
|
+
splits the changed files into those still byte-identical to the
|
|
31
|
+
green TDD seal (file_hashes in tdd/green-0.json) and those that
|
|
32
|
+
drifted or were never sealed. A review pass re-reads only the
|
|
33
|
+
drifted set; the sealed set is exactly what the TDD gate verified.
|
|
29
34
|
|
|
30
35
|
Self-contained on purpose: this atom does not import tdd_lib so it can
|
|
31
36
|
run standalone in a reviewed tree. The file/symbol/assertion semantics
|
|
32
|
-
mirror tdd_lib so verdicts agree with I6c.
|
|
37
|
+
mirror tdd_lib so verdicts agree with I6c. File reads go through a
|
|
38
|
+
per-process cache keyed by the P1-3 stat anchor (issue #5, P1-2
|
|
39
|
+
follow-up): the same unchanged file is read from disk once per run
|
|
40
|
+
and served from memory on later reads; a changed file is re-read.
|
|
33
41
|
|
|
34
42
|
The proof is written to specs/changes/<change_id>/cr/review-0.json; that
|
|
35
43
|
directory is excluded from the reviewed diff.
|
|
36
44
|
"""
|
|
37
45
|
import argparse
|
|
38
46
|
import fnmatch
|
|
47
|
+
import hashlib
|
|
39
48
|
import json
|
|
40
49
|
import os
|
|
41
50
|
import re
|
|
@@ -113,10 +122,64 @@ def assertion_count(content: str) -> int:
|
|
|
113
122
|
|
|
114
123
|
|
|
115
124
|
def read_text(path: Path) -> str:
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
125
|
+
return _read_cache.read_text(path)
|
|
126
|
+
|
|
127
|
+
|
|
128
|
+
class FileReadCache:
|
|
129
|
+
"""Per-process cache of file text, keyed by the stat anchor.
|
|
130
|
+
|
|
131
|
+
Local mirror of tdd_lib.FileReadCache (issue #5, P1-2 follow-up to
|
|
132
|
+
#34) — this atom stays self-contained and does not import tdd_lib.
|
|
133
|
+
One gate run reads the same source file several times: spec
|
|
134
|
+
alignment, then test follow (which re-reads every test file once
|
|
135
|
+
per changed code file). The first read stores the text under the
|
|
136
|
+
file's stat anchor (size, mtime_ns, inode, the P1-3 identity); a
|
|
137
|
+
later read whose anchor still matches is served from memory without
|
|
138
|
+
touching the bytes. Any anchor mismatch re-reads from disk, so the
|
|
139
|
+
cache can never serve stale content; a mid-read change is retried
|
|
140
|
+
once, so the stored anchor always describes the stored text.
|
|
141
|
+
Unreadable files are never cached and never counted as disk reads.
|
|
142
|
+
`stats()` reports {"disk_reads": n, "cache_hits": n} for the proof.
|
|
143
|
+
"""
|
|
144
|
+
|
|
145
|
+
def __init__(self) -> None:
|
|
146
|
+
self._entries: dict[str, tuple[list, str]] = {}
|
|
147
|
+
self.disk_reads = 0
|
|
148
|
+
self.cache_hits = 0
|
|
149
|
+
|
|
150
|
+
def get(self, path: Path) -> str | None:
|
|
151
|
+
"""Cached text for `path`, or None when it is unreadable."""
|
|
152
|
+
anchor = stat_anchor(path)
|
|
153
|
+
key = str(path)
|
|
154
|
+
entry = self._entries.get(key)
|
|
155
|
+
if anchor is not None and entry is not None and list(entry[0]) == list(anchor):
|
|
156
|
+
self.cache_hits += 1
|
|
157
|
+
return entry[1]
|
|
158
|
+
try:
|
|
159
|
+
data = path.read_bytes()
|
|
160
|
+
after = stat_anchor(path)
|
|
161
|
+
if anchor is not None and after is not None and list(after) != list(anchor):
|
|
162
|
+
data = path.read_bytes()
|
|
163
|
+
after = stat_anchor(path)
|
|
164
|
+
except OSError:
|
|
165
|
+
return None
|
|
166
|
+
self.disk_reads += 1
|
|
167
|
+
text = data.decode("utf-8", errors="replace")
|
|
168
|
+
if after is not None:
|
|
169
|
+
self._entries[key] = (after, text)
|
|
170
|
+
return text
|
|
171
|
+
|
|
172
|
+
def read_text(self, path: Path) -> str:
|
|
173
|
+
"""Cached text for `path`, or "" when it is unreadable."""
|
|
174
|
+
text = self.get(path)
|
|
175
|
+
return "" if text is None else text
|
|
176
|
+
|
|
177
|
+
def stats(self) -> dict:
|
|
178
|
+
return {"disk_reads": self.disk_reads, "cache_hits": self.cache_hits}
|
|
179
|
+
|
|
180
|
+
|
|
181
|
+
# One cache per gate run (reset in main).
|
|
182
|
+
_read_cache = FileReadCache()
|
|
120
183
|
|
|
121
184
|
|
|
122
185
|
def read_json(path: Path) -> dict:
|
|
@@ -200,6 +263,74 @@ def assertion_delta(project_root: Path, diff_ref: str, test_files: list, deleted
|
|
|
200
263
|
return added, removed
|
|
201
264
|
|
|
202
265
|
|
|
266
|
+
def stat_anchor(path: Path):
|
|
267
|
+
"""[size, mtime_ns, inode] for a file, or None when it cannot be stat'ed.
|
|
268
|
+
|
|
269
|
+
Same anchor the TDD seal records in the green proof's snapshot scope
|
|
270
|
+
(issue #5, P1-3): a matching anchor means the sealed bytes are still
|
|
271
|
+
there without re-reading them. Kept local — this atom stays
|
|
272
|
+
self-contained and does not import tdd_lib.
|
|
273
|
+
"""
|
|
274
|
+
try:
|
|
275
|
+
st = path.stat()
|
|
276
|
+
except OSError:
|
|
277
|
+
return None
|
|
278
|
+
if not path.is_file():
|
|
279
|
+
return None
|
|
280
|
+
return [st.st_size, st.st_mtime_ns, st.st_ino]
|
|
281
|
+
|
|
282
|
+
|
|
283
|
+
def seal_partition(project_root: Path, change_dir: Path, files: list):
|
|
284
|
+
"""Split changed files by whether they still match the green seal.
|
|
285
|
+
|
|
286
|
+
The green TDD proof (tdd/green-0.json) records the sha256 of every
|
|
287
|
+
file I6a sealed when the proof was produced (issue #5, P1-2). A
|
|
288
|
+
changed file whose current bytes still hash to the sealed value is
|
|
289
|
+
byte-for-byte what the TDD gate verified, so a review pass does not
|
|
290
|
+
need to re-read it; anything else — drifted after the seal, or never
|
|
291
|
+
sealed — must be re-read. Returns None when no usable green seal
|
|
292
|
+
exists (the review then re-reads everything).
|
|
293
|
+
|
|
294
|
+
When the green proof carries the seal-time stat anchors (snapshot
|
|
295
|
+
scope file_stats, issue #5 P1-3), a file whose anchor still matches
|
|
296
|
+
is accepted without hashing it at all; only anchor mismatches are
|
|
297
|
+
re-read and hashed. `stat_checked` counts the files accepted this
|
|
298
|
+
way; `reread_count` keeps its P1-2 meaning (files the review must
|
|
299
|
+
re-read) and is unchanged by the fast path. Proofs sealed before
|
|
300
|
+
the anchors existed fall back to hashing every file.
|
|
301
|
+
"""
|
|
302
|
+
green = read_json(change_dir / "tdd" / "green-0.json")
|
|
303
|
+
hashes = green.get("file_hashes")
|
|
304
|
+
if not isinstance(hashes, dict) or not hashes:
|
|
305
|
+
return None
|
|
306
|
+
scope = green.get("snapshot_scope")
|
|
307
|
+
stats = scope.get("file_stats") if isinstance(scope, dict) else None
|
|
308
|
+
sealed, drifted = [], []
|
|
309
|
+
stat_checked = 0
|
|
310
|
+
for relative in files:
|
|
311
|
+
full = project_root / relative
|
|
312
|
+
anchor = stats.get(relative) if isinstance(stats, dict) else None
|
|
313
|
+
if anchor is not None:
|
|
314
|
+
current = stat_anchor(full)
|
|
315
|
+
if current is not None and list(current) == list(anchor):
|
|
316
|
+
stat_checked += 1
|
|
317
|
+
sealed.append(relative)
|
|
318
|
+
continue
|
|
319
|
+
digest = None
|
|
320
|
+
if full.is_file():
|
|
321
|
+
digest = hashlib.sha256(full.read_bytes()).hexdigest()
|
|
322
|
+
if digest is not None and hashes.get(relative) == digest:
|
|
323
|
+
sealed.append(relative)
|
|
324
|
+
else:
|
|
325
|
+
drifted.append(relative)
|
|
326
|
+
return {
|
|
327
|
+
"sealed_unchanged": sealed,
|
|
328
|
+
"seal_drifted": drifted,
|
|
329
|
+
"reread_count": len(drifted),
|
|
330
|
+
"stat_checked": stat_checked,
|
|
331
|
+
}
|
|
332
|
+
|
|
333
|
+
|
|
203
334
|
def tokenize(text: str) -> set:
|
|
204
335
|
tokens = set()
|
|
205
336
|
for raw in re.findall(r"[A-Za-z_][A-Za-z0-9_]*", text):
|
|
@@ -248,6 +379,8 @@ def iter_repo_tests(project_root: Path) -> list:
|
|
|
248
379
|
|
|
249
380
|
|
|
250
381
|
def main() -> None:
|
|
382
|
+
global _read_cache
|
|
383
|
+
_read_cache = FileReadCache()
|
|
251
384
|
parser = argparse.ArgumentParser()
|
|
252
385
|
parser.add_argument("--project-root", default=".")
|
|
253
386
|
parser.add_argument("--change-id", required=True)
|
|
@@ -416,6 +549,8 @@ def main() -> None:
|
|
|
416
549
|
"changed_files": all_files,
|
|
417
550
|
"code_files": code_files,
|
|
418
551
|
"test_files": test_files,
|
|
552
|
+
"read_cache": _read_cache.stats(),
|
|
553
|
+
"seal_partition": seal_partition(project_root, change_dir, all_files),
|
|
419
554
|
"spec_aligned_files": aligned_files,
|
|
420
555
|
"assertions": {"added": added_assertions, "removed": removed_assertions},
|
|
421
556
|
"budget": budget,
|
|
@@ -432,6 +567,7 @@ def main() -> None:
|
|
|
432
567
|
"ok": proof["ok"],
|
|
433
568
|
"verdict": verdict,
|
|
434
569
|
"proof_file": str(proof_file),
|
|
570
|
+
"read_cache": _read_cache.stats(),
|
|
435
571
|
"failures": sum(1 for f in findings if f.get("severity") == "fail"),
|
|
436
572
|
"warnings": sum(1 for f in findings if f.get("severity") == "warn"),
|
|
437
573
|
"failed_codes": [f.get("code") for f in findings if f.get("severity") == "fail"],
|
package/xflow/atoms/tdd_lib.py
CHANGED
|
@@ -142,6 +142,126 @@ def sha256_bytes(data: bytes) -> str:
|
|
|
142
142
|
return hashlib.sha256(data).hexdigest()
|
|
143
143
|
|
|
144
144
|
|
|
145
|
+
def stat_anchor(path: Path) -> list | None:
|
|
146
|
+
"""[size, mtime_ns, inode] for a file, or None when it cannot be stat'ed.
|
|
147
|
+
|
|
148
|
+
The anchor is the cheap identity a later gate stats first: when the
|
|
149
|
+
anchor still matches the one recorded at seal time, the bytes the seal
|
|
150
|
+
hashed cannot have changed without a same-size same-nanosecond rewrite
|
|
151
|
+
(the same heuristic git uses for its index). Any anchor mismatch falls
|
|
152
|
+
back to a full re-read and re-hash, so the check never weakens.
|
|
153
|
+
"""
|
|
154
|
+
try:
|
|
155
|
+
st = path.stat()
|
|
156
|
+
except OSError:
|
|
157
|
+
return None
|
|
158
|
+
if not path.is_file():
|
|
159
|
+
return None
|
|
160
|
+
return [st.st_size, st.st_mtime_ns, st.st_ino]
|
|
161
|
+
|
|
162
|
+
|
|
163
|
+
def read_with_anchor(path: Path) -> tuple[bytes, list | None]:
|
|
164
|
+
"""Read a file and return the bytes plus the stat anchor they were read under.
|
|
165
|
+
|
|
166
|
+
Stats before and after the read; when the file changed mid-read the
|
|
167
|
+
read is retried once, so the returned anchor always describes the
|
|
168
|
+
returned bytes.
|
|
169
|
+
"""
|
|
170
|
+
before = stat_anchor(path)
|
|
171
|
+
data = path.read_bytes()
|
|
172
|
+
after = stat_anchor(path)
|
|
173
|
+
if before is not None and after is not None and before != after:
|
|
174
|
+
data = path.read_bytes()
|
|
175
|
+
after = stat_anchor(path)
|
|
176
|
+
return data, after
|
|
177
|
+
|
|
178
|
+
|
|
179
|
+
class FileReadCache:
|
|
180
|
+
"""Per-process cache of file text, keyed by the stat anchor.
|
|
181
|
+
|
|
182
|
+
One gate run reads the same source file several times — symbols,
|
|
183
|
+
assertions, references (issue #5, P1-2 follow-up to #34). The first
|
|
184
|
+
read stores the text under the file's stat anchor (size, mtime_ns,
|
|
185
|
+
inode, the P1-3 identity); a later read whose anchor still matches
|
|
186
|
+
is served from memory without touching the bytes. Any anchor
|
|
187
|
+
mismatch — the file changed, or it was never read — re-reads from
|
|
188
|
+
disk, so the cache can never serve stale content; a mid-read change
|
|
189
|
+
is retried once, like read_with_anchor, so the stored anchor always
|
|
190
|
+
describes the stored text. Unreadable files are never cached and
|
|
191
|
+
never counted as disk reads.
|
|
192
|
+
|
|
193
|
+
`stats()` reports {"disk_reads": n, "cache_hits": n} so a proof can
|
|
194
|
+
show the skipped re-reads; the counts are per cache instance, and a
|
|
195
|
+
gate creates one instance per run.
|
|
196
|
+
"""
|
|
197
|
+
|
|
198
|
+
def __init__(self) -> None:
|
|
199
|
+
self._entries: dict[str, tuple[list, str]] = {}
|
|
200
|
+
self.disk_reads = 0
|
|
201
|
+
self.cache_hits = 0
|
|
202
|
+
|
|
203
|
+
def get(self, path) -> str | None:
|
|
204
|
+
"""Cached text for `path`, or None when it is unreadable."""
|
|
205
|
+
path = Path(path)
|
|
206
|
+
anchor = stat_anchor(path)
|
|
207
|
+
key = str(path)
|
|
208
|
+
entry = self._entries.get(key)
|
|
209
|
+
if anchor is not None and entry is not None and list(entry[0]) == list(anchor):
|
|
210
|
+
self.cache_hits += 1
|
|
211
|
+
return entry[1]
|
|
212
|
+
try:
|
|
213
|
+
data = path.read_bytes()
|
|
214
|
+
after = stat_anchor(path)
|
|
215
|
+
if anchor is not None and after is not None and list(after) != list(anchor):
|
|
216
|
+
data = path.read_bytes()
|
|
217
|
+
after = stat_anchor(path)
|
|
218
|
+
except OSError:
|
|
219
|
+
return None
|
|
220
|
+
self.disk_reads += 1
|
|
221
|
+
text = data.decode("utf-8", errors="replace")
|
|
222
|
+
if after is not None:
|
|
223
|
+
self._entries[key] = (after, text)
|
|
224
|
+
return text
|
|
225
|
+
|
|
226
|
+
def read_text(self, path) -> str:
|
|
227
|
+
"""Cached text for `path`, or "" when it is unreadable."""
|
|
228
|
+
text = self.get(path)
|
|
229
|
+
return "" if text is None else text
|
|
230
|
+
|
|
231
|
+
def stats(self) -> dict:
|
|
232
|
+
return {"disk_reads": self.disk_reads, "cache_hits": self.cache_hits}
|
|
233
|
+
|
|
234
|
+
|
|
235
|
+
def verify_hashes_by_stat(root: Path, hashes: dict, stats: dict | None) -> dict:
|
|
236
|
+
"""Check recorded sha256 hashes, re-reading only files whose stat moved.
|
|
237
|
+
|
|
238
|
+
`stats` maps rel -> [size, mtime_ns, ino] recorded when the hash was
|
|
239
|
+
computed (issue #5, P1-3). A file whose current anchor matches is
|
|
240
|
+
accepted without reading its bytes; anything else — missing anchor,
|
|
241
|
+
stat mismatch, missing file — is re-read and hashed. Returns
|
|
242
|
+
{"mismatched": [...], "reread": [...], "stat_checked": n}.
|
|
243
|
+
"""
|
|
244
|
+
mismatched: list[str] = []
|
|
245
|
+
reread: list[str] = []
|
|
246
|
+
stat_checked = 0
|
|
247
|
+
anchors = stats if isinstance(stats, dict) else {}
|
|
248
|
+
for rel, digest in hashes.items():
|
|
249
|
+
full = root / rel
|
|
250
|
+
anchor = anchors.get(rel)
|
|
251
|
+
if anchor is not None:
|
|
252
|
+
current = stat_anchor(full)
|
|
253
|
+
if current is not None and list(current) == list(anchor):
|
|
254
|
+
stat_checked += 1
|
|
255
|
+
continue
|
|
256
|
+
reread.append(rel)
|
|
257
|
+
if not full.is_file():
|
|
258
|
+
mismatched.append(rel)
|
|
259
|
+
continue
|
|
260
|
+
if sha256_bytes(full.read_bytes()) != digest:
|
|
261
|
+
mismatched.append(rel)
|
|
262
|
+
return {"mismatched": sorted(mismatched), "reread": sorted(reread), "stat_checked": stat_checked}
|
|
263
|
+
|
|
264
|
+
|
|
145
265
|
# Optional binding fields join the payload only when present, so proofs
|
|
146
266
|
# sealed before the field existed keep verifying byte-for-byte.
|
|
147
267
|
OPTIONAL_BINDING_FIELDS = {"snapshot_scope"}
|
|
@@ -313,16 +433,23 @@ def write_snapshot(project_root: Path, change_id: str, phase: str, attempt: int,
|
|
|
313
433
|
target.parent.mkdir(parents=True, exist_ok=True)
|
|
314
434
|
target.write_bytes(data)
|
|
315
435
|
return hashes, None
|
|
436
|
+
file_stats: dict[str, list] = {}
|
|
437
|
+
copy_stats: dict[str, list] = {}
|
|
316
438
|
for rel in scope["files"]:
|
|
317
439
|
full = project_root / rel
|
|
318
440
|
if full.is_symlink() or not full.is_file():
|
|
319
441
|
continue
|
|
320
|
-
data = full
|
|
442
|
+
data, anchor = read_with_anchor(full)
|
|
443
|
+
if anchor is not None:
|
|
444
|
+
file_stats[rel] = anchor
|
|
321
445
|
hashes[rel] = sha256_bytes(data)
|
|
322
446
|
target = destination / rel
|
|
323
447
|
target.parent.mkdir(parents=True, exist_ok=True)
|
|
324
448
|
target.write_bytes(data)
|
|
325
|
-
|
|
449
|
+
copy_anchor = stat_anchor(target)
|
|
450
|
+
if copy_anchor is not None:
|
|
451
|
+
copy_stats[rel] = copy_anchor
|
|
452
|
+
scope = {**scope, "files": sorted(hashes), "file_stats": file_stats, "copy_stats": copy_stats}
|
|
326
453
|
return hashes, scope
|
|
327
454
|
|
|
328
455
|
|
package/xflow/yolo/SKILL.md
CHANGED
|
@@ -61,6 +61,14 @@ pass that writes the plan also writes the failing test, so `tdd-red` only
|
|
|
61
61
|
runs and seals it — no second exploration pass, and the issue/branch
|
|
62
62
|
ceremony no longer sits between planning and the red proof.
|
|
63
63
|
|
|
64
|
+
In `verify-consistency`, run the deterministic gates first. The J6 proof
|
|
65
|
+
(`specs/changes/${CHANGE_ID}/cr/review-0.json`) carries a `seal_partition`:
|
|
66
|
+
files whose bytes still match the green proof's sealed hashes
|
|
67
|
+
(`sealed_unchanged`) are exactly what the TDD gate verified — the review
|
|
68
|
+
pass does not re-read them. Only `seal_drifted` files (changed after the
|
|
69
|
+
green seal, or never sealed) get a fresh read. When `seal_partition` is
|
|
70
|
+
null there is no green seal to trust, and the review re-reads everything.
|
|
71
|
+
|
|
64
72
|
The workflow YAML is the canonical execution source for atom order, gates, and
|
|
65
73
|
artifact checks. This skill only explains the contract.
|
|
66
74
|
|