@surlymochan/evalid 0.4.1 → 0.5.0

package/CHANGELOG.md

@@ -5,6 +5,14 @@ All notable changes to this project are documented here. Detailed notes live in
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-11-06
+
+### Added
+- **CR review seal partition (issue #5, P1-2)**: the J6 review proof (`cr/review-0.json`) now splits the changed files into `sealed_unchanged` (bytes still identical to the green TDD proof's sealed hashes) and `seal_drifted`, with a `reread_count`. The yolo skill's verify-consistency guidance uses the partition so the review pass re-reads only drifted files instead of everything the gate already verified. Reporting only — verdicts are unchanged, and without a green seal the partition is `null` and everything is re-read.
+- **Bench Verified set extended to 15 instances (issue #5, P2-2)**: `bench/swebench/instances.json` grows from 12 to 15, adding sympy-15599, sympy-19040 and sympy-24213, and a structural validation of the instance set (`bench/swebench/test_instances.py`) now runs in CI. Gold-patch checks for the three new instances stay pending the benchmark environment.
+- **Bench report aggregates median (IQR) (issue #5, P2-1)**: `bench/swebench/report.py` now aggregates per-arm wall-clock, token and cost results with q1/median/q3 and presents them as median (IQR); reps isolation and the aggregation itself are pinned by `bench/swebench/test_reps_report.py`. Live multi-rep runs stay pending the benchmark environment.
+- **P0-2 acceptance pinned against saved pre-fix patches (issue #5, P0-2)**: regression tests in `bench/swebench/test_diff_filter.py` run the saved pre-fix patches through the evidence filter (`specs/`, `.xflow/`, `.as-xflow/` excluded from solution patches). The saved 9.7 MB evidence-laden patch filters down to 4.3 KB, and every saved patch lands under the 200 KB acceptance line.
+
 ## [0.4.1] - 2026-11-06
 
 ### Fixed

package/RELEASE_NOTES.md

@@ -1,5 +1,19 @@
 # Release Notes
 
+## 0.5.0 - Review re-reads only drifted files; the bench set grows to 15
+
+Minor release. One product change — the CR review now spends its re-reads only where the seal says bytes moved — plus three bench-hygiene landings from the issue #5 backlog, each pinned by tests rather than by a live run.
+
+**Review re-reads only drifted files (#32, issue #5 P1-2).** The J6 review proof (`cr/review-0.json`) now carries a seal partition: every changed file is classified `sealed_unchanged` (bytes still identical to the green TDD proof's sealed hashes) or `seal_drifted`, with a `reread_count`. The yolo verify-consistency guidance consumes the partition, so the review pass re-reads only the drifted files instead of everything the gate already verified. This is reporting only — verdicts are unchanged, and without a green seal the partition is `null` and everything is re-read, exactly as before.
+
+**The Verified bench set grows from 12 to 15 instances (#30, issue #5 P2-2).** `bench/swebench/instances.json` adds sympy-15599, sympy-19040 and sympy-24213, and a structural validation of the instance set (`bench/swebench/test_instances.py`) now runs in CI, so a malformed instance entry fails the build instead of a bench run. Gold-patch checks for the three new instances are pending the benchmark environment.
+
+**Bench reports now aggregate as median (IQR) (#31, issue #5 P2-1).** `bench/swebench/report.py` aggregates per-arm wall-clock, token and cost results with q1/median/q3 and presents them in median (IQR) form. Reps isolation — repeat runs of the same cell never share grading state — and the aggregation itself are pinned by `bench/swebench/test_reps_report.py`.
+
+**P0-2 acceptance is now pinned by regression tests (#29, issue #5 P0-2).** The evidence exclusion itself landed earlier (solution patches drop `specs/`, `.xflow/` and `.as-xflow/`); what was missing was a test that would notice if it regressed. `bench/swebench/test_diff_filter.py` now runs the saved pre-fix patches through the filter: the saved 9.7 MB evidence-laden patch comes out at 4.3 KB, and every saved patch lands under the 200 KB acceptance line.
+
+Acceptance debt, stated plainly: the live-run acceptance for P1-2, P2-1 and P2-2 — multi-rep bench runs on the restored environment, including the wall-clock/token lines from issue #5 and gold checks for the new instances — is still pending the benchmark environment. No performance numbers are claimed in this release; what shipped is the reporting, the larger instance set, and the tests that pin their behaviour.
+
 ## 0.4.1 - Failed runs report phase costs; Node 24 test counts fail closed
 
 Patch release. Two fixes, both about the gate trusting evidence it had not actually read.

package/docs/launch-dossier.md

@@ -1,6 +1,6 @@
 # evalid Launch Dossier
 
-Generated: 2026-11-05T17:36:26.724Z
+Generated: 2026-11-05T19:40:37.700Z
 Mode: post_publish
 Verdict: conditionally ready: engine and adoption evidence are strong, registry publication remains required for a splash launch
 Launch ready: not yet
@@ -117,7 +117,7 @@ Evidence:
 - Adoption records: 1; status: pass
 - Goal audit: pass; goal: Keep evalid credible as an open-source agent delivery runtime: repo-owned workflow contracts, executable verification gates, durable project context, and honest benchmark evidence.
 - Package evidence: missing; source: registry_json
-- Package name/version: @surlymochan/evalid@0.4.1
+- Package name/version: @surlymochan/evalid@0.5.0
 
 ## Passing Evidence
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@surlymochan/evalid",
-  "version": "0.4.1",
+  "version": "0.5.0",
   "description": "Claimed != verified: a runtime verification layer for coding agents. It seals each test run and replays it from a clean snapshot, with a zero-judge gate pipeline (spec clarification, code review, TDD proof) and the xflow CLI.",
   "keywords": [
     "ai-agents",

package/xflow/atoms/j6_cr_review.py

@@ -26,6 +26,11 @@ Checks (findings are structured JSON, severity fail | warn):
    (--max-files / --max-lines, or status.json cr_max_files /
    cr_max_lines). Over budget is a warn with a split-the-spec
    recommendation; over 2x budget fails.
+5. Seal partition (reporting only, never a verdict input) - the proof
+   splits the changed files into those still byte-identical to the
+   green TDD seal (file_hashes in tdd/green-0.json) and those that
+   drifted or were never sealed. A review pass re-reads only the
+   drifted set; the sealed set is exactly what the TDD gate verified.
 
 Self-contained on purpose: this atom does not import tdd_lib so it can
 run standalone in a reviewed tree. The file/symbol/assertion semantics
@@ -36,6 +41,7 @@ directory is excluded from the reviewed diff.
 """
 import argparse
 import fnmatch
+import hashlib
 import json
 import os
 import re
@@ -200,6 +206,38 @@ def assertion_delta(project_root: Path, diff_ref: str, test_files: list, deleted
     return added, removed
 
 
+def seal_partition(project_root: Path, change_dir: Path, files: list):
+    """Split changed files by whether they still match the green seal.
+
+    The green TDD proof (tdd/green-0.json) records the sha256 of every
+    file I6a sealed when the proof was produced (issue #5, P1-2). A
+    changed file whose current bytes still hash to the sealed value is
+    byte-for-byte what the TDD gate verified, so a review pass does not
+    need to re-read it; anything else — drifted after the seal, or never
+    sealed — must be re-read. Returns None when no usable green seal
+    exists (the review then re-reads everything).
+    """
+    green = read_json(change_dir / "tdd" / "green-0.json")
+    hashes = green.get("file_hashes")
+    if not isinstance(hashes, dict) or not hashes:
+        return None
+    sealed, drifted = [], []
+    for relative in files:
+        full = project_root / relative
+        digest = None
+        if full.is_file():
+            digest = hashlib.sha256(full.read_bytes()).hexdigest()
+        if digest is not None and hashes.get(relative) == digest:
+            sealed.append(relative)
+        else:
+            drifted.append(relative)
+    return {
+        "sealed_unchanged": sealed,
+        "seal_drifted": drifted,
+        "reread_count": len(drifted),
+    }
+
+
 def tokenize(text: str) -> set:
     tokens = set()
     for raw in re.findall(r"[A-Za-z_][A-Za-z0-9_]*", text):
@@ -416,6 +454,7 @@ def main() -> None:
         "changed_files": all_files,
         "code_files": code_files,
         "test_files": test_files,
+        "seal_partition": seal_partition(project_root, change_dir, all_files),
         "spec_aligned_files": aligned_files,
         "assertions": {"added": added_assertions, "removed": removed_assertions},
         "budget": budget,

package/xflow/yolo/SKILL.md

@@ -61,6 +61,14 @@ pass that writes the plan also writes the failing test, so `tdd-red` only
 runs and seals it — no second exploration pass, and the issue/branch
 ceremony no longer sits between planning and the red proof.
 
+In `verify-consistency`, run the deterministic gates first. The J6 proof
+(`specs/changes/${CHANGE_ID}/cr/review-0.json`) carries a `seal_partition`:
+files whose bytes still match the green proof's sealed hashes
+(`sealed_unchanged`) are exactly what the TDD gate verified — the review
+pass does not re-read them. Only `seal_drifted` files (changed after the
+green seal, or never sealed) get a fresh read. When `seal_partition` is
+null there is no green seal to trust, and the review re-reads everything.
+
 The workflow YAML is the canonical execution source for atom order, gates, and
 artifact checks. This skill only explains the contract.