@surlymochan/evalid 0.5.16 → 0.5.18

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -3,6 +3,26 @@
3
3
  All notable changes to this project are documented here. Detailed notes live in
4
4
  [RELEASE_NOTES.md](RELEASE_NOTES.md).
5
5
 
6
+ ## [0.5.18] - 2026-11-09
7
+
8
+ ### Added
9
+ - **Export proof verdicts as SARIF for code scanning (#92, closes #73)**: sealed-proof failures now surface as code scanning alerts, not just a red check and a PR comment. `action/sarif.mjs` maps each proof-failure verdict to a SARIF 2.1.0 rule — binding_mismatch / snapshot_mismatch / green_tree_mismatch / reexec_* and the other validator failures (level error; warning for tests_skipped_or_todo / tests_counts_unknown) — one result per named file (file-level location; the validator records no line numbers), degrading to the proof file itself when a verdict has no file location. Stable `partialFingerprints` (`change-id/phase/attempt/verdict/file`) so re-runs of the same proof update the existing alert instead of opening a duplicate; merging upserts per proof, so a valid proof clears its alert. `valid` / `not_required` / infrastructure verdicts produce zero results. `verify-proof.mjs` gains `--sarif-out <path>` (with matching `sarif-out` input and `sarif_path` / `sarif_written` outputs in action.yml); `proof-ledger.yml` passes `--sarif-out evalid-proof.sarif` per proof and uploads with `github/codeql-action/upload-sarif` (`security-events: write`; upload runs on `always()` so it fires exactly when a proof failed, and is `continue-on-error` so fork PRs stay green). Test evidence: 11 new tests in `test/github-action-sarif.test.js` (verdict→rule mapping + levels, per-file locations, no-file degradation, fingerprint stability, merge upsert / valid-clears-alert, SARIF 2.1.0 shape, action.yml declarations, CLI end-to-end on a real sealed proof); PR CI runs verify 42567046823 and ledger 42567046908 both pass; merged main CI run 42567609280 success.
10
+
11
+ Issue #73 — **closed with end-to-end evidence** (comment 2026-11-09T05:46:16Z, demo PR #94, closed unmerged): a sealed green proof with a tampered `test/demo-sarif.test.js` went red — ledger run 42568411445 fail, verdict `green_tree_mismatch`, files `[test/demo-sarif.test.js]` only; SARIF upload success, analysis 2124250892 (results_count 1), code scanning alert #1 visible in the Security tab (rule `evalid/green-tree-mismatch`, severity error). Re-run (attempt 2, analysis 2124258699) opened no duplicate alert, and the same fingerprint from the earlier demo PR #93 did not create a new alert either. Restoring the test file cleared it — ledger run 42568823086 pass, verdict `valid`, analysis 2124265677 (results_count 0), alert #1 state `fixed` on refs/pull/94/merge. A green PR produces no alert (PR #92 itself has no sealed proof: no SARIF upload, no analysis; main branch alerts 0).
12
+
13
+ Acceptance debt, stated plainly: the bench-number acceptance for P1-2 / P1-3 / P2-1 / P2-3 still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
14
+
15
+ ## [0.5.17] - 2026-11-09
16
+
17
+ ### Added
18
+ - **M3 slice 3: wire I6d into xflow audit replay and the I6c gate (#89)**: `xflow audit` now runs the I6d gate in the scratch clone when the diff touches Python test files (or a pytest claim exists) and the audited repo carries the I6d plugin — an unexecuted assertion inside the diff flips the verdict to `mismatch` (exit 1) even when the replay itself is fully green; no plugin / no Python test change / pytest unavailable is recorded `not applicable` and leaves the verdict alone (`--no-i6d` skips it). The I6c gate folds `specs/changes/<id>/tdd/i6d-0.json` evidence into fail findings (code `unexecuted_assertions`, one line per assertion). Test evidence: local full `npm test` 496 tests / 495 pass / 0 fail / 1 skipped (pre-existing); merged main CI run 42551898032 success.
19
+ - **M3 slice 4: acceptance demo for the I6d assertion-execution gate (#90)**: `scripts/demo-i6d-gate.sh` proves the gate end to end through the real `xflow audit` CLI (not the JS API) with a RED/GREEN double control — RED: a diff adding a never-called callback (`assert value == 42` never executes) with a claimed all-green pytest run is judged `mismatch`, exit 1, `i6d: unexecuted_assertions` (1/3 executed, 1 unexecuted); GREEN: a diff adding only really-executed assertions is judged `verified`, exit 0, `i6d: pass`. Test evidence: new test `test/i6d-acceptance-demo.test.js` 1/1 pass; `test/xflow-audit.test.js` 15/15 pass; local full `npm test` 511 tests / 510 pass / 0 fail / 1 skipped (pre-existing); merged main CI run 42557796622 success.
20
+ - **Concurrency policy test coverage, external contribution (#88, closes #87)**: 14 new tests in `test/concurrency-policy.test.js` covering `createConcurrencyPolicy()` — default settings, configured maxima and fractional normalization, invalid limits with fallback, mission-level overrides, atom/registry parallel weights, fixed and weighted charging. No production code changed. Test evidence: contributor ran the new file 14/14 pass (full suite was blocked locally by a missing dependency on their machine); full-suite verification after merge comes from main CI run 42556898257 success.
21
+
22
+ Milestone M3 "I6d assertion-execution gate" — **accepted** 2026-11-09. Acceptance evidence (issue #69, comment 6731688418): the slice-4 demo proves the gate's acceptance standard end to end — every new/changed assertion must actually execute during the green run, and an unexecuted assertion does not count as green (RED case: assertion never executed while the tests all pass, still `mismatch`; GREEN case: all assertions executed, `verified`). Demo script `scripts/demo-i6d-gate.sh` is in main and locked into the suite by `test/i6d-acceptance-demo.test.js`; the #69 checklist box is ticked.
23
+
24
+ Acceptance debt, stated plainly: the bench-number acceptance for P1-2 / P1-3 / P2-1 / P2-3 still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
25
+
6
26
  ## [0.5.16] - 2026-11-09
7
27
 
8
28
  ### Added
package/README.md CHANGED
@@ -231,6 +231,13 @@ claims found in the supplied text), `unverifiable` (a claimed command
231
231
  could not run). Exit codes are 0 / 1 / 3 / 1 respectively (2 = misuse).
232
232
  `--json` prints the machine-readable report; `--out <file>` writes it.
233
233
 
234
+ When the diff touches Python test files and the audited repo carries the
235
+ I6d plugin (`xflow/atoms/i6d_assertion_execution.py`), the audit also runs
236
+ the assertion-execution gate: an added or changed assertion that never
237
+ executes during the green replay flips the verdict to `mismatch`, even
238
+ when every claimed run replays green. `scripts/demo-i6d-gate.sh` is a
239
+ self-contained red/green demonstration of that gate (M3).
240
+
234
241
  ## Commands
235
242
 
236
243
  The installed binary is `xflow` (`npm install -g evalid` installs the `xflow` CLI). Every
package/RELEASE_NOTES.md CHANGED
@@ -1,5 +1,29 @@
1
1
  # Release Notes
2
2
 
3
+ ## 0.5.18 - Proof verdicts as SARIF for code scanning (issue #73 closed)
4
+
5
+ Patch release. One merged PR since 0.5.17, verified end to end in CI and locally, with the Security-tab acceptance evidence from a demo PR.
6
+
7
+ **Export proof verdicts as SARIF for code scanning (#92, closes #73).** Sealed-proof failures now surface as code scanning alerts, not just a red check and a PR comment. `action/sarif.mjs` maps each proof-failure verdict to a SARIF 2.1.0 rule (binding / snapshot / green-tree / reexec failures at error level; skipped-tests and unknown-counts at warning), one result per named file, with stable fingerprints so re-runs update the existing alert instead of duplicating it and a valid proof clears its alert. `verify-proof.mjs` gains `--sarif-out`, and `proof-ledger.yml` uploads the SARIF per proof (fires exactly when a proof failed; fork PRs stay green). Test evidence: 11 new tests in `test/github-action-sarif.test.js`; PR CI runs verify 42567046823 and ledger 42567046908 both pass; merged main CI run 42567609280 success.
8
+
9
+ **Issue #73 — closed with end-to-end evidence** (demo PR #94, closed unmerged): a sealed green proof with a tampered `test/demo-sarif.test.js` went red — ledger run 42568411445 fail, verdict `green_tree_mismatch`, files `[test/demo-sarif.test.js]` only; SARIF upload success, analysis 2124250892 (results_count 1), code scanning alert #1 visible in the Security tab (rule `evalid/green-tree-mismatch`, severity error). A re-run opened no duplicate alert, and restoring the test file cleared it — ledger run 42568823086 pass, verdict `valid`, analysis 2124265677 (results_count 0), alert #1 state `fixed` on refs/pull/94/merge. A green PR produces no alert (main branch alerts 0).
10
+
11
+ Acceptance debt, stated plainly: the bench-number acceptance for P1-2, P1-3, P2-1 and P2-3 — the wall-clock/token/resolved lines from issue #5 — still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
12
+
13
+ ## 0.5.17 - I6d assertion-execution gate (milestone M3 accepted)
14
+
15
+ Patch release. Three merged PRs since 0.5.16, all verified end to end in CI and locally.
16
+
17
+ **M3 slice 3: I6d wired into `xflow audit` replay and the I6c gate (#89).** When an audited diff touches Python test files, the audit now runs the I6d gate in the scratch clone: an unexecuted assertion inside the diff flips the verdict to `mismatch` even when the replay itself is fully green; repos without the I6d plugin are unaffected. The I6c gate folds the same evidence into fail findings naming each unexecuted assertion. Test evidence: local full `npm test` 496 tests / 495 pass / 0 fail / 1 skipped (pre-existing); merged main CI run 42551898032 success.
18
+
19
+ **M3 slice 4: acceptance demo (#90).** `scripts/demo-i6d-gate.sh` proves the gate through the real `xflow audit` CLI with a RED/GREEN double control — RED: a never-called callback's `assert value == 42` with a claimed all-green pytest run is judged `mismatch`, exit 1, `i6d: unexecuted_assertions`; GREEN: only really-executed assertions are judged `verified`, exit 0, `i6d: pass`. Test evidence: new test `test/i6d-acceptance-demo.test.js` 1/1 pass; `test/xflow-audit.test.js` 15/15 pass; local full `npm test` 511 tests / 510 pass / 0 fail / 1 skipped (pre-existing); merged main CI run 42557796622 success.
20
+
21
+ **Concurrency policy test coverage, external contribution (#88, closes #87).** 14 new tests in `test/concurrency-policy.test.js` covering `createConcurrencyPolicy()` limits, weights and overrides; no production code changed. Test evidence: contributor ran the new file 14/14 pass; full-suite verification after merge comes from main CI run 42556898257 success.
22
+
23
+ **Milestone M3 "I6d assertion-execution gate" — accepted.** Evidence in issue #69, comment 6731688418: the slice-4 demo proves the acceptance standard end to end — every new/changed assertion must actually execute during the green run, and an unexecuted assertion does not count as green. The demo script is in main and locked into the suite; the #69 checklist box is ticked.
24
+
25
+ Acceptance debt, stated plainly: the bench-number acceptance for P1-2, P1-3, P2-1 and P2-3 — the wall-clock/token/resolved lines from issue #5 — still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
26
+
3
27
  ## 0.5.16 - M3 slices 1–2: pytest assertion-execution tracking, diff-scoped (milestone M3 not yet accepted)
4
28
 
5
29
  Patch release. Two merged slices since 0.5.15, both verified end to end in CI and locally.
@@ -238,6 +238,19 @@
238
238
  "tdd_mode": "red-green|refactor|docs?"
239
239
  }
240
240
  },
241
+ "I6d.tdd.assertion_execution": {
242
+ "track": "lite",
243
+ "type": "python",
244
+ "script": "xflow/atoms/i6d_assertion_execution.py",
245
+ "description": "Prove every new/changed pytest assertion really executed during the green run (M3 slice 3). Runs after I6b and before I6c, so I6c can fold its verdict into quality-0.json.",
246
+ "input_schema": {
247
+ "project_root": "string",
248
+ "change_id": "string",
249
+ "only_changed": "boolean?",
250
+ "diff_base": "string?",
251
+ "include_uncommitted": "boolean?"
252
+ }
253
+ },
241
254
  "J4a.spec_consistency.structural": {
242
255
  "track": "lite",
243
256
  "type": "python",
package/bin/xflow.js CHANGED
@@ -5810,6 +5810,8 @@ async function handleAudit(args) {
5810
5810
  headRef,
5811
5811
  claimsText,
5812
5812
  timeoutMs,
5813
+ python: process.env.PYTHON || 'python3',
5814
+ i6d: !parsedArgs.no_i6d,
5813
5815
  keepWorkdir: Boolean(parsedArgs.keep_workdir),
5814
5816
  });
5815
5817
  if (parsedArgs.out) writeFileSync(resolve(parsedArgs.out), `${JSON.stringify(report, null, 2)}\n`);
@@ -6740,7 +6742,11 @@ Audit any git diff / PR without adopting evalid workflows: extract the test
6740
6742
  runs the change claims to have passed, replay each one in a clean clone at
6741
6743
  the change's head, and compare exit codes and pass/fail counts against the
6742
6744
  claim. Diffs carrying an evalid proof seal are replayed through the existing
6743
- i6b seal validator. No seal semantics, CLI name, or bin path changes.
6745
+ i6b seal validator. When the diff touches Python test files and the audited
6746
+ repo carries the I6d plugin, the I6d assertion-execution gate also runs in
6747
+ the scratch clone (scoped to the diff): unexecuted in-diff assertions flip
6748
+ the verdict to mismatch, since a green run that never executed an assertion
6749
+ must not count as green. No seal semantics, CLI name, or bin path changes.
6744
6750
 
6745
6751
  Verdicts / exit codes: verified 0, mismatch 1, unverifiable 1, misuse 2,
6746
6752
  no_claims 3 (no test claims found in the supplied text).
@@ -6754,6 +6760,7 @@ Options:
6754
6760
  --claims TEXT Claims text (default: PR body for --pr)
6755
6761
  --claims-file FILE Read claims text from a file
6756
6762
  --timeout SEC Per-command replay timeout, default: 300
6763
+ --no-i6d Skip the I6d assertion-execution gate
6757
6764
  --json Print the machine-readable report
6758
6765
  --out FILE Write the JSON report to a file
6759
6766
  --keep-workdir Keep the scratch clone for debugging
@@ -1,6 +1,6 @@
1
1
  # evalid Launch Dossier
2
2
 
3
- Generated: 2026-11-08T23:43:07.000Z
3
+ Generated: 2026-11-09T05:51:48.645Z
4
4
  Mode: post_publish
5
5
  Verdict: conditionally ready: engine and adoption evidence are strong, registry publication remains required for a splash launch
6
6
  Launch ready: not yet
@@ -117,7 +117,7 @@ Evidence:
117
117
  - Adoption records: 1; status: pass
118
118
  - Goal audit: pass; goal: Keep evalid credible as an open-source agent delivery runtime: repo-owned workflow contracts, executable verification gates, durable project context, and honest benchmark evidence.
119
119
  - Package evidence: missing; source: registry_json
120
- - Package name/version: @surlymochan/evalid@0.5.16
120
+ - Package name/version: @surlymochan/evalid@0.5.18
121
121
 
122
122
  ## Passing Evidence
123
123
 
@@ -49,7 +49,7 @@
49
49
  └────────────────────────────┬────────────────────────────────────┘
50
50
  │ atom dispatch
51
51
  ┌────────────────────────────▼────────────────────────────────────┐
52
- │ L0 原子组件(85 atoms,每个只有一处实现) │
52
+ │ L0 原子组件(86 atoms,每个只有一处实现) │
53
53
  │ │
54
54
  │ xflow/atoms/*.py A/B/C/E6/H1/I6/J/K (lite) │
55
55
  │ src/core/atoms/**/*.js D/E/F/G/E3/E4/E7 (heavy) │
@@ -219,7 +219,7 @@ CHANGE_ID=<id> xflow workflow run workflows/yolo.yaml
219
219
  │ ack
220
220
  ▼
221
221
  ┌──────────────────────┐
222
- │ 21b tdd-green │ atoms: I6a(green) + I6b(green) + I6c quality
222
+ │ 21b tdd-green │ atoms: I6a(green) + I6b(green) + I6d assertion + I6c quality
223
223
  │ gate: artifact-verify│◄─── green/refactor 必须通过
224
224
  └──────────┬───────────┘
225
225
  │ exist
@@ -251,6 +251,7 @@ specs/changes/<change_id>/
251
251
  ├── tdd/
252
252
  │ ├── red-0.json (I6a/I6b @ phase 21a, passed:false)
253
253
  │ ├── green-0.json (I6a/I6b @ phase 21b, passed:true)
254
+ │ ├── i6d-0.json (I6d @ phase 21b, assertion execution)
254
255
  │ └── quality-0.json (I6c @ phase 21b, test quality review)
255
256
  ├── merge-product.md (A5 @ archive)
256
257
  ├── merge-architecture.md
@@ -377,8 +378,8 @@ CHANGE_ID=<id> xflow workflow run workflows/corps.yaml
377
378
  ┌──────────────────┐
378
379
  │ 21a tdd-red │ I6a(red) + I6b(red)
379
380
  │ 22 execute │ implementation happens here
380
- │ 21b tdd-green │ I6a(green) + I6b(green) + I6c quality
381
- │ gate: artifact │◄── red-0.json + green-0.json + quality-0.json
381
+ │ 21b tdd-green │ I6a(green) + I6b(green) + I6d assertion + I6c quality
382
+ │ gate: artifact │◄── red-0.json + green-0.json + i6d-0.json + quality-0.json
382
383
  └────────┬─────────┘
383
384
  │
384
385
  ▼
@@ -627,6 +628,8 @@ J 管验证 K 管收尾归档
627
628
  │ │ I6b tdd.proof_validate 重算 proof 绑定,在快照上重跑命令(lite) │
628
629
  │ │ I6c tdd.quality_review 测试质量、红绿命令、skip/弱断言; │
629
630
  │ │ tdd_mode=refactor|docs 时替代 red 证明 │
631
+ │ │ I6d tdd.assertion_execution 新增/改动 pytest 断言真实执行证明; │
632
+ │ │ 无改动 pytest 测试时直接通过(lite) │
630
633
  ├────┼──────────────────────────────────────────────────────────────────────┤
631
634
  │ J │ Judge / 验证层:确定性测试 + 漂移扫描 + 证据收集 mixed 3 │
632
635
  │ │ Verify │
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@surlymochan/evalid",
3
- "version": "0.5.16",
3
+ "version": "0.5.18",
4
4
  "description": "Claimed != verified: a runtime verification layer for coding agents. It seals each test run and replays it from a clean snapshot, with a zero-judge gate pipeline (spec clarification, code review, TDD proof) and the xflow CLI.",
5
5
  "keywords": [
6
6
  "ai-agents",
@@ -0,0 +1,104 @@
1
+ #!/usr/bin/env bash
2
+ # M3 slice 4 — acceptance demo for the I6d assertion-execution gate.
3
+ #
4
+ # End-to-end proof, through the real `xflow audit` CLI (not the JS API):
5
+ # RED — a diff whose new test defines a callback that is never called,
6
+ # so its `assert value == 42` never executes, while the claimed
7
+ # pytest run still passes -> audit must say mismatch, exit 1.
8
+ # GREEN — a diff whose new assertions all execute, same green claim
9
+ # -> audit must say verified, exit 0.
10
+ #
11
+ # The gate's job (M3, parent #69): an assertion that never runs must not
12
+ # count as green. The RED case is exactly the README gap ("assertions in
13
+ # callbacks that never run").
14
+ #
15
+ # Usage: scripts/demo-i6d-gate.sh
16
+ # Everything happens in a throwaway repo under $TMPDIR; nothing is pushed.
17
+ # Needs a python with pytest (ambient, else a throwaway venv is built and
18
+ # pip installs pytest — same approach as test/xflow-audit.test.js).
19
+ set -euo pipefail
20
+
21
+ EVALID_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
22
+ WORK="$(mktemp -d "${TMPDIR:-/tmp}/evalid-i6d-demo-XXXXXX")"
23
+ trap 'rm -rf "$WORK"' EXIT
24
+
25
+ # --- python with pytest -------------------------------------------------
26
+ # EVALID_I6D_PYTEST_PYTHON lets a caller (e.g. the node test) hand over an
27
+ # already-provisioned interpreter instead of building a second venv.
28
+ PY="${EVALID_I6D_PYTEST_PYTHON:-python3}"
29
+ if ! "$PY" -c 'import pytest' 2>/dev/null; then
30
+ echo '[demo] no ambient pytest; building a throwaway venv…'
31
+ "$PY" -m venv "$WORK/venv"
32
+ "$WORK/venv/bin/python" -m pip install --quiet pytest
33
+ PY="$WORK/venv/bin/python"
34
+ fi
35
+ export PYTHON="$PY"
36
+ export PATH="$(dirname "$PY"):$PATH"
37
+ echo "[demo] pytest python: $PY"
38
+
39
+ # --- throwaway demo repo (carries the I6d plugin, like a real adopter) ---
40
+ DEMO="$WORK/repo"
41
+ mkdir -p "$DEMO/xflow/atoms"
42
+ cp "$EVALID_ROOT/xflow/atoms/i6d_assertion_execution.py" "$DEMO/xflow/atoms/"
43
+ cd "$DEMO"
44
+ git init -q -b main
45
+ git config user.email 'i6d-demo@example.test'
46
+ git config user.name 'i6d-demo'
47
+ printf 'def test_normal():\n assert 1 + 1 == 2\n' > test_dead.py
48
+ git add .
49
+ git commit -qm base
50
+ BASE="$(git rev-parse HEAD)"
51
+
52
+ run_case() { # $1 = RED|GREEN, $2 = file content, $3 = expected verdict, $4 = expected exit
53
+ local name="$1" content="$2" want_verdict="$3" want_exit="$4"
54
+ printf '%s' "$content" > test_dead.py
55
+ git diff > "$WORK/$name.diff"
56
+ printf 'Tests: `pytest test_dead.py` — 2 tests, all pass.\n' > "$WORK/$name.claims.txt"
57
+ git checkout -q -- test_dead.py
58
+ echo "=== $name case: xflow audit --diff $name.diff --base $BASE ==="
59
+ set +e
60
+ node "$EVALID_ROOT/bin/xflow.js" audit --repo "$DEMO" \
61
+ --diff "$WORK/$name.diff" --base "$BASE" \
62
+ --claims-file "$WORK/$name.claims.txt" --out "$WORK/$name.json"
63
+ local rc=$?
64
+ set -e
65
+ local verdict
66
+ verdict="$(node -p "require('$WORK/$name.json').verdict")"
67
+ local i6d_verdict unexecuted
68
+ i6d_verdict="$(node -p "require('$WORK/$name.json').i6d.verdict")"
69
+ unexecuted="$(node -p "require('$WORK/$name.json').i6d.unexecuted.map(u => u.file + ':' + u.line + ' ' + u.source).join('; ')")"
70
+ echo "[$name] audit exit=$rc verdict=$verdict i6d=$i6d_verdict unexecuted=[$unexecuted]"
71
+ if [ "$verdict" != "$want_verdict" ] || [ "$rc" != "$want_exit" ]; then
72
+ echo "[$name] FAIL: expected verdict=$want_verdict exit=$want_exit" >&2
73
+ return 1
74
+ fi
75
+ echo "[$name] PASS"
76
+ }
77
+
78
+ RED_CONTENT='def test_callback_never_runs():
79
+ def on_done(value):
80
+ assert value == 42
81
+ # on_done is defined but never called: its assert never executes.
82
+ assert True
83
+
84
+
85
+ def test_normal():
86
+ assert 1 + 1 == 2
87
+ '
88
+
89
+ GREEN_CONTENT='def test_normal():
90
+ assert 1 + 1 == 2
91
+
92
+
93
+ def test_new_ok():
94
+ assert 2 + 2 == 4
95
+ '
96
+
97
+ run_case RED "$RED_CONTENT" mismatch 1
98
+ run_case GREEN "$GREEN_CONTENT" verified 0
99
+
100
+ echo
101
+ echo '=== M3 acceptance demo: PASS ==='
102
+ echo 'RED diff (dead callback assert) -> mismatch, exit 1 (i6d: unexecuted_assertions)'
103
+ echo 'GREEN diff (all asserts execute) -> verified, exit 0 (i6d: pass)'
104
+ echo 'A never-executed assertion does not count as green.'
package/src/core/audit.js CHANGED
@@ -18,6 +18,17 @@
18
18
  * no_claims — no test claims found in the supplied text
19
19
  * unverifiable — a claimed command could not run (missing runner, timeout)
20
20
  *
21
+ * Slice 3 (M3, issue #86) adds the I6d assertion-execution gate to the
22
+ * replay: when the diff touches Python test files (or a pytest run is
23
+ * claimed) and the audited repo carries the I6d plugin
24
+ * (xflow/atoms/i6d_assertion_execution.py), the gate runs in the scratch
25
+ * clone, scoped to the diff. A pytest run can exit 0 while an assertion
26
+ * inside a never-called callback never executes -- that change must not
27
+ * count as green, so unexecuted in-diff assertions flip the verdict to
28
+ * mismatch. When the plugin is absent, the diff touches no Python tests,
29
+ * or pytest is unavailable, the gate reports not applicable and the
30
+ * verdict is unchanged (the audit stays usable for any repo).
31
+ *
21
32
  * Exit codes mirror the verdicts: 0 verified, 1 mismatch/unverifiable,
22
33
  * 2 misuse, 3 no_claims (inconclusive, not a failure of the change).
23
34
  */
@@ -54,6 +65,101 @@ export function isTestFile(path) {
54
65
  return /(^|\/)test(s)?\/|\.test\.[cm]?[jt]sx?$|_test\.go$|^test_[^/]*\.py$|_test\.py$|\.spec\.[cm]?[jt]sx?$/.test(path);
55
66
  }
56
67
 
68
+ /** Python files pytest collects as tests by default (test_*.py / *_test.py). */
69
+ export function isPythonTestFile(path) {
70
+ const name = String(path || '').split('/').pop();
71
+ return /\.py$/.test(name) && (name.startsWith('test_') || name.endsWith('_test.py'));
72
+ }
73
+
74
+ /**
75
+ * Run the I6d assertion-execution gate in an audit scratch clone.
76
+ * Scope comes from the audit's diff text (--diff-file in the I6d CLI),
77
+ * so uncommitted diffs work without committing anything.
78
+ *
79
+ * Returns `{ applicable: false, reason }` when the gate cannot apply
80
+ * (no plugin in the audited repo, no Python test changes and no pytest
81
+ * claim, pytest unavailable), and `{ applicable: true, verdict, ... }`
82
+ * otherwise. `pytest_failed` in the gate run does NOT flip the audit
83
+ * verdict by itself -- a failing pytest run is the claim replay's job
84
+ * to judge; only `unexecuted_assertions` (and gate errors) change it.
85
+ */
86
+ export function runI6dCheck(workdir, options = {}) {
87
+ const {
88
+ diffText = '',
89
+ claims = [],
90
+ python = process.env.PYTHON || 'python3',
91
+ timeoutMs = 300000,
92
+ } = options;
93
+ const plugin = join(workdir, 'xflow', 'atoms', 'i6d_assertion_execution.py');
94
+ if (!existsSync(plugin)) {
95
+ return { applicable: false, reason: 'plugin_not_present' };
96
+ }
97
+ const { files } = parseDiffFiles(diffText);
98
+ const changedPyTests = files.filter(isPythonTestFile);
99
+ const hasPytestClaim = claims.some((claim) => /\bpytest\b/.test(claim.command || ''));
100
+ if (changedPyTests.length === 0 && !hasPytestClaim) {
101
+ return { applicable: false, reason: 'no_python_test_changes' };
102
+ }
103
+ const targets = changedPyTests.filter((file) => existsSync(join(workdir, file)));
104
+ if (targets.length === 0 && !hasPytestClaim) {
105
+ return { applicable: false, reason: 'no_python_test_changes' };
106
+ }
107
+ const pytestCheck = spawnSync(python, ['-c', 'import pytest'], { cwd: workdir, encoding: 'utf8', timeout: 60000 });
108
+ if (pytestCheck.status !== 0) {
109
+ return { applicable: false, reason: 'pytest_unavailable' };
110
+ }
111
+ const scratch = mkdtempSync(join(tmpdir(), 'evalid-audit-i6d-'));
112
+ try {
113
+ const diffPath = join(scratch, 'audit.diff');
114
+ const evidencePath = join(scratch, 'i6d-evidence.json');
115
+ writeFileSync(diffPath, diffText);
116
+ const result = spawnSync(python, [
117
+ plugin,
118
+ '--root', workdir,
119
+ '--output', evidencePath,
120
+ '--pytest-python', python,
121
+ '--only-changed',
122
+ '--diff-file', diffPath,
123
+ '--',
124
+ ...targets,
125
+ ], { cwd: workdir, encoding: 'utf8', timeout: timeoutMs });
126
+ if (!existsSync(evidencePath)) {
127
+ return {
128
+ applicable: true,
129
+ verdict: 'i6d_error',
130
+ error: `i6d gate wrote no evidence (exit ${result.status}): ${(result.stderr || result.stdout || '').trim().slice(-300)}`,
131
+ };
132
+ }
133
+ let evidence;
134
+ try {
135
+ evidence = JSON.parse(readFileSync(evidencePath, 'utf8'));
136
+ } catch {
137
+ return { applicable: true, verdict: 'i6d_error', error: 'i6d evidence JSON unparseable' };
138
+ }
139
+ const unexecuted = (evidence.unexecuted || []).map((entry) => ({
140
+ file: String(entry.file || '').startsWith(workdir)
141
+ ? String(entry.file).slice(workdir.length + 1)
142
+ : entry.file,
143
+ line: entry.line,
144
+ test: entry.test,
145
+ source: entry.source,
146
+ }));
147
+ return {
148
+ applicable: true,
149
+ verdict: evidence.verdict,
150
+ exit: result.status,
151
+ executed: evidence.executed ?? 0,
152
+ total: evidence.total ?? 0,
153
+ unexecuted,
154
+ skipped: (evidence.skipped || []).length,
155
+ outOfScope: (evidence.outOfScope || []).length,
156
+ pytestExit: evidence.pytestExit ?? null,
157
+ };
158
+ } finally {
159
+ rmSync(scratch, { recursive: true, force: true });
160
+ }
161
+ }
162
+
57
163
  function normalizeCommand(raw) {
58
164
  return String(raw || '').replace(/\s+/g, ' ').trim();
59
165
  }
@@ -322,6 +428,10 @@ function replaySeal(workdir, repoRoot, seal, python, timeoutMs) {
322
428
  * timeoutMs — per-command timeout (default 300_000)
323
429
  * python — python binary for the i6b validator (default 'python3')
324
430
  * keepWorkdir — keep the scratch clone for debugging (default false)
431
+ * python — python binary for the i6b validator and the i6d gate
432
+ * (default 'python3' / $PYTHON)
433
+ * i6d — run the I6d assertion-execution gate when it applies
434
+ * (default true; --no-i6d on the CLI disables it)
325
435
  */
326
436
  export function runAudit(options = {}) {
327
437
  const {
@@ -333,6 +443,7 @@ export function runAudit(options = {}) {
333
443
  timeoutMs = 300000,
334
444
  python = process.env.PYTHON || 'python3',
335
445
  keepWorkdir = false,
446
+ i6d = true,
336
447
  } = options;
337
448
  if (!repoDir || !existsSync(repoDir)) {
338
449
  return { ok: false, verdict: 'misuse', error: `repo not found: ${repoDir}`, exitCode: 2 };
@@ -432,14 +543,28 @@ export function runAudit(options = {}) {
432
543
  const sealResults = diff.seals.map((seal) => replaySeal(workdir, resolve(repoDir), seal, python, timeoutMs));
433
544
  const badSeal = sealResults.find((seal) => !seal.ok);
434
545
 
546
+ // Slice 3 (M3): the I6d gate judges whether the diff's new/changed
547
+ // assertions really executed during the green run. A replay can exit 0
548
+ // while an assertion in a never-called callback never ran -- that
549
+ // change must not count as green.
550
+ const i6dResult = i6d === false
551
+ ? { applicable: false, reason: 'disabled' }
552
+ : runI6dCheck(workdir, { diffText, claims, python, timeoutMs });
553
+
435
554
  let verdict;
436
555
  let exitCode;
437
- if (claims.length === 0 && sealResults.length === 0) {
556
+ if (i6dResult.applicable && i6dResult.verdict === 'unexecuted_assertions') {
557
+ verdict = 'mismatch';
558
+ exitCode = 1;
559
+ } else if (claims.length === 0 && sealResults.length === 0) {
438
560
  verdict = 'no_claims';
439
561
  exitCode = 3;
440
562
  } else if (mismatch || badSeal) {
441
563
  verdict = 'mismatch';
442
564
  exitCode = 1;
565
+ } else if (i6dResult.applicable && (i6dResult.verdict === 'diff_error' || i6dResult.verdict === 'i6d_error')) {
566
+ verdict = 'unverifiable';
567
+ exitCode = 1;
443
568
  } else if (claimResults.some((entry) => entry.replay && (entry.replay.timedOut || entry.replay.spawnError))) {
444
569
  verdict = 'unverifiable';
445
570
  exitCode = 1;
@@ -454,6 +579,7 @@ export function runAudit(options = {}) {
454
579
  exitCode,
455
580
  claims: claimResults,
456
581
  seals: sealResults,
582
+ i6d: i6dResult,
457
583
  files: diff.files,
458
584
  testFiles: diff.testFiles,
459
585
  setup,
@@ -492,6 +618,21 @@ export function renderAuditReport(report) {
492
618
  for (const seal of report.seals || []) {
493
619
  lines.push(`- ${seal.ok ? '✓' : '✗'} seal ${seal.changeId}/${seal.phase}/${seal.attempt}: ${seal.verdict}`);
494
620
  }
621
+ const gate = report.i6d;
622
+ if (gate) {
623
+ if (!gate.applicable) {
624
+ lines.push(`i6d: not applicable (${gate.reason})`);
625
+ } else if (gate.verdict === 'pass') {
626
+ lines.push(`i6d: pass (${gate.executed}/${gate.total} assertions executed, ${gate.skipped} skipped, ${gate.outOfScope} out of scope)`);
627
+ } else if (gate.verdict === 'unexecuted_assertions') {
628
+ lines.push(`i6d: ✗ unexecuted_assertions (${gate.executed}/${gate.total} executed, ${gate.unexecuted.length} unexecuted)`);
629
+ for (const entry of gate.unexecuted) {
630
+ lines.push(` unexecuted: ${entry.file}:${entry.line}: ${entry.source}`);
631
+ }
632
+ } else {
633
+ lines.push(`i6d: ${gate.verdict}${gate.error ? `: ${gate.error}` : ''}`);
634
+ }
635
+ }
495
636
  if (report.durationMs !== undefined) lines.push(`duration: ${(report.durationMs / 1000).toFixed(1)}s`);
496
637
  return lines.join('\n');
497
638
  }
@@ -398,10 +398,13 @@ phases:
398
398
  with: { phase: "green", test_command: "${input.tdd_green_command}", expect: "pass" }
399
399
  - id: I6b.tdd.proof_validate
400
400
  with: { phase: "green", tdd_mode: "${input.tdd_mode}" }
401
+ - id: I6d.tdd.assertion_execution
402
+ with: { only_changed: true, include_uncommitted: true }
401
403
  - id: I6c.tdd.quality_review
402
404
  with: { tdd_mode: "${input.tdd_mode}" }
403
405
  artifacts:
404
406
  - { path: "specs/changes/${change_id}/tdd/green-0.json", optional: false }
407
+ - { path: "specs/changes/${change_id}/tdd/i6d-0.json", optional: false }
405
408
  - { path: "specs/changes/${change_id}/tdd/quality-0.json", optional: false }
406
409
  gate:
407
410
  type: artifact-verify
@@ -187,10 +187,13 @@ phases:
187
187
  with: { phase: "green", test_command: "${input.tdd_green_command}", expect: "pass" }
188
188
  - id: I6b.tdd.proof_validate
189
189
  with: { phase: "green", tdd_mode: "${input.tdd_mode}" }
190
+ - id: I6d.tdd.assertion_execution
191
+ with: { only_changed: true, include_uncommitted: true }
190
192
  - id: I6c.tdd.quality_review
191
193
  with: { tdd_mode: "${input.tdd_mode}" }
192
194
  artifacts:
193
195
  - { path: "specs/changes/${change_id}/tdd/green-0.json", optional: false }
196
+ - { path: "specs/changes/${change_id}/tdd/i6d-0.json", optional: false }
194
197
  - { path: "specs/changes/${change_id}/tdd/quality-0.json", optional: false }
195
198
  gate:
196
199
  type: artifact-verify
@@ -21,6 +21,14 @@ shapes of it: expected values taken from the implementation's own constants
21
21
  named callbacks that are never referenced again (they execute zero times),
22
22
  and self-comparison assertions (x == x). Type-only or existence-only
23
23
  assertions are flagged only when they are the sole assertions.
24
+
25
+ Slice 3 (M3, issue #86) folds the I6d assertion-execution evidence into
26
+ this review: when the I6d atom ran first (tdd-green phase order is I6a,
27
+ I6b, I6d, I6c) and wrote ``specs/changes/<id>/tdd/i6d-0.json``, an
28
+ ``unexecuted_assertions`` verdict becomes one fail finding per unexecuted
29
+ assert, and a ``pytest_unavailable`` verdict becomes a fail finding of its
30
+ own. When no i6d-0.json exists (I6d not run, e.g. a change with no Python
31
+ tests) this review behaves exactly as before.
24
32
  """
25
33
  import argparse
26
34
  import json
@@ -343,6 +351,43 @@ def main() -> None:
343
351
  for test_file in sorted(reviewed):
344
352
  findings.extend(review_test_file(project_root, test_file, implementation_sources))
345
353
 
354
+ # Slice 3 (M3): fold the I6d assertion-execution evidence in, when the
355
+ # I6d atom ran first and wrote i6d-0.json. An assertion that never
356
+ # executed during the green run must fail the quality gate, even when
357
+ # the run itself was green. No i6d-0.json means I6d did not run (no
358
+ # Python test changes): behaviour is unchanged, no finding added.
359
+ i6d_evidence = read_json(proof_dir / "i6d-0.json")
360
+ i6d_summary = None
361
+ if i6d_evidence:
362
+ i6d_summary = {
363
+ "verdict": i6d_evidence.get("verdict"),
364
+ "executed": i6d_evidence.get("executed"),
365
+ "total": i6d_evidence.get("total"),
366
+ "unexecuted": len(i6d_evidence.get("unexecuted") or []),
367
+ "skipped": len(i6d_evidence.get("skipped") or []),
368
+ }
369
+ if i6d_evidence.get("verdict") == "unexecuted_assertions":
370
+ for entry in i6d_evidence.get("unexecuted") or []:
371
+ findings.append({
372
+ "severity": "fail",
373
+ "code": "unexecuted_assertions",
374
+ "path": entry.get("file"),
375
+ "line": entry.get("line"),
376
+ "test": entry.get("test"),
377
+ "source": entry.get("source"),
378
+ "message": "An assertion added/changed by this diff never "
379
+ "executed during the green run; it must not count as green.",
380
+ })
381
+ elif i6d_evidence.get("verdict") == "pytest_unavailable":
382
+ findings.append({
383
+ "severity": "fail",
384
+ "code": "i6d_pytest_unavailable",
385
+ "message": "The I6d assertion-execution gate could not run: "
386
+ "pytest is unavailable, so added/changed assertions could "
387
+ "not be proven executed.",
388
+ "assertions_at_stake": i6d_evidence.get("total"),
389
+ })
390
+
346
391
  failed = any(finding.get("severity") == "fail" for finding in findings)
347
392
  verdict = "failed" if failed else "passed"
348
393
  proof = {
@@ -354,6 +399,7 @@ def main() -> None:
354
399
  "code_files": code_files,
355
400
  "test_files": test_files,
356
401
  "read_cache": _read_cache.stats(),
402
+ "i6d": i6d_summary,
357
403
  "findings": findings,
358
404
  "recorded_at": datetime.now(timezone.utc).isoformat(),
359
405
  }
@@ -368,6 +414,7 @@ def main() -> None:
368
414
  "tdd_mode": mode,
369
415
  "proof_file": str(proof_file),
370
416
  "read_cache": _read_cache.stats(),
417
+ "i6d": i6d_summary,
371
418
  "failures": sum(1 for finding in findings if finding.get("severity") == "fail"),
372
419
  "warnings": sum(1 for finding in findings if finding.get("severity") == "warn"),
373
420
  "failed_codes": [
@@ -39,6 +39,22 @@ and the CLI exposes the same as ``--only-changed`` / ``--diff-base``.
39
39
  With scoping on, asserts outside the diff are reported as ``outOfScope``
40
40
  (never red), an empty diff passes (exit 0), and only in-scope unexecuted
41
41
  asserts fail the gate. Without the flag, slice 1 behaviour is unchanged.
42
+
43
+ Slice 3 wires the gate into ``xflow audit`` replay and the I6c gate:
44
+
45
+ * ``changed_lines_from_diff(diff_text, repo)`` -- the same changed-lines
46
+ shape as ``changed_lines`` but parsed from a unified diff *text* (the
47
+ audit's --diff / PR diff, which may be uncommitted or carry context
48
+ lines), so the audit can scope the gate without committing anything;
49
+ * ``changed_lines(..., uncommitted=True)`` -- additionally unions
50
+ ``git diff HEAD`` (working tree + staged), for the workflow atom, where
51
+ the change under review is still uncommitted;
52
+ * atom mode (``--change-id`` / ``--project-root``, as invoked by the
53
+ workflow executor): evidence lands in
54
+ ``specs/changes/<id>/tdd/i6d-0.json`` and stdout is a single JSON line
55
+ (``{ok, verdict, proof_file, ...}``), so the executor can parse it and
56
+ the I6c review can fold the verdict into quality-0.json. With no
57
+ changed Python test files the atom passes without running pytest.
42
58
  """
43
59
 
44
60
  import argparse
@@ -53,7 +69,22 @@ from pathlib import Path
53
69
  EVIDENCE_ENV = "EVALID_I6D_EVIDENCE"
54
70
  DIFF_ONLY_ENV = "EVALID_I6D_ONLY_CHANGED"
55
71
  DIFF_BASE_ENV = "EVALID_I6D_DIFF_BASE"
56
- VERSION = 2
72
+ DIFF_FILE_ENV = "EVALID_I6D_DIFF_FILE"
73
+ INCLUDE_UNCOMMITTED_ENV = "EVALID_I6D_INCLUDE_UNCOMMITTED"
74
+ VERSION = 3
75
+
76
+
77
+ def is_python_test_file(path):
78
+ """True for files pytest collects as tests by default.
79
+
80
+ Mirrors pytest's default ``python_files`` (``test_*.py`` /
81
+ ``*_test.py``); kept in sync with ``isPythonTestFile`` in
82
+ ``src/core/audit.js``.
83
+ """
84
+ name = os.path.basename(str(path))
85
+ return name.endswith(".py") and (
86
+ name.startswith("test_") or name.endswith("_test.py")
87
+ )
57
88
 
58
89
 
59
90
  # ---------------------------------------------------------------------------
@@ -127,23 +158,25 @@ def resolve_diff_base(repo, base="main"):
127
158
  _HUNK_RE = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")
128
159
 
129
160
 
130
- def changed_lines(repo, base="main"):
131
- """Return ``(merge_base_sha, {abspath: {new-side line numbers}})``.
161
+ def _parse_unified_diff(diff_text, repo):
162
+ """``{abspath: {new-side line numbers}}`` from a unified diff text.
132
163
 
133
- The diff is ``git diff <merge-base> HEAD -U0``: committed changes on
134
- this branch only. A file's changed lines are the new-side lines of
135
- every hunk (for a brand-new file, every line). Deleted files contribute
136
- nothing; uncommitted working-tree edits are not included.
164
+ Only lines actually added (``+`` lines inside a hunk) count -- context
165
+ lines and hunk-header ranges do not, so a -U3 diff (``gh pr diff``,
166
+ the audit's --diff input) scopes exactly like a -U0 one. Deleted
167
+ lines do not advance the new-side number; ``\\ No newline`` lines
168
+ neither. Deleted files and files with no added lines contribute
169
+ nothing.
137
170
  """
138
- merge_base = resolve_diff_base(repo, base)
139
- raw = _git(repo, "diff", merge_base, "HEAD", "-U0", "--")
140
171
  changed = {}
141
172
  current = None
142
- for line in raw.splitlines():
173
+ new_lineno = None
174
+ for line in diff_text.splitlines():
143
175
  if line.startswith("+++ "):
144
176
  target = line[4:].strip()
145
177
  if target in ("/dev/null", ""):
146
178
  current = None
179
+ new_lineno = None
147
180
  continue
148
181
  if target.startswith("b/"):
149
182
  target = target[2:]
@@ -151,14 +184,54 @@ def changed_lines(repo, base="main"):
151
184
  target = json.loads(target)
152
185
  current = os.path.realpath(os.path.join(repo, target))
153
186
  changed.setdefault(current, set())
187
+ new_lineno = None
154
188
  elif line.startswith("@@ ") and current is not None:
155
189
  match = _HUNK_RE.match(line)
156
- if match is None:
157
- continue
158
- start = int(match.group(1))
159
- count = int(match.group(2)) if match.group(2) is not None else 1
160
- changed[current].update(range(start, start + count))
161
- return merge_base, {path: lines for path, lines in changed.items() if lines}
190
+ new_lineno = int(match.group(1)) if match else None
191
+ elif current is None or new_lineno is None:
192
+ continue
193
+ elif line.startswith("+") and not line.startswith("+++"):
194
+ changed[current].add(new_lineno)
195
+ new_lineno += 1
196
+ elif line.startswith("-") and not line.startswith("---"):
197
+ pass # deleted line: the new-side number does not advance
198
+ elif line.startswith("\\"):
199
+ pass # "": no advance
200
+ else:
201
+ new_lineno += 1 # context line
202
+ return {path: lines for path, lines in changed.items() if lines}
203
+
204
+
205
+ def changed_lines_from_diff(diff_text, repo):
206
+ """Changed lines from a unified diff *text* (slice 3, for the audit).
207
+
208
+ Unlike :func:`changed_lines` this needs no git state: the audit
209
+ replays an uncommitted diff (or a PR diff) in a scratch clone, where
210
+ the change is not a commit yet. Same return shape as
211
+ :func:`changed_lines` minus the merge-base.
212
+ """
213
+ return _parse_unified_diff(diff_text, repo)
214
+
215
+
216
+ def changed_lines(repo, base="main", uncommitted=False):
217
+ """Return ``(merge_base_sha, {abspath: {new-side line numbers}})``.
218
+
219
+ The diff is ``git diff <merge-base> HEAD -U0``: committed changes on
220
+ this branch only. With ``uncommitted=True`` the working tree + staged
221
+ changes vs HEAD (``git diff HEAD -U0``) are unioned in, for the
222
+ workflow atom, where the change under review is still uncommitted.
223
+ A file's changed lines are its added lines. Deleted files contribute
224
+ nothing; without ``uncommitted``, uncommitted working-tree edits are
225
+ not included.
226
+ """
227
+ merge_base = resolve_diff_base(repo, base)
228
+ raw = _git(repo, "diff", merge_base, "HEAD", "-U0", "--")
229
+ changed = _parse_unified_diff(raw, repo)
230
+ if uncommitted:
231
+ raw_wt = _git(repo, "diff", "HEAD", "-U0", "--")
232
+ for path, lines in _parse_unified_diff(raw_wt, repo).items():
233
+ changed.setdefault(path, set()).update(lines)
234
+ return merge_base, changed
162
235
 
163
236
 
164
237
  def assert_in_scope(entry, changed):
@@ -172,9 +245,9 @@ def assert_in_scope(entry, changed):
172
245
  )
173
246
 
174
247
 
175
- def load_diff_scope(repo, base="main"):
248
+ def load_diff_scope(repo, base="main", uncommitted=False):
176
249
  """``{'base': <merge-base sha>, 'changed': {path: [lines]}}`` (JSON-safe)."""
177
- merge_base, changed = changed_lines(repo, base)
250
+ merge_base, changed = changed_lines(repo, base, uncommitted=uncommitted)
178
251
  return {
179
252
  "base": merge_base,
180
253
  "changed": {path: sorted(lines) for path, lines in changed.items()},
@@ -218,11 +291,21 @@ class _AssertionExecutionPlugin:
218
291
  def pytest_sessionstart(self, session):
219
292
  self._old_trace = sys.gettrace()
220
293
  sys.settrace(self._global_trace)
221
- if os.environ.get(DIFF_ONLY_ENV):
294
+ if os.environ.get(DIFF_FILE_ENV):
295
+ # Slice 3 (audit): scope from a diff text, not from git state --
296
+ # the audited change may be uncommitted in a scratch clone.
297
+ try:
298
+ text = Path(os.environ[DIFF_FILE_ENV]).read_text(encoding="utf-8")
299
+ self.diff_changed = changed_lines_from_diff(text, self.rootdir)
300
+ self.diff_base = None
301
+ except Exception as exc: # noqa: BLE001 - surface as evidence, not a crash
302
+ self.diff_error = str(exc)
303
+ elif os.environ.get(DIFF_ONLY_ENV):
222
304
  try:
223
305
  base = os.environ.get(DIFF_BASE_ENV) or "main"
224
306
  self.diff_base, self.diff_changed = changed_lines(
225
- self.rootdir, base
307
+ self.rootdir, base,
308
+ uncommitted=bool(os.environ.get(INCLUDE_UNCOMMITTED_ENV)),
226
309
  )
227
310
  except Exception as exc: # noqa: BLE001 - surface as evidence, not a crash
228
311
  self.diff_error = str(exc)
@@ -373,34 +456,234 @@ def write_evidence(evidence):
373
456
  # CLI: run pytest under the plugin, then judge the evidence
374
457
  # ---------------------------------------------------------------------------
375
458
 
459
+ def _pytest_available(pytest_python):
460
+ """True when ``pytest_python`` can import pytest."""
461
+ try:
462
+ proc = subprocess.run(
463
+ [pytest_python, "-c", "import pytest"],
464
+ capture_output=True, text=True, timeout=60,
465
+ )
466
+ except OSError:
467
+ return False
468
+ return proc.returncode == 0
469
+
470
+
471
+ def _atom_summary(*, ok, verdict, proof_file, evidence, pytest_exit):
472
+ return {
473
+ "ok": ok,
474
+ "verdict": verdict,
475
+ "proof_file": str(proof_file),
476
+ "executed": evidence.get("executed", 0),
477
+ "total": evidence.get("total", 0),
478
+ "unexecuted": len(evidence.get("unexecuted", [])),
479
+ "skipped": len(evidence.get("skipped", [])),
480
+ "pytest_exit": pytest_exit,
481
+ }
482
+
483
+
376
484
  def run_gate(*, root, output, pytest_python, pytest_args,
377
- only_changed=False, diff_base="main"):
485
+ only_changed=False, diff_base="main", diff_file=None,
486
+ include_uncommitted=False, change_id=None, project_root=None):
487
+ """Run pytest under the plugin, then judge the evidence.
488
+
489
+ Slice 3 additions: ``diff_file`` scopes from a diff text instead of a
490
+ git range (the audit's case); ``include_uncommitted`` unions the
491
+ working tree into the git-range scope; ``change_id`` switches on atom
492
+ mode (evidence to ``specs/changes/<id>/tdd/i6d-0.json``, stdout a
493
+ single JSON line for the workflow executor). Atom mode always counts
494
+ uncommitted changes -- the change under review is still uncommitted
495
+ when the workflow runs -- and passes without running pytest when the
496
+ diff touches no Python test files.
497
+ """
498
+ atom_mode = change_id is not None
499
+ if atom_mode:
500
+ root = project_root or root
501
+ include_uncommitted = True
502
+ root = os.path.realpath(root)
503
+ output = os.path.abspath(output)
504
+ parent = os.path.dirname(output)
505
+ if parent:
506
+ os.makedirs(parent, exist_ok=True)
507
+
508
+ # Resolve the diff scope before running pytest: the audit and the
509
+ # atom both need the changed Python test files as pytest targets, and
510
+ # the atom short-circuits when there are none.
511
+ changed = None
512
+ diff_base_out = None
513
+ diff_source = None
514
+ diff_error = None
515
+ if only_changed:
516
+ try:
517
+ if diff_file is not None:
518
+ with open(diff_file, encoding="utf-8") as handle:
519
+ changed = changed_lines_from_diff(handle.read(), root)
520
+ diff_source = "diff-file"
521
+ else:
522
+ diff_base_out, changed = changed_lines(
523
+ root, diff_base, uncommitted=include_uncommitted,
524
+ )
525
+ diff_source = (
526
+ "git-range+uncommitted" if include_uncommitted
527
+ else "git-range"
528
+ )
529
+ except Exception as exc: # noqa: BLE001 - surface as evidence, not a crash
530
+ diff_error = str(exc)
531
+
532
+ targets = []
533
+ if only_changed and changed is not None and not pytest_args:
534
+ targets = sorted(
535
+ os.path.relpath(path, root)
536
+ for path, lines in changed.items()
537
+ if is_python_test_file(path) and lines
538
+ and os.path.exists(path)
539
+ )
540
+ pytest_args = targets
541
+
542
+ if atom_mode and only_changed and not pytest_args and diff_error is None:
543
+ # No changed Python test files: nothing for the gate to judge.
544
+ # Slice 2's empty-diff contract is "pass", so the atom passes
545
+ # without running pytest at all.
546
+ evidence = {
547
+ "tool": "i6d_assertion_execution",
548
+ "version": VERSION,
549
+ "root": root,
550
+ "pytestExit": None,
551
+ "verdict": "pass",
552
+ "total": 0,
553
+ "executed": 0,
554
+ "unexecuted": [],
555
+ "skipped": [],
556
+ "diffBase": diff_base_out,
557
+ "diffSource": diff_source,
558
+ "inScope": 0,
559
+ "outOfScope": [],
560
+ "notApplicable": "no_python_test_changes",
561
+ }
562
+ with open(output, "w", encoding="utf-8") as handle:
563
+ json.dump(evidence, handle, indent=2)
564
+ print(json.dumps(_atom_summary(
565
+ ok=True, verdict="pass", proof_file=output,
566
+ evidence=evidence, pytest_exit=None,
567
+ )))
568
+ return 0
569
+
570
+ if diff_error is not None:
571
+ evidence = {
572
+ "tool": "i6d_assertion_execution",
573
+ "version": VERSION,
574
+ "root": root,
575
+ "pytestExit": None,
576
+ "verdict": "diff_error",
577
+ "total": 0,
578
+ "executed": 0,
579
+ "unexecuted": [],
580
+ "skipped": [],
581
+ "diffBase": diff_base_out,
582
+ "diffSource": diff_source,
583
+ "diffError": diff_error,
584
+ }
585
+ with open(output, "w", encoding="utf-8") as handle:
586
+ json.dump(evidence, handle, indent=2)
587
+ if atom_mode:
588
+ print(json.dumps(_atom_summary(
589
+ ok=False, verdict="diff_error", proof_file=output,
590
+ evidence=evidence, pytest_exit=None,
591
+ )))
592
+ else:
593
+ print(f"i6d: diff_error: {diff_error}", file=sys.stderr)
594
+ return 2
595
+
378
596
  atoms_dir = str(Path(__file__).resolve().parent)
379
597
  env = dict(os.environ)
380
- env[EVIDENCE_ENV] = os.path.abspath(output)
598
+ env[EVIDENCE_ENV] = output
381
599
  if only_changed:
382
600
  env[DIFF_ONLY_ENV] = "1"
383
601
  env[DIFF_BASE_ENV] = diff_base
602
+ if diff_file is not None:
603
+ env[DIFF_FILE_ENV] = os.path.abspath(diff_file)
604
+ if include_uncommitted:
605
+ env[INCLUDE_UNCOMMITTED_ENV] = "1"
384
606
  env["PYTHONPATH"] = atoms_dir + os.pathsep + env.get("PYTHONPATH", "")
607
+
608
+ if not _pytest_available(pytest_python):
609
+ # The gate cannot judge anything without pytest. When the diff
610
+ # touches Python test files this is a real gap (exit 2), not a
611
+ # silent pass; statically count the asserts at stake for the
612
+ # evidence so the failure names what could not be judged.
613
+ total = 0
614
+ for target in (pytest_args or []):
615
+ candidate = target if os.path.isabs(target) else os.path.join(root, target)
616
+ if is_python_test_file(candidate) and os.path.exists(candidate):
617
+ try:
618
+ total += len(collect_assertions(candidate))
619
+ except Exception: # noqa: BLE001 - best effort only
620
+ pass
621
+ evidence = {
622
+ "tool": "i6d_assertion_execution",
623
+ "version": VERSION,
624
+ "root": root,
625
+ "pytestExit": None,
626
+ "verdict": "pytest_unavailable",
627
+ "total": total,
628
+ "executed": 0,
629
+ "unexecuted": [],
630
+ "skipped": [],
631
+ "diffBase": diff_base_out,
632
+ "diffSource": diff_source,
633
+ "inScope": total if only_changed else None,
634
+ "outOfScope": [],
635
+ }
636
+ with open(output, "w", encoding="utf-8") as handle:
637
+ json.dump(evidence, handle, indent=2)
638
+ if atom_mode:
639
+ print(json.dumps(_atom_summary(
640
+ ok=False, verdict="pytest_unavailable", proof_file=output,
641
+ evidence=evidence, pytest_exit=None,
642
+ )))
643
+ else:
644
+ print("i6d: pytest_unavailable: "
645
+ f"{pytest_python} cannot import pytest", file=sys.stderr)
646
+ return 2
647
+
385
648
  command = [pytest_python, "-m", "pytest", "-p", "i6d_assertion_execution", *pytest_args]
386
649
  try:
387
- proc = subprocess.run(command, cwd=root, env=env)
650
+ # Atom mode: the workflow executor parses stdout as one JSON line,
651
+ # so pytest's own session output must not leak onto stdout. Capture
652
+ # it; the evidence JSON carries the verdict, and the tail is only
653
+ # surfaced when no evidence was written.
654
+ if atom_mode:
655
+ proc = subprocess.run(command, cwd=root, env=env,
656
+ capture_output=True, text=True)
657
+ else:
658
+ proc = subprocess.run(command, cwd=root, env=env)
388
659
  except OSError as exc:
389
660
  print(f"i6d: could not run pytest: {exc}", file=sys.stderr)
390
661
  return 2
391
662
  try:
392
- with open(os.path.abspath(output), encoding="utf-8") as handle:
663
+ with open(output, encoding="utf-8") as handle:
393
664
  evidence = json.load(handle)
394
665
  except (OSError, json.JSONDecodeError) as exc:
395
- print(f"i6d: no evidence written ({exc}); pytest exit {proc.returncode}", file=sys.stderr)
666
+ detail = f"pytest exit {proc.returncode}"
667
+ if atom_mode and getattr(proc, "stdout", None):
668
+ detail += f"; pytest output tail: {(proc.stdout or '')[-500:]}"
669
+ print(f"i6d: no evidence written ({exc}); {detail}", file=sys.stderr)
396
670
  return 2
397
671
  evidence["pytestExit"] = proc.returncode
672
+ if diff_source is not None:
673
+ evidence["diffSource"] = diff_source
398
674
  if evidence["verdict"] == "unexecuted_assertions" and proc.returncode == 0:
399
675
  pass # verdict already reflects the gate; pytest itself passed.
400
676
  elif proc.returncode != 0 and not evidence["unexecuted"]:
401
677
  evidence["verdict"] = "pytest_failed"
402
- with open(os.path.abspath(output), "w", encoding="utf-8") as handle:
678
+ with open(output, "w", encoding="utf-8") as handle:
403
679
  json.dump(evidence, handle, indent=2)
680
+ if atom_mode:
681
+ ok = evidence["verdict"] == "pass"
682
+ print(json.dumps(_atom_summary(
683
+ ok=ok, verdict=evidence["verdict"], proof_file=output,
684
+ evidence=evidence, pytest_exit=proc.returncode,
685
+ )))
686
+ return 0 if ok else 1
404
687
  summary = (
405
688
  f"i6d: {evidence['verdict']} "
406
689
  f"({evidence['executed']}/{evidence['total']} assertions executed, "
@@ -422,27 +705,52 @@ def main(argv=None):
422
705
  parser = argparse.ArgumentParser(
423
706
  description="I6d assertion-execution gate: prove every assert in "
424
707
  "the pytest run really executed (slice 2 adds --only-changed to "
425
- "judge only asserts added/changed vs a diff base, default main)."
708
+ "judge only asserts added/changed vs a diff base, default main; "
709
+ "slice 3 adds --diff-file to scope from a diff text, and "
710
+ "--change-id atom mode for the workflow executor)."
426
711
  )
427
712
  parser.add_argument("--root", default=os.getcwd(), help="project root to run pytest in")
428
- parser.add_argument("--output", default=".evalid-i6d-evidence.json",
429
- help="where to write the evidence JSON")
713
+ parser.add_argument("--project-root", default=None,
714
+ help="alias for --root (workflow executor passes this)")
715
+ parser.add_argument("--output", default=None,
716
+ help="where to write the evidence JSON "
717
+ "(default: specs/changes/<id>/tdd/i6d-0.json in atom mode, "
718
+ ".evalid-i6d-evidence.json otherwise)")
430
719
  parser.add_argument("--pytest-python", default=sys.executable,
431
720
  help="python binary to run pytest with")
432
721
  parser.add_argument("--only-changed", action="store_true",
433
722
  help="only judge asserts in the diff vs --diff-base")
434
723
  parser.add_argument("--diff-base", default="main",
435
724
  help="diff base ref (merge-base of HEAD and this ref; default main)")
725
+ parser.add_argument("--diff-file", default=None,
726
+ help="scope from a unified diff file instead of a git range")
727
+ parser.add_argument("--include-uncommitted", action="store_true",
728
+ help="union working-tree + staged changes vs HEAD into the scope")
729
+ parser.add_argument("--change-id", default=None,
730
+ help="atom mode: write evidence to specs/changes/<id>/tdd/i6d-0.json "
731
+ "and print a single JSON summary line")
436
732
  parser.add_argument("pytest_args", nargs="*", help="arguments passed to pytest (after --)")
437
733
  args = parser.parse_args(argv)
734
+ root = args.project_root or args.root
735
+ if args.output:
736
+ output = args.output
737
+ elif args.change_id:
738
+ output = os.path.join(
739
+ root, "specs", "changes", args.change_id, "tdd", "i6d-0.json")
740
+ else:
741
+ output = ".evalid-i6d-evidence.json"
438
742
  # argparse keeps a literal "--" out of pytest_args already.
439
743
  return run_gate(
440
- root=args.root,
441
- output=args.output,
744
+ root=root,
745
+ output=output,
442
746
  pytest_python=args.pytest_python,
443
747
  pytest_args=args.pytest_args,
444
748
  only_changed=args.only_changed,
445
749
  diff_base=args.diff_base,
750
+ diff_file=args.diff_file,
751
+ include_uncommitted=args.include_uncommitted,
752
+ change_id=args.change_id,
753
+ project_root=args.project_root,
446
754
  )
447
755
 
448
756