@surlymochan/evalid 0.5.16 → 0.5.18
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +20 -0
- package/README.md +7 -0
- package/RELEASE_NOTES.md +24 -0
- package/atoms/registry.json +13 -0
- package/bin/xflow.js +8 -1
- package/docs/launch-dossier.md +2 -2
- package/docs/workflow-manual.md +7 -4
- package/package.json +1 -1
- package/scripts/demo-i6d-gate.sh +104 -0
- package/src/core/audit.js +142 -1
- package/workflows/corps.yaml +3 -0
- package/workflows/yolo.yaml +3 -0
- package/xflow/atoms/i6c_tdd_quality_review.py +47 -0
- package/xflow/atoms/i6d_assertion_execution.py +339 -31
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,26 @@
|
|
|
3
3
|
All notable changes to this project are documented here. Detailed notes live in
|
|
4
4
|
[RELEASE_NOTES.md](RELEASE_NOTES.md).
|
|
5
5
|
|
|
6
|
+
## [0.5.18] - 2026-11-09
|
|
7
|
+
|
|
8
|
+
### Added
|
|
9
|
+
- **Export proof verdicts as SARIF for code scanning (#92, closes #73)**: sealed-proof failures now surface as code scanning alerts, not just a red check and a PR comment. `action/sarif.mjs` maps each proof-failure verdict to a SARIF 2.1.0 rule — binding_mismatch / snapshot_mismatch / green_tree_mismatch / reexec_* and the other validator failures (level error; warning for tests_skipped_or_todo / tests_counts_unknown) — one result per named file (file-level location; the validator records no line numbers), degrading to the proof file itself when a verdict has no file location. Stable `partialFingerprints` (`change-id/phase/attempt/verdict/file`) so re-runs of the same proof update the existing alert instead of opening a duplicate; merging upserts per proof, so a valid proof clears its alert. `valid` / `not_required` / infrastructure verdicts produce zero results. `verify-proof.mjs` gains `--sarif-out <path>` (with matching `sarif-out` input and `sarif_path` / `sarif_written` outputs in action.yml); `proof-ledger.yml` passes `--sarif-out evalid-proof.sarif` per proof and uploads with `github/codeql-action/upload-sarif` (`security-events: write`; upload runs on `always()` so it fires exactly when a proof failed, and is `continue-on-error` so fork PRs stay green). Test evidence: 11 new tests in `test/github-action-sarif.test.js` (verdict→rule mapping + levels, per-file locations, no-file degradation, fingerprint stability, merge upsert / valid-clears-alert, SARIF 2.1.0 shape, action.yml declarations, CLI end-to-end on a real sealed proof); PR CI runs verify 42567046823 and ledger 42567046908 both pass; merged main CI run 42567609280 success.
|
|
10
|
+
|
|
11
|
+
Issue #73 — **closed with end-to-end evidence** (comment 2026-11-09T05:46:16Z, demo PR #94, closed unmerged): a sealed green proof with a tampered `test/demo-sarif.test.js` went red — ledger run 42568411445 fail, verdict `green_tree_mismatch`, files `[test/demo-sarif.test.js]` only; SARIF upload success, analysis 2124250892 (results_count 1), code scanning alert #1 visible in the Security tab (rule `evalid/green-tree-mismatch`, severity error). Re-run (attempt 2, analysis 2124258699) opened no duplicate alert, and the same fingerprint from the earlier demo PR #93 did not create a new alert either. Restoring the test file cleared it — ledger run 42568823086 pass, verdict `valid`, analysis 2124265677 (results_count 0), alert #1 state `fixed` on refs/pull/94/merge. A green PR produces no alert (PR #92 itself has no sealed proof: no SARIF upload, no analysis; main branch alerts 0).
|
|
12
|
+
|
|
13
|
+
Acceptance debt, stated plainly: the bench-number acceptance for P1-2 / P1-3 / P2-1 / P2-3 still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
|
|
14
|
+
|
|
15
|
+
## [0.5.17] - 2026-11-09
|
|
16
|
+
|
|
17
|
+
### Added
|
|
18
|
+
- **M3 slice 3: wire I6d into xflow audit replay and the I6c gate (#89)**: `xflow audit` now runs the I6d gate in the scratch clone when the diff touches Python test files (or a pytest claim exists) and the audited repo carries the I6d plugin — an unexecuted assertion inside the diff flips the verdict to `mismatch` (exit 1) even when the replay itself is fully green; no plugin / no Python test change / pytest unavailable is recorded `not applicable` and leaves the verdict alone (`--no-i6d` skips it). The I6c gate folds `specs/changes/<id>/tdd/i6d-0.json` evidence into fail findings (code `unexecuted_assertions`, one line per assertion). Test evidence: local full `npm test` 496 tests / 495 pass / 0 fail / 1 skipped (pre-existing); merged main CI run 42551898032 success.
|
|
19
|
+
- **M3 slice 4: acceptance demo for the I6d assertion-execution gate (#90)**: `scripts/demo-i6d-gate.sh` proves the gate end to end through the real `xflow audit` CLI (not the JS API) with a RED/GREEN double control — RED: a diff adding a never-called callback (`assert value == 42` never executes) with a claimed all-green pytest run is judged `mismatch`, exit 1, `i6d: unexecuted_assertions` (1/3 executed, 1 unexecuted); GREEN: a diff adding only really-executed assertions is judged `verified`, exit 0, `i6d: pass`. Test evidence: new test `test/i6d-acceptance-demo.test.js` 1/1 pass; `test/xflow-audit.test.js` 15/15 pass; local full `npm test` 511 tests / 510 pass / 0 fail / 1 skipped (pre-existing); merged main CI run 42557796622 success.
|
|
20
|
+
- **Concurrency policy test coverage, external contribution (#88, closes #87)**: 14 new tests in `test/concurrency-policy.test.js` covering `createConcurrencyPolicy()` — default settings, configured maxima and fractional normalization, invalid limits with fallback, mission-level overrides, atom/registry parallel weights, fixed and weighted charging. No production code changed. Test evidence: contributor ran the new file 14/14 pass (full suite was blocked locally by a missing dependency on their machine); full-suite verification after merge comes from main CI run 42556898257 success.
|
|
21
|
+
|
|
22
|
+
Milestone M3 "I6d assertion-execution gate" — **accepted** 2026-11-09. Acceptance evidence (issue #69, comment 6731688418): the slice-4 demo proves the gate's acceptance standard end to end — every new/changed assertion must actually execute during the green run, and an unexecuted assertion does not count as green (RED case: assertion never executed while the tests all pass, still `mismatch`; GREEN case: all assertions executed, `verified`). Demo script `scripts/demo-i6d-gate.sh` is in main and locked into the suite by `test/i6d-acceptance-demo.test.js`; the #69 checklist box is ticked.
|
|
23
|
+
|
|
24
|
+
Acceptance debt, stated plainly: the bench-number acceptance for P1-2 / P1-3 / P2-1 / P2-3 still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
|
|
25
|
+
|
|
6
26
|
## [0.5.16] - 2026-11-09
|
|
7
27
|
|
|
8
28
|
### Added
|
package/README.md
CHANGED
|
@@ -231,6 +231,13 @@ claims found in the supplied text), `unverifiable` (a claimed command
|
|
|
231
231
|
could not run). Exit codes are 0 / 1 / 3 / 1 respectively (2 = misuse).
|
|
232
232
|
`--json` prints the machine-readable report; `--out <file>` writes it.
|
|
233
233
|
|
|
234
|
+
When the diff touches Python test files and the audited repo carries the
|
|
235
|
+
I6d plugin (`xflow/atoms/i6d_assertion_execution.py`), the audit also runs
|
|
236
|
+
the assertion-execution gate: an added or changed assertion that never
|
|
237
|
+
executes during the green replay flips the verdict to `mismatch`, even
|
|
238
|
+
when every claimed run replays green. `scripts/demo-i6d-gate.sh` is a
|
|
239
|
+
self-contained red/green demonstration of that gate (M3).
|
|
240
|
+
|
|
234
241
|
## Commands
|
|
235
242
|
|
|
236
243
|
The installed binary is `xflow` (`npm install -g evalid` installs the `xflow` CLI). Every
|
package/RELEASE_NOTES.md
CHANGED
|
@@ -1,5 +1,29 @@
|
|
|
1
1
|
# Release Notes
|
|
2
2
|
|
|
3
|
+
## 0.5.18 - Proof verdicts as SARIF for code scanning (issue #73 closed)
|
|
4
|
+
|
|
5
|
+
Patch release. One merged PR since 0.5.17, verified end to end in CI and locally, with the Security-tab acceptance evidence from a demo PR.
|
|
6
|
+
|
|
7
|
+
**Export proof verdicts as SARIF for code scanning (#92, closes #73).** Sealed-proof failures now surface as code scanning alerts, not just a red check and a PR comment. `action/sarif.mjs` maps each proof-failure verdict to a SARIF 2.1.0 rule (binding / snapshot / green-tree / reexec failures at error level; skipped-tests and unknown-counts at warning), one result per named file, with stable fingerprints so re-runs update the existing alert instead of duplicating it and a valid proof clears its alert. `verify-proof.mjs` gains `--sarif-out`, and `proof-ledger.yml` uploads the SARIF per proof (fires exactly when a proof failed; fork PRs stay green). Test evidence: 11 new tests in `test/github-action-sarif.test.js`; PR CI runs verify 42567046823 and ledger 42567046908 both pass; merged main CI run 42567609280 success.
|
|
8
|
+
|
|
9
|
+
**Issue #73 — closed with end-to-end evidence** (demo PR #94, closed unmerged): a sealed green proof with a tampered `test/demo-sarif.test.js` went red — ledger run 42568411445 fail, verdict `green_tree_mismatch`, files `[test/demo-sarif.test.js]` only; SARIF upload success, analysis 2124250892 (results_count 1), code scanning alert #1 visible in the Security tab (rule `evalid/green-tree-mismatch`, severity error). A re-run opened no duplicate alert, and restoring the test file cleared it — ledger run 42568823086 pass, verdict `valid`, analysis 2124265677 (results_count 0), alert #1 state `fixed` on refs/pull/94/merge. A green PR produces no alert (main branch alerts 0).
|
|
10
|
+
|
|
11
|
+
Acceptance debt, stated plainly: the bench-number acceptance for P1-2, P1-3, P2-1 and P2-3 — the wall-clock/token/resolved lines from issue #5 — still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
|
|
12
|
+
|
|
13
|
+
## 0.5.17 - I6d assertion-execution gate (milestone M3 accepted)
|
|
14
|
+
|
|
15
|
+
Patch release. Three merged PRs since 0.5.16, all verified end to end in CI and locally.
|
|
16
|
+
|
|
17
|
+
**M3 slice 3: I6d wired into `xflow audit` replay and the I6c gate (#89).** When an audited diff touches Python test files, the audit now runs the I6d gate in the scratch clone: an unexecuted assertion inside the diff flips the verdict to `mismatch` even when the replay itself is fully green; repos without the I6d plugin are unaffected. The I6c gate folds the same evidence into fail findings naming each unexecuted assertion. Test evidence: local full `npm test` 496 tests / 495 pass / 0 fail / 1 skipped (pre-existing); merged main CI run 42551898032 success.
|
|
18
|
+
|
|
19
|
+
**M3 slice 4: acceptance demo (#90).** `scripts/demo-i6d-gate.sh` proves the gate through the real `xflow audit` CLI with a RED/GREEN double control — RED: a never-called callback's `assert value == 42` with a claimed all-green pytest run is judged `mismatch`, exit 1, `i6d: unexecuted_assertions`; GREEN: only really-executed assertions are judged `verified`, exit 0, `i6d: pass`. Test evidence: new test `test/i6d-acceptance-demo.test.js` 1/1 pass; `test/xflow-audit.test.js` 15/15 pass; local full `npm test` 511 tests / 510 pass / 0 fail / 1 skipped (pre-existing); merged main CI run 42557796622 success.
|
|
20
|
+
|
|
21
|
+
**Concurrency policy test coverage, external contribution (#88, closes #87).** 14 new tests in `test/concurrency-policy.test.js` covering `createConcurrencyPolicy()` limits, weights and overrides; no production code changed. Test evidence: contributor ran the new file 14/14 pass; full-suite verification after merge comes from main CI run 42556898257 success.
|
|
22
|
+
|
|
23
|
+
**Milestone M3 "I6d assertion-execution gate" — accepted.** Evidence in issue #69, comment 6731688418: the slice-4 demo proves the acceptance standard end to end — every new/changed assertion must actually execute during the green run, and an unexecuted assertion does not count as green. The demo script is in main and locked into the suite; the #69 checklist box is ticked.
|
|
24
|
+
|
|
25
|
+
Acceptance debt, stated plainly: the bench-number acceptance for P1-2, P1-3, P2-1 and P2-3 — the wall-clock/token/resolved lines from issue #5 — still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
|
|
26
|
+
|
|
3
27
|
## 0.5.16 - M3 slices 1–2: pytest assertion-execution tracking, diff-scoped (milestone M3 not yet accepted)
|
|
4
28
|
|
|
5
29
|
Patch release. Two merged slices since 0.5.15, both verified end to end in CI and locally.
|
package/atoms/registry.json
CHANGED
|
@@ -238,6 +238,19 @@
|
|
|
238
238
|
"tdd_mode": "red-green|refactor|docs?"
|
|
239
239
|
}
|
|
240
240
|
},
|
|
241
|
+
"I6d.tdd.assertion_execution": {
|
|
242
|
+
"track": "lite",
|
|
243
|
+
"type": "python",
|
|
244
|
+
"script": "xflow/atoms/i6d_assertion_execution.py",
|
|
245
|
+
"description": "Prove every new/changed pytest assertion really executed during the green run (M3 slice 3). Runs after I6b and before I6c, so I6c can fold its verdict into quality-0.json.",
|
|
246
|
+
"input_schema": {
|
|
247
|
+
"project_root": "string",
|
|
248
|
+
"change_id": "string",
|
|
249
|
+
"only_changed": "boolean?",
|
|
250
|
+
"diff_base": "string?",
|
|
251
|
+
"include_uncommitted": "boolean?"
|
|
252
|
+
}
|
|
253
|
+
},
|
|
241
254
|
"J4a.spec_consistency.structural": {
|
|
242
255
|
"track": "lite",
|
|
243
256
|
"type": "python",
|
package/bin/xflow.js
CHANGED
|
@@ -5810,6 +5810,8 @@ async function handleAudit(args) {
|
|
|
5810
5810
|
headRef,
|
|
5811
5811
|
claimsText,
|
|
5812
5812
|
timeoutMs,
|
|
5813
|
+
python: process.env.PYTHON || 'python3',
|
|
5814
|
+
i6d: !parsedArgs.no_i6d,
|
|
5813
5815
|
keepWorkdir: Boolean(parsedArgs.keep_workdir),
|
|
5814
5816
|
});
|
|
5815
5817
|
if (parsedArgs.out) writeFileSync(resolve(parsedArgs.out), `${JSON.stringify(report, null, 2)}\n`);
|
|
@@ -6740,7 +6742,11 @@ Audit any git diff / PR without adopting evalid workflows: extract the test
|
|
|
6740
6742
|
runs the change claims to have passed, replay each one in a clean clone at
|
|
6741
6743
|
the change's head, and compare exit codes and pass/fail counts against the
|
|
6742
6744
|
claim. Diffs carrying an evalid proof seal are replayed through the existing
|
|
6743
|
-
i6b seal validator.
|
|
6745
|
+
i6b seal validator. When the diff touches Python test files and the audited
|
|
6746
|
+
repo carries the I6d plugin, the I6d assertion-execution gate also runs in
|
|
6747
|
+
the scratch clone (scoped to the diff): unexecuted in-diff assertions flip
|
|
6748
|
+
the verdict to mismatch, since a green run that never executed an assertion
|
|
6749
|
+
must not count as green. No seal semantics, CLI name, or bin path changes.
|
|
6744
6750
|
|
|
6745
6751
|
Verdicts / exit codes: verified 0, mismatch 1, unverifiable 1, misuse 2,
|
|
6746
6752
|
no_claims 3 (no test claims found in the supplied text).
|
|
@@ -6754,6 +6760,7 @@ Options:
|
|
|
6754
6760
|
--claims TEXT Claims text (default: PR body for --pr)
|
|
6755
6761
|
--claims-file FILE Read claims text from a file
|
|
6756
6762
|
--timeout SEC Per-command replay timeout, default: 300
|
|
6763
|
+
--no-i6d Skip the I6d assertion-execution gate
|
|
6757
6764
|
--json Print the machine-readable report
|
|
6758
6765
|
--out FILE Write the JSON report to a file
|
|
6759
6766
|
--keep-workdir Keep the scratch clone for debugging
|
package/docs/launch-dossier.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# evalid Launch Dossier
|
|
2
2
|
|
|
3
|
-
Generated: 2026-11-
|
|
3
|
+
Generated: 2026-11-09T05:51:48.645Z
|
|
4
4
|
Mode: post_publish
|
|
5
5
|
Verdict: conditionally ready: engine and adoption evidence are strong, registry publication remains required for a splash launch
|
|
6
6
|
Launch ready: not yet
|
|
@@ -117,7 +117,7 @@ Evidence:
|
|
|
117
117
|
- Adoption records: 1; status: pass
|
|
118
118
|
- Goal audit: pass; goal: Keep evalid credible as an open-source agent delivery runtime: repo-owned workflow contracts, executable verification gates, durable project context, and honest benchmark evidence.
|
|
119
119
|
- Package evidence: missing; source: registry_json
|
|
120
|
-
- Package name/version: @surlymochan/evalid@0.5.
|
|
120
|
+
- Package name/version: @surlymochan/evalid@0.5.18
|
|
121
121
|
|
|
122
122
|
## Passing Evidence
|
|
123
123
|
|
package/docs/workflow-manual.md
CHANGED
|
@@ -49,7 +49,7 @@
|
|
|
49
49
|
└────────────────────────────┬────────────────────────────────────┘
|
|
50
50
|
│ atom dispatch
|
|
51
51
|
┌────────────────────────────▼────────────────────────────────────┐
|
|
52
|
-
│ L0 原子组件(
|
|
52
|
+
│ L0 原子组件(86 atoms,每个只有一处实现) │
|
|
53
53
|
│ │
|
|
54
54
|
│ xflow/atoms/*.py A/B/C/E6/H1/I6/J/K (lite) │
|
|
55
55
|
│ src/core/atoms/**/*.js D/E/F/G/E3/E4/E7 (heavy) │
|
|
@@ -219,7 +219,7 @@ CHANGE_ID=<id> xflow workflow run workflows/yolo.yaml
|
|
|
219
219
|
│ ack
|
|
220
220
|
▼
|
|
221
221
|
┌──────────────────────┐
|
|
222
|
-
│ 21b tdd-green │ atoms: I6a(green) + I6b(green) + I6c quality
|
|
222
|
+
│ 21b tdd-green │ atoms: I6a(green) + I6b(green) + I6d assertion + I6c quality
|
|
223
223
|
│ gate: artifact-verify│◄─── green/refactor 必须通过
|
|
224
224
|
└──────────┬───────────┘
|
|
225
225
|
│ exist
|
|
@@ -251,6 +251,7 @@ specs/changes/<change_id>/
|
|
|
251
251
|
├── tdd/
|
|
252
252
|
│ ├── red-0.json (I6a/I6b @ phase 21a, passed:false)
|
|
253
253
|
│ ├── green-0.json (I6a/I6b @ phase 21b, passed:true)
|
|
254
|
+
│ ├── i6d-0.json (I6d @ phase 21b, assertion execution)
|
|
254
255
|
│ └── quality-0.json (I6c @ phase 21b, test quality review)
|
|
255
256
|
├── merge-product.md (A5 @ archive)
|
|
256
257
|
├── merge-architecture.md
|
|
@@ -377,8 +378,8 @@ CHANGE_ID=<id> xflow workflow run workflows/corps.yaml
|
|
|
377
378
|
┌──────────────────┐
|
|
378
379
|
│ 21a tdd-red │ I6a(red) + I6b(red)
|
|
379
380
|
│ 22 execute │ implementation happens here
|
|
380
|
-
│ 21b tdd-green │ I6a(green) + I6b(green) + I6c quality
|
|
381
|
-
│ gate: artifact │◄── red-0.json + green-0.json + quality-0.json
|
|
381
|
+
│ 21b tdd-green │ I6a(green) + I6b(green) + I6d assertion + I6c quality
|
|
382
|
+
│ gate: artifact │◄── red-0.json + green-0.json + i6d-0.json + quality-0.json
|
|
382
383
|
└────────┬─────────┘
|
|
383
384
|
│
|
|
384
385
|
▼
|
|
@@ -627,6 +628,8 @@ J 管验证 K 管收尾归档
|
|
|
627
628
|
│ │ I6b tdd.proof_validate 重算 proof 绑定,在快照上重跑命令(lite) │
|
|
628
629
|
│ │ I6c tdd.quality_review 测试质量、红绿命令、skip/弱断言; │
|
|
629
630
|
│ │ tdd_mode=refactor|docs 时替代 red 证明 │
|
|
631
|
+
│ │ I6d tdd.assertion_execution 新增/改动 pytest 断言真实执行证明; │
|
|
632
|
+
│ │ 无改动 pytest 测试时直接通过(lite) │
|
|
630
633
|
├────┼──────────────────────────────────────────────────────────────────────┤
|
|
631
634
|
│ J │ Judge / 验证层:确定性测试 + 漂移扫描 + 证据收集 mixed 3 │
|
|
632
635
|
│ │ Verify │
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@surlymochan/evalid",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.18",
|
|
4
4
|
"description": "Claimed != verified: a runtime verification layer for coding agents. It seals each test run and replays it from a clean snapshot, with a zero-judge gate pipeline (spec clarification, code review, TDD proof) and the xflow CLI.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai-agents",
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# M3 slice 4 — acceptance demo for the I6d assertion-execution gate.
|
|
3
|
+
#
|
|
4
|
+
# End-to-end proof, through the real `xflow audit` CLI (not the JS API):
|
|
5
|
+
# RED — a diff whose new test defines a callback that is never called,
|
|
6
|
+
# so its `assert value == 42` never executes, while the claimed
|
|
7
|
+
# pytest run still passes -> audit must say mismatch, exit 1.
|
|
8
|
+
# GREEN — a diff whose new assertions all execute, same green claim
|
|
9
|
+
# -> audit must say verified, exit 0.
|
|
10
|
+
#
|
|
11
|
+
# The gate's job (M3, parent #69): an assertion that never runs must not
|
|
12
|
+
# count as green. The RED case is exactly the README gap ("assertions in
|
|
13
|
+
# callbacks that never run").
|
|
14
|
+
#
|
|
15
|
+
# Usage: scripts/demo-i6d-gate.sh
|
|
16
|
+
# Everything happens in a throwaway repo under $TMPDIR; nothing is pushed.
|
|
17
|
+
# Needs a python with pytest (ambient, else a throwaway venv is built and
|
|
18
|
+
# pip installs pytest — same approach as test/xflow-audit.test.js).
|
|
19
|
+
set -euo pipefail
|
|
20
|
+
|
|
21
|
+
EVALID_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
22
|
+
WORK="$(mktemp -d "${TMPDIR:-/tmp}/evalid-i6d-demo-XXXXXX")"
|
|
23
|
+
trap 'rm -rf "$WORK"' EXIT
|
|
24
|
+
|
|
25
|
+
# --- python with pytest -------------------------------------------------
|
|
26
|
+
# EVALID_I6D_PYTEST_PYTHON lets a caller (e.g. the node test) hand over an
|
|
27
|
+
# already-provisioned interpreter instead of building a second venv.
|
|
28
|
+
PY="${EVALID_I6D_PYTEST_PYTHON:-python3}"
|
|
29
|
+
if ! "$PY" -c 'import pytest' 2>/dev/null; then
|
|
30
|
+
echo '[demo] no ambient pytest; building a throwaway venv…'
|
|
31
|
+
"$PY" -m venv "$WORK/venv"
|
|
32
|
+
"$WORK/venv/bin/python" -m pip install --quiet pytest
|
|
33
|
+
PY="$WORK/venv/bin/python"
|
|
34
|
+
fi
|
|
35
|
+
export PYTHON="$PY"
|
|
36
|
+
export PATH="$(dirname "$PY"):$PATH"
|
|
37
|
+
echo "[demo] pytest python: $PY"
|
|
38
|
+
|
|
39
|
+
# --- throwaway demo repo (carries the I6d plugin, like a real adopter) ---
|
|
40
|
+
DEMO="$WORK/repo"
|
|
41
|
+
mkdir -p "$DEMO/xflow/atoms"
|
|
42
|
+
cp "$EVALID_ROOT/xflow/atoms/i6d_assertion_execution.py" "$DEMO/xflow/atoms/"
|
|
43
|
+
cd "$DEMO"
|
|
44
|
+
git init -q -b main
|
|
45
|
+
git config user.email 'i6d-demo@example.test'
|
|
46
|
+
git config user.name 'i6d-demo'
|
|
47
|
+
printf 'def test_normal():\n assert 1 + 1 == 2\n' > test_dead.py
|
|
48
|
+
git add .
|
|
49
|
+
git commit -qm base
|
|
50
|
+
BASE="$(git rev-parse HEAD)"
|
|
51
|
+
|
|
52
|
+
run_case() { # $1 = RED|GREEN, $2 = file content, $3 = expected verdict, $4 = expected exit
|
|
53
|
+
local name="$1" content="$2" want_verdict="$3" want_exit="$4"
|
|
54
|
+
printf '%s' "$content" > test_dead.py
|
|
55
|
+
git diff > "$WORK/$name.diff"
|
|
56
|
+
printf 'Tests: `pytest test_dead.py` — 2 tests, all pass.\n' > "$WORK/$name.claims.txt"
|
|
57
|
+
git checkout -q -- test_dead.py
|
|
58
|
+
echo "=== $name case: xflow audit --diff $name.diff --base $BASE ==="
|
|
59
|
+
set +e
|
|
60
|
+
node "$EVALID_ROOT/bin/xflow.js" audit --repo "$DEMO" \
|
|
61
|
+
--diff "$WORK/$name.diff" --base "$BASE" \
|
|
62
|
+
--claims-file "$WORK/$name.claims.txt" --out "$WORK/$name.json"
|
|
63
|
+
local rc=$?
|
|
64
|
+
set -e
|
|
65
|
+
local verdict
|
|
66
|
+
verdict="$(node -p "require('$WORK/$name.json').verdict")"
|
|
67
|
+
local i6d_verdict unexecuted
|
|
68
|
+
i6d_verdict="$(node -p "require('$WORK/$name.json').i6d.verdict")"
|
|
69
|
+
unexecuted="$(node -p "require('$WORK/$name.json').i6d.unexecuted.map(u => u.file + ':' + u.line + ' ' + u.source).join('; ')")"
|
|
70
|
+
echo "[$name] audit exit=$rc verdict=$verdict i6d=$i6d_verdict unexecuted=[$unexecuted]"
|
|
71
|
+
if [ "$verdict" != "$want_verdict" ] || [ "$rc" != "$want_exit" ]; then
|
|
72
|
+
echo "[$name] FAIL: expected verdict=$want_verdict exit=$want_exit" >&2
|
|
73
|
+
return 1
|
|
74
|
+
fi
|
|
75
|
+
echo "[$name] PASS"
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
RED_CONTENT='def test_callback_never_runs():
|
|
79
|
+
def on_done(value):
|
|
80
|
+
assert value == 42
|
|
81
|
+
# on_done is defined but never called: its assert never executes.
|
|
82
|
+
assert True
|
|
83
|
+
|
|
84
|
+
|
|
85
|
+
def test_normal():
|
|
86
|
+
assert 1 + 1 == 2
|
|
87
|
+
'
|
|
88
|
+
|
|
89
|
+
GREEN_CONTENT='def test_normal():
|
|
90
|
+
assert 1 + 1 == 2
|
|
91
|
+
|
|
92
|
+
|
|
93
|
+
def test_new_ok():
|
|
94
|
+
assert 2 + 2 == 4
|
|
95
|
+
'
|
|
96
|
+
|
|
97
|
+
run_case RED "$RED_CONTENT" mismatch 1
|
|
98
|
+
run_case GREEN "$GREEN_CONTENT" verified 0
|
|
99
|
+
|
|
100
|
+
echo
|
|
101
|
+
echo '=== M3 acceptance demo: PASS ==='
|
|
102
|
+
echo 'RED diff (dead callback assert) -> mismatch, exit 1 (i6d: unexecuted_assertions)'
|
|
103
|
+
echo 'GREEN diff (all asserts execute) -> verified, exit 0 (i6d: pass)'
|
|
104
|
+
echo 'A never-executed assertion does not count as green.'
|
package/src/core/audit.js
CHANGED
|
@@ -18,6 +18,17 @@
|
|
|
18
18
|
* no_claims — no test claims found in the supplied text
|
|
19
19
|
* unverifiable — a claimed command could not run (missing runner, timeout)
|
|
20
20
|
*
|
|
21
|
+
* Slice 3 (M3, issue #86) adds the I6d assertion-execution gate to the
|
|
22
|
+
* replay: when the diff touches Python test files (or a pytest run is
|
|
23
|
+
* claimed) and the audited repo carries the I6d plugin
|
|
24
|
+
* (xflow/atoms/i6d_assertion_execution.py), the gate runs in the scratch
|
|
25
|
+
* clone, scoped to the diff. A pytest run can exit 0 while an assertion
|
|
26
|
+
* inside a never-called callback never executes -- that change must not
|
|
27
|
+
* count as green, so unexecuted in-diff assertions flip the verdict to
|
|
28
|
+
* mismatch. When the plugin is absent, the diff touches no Python tests,
|
|
29
|
+
* or pytest is unavailable, the gate reports not applicable and the
|
|
30
|
+
* verdict is unchanged (the audit stays usable for any repo).
|
|
31
|
+
*
|
|
21
32
|
* Exit codes mirror the verdicts: 0 verified, 1 mismatch/unverifiable,
|
|
22
33
|
* 2 misuse, 3 no_claims (inconclusive, not a failure of the change).
|
|
23
34
|
*/
|
|
@@ -54,6 +65,101 @@ export function isTestFile(path) {
|
|
|
54
65
|
return /(^|\/)test(s)?\/|\.test\.[cm]?[jt]sx?$|_test\.go$|^test_[^/]*\.py$|_test\.py$|\.spec\.[cm]?[jt]sx?$/.test(path);
|
|
55
66
|
}
|
|
56
67
|
|
|
68
|
+
/** Python files pytest collects as tests by default (test_*.py / *_test.py). */
|
|
69
|
+
export function isPythonTestFile(path) {
|
|
70
|
+
const name = String(path || '').split('/').pop();
|
|
71
|
+
return /\.py$/.test(name) && (name.startsWith('test_') || name.endsWith('_test.py'));
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
/**
|
|
75
|
+
* Run the I6d assertion-execution gate in an audit scratch clone.
|
|
76
|
+
* Scope comes from the audit's diff text (--diff-file in the I6d CLI),
|
|
77
|
+
* so uncommitted diffs work without committing anything.
|
|
78
|
+
*
|
|
79
|
+
* Returns `{ applicable: false, reason }` when the gate cannot apply
|
|
80
|
+
* (no plugin in the audited repo, no Python test changes and no pytest
|
|
81
|
+
* claim, pytest unavailable), and `{ applicable: true, verdict, ... }`
|
|
82
|
+
* otherwise. `pytest_failed` in the gate run does NOT flip the audit
|
|
83
|
+
* verdict by itself -- a failing pytest run is the claim replay's job
|
|
84
|
+
* to judge; only `unexecuted_assertions` (and gate errors) change it.
|
|
85
|
+
*/
|
|
86
|
+
export function runI6dCheck(workdir, options = {}) {
|
|
87
|
+
const {
|
|
88
|
+
diffText = '',
|
|
89
|
+
claims = [],
|
|
90
|
+
python = process.env.PYTHON || 'python3',
|
|
91
|
+
timeoutMs = 300000,
|
|
92
|
+
} = options;
|
|
93
|
+
const plugin = join(workdir, 'xflow', 'atoms', 'i6d_assertion_execution.py');
|
|
94
|
+
if (!existsSync(plugin)) {
|
|
95
|
+
return { applicable: false, reason: 'plugin_not_present' };
|
|
96
|
+
}
|
|
97
|
+
const { files } = parseDiffFiles(diffText);
|
|
98
|
+
const changedPyTests = files.filter(isPythonTestFile);
|
|
99
|
+
const hasPytestClaim = claims.some((claim) => /\bpytest\b/.test(claim.command || ''));
|
|
100
|
+
if (changedPyTests.length === 0 && !hasPytestClaim) {
|
|
101
|
+
return { applicable: false, reason: 'no_python_test_changes' };
|
|
102
|
+
}
|
|
103
|
+
const targets = changedPyTests.filter((file) => existsSync(join(workdir, file)));
|
|
104
|
+
if (targets.length === 0 && !hasPytestClaim) {
|
|
105
|
+
return { applicable: false, reason: 'no_python_test_changes' };
|
|
106
|
+
}
|
|
107
|
+
const pytestCheck = spawnSync(python, ['-c', 'import pytest'], { cwd: workdir, encoding: 'utf8', timeout: 60000 });
|
|
108
|
+
if (pytestCheck.status !== 0) {
|
|
109
|
+
return { applicable: false, reason: 'pytest_unavailable' };
|
|
110
|
+
}
|
|
111
|
+
const scratch = mkdtempSync(join(tmpdir(), 'evalid-audit-i6d-'));
|
|
112
|
+
try {
|
|
113
|
+
const diffPath = join(scratch, 'audit.diff');
|
|
114
|
+
const evidencePath = join(scratch, 'i6d-evidence.json');
|
|
115
|
+
writeFileSync(diffPath, diffText);
|
|
116
|
+
const result = spawnSync(python, [
|
|
117
|
+
plugin,
|
|
118
|
+
'--root', workdir,
|
|
119
|
+
'--output', evidencePath,
|
|
120
|
+
'--pytest-python', python,
|
|
121
|
+
'--only-changed',
|
|
122
|
+
'--diff-file', diffPath,
|
|
123
|
+
'--',
|
|
124
|
+
...targets,
|
|
125
|
+
], { cwd: workdir, encoding: 'utf8', timeout: timeoutMs });
|
|
126
|
+
if (!existsSync(evidencePath)) {
|
|
127
|
+
return {
|
|
128
|
+
applicable: true,
|
|
129
|
+
verdict: 'i6d_error',
|
|
130
|
+
error: `i6d gate wrote no evidence (exit ${result.status}): ${(result.stderr || result.stdout || '').trim().slice(-300)}`,
|
|
131
|
+
};
|
|
132
|
+
}
|
|
133
|
+
let evidence;
|
|
134
|
+
try {
|
|
135
|
+
evidence = JSON.parse(readFileSync(evidencePath, 'utf8'));
|
|
136
|
+
} catch {
|
|
137
|
+
return { applicable: true, verdict: 'i6d_error', error: 'i6d evidence JSON unparseable' };
|
|
138
|
+
}
|
|
139
|
+
const unexecuted = (evidence.unexecuted || []).map((entry) => ({
|
|
140
|
+
file: String(entry.file || '').startsWith(workdir)
|
|
141
|
+
? String(entry.file).slice(workdir.length + 1)
|
|
142
|
+
: entry.file,
|
|
143
|
+
line: entry.line,
|
|
144
|
+
test: entry.test,
|
|
145
|
+
source: entry.source,
|
|
146
|
+
}));
|
|
147
|
+
return {
|
|
148
|
+
applicable: true,
|
|
149
|
+
verdict: evidence.verdict,
|
|
150
|
+
exit: result.status,
|
|
151
|
+
executed: evidence.executed ?? 0,
|
|
152
|
+
total: evidence.total ?? 0,
|
|
153
|
+
unexecuted,
|
|
154
|
+
skipped: (evidence.skipped || []).length,
|
|
155
|
+
outOfScope: (evidence.outOfScope || []).length,
|
|
156
|
+
pytestExit: evidence.pytestExit ?? null,
|
|
157
|
+
};
|
|
158
|
+
} finally {
|
|
159
|
+
rmSync(scratch, { recursive: true, force: true });
|
|
160
|
+
}
|
|
161
|
+
}
|
|
162
|
+
|
|
57
163
|
function normalizeCommand(raw) {
|
|
58
164
|
return String(raw || '').replace(/\s+/g, ' ').trim();
|
|
59
165
|
}
|
|
@@ -322,6 +428,10 @@ function replaySeal(workdir, repoRoot, seal, python, timeoutMs) {
|
|
|
322
428
|
* timeoutMs — per-command timeout (default 300_000)
|
|
323
429
|
* python — python binary for the i6b validator (default 'python3')
|
|
324
430
|
* keepWorkdir — keep the scratch clone for debugging (default false)
|
|
431
|
+
* python — python binary for the i6b validator and the i6d gate
|
|
432
|
+
* (default 'python3' / $PYTHON)
|
|
433
|
+
* i6d — run the I6d assertion-execution gate when it applies
|
|
434
|
+
* (default true; --no-i6d on the CLI disables it)
|
|
325
435
|
*/
|
|
326
436
|
export function runAudit(options = {}) {
|
|
327
437
|
const {
|
|
@@ -333,6 +443,7 @@ export function runAudit(options = {}) {
|
|
|
333
443
|
timeoutMs = 300000,
|
|
334
444
|
python = process.env.PYTHON || 'python3',
|
|
335
445
|
keepWorkdir = false,
|
|
446
|
+
i6d = true,
|
|
336
447
|
} = options;
|
|
337
448
|
if (!repoDir || !existsSync(repoDir)) {
|
|
338
449
|
return { ok: false, verdict: 'misuse', error: `repo not found: ${repoDir}`, exitCode: 2 };
|
|
@@ -432,14 +543,28 @@ export function runAudit(options = {}) {
|
|
|
432
543
|
const sealResults = diff.seals.map((seal) => replaySeal(workdir, resolve(repoDir), seal, python, timeoutMs));
|
|
433
544
|
const badSeal = sealResults.find((seal) => !seal.ok);
|
|
434
545
|
|
|
546
|
+
// Slice 3 (M3): the I6d gate judges whether the diff's new/changed
|
|
547
|
+
// assertions really executed during the green run. A replay can exit 0
|
|
548
|
+
// while an assertion in a never-called callback never ran -- that
|
|
549
|
+
// change must not count as green.
|
|
550
|
+
const i6dResult = i6d === false
|
|
551
|
+
? { applicable: false, reason: 'disabled' }
|
|
552
|
+
: runI6dCheck(workdir, { diffText, claims, python, timeoutMs });
|
|
553
|
+
|
|
435
554
|
let verdict;
|
|
436
555
|
let exitCode;
|
|
437
|
-
if (
|
|
556
|
+
if (i6dResult.applicable && i6dResult.verdict === 'unexecuted_assertions') {
|
|
557
|
+
verdict = 'mismatch';
|
|
558
|
+
exitCode = 1;
|
|
559
|
+
} else if (claims.length === 0 && sealResults.length === 0) {
|
|
438
560
|
verdict = 'no_claims';
|
|
439
561
|
exitCode = 3;
|
|
440
562
|
} else if (mismatch || badSeal) {
|
|
441
563
|
verdict = 'mismatch';
|
|
442
564
|
exitCode = 1;
|
|
565
|
+
} else if (i6dResult.applicable && (i6dResult.verdict === 'diff_error' || i6dResult.verdict === 'i6d_error')) {
|
|
566
|
+
verdict = 'unverifiable';
|
|
567
|
+
exitCode = 1;
|
|
443
568
|
} else if (claimResults.some((entry) => entry.replay && (entry.replay.timedOut || entry.replay.spawnError))) {
|
|
444
569
|
verdict = 'unverifiable';
|
|
445
570
|
exitCode = 1;
|
|
@@ -454,6 +579,7 @@ export function runAudit(options = {}) {
|
|
|
454
579
|
exitCode,
|
|
455
580
|
claims: claimResults,
|
|
456
581
|
seals: sealResults,
|
|
582
|
+
i6d: i6dResult,
|
|
457
583
|
files: diff.files,
|
|
458
584
|
testFiles: diff.testFiles,
|
|
459
585
|
setup,
|
|
@@ -492,6 +618,21 @@ export function renderAuditReport(report) {
|
|
|
492
618
|
for (const seal of report.seals || []) {
|
|
493
619
|
lines.push(`- ${seal.ok ? '✓' : '✗'} seal ${seal.changeId}/${seal.phase}/${seal.attempt}: ${seal.verdict}`);
|
|
494
620
|
}
|
|
621
|
+
const gate = report.i6d;
|
|
622
|
+
if (gate) {
|
|
623
|
+
if (!gate.applicable) {
|
|
624
|
+
lines.push(`i6d: not applicable (${gate.reason})`);
|
|
625
|
+
} else if (gate.verdict === 'pass') {
|
|
626
|
+
lines.push(`i6d: pass (${gate.executed}/${gate.total} assertions executed, ${gate.skipped} skipped, ${gate.outOfScope} out of scope)`);
|
|
627
|
+
} else if (gate.verdict === 'unexecuted_assertions') {
|
|
628
|
+
lines.push(`i6d: ✗ unexecuted_assertions (${gate.executed}/${gate.total} executed, ${gate.unexecuted.length} unexecuted)`);
|
|
629
|
+
for (const entry of gate.unexecuted) {
|
|
630
|
+
lines.push(` unexecuted: ${entry.file}:${entry.line}: ${entry.source}`);
|
|
631
|
+
}
|
|
632
|
+
} else {
|
|
633
|
+
lines.push(`i6d: ${gate.verdict}${gate.error ? `: ${gate.error}` : ''}`);
|
|
634
|
+
}
|
|
635
|
+
}
|
|
495
636
|
if (report.durationMs !== undefined) lines.push(`duration: ${(report.durationMs / 1000).toFixed(1)}s`);
|
|
496
637
|
return lines.join('\n');
|
|
497
638
|
}
|
package/workflows/corps.yaml
CHANGED
|
@@ -398,10 +398,13 @@ phases:
|
|
|
398
398
|
with: { phase: "green", test_command: "${input.tdd_green_command}", expect: "pass" }
|
|
399
399
|
- id: I6b.tdd.proof_validate
|
|
400
400
|
with: { phase: "green", tdd_mode: "${input.tdd_mode}" }
|
|
401
|
+
- id: I6d.tdd.assertion_execution
|
|
402
|
+
with: { only_changed: true, include_uncommitted: true }
|
|
401
403
|
- id: I6c.tdd.quality_review
|
|
402
404
|
with: { tdd_mode: "${input.tdd_mode}" }
|
|
403
405
|
artifacts:
|
|
404
406
|
- { path: "specs/changes/${change_id}/tdd/green-0.json", optional: false }
|
|
407
|
+
- { path: "specs/changes/${change_id}/tdd/i6d-0.json", optional: false }
|
|
405
408
|
- { path: "specs/changes/${change_id}/tdd/quality-0.json", optional: false }
|
|
406
409
|
gate:
|
|
407
410
|
type: artifact-verify
|
package/workflows/yolo.yaml
CHANGED
|
@@ -187,10 +187,13 @@ phases:
|
|
|
187
187
|
with: { phase: "green", test_command: "${input.tdd_green_command}", expect: "pass" }
|
|
188
188
|
- id: I6b.tdd.proof_validate
|
|
189
189
|
with: { phase: "green", tdd_mode: "${input.tdd_mode}" }
|
|
190
|
+
- id: I6d.tdd.assertion_execution
|
|
191
|
+
with: { only_changed: true, include_uncommitted: true }
|
|
190
192
|
- id: I6c.tdd.quality_review
|
|
191
193
|
with: { tdd_mode: "${input.tdd_mode}" }
|
|
192
194
|
artifacts:
|
|
193
195
|
- { path: "specs/changes/${change_id}/tdd/green-0.json", optional: false }
|
|
196
|
+
- { path: "specs/changes/${change_id}/tdd/i6d-0.json", optional: false }
|
|
194
197
|
- { path: "specs/changes/${change_id}/tdd/quality-0.json", optional: false }
|
|
195
198
|
gate:
|
|
196
199
|
type: artifact-verify
|
|
@@ -21,6 +21,14 @@ shapes of it: expected values taken from the implementation's own constants
|
|
|
21
21
|
named callbacks that are never referenced again (they execute zero times),
|
|
22
22
|
and self-comparison assertions (x == x). Type-only or existence-only
|
|
23
23
|
assertions are flagged only when they are the sole assertions.
|
|
24
|
+
|
|
25
|
+
Slice 3 (M3, issue #86) folds the I6d assertion-execution evidence into
|
|
26
|
+
this review: when the I6d atom ran first (tdd-green phase order is I6a,
|
|
27
|
+
I6b, I6d, I6c) and wrote ``specs/changes/<id>/tdd/i6d-0.json``, an
|
|
28
|
+
``unexecuted_assertions`` verdict becomes one fail finding per unexecuted
|
|
29
|
+
assert, and a ``pytest_unavailable`` verdict becomes a fail finding of its
|
|
30
|
+
own. When no i6d-0.json exists (I6d not run, e.g. a change with no Python
|
|
31
|
+
tests) this review behaves exactly as before.
|
|
24
32
|
"""
|
|
25
33
|
import argparse
|
|
26
34
|
import json
|
|
@@ -343,6 +351,43 @@ def main() -> None:
|
|
|
343
351
|
for test_file in sorted(reviewed):
|
|
344
352
|
findings.extend(review_test_file(project_root, test_file, implementation_sources))
|
|
345
353
|
|
|
354
|
+
# Slice 3 (M3): fold the I6d assertion-execution evidence in, when the
|
|
355
|
+
# I6d atom ran first and wrote i6d-0.json. An assertion that never
|
|
356
|
+
# executed during the green run must fail the quality gate, even when
|
|
357
|
+
# the run itself was green. No i6d-0.json means I6d did not run (no
|
|
358
|
+
# Python test changes): behaviour is unchanged, no finding added.
|
|
359
|
+
i6d_evidence = read_json(proof_dir / "i6d-0.json")
|
|
360
|
+
i6d_summary = None
|
|
361
|
+
if i6d_evidence:
|
|
362
|
+
i6d_summary = {
|
|
363
|
+
"verdict": i6d_evidence.get("verdict"),
|
|
364
|
+
"executed": i6d_evidence.get("executed"),
|
|
365
|
+
"total": i6d_evidence.get("total"),
|
|
366
|
+
"unexecuted": len(i6d_evidence.get("unexecuted") or []),
|
|
367
|
+
"skipped": len(i6d_evidence.get("skipped") or []),
|
|
368
|
+
}
|
|
369
|
+
if i6d_evidence.get("verdict") == "unexecuted_assertions":
|
|
370
|
+
for entry in i6d_evidence.get("unexecuted") or []:
|
|
371
|
+
findings.append({
|
|
372
|
+
"severity": "fail",
|
|
373
|
+
"code": "unexecuted_assertions",
|
|
374
|
+
"path": entry.get("file"),
|
|
375
|
+
"line": entry.get("line"),
|
|
376
|
+
"test": entry.get("test"),
|
|
377
|
+
"source": entry.get("source"),
|
|
378
|
+
"message": "An assertion added/changed by this diff never "
|
|
379
|
+
"executed during the green run; it must not count as green.",
|
|
380
|
+
})
|
|
381
|
+
elif i6d_evidence.get("verdict") == "pytest_unavailable":
|
|
382
|
+
findings.append({
|
|
383
|
+
"severity": "fail",
|
|
384
|
+
"code": "i6d_pytest_unavailable",
|
|
385
|
+
"message": "The I6d assertion-execution gate could not run: "
|
|
386
|
+
"pytest is unavailable, so added/changed assertions could "
|
|
387
|
+
"not be proven executed.",
|
|
388
|
+
"assertions_at_stake": i6d_evidence.get("total"),
|
|
389
|
+
})
|
|
390
|
+
|
|
346
391
|
failed = any(finding.get("severity") == "fail" for finding in findings)
|
|
347
392
|
verdict = "failed" if failed else "passed"
|
|
348
393
|
proof = {
|
|
@@ -354,6 +399,7 @@ def main() -> None:
|
|
|
354
399
|
"code_files": code_files,
|
|
355
400
|
"test_files": test_files,
|
|
356
401
|
"read_cache": _read_cache.stats(),
|
|
402
|
+
"i6d": i6d_summary,
|
|
357
403
|
"findings": findings,
|
|
358
404
|
"recorded_at": datetime.now(timezone.utc).isoformat(),
|
|
359
405
|
}
|
|
@@ -368,6 +414,7 @@ def main() -> None:
|
|
|
368
414
|
"tdd_mode": mode,
|
|
369
415
|
"proof_file": str(proof_file),
|
|
370
416
|
"read_cache": _read_cache.stats(),
|
|
417
|
+
"i6d": i6d_summary,
|
|
371
418
|
"failures": sum(1 for finding in findings if finding.get("severity") == "fail"),
|
|
372
419
|
"warnings": sum(1 for finding in findings if finding.get("severity") == "warn"),
|
|
373
420
|
"failed_codes": [
|
|
@@ -39,6 +39,22 @@ and the CLI exposes the same as ``--only-changed`` / ``--diff-base``.
|
|
|
39
39
|
With scoping on, asserts outside the diff are reported as ``outOfScope``
|
|
40
40
|
(never red), an empty diff passes (exit 0), and only in-scope unexecuted
|
|
41
41
|
asserts fail the gate. Without the flag, slice 1 behaviour is unchanged.
|
|
42
|
+
|
|
43
|
+
Slice 3 wires the gate into ``xflow audit`` replay and the I6c gate:
|
|
44
|
+
|
|
45
|
+
* ``changed_lines_from_diff(diff_text, repo)`` -- the same changed-lines
|
|
46
|
+
shape as ``changed_lines`` but parsed from a unified diff *text* (the
|
|
47
|
+
audit's --diff / PR diff, which may be uncommitted or carry context
|
|
48
|
+
lines), so the audit can scope the gate without committing anything;
|
|
49
|
+
* ``changed_lines(..., uncommitted=True)`` -- additionally unions
|
|
50
|
+
``git diff HEAD`` (working tree + staged), for the workflow atom, where
|
|
51
|
+
the change under review is still uncommitted;
|
|
52
|
+
* atom mode (``--change-id`` / ``--project-root``, as invoked by the
|
|
53
|
+
workflow executor): evidence lands in
|
|
54
|
+
``specs/changes/<id>/tdd/i6d-0.json`` and stdout is a single JSON line
|
|
55
|
+
(``{ok, verdict, proof_file, ...}``), so the executor can parse it and
|
|
56
|
+
the I6c review can fold the verdict into quality-0.json. With no
|
|
57
|
+
changed Python test files the atom passes without running pytest.
|
|
42
58
|
"""
|
|
43
59
|
|
|
44
60
|
import argparse
|
|
@@ -53,7 +69,22 @@ from pathlib import Path
|
|
|
53
69
|
EVIDENCE_ENV = "EVALID_I6D_EVIDENCE"
|
|
54
70
|
DIFF_ONLY_ENV = "EVALID_I6D_ONLY_CHANGED"
|
|
55
71
|
DIFF_BASE_ENV = "EVALID_I6D_DIFF_BASE"
|
|
56
|
-
|
|
72
|
+
DIFF_FILE_ENV = "EVALID_I6D_DIFF_FILE"
|
|
73
|
+
INCLUDE_UNCOMMITTED_ENV = "EVALID_I6D_INCLUDE_UNCOMMITTED"
|
|
74
|
+
VERSION = 3
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
def is_python_test_file(path):
|
|
78
|
+
"""True for files pytest collects as tests by default.
|
|
79
|
+
|
|
80
|
+
Mirrors pytest's default ``python_files`` (``test_*.py`` /
|
|
81
|
+
``*_test.py``); kept in sync with ``isPythonTestFile`` in
|
|
82
|
+
``src/core/audit.js``.
|
|
83
|
+
"""
|
|
84
|
+
name = os.path.basename(str(path))
|
|
85
|
+
return name.endswith(".py") and (
|
|
86
|
+
name.startswith("test_") or name.endswith("_test.py")
|
|
87
|
+
)
|
|
57
88
|
|
|
58
89
|
|
|
59
90
|
# ---------------------------------------------------------------------------
|
|
@@ -127,23 +158,25 @@ def resolve_diff_base(repo, base="main"):
|
|
|
127
158
|
_HUNK_RE = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,(\d+))? @@")
|
|
128
159
|
|
|
129
160
|
|
|
130
|
-
def
|
|
131
|
-
"""
|
|
161
|
+
def _parse_unified_diff(diff_text, repo):
|
|
162
|
+
"""``{abspath: {new-side line numbers}}`` from a unified diff text.
|
|
132
163
|
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
164
|
+
Only lines actually added (``+`` lines inside a hunk) count -- context
|
|
165
|
+
lines and hunk-header ranges do not, so a -U3 diff (``gh pr diff``,
|
|
166
|
+
the audit's --diff input) scopes exactly like a -U0 one. Deleted
|
|
167
|
+
lines do not advance the new-side number; ``\\ No newline`` lines
|
|
168
|
+
neither. Deleted files and files with no added lines contribute
|
|
169
|
+
nothing.
|
|
137
170
|
"""
|
|
138
|
-
merge_base = resolve_diff_base(repo, base)
|
|
139
|
-
raw = _git(repo, "diff", merge_base, "HEAD", "-U0", "--")
|
|
140
171
|
changed = {}
|
|
141
172
|
current = None
|
|
142
|
-
|
|
173
|
+
new_lineno = None
|
|
174
|
+
for line in diff_text.splitlines():
|
|
143
175
|
if line.startswith("+++ "):
|
|
144
176
|
target = line[4:].strip()
|
|
145
177
|
if target in ("/dev/null", ""):
|
|
146
178
|
current = None
|
|
179
|
+
new_lineno = None
|
|
147
180
|
continue
|
|
148
181
|
if target.startswith("b/"):
|
|
149
182
|
target = target[2:]
|
|
@@ -151,14 +184,54 @@ def changed_lines(repo, base="main"):
|
|
|
151
184
|
target = json.loads(target)
|
|
152
185
|
current = os.path.realpath(os.path.join(repo, target))
|
|
153
186
|
changed.setdefault(current, set())
|
|
187
|
+
new_lineno = None
|
|
154
188
|
elif line.startswith("@@ ") and current is not None:
|
|
155
189
|
match = _HUNK_RE.match(line)
|
|
156
|
-
if match
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
changed[current].
|
|
161
|
-
|
|
190
|
+
new_lineno = int(match.group(1)) if match else None
|
|
191
|
+
elif current is None or new_lineno is None:
|
|
192
|
+
continue
|
|
193
|
+
elif line.startswith("+") and not line.startswith("+++"):
|
|
194
|
+
changed[current].add(new_lineno)
|
|
195
|
+
new_lineno += 1
|
|
196
|
+
elif line.startswith("-") and not line.startswith("---"):
|
|
197
|
+
pass # deleted line: the new-side number does not advance
|
|
198
|
+
elif line.startswith("\\"):
|
|
199
|
+
pass # "": no advance
|
|
200
|
+
else:
|
|
201
|
+
new_lineno += 1 # context line
|
|
202
|
+
return {path: lines for path, lines in changed.items() if lines}
|
|
203
|
+
|
|
204
|
+
|
|
205
|
+
def changed_lines_from_diff(diff_text, repo):
|
|
206
|
+
"""Changed lines from a unified diff *text* (slice 3, for the audit).
|
|
207
|
+
|
|
208
|
+
Unlike :func:`changed_lines` this needs no git state: the audit
|
|
209
|
+
replays an uncommitted diff (or a PR diff) in a scratch clone, where
|
|
210
|
+
the change is not a commit yet. Same return shape as
|
|
211
|
+
:func:`changed_lines` minus the merge-base.
|
|
212
|
+
"""
|
|
213
|
+
return _parse_unified_diff(diff_text, repo)
|
|
214
|
+
|
|
215
|
+
|
|
216
|
+
def changed_lines(repo, base="main", uncommitted=False):
|
|
217
|
+
"""Return ``(merge_base_sha, {abspath: {new-side line numbers}})``.
|
|
218
|
+
|
|
219
|
+
The diff is ``git diff <merge-base> HEAD -U0``: committed changes on
|
|
220
|
+
this branch only. With ``uncommitted=True`` the working tree + staged
|
|
221
|
+
changes vs HEAD (``git diff HEAD -U0``) are unioned in, for the
|
|
222
|
+
workflow atom, where the change under review is still uncommitted.
|
|
223
|
+
A file's changed lines are its added lines. Deleted files contribute
|
|
224
|
+
nothing; without ``uncommitted``, uncommitted working-tree edits are
|
|
225
|
+
not included.
|
|
226
|
+
"""
|
|
227
|
+
merge_base = resolve_diff_base(repo, base)
|
|
228
|
+
raw = _git(repo, "diff", merge_base, "HEAD", "-U0", "--")
|
|
229
|
+
changed = _parse_unified_diff(raw, repo)
|
|
230
|
+
if uncommitted:
|
|
231
|
+
raw_wt = _git(repo, "diff", "HEAD", "-U0", "--")
|
|
232
|
+
for path, lines in _parse_unified_diff(raw_wt, repo).items():
|
|
233
|
+
changed.setdefault(path, set()).update(lines)
|
|
234
|
+
return merge_base, changed
|
|
162
235
|
|
|
163
236
|
|
|
164
237
|
def assert_in_scope(entry, changed):
|
|
@@ -172,9 +245,9 @@ def assert_in_scope(entry, changed):
|
|
|
172
245
|
)
|
|
173
246
|
|
|
174
247
|
|
|
175
|
-
def load_diff_scope(repo, base="main"):
|
|
248
|
+
def load_diff_scope(repo, base="main", uncommitted=False):
|
|
176
249
|
"""``{'base': <merge-base sha>, 'changed': {path: [lines]}}`` (JSON-safe)."""
|
|
177
|
-
merge_base, changed = changed_lines(repo, base)
|
|
250
|
+
merge_base, changed = changed_lines(repo, base, uncommitted=uncommitted)
|
|
178
251
|
return {
|
|
179
252
|
"base": merge_base,
|
|
180
253
|
"changed": {path: sorted(lines) for path, lines in changed.items()},
|
|
@@ -218,11 +291,21 @@ class _AssertionExecutionPlugin:
|
|
|
218
291
|
def pytest_sessionstart(self, session):
|
|
219
292
|
self._old_trace = sys.gettrace()
|
|
220
293
|
sys.settrace(self._global_trace)
|
|
221
|
-
if os.environ.get(
|
|
294
|
+
if os.environ.get(DIFF_FILE_ENV):
|
|
295
|
+
# Slice 3 (audit): scope from a diff text, not from git state --
|
|
296
|
+
# the audited change may be uncommitted in a scratch clone.
|
|
297
|
+
try:
|
|
298
|
+
text = Path(os.environ[DIFF_FILE_ENV]).read_text(encoding="utf-8")
|
|
299
|
+
self.diff_changed = changed_lines_from_diff(text, self.rootdir)
|
|
300
|
+
self.diff_base = None
|
|
301
|
+
except Exception as exc: # noqa: BLE001 - surface as evidence, not a crash
|
|
302
|
+
self.diff_error = str(exc)
|
|
303
|
+
elif os.environ.get(DIFF_ONLY_ENV):
|
|
222
304
|
try:
|
|
223
305
|
base = os.environ.get(DIFF_BASE_ENV) or "main"
|
|
224
306
|
self.diff_base, self.diff_changed = changed_lines(
|
|
225
|
-
self.rootdir, base
|
|
307
|
+
self.rootdir, base,
|
|
308
|
+
uncommitted=bool(os.environ.get(INCLUDE_UNCOMMITTED_ENV)),
|
|
226
309
|
)
|
|
227
310
|
except Exception as exc: # noqa: BLE001 - surface as evidence, not a crash
|
|
228
311
|
self.diff_error = str(exc)
|
|
@@ -373,34 +456,234 @@ def write_evidence(evidence):
|
|
|
373
456
|
# CLI: run pytest under the plugin, then judge the evidence
|
|
374
457
|
# ---------------------------------------------------------------------------
|
|
375
458
|
|
|
459
|
+
def _pytest_available(pytest_python):
|
|
460
|
+
"""True when ``pytest_python`` can import pytest."""
|
|
461
|
+
try:
|
|
462
|
+
proc = subprocess.run(
|
|
463
|
+
[pytest_python, "-c", "import pytest"],
|
|
464
|
+
capture_output=True, text=True, timeout=60,
|
|
465
|
+
)
|
|
466
|
+
except OSError:
|
|
467
|
+
return False
|
|
468
|
+
return proc.returncode == 0
|
|
469
|
+
|
|
470
|
+
|
|
471
|
+
def _atom_summary(*, ok, verdict, proof_file, evidence, pytest_exit):
|
|
472
|
+
return {
|
|
473
|
+
"ok": ok,
|
|
474
|
+
"verdict": verdict,
|
|
475
|
+
"proof_file": str(proof_file),
|
|
476
|
+
"executed": evidence.get("executed", 0),
|
|
477
|
+
"total": evidence.get("total", 0),
|
|
478
|
+
"unexecuted": len(evidence.get("unexecuted", [])),
|
|
479
|
+
"skipped": len(evidence.get("skipped", [])),
|
|
480
|
+
"pytest_exit": pytest_exit,
|
|
481
|
+
}
|
|
482
|
+
|
|
483
|
+
|
|
376
484
|
def run_gate(*, root, output, pytest_python, pytest_args,
|
|
377
|
-
only_changed=False, diff_base="main"
|
|
485
|
+
only_changed=False, diff_base="main", diff_file=None,
|
|
486
|
+
include_uncommitted=False, change_id=None, project_root=None):
|
|
487
|
+
"""Run pytest under the plugin, then judge the evidence.
|
|
488
|
+
|
|
489
|
+
Slice 3 additions: ``diff_file`` scopes from a diff text instead of a
|
|
490
|
+
git range (the audit's case); ``include_uncommitted`` unions the
|
|
491
|
+
working tree into the git-range scope; ``change_id`` switches on atom
|
|
492
|
+
mode (evidence to ``specs/changes/<id>/tdd/i6d-0.json``, stdout a
|
|
493
|
+
single JSON line for the workflow executor). Atom mode always counts
|
|
494
|
+
uncommitted changes -- the change under review is still uncommitted
|
|
495
|
+
when the workflow runs -- and passes without running pytest when the
|
|
496
|
+
diff touches no Python test files.
|
|
497
|
+
"""
|
|
498
|
+
atom_mode = change_id is not None
|
|
499
|
+
if atom_mode:
|
|
500
|
+
root = project_root or root
|
|
501
|
+
include_uncommitted = True
|
|
502
|
+
root = os.path.realpath(root)
|
|
503
|
+
output = os.path.abspath(output)
|
|
504
|
+
parent = os.path.dirname(output)
|
|
505
|
+
if parent:
|
|
506
|
+
os.makedirs(parent, exist_ok=True)
|
|
507
|
+
|
|
508
|
+
# Resolve the diff scope before running pytest: the audit and the
|
|
509
|
+
# atom both need the changed Python test files as pytest targets, and
|
|
510
|
+
# the atom short-circuits when there are none.
|
|
511
|
+
changed = None
|
|
512
|
+
diff_base_out = None
|
|
513
|
+
diff_source = None
|
|
514
|
+
diff_error = None
|
|
515
|
+
if only_changed:
|
|
516
|
+
try:
|
|
517
|
+
if diff_file is not None:
|
|
518
|
+
with open(diff_file, encoding="utf-8") as handle:
|
|
519
|
+
changed = changed_lines_from_diff(handle.read(), root)
|
|
520
|
+
diff_source = "diff-file"
|
|
521
|
+
else:
|
|
522
|
+
diff_base_out, changed = changed_lines(
|
|
523
|
+
root, diff_base, uncommitted=include_uncommitted,
|
|
524
|
+
)
|
|
525
|
+
diff_source = (
|
|
526
|
+
"git-range+uncommitted" if include_uncommitted
|
|
527
|
+
else "git-range"
|
|
528
|
+
)
|
|
529
|
+
except Exception as exc: # noqa: BLE001 - surface as evidence, not a crash
|
|
530
|
+
diff_error = str(exc)
|
|
531
|
+
|
|
532
|
+
targets = []
|
|
533
|
+
if only_changed and changed is not None and not pytest_args:
|
|
534
|
+
targets = sorted(
|
|
535
|
+
os.path.relpath(path, root)
|
|
536
|
+
for path, lines in changed.items()
|
|
537
|
+
if is_python_test_file(path) and lines
|
|
538
|
+
and os.path.exists(path)
|
|
539
|
+
)
|
|
540
|
+
pytest_args = targets
|
|
541
|
+
|
|
542
|
+
if atom_mode and only_changed and not pytest_args and diff_error is None:
|
|
543
|
+
# No changed Python test files: nothing for the gate to judge.
|
|
544
|
+
# Slice 2's empty-diff contract is "pass", so the atom passes
|
|
545
|
+
# without running pytest at all.
|
|
546
|
+
evidence = {
|
|
547
|
+
"tool": "i6d_assertion_execution",
|
|
548
|
+
"version": VERSION,
|
|
549
|
+
"root": root,
|
|
550
|
+
"pytestExit": None,
|
|
551
|
+
"verdict": "pass",
|
|
552
|
+
"total": 0,
|
|
553
|
+
"executed": 0,
|
|
554
|
+
"unexecuted": [],
|
|
555
|
+
"skipped": [],
|
|
556
|
+
"diffBase": diff_base_out,
|
|
557
|
+
"diffSource": diff_source,
|
|
558
|
+
"inScope": 0,
|
|
559
|
+
"outOfScope": [],
|
|
560
|
+
"notApplicable": "no_python_test_changes",
|
|
561
|
+
}
|
|
562
|
+
with open(output, "w", encoding="utf-8") as handle:
|
|
563
|
+
json.dump(evidence, handle, indent=2)
|
|
564
|
+
print(json.dumps(_atom_summary(
|
|
565
|
+
ok=True, verdict="pass", proof_file=output,
|
|
566
|
+
evidence=evidence, pytest_exit=None,
|
|
567
|
+
)))
|
|
568
|
+
return 0
|
|
569
|
+
|
|
570
|
+
if diff_error is not None:
|
|
571
|
+
evidence = {
|
|
572
|
+
"tool": "i6d_assertion_execution",
|
|
573
|
+
"version": VERSION,
|
|
574
|
+
"root": root,
|
|
575
|
+
"pytestExit": None,
|
|
576
|
+
"verdict": "diff_error",
|
|
577
|
+
"total": 0,
|
|
578
|
+
"executed": 0,
|
|
579
|
+
"unexecuted": [],
|
|
580
|
+
"skipped": [],
|
|
581
|
+
"diffBase": diff_base_out,
|
|
582
|
+
"diffSource": diff_source,
|
|
583
|
+
"diffError": diff_error,
|
|
584
|
+
}
|
|
585
|
+
with open(output, "w", encoding="utf-8") as handle:
|
|
586
|
+
json.dump(evidence, handle, indent=2)
|
|
587
|
+
if atom_mode:
|
|
588
|
+
print(json.dumps(_atom_summary(
|
|
589
|
+
ok=False, verdict="diff_error", proof_file=output,
|
|
590
|
+
evidence=evidence, pytest_exit=None,
|
|
591
|
+
)))
|
|
592
|
+
else:
|
|
593
|
+
print(f"i6d: diff_error: {diff_error}", file=sys.stderr)
|
|
594
|
+
return 2
|
|
595
|
+
|
|
378
596
|
atoms_dir = str(Path(__file__).resolve().parent)
|
|
379
597
|
env = dict(os.environ)
|
|
380
|
-
env[EVIDENCE_ENV] =
|
|
598
|
+
env[EVIDENCE_ENV] = output
|
|
381
599
|
if only_changed:
|
|
382
600
|
env[DIFF_ONLY_ENV] = "1"
|
|
383
601
|
env[DIFF_BASE_ENV] = diff_base
|
|
602
|
+
if diff_file is not None:
|
|
603
|
+
env[DIFF_FILE_ENV] = os.path.abspath(diff_file)
|
|
604
|
+
if include_uncommitted:
|
|
605
|
+
env[INCLUDE_UNCOMMITTED_ENV] = "1"
|
|
384
606
|
env["PYTHONPATH"] = atoms_dir + os.pathsep + env.get("PYTHONPATH", "")
|
|
607
|
+
|
|
608
|
+
if not _pytest_available(pytest_python):
|
|
609
|
+
# The gate cannot judge anything without pytest. When the diff
|
|
610
|
+
# touches Python test files this is a real gap (exit 2), not a
|
|
611
|
+
# silent pass; statically count the asserts at stake for the
|
|
612
|
+
# evidence so the failure names what could not be judged.
|
|
613
|
+
total = 0
|
|
614
|
+
for target in (pytest_args or []):
|
|
615
|
+
candidate = target if os.path.isabs(target) else os.path.join(root, target)
|
|
616
|
+
if is_python_test_file(candidate) and os.path.exists(candidate):
|
|
617
|
+
try:
|
|
618
|
+
total += len(collect_assertions(candidate))
|
|
619
|
+
except Exception: # noqa: BLE001 - best effort only
|
|
620
|
+
pass
|
|
621
|
+
evidence = {
|
|
622
|
+
"tool": "i6d_assertion_execution",
|
|
623
|
+
"version": VERSION,
|
|
624
|
+
"root": root,
|
|
625
|
+
"pytestExit": None,
|
|
626
|
+
"verdict": "pytest_unavailable",
|
|
627
|
+
"total": total,
|
|
628
|
+
"executed": 0,
|
|
629
|
+
"unexecuted": [],
|
|
630
|
+
"skipped": [],
|
|
631
|
+
"diffBase": diff_base_out,
|
|
632
|
+
"diffSource": diff_source,
|
|
633
|
+
"inScope": total if only_changed else None,
|
|
634
|
+
"outOfScope": [],
|
|
635
|
+
}
|
|
636
|
+
with open(output, "w", encoding="utf-8") as handle:
|
|
637
|
+
json.dump(evidence, handle, indent=2)
|
|
638
|
+
if atom_mode:
|
|
639
|
+
print(json.dumps(_atom_summary(
|
|
640
|
+
ok=False, verdict="pytest_unavailable", proof_file=output,
|
|
641
|
+
evidence=evidence, pytest_exit=None,
|
|
642
|
+
)))
|
|
643
|
+
else:
|
|
644
|
+
print("i6d: pytest_unavailable: "
|
|
645
|
+
f"{pytest_python} cannot import pytest", file=sys.stderr)
|
|
646
|
+
return 2
|
|
647
|
+
|
|
385
648
|
command = [pytest_python, "-m", "pytest", "-p", "i6d_assertion_execution", *pytest_args]
|
|
386
649
|
try:
|
|
387
|
-
|
|
650
|
+
# Atom mode: the workflow executor parses stdout as one JSON line,
|
|
651
|
+
# so pytest's own session output must not leak onto stdout. Capture
|
|
652
|
+
# it; the evidence JSON carries the verdict, and the tail is only
|
|
653
|
+
# surfaced when no evidence was written.
|
|
654
|
+
if atom_mode:
|
|
655
|
+
proc = subprocess.run(command, cwd=root, env=env,
|
|
656
|
+
capture_output=True, text=True)
|
|
657
|
+
else:
|
|
658
|
+
proc = subprocess.run(command, cwd=root, env=env)
|
|
388
659
|
except OSError as exc:
|
|
389
660
|
print(f"i6d: could not run pytest: {exc}", file=sys.stderr)
|
|
390
661
|
return 2
|
|
391
662
|
try:
|
|
392
|
-
with open(
|
|
663
|
+
with open(output, encoding="utf-8") as handle:
|
|
393
664
|
evidence = json.load(handle)
|
|
394
665
|
except (OSError, json.JSONDecodeError) as exc:
|
|
395
|
-
|
|
666
|
+
detail = f"pytest exit {proc.returncode}"
|
|
667
|
+
if atom_mode and getattr(proc, "stdout", None):
|
|
668
|
+
detail += f"; pytest output tail: {(proc.stdout or '')[-500:]}"
|
|
669
|
+
print(f"i6d: no evidence written ({exc}); {detail}", file=sys.stderr)
|
|
396
670
|
return 2
|
|
397
671
|
evidence["pytestExit"] = proc.returncode
|
|
672
|
+
if diff_source is not None:
|
|
673
|
+
evidence["diffSource"] = diff_source
|
|
398
674
|
if evidence["verdict"] == "unexecuted_assertions" and proc.returncode == 0:
|
|
399
675
|
pass # verdict already reflects the gate; pytest itself passed.
|
|
400
676
|
elif proc.returncode != 0 and not evidence["unexecuted"]:
|
|
401
677
|
evidence["verdict"] = "pytest_failed"
|
|
402
|
-
with open(
|
|
678
|
+
with open(output, "w", encoding="utf-8") as handle:
|
|
403
679
|
json.dump(evidence, handle, indent=2)
|
|
680
|
+
if atom_mode:
|
|
681
|
+
ok = evidence["verdict"] == "pass"
|
|
682
|
+
print(json.dumps(_atom_summary(
|
|
683
|
+
ok=ok, verdict=evidence["verdict"], proof_file=output,
|
|
684
|
+
evidence=evidence, pytest_exit=proc.returncode,
|
|
685
|
+
)))
|
|
686
|
+
return 0 if ok else 1
|
|
404
687
|
summary = (
|
|
405
688
|
f"i6d: {evidence['verdict']} "
|
|
406
689
|
f"({evidence['executed']}/{evidence['total']} assertions executed, "
|
|
@@ -422,27 +705,52 @@ def main(argv=None):
|
|
|
422
705
|
parser = argparse.ArgumentParser(
|
|
423
706
|
description="I6d assertion-execution gate: prove every assert in "
|
|
424
707
|
"the pytest run really executed (slice 2 adds --only-changed to "
|
|
425
|
-
"judge only asserts added/changed vs a diff base, default main
|
|
708
|
+
"judge only asserts added/changed vs a diff base, default main; "
|
|
709
|
+
"slice 3 adds --diff-file to scope from a diff text, and "
|
|
710
|
+
"--change-id atom mode for the workflow executor)."
|
|
426
711
|
)
|
|
427
712
|
parser.add_argument("--root", default=os.getcwd(), help="project root to run pytest in")
|
|
428
|
-
parser.add_argument("--
|
|
429
|
-
help="
|
|
713
|
+
parser.add_argument("--project-root", default=None,
|
|
714
|
+
help="alias for --root (workflow executor passes this)")
|
|
715
|
+
parser.add_argument("--output", default=None,
|
|
716
|
+
help="where to write the evidence JSON "
|
|
717
|
+
"(default: specs/changes/<id>/tdd/i6d-0.json in atom mode, "
|
|
718
|
+
".evalid-i6d-evidence.json otherwise)")
|
|
430
719
|
parser.add_argument("--pytest-python", default=sys.executable,
|
|
431
720
|
help="python binary to run pytest with")
|
|
432
721
|
parser.add_argument("--only-changed", action="store_true",
|
|
433
722
|
help="only judge asserts in the diff vs --diff-base")
|
|
434
723
|
parser.add_argument("--diff-base", default="main",
|
|
435
724
|
help="diff base ref (merge-base of HEAD and this ref; default main)")
|
|
725
|
+
parser.add_argument("--diff-file", default=None,
|
|
726
|
+
help="scope from a unified diff file instead of a git range")
|
|
727
|
+
parser.add_argument("--include-uncommitted", action="store_true",
|
|
728
|
+
help="union working-tree + staged changes vs HEAD into the scope")
|
|
729
|
+
parser.add_argument("--change-id", default=None,
|
|
730
|
+
help="atom mode: write evidence to specs/changes/<id>/tdd/i6d-0.json "
|
|
731
|
+
"and print a single JSON summary line")
|
|
436
732
|
parser.add_argument("pytest_args", nargs="*", help="arguments passed to pytest (after --)")
|
|
437
733
|
args = parser.parse_args(argv)
|
|
734
|
+
root = args.project_root or args.root
|
|
735
|
+
if args.output:
|
|
736
|
+
output = args.output
|
|
737
|
+
elif args.change_id:
|
|
738
|
+
output = os.path.join(
|
|
739
|
+
root, "specs", "changes", args.change_id, "tdd", "i6d-0.json")
|
|
740
|
+
else:
|
|
741
|
+
output = ".evalid-i6d-evidence.json"
|
|
438
742
|
# argparse keeps a literal "--" out of pytest_args already.
|
|
439
743
|
return run_gate(
|
|
440
|
-
root=
|
|
441
|
-
output=
|
|
744
|
+
root=root,
|
|
745
|
+
output=output,
|
|
442
746
|
pytest_python=args.pytest_python,
|
|
443
747
|
pytest_args=args.pytest_args,
|
|
444
748
|
only_changed=args.only_changed,
|
|
445
749
|
diff_base=args.diff_base,
|
|
750
|
+
diff_file=args.diff_file,
|
|
751
|
+
include_uncommitted=args.include_uncommitted,
|
|
752
|
+
change_id=args.change_id,
|
|
753
|
+
project_root=args.project_root,
|
|
446
754
|
)
|
|
447
755
|
|
|
448
756
|
|