@surlymochan/evalid 0.5.19 → 0.5.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/RELEASE_NOTES.md +14 -0
- package/bin/xflow.js +27 -3
- package/docs/evidence/m4-slice4-auto-resume.md +96 -0
- package/docs/launch-dossier.md +1 -1
- package/package.json +1 -1
- package/src/core/lite-track-auto.js +126 -0
- package/src/core/workflow-executor.js +12 -3
- package/xflow/atoms/lite_track.py +46 -0
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,18 @@
|
|
|
3
3
|
All notable changes to this project are documented here. Detailed notes live in
|
|
4
4
|
[RELEASE_NOTES.md](RELEASE_NOTES.md).
|
|
5
5
|
|
|
6
|
+
## [0.5.20] - 2026-11-10
|
|
7
|
+
|
|
8
|
+
### Added
|
|
9
|
+
- **M4 slice 3: workflow run auto-selects lite (#102)**: `xflow workflow run` with no explicit flag runs the slice-1 classifier first and picks the track itself — small diff → lite (spec ritual skipped, seal + replay only), big diff → full, explicit `--lite` / `--full` always wins (both flags together resolve to full), classifier failure falls back to full and records the failure, never silently lite. The decision rides in the workflow events as `lite_track_decision` (`{source, track, reason, files, added, removed, total, base, force_full}`, source `auto` / `explicit-lite` / `explicit-full` / `auto-fallback-full`), next to slice 1's `lite_fast_track` field; the CLI prints `Lite fast track: <source> → <track>`. Test evidence: 19 new tests in `test/lite-track-auto.test.js` (resolver, flag priority, executor event recording with mock atoms, CLI dry-run wiring) RED→GREEN; PR CI runs verify 42702996913 (3m31s) and ledger 42702997046 (8s) both pass; merged main CI run 42703634292 success.
|
|
10
|
+
- **M4 slice 4: classifier ignores workflow-written change artifacts (#104)**: the auto end-to-end acceptance demo on main HEAD caught a real bug — run 1 auto-decided lite and sealed the red proof, but the resume flipped to full because run 1 had rewritten the tracked `specs/changes/<id>/status.json` itself (+268/−2 lines) and the force-full rule fired on the `specs/` path. `classify_repo` in `xflow/atoms/lite_track.py` now excludes workflow-generated artifacts under `specs/changes/<id>/` (`status.json`, `verify_proof.json`, `spec_delta_review.json/.md`, `tdd/`, `cr/`) from the measured stats before the verdict; human-authored spec files (`proposal.md` / `plan.md` / `tasks.md`) still force full. Test evidence: 4 new classifier unit tests in `test/lite-track.test.js` (status.json churn, generated proof artifacts, proposal.md still forces full, committed status.json churn) 4/4 fail without the fix and 4/4 pass with it; new end-to-end regression test `test/lite-track-auto-acceptance.test.js` (real small fix, no flag, red → fix → resume stays lite with `source: auto, track: lite` in both runs' events, seal + clean replay with binding hash; 400-line control stays auto-full across resume) fails on main HEAD before the fix and passes after it; full `npm run verify` exit 0 — 61 test files, 560 tests, 559 pass, 0 fail (1 skipped); PR CI runs verify 42722307306 (3m31s) and ledger 42722307186 (8s) both pass; merged main CI run 42722893282 success.
|
|
11
|
+
|
|
12
|
+
Wall clock, stated plainly as preliminary (docs/evidence/m4-slice4-auto-resume.md, n=1 per track, same auto demo flow, Linux VM / node v24.22.1 / python 3.12.3): auto demo lite 4,995 ms vs full 6,363 ms (ratio 0.785) isolated, and lite 8,239 ms vs full 7,124 ms (ratio 1.157) inside `npm run verify`; the slice-2 demo re-measured 1.291 in the same suite run. Same demo, same machine, ratios 0.785–1.291 minutes apart — n=1 demo timings are dominated by per-process startup and machine load (the suite runs test files in parallel), not by the spec ritual. The formal ≤1.3× verdict for issue #5 P1-2 waits for the bench full-matrix numbers; P1-2 stays unticked, and this release makes no ≤1.3× claim.
|
|
13
|
+
|
|
14
|
+
Milestone M4 "lite fast track" — **not yet accepted**: slices 1–4 are merged checkpoints; the auto end-to-end acceptance is now demonstrated (real small change through auto-lite across the red→green resume, real large change judged auto-full, seal + clean replay), but the wall-clock verdict still awaits bench numbers. This release makes no M4 acceptance claim.
|
|
15
|
+
|
|
16
|
+
Acceptance debt, stated plainly: the bench-number acceptance for P1-2 / P1-3 / P2-1 / P2-3 still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is at 82/135 with the 12th relaunch in flight. No bench numbers are claimed in this release.
|
|
17
|
+
|
|
6
18
|
## [0.5.19] - 2026-11-10
|
|
7
19
|
|
|
8
20
|
### Added
|
package/RELEASE_NOTES.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# Release Notes
|
|
2
2
|
|
|
3
|
+
## 0.5.20 - M4 lite fast track, slices 3–4 (milestone M4 not yet accepted)
|
|
4
|
+
|
|
5
|
+
Patch release. Two merged PRs since 0.5.19, both verified end to end in CI and locally.
|
|
6
|
+
|
|
7
|
+
**M4 slice 3: workflow run auto-selects lite (#102).** `xflow workflow run` with no explicit flag runs the classifier first and picks the track itself — small diff → lite, big diff → full, explicit `--lite` / `--full` always wins, classifier failure falls back to full and records it. The decision is written into the workflow events as `lite_track_decision`, next to `lite_fast_track`. Test evidence: 19 new tests in `test/lite-track-auto.test.js` RED→GREEN; PR CI runs verify 42702996913 and ledger 42702997046 both pass; merged main CI run 42703634292 success.
|
|
8
|
+
|
|
9
|
+
**M4 slice 4: classifier ignores workflow-written change artifacts (#104).** The auto end-to-end acceptance demo on main HEAD caught a real bug: run 1 auto-decided lite and sealed the red proof, but the resume flipped to full because run 1 had rewritten the tracked `specs/changes/<id>/status.json` itself (+268/−2 lines) and the force-full rule fired on the `specs/` path. `classify_repo` now excludes workflow-generated artifacts under `specs/changes/<id>/` (`status.json`, `verify_proof.json`, `spec_delta_review.*`, `tdd/`, `cr/`) from the measured stats; human spec files still force full. Test evidence: 4 new classifier unit tests RED→GREEN (4/4 fail without the fix); new end-to-end regression test `test/lite-track-auto-acceptance.test.js` (real small fix, no flag, resume stays lite; 400-line control stays auto-full) fails before the fix, passes after; full `npm run verify` exit 0 — 61 test files, 560 tests, 559 pass, 0 fail; PR CI runs verify 42722307306 and ledger 42722307186 both pass; merged main CI run 42722893282 success.
|
|
10
|
+
|
|
11
|
+
Wall clock, stated plainly as preliminary (docs/evidence/m4-slice4-auto-resume.md, n=1 per track, same auto demo flow, Linux VM / node v24.22.1 / python 3.12.3): auto demo lite 4,995 ms vs full 6,363 ms (ratio 0.785) isolated, lite 8,239 ms vs full 7,124 ms (ratio 1.157) inside `npm run verify`; slice-2 demo re-measured 1.291 in the same suite run. Same demo, same machine, ratios 0.785–1.291 — n=1 demo timings are dominated by startup and machine load, not by the spec ritual. The formal ≤1.3× verdict for issue #5 P1-2 waits for the bench full-matrix numbers; P1-2 stays unticked, and this release makes no ≤1.3× claim.
|
|
12
|
+
|
|
13
|
+
**Milestone M4 "lite fast track" — not yet accepted.** Slices 1–4 are merged checkpoints; the auto end-to-end acceptance is now demonstrated (real small change through auto-lite across the red→green resume, real large change judged auto-full, seal + clean replay), but the wall-clock verdict still awaits bench numbers.
|
|
14
|
+
|
|
15
|
+
Acceptance debt, stated plainly: the bench-number acceptance for P1-2, P1-3, P2-1 and P2-3 — the wall-clock/token/resolved lines from issue #5 — still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), at 82/135 with the 12th relaunch in flight. No bench numbers are claimed in this release.
|
|
16
|
+
|
|
3
17
|
## 0.5.19 - M4 lite fast track, slices 1–2 (milestone M4 not yet accepted)
|
|
4
18
|
|
|
5
19
|
Patch release. Two merged PRs since 0.5.18, both verified end to end in CI and locally.
|
package/bin/xflow.js
CHANGED
|
@@ -5478,6 +5478,21 @@ async function handleWorkflow(sub, args) {
|
|
|
5478
5478
|
}
|
|
5479
5479
|
}
|
|
5480
5480
|
ensureChangeId(changeId, 'workflow run', projectRoot);
|
|
5481
|
+
// M4 slice 3 (evalid #101): auto-select the lite track. With no
|
|
5482
|
+
// explicit flag the classifier decides (small diff -> lite, big diff
|
|
5483
|
+
// -> full, classifier failure -> full); --lite / --full always win.
|
|
5484
|
+
// The decision is attached to the workflow input so the executor
|
|
5485
|
+
// records it in the workflow_started / workflow_completed events.
|
|
5486
|
+
const { decideLiteTrack, resolveLiteTrackAuto } = await import('../src/core/lite-track-auto.js');
|
|
5487
|
+
const autoDecision = (parsedArgs.lite === true || parsedArgs.full === true)
|
|
5488
|
+
? null
|
|
5489
|
+
: resolveLiteTrackAuto({ projectRoot });
|
|
5490
|
+
const { lite: effectiveLite, decision: liteDecision } = decideLiteTrack({
|
|
5491
|
+
lite: parsedArgs.lite, full: parsedArgs.full, autoDecision,
|
|
5492
|
+
});
|
|
5493
|
+
if (effectiveLite) input.lite = true;
|
|
5494
|
+
input.lite_track_decision = liteDecision;
|
|
5495
|
+
console.log(`Lite fast track: ${liteDecision.source} → ${liteDecision.track} (${liteDecision.reason})`);
|
|
5481
5496
|
const { load, dryRun } = await import('../src/core/workflow-loader.js');
|
|
5482
5497
|
if (isDryRun) {
|
|
5483
5498
|
dryRun(yamlPath);
|
|
@@ -6694,9 +6709,18 @@ Commands:
|
|
|
6694
6709
|
Options:
|
|
6695
6710
|
--lite M4 lite fast track: skip spec-ritual steps
|
|
6696
6711
|
(proposal-consistency-check phase, J4a/J4b atoms)
|
|
6697
|
-
and keep seal + replay only.
|
|
6698
|
-
|
|
6699
|
-
|
|
6712
|
+
and keep seal + replay only. Always wins over the
|
|
6713
|
+
auto verdict (see below).
|
|
6714
|
+
--full M4 lite fast track: force the full track, skipping
|
|
6715
|
+
the auto verdict. Wins over --lite when both are
|
|
6716
|
+
given.
|
|
6717
|
+
With neither --lite nor --full, the run classifies
|
|
6718
|
+
the working-tree diff first
|
|
6719
|
+
(python3 xflow/atoms/lite_track.py --repo .)
|
|
6720
|
+
and auto-selects lite for a small diff, full for a
|
|
6721
|
+
big one; a classifier failure falls back to full.
|
|
6722
|
+
The decision is printed and recorded in the run's
|
|
6723
|
+
workflow events as lite_track_decision.
|
|
6700
6724
|
help, -h, --help Show this help message
|
|
6701
6725
|
`);
|
|
6702
6726
|
}
|
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
# M4 slice 4: auto-lite resume flips to full on workflow-written status.json
|
|
2
|
+
|
|
3
|
+
Date: 2026-11-10 (Asia/Singapore). Source of truth for the demo itself is
|
|
4
|
+
`test/lite-track-auto-acceptance.test.js` (re-run it to reproduce).
|
|
5
|
+
Parent issue: #97; slice issue: #103; PR: slice 4 (this change).
|
|
6
|
+
|
|
7
|
+
## Status: preliminary wall clock — NOT the ≤1.3× conclusion
|
|
8
|
+
|
|
9
|
+
The wall-clock numbers below are local n=1 measurements of the acceptance
|
|
10
|
+
demo harness. The formal wall-clock verdict for issue #5 P1-2 (small tasks
|
|
11
|
+
≤1.3× baseline) waits for the bench full-matrix numbers; P1-2 stays
|
|
12
|
+
unticked and no ≤1.3× claim is made here.
|
|
13
|
+
|
|
14
|
+
## The bug (found by the auto acceptance demo on main HEAD 4668b6ab)
|
|
15
|
+
|
|
16
|
+
A real small fix through `xflow workflow run` with **no flag**: run 1
|
|
17
|
+
(pre-fix, clean tree) auto-decides lite, seals the red proof, stops at
|
|
18
|
+
tdd-green. The fix is applied, run 2 (resume) re-runs the classifier —
|
|
19
|
+
and flips to **full**:
|
|
20
|
+
|
|
21
|
+
| Run | `lite_track_decision` (from the workflow events) | `lite_fast_track` |
|
|
22
|
+
| --- | --- | --- |
|
|
23
|
+
| Run 1, pre-fix | `{source: auto, track: lite, reason: 'empty diff: nothing to ritualise', files: 0, total: 0}` | `true` |
|
|
24
|
+
| Run 2, resume (pre-fix code) | `{source: auto, track: full, reason: 'touches specs/ paths (specs/changes/chg-m4-auto-demo/status.json): spec/workflow drift needs the full track', files: 3, total: 281}` | `false` |
|
|
25
|
+
|
|
26
|
+
Root cause: run 1 rewrites the tracked
|
|
27
|
+
`specs/changes/<id>/status.json` itself (stage transitions +
|
|
28
|
+
`workflow_integrity` block, +268/−2 lines in the demo; the worktree diff
|
|
29
|
+
at resume time is 3 files, 277 insertions / 13 deletions). The classifier
|
|
30
|
+
measures the worktree diff against the merge-base, sees a `specs/` path,
|
|
31
|
+
and the force-full rule fires. The actual user change is still the same
|
|
32
|
+
2-file, 20-line fix. The two halves of one change then run on different
|
|
33
|
+
tracks: the resume re-executes the spec ritual (J4a/J4b) that run 1
|
|
34
|
+
skipped.
|
|
35
|
+
|
|
36
|
+
Scope note: evalid's own repo gitignores `specs/changes/`, so dogfooding
|
|
37
|
+
does not hit this — but any repo that tracks change state does, and the
|
|
38
|
+
repo's own acceptance demo (the M4 bar) fails on main HEAD. The defect is
|
|
39
|
+
in the shipped classifier, not the demo, so the fix is in the classifier.
|
|
40
|
+
|
|
41
|
+
## The fix
|
|
42
|
+
|
|
43
|
+
`classify_repo` in `xflow/atoms/lite_track.py` now excludes
|
|
44
|
+
workflow-generated artifacts under `specs/changes/<id>/` from the measured
|
|
45
|
+
stats before the verdict: `status.json`, `verify_proof.json`,
|
|
46
|
+
`spec_delta_review.json` / `.md`, and the `tdd/` / `cr/` trees.
|
|
47
|
+
Human-authored spec files (`proposal.md` / `plan.md` / `tasks.md`) still
|
|
48
|
+
force full. No CLI / executor / env / phanes call-surface change.
|
|
49
|
+
|
|
50
|
+
## Numbers (post-fix demo runs, n=1 each, preliminary)
|
|
51
|
+
|
|
52
|
+
| Run | lite (first + resume) | full (first + resume) | lite/full |
|
|
53
|
+
| --- | --- | --- | --- |
|
|
54
|
+
| Auto acceptance test, isolated | 2,446 ms + 2,549 ms = 4,995 ms | 2,854 ms + 3,509 ms = 6,363 ms | 0.785 |
|
|
55
|
+
| Auto acceptance test, inside `npm run verify` | 4,135 ms + 4,104 ms = 8,239 ms | 4,126 ms + 2,998 ms = 7,124 ms | 1.157 |
|
|
56
|
+
|
|
57
|
+
Same demo, same machine, two runs minutes apart: 0.785 vs 1.157. The
|
|
58
|
+
slice-2 demo re-measured inside the same suite run gives 1.291 this time
|
|
59
|
+
(7,655 ms vs 5,928 ms), against its original 0.644 / 0.798. The spread
|
|
60
|
+
itself is the finding: n=1 demo timings are dominated by per-process
|
|
61
|
+
startup and machine load (the suite runs test files in parallel), not by
|
|
62
|
+
the spec ritual. Do not quote any of these ratios as the lite-track
|
|
63
|
+
speedup; the formal ≤1.3× verdict still waits for the bench full-matrix
|
|
64
|
+
numbers.
|
|
65
|
+
|
|
66
|
+
Machine for all runs above: Linux x86_64 VM, AMD EPYC 9D25, 4 vCPU;
|
|
67
|
+
node v24.22.1, python 3.12.3.
|
|
68
|
+
|
|
69
|
+
## What the demo proved (functional, not timing)
|
|
70
|
+
|
|
71
|
+
- Post-fix resume decision: `{source: auto, track: lite, reason:
|
|
72
|
+
'within thresholds (2 files, 20 lines <= 10 files, 300 lines)',
|
|
73
|
+
files: 2, added: 9, removed: 11, total: 20, force_full: []}` — the
|
|
74
|
+
resume stays lite on the same fix that run 1 lite'd.
|
|
75
|
+
- Lite run: `proposal-consistency-check` logged `phase_skipped` (reason
|
|
76
|
+
`lite_track`); `J4a` / `J4b` logged `atom_skipped` (reason
|
|
77
|
+
`lite_track`); the `verify-consistency` gate passed with verdict
|
|
78
|
+
`lite_skipped`; `workflow_completed` recorded `lite_fast_track: true`
|
|
79
|
+
with the auto decision in both `workflow_started` and
|
|
80
|
+
`workflow_completed`.
|
|
81
|
+
- Seal + clean replay: red proof `passed: false`; green proof
|
|
82
|
+
`passed: true` with binding hash `8f5f78b44be55c3d` and snapshot scope
|
|
83
|
+
(`lite_track.py`, `test/demo.test.mjs`); I6b validated both; I6d, I6c,
|
|
84
|
+
J1 and J6 all passed.
|
|
85
|
+
- Full control (400-line file staged, no flag): first run auto-full
|
|
86
|
+
(`400 lines > 300 max_lines`, 1 file), resume auto-full (420 lines,
|
|
87
|
+
3 files); J4a / J4d / J4b all executed and passed; no `lite_track`
|
|
88
|
+
skip anywhere; `workflow_completed` recorded `lite_fast_track: false`.
|
|
89
|
+
|
|
90
|
+
## Reproduce
|
|
91
|
+
|
|
92
|
+
```sh
|
|
93
|
+
node --test test/lite-track-auto-acceptance.test.js # prints [m4-auto-*] lines
|
|
94
|
+
node --test test/lite-track.test.js # classifier unit tests
|
|
95
|
+
npm run verify # full suite
|
|
96
|
+
```
|
package/docs/launch-dossier.md
CHANGED
|
@@ -117,7 +117,7 @@ Evidence:
|
|
|
117
117
|
- Adoption records: 1; status: pass
|
|
118
118
|
- Goal audit: pass; goal: Keep evalid credible as an open-source agent delivery runtime: repo-owned workflow contracts, executable verification gates, durable project context, and honest benchmark evidence.
|
|
119
119
|
- Package evidence: missing; source: registry_json
|
|
120
|
-
- Package name/version: @surlymochan/evalid@0.5.
|
|
120
|
+
- Package name/version: @surlymochan/evalid@0.5.20
|
|
121
121
|
|
|
122
122
|
## Passing Evidence
|
|
123
123
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@surlymochan/evalid",
|
|
3
|
-
"version": "0.5.
|
|
3
|
+
"version": "0.5.20",
|
|
4
4
|
"description": "Claimed != verified: a runtime verification layer for coding agents. It seals each test run and replays it from a clean snapshot, with a zero-judge gate pipeline (spec clarification, code review, TDD proof) and the xflow CLI.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai-agents",
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* M4 lite fast track, slice 3 (evalid #101, parent #97): auto-select the
|
|
3
|
+
* track for `xflow workflow run`.
|
|
4
|
+
*
|
|
5
|
+
* With no explicit flag the run classifies the change first (slice 1's
|
|
6
|
+
* `xflow/atoms/lite_track.py`, working-tree diff against the merge-base,
|
|
7
|
+
* `--uncommitted`): a small diff walks the lite track, a big diff stays
|
|
8
|
+
* full. Explicit `--lite` / `--full` always wins; both flags together
|
|
9
|
+
* resolve to full (conservative). Any classifier failure falls back to
|
|
10
|
+
* full and records the failure -- a run must never silently go lite.
|
|
11
|
+
*
|
|
12
|
+
* The returned decision object is JSON-serialisable on purpose: the CLI
|
|
13
|
+
* attaches it to the workflow input as `lite_track_decision` and the
|
|
14
|
+
* executor records it in the `workflow_started` / `workflow_completed`
|
|
15
|
+
* events, next to slice 1's `lite_fast_track` field.
|
|
16
|
+
*/
|
|
17
|
+
import { spawnSync } from 'node:child_process';
|
|
18
|
+
import { dirname, resolve } from 'node:path';
|
|
19
|
+
import { fileURLToPath } from 'node:url';
|
|
20
|
+
|
|
21
|
+
const __dirname = dirname(fileURLToPath(import.meta.url));
|
|
22
|
+
export const DEFAULT_CLASSIFIER_PATH = resolve(__dirname, '..', '..', 'xflow', 'atoms', 'lite_track.py');
|
|
23
|
+
const DEFAULT_TIMEOUT_MS = 60_000;
|
|
24
|
+
|
|
25
|
+
/**
|
|
26
|
+
* Run the classifier against `projectRoot` and return the auto decision.
|
|
27
|
+
*
|
|
28
|
+
* `runClassifier` is injectable (defaults to `spawnSync`) so tests can
|
|
29
|
+
* stub the classifier invocation without touching git or python.
|
|
30
|
+
*/
|
|
31
|
+
export function resolveLiteTrackAuto({
|
|
32
|
+
projectRoot,
|
|
33
|
+
classifierPath = DEFAULT_CLASSIFIER_PATH,
|
|
34
|
+
python = process.env.PYTHON || 'python3',
|
|
35
|
+
runClassifier = spawnSync,
|
|
36
|
+
timeoutMs = DEFAULT_TIMEOUT_MS,
|
|
37
|
+
} = {}) {
|
|
38
|
+
const fallback = (detail) => ({
|
|
39
|
+
source: 'auto-fallback-full',
|
|
40
|
+
track: 'full',
|
|
41
|
+
reason: `classifier failed (${detail}); falling back to full track`,
|
|
42
|
+
files: null,
|
|
43
|
+
added: null,
|
|
44
|
+
removed: null,
|
|
45
|
+
total: null,
|
|
46
|
+
base: null,
|
|
47
|
+
force_full: [],
|
|
48
|
+
});
|
|
49
|
+
|
|
50
|
+
let result;
|
|
51
|
+
try {
|
|
52
|
+
result = runClassifier(
|
|
53
|
+
python,
|
|
54
|
+
[classifierPath, '--repo', projectRoot, '--uncommitted', '--json'],
|
|
55
|
+
{ encoding: 'utf8', timeout: timeoutMs },
|
|
56
|
+
);
|
|
57
|
+
} catch (err) {
|
|
58
|
+
return fallback(`spawn error: ${err?.message || String(err)}`);
|
|
59
|
+
}
|
|
60
|
+
if (!result || result.status !== 0) {
|
|
61
|
+
const detail = String(
|
|
62
|
+
result?.stderr?.trim() || result?.error?.message || `exit ${result?.status ?? 'unknown'}`,
|
|
63
|
+
).slice(0, 200);
|
|
64
|
+
return fallback(detail);
|
|
65
|
+
}
|
|
66
|
+
let verdict;
|
|
67
|
+
try {
|
|
68
|
+
verdict = JSON.parse(result.stdout);
|
|
69
|
+
} catch {
|
|
70
|
+
return fallback('unparseable classifier output');
|
|
71
|
+
}
|
|
72
|
+
if (verdict?.track !== 'lite' && verdict?.track !== 'full') {
|
|
73
|
+
return fallback(`unexpected verdict: ${JSON.stringify(verdict).slice(0, 200)}`);
|
|
74
|
+
}
|
|
75
|
+
return {
|
|
76
|
+
source: 'auto',
|
|
77
|
+
track: verdict.track,
|
|
78
|
+
reason: verdict.reason || '',
|
|
79
|
+
files: verdict.files ?? null,
|
|
80
|
+
added: verdict.added ?? null,
|
|
81
|
+
removed: verdict.removed ?? null,
|
|
82
|
+
total: verdict.total ?? null,
|
|
83
|
+
base: verdict.base ?? null,
|
|
84
|
+
force_full: verdict.force_full || [],
|
|
85
|
+
};
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
/**
|
|
89
|
+
* Resolve the effective track from explicit flags and (when neither flag
|
|
90
|
+
* is given) the auto decision. Returns `{ lite, decision }`.
|
|
91
|
+
*/
|
|
92
|
+
export function decideLiteTrack({ lite, full, autoDecision }) {
|
|
93
|
+
if (lite === true && full === true) {
|
|
94
|
+
return {
|
|
95
|
+
lite: false,
|
|
96
|
+
decision: {
|
|
97
|
+
source: 'explicit-full',
|
|
98
|
+
track: 'full',
|
|
99
|
+
reason: 'both --lite and --full given; --full wins (conservative)',
|
|
100
|
+
},
|
|
101
|
+
};
|
|
102
|
+
}
|
|
103
|
+
if (lite === true) {
|
|
104
|
+
return {
|
|
105
|
+
lite: true,
|
|
106
|
+
decision: { source: 'explicit-lite', track: 'lite', reason: 'explicit --lite flag' },
|
|
107
|
+
};
|
|
108
|
+
}
|
|
109
|
+
if (full === true) {
|
|
110
|
+
return {
|
|
111
|
+
lite: false,
|
|
112
|
+
decision: { source: 'explicit-full', track: 'full', reason: 'explicit --full flag' },
|
|
113
|
+
};
|
|
114
|
+
}
|
|
115
|
+
if (!autoDecision) {
|
|
116
|
+
return {
|
|
117
|
+
lite: false,
|
|
118
|
+
decision: {
|
|
119
|
+
source: 'auto-fallback-full',
|
|
120
|
+
track: 'full',
|
|
121
|
+
reason: 'no classifier verdict available; falling back to full track',
|
|
122
|
+
},
|
|
123
|
+
};
|
|
124
|
+
}
|
|
125
|
+
return { lite: autoDecision.track === 'lite', decision: autoDecision };
|
|
126
|
+
}
|
|
@@ -75,9 +75,13 @@ export class WorkflowExecutor {
|
|
|
75
75
|
|
|
76
76
|
// ─── Main execution loop ─────────────────────────────────────────────────
|
|
77
77
|
|
|
78
|
-
// M4 lite fast track:
|
|
79
|
-
//
|
|
80
|
-
//
|
|
78
|
+
// M4 lite fast track: `input.lite === true` engages it. Slice 1 set this
|
|
79
|
+
// only from the explicit `--lite` flag; slice 3 (evalid #101) lets the
|
|
80
|
+
// CLI resolve it first -- `decideLiteTrack` in lite-track-auto.js runs
|
|
81
|
+
// the classifier when no explicit flag is given and sets `input.lite`
|
|
82
|
+
// when the verdict is lite, so the executor itself never decides. The
|
|
83
|
+
// decision itself arrives as `input.lite_track_decision` and is recorded
|
|
84
|
+
// in the workflow_started / workflow_completed events.
|
|
81
85
|
isLiteFastTrack() {
|
|
82
86
|
return this.input?.lite === true;
|
|
83
87
|
}
|
|
@@ -137,6 +141,10 @@ export class WorkflowExecutor {
|
|
|
137
141
|
workflow: this.workflow.name,
|
|
138
142
|
track: this.workflow.track,
|
|
139
143
|
lite_fast_track: this.isLiteFastTrack(),
|
|
144
|
+
// M4 slice 3: how the track was chosen (auto / explicit-lite /
|
|
145
|
+
// explicit-full / auto-fallback-full); null when the run was started
|
|
146
|
+
// without the CLI decision step (e.g. direct executor use).
|
|
147
|
+
lite_track_decision: this.input?.lite_track_decision ?? null,
|
|
140
148
|
workflow_integrity: this.workflowManifest,
|
|
141
149
|
max_parallel_agents: this.maxParallelAgents,
|
|
142
150
|
parallel_policy: this.concurrencyPolicy.mode,
|
|
@@ -351,6 +359,7 @@ export class WorkflowExecutor {
|
|
|
351
359
|
workflow: this.workflow.name,
|
|
352
360
|
track: this.workflow.track,
|
|
353
361
|
lite_fast_track: this.isLiteFastTrack(),
|
|
362
|
+
lite_track_decision: this.input?.lite_track_decision ?? null,
|
|
354
363
|
workflow_integrity: this.workflowManifest,
|
|
355
364
|
change_id: this.changeId || null,
|
|
356
365
|
mission_id: this.runtime.missionId || null,
|
|
@@ -21,6 +21,13 @@ Thresholds (initial values, see evalid #97 for the calibration):
|
|
|
21
21
|
flips the verdict to ``full`` -- the spec ritual exists precisely to catch
|
|
22
22
|
drift there, so a small diff touching spec/workflow definitions must not
|
|
23
23
|
skip it. An empty diff classifies ``lite``.
|
|
24
|
+
* Workflow-generated artifacts under ``specs/changes/<id>/`` do not count
|
|
25
|
+
(slice 4, evalid #103): the workflow itself rewrites ``status.json``
|
|
26
|
+
(+268 lines in the M4 demo) and writes proof artifacts (``tdd/``,
|
|
27
|
+
``cr/``, ``verify_proof.json``, ``spec_delta_review.*``) during a run,
|
|
28
|
+
so a resume re-measuring the worktree would otherwise flip an auto-lite
|
|
29
|
+
change to full on its own state. Human-authored spec files
|
|
30
|
+
(``proposal.md`` / ``plan.md`` / ``tasks.md``) still force full.
|
|
24
31
|
|
|
25
32
|
CLI (always prints one JSON object; exit 0, exit 2 on git/usage errors)::
|
|
26
33
|
|
|
@@ -53,6 +60,36 @@ DEFAULT_MAX_LINES = 300
|
|
|
53
60
|
FORCE_FULL_PREFIXES = ("specs/", "workflows/")
|
|
54
61
|
VERSION = 1
|
|
55
62
|
|
|
63
|
+
# Files the workflow itself writes under specs/changes/<id>/ -- machine
|
|
64
|
+
# state and proof artifacts, not spec prose (slice 4, evalid #103).
|
|
65
|
+
GENERATED_CHANGE_FILES = (
|
|
66
|
+
"status.json",
|
|
67
|
+
"verify_proof.json",
|
|
68
|
+
"spec_delta_review.json",
|
|
69
|
+
"spec_delta_review.md",
|
|
70
|
+
)
|
|
71
|
+
GENERATED_CHANGE_DIRS = ("tdd/", "cr/")
|
|
72
|
+
|
|
73
|
+
|
|
74
|
+
def _is_generated_change_artifact(path):
|
|
75
|
+
"""True when ``path`` is workflow-generated state under a change dir.
|
|
76
|
+
|
|
77
|
+
``specs/changes/<id>/status.json`` is rewritten by the workflow on
|
|
78
|
+
every run (stage transitions, integrity block); ``tdd/`` / ``cr/``
|
|
79
|
+
hold sealed proofs and review output, ``verify_proof.json`` and
|
|
80
|
+
``spec_delta_review.*`` likewise. None of these is a human spec edit,
|
|
81
|
+
so none of them may force the full track or inflate the measured
|
|
82
|
+
diff. Anything else under ``specs/changes/<id>/`` -- in particular
|
|
83
|
+
``proposal.md`` / ``plan.md`` / ``tasks.md`` -- still counts.
|
|
84
|
+
"""
|
|
85
|
+
parts = path.split("/")
|
|
86
|
+
if len(parts) < 4 or parts[0] != "specs" or parts[1] != "changes":
|
|
87
|
+
return False
|
|
88
|
+
rest = parts[3:]
|
|
89
|
+
if len(rest) == 1 and rest[0] in GENERATED_CHANGE_FILES:
|
|
90
|
+
return True
|
|
91
|
+
return bool(rest) and (rest[0] + "/") in GENERATED_CHANGE_DIRS
|
|
92
|
+
|
|
56
93
|
|
|
57
94
|
def _git(repo, *argv):
|
|
58
95
|
proc = subprocess.run(
|
|
@@ -172,6 +209,15 @@ def classify_repo(repo, base="main", max_files=DEFAULT_MAX_FILES,
|
|
|
172
209
|
stats = _numstat_worktree(repo, base_sha)
|
|
173
210
|
else:
|
|
174
211
|
stats = _numstat(repo, base_sha)
|
|
212
|
+
# Workflow-generated artifacts under specs/changes/<id>/ are the
|
|
213
|
+
# workflow's own state, not the change under review (slice 4): drop
|
|
214
|
+
# them before the verdict so a resume cannot flip to full on its own
|
|
215
|
+
# status.json rewrite, and so their lines cannot inflate the budget.
|
|
216
|
+
stats = {
|
|
217
|
+
path: counts
|
|
218
|
+
for path, counts in stats.items()
|
|
219
|
+
if not _is_generated_change_artifact(path)
|
|
220
|
+
}
|
|
175
221
|
files = sorted(stats)
|
|
176
222
|
added = sum(a for a, _ in stats.values())
|
|
177
223
|
removed = sum(r for _, r in stats.values())
|