bmad-module-ultracode-goal 2.1.0 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +2 -1
- package/docs/_internal/STABILITY.md +5 -3
- package/docs/architecture.md +2 -1
- package/docs/gate-model.md +24 -0
- package/docs/how-it-works.md +2 -2
- package/docs/operating-tips.md +3 -1
- package/docs/troubleshooting.md +4 -0
- package/package.json +1 -1
- package/skills/ultracode-goal/customize.toml +1 -1
- package/skills/ultracode-goal/references/execute.md +12 -6
- package/skills/ultracode-goal/references/gate.md +19 -5
- package/skills/ultracode-goal/references/preflight.md +1 -1
- package/skills/ultracode-goal/scripts/gate_eval.py +166 -2
- package/skills/ultracode-goal/scripts/mutate_check.py +353 -0
|
@@ -19,7 +19,7 @@
|
|
|
19
19
|
"name": "bmad-module-ultracode-goal",
|
|
20
20
|
"source": "./",
|
|
21
21
|
"description": "Supercharges BMAD with Claude Code Goal Mode, Auto Mode, and Auto Memory: autonomous Epic execution to a machine-checked, TEA-gated Definition-of-Done. Execute Epics. Accumulate Knowledge. Deliver Outcomes.",
|
|
22
|
-
"version": "2.
|
|
22
|
+
"version": "2.2.0",
|
|
23
23
|
"author": {
|
|
24
24
|
"name": "Armel"
|
|
25
25
|
},
|
package/README.md
CHANGED
|
@@ -50,6 +50,7 @@ That verdict is a small, real JSON object; here is the actual output of `gate_ev
|
|
|
50
50
|
"overall_status": "88%",
|
|
51
51
|
"nfr_status": null,
|
|
52
52
|
"review_score": null,
|
|
53
|
+
"epic_level": false,
|
|
53
54
|
"reasons": [
|
|
54
55
|
"gate read from gate-decision.json",
|
|
55
56
|
"gate_status PASS -> advance"
|
|
@@ -104,7 +105,7 @@ UCG's closest peer is [bmad-loop](https://github.com/bmad-code-org/bmad-loop) (f
|
|
|
104
105
|
|
|
105
106
|
You don't have to take the gate's word for it. The deterministic pieces ship with a test suite, and you can run the evaluator on a real artifact yourself.
|
|
106
107
|
|
|
107
|
-
Run the Python suite (
|
|
108
|
+
Run the Python suite (1,700+ tests across the preflight, gate, hook, readiness, memory, mutation-runner, and customization scripts):
|
|
108
109
|
|
|
109
110
|
```bash
|
|
110
111
|
uv run --with pytest==9.1.1 --with pytest-xdist==3.8.0 pytest skills/ultracode-goal/scripts/tests/ -n auto
|
|
@@ -60,8 +60,10 @@ The skill name `ultracode-goal` and its documented invocation phrases ("run an e
|
|
|
60
60
|
|
|
61
61
|
`scripts/gate_eval.py` is the deterministic completion authority. Covered:
|
|
62
62
|
|
|
63
|
-
- **CLI flags**: `--trace-output` (required), `--profile` (`light` | `production`, required), `--story`, `--nfr`, `--test-review` (production only), `--epic-level`, `--rollup` and `--sprint-status` (roll-up only: `--rollup` requires `--epic-level`, `--story <epic id>` and `--sprint-status`, and `--sprint-status` is rejected without `--rollup`; every roll-up refusal resolves to `gate_status: NOT_EVALUATED` -> `escalate`, never to a silent narrower answer).
|
|
63
|
+
- **CLI flags**: `--trace-output` (required), `--profile` (`light` | `production`, required), `--story`, `--nfr`, `--test-review` (production only), `--tests-ran` (per-story only), `--cycle-profile` (per-story only), `--epic-level`, `--rollup` and `--sprint-status` (roll-up only: `--rollup` requires `--epic-level`, `--story <epic id>` and `--sprint-status`, and `--sprint-status` is rejected without `--rollup`; every roll-up refusal resolves to `gate_status: NOT_EVALUATED` -> `escalate`, never to a silent narrower answer).
|
|
64
64
|
- **The production signals are required, not optional, on a per-story gate.** Under `--profile production` an omitted `--nfr` or `--test-review` is a *failing* signal, exactly as a supplied-but-missing path already was. `--epic-level` is the one sanctioned omission: it declares the epic roll-up, where TEA writes no aggregate to AND, and it cannot be combined with either path (doing so is an invocation error, exit 2). Under `--profile light` the flag is a no-op.
|
|
65
|
+
- **The sweep-scope cap** (`--tests-ran`): optional PATH to the story's `.tests-ran-<story_id>` marker, read in both profiles. The marker's `scope=` line is read fail-closed: `scope=full` passes the verdict through; `scope=scoped`, a missing marker, a missing `scope=` line, or an unrecognised value caps `advance`/`defer` at `reloop` (defer moves too, because its route advances the story). It cannot be combined with `--epic-level` (an invocation error, exit 2). **Omitting the flag skips the check**: invocations that predate the flag keep their verdicts, per the breaking-change rule below. The verdict JSON's `sweep_scope` key carries the recognised value or `null`, and is present exactly when the flag was supplied, so the printed shape of pre-flag invocations is unchanged.
|
|
66
|
+
- **The delta cycle profile** (`--cycle-profile`, `full` | `delta`): `delta` declares a mid-loop gate that re-ran only the previously failed assessor dimension(s); its verdict is capped at `reloop` (`advance` and `defer` both move to `reloop`). An unrecognised value is an argparse error (exit 2), and the flag cannot be combined with `--epic-level` (exit 2). Omission is the full behaviour. The `cycle_profile` JSON key is present exactly when the flag was supplied.
|
|
65
67
|
- **Verdict vocabulary**: the `verdict` values `advance` / `defer` / `reloop` / `escalate`, and the `gate_status` values `PASS` / `CONCERNS` / `FAIL` / `WAIVED` / `NOT_EVALUATED`, plus the mapping between them. See the [gate model](../gate-model.md).
|
|
66
68
|
- **`--story` fail-closed resolution**: when `--story` matches no artifact in a trace dir whose reports or gate-decision files are named per story, the result is `NOT_EVALUATED` (so, `escalate`), never a fallback to another story's gate. A trace dir whose candidates are all generically named (`trace.md`, `gate-decision.json`) still resolves unscoped. Which file the resolver picks, and how it decides that a name carries a story id, are internal.
|
|
67
69
|
|
|
@@ -78,7 +80,7 @@ flowchart LR
|
|
|
78
80
|
class ADV,DEF,REL,ESC verdict
|
|
79
81
|
```
|
|
80
82
|
|
|
81
|
-
The printed JSON object's key set (`verdict`, `gate_status`, `p0_status`, `p1_status`, `overall_status`, `nfr_status`, `review_score`, `epic_level`, `reasons`) is the consumable shape; the human-readable `reasons` strings are not contractual wording. `epic_level` is how a consumer tells an `advance` that ANDed both production signals from one that skipped them by declaration: `nfr_status: null` alone cannot say which, and the distinction is not readable from `reasons` precisely because that wording is not contractual.
|
|
83
|
+
The printed JSON object's key set (`verdict`, `gate_status`, `p0_status`, `p1_status`, `overall_status`, `nfr_status`, `review_score`, `epic_level`, `reasons`) is the consumable shape; the human-readable `reasons` strings are not contractual wording. Two keys are conditional by design: `sweep_scope` appears exactly when `--tests-ran` was supplied (its value the recognised `scope=` reading, or `null`) and `cycle_profile` exactly when `--cycle-profile` was; each is absent otherwise, so an invocation that predates the flags prints the same shape it always did. `epic_level` is how a consumer tells an `advance` that ANDed both production signals from one that skipped them by declaration: `nfr_status: null` alone cannot say which, and the distinction is not readable from `reasons` precisely because that wording is not contractual.
|
|
82
84
|
|
|
83
85
|
## @internal: not covered
|
|
84
86
|
|
|
@@ -98,7 +100,7 @@ What counts as breaking, stated so it is not re-litigated per release:
|
|
|
98
100
|
|
|
99
101
|
- Removing or renaming an enumerated CLI flag, subcommand, config key, or JSON key.
|
|
100
102
|
- **Changing the verdict an unchanged, previously-documented invocation returns.** This is the one that bites, and it bit at 2.0.0: `gate_eval.py --profile production` with neither signal flag was the documented epic-level invocation and returned `advance`; it now returns `reloop` unless `--epic-level` is passed. That the old answer was unearned did not make the change non-breaking — a consumer's pipeline still changed behaviour on an upgrade they did not ask for.
|
|
101
|
-
- Adding a key to the
|
|
103
|
+
- Adding a key to the JSON an UNCHANGED invocation prints (the verdict object or the headless envelope), for a consumer validating the shape strictly. A key that appears only when a newly introduced flag is supplied is additive, not breaking: `sweep_scope` and `cycle_profile` are the worked example, present exactly when `--tests-ran` / `--cycle-profile` are passed and absent from every pre-flag invocation's output.
|
|
102
104
|
|
|
103
105
|
What does not:
|
|
104
106
|
|
package/docs/architecture.md
CHANGED
|
@@ -21,7 +21,7 @@ These are the module's non-negotiables. Each exists because the documented mecha
|
|
|
21
21
|
|
|
22
22
|
### 1. Deterministic gate truth
|
|
23
23
|
|
|
24
|
-
`scripts/gate_eval.py` reads TEA's `gate-decision.json` and maps its gate status to a routing verdict (`PASS`/`WAIVED` → advance, `CONCERNS` → defer, `FAIL` → reloop, `NOT_EVALUATED` → escalate). It never re-derives TEA's thresholds and never reads the transcript. The `/goal` evaluator that drives execution can only see what the run surfaces, it cannot open the gate file, so it is structurally incapable of being the completion authority. The script is. In production, two extra signals can only downgrade an `advance`, never lift a lower verdict. See the [gate model](./gate-model.md) for the full mapping diagram, thresholds, and the fail-closed contract.
|
|
24
|
+
`scripts/gate_eval.py` reads TEA's `gate-decision.json` and maps its gate status to a routing verdict (`PASS`/`WAIVED` → advance, `CONCERNS` → defer, `FAIL` → reloop, `NOT_EVALUATED` → escalate). It never re-derives TEA's thresholds and never reads the transcript. The `/goal` evaluator that drives execution can only see what the run surfaces, it cannot open the gate file, so it is structurally incapable of being the completion authority. The script is. In production, two extra signals can only downgrade an `advance`, never lift a lower verdict; in both profiles a `scope=scoped` tests-ran marker caps a story-moving verdict at `reloop`, so a scoped re-loop sweep can never advance a story. See the [gate model](./gate-model.md) for the full mapping diagram, thresholds, and the fail-closed contract.
|
|
25
25
|
|
|
26
26
|
### 2. Hooks as invariants
|
|
27
27
|
|
|
@@ -56,6 +56,7 @@ skills/ultracode-goal/
|
|
|
56
56
|
├── scripts/ # Deterministic truth (run via `uv`)
|
|
57
57
|
│ ├── preflight_check.py # mechanical preflight facts + blocker budget
|
|
58
58
|
│ ├── gate_eval.py # gate status -> verdict (the completion authority)
|
|
59
|
+
│ ├── mutate_check.py # mutation harness: attributable-red proof for the story's guards
|
|
59
60
|
│ ├── gate_trail.py # per-story evidence trail (gate-trail.md) at finalize
|
|
60
61
|
│ ├── formalize_check.py # readiness kernel behind the /ucg-formalize gate
|
|
61
62
|
│ ├── status_render.py # read-side render behind /ucg-status
|
package/docs/gate-model.md
CHANGED
|
@@ -86,6 +86,21 @@ flowchart TD
|
|
|
86
86
|
|
|
87
87
|
Under `--profile light` none of this applies: the trace gate is the whole decision, and `--epic-level` is a no-op there.
|
|
88
88
|
|
|
89
|
+
## The sweep-scope cap
|
|
90
|
+
|
|
91
|
+
Independent of profile, a per-story gate is also handed the story's `.tests-ran-<story_id>` marker via `--tests-ran`. Execute writes a `scope=<full|scoped>` line into that marker beside `baseline=`, naming which sweep form the green run behind this gate was: the full three-axis sweep, or a re-loop iteration's affected-set sweep (changed packages plus their dependents, still cache-defeated, with the resolved package list printed and recorded on a `packages=` line). The script reads the line and applies one rule: **a story only moves forward off a full-scope sweep.** `scope=full` passes the verdict through; `scope=scoped` caps `advance` at `reloop`, and caps `defer` too, because defer's route advances the story after parking its concerns. A missing marker, a marker with no `scope=` line, or an unrecognised value gets the same cap, fail-closed, with a reason naming what was wrong.
|
|
92
|
+
|
|
93
|
+
Two boundaries:
|
|
94
|
+
|
|
95
|
+
- `--epic-level` refuses `--tests-ran` outright (an invocation error, exit 2): the epic roll-up gates no sweep of its own, since every story proved its own scope before reaching `done`.
|
|
96
|
+
- Omitting the flag skips the check. That is deliberate: making omission failing would change the verdict of documented invocations that predate the flag, which is exactly the 2.0.0 lesson in the [stability policy](https://github.com/armelhbobdad/bmad-module-ultracode-goal/blob/main/docs/_internal/STABILITY.md). The skill instructions require the flag on every per-story gate; the script does not retrofit that requirement onto older callers.
|
|
97
|
+
|
|
98
|
+
The verdict JSON carries the recognised value as `sweep_scope` (or `null`), so an advance that proved a full sweep is distinguishable from one that was never asked about its sweep.
|
|
99
|
+
|
|
100
|
+
## The delta cycle profile
|
|
101
|
+
|
|
102
|
+
A mid-loop re-gate may declare `--cycle-profile delta`: only the assessor dimension(s) whose findings produced the previous reloop re-run in full, while the other assessors re-check their prior artifact against the diff since the commit they assessed. The script caps a delta gate's verdict at `reloop` (defer too, for the sweep-cap reason): a delta gate can say "not yet, and here is why", never move the story. Advancing requires the full instrument, every assessor over a `scope=full` sweep, so the oracle's finality is untouched and a wrong guess costs one extra full gate rather than an unearned advance. The fallback is mechanical: any prior finding without a clean dimension attribution means the full gate runs. `--epic-level` refuses the flag (invocation error, exit 2); omitting it is the full behaviour, and the `cycle_profile` JSON key appears exactly when the flag was supplied.
|
|
103
|
+
|
|
89
104
|
## The thresholds
|
|
90
105
|
|
|
91
106
|
The P0/P1/overall percentage thresholds (**P0 = 100%, P1 >= 90%, overall >= 80%**) are decided **upstream by the TEA trace workflow** and written into the gate artifact; `gate_eval.py` reads the resulting `gate_status`, `p0_status`, `p1_status`, and `overall_status` rather than recomputing the percentages. The script's own production AND adds the two coarser signals above (NFR != FAIL, review score >= 80 and recommendation != Block). Do not restate or recompute the TEA percentages elsewhere; they are TEA-owned, and the test-design stage's job is only to assign the P0-P3 priorities honestly so those upstream thresholds key off real priorities.
|
|
@@ -109,10 +124,15 @@ The script prints one JSON object (`evaluate()` in the script):
|
|
|
109
124
|
"overall_status": "...",
|
|
110
125
|
"nfr_status": "...",
|
|
111
126
|
"review_score": 0,
|
|
127
|
+
"epic_level": false,
|
|
128
|
+
"sweep_scope": "full|scoped|null",
|
|
129
|
+
"cycle_profile": "full|delta",
|
|
112
130
|
"reasons": ["..."]
|
|
113
131
|
}
|
|
114
132
|
```
|
|
115
133
|
|
|
134
|
+
`epic_level` is always present. The `sweep_scope` and `cycle_profile` keys are conditional by the same contract: each is present exactly when its flag (`--tests-ran`, `--cycle-profile`) was supplied and absent otherwise, so an invocation that predates the flags prints the same shape it always did.
|
|
135
|
+
|
|
116
136
|
### Example: a clean production advance
|
|
117
137
|
|
|
118
138
|
A story whose slim gate file reads `PASS`, with an NFR audit of `PASS` and a test review scoring 92 with an Approve recommendation, produces:
|
|
@@ -126,6 +146,8 @@ A story whose slim gate file reads `PASS`, with an NFR audit of `PASS` and a tes
|
|
|
126
146
|
"overall_status": "PASS",
|
|
127
147
|
"nfr_status": "PASS",
|
|
128
148
|
"review_score": 92,
|
|
149
|
+
"epic_level": false,
|
|
150
|
+
"sweep_scope": "full",
|
|
129
151
|
"reasons": [
|
|
130
152
|
"gate read from gate-decision.json",
|
|
131
153
|
"gate_status PASS -> advance"
|
|
@@ -146,6 +168,8 @@ The same `PASS` gate, but with a test review scoring 74, downgrades to `reloop`;
|
|
|
146
168
|
"overall_status": "PASS",
|
|
147
169
|
"nfr_status": "PASS",
|
|
148
170
|
"review_score": 74,
|
|
171
|
+
"epic_level": false,
|
|
172
|
+
"sweep_scope": "full",
|
|
149
173
|
"reasons": [
|
|
150
174
|
"gate read from gate-decision.json",
|
|
151
175
|
"gate_status PASS -> advance",
|
package/docs/how-it-works.md
CHANGED
|
@@ -78,11 +78,11 @@ Turn the Epic's acceptance criteria into **executable, red-phase acceptance test
|
|
|
78
78
|
|
|
79
79
|
### Stage 4: Execute
|
|
80
80
|
|
|
81
|
-
Drive each in-scope story from its red-phase tests to a green, committed state. Execution is the **sequential `/goal` spine**; per story, in sprint order: set the current story (so the PreToolUse hook can find its marker) → `bmad-dev-story` implements the feature and un-skips the story's ATDD tests → run tests/lint/build and **print the raw output** as evidence → (production) `bmad-testarch-test-review` then `bmad-code-review` → commit at green (one commit per green story) → **re-run the
|
|
81
|
+
Drive each in-scope story from its red-phase tests to a green, committed state. Execution is the **sequential `/goal` spine**; per story, in sprint order: set the current story (so the PreToolUse hook can find its marker) → `bmad-dev-story` implements the feature and un-skips the story's ATDD tests → run tests/lint/build and **print the raw output** as evidence → (production) `bmad-testarch-test-review` then `bmad-code-review` → commit at green (one commit per green story) → **re-run the suite at this cycle's scope on the committed HEAD (a story only advances off a full-scope sweep)** and print it, because a story's new files are untracked until the commit, so a tracked-files conformance gate can only red once they land; a red here means the story is not done and must be remediated and re-committed before it advances. The loop is wrapped in a single `/goal` whose condition encodes the per-story Definition-of-Done and carries the literal "…or stop after N turns" escape clause. The printed evidence keeps the run judgeable mid-flight, but **passing the `/goal` condition is not completion**; the authoritative verdict is Stage 5. As the spine advances it overwrites a `run-status.json` heartbeat for pollers. See [`references/execute.md`](https://github.com/armelhbobdad/bmad-module-ultracode-goal/blob/main/skills/ultracode-goal/references/execute.md).
|
|
82
82
|
|
|
83
83
|
### Stage 5: Gate
|
|
84
84
|
|
|
85
|
-
Decide whether a story (or, after the last story, the Epic) advances, by a deterministic artifact read. In production, the skill first backfills the evidence in order: `bmad-testarch-automate`, `bmad-testarch-trace` (which writes the gate decision), `bmad-testarch-nfr`. Then it runs `gate_eval.py
|
|
85
|
+
Decide whether a story (or, after the last story, the Epic) advances, by a deterministic artifact read. In production, the skill first backfills the evidence in order: `bmad-testarch-automate`, `bmad-testarch-trace` (which writes the gate decision), `bmad-testarch-nfr`. Assessors that run or time anything measure a worktree of the commit under assessment, never the shared working tree. Then it runs `gate_eval.py`; a mid-loop re-gate may declare a delta profile (only the previously failed dimension re-runs in full) whose verdict is capped at reloop, so only the full instrument ever advances a story. The script reads TEA's `gate-decision.json` and returns a verdict the skill executes: `advance` (move to the next story), `defer` (append non-blocking items to the ledger and advance anyway), `reloop` (generalize each finding to its defect class against the Epic's class ledger, then run `bmad-correct-course` and re-run the story within the remaining budget), or `escalate` (stop). The invariant: **a P0/critical FAIL never defers**; it re-loops within budget or escalates. See the [gate model](./gate-model.md) and [`references/gate.md`](https://github.com/armelhbobdad/bmad-module-ultracode-goal/blob/main/skills/ultracode-goal/references/gate.md).
|
|
86
86
|
|
|
87
87
|
This is how the verdict is read deterministically; the conductor never grades the work itself, it runs the script and routes on what comes back:
|
|
88
88
|
|
package/docs/operating-tips.md
CHANGED
|
@@ -150,7 +150,9 @@ The gate is a real verdict and worth trusting. The prose around it is not eviden
|
|
|
150
150
|
|
|
151
151
|
**Prefer executed mutation over a modeled mutant.** A hand-written mutant that differs from the real pre-fix code proves nothing about the real branch. Where a story claims a mutation-proof, re-run one yourself against the shipped code.
|
|
152
152
|
|
|
153
|
-
**
|
|
153
|
+
**Use the shipped runner rather than hand-rolling the harness.** `uv run <skill-root>/scripts/mutate_check.py --table <rows.json>` builds the properties every hand-rolled harness eventually lost in as refusals: a green-baseline precheck, per-row anchor uniqueness (a missing anchor is reported as STALE, the formatter-reflow signature, never silently skipped), an attributable red, a byte-identical restore, and a generated `mutation-ledger-<story_id>.md` that leads with GREEN survivors. The two tips below are what the runner automates; they remain worth knowing for reading its output and for the rare harness it cannot drive.
|
|
154
|
+
|
|
155
|
+
**Make any hand-rolled harness hard-error on a missing anchor.** Observed: a harness silently skipped its mutation twice when a refactor moved the anchor, and reported the suite's ordinary green as a passing twin.
|
|
154
156
|
|
|
155
157
|
**Back files up with `cp` and restore from the backup.** Never `git checkout -- <path>` or `git restore <path>` to undo a mutation: both restore from the *index*, and an intent-to-add entry (`git add -N`) holds the empty blob, so they truncate the file to zero bytes and exit 0 (measured on git 2.47). For a file that has a committed version, `git checkout HEAD -- <path>` is safe. For a file the story just created, nothing git offers is: `git checkout HEAD --` refuses, and `git restore --source=HEAD --` deletes the file with exit 0. The `cp` backup is the only restore that always works.
|
|
156
158
|
|
package/docs/troubleshooting.md
CHANGED
|
@@ -53,6 +53,10 @@ The decision log carries the full blocker list with what each needs to clear. Re
|
|
|
53
53
|
|
|
54
54
|
Note this is fail-closed on purpose: a missing or unreadable gate artifact escalates rather than being assumed green. The slim file's *absence alone* is not the problem: the script falls back to the summary, and that fallback is explicitly not a failure. See the [gate model](./gate-model.md) for how the artifact resolves into a `gate_status` and the full verdict mapping.
|
|
55
55
|
|
|
56
|
+
## Gate re-loops on a sweep-scope or delta cap
|
|
57
|
+
|
|
58
|
+
A per-story gate can return `reloop` with a `PASS` gate status when a cap fired rather than a finding: the `reasons` name either the sweep-scope cap (the `.tests-ran` marker says `scope=scoped`, or the marker/`scope=` line is missing) or the delta-profile cap (`--cycle-profile delta` was declared). Neither is an error; both mean "not yet, and here is why": a story only advances off a full-scope sweep judged by the full gate instrument. The remedy is the cap-only re-loop, which skips the generalize ceremony entirely: re-run the full three-axis sweep, refresh the marker with `scope=full`, and re-run the gate without `--cycle-profile`. See the cap sections in the [gate model](./gate-model.md).
|
|
59
|
+
|
|
56
60
|
## Hooks not firing
|
|
57
61
|
|
|
58
62
|
**Symptom.** A commit lands on a protected branch, or a commit lands before a story's tests ran: the invariants the PreToolUse hook should enforce did not block.
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-module-ultracode-goal",
|
|
4
|
-
"version": "2.
|
|
4
|
+
"version": "2.2.0",
|
|
5
5
|
"description": "BMAD module — Run a BMAD Epic autonomously to a machine-checked, TEA-gated Definition-of-Done.",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
|
@@ -36,7 +36,7 @@ parallel_max_concurrency = 8
|
|
|
36
36
|
# per-story baseline SHA with `git rev-parse HEAD` before any implementation, and the evidence
|
|
37
37
|
# trail bounds a story's commit range from it. Omitting them makes an unattended run stop for a
|
|
38
38
|
# permission prompt exactly where walk-away autonomy is claimed.
|
|
39
|
-
allowlist_commands = ["git status", "git add", "git commit", "git rev-parse", "git diff", "npm test", "npm run lint", "npm run build", "pytest", "uv run"]
|
|
39
|
+
allowlist_commands = ["git status", "git add", "git commit", "git rev-parse", "git diff", "npm test", "npm run lint", "npm run build", "pytest", "uv run", "date"]
|
|
40
40
|
|
|
41
41
|
# --- Hook fired at epic completion (prompt or shell command); empty = none ---
|
|
42
42
|
on_epic_complete = ""
|
|
@@ -8,9 +8,11 @@ Preconditions (asserted, not assumed): you are on the Epic branch (`{workflow.ep
|
|
|
8
8
|
|
|
9
9
|
**On resume, re-enter Execute at the first story whose last logged gate verdict is not advance; advanced stories are not re-run; re-assert (do not rebuild) the Epic branch, hooks, the allowlist, the `.mem-state.json` recall latch, and the in-flight story's baseline marker before continuing** — and "re-assert the hooks" means preflight step 5's assertion, not a presence check: both script paths appear in the resolved settings **and** the PreToolUse group's matcher is the all-tools form (`references/preflight.md`, "Arm the PreToolUse group with a matcher that covers EVERY tool"). A presence-only read passes on a prior run's `Bash`-only arming, and a resume that skips Stage 2 never meets the bullet where that requirement lives — so this is the one place it binds a resumed run, and without it the Cross-Session Recall invariant enforces nothing for the whole session while every log line says the hooks are armed. The latch is *re-written* by re-running Stage 1's latch step (`references/ingest-and-scope.md`), because Stage 6 removes it at every close-out and the PreToolUse hook reads an absent latch as "no run active" and stops gating claude-mem entirely — and a latch already on disk whose `run_id` is not this run's is **another run's (or a spawning wrapper's) artifact, never evidence**: re-latch from this run's own measurement and never inherit its `claude_mem` value, because an inherited green latch is a capability claim this run never made, which is exactly what ingest-and-scope.md's never-synthesize-a-probe rule forbids. The baseline marker is the opposite of the latch — split by which case the resume is, and the test is simply whether `.baseline-<story_id>` exists. **Mid-story (it exists):** *re-read* and never regenerate — the story may already carry commits, and a rebuilt baseline silently re-anchors its evidence range to a mid-story HEAD. **Story boundary (it does not exist** — the NORMAL case under `--max-stories N`, where every invocation after the first resumes at a fresh story): there is nothing to re-read; write it fresh at `HEAD` exactly as step 0 does for a first-run story, because the PreToolUse marker-freshness gate fail-closes without it.
|
|
10
10
|
|
|
11
|
+
**A resume owes the stall check first.** Before the re-assertions below, apply the per-step heartbeat's gap rule (the "Run-status heartbeat" section): when the in-flight story's `.heartbeat-<story_id>` exists and its last line is older than 30 minutes, write the `.stall-<story_id>` line before driving anything — the resume is the only reader positioned to see the gap while its cause is still nearby.
|
|
12
|
+
|
|
11
13
|
**The re-assertion sweep is one script call, not a tool action per fact.** Run `uv run {skill-root}/scripts/preflight_check.py --assert-armed --project-root {project-root} --impl-artifacts {workflow.implementation_artifacts} --epic-branch-prefix {workflow.epic_branch_prefix} --story <the story about to be driven> --run-id <this run's id> --trace-output {workflow.trace_output_dir}` (plus one `--protected-branch <name>` per protected branch, as Stage 2's step 1 passes them — the prefix and the protected list are resolved scalars, and omitting either would assert the branch against a default the operator may have overridden). Exit 0 asserts the branch, both hooks with the all-tools matcher, and the five injected hook env vars in one read, and the `story` block returns the artifact-derived route this paragraph defines — `story-boundary`, `mid-story`, or `gate-owed` (the committed-green-un-gated third case below; the script only ever returns it when `--trace-output` named the run's real, readable trace dir and the artifact evidence held in full — an unverifiable location reads `unknown` and routes to the ordinary loop, so lack of evidence never skips steps 1 through 4). Three things the manifest deliberately does **not** cover: the latch is **reported, never certified** — whatever the JSON shows, re-latch from this run's own measurement exactly as above; a boundary story's fresh baseline is still written by step 0; and the **allowlist** is neither asserted nor reported (it lives in permission settings the script does not read), so re-assert it exactly as the list at the top of this paragraph requires. A non-zero exit is the resume's RED: fix what the `reasons` name before continuing, and never narrate around it.
|
|
12
14
|
|
|
13
|
-
**A third case: a story already committed, green and un-gated.** The routing above sorts stories into two classes — re-drive, and do not re-run — and a session that died between its commit and its gate belongs to neither. Recognise it by artifacts, never by prose: the row carries commits past its baseline marker **and** a `.tests-ran-<story_id>` marker whose `baseline=` line still matches that marker, **and** neither `trace-<story_id>.md` nor `gate-decision-<story_id>.json` exists under `{workflow.trace_output_dir}`. That story owes **evidence and a gate, not implementation**. Re-set the `.current-story` pointer (the pointer half of step 0 only — the baseline marker is still re-read, never rewritten), skip steps 1 through 4, and enter at **step 5**, re-measuring the committed HEAD rather than reading the tests-ran marker's numbers back; then continue into Stage 5 as normal. Trust the artifacts and the `sprint-status.yaml` row, never the story file's own prose: such a file routinely carries a close-out asserting a gate `PASS` that no artifact backs, which is precisely the claim this stage does not get to make. **The discriminator must hold in full.** A tests-ran marker with no commits, or commits with no marker, is a story that died before or during the sweep, and it takes the ordinary loop from step 0.
|
|
15
|
+
**A third case: a story already committed, green and un-gated.** The routing above sorts stories into two classes — re-drive, and do not re-run — and a session that died between its commit and its gate belongs to neither. Recognise it by artifacts, never by prose: the row carries commits past its baseline marker **and** a `.tests-ran-<story_id>` marker whose `baseline=` line still matches that marker, **and** neither `trace-<story_id>.md` nor `gate-decision-<story_id>.json` exists under `{workflow.trace_output_dir}`. That story owes **evidence and a gate, not implementation**. Re-set the `.current-story` pointer (the pointer half of step 0 only — the baseline marker is still re-read, never rewritten), skip steps 1 through 4, and enter at **step 5**, re-measuring the committed HEAD rather than reading the tests-ran marker's numbers back — at FULL scope, since no step 2 ran this session to inherit from — and once green, rewrite the tests-ran marker fresh (same `baseline=` line, `scope=full`) **even on a green first try**: the inherited marker may predate the scope contract or carry a mid-loop `scope=scoped`, and the gate reads the marker THIS session leaves behind, not the dead one's; then continue into Stage 5 as normal. Trust the artifacts and the `sprint-status.yaml` row, never the story file's own prose: such a file routinely carries a close-out asserting a gate `PASS` that no artifact backs, which is precisely the claim this stage does not get to make. **The discriminator must hold in full.** A tests-ran marker with no commits, or commits with no marker, is a story that died before or during the sweep, and it takes the ordinary loop from step 0.
|
|
14
16
|
|
|
15
17
|
## Default — Sequential `/goal` spine
|
|
16
18
|
|
|
@@ -43,15 +45,17 @@ Then terminate as the work-bound section below describes for an invocation that
|
|
|
43
45
|
For each in-scope story, in sprint order, run this loop:
|
|
44
46
|
|
|
45
47
|
0. **Set the current story and record its baseline commit** — both **before any implementation**, before step 1 touches a single file. First, so the PreToolUse hook can locate this story's marker: write the story id to `{workflow.implementation_artifacts}/.current-story` (or export `ULTRACODE_STORY_ID=<story_id>` for the run). The guard reads the story id from `ULTRACODE_STORY_ID`, falling back to that `.current-story` file — without one set, it cannot find the marker and denies every commit. Then anchor the story: **if `{workflow.implementation_artifacts}/.baseline-<story_id>` is absent, write it now** with the output of `git rev-parse HEAD` — **exactly one newline-terminated line carrying the full 40-hex SHA**. This format is a contract, not a style choice: later stages bound this story's commit range from it and compare it as a verbatim whitespace-stripped string, so a `git rev-parse --short` abbreviation or a write with no trailing newline is a silent breakage, not a cosmetic one. **If the marker is already present, this story is a resume: read it and keep it — never overwrite it.** No later step refreshes it; a multi-commit story must keep the SHA it started from even after HEAD moves. **Scratch build space, if the story needs any.** A story that must build and run a throwaway program to produce a committed artifact puts it in `{workflow.implementation_artifacts}/.scratch-<story_id>/` and nowhere else. Not an improvised repo-root directory — that is one careless `git add` away from being committed, and only step 4's explicit per-path staging would catch it. Not `/tmp` either: it survives no audit, belongs to no run, and buys nothing the run folder does not. What this location buys is containment: the dot prefix and the run folder keep it out of every default source glob and out of the commit. What it does **not** buy is runnability — a headless permission layer gates on command **shape**, not path, so a compound `bash <scratch>/script.sh 2>&1 | tee …`, a `for` loop over gate names, or a `$(…)` substitution can each be refused wherever the file sits. Keep scratch work invocable as one simple command per call, and expect to run its steps as separate calls rather than through one orchestrating script. The directory outlives the step-4 commit **deliberately**: step 5 re-runs the same declared-gate sweep on the committed tree, so a runner or harness built here is still needed there, and one deleted at step 4 gets re-authored verbatim at step 5 — a real, measured cost. **Remove the directory after step 5's re-verify goes green, before the story advances** — never earlier, and never later: the next story's step 0 must not find it.
|
|
46
|
-
1. `bmad-create-story` (if not already current) → `bmad-dev-story` implementing the feature and **un-skipping** the story's ATDD acceptance tests (remove `test.skip()`) — **production only**; under `--light` there are no ATDD tests to un-skip, so the DoD is the story's acceptance criteria satisfied with lint and build green. **For a decomposed child row, skip `bmad-create-story` entirely** — the child has no story file and must never get one (running the sub-skill would duplicate the contract, which the sizing conventions above forbid); its owed criteria are its sprint-status row comment plus the parent story file it cites by AC number, and the parent file is read for the AC text only. **On a stack where Stage 3 could not generate the ATDD scaffold** (the non-web case), the story still owes a red phase and it is owed **now, before the first implementation edit**: author the acceptance cases with real assertions, RUN them, and capture that run's raw output to a file under `{workflow.implementation_artifacts}` — `references/gate.md`'s non-web section is the authority on the shape, but by the time that file is read at Stage 5 the window this rule governs has already closed. Before delegating, check the run's `.decision-log.md` for an **Operator notes** entry naming this story (a pre-launch hint like "watch the auth flow in story 3") and surface that note into the `bmad-dev-story` / review context for this story so the operator's last words actually reach the work.
|
|
47
|
-
2. Run the tests, lint, and build, and **PRINT the raw output** (pass/fail counts, lint result, build result) into the transcript. **In a monorepo, scope that test run wide enough to catch cross-package regressions** — run the full suite, or the changed package(s) plus everything downstream of them (e.g. an `--affected` set *and its dependents*), not just this story's own package. A story's change can regress a *sibling* package's conformance gate (a cross-package RBAC, ownership, or architecture-boundary check), and a run scoped to only this story's package reports green while that sibling gate is red — deferring the red (and its fix) to a later story or the Epic gate instead of catching it at the story that caused it. This printed evidence is what keeps the run judgeable mid-flight and what the `/goal` driver reads to pace itself — but it is *evidence*, not the gate. **Once all three go green, write the tests-ran marker** `{workflow.implementation_artifacts}/.tests-ran-<story_id>` — the PreToolUse guard denies the step-4 commit unless this marker exists for the current story, so write it only after a real green run. **That marker must carry a `baseline=<sha>` line** whose value is the verbatim, whitespace-stripped content of `{workflow.implementation_artifacts}/.baseline-<story_id>` written at step 0 — **copied, never re-derived from `git rev-parse HEAD`** at marker-write time. The guard compares the two strings character for character and denies the commit when they differ or when the line is missing, which is what stops a marker written against an older code state from being replayed to satisfy a later commit. Re-deriving the value here instead of copying it is the trap: on a story that commits more than once, HEAD has already moved past the baseline, so a re-derived line denies the story's own second commit. **Write that marker with the Write tool**, after Reading the baseline file and pasting its line unchanged — for the same reason step 4 writes the decision log with Write/Edit. The obvious Bash form, `printf 'baseline=%s\n' "$(cat …/.baseline-<story_id>)" > …/.tests-ran-<story_id>`, is a command substitution that a harness which statically analyses Bash strings refuses outright (the PreToolUse guard permits it; the refusal comes from above). Hand-retyping the forty hex characters instead is precisely the transcription error the freshness clause exists to catch, and its denial says only that the SHAs differ, never that one was mistyped.
|
|
48
|
+
1. `bmad-create-story` (if not already current) → `bmad-dev-story` implementing the feature and **un-skipping** the story's ATDD acceptance tests (remove `test.skip()`) — **production only**; under `--light` there are no ATDD tests to un-skip, so the DoD is the story's acceptance criteria satisfied with lint and build green. **For a decomposed child row, skip `bmad-create-story` entirely** — the child has no story file and must never get one (running the sub-skill would duplicate the contract, which the sizing conventions above forbid); its owed criteria are its sprint-status row comment plus the parent story file it cites by AC number, and the parent file is read for the AC text only. **On a stack where Stage 3 could not generate the ATDD scaffold** (the non-web case), the story still owes a red phase and it is owed **now, before the first implementation edit**: author the acceptance cases with real assertions, RUN them, and capture that run's raw output to a file under `{workflow.implementation_artifacts}` — `references/gate.md`'s non-web section is the authority on the shape, but by the time that file is read at Stage 5 the window this rule governs has already closed. Before delegating, check the run's `.decision-log.md` for an **Operator notes** entry naming this story (a pre-launch hint like "watch the auth flow in story 3") and surface that note into the `bmad-dev-story` / review context for this story so the operator's last words actually reach the work. Read `{workflow.implementation_artifacts}/.class-ledger-<epic_id>.md` the same way when it exists — the defect classes this Epic's reviews have already caught, appended by the gate's re-loop route (`references/gate.md`) — and surface it into the same context: a class named there is a class the reviewers will look for again, and code written blind to it is a re-loop pre-booked. (A story otherwise inherits nothing from its siblings' reviews — a measured run re-found the previous story's review finding on its very first cycle — so the ledger is the one artifact that crosses that boundary.)
|
|
49
|
+
2. Run the tests, lint, and build, and **PRINT the raw output** (pass/fail counts, lint result, build result) into the transcript. **In a monorepo, scope that test run wide enough to catch cross-package regressions** — run the full suite, or the changed package(s) plus everything downstream of them (e.g. an `--affected` set *and its dependents*), not just this story's own package. A story's change can regress a *sibling* package's conformance gate (a cross-package RBAC, ownership, or architecture-boundary check), and a run scoped to only this story's package reports green while that sibling gate is red — deferring the red (and its fix) to a later story or the Epic gate instead of catching it at the story that caused it. This printed evidence is what keeps the run judgeable mid-flight and what the `/goal` driver reads to pace itself — but it is *evidence*, not the gate. **Once all three go green, write the tests-ran marker** `{workflow.implementation_artifacts}/.tests-ran-<story_id>` — the PreToolUse guard denies the step-4 commit unless this marker exists for the current story, so write it only after a real green run. **That marker must carry a `baseline=<sha>` line** whose value is the verbatim, whitespace-stripped content of `{workflow.implementation_artifacts}/.baseline-<story_id>` written at step 0 — **copied, never re-derived from `git rev-parse HEAD`** at marker-write time. The guard compares the two strings character for character and denies the commit when they differ or when the line is missing, which is what stops a marker written against an older code state from being replayed to satisfy a later commit. Re-deriving the value here instead of copying it is the trap: on a story that commits more than once, HEAD has already moved past the baseline, so a re-derived line denies the story's own second commit. **Write that marker with the Write tool**, after Reading the baseline file and pasting its line unchanged — for the same reason step 4 writes the decision log with Write/Edit. The obvious Bash form, `printf 'baseline=%s\n' "$(cat …/.baseline-<story_id>)" > …/.tests-ran-<story_id>`, is a command substitution that a harness which statically analyses Bash strings refuses outright (the PreToolUse guard permits it; the refusal comes from above). Hand-retyping the forty hex characters instead is precisely the transcription error the freshness clause exists to catch, and its denial says only that the SHAs differ, never that one was mistyped. **The marker carries the sweep's scope beside its baseline** — two more lines, same file: `scope=<full|scoped>` naming which sweep form this green run was (the re-loop rule below is the only thing that ever writes `scoped`; a first pass writes `scope=full`), and — scoped only — `packages=<the resolved package list>` copied from the runner's own printout, never hand-typed. The gate reads the `scope=` line through `gate_eval.py --tests-ran` and caps a scoped cycle's verdict at `reloop` (`references/gate.md`, the sweep-scope AND), which is what stops a scoped sweep from impersonating a full one at the only place the distinction pays.
|
|
48
50
|
|
|
49
51
|
**Scope the sweep on three axes, not one.** Package breadth above is the first; the other two are how a green sweep is most often blind:
|
|
50
52
|
|
|
51
53
|
- **Language.** An aggregate task runner only runs the tasks packages *declare* to it, so in a polyglot repo the wide run can report every task green while an entire language's suite was never invoked. Read the runner's executed-task list against the languages actually present: a Rust crate with no `test` task simply does not appear in a turbo task count, and "8 of 8 tasks passed" says nothing about it. Run each uncovered language's suite explicitly (`cargo test`, `go test ./...`, `pytest`) and print those results alongside. **A task count is not a coverage claim.**
|
|
52
54
|
- **The repo's DECLARED gate set, not its default task.** Enumerate the quality tasks the repo actually declares (its `package.json` scripts, the turbo/nx pipeline, the CI workflow's jobs) and name, in the printed evidence, every one the default task excludes. A `test:storybook` or a `test:coverage`/coverage-ratchet that is a *sibling* task rather than a dependency of `test` is invisible to `run test`, and can sit red across an entire Epic while every sweep prints green. Run the excluded ones too, or state explicitly which you are not running and why.
|
|
53
55
|
|
|
54
|
-
**Defeat the build cache at least once.** In a content-addressed task runner (turbo, nx, bazel) a cache hit is precisely the state in which a sibling package's gate is *not* re-executed, which is the cross-package regression this step exists to catch. Run the authoritative sweep cache-defeating (turbo's `--force`, nx's `--skip-nx-cache`, bazel's `--nocache_test_results`, or the package's own runner directly) and **state in the printed evidence whether the run was cached** — a task count that is entirely cached, returning in under a second, is the signature of a replay (turbo prints a `FULL TURBO` line for it), and a run that is green cached and red forced is indistinguishable from a green one in the transcript. **The same rule binds any harness the story itself drives through the task runner** — mutation, fault injection, a twin-verification loop — and there the replay is worse than a false green: an unchanged task hash replays the pre-mutation GREEN for every mutation, so every mutation "survives" with zero failing cases, which reads as a devastating vacuity finding about the suite rather than as a cache hit. A subject **outside** the filtered package's tree (a repo-root script, a shared config, a generated artifact) never changes that package's task hash at all, so filter-plus-mutate is the specific combination that always needs the cache defeated. A string-replacement harness additionally owes its anchors a uniqueness assertion before mutating: an anchor that occurs more than once mutates a different site than it reports, and a `KILLED` earned against the wrong layer is the same silent inversion from the other side.
|
|
56
|
+
**Defeat the build cache at least once.** In a content-addressed task runner (turbo, nx, bazel) a cache hit is precisely the state in which a sibling package's gate is *not* re-executed, which is the cross-package regression this step exists to catch. Run the authoritative sweep cache-defeating (turbo's `--force`, nx's `--skip-nx-cache`, bazel's `--nocache_test_results`, or the package's own runner directly) and **state in the printed evidence whether the run was cached** — a task count that is entirely cached, returning in under a second, is the signature of a replay (turbo prints a `FULL TURBO` line for it), and a run that is green cached and red forced is indistinguishable from a green one in the transcript. **The same rule binds any harness the story itself drives through the task runner** — mutation, fault injection, a twin-verification loop — and there the replay is worse than a false green: an unchanged task hash replays the pre-mutation GREEN for every mutation, so every mutation "survives" with zero failing cases, which reads as a devastating vacuity finding about the suite rather than as a cache hit. A subject **outside** the filtered package's tree (a repo-root script, a shared config, a generated artifact) never changes that package's task hash at all, so filter-plus-mutate is the specific combination that always needs the cache defeated. A string-replacement harness additionally owes its anchors a uniqueness assertion before mutating: an anchor that occurs more than once mutates a different site than it reports, and a `KILLED` earned against the wrong layer is the same silent inversion from the other side. **The module ships that harness — use it instead of hand-rolling one into scratch space.** `uv run {skill-root}/scripts/mutate_check.py --table <rows.json>` drives a JSON table of `{file, anchor, replacement, command, expect_red}` rows with the properties every hand-rolled version eventually lost, built in as refusals: a green-baseline precheck (a mutation over a red baseline proves nothing, exit 2), per-row anchor uniqueness (0 occurrences reported distinctly as the formatter-reflow signature), an ATTRIBUTABLE red (the expected case named in the output, extra reds surfaced), and a byte-identical restore verification. Its generated ledger leads with GREEN survivors, because each green is a finding — a guard that cannot fail — and exit 1 on any green, misattributed red, or unheld anchor makes the runner gate-able. **Every guard or assertion this story adds or changes appears as a table row with an attributable red BEFORE the gate runs**, and the generated ledger is a story artifact the reviewers read: write it with `--output {workflow.implementation_artifacts}/mutation-ledger-<story_id>.md` — name-pinned per story like every artifact a later stage resolves, never scratch space (scratch dies at story end, which is exactly how the harness kept getting re-rolled) — and surface it into the step-3 test-review context in production, citing it in the story record either way. The cache rule above binds the table's `command` verbatim: the runner cannot defeat the task-runner cache for you. A repo may substitute its own runner only when it meets the same properties.
|
|
57
|
+
|
|
58
|
+
**A re-loop iteration may scope the sweep; the cycle that expects to advance owes the full one.** The first pass through this loop runs the full three-axis sweep above. A story re-entered from a `reloop` verdict may instead run the affected set — the changed package(s) *and their dependents*, computed against the story's last recorded commit (`{workflow.implementation_artifacts}/.commit-<story_id>`, the previous cycle's end marker): e.g. turbo's `--filter=...[<that sha>]`, still cache-defeated, still with raw output printed. Two obligations keep the scoped form honest. **Print the resolved package list** — the runner computes it; copy its own output into the transcript and the marker's `packages=` line, because a wrong baseline ref narrows coverage *silently*, and the printed list is the only place a narrowing shows. And **write the marker with `scope=scoped`**, so the record says which form ran — a scoped sweep reported as a full one is the same class of false claim reviews keep finding elsewhere. When the re-loop believes its findings are closed and the next gate should advance, run the full sweep and write `scope=full`: the gate refuses to advance a story off a `scope=scoped` marker (`references/gate.md`, the sweep-scope AND), so a scoped cycle that unexpectedly goes fully green costs one extra full-sweep cycle rather than producing an advance nothing re-proved. When in doubt, run it full — `scope=full` is never wrong, only slower. (Measured, on a seven-cycle story: the whole-repo cache-defeated sweep re-proved untouched packages seven times, roughly six of every eight sweep minutes; this split is that cost's fix, at no loss of gate authority. A repo whose runner cannot compute an affected set simply keeps running the full sweep — the scoped form is an optimisation, never an obligation.)
|
|
55
59
|
|
|
56
60
|
**A wide sweep is also a loaded machine.** Running every language's suite plus the declared gate set, cache-defeating, means suites share a host with concurrent native builds — and a case bounded by a wall-clock timeout can fail from contention alone (measured: a 180-second-bounded case took 197s beside a concurrent `cargo test` + clippy + type-check, and 93s alone). A timeout in a wide sweep is therefore not yet a red: re-run that package in isolation before treating it as one, and where the two can be sequenced, do not schedule a long native build beside a wall-clock-bounded suite.
|
|
57
61
|
|
|
@@ -60,8 +64,8 @@ For each in-scope story, in sprint order, run this loop:
|
|
|
60
64
|
**A repo with commit hooks carries gates this sweep structurally cannot see.** `lefthook` / `husky` / `.git/hooks/pre-commit` jobs — typos, formatters, staged-file linters, clippy — run against the staged index and are reachable from no script you can invoke here. So the step-4 commit attempt is itself part of verification rather than the end of it: a rejection there is expected traffic, not a failure of this sweep. What that rejection does to the tests-ran marker is step 4's problem, and it is not the obvious answer.
|
|
61
65
|
|
|
62
66
|
3. **Production profile only:** `bmad-testarch-test-review`, then `bmad-code-review`. A `CONCERNS`-grade or non-critical finding appends to `{workflow.deferred_work_path}` and the story proceeds. A **P0 / critical FAIL never defers** — re-loop within the turn budget or escalate.
|
|
63
|
-
4. **Commit at green** (one commit per green story). The PreToolUse hook blocks a commit on a protected branch, blocks a commit when no "tests-ran" marker exists for the current story, and — its marker-freshness clause — blocks a commit when that marker's `baseline=` line does not match the story's recorded baseline, so step 0 must have set the current story and step 2 must have actually run and written `{workflow.implementation_artifacts}/.tests-ran-<story_id>`, in the format step 2 pins, **in a prior, SEPARATE command**. The guard pre-evaluates the command *string* before it runs, so a marker written by a `printf`/`echo` in the SAME compound command as the commit (e.g. `printf … > .tests-ran-<id> && git commit …`) does not yet exist at hook-eval time and the commit is denied. Write the marker, then commit — two distinct tool calls. **A commit can also be rejected by something that is NOT the guard, and the marker rule inverts there.** A repo-side `pre-commit` hook (lefthook, husky, a plain `.git/hooks/pre-commit`) running clippy, rustfmt, typos or a staged-file linter rejects the commit without moving anything: HEAD is where it was, so the tests-ran marker's `baseline=` line still matches character for character and the guard will wave the retry straight through — carrying evidence collected *before* the fix. The freshness clause keys on the BASELINE, not on the code state, so it cannot see this case by construction. Treat a hook rejection exactly as step 5 treats a post-commit red: remediate, re-run the story's tests/lint/build to green, write a **fresh** `.tests-ran-<story_id>` (same `baseline=` line, still copied and never re-derived), and only then retry the commit. **Append `.decision-log.md` with Write/Edit, never a Bash heredoc.** Stages 4 and 6 require a log of what happened during the run, and what happened *is* commit behaviour — so the log quotes the verbs. The guard evaluates Bash command strings, and a heredoc carrying that prose is a Bash command string: a line leading with the verb is a verb-leading segment, and backticked identifiers or an apostrophe send the whole segment down the fail-closed matches-anywhere path. The guard **evaluates Bash command strings only**, so writing the log with the file tools removes the collision instead of phrasing around it, and keeps Bash for commands you actually mean to run. Said that way rather than "the guard never sees Write/Edit", because preflight arms the hook for EVERY tool (its Cross-Session Recall invariant gates an MCP call, which a `Bash`-only matcher never reaches): the hook IS invoked for a `Write`, it simply has nothing to evaluate and returns no decision. The conclusion is unchanged, but a reader who believes the guard cannot see a `Write` and then meets the all-tools matcher will try to resolve the contradiction by narrowing the matcher back. **Once the log is too large to Read affordably, do not Read it.** It grows monotonically, one session's prose per invocation, and both file tools refuse a file that has not been Read in this session — so on a long-running Epic (288 KB by its eleventh invocation) the two sanctioned paths are both closed at once. Compose this session's entry with Write into a NEW sibling file in the run folder, `{workflow.implementation_artifacts}/dlog-slice-<story_id>-<n>.md`, where `<n>` increments per append — the name has to be unique per append, not per story, or a story's SECOND entry finds the slice already on disk and walks straight back into the Read it was escaping; then append it in one Bash call, `cat <slice> >> .decision-log.md`. That command's leading token is `cat`, it carries no backtick, no `$(`, and no verb-leading segment, so the guard makes no decision on it and the prose never enters a Bash command string at all. Keep the slice beside the log rather than deleting it: it is the only per-session copy if the append is ever interrupted. **Stage only the current story's paths.** In production, Stage 3 generated *every* in-scope story's red-phase tests up front, so a sibling story's still-skipped tests can co-reside in the same shared dirs as this story's (one `test/` package may hold two stories' acceptance specs before either commits). A blanket `git add -A` here sweeps those uncommitted sibling tests into this story's commit and pollutes the per-story boundary — stage this story's source + test paths explicitly (`git add <this story's paths>`) instead. The allowlist ships plain `git add` (a prefix covering both `-A` and a scoped `git add <paths>`) so the targeted add needs no extra approval. **Staging is itself a prior, SEPARATE tool call**: `git add <this story's paths>` as one call, then `git commit` as a second — the same two-distinct-tool-calls rule the marker follows, and for the same reason. The guard also denies a commit whose staged index is empty, and a chained `git add … && git commit` has staged nothing at the moment it pre-evaluates that command string, so the chained form is denied even though the add would have worked. `git commit -a` and commit-time pathspecs (`git commit <paths>`) are unsupported inside the story loop for exactly that reason. (Under `--light` no ATDD tests are generated, so the collision cannot arise.) **Once the commit lands, record the story's END sha.** Write `{workflow.implementation_artifacts}/.commit-<story_id>` with the output of `git rev-parse HEAD`, in exactly the format step 0 pins for the baseline: one newline-terminated line carrying the full 40-hex SHA. Unlike the baseline, **this marker is refreshed, not preserved** — step 5 rewrites it after any remediation re-commit or amend, so it always names the story's last commit rather than its first. It exists to close the story's commit range explicitly: without it the evidence trail has to *infer* a range end from the NEXT story's baseline marker, and a later story anchored at this story's own SHA collapses that range to `X..X`, so a story that did commit renders as having committed nothing.
|
|
64
|
-
5. **Re-verify on the committed HEAD.** Re-run the authoritative
|
|
67
|
+
4. **Commit at green** (one commit per green story). The PreToolUse hook blocks a commit on a protected branch, blocks a commit when no "tests-ran" marker exists for the current story, and — its marker-freshness clause — blocks a commit when that marker's `baseline=` line does not match the story's recorded baseline, so step 0 must have set the current story and step 2 must have actually run and written `{workflow.implementation_artifacts}/.tests-ran-<story_id>`, in the format step 2 pins, **in a prior, SEPARATE command**. The guard pre-evaluates the command *string* before it runs, so a marker written by a `printf`/`echo` in the SAME compound command as the commit (e.g. `printf … > .tests-ran-<id> && git commit …`) does not yet exist at hook-eval time and the commit is denied. Write the marker, then commit — two distinct tool calls. **A commit can also be rejected by something that is NOT the guard, and the marker rule inverts there.** A repo-side `pre-commit` hook (lefthook, husky, a plain `.git/hooks/pre-commit`) running clippy, rustfmt, typos or a staged-file linter rejects the commit without moving anything: HEAD is where it was, so the tests-ran marker's `baseline=` line still matches character for character and the guard will wave the retry straight through — carrying evidence collected *before* the fix. The freshness clause keys on the BASELINE, not on the code state, so it cannot see this case by construction. Treat a hook rejection exactly as step 5 treats a post-commit red: remediate, re-run the story's tests/lint/build to green, write a **fresh** `.tests-ran-<story_id>` (same `baseline=` line, still copied and never re-derived; `scope=`/`packages=` re-stated for what the re-run actually was), and only then retry the commit. **Append `.decision-log.md` with Write/Edit, never a Bash heredoc.** Stages 4 and 6 require a log of what happened during the run, and what happened *is* commit behaviour — so the log quotes the verbs. The guard evaluates Bash command strings, and a heredoc carrying that prose is a Bash command string: a line leading with the verb is a verb-leading segment, and backticked identifiers or an apostrophe send the whole segment down the fail-closed matches-anywhere path. The guard **evaluates Bash command strings only**, so writing the log with the file tools removes the collision instead of phrasing around it, and keeps Bash for commands you actually mean to run. Said that way rather than "the guard never sees Write/Edit", because preflight arms the hook for EVERY tool (its Cross-Session Recall invariant gates an MCP call, which a `Bash`-only matcher never reaches): the hook IS invoked for a `Write`, it simply has nothing to evaluate and returns no decision. The conclusion is unchanged, but a reader who believes the guard cannot see a `Write` and then meets the all-tools matcher will try to resolve the contradiction by narrowing the matcher back. **Once the log is too large to Read affordably, do not Read it.** It grows monotonically, one session's prose per invocation, and both file tools refuse a file that has not been Read in this session — so on a long-running Epic (288 KB by its eleventh invocation) the two sanctioned paths are both closed at once. Compose this session's entry with Write into a NEW sibling file in the run folder, `{workflow.implementation_artifacts}/dlog-slice-<story_id>-<n>.md`, where `<n>` increments per append — the name has to be unique per append, not per story, or a story's SECOND entry finds the slice already on disk and walks straight back into the Read it was escaping; then append it in one Bash call, `cat <slice> >> .decision-log.md`. That command's leading token is `cat`, it carries no backtick, no `$(`, and no verb-leading segment, so the guard makes no decision on it and the prose never enters a Bash command string at all. Keep the slice beside the log rather than deleting it: it is the only per-session copy if the append is ever interrupted. **Stage only the current story's paths.** In production, Stage 3 generated *every* in-scope story's red-phase tests up front, so a sibling story's still-skipped tests can co-reside in the same shared dirs as this story's (one `test/` package may hold two stories' acceptance specs before either commits). A blanket `git add -A` here sweeps those uncommitted sibling tests into this story's commit and pollutes the per-story boundary — stage this story's source + test paths explicitly (`git add <this story's paths>`) instead. The allowlist ships plain `git add` (a prefix covering both `-A` and a scoped `git add <paths>`) so the targeted add needs no extra approval. **Staging is itself a prior, SEPARATE tool call**: `git add <this story's paths>` as one call, then `git commit` as a second — the same two-distinct-tool-calls rule the marker follows, and for the same reason. The guard also denies a commit whose staged index is empty, and a chained `git add … && git commit` has staged nothing at the moment it pre-evaluates that command string, so the chained form is denied even though the add would have worked. `git commit -a` and commit-time pathspecs (`git commit <paths>`) are unsupported inside the story loop for exactly that reason. (Under `--light` no ATDD tests are generated, so the collision cannot arise.) **Once the commit lands, record the story's END sha.** Write `{workflow.implementation_artifacts}/.commit-<story_id>` with the output of `git rev-parse HEAD`, in exactly the format step 0 pins for the baseline: one newline-terminated line carrying the full 40-hex SHA. Unlike the baseline, **this marker is refreshed, not preserved** — step 5 rewrites it after any remediation re-commit or amend, so it always names the story's last commit rather than its first. It exists to close the story's commit range explicitly: without it the evidence trail has to *infer* a range end from the NEXT story's baseline marker, and a later story anchored at this story's own SHA collapses that range to `X..X`, so a story that did commit renders as having committed nothing.
|
|
68
|
+
5. **Re-verify on the committed HEAD.** Re-run the authoritative suite (tests + lint + build, the same scope step 2 ran this cycle — all three axes including the declared gate set when full; the same resolved package set when this re-loop cycle legitimately ran scoped, per step 2's re-loop rule) on the just-committed, clean tree and **PRINT** the result before advancing. **Run it cache-defeating.** Committing changes a file's git *tracking state* and not one of its bytes, so a content-addressed cache key is unchanged and this re-run replays the pre-commit answer verbatim, observing nothing: eight of eight tasks reported cached, returning in 719ms immediately after a commit, is the whole failure. Pass `--force` (or invoke the package's runner directly), because the one thing this step must actually execute is the tracked-file gates against the newly tracked fileset. Step 2's pre-commit run cannot see a **conformance gate scoped to *tracked* files** — a `git ls-files`-based ref/ownership/architecture-boundary check: a story's **new** files are still untracked at step 2, so the gate skips them and reports green, then reds the instant the commit tracks them (generated or `.gitignore`d files hide the same way). This committed tree is the exact fileset CI and the Stage-5 gate will see, so a story that is green pre-commit but red here is a **false green** the pre-commit run and the `/goal` transcript evaluator both miss. A red means the story is **not** done: remediate, write a fresh `.tests-ran-<story_id>` marker, and re-commit (amend or a follow-up within the story boundary) until the committed HEAD is green — only then advance. **That fresh marker carries the same `baseline=<sha>` line as step 2's**: the verbatim, whitespace-stripped content of `{workflow.implementation_artifacts}/.baseline-<story_id>` — **copied, never re-derived from `git rev-parse HEAD`**. The story's baseline has not moved; HEAD has, because step 4 just committed. Re-deriving it here would re-anchor the story to that commit and quietly retire the freshness net for the rest of it, which is the very replayed-marker hole the line exists to close — and without the line the guard denies the re-commit outright, so a story that legitimately needs remediation could never close. The fresh marker also re-states `scope=` (and `packages=` when scoped) to match what THIS re-verify actually ran, because the `scope=` line the gate will read through `--tests-ran` is the one written here, not step 2's. **Rewrite `{workflow.implementation_artifacts}/.commit-<story_id>` from `git rev-parse HEAD` once this step's HEAD is green**, in step 4's format — every time, including when nothing was remediated, because an amend replaces the SHA step 4 recorded and a follow-up commit moves it. This marker is the one step-0 artifact whose rule is *refresh*, not *keep*: the baseline names where the story started and must never move, this one names where it ended and must never be stale. That re-commit carries the SAME staging rule as step 4: stage in a prior, SEPARATE tool call (`git add <this story's paths>`), then run the commit (or `git commit --amend`) as a second, distinct call. `git commit -a` and a chained `git add … && git commit` are unsupported here too — the staged index is still empty when the guard pre-evaluates the command string, so the re-commit is denied.
|
|
65
69
|
|
|
66
70
|
**Sub-skill halt catch-all.** If any orchestrated sub-skill blocks on interactive input mid-run, treat it as escalate for that story — write the typed escalation sidecar `{workflow.implementation_artifacts}/escalation-<story_id>.json` (shape below, under "Escalation sidecar") and stop; do not answer its prompt blind.
|
|
67
71
|
|
|
@@ -162,4 +166,6 @@ Before the Stop hook has fired for a story, no `.budget-<story>.json` exists yet
|
|
|
162
166
|
|
|
163
167
|
This is the file the Stage 2 launch briefing points the operator at ("where to watch"); Stage 6 (finalize) records the terminal state into it when the run closes, carrying every key above forward rather than overwriting the file with a narrower object.
|
|
164
168
|
|
|
169
|
+
**The per-step heartbeat sidecar, and the stall marker its gaps earn.** `run-status.json` says where the spine IS; it cannot say how long nothing has happened, because it is overwritten on story/verdict cadence and a five-hour hole between two writes renders exactly like five busy minutes. So, additionally, append one line — the step name and a UTC timestamp — to `{workflow.implementation_artifacts}/.heartbeat-<story_id>` at every numbered story-loop boundary (step 0 through step 5, plus gate start and gate end). One command, no substitution, guard-inert: `date -u "+step-2 %FT%TZ" >> {workflow.implementation_artifacts}/.heartbeat-<story_id>` (the step label rides the format string). Append-only, observability only: the file is never gate evidence, is read by no guard, and is purged at fresh-run arming exactly as `.budget-*` is (`references/preflight.md`), so a prior run's gaps can never fire this run's stall rule. Everywhere else in this file, "the heartbeat" with no qualifier still means `run-status.json` — the attended ticker above rides ITS story/verdict cadence, never this sidecar's per-step one. Its consumer is the RESUME: on re-entry, when the story's heartbeat exists and the gap between its last line and now exceeds 30 minutes, append one line to `{workflow.implementation_artifacts}/.stall-<story_id>` recording the gap, the last step reached, and the best-known cause — a rate-limit message if one is in context, an operator pause if told, `unknown` otherwise — before driving anything. A measured story carried a five-hour zero-write hole between two gate cycles that no artifact explained, and the investigation had to reconstruct it from file mtimes; `unknown, 5h00m, after gate` would have made it a record instead. Leave both sidecars in place at story end — finalize does not remove them, and the run report may state active versus idle time from them honestly.
|
|
170
|
+
|
|
165
171
|
**Attended runs also get a ticker.** Each time you write the heartbeat, print one line into the transcript — `epic-7 ▸ story 3/6 — last verdict: advance` — so the watching human sees motion without opening a JSON file. Skip the ticker in headless (`-H`): the file is the interface there, and transcript prose has no reader.
|
|
@@ -13,6 +13,16 @@ The gate reads artifacts TEA produces. In **production**, before running the gat
|
|
|
13
13
|
|
|
14
14
|
`bmad-testarch-nfr` (produces `nfr-assessment.md`) and `bmad-testarch-test-review` are independent — of each other *and* of the automate→trace chain — so run them in any order, or concurrently with it; `gate_eval.py` consumes all three artifacts without caring how they were produced. (`bmad-testarch-test-review` normally runs in Execute per story — run it here only if you lack a current `test-review.md` for the story.) In **`--light`**, skip all of the above and run only `bmad-testarch-trace`, then the gate with `--profile light` (trace gate only — no NFR/review AND).
|
|
15
15
|
|
|
16
|
+
## Measure the commit, not the working tree
|
|
17
|
+
|
|
18
|
+
Stage 5's subject is a COMMIT, but an assessor that runs or times anything by default measures the WORKING TREE — which is where the next remediation, or a sibling assessor's mutation window, happens. A measured run recorded the consequence three times in one story even under a strict no-concurrent-work rule: one assessor's mutation window contaminated a sibling's timing run (discarded and re-run), and two assessors invented their own ad hoc isolation mid-gate because they could not trust the shared tree. Make the isolation structural instead of a discipline each assessor re-derives: create a worktree of the commit under assessment — `git worktree add <tmp-path> <commit>`, with `<tmp-path>` under `{workflow.implementation_artifacts}/.gate-worktree-<story_id>` (the same containment Execute's scratch rule buys; never an improvised repo-root directory) — and point every assessor's reads, runs and measurements THERE. Their OUTPUTS go home: every artifact an assessor writes — trace report, gate decision, NFR assessment, test review, `bmad-testarch-automate`'s backfilled coverage — lands in the MAIN tree's resolved dirs (`{workflow.trace_output_dir}`, `{workflow.implementation_artifacts}`, resolved to absolute paths against `{project-root}` BEFORE any assessor is pointed at the worktree), never under the worktree. The worktree is the measurement surface, the main tree is the record — an artifact written into the worktree is destroyed with it, after which the gate re-reads the PREVIOUS cycle's same-named artifact as current (resolution is by name, with no freshness check), which is a stale PASS advancing a story whose fresh assessment was never read. Once the story's verdict is recorded, remove the worktree with `git worktree remove --force <tmp-path>` — the copied-in untracked inputs below make the unforced form refuse, and a measurement surface holds nothing to preserve — then `git worktree prune`; a resume that finds a dead session's stale `.gate-worktree-<story_id>` removes it the same way before creating its own. A step that only authors an artifact from already-recorded evidence has nothing to measure and needs no worktree; the rule binds every assessor that runs or times anything. Three caveats the setup owes, each a measured failure when skipped:
|
|
19
|
+
|
|
20
|
+
- **Dependency directories do not follow the commit.** A worktree carries tracked files only, so an installed dependency tree (`node_modules` and its kin) is absent there. Symlink each such directory the repo has (root *and* nested) from the main tree into the worktree — and prefer the symlink over resolver environment variables: Node's ESM resolver ignores `NODE_PATH` (it is honoured by the CJS resolver only), so a `NODE_PATH` bridge quietly breaks exactly the modern half of a JS gate set. Where a symlink cannot serve, an offline install from the package manager's shared store is the fallback; measure both once per repo before committing to one.
|
|
21
|
+
- **Untracked inputs the gate set reads.** A scratch browser config, an env file, a generated fixture: enumerate what the declared gate commands actually read that the commit does not carry, and copy those in deliberately — an isolated gate that silently lost its config measures a different harness and reports the result as this one's.
|
|
22
|
+
- **The ordering rule stands.** Isolation removes the contamination, not the sequence: the run still gates story N to a verdict before touching story N+1 (the sequential spine is unchanged), so this section licenses no overlap. What it buys today is that no assessor can contaminate another's measurement and no stray tree state can contaminate a gate mid-read.
|
|
23
|
+
|
|
24
|
+
Record in `.decision-log.md`, once per gated story, whether the gate measured a worktree or the main tree — the same honesty line the sweep's `scope=` carries: an assessment reported as isolated that actually ran on the shared tree is a false claim about the evidence.
|
|
25
|
+
|
|
16
26
|
## Non-web stack (web-only TEA chain): author the deterministic trace artifacts
|
|
17
27
|
|
|
18
28
|
When the module's TEA chain is web-only, `bmad-testarch-trace` cannot run on a non-web stack and so cannot emit the binding gate decision. Do **not** reverse-engineer a prior run's leftover artifacts — author the two files to the exact shape `scripts/gate_eval.py` reads; every other field in real TEA output (`target`, `links`/`trace_report_path`, `rationale`, `schema_version`) is decoration the reader never touches. **Name all four for the story** — `trace-<id>.md`, `gate-decision-<id>.json`, and on the production path `nfr-assessment-<id>.md` and `test-review-<id>.md`. The first pair must be per-story because `--story <id>` scoping resolves them by name in a shared `--trace-output` dir. The second pair binds nothing mechanically (the `--nfr` / `--test-review` flags take an explicit path), which is exactly why it needs stating: unnamed, each story's NFR and review overwrite the last one's in that shared directory, and the audit trail keeps only whichever story finished most recently.
|
|
@@ -46,7 +56,7 @@ Write a PASS only when the story is demonstrably done to its profile's Definitio
|
|
|
46
56
|
4. **The red count recorded in the trace report, with a one-line justification for every case that was already green.** Where a shared acceptance file accumulates already-shipped siblings' cases, those green-by-design cases may be justified as one line naming that class rather than case-by-case — the per-case justification owed is for THIS story's cases that never went red.
|
|
47
57
|
5. **`gate-provenance: hand-authored`, logged per story** exactly as "Run the gate" below requires — a run that takes this branch on every row produces an Epic whose every verdict rests on a file the run wrote for itself, and nothing else in the artifacts would say so.
|
|
48
58
|
|
|
49
|
-
`bmad-testarch-nfr` and `bmad-testarch-test-review` are independent of that browser pair (see "Backfill the gate evidence" above), so still produce `nfr-assessment.md` and `test-review.md` the normal way and run `gate_eval.py --profile production --story <story_id> --nfr <…/nfr-assessment-<story_id>.md> --test-review <…/test-review-<story_id>.md>`. Pass **both** flags. `gate_eval.py` fail-closes a `--nfr`/`--test-review` path that is given-but-missing **and one you simply omit**: an omitted flag on a per-story production gate is itself a failing signal, with a reason naming the flag. That is deliberate — a forgotten flag used to be skipped in silence, so the AND never ran, the field rendered `null`, and the verdict was computed without it, which bought a *higher* verdict than supplying a failing artifact would have. If a signal genuinely cannot be produced on the stack, that is a CONCERNS/`defer` or a `reloop`, never a dropped flag. The one legitimate omission is the epic roll-up, which declares itself with `--epic-level` (see "Route on the verdict") rather than by leaving the flags off. A stack that cannot meet the production acceptance-test bar at all belongs under `--light` (see the framework fitness caveat in `references/preflight.md`), not a hand-waved production PASS. The honesty bar is unchanged: a PASS requires full AC coverage by passing tests, never to dodge an AND.
|
|
59
|
+
`bmad-testarch-nfr` and `bmad-testarch-test-review` are independent of that browser pair (see "Backfill the gate evidence" above), so still produce `nfr-assessment.md` and `test-review.md` the normal way and run `gate_eval.py --profile production --story <story_id> --nfr <…/nfr-assessment-<story_id>.md> --test-review <…/test-review-<story_id>.md> --tests-ran {workflow.implementation_artifacts}/.tests-ran-<story_id>`. Pass **both** signal flags (and `--tests-ran`, which every per-story gate carries — the sweep-scope AND binds this path unchanged). `gate_eval.py` fail-closes a `--nfr`/`--test-review` path that is given-but-missing **and one you simply omit**: an omitted flag on a per-story production gate is itself a failing signal, with a reason naming the flag. That is deliberate — a forgotten flag used to be skipped in silence, so the AND never ran, the field rendered `null`, and the verdict was computed without it, which bought a *higher* verdict than supplying a failing artifact would have. If a signal genuinely cannot be produced on the stack, that is a CONCERNS/`defer` or a `reloop`, never a dropped flag. The one legitimate omission is the epic roll-up, which declares itself with `--epic-level` (see "Route on the verdict") rather than by leaving the flags off. A stack that cannot meet the production acceptance-test bar at all belongs under `--light` (see the framework fitness caveat in `references/preflight.md`), not a hand-waved production PASS. The honesty bar is unchanged: a PASS requires full AC coverage by passing tests, never to dodge an AND.
|
|
50
60
|
|
|
51
61
|
**The epic-level roll-up needs no hand-authoring at all where sprint-status is available: run it with `--rollup`** (see "Route on the verdict"), which computes the aggregate from the per-story artifacts this section already had you author — no `trace-<epic>.md` required. Hand-author the epic pair only where the gate cannot be given sprint-status. **In that fallback the roll-up needs the same hand-authoring, named for the EPIC.** Everything above is written per story, and on this stack TEA never runs — so nothing ever writes the epic's own `trace-<epic>.md`, while the omit-`--rollup` form of the gate below presumes it exists: `--story <epic_id>` in a per-story-named directory resolves nothing and fail-closes a completed Epic to `escalate`. When every story is `done`, hand-author the same two files named for the epic id — `trace-<epic>.md`, its frontmatter pointing at `gate-decision-<epic>.json` — with a `gate_status` that honestly summarises the per-story record: a PASS only when every story advanced, never a roll-up dressed over a story that did not. It takes no NFR and no test-review; the roll-up declares that omission with `--epic-level`, per "Route on the verdict" below.
|
|
52
62
|
|
|
@@ -74,13 +84,13 @@ The score **must** carry the `/100` denominator: a bare `score: 89` matches noth
|
|
|
74
84
|
Production:
|
|
75
85
|
|
|
76
86
|
```
|
|
77
|
-
uv run {skill-root}/scripts/gate_eval.py --trace-output {workflow.trace_output_dir} --story <story_id> --profile production --nfr {nfr-assessment.md} --test-review {test-review.md}
|
|
87
|
+
uv run {skill-root}/scripts/gate_eval.py --trace-output {workflow.trace_output_dir} --story <story_id> --profile production --nfr {nfr-assessment.md} --test-review {test-review.md} --tests-ran {workflow.implementation_artifacts}/.tests-ran-<story_id>
|
|
78
88
|
```
|
|
79
89
|
|
|
80
90
|
Light:
|
|
81
91
|
|
|
82
92
|
```
|
|
83
|
-
uv run {skill-root}/scripts/gate_eval.py --trace-output {workflow.trace_output_dir} --story <story_id> --profile light
|
|
93
|
+
uv run {skill-root}/scripts/gate_eval.py --trace-output {workflow.trace_output_dir} --story <story_id> --profile light --tests-ran {workflow.implementation_artifacts}/.tests-ran-<story_id>
|
|
84
94
|
```
|
|
85
95
|
|
|
86
96
|
Resolve `{nfr-assessment.md}` and `{test-review.md}` to the paths TEA wrote them to (under `{workflow.trace_output_dir}` or the TEA output root); pass the production-only flags only in production. The script reads `gate-decision.json` (resolving its filename from the trace report frontmatter, falling back to the `e2e-trace-summary.json` gate fields when the slim file is absent — that fallback is **not** a failure).
|
|
@@ -96,7 +106,11 @@ Resolve `{nfr-assessment.md}` and `{test-review.md}` to the paths TEA wrote them
|
|
|
96
106
|
|
|
97
107
|
Do not recompute TEA thresholds or re-judge `gate_status` — read it as given. The script already ANDs the production signals (an `advance` is downgraded to `reloop` if `nfr-assessment.md` overallStatus is FAIL, or `test-review.md` score < 80 or recommendation is Block). Record the full verdict JSON and its `reasons` in `.decision-log.md` for this story.
|
|
98
108
|
|
|
99
|
-
**
|
|
109
|
+
**The sweep-scope AND (`--tests-ran`, per-story gates, both profiles).** Pass `--tests-ran {workflow.implementation_artifacts}/.tests-ran-<story_id>` on every per-story gate, exactly as the commands above show. The script reads the marker's `scope=` line (`references/execute.md` steps 2 and 5 write it beside `baseline=`): `scope=full` asserts the green run behind this gate was the full three-axis sweep and passes the verdict through; `scope=scoped` is legitimate mid-loop evidence, so the verdict is **capped at `reloop`** — a story may only advance off a full-scope sweep, and the cap moves `defer` too, because defer's route advances the story. Fail-closed like every other signal: a missing marker, a marker with no `scope=` line, or a value the reader does not recognise all get the same cap, with a reason naming what was wrong. The verdict JSON carries the recognised value as `sweep_scope` so an advance that proved its sweep is distinguishable from one that was never asked. Two boundaries: the epic roll-up gates no sweep of its own, so `--epic-level` refuses the flag outright (an invocation error, exit 2 — the same lane as `--epic-level` with `--nfr`); and *omitting* the flag skips the check rather than failing it, deliberately — making omission failing would change the verdict of documented invocations that predate the flag, the exact 2.0.0 lesson `docs/_internal/STABILITY.md` records — so the always-pass-it rule lives here, in the instruction, and a per-story gate invoked without the flag is a drift to fix, not a laxer mode to use.
|
|
110
|
+
|
|
111
|
+
**The delta cycle profile (`--cycle-profile`, mid-loop gates, production).** A gate whose previous evaluation of this story was `reloop` with findings cleanly attributed to their assessor dimension may re-run as a **delta gate**: re-run in full only the assessor(s) whose dimension produced the failing findings, while the others verify their own prior artifact against the diff from the previous gate's `assessed-commit` line to the commit under assessment (a concerns-only re-check; a previous verdict record without that line means the fallback below — run the full gate) instead of re-deriving the story from zero. Declare it — pass `--cycle-profile delta` — and the script caps the verdict at `reloop`: a delta gate may say "not yet, and here is why", never `advance` or `defer`. ADVANCE requires the full instrument (every assessor, over a `scope=full` sweep), exactly as before, so the oracle's finality is untouched, and a delta cycle whose findings all close costs one extra full gate — the confirming cycle a multi-cycle story pays anyway. The fallback is mechanical, not judgment: when any prior finding lacks a clean dimension attribution (which assessor artifact it came from), run the full gate. Pair it with the sweep split: mid-loop cycles ride `scope=scoped` sweeps and delta gates; the advance-bound cycle runs the full sweep and the full gate. (A measured seven-cycle story paid a flat ~20 minutes of full three-assessor gating per cycle regardless of fix size — a nine-minute confirming fix gated at the same cost as a 77-minute one.) Under `--light` the gate is already a single trace read, so the profile buys nothing there; omit it.
|
|
112
|
+
|
|
113
|
+
**Record who authored the artifacts the verdict rests on.** Alongside that JSON, log one line naming the provenance of this story's `trace-<id>.md` / `gate-decision-<id>.json`: **`gate-provenance: tea`** when `bmad-testarch-trace` wrote them, or **`gate-provenance: hand-authored`** when the non-web section above did. This is not bookkeeping. The module's central non-negotiable is that completion is decided by a script reading TEA's artifact rather than by the model's judgment, and on the hand-authored path the model *writes the file the gate then reads* — a defensible substitution the non-web section sanctions, but one that is otherwise **indistinguishable** in the decision log and the run report from a verdict grounded in a TEA-authored gate. An Epic can run to completion entirely on the hand-authored path, story after story, with nothing anywhere recording that it did. Provenance is what keeps "the JSON is the truth" honest about *whose* JSON it was; it changes no verdict and gates nothing. Beside it, log `assessed-commit: <sha>` — the commit the assessors measured — because the delta cycle profile's re-check computes its diff from the PREVIOUS gate's `assessed-commit` line, and a story whose record lacks one falls back to the full gate.
|
|
100
114
|
|
|
101
115
|
**Attended runs also print the verdict, once.** In an attended run, immediately after `gate_eval.py` returns and alongside the `.decision-log.md` record above, print **exactly one** transcript line for this story — `story 4-2 — gate_status PASS — verdict advance` — carrying the story id, the `gate_status`, and the routing verdict. The line fires at the moment the verdict is formed, not on the heartbeat cadence: **one line per story, per gate evaluation**, so a watching human sees every story's completion decision without opening a file, and a re-looped story prints its own line for its second evaluation rather than restating the first. Skip this print in headless (`-H`): there the artifacts and `run-result.json` are the interface, and transcript prose has no reader. The line is **additive to the `.decision-log.md` record and never a substitute for it** — the recorded JSON is the completion evidence, a transcript line is not, and a reader who treats the line as the record has lost the `reasons`.
|
|
102
116
|
|
|
@@ -114,7 +128,7 @@ uv run {skill-root}/scripts/gate_eval.py --trace-output {workflow.trace_output_d
|
|
|
114
128
|
|
|
115
129
|
- **`defer`** (gate_status CONCERNS, or non-critical code-review / NFR findings that did not flip the gate) → append the open items to the ledger at `{workflow.deferred_work_path}` using the schema below, then **advance** anyway. The Epic keeps moving; the parked work is visible.
|
|
116
130
|
|
|
117
|
-
- **`reloop`** (gate_status FAIL, or a production signal downgraded an advance) → run `bmad-correct-course` to diagnose and adjust, then re-run the story (back through Execute, `references/execute.md`) — **within the remaining turn budget**. Re-run the gate after. If the re-loop would exceed `{workflow.max_turns_per_story}`, treat it as `escalate` instead.
|
|
131
|
+
- **`reloop`** (gate_status FAIL, or a production signal downgraded an advance, or a cap — sweep-scope or delta-profile) → **generalize before fixing.** Before `bmad-correct-course` writes any fix, take the verdict's `reasons` and the review's findings and, for each finding: (1) name, in ONE line, the defect CLASS the finding is an instance of; (2) state whether any earlier review of this story or this Epic already found that class — the previous gate artifacts and `{workflow.implementation_artifacts}/.class-ledger-<epic_id>.md` are the evidence, and they are already on disk, so this is a read, not a judgment call; (3) where the class is a repeat, the fix must close the CLASS — every sibling instance the same seam admits, not the one instance the reviewer demonstrated — and the log records why the class-level fix was not taken when the class was first named. A measured run spent five of six reviews re-finding one class ("an assertion that cannot go red") one seam over each time, with the class-killing move available from day one; the generalize step is what makes cycle N read cycle N-1 instead of re-buying it. Record one `class:` line per finding in this re-loop's `.decision-log.md` entry, and append any class not already present to `.class-ledger-<epic_id>.md` (create on first use; named for the Epic, so a later Epic starts a fresh ledger while this Epic's survives across its own invocations for exactly the resume case; one line per class, `- <class, one line> — first named <story_id>, cycle <n>`; **append-only** — Execute step 1 reads it before every story and re-loop, so a class named once is warned about everywhere after, including across stories, which no per-story artifact can do). **A score that does not move indicts the FIX STRATEGY, not the code:** when two consecutive gate evaluations of this story have moved `review_score` by less than 3 (production — under `--light` no review score exists, so the trigger there is two consecutive re-loops whose findings name any class already in the ledger), the generalize answers above are mandatory *in writing* before any further fix — a re-loop whose findings all name the same class as the previous re-loop's is a re-loop about to be spent badly. **A cap-only re-loop skips the ceremony.** When a cap alone produced the verdict — the `reasons` show only the sweep-scope cap, the delta-profile cap, or both, with no gate FAIL and no failed production signal — there is no finding to generalize and nothing to diagnose: re-run the full three-axis sweep where the sweep cap fired and refresh the marker with `scope=full`, then re-run the gate as the FULL instrument — **dropping `--cycle-profile`**, because the delta cap is lifted by the flag's absence and a re-gate that keeps it caps again forever — writing no `class:` lines, appending nothing to the ledger, and skipping `bmad-correct-course` entirely; such evaluations do not count toward the flat-score tripwire, and a capped CONCERNS keeps its findings for the next gate's own `defer` route — parking them is that gate's job, not this re-loop's. Otherwise, run `bmad-correct-course` to diagnose and adjust, then re-run the story (back through Execute, `references/execute.md`) — **within the remaining turn budget**. Re-run the gate after. If the re-loop would exceed `{workflow.max_turns_per_story}`, treat it as `escalate` instead.
|
|
118
132
|
|
|
119
133
|
- **`escalate`** (gate_status NOT_EVALUATED — the gate could not be read — or budget exhausted on a FAIL) → **stop.** Do not advance, do not defer the failing item. Record the reason and the verdict JSON in `.decision-log.md`, and write the typed escalation sidecar `{workflow.implementation_artifacts}/escalation-<story_id>.json` (shape in `references/execute.md`, under "Escalation sidecar") so the pending decision is readable without the transcript. In an attended run, surface the blocker to the user. In headless, this is a `blocked` outcome — emit the JSON in Stage 6 (`references/finalize.md`).
|
|
120
134
|
|
|
@@ -138,7 +138,7 @@ Pass `impl_artifacts={workflow.implementation_artifacts}` to that call: `scripts
|
|
|
138
138
|
Do these in order; each must be asserted, not assumed:
|
|
139
139
|
|
|
140
140
|
- **Epic branch.** Assert you are on `{workflow.epic_branch_prefix}<epic-id>`. When step 1 reported `git_branch`, step 2's remediation already created it, and this is a **re-assertion, not a rebuild** — the same treatment the resume path gives it. Create it here when it does not yet exist — and note that `budget == 0` does **not** by itself put you on it: `git_branch` fires on a protected branch and nothing else, so a run started on some unrelated feature branch clears the gate having never been asked about its branch, and step 2 has no bullet that fires for it. That case lands here, so resolve it here rather than routing back: check out the base this Epic should branch from (a `{workflow.protected_branches}` branch, per execute.md's "off a protected branch" precondition — or the base the operator named in Stage 1) and cut `{workflow.epic_branch_prefix}<epic-id>` from there. Do **not** branch from wherever HEAD happens to sit; an Epic branch cut off an unrelated feature branch carries that branch's uncommitted work into the run's history and makes the branch-deletion rollback destructive. Rollback for this run is git (per-story commits) — `/rewind` checkpoints miss Bash-driven changes, so the branch is the real undo. If the tree is dirty (`git_clean: false`), resolve before branching — the last budget re-run was at the end of step 2, and step 3 has since written its scan sidecar and appended to `.decision-log.md`, so `budget == 0` does **not** imply a clean tree by the time you arrive here (`git_clean` counts untracked files).
|
|
141
|
-
- **Purge stale run markers.** Delete any `{workflow.implementation_artifacts}/.escalation-*`, `{workflow.implementation_artifacts}/.on-escalation-fired-*` and `{workflow.implementation_artifacts}/.
|
|
141
|
+
- **Purge stale run markers.** Delete any `{workflow.implementation_artifacts}/.escalation-*`, `{workflow.implementation_artifacts}/.on-escalation-fired-*`, `{workflow.implementation_artifacts}/.budget-*`, `{workflow.implementation_artifacts}/.heartbeat-*` and `{workflow.implementation_artifacts}/.stall-*` an earlier run left behind, before the hooks that write them are armed — a stale escalation marker makes this run look like it already escalated, a stale fired-marker silently swallows this run's first escalation ping, which is the one an absent operator is waiting on, and a stale turn counter is already at or past the ceiling, so the Stop hook escalates on this run's **first** Stop event before it has done a single turn of work — and a stale heartbeat's gap since the PRIOR run would fire this run's stall rule on its first resume about an idle nobody had. Those five dot-prefixed families only. **On a resume, purge nothing.** A resume re-asserts the environment rather than rebuilding it, and the turn counter is the one piece of that environment a resume must inherit: zeroing `.budget-*` on re-entry would hand an in-flight story a fresh ceiling every time it stopped, so a run could evade the budget indefinitely by resuming. A counter the operator has deliberately cleared is a different thing, and `/ucg-resolve` clears exactly that one when it closes a `budget-overrun` escalation. In particular do **not** delete `escalation-*.json`: the typed sidecar is an operator-facing artifact like `run-status.json` and `run-result.json`, neither of which is purged at arming either, and it is named per story rather than per run, so arming cannot tell a deliberately-pending sidecar from a stale one. A decision the operator chose to leave pending lives in exactly that file, and a prior run's unresolved escalation surfacing at `/ucg-resolve` is correct: it genuinely is still pending. Sidecar staleness is handled at read time instead, by the run-scoped marker scan in `references/execute.md`.
|
|
142
142
|
- **Hooks.** Idempotently merge the **PreToolUse** guard (`scripts/hooks/guard_pretooluse.py`) and the **Stop** budget hook (`scripts/hooks/budget_stop.py`) into `{project-root}/.claude/settings.local.json` (gitignored, machine-local, honored after the workspace trust dialog). Re-merge every run — do not assume a prior run left them.
|
|
143
143
|
|
|
144
144
|
**Arm the PreToolUse group with a matcher that covers EVERY tool, not just `Bash`.** Use `"matcher": "*"`, or omit the `matcher` key entirely the way the `Stop` group already does. Five of the guard's six invariants are Bash/git, so `Bash` is the natural choice and it is the choice both known installs made — but the sixth, the Cross-Session Recall latch, exists to gate an **MCP tool call**. Claude Code only invokes a PreToolUse group for tools its matcher selects, so under `Bash` the hook is never run for `mcp__plugin_claude-mem_*__*` at all: the guard would deny that call correctly if it were asked, and it is never asked. The invariant then enforces nothing while every log line says the hooks are armed, which is the worst of both — a control that reports itself active and is not.
|
|
@@ -833,6 +833,88 @@ def apply_production_and(
|
|
|
833
833
|
return verdict, nfr_status, review_score
|
|
834
834
|
|
|
835
835
|
|
|
836
|
+
# The marker's scope line, written by execute.md steps 2/5 beside baseline=.
|
|
837
|
+
_SWEEP_SCOPE_RE = re.compile(r"^scope=(\S+)\s*$", re.MULTILINE)
|
|
838
|
+
|
|
839
|
+
|
|
840
|
+
def apply_sweep_scope(
|
|
841
|
+
verdict: str, tests_ran_path: Path, reasons: list[str]
|
|
842
|
+
) -> tuple[str, str | None]:
|
|
843
|
+
"""Cap a story-moving verdict at reloop unless the sweep behind it was full.
|
|
844
|
+
|
|
845
|
+
Reads the story's ``.tests-ran-<story_id>`` marker for its
|
|
846
|
+
``scope=<full|scoped>`` line. ``full`` passes the verdict through;
|
|
847
|
+
``scoped`` is legitimate mid-loop evidence whose verdict is capped at
|
|
848
|
+
reloop - a scoped sweep must not impersonate a full one at the one place
|
|
849
|
+
the distinction pays. Unlike the production AND's floor, the cap moves
|
|
850
|
+
``defer`` as well as ``advance``, because defer's route ADVANCES the story
|
|
851
|
+
(ledger, then advance anyway), and a story moving forward off mid-loop
|
|
852
|
+
evidence is exactly what the cap exists to refuse.
|
|
853
|
+
|
|
854
|
+
FAIL-CLOSED like every other signal: a missing or unreadable marker, a
|
|
855
|
+
marker with no ``scope=`` line, and a value this reader does not recognise
|
|
856
|
+
all get the same cap, with a reason naming what was wrong. Only the OMITTED
|
|
857
|
+
FLAG is a no-op, and that is deliberate rather than a soft spot: making
|
|
858
|
+
omission failing would change the verdict of documented invocations that
|
|
859
|
+
predate the flag (docs/_internal/STABILITY.md carries the 2.0.0 lesson on
|
|
860
|
+
exactly that), so the always-pass-it rule is the instruction layer's to
|
|
861
|
+
state, not this reader's to retrofit.
|
|
862
|
+
|
|
863
|
+
Returns ``(verdict, sweep_scope)`` where ``sweep_scope`` is the recognised
|
|
864
|
+
value read from the marker, or None when none was.
|
|
865
|
+
"""
|
|
866
|
+
scope: str | None = None
|
|
867
|
+
problem: str | None = None
|
|
868
|
+
text = _read_text(tests_ran_path) if tests_ran_path.is_file() else None
|
|
869
|
+
if text is None:
|
|
870
|
+
problem = f"tests-ran marker {tests_ran_path} missing or unreadable"
|
|
871
|
+
else:
|
|
872
|
+
values = _SWEEP_SCOPE_RE.findall(text)
|
|
873
|
+
if not values:
|
|
874
|
+
problem = (
|
|
875
|
+
f"tests-ran marker {tests_ran_path.name} carries no scope= line "
|
|
876
|
+
"(execute.md step 2 writes scope=full|scoped beside baseline=)"
|
|
877
|
+
)
|
|
878
|
+
elif len(set(values)) > 1:
|
|
879
|
+
# Disagreeing duplicates are a marker that was APPENDED to rather
|
|
880
|
+
# than rewritten - and picking either line would be a guess. Taking
|
|
881
|
+
# the first is the trap specifically: a re-loop cycle appending its
|
|
882
|
+
# scoped line under a first pass's scope=full would read as full,
|
|
883
|
+
# which is the one impersonation this reader exists to refuse.
|
|
884
|
+
problem = (
|
|
885
|
+
f"tests-ran marker {tests_ran_path.name} carries "
|
|
886
|
+
f"{len(values)} disagreeing scope= lines ({', '.join(values)}); "
|
|
887
|
+
"rewrite the marker with exactly one"
|
|
888
|
+
)
|
|
889
|
+
elif values[0] not in ("full", "scoped"):
|
|
890
|
+
problem = (
|
|
891
|
+
f"tests-ran marker {tests_ran_path.name} carries unrecognised "
|
|
892
|
+
f"scope={values[0]!r}"
|
|
893
|
+
)
|
|
894
|
+
else:
|
|
895
|
+
scope = values[0]
|
|
896
|
+
|
|
897
|
+
if scope == "full":
|
|
898
|
+
return verdict, scope
|
|
899
|
+
if verdict in ("advance", "defer"):
|
|
900
|
+
reasons.append(
|
|
901
|
+
f"{problem}; treated as failing" if problem is not None else "sweep scope is scoped"
|
|
902
|
+
)
|
|
903
|
+
reasons.append(
|
|
904
|
+
f"{verdict} capped at reloop: a story only advances off a full-scope "
|
|
905
|
+
"sweep - re-run the full three-axis sweep, refresh the marker with "
|
|
906
|
+
"scope=full, and re-run the gate"
|
|
907
|
+
)
|
|
908
|
+
verdict = "reloop"
|
|
909
|
+
elif problem is not None:
|
|
910
|
+
# A verdict already at reloop/escalate is not moved, but the marker
|
|
911
|
+
# problem still gets a line - the re-loop should fix the marker too.
|
|
912
|
+
reasons.append(f"{problem} (verdict already {verdict}; not moved)")
|
|
913
|
+
else:
|
|
914
|
+
reasons.append(f"sweep scope scoped (verdict already {verdict}; not moved)")
|
|
915
|
+
return verdict, scope
|
|
916
|
+
|
|
917
|
+
|
|
836
918
|
# Severity order for the roll-up aggregate: the epic's status is its WORST
|
|
837
919
|
# story's status, and an unknown status is treated as the worst there is.
|
|
838
920
|
_ROLLUP_SEVERITY = ("PASS", "WAIVED", "CONCERNS", "FAIL", "NOT_EVALUATED")
|
|
@@ -984,7 +1066,30 @@ def evaluate(args: argparse.Namespace) -> dict:
|
|
|
984
1066
|
verdict, nfr_path, review_path, reasons
|
|
985
1067
|
)
|
|
986
1068
|
|
|
987
|
-
|
|
1069
|
+
# `is not None`, never truthiness: an explicitly-passed empty value is a
|
|
1070
|
+
# caller error to fail closed on (Path("") is never a file, so it caps),
|
|
1071
|
+
# not a silent way back to the pre-flag behaviour. Only true omission skips.
|
|
1072
|
+
tests_ran = getattr(args, "tests_ran", None)
|
|
1073
|
+
sweep_scope: str | None = None
|
|
1074
|
+
if tests_ran is not None:
|
|
1075
|
+
verdict, sweep_scope = apply_sweep_scope(verdict, Path(tests_ran), reasons)
|
|
1076
|
+
|
|
1077
|
+
# The delta cycle profile: a mid-loop gate that re-ran only the previously
|
|
1078
|
+
# failed assessor dimension(s) declares itself and is capped exactly as a
|
|
1079
|
+
# scoped sweep is - it may say "not yet, and here is why", never advance.
|
|
1080
|
+
# defer moves too, for the same reason as the sweep cap: defer's route
|
|
1081
|
+
# advances the story. The choices list means an unrecognised value is an
|
|
1082
|
+
# argparse error (exit 2), never a silently-full gate.
|
|
1083
|
+
cycle_profile = getattr(args, "cycle_profile", None)
|
|
1084
|
+
if cycle_profile == "delta" and verdict in ("advance", "defer"):
|
|
1085
|
+
reasons.append(
|
|
1086
|
+
f"{verdict} capped at reloop: a delta-profile gate re-ran only the "
|
|
1087
|
+
"previously failed dimension(s), so it may not move the story - "
|
|
1088
|
+
"run the full gate (every assessor, over a scope=full sweep) to advance"
|
|
1089
|
+
)
|
|
1090
|
+
verdict = "reloop"
|
|
1091
|
+
|
|
1092
|
+
result = {
|
|
988
1093
|
"verdict": verdict,
|
|
989
1094
|
"gate_status": gate_status,
|
|
990
1095
|
"p0_status": gate["p0_status"],
|
|
@@ -997,8 +1102,22 @@ def evaluate(args: argparse.Namespace) -> dict:
|
|
|
997
1102
|
# an advance that ANDed both production signals from one that skipped
|
|
998
1103
|
# them, and `nfr_status: null` alone does not say which.
|
|
999
1104
|
"epic_level": bool(getattr(args, "epic_level", False)),
|
|
1000
|
-
"reasons": reasons,
|
|
1001
1105
|
}
|
|
1106
|
+
if tests_ran is not None:
|
|
1107
|
+
# Present exactly when --tests-ran was supplied, absent otherwise - so
|
|
1108
|
+
# the printed shape of every pre-flag invocation is unchanged (the
|
|
1109
|
+
# stability policy classes a new key in a documented invocation's JSON
|
|
1110
|
+
# as a shape change; a caller only sees this one by opting in). The
|
|
1111
|
+
# value is the recognised scope= reading, or null when none was.
|
|
1112
|
+
result["sweep_scope"] = sweep_scope
|
|
1113
|
+
if cycle_profile is not None:
|
|
1114
|
+
# Same conditional-presence contract as sweep_scope, same reason.
|
|
1115
|
+
result["cycle_profile"] = cycle_profile
|
|
1116
|
+
# reasons goes last, after any conditional keys, so the printed object
|
|
1117
|
+
# reads verdict-first, evidence-last exactly as the documented examples
|
|
1118
|
+
# show. Key ORDER is cosmetic; the contract is the key SET.
|
|
1119
|
+
result["reasons"] = reasons
|
|
1120
|
+
return result
|
|
1002
1121
|
|
|
1003
1122
|
|
|
1004
1123
|
def main(argv: list[str] | None = None) -> int:
|
|
@@ -1026,6 +1145,29 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1026
1145
|
"value passed here instead of a path is read as a missing file and "
|
|
1027
1146
|
"treated as failing.",
|
|
1028
1147
|
)
|
|
1148
|
+
parser.add_argument(
|
|
1149
|
+
"--tests-ran",
|
|
1150
|
+
help="PATH to the story's .tests-ran-<story_id> marker (per-story gates "
|
|
1151
|
+
"only). The marker must carry a scope=<full|scoped> line naming which "
|
|
1152
|
+
"sweep form its green run was (execute.md steps 2/5 write it beside "
|
|
1153
|
+
"baseline=). scope=full passes the verdict through; scope=scoped - and, "
|
|
1154
|
+
"fail-closed, a missing marker, a missing scope= line, or an "
|
|
1155
|
+
"unrecognised value - caps advance/defer at reloop. Omitting the flag "
|
|
1156
|
+
"skips the check (invocations that predate the flag keep their "
|
|
1157
|
+
"verdicts); the skill instructions require it on every per-story gate. "
|
|
1158
|
+
"Cannot be combined with --epic-level.",
|
|
1159
|
+
)
|
|
1160
|
+
parser.add_argument(
|
|
1161
|
+
"--cycle-profile",
|
|
1162
|
+
choices=["full", "delta"],
|
|
1163
|
+
help="Which gate cycle this is (per-story gates only). 'delta' declares "
|
|
1164
|
+
"a mid-loop gate that re-ran in full only the previously failed "
|
|
1165
|
+
"assessor dimension(s), the others re-checking their prior artifact "
|
|
1166
|
+
"against the diff: its verdict is capped at reloop - advance/defer "
|
|
1167
|
+
"require the full gate. Omitting the flag is the full behaviour; the "
|
|
1168
|
+
"JSON carries cycle_profile exactly when the flag was supplied. Cannot "
|
|
1169
|
+
"be combined with --epic-level.",
|
|
1170
|
+
)
|
|
1029
1171
|
parser.add_argument(
|
|
1030
1172
|
"--epic-level",
|
|
1031
1173
|
action="store_true",
|
|
@@ -1070,6 +1212,28 @@ def main(argv: list[str] | None = None) -> int:
|
|
|
1070
1212
|
"per-story gate, or drop the paths for the epic roll-up"
|
|
1071
1213
|
)
|
|
1072
1214
|
|
|
1215
|
+
# Same lane as the hybrid above: the epic roll-up gates no sweep of its own
|
|
1216
|
+
# (every story proved its own scope before reaching `done`), so an
|
|
1217
|
+
# epic-level --tests-ran is a contradiction to refuse, never to evaluate.
|
|
1218
|
+
# `is not None`, not truthiness: --epic-level --tests-ran "" is the same
|
|
1219
|
+
# contradiction spelled with an empty value, not a way around the refusal.
|
|
1220
|
+
if args.epic_level and args.tests_ran is not None:
|
|
1221
|
+
parser.error(
|
|
1222
|
+
"--epic-level is the epic roll-up: it gates no sweep of its own, so "
|
|
1223
|
+
"it cannot be combined with --tests-ran; drop the flag for a "
|
|
1224
|
+
"per-story gate, or drop the path for the epic roll-up"
|
|
1225
|
+
)
|
|
1226
|
+
|
|
1227
|
+
# The roll-up is by definition the final full instrument over the per-story
|
|
1228
|
+
# record - a delta roll-up is a contradiction, and even an explicit 'full'
|
|
1229
|
+
# is per-story vocabulary the roll-up does not speak. Same refusal lane.
|
|
1230
|
+
if args.epic_level and args.cycle_profile is not None:
|
|
1231
|
+
parser.error(
|
|
1232
|
+
"--epic-level is the epic roll-up, always the full instrument: it "
|
|
1233
|
+
"cannot be combined with --cycle-profile; drop the flag for a "
|
|
1234
|
+
"per-story gate"
|
|
1235
|
+
)
|
|
1236
|
+
|
|
1073
1237
|
result = evaluate(args)
|
|
1074
1238
|
print(json.dumps(result, indent=2))
|
|
1075
1239
|
return 0
|
|
@@ -0,0 +1,353 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Drive a table of mutations against a green baseline, and prove each one red.
|
|
3
|
+
|
|
4
|
+
The module's anti-vacuity discipline says a guard is non-vacuous only under
|
|
5
|
+
mutation. Every run used to hand-roll the harness for that into scratch space
|
|
6
|
+
(and delete it at story end, so the next story hand-rolled it again). This is
|
|
7
|
+
the tracked form: a table in, a ledger out, with the properties the hand-rolled
|
|
8
|
+
versions kept losing built in as refusals rather than reminders.
|
|
9
|
+
|
|
10
|
+
Table (JSON array; every key a string, all five required, ``note`` optional):
|
|
11
|
+
|
|
12
|
+
[{"file": "src/guard.py",
|
|
13
|
+
"anchor": "if depth > MAX_DEPTH:",
|
|
14
|
+
"replacement": "if False:",
|
|
15
|
+
"command": "npx turbo run test --force --filter=guard",
|
|
16
|
+
"expect_red": "test_depth_guard_fires",
|
|
17
|
+
"note": "AC-3 twin"}]
|
|
18
|
+
|
|
19
|
+
Per row, in order: assert the anchor occurs EXACTLY once (0 occurrences is
|
|
20
|
+
reported distinctly as STALE - the formatter-reflow signature - and more than
|
|
21
|
+
one as NOT UNIQUE, because a duplicated anchor mutates a different site than it
|
|
22
|
+
reports); back the file up; apply; run the command; classify; restore from the
|
|
23
|
+
backup and verify the restore BYTE-IDENTICAL. Before any row: run every
|
|
24
|
+
distinct command once and refuse the whole table unless all are green, because
|
|
25
|
+
a mutation over a red baseline proves nothing.
|
|
26
|
+
|
|
27
|
+
Classification, per row:
|
|
28
|
+
|
|
29
|
+
- GREEN the command stayed green: the mutation SURVIVED. This is
|
|
30
|
+
the output that matters - a guard that cannot fail - so
|
|
31
|
+
the ledger leads with these.
|
|
32
|
+
- KILLED red, the expected case named in the output, and exactly
|
|
33
|
+
one failure counted.
|
|
34
|
+
- KILLED+others red and named, but more than one case failed (or the
|
|
35
|
+
count could not be parsed): the kill stands, the extra
|
|
36
|
+
reds are surfaced rather than silently absorbed.
|
|
37
|
+
- MISATTRIBUTED red, but the expected case never appears beside a
|
|
38
|
+
failure token (or the output's own count says 0 failed):
|
|
39
|
+
something failed, and it was not the observer this row
|
|
40
|
+
claims to test. The name must sit on a line that speaks
|
|
41
|
+
of failure, because verbose reporters echo PASSING names.
|
|
42
|
+
- STALE / NOT UNIQUE / UNDECODABLE the anchor (or the file) did not hold;
|
|
43
|
+
nothing was mutated for the row.
|
|
44
|
+
|
|
45
|
+
Exit code: 0 when every row KILLED (with or without +others); 1 when any row
|
|
46
|
+
is GREEN, MISATTRIBUTED, STALE, NOT UNIQUE or UNDECODABLE (each is a finding
|
|
47
|
+
about the suite or the table); 2 on a refusal - invalid table, red baseline, or a
|
|
48
|
+
restore that did not verify byte-identical (the one failure mode that leaves
|
|
49
|
+
the tree dirty, so it aborts the run rather than annotating it).
|
|
50
|
+
|
|
51
|
+
The task-runner cache rule binds the table's ``command`` verbatim: an
|
|
52
|
+
unchanged task hash replays the pre-mutation green, so every mutation
|
|
53
|
+
"survives" with zero failing cases. Defeat the cache in the command itself
|
|
54
|
+
(``--force`` and friends); this runner cannot do it for you.
|
|
55
|
+
"""
|
|
56
|
+
|
|
57
|
+
from __future__ import annotations
|
|
58
|
+
|
|
59
|
+
import argparse
|
|
60
|
+
import json
|
|
61
|
+
import re
|
|
62
|
+
import signal
|
|
63
|
+
import subprocess
|
|
64
|
+
import sys
|
|
65
|
+
from pathlib import Path
|
|
66
|
+
|
|
67
|
+
REQUIRED_KEYS = ("file", "anchor", "replacement", "command", "expect_red")
|
|
68
|
+
|
|
69
|
+
|
|
70
|
+
def _refuse(message: str) -> "SystemExit":
|
|
71
|
+
"""Exit 2, the refusal lane. `raise SystemExit(str)` would exit 1 and be
|
|
72
|
+
indistinguishable from a findings run; the docstring promises 2."""
|
|
73
|
+
print(f"REFUSED: {message}", file=sys.stderr)
|
|
74
|
+
return SystemExit(2)
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
def _terminated(signum, _frame):
|
|
78
|
+
raise SystemExit(128 + signum)
|
|
79
|
+
|
|
80
|
+
|
|
81
|
+
def _install_signal_unwind() -> None:
|
|
82
|
+
"""Convert termination signals into SystemExit so `finally` restores run.
|
|
83
|
+
|
|
84
|
+
SIGTERM's default disposition kills the interpreter WITHOUT unwinding, so
|
|
85
|
+
the finally block that restores a mutated file never executes - and TERM
|
|
86
|
+
is the exact kill a harness timeout delivers mid-command. SIGKILL cannot
|
|
87
|
+
be caught: after any abnormal end, `git status` before trusting the tree.
|
|
88
|
+
"""
|
|
89
|
+
for name in ("SIGTERM", "SIGHUP", "SIGBREAK"):
|
|
90
|
+
sig = getattr(signal, name, None)
|
|
91
|
+
if sig is None:
|
|
92
|
+
continue
|
|
93
|
+
try:
|
|
94
|
+
signal.signal(sig, _terminated)
|
|
95
|
+
except (ValueError, OSError):
|
|
96
|
+
continue
|
|
97
|
+
|
|
98
|
+
# Failure-count shapes across the common runners (pytest, vitest/jest, mocha,
|
|
99
|
+
# cargo, go). First match wins; no match means the count is unparseable and the
|
|
100
|
+
# kill is reported without the exactly-one claim.
|
|
101
|
+
_FAIL_COUNT_RES = (
|
|
102
|
+
re.compile(r"(\d+)\s+failed", re.IGNORECASE),
|
|
103
|
+
re.compile(r"failures[:=]\s*(\d+)", re.IGNORECASE),
|
|
104
|
+
re.compile(r"(\d+)\s+failing", re.IGNORECASE),
|
|
105
|
+
)
|
|
106
|
+
|
|
107
|
+
|
|
108
|
+
def _load_table(path: Path) -> list[dict]:
|
|
109
|
+
try:
|
|
110
|
+
raw = json.loads(path.read_text(encoding="utf-8"))
|
|
111
|
+
except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc:
|
|
112
|
+
raise _refuse(f"table {path} unreadable or not JSON: {exc}")
|
|
113
|
+
if not isinstance(raw, list) or not raw:
|
|
114
|
+
raise _refuse(f"table {path} must be a non-empty JSON array of rows")
|
|
115
|
+
for index, row in enumerate(raw):
|
|
116
|
+
if not isinstance(row, dict):
|
|
117
|
+
raise _refuse(f"row {index} is not an object")
|
|
118
|
+
for key in REQUIRED_KEYS:
|
|
119
|
+
if not isinstance(row.get(key), str) or not row[key]:
|
|
120
|
+
raise _refuse(f"row {index} is missing required string key {key!r}")
|
|
121
|
+
if row["anchor"] == row["replacement"]:
|
|
122
|
+
raise _refuse(
|
|
123
|
+
f"row {index} anchor and replacement are identical - "
|
|
124
|
+
"that mutation mutates nothing"
|
|
125
|
+
)
|
|
126
|
+
return raw
|
|
127
|
+
|
|
128
|
+
|
|
129
|
+
def _run(command: str, cwd: Path) -> tuple[int, str]:
|
|
130
|
+
proc = subprocess.run(
|
|
131
|
+
command,
|
|
132
|
+
shell=True,
|
|
133
|
+
cwd=cwd,
|
|
134
|
+
capture_output=True,
|
|
135
|
+
# utf-8 + replace, never the strict locale codec: a failing assertion
|
|
136
|
+
# that dumps raw bytes must classify as a red, not crash the run - and
|
|
137
|
+
# on Windows the locale default (cp1252) also mojibakes the very names
|
|
138
|
+
# attribution matches on.
|
|
139
|
+
encoding="utf-8",
|
|
140
|
+
errors="replace",
|
|
141
|
+
check=False,
|
|
142
|
+
)
|
|
143
|
+
return proc.returncode, (proc.stdout or "") + (proc.stderr or "")
|
|
144
|
+
|
|
145
|
+
|
|
146
|
+
def _fail_count(output: str) -> int | None:
|
|
147
|
+
for pattern in _FAIL_COUNT_RES:
|
|
148
|
+
match = pattern.search(output)
|
|
149
|
+
if match:
|
|
150
|
+
return int(match.group(1))
|
|
151
|
+
return None
|
|
152
|
+
|
|
153
|
+
|
|
154
|
+
_FAIL_TOKEN_RE = re.compile(r"fail|error|not ok|assert|✗|✘", re.IGNORECASE)
|
|
155
|
+
|
|
156
|
+
|
|
157
|
+
def _named_in_failure_context(output: str, expect_red: str) -> bool:
|
|
158
|
+
"""True when the expected case is named on a line that speaks of failure.
|
|
159
|
+
|
|
160
|
+
A whole-output substring match is defeated by any verbose reporter that
|
|
161
|
+
echoes PASSING names (`pytest -v` prints every test), which made a wrongly
|
|
162
|
+
attributed kill read as a clean one. So the name only counts on a line
|
|
163
|
+
that also carries a failure token. Heuristic, and documented as one: a
|
|
164
|
+
runner whose failure lines carry none of the tokens reads MISATTRIBUTED,
|
|
165
|
+
which fails toward the honest side - a human looks.
|
|
166
|
+
"""
|
|
167
|
+
return any(
|
|
168
|
+
expect_red in line and _FAIL_TOKEN_RE.search(line)
|
|
169
|
+
for line in output.splitlines()
|
|
170
|
+
)
|
|
171
|
+
|
|
172
|
+
|
|
173
|
+
def _classify_red(output: str, expect_red: str) -> tuple[str, str, bool]:
|
|
174
|
+
"""(status, detail, is_finding) for a red run."""
|
|
175
|
+
if not _named_in_failure_context(output, expect_red):
|
|
176
|
+
return (
|
|
177
|
+
"MISATTRIBUTED",
|
|
178
|
+
f"red, but {expect_red!r} never appears beside a failure token - "
|
|
179
|
+
"whatever failed, it was not the observer this row claims",
|
|
180
|
+
True,
|
|
181
|
+
)
|
|
182
|
+
count = _fail_count(output)
|
|
183
|
+
if count == 1:
|
|
184
|
+
return "KILLED", f"exactly {expect_red!r} went red", False
|
|
185
|
+
if count == 0:
|
|
186
|
+
# A red exit whose own summary counts zero failures is a collection or
|
|
187
|
+
# harness error wearing a red exit code, not the observer firing.
|
|
188
|
+
return (
|
|
189
|
+
"MISATTRIBUTED",
|
|
190
|
+
"red, but the output's own count says 0 failed - a collection or "
|
|
191
|
+
"harness error, not the observer",
|
|
192
|
+
True,
|
|
193
|
+
)
|
|
194
|
+
if count is None:
|
|
195
|
+
return "KILLED+others", f"{expect_red!r} went red; failure count unparseable", False
|
|
196
|
+
return "KILLED+others", f"{expect_red!r} went red among {count} failures", False
|
|
197
|
+
|
|
198
|
+
|
|
199
|
+
def run_table(table: list[dict], project_root: Path) -> tuple[list[dict], int]:
|
|
200
|
+
"""Execute the table; return (ledger rows, exit code)."""
|
|
201
|
+
# Baseline first, each distinct command once, in first-appearance order.
|
|
202
|
+
seen: list[str] = []
|
|
203
|
+
for row in table:
|
|
204
|
+
if row["command"] not in seen:
|
|
205
|
+
seen.append(row["command"])
|
|
206
|
+
for command in seen:
|
|
207
|
+
code, output = _run(command, project_root)
|
|
208
|
+
if code != 0:
|
|
209
|
+
raise _refuse(
|
|
210
|
+
"baseline is not green for command "
|
|
211
|
+
f"{command!r} (exit {code}) - a mutation over a red baseline "
|
|
212
|
+
f"proves nothing. Last lines:\n{output[-2000:]}"
|
|
213
|
+
)
|
|
214
|
+
|
|
215
|
+
rows: list[dict] = []
|
|
216
|
+
findings = False
|
|
217
|
+
for index, row in enumerate(table):
|
|
218
|
+
target = project_root / row["file"]
|
|
219
|
+
entry = {
|
|
220
|
+
"index": index,
|
|
221
|
+
"file": row["file"],
|
|
222
|
+
"expect_red": row["expect_red"],
|
|
223
|
+
"note": row.get("note", ""),
|
|
224
|
+
"restore": "n/a",
|
|
225
|
+
}
|
|
226
|
+
try:
|
|
227
|
+
original = target.read_bytes()
|
|
228
|
+
except OSError as exc:
|
|
229
|
+
entry.update(status="STALE", detail=f"file unreadable: {exc}")
|
|
230
|
+
rows.append(entry)
|
|
231
|
+
findings = True
|
|
232
|
+
continue
|
|
233
|
+
try:
|
|
234
|
+
text = original.decode("utf-8")
|
|
235
|
+
except UnicodeDecodeError as exc:
|
|
236
|
+
entry.update(status="UNDECODABLE", detail=f"file is not UTF-8: {exc}")
|
|
237
|
+
rows.append(entry)
|
|
238
|
+
findings = True
|
|
239
|
+
continue
|
|
240
|
+
occurrences = text.count(row["anchor"])
|
|
241
|
+
if occurrences != 1:
|
|
242
|
+
entry.update(
|
|
243
|
+
status="STALE" if occurrences == 0 else "NOT UNIQUE",
|
|
244
|
+
detail=(
|
|
245
|
+
f"anchor occurs {occurrences} times - "
|
|
246
|
+
+ (
|
|
247
|
+
"0 is the formatter-reflow signature; re-derive the anchor"
|
|
248
|
+
if occurrences == 0
|
|
249
|
+
else "a duplicated anchor mutates a different site than it reports"
|
|
250
|
+
)
|
|
251
|
+
),
|
|
252
|
+
)
|
|
253
|
+
rows.append(entry)
|
|
254
|
+
findings = True
|
|
255
|
+
continue
|
|
256
|
+
|
|
257
|
+
target.write_bytes(text.replace(row["anchor"], row["replacement"], 1).encode("utf-8"))
|
|
258
|
+
try:
|
|
259
|
+
code, output = _run(row["command"], project_root)
|
|
260
|
+
finally:
|
|
261
|
+
# Restore is unconditional and verified, never assumed: an earlier
|
|
262
|
+
# hand-rolled run in the field lost a file to its own cleanup. A
|
|
263
|
+
# restore WRITE that fails is the tree-dirty case and must land in
|
|
264
|
+
# the exit-2 refusal lane, not propagate as a traceback into the
|
|
265
|
+
# findings exit code.
|
|
266
|
+
try:
|
|
267
|
+
target.write_bytes(original)
|
|
268
|
+
except OSError as exc:
|
|
269
|
+
raise _refuse(
|
|
270
|
+
f"restore of {row['file']} FAILED ({exc}) - the tree still "
|
|
271
|
+
"carries the mutation; fix the tree before trusting "
|
|
272
|
+
"anything else this run printed"
|
|
273
|
+
) from exc
|
|
274
|
+
if target.read_bytes() != original:
|
|
275
|
+
raise _refuse(
|
|
276
|
+
f"restore of {row['file']} did not verify byte-identical - "
|
|
277
|
+
"the tree is dirty; fix that before trusting anything else this run printed"
|
|
278
|
+
)
|
|
279
|
+
entry["restore"] = "verified"
|
|
280
|
+
|
|
281
|
+
if code == 0:
|
|
282
|
+
entry.update(
|
|
283
|
+
status="GREEN",
|
|
284
|
+
detail="the mutation SURVIVED: nothing in the suite can tell - this is a finding",
|
|
285
|
+
)
|
|
286
|
+
findings = True
|
|
287
|
+
else:
|
|
288
|
+
status, detail, is_finding = _classify_red(output, row["expect_red"])
|
|
289
|
+
entry.update(status=status, detail=detail)
|
|
290
|
+
findings = findings or is_finding
|
|
291
|
+
rows.append(entry)
|
|
292
|
+
|
|
293
|
+
return rows, 1 if findings else 0
|
|
294
|
+
|
|
295
|
+
|
|
296
|
+
_LEDGER_ORDER = {"GREEN": 0, "MISATTRIBUTED": 1, "STALE": 2, "NOT UNIQUE": 2, "UNDECODABLE": 2}
|
|
297
|
+
|
|
298
|
+
|
|
299
|
+
def render_ledger(rows: list[dict]) -> str:
|
|
300
|
+
"""The generated markdown ledger, GREEN survivors first."""
|
|
301
|
+
ordered = sorted(rows, key=lambda r: (_LEDGER_ORDER.get(r["status"], 9), r["index"]))
|
|
302
|
+
greens = sum(1 for r in rows if r["status"] == "GREEN")
|
|
303
|
+
killed = sum(1 for r in rows if r["status"].startswith("KILLED"))
|
|
304
|
+
lines = [
|
|
305
|
+
"# Mutation ledger",
|
|
306
|
+
"",
|
|
307
|
+
f"{len(rows)} rows: {greens} GREEN (survived), {killed} killed, "
|
|
308
|
+
f"{len(rows) - greens - killed} not applied.",
|
|
309
|
+
"",
|
|
310
|
+
"| # | status | file | expected red | detail | restore | note |",
|
|
311
|
+
"|---|--------|------|--------------|--------|---------|------|",
|
|
312
|
+
]
|
|
313
|
+
for r in ordered:
|
|
314
|
+
lines.append(
|
|
315
|
+
f"| {r['index']} | {r['status']} | `{r['file']}` | `{r['expect_red']}` "
|
|
316
|
+
f"| {r['detail']} | {r['restore']} | {r['note']} |"
|
|
317
|
+
)
|
|
318
|
+
lines.append("")
|
|
319
|
+
return "\n".join(lines)
|
|
320
|
+
|
|
321
|
+
|
|
322
|
+
def main(argv: list[str] | None = None) -> int:
|
|
323
|
+
parser = argparse.ArgumentParser(
|
|
324
|
+
description="Run a mutation table against a green baseline; ledger out, greens first."
|
|
325
|
+
)
|
|
326
|
+
parser.add_argument("--table", required=True, help="PATH to the JSON mutation table.")
|
|
327
|
+
parser.add_argument(
|
|
328
|
+
"--project-root",
|
|
329
|
+
default=".",
|
|
330
|
+
help="Directory row file paths resolve against and commands run in (default: cwd).",
|
|
331
|
+
)
|
|
332
|
+
parser.add_argument(
|
|
333
|
+
"--output",
|
|
334
|
+
help="Also write the markdown ledger to this PATH (it always prints to stdout).",
|
|
335
|
+
)
|
|
336
|
+
args = parser.parse_args(argv)
|
|
337
|
+
|
|
338
|
+
_install_signal_unwind()
|
|
339
|
+
table = _load_table(Path(args.table))
|
|
340
|
+
project_root = Path(args.project_root).resolve()
|
|
341
|
+
if not project_root.is_dir():
|
|
342
|
+
raise _refuse(f"project root {project_root} is not a directory")
|
|
343
|
+
|
|
344
|
+
rows, exit_code = run_table(table, project_root)
|
|
345
|
+
ledger = render_ledger(rows)
|
|
346
|
+
print(ledger)
|
|
347
|
+
if args.output:
|
|
348
|
+
Path(args.output).write_text(ledger, encoding="utf-8")
|
|
349
|
+
return exit_code
|
|
350
|
+
|
|
351
|
+
|
|
352
|
+
if __name__ == "__main__":
|
|
353
|
+
sys.exit(main())
|