bmad-method-test-architecture-enterprise 2.0.1 → 2.0.2-next.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +4 -3
- package/cli/lib/resolve-tea-config.js +6 -9
- package/cli/lib/trace-command.js +628 -0
- package/cli/lib/trace-live.js +101 -0
- package/cli/lib/workflow-cli.js +250 -0
- package/cli/trace.js +258 -0
- package/package.json +4 -2
- package/skills/bmad-testarch-trace/checklist.md +35 -38
- package/skills/bmad-testarch-trace/steps-c/step-01-load-context.md +5 -0
- package/skills/bmad-testarch-trace/steps-c/step-02-discover-tests.md +1 -0
- package/skills/bmad-testarch-trace/steps-c/step-03-map-criteria.md +7 -0
- package/skills/bmad-testarch-trace/steps-c/step-04-analyze-gaps.md +40 -1
- package/skills/bmad-testarch-trace/steps-c/step-05-gate-decision.md +22 -1
- package/skills/bmad-testarch-trace/trace-template.md +1 -0
- package/skills/bmad-testarch-trace/workflow.yaml +3 -3
- package/skills/bmod-tea/bmod.toml +1 -1
|
@@ -254,6 +254,8 @@ Knowledge fragments referenced:
|
|
|
254
254
|
|
|
255
255
|
## PHASE 2: QUALITY GATE DECISION
|
|
256
256
|
|
|
257
|
+
The automated gate applies Step 5's coverage thresholds, oracle-confidence overlay, and live-evidence overlay. Test execution reports, NFR audits, code coverage, and burn-in results supply supporting context when available. Record their findings and missing evidence; these assessments have their own quality decisions. Report filed waivers separately and preserve the automated Trace decision.
|
|
258
|
+
|
|
257
259
|
**Note**: Phase 2 always emits `{e2e_trace_summary_output}` (`trace/e2e-trace-summary-{run_key}.json`); gate decision fields are populated only when `allow_gate: true` and `collection_status` resolves to `COLLECTED`.
|
|
258
260
|
|
|
259
261
|
---
|
|
@@ -262,7 +264,7 @@ Knowledge fragments referenced:
|
|
|
262
264
|
|
|
263
265
|
### Evidence Gathering
|
|
264
266
|
|
|
265
|
-
- [ ]
|
|
267
|
+
- [ ] Available test execution results read and cited (CI/CD pipeline, test framework reports)
|
|
266
268
|
- [ ] Story/epic/release file identified and read
|
|
267
269
|
- [ ] Test design document discovered in `{test_artifacts}/test-design/` (legacy: the root of `{test_artifacts}`) or explicitly provided (if available)
|
|
268
270
|
- [ ] Traceability matrix discovered or explicitly provided (available from Phase 1)
|
|
@@ -274,16 +276,15 @@ Knowledge fragments referenced:
|
|
|
274
276
|
### Evidence Validation
|
|
275
277
|
|
|
276
278
|
- [ ] Evidence freshness validated (warn if >7 days old, recommend re-running workflows)
|
|
277
|
-
- [ ]
|
|
278
|
-
- [ ]
|
|
279
|
-
- [ ] Test results match current codebase (not from outdated branch)
|
|
279
|
+
- [ ] Phase 1 coverage matrix available; missing supporting assessments recorded
|
|
280
|
+
- [ ] Available test execution reports are complete and match the current codebase
|
|
280
281
|
|
|
281
282
|
### Knowledge Base Loading
|
|
282
283
|
|
|
283
284
|
- [ ] `risk-governance.md` loaded successfully
|
|
284
285
|
- [ ] `probability-impact.md` loaded successfully
|
|
285
286
|
- [ ] `test-quality.md` loaded successfully
|
|
286
|
-
- [ ] `test-priorities.md` loaded successfully
|
|
287
|
+
- [ ] `test-priorities-matrix.md` loaded successfully
|
|
287
288
|
- [ ] `ci-burn-in.md` loaded (if burn-in results available)
|
|
288
289
|
|
|
289
290
|
---
|
|
@@ -300,7 +301,7 @@ Knowledge fragments referenced:
|
|
|
300
301
|
|
|
301
302
|
### Step 2: Evidence Parsing
|
|
302
303
|
|
|
303
|
-
**Test Results:**
|
|
304
|
+
**Test Results (when execution reports are available):**
|
|
304
305
|
|
|
305
306
|
- [ ] Total test count extracted
|
|
306
307
|
- [ ] Passed test count extracted
|
|
@@ -337,31 +338,28 @@ Knowledge fragments referenced:
|
|
|
337
338
|
|
|
338
339
|
**P0 Criteria Evaluation:**
|
|
339
340
|
|
|
340
|
-
- [ ] P0 test pass rate evaluated (must be 100%)
|
|
341
341
|
- [ ] P0 oracle-item coverage evaluated (must be 100%)
|
|
342
|
-
- [ ]
|
|
343
|
-
- [ ] Critical NFR failures evaluated (must be 0)
|
|
344
|
-
- [ ] Flaky tests evaluated (must be 0 if burn-in enabled)
|
|
342
|
+
- [ ] Available execution, security, NFR, and stability findings recorded as supporting evidence
|
|
345
343
|
- [ ] P0 decision recorded: PASS or FAIL
|
|
346
344
|
|
|
347
345
|
**P1 Criteria Evaluation:**
|
|
348
346
|
|
|
349
|
-
- [ ] P1 test pass rate evaluated (threshold: min_p1_pass_rate)
|
|
350
347
|
- [ ] P1 oracle-item coverage evaluated (PASS >=90%, CONCERNS 80-89%, FAIL <80%)
|
|
351
|
-
- [ ] Overall test pass rate evaluated (threshold: min_overall_pass_rate)
|
|
352
348
|
- [ ] Overall oracle coverage evaluated (threshold: >=80%)
|
|
353
349
|
- [ ] Code coverage considered if available (informational unless explicitly required by policy)
|
|
354
|
-
- [ ] P1 decision recorded: PASS or
|
|
350
|
+
- [ ] P1 decision recorded: PASS, CONCERNS, or FAIL
|
|
355
351
|
|
|
356
352
|
**P2/P3 Criteria Evaluation:**
|
|
357
353
|
|
|
358
|
-
- [ ] P2
|
|
359
|
-
- [ ] P3 failures
|
|
354
|
+
- [ ] P2/P3 coverage included in the overall coverage threshold
|
|
355
|
+
- [ ] Available P2/P3 execution failures recorded as supporting evidence
|
|
360
356
|
- [ ] Residual risks documented
|
|
361
357
|
|
|
362
358
|
**Final Decision:**
|
|
363
359
|
|
|
364
|
-
- [ ]
|
|
360
|
+
- [ ] Eligible automated decision determined: PASS / CONCERNS / FAIL
|
|
361
|
+
- [ ] Oracle-confidence and live-evidence overlays applied in Step 5's order
|
|
362
|
+
- [ ] Ineligible runs omit gate signals and record why the gate was skipped
|
|
365
363
|
- [ ] Decision rationale documented
|
|
366
364
|
- [ ] Decision is deterministic (follows rules, not arbitrary)
|
|
367
365
|
|
|
@@ -370,14 +368,14 @@ Knowledge fragments referenced:
|
|
|
370
368
|
**Gate Decision Document Created:**
|
|
371
369
|
|
|
372
370
|
- [ ] Story/epic/release info section complete (ID, title, description, links)
|
|
373
|
-
- [ ]
|
|
371
|
+
- [ ] Automated decision clearly stated (PASS / CONCERNS / FAIL, or gate skipped)
|
|
374
372
|
- [ ] Decision date recorded
|
|
375
373
|
- [ ] Evaluator recorded (user or agent name)
|
|
376
374
|
|
|
377
375
|
**Evidence Summary Documented:**
|
|
378
376
|
|
|
379
|
-
- [ ]
|
|
380
|
-
- [ ] Coverage summary complete (P0/P1 criteria, code coverage)
|
|
377
|
+
- [ ] Available test execution results summarized; missing reports recorded
|
|
378
|
+
- [ ] Coverage summary complete (P0/P1 criteria, code coverage when available)
|
|
381
379
|
- [ ] NFR validation summary complete (security, performance, reliability, maintainability)
|
|
382
380
|
- [ ] Flakiness summary complete (burn-in iterations, flaky test count)
|
|
383
381
|
|
|
@@ -417,12 +415,11 @@ Knowledge fragments referenced:
|
|
|
417
415
|
|
|
418
416
|
### Step 5: Status Updates and Notifications
|
|
419
417
|
|
|
420
|
-
**Gate
|
|
418
|
+
**Gate JSON Created When Eligible:**
|
|
421
419
|
|
|
422
|
-
- [ ] Gate
|
|
423
|
-
- [ ] Evidence references included in
|
|
424
|
-
- [ ]
|
|
425
|
-
- [ ] YAML file saved to output folder
|
|
420
|
+
- [ ] Gate JSON generated with decision and criteria
|
|
421
|
+
- [ ] Evidence references and rationale included in JSON
|
|
422
|
+
- [ ] JSON file saved to `{gate_decision_output}`
|
|
426
423
|
|
|
427
424
|
**Stakeholder Notification Generated:**
|
|
428
425
|
|
|
@@ -454,7 +451,7 @@ Knowledge fragments referenced:
|
|
|
454
451
|
**Accuracy:**
|
|
455
452
|
|
|
456
453
|
- [ ] Decision matches applied criteria rules
|
|
457
|
-
- [ ]
|
|
454
|
+
- [ ] Available test execution results match cited CI/CD pipeline output
|
|
458
455
|
- [ ] Coverage percentages match reports
|
|
459
456
|
- [ ] NFR status matches assessment document
|
|
460
457
|
- [ ] No contradictions or inconsistencies
|
|
@@ -466,12 +463,12 @@ Knowledge fragments referenced:
|
|
|
466
463
|
- [ ] Stakeholders can understand next steps
|
|
467
464
|
- [ ] Recommendations are actionable
|
|
468
465
|
|
|
469
|
-
### Gate
|
|
466
|
+
### Gate JSON
|
|
470
467
|
|
|
471
468
|
**Format:**
|
|
472
469
|
|
|
473
|
-
- [ ]
|
|
474
|
-
- [ ]
|
|
470
|
+
- [ ] JSON is valid (no syntax errors)
|
|
471
|
+
- [ ] Schema 0.1.0 fields match Step 5's slim gate contract
|
|
475
472
|
- [ ] Field values are correct data types (numbers, strings, dates)
|
|
476
473
|
|
|
477
474
|
**Content:**
|
|
@@ -487,14 +484,14 @@ Knowledge fragments referenced:
|
|
|
487
484
|
### Decision Integrity
|
|
488
485
|
|
|
489
486
|
- [ ] Decision is deterministic (follows rules, not arbitrary)
|
|
490
|
-
- [ ] P0
|
|
491
|
-
- [ ]
|
|
492
|
-
- [ ]
|
|
487
|
+
- [ ] P0 coverage below 100% results in FAIL
|
|
488
|
+
- [ ] Fresh live failures cap a passing decision at CONCERNS as Step 5 defines
|
|
489
|
+
- [ ] Filed waivers are validated and reported separately from the derived decision
|
|
493
490
|
- [ ] Residual risks are documented (if CONCERNS or WAIVED)
|
|
494
491
|
|
|
495
492
|
### Evidence-Based
|
|
496
493
|
|
|
497
|
-
- [ ] Decision
|
|
494
|
+
- [ ] Decision follows the coverage matrix and available test execution results
|
|
498
495
|
- [ ] All claims are supported by evidence
|
|
499
496
|
- [ ] No assumptions without documentation
|
|
500
497
|
- [ ] Evidence sources are cited (CI run IDs, report URLs)
|
|
@@ -519,8 +516,8 @@ Knowledge fragments referenced:
|
|
|
519
516
|
|
|
520
517
|
### CI/CD Pipeline
|
|
521
518
|
|
|
522
|
-
- [ ] Gate
|
|
523
|
-
- [ ]
|
|
519
|
+
- [ ] Gate JSON is CI/CD-compatible
|
|
520
|
+
- [ ] JSON can be parsed by pipeline automation
|
|
524
521
|
- [ ] Decision can be used to block/allow deployments
|
|
525
522
|
- [ ] Evidence references are accessible to pipeline
|
|
526
523
|
|
|
@@ -546,7 +543,7 @@ Knowledge fragments referenced:
|
|
|
546
543
|
### Traceability
|
|
547
544
|
|
|
548
545
|
- [ ] Gate decision traceable to story/epic/release
|
|
549
|
-
- [ ] Evidence traceable to specific
|
|
546
|
+
- [ ] Evidence traceable to cited tests and to specific runs when execution reports exist
|
|
550
547
|
- [ ] Assessments traceable to workflows that created them
|
|
551
548
|
- [ ] Waiver traceable to approver (if applicable)
|
|
552
549
|
|
|
@@ -563,8 +560,8 @@ Knowledge fragments referenced:
|
|
|
563
560
|
|
|
564
561
|
### Missing Evidence
|
|
565
562
|
|
|
566
|
-
- [ ] If no test design doc exists in `{test_artifacts}/test-design/` or the legacy root, decision still possible
|
|
567
|
-
- [ ] If
|
|
563
|
+
- [ ] If no test design doc exists in `{test_artifacts}/test-design/` or the legacy root, decision still possible from the Phase 1 matrix
|
|
564
|
+
- [ ] If no test execution report exists, decision follows the Phase 1 matrix and records the missing runtime evidence
|
|
568
565
|
- [ ] If no NFR assessment exists in `{test_artifacts}/nfr/` or the legacy root, NFR validation marked as NOT ASSESSED
|
|
569
566
|
- [ ] If code coverage missing, coverage criterion marked as NOT ASSESSED
|
|
570
567
|
- [ ] User acknowledged gaps in evidence or provided alternative proof
|
|
@@ -578,7 +575,7 @@ Knowledge fragments referenced:
|
|
|
578
575
|
|
|
579
576
|
### Conflicting Evidence
|
|
580
577
|
|
|
581
|
-
- [ ] Conflicts between test results and assessments resolved
|
|
578
|
+
- [ ] Conflicts between available test results and assessments resolved
|
|
582
579
|
- [ ] Most recent/authoritative source identified
|
|
583
580
|
- [ ] Conflict resolution documented in decision rationale
|
|
584
581
|
- [ ] User consulted if conflict cannot be resolved
|
|
@@ -82,6 +82,10 @@ Resolve the oracle in this order:
|
|
|
82
82
|
- `P2`: secondary workflows and edge scenarios
|
|
83
83
|
- `P3`: low-risk polish or optional flows
|
|
84
84
|
|
|
85
|
+
Persist `oracleLedger` in this step's progress YAML frontmatter as an array of `{id, requirement, priority, source}` records: one row per item with its stable ID, requirement text, priority, and source reference. An explicit priority in the selected oracle is authoritative for that item's coverage statistics. Infer a priority with `test-priorities-matrix.md` only when the oracle leaves it unspecified, and record that inference. A mapped test's own priority describes the test and never changes the oracle item's priority. Carry this ledger through mapping and gap analysis, including resumed runs.
|
|
86
|
+
|
|
87
|
+
For local source files, read numbered lines before recording each ledger source reference. When the source declares an item's ID and priority, cite that line. For an inferred item, cite numbered source lines that support its requirement; the generated ID and priority stay in the ledger. Check each source line independently because headings and criteria may span different numbers of lines.
|
|
88
|
+
|
|
85
89
|
Record the resolved oracle metadata in step output/frontmatter using consistent keys:
|
|
86
90
|
|
|
87
91
|
- `coverageBasis` (`acceptance_criteria` | `synthetic_requirements` | `openapi_endpoints` | `user_journeys`) — the type of oracle selected for coverage tracing
|
|
@@ -218,6 +222,7 @@ oracleConfidence: '{resolved oracle_confidence}'
|
|
|
218
222
|
oracleResolutionMode: '{resolved oracle_resolution_mode}'
|
|
219
223
|
oracleSources: ['{resolved oracle source 1}', '{resolved oracle source 2}']
|
|
220
224
|
externalPointerStatus: '{resolved external_pointer_status}'
|
|
225
|
+
oracleLedger: [{ id: '{oracle item id}', requirement: '{requirement text}', priority: '{P0-P3}', source: '{source reference}' }]
|
|
221
226
|
---
|
|
222
227
|
```
|
|
223
228
|
|
|
@@ -234,6 +234,7 @@ Capture explicit coverage signals so Phase 1 can detect common blind spots:
|
|
|
234
234
|
- API endpoint coverage
|
|
235
235
|
- Inventory endpoints referenced by requirements/specs and endpoints exercised by API tests
|
|
236
236
|
- Mark endpoints with no direct tests
|
|
237
|
+
- Count an endpoint as directly tested when an API test calls its route and asserts a response. A test can exercise the endpoint while leaving a specific requirement unproved. Record missing permission-denied assertions under authentication/authorization coverage.
|
|
237
238
|
- Authentication/authorization coverage
|
|
238
239
|
- Detect tests for login/session/token flows and permission-denied paths
|
|
239
240
|
- Mark auth/authz requirements with missing negative-path tests
|
|
@@ -39,6 +39,7 @@ Create the traceability matrix linking the resolved oracle items to tests.
|
|
|
39
39
|
|
|
40
40
|
For each resolved oracle item (formal requirement, endpoint/spec item, or synthetic journey):
|
|
41
41
|
|
|
42
|
+
- Copy its ID, requirement text, and priority from Step 1's persisted oracle ledger. Preserve explicit source priorities through every matrix heading and statistics bucket. Keep each mapped test's priority in the test record as a separate field.
|
|
42
43
|
- Map to matching tests
|
|
43
44
|
- Mark coverage status: FULL / PARTIAL / NONE / UNIT-ONLY / INTEGRATION-ONLY. `checklist.md`'s "Coverage Classification" section is the definition; **read it before classifying anything**, and apply the rule it turns on:
|
|
44
45
|
- Classification is decided by what the evidence **establishes** about the criterion. The number of levels the evidence spans does not set the status.
|
|
@@ -46,6 +47,10 @@ For each resolved oracle item (formal requirement, endpoint/spec item, or synthe
|
|
|
46
47
|
- **UNIT-ONLY** and **INTEGRATION-ONLY** are for the case where a _missing_ level is what leaves the criterion unestablished: unit tests only, against a criterion stating an HTTP status or a rendered state; or API/component tests only, against a criterion stating branch-level logic that needs unit proof. "The only evidence is API-level" is not by itself INTEGRATION-ONLY.
|
|
47
48
|
- Cite every mapped test as a backticked `` `path:line` `` pair, the form `trace-template.md` and `resources/traceability-matrix.example.md` both use. The backticks are the delimiter, so a path holding a space survives whole. An undelimited pair written into prose has no boundary but the file extension, and a path with a space in it is then read only from its last space-free segment onward.
|
|
48
49
|
- Record test level and priority
|
|
50
|
+
- For every accepted static test, cite its declaration once and each assertion that establishes a fact named by the criterion once within that criterion's evidence block. Cite both an asserted state and an asserted property that ties it to the stated condition, such as `expired` state and the token's expiry timestamp.
|
|
51
|
+
- A status assertion on a prerequisite request is setup evidence. A 200 response before revocation does not establish that a later request is rejected. Cite a response status when the criterion requires it or the status itself establishes the outcome, such as 403 for denied access. When a later state or value assertion proves the outcome, omit an earlier 2xx response assertion on the action that produced it.
|
|
52
|
+
- For a criterion that asks for a ready job and a signed link, cite the ready-state and signed-link assertions. An earlier 202 response only shows that creation was accepted. Apply the same rule to other intermediate 2xx responses.
|
|
53
|
+
- Use the exact assertion line; for a multiline assertion, use its opening line. Use one project-relative path spelling throughout, including rejected evidence. Strip the project root directory from paths inside that project. Do not cite setup or action lines as separate anchors. Explain their role beside the assertion anchors. Before Step 4, remove duplicate `path:line` pairs from each criterion block, including pairs repeated in prose.
|
|
49
54
|
- Preserve each mapped test's stable identity fields (`id`, `title`, `file`, `line`, `level`, status flags) so Phase 1 can deduplicate unique tests before JSON export
|
|
50
55
|
- Record heuristic signals:
|
|
51
56
|
- Endpoint coverage present/missing (for API-impacting items)
|
|
@@ -64,6 +69,8 @@ That is the rule for this workflow, and it fixes what every count derived from `
|
|
|
64
69
|
|
|
65
70
|
Record every such test in `rejectedEvidence` against the criterion it claims, with the reason its assertions fall short. Carry the list forward for Step 4 and persist it into the progress document so a resumed Step 4 can read it. A reader needs to see that the test was read and turned down: a test that claims a criterion and then vanishes from the report is indistinguishable from a test nobody found.
|
|
66
71
|
|
|
72
|
+
Use the test's explicit ID when its source declares one. Otherwise set `test_id` to its project-relative `file:line` declaration anchor. Do not assign a new sequence number to a rejected test.
|
|
73
|
+
|
|
67
74
|
```javascript
|
|
68
75
|
// One entry per test whose name claims a criterion its assertions do not establish.
|
|
69
76
|
// Shape only; the entries come from reading the tests against the criterion text.
|
|
@@ -116,6 +116,10 @@ Resolution precedence:
|
|
|
116
116
|
2. `tea_execution_mode` from config
|
|
117
117
|
3. Runtime capability fallback (when probing enabled)
|
|
118
118
|
|
|
119
|
+
### 0b. Reconcile the Oracle Ledger
|
|
120
|
+
|
|
121
|
+
Before classifying gaps, compare every matrix item's ID and priority with Step 1's persisted oracle ledger. Require exactly one matrix item per oracle item and unchanged priorities. Correct any drift from the ledger before using the matrix in Sections 1 and 3. Repeat this check when resuming a saved mapping.
|
|
122
|
+
|
|
119
123
|
### 1. Gap Analysis
|
|
120
124
|
|
|
121
125
|
**Identify uncovered requirements:**
|
|
@@ -147,7 +151,7 @@ const lowGaps = uncoveredRequirements.filter((req) => req.priority === 'P3');
|
|
|
147
151
|
|
|
148
152
|
### 2. Coverage Heuristics Checks
|
|
149
153
|
|
|
150
|
-
Use the heuristics inventory from Step 2 and mapped criteria from Step 3 to flag common coverage blind spots:
|
|
154
|
+
Use the heuristics inventory from Step 2 and mapped criteria from Step 3 to flag common coverage blind spots. Carry Step 2's endpoint inventory through unchanged: endpoint reachability comes from API calls and response assertions, while missing permission-denied assertions appear in the auth gap list.
|
|
151
155
|
|
|
152
156
|
```javascript
|
|
153
157
|
const endpointCoverageGaps = coverageHeuristics?.endpoints_without_tests || [];
|
|
@@ -286,6 +290,20 @@ const liveEvidence = {
|
|
|
286
290
|
stale: staleCount,
|
|
287
291
|
unverifiable: unverifiableCount,
|
|
288
292
|
failed: countDisposition('fail'),
|
|
293
|
+
fresh_failed: liveRecordsForRollup.filter((record) => {
|
|
294
|
+
if (record.disposition !== 'fail') return false;
|
|
295
|
+
const recorded = String(record.recorded_source_sha || '')
|
|
296
|
+
.trim()
|
|
297
|
+
.toLowerCase();
|
|
298
|
+
const current = String(currentSourceSha || '')
|
|
299
|
+
.trim()
|
|
300
|
+
.toLowerCase();
|
|
301
|
+
return (
|
|
302
|
+
/^[0-9a-f]{7,64}$/.test(recorded) &&
|
|
303
|
+
/^[0-9a-f]{7,64}$/.test(current) &&
|
|
304
|
+
(recorded.startsWith(current) || current.startsWith(recorded))
|
|
305
|
+
);
|
|
306
|
+
}).length,
|
|
289
307
|
contradicted: countDisposition('contradicted'),
|
|
290
308
|
blocked: countDisposition('blocked'),
|
|
291
309
|
skipped: countDisposition('skipped'),
|
|
@@ -690,7 +708,28 @@ const coverageMatrix = {
|
|
|
690
708
|
|
|
691
709
|
**Write to temp file for Phase 2:**
|
|
692
710
|
|
|
711
|
+
Recount `FULL` rows before saving. The overall count must equal the sum of the four priority counts and the number of `FULL` requirement rows. If any count differs, correct the statistics from the mapped rows and repeat this check. `PARTIAL`, `UNIT-ONLY`, and `INTEGRATION-ONLY` do not count as fully covered.
|
|
712
|
+
|
|
693
713
|
```javascript
|
|
714
|
+
const rows = coverageMatrix.requirements;
|
|
715
|
+
const statistics = coverageMatrix.coverage_statistics;
|
|
716
|
+
const priorities = ['P0', 'P1', 'P2', 'P3'];
|
|
717
|
+
const fullRows = rows.filter((row) => row.coverage === 'FULL');
|
|
718
|
+
const countedByPriority = priorities.reduce((count, priority) => count + statistics.priority_breakdown[priority].covered, 0);
|
|
719
|
+
if (
|
|
720
|
+
statistics.total_requirements !== rows.length ||
|
|
721
|
+
statistics.fully_covered !== fullRows.length ||
|
|
722
|
+
statistics.fully_covered !== countedByPriority ||
|
|
723
|
+
statistics.overall_coverage_percentage !== safePct(fullRows.length, rows.length) ||
|
|
724
|
+
priorities.some(
|
|
725
|
+
(priority) =>
|
|
726
|
+
statistics.priority_breakdown[priority].total !== rows.filter((row) => row.priority === priority).length ||
|
|
727
|
+
statistics.priority_breakdown[priority].covered !== fullRows.filter((row) => row.priority === priority).length,
|
|
728
|
+
)
|
|
729
|
+
) {
|
|
730
|
+
throw new Error('Coverage totals disagree with mapped requirement rows; correct Phase 1 statistics before saving');
|
|
731
|
+
}
|
|
732
|
+
|
|
694
733
|
const outputPath = '{tempOutputFile}';
|
|
695
734
|
fs.writeFileSync(outputPath, JSON.stringify(coverageMatrix, null, 2), 'utf8');
|
|
696
735
|
|
|
@@ -92,6 +92,21 @@ if (
|
|
|
92
92
|
);
|
|
93
93
|
}
|
|
94
94
|
const priorityBreakdown = stats.priority_breakdown;
|
|
95
|
+
// Check Phase 1 counts again before they become the published summary and gate inputs.
|
|
96
|
+
// Repair the Phase 1 matrix from its requirement rows and rerun this check on mismatch.
|
|
97
|
+
const requirementRows = coverageMatrix.requirements || [];
|
|
98
|
+
const fullRequirementRows = requirementRows.filter((row) => row.coverage === 'FULL');
|
|
99
|
+
const priorityNames = ['P0', 'P1', 'P2', 'P3'];
|
|
100
|
+
if (
|
|
101
|
+
stats.total_requirements !== requirementRows.length ||
|
|
102
|
+
stats.fully_covered !== fullRequirementRows.length ||
|
|
103
|
+
priorityNames.reduce((sum, priority) => sum + priorityBreakdown[priority].covered, 0) !== fullRequirementRows.length ||
|
|
104
|
+
priorityNames.some(
|
|
105
|
+
(priority) => priorityBreakdown[priority].covered !== fullRequirementRows.filter((row) => row.priority === priority).length,
|
|
106
|
+
)
|
|
107
|
+
) {
|
|
108
|
+
throw new Error('Phase 1 coverage totals disagree with mapped requirement rows; correct the matrix before gate evaluation');
|
|
109
|
+
}
|
|
95
110
|
const p0Coverage = priorityBreakdown.P0.percentage;
|
|
96
111
|
const p1Coverage = priorityBreakdown.P1.percentage;
|
|
97
112
|
const hasP1Requirements = (priorityBreakdown.P1.total || 0) > 0;
|
|
@@ -328,6 +343,12 @@ if (!gateEligible) {
|
|
|
328
343
|
// A live result is a one-time observation of one commit with no artifact anyone can re-run, so a
|
|
329
344
|
// requirement resting only on it is capped at CONCERNS. This overlay only ever lowers PASS or
|
|
330
345
|
// annotates an existing CONCERNS; it can never lift a FAIL.
|
|
346
|
+
const freshLiveFailures = liveEvidence.fresh_failed ?? (liveEvidence.freshness === 'fresh' ? liveEvidence.failed : 0);
|
|
347
|
+
if (freshLiveFailures > 0 && gateDecision === 'PASS') {
|
|
348
|
+
gateDecision = 'CONCERNS';
|
|
349
|
+
rationale = `${rationale} ${freshLiveFailures} fresh live verification failure(s) require investigation.`;
|
|
350
|
+
}
|
|
351
|
+
|
|
331
352
|
if (liveOnlyCoveredRequirements > 0 && ['PASS', 'CONCERNS'].includes(gateDecision)) {
|
|
332
353
|
gateDecision = 'CONCERNS';
|
|
333
354
|
// Appended rather than replaced so the coverage numbers that produced the base decision survive.
|
|
@@ -741,7 +762,7 @@ fs.writeFileSync('{e2e_trace_summary_output}', JSON.stringify(e2eTraceSummary, n
|
|
|
741
762
|
console.log(`✅ e2e trace summary written to {e2e_trace_summary_output}`);
|
|
742
763
|
```
|
|
743
764
|
|
|
744
|
-
**
|
|
765
|
+
**Emit `gate-decision-{run_key}.json` whenever gate-eligible** to `{gate_decision_output}` for pipelines that consume the gate signal:
|
|
745
766
|
|
|
746
767
|
```javascript
|
|
747
768
|
// Construct and write only when gate evaluation was performed and produced a meaningful decision.
|
|
@@ -37,7 +37,7 @@ variables:
|
|
|
37
37
|
# writes another scope's files. Inputs stay at the root of {test_artifacts}.
|
|
38
38
|
default_output_file: "{test_artifacts}/trace/traceability-matrix-{run_key}.md"
|
|
39
39
|
e2e_trace_summary_output: "{test_artifacts}/trace/e2e-trace-summary-{run_key}.json" # Machine-readable summary consumed by CI/CD and reporting pipelines
|
|
40
|
-
gate_decision_output: "{test_artifacts}/trace/gate-decision-{run_key}.json" #
|
|
40
|
+
gate_decision_output: "{test_artifacts}/trace/gate-decision-{run_key}.json" # Required downstream gate signal when gate-eligible
|
|
41
41
|
|
|
42
42
|
# e2e-trace-summary-{run_key}.json schema (emitted by step-05 to {e2e_trace_summary_output} at workflow completion)
|
|
43
43
|
# schema_version: "0.3.0" (0.2.0 added the live_evidence block and the by_level.live bucket; 0.3.0 added the waivers block)
|
|
@@ -45,7 +45,7 @@ gate_decision_output: "{test_artifacts}/trace/gate-decision-{run_key}.json" # Op
|
|
|
45
45
|
# repo / collection_mode / collection_status / inventory_basis / gate_basis / source_sha
|
|
46
46
|
# target: { type, id, label } / decision_mode / evaluator / confidence
|
|
47
47
|
# oracle: { resolution_mode, confidence, sources[], external_pointer_status, synthetic }
|
|
48
|
-
# gate_status: PASS | CONCERNS | FAIL
|
|
48
|
+
# gate_status: PASS | CONCERNS | FAIL (only when allow_gate is true and collection_status is COLLECTED)
|
|
49
49
|
# coverage: { inventory{covered,total,pct}, priority_breakdown{P0..P3}, by_level{e2e,api,component,unit,live,other} }
|
|
50
50
|
# tests: { files, cases, skipped_cases, fixme_cases, pending_cases }
|
|
51
51
|
# risk_summary: { critical_open, high_open, medium_open, low_open }
|
|
@@ -87,7 +87,7 @@ gate_decision_output: "{test_artifacts}/trace/gate-decision-{run_key}.json" # Op
|
|
|
87
87
|
# Required tools
|
|
88
88
|
required_tools:
|
|
89
89
|
- read_file # Read story, test files, BMad artifacts
|
|
90
|
-
- write_file # Create traceability matrix, gate
|
|
90
|
+
- write_file # Create traceability matrix, summary JSON, gate JSON
|
|
91
91
|
- list_files # Discover test files
|
|
92
92
|
- search_repo # Find tests by test ID, describe blocks
|
|
93
93
|
- glob # Find test files matching patterns
|