bmad-method-test-architecture-enterprise 1.27.2-next.3 → 1.27.2-next.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -31,7 +31,7 @@
31
31
  "name": "bmad-method-test-architecture-enterprise",
32
32
  "source": "./",
33
33
  "description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
34
- "version": "1.27.2-next.3",
34
+ "version": "1.27.2-next.4",
35
35
  "author": {
36
36
  "name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
37
37
  },
package/README.md CHANGED
@@ -392,7 +392,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
392
392
 
393
393
  ### Deterministic Checks
394
394
 
395
- `npm test` chains seventy-four checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Ten of the seventy-four keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
395
+ `npm test` chains seventy-five checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Eleven of the seventy-five keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
396
396
 
397
397
  - `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
398
398
  - `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, and the fragment-selection case count. A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
@@ -403,6 +403,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
403
403
  - `test:eval-routing-boundaries` mutates the facts, candidate codes, and missing deciding information in each source-bound routing ambiguity rule. It also rejects duplicate and orphan boundaries, plus any boundary that would turn a clear intent into a clarification.
404
404
  - `test:eval-schemas` checks this repository's own eval suite manifest against its schema, confirms that every threshold it declares is the threshold the harness actually applies, that the runner capabilities it declares are the ones the harness grants its runner, and that the preflight argv it declares fails on a missing runner and passes on a present one, and fails when a TEA skill has neither a behavioral suite nor a deferred declaration. A suite list that omits a skill reads as coverage, so the omission has to be an error rather than a silence.
405
405
  - `test:eval-diagnostics` checks that current result writers emit bounded case and repetition diagnostics, that quality and environment outcomes stay separate, that unstable signatures remain visible, and that the protected 1.3.0 baseline remains readable and byte-for-byte unchanged.
406
+ - `test:nfr-evidence-guidance` holds every NFR worker and the publication audit to the supplied-evidence ledger, exercises supported, missing, and mismatched evidence, rejects contaminating examples, and binds the protected baseline and focused remediation records by digest.
406
407
  - `test:contract-oracles` evaluates every oracle in every eval contract with `eval-quality`'s own evaluator, over the stored replay outputs and over constructed selections for all 24 fragment-selection cases, and fails when an oracle faults or disagrees with the harness scorer on the same evidence. The first run found that every regex oracle in the `test-review` contract was refused by the evaluator before it matched anything.
407
408
 
408
409
  These checks produce the same answer from the same repository state. They need no agent credential, network call, or model budget. `test:eval-data` runs through `npm test`, the local pre-commit hook, pull-request quality checks, and the publish workflow.
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "$schema": "https://json.schemastore.org/package.json",
3
3
  "name": "bmad-method-test-architecture-enterprise",
4
- "version": "1.27.2-next.3",
4
+ "version": "1.27.2-next.4",
5
5
  "description": "Master Test Architect for quality strategy, test automation, and release gates",
6
6
  "keywords": [
7
7
  "bmad",
@@ -83,7 +83,7 @@
83
83
  "release:minor": "gh workflow run publish.yaml -f channel=latest -f bump=minor",
84
84
  "release:next": "gh workflow run publish.yaml -f channel=next",
85
85
  "release:patch": "gh workflow run publish.yaml -f channel=latest -f bump=patch",
86
- "test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
86
+ "test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:nfr-evidence-guidance && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
87
87
  "test:atdd-isolation": "node test/test-atdd-isolation.js",
88
88
  "test:atdd-net-guard": "node test/test-atdd-net-guard.js",
89
89
  "test:atdd-scoring": "node test/test-atdd-scoring.js",
@@ -142,6 +142,7 @@
142
142
  "test:lineage": "eval-quality-gates field-ownership",
143
143
  "test:lockfile-age": "eval-quality-gates lockfile-age",
144
144
  "test:lockfile-age-cache": "node test/test-generate-lockfile-age-cache.js",
145
+ "test:nfr-evidence-guidance": "node test/test-nfr-evidence-guidance.js",
145
146
  "test:nfr-validate-corpus": "node test/test-nfr-validate-corpus.js",
146
147
  "test:port-totality": "node test/test-port-totality.js",
147
148
  "test:probe-conformance": "node test/test-probe-conformance.js",
@@ -35,7 +35,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
35
35
  - **Status:** {STATUS} {STATUS_ICON}
36
36
  - **Threshold:** {THRESHOLD_VALUE}
37
37
  - **Actual:** {ACTUAL_VALUE}
38
- - **Evidence:** {EVIDENCE_SOURCE}
38
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
39
+ - **Supports:** {SUPPORTED_OBSERVATION}
39
40
  - **Findings:** {FINDINGS_DESCRIPTION}
40
41
 
41
42
  ### Throughput
@@ -43,7 +44,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
43
44
  - **Status:** {STATUS} {STATUS_ICON}
44
45
  - **Threshold:** {THRESHOLD_VALUE}
45
46
  - **Actual:** {ACTUAL_VALUE}
46
- - **Evidence:** {EVIDENCE_SOURCE}
47
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
48
+ - **Supports:** {SUPPORTED_OBSERVATION}
47
49
  - **Findings:** {FINDINGS_DESCRIPTION}
48
50
 
49
51
  ### Resource Usage
@@ -52,13 +54,15 @@ Note: This audit summarizes existing implementation evidence; it does not run te
52
54
  - **Status:** {STATUS} {STATUS_ICON}
53
55
  - **Threshold:** {THRESHOLD_VALUE}
54
56
  - **Actual:** {ACTUAL_VALUE}
55
- - **Evidence:** {EVIDENCE_SOURCE}
57
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
58
+ - **Supports:** {SUPPORTED_OBSERVATION}
56
59
 
57
60
  - **Memory Usage**
58
61
  - **Status:** {STATUS} {STATUS_ICON}
59
62
  - **Threshold:** {THRESHOLD_VALUE}
60
63
  - **Actual:** {ACTUAL_VALUE}
61
- - **Evidence:** {EVIDENCE_SOURCE}
64
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
65
+ - **Supports:** {SUPPORTED_OBSERVATION}
62
66
 
63
67
  ---
64
68
 
@@ -69,7 +73,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
69
73
  - **Status:** {STATUS} {STATUS_ICON}
70
74
  - **Threshold:** {THRESHOLD_DESCRIPTION}
71
75
  - **Actual:** {ACTUAL_DESCRIPTION}
72
- - **Evidence:** {EVIDENCE_SOURCE}
76
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
77
+ - **Supports:** {SUPPORTED_OBSERVATION}
73
78
  - **Findings:** {FINDINGS_DESCRIPTION}
74
79
  - **Recommendation:** {RECOMMENDATION} (if CONCERNS or FAIL)
75
80
 
@@ -78,7 +83,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
78
83
  - **Status:** {STATUS} {STATUS_ICON}
79
84
  - **Threshold:** {THRESHOLD_DESCRIPTION}
80
85
  - **Actual:** {ACTUAL_DESCRIPTION}
81
- - **Evidence:** {EVIDENCE_SOURCE}
86
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
87
+ - **Supports:** {SUPPORTED_OBSERVATION}
82
88
  - **Findings:** {FINDINGS_DESCRIPTION}
83
89
 
84
90
  ### Data Protection
@@ -86,23 +92,26 @@ Note: This audit summarizes existing implementation evidence; it does not run te
86
92
  - **Status:** {STATUS} {STATUS_ICON}
87
93
  - **Threshold:** {THRESHOLD_DESCRIPTION}
88
94
  - **Actual:** {ACTUAL_DESCRIPTION}
89
- - **Evidence:** {EVIDENCE_SOURCE}
95
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
96
+ - **Supports:** {SUPPORTED_OBSERVATION}
90
97
  - **Findings:** {FINDINGS_DESCRIPTION}
91
98
 
92
99
  ### Vulnerability Management
93
100
 
94
101
  - **Status:** {STATUS} {STATUS_ICON}
95
- - **Threshold:** {THRESHOLD_DESCRIPTION} (e.g., "0 critical, <3 high vulnerabilities")
96
- - **Actual:** {ACTUAL_DESCRIPTION} (e.g., "0 critical, 1 high, 5 medium vulnerabilities")
97
- - **Evidence:** {EVIDENCE_SOURCE} (e.g., "Snyk scan results - scan-2025-10-14.json")
102
+ - **Threshold:** {THRESHOLD_DESCRIPTION}
103
+ - **Actual:** {ACTUAL_DESCRIPTION}
104
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
105
+ - **Supports:** {SUPPORTED_OBSERVATION}
98
106
  - **Findings:** {FINDINGS_DESCRIPTION}
99
107
 
100
108
  ### Compliance (if applicable)
101
109
 
102
110
  - **Status:** {STATUS} {STATUS_ICON}
103
- - **Standards:** {COMPLIANCE_STANDARDS} (e.g., "GDPR, HIPAA, PCI-DSS")
111
+ - **Standards:** {COMPLIANCE_STANDARDS}
104
112
  - **Actual:** {ACTUAL_COMPLIANCE_STATUS}
105
- - **Evidence:** {EVIDENCE_SOURCE}
113
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
114
+ - **Supports:** {SUPPORTED_OBSERVATION}
106
115
  - **Findings:** {FINDINGS_DESCRIPTION}
107
116
 
108
117
  ---
@@ -112,25 +121,28 @@ Note: This audit summarizes existing implementation evidence; it does not run te
112
121
  ### Availability (Uptime)
113
122
 
114
123
  - **Status:** {STATUS} {STATUS_ICON}
115
- - **Threshold:** {THRESHOLD_VALUE} (e.g., "99.9%")
116
- - **Actual:** {ACTUAL_VALUE} (e.g., "99.95%")
117
- - **Evidence:** {EVIDENCE_SOURCE} (e.g., "Uptime monitoring - uptime-report-2025-10-14.csv")
124
+ - **Threshold:** {THRESHOLD_VALUE}
125
+ - **Actual:** {ACTUAL_VALUE}
126
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
127
+ - **Supports:** {SUPPORTED_OBSERVATION}
118
128
  - **Findings:** {FINDINGS_DESCRIPTION}
119
129
 
120
130
  ### Error Rate
121
131
 
122
132
  - **Status:** {STATUS} {STATUS_ICON}
123
- - **Threshold:** {THRESHOLD_VALUE} (e.g., "<0.1%")
124
- - **Actual:** {ACTUAL_VALUE} (e.g., "0.05%")
125
- - **Evidence:** {EVIDENCE_SOURCE} (e.g., "Error logs - logs/errors-2025-10.log")
133
+ - **Threshold:** {THRESHOLD_VALUE}
134
+ - **Actual:** {ACTUAL_VALUE}
135
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
136
+ - **Supports:** {SUPPORTED_OBSERVATION}
126
137
  - **Findings:** {FINDINGS_DESCRIPTION}
127
138
 
128
139
  ### MTTR (Mean Time To Recovery)
129
140
 
130
141
  - **Status:** {STATUS} {STATUS_ICON}
131
- - **Threshold:** {THRESHOLD_VALUE} (e.g., "<15 minutes")
132
- - **Actual:** {ACTUAL_VALUE} (e.g., "12 minutes")
133
- - **Evidence:** {EVIDENCE_SOURCE} (e.g., "Incident reports - incidents/")
142
+ - **Threshold:** {THRESHOLD_VALUE}
143
+ - **Actual:** {ACTUAL_VALUE}
144
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
145
+ - **Supports:** {SUPPORTED_OBSERVATION}
134
146
  - **Findings:** {FINDINGS_DESCRIPTION}
135
147
 
136
148
  ### Fault Tolerance
@@ -138,15 +150,17 @@ Note: This audit summarizes existing implementation evidence; it does not run te
138
150
  - **Status:** {STATUS} {STATUS_ICON}
139
151
  - **Threshold:** {THRESHOLD_DESCRIPTION}
140
152
  - **Actual:** {ACTUAL_DESCRIPTION}
141
- - **Evidence:** {EVIDENCE_SOURCE}
153
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
154
+ - **Supports:** {SUPPORTED_OBSERVATION}
142
155
  - **Findings:** {FINDINGS_DESCRIPTION}
143
156
 
144
157
  ### CI Burn-In (Stability)
145
158
 
146
159
  - **Status:** {STATUS} {STATUS_ICON}
147
- - **Threshold:** {THRESHOLD_VALUE} (e.g., "100 consecutive successful runs")
148
- - **Actual:** {ACTUAL_VALUE} (e.g., "150 consecutive successful runs")
149
- - **Evidence:** {EVIDENCE_SOURCE} (e.g., "CI burn-in results - ci-burn-in-2025-10-14.log")
160
+ - **Threshold:** {THRESHOLD_VALUE}
161
+ - **Actual:** {ACTUAL_VALUE}
162
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
163
+ - **Supports:** {SUPPORTED_OBSERVATION}
150
164
  - **Findings:** {FINDINGS_DESCRIPTION}
151
165
 
152
166
  ### Disaster Recovery (if applicable)
@@ -155,13 +169,15 @@ Note: This audit summarizes existing implementation evidence; it does not run te
155
169
  - **Status:** {STATUS} {STATUS_ICON}
156
170
  - **Threshold:** {THRESHOLD_VALUE}
157
171
  - **Actual:** {ACTUAL_VALUE}
158
- - **Evidence:** {EVIDENCE_SOURCE}
172
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
173
+ - **Supports:** {SUPPORTED_OBSERVATION}
159
174
 
160
175
  - **RPO (Recovery Point Objective)**
161
176
  - **Status:** {STATUS} {STATUS_ICON}
162
177
  - **Threshold:** {THRESHOLD_VALUE}
163
178
  - **Actual:** {ACTUAL_VALUE}
164
- - **Evidence:** {EVIDENCE_SOURCE}
179
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
180
+ - **Supports:** {SUPPORTED_OBSERVATION}
165
181
 
166
182
  ---
167
183
 
@@ -170,33 +186,37 @@ Note: This audit summarizes existing implementation evidence; it does not run te
170
186
  ### Test Coverage
171
187
 
172
188
  - **Status:** {STATUS} {STATUS_ICON}
173
- - **Threshold:** {THRESHOLD_DESCRIPTION} (e.g., "≥80% coverage")
189
+ - **Threshold:** {THRESHOLD_DESCRIPTION}
174
190
  - **Actual:** {ACTUAL_DESCRIPTION}
175
- - **Evidence:** {EVIDENCE_SOURCE} (e.g., "CI coverage report - coverage/lcov-report/index.html")
191
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
192
+ - **Supports:** {SUPPORTED_OBSERVATION}
176
193
  - **Findings:** {FINDINGS_DESCRIPTION}
177
194
 
178
195
  ### Code Duplication
179
196
 
180
197
  - **Status:** {STATUS} {STATUS_ICON}
181
- - **Threshold:** {THRESHOLD_DESCRIPTION} (e.g., "<5% duplication")
198
+ - **Threshold:** {THRESHOLD_DESCRIPTION}
182
199
  - **Actual:** {ACTUAL_DESCRIPTION}
183
- - **Evidence:** {EVIDENCE_SOURCE} (e.g., "jscpd report - reports/jscpd/jscpd-report.json")
200
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
201
+ - **Supports:** {SUPPORTED_OBSERVATION}
184
202
  - **Findings:** {FINDINGS_DESCRIPTION}
185
203
 
186
204
  ### Vulnerability Scan
187
205
 
188
206
  - **Status:** {STATUS} {STATUS_ICON}
189
- - **Threshold:** {THRESHOLD_DESCRIPTION} (e.g., "0 critical, 0 high vulnerabilities")
207
+ - **Threshold:** {THRESHOLD_DESCRIPTION}
190
208
  - **Actual:** {ACTUAL_DESCRIPTION}
191
- - **Evidence:** {EVIDENCE_SOURCE} (e.g., "npm audit CI job output")
209
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
210
+ - **Supports:** {SUPPORTED_OBSERVATION}
192
211
  - **Findings:** {FINDINGS_DESCRIPTION}
193
212
 
194
213
  ### Observability
195
214
 
196
215
  - **Status:** {STATUS} {STATUS_ICON}
197
- - **Threshold:** {THRESHOLD_DESCRIPTION} (e.g., "structured logging + error tracking configured")
216
+ - **Threshold:** {THRESHOLD_DESCRIPTION}
198
217
  - **Actual:** {ACTUAL_DESCRIPTION}
199
- - **Evidence:** {EVIDENCE_SOURCE}
218
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
219
+ - **Supports:** {SUPPORTED_OBSERVATION}
200
220
  - **Findings:** {FINDINGS_DESCRIPTION}
201
221
 
202
222
  ---
@@ -208,7 +228,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
208
228
  - **Status:** {STATUS} {STATUS_ICON}
209
229
  - **Threshold:** {THRESHOLD_DESCRIPTION}
210
230
  - **Actual:** {ACTUAL_DESCRIPTION}
211
- - **Evidence:** {EVIDENCE_SOURCE}
231
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
232
+ - **Supports:** {SUPPORTED_OBSERVATION}
212
233
  - **Findings:** {FINDINGS_DESCRIPTION}
213
234
 
214
235
  ### {CUSTOM_NFR_NAME_2}
@@ -216,7 +237,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
216
237
  - **Status:** {STATUS} {STATUS_ICON}
217
238
  - **Threshold:** {THRESHOLD_DESCRIPTION}
218
239
  - **Actual:** {ACTUAL_DESCRIPTION}
219
- - **Evidence:** {EVIDENCE_SOURCE}
240
+ - **Evidence:** {PROJECT_RELATIVE_PATH}
241
+ - **Supports:** {SUPPORTED_OBSERVATION}
220
242
  - **Findings:** {FINDINGS_DESCRIPTION}
221
243
 
222
244
  ---
@@ -49,7 +49,8 @@ Before deriving thresholds from raw documents, check if a `test-design` output e
49
49
 
50
50
  ## 1. Select Categories
51
51
 
52
- Use the ADR Quality Readiness Checklist (8 categories) as the elicitation source for thresholds:
52
+ Use the ADR Quality Readiness Checklist (8 categories) only as an elicitation
53
+ source for finding explicitly declared requirements:
53
54
 
54
55
  1. Testability & Automation
55
56
  2. Test Data Strategy
@@ -62,13 +63,51 @@ Use the ADR Quality Readiness Checklist (8 categories) as the elicitation source
62
63
 
63
64
  Add any `custom_nfr_categories` if provided.
64
65
 
66
+ The checklist never creates an assessment criterion by itself. Build a canonical
67
+ `declared_nfr_criteria` map containing only requirements in the four automated
68
+ audit domains: Security, Performance, Reliability, and Maintainability. Put
69
+ explicit requirements from every other category in a separate
70
+ `recorded_only_nfr_criteria` list. Both collections come only from the supplied
71
+ test-design plan, tech spec, PRD, story, or custom configuration.
72
+ For each criterion record a stable ID, its source-order position, exact label,
73
+ domain, threshold, and canonical threshold-source path. Preserve source order.
74
+ Do not add a broad checklist category, evidence gap, or CONCERNS finding when
75
+ the supplied requirements do not declare it.
76
+
77
+ Derive the stable ID mechanically from the domain plus the exact declared label:
78
+ lowercase both, replace each run of non-alphanumeric characters with one hyphen,
79
+ and trim leading or trailing hyphens. Stop with an explicit ambiguity error if
80
+ two criteria produce the same ID. Repeated runs over the same requirements must
81
+ produce the same IDs and order.
82
+
83
+ ```json
84
+ {
85
+ "{NFR_DOMAIN}": [
86
+ {
87
+ "id": "{STABLE_CRITERION_ID}",
88
+ "order": "{SOURCE_ORDER}",
89
+ "label": "{DECLARED_REQUIREMENT_LABEL}",
90
+ "threshold": "{DECLARED_THRESHOLD_OR_UNKNOWN}",
91
+ "threshold_source": "{PATH_RELATIVE_TO_SUPPLIED_PROJECT_ROOT}"
92
+ }
93
+ ]
94
+ }
95
+ ```
96
+
97
+ Each recorded-only entry uses the same fields and stable-ID derivation, plus
98
+ `assessment_mode: "recorded-only"`. Recorded-only criteria are preserved for
99
+ publication with their exact threshold and threshold source. They never enter a
100
+ worker payload, finding, domain status, evidence gap, risk rollup, compliance
101
+ summary, or gate decision.
102
+
65
103
  **Note:** This workflow's automated evidence audit in Step 4 evaluates exactly four domains: Security, Performance, Reliability, and Maintainability. Security and Performance draw their thresholds from the ADR-8 list above. The Reliability worker consumes availability, error-rate, fault-tolerance, and MTTR thresholds. Disaster Recovery remains a separate ADR-8 category for RTO, RPO, backup, restore, and regional recovery evidence; no automated Step 4 worker evaluates it. Maintainability criteria (test coverage, code duplication, dependency vulnerabilities, structured logging, and error tracking) come directly from `nfr-criteria.md`. Thresholds gathered here for Test Data Strategy, Disaster Recovery, Monitorability, QoS/QoE, and Deployability are recorded for the report and are not evaluated by an automated subagent in this workflow.
66
104
 
67
105
  ---
68
106
 
69
107
  ## 2. Define Thresholds
70
108
 
71
- For each category, use thresholds from the test-design NFR plan (step 0) where available. For any remaining UNKNOWN or missing thresholds, extract from:
109
+ For each declared criterion, use thresholds from the test-design NFR plan (step 0) where available. For any remaining UNKNOWN or missing thresholds, extract
110
+ from:
72
111
 
73
112
  - tech-spec (primary)
74
113
  - PRD (secondary)
@@ -76,11 +115,19 @@ For each category, use thresholds from the test-design NFR plan (step 0) where a
76
115
 
77
116
  If a threshold is still unknown after checking all sources, mark it **UNKNOWN** and plan to report **CONCERNS**.
78
117
 
118
+ A requirements document proves only the declared target. Record it as
119
+ `threshold_source`; never treat it as implementation evidence or place it in a
120
+ finding's `evidence` array.
121
+
79
122
  ---
80
123
 
81
124
  ## 3. Confirm NFR Matrix
82
125
 
83
- List each NFR category with its threshold or UNKNOWN status.
126
+ List each declared NFR criterion with its stable ID, domain, source order,
127
+ threshold or UNKNOWN status, threshold source, and assessment mode. The ordered
128
+ `declared_nfr_criteria` map is the complete worker assessment scope passed to
129
+ Steps 3 through 5. The ordered `recorded_only_nfr_criteria` list is passed to
130
+ Step 5 solely for the recorded-only report table.
84
131
 
85
132
  ---
86
133
 
@@ -37,13 +37,79 @@ Collect measurable evidence to evaluate each NFR category.
37
37
 
38
38
  ## 1. Evidence Sources
39
39
 
40
- Collect evidence for:
40
+ For the criteria present in `declared_nfr_criteria`, collect relevant evidence
41
+ from these source families:
41
42
 
42
43
  - **Performance**: load tests, metrics, response time data
43
44
  - **Security**: scans, auth tests, vuln reports
44
45
  - **Reliability**: error rates, burn-in runs, failover tests
45
46
  - **Maintainability**: CI coverage report, jscpd duplication report, npm audit results, structured logging and error-tracking config
46
- - **Other categories**: logs, monitoring, DR drills, deployability checks
47
+ - **Other declared categories**: logs, monitoring, DR drills, deployability checks
48
+
49
+ These source families are discovery hints. An item in this list that has no
50
+ matching declared criterion creates no finding and no evidence gap.
51
+
52
+ ### Build the supplied-evidence ledger
53
+
54
+ First identify `supplied_project_root`: the root directory of the project being
55
+ audited, containing its supplied `docs`, `evidence`, and configuration paths.
56
+ This can be a child of the runner workspace. Build one canonical
57
+ `supplied_evidence_ledger` before any domain evaluation. The ledger is the
58
+ complete allowlist for factual NFR claims. Include only files that exist under
59
+ `supplied_project_root` and were supplied for this audit. Exclude the skill,
60
+ templates, temporary worker output, and `{outputFile}`.
61
+
62
+ For every ledger entry, record:
63
+
64
+ ```json
65
+ {
66
+ "path": "{PROJECT_RELATIVE_PATH}",
67
+ "observations": [
68
+ {
69
+ "criterion_id": "{STABLE_CRITERION_ID_FROM_DECLARED_SCOPE}",
70
+ "supports": "{OBSERVATION_SUPPORTED_BY_THIS_FILE}"
71
+ }
72
+ ],
73
+ "domains": ["{NFR_DOMAIN}"],
74
+ "source": "supplied",
75
+ "source_type": "implementation-evidence"
76
+ }
77
+ ```
78
+
79
+ Rules:
80
+
81
+ - `path` is relative to `supplied_project_root`, uses `/` separators, and has no
82
+ leading `./`, absolute prefix, or supplied project-directory prefix. For a
83
+ supplied root ending in `{PROJECT_DIRECTORY}`, the one canonical spelling is
84
+ `evidence/{FILE}`, never `{PROJECT_DIRECTORY}/evidence/{FILE}`.
85
+ - Resolve the canonical path under `supplied_project_root`, confirm it remains
86
+ inside that root, and confirm it is a regular file before adding it.
87
+ - Every `observations` item binds one supported statement to the exact stable
88
+ `criterion_id` from `declared_nfr_criteria`. Read the file before recording
89
+ the observation. A statement that supports two declared criteria is recorded
90
+ once for each criterion ID.
91
+ - Keep requirements and planning documents in
92
+ `declared_nfr_criteria[].threshold_source`. Exclude them from the
93
+ implementation-evidence ledger. A requirement or threshold proves the target
94
+ only. It never proves implementation, an achieved measurement, or PASS.
95
+ - Sort ledger entries by `path`. Within each entry, de-duplicate observations by
96
+ the pair `criterion_id` plus `supports`, then sort by `criterion_id` and
97
+ `supports`. Order `domains` as security, performance, reliability,
98
+ maintainability. Equivalent supplied evidence must produce identical ledger
99
+ bytes.
100
+ - Record an explicit evidence gap only for a criterion in
101
+ `declared_nfr_criteria` whose required implementation observation has no
102
+ supporting ledger observation. Emit exactly one structured gap in declared
103
+ source order:
104
+ `{"criterion_id":"{criterion.id}","message":"{criterion.label}: no supplied implementation evidence"}`.
105
+ Never split one
106
+ unsupported criterion into narrower gaps or combine several criteria into a
107
+ broader gap.
108
+ - Pass the complete ledger and gaps to Step 4 as
109
+ `subagentContext.supplied_project_root`,
110
+ `subagentContext.declared_nfr_criteria`,
111
+ `subagentContext.supplied_evidence_ledger` and
112
+ `subagentContext.evidence_gaps`.
47
113
 
48
114
  ---
49
115
 
@@ -68,7 +134,11 @@ Store artifacts under `{test_artifacts}/nfr/`
68
134
 
69
135
  ## 3. Evidence Gaps
70
136
 
137
+ Here, "category" means one criterion present in `declared_nfr_criteria`.
71
138
  If evidence is missing for a category, mark that category as **CONCERNS**.
139
+ Name the missing observation in `evidence_gaps`.
140
+ Ignore discovery-checklist items absent from `declared_nfr_criteria`. Never add
141
+ a remembered, inferred, or example path or value to the ledger.
72
142
 
73
143
  ---
74
144
 
@@ -55,7 +55,11 @@ const parseBooleanFlag = (value, defaultValue = true) => {
55
55
  const subagentContext = {
56
56
  system_context: /* from Step 1 */,
57
57
  nfr_thresholds: /* from Step 2 */,
58
+ declared_nfr_criteria: /* ordered, explicit assessment scope from Step 2 */,
58
59
  evidence_gathered: /* from Step 3 */,
60
+ supplied_project_root: /* audited project root from Step 3 */,
61
+ supplied_evidence_ledger: /* canonical ledger from Step 3 */,
62
+ evidence_gaps: /* explicit gaps from Step 3 */,
59
63
  config: {
60
64
  execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
61
65
  capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
@@ -141,6 +145,22 @@ If probing is disabled, honor the requested mode strictly. If that mode cannot b
141
145
 
142
146
  ### 3. Dispatch 4 NFR Workers
143
147
 
148
+ Every worker receives the complete `supplied_evidence_ledger` and
149
+ `evidence_gaps`, plus the ordered `declared_nfr_criteria` scope. A worker emits
150
+ exactly one finding for each declared criterion in its domain, in declared
151
+ source order. It must omit every broad checklist or worker category absent from
152
+ that scope; omitted categories create no gap and cannot affect domain status.
153
+ A worker may cite only an exact `path` from the implementation-evidence ledger
154
+ and may state only an observation listed in that entry's `supports` array.
155
+ Threshold sources remain separate and never appear in a finding's `evidence`
156
+ array. A declared criterion with no matching implementation support cannot
157
+ receive PASS. The worker must emit CONCERNS, or the workflow's declared
158
+ undecidable state, and add the missing observation to the evidence gaps.
159
+
160
+ Before writing output, de-duplicate and sort every finding's evidence by
161
+ `path`, then `supports`; de-duplicate gaps while preserving declared criterion
162
+ order. These rules apply in every execution mode.
163
+
144
164
  #### Subagent A: Security Evidence Audit
145
165
 
146
166
  - File: `./step-04a-subagent-security.md`