bmad-method-test-architecture-enterprise 1.27.2-next.3 → 1.27.2-next.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +2 -1
- package/package.json +3 -2
- package/src/workflows/testarch/bmad-testarch-nfr/nfr-report-template.md +59 -37
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-02-define-thresholds.md +50 -3
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-03-gather-evidence.md +72 -2
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04-evaluate-and-score.md +20 -0
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04a-subagent-security.md +55 -75
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04b-subagent-performance.md +56 -48
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04c-subagent-reliability.md +56 -44
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04d-subagent-maintainability.md +56 -55
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04e-aggregate-nfr.md +275 -72
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-05-generate-report.md +54 -0
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "1.27.2-next.
|
|
34
|
+
"version": "1.27.2-next.4",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/README.md
CHANGED
|
@@ -392,7 +392,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
|
|
|
392
392
|
|
|
393
393
|
### Deterministic Checks
|
|
394
394
|
|
|
395
|
-
`npm test` chains seventy-
|
|
395
|
+
`npm test` chains seventy-five checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Eleven of the seventy-five keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
|
|
396
396
|
|
|
397
397
|
- `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
|
|
398
398
|
- `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, and the fragment-selection case count. A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
|
|
@@ -403,6 +403,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
|
|
|
403
403
|
- `test:eval-routing-boundaries` mutates the facts, candidate codes, and missing deciding information in each source-bound routing ambiguity rule. It also rejects duplicate and orphan boundaries, plus any boundary that would turn a clear intent into a clarification.
|
|
404
404
|
- `test:eval-schemas` checks this repository's own eval suite manifest against its schema, confirms that every threshold it declares is the threshold the harness actually applies, that the runner capabilities it declares are the ones the harness grants its runner, and that the preflight argv it declares fails on a missing runner and passes on a present one, and fails when a TEA skill has neither a behavioral suite nor a deferred declaration. A suite list that omits a skill reads as coverage, so the omission has to be an error rather than a silence.
|
|
405
405
|
- `test:eval-diagnostics` checks that current result writers emit bounded case and repetition diagnostics, that quality and environment outcomes stay separate, that unstable signatures remain visible, and that the protected 1.3.0 baseline remains readable and byte-for-byte unchanged.
|
|
406
|
+
- `test:nfr-evidence-guidance` holds every NFR worker and the publication audit to the supplied-evidence ledger, exercises supported, missing, and mismatched evidence, rejects contaminating examples, and binds the protected baseline and focused remediation records by digest.
|
|
406
407
|
- `test:contract-oracles` evaluates every oracle in every eval contract with `eval-quality`'s own evaluator, over the stored replay outputs and over constructed selections for all 24 fragment-selection cases, and fails when an oracle faults or disagrees with the harness scorer on the same evidence. The first run found that every regex oracle in the `test-review` contract was refused by the evaluator before it matched anything.
|
|
407
408
|
|
|
408
409
|
These checks produce the same answer from the same repository state. They need no agent credential, network call, or model budget. `test:eval-data` runs through `npm test`, the local pre-commit hook, pull-request quality checks, and the publish workflow.
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.27.2-next.
|
|
4
|
+
"version": "1.27.2-next.4",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
|
@@ -83,7 +83,7 @@
|
|
|
83
83
|
"release:minor": "gh workflow run publish.yaml -f channel=latest -f bump=minor",
|
|
84
84
|
"release:next": "gh workflow run publish.yaml -f channel=next",
|
|
85
85
|
"release:patch": "gh workflow run publish.yaml -f channel=latest -f bump=patch",
|
|
86
|
-
"test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
|
|
86
|
+
"test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:nfr-evidence-guidance && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
|
|
87
87
|
"test:atdd-isolation": "node test/test-atdd-isolation.js",
|
|
88
88
|
"test:atdd-net-guard": "node test/test-atdd-net-guard.js",
|
|
89
89
|
"test:atdd-scoring": "node test/test-atdd-scoring.js",
|
|
@@ -142,6 +142,7 @@
|
|
|
142
142
|
"test:lineage": "eval-quality-gates field-ownership",
|
|
143
143
|
"test:lockfile-age": "eval-quality-gates lockfile-age",
|
|
144
144
|
"test:lockfile-age-cache": "node test/test-generate-lockfile-age-cache.js",
|
|
145
|
+
"test:nfr-evidence-guidance": "node test/test-nfr-evidence-guidance.js",
|
|
145
146
|
"test:nfr-validate-corpus": "node test/test-nfr-validate-corpus.js",
|
|
146
147
|
"test:port-totality": "node test/test-port-totality.js",
|
|
147
148
|
"test:probe-conformance": "node test/test-probe-conformance.js",
|
|
@@ -35,7 +35,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
35
35
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
36
36
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
37
37
|
- **Actual:** {ACTUAL_VALUE}
|
|
38
|
-
- **Evidence:** {
|
|
38
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
39
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
39
40
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
40
41
|
|
|
41
42
|
### Throughput
|
|
@@ -43,7 +44,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
43
44
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
44
45
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
45
46
|
- **Actual:** {ACTUAL_VALUE}
|
|
46
|
-
- **Evidence:** {
|
|
47
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
48
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
47
49
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
48
50
|
|
|
49
51
|
### Resource Usage
|
|
@@ -52,13 +54,15 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
52
54
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
53
55
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
54
56
|
- **Actual:** {ACTUAL_VALUE}
|
|
55
|
-
- **Evidence:** {
|
|
57
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
58
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
56
59
|
|
|
57
60
|
- **Memory Usage**
|
|
58
61
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
59
62
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
60
63
|
- **Actual:** {ACTUAL_VALUE}
|
|
61
|
-
- **Evidence:** {
|
|
64
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
65
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
62
66
|
|
|
63
67
|
---
|
|
64
68
|
|
|
@@ -69,7 +73,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
69
73
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
70
74
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
71
75
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
72
|
-
- **Evidence:** {
|
|
76
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
77
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
73
78
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
74
79
|
- **Recommendation:** {RECOMMENDATION} (if CONCERNS or FAIL)
|
|
75
80
|
|
|
@@ -78,7 +83,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
78
83
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
79
84
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
80
85
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
81
|
-
- **Evidence:** {
|
|
86
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
87
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
82
88
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
83
89
|
|
|
84
90
|
### Data Protection
|
|
@@ -86,23 +92,26 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
86
92
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
87
93
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
88
94
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
89
|
-
- **Evidence:** {
|
|
95
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
96
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
90
97
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
91
98
|
|
|
92
99
|
### Vulnerability Management
|
|
93
100
|
|
|
94
101
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
95
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
96
|
-
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
97
|
-
- **Evidence:** {
|
|
102
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
103
|
+
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
104
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
105
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
98
106
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
99
107
|
|
|
100
108
|
### Compliance (if applicable)
|
|
101
109
|
|
|
102
110
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
103
|
-
- **Standards:** {COMPLIANCE_STANDARDS}
|
|
111
|
+
- **Standards:** {COMPLIANCE_STANDARDS}
|
|
104
112
|
- **Actual:** {ACTUAL_COMPLIANCE_STATUS}
|
|
105
|
-
- **Evidence:** {
|
|
113
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
114
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
106
115
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
107
116
|
|
|
108
117
|
---
|
|
@@ -112,25 +121,28 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
112
121
|
### Availability (Uptime)
|
|
113
122
|
|
|
114
123
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
115
|
-
- **Threshold:** {THRESHOLD_VALUE}
|
|
116
|
-
- **Actual:** {ACTUAL_VALUE}
|
|
117
|
-
- **Evidence:** {
|
|
124
|
+
- **Threshold:** {THRESHOLD_VALUE}
|
|
125
|
+
- **Actual:** {ACTUAL_VALUE}
|
|
126
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
127
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
118
128
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
119
129
|
|
|
120
130
|
### Error Rate
|
|
121
131
|
|
|
122
132
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
123
|
-
- **Threshold:** {THRESHOLD_VALUE}
|
|
124
|
-
- **Actual:** {ACTUAL_VALUE}
|
|
125
|
-
- **Evidence:** {
|
|
133
|
+
- **Threshold:** {THRESHOLD_VALUE}
|
|
134
|
+
- **Actual:** {ACTUAL_VALUE}
|
|
135
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
136
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
126
137
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
127
138
|
|
|
128
139
|
### MTTR (Mean Time To Recovery)
|
|
129
140
|
|
|
130
141
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
131
|
-
- **Threshold:** {THRESHOLD_VALUE}
|
|
132
|
-
- **Actual:** {ACTUAL_VALUE}
|
|
133
|
-
- **Evidence:** {
|
|
142
|
+
- **Threshold:** {THRESHOLD_VALUE}
|
|
143
|
+
- **Actual:** {ACTUAL_VALUE}
|
|
144
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
145
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
134
146
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
135
147
|
|
|
136
148
|
### Fault Tolerance
|
|
@@ -138,15 +150,17 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
138
150
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
139
151
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
140
152
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
141
|
-
- **Evidence:** {
|
|
153
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
154
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
142
155
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
143
156
|
|
|
144
157
|
### CI Burn-In (Stability)
|
|
145
158
|
|
|
146
159
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
147
|
-
- **Threshold:** {THRESHOLD_VALUE}
|
|
148
|
-
- **Actual:** {ACTUAL_VALUE}
|
|
149
|
-
- **Evidence:** {
|
|
160
|
+
- **Threshold:** {THRESHOLD_VALUE}
|
|
161
|
+
- **Actual:** {ACTUAL_VALUE}
|
|
162
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
163
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
150
164
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
151
165
|
|
|
152
166
|
### Disaster Recovery (if applicable)
|
|
@@ -155,13 +169,15 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
155
169
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
156
170
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
157
171
|
- **Actual:** {ACTUAL_VALUE}
|
|
158
|
-
- **Evidence:** {
|
|
172
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
173
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
159
174
|
|
|
160
175
|
- **RPO (Recovery Point Objective)**
|
|
161
176
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
162
177
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
163
178
|
- **Actual:** {ACTUAL_VALUE}
|
|
164
|
-
- **Evidence:** {
|
|
179
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
180
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
165
181
|
|
|
166
182
|
---
|
|
167
183
|
|
|
@@ -170,33 +186,37 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
170
186
|
### Test Coverage
|
|
171
187
|
|
|
172
188
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
173
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
189
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
174
190
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
175
|
-
- **Evidence:** {
|
|
191
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
192
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
176
193
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
177
194
|
|
|
178
195
|
### Code Duplication
|
|
179
196
|
|
|
180
197
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
181
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
198
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
182
199
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
183
|
-
- **Evidence:** {
|
|
200
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
201
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
184
202
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
185
203
|
|
|
186
204
|
### Vulnerability Scan
|
|
187
205
|
|
|
188
206
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
189
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
207
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
190
208
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
191
|
-
- **Evidence:** {
|
|
209
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
210
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
192
211
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
193
212
|
|
|
194
213
|
### Observability
|
|
195
214
|
|
|
196
215
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
197
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
216
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
198
217
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
199
|
-
- **Evidence:** {
|
|
218
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
219
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
200
220
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
201
221
|
|
|
202
222
|
---
|
|
@@ -208,7 +228,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
208
228
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
209
229
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
210
230
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
211
|
-
- **Evidence:** {
|
|
231
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
232
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
212
233
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
213
234
|
|
|
214
235
|
### {CUSTOM_NFR_NAME_2}
|
|
@@ -216,7 +237,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
216
237
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
217
238
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
218
239
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
219
|
-
- **Evidence:** {
|
|
240
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
241
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
220
242
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
221
243
|
|
|
222
244
|
---
|
|
@@ -49,7 +49,8 @@ Before deriving thresholds from raw documents, check if a `test-design` output e
|
|
|
49
49
|
|
|
50
50
|
## 1. Select Categories
|
|
51
51
|
|
|
52
|
-
Use the ADR Quality Readiness Checklist (8 categories) as
|
|
52
|
+
Use the ADR Quality Readiness Checklist (8 categories) only as an elicitation
|
|
53
|
+
source for finding explicitly declared requirements:
|
|
53
54
|
|
|
54
55
|
1. Testability & Automation
|
|
55
56
|
2. Test Data Strategy
|
|
@@ -62,13 +63,51 @@ Use the ADR Quality Readiness Checklist (8 categories) as the elicitation source
|
|
|
62
63
|
|
|
63
64
|
Add any `custom_nfr_categories` if provided.
|
|
64
65
|
|
|
66
|
+
The checklist never creates an assessment criterion by itself. Build a canonical
|
|
67
|
+
`declared_nfr_criteria` map containing only requirements in the four automated
|
|
68
|
+
audit domains: Security, Performance, Reliability, and Maintainability. Put
|
|
69
|
+
explicit requirements from every other category in a separate
|
|
70
|
+
`recorded_only_nfr_criteria` list. Both collections come only from the supplied
|
|
71
|
+
test-design plan, tech spec, PRD, story, or custom configuration.
|
|
72
|
+
For each criterion record a stable ID, its source-order position, exact label,
|
|
73
|
+
domain, threshold, and canonical threshold-source path. Preserve source order.
|
|
74
|
+
Do not add a broad checklist category, evidence gap, or CONCERNS finding when
|
|
75
|
+
the supplied requirements do not declare it.
|
|
76
|
+
|
|
77
|
+
Derive the stable ID mechanically from the domain plus the exact declared label:
|
|
78
|
+
lowercase both, replace each run of non-alphanumeric characters with one hyphen,
|
|
79
|
+
and trim leading or trailing hyphens. Stop with an explicit ambiguity error if
|
|
80
|
+
two criteria produce the same ID. Repeated runs over the same requirements must
|
|
81
|
+
produce the same IDs and order.
|
|
82
|
+
|
|
83
|
+
```json
|
|
84
|
+
{
|
|
85
|
+
"{NFR_DOMAIN}": [
|
|
86
|
+
{
|
|
87
|
+
"id": "{STABLE_CRITERION_ID}",
|
|
88
|
+
"order": "{SOURCE_ORDER}",
|
|
89
|
+
"label": "{DECLARED_REQUIREMENT_LABEL}",
|
|
90
|
+
"threshold": "{DECLARED_THRESHOLD_OR_UNKNOWN}",
|
|
91
|
+
"threshold_source": "{PATH_RELATIVE_TO_SUPPLIED_PROJECT_ROOT}"
|
|
92
|
+
}
|
|
93
|
+
]
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Each recorded-only entry uses the same fields and stable-ID derivation, plus
|
|
98
|
+
`assessment_mode: "recorded-only"`. Recorded-only criteria are preserved for
|
|
99
|
+
publication with their exact threshold and threshold source. They never enter a
|
|
100
|
+
worker payload, finding, domain status, evidence gap, risk rollup, compliance
|
|
101
|
+
summary, or gate decision.
|
|
102
|
+
|
|
65
103
|
**Note:** This workflow's automated evidence audit in Step 4 evaluates exactly four domains: Security, Performance, Reliability, and Maintainability. Security and Performance draw their thresholds from the ADR-8 list above. The Reliability worker consumes availability, error-rate, fault-tolerance, and MTTR thresholds. Disaster Recovery remains a separate ADR-8 category for RTO, RPO, backup, restore, and regional recovery evidence; no automated Step 4 worker evaluates it. Maintainability criteria (test coverage, code duplication, dependency vulnerabilities, structured logging, and error tracking) come directly from `nfr-criteria.md`. Thresholds gathered here for Test Data Strategy, Disaster Recovery, Monitorability, QoS/QoE, and Deployability are recorded for the report and are not evaluated by an automated subagent in this workflow.
|
|
66
104
|
|
|
67
105
|
---
|
|
68
106
|
|
|
69
107
|
## 2. Define Thresholds
|
|
70
108
|
|
|
71
|
-
For each
|
|
109
|
+
For each declared criterion, use thresholds from the test-design NFR plan (step 0) where available. For any remaining UNKNOWN or missing thresholds, extract
|
|
110
|
+
from:
|
|
72
111
|
|
|
73
112
|
- tech-spec (primary)
|
|
74
113
|
- PRD (secondary)
|
|
@@ -76,11 +115,19 @@ For each category, use thresholds from the test-design NFR plan (step 0) where a
|
|
|
76
115
|
|
|
77
116
|
If a threshold is still unknown after checking all sources, mark it **UNKNOWN** and plan to report **CONCERNS**.
|
|
78
117
|
|
|
118
|
+
A requirements document proves only the declared target. Record it as
|
|
119
|
+
`threshold_source`; never treat it as implementation evidence or place it in a
|
|
120
|
+
finding's `evidence` array.
|
|
121
|
+
|
|
79
122
|
---
|
|
80
123
|
|
|
81
124
|
## 3. Confirm NFR Matrix
|
|
82
125
|
|
|
83
|
-
List each NFR
|
|
126
|
+
List each declared NFR criterion with its stable ID, domain, source order,
|
|
127
|
+
threshold or UNKNOWN status, threshold source, and assessment mode. The ordered
|
|
128
|
+
`declared_nfr_criteria` map is the complete worker assessment scope passed to
|
|
129
|
+
Steps 3 through 5. The ordered `recorded_only_nfr_criteria` list is passed to
|
|
130
|
+
Step 5 solely for the recorded-only report table.
|
|
84
131
|
|
|
85
132
|
---
|
|
86
133
|
|
|
@@ -37,13 +37,79 @@ Collect measurable evidence to evaluate each NFR category.
|
|
|
37
37
|
|
|
38
38
|
## 1. Evidence Sources
|
|
39
39
|
|
|
40
|
-
|
|
40
|
+
For the criteria present in `declared_nfr_criteria`, collect relevant evidence
|
|
41
|
+
from these source families:
|
|
41
42
|
|
|
42
43
|
- **Performance**: load tests, metrics, response time data
|
|
43
44
|
- **Security**: scans, auth tests, vuln reports
|
|
44
45
|
- **Reliability**: error rates, burn-in runs, failover tests
|
|
45
46
|
- **Maintainability**: CI coverage report, jscpd duplication report, npm audit results, structured logging and error-tracking config
|
|
46
|
-
- **Other categories**: logs, monitoring, DR drills, deployability checks
|
|
47
|
+
- **Other declared categories**: logs, monitoring, DR drills, deployability checks
|
|
48
|
+
|
|
49
|
+
These source families are discovery hints. An item in this list that has no
|
|
50
|
+
matching declared criterion creates no finding and no evidence gap.
|
|
51
|
+
|
|
52
|
+
### Build the supplied-evidence ledger
|
|
53
|
+
|
|
54
|
+
First identify `supplied_project_root`: the root directory of the project being
|
|
55
|
+
audited, containing its supplied `docs`, `evidence`, and configuration paths.
|
|
56
|
+
This can be a child of the runner workspace. Build one canonical
|
|
57
|
+
`supplied_evidence_ledger` before any domain evaluation. The ledger is the
|
|
58
|
+
complete allowlist for factual NFR claims. Include only files that exist under
|
|
59
|
+
`supplied_project_root` and were supplied for this audit. Exclude the skill,
|
|
60
|
+
templates, temporary worker output, and `{outputFile}`.
|
|
61
|
+
|
|
62
|
+
For every ledger entry, record:
|
|
63
|
+
|
|
64
|
+
```json
|
|
65
|
+
{
|
|
66
|
+
"path": "{PROJECT_RELATIVE_PATH}",
|
|
67
|
+
"observations": [
|
|
68
|
+
{
|
|
69
|
+
"criterion_id": "{STABLE_CRITERION_ID_FROM_DECLARED_SCOPE}",
|
|
70
|
+
"supports": "{OBSERVATION_SUPPORTED_BY_THIS_FILE}"
|
|
71
|
+
}
|
|
72
|
+
],
|
|
73
|
+
"domains": ["{NFR_DOMAIN}"],
|
|
74
|
+
"source": "supplied",
|
|
75
|
+
"source_type": "implementation-evidence"
|
|
76
|
+
}
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Rules:
|
|
80
|
+
|
|
81
|
+
- `path` is relative to `supplied_project_root`, uses `/` separators, and has no
|
|
82
|
+
leading `./`, absolute prefix, or supplied project-directory prefix. For a
|
|
83
|
+
supplied root ending in `{PROJECT_DIRECTORY}`, the one canonical spelling is
|
|
84
|
+
`evidence/{FILE}`, never `{PROJECT_DIRECTORY}/evidence/{FILE}`.
|
|
85
|
+
- Resolve the canonical path under `supplied_project_root`, confirm it remains
|
|
86
|
+
inside that root, and confirm it is a regular file before adding it.
|
|
87
|
+
- Every `observations` item binds one supported statement to the exact stable
|
|
88
|
+
`criterion_id` from `declared_nfr_criteria`. Read the file before recording
|
|
89
|
+
the observation. A statement that supports two declared criteria is recorded
|
|
90
|
+
once for each criterion ID.
|
|
91
|
+
- Keep requirements and planning documents in
|
|
92
|
+
`declared_nfr_criteria[].threshold_source`. Exclude them from the
|
|
93
|
+
implementation-evidence ledger. A requirement or threshold proves the target
|
|
94
|
+
only. It never proves implementation, an achieved measurement, or PASS.
|
|
95
|
+
- Sort ledger entries by `path`. Within each entry, de-duplicate observations by
|
|
96
|
+
the pair `criterion_id` plus `supports`, then sort by `criterion_id` and
|
|
97
|
+
`supports`. Order `domains` as security, performance, reliability,
|
|
98
|
+
maintainability. Equivalent supplied evidence must produce identical ledger
|
|
99
|
+
bytes.
|
|
100
|
+
- Record an explicit evidence gap only for a criterion in
|
|
101
|
+
`declared_nfr_criteria` whose required implementation observation has no
|
|
102
|
+
supporting ledger observation. Emit exactly one structured gap in declared
|
|
103
|
+
source order:
|
|
104
|
+
`{"criterion_id":"{criterion.id}","message":"{criterion.label}: no supplied implementation evidence"}`.
|
|
105
|
+
Never split one
|
|
106
|
+
unsupported criterion into narrower gaps or combine several criteria into a
|
|
107
|
+
broader gap.
|
|
108
|
+
- Pass the complete ledger and gaps to Step 4 as
|
|
109
|
+
`subagentContext.supplied_project_root`,
|
|
110
|
+
`subagentContext.declared_nfr_criteria`,
|
|
111
|
+
`subagentContext.supplied_evidence_ledger` and
|
|
112
|
+
`subagentContext.evidence_gaps`.
|
|
47
113
|
|
|
48
114
|
---
|
|
49
115
|
|
|
@@ -68,7 +134,11 @@ Store artifacts under `{test_artifacts}/nfr/`
|
|
|
68
134
|
|
|
69
135
|
## 3. Evidence Gaps
|
|
70
136
|
|
|
137
|
+
Here, "category" means one criterion present in `declared_nfr_criteria`.
|
|
71
138
|
If evidence is missing for a category, mark that category as **CONCERNS**.
|
|
139
|
+
Name the missing observation in `evidence_gaps`.
|
|
140
|
+
Ignore discovery-checklist items absent from `declared_nfr_criteria`. Never add
|
|
141
|
+
a remembered, inferred, or example path or value to the ledger.
|
|
72
142
|
|
|
73
143
|
---
|
|
74
144
|
|
|
@@ -55,7 +55,11 @@ const parseBooleanFlag = (value, defaultValue = true) => {
|
|
|
55
55
|
const subagentContext = {
|
|
56
56
|
system_context: /* from Step 1 */,
|
|
57
57
|
nfr_thresholds: /* from Step 2 */,
|
|
58
|
+
declared_nfr_criteria: /* ordered, explicit assessment scope from Step 2 */,
|
|
58
59
|
evidence_gathered: /* from Step 3 */,
|
|
60
|
+
supplied_project_root: /* audited project root from Step 3 */,
|
|
61
|
+
supplied_evidence_ledger: /* canonical ledger from Step 3 */,
|
|
62
|
+
evidence_gaps: /* explicit gaps from Step 3 */,
|
|
59
63
|
config: {
|
|
60
64
|
execution_mode: config.tea_execution_mode || 'auto', // "auto" | "subagent" | "agent-team" | "sequential"
|
|
61
65
|
capability_probe: parseBooleanFlag(config.tea_capability_probe, true), // supports booleans and "false"/"true" strings
|
|
@@ -141,6 +145,22 @@ If probing is disabled, honor the requested mode strictly. If that mode cannot b
|
|
|
141
145
|
|
|
142
146
|
### 3. Dispatch 4 NFR Workers
|
|
143
147
|
|
|
148
|
+
Every worker receives the complete `supplied_evidence_ledger` and
|
|
149
|
+
`evidence_gaps`, plus the ordered `declared_nfr_criteria` scope. A worker emits
|
|
150
|
+
exactly one finding for each declared criterion in its domain, in declared
|
|
151
|
+
source order. It must omit every broad checklist or worker category absent from
|
|
152
|
+
that scope; omitted categories create no gap and cannot affect domain status.
|
|
153
|
+
A worker may cite only an exact `path` from the implementation-evidence ledger
|
|
154
|
+
and may state only an observation listed in that entry's `supports` array.
|
|
155
|
+
Threshold sources remain separate and never appear in a finding's `evidence`
|
|
156
|
+
array. A declared criterion with no matching implementation support cannot
|
|
157
|
+
receive PASS. The worker must emit CONCERNS, or the workflow's declared
|
|
158
|
+
undecidable state, and add the missing observation to the evidence gaps.
|
|
159
|
+
|
|
160
|
+
Before writing output, de-duplicate and sort every finding's evidence by
|
|
161
|
+
`path`, then `supports`; de-duplicate gaps while preserving declared criterion
|
|
162
|
+
order. These rules apply in every execution mode.
|
|
163
|
+
|
|
144
164
|
#### Subagent A: Security Evidence Audit
|
|
145
165
|
|
|
146
166
|
- File: `./step-04a-subagent-security.md`
|