bmad-method-test-architecture-enterprise 1.27.2-next.2 → 1.27.2-next.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +2 -1
- package/package.json +3 -2
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-01-preflight-and-context.md +8 -46
- package/src/workflows/testarch/bmad-testarch-automate/steps-c/step-01-preflight-and-context.md +15 -54
- package/src/workflows/testarch/bmad-testarch-ci/steps-c/step-02-generate-pipeline.md +6 -6
- package/src/workflows/testarch/bmad-testarch-ci/steps-c/step-03-configure-quality-gates.md +8 -2
- package/src/workflows/testarch/bmad-testarch-framework/steps-c/step-03-scaffold-framework.md +7 -1
- package/src/workflows/testarch/bmad-testarch-nfr/nfr-report-template.md +59 -37
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-01-load-context.md +2 -8
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-02-define-thresholds.md +50 -3
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-03-gather-evidence.md +72 -2
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04-evaluate-and-score.md +20 -0
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04a-subagent-security.md +55 -75
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04b-subagent-performance.md +56 -48
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04c-subagent-reliability.md +56 -44
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04d-subagent-maintainability.md +56 -55
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04e-aggregate-nfr.md +275 -72
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-05-generate-report.md +54 -0
- package/src/workflows/testarch/bmad-testarch-test-design/steps-c/step-02-load-context.md +7 -45
- package/src/workflows/testarch/bmad-testarch-test-review/steps-c/step-01-load-context.md +8 -54
- package/src/workflows/testarch/bmad-testarch-trace/steps-c/step-01-load-context.md +4 -0
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "1.27.2-next.
|
|
34
|
+
"version": "1.27.2-next.4",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/README.md
CHANGED
|
@@ -392,7 +392,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
|
|
|
392
392
|
|
|
393
393
|
### Deterministic Checks
|
|
394
394
|
|
|
395
|
-
`npm test` chains seventy-
|
|
395
|
+
`npm test` chains seventy-five checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Eleven of the seventy-five keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
|
|
396
396
|
|
|
397
397
|
- `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
|
|
398
398
|
- `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, and the fragment-selection case count. A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
|
|
@@ -403,6 +403,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
|
|
|
403
403
|
- `test:eval-routing-boundaries` mutates the facts, candidate codes, and missing deciding information in each source-bound routing ambiguity rule. It also rejects duplicate and orphan boundaries, plus any boundary that would turn a clear intent into a clarification.
|
|
404
404
|
- `test:eval-schemas` checks this repository's own eval suite manifest against its schema, confirms that every threshold it declares is the threshold the harness actually applies, that the runner capabilities it declares are the ones the harness grants its runner, and that the preflight argv it declares fails on a missing runner and passes on a present one, and fails when a TEA skill has neither a behavioral suite nor a deferred declaration. A suite list that omits a skill reads as coverage, so the omission has to be an error rather than a silence.
|
|
405
405
|
- `test:eval-diagnostics` checks that current result writers emit bounded case and repetition diagnostics, that quality and environment outcomes stay separate, that unstable signatures remain visible, and that the protected 1.3.0 baseline remains readable and byte-for-byte unchanged.
|
|
406
|
+
- `test:nfr-evidence-guidance` holds every NFR worker and the publication audit to the supplied-evidence ledger, exercises supported, missing, and mismatched evidence, rejects contaminating examples, and binds the protected baseline and focused remediation records by digest.
|
|
406
407
|
- `test:contract-oracles` evaluates every oracle in every eval contract with `eval-quality`'s own evaluator, over the stored replay outputs and over constructed selections for all 24 fragment-selection cases, and fails when an oracle faults or disagrees with the harness scorer on the same evidence. The first run found that every regex oracle in the `test-review` contract was refused by the evaluator before it matched anything.
|
|
407
408
|
|
|
408
409
|
These checks produce the same answer from the same repository state. They need no agent credential, network call, or model budget. `test:eval-data` runs through `npm test`, the local pre-commit hook, pull-request quality checks, and the publish workflow.
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.27.2-next.
|
|
4
|
+
"version": "1.27.2-next.4",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
|
@@ -83,7 +83,7 @@
|
|
|
83
83
|
"release:minor": "gh workflow run publish.yaml -f channel=latest -f bump=minor",
|
|
84
84
|
"release:next": "gh workflow run publish.yaml -f channel=next",
|
|
85
85
|
"release:patch": "gh workflow run publish.yaml -f channel=latest -f bump=patch",
|
|
86
|
-
"test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
|
|
86
|
+
"test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:nfr-evidence-guidance && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
|
|
87
87
|
"test:atdd-isolation": "node test/test-atdd-isolation.js",
|
|
88
88
|
"test:atdd-net-guard": "node test/test-atdd-net-guard.js",
|
|
89
89
|
"test:atdd-scoring": "node test/test-atdd-scoring.js",
|
|
@@ -142,6 +142,7 @@
|
|
|
142
142
|
"test:lineage": "eval-quality-gates field-ownership",
|
|
143
143
|
"test:lockfile-age": "eval-quality-gates lockfile-age",
|
|
144
144
|
"test:lockfile-age-cache": "node test/test-generate-lockfile-age-cache.js",
|
|
145
|
+
"test:nfr-evidence-guidance": "node test/test-nfr-evidence-guidance.js",
|
|
145
146
|
"test:nfr-validate-corpus": "node test/test-nfr-validate-corpus.js",
|
|
146
147
|
"test:port-totality": "node test/test-port-totality.js",
|
|
147
148
|
"test:probe-conformance": "node test/test-probe-conformance.js",
|
|
@@ -109,49 +109,11 @@ From `{config_source}`:
|
|
|
109
109
|
|
|
110
110
|
---
|
|
111
111
|
|
|
112
|
-
###
|
|
112
|
+
### Deterministic Knowledge Selection
|
|
113
113
|
|
|
114
|
-
Load
|
|
114
|
+
The fragment list for this step is a closed set. Start empty, evaluate the complete conditions under **Load Knowledge Base Fragments**, and add every fragment from each matching list. A config flag opens a branch only when every stack, runner, package, and relevance condition on that branch also matches. Do not add fragments from tier labels, index descriptions, nearby mentions, general usefulness, or possible future need. Deduplicate while preserving the order below. Identical facts and config must produce an identical list.
|
|
115
115
|
|
|
116
|
-
|
|
117
|
-
2. **Extended tier** (load on-demand): Load when deeper analysis is needed or when the user's context requires it
|
|
118
|
-
3. **Specialized tier** (load only when relevant): Load only when the specific use case matches (e.g., contract-testing only for microservices, email-auth only for email flows)
|
|
119
|
-
|
|
120
|
-
> **Context Efficiency**: Loading only core fragments reduces context usage by 40-50% compared to loading all fragments.
|
|
121
|
-
|
|
122
|
-
### Playwright Utils Loading Profiles
|
|
123
|
-
|
|
124
|
-
**If `tea_use_playwright_utils` is enabled**, load `playwright-utils-mandate.md` FIRST, before any profile below. It is the binding rule for this run: playwright-utils is the default implementation for every capability it covers, and a vanilla Playwright equivalent is a deviation that must be justified in the output. Red-phase scaffolds are still real test code, so the mandate applies to them exactly as it applies to green tests.
|
|
125
|
-
|
|
126
|
-
Then select the appropriate loading profile:
|
|
127
|
-
|
|
128
|
-
- **API-only profile** (when `{detected_stack}` is `backend` or no `page.goto`/`page.locator` found in test files):
|
|
129
|
-
Load: `playwright-utils-mandate`, `overview`, `api-request`, `auth-session`, `recurse` (~2,100 lines)
|
|
130
|
-
|
|
131
|
-
- **Full UI+API profile** (when `{detected_stack}` is `frontend`/`fullstack` or browser tests detected):
|
|
132
|
-
Load: `playwright-utils-mandate` plus all Playwright Utils core fragments (~4,800 lines)
|
|
133
|
-
|
|
134
|
-
**Detection**: Scan `{test_dir}` for files containing `page.goto` or `page.locator`. If none found, use API-only profile.
|
|
135
|
-
|
|
136
|
-
### Pact.js Utils Loading
|
|
137
|
-
|
|
138
|
-
**If `tea_use_pactjs_utils` is enabled** (and `{detected_stack}` is `backend` or `fullstack`, or a microservices layout is detected, `pactjs-utils-mandate.md`'s own definition: two or more independently deployable services in this repo that call each other):
|
|
139
|
-
|
|
140
|
-
Load `pactjs-utils-mandate.md` FIRST. It is the binding rule for any Pact artifact this run produces, and it carries the relevance gate: the flag defaults to `true` and means "use these utilities when contract tests are written", never "add contract tests to this project".
|
|
141
|
-
|
|
142
|
-
Then load: `pactjs-utils-overview.md`, `pactjs-utils-consumer-helpers.md`, `pactjs-utils-provider-verifier.md`, `pactjs-utils-request-filter.md`, `pactjs-utils-zod-to-pact.md`, `pact-consumer-di.md`, `pact-consumer-framework-setup.md`, `pact-broker-webhooks.md`
|
|
143
|
-
|
|
144
|
-
**If `tea_use_pactjs_utils` is disabled** but contract testing is relevant:
|
|
145
|
-
|
|
146
|
-
Load: `contract-testing.md`
|
|
147
|
-
|
|
148
|
-
### Pact MCP Loading
|
|
149
|
-
|
|
150
|
-
**If `tea_pact_mcp` is `"mcp"`:**
|
|
151
|
-
|
|
152
|
-
Load: `pact-mcp.md`
|
|
153
|
-
|
|
154
|
-
**`tea_pact_mcp` defaults to `"mcp"`, and Pact artifacts are gated on relevance, not on this flag.** Follow `pact-mcp.md` § _When the Tools Are Not Reachable_: the probe is a tool-list check and never a broker call, its result is recorded once per run as `pact_mcp_reachable`, and the fallback order is provider source, then an OpenAPI spec, then `confidence-gate.md`. Report the outcome once and continue; never block, never retry, never present inferred provider states as broker data.
|
|
116
|
+
Contract testing is relevant only when repository facts show existing Pact artifacts, dependencies, configuration, or broker variables, or when the task explicitly requests contract testing. A service count or target-state architecture alone does not open a contract branch.
|
|
155
117
|
|
|
156
118
|
## 5. Load Knowledge Base Fragments
|
|
157
119
|
|
|
@@ -169,7 +131,7 @@ Use `{knowledgeIndex}` to load:
|
|
|
169
131
|
- `selector-resilience.md`
|
|
170
132
|
- `timing-debugging.md`
|
|
171
133
|
|
|
172
|
-
**Playwright Utils (if enabled and {detected_stack} is `frontend` or `fullstack`):**
|
|
134
|
+
**Playwright Utils (if enabled, `@seontechnologies/playwright-utils` is in `package.json`, the test files run on the Playwright runner, and {detected_stack} is `frontend` or `fullstack`):**
|
|
173
135
|
|
|
174
136
|
- `playwright-utils-mandate.md` (load first — it governs how the fragments below are applied)
|
|
175
137
|
- `overview.md`, `api-request.md`, `network-recorder.md`, `auth-session.md`, `intercept-network-call.md`, `recurse.md`, `log.md`, `file-utils.md`, `network-error-monitor.md`, `fixtures-composition.md`
|
|
@@ -183,7 +145,7 @@ Use `{knowledgeIndex}` to load:
|
|
|
183
145
|
|
|
184
146
|
- (existing MCP-related fragments, if any are added in future)
|
|
185
147
|
|
|
186
|
-
**Traditional Patterns (if
|
|
148
|
+
**Traditional Patterns (if the Playwright Utils applicability gate above did not open and {detected_stack} is `frontend` or `fullstack`):**
|
|
187
149
|
|
|
188
150
|
- `fixture-architecture.md`
|
|
189
151
|
- `network-first.md`
|
|
@@ -194,16 +156,16 @@ Use `{knowledgeIndex}` to load:
|
|
|
194
156
|
- `test-priorities-matrix.md`
|
|
195
157
|
- `ci-burn-in.md`
|
|
196
158
|
|
|
197
|
-
**Pact.js Utils (if enabled and contract testing is relevant):**
|
|
159
|
+
**Pact.js Utils (if enabled, `@seontechnologies/pactjs-utils` is in `package.json`, and contract testing is relevant):**
|
|
198
160
|
|
|
199
161
|
- `pactjs-utils-mandate.md` (load first — it governs how the fragments below are applied)
|
|
200
162
|
- `pactjs-utils-overview.md`, `pactjs-utils-consumer-helpers.md`, `pactjs-utils-provider-verifier.md`, `pactjs-utils-request-filter.md`, `pactjs-utils-zod-to-pact.md`, `pact-consumer-di.md`, `pact-consumer-framework-setup.md`, `pact-broker-webhooks.md`
|
|
201
163
|
|
|
202
|
-
**Contract Testing (if
|
|
164
|
+
**Contract Testing (if Pact.js Utils is disabled or not installed, and contract testing is relevant):**
|
|
203
165
|
|
|
204
166
|
- `contract-testing.md`
|
|
205
167
|
|
|
206
|
-
**Pact MCP (if tea_pact_mcp is "mcp"):**
|
|
168
|
+
**Pact MCP (if tea_pact_mcp is "mcp" and contract testing is relevant):**
|
|
207
169
|
|
|
208
170
|
- `pact-mcp.md`
|
|
209
171
|
|
package/src/workflows/testarch/bmad-testarch-automate/steps-c/step-01-preflight-and-context.md
CHANGED
|
@@ -113,56 +113,11 @@ If required framework configuration is missing: **HALT** with message "Run `fram
|
|
|
113
113
|
|
|
114
114
|
---
|
|
115
115
|
|
|
116
|
-
###
|
|
116
|
+
### Deterministic Knowledge Selection
|
|
117
117
|
|
|
118
|
-
Load
|
|
118
|
+
The fragment list for this step is a closed set. Start empty, evaluate the complete conditions under **Load Knowledge Base Fragments**, and add every fragment from each matching list. A config flag opens a branch only when every stack, runner, package, and relevance condition on that branch also matches. Do not add fragments from tier labels, index descriptions, nearby mentions, general usefulness, or possible future need. Deduplicate while preserving the order below. Identical facts and config must produce an identical list.
|
|
119
119
|
|
|
120
|
-
|
|
121
|
-
2. **Extended tier** (load on-demand): Load when deeper analysis is needed or when the user's context requires it
|
|
122
|
-
3. **Specialized tier** (load only when relevant): Load only when the specific use case matches (e.g., contract-testing only for microservices, email-auth only for email flows)
|
|
123
|
-
|
|
124
|
-
> **Context Efficiency**: Loading only core fragments reduces context usage by 40-50% compared to loading all fragments.
|
|
125
|
-
|
|
126
|
-
### Playwright Utils Loading Profiles
|
|
127
|
-
|
|
128
|
-
**If `tea_use_playwright_utils` is enabled**, load `playwright-utils-mandate.md` FIRST, before any profile below. It is the binding rule for this run: playwright-utils is the default implementation for every capability it covers, and a vanilla Playwright equivalent is a deviation that must be justified in the output. Every worker step dispatched from this workflow inherits that rule.
|
|
129
|
-
|
|
130
|
-
Then select the appropriate loading profile:
|
|
131
|
-
|
|
132
|
-
- **API-only profile** (when `{detected_stack}` is `backend` or no `page.goto`/`page.locator` found in test files):
|
|
133
|
-
Load: `playwright-utils-mandate`, `overview`, `api-request`, `auth-session`, `recurse` (~2,100 lines)
|
|
134
|
-
|
|
135
|
-
- **Full UI+API profile** (when `{detected_stack}` is `frontend`/`fullstack` or browser tests detected):
|
|
136
|
-
Load: `playwright-utils-mandate` plus all Playwright Utils core fragments (~4,800 lines)
|
|
137
|
-
|
|
138
|
-
- **Mobile profile** (when `{detected_stack}` is `mobile`):
|
|
139
|
-
Load: `mobile-test-strategy`, `maestro-flows`, `mobile-ci-device-lab`, `test-levels-framework`, `test-priorities-matrix`, `test-quality`, plus `playwright-utils-mandate`, `overview`, `api-request`, `auth-session`, `recurse` for the app's HTTP boundary.
|
|
140
|
-
Do NOT load the browser fragments (`network-first`, `playwright-config`, `intercept-network-call`, `selector-resilience`): a device flow has no DOM and no request interceptor, and loading them invites browser patterns into a Maestro flow.
|
|
141
|
-
The mandate is loaded here for the HTTP-boundary tests only. It has no bearing on a Maestro flow, and its own scope section says so.
|
|
142
|
-
|
|
143
|
-
**Detection**: Scan `{test_dir}` for files containing `page.goto` or `page.locator`. If none found, use API-only profile. A `maestro/` or `.maestro/` directory selects the mobile profile regardless of what `{test_dir}` holds.
|
|
144
|
-
|
|
145
|
-
### Pact.js Utils Loading
|
|
146
|
-
|
|
147
|
-
**If `tea_use_pactjs_utils` is enabled** (and `{detected_stack}` is `backend` or `fullstack`, or a microservices layout is detected, `pactjs-utils-mandate.md`'s own definition: two or more independently deployable services in this repo that call each other):
|
|
148
|
-
|
|
149
|
-
Load `pactjs-utils-mandate.md` FIRST. It is the binding rule for any Pact artifact this run produces, and it carries the relevance gate: the flag defaults to `true` and means "use these utilities when contract tests are written", never "add contract tests to this project".
|
|
150
|
-
|
|
151
|
-
Then load: `pactjs-utils-overview.md`, `pactjs-utils-consumer-helpers.md`, `pactjs-utils-provider-verifier.md`, `pactjs-utils-request-filter.md`, `pactjs-utils-zod-to-pact.md` (~1,100 lines)
|
|
152
|
-
|
|
153
|
-
**If `tea_use_pactjs_utils` is disabled** but contract testing is relevant (a microservices layout detected per the definition above, or existing Pact config found):
|
|
154
|
-
|
|
155
|
-
Load: `contract-testing.md` (~960 lines)
|
|
156
|
-
|
|
157
|
-
**Detection**: Scan `{project-root}` for Pact indicators: `pact/` directory, `@pact-foundation/pact` in `package.json`, `pactUrls` in test files, `PACT_BROKER` in env files.
|
|
158
|
-
|
|
159
|
-
### Pact MCP Loading
|
|
160
|
-
|
|
161
|
-
**If `tea_pact_mcp` is `"mcp"`:**
|
|
162
|
-
|
|
163
|
-
Load: `pact-mcp.md` (~150 lines) — enables agent to use SmartBear MCP tools for fetching provider states and generating pact tests during automation.
|
|
164
|
-
|
|
165
|
-
**`tea_pact_mcp` defaults to `"mcp"`, and Pact artifacts are gated on relevance, not on this flag.** Follow `pact-mcp.md` § _When the Tools Are Not Reachable_: the probe is a tool-list check and never a broker call, its result is recorded once per run as `pact_mcp_reachable`, and the fallback order is provider source, then an OpenAPI spec, then `confidence-gate.md`. Report the outcome once and continue; never block, never retry, never present inferred provider states as broker data.
|
|
120
|
+
Contract testing is relevant only when repository facts show existing Pact artifacts, dependencies, configuration, or broker variables, or when the task explicitly requests contract testing. A service count or target-state architecture alone does not open a contract branch.
|
|
166
121
|
|
|
167
122
|
## 4. Load Knowledge Base Fragments
|
|
168
123
|
|
|
@@ -177,27 +132,33 @@ Use `{knowledgeIndex}` and load only what is required.
|
|
|
177
132
|
- `ci-burn-in.md`
|
|
178
133
|
- `test-quality.md`
|
|
179
134
|
|
|
180
|
-
**
|
|
135
|
+
**Mobile (if `{detected_stack}` is `mobile`):**
|
|
136
|
+
|
|
137
|
+
- `mobile-test-strategy.md`
|
|
138
|
+
- `maestro-flows.md`
|
|
139
|
+
- `mobile-ci-device-lab.md`
|
|
140
|
+
|
|
141
|
+
**Playwright Utils (if enabled, `@seontechnologies/playwright-utils` is in `package.json`, and the test files run on the Playwright runner):**
|
|
181
142
|
|
|
182
143
|
- `playwright-utils-mandate.md` (load first — it governs how the fragments below are applied)
|
|
183
144
|
- `overview.md`, `api-request.md`, `network-recorder.md`, `auth-session.md`, `intercept-network-call.md`, `recurse.md`, `log.md`, `file-utils.md`, `burn-in.md`, `network-error-monitor.md`, `fixtures-composition.md`
|
|
184
145
|
- `fixture-architecture.md` and `network-first.md` for their principles only. Under the mandate the mechanism comes from the playwright-utils fragments: interception is `interceptNetworkCall` declared before `page.goto`, and composition is `mergeTests`.
|
|
185
146
|
|
|
186
|
-
**Traditional Patterns (if Playwright Utils
|
|
147
|
+
**Traditional Patterns (if the Playwright Utils applicability gate above did not open and the test files run on the Playwright runner):**
|
|
187
148
|
|
|
188
149
|
- `fixture-architecture.md`
|
|
189
150
|
- `network-first.md`
|
|
190
151
|
|
|
191
|
-
**Pact.js Utils (if enabled and contract testing is relevant):**
|
|
152
|
+
**Pact.js Utils (if enabled, `@seontechnologies/pactjs-utils` is in `package.json`, and contract testing is relevant):**
|
|
192
153
|
|
|
193
154
|
- `pactjs-utils-mandate.md` (load first — it governs how the fragments below are applied)
|
|
194
155
|
- `pactjs-utils-overview.md`, `pactjs-utils-consumer-helpers.md`, `pactjs-utils-provider-verifier.md`, `pactjs-utils-request-filter.md`, `pactjs-utils-zod-to-pact.md`
|
|
195
156
|
|
|
196
|
-
**Contract Testing (if
|
|
157
|
+
**Contract Testing (if Pact.js Utils is disabled or not installed, and contract testing is relevant):**
|
|
197
158
|
|
|
198
159
|
- `contract-testing.md`
|
|
199
160
|
|
|
200
|
-
**Pact MCP (if tea_pact_mcp is "mcp"):**
|
|
161
|
+
**Pact MCP (if tea_pact_mcp is "mcp" and contract testing is relevant):**
|
|
201
162
|
|
|
202
163
|
- `pact-mcp.md`
|
|
203
164
|
|
|
@@ -207,7 +168,7 @@ Use `{knowledgeIndex}` and load only what is required.
|
|
|
207
168
|
- `selector-resilience.md`
|
|
208
169
|
- `timing-debugging.md`
|
|
209
170
|
|
|
210
|
-
**Playwright CLI (if tea_browser_automation is "cli" or "auto"):**
|
|
171
|
+
**Playwright CLI (if tea_browser_automation is "cli" or "auto" and the test files run on the Playwright runner):**
|
|
211
172
|
|
|
212
173
|
- `playwright-cli.md`
|
|
213
174
|
|
|
@@ -164,7 +164,7 @@ Include stages:
|
|
|
164
164
|
|
|
165
165
|
- lint
|
|
166
166
|
- test (parallel shards)
|
|
167
|
-
- contract-test (if `tea_use_pactjs_utils` enabled)
|
|
167
|
+
- contract-test (if `tea_use_pactjs_utils` is enabled and Step 1 found contract testing relevant)
|
|
168
168
|
- burn-in (flaky detection)
|
|
169
169
|
- report (aggregate + publish)
|
|
170
170
|
|
|
@@ -197,9 +197,9 @@ Write the selected pipeline configuration to the resolved output path from step
|
|
|
197
197
|
- **Artifacts**: upload the per-step statuses, hierarchy dumps, screenshots, video, AND device logs on failure, resolving the newest run directory rather than globbing a flat filename that recent versions no longer write. The hierarchy dump captured at failure is what identifies a selector break; the failure screenshot is taken after teardown and frequently shows the launcher, so do not lead with it.
|
|
198
198
|
- **Cache**: Gradle or CocoaPods plus the JS toolchain for React Native and Expo.
|
|
199
199
|
|
|
200
|
-
### Contract Testing Pipeline (if `tea_use_pactjs_utils` enabled)
|
|
200
|
+
### Contract Testing Pipeline (if `tea_use_pactjs_utils` is enabled and Step 1 found contract testing relevant)
|
|
201
201
|
|
|
202
|
-
**If `tea_use_pactjs_utils` is enabled**, use `{knowledgeIndex}` to load:
|
|
202
|
+
**If `tea_use_pactjs_utils` is enabled and Step 1 found contract testing relevant**, use `{knowledgeIndex}` to load:
|
|
203
203
|
|
|
204
204
|
- `pact-consumer-framework-setup.md` — determinism gate, `jq -S` publish normalization, PR-only provider branch detection, additive branch-aware `can-i-deploy`, 1:1 local/CI parity
|
|
205
205
|
- `pactjs-utils-consumer-helpers.md` — one-interaction-per-`it()` determinism rule
|
|
@@ -207,7 +207,7 @@ Write the selected pipeline configuration to the resolved output path from step
|
|
|
207
207
|
- `pactjs-utils-request-filter.md` — `createRequestFilter` auth injection patterns for CI pipeline auth setup
|
|
208
208
|
- `pact-broker-webhooks.md` — PactFlow → GitHub webhook auth, exact registered provider target checkout, rotation runbook, and staleness monitoring
|
|
209
209
|
|
|
210
|
-
When `tea_use_pactjs_utils` is enabled, add a `contract-test` stage after `test`:
|
|
210
|
+
When `tea_use_pactjs_utils` is enabled and Step 1 found contract testing relevant, add a `contract-test` stage after `test`:
|
|
211
211
|
|
|
212
212
|
**Required env block** (add to the generated pipeline):
|
|
213
213
|
|
|
@@ -256,7 +256,7 @@ env:
|
|
|
256
256
|
|
|
257
257
|
Required CI secrets: `PACT_BROKER_BASE_URL`, `PACT_BROKER_TOKEN`
|
|
258
258
|
|
|
259
|
-
**If `tea_pact_mcp` is `"mcp"
|
|
259
|
+
**If `tea_pact_mcp` is `"mcp"` and contract testing is relevant:** Reference the SmartBear MCP `Can I Deploy` and `Matrix` tools for pipeline guidance in `pact-mcp.md`.
|
|
260
260
|
|
|
261
261
|
**`tea_pact_mcp` defaults to `"mcp"`, and Pact artifacts are gated on relevance, not on this flag.** Follow `pact-mcp.md` § _When the Tools Are Not Reachable_: the probe is a tool-list check and never a broker call, its result is recorded once per run as `pact_mcp_reachable`, and the fallback order is provider source, then an OpenAPI spec, then `confidence-gate.md`. Report the outcome once and continue; never block, never retry, never present inferred provider states as broker data.
|
|
262
262
|
|
|
@@ -290,7 +290,7 @@ For this step, treat these work units as parallelizable when `resolvedMode` is `
|
|
|
290
290
|
|
|
291
291
|
- Worker A: resolve platform path/template and produce base pipeline skeleton (section 1)
|
|
292
292
|
- Worker B: construct stage definitions and test execution blocks (sections 2-3)
|
|
293
|
-
- Worker C: contract-testing block (only when `tea_use_pactjs_utils` is true)
|
|
293
|
+
- Worker C: contract-testing block (only when `tea_use_pactjs_utils` is true and Step 1 found contract testing relevant)
|
|
294
294
|
|
|
295
295
|
If `resolvedMode` is `sequential`, execute sections 1→4 in order.
|
|
296
296
|
|
|
@@ -36,6 +36,12 @@ Configure burn-in loops, quality thresholds, and notification hooks.
|
|
|
36
36
|
|
|
37
37
|
**CRITICAL:** Follow this sequence exactly. Do not skip, reorder, or improvise.
|
|
38
38
|
|
|
39
|
+
### Deterministic Knowledge Selection
|
|
40
|
+
|
|
41
|
+
The fragment list for this step is a closed set. Start empty, evaluate the complete conditions in the numbered sections below, and add every fragment from each matching list. A config flag opens a branch only when every stack, runner, package, and relevance condition on that branch also matches. Do not add fragments from tier labels, index descriptions, nearby mentions, general usefulness, or possible future need. Deduplicate while preserving the order below. Identical facts and config must produce an identical list.
|
|
42
|
+
|
|
43
|
+
Contract testing is relevant only when repository facts show existing Pact artifacts, dependencies, configuration, or broker variables, or when the task explicitly requests contract testing. A service count or target-state architecture alone does not open a contract branch.
|
|
44
|
+
|
|
39
45
|
## 1. Burn-In Configuration
|
|
40
46
|
|
|
41
47
|
Use `{knowledgeIndex}` to load `ci-burn-in.md` guidance:
|
|
@@ -65,7 +71,7 @@ Skip this for Cypress, Maestro, and non-Playwright backend suites; those keep th
|
|
|
65
71
|
- **Backend only** (`test_stack_type` is `backend`): Skip burn-in by default. Backend tests (unit, integration, API) are deterministic and rarely exhibit UI-related flakiness. If the user explicitly requests burn-in for backend, honor that override.
|
|
66
72
|
- **Mobile** (`test_stack_type` is `mobile`): Enable burn-in by default, and scope it to new and changed Maestro flows only. Device flows are the most flake-prone level in any suite (emulator boot, app install, animation timing, real network), so a new flow that has not survived repeated runs is not evidence. Never burn in the whole flow suite on a PR: run the changed flows N times on the primary target, and leave the full matrix to the nightly job.
|
|
67
73
|
|
|
68
|
-
**The gate must be able to fail.**
|
|
74
|
+
**The gate must be able to fail.** `continue-on-error` belongs on artifact collection and never on a step that runs tests, and a runner manifest that names a subset of the discovered test files is a silent coverage hole rather than a configuration choice. Reconcile the executed count against the discovered count in the job, so a suite that quietly stopped running most of itself fails instead of passing faster.
|
|
69
75
|
|
|
70
76
|
**Security: Script injection prevention for reusable burn-in workflows:**
|
|
71
77
|
|
|
@@ -100,7 +106,7 @@ Define:
|
|
|
100
106
|
- Fail CI on critical test failures
|
|
101
107
|
- Optional: require traceability or nfr-assess output before release
|
|
102
108
|
|
|
103
|
-
**Contract testing gate** (if `tea_use_pactjs_utils` is enabled):
|
|
109
|
+
**Contract testing gate** (if `tea_use_pactjs_utils` is enabled and contract testing is relevant):
|
|
104
110
|
|
|
105
111
|
Use `{knowledgeIndex}` to load:
|
|
106
112
|
|
package/src/workflows/testarch/bmad-testarch-framework/steps-c/step-03-scaffold-framework.md
CHANGED
|
@@ -236,6 +236,12 @@ Create the idiomatic version file for the detected language:
|
|
|
236
236
|
|
|
237
237
|
Read `{config_source}` and use `{knowledgeIndex}` to load fragments based on `{detected_stack}`:
|
|
238
238
|
|
|
239
|
+
### Deterministic Knowledge Selection
|
|
240
|
+
|
|
241
|
+
The fragment list for this step is a closed set. Start empty, evaluate the complete conditions in this section, and add every fragment from each matching list. A config flag opens a branch only when every stack, runner, package, and relevance condition on that branch also matches. Do not add fragments from tier labels, index descriptions, nearby mentions elsewhere in this step, general usefulness, or possible future need. Deduplicate while preserving the order below. Identical facts and config must produce an identical list.
|
|
242
|
+
|
|
243
|
+
Contract testing is relevant only when repository facts show existing Pact artifacts, dependencies, configuration, or broker variables, or when the task explicitly requests contract testing. A service count or target-state architecture alone does not open a contract branch.
|
|
244
|
+
|
|
239
245
|
**If `{detected_stack}` is `mobile`:**
|
|
240
246
|
|
|
241
247
|
- `mobile-test-strategy.md` (CRITICAL: load this first — it decides which behaviors become device flows at all, and most should not)
|
|
@@ -289,7 +295,7 @@ Read `{config_source}` and use `{knowledgeIndex}` to load fragments based on `{d
|
|
|
289
295
|
|
|
290
296
|
- `contract-testing.md`
|
|
291
297
|
|
|
292
|
-
**If Pact MCP enabled** (`config.tea_pact_mcp` is `"mcp"`):
|
|
298
|
+
**If Pact MCP is enabled and the relevance gate in section 1 opened** (`config.tea_pact_mcp` is `"mcp"`):
|
|
293
299
|
|
|
294
300
|
- `pact-mcp.md`
|
|
295
301
|
|
|
@@ -35,7 +35,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
35
35
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
36
36
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
37
37
|
- **Actual:** {ACTUAL_VALUE}
|
|
38
|
-
- **Evidence:** {
|
|
38
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
39
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
39
40
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
40
41
|
|
|
41
42
|
### Throughput
|
|
@@ -43,7 +44,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
43
44
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
44
45
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
45
46
|
- **Actual:** {ACTUAL_VALUE}
|
|
46
|
-
- **Evidence:** {
|
|
47
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
48
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
47
49
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
48
50
|
|
|
49
51
|
### Resource Usage
|
|
@@ -52,13 +54,15 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
52
54
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
53
55
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
54
56
|
- **Actual:** {ACTUAL_VALUE}
|
|
55
|
-
- **Evidence:** {
|
|
57
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
58
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
56
59
|
|
|
57
60
|
- **Memory Usage**
|
|
58
61
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
59
62
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
60
63
|
- **Actual:** {ACTUAL_VALUE}
|
|
61
|
-
- **Evidence:** {
|
|
64
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
65
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
62
66
|
|
|
63
67
|
---
|
|
64
68
|
|
|
@@ -69,7 +73,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
69
73
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
70
74
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
71
75
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
72
|
-
- **Evidence:** {
|
|
76
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
77
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
73
78
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
74
79
|
- **Recommendation:** {RECOMMENDATION} (if CONCERNS or FAIL)
|
|
75
80
|
|
|
@@ -78,7 +83,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
78
83
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
79
84
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
80
85
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
81
|
-
- **Evidence:** {
|
|
86
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
87
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
82
88
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
83
89
|
|
|
84
90
|
### Data Protection
|
|
@@ -86,23 +92,26 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
86
92
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
87
93
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
88
94
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
89
|
-
- **Evidence:** {
|
|
95
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
96
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
90
97
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
91
98
|
|
|
92
99
|
### Vulnerability Management
|
|
93
100
|
|
|
94
101
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
95
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
96
|
-
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
97
|
-
- **Evidence:** {
|
|
102
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
103
|
+
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
104
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
105
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
98
106
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
99
107
|
|
|
100
108
|
### Compliance (if applicable)
|
|
101
109
|
|
|
102
110
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
103
|
-
- **Standards:** {COMPLIANCE_STANDARDS}
|
|
111
|
+
- **Standards:** {COMPLIANCE_STANDARDS}
|
|
104
112
|
- **Actual:** {ACTUAL_COMPLIANCE_STATUS}
|
|
105
|
-
- **Evidence:** {
|
|
113
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
114
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
106
115
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
107
116
|
|
|
108
117
|
---
|
|
@@ -112,25 +121,28 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
112
121
|
### Availability (Uptime)
|
|
113
122
|
|
|
114
123
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
115
|
-
- **Threshold:** {THRESHOLD_VALUE}
|
|
116
|
-
- **Actual:** {ACTUAL_VALUE}
|
|
117
|
-
- **Evidence:** {
|
|
124
|
+
- **Threshold:** {THRESHOLD_VALUE}
|
|
125
|
+
- **Actual:** {ACTUAL_VALUE}
|
|
126
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
127
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
118
128
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
119
129
|
|
|
120
130
|
### Error Rate
|
|
121
131
|
|
|
122
132
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
123
|
-
- **Threshold:** {THRESHOLD_VALUE}
|
|
124
|
-
- **Actual:** {ACTUAL_VALUE}
|
|
125
|
-
- **Evidence:** {
|
|
133
|
+
- **Threshold:** {THRESHOLD_VALUE}
|
|
134
|
+
- **Actual:** {ACTUAL_VALUE}
|
|
135
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
136
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
126
137
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
127
138
|
|
|
128
139
|
### MTTR (Mean Time To Recovery)
|
|
129
140
|
|
|
130
141
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
131
|
-
- **Threshold:** {THRESHOLD_VALUE}
|
|
132
|
-
- **Actual:** {ACTUAL_VALUE}
|
|
133
|
-
- **Evidence:** {
|
|
142
|
+
- **Threshold:** {THRESHOLD_VALUE}
|
|
143
|
+
- **Actual:** {ACTUAL_VALUE}
|
|
144
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
145
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
134
146
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
135
147
|
|
|
136
148
|
### Fault Tolerance
|
|
@@ -138,15 +150,17 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
138
150
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
139
151
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
140
152
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
141
|
-
- **Evidence:** {
|
|
153
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
154
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
142
155
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
143
156
|
|
|
144
157
|
### CI Burn-In (Stability)
|
|
145
158
|
|
|
146
159
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
147
|
-
- **Threshold:** {THRESHOLD_VALUE}
|
|
148
|
-
- **Actual:** {ACTUAL_VALUE}
|
|
149
|
-
- **Evidence:** {
|
|
160
|
+
- **Threshold:** {THRESHOLD_VALUE}
|
|
161
|
+
- **Actual:** {ACTUAL_VALUE}
|
|
162
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
163
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
150
164
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
151
165
|
|
|
152
166
|
### Disaster Recovery (if applicable)
|
|
@@ -155,13 +169,15 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
155
169
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
156
170
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
157
171
|
- **Actual:** {ACTUAL_VALUE}
|
|
158
|
-
- **Evidence:** {
|
|
172
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
173
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
159
174
|
|
|
160
175
|
- **RPO (Recovery Point Objective)**
|
|
161
176
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
162
177
|
- **Threshold:** {THRESHOLD_VALUE}
|
|
163
178
|
- **Actual:** {ACTUAL_VALUE}
|
|
164
|
-
- **Evidence:** {
|
|
179
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
180
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
165
181
|
|
|
166
182
|
---
|
|
167
183
|
|
|
@@ -170,33 +186,37 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
170
186
|
### Test Coverage
|
|
171
187
|
|
|
172
188
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
173
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
189
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
174
190
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
175
|
-
- **Evidence:** {
|
|
191
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
192
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
176
193
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
177
194
|
|
|
178
195
|
### Code Duplication
|
|
179
196
|
|
|
180
197
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
181
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
198
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
182
199
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
183
|
-
- **Evidence:** {
|
|
200
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
201
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
184
202
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
185
203
|
|
|
186
204
|
### Vulnerability Scan
|
|
187
205
|
|
|
188
206
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
189
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
207
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
190
208
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
191
|
-
- **Evidence:** {
|
|
209
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
210
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
192
211
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
193
212
|
|
|
194
213
|
### Observability
|
|
195
214
|
|
|
196
215
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
197
|
-
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
216
|
+
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
198
217
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
199
|
-
- **Evidence:** {
|
|
218
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
219
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
200
220
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
201
221
|
|
|
202
222
|
---
|
|
@@ -208,7 +228,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
208
228
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
209
229
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
210
230
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
211
|
-
- **Evidence:** {
|
|
231
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
232
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
212
233
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
213
234
|
|
|
214
235
|
### {CUSTOM_NFR_NAME_2}
|
|
@@ -216,7 +237,8 @@ Note: This audit summarizes existing implementation evidence; it does not run te
|
|
|
216
237
|
- **Status:** {STATUS} {STATUS_ICON}
|
|
217
238
|
- **Threshold:** {THRESHOLD_DESCRIPTION}
|
|
218
239
|
- **Actual:** {ACTUAL_DESCRIPTION}
|
|
219
|
-
- **Evidence:** {
|
|
240
|
+
- **Evidence:** {PROJECT_RELATIVE_PATH}
|
|
241
|
+
- **Supports:** {SUPPORTED_OBSERVATION}
|
|
220
242
|
- **Findings:** {FINDINGS_DESCRIPTION}
|
|
221
243
|
|
|
222
244
|
---
|
|
@@ -54,15 +54,9 @@ From `{config_source}`:
|
|
|
54
54
|
|
|
55
55
|
---
|
|
56
56
|
|
|
57
|
-
###
|
|
57
|
+
### Deterministic Knowledge Selection
|
|
58
58
|
|
|
59
|
-
Load
|
|
60
|
-
|
|
61
|
-
1. **Core tier** (always load): Foundational fragments required for this workflow
|
|
62
|
-
2. **Extended tier** (load on-demand): Load when deeper analysis is needed or when the user's context requires it
|
|
63
|
-
3. **Specialized tier** (load only when relevant): Load only when the specific use case matches (e.g., contract-testing only for microservices, email-auth only for email flows)
|
|
64
|
-
|
|
65
|
-
> **Context Efficiency**: Loading only core fragments reduces context usage by 40-50% compared to loading all fragments.
|
|
59
|
+
The fragment list for this step is a closed set. Start empty, evaluate the complete conditions under **Load Knowledge Base Fragments**, and add every fragment from each matching list. A config flag opens a branch only when every stack, runner, package, and relevance condition on that branch also matches. Do not add fragments from tier labels, index descriptions, nearby mentions, general usefulness, or possible future need. Deduplicate while preserving the order below. Identical facts and config must produce an identical list.
|
|
66
60
|
|
|
67
61
|
## 3. Load Knowledge Base Fragments
|
|
68
62
|
|