bmad-method-test-architecture-enterprise 1.27.2-next.0 → 1.27.2-next.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +4 -2
- package/package.json +5 -2
- package/src/agents/bmad-tea/SKILL.md +37 -0
- package/src/workflows/testarch/bmad-testarch-atdd/atdd-checklist-template.md +6 -4
- package/src/workflows/testarch/bmad-testarch-atdd/checklist.md +9 -0
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-01-preflight-and-context.md +6 -1
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-03-test-strategy.md +8 -1
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-04a-subagent-api-failing.md +14 -5
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-04b-subagent-e2e-failing.md +54 -44
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-04c-aggregate.md +18 -0
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-05-validate-and-complete.md +8 -0
- package/src/workflows/testarch/bmad-testarch-atdd/steps-v/step-01-validate.md +2 -0
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "1.27.2-next.
|
|
34
|
+
"version": "1.27.2-next.2",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/README.md
CHANGED
|
@@ -182,7 +182,7 @@ flowchart TB
|
|
|
182
182
|
|
|
183
183
|
### How It Works at Runtime
|
|
184
184
|
|
|
185
|
-
**1. Activation.** `/bmad-tea` or `$bmad-tea` loads the agent skill. It resolves its customization block across base, team, and user layers, adopts the persona, loads persistent facts and `_bmad/tea/config.yaml`, greets you, and renders `{agent.menu}` as a numbered table. Naming
|
|
185
|
+
**1. Activation.** `/bmad-tea` or `$bmad-tea` loads the agent skill. It resolves its customization block across base, team, and user layers, adopts the persona, loads persistent facts and `_bmad/tea/config.yaml`, greets you, and renders `{agent.menu}` as a numbered table. Naming one clear intent in your first message ("let's design tests for this epic") skips the menu and dispatches directly. A message that supports several menu choices triggers one focused question naming the relevant choices before any workflow starts.
|
|
186
186
|
|
|
187
187
|
**2. Workflow entry.** Direct workflow commands use the installed skill name, such as `/bmad-testarch-automate` or `$bmad-testarch-automate`, depending on the host's invocation syntax. `TA` is the equivalent agent-menu code, available only once TEA is active. Either way, the workflow's `SKILL.md` resolves its own `[workflow]` customization block and asks which mode to run: Create, Resume, Validate, or Edit. Create and Resume both route into `steps-c/`; Validate into `steps-v/`; Edit into `steps-e/`. `test-review` alone supports `headless: true`, which skips the greeting and the menu and runs Create directly. That is how the CLI drives it in CI.
|
|
188
188
|
|
|
@@ -392,13 +392,15 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
|
|
|
392
392
|
|
|
393
393
|
### Deterministic Checks
|
|
394
394
|
|
|
395
|
-
`npm test` chains seventy-
|
|
395
|
+
`npm test` chains seventy-four checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Ten of the seventy-four keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
|
|
396
396
|
|
|
397
397
|
- `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
|
|
398
398
|
- `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, and the fragment-selection case count. A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
|
|
399
399
|
- `test:doc-claims` runs `eval-quality-gates doc-claims`, which holds a hand-written prose claim — a named symbol, a transcribed list, a worked JSON example, a named exit code, or a claim whose truth depends on when it was written — against the artifact it describes. A registered dated claim whose sentence was rewritten out from under it fails the same way a wrong symbol name does, so the entry cannot go stale by drifting out from under its own key either.
|
|
400
400
|
- `test:enforce-hook` fails when a new Absolute registry row appears in neither the hook's enforced list nor its deferred list. This prevents a rule from being added without an explicit write-time enforcement decision.
|
|
401
|
+
- `test:atdd-workflow-guidance` holds ATDD generation to one supplied criterion id per executable leaf, one primary scaffold per criterion, a criterion-defining first assertion, and a transition-bearing primary branch for state changes. It also checks every literal API and E2E example title against the same criterion-id format.
|
|
401
402
|
- `test:eval-data` checks that all 24 fragment-selection cases are structurally usable: their workflow context files exist, every expected fragment exists and is indexed for that workflow, and the required and forbidden sets do not overlap. The expected sets come from the workflow step files. This check does not ask an agent to select anything.
|
|
403
|
+
- `test:eval-routing-boundaries` mutates the facts, candidate codes, and missing deciding information in each source-bound routing ambiguity rule. It also rejects duplicate and orphan boundaries, plus any boundary that would turn a clear intent into a clarification.
|
|
402
404
|
- `test:eval-schemas` checks this repository's own eval suite manifest against its schema, confirms that every threshold it declares is the threshold the harness actually applies, that the runner capabilities it declares are the ones the harness grants its runner, and that the preflight argv it declares fails on a missing runner and passes on a present one, and fails when a TEA skill has neither a behavioral suite nor a deferred declaration. A suite list that omits a skill reads as coverage, so the omission has to be an error rather than a silence.
|
|
403
405
|
- `test:eval-diagnostics` checks that current result writers emit bounded case and repetition diagnostics, that quality and environment outcomes stay separate, that unstable signatures remain visible, and that the protected 1.3.0 baseline remains readable and byte-for-byte unchanged.
|
|
404
406
|
- `test:contract-oracles` evaluates every oracle in every eval contract with `eval-quality`'s own evaluator, over the stored replay outputs and over constructed selections for all 24 fragment-selection cases, and fails when an oracle faults or disagrees with the harness scorer on the same evidence. The first run found that every regex oracle in the `test-review` contract was refused by the evaluator before it matched anything.
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.27.2-next.
|
|
4
|
+
"version": "1.27.2-next.2",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
|
@@ -83,10 +83,11 @@
|
|
|
83
83
|
"release:minor": "gh workflow run publish.yaml -f channel=latest -f bump=minor",
|
|
84
84
|
"release:next": "gh workflow run publish.yaml -f channel=next",
|
|
85
85
|
"release:patch": "gh workflow run publish.yaml -f channel=latest -f bump=patch",
|
|
86
|
-
"test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
|
|
86
|
+
"test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
|
|
87
87
|
"test:atdd-isolation": "node test/test-atdd-isolation.js",
|
|
88
88
|
"test:atdd-net-guard": "node test/test-atdd-net-guard.js",
|
|
89
89
|
"test:atdd-scoring": "node test/test-atdd-scoring.js",
|
|
90
|
+
"test:atdd-workflow-guidance": "node test/test-atdd-workflow-guidance.js",
|
|
90
91
|
"test:automate-eval-fixture": "node test/test-automate-eval-fixture.js",
|
|
91
92
|
"test:bmad-output-gated": "node test/test-bmad-output-gated.js",
|
|
92
93
|
"test:boundary": "eval-quality-gates package-boundary",
|
|
@@ -121,7 +122,9 @@
|
|
|
121
122
|
"test:eval-nfr-data": "node test/eval-nfr.js --validate-only",
|
|
122
123
|
"test:eval-quality-corpus": "node test/test-eval-quality-corpus.js",
|
|
123
124
|
"test:eval-replay": "node test/test-eval-replay.js",
|
|
125
|
+
"test:eval-routing-boundaries": "node test/test-routing-ambiguity-boundaries.js",
|
|
124
126
|
"test:eval-routing-data": "node test/eval-bmad-tea-routing.js --validate-only",
|
|
127
|
+
"test:eval-routing-evidence": "node test/test-routing-evidence.js",
|
|
125
128
|
"test:eval-schemas": "node tools/validate-eval-schemas.js",
|
|
126
129
|
"test:eval-teach-me-testing-data": "node test/eval-teach-me-testing.js --validate-only",
|
|
127
130
|
"test:eval-test-design-data": "node test/eval-test-design.js --validate-only",
|
|
@@ -71,6 +71,43 @@ Otherwise render `{agent.menu}` as a numbered table: `Code`, `Description`, `Act
|
|
|
71
71
|
|
|
72
72
|
Dispatch on a clear match by invoking the item's `skill` or executing its `prompt`. Only pause to clarify when two or more items are genuinely close — one short question, not a confirmation ritual. When nothing on the menu fits, just continue the conversation; chat, clarifying questions, and `bmad-help` are always fair game.
|
|
73
73
|
|
|
74
|
+
### Routing Ambiguity Boundaries
|
|
75
|
+
|
|
76
|
+
Before dispatching, list the menu items directly supported by facts in the user's message. One supported item is a clear route. Two or more supported items require the missing deciding information when the user has supplied no priority, sequence, or requested deliverable that selects one.
|
|
77
|
+
|
|
78
|
+
Ask one short question that names every supported choice in user-facing language and preserves any epic, story, feature, or file-set scope the user named. Keep the menu code and workflow unset until the user answers. Do not invoke any candidate while asking.
|
|
79
|
+
|
|
80
|
+
In routing, `spec files` means written test files when the request asks which files are badly written and what to fix. That fact pattern is a clear Review Tests (`RV`) route. A request that identifies product requirements or design specifications falls outside this rule. Direct requests to judge existing tests, identify badly written tests, or recommend fixes for those tests also route to Review Tests. Fix recommendations remain part of the review. The Review Tests versus Trace Coverage boundary applies when the same request also asks which requirements or risks the tests cover, or whether that coverage supports shipping.
|
|
81
|
+
|
|
82
|
+
<!-- routing-ambiguity-boundaries:start -->
|
|
83
|
+
|
|
84
|
+
| Source case | Facts supplied by the user | Supported choices | Missing deciding information |
|
|
85
|
+
| :-------------------------------- | :--------------------------------------------------------------------------------- | :---------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------- |
|
|
86
|
+
| `good-or-covering-what-matters` | Existing tests raise both writing-quality and requirements-coverage concerns | Review Tests (`RV`), Trace Coverage (`TR`) | Whether to assess how well the tests are written or map what they cover and evaluate ship readiness |
|
|
87
|
+
| `measured-some-nfrs-planned-none` | Measured NFR evidence exists for current work and future NFR coverage is unplanned | NFR Evidence Audit (`NR`), Test Design (`TD`) | Whether to audit the existing measurements or plan validation for the future scope first |
|
|
88
|
+
| `thin-coverage-on-payments` | Coverage is described as thin with no requested activity | Test Design (`TD`), Test Automation (`TA`), Trace Coverage (`TR`) | Whether to plan coverage, generate tests, or measure current requirement coverage |
|
|
89
|
+
| `story-half-done` | One scope contains an implemented part and an unbuilt part | ATDD (`AT`), Test Automation (`TA`) | Whether to create failing acceptance tests for the unbuilt part or automate the implemented part first |
|
|
90
|
+
|
|
91
|
+
<!-- routing-ambiguity-boundaries:end -->
|
|
92
|
+
|
|
93
|
+
### Unservable Request Boundaries
|
|
94
|
+
|
|
95
|
+
When the facts support no menu item, keep the menu code and workflow unset. State the
|
|
96
|
+
capability TEA's menu lacks and continue the conversation without activating a workflow.
|
|
97
|
+
The closest-sounding menu item remains unavailable when its declared action cannot
|
|
98
|
+
produce the requested result.
|
|
99
|
+
|
|
100
|
+
<!-- routing-unservable-boundaries:start -->
|
|
101
|
+
|
|
102
|
+
| Source case | Requested result | Menu boundary | Missing capability |
|
|
103
|
+
| :------------------------- | :------------------------------------ | :-------------------------------------------------------------------- | :------------------------------------- |
|
|
104
|
+
| `run-and-fix-ci-failures` | Execute the suite and repair failures | Continuous Integration scaffolds pipelines; Review Tests judges tests | Suite execution and failure repair |
|
|
105
|
+
| `write-production-code` | Implement a production endpoint | ATDD generates failing acceptance tests | Production implementation |
|
|
106
|
+
| `penetration-test-staging` | Perform a live penetration test | NFR Evidence Audit assesses evidence already gathered | Security testing against a live target |
|
|
107
|
+
| `hire-a-qa-lead` | Produce hiring materials | The menu serves testing and quality-engineering workflows | Recruiting and interview design |
|
|
108
|
+
|
|
109
|
+
<!-- routing-unservable-boundaries:end -->
|
|
110
|
+
|
|
74
111
|
## Critical Actions
|
|
75
112
|
|
|
76
113
|
- Consult `./resources/tea-index.csv` to select knowledge fragments under `resources/knowledge/` and load only the files needed for the current task.
|
|
@@ -34,11 +34,13 @@ inputDocuments: []
|
|
|
34
34
|
|
|
35
35
|
## Acceptance Criteria
|
|
36
36
|
|
|
37
|
-
{List
|
|
37
|
+
{List the persisted criterion registry. Preserve supplied ids. For unnamed criteria, assign the lowest unused `AC-<n>` id in source order.}
|
|
38
38
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
39
|
+
| ID | ID source | Acceptance criterion |
|
|
40
|
+
| ---- | ------------------ | ------------------------ |
|
|
41
|
+
| AC-1 | supplied/generated | {Acceptance criterion 1} |
|
|
42
|
+
| AC-2 | supplied/generated | {Acceptance criterion 2} |
|
|
43
|
+
| AC-3 | supplied/generated | {Acceptance criterion 3} |
|
|
42
44
|
|
|
43
45
|
---
|
|
44
46
|
|
|
@@ -20,6 +20,8 @@ Before starting this workflow, verify:
|
|
|
20
20
|
|
|
21
21
|
- [ ] Story markdown file loaded and parsed successfully
|
|
22
22
|
- [ ] All acceptance criteria identified and extracted
|
|
23
|
+
- [ ] Supplied criterion ids preserved; unnamed criteria assigned the lowest unused `AC-<n>` ids in source order
|
|
24
|
+
- [ ] Criterion registry persisted with `id`, `idSource`, and criterion text
|
|
23
25
|
- [ ] Affected systems and components identified
|
|
24
26
|
- [ ] Technical constraints documented
|
|
25
27
|
- [ ] Framework configuration loaded (playwright.config.ts or cypress.config.ts)
|
|
@@ -95,6 +97,13 @@ Before starting this workflow, verify:
|
|
|
95
97
|
|
|
96
98
|
- [ ] All tests use Given-When-Then structure with clear comments
|
|
97
99
|
- [ ] All tests have descriptive names explaining what they test
|
|
100
|
+
- [ ] Every executable leaf title carries exactly one declared `AC-<n>` id from the criterion registry
|
|
101
|
+
- [ ] Every declared acceptance criterion has exactly one red-phase leaf scaffold
|
|
102
|
+
- [ ] Secondary branches and journeys are recorded for green-phase automation
|
|
103
|
+
- [ ] Criterion-defining assertion is the first assertion that can fail and directly isolates the exact newly promised status, scalar, or property
|
|
104
|
+
- [ ] API setup responses from unimplemented endpoints remain opaque before the criterion assertion
|
|
105
|
+
- [ ] E2E criterion-defining assertion is the first potentially failing operation and owns the browser journey
|
|
106
|
+
- [ ] State-transition criteria exercise the transition-bearing branch first
|
|
98
107
|
- [ ] No duplicate tests (same behavior tested multiple times)
|
|
99
108
|
- [ ] No flaky patterns (race conditions, timing issues)
|
|
100
109
|
- [ ] No test interdependencies (tests can run in any order)
|
|
@@ -72,7 +72,11 @@ If any are missing: **HALT** and notify the user.
|
|
|
72
72
|
## 3. Load Story Context
|
|
73
73
|
|
|
74
74
|
- Read story markdown from `{story_file}` (or ask user if not provided)
|
|
75
|
-
- Extract acceptance criteria and constraints
|
|
75
|
+
- Extract acceptance criteria and constraints into a criterion registry in source order
|
|
76
|
+
- Preserve every supplied criterion id that matches `AC-<positive integer>` exactly
|
|
77
|
+
- Reject duplicate supplied ids
|
|
78
|
+
- Assign ids to unnamed criteria deterministically. Reserve all supplied ids first, then visit unnamed criteria in source order and assign the lowest unused `AC-<positive integer>` id
|
|
79
|
+
- Persist each registry row as `{ id, idSource: supplied | generated, text }` and persist the exact ordered id set as `{criterion_ids}`. The same story content must always produce the same registry
|
|
76
80
|
- Identify affected components and integrations
|
|
77
81
|
- Derive and store `story_key` from the story filename when available (for BMM stories, this is the filename without `.md`, e.g. `1-2-user-authentication`)
|
|
78
82
|
- Derive and store `story_id` from story metadata, the H1 heading, or the filename when available (for BMM stories, this is typically `{epic_num}.{story_num}`)
|
|
@@ -241,6 +245,7 @@ Summarize loaded inputs and confirm with the user. Then proceed.
|
|
|
241
245
|
- Set `atddChecklistPath` to `{outputFile}`
|
|
242
246
|
- Initialize `generatedTestFiles` to `[]`
|
|
243
247
|
- Set `inputDocuments` to the list of artifact paths loaded in this step (e.g., knowledge fragments, test design documents, configuration files)
|
|
248
|
+
- Set `acceptanceCriteria` to the persisted criterion registry, including generated ids and `idSource`
|
|
244
249
|
|
|
245
250
|
Load next step: `{nextStepFile}`
|
|
246
251
|
|
|
@@ -38,7 +38,12 @@ Translate acceptance criteria into a prioritized, level-appropriate test plan.
|
|
|
38
38
|
|
|
39
39
|
## 1. Map Acceptance Criteria
|
|
40
40
|
|
|
41
|
-
- Convert each acceptance criterion into test scenarios
|
|
41
|
+
- Convert each acceptance criterion in the persisted registry into test scenarios
|
|
42
|
+
- Record the exact declared acceptance criterion id on every scenario. Preserve supplied ids and use the preflight-generated id for an unnamed criterion
|
|
43
|
+
- Plan exactly one primary red-phase scaffold for every declared criterion. Record secondary branches for green-phase automation without emitting extra red-phase leaves
|
|
44
|
+
- Identify the one smallest criterion-defining assertion that must fail first. Assert the exact newly promised status, scalar, or property before broad object, schema, or secondary assertions
|
|
45
|
+
- Establish prerequisite state through an existing fixture, provider state, or unasserted setup action. Keep an unimplemented setup response opaque before the criterion assertion: do not parse it, branch on it, throw from it, assert it, or derive cleanup data from it
|
|
46
|
+
- For state-transition criteria, choose the transition-bearing branch as the primary red-phase scenario
|
|
42
47
|
- Include negative and edge cases where risk is high
|
|
43
48
|
|
|
44
49
|
---
|
|
@@ -72,6 +77,8 @@ Assign P0–P3 priorities using risk and business impact.
|
|
|
72
77
|
|
|
73
78
|
Ensure all tests are designed to **fail before implementation** (TDD red phase).
|
|
74
79
|
|
|
80
|
+
For each declared criterion, confirm that its single red-phase scaffold reaches the criterion-defining assertion before any other assertion can fail. The first assertion must isolate the exact newly promised status, scalar, or property. Broad object, schema, and secondary assertions follow it within that leaf only when they cannot change its first failure. A criterion describing behavior after a state transition must exercise that transition. Record baseline branches and additional cases in the checklist for green-phase automation.
|
|
81
|
+
|
|
75
82
|
---
|
|
76
83
|
|
|
77
84
|
## 5. Save Progress
|
|
@@ -13,7 +13,7 @@ This is an **isolated subagent** running in parallel with E2E red-phase test gen
|
|
|
13
13
|
|
|
14
14
|
**What you have from parent workflow:**
|
|
15
15
|
|
|
16
|
-
- Story acceptance
|
|
16
|
+
- Story acceptance criterion registry from Step 1, including preserved supplied ids and deterministic ids for unnamed criteria
|
|
17
17
|
- Test strategy and scenarios from Step 3
|
|
18
18
|
- Knowledge fragments loaded: playwright-utils-mandate, overview, api-request, recurse, log, auth-session, data-factories, api-testing-patterns
|
|
19
19
|
- Config: test framework, `use_playwright_utils` (default `true`), Pact.js Utils enabled/disabled (`use_pactjs_utils`), Pact MCP mode (`pact_mcp`)
|
|
@@ -72,7 +72,7 @@ import { test, expect } from '../support/merged-fixtures';
|
|
|
72
72
|
import { registrationPayload } from '../support/factories';
|
|
73
73
|
|
|
74
74
|
test.describe('[Story Name] API Tests (ATDD)', () => {
|
|
75
|
-
test.skip('[P0] should register new user successfully', async ({ apiRequest }) => {
|
|
75
|
+
test.skip('[P0] AC-1 should register new user successfully', async ({ apiRequest }) => {
|
|
76
76
|
// THIS TEST WILL FAIL - Endpoint not implemented yet
|
|
77
77
|
const { status, body } = await apiRequest<RegisteredUser>({
|
|
78
78
|
method: 'POST',
|
|
@@ -88,7 +88,7 @@ test.describe('[Story Name] API Tests (ATDD)', () => {
|
|
|
88
88
|
});
|
|
89
89
|
});
|
|
90
90
|
|
|
91
|
-
test.skip('[P1] should return 400 if email exists', async ({ apiRequest }) => {
|
|
91
|
+
test.skip('[P1] AC-2 should return 400 if email exists', async ({ apiRequest }) => {
|
|
92
92
|
// THIS TEST WILL FAIL - Endpoint not implemented yet
|
|
93
93
|
const { status, body } = await apiRequest({
|
|
94
94
|
method: 'POST',
|
|
@@ -110,7 +110,7 @@ test.describe('[Story Name] API Tests (ATDD)', () => {
|
|
|
110
110
|
import { test, expect } from '@playwright/test';
|
|
111
111
|
|
|
112
112
|
test.describe('[Story Name] API Tests (ATDD)', () => {
|
|
113
|
-
test.skip('[P0] should register new user successfully', async ({ request }) => {
|
|
113
|
+
test.skip('[P0] AC-1 should register new user successfully', async ({ request }) => {
|
|
114
114
|
const response = await request.post('/api/users/register', {
|
|
115
115
|
data: { email: 'newuser@example.com', password: 'SecurePass123!' },
|
|
116
116
|
});
|
|
@@ -139,6 +139,11 @@ If the merged-fixtures file does not exist yet, generate the import against `../
|
|
|
139
139
|
**CRITICAL ATDD Requirements:**
|
|
140
140
|
|
|
141
141
|
- ✅ Use `test.skip()` to mark tests as red-phase scaffolds
|
|
142
|
+
- ✅ Every leaf `test.skip()` title MUST include exactly one declared acceptance criterion id from the Step 1 registry in the form `[P#] AC-<n> description`; an id on `test.describe()` does not map the leaf test
|
|
143
|
+
- ✅ Generate exactly one red-phase leaf scaffold for every declared acceptance criterion. Record secondary branches and edge cases as green-phase checklist work; do not emit additional `test.skip()` leaves for the same criterion
|
|
144
|
+
- ✅ The criterion-defining assertion MUST be the first assertion that can fail and MUST isolate the exact newly promised status, scalar, or property with a direct matcher. Put broad object, schema, and secondary assertions after it
|
|
145
|
+
- ✅ Establish prerequisites through fixtures, provider state, or unasserted setup calls. Treat an unimplemented setup response as opaque before the criterion assertion: do not parse its body, branch on its result, throw from it, assert its status, or derive cleanup data from it. Use inputs known before the setup call and perform cleanup after the criterion assertion or in fixture teardown
|
|
146
|
+
- ✅ For a state-transition criterion, choose the transition-bearing branch for the primary scaffold and assert the newly promised state directly
|
|
142
147
|
- ✅ Write assertions for EXPECTED behavior (even though not implemented)
|
|
143
148
|
- ✅ Use realistic test data (not placeholder data)
|
|
144
149
|
- ✅ Test both happy path and error scenarios from acceptance criteria
|
|
@@ -185,7 +190,7 @@ When generating Pact consumer contract tests in the ATDD red phase, provider scr
|
|
|
185
190
|
* - Status: 201 for success, 400 for duplicate email, 422 for validation error
|
|
186
191
|
* - Response: { id: number, email: string, createdAt: string }
|
|
187
192
|
*/
|
|
188
|
-
test.skip('[P0] should generate consumer contract for user registration', async () => {
|
|
193
|
+
test.skip('[P0] AC-1 should generate consumer contract for user registration', async () => {
|
|
189
194
|
await provider
|
|
190
195
|
.given('no users exist')
|
|
191
196
|
.uponReceiving('a request to register a new user')
|
|
@@ -238,6 +243,10 @@ Write JSON to temp file: `/tmp/tea-atdd-api-tests-{{timestamp}}.json`
|
|
|
238
243
|
{
|
|
239
244
|
"success": true,
|
|
240
245
|
"subagent": "atdd-api-tests",
|
|
246
|
+
"criterion_registry": [
|
|
247
|
+
{ "id": "AC-1", "idSource": "supplied", "text": "A new user can register" },
|
|
248
|
+
{ "id": "AC-2", "idSource": "generated", "text": "A duplicate email is rejected" }
|
|
249
|
+
],
|
|
241
250
|
"tests": [
|
|
242
251
|
{
|
|
243
252
|
"file": "tests/api/user-registration.spec.ts",
|
|
@@ -13,7 +13,7 @@ This is an **isolated subagent** running in parallel with API red-phase test gen
|
|
|
13
13
|
|
|
14
14
|
**What you have from parent workflow:**
|
|
15
15
|
|
|
16
|
-
- Story acceptance
|
|
16
|
+
- Story acceptance criterion registry from Step 1, including preserved supplied ids and deterministic ids for unnamed criteria
|
|
17
17
|
- Test strategy and user journey scenarios from Step 3
|
|
18
18
|
- Knowledge fragments loaded: playwright-utils-mandate, overview, intercept-network-call, network-error-monitor, fixtures-composition, log, auth-session, fixture-architecture, network-first, selector-resilience
|
|
19
19
|
- Config: test framework, `use_playwright_utils` (default `true`)
|
|
@@ -101,46 +101,46 @@ For each user journey, create test file in `tests/e2e/[feature].spec.ts`:
|
|
|
101
101
|
import { test, expect } from '../support/merged-fixtures';
|
|
102
102
|
|
|
103
103
|
test.describe('[Story Name] E2E User Journey (ATDD)', () => {
|
|
104
|
-
test.skip('[P0] should complete user registration successfully', async ({ page, interceptNetworkCall }) => {
|
|
105
|
-
//
|
|
106
|
-
//
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
104
|
+
test.skip('[P0] AC-1 should complete user registration successfully', async ({ page, interceptNetworkCall }) => {
|
|
105
|
+
// The outer assertion is the first potentially failing operation. It owns
|
|
106
|
+
// every navigation, interaction, network, and outcome failure in this AC.
|
|
107
|
+
await expect(
|
|
108
|
+
(async () => {
|
|
109
|
+
const registerCall = interceptNetworkCall({ url: '**/api/users/register', method: 'POST' });
|
|
110
|
+
await page.goto('/register');
|
|
111
|
+
await page.getByLabel('Email').fill('newuser@example.com');
|
|
112
|
+
await page.getByLabel('Password').fill('SecurePass123!');
|
|
113
|
+
await page.getByRole('button', { name: 'Register' }).click();
|
|
114
|
+
|
|
115
|
+
const { status } = await registerCall;
|
|
116
|
+
expect(status).toBe(201);
|
|
117
|
+
await expect(page.getByText('Registration successful!')).toBeVisible();
|
|
118
|
+
await page.waitForURL('/dashboard');
|
|
119
|
+
})(),
|
|
120
|
+
).resolves.toBeUndefined();
|
|
121
121
|
});
|
|
122
122
|
|
|
123
123
|
// Stubs a 409 on purpose, so it opts out of network monitoring.
|
|
124
124
|
test.skip(
|
|
125
|
-
'[P1] should show error if email exists',
|
|
125
|
+
'[P1] AC-2 should show error if email exists',
|
|
126
126
|
{ annotation: [{ type: 'skipNetworkMonitoring' }] },
|
|
127
127
|
async ({ page, interceptNetworkCall }) => {
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
128
|
+
await expect(
|
|
129
|
+
(async () => {
|
|
130
|
+
const conflictCall = interceptNetworkCall({
|
|
131
|
+
url: '**/api/users/register',
|
|
132
|
+
method: 'POST',
|
|
133
|
+
fulfillResponse: { status: 409, body: { message: 'Email already exists' } },
|
|
134
|
+
});
|
|
135
|
+
await page.goto('/register');
|
|
136
|
+
await page.getByLabel('Email').fill('existing@example.com');
|
|
137
|
+
await page.getByLabel('Password').fill('SecurePass123!');
|
|
138
|
+
await page.getByRole('button', { name: 'Register' }).click();
|
|
139
|
+
|
|
140
|
+
await conflictCall;
|
|
141
|
+
await expect(page.getByText('Email already exists')).toBeVisible();
|
|
142
|
+
})(),
|
|
143
|
+
).resolves.toBeUndefined();
|
|
144
144
|
},
|
|
145
145
|
);
|
|
146
146
|
});
|
|
@@ -154,15 +154,17 @@ Note the selectors: `getByLabel` and `getByRole`, never `[name="email"]` or `but
|
|
|
154
154
|
import { test, expect } from '@playwright/test';
|
|
155
155
|
|
|
156
156
|
test.describe('[Story Name] E2E User Journey (ATDD)', () => {
|
|
157
|
-
test.skip('[P0] should complete user registration successfully', async ({ page }) => {
|
|
158
|
-
await
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
157
|
+
test.skip('[P0] AC-1 should complete user registration successfully', async ({ page }) => {
|
|
158
|
+
await expect(
|
|
159
|
+
(async () => {
|
|
160
|
+
await page.route('**/api/users/register', (route) => route.continue());
|
|
161
|
+
await page.goto('/register');
|
|
162
|
+
await page.getByLabel('Email').fill('newuser@example.com');
|
|
163
|
+
await page.getByLabel('Password').fill('SecurePass123!');
|
|
164
|
+
await page.getByRole('button', { name: 'Register' }).click();
|
|
165
|
+
await expect(page.getByText('Registration successful!')).toBeVisible();
|
|
166
|
+
})(),
|
|
167
|
+
).resolves.toBeUndefined();
|
|
166
168
|
});
|
|
167
169
|
});
|
|
168
170
|
```
|
|
@@ -187,6 +189,10 @@ If the merged-fixtures file does not exist yet, generate the import against `../
|
|
|
187
189
|
**CRITICAL ATDD Requirements:**
|
|
188
190
|
|
|
189
191
|
- ✅ Use `test.skip()` to mark tests as red-phase scaffolds
|
|
192
|
+
- ✅ Every leaf `test.skip()` title MUST include exactly one declared acceptance criterion id from the Step 1 registry in the form `[P#] AC-<n> description`; an id on `test.describe()` does not map the leaf test
|
|
193
|
+
- ✅ Generate exactly one red-phase leaf scaffold for every declared acceptance criterion. Record secondary journeys as green-phase checklist work; do not emit additional `test.skip()` leaves for the same criterion
|
|
194
|
+
- ✅ The criterion-defining assertion MUST be the first potentially failing operation. Wrap the complete browser journey in one `expect((async () => { ... })()).resolves.toBeUndefined()` assertion so navigation, locator, interaction, network, and outcome failures retain criterion provenance without retrying stateful steps
|
|
195
|
+
- ✅ For a state-transition criterion, choose the transition-bearing branch for the primary scaffold and assert the newly promised state directly
|
|
190
196
|
- ✅ Write assertions for EXPECTED UI behavior (even though not implemented)
|
|
191
197
|
- ✅ Use resilient selectors: getByRole, getByText, getByLabel (from selector-resilience). Never `[name="..."]`, `button:has-text(...)`, CSS classes, or XPath
|
|
192
198
|
- ✅ Network-first: interception declared before navigation when API calls are involved (from network-first; mechanism per the mandate when enabled)
|
|
@@ -222,6 +228,10 @@ Write JSON to temp file: `/tmp/tea-atdd-e2e-tests-{{timestamp}}.json`
|
|
|
222
228
|
{
|
|
223
229
|
"success": true,
|
|
224
230
|
"subagent": "atdd-e2e-tests",
|
|
231
|
+
"criterion_registry": [
|
|
232
|
+
{ "id": "AC-1", "idSource": "supplied", "text": "A new user can register" },
|
|
233
|
+
{ "id": "AC-2", "idSource": "generated", "text": "A duplicate email is rejected" }
|
|
234
|
+
],
|
|
225
235
|
"tests": [
|
|
226
236
|
{
|
|
227
237
|
"file": "tests/e2e/user-registration.spec.ts",
|
|
@@ -66,6 +66,7 @@ const e2eTestsOutput = JSON.parse(fs.readFileSync(e2eTestsPath, 'utf8'));
|
|
|
66
66
|
- Check `apiTestsOutput.success === true`
|
|
67
67
|
- Check `e2eTestsOutput.success === true`
|
|
68
68
|
- If either failed, report error and stop (don't proceed)
|
|
69
|
+
- Require both workers' `criterion_registry` to match the persisted Step 1 registry exactly, including order, ids, `idSource`, and text
|
|
69
70
|
|
|
70
71
|
---
|
|
71
72
|
|
|
@@ -73,6 +74,20 @@ const e2eTestsOutput = JSON.parse(fs.readFileSync(e2eTestsPath, 'utf8'));
|
|
|
73
74
|
|
|
74
75
|
**CRITICAL TDD Validation:**
|
|
75
76
|
|
|
77
|
+
Load the exact criterion registry persisted by Step 1. For every leaf `test.skip()` title from both worker outputs, extract all tokens matching `\bAC-\d+\b`. Require exactly one token, then require that token to be a member of the registry. Reject titles with zero ids, multiple ids, repeated copies of one id, or an undeclared id. Count mapped leaf titles per criterion across both worker outputs. Require exactly one red-phase leaf for every declared criterion. An id present only on a containing `test.describe()` does not count. Record secondary branches and journeys in the implementation checklist for green-phase automation.
|
|
78
|
+
|
|
79
|
+
For each criterion's primary scaffold, verify that the criterion-defining assertion is the first assertion that can fail and directly isolates the exact newly promised status, scalar, or property. Broad object, schema, and secondary assertions must follow it. An API setup call to an unimplemented endpoint must remain opaque before this assertion: no body parsing, result branching, explicit throw, status assertion, or cleanup-data derivation. In E2E scaffolds the criterion assertion must also be the first potentially failing operation and must own the complete browser journey. For a state-transition criterion, verify that the primary scaffold exercises the transition-bearing branch and directly asserts the newly promised state.
|
|
80
|
+
|
|
81
|
+
Use this title guard as written:
|
|
82
|
+
|
|
83
|
+
```javascript
|
|
84
|
+
const declaredIds = new Set(criterionRegistry.map(({ id }) => id));
|
|
85
|
+
const ids = title.match(/\bAC-\d+\b/g) ?? [];
|
|
86
|
+
if (ids.length !== 1 || !declaredIds.has(ids[0])) {
|
|
87
|
+
throw new Error(`ATDD ERROR: leaf title must carry exactly one declared criterion id: ${title}`);
|
|
88
|
+
}
|
|
89
|
+
```
|
|
90
|
+
|
|
76
91
|
**Check API tests:**
|
|
77
92
|
|
|
78
93
|
```javascript
|
|
@@ -118,6 +133,9 @@ e2eTestsOutput.tests.forEach((test) => {
|
|
|
118
133
|
```text
|
|
119
134
|
✅ TDD Red Phase Validation: PASS
|
|
120
135
|
- All tests use test.skip()
|
|
136
|
+
- Every leaf test maps to exactly one declared acceptance criterion
|
|
137
|
+
- Every declared acceptance criterion has exactly one red-phase leaf scaffold
|
|
138
|
+
- Each primary scaffold reaches a direct, exact criterion-defining assertion first
|
|
121
139
|
- All tests assert expected behavior (not placeholders)
|
|
122
140
|
- All tests marked as expected_to_fail
|
|
123
141
|
```
|
|
@@ -43,6 +43,14 @@ Use `checklist.md` to validate:
|
|
|
43
43
|
- Test files created correctly
|
|
44
44
|
- Checklist matches acceptance criteria
|
|
45
45
|
- Tests are generated as red-phase scaffolds and marked with `test.skip()`
|
|
46
|
+
- Preflight preserved supplied criterion ids and assigned stable, collision-free ids to unnamed criteria
|
|
47
|
+
- Every executable leaf title carries exactly one id from the persisted criterion registry
|
|
48
|
+
- Every declared acceptance criterion has exactly one red-phase leaf scaffold
|
|
49
|
+
- Secondary branches and journeys are checklist work for green-phase automation
|
|
50
|
+
- The criterion-defining assertion is the first assertion that can fail and directly isolates the exact newly promised status, scalar, or property
|
|
51
|
+
- API setup responses from unimplemented endpoints remain opaque before the criterion assertion
|
|
52
|
+
- Each E2E primary scaffold begins with the criterion-defining assertion as its first potentially failing operation
|
|
53
|
+
- State-transition criteria exercise the transition-bearing branch in their primary scaffold
|
|
46
54
|
- Story metadata and handoff paths are captured for downstream workflows
|
|
47
55
|
- [ ] CLI sessions cleaned up (no orphaned browsers)
|
|
48
56
|
- [ ] Temp artifacts stored in `{test_artifacts}/` not random locations
|
|
@@ -59,6 +59,8 @@ Read `{validationChecklist}` and list all criteria.
|
|
|
59
59
|
|
|
60
60
|
Evaluate outputs against each checklist item.
|
|
61
61
|
|
|
62
|
+
For ATDD test artifacts, reconstruct the criterion registry from the selected checklist. Confirm supplied ids were preserved and generated ids follow the lowest-unused `AC-<n>` rule in source order. Extract all `\bAC-\d+\b` tokens from every executable leaf title. Require exactly one token and require it to exist in the registry. Require exactly one red-phase leaf per declared criterion. For every scaffold, confirm the first assertion directly isolates the exact newly promised status, scalar, or property. Confirm API setup responses from unimplemented endpoints remain opaque before that assertion. For each E2E scaffold, confirm the criterion-defining assertion is the first potentially failing operation and owns the complete browser journey.
|
|
63
|
+
|
|
62
64
|
### 4. Write Report
|
|
63
65
|
|
|
64
66
|
Replace the `IN_PROGRESS` body in this run's reserved `{outputFile}` with the final validation report. Include PASS/WARN/FAIL per section plus the original `validation_scope`, `run_timestamp`, and `validated_artifacts` metadata. Record every selected artifact using its exact project-relative path.
|