bmad-method-test-architecture-enterprise 1.3.0 β 1.3.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/explanation/testing-as-engineering.md +39 -35
- package/docs/how-to/customization/configure-browser-automation.md +1 -1
- package/docs/reference/configuration.md +2 -2
- package/docs/reference/troubleshooting.md +1 -1
- package/package.json +1 -1
- package/release_notes.md +9 -6
- package/src/testarch/knowledge/pactjs-utils-consumer-helpers.md +1 -0
- package/src/testarch/knowledge/pactjs-utils-provider-verifier.md +2 -2
- package/src/testarch/knowledge/pactjs-utils-request-filter.md +20 -13
- package/src/workflows/testarch/framework/steps-c/step-03-scaffold-framework.md +2 -2
- package/tools/validate-doc-links.js +1 -1
- package/docs/how-to/customization/enable-tea-mcp-enhancements.md +0 -23
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
---
|
|
2
2
|
title: 'AI-Generated Testing: Why Most Approaches Fail'
|
|
3
|
-
description: How Playwright-Utils, TEA workflows, and
|
|
3
|
+
description: How Playwright-Utils, pactjs-utils, TEA workflows, Playwright CLI, and MCPs solve AI test quality problems
|
|
4
4
|
---
|
|
5
5
|
|
|
6
|
-
AI-generated tests frequently fail in production because they lack systematic quality standards. This document explains the problem and presents a solution combining
|
|
6
|
+
AI-generated tests frequently fail in production because they lack systematic quality standards. This document explains the problem and presents a solution combining utility standards, TEA workflows, and automation interfaces across both UI, API and contract testing.
|
|
7
7
|
|
|
8
8
|
:::note[Source]
|
|
9
9
|
This article is adapted from [The Testing Meta Most Teams Have Not Caught Up To Yet](https://dev.to/muratkeremozcan/the-testing-meta-most-teams-have-not-caught-up-to-yet-5765) by Murat K Ozcan.
|
|
@@ -30,25 +30,22 @@ AI excels at generating code quickly, but testing requires precision and consist
|
|
|
30
30
|
|
|
31
31
|
The solution combines three components that work together to enforce quality:
|
|
32
32
|
|
|
33
|
-
### Playwright-Utils
|
|
33
|
+
### 1. Utilities: Playwright-Utils + Pact.js Utils
|
|
34
34
|
|
|
35
|
-
|
|
35
|
+
`@seontechnologies/playwright-utils` standardizes commonly reinvented testing primitives across UI, API, web, and non-web flows. `@seontechnologies/pactjs-utils` standardizes Pact.js contract-testing primitives for provider state setup, request filtering, and provider/message verifier configuration.
|
|
36
36
|
|
|
37
|
-
| Utility
|
|
38
|
-
|
|
|
39
|
-
|
|
|
40
|
-
|
|
|
41
|
-
| intercept-network-call | Network mocking and interception |
|
|
42
|
-
| recurse | Retry logic and polling |
|
|
43
|
-
| log | Structured logging |
|
|
44
|
-
| network-recorder | Record and replay network traffic |
|
|
45
|
-
| burn-in | Smart test selection for CI |
|
|
46
|
-
| network-error-monitor | HTTP error detection |
|
|
47
|
-
| file-utils | CSV/PDF handling |
|
|
37
|
+
| Track | Utility Layer | Purpose |
|
|
38
|
+
| ------------------ | ------------------------------------ | -------------------------------------------------------- |
|
|
39
|
+
| UI/API/Web/Non-web | `@seontechnologies/playwright-utils` | Reusable testing primitives and fixtures |
|
|
40
|
+
| Contract | `@seontechnologies/pactjs-utils` | Reusable Pact consumer/provider helpers and verification |
|
|
48
41
|
|
|
49
|
-
|
|
42
|
+
**Playwright-Utils examples:** `api-request`, `auth-session`, `intercept-network-call`, `recurse`, `log`, `network-recorder`, `burn-in`, `network-error-monitor`, `file-utils`.
|
|
50
43
|
|
|
51
|
-
|
|
44
|
+
**pactjs-utils examples:** `createProviderState`, `toJsonMap`, `createRequestFilter`, `noOpRequestFilter`, `buildVerifierOptions`, `buildMessageVerifierOptions`.
|
|
45
|
+
|
|
46
|
+
Together, these utility libraries eliminate the need to reinvent core testing primitives across UI, API, web, non-web, and contract testing.
|
|
47
|
+
|
|
48
|
+
### 2. Process: TEA (Test Architect Agent)
|
|
52
49
|
|
|
53
50
|
A quality operating model packaged as eight executable workflows spanning test design, CI/CD gates, and release readiness. TEA encodes test architecture expertise into repeatable processes.
|
|
54
51
|
|
|
@@ -67,28 +64,35 @@ A quality operating model packaged as eight executable workflows spanning test d
|
|
|
67
64
|
TEA doesn't just generate testsβit provides a complete quality operating model with workflows for planning, execution, and release gates.
|
|
68
65
|
:::
|
|
69
66
|
|
|
70
|
-
### Playwright MCPs
|
|
67
|
+
### 3. Automation Interfaces: Playwright CLI + MCPs
|
|
68
|
+
|
|
69
|
+
Automation interfaces enable real-time verification during test generation and review across browser and contract tracks:
|
|
70
|
+
|
|
71
|
+
- **Playwright CLI**: token-efficient browser automation for stateless execution and fast checks in workflows.
|
|
72
|
+
- **Playwright MCP**: stateful browser automation with richer context for interactive exploration and DOM validation.
|
|
73
|
+
- **Pact MCP**: broker-aware contract automation for verification matrix queries, provider-state discovery, compatibility analysis, and `can-i-deploy` deployment decisions.
|
|
71
74
|
|
|
72
|
-
|
|
75
|
+
Instead of inferring behavior from documentation alone, these interfaces allow agents to:
|
|
73
76
|
|
|
74
|
-
- Run flows and confirm the DOM against the accessibility tree
|
|
75
|
-
- Validate network
|
|
76
|
-
-
|
|
77
|
-
-
|
|
77
|
+
- Run browser flows and confirm the DOM against the accessibility tree
|
|
78
|
+
- Validate UI/API network behavior in real-time
|
|
79
|
+
- Query Pact verification matrix results across consumer/provider versions
|
|
80
|
+
- Check provider states and contract compatibility before release
|
|
81
|
+
- Execute `can-i-deploy` checks against target environments
|
|
78
82
|
|
|
79
83
|
## How They Work Together
|
|
80
84
|
|
|
81
85
|
The three components form a quality pipeline:
|
|
82
86
|
|
|
83
|
-
| Stage | Component
|
|
84
|
-
| ------------ |
|
|
85
|
-
| Standards | Playwright-Utils | Provides production-ready patterns and
|
|
86
|
-
| Process | TEA Workflows
|
|
87
|
-
| Verification | Playwright
|
|
87
|
+
| Stage | Component | Action |
|
|
88
|
+
| ------------ | ------------------------------------------ | ------------------------------------------------------------ |
|
|
89
|
+
| Standards | Playwright-Utils + pactjs-utils | Provides production-ready patterns for UI and contract tests |
|
|
90
|
+
| Process | TEA Workflows | Enforces systematic test planning and review |
|
|
91
|
+
| Verification | Playwright CLI + Playwright MCP + Pact MCP | Validates tests and contracts against live systems |
|
|
88
92
|
|
|
89
93
|
**Before (AI-only):** 20 tests with redundant coverage, incorrect assertions, and flaky behavior.
|
|
90
94
|
|
|
91
|
-
**After (Full Stack):** Risk-based selection, verified selectors, validated behavior, reviewable code.
|
|
95
|
+
**After (Full Stack):** Risk-based selection, verified selectors, validated behavior, contract compatibility checks, reviewable code.
|
|
92
96
|
|
|
93
97
|
## Why This Matters
|
|
94
98
|
|
|
@@ -101,11 +105,11 @@ Traditional AI testing approaches fail because they:
|
|
|
101
105
|
|
|
102
106
|
The three-part stack addresses each gap:
|
|
103
107
|
|
|
104
|
-
| Gap | Solution
|
|
105
|
-
| --------------- |
|
|
106
|
-
| No standards | Playwright-Utils
|
|
107
|
-
| No planning | TEA `test-design` creates risk-based test plans
|
|
108
|
-
| No verification | Playwright
|
|
109
|
-
| No review | TEA `test-review` audits quality with scoring
|
|
108
|
+
| Gap | Solution |
|
|
109
|
+
| --------------- | ------------------------------------------------------------------------ |
|
|
110
|
+
| No standards | Playwright-Utils + pactjs-utils provide production-ready patterns |
|
|
111
|
+
| No planning | TEA `test-design` creates risk-based test plans |
|
|
112
|
+
| No verification | Playwright CLI + Playwright MCP + Pact MCP validate against live systems |
|
|
113
|
+
| No review | TEA `test-review` audits quality with scoring |
|
|
110
114
|
|
|
111
115
|
This approach is sometimes called _context engineering_βloading domain-specific standards into AI context automatically rather than relying on prompts alone. TEA's `tea-index.csv` manifest loads relevant knowledge fragments so the AI doesn't relearn testing patterns each session.
|
|
@@ -62,7 +62,7 @@ Add these MCP server entries to your tool's configuration file:
|
|
|
62
62
|
}
|
|
63
63
|
```
|
|
64
64
|
|
|
65
|
-
The `smartbear` server is optional β only needed if you use the [Pact MCP integration](/docs/
|
|
65
|
+
The `smartbear` server is optional β only needed if you use the [Pact MCP integration](/docs/reference/configuration.md#tea_pact_mcp) for contract testing workflows. See the [pact-mcp knowledge fragment](/docs/reference/knowledge-base.md#pact-contract-testing-integration) for details.
|
|
66
66
|
|
|
67
67
|
#### Where to put the config
|
|
68
68
|
|
|
@@ -141,7 +141,7 @@ Enable Pact.js Utils integration for production-ready contract testing utilities
|
|
|
141
141
|
|
|
142
142
|
**Type:** `boolean`
|
|
143
143
|
|
|
144
|
-
**Default:** `
|
|
144
|
+
**Default:** `true`
|
|
145
145
|
|
|
146
146
|
**Installer Prompt:**
|
|
147
147
|
|
|
@@ -201,7 +201,7 @@ Pact MCP strategy for broker interaction during contract testing workflows.
|
|
|
201
201
|
|
|
202
202
|
**Type:** `string`
|
|
203
203
|
|
|
204
|
-
**Default:** `"
|
|
204
|
+
**Default:** `"mcp"`
|
|
205
205
|
|
|
206
206
|
**Options:** `"mcp"` | `"none"`
|
|
207
207
|
|
|
@@ -593,7 +593,7 @@ If the BMAD installer can run but cannot fetch the Test Architect module from Gi
|
|
|
593
593
|
|
|
594
594
|
**Causes**:
|
|
595
595
|
|
|
596
|
-
-
|
|
596
|
+
- Up to 40 fragments loading at once (depends on workflow and enabled integrations)
|
|
597
597
|
- Large fragment file sizes
|
|
598
598
|
- Disk I/O bottleneck
|
|
599
599
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.3.
|
|
4
|
+
"version": "1.3.1",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
package/release_notes.md
CHANGED
|
@@ -1,10 +1,13 @@
|
|
|
1
|
-
## π What's New in v1.3.
|
|
2
|
-
|
|
3
|
-
### β¨ New Features
|
|
4
|
-
- feat: pactjs utils support
|
|
1
|
+
## π What's New in v1.3.1
|
|
5
2
|
|
|
6
3
|
### π¦ Other Changes
|
|
7
|
-
-
|
|
4
|
+
- docs: update pactjs utils
|
|
5
|
+
- addressed review commments
|
|
6
|
+
- Merge pull request #42 from bmad-code-org/docs/update-pactjs-utils
|
|
7
|
+
- Merge pull request #43 from bmad-code-org/chore/refresh-lock
|
|
8
|
+
|
|
9
|
+
### π§ Maintenance
|
|
10
|
+
- chore: refresh lock
|
|
8
11
|
|
|
9
12
|
|
|
10
13
|
## π¦ Installation
|
|
@@ -14,4 +17,4 @@ npx bmad-method install
|
|
|
14
17
|
# Select "Test Architect" from module menu
|
|
15
18
|
```
|
|
16
19
|
|
|
17
|
-
**Full Changelog**: https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/compare/v1.
|
|
20
|
+
**Full Changelog**: https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/compare/v1.3.0...v1.3.1
|
|
@@ -154,6 +154,7 @@ await provider
|
|
|
154
154
|
- **Null handling**: `null` becomes `"null"` string in JsonMap (Pact requirement)
|
|
155
155
|
- **Date handling**: Date objects become ISO 8601 strings
|
|
156
156
|
- **No nested objects in JsonMap**: Nested objects are JSON-stringified β provider state handlers must parse them
|
|
157
|
+
- **Array serialization is lossy**: Arrays are converted via `String()` (e.g., `[1,2,3]` β `"1,2,3"`) β prefer passing arrays as JSON-stringified objects for round-trip safety
|
|
157
158
|
- **Message pacts**: Works identically with `MessageConsumerPact` β same `.given()` API
|
|
158
159
|
|
|
159
160
|
## Related Fragments
|
|
@@ -265,8 +265,8 @@ const opts: VerifierOptions = {
|
|
|
265
265
|
providerVersionBranch: process.env.GITHUB_HEAD_REF || process.env.GITHUB_REF_NAME,
|
|
266
266
|
consumerVersionSelectors:
|
|
267
267
|
process.env.PACT_BREAKING_CHANGE === 'true'
|
|
268
|
-
? [{
|
|
269
|
-
: [{
|
|
268
|
+
? [{ matchingBranch: true }]
|
|
269
|
+
: [{ matchingBranch: true }, { mainBranch: true }, { deployedOrReleased: true }],
|
|
270
270
|
pactUrls: process.env.PACT_PAYLOAD_URL ? [process.env.PACT_PAYLOAD_URL] : undefined,
|
|
271
271
|
stateHandlers: {
|
|
272
272
|
/* ... */
|
|
@@ -198,20 +198,27 @@ const opts: VerifierOptions = {
|
|
|
198
198
|
### Right: Separate auth into createRequestFilter
|
|
199
199
|
|
|
200
200
|
```typescript
|
|
201
|
-
// β
Clean separation
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
});
|
|
201
|
+
// β
Clean separation β async setup wraps token fetch (CommonJS-safe)
|
|
202
|
+
async function setupVerifierOptions() {
|
|
203
|
+
const token = await fetchAuthToken(); // Resolve async token BEFORE creating filter
|
|
205
204
|
|
|
206
|
-
const
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
205
|
+
const requestFilter = createRequestFilter({
|
|
206
|
+
tokenGenerator: () => token, // Synchronous β returns pre-fetched value
|
|
207
|
+
});
|
|
208
|
+
|
|
209
|
+
return buildVerifierOptions({
|
|
210
|
+
provider: 'my-api',
|
|
211
|
+
port: '3001',
|
|
212
|
+
includeMainAndDeployed: true,
|
|
213
|
+
requestFilter,
|
|
214
|
+
stateHandlers: {
|
|
215
|
+
/* ... */
|
|
216
|
+
},
|
|
217
|
+
});
|
|
218
|
+
}
|
|
219
|
+
|
|
220
|
+
// In tests/hooks, callers can await setupVerifierOptions():
|
|
221
|
+
// const opts = await setupVerifierOptions();
|
|
215
222
|
```
|
|
216
223
|
|
|
217
224
|
_Source: @seontechnologies/pactjs-utils request-filter module, pact-js-example-provider verification tests_
|
|
@@ -59,7 +59,7 @@ Create the idiomatic test directory for the detected language:
|
|
|
59
59
|
- **Ruby (RSpec)**: `spec/` with `spec/unit/`, `spec/integration/`, `spec/api/`, `spec/support/`
|
|
60
60
|
- **Rust**: `tests/` for integration tests, inline `#[cfg(test)]` modules for unit tests
|
|
61
61
|
|
|
62
|
-
**If `tea_use_pactjs_utils` is enabled** (and `{detected_stack}` is `backend` or `fullstack`):
|
|
62
|
+
**If `config.tea_use_pactjs_utils` is enabled** (and `{detected_stack}` is `backend` or `fullstack`):
|
|
63
63
|
|
|
64
64
|
Create contract testing directory structure:
|
|
65
65
|
|
|
@@ -180,7 +180,7 @@ Create helpers for:
|
|
|
180
180
|
- Auth helpers
|
|
181
181
|
- Test data factories (language-idiomatic patterns)
|
|
182
182
|
|
|
183
|
-
**If `tea_use_pactjs_utils` is enabled** (and `{detected_stack}` is `backend` or `fullstack`):
|
|
183
|
+
**If `config.tea_use_pactjs_utils` is enabled** (and `{detected_stack}` is `backend` or `fullstack`):
|
|
184
184
|
|
|
185
185
|
Create contract test samples in `pact/` directory:
|
|
186
186
|
|
|
@@ -279,7 +279,7 @@ function applyFixes(content, issues) {
|
|
|
279
279
|
if (issue.status === 'auto-fixable' && issue.suggestedFix) {
|
|
280
280
|
const oldLink = `[${issue.linkText}](${issue.href})`;
|
|
281
281
|
const newLink = `[${issue.linkText}](${issue.suggestedFix})`;
|
|
282
|
-
updated = updated.
|
|
282
|
+
updated = updated.replaceAll(oldLink, newLink);
|
|
283
283
|
}
|
|
284
284
|
}
|
|
285
285
|
|
|
@@ -1,23 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: 'Enable TEA MCP Enhancements (Deprecated)'
|
|
3
|
-
description: This guide has been superseded by Configure Browser Automation
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
# Enable TEA MCP Enhancements
|
|
7
|
-
|
|
8
|
-
> **Deprecated:** This guide has been replaced by [Configure Browser Automation](/docs/how-to/customization/configure-browser-automation.md).
|
|
9
|
-
|
|
10
|
-
The `tea_use_mcp_enhancements` boolean flag has been replaced by `tea_browser_automation`, which supports four modes: `auto`, `cli`, `mcp`, and `none`.
|
|
11
|
-
|
|
12
|
-
**Migration:**
|
|
13
|
-
|
|
14
|
-
| Old Setting | New Equivalent |
|
|
15
|
-
| --------------------------------- | -------------------------------- |
|
|
16
|
-
| `tea_use_mcp_enhancements: true` | `tea_browser_automation: "auto"` |
|
|
17
|
-
| `tea_use_mcp_enhancements: false` | `tea_browser_automation: "none"` |
|
|
18
|
-
|
|
19
|
-
Please see [Configure Browser Automation](/docs/how-to/customization/configure-browser-automation.md) for full setup instructions.
|
|
20
|
-
|
|
21
|
-
---
|
|
22
|
-
|
|
23
|
-
Generated with [BMad Method](https://bmad-method.org) - TEA (Test Engineering Architect)
|