bmad-method-test-architecture-enterprise 2.0.1-next.0 → 2.0.1-next.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -31,7 +31,7 @@
31
31
  "name": "bmad-method-test-architecture-enterprise",
32
32
  "source": "./",
33
33
  "description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
34
- "version": "2.0.1-next.0",
34
+ "version": "2.0.1-next.1",
35
35
  "author": {
36
36
  "name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
37
37
  },
package/README.md CHANGED
@@ -1,859 +1,134 @@
1
1
  # TEA: Test Engineering Architect
2
2
 
3
- [Node Version](https://nodejs.org)
4
- [License: MIT](./LICENSE)
3
+ [Documentation](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/) · [Getting started](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/tea-lite-quickstart/) · [Contributing](./CONTRIBUTING.md) · [MIT license](./LICENSE)
5
4
 
6
- **TEA** stands for **Test Engineering Architect**. The npm package and repository slug `bmad-method-test-architecture-enterprise` is a package name and never an expansion of the acronym.
5
+ TEA is a BMad module for test planning, automation, evaluation, and release decisions.
6
+ Its agent, Murat, uses ten workflows and a shared testing knowledge base to turn requirements and risks into tests and evidence.
7
+ You can use TEA on its own or alongside BMad Method.
7
8
 
8
- TEA is a standalone BMAD module that delivers risk-based test strategy, test automation guidance, and release gate decisions. It ships:
9
+ TEA helps you:
9
10
 
10
- - one expert agent, Murat, Master Test Architect and Quality Advisor
11
- - ten workflows spanning Teach Me Testing (TEA Academy), test design, framework setup, CI guidance, ATDD, automation, test review, NFR Evidence Audit, traceability, and Evaluate ([evaluate your first skill](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/evaluate-your-first-skill/))
12
- - a 35-row criteria registry that fixes the severity of every reviewable violation, so a score is a lookup rather than a judgment call
13
- - `tea-test-review`, a headless CLI that runs the review workflow as a CI gate with real exit codes
14
- - a write-time enforcement hook that blocks the mechanically decidable violations before they reach disk
15
-
16
- TEA is two layers. **TEA Core** decides what must be verified, at what depth, with what evidence, and whether that evidence is sufficient to release; it assumes nothing about your language, framework, or platform. **Execution targets** turn those decisions into runnable tests on a specific stack, and that layer is swappable. See [Verification Architecture](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/verification-architecture/) for the split, and [Execution Targets](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/execution-targets/) for exactly which stacks are covered at which depth.
17
-
18
- Docs: [https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/)
19
-
20
- ## Why TEA
21
-
22
- - Risk-based prioritization (P0-P3) from probability × impact, with measurable quality gates
23
- - Requirements traced to evidence, and PASS / CONCERNS / FAIL / WAIVED release decisions that survive an audit
24
- - NFR thresholds set at design time and audited against real evidence, defaulting to CONCERNS when evidence is missing
25
- - Consistent, knowledge-base driven outputs instead of whatever the model felt like producing
26
- - Stack-aware execution: Playwright and Cypress for browsers, Maestro for mobile native, Pact for contracts, pytest / JUnit / Go test / xUnit / RSpec for backend services, k6 and scanners as NFR evidence
27
- - Three enforcement points rather than one: fragments steer generation, a hook blocks the write, and `test-review` scores what actually landed
28
-
29
- ## How BMad Works
30
-
31
- BMad works because it turns big, fuzzy work into **repeatable workflows**. Each workflow is broken into small steps with clear instructions, so the AI follows the same path every time. It also uses a **shared knowledge base** (standards and patterns) so outputs are consistent, not random. In short: **structured steps + shared standards = reliable results**.
32
-
33
- ## How TEA Fits In
34
-
35
- TEA plugs into BMad the same way a specialist plugs into a team. It uses the same step‑by‑step workflow engine and shared standards, but focuses exclusively on testing and quality gates. That means you get a **risk‑based test plan**, **automation guidance**, and **go/no‑go decisions** that align with the rest of the BMad process.
36
-
37
- ## How It Actually Works
38
-
39
- ### The problem it is built around
40
-
41
- Ask a model to "write tests for this feature" and you reliably get four things: redundant coverage, incorrect assertions, flaky tests, and diffs nobody can review. The cause is a category error. Prompt-driven generation is nondeterministic, and it is being pointed at the one artifact whose entire job is determinism.
42
-
43
- TEA's answer is not a better prompt. It is to make the work repeatable at three levels.
44
-
45
- **Repeatable instructions.** A single 5,000-word instruction file fails predictably: the model skims it, improvises past the vague parts ("analyze codebase then generate tests" specifies nothing), keeps going because nothing told it where to stop, and returns something different next run. So every workflow is cut into step files that each do one thing, state what "finished" means, restate the context they need, list what they must not do, and load one at a time. Consistent output for the same input is what makes everything else possible: you cannot parallelize work whose boundaries are undefined.
46
-
47
- **Repeatable standards.** 59 knowledge fragments carry the patterns, and an index decides which ones enter context for the task at hand. The model is not asked to remember how fixtures compose or what network-first means. It is handed the fragment.
48
-
49
- **Repeatable judgment.** Risk scores, priorities, quality scores, and gate decisions are computed from stated rules rather than produced as opinions. This is the part most tools skip, and it is the difference between a review you can act on and a review you have to re-litigate.
50
-
51
- ### The order you run things
52
-
53
- The ten workflows are a directed graph, not a menu. `skills/bmod-tea/help/help.md` describes it, and the `bmad` skill reads it to recommend the next step.
54
-
55
- ```text
56
- Phase 3, solutioning, once per project
57
- TD test-design (system-level) → TF framework → CI ci
58
-
59
- Phase 4, implementation, per story
60
- create-story → AT atdd → dev implements → TA automate
61
-
62
- Epic or release gate
63
- TA automate → RV test-review
64
- TA automate → NR nfr
65
- TA automate → EV evaluate → CI ci
66
- RV test-review → TR trace (Phase 2 gate decision)
67
- ```
68
-
69
- Phase 3 order matters and is deliberate: run `test-design` first so NFR evidence needs can shape the infrastructure, then `framework` once the architecture and the test design have settled the stack, then `ci` once the framework exists so the pipeline wires to real commands.
70
-
71
- `test-design` is dual-mode. At system level it produces an architecture-facing document and a QA-facing one. Per epic it produces `test-design/test-design-epic-N.md`. `teach-me-testing` sits outside the lifecycle and runs once per learner.
72
-
73
- For the full lifecycle diagram including the BMad phases around TEA, see [TEA Overview](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/tea-overview/).
74
-
75
- ### One epic, end to end
76
-
77
- Here is the same feature moving through the chain, with the rules TEA actually applies at each step.
78
-
79
- **1. Risk, in** `test-design`**.** Every identified risk gets a probability of 1 to 3 (unlikely, possible, likely) and an impact of 1 to 3 (minor, degraded, critical). Score is the product, so the range is 1 to 9, and the score determines the action:
80
-
81
- | Score | Action | Gate impact |
82
- | ----- | -------- | -------------------- |
83
- | 1-3 | DOCUMENT | none |
84
- | 4-5 | MONITOR | none, watch closely |
85
- | 6-8 | MITIGATE | CONCERNS at the gate |
86
- | 9 | BLOCK | automatic FAIL |
87
-
88
- A checkout risk scored `probability 2 × impact 3 = 6` lands in MITIGATE. It gets a row in the test design with a named owner and a date, and it will surface as CONCERNS at the gate until the mitigation is real.
89
-
90
- Priority is a separate judgment that the risk score informs rather than determines. P0 is revenue-critical, security-critical, data-integrity, regulatory, or previously broken. P1 is core journeys and complex logic. P2 is secondary features. P3 is nice-to-have. The design pins the effort too: P0 tests are budgeted at 2 hours each, P1 at 1, P2 at 0.5, P3 at 0.25.
91
-
92
- **2. Test level, still in** `test-design`**.** Favor unit when logic can be isolated with no side effects; integration for persistence, service contracts, and component boundaries; E2E for user-facing critical paths and multi-system interactions. Before adding any test, the duplicate-coverage guard asks whether a lower level already covers it. Overlap is allowed only for genuinely different aspects, defense in depth on critical paths, or regression prevention on something that broke before.
93
-
94
- **3. Red tests, in** `atdd` (optional). Run before implementation. It generates acceptance tests that all carry `test.skip()`, plus data factories, fixtures, and an implementation checklist that lists, per test, the tasks required to make it pass and the command to run it. The developer un-skips one test, confirms it fails, then makes it pass. The red phase is the point: a test that has never failed has proven nothing.
95
-
96
- **4. Coverage, in** `automate`**.** Run after implementation. Workers generate API, E2E, backend, and mobile tests in parallel depending on the detected stack, and the aggregation step reports the totals broken down by priority. It also rolls up every deviation from an active integration mandate as `file:line: reason`, and writes `None` when there are none, because a reader cannot tell an empty section from a forgotten one.
97
-
98
- **5. Quality, in** `test-review`**.** Every finding must cite a registry row (`C1`, `H2`, `M4`), and the row carries the severity. Score starts at 100:
99
-
100
- ```text
101
- Starting Score: 100
102
- Critical Violations: -{count} × 10
103
- High Violations: -{count} × 5
104
- Medium Violations: -{count} × 2
105
- Low Violations: -{count} × 1
106
- Bonus (5 categories, each 0 or 5, max +25)
107
- Final Score: clamped to 0-100 Grade: A ≥90, B ≥80, C ≥70, D ≥60, else F
108
- ```
109
-
110
- The verdict is then derived from the findings, not written by the model:
111
-
112
- ```javascript
113
- if (CRITICAL > 0) return 'Block'; // a test that cannot fail is not a suggestion
114
- if (HIGH > 0) return 'Request Changes';
115
- if (score < 70) return 'Request Changes'; // volume of MEDIUM/LOW can also fail the bar
116
- if (MEDIUM + LOW > 0) return 'Approve with Comments';
117
- return 'Approve';
118
- ```
119
-
120
- This is the part worth sitting with. A suite with one CRITICAL and three MEDIUM findings, earning two bonus categories, scores `100 - 10 - 6 + 10 = 94`, a grade A, and is still a **Block**. Score measures the suite. The verdict answers a different question: is there anything here that makes the suite lie? One committed `.skip` on the test that mattered, or one `expect(true).toBe(true)`, means green proves nothing, which is worse than an absent test because it buys false confidence.
121
-
122
- That separation exists because it was measured. Two reviewers of the same four files scored 82 and 85, which is noise, and returned opposite verdicts, which is not. `--fail-on request-changes` acts on the verdict, so the gate was being decided by the unpinned half of the report. `DESIGN-CRITERIA-REGISTRY.md` records the whole investigation.
123
-
124
- **6. Gate, in** `trace` **Phase 2.** Requirements are mapped to tests with Given/When/Then, coverage is computed per priority, and the decision is deterministic:
125
-
126
- | Condition | Decision |
127
- | --------------------------------------------------------------- | ------------ |
128
- | P0 coverage below 100% | **FAIL** |
129
- | Overall coverage below 80% | **FAIL** |
130
- | P1 coverage below 80% | **FAIL** |
131
- | P0 at 100%, overall ≥ 80%, P1 ≥ 90% | **PASS** |
132
- | P0 at 100%, overall ≥ 80%, P1 between 80% and 89% | **CONCERNS** |
133
- | Stakeholder-approved waiver with the complete approval contract | **WAIVED** |
134
-
135
- Our epic finishes at P0 100%, P1 87%, overall 84%, so it gates at CONCERNS with the residual risk named rather than passing quietly. Two overlays can lower that result further and can never raise it: a requirement resting only on recorded live verification caps at CONCERNS, and a synthetic coverage oracle below high confidence does the same. WAIVED is never derived; a human sets it, and the artifact demands an approver, approval date, reason, expiry, monitoring plan, remediation owner, and fix target. The authoritative contract is in the [Traceability template](./skills/bmad-testarch-trace/trace-template.md#waiver-details).
136
-
137
- If the run is not gate-eligible at all, because evidence collection was waived, restricted, inaccessible, or deferred, TEA emits no decision rather than computing one on partial evidence.
138
-
139
- ## Architecture & Flow
140
-
141
- BMad is a small **agent + workflow engine**. There is no external orchestrator; everything runs inside the LLM context window through structured instructions. TEA adds two pieces that run outside it: a Node hook that intercepts writes in your project, and a CLI that drives the review workflow headlessly in CI.
142
-
143
- ### Building Blocks
144
-
145
- | File / Scope | What it does | When it loads |
146
- | --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- |
147
- | `skills/bmad-tea/SKILL.md` | Murat's activation sequence and critical actions; renders the `{agent.menu}` placeholder | First, activates the TEA agent |
148
- | `skills/bmad-tea/customize.toml` | Agent customization surface: `[[agent.menu]]` items (code to skill), persona fields, persistent facts, hooks | During agent activation |
149
- | `skills/<workflow>/SKILL.md` | Workflow entrypoint: resolves workflow customization, picks mode, routes to the first step | When a TEA workflow is invoked |
150
- | `skills/<workflow>/customize.toml` | Workflow customization surface: activation hooks, persistent facts, optional `on_complete` behavior | During workflow activation |
151
- | `skills/<workflow>/workflow.yaml` | Machine-readable metadata: run variables, tool hints, output paths | Tooling and workflow metadata lookups |
152
- | `instructions.md` | Workflow-specific summary and operator notes | On demand |
153
- | `steps-c/*.md` | **Create** steps: primary execution, 5 to 12 files per workflow, 75 across the module | One at a time (just-in-time) |
154
- | `steps-c/step-NNx-subagent-*.md` | **Worker** steps: one isolated dimension each, dispatched in parallel | When an orchestrator step delegates |
155
- | `steps-e/*.md` | **Edit** steps: always 2 files, assess target then apply edit | One at a time |
156
- | `steps-v/*.md` | **Validate** steps: always 1 file, evaluate against the checklist | On demand |
157
- | `checklist.md` | Validation criteria: what "done" looks like for this workflow | Read by steps-v |
158
- | `*-template.md` | Output skeleton with `{PLACEHOLDER}` vars, filled in by steps to produce the artifact | Read by steps-c when generating output |
159
- | `bmad-testarch-test-review/steps-c/criteria-registry.md` | The 35 scoreable rows, each with a fixed severity and gate class. Severity is read here | Read by every review worker before it scores anything |
160
- | `bmad-testarch-framework/resources/hooks/tea-enforce.cjs` | Project-level guardrail for mechanically decidable test-quality violations; framework Create scaffolds it | Before writes, after writes or shell commands, and when an agent turn stops |
161
- | `skills/bmod-tea/bmod.toml` | Module record: version, member skills, setup questions, install messages | Read by `bmad setup tea` and `bmad` |
162
- | `skills/bmod-tea/roster.toml` | Lists Murat as the module's agent | Read by roster-aware skills such as party mode |
163
- | `skills/bmod-tea/help/*.md` | What each TEA skill is for, when to recommend it, and what each setup answer does | Read by the `bmad` skill when routing |
164
- | `skills/bmod-tea/knowledge/tea-index.csv` | Knowledge fragment index: id, name, description, tags, tier, path. 59 rows | Read before recommendations and by knowledge-loading steps |
165
- | `skills/bmod-tea/knowledge/*.md` | 59 reusable fragments: standards, patterns, API references, integration mandates | Selectively read into context by tier and config flags |
166
-
167
- The knowledge base has one copy, in the `bmod-tea` skill. The agent and every workflow that consults it read it as a sibling skill folder, `{skill-root}/../bmod-tea/knowledge`. If `bmod-tea` is missing, the skill says so and offers to install it. `bmad-teach-me-testing` carries a curated pointer file at `data/tea-resources-index.yaml` instead of reading the fragments.
168
-
169
- ```mermaid
170
- flowchart TB
171
- U[User] --> A[Agent activation<br/>persona + config + menu]
172
- A --> W[Workflow entry: SKILL.md<br/>mode: Create / Resume / Validate / Edit]
173
- W --> S[Step files<br/>steps-c / steps-e / steps-v]
174
- S --> K[Knowledge fragments<br/>tea-index.csv to knowledge/*.md]
175
- S --> T[Templates & checklists]
176
- S --> P[Orchestrator step]
177
- P --> X[Isolated workers<br/>one dimension each]
178
- X --> G[Aggregation step<br/>scored against criteria-registry.md]
179
- S --> O[Outputs: plans, tests, reports]
180
- G --> O
181
- O --> V[Validation: steps-v + checklist.md]
182
- O --> C[Checkpoint frontmatter<br/>resume where it stopped]
183
- ```
184
-
185
- ### How It Works at Runtime
186
-
187
- **1. Activation.** `/bmad-tea` or `$bmad-tea` loads the agent skill. It resolves its customization block across base, team, and user layers, adopts the persona, loads persistent facts and the `core` and `modules.tea` config from `_bmad/config.toml` and its `_bmad/custom/` layers, greets you, and renders `{agent.menu}` as a numbered table. Naming one clear intent in your first message ("let's design tests for this epic") skips the menu and dispatches directly. A message that supports several menu choices triggers one focused question naming the relevant choices before any workflow starts.
188
-
189
- **2. Workflow entry.** Direct workflow commands use the installed skill name, such as `/bmad-testarch-automate` or `$bmad-testarch-automate`, depending on the host's invocation syntax. `TA` is the equivalent agent-menu code, available only once TEA is active. Either way, the workflow's `SKILL.md` resolves its own `[workflow]` customization block and asks which mode to run: Create, Resume, Validate, or Edit. Create and Resume both route into `steps-c/`; Validate into `steps-v/`; Edit into `steps-e/`. `test-review` alone supports `headless: true`, which skips the greeting and the menu and runs Create directly. That is how the CLI drives it in CI.
190
-
191
- **3. Steps.** Each step file declares its own wiring in YAML frontmatter: `outputFile`, `nextStepFile`, and where relevant `knowledgeIndex` and `resumeStepFile`. A step loads on its own, pulls only the fragments its mode and config flags call for, fills any `*-template.md` placeholders, writes its output, and names the next step. Nothing loads the whole workflow at once, and the step files say so in as many words: "Do not load the next step until this step is complete."
192
-
193
- **4. Progress and resume.** Every create step appends itself to a checkpoint file's YAML frontmatter (`stepsCompleted`, `lastStep`, `lastSaved`), so an interrupted run resumes at the next incomplete step. Every workflow that runs once per scope also records run identity: `runScope` and `runKey` are resolved before anything is saved, the file name carries the `run_key` (`test-design/test-design-progress-epic-3.md`, `trace/traceability-matrix-epic-16.md`), and Resume refuses to continue a file whose `runKey` belongs to a different run. A trace run for epic 16 never opens epic 15's matrix or gate decision, and interrupting a system-level run to start an epic-level one leaves the first intact. `framework` and `ci` scaffold once per project, so each keeps a single fixed checkpoint in its own folder.
194
-
195
- **5. Validation.** `steps-v/` scores the finished output against `checklist.md`.
196
-
197
- See [Step-File Architecture](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/step-file-architecture/) for the loading model, worker isolation, and the per-workflow step patterns.
198
-
199
- ### Parallel Workers and Execution Modes
200
-
201
- Five workflows split their heaviest step across isolated workers. An orchestrator step does no work of its own; it resolves the mode, dispatches, and hands off to an aggregation step.
202
-
203
- | Workflow | Workers |
204
- | ------------- | -------------------------------------------------------------- |
205
- | `test-review` | determinism, isolation, maintainability, performance |
206
- | `nfr` | security, performance, reliability, maintainability |
207
- | `automate` | API, E2E, backend, mobile. Stack-gated, so 1 to 3 of the 4 run |
208
- | `atdd` | failing API tests, failing E2E tests |
209
- | `test-design` | system-level mode may generate its two documents in parallel |
210
-
211
- `tea_execution_mode` decides how they run: `auto`, `agent-team`, `subagent`, or `sequential`. With `tea_capability_probe` at its default of `true`, `auto` probes the runtime and prefers agent-team, then subagent, then sequential, which keeps behavior portable across supported agent runtimes. With probing off, TEA honors the configured mode strictly and fails with an explicit error rather than falling back silently. Mode changes orchestration only. The output schema, the validation rules, and the aggregation contract are identical in every mode.
212
-
213
- Workers exchange nothing directly. Each writes a JSON file under `/tmp` keyed by a shared run timestamp, and the aggregation step asserts every expected file exists before it scores anything. Isolation is what makes a parallel review honest. The shared criteria registry is what stops two isolated workers from disagreeing about what a finding is worth: every worker loads it, and none of them choose a severity.
214
-
215
- ### How Knowledge Gets Selected
216
-
217
- `tea-index.csv` classifies all 59 fragments into three tiers: **core** (24, always loaded), **extended** (19, loaded when deeper analysis is called for), and **specialized** (16, loaded only when the case matches, such as contract testing on a real consumer-provider boundary). Steps name the fragments they need, and they name their exclusions just as explicitly. A Maestro run is told not to load the browser fragments, because a device flow has no DOM and no request interceptor, and loading them invites browser patterns into a device flow.
218
-
219
- Over-loading is treated as a real defect, not a harmless cost. `npm run eval:fragment-selection` measures both directions: recall of the fragments a step requires, and the rate at which a run pulls one the step excludes by name.
220
-
221
- ### The Three Control Points
222
-
223
- A test rule can be enforced at three moments, and most tools occupy one of them. TEA occupies all three. The release gate is the fourth row below because it consumes what the other three produce, rather than being a fourth place to enforce a rule.
224
-
225
- | Point | When | Mechanism | What it closes |
226
- | -------------- | ---------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------- |
227
- | **Generation** | before the test exists | knowledge fragments and integration mandates | the model improvising a pattern TEA already has a standard for |
228
- | **Write** | as the file lands | `tea-enforce.cjs` on PreToolUse, PostToolUse, and Stop | a `.only`, a hard wait, or a tautological assertion getting committed at all |
229
- | **Review** | after the fact | `test-review` scored against the criteria registry | severity drifting with whichever model happened to run the review |
230
- | **Gate** | at release | `trace` Phase 2, PASS / CONCERNS / FAIL / WAIVED | shipping on evidence nobody checked was sufficient |
231
-
232
- The hook has separate installation and runtime lifecycles. The `bmad-testarch-framework` Create path installs it during its documentation and scripts step. Resume reaches the same step when installation is still incomplete. Framework Validate and Edit do not install it, and no other TEA workflow calls it.
233
-
234
- Once installed, the hook is project-scoped rather than workflow-scoped. It runs on matching tool events across TEA workflows, other agents, and ordinary coding sessions without requiring Murat or a TEA workflow to be active.
235
-
236
- The three passes cover different user-visible moments. `--pre` checks content before a direct file write reaches disk and rejects a blocking violation with a fix. `--post` re-reads the affected file after direct file writes or shell commands, then reports violations that only become visible in the complete file. `--stop` scans recently modified test files when the agent turn finishes, including outputs a code generator did not name in its command. All three passes are limited to the test and Pact configuration globs written for the detected stack in `.tea/enforce-config.json`.
237
-
238
- This closes the gap between advisory generation guidance and a later `test-review`. It blocks seven mechanically decidable Absolute rules and warns on one: focused tests, tautological assertions, hard waits, oversized test files, Maestro flows that cannot fail, two Pact parallelism rules, and undocumented disabled tests as the warning. Rules that require semantic judgment stay in `test-review`. The hook fails open on its own errors so a broken guardrail cannot lock the agent out of writing. Agent platforms without a write-time hook API skip installation and rely on `test-review` for enforcement.
239
-
240
- **How workflows become commands.** `npx skills add` copies each TEA skill into the host's skill directory under its own name. Invoking that name loads the skill, and the step-file process takes over. The skill name is the same on every host.
11
+ - Plan coverage by risk and priority, from P0 to P3.
12
+ - Set up a test framework and CI, write acceptance tests, and expand automation.
13
+ - Review test quality against a fixed criteria registry.
14
+ - Evaluate skills, agents, and AI features with scored probes and regression checks.
15
+ - Trace requirements to evidence and make PASS, CONCERNS, or FAIL release decisions, with waivers recorded separately.
241
16
 
242
17
  ## Install
243
18
 
19
+ Install the skills from GitHub:
20
+
244
21
  ```bash
245
22
  npx skills add bmad-code-org/bmad-method-test-architecture-enterprise
246
23
  ```
247
24
 
248
- This adds the TEA skills and `bmod-tea`, the module record that holds the setup questions and the knowledge base. Then, in your assistant chat, run:
249
-
250
- ```text
251
- bmad setup tea
252
- ```
253
-
254
- `bmad setup` runs through [uv](https://docs.astral.sh/uv/) (Python 3.11 or later, which uv fetches when it is missing), so install uv first. The `bmad` skill comes from BMad Method core. If you do not have it, add it with `npx skills add bmad-code-org/BMAD-METHOD --skill bmad bmod-core-tools`. Setup asks TEA's questions and writes the answers to `_bmad/config.toml`. Run `bmad setup tea` again later to see or change them.
255
-
256
- `npx skills add` installs the head of `main`. To install a release, point it at the release tag instead: `npx skills add https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/tree/v<version>/skills`.
257
-
258
- Murat is listed in the module's roster, so party mode can bring him into a group discussion alongside other BMad agents.
259
-
260
- ### Invocation Syntax
261
-
262
- | Host convention | Example |
263
- | --------------------- | ---------------------------------------- |
264
- | Slash command | `/bmad-testarch-automate` |
265
- | Dollar-prefixed skill | `$bmad-tea` or `$bmad-testarch-automate` |
266
-
267
- ## Quickstart
268
-
269
- 1. Install TEA (above)
270
- 2. Load the TEA menu with `/bmad-tea` or `$bmad-tea` if you want a conversational entrypoint.
271
- 3. Run one of the core workflows:
272
-
273
- - `TD` / `/bmad-testarch-test-design` / `$bmad-testarch-test-design` — test design, risk assessment, and NFR planning
274
- - `AT` / `/bmad-testarch-atdd` / `$bmad-testarch-atdd` — failing acceptance tests first (TDD red phase)
275
- - `TA` / `/bmad-testarch-automate` / `$bmad-testarch-automate` — expand automation coverage
276
-
277
- 1. Or use in party mode: `/party` to include TEA with other agents
278
-
279
- ## Engagement Models
280
-
281
- - **No TEA**: Use your existing testing approach
282
- - **TEA Solo**: Standalone use on non-BMad projects
283
- - **TEA Lite**: Start with `automate` only for fast onboarding
284
- - **Integrated (BMad Method / Enterprise)**: Use TEA in Phases 3–4 and release gates
285
-
286
- ## Workflows
287
-
288
- | Trigger | Slash Command | Dollar Skill | Purpose |
289
- | ------- | ---------------------------- | ---------------------------- | --------------------------------------------------------------------------- |
290
- | TMT | `/bmad-teach-me-testing` | `$bmad-teach-me-testing` | Teach Me Testing (TEA Academy) |
291
- | TD | `/bmad-testarch-test-design` | `$bmad-testarch-test-design` | System-level or epic-level test design and NFR planning |
292
- | TF | `/bmad-testarch-framework` | `$bmad-testarch-framework` | Scaffold test framework (frontend, backend, fullstack, or mobile) |
293
- | CI | `/bmad-testarch-ci` | `$bmad-testarch-ci` | Set up CI/CD quality pipeline (multi-platform) |
294
- | AT | `/bmad-testarch-atdd` | `$bmad-testarch-atdd` | Generate failing acceptance tests + checklist |
295
- | TA | `/bmad-testarch-automate` | `$bmad-testarch-automate` | Expand test automation coverage |
296
- | RV | `/bmad-testarch-test-review` | `$bmad-testarch-test-review` | Review test quality and score |
297
- | NR | `/bmad-testarch-nfr` | `$bmad-testarch-nfr` | Audit implemented NFR evidence |
298
- | TR | `/bmad-testarch-trace` | `$bmad-testarch-trace` | Trace requirements to tests + gate decision |
299
- | GATE | agent menu only | agent menu only | Route the release gate: test review, NFR evidence audit, then trace Phase 2 |
300
-
301
- `GATE` is a routing prompt on the agent menu, so it has no standalone command. Load the agent with `/bmad-tea` or `$bmad-tea` and pick it there.
302
-
303
- ## The Release Gate
304
-
305
- `trace` Phase 2 produces the decision: PASS, CONCERNS, FAIL, or WAIVED. Two mechanics sit under that vocabulary and are easy to miss.
306
-
307
- **Live evidence is capped.** A requirement covered only by recorded live verification forces PASS down to CONCERNS, with a rationale naming the recorded source SHA. The overlay only ever lowers a PASS or annotates an existing CONCERNS. It can never lift a FAIL. Only a `pass` recorded against the commit under trace counts; `stale`, `unverifiable`, `contradicted`, `blocked`, and the rest are reported as blockers. The JSON contract is published at [Live Verification Results](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/live-verification-results/), so any runner can produce it. Trace reads that file and never runs anything itself.
308
-
309
- **Some runs are not gate-eligible at all.** A collection status of `waived`, `restricted`, `inaccessible`, or `deferred_shared` means no decision is emitted rather than a decision computed on partial evidence. A missing manifest resolves to `INACCESSIBLE`, not to 0% coverage.
310
-
311
- ### `tea-test-review` in CI
312
-
313
- Installing this package also installs a `tea-test-review` binary that runs the review workflow headlessly against a pull request diff.
25
+ Setup needs [uv](https://docs.astral.sh/uv/) and the `bmad` and `bmod-core-tools` skills from BMad Method core.
26
+ If those skills are missing, install them:
314
27
 
315
28
  ```bash
316
- npx tea-test-review --base origin/main --min-score 80
29
+ npx skills add bmad-code-org/BMAD-METHOD --skill bmad bmod-core-tools
317
30
  ```
318
31
 
319
- It reviews with the skill shipped in its own package, scopes to changed tests (`--base`, `--files`, or `--pr` to look up the base branch), runs through an agent adapter with a pinned review model, isolates the filesystem, emits a JSON verdict, and separates its exit codes: `0` pass, `1` verdict failure, `2` environment or configuration failure, and `3` agent failure or an unparseable or untrusted report. For example, a missing credential exits `2`, while a runner crash after launch exits `3`.
320
-
321
- The recommendation is derived from the findings rather than taken from the agent's prose. Any CRITICAL derives Block. Any HIGH, or a score under 70, derives Request Changes. The agent's own stated recommendation is preserved as `reportedRecommendation` when the two disagree. `--waive` exists for the exceptions and requires an expiry.
322
-
323
- `tea-test-review --github` also opens a check run and keeps one short pull-request comment up to date, and `tea-test-review render` prints the same comment, check-run text or summary from a verdict file for any other CI. Copy-paste workflows live at `cli/examples/pr-test-review.yml` (GitHub Actions) and `cli/examples/gitlab-ci.yaml` (GitLab), and the full flag, exit-code, and security reference is at [tea-test-review CLI](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/tea-test-review-cli/).
324
-
325
- ## Configuration
326
-
327
- TEA's setup questions are declared in `skills/bmod-tea/bmod.toml` and asked by `bmad setup tea`, which writes the answers to `modules.tea` in `_bmad/config.toml`. Every one is read by a workflow today. Team overrides go in `_bmad/custom/config.toml` and personal ones in `_bmad/custom/config.user.toml`; run `bmad setup tea` again to see or change the answers.
328
-
329
- - `test_artifacts` — base output folder for test artifacts. Each workflow writes into its own folder under it (`test-design/`, `atdd/`, `automate/`, `test-review/`, `nfr/`, `trace/`, `ci/`, `framework/`), and a file produced once per story, epic, or release carries that scope in its name, such as `trace/gate-decision-epic-16.json`. See [Output Layout](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/configuration/#output-layout)
330
- - `tea_use_playwright_utils` — enable Playwright Utils integration (boolean, default true). When true **and the package is installed**, `@seontechnologies/playwright-utils` becomes the default implementation for everything it covers: generated Playwright tests use `interceptNetworkCall`, `apiRequest`, `recurse`, and `log` without being asked, and `test-review` flags a vanilla equivalent that carries no stated reason. See [Integrate Playwright Utils](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/customization/integrate-playwright-utils/)
331
- - `tea_use_pactjs_utils` — enable Pact.js Utils integration for contract testing (boolean, default true). It decides how Pact suites are written, not whether a project gets one: TEA still requires a real consumer-provider boundary before scaffolding a contract test. When on **and the package is installed**, generated Pact code uses `createProviderState`, `buildVerifierOptions`, and `createRequestFilter` rather than raw Pact boilerplate. A flag with no install generates the raw path and reports one recommendation rather than flagging every file
332
- - `tea_pact_mcp` — SmartBear MCP for PactFlow/Broker interaction: mcp, none (string, default mcp). Safe without a broker: every broker-dependent step degrades to provider source or an OpenAPI spec and reports that the broker was unreachable
333
- - `tea_browser_automation` — browser automation mode: auto, cli, mcp, none (string, default auto)
334
- - `tea_execution_mode` — how TEA orchestrates multi-step generation and evaluation: auto, subagent, agent-team, sequential (string, default auto)
335
- - `tea_capability_probe` — probe the runtime before selecting an execution mode (boolean, default true). With it off, TEA honors the configured mode strictly and fails loudly instead of falling back
336
- - `test_stack_type` — detected or configured stack type (auto, frontend, backend, fullstack, mobile). Mobile is checked before frontend, because a React Native project carries React in `package.json` and would otherwise misdetect as web
337
- - `test_framework` — detected or configured test framework (auto, Playwright, Cypress, Jest, Vitest, pytest, JUnit, Go test, dotnet test, RSpec, Maestro, other)
338
-
339
- Two settings that only one workflow reads are set through that workflow's customization file instead, under `[workflow]`:
340
-
341
- - `evaluations_folder` — base folder for Evaluate's own evaluation folders (default `evals`, relative to the project root). Set it in `_bmad/custom/bmad-testarch-evaluate.toml`
342
- - `ci_platform` — CI platform (auto, github-actions, gitlab-ci, jenkins, azure-devops, harness, circle-ci, other; default auto). Set it in `_bmad/custom/bmad-testarch-ci.toml`
343
-
344
- Full option reference: [Configuration](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/configuration/).
345
-
346
- ## Knowledge Base
347
-
348
- TEA relies on a curated testing knowledge base of 59 fragments, indexed by tier:
349
-
350
- - Index: `skills/bmod-tea/knowledge/tea-index.csv`
351
- - Fragments: `skills/bmod-tea/knowledge/`
352
- - Tiers: 24 core, 19 extended, 16 specialized
353
-
354
- Workflows load only the fragments required for the current task, and the required set is named in the step file rather than inferred from index tags. See [Knowledge Base](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/knowledge-base/).
355
-
356
- ## Repository Layout
32
+ Then type this in your assistant chat:
357
33
 
358
34
  ```text
359
- skills/ # the shipped module, one folder per skill
360
- ├── bmod-tea/ # module record: bmod.toml, roster.toml, help/, knowledge/{tea-index.csv, *.md}
361
- ├── bmad-tea/ # the agent: SKILL.md, customize.toml
362
- ├── bmad-teach-me-testing/
363
- ├── bmad-testarch-atdd/
364
- ├── bmad-testarch-automate/
365
- ├── bmad-testarch-ci/
366
- ├── bmad-testarch-evaluate/
367
- ├── bmad-testarch-framework/ # resources/hooks/tea-enforce.cjs lives here
368
- ├── bmad-testarch-nfr/
369
- ├── bmad-testarch-test-design/
370
- ├── bmad-testarch-test-review/ # steps-c/criteria-registry.md lives here
371
- └── bmad-testarch-trace/
372
-
373
- cli/ # tea-test-review: the headless CI gate
374
- docs/ # source of truth for the docs site
375
- website/ # Astro + Starlight, consumes docs/ through a symlink
376
- tools/ # validators, doc build, changelog stamping
377
- test/ # quality gate suites and the six eval harnesses
378
- ```
379
-
380
- ## How TEA Keeps Itself Honest
381
-
382
- TEA has deterministic checks and live evals. All eleven of TEA's skills have coverage registered in the suite manifest: ten behavioral suites, and the suite Evaluate authored for `bmad-testarch-evaluate` itself.
383
-
384
- ### Current Eval Coverage
385
-
386
- The eight suites under `test/evals/` measure one decision inside each knowledge-bearing workflow: whether the agent selects the required knowledge fragments and avoids fragments the workflow excludes. They do not execute the complete workflow or grade its final artifact.
387
-
388
- The `atdd`, `automate`, `ci`, `framework`, `nfr`, `teach-me-testing`, `test-design`, `test-review`, and `trace` suites add behavioral evals, and so does `bmad-tea`'s own routing suite. `atdd` generates red-phase scaffolds against an unimplemented fixture, activates them, executes them under NFR9's isolation, and scores whether each one fails for the reason its criterion states. `automate` runs four hand-authored spec sets, standing in for generated coverage, against a fixed voucher-redemption service and against a scratch copy carrying its seeded minimum-spend regression, and scores whether the one spec set that tests the boundary catches the regression and whether the other three are correctly reported as missing it, vacuous, or duplicated rather than as clean passes; it spends no vendor call, since there is no live-agent mode to build one for. `ci` scaffolds a pipeline for a project whose team wrote down what it needs, then parses the generated workflow, lints it with `actionlint`, and scores every requested trigger, permission, command, gate, and artifact against a minimal-request control. `framework` installs and smoke-tests a scaffolded framework in a network-isolated sandbox, proving it runs rather than merely reading as valid. `nfr` audits one evidence bundle with known gaps and one clean bundle. `teach-me-testing` tests multi-turn teaching sessions, seeded quiz corrections, and persistent learner progress through `tea-transcript-runner`. `test-design` analyzes one seeded epic and one clean control epic. `test-review` runs the complete review against files containing nine planted defects plus one clean file, then scores recall, the non-false-positive rate, score variance, and verdict stability. `trace` runs the complete traceability workflow against a ten-criterion seeded set whose declared gate is FAIL and a five-criterion clean set whose declared gate is PASS, then scores criterion status, the gate decision and the criteria behind it, the coverage arithmetic, evidence citation, and case stability. `bmad-tea` routes nineteen intents to the right menu item, or declines the ones nothing on the menu serves.
389
-
390
- | Skill | Fragment-selection cases | Full behavioral eval |
391
- | --------------------------- | ------------------------ | ----------------------------------------------------------------------------------- |
392
- | `bmad-tea` | N/A | Yes; nineteen intents, eleven with a right answer and eight controls |
393
- | `bmad-teach-me-testing` | N/A | Yes; multi-turn session with seeded wrong quiz answer, review loop, and progress |
394
- | `bmad-testarch-atdd` | 3 | Yes; one unimplemented fixture and five acceptance criteria |
395
- | `bmad-testarch-automate` | 5 | Yes; four hand-authored spec sets against a fixed service and its seeded regression |
396
- | `bmad-testarch-ci` | 2 | Yes; one full request, one minimal request and one request over an evaluation plan |
397
- | `bmad-testarch-evaluate` | N/A | Yes; Evaluate-authored, five seeded probes and four clean controls |
398
- | `bmad-testarch-framework` | 3 | Yes; install and smoke-test generated scaffold in network-isolated sandbox |
399
- | `bmad-testarch-nfr` | 2 | Yes; one evidence bundle with known gaps and one clean bundle |
400
- | `bmad-testarch-test-design` | 5 | Yes; one seeded epic and one clean control epic |
401
- | `bmad-testarch-test-review` | 2 | Yes; three files, nine planted defects, and one clean file |
402
- | `bmad-testarch-trace` | 2 | Yes; a ten-criterion seeded set and a five-criterion clean set |
403
-
404
- A passing fragment-selection eval means the workflow loaded the right knowledge. All eleven of TEA's skills have coverage registered in the suite manifest; `bmad-testarch-evaluate` is covered by the `evaluate-authored` suite Evaluate wrote and ran on itself, so the manifest's deferred array is empty. [How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) explains the testing hierarchy, exit classes, and evidence model in plain language. The source-controlled [Eval Quality and Behavioral Coverage Roadmap](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-roadmap/) records the per-skill contracts, runner work, CI plan, and intended boundary with the upcoming standalone `eval-quality` project. [The eval-quality Command Adapter](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-command-adapter/) records how TEA runs the commands those evals measure, and [Adopting eval-quality, One Skill at a Time](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-adoption-guide/) is the guide for taking one skill in another BMAD module from asserted quality to measured quality.
405
-
406
- ### Deterministic Checks
407
-
408
- `npm test` chains 164 checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Twelve of the 164 keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
409
-
410
- - `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 35 rows are currently mapped across 51 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
411
- - `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, the fragment-selection case count, the replay totals that the test README and the header of the replay suite state, the story count, epic count and appended-story count in the overview of the Evaluate plan, and the lane count in its parallel-lanes section.
412
- The plan's source module also refuses a plan whose five lane lists differ between the plan's epics file and its sprint status file, so the gate fails on a story moved, dropped, duplicated or reordered in one file only.
413
- A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
414
- - `test:doc-claims` runs `eval-quality-gates doc-claims`, which holds a hand-written prose claim — a named symbol, a transcribed list, a worked JSON example, a named exit code, or a claim whose truth depends on when it was written — against the artifact it describes. A registered dated claim whose sentence was rewritten out from under it fails the same way a wrong symbol name does, so the entry cannot go stale by drifting out from under its own key either.
415
- - `test:enforce-hook` fails when a new Absolute registry row appears in neither the hook's enforced list nor its deferred list. This prevents a rule from being added without an explicit write-time enforcement decision.
416
- - `test:atdd-workflow-guidance` holds ATDD generation to one supplied criterion id per executable leaf, one primary scaffold per criterion, a criterion-defining first assertion, and a transition-bearing primary branch for state changes. It also checks every literal API and E2E example title against the same criterion-id format.
417
- - `test:eval-data` checks that all 24 fragment-selection cases are structurally usable: their workflow context files exist, every expected fragment exists and is indexed for that workflow, and the required and forbidden sets do not overlap. The expected sets come from the workflow step files. This check does not ask an agent to select anything.
418
- - `test:eval-routing-boundaries` mutates the facts, candidate codes, and missing deciding information in each source-bound routing ambiguity rule. It also rejects duplicate and orphan boundaries, plus any boundary that would turn a clear intent into a clarification.
419
- - `test:eval-schemas` checks this repository's own eval suite manifest against its schema, confirms that every threshold it declares is the threshold the harness actually applies, that the runner capabilities it declares are the ones the harness grants its runner, and that the preflight argv it declares fails on a missing runner and passes on a present one, and fails when a TEA skill has neither a behavioral or Evaluate-authored suite nor a deferred declaration. A suite list that omits a skill reads as coverage, so the omission has to be an error rather than a silence.
420
- - `test:eval-diagnostics` checks that current result writers emit bounded case and repetition diagnostics, that quality and environment outcomes stay separate, that unstable signatures remain visible, and that the protected 1.3.0 baseline remains readable and byte-for-byte unchanged.
421
- - `test:nfr-evidence-guidance` holds every NFR worker and the publication audit to the supplied-evidence ledger, exercises supported, missing, and mismatched evidence, rejects contaminating examples, and binds the protected baseline and focused remediation records by digest.
422
- - `test:contract-oracles` evaluates every oracle in every eval contract with `eval-quality`'s own evaluator, over the stored replay outputs and over constructed selections for all 24 fragment-selection cases, and fails when an oracle faults or disagrees with the harness scorer on the same evidence. The first run found that every regex oracle in the `test-review` contract was refused by the evaluator before it matched anything.
423
- - `test:evaluate-guidance` parses the Evaluate skill's `SKILL.md` stage list and fails when a stage entry is missing, out of order, or points at a `references/` guide that does not exist.
424
-
425
- Four more checks hold the Evaluate documentation:
426
-
427
- - `test:docs-build-names` scans every page under `docs/` and fails on a name from how TEA was built: a story id, a decision id, a line of epic narrative, a planning path, a pull request number or a word from the review process.
428
- It also fails when an executable command block (a fenced `bash`, `sh`, `shell` or `zsh` block) holds a timestamped run or score ID, which a reader's own run never has.
429
- - `test:docs-evaluate-sidebar` fails when a page that names `tea-evaluate` or `bmad-testarch-evaluate` is missing from the site sidebar, and when the TEA overview links no page that Evaluate owns.
430
- - `test:docs-gap-reader` lifts the held-out reader command out of the gaps how-to, runs it over a `gap-view.json` built with a caught, an uncaught and a withheld row, and fails when it crashes or leaves a row unprinted.
431
- - `test:docs-tutorial` runs the commands of the Evaluate tutorial against its fixture and fails on a command that errors or an output line the page promises and the run does not print.
432
-
433
- `test:workflows-lint` runs `actionlint` over every workflow in `.github/workflows/` and the `cli/examples/pr-test-review.yml` template, and fails on any finding. It disables the shellcheck and pyflakes integrations so the answer does not depend on which of those binaries a machine has. With no `actionlint` on `PATH` it fails and names `tools/install-actionlint.sh` rather than skipping, and it proves on every run that a workflow reading a step that does not exist is refused.
434
-
435
- These checks produce the same answer from the same repository state. They need no agent credential, network call, or model budget. `test:eval-data` runs through `npm test`, the local pre-commit hook, pull-request quality checks, and the publish workflow.
436
-
437
- `test:doc-invocations` runs `eval-quality-gates doc-invocations`, which re-runs every `npm run <script>` this README, the adoption guide, and `test/README.md` document in their CI-usage and individual-suites blocks against the real repository, so a renamed script or a broken example fails the build instead of a reader. Each of the eighteen documented scripts is declared as a full literal spelling, never a shared `npm run` prefix, which is what keeps the gate from ever matching, let alone running, `npm run eval:all` or `npm run release:next` on the same pages. The gate's own entry script is the one real program it is allowed to reach; it refuses anything outside its own hardcoded allowlist before invoking real `npm`, and `test:doc-invocation-entry` holds that allowlist's comment-to-script lookup against all three pages' literal text, line by line, so a swapped or drifted comment fails there even when the script it resolves to would still run.
438
-
439
- ### Start Here
440
-
441
- You do not start an interactive agent session. A live eval launches the selected agent CLI as a headless subprocess, sends it each prompt, waits for the result, and scores the result.
442
-
443
- The normal path is one command. It runs fragment selection across all eight covered workflow skills, then runs the behavioral `bmad-tea-routing`, `ci`, `nfr`, `test-design`, `test-review`, `trace`, `atdd` and `automate` evals:
444
-
445
- ```bash
446
- npm run eval:all -- --agent codex
447
- ```
448
-
449
- Use `claude` or `agy` instead, or run all three built-in adapters:
450
-
451
- ```bash
452
- npm run eval:all -- --agent claude
453
- npm run eval:all -- --agent agy
454
- npm run eval:all -- --agent agy --agent claude --agent codex
455
- ```
456
-
457
- `eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `ci` project, per `test-design` epic, per `trace` fixture set, and per `atdd` story. One runner makes 115 agent calls: 48 fragment selections, 38 routing intents, 3 reviews, 4 audits, 12 pipelines, 4 test designs, 4 traces, and 2 ATDD generations. All three built-in runners make 345 calls.
458
-
459
- Check the data, executable, login, fixtures, and expected results without making a model call:
460
-
461
- ```bash
462
- npm run eval:all -- --agent codex --preflight-only
463
- npm run eval:all -- --agent claude --preflight-only
464
- npm run eval:all -- --agent agy --preflight-only
35
+ bmad setup tea
465
36
  ```
466
37
 
467
- Output ending with `nothing measured` is expected in preflight mode. It means the static eval data is valid and, for ten of the twelve suites, the selected executable is on `PATH`, answers `--version`, and a built-in vendor has a credential. `automate` and `framework` invoke no agent at all, so their preflight only checks that their own corpus and dependencies resolve, never a vendor or a credential. Some runners cannot expose session authentication to this probe, so a preflight pass does not guarantee that the later live call will authenticate. The flag intentionally exits before launching the agent. `npm run test:eval-schemas` runs each suite's preflight with a missing runner and with a present one, so a preflight that stopped probing the runner would fail `npm test`.
468
-
469
- ### A La Carte Live Evals
38
+ Setup asks about your test stack, integrations, and output folder, then writes the answers to `_bmad/config.toml`.
39
+ Run it again to change those answers.
40
+ The [configuration reference](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/configuration/) covers team and personal overrides.
470
41
 
471
- Use the focused commands when debugging one metric or skill. A one-call review smoke test is:
42
+ The GitHub install follows `main`.
43
+ To install a tagged release, replace `<version>` with the release number:
472
44
 
473
45
  ```bash
474
- # One review. Recall and the non-false-positive rate are measured; variance and stability are not.
475
- npm run eval:test-review -- --agent codex --runs 1
476
-
477
- # Complete eval with one runner.
478
- npm run eval:test-review -- --agent codex
479
- npm run eval:test-review -- --agent claude
480
-
481
- # Complete eval with all three built-in runners. This makes nine review calls.
482
- npm run eval:test-review -- --agent agy --agent claude --agent codex
46
+ npx skills add https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/tree/v<version>/skills
483
47
  ```
484
48
 
485
- The `trace` suite runs the same way, one call per fixture set per repetition:
49
+ ## Start with one workflow
486
50
 
487
- ```bash
488
- # One trace per fixture set. Accuracy is measured; stability is not.
489
- npm run eval:trace -- --agent codex --runs 1
51
+ In Claude Code, Cursor, and Windsurf, type `/bmad-testarch-test-design` in chat.
52
+ In Codex, type `$bmad-testarch-test-design`.
53
+ You can also load `/bmad-tea` or `$bmad-tea` and choose a menu code.
54
+ Each workflow can run directly in a fresh session.
490
55
 
491
- # One fixture set only.
492
- npm run eval:trace -- --agent codex --set seeded-tenant-data-export
56
+ | Workflow | Command | Menu | Use it to |
57
+ | ---------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | ---- | ------------------------------------------------------ |
58
+ | [Teach Me Testing](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/teach-me-testing/) | `/bmad-teach-me-testing` | TMT | Learn testing through seven sessions |
59
+ | [Test Design](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-test-design/) | `/bmad-testarch-test-design` | TD | Plan risks, coverage, and NFR evidence |
60
+ | [Framework Setup](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/setup-test-framework/) | `/bmad-testarch-framework` | TF | Scaffold a test framework |
61
+ | [CI Setup](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/setup-ci/) | `/bmad-testarch-ci` | CI | Connect tests and quality checks to CI |
62
+ | [ATDD](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-atdd/) | `/bmad-testarch-atdd` | AT | Write acceptance scaffolds before implementation |
63
+ | [Automate](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-automate/) | `/bmad-testarch-automate` | TA | Add coverage to implemented features |
64
+ | [Test Review](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-test-review/) | `/bmad-testarch-test-review` | RV | Audit test quality and score findings |
65
+ | [NFR Evidence Audit](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-nfr-assess/) | `/bmad-testarch-nfr` | NR | Assess performance, security, and reliability evidence |
66
+ | [Evaluate](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/evaluate-your-first-skill/) | `/bmad-testarch-evaluate` | EV | Build and run a behavioral evaluation |
67
+ | [Trace](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-trace/) | `/bmad-testarch-trace` | TR | Map requirements to tests and decide a release gate |
493
68
 
494
- # Complete eval with one runner. Two fixture sets twice: 4 calls.
495
- npm run eval:trace -- --agent codex
496
- ```
69
+ For Codex, replace the leading `/` in the table with `$`.
70
+ The agent menu also accepts `GATE` to route you through test review, NFR evidence audit, and trace Phase 2.
497
71
 
498
- ### Run Fragment Selection by Skill
72
+ Choose a starting path:
499
73
 
500
- Each command below runs one repetition. Use `--runs 2` for the complete stability measurement.
74
+ - [TEA Lite](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/tea-lite-quickstart/) for an existing project that needs more test coverage.
75
+ - [TEA Academy](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/learn-testing-tea-academy/) to learn testing.
76
+ - [TEA Overview](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/tea-overview/) for the lifecycle and workflow order.
77
+ - [Evaluate Your First Skill](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/evaluate-your-first-skill/) for a runnable evaluation tutorial.
501
78
 
502
- | Skill | Copy-paste command |
503
- | --------------------- | ------------------------------------------------------------------------------------------------ |
504
- | `atdd` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-atdd --runs 1` |
505
- | `automate` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-automate --runs 1` |
506
- | `ci` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-ci --runs 1` |
507
- | `framework` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-framework --runs 1` |
508
- | `nfr` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-nfr --runs 1` |
509
- | `test-design` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-test-design --runs 1` |
510
- | `test-review` routing | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-test-review --runs 1` |
511
- | `trace` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-trace --runs 1` |
79
+ ## How it works
512
80
 
513
- Run every suite with one or all built-in runners:
81
+ Each workflow loads one step file at a time.
82
+ The steps define the task, the evidence to collect, and when to stop.
83
+ They load the knowledge fragments needed for that task from `skills/bmod-tea/knowledge/tea-index.csv`.
84
+ Workflows can delegate independent tasks when the assistant supports workers; execution mode is configurable.
514
85
 
515
- ```bash
516
- # 24 cases run twice: 48 calls.
517
- npm run eval:fragment-selection -- --agent codex
518
- npm run eval:fragment-selection -- --agent claude
86
+ Review findings use fixed criteria and severity rules.
87
+ Trace applies coverage thresholds and evidence checks to the release decision.
88
+ Generated tests and reports still need review, and a release gate depends on the evidence collected.
89
+ See [step-file architecture](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/step-file-architecture/), [test quality standards](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/test-quality-standards/), and [risk-based testing](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/risk-based-testing/).
519
90
 
520
- # All three runners: 144 calls.
521
- npm run eval:fragment-selection -- --agent agy --agent claude --agent codex
522
- ```
91
+ Planning and traceability work across stacks.
92
+ Test generation and execution support vary by target.
93
+ The [execution-target matrix](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/execution-targets/) lists the supported stacks and their limits.
523
94
 
524
- ### Antigravity, Claude, Codex, and Custom Agent CLIs
95
+ ## Command-line tools
525
96
 
526
- The built-in adapters are `claude`, `codex`, and `agy` (Antigravity CLI). Run live evals with any built-in adapter:
97
+ TEA also publishes an npm package for CI and terminal use.
98
+ It requires Node.js 22.20.0 or later:
527
99
 
528
100
  ```bash
529
- npm run eval:all -- --agent agy
530
- npm run eval:all -- --agent claude
531
- npm run eval:all -- --agent codex
101
+ npm install --save-dev bmad-method-test-architecture-enterprise
532
102
  ```
533
103
 
534
- Any other headless CLI can use `--agent custom`. The runner must:
104
+ - [`tea-test-review`](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/tea-test-review-cli/) runs the review workflow against changed tests and returns a gate verdict.
105
+ - [`tea-evaluate`](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/tea-evaluate-cli/) checks, runs, scores, and compares behavioral evaluations.
535
106
 
536
- 1. Read the complete prompt from standard input.
537
- 2. Run non-interactively in the repository working directory.
538
- 3. Print its final response to standard output. The review eval must also allow the agent to write the report path named in the prompt.
539
- 4. Exit with a nonzero status when the agent call fails.
540
-
541
- [Gemini CLI headless mode](https://geminicli.com/docs/cli/headless/) accepts standard input alongside a `-p` prompt. Once Gemini is installed and authenticated, run every live eval with:
542
-
543
- ```bash
544
- npm run eval:all -- \
545
- --agent custom \
546
- --agent-cmd gemini \
547
- --agent-arg -p \
548
- --agent-arg "Follow the complete instructions from standard input." \
549
- --agent-arg --output-format \
550
- --agent-arg text \
551
- --agent-arg --approval-mode \
552
- --agent-arg yolo \
553
- --agent-arg --skip-trust \
554
- --env-pass GEMINI_API_KEY \
555
- --env-pass GOOGLE_API_KEY
556
- ```
557
-
558
- This uses `yolo` because the review eval must write its report. To run fragment selection alone with a read-only policy:
107
+ For `tea-evaluate`, install `eval-quality` alongside TEA in the evaluations folder:
559
108
 
560
109
  ```bash
561
- npm run eval:fragment-selection -- \
562
- --agent custom \
563
- --agent-cmd gemini \
564
- --agent-arg -p \
565
- --agent-arg "Follow the complete instructions from standard input." \
566
- --agent-arg --output-format \
567
- --agent-arg text \
568
- --agent-arg --approval-mode \
569
- --agent-arg plan \
570
- --agent-arg --skip-trust \
571
- --env-pass GEMINI_API_KEY \
572
- --env-pass GOOGLE_API_KEY
110
+ npm install --prefix evals bmad-method-test-architecture-enterprise eval-quality
111
+ npm exec --prefix evals -- tea-evaluate --help
573
112
  ```
574
113
 
575
- `--env-pass` is required only for credentials stored in environment variables. Stored CLI logins use the home directory that the harness already passes through. Model selection for a custom runner is also explicit, using repeated `--agent-arg` values for that CLI's model flag and value.
576
-
577
- ### What Has to Pass
578
-
579
- Every row below is a declared gate. Ten of TEA's eleven skills have behavioral suites registered in the suite manifest, and `bmad-testarch-evaluate` has the Evaluate-authored suite. Recorded live baselines preserve timestamped history, and [How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) describes the evidence model. The live thresholds are the ones this repository's own eval suite manifest declares, and `npm run test:eval-schemas` fails when a harness constant and the manifest disagree.
580
-
581
- | Eval | Declared threshold | Declared volume |
582
- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
583
- | `npm run eval:all -- --agent ...` | All eight live suites below meet their thresholds for the selected runner | 48 selections, 38 routing calls, 3 reviews, 4 audits, 6 pipelines, 4 test designs, 4 traces, 2 generations |
584
- | Fragment selection | At least 90% required-fragment recall, at most 10% forbidden-fragment selection, and stable choices across repeated cases | 24 cases twice: 48 calls |
585
- | Test review | At least 70% overall recall, 100% CRITICAL recall, an 80% non-false-positive rate, score standard deviation no higher than 3, and a stable verdict | Three complete reviews |
586
- | Trace | At least 90% criterion-status accuracy and 90% evidence-citation precision; 100% on both discriminating criteria, the gate decision, the gate criteria, the coverage arithmetic, the oracle resolution, the rejected evidence, the waiver validity, and the live evidence; no clean-set false positive, no unstable case, and no fixture mutation | Two fixture sets twice: 4 calls |
587
- | bmad-tea routing | At least 90% route accuracy, 80% reason-token recall, 87.5% scope fidelity, and 75% on both clarification and decline recall; zero confident routes on an intent nothing on the menu serves, zero on one whose menu items are genuinely close, zero clear intents left unrouted, and at most two intents whose answer changes between repetitions | 19 intents twice: 38 calls |
588
- | NFR | At least 87.5% domain-status accuracy and 90% threshold fidelity; 100% on undecidable domains, the overall status, and domain coverage; no unsupported PASS, no fabricated evidence, no clean-bundle false positive, no duplicate domain section, no disagreement between the gate artifact and the assessment it summarizes, no unstable case, and no fixture mutation | Two evidence bundles twice: 4 calls |
589
- | CI | Zero parse failures and zero `actionlint` findings; at least 97% requested-element recall and 100% trigger accuracy; zero unrequested elements, zero workflow rule violations, no unstable case, and no fixture mutation | Six projects twice: 12 calls |
590
- | Test design | At least 80% grounded-risk recall, 80% risk precision, and 80% coverage mapping; 100% on the scale compliance, the score arithmetic, the category validity, the band placement, the risk-id shape, the risk-link resolution, and the priority ordering; no unscored risk table, no ungrounded risk, no missed top-severity risk, no risk-ceiling excess, no unstable case, and no fixture mutation | Two epics twice: 4 calls |
591
- | ATDD | 100% red-for-intended-reason rate and criteria coverage; no vacuous pass, no scaffold still skipped after activation, no load error, no non-assertion exit, no unmapped test, no production mutation, and no unstable case | One story twice: 2 generation calls, plus execution under NFR9's isolation, which spends no call |
592
- | `npm run test:eval-data` | Every case references valid workflow files and indexed fragments; required and forbidden sets do not overlap | No agent calls |
593
- | `npm run test:eval-schemas` | The suite manifest matches its schema, declares the thresholds the harnesses apply, and accounts for every TEA skill | No agent calls |
594
-
595
- Every declared repetition has to complete. A run that loses one to a timeout, a transport error, or an unparseable reply cannot measure variance or stability, so it exits `2` rather than reporting a lower score.
596
-
597
- The non-false-positive rate is the share of reported findings that are not definite false positives, and only the clean fixture makes a false positive definite. A finding on a seeded fixture that matches no planted row is reported separately as `unattributed`: the fixture may carry an incidental real defect nobody planted, so counting it either way would be a guess.
598
-
599
- ### The Suite Manifest and Machine-Readable Results
600
-
601
- This repository's own eval suite manifest registers every suite `eval:all` runs, with its skills, fixtures, ground truth, contract, thresholds, repetition count, CI tier, and the capabilities its runner needs. It also carries a `deferred` list: one entry per skill with no behavioral suite, naming the owner, the missing evidence, and the condition that retires the entry. `eval:all` reads the suite list from that file and refuses to run when a TEA skill appears in neither list.
602
-
603
- Add `--json <path>` to any of the six harnesses to write a result record alongside the console output:
604
-
605
- ```bash
606
- npm run eval:all -- --agent codex --json results/eval-all.json
607
- npm run eval:test-review -- --agent codex --json results/test-review.json
608
- ```
609
-
610
- The record carries the repository commit, the suite and case IDs, the runner executable and version, the resolved model and parameters, the fixture and prompt digests, the expected and completed repetitions, the measurements, the duration, and the final failure class. A generated JSON Schema in this repository's own dev tree describes it, and every record is validated against that schema before it is written.
611
-
612
- ### CI Usage
613
-
614
- Run the eval's deterministic checks on every pull request. `npm test` runs all of these along with the rest of the chain:
615
-
616
- ```bash
617
- npm ci
618
- npm run test:eval-data # fragment-selection corpus, static
619
- npm run test:eval-trace-data # trace corpus, static
620
- npm run test:eval-schemas # manifest against harness constants, and the preflight argv
621
- npm run test:eval-replay # stored outputs against the scorers
622
- ```
623
-
624
- Run live evals in a scheduled or manually triggered CI job after installing and authenticating the selected agent CLI:
625
-
626
- ```bash
627
- npm ci
628
- npm run eval:all -- --agent codex
629
- ```
630
-
631
- The eval harnesses use CI-compatible exit codes: `0` means every threshold passed, `1` means a measured result missed a threshold, and `2` means the environment could not run the eval. Live jobs consume model quota and can vary as models change, so keep their result separate from the deterministic pull-request gate until the team chooses to make model quality a required check.
632
-
633
- TEA applies the same evidence rule to its documentation. Unproven explanations are labeled as hypotheses, and workarounds are labeled as countermeasures. `DESIGN-CRITERIA-REGISTRY.md` records the investigations behind the review rules and scoring decisions.
634
-
635
- ## Extending TEA
636
-
637
- Custom workflows are still compatible with TEA, but they are no longer implicitly absorbed into TEA core. The supported path is:
638
-
639
- 1. Package the workflow as custom content or a custom module.
640
- 2. Attach it to `bmad-tea` using the agent customization flow.
641
- 3. Install the workflow skill with `npx skills add`, then start a fresh chat so the new menu item is picked up.
642
-
643
- See [Extend TEA with Custom Workflows](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/customization/extend-tea-with-custom-workflows/) and the BMAD customization guide at [BMAD-METHOD/docs/how-to/customize-bmad.md](https://github.com/bmad-code-org/BMAD-METHOD/blob/main/docs/how-to/customize-bmad.md).
114
+ The npm package includes the workflow skills used by the tools.
115
+ See the [CI examples](./cli/examples/README.md) for test-review integration.
644
116
 
645
117
  ## Contributing
646
118
 
647
- See `CONTRIBUTING.md` for guidelines.
648
-
649
- ---
650
-
651
- ## 📦 Publishing TEA to NPM (for Maintainers)
652
-
653
- TEA uses an automated publish workflow modeled after the main `BMAD-METHOD` repo. It supports:
654
-
655
- - `next` prereleases published automatically from `main`
656
- - manual stable releases on the `latest` dist-tag
657
- - trusted npm publishing (no `NPM_TOKEN` secret)
658
- - metadata sync for `package.json`, `package-lock.json`, and `.claude-plugin/marketplace.json`
659
-
660
- ### Prerequisites (One-Time Setup)
661
-
662
- 1. **npm Trusted Publishing:**
663
-
664
- - In npm package settings for `bmad-method-test-architecture-enterprise`, configure Trusted Publishers for this GitHub repository
665
- - Allow publishes from the `bmad-code-org/bmad-method-test-architecture-enterprise` repo and the `.github/workflows/publish.yaml` workflow
666
- - GitHub Actions must be able to request an OIDC token (`id-token: write`), which the workflow already does
667
-
668
- 1. **GitHub App Secrets for Stable Releases:**
669
-
670
- - Add `RELEASE_APP_ID`
671
- - Add `RELEASE_APP_PRIVATE_KEY`
672
- - Install the corresponding GitHub App on this repository with contents write access
673
- - If `main` is protected, ensure the app is allowed to push the release commit and tag
674
- - These are used only for manual stable releases so the workflow can push the version bump commit and tag back to `main`
675
-
676
- 1. **Verify Package Configuration:**
677
-
678
- ```bash
679
- # Check package.json settings
680
- cat package.json | grep -A 3 "publishConfig"
681
- # Should show: "access": "public"
682
- if grep -Eq '"private"[[:space:]]*:[[:space:]]*true' package.json; then
683
- echo '❌ package.json must not set "private": true'
684
- else
685
- echo '✅ package.json is publishable ("private": true not present)'
686
- fi
687
- ```
688
-
689
- ### Release Process
690
-
691
- #### Option 1: Using npm Scripts (Recommended)
692
-
693
- From your local terminal after merging to `main`:
694
-
695
- ```bash
696
- # Publish the next prerelease from current main
697
- npm run release:next
698
-
699
- # Publish a stable patch release
700
- npm run release:patch
701
-
702
- # Publish a stable minor release
703
- npm run release:minor
704
-
705
- # Publish a stable major release
706
- npm run release:major
707
- ```
708
-
709
- #### Option 2: Manual Workflow Trigger
710
-
711
- 1. Go to **Actions** tab in GitHub
712
- 2. Click **"Publish"** workflow
713
- 3. Click **"Run workflow"**
714
- 4. Choose the branch to release, typically `main`
715
- 5. Select channel:
716
-
717
- - `next` for a prerelease publish
718
- - `latest` for a stable release
719
-
720
- 1. If using `latest`, choose the bump type (`patch`, `minor`, `major`)
721
- 2. Click **"Run workflow"**
722
-
723
- ### What Happens Automatically
724
-
725
- The workflow performs these steps:
726
-
727
- 1. ✅ **Validation**: Runs the full `npm test` chain: schema checks, install tests, knowledge checks, criteria-to-fragment traceability, enforce-hook coverage, eval data validation, release metadata, changelog, workflow descriptions, linting, markdown linting, and formatting. The chain includes the CLI suite (`npm run test:cli`).
728
- 2. ✅ **Version Bump**:
729
-
730
- - `next`: derives the next prerelease version and publishes it with dist-tag `next`
731
- - `latest`: bumps the stable version (`patch`, `minor`, or `major`)
732
-
733
- 1. ✅ **Metadata Sync**: Updates `.claude-plugin/marketplace.json` to match the package version before publishing
734
- 2. ✅ **Publish**: Publishes to npm with provenance enabled
735
-
736
- - `next` → `npm publish --tag next --provenance`
737
- - `latest` → `npm publish --tag latest --provenance`
738
-
739
- 1. ✅ **Stable Release Finalization**: For `latest`, creates a version bump commit, tags it, pushes it to `main`, and creates a GitHub Release
740
-
741
- ### Channel Strategy
742
-
743
- - `next`: prerelease channel for the newest merged changes
744
- - `latest`: stable channel for intentional releases
745
- - `patch`: bug fixes, no breaking changes
746
- - `minor`: new features, backwards compatible
747
- - `major`: breaking changes
748
-
749
- **Recommended Release Path:**
750
-
751
- 1. Merge releasable work to `main`
752
- 2. Let `next` publish for early validation
753
- 3. When ready, cut a stable `latest` release via `patch`, `minor`, or `major`
754
-
755
- ### Verify Publication
756
-
757
- **Check NPM:**
758
-
759
- ```bash
760
- npm view bmad-method-test-architecture-enterprise
761
- npm view bmad-method-test-architecture-enterprise dist-tags
762
- ```
763
-
764
- **Install TEA:**
765
-
766
- ```bash
767
- npx skills add bmad-code-org/bmad-method-test-architecture-enterprise
768
- ```
769
-
770
- Then run `bmad setup tea` in the assistant chat.
771
-
772
- **Test Workflows:** type these in the assistant chat, not in a shell.
773
-
774
- ```text
775
- /bmad-tea # load the agent persona and menu
776
- /bmad-testarch-test-design # run a workflow directly
777
- ```
778
-
779
- Hosts that use dollar-prefixed skills use `$` in place of `/`.
780
-
781
- ### Rollback a Release (if needed)
782
-
783
- If you need to unpublish a version:
784
-
785
- ```bash
786
- # Unpublish specific version (within 72 hours)
787
- npm unpublish bmad-method-test-architecture-enterprise@1.13.2-next.0
788
-
789
- # Deprecate version (preferred for older releases)
790
- npm deprecate bmad-method-test-architecture-enterprise@1.13.2-next.0 "Use version X.Y.Z instead"
791
- ```
792
-
793
- ### Troubleshooting
794
-
795
- **Trusted publishing failed:**
796
-
797
- - Verify npm Trusted Publishing is configured for this repository and workflow
798
- - Verify the workflow has `id-token: write`
799
- - Confirm the publish is running from the canonical repository, not a fork
800
-
801
- **"Package already exists":**
802
-
803
- - Check if package name is already taken on NPM
804
- - Update `name` in `package.json` if needed
805
-
806
- **"Version push failed":**
807
-
808
- - Verify `RELEASE_APP_ID` and `RELEASE_APP_PRIVATE_KEY` are configured
809
- - Verify the GitHub App is installed on this repository with contents write access
810
- - If branch protection is enabled on `main`, verify the app is allowed to push the release commit and tag
811
-
812
- **"Tests failed":**
813
-
814
- - Fix failing tests before release
815
- - Run `npm test` locally to verify
816
-
817
- **"Git push failed (protected branch)":**
818
-
819
- - This is not expected once the release GitHub App is configured correctly
820
- - Verify branch protection allows the app to push the release commit and tag
821
- - If needed, create the GitHub Release manually after resolving the app permissions
822
-
823
- ### Release Checklist
824
-
825
- Before releasing:
826
-
827
- - [ ] All tests passing: `npm test`
828
- - [ ] Documentation up to date
829
- - [ ] CHANGELOG.md updated
830
- - [ ] No uncommitted changes
831
- - [ ] On `main` branch
832
- - [ ] npm Trusted Publishing configured
833
- - [ ] `RELEASE_APP_ID` and `RELEASE_APP_PRIVATE_KEY` configured
834
- - [ ] Package name available on NPM
835
-
836
- After releasing:
837
-
838
- - [ ] Verify NPM publication: `npm view bmad-method-test-architecture-enterprise`
839
- - [ ] Test installation: `npx skills add bmad-code-org/bmad-method-test-architecture-enterprise`, then `bmad setup tea`
840
- - [ ] Verify workflows work
841
- - [ ] Check GitHub Release created
842
- - [ ] Monitor for issues
843
-
844
- ---
119
+ [CONTRIBUTING.md](./CONTRIBUTING.md) covers local checks, pull requests, and publishing.
120
+ [How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) explains the test and evaluation layers.
121
+ The [test suite guide](./test/README.md) has commands for running individual suites and live evaluations.
845
122
 
846
123
  ## Community
847
124
 
848
- - [Discord](https://discord.gg/gk8jAdXWmj) — Get help, share ideas, collaborate
849
- - [YouTube](https://youtube.com/@BMadCode) — Tutorials, master class, and more
850
- - [X / Twitter](https://x.com/BMadCode)
851
- - [Website](https://bmadcode.com)
852
-
853
- ## Support BMad
125
+ - [Discord](https://discord.gg/gk8jAdXWmj) for help and discussion.
126
+ - [GitHub Issues](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/issues) for bugs and feature requests.
127
+ - [YouTube](https://youtube.com/@BMadCode) for tutorials.
128
+ - [BMad website](https://bmadcode.com) and [X](https://x.com/BMadCode).
854
129
 
855
- BMad is free for everyone and always will be. Star this repo, [buy me a coffee](https://buymeacoffee.com/bmad), or email [contact@bmadcode.com](mailto:contact@bmadcode.com) for corporate sponsorship.
130
+ You can support BMad by starring the repository, [buying a coffee](https://buymeacoffee.com/bmad), or contacting [contact@bmadcode.com](mailto:contact@bmadcode.com) about sponsorship.
856
131
 
857
132
  ## License
858
133
 
859
- See `LICENSE`.
134
+ [MIT](./LICENSE).
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "$schema": "https://json.schemastore.org/package.json",
3
3
  "name": "bmad-method-test-architecture-enterprise",
4
- "version": "2.0.1-next.0",
4
+ "version": "2.0.1-next.1",
5
5
  "description": "Master Test Architect for quality strategy, test automation, and release gates",
6
6
  "keywords": [
7
7
  "bmad",
@@ -66,7 +66,7 @@ sessions:
66
66
  duration: "45 min"
67
67
  difficulty: intermediate
68
68
  objective: "Audit quality and ensure traceability"
69
- description: "Test Review (5 dimensions), Trace workflow, quality metrics"
69
+ description: "Test Review (rubric 5.0 scoring), Trace workflow, quality metrics"
70
70
  recommended_for:
71
71
  - intermediate
72
72
  - experienced
@@ -169,15 +169,15 @@ session-05-atdd-automate:
169
169
  session-06-quality-trace:
170
170
  passing_score: 70
171
171
  questions:
172
- - id: q1-five-dimensions
173
- question: "What are the 5 dimensions in Test Review workflow?"
172
+ - id: q1-review-scoring
173
+ question: "How does rubric 5.0 calculate the test-review score?"
174
174
  options:
175
- A: "Speed, cost, coverage, bugs, time"
176
- B: "Determinism, Isolation, Assertions, Structure, Performance"
177
- C: "Unit, integration, E2E, manual, exploratory"
178
- D: "P0, P1, P2, P3, P4"
175
+ A: "Average five dimension scores"
176
+ B: "Start at 100, apply severity deductions and up to 25 bonus points, then apply the highest-severity cap"
177
+ C: "Count the number of tests"
178
+ D: "Use line coverage alone"
179
179
  correct: B
180
- explanation: "Test Review evaluates 5 dimensions: Determinism (no flakiness), Isolation (parallel-safe), Assertions (correct checks), Structure (readable/maintainable organization), Performance (speed)."
180
+ explanation: "Registry findings deduct 10/5/2/1 points by severity. Bonuses total at most 25. The effective score is capped at 69/79/89/99 when CRITICAL/HIGH/MEDIUM/LOW findings are present."
181
181
 
182
182
  - id: q2-release-gate
183
183
  question: "When should the Trace workflow gate decision be RED (block release)?"
@@ -147,7 +147,7 @@ sessions:
147
147
  - test-review
148
148
  - trace
149
149
  key_concepts:
150
- - "5 dimensions of test quality"
150
+ - "Registry criteria, severity deductions, bonuses, and score caps"
151
151
  - "Quality scoring (0-100)"
152
152
  - "Requirements traceability"
153
153
  - "Release gate decisions"
@@ -49,7 +49,7 @@ Dana Whitfield has demonstrated proficiency in:
49
49
  - ✅ **Architecture Patterns:** Fixture composition, network-first patterns, data factories, step-file architecture
50
50
  - ✅ **Test Design:** Risk assessment (Probability × Impact), coverage planning, test levels framework
51
51
  - ✅ **Test Development:** ATDD red-green TDD approach, test automation, API testing patterns
52
- - ✅ **Quality Assurance:** Test review (5 dimensions), traceability, release gates, quality metrics
52
+ - ✅ **Quality Assurance:** Test review (rubric 5.0 scoring), traceability, release gates, quality metrics
53
53
  - ✅ **Advanced Techniques:** 47 of 59 knowledge fragments explored, with deep dives on flaky test diagnosis and contract testing, plus Playwright Utils integration
54
54
 
55
55
  ---
@@ -93,8 +93,8 @@ Present this content (mostly autonomous, clear and educational):
93
93
 
94
94
  TEA is a comprehensive test architecture framework that provides:
95
95
 
96
- - **9 Workflows:** Teach Me Testing, Test Design, Framework, CI, ATDD, Automate, Test Review, NFR Evidence Audit, Trace
97
- - **35 Knowledge Fragments:** Distilled expertise on patterns, best practices, Playwright Utils
96
+ - **10 Workflows:** Teach Me Testing, Test Design, Framework, CI, ATDD, Automate, Test Review, NFR Evidence Audit, Trace, Evaluate
97
+ - **59 Knowledge Fragments:** Distilled expertise on patterns, best practices, Playwright Utils
98
98
  - **Quality Standards:** Definition of Done with execution limits (no flaky tests, no hard waits, etc.)
99
99
  - **Risk-Based Testing:** P0-P3 matrix for prioritizing test coverage
100
100
 
@@ -102,7 +102,7 @@ TEA is a comprehensive test architecture framework that provides:
102
102
  Testing knowledge doesn't scale through manual teaching. TEA makes testing expertise accessible through:
103
103
 
104
104
  - Structured workflows that guide you step-by-step
105
- - Documentation (32 docs) organized by type (tutorials, how-to, explanation, reference)
105
+ - Documentation organized by type (tutorials, how-to, explanation, reference)
106
106
  - Knowledge fragments for just-in-time learning
107
107
  - Online resources: <https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/>
108
108
 
@@ -50,7 +50,7 @@ To teach test quality auditing and requirements traceability using Test Review a
50
50
  **What you'll learn:**
51
51
 
52
52
  - Test Review workflow (quality scoring)
53
- - 5 dimensions of test quality
53
+ - Registry criteria, severity deductions, bonuses, and score caps
54
54
  - Trace workflow (requirements traceability)
55
55
  - Release gate decisions
56
56
 
@@ -66,39 +66,23 @@ Set session-06-quality-trace `status: 'in-progress'`.
66
66
 
67
67
  **Purpose:** Audit test quality with 0-100 scoring
68
68
 
69
- **5 Dimensions of Quality:**
69
+ #### Rubric 5.0: criteria and severity scoring
70
70
 
71
- #### 1. Determinism (0-100)
71
+ The review applies the 35 rows in the test-review criteria registry. Each finding names its row, evidence, and fixed severity.
72
+ Start at 100 and deduct for every finding:
72
73
 
73
- - Tests pass/fail consistently
74
- - No flakiness, no randomness
75
- - Proper async handling
74
+ | Severity | Deduction | Score cap when present |
75
+ | -------- | --------- | ---------------------- |
76
+ | CRITICAL | 10 | 69 |
77
+ | HIGH | 5 | 79 |
78
+ | MEDIUM | 2 | 89 |
79
+ | LOW | 1 | 99 |
76
80
 
77
- #### 2. Isolation (0-100)
81
+ Five bonus categories can each add 5 points: fixture setup, data factories, network-first ordering, isolation, and stable test IDs. Each bonus must hold across every reviewed file, for a maximum of 25 points.
82
+ Clamp the raw score to 0–100, then apply the cap for the highest finding severity.
83
+ A CRITICAL finding recommends Block; a HIGH finding or score below 70 recommends Request Changes. MEDIUM or LOW findings at 70 or above recommend Approve with Comments. A review with no findings recommends Approve.
78
84
 
79
- - Tests run independently
80
- - No shared state
81
- - Parallelizable
82
-
83
- #### 3. Assertions (0-100)
84
-
85
- - Correct checks for expected behavior
86
- - Meaningful assertions (not just presence)
87
- - Fails for the right reasons
88
-
89
- #### 4. Structure (0-100)
90
-
91
- - Readable test code
92
- - Clear organization and naming
93
- - Minimal duplication
94
-
95
- #### 5. Performance (0-100)
96
-
97
- - Test execution speed
98
- - Resource usage
99
- - Parallel efficiency
100
-
101
- **Overall Score = Average of 5 dimensions.**
85
+ Example: one HIGH finding deducts 5 points. The raw score is 95, and the HIGH cap makes the effective score 79.
102
86
 
103
87
  {Role-adapted example}
104
88
 
@@ -151,11 +135,11 @@ Set session-06-quality-trace `status: 'in-progress'`.
151
135
 
152
136
  ### 6. Quiz (3 questions)
153
137
 
154
- **Q1:** "What are the 5 dimensions in Test Review workflow?
155
- A) Speed, cost, coverage, bugs, time
156
- B) Determinism, Isolation, Assertions, Structure, Performance
157
- C) Unit, integration, E2E, manual, exploratory
158
- D) P0, P1, P2, P3, P4"
138
+ **Q1:** "How does rubric 5.0 calculate the test-review score?
139
+ A) Average five dimension scores
140
+ B) Start at 100, apply severity deductions and up to 25 bonus points, then apply the highest-severity cap
141
+ C) Count the number of tests
142
+ D) Use line coverage alone"
159
143
 
160
144
  Correct: B
161
145
 
@@ -204,4 +188,4 @@ Return to {nextStepFile}.
204
188
 
205
189
  ## 🚨 SUCCESS METRICS
206
190
 
207
- ✅ Test Review and Trace taught, quality dimensions explained, quiz passed, notes generated, returned to hub.
191
+ ✅ Test Review and Trace taught, severity scoring explained, quiz passed, notes generated, returned to hub.
@@ -174,7 +174,7 @@ average_score: { average_score }
174
174
  - ✅ **Architecture Patterns:** Fixture composition, network-first patterns, data factories, step-file architecture
175
175
  - ✅ **Test Design:** Risk assessment (Probability × Impact), coverage planning, test levels framework
176
176
  - ✅ **Test Development:** ATDD red-green TDD approach, test automation, API testing patterns
177
- - ✅ **Quality Assurance:** Test review (5 dimensions), traceability, release gates, quality metrics
177
+ - ✅ **Quality Assurance:** Test review (rubric 5.0 scoring), traceability, release gates, quality metrics
178
178
  - ✅ **Advanced Techniques:** Knowledge fragments explored, Playwright Utils integration
179
179
 
180
180
  ---
@@ -2,7 +2,7 @@
2
2
 
3
3
  [bmod]
4
4
  code = "tea"
5
- version = "2.0.1-next.0"
5
+ version = "2.0.1-next.1"
6
6
  update_source = "github:bmad-code-org/bmad-method-test-architecture-enterprise/skills"
7
7
  skills = [
8
8
  "bmad-tea",