bmad-method-test-architecture-enterprise 2.0.1-next.0 → 2.0.1-next.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +79 -804
- package/package.json +1 -1
- package/skills/bmad-teach-me-testing/data/curriculum.yaml +1 -1
- package/skills/bmad-teach-me-testing/data/quiz-questions.yaml +7 -7
- package/skills/bmad-teach-me-testing/data/session-content-map.yaml +1 -1
- package/skills/bmad-teach-me-testing/resources/tea-completion-summary.example.md +1 -1
- package/skills/bmad-teach-me-testing/steps-c/step-04-session-01.md +3 -3
- package/skills/bmad-teach-me-testing/steps-c/step-04-session-06.md +20 -36
- package/skills/bmad-teach-me-testing/steps-c/step-05-completion.md +1 -1
- package/skills/bmod-tea/bmod.toml +1 -1
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "2.0.1-next.
|
|
34
|
+
"version": "2.0.1-next.1",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/README.md
CHANGED
|
@@ -1,859 +1,134 @@
|
|
|
1
1
|
# TEA: Test Engineering Architect
|
|
2
2
|
|
|
3
|
-
[
|
|
4
|
-
[License: MIT](./LICENSE)
|
|
3
|
+
[Documentation](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/) · [Getting started](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/tea-lite-quickstart/) · [Contributing](./CONTRIBUTING.md) · [MIT license](./LICENSE)
|
|
5
4
|
|
|
6
|
-
|
|
5
|
+
TEA is a BMad module for test planning, automation, evaluation, and release decisions.
|
|
6
|
+
Its agent, Murat, uses ten workflows and a shared testing knowledge base to turn requirements and risks into tests and evidence.
|
|
7
|
+
You can use TEA on its own or alongside BMad Method.
|
|
7
8
|
|
|
8
|
-
TEA
|
|
9
|
+
TEA helps you:
|
|
9
10
|
|
|
10
|
-
-
|
|
11
|
-
-
|
|
12
|
-
-
|
|
13
|
-
-
|
|
14
|
-
-
|
|
15
|
-
|
|
16
|
-
TEA is two layers. **TEA Core** decides what must be verified, at what depth, with what evidence, and whether that evidence is sufficient to release; it assumes nothing about your language, framework, or platform. **Execution targets** turn those decisions into runnable tests on a specific stack, and that layer is swappable. See [Verification Architecture](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/verification-architecture/) for the split, and [Execution Targets](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/execution-targets/) for exactly which stacks are covered at which depth.
|
|
17
|
-
|
|
18
|
-
Docs: [https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/)
|
|
19
|
-
|
|
20
|
-
## Why TEA
|
|
21
|
-
|
|
22
|
-
- Risk-based prioritization (P0-P3) from probability × impact, with measurable quality gates
|
|
23
|
-
- Requirements traced to evidence, and PASS / CONCERNS / FAIL / WAIVED release decisions that survive an audit
|
|
24
|
-
- NFR thresholds set at design time and audited against real evidence, defaulting to CONCERNS when evidence is missing
|
|
25
|
-
- Consistent, knowledge-base driven outputs instead of whatever the model felt like producing
|
|
26
|
-
- Stack-aware execution: Playwright and Cypress for browsers, Maestro for mobile native, Pact for contracts, pytest / JUnit / Go test / xUnit / RSpec for backend services, k6 and scanners as NFR evidence
|
|
27
|
-
- Three enforcement points rather than one: fragments steer generation, a hook blocks the write, and `test-review` scores what actually landed
|
|
28
|
-
|
|
29
|
-
## How BMad Works
|
|
30
|
-
|
|
31
|
-
BMad works because it turns big, fuzzy work into **repeatable workflows**. Each workflow is broken into small steps with clear instructions, so the AI follows the same path every time. It also uses a **shared knowledge base** (standards and patterns) so outputs are consistent, not random. In short: **structured steps + shared standards = reliable results**.
|
|
32
|
-
|
|
33
|
-
## How TEA Fits In
|
|
34
|
-
|
|
35
|
-
TEA plugs into BMad the same way a specialist plugs into a team. It uses the same step‑by‑step workflow engine and shared standards, but focuses exclusively on testing and quality gates. That means you get a **risk‑based test plan**, **automation guidance**, and **go/no‑go decisions** that align with the rest of the BMad process.
|
|
36
|
-
|
|
37
|
-
## How It Actually Works
|
|
38
|
-
|
|
39
|
-
### The problem it is built around
|
|
40
|
-
|
|
41
|
-
Ask a model to "write tests for this feature" and you reliably get four things: redundant coverage, incorrect assertions, flaky tests, and diffs nobody can review. The cause is a category error. Prompt-driven generation is nondeterministic, and it is being pointed at the one artifact whose entire job is determinism.
|
|
42
|
-
|
|
43
|
-
TEA's answer is not a better prompt. It is to make the work repeatable at three levels.
|
|
44
|
-
|
|
45
|
-
**Repeatable instructions.** A single 5,000-word instruction file fails predictably: the model skims it, improvises past the vague parts ("analyze codebase then generate tests" specifies nothing), keeps going because nothing told it where to stop, and returns something different next run. So every workflow is cut into step files that each do one thing, state what "finished" means, restate the context they need, list what they must not do, and load one at a time. Consistent output for the same input is what makes everything else possible: you cannot parallelize work whose boundaries are undefined.
|
|
46
|
-
|
|
47
|
-
**Repeatable standards.** 59 knowledge fragments carry the patterns, and an index decides which ones enter context for the task at hand. The model is not asked to remember how fixtures compose or what network-first means. It is handed the fragment.
|
|
48
|
-
|
|
49
|
-
**Repeatable judgment.** Risk scores, priorities, quality scores, and gate decisions are computed from stated rules rather than produced as opinions. This is the part most tools skip, and it is the difference between a review you can act on and a review you have to re-litigate.
|
|
50
|
-
|
|
51
|
-
### The order you run things
|
|
52
|
-
|
|
53
|
-
The ten workflows are a directed graph, not a menu. `skills/bmod-tea/help/help.md` describes it, and the `bmad` skill reads it to recommend the next step.
|
|
54
|
-
|
|
55
|
-
```text
|
|
56
|
-
Phase 3, solutioning, once per project
|
|
57
|
-
TD test-design (system-level) → TF framework → CI ci
|
|
58
|
-
|
|
59
|
-
Phase 4, implementation, per story
|
|
60
|
-
create-story → AT atdd → dev implements → TA automate
|
|
61
|
-
|
|
62
|
-
Epic or release gate
|
|
63
|
-
TA automate → RV test-review
|
|
64
|
-
TA automate → NR nfr
|
|
65
|
-
TA automate → EV evaluate → CI ci
|
|
66
|
-
RV test-review → TR trace (Phase 2 gate decision)
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
Phase 3 order matters and is deliberate: run `test-design` first so NFR evidence needs can shape the infrastructure, then `framework` once the architecture and the test design have settled the stack, then `ci` once the framework exists so the pipeline wires to real commands.
|
|
70
|
-
|
|
71
|
-
`test-design` is dual-mode. At system level it produces an architecture-facing document and a QA-facing one. Per epic it produces `test-design/test-design-epic-N.md`. `teach-me-testing` sits outside the lifecycle and runs once per learner.
|
|
72
|
-
|
|
73
|
-
For the full lifecycle diagram including the BMad phases around TEA, see [TEA Overview](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/tea-overview/).
|
|
74
|
-
|
|
75
|
-
### One epic, end to end
|
|
76
|
-
|
|
77
|
-
Here is the same feature moving through the chain, with the rules TEA actually applies at each step.
|
|
78
|
-
|
|
79
|
-
**1. Risk, in** `test-design`**.** Every identified risk gets a probability of 1 to 3 (unlikely, possible, likely) and an impact of 1 to 3 (minor, degraded, critical). Score is the product, so the range is 1 to 9, and the score determines the action:
|
|
80
|
-
|
|
81
|
-
| Score | Action | Gate impact |
|
|
82
|
-
| ----- | -------- | -------------------- |
|
|
83
|
-
| 1-3 | DOCUMENT | none |
|
|
84
|
-
| 4-5 | MONITOR | none, watch closely |
|
|
85
|
-
| 6-8 | MITIGATE | CONCERNS at the gate |
|
|
86
|
-
| 9 | BLOCK | automatic FAIL |
|
|
87
|
-
|
|
88
|
-
A checkout risk scored `probability 2 × impact 3 = 6` lands in MITIGATE. It gets a row in the test design with a named owner and a date, and it will surface as CONCERNS at the gate until the mitigation is real.
|
|
89
|
-
|
|
90
|
-
Priority is a separate judgment that the risk score informs rather than determines. P0 is revenue-critical, security-critical, data-integrity, regulatory, or previously broken. P1 is core journeys and complex logic. P2 is secondary features. P3 is nice-to-have. The design pins the effort too: P0 tests are budgeted at 2 hours each, P1 at 1, P2 at 0.5, P3 at 0.25.
|
|
91
|
-
|
|
92
|
-
**2. Test level, still in** `test-design`**.** Favor unit when logic can be isolated with no side effects; integration for persistence, service contracts, and component boundaries; E2E for user-facing critical paths and multi-system interactions. Before adding any test, the duplicate-coverage guard asks whether a lower level already covers it. Overlap is allowed only for genuinely different aspects, defense in depth on critical paths, or regression prevention on something that broke before.
|
|
93
|
-
|
|
94
|
-
**3. Red tests, in** `atdd` (optional). Run before implementation. It generates acceptance tests that all carry `test.skip()`, plus data factories, fixtures, and an implementation checklist that lists, per test, the tasks required to make it pass and the command to run it. The developer un-skips one test, confirms it fails, then makes it pass. The red phase is the point: a test that has never failed has proven nothing.
|
|
95
|
-
|
|
96
|
-
**4. Coverage, in** `automate`**.** Run after implementation. Workers generate API, E2E, backend, and mobile tests in parallel depending on the detected stack, and the aggregation step reports the totals broken down by priority. It also rolls up every deviation from an active integration mandate as `file:line: reason`, and writes `None` when there are none, because a reader cannot tell an empty section from a forgotten one.
|
|
97
|
-
|
|
98
|
-
**5. Quality, in** `test-review`**.** Every finding must cite a registry row (`C1`, `H2`, `M4`), and the row carries the severity. Score starts at 100:
|
|
99
|
-
|
|
100
|
-
```text
|
|
101
|
-
Starting Score: 100
|
|
102
|
-
Critical Violations: -{count} × 10
|
|
103
|
-
High Violations: -{count} × 5
|
|
104
|
-
Medium Violations: -{count} × 2
|
|
105
|
-
Low Violations: -{count} × 1
|
|
106
|
-
Bonus (5 categories, each 0 or 5, max +25)
|
|
107
|
-
Final Score: clamped to 0-100 Grade: A ≥90, B ≥80, C ≥70, D ≥60, else F
|
|
108
|
-
```
|
|
109
|
-
|
|
110
|
-
The verdict is then derived from the findings, not written by the model:
|
|
111
|
-
|
|
112
|
-
```javascript
|
|
113
|
-
if (CRITICAL > 0) return 'Block'; // a test that cannot fail is not a suggestion
|
|
114
|
-
if (HIGH > 0) return 'Request Changes';
|
|
115
|
-
if (score < 70) return 'Request Changes'; // volume of MEDIUM/LOW can also fail the bar
|
|
116
|
-
if (MEDIUM + LOW > 0) return 'Approve with Comments';
|
|
117
|
-
return 'Approve';
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
This is the part worth sitting with. A suite with one CRITICAL and three MEDIUM findings, earning two bonus categories, scores `100 - 10 - 6 + 10 = 94`, a grade A, and is still a **Block**. Score measures the suite. The verdict answers a different question: is there anything here that makes the suite lie? One committed `.skip` on the test that mattered, or one `expect(true).toBe(true)`, means green proves nothing, which is worse than an absent test because it buys false confidence.
|
|
121
|
-
|
|
122
|
-
That separation exists because it was measured. Two reviewers of the same four files scored 82 and 85, which is noise, and returned opposite verdicts, which is not. `--fail-on request-changes` acts on the verdict, so the gate was being decided by the unpinned half of the report. `DESIGN-CRITERIA-REGISTRY.md` records the whole investigation.
|
|
123
|
-
|
|
124
|
-
**6. Gate, in** `trace` **Phase 2.** Requirements are mapped to tests with Given/When/Then, coverage is computed per priority, and the decision is deterministic:
|
|
125
|
-
|
|
126
|
-
| Condition | Decision |
|
|
127
|
-
| --------------------------------------------------------------- | ------------ |
|
|
128
|
-
| P0 coverage below 100% | **FAIL** |
|
|
129
|
-
| Overall coverage below 80% | **FAIL** |
|
|
130
|
-
| P1 coverage below 80% | **FAIL** |
|
|
131
|
-
| P0 at 100%, overall ≥ 80%, P1 ≥ 90% | **PASS** |
|
|
132
|
-
| P0 at 100%, overall ≥ 80%, P1 between 80% and 89% | **CONCERNS** |
|
|
133
|
-
| Stakeholder-approved waiver with the complete approval contract | **WAIVED** |
|
|
134
|
-
|
|
135
|
-
Our epic finishes at P0 100%, P1 87%, overall 84%, so it gates at CONCERNS with the residual risk named rather than passing quietly. Two overlays can lower that result further and can never raise it: a requirement resting only on recorded live verification caps at CONCERNS, and a synthetic coverage oracle below high confidence does the same. WAIVED is never derived; a human sets it, and the artifact demands an approver, approval date, reason, expiry, monitoring plan, remediation owner, and fix target. The authoritative contract is in the [Traceability template](./skills/bmad-testarch-trace/trace-template.md#waiver-details).
|
|
136
|
-
|
|
137
|
-
If the run is not gate-eligible at all, because evidence collection was waived, restricted, inaccessible, or deferred, TEA emits no decision rather than computing one on partial evidence.
|
|
138
|
-
|
|
139
|
-
## Architecture & Flow
|
|
140
|
-
|
|
141
|
-
BMad is a small **agent + workflow engine**. There is no external orchestrator; everything runs inside the LLM context window through structured instructions. TEA adds two pieces that run outside it: a Node hook that intercepts writes in your project, and a CLI that drives the review workflow headlessly in CI.
|
|
142
|
-
|
|
143
|
-
### Building Blocks
|
|
144
|
-
|
|
145
|
-
| File / Scope | What it does | When it loads |
|
|
146
|
-
| --------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------- |
|
|
147
|
-
| `skills/bmad-tea/SKILL.md` | Murat's activation sequence and critical actions; renders the `{agent.menu}` placeholder | First, activates the TEA agent |
|
|
148
|
-
| `skills/bmad-tea/customize.toml` | Agent customization surface: `[[agent.menu]]` items (code to skill), persona fields, persistent facts, hooks | During agent activation |
|
|
149
|
-
| `skills/<workflow>/SKILL.md` | Workflow entrypoint: resolves workflow customization, picks mode, routes to the first step | When a TEA workflow is invoked |
|
|
150
|
-
| `skills/<workflow>/customize.toml` | Workflow customization surface: activation hooks, persistent facts, optional `on_complete` behavior | During workflow activation |
|
|
151
|
-
| `skills/<workflow>/workflow.yaml` | Machine-readable metadata: run variables, tool hints, output paths | Tooling and workflow metadata lookups |
|
|
152
|
-
| `instructions.md` | Workflow-specific summary and operator notes | On demand |
|
|
153
|
-
| `steps-c/*.md` | **Create** steps: primary execution, 5 to 12 files per workflow, 75 across the module | One at a time (just-in-time) |
|
|
154
|
-
| `steps-c/step-NNx-subagent-*.md` | **Worker** steps: one isolated dimension each, dispatched in parallel | When an orchestrator step delegates |
|
|
155
|
-
| `steps-e/*.md` | **Edit** steps: always 2 files, assess target then apply edit | One at a time |
|
|
156
|
-
| `steps-v/*.md` | **Validate** steps: always 1 file, evaluate against the checklist | On demand |
|
|
157
|
-
| `checklist.md` | Validation criteria: what "done" looks like for this workflow | Read by steps-v |
|
|
158
|
-
| `*-template.md` | Output skeleton with `{PLACEHOLDER}` vars, filled in by steps to produce the artifact | Read by steps-c when generating output |
|
|
159
|
-
| `bmad-testarch-test-review/steps-c/criteria-registry.md` | The 35 scoreable rows, each with a fixed severity and gate class. Severity is read here | Read by every review worker before it scores anything |
|
|
160
|
-
| `bmad-testarch-framework/resources/hooks/tea-enforce.cjs` | Project-level guardrail for mechanically decidable test-quality violations; framework Create scaffolds it | Before writes, after writes or shell commands, and when an agent turn stops |
|
|
161
|
-
| `skills/bmod-tea/bmod.toml` | Module record: version, member skills, setup questions, install messages | Read by `bmad setup tea` and `bmad` |
|
|
162
|
-
| `skills/bmod-tea/roster.toml` | Lists Murat as the module's agent | Read by roster-aware skills such as party mode |
|
|
163
|
-
| `skills/bmod-tea/help/*.md` | What each TEA skill is for, when to recommend it, and what each setup answer does | Read by the `bmad` skill when routing |
|
|
164
|
-
| `skills/bmod-tea/knowledge/tea-index.csv` | Knowledge fragment index: id, name, description, tags, tier, path. 59 rows | Read before recommendations and by knowledge-loading steps |
|
|
165
|
-
| `skills/bmod-tea/knowledge/*.md` | 59 reusable fragments: standards, patterns, API references, integration mandates | Selectively read into context by tier and config flags |
|
|
166
|
-
|
|
167
|
-
The knowledge base has one copy, in the `bmod-tea` skill. The agent and every workflow that consults it read it as a sibling skill folder, `{skill-root}/../bmod-tea/knowledge`. If `bmod-tea` is missing, the skill says so and offers to install it. `bmad-teach-me-testing` carries a curated pointer file at `data/tea-resources-index.yaml` instead of reading the fragments.
|
|
168
|
-
|
|
169
|
-
```mermaid
|
|
170
|
-
flowchart TB
|
|
171
|
-
U[User] --> A[Agent activation<br/>persona + config + menu]
|
|
172
|
-
A --> W[Workflow entry: SKILL.md<br/>mode: Create / Resume / Validate / Edit]
|
|
173
|
-
W --> S[Step files<br/>steps-c / steps-e / steps-v]
|
|
174
|
-
S --> K[Knowledge fragments<br/>tea-index.csv to knowledge/*.md]
|
|
175
|
-
S --> T[Templates & checklists]
|
|
176
|
-
S --> P[Orchestrator step]
|
|
177
|
-
P --> X[Isolated workers<br/>one dimension each]
|
|
178
|
-
X --> G[Aggregation step<br/>scored against criteria-registry.md]
|
|
179
|
-
S --> O[Outputs: plans, tests, reports]
|
|
180
|
-
G --> O
|
|
181
|
-
O --> V[Validation: steps-v + checklist.md]
|
|
182
|
-
O --> C[Checkpoint frontmatter<br/>resume where it stopped]
|
|
183
|
-
```
|
|
184
|
-
|
|
185
|
-
### How It Works at Runtime
|
|
186
|
-
|
|
187
|
-
**1. Activation.** `/bmad-tea` or `$bmad-tea` loads the agent skill. It resolves its customization block across base, team, and user layers, adopts the persona, loads persistent facts and the `core` and `modules.tea` config from `_bmad/config.toml` and its `_bmad/custom/` layers, greets you, and renders `{agent.menu}` as a numbered table. Naming one clear intent in your first message ("let's design tests for this epic") skips the menu and dispatches directly. A message that supports several menu choices triggers one focused question naming the relevant choices before any workflow starts.
|
|
188
|
-
|
|
189
|
-
**2. Workflow entry.** Direct workflow commands use the installed skill name, such as `/bmad-testarch-automate` or `$bmad-testarch-automate`, depending on the host's invocation syntax. `TA` is the equivalent agent-menu code, available only once TEA is active. Either way, the workflow's `SKILL.md` resolves its own `[workflow]` customization block and asks which mode to run: Create, Resume, Validate, or Edit. Create and Resume both route into `steps-c/`; Validate into `steps-v/`; Edit into `steps-e/`. `test-review` alone supports `headless: true`, which skips the greeting and the menu and runs Create directly. That is how the CLI drives it in CI.
|
|
190
|
-
|
|
191
|
-
**3. Steps.** Each step file declares its own wiring in YAML frontmatter: `outputFile`, `nextStepFile`, and where relevant `knowledgeIndex` and `resumeStepFile`. A step loads on its own, pulls only the fragments its mode and config flags call for, fills any `*-template.md` placeholders, writes its output, and names the next step. Nothing loads the whole workflow at once, and the step files say so in as many words: "Do not load the next step until this step is complete."
|
|
192
|
-
|
|
193
|
-
**4. Progress and resume.** Every create step appends itself to a checkpoint file's YAML frontmatter (`stepsCompleted`, `lastStep`, `lastSaved`), so an interrupted run resumes at the next incomplete step. Every workflow that runs once per scope also records run identity: `runScope` and `runKey` are resolved before anything is saved, the file name carries the `run_key` (`test-design/test-design-progress-epic-3.md`, `trace/traceability-matrix-epic-16.md`), and Resume refuses to continue a file whose `runKey` belongs to a different run. A trace run for epic 16 never opens epic 15's matrix or gate decision, and interrupting a system-level run to start an epic-level one leaves the first intact. `framework` and `ci` scaffold once per project, so each keeps a single fixed checkpoint in its own folder.
|
|
194
|
-
|
|
195
|
-
**5. Validation.** `steps-v/` scores the finished output against `checklist.md`.
|
|
196
|
-
|
|
197
|
-
See [Step-File Architecture](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/step-file-architecture/) for the loading model, worker isolation, and the per-workflow step patterns.
|
|
198
|
-
|
|
199
|
-
### Parallel Workers and Execution Modes
|
|
200
|
-
|
|
201
|
-
Five workflows split their heaviest step across isolated workers. An orchestrator step does no work of its own; it resolves the mode, dispatches, and hands off to an aggregation step.
|
|
202
|
-
|
|
203
|
-
| Workflow | Workers |
|
|
204
|
-
| ------------- | -------------------------------------------------------------- |
|
|
205
|
-
| `test-review` | determinism, isolation, maintainability, performance |
|
|
206
|
-
| `nfr` | security, performance, reliability, maintainability |
|
|
207
|
-
| `automate` | API, E2E, backend, mobile. Stack-gated, so 1 to 3 of the 4 run |
|
|
208
|
-
| `atdd` | failing API tests, failing E2E tests |
|
|
209
|
-
| `test-design` | system-level mode may generate its two documents in parallel |
|
|
210
|
-
|
|
211
|
-
`tea_execution_mode` decides how they run: `auto`, `agent-team`, `subagent`, or `sequential`. With `tea_capability_probe` at its default of `true`, `auto` probes the runtime and prefers agent-team, then subagent, then sequential, which keeps behavior portable across supported agent runtimes. With probing off, TEA honors the configured mode strictly and fails with an explicit error rather than falling back silently. Mode changes orchestration only. The output schema, the validation rules, and the aggregation contract are identical in every mode.
|
|
212
|
-
|
|
213
|
-
Workers exchange nothing directly. Each writes a JSON file under `/tmp` keyed by a shared run timestamp, and the aggregation step asserts every expected file exists before it scores anything. Isolation is what makes a parallel review honest. The shared criteria registry is what stops two isolated workers from disagreeing about what a finding is worth: every worker loads it, and none of them choose a severity.
|
|
214
|
-
|
|
215
|
-
### How Knowledge Gets Selected
|
|
216
|
-
|
|
217
|
-
`tea-index.csv` classifies all 59 fragments into three tiers: **core** (24, always loaded), **extended** (19, loaded when deeper analysis is called for), and **specialized** (16, loaded only when the case matches, such as contract testing on a real consumer-provider boundary). Steps name the fragments they need, and they name their exclusions just as explicitly. A Maestro run is told not to load the browser fragments, because a device flow has no DOM and no request interceptor, and loading them invites browser patterns into a device flow.
|
|
218
|
-
|
|
219
|
-
Over-loading is treated as a real defect, not a harmless cost. `npm run eval:fragment-selection` measures both directions: recall of the fragments a step requires, and the rate at which a run pulls one the step excludes by name.
|
|
220
|
-
|
|
221
|
-
### The Three Control Points
|
|
222
|
-
|
|
223
|
-
A test rule can be enforced at three moments, and most tools occupy one of them. TEA occupies all three. The release gate is the fourth row below because it consumes what the other three produce, rather than being a fourth place to enforce a rule.
|
|
224
|
-
|
|
225
|
-
| Point | When | Mechanism | What it closes |
|
|
226
|
-
| -------------- | ---------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------- |
|
|
227
|
-
| **Generation** | before the test exists | knowledge fragments and integration mandates | the model improvising a pattern TEA already has a standard for |
|
|
228
|
-
| **Write** | as the file lands | `tea-enforce.cjs` on PreToolUse, PostToolUse, and Stop | a `.only`, a hard wait, or a tautological assertion getting committed at all |
|
|
229
|
-
| **Review** | after the fact | `test-review` scored against the criteria registry | severity drifting with whichever model happened to run the review |
|
|
230
|
-
| **Gate** | at release | `trace` Phase 2, PASS / CONCERNS / FAIL / WAIVED | shipping on evidence nobody checked was sufficient |
|
|
231
|
-
|
|
232
|
-
The hook has separate installation and runtime lifecycles. The `bmad-testarch-framework` Create path installs it during its documentation and scripts step. Resume reaches the same step when installation is still incomplete. Framework Validate and Edit do not install it, and no other TEA workflow calls it.
|
|
233
|
-
|
|
234
|
-
Once installed, the hook is project-scoped rather than workflow-scoped. It runs on matching tool events across TEA workflows, other agents, and ordinary coding sessions without requiring Murat or a TEA workflow to be active.
|
|
235
|
-
|
|
236
|
-
The three passes cover different user-visible moments. `--pre` checks content before a direct file write reaches disk and rejects a blocking violation with a fix. `--post` re-reads the affected file after direct file writes or shell commands, then reports violations that only become visible in the complete file. `--stop` scans recently modified test files when the agent turn finishes, including outputs a code generator did not name in its command. All three passes are limited to the test and Pact configuration globs written for the detected stack in `.tea/enforce-config.json`.
|
|
237
|
-
|
|
238
|
-
This closes the gap between advisory generation guidance and a later `test-review`. It blocks seven mechanically decidable Absolute rules and warns on one: focused tests, tautological assertions, hard waits, oversized test files, Maestro flows that cannot fail, two Pact parallelism rules, and undocumented disabled tests as the warning. Rules that require semantic judgment stay in `test-review`. The hook fails open on its own errors so a broken guardrail cannot lock the agent out of writing. Agent platforms without a write-time hook API skip installation and rely on `test-review` for enforcement.
|
|
239
|
-
|
|
240
|
-
**How workflows become commands.** `npx skills add` copies each TEA skill into the host's skill directory under its own name. Invoking that name loads the skill, and the step-file process takes over. The skill name is the same on every host.
|
|
11
|
+
- Plan coverage by risk and priority, from P0 to P3.
|
|
12
|
+
- Set up a test framework and CI, write acceptance tests, and expand automation.
|
|
13
|
+
- Review test quality against a fixed criteria registry.
|
|
14
|
+
- Evaluate skills, agents, and AI features with scored probes and regression checks.
|
|
15
|
+
- Trace requirements to evidence and make PASS, CONCERNS, or FAIL release decisions, with waivers recorded separately.
|
|
241
16
|
|
|
242
17
|
## Install
|
|
243
18
|
|
|
19
|
+
Install the skills from GitHub:
|
|
20
|
+
|
|
244
21
|
```bash
|
|
245
22
|
npx skills add bmad-code-org/bmad-method-test-architecture-enterprise
|
|
246
23
|
```
|
|
247
24
|
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
```text
|
|
251
|
-
bmad setup tea
|
|
252
|
-
```
|
|
253
|
-
|
|
254
|
-
`bmad setup` runs through [uv](https://docs.astral.sh/uv/) (Python 3.11 or later, which uv fetches when it is missing), so install uv first. The `bmad` skill comes from BMad Method core. If you do not have it, add it with `npx skills add bmad-code-org/BMAD-METHOD --skill bmad bmod-core-tools`. Setup asks TEA's questions and writes the answers to `_bmad/config.toml`. Run `bmad setup tea` again later to see or change them.
|
|
255
|
-
|
|
256
|
-
`npx skills add` installs the head of `main`. To install a release, point it at the release tag instead: `npx skills add https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/tree/v<version>/skills`.
|
|
257
|
-
|
|
258
|
-
Murat is listed in the module's roster, so party mode can bring him into a group discussion alongside other BMad agents.
|
|
259
|
-
|
|
260
|
-
### Invocation Syntax
|
|
261
|
-
|
|
262
|
-
| Host convention | Example |
|
|
263
|
-
| --------------------- | ---------------------------------------- |
|
|
264
|
-
| Slash command | `/bmad-testarch-automate` |
|
|
265
|
-
| Dollar-prefixed skill | `$bmad-tea` or `$bmad-testarch-automate` |
|
|
266
|
-
|
|
267
|
-
## Quickstart
|
|
268
|
-
|
|
269
|
-
1. Install TEA (above)
|
|
270
|
-
2. Load the TEA menu with `/bmad-tea` or `$bmad-tea` if you want a conversational entrypoint.
|
|
271
|
-
3. Run one of the core workflows:
|
|
272
|
-
|
|
273
|
-
- `TD` / `/bmad-testarch-test-design` / `$bmad-testarch-test-design` — test design, risk assessment, and NFR planning
|
|
274
|
-
- `AT` / `/bmad-testarch-atdd` / `$bmad-testarch-atdd` — failing acceptance tests first (TDD red phase)
|
|
275
|
-
- `TA` / `/bmad-testarch-automate` / `$bmad-testarch-automate` — expand automation coverage
|
|
276
|
-
|
|
277
|
-
1. Or use in party mode: `/party` to include TEA with other agents
|
|
278
|
-
|
|
279
|
-
## Engagement Models
|
|
280
|
-
|
|
281
|
-
- **No TEA**: Use your existing testing approach
|
|
282
|
-
- **TEA Solo**: Standalone use on non-BMad projects
|
|
283
|
-
- **TEA Lite**: Start with `automate` only for fast onboarding
|
|
284
|
-
- **Integrated (BMad Method / Enterprise)**: Use TEA in Phases 3–4 and release gates
|
|
285
|
-
|
|
286
|
-
## Workflows
|
|
287
|
-
|
|
288
|
-
| Trigger | Slash Command | Dollar Skill | Purpose |
|
|
289
|
-
| ------- | ---------------------------- | ---------------------------- | --------------------------------------------------------------------------- |
|
|
290
|
-
| TMT | `/bmad-teach-me-testing` | `$bmad-teach-me-testing` | Teach Me Testing (TEA Academy) |
|
|
291
|
-
| TD | `/bmad-testarch-test-design` | `$bmad-testarch-test-design` | System-level or epic-level test design and NFR planning |
|
|
292
|
-
| TF | `/bmad-testarch-framework` | `$bmad-testarch-framework` | Scaffold test framework (frontend, backend, fullstack, or mobile) |
|
|
293
|
-
| CI | `/bmad-testarch-ci` | `$bmad-testarch-ci` | Set up CI/CD quality pipeline (multi-platform) |
|
|
294
|
-
| AT | `/bmad-testarch-atdd` | `$bmad-testarch-atdd` | Generate failing acceptance tests + checklist |
|
|
295
|
-
| TA | `/bmad-testarch-automate` | `$bmad-testarch-automate` | Expand test automation coverage |
|
|
296
|
-
| RV | `/bmad-testarch-test-review` | `$bmad-testarch-test-review` | Review test quality and score |
|
|
297
|
-
| NR | `/bmad-testarch-nfr` | `$bmad-testarch-nfr` | Audit implemented NFR evidence |
|
|
298
|
-
| TR | `/bmad-testarch-trace` | `$bmad-testarch-trace` | Trace requirements to tests + gate decision |
|
|
299
|
-
| GATE | agent menu only | agent menu only | Route the release gate: test review, NFR evidence audit, then trace Phase 2 |
|
|
300
|
-
|
|
301
|
-
`GATE` is a routing prompt on the agent menu, so it has no standalone command. Load the agent with `/bmad-tea` or `$bmad-tea` and pick it there.
|
|
302
|
-
|
|
303
|
-
## The Release Gate
|
|
304
|
-
|
|
305
|
-
`trace` Phase 2 produces the decision: PASS, CONCERNS, FAIL, or WAIVED. Two mechanics sit under that vocabulary and are easy to miss.
|
|
306
|
-
|
|
307
|
-
**Live evidence is capped.** A requirement covered only by recorded live verification forces PASS down to CONCERNS, with a rationale naming the recorded source SHA. The overlay only ever lowers a PASS or annotates an existing CONCERNS. It can never lift a FAIL. Only a `pass` recorded against the commit under trace counts; `stale`, `unverifiable`, `contradicted`, `blocked`, and the rest are reported as blockers. The JSON contract is published at [Live Verification Results](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/live-verification-results/), so any runner can produce it. Trace reads that file and never runs anything itself.
|
|
308
|
-
|
|
309
|
-
**Some runs are not gate-eligible at all.** A collection status of `waived`, `restricted`, `inaccessible`, or `deferred_shared` means no decision is emitted rather than a decision computed on partial evidence. A missing manifest resolves to `INACCESSIBLE`, not to 0% coverage.
|
|
310
|
-
|
|
311
|
-
### `tea-test-review` in CI
|
|
312
|
-
|
|
313
|
-
Installing this package also installs a `tea-test-review` binary that runs the review workflow headlessly against a pull request diff.
|
|
25
|
+
Setup needs [uv](https://docs.astral.sh/uv/) and the `bmad` and `bmod-core-tools` skills from BMad Method core.
|
|
26
|
+
If those skills are missing, install them:
|
|
314
27
|
|
|
315
28
|
```bash
|
|
316
|
-
npx
|
|
29
|
+
npx skills add bmad-code-org/BMAD-METHOD --skill bmad bmod-core-tools
|
|
317
30
|
```
|
|
318
31
|
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
The recommendation is derived from the findings rather than taken from the agent's prose. Any CRITICAL derives Block. Any HIGH, or a score under 70, derives Request Changes. The agent's own stated recommendation is preserved as `reportedRecommendation` when the two disagree. `--waive` exists for the exceptions and requires an expiry.
|
|
322
|
-
|
|
323
|
-
`tea-test-review --github` also opens a check run and keeps one short pull-request comment up to date, and `tea-test-review render` prints the same comment, check-run text or summary from a verdict file for any other CI. Copy-paste workflows live at `cli/examples/pr-test-review.yml` (GitHub Actions) and `cli/examples/gitlab-ci.yaml` (GitLab), and the full flag, exit-code, and security reference is at [tea-test-review CLI](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/tea-test-review-cli/).
|
|
324
|
-
|
|
325
|
-
## Configuration
|
|
326
|
-
|
|
327
|
-
TEA's setup questions are declared in `skills/bmod-tea/bmod.toml` and asked by `bmad setup tea`, which writes the answers to `modules.tea` in `_bmad/config.toml`. Every one is read by a workflow today. Team overrides go in `_bmad/custom/config.toml` and personal ones in `_bmad/custom/config.user.toml`; run `bmad setup tea` again to see or change the answers.
|
|
328
|
-
|
|
329
|
-
- `test_artifacts` — base output folder for test artifacts. Each workflow writes into its own folder under it (`test-design/`, `atdd/`, `automate/`, `test-review/`, `nfr/`, `trace/`, `ci/`, `framework/`), and a file produced once per story, epic, or release carries that scope in its name, such as `trace/gate-decision-epic-16.json`. See [Output Layout](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/configuration/#output-layout)
|
|
330
|
-
- `tea_use_playwright_utils` — enable Playwright Utils integration (boolean, default true). When true **and the package is installed**, `@seontechnologies/playwright-utils` becomes the default implementation for everything it covers: generated Playwright tests use `interceptNetworkCall`, `apiRequest`, `recurse`, and `log` without being asked, and `test-review` flags a vanilla equivalent that carries no stated reason. See [Integrate Playwright Utils](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/customization/integrate-playwright-utils/)
|
|
331
|
-
- `tea_use_pactjs_utils` — enable Pact.js Utils integration for contract testing (boolean, default true). It decides how Pact suites are written, not whether a project gets one: TEA still requires a real consumer-provider boundary before scaffolding a contract test. When on **and the package is installed**, generated Pact code uses `createProviderState`, `buildVerifierOptions`, and `createRequestFilter` rather than raw Pact boilerplate. A flag with no install generates the raw path and reports one recommendation rather than flagging every file
|
|
332
|
-
- `tea_pact_mcp` — SmartBear MCP for PactFlow/Broker interaction: mcp, none (string, default mcp). Safe without a broker: every broker-dependent step degrades to provider source or an OpenAPI spec and reports that the broker was unreachable
|
|
333
|
-
- `tea_browser_automation` — browser automation mode: auto, cli, mcp, none (string, default auto)
|
|
334
|
-
- `tea_execution_mode` — how TEA orchestrates multi-step generation and evaluation: auto, subagent, agent-team, sequential (string, default auto)
|
|
335
|
-
- `tea_capability_probe` — probe the runtime before selecting an execution mode (boolean, default true). With it off, TEA honors the configured mode strictly and fails loudly instead of falling back
|
|
336
|
-
- `test_stack_type` — detected or configured stack type (auto, frontend, backend, fullstack, mobile). Mobile is checked before frontend, because a React Native project carries React in `package.json` and would otherwise misdetect as web
|
|
337
|
-
- `test_framework` — detected or configured test framework (auto, Playwright, Cypress, Jest, Vitest, pytest, JUnit, Go test, dotnet test, RSpec, Maestro, other)
|
|
338
|
-
|
|
339
|
-
Two settings that only one workflow reads are set through that workflow's customization file instead, under `[workflow]`:
|
|
340
|
-
|
|
341
|
-
- `evaluations_folder` — base folder for Evaluate's own evaluation folders (default `evals`, relative to the project root). Set it in `_bmad/custom/bmad-testarch-evaluate.toml`
|
|
342
|
-
- `ci_platform` — CI platform (auto, github-actions, gitlab-ci, jenkins, azure-devops, harness, circle-ci, other; default auto). Set it in `_bmad/custom/bmad-testarch-ci.toml`
|
|
343
|
-
|
|
344
|
-
Full option reference: [Configuration](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/configuration/).
|
|
345
|
-
|
|
346
|
-
## Knowledge Base
|
|
347
|
-
|
|
348
|
-
TEA relies on a curated testing knowledge base of 59 fragments, indexed by tier:
|
|
349
|
-
|
|
350
|
-
- Index: `skills/bmod-tea/knowledge/tea-index.csv`
|
|
351
|
-
- Fragments: `skills/bmod-tea/knowledge/`
|
|
352
|
-
- Tiers: 24 core, 19 extended, 16 specialized
|
|
353
|
-
|
|
354
|
-
Workflows load only the fragments required for the current task, and the required set is named in the step file rather than inferred from index tags. See [Knowledge Base](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/knowledge-base/).
|
|
355
|
-
|
|
356
|
-
## Repository Layout
|
|
32
|
+
Then type this in your assistant chat:
|
|
357
33
|
|
|
358
34
|
```text
|
|
359
|
-
|
|
360
|
-
├── bmod-tea/ # module record: bmod.toml, roster.toml, help/, knowledge/{tea-index.csv, *.md}
|
|
361
|
-
├── bmad-tea/ # the agent: SKILL.md, customize.toml
|
|
362
|
-
├── bmad-teach-me-testing/
|
|
363
|
-
├── bmad-testarch-atdd/
|
|
364
|
-
├── bmad-testarch-automate/
|
|
365
|
-
├── bmad-testarch-ci/
|
|
366
|
-
├── bmad-testarch-evaluate/
|
|
367
|
-
├── bmad-testarch-framework/ # resources/hooks/tea-enforce.cjs lives here
|
|
368
|
-
├── bmad-testarch-nfr/
|
|
369
|
-
├── bmad-testarch-test-design/
|
|
370
|
-
├── bmad-testarch-test-review/ # steps-c/criteria-registry.md lives here
|
|
371
|
-
└── bmad-testarch-trace/
|
|
372
|
-
|
|
373
|
-
cli/ # tea-test-review: the headless CI gate
|
|
374
|
-
docs/ # source of truth for the docs site
|
|
375
|
-
website/ # Astro + Starlight, consumes docs/ through a symlink
|
|
376
|
-
tools/ # validators, doc build, changelog stamping
|
|
377
|
-
test/ # quality gate suites and the six eval harnesses
|
|
378
|
-
```
|
|
379
|
-
|
|
380
|
-
## How TEA Keeps Itself Honest
|
|
381
|
-
|
|
382
|
-
TEA has deterministic checks and live evals. All eleven of TEA's skills have coverage registered in the suite manifest: ten behavioral suites, and the suite Evaluate authored for `bmad-testarch-evaluate` itself.
|
|
383
|
-
|
|
384
|
-
### Current Eval Coverage
|
|
385
|
-
|
|
386
|
-
The eight suites under `test/evals/` measure one decision inside each knowledge-bearing workflow: whether the agent selects the required knowledge fragments and avoids fragments the workflow excludes. They do not execute the complete workflow or grade its final artifact.
|
|
387
|
-
|
|
388
|
-
The `atdd`, `automate`, `ci`, `framework`, `nfr`, `teach-me-testing`, `test-design`, `test-review`, and `trace` suites add behavioral evals, and so does `bmad-tea`'s own routing suite. `atdd` generates red-phase scaffolds against an unimplemented fixture, activates them, executes them under NFR9's isolation, and scores whether each one fails for the reason its criterion states. `automate` runs four hand-authored spec sets, standing in for generated coverage, against a fixed voucher-redemption service and against a scratch copy carrying its seeded minimum-spend regression, and scores whether the one spec set that tests the boundary catches the regression and whether the other three are correctly reported as missing it, vacuous, or duplicated rather than as clean passes; it spends no vendor call, since there is no live-agent mode to build one for. `ci` scaffolds a pipeline for a project whose team wrote down what it needs, then parses the generated workflow, lints it with `actionlint`, and scores every requested trigger, permission, command, gate, and artifact against a minimal-request control. `framework` installs and smoke-tests a scaffolded framework in a network-isolated sandbox, proving it runs rather than merely reading as valid. `nfr` audits one evidence bundle with known gaps and one clean bundle. `teach-me-testing` tests multi-turn teaching sessions, seeded quiz corrections, and persistent learner progress through `tea-transcript-runner`. `test-design` analyzes one seeded epic and one clean control epic. `test-review` runs the complete review against files containing nine planted defects plus one clean file, then scores recall, the non-false-positive rate, score variance, and verdict stability. `trace` runs the complete traceability workflow against a ten-criterion seeded set whose declared gate is FAIL and a five-criterion clean set whose declared gate is PASS, then scores criterion status, the gate decision and the criteria behind it, the coverage arithmetic, evidence citation, and case stability. `bmad-tea` routes nineteen intents to the right menu item, or declines the ones nothing on the menu serves.
|
|
389
|
-
|
|
390
|
-
| Skill | Fragment-selection cases | Full behavioral eval |
|
|
391
|
-
| --------------------------- | ------------------------ | ----------------------------------------------------------------------------------- |
|
|
392
|
-
| `bmad-tea` | N/A | Yes; nineteen intents, eleven with a right answer and eight controls |
|
|
393
|
-
| `bmad-teach-me-testing` | N/A | Yes; multi-turn session with seeded wrong quiz answer, review loop, and progress |
|
|
394
|
-
| `bmad-testarch-atdd` | 3 | Yes; one unimplemented fixture and five acceptance criteria |
|
|
395
|
-
| `bmad-testarch-automate` | 5 | Yes; four hand-authored spec sets against a fixed service and its seeded regression |
|
|
396
|
-
| `bmad-testarch-ci` | 2 | Yes; one full request, one minimal request and one request over an evaluation plan |
|
|
397
|
-
| `bmad-testarch-evaluate` | N/A | Yes; Evaluate-authored, five seeded probes and four clean controls |
|
|
398
|
-
| `bmad-testarch-framework` | 3 | Yes; install and smoke-test generated scaffold in network-isolated sandbox |
|
|
399
|
-
| `bmad-testarch-nfr` | 2 | Yes; one evidence bundle with known gaps and one clean bundle |
|
|
400
|
-
| `bmad-testarch-test-design` | 5 | Yes; one seeded epic and one clean control epic |
|
|
401
|
-
| `bmad-testarch-test-review` | 2 | Yes; three files, nine planted defects, and one clean file |
|
|
402
|
-
| `bmad-testarch-trace` | 2 | Yes; a ten-criterion seeded set and a five-criterion clean set |
|
|
403
|
-
|
|
404
|
-
A passing fragment-selection eval means the workflow loaded the right knowledge. All eleven of TEA's skills have coverage registered in the suite manifest; `bmad-testarch-evaluate` is covered by the `evaluate-authored` suite Evaluate wrote and ran on itself, so the manifest's deferred array is empty. [How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) explains the testing hierarchy, exit classes, and evidence model in plain language. The source-controlled [Eval Quality and Behavioral Coverage Roadmap](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-roadmap/) records the per-skill contracts, runner work, CI plan, and intended boundary with the upcoming standalone `eval-quality` project. [The eval-quality Command Adapter](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-command-adapter/) records how TEA runs the commands those evals measure, and [Adopting eval-quality, One Skill at a Time](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-adoption-guide/) is the guide for taking one skill in another BMAD module from asserted quality to measured quality.
|
|
405
|
-
|
|
406
|
-
### Deterministic Checks
|
|
407
|
-
|
|
408
|
-
`npm test` chains 164 checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Twelve of the 164 keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
|
|
409
|
-
|
|
410
|
-
- `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 35 rows are currently mapped across 51 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
|
|
411
|
-
- `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, the fragment-selection case count, the replay totals that the test README and the header of the replay suite state, the story count, epic count and appended-story count in the overview of the Evaluate plan, and the lane count in its parallel-lanes section.
|
|
412
|
-
The plan's source module also refuses a plan whose five lane lists differ between the plan's epics file and its sprint status file, so the gate fails on a story moved, dropped, duplicated or reordered in one file only.
|
|
413
|
-
A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
|
|
414
|
-
- `test:doc-claims` runs `eval-quality-gates doc-claims`, which holds a hand-written prose claim — a named symbol, a transcribed list, a worked JSON example, a named exit code, or a claim whose truth depends on when it was written — against the artifact it describes. A registered dated claim whose sentence was rewritten out from under it fails the same way a wrong symbol name does, so the entry cannot go stale by drifting out from under its own key either.
|
|
415
|
-
- `test:enforce-hook` fails when a new Absolute registry row appears in neither the hook's enforced list nor its deferred list. This prevents a rule from being added without an explicit write-time enforcement decision.
|
|
416
|
-
- `test:atdd-workflow-guidance` holds ATDD generation to one supplied criterion id per executable leaf, one primary scaffold per criterion, a criterion-defining first assertion, and a transition-bearing primary branch for state changes. It also checks every literal API and E2E example title against the same criterion-id format.
|
|
417
|
-
- `test:eval-data` checks that all 24 fragment-selection cases are structurally usable: their workflow context files exist, every expected fragment exists and is indexed for that workflow, and the required and forbidden sets do not overlap. The expected sets come from the workflow step files. This check does not ask an agent to select anything.
|
|
418
|
-
- `test:eval-routing-boundaries` mutates the facts, candidate codes, and missing deciding information in each source-bound routing ambiguity rule. It also rejects duplicate and orphan boundaries, plus any boundary that would turn a clear intent into a clarification.
|
|
419
|
-
- `test:eval-schemas` checks this repository's own eval suite manifest against its schema, confirms that every threshold it declares is the threshold the harness actually applies, that the runner capabilities it declares are the ones the harness grants its runner, and that the preflight argv it declares fails on a missing runner and passes on a present one, and fails when a TEA skill has neither a behavioral or Evaluate-authored suite nor a deferred declaration. A suite list that omits a skill reads as coverage, so the omission has to be an error rather than a silence.
|
|
420
|
-
- `test:eval-diagnostics` checks that current result writers emit bounded case and repetition diagnostics, that quality and environment outcomes stay separate, that unstable signatures remain visible, and that the protected 1.3.0 baseline remains readable and byte-for-byte unchanged.
|
|
421
|
-
- `test:nfr-evidence-guidance` holds every NFR worker and the publication audit to the supplied-evidence ledger, exercises supported, missing, and mismatched evidence, rejects contaminating examples, and binds the protected baseline and focused remediation records by digest.
|
|
422
|
-
- `test:contract-oracles` evaluates every oracle in every eval contract with `eval-quality`'s own evaluator, over the stored replay outputs and over constructed selections for all 24 fragment-selection cases, and fails when an oracle faults or disagrees with the harness scorer on the same evidence. The first run found that every regex oracle in the `test-review` contract was refused by the evaluator before it matched anything.
|
|
423
|
-
- `test:evaluate-guidance` parses the Evaluate skill's `SKILL.md` stage list and fails when a stage entry is missing, out of order, or points at a `references/` guide that does not exist.
|
|
424
|
-
|
|
425
|
-
Four more checks hold the Evaluate documentation:
|
|
426
|
-
|
|
427
|
-
- `test:docs-build-names` scans every page under `docs/` and fails on a name from how TEA was built: a story id, a decision id, a line of epic narrative, a planning path, a pull request number or a word from the review process.
|
|
428
|
-
It also fails when an executable command block (a fenced `bash`, `sh`, `shell` or `zsh` block) holds a timestamped run or score ID, which a reader's own run never has.
|
|
429
|
-
- `test:docs-evaluate-sidebar` fails when a page that names `tea-evaluate` or `bmad-testarch-evaluate` is missing from the site sidebar, and when the TEA overview links no page that Evaluate owns.
|
|
430
|
-
- `test:docs-gap-reader` lifts the held-out reader command out of the gaps how-to, runs it over a `gap-view.json` built with a caught, an uncaught and a withheld row, and fails when it crashes or leaves a row unprinted.
|
|
431
|
-
- `test:docs-tutorial` runs the commands of the Evaluate tutorial against its fixture and fails on a command that errors or an output line the page promises and the run does not print.
|
|
432
|
-
|
|
433
|
-
`test:workflows-lint` runs `actionlint` over every workflow in `.github/workflows/` and the `cli/examples/pr-test-review.yml` template, and fails on any finding. It disables the shellcheck and pyflakes integrations so the answer does not depend on which of those binaries a machine has. With no `actionlint` on `PATH` it fails and names `tools/install-actionlint.sh` rather than skipping, and it proves on every run that a workflow reading a step that does not exist is refused.
|
|
434
|
-
|
|
435
|
-
These checks produce the same answer from the same repository state. They need no agent credential, network call, or model budget. `test:eval-data` runs through `npm test`, the local pre-commit hook, pull-request quality checks, and the publish workflow.
|
|
436
|
-
|
|
437
|
-
`test:doc-invocations` runs `eval-quality-gates doc-invocations`, which re-runs every `npm run <script>` this README, the adoption guide, and `test/README.md` document in their CI-usage and individual-suites blocks against the real repository, so a renamed script or a broken example fails the build instead of a reader. Each of the eighteen documented scripts is declared as a full literal spelling, never a shared `npm run` prefix, which is what keeps the gate from ever matching, let alone running, `npm run eval:all` or `npm run release:next` on the same pages. The gate's own entry script is the one real program it is allowed to reach; it refuses anything outside its own hardcoded allowlist before invoking real `npm`, and `test:doc-invocation-entry` holds that allowlist's comment-to-script lookup against all three pages' literal text, line by line, so a swapped or drifted comment fails there even when the script it resolves to would still run.
|
|
438
|
-
|
|
439
|
-
### Start Here
|
|
440
|
-
|
|
441
|
-
You do not start an interactive agent session. A live eval launches the selected agent CLI as a headless subprocess, sends it each prompt, waits for the result, and scores the result.
|
|
442
|
-
|
|
443
|
-
The normal path is one command. It runs fragment selection across all eight covered workflow skills, then runs the behavioral `bmad-tea-routing`, `ci`, `nfr`, `test-design`, `test-review`, `trace`, `atdd` and `automate` evals:
|
|
444
|
-
|
|
445
|
-
```bash
|
|
446
|
-
npm run eval:all -- --agent codex
|
|
447
|
-
```
|
|
448
|
-
|
|
449
|
-
Use `claude` or `agy` instead, or run all three built-in adapters:
|
|
450
|
-
|
|
451
|
-
```bash
|
|
452
|
-
npm run eval:all -- --agent claude
|
|
453
|
-
npm run eval:all -- --agent agy
|
|
454
|
-
npm run eval:all -- --agent agy --agent claude --agent codex
|
|
455
|
-
```
|
|
456
|
-
|
|
457
|
-
`eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `ci` project, per `test-design` epic, per `trace` fixture set, and per `atdd` story. One runner makes 115 agent calls: 48 fragment selections, 38 routing intents, 3 reviews, 4 audits, 12 pipelines, 4 test designs, 4 traces, and 2 ATDD generations. All three built-in runners make 345 calls.
|
|
458
|
-
|
|
459
|
-
Check the data, executable, login, fixtures, and expected results without making a model call:
|
|
460
|
-
|
|
461
|
-
```bash
|
|
462
|
-
npm run eval:all -- --agent codex --preflight-only
|
|
463
|
-
npm run eval:all -- --agent claude --preflight-only
|
|
464
|
-
npm run eval:all -- --agent agy --preflight-only
|
|
35
|
+
bmad setup tea
|
|
465
36
|
```
|
|
466
37
|
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
38
|
+
Setup asks about your test stack, integrations, and output folder, then writes the answers to `_bmad/config.toml`.
|
|
39
|
+
Run it again to change those answers.
|
|
40
|
+
The [configuration reference](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/configuration/) covers team and personal overrides.
|
|
470
41
|
|
|
471
|
-
|
|
42
|
+
The GitHub install follows `main`.
|
|
43
|
+
To install a tagged release, replace `<version>` with the release number:
|
|
472
44
|
|
|
473
45
|
```bash
|
|
474
|
-
|
|
475
|
-
npm run eval:test-review -- --agent codex --runs 1
|
|
476
|
-
|
|
477
|
-
# Complete eval with one runner.
|
|
478
|
-
npm run eval:test-review -- --agent codex
|
|
479
|
-
npm run eval:test-review -- --agent claude
|
|
480
|
-
|
|
481
|
-
# Complete eval with all three built-in runners. This makes nine review calls.
|
|
482
|
-
npm run eval:test-review -- --agent agy --agent claude --agent codex
|
|
46
|
+
npx skills add https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/tree/v<version>/skills
|
|
483
47
|
```
|
|
484
48
|
|
|
485
|
-
|
|
49
|
+
## Start with one workflow
|
|
486
50
|
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
51
|
+
In Claude Code, Cursor, and Windsurf, type `/bmad-testarch-test-design` in chat.
|
|
52
|
+
In Codex, type `$bmad-testarch-test-design`.
|
|
53
|
+
You can also load `/bmad-tea` or `$bmad-tea` and choose a menu code.
|
|
54
|
+
Each workflow can run directly in a fresh session.
|
|
490
55
|
|
|
491
|
-
|
|
492
|
-
|
|
56
|
+
| Workflow | Command | Menu | Use it to |
|
|
57
|
+
| ---------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | ---- | ------------------------------------------------------ |
|
|
58
|
+
| [Teach Me Testing](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/teach-me-testing/) | `/bmad-teach-me-testing` | TMT | Learn testing through seven sessions |
|
|
59
|
+
| [Test Design](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-test-design/) | `/bmad-testarch-test-design` | TD | Plan risks, coverage, and NFR evidence |
|
|
60
|
+
| [Framework Setup](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/setup-test-framework/) | `/bmad-testarch-framework` | TF | Scaffold a test framework |
|
|
61
|
+
| [CI Setup](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/setup-ci/) | `/bmad-testarch-ci` | CI | Connect tests and quality checks to CI |
|
|
62
|
+
| [ATDD](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-atdd/) | `/bmad-testarch-atdd` | AT | Write acceptance scaffolds before implementation |
|
|
63
|
+
| [Automate](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-automate/) | `/bmad-testarch-automate` | TA | Add coverage to implemented features |
|
|
64
|
+
| [Test Review](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-test-review/) | `/bmad-testarch-test-review` | RV | Audit test quality and score findings |
|
|
65
|
+
| [NFR Evidence Audit](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-nfr-assess/) | `/bmad-testarch-nfr` | NR | Assess performance, security, and reliability evidence |
|
|
66
|
+
| [Evaluate](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/evaluate-your-first-skill/) | `/bmad-testarch-evaluate` | EV | Build and run a behavioral evaluation |
|
|
67
|
+
| [Trace](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/workflows/run-trace/) | `/bmad-testarch-trace` | TR | Map requirements to tests and decide a release gate |
|
|
493
68
|
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
```
|
|
69
|
+
For Codex, replace the leading `/` in the table with `$`.
|
|
70
|
+
The agent menu also accepts `GATE` to route you through test review, NFR evidence audit, and trace Phase 2.
|
|
497
71
|
|
|
498
|
-
|
|
72
|
+
Choose a starting path:
|
|
499
73
|
|
|
500
|
-
|
|
74
|
+
- [TEA Lite](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/tea-lite-quickstart/) for an existing project that needs more test coverage.
|
|
75
|
+
- [TEA Academy](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/learn-testing-tea-academy/) to learn testing.
|
|
76
|
+
- [TEA Overview](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/tea-overview/) for the lifecycle and workflow order.
|
|
77
|
+
- [Evaluate Your First Skill](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/tutorials/evaluate-your-first-skill/) for a runnable evaluation tutorial.
|
|
501
78
|
|
|
502
|
-
|
|
503
|
-
| --------------------- | ------------------------------------------------------------------------------------------------ |
|
|
504
|
-
| `atdd` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-atdd --runs 1` |
|
|
505
|
-
| `automate` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-automate --runs 1` |
|
|
506
|
-
| `ci` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-ci --runs 1` |
|
|
507
|
-
| `framework` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-framework --runs 1` |
|
|
508
|
-
| `nfr` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-nfr --runs 1` |
|
|
509
|
-
| `test-design` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-test-design --runs 1` |
|
|
510
|
-
| `test-review` routing | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-test-review --runs 1` |
|
|
511
|
-
| `trace` | `npm run eval:fragment-selection -- --agent codex --workflow bmad-testarch-trace --runs 1` |
|
|
79
|
+
## How it works
|
|
512
80
|
|
|
513
|
-
|
|
81
|
+
Each workflow loads one step file at a time.
|
|
82
|
+
The steps define the task, the evidence to collect, and when to stop.
|
|
83
|
+
They load the knowledge fragments needed for that task from `skills/bmod-tea/knowledge/tea-index.csv`.
|
|
84
|
+
Workflows can delegate independent tasks when the assistant supports workers; execution mode is configurable.
|
|
514
85
|
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
|
|
86
|
+
Review findings use fixed criteria and severity rules.
|
|
87
|
+
Trace applies coverage thresholds and evidence checks to the release decision.
|
|
88
|
+
Generated tests and reports still need review, and a release gate depends on the evidence collected.
|
|
89
|
+
See [step-file architecture](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/step-file-architecture/), [test quality standards](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/test-quality-standards/), and [risk-based testing](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/risk-based-testing/).
|
|
519
90
|
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
91
|
+
Planning and traceability work across stacks.
|
|
92
|
+
Test generation and execution support vary by target.
|
|
93
|
+
The [execution-target matrix](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/execution-targets/) lists the supported stacks and their limits.
|
|
523
94
|
|
|
524
|
-
|
|
95
|
+
## Command-line tools
|
|
525
96
|
|
|
526
|
-
|
|
97
|
+
TEA also publishes an npm package for CI and terminal use.
|
|
98
|
+
It requires Node.js 22.20.0 or later:
|
|
527
99
|
|
|
528
100
|
```bash
|
|
529
|
-
npm
|
|
530
|
-
npm run eval:all -- --agent claude
|
|
531
|
-
npm run eval:all -- --agent codex
|
|
101
|
+
npm install --save-dev bmad-method-test-architecture-enterprise
|
|
532
102
|
```
|
|
533
103
|
|
|
534
|
-
|
|
104
|
+
- [`tea-test-review`](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/tea-test-review-cli/) runs the review workflow against changed tests and returns a gate verdict.
|
|
105
|
+
- [`tea-evaluate`](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/tea-evaluate-cli/) checks, runs, scores, and compares behavioral evaluations.
|
|
535
106
|
|
|
536
|
-
|
|
537
|
-
2. Run non-interactively in the repository working directory.
|
|
538
|
-
3. Print its final response to standard output. The review eval must also allow the agent to write the report path named in the prompt.
|
|
539
|
-
4. Exit with a nonzero status when the agent call fails.
|
|
540
|
-
|
|
541
|
-
[Gemini CLI headless mode](https://geminicli.com/docs/cli/headless/) accepts standard input alongside a `-p` prompt. Once Gemini is installed and authenticated, run every live eval with:
|
|
542
|
-
|
|
543
|
-
```bash
|
|
544
|
-
npm run eval:all -- \
|
|
545
|
-
--agent custom \
|
|
546
|
-
--agent-cmd gemini \
|
|
547
|
-
--agent-arg -p \
|
|
548
|
-
--agent-arg "Follow the complete instructions from standard input." \
|
|
549
|
-
--agent-arg --output-format \
|
|
550
|
-
--agent-arg text \
|
|
551
|
-
--agent-arg --approval-mode \
|
|
552
|
-
--agent-arg yolo \
|
|
553
|
-
--agent-arg --skip-trust \
|
|
554
|
-
--env-pass GEMINI_API_KEY \
|
|
555
|
-
--env-pass GOOGLE_API_KEY
|
|
556
|
-
```
|
|
557
|
-
|
|
558
|
-
This uses `yolo` because the review eval must write its report. To run fragment selection alone with a read-only policy:
|
|
107
|
+
For `tea-evaluate`, install `eval-quality` alongside TEA in the evaluations folder:
|
|
559
108
|
|
|
560
109
|
```bash
|
|
561
|
-
npm
|
|
562
|
-
|
|
563
|
-
--agent-cmd gemini \
|
|
564
|
-
--agent-arg -p \
|
|
565
|
-
--agent-arg "Follow the complete instructions from standard input." \
|
|
566
|
-
--agent-arg --output-format \
|
|
567
|
-
--agent-arg text \
|
|
568
|
-
--agent-arg --approval-mode \
|
|
569
|
-
--agent-arg plan \
|
|
570
|
-
--agent-arg --skip-trust \
|
|
571
|
-
--env-pass GEMINI_API_KEY \
|
|
572
|
-
--env-pass GOOGLE_API_KEY
|
|
110
|
+
npm install --prefix evals bmad-method-test-architecture-enterprise eval-quality
|
|
111
|
+
npm exec --prefix evals -- tea-evaluate --help
|
|
573
112
|
```
|
|
574
113
|
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
### What Has to Pass
|
|
578
|
-
|
|
579
|
-
Every row below is a declared gate. Ten of TEA's eleven skills have behavioral suites registered in the suite manifest, and `bmad-testarch-evaluate` has the Evaluate-authored suite. Recorded live baselines preserve timestamped history, and [How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) describes the evidence model. The live thresholds are the ones this repository's own eval suite manifest declares, and `npm run test:eval-schemas` fails when a harness constant and the manifest disagree.
|
|
580
|
-
|
|
581
|
-
| Eval | Declared threshold | Declared volume |
|
|
582
|
-
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
|
|
583
|
-
| `npm run eval:all -- --agent ...` | All eight live suites below meet their thresholds for the selected runner | 48 selections, 38 routing calls, 3 reviews, 4 audits, 6 pipelines, 4 test designs, 4 traces, 2 generations |
|
|
584
|
-
| Fragment selection | At least 90% required-fragment recall, at most 10% forbidden-fragment selection, and stable choices across repeated cases | 24 cases twice: 48 calls |
|
|
585
|
-
| Test review | At least 70% overall recall, 100% CRITICAL recall, an 80% non-false-positive rate, score standard deviation no higher than 3, and a stable verdict | Three complete reviews |
|
|
586
|
-
| Trace | At least 90% criterion-status accuracy and 90% evidence-citation precision; 100% on both discriminating criteria, the gate decision, the gate criteria, the coverage arithmetic, the oracle resolution, the rejected evidence, the waiver validity, and the live evidence; no clean-set false positive, no unstable case, and no fixture mutation | Two fixture sets twice: 4 calls |
|
|
587
|
-
| bmad-tea routing | At least 90% route accuracy, 80% reason-token recall, 87.5% scope fidelity, and 75% on both clarification and decline recall; zero confident routes on an intent nothing on the menu serves, zero on one whose menu items are genuinely close, zero clear intents left unrouted, and at most two intents whose answer changes between repetitions | 19 intents twice: 38 calls |
|
|
588
|
-
| NFR | At least 87.5% domain-status accuracy and 90% threshold fidelity; 100% on undecidable domains, the overall status, and domain coverage; no unsupported PASS, no fabricated evidence, no clean-bundle false positive, no duplicate domain section, no disagreement between the gate artifact and the assessment it summarizes, no unstable case, and no fixture mutation | Two evidence bundles twice: 4 calls |
|
|
589
|
-
| CI | Zero parse failures and zero `actionlint` findings; at least 97% requested-element recall and 100% trigger accuracy; zero unrequested elements, zero workflow rule violations, no unstable case, and no fixture mutation | Six projects twice: 12 calls |
|
|
590
|
-
| Test design | At least 80% grounded-risk recall, 80% risk precision, and 80% coverage mapping; 100% on the scale compliance, the score arithmetic, the category validity, the band placement, the risk-id shape, the risk-link resolution, and the priority ordering; no unscored risk table, no ungrounded risk, no missed top-severity risk, no risk-ceiling excess, no unstable case, and no fixture mutation | Two epics twice: 4 calls |
|
|
591
|
-
| ATDD | 100% red-for-intended-reason rate and criteria coverage; no vacuous pass, no scaffold still skipped after activation, no load error, no non-assertion exit, no unmapped test, no production mutation, and no unstable case | One story twice: 2 generation calls, plus execution under NFR9's isolation, which spends no call |
|
|
592
|
-
| `npm run test:eval-data` | Every case references valid workflow files and indexed fragments; required and forbidden sets do not overlap | No agent calls |
|
|
593
|
-
| `npm run test:eval-schemas` | The suite manifest matches its schema, declares the thresholds the harnesses apply, and accounts for every TEA skill | No agent calls |
|
|
594
|
-
|
|
595
|
-
Every declared repetition has to complete. A run that loses one to a timeout, a transport error, or an unparseable reply cannot measure variance or stability, so it exits `2` rather than reporting a lower score.
|
|
596
|
-
|
|
597
|
-
The non-false-positive rate is the share of reported findings that are not definite false positives, and only the clean fixture makes a false positive definite. A finding on a seeded fixture that matches no planted row is reported separately as `unattributed`: the fixture may carry an incidental real defect nobody planted, so counting it either way would be a guess.
|
|
598
|
-
|
|
599
|
-
### The Suite Manifest and Machine-Readable Results
|
|
600
|
-
|
|
601
|
-
This repository's own eval suite manifest registers every suite `eval:all` runs, with its skills, fixtures, ground truth, contract, thresholds, repetition count, CI tier, and the capabilities its runner needs. It also carries a `deferred` list: one entry per skill with no behavioral suite, naming the owner, the missing evidence, and the condition that retires the entry. `eval:all` reads the suite list from that file and refuses to run when a TEA skill appears in neither list.
|
|
602
|
-
|
|
603
|
-
Add `--json <path>` to any of the six harnesses to write a result record alongside the console output:
|
|
604
|
-
|
|
605
|
-
```bash
|
|
606
|
-
npm run eval:all -- --agent codex --json results/eval-all.json
|
|
607
|
-
npm run eval:test-review -- --agent codex --json results/test-review.json
|
|
608
|
-
```
|
|
609
|
-
|
|
610
|
-
The record carries the repository commit, the suite and case IDs, the runner executable and version, the resolved model and parameters, the fixture and prompt digests, the expected and completed repetitions, the measurements, the duration, and the final failure class. A generated JSON Schema in this repository's own dev tree describes it, and every record is validated against that schema before it is written.
|
|
611
|
-
|
|
612
|
-
### CI Usage
|
|
613
|
-
|
|
614
|
-
Run the eval's deterministic checks on every pull request. `npm test` runs all of these along with the rest of the chain:
|
|
615
|
-
|
|
616
|
-
```bash
|
|
617
|
-
npm ci
|
|
618
|
-
npm run test:eval-data # fragment-selection corpus, static
|
|
619
|
-
npm run test:eval-trace-data # trace corpus, static
|
|
620
|
-
npm run test:eval-schemas # manifest against harness constants, and the preflight argv
|
|
621
|
-
npm run test:eval-replay # stored outputs against the scorers
|
|
622
|
-
```
|
|
623
|
-
|
|
624
|
-
Run live evals in a scheduled or manually triggered CI job after installing and authenticating the selected agent CLI:
|
|
625
|
-
|
|
626
|
-
```bash
|
|
627
|
-
npm ci
|
|
628
|
-
npm run eval:all -- --agent codex
|
|
629
|
-
```
|
|
630
|
-
|
|
631
|
-
The eval harnesses use CI-compatible exit codes: `0` means every threshold passed, `1` means a measured result missed a threshold, and `2` means the environment could not run the eval. Live jobs consume model quota and can vary as models change, so keep their result separate from the deterministic pull-request gate until the team chooses to make model quality a required check.
|
|
632
|
-
|
|
633
|
-
TEA applies the same evidence rule to its documentation. Unproven explanations are labeled as hypotheses, and workarounds are labeled as countermeasures. `DESIGN-CRITERIA-REGISTRY.md` records the investigations behind the review rules and scoring decisions.
|
|
634
|
-
|
|
635
|
-
## Extending TEA
|
|
636
|
-
|
|
637
|
-
Custom workflows are still compatible with TEA, but they are no longer implicitly absorbed into TEA core. The supported path is:
|
|
638
|
-
|
|
639
|
-
1. Package the workflow as custom content or a custom module.
|
|
640
|
-
2. Attach it to `bmad-tea` using the agent customization flow.
|
|
641
|
-
3. Install the workflow skill with `npx skills add`, then start a fresh chat so the new menu item is picked up.
|
|
642
|
-
|
|
643
|
-
See [Extend TEA with Custom Workflows](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/customization/extend-tea-with-custom-workflows/) and the BMAD customization guide at [BMAD-METHOD/docs/how-to/customize-bmad.md](https://github.com/bmad-code-org/BMAD-METHOD/blob/main/docs/how-to/customize-bmad.md).
|
|
114
|
+
The npm package includes the workflow skills used by the tools.
|
|
115
|
+
See the [CI examples](./cli/examples/README.md) for test-review integration.
|
|
644
116
|
|
|
645
117
|
## Contributing
|
|
646
118
|
|
|
647
|
-
|
|
648
|
-
|
|
649
|
-
|
|
650
|
-
|
|
651
|
-
## 📦 Publishing TEA to NPM (for Maintainers)
|
|
652
|
-
|
|
653
|
-
TEA uses an automated publish workflow modeled after the main `BMAD-METHOD` repo. It supports:
|
|
654
|
-
|
|
655
|
-
- `next` prereleases published automatically from `main`
|
|
656
|
-
- manual stable releases on the `latest` dist-tag
|
|
657
|
-
- trusted npm publishing (no `NPM_TOKEN` secret)
|
|
658
|
-
- metadata sync for `package.json`, `package-lock.json`, and `.claude-plugin/marketplace.json`
|
|
659
|
-
|
|
660
|
-
### Prerequisites (One-Time Setup)
|
|
661
|
-
|
|
662
|
-
1. **npm Trusted Publishing:**
|
|
663
|
-
|
|
664
|
-
- In npm package settings for `bmad-method-test-architecture-enterprise`, configure Trusted Publishers for this GitHub repository
|
|
665
|
-
- Allow publishes from the `bmad-code-org/bmad-method-test-architecture-enterprise` repo and the `.github/workflows/publish.yaml` workflow
|
|
666
|
-
- GitHub Actions must be able to request an OIDC token (`id-token: write`), which the workflow already does
|
|
667
|
-
|
|
668
|
-
1. **GitHub App Secrets for Stable Releases:**
|
|
669
|
-
|
|
670
|
-
- Add `RELEASE_APP_ID`
|
|
671
|
-
- Add `RELEASE_APP_PRIVATE_KEY`
|
|
672
|
-
- Install the corresponding GitHub App on this repository with contents write access
|
|
673
|
-
- If `main` is protected, ensure the app is allowed to push the release commit and tag
|
|
674
|
-
- These are used only for manual stable releases so the workflow can push the version bump commit and tag back to `main`
|
|
675
|
-
|
|
676
|
-
1. **Verify Package Configuration:**
|
|
677
|
-
|
|
678
|
-
```bash
|
|
679
|
-
# Check package.json settings
|
|
680
|
-
cat package.json | grep -A 3 "publishConfig"
|
|
681
|
-
# Should show: "access": "public"
|
|
682
|
-
if grep -Eq '"private"[[:space:]]*:[[:space:]]*true' package.json; then
|
|
683
|
-
echo '❌ package.json must not set "private": true'
|
|
684
|
-
else
|
|
685
|
-
echo '✅ package.json is publishable ("private": true not present)'
|
|
686
|
-
fi
|
|
687
|
-
```
|
|
688
|
-
|
|
689
|
-
### Release Process
|
|
690
|
-
|
|
691
|
-
#### Option 1: Using npm Scripts (Recommended)
|
|
692
|
-
|
|
693
|
-
From your local terminal after merging to `main`:
|
|
694
|
-
|
|
695
|
-
```bash
|
|
696
|
-
# Publish the next prerelease from current main
|
|
697
|
-
npm run release:next
|
|
698
|
-
|
|
699
|
-
# Publish a stable patch release
|
|
700
|
-
npm run release:patch
|
|
701
|
-
|
|
702
|
-
# Publish a stable minor release
|
|
703
|
-
npm run release:minor
|
|
704
|
-
|
|
705
|
-
# Publish a stable major release
|
|
706
|
-
npm run release:major
|
|
707
|
-
```
|
|
708
|
-
|
|
709
|
-
#### Option 2: Manual Workflow Trigger
|
|
710
|
-
|
|
711
|
-
1. Go to **Actions** tab in GitHub
|
|
712
|
-
2. Click **"Publish"** workflow
|
|
713
|
-
3. Click **"Run workflow"**
|
|
714
|
-
4. Choose the branch to release, typically `main`
|
|
715
|
-
5. Select channel:
|
|
716
|
-
|
|
717
|
-
- `next` for a prerelease publish
|
|
718
|
-
- `latest` for a stable release
|
|
719
|
-
|
|
720
|
-
1. If using `latest`, choose the bump type (`patch`, `minor`, `major`)
|
|
721
|
-
2. Click **"Run workflow"**
|
|
722
|
-
|
|
723
|
-
### What Happens Automatically
|
|
724
|
-
|
|
725
|
-
The workflow performs these steps:
|
|
726
|
-
|
|
727
|
-
1. ✅ **Validation**: Runs the full `npm test` chain: schema checks, install tests, knowledge checks, criteria-to-fragment traceability, enforce-hook coverage, eval data validation, release metadata, changelog, workflow descriptions, linting, markdown linting, and formatting. The chain includes the CLI suite (`npm run test:cli`).
|
|
728
|
-
2. ✅ **Version Bump**:
|
|
729
|
-
|
|
730
|
-
- `next`: derives the next prerelease version and publishes it with dist-tag `next`
|
|
731
|
-
- `latest`: bumps the stable version (`patch`, `minor`, or `major`)
|
|
732
|
-
|
|
733
|
-
1. ✅ **Metadata Sync**: Updates `.claude-plugin/marketplace.json` to match the package version before publishing
|
|
734
|
-
2. ✅ **Publish**: Publishes to npm with provenance enabled
|
|
735
|
-
|
|
736
|
-
- `next` → `npm publish --tag next --provenance`
|
|
737
|
-
- `latest` → `npm publish --tag latest --provenance`
|
|
738
|
-
|
|
739
|
-
1. ✅ **Stable Release Finalization**: For `latest`, creates a version bump commit, tags it, pushes it to `main`, and creates a GitHub Release
|
|
740
|
-
|
|
741
|
-
### Channel Strategy
|
|
742
|
-
|
|
743
|
-
- `next`: prerelease channel for the newest merged changes
|
|
744
|
-
- `latest`: stable channel for intentional releases
|
|
745
|
-
- `patch`: bug fixes, no breaking changes
|
|
746
|
-
- `minor`: new features, backwards compatible
|
|
747
|
-
- `major`: breaking changes
|
|
748
|
-
|
|
749
|
-
**Recommended Release Path:**
|
|
750
|
-
|
|
751
|
-
1. Merge releasable work to `main`
|
|
752
|
-
2. Let `next` publish for early validation
|
|
753
|
-
3. When ready, cut a stable `latest` release via `patch`, `minor`, or `major`
|
|
754
|
-
|
|
755
|
-
### Verify Publication
|
|
756
|
-
|
|
757
|
-
**Check NPM:**
|
|
758
|
-
|
|
759
|
-
```bash
|
|
760
|
-
npm view bmad-method-test-architecture-enterprise
|
|
761
|
-
npm view bmad-method-test-architecture-enterprise dist-tags
|
|
762
|
-
```
|
|
763
|
-
|
|
764
|
-
**Install TEA:**
|
|
765
|
-
|
|
766
|
-
```bash
|
|
767
|
-
npx skills add bmad-code-org/bmad-method-test-architecture-enterprise
|
|
768
|
-
```
|
|
769
|
-
|
|
770
|
-
Then run `bmad setup tea` in the assistant chat.
|
|
771
|
-
|
|
772
|
-
**Test Workflows:** type these in the assistant chat, not in a shell.
|
|
773
|
-
|
|
774
|
-
```text
|
|
775
|
-
/bmad-tea # load the agent persona and menu
|
|
776
|
-
/bmad-testarch-test-design # run a workflow directly
|
|
777
|
-
```
|
|
778
|
-
|
|
779
|
-
Hosts that use dollar-prefixed skills use `$` in place of `/`.
|
|
780
|
-
|
|
781
|
-
### Rollback a Release (if needed)
|
|
782
|
-
|
|
783
|
-
If you need to unpublish a version:
|
|
784
|
-
|
|
785
|
-
```bash
|
|
786
|
-
# Unpublish specific version (within 72 hours)
|
|
787
|
-
npm unpublish bmad-method-test-architecture-enterprise@1.13.2-next.0
|
|
788
|
-
|
|
789
|
-
# Deprecate version (preferred for older releases)
|
|
790
|
-
npm deprecate bmad-method-test-architecture-enterprise@1.13.2-next.0 "Use version X.Y.Z instead"
|
|
791
|
-
```
|
|
792
|
-
|
|
793
|
-
### Troubleshooting
|
|
794
|
-
|
|
795
|
-
**Trusted publishing failed:**
|
|
796
|
-
|
|
797
|
-
- Verify npm Trusted Publishing is configured for this repository and workflow
|
|
798
|
-
- Verify the workflow has `id-token: write`
|
|
799
|
-
- Confirm the publish is running from the canonical repository, not a fork
|
|
800
|
-
|
|
801
|
-
**"Package already exists":**
|
|
802
|
-
|
|
803
|
-
- Check if package name is already taken on NPM
|
|
804
|
-
- Update `name` in `package.json` if needed
|
|
805
|
-
|
|
806
|
-
**"Version push failed":**
|
|
807
|
-
|
|
808
|
-
- Verify `RELEASE_APP_ID` and `RELEASE_APP_PRIVATE_KEY` are configured
|
|
809
|
-
- Verify the GitHub App is installed on this repository with contents write access
|
|
810
|
-
- If branch protection is enabled on `main`, verify the app is allowed to push the release commit and tag
|
|
811
|
-
|
|
812
|
-
**"Tests failed":**
|
|
813
|
-
|
|
814
|
-
- Fix failing tests before release
|
|
815
|
-
- Run `npm test` locally to verify
|
|
816
|
-
|
|
817
|
-
**"Git push failed (protected branch)":**
|
|
818
|
-
|
|
819
|
-
- This is not expected once the release GitHub App is configured correctly
|
|
820
|
-
- Verify branch protection allows the app to push the release commit and tag
|
|
821
|
-
- If needed, create the GitHub Release manually after resolving the app permissions
|
|
822
|
-
|
|
823
|
-
### Release Checklist
|
|
824
|
-
|
|
825
|
-
Before releasing:
|
|
826
|
-
|
|
827
|
-
- [ ] All tests passing: `npm test`
|
|
828
|
-
- [ ] Documentation up to date
|
|
829
|
-
- [ ] CHANGELOG.md updated
|
|
830
|
-
- [ ] No uncommitted changes
|
|
831
|
-
- [ ] On `main` branch
|
|
832
|
-
- [ ] npm Trusted Publishing configured
|
|
833
|
-
- [ ] `RELEASE_APP_ID` and `RELEASE_APP_PRIVATE_KEY` configured
|
|
834
|
-
- [ ] Package name available on NPM
|
|
835
|
-
|
|
836
|
-
After releasing:
|
|
837
|
-
|
|
838
|
-
- [ ] Verify NPM publication: `npm view bmad-method-test-architecture-enterprise`
|
|
839
|
-
- [ ] Test installation: `npx skills add bmad-code-org/bmad-method-test-architecture-enterprise`, then `bmad setup tea`
|
|
840
|
-
- [ ] Verify workflows work
|
|
841
|
-
- [ ] Check GitHub Release created
|
|
842
|
-
- [ ] Monitor for issues
|
|
843
|
-
|
|
844
|
-
---
|
|
119
|
+
[CONTRIBUTING.md](./CONTRIBUTING.md) covers local checks, pull requests, and publishing.
|
|
120
|
+
[How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) explains the test and evaluation layers.
|
|
121
|
+
The [test suite guide](./test/README.md) has commands for running individual suites and live evaluations.
|
|
845
122
|
|
|
846
123
|
## Community
|
|
847
124
|
|
|
848
|
-
- [Discord](https://discord.gg/gk8jAdXWmj)
|
|
849
|
-
- [
|
|
850
|
-
- [
|
|
851
|
-
- [
|
|
852
|
-
|
|
853
|
-
## Support BMad
|
|
125
|
+
- [Discord](https://discord.gg/gk8jAdXWmj) for help and discussion.
|
|
126
|
+
- [GitHub Issues](https://github.com/bmad-code-org/bmad-method-test-architecture-enterprise/issues) for bugs and feature requests.
|
|
127
|
+
- [YouTube](https://youtube.com/@BMadCode) for tutorials.
|
|
128
|
+
- [BMad website](https://bmadcode.com) and [X](https://x.com/BMadCode).
|
|
854
129
|
|
|
855
|
-
|
|
130
|
+
You can support BMad by starring the repository, [buying a coffee](https://buymeacoffee.com/bmad), or contacting [contact@bmadcode.com](mailto:contact@bmadcode.com) about sponsorship.
|
|
856
131
|
|
|
857
132
|
## License
|
|
858
133
|
|
|
859
|
-
|
|
134
|
+
[MIT](./LICENSE).
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "2.0.1-next.
|
|
4
|
+
"version": "2.0.1-next.1",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
|
@@ -66,7 +66,7 @@ sessions:
|
|
|
66
66
|
duration: "45 min"
|
|
67
67
|
difficulty: intermediate
|
|
68
68
|
objective: "Audit quality and ensure traceability"
|
|
69
|
-
description: "Test Review (5
|
|
69
|
+
description: "Test Review (rubric 5.0 scoring), Trace workflow, quality metrics"
|
|
70
70
|
recommended_for:
|
|
71
71
|
- intermediate
|
|
72
72
|
- experienced
|
|
@@ -169,15 +169,15 @@ session-05-atdd-automate:
|
|
|
169
169
|
session-06-quality-trace:
|
|
170
170
|
passing_score: 70
|
|
171
171
|
questions:
|
|
172
|
-
- id: q1-
|
|
173
|
-
question: "
|
|
172
|
+
- id: q1-review-scoring
|
|
173
|
+
question: "How does rubric 5.0 calculate the test-review score?"
|
|
174
174
|
options:
|
|
175
|
-
A: "
|
|
176
|
-
B: "
|
|
177
|
-
C: "
|
|
178
|
-
D: "
|
|
175
|
+
A: "Average five dimension scores"
|
|
176
|
+
B: "Start at 100, apply severity deductions and up to 25 bonus points, then apply the highest-severity cap"
|
|
177
|
+
C: "Count the number of tests"
|
|
178
|
+
D: "Use line coverage alone"
|
|
179
179
|
correct: B
|
|
180
|
-
explanation: "
|
|
180
|
+
explanation: "Registry findings deduct 10/5/2/1 points by severity. Bonuses total at most 25. The effective score is capped at 69/79/89/99 when CRITICAL/HIGH/MEDIUM/LOW findings are present."
|
|
181
181
|
|
|
182
182
|
- id: q2-release-gate
|
|
183
183
|
question: "When should the Trace workflow gate decision be RED (block release)?"
|
|
@@ -147,7 +147,7 @@ sessions:
|
|
|
147
147
|
- test-review
|
|
148
148
|
- trace
|
|
149
149
|
key_concepts:
|
|
150
|
-
- "
|
|
150
|
+
- "Registry criteria, severity deductions, bonuses, and score caps"
|
|
151
151
|
- "Quality scoring (0-100)"
|
|
152
152
|
- "Requirements traceability"
|
|
153
153
|
- "Release gate decisions"
|
|
@@ -49,7 +49,7 @@ Dana Whitfield has demonstrated proficiency in:
|
|
|
49
49
|
- ✅ **Architecture Patterns:** Fixture composition, network-first patterns, data factories, step-file architecture
|
|
50
50
|
- ✅ **Test Design:** Risk assessment (Probability × Impact), coverage planning, test levels framework
|
|
51
51
|
- ✅ **Test Development:** ATDD red-green TDD approach, test automation, API testing patterns
|
|
52
|
-
- ✅ **Quality Assurance:** Test review (5
|
|
52
|
+
- ✅ **Quality Assurance:** Test review (rubric 5.0 scoring), traceability, release gates, quality metrics
|
|
53
53
|
- ✅ **Advanced Techniques:** 47 of 59 knowledge fragments explored, with deep dives on flaky test diagnosis and contract testing, plus Playwright Utils integration
|
|
54
54
|
|
|
55
55
|
---
|
|
@@ -93,8 +93,8 @@ Present this content (mostly autonomous, clear and educational):
|
|
|
93
93
|
|
|
94
94
|
TEA is a comprehensive test architecture framework that provides:
|
|
95
95
|
|
|
96
|
-
- **
|
|
97
|
-
- **
|
|
96
|
+
- **10 Workflows:** Teach Me Testing, Test Design, Framework, CI, ATDD, Automate, Test Review, NFR Evidence Audit, Trace, Evaluate
|
|
97
|
+
- **59 Knowledge Fragments:** Distilled expertise on patterns, best practices, Playwright Utils
|
|
98
98
|
- **Quality Standards:** Definition of Done with execution limits (no flaky tests, no hard waits, etc.)
|
|
99
99
|
- **Risk-Based Testing:** P0-P3 matrix for prioritizing test coverage
|
|
100
100
|
|
|
@@ -102,7 +102,7 @@ TEA is a comprehensive test architecture framework that provides:
|
|
|
102
102
|
Testing knowledge doesn't scale through manual teaching. TEA makes testing expertise accessible through:
|
|
103
103
|
|
|
104
104
|
- Structured workflows that guide you step-by-step
|
|
105
|
-
- Documentation
|
|
105
|
+
- Documentation organized by type (tutorials, how-to, explanation, reference)
|
|
106
106
|
- Knowledge fragments for just-in-time learning
|
|
107
107
|
- Online resources: <https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/>
|
|
108
108
|
|
|
@@ -50,7 +50,7 @@ To teach test quality auditing and requirements traceability using Test Review a
|
|
|
50
50
|
**What you'll learn:**
|
|
51
51
|
|
|
52
52
|
- Test Review workflow (quality scoring)
|
|
53
|
-
-
|
|
53
|
+
- Registry criteria, severity deductions, bonuses, and score caps
|
|
54
54
|
- Trace workflow (requirements traceability)
|
|
55
55
|
- Release gate decisions
|
|
56
56
|
|
|
@@ -66,39 +66,23 @@ Set session-06-quality-trace `status: 'in-progress'`.
|
|
|
66
66
|
|
|
67
67
|
**Purpose:** Audit test quality with 0-100 scoring
|
|
68
68
|
|
|
69
|
-
|
|
69
|
+
#### Rubric 5.0: criteria and severity scoring
|
|
70
70
|
|
|
71
|
-
|
|
71
|
+
The review applies the 35 rows in the test-review criteria registry. Each finding names its row, evidence, and fixed severity.
|
|
72
|
+
Start at 100 and deduct for every finding:
|
|
72
73
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
74
|
+
| Severity | Deduction | Score cap when present |
|
|
75
|
+
| -------- | --------- | ---------------------- |
|
|
76
|
+
| CRITICAL | 10 | 69 |
|
|
77
|
+
| HIGH | 5 | 79 |
|
|
78
|
+
| MEDIUM | 2 | 89 |
|
|
79
|
+
| LOW | 1 | 99 |
|
|
76
80
|
|
|
77
|
-
|
|
81
|
+
Five bonus categories can each add 5 points: fixture setup, data factories, network-first ordering, isolation, and stable test IDs. Each bonus must hold across every reviewed file, for a maximum of 25 points.
|
|
82
|
+
Clamp the raw score to 0–100, then apply the cap for the highest finding severity.
|
|
83
|
+
A CRITICAL finding recommends Block; a HIGH finding or score below 70 recommends Request Changes. MEDIUM or LOW findings at 70 or above recommend Approve with Comments. A review with no findings recommends Approve.
|
|
78
84
|
|
|
79
|
-
|
|
80
|
-
- No shared state
|
|
81
|
-
- Parallelizable
|
|
82
|
-
|
|
83
|
-
#### 3. Assertions (0-100)
|
|
84
|
-
|
|
85
|
-
- Correct checks for expected behavior
|
|
86
|
-
- Meaningful assertions (not just presence)
|
|
87
|
-
- Fails for the right reasons
|
|
88
|
-
|
|
89
|
-
#### 4. Structure (0-100)
|
|
90
|
-
|
|
91
|
-
- Readable test code
|
|
92
|
-
- Clear organization and naming
|
|
93
|
-
- Minimal duplication
|
|
94
|
-
|
|
95
|
-
#### 5. Performance (0-100)
|
|
96
|
-
|
|
97
|
-
- Test execution speed
|
|
98
|
-
- Resource usage
|
|
99
|
-
- Parallel efficiency
|
|
100
|
-
|
|
101
|
-
**Overall Score = Average of 5 dimensions.**
|
|
85
|
+
Example: one HIGH finding deducts 5 points. The raw score is 95, and the HIGH cap makes the effective score 79.
|
|
102
86
|
|
|
103
87
|
{Role-adapted example}
|
|
104
88
|
|
|
@@ -151,11 +135,11 @@ Set session-06-quality-trace `status: 'in-progress'`.
|
|
|
151
135
|
|
|
152
136
|
### 6. Quiz (3 questions)
|
|
153
137
|
|
|
154
|
-
**Q1:** "
|
|
155
|
-
A)
|
|
156
|
-
B)
|
|
157
|
-
C)
|
|
158
|
-
D)
|
|
138
|
+
**Q1:** "How does rubric 5.0 calculate the test-review score?
|
|
139
|
+
A) Average five dimension scores
|
|
140
|
+
B) Start at 100, apply severity deductions and up to 25 bonus points, then apply the highest-severity cap
|
|
141
|
+
C) Count the number of tests
|
|
142
|
+
D) Use line coverage alone"
|
|
159
143
|
|
|
160
144
|
Correct: B
|
|
161
145
|
|
|
@@ -204,4 +188,4 @@ Return to {nextStepFile}.
|
|
|
204
188
|
|
|
205
189
|
## 🚨 SUCCESS METRICS
|
|
206
190
|
|
|
207
|
-
✅ Test Review and Trace taught,
|
|
191
|
+
✅ Test Review and Trace taught, severity scoring explained, quiz passed, notes generated, returned to hub.
|
|
@@ -174,7 +174,7 @@ average_score: { average_score }
|
|
|
174
174
|
- ✅ **Architecture Patterns:** Fixture composition, network-first patterns, data factories, step-file architecture
|
|
175
175
|
- ✅ **Test Design:** Risk assessment (Probability × Impact), coverage planning, test levels framework
|
|
176
176
|
- ✅ **Test Development:** ATDD red-green TDD approach, test automation, API testing patterns
|
|
177
|
-
- ✅ **Quality Assurance:** Test review (5
|
|
177
|
+
- ✅ **Quality Assurance:** Test review (rubric 5.0 scoring), traceability, release gates, quality metrics
|
|
178
178
|
- ✅ **Advanced Techniques:** Knowledge fragments explored, Playwright Utils integration
|
|
179
179
|
|
|
180
180
|
---
|