bmad-method-test-architecture-enterprise 2.0.2 → 2.0.3-next.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -3
- package/README.md +3 -3
- package/cli/evaluate.js +25 -2
- package/cli/lib/evaluate/preflight.js +77 -1
- package/cli/lib/evaluate/run.js +43 -4
- package/cli/lib/evaluate/status.js +184 -0
- package/package.json +3 -2
- package/skills/bmad-tea/SKILL.md +2 -2
- package/skills/bmad-tea/customize.toml +4 -4
- package/skills/bmad-teach-me-testing/steps-c/step-04-session-01.md +2 -2
- package/skills/bmad-teach-me-testing/steps-c/step-04-session-03.md +1 -1
- package/skills/bmad-teach-me-testing/steps-c/step-04-session-05.md +2 -2
- package/skills/bmad-testarch-automate/red/instructions.md +1 -2
- package/skills/bmad-testarch-automate/resources/test-generation-routing.md +17 -21
- package/skills/bmad-testarch-evaluate/SKILL.md +5 -3
- package/skills/bmad-testarch-evaluate/assets/inspection-record.md +1 -1
- package/skills/bmad-testarch-evaluate/references/authoring-routing.md +29 -0
- package/skills/bmad-testarch-evaluate/references/ci.md +5 -5
- package/skills/bmad-testarch-evaluate/references/run.md +19 -0
- package/skills/bmad-testarch-framework/SKILL.md +2 -2
- package/skills/bmad-testarch-framework/checklist.md +4 -4
- package/skills/bmad-testarch-framework/ci/checklist.md +2 -2
- package/skills/bmad-testarch-framework/ci/instructions.md +1 -1
- package/skills/bmad-testarch-framework/resources/setup-routing.md +4 -4
- package/skills/bmad-testarch-test-design/checklist.md +6 -6
- package/skills/bmad-testarch-test-design/resources/test-design-epic-3.example.md +1 -1
- package/skills/bmad-testarch-test-design/test-design-template.md +1 -1
- package/skills/bmad-testarch-trace/checklist.md +2 -2
- package/skills/bmad-testarch-trace/steps-c/step-04-analyze-gaps.md +1 -1
- package/skills/bmad-testarch-trace/trace-template.md +1 -1
- package/skills/bmod-tea/bmod.toml +2 -4
- package/skills/bmod-tea/help/help.md +1 -3
- package/skills/bmod-tea/help/integrations.md +1 -1
- package/skills/bmod-tea/knowledge/burn-in.md +1 -1
- package/skills/bmod-tea/migration-1.toml +1 -1
- package/skills/bmad-testarch-atdd/SKILL.md +0 -16
- package/skills/bmad-testarch-atdd/bmod.toml +0 -4
- package/skills/bmad-testarch-atdd/customize.toml +0 -38
- package/skills/bmad-testarch-atdd/workflow.yaml +0 -40
- package/skills/bmad-testarch-ci/SKILL.md +0 -14
- package/skills/bmad-testarch-ci/bmod.toml +0 -4
- package/skills/bmad-testarch-ci/customize.toml +0 -43
- package/skills/bmad-testarch-ci/workflow.yaml +0 -44
package/skills/bmad-tea/SKILL.md
CHANGED
|
@@ -78,11 +78,11 @@ Dispatch on a clear match by invoking the item's `skill` or executing its `promp
|
|
|
78
78
|
|
|
79
79
|
### Framework and CI Setup Routing
|
|
80
80
|
|
|
81
|
-
TF invokes `bmad-testarch-framework
|
|
81
|
+
TF invokes `bmad-testarch-framework` for framework setup. CI invokes the same skill with `setup_scope = ci`. Preserve both codes, custom menu overrides, requested scope and operation. A request for framework and pipeline setup routes to TF with `setup_scope = both`. The shared router asks “Do you want CI too?” once when setup leaves CI intent unclear.
|
|
82
82
|
|
|
83
83
|
### Test Generation Routing
|
|
84
84
|
|
|
85
|
-
AT invokes `bmad-testarch-
|
|
85
|
+
AT invokes `bmad-testarch-automate` with `test_mode = red`. TA invokes the same skill with expand as its default. Preserve both menu codes, custom menu overrides, the requested operation and any explicit red or expand intent. Red writes acceptance tests before implementation and checks the intended failure; expand adds passing coverage for implemented code. Create runs run-and-heal by default in both modes. Validate and Edit report their own checks without repair. A mixed implemented/unimplemented request still needs the deciding scope or sequence when the user has supplied neither.
|
|
86
86
|
|
|
87
87
|
### Routing Ambiguity Boundaries
|
|
88
88
|
|
|
@@ -74,13 +74,13 @@ skill = "bmad-testarch-framework"
|
|
|
74
74
|
|
|
75
75
|
[[agent.menu]]
|
|
76
76
|
code = "CI"
|
|
77
|
-
description = "Continuous Integration: scaffold CI/CD quality pipeline through
|
|
78
|
-
skill = "bmad-testarch-
|
|
77
|
+
description = "Continuous Integration: scaffold a CI/CD quality pipeline through Framework CI scope"
|
|
78
|
+
skill = "bmad-testarch-framework"
|
|
79
79
|
|
|
80
80
|
[[agent.menu]]
|
|
81
81
|
code = "AT"
|
|
82
|
-
description = "ATDD: generate and verify failing acceptance tests
|
|
83
|
-
skill = "bmad-testarch-
|
|
82
|
+
description = "ATDD: generate and verify failing acceptance tests through Automate red mode"
|
|
83
|
+
skill = "bmad-testarch-automate"
|
|
84
84
|
|
|
85
85
|
[[agent.menu]]
|
|
86
86
|
code = "TA"
|
|
@@ -93,8 +93,8 @@ Present this content (mostly autonomous, clear and educational):
|
|
|
93
93
|
|
|
94
94
|
TEA is a comprehensive test architecture framework that provides:
|
|
95
95
|
|
|
96
|
-
- **8 Workflows:** Teach Me Testing, Test Design, Framework and CI Setup, Automation (red and expand modes), Test Review, NFR Evidence Audit, Trace, Evaluate. The
|
|
97
|
-
- **Framework and CI setup:** One `bmad-testarch-framework` skill handles framework only, CI only, or both.
|
|
96
|
+
- **8 Workflows:** Teach Me Testing, Test Design, Framework and CI Setup, Automation (red and expand modes), Test Review, NFR Evidence Audit, Trace, Evaluate. The agent's `AT` and `CI` menu codes remain available for Automate red mode and Framework CI scope.
|
|
97
|
+
- **Framework and CI setup:** One `bmad-testarch-framework` skill handles framework only, CI only, or both. Ask that skill for CI setup, or use the agent's `CI` menu code. TEA infers the scope from your prompt and asks "Do you want CI too?" once when unclear in an interactive session. An unattended request with unclear scope sets up the framework only. Create, Resume, Validate, and Edit choose the operation.
|
|
98
98
|
- **59 Knowledge Fragments:** Distilled expertise on patterns, best practices, Playwright Utils
|
|
99
99
|
- **Quality Standards:** Definition of Done with execution limits (no flaky tests, no hard waits, etc.)
|
|
100
100
|
- **Risk-Based Testing:** P0-P3 matrix for prioritizing test coverage
|
|
@@ -220,7 +220,7 @@ The prompt determines scope; if CI scope is unclear in an interactive session, a
|
|
|
220
220
|
Framework scope covers stack selection, fixtures, scaffold, scripts, and the write-time quality hook.
|
|
221
221
|
CI scope covers pipeline generation, quality gates, and evaluation-plan rendering.
|
|
222
222
|
For both, agree the stack, framework, and test commands first. Scaffold and pipeline generation can then run in parallel, with joint validation after both finish. A CI request without a framework offers the framework phase first.
|
|
223
|
-
`bmad-testarch-
|
|
223
|
+
`bmad-testarch-framework` handles CI setup when requested; the agent's `CI` code selects that scope, while `TF` starts framework setup.
|
|
224
224
|
Create starts setup, Validate reports checks without repairing outputs, and Edit revises the selected outputs and checks those changes. Resume continues an interrupted run, whatever its scope and operation.
|
|
225
225
|
|
|
226
226
|
### 7. Quiz (3 questions)
|
|
@@ -71,7 +71,7 @@ Save the updated progress file.
|
|
|
71
71
|
|
|
72
72
|
**TDD Red Phase:** Write failing tests FIRST
|
|
73
73
|
|
|
74
|
-
**Automation red mode:** `/bmad-testarch-
|
|
74
|
+
**Automation red mode:** Ask `/bmad-testarch-automate` for acceptance tests before implementation, or use the agent's `AT` code to select red.
|
|
75
75
|
|
|
76
76
|
**Create sequence:**
|
|
77
77
|
|
|
@@ -97,7 +97,7 @@ Save the updated progress file.
|
|
|
97
97
|
|
|
98
98
|
**Purpose:** Generate tests for existing features
|
|
99
99
|
|
|
100
|
-
**Automation expand mode:** `/bmad-testarch-automate` and `TA` select expand by default. An explicit mode in the prompt selects red or expand. Unattended runs
|
|
100
|
+
**Automation expand mode:** `/bmad-testarch-automate` and `TA` select expand by default. An explicit mode in the prompt selects red or expand. Unattended runs use expand when mode is unclear.
|
|
101
101
|
|
|
102
102
|
**Create sequence:**
|
|
103
103
|
|
|
@@ -26,8 +26,7 @@ This workflow uses **step-file architecture**:
|
|
|
26
26
|
|
|
27
27
|
This instruction file owns red mode after canonical activation.
|
|
28
28
|
Set `test_mode = red` when a direct runner loads this file without the shared router; keep `{skill-root}` canonical.
|
|
29
|
-
Resolve red customization
|
|
30
|
-
When that sibling is absent, merge `{skill-root}/red/customize.toml` with `_bmad/custom/bmad-testarch-atdd.toml` and `.user.toml` manually and retain its workflow block.
|
|
29
|
+
Resolve red customization by merging `{skill-root}/red/customize.toml` with `_bmad/custom/bmad-testarch-atdd.toml` and `.user.toml` manually and retain its workflow block.
|
|
31
30
|
Entry activation and operation routing remain in `{skill-root}/SKILL.md`.
|
|
32
31
|
|
|
33
32
|
### 1. Configuration Loading
|
|
@@ -7,33 +7,31 @@ These are separate from the worker execution mode (`tea_execution_mode`) and the
|
|
|
7
7
|
## 1. Choose the Operation and Mode
|
|
8
8
|
|
|
9
9
|
Keep the original request and supplied files.
|
|
10
|
-
|
|
10
|
+
Use `bmad-testarch-automate` as `test_entry`. Preserve a legacy `testEntry` value when resuming an existing checkpoint. An AT menu invocation presets red mode; a single explicit user mode overrides that preset.
|
|
11
11
|
Resolve explicit Create, Resume, Validate or Edit wording first.
|
|
12
12
|
A request to generate tests means Create.
|
|
13
13
|
In a headless or autonomous invocation, an unspecified operation defaults to Create without a menu.
|
|
14
14
|
|
|
15
15
|
Start with `test_mode_defaulted = false`.
|
|
16
16
|
A single explicit `red mode` or `expand mode` is authoritative.
|
|
17
|
-
Otherwise
|
|
17
|
+
Otherwise an AT menu invocation selects red. Acceptance tests before implementation, `write acceptance tests`, `acceptance test scaffolds`, ATDD, or the TDD red phase also selects red; coverage expansion, tests for existing code, or automation selects expand.
|
|
18
18
|
Treat the invoked command name as an entry default, separate from the user's task wording: mentioning `/bmad-testarch-automate` in an acceptance-test request cannot count as an expand signal.
|
|
19
|
-
When no task mode signal is present,
|
|
20
|
-
Interactive AT, TA and slash-command invocations with no task wording use those entry defaults without a mode question.
|
|
19
|
+
When no task mode signal is present, default to expand.
|
|
21
20
|
Ask once which mode to use only when explicit task mode signals conflict in an interactive run, including a request that names both `red mode` and `expand mode`.
|
|
22
|
-
A headless or autonomous run with conflicting signals
|
|
23
|
-
For
|
|
24
|
-
An explicit mode in the user's request overrides the
|
|
21
|
+
A headless or autonomous run with conflicting signals defaults to expand without a question.
|
|
22
|
+
For a defaulted mode, set `test_mode_defaulted = true` and state the selected default in the final summary.
|
|
23
|
+
An explicit mode in the user's request overrides the default.
|
|
25
24
|
|
|
26
25
|
The following routing examples apply before customization activation:
|
|
27
26
|
|
|
28
|
-
| Invocation
|
|
29
|
-
|
|
|
30
|
-
|
|
|
31
|
-
|
|
|
32
|
-
|
|
|
33
|
-
|
|
|
34
|
-
|
|
|
35
|
-
|
|
|
36
|
-
| TA or `/bmad-testarch-automate` | red and expand task signals | headless | expand (entry default) | no |
|
|
27
|
+
| Invocation | Task signal | Run mode | Selected mode | Mode question |
|
|
28
|
+
| ------------------------ | --------------------------- | ----------------------- | ---------------- | ------------- |
|
|
29
|
+
| `bmad-testarch-automate` | none | interactive | expand (default) | no |
|
|
30
|
+
| `bmad-testarch-automate` | lets write acceptance tests | interactive or headless | red | no |
|
|
31
|
+
| `bmad-testarch-automate` | acceptance test scaffolds | interactive or headless | red | no |
|
|
32
|
+
| `bmad-testarch-automate` | red and expand task signals | interactive | unresolved | ask once |
|
|
33
|
+
| `bmad-testarch-automate` | red and expand task signals | headless | expand (default) | no |
|
|
34
|
+
| `bmad-tea` AT menu | none | interactive or headless | red | no |
|
|
37
35
|
|
|
38
36
|
For Resume with an exact checkpoint path, resolve it against the project root, verify that it exists and is readable, and retain it as `resume_checkpoint_path`. A missing supplied file stops before activation; do not search for a replacement. The checkpoint path itself is not a new target scope.
|
|
39
37
|
Infer the original mode from that supplied checkpoint's `testMode` or its owning artifact folder: `atdd/` is red and `automate/` is expand; legacy `atdd-checklist-*.md` is red and `automation-summary.md` is expand.
|
|
@@ -45,12 +43,10 @@ Do not restart Create or execute a completed generation step on an explicit Resu
|
|
|
45
43
|
|
|
46
44
|
## 2. Choose Exactly One Customization Surface
|
|
47
45
|
|
|
48
|
-
For red set `workflow-skill-root = {skill-root}
|
|
46
|
+
For red set `workflow-skill-root = {skill-root}/red` and `workflow-skill-name = bmad-testarch-atdd` to preserve existing customization filenames.
|
|
49
47
|
For expand set `workflow-skill-root = {skill-root}` and `workflow-skill-name = bmad-testarch-automate`.
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
Canonical-only installs can therefore run red with existing ATDD overrides.
|
|
53
|
-
Record `workflow_customization_manual = true` for this fallback and retain the resolved workflow block for its terminal hook.
|
|
48
|
+
Resolve `{skill-root}/red/customize.toml` plus `_bmad/custom/bmad-testarch-atdd.toml` and `.user.toml` manually with the activation merge rules; the resolver's directory-name inference cannot resolve this customization namespace from `red/`.
|
|
49
|
+
Record `workflow_customization_manual = true` and retain the resolved workflow block for its terminal hook.
|
|
54
50
|
Keep the resolver's workflow namespace and structural merge rules intact.
|
|
55
51
|
Resolve only the selected surface's defaults, team file and user file.
|
|
56
52
|
Red uses `_bmad/custom/bmad-testarch-atdd.toml` and `.user.toml`; expand uses `_bmad/custom/bmad-testarch-automate.toml` and `.user.toml`.
|
|
@@ -63,7 +63,9 @@ Activation is complete. Begin the workflow below.
|
|
|
63
63
|
|
|
64
64
|
## Workflow
|
|
65
65
|
|
|
66
|
-
Evaluate is one continuous loop over twelve stages, run inline rather than as separate step files: each stage's craft lives in its own `references/` guide, loaded when that stage is reached. Work under `{tea_evaluations_folder}` unless the adopter names another location. On resume, read the existing inspection record, `evaluation.json`, any `requirements.md`, any `<evaluation-folder>/ci/evaluation-ci-plan.json`, and any `{test_artifacts}/evaluate/<evaluationId>/gap-report.md` before choosing a stage. Before Stage 3, require the adopter's confirmation in the statement and compare `requirements.digest` with eval-quality's `digestBytes` over the committed file; otherwise return to intake. Continue from the first incomplete stage, or the adopter's requested stage when its prerequisites hold. `assets/README.md` says how to fill each template under `assets/`.
|
|
66
|
+
Evaluate is one continuous loop over twelve stages, run inline rather than as separate step files: each stage's craft lives in its own `references/` guide, loaded when that stage is reached. Work under `{tea_evaluations_folder}` unless the adopter names another location. On resume, read the existing inspection record, `evaluation.json`, any `requirements.md`, any `authoring-log.md`, any `<evaluation-folder>/ci/evaluation-ci-plan.json`, and any `{test_artifacts}/evaluate/<evaluationId>/gap-report.md` before choosing a stage. Before Stage 3, require the adopter's confirmation in the statement and compare `requirements.digest` with eval-quality's `digestBytes` over the committed file; otherwise return to intake. Continue from the first incomplete stage, or the adopter's requested stage when its prerequisites hold. `assets/README.md` says how to fill each template under `assets/`.
|
|
67
|
+
|
|
68
|
+
Read `references/authoring-routing.md` before delegating any authoring work. Use its bounded worker tasks and authoring ledger throughout the loop when the host supports subagents.
|
|
67
69
|
|
|
68
70
|
Never improvise the stage's craft yourself, and never compute a verdict, score, or pass/fail decision outside `eval-quality`'s own CLI (AD-6): a verdict this skill computed itself would not be one `eval-quality` sealed.
|
|
69
71
|
|
|
@@ -97,7 +99,7 @@ Before the first `tea-evaluate check`, load `references/mutation.md` and author
|
|
|
97
99
|
|
|
98
100
|
For an adopter installation, run `npm exec --prefix {tea_evaluations_folder} -- tea-evaluate check --evaluation <evaluation-folder>`, then `npm exec --prefix {tea_evaluations_folder} -- eval-quality compile --in <evaluation-folder>/contract.json --out <evaluation-folder>/compiled-contract.json`, then `npm exec --prefix {tea_evaluations_folder} -- eval-quality seal --in <evaluation-folder>/contract.json --out <evaluation-folder>/sealed-brief.json`. For TeA's own package, run `node cli/evaluate.js check --evaluation <evaluation-folder>`, then `./node_modules/.bin/eval-quality compile --in <evaluation-folder>/contract.json --out <evaluation-folder>/compiled-contract.json`, then `./node_modules/.bin/eval-quality seal --in <evaluation-folder>/contract.json --out <evaluation-folder>/sealed-brief.json` from the repository root. The local CLI and binary use that root's one eval-quality installation. Stop at the first nonzero exit. Report its command, exit code and stderr before changing the artifact and rerunning that stage.
|
|
99
101
|
|
|
100
|
-
After seal succeeds, copy `assets/evaluation-folder.gitignore` to `<evaluation-folder>/.gitignore` before the first preflight creates `runs/`. For an adopter installation, run `npm exec --prefix {tea_evaluations_folder} -- tea-evaluate preflight --evaluation <evaluation-folder> --partition development`; for TeA's own package, run `node cli/evaluate.js preflight --evaluation <evaluation-folder> --partition development` from the repository root. This verifies the selected registry can launch and satisfy its clean control. The flag is required only when `evaluation.json` declares a `partitionPlan`, and a preflight with no `--partition` over one exits 64. Held-out preflight (`--partition held-out`) runs only after the development review. Stop and report the command, exit code and stderr on failure; complete adapter wiring only after preflight succeeds.
|
|
102
|
+
After seal succeeds, copy `assets/evaluation-folder.gitignore` to `<evaluation-folder>/.gitignore` before the first preflight creates `runs/`. Before a live preflight, follow the Git isolation procedure in `references/run.md`; Git worktrees share refs even when their working files are separate. For an adopter installation, run `npm exec --prefix {tea_evaluations_folder} -- tea-evaluate preflight --evaluation <evaluation-folder> --partition development`; for TeA's own package, run `node cli/evaluate.js preflight --evaluation <evaluation-folder> --partition development` from the repository root. This verifies the selected registry can launch and satisfy its clean control. The flag is required only when `evaluation.json` declares a `partitionPlan`, and a preflight with no `--partition` over one exits 64. Held-out preflight (`--partition held-out`) runs only after the development review. Stop and report the command, exit code and stderr on failure; complete adapter wiring only after preflight succeeds.
|
|
101
103
|
|
|
102
104
|
### Stage 7: Evaluator
|
|
103
105
|
|
|
@@ -121,7 +123,7 @@ Interpret a weak result and close the gap. Load `references/gaps.md`.
|
|
|
121
123
|
|
|
122
124
|
### Stage 12: CI
|
|
123
125
|
|
|
124
|
-
Inspect the adopter's repository, place each check in a tier, write `<evaluation-folder>/ci/evaluation-ci-plan.json` and hand it to `bmad-testarch-framework` with CI setup scope.
|
|
126
|
+
Inspect the adopter's repository, place each check in a tier, write `<evaluation-folder>/ci/evaluation-ci-plan.json` and hand it to `bmad-testarch-framework` with CI setup scope. Load `references/ci.md`.
|
|
125
127
|
|
|
126
128
|
## On Complete
|
|
127
129
|
|
|
@@ -51,6 +51,6 @@ Stage 12 appends this section as each inspection finishes.
|
|
|
51
51
|
- Release flow: `release.yml` on `v*` tags, with `docs/RELEASING.md` naming the tag as the gate.
|
|
52
52
|
- Risk profile: one `critical` behavior (`contract.json`), about forty model calls per live run, nightly and release accepted and per pull request not (the adopter's answer), a missed defect ships within a day (`docs/RELEASING.md`).
|
|
53
53
|
- Tier exits: `pr` 0; `scheduled` not run, `RESERVATION_MODEL_KEY` is absent on this machine; `release` 2, repair owned by Stage 3.
|
|
54
|
-
- Hand-off status: framework skill invoked with CI scope and Create or Edit, or an open item such as a declined baseline or a missing `bmad-testarch-framework
|
|
54
|
+
- Hand-off status: framework skill invoked with CI scope and Create or Edit, or an open item such as a declined baseline or a missing `bmad-testarch-framework`.
|
|
55
55
|
|
|
56
56
|
Replace this worked record with the inspected target's facts and source paths. Record the selected target kind and interface in `evaluation.json` alone.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Authoring model routing
|
|
2
|
+
|
|
3
|
+
Use economical subagents for bounded research and candidate drafting. The parent Test Architect owns requirements confirmation, corpus selection, contract and oracle decisions, artifact writes, every CLI gate, score interpretation, and the final report. A worker's answer is a proposal with file and line evidence. It is never a qualified probe or sealed verdict.
|
|
4
|
+
|
|
5
|
+
## Select a worker at dispatch time
|
|
6
|
+
|
|
7
|
+
Inspect the current host's available subagent models immediately before dispatch. On Claude, select the latest available Haiku tier. On Codex, select the latest available Luna tier. Do not put a versioned model ID in the dispatch rule or reuse a recorded ID as the next default. If the host does not expose that tier or an exact resolved model, do the task in the parent session and record the fallback. Do not silently substitute another model tier.
|
|
8
|
+
|
|
9
|
+
Record the provider, requested tier, exact resolved model, host or CLI version when available, and the selection time in `<evaluation-folder>/authoring-log.md`. The logged versioned ID is an audit observation, not a model pin for later dispatches. Recheck availability for a later session.
|
|
10
|
+
|
|
11
|
+
This routing applies to the agent **authoring** the evaluation. The target model, system prompt, evaluator framework, rubric judge, decoding settings, and isolation conditions are fixed scored conditions. Keep their recorded snapshots and values identical across clean and mutated arms. A model change to any scored condition needs a new comparable run, never an unrecorded worker substitution.
|
|
12
|
+
|
|
13
|
+
When the adopter asks for the latest target model or model tier, resolve it from the current provider or host catalog before the first live preflight. Do not inherit a versioned adapter default from an older implementation, reuse a prior evaluation's model ID, or assume the host's configured default is current. Record the requested tier, source and time of resolution, exact ID passed to the target, and the same ID in evaluator conditions. Recheck before a new baseline; if the resolved ID changes, update the condition and repeat check, compile, seal, preflight, run, and score. Historical run records retain their original IDs.
|
|
14
|
+
|
|
15
|
+
## Delegate bounded tasks
|
|
16
|
+
|
|
17
|
+
Use at most two read-only workers at once. Give each a disjoint source scope, the confirmed requirements statement, a concrete question, a time bound, and this output shape: `candidate`, `source path and line`, `uncertainty`, and `files inspected`. Good tasks are entry-point inventory, failure-history search, candidate boundary cases, candidate mutation locations, and summaries of public evaluator documentation. Workers may draft a probe or oracle fragment as text, but the parent decides whether it enters the corpus or contract.
|
|
18
|
+
|
|
19
|
+
Keep one writer: the parent. Instruct workers to avoid editing the repository, evaluation folder, policy, or run artifacts; launching targets; running `tea-evaluate` or `eval-quality`; or contacting other workers. Some hosts give subagents writable tools even for a read-only request. Pause parent writes during a worker group. Compare the scoped source and evaluation-file inventory and digests before dispatch and after every worker returns. If files changed, reject the affected drafts, identify the writer, and repair worker-attributed edits before continuing. Never send workers secrets, credentials, private user data, `corpus/held-out/`, a held-out plan, held-out results, or a complete held-out request. When a task cannot be split cleanly, keep it in the parent session.
|
|
20
|
+
|
|
21
|
+
The parent checks every file and line claim before adopting it. Draft probes still pass `tea-evaluate check`, `eval-quality compile` and `seal`, preflight, scored `run`, and `score` at their normal stages. A cheap worker's confidence or self-reported pass does not replace any gate. If a draft fails a gate, record the failure and repair time in the authoring log.
|
|
22
|
+
|
|
23
|
+
## Show authoring progress and cost
|
|
24
|
+
|
|
25
|
+
Start an authoring log in `{test_artifacts}/evaluate/` before Stage 1 inspection, then carry it to `<evaluation-folder>/authoring-log.md` when that folder exists. Record each stage's start and completion time, its artifact paths, and its next gate. For each worker task, record dispatch and completion time, scope, requested tier, resolved model, outcome, retries, parent verification and repair time, and reported input/output tokens or cost when available. Write `unknown` for missing usage. Count parent work separately from worker work. On resume, read the existing log, mark any unfinished worker task interrupted, and continue with new entries rather than replacing prior timings. Keep this log outside `corpus/`, `probes/`, `mutations/`, and `runs/` so it does not alter a corpus digest or sealed run.
|
|
26
|
+
|
|
27
|
+
For a long target run, use the runtime's progress command when available and report its current phase, completed/planned trials, whole-run elapsed time, and last heartbeat. Report scored trial usage separately from authoring usage; a subtotal is not full-run cost. Continue through `score` before reporting a verdict.
|
|
28
|
+
|
|
29
|
+
To test savings, compare a parent-only authoring pass with a routed pass using the same confirmed requirements, target revision and model, scoring policy, evaluator conditions, and probe scope. Compare authoring wall time, reported tokens and cost, retries, parent repair time, and gate failures. Compare the resulting clean-control, seeded-defect, and held-out outcomes from eval-quality. Run time and target call cost are separate measurements; authoring routing alone does not shorten a scored target call. Record unknown usage explicitly and make no savings claim from incomplete cost data.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# CI
|
|
2
2
|
|
|
3
|
-
Run this stage after the evaluation checks clean and a scored run exists (Stages 1 to 11). It writes `<evaluation-folder>/ci/evaluation-ci-plan.json`, the platform-neutral plan that is the only definition of which check runs when and what blocks, then hands the plan to `bmad-testarch-framework` with explicit CI setup scope.
|
|
3
|
+
Run this stage after the evaluation checks clean and a scored run exists (Stages 1 to 11). It writes `<evaluation-folder>/ci/evaluation-ci-plan.json`, the platform-neutral plan that is the only definition of which check runs when and what blocks, then hands the plan to `bmad-testarch-framework` with explicit CI setup scope. Never write a pipeline file here. The plan's schema and placement rules belong to the installed runtime, in `{tea_evaluations_folder}/node_modules/bmad-method-test-architecture-enterprise/cli/lib/evaluate/schemas/evaluation-ci-plan.schema.json` and `{tea_evaluations_folder}/node_modules/bmad-method-test-architecture-enterprise/cli/lib/evaluate/ci-plan.js` (`cli/lib/evaluate/...` from the repository root inside TeA's own package); read them there and copy neither. `npm exec --prefix {tea_evaluations_folder} -- tea-evaluate check --evaluation <evaluation-folder>` reports the plan's findings under the `ci-plan` rule family, so the command that validates the rest of the evaluation validates the plan.
|
|
4
4
|
|
|
5
5
|
If `check` reports a finding outside the `ci-plan` family, or no scored run exists, name the missing prerequisite and return to its stage before inspecting CI.
|
|
6
6
|
|
|
@@ -48,7 +48,7 @@ The `release` tier is the usual home, since its event is the one that starts the
|
|
|
48
48
|
Name in the `trigger` of every check on the gating tier each event that starts the gated job's workflow.
|
|
49
49
|
When that workflow also starts on `workflow_dispatch`, as a nightly `deploy.yml` does, list `manual-dispatch` beside `release`, since the CI skill refuses a gate whose job already ran on an event where the tier's evaluation job is skipped.
|
|
50
50
|
The jobs a tier gates are the union of `gates` over the checks the plan places on that tier, so name a job once, on any check of the tier, and a name repeated across the tier's checks is merged.
|
|
51
|
-
`check` accepts a job id and reports any other name under rule `gates`; it cannot tell whether the repository holds that job, which `bmad-testarch-
|
|
51
|
+
`check` accepts a job id and reports any other name under rule `gates`; it cannot tell whether the repository holds that job, which `bmad-testarch-framework` answers when it renders the wait.
|
|
52
52
|
The wait holds best when the job lives in the workflow file the CI skill edits, which holds the evaluation jobs too: the job then waits through `needs`, and the step refuses the plan when the job already ran in a run where the tier's evaluation job is skipped, such as a deploy on every `push` beside an evaluation job guarded to tags, or when the evaluation job runs on a run the job did not run on before, such as a `workflow_dispatch` the plan adds to a tag-push release file, while a run the render adds that the evaluation job skips is skipped through the wait and is no conflict.
|
|
53
53
|
A job in another workflow file waits through a `workflow_run` trigger, which the step refuses for a pull request tier, for a tier whose evaluation job carries a ref or cron guard, and for a job with its own `needs`, `if:` or ref and event contexts, so hand the CI skill the workflow file that holds the job when you can.
|
|
54
54
|
Leave `gates` out when the inspection found no such job or the adopter declined the gate, and say which in the `reason` of the release entries.
|
|
@@ -62,7 +62,7 @@ Place the gameability arm (`gameability`), contract-source freshness (part of `c
|
|
|
62
62
|
|
|
63
63
|
Put the live checks where the inspection found a secret and an event: a live `preflight-live` on `merge` when the target needs no secret and the merge flow allows its run time, the held-out partition (`held-out`) on `scheduled` and `release`, judge calibration (`judge-calibration`) on `scheduled` and `release` whenever the contract declares a rubric, and `twin-run` and `strength-comparison` with them. Never place a live check on `pr`. A skill or agent target always needs the runner's model credentials, so its live tiers are `scheduled`, `release` and manual dispatch only; its `trigger` lists `schedule` and `manual-dispatch` for `scheduled` and `release` for `release`, with `manual-dispatch` beside it when the gated workflow starts on `workflow_dispatch`. Without a schedule trigger in the repository, say so in the `reason` and put the set on `release`.
|
|
64
64
|
|
|
65
|
-
Declare the runner's credential keys as `permittedEnvironmentKeys` through the target's registry `environmentKeys`, which Stage 6 scaffolds and the runtime compiles into the authorization: read them, and add a key name only when a live check needs a credential the registry omits. Keys carry names alone. The plan carries no credential and
|
|
65
|
+
Declare the runner's credential keys as `permittedEnvironmentKeys` through the target's registry `environmentKeys`, which Stage 6 scaffolds and the runtime compiles into the authorization: read them, and add a key name only when a live check needs a credential the registry omits. Keys carry names alone. The plan carries no credential and Framework CI setup wires none, so give the adopter the same names as the CI secrets to add for the live tiers.
|
|
66
66
|
|
|
67
67
|
On Linux the live checks also need the target's registry entry to carry the `egress` authorization for the hosts it reaches.
|
|
68
68
|
A confined Linux target runs in a network namespace with a loopback and nothing else, so read the entry's `egress` and add the model provider's host, port and the addresses the host resolves to now when an item is missing, as the example shows.
|
|
@@ -250,7 +250,7 @@ With no accepted baseline, skip the tier runs and record that in the `## CI` sec
|
|
|
250
250
|
|
|
251
251
|
## Hand the plan to framework setup with CI scope
|
|
252
252
|
|
|
253
|
-
Invoke `bmad-testarch-framework` with explicit CI setup scope and Edit operation on the adopter's pipeline file, naming the plan's path, or Create when the inspection found no pipeline file.
|
|
253
|
+
Invoke `bmad-testarch-framework` with explicit CI setup scope and Edit operation on the adopter's pipeline file, naming the plan's path, or Create when the inspection found no pipeline file. Its CI phase detects `ci/evaluation-ci-plan.json` and validates it; the rendering rules belong to `skills/bmad-testarch-framework/ci/steps-c/step-03b-render-evaluation-plans.md` from TEA's repository root. In an installed skill set, resolve it as `{skill-root}/../bmad-testarch-framework/ci/steps-c/step-03b-render-evaluation-plans.md`, where `{skill-root}` is the Evaluate skill root. Setup scope and operation are independent, so this handoff selects CI and preserves an existing framework. CI Edit validates the changed pipeline syntax, injection safety and evaluation-plan rendering/gates; it does not require the adopter's full suite to pass or activate framework hooks. CI Create uses the existing framework contract and its execution gate. Name in the request the concrete event of this repository for each tier it should render, such as `release`: the push of `v*` tags that starts `.github/workflows/release.yml`.
|
|
254
254
|
That step renders each `gates` entry as a wait of the named job on the evaluation job of the tier, and its summary reports the edit to the job, a name that matches no job and a conflicting gate, so relay that report to the adopter.
|
|
255
255
|
When the plan gates a job, invoke it on the workflow file that holds that job.
|
|
256
|
-
When that skill is not installed, give the adopter the request and the plan's path and record the hand-off as an open item in the inspection record. Stage 12 is complete when the plan passes `check` and the hand-off is made; a declined baseline or a missing `bmad-testarch-framework`
|
|
256
|
+
When that skill is not installed, give the adopter the request and the plan's path and record the hand-off as an open item in the inspection record. Stage 12 is complete when the plan passes `check` and the hand-off is made; a declined baseline or a missing `bmad-testarch-framework` stays a named open item in the `## CI` section and does not reopen the stage. Finish with the closing summary: the placement table, the deviations and their reasons, the secrets the live tiers need (the names from `environmentKeys`), the gates adopted, the baseline still to accept, every tier that exited non-zero with the stage that owns its repair, and each publish or deploy job that waits for the evaluation job through `gates`, with every such job the plan leaves ungated and the reason.
|
|
@@ -2,6 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
Run from the sealed evaluation folder after its target registry, evaluator, mutations, policy and conditions pass `check`. Save each command, exit status, stderr and resulting artifact path in the run log. Stop the command sequence on a nonzero stage exit; retain diagnostics and route the failure through the gaps guide before retrying. eval-quality alone decides verdict and strength.
|
|
4
4
|
|
|
5
|
+
## Protect the target's Git state
|
|
6
|
+
|
|
7
|
+
A `git` workspace makes detached worktrees from `launch.root`. Every worktree of one repository shares its Git common directory, including refs, config and hooks. When confinement is disabled, Evaluate checks that shared state before and after qualification and trials. A branch creation, commit, rebase, fetch, push that updates tracking refs, worktree addition, hook edit, or test that writes refs in another worktree can stop an unconfined run with exit 12 after a costly target call. Separate worktrees alone do not isolate this state.
|
|
8
|
+
|
|
9
|
+
Before the first live preflight, resolve `launch.root` and run `git -C <launch-root> rev-parse --git-common-dir`. Record the resolved common directory and target commit in the run log. If development or CI-related Git work must continue in the same repository during a long evaluation, use a dedicated clone with its own `.git` directory as `launch.root`. Confirm its common directory differs from the active development repository, update `evaluation.json`, then repeat `check`, `compile`, `seal` and `preflight`. A local clone may share object files but has independent refs; do not add a worktree to the active repository as the substitute.
|
|
10
|
+
|
|
11
|
+
When the target uses the active repository, reserve a repository-wide Git quiet period before the first live preflight and keep it through `run` and `score`. During that period, do not commit, branch, rebase, fetch, push, add worktrees, edit hooks or config, or run tools that change refs in any worktree sharing its common directory. Read-only status inspection and `tea-evaluate status --follow` can continue. If a run stops for shared Git state, record its command, exit 12 and diagnostic as an infrastructure stop, establish an independent clone or quiet period, and rerun; it has no scored verdict.
|
|
12
|
+
|
|
5
13
|
## Install the private latest-spec runtime
|
|
6
14
|
|
|
7
15
|
In `{tea_evaluations_folder}`, create a private `package.json` with TeA and eval-quality as `devDependencies` at the `latest` spec. The adopter decides whether to track this manifest and its lockfile for reproducibility. Keep the installed versions in the run log. Node >=22.20.0 is required. Install with `npm install --prefix {tea_evaluations_folder}`. Stage 6 installs `assets/evaluation-folder.gitignore` as `<evaluation-folder>/.gitignore` before the first preflight; verify that it still ignores `runs/`, `compiled-contract.json` and `sealed-brief.json` before this run. When `{project-root}` is TeA's own package, invoke `node cli/evaluate.js` from the repository root for TeA commands so the local source is tested. Use the same local engine installation for `eval-quality`.
|
|
@@ -56,6 +64,17 @@ npm exec --prefix {tea_evaluations_folder} -- tea-evaluate score --evaluation <e
|
|
|
56
64
|
|
|
57
65
|
Use the `invocationId` emitted by `run`; it names `runs/<invocationId>/` and is the value `score --run` accepts. Each trial set has its own `runId`, which is a different identifier. The runtime qualifies mutations against a passing baseline, calibrates any rubric judge, qualifies a sealed-brief agent evaluator on the clean and mutated arms, executes clean and mutated trials through the chosen evaluator, verifies rollback, and seals records and isolation manifests. `score` delegates every probe to eval-quality and persists per-probe diagnostics under `runs/<invocationId>/scores/<scoreInvocationId>/`, plus evidence artifacts, the run-wide `strength-aggregate.json`, `interpretation.json`, `partitions.json` and `gap-view.json`. Preserve both the command exit and each per-probe exit. A clean control should resolve `passed-clean-control` and the seeded defect `caught`; read those states from the evidence artifact. If a development `score` exits 3 with no artifact, read its persisted `score.json`, stdout and stderr diagnostics and route the fault through the gaps guide. For a held-out fault, read only `gap-view.json` and reproduce the class failure in development before inspecting diagnostics.
|
|
58
66
|
|
|
67
|
+
For a long run, use its invocation ID from another terminal to read progress without disturbing the run:
|
|
68
|
+
|
|
69
|
+
```sh
|
|
70
|
+
npm exec --prefix {tea_evaluations_folder} -- tea-evaluate status --evaluation <evaluation-folder> --run <invocationId>
|
|
71
|
+
npm exec --prefix {tea_evaluations_folder} -- tea-evaluate status --evaluation <evaluation-folder> --run <invocationId> --follow
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Inside TeA, use `node cli/evaluate.js status` with the same arguments. `status` reads the current phase, time spent in that phase, completed and planned trials, whole-run elapsed time, completed phase durations, last heartbeat, and trial usage from the run directory. `--follow` keeps reporting changes until the run ends. The running `run` command also prints periodic heartbeat lines during a long call. If a heartbeat is old, report when it was last seen without claiming the process is still alive. Show the whole-run elapsed time alongside phase timing; the sum of trial durations omits qualification and preflight. Count a shared arm's trials once when reporting totals.
|
|
75
|
+
|
|
76
|
+
Usage in progress covers scored trials only. Qualification and preflight costs are not fully captured, so full-run cost is unknown even when all trials report. If a trial call lacks complete usage, trial cost is also unknown; the reported amount is a subtotal. An explicit reported zero is a measured zero. Progress carries no verdict and provides no ETA. Continue through `score` to read the evaluation result.
|
|
77
|
+
|
|
59
78
|
## Record a before state
|
|
60
79
|
|
|
61
80
|
A target with known defects fails its clean controls, so `run` stops with exit 11 on the first. Record that starting point with `--before-state` and keep the ordinary run for the fixed target.
|
|
@@ -68,9 +68,9 @@ Activation is complete. Begin the workflow below.
|
|
|
68
68
|
|
|
69
69
|
## Workflow Architecture
|
|
70
70
|
|
|
71
|
-
The existing framework steps at the skill root and the CI steps under `ci/` share scope and operation routing. `bmad-testarch-
|
|
71
|
+
The existing framework steps at the skill root and the CI steps under `ci/` share scope and operation routing. Select CI setup by asking `bmad-testarch-framework` for CI scope.
|
|
72
72
|
|
|
73
|
-
Create and edit CI phases detect `ci/evaluation-ci-plan.json` and render it through `{skill-root}/ci/steps-c/step-03b-render-evaluation-plans.md`. Evaluate
|
|
73
|
+
Create and edit CI phases detect `ci/evaluation-ci-plan.json` and render it through `{skill-root}/ci/steps-c/step-03b-render-evaluation-plans.md`. Evaluate invokes `bmad-testarch-framework` with CI scope in create or edit mode.
|
|
74
74
|
|
|
75
75
|
## Initialization Sequence
|
|
76
76
|
|
|
@@ -266,9 +266,9 @@ Per `playwright-utils-mandate.md`. Skip for Maestro, Cypress, and non-Playwright
|
|
|
266
266
|
|
|
267
267
|
### Workflow Dependencies
|
|
268
268
|
|
|
269
|
-
- [ ] Can proceed to
|
|
269
|
+
- [ ] Can proceed to Framework CI scope after completion
|
|
270
270
|
- [ ] Can proceed to `test-design` workflow after completion
|
|
271
|
-
- [ ] Can proceed to
|
|
271
|
+
- [ ] Can proceed to Automate red mode after completion
|
|
272
272
|
- [ ] Framework setup compatible with downstream workflows
|
|
273
273
|
|
|
274
274
|
---
|
|
@@ -301,9 +301,9 @@ Per `playwright-utils-mandate.md`. Skip for Maestro, Cypress, and non-Playwright
|
|
|
301
301
|
|
|
302
302
|
**Recommended next workflows:**
|
|
303
303
|
|
|
304
|
-
1. [ ] Run
|
|
304
|
+
1. [ ] Run Framework with CI scope to set up the CI/CD pipeline
|
|
305
305
|
2. [ ] Run `test-design` workflow to plan test coverage
|
|
306
|
-
3. [ ] Run
|
|
306
|
+
3. [ ] Run Automate in red mode when ready to develop stories
|
|
307
307
|
|
|
308
308
|
---
|
|
309
309
|
|
|
@@ -193,7 +193,7 @@ Note: CI setup is typically a one-time task per repo and can be run any time aft
|
|
|
193
193
|
### Workflow Dependencies
|
|
194
194
|
|
|
195
195
|
- [ ] Existing or newly created framework validated against the agreed contract in this run (CI-only existing; both-scope new or reused)
|
|
196
|
-
- [ ] Can proceed to
|
|
196
|
+
- [ ] Can proceed to Automate red mode after CI setup
|
|
197
197
|
- [ ] Can proceed to `automate` workflow
|
|
198
198
|
- [ ] CI integrates with `gate` workflow
|
|
199
199
|
|
|
@@ -224,7 +224,7 @@ Note: CI setup is typically a one-time task per repo and can be run any time aft
|
|
|
224
224
|
|
|
225
225
|
**Recommended next workflows:**
|
|
226
226
|
|
|
227
|
-
1. [ ] Run
|
|
227
|
+
1. [ ] Run Automate in red mode for acceptance test generation
|
|
228
228
|
2. [ ] Run `automate` workflow for coverage expansion
|
|
229
229
|
3. [ ] Run `gate` workflow for quality gates
|
|
230
230
|
|
|
@@ -33,7 +33,7 @@ From `workflow.yaml`, resolve:
|
|
|
33
33
|
|
|
34
34
|
### 2. Canonical CI Activation
|
|
35
35
|
|
|
36
|
-
If invoked through legacy workflow metadata, preset `setup_scope = ci
|
|
36
|
+
If invoked through legacy workflow metadata, preset `setup_scope = ci`, then load `{skill-root}/SKILL.md` and its shared router before executing a phase. Keep the original operation and CI customization. CI instructions, templates and steps resolve under `{skill-root}/ci/`.
|
|
37
37
|
|
|
38
38
|
### 3. First Step
|
|
39
39
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Shared Setup Router
|
|
2
2
|
|
|
3
|
-
Read this resource and `{skill-root}/resources/setup-state.md` completely before activation. `{skill-root}` always names `bmad-testarch-framework
|
|
3
|
+
Read this resource and `{skill-root}/resources/setup-state.md` completely before activation. `{skill-root}` always names `bmad-testarch-framework`. Keep the original request and supplied artifact paths through every handoff.
|
|
4
4
|
|
|
5
5
|
## 1. Read-Only Request Gate: Before Activation
|
|
6
6
|
|
|
@@ -11,7 +11,7 @@ Read effective config and all applicable run/phase history and execute `setup-st
|
|
|
11
11
|
Resolve `setup_scope` as `framework`, `ci`, or `both`:
|
|
12
12
|
|
|
13
13
|
- Explicit scope and a supplied operation take priority. Requests mentioning framework and pipeline setup select `both`.
|
|
14
|
-
- A CI pipeline, quality gates, CI artifact target
|
|
14
|
+
- A CI pipeline, quality gates, or CI artifact target selects `ci`.
|
|
15
15
|
- Framework-only wording or a supplied framework artifact selects `framework`.
|
|
16
16
|
- For setup requests whose CI intent remains unclear, ask exactly once: **“Do you want CI too?”** A yes selects `both`; a no selects `framework`. Record `ci_scope_answered = true`. Wait for the answer before any project writes, including custom activation hooks. An unattended/headless run with no scope answer defaults to `framework`, records `ci_scope_defaulted = true`, and says in its completion summary that CI was not included. Explicit or inferred CI intent still selects `ci` or `both` in unattended runs.
|
|
17
17
|
|
|
@@ -27,8 +27,8 @@ Before SKILL.md executes prepend hooks or greets, resolve settings for every req
|
|
|
27
27
|
|
|
28
28
|
When scope includes CI:
|
|
29
29
|
|
|
30
|
-
1. Start with
|
|
31
|
-
2.
|
|
30
|
+
1. Start with CI defaults: `ci_platform = auto`, empty `activation_steps_prepend`, `activation_steps_append`, `persistent_facts`, and `on_complete`.
|
|
31
|
+
2. Structurally merge the CI defaults, `_bmad/custom/bmad-testarch-ci.toml`, and `_bmad/custom/bmad-testarch-ci.user.toml`, skipping missing files. Existing CI overrides, including migrated platform settings, remain effective.
|
|
32
32
|
3. Store the result as `ci_workflow`. SKILL.md executes applicable prepend entries before greeting, loads applicable facts/globs in lexical order, and executes append entries after greeting through `setup-state.md` section 3's individual durable hook protocol. On Resume, resolve current settings and reload applicable facts; skip completed hook entries and halt on uncertain started entries. Save each applicable phase in `activation_completed` only after its activation succeeds. This section resolves settings; it does not replay activation at phase or worker handoffs.
|
|
33
33
|
4. Framework steps use canonical `workflow`. CI steps read `ci_workflow` for their legacy hooks/facts and the effective platform below. Keep `{skill-root}` bound to the canonical directory for every CI asset.
|
|
34
34
|
|
|
@@ -188,11 +188,11 @@
|
|
|
188
188
|
|
|
189
189
|
### Workflow Dependencies
|
|
190
190
|
|
|
191
|
-
- [ ] Can proceed to `/bmad-testarch-
|
|
192
|
-
- [ ] `/bmad-testarch-
|
|
191
|
+
- [ ] Can proceed to `/bmad-testarch-automate` red mode with P0 scenarios
|
|
192
|
+
- [ ] `/bmad-testarch-automate` is a separate workflow and must be run explicitly (not auto-run)
|
|
193
193
|
- [ ] Can proceed to `automate` workflow with full coverage plan
|
|
194
194
|
- [ ] Risk assessment informs `gate` workflow criteria
|
|
195
|
-
- [ ] Integrates with
|
|
195
|
+
- [ ] Integrates with Framework CI scope execution order
|
|
196
196
|
|
|
197
197
|
## Accountability & Logistics
|
|
198
198
|
|
|
@@ -415,15 +415,15 @@
|
|
|
415
415
|
1. [ ] Review risk assessment with team
|
|
416
416
|
2. [ ] Prioritize mitigation for high-priority risks (score ≥6)
|
|
417
417
|
3. [ ] Allocate resources per estimates
|
|
418
|
-
4. [ ] Run `/bmad-testarch-
|
|
418
|
+
4. [ ] Run `/bmad-testarch-automate` in red mode to generate P0 tests
|
|
419
419
|
5. [ ] Set up test data factories and fixtures
|
|
420
420
|
6. [ ] Schedule team review of test design document
|
|
421
421
|
|
|
422
422
|
**Recommended next workflows:**
|
|
423
423
|
|
|
424
|
-
1. [ ] Run
|
|
424
|
+
1. [ ] Run Automate in red mode for P0 test generation
|
|
425
425
|
2. [ ] Run `framework` workflow if not already done
|
|
426
|
-
3. [ ] Run
|
|
426
|
+
3. [ ] Run Framework with CI scope to configure pipeline stages
|
|
427
427
|
|
|
428
428
|
## Rollback Procedure
|
|
429
429
|
|
|
@@ -258,7 +258,7 @@ P0, P1, P2, and P3 describe priority. Execution timing is defined separately bel
|
|
|
258
258
|
|
|
259
259
|
## Follow-on Workflows
|
|
260
260
|
|
|
261
|
-
- Run `/bmad-testarch-
|
|
261
|
+
- Run `/bmad-testarch-automate` in red mode explicitly for the four P0 scenarios.
|
|
262
262
|
- Run `/bmad-testarch-automate` for the remaining functional coverage after implementation exists.
|
|
263
263
|
- Run `/bmad-testarch-nfr` when evidence exists for all four NFR domains.
|
|
264
264
|
- Run `/bmad-testarch-trace` to map the approved oracle and apply the release gate.
|
|
@@ -290,7 +290,7 @@ Do not re-list the individual tests here. They are already in the coverage plan.
|
|
|
290
290
|
|
|
291
291
|
## Follow-on Workflows (Manual)
|
|
292
292
|
|
|
293
|
-
- Run `/bmad-testarch-
|
|
293
|
+
- Run `/bmad-testarch-automate` in red mode to generate failing P0 tests (separate workflow; not auto-run).
|
|
294
294
|
- Run `/bmad-testarch-automate` for broader coverage once implementation exists.
|
|
295
295
|
|
|
296
296
|
---
|
|
@@ -16,7 +16,7 @@ This checklist covers **two sequential phases**:
|
|
|
16
16
|
|
|
17
17
|
- [ ] A coverage oracle is available or inferred (formal requirements, spec, resolvable external pointer, or synthetic journeys)
|
|
18
18
|
- [ ] Test suite exists (or gaps are acknowledged and documented)
|
|
19
|
-
- [ ] If tests are missing, recommend `/bmad-testarch-
|
|
19
|
+
- [ ] If tests are missing, recommend `/bmad-testarch-automate` with the applicable red or expand mode (trace does not run it automatically)
|
|
20
20
|
- [ ] Test directory path is correct (`test_dir` variable)
|
|
21
21
|
- [ ] Story file is accessible (if using BMad mode)
|
|
22
22
|
- [ ] Knowledge base is loaded (test-priorities, traceability, risk-governance)
|
|
@@ -673,7 +673,7 @@ This list is the definition of waiver validity. `steps-c/step-05-gate-decision.m
|
|
|
673
673
|
|
|
674
674
|
- If PASS (both phases): Proceed to deployment
|
|
675
675
|
- If WARN/CONCERNS: Address gaps/issues, proceed with monitoring
|
|
676
|
-
- If FAIL (either phase): Run `/bmad-testarch-
|
|
676
|
+
- If FAIL (either phase): Run `/bmad-testarch-automate` for missing tests, fix issues, re-run `/bmad-testarch-trace`
|
|
677
677
|
- If WAIVED: Deploy with approved waiver, schedule remediation
|
|
678
678
|
|
|
679
679
|
---
|
|
@@ -374,7 +374,7 @@ const recommendations = [];
|
|
|
374
374
|
if (criticalGaps.length > 0) {
|
|
375
375
|
recommendations.push({
|
|
376
376
|
priority: 'URGENT',
|
|
377
|
-
action: `Run /bmad-testarch-
|
|
377
|
+
action: `Run /bmad-testarch-automate for ${criticalGaps.length} P0 requirements: red mode where behavior is missing; expand mode where behavior exists but tests are incomplete`,
|
|
378
378
|
requirements: criticalGaps.map((r) => r.id),
|
|
379
379
|
});
|
|
380
380
|
}
|
|
@@ -29,7 +29,7 @@ oracleLedger: []
|
|
|
29
29
|
|
|
30
30
|
---
|
|
31
31
|
|
|
32
|
-
Note: This workflow does not generate tests. If gaps exist, run `/bmad-testarch-
|
|
32
|
+
Note: This workflow does not generate tests. If gaps exist, run `/bmad-testarch-automate` in red mode before implementation or expand mode for existing code.
|
|
33
33
|
|
|
34
34
|
## PHASE 1: REQUIREMENTS TRACEABILITY
|
|
35
35
|
|
|
@@ -2,14 +2,12 @@
|
|
|
2
2
|
|
|
3
3
|
[bmod]
|
|
4
4
|
code = "tea"
|
|
5
|
-
version = "2.0.
|
|
5
|
+
version = "2.0.3-next.0"
|
|
6
6
|
update_source = "github:bmad-code-org/bmad-method-test-architecture-enterprise/skills"
|
|
7
7
|
skills = [
|
|
8
8
|
"bmad-tea",
|
|
9
9
|
"bmad-teach-me-testing",
|
|
10
|
-
"bmad-testarch-atdd",
|
|
11
10
|
"bmad-testarch-automate",
|
|
12
|
-
"bmad-testarch-ci",
|
|
13
11
|
"bmad-testarch-evaluate",
|
|
14
12
|
"bmad-testarch-framework",
|
|
15
13
|
"bmad-testarch-nfr",
|
|
@@ -22,7 +20,7 @@ pre_install_message = ""
|
|
|
22
20
|
post_install_message = '''
|
|
23
21
|
TEA is ready. Talk to Murat (bmad-tea) or start with Test Design (bmad-testarch-test-design).
|
|
24
22
|
|
|
25
|
-
Playwright Utils, Pact.js Utils, the Playwright CLI or MCP servers, and the SmartBear MCP each need their own install. Test Framework (bmad-testarch-framework) installs the utils packages and owns framework and CI setup.
|
|
23
|
+
Playwright Utils, Pact.js Utils, the Playwright CLI or MCP servers, and the SmartBear MCP each need their own install. Test Framework (bmad-testarch-framework) installs the utils packages and owns framework and CI setup. Test Automation (bmad-testarch-automate) owns red and expand test generation with run-and-heal enabled by default. Existing CI and ATDD customizations keep working through their respective modes. Ask bmad how to set up the rest.
|
|
26
24
|
'''
|
|
27
25
|
|
|
28
26
|
# Asked by `bmad setup` in this order. Members read each answer as modules.tea.<key>.
|
|
@@ -8,9 +8,7 @@ This document covers the skills of the `tea` module: what each one gives the use
|
|
|
8
8
|
- `bmad-teach-me-testing`: teaches testing fundamentals in 7 sessions (TEA Academy), with a progress file, session notes and a completion summary. Recommend it to someone learning testing, at any phase.
|
|
9
9
|
- `bmad-testarch-test-design`: risk-based test planning, producing a test design document. Recommend it during solutioning, before tests or a framework exist. Afterwards offer `bmad-testarch-framework`.
|
|
10
10
|
- `bmad-testarch-framework`: owns production-ready framework and CI setup, with framework-only, CI-only, or both scope. It installs Playwright Utils or Pact.js Utils when enabled. Recommend it during solutioning, after test design. When CI scope cannot be inferred from the request, interactive runs ask whether to include CI. Unattended runs default to framework only and state in the summary that CI was excluded.
|
|
11
|
-
- `bmad-testarch-
|
|
12
|
-
- `bmad-testarch-atdd`: starts the combined test-generation skill in red mode, writes failing acceptance tests and an ATDD checklist before a story is implemented, and verifies the intended missing behavior. Recommend it after a story is created. Afterwards offer the dev story skill.
|
|
13
|
-
- `bmad-testarch-automate`: owns test generation in red or expand mode. Expand mode writes tests for implemented code, runs them, heals test defects for up to three rounds by default, and reports product defects. Recommend it during implementation, after ATDD or once code exists. Recommend it with red mode when the user asks for acceptance tests before implementation.
|
|
11
|
+
- `bmad-testarch-automate`: owns test generation in red or expand mode. Red mode writes failing acceptance tests and an ATDD checklist before implementation, then verifies the intended missing behavior. Expand mode writes tests for implemented code, runs them, heals test defects for up to three rounds by default, and reports product defects. Recommend red mode after a story is created and expand mode once code exists.
|
|
14
12
|
- `bmad-testarch-evaluate`: builds and runs a scored, evidence-backed evaluation of an agent, skill, workflow or tool-use system. Recommend it after automation when the target's behavior needs proof. Afterwards offer `bmad-testarch-framework` with CI scope to enforce it.
|
|
15
13
|
- `bmad-testarch-test-review`: audits test quality with a 0-100 score and a review report. Recommend it after tests are written. Afterwards offer `bmad-testarch-trace`.
|
|
16
14
|
- `bmad-testarch-nfr`: audits the evidence for non-functional requirements and writes an NFR report. Recommend it after automation, before a release.
|
|
@@ -6,7 +6,7 @@ Each setup answer is `modules.tea.<key>` in `_bmad/config.toml`. `bmad setup tea
|
|
|
6
6
|
|
|
7
7
|
- `test_stack_type = "auto"` detects the project type from manifest files. Mobile: `.maestro/` or `maestro/` flows, `app.json` with expo or react-native, `Podfile`, `android/app/build.gradle`, `*.xcodeproj`, `pubspec.yaml`. Frontend: `package.json` with react, vue or angular, `playwright.config.*`, `vite.config.*`. Backend: `pyproject.toml`, `pom.xml`, `go.mod`, `*.csproj`, `Gemfile`, `Cargo.toml`. Full-stack: both frontend and backend indicators. Mobile is checked first, because a React Native project also has `package.json` with react.
|
|
8
8
|
- `test_framework = "auto"` detects the framework the same way.
|
|
9
|
-
- The CI platform is detected by `bmad-testarch-
|
|
9
|
+
- The CI platform is detected by `bmad-testarch-framework` from `.github/workflows/`, `.gitlab-ci.yml`, `Jenkinsfile`, `azure-pipelines.yml`, `.harness/` or `.circleci/config.yml`. To override detection, set `ci_platform` in `_bmad/custom/bmad-testarch-framework.toml`.
|
|
10
10
|
|
|
11
11
|
## Execution mode
|
|
12
12
|
|