bmad-method 6.0.0-alpha.1 → 6.0.0-alpha.11

package/src/modules/bmm/workflows/testarch/test-design/instructions.md

@@ -2,31 +2,84 @@
 
 # Test Design and Risk Assessment
 
-**Workflow ID**: `bmad/bmm/testarch/test-design`
+**Workflow ID**: `{bmad_folder}/bmm/testarch/test-design`
 **Version**: 4.0 (BMad v6)
 
 ---
 
 ## Overview
 
-Plans comprehensive test coverage strategy with risk assessment, priority classification, and execution ordering. This workflow generates a test design document that identifies high-risk areas, maps requirements to test levels, prioritizes scenarios (P0-P3), and provides resource estimates for the testing effort.
+Plans comprehensive test coverage strategy with risk assessment, priority classification, and execution ordering. This workflow operates in **two modes**:
+
+- **System-Level Mode (Phase 3)**: Testability review of architecture before solutioning gate check
+- **Epic-Level Mode (Phase 4)**: Per-epic test planning with risk assessment (current behavior)
+
+The workflow auto-detects which mode to use based on project phase.
 
 ---
 
-## Preflight Requirements
+## Preflight: Detect Mode and Load Context
+
+**Critical:** Determine mode before proceeding.
+
+### Mode Detection
+
+1. **Check for sprint-status.yaml**
+   - If `{output_folder}/bmm-sprint-status.yaml` exists → **Epic-Level Mode** (Phase 4)
+   - If NOT exists → Check workflow status
+
+2. **Check workflow-status.yaml**
+   - Read `{output_folder}/bmm-workflow-status.yaml`
+   - If `implementation-readiness: required` or `implementation-readiness: recommended` → **System-Level Mode** (Phase 3)
+   - Otherwise → **Epic-Level Mode** (Phase 4 without sprint status yet)
+
+3. **Mode-Specific Requirements**
 
-**Critical:** Verify these requirements before proceeding. If any fail, HALT and notify the user.
+   **System-Level Mode (Phase 3 - Testability Review):**
+   - ✅ Architecture document exists (architecture.md or tech-spec)
+   - ✅ PRD exists with functional and non-functional requirements
+   - ✅ Epics documented (epics.md)
+   - ⚠️ Output: `{output_folder}/test-design-system.md`
 
-- ✅ Story markdown with acceptance criteria available
-- ✅ PRD or epic documentation exists for context
-- ✅ Architecture documents available (optional but recommended)
-- ✅ Requirements are clear and testable
+   **Epic-Level Mode (Phase 4 - Per-Epic Planning):**
+   - ✅ Story markdown with acceptance criteria available
+   - ✅ PRD or epic documentation exists for context
+   - ✅ Architecture documents available (optional but recommended)
+   - ✅ Requirements are clear and testable
+   - ⚠️ Output: `{output_folder}/test-design-epic-{epic_num}.md`
+
+**Halt Condition:** If mode cannot be determined or required files missing, HALT and notify user with missing prerequisites.
 
 ---
 
-## Step 1: Load Context and Requirements
+## Step 1: Load Context (Mode-Aware)
 
-### Actions
+**Mode-Specific Loading:**
+
+### System-Level Mode (Phase 3)
+
+1. **Read Architecture Documentation**
+   - Load architecture.md or tech-spec (REQUIRED)
+   - Load PRD.md for functional and non-functional requirements
+   - Load epics.md for feature scope
+   - Identify technology stack decisions (frameworks, databases, deployment targets)
+   - Note integration points and external system dependencies
+   - Extract NFR requirements (performance SLOs, security requirements, etc.)
+
+2. **Load Knowledge Base Fragments (System-Level)**
+
+   **Critical:** Consult `{project-root}/{bmad_folder}/bmm/testarch/tea-index.csv` to load:
+   - `nfr-criteria.md` - NFR validation approach (security, performance, reliability, maintainability)
+   - `test-levels-framework.md` - Test levels strategy guidance
+   - `risk-governance.md` - Testability risk identification
+   - `test-quality.md` - Quality standards and Definition of Done
+
+3. **Analyze Existing Test Setup (if brownfield)**
+   - Search for existing test directories
+   - Identify current test framework (if any)
+   - Note testability concerns in existing codebase
+
+### Epic-Level Mode (Phase 4)
 
 1. **Read Requirements Documentation**
    - Load PRD.md for high-level product requirements
@@ -37,6 +90,7 @@ Plans comprehensive test coverage strategy with risk assessment, priority classi
 2. **Load Architecture Context**
    - Read architecture.md for system design
    - Read tech-spec for implementation details
+   - Read test-design-system.md (if exists from Phase 3)
    - Identify technical constraints and dependencies
    - Note integration points and external systems
 
@@ -46,19 +100,126 @@ Plans comprehensive test coverage strategy with risk assessment, priority classi
    - Note areas with insufficient testing
    - Check for flaky or outdated tests
 
-4. **Load Knowledge Base Fragments**
+4. **Load Knowledge Base Fragments (Epic-Level)**
 
-   **Critical:** Consult `{project-root}/bmad/bmm/testarch/tea-index.csv` to load:
+   **Critical:** Consult `{project-root}/{bmad_folder}/bmm/testarch/tea-index.csv` to load:
    - `risk-governance.md` - Risk classification framework (6 categories: TECH, SEC, PERF, DATA, BUS, OPS), automated scoring, gate decision engine, owner tracking (625 lines, 4 examples)
    - `probability-impact.md` - Risk scoring methodology (probability × impact matrix, automated classification, dynamic re-assessment, gate integration, 604 lines, 4 examples)
    - `test-levels-framework.md` - Test level selection guidance (E2E vs API vs Component vs Unit with decision matrix, characteristics, when to use each, 467 lines, 4 examples)
    - `test-priorities-matrix.md` - P0-P3 prioritization criteria (automated priority calculation, risk-based mapping, tagging strategy, time budgets, 389 lines, 2 examples)
 
-**Halt Condition:** If story data or acceptance criteria are missing, check if brownfield exploration is needed. If neither requirements NOR exploration possible, HALT with message: "Test design requires clear requirements, acceptance criteria, or brownfield app URL for exploration"
+**Halt Condition (Epic-Level only):** If story data or acceptance criteria are missing, check if brownfield exploration is needed. If neither requirements NOR exploration possible, HALT with message: "Epic-level test design requires clear requirements, acceptance criteria, or brownfield app URL for exploration"
+
+---
+
+## Step 1.5: System-Level Testability Review (Phase 3 Only)
+
+**Skip this step if Epic-Level Mode.** This step only executes in System-Level Mode.
+
+### Actions
+
+1. **Review Architecture for Testability**
+
+   Evaluate architecture against these criteria:
+
+   **Controllability:**
+   - Can we control system state for testing? (API seeding, factories, database reset)
+   - Are external dependencies mockable? (interfaces, dependency injection)
+   - Can we trigger error conditions? (chaos engineering, fault injection)
+
+   **Observability:**
+   - Can we inspect system state? (logging, metrics, traces)
+   - Are test results deterministic? (no race conditions, clear success/failure)
+   - Can we validate NFRs? (performance metrics, security audit logs)
+
+   **Reliability:**
+   - Are tests isolated? (parallel-safe, stateless, cleanup discipline)
+   - Can we reproduce failures? (deterministic waits, HAR capture, seed data)
+   - Are components loosely coupled? (mockable, testable boundaries)
+
+2. **Identify Architecturally Significant Requirements (ASRs)**
+
+   From PRD NFRs and architecture decisions, identify quality requirements that:
+   - Drive architecture decisions (e.g., "Must handle 10K concurrent users" → caching architecture)
+   - Pose testability challenges (e.g., "Sub-second response time" → performance test infrastructure)
+   - Require special test environments (e.g., "Multi-region deployment" → regional test instances)
+
+   Score each ASR using risk matrix (probability × impact).
+
+3. **Define Test Levels Strategy**
+
+   Based on architecture (mobile, web, API, microservices, monolith):
+   - Recommend unit/integration/E2E split (e.g., 70/20/10 for API-heavy, 40/30/30 for UI-heavy)
+   - Identify test environment needs (local, staging, ephemeral, production-like)
+   - Define testing approach per technology (Playwright for web, Maestro for mobile, k6 for performance)
+
+4. **Assess NFR Testing Approach**
+
+   For each NFR category:
+   - **Security**: Auth/authz tests, OWASP validation, secret handling (Playwright E2E + security tools)
+   - **Performance**: Load/stress/spike testing with k6, SLO/SLA thresholds
+   - **Reliability**: Error handling, retries, circuit breakers, health checks (Playwright + API tests)
+   - **Maintainability**: Coverage targets, code quality gates, observability validation
+
+5. **Flag Testability Concerns**
+
+   Identify architecture decisions that harm testability:
+   - ❌ Tight coupling (no interfaces, hard dependencies)
+   - ❌ No dependency injection (can't mock external services)
+   - ❌ Hardcoded configurations (can't test different envs)
+   - ❌ Missing observability (can't validate NFRs)
+   - ❌ Stateful designs (can't parallelize tests)
+
+   **Critical:** If testability concerns are blockers (e.g., "Architecture makes performance testing impossible"), document as CONCERNS or FAIL recommendation for gate check.
+
+6. **Output System-Level Test Design**
+
+   Write to `{output_folder}/test-design-system.md` containing:
+
+   ```markdown
+   # System-Level Test Design
+
+   ## Testability Assessment
+
+   - Controllability: [PASS/CONCERNS/FAIL with details]
+   - Observability: [PASS/CONCERNS/FAIL with details]
+   - Reliability: [PASS/CONCERNS/FAIL with details]
+
+   ## Architecturally Significant Requirements (ASRs)
+
+   [Risk-scored quality requirements]
+
+   ## Test Levels Strategy
+
+   - Unit: [X%] - [Rationale]
+   - Integration: [Y%] - [Rationale]
+   - E2E: [Z%] - [Rationale]
+
+   ## NFR Testing Approach
+
+   - Security: [Approach with tools]
+   - Performance: [Approach with tools]
+   - Reliability: [Approach with tools]
+   - Maintainability: [Approach with tools]
+
+   ## Test Environment Requirements
+
+   [Infrastructure needs based on deployment architecture]
+
+   ## Testability Concerns (if any)
+
+   [Blockers or concerns that should inform solutioning gate check]
+
+   ## Recommendations for Sprint 0
+
+   [Specific actions for *framework and *ci workflows]
+   ```
+
+**After System-Level Mode:** Skip to Step 4 (Generate Deliverables) - Steps 2-3 are epic-level only.
 
 ---
 
-## Step 1.5: Mode Selection (NEW - Phase 2.5)
+## Step 1.6: Exploratory Mode Selection (Epic-Level Only)
 
 ### Actions
 

package/src/modules/bmm/workflows/testarch/test-design/test-design-template.md

@@ -250,9 +250,9 @@
 
 **Test Design Approved By:**
 
-- [ ] Product Manager: **\*\***\_\_\_**\*\*** Date: **\*\***\_\_\_**\*\***
-- [ ] Tech Lead: **\*\***\_\_\_**\*\*** Date: **\*\***\_\_\_**\*\***
-- [ ] QA Lead: **\*\***\_\_\_**\*\*** Date: **\*\***\_\_\_**\*\***
+- [ ] Product Manager: {name} Date: {date}
+- [ ] Tech Lead: {name} Date: {date}
+- [ ] QA Lead: {name} Date: {date}
 
 **Comments:**
 
@@ -281,5 +281,5 @@
 ---
 
 **Generated by**: BMad TEA Agent - Test Architect Module
-**Workflow**: `bmad/bmm/testarch/test-design`
+**Workflow**: `{bmad_folder}/bmm/testarch/test-design`
 **Version**: 4.0 (BMad v6)

package/src/modules/bmm/workflows/testarch/test-design/workflow.yaml

@@ -1,10 +1,10 @@
 # Test Architect workflow: test-design
 name: testarch-test-design
-description: "Plan risk mitigation and test coverage strategy before development with risk assessment and prioritization"
+description: "Dual-mode workflow: (1) System-level testability review in Solutioning phase, or (2) Epic-level test planning in Implementation phase. Auto-detects mode based on project phase."
 author: "BMad"
 
 # Critical variables from config
-config_source: "{project-root}/bmad/bmm/config.yaml"
+config_source: "{project-root}/{bmad_folder}/bmm/config.yaml"
 output_folder: "{config_source}:output_folder"
 user_name: "{config_source}:user_name"
 communication_language: "{config_source}:communication_language"
@@ -12,7 +12,7 @@ document_output_language: "{config_source}:document_output_language"
 date: system-generated
 
 # Workflow components
-installed_path: "{project-root}/bmad/bmm/workflows/testarch/test-design"
+installed_path: "{project-root}/{bmad_folder}/bmm/workflows/testarch/test-design"
 instructions: "{installed_path}/instructions.md"
 validation: "{installed_path}/checklist.md"
 template: "{installed_path}/test-design-template.md"
@@ -20,8 +20,12 @@ template: "{installed_path}/test-design-template.md"
 # Variables and inputs
 variables:
   design_level: "full" # full, targeted, minimal - scope of design effort
+  mode: "auto-detect" # auto-detect (default), system-level, epic-level
 
 # Output configuration
+# Note: Actual output file determined dynamically based on mode detection
+# - System-Level (Phase 3): {output_folder}/test-design-system.md
+# - Epic-Level (Phase 4): {output_folder}/test-design-epic-{epic_num}.md
 default_output_file: "{output_folder}/test-design-epic-{epic_num}.md"
 
 # Required tools
@@ -31,14 +35,6 @@ required_tools:
   - list_files # Find related documentation
   - search_repo # Search for existing tests and patterns
 
-# Recommended inputs
-recommended_inputs:
-  - prd: "Product Requirements Document for context"
-  - epics: "Epic documentation (epics.md or specific epic)"
-  - story: "Story markdown with acceptance criteria"
-  - architecture: "Architecture documents (architecture.md, tech-spec)"
-  - existing_tests: "Current test coverage for gap analysis"
-
 tags:
   - qa
   - planning

package/src/modules/bmm/workflows/testarch/test-review/instructions.md

@@ -49,7 +49,7 @@ This workflow performs comprehensive test quality reviews using TEA's knowledge
 
 **Actions:**
 
-1. Load relevant knowledge fragments from `{project-root}/bmad/bmm/testarch/tea-index.csv`:
+1. Load relevant knowledge fragments from `{project-root}/{bmad_folder}/bmm/testarch/tea-index.csv`:
    - `test-quality.md` - Definition of Done (deterministic tests, isolated with cleanup, explicit assertions, <300 lines, <1.5 min, 658 lines, 5 examples)
    - `fixture-architecture.md` - Pure function → Fixture → mergeTests composition with auto-cleanup (406 lines, 5 examples)
    - `network-first.md` - Route intercept before navigate to prevent race conditions (intercept before navigate, HAR capture, deterministic waiting, 489 lines, 5 examples)

package/src/modules/bmm/workflows/testarch/test-review/workflow.yaml

@@ -4,7 +4,7 @@ description: "Review test quality using comprehensive knowledge base and best pr
 author: "BMad"
 
 # Critical variables from config
-config_source: "{project-root}/bmad/bmm/config.yaml"
+config_source: "{project-root}/{bmad_folder}/bmm/config.yaml"
 output_folder: "{config_source}:output_folder"
 user_name: "{config_source}:user_name"
 communication_language: "{config_source}:communication_language"
@@ -12,7 +12,7 @@ document_output_language: "{config_source}:document_output_language"
 date: system-generated
 
 # Workflow components
-installed_path: "{project-root}/bmad/bmm/workflows/testarch/test-review"
+installed_path: "{project-root}/{bmad_folder}/bmm/workflows/testarch/test-review"
 instructions: "{installed_path}/instructions.md"
 validation: "{installed_path}/checklist.md"
 template: "{installed_path}/test-review-template.md"
@@ -33,13 +33,6 @@ required_tools:
   - search_repo # Find tests by patterns
   - glob # Find test files matching patterns
 
-# Recommended inputs
-recommended_inputs:
-  - test_file: "Test file to review (single file mode)"
-  - test_dir: "Directory of tests to review (directory mode)"
-  - story: "Related story for acceptance criteria context (optional)"
-  - test_design: "Test design for priority context (optional)"
-
 tags:
   - qa
   - test-architect

package/src/modules/bmm/workflows/testarch/trace/instructions.md

@@ -64,7 +64,7 @@ This phase focuses on mapping requirements to tests, analyzing coverage, and ide
 
 **Actions:**
 
-1. Load relevant knowledge fragments from `{project-root}/bmad/bmm/testarch/tea-index.csv`:
+1. Load relevant knowledge fragments from `{project-root}/{bmad_folder}/bmm/testarch/tea-index.csv`:
    - `test-priorities-matrix.md` - P0/P1/P2/P3 risk framework with automated priority calculation, risk-based mapping, tagging strategy (389 lines, 2 examples)
    - `risk-governance.md` - Risk-based testing approach: 6 categories (TECH, SEC, PERF, DATA, BUS, OPS), automated scoring, gate decision engine, coverage traceability (625 lines, 4 examples)
    - `probability-impact.md` - Risk scoring methodology: probability × impact matrix, automated classification, dynamic re-assessment, gate integration (604 lines, 4 examples)
@@ -126,9 +126,9 @@ This phase focuses on mapping requirements to tests, analyzing coverage, and ide
 2. Build traceability matrix:
 
    ```
-   | Criterion ID | Description | Test ID | Test File | Test Level | Coverage Status |
-   |--------------|-------------|---------|-----------|------------|-----------------|
-   | AC-1         | User can... | 1.3-E2E-001 | e2e/auth.spec.ts | E2E | FULL |
+   | Criterion ID | Description | Test ID     | Test File        | Test Level | Coverage Status |
+   | ------------ | ----------- | ----------- | ---------------- | ---------- | --------------- |
+   | AC-1         | User can... | 1.3-E2E-001 | e2e/auth.spec.ts | E2E        | FULL            |
    ```
 
 3. Classify coverage status for each criterion:
@@ -476,10 +476,10 @@ This phase uses traceability results to make a quality gate decision (PASS/CONCE
 
 ## References
 
-- Traceability Matrix: `bmad/output/traceability-matrix.md`
-- Test Design: `bmad/output/test-design-epic-2.md`
+- Traceability Matrix: `{bmad_folder}/output/traceability-matrix.md`
+- Test Design: `{bmad_folder}/output/test-design-epic-2.md`
 - Test Results: `ci-artifacts/test-report-2025-01-15.xml`
-- NFR Assessment: `bmad/output/nfr-assessment-release-1.2.md`
+- NFR Assessment: `{bmad_folder}/output/nfr-assessment-release-1.2.md`
 ```
 
 3. **Include evidence links** (if `require_evidence: true`):
@@ -515,7 +515,7 @@ This phase uses traceability results to make a quality gate decision (PASS/CONCE
 
      - **Decision**: CONCERNS
      - **Reason**: P1 coverage 88% (below 90%)
-     - **Document**: [gate-decision-story-1.3.md](bmad/output/gate-decision-story-1.3.md)
+     - **Document**: [gate-decision-story-1.3.md]({bmad_folder}/output/gate-decision-story-1.3.md)
      - **Action**: Deploy with follow-up story for AC-5
      ```
 
@@ -536,7 +536,7 @@ This phase uses traceability results to make a quality gate decision (PASS/CONCE
    - Create follow-up story for AC-5 E2E test
    - Deploy to staging for validation
 
-   Full Report: bmad/output/gate-decision-story-1.3.md
+   Full Report: {bmad_folder}/output/gate-decision-story-1.3.md
    ```
 
 3. **Request sign-off** (if `require_sign_off: true`):

package/src/modules/bmm/workflows/testarch/trace/workflow.yaml

@@ -4,7 +4,7 @@ description: "Generate requirements-to-tests traceability matrix, analyze covera
 author: "BMad"
 
 # Critical variables from config
-config_source: "{project-root}/bmad/bmm/config.yaml"
+config_source: "{project-root}/{bmad_folder}/bmm/config.yaml"
 output_folder: "{config_source}:output_folder"
 user_name: "{config_source}:user_name"
 communication_language: "{config_source}:communication_language"
@@ -12,7 +12,7 @@ document_output_language: "{config_source}:document_output_language"
 date: system-generated
 
 # Workflow components
-installed_path: "{project-root}/bmad/bmm/workflows/testarch/trace"
+installed_path: "{project-root}/{bmad_folder}/bmm/workflows/testarch/trace"
 instructions: "{installed_path}/instructions.md"
 validation: "{installed_path}/checklist.md"
 template: "{installed_path}/trace-template.md"
@@ -39,17 +39,6 @@ required_tools:
   - search_repo # Find tests by test ID, describe blocks
   - glob # Find test files matching patterns
 
-# Recommended inputs
-recommended_inputs:
-  - story: "Story markdown with acceptance criteria (required for BMad mode)"
-  - test_files: "Test suite for the feature (auto-discovered if not provided)"
-  - test_design: "Test design with risk/priority assessment (required for Phase 2 gate)"
-  - tech_spec: "Technical specification (optional)"
-  - existing_tests: "Current test suite for analysis"
-  - test_results: "CI/CD test execution results (required for Phase 2 gate)"
-  - nfr_assess: "Non-functional requirements validation (recommended for release gates)"
-  - code_coverage: "Code coverage report (optional)"
-
 tags:
   - qa
   - traceability