agentvibes 2.10.0 → 2.12.0

package/.bmad/bmm/workflows/testarch/trace/checklist.md

@@ -1,654 +0,0 @@
-# Requirements Traceability & Gate Decision - Validation Checklist
-
-**Workflow:** `testarch-trace`
-**Purpose:** Ensure complete traceability matrix with actionable gap analysis AND make deployment readiness decision (PASS/CONCERNS/FAIL/WAIVED)
-
-This checklist covers **two sequential phases**:
-
-- **PHASE 1**: Requirements Traceability (always executed)
-- **PHASE 2**: Quality Gate Decision (executed if `enable_gate_decision: true`)
-
----
-
-# PHASE 1: REQUIREMENTS TRACEABILITY
-
-## Prerequisites Validation
-
-- [ ] Acceptance criteria are available (from story file OR inline)
-- [ ] Test suite exists (or gaps are acknowledged and documented)
-- [ ] Test directory path is correct (`test_dir` variable)
-- [ ] Story file is accessible (if using BMad mode)
-- [ ] Knowledge base is loaded (test-priorities, traceability, risk-governance)
-
----
-
-## Context Loading
-
-- [ ] Story file read successfully (if applicable)
-- [ ] Acceptance criteria extracted correctly
-- [ ] Story ID identified (e.g., 1.3)
-- [ ] `test-design.md` loaded (if available)
-- [ ] `tech-spec.md` loaded (if available)
-- [ ] `PRD.md` loaded (if available)
-- [ ] Relevant knowledge fragments loaded from `tea-index.csv`
-
----
-
-## Test Discovery and Cataloging
-
-- [ ] Tests auto-discovered using multiple strategies (test IDs, describe blocks, file paths)
-- [ ] Tests categorized by level (E2E, API, Component, Unit)
-- [ ] Test metadata extracted:
-  - [ ] Test IDs (e.g., 1.3-E2E-001)
-  - [ ] Describe/context blocks
-  - [ ] It blocks (individual test cases)
-  - [ ] Given-When-Then structure (if BDD)
-  - [ ] Priority markers (P0/P1/P2/P3)
-- [ ] All relevant test files found (no tests missed due to naming conventions)
-
----
-
-## Criteria-to-Test Mapping
-
-- [ ] Each acceptance criterion mapped to tests (or marked as NONE)
-- [ ] Explicit references found (test IDs, describe blocks mentioning criterion)
-- [ ] Test level documented (E2E, API, Component, Unit)
-- [ ] Given-When-Then narrative verified for alignment
-- [ ] Traceability matrix table generated:
-  - [ ] Criterion ID
-  - [ ] Description
-  - [ ] Test ID
-  - [ ] Test File
-  - [ ] Test Level
-  - [ ] Coverage Status
-
----
-
-## Coverage Classification
-
-- [ ] Coverage status classified for each criterion:
-  - [ ] **FULL** - All scenarios validated at appropriate level(s)
-  - [ ] **PARTIAL** - Some coverage but missing edge cases or levels
-  - [ ] **NONE** - No test coverage at any level
-  - [ ] **UNIT-ONLY** - Only unit tests (missing integration/E2E validation)
-  - [ ] **INTEGRATION-ONLY** - Only API/Component tests (missing unit confidence)
-- [ ] Classification justifications provided
-- [ ] Edge cases considered in FULL vs PARTIAL determination
-
----
-
-## Duplicate Coverage Detection
-
-- [ ] Duplicate coverage checked across test levels
-- [ ] Acceptable overlap identified (defense in depth for critical paths)
-- [ ] Unacceptable duplication flagged (same validation at multiple levels)
-- [ ] Recommendations provided for consolidation
-- [ ] Selective testing principles applied
-
----
-
-## Gap Analysis
-
-- [ ] Coverage gaps identified:
-  - [ ] Criteria with NONE status
-  - [ ] Criteria with PARTIAL status
-  - [ ] Criteria with UNIT-ONLY status
-  - [ ] Criteria with INTEGRATION-ONLY status
-- [ ] Gaps prioritized by risk level using test-priorities framework:
-  - [ ] **CRITICAL** - P0 criteria without FULL coverage (BLOCKER)
-  - [ ] **HIGH** - P1 criteria without FULL coverage (PR blocker)
-  - [ ] **MEDIUM** - P2 criteria without FULL coverage (nightly gap)
-  - [ ] **LOW** - P3 criteria without FULL coverage (acceptable)
-- [ ] Specific test recommendations provided for each gap:
-  - [ ] Suggested test level (E2E, API, Component, Unit)
-  - [ ] Test description (Given-When-Then)
-  - [ ] Recommended test ID (e.g., 1.3-E2E-004)
-  - [ ] Explanation of why test is needed
-
----
-
-## Coverage Metrics
-
-- [ ] Overall coverage percentage calculated (FULL coverage / total criteria)
-- [ ] P0 coverage percentage calculated
-- [ ] P1 coverage percentage calculated
-- [ ] P2 coverage percentage calculated (if applicable)
-- [ ] Coverage by level calculated:
-  - [ ] E2E coverage %
-  - [ ] API coverage %
-  - [ ] Component coverage %
-  - [ ] Unit coverage %
-
----
-
-## Test Quality Verification
-
-For each mapped test, verify:
-
-- [ ] Explicit assertions are present (not hidden in helpers)
-- [ ] Test follows Given-When-Then structure
-- [ ] No hard waits or sleeps (deterministic waiting only)
-- [ ] Self-cleaning (test cleans up its data)
-- [ ] File size < 300 lines
-- [ ] Test duration < 90 seconds
-
-Quality issues flagged:
-
-- [ ] **BLOCKER** issues identified (missing assertions, hard waits, flaky patterns)
-- [ ] **WARNING** issues identified (large files, slow tests, unclear structure)
-- [ ] **INFO** issues identified (style inconsistencies, missing documentation)
-
-Knowledge fragments referenced:
-
-- [ ] `test-quality.md` for Definition of Done
-- [ ] `fixture-architecture.md` for self-cleaning patterns
-- [ ] `network-first.md` for Playwright best practices
-- [ ] `data-factories.md` for test data patterns
-
----
-
-## Phase 1 Deliverables Generated
-
-### Traceability Matrix Markdown
-
-- [ ] File created at `{output_folder}/traceability-matrix.md`
-- [ ] Template from `trace-template.md` used
-- [ ] Full mapping table included
-- [ ] Coverage status section included
-- [ ] Gap analysis section included
-- [ ] Quality assessment section included
-- [ ] Recommendations section included
-
-### Coverage Badge/Metric (if enabled)
-
-- [ ] Badge markdown generated
-- [ ] Metrics exported to JSON for CI/CD integration
-
-### Updated Story File (if enabled)
-
-- [ ] "Traceability" section added to story markdown
-- [ ] Link to traceability matrix included
-- [ ] Coverage summary included
-
----
-
-## Phase 1 Quality Assurance
-
-### Accuracy Checks
-
-- [ ] All acceptance criteria accounted for (none skipped)
-- [ ] Test IDs correctly formatted (e.g., 1.3-E2E-001)
-- [ ] File paths are correct and accessible
-- [ ] Coverage percentages calculated correctly
-- [ ] No false positives (tests incorrectly mapped to criteria)
-- [ ] No false negatives (existing tests missed in mapping)
-
-### Completeness Checks
-
-- [ ] All test levels considered (E2E, API, Component, Unit)
-- [ ] All priorities considered (P0, P1, P2, P3)
-- [ ] All coverage statuses used appropriately (FULL, PARTIAL, NONE, UNIT-ONLY, INTEGRATION-ONLY)
-- [ ] All gaps have recommendations
-- [ ] All quality issues have severity and remediation guidance
-
-### Actionability Checks
-
-- [ ] Recommendations are specific (not generic)
-- [ ] Test IDs suggested for new tests
-- [ ] Given-When-Then provided for recommended tests
-- [ ] Impact explained for each gap
-- [ ] Priorities clear (CRITICAL, HIGH, MEDIUM, LOW)
-
----
-
-## Phase 1 Documentation
-
-- [ ] Traceability matrix is readable and well-formatted
-- [ ] Tables render correctly in markdown
-- [ ] Code blocks have proper syntax highlighting
-- [ ] Links are valid and accessible
-- [ ] Recommendations are clear and prioritized
-
----
-
-# PHASE 2: QUALITY GATE DECISION
-
-**Note**: Phase 2 executes only if `enable_gate_decision: true` in workflow.yaml
-
----
-
-## Prerequisites
-
-### Evidence Gathering
-
-- [ ] Test execution results obtained (CI/CD pipeline, test framework reports)
-- [ ] Story/epic/release file identified and read
-- [ ] Test design document discovered or explicitly provided (if available)
-- [ ] Traceability matrix discovered or explicitly provided (available from Phase 1)
-- [ ] NFR assessment discovered or explicitly provided (if available)
-- [ ] Code coverage report discovered or explicitly provided (if available)
-- [ ] Burn-in results discovered or explicitly provided (if available)
-
-### Evidence Validation
-
-- [ ] Evidence freshness validated (warn if >7 days old, recommend re-running workflows)
-- [ ] All required assessments available or user acknowledged gaps
-- [ ] Test results are complete (not partial or interrupted runs)
-- [ ] Test results match current codebase (not from outdated branch)
-
-### Knowledge Base Loading
-
-- [ ] `risk-governance.md` loaded successfully
-- [ ] `probability-impact.md` loaded successfully
-- [ ] `test-quality.md` loaded successfully
-- [ ] `test-priorities.md` loaded successfully
-- [ ] `ci-burn-in.md` loaded (if burn-in results available)
-
----
-
-## Process Steps
-
-### Step 1: Context Loading
-
-- [ ] Gate type identified (story/epic/release/hotfix)
-- [ ] Target ID extracted (story_id, epic_num, or release_version)
-- [ ] Decision thresholds loaded from workflow variables
-- [ ] Risk tolerance configuration loaded
-- [ ] Waiver policy loaded
-
-### Step 2: Evidence Parsing
-
-**Test Results:**
-
-- [ ] Total test count extracted
-- [ ] Passed test count extracted
-- [ ] Failed test count extracted
-- [ ] Skipped test count extracted
-- [ ] Test duration extracted
-- [ ] P0 test pass rate calculated
-- [ ] P1 test pass rate calculated
-- [ ] Overall test pass rate calculated
-
-**Quality Assessments:**
-
-- [ ] P0/P1/P2/P3 scenarios extracted from test-design.md (if available)
-- [ ] Risk scores extracted from test-design.md (if available)
-- [ ] Coverage percentages extracted from traceability-matrix.md (available from Phase 1)
-- [ ] Coverage gaps extracted from traceability-matrix.md (available from Phase 1)
-- [ ] NFR status extracted from nfr-assessment.md (if available)
-- [ ] Security issues count extracted from nfr-assessment.md (if available)
-
-**Code Coverage:**
-
-- [ ] Line coverage percentage extracted (if available)
-- [ ] Branch coverage percentage extracted (if available)
-- [ ] Function coverage percentage extracted (if available)
-- [ ] Critical path coverage validated (if available)
-
-**Burn-in Results:**
-
-- [ ] Burn-in iterations count extracted (if available)
-- [ ] Flaky tests count extracted (if available)
-- [ ] Stability score calculated (if available)
-
-### Step 3: Decision Rules Application
-
-**P0 Criteria Evaluation:**
-
-- [ ] P0 test pass rate evaluated (must be 100%)
-- [ ] P0 acceptance criteria coverage evaluated (must be 100%)
-- [ ] Security issues count evaluated (must be 0)
-- [ ] Critical NFR failures evaluated (must be 0)
-- [ ] Flaky tests evaluated (must be 0 if burn-in enabled)
-- [ ] P0 decision recorded: PASS or FAIL
-
-**P1 Criteria Evaluation:**
-
-- [ ] P1 test pass rate evaluated (threshold: min_p1_pass_rate)
-- [ ] P1 acceptance criteria coverage evaluated (threshold: 95%)
-- [ ] Overall test pass rate evaluated (threshold: min_overall_pass_rate)
-- [ ] Code coverage evaluated (threshold: min_coverage)
-- [ ] P1 decision recorded: PASS or CONCERNS
-
-**P2/P3 Criteria Evaluation:**
-
-- [ ] P2 failures tracked (informational, don't block if allow_p2_failures: true)
-- [ ] P3 failures tracked (informational, don't block if allow_p3_failures: true)
-- [ ] Residual risks documented
-
-**Final Decision:**
-
-- [ ] Decision determined: PASS / CONCERNS / FAIL / WAIVED
-- [ ] Decision rationale documented
-- [ ] Decision is deterministic (follows rules, not arbitrary)
-
-### Step 4: Documentation
-
-**Gate Decision Document Created:**
-
-- [ ] Story/epic/release info section complete (ID, title, description, links)
-- [ ] Decision clearly stated (PASS / CONCERNS / FAIL / WAIVED)
-- [ ] Decision date recorded
-- [ ] Evaluator recorded (user or agent name)
-
-**Evidence Summary Documented:**
-
-- [ ] Test results summary complete (total, passed, failed, pass rates)
-- [ ] Coverage summary complete (P0/P1 criteria, code coverage)
-- [ ] NFR validation summary complete (security, performance, reliability, maintainability)
-- [ ] Flakiness summary complete (burn-in iterations, flaky test count)
-
-**Rationale Documented:**
-
-- [ ] Decision rationale clearly explained
-- [ ] Key evidence highlighted
-- [ ] Assumptions and caveats noted (if any)
-
-**Residual Risks Documented (if CONCERNS or WAIVED):**
-
-- [ ] Unresolved P1/P2 issues listed
-- [ ] Probability × impact estimated for each risk
-- [ ] Mitigations or workarounds described
-
-**Waivers Documented (if WAIVED):**
-
-- [ ] Waiver reason documented (business justification)
-- [ ] Waiver approver documented (name, role)
-- [ ] Waiver expiry date documented
-- [ ] Remediation plan documented (fix in next release, due date)
-- [ ] Monitoring plan documented
-
-**Critical Issues Documented (if FAIL or CONCERNS):**
-
-- [ ] Top 5-10 critical issues listed
-- [ ] Priority assigned to each issue (P0/P1/P2)
-- [ ] Owner assigned to each issue
-- [ ] Due date assigned to each issue
-
-**Recommendations Documented:**
-
-- [ ] Next steps clearly stated for decision type
-- [ ] Deployment recommendation provided
-- [ ] Monitoring recommendations provided (if applicable)
-- [ ] Remediation recommendations provided (if applicable)
-
-### Step 5: Status Updates and Notifications
-
-**Status File Updated:**
-
-- [ ] Gate decision appended to bmm-workflow-status.md (if append_to_history: true)
-- [ ] Format correct: `[DATE] Gate Decision: DECISION - Target {ID} - {rationale}`
-- [ ] Status file committed or staged for commit
-
-**Gate YAML Created:**
-
-- [ ] Gate YAML snippet generated with decision and criteria
-- [ ] Evidence references included in YAML
-- [ ] Next steps included in YAML
-- [ ] YAML file saved to output folder
-
-**Stakeholder Notification Generated:**
-
-- [ ] Notification subject line created
-- [ ] Notification body created with summary
-- [ ] Recipients identified (PM, SM, DEV lead, stakeholders)
-- [ ] Notification ready for delivery (if notify_stakeholders: true)
-
-**Outputs Saved:**
-
-- [ ] Gate decision document saved to `{output_file}`
-- [ ] Gate YAML saved to `{output_folder}/gate-decision-{target}.yaml`
-- [ ] All outputs are valid and readable
-
----
-
-## Phase 2 Output Validation
-
-### Gate Decision Document
-
-**Completeness:**
-
-- [ ] All required sections present (info, decision, evidence, rationale, next steps)
-- [ ] No placeholder text or TODOs left in document
-- [ ] All evidence references are accurate and complete
-- [ ] All links to artifacts are valid
-
-**Accuracy:**
-
-- [ ] Decision matches applied criteria rules
-- [ ] Test results match CI/CD pipeline output
-- [ ] Coverage percentages match reports
-- [ ] NFR status matches assessment document
-- [ ] No contradictions or inconsistencies
-
-**Clarity:**
-
-- [ ] Decision rationale is clear and unambiguous
-- [ ] Technical jargon is explained or avoided
-- [ ] Stakeholders can understand next steps
-- [ ] Recommendations are actionable
-
-### Gate YAML
-
-**Format:**
-
-- [ ] YAML is valid (no syntax errors)
-- [ ] All required fields present (target, decision, date, evaluator, criteria, evidence)
-- [ ] Field values are correct data types (numbers, strings, dates)
-
-**Content:**
-
-- [ ] Criteria values match decision document
-- [ ] Evidence references are accurate
-- [ ] Next steps align with decision type
-
----
-
-## Phase 2 Quality Checks
-
-### Decision Integrity
-
-- [ ] Decision is deterministic (follows rules, not arbitrary)
-- [ ] P0 failures result in FAIL decision (unless waived)
-- [ ] Security issues result in FAIL decision (unless waived - but should never be waived)
-- [ ] Waivers have business justification and approver (if WAIVED)
-- [ ] Residual risks are documented (if CONCERNS or WAIVED)
-
-### Evidence-Based
-
-- [ ] Decision is based on actual test results (not guesses)
-- [ ] All claims are supported by evidence
-- [ ] No assumptions without documentation
-- [ ] Evidence sources are cited (CI run IDs, report URLs)
-
-### Transparency
-
-- [ ] Decision rationale is transparent and auditable
-- [ ] Criteria evaluation is documented step-by-step
-- [ ] Any deviations from standard process are explained
-- [ ] Waiver justifications are clear (if applicable)
-
-### Consistency
-
-- [ ] Decision aligns with risk-governance knowledge fragment
-- [ ] Priority framework (P0/P1/P2/P3) applied consistently
-- [ ] Terminology consistent with test-quality knowledge fragment
-- [ ] Decision matrix followed correctly
-
----
-
-## Phase 2 Integration Points
-
-### BMad Workflow Status
-
-- [ ] Gate decision added to `bmm-workflow-status.md`
-- [ ] Format matches existing gate history entries
-- [ ] Timestamp is accurate
-- [ ] Decision summary is concise (<80 chars)
-
-### CI/CD Pipeline
-
-- [ ] Gate YAML is CI/CD-compatible
-- [ ] YAML can be parsed by pipeline automation
-- [ ] Decision can be used to block/allow deployments
-- [ ] Evidence references are accessible to pipeline
-
-### Stakeholders
-
-- [ ] Notification message is clear and actionable
-- [ ] Decision is explained in non-technical terms
-- [ ] Next steps are specific and time-bound
-- [ ] Recipients are appropriate for decision type
-
----
-
-## Phase 2 Compliance and Audit
-
-### Audit Trail
-
-- [ ] Decision date and time recorded
-- [ ] Evaluator identified (user or agent)
-- [ ] All evidence sources cited
-- [ ] Decision criteria documented
-- [ ] Rationale clearly explained
-
-### Traceability
-
-- [ ] Gate decision traceable to story/epic/release
-- [ ] Evidence traceable to specific test runs
-- [ ] Assessments traceable to workflows that created them
-- [ ] Waiver traceable to approver (if applicable)
-
-### Compliance
-
-- [ ] Security requirements validated (no unresolved vulnerabilities)
-- [ ] Quality standards met or waived with justification
-- [ ] Regulatory requirements addressed (if applicable)
-- [ ] Documentation sufficient for external audit
-
----
-
-## Phase 2 Edge Cases and Exceptions
-
-### Missing Evidence
-
-- [ ] If test-design.md missing, decision still possible with test results + trace
-- [ ] If traceability-matrix.md missing, decision still possible with test results (but Phase 1 should provide it)
-- [ ] If nfr-assessment.md missing, NFR validation marked as NOT ASSESSED
-- [ ] If code coverage missing, coverage criterion marked as NOT ASSESSED
-- [ ] User acknowledged gaps in evidence or provided alternative proof
-
-### Stale Evidence
-
-- [ ] Evidence freshness checked (if validate_evidence_freshness: true)
-- [ ] Warnings issued for assessments >7 days old
-- [ ] User acknowledged stale evidence or re-ran workflows
-- [ ] Decision document notes any stale evidence used
-
-### Conflicting Evidence
-
-- [ ] Conflicts between test results and assessments resolved
-- [ ] Most recent/authoritative source identified
-- [ ] Conflict resolution documented in decision rationale
-- [ ] User consulted if conflict cannot be resolved
-
-### Waiver Scenarios
-
-- [ ] Waiver only used for FAIL decision (not PASS or CONCERNS)
-- [ ] Waiver has business justification (not technical convenience)
-- [ ] Waiver has named approver with authority (VP/CTO/PO)
-- [ ] Waiver has expiry date (does NOT apply to future releases)
-- [ ] Waiver has remediation plan with concrete due date
-- [ ] Security vulnerabilities are NOT waived (enforced)
-
----
-
-# FINAL VALIDATION (Both Phases)
-
-## Non-Prescriptive Validation
-
-- [ ] Traceability format adapted to team needs (not rigid template)
-- [ ] Examples are minimal and focused on patterns
-- [ ] Teams can extend with custom classifications
-- [ ] Integration with external systems supported (JIRA, Azure DevOps)
-- [ ] Compliance requirements considered (if applicable)
-
----
-
-## Documentation and Communication
-
-- [ ] All documents are readable and well-formatted
-- [ ] Tables render correctly in markdown
-- [ ] Code blocks have proper syntax highlighting
-- [ ] Links are valid and accessible
-- [ ] Recommendations are clear and prioritized
-- [ ] Gate decision is prominent and unambiguous (Phase 2)
-
----
-
-## Final Validation
-
-**Phase 1 (Traceability):**
-
-- [ ] All prerequisites met
-- [ ] All acceptance criteria mapped or gaps documented
-- [ ] P0 coverage is 100% OR documented as BLOCKER
-- [ ] Gap analysis is complete and prioritized
-- [ ] Test quality issues identified and flagged
-- [ ] Deliverables generated and saved
-
-**Phase 2 (Gate Decision):**
-
-- [ ] All quality evidence gathered
-- [ ] Decision criteria applied correctly
-- [ ] Decision rationale documented
-- [ ] Gate YAML ready for CI/CD integration
-- [ ] Status file updated (if enabled)
-- [ ] Stakeholders notified (if enabled)
-
-**Workflow Complete:**
-
-- [ ] Phase 1 completed successfully
-- [ ] Phase 2 completed successfully (if enabled)
-- [ ] All outputs validated and saved
-- [ ] Ready to proceed based on gate decision
-
----
-
-## Sign-Off
-
-**Phase 1 - Traceability Status:**
-
-- [ ] ✅ PASS - All quality gates met, no critical gaps
-- [ ] ⚠️ WARN - P1 gaps exist, address before PR merge
-- [ ] ❌ FAIL - P0 gaps exist, BLOCKER for release
-
-**Phase 2 - Gate Decision Status (if enabled):**
-
-- [ ] ✅ PASS - Deploy to production
-- [ ] ⚠️ CONCERNS - Deploy with monitoring
-- [ ] ❌ FAIL - Block deployment, fix issues
-- [ ] 🔓 WAIVED - Deploy with business approval and remediation plan
-
-**Next Actions:**
-
-- If PASS (both phases): Proceed to deployment
-- If WARN/CONCERNS: Address gaps/issues, proceed with monitoring
-- If FAIL (either phase): Run `*atdd` for missing tests, fix issues, re-run `*trace`
-- If WAIVED: Deploy with approved waiver, schedule remediation
-
----
-
-## Notes
-
-Record any issues, deviations, or important observations during workflow execution:
-
-- **Phase 1 Issues**: [Note any traceability mapping challenges, missing tests, quality concerns]
-- **Phase 2 Issues**: [Note any missing, stale, or conflicting evidence]
-- **Decision Rationale**: [Document any nuanced reasoning or edge cases]
-- **Waiver Details**: [Document waiver negotiations or approvals]
-- **Follow-up Actions**: [List any actions required after gate decision]
-
----
-
-<!-- Powered by BMAD-CORE™ -->