bmad-method-test-architecture-enterprise 1.26.1-next.8 → 1.27.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +53 -49
- package/cli/atdd-red-check.js +546 -0
- package/cli/atdd-runner.js +166 -0
- package/cli/ci-runner.js +192 -0
- package/cli/examples/pr-test-review.yml +3 -2
- package/cli/fragment-selection-runner.js +3 -3
- package/cli/lib/agent-adapters.js +5 -4
- package/cli/lib/atdd-isolation.js +503 -0
- package/cli/lib/atdd-net-guard.cjs +204 -0
- package/cli/lib/parse-report.js +8 -3
- package/cli/lib/parse-routing.js +6 -6
- package/cli/lib/parse-selection.js +1 -1
- package/cli/lib/registry-rows.js +3 -2
- package/cli/lib/review-provenance.js +5 -1
- package/cli/lib/run-agent.js +4 -2
- package/cli/lib/runner-exit-codes.js +12 -12
- package/cli/nfr-runner.js +3 -3
- package/cli/routing-runner.js +8 -3
- package/cli/test-design-runner.js +4 -4
- package/cli/test-review.js +9 -4
- package/cli/trace-runner.js +5 -5
- package/cli/transcript-runner.js +186 -0
- package/package.json +79 -8
- package/src/agents/bmad-tea/resources/knowledge/burn-in.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/ci-burn-in.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/confidence-gate.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/contract-testing.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/email-auth.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/error-handling.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/evidence-integrity.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/feature-flags.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/fixtures-composition.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/library-integration-mandate.md +2 -2
- package/src/agents/bmad-tea/resources/knowledge/mobile-test-strategy.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/network-error-monitor.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/network-recorder.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/nfr-criteria.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/pact-broker-webhooks.md +6 -6
- package/src/agents/bmad-tea/resources/knowledge/pact-consumer-framework-setup.md +2 -2
- package/src/agents/bmad-tea/resources/knowledge/pact-mcp.md +6 -6
- package/src/agents/bmad-tea/resources/knowledge/pactjs-utils-consumer-helpers.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/pactjs-utils-overview.md +2 -2
- package/src/agents/bmad-tea/resources/knowledge/pactjs-utils-provider-verifier.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/pactjs-utils-request-filter.md +2 -2
- package/src/agents/bmad-tea/resources/knowledge/playwright-cli.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/probability-impact.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/risk-governance.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/selective-testing.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/selector-resilience.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/test-healing-patterns.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/test-levels-framework.md +2 -2
- package/src/agents/bmad-tea/resources/knowledge/test-priorities-matrix.md +2 -2
- package/src/agents/bmad-tea/resources/knowledge/timing-debugging.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/visual-debugging.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/webhook-risk-guidance.md +1 -1
- package/src/agents/bmad-tea/resources/knowledge/webhook-timeout-error.md +1 -1
- package/src/workflows/testarch/bmad-teach-me-testing/resources/tea-completion-summary.example.md +1 -1
- package/src/workflows/testarch/bmad-teach-me-testing/steps-c/step-04-session-01.md +3 -3
- package/src/workflows/testarch/bmad-teach-me-testing/steps-c/step-04-session-02.md +18 -18
- package/src/workflows/testarch/bmad-teach-me-testing/steps-c/step-04-session-04.md +2 -2
- package/src/workflows/testarch/bmad-teach-me-testing/steps-c/step-04-session-06.md +6 -6
- package/src/workflows/testarch/bmad-teach-me-testing/steps-c/step-04-session-07.md +7 -7
- package/src/workflows/testarch/bmad-teach-me-testing/templates/completion-summary-template.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/atdd-checklist-template.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/atdd-checklist-2-4-guest-checkout.example.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/ci-burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/confidence-gate.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/contract-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/email-auth.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/error-handling.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/evidence-integrity.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/feature-flags.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/fixtures-composition.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/library-integration-mandate.md +2 -2
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/mobile-test-strategy.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/network-error-monitor.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/network-recorder.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/nfr-criteria.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/pact-broker-webhooks.md +6 -6
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/pact-consumer-framework-setup.md +2 -2
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/pact-mcp.md +6 -6
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/pactjs-utils-consumer-helpers.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/pactjs-utils-overview.md +2 -2
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/pactjs-utils-provider-verifier.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/pactjs-utils-request-filter.md +2 -2
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/playwright-cli.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/probability-impact.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/risk-governance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/selective-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/selector-resilience.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/test-healing-patterns.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/test-levels-framework.md +2 -2
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/test-priorities-matrix.md +2 -2
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/timing-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/visual-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/webhook-risk-guidance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/resources/knowledge/webhook-timeout-error.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-04-generate-tests.md +6 -6
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-04a-subagent-api-failing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-04b-subagent-e2e-failing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-atdd/steps-c/step-04c-aggregate.md +2 -2
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/ci-burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/confidence-gate.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/contract-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/email-auth.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/error-handling.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/evidence-integrity.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/feature-flags.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/fixtures-composition.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/library-integration-mandate.md +2 -2
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/mobile-test-strategy.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/network-error-monitor.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/network-recorder.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/nfr-criteria.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/pact-broker-webhooks.md +6 -6
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/pact-consumer-framework-setup.md +2 -2
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/pact-mcp.md +6 -6
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/pactjs-utils-consumer-helpers.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/pactjs-utils-overview.md +2 -2
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/pactjs-utils-provider-verifier.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/pactjs-utils-request-filter.md +2 -2
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/playwright-cli.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/probability-impact.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/risk-governance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/selective-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/selector-resilience.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/test-healing-patterns.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/test-levels-framework.md +2 -2
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/test-priorities-matrix.md +2 -2
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/timing-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/visual-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/webhook-risk-guidance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/resources/knowledge/webhook-timeout-error.md +1 -1
- package/src/workflows/testarch/bmad-testarch-automate/steps-c/step-02-identify-targets.md +6 -6
- package/src/workflows/testarch/bmad-testarch-automate/steps-c/step-03-generate-tests.md +8 -8
- package/src/workflows/testarch/bmad-testarch-automate/steps-c/step-03a-subagent-api.md +11 -11
- package/src/workflows/testarch/bmad-testarch-automate/steps-c/step-03c-aggregate.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/ci-burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/confidence-gate.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/contract-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/email-auth.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/error-handling.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/evidence-integrity.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/feature-flags.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/fixtures-composition.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/library-integration-mandate.md +2 -2
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/mobile-test-strategy.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/network-error-monitor.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/network-recorder.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/nfr-criteria.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/pact-broker-webhooks.md +6 -6
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/pact-consumer-framework-setup.md +2 -2
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/pact-mcp.md +6 -6
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/pactjs-utils-consumer-helpers.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/pactjs-utils-overview.md +2 -2
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/pactjs-utils-provider-verifier.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/pactjs-utils-request-filter.md +2 -2
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/playwright-cli.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/probability-impact.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/risk-governance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/selective-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/selector-resilience.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/test-healing-patterns.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/test-levels-framework.md +2 -2
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/test-priorities-matrix.md +2 -2
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/timing-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/visual-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/webhook-risk-guidance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-ci/resources/knowledge/webhook-timeout-error.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/hooks/tea-enforce.cjs +8 -6
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/ci-burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/confidence-gate.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/contract-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/email-auth.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/error-handling.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/evidence-integrity.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/feature-flags.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/fixtures-composition.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/library-integration-mandate.md +2 -2
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/mobile-test-strategy.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/network-error-monitor.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/network-recorder.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/nfr-criteria.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/pact-broker-webhooks.md +6 -6
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/pact-consumer-framework-setup.md +2 -2
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/pact-mcp.md +6 -6
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/pactjs-utils-consumer-helpers.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/pactjs-utils-overview.md +2 -2
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/pactjs-utils-provider-verifier.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/pactjs-utils-request-filter.md +2 -2
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/playwright-cli.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/probability-impact.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/risk-governance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/selective-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/selector-resilience.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/test-healing-patterns.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/test-levels-framework.md +2 -2
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/test-priorities-matrix.md +2 -2
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/timing-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/visual-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/webhook-risk-guidance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-framework/resources/knowledge/webhook-timeout-error.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/checklist.md +6 -2
- package/src/workflows/testarch/bmad-testarch-nfr/nfr-report-template.md +6 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/ci-burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/confidence-gate.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/contract-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/email-auth.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/error-handling.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/evidence-integrity.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/feature-flags.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/fixtures-composition.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/library-integration-mandate.md +2 -2
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/mobile-test-strategy.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/network-error-monitor.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/network-recorder.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/nfr-criteria.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/pact-broker-webhooks.md +6 -6
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/pact-consumer-framework-setup.md +2 -2
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/pact-mcp.md +6 -6
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/pactjs-utils-consumer-helpers.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/pactjs-utils-overview.md +2 -2
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/pactjs-utils-provider-verifier.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/pactjs-utils-request-filter.md +2 -2
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/playwright-cli.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/probability-impact.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/risk-governance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/selective-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/selector-resilience.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/test-healing-patterns.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/test-levels-framework.md +2 -2
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/test-priorities-matrix.md +2 -2
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/timing-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/visual-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/webhook-risk-guidance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/knowledge/webhook-timeout-error.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/resources/nfr-assessment.example.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/nfr-status-definitions.md +8 -0
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-01b-resume.md +1 -1
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04-evaluate-and-score.md +7 -7
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04e-aggregate-nfr.md +38 -1
- package/src/workflows/testarch/bmad-testarch-nfr/steps-c/step-05-generate-report.md +7 -1
- package/src/workflows/testarch/bmad-testarch-test-design/checklist.md +5 -5
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/ci-burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/confidence-gate.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/contract-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/email-auth.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/error-handling.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/evidence-integrity.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/feature-flags.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/fixtures-composition.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/library-integration-mandate.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/mobile-test-strategy.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/network-error-monitor.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/network-recorder.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/nfr-criteria.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/pact-broker-webhooks.md +6 -6
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/pact-consumer-framework-setup.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/pact-mcp.md +6 -6
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/pactjs-utils-consumer-helpers.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/pactjs-utils-overview.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/pactjs-utils-provider-verifier.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/pactjs-utils-request-filter.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/playwright-cli.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/probability-impact.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/risk-governance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/selective-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/selector-resilience.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/test-healing-patterns.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/test-levels-framework.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/test-priorities-matrix.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/timing-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/visual-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/webhook-risk-guidance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/resources/knowledge/webhook-timeout-error.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-design/test-design-architecture-template.md +3 -3
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/ci-burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/confidence-gate.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/contract-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/email-auth.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/error-handling.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/evidence-integrity.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/feature-flags.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/fixtures-composition.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/library-integration-mandate.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/mobile-test-strategy.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/network-error-monitor.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/network-recorder.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/nfr-criteria.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/pact-broker-webhooks.md +6 -6
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/pact-consumer-framework-setup.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/pact-mcp.md +6 -6
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/pactjs-utils-consumer-helpers.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/pactjs-utils-overview.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/pactjs-utils-provider-verifier.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/pactjs-utils-request-filter.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/playwright-cli.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/probability-impact.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/risk-governance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/selective-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/selector-resilience.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/test-healing-patterns.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/test-levels-framework.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/test-priorities-matrix.md +2 -2
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/timing-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/visual-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/webhook-risk-guidance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/knowledge/webhook-timeout-error.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/resources/test-review.example.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/steps-c/step-01b-resume.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/steps-c/step-02-discover-tests.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/steps-c/step-03-quality-evaluation.md +7 -7
- package/src/workflows/testarch/bmad-testarch-test-review/steps-c/step-03f-aggregate-scores.md +1 -1
- package/src/workflows/testarch/bmad-testarch-test-review/test-review-template.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/checklist.md +3 -3
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/ci-burn-in.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/confidence-gate.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/contract-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/email-auth.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/error-handling.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/evidence-integrity.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/feature-flags.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/fixtures-composition.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/library-integration-mandate.md +2 -2
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/mobile-test-strategy.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/network-error-monitor.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/network-recorder.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/nfr-criteria.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/pact-broker-webhooks.md +6 -6
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/pact-consumer-framework-setup.md +2 -2
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/pact-mcp.md +6 -6
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/pactjs-utils-consumer-helpers.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/pactjs-utils-overview.md +2 -2
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/pactjs-utils-provider-verifier.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/pactjs-utils-request-filter.md +2 -2
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/playwright-cli.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/probability-impact.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/risk-governance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/selective-testing.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/selector-resilience.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/test-healing-patterns.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/test-levels-framework.md +2 -2
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/test-priorities-matrix.md +2 -2
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/timing-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/visual-debugging.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/webhook-risk-guidance.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/resources/knowledge/webhook-timeout-error.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/steps-c/step-02-discover-tests.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/steps-c/step-04-analyze-gaps.md +2 -2
- package/src/workflows/testarch/bmad-testarch-trace/steps-c/step-05-gate-decision.md +1 -1
- package/src/workflows/testarch/bmad-testarch-trace/trace-template.md +3 -3
- package/src/workflows/testarch/bmad-testarch-trace/workflow.yaml +3 -2
- package/.coderabbit.yaml +0 -40
- package/.gitattributes +0 -21
- package/.github/CODE_OF_CONDUCT.md +0 -128
- package/.github/FUNDING.yaml +0 -15
- package/.github/ISSUE_TEMPLATE/config.yaml +0 -8
- package/.github/ISSUE_TEMPLATE/feature_request.md +0 -70
- package/.github/ISSUE_TEMPLATE/issue.md +0 -61
- package/.github/ISSUE_TEMPLATE/rule_quality_report.md +0 -46
- package/.github/workflows/docs.yaml +0 -66
- package/.github/workflows/publish.yaml +0 -193
- package/.github/workflows/quality.yaml +0 -257
- package/.github/workflows/tea-test-review.yaml +0 -279
- package/.husky/pre-commit +0 -20
- package/.markdownlint-cli2.yaml +0 -36
- package/.nvmrc +0 -1
- package/.prettierignore +0 -14
- package/.vscode/settings.json +0 -47
- package/AGENTS.md +0 -32
- package/CHANGELOG.md +0 -976
- package/CONTRIBUTING.md +0 -268
- package/DESIGN-CRITERIA-REGISTRY.md +0 -350
- package/SECURITY.md +0 -85
- package/docs/404.md +0 -20
- package/docs/explanation/engagement-models.md +0 -191
- package/docs/explanation/eval-quality-adoption-guide.md +0 -390
- package/docs/explanation/eval-quality-command-adapter.md +0 -418
- package/docs/explanation/eval-quality-roadmap.md +0 -176
- package/docs/explanation/fixture-architecture.md +0 -350
- package/docs/explanation/knowledge-base-system.md +0 -158
- package/docs/explanation/network-first-patterns.md +0 -342
- package/docs/explanation/risk-based-testing.md +0 -189
- package/docs/explanation/step-file-architecture.md +0 -227
- package/docs/explanation/tea-overview.md +0 -244
- package/docs/explanation/test-quality-standards.md +0 -475
- package/docs/explanation/test-review-cli-architecture.md +0 -159
- package/docs/explanation/testing-as-engineering.md +0 -124
- package/docs/explanation/verification-architecture.md +0 -129
- package/docs/glossary/index.md +0 -160
- package/docs/how-to/brownfield/use-tea-for-enterprise.md +0 -394
- package/docs/how-to/brownfield/use-tea-with-existing-tests.md +0 -499
- package/docs/how-to/customization/configure-browser-automation.md +0 -213
- package/docs/how-to/customization/extend-tea-with-custom-workflows.md +0 -102
- package/docs/how-to/customization/integrate-pactjs-utils.md +0 -231
- package/docs/how-to/customization/integrate-playwright-utils.md +0 -886
- package/docs/how-to/install-behind-firewall.md +0 -43
- package/docs/how-to/workflows/run-atdd.md +0 -453
- package/docs/how-to/workflows/run-automate.md +0 -673
- package/docs/how-to/workflows/run-nfr-assess.md +0 -723
- package/docs/how-to/workflows/run-test-design.md +0 -155
- package/docs/how-to/workflows/run-test-review.md +0 -622
- package/docs/how-to/workflows/run-trace.md +0 -973
- package/docs/how-to/workflows/setup-ci.md +0 -741
- package/docs/how-to/workflows/setup-test-framework.md +0 -130
- package/docs/how-to/workflows/teach-me-testing.md +0 -287
- package/docs/index.md +0 -67
- package/docs/reference/commands.md +0 -394
- package/docs/reference/configuration.md +0 -561
- package/docs/reference/execution-targets.md +0 -106
- package/docs/reference/knowledge-base.md +0 -271
- package/docs/reference/live-verification-results.md +0 -193
- package/docs/reference/tea-test-review-cli.md +0 -376
- package/docs/reference/troubleshooting.md +0 -362
- package/docs/tutorials/learn-testing-tea-academy.md +0 -67
- package/docs/tutorials/tea-lite-quickstart.md +0 -459
- package/eslint.config.mjs +0 -146
- package/prettier.config.mjs +0 -32
- package/test/README.md +0 -348
- package/test/contracts/README.md +0 -436
- package/test/contracts/expected-status.json +0 -58
- package/test/contracts/fragment-selection/bmad-testarch-atdd.contract.json +0 -562
- package/test/contracts/fragment-selection/bmad-testarch-automate.contract.json +0 -804
- package/test/contracts/fragment-selection/bmad-testarch-ci.contract.json +0 -459
- package/test/contracts/fragment-selection/bmad-testarch-framework.contract.json +0 -563
- package/test/contracts/fragment-selection/bmad-testarch-nfr.contract.json +0 -434
- package/test/contracts/fragment-selection/bmad-testarch-test-design.contract.json +0 -771
- package/test/contracts/fragment-selection/bmad-testarch-test-review.contract.json +0 -431
- package/test/contracts/fragment-selection/bmad-testarch-trace.contract.json +0 -428
- package/test/contracts/nfr.contract.json +0 -730
- package/test/contracts/tea-routing-controls.contract.json +0 -1633
- package/test/contracts/tea-routing-intents.contract.json +0 -2468
- package/test/contracts/test-design.contract.json +0 -1021
- package/test/contracts/test-review.contract.json +0 -1264
- package/test/contracts/trace.contract.json +0 -2764
- package/test/eval-all.js +0 -413
- package/test/eval-bmad-tea-routing.js +0 -1275
- package/test/eval-contract-strength.js +0 -613
- package/test/eval-fragment-selection.js +0 -830
- package/test/eval-nfr.js +0 -2079
- package/test/eval-test-design.js +0 -2008
- package/test/eval-test-review.js +0 -1001
- package/test/eval-trace.js +0 -2710
- package/test/evals/bmad-testarch-atdd/evals.json +0 -98
- package/test/evals/bmad-testarch-automate/evals.json +0 -198
- package/test/evals/bmad-testarch-ci/evals.json +0 -78
- package/test/evals/bmad-testarch-framework/evals.json +0 -111
- package/test/evals/bmad-testarch-nfr/evals.json +0 -79
- package/test/evals/bmad-testarch-test-design/evals.json +0 -136
- package/test/evals/bmad-testarch-test-review/evals.json +0 -76
- package/test/evals/bmad-testarch-trace/evals.json +0 -62
- package/test/evals/suite-manifest.json +0 -407
- package/test/fixtures/agent-schema/invalid/critical-actions/actions-as-string.agent.yaml +0 -27
- package/test/fixtures/agent-schema/invalid/critical-actions/empty-string-in-actions.agent.yaml +0 -30
- package/test/fixtures/agent-schema/invalid/menu/empty-menu.agent.yaml +0 -22
- package/test/fixtures/agent-schema/invalid/menu/missing-menu.agent.yaml +0 -20
- package/test/fixtures/agent-schema/invalid/menu-commands/empty-command-target.agent.yaml +0 -25
- package/test/fixtures/agent-schema/invalid/menu-commands/no-command-target.agent.yaml +0 -24
- package/test/fixtures/agent-schema/invalid/menu-triggers/camel-case.agent.yaml +0 -25
- package/test/fixtures/agent-schema/invalid/menu-triggers/compound-invalid-format.agent.yaml +0 -25
- package/test/fixtures/agent-schema/invalid/menu-triggers/compound-mismatched-kebab.agent.yaml +0 -25
- package/test/fixtures/agent-schema/invalid/menu-triggers/duplicate-triggers.agent.yaml +0 -31
- package/test/fixtures/agent-schema/invalid/menu-triggers/empty-trigger.agent.yaml +0 -25
- package/test/fixtures/agent-schema/invalid/menu-triggers/leading-asterisk.agent.yaml +0 -25
- package/test/fixtures/agent-schema/invalid/menu-triggers/snake-case.agent.yaml +0 -25
- package/test/fixtures/agent-schema/invalid/menu-triggers/trigger-with-spaces.agent.yaml +0 -25
- package/test/fixtures/agent-schema/invalid/metadata/empty-module-string.agent.yaml +0 -26
- package/test/fixtures/agent-schema/invalid/metadata/empty-name.agent.yaml +0 -24
- package/test/fixtures/agent-schema/invalid/metadata/extra-metadata-fields.agent.yaml +0 -27
- package/test/fixtures/agent-schema/invalid/metadata/missing-id.agent.yaml +0 -23
- package/test/fixtures/agent-schema/invalid/persona/empty-principles-array.agent.yaml +0 -24
- package/test/fixtures/agent-schema/invalid/persona/empty-string-in-principles.agent.yaml +0 -27
- package/test/fixtures/agent-schema/invalid/persona/extra-persona-fields.agent.yaml +0 -27
- package/test/fixtures/agent-schema/invalid/persona/missing-role.agent.yaml +0 -24
- package/test/fixtures/agent-schema/invalid/prompts/empty-content.agent.yaml +0 -29
- package/test/fixtures/agent-schema/invalid/prompts/extra-prompt-fields.agent.yaml +0 -31
- package/test/fixtures/agent-schema/invalid/prompts/missing-content.agent.yaml +0 -28
- package/test/fixtures/agent-schema/invalid/prompts/missing-id.agent.yaml +0 -28
- package/test/fixtures/agent-schema/invalid/top-level/empty-file.agent.yaml +0 -5
- package/test/fixtures/agent-schema/invalid/top-level/extra-top-level-keys.agent.yaml +0 -28
- package/test/fixtures/agent-schema/invalid/top-level/missing-agent-key.agent.yaml +0 -11
- package/test/fixtures/agent-schema/invalid/yaml-errors/invalid-indentation.agent.yaml +0 -19
- package/test/fixtures/agent-schema/invalid/yaml-errors/malformed-yaml.agent.yaml +0 -18
- package/test/fixtures/agent-schema/valid/critical-actions/empty-critical-actions.agent.yaml +0 -24
- package/test/fixtures/agent-schema/valid/critical-actions/no-critical-actions.agent.yaml +0 -22
- package/test/fixtures/agent-schema/valid/critical-actions/valid-critical-actions.agent.yaml +0 -27
- package/test/fixtures/agent-schema/valid/menu/multiple-menu-items.agent.yaml +0 -31
- package/test/fixtures/agent-schema/valid/menu/single-menu-item.agent.yaml +0 -22
- package/test/fixtures/agent-schema/valid/menu-commands/all-command-types.agent.yaml +0 -38
- package/test/fixtures/agent-schema/valid/menu-commands/multiple-commands.agent.yaml +0 -24
- package/test/fixtures/agent-schema/valid/menu-triggers/compound-triggers.agent.yaml +0 -31
- package/test/fixtures/agent-schema/valid/menu-triggers/kebab-case-triggers.agent.yaml +0 -34
- package/test/fixtures/agent-schema/valid/metadata/core-agent-with-module.agent.yaml +0 -24
- package/test/fixtures/agent-schema/valid/metadata/empty-module-name-in-path.agent.yaml +0 -24
- package/test/fixtures/agent-schema/valid/metadata/malformed-path-treated-as-core.agent.yaml +0 -24
- package/test/fixtures/agent-schema/valid/metadata/module-agent-correct.agent.yaml +0 -24
- package/test/fixtures/agent-schema/valid/metadata/module-agent-missing-module.agent.yaml +0 -23
- package/test/fixtures/agent-schema/valid/metadata/wrong-module-value.agent.yaml +0 -24
- package/test/fixtures/agent-schema/valid/persona/complete-persona.agent.yaml +0 -24
- package/test/fixtures/agent-schema/valid/prompts/empty-prompts.agent.yaml +0 -24
- package/test/fixtures/agent-schema/valid/prompts/no-prompts.agent.yaml +0 -22
- package/test/fixtures/agent-schema/valid/prompts/valid-prompts-minimal.agent.yaml +0 -28
- package/test/fixtures/agent-schema/valid/prompts/valid-prompts-with-description.agent.yaml +0 -30
- package/test/fixtures/agent-schema/valid/top-level/minimal-core-agent.agent.yaml +0 -24
- package/test/fixtures/fragment-selection-runner/stub-agent.js +0 -78
- package/test/fixtures/nfr-eval/README.md +0 -64
- package/test/fixtures/nfr-eval/clean/config/logging.json +0 -14
- package/test/fixtures/nfr-eval/clean/docs/tech-spec.md +0 -65
- package/test/fixtures/nfr-eval/clean/evidence/coverage-summary-2026-09-01.json +0 -15
- package/test/fixtures/nfr-eval/clean/evidence/dependency-scan-2026-09-01.json +0 -10
- package/test/fixtures/nfr-eval/clean/evidence/duplication-report-2026-09-01.json +0 -10
- package/test/fixtures/nfr-eval/clean/evidence/load-test-2026-09-02.json +0 -20
- package/test/fixtures/nfr-eval/clean/evidence/reliability-2026-09.json +0 -17
- package/test/fixtures/nfr-eval/clean/evidence/security-review-2026-09-01.md +0 -55
- package/test/fixtures/nfr-eval/gapped/config/logging.json +0 -14
- package/test/fixtures/nfr-eval/gapped/docs/tech-spec.md +0 -67
- package/test/fixtures/nfr-eval/gapped/evidence/dependency-scan-2026-08-30.json +0 -12
- package/test/fixtures/nfr-eval/gapped/evidence/load-test-2026-08-28.json +0 -19
- package/test/fixtures/nfr-eval/gapped/evidence/reliability-2026-08.json +0 -20
- package/test/fixtures/nfr-eval/gapped/evidence/security-review-2026-08-29.md +0 -50
- package/test/fixtures/nfr-eval/ground-truth.json +0 -619
- package/test/fixtures/nfr-runner/stub-agent.js +0 -102
- package/test/fixtures/probe-conformance/fixture-command.js +0 -56
- package/test/fixtures/tea-routing-eval/ground-truth.json +0 -204
- package/test/fixtures/tea-routing-eval/intents.json +0 -94
- package/test/fixtures/test-design-eval/clean/docs/epics/epic-9-last-sync-indicator.md +0 -47
- package/test/fixtures/test-design-eval/ground-truth.json +0 -478
- package/test/fixtures/test-design-eval/seeded/docs/epics/epic-7-offline-order-capture.md +0 -75
- package/test/fixtures/test-design-runner/stub-agent.js +0 -134
- package/test/fixtures/test-review-cli/project/_bmad/tea/workflows/testarch/bmad-testarch-test-review/SKILL.md +0 -1
- package/test/fixtures/test-review-cli/project-claude/.claude/skills/bmad-testarch-test-review/SKILL.md +0 -1
- package/test/fixtures/test-review-cli/project-empty/.gitkeep +0 -0
- package/test/fixtures/test-review-cli/reports/approve-low-score.md +0 -54
- package/test/fixtures/test-review-cli/reports/approve-with-high.md +0 -61
- package/test/fixtures/test-review-cli/reports/approve.md +0 -54
- package/test/fixtures/test-review-cli/reports/bad-value.md +0 -38
- package/test/fixtures/test-review-cli/reports/block.md +0 -102
- package/test/fixtures/test-review-cli/reports/bonus-not-multiple.md +0 -71
- package/test/fixtures/test-review-cli/reports/colon-in-bold.md +0 -51
- package/test/fixtures/test-review-cli/reports/conflicting.md +0 -38
- package/test/fixtures/test-review-cli/reports/context-basis-without-manifest.md +0 -61
- package/test/fixtures/test-review-cli/reports/context-none-with-manifest.md +0 -66
- package/test/fixtures/test-review-cli/reports/context-overlaps-reviewed.md +0 -67
- package/test/fixtures/test-review-cli/reports/context-pr-diff.md +0 -86
- package/test/fixtures/test-review-cli/reports/convention-baseline-unavailable.md +0 -63
- package/test/fixtures/test-review-cli/reports/convention-fabricated.md +0 -64
- package/test/fixtures/test-review-cli/reports/convention-honest-absent.md +0 -73
- package/test/fixtures/test-review-cli/reports/critical-approve.md +0 -38
- package/test/fixtures/test-review-cli/reports/critical-raw-100.md +0 -51
- package/test/fixtures/test-review-cli/reports/duplicate-breakdown-heading.md +0 -87
- package/test/fixtures/test-review-cli/reports/duplicate-finding.md +0 -69
- package/test/fixtures/test-review-cli/reports/empty-steps-flow.md +0 -56
- package/test/fixtures/test-review-cli/reports/fenced-fake-finding.md +0 -72
- package/test/fixtures/test-review-cli/reports/fenced-recommendation-crlf.md +0 -59
- package/test/fixtures/test-review-cli/reports/fenced-recommendation.md +0 -59
- package/test/fixtures/test-review-cli/reports/finding-without-location.md +0 -67
- package/test/fixtures/test-review-cli/reports/findings-multi-severity.md +0 -94
- package/test/fixtures/test-review-cli/reports/key-strengths-weaknesses.md +0 -78
- package/test/fixtures/test-review-cli/reports/lowercase.md +0 -53
- package/test/fixtures/test-review-cli/reports/malformed.md +0 -12
- package/test/fixtures/test-review-cli/reports/missing-breakdown.md +0 -46
- package/test/fixtures/test-review-cli/reports/missing-context-basis.md +0 -59
- package/test/fixtures/test-review-cli/reports/missing-decision.md +0 -53
- package/test/fixtures/test-review-cli/reports/missing-frontmatter.md +0 -45
- package/test/fixtures/test-review-cli/reports/missing-reviewed-files.md +0 -49
- package/test/fixtures/test-review-cli/reports/missing-score.md +0 -52
- package/test/fixtures/test-review-cli/reports/missing-violations.md +0 -36
- package/test/fixtures/test-review-cli/reports/mixed-fence-recommendation.md +0 -64
- package/test/fixtures/test-review-cli/reports/nested-fence-recommendation.md +0 -66
- package/test/fixtures/test-review-cli/reports/plain-bullets-key-strengths.md +0 -62
- package/test/fixtures/test-review-cli/reports/request-changes-critical.md +0 -98
- package/test/fixtures/test-review-cli/reports/request-changes.md +0 -91
- package/test/fixtures/test-review-cli/reports/score-140.md +0 -38
- package/test/fixtures/test-review-cli/reports/score-mismatch.md +0 -57
- package/test/fixtures/test-review-cli/reports/spaced-path-location.md +0 -68
- package/test/fixtures/test-review-cli/reports/table-breakdown.md +0 -65
- package/test/fixtures/test-review-cli/reports/tilde-fence-recommendation.md +0 -61
- package/test/fixtures/test-review-cli/reports/wrapped-steps-flow.md +0 -58
- package/test/fixtures/test-review-cli/stub-agent.js +0 -420
- package/test/fixtures/test-review-eval/clean/profile.spec.ts +0 -36
- package/test/fixtures/test-review-eval/ground-truth.json +0 -200
- package/test/fixtures/test-review-eval/seeded/checkout.spec.ts +0 -42
- package/test/fixtures/test-review-eval/seeded/orders.service.spec.ts +0 -40
- package/test/fixtures/trace-eval/README.md +0 -50
- package/test/fixtures/trace-eval/clean/docs/epics/epic-5-api-token-lifecycle.md +0 -50
- package/test/fixtures/trace-eval/clean/src/tokens/token-list.tsx +0 -23
- package/test/fixtures/trace-eval/clean/tests/api/credential-lifetime.api.spec.ts +0 -43
- package/test/fixtures/trace-eval/clean/tests/api/token-revocation.api.spec.ts +0 -49
- package/test/fixtures/trace-eval/clean/tests/component/token-list.spec.tsx +0 -23
- package/test/fixtures/trace-eval/clean/tests/e2e/token-management.spec.ts +0 -23
- package/test/fixtures/trace-eval/ground-truth.json +0 -963
- package/test/fixtures/trace-eval/seeded/docs/epics/epic-4-tenant-data-export-and-erasure.md +0 -74
- package/test/fixtures/trace-eval/seeded/src/exports/manifest.ts +0 -20
- package/test/fixtures/trace-eval/seeded/test-artifacts/gate-waivers.md +0 -55
- package/test/fixtures/trace-eval/seeded/test-artifacts/live-verification-results.json +0 -22
- package/test/fixtures/trace-eval/seeded/tests/api/audit-log.api.spec.ts +0 -26
- package/test/fixtures/trace-eval/seeded/tests/api/tenant-export.api.spec.ts +0 -96
- package/test/fixtures/trace-eval/seeded/tests/e2e/tenant-export.spec.ts +0 -46
- package/test/fixtures/trace-eval/seeded/tests/unit/export-manifest.spec.ts +0 -19
- package/test/fixtures/trace-runner/stub-agent.js +0 -126
- package/test/lib/bounded-probe.js +0 -72
- package/test/lib/clock.js +0 -151
- package/test/lib/conformance-counts.js +0 -82
- package/test/lib/corpus-port.js +0 -151
- package/test/lib/eval-quality-inputs.js +0 -346
- package/test/lib/eval-record.js +0 -364
- package/test/lib/file-system-port.js +0 -251
- package/test/lib/probe-scoring.js +0 -1311
- package/test/lib/probe-targets.js +0 -546
- package/test/lib/runner-capabilities.js +0 -80
- package/test/lib/suite-manifest.js +0 -98
- package/test/lib/tea-skills.js +0 -45
- package/test/probes/README.md +0 -189
- package/test/probes/expected-strength.json +0 -1278
- package/test/probes/fragment-selection/bmad-testarch-atdd.probes.json +0 -107
- package/test/probes/fragment-selection/bmad-testarch-automate.probes.json +0 -107
- package/test/probes/fragment-selection/bmad-testarch-ci.probes.json +0 -114
- package/test/probes/fragment-selection/bmad-testarch-framework.probes.json +0 -107
- package/test/probes/fragment-selection/bmad-testarch-nfr.probes.json +0 -114
- package/test/probes/fragment-selection/bmad-testarch-test-design.probes.json +0 -107
- package/test/probes/fragment-selection/bmad-testarch-test-review.probes.json +0 -107
- package/test/probes/fragment-selection/bmad-testarch-trace.probes.json +0 -114
- package/test/probes/nfr.probes.json +0 -526
- package/test/probes/scoring-policy.json +0 -13
- package/test/probes/tea-routing-controls.probes.json +0 -118
- package/test/probes/tea-routing-intents.probes.json +0 -106
- package/test/probes/test-design.probes.json +0 -2627
- package/test/probes/test-review.probes.json +0 -1571
- package/test/probes/trace.probes.json +0 -391
- package/test/replay/bmad-tea-routing/clarification-correct/expected.json +0 -146
- package/test/replay/bmad-tea-routing/clarification-correct/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/clarification-names-both-by-label/expected.json +0 -144
- package/test/replay/bmad-tea-routing/clarification-names-both-by-label/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/clarification-names-one-side/expected.json +0 -144
- package/test/replay/bmad-tea-routing/clarification-names-one-side/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/clear-intent-left-unrouted/expected.json +0 -147
- package/test/replay/bmad-tea-routing/clear-intent-left-unrouted/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/confident-route-on-ambiguous/expected.json +0 -152
- package/test/replay/bmad-tea-routing/confident-route-on-ambiguous/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/confident-route-on-unservable/expected.json +0 -145
- package/test/replay/bmad-tea-routing/confident-route-on-unservable/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/correct-route/expected.json +0 -146
- package/test/replay/bmad-tea-routing/correct-route/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/decline-correct/expected.json +0 -143
- package/test/replay/bmad-tea-routing/decline-correct/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/decline-without-saying-what-is-missing/expected.json +0 -142
- package/test/replay/bmad-tea-routing/decline-without-saying-what-is-missing/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/fenced-json/expected.json +0 -144
- package/test/replay/bmad-tea-routing/fenced-json/stdout.txt +0 -13
- package/test/replay/bmad-tea-routing/gate-routed-as-a-workflow/expected.json +0 -146
- package/test/replay/bmad-tea-routing/gate-routed-as-a-workflow/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/prose-no-json/expected.json +0 -109
- package/test/replay/bmad-tea-routing/prose-no-json/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/reason-not-grounded/expected.json +0 -144
- package/test/replay/bmad-tea-routing/reason-not-grounded/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/route-in-the-messages-own-words/expected.json +0 -142
- package/test/replay/bmad-tea-routing/route-in-the-messages-own-words/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/scope-dropped/expected.json +0 -143
- package/test/replay/bmad-tea-routing/scope-dropped/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/scope-echoes-the-message/expected.json +0 -144
- package/test/replay/bmad-tea-routing/scope-echoes-the-message/stdout.txt +0 -1
- package/test/replay/bmad-tea-routing/wrong-menu-code/expected.json +0 -148
- package/test/replay/bmad-tea-routing/wrong-menu-code/stdout.txt +0 -1
- package/test/replay/fragment-selection/bare-json/expected.json +0 -73
- package/test/replay/fragment-selection/bare-json/stdout.txt +0 -1
- package/test/replay/fragment-selection/fenced-json/expected.json +0 -74
- package/test/replay/fragment-selection/fenced-json/stdout.txt +0 -17
- package/test/replay/fragment-selection/prose-no-json/expected.json +0 -52
- package/test/replay/fragment-selection/prose-no-json/stdout.txt +0 -5
- package/test/replay/nfr/clean-correct-audit/expected.json +0 -113
- package/test/replay/nfr/clean-correct-audit/test-artifacts/nfr-assessment.md +0 -222
- package/test/replay/nfr/clean-gaps-invented/expected.json +0 -112
- package/test/replay/nfr/clean-gaps-invented/test-artifacts/nfr-assessment.md +0 -227
- package/test/replay/nfr/clean-gaps-invented-off-template/expected.json +0 -136
- package/test/replay/nfr/clean-gaps-invented-off-template/test-artifacts/nfr-assessment.md +0 -235
- package/test/replay/nfr/gapped-correct-audit/expected.json +0 -114
- package/test/replay/nfr/gapped-correct-audit/test-artifacts/nfr-assessment.md +0 -238
- package/test/replay/nfr/gapped-domain-omitted/expected.json +0 -112
- package/test/replay/nfr/gapped-domain-omitted/test-artifacts/nfr-assessment.md +0 -201
- package/test/replay/nfr/gapped-example-quoted-in-fence/expected.json +0 -133
- package/test/replay/nfr/gapped-example-quoted-in-fence/test-artifacts/nfr-assessment.md +0 -466
- package/test/replay/nfr/gapped-fabricated-evidence/expected.json +0 -108
- package/test/replay/nfr/gapped-fabricated-evidence/test-artifacts/nfr-assessment.md +0 -238
- package/test/replay/nfr/gapped-fabricated-evidence-on-a-source-label/expected.json +0 -132
- package/test/replay/nfr/gapped-fabricated-evidence-on-a-source-label/test-artifacts/nfr-assessment.md +0 -238
- package/test/replay/nfr/gapped-fault-tolerance-grounded-on-spec/expected.json +0 -126
- package/test/replay/nfr/gapped-fault-tolerance-grounded-on-spec/test-artifacts/nfr-assessment.md +0 -241
- package/test/replay/nfr/gapped-maintainability-passed-on-prose/expected.json +0 -110
- package/test/replay/nfr/gapped-maintainability-passed-on-prose/test-artifacts/nfr-assessment.md +0 -229
- package/test/replay/nfr/gapped-maintainability-status-unreadable/expected.json +0 -116
- package/test/replay/nfr/gapped-maintainability-status-unreadable/test-artifacts/nfr-assessment.md +0 -238
- package/test/replay/nfr/gapped-performance-passed/expected.json +0 -112
- package/test/replay/nfr/gapped-performance-passed/test-artifacts/nfr-assessment.md +0 -235
- package/test/replay/nfr/gapped-report-without-sections/expected.json +0 -32
- package/test/replay/nfr/gapped-report-without-sections/test-artifacts/nfr-assessment.md +0 -28
- package/test/replay/nfr/gapped-security-assessed-twice/expected.json +0 -116
- package/test/replay/nfr/gapped-security-assessed-twice/test-artifacts/nfr-assessment.md +0 -264
- package/test/replay/nfr/gapped-two-domains-misjudged/expected.json +0 -118
- package/test/replay/nfr/gapped-two-domains-misjudged/test-artifacts/nfr-assessment.md +0 -242
- package/test/replay/test-design/clean-correct-run/design.md +0 -29
- package/test/replay/test-design/clean-correct-run/expected.json +0 -105
- package/test/replay/test-design/clean-over-reported/design.md +0 -44
- package/test/replay/test-design/clean-over-reported/expected.json +0 -130
- package/test/replay/test-design/seeded-arithmetic-off/design.md +0 -45
- package/test/replay/test-design/seeded-arithmetic-off/expected.json +0 -125
- package/test/replay/test-design/seeded-band-misfiled/design.md +0 -45
- package/test/replay/test-design/seeded-band-misfiled/expected.json +0 -117
- package/test/replay/test-design/seeded-correct-run/design.md +0 -45
- package/test/replay/test-design/seeded-correct-run/expected.json +0 -120
- package/test/replay/test-design/seeded-coverage-level-unadmitted/design.md +0 -45
- package/test/replay/test-design/seeded-coverage-level-unadmitted/expected.json +0 -115
- package/test/replay/test-design/seeded-generic-register/design.md +0 -38
- package/test/replay/test-design/seeded-generic-register/expected.json +0 -145
- package/test/replay/test-design/seeded-priority-inverted/design.md +0 -45
- package/test/replay/test-design/seeded-priority-inverted/expected.json +0 -141
- package/test/replay/test-design/seeded-register-absent/design.md +0 -14
- package/test/replay/test-design/seeded-register-absent/expected.json +0 -45
- package/test/replay/test-design/seeded-shape-defects/design.md +0 -35
- package/test/replay/test-design/seeded-shape-defects/expected.json +0 -153
- package/test/replay/test-design/seeded-unbanded-register/design.md +0 -35
- package/test/replay/test-design/seeded-unbanded-register/expected.json +0 -141
- package/test/replay/test-review/approved-with-no-findings/expected.json +0 -77
- package/test/replay/test-review/approved-with-no-findings/verdict.json +0 -61
- package/test/replay/test-review/clean-false-positive/expected.json +0 -59
- package/test/replay/test-review/clean-false-positive/report.md +0 -126
- package/test/replay/test-review/clean-false-positive/verdict.json +0 -218
- package/test/replay/test-review/enclosing-declarations/expected.json +0 -70
- package/test/replay/test-review/enclosing-declarations/report.md +0 -114
- package/test/replay/test-review/enclosing-declarations/verdict.json +0 -190
- package/test/replay/test-review/full-recall/expected.json +0 -58
- package/test/replay/test-review/full-recall/report.md +0 -114
- package/test/replay/test-review/full-recall/verdict.json +0 -195
- package/test/replay/test-review/misses-two/expected.json +0 -60
- package/test/replay/test-review/misses-two/report.md +0 -106
- package/test/replay/test-review/misses-two/verdict.json +0 -162
- package/test/replay/test-review/out-of-scope-finding/expected.json +0 -54
- package/test/replay/test-review/out-of-scope-finding/report.md +0 -120
- package/test/replay/test-review/out-of-scope-finding/verdict.json +0 -204
- package/test/replay/test-review/partially-attributed/expected.json +0 -72
- package/test/replay/test-review/partially-attributed/report.md +0 -106
- package/test/replay/test-review/partially-attributed/verdict.json +0 -167
- package/test/replay/test-review/row-without-location/expected.json +0 -69
- package/test/replay/test-review/row-without-location/report.md +0 -119
- package/test/replay/test-review/row-without-location/verdict.json +0 -209
- package/test/replay/test-review/unattributed-violations/expected.json +0 -81
- package/test/replay/test-review/unattributed-violations/verdict.json +0 -63
- package/test/replay/test-review/verdict-without-findings/expected.json +0 -39
- package/test/replay/test-review/verdict-without-findings/verdict.json +0 -62
- package/test/replay/trace/clean-ac4-name-matched/expected.json +0 -91
- package/test/replay/trace/clean-ac4-name-matched/test-artifacts/e2e-trace-summary.json +0 -151
- package/test/replay/trace/clean-ac4-name-matched/test-artifacts/traceability-matrix.md +0 -82
- package/test/replay/trace/clean-correct-run/expected.json +0 -71
- package/test/replay/trace/clean-correct-run/test-artifacts/e2e-trace-summary.json +0 -146
- package/test/replay/trace/clean-correct-run/test-artifacts/traceability-matrix.md +0 -81
- package/test/replay/trace/clean-empty-waivers-block/expected.json +0 -61
- package/test/replay/trace/clean-empty-waivers-block/test-artifacts/e2e-trace-summary.json +0 -152
- package/test/replay/trace/clean-empty-waivers-block/test-artifacts/traceability-matrix.md +0 -81
- package/test/replay/trace/clean-matrix-without-sections/expected.json +0 -35
- package/test/replay/trace/clean-matrix-without-sections/test-artifacts/e2e-trace-summary.json +0 -146
- package/test/replay/trace/clean-matrix-without-sections/test-artifacts/traceability-matrix.md +0 -53
- package/test/replay/trace/seeded-ac2-title-matched/expected.json +0 -124
- package/test/replay/trace/seeded-ac2-title-matched/test-artifacts/e2e-trace-summary.json +0 -203
- package/test/replay/trace/seeded-ac2-title-matched/test-artifacts/traceability-matrix.md +0 -135
- package/test/replay/trace/seeded-arithmetic-off/expected.json +0 -69
- package/test/replay/trace/seeded-arithmetic-off/test-artifacts/e2e-trace-summary.json +0 -217
- package/test/replay/trace/seeded-arithmetic-off/test-artifacts/traceability-matrix.md +0 -138
- package/test/replay/trace/seeded-correct-run/expected.json +0 -90
- package/test/replay/trace/seeded-correct-run/test-artifacts/e2e-trace-summary.json +0 -217
- package/test/replay/trace/seeded-correct-run/test-artifacts/traceability-matrix.md +0 -138
- package/test/replay/trace/seeded-invented-and-duplicate-sections/expected.json +0 -67
- package/test/replay/trace/seeded-invented-and-duplicate-sections/test-artifacts/e2e-trace-summary.json +0 -217
- package/test/replay/trace/seeded-invented-and-duplicate-sections/test-artifacts/traceability-matrix.md +0 -153
- package/test/replay/trace/seeded-live-unverifiable/expected.json +0 -64
- package/test/replay/trace/seeded-live-unverifiable/test-artifacts/e2e-trace-summary.json +0 -217
- package/test/replay/trace/seeded-live-unverifiable/test-artifacts/traceability-matrix.md +0 -138
- package/test/replay/trace/seeded-matrix-lines-rejected/expected.json +0 -76
- package/test/replay/trace/seeded-matrix-lines-rejected/test-artifacts/e2e-trace-summary.json +0 -217
- package/test/replay/trace/seeded-matrix-lines-rejected/test-artifacts/traceability-matrix.md +0 -153
- package/test/replay/trace/seeded-missing-oracle-source/expected.json +0 -64
- package/test/replay/trace/seeded-missing-oracle-source/test-artifacts/e2e-trace-summary.json +0 -217
- package/test/replay/trace/seeded-missing-oracle-source/test-artifacts/traceability-matrix.md +0 -138
- package/test/replay/trace/seeded-rejected-evidence-omitted/expected.json +0 -67
- package/test/replay/trace/seeded-rejected-evidence-omitted/test-artifacts/e2e-trace-summary.json +0 -206
- package/test/replay/trace/seeded-rejected-evidence-omitted/test-artifacts/traceability-matrix.md +0 -138
- package/test/replay/trace/seeded-summary-schema-0-2/expected.json +0 -32
- package/test/replay/trace/seeded-summary-schema-0-2/test-artifacts/e2e-trace-summary.json +0 -186
- package/test/replay/trace/seeded-summary-schema-0-2/test-artifacts/traceability-matrix.md +0 -138
- package/test/replay/trace/seeded-waiver-misjudged/expected.json +0 -76
- package/test/replay/trace/seeded-waiver-misjudged/test-artifacts/e2e-trace-summary.json +0 -210
- package/test/replay/trace/seeded-waiver-misjudged/test-artifacts/traceability-matrix.md +0 -138
- package/test/schema/agent.js +0 -491
- package/test/schema/eval-result.js +0 -241
- package/test/schema/eval-result.schema.json +0 -476
- package/test/schema/suite-manifest.js +0 -188
- package/test/test-agent-schema.js +0 -387
- package/test/test-clock-port.js +0 -342
- package/test/test-contract-oracles.js +0 -1035
- package/test/test-contracts.js +0 -450
- package/test/test-corpus-conformance.js +0 -283
- package/test/test-enforce-hook.js +0 -413
- package/test/test-eval-quality-corpus.js +0 -301
- package/test/test-eval-replay.js +0 -1425
- package/test/test-file-system-conformance.js +0 -260
- package/test/test-file-system-port.js +0 -236
- package/test/test-installation-components.js +0 -510
- package/test/test-knowledge-base.js +0 -445
- package/test/test-port-totality.js +0 -396
- package/test/test-probe-conformance.js +0 -315
- package/test/test-probe-corpus.js +0 -320
- package/test/test-probe-targets.js +0 -1322
- package/test/test-release-metadata.js +0 -71
- package/test/test-stamp-changelog.js +0 -102
- package/test/test-test-review-cli.js +0 -6139
- package/tools/build-docs.js +0 -588
- package/tools/fix-doc-links.js +0 -288
- package/tools/generate-contracts.js +0 -3814
- package/tools/generate-probes.js +0 -1665
- package/tools/schema/agent.js +0 -491
- package/tools/stamp-changelog.js +0 -123
- package/tools/validate-agent-schema.js +0 -284
- package/tools/validate-ci-coverage.js +0 -185
- package/tools/validate-criteria-fragments.js +0 -453
- package/tools/validate-doc-links.js +0 -440
- package/tools/validate-eval-schemas.js +0 -345
- package/tools/validate-no-conflict-markers.js +0 -137
- package/tools/validate-tea-workflow-descriptions.js +0 -123
- package/website/README.md +0 -137
- package/website/astro.config.mjs +0 -191
- package/website/package-lock.json +0 -6856
- package/website/package.json +0 -24
- package/website/public/favicon.ico +0 -0
- package/website/public/img/tea-logo.svg +0 -7
- package/website/public/robots.txt +0 -37
- package/website/src/components/Banner.astro +0 -74
- package/website/src/components/Header.astro +0 -121
- package/website/src/components/MobileMenuFooter.astro +0 -53
- package/website/src/content/config.ts +0 -6
- package/website/src/lib/site-url.js +0 -25
- package/website/src/pages/404.astro +0 -11
- package/website/src/rehype-base-paths.js +0 -89
- package/website/src/rehype-markdown-links.js +0 -117
- package/website/src/styles/custom.css +0 -518
- package/website/tsconfig.json +0 -9
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "1.
|
|
34
|
+
"version": "1.27.0",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/README.md
CHANGED
|
@@ -13,7 +13,7 @@ TEA is a standalone BMAD module that delivers risk-based test strategy, test aut
|
|
|
13
13
|
- `tea-test-review`, a headless CLI that runs the review workflow as a CI gate with real exit codes
|
|
14
14
|
- a write-time enforcement hook that blocks the mechanically decidable violations before they reach disk
|
|
15
15
|
|
|
16
|
-
TEA is two layers. **TEA Core** decides what must be verified, at what depth, with what evidence, and whether that evidence is sufficient to release; it assumes nothing about your language, framework, or platform. **Execution targets** turn those decisions into runnable tests on a specific stack, and that layer is swappable. See [Verification Architecture](
|
|
16
|
+
TEA is two layers. **TEA Core** decides what must be verified, at what depth, with what evidence, and whether that evidence is sufficient to release; it assumes nothing about your language, framework, or platform. **Execution targets** turn those decisions into runnable tests on a specific stack, and that layer is swappable. See [Verification Architecture](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/verification-architecture/) for the split, and [Execution Targets](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/execution-targets/) for exactly which stacks are covered at which depth.
|
|
17
17
|
|
|
18
18
|
Docs: [https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/)
|
|
19
19
|
|
|
@@ -71,7 +71,7 @@ Phase 3 order matters and is deliberate: run `test-design` first so NFR evidence
|
|
|
71
71
|
|
|
72
72
|
`module-help.csv`'s single `phase` column records the phase a workflow's catalog row is sequenced under (its `preceded-by`/`followed-by` chain), not every phase the workflow can run in. `test-design`'s row is `3-solutioning` because that is the chain the row encodes (`test-design` → `framework`); the epic-level Phase 4 invocation above has no dependency edges of its own and so gets no second row, only this prose.
|
|
73
73
|
|
|
74
|
-
For the full lifecycle diagram including the BMad phases around TEA, see [TEA Overview](
|
|
74
|
+
For the full lifecycle diagram including the BMad phases around TEA, see [TEA Overview](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/tea-overview/).
|
|
75
75
|
|
|
76
76
|
### One epic, end to end
|
|
77
77
|
|
|
@@ -192,7 +192,7 @@ flowchart TB
|
|
|
192
192
|
|
|
193
193
|
**5. Validation.** `steps-v/` scores the finished output against `checklist.md`.
|
|
194
194
|
|
|
195
|
-
See [Step-File Architecture](
|
|
195
|
+
See [Step-File Architecture](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/step-file-architecture/) for the loading model, worker isolation, and the per-workflow step patterns.
|
|
196
196
|
|
|
197
197
|
### Parallel Workers and Execution Modes
|
|
198
198
|
|
|
@@ -293,7 +293,7 @@ npx bmad-method install
|
|
|
293
293
|
|
|
294
294
|
`trace` Phase 2 produces the decision: PASS, CONCERNS, FAIL, or WAIVED. Two mechanics sit under that vocabulary and are easy to miss.
|
|
295
295
|
|
|
296
|
-
**Live evidence is capped.** A requirement covered only by recorded live verification forces PASS down to CONCERNS, with a rationale naming the recorded source SHA. The overlay only ever lowers a PASS or annotates an existing CONCERNS. It can never lift a FAIL. Only a `pass` recorded against the commit under trace counts; `stale`, `unverifiable`, `contradicted`, `blocked`, and the rest are reported as blockers. The JSON contract is published at [Live Verification Results](
|
|
296
|
+
**Live evidence is capped.** A requirement covered only by recorded live verification forces PASS down to CONCERNS, with a rationale naming the recorded source SHA. The overlay only ever lowers a PASS or annotates an existing CONCERNS. It can never lift a FAIL. Only a `pass` recorded against the commit under trace counts; `stale`, `unverifiable`, `contradicted`, `blocked`, and the rest are reported as blockers. The JSON contract is published at [Live Verification Results](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/live-verification-results/), so any runner can produce it. Trace reads that file and never runs anything itself.
|
|
297
297
|
|
|
298
298
|
**Some runs are not gate-eligible at all.** A collection status of `waived`, `restricted`, `inaccessible`, or `deferred_shared` means no decision is emitted rather than a decision computed on partial evidence. A missing manifest resolves to `INACCESSIBLE`, not to 0% coverage.
|
|
299
299
|
|
|
@@ -309,7 +309,7 @@ It scopes to changed tests (`--base`, `--files`), runs through an agent adapter
|
|
|
309
309
|
|
|
310
310
|
The recommendation is derived from the findings rather than taken from the agent's prose. Any CRITICAL derives Block. Any HIGH, or a score under 70, derives Request Changes. The agent's own stated recommendation is preserved as `reportedRecommendation` when the two disagree. `--waive` exists for the exceptions and requires an expiry.
|
|
311
311
|
|
|
312
|
-
A copy-paste workflow lives at `cli/examples/pr-test-review.yml`, and the full flag, exit-code, and security reference is at [tea-test-review CLI](
|
|
312
|
+
A copy-paste workflow lives at `cli/examples/pr-test-review.yml`, and the full flag, exit-code, and security reference is at [tea-test-review CLI](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/tea-test-review-cli/).
|
|
313
313
|
|
|
314
314
|
## Configuration
|
|
315
315
|
|
|
@@ -328,7 +328,7 @@ TEA variables are defined in `src/module.yaml` and prompted during install. Ten
|
|
|
328
328
|
- `risk_threshold` — risk cutoff for mandatory testing. Prompted at install, not yet read by any workflow
|
|
329
329
|
- `test_design_output`, `test_review_output`, `trace_output` — subfolders under `test_artifacts`. Prompted at install, not yet read by any workflow
|
|
330
330
|
|
|
331
|
-
Full option reference: [Configuration](
|
|
331
|
+
Full option reference: [Configuration](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/configuration/).
|
|
332
332
|
|
|
333
333
|
## Knowledge Base
|
|
334
334
|
|
|
@@ -338,7 +338,7 @@ TEA relies on a curated testing knowledge base of 59 fragments, indexed by tier:
|
|
|
338
338
|
- Fragments: `src/agents/bmad-tea/resources/knowledge/`
|
|
339
339
|
- Tiers: 24 core, 19 extended, 16 specialized
|
|
340
340
|
|
|
341
|
-
Workflows load only the fragments required for the current task, and the required set is named in the step file rather than inferred from index tags. See [Knowledge Base](
|
|
341
|
+
Workflows load only the fragments required for the current task, and the required set is named in the step file rather than inferred from index tags. See [Knowledge Base](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/reference/knowledge-base/).
|
|
342
342
|
|
|
343
343
|
## Repository Layout
|
|
344
344
|
|
|
@@ -373,40 +373,44 @@ TEA has deterministic checks and live evals. These cover specific risks. They ar
|
|
|
373
373
|
|
|
374
374
|
The eight suites under `test/evals/` measure one decision inside each knowledge-bearing workflow: whether the agent selects the required knowledge fragments and avoids fragments the workflow excludes. They do not execute the complete workflow or grade its final artifact.
|
|
375
375
|
|
|
376
|
-
The `nfr`, `test-design`, `test-review`, and `trace` suites add behavioral evals. `nfr` audits one evidence bundle with known gaps and one clean bundle. `test-design` analyzes one seeded epic and one clean control epic. `test-review` runs the complete review against files containing nine planted defects plus one clean file, then scores recall, the non-false-positive rate, score variance, and verdict stability. `trace` runs the complete traceability workflow against a ten-criterion seeded set whose declared gate is FAIL and a five-criterion clean set whose declared gate is PASS, then scores criterion status, the gate decision and the criteria behind it, the coverage arithmetic, evidence citation, and case stability.
|
|
376
|
+
The `atdd`, `automate`, `ci`, `nfr`, `test-design`, `test-review`, and `trace` suites add behavioral evals, and so does `bmad-tea`'s own routing suite. `atdd` generates red-phase scaffolds against an unimplemented fixture, activates them, executes them under NFR9's isolation, and scores whether each one fails for the reason its criterion states. `automate` runs four hand-authored spec sets, standing in for generated coverage, against a fixed voucher-redemption service and against a scratch copy carrying its seeded minimum-spend regression, and scores whether the one spec set that tests the boundary catches the regression and whether the other three are correctly reported as missing it, vacuous, or duplicated rather than as clean passes; it spends no vendor call, since there is no live-agent mode to build one for. `ci` scaffolds a pipeline for a project whose team wrote down what it needs, then parses the generated workflow, lints it with `actionlint`, and scores every requested trigger, permission, command, gate, and artifact against a minimal-request control. `nfr` audits one evidence bundle with known gaps and one clean bundle. `test-design` analyzes one seeded epic and one clean control epic. `test-review` runs the complete review against files containing nine planted defects plus one clean file, then scores recall, the non-false-positive rate, score variance, and verdict stability. `trace` runs the complete traceability workflow against a ten-criterion seeded set whose declared gate is FAIL and a five-criterion clean set whose declared gate is PASS, then scores criterion status, the gate decision and the criteria behind it, the coverage arithmetic, evidence citation, and case stability. `bmad-tea` routes eighteen intents to the right menu item, or declines the ones nothing on the menu serves.
|
|
377
377
|
|
|
378
|
-
| Skill | Fragment-selection cases | Full behavioral eval
|
|
379
|
-
| --------------------------- | ------------------------ |
|
|
380
|
-
| `bmad-tea` | N/A |
|
|
381
|
-
| `bmad-teach-me-testing` | N/A | None; this skill has no workflow knowledge index
|
|
382
|
-
| `bmad-testarch-atdd` | 3 |
|
|
383
|
-
| `bmad-testarch-automate` | 5 |
|
|
384
|
-
| `bmad-testarch-ci` | 2 |
|
|
385
|
-
| `bmad-testarch-framework` | 3 | None
|
|
386
|
-
| `bmad-testarch-nfr` | 2 | Yes; one evidence bundle with known gaps and one clean bundle
|
|
387
|
-
| `bmad-testarch-test-design` | 5 | Yes; one seeded epic and one clean control epic
|
|
388
|
-
| `bmad-testarch-test-review` | 2 | Yes; three files, nine planted defects, and one clean file
|
|
389
|
-
| `bmad-testarch-trace` | 2 | Yes; a ten-criterion seeded set and a five-criterion clean set
|
|
378
|
+
| Skill | Fragment-selection cases | Full behavioral eval |
|
|
379
|
+
| --------------------------- | ------------------------ | ----------------------------------------------------------------------------------- |
|
|
380
|
+
| `bmad-tea` | N/A | Yes; eighteen intents, ten with a right answer and eight controls |
|
|
381
|
+
| `bmad-teach-me-testing` | N/A | None; this skill has no workflow knowledge index |
|
|
382
|
+
| `bmad-testarch-atdd` | 3 | Yes; one unimplemented fixture and five acceptance criteria |
|
|
383
|
+
| `bmad-testarch-automate` | 5 | Yes; four hand-authored spec sets against a fixed service and its seeded regression |
|
|
384
|
+
| `bmad-testarch-ci` | 2 | Yes; one full request and one minimal request |
|
|
385
|
+
| `bmad-testarch-framework` | 3 | None |
|
|
386
|
+
| `bmad-testarch-nfr` | 2 | Yes; one evidence bundle with known gaps and one clean bundle |
|
|
387
|
+
| `bmad-testarch-test-design` | 5 | Yes; one seeded epic and one clean control epic |
|
|
388
|
+
| `bmad-testarch-test-review` | 2 | Yes; three files, nine planted defects, and one clean file |
|
|
389
|
+
| `bmad-testarch-trace` | 2 | Yes; a ten-criterion seeded set and a five-criterion clean set |
|
|
390
390
|
|
|
391
|
-
A passing fragment-selection eval means the workflow loaded the right knowledge. It makes no claim about the quality of the workflow's final output.
|
|
391
|
+
A passing fragment-selection eval means the workflow loaded the right knowledge. It makes no claim about the quality of the workflow's final output. A full behavioral eval for the remaining skill, `bmad-testarch-framework`, is a coverage gap. The source-controlled [Eval Quality and Behavioral Coverage Roadmap](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-roadmap/) records the per-skill contracts, runner work, CI plan, and intended boundary with the upcoming standalone `eval-quality` project. [The eval-quality Command Adapter](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-command-adapter/) records how TEA runs the commands those evals measure, and [Adopting eval-quality, One Skill at a Time](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-adoption-guide/) is the guide for taking one skill in another BMAD module from asserted quality to measured quality.
|
|
392
392
|
|
|
393
393
|
### Deterministic Checks
|
|
394
394
|
|
|
395
|
-
`npm test` chains
|
|
395
|
+
`npm test` chains seventy checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Seven of the seventy keep the rules, guidance, hook, eval data, eval contracts, and documentation aligned:
|
|
396
396
|
|
|
397
397
|
- `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
|
|
398
|
+
- `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, and the fragment-selection case count. A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
|
|
399
|
+
- `test:doc-claims` runs `eval-quality-gates doc-claims`, which holds a hand-written prose claim — a named symbol, a transcribed list, a worked JSON example, a named exit code, or a claim whose truth depends on when it was written — against the artifact it describes. A registered dated claim whose sentence was rewritten out from under it fails the same way a wrong symbol name does, so the entry cannot go stale by drifting out from under its own key either.
|
|
398
400
|
- `test:enforce-hook` fails when a new Absolute registry row appears in neither the hook's enforced list nor its deferred list. This prevents a rule from being added without an explicit write-time enforcement decision.
|
|
399
401
|
- `test:eval-data` checks that all 24 fragment-selection cases are structurally usable: their workflow context files exist, every expected fragment exists and is indexed for that workflow, and the required and forbidden sets do not overlap. The expected sets come from the workflow step files. This check does not ask an agent to select anything.
|
|
400
|
-
- `test:eval-schemas` checks
|
|
402
|
+
- `test:eval-schemas` checks this repository's own eval suite manifest against its schema, confirms that every threshold it declares is the threshold the harness actually applies, that the runner capabilities it declares are the ones the harness grants its runner, and that the preflight argv it declares fails on a missing runner and passes on a present one, and fails when a TEA skill has neither a behavioral suite nor a deferred declaration. A suite list that omits a skill reads as coverage, so the omission has to be an error rather than a silence.
|
|
401
403
|
- `test:contract-oracles` evaluates every oracle in every eval contract with `eval-quality`'s own evaluator, over the stored replay outputs and over constructed selections for all 24 fragment-selection cases, and fails when an oracle faults or disagrees with the harness scorer on the same evidence. The first run found that every regex oracle in the `test-review` contract was refused by the evaluator before it matched anything.
|
|
402
404
|
|
|
403
405
|
These checks produce the same answer from the same repository state. They need no agent credential, network call, or model budget. `test:eval-data` runs through `npm test`, the local pre-commit hook, pull-request quality checks, and the publish workflow.
|
|
404
406
|
|
|
407
|
+
`test:doc-invocations` runs `eval-quality-gates doc-invocations`, which re-runs every `npm run <script>` this README, the adoption guide, and `test/README.md` document in their CI-usage and individual-suites blocks against the real repository, so a renamed script or a broken example fails the build instead of a reader. Each of the seventeen documented scripts is declared as a full literal spelling, never a shared `npm run` prefix, which is what keeps the gate from ever matching, let alone running, `npm run eval:all` or `npm run release:next` on the same pages; `test:cli` is deliberately left undeclared, since it is a separately sharded suite that takes 12-15 minutes, the wrong fit for a bounded documentation gate. The gate's own entry script is the one real program it is allowed to reach; it refuses anything outside its own hardcoded allowlist before invoking real `npm`, and `test:doc-invocation-entry` holds that allowlist's comment-to-script lookup against all three pages' literal text, line by line, so a swapped or drifted comment fails there even when the script it resolves to would still run.
|
|
408
|
+
|
|
405
409
|
### Start Here
|
|
406
410
|
|
|
407
411
|
You do not start an interactive agent session. A live eval launches the selected agent CLI as a headless subprocess, sends it each prompt, waits for the result, and scores the result.
|
|
408
412
|
|
|
409
|
-
The normal path is one command. It runs fragment selection across all eight covered workflow skills, then runs the behavioral `bmad-tea-routing`, `nfr`, `test-design`, `test-review` and `
|
|
413
|
+
The normal path is one command. It runs fragment selection across all eight covered workflow skills, then runs the behavioral `bmad-tea-routing`, `ci`, `nfr`, `test-design`, `test-review`, `trace`, `atdd` and `automate` evals:
|
|
410
414
|
|
|
411
415
|
```bash
|
|
412
416
|
npm run eval:all -- --agent codex
|
|
@@ -420,7 +424,7 @@ npm run eval:all -- --agent agy
|
|
|
420
424
|
npm run eval:all -- --agent agy --agent claude --agent codex
|
|
421
425
|
```
|
|
422
426
|
|
|
423
|
-
`eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `test-design` epic,
|
|
427
|
+
`eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `ci` project, per `test-design` epic, per `trace` fixture set, and per `atdd` story. One runner makes 105 agent calls: 48 fragment selections, 36 routing intents, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, and 2 ATDD generations. All three built-in runners make 315 calls.
|
|
424
428
|
|
|
425
429
|
Check the data, executable, login, fixtures, and expected results without making a model call:
|
|
426
430
|
|
|
@@ -430,7 +434,7 @@ npm run eval:all -- --agent claude --preflight-only
|
|
|
430
434
|
npm run eval:all -- --agent agy --preflight-only
|
|
431
435
|
```
|
|
432
436
|
|
|
433
|
-
Output ending with `nothing measured` is expected in preflight mode. It means the static eval data is valid and, for
|
|
437
|
+
Output ending with `nothing measured` is expected in preflight mode. It means the static eval data is valid and, for ten of the twelve suites, the selected executable is on `PATH`, answers `--version`, and a built-in vendor has a credential. `automate` and `framework` invoke no agent at all, so their preflight only checks that their own corpus and dependencies resolve, never a vendor or a credential. Some runners cannot expose session authentication to this probe, so a preflight pass does not guarantee that the later live call will authenticate. The flag intentionally exits before launching the agent. `npm run test:eval-schemas` runs each suite's preflight with a missing runner and with a present one, so a preflight that stopped probing the runner would fail `npm test`.
|
|
434
438
|
|
|
435
439
|
### A La Carte Live Evals
|
|
436
440
|
|
|
@@ -542,19 +546,21 @@ npm run eval:fragment-selection -- \
|
|
|
542
546
|
|
|
543
547
|
### What Has to Pass
|
|
544
548
|
|
|
545
|
-
Every row below is a declared gate. The three suites that existed then were measured live for the first time on 2026-09-08 against `claude`/`sonnet`, and
|
|
546
|
-
|
|
547
|
-
| Eval | Declared threshold | Declared volume
|
|
548
|
-
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
549
|
-
| `npm run eval:all -- --agent ...` | All
|
|
550
|
-
| Fragment selection | At least 90% required-fragment recall, at most 10% forbidden-fragment selection, and stable choices across repeated cases | 24 cases twice: 48 calls
|
|
551
|
-
| Test review | At least 70% overall recall, 100% CRITICAL recall, an 80% non-false-positive rate, score standard deviation no higher than 3, and a stable verdict | Three complete reviews
|
|
552
|
-
| Trace | At least 90% criterion-status accuracy and 90% evidence-citation precision; 100% on both discriminating criteria, the gate decision, the gate criteria, the coverage arithmetic, the oracle resolution, the rejected evidence, the waiver validity, and the live evidence; no clean-set false positive, no unstable case, and no fixture mutation | Two fixture sets twice: 4 calls
|
|
553
|
-
| bmad-tea routing | At least 90% route accuracy, 80% reason-token recall, 87.5% scope fidelity, and 75% on both clarification and decline recall; zero confident routes on an intent nothing on the menu serves, zero on one whose menu items are genuinely close, zero clear intents left unrouted, and at most two intents whose answer changes between repetitions | 18 intents twice: 36 calls
|
|
554
|
-
| NFR | At least 87.5% domain-status accuracy and 90% threshold fidelity; 100% on undecidable domains, the overall status, and domain coverage; no unsupported PASS, no fabricated evidence, no clean-bundle false positive, no duplicate domain section, no unstable case, and no fixture mutation
|
|
555
|
-
|
|
|
556
|
-
|
|
|
557
|
-
|
|
|
549
|
+
Every row below is a declared gate. The three suites that existed then were measured live for the first time on 2026-09-08 against `claude`/`sonnet`, and the [Eval Quality and Behavioral Coverage Roadmap](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-roadmap/) records what each one returned; `bmad-tea-routing`, `nfr`, `test-design`, `ci` and `atdd` have been added since and have never been run live, so their rows are declared targets and nothing more; no result artifact is committed, so quote that section rather than a number from memory. The live thresholds are the ones this repository's own eval suite manifest declares, and `npm run test:eval-schemas` fails when a harness constant and the manifest disagree.
|
|
550
|
+
|
|
551
|
+
| Eval | Declared threshold | Declared volume |
|
|
552
|
+
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
|
|
553
|
+
| `npm run eval:all -- --agent ...` | All eight live suites below meet their thresholds for the selected runner | 48 selections, 36 routing calls, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, 2 generations |
|
|
554
|
+
| Fragment selection | At least 90% required-fragment recall, at most 10% forbidden-fragment selection, and stable choices across repeated cases | 24 cases twice: 48 calls |
|
|
555
|
+
| Test review | At least 70% overall recall, 100% CRITICAL recall, an 80% non-false-positive rate, score standard deviation no higher than 3, and a stable verdict | Three complete reviews |
|
|
556
|
+
| Trace | At least 90% criterion-status accuracy and 90% evidence-citation precision; 100% on both discriminating criteria, the gate decision, the gate criteria, the coverage arithmetic, the oracle resolution, the rejected evidence, the waiver validity, and the live evidence; no clean-set false positive, no unstable case, and no fixture mutation | Two fixture sets twice: 4 calls |
|
|
557
|
+
| bmad-tea routing | At least 90% route accuracy, 80% reason-token recall, 87.5% scope fidelity, and 75% on both clarification and decline recall; zero confident routes on an intent nothing on the menu serves, zero on one whose menu items are genuinely close, zero clear intents left unrouted, and at most two intents whose answer changes between repetitions | 18 intents twice: 36 calls |
|
|
558
|
+
| NFR | At least 87.5% domain-status accuracy and 90% threshold fidelity; 100% on undecidable domains, the overall status, and domain coverage; no unsupported PASS, no fabricated evidence, no clean-bundle false positive, no duplicate domain section, no disagreement between the gate artifact and the assessment it summarizes, no unstable case, and no fixture mutation | Two evidence bundles twice: 4 calls |
|
|
559
|
+
| CI | Zero parse failures and zero `actionlint` findings; at least 90% requested-element recall and 100% trigger accuracy; zero unrequested elements, zero workflow rule violations, no unstable case, and no fixture mutation | Two projects twice: 4 calls |
|
|
560
|
+
| Test design | At least 80% grounded-risk recall, 80% risk precision, and 80% coverage mapping; 100% on the scale compliance, the score arithmetic, the category validity, the band placement, the risk-id shape, the risk-link resolution, and the priority ordering; no unscored risk table, no ungrounded risk, no missed top-severity risk, no risk-ceiling excess, no unstable case, and no fixture mutation | Two epics twice: 4 calls |
|
|
561
|
+
| ATDD | 100% red-for-intended-reason rate and criteria coverage; no vacuous pass, no scaffold still skipped after activation, no load error, no non-assertion exit, no unmapped test, no production mutation, and no unstable case | One story twice: 2 generation calls, plus execution under NFR9's isolation, which spends no call |
|
|
562
|
+
| `npm run test:eval-data` | Every case references valid workflow files and indexed fragments; required and forbidden sets do not overlap | No agent calls |
|
|
563
|
+
| `npm run test:eval-schemas` | The suite manifest matches its schema, declares the thresholds the harnesses apply, and accounts for every TEA skill | No agent calls |
|
|
558
564
|
|
|
559
565
|
Every declared repetition has to complete. A run that loses one to a timeout, a transport error, or an unparseable reply cannot measure variance or stability, so it exits `2` rather than reporting a lower score.
|
|
560
566
|
|
|
@@ -562,7 +568,7 @@ The non-false-positive rate is the share of reported findings that are not defin
|
|
|
562
568
|
|
|
563
569
|
### The Suite Manifest and Machine-Readable Results
|
|
564
570
|
|
|
565
|
-
|
|
571
|
+
This repository's own eval suite manifest registers every suite `eval:all` runs, with its skills, fixtures, ground truth, contract, thresholds, repetition count, CI tier, and the capabilities its runner needs. It also carries a `deferred` list: one entry per skill with no behavioral suite, naming the owner, the missing evidence, and the condition that retires the entry. `eval:all` reads the suite list from that file and refuses to run when a TEA skill appears in neither list.
|
|
566
572
|
|
|
567
573
|
Add `--json <path>` to any of the six harnesses to write a result record alongside the console output:
|
|
568
574
|
|
|
@@ -571,7 +577,7 @@ npm run eval:all -- --agent codex --json results/eval-all.json
|
|
|
571
577
|
npm run eval:test-review -- --agent codex --json results/test-review.json
|
|
572
578
|
```
|
|
573
579
|
|
|
574
|
-
The record carries the repository commit, the suite and case IDs, the runner executable and version, the resolved model and parameters, the fixture and prompt digests, the expected and completed repetitions, the measurements, the duration, and the final failure class.
|
|
580
|
+
The record carries the repository commit, the suite and case IDs, the runner executable and version, the resolved model and parameters, the fixture and prompt digests, the expected and completed repetitions, the measurements, the duration, and the final failure class. A generated JSON Schema in this repository's own dev tree describes it, and every record is validated against that schema before it is written.
|
|
575
581
|
|
|
576
582
|
### CI Usage
|
|
577
583
|
|
|
@@ -579,10 +585,10 @@ Run the eval's deterministic checks on every pull request. `npm test` runs all o
|
|
|
579
585
|
|
|
580
586
|
```bash
|
|
581
587
|
npm ci
|
|
582
|
-
npm run test:eval-data
|
|
583
|
-
npm run test:eval-trace-data
|
|
584
|
-
npm run test:eval-schemas
|
|
585
|
-
npm run test:eval-replay
|
|
588
|
+
npm run test:eval-data # fragment-selection corpus, static
|
|
589
|
+
npm run test:eval-trace-data # trace corpus, static
|
|
590
|
+
npm run test:eval-schemas # manifest against harness constants, and the preflight argv
|
|
591
|
+
npm run test:eval-replay # 108 stored outputs against the scorers
|
|
586
592
|
```
|
|
587
593
|
|
|
588
594
|
Run live evals in a scheduled or manually triggered CI job after installing and authenticating the selected agent CLI:
|
|
@@ -604,7 +610,7 @@ Custom workflows are still compatible with TEA, but they are no longer implicitl
|
|
|
604
610
|
2. Attach it to `bmad-tea` using the agent customization flow.
|
|
605
611
|
3. Reinstall/update BMAD so the new menu item and workflow are registered.
|
|
606
612
|
|
|
607
|
-
See [Extend TEA with Custom Workflows](
|
|
613
|
+
See [Extend TEA with Custom Workflows](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/customization/extend-tea-with-custom-workflows/) and the BMAD customization guide at [BMAD-METHOD/docs/how-to/customize-bmad.md](https://github.com/bmad-code-org/BMAD-METHOD/blob/main/docs/how-to/customize-bmad.md).
|
|
608
614
|
|
|
609
615
|
## Contributing
|
|
610
616
|
|
|
@@ -612,9 +618,7 @@ See `CONTRIBUTING.md` for guidelines.
|
|
|
612
618
|
|
|
613
619
|
---
|
|
614
620
|
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
## Publishing TEA to NPM
|
|
621
|
+
## 📦 Publishing TEA to NPM (for Maintainers)
|
|
618
622
|
|
|
619
623
|
TEA uses an automated publish workflow modeled after the main `BMAD-METHOD` repo. It supports:
|
|
620
624
|
|