bmad-method-test-architecture-enterprise 1.23.1 → 1.23.2

package/.claude-plugin/marketplace.json

@@ -31,7 +31,7 @@
       "name": "bmad-method-test-architecture-enterprise",
       "source": "./",
       "description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
-      "version": "1.23.1",
+      "version": "1.23.2",
       "author": {
         "name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
       },

package/CHANGELOG.md

@@ -27,18 +27,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- `docs/explanation/eval-quality-roadmap.md`: a source-controlled handoff for the nine skills that still lack full behavioral coverage, the runner and CI work needed to scale `eval:all`, and the intended one-way integration with the upcoming standalone `eval-quality` contract and scoring layer. It defines per-skill behavioral contracts, a phased rollout, runner admission and manual-use requirements, CI tiers, and the evidence required before a skill can be called behaviorally covered.
+- Built-in `agy` (Antigravity CLI) adapter for `npm run eval:all`, `eval-fragment-selection`, and `tea-test-review`. Handles argv-based prompt passing natively (`promptViaArgv`) without stdin pipe truncation or `EPIPE` errors, enabling headless evaluation in Google Antigravity environments.
+- `npm run eval:all -- --agent <runner>`: one CI-compatible entrypoint that runs fragment selection across every covered workflow skill, then runs the behavioral `test-review` eval with the same selected runner. It preserves the harness-specific repetition defaults, supports focused workflow and repetition overrides, and offers a no-model `--preflight-only` path.
+- Portable `custom` agent adapter for `tea-test-review` and both live eval harnesses. `--agent-cmd`, repeatable `--agent-arg`, and repeatable `--env-pass` let any stdin-driven headless agent CLI use the same prompt, scoring, and exit-code path as the built-in adapters. The custom runner has no implicit executable, arguments, model, credential variables, or approval policy.
 - Write-time enforcement hook, scaffolded by the `framework` workflow. TEA was advisory (knowledge fragments) plus post-hoc (`test-review` scoring) with nothing at the write itself, so a `.only`, a `waitForTimeout`, or a `Thread.sleep` could land, be committed, and only surface at review. `src/workflows/testarch/bmad-testarch-framework/resources/hooks/tea-enforce.cjs` blocks the write instead, and `steps-c/step-04-docs-and-scripts.md` installs it plus its `.claude/settings.json` registration into the target project. Rules are the mechanically decidable `Absolute` rows of `criteria-registry.md` (C2, C3, C4 for Maestro flows, H1, H5, H6, H8 block; C1 warns, because its row is conditioned on a documented, still-true reason that no pattern can check). Multi-language by construction: Playwright, Cypress, Vitest/Jest, Pact, pytest, JUnit, Go test, and Maestro flows each get their own predicates and their own comment and string stripping, so a `waitForTimeout` inside a comment or a doc example is not a violation. Three passes rather than one: `--pre` blocks the fragment about to be written, `--post` re-reads the whole file from disk (which is what catches writes made through Bash, violations split across two edits, and whole-file rules), and `--stop` sweeps test files modified during the turn (which is what catches a codegen script that wrote files it never named). The hook fails open on any error of its own.
 - The hook honours the registry's `Gate` column structurally. `.tea/enforce-config.json` carries only the globs for the stack the `framework` workflow actually detected, so a repo with no Maestro flows cannot fire the Maestro rows and a repo with no pact config cannot fire H6 or H8. `excludeGlobs` keeps k6 scripts out, where `sleep(1)` is the documented way to model think-time and H1 would be confidently wrong. A closed gate is not a violation, which is defect #1 from the couture-cast PR #103 postmortem in `DESIGN-CRITERIA-REGISTRY.md`, applied to a second enforcement surface.
 - The scaffold records the hook's sha256 in `.tea/enforce-config.json`, and the hook compares its own file against it on `--stop` only, warning once and never blocking. The instruction to copy the script byte for byte pointed at a test that lives in the TEA repository, so nothing inside the target project would have noticed a locally edited copy — and a locally edited copy is exactly the one that is no longer covered by the test keeping its rules in agreement with the registry.
-- `tools/validate-criteria-fragments.js`: traceability between `criteria-registry.md` and the knowledge fragments. Nothing asserted that a registry row still had a fragment teaching it, nor that a mapped fragment still carried its claim at the registry's pinned severity. It fails on a mapped fragment that lost its anchor, a manifest row pointing at a fragment that does not exist or is not indexed in `tea-index.csv`, a severity that no longer matches the registry, and a registry row that is neither mapped nor declared a gap. It found 14 rows with no fragment teaching them at all, which are closed in this same release (below), taking coverage from 21/35 to 35/35 over 46 anchors. The gap-declaration mechanism stays: a registry row that is neither mapped to a fragment nor declared a gap fails the build, and with no real gap left to exercise that path the tool now self-checks it against a synthetic row so a refactor cannot quietly kill the guard.
+- `tools/validate-criteria-fragments.js`: traceability between `criteria-registry.md` and the knowledge fragments. Nothing asserted that a registry row still had a fragment teaching it, nor that a mapped fragment still carried its claim at the registry's pinned severity. It fails on a mapped fragment that lost its anchor, a manifest row pointing at a fragment that does not exist or is not indexed in `tea-index.csv`, a severity that no longer matches the registry, and a registry row that is neither mapped nor declared a gap. It found 14 rows with no fragment teaching them at all, which are closed in this same release (below), taking coverage from 21/35 to 35/35 over 48 anchors. The gap-declaration mechanism stays: a registry row that is neither mapped to a fragment nor declared a gap fails the build, and with no real gap left to exercise that path the tool now self-checks it against a synthetic row so a refactor cannot quietly kill the guard.
 - Knowledge for the 14 registry rows nothing taught, added to the four fragments that already own the surrounding material rather than as 14 new files, since fragmenting the base makes selection worse. `test-quality.md` gains committed skips and committed focus (C1, C2), assertions that cannot fail (C3 tautological, C5 asserted against the test's own mock, C6 unreachable), and suite structure and naming (M3 counted by subject rather than by `expect` call, M4 grouping, M7 nesting, L5 behavioral names, L7 one assertion dialect). `timing-debugging.md` gains wall-clock fixtures with fake timers as the fix (H2) and unawaited promises in test bodies (M6). `component-tdd.md` gains user-level interaction over raw event dispatch (M5), gated on the project already depending on such an API. `data-factories.md` gains naming the domain literals a test hardcodes on purpose (L6), which is the half the factory patterns never covered. Examples appear in the languages each row's own predicate names — Python and JUnit skip forms for C1, `assert x == x` for C3, `freeze_time` for H2 — rather than in TypeScript alone. The `tags` and `description` columns for all four fragments gained the terms that make the new material selectable, in the agent index and all eight workflow copies.
 - `test/eval-fragment-selection.js` plus per-workflow eval data under `test/evals/`: measures whether the right fragment comes out of `tea-index.csv` for a given task, which is the failure mode where an agent loads the wrong fragment, or none, and answers from prior. Twenty-four cases across all eight workflows that ship a knowledge base (`automate`, `test-design`, `atdd`, `ci`, `framework`, `nfr`, `test-review`, `trace`), seeded from Playwright, Cypress, pytest, JUnit, Go, Pact, GitLab CI, and Maestro scenarios rather than from Playwright alone, since fragment selection for a non-JavaScript stack is where TEA is most likely to route wrong. `teach-me-testing` is excluded because it ships no `resources/knowledge`: its fragment browsing is a menu the learner drives rather than a routing decision the agent makes. Every non-JavaScript case asserts the run-level precondition that nothing else was measuring: both library flags default true, and in a repo with no JavaScript manifest neither package can be installed, so both mandates must stay closed. Ground truth is quoted from each workflow's own step files, never from `tea-index.csv` tags, because tags describe a fragment while step files decide what a run loads. `--validate-only` runs in CI with no vendor cost and rejects a name in either direction that does not exist or is not indexed for that workflow, so a typo in a forbidden list cannot pass vacuously; the scored run needs a logged-in `claude` or `codex`.
 - `.github/ISSUE_TEMPLATE/rule_quality_report.md`: a low-friction report for the failure that matters most in a rules product, an agent misreading, ignoring, or being misled by a TEA rule or knowledge fragment. Captures the agent and model, the rule and section, the workflow, the prompt, the non-compliant output, and what should have happened.
+- Nine populated workflow output examples, one beside every TEA workflow: Academy completion, epic test design, framework setup, CI pipeline setup, ATDD red-phase checklist, automation summary, test review, four-domain NFR evidence audit, and traceability plus deterministic gate decision. Each example uses an independent scenario, complete progress frontmatter, repository-relative references, and the current output contract. Together they demonstrate that test priority remains a judgment separate from risk score, NFR statuses use `CONCERNS` across Security, Performance, Reliability, and Maintainability, and `WAIVED` requires a complete human override rather than a derived code path.
+
+### Changed
+
+- `README.md` now separates installation from runtime for `tea-enforce.cjs`: framework Create installs it, Resume can complete that step, and the hook then runs at project scope for matching tool events from any workflow, agent, or ordinary coding prompt. The lifecycle description also explains what each pass protects and why semantic findings remain in `test-review`. Conceptual descriptions use platform-neutral language while exact host details remain where invocation or release instructions require them. The self-validation section now presents deterministic checks and live evals as two distinct layers. It includes exact commands for each existing per-skill routing suite, the built-in runner combinations, Gemini through the custom runner contract, CI usage, passing thresholds, default call volume, and an explicit account of which skills still lack behavioral eval coverage.
 
 ### Fixed
 
+- CodeRabbit follow-up aligned the public runner contract with the built-in `agy` adapter, corrected the README's waiver, Automate deviation, built-in runner, custom stdin, and customization-link documentation, and removed the stdin-only confidentiality claim for argv-based adapters.
+- CodeRabbit follow-up synchronized shared priority examples and coverage-gate callouts, tightened eval preflight and custom-model validation, aligned NFR threshold and compliance contracts, and corrected the affected workflow examples.
+- Worked-example validation exposed two stale workflow contracts. The `test-review` checklist still advertised A+/A/B/C/F while the executing scoring step, report parser, and template use A/B/C/D/F; it now matches the live grade function. The Automate final step described deviation entries with an em dash; it now uses the repository-compliant `file:line: reason` shape used by the example.
+
+- Risk score to test priority had three incompatible statements. `probability-impact.md` defined a deterministic `mapRiskToPriority()` sending a score of 6-8 to P1; `test-priorities-matrix.md`'s Integration with Risk Scoring table called the same band "P0 or P1"; and `test-design-template.md` and `checklist.md` both wrote the P0 criterion as risk score `≥6`, which is P1 under the deleted function. An agent's answer to "what priority does this risk score imply" depended on which fragment it loaded. Resolved as: priority is a judgment the risk score informs, not a value the score determines. `mapRiskToPriority()` and its call site are removed from `probability-impact.md` (all nine copies); risk score continues to classify the remediation action (DOCUMENT/MONITOR/MITIGATE/BLOCK) via the untouched `classifyRiskAction()`. The Integration with Risk Scoring table in `test-priorities-matrix.md` (all nine copies) now states plainly that it is a sanity check on a priority already assigned by the Priority Decision Tree, not an assignment rule. `test-design-template.md` and `checklist.md` drop the numeric risk-score anchor from the P0/P1/P2 criteria lines in favor of the qualitative risk levels the decision tree already uses. `README.md`'s existing description of this relationship needed no change; it already described the model this converges on.
+
+- The NFR evidence audit named its four domains three different ways. `nfr-criteria.md` (all nine copies) taught Security/Performance/Reliability/**Maintainability** criteria and gate rows, but the `nfr` workflow's four evaluation subagents, its aggregation step, `trace-template.md`'s Phase 2 evidence summary, and several docs pages all executed and described Security/Performance/Reliability/**Scalability** instead — an agent auditing evidence ran a domain (Scalability) the knowledge base never defined PASS/CONCERNS/FAIL criteria for, while the domain the knowledge base did define (Maintainability) was never audited. Separately, `step-02-define-thresholds.md` offered the 8-category ADR Quality Readiness Checklist as if it were the source for all four audited domains, which doesn't hold: Maintainability isn't one of the 8 ADR categories. Resolved in favor of the knowledge base's existing Maintainability domain: `step-04d-subagent-scalability.md` is replaced by `step-04d-subagent-maintainability.md`, auditing test coverage, code duplication, dependency vulnerabilities, and observability instead of horizontal/vertical/data scaling; `step-04e-aggregate-nfr.md`'s domain list, cross-domain risk example, and risk-breakdown output follow; `nfr-report-template.md`'s Scalability Assessment section and Auto-Scaling fail-fast item become Maintainability equivalents (the separate 8-category ADR scorecard table, which was never part of this inconsistency, is untouched); `checklist.md` and `trace-template.md` follow the same rename. `step-02-define-thresholds.md` now states plainly that Maintainability's criteria come from `nfr-criteria.md` directly rather than from the ADR-8 list, which remains the elicitation source for Security and Performance. `README.md`, `docs/reference/commands.md`, `docs/explanation/step-file-architecture.md`, and `docs/reference/execution-targets.md` are updated to match; the last of these was also missing a Maintainability row entirely, having folded Reliability and Scalability into one row.
+
+- Only one of the `nfr` workflow's four evidence-audit subagents wrote down what PASS, CONCERN, FAIL, and N/A mean. `step-04a-subagent-security.md` spelled the definitions out; `step-04b-subagent-performance.md`, `step-04c-subagent-reliability.md`, and (after the rename above) `step-04d-subagent-maintainability.md` used the same four values only inside their JSON output examples, so a worker choosing between CONCERN and FAIL for those three domains had no written rule to apply. A new shared file, `steps-c/nfr-status-definitions.md`, states the four definitions once; all four worker step files now load it instead of each carrying (or, for three of them, lacking) their own copy — the same fix already applied to severity via `criteria-registry.md` in the `test-review` workflow, applied here before four independently worded copies could drift the way that severity drift did. Separately, `step-02-define-thresholds.md` has instructed since before this fix that a threshold still UNKNOWN after checking every source should be reported as CONCERNS, but nothing checked that a worker actually did that. `step-04e-aggregate-nfr.md` now enforces it after all four workers report: any finding under a domain whose threshold was UNKNOWN is downgraded from PASS to CONCERNS, with the finding's description noting why.
+
+- The `nfr` workflow's evidence-audit status enum was spelled two ways: `step-04a-subagent-security.md` and the new shared `nfr-status-definitions.md` used the singular `CONCERN`, while `nfr-criteria.md`, the `trace` workflow's gate step, and its trace template all used the plural `CONCERNS`. `step-04e-aggregate-nfr.md`'s aggregation predicate papered over the mismatch by testing for both spellings instead of resolving it. Converged on `CONCERNS`, the form already used by all nine copies of `nfr-criteria.md` and by the gate's `GateDecision` type: the shared status-definitions file, all four worker step files (`step-04a`-`step-04d`, including their JSON output examples), and the aggregation predicate now agree. `grep -rn "'CONCERN'" src/` returns nothing.
+
+- The `trace` workflow's gate decision step carried a Rule 6 that was only a comment: `// Rule 6: Manual waiver — set gateDecision = 'WAIVED' and update rationale here if a stakeholder-approved waiver applies (wired through config or user input upstream)`, with no code assigning `'WAIVED'` and no waiver field anywhere upstream in the step's inputs. Meanwhile `trace-template.md`'s Waiver Details section and `checklist.md`'s Decision Integrity and Waiver Scenarios checks already specify and validate a full waiver contract (approver, approval date, reason, expiry, monitoring plan, remediation owner, fix target), so the artifact-level contract existed for a code path that didn't. Resolved as intentional: the comment is replaced with a stated rule that Rules 1-5 are the only rules that set `gateDecision` automatically, and `WAIVED` is never derived by this step from coverage data or any other input, only applied by a human overriding the automated decision, with the resulting artifact required to carry the waiver contract. `risk-governance.md`'s unrelated `requestWaiver()` takes `expiryDays` as a caller-supplied parameter with no default anywhere in the repo, confirming there is no implicit expiry to wire in either.
+
+- `risk-governance.md`'s Example 2 (`evaluateGate()`, all nine copies) is a complete gate decision function keyed on risk scores: FAIL on any score-9 risk or unresolved coverage gap, WAIVED when every risk is waived by an approver, CONCERNS on scores 6-8 with mitigation plans and owners, PASS otherwise. It produces the same four words (PASS/CONCERNS/FAIL/WAIVED) as the gate the `trace` workflow actually runs (`step-05-gate-decision.md`, keyed on coverage percentages: P0 at 100%, overall at 80%, P1 at 90%/80%), so an agent asked "how does the gate decide" after loading this fragment could answer with the engine that never executes. Kept, since the risk-driven framing teaches something the coverage engine doesn't, but labelled: a callout now sits immediately under the Example 2 heading naming `step-05-gate-decision.md` as the executed rule set and noting the two engines' different inputs, and the section's own Key Points list gets a matching bullet.
+
+- `module-help.csv` records `test-design` in phase `3-solutioning` only, while the workflow is dual-mode and also runs per epic in Phase 4, and the CSV has a single `phase` column with no precedent for a multi-value entry. Decided the column means the phase the row's `preceded-by`/`followed-by` dependency chain belongs to, not every phase the workflow can run in: no code in the repository parses this column, and the Phase 4 per-epic invocation has no dependency edges of its own to encode. `README.md`'s "order you run things" diagram already matched this reading; added one sentence stating it explicitly so the relationship between the prose, the diagram, and the CSV isn't left for a reader to infer.
+
 - `teach-me-testing` session 7 could reach only 42 of the 59 knowledge fragments. Every mobile fragment (`maestro-flows`, `mobile-test-strategy`, `mobile-ci-device-lab`), the entire seven-fragment webhook family, both integration mandates, `library-integration-mandate`, `confidence-gate`, and `evidence-integrity` had no category to appear under, so the only browsable view of the knowledge base silently hid a third of it. The menu gains a Mobile category and a Webhooks category, the existing categories absorb the rest, and the sixteen places that advertised "42 fragments" now state the real number. `test-knowledge-base.js` Test Suite 6 asserts all of it: every fragment appears exactly once, no phantom entries, each category subtotal matches its own list, and every stated total matches the base. The count matching the short menu is what kept the shortfall invisible.
 
+- `README.md` described a version of TEA that stopped existing around 1.20.0. The Architecture and Flow section predated parallel workers, so it never mentioned `tea_execution_mode`, `tea_capability_probe`, the capability probe's fallback order, or the fact that five workflows split their heaviest step across isolated workers that communicate only through validated JSON. It never mentioned `criteria-registry.md`, the `tea-test-review` CLI (a published `bin` of this package), or the write-time enforcement hook, so the three control points TEA now occupies read as one. The building-blocks table said `steps-c/` held "5-9 sequential files" when the real range is 5 to 12 and 75 in total; the runtime section claimed a backend project pulls "~1,800 lines of Playwright Utils fragments," an unverifiable number of the class removed in 1.22.1 and backwards besides; the configuration list omitted `tea_execution_mode` and `tea_capability_probe`, both prompted at install; `risk_threshold` was labelled "(future)" when nothing reads it at all; the release guide understated `npm test` as seven checks when it chains thirteen; and the "45 platforms" claim was transcribed from the installer's repository, where it can change without anything here noticing. Adds a release-gate section covering the live-evidence CONCERNS cap and gate ineligibility, a repository-layout tree that includes `cli/`, `tools/`, `test/`, and `website/`, and a section on the checks that keep TEA's own rules and knowledge in agreement.
+
 - `quality.yaml` did not run the full test suite. `test:knowledge`, `test:changelog`, and `test:tea-workflow-descriptions` ran only from `.husky/pre-commit`, which `git commit --no-verify` skips and which no GitHub web-UI edit ever reaches. `test:knowledge` is the suite carrying the knowledge-base parity check across the eight workflow copies, so until now nothing in CI stopped the workflow copies from silently diverging from the agent's. All three now run in the `validate` job, alongside the three new checks above. `test:cli` moved to its own job: it takes 12m21s measured, and leaving it in line made every fast check behind it report twelve minutes late. It stays out of `npm test` for the same reason, since `npm test` runs on every commit.
 
 ## [1.22.6] - 2026-08-14