bmad-method-test-architecture-enterprise 1.27.2 → 1.27.3-next.1

package/.claude-plugin/marketplace.json

@@ -31,7 +31,7 @@
       "name": "bmad-method-test-architecture-enterprise",
       "source": "./",
       "description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
-      "version": "1.27.2",
+      "version": "1.27.3-next.1",
       "author": {
         "name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
       },
@@ -40,6 +40,7 @@
         "./src/workflows/testarch/bmad-testarch-atdd",
         "./src/workflows/testarch/bmad-testarch-automate",
         "./src/workflows/testarch/bmad-testarch-ci",
+        "./src/workflows/testarch/bmad-testarch-evaluate",
         "./src/workflows/testarch/bmad-testarch-framework",
         "./src/workflows/testarch/bmad-testarch-nfr",
         "./src/workflows/testarch/bmad-testarch-test-design",

package/README.md

@@ -8,7 +8,7 @@
 TEA is a standalone BMAD module that delivers risk-based test strategy, test automation guidance, and release gate decisions. It ships:
 
 - one expert agent, Murat, Master Test Architect and Quality Advisor
-- nine workflows spanning Teach Me Testing (TEA Academy), test design, framework setup, CI guidance, ATDD, automation, test review, NFR Evidence Audit, and traceability
+- ten workflows spanning Teach Me Testing (TEA Academy), test design, framework setup, CI guidance, ATDD, automation, test review, NFR Evidence Audit, traceability, and Evaluate
 - a 36-row criteria registry that fixes the severity of every reviewable violation, so a score is a lookup rather than a judgment call
 - `tea-test-review`, a headless CLI that runs the review workflow as a CI gate with real exit codes
 - a write-time enforcement hook that blocks the mechanically decidable violations before they reach disk
@@ -50,7 +50,7 @@ TEA's answer is not a better prompt. It is to make the work repeatable at three
 
 ### The order you run things
 
-The nine workflows are a directed graph, not a menu. `src/module-help.csv` encodes it.
+The ten workflows are a directed graph, not a menu. `src/module-help.csv` encodes it.
 
 ```text
 Phase 3, solutioning, once per project
@@ -62,6 +62,7 @@ Phase 4, implementation, per story
 Epic or release gate
   TA  automate  →  RV  test-review
   TA  automate  →  NR  nfr
+  TA  automate  →  EV  evaluate  →  CI  ci
   RV  test-review  →  TR  trace (Phase 2 gate decision)
 ```
 
@@ -313,9 +314,10 @@ A copy-paste workflow lives at `cli/examples/pr-test-review.yml`, and the full f
 
 ## Configuration
 
-TEA variables are defined in `src/module.yaml` and prompted during install. Ten are wired into workflows today; the last four are placeholders that nothing reads yet.
+TEA variables are defined in `src/module.yaml` and prompted during install. Eleven are wired into workflows today; the last four are placeholders that nothing reads yet.
 
 - `test_artifacts` — base output folder for test artifacts
+- `tea_evaluations_folder` — base folder for Evaluate's own evaluation folders (string, default `evals`), resolved as `{project-root}/{value}`
 - `tea_use_playwright_utils` — enable Playwright Utils integration (boolean, default true). When true **and the package is installed**, `@seontechnologies/playwright-utils` becomes the default implementation for everything it covers: generated Playwright tests use `interceptNetworkCall`, `apiRequest`, `recurse`, and `log` without being asked, and `test-review` flags a vanilla equivalent that carries no stated reason. See [Integrate Playwright Utils](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/how-to/customization/integrate-playwright-utils/)
 - `tea_use_pactjs_utils` — enable Pact.js Utils integration for contract testing (boolean, default true). It decides how Pact suites are written, not whether a project gets one: TEA still requires a real consumer-provider boundary before scaffolding a contract test. When on **and the package is installed**, generated Pact code uses `createProviderState`, `buildVerifierOptions`, and `createRequestFilter` rather than raw Pact boilerplate. A flag with no install generates the raw path and reports one recommendation rather than flagging every file
 - `tea_pact_mcp` — SmartBear MCP for PactFlow/Broker interaction: mcp, none (string, default mcp). Safe without a broker: every broker-dependent step degrades to provider source or an OpenAPI spec and reports that the broker was unreachable
@@ -347,11 +349,12 @@ src/                     # the shipped module
 ├── module.yaml          # install-time variables and post-install notes
 ├── module-help.csv      # workflow catalog: menu codes, phases, ordering
 ├── agents/bmad-tea/     # SKILL.md, customize.toml, resources/{tea-index.csv, knowledge/}
-└── workflows/testarch/  # nine self-contained workflow skills
+└── workflows/testarch/  # ten self-contained workflow skills
     ├── bmad-teach-me-testing/
     ├── bmad-testarch-atdd/
     ├── bmad-testarch-automate/
     ├── bmad-testarch-ci/
+    ├── bmad-testarch-evaluate/
     ├── bmad-testarch-framework/        # resources/hooks/tea-enforce.cjs lives here
     ├── bmad-testarch-nfr/
     ├── bmad-testarch-test-design/
@@ -367,32 +370,33 @@ test/                    # quality gate suites and the six eval harnesses
 
 ## How TEA Keeps Itself Honest
 
-TEA has deterministic checks and live evals. These cover specific risks. They are not end-to-end evals of every skill.
+TEA has deterministic checks and live evals. Ten of TEA's eleven skills have behavioral coverage registered in the suite manifest; `bmad-testarch-evaluate` is deferred there until Story 1.16 authors its own suite.
 
 ### Current Eval Coverage
 
 The eight suites under `test/evals/` measure one decision inside each knowledge-bearing workflow: whether the agent selects the required knowledge fragments and avoids fragments the workflow excludes. They do not execute the complete workflow or grade its final artifact.
 
-The `atdd`, `automate`, `ci`, `nfr`, `test-design`, `test-review`, and `trace` suites add behavioral evals, and so does `bmad-tea`'s own routing suite. `atdd` generates red-phase scaffolds against an unimplemented fixture, activates them, executes them under NFR9's isolation, and scores whether each one fails for the reason its criterion states. `automate` runs four hand-authored spec sets, standing in for generated coverage, against a fixed voucher-redemption service and against a scratch copy carrying its seeded minimum-spend regression, and scores whether the one spec set that tests the boundary catches the regression and whether the other three are correctly reported as missing it, vacuous, or duplicated rather than as clean passes; it spends no vendor call, since there is no live-agent mode to build one for. `ci` scaffolds a pipeline for a project whose team wrote down what it needs, then parses the generated workflow, lints it with `actionlint`, and scores every requested trigger, permission, command, gate, and artifact against a minimal-request control. `nfr` audits one evidence bundle with known gaps and one clean bundle. `test-design` analyzes one seeded epic and one clean control epic. `test-review` runs the complete review against files containing nine planted defects plus one clean file, then scores recall, the non-false-positive rate, score variance, and verdict stability. `trace` runs the complete traceability workflow against a ten-criterion seeded set whose declared gate is FAIL and a five-criterion clean set whose declared gate is PASS, then scores criterion status, the gate decision and the criteria behind it, the coverage arithmetic, evidence citation, and case stability. `bmad-tea` routes eighteen intents to the right menu item, or declines the ones nothing on the menu serves.
+The `atdd`, `automate`, `ci`, `framework`, `nfr`, `teach-me-testing`, `test-design`, `test-review`, and `trace` suites add behavioral evals, and so does `bmad-tea`'s own routing suite. `atdd` generates red-phase scaffolds against an unimplemented fixture, activates them, executes them under NFR9's isolation, and scores whether each one fails for the reason its criterion states. `automate` runs four hand-authored spec sets, standing in for generated coverage, against a fixed voucher-redemption service and against a scratch copy carrying its seeded minimum-spend regression, and scores whether the one spec set that tests the boundary catches the regression and whether the other three are correctly reported as missing it, vacuous, or duplicated rather than as clean passes; it spends no vendor call, since there is no live-agent mode to build one for. `ci` scaffolds a pipeline for a project whose team wrote down what it needs, then parses the generated workflow, lints it with `actionlint`, and scores every requested trigger, permission, command, gate, and artifact against a minimal-request control. `framework` installs and smoke-tests a scaffolded framework in a network-isolated sandbox, proving it runs rather than merely reading as valid. `nfr` audits one evidence bundle with known gaps and one clean bundle. `teach-me-testing` tests multi-turn teaching sessions, seeded quiz corrections, and persistent learner progress through `tea-transcript-runner`. `test-design` analyzes one seeded epic and one clean control epic. `test-review` runs the complete review against files containing nine planted defects plus one clean file, then scores recall, the non-false-positive rate, score variance, and verdict stability. `trace` runs the complete traceability workflow against a ten-criterion seeded set whose declared gate is FAIL and a five-criterion clean set whose declared gate is PASS, then scores criterion status, the gate decision and the criteria behind it, the coverage arithmetic, evidence citation, and case stability. `bmad-tea` routes nineteen intents to the right menu item, or declines the ones nothing on the menu serves.
 
 | Skill                       | Fragment-selection cases | Full behavioral eval                                                                |
 | --------------------------- | ------------------------ | ----------------------------------------------------------------------------------- |
-| `bmad-tea`                  | N/A                      | Yes; eighteen intents, ten with a right answer and eight controls                   |
-| `bmad-teach-me-testing`     | N/A                      | None; this skill has no workflow knowledge index                                    |
+| `bmad-tea`                  | N/A                      | Yes; nineteen intents, eleven with a right answer and eight controls                |
+| `bmad-teach-me-testing`     | N/A                      | Yes; multi-turn session with seeded wrong quiz answer, review loop, and progress    |
 | `bmad-testarch-atdd`        | 3                        | Yes; one unimplemented fixture and five acceptance criteria                         |
 | `bmad-testarch-automate`    | 5                        | Yes; four hand-authored spec sets against a fixed service and its seeded regression |
 | `bmad-testarch-ci`          | 2                        | Yes; one full request and one minimal request                                       |
-| `bmad-testarch-framework`   | 3                        | None                                                                                |
+| `bmad-testarch-evaluate`    | N/A                      | No; deferred until Story 1.16 authors its own suite                                 |
+| `bmad-testarch-framework`   | 3                        | Yes; install and smoke-test generated scaffold in network-isolated sandbox          |
 | `bmad-testarch-nfr`         | 2                        | Yes; one evidence bundle with known gaps and one clean bundle                       |
 | `bmad-testarch-test-design` | 5                        | Yes; one seeded epic and one clean control epic                                     |
 | `bmad-testarch-test-review` | 2                        | Yes; three files, nine planted defects, and one clean file                          |
 | `bmad-testarch-trace`       | 2                        | Yes; a ten-criterion seeded set and a five-criterion clean set                      |
 
-A passing fragment-selection eval means the workflow loaded the right knowledge. It makes no claim about the quality of the workflow's final output. A full behavioral eval for the remaining skill, `bmad-testarch-framework`, is a coverage gap. The source-controlled [Eval Quality and Behavioral Coverage Roadmap](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-roadmap/) records the per-skill contracts, runner work, CI plan, and intended boundary with the upcoming standalone `eval-quality` project. [The eval-quality Command Adapter](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-command-adapter/) records how TEA runs the commands those evals measure, and [Adopting eval-quality, One Skill at a Time](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-adoption-guide/) is the guide for taking one skill in another BMAD module from asserted quality to measured quality.
+A passing fragment-selection eval means the workflow loaded the right knowledge. Ten of TEA's eleven skills have behavioral coverage registered in the suite manifest; `bmad-testarch-evaluate` is named in the manifest's deferred array with its owner, missing evidence, and exit condition until Story 1.16 lands its own suite. [How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) explains the testing hierarchy, exit classes, and evidence model in plain language. The source-controlled [Eval Quality and Behavioral Coverage Roadmap](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-roadmap/) records the per-skill contracts, runner work, CI plan, and intended boundary with the upcoming standalone `eval-quality` project. [The eval-quality Command Adapter](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-command-adapter/) records how TEA runs the commands those evals measure, and [Adopting eval-quality, One Skill at a Time](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-adoption-guide/) is the guide for taking one skill in another BMAD module from asserted quality to measured quality.
 
 ### Deterministic Checks
 
-`npm test` chains seventy-five checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Eleven of the seventy-five keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
+`npm test` chains seventy-seven checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Twelve of the seventy-seven keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
 
 - `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
 - `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, and the fragment-selection case count. A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
@@ -405,6 +409,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
 - `test:eval-diagnostics` checks that current result writers emit bounded case and repetition diagnostics, that quality and environment outcomes stay separate, that unstable signatures remain visible, and that the protected 1.3.0 baseline remains readable and byte-for-byte unchanged.
 - `test:nfr-evidence-guidance` holds every NFR worker and the publication audit to the supplied-evidence ledger, exercises supported, missing, and mismatched evidence, rejects contaminating examples, and binds the protected baseline and focused remediation records by digest.
 - `test:contract-oracles` evaluates every oracle in every eval contract with `eval-quality`'s own evaluator, over the stored replay outputs and over constructed selections for all 24 fragment-selection cases, and fails when an oracle faults or disagrees with the harness scorer on the same evidence. The first run found that every regex oracle in the `test-review` contract was refused by the evaluator before it matched anything.
+- `test:evaluate-guidance` parses the Evaluate skill's `SKILL.md` stage list and fails when a stage entry is missing, out of order, or points at a `references/` guide that does not exist.
 
 These checks produce the same answer from the same repository state. They need no agent credential, network call, or model budget. `test:eval-data` runs through `npm test`, the local pre-commit hook, pull-request quality checks, and the publish workflow.
 
@@ -428,7 +433,7 @@ npm run eval:all -- --agent agy
 npm run eval:all -- --agent agy --agent claude --agent codex
 ```
 
-`eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `ci` project, per `test-design` epic, per `trace` fixture set, and per `atdd` story. One runner makes 105 agent calls: 48 fragment selections, 36 routing intents, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, and 2 ATDD generations. All three built-in runners make 315 calls.
+`eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `ci` project, per `test-design` epic, per `trace` fixture set, and per `atdd` story. One runner makes 107 agent calls: 48 fragment selections, 38 routing intents, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, and 2 ATDD generations. All three built-in runners make 321 calls.
 
 Check the data, executable, login, fixtures, and expected results without making a model call:
 
@@ -550,15 +555,15 @@ npm run eval:fragment-selection -- \
 
 ### What Has to Pass
 
-Every row below is a declared gate. The three suites that existed then were measured live for the first time on 2026-09-08 against `claude`/`sonnet`, and the [Eval Quality and Behavioral Coverage Roadmap](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-roadmap/) records what each one returned; `bmad-tea-routing`, `nfr`, `test-design`, `ci` and `atdd` have been added since and have never been run live, so their rows are declared targets and nothing more; no result artifact is committed, so quote that section rather than a number from memory. The live thresholds are the ones this repository's own eval suite manifest declares, and `npm run test:eval-schemas` fails when a harness constant and the manifest disagree.
+Every row below is a declared gate. Ten of TEA's eleven skills have behavioral suites registered in the suite manifest; `bmad-testarch-evaluate` is deferred until Story 1.16. Recorded live baselines preserve timestamped history, and [How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) describes the evidence model. The live thresholds are the ones this repository's own eval suite manifest declares, and `npm run test:eval-schemas` fails when a harness constant and the manifest disagree.
 
 | Eval                              | Declared threshold                                                                                                                                                                                                                                                                                                                                                                                 | Declared volume                                                                                            |
 | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
-| `npm run eval:all -- --agent ...` | All eight live suites below meet their thresholds for the selected runner                                                                                                                                                                                                                                                                                                                          | 48 selections, 36 routing calls, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, 2 generations |
+| `npm run eval:all -- --agent ...` | All eight live suites below meet their thresholds for the selected runner                                                                                                                                                                                                                                                                                                                          | 48 selections, 38 routing calls, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, 2 generations |
 | Fragment selection                | At least 90% required-fragment recall, at most 10% forbidden-fragment selection, and stable choices across repeated cases                                                                                                                                                                                                                                                                          | 24 cases twice: 48 calls                                                                                   |
 | Test review                       | At least 70% overall recall, 100% CRITICAL recall, an 80% non-false-positive rate, score standard deviation no higher than 3, and a stable verdict                                                                                                                                                                                                                                                 | Three complete reviews                                                                                     |
 | Trace                             | At least 90% criterion-status accuracy and 90% evidence-citation precision; 100% on both discriminating criteria, the gate decision, the gate criteria, the coverage arithmetic, the oracle resolution, the rejected evidence, the waiver validity, and the live evidence; no clean-set false positive, no unstable case, and no fixture mutation                                                  | Two fixture sets twice: 4 calls                                                                            |
-| bmad-tea routing                  | At least 90% route accuracy, 80% reason-token recall, 87.5% scope fidelity, and 75% on both clarification and decline recall; zero confident routes on an intent nothing on the menu serves, zero on one whose menu items are genuinely close, zero clear intents left unrouted, and at most two intents whose answer changes between repetitions                                                  | 18 intents twice: 36 calls                                                                                 |
+| bmad-tea routing                  | At least 90% route accuracy, 80% reason-token recall, 87.5% scope fidelity, and 75% on both clarification and decline recall; zero confident routes on an intent nothing on the menu serves, zero on one whose menu items are genuinely close, zero clear intents left unrouted, and at most two intents whose answer changes between repetitions                                                  | 19 intents twice: 38 calls                                                                                 |
 | NFR                               | At least 87.5% domain-status accuracy and 90% threshold fidelity; 100% on undecidable domains, the overall status, and domain coverage; no unsupported PASS, no fabricated evidence, no clean-bundle false positive, no duplicate domain section, no disagreement between the gate artifact and the assessment it summarizes, no unstable case, and no fixture mutation                            | Two evidence bundles twice: 4 calls                                                                        |
 | CI                                | Zero parse failures and zero `actionlint` findings; at least 90% requested-element recall and 100% trigger accuracy; zero unrequested elements, zero workflow rule violations, no unstable case, and no fixture mutation                                                                                                                                                                           | Two projects twice: 4 calls                                                                                |
 | Test design                       | At least 80% grounded-risk recall, 80% risk precision, and 80% coverage mapping; 100% on the scale compliance, the score arithmetic, the category validity, the band placement, the risk-id shape, the risk-link resolution, and the priority ordering; no unscored risk table, no ungrounded risk, no missed top-severity risk, no risk-ceiling excess, no unstable case, and no fixture mutation | Two epics twice: 4 calls                                                                                   |

package/package.json

@@ -1,7 +1,7 @@
 {
   "$schema": "https://json.schemastore.org/package.json",
   "name": "bmad-method-test-architecture-enterprise",
-  "version": "1.27.2",
+  "version": "1.27.3-next.1",
   "description": "Master Test Architect for quality strategy, test automation, and release gates",
   "keywords": [
     "bmad",
@@ -47,7 +47,9 @@
     "cli",
     ".claude-plugin",
     "README.md",
-    "LICENSE"
+    "LICENSE",
+    "!src/workflows/testarch/*/.memlog.md",
+    "!src/workflows/testarch/*/.analysis/**"
   ],
   "scripts": {
     "docs:build": "node tools/build-docs.js",
@@ -83,7 +85,7 @@
     "release:minor": "gh workflow run publish.yaml -f channel=latest -f bump=minor",
     "release:next": "gh workflow run publish.yaml -f channel=next",
     "release:patch": "gh workflow run publish.yaml -f channel=latest -f bump=patch",
-    "test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:nfr-evidence-guidance && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
+    "test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:evaluate-guidance && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:nfr-evidence-guidance && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:trial-set-scoring && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
     "test:atdd-isolation": "node test/test-atdd-isolation.js",
     "test:atdd-net-guard": "node test/test-atdd-net-guard.js",
     "test:atdd-scoring": "node test/test-atdd-scoring.js",
@@ -130,6 +132,7 @@
     "test:eval-test-design-data": "node test/eval-test-design.js --validate-only && node test/test-test-design-grounding.js",
     "test:eval-trace-data": "node test/eval-trace.js --validate-only",
     "test:eval-transcript-data": "node test/eval-transcript.js --validate-only",
+    "test:evaluate-guidance": "node test/test-evaluate-guidance.js",
     "test:file-system-conformance": "node test/test-file-system-conformance.js",
     "test:file-system-port": "node test/test-file-system-port.js",
     "test:framework-scaffold": "node test/test-framework-scaffold.js",
@@ -154,9 +157,10 @@
     "test:schemas": "node test/test-agent-schema.js",
     "test:suite-manifest": "node test/test-suite-manifest.js",
     "test:supply-chain": "node test/test-supply-chain.js",
-    "test:tea-workflow-descriptions": "node tools/validate-tea-workflow-descriptions.js",
+    "test:tea-workflow-descriptions": "node test/test-tea-workflow-descriptions.js",
     "test:teach-me-testing-scoring": "node test/test-teach-me-testing-scoring.js",
     "test:transcript-harness": "node test/test-transcript-harness.js",
+    "test:trial-set-scoring": "node test/test-trial-set-scoring.js",
     "validate:schemas": "node tools/validate-agent-schema.js"
   },
   "lint-staged": {
@@ -228,7 +232,7 @@
     "eslint-plugin-n": "^17.21.3",
     "eslint-plugin-unicorn": "^60.0.0",
     "eslint-plugin-yml": "^1.18.0",
-    "eval-quality": "3.4.0",
+    "eval-quality": "4.0.0",
     "husky": "^9.1.7",
     "jest": "^30.0.4",
     "lint-staged": "^16.1.1",

package/src/agents/bmad-tea/customize.toml

@@ -87,6 +87,11 @@ code = "TA"
 description = "Test Automation — generate prioritized API/E2E tests, fixtures, and DoD summary for a story or feature"
 skill = "bmad-testarch-automate"
 
+[[agent.menu]]
+code = "EV"
+description = "Evaluate — build and run a scored, evidence-backed evaluation of an agent, skill, workflow, or tool-use system, with seeded-defect proof and CI enforcement"
+skill = "bmad-testarch-evaluate"
+
 [[agent.menu]]
 code = "GATE"
 description = "Release Gate — route final audit, NFR evidence audit, and trace gate decision"

package/src/module-help.csv

@@ -6,6 +6,7 @@ Test Architecture Enterprise,bmad-testarch-framework,Test Framework,TF,Initializ
 Test Architecture Enterprise,bmad-testarch-ci,CI Setup,CI,Configure CI/CD quality pipeline.,,,3-solutioning,bmad-testarch-framework,,false,test_artifacts,ci config
 Test Architecture Enterprise,bmad-testarch-atdd,ATDD,AT,Generate red-phase acceptance test scaffolds before implementation.,,,4-implementation,bmad-create-story:create,bmad-dev-story,false,test_artifacts,atdd-checklist|red-phase acceptance tests
 Test Architecture Enterprise,bmad-testarch-automate,Test Automation,TA,Expand test coverage.,,,4-implementation,bmad-testarch-atdd,,false,test_artifacts,test suite
+Test Architecture Enterprise,bmad-testarch-evaluate,Evaluate,EV,"Build and run a scored, evidence-backed evaluation of a target.",,,4-implementation,bmad-testarch-automate,bmad-testarch-ci,false,tea_evaluations_folder,evaluation folder|scored contract
 Test Architecture Enterprise,bmad-testarch-test-review,Test Review,RV,Quality audit (0-100 scoring).,,,4-implementation,bmad-testarch-automate,,false,test_artifacts,review report
 Test Architecture Enterprise,bmad-testarch-nfr,NFR Evidence Audit,NR,Audit non-functional requirement evidence.,,,4-implementation,bmad-testarch-automate,,false,test_artifacts,nfr report
 Test Architecture Enterprise,bmad-testarch-trace,Traceability,TR,Coverage traceability and gate.,,,4-implementation,bmad-testarch-test-review,,false,test_artifacts,traceability matrix|gate decision

package/src/module.yaml

@@ -24,7 +24,7 @@ agents:
 # TEA-specific configuration variables
 # IMPORTANT: tea_use_playwright_utils, tea_use_pactjs_utils, tea_pact_mcp, tea_browser_automation,
 # tea_execution_mode, tea_capability_probe, test_stack_type, ci_platform,
-# and test_framework are ACTIVELY USED in workflows
+# test_framework, and tea_evaluations_folder are ACTIVELY USED in workflows
 # Other variables are FUTURE placeholders for Phase 10 enhancements
 
 test_artifacts:
@@ -32,6 +32,13 @@ test_artifacts:
   default: "{output_folder}/test-artifacts"
   result: "{project-root}/{value}"
 
+# ✅ ACTIVELY USED - Evaluate (bmad-testarch-evaluate) reads this on activation and
+# scaffolds every evaluation folder under it.
+tea_evaluations_folder:
+  prompt: "Where should Evaluate store evaluation folders?"
+  default: "evals"
+  result: "{project-root}/{value}"
+
 # ✅ ACTIVELY USED - Playwright Utils integration (referenced in 6 workflows: automate, atdd, framework, test-design, test-review, ci)
 # CRITICAL: Must be boolean (true/false) not string - workflows check "if config.tea_use_playwright_utils: true"
 # When true, playwright-utils-mandate.md binds generation and review: the package is the default

package/src/workflows/testarch/bmad-testarch-evaluate/SKILL.md

@@ -0,0 +1,122 @@
+---
+name: bmad-testarch-evaluate
+description: 'Build and run a scored, evidence-backed evaluation of a target. Use when the user says "evaluate this agent" or "I want proof this behaves correctly"'
+---
+
+# Evaluate
+
+**Goal:** Take an adopter from a described target to a compiling, sealed, preflighted, scored Behavioral Evaluation Contract whose clean arm passes and whose mutated arm catches every seeded defect, then wire the proof into CI.
+
+**Role:** You are the Master Test Architect.
+
+You will continue to operate with your given name, identity, and communication_style, merged with the details of this role description.
+
+## Conventions
+
+- Bare paths (e.g. `references/inspection.md`) resolve from the skill root.
+- `{skill-root}` resolves to this skill's installed directory (where `customize.toml` lives).
+- `{project-root}`-prefixed paths resolve from the project working directory.
+- `{skill-name}` resolves to the skill directory's basename.
+- Resolve sibling files such as `references/...` and `assets/...` from `{skill-root}`.
+
+## On Activation
+
+### Step 1: Resolve the Workflow Block
+
+Run: `uv run {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --project-root {project-root} --key workflow`
+
+**If the script fails**, resolve the `workflow` block yourself by reading these three files in base → team → user order and applying the same structural merge rules as the resolver:
+
+1. `{skill-root}/customize.toml` — defaults
+2. `{project-root}/_bmad/custom/{skill-name}.toml` — team overrides
+3. `{project-root}/_bmad/custom/{skill-name}.user.toml` — personal overrides
+
+Any missing file is skipped. Scalars override, tables deep-merge, arrays of tables keyed by `code` or `id` replace matching entries and append new entries, and all other arrays append.
+
+### Step 2: Execute Prepend Steps
+
+Execute each entry in `{workflow.activation_steps_prepend}` in order before proceeding.
+
+### Step 3: Load Persistent Facts
+
+Treat every entry in `{workflow.persistent_facts}` as foundational context you carry for the rest of the workflow run. Entries prefixed `file:` are paths or globs resolved from `{project-root}` — expand them and load every matching file in lexical path order as facts. All other entries are facts verbatim.
+
+### Step 4: Load Config
+
+Load config from `{project-root}/_bmad/tea/config.yaml` and resolve:
+
+- `user_name`
+- `communication_language`
+- `tea_evaluations_folder`
+
+### Step 5: Greet the User
+
+Greet `{user_name}`, speaking in `{communication_language}`.
+
+### Step 6: Execute Append Steps
+
+Execute each entry in `{workflow.activation_steps_append}` in order.
+
+Activation is complete. Begin the workflow below.
+
+## Workflow
+
+Evaluate is one continuous loop over twelve stages, run inline rather than as separate step files: each stage's craft lives in its own `references/` guide, loaded when that stage is reached. Ask the adopter which stage to start from when resuming earlier work; otherwise start at Stage 1. Work under `{tea_evaluations_folder}` unless the adopter names another location.
+
+If the loaded `references/<stage>.md` guide is a placeholder (it says "Placeholder." and names the story that fills it), tell the adopter that stage is not yet available and stop there. Never improvise the stage's craft yourself, and never compute a verdict, score, or pass/fail decision outside `eval-quality`'s own CLI (AD-6): a placeholder stage has no craft to improvise from, and a verdict this skill computed itself would not be one `eval-quality` sealed.
+
+### Stage 1: Inspection
+
+Inspect the target and identify its kind. Load `references/inspection.md`.
+
+### Stage 2: Intake
+
+Capture behavioral requirements and constraints the adopter confirms. Load `references/intake.md`.
+
+### Stage 3: Corpus
+
+Design the probe corpus. Load `references/corpus.md`.
+
+### Stage 4: Contract
+
+Author the Behavioral Evaluation Contract. Load `references/contract.md`.
+
+### Stage 5: Oracles
+
+Design oracles and rubrics. Load `references/oracles.md`.
+
+### Stage 6: Adapters
+
+Scaffold the execution-target registry and adapters. Load `references/adapters.md`.
+
+### Stage 7: Evaluator
+
+Choose or build the evaluation layer. Load `references/evaluator.md`.
+
+### Stage 8: Mutation
+
+Apply and roll back controlled mutations. Load `references/mutation.md`.
+
+### Stage 9: Harness
+
+Scaffold the runner and scoring policy. Load `references/harness.md`.
+
+### Stage 10: Run
+
+Run the clean and mutated arms and score them. Load `references/run.md`.
+
+### Stage 11: Gaps
+
+Interpret a weak result and close the gap. Load `references/gaps.md`.
+
+### Stage 12: CI
+
+Wire continuous proof into the adopter's CI. Load `references/ci.md`.
+
+## On Complete
+
+Run: `uv run {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --project-root {project-root} --key workflow.on_complete`
+
+If the resolver succeeds and returns a non-empty `workflow.on_complete`, execute that value as the final terminal instruction before exiting.
+
+If the resolver fails, returns no output, or resolves an empty value, skip the hook and exit normally.

package/src/workflows/testarch/bmad-testarch-evaluate/assets/README.md

@@ -0,0 +1,6 @@
+# Evaluate assets
+
+Templates an authored evaluation validates against: scoring policy, evaluator
+configuration, and isolation manifest templates, and the `evaluation.json`
+folder scaffold. Starting with Story 1.8, each template ships with no
+threshold or identity values filled in, so a filled copy is the adopter's own.

package/src/workflows/testarch/bmad-testarch-evaluate/customize.toml

@@ -0,0 +1,38 @@
+# DO NOT EDIT -- overwritten on every update.
+#
+# Workflow customization surface for bmad-testarch-evaluate. Mirrors the
+# agent customization shape under the [workflow] namespace.
+
+[workflow]
+
+# --- Configurable below. Overrides merge per BMad structural rules: ---
+#   scalars: override wins • arrays (persistent_facts, activation_steps_*): append
+
+# Steps to run before the standard activation (config load, greet).
+# Overrides append. Use for pre-flight loads, compliance checks, etc.
+
+activation_steps_prepend = []
+
+# Steps to run after greet but before the workflow begins.
+# Overrides append. Use for context-heavy setup that should happen
+# once the user has been acknowledged.
+
+activation_steps_append = []
+
+# Persistent facts the workflow keeps in mind for the whole run
+# (target conventions, evaluation policy, prior findings). Distinct
+# from a run's own evidence -- these are static context loaded on
+# activation. Overrides append.
+#
+# Each entry is either:
+#   - a literal sentence, e.g. "Every probe corpus needs one clean control."
+#   - a file reference prefixed with `file:`, e.g. "file:{project-root}/docs/eval-standards.md"
+#     (glob patterns are supported; matching files load in lexical path order as facts).
+
+persistent_facts = []
+
+# Scalar: executed when the workflow reaches its terminal stage,
+# after the final outputs are produced. Override wins. Leave empty
+# for no custom post-completion behavior.
+
+on_complete = ""

package/src/workflows/testarch/bmad-testarch-evaluate/references/adapters.md

@@ -0,0 +1,6 @@
+# Adapters
+
+Placeholder. This guide will teach how to scaffold the runner command, MCP
+wiring, or HTTP port and register it in the execution-target registry, so
+preflight reduces a real observation with no authorization denial (FR6).
+Filled by Story 1.13.

package/src/workflows/testarch/bmad-testarch-evaluate/references/ci.md

@@ -0,0 +1,5 @@
+# CI
+
+Placeholder. This guide will teach how to wire continuous proof into the
+adopter's CI, with tiers placed from an inspection of the adopter's
+repository and a per-check enforcement class (FR11). Filled by Story 2.4.

package/src/workflows/testarch/bmad-testarch-evaluate/references/contract.md

@@ -0,0 +1,6 @@
+# Contract
+
+Placeholder. This guide will teach how to author the Behavioral Evaluation
+Contract under eval-quality's authoring discipline, so it compiles, seals, and
+carries a non-null observable success criterion on every behavior (FR4).
+Filled by Story 1.13.

package/src/workflows/testarch/bmad-testarch-evaluate/references/corpus.md

@@ -0,0 +1,6 @@
+# Corpus
+
+Placeholder. This guide will teach how to design the probe corpus per target
+kind: representative, negative and malformed inputs, held-out probes, clean
+controls, seeded-defect probes, zero-action defect probes, and gameability
+probes, plus the corpus layout and digest (FR3, AD-9). Filled by Story 1.12.

package/src/workflows/testarch/bmad-testarch-evaluate/references/evaluator.md

@@ -0,0 +1,6 @@
+# Evaluator
+
+Placeholder. This guide will teach how to choose or build the evaluation
+layer (the deterministic evaluator, a sealed-brief agent, an adopter harness,
+a skill-specific evaluator, custom code, or an external framework) behind the
+one framework-neutral import contract (FR13, AD-21). Filled by Story 1.23.

package/src/workflows/testarch/bmad-testarch-evaluate/references/gaps.md

@@ -0,0 +1,6 @@
+# Gaps
+
+Placeholder. This guide will teach how to interpret a CONCERNS, FAIL, or
+Invalid result: name the unsatisfied rule or missing evidence, propose and
+author the probe, control, or oracle that closes the gap, then rerun and
+rescore (FR10). Filled by Story 2.4.

package/src/workflows/testarch/bmad-testarch-evaluate/references/harness.md

@@ -0,0 +1,5 @@
+# Harness
+
+Placeholder. This guide will teach how to generate the scoring policy,
+evaluator configuration, and isolation manifest, with every threshold set by
+the adopter (FR8). Filled by Story 1.14.

package/src/workflows/testarch/bmad-testarch-evaluate/references/inspection.md

@@ -0,0 +1,7 @@
+# Inspection
+
+Placeholder. This guide will teach how to inspect a target (entry points,
+behaviors, surfaces, existing tests, failure history), identify its target
+kind, and map it to an interface kind and adapter shape, redirecting a request
+to evaluate a vendor model itself to the adopter's own use of it (FR1, AD-4).
+Filled by Story 1.12.

package/src/workflows/testarch/bmad-testarch-evaluate/references/intake.md

@@ -0,0 +1,5 @@
+# Intake
+
+Placeholder. This guide will teach how to capture the behavioral requirements
+and constraints inspection cannot infer, as a written statement the adopter
+confirms before corpus design (FR2). Filled by Story 1.12.

package/src/workflows/testarch/bmad-testarch-evaluate/references/mutation.md

@@ -0,0 +1,5 @@
+# Mutation
+
+Placeholder. This guide will teach how to apply and roll back each controlled
+mutation in a disposable copy, with baseline-pass and mutated-fail evidence
+and a measured `rollbackVerified` (FR7, AD-8). Filled by Story 1.14.

package/src/workflows/testarch/bmad-testarch-evaluate/references/oracles.md

@@ -0,0 +1,5 @@
+# Oracles
+
+Placeholder. This guide will teach how to design one oracle per discharged
+behavior and an anchored rubric wherever judgment needs a scale, judged by a
+calibrated judge (FR5). Filled by Story 1.13.

package/src/workflows/testarch/bmad-testarch-evaluate/references/run.md

@@ -0,0 +1,6 @@
+# Run
+
+Placeholder. This guide will teach how to run the clean and mutated arms
+through the chosen evaluation layer and drive compile, seal, preflight, and
+score end to end, so the clean arm resolves `passed-clean-control` and the
+seeded probe resolves `caught` (FR9). Filled by Story 1.14.