bmad-method-test-architecture-enterprise 1.27.2 → 1.27.3-next.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/README.md +7 -7
- package/package.json +4 -3
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "1.27.
|
|
34
|
+
"version": "1.27.3-next.0",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/README.md
CHANGED
|
@@ -367,32 +367,32 @@ test/ # quality gate suites and the six eval harnesses
|
|
|
367
367
|
|
|
368
368
|
## How TEA Keeps Itself Honest
|
|
369
369
|
|
|
370
|
-
TEA has deterministic checks and live evals.
|
|
370
|
+
TEA has deterministic checks and live evals. All ten skills have behavioral coverage registered in the suite manifest.
|
|
371
371
|
|
|
372
372
|
### Current Eval Coverage
|
|
373
373
|
|
|
374
374
|
The eight suites under `test/evals/` measure one decision inside each knowledge-bearing workflow: whether the agent selects the required knowledge fragments and avoids fragments the workflow excludes. They do not execute the complete workflow or grade its final artifact.
|
|
375
375
|
|
|
376
|
-
The `atdd`, `automate`, `ci`, `nfr`, `test-design`, `test-review`, and `trace` suites add behavioral evals, and so does `bmad-tea`'s own routing suite. `atdd` generates red-phase scaffolds against an unimplemented fixture, activates them, executes them under NFR9's isolation, and scores whether each one fails for the reason its criterion states. `automate` runs four hand-authored spec sets, standing in for generated coverage, against a fixed voucher-redemption service and against a scratch copy carrying its seeded minimum-spend regression, and scores whether the one spec set that tests the boundary catches the regression and whether the other three are correctly reported as missing it, vacuous, or duplicated rather than as clean passes; it spends no vendor call, since there is no live-agent mode to build one for. `ci` scaffolds a pipeline for a project whose team wrote down what it needs, then parses the generated workflow, lints it with `actionlint`, and scores every requested trigger, permission, command, gate, and artifact against a minimal-request control. `nfr` audits one evidence bundle with known gaps and one clean bundle. `test-design` analyzes one seeded epic and one clean control epic. `test-review` runs the complete review against files containing nine planted defects plus one clean file, then scores recall, the non-false-positive rate, score variance, and verdict stability. `trace` runs the complete traceability workflow against a ten-criterion seeded set whose declared gate is FAIL and a five-criterion clean set whose declared gate is PASS, then scores criterion status, the gate decision and the criteria behind it, the coverage arithmetic, evidence citation, and case stability. `bmad-tea` routes eighteen intents to the right menu item, or declines the ones nothing on the menu serves.
|
|
376
|
+
The `atdd`, `automate`, `ci`, `framework`, `nfr`, `teach-me-testing`, `test-design`, `test-review`, and `trace` suites add behavioral evals, and so does `bmad-tea`'s own routing suite. `atdd` generates red-phase scaffolds against an unimplemented fixture, activates them, executes them under NFR9's isolation, and scores whether each one fails for the reason its criterion states. `automate` runs four hand-authored spec sets, standing in for generated coverage, against a fixed voucher-redemption service and against a scratch copy carrying its seeded minimum-spend regression, and scores whether the one spec set that tests the boundary catches the regression and whether the other three are correctly reported as missing it, vacuous, or duplicated rather than as clean passes; it spends no vendor call, since there is no live-agent mode to build one for. `ci` scaffolds a pipeline for a project whose team wrote down what it needs, then parses the generated workflow, lints it with `actionlint`, and scores every requested trigger, permission, command, gate, and artifact against a minimal-request control. `framework` installs and smoke-tests a scaffolded framework in a network-isolated sandbox, proving it runs rather than merely reading as valid. `nfr` audits one evidence bundle with known gaps and one clean bundle. `teach-me-testing` tests multi-turn teaching sessions, seeded quiz corrections, and persistent learner progress through `tea-transcript-runner`. `test-design` analyzes one seeded epic and one clean control epic. `test-review` runs the complete review against files containing nine planted defects plus one clean file, then scores recall, the non-false-positive rate, score variance, and verdict stability. `trace` runs the complete traceability workflow against a ten-criterion seeded set whose declared gate is FAIL and a five-criterion clean set whose declared gate is PASS, then scores criterion status, the gate decision and the criteria behind it, the coverage arithmetic, evidence citation, and case stability. `bmad-tea` routes eighteen intents to the right menu item, or declines the ones nothing on the menu serves.
|
|
377
377
|
|
|
378
378
|
| Skill | Fragment-selection cases | Full behavioral eval |
|
|
379
379
|
| --------------------------- | ------------------------ | ----------------------------------------------------------------------------------- |
|
|
380
380
|
| `bmad-tea` | N/A | Yes; eighteen intents, ten with a right answer and eight controls |
|
|
381
|
-
| `bmad-teach-me-testing` | N/A |
|
|
381
|
+
| `bmad-teach-me-testing` | N/A | Yes; multi-turn session with seeded wrong quiz answer, review loop, and progress |
|
|
382
382
|
| `bmad-testarch-atdd` | 3 | Yes; one unimplemented fixture and five acceptance criteria |
|
|
383
383
|
| `bmad-testarch-automate` | 5 | Yes; four hand-authored spec sets against a fixed service and its seeded regression |
|
|
384
384
|
| `bmad-testarch-ci` | 2 | Yes; one full request and one minimal request |
|
|
385
|
-
| `bmad-testarch-framework` | 3 |
|
|
385
|
+
| `bmad-testarch-framework` | 3 | Yes; install and smoke-test generated scaffold in network-isolated sandbox |
|
|
386
386
|
| `bmad-testarch-nfr` | 2 | Yes; one evidence bundle with known gaps and one clean bundle |
|
|
387
387
|
| `bmad-testarch-test-design` | 5 | Yes; one seeded epic and one clean control epic |
|
|
388
388
|
| `bmad-testarch-test-review` | 2 | Yes; three files, nine planted defects, and one clean file |
|
|
389
389
|
| `bmad-testarch-trace` | 2 | Yes; a ten-criterion seeded set and a five-criterion clean set |
|
|
390
390
|
|
|
391
|
-
A passing fragment-selection eval means the workflow loaded the right knowledge.
|
|
391
|
+
A passing fragment-selection eval means the workflow loaded the right knowledge. All ten TEA skills now have behavioral coverage registered in the suite manifest, and the manifest's deferred array is empty. [How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) explains the testing hierarchy, exit classes, and evidence model in plain language. The source-controlled [Eval Quality and Behavioral Coverage Roadmap](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-roadmap/) records the per-skill contracts, runner work, CI plan, and intended boundary with the upcoming standalone `eval-quality` project. [The eval-quality Command Adapter](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-command-adapter/) records how TEA runs the commands those evals measure, and [Adopting eval-quality, One Skill at a Time](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/eval-quality-adoption-guide/) is the guide for taking one skill in another BMAD module from asserted quality to measured quality.
|
|
392
392
|
|
|
393
393
|
### Deterministic Checks
|
|
394
394
|
|
|
395
|
-
`npm test` chains seventy-
|
|
395
|
+
`npm test` chains seventy-six checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Eleven of the seventy-six keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
|
|
396
396
|
|
|
397
397
|
- `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
|
|
398
398
|
- `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, and the fragment-selection case count. A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
|
|
@@ -550,7 +550,7 @@ npm run eval:fragment-selection -- \
|
|
|
550
550
|
|
|
551
551
|
### What Has to Pass
|
|
552
552
|
|
|
553
|
-
Every row below is a declared gate.
|
|
553
|
+
Every row below is a declared gate. All ten skills now have behavioral suites registered in the suite manifest. Recorded live baselines preserve timestamped history, and [How TEA Is Tested](https://bmad-code-org.github.io/bmad-method-test-architecture-enterprise/explanation/how-tea-is-tested/) describes the evidence model. The live thresholds are the ones this repository's own eval suite manifest declares, and `npm run test:eval-schemas` fails when a harness constant and the manifest disagree.
|
|
554
554
|
|
|
555
555
|
| Eval | Declared threshold | Declared volume |
|
|
556
556
|
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.27.
|
|
4
|
+
"version": "1.27.3-next.0",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
|
@@ -83,7 +83,7 @@
|
|
|
83
83
|
"release:minor": "gh workflow run publish.yaml -f channel=latest -f bump=minor",
|
|
84
84
|
"release:next": "gh workflow run publish.yaml -f channel=next",
|
|
85
85
|
"release:patch": "gh workflow run publish.yaml -f channel=latest -f bump=patch",
|
|
86
|
-
"test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:nfr-evidence-guidance && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
|
|
86
|
+
"test": "npm run test:conflict-markers && npm run test:schemas && npm run test:install && npm run test:knowledge && npm run test:criteria-fragments && npm run test:doc-count-sources && npm run test:doc-counts && npm run test:doc-claim-sources && npm run test:doc-claims && npm run test:doc-invocation-entry && npm run test:doc-invocations && npm run test:enforce-hook && npm run test:framework-scaffold && npm run test:atdd-isolation && npm run test:atdd-net-guard && npm run test:atdd-scoring && npm run test:atdd-workflow-guidance && npm run test:framework-scaffold-install-isolation && npm run test:transcript-harness && npm run test:suite-manifest && npm run test:teach-me-testing-scoring && npm run test:eval-data && npm run test:eval-atdd-data && npm run test:eval-ci-data && npm run test:eval-framework-scaffold-data && npm run test:eval-nfr-data && npm run test:nfr-validate-corpus && npm run test:nfr-evidence-guidance && npm run test:eval-test-design-data && npm run test:eval-trace-data && npm run test:eval-routing-data && npm run test:eval-routing-boundaries && npm run test:eval-routing-evidence && npm run test:eval-transcript-data && npm run test:eval-teach-me-testing-data && npm run test:eval-schemas && npm run test:eval-diagnostics && npm run test:eval-replay && npm run test:eval-quality-corpus && npm run test:schema-versions && npm run test:contract-sources && npm run test:contracts && npm run test:contract-oracles && npm run test:probe-sources && npm run test:probe-corpus && npm run test:compare-dominance && npm run test:compare-eval-runs && npm run test:trial-set-scoring && npm run test:automate-eval-fixture && npm run test:eval-automate-data && npm run test:clock-port && npm run test:probe-conformance && npm run test:corpus-conformance && npm run test:file-system-conformance && npm run test:file-system-port && npm run test:probe-targets && npm run test:port-totality && npm run test:ci-coverage && npm run test:ci-coverage-filters && npm run test:bmad-output-gated && npm run test:release-metadata && npm run test:lockfile-age && npm run test:lockfile-age-cache && npm run test:licences && npm run test:supply-chain && npm run test:boundary && npm run test:direction && npm run test:lineage && npm run test:guard-publish && npm run test:layering-boundary-lineage && npm run test:changelog && npm run test:tea-workflow-descriptions && npm run validate:schemas && npm run lint && npm run lint:md && npm run format:check",
|
|
87
87
|
"test:atdd-isolation": "node test/test-atdd-isolation.js",
|
|
88
88
|
"test:atdd-net-guard": "node test/test-atdd-net-guard.js",
|
|
89
89
|
"test:atdd-scoring": "node test/test-atdd-scoring.js",
|
|
@@ -157,6 +157,7 @@
|
|
|
157
157
|
"test:tea-workflow-descriptions": "node tools/validate-tea-workflow-descriptions.js",
|
|
158
158
|
"test:teach-me-testing-scoring": "node test/test-teach-me-testing-scoring.js",
|
|
159
159
|
"test:transcript-harness": "node test/test-transcript-harness.js",
|
|
160
|
+
"test:trial-set-scoring": "node test/test-trial-set-scoring.js",
|
|
160
161
|
"validate:schemas": "node tools/validate-agent-schema.js"
|
|
161
162
|
},
|
|
162
163
|
"lint-staged": {
|
|
@@ -228,7 +229,7 @@
|
|
|
228
229
|
"eslint-plugin-n": "^17.21.3",
|
|
229
230
|
"eslint-plugin-unicorn": "^60.0.0",
|
|
230
231
|
"eslint-plugin-yml": "^1.18.0",
|
|
231
|
-
"eval-quality": "
|
|
232
|
+
"eval-quality": "4.0.0",
|
|
232
233
|
"husky": "^9.1.7",
|
|
233
234
|
"jest": "^30.0.4",
|
|
234
235
|
"lint-staged": "^16.1.1",
|