bmad-method-test-architecture-enterprise 1.27.3-next.21 → 1.27.3-next.22

package/.claude-plugin/marketplace.json

@@ -31,7 +31,7 @@
       "name": "bmad-method-test-architecture-enterprise",
       "source": "./",
       "description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
-      "version": "1.27.3-next.21",
+      "version": "1.27.3-next.22",
       "author": {
         "name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
       },

package/README.md

@@ -383,7 +383,7 @@ The `atdd`, `automate`, `ci`, `framework`, `nfr`, `teach-me-testing`, `test-desi
 | `bmad-teach-me-testing`     | N/A                      | Yes; multi-turn session with seeded wrong quiz answer, review loop, and progress    |
 | `bmad-testarch-atdd`        | 3                        | Yes; one unimplemented fixture and five acceptance criteria                         |
 | `bmad-testarch-automate`    | 5                        | Yes; four hand-authored spec sets against a fixed service and its seeded regression |
-| `bmad-testarch-ci`          | 2                        | Yes; one full request and one minimal request                                       |
+| `bmad-testarch-ci`          | 2                        | Yes; one full request, one minimal request and one request over an evaluation plan  |
 | `bmad-testarch-evaluate`    | N/A                      | Yes; Evaluate-authored, a gap-guide class swap seeded and a second held out         |
 | `bmad-testarch-framework`   | 3                        | Yes; install and smoke-test generated scaffold in network-isolated sandbox          |
 | `bmad-testarch-nfr`         | 2                        | Yes; one evidence bundle with known gaps and one clean bundle                       |
@@ -395,7 +395,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
 
 ### Deterministic Checks
 
-`npm test` chains 106 checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Twelve of the 106 keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
+`npm test` chains 107 checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Twelve of the 107 keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
 
 - `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
 - `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, and the fragment-selection case count. A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
@@ -432,7 +432,7 @@ npm run eval:all -- --agent agy
 npm run eval:all -- --agent agy --agent claude --agent codex
 ```
 
-`eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `ci` project, per `test-design` epic, per `trace` fixture set, and per `atdd` story. One runner makes 107 agent calls: 48 fragment selections, 38 routing intents, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, and 2 ATDD generations. All three built-in runners make 321 calls.
+`eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `ci` project, per `test-design` epic, per `trace` fixture set, and per `atdd` story. One runner makes 109 agent calls: 48 fragment selections, 38 routing intents, 3 reviews, 4 audits, 6 pipelines, 4 test designs, 4 traces, and 2 ATDD generations. All three built-in runners make 327 calls.
 
 Check the data, executable, login, fixtures, and expected results without making a model call:
 
@@ -558,7 +558,7 @@ Every row below is a declared gate. Ten of TEA's eleven skills have behavioral s
 
 | Eval                              | Declared threshold                                                                                                                                                                                                                                                                                                                                                                                 | Declared volume                                                                                            |
 | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
-| `npm run eval:all -- --agent ...` | All eight live suites below meet their thresholds for the selected runner                                                                                                                                                                                                                                                                                                                          | 48 selections, 38 routing calls, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, 2 generations |
+| `npm run eval:all -- --agent ...` | All eight live suites below meet their thresholds for the selected runner                                                                                                                                                                                                                                                                                                                          | 48 selections, 38 routing calls, 3 reviews, 4 audits, 6 pipelines, 4 test designs, 4 traces, 2 generations |
 | Fragment selection                | At least 90% required-fragment recall, at most 10% forbidden-fragment selection, and stable choices across repeated cases                                                                                                                                                                                                                                                                          | 24 cases twice: 48 calls                                                                                   |
 | Test review                       | At least 70% overall recall, 100% CRITICAL recall, an 80% non-false-positive rate, score standard deviation no higher than 3, and a stable verdict                                                                                                                                                                                                                                                 | Three complete reviews                                                                                     |
 | Trace                             | At least 90% criterion-status accuracy and 90% evidence-citation precision; 100% on both discriminating criteria, the gate decision, the gate criteria, the coverage arithmetic, the oracle resolution, the rejected evidence, the waiver validity, and the live evidence; no clean-set false positive, no unstable case, and no fixture mutation                                                  | Two fixture sets twice: 4 calls                                                                            |
@@ -596,7 +596,7 @@ npm ci
 npm run test:eval-data          # fragment-selection corpus, static
 npm run test:eval-trace-data    # trace corpus, static
 npm run test:eval-schemas       # manifest against harness constants, and the preflight argv
-npm run test:eval-replay        # 123 stored outputs against the scorers
+npm run test:eval-replay        # stored outputs against the scorers
 ```
 
 Run live evals in a scheduled or manually triggered CI job after installing and authenticating the selected agent CLI:

package/cli/lib/evaluate/arm.js

@@ -379,6 +379,8 @@ function bodyValue(body) {
  */
 function hostEnvironmentPort({ port, registry }) {
   return {
+    /** Empties the private home the port's sandbox keeps, where it has one (`registry.js` `createProbePort`). */
+    resetHome: () => port.resetHome?.(),
     async probe(request, signal) {
       const registered = request?.kind === 'cli' && registry.targetFor(request.interfaceId, request.executable) !== undefined;
       const injected = registered ? registry.hostEnvironment(request.interfaceId, [], request.executable) : {};
@@ -720,6 +722,9 @@ async function runArm({
   signal,
   seed = 'tea-evaluate-default-seed',
 }) {
+  // An arm is independent of the arms before it on a shared port, so its target starts with an empty private home; the
+  // steps of this arm share it.
+  port.resetHome?.();
   const operations = operationsById(contract);
   const plan = contract.interactionPlan ?? [];
   const declared = new Set(plan.map((step) => step.stepId));

package/cli/lib/evaluate/check.js

@@ -70,6 +70,10 @@
  * - `evaluator` (Story 1.34): a `sealed-brief-agent` evaluator whose `evaluation.json` has no
  *   `evaluatorQualification` (`attempts`, `minimumAgreement`), which `run` needs to qualify the agent
  *   before its verdicts count; and an `evaluatorQualification` beside any other kind, where nothing would use it.
+ * - `evaluator` (Story 1.44, AD-21): a `command` evaluator has no tracked `evaluator/frameworks.json` (its declared
+ *   installed frameworks, an empty list for none), one whose version probe is not a tracked regular executable under
+ *   `evaluator/`, or an `evaluator/LEARNED.md` whose `package@version` records disagree with a declaration; a declaration
+ *   that fails its shape is reported under `schema` (or `json`). `check` runs no probe (`run` does, `frameworks.js`).
  * - `evaluator` (Story 1.17, AD-21): a `command` or `sealed-brief-agent` evaluator has no
  *   `evaluator/mapping.json`, or one binding an oracle, behavior or rubric criterion the contract does not
  *   declare, an oracle to a behavior that does not declare it, levels other than the criterion's anchored
@@ -140,6 +144,7 @@ const {
 } = require('./registry');
 const { AGENT_ADAPTERS, bridgedArgsRefused, resolveModel } = require('../agent-adapters');
 const { EVALUATOR_DIRECTORY, EvaluatorLayerError, evaluatorFiles, evaluatorOf, isKnownEvaluator } = require('./evaluators');
+const { FRAMEWORKS_PATH, LEARNED_PATH, declarationProblems, declaredFrameworks, learnedProblems } = require('./frameworks');
 const { answeredKind, degenerateResponsePath } = require('./gameability');
 
 /** How a finding names a call of each interface kind. */
@@ -1239,6 +1244,82 @@ function checkRecordsCalibration(report, folder, evaluator, evaluation, contract
 /** The mode bits that let anyone execute a file. */
 const EXECUTE_BITS = 0o111;
 
+/**
+ * A `command` evaluator's declared framework dependencies (Story 1.44):
+ * `evaluator/frameworks.json` must be there, tracked, and meet its shape (an
+ * empty list for an evaluator with no installed framework), every version
+ * probe must be a tracked regular executable of the layer, and
+ * `evaluator/LEARNED.md` must record the version each nonempty declaration
+ * names. `run` observes what is installed; `check` runs no probe.
+ */
+function checkFrameworks(report, folder, layer, untracked) {
+  if (untracked(FRAMEWORKS_PATH)) {
+    report.add(
+      FRAMEWORKS_PATH,
+      'evaluator',
+      `${FRAMEWORKS_PATH} is not tracked by git, and a run reads only the files git tracks under evaluator/; git add it`,
+    );
+    return;
+  }
+  if (!fs.existsSync(path.join(folder, ...FRAMEWORKS_PATH.split('/')))) {
+    report.add(
+      FRAMEWORKS_PATH,
+      'evaluator',
+      `evaluation.json's evaluator is command, which declares the installed frameworks it depends on in ${FRAMEWORKS_PATH}, and the folder has none; declare each dependency, or an empty list for an evaluator with none`,
+    );
+    return;
+  }
+  const declaration = parseInto(report, folder, FRAMEWORKS_PATH);
+  if (declaration === undefined) return;
+  const shape = declarationProblems(declaration);
+  for (const problem of shape) report.add(FRAMEWORKS_PATH, 'schema', problem);
+  if (shape.length > 0) return;
+  const frameworks = declaredFrameworks(declaration);
+  for (const { package: name, probe } of frameworks) {
+    if (untracked(probe.command)) {
+      report.add(
+        FRAMEWORKS_PATH,
+        'evaluator',
+        `the version probe of ${name} names ${probe.command}, which git does not track, and a run reads only the files git tracks under evaluator/; git add it`,
+      );
+      continue;
+    }
+    let stats;
+    try {
+      stats = fs.lstatSync(path.join(folder, ...probe.command.split('/')));
+    } catch {
+      stats = null;
+    }
+    if (stats === null || !stats.isFile()) {
+      report.add(
+        FRAMEWORKS_PATH,
+        'evaluator',
+        `the version probe of ${name} names ${probe.command}, which is not a regular file the evaluation folder holds`,
+      );
+    } else if (process.platform !== 'win32' && (stats.mode & EXECUTE_BITS) === 0) {
+      report.add(
+        FRAMEWORKS_PATH,
+        'evaluator',
+        `the version probe of ${name} names ${probe.command}, which is not executable; set its execute bit`,
+      );
+    }
+  }
+  // Where the layer could not be read, the layer's own finding already says so.
+  if (layer === null) return;
+  if (untracked(LEARNED_PATH)) {
+    report.add(
+      LEARNED_PATH,
+      'evaluator',
+      `${LEARNED_PATH} is not tracked by git, and a run reads only the files git tracks under evaluator/; git add it`,
+    );
+    return;
+  }
+  const learned = layer.files.find((file) => file.path === LEARNED_PATH);
+  for (const problem of learnedProblems(frameworks, learned === undefined ? null : learned.bytes.toString('utf8'))) {
+    report.add(LEARNED_PATH, 'evaluator', problem);
+  }
+}
+
 /**
  * `evaluation.json`'s `evaluator` (AD-21) held to the folder and the
  * contract: a `command` or `sealed-brief-agent` evaluator needs
@@ -1339,6 +1420,7 @@ function checkEvaluator(report, folder, evaluation, contract, conditions, engine
       `evaluation.json's evaluator is ${kind}, whose judgment rows convert through ${MAPPING_PATH}, and the folder has none`,
     );
   }
+  if (kind === 'command') checkFrameworks(report, folder, layer, untracked);
   if (kind === 'command') {
     if (typeof evaluator.command !== 'string') return;
     if (untracked(evaluator.command)) {

package/cli/lib/evaluate/ci.js

@@ -5,7 +5,7 @@
  * The plan (`ci-plan.js`) is the only definition of tier membership. The runtime reads it, validates its placement
  * rules (exit 10 on a finding, 64 when the plan is absent) and runs exactly the checks whose `placement.tier` is the
  * tier asked for, in plan order. Every check runs, whether or not an earlier one failed, so the evidence bundle is
- * complete. An `evaluate` check is run by its id; its `command` is the `tea-evaluate` argv a pipeline step renders. A
+ * complete. An `evaluate` check is run by its id; its `command` records the `tea-evaluate` argv a reader can run by hand, and a pipeline runs `tea-evaluate ci --tier <tier>` once per tier. A
  * `gate` check is an `eval-quality-gates` command the adopter adopted, run as a child process with no shell and the
  * plan's argv, in the evaluation folder, as the leader of a process group of its own: it ends at the plan check's
  * `timeoutMs` or past 64 MiB of output (exit 12, what it printed kept), and a signal to `ci` reaches the group first.

package/cli/lib/evaluate/command-evaluator.js

@@ -27,6 +27,11 @@
  * (`confinement.js` `layerPrefix`), so neither it nor any process it starts
  * can write under `evaluator/` (Story 1.31).
  *
+ * The frameworks it depends on are declared in `evaluator/frameworks.json` and read through the same launch path
+ * (`observeFrameworks`, Story 1.44): each declared version probe starts as the evaluator does, with the same
+ * environment, an empty private working directory of its own and the same confinement, and prints the installed
+ * package identity and version (`frameworks.js`).
+ *
  * Its stdout is read as UTF-8 (a byte sequence that is not UTF-8 reads as
  * U+FFFD), and what it printed is kept as the bytes it wrote. One that cannot
  * start, is still running at its wall clock, exits other than 0, or prints
@@ -39,9 +44,81 @@
 const path = require('node:path');
 
 const { buildMinimalEnv, runSupervised } = require('../run-agent');
+const { readProbeAnswer, stderrNote } = require('./frameworks');
 const { EvaluatorError, readAnswer } = require('./judgment-rows');
 const { releaseScratchDirectory, makeScratchDirectory } = require('./workspace');
 
+/**
+ * Starts one executable under the evaluation folder's `evaluator/` exactly as
+ * the evaluator itself starts: through `spawnPrefix` when the run confines it,
+ * under the supervisor, with an empty private working directory in `scratch`
+ * removed afterwards, the base environment plus `evaluator.environmentKeys`,
+ * and `evaluator.timeoutMs` as its wall clock. The evaluator's launch and a
+ * framework's version probe (`observeFrameworks`) both go through it, so a
+ * probe sees what the evaluator sees.
+ *
+ * @returns {Promise<object>} `runSupervised`'s report: the outcome and the streams
+ */
+async function launchExecutable({ folder, evaluator, command, args, input, scratch, env, spawnPrefix }) {
+  const executable = path.join(folder, ...command.split('/'));
+  const cwd = makeScratchDirectory(scratch, 'tea-evaluate-command-');
+  try {
+    return await runSupervised({
+      command: spawnPrefix.length === 0 ? executable : spawnPrefix[0],
+      args: [...spawnPrefix.slice(1), ...(spawnPrefix.length === 0 ? [] : [executable]), ...args],
+      input,
+      cwd,
+      env: buildMinimalEnv(evaluator.environmentKeys ?? [], env),
+      timeout: evaluator.timeoutMs,
+    });
+  } finally {
+    releaseScratchDirectory(scratch, cwd);
+  }
+}
+
+/**
+ * Reads the installed version of each declared framework (Story 1.44): runs
+ * the dependency's version probe through the evaluator's own launch path
+ * (`launchExecutable`) and reads the `{ package, version }` it prints
+ * (`frameworks.js`). A probe that cannot start, outlives the timeout, exits
+ * other than 0 or prints another shape yields no observation for that
+ * dependency and a `fault` naming why, with what it printed. Nothing is
+ * thrown for a dependency that is missing: the caller compares the entries
+ * with the declaration.
+ *
+ * @param {object} options
+ * @param {string} options.folder the evaluation folder
+ * @param {object} options.evaluator `evaluation.json`'s `command` evaluator
+ * @param {Array<{ package: string, probe: { command: string, args: string[] } }>} options.frameworks
+ * @param {string[]} options.scratch
+ * @param {NodeJS.ProcessEnv} [options.env]
+ * @param {string[]} [options.spawnPrefix]
+ * @returns {Promise<Array<{ package: string, observed: { package: string, version: string }|null, fault: string|null, stdout: string, stderr: string }>>}
+ *   one entry per declared framework, in the order given
+ */
+async function observeFrameworks({ folder, evaluator, frameworks, scratch, env = process.env, spawnPrefix = [] }) {
+  const entries = [];
+  for (const framework of frameworks) {
+    const { command, args } = framework.probe;
+    const label = `the version probe of ${framework.package} (${command})`;
+    const ended = await launchExecutable({ folder, evaluator, command, args, input: '', scratch, env, spawnPrefix });
+    const { outcome, stdout, stderr } = ended;
+    const entry = { package: framework.package, observed: null, fault: null, stdout, stderr };
+    if (outcome.spawnError) entry.fault = `${label} could not start: ${outcome.spawnError.message}`;
+    else if (outcome.timedOut) entry.fault = `${label} was still running at the evaluator's ${evaluator.timeoutMs}ms timeout`;
+    else if (outcome.failure) entry.fault = `${label} did not finish: ${outcome.failure}`;
+    else if (outcome.status === 0) {
+      const answer = readProbeAnswer(framework.package, stdout);
+      if ('fault' in answer) entry.fault = `${label}: ${answer.fault}`;
+      else entry.observed = answer;
+    } else {
+      entry.fault = `${label} ${outcome.signal ? `was killed by signal ${outcome.signal}` : `exited ${outcome.status}`}${stderrNote(stderr)}`;
+    }
+    entries.push(entry);
+  }
+  return entries;
+}
+
 /**
  * Runs the evaluator over one trial.
  *
@@ -70,21 +147,16 @@ async function runCommandEvaluator({
   env = process.env,
   spawnPrefix = [],
 }) {
-  const executable = path.join(folder, ...evaluator.command.split('/'));
-  const cwd = makeScratchDirectory(scratch, 'tea-evaluate-command-');
-  let ended;
-  try {
-    ended = await runSupervised({
-      command: spawnPrefix.length === 0 ? executable : spawnPrefix[0],
-      args: [...spawnPrefix.slice(1), ...(spawnPrefix.length === 0 ? [] : [executable]), ...(evaluator.args ?? [])],
-      input: `${JSON.stringify({ sealedBrief, observations })}\n`,
-      cwd,
-      env: buildMinimalEnv(evaluator.environmentKeys ?? [], env),
-      timeout: evaluator.timeoutMs,
-    });
-  } finally {
-    releaseScratchDirectory(scratch, cwd);
-  }
+  const ended = await launchExecutable({
+    folder,
+    evaluator,
+    command: evaluator.command,
+    args: evaluator.args ?? [],
+    input: `${JSON.stringify({ sealedBrief, observations })}\n`,
+    scratch,
+    env,
+    spawnPrefix,
+  });
   const { outcome, stdout, stderr, stdoutBytes, stderrBytes } = ended;
   const streams = { stdout, stderr, stdoutBytes, stderrBytes };
   const fail = (message) => Object.assign(new EvaluatorError(message, streams), { outcome });
@@ -108,4 +180,4 @@ async function runCommandEvaluator({
   }
 }
 
-module.exports = { runCommandEvaluator };
+module.exports = { observeFrameworks, runCommandEvaluator };