bmad-method-test-architecture-enterprise 1.27.3-next.21 → 1.27.3-next.23

package/.claude-plugin/marketplace.json

@@ -31,7 +31,7 @@
       "name": "bmad-method-test-architecture-enterprise",
       "source": "./",
       "description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
-      "version": "1.27.3-next.21",
+      "version": "1.27.3-next.23",
       "author": {
         "name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
       },

package/README.md

@@ -383,7 +383,7 @@ The `atdd`, `automate`, `ci`, `framework`, `nfr`, `teach-me-testing`, `test-desi
 | `bmad-teach-me-testing`     | N/A                      | Yes; multi-turn session with seeded wrong quiz answer, review loop, and progress    |
 | `bmad-testarch-atdd`        | 3                        | Yes; one unimplemented fixture and five acceptance criteria                         |
 | `bmad-testarch-automate`    | 5                        | Yes; four hand-authored spec sets against a fixed service and its seeded regression |
-| `bmad-testarch-ci`          | 2                        | Yes; one full request and one minimal request                                       |
+| `bmad-testarch-ci`          | 2                        | Yes; one full request, one minimal request and one request over an evaluation plan  |
 | `bmad-testarch-evaluate`    | N/A                      | Yes; Evaluate-authored, a gap-guide class swap seeded and a second held out         |
 | `bmad-testarch-framework`   | 3                        | Yes; install and smoke-test generated scaffold in network-isolated sandbox          |
 | `bmad-testarch-nfr`         | 2                        | Yes; one evidence bundle with known gaps and one clean bundle                       |
@@ -395,7 +395,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
 
 ### Deterministic Checks
 
-`npm test` chains 106 checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Twelve of the 106 keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
+`npm test` chains 107 checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Twelve of the 107 keep the rules, guidance, hook, eval data, eval contracts, diagnostics, and documentation aligned:
 
 - `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
 - `test:doc-counts` runs `eval-quality-gates doc-counts`, which holds a hand-written count on a published page against the source that computes it: the roadmap's per-suite `eval:all` call counts, the knowledge-fragment tier breakdown, this section's own npm-test-chain length, and the fragment-selection case count. A pattern matching no sentence, or more than one, fails the same way a wrong number does, so the entry cannot go stale by drifting out from under its own pattern either.
@@ -432,7 +432,7 @@ npm run eval:all -- --agent agy
 npm run eval:all -- --agent agy --agent claude --agent codex
 ```
 
-`eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `ci` project, per `test-design` epic, per `trace` fixture set, and per `atdd` story. One runner makes 107 agent calls: 48 fragment selections, 38 routing intents, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, and 2 ATDD generations. All three built-in runners make 321 calls.
+`eval:all` uses two repetitions per fragment-selection case and per routing intent, three repetitions for `test-review`, and two repetitions per `nfr` evidence bundle, per `ci` project, per `test-design` epic, per `trace` fixture set, and per `atdd` story. One runner makes 109 agent calls: 48 fragment selections, 38 routing intents, 3 reviews, 4 audits, 6 pipelines, 4 test designs, 4 traces, and 2 ATDD generations. All three built-in runners make 327 calls.
 
 Check the data, executable, login, fixtures, and expected results without making a model call:
 
@@ -558,7 +558,7 @@ Every row below is a declared gate. Ten of TEA's eleven skills have behavioral s
 
 | Eval                              | Declared threshold                                                                                                                                                                                                                                                                                                                                                                                 | Declared volume                                                                                            |
 | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- |
-| `npm run eval:all -- --agent ...` | All eight live suites below meet their thresholds for the selected runner                                                                                                                                                                                                                                                                                                                          | 48 selections, 38 routing calls, 3 reviews, 4 audits, 4 pipelines, 4 test designs, 4 traces, 2 generations |
+| `npm run eval:all -- --agent ...` | All eight live suites below meet their thresholds for the selected runner                                                                                                                                                                                                                                                                                                                          | 48 selections, 38 routing calls, 3 reviews, 4 audits, 6 pipelines, 4 test designs, 4 traces, 2 generations |
 | Fragment selection                | At least 90% required-fragment recall, at most 10% forbidden-fragment selection, and stable choices across repeated cases                                                                                                                                                                                                                                                                          | 24 cases twice: 48 calls                                                                                   |
 | Test review                       | At least 70% overall recall, 100% CRITICAL recall, an 80% non-false-positive rate, score standard deviation no higher than 3, and a stable verdict                                                                                                                                                                                                                                                 | Three complete reviews                                                                                     |
 | Trace                             | At least 90% criterion-status accuracy and 90% evidence-citation precision; 100% on both discriminating criteria, the gate decision, the gate criteria, the coverage arithmetic, the oracle resolution, the rejected evidence, the waiver validity, and the live evidence; no clean-set false positive, no unstable case, and no fixture mutation                                                  | Two fixture sets twice: 4 calls                                                                            |
@@ -596,7 +596,7 @@ npm ci
 npm run test:eval-data          # fragment-selection corpus, static
 npm run test:eval-trace-data    # trace corpus, static
 npm run test:eval-schemas       # manifest against harness constants, and the preflight argv
-npm run test:eval-replay        # 123 stored outputs against the scorers
+npm run test:eval-replay        # stored outputs against the scorers
 ```
 
 Run live evals in a scheduled or manually triggered CI job after installing and authenticating the selected agent CLI:

package/cli/evaluate.js

@@ -4,7 +4,12 @@
  *
  * Subcommands in this release:
  *   tea-evaluate check  --evaluation <path>   validate the folder; exit 10 on any authoring defect
- *   tea-evaluate digest --evaluation <path>   write corpus-index.json and print corpusDigest
+ *   tea-evaluate digest --evaluation <path> [--calibration-inputs]
+ *                                             write corpus-index.json and print corpusDigest; with
+ *                                             --calibration-inputs write nothing and print, as JSON, the values a
+ *                                             records harness copies into calibration-judgments.json: the labelled
+ *                                             file's digest, the scorer configuration digest and each labelled
+ *                                             item's label-free scorerInput
  *   tea-evaluate preflight --evaluation <path> [--from-working-tree]
  *                                             qualify the seeded probes in a disposable workspace, drive the
  *                                             preflight legs and take the verdict from eval-quality
@@ -38,7 +43,9 @@
  *   2-5 also ci: the exit of each stage it runs (compile, seal, the replay's preflight and score, a live check's
  *       preflight, run and score), passed through the same way; 2 is also a probe class below its strength floor
  *       on the release tier
- *   10  authoring defect: every finding is printed, one per line (digest: an indexed entry it cannot digest;
+ *   10  authoring defect: every finding is printed, one per line (digest: an indexed entry it cannot digest, and with
+ *       --calibration-inputs an evaluator that is not records, a contract with no rubric, an unusable labelled file
+ *       or a harness configuration that is absent, a link, not JSON or not an EvaluatorConfiguration;
  *       preflight: a leg the registry does not authorize, or a mutation whose find text does not occur
  *       exactly once)
  *   10  also compare: a run whose scores or members cannot be read as files the run wrote, a baseline holding a link
@@ -103,6 +110,7 @@ const { runScoreCommand } = require('./lib/evaluate/score');
 const { runCiCommand } = require('./lib/evaluate/ci');
 const { TIERS } = require('./lib/evaluate/ci-plan');
 const { escapeUnprintable, findingLine } = require('./lib/evaluate/finding-lines');
+const { calibrationInputsOf } = require('./lib/evaluate/records-calibration');
 
 const EXIT_CODES = {
   ok: 0,
@@ -135,8 +143,21 @@ async function runCheck(options) {
   return EXIT_CODES.authoring;
 }
 
+/** Prints the values a records harness copies into its judgments file; writes nothing under the folder. */
+async function runCalibrationInputs(folder) {
+  const { problems, inputs } = await calibrationInputsOf(folder);
+  if (inputs === null) {
+    for (const problem of problems) process.stdout.write(findingLine(problem.file, 'judge-calibration', problem.message));
+    process.stderr.write(`${NAME} digest --calibration-inputs: ${problems.length} authoring defect(s) in ${folder}\n`);
+    return EXIT_CODES.authoring;
+  }
+  process.stdout.write(`${JSON.stringify(inputs, null, 2)}\n`);
+  return EXIT_CODES.ok;
+}
+
 async function runDigest(options) {
   const folder = folderFrom(options);
+  if (options.calibrationInputs === true) return runCalibrationInputs(folder);
   const { index, corpusDigest, indexPath } = await writeCorpusIndex(folder);
   process.stderr.write(`${NAME} digest: wrote ${indexPath} (${index.length} file(s))\n`);
   process.stdout.write(`${corpusDigest}\n`);
@@ -220,8 +241,14 @@ function buildProgram(run) {
     .action((options) => run(runCheck, options));
   program
     .command('digest')
-    .description('Write corpus-index.json over corpus/, probes/ and mutations/, and print corpusDigest.')
+    .description(
+      'Write corpus-index.json over corpus/, probes/ and mutations/, and print corpusDigest; with --calibration-inputs, print the values a records harness copies instead.',
+    )
     .option('--evaluation <path>', 'the evaluation folder, or its evaluation.json')
+    .option(
+      '--calibration-inputs',
+      "print, and write nothing: the labelled file's digest, the scorer configuration digest and each item's scorerInput",
+    )
     .action((options) => run(runDigest, options));
   program
     .command('preflight')

package/cli/lib/evaluate/arm.js

@@ -379,6 +379,8 @@ function bodyValue(body) {
  */
 function hostEnvironmentPort({ port, registry }) {
   return {
+    /** Empties the private home the port's sandbox keeps, where it has one (`registry.js` `createProbePort`). */
+    resetHome: () => port.resetHome?.(),
     async probe(request, signal) {
       const registered = request?.kind === 'cli' && registry.targetFor(request.interfaceId, request.executable) !== undefined;
       const injected = registered ? registry.hostEnvironment(request.interfaceId, [], request.executable) : {};
@@ -720,6 +722,9 @@ async function runArm({
   signal,
   seed = 'tea-evaluate-default-seed',
 }) {
+  // An arm is independent of the arms before it on a shared port, so its target starts with an empty private home; the
+  // steps of this arm share it.
+  port.resetHome?.();
   const operations = operationsById(contract);
   const plan = contract.interactionPlan ?? [];
   const declared = new Set(plan.map((step) => step.stepId));

package/cli/lib/evaluate/calibration.js

@@ -203,6 +203,11 @@ function readCalibration(folder) {
   }
 }
 
+/** The digest of the labelled file's bytes: what a configuration binds as `tea.judgeCalibrationDigest` and a run records. */
+function labelledDigest(labelled, engine) {
+  return engine.digestBytes(labelled.bytes);
+}
+
 /**
  * The criteria of a calibration report whose agreement is below its `minimumAgreement`, each as a sentence; empty when
  * every criterion meets it. `run` stops with exit 11 on any, and `tea-evaluate ci` reads the same report with this.
@@ -258,7 +263,7 @@ async function runCalibration({ calibration, evaluation, contract, engine, write
       exitCode: 11,
       message: `judge calibration agreement fell below ${report.minimumAgreement}; see judge-calibration.json`,
     });
-  return { digest: engine.digestBytes(calibration.bytes), report };
+  return { digest: labelledDigest(calibration, engine), report };
 }
 
 module.exports = {
@@ -267,6 +272,7 @@ module.exports = {
   calibrationOperationId,
   calibrationProblems,
   calibrationShortfalls,
+  labelledDigest,
   readCalibration,
   runCalibration,
 };

package/cli/lib/evaluate/check.js

@@ -70,6 +70,10 @@
  * - `evaluator` (Story 1.34): a `sealed-brief-agent` evaluator whose `evaluation.json` has no
  *   `evaluatorQualification` (`attempts`, `minimumAgreement`), which `run` needs to qualify the agent
  *   before its verdicts count; and an `evaluatorQualification` beside any other kind, where nothing would use it.
+ * - `evaluator` (Story 1.44, AD-21): a `command` evaluator has no tracked `evaluator/frameworks.json` (its declared
+ *   installed frameworks, an empty list for none), one whose version probe is not a tracked regular executable under
+ *   `evaluator/`, or an `evaluator/LEARNED.md` whose `package@version` records disagree with a declaration; a declaration
+ *   that fails its shape is reported under `schema` (or `json`). `check` runs no probe (`run` does, `frameworks.js`).
  * - `evaluator` (Story 1.17, AD-21): a `command` or `sealed-brief-agent` evaluator has no
  *   `evaluator/mapping.json`, or one binding an oracle, behavior or rubric criterion the contract does not
  *   declare, an oracle to a behavior that does not declare it, levels other than the criterion's anchored
@@ -123,7 +127,7 @@ const AjvModule = require('ajv/dist/2020');
 
 const { engineSchemaPath, loadEngine, schemaVersionProblems } = require('./engine');
 const { CALIBRATION_PATH, calibrationProblems, readCalibration } = require('./calibration');
-const { verifyRecordsCalibration } = require('./records-calibration');
+const { readConfiguration, recordsDirectory, verifyRecordsCalibration } = require('./records-calibration');
 const { readPlan } = require('./ci-plan');
 const { MANIFEST_NAME } = require('./folder');
 const { addFormats } = require('./formats');
@@ -140,6 +144,7 @@ const {
 } = require('./registry');
 const { AGENT_ADAPTERS, bridgedArgsRefused, resolveModel } = require('../agent-adapters');
 const { EVALUATOR_DIRECTORY, EvaluatorLayerError, evaluatorFiles, evaluatorOf, isKnownEvaluator } = require('./evaluators');
+const { FRAMEWORKS_PATH, LEARNED_PATH, declarationProblems, declaredFrameworks, learnedProblems } = require('./frameworks');
 const { answeredKind, degenerateResponsePath } = require('./gameability');
 
 /** How a finding names a call of each interface kind. */
@@ -1213,9 +1218,7 @@ function checkRecordsCalibration(report, folder, evaluator, evaluation, contract
   if (calibrationProblems(evaluation, contract, labelled?.value, engine).length > 0) return;
   let configuration;
   try {
-    const stats = fs.lstatSync(path.join(root, 'evaluator-configuration.json'));
-    if (!stats.isFile()) throw new Error('it is not a regular file');
-    configuration = JSON.parse(fs.readFileSync(path.join(root, 'evaluator-configuration.json'), 'utf8'));
+    configuration = readConfiguration(root);
   } catch (error) {
     report.add(
       configurationFile,
@@ -1239,6 +1242,82 @@ function checkRecordsCalibration(report, folder, evaluator, evaluation, contract
 /** The mode bits that let anyone execute a file. */
 const EXECUTE_BITS = 0o111;
 
+/**
+ * A `command` evaluator's declared framework dependencies (Story 1.44):
+ * `evaluator/frameworks.json` must be there, tracked, and meet its shape (an
+ * empty list for an evaluator with no installed framework), every version
+ * probe must be a tracked regular executable of the layer, and
+ * `evaluator/LEARNED.md` must record the version each nonempty declaration
+ * names. `run` observes what is installed; `check` runs no probe.
+ */
+function checkFrameworks(report, folder, layer, untracked) {
+  if (untracked(FRAMEWORKS_PATH)) {
+    report.add(
+      FRAMEWORKS_PATH,
+      'evaluator',
+      `${FRAMEWORKS_PATH} is not tracked by git, and a run reads only the files git tracks under evaluator/; git add it`,
+    );
+    return;
+  }
+  if (!fs.existsSync(path.join(folder, ...FRAMEWORKS_PATH.split('/')))) {
+    report.add(
+      FRAMEWORKS_PATH,
+      'evaluator',
+      `evaluation.json's evaluator is command, which declares the installed frameworks it depends on in ${FRAMEWORKS_PATH}, and the folder has none; declare each dependency, or an empty list for an evaluator with none`,
+    );
+    return;
+  }
+  const declaration = parseInto(report, folder, FRAMEWORKS_PATH);
+  if (declaration === undefined) return;
+  const shape = declarationProblems(declaration);
+  for (const problem of shape) report.add(FRAMEWORKS_PATH, 'schema', problem);
+  if (shape.length > 0) return;
+  const frameworks = declaredFrameworks(declaration);
+  for (const { package: name, probe } of frameworks) {
+    if (untracked(probe.command)) {
+      report.add(
+        FRAMEWORKS_PATH,
+        'evaluator',
+        `the version probe of ${name} names ${probe.command}, which git does not track, and a run reads only the files git tracks under evaluator/; git add it`,
+      );
+      continue;
+    }
+    let stats;
+    try {
+      stats = fs.lstatSync(path.join(folder, ...probe.command.split('/')));
+    } catch {
+      stats = null;
+    }
+    if (stats === null || !stats.isFile()) {
+      report.add(
+        FRAMEWORKS_PATH,
+        'evaluator',
+        `the version probe of ${name} names ${probe.command}, which is not a regular file the evaluation folder holds`,
+      );
+    } else if (process.platform !== 'win32' && (stats.mode & EXECUTE_BITS) === 0) {
+      report.add(
+        FRAMEWORKS_PATH,
+        'evaluator',
+        `the version probe of ${name} names ${probe.command}, which is not executable; set its execute bit`,
+      );
+    }
+  }
+  // Where the layer could not be read, the layer's own finding already says so.
+  if (layer === null) return;
+  if (untracked(LEARNED_PATH)) {
+    report.add(
+      LEARNED_PATH,
+      'evaluator',
+      `${LEARNED_PATH} is not tracked by git, and a run reads only the files git tracks under evaluator/; git add it`,
+    );
+    return;
+  }
+  const learned = layer.files.find((file) => file.path === LEARNED_PATH);
+  for (const problem of learnedProblems(frameworks, learned === undefined ? null : learned.bytes.toString('utf8'))) {
+    report.add(LEARNED_PATH, 'evaluator', problem);
+  }
+}
+
 /**
  * `evaluation.json`'s `evaluator` (AD-21) held to the folder and the
  * contract: a `command` or `sealed-brief-agent` evaluator needs
@@ -1287,14 +1366,8 @@ function checkEvaluator(report, folder, evaluation, contract, conditions, engine
   if (kind === 'records') {
     if (typeof evaluator.records !== 'string') return;
     // Only a directory inside the folder, reached through no link, is the folder's own.
-    const spelled = path.join(fs.realpathSync(folder), ...evaluator.records.split('/'));
-    let real;
-    try {
-      real = fs.realpathSync(path.join(folder, ...evaluator.records.split('/')));
-    } catch {
-      real = null;
-    }
-    if (real !== spelled || !fs.statSync(real).isDirectory()) {
+    const real = recordsDirectory(folder, evaluator.records);
+    if (real === null) {
       report.add(
         MANIFEST_NAME,
         'evaluator',
@@ -1339,6 +1412,7 @@ function checkEvaluator(report, folder, evaluation, contract, conditions, engine
       `evaluation.json's evaluator is ${kind}, whose judgment rows convert through ${MAPPING_PATH}, and the folder has none`,
     );
   }
+  if (kind === 'command') checkFrameworks(report, folder, layer, untracked);
   if (kind === 'command') {
     if (typeof evaluator.command !== 'string') return;
     if (untracked(evaluator.command)) {

package/cli/lib/evaluate/ci.js

@@ -5,7 +5,7 @@
  * The plan (`ci-plan.js`) is the only definition of tier membership. The runtime reads it, validates its placement
  * rules (exit 10 on a finding, 64 when the plan is absent) and runs exactly the checks whose `placement.tier` is the
  * tier asked for, in plan order. Every check runs, whether or not an earlier one failed, so the evidence bundle is
- * complete. An `evaluate` check is run by its id; its `command` is the `tea-evaluate` argv a pipeline step renders. A
+ * complete. An `evaluate` check is run by its id; its `command` records the `tea-evaluate` argv a reader can run by hand, and a pipeline runs `tea-evaluate ci --tier <tier>` once per tier. A
  * `gate` check is an `eval-quality-gates` command the adopter adopted, run as a child process with no shell and the
  * plan's argv, in the evaluation folder, as the leader of a process group of its own: it ends at the plan check's
  * `timeoutMs` or past 64 MiB of output (exit 12, what it printed kept), and a signal to `ci` reaches the group first.

package/cli/lib/evaluate/command-evaluator.js

@@ -27,6 +27,11 @@
  * (`confinement.js` `layerPrefix`), so neither it nor any process it starts
  * can write under `evaluator/` (Story 1.31).
  *
+ * The frameworks it depends on are declared in `evaluator/frameworks.json` and read through the same launch path
+ * (`observeFrameworks`, Story 1.44): each declared version probe starts as the evaluator does, with the same
+ * environment, an empty private working directory of its own and the same confinement, and prints the installed
+ * package identity and version (`frameworks.js`).
+ *
  * Its stdout is read as UTF-8 (a byte sequence that is not UTF-8 reads as
  * U+FFFD), and what it printed is kept as the bytes it wrote. One that cannot
  * start, is still running at its wall clock, exits other than 0, or prints
@@ -39,9 +44,81 @@
 const path = require('node:path');
 
 const { buildMinimalEnv, runSupervised } = require('../run-agent');
+const { readProbeAnswer, stderrNote } = require('./frameworks');
 const { EvaluatorError, readAnswer } = require('./judgment-rows');
 const { releaseScratchDirectory, makeScratchDirectory } = require('./workspace');
 
+/**
+ * Starts one executable under the evaluation folder's `evaluator/` exactly as
+ * the evaluator itself starts: through `spawnPrefix` when the run confines it,
+ * under the supervisor, with an empty private working directory in `scratch`
+ * removed afterwards, the base environment plus `evaluator.environmentKeys`,
+ * and `evaluator.timeoutMs` as its wall clock. The evaluator's launch and a
+ * framework's version probe (`observeFrameworks`) both go through it, so a
+ * probe sees what the evaluator sees.
+ *
+ * @returns {Promise<object>} `runSupervised`'s report: the outcome and the streams
+ */
+async function launchExecutable({ folder, evaluator, command, args, input, scratch, env, spawnPrefix }) {
+  const executable = path.join(folder, ...command.split('/'));
+  const cwd = makeScratchDirectory(scratch, 'tea-evaluate-command-');
+  try {
+    return await runSupervised({
+      command: spawnPrefix.length === 0 ? executable : spawnPrefix[0],
+      args: [...spawnPrefix.slice(1), ...(spawnPrefix.length === 0 ? [] : [executable]), ...args],
+      input,
+      cwd,
+      env: buildMinimalEnv(evaluator.environmentKeys ?? [], env),
+      timeout: evaluator.timeoutMs,
+    });
+  } finally {
+    releaseScratchDirectory(scratch, cwd);
+  }
+}
+
+/**
+ * Reads the installed version of each declared framework (Story 1.44): runs
+ * the dependency's version probe through the evaluator's own launch path
+ * (`launchExecutable`) and reads the `{ package, version }` it prints
+ * (`frameworks.js`). A probe that cannot start, outlives the timeout, exits
+ * other than 0 or prints another shape yields no observation for that
+ * dependency and a `fault` naming why, with what it printed. Nothing is
+ * thrown for a dependency that is missing: the caller compares the entries
+ * with the declaration.
+ *
+ * @param {object} options
+ * @param {string} options.folder the evaluation folder
+ * @param {object} options.evaluator `evaluation.json`'s `command` evaluator
+ * @param {Array<{ package: string, probe: { command: string, args: string[] } }>} options.frameworks
+ * @param {string[]} options.scratch
+ * @param {NodeJS.ProcessEnv} [options.env]
+ * @param {string[]} [options.spawnPrefix]
+ * @returns {Promise<Array<{ package: string, observed: { package: string, version: string }|null, fault: string|null, stdout: string, stderr: string }>>}
+ *   one entry per declared framework, in the order given
+ */
+async function observeFrameworks({ folder, evaluator, frameworks, scratch, env = process.env, spawnPrefix = [] }) {
+  const entries = [];
+  for (const framework of frameworks) {
+    const { command, args } = framework.probe;
+    const label = `the version probe of ${framework.package} (${command})`;
+    const ended = await launchExecutable({ folder, evaluator, command, args, input: '', scratch, env, spawnPrefix });
+    const { outcome, stdout, stderr } = ended;
+    const entry = { package: framework.package, observed: null, fault: null, stdout, stderr };
+    if (outcome.spawnError) entry.fault = `${label} could not start: ${outcome.spawnError.message}`;
+    else if (outcome.timedOut) entry.fault = `${label} was still running at the evaluator's ${evaluator.timeoutMs}ms timeout`;
+    else if (outcome.failure) entry.fault = `${label} did not finish: ${outcome.failure}`;
+    else if (outcome.status === 0) {
+      const answer = readProbeAnswer(framework.package, stdout);
+      if ('fault' in answer) entry.fault = `${label}: ${answer.fault}`;
+      else entry.observed = answer;
+    } else {
+      entry.fault = `${label} ${outcome.signal ? `was killed by signal ${outcome.signal}` : `exited ${outcome.status}`}${stderrNote(stderr)}`;
+    }
+    entries.push(entry);
+  }
+  return entries;
+}
+
 /**
  * Runs the evaluator over one trial.
  *
@@ -70,21 +147,16 @@ async function runCommandEvaluator({
   env = process.env,
   spawnPrefix = [],
 }) {
-  const executable = path.join(folder, ...evaluator.command.split('/'));
-  const cwd = makeScratchDirectory(scratch, 'tea-evaluate-command-');
-  let ended;
-  try {
-    ended = await runSupervised({
-      command: spawnPrefix.length === 0 ? executable : spawnPrefix[0],
-      args: [...spawnPrefix.slice(1), ...(spawnPrefix.length === 0 ? [] : [executable]), ...(evaluator.args ?? [])],
-      input: `${JSON.stringify({ sealedBrief, observations })}\n`,
-      cwd,
-      env: buildMinimalEnv(evaluator.environmentKeys ?? [], env),
-      timeout: evaluator.timeoutMs,
-    });
-  } finally {
-    releaseScratchDirectory(scratch, cwd);
-  }
+  const ended = await launchExecutable({
+    folder,
+    evaluator,
+    command: evaluator.command,
+    args: evaluator.args ?? [],
+    input: `${JSON.stringify({ sealedBrief, observations })}\n`,
+    scratch,
+    env,
+    spawnPrefix,
+  });
   const { outcome, stdout, stderr, stdoutBytes, stderrBytes } = ended;
   const streams = { stdout, stderr, stdoutBytes, stderrBytes };
   const fail = (message) => Object.assign(new EvaluatorError(message, streams), { outcome });
@@ -108,4 +180,4 @@ async function runCommandEvaluator({
   }
 }
 
-module.exports = { runCommandEvaluator };
+module.exports = { observeFrameworks, runCommandEvaluator };