bmad-method-test-architecture-enterprise 1.25.2 → 1.26.0

package/.claude-plugin/marketplace.json

@@ -31,7 +31,7 @@
       "name": "bmad-method-test-architecture-enterprise",
       "source": "./",
       "description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
-      "version": "1.25.2",
+      "version": "1.26.0",
       "author": {
         "name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
       },

package/CHANGELOG.md

@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [1.26.0] - 2026-09-09
+
+### Changed
+
+- `tea-test-review` spent most of a small review reading files that were not in the pull request. `step-02-discover-tests.md` §2b's convention sample was capped at 40 files, and the agent runs with no shell, so each sampled file cost its own `Read` turn on every run whatever the size of the diff. The agent now reads 8. Measured on a 50-file corpus with a one-file review set: 40 files and ~233 KB before, 8 files and ~47 KB after, five times fewer reads and a fifth of the bytes. Eight clears §2b's `sampled < 4` "corpus too small to infer a house rule" floor with margin and leaves the 0.5 established/emerging ratio a denominator that means something. The sampled count is published in the report and cross-checked by `parse-report.js`, so every report states the corpus it was judged against.
+- The mechanical detector scan keeps the corpus it always had. It is a separate budget from the read set: `cli/lib/convention-baseline.js` opens those files itself and pays no agent turn, so it still scans the closest 40 while the agent reads 8 of them. `mechanicalSignal: false` is what forbids a report from citing any adoption for a key, and a zero observed over 8 files is much weaker evidence of absence than a zero observed over 40. Measured on this repository before the two budgets were split, `networkFirst` went from 1 of 40 to 0 of 8, which turns `emerging` into `absent` and removes the deduction entirely.
+- The 8 are spread evenly across the closest 40. Every file in the reviewed file's own directory ranks at distance 0, so among those the ranking is the alphabetical tie-break and nothing else. At a cap of 40 against a 45-file corpus that never showed; at 8 the tie-break became the selector. Measured on this repository, a review anchored in `test/` sampled `eval-all.js` through `test-contracts.js` and nothing named `test-e*` or later was reachable by any review anchored there, on any run. Taking every `floor(i × 40 / 8)`-th file keeps the count, the determinism and the neighborhood, and the sample now spans `test/`, `test/lib/`, `test/schema/` and `test/fixtures/`.
+- The convention corpus counts a file only when its name says it is a test, and skips `fixtures`, `__fixtures__`, `__mocks__` and `testdata` trees. `isTestFile` is deliberately wider, because the review set has to catch a changed helper under a test directory, and a directory-only match was never right for a baseline: on this repository it admitted `stub-agent.js` and a React component under `trace-eval/clean/src/tokens/`, and it admitted the seeded-defect specs under `test/fixtures/test-review-eval/seeded/` that the evals score against, so the house convention was partly established from deliberate anti-patterns. This repository's own corpus is now empty and its self-review reports the baseline as unavailable, which is the truthful answer for a repository whose only spec files are eval fixtures.
+- The four quality workers in `step-03-quality-evaluation.md` can now actually run in parallel. Three things had to change together. `cli/lib/build-prompt.js` pinned `tea_execution_mode=sequential`; the `claude` adapter's tool list carried no delegation tool, so step-03's capability probe found no launcher; and `tea_capability_probe` was never stated at all, which resolves `auto` to `sequential` on every run whatever the mode says. The prompt now states both orchestration keys, and `cli/lib/agent-adapters.js` grants `Task` from the `scoped-artifact-writes` tier upward in both `--tools` and `--allowedTools`. A launched subagent inherits that list exactly, verified live against claude 2.1.266: the child reported `Agent, Edit, Glob, Grep, Read, Write` and no shell, so "the shell only under command-execution" holds at the workers too. `read-only` runners get no delegation, since the workers' declared outputs cannot exist at that tier, and `codex exec` exposes no subagent launcher, so `auto` resolves to `sequential` there and that adapter's run is unchanged.
+- `tea_execution_mode` and `tea_capability_probe` resolve through the same chain the fragment keys do: `--execution-mode` / `--capability-probe` / `--no-capability-probe`, then `_bmad/tea/config.yaml`, then `src/module.yaml`. Both were previously stated in the prompt as literals, and the prompt declares that stated values beat `config.yaml`, so `tea_execution_mode: sequential` in the file that `docs/reference/troubleshooting.md` tells users to edit was being ignored.
+- The report carries a `**Execution Mode**:` line and the verdict JSON carries `executionMode`, required on any run that measured a convention baseline. step-03 resolves its mode from a runtime capability probe and printed the result only to the agent's stdout, which the CLI reads only when the report is missing, so a run that asked for parallel workers and silently got `sequential` was indistinguishable afterwards from one that got what it asked for. `step-03f-aggregate-scores.md` also stopped hardcoding `subagent_execution: 'PARALLEL (4 quality dimensions)'` and `performance_gain: '~60% faster than sequential'` regardless of what ran.
+- The CLI mints a `tea_run_id` and the workflow uses it as the timestamp in its four worker output paths. step-03 asked the agent to generate one with `new Date().toISOString()`, which an agent with no clock and no shell cannot execute, so it emitted a plausible string; nothing cleans `/tmp/tea-test-review-*` and step-03 checks only that the four files exist, so two runs landing on the same invented value would aggregate each other's scores.
+- The default agent wall-clock timeout scales with the review set: 20 minutes plus 2 minutes per reviewed file, capped at the 30 minutes it used to be flat at, with `--timeout-ms` still overriding it. Nothing bounds how many turns the agent takes and no supported vendor CLI offers a turn cap (`claude` 2.1.266 has `--max-budget-usd` and no `--max-turns`; `codex exec` has neither), so the wall clock is the only vendor-agnostic bound. A one-file review that stopped making progress used to be indistinguishable from a slow one for half an hour.
+
+### Fixed
+
+- Three statements of the module defaults that disagreed with `src/module.yaml` and with `cli/lib/resolve-tea-config.js`'s `MODULE_DEFAULTS`, which a test already holds equal to each other. `--use-pactjs-utils`'s help said the default is `false` and `--pact-mcp`'s said `none`; both are `true` and `"mcp"`. `docs/reference/tea-test-review-cli.md` repeated the same two wrong values and then advised the opposite of what the CLI does, telling a contract-testing repo it gets `tea_use_pactjs_utils: false` by default and should pass `--use-pactjs-utils` to change it. A repo on raw `@pact-foundation/pact` needs `--no-use-pactjs-utils`.
+
 ## [1.25.2] - 2026-09-09
 
 ### Fixed

package/DESIGN-CRITERIA-REGISTRY.md

@@ -63,7 +63,7 @@ or label-based locator now satisfies L1 outright.**
 ### The baseline is measured before it is judged against
 
 `step-02-discover-tests` §2b samples the corpus **outside the review set** (a PR must
-not establish or dilute the convention it is judged against), capped at 40 files
+not establish or dilute the convention it is judged against), capped at 8 files
 closest-first, and classifies each convention on a pinned threshold: `< 4` files is
 `unknown`, `0` adopted is `absent`, `>= 50%` is `established`, otherwise `emerging`.
 Unmeasurable means every Convention row passes as `n/a` and the report says so.
@@ -269,9 +269,13 @@ code path that looked for one.
 Same fix as the score and the recommendation before it: stop trusting the agent to
 compute a number the CLI can compute itself. `cli/lib/convention-baseline.js` now
 performs §2b's sampling deterministically — `git ls-files`, the review set excluded,
-ranked closest-first by directory distance, capped at 40 — and, for the five keys
+ranked closest-first by directory distance, capped at 8 — and, for the six keys
 with a literal recognized form (`priorityMarkers`, `testIds`, `networkFirst`,
-`dataFactories`, `fixtures`), scans the real sampled files for it. `bddNaming` and
+`dataFactories`, `fixtures`, `playwrightUtils`), scans the real file content for it.
+The scan reads a wider slice of the same ranking than the agent does, 40 files
+against 8, because the CLI opens them itself and the agent pays a turn per file.
+The zero-signal floor below is only as strong as the corpus it observed nothing in,
+so the read set got cheaper and the scanned corpus stayed where it was. `bddNaming` and
 `assertionStyle` get no mechanical signal (no single token separates "adopted" from
 "not" for a naming style or a dialect choice) and stay agent-judged; the
 sampled/corpusSize grounding still applies to them.
@@ -282,7 +286,7 @@ already does, and `parse-report.js`'s `verifyConventionBaseline` binds every
 `**Convention Baseline**:` line to it, one direction strictly (the sampled/corpusSize
 counts must match exactly — they're 100% mechanical) and one direction only
 downward (a citation claiming nonzero adoption for a key the CLI's own scan found
-zero real occurrences of anywhere in the sampled corpus is rejected outright; a lower
+zero real occurrences of anywhere in the scanned corpus is rejected outright; a lower
 or judgment-based count is left alone, because a regex cannot know intent and was
 never used to force a number up). Applied to #106's actual corpus, this scan finds
 zero `priorityMarkers` signal too — the same zero grep found — so the fabricated
@@ -292,8 +296,10 @@ Suite 8 reproduce this end-to-end against a real temp git repo built to the same
 shape (real neighbor test files, zero real priority markers anywhere).
 
 Not fixed here, and worth naming: the "seven keys" §2b measures don't map onto the
-registry's Convention gate class evenly. Only three published criteria are actually
-Convention-gated (`priorityMarkers` → L2, `testIds` → L3, `bddNaming` → L5);
+registry's Convention gate class evenly. Four published criteria are
+Convention-gated (`priorityMarkers` → L2, `testIds` → L3, `bddNaming` → L5, and
+`playwrightUtils` → M9 and L9, added later and the only one whose classification
+moves the score: M9 is MEDIUM at `established` and LOW at `emerging`);
 `networkFirst`/`dataFactories`/`fixtures` deliberately became Applicability-gated
 instead (see "The baseline is measured before it is judged against" above —
 popularity shouldn't demote a navigation race), and `assertionStyle` → L7 has no

package/cli/lib/agent-adapters.js

@@ -72,6 +72,25 @@ const DEFAULT_CAPABILITIES = ['scoped-artifact-writes'];
 
 const WRITE_TOOLS = ['Write', 'Edit'];
 const COMMAND_TOOLS = ['Bash'];
+// Delegation, which claude spells `Task` on the command line and exposes to the
+// model as `Agent`. step-03-quality-evaluation.md dispatches four quality workers
+// and resolves its execution mode from a runtime capability probe, so without this
+// the probe finds no launcher and every headless run collapses to `sequential` no
+// matter what tea_execution_mode says.
+//
+// A launched subagent inherits this exact list, verified live against claude
+// 2.1.266 under the argv below: the child reported `Agent, Edit, Glob, Grep, Read,
+// Write` and no shell. So "the shell only under command-execution" holds at the four
+// workers as well as at the parent, and granting delegation does not widen the tool
+// surface by the back door.
+//
+// It is granted from `scoped-artifact-writes` upward, that being the first tier
+// at which the workers' declared outputs can exist: each writes
+// /tmp/tea-test-review-<dimension>-<timestamp>.json, and step-03 section 5 aborts
+// the workflow when one of them is missing. A `read-only` runner cannot finish
+// that step in any mode, so handing it a launcher would widen the tool surface
+// while enabling nothing.
+const DELEGATE_TOOLS = ['Task'];
 
 /** The tier a capability list resolves to: the strongest one named. */
 function strongestCapability(capabilities = DEFAULT_CAPABILITIES) {
@@ -90,6 +109,7 @@ function claudeTools(capabilities) {
     ...(tier === 'read-only' ? [] : WRITE_TOOLS),
     'Glob',
     'Grep',
+    ...(tier === 'read-only' ? [] : DELEGATE_TOOLS),
     ...(tier === 'command-execution' ? COMMAND_TOOLS : []),
   ].join(',');
 }
@@ -196,8 +216,19 @@ const AGENT_ADAPTERS = {
     modelFlags: ['--model'],
     // --safe-mode strips repo customizations for the review run; --tools/
     // --allowedTools scope the run to the tool surface the caller's declared
-    // capabilities allow: search and read always, write only above read-only,
-    // the shell only under command-execution.
+    // capabilities allow: search and read always, write and delegation only
+    // above read-only, the shell only under command-execution. Both flags carry
+    // the same list because --tools decides what exists and --allowedTools
+    // decides what runs without a prompt; naming a tool in one and not the other
+    // either hides it or stops the headless run to ask about it.
+    //
+    // claude 2.1.266 has no turn cap to pair with these. `--max-budget-usd`
+    // is the nearest vendor bound and is reachable through the passthrough
+    // (--agent-arg --max-budget-usd --agent-arg 2.00); it stays out of this
+    // table for the same reason codex's reasoning effort does, since it is one
+    // vendor's flag and the other adapters cannot honor it. The vendor-agnostic
+    // bound is the wall-clock timeout in run-agent.js, which the CLI scales to
+    // the size of the review set.
     buildArgv: (extra = [], model, capabilities = DEFAULT_CAPABILITIES) => [
       '-p',
       '--output-format',
@@ -226,6 +257,14 @@ const AGENT_ADAPTERS = {
     // --skip-git-repo-check matters under --isolate, where the agent's cwd
     // is a fresh tmpdir with no .git.
     //
+    // No parallel-worker argv here, deliberately. `codex exec` exposes no
+    // subagent launcher for step-03's four quality workers to run on, so the
+    // prompt's tea_execution_mode=auto resolves through the capability probe to
+    // sequential on this adapter, which is the same run codex did before. The
+    // output contract, the aggregation, and the score are identical in either
+    // mode (step-03: "Mode changes orchestration only"), so codex loses the
+    // wall-clock gain and nothing else.
+    //
     // Reasoning effort is deliberately not pinned here. It is a second
     // unstated input (a local model_reasoning_effort = "max" costs ~10s even
     // on a one-word prompt, measured 2026-08-03), but it is codex-only, so

package/cli/lib/build-prompt.js

@@ -10,9 +10,11 @@
  * context_files stays an invocation-only wire so PR evidence can never become
  * a persistent user preference.
  *
- * It also states every TEA config key that step-01 branches on, resolved by
- * resolve-tea-config. An unstated key is one the agent decides for itself, which
- * makes knowledge loading differ between runs over identical files.
+ * It also states every TEA config key the workflow branches on: the fragment-loading
+ * keys step-01 reads, resolved by resolve-tea-config, and the orchestration pair
+ * step-03 reads (tea_execution_mode, tea_capability_probe). An unstated key is one
+ * the agent decides for itself. Two runs over identical files would then load
+ * different knowledge, or dispatch a different number of workers.
  *
  * Two file lists travel in the prompt, each as a JSON array inside its own
  * delimiters so paths are unambiguously data: the review set, which is scored,
@@ -67,20 +69,25 @@ function conventionBaselinePromptLines(conventionBaseline) {
     }
     return measured.mechanicalSignal
       ? [
-          `- ${key}: mechanically scanned; at least one sampled file contains a recognized form. Read the sampled files`,
-          `  yourself to judge the true adopted count (0-${conventionBaseline.sampled}) and record the observed form.`,
+          `- ${key}: mechanically scanned; at least one file in the wider scanned corpus contains a recognized form, so`,
+          '  this key is NOT ruled out. That corpus is larger than the sampled files below, so the form may well live in a',
+          `  file you were not asked to read. Judge the adopted count (0-${conventionBaseline.sampled}) from the sampled files alone and`,
+          '  record the observed form; 0 is a legitimate answer here and does not contradict the scan.',
         ]
       : [
-          `- ${key}: mechanically scanned across all ${conventionBaseline.sampled} sampled files; zero occurrences of any`,
+          `- ${key}: mechanically scanned across all ${conventionBaseline.scanned} scanned files; zero occurrences of any`,
           '  recognized form were found. This convention MUST be reported as absent: adopted = 0. A report claiming ANY',
-          `  nonzero adoption for ${key} will be rejected — the CLI already read every sampled file and found nothing.`,
+          `  nonzero adoption for ${key} will be rejected — the CLI already read every scanned file and found nothing.`,
         ];
   });
   return [
     "step-02-discover-tests.md §2b's convention baseline has already been computed for this run. Do not sample, glob,",
     'or guess this yourself — the corpus and the counts below came from actually reading the files named, not the',
     'reviewed files themselves (sampling the review set to judge the review set would be circular).',
-    `- corpusSize: ${conventionBaseline.corpusSize}, sampled: ${conventionBaseline.sampled}`,
+    `- corpusSize: ${conventionBaseline.corpusSize}, sampled: ${conventionBaseline.sampled}, scanned: ${conventionBaseline.scanned}`,
+    'sampled is the list below, the files you read. scanned is how many files the CLI ran its own mechanical detectors',
+    'over, which is wider because it costs the CLI a file read and costs you nothing. Cite sampled, never scanned:',
+    'the scanned corpus only decides which conventions the CLI has already ruled out for you.',
     `The "**Convention Baseline**:" line must read exactly: ${conventionBaseline.sampled} test files sampled outside the review set`,
     'Sampled files (read exactly these; do not substitute, add, or drop any):',
     '---BEGIN CONVENTION CORPUS---',
@@ -126,6 +133,14 @@ function conventionBaselinePromptLines(conventionBaseline) {
  *   only --test-glob put there and that no built-in rule recognizes. The CLI
  *   cannot know whether a registry row attached, so it names them and the agent
  *   applies criteria-registry rule 4 rather than publishing 100 - 0 = 100.
+ * @param {string} [options.runId] - Unique id for this run, minted by the CLI. The
+ *   workflow's step-03 uses it verbatim as the `timestamp` in its worker output
+ *   paths. step-03 asks the agent to generate one with `new Date().toISOString()`,
+ *   which a headless agent with no shell cannot execute: it emits a plausible
+ *   string instead, and two runs on one machine can land on the same one. Nothing
+ *   cleans /tmp/tea-test-review-*, and step-03 section 5 checks existence only, so
+ *   a repeat would aggregate a previous run's scores. A real unique value from the
+ *   caller removes the class.
  * @param {object} [options.conventionBaseline] - step-02-discover-tests.md §2b's
  *   "convention baseline", pre-computed by cli/lib/convention-baseline.js instead of
  *   left to the agent to sample. `{ baselineUnavailable: true, reason }` or
@@ -149,6 +164,7 @@ function buildPrompt({
   unscorableTestArtifacts = [],
   forcedUnscorableCandidates = [],
   conventionBaseline,
+  runId = '',
 }) {
   const absoluteSkillRoot = path.resolve(skillRoot);
   const absoluteOutputPath = path.resolve(outputPath);
@@ -186,7 +202,9 @@ function buildPrompt({
     `review_scope=${reviewScope}`,
     `test_dir=${testDir}`,
     'tea_browser_automation=none',
-    'tea_execution_mode=sequential',
+    `tea_execution_mode=${teaConfig.tea_execution_mode}`,
+    `tea_capability_probe=${teaConfig.tea_capability_probe}`,
+    ...(runId ? [`tea_run_id=${runId}`] : []),
     `tea_use_playwright_utils=${teaConfig.tea_use_playwright_utils}`,
     `tea_use_pactjs_utils=${teaConfig.tea_use_pactjs_utils}`,
     `tea_pact_mcp=${teaConfig.tea_pact_mcp}`,
@@ -197,6 +215,25 @@ function buildPrompt({
     'fragment set, Pact MCP) instead of inferring the flags.',
     'The two *_installed values above were read from the project manifest by the CLI. Do not re-derive them, and do',
     'not open package.json: they are the second half of each mandate gate, stated here for the same reason the flags are.',
+    'tea_execution_mode and tea_capability_probe are the orchestration pair step-03-quality-evaluation.md branches on,',
+    'resolved the same way the fragment keys above are: an explicit CLI flag, then _bmad/tea/config.yaml, then',
+    'src/module.yaml\'s default. Both are stated because step-03 reads both, and "auto" with probing off resolves to',
+    'sequential on every run.',
+    'Whichever mode resolves, dispatch every quality worker with the full subagentContext step-03 section 1 assembles,',
+    'written out in the launch prompt itself: the review set, the criteria-registry path resolved to an absolute path,',
+    'the convention baseline block stated above verbatim, and the playwright_utils_installed / pactjs_utils_installed',
+    'values. A worker that has to guess any of those scores rows it cannot see, which silently removes deductions.',
+    ...(runId
+      ? [
+          `Use tea_run_id (${runId}) verbatim wherever step-03-quality-evaluation.md section 1 says to generate a`,
+          '`timestamp`, and pass that same value to every worker and to step 3F. Do not generate one: you have no clock,',
+          'nothing cleans /tmp/tea-test-review-*, and step-03 section 5 checks only that the files exist, so an invented',
+          "value that collided with an earlier run's would aggregate that run's scores into this report.",
+        ]
+      : []),
+    'Report the mode you actually ran in as exactly one "**Execution Mode**: <mode>" line in the Executive Summary,',
+    'where <mode> is agent-team, subagent, or sequential. Never write "auto" there: auto is the request, and this line',
+    'records what the capability probe resolved it to.',
     'playwrightUtilsActive = tea_use_playwright_utils AND playwright_utils_installed; when true, load',
     'playwright-utils-mandate.md and score registry rows M9 and L9. pactjsUtilsActive = tea_use_pactjs_utils AND',
     'pactjs_utils_installed; when true, load pactjs-utils-mandate.md and score registry row M10.',

package/cli/lib/convention-baseline.js

@@ -18,19 +18,24 @@
  * is never asked to produce a number this module can produce itself.
  *
  * Two tiers of grounding:
- * - corpusSize / sampled / sampledFiles: 100% mechanical (git ls-files, isTestFile,
- *   directory-distance ranking, a 40-file cap) — this is exactly step-02 §2b's
- *   sampling rules, just executed by code instead of described in prose. A report
- *   that cites a different sampled count, or a different corpus, is provably wrong
- *   and rejected outright.
+ * - corpusSize / sampled / sampledFiles: 100% mechanical (git ls-files, a real test
+ *   filename with fixture trees excluded, directory-distance ranking, then a stride
+ *   of 8 across the closest 40) — this is exactly step-02 §2b's sampling rules, just
+ *   executed by code instead of described in prose. A report
+ *   that cites a different sampled count is provably wrong and rejected outright.
+ *   (verifyConventionBaseline checks the count; the corpus itself travels in the
+ *   prompt as a do-not-substitute list and is not re-derivable from the report.)
  * - adopted counts for the six keys with a concrete, literal recognized form
  *   (priorityMarkers, testIds, networkFirst, dataFactories, fixtures, playwrightUtils): a generous,
  *   high-recall regex scan over the real sampled files' real content. This is not
  *   claimed to be a precise semantic judgment — a regex cannot know intent — but it
  *   gives a safe one-directional floor: when the scan finds ZERO occurrences of any
- *   recognized form anywhere in the entire sampled corpus, the true adopted count is
+ *   recognized form anywhere in the entire scanned corpus, the true adopted count is
  *   zero with very high confidence (a looser scan finding nothing all but rules out
  *   a careful reader finding something), so the report may never claim otherwise.
+ *   That scan reads up to 40 files, wider than the 8 the agent is asked to read,
+ *   because this module opens them itself and pays no agent turn for any of them.
+ *   The floor is only as strong as the corpus it observed nothing in.
  *   The two keys with no literal recognized form (bddNaming — a naming *style*, not
  *   a token; assertionStyle — dialect consistency) get no mechanical signal and stay
  *   fully agent-judged; only their sampled/corpusSize grounding applies. This is an
@@ -59,7 +64,31 @@ const CONVENTION_KEYS = [
   'playwrightUtils',
 ];
 
-const MAX_SAMPLED_FILES = 40;
+// The sample is the review's single largest input, and it is paid on every run
+// regardless of how small the pull request is. The agent has no shell (see
+// agent-adapters.js's claude tool list), so a sampled file can only be opened one
+// Read at a time: the cap is a turn budget as much as a byte budget. At 40, and
+// measured on this repository against a one-file review set, the sample was
+// ~609 KB of files that were not in the diff; at 8 it is ~289 KB, over five times
+// fewer reads.
+//
+// Eight keeps the two properties the cap has to keep. It stays above
+// step-02-discover-tests.md §2b's `sampled < 4` "corpus too small to infer a
+// house rule" floor with margin, and it leaves enough files for §2b's 0.5
+// established/emerging ratio to mean something. Changing this changes the number
+// every report cites, so step-02 §2b's "Sampling rules" states the same figure.
+const MAX_SAMPLED_FILES = 8;
+
+// The mechanical scan is a different budget from the read set, so it gets its own
+// cap. Nothing the agent does bounds it: this module opens the files itself with
+// fs.readFileSync and runs six regexes, which costs milliseconds and no turns at
+// all. Shrinking it alongside the read set would have quietly weakened the one
+// claim the scan exists to make. `mechanicalSignal: false` is what forbids the
+// report from citing any adoption for a key (see conventionBaselinePromptLines
+// and parse-report.js's verifyConventionBaseline), and a zero observed over 8
+// files is much weaker evidence of absence than a zero observed over 40. The
+// floor keeps its old corpus; only the agent's reading got cheaper.
+const MAX_SCANNED_FILES = 40;
 const MIN_CORPUS_TO_ATTEMPT = 1; // 0 eligible files outside the review set: nothing to sample at all.
 
 // One regex per mechanically-recognizable key, matched against a sampled file's raw
@@ -85,6 +114,48 @@ const MECHANICAL_DETECTORS = {
   playwrightUtils: /(?:from|require\s*\(|import\s*\()\s*['"`]@seontechnologies\/playwright-utils/i,
 };
 
+// §2b says "Sample test files". `isTestFile` is deliberately wider than that,
+// because the review set has to catch a changed helper sitting under a test
+// directory, and a directory-only match is enough there. It is not enough here. A
+// convention baseline built from files that are not tests measures the wrong thing:
+// on this repository the directory rule pulled in `stub-agent.js` and a React
+// component under `trace-eval/clean/src/tokens/`, and the house Playwright
+// convention was being established partly from those.
+const NAMED_TEST_FILE_PATTERNS = [
+  /^test_.*\.py$/,
+  /_test\.py$/,
+  /_test\.go$/,
+  /_spec\.rb$/,
+  /\.cy\.[^.]+$/,
+  /Test\.php$/,
+  /Tests?\.cs$/,
+  /Tests?\.java$/,
+  /\.pacttest\.[^.]+$/,
+  /\.test\.[^.]+$/,
+  /\.spec\.[^.]+$/,
+  /\.test-[^.]+\.[^.]+$/,
+];
+
+// A repository's test-tooling fixtures are not its test suite. A spec under a
+// directory that declares itself fixture data is an input to some other test, and
+// the sharpest case is a seeded-defect corpus: this repository keeps deliberately
+// bad specs under `test/fixtures/test-review-eval/seeded/` for its own evals to
+// score against, and sampling those to establish "what this repo does" would
+// establish the anti-patterns. Costs a consumer repository nothing, since spec
+// files rarely live under a fixtures directory there, and when the exclusion empties
+// the corpus the baseline reports itself unavailable rather than guessing.
+const FIXTURE_TREE_PATTERN = /(^|\/)(fixtures|__fixtures__|__mocks__|testdata)(\/|$)/;
+
+/** Whether a path is a real test file for baseline purposes: named like one, and not fixture data. */
+function isConventionCorpusFile(filePath) {
+  const normalized = filePath.replaceAll('\\', '/');
+  if (FIXTURE_TREE_PATTERN.test(normalized)) {
+    return false;
+  }
+  const basename = normalized.slice(normalized.lastIndexOf('/') + 1);
+  return NAMED_TEST_FILE_PATTERNS.some((pattern) => pattern.test(basename));
+}
+
 const MECHANICAL_CONVENTION_KEYS = Object.keys(MECHANICAL_DETECTORS);
 // bddNaming and assertionStyle: no literal token distinguishes "adopted" from "not",
 // so no mechanical signal is offered for them. Grounded on sampled/corpusSize only.
@@ -138,16 +209,18 @@ function closestDistance(candidateDir, reviewedDirs) {
 }
 
 /**
- * Mechanically measure adoption of the five literal-form conventions across the
- * already-sampled files' real content.
+ * Mechanically measure adoption of the six literal-form conventions across the
+ * scanned files' real content.
  *
  * @param {object} options
  * @param {string} options.projectRoot
  * @param {string[]} options.sampledFiles - Repo-relative paths, already capped/ranked.
+ *   This is the scanned corpus, which is wider than the agent's read set; see
+ *   computeConventionBaseline's `scanned` field.
  * @returns {object} One entry per CONVENTION_KEYS, `{ mechanical: false }` for the
  *   two judgment-only keys, `{ mechanical: true, adopted, mechanicalSignal }` for
  *   the rest. `mechanicalSignal` is `adopted > 0`, i.e. whether the scan found the
- *   convention anywhere at all in the sampled corpus.
+ *   convention anywhere at all in the scanned corpus.
  */
 function measureConventions({ projectRoot, sampledFiles }) {
   const conventions = {};
@@ -174,36 +247,82 @@ function measureConventions({ projectRoot, sampledFiles }) {
   return conventions;
 }
 
+/**
+ * Take `count` entries spread evenly across `ranked`, always including its head.
+ *
+ * Taking the first `count` instead would be wrong in a way that hides itself.
+ * Every file in the reviewed file's own directory has distance 0, so within that
+ * directory the ranking is the alphabetical tie-break and nothing else. At a cap of
+ * 40 against a 45-file corpus that never showed, because the sample was almost the
+ * whole corpus. At a cap of 8 the tie-break becomes the selector: measured on this
+ * repository, a review anchored in `test/` sampled `eval-all.js` through
+ * `test-contracts.js` and nothing named `test-e*` or later was reachable by any
+ * review, ever. That is a systematic wrong draw rather than a noisy one, and a
+ * systematic draw has no error bars to argue about.
+ *
+ * Striding keeps every property the head had. It is deterministic (same ranking,
+ * same indices, same files on every machine), it stays inside the reviewed file's
+ * neighborhood (the whole stride runs over the closest `scanCap` files), and it
+ * costs the same `count` reads. What it adds is that the sample spans the corpus
+ * it claims to describe.
+ *
+ * @param {string[]} ranked - Ranked file list, closest-first.
+ * @param {number} count - How many to take.
+ * @returns {string[]} Up to `count` files, in ranked order.
+ */
+function strideSelect(ranked, count) {
+  if (count >= ranked.length) {
+    return [...ranked];
+  }
+  const picked = [];
+  for (let index = 0; index < count; index++) {
+    picked.push(ranked[Math.floor((index * ranked.length) / count)]);
+  }
+  return picked;
+}
+
 function unavailable(reason) {
-  return { baselineUnavailable: true, reason, corpusSize: 0, sampled: 0, sampledFiles: [], conventions: {} };
+  return { baselineUnavailable: true, reason, corpusSize: 0, sampled: 0, scanned: 0, sampledFiles: [], conventions: {} };
 }
 
 /**
  * Compute the convention baseline for a review run: the real corpus outside the
- * review set, ranked closest-first by directory distance and capped at 40 (per
- * step-02-discover-tests.md §2b's sampling rules), plus a mechanical adoption scan
- * over the sampled files' real content.
+ * review set, ranked closest-first by directory distance, plus a mechanical
+ * adoption scan over real file content.
+ *
+ * Two corpora come out of the one ranking, and they are different sizes on purpose.
+ * `scanned` (the closest 40) is how many files this module opened itself for the
+ * mechanical detectors. `sampledFiles` (8, per step-02-discover-tests.md §2b's
+ * sampling rules) is a stride across those 40, and it is what the agent is told to
+ * read and the denominator every report cites. The first is not a turn budget and
+ * the second is, so keeping the scan wide costs milliseconds and keeps the
+ * zero-signal floor standing on the evidence it always had.
  *
  * @param {object} options
  * @param {string} options.projectRoot - Repo root (git ls-files runs here).
  * @param {string[]} options.reviewFiles - The review set; excluded from sampling and
  *   used as the distance anchor.
- * @param {number} [options.cap] - Sample size cap (default 40, matching step-02).
+ * @param {number} [options.cap] - Agent read-set cap (default 8, matching step-02).
+ * @param {number} [options.scanCap] - Mechanical scan cap (default 40). Raised to
+ *   `cap` when it is smaller, since the read set is always drawn from the scanned
+ *   corpus.
  * @returns {object} `{ baselineUnavailable: true, reason, ... }` when no corpus
  *   exists outside the review set (or git ls-files failed), otherwise
- *   `{ baselineUnavailable: false, corpusSize, sampled, sampledFiles, conventions }`.
+ *   `{ baselineUnavailable: false, corpusSize, sampled, scanned, sampledFiles, conventions }`.
  */
-function computeConventionBaseline({ projectRoot, reviewFiles, cap = MAX_SAMPLED_FILES }) {
+function computeConventionBaseline({ projectRoot, reviewFiles, cap = MAX_SAMPLED_FILES, scanCap = MAX_SCANNED_FILES }) {
   const tracked = listGitTrackedFiles(projectRoot);
   if (tracked === null) {
     return unavailable('could not list repository files (git ls-files failed; not a git repo, or a git error)');
   }
 
   const reviewSet = new Set(reviewFiles.map((file) => file.replaceAll('\\', '/')));
-  const eligible = tracked.filter((file) => isTestFile(file) && !reviewSet.has(file));
+  const eligible = tracked.filter((file) => isTestFile(file) && isConventionCorpusFile(file) && !reviewSet.has(file));
   const corpusSize = eligible.length;
   if (corpusSize < MIN_CORPUS_TO_ATTEMPT) {
-    return unavailable('no test files exist outside the review set to measure a house convention against');
+    return unavailable(
+      'no test files exist outside the review set to measure a house convention against (fixture and test-data trees are excluded)',
+    );
   }
 
   const reviewedDirs = reviewFiles.length > 0 ? reviewFiles.map(directoryOf) : [''];
@@ -213,15 +332,17 @@ function computeConventionBaseline({ projectRoot, reviewFiles, cap = MAX_SAMPLED
     // the same sampled set on every machine regardless of the runtime's ICU/locale
     // data, or "deterministic sampling" would itself be an environment-dependent claim.
     .sort((a, b) => a.distance - b.distance || (a.file < b.file ? -1 : a.file > b.file ? 1 : 0));
-  const sampledFiles = ranked.slice(0, cap).map((entry) => entry.file);
+  const scannedFiles = ranked.slice(0, Math.max(cap, scanCap)).map((entry) => entry.file);
+  const sampledFiles = strideSelect(scannedFiles, cap);
 
   return {
     baselineUnavailable: false,
     reason: null,
     corpusSize,
     sampled: sampledFiles.length,
+    scanned: scannedFiles.length,
     sampledFiles,
-    conventions: measureConventions({ projectRoot, sampledFiles }),
+    conventions: measureConventions({ projectRoot, sampledFiles: scannedFiles }),
   };
 }
 
@@ -234,4 +355,6 @@ module.exports = {
   directoryDistance,
   directoryOf,
   measureConventions,
+  strideSelect,
+  isConventionCorpusFile,
 };

package/cli/lib/parse-report.js

@@ -71,6 +71,11 @@ const { SEVERITY_ENUM } = require('./registry-rows');
 const RECOMMENDATION_ENUM = ['Approve', 'Approve with Comments', 'Request Changes', 'Block'];
 const RECOMMENDATION_LINE = /^[ \t]*(?:\*\*Recommendation\*\*:|\*\*Recommendation:\*\*|Recommendation:)[ \t]*([^\r\n]+?)[ \t]*$/m;
 const CONTEXT_BASIS_ENUM = ['none', 'pr_diff', 'pr_diff_truncated'];
+// step-03-quality-evaluation.md's own three resolved modes. `auto` is a request,
+// never a resolution, so it is deliberately absent: a report that says `auto` never
+// ran the probe.
+const EXECUTION_MODE_ENUM = ['agent-team', 'subagent', 'sequential'];
+const EXECUTION_MODE_LINE_SOURCE = String.raw`^[ \t]*\*\*Execution Mode:?\*\*:?[ \t]*([^\r\n]+?)[ \t]*$`;
 const CONTEXT_BASIS_LINE_SOURCE = String.raw`^[ \t]*\*\*Context Basis:?\*\*:?[ \t]*([^\r\n]+)[ \t]*$`;
 const CONTEXT_WAIVERS_LINE_SOURCE = String.raw`^[ \t]*\*\*Context Waivers Applied:?\*\*:?[ \t]*([^\r\n]+)[ \t]*$`;
 const SCORE_PATTERN = /\*\*Quality Score\*\*:\s*(\d+)\s*\/\s*100(?:[ \t]*\([ \t]*([A-F])(?=[ \t)-]))?/;
@@ -155,6 +160,7 @@ const PARSED_VERDICT_KEYS = {
   },
   conditional: {
     conventionBaseline: 'object',
+    executionMode: 'string',
     reportedQualityScore: 'number',
     reportedRecommendation: 'string',
   },
@@ -696,6 +702,51 @@ function parseConventionCitations(text) {
   }));
 }
 
+/**
+ * Parse the at-most-one "**Execution Mode**:" line, required on any run that
+ * measured a convention baseline.
+ *
+ * `step-03-quality-evaluation.md` resolves its own execution mode from a runtime
+ * capability probe and prints the result to the agent's stdout, which
+ * `cli/lib/run-agent.js` captures and `cli/test-review.js` reads only when the
+ * report is missing. So a run that asked for parallel workers and silently got
+ * `sequential` was indistinguishable, after the fact, from one that got what it
+ * asked for. That made "this change made the review faster" unfalsifiable: the
+ * gain could have come from anywhere. The mode is a run input the same way the
+ * model and the convention baseline are, so it travels in the verdict with them.
+ *
+ * Required exactly when the run supplied a convention baseline, which every real
+ * CLI run does. Making it unconditionally optional would have left the hole it
+ * exists to close: a report that omits the line is as silent about its mode as one
+ * written before the line existed, and the failure gradient would run the wrong way,
+ * with an omission passing and a malformed value exiting 3. A bare `parseReport`
+ * call in a unit test supplies no baseline and needs no mode.
+ *
+ * @param {string} text - Full report.
+ * @param {boolean} required - Whether this run has to state a mode.
+ * @returns {string|null} The resolved mode, or null when none is stated and none is required.
+ */
+function parseExecutionMode(text, required) {
+  const matches = [...text.matchAll(new RegExp(EXECUTION_MODE_LINE_SOURCE, 'gm'))];
+  if (matches.length === 0) {
+    if (required) {
+      unparseable(
+        'Report is missing the "**Execution Mode**:" line; state the mode step-03 actually resolved ' +
+          `(${EXECUTION_MODE_ENUM.join(' | ')}), so a run that fell back to sequential says so in its own artifact`,
+      );
+    }
+    return null;
+  }
+  if (matches.length > 1) {
+    unparseable(`Report must contain at most one "**Execution Mode**:" line; found ${matches.length}`);
+  }
+  const cleaned = stripWrappers(matches[0][1].replaceAll(/\s+/g, ' ')).toLowerCase();
+  if (!EXECUTION_MODE_ENUM.includes(cleaned)) {
+    unparseable(`Report Execution Mode "${cleaned}" is not one of: ${EXECUTION_MODE_ENUM.join(' | ')}`);
+  }
+  return cleaned;
+}
+
 /** Extract the optional, at-most-one "**Convention Baseline**:" line's raw value. */
 function parseConventionBaselineLine(text) {
   const matches = [...text.matchAll(new RegExp(CONVENTION_BASELINE_LINE_SOURCE, 'gm'))];
@@ -795,8 +846,9 @@ function verifyConventionBaseline(text, conventionBaselineContract) {
     if (measured && measured.mechanical && measured.mechanicalSignal === false && citation.adopted > 0) {
       unparseable(
         `Report Convention citation "${citation.key} (${citation.adopted} of ${citation.sampled} sampled)" claims adoption, but this run ` +
-          `scanned every one of the ${conventionBaselineContract.sampled} sampled files for the recognized forms and found zero occurrences; ` +
-          'a convention with no real evidence in the sampled corpus must be reported as absent (0 adopted), never a fabricated nonzero count',
+          `scanned every one of the ${conventionBaselineContract.scanned ?? conventionBaselineContract.sampled} files in its scanned corpus ` +
+          'for the recognized forms and found zero occurrences; a convention with no real evidence anywhere in that corpus must be ' +
+          'reported as absent (0 adopted), never a fabricated nonzero count',
       );
     }
   }
@@ -1465,6 +1517,7 @@ function parseReport(reportText, runContract = {}) {
 
   verifyRunContract({ reviewedFiles, contextBasis, contextFiles, excludedFiles }, runContract);
   verifyConventionBaseline(text, runContract.conventionBaseline);
+  const executionMode = parseExecutionMode(text, Boolean(runContract.conventionBaseline));
 
   const executiveSection = extractSection(text, 'Executive Summary');
   const keyStrengths = extractBullets(extractSubsection(executiveSection, 'Key Strengths'), '✅');
@@ -1500,6 +1553,9 @@ function parseReport(reportText, runContract = {}) {
   if (runContract.conventionBaseline) {
     setConditionalKey(parsed, 'conventionBaseline', runContract.conventionBaseline);
   }
+  if (executionMode !== null) {
+    setConditionalKey(parsed, 'executionMode', executionMode);
+  }
   if (
     reportedQualityScore !== qualityScore ||
     (reportedQualityGrade !== undefined && reportedQualityGrade !== gradeForScore(qualityScore))
@@ -1575,6 +1631,8 @@ module.exports = {
   CONTEXT_BASIS_ENUM,
   verifyConventionBaseline,
   parseConventionCitations,
+  parseExecutionMode,
+  EXECUTION_MODE_ENUM,
   verifyFindingSeverityCounts,
   extractFindings,
   FINDING_KEYS,