npm - bmad-method-test-architecture-enterprise - Versions diffs - 1.26.0 → 1.26.1-next.1 - Mend

bmad-method-test-architecture-enterprise 1.26.0 → 1.26.1-next.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (94) hide show

package/.claude-plugin/marketplace.json CHANGED Viewed

@@ -31,7 +31,7 @@
       "name": "bmad-method-test-architecture-enterprise",
       "source": "./",
       "description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
-      "version": "1.26.0",
+      "version": "1.26.1-next.1",
       "author": {
         "name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
       },

package/.github/workflows/quality.yaml CHANGED Viewed

@@ -148,12 +148,18 @@ jobs:
       - name: Validate trace eval corpus
         run: npm run test:eval-trace-data
+      - name: Validate the bmad-tea routing eval corpus
+        run: npm run test:eval-routing-data
       - name: Validate the eval suite manifest and result schema
         run: npm run test:eval-schemas
       - name: Replay stored eval outputs against the scorers
         run: npm run test:eval-replay
+      - name: Compile eval-quality's own published corpus against the pinned release
+        run: npm run test:eval-quality-corpus
       - name: Check the eval contracts against the sources they are generated from
         run: npm run test:contract-sources
@@ -175,6 +181,9 @@ jobs:
       - name: Probe every TEA command target through the eval-quality adapter
         run: npm run test:probe-targets
+      - name: Check TEA's branches are total over the published port vocabularies
+        run: npm run test:port-totality
       - name: Test changelog stamping
         run: npm run test:changelog

package/CHANGELOG.md CHANGED Viewed

@@ -7,6 +7,80 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
+### Added
+- `npm run test:eval-quality-corpus` compiles `eval-quality`'s own published corpus against the pinned release, before any TEA artifact is asked to run on it.
+  TEA's every other check feeds the package TEA's own bytes, so a package regression and a bad migration read identically, and this upgrade moved 31 probes, 10 contracts and both command policies at once.
+  This check feeds the package nothing of TEA's. The `./corpus/*` subpath ships `corpus/dev/index.json`, which names 27 entries with a `sha256:` digest each: 24 contracts under `contracts/`, the compile-and-seal example's contract and its sealed brief, and the corpus README. The index also records the failure code each of the three deliberately-refused contracts raises, and it carries no digest of itself, which is the one corpus file it cannot list.
+  So the corpus states the expected result and the package computes the observed one. Measured against 3.0.0: 27 entry digests match the shipped bytes, 22 contracts compile, 3 are refused with the code the index records, and the sealed brief the example produces is byte for byte the shipped `brief.json`.
+  It also holds the resolved version against the exact pin in `package.json`, so a tree that is not the tree the repository declares fails here rather than somewhere downstream.
+  Exit 1 is a moved status, a moved digest or a moved brief; exit 2 is a package, subpath or index that could not be read, an index naming no contract, or a tree resolving a version other than the pin. So an unresolvable install, an empty corpus and a tree nobody installed are each reported as nothing measured rather than as a package regression.
+- `npm run test:port-totality` holds TEA's branches against the port vocabularies the installed package declares.
+  `ProbeRequest` and `ProbeObservation` are three-member unions tagged by `kind` and the conformance surface publishes six arms, and TEA is CommonJS with no typechecker, so no compiler and no reviewer will notice the day a fourth member arrives: the failure mode is code that keeps working on the members it knows.
+  Every member is now handled or declined with the reason recorded, every arm is run or recorded as not yet run, and a member or arm in neither fails the check by name.
+  `probeCommand` narrows every observation to the `cli` member and raises a named error otherwise, which the check proves by calling the narrowing with the two members TEA declines.
+  Three of the four harnesses reach the port through `probeCommand`; `test/eval-contract-strength.js` caches every observation itself, so it narrows at its own boundary, on the live answer and on the cached one.
+  The narrowing throws outside the fault handler on purpose: a member TEA cannot read is a defect in TEA, and classifying it as an environment failure would file that defect as a lost run.
+- `bmad-tea` has a behavioral eval suite and is no longer declared `deferred` in `test/evals/suite-manifest.json`. It was the only TEA skill with no eval of any kind, and the one thing it does is the step every workflow runs after: turn a sentence a user wrote into one item on its ten-item menu. A wrong route is the most expensive mistake the module can make and the cheapest one to miss, because every artifact downstream of it is well-formed, cites the repository, and answers a question nobody asked. `test/eval-bmad-tea-routing.js` puts eighteen intents to the skill through `cli/routing-runner.js`, a new `read-only` command registered as `tea-routing-runner`: ten with a single right menu item, four where two or more items are genuinely close and the right answer is to ask, and four nothing on the menu serves. Each ambiguous intent is ambiguous from what the agent is actually shown, which is the message plus `SKILL.md` and `customize.toml`; an intent whose second reading only opens once you have read a workflow's frontmatter is not ambiguous to the agent, and scoring it as a clarification would measure a fact it was never given.
+- Two of the three answers are Step 8 of `src/agents/bmad-tea/SKILL.md` verbatim and the third is not, which is worth stating rather than glossing. Step 8 says to dispatch a clear match directly and to pause only when two or more items are genuinely close, so `route` and `clarify` are the skill's own rules. Its third clause says that when nothing on the menu fits the agent should just continue the conversation, and names chat, clarifying questions and `bmad-help` as fair game: it forbids a dispatch and asks for no refusal. The requirement that an unservable intent be declined and that the answer say what is missing is the suite's own exit condition, and the prompt states it outright, so the skill is scored against a rule it was handed rather than one inferred from text that does not carry it. The skill itself is not edited.
+- Five metrics and four counts, and three of the counts are the controls. `routeAccuracy`, `reasonTokenRecall`, `scopeFidelity`, `clarifyRecall` and `declineRecall` measure the decision, the stated reason, the scope the user named, the clarification and the decline. `maxConfidentRoutesOnUnservable`, `maxConfidentRoutesOnAmbiguous` and `maxUnroutedClearIntents` are all zero and they close the cells recall leaves open: a skill that guesses on everything fails the first two, and one that asks about everything fails the third, which Step 8 rules out by name as a confirmation ritual. `maxUnstableCases` is two of eighteen at two repetitions. Every threshold sits on a value its denominator can actually produce, because a threshold between two reachable values is not the gate it reads as, and every one of them except stability has a stored replay case that fails it and nothing else. Stability cannot have one, since it is a property of a pair of repetitions and a stored case is one reply; what is pinned instead is the signature the live run measures it with.
+- The stated reason is scored by token containment and never by judgment, and every token has to be a word of the user's own message. The harness refuses one that is not, and that rule is what makes the number mean anything: the prompt asks the agent to name the deciding feature in the message's own words, so a token reachable only from the menu row it is being scored for choosing could be satisfied by echoing that row without reading the message, and a token in neither place would make the threshold unreachable for a skill doing exactly what it was told. `test/eval-bmad-tea-routing.js` owns the one pattern rule and `tools/generate-contracts.js` writes the same pattern sources into the contracts, so `npm run test:contract-oracles` compares an oracle and a scorer that cannot spell it two ways. Scope is held to containment and to a length bound together, because the same prompt asks for the scope in the message's own words and echoing the whole message would otherwise satisfy any token drawn from it.
+- The answers never reach the agent. `test/fixtures/tea-routing-eval/intents.json` holds the eighteen messages and is the only file the prompt carries; `ground-truth.json` holds the expected action, menu code, workflow, deciding tokens, scope tokens and clarification candidates. Every assembled prompt is searched for each ground-truth-only key and for the file's own bytes before a model call is spent, and that check runs under `--validate-only`, so it is in the pull-request gate rather than only in the runs that cost money. Every expected menu code is checked against the live `src/agents/bmad-tea/customize.toml`, so editing the menu fails the corpus instead of quietly invalidating it.
+- Seventeen stored replies under `test/replay/bmad-tea-routing/` score with no model call, and they are the only part of the suite the deterministic gate can run end to end. They cover a correct route, a wrong menu code, a route whose scope is the whole message copied back, a grounded route with an ungrounded reason, a route that dropped the epic the user named, a `GATE` dispatch that named a workflow behind a menu item whose target is a prompt, a route whose reason is written the way a person writes rather than as a list of the fixture's tokens, a clarification that names nothing to choose between, one that names all three candidates, one that names both by menu label and never by code, a confident route on a genuinely ambiguous intent, a confident route on an unservable one, a decline that says what is missing and one that does not, a clear intent left unrouted, an answer inside a fenced block, and a reply carrying no answer at all. Every stored result was derived by hand.
+- Two eval contracts and two probe corpora, generated. The suite is expressed as `tea-routing-intents.contract.json` and `tea-routing-controls.contract.json` because `eval-quality`'s AD-39 scripting bound caps an interaction plan at sixteen steps and the corpus is eighteen intents, one step each; the manifest's `contracts` field is an array for exactly this. Both bind standard input as a literal prompt per step, which is what tells eighteen steps declaring one operation apart, and what the pair costs in bytes. Seventy-two oracles between them read the action, the menu code, the workflow behind it, the stated reason, the scope and the clarifying question, and every one of them was seen resolving false on some answer as well as true on another. The probe corpora carry a gameability probe and a clean control each and no defect probe: a defect probe needs a controlled mutation of the system under test, and the system here is the skill, which this suite must not edit.
+### Changed
+- `eval-quality` moves from 1.4.0 to 3.0.0.
+  The pin alone left four checks red, which is why every migration below lands with it.
+- Every `CommandTargetPolicy` TEA builds declares `permittedEnvironmentKeys`, required on 3.0.0 with no default.
+  The environment channel was the one command channel with no operator bound: the contract author declared it and everything declared reached the process.
+  The list is stated per target rather than once for every command TEA ships, because a key a command never reads has no business reaching it. All three permit exactly what their own contracts declare, the vendor names plus `HOME` and `USER`, read from `vendorEnvironmentNames()` so the contract and the policy cannot drift.
+  `PATH` is on no list, refused where the widening happens as well as by the schema and by the adapter, because `target` may name a bare command and a declared `PATH` would then choose which binary runs.
+  A per-interface override naming an interface the policy does not authorize is refused too: it widens nothing, and unnoticed it leaves every request of that harness carrying names the authorization never permitted.
+  A caller widens an authorization one call at a time through `environmentKeys`, which is how an operator's `--env-pass` names and this repository's two stub variables reach a process.
+- `CI` reaches no measured run, and its absence is what makes a review reproducible.
+  `cli/test-review.js` reads it to decide filesystem isolation when `--isolate` is not stated, and no contract declares it, so passing it through let the host decide how the measured run executed: isolation on in GitHub Actions, off on a laptop, with the two sealed records indistinguishable afterwards.
+  TEA's harness had been sending it since before the environment channel had an operator bound, and `test/eval-contract-strength.js` had been filtering it back out against the contract, so the two halves of the same suite were already measuring different commands.
+  A command run through the port now sees no `CI` at all and isolates the same way on every host; `test/eval-test-review.js` states `--isolate` explicitly and never depended on the variable.
+  `test/test-probe-targets.js` holds each contract's declared keys equal to its authorization's permitted keys, in both directions, so neither side can drift from the other again.
+- All 31 probes carry `schemaVersion: 5` and all 10 contracts carry `5`, written by their generators rather than by hand.
+  On 3.0.0 a stale stamp is a `schema-version-mismatch` runtime fault at exit 5 in both `preflight` and `score`, so every probe corpus and every contract compile was failing on the version alone.
+  The load-bearing edit is the ninth input channel: each of the 21 `inputBinding` selectors gains `"arguments": null`, which version 5 added so a signature against a tool call can filter on what the call supplied.
+  `tools/generate-contracts.js` now names its stamp `EVAL_CONTRACT_SCHEMA_VERSION` instead of repeating the literal in three places.
+- TEA's sealed run records carry `schemaVersion: 6` and no `invalidReason`, and their observations carry the ninth `arguments` call-input channel.
+  Version 6 drops `invalidReason` because nothing in the package ever read it, so a caller attesting that a run was invalid was ignored by every stage while the field looked like a supported channel.
+  `SCHEMA_VERSIONS` in `test/lib/eval-quality-inputs.js` read 3 for both the probe and the record; the record entry is the one with a reader, and it was emitting version-3 records against a version-6 parser.
+- The command-probe conformance arm reports 16 outcomes where it reported 15. The count is the package's and moved with the version; what TEA supplies is `unauthorizedEnvironmentKeyRequest`, which is what makes the sixteenth assertion pass rather than fail.
+  That request is authorized in every other respect and declares exactly one environment key its authorization omits, and the suite checks that the refusal costs zero calls into the underlying mechanism, so no process is spawned.
+  TEA reads the expected count from `CONFORMANCE_OUTCOME_COUNTS` rather than stating it, so the next assertion the package adds fails this check until TEA answers it.
+  The conformance policy permits no environment key at all, which is both the honest allowlist for a fixture command that reads none and AD-35's default-deny base case.
+- Scores computed before and after this release are comparable on every input, with one exception that nothing recorded depends on.
+  Every replay-scored result is comparable across this change, and none of them moved.
+  `npm run test:probe-corpus` recomputes each probe's pre-flight, verdict, exit code, basis, qualification and strength vector from the stored evidence under 3.0.0, against `test/probes/expected-strength.json` unchanged by this release.
+  The probe and contract bytes moved, so `corpusDigest` and `contractDigest` move with them at scoring time, and nothing under `test/replay/` pins either or pins a scoring version, which is why no stored score moved.
+  No digest stored inside a probe or a contract moved either: those are taken over the harness implementation and its artifacts rather than over the files themselves.
+  The live `eval:test-review` numbers in `docs/explanation/eval-quality-roadmap.md` are comparable.
+  That harness states `--isolate` explicitly on every run, so its filesystem isolation never depended on `CI`, before or after.
+  A live `eval:preflight` or `eval:contract-strength` measurement of `tea-test-review` taken on a host where `CI` was set is not comparable with one taken after this release, because such a run isolated where this one does not.
+  No such measurement is recorded here: no workflow invokes any `eval:` script, the pre-flight cache under `test/eval-artifacts/` is not committed, and every contract-strength result this repository records is a finding rather than a figure.
+  So nothing recorded is invalidated; what changes is that the next contract-strength run and the next eval run measure the same command on every host.
+  `tea-fragment-selection-runner` and `tea-trace-runner` stop carrying `CI` as well, and nothing observable moves.
+  Only `cli/test-review.js` reads it, and `buildMinimalEnv` in `cli/lib/run-agent.js` never passes it to a vendor, so no measured run of those two could have read it.
+### Fixed
+- Two statements in `docs/explanation/eval-quality-roadmap.md` and `docs/explanation/eval-quality-command-adapter.md` that this upgrade falsified or that were already false.
+  The roadmap counted the published conformance suite at fifteen assertions, which is sixteen now.
+  The adapter page said `tea-test-review` permits neither `HOME` nor `USER`, so its live pre-flight must use an API key. That stopped being true one release earlier when its request shape moved to `vendorEnvironmentNames()`, and the same page recorded the correction two hundred lines further down while the first statement stayed, so the page contradicted itself.
+  The page's adoption inventory also listed `serializeArtifact` and `digestBytes` as unused, which the new corpus check falsifies, and its "done, and covered by npm test" section named none of the three checks this change adds.
+- `npm run test:ci-coverage` now reconciles the CLI job's shard partition against the suites it splits, and a deliberately dropped suite number proves it fires. That job runs `test/test-test-review-cli.js` across three parallel shards by suite number, and the three lists were hardcoded in `.github/workflows/quality.yaml` with nothing holding their union to the suites the file defines. A suite added to the file and to no list is enabled in none of the three: every shard prints "skipped", all three jobs go green, and the suite runs nowhere at all. The file's own 1-to-13 bound catches a number that is not a suite and cannot catch a suite that is in no list. The check reads the suite numbers off the file's own `suiteEnabled()` call sites and fails in both directions. Found incidentally while the positional assertion below was being fixed; it is the same defect one layer up, with a worse blast radius.
+- `test/test-test-review-cli.js` asserted `eval:all`'s invocation list by position, indexing `[0]` and `[1]` for the two suites that declare repetition overrides. Nothing held that order. It passed for three suites by accident of the order they were added in, and the first suite whose id sorted ahead of `fragment-selection` broke an assertion that is not about order at all. Both now look the suite up by id. Found while adding the routing suite, whose id does sort ahead of it.
+- Seventeen stale counts and claims in prose, every one of which this change would have made staler. `npm test` chains twenty-five checks and the README said twenty-one; `test/contracts/` holds twelve contracts and three files said ten; `test/probes/` holds thirty-five probes across twelve corpora and two files said thirty-one across ten; the replay corpus holds forty-four cases and three files said twenty-seven; TEA ships four commands and four harnesses and five statements said three; and the README, the adoption guide and the roadmap each said that every suite had been measured live, which stops being true the moment a suite is added. These are hand-maintained counts and they will go stale again. FR23 and Story 4.1 are what end that; this is not it.
 ## [1.26.0] - 2026-09-09
 ### Changed

package/README.md CHANGED Viewed

@@ -362,7 +362,7 @@ cli/                     # tea-test-review: the headless CI gate
 docs/                    # source of truth for the docs site
 website/                 # Astro + Starlight, consumes docs/ through a symlink
 tools/                   # validators, doc build, changelog stamping
-test/                    # quality gate suites and the three eval harnesses
+test/                    # quality gate suites and the four eval harnesses
 ```
 ## How TEA Keeps Itself Honest
@@ -392,7 +392,7 @@ A passing fragment-selection eval means the workflow loaded the right knowledge.
 ### Deterministic Checks
-`npm test` chains twenty-one checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Five of the twenty-one keep the rules, guidance, hook, eval data, and eval contracts aligned:
+`npm test` chains twenty-five checks. That count covers the whole chain: every entry in it is deterministic and credential-free, so the chain and its credential-free subset are the same list. `npm run test:ci-coverage` derives the count from `package.json` and prints it. Five of the twenty-five keep the rules, guidance, hook, eval data, and eval contracts aligned:
 - `test:criteria-fragments` fails when a registry row is neither mapped to a knowledge fragment nor declared a known gap. A rule the reviewer scores but no fragment teaches is a rule TEA punishes without ever having explained it. All 36 rows are currently mapped across 50 anchors. Because the declared-gap list is empty, the validator feeds itself a synthetic unmapped row on every run to prove that path still works.
 - `test:enforce-hook` fails when a new Absolute registry row appears in neither the hook's enforced list nor its deferred list. This prevents a rule from being added without an explicit write-time enforcement decision.
@@ -430,7 +430,7 @@ npm run eval:all -- --agent claude --preflight-only
 npm run eval:all -- --agent agy --preflight-only
 ```
-Output ending with `nothing measured` is expected in preflight mode. It means the static eval data is valid and, for every one of the three suites, the selected executable is on `PATH`, answers `--version`, and a built-in vendor has a credential. Some runners cannot expose session authentication to this probe, so a preflight pass does not guarantee that the later live call will authenticate. The flag intentionally exits before launching the agent. `npm run test:eval-schemas` runs each suite's preflight with a missing runner and with a present one, so a preflight that stopped probing the runner would fail `npm test`.
+Output ending with `nothing measured` is expected in preflight mode. It means the static eval data is valid and, for every one of the four suites, the selected executable is on `PATH`, answers `--version`, and a built-in vendor has a credential. Some runners cannot expose session authentication to this probe, so a preflight pass does not guarantee that the later live call will authenticate. The flag intentionally exits before launching the agent. `npm run test:eval-schemas` runs each suite's preflight with a missing runner and with a present one, so a preflight that stopped probing the runner would fail `npm test`.
 ### A La Carte Live Evals
@@ -542,16 +542,17 @@ npm run eval:fragment-selection -- \
 ### What Has to Pass
-Every row below is a declared gate. All three suites were measured live for the first time on 2026-09-08 against `claude`/`sonnet`, and `docs/explanation/eval-quality-roadmap.md` records what each one returned; no result artifact is committed, so quote that section rather than a number from memory. The live thresholds are the ones `test/evals/suite-manifest.json` declares, and `npm run test:eval-schemas` fails when a harness constant and the manifest disagree.
+Every row below is a declared gate. The three suites that existed then were measured live for the first time on 2026-09-08 against `claude`/`sonnet`, and `docs/explanation/eval-quality-roadmap.md` records what each one returned; `bmad-tea-routing` has been added since and has never been run live, so its row is a declared target and nothing more; no result artifact is committed, so quote that section rather than a number from memory. The live thresholds are the ones `test/evals/suite-manifest.json` declares, and `npm run test:eval-schemas` fails when a harness constant and the manifest disagree.
-| Eval                              | Declared threshold                                                                                                                                                                                                                                                                                                                                | Declared volume                        |
-| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- |
-| `npm run eval:all -- --agent ...` | All three live suites below meet their thresholds for the selected runner                                                                                                                                                                                                                                                                         | 48 selections, 3 reviews, and 4 traces |
-| Fragment selection                | At least 90% required-fragment recall, at most 10% forbidden-fragment selection, and stable choices across repeated cases                                                                                                                                                                                                                         | 24 cases twice: 48 calls               |
-| Test review                       | At least 70% overall recall, 100% CRITICAL recall, an 80% non-false-positive rate, score standard deviation no higher than 3, and a stable verdict                                                                                                                                                                                                | Three complete reviews                 |
-| Trace                             | At least 90% criterion-status accuracy and 90% evidence-citation precision; 100% on both discriminating criteria, the gate decision, the gate criteria, the coverage arithmetic, the oracle resolution, the rejected evidence, the waiver validity, and the live evidence; no clean-set false positive, no unstable case, and no fixture mutation | Two fixture sets twice: 4 calls        |
-| `npm run test:eval-data`          | Every case references valid workflow files and indexed fragments; required and forbidden sets do not overlap                                                                                                                                                                                                                                      | No agent calls                         |
-| `npm run test:eval-schemas`       | The suite manifest matches its schema, declares the thresholds the harnesses apply, and accounts for every TEA skill                                                                                                                                                                                                                              | No agent calls                         |
+| Eval                              | Declared threshold                                                                                                                                                                                                                                                                                                                                | Declared volume                                      |
+| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
+| `npm run eval:all -- --agent ...` | All four live suites below meet their thresholds for the selected runner                                                                                                                                                                                                                                                                          | 48 selections, 3 reviews, 4 traces, 36 routing calls |
+| Fragment selection                | At least 90% required-fragment recall, at most 10% forbidden-fragment selection, and stable choices across repeated cases                                                                                                                                                                                                                         | 24 cases twice: 48 calls                             |
+| Test review                       | At least 70% overall recall, 100% CRITICAL recall, an 80% non-false-positive rate, score standard deviation no higher than 3, and a stable verdict                                                                                                                                                                                                | Three complete reviews                               |
+| Trace                             | At least 90% criterion-status accuracy and 90% evidence-citation precision; 100% on both discriminating criteria, the gate decision, the gate criteria, the coverage arithmetic, the oracle resolution, the rejected evidence, the waiver validity, and the live evidence; no clean-set false positive, no unstable case, and no fixture mutation | Two fixture sets twice: 4 calls                      |
+| bmad-tea routing                  | At least 90% route accuracy, 80% reason-token recall, 87.5% scope fidelity, and 75% on both clarification and decline recall; zero confident routes on an intent nothing on the menu serves, zero on one whose menu items are genuinely close, zero clear intents left unrouted, and at most two intents whose answer changes between repetitions | 18 intents twice: 36 calls                           |
+| `npm run test:eval-data`          | Every case references valid workflow files and indexed fragments; required and forbidden sets do not overlap                                                                                                                                                                                                                                      | No agent calls                                       |
+| `npm run test:eval-schemas`       | The suite manifest matches its schema, declares the thresholds the harnesses apply, and accounts for every TEA skill                                                                                                                                                                                                                              | No agent calls                                       |
 Every declared repetition has to complete. A run that loses one to a timeout, a transport error, or an unparseable reply cannot measure variance or stability, so it exits `2` rather than reporting a lower score.
@@ -561,7 +562,7 @@ The non-false-positive rate is the share of reported findings that are not defin
 `test/evals/suite-manifest.json` registers every suite `eval:all` runs, with its skills, fixtures, ground truth, contract, thresholds, repetition count, CI tier, and the capabilities its runner needs. It also carries a `deferred` list: one entry per skill with no behavioral suite, naming the owner, the missing evidence, and the condition that retires the entry. `eval:all` reads the suite list from that file and refuses to run when a TEA skill appears in neither list.
-Add `--json <path>` to any of the three harnesses to write a result record alongside the console output:
+Add `--json <path>` to any of the four harnesses to write a result record alongside the console output:
 ```bash
 npm run eval:all -- --agent codex --json results/eval-all.json

package/cli/lib/parse-routing.js ADDED Viewed

@@ -0,0 +1,67 @@
+/**
+ * The routing answer out of a runner reply.
+ *
+ * Lives here rather than inside the harness for the reason cli/lib/parse-selection.js
+ * records: `cli/routing-runner.js` has to accept exactly the spellings
+ * `test/eval-bmad-tea-routing.js` and `test/test-eval-replay.js` score, and two
+ * copies of that rule would let a fenced-block reply be accepted by the runner
+ * and rejected by the scorer. This file owns how a reply is parsed and nothing
+ * about how one is scored; the pattern rules the oracles share live in
+ * `test/eval-bmad-tea-routing.js`.
+ *
+ * Returns null when nothing parseable came back, so "the reply was prose" never
+ * reports as "the agent declined", which is a real and very different answer.
+ * Everything past `action` is normalized rather than rejected: a reply that names
+ * an action and forgets to say why is a measurement of a bad routing answer, and
+ * a reply that is not a routing answer at all is a broken pipe. Collapsing the
+ * two would let a parser change show up as a quality score.
+ */
+'use strict';
+/** The three answers Step 8 of src/agents/bmad-tea/SKILL.md allows. */
+const ROUTING_ACTIONS = ['route', 'clarify', 'decline'];
+/** @returns {string|null} A trimmed non-empty string, or null for anything else. */
+function text(value) {
+  if (typeof value !== 'string') return null;
+  const trimmed = value.trim();
+  return trimmed.length === 0 ? null : trimmed;
+}
+/**
+ * @param {string} stdout - The runner reply, in full.
+ * @returns {{action: string, menuCode: string|null, workflow: string|null, scope: string|null, reason: string|null, question: string|null, missing: string|null}|null}
+ */
+function parseRouting(stdout) {
+  const source = String(stdout || '');
+  const candidates = [];
+  const fenced = /```(?:json)?\s*([\s\S]*?)```/g;
+  for (const match of source.matchAll(fenced)) candidates.push(match[1]);
+  const braced = /\{[\s\S]*"action"[\s\S]*\}/.exec(source);
+  if (braced) candidates.push(braced[0]);
+  candidates.push(source);
+  for (const candidate of candidates) {
+    let parsed;
+    try {
+      parsed = JSON.parse(candidate.trim());
+    } catch {
+      continue;
+    }
+    const action = text(parsed?.action)?.toLowerCase();
+    if (action === undefined || !ROUTING_ACTIONS.includes(action)) continue;
+    return {
+      action,
+      menuCode: text(parsed?.menuCode)?.toUpperCase() ?? null,
+      workflow: text(parsed?.workflow),
+      scope: text(parsed?.scope),
+      reason: text(parsed?.reason),
+      question: text(parsed?.question),
+      missing: text(parsed?.missing),
+    };
+  }
+  return null;
+}
+module.exports = { ROUTING_ACTIONS, parseRouting };

package/cli/routing-runner.js ADDED Viewed

@@ -0,0 +1,197 @@
+#!/usr/bin/env node
+/**
+ * tea-routing-runner — the command the `bmad-tea` routing contract names.
+ *
+ * `bmad-tea` is the agent a user meets first, and until this suite existed
+ * nothing measured the only thing it does: turn a sentence a person wrote into
+ * one item on its menu. The whole surface is one turn:
+ *
+ *   prompt on standard input -> {"action": "...", "menuCode": "...", ...} on standard output
+ *
+ * which is what the contract's operation declares: `stdin` carries the one
+ * required key, `descriptorChannel` is the stdout stream, and the response
+ * descriptor requires `action` and `reason` and permits the five optional keys
+ * beside them. The prompt stays the harness's to assemble, because it is built
+ * from the skill's own SKILL.md and customize.toml plus one corpus intent, and
+ * those belong to the eval corpus rather than to this command.
+ *
+ * The agent call is TEA's own (cli/lib/run-agent.js), so vendor argv, the
+ * minimal child environment, the model pin, and the credential shape are decided
+ * in the one place that already decides them for the other three commands. This
+ * file adds no vendor knowledge. It declares the one capability a routing
+ * decision needs, `read-only`: the answer is a JSON object on standard output
+ * and nothing about it requires a file, so claude runs with no write tool and
+ * codex under its read-only sandbox, which is what the suite declares in
+ * test/evals/suite-manifest.json.
+ *
+ * Two nested wall clocks are in play when this runs behind eval-quality's
+ * command-line adapter, and the inner one has to be the shorter of the two:
+ * --timeout-ms bounds the vendor call and reports a timeout as exit 5, while the
+ * adapter's own maxElapsedMs SIGKILLs this process and reports `budget-exhausted`
+ * with no exit code at all. A run that hits the outer bound first loses the
+ * classification the inner one would have produced. test/lib/probe-targets.js
+ * sets the outer bound a minute above the harness's own for that reason.
+ *
+ * Usage:
+ *   tea-routing-runner --agent codex < prompt.txt
+ *   tea-routing-runner --agent custom --agent-cmd ./my-runner < prompt.txt
+ *
+ * Exit codes: 0 a routing answer was produced, and the classes in
+ * cli/lib/runner-exit-codes.js otherwise. They are the exit codes because the
+ * caller is a probe: the adapter records an exit code as an observation and never
+ * as a fault, so the code is the only channel that survives the boundary intact.
+ * The table is shared with the other runner commands so all of them spell one
+ * class with one number.
+ */
+'use strict';
+const fs = require('node:fs');
+const { Command } = require('commander');
+const { AGENT_ADAPTERS } = require('./lib/agent-adapters');
+const { parseRouting } = require('./lib/parse-routing');
+const { runAgent } = require('./lib/run-agent');
+const { EXIT_CODES, classOfAgentError, failureClassForExit, vendorEnvironmentNames } = require('./lib/runner-exit-codes');
+/** The same default the other three commands declare, so this command changes no run that omits `--agent`. */
+const DEFAULT_AGENT = 'claude';
+/** Five minutes, matching RUN_TIMEOUT_MS in test/eval-bmad-tea-routing.js, which is the only caller that measures. */
+const DEFAULT_TIMEOUT_MS = 5 * 60_000;
+/** A routing decision is a reply, so the runner needs to write nothing; see the header. */
+const RUNNER_CAPABILITIES = ['read-only'];
+/**
+ * The request shape this command accepts, as the contract generator needs to
+ * declare it.
+ *
+ * Read by tools/generate-contracts.js rather than transcribed there, which is
+ * the rule test/contracts/README.md records for every declaration a command
+ * owns. The permitted environment names are the vendor variables plus HOME and
+ * USER; cli/lib/runner-exit-codes.js states why those two are load-bearing.
+ */
+const ROUTING_REQUEST_KEYS = {
+  argument: { required: [], permitted: [] },
+  option: {
+    required: ['agent'],
+    permitted: ['agent', 'agent-cmd', 'agent-arg', 'env-pass', 'model', 'timeout-ms'],
+  },
+  environment: { required: [], permitted: vendorEnvironmentNames() },
+  stdin: { required: ['prompt'], permitted: ['prompt'] },
+};
+/**
+ * The response shape this command prints, as the contract generator needs to
+ * declare it.
+ *
+ * `action` and `reason` are required because an answer with neither is not a
+ * routing decision. The other five are permitted and not required because which
+ * of them carries the answer depends on the action: a route names a menu code, a
+ * clarification names a question, a decline names what is missing, and forcing
+ * all three onto every reply would make an empty string the normal way to say
+ * "not applicable".
+ */
+const ROUTING_RESPONSE_KEYS = {
+  required: ['action', 'reason'],
+  permitted: ['action', 'menuCode', 'workflow', 'scope', 'reason', 'question', 'missing'],
+};
+function collect(value, previous) {
+  return [...previous, value];
+}
+/** Everything this process prints on stderr is diagnostic; stdout carries the routing answer alone. */
+function fail(failureClass, message) {
+  process.stderr.write(`tea-routing-runner: ${message}\n`);
+  process.exit(EXIT_CODES[failureClass]);
+}
+function readPrompt() {
+  try {
+    return fs.readFileSync(0, 'utf8');
+  } catch (error) {
+    fail('usage', `could not read the prompt from standard input: ${error.message}`);
+    return '';
+  }
+}
+function main(argv) {
+  const program = new Command();
+  program
+    .name('tea-routing-runner')
+    .description("Ask a headless agent which bmad-tea menu item a user's intent belongs to, and print the answer as JSON.")
+    .option('--agent <name>', `agent adapter (${Object.keys(AGENT_ADAPTERS).join('|')})`, DEFAULT_AGENT)
+    .option('--agent-cmd <path>', 'executable override for the selected adapter')
+    .option('--agent-arg <arg>', 'extra argument appended to the agent CLI argv (repeatable)', collect, [])
+    .option('--env-pass <NAME>', 'environment variable name allowed through to the agent (repeatable)', collect, [])
+    .option('--model <name>', 'model to pin for this run; defaults to the adapter default')
+    .option('--timeout-ms <n>', 'agent wall-clock timeout in milliseconds', String(DEFAULT_TIMEOUT_MS));
+  program.exitOverride();
+  try {
+    program.parse(argv);
+  } catch (error) {
+    // commander's own --help and --version exits are not usage errors.
+    if (error.exitCode === 0) return;
+    fail('usage', error.message);
+  }
+  const options = program.opts();
+  if (!Object.hasOwn(AGENT_ADAPTERS, options.agent)) {
+    fail('environment-configuration', `unknown agent "${options.agent}"; expected one of ${Object.keys(AGENT_ADAPTERS).join(', ')}`);
+  }
+  const timeout = Number.parseInt(options.timeoutMs, 10);
+  if (!Number.isInteger(timeout) || timeout <= 0) {
+    fail('usage', `--timeout-ms must be a positive integer; got ${JSON.stringify(options.timeoutMs)}`);
+  }
+  const prompt = readPrompt();
+  if (prompt.trim().length === 0) {
+    fail('usage', 'the prompt is empty; this command reads the complete prompt from standard input');
+  }
+  let stdout = '';
+  try {
+    ({ stdout } = runAgent(prompt, {
+      agent: options.agent,
+      agentCommand: options.agentCmd,
+      agentArgs: options.agentArg,
+      envPass: options.envPass,
+      model: options.model,
+      timeout,
+      cwd: process.cwd(),
+      capabilities: RUNNER_CAPABILITIES,
+    }));
+  } catch (error) {
+    fail(classOfAgentError(error), error.message);
+  }
+  const routing = parseRouting(stdout);
+  if (routing === null) {
+    fail('environment-parser', 'the agent reply carries no routing answer');
+  }
+  // Exactly the response descriptor's key set, and nothing beside it, for the
+  // reason tea-fragment-selection-runner gives: a diagnostic field added here
+  // would fail the contract it exists to satisfy.
+  process.stdout.write(`${JSON.stringify(routing)}\n`);
+}
+// Guarded so tools/generate-contracts.js can read the declarations above without
+// running a command, the same way it reads VERDICT_KEYS out of cli/test-review.js.
+if (require.main === module) {
+  main(process.argv);
+}
+module.exports = {
+  DEFAULT_AGENT,
+  DEFAULT_TIMEOUT_MS,
+  EXIT_CODES,
+  ROUTING_REQUEST_KEYS,
+  ROUTING_RESPONSE_KEYS,
+  RUNNER_CAPABILITIES,
+  classOfAgentError,
+  failureClassForExit,
+};

package/docs/explanation/eval-quality-adoption-guide.md CHANGED Viewed

@@ -26,7 +26,7 @@ TEA's state today, as `test/evals/suite-manifest.json` registers it:
 Two of the ten skills, `bmad-tea` and `bmad-teach-me-testing`, have no suite of any kind. They carry deferred entries in the manifest naming their owner, their missing evidence, and the condition that retires the entry, because a suite list with a skill quietly missing from it reads as coverage.
-All three suites were measured live for the first time on 2026-09-08 and all three were green. No result artifact is committed, so the numbers live in `docs/explanation/eval-quality-roadmap.md` and any claim about them needs a fresh measurement.
+The three suites that existed then were measured live for the first time on 2026-09-08 and all three were green. No result artifact is committed, so the numbers live in `docs/explanation/eval-quality-roadmap.md` and any claim about them needs a fresh measurement. The `bmad-tea-routing` suite was added afterwards and has never been run live, so it has a declared threshold and no measurement behind it.
 ## 1. What you need before you start

package/docs/explanation/eval-quality-command-adapter.md CHANGED Viewed

@@ -7,7 +7,7 @@ description: 'Why TEA probes every measured command through eval-quality, what t
 TEA measures a skill by running a command and reading what it wrote. Every harness owned that mechanism itself: its own `spawnSync`, argv, timeout, and `existsSync` plus `JSON.parse` over the file the run produced. Three copies, disagreeing about what a command may do, none capping output.
-`eval-quality` 1.0.0 ships that mechanism as `createCommandLineAdapter`, a real `EnvironmentProbePort` over a child process with 15 conformance outcomes behind it. TEA uses it and deletes what it invented. See the [roadmap](./eval-quality-roadmap.md) for the surrounding plan and `test/contracts/README.md` for the contracts.
+`eval-quality` 1.0.0 ships that mechanism as `createCommandLineAdapter`, a real `EnvironmentProbePort` over a child process with a conformance arm behind it, 16 outcomes on the 3.0.0 TEA now runs. TEA uses it and deletes what it invented. See the [roadmap](./eval-quality-roadmap.md) for the surrounding plan and `test/contracts/README.md` for the contracts.
 "Runner" has meant three processes here, which is most of why the boundary was unclear: the vendor agent doing the skill's work, the TEA command under evaluation, and `eval-all.js` orchestrating one child per suite. `eval-quality` has an opinion about the middle one only. It probes a declared interface and returns an observation, never launching an agent and holding no view on vendor routing or credential shape.
@@ -36,11 +36,11 @@ A contract names a logical executable and `ProbeRequest` enforces it: `executabl
 Two couplings the seam does not remove, both found by running it:
 - **`--json` and `--output` resolve against `--project-root`; the artifact map resolves against the policy `cwd`.** `test-review.contract.json`'s witness legs pass a bare `verdict.json` and name no project root, so its live pre-flight writes into whichever directory also has to satisfy its repository-relative `--files`. Those legs need an explicit project root and run-scoped artifact paths.
-- **The child environment is closed** to `PATH` plus what the request declares. Both vendors resolve a stored login through `HOME`, so the selection runner permits `HOME` and `USER`. `tea-test-review` does not, so its live pre-flight must use an API key.
+- **The child environment is closed** to `PATH` plus what the request declares, and from `eval-quality` 3.0.0 the authorization declares which of those keys a request may carry at all. All three commands permit the vendor names plus `HOME` and `USER`, because both vendors resolve a stored login through `HOME`; `tea-test-review` permits `CI` on top of them, which it reads to decide filesystem isolation. A key outside a command's list is refused before a process spawns, and `PATH` is on no list: `target` may name a bare command, so a declared `PATH` would choose which binary runs. The adapter supplies its own.
 ## Reaching more than one skill
-Ten contracts declare three logical executables, each a command TEA ships. When there were nine they declared two, and one did not exist: `tea-fragment-selection-runner`, named by eight of them, was fiction, which is worse than a declared gap: the contract compiles, pre-flight schedules a leg against it, and the gate stays green over a command nobody can run.
+Twelve contracts declare four logical executables, each a command TEA ships. When there were nine they declared two, and one did not exist: `tea-fragment-selection-runner`, named by eight of them, was fiction, which is worse than a declared gap: the contract compiles, pre-flight schedules a leg against it, and the gate stays green over a command nobody can run.
 `cli/fragment-selection-runner.js` is that command now. Its whole surface is the one turn those contracts declared: a prompt on standard input, `{"fragments": [...]}` on standard output. It builds no prompt, because a prompt belongs to the eval corpus, and knows no vendor, because it calls `runAgent`. `test/test-probe-targets.js` keeps the fiction from returning: every declared interface, executable, and subcommand path must be one the registry carries, and every registered command must be named by some contract.
@@ -64,22 +64,23 @@ The package has three stages: compile a contract, probe an environment, then sco
 TEA used the first two and none of the third. It uses all three now, and this section is the
 inventory, kept honest by being a list of what is still unused rather than a list of what is.
-| Published surface                                                                                          | TEA's use                                                                                                                                                                                                                                                                 |
-| ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `compile`, through the `eval-quality` binary                                                               | `npm run test:contracts` compiles all ten contracts against `test/contracts/expected-status.json`                                                                                                                                                                         |
-| `createCommandLineAdapter`, `nodeCommandMechanism`, `CommandTargetPolicy`                                  | `test/lib/probe-targets.js` maps three logical executables to three real commands                                                                                                                                                                                         |
-| `runPreflight`                                                                                             | `npm run eval:preflight` drives every contract's witness legs through the adapter for real                                                                                                                                                                                |
-| `runScore`                                                                                                 | `npm run test:probe-corpus` scores 31 probes across ten corpora; `npm run eval:contract-strength` scores them under a live pre-flight verdict                                                                                                                             |
-| `seal`                                                                                                     | one sealed evaluator brief per contract, written by the same script                                                                                                                                                                                                       |
-| `digestArtifact`                                                                                           | every artifact digest the run record and the isolation manifest declare                                                                                                                                                                                                   |
-| the published JSON Schemas                                                                                 | `test/lib/eval-quality-inputs.js` validates every artifact TEA builds or receives against `eval-quality/schemas/*`                                                                                                                                                        |
-| `preflightFromObservations`                                                                                | unused, and it cannot be used: a caller has to key its observations by leg identifier, and the leg identifiers are minted by the plan `runPreflight` builds. TEA holds a port for both halves, so the port entry point answers the same question with no ordering problem |
-| `validateLineageChain`, `INTERCHANGE_ARTIFACT_KEYS`, `serializeArtifact`, `digestBytes`, `digestComposite` | unused. Every TEA artifact is `revisionCount: 0` with a null parent, so there is no chain to validate, and the digest helpers TEA needs are the artifact one and its own file digest                                                                                      |
-| `eval-quality/conformance`                                                                                 | `npm run test:probe-conformance` runs the published command-line arm against a fixture command                                                                                                                                                                            |
+| Published surface                                                                 | TEA's use                                                                                                                                                                                                                                                                                               |
+| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `compile`, through the `eval-quality` binary                                      | `npm run test:contracts` compiles all twelve contracts against `test/contracts/expected-status.json`                                                                                                                                                                                                    |
+| `createCommandLineAdapter`, `nodeCommandMechanism`, `CommandTargetPolicy`         | `test/lib/probe-targets.js` maps four logical executables to four real commands                                                                                                                                                                                                                         |
+| `runPreflight`                                                                    | `npm run eval:preflight` drives every contract's witness legs through the adapter for real                                                                                                                                                                                                              |
+| `runScore`                                                                        | `npm run test:probe-corpus` scores 35 probes across twelve corpora; `npm run eval:contract-strength` scores them under a live pre-flight verdict                                                                                                                                                        |
+| `seal`                                                                            | one sealed evaluator brief per contract, written by the same script                                                                                                                                                                                                                                     |
+| `digestArtifact`                                                                  | every artifact digest the run record and the isolation manifest declare                                                                                                                                                                                                                                 |
+| the published JSON Schemas                                                        | `test/lib/eval-quality-inputs.js` validates every artifact TEA builds or receives against `eval-quality/schemas/*`                                                                                                                                                                                      |
+| `preflightFromObservations`                                                       | unused, and it cannot be used: a caller has to key its observations by leg identifier, and the leg identifiers are minted by the plan `runPreflight` builds. TEA holds a port for both halves, so the port entry point answers the same question with no ordering problem                               |
+| `validateLineageChain`, `INTERCHANGE_ARTIFACT_KEYS`, `digestComposite`            | unused. Every TEA artifact is `revisionCount: 0` with a null parent, so there is no chain to validate, and the digest helper TEA needs is the artifact one                                                                                                                                              |
+| `serializeArtifact`, `digestBytes`, `StructuralFailure`, the `./corpus/*` subpath | `npm run test:eval-quality-corpus` seals the package's own compile-and-seal example and compares the serialized bytes with the shipped brief, digests every corpus file against the digest `corpus/dev/index.json` records, and reads a refused compile by its failure class rather than by duck-typing |
+| `eval-quality/conformance`                                                        | `npm run test:probe-conformance` runs the published command-line arm against a fixture command                                                                                                                                                                                                          |
 ### What the corpus is
-`tools/generate-probes.js` writes 31 probes from the ground truth this repository already keeps.
+`tools/generate-probes.js` writes 35 probes from the ground truth this repository already keeps.
 Nine defect probes for `test-review`, one per planted registry row, each a controlled mutation whose
 target artifact, baseline-pass evidence and mutated-fail evidence are files on disk. Three for
 `trace`, one per criterion the seeded set deliberately leaves short. A clean control for every
@@ -126,8 +127,9 @@ been zero by construction whatever the reviewer did.
 `test-review.contract.json` grouped its nine plant oracles into three behaviors by severity, and each
 fragment-selection contract grouped its two oracles per case into one behavior. Both are one behavior
 per oracle now. The demand is unchanged: the same oracles, all required, at the same severities, with
-all ten contracts still compiling and all 611 oracle checks still agreeing with their scorers. Only
-the grouping moved.
+all contracts still compiling and all 611 oracle checks of the day still agreeing with their scorers.
+Only the grouping moved. Two routing contracts and their oracles have joined the corpus since, so the
+count is a record of that change rather than a running total.
 `trace.contract.json` is split the same way, twenty-six behaviors for twenty-six oracles, minted from
 the oracle identifiers so `B-00n` and `O-00n` are one thing. Its authored groups still carry the
@@ -379,11 +381,14 @@ for the same correctness.
 The adapter closes the child environment to `PATH` plus what the request declares, and
 `cli/test-review.js` resolves a stored login through `HOME`, so a machine with a keychain login could
 not run that contract's own pre-flight. The shape is read from `vendorEnvironmentNames()` now, the
-same source the other two commands use.
+same source the other two commands use, and on 3.0.0 the authorization permits exactly that list,
+asserted equal in both directions by `npm run test:probe-targets`.
 ## Done, and owed
-Done, and covered by `npm test`: the registry, policy, port, and fault-to-failure-class mapping in `test/lib/probe-targets.js`; the runner, whose request shape and default agent `tools/generate-contracts.js` reads rather than transcribes; and `npm run test:probe-targets`, which drives all three real commands through the real adapter against checked-in fixtures with a stub vendor. It asserts default-deny, the observation shape, artifact read-back, an absent artifact, a real budget kill classified as a timeout, and contract-to-registry agreement both ways, with no model call and no credential.
+Done, and covered by `npm test`: the registry, policy, port, and fault-to-failure-class mapping in `test/lib/probe-targets.js`; the runner, whose request shape and default agent `tools/generate-contracts.js` reads rather than transcribes; and `npm run test:probe-targets`, which drives every real command through the real adapter against checked-in fixtures with a stub vendor. It asserts default-deny, the observation shape, artifact read-back, an absent artifact, a real budget kill classified as a timeout, and contract-to-registry agreement both ways, with no model call and no credential.
+Three more checks joined that list with the move to 3.0.0. `npm run test:probe-targets` now also holds each contract's declared environment keys equal to its authorization's permitted keys in both directions, and asserts that an unpermitted key is denied before a process spawns and that a malformed key fails at the port parse. `npm run test:eval-quality-corpus` compiles the package's own published corpus, which is the one check here that feeds the package nothing of TEA's. `npm run test:port-totality` holds TEA's branches total over both probe unions and all six published conformance arms.
 Also done: `eval-test-review.js`'s `runReview` and `promptDigestFromCli` drive `tea-test-review` through the port, and `eval-fragment-selection.js` runs `cli/fragment-selection-runner.js` rather than calling `runAgent` in process, so its live path and its contract describe one thing. Both were verified end to end against their stub agents, with no model call: the review harness reads a real verdict artifact back as JSON and scores it, and the selection harness scores a real reply off stdout.