bmad-method-test-architecture-enterprise 1.25.1 → 1.25.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -31,7 +31,7 @@
31
31
  "name": "bmad-method-test-architecture-enterprise",
32
32
  "source": "./",
33
33
  "description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
34
- "version": "1.25.1",
34
+ "version": "1.25.2",
35
35
  "author": {
36
36
  "name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
37
37
  },
package/CHANGELOG.md CHANGED
@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
7
7
 
8
8
  ## [Unreleased]
9
9
 
10
+ ## [1.25.2] - 2026-09-09
11
+
12
+ ### Fixed
13
+
14
+ - `trace`'s clean control scored FAIL at exit 2 because five of its twenty-six oracles abstained. `O-009`, `O-010`, `O-011`, `O-013` and `O-014` are `for-any` quantifiers over the seeded set's summary, and `P-004`'s record carried one observation that both plan steps selected, so those five quantified over the clean summary's empty or absent collections. Two changes close it and they work only together: each plan step binds `stdin.prompt` as a literal of that set's own prompt, and the clean control's record carries both fixture sets' runs. Measured on the stored replay, `P-004` moves from FAIL at exit 2 with five abstentions to CONCERNS at exit 0 with all twenty-six oracles resolving `passed-clean-control`; four unsatisfied AD-20 coverage rules hold it at CONCERNS and are separate work. Either half alone is worse than neither, and both were run before this was accepted: two observations under the old matcher bindings leave every observation satisfying both steps, so `exactly-one` reports selector ambiguity on all twenty-six oracles at exit 3; literals over one observation select nothing, so all twenty-six resolve `unreached` and the run reports CONCERNS at exit 0 having examined no evidence, which is a silent green worse than the FAIL it replaces. A literal is compared with `deepEquals`, so the contract's literal and the record's prompt are both `buildPrompt` from `test/eval-trace.js`, and `traceEvidence` throws when the contract on disk binds any other bytes. The record used to carry the placeholder `The prompt the trace harness assembles for <case>.`, which is a prompt no run has ever sent. The control still catches a false positive: a clean summary damaged to derive FAIL with an invented blocker, reported as a defect finding, resolves `O-016` to `false-positive` and the probe to FAIL at exit 2. `test/contracts/trace.contract.json` grows 3.6 kilobytes, the size of the two prompts; the eight fragment-selection contracts keep the matcher binding, where the same fix costs 42 to 143 kilobytes each and roughly doubles every file.
15
+ - A stale reason in `test/contracts/README.md`, surfaced by the contradiction the entry above created. The paragraph on `trace.contract.json`'s witness said its two plan steps share one prompt on purpose because `test/eval-trace.js` names no fact about either fixture set in it, and it said the witness legs are runnable only against a staged workspace of the seeded set. Both stopped being true one release earlier: each set declares its own `projectRoot`, the whole prompt is written against it, and both witness legs stage the clean set. The `allow_gate` reasoning, the differential-and-invariance argument, and the pointer to `testData.setup` all survive and are kept. `testData.setup` in the contract itself was already correct and needed no change.
16
+ - A stale illustration in `docs/explanation/eval-quality-command-adapter.md`. The passage on AD-4 said `trace`'s five `for-any` oracles over the seeded export "still abstain on a clean run", which stopped being true once each plan step selected its own set's run. AD-4's rule is unchanged and the sentence now dates the abstention it describes.
17
+
10
18
  ## [1.25.1] - 2026-09-09
11
19
 
12
20
  ### Changed
@@ -249,8 +249,9 @@ spelling to have committed to.
249
249
 
250
250
  One limit is worth knowing before writing a new oracle. Every quantifier still abstains over an empty
251
251
  collection, which is AD-4's whole purpose and is why `trace`'s five `for-any` oracles over the seeded
252
- export still abstain on a clean run: they ask whether some element exists, and an empty collection is
253
- an honest "nothing was checked". `deep-equality` against a literal `[]` also still abstains, so the
252
+ export abstained on the clean control for as long as they were resolved against the clean set's
253
+ summary: they ask whether some element exists, and an empty collection is an honest "nothing was
254
+ checked". `deep-equality` against a literal `[]` also still abstains, so the
254
255
  two spellings of "this collection is empty" disagree. eval-quality records that disagreement in AD-4
255
256
  rather than hiding it. The bare `count-tolerance` assertion is the one to write.
256
257
 
@@ -273,33 +274,47 @@ Three findings, all measured, none of them tuned away.
273
274
  before. `test-review` leaves `whole-body`, `malformed-input` and `state-change-read-back`
274
275
  unsatisfied; every fragment-selection contract leaves `malformed-input` unsatisfied. Each scores the
275
276
  run down to CONCERNS without blocking it, which is exactly the weight AD-20 gives a coverage gap.
276
- - **`trace`'s clean control scores FAIL, and what remains of it is one thing.** Measured on the
277
- stored replay, five of its twenty-six oracles abstain on `P-004`: `O-009`, `O-010`, `O-011`,
277
+ - **`trace`'s clean control scored FAIL. Closed, and the two halves that closed it work only
278
+ together.** Five of its twenty-six oracles abstained on `P-004`: `O-009`, `O-010`, `O-011`,
278
279
  `O-013` and `O-014`, every one a `for-any` quantifier over the seeded export, and every one named
279
280
  in the artifact's `verdictBasis`. It was seven. `O-023` and `O-024`, the two asserting that a
280
281
  collection is empty, moved to `passed-clean-control` when `eval-quality` 1.4.0 gave that claim a
281
282
  spelling.
282
283
 
283
- The five abstain because the record carries only the clean set's observation and both plan steps
284
- select it. The reason recorded here used to be that no request shape names a fixture set; that is
285
- no longer true, since each set is staged under its own project root and the prompt is written
286
- against it. What is left is the binding: `stdin.prompt` is bound `{matcher: 'any'}`, so one
287
- observation is selected by both steps, which is the same limit as the bullet below. The trace
288
- prompt is 1.8 kilobytes rather than fragment selection's 28, so the size argument against a literal
289
- does not apply here either. What a literal binding needs beside it is a record carrying both sets'
290
- observations, and whether one probe's record may carry the run of a set it did not seed is a
291
- question about what a clean control means rather than a mechanical change, so it is not made here.
292
- Neither half is a case for giving the clean set a member so an oracle has something to read: that
293
- would mutate a control to make the instrument work.
284
+ The five abstained because the record carried one observation and both plan steps selected it, so
285
+ the seeded set's oracles quantified over the clean summary's empty or absent collections. Each
286
+ step binds its own set's prompt as a literal now, and the clean control's record carries both
287
+ sets' runs, so the seeded step selects the seeded run and the clean step selects the clean one.
288
+ Measured on the stored replay, all twenty-six oracles resolve `passed-clean-control` and `P-004`
289
+ moves from FAIL at exit 2 to CONCERNS at exit 0. What holds it at CONCERNS is the four unsatisfied
290
+ AD-20 coverage rules the bullet above describes, which is separate work.
291
+
292
+ Either half on its own is worse than neither, which is why they landed together. Two observations
293
+ under the old matcher bindings leave every observation satisfying both steps, and `exactly-one`
294
+ then reports selector ambiguity on all twenty-six oracles at exit 3. Literals over one observation
295
+ select nothing: all twenty-six resolve `unreached` and the run reports CONCERNS at exit 0 having
296
+ examined no evidence at all, which is a silent green and is worse than the FAIL it replaces. A
297
+ literal is compared with `deepEquals`, so the contract's literal and the record's prompt are both
298
+ `buildPrompt` from `test/eval-trace.js`, and `traceEvidence` in `test/lib/probe-scoring.js` throws
299
+ when the contract on disk binds any other bytes. Both states above were run before the change was
300
+ accepted, so the failure mode is one somebody has seen.
294
301
 
295
302
  - **A plan cannot tell two steps apart when both bind their inputs by matcher.** Each
296
303
  fragment-selection contract declares one plan step per case, distinguished only by the prompt, and
297
- the prompt is bound `{matcher: 'any'}` because the alternative is a 28-kilobyte literal per step. A
298
- record carrying one observation is therefore selected by every step, and the oracles of the other
299
- cases resolve against evidence that is not theirs. The designated oracle still votes correctly, so
300
- the strength vector is unaffected, and the surrounding outcome rows are noise. This is the limit
301
- `test/contracts/README.md` records as "a plan cannot declare that two steps must receive different
302
- inputs", with its consequence now measured.
304
+ the prompt is bound `{matcher: 'any'}`. A record carrying one observation is therefore selected by
305
+ every step, and the oracles of the other cases resolve against evidence that is not theirs. The
306
+ designated oracle still votes correctly, so the strength vector is unaffected, and the surrounding
307
+ outcome rows are noise. This is the limit `test/contracts/README.md` records as "a plan cannot
308
+ declare that two steps must receive different inputs", with its consequence now measured.
309
+
310
+ The trace contract closed this with literals; the eight fragment-selection contracts keep the
311
+ matcher, and the reason is size. Their prompts carry the workflow's knowledge-loading rules and its
312
+ whole fragment index, 21 to 43 kilobytes per step, so binding each step's prompt as a literal adds
313
+ between 42 and 143 kilobytes to a contract and roughly doubles every one of the eight files: 1.7x
314
+ for `bmad-testarch-nfr`, 2.7x for `bmad-testarch-automate`, measured from the prompts those
315
+ contracts already carry on their witness legs. Trace pays 3.6 kilobytes on 108 for the same fix,
316
+ because its two prompts are 1.8 kilobytes each, the size of the prompt the file already carries on
317
+ each of its two witness legs.
303
318
 
304
319
  ### What the live pre-flight measured
305
320
 
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "$schema": "https://json.schemastore.org/package.json",
3
3
  "name": "bmad-method-test-architecture-enterprise",
4
- "version": "1.25.1",
4
+ "version": "1.25.2",
5
5
  "description": "Master Test Architect for quality strategy, test automation, and release gates",
6
6
  "keywords": [
7
7
  "bmad",
@@ -116,19 +116,21 @@ input-insensitive by design, and the contract now states that as its own claim i
116
116
  nothing.
117
117
 
118
118
  `trace.contract.json`, the tenth, states its witness over standard input on one prompt value,
119
- `allow_gate`. Its two plan steps share one prompt on purpose: `test/eval-trace.js` names no fact about
120
- either fixture set in it, so the seeded and clean summaries differ because of the staged workspace and
121
- never because of the prompt. A differential between the two steps would attribute the difference to
122
- the wrong input, and an invariance claim would be false. `allow_gate` is the one prompt value the
123
- ground truth establishes an effect for, through `skillRuleCitations.gateEligibility`: step-05
124
- evaluates a gate only when it is true and writes `gate_basis` as `none` otherwise, so two prompts
125
- differing in that value, in one staged workspace, produce two `gate_basis` values. That is a true and
126
- checkable claim that the command reads its standard input, and it is the claim the witness makes. The
127
- legs are runnable only against a staged workspace of the seeded set, which is the coupling
128
- `docs/explanation/eval-quality-command-adapter.md` records for every artifact-writing command. The
129
- contract itself states this reasoning in `testData.setup`, beside the sentence about the shared
130
- prompt, because `SensitivityWitness` is a strict object with no prose field of its own; do not look for
131
- it on the witness.
119
+ `allow_gate`. Its two plan steps send two prompts, each written against its own fixture set's project
120
+ root, and each step binds that prompt as its `stdin.prompt` literal. The summaries the two sets
121
+ produce differ because of the files staged under those roots, so a differential across the two sets
122
+ would attribute to the prompt a difference the staged workspace produced, and an invariance claim over
123
+ them would be false. `allow_gate` is the one prompt value the ground truth establishes an effect for,
124
+ through `skillRuleCitations.gateEligibility`: step-05 evaluates a gate only when it is true and writes
125
+ `gate_basis` as `none` otherwise, so two prompts differing in that value, over one staged fixture set,
126
+ produce two `gate_basis` values. That is a true and checkable claim that the command reads its
127
+ standard input, and it is the claim the witness makes. Both its legs stage the clean set, because AD-10
128
+ reads every other leg of an operation as a clean leg, and they are runnable only against that staged
129
+ workspace, which is the coupling `docs/explanation/eval-quality-command-adapter.md` records for every
130
+ artifact-writing command. The contract itself states this reasoning in `testData.setup`, because
131
+ `SensitivityWitness` is a strict object with no prose field of its own; do not look for it on the
132
+ witness. The section "A plan cannot declare that two steps must receive different inputs" below
133
+ records what those literals cost and what they bought.
132
134
 
133
135
  ## What the operator vocabulary cannot say
134
136
 
@@ -145,6 +147,18 @@ cases differ only in the prompt sent to the runner, and that is the property tha
145
147
  mean anything. `sensitivityWitness` expresses a two-leg version of this inside one operation and
146
148
  nothing expresses it across plan steps.
147
149
 
150
+ Answered for `trace`, at a price the eight fragment-selection contracts cannot pay. Binding each
151
+ step's `stdin.prompt` as a literal of that step's own prompt is what tells the two steps apart: the
152
+ seeded step selects the seeded run, the clean step selects the clean one, and `trace`'s clean control
153
+ moved from FAIL at exit 2 with five oracles abstaining to CONCERNS at exit 0 with all twenty-six
154
+ resolving `passed-clean-control`. It costs 3.6 kilobytes on `trace.contract.json` and 42 to 143
155
+ kilobytes on a fragment-selection contract, which roughly doubles each of those eight files, so they
156
+ keep the matcher. A literal is compared with `deepEquals`, so both sides of one have to come from a
157
+ single function: `tools/generate-contracts.js` and `test/lib/probe-scoring.js` both call `buildPrompt`
158
+ from `test/eval-trace.js`, and the record builder throws when the contract on disk carries any other
159
+ bytes. A prompt restated in either place would select nothing and every oracle would resolve
160
+ `unreached`, which reads as a clean run at exit 0.
161
+
148
162
  **An empty collection is no evidence.** AD-4 resolves a quantifier over an
149
163
  empty collection to `insufficient-evidence` with an `empty-collection` introduction condition. The
150
164
  `test-review` harness reads a verdict whose findings array is empty as a reviewer that named nothing,
@@ -2701,7 +2701,7 @@
2701
2701
  "environment": null,
2702
2702
  "stdin": {
2703
2703
  "prompt": {
2704
- "matcher": "any"
2704
+ "literal": "You are running the TEA workflow `bmad-testarch-trace` against the project in `tenant-data-export/`.\n\nThe workflow is in `skill/`. Read `skill/instructions.md` first, then execute every step file it\nnames in order, in full, without skipping or reordering. The step files are under `skill/steps-c/`.\n\n----- run configuration -----\nResolve the workflow placeholders to these values:\n\n- `{project-root}`: `tenant-data-export`\n- `{config_source}`: `tenant-data-export/_bmad/tea/config.yaml`\n- `{test_artifacts}`: `tenant-data-export/test-artifacts`\n- `{test_dir}`: `tenant-data-export/tests`\n- `{source_dir}`: `tenant-data-export/src`\n- `{skill-root}`: `skill`\n- `gate_type`: `epic`\n- `decision_mode`: `deterministic`\n- `collection_mode`: `contract_static`\n- `allow_gate`: `true`\n- `coverage_basis`: `auto`\n- `summary_confidence`: `auto`\n- `coverage_levels`: `e2e,api,component,unit,live`\n\nThe trace target is the epic under `tenant-data-export/docs/epics/`. Resolve the coverage oracle from it the\nway step-01 says to.\n\n----- what to produce -----\nWrite both deliverables the workflow declares:\n\n- `tenant-data-export/test-artifacts/traceability-matrix.md`, from `skill/trace-template.md`, carrying the\n detailed mapping with one section per criterion, each stating its coverage status and the tests\n that establish it as `file:line`.\n- `tenant-data-export/test-artifacts/e2e-trace-summary.json` at schema_version 0.3.0, exactly as\n `skill/steps-c/step-05-gate-decision.md` section 3b defines it.\n\nDo not add, edit, or delete any file under `tenant-data-export/docs/`, `tenant-data-export/src/`, or `tenant-data-export/tests/`.\nThis workflow does not generate tests.\n\nWhen you are done, print one line naming the two files you wrote. Nothing else you print is read."
2705
2705
  }
2706
2706
  }
2707
2707
  }
@@ -2721,7 +2721,7 @@
2721
2721
  "environment": null,
2722
2722
  "stdin": {
2723
2723
  "prompt": {
2724
- "matcher": "any"
2724
+ "literal": "You are running the TEA workflow `bmad-testarch-trace` against the project in `api-token-lifecycle/`.\n\nThe workflow is in `skill/`. Read `skill/instructions.md` first, then execute every step file it\nnames in order, in full, without skipping or reordering. The step files are under `skill/steps-c/`.\n\n----- run configuration -----\nResolve the workflow placeholders to these values:\n\n- `{project-root}`: `api-token-lifecycle`\n- `{config_source}`: `api-token-lifecycle/_bmad/tea/config.yaml`\n- `{test_artifacts}`: `api-token-lifecycle/test-artifacts`\n- `{test_dir}`: `api-token-lifecycle/tests`\n- `{source_dir}`: `api-token-lifecycle/src`\n- `{skill-root}`: `skill`\n- `gate_type`: `epic`\n- `decision_mode`: `deterministic`\n- `collection_mode`: `contract_static`\n- `allow_gate`: `true`\n- `coverage_basis`: `auto`\n- `summary_confidence`: `auto`\n- `coverage_levels`: `e2e,api,component,unit,live`\n\nThe trace target is the epic under `api-token-lifecycle/docs/epics/`. Resolve the coverage oracle from it the\nway step-01 says to.\n\n----- what to produce -----\nWrite both deliverables the workflow declares:\n\n- `api-token-lifecycle/test-artifacts/traceability-matrix.md`, from `skill/trace-template.md`, carrying the\n detailed mapping with one section per criterion, each stating its coverage status and the tests\n that establish it as `file:line`.\n- `api-token-lifecycle/test-artifacts/e2e-trace-summary.json` at schema_version 0.3.0, exactly as\n `skill/steps-c/step-05-gate-decision.md` section 3b defines it.\n\nDo not add, edit, or delete any file under `api-token-lifecycle/docs/`, `api-token-lifecycle/src/`, or `api-token-lifecycle/tests/`.\nThis workflow does not generate tests.\n\nWhen you are done, print one line naming the two files you wrote. Nothing else you print is read."
2725
2725
  }
2726
2726
  }
2727
2727
  }
@@ -41,6 +41,11 @@ const fs = require('node:fs');
41
41
  const path = require('node:path');
42
42
 
43
43
  const { digest } = require('./eval-record');
44
+ // The prompt a trace observation carries is the prompt the harness assembles, and
45
+ // tools/generate-contracts.js binds the same function's output as each trace plan
46
+ // step's stdin literal. One function on both sides is the whole guard; `legs` in
47
+ // traceEvidence says what a restated prompt would cost.
48
+ const { buildPrompt: buildTracePrompt } = require('../eval-trace');
44
49
  const {
45
50
  evaluatorConfiguration,
46
51
  isolationManifest,
@@ -276,12 +281,42 @@ function traceArtifacts(caseId) {
276
281
  function traceEvidence(contract) {
277
282
  const [seededStep, cleanStep] = contract.interactionPlan;
278
283
  const groundTruth = readJson(path.join(PROJECT_ROOT, 'test', 'fixtures', 'trace-eval', 'ground-truth.json'));
284
+
285
+ /**
286
+ * One entry per fixture set: the plan step that selects it, the stored run its
287
+ * project root names, and the prompt the harness assembles for it.
288
+ *
289
+ * `buildTracePrompt` is test/eval-trace.js's own `buildPrompt`, which is also
290
+ * what tools/generate-contracts.js calls to build each step's stdin literal. A
291
+ * literal is compared with `deepEquals`, so a prompt restated here in any other
292
+ * form would select nothing, all twenty-six oracles would resolve `unreached`,
293
+ * and the run would report clean at exit 0 having examined no evidence. The
294
+ * equality below is the tripwire: the prompt this record will carry is checked
295
+ * against the literal the contract on disk binds, so a divergence fails the run
296
+ * where it would otherwise pass it silently.
297
+ */
298
+ const legs = groundTruth.fixtureSets.map((set) => {
299
+ const seeded = set.id.startsWith('seeded');
300
+ const step = seeded ? seededStep : cleanStep;
301
+ const prompt = buildTracePrompt(set);
302
+ if (step.inputBinding.stdin?.prompt?.literal !== prompt) {
303
+ throw new Error(
304
+ `${step.stepId} binds a stdin literal that is not the prompt the harness assembles for ${set.id}; run node tools/generate-contracts.js`,
305
+ );
306
+ }
307
+ return {
308
+ caseId: seeded ? 'seeded-correct-run' : 'clean-correct-run',
309
+ observationId: seeded ? 'trace-seeded-run' : 'trace-clean-run',
310
+ projectRoot: set.projectRoot,
311
+ step,
312
+ prompt,
313
+ };
314
+ });
315
+
279
316
  // The stored run each fixture set's project root names. A trace prompt is written
280
317
  // against one project root, and that root is the only thing in the request that
281
318
  // says which set the leg is asking for, so it is what the port stages against.
282
- const caseByProjectRoot = new Map(
283
- groundTruth.fixtureSets.map((set) => [set.projectRoot, set.id.startsWith('seeded') ? 'seeded-correct-run' : 'clean-correct-run']),
284
- );
319
+ const caseByProjectRoot = new Map(legs.map((leg) => [leg.projectRoot, leg.caseId]));
285
320
 
286
321
  return {
287
322
  /**
@@ -313,41 +348,61 @@ function traceEvidence(contract) {
313
348
  },
314
349
  recordInputs(probe) {
315
350
  const clean = probe.expectedClean;
316
- const caseId = clean ? 'clean-correct-run' : 'seeded-correct-run';
317
- // One observation, and the other step's oracles then have nothing of their
318
- // own to read. Both steps declare the same operation and send the same
319
- // prompt, because what makes a trace run the seeded set or the clean set is
320
- // the staged workspace and no request shape names one, so a record carrying
321
- // both runs is ambiguous under `exactly-one` and a record carrying one
322
- // leaves five seeded-set oracles quantifying over an absent collection. The
323
- // clean control's FAIL is that, plus two oracles asserting an empty array.
324
- const step = clean ? cleanStep : seededStep;
325
- const observationId = clean ? 'trace-clean-run' : 'trace-seeded-run';
326
- const observations = [
351
+ // The clean control's record carries both runs. Each step binds its own set's
352
+ // prompt as a literal, so the seeded step selects the seeded observation and
353
+ // the clean step selects the clean one, and the oracles of each quantify over
354
+ // the summary their own set produced. With one observation both steps selected
355
+ // it, so five of the seeded step's `for-any` oracles quantified over the clean
356
+ // summary's empty or absent collections and abstained, which was the control's
357
+ // FAIL at exit 2. Two observations under a matcher binding are worse: every
358
+ // observation satisfies both steps, and `exactly-one` then reports selector
359
+ // ambiguity on all twenty-six oracles. The literals and the second observation
360
+ // work only together.
361
+ //
362
+ // A defect probe carries its seeded run alone. AD-9's qualification gate
363
+ // resolves every one of its oracles before a selection is read, so a second
364
+ // observation would add evidence nothing reaches.
365
+ //
366
+ // Two things this record still states loosely, and nothing reads either today.
367
+ // `conditionArm` stays `clean-correct-run` for a record that now carries two
368
+ // runs, and P-004's `baselinePassEvidence` in test/probes/trace.probes.json
369
+ // names the clean summary's digest alone.
370
+ const selected = clean ? legs : legs.filter((leg) => leg.caseId === 'seeded-correct-run');
371
+ const observations = selected.map((leg, index) =>
327
372
  recordObservation({
328
- observationId,
329
- sequence: 1,
330
- operationId: step.operationId,
331
- callInputs: { option: { agent: 'claude' }, stdin: { prompt: `The prompt the trace harness assembles for ${caseId}.` } },
373
+ observationId: leg.observationId,
374
+ sequence: index + 1,
375
+ operationId: leg.step.operationId,
376
+ callInputs: { option: { agent: 'claude' }, stdin: { prompt: leg.prompt } },
332
377
  stdout: { kind: 'text', value: '' },
333
378
  stderr: { kind: 'text', value: '' },
334
379
  exitCode: 0,
335
- artifacts: traceArtifacts(caseId),
380
+ artifacts: traceArtifacts(leg.caseId),
336
381
  }),
337
- ];
382
+ );
383
+ // Every trace oracle reads one step's interaction, so its disposition cites
384
+ // that step's observation. A record carrying one run has nothing of the other
385
+ // step's to cite, and a `held` disposition citing nothing is scored as an
386
+ // unsupported claim, so those oracles cite the run the record does carry.
387
+ const observationIdByStep = new Map(selected.map((leg) => [leg.step.stepId, leg.observationId]));
388
+ const stepOf = (pointer) => String(pointer).split('/')[2];
389
+ const citedObservationId = (oracle) => {
390
+ const target = (oracle.direction?.evidenceTargets ?? []).find((pointer) => observationIdByStep.has(stepOf(pointer)));
391
+ return target === undefined ? observations[0].observationId : observationIdByStep.get(stepOf(target));
392
+ };
338
393
  return {
339
394
  observations,
340
395
  findings: [],
341
- // Both stored runs are recorded as scoring every check they were given,
342
- // so every oracle held on the evidence the harness read.
396
+ // Every stored run in this record is recorded as scoring every check it was
397
+ // given, so every oracle held on the evidence the harness read.
343
398
  oracleDispositions: contract.oracles.map((oracle) => ({
344
399
  oracleId: oracle.id,
345
400
  disposition: 'held',
346
- observationIds: [observationId],
401
+ observationIds: [citedObservationId(oracle)],
347
402
  note: null,
348
403
  })),
349
404
  evaluatorRecommendation: 'PASS',
350
- conditionArm: caseId,
405
+ conditionArm: clean ? 'clean-correct-run' : 'seeded-correct-run',
351
406
  };
352
407
  },
353
408
  };
@@ -308,9 +308,14 @@
308
308
  "expectedClean": true,
309
309
  "behaviorId": "B-016",
310
310
  "preflight": "passed",
311
- "verdict": "FAIL",
312
- "exitCode": 2,
313
- "basis": ["oracle resolved abstained at or above the severity floor"],
311
+ "verdict": "CONCERNS",
312
+ "exitCode": 0,
313
+ "basis": [
314
+ "coverage gap malformed-input unsatisfied at or above the severity floor",
315
+ "coverage gap state-change-read-back unsatisfied at or above the severity floor",
316
+ "coverage gap success-indicator-separation unsatisfied at or above the severity floor",
317
+ "coverage gap whole-body unsatisfied at or above the severity floor"
318
+ ],
314
319
  "qualification": null,
315
320
  "strength": {
316
321
  "defect": null,
@@ -22,7 +22,8 @@
22
22
  * name for that contract's sourceSpecDigest,
23
23
  * - cli/trace-runner.js's TRACE_REQUEST_KEYS and DEFAULT_AGENT for that
24
24
  * contract's request shape and witness legs, and test/eval-trace.js's
25
- * buildPrompt for the two prompts the witness legs send, and
25
+ * buildPrompt for the two prompts the witness legs send and for the literal
26
+ * each of its two plan steps binds on standard input, and
26
27
  * - the trace workflow's step-05 for the key set of the summary the run writes.
27
28
  *
28
29
  * The prose that is genuinely authored (an oracle's commentary, a behavior's lead
@@ -69,8 +70,8 @@ const { VERDICT_KEYS, DEFAULT_AGENT } = require('../cli/test-review');
69
70
  // owns its own request shape and its own default agent, so both are read from it.
70
71
  const { SELECTION_REQUEST_KEYS, DEFAULT_AGENT: SELECTION_DEFAULT_AGENT } = require('../cli/fragment-selection-runner');
71
72
  // And for the trace command: its request shape and default agent are its own, and
72
- // the prompt its witness legs send is the harness's, because the harness is the
73
- // only thing that assembles one.
73
+ // the prompt its witness legs send and the literal each of its plan steps binds
74
+ // are the harness's, because the harness is the only thing that assembles one.
74
75
  const { TRACE_REQUEST_KEYS, DEFAULT_AGENT: TRACE_DEFAULT_AGENT, EXIT_CODES: TRACE_EXIT_CODES } = require('../cli/trace-runner');
75
76
  const { vendorEnvironmentNames } = require('../cli/lib/runner-exit-codes');
76
77
  const {
@@ -2278,20 +2279,31 @@ function buildTraceContract() {
2278
2279
  operationId: TRACE_OPERATION,
2279
2280
  after: null,
2280
2281
  cardinality: 'exactly-one',
2281
- // The agent is bound as `any` rather than pinned: which vendor answered is
2282
- // the runner record's to state. The prompt is bound as `any` for the reason
2283
- // the fragment-selection contracts give: its bytes are evidence, and the
2284
- // sealed record carries their digest.
2282
+ // The agent is bound as `any`: which vendor answered is the runner record's
2283
+ // to state.
2285
2284
  //
2286
- // The two steps no longer send the same prompt. Each fixture set is staged
2287
- // under its own project root and the prompt is written against it, so the
2288
- // request now says which set a run traces. What the matcher leaves open is
2289
- // which observation a step selects: a record carrying one observation is
2290
- // selected by both steps, so the other set's oracles quantify over evidence
2291
- // that is not theirs. That is the limit test/contracts/README.md records as
2292
- // "a plan cannot declare that two steps must receive different inputs", and
2293
- // it is a property of the matcher rather than of the prompts.
2294
- inputBinding: { argument: null, option: { agent: { matcher: 'any' } }, environment: null, stdin: { prompt: { matcher: 'any' } } },
2285
+ // Standard input is bound as a literal, and the literal is the prompt the
2286
+ // harness assembles for this fixture set. Both steps declare the same
2287
+ // operation, so under a matcher binding every observation satisfies both
2288
+ // steps: one observation is selected twice and the other set's oracles
2289
+ // quantify over evidence that is not theirs, which is what held the clean
2290
+ // control at FAIL with five abstentions. The prompt is the only part of the
2291
+ // request that tells the two steps apart, because the project root it is
2292
+ // written against is the one fact about the set the request carries.
2293
+ //
2294
+ // buildTracePrompt is test/eval-trace.js's own buildPrompt, the function
2295
+ // that assembles the prompt the live run sends and the prompt
2296
+ // test/lib/probe-scoring.js puts on a trace observation. A literal is
2297
+ // compared with deepEquals, so a prompt restated here in any other form
2298
+ // would select nothing, every oracle would resolve `unreached`, and a run
2299
+ // that examined no evidence at all would report clean at exit 0. One
2300
+ // function on both sides is what makes that unrepresentable.
2301
+ inputBinding: {
2302
+ argument: null,
2303
+ option: { agent: { matcher: 'any' } },
2304
+ environment: null,
2305
+ stdin: { prompt: { literal: buildTracePrompt(set) } },
2306
+ },
2295
2307
  })),
2296
2308
  scopedResources: null,
2297
2309
  forbiddenInputs: FORBIDDEN_INPUTS,