bmad-method-test-architecture-enterprise 1.25.1 → 1.25.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/marketplace.json +1 -1
- package/CHANGELOG.md +8 -0
- package/docs/explanation/eval-quality-command-adapter.md +36 -21
- package/package.json +1 -1
- package/test/contracts/README.md +27 -13
- package/test/contracts/trace.contract.json +2 -2
- package/test/lib/probe-scoring.js +79 -24
- package/test/probes/expected-strength.json +8 -3
- package/tools/generate-contracts.js +28 -16
|
@@ -31,7 +31,7 @@
|
|
|
31
31
|
"name": "bmad-method-test-architecture-enterprise",
|
|
32
32
|
"source": "./",
|
|
33
33
|
"description": "Master Test Architect module for quality strategy, test automation, CI/CD quality gates, and structured testing education. Part of the BMad Method ecosystem.",
|
|
34
|
-
"version": "1.25.
|
|
34
|
+
"version": "1.25.2",
|
|
35
35
|
"author": {
|
|
36
36
|
"name": "Murat K Ozcan (TEA Creator) & Brian (BMad) Madison"
|
|
37
37
|
},
|
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
7
7
|
|
|
8
8
|
## [Unreleased]
|
|
9
9
|
|
|
10
|
+
## [1.25.2] - 2026-09-09
|
|
11
|
+
|
|
12
|
+
### Fixed
|
|
13
|
+
|
|
14
|
+
- `trace`'s clean control scored FAIL at exit 2 because five of its twenty-six oracles abstained. `O-009`, `O-010`, `O-011`, `O-013` and `O-014` are `for-any` quantifiers over the seeded set's summary, and `P-004`'s record carried one observation that both plan steps selected, so those five quantified over the clean summary's empty or absent collections. Two changes close it and they work only together: each plan step binds `stdin.prompt` as a literal of that set's own prompt, and the clean control's record carries both fixture sets' runs. Measured on the stored replay, `P-004` moves from FAIL at exit 2 with five abstentions to CONCERNS at exit 0 with all twenty-six oracles resolving `passed-clean-control`; four unsatisfied AD-20 coverage rules hold it at CONCERNS and are separate work. Either half alone is worse than neither, and both were run before this was accepted: two observations under the old matcher bindings leave every observation satisfying both steps, so `exactly-one` reports selector ambiguity on all twenty-six oracles at exit 3; literals over one observation select nothing, so all twenty-six resolve `unreached` and the run reports CONCERNS at exit 0 having examined no evidence, which is a silent green worse than the FAIL it replaces. A literal is compared with `deepEquals`, so the contract's literal and the record's prompt are both `buildPrompt` from `test/eval-trace.js`, and `traceEvidence` throws when the contract on disk binds any other bytes. The record used to carry the placeholder `The prompt the trace harness assembles for <case>.`, which is a prompt no run has ever sent. The control still catches a false positive: a clean summary damaged to derive FAIL with an invented blocker, reported as a defect finding, resolves `O-016` to `false-positive` and the probe to FAIL at exit 2. `test/contracts/trace.contract.json` grows 3.6 kilobytes, the size of the two prompts; the eight fragment-selection contracts keep the matcher binding, where the same fix costs 42 to 143 kilobytes each and roughly doubles every file.
|
|
15
|
+
- A stale reason in `test/contracts/README.md`, surfaced by the contradiction the entry above created. The paragraph on `trace.contract.json`'s witness said its two plan steps share one prompt on purpose because `test/eval-trace.js` names no fact about either fixture set in it, and it said the witness legs are runnable only against a staged workspace of the seeded set. Both stopped being true one release earlier: each set declares its own `projectRoot`, the whole prompt is written against it, and both witness legs stage the clean set. The `allow_gate` reasoning, the differential-and-invariance argument, and the pointer to `testData.setup` all survive and are kept. `testData.setup` in the contract itself was already correct and needed no change.
|
|
16
|
+
- A stale illustration in `docs/explanation/eval-quality-command-adapter.md`. The passage on AD-4 said `trace`'s five `for-any` oracles over the seeded export "still abstain on a clean run", which stopped being true once each plan step selected its own set's run. AD-4's rule is unchanged and the sentence now dates the abstention it describes.
|
|
17
|
+
|
|
10
18
|
## [1.25.1] - 2026-09-09
|
|
11
19
|
|
|
12
20
|
### Changed
|
|
@@ -249,8 +249,9 @@ spelling to have committed to.
|
|
|
249
249
|
|
|
250
250
|
One limit is worth knowing before writing a new oracle. Every quantifier still abstains over an empty
|
|
251
251
|
collection, which is AD-4's whole purpose and is why `trace`'s five `for-any` oracles over the seeded
|
|
252
|
-
export
|
|
253
|
-
|
|
252
|
+
export abstained on the clean control for as long as they were resolved against the clean set's
|
|
253
|
+
summary: they ask whether some element exists, and an empty collection is an honest "nothing was
|
|
254
|
+
checked". `deep-equality` against a literal `[]` also still abstains, so the
|
|
254
255
|
two spellings of "this collection is empty" disagree. eval-quality records that disagreement in AD-4
|
|
255
256
|
rather than hiding it. The bare `count-tolerance` assertion is the one to write.
|
|
256
257
|
|
|
@@ -273,33 +274,47 @@ Three findings, all measured, none of them tuned away.
|
|
|
273
274
|
before. `test-review` leaves `whole-body`, `malformed-input` and `state-change-read-back`
|
|
274
275
|
unsatisfied; every fragment-selection contract leaves `malformed-input` unsatisfied. Each scores the
|
|
275
276
|
run down to CONCERNS without blocking it, which is exactly the weight AD-20 gives a coverage gap.
|
|
276
|
-
- **`trace`'s clean control
|
|
277
|
-
|
|
277
|
+
- **`trace`'s clean control scored FAIL. Closed, and the two halves that closed it work only
|
|
278
|
+
together.** Five of its twenty-six oracles abstained on `P-004`: `O-009`, `O-010`, `O-011`,
|
|
278
279
|
`O-013` and `O-014`, every one a `for-any` quantifier over the seeded export, and every one named
|
|
279
280
|
in the artifact's `verdictBasis`. It was seven. `O-023` and `O-024`, the two asserting that a
|
|
280
281
|
collection is empty, moved to `passed-clean-control` when `eval-quality` 1.4.0 gave that claim a
|
|
281
282
|
spelling.
|
|
282
283
|
|
|
283
|
-
The five
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
284
|
+
The five abstained because the record carried one observation and both plan steps selected it, so
|
|
285
|
+
the seeded set's oracles quantified over the clean summary's empty or absent collections. Each
|
|
286
|
+
step binds its own set's prompt as a literal now, and the clean control's record carries both
|
|
287
|
+
sets' runs, so the seeded step selects the seeded run and the clean step selects the clean one.
|
|
288
|
+
Measured on the stored replay, all twenty-six oracles resolve `passed-clean-control` and `P-004`
|
|
289
|
+
moves from FAIL at exit 2 to CONCERNS at exit 0. What holds it at CONCERNS is the four unsatisfied
|
|
290
|
+
AD-20 coverage rules the bullet above describes, which is separate work.
|
|
291
|
+
|
|
292
|
+
Either half on its own is worse than neither, which is why they landed together. Two observations
|
|
293
|
+
under the old matcher bindings leave every observation satisfying both steps, and `exactly-one`
|
|
294
|
+
then reports selector ambiguity on all twenty-six oracles at exit 3. Literals over one observation
|
|
295
|
+
select nothing: all twenty-six resolve `unreached` and the run reports CONCERNS at exit 0 having
|
|
296
|
+
examined no evidence at all, which is a silent green and is worse than the FAIL it replaces. A
|
|
297
|
+
literal is compared with `deepEquals`, so the contract's literal and the record's prompt are both
|
|
298
|
+
`buildPrompt` from `test/eval-trace.js`, and `traceEvidence` in `test/lib/probe-scoring.js` throws
|
|
299
|
+
when the contract on disk binds any other bytes. Both states above were run before the change was
|
|
300
|
+
accepted, so the failure mode is one somebody has seen.
|
|
294
301
|
|
|
295
302
|
- **A plan cannot tell two steps apart when both bind their inputs by matcher.** Each
|
|
296
303
|
fragment-selection contract declares one plan step per case, distinguished only by the prompt, and
|
|
297
|
-
the prompt is bound `{matcher: 'any'}
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
304
|
+
the prompt is bound `{matcher: 'any'}`. A record carrying one observation is therefore selected by
|
|
305
|
+
every step, and the oracles of the other cases resolve against evidence that is not theirs. The
|
|
306
|
+
designated oracle still votes correctly, so the strength vector is unaffected, and the surrounding
|
|
307
|
+
outcome rows are noise. This is the limit `test/contracts/README.md` records as "a plan cannot
|
|
308
|
+
declare that two steps must receive different inputs", with its consequence now measured.
|
|
309
|
+
|
|
310
|
+
The trace contract closed this with literals; the eight fragment-selection contracts keep the
|
|
311
|
+
matcher, and the reason is size. Their prompts carry the workflow's knowledge-loading rules and its
|
|
312
|
+
whole fragment index, 21 to 43 kilobytes per step, so binding each step's prompt as a literal adds
|
|
313
|
+
between 42 and 143 kilobytes to a contract and roughly doubles every one of the eight files: 1.7x
|
|
314
|
+
for `bmad-testarch-nfr`, 2.7x for `bmad-testarch-automate`, measured from the prompts those
|
|
315
|
+
contracts already carry on their witness legs. Trace pays 3.6 kilobytes on 108 for the same fix,
|
|
316
|
+
because its two prompts are 1.8 kilobytes each, the size of the prompt the file already carries on
|
|
317
|
+
each of its two witness legs.
|
|
303
318
|
|
|
304
319
|
### What the live pre-flight measured
|
|
305
320
|
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$schema": "https://json.schemastore.org/package.json",
|
|
3
3
|
"name": "bmad-method-test-architecture-enterprise",
|
|
4
|
-
"version": "1.25.
|
|
4
|
+
"version": "1.25.2",
|
|
5
5
|
"description": "Master Test Architect for quality strategy, test automation, and release gates",
|
|
6
6
|
"keywords": [
|
|
7
7
|
"bmad",
|
package/test/contracts/README.md
CHANGED
|
@@ -116,19 +116,21 @@ input-insensitive by design, and the contract now states that as its own claim i
|
|
|
116
116
|
nothing.
|
|
117
117
|
|
|
118
118
|
`trace.contract.json`, the tenth, states its witness over standard input on one prompt value,
|
|
119
|
-
`allow_gate`. Its two plan steps
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
ground truth establishes an effect for,
|
|
124
|
-
evaluates a gate only when it is true and writes
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
it on the
|
|
119
|
+
`allow_gate`. Its two plan steps send two prompts, each written against its own fixture set's project
|
|
120
|
+
root, and each step binds that prompt as its `stdin.prompt` literal. The summaries the two sets
|
|
121
|
+
produce differ because of the files staged under those roots, so a differential across the two sets
|
|
122
|
+
would attribute to the prompt a difference the staged workspace produced, and an invariance claim over
|
|
123
|
+
them would be false. `allow_gate` is the one prompt value the ground truth establishes an effect for,
|
|
124
|
+
through `skillRuleCitations.gateEligibility`: step-05 evaluates a gate only when it is true and writes
|
|
125
|
+
`gate_basis` as `none` otherwise, so two prompts differing in that value, over one staged fixture set,
|
|
126
|
+
produce two `gate_basis` values. That is a true and checkable claim that the command reads its
|
|
127
|
+
standard input, and it is the claim the witness makes. Both its legs stage the clean set, because AD-10
|
|
128
|
+
reads every other leg of an operation as a clean leg, and they are runnable only against that staged
|
|
129
|
+
workspace, which is the coupling `docs/explanation/eval-quality-command-adapter.md` records for every
|
|
130
|
+
artifact-writing command. The contract itself states this reasoning in `testData.setup`, because
|
|
131
|
+
`SensitivityWitness` is a strict object with no prose field of its own; do not look for it on the
|
|
132
|
+
witness. The section "A plan cannot declare that two steps must receive different inputs" below
|
|
133
|
+
records what those literals cost and what they bought.
|
|
132
134
|
|
|
133
135
|
## What the operator vocabulary cannot say
|
|
134
136
|
|
|
@@ -145,6 +147,18 @@ cases differ only in the prompt sent to the runner, and that is the property tha
|
|
|
145
147
|
mean anything. `sensitivityWitness` expresses a two-leg version of this inside one operation and
|
|
146
148
|
nothing expresses it across plan steps.
|
|
147
149
|
|
|
150
|
+
Answered for `trace`, at a price the eight fragment-selection contracts cannot pay. Binding each
|
|
151
|
+
step's `stdin.prompt` as a literal of that step's own prompt is what tells the two steps apart: the
|
|
152
|
+
seeded step selects the seeded run, the clean step selects the clean one, and `trace`'s clean control
|
|
153
|
+
moved from FAIL at exit 2 with five oracles abstaining to CONCERNS at exit 0 with all twenty-six
|
|
154
|
+
resolving `passed-clean-control`. It costs 3.6 kilobytes on `trace.contract.json` and 42 to 143
|
|
155
|
+
kilobytes on a fragment-selection contract, which roughly doubles each of those eight files, so they
|
|
156
|
+
keep the matcher. A literal is compared with `deepEquals`, so both sides of one have to come from a
|
|
157
|
+
single function: `tools/generate-contracts.js` and `test/lib/probe-scoring.js` both call `buildPrompt`
|
|
158
|
+
from `test/eval-trace.js`, and the record builder throws when the contract on disk carries any other
|
|
159
|
+
bytes. A prompt restated in either place would select nothing and every oracle would resolve
|
|
160
|
+
`unreached`, which reads as a clean run at exit 0.
|
|
161
|
+
|
|
148
162
|
**An empty collection is no evidence.** AD-4 resolves a quantifier over an
|
|
149
163
|
empty collection to `insufficient-evidence` with an `empty-collection` introduction condition. The
|
|
150
164
|
`test-review` harness reads a verdict whose findings array is empty as a reviewer that named nothing,
|
|
@@ -2701,7 +2701,7 @@
|
|
|
2701
2701
|
"environment": null,
|
|
2702
2702
|
"stdin": {
|
|
2703
2703
|
"prompt": {
|
|
2704
|
-
"
|
|
2704
|
+
"literal": "You are running the TEA workflow `bmad-testarch-trace` against the project in `tenant-data-export/`.\n\nThe workflow is in `skill/`. Read `skill/instructions.md` first, then execute every step file it\nnames in order, in full, without skipping or reordering. The step files are under `skill/steps-c/`.\n\n----- run configuration -----\nResolve the workflow placeholders to these values:\n\n- `{project-root}`: `tenant-data-export`\n- `{config_source}`: `tenant-data-export/_bmad/tea/config.yaml`\n- `{test_artifacts}`: `tenant-data-export/test-artifacts`\n- `{test_dir}`: `tenant-data-export/tests`\n- `{source_dir}`: `tenant-data-export/src`\n- `{skill-root}`: `skill`\n- `gate_type`: `epic`\n- `decision_mode`: `deterministic`\n- `collection_mode`: `contract_static`\n- `allow_gate`: `true`\n- `coverage_basis`: `auto`\n- `summary_confidence`: `auto`\n- `coverage_levels`: `e2e,api,component,unit,live`\n\nThe trace target is the epic under `tenant-data-export/docs/epics/`. Resolve the coverage oracle from it the\nway step-01 says to.\n\n----- what to produce -----\nWrite both deliverables the workflow declares:\n\n- `tenant-data-export/test-artifacts/traceability-matrix.md`, from `skill/trace-template.md`, carrying the\n detailed mapping with one section per criterion, each stating its coverage status and the tests\n that establish it as `file:line`.\n- `tenant-data-export/test-artifacts/e2e-trace-summary.json` at schema_version 0.3.0, exactly as\n `skill/steps-c/step-05-gate-decision.md` section 3b defines it.\n\nDo not add, edit, or delete any file under `tenant-data-export/docs/`, `tenant-data-export/src/`, or `tenant-data-export/tests/`.\nThis workflow does not generate tests.\n\nWhen you are done, print one line naming the two files you wrote. Nothing else you print is read."
|
|
2705
2705
|
}
|
|
2706
2706
|
}
|
|
2707
2707
|
}
|
|
@@ -2721,7 +2721,7 @@
|
|
|
2721
2721
|
"environment": null,
|
|
2722
2722
|
"stdin": {
|
|
2723
2723
|
"prompt": {
|
|
2724
|
-
"
|
|
2724
|
+
"literal": "You are running the TEA workflow `bmad-testarch-trace` against the project in `api-token-lifecycle/`.\n\nThe workflow is in `skill/`. Read `skill/instructions.md` first, then execute every step file it\nnames in order, in full, without skipping or reordering. The step files are under `skill/steps-c/`.\n\n----- run configuration -----\nResolve the workflow placeholders to these values:\n\n- `{project-root}`: `api-token-lifecycle`\n- `{config_source}`: `api-token-lifecycle/_bmad/tea/config.yaml`\n- `{test_artifacts}`: `api-token-lifecycle/test-artifacts`\n- `{test_dir}`: `api-token-lifecycle/tests`\n- `{source_dir}`: `api-token-lifecycle/src`\n- `{skill-root}`: `skill`\n- `gate_type`: `epic`\n- `decision_mode`: `deterministic`\n- `collection_mode`: `contract_static`\n- `allow_gate`: `true`\n- `coverage_basis`: `auto`\n- `summary_confidence`: `auto`\n- `coverage_levels`: `e2e,api,component,unit,live`\n\nThe trace target is the epic under `api-token-lifecycle/docs/epics/`. Resolve the coverage oracle from it the\nway step-01 says to.\n\n----- what to produce -----\nWrite both deliverables the workflow declares:\n\n- `api-token-lifecycle/test-artifacts/traceability-matrix.md`, from `skill/trace-template.md`, carrying the\n detailed mapping with one section per criterion, each stating its coverage status and the tests\n that establish it as `file:line`.\n- `api-token-lifecycle/test-artifacts/e2e-trace-summary.json` at schema_version 0.3.0, exactly as\n `skill/steps-c/step-05-gate-decision.md` section 3b defines it.\n\nDo not add, edit, or delete any file under `api-token-lifecycle/docs/`, `api-token-lifecycle/src/`, or `api-token-lifecycle/tests/`.\nThis workflow does not generate tests.\n\nWhen you are done, print one line naming the two files you wrote. Nothing else you print is read."
|
|
2725
2725
|
}
|
|
2726
2726
|
}
|
|
2727
2727
|
}
|
|
@@ -41,6 +41,11 @@ const fs = require('node:fs');
|
|
|
41
41
|
const path = require('node:path');
|
|
42
42
|
|
|
43
43
|
const { digest } = require('./eval-record');
|
|
44
|
+
// The prompt a trace observation carries is the prompt the harness assembles, and
|
|
45
|
+
// tools/generate-contracts.js binds the same function's output as each trace plan
|
|
46
|
+
// step's stdin literal. One function on both sides is the whole guard; `legs` in
|
|
47
|
+
// traceEvidence says what a restated prompt would cost.
|
|
48
|
+
const { buildPrompt: buildTracePrompt } = require('../eval-trace');
|
|
44
49
|
const {
|
|
45
50
|
evaluatorConfiguration,
|
|
46
51
|
isolationManifest,
|
|
@@ -276,12 +281,42 @@ function traceArtifacts(caseId) {
|
|
|
276
281
|
function traceEvidence(contract) {
|
|
277
282
|
const [seededStep, cleanStep] = contract.interactionPlan;
|
|
278
283
|
const groundTruth = readJson(path.join(PROJECT_ROOT, 'test', 'fixtures', 'trace-eval', 'ground-truth.json'));
|
|
284
|
+
|
|
285
|
+
/**
|
|
286
|
+
* One entry per fixture set: the plan step that selects it, the stored run its
|
|
287
|
+
* project root names, and the prompt the harness assembles for it.
|
|
288
|
+
*
|
|
289
|
+
* `buildTracePrompt` is test/eval-trace.js's own `buildPrompt`, which is also
|
|
290
|
+
* what tools/generate-contracts.js calls to build each step's stdin literal. A
|
|
291
|
+
* literal is compared with `deepEquals`, so a prompt restated here in any other
|
|
292
|
+
* form would select nothing, all twenty-six oracles would resolve `unreached`,
|
|
293
|
+
* and the run would report clean at exit 0 having examined no evidence. The
|
|
294
|
+
* equality below is the tripwire: the prompt this record will carry is checked
|
|
295
|
+
* against the literal the contract on disk binds, so a divergence fails the run
|
|
296
|
+
* where it would otherwise pass it silently.
|
|
297
|
+
*/
|
|
298
|
+
const legs = groundTruth.fixtureSets.map((set) => {
|
|
299
|
+
const seeded = set.id.startsWith('seeded');
|
|
300
|
+
const step = seeded ? seededStep : cleanStep;
|
|
301
|
+
const prompt = buildTracePrompt(set);
|
|
302
|
+
if (step.inputBinding.stdin?.prompt?.literal !== prompt) {
|
|
303
|
+
throw new Error(
|
|
304
|
+
`${step.stepId} binds a stdin literal that is not the prompt the harness assembles for ${set.id}; run node tools/generate-contracts.js`,
|
|
305
|
+
);
|
|
306
|
+
}
|
|
307
|
+
return {
|
|
308
|
+
caseId: seeded ? 'seeded-correct-run' : 'clean-correct-run',
|
|
309
|
+
observationId: seeded ? 'trace-seeded-run' : 'trace-clean-run',
|
|
310
|
+
projectRoot: set.projectRoot,
|
|
311
|
+
step,
|
|
312
|
+
prompt,
|
|
313
|
+
};
|
|
314
|
+
});
|
|
315
|
+
|
|
279
316
|
// The stored run each fixture set's project root names. A trace prompt is written
|
|
280
317
|
// against one project root, and that root is the only thing in the request that
|
|
281
318
|
// says which set the leg is asking for, so it is what the port stages against.
|
|
282
|
-
const caseByProjectRoot = new Map(
|
|
283
|
-
groundTruth.fixtureSets.map((set) => [set.projectRoot, set.id.startsWith('seeded') ? 'seeded-correct-run' : 'clean-correct-run']),
|
|
284
|
-
);
|
|
319
|
+
const caseByProjectRoot = new Map(legs.map((leg) => [leg.projectRoot, leg.caseId]));
|
|
285
320
|
|
|
286
321
|
return {
|
|
287
322
|
/**
|
|
@@ -313,41 +348,61 @@ function traceEvidence(contract) {
|
|
|
313
348
|
},
|
|
314
349
|
recordInputs(probe) {
|
|
315
350
|
const clean = probe.expectedClean;
|
|
316
|
-
|
|
317
|
-
//
|
|
318
|
-
//
|
|
319
|
-
//
|
|
320
|
-
//
|
|
321
|
-
//
|
|
322
|
-
//
|
|
323
|
-
//
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
351
|
+
// The clean control's record carries both runs. Each step binds its own set's
|
|
352
|
+
// prompt as a literal, so the seeded step selects the seeded observation and
|
|
353
|
+
// the clean step selects the clean one, and the oracles of each quantify over
|
|
354
|
+
// the summary their own set produced. With one observation both steps selected
|
|
355
|
+
// it, so five of the seeded step's `for-any` oracles quantified over the clean
|
|
356
|
+
// summary's empty or absent collections and abstained, which was the control's
|
|
357
|
+
// FAIL at exit 2. Two observations under a matcher binding are worse: every
|
|
358
|
+
// observation satisfies both steps, and `exactly-one` then reports selector
|
|
359
|
+
// ambiguity on all twenty-six oracles. The literals and the second observation
|
|
360
|
+
// work only together.
|
|
361
|
+
//
|
|
362
|
+
// A defect probe carries its seeded run alone. AD-9's qualification gate
|
|
363
|
+
// resolves every one of its oracles before a selection is read, so a second
|
|
364
|
+
// observation would add evidence nothing reaches.
|
|
365
|
+
//
|
|
366
|
+
// Two things this record still states loosely, and nothing reads either today.
|
|
367
|
+
// `conditionArm` stays `clean-correct-run` for a record that now carries two
|
|
368
|
+
// runs, and P-004's `baselinePassEvidence` in test/probes/trace.probes.json
|
|
369
|
+
// names the clean summary's digest alone.
|
|
370
|
+
const selected = clean ? legs : legs.filter((leg) => leg.caseId === 'seeded-correct-run');
|
|
371
|
+
const observations = selected.map((leg, index) =>
|
|
327
372
|
recordObservation({
|
|
328
|
-
observationId,
|
|
329
|
-
sequence: 1,
|
|
330
|
-
operationId: step.operationId,
|
|
331
|
-
callInputs: { option: { agent: 'claude' }, stdin: { prompt:
|
|
373
|
+
observationId: leg.observationId,
|
|
374
|
+
sequence: index + 1,
|
|
375
|
+
operationId: leg.step.operationId,
|
|
376
|
+
callInputs: { option: { agent: 'claude' }, stdin: { prompt: leg.prompt } },
|
|
332
377
|
stdout: { kind: 'text', value: '' },
|
|
333
378
|
stderr: { kind: 'text', value: '' },
|
|
334
379
|
exitCode: 0,
|
|
335
|
-
artifacts: traceArtifacts(caseId),
|
|
380
|
+
artifacts: traceArtifacts(leg.caseId),
|
|
336
381
|
}),
|
|
337
|
-
|
|
382
|
+
);
|
|
383
|
+
// Every trace oracle reads one step's interaction, so its disposition cites
|
|
384
|
+
// that step's observation. A record carrying one run has nothing of the other
|
|
385
|
+
// step's to cite, and a `held` disposition citing nothing is scored as an
|
|
386
|
+
// unsupported claim, so those oracles cite the run the record does carry.
|
|
387
|
+
const observationIdByStep = new Map(selected.map((leg) => [leg.step.stepId, leg.observationId]));
|
|
388
|
+
const stepOf = (pointer) => String(pointer).split('/')[2];
|
|
389
|
+
const citedObservationId = (oracle) => {
|
|
390
|
+
const target = (oracle.direction?.evidenceTargets ?? []).find((pointer) => observationIdByStep.has(stepOf(pointer)));
|
|
391
|
+
return target === undefined ? observations[0].observationId : observationIdByStep.get(stepOf(target));
|
|
392
|
+
};
|
|
338
393
|
return {
|
|
339
394
|
observations,
|
|
340
395
|
findings: [],
|
|
341
|
-
//
|
|
342
|
-
// so every oracle held on the evidence the harness read.
|
|
396
|
+
// Every stored run in this record is recorded as scoring every check it was
|
|
397
|
+
// given, so every oracle held on the evidence the harness read.
|
|
343
398
|
oracleDispositions: contract.oracles.map((oracle) => ({
|
|
344
399
|
oracleId: oracle.id,
|
|
345
400
|
disposition: 'held',
|
|
346
|
-
observationIds: [
|
|
401
|
+
observationIds: [citedObservationId(oracle)],
|
|
347
402
|
note: null,
|
|
348
403
|
})),
|
|
349
404
|
evaluatorRecommendation: 'PASS',
|
|
350
|
-
conditionArm:
|
|
405
|
+
conditionArm: clean ? 'clean-correct-run' : 'seeded-correct-run',
|
|
351
406
|
};
|
|
352
407
|
},
|
|
353
408
|
};
|
|
@@ -308,9 +308,14 @@
|
|
|
308
308
|
"expectedClean": true,
|
|
309
309
|
"behaviorId": "B-016",
|
|
310
310
|
"preflight": "passed",
|
|
311
|
-
"verdict": "
|
|
312
|
-
"exitCode":
|
|
313
|
-
"basis": [
|
|
311
|
+
"verdict": "CONCERNS",
|
|
312
|
+
"exitCode": 0,
|
|
313
|
+
"basis": [
|
|
314
|
+
"coverage gap malformed-input unsatisfied at or above the severity floor",
|
|
315
|
+
"coverage gap state-change-read-back unsatisfied at or above the severity floor",
|
|
316
|
+
"coverage gap success-indicator-separation unsatisfied at or above the severity floor",
|
|
317
|
+
"coverage gap whole-body unsatisfied at or above the severity floor"
|
|
318
|
+
],
|
|
314
319
|
"qualification": null,
|
|
315
320
|
"strength": {
|
|
316
321
|
"defect": null,
|
|
@@ -22,7 +22,8 @@
|
|
|
22
22
|
* name for that contract's sourceSpecDigest,
|
|
23
23
|
* - cli/trace-runner.js's TRACE_REQUEST_KEYS and DEFAULT_AGENT for that
|
|
24
24
|
* contract's request shape and witness legs, and test/eval-trace.js's
|
|
25
|
-
* buildPrompt for the two prompts the witness legs send
|
|
25
|
+
* buildPrompt for the two prompts the witness legs send and for the literal
|
|
26
|
+
* each of its two plan steps binds on standard input, and
|
|
26
27
|
* - the trace workflow's step-05 for the key set of the summary the run writes.
|
|
27
28
|
*
|
|
28
29
|
* The prose that is genuinely authored (an oracle's commentary, a behavior's lead
|
|
@@ -69,8 +70,8 @@ const { VERDICT_KEYS, DEFAULT_AGENT } = require('../cli/test-review');
|
|
|
69
70
|
// owns its own request shape and its own default agent, so both are read from it.
|
|
70
71
|
const { SELECTION_REQUEST_KEYS, DEFAULT_AGENT: SELECTION_DEFAULT_AGENT } = require('../cli/fragment-selection-runner');
|
|
71
72
|
// And for the trace command: its request shape and default agent are its own, and
|
|
72
|
-
// the prompt its witness legs send
|
|
73
|
-
// only thing that assembles one.
|
|
73
|
+
// the prompt its witness legs send and the literal each of its plan steps binds
|
|
74
|
+
// are the harness's, because the harness is the only thing that assembles one.
|
|
74
75
|
const { TRACE_REQUEST_KEYS, DEFAULT_AGENT: TRACE_DEFAULT_AGENT, EXIT_CODES: TRACE_EXIT_CODES } = require('../cli/trace-runner');
|
|
75
76
|
const { vendorEnvironmentNames } = require('../cli/lib/runner-exit-codes');
|
|
76
77
|
const {
|
|
@@ -2278,20 +2279,31 @@ function buildTraceContract() {
|
|
|
2278
2279
|
operationId: TRACE_OPERATION,
|
|
2279
2280
|
after: null,
|
|
2280
2281
|
cardinality: 'exactly-one',
|
|
2281
|
-
// The agent is bound as `any
|
|
2282
|
-
//
|
|
2283
|
-
// the fragment-selection contracts give: its bytes are evidence, and the
|
|
2284
|
-
// sealed record carries their digest.
|
|
2282
|
+
// The agent is bound as `any`: which vendor answered is the runner record's
|
|
2283
|
+
// to state.
|
|
2285
2284
|
//
|
|
2286
|
-
//
|
|
2287
|
-
//
|
|
2288
|
-
//
|
|
2289
|
-
//
|
|
2290
|
-
//
|
|
2291
|
-
//
|
|
2292
|
-
//
|
|
2293
|
-
//
|
|
2294
|
-
|
|
2285
|
+
// Standard input is bound as a literal, and the literal is the prompt the
|
|
2286
|
+
// harness assembles for this fixture set. Both steps declare the same
|
|
2287
|
+
// operation, so under a matcher binding every observation satisfies both
|
|
2288
|
+
// steps: one observation is selected twice and the other set's oracles
|
|
2289
|
+
// quantify over evidence that is not theirs, which is what held the clean
|
|
2290
|
+
// control at FAIL with five abstentions. The prompt is the only part of the
|
|
2291
|
+
// request that tells the two steps apart, because the project root it is
|
|
2292
|
+
// written against is the one fact about the set the request carries.
|
|
2293
|
+
//
|
|
2294
|
+
// buildTracePrompt is test/eval-trace.js's own buildPrompt, the function
|
|
2295
|
+
// that assembles the prompt the live run sends and the prompt
|
|
2296
|
+
// test/lib/probe-scoring.js puts on a trace observation. A literal is
|
|
2297
|
+
// compared with deepEquals, so a prompt restated here in any other form
|
|
2298
|
+
// would select nothing, every oracle would resolve `unreached`, and a run
|
|
2299
|
+
// that examined no evidence at all would report clean at exit 0. One
|
|
2300
|
+
// function on both sides is what makes that unrepresentable.
|
|
2301
|
+
inputBinding: {
|
|
2302
|
+
argument: null,
|
|
2303
|
+
option: { agent: { matcher: 'any' } },
|
|
2304
|
+
environment: null,
|
|
2305
|
+
stdin: { prompt: { literal: buildTracePrompt(set) } },
|
|
2306
|
+
},
|
|
2295
2307
|
})),
|
|
2296
2308
|
scopedResources: null,
|
|
2297
2309
|
forbiddenInputs: FORBIDDEN_INPUTS,
|