eval-quality 1.0.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +77 -253
- package/corpus/dev/README.md +27 -31
- package/corpus/dev/index.json +1 -1
- package/dist/cli/render.d.ts +1 -1
- package/dist/cli/render.js +4 -4
- package/dist/index.d.ts +1 -1
- package/dist/index.js +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,36 +1,40 @@
|
|
|
1
1
|
# `eval-quality`
|
|
2
2
|
|
|
3
3
|
**[Documentation](https://bmad-code-org.github.io/bmad-eval-quality/)** ·
|
|
4
|
-
[
|
|
4
|
+
[Start here](https://bmad-code-org.github.io/bmad-eval-quality/tutorials/getting-started/) ·
|
|
5
5
|
[CLI reference](https://bmad-code-org.github.io/bmad-eval-quality/reference/cli-commands/) ·
|
|
6
6
|
[npm](https://www.npmjs.com/package/eval-quality)
|
|
7
7
|
|
|
8
|
+
Write the eval. Hide the bug. See if the eval catches it.
|
|
9
|
+
|
|
8
10
|
```bash
|
|
9
11
|
npx eval-quality --help
|
|
10
12
|
```
|
|
11
13
|
|
|
12
|
-
|
|
14
|
+
## The idea
|
|
13
15
|
|
|
14
|
-
|
|
15
|
-
2. **Seal**: render the brief for the independent evaluator while hiding the planted bug and scoring answer.
|
|
16
|
-
3. **Preflight**: verify baseline environment readiness and probe reachability before running an evaluator.
|
|
17
|
-
4. **Score**: compare the evaluator's completed findings against the hidden bug signature and mint a versioned evidence artifact.
|
|
16
|
+
An AI evaluation can pass and prove nothing. It sends a request, sees something plausible come back, and reports success while the failure it was written to catch sits right next to the thing it looked at. That is a blind spot, and a green run never shows you one.
|
|
18
17
|
|
|
19
|
-
|
|
18
|
+
Mutation testing finds blind spots in ordinary tests: keep the tests fixed, plant a known defect in the code, run the tests again, and ask whether they caught it. `eval-quality` points the same idea at evaluations.
|
|
20
19
|
|
|
21
|
-
|
|
20
|
+
```text
|
|
21
|
+
clean system → evaluation → should pass
|
|
22
|
+
mutated system → evaluation → should degrade
|
|
23
|
+
↓
|
|
24
|
+
did the evaluation catch it?
|
|
25
|
+
```
|
|
22
26
|
|
|
23
|
-
|
|
27
|
+
An evaluation that caught the planted defect is sensitive to that failure. One that stayed green has a blind spot, and now you know where.
|
|
24
28
|
|
|
25
|
-
|
|
29
|
+
## The evaluation contract
|
|
26
30
|
|
|
27
|
-
It
|
|
31
|
+
The contract is the test: the evaluator's instructions for how to expose a failure and what evidence counts as finding it. The long name is Behavioral Evaluation Contract; the docs shorten it to eval contract. It is a JSON document that declares:
|
|
28
32
|
|
|
29
33
|
- the behavior being evaluated;
|
|
30
34
|
- the probes the evaluator should perform;
|
|
31
35
|
- the evidence it should inspect;
|
|
32
36
|
- the negative behavior it must rule out;
|
|
33
|
-
- the oracle that
|
|
37
|
+
- the oracle that decides pass or fail.
|
|
34
38
|
|
|
35
39
|
For example:
|
|
36
40
|
|
|
@@ -48,10 +52,6 @@ A weak eval checks only the response and misses the bug.
|
|
|
48
52
|
|
|
49
53
|
A strong eval checks the response **and** persistence, so it catches the bug.
|
|
50
54
|
|
|
51
|
-
### Caveman summary
|
|
52
|
-
|
|
53
|
-
Write the eval. Hide the bug. See if the eval catches it.
|
|
54
|
-
|
|
55
55
|
## The core flow, in eight nouns
|
|
56
56
|
|
|
57
57
|
Every run of an evaluation walks the same order:
|
|
@@ -66,14 +66,16 @@ evaluation contract → probe → observation → preflight → evidence → ora
|
|
|
66
66
|
| **Probe** | How to poke the system to produce evidence: a test case, a call, a step. In scoring, a probe also names the defect it seeded. | "Update note n-1, then read it back." |
|
|
67
67
|
| **Observation** | What actually happened when the system was poked: the recorded status, headers, and body of one call. | `PATCH` returned 200 with the new title; the later `GET` returned the old one. |
|
|
68
68
|
| **Preflight** | Whether the environment and the observations are fit for meaningful measurement. | Both operations reachable, the fixture reset, the clean control clean. |
|
|
69
|
-
| **Evidence** | The recorded output, trajectory, and artifacts from the evaluation run: what the
|
|
69
|
+
| **Evidence** | The recorded output, trajectory, and artifacts from the evaluation run: what the oracles resolve against. | A finding citing the two observations above. |
|
|
70
70
|
| **Oracle** | The assertion: the relation that has to hold over the evidence. | The title sent equals the title read back. |
|
|
71
71
|
| **Rubric** | The grading guide for judgment-heavy quality, with anchored criteria a judge scores against. The judge runs outside the package and its scores arrive in the sealed run record. | Present only when a contract declares one. |
|
|
72
72
|
| **Score / verdict** | The combined result: did the evaluation catch the planted defect? `PASS`, `WAIVED`, `CONCERNS`, or `FAIL`, or Invalid when the run produced no verdict. | `FAIL`, exit code 2. |
|
|
73
73
|
|
|
74
74
|
The word evidence is used twice on purpose. The evidence in the flow is what the evaluator produced, and it reaches `score` inside a sealed run record. The evidence artifact is what `score` mints at the end: the outcomes, the verdict, the strength vector, and the exit code.
|
|
75
75
|
|
|
76
|
-
|
|
76
|
+
## What the tool does
|
|
77
|
+
|
|
78
|
+
`eval-quality` is a Node package and a command line binary. It gives you four commands, which sit on the flow like this:
|
|
77
79
|
|
|
78
80
|
| Command | Reads | Writes |
|
|
79
81
|
| --- | --- | --- |
|
|
@@ -82,69 +84,11 @@ The four commands sit on that flow like this:
|
|
|
82
84
|
| `preflight` | a contract, a probe list, observations | `preflight-verdict.json`, fit or unfit to measure |
|
|
83
85
|
| `score` | a sealed run record, the contract, a probe, the preflight verdict, a scoring policy, a caller-attested corpus digest, and the isolation manifest and evaluator configuration the record was produced under | `evidence-artifact.json`, and the verdict's own exit code |
|
|
84
86
|
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
Compile disciplined agent eval contracts, then check whether those contracts can catch known bugs.
|
|
88
|
-
|
|
89
|
-
An agent can produce an answer that reads as correct and is materially wrong. An eval can make the same mistake.
|
|
90
|
-
|
|
91
|
-
Weak oracle:
|
|
92
|
-
|
|
93
|
-
```text
|
|
94
|
-
Check malformed input is handled correctly.
|
|
95
|
-
```
|
|
96
|
-
|
|
97
|
-
An evaluator given that instruction sends one malformed request, sees an error come back, and reports success. The record that should never have been created was created anyway. Nobody looked.
|
|
98
|
-
|
|
99
|
-
Strong oracle:
|
|
100
|
-
|
|
101
|
-
```text
|
|
102
|
-
Send malformed input. Verify the request fails, inspect the full response body,
|
|
103
|
-
confirm the specific error, and confirm no record was created.
|
|
104
|
-
```
|
|
105
|
-
|
|
106
|
-
A passing eval says little when the contract never asked for the probe that would expose the failure. Testing whether the eval can catch a failure you already know about is the first check worth running.
|
|
107
|
-
|
|
108
|
-
The loop that does that is a twin run. Keep the contract, the probes, the oracles, and the scoring policy fixed. Run the evaluator once against the clean system and once against the same system carrying one known defect. Score both runs. The contract is strong when the clean run passes and the mutated run degrades, and it has a blind spot when both stay green.
|
|
109
|
-
|
|
110
|
-
## What each part provides
|
|
111
|
-
|
|
112
|
-
`eval-quality` provides:
|
|
113
|
-
|
|
114
|
-
- the Behavioral Evaluation Contract schema
|
|
115
|
-
- the oracle vocabulary and authoring rules
|
|
116
|
-
- the contract compiler
|
|
117
|
-
- the environment pre-flight
|
|
118
|
-
- Eval Contract strength scoring: the AD-7 rate vector and dominance relation, implemented in
|
|
119
|
-
`src/core/score/strength.ts` and reached by the `score` command
|
|
120
|
-
- PASS / WAIVED / CONCERNS / FAIL governance: both verdict ladders, implemented and total in
|
|
121
|
-
`src/core/score/ladder.ts`, and the `score` command's own exit code
|
|
122
|
-
- versioned evidence output: `evidence-artifact.json`, implemented in `src/core/emit/emit.ts` and
|
|
123
|
-
minted by the `score` command
|
|
124
|
-
|
|
125
|
-
The caller provides:
|
|
126
|
-
|
|
127
|
-
- execution of its chosen agent, harness, or person
|
|
128
|
-
- repeated trials
|
|
129
|
-
- cost accounting
|
|
130
|
-
- the live system and environment-probe implementation
|
|
131
|
-
- a sealed run record returned for ingestion
|
|
132
|
-
|
|
133
|
-
`eval-quality` executes nothing: it never spawns a process, calls a model, drives a system under test,
|
|
134
|
-
or invokes a judge. Its six stages are compile, seal, ingest, pre-flight, score, and emit, all pure,
|
|
135
|
-
and every one is reachable through the CLI and the library alike. That list is the declared stage
|
|
136
|
-
order; on the clock, ingest follows the evaluator run, so it sits after pre-flight and just before
|
|
137
|
-
score. Compile, seal, and pre-flight each
|
|
138
|
-
have their own command and their own exported function. `ingest`, `score`, and `emit` are reached
|
|
139
|
-
through the one `score` command and the one exported `runScore` call that chains them, per AD-14's
|
|
140
|
-
rule that a command exposes no more than the library itself calls. Pre-flight probes the fixture through the environment-probe port, so a contract that declares a fixture reset
|
|
141
|
-
needs the caller's probe policy to authorize that operation's method as well as the read methods
|
|
142
|
-
every other pre-flight leg uses. Engine integration is a later adapter behind a port. See
|
|
143
|
-
[ADR-004](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-004-execution-boundary.md).
|
|
87
|
+
It executes nothing. No agent, no judge, and no system under test runs inside it. You run the agent or harness, the repeated trials, and the live system with its environment probe, and you hand over a sealed run record. `eval-quality` compiles, seals, preflights, and scores.
|
|
144
88
|
|
|
145
89
|
## Who it is for
|
|
146
90
|
|
|
147
|
-
Teams shipping AI agents, coding skills, review bots, MCP-based assistants, or automated test-generation systems, and teams
|
|
91
|
+
Teams shipping AI agents, coding skills, review bots, MCP-based assistants, or automated test-generation systems, and teams that let agents ship with little or no human review.
|
|
148
92
|
|
|
149
93
|
Use `eval-quality` when all three are true:
|
|
150
94
|
|
|
@@ -154,57 +98,43 @@ Use `eval-quality` when all three are true:
|
|
|
154
98
|
|
|
155
99
|
Deterministic work does not need it and already has cheaper, stronger evidence from unit, integration, contract, E2E and performance testing.
|
|
156
100
|
|
|
157
|
-
##
|
|
101
|
+
## The authoring discipline
|
|
158
102
|
|
|
159
|
-
A
|
|
103
|
+
A contract carries no prescribed action sequence; the evaluator chooses its own path. What the contract fixes is seven rules that survived the experiments: separate the success indicator from the body, read the whole body, probe malformed and negative inputs, verify per record, cross-check sibling parameters and sibling tools, check for omissions and completeness, and read a state change back after writing it.
|
|
160
104
|
|
|
161
|
-
The
|
|
105
|
+
The compiler enforces those rules against the contract artifact, in three classes. Structural errors fail compilation. Coverage gaps score down without blocking. A waived rule is allowed when the waiver records the rule name, a rationale, a machine-checkable condition, and the approval.
|
|
162
106
|
|
|
163
|
-
|
|
107
|
+
Rubrics compile under the same discipline: an anchored scale, named criteria that each state a question, and evidence pointers that resolve against the declared interfaces.
|
|
164
108
|
|
|
165
|
-
|
|
109
|
+
## How contract strength is scored
|
|
166
110
|
|
|
167
|
-
|
|
111
|
+
Do not trust a contract because it looks thorough. Put a known defect behind it, run the evaluator, and check whether the contract's oracles caused the defect to be caught.
|
|
168
112
|
|
|
169
|
-
|
|
170
|
-
runs the same functions over the committed worked chain, and
|
|
171
|
-
[the full walkthrough](https://bmad-code-org.github.io/bmad-eval-quality/how-to/author-behavioral-contracts/)
|
|
172
|
-
reads the result field by field. Do not trust a contract because it looks thorough. Put a known defect behind it, run the evaluator, and check whether the contract's oracles caused the defect to be caught.
|
|
173
|
-
|
|
174
|
-
Two probe classes go behind a contract, and a strong contract rejects both:
|
|
113
|
+
Three probe classes go behind a contract, and a strong contract catches all three:
|
|
175
114
|
|
|
176
115
|
- **Defect probes**, where the behavior is simply wrong.
|
|
177
116
|
- **Gameability probes**, where the behavior looks compliant while dodging the oracle's intent. A test that raises coverage while asserting nothing is the familiar version of this.
|
|
117
|
+
- **Zero-action probes**, where the system does nothing and reports success.
|
|
178
118
|
|
|
179
|
-
|
|
119
|
+
Canary probes and clean controls are run too, and never enter the strength vector.
|
|
180
120
|
|
|
181
|
-
Every required oracle check resolves to exactly one
|
|
182
|
-
"the check reported" is never sufficient on its own: `caught`, `confirmed`, `missed`,
|
|
183
|
-
`passed-clean-control`, `false-positive`, `abstained`, `bypassed`, `unreached`, `oracle-error`,
|
|
184
|
-
`judge-error`, `infrastructure-error`, or `not-applicable`.
|
|
121
|
+
Every required oracle check resolves to exactly one of twelve states, and the state travels with the result: `caught`, `confirmed`, `missed`, `passed-clean-control`, `false-positive`, `abstained`, `bypassed`, `unreached`, `oracle-error`, `judge-error`, `infrastructure-error`, or `not-applicable`. A required oracle that missed, abstained, errored, or is absent prevents PASS, and a high overall score never overrides it. An infrastructure error or a failed pre-flight invalidates the run; run it again.
|
|
185
122
|
|
|
186
|
-
A
|
|
123
|
+
A caught defect is decided by evidence. A finding counts as detection only when the probe's declared defect signature matches an observation the finding cites; the evaluator's own claim does not settle it.
|
|
187
124
|
|
|
188
|
-
Repeated runs of one probe are trials,
|
|
125
|
+
Repeated runs of one probe are trials, reduced to one result per probe before any rate is computed. The `score` command and `runScore` hand the stage one sealed run record per call, so against a policy that asks for more than one trial the strength vector is reported and marked non-comparable.
|
|
189
126
|
|
|
190
|
-
|
|
127
|
+
[The full walkthrough](https://bmad-code-org.github.io/bmad-eval-quality/how-to/author-behavioral-contracts/) reads a scored run field by field, down to its verdict and exit code.
|
|
191
128
|
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
The **library** is the primary surface. It exports the artifact types, the compiler, the pre-flight, `runScore`, the canonical digest, the lineage validator, and the failure-code and verdict registries. The Zod schemas themselves are not exported; they are published as JSON Schema under `eval-quality/schemas/*`. The published typed schema is what lets coding agents author contracts correctly by default, which is how the discipline scales beyond the people who went looking for the tool.
|
|
195
|
-
|
|
196
|
-
The **CLI** wraps the same library for callers that cannot import TypeScript: CI jobs, GitHub Actions, PR-review and unit-test bots, other frameworks' skills, and any agent permitted to run a shell command.
|
|
197
|
-
|
|
198
|
-
### What the CLI Commands Do
|
|
129
|
+
## Using it
|
|
199
130
|
|
|
200
|
-
|
|
201
|
-
- **`seal`**: Generates a `sealed-evaluator-brief.json` by stripping secret defect signatures, planted answers, and author commentary. The brief carries only the directions and safety bounds the evaluator needs.
|
|
202
|
-
- **`preflight`**: Reduces caller-supplied probe observations against the contract to verify environment baseline readiness and probe reachability. All four of `--contract`, `--probes`, `--observations`, and `--run-id` are required. Halts early with exit code `3` if the environment is unready. `schemas/probe.schema.json` gives the shape of one probe in the list; each observation echoes a planned leg's id back as `probeId`, and the getting-started tutorial writes six by hand.
|
|
203
|
-
- **`score`**: Chains ingest, score, and emit over one sealed run record, minting `evidence-artifact.json` and exiting with the AD-21 verdict's own exit code. `schemas/sealed-run-record.schema.json` gives the record's shape, and the committed worked chain under `_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/spike-worked-example/` carries one, scored. `--record`, `--contract`, `--probe`, `--preflight-verdict`, `--policy`, and `--corpus-digest` are required; `--isolation-manifest` and `--evaluator-configuration` are each optional and their absence invalidates the run and the command still parses; `--private-manifest` is optional and, when given, each entry's declared digest is checked against its resolved bytes. `--corpus-root` names the directory a private reference resolves under, and is required only when `--private-manifest` or a private-storage isolation-manifest reference is actually present.
|
|
131
|
+
Node.js 22.20.0 or newer. `zod` is the only production dependency.
|
|
204
132
|
|
|
205
|
-
|
|
133
|
+
```bash
|
|
134
|
+
npm install eval-quality
|
|
135
|
+
```
|
|
206
136
|
|
|
207
|
-
Every command runs through `npx
|
|
137
|
+
Every command runs through `npx`:
|
|
208
138
|
|
|
209
139
|
```bash
|
|
210
140
|
npx eval-quality compile --in contract.json --out ./eval-out
|
|
@@ -221,18 +151,7 @@ npx eval-quality score --record record.json --contract contract.json \
|
|
|
221
151
|
--out ./eval-out
|
|
222
152
|
```
|
|
223
153
|
|
|
224
|
-
Every command is non-interactive
|
|
225
|
-
stdin is a pipe. Each one is a single call into the library plus artifact serialization.
|
|
226
|
-
|
|
227
|
-
**Input and output.** `--in` is the only input that falls back to stdin: `compile` and `seal` read it
|
|
228
|
-
when `--in` is left out. `-` names stdin explicitly on any input, and at most one input may be `-` per
|
|
229
|
-
invocation. `compile` and `seal` each take one input; `preflight` takes three, all required; `score`
|
|
230
|
-
takes eight, three of them optional (`--isolation-manifest`, `--evaluator-configuration`, and
|
|
231
|
-
`--private-manifest`). Without `--out` the artifact goes to stdout, so a command composes with a pipe.
|
|
232
|
-
An `--out` ending in `.json` is a file path; anything else is a directory, and the artifact is
|
|
233
|
-
written to `<target>/<kind>.json` where `kind` is `eval-contract`, `sealed-evaluator-brief`,
|
|
234
|
-
`preflight-verdict`, or `evidence-artifact`. Diagnostics and errors go to stderr, always, so stdout
|
|
235
|
-
carries the artifact alone.
|
|
154
|
+
Every command is non-interactive. Without `--out` the artifact goes to stdout, so a command composes with a pipe. An `--out` ending in `.json` is a file path; anything else is a directory, and the artifact lands at `<target>/<kind>.json`. Diagnostics and errors go to stderr, so stdout carries the artifact alone.
|
|
236
155
|
|
|
237
156
|
**Exit codes.**
|
|
238
157
|
|
|
@@ -246,81 +165,51 @@ carries the artifact alone.
|
|
|
246
165
|
| `5` | runtime fault |
|
|
247
166
|
| `64` | usage error |
|
|
248
167
|
|
|
249
|
-
`--strict` never promotes a CONCERNS whose firing conditions are all evidence conditions: those
|
|
250
|
-
|
|
251
|
-
`
|
|
252
|
-
condition behind code 3 is reachable through `score` too, alongside the failed pre-flight `preflight`
|
|
253
|
-
itself reports.
|
|
168
|
+
`--strict` never promotes a CONCERNS whose firing conditions are all evidence conditions: those report that the measurement fell short of the policy. Codes 1 and 2 come from `score`'s verdict ladder. Code 3 comes from a failed pre-flight, which `preflight` reports, or from any other invalidating condition `score` finds.
|
|
169
|
+
|
|
170
|
+
`--strict` is accepted on every command. `--strict-inputs` and `--no-strict-inputs` are a different switch: they set the compiler's input strictness, on by default, and only `compile` and `seal` accept them.
|
|
254
171
|
|
|
255
|
-
|
|
256
|
-
`--no-strict-inputs` are a different switch: they set the compiler's input strictness, which is on
|
|
257
|
-
by default, and `preflight` and `score` each reject both with exit `64` because neither has a compile
|
|
258
|
-
step.
|
|
172
|
+
The [CLI reference](https://bmad-code-org.github.io/bmad-eval-quality/reference/cli-commands/) has every flag, every parsing rule, and the package exports.
|
|
259
173
|
|
|
260
|
-
**The published JSON Schema
|
|
261
|
-
twelve generated documents, published at the `eval-quality/schemas/*` subpath:
|
|
174
|
+
**The library** exports the same stages as functions, plus the artifact types, the canonical digest, the lineage validator, and the failure-code and verdict registries. The schemas are published as JSON Schema under `eval-quality/schemas/*`, which is what lets a coding agent author contracts correctly by default:
|
|
262
175
|
|
|
263
176
|
```ts
|
|
264
177
|
import spec from 'eval-quality/schemas/eval-contract.schema.json' with { type: 'json' }
|
|
265
178
|
```
|
|
266
179
|
|
|
267
|
-
The import attribute is required: ESM on Node 22 and 24 both throw `ERR_IMPORT_ATTRIBUTE_MISSING`
|
|
268
|
-
without it. The development corpus ships the same way, at `eval-quality/corpus/dev/`, so an adopter
|
|
269
|
-
can read real compiled contracts and one compiled-and-sealed pair without cloning this repository.
|
|
270
|
-
`eval-quality/adapters` is one of the five published subpaths, holding the four reference adapters
|
|
271
|
-
the conformance suite runs against.
|
|
272
|
-
|
|
273
|
-
Eleven of the twelve published schemas carry a `schemaVersion`. `artifact-reference` is exempt: it
|
|
274
|
-
is embedded inside other artifacts, so it has no version to break.
|
|
180
|
+
The import attribute is required: ESM on Node 22 and 24 both throw `ERR_IMPORT_ATTRIBUTE_MISSING` without it. The development corpus ships the same way, at `eval-quality/corpus/dev/`, so you can read twenty-one real contracts and one compiled-and-sealed pair without cloning this repository.
|
|
275
181
|
|
|
276
|
-
|
|
277
|
-
parses. The bumps in the next release each add a required field, which is why an older document
|
|
278
|
-
fails; the version itself is compared in exactly one place, `validateLineageChain`, over
|
|
279
|
-
lineage-chain members, and nowhere on the command path. The package is pre-1.0, so pin exactly.
|
|
280
|
-
`CHANGELOG.md` records what each release breaks.
|
|
182
|
+
Version 1.0 is out and the published surface is stable: a breaking change to a command, an export, or a schema is a major version bump. `compile` refuses a contract whose `schemaVersion` differs from the one this build reads, so check the stamp on anything you did not author against this version. `CHANGELOG.md` records what each release breaks.
|
|
281
183
|
|
|
282
184
|
## Relationship with BMad and TEA
|
|
283
185
|
|
|
284
|
-
The dependency runs one way: TEA uses `eval-quality`, and `eval-quality` knows nothing about TEA.
|
|
186
|
+
TEA is the BMad Test Architect. The dependency runs one way: TEA uses `eval-quality`, and `eval-quality` knows nothing about TEA.
|
|
285
187
|
|
|
286
188
|
```mermaid
|
|
287
189
|
graph LR
|
|
288
190
|
TEA["TEA<br/>(reference authoring client)"] -- "drafts a contract, then calls" --> EQ["eval-quality<br/>(this package)"]
|
|
289
191
|
```
|
|
290
192
|
|
|
291
|
-
TEA is the reference authoring client. It reads BMad planning artifacts, notices eval-relevant work, drafts a contract, and calls this package.
|
|
193
|
+
TEA is the reference authoring client. It reads BMad planning artifacts, notices eval-relevant work, drafts a contract, and calls this package. Any human, bot, CI job, skill, or other framework can author a contract and use `eval-quality` directly. The compiler judges the artifact, whoever produced it.
|
|
292
194
|
|
|
293
|
-
|
|
195
|
+
### Example: testing a `bmad-tea` knowledge harness
|
|
294
196
|
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
3. Run `eval-quality seal --in contract.json --out ./run` to generate `sealed-evaluator-brief.json`.
|
|
301
|
-
4. Probe the harness's environment and run `eval-quality preflight` over the observations; a verdict that does not pass is exit `3`, and the run stops there.
|
|
302
|
-
5. Pass `sealed-evaluator-brief.json` to `bmad-tea` to execute the task without seeing answer keys, once against the clean harness and once against a harness with one known step file removed.
|
|
197
|
+
1. Author an `eval-contract.json` declaring the required knowledge step files (for example `playwright-utils-mandate.md`).
|
|
198
|
+
2. Run `eval-quality compile --in contract.json` to check the contract's structure and discipline rules.
|
|
199
|
+
3. Run `eval-quality seal --in contract.json --out ./run` to produce `sealed-evaluator-brief.json`.
|
|
200
|
+
4. Probe the harness's environment and run `eval-quality preflight` over the observations. A verdict that does not pass is exit `3`, and the run stops there.
|
|
201
|
+
5. Hand `sealed-evaluator-brief.json` to `bmad-tea` to execute the task without seeing answer keys, once against the clean harness and once against a harness with one known step file removed.
|
|
303
202
|
6. Seal each evaluator run into a `sealed-run-record.json` and run `eval-quality score` over it with the probe that names the removed file as the seeded defect. The clean run should pass; the mutated run should degrade, and the exit code says which.
|
|
304
203
|
|
|
305
204
|
## Evidence and limitations
|
|
306
205
|
|
|
307
|
-
Holding the model, the budget, the system, and the defects fixed, and changing only how the
|
|
206
|
+
Holding the model, the budget, the system, and the defects fixed, and changing only how the eval contract was authored, sealed-evaluator detection moved from **0.33 to 1.00** across three naturally occurring defects, three repetitions per arm, 19 scored runs.
|
|
308
207
|
|
|
309
|
-
Both experiment rounds missed at least one preregistered gate
|
|
208
|
+
Both experiment rounds missed at least one preregistered gate; round 2 failed one gate of five on a single unreplicated clean control. The separation comes from two of the three defects, since both arms detected the third in every repetition, and both separating cases carry a recorded measurement-layer confound. The sample covers three defects, one system, and one model. This supports a product-direction decision at narrow scale. Certification would require broader replication.
|
|
310
209
|
|
|
311
210
|
Read the [product brief](_bmad-output/planning-artifacts/briefs/brief-eval-quality-2026-07-17/brief.md) for the product rationale and the [PRD](_bmad-output/planning-artifacts/prds/prd-eval-quality-2026-07-17/prd.md) for build requirements. The experiment record includes the [round 1 verdict](experiments/hypothesis-validation/DECISION.md), [round 2 results](experiments/hypothesis-validation/PHASE2-RESULTS.md), [metric summary](experiments/hypothesis-validation/results/summary.md), and [protocol](experiments/hypothesis-validation/HYPOTHESIS_VALIDATION_PLAN.md).
|
|
312
211
|
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
The [architecture spine](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ARCHITECTURE-SPINE.md) is split by pipeline half, and its own status line still reads that the compile-and-seal half is epic-ready while the score half is not. The code has moved past that line. Epic 7 delivered AD-21, AD-33, and AD-40 as pure functions with generated tables, and epic 8 shipped the `ingest`, `score`, and `emit` stages, the `score` command, and `runScore` over them, which closes every item the spine's *Owed to the reference implementation* section listed. The `score` stage consumes a trial set; the command and `runScore` hand it one record per call, so a run scored from the published surface completes one trial, and whenever the policy's declared minimum exceeds one its strength vector is reported and marked non-comparable. Gate C closed at zero blocking authoring points and 14 of 14 declaration-only predicates. Gate D's generated-current-fields arm matched the hand-written positive control at 3 of 3 seeded-defect catches, so `seal` joins the stage-one order without adding an evidence-precondition field.
|
|
316
|
-
|
|
317
|
-
Contract strength scoring has been open since [ADR-007](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-007-compile-score-split.md): three rounds of external review established that the catch rate was 1.00 by construction, because nothing matched a finding to the defect its probe seeded. That input now exists, and so does the mapping that reads it: `src/core/score/witness.ts` is AD-40's witness match, delivered by epic 7. What is still owed is its validation against the block-2 replication, which the spine records as committed and not yet run.
|
|
318
|
-
|
|
319
|
-
Contract compilation was declared ready in ADR-007 and a fourth review withdrew that claim in [ADR-008](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-008-compile-half-owed-to-calibration.md). The named calibration is now complete. The absent local-only mut2 arm was reconstructed from its recorded base, reproduced its prior black-box behavior, and ran under a pre-registered three-arm, three-repetition design. All three arms composed filters and detected the seeded defect in every valid repetition. This closes the calibration gate narrowly; it does not generalize the historical 0.33-to-1.00 effect beyond one behavior and one controlled mutation.
|
|
320
|
-
|
|
321
|
-
Both are documented as defects, because four rounds have shown that a confidently worded revision is the thing that goes wrong here.
|
|
322
|
-
|
|
323
|
-
The decision record, in order: [ADR-001](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-22/ADR-001-evaluator-isolation-boundary.md) on evaluator isolation, [ADR-002](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-22/ADR-002-contract-authoring-discipline.md) on why authoring discipline is the product, [ADR-003](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-003-measurement-mechanics.md) on measurement mechanics, [ADR-004](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-004-execution-boundary.md) on why this package executes nothing, [ADR-005](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-005-review-round-corrections.md) and [ADR-006](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-006-interaction-plan.md) on what review and hand-authoring corrected, [ADR-007](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-007-compile-score-split.md) on the split, [ADR-008](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-008-compile-half-owed-to-calibration.md) on why the other half stopped claiming to be finished too, and [ADR-009](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ADR-009-adversarial-gate-corrections.md) on the seventeen places where two conforming implementations still disagreed. Review triage lives in [`reviews/`](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/reviews/).
|
|
212
|
+
The design record is the [architecture spine](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/ARCHITECTURE-SPINE.md), its nine decision records, and the review triage in [`reviews/`](_bmad-output/planning-artifacts/architecture/architecture-eval-quality-2026-07-29/reviews/).
|
|
324
213
|
|
|
325
214
|
## Not building now
|
|
326
215
|
|
|
@@ -330,97 +219,32 @@ Out of scope entirely: a new eval engine, a hosted service, a dashboard or GUI,
|
|
|
330
219
|
|
|
331
220
|
## Development
|
|
332
221
|
|
|
333
|
-
Node `>=22.20.0`, which `package.json` declares as the engine floor. `zod` is the only production
|
|
334
|
-
dependency.
|
|
335
|
-
|
|
336
222
|
```bash
|
|
337
223
|
npm install
|
|
338
|
-
npm run validate #
|
|
224
|
+
npm run validate # the whole gate: build, typecheck, lint, every drift check, tests with coverage
|
|
339
225
|
npm run build # emit to dist/
|
|
226
|
+
npm run test # run the suite once
|
|
340
227
|
npm run lint:fix # auto-fix with Biome
|
|
341
|
-
npm run test:coverage # run the suite and fail below AD-30's 90 percent statement and branch floor on core/
|
|
342
|
-
npm run generate:schemas # rebuild schemas/*.schema.json from the Zod source
|
|
343
|
-
npm run check:schemas # fail if the committed schemas differ from the source by one byte
|
|
344
|
-
npm run check:ad5-registry # fail if the compile-time failure-code list drifts from the AD-5 table
|
|
345
|
-
npm run check:ad28-registry # fail if the runtime fault-code list drifts from the AD-28 table
|
|
346
|
-
npm run check:lineage # fail if a module outside the stage table writes an artifact's lineage fields
|
|
347
|
-
npm run check:boundary # fail if anything the tarball carries references the planning system that produced it
|
|
348
|
-
npm run generate:ad21-table # rebuild docs/ad21-verdict-decision.generated.md from the two verdict ladders
|
|
349
|
-
npm run check:ad21-table # fail if the committed AD-21 table differs from the builder by one byte
|
|
350
|
-
npm run generate:ad31-table # rebuild docs/ad31-coverage-predicates.generated.md from the predicates
|
|
351
|
-
npm run check:ad31-table # fail if the committed AD-31 table differs from the builder by one byte
|
|
352
|
-
npm run generate:ad33-table # rebuild docs/ad33-outcome-decision.generated.md from the decision procedure
|
|
353
|
-
npm run check:ad33-table # fail if the committed AD-33 table differs from the builder by one byte
|
|
354
|
-
npm run generate:dev-corpus # rebuild corpus/dev/ from the contract fixtures through the shipped compile and seal
|
|
355
|
-
npm run check:corpus # fail if the committed corpus differs from the builder by one byte
|
|
356
|
-
npm run generate:worked-example # rebuild the spike worked chain by running the compile, seal, score, and emit functions over it
|
|
357
|
-
npm run check:worked-example # fail if the committed worked chain differs from the builder by one byte
|
|
358
|
-
npm run build:shareable # render the planning artifacts to self-contained HTML
|
|
359
228
|
npm run test:conformance # run the published port conformance suite against every shipped adapter
|
|
360
229
|
```
|
|
361
230
|
|
|
362
|
-
|
|
363
|
-
committed. They are the contract for consumers who do not read TypeScript, so they are proven
|
|
364
|
-
equivalent to the source: a byte-exact drift check, a rejection suite
|
|
365
|
-
asserting the validator keyword and instance path for every negative fixture, a differential check
|
|
366
|
-
comparing Zod's verdict against a third-party validator's over a generated corpus, and a
|
|
367
|
-
keyword-mutation sweep that deletes each published constraint and requires some fixture to notice.
|
|
368
|
-
Edit the Zod schema and regenerate; never hand-edit a file under `schemas/`.
|
|
369
|
-
|
|
370
|
-
Every artifact the library hands back is deep-frozen, so it cannot be changed in place. This package
|
|
371
|
-
is ES modules, which are always strict, so an attempt throws a `TypeError` there; a sloppy-mode
|
|
372
|
-
caller sees the write fail silently. A revision is minted as a new artifact carrying its parent's
|
|
373
|
-
digest and a revision count one greater. `check:lineage` fails the build when a lineage field is
|
|
374
|
-
written outside `src/core/schemas/`, `src/core/lineage/`, and the modules the AD-24 stage table
|
|
375
|
-
names as that artifact's producer, which today are `src/core/seal/seal.ts`,
|
|
376
|
-
`src/core/preflight/reduce.ts`, and `src/core/emit/emit.ts`.
|
|
377
|
-
|
|
378
|
-
The `eval-quality/conformance` subpath publishes the port boundary: the four port types, the message
|
|
379
|
-
shapes they carry, the AD-28 `RUNTIME_FAULT_CODES` registry and `RuntimeFaultCode` type a conforming
|
|
380
|
-
adapter throws against, and an executable conformance suite. An adapter is conforming when
|
|
381
|
-
`runCorpusPortConformance`, `runClockPortConformance`, `runFileSystemPortConformance`,
|
|
382
|
-
`runEnvironmentProbePortConformance`, or `runCommandLineProbeConformance` returns a report whose
|
|
383
|
-
`passed` is true, which is the definition; each returns a report, so the suite carries no test
|
|
384
|
-
framework and runs under whichever one you already use.
|
|
231
|
+
Several files are generated from the code and guarded byte for byte, so a hand edit fails the build. Regenerate them:
|
|
385
232
|
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
233
|
+
| Generated | Rebuild | Guard |
|
|
234
|
+
| --- | --- | --- |
|
|
235
|
+
| `schemas/*.schema.json`, from the Zod source | `npm run generate:schemas` | `npm run check:schemas` |
|
|
236
|
+
| `corpus/dev/`, from the contract fixtures | `npm run generate:dev-corpus` | `npm run check:corpus` |
|
|
237
|
+
| `docs/ad21-verdict-decision.generated.md`, the two verdict ladders | `npm run generate:ad21-table` | `npm run check:ad21-table` |
|
|
238
|
+
| `docs/ad31-coverage-predicates.generated.md`, the coverage predicates | `npm run generate:ad31-table` | `npm run check:ad31-table` |
|
|
239
|
+
| `docs/ad33-outcome-decision.generated.md`, the outcome decision procedure | `npm run generate:ad33-table` | `npm run check:ad33-table` |
|
|
240
|
+
| the committed worked chain | `npm run generate:worked-example` | `npm run check:worked-example` |
|
|
241
|
+
| `_bmad-output/shareable/`, this README, CONTRIBUTING, and the planning artifacts as standalone HTML | `npm run build:shareable` | `npm run check:shareable` |
|
|
242
|
+
|
|
243
|
+
Every artifact the library hands back is deep-frozen. A revision is a new artifact carrying its parent's digest and a revision count one greater, and `npm run check:lineage` fails the build when a lineage field is written outside the modules that own it. `npm run check:boundary` fails it when anything the tarball carries references the planning system that produced it.
|
|
244
|
+
|
|
245
|
+
`eval-quality/conformance` publishes the port boundary: the four port types, the fault registry a conforming adapter throws against, and an executable conformance suite that returns a report and runs under whichever test framework you already use. The [CLI reference](https://bmad-code-org.github.io/bmad-eval-quality/reference/cli-commands/) describes the ports and the suite.
|
|
389
246
|
|
|
390
|
-
|
|
391
|
-
mechanism failure is a typed fault, exactly one underlying call happens on success and on failure, an
|
|
392
|
-
aborted signal rejects promptly, an in-band error value is thrown as a fault, and a
|
|
393
|
-
successful call returns a response the published schema accepts. `EnvironmentProbePort` adds a second
|
|
394
|
-
arm per mechanism: `runEnvironmentProbePortConformance` (`api`) adds thirteen more from AD-35's
|
|
395
|
-
default-deny target policy, and `runCommandLineProbeConformance` (`cli`) adds nine, once
|
|
396
|
-
`CommandTargetPolicy` gave that mechanism something to authorize — an unmapped interface, an
|
|
397
|
-
unmapped executable, an unauthorized subcommand path, a non-zero exit read as an observation, a
|
|
398
|
-
shell-metacharacter argument proven to reach the process as one literal token, a declared artifact
|
|
399
|
-
captured, and both caps enforced. `npm run test:conformance` runs the suite against the four
|
|
400
|
-
adapters this package ships and against two in-repository probe subjects, one per mechanism, that
|
|
401
|
-
exist only as the suite's own subjects.
|
|
402
|
-
|
|
403
|
-
`docs/ad21-verdict-decision.generated.md` holds AD-21's two published verdict ladders, production and
|
|
404
|
-
contract-scoring, emitted from the rule tables in `src/core/score/ladder.ts` together with the
|
|
405
|
-
fixtures that exercise them, and guarded by `npm run check:ad21-table`. Each row carries its
|
|
406
|
-
condition, its rung, the guard in prose, and whether `--strict` may promote it.
|
|
407
|
-
|
|
408
|
-
`docs/ad31-coverage-predicates.generated.md` holds AD-31's published predicate table, emitted from
|
|
409
|
-
the seven relevance predicates and their seven satisfaction twins run over a hand-authored contract
|
|
410
|
-
corpus. It is generated by `npm run generate:ad31-table` and guarded by `npm run check:ad31-table`,
|
|
411
|
-
a byte-exact drift check that fails when a predicate changes and the committed document does not, so
|
|
412
|
-
the table is evidence the predicates produce. A hand edit fails the check; regenerate.
|
|
413
|
-
|
|
414
|
-
`docs/ad33-outcome-decision.generated.md` holds AD-33's published decision table: the ten
|
|
415
|
-
invalidating conditions, the twenty-row outcome ladder, the two waiver rules, the eight
|
|
416
|
-
corroboration rules, the named structural constraints with the infeasible input pairs derived from
|
|
417
|
-
them, and five censuses over the fixture set. It is generated by `npm run generate:ad33-table` and
|
|
418
|
-
guarded by `npm run check:ad33-table`, the same byte-exact drift check, and the builder refuses to
|
|
419
|
-
publish a census cell at zero, so a rule or a state losing its last fixture fails the build. AD-33
|
|
420
|
-
puts a cell-per-input-tuple table out of arithmetic reach, so what is published is the enumerated
|
|
421
|
-
output of the total function itself. A hand edit fails the check; regenerate.
|
|
422
|
-
|
|
423
|
-
`build:shareable` renders this README, the product brief, the PRD, the architecture spine, all nine ADRs, and every document those pages link to (contributing, code of conduct, security, licence, and the four experiment records) to `_bmad-output/shareable/` as standalone styled HTML for sharing outside the repo. Rendering the linked documents is what lets a recipient without repository access follow the evidence, contribution, security, and licence links; anything that has no page of its own, such as a directory, is marked in the export as needing repository access. A hand edit fails the check; regenerate: `check:shareable` fails the build when the committed export is stale or carries a repository URL that is not the canonical one. Mermaid diagrams render as code blocks there, which is a known limitation.
|
|
247
|
+
[CONTRIBUTING.md](CONTRIBUTING.md) covers the gate and the release process.
|
|
424
248
|
|
|
425
249
|
## Contributing
|
|
426
250
|
|
package/corpus/dev/README.md
CHANGED
|
@@ -1,43 +1,39 @@
|
|
|
1
1
|
# Development corpus
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Twenty-one contracts and one compiled-and-sealed pair, published so an adopter can read real input to
|
|
4
4
|
this package without cloning the repository. Everything here is generated by
|
|
5
5
|
`npm run generate:dev-corpus` and checked byte for byte by `npm run check:corpus`.
|
|
6
6
|
|
|
7
7
|
## What is here
|
|
8
8
|
|
|
9
|
-
- `contracts/<contractId>.json`: twenty-one contracts. Nineteen are one per
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
9
|
+
- `contracts/<contractId>.json`: twenty-one contracts. Nineteen are one per discipline rule in each
|
|
10
|
+
declaration state, and two describe a system under test that runs behind a command. Eighteen are
|
|
11
|
+
published only after this package's own compile stage accepts them, so every one of those is a
|
|
12
|
+
contract the compiler admits. Three fail compilation by design; those ship as authored input, and
|
|
13
|
+
`index.json` records the failure code each one raises.
|
|
14
14
|
- `compile-seal-example/contract.json` and `compile-seal-example/brief.json`: one contract and
|
|
15
15
|
the brief this package's compile-then-seal boundary produces from it.
|
|
16
|
-
- `index.json`: every file above, its kind, and the
|
|
16
|
+
- `index.json`: every file above, its kind, and the digest of the exact bytes on disk.
|
|
17
17
|
|
|
18
18
|
## These contracts are visible and diagnostic
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
## What is absent
|
|
25
|
-
|
|
26
|
-
**The qualified-probe dimensions are absent.**
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
`ingest
|
|
39
|
-
|
|
40
|
-
configuration exist only as the authored values that build passes to `ingest`, never serialized to
|
|
41
|
-
a file. This directory still ships only the compile-and-seal pair, scoped to what a corpus of
|
|
42
|
-
contracts needs,
|
|
43
|
-
under a name that does not claim AD-38's term.
|
|
20
|
+
Nothing here is a holdout: every contract is published, readable, and meant to be read while writing
|
|
21
|
+
your own. A holdout set that measures a contract's strength is a separate thing this package does not
|
|
22
|
+
ship.
|
|
23
|
+
|
|
24
|
+
## What is absent
|
|
25
|
+
|
|
26
|
+
**The qualified-probe dimensions are absent.** A qualified probe is one whose qualification record and
|
|
27
|
+
defect signature agree with its class, and a probe corpus would carry at least one per probe class and
|
|
28
|
+
per `expectedClean` state. The probe schema carries both halves, and the trial reducer and the
|
|
29
|
+
score stage that reads it are both shipped, so an admitted probe can be scored end to end today. This
|
|
30
|
+
directory's own gate does not yet require one.
|
|
31
|
+
|
|
32
|
+
**Three of the four artifacts in an end-to-end example are absent here.** Such an example is a sealed
|
|
33
|
+
brief, a conforming sealed run record, an isolation manifest, and an evaluator configuration. The last
|
|
34
|
+
three are inputs the shipped `ingest` stage consumes, authored for the worked example in
|
|
35
|
+
`scripts/worked-example-target.ts` and exercised there through `ingest`/`score`/`emit`. Only
|
|
36
|
+
the run record among them is committed, as `spike-worked-example/sealed-run-record.json`; the
|
|
37
|
+
isolation manifest and the evaluator configuration exist only as the authored values that build
|
|
38
|
+
passes to `ingest`. This directory ships only the compile-and-seal pair, scoped to what a corpus of
|
|
39
|
+
contracts needs.
|
package/corpus/dev/index.json
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"entries":[{"digest":"sha256:
|
|
1
|
+
{"entries":[{"digest":"sha256:419dfab17baf109b84e2f971ab85c399ff94712f562bf6e74d1f3e9093b68b18","kind":"readme","path":"corpus/dev/README.md"},{"digest":"sha256:2a5a248cea57e55ba6f79eaddc5e48e156917b4b91866fa349985c8d651a94cb","kind":"sealed-evaluator-brief","path":"corpus/dev/compile-seal-example/brief.json"},{"digest":"sha256:8f9dea3220baedd8a07da83ca4dd804c943cebda99c6754734694e48b8007e1c","kind":"contract","path":"corpus/dev/compile-seal-example/contract.json"},{"digest":"sha256:ec77b19ba1d3c8537ff74cc656f67c4ac46f0d0040e65422b3d742687a97e39c","kind":"contract","path":"corpus/dev/contracts/absent-collection-locations.json"},{"digest":"sha256:411c6a8c074ab6e524368605812532cc932fffc1ff20e68a65b50145fc294181","kind":"contract","path":"corpus/dev/contracts/absent-sibling-groups.json"},{"digest":"sha256:b6d22168553303903df5513da24edd01bf2f780a7f8dc3c0ed0d66a0bc6c7ba8","kind":"contract","path":"corpus/dev/contracts/absent-success-indicator.json"},{"digest":"sha256:eca39a841b40a67595e06343119f63be471af8110139b0245d6b18d18d0be4bb","kind":"contract","path":"corpus/dev/contracts/empty-channel-roles.json"},{"digest":"sha256:56aa60b674f70b4f96be8b982ffe384b7825caf886d1ae8107e2019beeee5d19","kind":"contract","path":"corpus/dev/contracts/empty-collection-locations.json"},{"digest":"sha256:c6ba5efeac1e3c5dfa096375225b98fd80a6fee4f28f4546df21f18baacce663","kind":"contract","path":"corpus/dev/contracts/empty-request-shapes.json","structuralFailure":"unreachable-check-evidence"},{"digest":"sha256:703d3e1468b6e058d728fd909d170f130d9539dc27822b16c539c5e9a3c908c1","kind":"contract","path":"corpus/dev/contracts/empty-sibling-groups.json"},{"digest":"sha256:f7983813b294cc114c35efd3efeb1db315d40ada56ae24f22b575046e6a76a14","kind":"contract","path":"corpus/dev/contracts/fragment-selection.json"},{"digest":"sha256:3e912f4423aa529fec21364155197334506316cdd7435ed1b59969eaab9cadf4","kind":"contract","path":"corpus/dev/contracts/no-collection-quantifier.json"},{"digest":"sha256:10e142a5ea8130c83c8dbe3be1c8df67c2997e8b57da0394cccbcfc43f44326f","kind":"contract","path":"corpus/dev/contracts/no-operation-inventory.json","structuralFailure":"unreachable-check-evidence"},{"digest":"sha256:8eace1550e26534eb886c9f217d4a8999891072643535b11afda44efc21d736f","kind":"contract","path":"corpus/dev/contracts/no-read-back-relation.json"},{"digest":"sha256:f66980583dde3213b65d5460491fe2dc34961cc9ee44600a40a2d10ba260efed","kind":"contract","path":"corpus/dev/contracts/no-state-change-marker.json","structuralFailure":"undeclared-mandatory-input"},{"digest":"sha256:deb36540d39aa99ae1d5d7e2e2808211ed40319fcc0d5519500e478659d8d232","kind":"contract","path":"corpus/dev/contracts/no-type-violating-step.json"},{"digest":"sha256:aa0ffb76c69870798b949e2e5a6a93b0bd46507fa7124505878b45a0672bc7e5","kind":"contract","path":"corpus/dev/contracts/per-key-split-oracles.json"},{"digest":"sha256:e4eaeb0e38357912dffb4bc7d29730797f9a1e485b4d98616fc934e83a57b732","kind":"contract","path":"corpus/dev/contracts/review-corpus.json"},{"digest":"sha256:8f9dea3220baedd8a07da83ca4dd804c943cebda99c6754734694e48b8007e1c","kind":"contract","path":"corpus/dev/contracts/satisfied-declarations.json"},{"digest":"sha256:951a90c269944c9e29c1b06ba4d634dbe6d2a1847308a951e7a5b6e4d271e2ff","kind":"contract","path":"corpus/dev/contracts/single-required-response-key.json"},{"digest":"sha256:fd696c819b7e2ada4dfb821fb4bbeeb0210a7826f0dd3c845e70476e7cab6a4e","kind":"contract","path":"corpus/dev/contracts/split-indicator-oracle.json"},{"digest":"sha256:70fa47876aa4318fb758203c77d2f234cdf26fe404d382dcc49e8f7dbf0c378e","kind":"contract","path":"corpus/dev/contracts/unaddressed-parameter-sibling.json"},{"digest":"sha256:9a64e4c32aad92a08da1f64a7d5616b4b96b60bd7070cdeaef7afbd9723e573c","kind":"contract","path":"corpus/dev/contracts/unnamed-reference-set.json"},{"digest":"sha256:fbb9e557e77311f8d608e003e019fe9c59fcf3e11fd737bf97cd0a3ff1e2e6b6","kind":"contract","path":"corpus/dev/contracts/wrong-cardinality-form.json"}]}
|
package/dist/cli/render.d.ts
CHANGED
|
@@ -18,6 +18,6 @@ export declare function renderError(error: unknown): string;
|
|
|
18
18
|
* AD-21's seven exit codes, one line each. The `--help` output and the README
|
|
19
19
|
* table are this text, so the two cannot drift.
|
|
20
20
|
*/
|
|
21
|
-
export declare const EXIT_CODE_TABLE = "Exit codes (AD-21):\n 0 success, and every verdict other than FAIL or a promoted CONCERNS\n 1 CONCERNS promoted by --strict\n 2 FAIL\n 3 invalid: a failed pre-flight, or any other AD-21 invalidating condition\n 4 structural failure\n 5 runtime fault\n 64 usage error\n\n --strict never promotes a CONCERNS whose firing conditions are all evidence\n conditions: those
|
|
21
|
+
export declare const EXIT_CODE_TABLE = "Exit codes (AD-21):\n 0 success, and every verdict other than FAIL or a promoted CONCERNS\n 1 CONCERNS promoted by --strict\n 2 FAIL\n 3 invalid: a failed pre-flight, or any other AD-21 invalidating condition\n 4 structural failure\n 5 runtime fault\n 64 usage error\n\n --strict never promotes a CONCERNS whose firing conditions are all evidence\n conditions: those report that the measurement fell short of the policy.\n 1 and 2 come from the score command's verdict ladder. 3 comes from a failed\n pre-flight, which the preflight command reports, or from any other\n invalidating condition score finds.";
|
|
22
22
|
/** `eval-quality: usage: <message>` */
|
|
23
23
|
export declare function renderUsage(message: string): string;
|
package/dist/cli/render.js
CHANGED
|
@@ -95,10 +95,10 @@ export const EXIT_CODE_TABLE = `Exit codes (AD-21):
|
|
|
95
95
|
64 usage error
|
|
96
96
|
|
|
97
97
|
--strict never promotes a CONCERNS whose firing conditions are all evidence
|
|
98
|
-
conditions: those
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
98
|
+
conditions: those report that the measurement fell short of the policy.
|
|
99
|
+
1 and 2 come from the score command's verdict ladder. 3 comes from a failed
|
|
100
|
+
pre-flight, which the preflight command reports, or from any other
|
|
101
|
+
invalidating condition score finds.`;
|
|
102
102
|
/** `eval-quality: usage: <message>` */
|
|
103
103
|
export function renderUsage(message) {
|
|
104
104
|
return `${PREFIX}: usage: ${message}`;
|
package/dist/index.d.ts
CHANGED
|
@@ -12,4 +12,4 @@ export type { ScoringPolicy } from './core/schemas/scoring-policy.ts';
|
|
|
12
12
|
export type { SealedEvaluatorBrief } from './core/schemas/sealed-evaluator-brief.ts';
|
|
13
13
|
export type { SealedRunRecord } from './core/schemas/sealed-run-record.ts';
|
|
14
14
|
export type { FixtureReset, ManifestationWitness, SensitivityWitness, SensitivityWitnessLeg, WitnessChannel, WitnessInputs, } from './core/schemas/sensitivity-witness.ts';
|
|
15
|
-
export declare const VERSION = "1.
|
|
15
|
+
export declare const VERSION = "1.1.0";
|
package/dist/index.js
CHANGED
package/package.json
CHANGED