@openwop/openwop-conformance 2.46.18 → 2.46.20

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,3 +1,16 @@
1
+ ## [2.46.20] — 2026-10-24 — RFC 0081 eval envelope legs; RFC 0249 Draft
2
+
3
+ - **Two new legs in `v2-eval-trajectory-trials` (RFC 0081 §C)**, on the existing `conformance-eval-trajectory` run, so no extra run: `0081.eval-envelope` (one `eval.started` and one `eval.completed` bracket every `eval.scored`, `eval.completed` precedes `run.completed`, and its counts equal the summary's) and `0081.summary-served` (the served `EvalSummary` validates and its counts match its rows). Judges in `lib/eval-envelope-witness.ts`, whose self-tests fail each wrong host answer. Before this, no v2 leg read `eval.completed`.
4
+ - **Host impact:** hosts that advertise `agents.evalSuite.trajectory` and list the fixture (openwop-app, MyndHyve) run both legs. A host that omits `eval.completed` fails `0081.eval-envelope`.
5
+ - RFC 0249 (v2 proposal operations and an install read-back for RFC 0096) is filed as `Draft`. Its gap register lands in the packed `spec/v1/gaps.json`, which opened this cycle.
6
+
7
+ ## [2.46.19] — 2026-10-22 — RFC 0248: ctx.callAI attempt accounting and retry policy
8
+
9
+ - **New scenario `v2-callai-attempt-accounting`** with eight legs: `0248.attempts-count`, `output-tokens-sum`, `usage-incomplete`, `structured-output`, `attempts-replayed`, `max-attempts-cap`, `no-budget-growth` and `retry-malformed-refused`. They are gated on `aiProviders.callAccounting.attempts` / `.retryPolicy` and the fixture.
10
+ - **New fixture `conformance-callai-attempt-accounting`.** Its node `conformance.callai.echo-attempts` makes one `ctx.callAI` call through the host's conformance mock, which serves the run's `script`, and echoes `attempts`. The mock clamps reported `outputTokens` to each request's `maxTokens`. No seam is used: the legs read `provider.usage`, `envelope.retry-exhausted` and node outputs. The judges are in `lib/callai-attempts-witness.ts`, and their self-tests fail each wrong host answer.
11
+ - **`aiProviders.callAccounting {attempts, retryPolicy}`** is added to the facet, with the RFC 0248 rules in its descriptions.
12
+ - **Host impact: none today.** No host advertises `callAccounting` or lists the fixture.
13
+
1
14
  ## [2.46.18] — 2026-10-18 — RFC 0245: a success return need not carry outcome
2
15
 
3
16
  - **`0245.fixture-error-replayed` no longer requires `outcome` on the success return.** `outcome` is optional on `agent.tool-returned`, and RFC 0064 §F lets a host omit it. The leg now asserts that the second return carries no `error` (and no `status: "error"` under tool hooks). This is an erratum from openwop-app's implementation (ADR 0929). No host was judged under 2.46.17.
package/README.md CHANGED
@@ -137,7 +137,7 @@ Each scenario file records its disposition and assertion count in `<report-dir>/
137
137
 
138
138
  ## What's Covered
139
139
 
140
- The current suite has 658 scenario files under `src/scenarios/`. By target major ([`scenario-majors.json`](./scenario-majors.json)): 218 files run against a v2 host and 455 against a v1 host; 15 of them run at both.
140
+ The current suite has 659 scenario files under `src/scenarios/`. By target major ([`scenario-majors.json`](./scenario-majors.json)): 219 files run against a v2 host and 455 against a v1 host; 15 of them run at both.
141
141
 
142
142
  The v2 scenarios (`src/scenarios/v2-*.test.ts`) cover, by area:
143
143
 
@@ -166,7 +166,7 @@ Three kinds of scenario share the tree:
166
166
  - **Host** checks drive a live host over HTTP.
167
167
  - **Gated** checks run only when the host advertises the family, profile or fixture they need, and otherwise record `inapplicable` or `blocked` with a reason. `--require-behavior` turns an advertised-but-unwitnessable behaviour into a failure.
168
168
 
169
- Current source tree: 658 scenario files. [`coverage.md`](./coverage.md) maps each spec document and every OpenAPI operation to its scenarios. [`CHANGELOG.md`](./CHANGELOG.md) records when each scenario landed.
169
+ Current source tree: 659 scenario files. [`coverage.md`](./coverage.md) maps each spec document and every OpenAPI operation to its scenarios. [`CHANGELOG.md`](./CHANGELOG.md) records when each scenario landed.
170
170
 
171
171
  ## How the 2.x suite is built
172
172
 
@@ -1,5 +1,5 @@
1
1
  {
2
2
  "package": "@openwop/spec-artifacts",
3
- "version": "2.46.18",
4
- "stampSha256": "4e40e9463c8a0a9554e81e19d745f8dd90d59250be85da01cc3fc43eca1076f3"
3
+ "version": "2.46.20",
4
+ "stampSha256": "426586e18ceca126f915b8517b442ecc6178e3f2cecde3325bf00dcfd76fb37d"
5
5
  }
@@ -0,0 +1,24 @@
1
+ {
2
+ "id": "conformance-callai-attempt-accounting",
3
+ "name": "Conformance: ctx.callAI attempt accounting and retry policy (RFC 0248)",
4
+ "version": "1.0",
5
+ "description": "Fixture for `v2-callai-attempt-accounting.test.ts` (RFC 0248; the rules are the `aiProviders.callAccounting` facet descriptions). One node, `conformance.callai.echo-attempts` (conformance-RESERVED). It makes exactly one `ctx.callAI` call through the host's conformance mock provider, passing `inputs.request` (`prompt`, and optionally `responseSchema`, `maxTokens`, `retry`) unchanged. The mock serves `inputs.script` to that call's provider requests in order, one entry per request: `{ kind: \"invalid-json\" | \"truncated\" | \"valid\", outputTokens?, usage?, structuredMode? }`; `invalid-json` is the reply `{ this is not json }` (a parse error, never the natural-language fallback), `valid` is a minimal instance of `responseSchema` (or `ok` without one); past the end it repeats the last entry. The mock reports `outputTokens = min(entry.outputTokens, the maxTokens THAT request carried)` (no clamp when the request carried none), and serves an entry whose `outputTokens` exceeds that budget as truncated. `usage: false` means that request reports no usage. For a call with `responseSchema`, the mock uses `structuredMode` (default `prompt`) as the request's structured-output mode. The node completes with `outputs.attempts`, the `attempts` record of the `ctx.callAI` result verbatim, and does no counting of its own. A call that throws fails the node with the error's code and details, unchanged. A host that does not advertise `aiProviders.callAccounting` MUST NOT advertise this fixture.",
6
+ "nodes": [
7
+ {
8
+ "id": "callai-echo",
9
+ "typeId": "conformance.callai.echo-attempts",
10
+ "name": "Echo ctx.callAI attempts",
11
+ "position": { "x": 0, "y": 0 },
12
+ "config": {},
13
+ "inputs": {}
14
+ }
15
+ ],
16
+ "edges": [],
17
+ "triggers": [{ "id": "manual", "type": "manual", "enabled": true }],
18
+ "variables": [
19
+ { "name": "request", "type": "object" },
20
+ { "name": "script", "type": "array" }
21
+ ],
22
+ "metadata": { "tags": ["conformance", "aiProviders", "callAccounting"] },
23
+ "settings": { "timeout": 0 }
24
+ }
package/fixtures.md CHANGED
@@ -133,6 +133,7 @@ All fixtures MUST advertise:
133
133
  | v2 Storage Tenant-Isolation Probe | `conformance-storage-probe` | `storage.md` §Shared rules — one `core.conformance.storage-probe` node (conformance-RESERVED) calls the host's own storage `family` for the run's tenant: `write` stores `value` at `key`, `read` returns every value and key the family's isolated reads return for `key`, `delete` removes it. See §"The storage probe fixtures" | `completed`, or `failed` (`forbidden` / `not_found`) for a refused read | ≤ 30s |
134
134
  | v2 fs Sandbox Probe | `conformance-fs-probe` | `storage.md` §`fs` — one `core.conformance.fs-probe` node (conformance-RESERVED) calls the host's own `ctx.fs.read` / `write` with the run's `path`, unchanged; a rejection fails the node with its code and details. Operator symlink `<sandboxRoot>/conformance/escape-link` points outside the root. See §"The storage probe fixtures" | `completed` inside the root; `failed` (`forbidden`, `details.reason: path-outside-sandbox`) for an escape | ≤ 30s |
135
135
  | v2 tableStorage Schema Probe | `conformance-table-schema-probe` | `storage.md` §`tableStorage` — one `core.conformance.table-schema-probe` node (conformance-RESERVED) creates a table `{ k: string, n: number }` with the host's own `ctx.storage.table`, then inserts a well-typed row (`control`), a row whose `n` is a string (`insert-mistyped`), or updates `n` to a string (`update-mistyped`). See §"The tableStorage schema probe fixture" | `completed` for `control`; `failed` (`validation_error`, `details.service: tableStorage`) for a mistyped write | ≤ 30s |
136
+ | v2 ctx.callAI Attempt Accounting | `conformance-callai-attempt-accounting` | RFC 0248 — one `conformance.callai.echo-attempts` node (conformance-RESERVED) that makes one `ctx.callAI` call through the host's conformance mock, serves the run's `script`, and echoes the result's `attempts` (`v2-callai-attempt-accounting`) | each case run completes or fails as its script dictates | ≤ 30s |
136
137
  | v2 safeFetch SSRF Probe | `conformance-safefetch-probe` | `host-services.md` §`httpClient` — one `core.conformance.safefetch-probe` node (conformance-RESERVED) calls the host's own `ctx.http.safeFetch(url)` with the run's `url`, unchanged; completes with the response `status` only; a rejection fails the node with its code and details. See §"The safeFetch probe fixture" | `failed` (`egress_denied`, `details.reason: ssrf-blocked`) for a refused target | ≤ 60s |
137
138
  | A2A Task Roundtrip | `conformance-a2a-task-roundtrip` | Track 6 — host consumes the conformance suite's synthetic A2A peer; covers drift points #3 (`AUTH_REQUIRED`) and #4 (`REJECTED`). Node `core.conformance.a2a-invoke` is a **conformance-RESERVED** typeId (renamed 2026-08-16 from `core.a2a.invoke`, a multi-segment `core.*` id no host shipped and the corpus never defined): a host that consumes A2A peers MUST map it to its A2A bridge node (a host-extension node — `a2a-integration.md` §"State projection"); a host that does not consume A2A MUST NOT advertise this fixture | `failed` or `waiting-input` (per `driftScenario` input) | ≤ 30s |
138
139
  | WASM Pack Roundtrip | `conformance-wasm-pack-roundtrip` | RFC 0008 — invokes `vendor.openwop.rust-hello.greet` (loaded WASM pack); exercises required exports + at least one import | `completed` | ≤ 10s |
@@ -814,6 +815,26 @@ For `v2-tool-catalog-annotations`' unclassified-MCP leg, a host that advertises
814
815
 
815
816
  ---
816
817
 
818
+ ## The ctx.callAI attempt-accounting fixture (RFC 0248)
819
+
820
+ `conformance-callai-attempt-accounting` is what `v2-callai-attempt-accounting` drives. Its node, `conformance.callai.echo-attempts`, is conformance-RESERVED. A host that advertises `aiProviders.callAccounting` maps it to a node that makes exactly one `ctx.callAI` call through the host's conformance mock provider. It does no counting of its own. A host that does not advertise `callAccounting` MUST NOT advertise the fixture.
821
+
822
+ **Inputs.** `request` is passed to `ctx.callAI` unchanged: `prompt`, and optionally `responseSchema`, `maxTokens` and `retry`. `script` is the mock's replies.
823
+
824
+ **The mock.** It serves `script` to the call's provider requests in order, one entry per request, and repeats the last entry past the end. Each entry is `{ kind, outputTokens?, usage?, structuredMode? }`:
825
+
826
+ | Field | Meaning |
827
+ |---|---|
828
+ | `kind` | `invalid-json` (a clean stop with the reply `{ this is not json }`: a parse error, so it never engages the natural-language fallback), `truncated` (a `max_tokens` stop), or `valid` (a minimal instance of `responseSchema`, or `ok` without one) |
829
+ | `outputTokens` | The tokens the reply uses. The mock reports `min(outputTokens, the maxTokens that request carried)`, and serves an entry over that budget as truncated. With no `maxTokens` on the request, there is no clamp |
830
+ | `usage: false` | The request reports no usage |
831
+ | `structuredMode` | For a call with `responseSchema`: `native` or `prompt` (default) is the structured-output mode that request used |
832
+
833
+ **Output.** The node completes with `outputs.attempts`, the `attempts` record of the `ctx.callAI` result verbatim. A call that throws fails the node with the error's code and details unchanged: a malformed `retry` is `validation_error` with `details.field` naming the key.
834
+
835
+ **Why the clamp.** A real provider cannot exceed its budget. Clamping makes a host that grows `maxTokens` against `growMaxTokens: false` visible: the over-budget reply completes only under a grown budget, and its `provider.usage.outputTokens` exceeds the request's `maxTokens`.
836
+
837
+
817
838
  ## The tableStorage schema probe fixture (storage.md)
818
839
 
819
840
  `conformance-table-schema-probe` is what `v2-table-schema-enforcement` drives. Its node, `core.conformance.table-schema-probe`, is conformance-RESERVED: a host that advertises `tableStorage` maps it to a node that calls its OWN `ctx.storage.table` (`spec/v2/core/storage.md` §`tableStorage`) for the run's tenant, and nothing else. It is a fixture, not a seam: it runs through `POST /runs` under the caller's own credential.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@openwop/openwop-conformance",
3
- "version": "2.46.18",
3
+ "version": "2.46.20",
4
4
  "description": "Black-box conformance suite for OpenWOP hosts (v2 current; v1 scenarios retained for the frozen v1 tree).",
5
5
  "repository": {
6
6
  "type": "git",
@@ -56,6 +56,6 @@
56
56
  "@openwop/spec-artifacts": "file:../spec-artifacts"
57
57
  },
58
58
  "peerDependencies": {
59
- "@openwop/spec-artifacts": "2.46.18"
59
+ "@openwop/spec-artifacts": "2.46.20"
60
60
  }
61
61
  }
package/requirements.json CHANGED
@@ -2,11 +2,11 @@
2
2
  "$comment": "GENERATED by conformance/scripts/generate-requirement-registry.mjs — do not edit. One record per it()/test() in src/scenarios. Ids: openwop.it.<file-stem>.<title-slug>[~n] (src/lib/requirement-ids.ts). A record with id null has an interpolated title; its run-time row is keyed by the rendered title and maps here by file+line only. Renamed ids need a row in requirement-aliases.json.",
3
3
  "generatedFrom": "src/scenarios/*.test.ts",
4
4
  "counts": {
5
- "files": 746,
6
- "tests": 2814,
7
- "withStableId": 2814,
5
+ "files": 747,
6
+ "tests": 2824,
7
+ "withStableId": 2824,
8
8
  "interpolatedTitles": 0,
9
- "explicitIds": 2687
9
+ "explicitIds": 2697
10
10
  },
11
11
  "records": [
12
12
  {
@@ -29087,6 +29087,70 @@
29087
29087
  }
29088
29088
  ]
29089
29089
  },
29090
+ {
29091
+ "id": "openwop.it.v2-callai-attempt-accounting.attempts-count-equals-the-provider-requests-the-call-made",
29092
+ "file": "v2-callai-attempt-accounting.test.ts",
29093
+ "line": 91,
29094
+ "title": "attempts.count equals the provider requests the call made",
29095
+ "explicitId": "openwop.requirement.0248.attempts-count",
29096
+ "citations": []
29097
+ },
29098
+ {
29099
+ "id": "openwop.it.v2-callai-attempt-accounting.attempts-outputtokens-is-the-sum-over-every-attempt-and-usagecomplete-is-true-wh",
29100
+ "file": "v2-callai-attempt-accounting.test.ts",
29101
+ "line": 97,
29102
+ "title": "attempts.outputTokens is the sum over every attempt, and usageComplete is true when all reported",
29103
+ "explicitId": "openwop.requirement.0248.output-tokens-sum",
29104
+ "citations": []
29105
+ },
29106
+ {
29107
+ "id": "openwop.it.v2-callai-attempt-accounting.an-attempt-that-reported-no-usage-makes-usagecomplete-false",
29108
+ "file": "v2-callai-attempt-accounting.test.ts",
29109
+ "line": 103,
29110
+ "title": "an attempt that reported no usage makes usageComplete false",
29111
+ "explicitId": "openwop.requirement.0248.usage-incomplete",
29112
+ "citations": []
29113
+ },
29114
+ {
29115
+ "id": "openwop.it.v2-callai-attempt-accounting.structuredoutput-is-absent-without-responseschema-and-names-the-mode-the-final-a",
29116
+ "file": "v2-callai-attempt-accounting.test.ts",
29117
+ "line": 109,
29118
+ "title": "structuredOutput is absent without responseSchema and names the mode the final attempt used",
29119
+ "explicitId": "openwop.requirement.0248.structured-output",
29120
+ "citations": []
29121
+ },
29122
+ {
29123
+ "id": "openwop.it.v2-callai-attempt-accounting.a-replay-fork-reproduces-the-recorded-attempts",
29124
+ "file": "v2-callai-attempt-accounting.test.ts",
29125
+ "line": 117,
29126
+ "title": "a replay fork reproduces the recorded attempts",
29127
+ "explicitId": "openwop.requirement.0248.attempts-replayed",
29128
+ "citations": []
29129
+ },
29130
+ {
29131
+ "id": "openwop.it.v2-callai-attempt-accounting.maxattempts-caps-the-provider-requests",
29132
+ "file": "v2-callai-attempt-accounting.test.ts",
29133
+ "line": 131,
29134
+ "title": "maxAttempts caps the provider requests",
29135
+ "explicitId": "openwop.requirement.0248.max-attempts-cap",
29136
+ "citations": []
29137
+ },
29138
+ {
29139
+ "id": "openwop.it.v2-callai-attempt-accounting.growmaxtokens-false-keeps-every-attempt-within-the-request-budget",
29140
+ "file": "v2-callai-attempt-accounting.test.ts",
29141
+ "line": 137,
29142
+ "title": "growMaxTokens false keeps every attempt within the request budget",
29143
+ "explicitId": "openwop.requirement.0248.no-budget-growth",
29144
+ "citations": []
29145
+ },
29146
+ {
29147
+ "id": "openwop.it.v2-callai-attempt-accounting.a-malformed-retry-policy-is-refused-validation-error-naming-the-key",
29148
+ "file": "v2-callai-attempt-accounting.test.ts",
29149
+ "line": 143,
29150
+ "title": "a malformed retry policy is refused validation_error naming the key",
29151
+ "explicitId": "openwop.requirement.0248.retry-malformed-refused",
29152
+ "citations": []
29153
+ },
29090
29154
  {
29091
29155
  "id": "openwop.it.v2-callback-url-guarded.a-host-advertising-callback-delivery-refuses-a-callbackurl-its-egress-guard-woul",
29092
29156
  "file": "v2-callback-url-guarded.test.ts",
@@ -31028,7 +31092,7 @@
31028
31092
  {
31029
31093
  "id": "openwop.it.v2-eval-trajectory-trials.the-rfc-0244-suite-summary-and-event-fields-validate-and-the-negatives-are-refus",
31030
31094
  "file": "v2-eval-trajectory-trials.test.ts",
31031
- "line": 150,
31095
+ "line": 162,
31032
31096
  "title": "the RFC 0244 suite, summary and event fields validate and the negatives are refused",
31033
31097
  "explicitId": "openwop.requirement.0244.wire-shape",
31034
31098
  "citations": [
@@ -31067,7 +31131,7 @@
31067
31131
  {
31068
31132
  "id": "openwop.it.v2-eval-trajectory-trials.the-rfc-0244-additions-are-content-free-eval-summary-no-content-leak",
31069
31133
  "file": "v2-eval-trajectory-trials.test.ts",
31070
- "line": 173,
31134
+ "line": 185,
31071
31135
  "title": "the RFC 0244 additions are content-free (eval-summary-no-content-leak)",
31072
31136
  "explicitId": "openwop.requirement.0244.content-free",
31073
31137
  "citations": [
@@ -31084,7 +31148,7 @@
31084
31148
  {
31085
31149
  "id": "openwop.it.v2-eval-trajectory-trials.the-eval-suite-fixtures-validate-against-the-v2-suite-schema",
31086
31150
  "file": "v2-eval-trajectory-trials.test.ts",
31087
- "line": 180,
31151
+ "line": 192,
31088
31152
  "title": "the eval-suite fixtures validate against the v2 suite schema",
31089
31153
  "explicitId": "openwop.requirement.0244.fixtures-valid",
31090
31154
  "citations": [
@@ -31098,7 +31162,7 @@
31098
31162
  {
31099
31163
  "id": "openwop.it.v2-eval-trajectory-trials.the-rfc-0246-gating-fields-validate-and-stay-in-0-1",
31100
31164
  "file": "v2-eval-trajectory-trials.test.ts",
31101
- "line": 187,
31165
+ "line": 199,
31102
31166
  "title": "the RFC 0246 gating fields validate and stay in 0–1",
31103
31167
  "explicitId": "openwop.requirement.0246.gated-fields-shape",
31104
31168
  "citations": [
@@ -31120,7 +31184,7 @@
31120
31184
  {
31121
31185
  "id": "openwop.it.v2-eval-trajectory-trials.a-fixture-entry-may-be-marked-iserror-and-only-as-a-boolean-rfc-0245-a",
31122
31186
  "file": "v2-eval-trajectory-trials.test.ts",
31123
- "line": 195,
31187
+ "line": 207,
31124
31188
  "title": "a fixture entry may be marked isError, and only as a boolean (RFC 0245 §A)",
31125
31189
  "explicitId": "openwop.requirement.0245.fixture-error-shape",
31126
31190
  "citations": [
@@ -31142,7 +31206,7 @@
31142
31206
  {
31143
31207
  "id": "openwop.it.v2-eval-trajectory-trials.a-trajectory-suite-is-graded-per-the-reference-grader",
31144
31208
  "file": "v2-eval-trajectory-trials.test.ts",
31145
- "line": 205,
31209
+ "line": 217,
31146
31210
  "title": "a trajectory suite is graded per the reference grader",
31147
31211
  "explicitId": "openwop.requirement.0244.trajectory-graded",
31148
31212
  "citations": [
@@ -31173,10 +31237,38 @@
31173
31237
  }
31174
31238
  ]
31175
31239
  },
31240
+ {
31241
+ "id": "openwop.it.v2-eval-trajectory-trials.an-eval-run-brackets-its-scores-with-one-eval-started-and-one-eval-completed-rfc",
31242
+ "file": "v2-eval-trajectory-trials.test.ts",
31243
+ "line": 251,
31244
+ "title": "an eval run brackets its scores with one eval.started and one eval.completed (RFC 0081 §C)",
31245
+ "explicitId": "openwop.requirement.0081.eval-envelope",
31246
+ "citations": [
31247
+ {
31248
+ "section": "run-event-payloads evalStarted / evalCompleted, eval-summary.schema.json (RFC 0081 §C)",
31249
+ "requirement": null,
31250
+ "interpolated": true
31251
+ }
31252
+ ]
31253
+ },
31254
+ {
31255
+ "id": "openwop.it.v2-eval-trajectory-trials.the-served-evalsummary-validates-and-its-counts-match-its-rows-rfc-0081-c",
31256
+ "file": "v2-eval-trajectory-trials.test.ts",
31257
+ "line": 259,
31258
+ "title": "the served EvalSummary validates and its counts match its rows (RFC 0081 §C)",
31259
+ "explicitId": "openwop.requirement.0081.summary-served",
31260
+ "citations": [
31261
+ {
31262
+ "section": "run-event-payloads evalStarted / evalCompleted, eval-summary.schema.json (RFC 0081 §C)",
31263
+ "requirement": null,
31264
+ "interpolated": true
31265
+ }
31266
+ ]
31267
+ },
31176
31268
  {
31177
31269
  "id": "openwop.it.v2-eval-trajectory-trials.repeated-trials-emit-one-eval-scored-per-task-per-trial",
31178
31270
  "file": "v2-eval-trajectory-trials.test.ts",
31179
- "line": 241,
31271
+ "line": 280,
31180
31272
  "title": "repeated trials emit one eval.scored per task per trial",
31181
31273
  "explicitId": "openwop.requirement.0244.trials-per-task",
31182
31274
  "citations": [
@@ -31194,7 +31286,7 @@
31194
31286
  {
31195
31287
  "id": "openwop.it.v2-eval-trajectory-trials.the-per-task-trials-block-and-trialstats-follow-the-b-formulas",
31196
31288
  "file": "v2-eval-trajectory-trials.test.ts",
31197
- "line": 252,
31289
+ "line": 291,
31198
31290
  "title": "the per-task trials block and trialStats follow the §B formulas",
31199
31291
  "explicitId": "openwop.requirement.0244.trial-stats-formula",
31200
31292
  "citations": [
@@ -31233,7 +31325,7 @@
31233
31325
  {
31234
31326
  "id": "openwop.it.v2-eval-trajectory-trials.a-fixture-miss-under-onmiss-fail-fails-the-trial-and-is-counted",
31235
31327
  "file": "v2-eval-trajectory-trials.test.ts",
31236
- "line": 275,
31328
+ "line": 314,
31237
31329
  "title": "a fixture miss under onMiss \"fail\" fails the trial and is counted",
31238
31330
  "explicitId": "openwop.requirement.0244.fixture-miss-fails",
31239
31331
  "citations": [
@@ -31270,7 +31362,7 @@
31270
31362
  {
31271
31363
  "id": "openwop.it.v2-eval-trajectory-trials.a-suite-using-an-unadvertised-rfc-0244-feature-is-refused-at-createrun",
31272
31364
  "file": "v2-eval-trajectory-trials.test.ts",
31273
- "line": 295,
31365
+ "line": 334,
31274
31366
  "title": "a suite using an unadvertised RFC 0244 feature is refused at createRun",
31275
31367
  "explicitId": "openwop.requirement.0244.unadvertised-feature-refused",
31276
31368
  "citations": [
@@ -31288,7 +31380,7 @@
31288
31380
  {
31289
31381
  "id": "openwop.it.v2-eval-trajectory-trials.under-passrule-mean-a-task-whose-trials-failed-their-trajectory-does-not-pass-rf",
31290
31382
  "file": "v2-eval-trajectory-trials.test.ts",
31291
- "line": 311,
31383
+ "line": 350,
31292
31384
  "title": "under passRule \"mean\" a task whose trials failed their trajectory does not pass (RFC 0246)",
31293
31385
  "explicitId": "openwop.requirement.0246.mean-gated",
31294
31386
  "citations": [
@@ -31302,7 +31394,7 @@
31302
31394
  {
31303
31395
  "id": "openwop.it.v2-eval-trajectory-trials.a-suite-whose-tasks-failed-their-trajectory-does-not-pass-on-the-output-scores-r",
31304
31396
  "file": "v2-eval-trajectory-trials.test.ts",
31305
- "line": 324,
31397
+ "line": 363,
31306
31398
  "title": "a suite whose tasks failed their trajectory does not pass on the output scores (RFC 0246)",
31307
31399
  "explicitId": "openwop.requirement.0246.suite-gated",
31308
31400
  "citations": [
@@ -31316,7 +31408,7 @@
31316
31408
  {
31317
31409
  "id": "openwop.it.v2-eval-trajectory-trials.a-fixture-entry-marked-iserror-is-replayed-as-a-tool-error-not-a-success-rfc-024",
31318
31410
  "file": "v2-eval-trajectory-trials.test.ts",
31319
- "line": 335,
31411
+ "line": 374,
31320
31412
  "title": "a fixture entry marked isError is replayed as a tool error, not a success (RFC 0245)",
31321
31413
  "explicitId": "openwop.requirement.0245.fixture-error-replayed",
31322
31414
  "citations": [
@@ -31330,7 +31422,7 @@
31330
31422
  {
31331
31423
  "id": "openwop.it.v2-eval-trajectory-trials.a-suite-using-iserror-is-refused-at-createrun-without-fixtureerrors-rfc-0245-c",
31332
31424
  "file": "v2-eval-trajectory-trials.test.ts",
31333
- "line": 349,
31425
+ "line": 388,
31334
31426
  "title": "a suite using isError is refused at createRun without fixtureErrors (RFC 0245 §C)",
31335
31427
  "explicitId": "openwop.requirement.0245.unadvertised-refused",
31336
31428
  "citations": [
@@ -1,9 +1,9 @@
1
1
  {
2
2
  "$comment": "GENERATED by conformance/scripts/generate-scenario-majors.mjs (RFC 0168 §D.3). Do not edit; add a file to BOTH_MAJORS in the generator to target both majors.",
3
3
  "counts": {
4
- "files": 658,
4
+ "files": 659,
5
5
  "v1": 455,
6
- "v2": 218
6
+ "v2": 219
7
7
  },
8
8
  "majors": {
9
9
  "a2a-1-0-agent-card.test.ts": [
@@ -1356,6 +1356,9 @@
1356
1356
  "v2-bundle-v3-signed.test.ts": [
1357
1357
  2
1358
1358
  ],
1359
+ "v2-callai-attempt-accounting.test.ts": [
1360
+ 2
1361
+ ],
1359
1362
  "v2-callback-url-guarded.test.ts": [
1360
1363
  2
1361
1364
  ],
@@ -1,17 +1,17 @@
1
1
  {
2
2
  "_comment": "Provenance of @openwop/spec-artifacts (RFC 0168 §D.2). files: SHA-256 per file; the conformance suite compares the installed peer against dist/spec-artifacts.lock.json at start.",
3
3
  "package": "@openwop/spec-artifacts",
4
- "version": "2.46.18",
5
- "corpusTag": "v2.46.18",
4
+ "version": "2.46.20",
5
+ "corpusTag": "v2.46.20",
6
6
  "files": {
7
7
  "api/.redocly.lint-ignore.yaml": "bf5a8350b88a72fa43f59605ed8d903ed24b6cfccda5e45509c9f6ed9ee4e712",
8
8
  "api/asyncapi.yaml": "d5ecb9ee6114582be3b1f662c84bfac9ae96dae7bacb853e461168f70a8e1c7d",
9
9
  "api/grpc/openwop.proto": "c3e72bb17cba514ee98feb6434e6c9b6ea6795bfd086489ec69fd882dd1ad977",
10
10
  "api/openapi.yaml": "ac6de5303872664b78ff419c035915593e01599e02835e777a0123de1baeb398",
11
11
  "api/redocly.yaml": "b0604c89b2ca6d5076ec25725c539dad44a741a811fe524439ee6daef8baa09f",
12
- "api/seams-v2.yaml": "7d694b3cb21105110cff218f33bdf9c931cee03c6ae54be6e3781dbc2f5789a7",
13
- "api/v2/asyncapi.yaml": "5eae29f67aee9447008184f12c04673d84b7a2bed8616c5055b675d33c8a6d29",
14
- "api/v2/openapi.yaml": "49bcf512e19b7441d33819c4cb2fce26de0251296d78da3599d509217041e1c1",
12
+ "api/seams-v2.yaml": "14325981296aaf0ab2dcce8aa96527bd68935c1130bd9bf6103ee6b6c4df4f8e",
13
+ "api/v2/asyncapi.yaml": "b5f5369bbb35a2c3c75ff6c4f6c9281e77b141dd1d21e9495fef670d72b74a5b",
14
+ "api/v2/openapi.yaml": "3b17112c01e94987b0cef6a328cc9a6d3a0c0521bbafb14b1fc8abb4e10d9091",
15
15
  "api/v2/redocly.yaml": "1e66b60e6118ad11a823bb620678be464d99dfe50a40e3e6f93ec9429b88b34c",
16
16
  "schemas/README.md": "276fc1d51eb08e14e9127b9fd27761d7046ee33c12000f3a64a801322db8bf58",
17
17
  "schemas/a2a-task-state.schema.json": "75d5049dea8bd873ff0e7546f1c60c8d36c219bec7084264360be54a8be30ae1",
@@ -117,7 +117,7 @@
117
117
  "schemas/v2/artifact.schema.json": "85b9dc284d39b99d2a66fa8cbf47f2dd3d8526d3bd6cc411ad315b372de38975",
118
118
  "schemas/v2/audit-verify-result.schema.json": "14af8e9283510f0a40e48a2059c929ee44e0fc9fa3cc66a95bfbe90b17c48297",
119
119
  "schemas/v2/budget-policy.schema.json": "1de68095db0beb5c7a4be629de26e79f62ebbe748a88d46f36e768edfd1f0d68",
120
- "schemas/v2/capabilities.schema.json": "87faea080e8043abd16cba19dece777a810731fc3e4455c96c02d0bfd0997d84",
120
+ "schemas/v2/capabilities.schema.json": "fe28a95bc36767b99cdf529fe7db47dce49c49c5635a724e552c728871c48b06",
121
121
  "schemas/v2/certification-bundle.schema.json": "5cfa7fb5247160333e33a58b60d42bcd4e5269d8391d6a112d301e14828892ed",
122
122
  "schemas/v2/channel-presence-payload.schema.json": "fc03df0c20cc12a25f7ee6a47503a4c5931d79fe3d5ff24815e8fd7c1bcb9a87",
123
123
  "schemas/v2/channel-written-payload.schema.json": "ccecff3c71a3275ad8db035ac0e04db25e6f8d23cf091ddbcd5a60d1be1cbb6b",
@@ -208,14 +208,14 @@
208
208
  "schemas/workspace-file.schema.json": "464de85c2a068243084ee9c1d969bc7cd5d8f7948574e58450d6493c38a0e1e4",
209
209
  "spec/v1/alias-detectors.json": "22444c22bb88c568339ef512553f9fc38146e22b566d11ab8608f655d688d19c",
210
210
  "spec/v1/capability-declaration-classes.json": "e7729aed5c4b4e1dd02abab0530f14cc95f5d4070fe51fb139e7f5cccefa00c6",
211
- "spec/v1/core-standard-manifest.json": "df4bf8218c18e6a19c7fca67cd6fee1bee705e979c687b8071f8d7e23fb5c10f",
211
+ "spec/v1/core-standard-manifest.json": "d70cc792bd36812bfcec84c6a8b0d2cd096a3fbed19967afbc8f1dd1a7d26d87",
212
212
  "spec/v1/deprecations.json": "e723bc11d7272c71789ad1303bb7f21809a8a024abf57594f9d5c35dc199ece9",
213
213
  "spec/v1/deprecations.schema.json": "3e393c405d2a41b467d8c5e3c468549078df1ce6a6d2588fc488097b95d9b55b",
214
214
  "spec/v1/end-of-support.json": "35f0bf3b70fe78b16e2a2fd81a153fc27e703f6c16376c6fe7cdf1f4f2c4e6d4",
215
215
  "spec/v1/event-codemap.json": "3da60d884157793a360da532a9fcbbfb5285636db325a74cec94b34622186d97",
216
216
  "spec/v1/event-codemap.schema.json": "d05933b2e88103aff51a2f774df97b9bda065e0fc88dbd9b114cc47a32189174",
217
217
  "spec/v1/extensions.json": "79a60754aa16cbdbdb604f8e5c00038af535a1690a4d07bd5ff7c38f0256a7bc",
218
- "spec/v1/gaps.json": "572b68c8a4926339fb94017fcd8f9c78adfaa4900dc809fb8e76b094a7260ba0",
218
+ "spec/v1/gaps.json": "4655f3087d9917f3b6b7612af23c86a1b338c732a9c1a72ee4a705a1b4f4f7a9",
219
219
  "spec/v1/gaps.schema.json": "8fd83259f556553c9df0f53e7a82ca8c2a4771d471197f69b4e2ae0ceeacfd66",
220
220
  "spec/v1/migrations.json": "4c50760be70cb644d02427bcaf5749606700edf4513f423903cf3b1d12f5c300",
221
221
  "spec/v1/migrations.schema.json": "886779aa6c22e646db097f5df210adb018a4dd14a7b815465a18c8a7056c8f72",
@@ -232,7 +232,7 @@
232
232
  "spec/v2/core/execution.md": "4218dbb397534940f83ca5d46561f02bf8f3fdf68d2ae7fefe7f2eb2ed771103",
233
233
  "spec/v2/core/form-content-packs.md": "02931227899676fd4581a27017be2a6c4953904717be9b81c29d12f87ab3917d",
234
234
  "spec/v2/core/headers.md": "f69fe81e4a3ad21879bb85948660c16860eb05bf152928c31aa4898f6d7d1887",
235
- "spec/v2/core/host-services.md": "168a40935a03867ee82461b5d83d05578be25ee3e5af4eb080587b891231e242",
235
+ "spec/v2/core/host-services.md": "a6b5af882cb1e5c992dddc53c4de06feb36eee01bd566614330041213b3ce657",
236
236
  "spec/v2/core/i18n.md": "882904b7d59f992ab1e5f7346167161a54cf665b8fee47e64a3862cf9133a580",
237
237
  "spec/v2/core/idempotency.md": "adbd317a6ebb57c2759c076297b31f67def9e391576bccd0bbea787b13e85872",
238
238
  "spec/v2/core/identity.md": "d0e3d388484507dd6260601ebf8dd1447b0b0f26ce2fe76c8a5d49e61387cffd",
@@ -254,7 +254,7 @@
254
254
  "spec/v2/core/workflow-chain-packs.md": "fdce7a8477e269950a2bac05ee6eddbd66eb0b670b464d00054f7196ec05d29c",
255
255
  "spec/v2/corrections.json": "3a8d93e33865b692d26a3b6bfd3819766e206b2e0c2a69669baa4bf833d05a49",
256
256
  "spec/v2/corrections.schema.json": "46bbffb954d13bd9995aebc912ceeb2544f7064331abf25fd6ab85c81a5cec94",
257
- "spec/v2/declaration.json": "de8558d76a03d07101c5e26a7311d3e90b449f065c9ba418f4abd9c91864a58b",
257
+ "spec/v2/declaration.json": "5ccdcf9865ca02022c01a551583e1f343a4b7c73fd228049f1be620858a60ad5",
258
258
  "spec/v2/declaration.schema.json": "ef951633e4070899f64828f5cf1b5f4870270a0360f81dbb49b5b4d6cd407550",
259
259
  "spec/v2/eos-override.json": "4c72d2f6a658943b01c1de1643cf294642091933af7832f38ee5495df684cb2d",
260
260
  "spec/v2/errors.json": "1f0e29455edbd22e28f67f221766209c3c55ac4987beb77993b5702dd7cae131",
@@ -282,7 +282,7 @@
282
282
  "spec/v2/ext/webResearch/README.md": "b675e6f6646128d781b661f754c2393eed86e5935d16c3a22df582a5ce025b9b",
283
283
  "spec/v2/facets/a2a.schema.json": "b68bf5e29e455e46c2ad7cfcccf5963730ec2310ff9db81884550d8df30fd6ef",
284
284
  "spec/v2/facets/agents.schema.json": "88fe144572c25e1b09b99fd83e2ad12b2611d4d7366b5d53c3a50d0310faad93",
285
- "spec/v2/facets/aiProviders.schema.json": "4eb911c2403b8f618325194855a1326680fbda2549dfbfe86a70dd2743f8fb12",
285
+ "spec/v2/facets/aiProviders.schema.json": "6e4327bcd48110686cc1adfd26ddd6fbbf4ff529c476dcc74eeff8c5ae9965ce",
286
286
  "spec/v2/facets/auditLogIntegrity.schema.json": "9a7f81320c8f940c6f4129bc7dd895a4063231c5458a717a08c2cbf9bb886c46",
287
287
  "spec/v2/facets/auth.schema.json": "7d98bc81a0f7d307460d039eebfebd21388091bfe1d538762583940f94c0bd2a",
288
288
  "spec/v2/facets/connections.schema.json": "f998bb206a5814183bb4720cc34fa43b7ab64d61e39a79437b009c589ce8b78e",
@@ -314,9 +314,9 @@
314
314
  "spec/v2/path-manifest.json": "7b5679a4dad98c80d855361d88a98c8a4e045af15e310862a05b285b815dc9fc",
315
315
  "spec/v2/peer-dependency-aliases.json": "d10299280abee08258502925bc327293ee413e0108cd6e6ec75ff6110653308d",
316
316
  "spec/v2/profiles.json": "b382b261c7844798575f7c1b37c934f0da19846016a2337d883d4f5ef6e05479",
317
- "spec/v2/release.json": "a8ae6035c74fcc0f2b13f1c02bee4862287840cc495840febc867cd66041c8ed",
317
+ "spec/v2/release.json": "a8b6d20a4ed90b21b4f9feb57a9c2ec5fdb13c74a74f857e5e3aec5ee8e75e40",
318
318
  "spec/v2/retention-floors.json": "eaf3722d95c79947af1d4269ef85117e126518c588cfcf1a2b21b97269f51624",
319
- "spec/v2/surface-baseline.json": "619c530713063e556abeb0f232cd905942ef8465b73936b9bf4f89ca01fe4485"
319
+ "spec/v2/surface-baseline.json": "c5aee6bbf8011e11351a4f9e0aafaf444d60af55116c543f04b55b88878cdf83"
320
320
  },
321
- "corpusCommit": "e4a97e99e7fb158f4091df43da0422866140af2a"
321
+ "corpusCommit": "3ed6fc93890b4d00c44aafc0fe3ce75a24b8dcc7"
322
322
  }
@@ -0,0 +1,256 @@
1
+ /**
2
+ * RFC 0248 — `ctx.callAI` reports its provider attempts and takes a per-call
3
+ * retry policy. The rules are the `aiProviders.callAccounting` facet
4
+ * descriptions (`spec/v2/facets/aiProviders.schema.json`).
5
+ *
6
+ * No seam. The suite runs the `conformance-callai-attempt-accounting` fixture
7
+ * once per case. Its node makes one `ctx.callAI` call, the host's conformance
8
+ * mock serves the case's `script` to that call's provider requests, and the
9
+ * node echoes the result's `attempts`. The suite wrote the script, so it knows
10
+ * what every request returned. Two normative surfaces corroborate it where the
11
+ * host advertises them:
12
+ * - `provider.usage`: exactly one per provider invocation, with
13
+ * `outputTokens` (`events.md` §`providerUsage`);
14
+ * - `envelope.retry-exhausted.totalAttempts` (`events.md` §envelopes).
15
+ *
16
+ * The mock clamps each reported `outputTokens` to the `maxTokens` that request
17
+ * carried and serves an over-budget entry as truncated (fixtures.md), so a
18
+ * host that grows the budget against `growMaxTokens: false` shows on the log.
19
+ *
20
+ * Two halves, as in `table-schema-witness.ts`: {@link drive} observes and
21
+ * asserts nothing; the `judge*` functions are pure.
22
+ *
23
+ * @see RFCS/0248-callai-attempt-accounting-and-retry-policy.md
24
+ */
25
+
26
+ import { driver, type OpenWOPResponse } from './driver.js';
27
+ import { projectBoundId } from './bound-id.js';
28
+ import { readErrorCode } from './error-envelope.js';
29
+ import { isFixtureAdvertised } from './fixtures.js';
30
+
31
+ export const FIXTURE = 'conformance-callai-attempt-accounting';
32
+ export const NODE_ID = 'callai-echo';
33
+ const DOC_A = 'aiProviders.schema.json §callAccounting.attempts (RFC 0248 §A)';
34
+ const DOC_B = 'aiProviders.schema.json §callAccounting.retryPolicy (RFC 0248 §B)';
35
+
36
+ export interface ScriptEntry { readonly kind: 'invalid-json' | 'truncated' | 'valid'; readonly outputTokens?: number; readonly usage?: false; readonly structuredMode?: 'native' | 'prompt' }
37
+ export interface Case { readonly request: Record<string, unknown>; readonly script: readonly ScriptEntry[] }
38
+
39
+ const SCHEMA = { type: 'object', properties: { answer: { type: 'string' } }, required: ['answer'] };
40
+ const BUDGET = 100;
41
+ export const CASES = {
42
+ /** §A.1/A.2: a schema violation, then a valid reply. */
43
+ count: { request: { prompt: 'conformance', responseSchema: SCHEMA, maxTokens: BUDGET }, script: [{ kind: 'invalid-json', outputTokens: 10 }, { kind: 'valid', outputTokens: 20 }] },
44
+ /** §A.2: one reply that reports no usage. */
45
+ incomplete: { request: { prompt: 'conformance', maxTokens: BUDGET }, script: [{ kind: 'valid', usage: false }] },
46
+ /** §A.3: no responseSchema, so no structuredOutput. */
47
+ plain: { request: { prompt: 'conformance', maxTokens: BUDGET }, script: [{ kind: 'valid', outputTokens: 5 }] },
48
+ /** §A.3: the mock declares native structured output. */
49
+ native: { request: { prompt: 'conformance', responseSchema: SCHEMA, maxTokens: BUDGET }, script: [{ kind: 'valid', outputTokens: 5, structuredMode: 'native' }] },
50
+ /** §B.1: one attempt allowed, and it is invalid. */
51
+ cap: { request: { prompt: 'conformance', responseSchema: SCHEMA, maxTokens: BUDGET, retry: { maxAttempts: 1 } }, script: [{ kind: 'invalid-json', outputTokens: 10 }, { kind: 'valid', outputTokens: 20 }] },
52
+ /** §B.2: truncated, then a reply that fits only a grown budget. */
53
+ growth: { request: { prompt: 'conformance', responseSchema: SCHEMA, maxTokens: BUDGET, retry: { growMaxTokens: false } }, script: [{ kind: 'truncated', outputTokens: BUDGET }, { kind: 'valid', outputTokens: 150 }] },
54
+ /** §B.5: maxAttempts out of range. */
55
+ malformed: { request: { prompt: 'conformance', maxTokens: BUDGET, retry: { maxAttempts: 0 } }, script: [{ kind: 'valid', outputTokens: 5 }] },
56
+ } as const satisfies Record<string, Case>;
57
+ export type CaseName = keyof typeof CASES;
58
+
59
+ export interface CaseRun {
60
+ readonly runId: string;
61
+ readonly status: string;
62
+ readonly attempts: Record<string, unknown> | undefined;
63
+ readonly errorCode: string | null;
64
+ readonly errorField: unknown;
65
+ /** `provider.usage` payloads of the run (one node), in log order. */
66
+ readonly usage: readonly Record<string, unknown>[];
67
+ readonly exhaustedTotal: number | undefined;
68
+ readonly startedSeq: number | undefined;
69
+ }
70
+
71
+ export interface Finding { readonly ok: boolean; readonly doc: string; readonly message: string }
72
+ type Rec = Record<string, unknown>;
73
+ const isRec = (v: unknown): v is Rec => typeof v === 'object' && v !== null && !Array.isArray(v);
74
+ const f = (ok: boolean, doc: string, message: string): Finding => ({ ok, doc, message });
75
+ const TERMINAL = new Set(['completed', 'failed', 'cancelled']);
76
+
77
+ async function http(fn: () => Promise<OpenWOPResponse>): Promise<OpenWOPResponse | null> {
78
+ try { return await fn(); } catch { return null; }
79
+ }
80
+
81
+ async function terminal(runId: string, timeoutMs: number): Promise<string | null> {
82
+ const deadline = Date.now() + timeoutMs;
83
+ for (;;) {
84
+ const snap = await http(() => driver.get(`/runs/${projectBoundId(runId)}`));
85
+ const status = String((snap?.json as { status?: unknown } | undefined)?.status ?? '');
86
+ if (TERMINAL.has(status)) return status;
87
+ if (Date.now() > deadline) {
88
+ await http(() => driver.post(`/runs/${projectBoundId(runId)}/cancel`, {}));
89
+ return null;
90
+ }
91
+ await new Promise((r) => setTimeout(r, 200));
92
+ }
93
+ }
94
+
95
+ /** Read one finished run's log into a CaseRun. Asserts nothing. */
96
+ export async function observe(runId: string, status: string): Promise<CaseRun | string> {
97
+ const ev = await http(() => driver.get(`/runs/${projectBoundId(runId)}/events/poll?timeout=1`));
98
+ const events = (ev?.json as { events?: unknown } | undefined)?.events;
99
+ if (ev?.status !== 200 || !Array.isArray(events)) return `GET /runs/{runId}/events/poll answered ${ev?.status ?? 'nothing'}`;
100
+ // The fixture has one node, so every event of the run is the echo node's.
101
+ // `provider.usage.nodeId` is optional; filtering on it would blame a host
102
+ // that omits it.
103
+ const mine = (events as unknown[]).filter(isRec);
104
+ const of = (type: string): Rec[] => mine.filter((e) => e['type'] === type).map((e) => (isRec(e['payload']) ? e['payload'] : {}));
105
+ const completed = of('node.completed')[0];
106
+ const outputs = isRec(completed?.['outputs']) ? completed['outputs'] : undefined;
107
+ const failed = of('node.failed')[0];
108
+ const error = isRec(failed?.['error']) ? failed['error'] : undefined;
109
+ const details = isRec(error?.['details']) ? error['details'] : undefined;
110
+ const exhausted = of('envelope.retry-exhausted')[0];
111
+ const started = mine.find((e) => e['type'] === 'node.started');
112
+ return {
113
+ runId, status,
114
+ attempts: isRec(outputs?.['attempts']) ? outputs['attempts'] : undefined,
115
+ errorCode: typeof error?.['code'] === 'string' ? error['code'] : null,
116
+ errorField: details?.['field'],
117
+ usage: of('provider.usage'),
118
+ exhaustedTotal: typeof exhausted?.['totalAttempts'] === 'number' ? exhausted['totalAttempts'] : undefined,
119
+ startedSeq: typeof started?.['sequence'] === 'number' ? started['sequence'] : undefined,
120
+ };
121
+ }
122
+
123
+ export type Drive = { readonly kind: 'skip'; readonly disposition: 'inapplicable' | 'blocked'; readonly reason: string } | { readonly kind: 'observed'; readonly run: CaseRun };
124
+
125
+ /** One fixture run of one case. Asserts nothing. */
126
+ export async function runCase(name: CaseName, timeoutMs = 60_000): Promise<Drive> {
127
+ if (!isFixtureAdvertised(FIXTURE)) return { kind: 'skip', disposition: 'inapplicable', reason: `the host does not advertise fixture ${FIXTURE}` };
128
+ const c = CASES[name];
129
+ const created = await http(() => driver.post('/runs', { workflowId: FIXTURE, inputs: { request: c.request, script: c.script } }));
130
+ if (created === null) return { kind: 'skip', disposition: 'blocked', reason: 'POST /runs unreachable (fetch failed)' };
131
+ if (created.status === 429) return { kind: 'skip', disposition: 'blocked', reason: 'POST /runs answered 429 — the run budget, not the wire' };
132
+ const runId = (created.json as { runId?: unknown } | undefined)?.runId;
133
+ if (created.status !== 201 || typeof runId !== 'string') return { kind: 'skip', disposition: 'blocked', reason: `POST /runs {workflowId: ${FIXTURE}} answered ${created.status} ${readErrorCode(created.json) ?? ''}`.trim() };
134
+ const status = await terminal(runId, timeoutMs);
135
+ if (status === null) return { kind: 'skip', disposition: 'blocked', reason: `the ${name} run did not reach a terminal status within ${timeoutMs} ms` };
136
+ const run = await observe(runId, status);
137
+ return typeof run === 'string' ? { kind: 'skip', disposition: 'blocked', reason: run } : { kind: 'observed', run };
138
+ }
139
+
140
+ /** Fork a finished run in replay mode from the node's start and read the fork. Asserts nothing. */
141
+ export async function replayOf(source: CaseRun, timeoutMs = 60_000): Promise<{ declined: number } | { run: CaseRun } | string> {
142
+ if (source.startedSeq === undefined) return 'the source run has no node.started for the echo node';
143
+ const res = await http(() => driver.post(`/runs/${projectBoundId(source.runId)}:fork`, { mode: 'replay', fromSeq: source.startedSeq }));
144
+ if (res === null) return 'POST /runs/{runId}:fork unreachable (fetch failed)';
145
+ if (res.status === 404 || res.status === 403 || res.status === 501) return { declined: res.status };
146
+ const forkId = (res.json as { runId?: unknown } | undefined)?.runId;
147
+ if (res.status !== 201 || typeof forkId !== 'string') return `POST /runs/{runId}:fork answered ${res.status} ${readErrorCode(res.json) ?? ''}`.trim();
148
+ const status = await terminal(forkId, timeoutMs);
149
+ if (status === null) return `the replay fork did not reach a terminal status within ${timeoutMs} ms`;
150
+ const run = await observe(forkId, status);
151
+ return typeof run === 'string' ? run : { run };
152
+ }
153
+
154
+ const tokensOf = (script: readonly ScriptEntry[], i: number, budget: number): number => {
155
+ const e = script[Math.min(i, script.length - 1)]!;
156
+ return Math.min(e.outputTokens ?? 0, budget);
157
+ };
158
+ const intAtLeast = (v: unknown, min: number): v is number => typeof v === 'number' && Number.isInteger(v) && v >= min;
159
+
160
+ const presence = (run: CaseRun): Finding[] => {
161
+ const a = run.attempts;
162
+ const out = [f(run.status === 'completed' && a !== undefined, DOC_A, `a completed ctx.callAI call MUST carry attempts on its result (run ${run.status}, attempts ${a === undefined ? 'absent' : 'present'})`)];
163
+ if (a !== undefined) out.push(f(intAtLeast(a['count'], 1), DOC_A, `attempts.count MUST be an integer ≥ 1 (got ${JSON.stringify(a['count'])})`));
164
+ return out;
165
+ };
166
+
167
+ /**
168
+ * §A.1: the record's count equals the provider requests — one `provider.usage`
169
+ * each where the host advertises it (`withUsageEvents`).
170
+ */
171
+ export function judgeCount(run: CaseRun, withUsageEvents: boolean): Finding[] {
172
+ const out = presence(run);
173
+ const count = run.attempts?.['count'];
174
+ if (!intAtLeast(count, 1)) return out;
175
+ if (withUsageEvents) out.push(f(run.usage.length === count, DOC_A, `attempts.count MUST equal the provider requests, one provider.usage each (events.md §providerUsage): ${run.usage.length} provider.usage vs count ${count}`));
176
+ return out;
177
+ }
178
+
179
+ /**
180
+ * §A.2: `outputTokens` is the sum over the requests the script served, equals
181
+ * the summed `provider.usage` where advertised, and `usageComplete` is true
182
+ * because every scripted reply reported usage.
183
+ */
184
+ export function judgeSum(run: CaseRun, c: Case, withUsageEvents: boolean): Finding[] {
185
+ const out = presence(run);
186
+ const a = run.attempts;
187
+ const count = a?.['count'];
188
+ if (a === undefined || !intAtLeast(count, 1)) return out;
189
+ const budget = typeof c.request['maxTokens'] === 'number' ? (c.request['maxTokens'] as number) : Number.MAX_SAFE_INTEGER;
190
+ const want = Array.from({ length: count }, (_, i) => tokensOf(c.script, i, budget)).reduce((x, y) => x + y, 0);
191
+ out.push(f(a['usageComplete'] === true, DOC_A, `every scripted request reported usage, so usageComplete MUST be true (got ${JSON.stringify(a['usageComplete'])})`));
192
+ out.push(f(a['outputTokens'] === want, DOC_A, `attempts.outputTokens MUST be the sum over the ${count} requests the script served (want ${want}, got ${JSON.stringify(a['outputTokens'])})`));
193
+ if (withUsageEvents) {
194
+ const sum = run.usage.reduce((x, u) => x + (typeof u['outputTokens'] === 'number' ? u['outputTokens'] : 0), 0);
195
+ out.push(f(sum === a['outputTokens'], DOC_A, `attempts.outputTokens MUST equal the summed provider.usage.outputTokens (${sum} vs ${JSON.stringify(a['outputTokens'])})`));
196
+ }
197
+ return out;
198
+ }
199
+
200
+ /** §A.2: a request that reported no usage makes usageComplete false. */
201
+ export function judgeIncomplete(run: CaseRun): Finding[] {
202
+ const a = run.attempts;
203
+ if (run.status !== 'completed' || a === undefined) return [f(false, DOC_A, `a completed ctx.callAI call MUST carry attempts (run ${run.status})`)];
204
+ return [f(a['usageComplete'] === false, DOC_A, `a request that reported no usage MUST make usageComplete false (got ${JSON.stringify(a['usageComplete'])})`)];
205
+ }
206
+
207
+ /** §A.3: absent without responseSchema; the declared mode with one. */
208
+ export function judgeStructuredOutput(plain: CaseRun, native: CaseRun): Finding[] {
209
+ const out: Finding[] = [];
210
+ for (const [run, label] of [[plain, 'plain'], [native, 'native']] as const) {
211
+ if (run.status !== 'completed' || run.attempts === undefined) out.push(f(false, DOC_A, `the ${label} call MUST complete with attempts (run ${run.status})`));
212
+ }
213
+ if (plain.attempts !== undefined) out.push(f(plain.attempts['structuredOutput'] === undefined, DOC_A, `without responseSchema attempts.structuredOutput MUST be absent (got ${JSON.stringify(plain.attempts['structuredOutput'])})`));
214
+ if (native.attempts !== undefined) out.push(f(native.attempts['structuredOutput'] === 'native', DOC_A, `the final request used the mock's native structured output, so structuredOutput MUST be "native" (got ${JSON.stringify(native.attempts['structuredOutput'])})`));
215
+ return out;
216
+ }
217
+
218
+ /** §B.1: maxAttempts 1 with an invalid first reply — one request, then failure. */
219
+ export function judgeCap(run: CaseRun, withUsageEvents: boolean): Finding[] {
220
+ const out: Finding[] = [f(run.status === 'failed', DOC_B, `with maxAttempts 1 and an invalid first reply the call MUST fail, since a completion needs a second request (run ${run.status})`)];
221
+ out.push(f(run.errorCode !== 'validation_error', DOC_B, 'a well-formed retry policy MUST NOT be refused as validation_error'));
222
+ if (withUsageEvents) out.push(f(run.usage.length <= 1, DOC_B, `the host MUST NOT send more than maxAttempts provider requests: ${run.usage.length} provider.usage for maxAttempts 1`));
223
+ if (run.exhaustedTotal !== undefined) out.push(f(run.exhaustedTotal === 1, DOC_B, `envelope.retry-exhausted.totalAttempts MUST be 1 under maxAttempts 1 (got ${run.exhaustedTotal})`));
224
+ return out;
225
+ }
226
+
227
+ /**
228
+ * §B.2: growMaxTokens false. The second reply fits only a grown budget, so a
229
+ * host that keeps the budget never completes, and every reported attempt stays
230
+ * within it.
231
+ */
232
+ export function judgeNoGrowth(run: CaseRun, withUsageEvents: boolean): Finding[] {
233
+ const out: Finding[] = [f(run.status !== 'completed', DOC_B, `under growMaxTokens false the over-budget reply MUST stay truncated, so the call MUST NOT complete — completing means an attempt's maxTokens exceeded ${BUDGET}`)];
234
+ out.push(f(run.errorCode !== 'validation_error', DOC_B, 'a well-formed retry policy MUST NOT be refused as validation_error'));
235
+ if (withUsageEvents) {
236
+ const over = run.usage.map((u) => u['outputTokens']).filter((t) => typeof t === 'number' && t > BUDGET);
237
+ out.push(f(over.length === 0, DOC_B, `no attempt MAY exceed the request's maxTokens ${BUDGET} under growMaxTokens false (provider.usage outputTokens over it: ${JSON.stringify(over)})`));
238
+ }
239
+ return out;
240
+ }
241
+
242
+ /** §B.5: a malformed retry is refused validation_error naming the key. */
243
+ export function judgeMalformed(run: CaseRun): Finding[] {
244
+ return [
245
+ f(run.status === 'failed' && run.errorCode === 'validation_error', DOC_B, `retry.maxAttempts 0 MUST be refused with validation_error, never ignored (run ${run.status} ${run.errorCode ?? ''})`.trim()),
246
+ f(run.errorField === 'retry.maxAttempts', DOC_B, `the refusal MUST name the key in details.field (got ${JSON.stringify(run.errorField)})`),
247
+ ];
248
+ }
249
+
250
+ /** §A.5: a replay fork reproduces the recorded attempts. */
251
+ export function judgeReplayed(source: CaseRun, fork: CaseRun): Finding[] {
252
+ return [
253
+ f(fork.status === 'completed' && fork.attempts !== undefined, DOC_A, `the replay fork MUST complete with attempts (fork ${fork.status})`),
254
+ f(JSON.stringify(fork.attempts) === JSON.stringify(source.attempts), DOC_A, `on replay attempts MUST equal the recorded value (source ${JSON.stringify(source.attempts)}, fork ${JSON.stringify(fork.attempts)})`),
255
+ ];
256
+ }
@@ -0,0 +1,74 @@
1
+ /**
2
+ * RFC 0081 §C — the `eval.*` envelope of an eval run, and the summary it
3
+ * serves. A pure judge over one eval run, for
4
+ * `v2-eval-trajectory-trials.test.ts`.
5
+ *
6
+ * The rules are the `run-event-payloads` descriptions, which the `agents`
7
+ * family lists as normative text:
8
+ * - `evalStarted` — emitted ONCE, at the start of the run;
9
+ * - `evalScored` — once per task (per trial under RFC 0244 §B);
10
+ * - `evalCompleted` — emitted ONCE, after all tasks and before
11
+ * `run.completed`; its aggregates describe the same run the summary does.
12
+ * The served `EvalSummary` validates against `eval-summary.schema.json`.
13
+ *
14
+ * Until 2.46.20 no v2 leg read `eval.completed`; the frozen v1
15
+ * `agent-eval-run.test.ts` did (RFC 0081 correction note, 2026-10-24).
16
+ *
17
+ * @see RFCS/0081-agent-evaluation-and-scorecards.md
18
+ */
19
+
20
+ export interface EnvelopeFinding { readonly ok: boolean; readonly doc: string; readonly message: string }
21
+
22
+ type Rec = Record<string, unknown>;
23
+ interface Event { readonly type: string; readonly sequence: number; readonly payload: Rec }
24
+
25
+ const isRec = (v: unknown): v is Rec => typeof v === 'object' && v !== null && !Array.isArray(v);
26
+ const f = (ok: boolean, doc: string, message: string): EnvelopeFinding => ({ ok, doc, message });
27
+
28
+ const C = 'run-event-payloads evalStarted / evalScored / evalCompleted (RFC 0081 §C)';
29
+
30
+ export interface EnvelopeOptions {
31
+ /** Validates an `eval.completed` payload against `run-event-payloads#/$defs/evalCompleted`. */
32
+ readonly completedValid: (payload: unknown) => { ok: boolean; errors: string };
33
+ }
34
+
35
+ export function judgeEvalEnvelope(events: readonly Event[], summary: Rec, opts: EnvelopeOptions): EnvelopeFinding[] {
36
+ const ordered = [...events].sort((a, b) => a.sequence - b.sequence);
37
+ const at = (type: string): number[] => ordered.flatMap((e, i) => (e.type === type ? [i] : []));
38
+ const started = at('eval.started');
39
+ const scored = at('eval.scored');
40
+ const completed = at('eval.completed');
41
+ const runCompleted = at('run.completed');
42
+ const out: EnvelopeFinding[] = [];
43
+
44
+ out.push(f(started.length === 1, C, `eval.started MUST be emitted once (got ${started.length})`));
45
+ out.push(f(completed.length === 1, C, `eval.completed MUST be emitted once (got ${completed.length})`));
46
+ out.push(f(scored.length > 0, C, 'the run MUST emit eval.scored for its tasks'));
47
+ if (started.length === 1 && scored.length > 0) {
48
+ out.push(f(started[0]! < scored[0]!, C, 'eval.started MUST precede every eval.scored'));
49
+ }
50
+ if (completed.length !== 1) return out;
51
+ const c = completed[0]!;
52
+ if (scored.length > 0) out.push(f(c > scored[scored.length - 1]!, C, 'eval.completed MUST follow every eval.scored ("after all tasks")'));
53
+ if (runCompleted.length > 0) out.push(f(c < runCompleted[0]!, C, 'eval.completed MUST precede run.completed'));
54
+
55
+ const payload = ordered[c]!.payload;
56
+ const v = opts.completedValid(payload);
57
+ out.push(f(v.ok, C, `eval.completed MUST validate: ${v.errors}`));
58
+ for (const k of ['taskCount', 'passedCount', 'passed'] as const) {
59
+ if (k in summary) out.push(f(payload[k] === summary[k], C, `eval.completed.${k} MUST equal the served summary's (event ${JSON.stringify(payload[k])}, summary ${JSON.stringify(summary[k])})`));
60
+ }
61
+ return out;
62
+ }
63
+
64
+ /** The served summary validates, and its per-task rows agree with its own counts. */
65
+ export function judgeSummaryServed(summary: Rec, valid: (s: unknown) => { ok: boolean; errors: string }): EnvelopeFinding[] {
66
+ const out: EnvelopeFinding[] = [];
67
+ const v = valid(summary);
68
+ out.push(f(v.ok, 'eval-summary.schema.json (RFC 0081 §C)', `the served EvalSummary MUST validate: ${v.errors}`));
69
+ const tasks = Array.isArray(summary['tasks']) ? (summary['tasks'] as unknown[]).filter(isRec) : [];
70
+ out.push(f(summary['taskCount'] === tasks.length, 'eval-summary.schema.json (RFC 0081 §C)', `taskCount MUST equal the number of tasks (got ${JSON.stringify(summary['taskCount'])}, ${tasks.length} tasks)`));
71
+ const passed = tasks.filter((t) => t['passed'] === true).length;
72
+ out.push(f(summary['passedCount'] === passed, 'eval-summary.schema.json (RFC 0081 §C)', `passedCount MUST count the passed tasks (got ${JSON.stringify(summary['passedCount'])}, ${passed} passed)`));
73
+ return out;
74
+ }
@@ -0,0 +1,148 @@
1
+ /**
2
+ * v2 — `ctx.callAI` reports its provider attempts and takes a per-call retry
3
+ * policy (RFC 0248; the rules are the `aiProviders.callAccounting` facet
4
+ * descriptions in `spec/v2/facets/aiProviders.schema.json`).
5
+ *
6
+ * Unaided, through the `conformance-callai-attempt-accounting` fixture: its
7
+ * node makes one `ctx.callAI` call, the host's conformance mock serves the
8
+ * case's script, and the node echoes `attempts`. The suite wrote the script,
9
+ * and `provider.usage` (one per provider invocation, `events.md`
10
+ * §`providerUsage`) corroborates it where advertised. Judges in
11
+ * `lib/callai-attempts-witness.ts`.
12
+ *
13
+ * attempts-count / output-tokens-sum invalid then valid → the count and the
14
+ * summed tokens match the script and log
15
+ * usage-incomplete a reply with no usage → usageComplete false
16
+ * structured-output absent without responseSchema; "native"
17
+ * when the mock declares it
18
+ * attempts-replayed a replay fork reproduces attempts
19
+ * max-attempts-cap maxAttempts 1, invalid first reply → fails
20
+ * after one request
21
+ * no-budget-growth growMaxTokens false → the over-budget
22
+ * reply stays truncated
23
+ * retry-malformed-refused maxAttempts 0 → validation_error,
24
+ * details.field retry.maxAttempts
25
+ *
26
+ * Dispositions: the flag not advertised, or the fixture not listed ⇒
27
+ * `inapplicable`. A run that cannot be created or read, or a count run that
28
+ * failed because the host does not retry a schema violation ⇒ `blocked`.
29
+ *
30
+ * @see RFCS/0248-callai-attempt-accounting-and-retry-policy.md
31
+ * @see conformance/fixtures.md §The ctx.callAI attempt-accounting fixture
32
+ */
33
+
34
+ import { describe, it, expect } from 'vitest';
35
+ import { v2Discovery } from '../lib/v2.js';
36
+ import { softSkip } from '../lib/soft-skip.js';
37
+ import { req } from '../lib/requirement-ids.js';
38
+ import { forkDeclined } from '../lib/fork-availability.js';
39
+ import {
40
+ CASES, runCase, replayOf, judgeCount, judgeSum, judgeIncomplete, judgeStructuredOutput, judgeCap, judgeNoGrowth, judgeMalformed, judgeReplayed,
41
+ type CaseName, type CaseRun, type Drive, type Finding,
42
+ } from '../lib/callai-attempts-witness.js';
43
+
44
+ const ID_COUNT = 'openwop.requirement.0248.attempts-count';
45
+ const ID_SUM = 'openwop.requirement.0248.output-tokens-sum';
46
+ const ID_INCOMPLETE = 'openwop.requirement.0248.usage-incomplete';
47
+ const ID_STRUCTURED = 'openwop.requirement.0248.structured-output';
48
+ const ID_REPLAYED = 'openwop.requirement.0248.attempts-replayed';
49
+ const ID_CAP = 'openwop.requirement.0248.max-attempts-cap';
50
+ const ID_GROWTH = 'openwop.requirement.0248.no-budget-growth';
51
+ const ID_MALFORMED = 'openwop.requirement.0248.retry-malformed-refused';
52
+ const UNREADABLE = 'v2 discovery unreachable — /.well-known/openwop did not answer 200 JSON under OpenWOP-Version: 2.0';
53
+
54
+ type Skip = readonly ['inapplicable' | 'blocked', string];
55
+ const isRec = (v: unknown): v is Record<string, unknown> => typeof v === 'object' && v !== null && !Array.isArray(v);
56
+
57
+ let disc: Promise<Record<string, unknown> | null> | undefined;
58
+ const discovery = (): Promise<Record<string, unknown> | null> => (disc ??= v2Discovery().catch(() => null));
59
+
60
+ async function gate(flag: 'attempts' | 'retryPolicy'): Promise<{ skip: Skip } | { usageEvents: boolean }> {
61
+ const doc = await discovery();
62
+ if (!doc) return { skip: ['blocked', UNREADABLE] };
63
+ const ai = doc['aiProviders'];
64
+ const acct = isRec(ai) && isRec(ai['callAccounting']) ? ai['callAccounting'] : undefined;
65
+ if (acct?.[flag] !== true) return { skip: ['inapplicable', `the host does not advertise aiProviders.callAccounting.${flag}`] };
66
+ return { usageEvents: isRec(doc['providerUsage']) };
67
+ }
68
+
69
+ const runs = new Map<CaseName, Promise<Drive>>();
70
+ const once = (name: CaseName): Promise<Drive> => { let p = runs.get(name); if (!p) { p = runCase(name); runs.set(name, p); } return p; };
71
+
72
+ async function observed(flag: 'attempts' | 'retryPolicy', name: CaseName): Promise<{ skip: Skip } | { run: CaseRun; usageEvents: boolean }> {
73
+ const g = await gate(flag);
74
+ if ('skip' in g) return g;
75
+ const d = await once(name);
76
+ if (d.kind === 'skip') return { skip: [d.disposition, d.reason] };
77
+ return { run: d.run, usageEvents: g.usageEvents };
78
+ }
79
+
80
+ /** The count run, or why it cannot be judged: a host that does not retry a schema violation fails the call and has no result to read. */
81
+ async function countRun(): Promise<{ skip: Skip } | { run: CaseRun; usageEvents: boolean }> {
82
+ const o = await observed('attempts', 'count');
83
+ if ('skip' in o) return o;
84
+ if (o.run.status === 'failed') return { skip: ['blocked', `the count call failed (${o.run.errorCode ?? 'no code'}): the host did not retry the scripted schema violation, so no multi-attempt result exists to read`] };
85
+ return o;
86
+ }
87
+
88
+ const assertAll = (id: string, fs: readonly Finding[]): void => { for (const x of fs) expect(x.ok, req(id, x.doc, x.message)).toBe(true); };
89
+
90
+ describe('v2 RFC 0248 — ctx.callAI attempt accounting (§A)', () => {
91
+ it('attempts.count equals the provider requests the call made', async () => {
92
+ const o = await countRun();
93
+ if ('skip' in o) return softSkip(...o.skip);
94
+ assertAll(ID_COUNT, judgeCount(o.run, o.usageEvents));
95
+ }, 180_000);
96
+
97
+ it('attempts.outputTokens is the sum over every attempt, and usageComplete is true when all reported', async () => {
98
+ const o = await countRun();
99
+ if ('skip' in o) return softSkip(...o.skip);
100
+ assertAll(ID_SUM, judgeSum(o.run, CASES.count, o.usageEvents));
101
+ }, 180_000);
102
+
103
+ it('an attempt that reported no usage makes usageComplete false', async () => {
104
+ const o = await observed('attempts', 'incomplete');
105
+ if ('skip' in o) return softSkip(...o.skip);
106
+ assertAll(ID_INCOMPLETE, judgeIncomplete(o.run));
107
+ }, 180_000);
108
+
109
+ it('structuredOutput is absent without responseSchema and names the mode the final attempt used', async () => {
110
+ const p = await observed('attempts', 'plain');
111
+ if ('skip' in p) return softSkip(...p.skip);
112
+ const n = await observed('attempts', 'native');
113
+ if ('skip' in n) return softSkip(...n.skip);
114
+ assertAll(ID_STRUCTURED, judgeStructuredOutput(p.run, n.run));
115
+ }, 180_000);
116
+
117
+ it('a replay fork reproduces the recorded attempts', async () => {
118
+ const o = await countRun();
119
+ if ('skip' in o) return softSkip(...o.skip);
120
+ const doc = await discovery();
121
+ const modes = isRec(doc?.['replay']) && Array.isArray((doc['replay'] as Record<string, unknown>)['modes']) ? ((doc['replay'] as Record<string, unknown>)['modes'] as unknown[]) : [];
122
+ if (!modes.includes('replay')) return softSkip('inapplicable', 'the host does not advertise replay.modes including "replay"');
123
+ const fork = await replayOf(o.run);
124
+ if (typeof fork === 'string') return softSkip('blocked', fork);
125
+ if ('declined' in fork) { forkDeclined(fork.declined, 'v2-callai-attempt-accounting replay'); return softSkip('blocked', `POST /runs/{runId}:fork declined with ${fork.declined}`); }
126
+ assertAll(ID_REPLAYED, judgeReplayed(o.run, fork.run));
127
+ }, 240_000);
128
+ });
129
+
130
+ describe('v2 RFC 0248 — ctx.callAI retry policy (§B)', () => {
131
+ it('maxAttempts caps the provider requests', async () => {
132
+ const o = await observed('retryPolicy', 'cap');
133
+ if ('skip' in o) return softSkip(...o.skip);
134
+ assertAll(ID_CAP, judgeCap(o.run, o.usageEvents));
135
+ }, 180_000);
136
+
137
+ it('growMaxTokens false keeps every attempt within the request budget', async () => {
138
+ const o = await observed('retryPolicy', 'growth');
139
+ if ('skip' in o) return softSkip(...o.skip);
140
+ assertAll(ID_GROWTH, judgeNoGrowth(o.run, o.usageEvents));
141
+ }, 180_000);
142
+
143
+ it('a malformed retry policy is refused validation_error naming the key', async () => {
144
+ const o = await observed('retryPolicy', 'malformed');
145
+ if ('skip' in o) return softSkip(...o.skip);
146
+ assertAll(ID_MALFORMED, judgeMalformed(o.run));
147
+ }, 180_000);
148
+ });
@@ -32,6 +32,13 @@
32
32
  * lists `conformance-eval-trials` refuses it at createRun with
33
33
  * `400 validation_error`.
34
34
  *
35
+ * RFC 0081 §C, on the `conformance-eval-trajectory` run (2.46.20). Judges in
36
+ * `lib/eval-envelope-witness.ts`.
37
+ * - `eval.started` and `eval.completed` are emitted once each, bracketing
38
+ * every `eval.scored`, and `eval.completed` precedes `run.completed`; its
39
+ * counts equal the served summary's.
40
+ * - The served `EvalSummary` validates, and its counts match its rows.
41
+ *
35
42
  * RFC 0246 (safety fix): a trial that failed its trajectory counts 0 in the
36
43
  * two pass decisions, while every reported score is unchanged. Judges in
37
44
  * `lib/eval-gate-witness.ts`.
@@ -73,6 +80,7 @@ import { majorProfile } from '../lib/major-profile.js';
73
80
  import { isFixtureAdvertised } from '../lib/fixtures.js';
74
81
  import { FIXTURES_DIR } from '../lib/paths.js';
75
82
  import { pollTerminal, readEvents, isRecord, type Ev } from '../lib/fixture-run-observer.js';
83
+ import { judgeEvalEnvelope, judgeSummaryServed } from '../lib/eval-envelope-witness.js';
76
84
  import { gradeTrajectory, passAtK, passHatK, scoreStderr, withinTolerance, type TrajectorySpec } from '../lib/eval-trajectory.js';
77
85
  import { judgeMeanGated, judgeSuiteGated } from '../lib/eval-gate-witness.js';
78
86
  import { judgeFixtureErrorReplayed } from '../lib/eval-fixture-error-witness.js';
@@ -97,6 +105,9 @@ const DOC_0246 = 'eval-summary.schema.json gatedScore / gatedAggregateScore (RFC
97
105
  const ID_FIXTURE_ERROR_SHAPE = 'openwop.requirement.0245.fixture-error-shape';
98
106
  const ID_FIXTURE_ERROR = 'openwop.requirement.0245.fixture-error-replayed';
99
107
  const ID_FIXTURE_ERROR_REFUSED = 'openwop.requirement.0245.unadvertised-refused';
108
+ const ID_ENVELOPE = 'openwop.requirement.0081.eval-envelope';
109
+ const ID_SUMMARY = 'openwop.requirement.0081.summary-served';
110
+ const DOC_0081 = 'run-event-payloads evalStarted / evalCompleted, eval-summary.schema.json (RFC 0081 §C)';
100
111
  const DOC_0245 = 'agents.schema.json §evalSuite.fixtureErrors (RFC 0245)';
101
112
 
102
113
  interface Task { taskId: string; input: { script: Array<{ tool: string; arguments?: unknown }>; output: string }; expected: { kind: string; trajectory?: TrajectorySpec }; fixtures?: { toolResponses?: Array<Record<string, unknown>> } }
@@ -107,6 +118,7 @@ const suiteSchema = v2Validator('agent-eval-suite');
107
118
  const summarySchema = v2Validator('eval-summary');
108
119
  const scored = v2RefValidator('run-event-payloads.schema.json#/$defs/evalScored');
109
120
  const started = v2RefValidator('run-event-payloads.schema.json#/$defs/evalStarted');
121
+ const completedPayload = v2RefValidator('run-event-payloads.schema.json#/$defs/evalCompleted');
110
122
 
111
123
  async function http(fn: () => Promise<OpenWOPResponse>): Promise<OpenWOPResponse | null> {
112
124
  try { return await fn(); } catch { return null; }
@@ -225,6 +237,33 @@ describe('v2 RFC 0244 — host behaviour', () => {
225
237
  expect(calls, req(ID_TRAJECTORY, DOC, `the log MUST carry one agent.tool-called per scripted call (${scripted})`)).toBe(scripted);
226
238
  });
227
239
 
240
+ async function trajectoryRun(): Promise<{ skip: [Parameters<typeof softSkip>[0], string] } | { o: Extract<Observed, { kind: 'ok' }> }> {
241
+ const { doc, facet } = await evalSuiteFacet();
242
+ if (!doc) return { skip: ['blocked', UNREADABLE] };
243
+ if (facet?.['trajectory'] !== true) return { skip: ['inapplicable', 'the host does not advertise agents.evalSuite.trajectory'] };
244
+ const id = 'conformance-eval-trajectory';
245
+ if (!isFixtureAdvertised(id)) return { skip: ['inapplicable', `the host does not advertise fixture ${id}`] };
246
+ const o = await runSuite(id);
247
+ if (o.kind === 'skip') return { skip: [o.disposition, o.reason] };
248
+ return { o };
249
+ }
250
+
251
+ it('an eval run brackets its scores with one eval.started and one eval.completed (RFC 0081 §C)', async () => {
252
+ const r = await trajectoryRun();
253
+ if ('skip' in r) return softSkip(...r.skip);
254
+ for (const x of judgeEvalEnvelope(r.o.events, r.o.summary, { completedValid: completedPayload })) {
255
+ expect(x.ok, req(ID_ENVELOPE, DOC_0081, x.message)).toBe(true);
256
+ }
257
+ });
258
+
259
+ it('the served EvalSummary validates and its counts match its rows (RFC 0081 §C)', async () => {
260
+ const r = await trajectoryRun();
261
+ if ('skip' in r) return softSkip(...r.skip);
262
+ for (const x of judgeSummaryServed(r.o.summary, summarySchema)) {
263
+ expect(x.ok, req(ID_SUMMARY, DOC_0081, x.message)).toBe(true);
264
+ }
265
+ });
266
+
228
267
  async function trialsRun(): Promise<{ skip: [Parameters<typeof softSkip>[0], string] } | { o: Extract<Observed, { kind: 'ok' }>; suite: Suite; n: number; k: number }> {
229
268
  const { doc, facet } = await evalSuiteFacet();
230
269
  if (!doc) return { skip: ['blocked', UNREADABLE] };