@openwop/openwop-conformance 2.48.0 → 2.49.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +16 -0
- package/README.md +2 -2
- package/dist/spec-artifacts.lock.json +2 -2
- package/fixtures.md +3 -2
- package/package.json +2 -2
- package/requirements.json +183 -8
- package/scenario-majors.json +11 -2
- package/schemas/CORPUS-STAMP.json +11 -11
- package/src/lib/budget-accounting-witness.ts +129 -0
- package/src/scenarios/v2-budget-accounting.test.ts +107 -0
- package/src/scenarios/v2-event-log-family.test.ts +237 -0
- package/src/scenarios/v2-production-profile.test.ts +118 -0
- package/src/scenarios/v2-secrets-run-witness.test.ts +16 -3
- package/src/scenarios/v2-telemetry-export.test.ts +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,19 @@
|
|
|
1
|
+
## [2.49.0] — 2026-11-14 — Gated legs for eventLog, budget and production
|
|
2
|
+
|
|
3
|
+
- **`v2-event-log-family`** (gated on `eventLog`). The era is stamped on runs created by createRun and by forkRun. The poll cursor contract holds at every position of a settled log. Event types are codemap names or registered vendor types.
|
|
4
|
+
- **`v2-budget-accounting`** (gated on `budget`).
|
|
5
|
+
- `consumed` is derived from `agent.tool-called`.
|
|
6
|
+
- The reserved limit is the one every accounting event names.
|
|
7
|
+
- `threshold-crossed` is emitted once, at the policy percent.
|
|
8
|
+
|
|
9
|
+
The judge is `lib/budget-accounting-witness.ts`, with a sabotage self-test.
|
|
10
|
+
- **`v2-production-profile`** (gated on `production`). Five same-key create retries replay without a second run. An expired run answers `404 not_found` or `410 run_expired`; this leg needs the operator.
|
|
11
|
+
- **`v2-secrets-run-witness`** gates on the `secrets` record before `secrets.runSecrets`. The verdict is unchanged; the change is so RFC 0264's derivation counts a host that passes.
|
|
12
|
+
|
|
13
|
+
## [2.48.1] — 2026-11-14 — Fix the telemetry shape leg
|
|
14
|
+
|
|
15
|
+
- **RFC 0262.** `v2-telemetry-export` (`0262.shape`): the positive records now carry `until`, because the schema requires it while a family is `experimental`. Before this fix the leg failed on every host. Reported by openwop-app-88 and openwop-app-14.
|
|
16
|
+
|
|
1
17
|
## [2.48.0] — 2026-11-13 — Compensation unwind and derived adoption
|
|
2
18
|
|
|
3
19
|
- **RFC 0265.** `v2-compensation-unwind`: five host legs gated on `compensation` and the `conformance-compensate-unwind` fixture (legs 3 and 5 also on `idempotency`), reading only the event log, `GET /runs/{runId}/compensation`, `GET /runs/{runId}/effects` and a replay fork. The judges live in `lib/compensation-unwind.ts`, with a self-test that includes sabotage cases.
|
package/README.md
CHANGED
|
@@ -137,7 +137,7 @@ Each scenario file records its disposition and assertion count in `<report-dir>/
|
|
|
137
137
|
|
|
138
138
|
## What's Covered
|
|
139
139
|
|
|
140
|
-
The current suite has
|
|
140
|
+
The current suite has 674 scenario files under `src/scenarios/`. By target major ([`scenario-majors.json`](./scenario-majors.json)): 234 files run against a v2 host and 455 against a v1 host; 15 of them run at both.
|
|
141
141
|
|
|
142
142
|
The v2 scenarios (`src/scenarios/v2-*.test.ts`) cover, by area:
|
|
143
143
|
|
|
@@ -166,7 +166,7 @@ Three kinds of scenario share the tree:
|
|
|
166
166
|
- **Host** checks drive a live host over HTTP.
|
|
167
167
|
- **Gated** checks run only when the host advertises the family, profile or fixture they need, and otherwise record `inapplicable` or `blocked` with a reason. `--require-behavior` turns an advertised-but-unwitnessable behaviour into a failure.
|
|
168
168
|
|
|
169
|
-
Current source tree:
|
|
169
|
+
Current source tree: 674 scenario files. [`coverage.md`](./coverage.md) maps each spec document and every OpenAPI operation to its scenarios. [`CHANGELOG.md`](./CHANGELOG.md) records when each scenario landed.
|
|
170
170
|
|
|
171
171
|
## How the 2.x suite is built
|
|
172
172
|
|
package/fixtures.md
CHANGED
|
@@ -405,6 +405,7 @@ The `messages`-mode stream fixture (AI token streaming) is covered by the determ
|
|
|
405
405
|
- **Expected behavior**:
|
|
406
406
|
- `POST /runs` with the same `Idempotency-Key` and same body twice → second response MUST replay the first (`openwop-Idempotent-Replay: true` header) and MUST NOT create a second run.
|
|
407
407
|
- Same `Idempotency-Key` with a different body → 409.
|
|
408
|
+
- **v2**: `v2-production-profile` replays one key across five retries (conformance.md §Production profile); each retry MUST return the first response's status and runId with `OpenWOP-Idempotent-Replay: true`.
|
|
408
409
|
- **Terminal status**: `completed`.
|
|
409
410
|
|
|
410
411
|
### `conformance-cancellable`
|
|
@@ -537,7 +538,7 @@ The `messages`-mode stream fixture (AI token streaming) is covered by the determ
|
|
|
537
538
|
|
|
538
539
|
### `conformance-budget-tool-calls`
|
|
539
540
|
|
|
540
|
-
- **Purpose**: spend a run budget unaided, so `budget` can be witnessed without a model or a seam (`v2-budget-enforcement`).
|
|
541
|
+
- **Purpose**: spend a run budget unaided, so `budget` can be witnessed without a model or a seam (`v2-budget-enforcement`, `v2-budget-accounting`).
|
|
541
542
|
- **Shape**: one `core.conformance.mock-agent` node with three `mockToolCalls`. Each call counts once against the `toolCalls` dimension.
|
|
542
543
|
- **Inputs**: none.
|
|
543
544
|
- **Expected behavior**: created with `configurable.budget = { maxToolCalls: 2, thresholdPercent: 50, onExhaustion: "fail" }`, the run emits `budget.reserved`, then `budget.threshold-crossed`, then `budget.exhausted`. A host advertising `enforce: "hard"` then emits `cap.breached` (`kind: "budget-tool-calls"`) and fails the run `budget_exhausted`; an `advisory` host lets it complete.
|
|
@@ -857,7 +858,7 @@ The scope leg sends a promote with `OPENWOP_TEST_LOW_SCOPE_KEY`, a key with no `
|
|
|
857
858
|
`conformance-compensate-unwind` is what `v2-compensation-unwind` drives. Its effect node, `core.conformance.compensable-effect`, is conformance-RESERVED. A host that advertises `compensation` and serves the fixture maps it to a node it records on the effect ledger (`GET /runs/{runId}/effects`) as exactly one effect per execution, forward and inverse alike, and runs it as the inverse action when its resolved inputs carry `inverse: true`. What the effect does is irrelevant. A host that does not advertise `compensation` MUST NOT advertise the fixture: it would refuse the workflow's `settings.compensation` with `capability_required`.
|
|
858
859
|
|
|
859
860
|
- **Shape**: `e1` → `e2` → `e3` (`core.conformance.compensable-effect`, each with node-level `compensation.nodeTypeId` = the same type and `inputMapping: { inverse: true }`) → `fail` (`core.fail`). `settings.compensation`: `orderingModel: "reverse-completion"`, `triggers: ["node-failure"]`.
|
|
860
|
-
- **Inputs**: none.
|
|
861
|
+
- **Inputs**: none. The `inputMapping` value `{ inverse: true }` is a literal constant, not a path into recorded outputs. The inverse is not a graph node and declares no compensation of its own, so naming the forward type as its own inverse is not a compensation cycle.
|
|
861
862
|
- **Expected behavior**:
|
|
862
863
|
1. `e1`, `e2` and `e3` complete in order; `fail` fails; the run ends `failed`.
|
|
863
864
|
2. The log holds `compensation.requested` before the first `compensation.started`, and ends the unwind with `compensation.completed`.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@openwop/openwop-conformance",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.49.0",
|
|
4
4
|
"description": "Black-box conformance suite for OpenWOP hosts (v2 current; v1 scenarios retained for the frozen v1 tree).",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -56,6 +56,6 @@
|
|
|
56
56
|
"@openwop/spec-artifacts": "file:../spec-artifacts"
|
|
57
57
|
},
|
|
58
58
|
"peerDependencies": {
|
|
59
|
-
"@openwop/spec-artifacts": "2.
|
|
59
|
+
"@openwop/spec-artifacts": "2.49.0"
|
|
60
60
|
}
|
|
61
61
|
}
|
package/requirements.json
CHANGED
|
@@ -2,11 +2,11 @@
|
|
|
2
2
|
"$comment": "GENERATED by conformance/scripts/generate-requirement-registry.mjs — do not edit. One record per it()/test() in src/scenarios. Ids: openwop.it.<file-stem>.<title-slug>[~n] (src/lib/requirement-ids.ts). A record with id null has an interpolated title; its run-time row is keyed by the rendered title and maps here by file+line only. Renamed ids need a row in requirement-aliases.json.",
|
|
3
3
|
"generatedFrom": "src/scenarios/*.test.ts",
|
|
4
4
|
"counts": {
|
|
5
|
-
"files":
|
|
6
|
-
"tests":
|
|
7
|
-
"withStableId":
|
|
5
|
+
"files": 759,
|
|
6
|
+
"tests": 2919,
|
|
7
|
+
"withStableId": 2919,
|
|
8
8
|
"interpolatedTitles": 0,
|
|
9
|
-
"explicitIds":
|
|
9
|
+
"explicitIds": 2763
|
|
10
10
|
},
|
|
11
11
|
"records": [
|
|
12
12
|
{
|
|
@@ -28932,6 +28932,48 @@
|
|
|
28932
28932
|
"explicitId": null,
|
|
28933
28933
|
"citations": []
|
|
28934
28934
|
},
|
|
28935
|
+
{
|
|
28936
|
+
"id": "openwop.it.v2-budget-accounting.every-toolcalls-accounting-event-reports-consumed-equal-to-the-agent-tool-called",
|
|
28937
|
+
"file": "v2-budget-accounting.test.ts",
|
|
28938
|
+
"line": 90,
|
|
28939
|
+
"title": "every toolCalls accounting event reports consumed equal to the agent.tool-called events before it",
|
|
28940
|
+
"explicitId": "openwop.requirement.0084.consumption-derived",
|
|
28941
|
+
"citations": [
|
|
28942
|
+
{
|
|
28943
|
+
"section": null,
|
|
28944
|
+
"requirement": null,
|
|
28945
|
+
"interpolated": true
|
|
28946
|
+
}
|
|
28947
|
+
]
|
|
28948
|
+
},
|
|
28949
|
+
{
|
|
28950
|
+
"id": "openwop.it.v2-budget-accounting.budget-reserved-records-an-effective-budget-within-the-run-ceiling-and-every-acc",
|
|
28951
|
+
"file": "v2-budget-accounting.test.ts",
|
|
28952
|
+
"line": 96,
|
|
28953
|
+
"title": "budget.reserved records an effective budget within the run ceiling, and every accounting event names that limit",
|
|
28954
|
+
"explicitId": "openwop.requirement.0084.reserved-limit-enforced",
|
|
28955
|
+
"citations": [
|
|
28956
|
+
{
|
|
28957
|
+
"section": null,
|
|
28958
|
+
"requirement": null,
|
|
28959
|
+
"interpolated": true
|
|
28960
|
+
}
|
|
28961
|
+
]
|
|
28962
|
+
},
|
|
28963
|
+
{
|
|
28964
|
+
"id": "openwop.it.v2-budget-accounting.budget-threshold-crossed-is-emitted-once-at-the-policy-percent-and-not-before-sp",
|
|
28965
|
+
"file": "v2-budget-accounting.test.ts",
|
|
28966
|
+
"line": 102,
|
|
28967
|
+
"title": "budget.threshold-crossed is emitted once, at the policy percent, and not before spend reached it",
|
|
28968
|
+
"explicitId": "openwop.requirement.0084.threshold-once",
|
|
28969
|
+
"citations": [
|
|
28970
|
+
{
|
|
28971
|
+
"section": null,
|
|
28972
|
+
"requirement": null,
|
|
28973
|
+
"interpolated": true
|
|
28974
|
+
}
|
|
28975
|
+
]
|
|
28976
|
+
},
|
|
28935
28977
|
{
|
|
28936
28978
|
"id": "openwop.it.v2-budget-enforcement.a-budgeted-run-emits-budget-reserved-budget-threshold-crossed-and-budget-exhaust",
|
|
28937
28979
|
"file": "v2-budget-enforcement.test.ts",
|
|
@@ -31700,6 +31742,92 @@
|
|
|
31700
31742
|
}
|
|
31701
31743
|
]
|
|
31702
31744
|
},
|
|
31745
|
+
{
|
|
31746
|
+
"id": "openwop.it.v2-event-log-family.every-run-the-host-creates-by-createrun-and-by-forkrun-carries-the-advertised-er",
|
|
31747
|
+
"file": "v2-event-log-family.test.ts",
|
|
31748
|
+
"line": 127,
|
|
31749
|
+
"title": "every run the host creates, by createRun and by forkRun, carries the advertised era 3",
|
|
31750
|
+
"explicitId": "openwop.requirement.0036.event-log-stamp-every-path",
|
|
31751
|
+
"citations": [
|
|
31752
|
+
{
|
|
31753
|
+
"section": "spec/v2/core/persistence.md §The era key",
|
|
31754
|
+
"requirement": null,
|
|
31755
|
+
"interpolated": true
|
|
31756
|
+
},
|
|
31757
|
+
{
|
|
31758
|
+
"section": "spec/v2/core/persistence.md §The eventLog family",
|
|
31759
|
+
"requirement": null,
|
|
31760
|
+
"interpolated": true
|
|
31761
|
+
},
|
|
31762
|
+
{
|
|
31763
|
+
"section": "spec/v2/core/persistence.md §The eventLog family",
|
|
31764
|
+
"requirement": null,
|
|
31765
|
+
"interpolated": true
|
|
31766
|
+
}
|
|
31767
|
+
]
|
|
31768
|
+
},
|
|
31769
|
+
{
|
|
31770
|
+
"id": "openwop.it.v2-event-log-family.a-cursor-at-every-position-of-a-settled-log-serves-the-poll-contract",
|
|
31771
|
+
"file": "v2-event-log-family.test.ts",
|
|
31772
|
+
"line": 154,
|
|
31773
|
+
"title": "a cursor at every position of a settled log serves the poll contract",
|
|
31774
|
+
"explicitId": "openwop.requirement.0036.event-log-cursor-contract",
|
|
31775
|
+
"citations": [
|
|
31776
|
+
{
|
|
31777
|
+
"section": "spec/v2/core/events.md §Poll",
|
|
31778
|
+
"requirement": null,
|
|
31779
|
+
"interpolated": true
|
|
31780
|
+
},
|
|
31781
|
+
{
|
|
31782
|
+
"section": "spec/v2/core/events.md §Poll",
|
|
31783
|
+
"requirement": null,
|
|
31784
|
+
"interpolated": true
|
|
31785
|
+
},
|
|
31786
|
+
{
|
|
31787
|
+
"section": "spec/v2/core/events.md §Poll",
|
|
31788
|
+
"requirement": null,
|
|
31789
|
+
"interpolated": true
|
|
31790
|
+
},
|
|
31791
|
+
{
|
|
31792
|
+
"section": "spec/v2/core/events.md §Poll",
|
|
31793
|
+
"requirement": null,
|
|
31794
|
+
"interpolated": true
|
|
31795
|
+
},
|
|
31796
|
+
{
|
|
31797
|
+
"section": "spec/v2/core/events.md §Poll",
|
|
31798
|
+
"requirement": null,
|
|
31799
|
+
"interpolated": true
|
|
31800
|
+
},
|
|
31801
|
+
{
|
|
31802
|
+
"section": "spec/v2/core/events.md §Poll",
|
|
31803
|
+
"requirement": null,
|
|
31804
|
+
"interpolated": true
|
|
31805
|
+
},
|
|
31806
|
+
{
|
|
31807
|
+
"section": "spec/v2/core/events.md §Poll",
|
|
31808
|
+
"requirement": "a cursor past the end of the log MUST answer 200"
|
|
31809
|
+
},
|
|
31810
|
+
{
|
|
31811
|
+
"section": "spec/v2/core/events.md §Poll",
|
|
31812
|
+
"requirement": null,
|
|
31813
|
+
"interpolated": true
|
|
31814
|
+
}
|
|
31815
|
+
]
|
|
31816
|
+
},
|
|
31817
|
+
{
|
|
31818
|
+
"id": "openwop.it.v2-event-log-family.every-event-type-on-the-lifecycle-fixtures-logs-is-a-codemap-name-or-a-registere",
|
|
31819
|
+
"file": "v2-event-log-family.test.ts",
|
|
31820
|
+
"line": 206,
|
|
31821
|
+
"title": "every event type on the lifecycle fixtures' logs is a codemap name or a registered vendor type",
|
|
31822
|
+
"explicitId": "openwop.requirement.0036.event-log-codemap-types",
|
|
31823
|
+
"citations": [
|
|
31824
|
+
{
|
|
31825
|
+
"section": "spec/v2/core/persistence.md §The codemap is data",
|
|
31826
|
+
"requirement": null,
|
|
31827
|
+
"interpolated": true
|
|
31828
|
+
}
|
|
31829
|
+
]
|
|
31830
|
+
},
|
|
31703
31831
|
{
|
|
31704
31832
|
"id": "openwop.it.v2-event-type-closed.every-emitted-type-is-a-registered-protocol-type-or-a-vendor-type-under-an-adver",
|
|
31705
31833
|
"file": "v2-event-type-closed.test.ts",
|
|
@@ -35768,6 +35896,53 @@
|
|
|
35768
35896
|
}
|
|
35769
35897
|
]
|
|
35770
35898
|
},
|
|
35899
|
+
{
|
|
35900
|
+
"id": "openwop.it.v2-production-profile.a-host-claiming-production-replays-one-idempotency-key-across-five-retries-witho",
|
|
35901
|
+
"file": "v2-production-profile.test.ts",
|
|
35902
|
+
"line": 67,
|
|
35903
|
+
"title": "a host claiming production replays one Idempotency-Key across five retries without a second run",
|
|
35904
|
+
"explicitId": "openwop.requirement.0009.idempotency-five-retries",
|
|
35905
|
+
"citations": [
|
|
35906
|
+
{
|
|
35907
|
+
"section": "spec/v2/core/conformance.md §Production profile",
|
|
35908
|
+
"requirement": null,
|
|
35909
|
+
"interpolated": true
|
|
35910
|
+
},
|
|
35911
|
+
{
|
|
35912
|
+
"section": "spec/v2/core/idempotency.md §Outcomes",
|
|
35913
|
+
"requirement": null,
|
|
35914
|
+
"interpolated": true
|
|
35915
|
+
},
|
|
35916
|
+
{
|
|
35917
|
+
"section": "spec/v2/core/runs.md §Request headers",
|
|
35918
|
+
"requirement": null,
|
|
35919
|
+
"interpolated": true
|
|
35920
|
+
},
|
|
35921
|
+
{
|
|
35922
|
+
"section": "spec/v2/core/idempotency.md §Outcomes",
|
|
35923
|
+
"requirement": null,
|
|
35924
|
+
"interpolated": true
|
|
35925
|
+
}
|
|
35926
|
+
]
|
|
35927
|
+
},
|
|
35928
|
+
{
|
|
35929
|
+
"id": "openwop.it.v2-production-profile.an-expired-run-answers-404-not-found-or-410-run-expired",
|
|
35930
|
+
"file": "v2-production-profile.test.ts",
|
|
35931
|
+
"line": 89,
|
|
35932
|
+
"title": "an expired run answers 404 not_found or 410 run_expired",
|
|
35933
|
+
"explicitId": "openwop.requirement.0009.retention-expired-run",
|
|
35934
|
+
"citations": [
|
|
35935
|
+
{
|
|
35936
|
+
"section": "spec/v2/core/conformance.md §Production profile",
|
|
35937
|
+
"requirement": null,
|
|
35938
|
+
"interpolated": true
|
|
35939
|
+
},
|
|
35940
|
+
{
|
|
35941
|
+
"section": "spec/v2/core/errors.md",
|
|
35942
|
+
"requirement": "the refusal is the canonical envelope: message is a string"
|
|
35943
|
+
}
|
|
35944
|
+
]
|
|
35945
|
+
},
|
|
35771
35946
|
{
|
|
35772
35947
|
"id": "openwop.it.v2-profiles-derived-only.the-v2-document-carries-no-root-profiles",
|
|
35773
35948
|
"file": "v2-profiles-derived-only.test.ts",
|
|
@@ -38009,7 +38184,7 @@
|
|
|
38009
38184
|
{
|
|
38010
38185
|
"id": "openwop.it.v2-secrets-run-witness.a-run-supplied-secret-resolves-intact-matched-true-for-its-digest-false-for-anot",
|
|
38011
38186
|
"file": "v2-secrets-run-witness.test.ts",
|
|
38012
|
-
"line":
|
|
38187
|
+
"line": 98,
|
|
38013
38188
|
"title": "a run-supplied secret resolves intact: matched true for its digest, false for another",
|
|
38014
38189
|
"explicitId": "openwop.requirement.secrets.run-witness-resolves",
|
|
38015
38190
|
"citations": [
|
|
@@ -38039,7 +38214,7 @@
|
|
|
38039
38214
|
{
|
|
38040
38215
|
"id": "openwop.it.v2-secrets-run-witness.a-run-supplied-secret-appears-on-no-readable-surface-in-any-encoding-or-as-its-d",
|
|
38041
38216
|
"file": "v2-secrets-run-witness.test.ts",
|
|
38042
|
-
"line":
|
|
38217
|
+
"line": 114,
|
|
38043
38218
|
"title": "a run-supplied secret appears on no readable surface, in any encoding or as its digest",
|
|
38044
38219
|
"explicitId": "openwop.requirement.secrets.run-witness-redacted",
|
|
38045
38220
|
"citations": [
|
|
@@ -38053,7 +38228,7 @@
|
|
|
38053
38228
|
{
|
|
38054
38229
|
"id": "openwop.it.v2-secrets-run-witness.a-run-ref-resolves-only-within-the-run-that-supplied-it-and-the-witness-refuses",
|
|
38055
38230
|
"file": "v2-secrets-run-witness.test.ts",
|
|
38056
|
-
"line":
|
|
38231
|
+
"line": 125,
|
|
38057
38232
|
"title": "a run: ref resolves only within the run that supplied it, and the witness refuses any other ref",
|
|
38058
38233
|
"explicitId": "openwop.requirement.secrets.run-witness-scope-bound",
|
|
38059
38234
|
"citations": [
|
|
@@ -38071,7 +38246,7 @@
|
|
|
38071
38246
|
{
|
|
38072
38247
|
"id": "openwop.it.v2-secrets-run-witness.the-idempotency-request-digest-excludes-runsecrets-a-retry-differing-only-there",
|
|
38073
38248
|
"file": "v2-secrets-run-witness.test.ts",
|
|
38074
|
-
"line":
|
|
38249
|
+
"line": 157,
|
|
38075
38250
|
"title": "the idempotency request digest excludes runSecrets: a retry differing only there replays, and its value is discarded",
|
|
38076
38251
|
"explicitId": "openwop.requirement.secrets.run-secrets-outside-request-digest",
|
|
38077
38252
|
"citations": [
|
package/scenario-majors.json
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$comment": "GENERATED by conformance/scripts/generate-scenario-majors.mjs (RFC 0168 §D.3). Do not edit; add a file to BOTH_MAJORS in the generator to target both majors.",
|
|
3
3
|
"counts": {
|
|
4
|
-
"files":
|
|
4
|
+
"files": 674,
|
|
5
5
|
"v1": 455,
|
|
6
|
-
"v2":
|
|
6
|
+
"v2": 234
|
|
7
7
|
},
|
|
8
8
|
"majors": {
|
|
9
9
|
"a2a-1-0-agent-card.test.ts": [
|
|
@@ -1347,6 +1347,9 @@
|
|
|
1347
1347
|
"v2-bounded-loopback.test.ts": [
|
|
1348
1348
|
2
|
|
1349
1349
|
],
|
|
1350
|
+
"v2-budget-accounting.test.ts": [
|
|
1351
|
+
2
|
|
1352
|
+
],
|
|
1350
1353
|
"v2-budget-enforcement.test.ts": [
|
|
1351
1354
|
2
|
|
1352
1355
|
],
|
|
@@ -1494,6 +1497,9 @@
|
|
|
1494
1497
|
"v2-eval-trajectory-trials.test.ts": [
|
|
1495
1498
|
2
|
|
1496
1499
|
],
|
|
1500
|
+
"v2-event-log-family.test.ts": [
|
|
1501
|
+
2
|
|
1502
|
+
],
|
|
1497
1503
|
"v2-event-type-closed.test.ts": [
|
|
1498
1504
|
2
|
|
1499
1505
|
],
|
|
@@ -1674,6 +1680,9 @@
|
|
|
1674
1680
|
"v2-production-backpressure.test.ts": [
|
|
1675
1681
|
2
|
|
1676
1682
|
],
|
|
1683
|
+
"v2-production-profile.test.ts": [
|
|
1684
|
+
2
|
|
1685
|
+
],
|
|
1677
1686
|
"v2-profiles-derived-only.test.ts": [
|
|
1678
1687
|
2
|
|
1679
1688
|
],
|
|
@@ -1,17 +1,17 @@
|
|
|
1
1
|
{
|
|
2
2
|
"_comment": "Provenance of @openwop/spec-artifacts (RFC 0168 §D.2). files: SHA-256 per file; the conformance suite compares the installed peer against dist/spec-artifacts.lock.json at start.",
|
|
3
3
|
"package": "@openwop/spec-artifacts",
|
|
4
|
-
"version": "2.
|
|
5
|
-
"corpusTag": "v2.
|
|
4
|
+
"version": "2.49.0",
|
|
5
|
+
"corpusTag": "v2.49.0",
|
|
6
6
|
"files": {
|
|
7
7
|
"api/.redocly.lint-ignore.yaml": "bf5a8350b88a72fa43f59605ed8d903ed24b6cfccda5e45509c9f6ed9ee4e712",
|
|
8
8
|
"api/asyncapi.yaml": "d5ecb9ee6114582be3b1f662c84bfac9ae96dae7bacb853e461168f70a8e1c7d",
|
|
9
9
|
"api/grpc/openwop.proto": "c3e72bb17cba514ee98feb6434e6c9b6ea6795bfd086489ec69fd882dd1ad977",
|
|
10
10
|
"api/openapi.yaml": "ac6de5303872664b78ff419c035915593e01599e02835e777a0123de1baeb398",
|
|
11
11
|
"api/redocly.yaml": "b0604c89b2ca6d5076ec25725c539dad44a741a811fe524439ee6daef8baa09f",
|
|
12
|
-
"api/seams-v2.yaml": "
|
|
13
|
-
"api/v2/asyncapi.yaml": "
|
|
14
|
-
"api/v2/openapi.yaml": "
|
|
12
|
+
"api/seams-v2.yaml": "a4fd4fe1ea1bcefe5f015b9cb6af5bd89ec9d225eeffc39ca6ec1473c0964203",
|
|
13
|
+
"api/v2/asyncapi.yaml": "9ec0cac22964427c635e2fd836ca6d94397ccf14245b7c30f96b69d77033330a",
|
|
14
|
+
"api/v2/openapi.yaml": "d9c5921a1800d2ffde4807e4652d97616aec5dd75c22ba70860c111e4e660e15",
|
|
15
15
|
"api/v2/redocly.yaml": "1e66b60e6118ad11a823bb620678be464d99dfe50a40e3e6f93ec9429b88b34c",
|
|
16
16
|
"schemas/README.md": "276fc1d51eb08e14e9127b9fd27761d7046ee33c12000f3a64a801322db8bf58",
|
|
17
17
|
"schemas/a2a-task-state.schema.json": "75d5049dea8bd873ff0e7546f1c60c8d36c219bec7084264360be54a8be30ae1",
|
|
@@ -115,7 +115,7 @@
|
|
|
115
115
|
"schemas/v2/artifact.schema.json": "85b9dc284d39b99d2a66fa8cbf47f2dd3d8526d3bd6cc411ad315b372de38975",
|
|
116
116
|
"schemas/v2/audit-verify-result.schema.json": "14af8e9283510f0a40e48a2059c929ee44e0fc9fa3cc66a95bfbe90b17c48297",
|
|
117
117
|
"schemas/v2/budget-policy.schema.json": "1de68095db0beb5c7a4be629de26e79f62ebbe748a88d46f36e768edfd1f0d68",
|
|
118
|
-
"schemas/v2/capabilities.schema.json": "
|
|
118
|
+
"schemas/v2/capabilities.schema.json": "f4b4fb4c32144026e97a1be4266cc7a87d349ac714dcac156e9d8512740d06a1",
|
|
119
119
|
"schemas/v2/certification-bundle.schema.json": "f9e233dc3584c3cef78809d518749ce37cd1ee79ebbfe58163dc67eeac7802b1",
|
|
120
120
|
"schemas/v2/channel-presence-payload.schema.json": "fc03df0c20cc12a25f7ee6a47503a4c5931d79fe3d5ff24815e8fd7c1bcb9a87",
|
|
121
121
|
"schemas/v2/channel-written-payload.schema.json": "ccecff3c71a3275ad8db035ac0e04db25e6f8d23cf091ddbcd5a60d1be1cbb6b",
|
|
@@ -203,7 +203,7 @@
|
|
|
203
203
|
"schemas/workspace-file.schema.json": "464de85c2a068243084ee9c1d969bc7cd5d8f7948574e58450d6493c38a0e1e4",
|
|
204
204
|
"spec/v1/alias-detectors.json": "202a443678e592a515e33aad7f6ab679a07d8408809f2d4b0f59d156c84ffc65",
|
|
205
205
|
"spec/v1/capability-declaration-classes.json": "e7729aed5c4b4e1dd02abab0530f14cc95f5d4070fe51fb139e7f5cccefa00c6",
|
|
206
|
-
"spec/v1/core-standard-manifest.json": "
|
|
206
|
+
"spec/v1/core-standard-manifest.json": "e6bbda0be8241175de9d93dc531c727184a02f5043b0a61a12bdfc6de4305451",
|
|
207
207
|
"spec/v1/deprecations.json": "901fdab0a7212627f1838cbb06ae7b7204f0dff528299e56d3d938deecea27c1",
|
|
208
208
|
"spec/v1/deprecations.schema.json": "3e393c405d2a41b467d8c5e3c468549078df1ce6a6d2588fc488097b95d9b55b",
|
|
209
209
|
"spec/v1/end-of-support.json": "35f0bf3b70fe78b16e2a2fd81a153fc27e703f6c16376c6fe7cdf1f4f2c4e6d4",
|
|
@@ -250,7 +250,7 @@
|
|
|
250
250
|
"spec/v2/core/workflow-chain-packs.md": "fdce7a8477e269950a2bac05ee6eddbd66eb0b670b464d00054f7196ec05d29c",
|
|
251
251
|
"spec/v2/corrections.json": "5ee76d1ee87677a86d5d9ee2f477b88fc29f9f9d37611040b9ccb08f8f30fb2d",
|
|
252
252
|
"spec/v2/corrections.schema.json": "02f6a0a9531e657693be311af0f59d6432f3efcf645f56ccbf39e5b141e6b641",
|
|
253
|
-
"spec/v2/declaration.json": "
|
|
253
|
+
"spec/v2/declaration.json": "d8971c070358eb9ff6a94fdfb0ebc5638fd8f61b7d1d9cb816424bf1e70b8079",
|
|
254
254
|
"spec/v2/declaration.schema.json": "ef951633e4070899f64828f5cf1b5f4870270a0360f81dbb49b5b4d6cd407550",
|
|
255
255
|
"spec/v2/eos-override.json": "4c72d2f6a658943b01c1de1643cf294642091933af7832f38ee5495df684cb2d",
|
|
256
256
|
"spec/v2/errors.json": "8b4d2ebab569b26f608323ca4c9adaf805ab515286d26532db23fcb97541787f",
|
|
@@ -309,9 +309,9 @@
|
|
|
309
309
|
"spec/v2/path-manifest.json": "3720e502614f301efc7556e0bfcbfa7b7ad8a37c579256ba6bca82417a2728f1",
|
|
310
310
|
"spec/v2/peer-dependency-aliases.json": "cbf62f551f9295b0ac7ba6cc0466abb153563354a57b0622f3ec6e6689cba47e",
|
|
311
311
|
"spec/v2/profiles.json": "b382b261c7844798575f7c1b37c934f0da19846016a2337d883d4f5ef6e05479",
|
|
312
|
-
"spec/v2/release.json": "
|
|
312
|
+
"spec/v2/release.json": "2d7d294978a1a23726e9608da583d81645e447a4315fcad45c990caed1a058a1",
|
|
313
313
|
"spec/v2/retention-floors.json": "eaf3722d95c79947af1d4269ef85117e126518c588cfcf1a2b21b97269f51624",
|
|
314
|
-
"spec/v2/surface-baseline.json": "
|
|
314
|
+
"spec/v2/surface-baseline.json": "735c19442cecd130148546a302c6eb8f838f83936b7e329e1988f2b12a547a9d"
|
|
315
315
|
},
|
|
316
|
-
"corpusCommit": "
|
|
316
|
+
"corpusCommit": "cc71a07af3655fd18a498211819a96c4e3b1130d"
|
|
317
317
|
}
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The budget ACCOUNTING witness: what the numbers on a budgeted run's log must
|
|
3
|
+
* say, beyond the events existing (`budget-witness.ts` owns that).
|
|
4
|
+
*
|
|
5
|
+
* It reads the same observation `budget-witness.ts` `drive` returns — a run of
|
|
6
|
+
* `conformance-budget-tool-calls` under `{ maxToolCalls: 2, thresholdPercent:
|
|
7
|
+
* 50, onExhaustion: "fail" }` — and judges three rules of `runs.md` §`budget`
|
|
8
|
+
* section and `schemas/v2/budget-policy.schema.json`. Pure: no I/O.
|
|
9
|
+
*
|
|
10
|
+
* derived consumption is derived from `agent.tool-called`, "never
|
|
11
|
+
* measured twice": every `toolCalls` `budget.consumed`,
|
|
12
|
+
* `budget.threshold-crossed` and `budget.exhausted` carries a
|
|
13
|
+
* `consumed` equal to the number of `agent.tool-called` events
|
|
14
|
+
* before it in the log. A host that counts a call twice, or
|
|
15
|
+
* counts `agent.tool-returned` as well, reports twice the calls.
|
|
16
|
+
* limit `budget.reserved` records the effective budget: its
|
|
17
|
+
* `effectiveBudget.maxToolCalls` is at most the run's own
|
|
18
|
+
* ceiling (the effective budget is the minimum across scopes),
|
|
19
|
+
* and every `toolCalls` accounting event's `limit` equals it. A
|
|
20
|
+
* host that enforces a number it did not record breaks replay,
|
|
21
|
+
* which re-reads `budget.reserved` and never re-resolves.
|
|
22
|
+
* threshold `budget.threshold-crossed` is emitted once for the dimension,
|
|
23
|
+
* at the policy's percent (`percent` equals `thresholdPercent`)
|
|
24
|
+
* and not before consumption reached it.
|
|
25
|
+
*
|
|
26
|
+
* An empty finding list means the log offers nothing to judge for that rule;
|
|
27
|
+
* the scenario records that as `blocked`, never as a pass.
|
|
28
|
+
*/
|
|
29
|
+
|
|
30
|
+
import type { BudgetObservation, RunEvent } from './budget-witness.js';
|
|
31
|
+
|
|
32
|
+
export const ACCOUNTING_DIMENSION = 'toolCalls';
|
|
33
|
+
|
|
34
|
+
export interface AccountingNames {
|
|
35
|
+
readonly toolCalled: string;
|
|
36
|
+
readonly consumed: string;
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
export interface AccountingPolicy {
|
|
40
|
+
readonly maxToolCalls: number;
|
|
41
|
+
readonly thresholdPercent: number;
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
export interface AccountingFinding {
|
|
45
|
+
readonly rule: 'derived' | 'limit' | 'threshold';
|
|
46
|
+
readonly ok: boolean;
|
|
47
|
+
readonly doc: string;
|
|
48
|
+
readonly message: string;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
const num = (v: unknown): number | undefined => (typeof v === 'number' && Number.isFinite(v) ? v : undefined);
|
|
52
|
+
|
|
53
|
+
/** The `toolCalls` accounting events: consumed, threshold-crossed, exhausted. */
|
|
54
|
+
function accountingEvents(o: BudgetObservation, names: AccountingNames): RunEvent[] {
|
|
55
|
+
const types = new Set([names.consumed, o.names.threshold, o.names.exhausted]);
|
|
56
|
+
return o.events.filter((e) => types.has(e.type) && e.payload['dimension'] === ACCOUNTING_DIMENSION);
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
/** Judge the accounting rules on one observed budgeted run. Pure. */
|
|
60
|
+
export function judgeAccounting(o: BudgetObservation, names: AccountingNames, policy: AccountingPolicy, home: string): AccountingFinding[] {
|
|
61
|
+
const out: AccountingFinding[] = [];
|
|
62
|
+
const calls = o.events.filter((e) => e.type === names.toolCalled).map((e) => e.sequence);
|
|
63
|
+
const accounting = accountingEvents(o, names);
|
|
64
|
+
|
|
65
|
+
// derived — judged only when the log carries calls and accounting to compare.
|
|
66
|
+
if (calls.length > 0) {
|
|
67
|
+
for (const e of accounting) {
|
|
68
|
+
const consumed = num(e.payload['consumed']);
|
|
69
|
+
const before = calls.filter((s) => s < e.sequence).length;
|
|
70
|
+
out.push({
|
|
71
|
+
rule: 'derived',
|
|
72
|
+
ok: consumed === before,
|
|
73
|
+
doc: home,
|
|
74
|
+
message: `consumption is derived from ${names.toolCalled} and never measured twice: ${e.type} at sequence ${e.sequence} MUST report consumed ${before} (the ${names.toolCalled} events before it), got ${String(e.payload['consumed'])}`,
|
|
75
|
+
});
|
|
76
|
+
}
|
|
77
|
+
}
|
|
78
|
+
|
|
79
|
+
// limit — the recorded effective budget is the one every accounting event names.
|
|
80
|
+
const reserved = o.events.find((e) => e.type === o.names.reserved);
|
|
81
|
+
if (reserved !== undefined) {
|
|
82
|
+
const eff = reserved.payload['effectiveBudget'];
|
|
83
|
+
const recorded = eff !== null && typeof eff === 'object' && !Array.isArray(eff) ? num((eff as Record<string, unknown>)['maxToolCalls']) : undefined;
|
|
84
|
+
out.push({
|
|
85
|
+
rule: 'limit',
|
|
86
|
+
ok: recorded !== undefined && Number.isInteger(recorded) && recorded <= policy.maxToolCalls,
|
|
87
|
+
doc: home,
|
|
88
|
+
message: `${o.names.reserved} MUST record the effective budget: effectiveBudget.maxToolCalls is the minimum across scopes, so at most the run's own ${policy.maxToolCalls} (got ${String(recorded)})`,
|
|
89
|
+
});
|
|
90
|
+
if (recorded !== undefined) {
|
|
91
|
+
for (const e of accounting) {
|
|
92
|
+
out.push({
|
|
93
|
+
rule: 'limit',
|
|
94
|
+
ok: num(e.payload['limit']) === recorded,
|
|
95
|
+
doc: home,
|
|
96
|
+
message: `${e.type} at sequence ${e.sequence} MUST name the limit ${o.names.reserved} recorded (${recorded}), got ${String(e.payload['limit'])} — a replay re-reads the reserved budget and never re-resolves it`,
|
|
97
|
+
});
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
// threshold — once, at the policy's percent, not early.
|
|
103
|
+
const crossings = o.events.filter((e) => e.type === o.names.threshold && e.payload['dimension'] === ACCOUNTING_DIMENSION);
|
|
104
|
+
if (crossings.length > 0) {
|
|
105
|
+
out.push({
|
|
106
|
+
rule: 'threshold',
|
|
107
|
+
ok: crossings.length === 1,
|
|
108
|
+
doc: 'schemas/v2/budget-policy.schema.json thresholdPercent',
|
|
109
|
+
message: `${o.names.threshold} MUST be emitted once per dimension (got ${crossings.length} for ${ACCOUNTING_DIMENSION} at sequences ${crossings.map((e) => e.sequence).join(', ')})`,
|
|
110
|
+
});
|
|
111
|
+
const first = crossings[0] as RunEvent;
|
|
112
|
+
const percent = num(first.payload['percent']);
|
|
113
|
+
const consumed = num(first.payload['consumed']);
|
|
114
|
+
const limit = num(first.payload['limit']);
|
|
115
|
+
out.push({
|
|
116
|
+
rule: 'threshold',
|
|
117
|
+
ok: percent === policy.thresholdPercent,
|
|
118
|
+
doc: home,
|
|
119
|
+
message: `${o.names.threshold}.percent MUST be the percent-of-limit threshold the policy set (${policy.thresholdPercent}), got ${String(first.payload['percent'])}`,
|
|
120
|
+
});
|
|
121
|
+
out.push({
|
|
122
|
+
rule: 'threshold',
|
|
123
|
+
ok: consumed !== undefined && limit !== undefined && limit > 0 && (consumed / limit) * 100 >= policy.thresholdPercent,
|
|
124
|
+
doc: 'schemas/v2/budget-policy.schema.json thresholdPercent',
|
|
125
|
+
message: `${o.names.threshold} MUST NOT precede the threshold: emitted at consumed ${String(first.payload['consumed'])} of limit ${String(first.payload['limit'])}, below ${policy.thresholdPercent}%`,
|
|
126
|
+
});
|
|
127
|
+
}
|
|
128
|
+
return out;
|
|
129
|
+
}
|
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* v2 — a run budget's numbers are derived, recorded and crossed once
|
|
3
|
+
* (`spec/v2/core/runs.md` §`budget` section;
|
|
4
|
+
* `schemas/v2/budget-policy.schema.json` `thresholdPercent`). Gated on
|
|
5
|
+
* `budget`. Unaided.
|
|
6
|
+
*
|
|
7
|
+
* `v2-budget-enforcement` witnesses that the budget events exist, in order,
|
|
8
|
+
* and that a hard budget stops the run. This file reads the same kind of run
|
|
9
|
+
* for what those events SAY — the accounting a replay depends on:
|
|
10
|
+
*
|
|
11
|
+
* derived every `toolCalls` `budget.consumed`, `budget.threshold-crossed`
|
|
12
|
+
* and `budget.exhausted` reports `consumed` equal to the
|
|
13
|
+
* `agent.tool-called` events before it ("derived from …
|
|
14
|
+
* `agent.toolCalled` …, never measured twice").
|
|
15
|
+
* limit `budget.reserved` records the effective budget, at most the
|
|
16
|
+
* run's own ceiling, and every accounting event names that limit.
|
|
17
|
+
* threshold `budget.threshold-crossed` is emitted once for the dimension, at
|
|
18
|
+
* the policy's `thresholdPercent`, and not before spend reached it.
|
|
19
|
+
*
|
|
20
|
+
* The run is `conformance-budget-tool-calls` (three scripted tool calls, no
|
|
21
|
+
* model) under `{ maxToolCalls: 2, thresholdPercent: 50, onExhaustion: "fail" }`
|
|
22
|
+
* on `configurable.budget`, driven by `lib/budget-witness.ts` `drive`. The
|
|
23
|
+
* judge is `lib/budget-accounting-witness.ts`, proven both ways in its
|
|
24
|
+
* self-test.
|
|
25
|
+
*
|
|
26
|
+
* Dispositions: no `budget` ⇒ `inapplicable` (gate `openwop.family.budget`);
|
|
27
|
+
* `toolCalls` not in `budget.dimensions`, or the fixture unadvertised ⇒
|
|
28
|
+
* `inapplicable` (the fixture is the opt-in, as in `v2-budget-enforcement`); a
|
|
29
|
+
* log with nothing to judge for a rule ⇒ `blocked`. A valid budgeted create
|
|
30
|
+
* that is refused fails here, as it does there.
|
|
31
|
+
*
|
|
32
|
+
* @see spec/v2/core/runs.md §budget section
|
|
33
|
+
* @see schemas/v2/budget-policy.schema.json
|
|
34
|
+
* @see conformance/fixtures.md §conformance-budget-tool-calls
|
|
35
|
+
*/
|
|
36
|
+
|
|
37
|
+
import { describe, it, expect } from 'vitest';
|
|
38
|
+
import { gateFamily, v2Discovery } from '../lib/v2.js';
|
|
39
|
+
import { softSkip } from '../lib/soft-skip.js';
|
|
40
|
+
import { req } from '../lib/requirement-ids.js';
|
|
41
|
+
import { majorProfile } from '../lib/major-profile.js';
|
|
42
|
+
import { BUDGET_POLICY, drive, type BudgetRun } from '../lib/budget-witness.js';
|
|
43
|
+
import { judgeAccounting, type AccountingFinding } from '../lib/budget-accounting-witness.js';
|
|
44
|
+
|
|
45
|
+
const PROFILE = majorProfile(2);
|
|
46
|
+
const DOC = 'spec/v2/core/runs.md §budget section';
|
|
47
|
+
const ID_DERIVED = 'openwop.requirement.0084.consumption-derived';
|
|
48
|
+
const ID_LIMIT = 'openwop.requirement.0084.reserved-limit-enforced';
|
|
49
|
+
const ID_THRESHOLD = 'openwop.requirement.0084.threshold-once';
|
|
50
|
+
|
|
51
|
+
/** One budgeted run per file: three legs read the same log. */
|
|
52
|
+
let once: Promise<BudgetRun> | undefined;
|
|
53
|
+
function budgetRun(): Promise<BudgetRun> {
|
|
54
|
+
once ??= (async (): Promise<BudgetRun> => {
|
|
55
|
+
let doc: Record<string, unknown> | null;
|
|
56
|
+
try { doc = await v2Discovery(); } catch { doc = null; }
|
|
57
|
+
if (!doc) return { kind: 'skip', disposition: 'blocked', reason: 'v2 discovery unreachable — /.well-known/openwop did not answer 200 with a JSON body under OpenWOP-Version: 2.0' };
|
|
58
|
+
return drive(PROFILE, doc);
|
|
59
|
+
})();
|
|
60
|
+
return once;
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
type Leg = { readonly skip: readonly ['inapplicable' | 'blocked', string] } | { readonly findings: AccountingFinding[] };
|
|
64
|
+
|
|
65
|
+
/** The findings for one rule, or why there are none. A refused create fails the leg here. */
|
|
66
|
+
async function leg(id: string, rule: AccountingFinding['rule']): Promise<Leg> {
|
|
67
|
+
if ((await gateFamily('budget')) === null) return { skip: ['inapplicable', 'the host does not advertise budget (gate recorded under openwop.family.budget)'] };
|
|
68
|
+
const r = await budgetRun();
|
|
69
|
+
if (r.kind === 'skip') return { skip: [r.disposition, r.reason] };
|
|
70
|
+
if (r.kind === 'refused') {
|
|
71
|
+
expect(r.status, req(id, DOC, `a host that advertises budget and the fixture MUST accept a valid configurable.budget (got ${r.status} ${r.code ?? ''})`.trim())).toBe(201);
|
|
72
|
+
return { findings: [] };
|
|
73
|
+
}
|
|
74
|
+
const toolCalled = PROFILE.eventType('agent.toolCalled');
|
|
75
|
+
const consumed = PROFILE.eventType('budget.consumed');
|
|
76
|
+
if (toolCalled === undefined || consumed === undefined) return { skip: ['blocked', 'the v2 event names for agent.toolCalled / budget.consumed are not in the codemap on disk — the witness will not guess them'] };
|
|
77
|
+
const findings = judgeAccounting(r.observation, { toolCalled, consumed }, BUDGET_POLICY, DOC).filter((f) => f.rule === rule);
|
|
78
|
+
if (findings.length === 0) {
|
|
79
|
+
const what: Record<AccountingFinding['rule'], string> = {
|
|
80
|
+
derived: `the log carries no ${toolCalled} event or no toolCalls accounting event — there is no consumed figure to compare with the calls made`,
|
|
81
|
+
limit: 'the log carries no budget.reserved — there is no recorded effective budget to compare with',
|
|
82
|
+
threshold: 'the log carries no toolCalls budget.threshold-crossed — there is no crossing to judge (v2-budget-enforcement fails its absence)',
|
|
83
|
+
};
|
|
84
|
+
return { skip: ['blocked', what[rule]] };
|
|
85
|
+
}
|
|
86
|
+
return { findings };
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
describe('v2 budget accounting (runs.md §budget section)', () => {
|
|
90
|
+
it('every toolCalls accounting event reports consumed equal to the agent.tool-called events before it', async () => {
|
|
91
|
+
const l = await leg(ID_DERIVED, 'derived');
|
|
92
|
+
if ('skip' in l) return softSkip(...l.skip);
|
|
93
|
+
for (const f of l.findings) expect(f.ok, req(ID_DERIVED, f.doc, f.message)).toBe(true);
|
|
94
|
+
});
|
|
95
|
+
|
|
96
|
+
it('budget.reserved records an effective budget within the run ceiling, and every accounting event names that limit', async () => {
|
|
97
|
+
const l = await leg(ID_LIMIT, 'limit');
|
|
98
|
+
if ('skip' in l) return softSkip(...l.skip);
|
|
99
|
+
for (const f of l.findings) expect(f.ok, req(ID_LIMIT, f.doc, f.message)).toBe(true);
|
|
100
|
+
});
|
|
101
|
+
|
|
102
|
+
it('budget.threshold-crossed is emitted once, at the policy percent, and not before spend reached it', async () => {
|
|
103
|
+
const l = await leg(ID_THRESHOLD, 'threshold');
|
|
104
|
+
if ('skip' in l) return softSkip(...l.skip);
|
|
105
|
+
for (const f of l.findings) expect(f.ok, req(ID_THRESHOLD, f.doc, f.message)).toBe(true);
|
|
106
|
+
});
|
|
107
|
+
});
|
|
@@ -0,0 +1,237 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* v2 — what a host advertising `eventLog` owes on its log
|
|
3
|
+
* (`spec/v2/core/persistence.md` §The `eventLog` family). Gated on `eventLog`.
|
|
4
|
+
* Unaided.
|
|
5
|
+
*
|
|
6
|
+
* The family's three MUSTs, one leg each:
|
|
7
|
+
*
|
|
8
|
+
* stamp "MUST stamp `eventLogSchemaVersion` on every run it creates".
|
|
9
|
+
* Every creation path the suite can drive from outside: a run
|
|
10
|
+
* made by `createRun`, and, where `replay` is advertised, a run
|
|
11
|
+
* made by `forkRun`. Each snapshot carries the era discovery
|
|
12
|
+
* advertises, and that era is `3` (§The era key). The ungated
|
|
13
|
+
* `v2-era-stamp-universal` checks `createRun` only; a host whose
|
|
14
|
+
* fork path writes no era fails here.
|
|
15
|
+
* cursor "MUST serve the cursor contract of events.md §Poll over that
|
|
16
|
+
* log". A cursor at EVERY position of a settled log returns
|
|
17
|
+
* exactly the tail after it (a page-limited host may return a
|
|
18
|
+
* non-empty prefix of it); chaining `afterSequence` through the
|
|
19
|
+
* pages rebuilds the log with no gap and no duplicate and ends
|
|
20
|
+
* on an empty page; each event keeps its `eventId` across reads;
|
|
21
|
+
* and every response's `lastSequence` is the highest sequence in
|
|
22
|
+
* the log, a cursor past the end included. `v2-poll-cursor-v2`
|
|
23
|
+
* checks one position.
|
|
24
|
+
* codemap "MUST NOT emit an event `type` the codemap does not name".
|
|
25
|
+
* Every type on the logs of the advertised lifecycle fixtures
|
|
26
|
+
* (`conformance-noop`, and `conformance-multi-node` and
|
|
27
|
+
* `conformance-failure` where seeded) is a v2 name in
|
|
28
|
+
* `spec/v2/event-codemap.json`, or a vendor type whose org is
|
|
29
|
+
* registered in `spec/v2/declaration.json` `extensions`
|
|
30
|
+
* (§The codemap is data). The failure fixture reaches
|
|
31
|
+
* `run.failed` / `node.failed`, which `v2-event-type-closed`'s
|
|
32
|
+
* noop log never does.
|
|
33
|
+
*
|
|
34
|
+
* Dispositions: no `eventLog` ⇒ `inapplicable` (gate `openwop.family.eventLog`).
|
|
35
|
+
* A create refused, a run that does not settle, or a log that grows between
|
|
36
|
+
* reads ⇒ `blocked`. A fork that is declined is noted after the `createRun`
|
|
37
|
+
* path is judged, so that row is a partial-witness pass.
|
|
38
|
+
*
|
|
39
|
+
* @see spec/v2/core/persistence.md §The eventLog family
|
|
40
|
+
* @see spec/v2/core/persistence.md §The era key
|
|
41
|
+
* @see spec/v2/core/events.md §Poll
|
|
42
|
+
*/
|
|
43
|
+
|
|
44
|
+
import { describe, it, expect } from 'vitest';
|
|
45
|
+
import { existsSync, readFileSync } from 'node:fs';
|
|
46
|
+
import { join } from 'node:path';
|
|
47
|
+
import { driver, type OpenWOPResponse } from '../lib/driver.js';
|
|
48
|
+
import { familyAdvertised, gateFamily, v2Discovery } from '../lib/v2.js';
|
|
49
|
+
import { readErrorCode } from '../lib/error-envelope.js';
|
|
50
|
+
import { isFixtureAdvertised } from '../lib/fixtures.js';
|
|
51
|
+
import { registeredOrgs } from '../lib/era2-seed.js';
|
|
52
|
+
import { SPEC_V2_DIR } from '../lib/paths.js';
|
|
53
|
+
import { blockedDespiteAssertions, softSkip } from '../lib/soft-skip.js';
|
|
54
|
+
import { req } from '../lib/requirement-ids.js';
|
|
55
|
+
|
|
56
|
+
const ID_STAMP = 'openwop.requirement.0036.event-log-stamp-every-path';
|
|
57
|
+
const ID_CURSOR = 'openwop.requirement.0036.event-log-cursor-contract';
|
|
58
|
+
const ID_CODEMAP = 'openwop.requirement.0036.event-log-codemap-types';
|
|
59
|
+
const DOC_FAMILY = 'spec/v2/core/persistence.md §The eventLog family';
|
|
60
|
+
const DOC_ERA = 'spec/v2/core/persistence.md §The era key';
|
|
61
|
+
const DOC_POLL = 'spec/v2/core/events.md §Poll';
|
|
62
|
+
const DOC_CODEMAP = 'spec/v2/core/persistence.md §The codemap is data';
|
|
63
|
+
const NOOP = 'conformance-noop';
|
|
64
|
+
const MULTI = 'conformance-multi-node';
|
|
65
|
+
const FAILURE = 'conformance-failure';
|
|
66
|
+
const V2_ERA = 3;
|
|
67
|
+
const TERMINAL = new Set(['completed', 'failed', 'cancelled']);
|
|
68
|
+
const SETTLE_MS = Number(process.env['OPENWOP_LIFECYCLE_TIMEOUT_MS'] ?? 15_000);
|
|
69
|
+
const MAX_PAGES = 200;
|
|
70
|
+
|
|
71
|
+
interface Read { readonly sequence: number; readonly eventId: string; readonly type: string }
|
|
72
|
+
interface Page { readonly status: number; readonly events: readonly Read[]; readonly lastSequence: unknown }
|
|
73
|
+
|
|
74
|
+
const enc = encodeURIComponent;
|
|
75
|
+
const isRecord = (v: unknown): v is Record<string, unknown> => v !== null && typeof v === 'object' && !Array.isArray(v);
|
|
76
|
+
async function http(fn: () => Promise<OpenWOPResponse>): Promise<OpenWOPResponse | null> {
|
|
77
|
+
try { return await fn(); } catch { return null; }
|
|
78
|
+
}
|
|
79
|
+
async function discovery(): Promise<Record<string, unknown> | null> {
|
|
80
|
+
try { return await v2Discovery(); } catch { return null; }
|
|
81
|
+
}
|
|
82
|
+
|
|
83
|
+
/** Create a run of `workflowId` and wait for it to settle. */
|
|
84
|
+
async function settledRun(workflowId: string): Promise<{ runId: string; snapshot: Record<string, unknown> } | { reason: string }> {
|
|
85
|
+
const created = await http(() => driver.post('/runs', { workflowId }));
|
|
86
|
+
if (created === null) return { reason: 'POST /runs unreachable (fetch failed)' };
|
|
87
|
+
const runId = isRecord(created.json) ? created.json['runId'] : undefined;
|
|
88
|
+
if (created.status !== 201 || typeof runId !== 'string') return { reason: `POST /runs {workflowId: ${workflowId}} answered ${created.status} ${readErrorCode(created.json) ?? ''} — the create was refused (fixture not seeded?)`.trim() };
|
|
89
|
+
return settle(runId);
|
|
90
|
+
}
|
|
91
|
+
async function settle(runId: string): Promise<{ runId: string; snapshot: Record<string, unknown> } | { reason: string }> {
|
|
92
|
+
const deadline = Date.now() + SETTLE_MS;
|
|
93
|
+
let last: Record<string, unknown> | null = null;
|
|
94
|
+
while (Date.now() < deadline) {
|
|
95
|
+
const res = await http(() => driver.get(`/runs/${enc(runId)}`));
|
|
96
|
+
last = res !== null && res.status === 200 && isRecord(res.json) ? res.json : null;
|
|
97
|
+
if (last !== null && TERMINAL.has(String(last['status']))) return { runId, snapshot: last };
|
|
98
|
+
await new Promise((r) => setTimeout(r, 250));
|
|
99
|
+
}
|
|
100
|
+
return { reason: `run ${runId} did not reach a terminal status within ${SETTLE_MS} ms (last: ${last === null ? 'unreadable' : String(last['status'])})` };
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
async function poll(runId: string, after: number | null): Promise<Page | null> {
|
|
104
|
+
const res = await http(() => driver.get(`/runs/${enc(runId)}/events/poll?timeout=1${after === null ? '' : `&afterSequence=${after}`}`));
|
|
105
|
+
if (res === null) return null;
|
|
106
|
+
const body = isRecord(res.json) ? res.json : {};
|
|
107
|
+
const raw = Array.isArray(body['events']) ? body['events'] : [];
|
|
108
|
+
const events = raw.filter(isRecord).map((e): Read => ({ sequence: typeof e['sequence'] === 'number' ? e['sequence'] : Number.NaN, eventId: String(e['eventId']), type: String(e['type']) }));
|
|
109
|
+
return { status: res.status, events, lastSequence: body['lastSequence'] };
|
|
110
|
+
}
|
|
111
|
+
const seqs = (p: Page): number[] => p.events.map((e) => e.sequence);
|
|
112
|
+
|
|
113
|
+
/** The v2 names of `spec/v2/event-codemap.json`, or null when the file is not on disk in this layout. */
|
|
114
|
+
function codemapV2Names(): Set<string> | null {
|
|
115
|
+
const file = SPEC_V2_DIR ? join(SPEC_V2_DIR, 'event-codemap.json') : null;
|
|
116
|
+
if (file === null || !existsSync(file)) return null;
|
|
117
|
+
try {
|
|
118
|
+
const doc = JSON.parse(readFileSync(file, 'utf8')) as { rows?: Array<{ v2?: unknown }> };
|
|
119
|
+
const names = new Set((doc.rows ?? []).map((r) => r.v2).filter((v): v is string => typeof v === 'string'));
|
|
120
|
+
return names.size > 0 ? names : null;
|
|
121
|
+
} catch {
|
|
122
|
+
return null;
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
describe('v2 eventLog family (persistence.md §The eventLog family)', () => {
|
|
127
|
+
it('every run the host creates, by createRun and by forkRun, carries the advertised era 3', async () => {
|
|
128
|
+
if ((await gateFamily('eventLog')) === null) return softSkip('inapplicable', 'the host does not advertise eventLog (gate recorded under openwop.family.eventLog)');
|
|
129
|
+
const doc = await discovery();
|
|
130
|
+
if (!doc) return softSkip('blocked', 'v2 discovery unreachable');
|
|
131
|
+
const advertised = doc['eventLogSchemaVersion'];
|
|
132
|
+
expect(advertised, req(ID_STAMP, DOC_ERA, `a host advertising eventLog MUST advertise the era it writes for new runs, and a v2 host writes ${V2_ERA} (discovery eventLogSchemaVersion: ${JSON.stringify(advertised)})`)).toBe(V2_ERA);
|
|
133
|
+
|
|
134
|
+
const source = await settledRun(NOOP);
|
|
135
|
+
if ('reason' in source) return softSkip('blocked', source.reason);
|
|
136
|
+
expect(source.snapshot['eventLogSchemaVersion'], req(ID_STAMP, DOC_FAMILY, `a run made by createRun MUST carry eventLogSchemaVersion ${V2_ERA} on its snapshot (got ${JSON.stringify(source.snapshot['eventLogSchemaVersion'])})`)).toBe(V2_ERA);
|
|
137
|
+
|
|
138
|
+
// The second creation path. forkRun is gated on `replay` (runs.md §Surface).
|
|
139
|
+
const replay = await familyAdvertised('replay');
|
|
140
|
+
const modes = Array.isArray(replay?.['modes']) ? (replay?.['modes'] as unknown[]).filter((m): m is string => typeof m === 'string') : [];
|
|
141
|
+
// partial-witness-ok: forkRun is a creation path only where replay advertises a fork mode; createRun, the only other, was judged above.
|
|
142
|
+
if (replay === null || modes.length === 0) return softSkip('inapplicable', 'replay is not advertised with a fork mode — createRun is the only creation path the suite can drive, and it was judged above');
|
|
143
|
+
const mode = modes.includes('branch') ? 'branch' : (modes[0] as string);
|
|
144
|
+
const fork = await http(() => driver.post(`/runs/${enc(source.runId)}:fork`, { mode, fromSeq: 0 }));
|
|
145
|
+
const forkId = fork !== null && isRecord(fork.json) ? fork.json['runId'] : undefined;
|
|
146
|
+
if (fork === null || fork.status !== 201 || typeof forkId !== 'string') {
|
|
147
|
+
return softSkip('blocked', `a ${mode} fork at fromSeq 0 answered ${fork?.status ?? 'no response'} ${fork === null ? '' : readErrorCode(fork.json) ?? ''} — the forkRun creation path was not reached (its refusals are v2-run-fork-refusals' to judge)`.trim());
|
|
148
|
+
}
|
|
149
|
+
const forked = await http(() => driver.get(`/runs/${enc(forkId)}`));
|
|
150
|
+
if (forked === null || forked.status !== 200 || !isRecord(forked.json)) return softSkip('blocked', `GET /runs/{fork} answered ${forked?.status ?? 'no response'} — the fork's snapshot is where its era is reported`);
|
|
151
|
+
expect(forked.json['eventLogSchemaVersion'], req(ID_STAMP, DOC_FAMILY, `a run made by forkRun (${mode}) MUST carry eventLogSchemaVersion ${V2_ERA} too: a creation path that writes no era makes runs every reader translates as the v1 era (got ${JSON.stringify(forked.json['eventLogSchemaVersion'])})`)).toBe(V2_ERA);
|
|
152
|
+
});
|
|
153
|
+
|
|
154
|
+
it('a cursor at every position of a settled log serves the poll contract', async () => {
|
|
155
|
+
if ((await gateFamily('eventLog')) === null) return softSkip('inapplicable', 'the host does not advertise eventLog (gate recorded under openwop.family.eventLog)');
|
|
156
|
+
if (!(await discovery())) return softSkip('blocked', 'v2 discovery unreachable');
|
|
157
|
+
const fixture = isFixtureAdvertised(MULTI) ? MULTI : NOOP;
|
|
158
|
+
const run = await settledRun(fixture);
|
|
159
|
+
if ('reason' in run) return softSkip('blocked', run.reason);
|
|
160
|
+
|
|
161
|
+
// Observe everything first; judge only a log that stayed put while it was read.
|
|
162
|
+
const whole = await poll(run.runId, null);
|
|
163
|
+
if (whole === null || whole.status !== 200) return softSkip('blocked', `GET /runs/{runId}/events/poll answered ${whole?.status ?? 'no response'}`);
|
|
164
|
+
const all = seqs(whole);
|
|
165
|
+
if (all.length < 2) return softSkip('blocked', `the ${fixture} log holds ${all.length} event(s) — a cursor walk needs an interior position`);
|
|
166
|
+
const atCursor = new Map<number, Page | null>();
|
|
167
|
+
for (const k of all) atCursor.set(k, await poll(run.runId, k));
|
|
168
|
+
const walk: Page[] = [];
|
|
169
|
+
let cursor: number | null = null;
|
|
170
|
+
for (let i = 0; i < MAX_PAGES; i++) {
|
|
171
|
+
const p = await poll(run.runId, cursor);
|
|
172
|
+
if (p === null) break;
|
|
173
|
+
walk.push(p);
|
|
174
|
+
const s = seqs(p);
|
|
175
|
+
if (p.status !== 200 || s.length === 0 || !Number.isFinite(s[s.length - 1] as number)) break;
|
|
176
|
+
cursor = s[s.length - 1] as number;
|
|
177
|
+
}
|
|
178
|
+
const pastEnd = await poll(run.runId, Math.max(...all) + 1000);
|
|
179
|
+
const again = await poll(run.runId, null);
|
|
180
|
+
if (again === null || again.status !== 200 || seqs(again).join(',') !== all.join(',')) {
|
|
181
|
+
return softSkip('blocked', `the log changed between reads (before [${all.join(',')}], after [${again === null ? 'unreadable' : seqs(again).join(',')}]) — a settled run is still appending, so reads of it are not comparable`);
|
|
182
|
+
}
|
|
183
|
+
|
|
184
|
+
const max = Math.max(...all);
|
|
185
|
+
const idAt = new Map<number, string>(whole.events.map((e): [number, string] => [e.sequence, e.eventId]));
|
|
186
|
+
const tail = (k: number): number[] => all.filter((s) => s > k);
|
|
187
|
+
for (const [k, p] of atCursor) {
|
|
188
|
+
expect(p?.status, req(ID_CURSOR, DOC_POLL, `afterSequence=${k} MUST answer 200`)).toBe(200);
|
|
189
|
+
const got = p === null ? [] : seqs(p);
|
|
190
|
+
const want = tail(k);
|
|
191
|
+
const prefix = got.length <= want.length && got.every((s, i) => s === want[i]);
|
|
192
|
+
expect(prefix && (want.length === 0 ? got.length === 0 : got.length > 0), req(ID_CURSOR, DOC_POLL, `afterSequence=${k} MUST return the events with sequence > ${k} in order — [${want.join(', ')}], or a non-empty prefix of it; got [${got.join(', ')}]`)).toBe(true);
|
|
193
|
+
expect(p?.lastSequence, req(ID_CURSOR, DOC_POLL, `lastSequence is the highest sequence in the log (${max}) whatever the cursor; afterSequence=${k} answered ${JSON.stringify(p?.lastSequence)}`)).toBe(max);
|
|
194
|
+
for (const e of p?.events ?? []) {
|
|
195
|
+
expect(e.eventId, req(ID_CURSOR, DOC_POLL, `the event at sequence ${e.sequence} MUST keep its eventId across reads — the log is not renumbered (whole read: ${idAt.get(e.sequence) ?? 'absent'}, cursor read: ${e.eventId})`)).toBe(idAt.get(e.sequence));
|
|
196
|
+
}
|
|
197
|
+
}
|
|
198
|
+
const walked = walk.flatMap(seqs);
|
|
199
|
+
expect(walked, req(ID_CURSOR, DOC_POLL, `chaining afterSequence through the pages MUST rebuild the log with no gap and no duplicate: log [${all.join(', ')}], walk [${walked.join(', ')}] over ${walk.length} page(s)`)).toEqual(all);
|
|
200
|
+
const lastPage = walk[walk.length - 1];
|
|
201
|
+
expect(lastPage !== undefined && lastPage.status === 200 && lastPage.events.length === 0, req(ID_CURSOR, DOC_POLL, `the walk MUST end on a 200 with an empty events array once the cursor reaches the end (last page: ${lastPage === undefined ? 'none' : `${lastPage.status}, ${lastPage.events.length} event(s)`}; ${walk.length} page(s), cap ${MAX_PAGES})`)).toBe(true);
|
|
202
|
+
expect(pastEnd?.status, req(ID_CURSOR, DOC_POLL, 'a cursor past the end of the log MUST answer 200')).toBe(200);
|
|
203
|
+
expect(pastEnd?.lastSequence, req(ID_CURSOR, DOC_POLL, `a cursor past the end returns no events, yet lastSequence is still the highest sequence in the log (${max}), not a figure computed from the empty page (got ${JSON.stringify(pastEnd?.lastSequence)})`)).toBe(max);
|
|
204
|
+
});
|
|
205
|
+
|
|
206
|
+
it('every event type on the lifecycle fixtures\' logs is a codemap name or a registered vendor type', async () => {
|
|
207
|
+
if ((await gateFamily('eventLog')) === null) return softSkip('inapplicable', 'the host does not advertise eventLog (gate recorded under openwop.family.eventLog)');
|
|
208
|
+
if (!(await discovery())) return softSkip('blocked', 'v2 discovery unreachable');
|
|
209
|
+
const names = codemapV2Names();
|
|
210
|
+
if (names === null) return softSkip('blocked', 'spec/v2/event-codemap.json is not on disk in this layout — the codemap is the authority this rule names, and the suite will not substitute a list of its own');
|
|
211
|
+
const orgs = registeredOrgs();
|
|
212
|
+
|
|
213
|
+
const fixtures = [NOOP, ...[MULTI, FAILURE].filter((f) => isFixtureAdvertised(f))];
|
|
214
|
+
const logs: Array<{ fixture: string; types: string[] }> = [];
|
|
215
|
+
for (const fixture of fixtures) {
|
|
216
|
+
const run = await settledRun(fixture);
|
|
217
|
+
if ('reason' in run) {
|
|
218
|
+
if (logs.length === 0) return softSkip('blocked', run.reason);
|
|
219
|
+
continue;
|
|
220
|
+
}
|
|
221
|
+
const page = await poll(run.runId, null);
|
|
222
|
+
if (page === null || page.status !== 200 || page.events.length === 0) continue;
|
|
223
|
+
logs.push({ fixture, types: page.events.map((e) => e.type) });
|
|
224
|
+
}
|
|
225
|
+
if (logs.length === 0) return softSkip('blocked', `no log of ${fixtures.join(', ')} could be read through the poll`);
|
|
226
|
+
|
|
227
|
+
for (const { fixture, types } of logs) {
|
|
228
|
+
for (const type of new Set(types)) {
|
|
229
|
+
const named = names.has(type);
|
|
230
|
+
if (!named && orgs === undefined) return blockedDespiteAssertions(`${fixture} emitted ${type}, which the codemap does not name, and spec/v2/declaration.json is not on disk to say whether its org is registered`);
|
|
231
|
+
const org = type.split('.')[0] ?? '';
|
|
232
|
+
const vendor = !named && org !== 'openwop' && orgs !== undefined && orgs.has(org);
|
|
233
|
+
expect(named || vendor, req(ID_CODEMAP, DOC_CODEMAP, `a host advertising eventLog MUST NOT emit a type the codemap does not name: ${fixture} emitted ${type}, which is not a v2 name in spec/v2/event-codemap.json, and its org "${org}" is not registered in spec/v2/declaration.json extensions`)).toBe(true);
|
|
234
|
+
}
|
|
235
|
+
}
|
|
236
|
+
});
|
|
237
|
+
});
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* v2 — the `production` bar beyond backpressure
|
|
3
|
+
* (`spec/v2/core/conformance.md` §Production profile). Gated on `production`.
|
|
4
|
+
*
|
|
5
|
+
* `v2-production-backpressure` witnesses the `backpressure` facet. This file
|
|
6
|
+
* takes the two other rules a suite can cause:
|
|
7
|
+
*
|
|
8
|
+
* idempotency "MUST tolerate at least five retries of one `Idempotency-Key`
|
|
9
|
+
* within its retention". Unaided: one `createRun` under a fresh
|
|
10
|
+
* key, then five sequential retries with the same key and the
|
|
11
|
+
* same bytes. Every retry answers the first response's status
|
|
12
|
+
* with the same `runId` and `OpenWOP-Idempotent-Replay: true`
|
|
13
|
+
* (`idempotency.md` §Outcomes, `runs.md` §Request headers). A
|
|
14
|
+
* host that evicts after a retry count, answers a later retry
|
|
15
|
+
* `409`, or mints a second run fails. Within retention: the
|
|
16
|
+
* retries land seconds after the first, and a record MUST be
|
|
17
|
+
* kept 24 hours.
|
|
18
|
+
* retention `retention` facet: "An expired run MUST answer `404
|
|
19
|
+
* not_found` or `410 run_expired`". Expiry takes at least seven
|
|
20
|
+
* days on a public host, so this leg is operator-aided, as at
|
|
21
|
+
* v1 (`production-retention-expiry`): `OPENWOP_TEST_EXPIRED_RUN_ID`
|
|
22
|
+
* names a run already expired, or — where the host advertises
|
|
23
|
+
* `retention.testForceExpire: true` —
|
|
24
|
+
* `OPENWOP_TEST_FORCE_EXPIRE_URL` (`{runId}` substituted, method
|
|
25
|
+
* `OPENWOP_TEST_FORCE_EXPIRE_METHOD`, default `POST`) expires a
|
|
26
|
+
* run the suite just created.
|
|
27
|
+
*
|
|
28
|
+
* Not here, and why: `debugBundle` has no v2 read (RFC 0207 §C.11 — no
|
|
29
|
+
* operation in `api/v2/openapi.yaml`), so its redaction and truncation rules
|
|
30
|
+
* have no wire to witness at major 2. Persistence across restart needs a
|
|
31
|
+
* process kill (RFC 0158's seam), and the logging rule is not observable.
|
|
32
|
+
*
|
|
33
|
+
* Dispositions: no `production` ⇒ `inapplicable` (gate
|
|
34
|
+
* `openwop.family.production`). No `retention` facet, or a host that offers no
|
|
35
|
+
* expiry seam and an operator who named no expired run ⇒ `inapplicable` (the
|
|
36
|
+
* facet is the opt-in, as a fixture is). `testForceExpire: true` with no URL
|
|
37
|
+
* supplied, a create refused, or a force-expire refused ⇒ `blocked`.
|
|
38
|
+
*
|
|
39
|
+
* @see spec/v2/core/conformance.md §Production profile
|
|
40
|
+
* @see spec/v2/core/idempotency.md §Outcomes
|
|
41
|
+
* @see conformance/fixtures.md §conformance-idempotent
|
|
42
|
+
*/
|
|
43
|
+
|
|
44
|
+
import { describe, it, expect } from 'vitest';
|
|
45
|
+
import { randomUUID } from 'node:crypto';
|
|
46
|
+
import { driver, type OpenWOPResponse } from '../lib/driver.js';
|
|
47
|
+
import { gateFamily } from '../lib/v2.js';
|
|
48
|
+
import { readErrorCode } from '../lib/error-envelope.js';
|
|
49
|
+
import { isFixtureAdvertised } from '../lib/fixtures.js';
|
|
50
|
+
import { softSkip } from '../lib/soft-skip.js';
|
|
51
|
+
import { req } from '../lib/requirement-ids.js';
|
|
52
|
+
|
|
53
|
+
const ID_RETRIES = 'openwop.requirement.0009.idempotency-five-retries';
|
|
54
|
+
const ID_EXPIRED = 'openwop.requirement.0009.retention-expired-run';
|
|
55
|
+
const DOC = 'spec/v2/core/conformance.md §Production profile';
|
|
56
|
+
const IDEMPOTENT = 'conformance-idempotent';
|
|
57
|
+
const NOOP = 'conformance-noop';
|
|
58
|
+
const RETRIES = 5;
|
|
59
|
+
|
|
60
|
+
const isRecord = (v: unknown): v is Record<string, unknown> => v !== null && typeof v === 'object' && !Array.isArray(v);
|
|
61
|
+
async function http(fn: () => Promise<OpenWOPResponse>): Promise<OpenWOPResponse | null> {
|
|
62
|
+
try { return await fn(); } catch { return null; }
|
|
63
|
+
}
|
|
64
|
+
const runIdOf = (r: OpenWOPResponse): unknown => (isRecord(r.json) ? r.json['runId'] : undefined);
|
|
65
|
+
|
|
66
|
+
describe('v2 production profile (conformance.md §Production profile)', () => {
|
|
67
|
+
it('a host claiming production replays one Idempotency-Key across five retries without a second run', async () => {
|
|
68
|
+
if ((await gateFamily('production')) === null) return softSkip('inapplicable', 'the host does not advertise production (gate recorded under openwop.family.production)');
|
|
69
|
+
// conformance-idempotent is the fixture made for this (fixtures.md); noop is the fallback every host seeds.
|
|
70
|
+
const body = isFixtureAdvertised(IDEMPOTENT) ? { workflowId: IDEMPOTENT, inputs: { nonce: randomUUID() } } : { workflowId: NOOP };
|
|
71
|
+
const headers = { 'Idempotency-Key': randomUUID() };
|
|
72
|
+
|
|
73
|
+
const first = await http(() => driver.post('/runs', body, { headers }));
|
|
74
|
+
if (first === null) return softSkip('blocked', 'POST /runs unreachable (fetch failed)');
|
|
75
|
+
if (first.status === 429 || first.status >= 500) return softSkip('blocked', `the first create answered ${first.status} — a retryable outcome is never cached, so there is no record to retry against`);
|
|
76
|
+
const runId = runIdOf(first);
|
|
77
|
+
if (first.status !== 201 || typeof runId !== 'string') return softSkip('blocked', `POST /runs {workflowId: ${body.workflowId}} answered ${first.status} ${readErrorCode(first.json) ?? ''} — the create was refused (fixture not seeded?)`.trim());
|
|
78
|
+
|
|
79
|
+
for (let i = 1; i <= RETRIES; i++) {
|
|
80
|
+
const retry = await http(() => driver.post('/runs', body, { headers }));
|
|
81
|
+
expect(retry, req(ID_RETRIES, DOC, `retry ${i} of ${RETRIES} under one Idempotency-Key MUST be answered (fetch failed)`)).not.toBeNull();
|
|
82
|
+
if (retry === null) continue;
|
|
83
|
+
expect(retry.status, req(ID_RETRIES, 'spec/v2/core/idempotency.md §Outcomes', `retry ${i} of ${RETRIES} MUST return the cached response — status ${first.status} — not ${retry.status} ${readErrorCode(retry.json) ?? ''}`.trim())).toBe(first.status);
|
|
84
|
+
expect(runIdOf(retry), req(ID_RETRIES, 'spec/v2/core/runs.md §Request headers', `retry ${i} of ${RETRIES}: a replayed create MUST NOT create a second run — the cached body names ${runId}`)).toBe(runId);
|
|
85
|
+
expect(retry.headers.get('openwop-idempotent-replay'), req(ID_RETRIES, 'spec/v2/core/idempotency.md §Outcomes', `retry ${i} of ${RETRIES} is served from cache and MUST carry OpenWOP-Idempotent-Replay: true`)).toBe('true');
|
|
86
|
+
}
|
|
87
|
+
});
|
|
88
|
+
|
|
89
|
+
it('an expired run answers 404 not_found or 410 run_expired', async () => {
|
|
90
|
+
const production = await gateFamily('production');
|
|
91
|
+
if (production === null) return softSkip('inapplicable', 'the host does not advertise production (gate recorded under openwop.family.production)');
|
|
92
|
+
const retention = production['retention'];
|
|
93
|
+
if (!isRecord(retention)) return softSkip('inapplicable', 'the host does not advertise production.retention — the expiry rule binds the retention facet');
|
|
94
|
+
|
|
95
|
+
let expired = process.env['OPENWOP_TEST_EXPIRED_RUN_ID'];
|
|
96
|
+
if (expired === undefined) {
|
|
97
|
+
if (retention['testForceExpire'] !== true) return softSkip('inapplicable', 'production.retention.testForceExpire is not true and OPENWOP_TEST_EXPIRED_RUN_ID is unset — the host offers no expiry seam and expiry takes at least 7 days, so the rule is not reachable from outside');
|
|
98
|
+
const url = process.env['OPENWOP_TEST_FORCE_EXPIRE_URL'];
|
|
99
|
+
if (url === undefined) return softSkip('blocked', 'the host advertises retention.testForceExpire, but OPENWOP_TEST_FORCE_EXPIRE_URL is unset — the operator has not said where the hook is');
|
|
100
|
+
const created = await http(() => driver.post('/runs', { workflowId: NOOP }));
|
|
101
|
+
const runId = created === null ? undefined : runIdOf(created);
|
|
102
|
+
if (created === null || created.status !== 201 || typeof runId !== 'string') return softSkip('blocked', `POST /runs answered ${created?.status ?? 'no response'} — there is no run to expire`);
|
|
103
|
+
const forced = await http(() => driver.request(process.env['OPENWOP_TEST_FORCE_EXPIRE_METHOD'] ?? 'POST', url.replace('{runId}', encodeURIComponent(runId))));
|
|
104
|
+
if (forced === null || forced.status < 200 || forced.status >= 300) return softSkip('blocked', `the force-expire hook answered ${forced?.status ?? 'no response'} — the run was not expired`);
|
|
105
|
+
expired = runId;
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
const target: string = expired;
|
|
109
|
+
const res = await http(() => driver.get(`/runs/${encodeURIComponent(target)}`));
|
|
110
|
+
if (res === null) return softSkip('blocked', 'GET /runs/{expiredRunId} unreachable (fetch failed)');
|
|
111
|
+
const code = readErrorCode(res.json);
|
|
112
|
+
expect(
|
|
113
|
+
(res.status === 404 && code === 'not_found') || (res.status === 410 && code === 'run_expired'),
|
|
114
|
+
req(ID_EXPIRED, DOC, `an expired run MUST answer 404 not_found or 410 run_expired through the error envelope (got ${res.status} ${code ?? 'no code'})`),
|
|
115
|
+
).toBe(true);
|
|
116
|
+
expect(typeof (isRecord(res.json) ? res.json['message'] : undefined), req(ID_EXPIRED, 'spec/v2/core/errors.md', 'the refusal is the canonical envelope: message is a string')).toBe('string');
|
|
117
|
+
});
|
|
118
|
+
});
|
|
@@ -42,10 +42,23 @@ import { req } from '../lib/requirement-ids.js';
|
|
|
42
42
|
import { readErrorCode } from '../lib/error-envelope.js';
|
|
43
43
|
import { blockedDespiteAssertions, softSkip } from '../lib/soft-skip.js';
|
|
44
44
|
import { CANARY_NAME, WITNESS_REF, WitnessHost, freshName, freshValue, hits, sha256Hex } from '../lib/run-secrets-witness.js';
|
|
45
|
+
import { familyAdvertised } from '../lib/v2.js';
|
|
45
46
|
|
|
46
47
|
const DOC = 'host-services.md §Run-supplied secrets (RFC 0229)';
|
|
47
48
|
const host = new WitnessHost(2);
|
|
48
49
|
|
|
50
|
+
/**
|
|
51
|
+
* The `secrets` record first, then the facet and fixture (`host.gate()`). With no
|
|
52
|
+
* record there is no `runSecrets`, so the verdict is unchanged; naming the family
|
|
53
|
+
* key here is what lets RFC 0264's derivation count a host that passes.
|
|
54
|
+
*/
|
|
55
|
+
async function gate(): ReturnType<WitnessHost['gate']> {
|
|
56
|
+
let record: Record<string, unknown> | null = null;
|
|
57
|
+
try { record = await familyAdvertised('secrets'); } catch { return host.gate(); }
|
|
58
|
+
if (record === null) return { ok: false, kind: 'inapplicable', reason: 'the secrets family is not advertised — no run-supplied secrets (RFC 0229 §D)' };
|
|
59
|
+
return host.gate();
|
|
60
|
+
}
|
|
61
|
+
|
|
49
62
|
interface Witnessed { runId: string; createText: string; status: string; matched: boolean | undefined; value: string; digestSupplied: boolean }
|
|
50
63
|
type Outcome = { runs: [Witnessed, Witnessed] } | { skip: 'inapplicable' | 'blocked'; reason: string } | { refused: { status: number; detail: string } };
|
|
51
64
|
|
|
@@ -53,7 +66,7 @@ type Outcome = { runs: [Witnessed, Witnessed] } | { skip: 'inapplicable' | 'bloc
|
|
|
53
66
|
let pair: Promise<Outcome> | undefined;
|
|
54
67
|
function witnessedPair(): Promise<Outcome> {
|
|
55
68
|
pair ??= (async (): Promise<Outcome> => {
|
|
56
|
-
const g = await
|
|
69
|
+
const g = await gate();
|
|
57
70
|
if (!g.ok) return { skip: g.kind, reason: g.reason };
|
|
58
71
|
const out: Witnessed[] = [];
|
|
59
72
|
for (const same of [true, false]) {
|
|
@@ -110,7 +123,7 @@ describe('RFC 0229 — v2-secrets-run-witness (gated on secrets.runSecrets + ope
|
|
|
110
123
|
}, 90_000);
|
|
111
124
|
|
|
112
125
|
it('a run: ref resolves only within the run that supplied it, and the witness refuses any other ref', async () => {
|
|
113
|
-
const g = await
|
|
126
|
+
const g = await gate();
|
|
114
127
|
if (!g.ok) return softSkip(g.kind, g.reason);
|
|
115
128
|
const id = 'openwop.requirement.secrets.run-witness-scope-bound';
|
|
116
129
|
const v = freshValue();
|
|
@@ -142,7 +155,7 @@ describe('RFC 0229 — v2-secrets-run-witness (gated on secrets.runSecrets + ope
|
|
|
142
155
|
}, 180_000);
|
|
143
156
|
|
|
144
157
|
it('the idempotency request digest excludes runSecrets: a retry differing only there replays, and its value is discarded', async () => {
|
|
145
|
-
const g = await
|
|
158
|
+
const g = await gate();
|
|
146
159
|
if (!g.ok) return softSkip(g.kind, g.reason);
|
|
147
160
|
const id = 'openwop.requirement.secrets.run-secrets-outside-request-digest';
|
|
148
161
|
const key = `openwop-witness-${randomUUID()}`;
|
|
@@ -101,7 +101,7 @@ function spanSurfaces(spans: readonly CapturedSpan[]): Surface[] {
|
|
|
101
101
|
describe('v2 RFC 0262 — telemetry facet shape (server-free)', () => {
|
|
102
102
|
it('the telemetry record requires otlp.protocols from the closed transports; genAiSemconv is semver', () => {
|
|
103
103
|
const v = v2RefValidator('capabilities.schema.json#/properties/telemetry');
|
|
104
|
-
const base = { status: 'experimental', since: '2.0', witness: 'witnessable-gated' };
|
|
104
|
+
const base = { status: 'experimental', since: '2.0', until: '2.1', witness: 'witnessable-gated' }; // capabilities.md: `until` is REQUIRED while experimental
|
|
105
105
|
expect(v({ ...base, otlp: { protocols: ['http/protobuf'] } }).ok, req(R('shape'), DOC, 'a record with otlp.protocols MUST validate')).toBe(true);
|
|
106
106
|
expect(v({ ...base, otlp: { protocols: ['grpc', 'http/json'] }, genAiSemconv: '1.37.0' }).ok, req(R('shape'), DOC, 'genAiSemconv as semver MUST validate')).toBe(true);
|
|
107
107
|
expect(v({ ...base }).ok, req(R('shape'), DOC, 'otlp is REQUIRED')).toBe(false);
|