@openwop/openwop-conformance 2.46.44 → 2.48.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +11 -0
- package/README.md +2 -2
- package/dist/spec-artifacts.lock.json +2 -2
- package/fixtures/conformance-compensate-unwind.json +70 -0
- package/fixtures/conformance-llm-cache-key.json +23 -0
- package/fixtures.md +20 -0
- package/package.json +2 -2
- package/requirements.json +538 -6
- package/scenario-majors.json +11 -2
- package/schemas/CORPUS-STAMP.json +22 -19
- package/src/lib/compensation-unwind.ts +173 -0
- package/src/lib/llm-cache-key-2.ts +114 -0
- package/src/scenarios/v2-compensation-unwind.test.ts +244 -0
- package/src/scenarios/v2-replay-llm-cache-key.test.ts +160 -0
- package/src/scenarios/v2-telemetry-export.test.ts +181 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,14 @@
|
|
|
1
|
+
## [2.48.0] — 2026-11-13 — Compensation unwind and derived adoption
|
|
2
|
+
|
|
3
|
+
- **RFC 0265.** `v2-compensation-unwind`: five host legs gated on `compensation` and the `conformance-compensate-unwind` fixture (legs 3 and 5 also on `idempotency`), reading only the event log, `GET /runs/{runId}/compensation`, `GET /runs/{runId}/effects` and a replay fork. The judges live in `lib/compensation-unwind.ts`, with a self-test that includes sabotage cases.
|
|
4
|
+
- **RFC 0264.** `v2-core-adoption-derived` (coherence) drives `scripts/derive-core-adoption.mjs` on fixture trees: a derived value against a hand edit, missing, stale and overdue review rows, and a renewal that only changes the date.
|
|
5
|
+
|
|
6
|
+
## [2.47.0] — 2026-11-13 — Cache-key recipe 2, telemetry export, sabotage ledger
|
|
7
|
+
|
|
8
|
+
- **RFC 0260.** `v2-replay-llm-cache-key`: a server-free shape leg, an unaided implies-idempotency leg, and four host legs on `GET /runs/{runId}/effects` (key recorded, parameters in the key, stable, portable), gated on `replay.llmCacheKeyRecipe` and the `conformance-llm-cache-key` fixture. Judges in `lib/llm-cache-key-2.ts`.
|
|
9
|
+
- **RFC 0262.** `v2-telemetry-export`: a server-free shape leg and five legs against the suite OTLP collector (run span, node span, `traceparent`, no secret in spans, GenAI attributes), gated on the `telemetry` family.
|
|
10
|
+
- **RFC 0263.** `v2-sabotage-ledger` (coherence) drives `check-accepted-predicate` rules 5 and 6 on fixture trees, with positive and negative controls.
|
|
11
|
+
|
|
1
12
|
## [2.46.44] — 2026-11-08 — Bounded loopback and resolvedVia go Active
|
|
2
13
|
|
|
3
14
|
- **RFC 0258.** `v2-bounded-loopback`: a server-free shape leg and six host legs (re-arm, history from the log, `ApprovalData.loopback`, withdraw, fail on exhaustion, replay re-arm), gated on the `loopback` family and the `conformance-loopback-refine` / `-fail` fixtures. Judges in `lib/bounded-loopback-witness.ts`.
|
package/README.md
CHANGED
|
@@ -137,7 +137,7 @@ Each scenario file records its disposition and assertion count in `<report-dir>/
|
|
|
137
137
|
|
|
138
138
|
## What's Covered
|
|
139
139
|
|
|
140
|
-
The current suite has
|
|
140
|
+
The current suite has 671 scenario files under `src/scenarios/`. By target major ([`scenario-majors.json`](./scenario-majors.json)): 231 files run against a v2 host and 455 against a v1 host; 15 of them run at both.
|
|
141
141
|
|
|
142
142
|
The v2 scenarios (`src/scenarios/v2-*.test.ts`) cover, by area:
|
|
143
143
|
|
|
@@ -166,7 +166,7 @@ Three kinds of scenario share the tree:
|
|
|
166
166
|
- **Host** checks drive a live host over HTTP.
|
|
167
167
|
- **Gated** checks run only when the host advertises the family, profile or fixture they need, and otherwise record `inapplicable` or `blocked` with a reason. `--require-behavior` turns an advertised-but-unwitnessable behaviour into a failure.
|
|
168
168
|
|
|
169
|
-
Current source tree:
|
|
169
|
+
Current source tree: 671 scenario files. [`coverage.md`](./coverage.md) maps each spec document and every OpenAPI operation to its scenarios. [`CHANGELOG.md`](./CHANGELOG.md) records when each scenario landed.
|
|
170
170
|
|
|
171
171
|
## How the 2.x suite is built
|
|
172
172
|
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
{
|
|
2
|
+
"id": "conformance-compensate-unwind",
|
|
3
|
+
"name": "Conformance: Compensate Unwind",
|
|
4
|
+
"version": "1.0",
|
|
5
|
+
"description": "RFC 0265 fixture for `v2-compensation-unwind.test.ts`: the condition an unwind exists for, caused by the suite unaided. Three effect nodes run in sequence (`e1` → `e2` → `e3`) and succeed; the last node, `fail` (`core.fail`, the always-fails node `conformance-failure` uses), then fails deterministically, so the run has committed effects and a `node-failure` trigger. Each effect node declares a node-level `compensation` whose inverse is the same reserved type with `inputMapping.inverse: true`, and `settings.compensation` asks for a `reverse-completion` unwind on `node-failure`. `core.conformance.compensable-effect` is a conformance-RESERVED typeId (same pattern as `core.conformance.side-effect`): a host that advertises `compensation` and serves this fixture MUST map it to a node it records on the Layer-2 effect ledger (`GET /runs/{runId}/effects`) as exactly one effect per execution, forward and inverse alike, and MUST run it as the inverse action when its resolved inputs carry `inverse: true`. What the effect does is irrelevant; that it is recorded once per execution is the whole contract. The suite reads the unwind through normative paths only: the event log (`compensation.requested` before `compensation.started`), `GET /runs/{runId}/compensation` (attempts in the order `e3, e2, e1`, the reverse of forward completion) and the effect ledger (three inverse records, none added by a `mode: \"replay\"` fork). Serving the fixture is OPTIONAL; a host that does not list it in discovery `fixtures[]` is `inapplicable` for every leg.",
|
|
6
|
+
"nodes": [
|
|
7
|
+
{
|
|
8
|
+
"id": "e1",
|
|
9
|
+
"typeId": "core.conformance.compensable-effect",
|
|
10
|
+
"name": "Effect 1",
|
|
11
|
+
"position": { "x": 0, "y": 0 },
|
|
12
|
+
"config": {},
|
|
13
|
+
"inputs": {},
|
|
14
|
+
"compensation": {
|
|
15
|
+
"nodeTypeId": "core.conformance.compensable-effect",
|
|
16
|
+
"inputMapping": { "inverse": true }
|
|
17
|
+
}
|
|
18
|
+
},
|
|
19
|
+
{
|
|
20
|
+
"id": "e2",
|
|
21
|
+
"typeId": "core.conformance.compensable-effect",
|
|
22
|
+
"name": "Effect 2",
|
|
23
|
+
"position": { "x": 200, "y": 0 },
|
|
24
|
+
"config": {},
|
|
25
|
+
"inputs": {},
|
|
26
|
+
"compensation": {
|
|
27
|
+
"nodeTypeId": "core.conformance.compensable-effect",
|
|
28
|
+
"inputMapping": { "inverse": true }
|
|
29
|
+
}
|
|
30
|
+
},
|
|
31
|
+
{
|
|
32
|
+
"id": "e3",
|
|
33
|
+
"typeId": "core.conformance.compensable-effect",
|
|
34
|
+
"name": "Effect 3",
|
|
35
|
+
"position": { "x": 400, "y": 0 },
|
|
36
|
+
"config": {},
|
|
37
|
+
"inputs": {},
|
|
38
|
+
"compensation": {
|
|
39
|
+
"nodeTypeId": "core.conformance.compensable-effect",
|
|
40
|
+
"inputMapping": { "inverse": true }
|
|
41
|
+
}
|
|
42
|
+
},
|
|
43
|
+
{
|
|
44
|
+
"id": "fail",
|
|
45
|
+
"typeId": "core.fail",
|
|
46
|
+
"name": "Fail",
|
|
47
|
+
"position": { "x": 600, "y": 0 },
|
|
48
|
+
"config": {},
|
|
49
|
+
"inputs": {}
|
|
50
|
+
}
|
|
51
|
+
],
|
|
52
|
+
"edges": [
|
|
53
|
+
{ "id": "e1-e2", "sourceNodeId": "e1", "targetNodeId": "e2" },
|
|
54
|
+
{ "id": "e2-e3", "sourceNodeId": "e2", "targetNodeId": "e3" },
|
|
55
|
+
{ "id": "e3-fail", "sourceNodeId": "e3", "targetNodeId": "fail" }
|
|
56
|
+
],
|
|
57
|
+
"triggers": [
|
|
58
|
+
{ "id": "manual", "type": "manual", "enabled": true }
|
|
59
|
+
],
|
|
60
|
+
"variables": [],
|
|
61
|
+
"metadata": { "tags": ["conformance"] },
|
|
62
|
+
"settings": {
|
|
63
|
+
"timeout": 30000,
|
|
64
|
+
"maxRetries": 0,
|
|
65
|
+
"compensation": {
|
|
66
|
+
"orderingModel": "reverse-completion",
|
|
67
|
+
"triggers": ["node-failure"]
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
}
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
{
|
|
2
|
+
"id": "conformance-llm-cache-key",
|
|
3
|
+
"name": "Conformance: LLM cache key recipe 2 (RFC 0260)",
|
|
4
|
+
"version": "1.0",
|
|
5
|
+
"description": "Fixture for `v2-replay-llm-cache-key.test.ts` (RFC 0260; the rules are the `replay.llmCacheKeyRecipe` facet description and the effect-ledger `cacheKey` description). One node, `conformance.callai.cache-key` (conformance-RESERVED). It makes exactly one `ctx.callAI` call through the host's conformance mock provider, whose provider id is `conformance-mock` and model id `conformance-mock-1`. The call sends one message `{role: \"user\", content: inputs.request.prompt}`, no tools, no toolChoice, no responseSchema, and exactly the parameters in `inputs.request.params` (keys from recipe 2's named set: `temperature`, `maxTokens`, `stop`, `seed`, `topP`, `topK`, `frequencyPenalty`, `presencePenalty`), nothing more. The mock replies `ok`. The host records the call on `GET /runs/{runId}/effects` under this node's id, carrying `cacheKey`. The node completes with `outputs.text`, the reply. A host that does not advertise `replay.llmCacheKeyRecipe` MUST NOT advertise this fixture.",
|
|
6
|
+
"nodes": [
|
|
7
|
+
{
|
|
8
|
+
"id": "llm-call",
|
|
9
|
+
"typeId": "conformance.callai.cache-key",
|
|
10
|
+
"name": "One ctx.callAI call through the conformance mock",
|
|
11
|
+
"position": { "x": 0, "y": 0 },
|
|
12
|
+
"config": {},
|
|
13
|
+
"inputs": {}
|
|
14
|
+
}
|
|
15
|
+
],
|
|
16
|
+
"edges": [],
|
|
17
|
+
"triggers": [{ "id": "manual", "type": "manual", "enabled": true }],
|
|
18
|
+
"variables": [
|
|
19
|
+
{ "name": "request", "type": "object" }
|
|
20
|
+
],
|
|
21
|
+
"metadata": { "tags": ["conformance", "replay", "llmCacheKeyRecipe"] },
|
|
22
|
+
"settings": { "timeout": 0 }
|
|
23
|
+
}
|
package/fixtures.md
CHANGED
|
@@ -150,6 +150,8 @@ All fixtures MUST advertise:
|
|
|
150
150
|
| v2 fs Sandbox Probe | `conformance-fs-probe` | `storage.md` §`fs` — one `core.conformance.fs-probe` node (conformance-RESERVED) calls the host's own `ctx.fs.read` / `write` with the run's `path`, unchanged; a rejection fails the node with its code and details. Operator symlink `<sandboxRoot>/conformance/escape-link` points outside the root. See §"The storage probe fixtures" | `completed` inside the root; `failed` (`forbidden`, `details.reason: path-outside-sandbox`) for an escape | ≤ 30s |
|
|
151
151
|
| v2 tableStorage Schema Probe | `conformance-table-schema-probe` | `storage.md` §`tableStorage` — one `core.conformance.table-schema-probe` node (conformance-RESERVED) creates a table `{ k: string, n: number }` with the host's own `ctx.storage.table`, then inserts a well-typed row (`control`), a row whose `n` is a string (`insert-mistyped`), or updates `n` to a string (`update-mistyped`). See §"The tableStorage schema probe fixture" | `completed` for `control`; `failed` (`validation_error`, `details.service: tableStorage`) for a mistyped write | ≤ 30s |
|
|
152
152
|
| v2 ctx.callAI Attempt Accounting | `conformance-callai-attempt-accounting` | RFC 0248 — one `conformance.callai.echo-attempts` node (conformance-RESERVED) that makes one `ctx.callAI` call through the host's conformance mock, serves the run's `script`, and echoes the result's `attempts` (`v2-callai-attempt-accounting`) | each case run completes or fails as its script dictates | ≤ 30s |
|
|
153
|
+
| v2 LLM Cache Key | `conformance-llm-cache-key` | RFC 0260 — one `conformance.callai.cache-key` node (conformance-RESERVED) making one `ctx.callAI` call through the suite's mock provider with the run's prompt and params unchanged (`v2-replay-llm-cache-key`) | completes; the call's effect record carries `cacheKey` | ≤ 30s |
|
|
154
|
+
| v2 Compensate Unwind | `conformance-compensate-unwind` | RFC 0265 — three `core.conformance.compensable-effect` nodes (conformance-RESERVED) `e1 → e2 → e3`, each declaring itself as its own inverse (`inputMapping.inverse: true`), then `core.fail`; `settings.compensation` is `reverse-completion` on `node-failure` (`v2-compensation-unwind`). See §"The compensate-unwind fixture" | `failed`, then an unwind: `compensation.requested` → `compensation.started` → `compensation.completed`; three inverse records on the effect ledger | ≤ 30s |
|
|
153
155
|
| v2 safeFetch SSRF Probe | `conformance-safefetch-probe` | `host-services.md` §`httpClient` — one `core.conformance.safefetch-probe` node (conformance-RESERVED) calls the host's own `ctx.http.safeFetch(url)` with the run's `url`, unchanged; completes with the response `status` only; a rejection fails the node with its code and details. See §"The safeFetch probe fixture" | `failed` (`egress_denied`, `details.reason: ssrf-blocked`) for a refused target | ≤ 60s |
|
|
154
156
|
| A2A Task Roundtrip | `conformance-a2a-task-roundtrip` | Track 6 — host consumes the conformance suite's synthetic A2A peer; covers drift points #3 (`AUTH_REQUIRED`) and #4 (`REJECTED`). Node `core.conformance.a2a-invoke` is a **conformance-RESERVED** typeId (renamed 2026-08-16 from `core.a2a.invoke`, a multi-segment `core.*` id no host shipped and the corpus never defined): a host that consumes A2A peers MUST map it to its A2A bridge node (a host-extension node — `a2a-integration.md` §"State projection"); a host that does not consume A2A MUST NOT advertise this fixture | `failed` or `waiting-input` (per `driftScenario` input) | ≤ 30s |
|
|
155
157
|
| WASM Pack Roundtrip | `conformance-wasm-pack-roundtrip` | RFC 0008 — invokes `vendor.openwop.rust-hello.greet` (loaded WASM pack); exercises required exports + at least one import | `completed` | ≤ 10s |
|
|
@@ -850,6 +852,21 @@ The scope leg sends a promote with `OPENWOP_TEST_LOW_SCOPE_KEY`, a key with no `
|
|
|
850
852
|
- **Gate**: `v2-interrupt-form-rules` runs each leg only when its fixtures are advertised. No capability or seam.
|
|
851
853
|
- **Timing**: the positive suspends; the probes settle within 10 s.
|
|
852
854
|
|
|
855
|
+
## The compensate-unwind fixture (RFC 0265)
|
|
856
|
+
|
|
857
|
+
`conformance-compensate-unwind` is what `v2-compensation-unwind` drives. Its effect node, `core.conformance.compensable-effect`, is conformance-RESERVED. A host that advertises `compensation` and serves the fixture maps it to a node it records on the effect ledger (`GET /runs/{runId}/effects`) as exactly one effect per execution, forward and inverse alike, and runs it as the inverse action when its resolved inputs carry `inverse: true`. What the effect does is irrelevant. A host that does not advertise `compensation` MUST NOT advertise the fixture: it would refuse the workflow's `settings.compensation` with `capability_required`.
|
|
858
|
+
|
|
859
|
+
- **Shape**: `e1` → `e2` → `e3` (`core.conformance.compensable-effect`, each with node-level `compensation.nodeTypeId` = the same type and `inputMapping: { inverse: true }`) → `fail` (`core.fail`). `settings.compensation`: `orderingModel: "reverse-completion"`, `triggers: ["node-failure"]`.
|
|
860
|
+
- **Inputs**: none.
|
|
861
|
+
- **Expected behavior**:
|
|
862
|
+
1. `e1`, `e2` and `e3` complete in order; `fail` fails; the run ends `failed`.
|
|
863
|
+
2. The log holds `compensation.requested` before the first `compensation.started`, and ends the unwind with `compensation.completed`.
|
|
864
|
+
3. `GET /runs/{runId}/compensation` reads `status: "completed"`, `plan[]` naming `e1`, `e2` and `e3`, and `attempts[]` in the order `e3, e2, e1`.
|
|
865
|
+
4. The effect ledger holds one forward record per effect node and exactly three inverse records.
|
|
866
|
+
5. A `mode: "replay"` fork records no effect the source run did not make.
|
|
867
|
+
- **Advertise only if** the host advertises `compensation`. Legs 3 and 5 also need `idempotency` (the ledger); leg 5 needs `replay.modes ∋ replay`.
|
|
868
|
+
- **Terminal status**: `failed`.
|
|
869
|
+
|
|
853
870
|
## NodeModule registration
|
|
854
871
|
|
|
855
872
|
The fixtures reference these typeIds:
|
|
@@ -857,6 +874,7 @@ The fixtures reference these typeIds:
|
|
|
857
874
|
| typeId | Required by | Behavior |
|
|
858
875
|
| ----------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
859
876
|
| `core.noop` | noop, multi-node, approval, clarification, cancellable | Immediate completion, no output |
|
|
877
|
+
| `core.conformance.compensable-effect` | compensate-unwind | Conformance-RESERVED (RFC 0265). Performs one effect the host records on the Layer-2 effect ledger, one record per execution; with `inverse: true` in its resolved inputs it is the inverse action. Completes with no output. |
|
|
860
878
|
| `core.identity` | identity | Echo `input.payload` to `output.payload` |
|
|
861
879
|
| `core.delay` | delay, cancellable | Sleep `config.delayMs` ms |
|
|
862
880
|
| `core.fail` | failure | Throw with `code: "example.conformance_failure"` (a vendor code under the registered `example` org; errors.md §The registry, openwop#1698), message: "Intentional conformance failure" |
|
|
@@ -924,6 +942,8 @@ For `v2-tool-catalog-annotations`' unclassified-MCP leg, a host that advertises
|
|
|
924
942
|
|
|
925
943
|
`conformance-callai-attempt-accounting` is what `v2-callai-attempt-accounting` drives. Its node, `conformance.callai.echo-attempts`, is conformance-RESERVED. A host that advertises `aiProviders.callAccounting` maps it to a node that makes exactly one `ctx.callAI` call through the host's conformance mock provider. It does no counting of its own. A host that does not advertise `callAccounting` MUST NOT advertise the fixture.
|
|
926
944
|
|
|
945
|
+
`conformance-llm-cache-key` is what `v2-replay-llm-cache-key` drives. Its node, `conformance.callai.cache-key`, is conformance-RESERVED. A host that advertises `replay.llmCacheKeyRecipe` maps it to a node that makes exactly one `ctx.callAI` call through the host's conformance mock provider, passing the run's prompt and parameters unchanged, so the suite can recompute the recipe-2 key from the inputs it sent. A host that does not advertise `llmCacheKeyRecipe` MUST NOT advertise the fixture.
|
|
946
|
+
|
|
927
947
|
**Inputs.** `request` is passed to `ctx.callAI` unchanged: `prompt`, and optionally `responseSchema`, `maxTokens` and `retry`. `script` is the mock's replies.
|
|
928
948
|
|
|
929
949
|
**The mock.** It serves `script` to the call's provider requests in order, one entry per request, and repeats the last entry past the end. Each entry is `{ kind, outputTokens?, usage?, structuredMode? }`:
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@openwop/openwop-conformance",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.48.0",
|
|
4
4
|
"description": "Black-box conformance suite for OpenWOP hosts (v2 current; v1 scenarios retained for the frozen v1 tree).",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -56,6 +56,6 @@
|
|
|
56
56
|
"@openwop/spec-artifacts": "file:../spec-artifacts"
|
|
57
57
|
},
|
|
58
58
|
"peerDependencies": {
|
|
59
|
-
"@openwop/spec-artifacts": "2.
|
|
59
|
+
"@openwop/spec-artifacts": "2.48.0"
|
|
60
60
|
}
|
|
61
61
|
}
|