@openwop/openwop-conformance 2.42.9 → 2.43.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +29 -0
- package/README.md +3 -3
- package/coverage.md +1 -1
- package/dist/cli.js +15 -12
- package/dist/lib/certification-bundle-v3.js +19 -0
- package/dist/spec-artifacts.lock.json +2 -2
- package/fixtures/conformance-approval-reject-loopback.json +33 -0
- package/fixtures/conformance-approval-reject-routed.json +32 -0
- package/fixtures/conformance-approval-timeout-approve.json +46 -0
- package/fixtures/conformance-approval-timeout.json +28 -0
- package/fixtures.md +59 -2
- package/package.json +2 -2
- package/requirements.json +535 -38
- package/scenario-majors.json +20 -5
- package/schemas/CORPUS-STAMP.json +55 -54
- package/src/cli.ts +15 -12
- package/src/lib/auditIntegrity.ts +45 -0
- package/src/lib/certification-bundle-v3.ts +25 -0
- package/src/lib/standard-webhooks.ts +53 -0
- package/src/scenarios/agent-org-chart-scoping.test.ts +63 -0
- package/src/scenarios/audit-anomaly-shape.test.ts +100 -0
- package/src/scenarios/audit-checkpoint-signature.test.ts +14 -15
- package/src/scenarios/audit-log-integrity.test.ts +102 -46
- package/src/scenarios/byok-roundtrip.test.ts +21 -7
- package/src/scenarios/v2-agent-org-chart-served-shape.test.ts +63 -0
- package/src/scenarios/v2-approval-reject-disposition.test.ts +262 -0
- package/src/scenarios/v2-error-registry.test.ts +59 -0
- package/src/scenarios/v2-manifest-ceiling-refused.test.ts +21 -9
- package/src/scenarios/v2-manifest-hatch-carried.test.ts +23 -7
- package/src/scenarios/v2-pack-isolation.test.ts +10 -6
- package/src/scenarios/v2-peer-dependency-declared.test.ts +18 -6
- package/src/scenarios/v2-registry-lifecycle.test.ts +232 -0
- package/src/scenarios/v2-webhook-secret-rotation.test.ts +28 -11
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,34 @@
|
|
|
1
1
|
# `@openwop/openwop-conformance` Changelog
|
|
2
2
|
|
|
3
|
+
## [2.43.1] — 2026-09-28 — major-1 certification works again, RFC 0223's routed and timed-out rejects are witnessed, and four Class 3 corrections
|
|
4
|
+
|
|
5
|
+
- **`v2-manifest-hatch-carried`'s agent fixture is schema-valid.** Its `agents[0]` had neither `systemPrompt` nor `systemPromptRef`, and `agent-manifest` (v1 and v2) requires exactly one. A conforming host therefore had to refuse the pack for a reason unrelated to the `x-` hatch, and `0177.manifest-hatch-carried.agents-x-field` could not pass on any host. The fixture now carries `systemPrompt`. Validated against `schemas/v2/agent-manifest.schema.json`: invalid before, valid after. Reported by MyndHyve.
|
|
6
|
+
- **RFC 0223 G1 and G2: the routed and the timed-out approval reject are witnessed** (#1700). Two new fixtures and two new rows in `v2-approval-reject-disposition.test.ts` (target major 2). Each row is `inapplicable` on a host that does not advertise its fixture.
|
|
7
|
+
- `conformance-approval-reject-routed` and `0223.reject-routed`: the gate has an `any_failed` edge to `on-reject` and an `all_success` edge to `on-accept`. Rejected, the gate still fails as a node (`approval_rejected`, `retryable: false`), `on-reject` completes, `on-accept` never runs, and the run completes. `node.skipped` is not required, because `interrupt.md` §Rejection says the edge is not satisfied, not how a host records it.
|
|
8
|
+
- `conformance-approval-timeout` and `0223.timeout-rejects`: the gate has a 1500 ms `timeoutMs` and no `onTimeout`. Left unresolved, the host resolves it with `action: "timeout"`, `decision: "rejected"`, `reason: "timeout"`, the gate fails with `approval_rejected`, and `run.failed` names it.
|
|
9
|
+
- Witnessed on the v2 reference host (openwop-examples `777b485`) with both fixtures added to its honoured set: all six rows `executed-pass`. Sabotaged, emptying the host's failure-admitting `triggerRule` set fails `0223.reject-routed`, and short-circuiting its timeout sweep fails `0223.timeout-rejects`. The host registers the two ids in a follow-up.
|
|
10
|
+
- **The pack isolation and manifest rows no longer gate on `packs`** (#1706, Class 3). `packs` advertises registry resolution, which a host that only loads local packs correctly omits, so these rows went unmeasured on exactly that host. `v2-pack-isolation` gates on `sandbox` alone. `v2-manifest-ceiling-refused`, `v2-manifest-hatch-carried` and `v2-peer-dependency-declared` gate on the seams-profile install path, and a publish-seam `404` records `inapplicable` when `packs` is absent and `blocked` when it is advertised. Requirement ids are unchanged.
|
|
11
|
+
- **A timeout never grants an approval gate** (#1696, Class 3). New fixture `conformance-approval-timeout-approve` (1500 ms `timeoutMs`, `onTimeout: "approve"`) and row `0223.timeout-never-grants` in `v2-approval-reject-disposition` (`inapplicable` without the fixture). The timed-out gate MUST resolve rejected (`action: timeout`, `decision: rejected`), fail with `approval_rejected`, and the run MUST fail rather than complete. Measured on the v2 reference host: `executed-fail` today, because its sweep skips a non-`reject` `onTimeout` and the run waits forever. It passes with that exclusion dropped, which is its follow-up.
|
|
12
|
+
- **An interrupt `key` is per visit** (#1697, Class 3). New fixture `conformance-approval-reject-loopback` (`start` → `gate`, `gate` → `revise` on `any_failed`, `revise` → `gate`) and row `0223.reject-loopback-reasks`. Rejected once, the gate is asked again under a different `key`, the run is `waiting-approval` again, and only one `interrupt.resolved` is recorded. The fixture is advertised only by a host that runs cycles; everyone else records `inapplicable`.
|
|
13
|
+
- **`v2-error-registry` checks the codes on events** (#1698, Class 3). New leg `openwop.requirement.errors.event-code-registered`. It runs `conformance-failure` and checks every `error.code` on `run.failed`, `node.failed` and the snapshot against the registry and the vendor pattern (the org must be registered in `declaration.json`). It is ADVISORY in 2.43.1: a violation records `executed-pass` with a `partial-witness` detail naming the codes, and becomes a failure in a later patch. On the v2 reference host it currently names `fixture_failure`. `fixtures.md`'s `core.fail` now throws `example.conformance_failure`.
|
|
14
|
+
- **RFC 0201 §E.20's post-overlap rule is witnessable on a long overlap.** New OPTIONAL seams-profile seam `POST /conformance/seams/sample/webhooks/rotation-overlap { webhookId, overlapSeconds }` (`host-sample-test-seams.md` §29; `api/seams-v2.yaml` `shortenRotationOverlap`) shortens the rotation overlap in progress on one subscription by moving the stored expiry the production signer reads. It is tenant-checked like `rotateWebhookSecret` and only ever shortens. `v2-webhook-secret-rotation` calls it only when the advertised `overlapSeconds` outlasts the wait cap: it sets 3 s, waits past the reported expiry, and asserts that only the new secret signs. Dispositions: seam served → real assertion; served but contract broken (wrong expiry, or a foreign-tenant id not refused `403 id_tenant_mismatch`) → `blocked`; not served, or no seams profile → the pre-seam partial witness, never `blocked`. Witnessed on the v2 reference host (openwop-examples#117) with a 3600 s overlap: `executed-pass` (17 assertions); seam unmounted → partial witness; a signer that ignores the expiry → `executed-fail`; a seam that misreports the expiry → `blocked`.
|
|
15
|
+
- **`--certify --bundle-version 3` no longer refuses its own bundle when a secret is redacted from a row.** The emitter digested and signed the raw requirement rows, then ran `scrubEvidence` over the finished bundle. A configured secret (the handed `--api-key`, an `OPENWOP_*` key/token/secret env value, or the canary) found in a row's `detail` or `evidence` was redacted AFTER `witnessSha256` was taken, so the written rows no longer hashed to it and self-verification exited 2 with `[witness-digest]` and no bundle. openwop-app hit it at major 1 on 2.43.0. The new `sealBundleV3` scrubs first, then digests and signs, so the signed rows are the shipped rows; the v3 path now also prints the `REDACTED … field(s)` line the v2 path always had. Reproduced on published 2.43.0 against the v1 SQLite reference host at major 1 with an `OPENWOP_*_SECRET` value that occurs in 274 row details: exit 2, `witnessSha256 0657c889365e` (the clean run's digest) against rows digesting to `803b02c17414`. With the fix the same run writes a bundle `--verify` accepts. New self-tests in `certification-bundle-v3.test.ts` pin both orders. Sabotaged, digesting the pre-scrub rows fails the new test. No wire, schema or digest-preimage change.
|
|
16
|
+
- **The org-chart read pair's served bodies are validated against their schemas** (RFC 0087 §A/§D). Only a server-free probe compiled `agent-org-chart.schema.json`; the live scenario walked the tree and member refs and never validated the body. openwop-app served its stored record (`{ tenantId, departments, members, updatedAt }`: `owner` missing, two extra keys on a closed object) at both majors, and every row passed (fixed in openwop-app #4189). New row `openwop.it.agent-org-chart-scoping.serves-the-org-chart-read-pair-in-the-exact-schema-shape` (major 1, same `openwop-org-chart-scoping` gate) and new file `v2-agent-org-chart-served-shape.test.ts` (major 2, `inapplicable` unless the v2 `agents` record carries `orgChart`) validate `GET /agents/org-chart` against `agent-org-chart.schema.json` and, when `responsibilityView` is advertised, `GET /agents/org-chart/{departmentId}` against `org-chart-responsibility-view.schema.json`. Witnessed on openwop-app `c4aad98c2` in-process (one seeded department): `executed-pass` at both majors; the pre-fix `routes/orgChart.ts` (`553355847`) → `executed-fail` at both majors on `owner` + extra keys while the old tree row still passed; an extra key on the department view alone → `executed-fail` at both majors. openwop-app does not advertise `agents` at major 2, so the v2 row records `inapplicable` there; the major-2 witness advertised `agents.orgChart` via a local, uncommitted discovery patch.
|
|
17
|
+
- **Suite `2.43.1`**. This adds scenario rows and fixtures, the same class as the 2.42.8 and 2.42.9 scenario changes. The only packed contract file that changes is `spec/v1/gaps.json`, where G1 and G2 close. No wire change. `@openwop/spec-artifacts` moves in lockstep at the same exact pin.
|
|
18
|
+
|
|
19
|
+
## [2.43.0] — 2026-09-28 — four RFCs: approval reject, registry operations, the audit anomaly shape, and a v2 home for audit-log integrity
|
|
20
|
+
|
|
21
|
+
- **RFC 0222: v2 registry lifecycle and signing keys, witnessed against a registry.** New `v2-registry-lifecycle.test.ts` (target major 2). It reads a registry, not the host: it runs when `OPENWOP_REGISTRY_URL` names one or `OPENWOP_TEST_PUBLIC_REGISTRY=true` names `packs.openwop.dev`, records `inapplicable` otherwise, and resolves every path through `.well-known` `endpoints.v2`. Leg 1 (`openwop.requirement.0222.yanked-version-lifecycle`): every pack index agrees with its version manifests on `yanked` and `versionDeprecated`, never names a yanked version `latest` while an unyanked one exists, and a yanked version's tarball and signature stay served. Leg 2 (`openwop.requirement.0222.signing-keys-cover-served-versions`): every served version names a key `signingKeys[]` still lists, whose `permittedNamespaces` admit it, and its integrity and signature over the in-tarball `pack.json` verify, whatever the key's `status`. Passes on `packs.openwop.dev` (198 versions; leg 1 `partial-witness`, no v2 version is yanked) and on a local tree with a yanked version; fails on seven sabotages, including the registry's previous `build-index.mjs`.
|
|
22
|
+
- **RFC 0220 `Active`: six extension families are `Stable`.** Packed `@openwop/spec-artifacts` files change (`spec/v2/declaration.json` `technical: stable` for chat, entities, kanban, knowledge, restTransport and webResearch; their `spec/v2/ext/*/README.md` read `Status: Stable.`). No scenario change.
|
|
23
|
+
- **RFC 0223: a rejected approval gate fails closed, and the failure is routable.** New `v2-approval-reject-disposition.test.ts` (target major 2, gated on the `interrupt` family). Four rows, so a host fails only what it misses. `0223.reject-fails-run`: `conformance-approval` resolved `reject` ends `failed`, with `approval_rejected` on the snapshot and on `run.failed`. `0223.reject-recorded`: `interrupt.resolved` carries `action: "reject"` and `decision: "rejected"`. `0223.reject-fails-node`: the gate's `node.failed` carries `approval_rejected` with `retryable: false`, and `run.failed.failedNodeId` is `gate`. `0223.quorum-reject-fails-run` (gated on `conformance-interrupt-quorum`, `inapplicable` when not advertised): one reject of three under `majority` leaves the run waiting and emits no `interrupt.resolved`, and the second fails the run with `approval_rejected`. Measured before release: the v2 reference host fails only `reject-recorded` (no `action`, fixed in its follow-up); openwop-app fails `reject-recorded` and `reject-fails-node`; MyndHyve fails on its `APPROVAL_REJECTED` spelling. The major-1 `interrupt-quorum-resolution` is unchanged: v1 registers no `approval_rejected`.
|
|
24
|
+
- **Coherence: `v2-error-registry-prose-parity` gains an RFC 0223 leg** (server-free, `0223.code-registered`): the registry row is a 422 that is not retriable, `interrupt.md` §Rejection names it, and `ApprovalData.onTimeout` declares no JSON-Schema `default`.
|
|
25
|
+
- **`fixtures.md`**: `conformance-approval` states its major-2 outcome after a reject; the quorum catalog row names the code.
|
|
26
|
+
- **RFC 0218 §C: new server-free `audit-anomaly-shape.test.ts` (both majors, requirement id `openwop.requirement.0218.anomaly-shape`).** It validates one anomaly of every `kind` against `audit-verify-result.schema.json` (v1 and v2), plus a legacy kind-less chain-break. It refuses the reference hosts' pre-§C `{ atSequence, kind, detail }`, a checkpoint kind without `checkpoint`, a `chain-break` without its hashes, mixed entries and an unknown `kind`. It also refuses `chainValid: true` beside an anomaly and `chainValid: false` with none. Five schema sabotages each turn a leg red.
|
|
27
|
+
- **`v2-surface-monotone-gate` gains four legs for the new-value conditional licence.** A `required` under a `then` selecting only new enum members passes and is reported. An old-value, mixed or unpinned selector is refused, and so is an unconditional `required` beside a licensed one.
|
|
28
|
+
- **RFC 0224: `audit-log-integrity` and `audit-checkpoint-signature` run at both majors.** At major 2 they gate on the new `auditLogIntegrity` family record (absent ⇒ `inapplicable` before any assertion) and call `/audit/verify`. Major 1 keeps its profile gate and `/v1/audit/verify`. Both resolve through the new `src/lib/auditIntegrity.ts`, and both are now in `BOTH_MAJORS`. At major 2 the shape leg validates the record against the generated family schema and parses the key as Ed25519 SPKI, and the verify leg validates against the v2 body schema. New cadence leg at both majors (`openwop.requirement.0224.checkpoint-cadence`): checkpoints are listed ascending, with no gap over the advertised `checkpointIntervalEntries`; it is `inapplicable` before a first checkpoint. Sabotage on the v2 reference host: signing the checkpoint object reddens the RFC 0218 signature row, and dropping a checkpoint from the verify body reddens the cadence row. The SQLite host at major 1 is executed-pass on both files.
|
|
29
|
+
- **`byok-roundtrip` records `blocked` instead of an unclassified return, so one unwitnessable floor no longer voids a whole cut.** The file was wrapped in `describe.skipIf(SKIP_NO_FIXTURE)`: on a host that does not advertise the `openwop-smoke-byok-roundtrip` fixture, every test was skipped before it could record a reason. A file that skips everything with no reason is an unclassified return (RFC 0148 §A), and an unclassified FLOOR row (`openwop.floor.byok-roundtrip`, `openwop-secrets`) in a claimed profile rejects the certification outright. Measured on openwop-app's 2026-09-28 post-deploy major-1 cut: 2,116 rows, `certified: none`, and that file the only unclassified row. The previous published cut (`f270cfb6`) carried the identical row. Each test now gates itself: with no fixture advertised it records `blocked`, naming the missing fixture and the seam behind it. A production posture that keeps `conformance.secret.echo` off therefore leaves `openwop-secrets` uncertified, which is honest, since the floor was not witnessed, and ONLY that profile: the cut's other claims certify. The first test's 404/422 branch moves from `inapplicable` to `blocked` in the same change. On a floor, `inapplicable` resolves to an `executed-pass` partial witness, so a host whose canary run cannot start was being credited with a floor it never demonstrated. The two sibling tests already answered the same condition with `blocked`. Run against app.openwop.dev (fixture not advertised): all three rows and `openwop.floor.byok-roundtrip` resolve `blocked`, with no unclassified return. `requirements.json` is regenerated for three moved line numbers only; no id or title changed. The other 31 files gated on a bare `SKIP_NO_FIXTURE` have the same shape and were not touched: none is a floor, so their unclassified rows do not decide a certification. They are left for a sweep rather than folded in here.
|
|
30
|
+
- **Suite `2.43.0`**, a minor: a new registry row changes the packed `@openwop/spec-artifacts` tree (`spec/v2/errors.json` and the generated `schemas/v2/error-envelope.schema.json`), and 2.42.9 is published. `@openwop/spec-artifacts` moves in lockstep at the same exact pin.
|
|
31
|
+
|
|
3
32
|
## [2.42.9] — 2026-09-28 — a v2 delivery's workspaceId is checked, and two test-isolation and SHOULD-strength fixes
|
|
4
33
|
|
|
5
34
|
- **Scenario files that share a mock-AI node no longer race.** The mock-AI program hook is keyed by node id, and a node id belongs to a fixture. `envelope-completion-distinguishes-truncation` drives both envelope fixtures that `envelope-truncated` and `envelope-retry-attempted` also drive, so under the default parallel run one file could overwrite another's program. New `lib/mock-node-lock.ts` holds a cross-process lock per node id for the whole file, so files sharing a node run one after another. Serial runs (`--certify`, openwop-app's harness) are unchanged.
|
package/README.md
CHANGED
|
@@ -11,7 +11,7 @@
|
|
|
11
11
|
# --legacy-peer-deps is REQUIRED, not optional: the exact peer pin is what npm's
|
|
12
12
|
# default resolver refuses. npm 10.9 fails outright with
|
|
13
13
|
# "Cannot read properties of null (reading 'edgesOut')" — use npm >= 11.
|
|
14
|
-
npm install --legacy-peer-deps @openwop/openwop-conformance@2.
|
|
14
|
+
npm install --legacy-peer-deps @openwop/openwop-conformance@2.43.1 @openwop/spec-artifacts@2.43.1
|
|
15
15
|
# or run without install:
|
|
16
16
|
npx @openwop/openwop-conformance --base-url https://api.example.com --api-key hk_test_...
|
|
17
17
|
```
|
|
@@ -135,7 +135,7 @@ Exit code is non-zero on any failed assertion. `--certify` distinguishes: `0`
|
|
|
135
135
|
|
|
136
136
|
## What's Covered
|
|
137
137
|
|
|
138
|
-
The current suite has
|
|
138
|
+
The current suite has 571 scenario files under `src/scenarios/`.
|
|
139
139
|
- 2026-09-23 (suite 2.37.0 cycle, RFC 0213): NEW `v2-sse-last-event-id-cursor.test.ts` (a `Last-Event-ID` past the log is an exclusive cursor; a malformed id, when refused, is `400 validation_error`; the cursor never changes the answer for an unknown or foreign-tenant run — public test of `event-cursor-after-authorization`), `v2-idempotency-in-flight.test.ts` (five concurrent same-key creates yield one run; each loser is a marked replay or `409 idempotency_in_flight` with no retry timing in `details`; `partial-witness` when no loser was refused in flight) and `v2-interrupt-resolve-terminal.test.ts` (a run-scoped resolve after cancel or completion is `409 interrupt_already_resolved`, never `interrupt_cancelled`). All three sit off the core-standard floor until measured on the three bundle hosts.
|
|
140
140
|
- 2026-09-03 (suite `1.157.0 -> 1.158.0`, gap G17): NEW `idempotency-concurrent-claim.test.ts` — drives the new `host-sample-test-seams.md` §25 concurrent duplicate-delivery seam for the RFC 0150 §B / `idempotency.md` §"Concurrent duplicates (Layer 2)" atomic-claim MUST, which is unconditional and had no witness of any kind. Asserts every executor mints the SAME `logicalInvocationId` **before** asserting `delivered === 1` — without the identity check a host passes by minting different ids and never colliding, one effect because nothing raced. Not profile-gated and so not opt-out-able (the obligation is unconditional); an unmounted seam records `blocked`, which is not certifiable. Graduates `layer2-invocation-claim-atomic` reference-impl -> protocol.
|
|
141
141
|
- 2026-08-19 (suite `1.137.0 → 1.138.0`): NEW `durability-poison-exhaustion.test.ts` — RFC 0158 §C.8, the FIRST row of that RFC's conformance table to land. Asserts what `failure-path.test.ts` cannot: not just that deterministically failing work reaches terminal, but that attempts STOP — counted on the log, re-counted after a scaled quiet window, asserted unchanged. A host still redelivering records more. Seam-gated on the existing event-log seam (`blocked` = unobservable, not unmet) and outside every profile floor.
|
|
@@ -481,7 +481,7 @@ Server-required (added in 1.7.0):
|
|
|
481
481
|
| ------------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
482
482
|
| **Redaction** | [`capabilities.md`](../spec/v1/capabilities.md) §"Secrets" + NFR-7 + §"aiProviders" | Vendor-neutral assertions that the server doesn't leak secret material. Three scenario groups: (a) discovery shape contract — `secrets` + `aiProviders` advertisements are well-formed regardless of `secrets.supported`; when `supported === true`, scopes MUST be non-empty + `resolution === 'host-managed'`; `byok ⊆ supported`. (b) bearer-token redaction — invalid Bearer canary in `Authorization` header is not echoed in the 401 response body. (c) credentialRef echo control — gated on `secrets.supported === true`; canary planted in `configurable.ai.credentialRef` MUST NOT appear in any RunEvent payload (poll-based capture; transport-agnostic). Uses runtime-built canary fixtures (`lib/canaries.ts`) that defeat static secret scanners. 6 scenarios. |
|
|
483
483
|
|
|
484
|
-
Current source tree:
|
|
484
|
+
Current source tree: 571 scenario files. Use [`coverage.md`](./coverage.md) for current grade/gap tracking.
|
|
485
485
|
|
|
486
486
|
## Remaining Gaps
|
|
487
487
|
|
package/coverage.md
CHANGED
|
@@ -37,7 +37,7 @@
|
|
|
37
37
|
| Per-workflow `configurableSchema` | `configurable-schema.test.ts` | A− | Negative validation + positive accepted-overlay + `GET /v1/workflows/{id}` schema-surface assertion all covered (CF-7 close-out, 2026-05-15). Grade moves C+ → A−. |
|
|
38
38
|
| Append-reducer ordering | `append-ordering.test.ts` | B | Intra-engine sequence-order check; remaining: cross-engine ordering under a multi-engine fixture. |
|
|
39
39
|
| Webhook signature algorithms | `webhook-sig-algorithm.test.ts`, `webhook-signed-delivery.test.ts`, `webhook-negative.test.ts`, `webhook-receiver-adversarial.test.ts` | A | Forward direction: discovery shape + end-to-end signed delivery with HMAC verification (`webhook-signed-delivery`); negative paths (SSRF guard / validation / unknown-unregister via `webhook-negative`). Reverse direction (CF-5 close-out 2026-05-15): `webhook-receiver-adversarial.test.ts` covers 6 paths — positive control + tampered body + tampered HMAC + stale timestamp + replayed signature + wrong algorithm + malformed signature header. Reference receiver implementation at `conformance/src/lib/webhook-receiver.ts` mirrors the SDK's `verifyWebhookSignature` helper. |
|
|
40
|
-
| Audit-log integrity profile | `audit-log-integrity.test.ts` | A | Profile claim + `/v1/audit/verify` shape; checkpoint-signature verification under the advertised key (`audit-checkpoint-signature.test.ts`, RFC 0218) and the pinned preimage vectors (`audit-checkpoint-vectors.test.ts`); chain re-walk with `chainValid` and `checkpointsValid` bits. Tamper detection covered host-internally at `openwop-examples:examples/hosts/sqlite/test/audit-tamper.test.ts` (mutate-entry + forge-signature paths). CF-11 close-out 2026-05-15: cross-host checkpoint export via `openwop-examples:examples/hosts/postgres/src/audit-export.ts` + standalone verifier at `scripts/verify-audit-checkpoints.mjs`; `openwop-examples:examples/hosts/postgres/test/audit-checkpoint-export.test.ts` verifies 7 paths (positive + tampered-signature + non-monotonic-atSequence). The standalone verifier is additionally regression-guarded in `openwop:check` (step 7) against the committed `conformance/audit-export-samples/{valid,tampered}.json` bundles — accept-valid (exit 0) + reject-tampered (exit 1) — so a verifier regression is caught at the spec gate, not only in the postgres host's own test. |
|
|
40
|
+
| Audit-log integrity profile | `audit-log-integrity.test.ts` | A | Profile claim + `/v1/audit/verify` shape; checkpoint-signature verification under the advertised key (`audit-checkpoint-signature.test.ts`, RFC 0218) and the pinned preimage vectors (`audit-checkpoint-vectors.test.ts`); chain re-walk with `chainValid` and `checkpointsValid` bits; the anomaly shape per `kind` (RFC 0218 §C) server-free in `audit-anomaly-shape.test.ts`. Tamper detection covered host-internally at `openwop-examples:examples/hosts/sqlite/test/audit-tamper.test.ts` (mutate-entry + forge-signature paths). CF-11 close-out 2026-05-15: cross-host checkpoint export via `openwop-examples:examples/hosts/postgres/src/audit-export.ts` + standalone verifier at `scripts/verify-audit-checkpoints.mjs`; `openwop-examples:examples/hosts/postgres/test/audit-checkpoint-export.test.ts` verifies 7 paths (positive + tampered-signature + non-monotonic-atSequence). The standalone verifier is additionally regression-guarded in `openwop:check` (step 7) against the committed `conformance/audit-export-samples/{valid,tampered}.json` bundles — accept-valid (exit 0) + reject-tampered (exit 1) — so a verifier regression is caught at the spec gate, not only in the postgres host's own test. |
|
|
41
41
|
| Multi-region idempotency capability | `multi-region-idempotency.test.ts` | C | Discovery enum coverage; remaining: cross-region partition simulation (requires multi-region harness). |
|
|
42
42
|
| Public hosted registry (`packs.openwop.dev`) | `registry-public.test.ts` | A− | Discovery, index, and per-pack manifest assertions against the public registry. Opt-in via `OPENWOP_TEST_PUBLIC_REGISTRY=true` so default conformance runs don't depend on outbound `packs.openwop.dev` reachability. Remaining: tarball-fetch + signature-verify roundtrip. |
|
|
43
43
|
| Workflow-chain packs (RFC 0013 — `spec/v1/workflow-chain-packs.md`) | `workflow-chain-pack-manifest-validation.test.ts`, `workflow-chain-pack-signature-verification.test.ts`, `workflow-chain-expansion.test.ts`, `workflow-chain-unresolvable-typeid.test.ts`, `workflow-chain-host-expansion.test.ts` | A (4 server-free + live-host harness-witnessed by openwop-app #1298, 6/6) | RFC 0013 promoted Draft → Active → Accepted 2026-05-18 on the 4 server-free scenarios (pure-library algorithm). **Erratum (2026-07-05):** the live-host leg (`workflow-chain-host-expansion.test.ts`, `POST /v1/host/sample/workflow-chain:expand`, 6 cases) actually **soft-skips against every host** — no reference host serves the seam and the `vendor.openwop.workflow-chain-sample` fixture was never bundled here — so it is re-gated onto the OPTIONAL `capabilities.workflowChainPacks.hostExpansionSeam` sub-flag (a test-harness advertisement), decoupled from the semantic `supported` claim so it no longer blocks the RFC 0124 `deferredParameters` flip. **A-lite (2026-07-05, suite `1.52.0`):** the `vendor.openwop.workflow-chain-sample` pack is now bundled at `fixtures/pack-manifests/workflow-chain-sample.pack.json` and the scenario **loads** it + derives expected output from the `expandChain()` reference library (no hardcode) — and a host wiring the `/…:expand` route + advertising `hostExpansionSeam` witnesses it. **Witnessed 2026-07-05:** openwop-app (#1298) serves the seam-gated route (404 prod, RFC 0117 posture) resolving the bundled `@1.52.0` fixture through its `expandChain()` → the 6 positive+negative legs run **6/6 non-vacuously** under `OPENWOP_REQUIRE_BEHAVIOR`, checked against the reference library via the host's returned `expansionId` — harness-tier (in-process, test-only seam; RFC 0035/0118 evidence tier). Closes RFC 0013's Phase-3 host-expansion acceptance item. RFC 0124's deferred path is separately covered by `workflow-chain-deferred-parameters.test.ts` (openwop-app `/chain/deferred-expand` seam #1292, gated legs 12/0). The matching spec doc `workflow-chain-packs.md` remains at `DRAFT v1.x` pending Phase B/C closure (parameter schema validation + cross-host expansion equivalence). |
|
package/dist/cli.js
CHANGED
|
@@ -45,7 +45,7 @@ import { readLedgerFile } from './lib/requirement-ledger.js';
|
|
|
45
45
|
import { deriveRung, emittedByNewerSuite } from './lib/durability-evidence.js';
|
|
46
46
|
import { deriveRequirementDispositions } from './lib/scenario-disposition.js';
|
|
47
47
|
import { scrubEvidence, evidenceSecretsFromEnv, verifyBundleV2 } from './lib/certification-bundle-verify.js';
|
|
48
|
-
import { publicKeyFromPrivate,
|
|
48
|
+
import { publicKeyFromPrivate, sealBundleV3, verifierSign, verifyBundleV3 } from './lib/certification-bundle-v3.js';
|
|
49
49
|
import { canonicalJSON, parseIJson } from './lib/jcs.js';
|
|
50
50
|
import { deriveProfiles, isCoreStandard, agentPlatformStatus, DEPRECATED_PROFILE_ALIASES, PROFILE_FLOOR_SCENARIOS, } from './lib/profiles.js';
|
|
51
51
|
import { setV2ProfileFloors, v2ProfileFloorFiles } from './lib/requirement-registry.js';
|
|
@@ -687,7 +687,7 @@ async function runCertify(args, baseUrl, apiKey) {
|
|
|
687
687
|
if (rows3.some((r) => r.id.startsWith('openwop.requirement.0158.') && r.result === 'executed-pass' && r.id !== 'openwop.requirement.0158.poison-exhaustion')) {
|
|
688
688
|
process.stderr.write(`openwop-conformance --certify: RFC 0158 rung — ${rung3.rung ?? 'NONE'} (${rung3.why})\n`);
|
|
689
689
|
}
|
|
690
|
-
const
|
|
690
|
+
const draft = {
|
|
691
691
|
bundleVersion: '3',
|
|
692
692
|
generatedAt: new Date().toISOString(),
|
|
693
693
|
suite: { name: '@openwop/openwop-conformance', version, targetMajor: target.major, specArtifactsVersion: lock?.version ?? 'repo-layout', ...(lock ? { stampSha256: lock.stampSha256 } : {}) },
|
|
@@ -699,7 +699,6 @@ async function runCertify(args, baseUrl, apiKey) {
|
|
|
699
699
|
discovery: { url: discoveryUrl, sha256, protocolVersions, preferredVersion, document },
|
|
700
700
|
claimedProfiles: claimed3,
|
|
701
701
|
results: { totals: totals3, requirements: rows3 },
|
|
702
|
-
witnessSha256: witnessDigest(rows3, relaxations, args.colocatedCompanion ? 'colocated-companion' : undefined),
|
|
703
702
|
assertionCount: rows3.reduce((n, r) => n + (r.assertions ?? 0), 0),
|
|
704
703
|
...(nonPass.length ? { detail: { nonPass: nonPass.map((r) => ({ id: r.id, result: r.result, reason: r.detail ?? '' })) } } : {}),
|
|
705
704
|
// RFC 0158 §D: claimed ONLY when these rows support it. The verifier
|
|
@@ -707,8 +706,18 @@ async function runCertify(args, baseUrl, apiKey) {
|
|
|
707
706
|
// claimed more would be writing a bundle its own `--verify` rejects.
|
|
708
707
|
...(rung3.rung === null ? {} : { durability: { rung: rung3.rung } }),
|
|
709
708
|
};
|
|
710
|
-
|
|
711
|
-
|
|
709
|
+
// The published keyId is never a secret, whichever variable carried it
|
|
710
|
+
// (evidenceSecretsFromEnv: the `_ID` exclusion is the rule, `except` is
|
|
711
|
+
// the belt for a host that names its env var differently). Scrubbed BEFORE
|
|
712
|
+
// the digest and the signature (sealBundleV3), so the signed rows are the
|
|
713
|
+
// written rows.
|
|
714
|
+
const secrets3 = evidenceSecretsFromEnv(process.env, [apiKey, signingKeyPem], [keyId]);
|
|
715
|
+
const sealed = sealBundleV3(draft, secrets3, signingKeyPem, keyId);
|
|
716
|
+
const { unsigned } = sealed;
|
|
717
|
+
const v3Out = sealed.bundle;
|
|
718
|
+
if (sealed.redactedAt.length > 0) {
|
|
719
|
+
process.stderr.write(`openwop-conformance --certify: REDACTED ${sealed.redactedAt.length} evidence field(s) that carried a configured secret or the conformance canary: ${sealed.redactedAt.slice(0, 8).join(', ')}${sealed.redactedAt.length > 8 ? ', …' : ''}\n`);
|
|
720
|
+
}
|
|
712
721
|
if (args.evidenceTier === 'independent') {
|
|
713
722
|
const vk = args.verifierKeyPath ? readFileSync(args.verifierKeyPath, 'utf8') : process.env['OPENWOP_BUNDLE_VERIFIER_KEY'];
|
|
714
723
|
const vkId = args.verifierKeyId ?? process.env['OPENWOP_BUNDLE_VERIFIER_KEY_ID'];
|
|
@@ -716,14 +725,8 @@ async function runCertify(args, baseUrl, apiKey) {
|
|
|
716
725
|
process.stderr.write('openwop-conformance --certify: --evidence-tier independent needs --verifier-key and --verifier-key-id (RFC 0168 §E.2)\n');
|
|
717
726
|
process.exit(2);
|
|
718
727
|
}
|
|
719
|
-
|
|
728
|
+
v3Out.verifierSignature = verifierSign(unsigned, vk, vkId);
|
|
720
729
|
}
|
|
721
|
-
// The published keyId is never a secret, whichever variable carried it
|
|
722
|
-
// (evidenceSecretsFromEnv: the `_ID` exclusion is the rule, `except` is
|
|
723
|
-
// the belt for a host that names its env var differently).
|
|
724
|
-
const secrets3 = evidenceSecretsFromEnv(process.env, [apiKey, signingKeyPem], [keyId]);
|
|
725
|
-
const scrubbed3 = scrubEvidence(v3, secrets3);
|
|
726
|
-
const v3Out = scrubbed3.value;
|
|
727
730
|
const audit3 = verifyBundleV3(v3Out, { hostPublicKeyPem: publicKeyFromPrivate(signingKeyPem) });
|
|
728
731
|
// `relaxed-profile-certified` is no longer excused: the emitter now computes
|
|
729
732
|
// the relaxation into `certified`, so if the verifier still finds that
|
|
@@ -22,6 +22,7 @@ import { profileDerivable } from './profiles.js';
|
|
|
22
22
|
import { checkRungClaim } from './durability-evidence.js';
|
|
23
23
|
import { profilesDeniedByObservedRelaxation, profilesRelaxedBy, v2RegistryAvailable } from './v2-profiles.js';
|
|
24
24
|
import { canonicalJSON, codeUnitCompare, JcsRefusal } from './jcs.js';
|
|
25
|
+
import { scrubEvidence } from './certification-bundle-verify.js';
|
|
25
26
|
export const SIGNATURE_OVER = ['witnessSha256', 'host.build', 'suite.version', 'discovery.sha256'];
|
|
26
27
|
/**
|
|
27
28
|
* RFC 0212 §A — the bytes every attestation and digest here covers are RFC 8785
|
|
@@ -82,6 +83,24 @@ export function verifierSign(bundle, privateKeyPem, keyId) {
|
|
|
82
83
|
const key = createPrivateKey(privateKeyPem);
|
|
83
84
|
return { alg: 'ed25519', keyId, sig: toBase64url(edSign(null, attestationPayload(bundle), key)) };
|
|
84
85
|
}
|
|
86
|
+
/**
|
|
87
|
+
* Scrub, THEN digest, THEN sign (RFC 0148 §C + RFC 0168 §E.2).
|
|
88
|
+
*
|
|
89
|
+
* Until 2.43.1 the emitter computed `witnessSha256` over the raw rows, signed,
|
|
90
|
+
* and only then ran `scrubEvidence` over the finished bundle. A secret found in
|
|
91
|
+
* a row's `detail` or `evidence` was redacted AFTER the digest was taken, so
|
|
92
|
+
* the written rows no longer hashed to `witnessSha256` and the emitter's own
|
|
93
|
+
* self-verification refused the bundle (`[witness-digest]`, exit 2, no bundle)
|
|
94
|
+
* — openwop-app at major 1 on suite 2.43.0, 2026-09-28. The redaction was
|
|
95
|
+
* right and the check was right; the order was wrong. Everything the digest and
|
|
96
|
+
* the attestation cover is now scrubbed before either is computed, so the
|
|
97
|
+
* signed bytes are the shipped bytes.
|
|
98
|
+
*/
|
|
99
|
+
export function sealBundleV3(draft, secrets, privateKeyPem, keyId) {
|
|
100
|
+
const { value: scrubbed, redactedAt } = scrubEvidence(draft, secrets);
|
|
101
|
+
const unsigned = { ...scrubbed, witnessSha256: witnessDigest(scrubbed.results.requirements, scrubbed.host.relaxations, scrubbed.host.deployment) };
|
|
102
|
+
return { unsigned, bundle: { ...unsigned, signature: signBundleV3(unsigned, privateKeyPem, keyId) }, redactedAt };
|
|
103
|
+
}
|
|
85
104
|
export function publicKeyFromPrivate(privateKeyPem) {
|
|
86
105
|
return createPublicKey(createPrivateKey(privateKeyPem)).export({ type: 'spki', format: 'pem' });
|
|
87
106
|
}
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
{
|
|
2
|
+
"id": "conformance-approval-reject-loopback",
|
|
3
|
+
"name": "Conformance: Approval (reject loops back)",
|
|
4
|
+
"version": "1.0",
|
|
5
|
+
"description": "openwop#1697. A rejected approval gate routed over an any_failed edge to revise, which loops back to the gate (bounded by settings.maxLoopbackIterations 2). Rejected once, the gate is visited again: the host MUST raise a new interrupt.requested with a different key and wait for a new decision, never replaying the first rejection. Conformance scenario: v2-approval-reject-disposition.test.ts.",
|
|
6
|
+
"nodes": [
|
|
7
|
+
{ "id": "start", "typeId": "core.noop", "name": "Start", "position": { "x": 0, "y": 0 }, "config": {}, "inputs": {} },
|
|
8
|
+
{
|
|
9
|
+
"id": "gate",
|
|
10
|
+
"typeId": "core.approvalGate",
|
|
11
|
+
"name": "Approval Gate",
|
|
12
|
+
"position": { "x": 200, "y": 0 },
|
|
13
|
+
"config": {
|
|
14
|
+
"title": "Conformance approval (reject loops back)",
|
|
15
|
+
"description": "Conformance suite — reject once; the gate is asked again.",
|
|
16
|
+
"actions": ["accept", "reject"]
|
|
17
|
+
},
|
|
18
|
+
"inputs": {}
|
|
19
|
+
},
|
|
20
|
+
{ "id": "revise", "typeId": "core.noop", "name": "Revise", "position": { "x": 400, "y": 0 }, "config": {}, "inputs": {} }
|
|
21
|
+
],
|
|
22
|
+
"edges": [
|
|
23
|
+
{ "id": "start-gate", "sourceNodeId": "start", "targetNodeId": "gate", "triggerRule": "any_success" },
|
|
24
|
+
{ "id": "gate-revise", "sourceNodeId": "gate", "targetNodeId": "revise", "triggerRule": "any_failed" },
|
|
25
|
+
{ "id": "revise-gate", "sourceNodeId": "revise", "targetNodeId": "gate", "triggerRule": "any_success" }
|
|
26
|
+
],
|
|
27
|
+
"triggers": [
|
|
28
|
+
{ "id": "manual", "type": "manual", "enabled": true }
|
|
29
|
+
],
|
|
30
|
+
"variables": [],
|
|
31
|
+
"metadata": { "tags": ["conformance", "hitl"] },
|
|
32
|
+
"settings": { "timeout": 0, "maxLoopbackIterations": 2 }
|
|
33
|
+
}
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
{
|
|
2
|
+
"id": "conformance-approval-reject-routed",
|
|
3
|
+
"name": "Conformance: Approval (reject routed)",
|
|
4
|
+
"version": "1.0",
|
|
5
|
+
"description": "RFC 0223 G1. An approval gate with two outgoing edges: an any_failed edge to on-reject and an all_success edge to on-accept. Rejected, the gate fails with approval_rejected, on-reject runs, on-accept never runs, and the run completes. Conformance scenario: v2-approval-reject-disposition.test.ts.",
|
|
6
|
+
"nodes": [
|
|
7
|
+
{
|
|
8
|
+
"id": "gate",
|
|
9
|
+
"typeId": "core.approvalGate",
|
|
10
|
+
"name": "Approval Gate",
|
|
11
|
+
"position": { "x": 0, "y": 0 },
|
|
12
|
+
"config": {
|
|
13
|
+
"title": "Conformance approval (reject routed)",
|
|
14
|
+
"description": "Conformance suite — reject to take the any_failed route.",
|
|
15
|
+
"actions": ["accept", "reject"]
|
|
16
|
+
},
|
|
17
|
+
"inputs": {}
|
|
18
|
+
},
|
|
19
|
+
{ "id": "on-reject", "typeId": "core.noop", "name": "On reject", "position": { "x": 200, "y": -80 }, "config": {}, "inputs": {} },
|
|
20
|
+
{ "id": "on-accept", "typeId": "core.noop", "name": "On accept", "position": { "x": 200, "y": 80 }, "config": {}, "inputs": {} }
|
|
21
|
+
],
|
|
22
|
+
"edges": [
|
|
23
|
+
{ "id": "gate-on-reject", "sourceNodeId": "gate", "targetNodeId": "on-reject", "triggerRule": "any_failed" },
|
|
24
|
+
{ "id": "gate-on-accept", "sourceNodeId": "gate", "targetNodeId": "on-accept", "triggerRule": "all_success" }
|
|
25
|
+
],
|
|
26
|
+
"triggers": [
|
|
27
|
+
{ "id": "manual", "type": "manual", "enabled": true }
|
|
28
|
+
],
|
|
29
|
+
"variables": [],
|
|
30
|
+
"metadata": { "tags": ["conformance", "hitl"] },
|
|
31
|
+
"settings": { "timeout": 0 }
|
|
32
|
+
}
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
{
|
|
2
|
+
"id": "conformance-approval-timeout-approve",
|
|
3
|
+
"name": "Conformance: Approval (timeout, onTimeout approve)",
|
|
4
|
+
"version": "1.0",
|
|
5
|
+
"description": "RFC 0223 / openwop#1696. An approval gate with a 1500 ms timeoutMs and onTimeout: approve. Nobody resolves it: a timeout never grants, so the host's timer resolves the gate rejected (action timeout, decision rejected, reason timeout), the gate fails with approval_rejected and the run fails. Conformance scenario: v2-approval-reject-disposition.test.ts.",
|
|
6
|
+
"nodes": [
|
|
7
|
+
{
|
|
8
|
+
"id": "gate",
|
|
9
|
+
"typeId": "core.approvalGate",
|
|
10
|
+
"name": "Approval Gate",
|
|
11
|
+
"position": {
|
|
12
|
+
"x": 0,
|
|
13
|
+
"y": 0
|
|
14
|
+
},
|
|
15
|
+
"config": {
|
|
16
|
+
"title": "Conformance approval (timeout, onTimeout approve)",
|
|
17
|
+
"description": "Conformance suite — do not resolve; the gate times out and MUST NOT be granted.",
|
|
18
|
+
"actions": [
|
|
19
|
+
"accept",
|
|
20
|
+
"reject"
|
|
21
|
+
],
|
|
22
|
+
"timeoutMs": 1500,
|
|
23
|
+
"onTimeout": "approve"
|
|
24
|
+
},
|
|
25
|
+
"inputs": {}
|
|
26
|
+
}
|
|
27
|
+
],
|
|
28
|
+
"edges": [],
|
|
29
|
+
"triggers": [
|
|
30
|
+
{
|
|
31
|
+
"id": "manual",
|
|
32
|
+
"type": "manual",
|
|
33
|
+
"enabled": true
|
|
34
|
+
}
|
|
35
|
+
],
|
|
36
|
+
"variables": [],
|
|
37
|
+
"metadata": {
|
|
38
|
+
"tags": [
|
|
39
|
+
"conformance",
|
|
40
|
+
"hitl"
|
|
41
|
+
]
|
|
42
|
+
},
|
|
43
|
+
"settings": {
|
|
44
|
+
"timeout": 0
|
|
45
|
+
}
|
|
46
|
+
}
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
{
|
|
2
|
+
"id": "conformance-approval-timeout",
|
|
3
|
+
"name": "Conformance: Approval (timeout)",
|
|
4
|
+
"version": "1.0",
|
|
5
|
+
"description": "RFC 0223 G2. An approval gate with a 1500 ms timeoutMs and no onTimeout. Nobody resolves it: the host's timer resolves the gate rejected (action timeout, decision rejected, reason timeout), the gate fails with approval_rejected and the run fails naming it. Conformance scenario: v2-approval-reject-disposition.test.ts.",
|
|
6
|
+
"nodes": [
|
|
7
|
+
{
|
|
8
|
+
"id": "gate",
|
|
9
|
+
"typeId": "core.approvalGate",
|
|
10
|
+
"name": "Approval Gate",
|
|
11
|
+
"position": { "x": 0, "y": 0 },
|
|
12
|
+
"config": {
|
|
13
|
+
"title": "Conformance approval (timeout)",
|
|
14
|
+
"description": "Conformance suite — do not resolve; the gate times out.",
|
|
15
|
+
"actions": ["accept", "reject"],
|
|
16
|
+
"timeoutMs": 1500
|
|
17
|
+
},
|
|
18
|
+
"inputs": {}
|
|
19
|
+
}
|
|
20
|
+
],
|
|
21
|
+
"edges": [],
|
|
22
|
+
"triggers": [
|
|
23
|
+
{ "id": "manual", "type": "manual", "enabled": true }
|
|
24
|
+
],
|
|
25
|
+
"variables": [],
|
|
26
|
+
"metadata": { "tags": ["conformance", "hitl"] },
|
|
27
|
+
"settings": { "timeout": 0 }
|
|
28
|
+
}
|
package/fixtures.md
CHANGED
|
@@ -49,6 +49,10 @@ All fixtures MUST advertise:
|
|
|
49
49
|
| Approval | `conformance-approval` | Verifies HITL approval interrupt + resume | `completed` after resolve | unbounded (suspends) |
|
|
50
50
|
| Approval (refine) | `conformance-approval-refine` | RFC 0183 — refine resolution carries action + refineFeedback |
|
|
51
51
|
| Approval (edit-accept) | `conformance-approval-edit-accept` | RFC 0183 — edit-accept resolution carries action + editedArtifactData |
|
|
52
|
+
| Approval (reject routed) | `conformance-approval-reject-routed` | RFC 0223 G1 — a rejected gate is routed over an `any_failed` edge; its `all_success` sibling never runs | `completed` after reject | unbounded (suspends) |
|
|
53
|
+
| Approval (timeout) | `conformance-approval-timeout` | RFC 0223 G2 — a 1500 ms `timeoutMs`, no `onTimeout`: the host's timer resolves the gate rejected | `failed` (`approval_rejected`) after the timeout | ~1.5 s + the host's sweep |
|
|
54
|
+
| Approval (timeout, onTimeout approve) | `conformance-approval-timeout-approve` | openwop#1696 — a 1500 ms `timeoutMs` with `onTimeout: approve`: a timeout never grants, so the gate still resolves rejected | `failed` (`approval_rejected`) after the timeout | ~1.5 s + the host's sweep |
|
|
55
|
+
| Approval (reject loops back) | `conformance-approval-reject-loopback` | openwop#1697 — a rejected gate routed back to itself through `revise` is asked again under a new `key` | `waiting-approval` again after one reject (the suite cancels) | unbounded (suspends) |
|
|
52
56
|
| Clarification | `conformance-clarification` | Verifies HITL clarification interrupt + resume | `completed` after resolve | unbounded (suspends) |
|
|
53
57
|
| Clarification (nested schema) | `conformance-clarification-nested` | RFC 0199 §D.2(d) probe — an answer schema an MCP mount MUST NOT bridge in form mode | `completed` after resolve | unbounded (suspends) |
|
|
54
58
|
| Clarification (sensitive field) | `conformance-clarification-sensitive` | RFC 0199 §D.2(d) probe — `format: "password"`; form mode MUST be refused | `completed` after resolve | unbounded (suspends) |
|
|
@@ -61,7 +65,7 @@ All fixtures MUST advertise:
|
|
|
61
65
|
| Prompt End-to-End | `conformance-prompt-end-to-end` | RFC 0027 + RFC 0029 end-to-end. Single `mock-ai` node with `config.systemPromptRef` set; host MUST emit `agent.promptResolved` + `prompt.composed` events during dispatch, then complete. Capability-gated on `capabilities.prompts.supported`. | `completed` | ≤ 10s |
|
|
62
66
|
| Prompt All Four Kinds | `conformance-prompt-all-four-kinds` | RFC 0027 §A four-kind dispatch coverage with a MULTI-ENTRY `fewShotPromptRefs[]` array. Single `mock-ai` node with one ref per singular-kind slot (`systemPromptRef`, `userPromptRef`, `schemaHintPromptRef`) + two distinct templateIds in `fewShotPromptRefs[]`; host MUST emit 5 `agent.promptResolved` events (one per slot) AND 5 `prompt.composed` events. Multi-entry few-shot is the regression pin for `fewShotPromptRefs[slotIndex]` per-index resolution — a host that hard-codes `[0]` would emit the same template twice in the few-shot events and fail the per-templateId assertion. Capability-gated on `capabilities.prompts.supported`. | `completed` | ≤ 10s |
|
|
63
67
|
| Dispatch Loop | `conformance-dispatch-loop` | Verifies `core.dispatch` loop mechanism | `completed` | ≤ 30s |
|
|
64
|
-
| Interrupt — Quorum | `conformance-interrupt-quorum` | Verifies `openwop-interrupt-quorum` profile (multi-approver, majority rejection) | `completed` after 3 accepts, `failed` after quorum reject
|
|
68
|
+
| Interrupt — Quorum | `conformance-interrupt-quorum` | Verifies `openwop-interrupt-quorum` profile (multi-approver, majority rejection) | `completed` after 3 accepts, `failed` after quorum reject (major 2: `approval_rejected`) | unbounded (suspends) |
|
|
65
69
|
| Interrupt — External Event | `conformance-interrupt-external-event` | Verifies `openwop-interrupt-external-event` profile (correlation-matched callback) | `completed` after matching POST, `failed` on timeout | ≤ 60s (timeoutMs configured) |
|
|
66
70
|
| Interrupt — Auth Required | `conformance-interrupt-auth-required` | Verifies `openwop-interrupt-auth-required` profile (bearer-token resume only) | `completed` after bearer resolve | unbounded (suspends) |
|
|
67
71
|
| Interrupt — Parent/Child Cancel | `conformance-interrupt-parent-child-cancel` + `conformance-interrupt-parent-child-cancel-child` | Verifies `openwop-interrupt-parent-child` cancel cascade | `cancelled` (both runs) | ≤ 30s |
|
|
@@ -182,6 +186,7 @@ The `messages`-mode stream fixture (AI token streaming) is covered by the determ
|
|
|
182
186
|
4. After client POSTs `{action: 'accept'}` to `/v1/runs/{runId}/interrupt`, server emits `approval.received` and resumes.
|
|
183
187
|
5. Run reaches `completed`.
|
|
184
188
|
- **Terminal status (after accept)**: `completed`.
|
|
189
|
+
- **Terminal status (after reject, major 2)**: `failed`, with `error.code` `approval_rejected` on the snapshot and on `run.failed`, and `failedNodeId: "gate"`. The fixture has no edges, so nothing routes the failure (`spec/v2/core/interrupt.md` §Rejection, RFC 0223; `v2-approval-reject-disposition.test.ts`).
|
|
185
190
|
- **Resolve schema**: `{action: "accept" | "reject"}`. Server MUST reject any other shape with 400.
|
|
186
191
|
|
|
187
192
|
### `conformance-approval-refine`
|
|
@@ -210,6 +215,54 @@ The `messages`-mode stream fixture (AI token streaming) is covered by the determ
|
|
|
210
215
|
5. An `edit-accept` resolution supplying no `editedArtifactData` MUST be refused.
|
|
211
216
|
- **Why it is separate**: same reason as `conformance-approval-refine` — widening a registered fixture's `actions` forces every host to re-register a test-only change.
|
|
212
217
|
|
|
218
|
+
### `conformance-approval-reject-routed`
|
|
219
|
+
|
|
220
|
+
- **Purpose**: witness that a rejected approval gate is a failed source a workflow can route (RFC 0223 §A, gap G1; `spec/v2/core/interrupt.md` §Rejection).
|
|
221
|
+
- **Inputs**: none.
|
|
222
|
+
- **Behavior**:
|
|
223
|
+
1. Run starts and reaches the `gate` (`core.approvalGate`, actions `accept | reject`).
|
|
224
|
+
2. Run status MUST be `waiting-approval`.
|
|
225
|
+
3. Client POSTs `{action: 'reject'}` to the interrupt.
|
|
226
|
+
4. `gate` MUST emit `node.failed` with `error.code` `approval_rejected` and `retryable: false`.
|
|
227
|
+
5. `on-reject` (incoming edge `triggerRule: any_failed`) MUST run to `node.completed`.
|
|
228
|
+
6. `on-accept` (incoming edge `triggerRule: all_success`) MUST NOT run: no `node.started` or `node.completed` for it. A host MAY record `node.skipped`; the spec does not require it.
|
|
229
|
+
7. Run reaches `completed`.
|
|
230
|
+
- **Terminal status (after reject)**: `completed`.
|
|
231
|
+
- **Why it is separate from `conformance-approval`**: that fixture has no edges, and giving it some would force every host to re-register a registered definition.
|
|
232
|
+
|
|
233
|
+
### `conformance-approval-timeout`
|
|
234
|
+
|
|
235
|
+
- **Purpose**: witness the timeout disposition (RFC 0223 §A, gap G2).
|
|
236
|
+
- **Inputs**: none.
|
|
237
|
+
- **Behavior**:
|
|
238
|
+
1. Run starts and reaches the `gate` (`core.approvalGate`, `config.timeoutMs: 1500`, no `onTimeout`). The host carries the node's `timeoutMs` onto the interrupt payload.
|
|
239
|
+
2. Run status MUST be `waiting-approval`.
|
|
240
|
+
3. Nobody resolves it. After the deadline, the host MUST resolve the gate itself: `interrupt.resolved` with `action: "timeout"`, `decision: "rejected"`, `reason: "timeout"`.
|
|
241
|
+
4. `gate` MUST emit `node.failed` with `error.code` `approval_rejected`, and `run.failed` MUST carry `approval_rejected` with `failedNodeId: "gate"`.
|
|
242
|
+
- **Terminal status**: `failed`.
|
|
243
|
+
- **Timing**: the suite waits up to 20 s (scaled) for the terminal state; a host's timer granularity is its own.
|
|
244
|
+
|
|
245
|
+
### `conformance-approval-timeout-approve`
|
|
246
|
+
|
|
247
|
+
- **Purpose**: witness that a timeout never grants a gate, whatever `onTimeout` holds (`spec/v2/core/interrupt.md` §Rejection; RFC 0223 §C as corrected by openwop#1696).
|
|
248
|
+
- **Inputs**: none.
|
|
249
|
+
- **Behavior**: as `conformance-approval-timeout`, with `config.onTimeout: "approve"`. After the deadline, the host MUST resolve the gate rejected (`action: "timeout"`, `decision: "rejected"`, `reason: "timeout"`), `gate` MUST fail with `approval_rejected`, and the run MUST NOT complete.
|
|
250
|
+
- **Terminal status**: `failed`.
|
|
251
|
+
- **Why it is separate**: `conformance-approval-timeout` witnesses the absent case; this one witnesses the value that used to read as permission to fail open.
|
|
252
|
+
|
|
253
|
+
### `conformance-approval-reject-loopback`
|
|
254
|
+
|
|
255
|
+
- **Purpose**: witness that `key` is per visit (`spec/v2/core/interrupt.md` §Re-entry and resume values, openwop#1697).
|
|
256
|
+
- **Inputs**: none.
|
|
257
|
+
- **Graph**: `start` (`core.noop`) → `gate` (`any_success`); `gate` → `revise` (`core.noop`, `any_failed`); `revise` → `gate` (`any_success`). `settings.maxLoopbackIterations: 2`.
|
|
258
|
+
- **Behavior**:
|
|
259
|
+
1. Run reaches `gate` and MUST be `waiting-approval`.
|
|
260
|
+
2. Client POSTs `{action: 'reject'}`. `gate` fails with `approval_rejected`, and `revise` runs.
|
|
261
|
+
3. `gate` runs again. The host MUST raise a second `interrupt.requested` for `gate` with a different `key`, and the run MUST be `waiting-approval` again.
|
|
262
|
+
4. The first rejection MUST NOT be replayed into the second visit: only one `interrupt.resolved` is on the log.
|
|
263
|
+
- **Terminal status**: none reached by the suite, which cancels the run.
|
|
264
|
+
- **Advertise only if** the host runs cycles. A host that does not MUST NOT advertise the fixture, and the row is `inapplicable`.
|
|
265
|
+
|
|
213
266
|
### `conformance-clarification`
|
|
214
267
|
|
|
215
268
|
- **Purpose**: verify HITL clarification interrupt + resume.
|
|
@@ -523,7 +576,7 @@ The fixtures reference these typeIds:
|
|
|
523
576
|
| `core.noop` | noop, multi-node, approval, clarification, cancellable | Immediate completion, no output |
|
|
524
577
|
| `core.identity` | identity | Echo `input.payload` to `output.payload` |
|
|
525
578
|
| `core.delay` | delay, cancellable | Sleep `config.delayMs` ms |
|
|
526
|
-
| `core.fail` | failure | Throw with `code: "
|
|
579
|
+
| `core.fail` | failure | Throw with `code: "example.conformance_failure"` (a vendor code under the registered `example` org; errors.md §The registry, openwop#1698), message: "Intentional conformance failure" |
|
|
527
580
|
| `core.approvalGate` | approval | Call `ctx.interrupt({kind: 'approval', ...})` |
|
|
528
581
|
| `core.clarificationGate` | clarification | Call `ctx.interrupt({kind: 'clarification', ...})` |
|
|
529
582
|
| `conformance.requiresMissing` | capability-missing | Declares `requires: ['conformance.never-provided']`; engine MUST refuse dispatch. Opt-in fixture registration is recommended so production deployments don't expose the fixture surface. |
|
|
@@ -587,6 +640,10 @@ conformance/
|
|
|
587
640
|
conformance-approval.json
|
|
588
641
|
conformance-approval-refine.json
|
|
589
642
|
conformance-approval-edit-accept.json
|
|
643
|
+
conformance-approval-reject-routed.json
|
|
644
|
+
conformance-approval-timeout.json
|
|
645
|
+
conformance-approval-timeout-approve.json
|
|
646
|
+
conformance-approval-reject-loopback.json
|
|
590
647
|
conformance-clarification.json
|
|
591
648
|
conformance-multi-node.json
|
|
592
649
|
conformance-idempotent.json
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@openwop/openwop-conformance",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.43.1",
|
|
4
4
|
"description": "Production-ready black-box conformance suite for OpenWOP v1.0 compliant servers.",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -56,6 +56,6 @@
|
|
|
56
56
|
"@openwop/spec-artifacts": "file:../spec-artifacts"
|
|
57
57
|
},
|
|
58
58
|
"peerDependencies": {
|
|
59
|
-
"@openwop/spec-artifacts": "2.
|
|
59
|
+
"@openwop/spec-artifacts": "2.43.1"
|
|
60
60
|
}
|
|
61
61
|
}
|