@ggui-ai/protocol-conformance 0.25.0 → 0.26.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +45 -25
- package/dist/cli.d.ts +4 -2
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +13 -8
- package/dist/dispatch-idempotency-conformance/index.d.ts +78 -0
- package/dist/dispatch-idempotency-conformance/index.d.ts.map +1 -0
- package/dist/dispatch-idempotency-conformance/index.js +173 -0
- package/dist/fixtures/index.d.ts +22 -4
- package/dist/fixtures/index.d.ts.map +1 -1
- package/dist/fixtures/index.js +23 -3
- package/dist/fixtures/reserved-channel-replay/fresh-subscribe-replays-reserved-preview.json +30 -0
- package/dist/fixtures/reserved-channel-replay/index.d.ts +4 -0
- package/dist/fixtures/reserved-channel-replay/index.d.ts.map +1 -0
- package/dist/fixtures/reserved-channel-replay/index.js +18 -0
- package/dist/host-helper-conformance/index.d.ts +144 -2
- package/dist/host-helper-conformance/index.d.ts.map +1 -1
- package/dist/host-helper-conformance/index.js +309 -1
- package/dist/index.d.ts +4 -2
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +4 -2
- package/dist/match-behavior.d.ts.map +1 -1
- package/dist/match-behavior.js +47 -8
- package/dist/n1-compat-conformance/cases/forward-app-theme-carry-uninterpretable.json +21 -0
- package/dist/n1-compat-conformance/cases/forward-ops-list-blueprints-cloned.json +88 -0
- package/dist/n1-compat-conformance/cases/forward-ops-list-blueprints-stamped.json +77 -0
- package/dist/n1-compat-conformance/cases/forward-view-proof-later-version.json +14 -0
- package/dist/n1-compat-conformance/cases/forward-view-proof-root-later-claim.json +14 -0
- package/dist/n1-compat-conformance/cases/release-14-ops-list-blueprints.json +41 -0
- package/dist/n1-compat-conformance/index.d.ts +1 -1
- package/dist/n1-compat-conformance/index.d.ts.map +1 -1
- package/dist/n1-compat-conformance/index.js +128 -22
- package/dist/reporter.d.ts +5 -0
- package/dist/reporter.d.ts.map +1 -1
- package/dist/reporter.js +30 -4
- package/dist/run-conformance.d.ts +37 -1
- package/dist/run-conformance.d.ts.map +1 -1
- package/dist/run-conformance.js +32 -17
- package/dist/types.d.ts +16 -0
- package/dist/types.d.ts.map +1 -1
- package/package.json +8 -3
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# @ggui-ai/protocol-conformance
|
|
2
2
|
|
|
3
|
-
Conformance test kit for the
|
|
3
|
+
Conformance test kit for the GGUI protocol. Ships JSON case catalogs + runners that drive them against a live implementation — the behavioral fixtures over WebSocket, the `resources/read` cases over the MCP binding — and report scorecard-style pass / fail / skip output.
|
|
4
4
|
|
|
5
5
|
This kit is the protocol's conformance surface: it lets a third-party MCP builder prove their implementation is protocol-conformant without reimplementing the test harness.
|
|
6
6
|
|
|
@@ -13,11 +13,13 @@ The kit grades protocol obligations through **four mechanisms**, each matched to
|
|
|
13
13
|
3. **Session-state host read-back** — stateful obligations with no wire response (e.g. `host_context_observed` persisting onto `GguiSession.hostContext`): the runner dispatches the C→S frame, waits out the observation window, then reads the GguiSession field back through `ConformanceHost.readSessionField()` and deep-equals it against the fixture's `expected`. No host (or a host without the method, or a read that throws) → the fixture SKIPS with the reason — a host that cannot read state cannot grade it, and the kit never converts that gap into a pass.
|
|
14
14
|
4. **MCP-binding scenario driver** — request/response obligations on the MCP surface, where the deployment shape is part of what is under test. The `resources/read` catalog is the first: a case declares a server shape, a caller identity and a seed set, the adopter's driver brings a server up that way, and the kit reads locators against it and grades the raw JSON-RPC frames. A driver that cannot express a scenario throws, and the case SKIPS — never passes.
|
|
15
15
|
|
|
16
|
+
**Obligation strength.** A fixture grades a MUST unless it declares `"level": "should"`. An unmet MUST is a failure. An unmet SHOULD is a **warning**: it lands in `result.warned` with the same evidence a failure carries, is printed as `WARN`, and never fails a run. A skip stays a skip at either level.
|
|
17
|
+
|
|
16
18
|
A fifth surface — the **Path-B browser-host driver** for DOM-level claims (`bootstrap-failure`, `props-update`) — is honestly absent: it is not yet packaged, and those fixtures skip wherever the kit runs (see below).
|
|
17
19
|
|
|
18
20
|
## Transport
|
|
19
21
|
|
|
20
|
-
**The behavioral fixture catalog is WebSocket-only.** The canonical
|
|
22
|
+
**The behavioral fixture catalog is WebSocket-only.** The canonical GGUI live-channel transport is WS (see SPEC §12 Transport Bindings). `TransportConfig` is shaped as an extensibly-closed union so later live-channel transports (HTTP long-poll) can be added without breaking the public API.
|
|
21
23
|
|
|
22
24
|
The MCP surface is graded separately, by catalog rather than by transport config — see `resource-read-conformance` below. Its cases are deliberately NOT registered in `fixturesByContract`: everything in that map is driven over a WebSocket, so an MCP case there would be a permanent skip on every WS run, which is the false gate the kit's exact skip-set pinning exists to prevent. `resources/read` and — since ggui#880 — one `tools/call` catalog (`domain-error`, SPEC §7.9 Plane 2) are the MCP methods bound today.
|
|
23
25
|
|
|
@@ -58,12 +60,12 @@ Each catalog ships its cases as raw JSON so a non-TypeScript implementer can gra
|
|
|
58
60
|
- **`@ggui-ai/protocol-conformance/resolution-conformance`** — which bundle + style URLs the server MUST compute for a gadget descriptor's transport fields (`bundleHost` precedence, default host, loopback `http` scheme).
|
|
59
61
|
- **`@ggui-ai/protocol-conformance/binding-conformance`** — which MCP tool-binding set a manifest resolves to (a declared `mcpTools` list wins entirely; blueprints without it derive bare tool-name entries from contract tool names) and which artifacts the registry `tool=` / `server=` search filters MUST match (SPEC §7.7.4.1).
|
|
60
62
|
- **`@ggui-ai/protocol-conformance/props-schema-conformance`** — the schema-precise render arbiter (SPEC §2.3.2): the enforced-props-schema builder (drift), its RFC 8785 sha256 hash, the grammar-safe profile verdict, and returned-schema AUTHORITY (sample validity under the case's own `propsSchema`, including the 2026-08-19 out-of-vocabulary enum incident pinned as a permanent sample).
|
|
61
|
-
- **`@ggui-ai/protocol-conformance/host-helper-conformance`** — grades a HOST-HELPER library (what mounts a
|
|
63
|
+
- **`@ggui-ai/protocol-conformance/host-helper-conformance`** — grades a HOST-HELPER library (what mounts a GGUI view and answers the MCP Apps bridge) through a JSON-RPC port the implementer supplies: a well-formed `ui/initialize` (H1), truthful capability advertisement (H2: an advertised `serverTools` answers `tools/call`, an advertised `serverResources` answers `resources/read`), honest and bounded refusals (H3/H4), and an unmodified `tools/call` relay (R1/R2), plus optional chrome (C1), theme-coverage (T1) and model-tool-set (M1 — the host's own visibility filter graded against `toolVisibleToModel`: withhold the app-only `ggui_runtime_*` tools, keep every model-visible one; ggui#1414), view-material (V1 — the host's own model-context rules, fed a render result and a render read body whose slice is marked in every value: none may reach the model, raw, percent-encoded or base64-encoded, while the result's own marker must, and no `ui://ggui/render` read may be offered to the model; ggui#1415) and view-binding (L1 — the host's own relay decision: a view's own reads, in every locator form, and every session-naming call a view may make relay, another session's refuse; ggui#1415) grades. M1, V1 and L1 grade the host's rule against fixtures, not a running host: they are self-certified. The scorecard names a tier: `relaying`, `read-only` (a legal grade), or `nonconforming`. Runner: `runHostHelperConformance(port)`.
|
|
62
64
|
- **`@ggui-ai/protocol-conformance/refusal-envelope-conformance`** — the tool result a server MUST project for a PRE-GENERATION refusal (SPEC §7.1's refused arm): an in-result `isError` (never a throw), `content[0].text` leading with the code, `structuredContent` carrying `{outcome: 'refused', refusal}` and nothing else, no `_meta` — and no envelope at all for a registered code whose surfaces exclude the render gate.
|
|
63
65
|
- **`@ggui-ai/protocol-conformance/registry-completeness`** — the structural obligations a deployment's closed refusal-code registry MUST satisfy: `surfaces` non-empty, `code === key`, every `after-fix` entry naming who acts (`fixBy`), and `retry` inside the closed four-value set. Grades DATA rather than a function — the caller passes their registry.
|
|
64
66
|
- **`@ggui-ai/protocol-conformance/transport-refusal-conformance`** — the JSON-RPC error a per-app MCP endpoint MUST answer with when it refuses a request for a typed reason (SPEC §7.1's endpoint-level refusal, ggui#825): HTTP 403 with `{ code: -32003, message: 'App not found', data: { refusal } }`, where `refusal` is the registry projection without the render-only fields (`code`, `message`, `fix`, `retry`) — and no envelope at all for a registered code whose surfaces exclude `mcp-endpoint` (an untyped authorization failure answers 403 with `-32007` and no `data`, by contract — ggui#836). Runner: `runTransportRefusalConformance(project)`; folded into `runConformance()` via `transportRefusalProjector`.
|
|
65
67
|
- **`@ggui-ai/protocol-conformance/theme-binding-conformance`** — the theme-binding total order (ggui#987): which host-announced / stamped / sidecar mode a read door MUST paint, and which overlays it MUST keep.
|
|
66
|
-
- **`@ggui-ai/protocol-conformance/n1-compat-conformance`** — N−1 wire compatibility (VERSION-POLICY §3.6, ggui#1014): BACKWARD cases are the previous release's payloads today's parsers MUST accept;
|
|
68
|
+
- **`@ggui-ai/protocol-conformance/n1-compat-conformance`** — N−1 wire compatibility (VERSION-POLICY §3.6, ggui#1014): BACKWARD cases are the previous release's payloads today's parsers MUST accept; a FORWARD case (ggui#1093) is a later release's payload today's READ door MUST keep, stripping and naming the unknown top-level member. The view-origin proof (ggui#1415) has two forward cases: a later proof version still fits the relay door shape and reads `version_unknown`, never `malformed`, and a v1 proof whose root carries a later claim still parses. Graded on every `runConformance()`.
|
|
67
69
|
|
|
68
70
|
```ts
|
|
69
71
|
import { runSchemaConformance } from "@ggui-ai/protocol-conformance/schema-conformance";
|
|
@@ -149,35 +151,49 @@ Vendor-neutral like `resources/read`: the kit imports no server. The adopter sup
|
|
|
149
151
|
|
|
150
152
|
The catalog exists for one receipt: before the slug led the text, every first-party server failed every case on `slug-leads` (executed, ggui#880). `src/cli-samples/tool-call-driver.mjs` is a spec-correct sample of what the flag imports.
|
|
151
153
|
|
|
154
|
+
## Dispatch-idempotency conformance — a retried dispatch is one gesture (SPEC §11.1, ggui#1519)
|
|
155
|
+
|
|
156
|
+
A host or relay that could not confirm a `ggui_runtime_submit_action` dispatch may retry it. The server treats a repeat of an earlier committed `(sessionId, actionId)` as the same gesture: one pending event for `ggui_consume` and one `user.submitted` row in the ledger `ggui_runtime_pull` serves, even with both requests in flight at once. An `actionId` reused for a different gesture keeps the first. Three cases: `sequential-retry-is-one-gesture`, `concurrent-retry-is-one-gesture`, `reused-id-keeps-the-first-gesture`. Each has a control: the SAME gesture under a distinct `actionId` is a second gesture, so a server that dedupes by content rather than by id fails it. The cases share their ids, each in a fresh session, so a server that dedupes ids across sessions fails from the second case on. A driver that throws, or a read that answers `isError`, is named on that case, and the run goes on.
|
|
157
|
+
|
|
158
|
+
**SHOULD level this release:** a case that does not hold is reported `warned`, never `failed`. §11.1 makes the promise a MUST in the next release, and the catalog moves every case to `failed` with it.
|
|
159
|
+
|
|
160
|
+
The adopter supplies the same `ToolCallDriver` the domain-error catalog takes, plus an `openSession()` seam: a fresh session that declares `DISPATCH_IDEMPOTENCY_ACTION_SPEC` and has a pending-event pipe, with a `close()` the kit calls after each case. Nothing on the wire mints such a session without generation, which is why it is a host seam. It is also why the CLI does not run this catalog: bind it in your own test, as `@ggui-ai/mcp-server`'s `dispatch-idempotency.conformance.test.ts` does. A host with no seam, or without the three tools, is SKIPPED with the reason named.
|
|
161
|
+
|
|
162
|
+
**What the reuse case grades, and what it cannot yet:** it grades that the first gesture stands. It cannot tell a server that detects the reuse from one that dedupes by id alone, because both keep the first. The difference is visible only in the server's own log until `ACTION_ID_REUSED` is emitted, from the next release.
|
|
163
|
+
|
|
164
|
+
**Not graded by the kit:** §11.1's spend clauses (a repeat, or a reused id, does not use up a one-shot action a second time). They are visible only inside the server; `@ggui-ai/mcp-server-handlers`' own tests grade them for the first-party server.
|
|
165
|
+
|
|
152
166
|
## Conformance status
|
|
153
167
|
|
|
154
168
|
What is actually graded against shipping code today, and what is still awaiting a driver. This is an inventory, not a roadmap — a mechanism is listed as proven only if a test in the repo drives it against the real implementation on every push.
|
|
155
169
|
|
|
156
|
-
Every test below is a plain `*.test.ts` inside a package's `src/`, so it runs in that package's ordinary vitest suite. The
|
|
157
|
-
|
|
158
|
-
| Obligation | Graded against | Driver
|
|
159
|
-
| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
160
|
-
| Gadget wire schema (§7.7.2) | the shipping `@ggui-ai/protocol` `clientCapabilitiesSpecSchema` | this kit's own `src/schema-conformance/schema-conformance.test.ts`
|
|
161
|
-
| Registration gate (§7.7.3, §7.9) | the shipping gate trio in `@ggui-ai/mcp-server-handlers` | `mcp-server-handlers/src/renders/assert-gadgets.conformance.test.ts`
|
|
162
|
-
| Bundle-URL resolution (§7.7.2) | the shipping `resolveGadgetUrls` in `@ggui-ai/mcp-server-handlers` | `mcp-server-handlers/src/renders/resolve-gadget-urls.conformance.test.ts`
|
|
163
|
-
| Path-A WS fixtures — first-party | the real `@ggui-ai/mcp-server` GguiSession channel, booted in-process on an ephemeral port | `mcp-server/src/ggui-session-channel.conformance.test.ts` — **
|
|
164
|
-
| Path-A WS fixtures — vendor-neutral | `@ggui-ai/protocol-reference-server` | `protocol-reference-server/src/conformance.test.ts` — **
|
|
165
|
-
| `resources/read` typed failures + mount invariant | the shipping `registerGguiRenderResourceTemplate` in `@ggui-ai/mcp-server` | `mcp-server/src/mcp-apps-outbound.resource-read.conformance.test.ts` — **16 pass / 0 skip / 0 fail** (the pass set is pinned as the whole catalog)
|
|
166
|
-
| `tools/call`
|
|
167
|
-
|
|
|
168
|
-
|
|
|
169
|
-
| Tool-binding
|
|
170
|
-
|
|
|
171
|
-
|
|
|
172
|
-
|
|
|
173
|
-
|
|
174
|
-
|
|
170
|
+
Every test below is a plain `*.test.ts` inside a package's `src/`, so it runs in that package's ordinary vitest suite. The GGUI repo's per-push CI runs `turbo run test` across all non-e2e packages, so **all fourteen rows below are gated on every push** — the kit needs no dedicated workflow, and adopters wiring it into their own CI need nothing more than their existing unit-test task.
|
|
171
|
+
|
|
172
|
+
| Obligation | Graded against | Driver |
|
|
173
|
+
| ----------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
174
|
+
| Gadget wire schema (§7.7.2) | the shipping `@ggui-ai/protocol` `clientCapabilitiesSpecSchema` | this kit's own `src/schema-conformance/schema-conformance.test.ts` |
|
|
175
|
+
| Registration gate (§7.7.3, §7.9) | the shipping gate trio in `@ggui-ai/mcp-server-handlers` | `mcp-server-handlers/src/renders/assert-gadgets.conformance.test.ts` |
|
|
176
|
+
| Bundle-URL resolution (§7.7.2) | the shipping `resolveGadgetUrls` in `@ggui-ai/mcp-server-handlers` | `mcp-server-handlers/src/renders/resolve-gadget-urls.conformance.test.ts` |
|
|
177
|
+
| Path-A WS fixtures — first-party | the real `@ggui-ai/mcp-server` GguiSession channel, booted in-process on an ephemeral port | `mcp-server/src/ggui-session-channel.conformance.test.ts` — **39 pass / 18 skip / 0 warn / 0 fail**, plus a no-replay control that warns exactly `fresh-subscribe-replays-reserved-preview` |
|
|
178
|
+
| Path-A WS fixtures — vendor-neutral | `@ggui-ai/protocol-reference-server` | `protocol-reference-server/src/conformance.test.ts` — **45 pass / 11 skip / 1 warn / 0 fail** (it declines the fresh-subscribe reserved replay, a SHOULD) |
|
|
179
|
+
| `resources/read` typed failures + mount invariant | the shipping `registerGguiRenderResourceTemplate` in `@ggui-ai/mcp-server` | `mcp-server/src/mcp-apps-outbound.resource-read.conformance.test.ts` — **16 pass / 0 skip / 0 fail** (the pass set is pinned as the whole catalog) |
|
|
180
|
+
| `tools/call` dispatch idempotency (§11.1, ggui#1519; SHOULD level) | the shipping `defaultHandlers` in `@ggui-ai/mcp-server` (`submit_action`, `consume`, `runtime_pull` over one store and pipe) | `mcp-server/src/dispatch-idempotency.conformance.test.ts` — **3 pass / 0 warn / 0 skip / 0 fail** |
|
|
181
|
+
| `tools/call` domain errors (§7.9 Plane 2, ggui#880) | the shipping `createGguiServer` in `@ggui-ai/mcp-server`, over Streamable HTTP | `mcp-server/src/domain-error.conformance.test.ts` — **6 pass / 0 skip / 0 fail** |
|
|
182
|
+
| Read-plane-only posture result (§7.10.6, ggui#1304) | the shipping `createGguiServer({ withholdResultMeta: true })` in `@ggui-ai/mcp-server`, over Streamable HTTP | `mcp-server/src/read-plane-only.conformance.test.ts` — **1 pass / 0 skip / 0 fail**, plus a posture-OFF control that fails exactly `slice-withheld` and `credential-withheld` |
|
|
183
|
+
| Tool-binding resolution — declared wins, contract derivation (§7.7.4.1) | the shipping `resolveMcpToolBindings` in `@ggui-ai/artifact-manifest` | `artifact-manifest/src/mcp-tool-bindings.conformance.test.ts` |
|
|
184
|
+
| Tool-binding search filters — `tool=` / `server=` semantics (§7.7.4.1) | the shipping `matchesMcpToolFilters` in `@ggui-ai/registry-core` | `registry-core/src/mcp-tool-filters.conformance.test.ts` |
|
|
185
|
+
| Pre-generation refusal envelope (§7.1, refused arm) | `projectRenderRefusal` in `@ggui-ai/protocol` — the one source both the shipping `ggui_render` handler (`@ggui-ai/mcp-server-handlers`) and `@ggui-ai/protocol-reference-server` answer through | `mcp-server-handlers/src/renders/render-refusal-projection.conformance.test.ts` |
|
|
186
|
+
| Refusal-registry completeness (ggui#786 ruling 5b) | the shipping `PRE_GENERATION_REFUSAL_CODES` in `@ggui-ai/protocol` | this kit's own `src/registry-completeness/registry-completeness.test.ts` |
|
|
187
|
+
| Endpoint-level refusal (§7.1, `mcp-endpoint` surface, ggui#825) | the catalog's kit-local reference projector only — no shipping deployment projector is graded yet: the OSS route is generic (`errorMapper`), the first-party server and the reference server supply no projector (SKIPPED, named), and a hosted deployment's projector lands after this slice | this kit's own `src/transport-refusal-conformance/transport-refusal-conformance.test.ts` (coherence + discrimination) |
|
|
188
|
+
|
|
189
|
+
Both WS drivers pin their pass set, their skip set _and_ their warned set as exact sets, and the resource-read driver pins its skip set as exactly empty. A fixture silently degrading to a skip fails the build, and so does a skipped fixture that quietly starts passing — a skip set that can grow unnoticed is a false gate, and re-pinning it is meant to be a deliberate act.
|
|
175
190
|
|
|
176
191
|
**Awaiting a driver:**
|
|
177
192
|
|
|
178
193
|
- **Path-B browser-host fixtures** — `bootstrap-bundle-fetch-failed`, `bootstrap-meta-missing` (both need MCP-Apps-host fault injection) and `props-update-roundtrip` (assertion is on rendered DOM). No browser-host adapter ships with the kit, so these skip on every WS driver above. Honestly absent, not silently passing.
|
|
179
194
|
- **`version-mismatch` on the first-party server** — needs a `server-version-override` seam the channel deliberately does not expose; adding a production seam with no production caller purely to drive a fixture was rejected. It passes on the reference server, which has a per-render override.
|
|
180
195
|
- **The `ggui_consume` retrieval half of the action loop, and `stream-delivery-roundtrip`** — both are `tools/call` obligations (`ggui_consume`, `ggui_emit`). The resource-read driver binds `resources/read` only, so neither is closed by it; both await a tool-call driver.
|
|
196
|
+
- **The client side of SPEC §12.2.1 invariant 3** — after a fresh ack a client applies every frame, including those at `seq ≤ streamSeq`, and dedupes against the highest `seq` it applied, never against `streamSeq`. The kit has no driver for a live-channel _client_ (the runner is the client here, and the host-helper kit grades MCP Apps host helpers, not a view's channel), so this obligation is ungraded.
|
|
181
197
|
- **`resources/read` on a second implementation** — the catalog is vendor-neutral by construction (the kit imports no server; the adopter supplies the driver), but it is graded against one implementation today. A second binding is what would ground the vendor-neutrality claim empirically, the way `protocol-reference-server` does for the WS catalog.
|
|
182
198
|
|
|
183
199
|
## Public API
|
|
@@ -192,9 +208,13 @@ const result = await runConformance({
|
|
|
192
208
|
|
|
193
209
|
// A failed fixture is a red build — and so is a run that executed
|
|
194
210
|
// ZERO fixtures (all skips prove nothing).
|
|
195
|
-
|
|
211
|
+
// A warned fixture executed too, so it counts against the zero-executed
|
|
212
|
+
// rule, but a warning never fails the run.
|
|
213
|
+
if (result.failed.length > 0 || result.passed.length + result.warned.length === 0) {
|
|
196
214
|
process.exit(1);
|
|
197
215
|
}
|
|
216
|
+
// `result.warned` lists the SHOULD fixtures the implementation
|
|
217
|
+
// declined. Read them, don't gate on them.
|
|
198
218
|
```
|
|
199
219
|
|
|
200
220
|
`serverUrl` accepts either a bare origin (`http://localhost:3000` — the runner derives `ws://localhost:3000/ws`) or the full live-channel endpoint (`ws://localhost:3000/ws` — used exactly as given; the runner never appends to an explicit path).
|
|
@@ -205,7 +225,7 @@ CLI equivalent:
|
|
|
205
225
|
npx @ggui-ai/protocol-conformance --url ws://localhost:3000/ws --auth bearer:$TOKEN
|
|
206
226
|
```
|
|
207
227
|
|
|
208
|
-
The CLI exits `0` only when at least one fixture executed and none failed; `1` on any fixture failure; `2` on invocation errors **or** when every fixture skipped (a zero-executed run never reads as success in CI).
|
|
228
|
+
The CLI exits `0` only when at least one fixture executed and none failed; `1` on any fixture failure; `2` on invocation errors **or** when every fixture skipped (a zero-executed run never reads as success in CI). A warning never changes the exit code, and a warned fixture counts as executed.
|
|
209
229
|
|
|
210
230
|
The three pure-function catalogs (`refusal-envelope`, `registry-completeness`, `transport-refusal`) grade an in-process function and a data table, not the wire, so a URL alone cannot reach them. The CLI hands them their inputs by flag — `--registry <file.json>` (the deployment's refusal registry, a JSON object keyed by code), `--projector <module>` (an ES module whose `project(refusal)` — named or default export — returns the SPEC §7.1 tool result, or `null` for a code with no render envelope) and `--transport-projector <module>` (same shape, returning the endpoint's `{ httpStatus, error }` or `null`). A flag left out leaves its catalog on the scorecard as SKIPPED with the flag named. Catalog rows count as executed fixtures, so a run given a catalog flag can exit 0 while every wire fixture is SKIPPED — read the scorecard for the wire signal. Minimal spec-correct samples of each input live in the repository at [`src/cli-samples/`](https://github.com/ggui-ai/ggui/tree/main/packages/protocol-conformance/src/cli-samples) (not in the npm tarball); the test suite pins that they pass their catalogs.
|
|
211
231
|
|
package/dist/cli.d.ts
CHANGED
|
@@ -40,8 +40,10 @@ export declare const USAGE: string;
|
|
|
40
40
|
declare function main(argv: readonly string[]): Promise<number>;
|
|
41
41
|
/**
|
|
42
42
|
* Map a {@link ConformanceResult} to the CLI exit code:
|
|
43
|
-
* - `1` — at least one fixture failed.
|
|
44
|
-
*
|
|
43
|
+
* - `1` — at least one fixture failed. A warning (a SHOULD fixture
|
|
44
|
+
* whose expectation was not met) never does.
|
|
45
|
+
* - `2` — zero fixtures executed (every fixture skipped). A warned
|
|
46
|
+
* fixture executed. A run
|
|
45
47
|
* that graded nothing carries no conformance signal and must not
|
|
46
48
|
* read as success in CI.
|
|
47
49
|
* - `0` — at least one fixture executed and none failed.
|
package/dist/cli.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AA6DA,OAAO,EAEL,KAAK,iBAAiB,EACtB,KAAK,oBAAoB,EAC1B,MAAM,sBAAsB,CAAC;AAK9B,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAM7C,UAAU,UAAU;IAClB,QAAQ,CAAC,GAAG,CAAC,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,IAAI,CAAC,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,IAAI,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;IAClC,QAAQ,CAAC,SAAS,CAAC,EAAE,MAAM,CAAC;IAC5B,kFAAkF;IAClF,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAC3B,sEAAsE;IACtE,QAAQ,CAAC,SAAS,CAAC,EAAE,MAAM,CAAC;IAC5B,2EAA2E;IAC3E,QAAQ,CAAC,kBAAkB,CAAC,EAAE,MAAM,CAAC;IACrC,0FAA0F;IAC1F,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,CAAC;IACjC,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;IAC1B,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC;CACxB;AAED,iBAAS,SAAS,CAAC,IAAI,EAAE,SAAS,MAAM,EAAE,GAAG,UAAU,CA8DtD;AAOD,iBAAS,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,UAAU,CAY3C;AAMD,+EAA+E;AAC/E,MAAM,MAAM,aAAa,GAAG,IAAI,CAC9B,oBAAoB,EACpB,iBAAiB,GAAG,kBAAkB,GAAG,2BAA2B,GAAG,gBAAgB,CACxF,CAAC;AAEF,qEAAqE;AACrE,MAAM,WAAW,iBAAiB;IAChC,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACvC,QAAQ,CAAC,SAAS,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACxC,QAAQ,CAAC,kBAAkB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACjD,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAC9C;AA6KD;;;;;;GAMG;AACH,wBAAsB,iBAAiB,CAAC,KAAK,EAAE,iBAAiB,GAAG,OAAO,CAAC,aAAa,CAAC,CAqCxF;AAMD,eAAO,MAAM,KAAK,QA8DL,CAAC;AAEd,iBAAe,IAAI,CAAC,IAAI,EAAE,SAAS,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,CAAC,
|
|
1
|
+
{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AA6DA,OAAO,EAEL,KAAK,iBAAiB,EACtB,KAAK,oBAAoB,EAC1B,MAAM,sBAAsB,CAAC;AAK9B,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAM7C,UAAU,UAAU;IAClB,QAAQ,CAAC,GAAG,CAAC,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,IAAI,CAAC,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,IAAI,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;IAClC,QAAQ,CAAC,SAAS,CAAC,EAAE,MAAM,CAAC;IAC5B,kFAAkF;IAClF,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;IAC3B,sEAAsE;IACtE,QAAQ,CAAC,SAAS,CAAC,EAAE,MAAM,CAAC;IAC5B,2EAA2E;IAC3E,QAAQ,CAAC,kBAAkB,CAAC,EAAE,MAAM,CAAC;IACrC,0FAA0F;IAC1F,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,CAAC;IACjC,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;IAC1B,QAAQ,CAAC,IAAI,EAAE,OAAO,CAAC;CACxB;AAED,iBAAS,SAAS,CAAC,IAAI,EAAE,SAAS,MAAM,EAAE,GAAG,UAAU,CA8DtD;AAOD,iBAAS,SAAS,CAAC,IAAI,EAAE,MAAM,GAAG,UAAU,CAY3C;AAMD,+EAA+E;AAC/E,MAAM,MAAM,aAAa,GAAG,IAAI,CAC9B,oBAAoB,EACpB,iBAAiB,GAAG,kBAAkB,GAAG,2BAA2B,GAAG,gBAAgB,CACxF,CAAC;AAEF,qEAAqE;AACrE,MAAM,WAAW,iBAAiB;IAChC,QAAQ,CAAC,QAAQ,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACvC,QAAQ,CAAC,SAAS,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACxC,QAAQ,CAAC,kBAAkB,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;IACjD,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CAC9C;AA6KD;;;;;;GAMG;AACH,wBAAsB,iBAAiB,CAAC,KAAK,EAAE,iBAAiB,GAAG,OAAO,CAAC,aAAa,CAAC,CAqCxF;AAMD,eAAO,MAAM,KAAK,QA8DL,CAAC;AAEd,iBAAe,IAAI,CAAC,IAAI,EAAE,SAAS,MAAM,EAAE,GAAG,OAAO,CAAC,MAAM,CAAC,CAsE5D;AAED;;;;;;;;;GASG;AACH,iBAAS,iBAAiB,CAAC,MAAM,EAAE,iBAAiB,GAAG,CAAC,GAAG,CAAC,GAAG,CAAC,CAI/D;AA+BD,OAAO,EAAE,iBAAiB,EAAE,IAAI,EAAE,SAAS,EAAE,SAAS,EAAE,CAAC"}
|
package/dist/cli.js
CHANGED
|
@@ -36,8 +36,8 @@
|
|
|
36
36
|
* design. To get real pass/fail signal against an implementation,
|
|
37
37
|
* consumers call `runConformance({ host: myImpl })` programmatically
|
|
38
38
|
* and apply the same guard:
|
|
39
|
-
* `result.failed.length > 0 || result.passed.length === 0`
|
|
40
|
-
* build. The CLI remains the lowest-friction wire-level smoke check
|
|
39
|
+
* `result.failed.length > 0 || result.passed.length + result.warned.length === 0`
|
|
40
|
+
* is a red build (a warned fixture executed, and never fails a run). The CLI remains the lowest-friction wire-level smoke check
|
|
41
41
|
* for setups where the server under test is provisioned out-of-band
|
|
42
42
|
* rather than through a `ConformanceHost` injection.
|
|
43
43
|
*/
|
|
@@ -45,7 +45,7 @@ import { readFile } from 'node:fs/promises';
|
|
|
45
45
|
import { resolve } from 'node:path';
|
|
46
46
|
import { pathToFileURL } from 'node:url';
|
|
47
47
|
import { isRawToolCallResult, } from './domain-error-conformance/index.js';
|
|
48
|
-
import { createDefaultReporter, formatFailures, formatSkips } from './reporter.js';
|
|
48
|
+
import { createDefaultReporter, formatFailures, formatSkips, formatWarnings } from './reporter.js';
|
|
49
49
|
import { runConformance, } from './run-conformance.js';
|
|
50
50
|
function parseArgs(argv) {
|
|
51
51
|
let url;
|
|
@@ -342,7 +342,7 @@ Options:
|
|
|
342
342
|
performs ONE tools/call and returns the raw result
|
|
343
343
|
(null = tool not bound). Grades the domain-error
|
|
344
344
|
catalog — SPEC §7.9 Plane 2 on the wire (ggui#880).
|
|
345
|
-
--verbose, -v Print failure details + skip reasons at the end.
|
|
345
|
+
--verbose, -v Print failure details, warnings + skip reasons at the end.
|
|
346
346
|
--help, -h Show this help.
|
|
347
347
|
|
|
348
348
|
Pure-function catalogs (refusal-envelope, registry-completeness, transport-refusal):
|
|
@@ -421,20 +421,25 @@ async function main(argv) {
|
|
|
421
421
|
const failures = formatFailures(result.failed);
|
|
422
422
|
if (failures.length > 0)
|
|
423
423
|
process.stdout.write(`${failures}\n`);
|
|
424
|
+
const warnings = formatWarnings(result.warned);
|
|
425
|
+
if (warnings.length > 0)
|
|
426
|
+
process.stdout.write(`${warnings}\n`);
|
|
424
427
|
const skips = formatSkips(result.skipped);
|
|
425
428
|
if (skips.length > 0)
|
|
426
429
|
process.stdout.write(`${skips}\n`);
|
|
427
430
|
}
|
|
428
431
|
const code = exitCodeForResult(result);
|
|
429
432
|
if (code === 2) {
|
|
430
|
-
process.stderr.write(`error: conformance run executed ZERO fixtures (0 passed, ${result.failed.length} failed, ${result.skipped.length} skipped) — an all-skip run proves nothing and must not read as success; exiting 2. Run with --verbose for skip reasons.\n`);
|
|
433
|
+
process.stderr.write(`error: conformance run executed ZERO fixtures (0 passed, ${result.failed.length} failed, 0 warned, ${result.skipped.length} skipped) — an all-skip run proves nothing and must not read as success; exiting 2. Run with --verbose for skip reasons.\n`);
|
|
431
434
|
}
|
|
432
435
|
return code;
|
|
433
436
|
}
|
|
434
437
|
/**
|
|
435
438
|
* Map a {@link ConformanceResult} to the CLI exit code:
|
|
436
|
-
* - `1` — at least one fixture failed.
|
|
437
|
-
*
|
|
439
|
+
* - `1` — at least one fixture failed. A warning (a SHOULD fixture
|
|
440
|
+
* whose expectation was not met) never does.
|
|
441
|
+
* - `2` — zero fixtures executed (every fixture skipped). A warned
|
|
442
|
+
* fixture executed. A run
|
|
438
443
|
* that graded nothing carries no conformance signal and must not
|
|
439
444
|
* read as success in CI.
|
|
440
445
|
* - `0` — at least one fixture executed and none failed.
|
|
@@ -442,7 +447,7 @@ async function main(argv) {
|
|
|
442
447
|
function exitCodeForResult(result) {
|
|
443
448
|
if (result.failed.length > 0)
|
|
444
449
|
return 1;
|
|
445
|
-
if (result.passed.length === 0)
|
|
450
|
+
if (result.passed.length === 0 && result.warned.length === 0)
|
|
446
451
|
return 2;
|
|
447
452
|
return 0;
|
|
448
453
|
}
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* `dispatch-idempotency` catalog — SPEC §11.1 "A retried dispatch is one
|
|
3
|
+
* gesture" (ggui#1519), graded on the `tools/call` wire.
|
|
4
|
+
*
|
|
5
|
+
* A host or relay that could not confirm a `ggui_runtime_submit_action`
|
|
6
|
+
* dispatch may retry it. The server treats a repeat of an earlier committed
|
|
7
|
+
* `(sessionId, actionId)` as the same gesture: one pending event for the
|
|
8
|
+
* agent's `ggui_consume`, one `user.submitted` row in the ledger that
|
|
9
|
+
* `ggui_runtime_pull` serves, even when both requests are in flight at once.
|
|
10
|
+
* An `actionId` reused for a DIFFERENT gesture keeps the first.
|
|
11
|
+
*
|
|
12
|
+
* SHOULD LEVEL THIS RELEASE. §11.1 states the promise as a SHOULD and makes
|
|
13
|
+
* it a MUST in the next release, so a case that does not hold is reported
|
|
14
|
+
* `warned`, never `failed`. The next release moves every case to `failed`.
|
|
15
|
+
*
|
|
16
|
+
* Vendor-neutral by construction: the kit imports no server. The adopter
|
|
17
|
+
* supplies a {@link ToolCallDriver} (the same one the `domain-error` catalog
|
|
18
|
+
* takes) and a {@link DispatchIdempotencyHost.openSession} seam that opens a
|
|
19
|
+
* live session declaring the kit's action, with a pending-event pipe, and
|
|
20
|
+
* later closes it. Nothing on the wire mints such a session without
|
|
21
|
+
* generation, which is why it is a host seam. A host that supplies no seam,
|
|
22
|
+
* or binds none of the three tools, is SKIPPED with the reason named.
|
|
23
|
+
*
|
|
24
|
+
* What the cases are built to catch, beyond "no dedupe at all": the control
|
|
25
|
+
* repeats the SAME gesture under a new id, so a server that dedupes by content
|
|
26
|
+
* rather than by id loses it; the ids repeat across cases, each in a fresh
|
|
27
|
+
* session, so a server that dedupes by id across sessions loses the later
|
|
28
|
+
* cases' first gesture; and the concurrent case puts both requests in flight
|
|
29
|
+
* at once, so a check-then-write dedupe lets both through. A driver that
|
|
30
|
+
* throws, or a read that answers `isError`, is named on its case rather than
|
|
31
|
+
* read as an empty pipe.
|
|
32
|
+
*
|
|
33
|
+
* Not graded here: whether a repeat re-spends a one-shot action. That clause
|
|
34
|
+
* of §11.1 is visible only inside the server, and the first-party server's
|
|
35
|
+
* own tests grade it.
|
|
36
|
+
*/
|
|
37
|
+
import type { ActionSpec } from '@ggui-ai/protocol';
|
|
38
|
+
import type { ToolCallDriver } from '../domain-error-conformance/index.js';
|
|
39
|
+
/** The one action every case dispatches; the host's session must declare it. */
|
|
40
|
+
export declare const DISPATCH_IDEMPOTENCY_ACTION: "pick";
|
|
41
|
+
/**
|
|
42
|
+
* The action spec the host's session declares: one action whose payload is
|
|
43
|
+
* `{ choice: string }`. Declared in full because a contract gate treats an
|
|
44
|
+
* object schema as closed, and the cases tell gestures apart by `choice`.
|
|
45
|
+
*/
|
|
46
|
+
export declare const DISPATCH_IDEMPOTENCY_ACTION_SPEC: ActionSpec;
|
|
47
|
+
/** A live session the host opened for one case, and how to release it. */
|
|
48
|
+
export interface DispatchIdempotencySession {
|
|
49
|
+
readonly sessionId: string;
|
|
50
|
+
readonly appId: string;
|
|
51
|
+
close(): Promise<void>;
|
|
52
|
+
}
|
|
53
|
+
/** What the adopter supplies. */
|
|
54
|
+
export interface DispatchIdempotencyHost {
|
|
55
|
+
readonly callTool: ToolCallDriver;
|
|
56
|
+
/**
|
|
57
|
+
* Open a fresh session declaring {@link DISPATCH_IDEMPOTENCY_ACTION_SPEC},
|
|
58
|
+
* with its pending-event pipe open. The seam owns cleanup: the kit calls
|
|
59
|
+
* `close()` after the case, whatever its outcome.
|
|
60
|
+
*/
|
|
61
|
+
readonly openSession?: () => Promise<DispatchIdempotencySession>;
|
|
62
|
+
}
|
|
63
|
+
export type DispatchIdempotencyCaseName = 'sequential-retry-is-one-gesture' | 'concurrent-retry-is-one-gesture' | 'reused-id-keeps-the-first-gesture';
|
|
64
|
+
export interface DispatchIdempotencyCaseResult {
|
|
65
|
+
/** Reported as `dispatch-idempotency/<name>`. */
|
|
66
|
+
readonly name: DispatchIdempotencyCaseName;
|
|
67
|
+
readonly detail: string;
|
|
68
|
+
}
|
|
69
|
+
export interface DispatchIdempotencyResult {
|
|
70
|
+
readonly passed: readonly DispatchIdempotencyCaseResult[];
|
|
71
|
+
/** SHOULD level: a case that does not hold lands here, not in `failed`. */
|
|
72
|
+
readonly warned: readonly DispatchIdempotencyCaseResult[];
|
|
73
|
+
readonly failed: readonly DispatchIdempotencyCaseResult[];
|
|
74
|
+
readonly skipped: readonly DispatchIdempotencyCaseResult[];
|
|
75
|
+
}
|
|
76
|
+
/** Run the catalog. Each case gets its own session from the host's seam. */
|
|
77
|
+
export declare function runDispatchIdempotencyConformance(host: DispatchIdempotencyHost): Promise<DispatchIdempotencyResult>;
|
|
78
|
+
//# sourceMappingURL=index.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/dispatch-idempotency-conformance/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAmCG;AACH,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,mBAAmB,CAAC;AACpD,OAAO,KAAK,EAAgB,cAAc,EAAqB,MAAM,sCAAsC,CAAC;AAE5G,gFAAgF;AAChF,eAAO,MAAM,2BAA2B,EAAG,MAAe,CAAC;AAE3D;;;;GAIG;AACH,eAAO,MAAM,gCAAgC,EAAE,UAK9C,CAAC;AAEF,0EAA0E;AAC1E,MAAM,WAAW,0BAA0B;IACzC,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC;IAC3B,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;CACxB;AAED,iCAAiC;AACjC,MAAM,WAAW,uBAAuB;IACtC,QAAQ,CAAC,QAAQ,EAAE,cAAc,CAAC;IAClC;;;;OAIG;IACH,QAAQ,CAAC,WAAW,CAAC,EAAE,MAAM,OAAO,CAAC,0BAA0B,CAAC,CAAC;CAClE;AAED,MAAM,MAAM,2BAA2B,GACnC,iCAAiC,GACjC,iCAAiC,GACjC,mCAAmC,CAAC;AAExC,MAAM,WAAW,6BAA6B;IAC5C,iDAAiD;IACjD,QAAQ,CAAC,IAAI,EAAE,2BAA2B,CAAC;IAC3C,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED,MAAM,WAAW,yBAAyB;IACxC,QAAQ,CAAC,MAAM,EAAE,SAAS,6BAA6B,EAAE,CAAC;IAC1D,2EAA2E;IAC3E,QAAQ,CAAC,MAAM,EAAE,SAAS,6BAA6B,EAAE,CAAC;IAC1D,QAAQ,CAAC,MAAM,EAAE,SAAS,6BAA6B,EAAE,CAAC;IAC1D,QAAQ,CAAC,OAAO,EAAE,SAAS,6BAA6B,EAAE,CAAC;CAC5D;AA0ID,4EAA4E;AAC5E,wBAAsB,iCAAiC,CACrD,IAAI,EAAE,uBAAuB,GAC5B,OAAO,CAAC,yBAAyB,CAAC,CAyBpC"}
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
/** The one action every case dispatches; the host's session must declare it. */
|
|
2
|
+
export const DISPATCH_IDEMPOTENCY_ACTION = 'pick';
|
|
3
|
+
/**
|
|
4
|
+
* The action spec the host's session declares: one action whose payload is
|
|
5
|
+
* `{ choice: string }`. Declared in full because a contract gate treats an
|
|
6
|
+
* object schema as closed, and the cases tell gestures apart by `choice`.
|
|
7
|
+
*/
|
|
8
|
+
export const DISPATCH_IDEMPOTENCY_ACTION_SPEC = {
|
|
9
|
+
[DISPATCH_IDEMPOTENCY_ACTION]: {
|
|
10
|
+
label: 'Pick',
|
|
11
|
+
schema: { type: 'object', properties: { choice: { type: 'string' } }, required: ['choice'] },
|
|
12
|
+
},
|
|
13
|
+
};
|
|
14
|
+
const CASES = [
|
|
15
|
+
'sequential-retry-is-one-gesture',
|
|
16
|
+
'concurrent-retry-is-one-gesture',
|
|
17
|
+
'reused-id-keeps-the-first-gesture',
|
|
18
|
+
];
|
|
19
|
+
/**
|
|
20
|
+
* The ids every case uses. The SAME ids in every case, each case in its own
|
|
21
|
+
* session: the promise is per `(sessionId, actionId)`, so a later case's first
|
|
22
|
+
* dispatch is a new gesture, and a server that remembers ids across sessions
|
|
23
|
+
* drops it.
|
|
24
|
+
*/
|
|
25
|
+
const X = 'kit-1519-x';
|
|
26
|
+
const Y = 'kit-1519-y';
|
|
27
|
+
/** Thrown inside a case when the host cannot run it at all (a tool unbound). */
|
|
28
|
+
class Unbound extends Error {
|
|
29
|
+
}
|
|
30
|
+
function field(value, key) {
|
|
31
|
+
return typeof value === 'object' && value !== null ? Reflect.get(value, key) : undefined;
|
|
32
|
+
}
|
|
33
|
+
function dispatchArgs(session, actionId, choice) {
|
|
34
|
+
return {
|
|
35
|
+
sessionId: session.sessionId,
|
|
36
|
+
appId: session.appId,
|
|
37
|
+
kind: 'dispatch',
|
|
38
|
+
actionId,
|
|
39
|
+
firedAt: '2026-09-29T00:00:00.000Z',
|
|
40
|
+
payload: { intent: DISPATCH_IDEMPOTENCY_ACTION, actionData: { choice }, uiContext: {} },
|
|
41
|
+
};
|
|
42
|
+
}
|
|
43
|
+
async function call(host, tool, args) {
|
|
44
|
+
const result = await host.callTool({ tool, args });
|
|
45
|
+
if (result === null)
|
|
46
|
+
throw new Unbound(`${tool} is not bound on this deployment`);
|
|
47
|
+
return result;
|
|
48
|
+
}
|
|
49
|
+
/** `ok: true` from submit_action, or the declared reuse answer where a case allows it. */
|
|
50
|
+
function answerOf(result) {
|
|
51
|
+
const structured = result.structuredContent;
|
|
52
|
+
if (field(structured, 'ok') === true)
|
|
53
|
+
return 'ok';
|
|
54
|
+
const code = field(structured, 'code');
|
|
55
|
+
return typeof code === 'string' ? code : result.isError === true ? 'isError' : 'unknown';
|
|
56
|
+
}
|
|
57
|
+
/**
|
|
58
|
+
* A read tool's `events`, or the reason there are none to count: a read that
|
|
59
|
+
* answered `isError`, or no `events` array, is a problem of its own and not an
|
|
60
|
+
* empty pipe (which would misreport the case as a delivery count).
|
|
61
|
+
*/
|
|
62
|
+
async function readEvents(host, tool, args) {
|
|
63
|
+
const result = await call(host, tool, args);
|
|
64
|
+
if (result.isError === true)
|
|
65
|
+
return { problem: `${tool} answered isError` };
|
|
66
|
+
const events = field(result.structuredContent, 'events');
|
|
67
|
+
if (!Array.isArray(events))
|
|
68
|
+
return { problem: `${tool} answered no events array` };
|
|
69
|
+
return { events };
|
|
70
|
+
}
|
|
71
|
+
/** Run one case against a fresh session. Returns the reasons it does not hold (empty = holds). */
|
|
72
|
+
async function runCase(name, host, session) {
|
|
73
|
+
const problems = [];
|
|
74
|
+
let answers;
|
|
75
|
+
switch (name) {
|
|
76
|
+
case 'sequential-retry-is-one-gesture':
|
|
77
|
+
answers = [
|
|
78
|
+
answerOf(await call(host, 'ggui_runtime_submit_action', dispatchArgs(session, X, 'a'))),
|
|
79
|
+
answerOf(await call(host, 'ggui_runtime_submit_action', dispatchArgs(session, X, 'a'))),
|
|
80
|
+
];
|
|
81
|
+
break;
|
|
82
|
+
case 'concurrent-retry-is-one-gesture':
|
|
83
|
+
answers = (await Promise.all([
|
|
84
|
+
call(host, 'ggui_runtime_submit_action', dispatchArgs(session, X, 'a')),
|
|
85
|
+
call(host, 'ggui_runtime_submit_action', dispatchArgs(session, X, 'a')),
|
|
86
|
+
])).map(answerOf);
|
|
87
|
+
break;
|
|
88
|
+
case 'reused-id-keeps-the-first-gesture':
|
|
89
|
+
answers = [
|
|
90
|
+
answerOf(await call(host, 'ggui_runtime_submit_action', dispatchArgs(session, X, 'a'))),
|
|
91
|
+
answerOf(await call(host, 'ggui_runtime_submit_action', dispatchArgs(session, X, 'b'))),
|
|
92
|
+
];
|
|
93
|
+
break;
|
|
94
|
+
}
|
|
95
|
+
// Control: the SAME gesture under a DISTINCT id is a second gesture, so a
|
|
96
|
+
// server that drops every dispatch, every second one, or every repeat of
|
|
97
|
+
// content, cannot pass.
|
|
98
|
+
const control = answerOf(await call(host, 'ggui_runtime_submit_action', dispatchArgs(session, Y, 'a')));
|
|
99
|
+
const reuseCase = name === 'reused-id-keeps-the-first-gesture';
|
|
100
|
+
if (answers[0] !== 'ok')
|
|
101
|
+
problems.push(`the first dispatch answered ${answers[0]}, not ok`);
|
|
102
|
+
// The repeat is answered as the first was; a reuse may instead be refused
|
|
103
|
+
// ACTION_ID_REUSED (declared this release, emitted from the next).
|
|
104
|
+
if (!(answers[1] === 'ok' || (reuseCase && answers[1] === 'ACTION_ID_REUSED'))) {
|
|
105
|
+
problems.push(`the repeat answered ${answers[1]}${reuseCase ? ', not ok or ACTION_ID_REUSED' : ', not as the first'}`);
|
|
106
|
+
}
|
|
107
|
+
if (control !== 'ok')
|
|
108
|
+
problems.push(`the control (a distinct actionId) answered ${control}, not ok`);
|
|
109
|
+
// One consume: it clears what it returns, so a second would read nothing.
|
|
110
|
+
const drained = await readEvents(host, 'ggui_consume', { sessionId: session.sessionId, timeout: 0 });
|
|
111
|
+
if ('problem' in drained) {
|
|
112
|
+
problems.push(drained.problem);
|
|
113
|
+
}
|
|
114
|
+
else {
|
|
115
|
+
const events = drained.events.filter((e) => field(e, 'actionId') === X).map((e) => field(e, 'actionData'));
|
|
116
|
+
if (events.length !== 1)
|
|
117
|
+
problems.push(`ggui_consume delivered ${events.length} events for the repeated actionId, not 1`);
|
|
118
|
+
if (reuseCase && events.length === 1 && field(events[0], 'choice') !== 'a') {
|
|
119
|
+
problems.push('the delivered event is not the first gesture');
|
|
120
|
+
}
|
|
121
|
+
const controlEvents = drained.events.filter((e) => field(e, 'actionId') === Y).length;
|
|
122
|
+
if (controlEvents !== 1)
|
|
123
|
+
problems.push(`the control's gesture was delivered ${controlEvents} times, not once`);
|
|
124
|
+
}
|
|
125
|
+
const ledger = await readEvents(host, 'ggui_runtime_pull', { sessionId: session.sessionId, sinceSequence: 0, limit: 100 });
|
|
126
|
+
if ('problem' in ledger) {
|
|
127
|
+
problems.push(ledger.problem);
|
|
128
|
+
}
|
|
129
|
+
else {
|
|
130
|
+
const rowsFor = (id) => ledger.events.filter((e) => field(e, 'type') === 'user.submitted' && field(field(e, 'data'), 'actionId') === id)
|
|
131
|
+
.length;
|
|
132
|
+
const rows = rowsFor(X);
|
|
133
|
+
if (rows !== 1)
|
|
134
|
+
problems.push(`the ledger holds ${rows} user.submitted rows for the repeated actionId, not 1`);
|
|
135
|
+
const controlRows = rowsFor(Y);
|
|
136
|
+
if (controlRows !== 1)
|
|
137
|
+
problems.push(`the control's gesture has ${controlRows} ledger rows, not 1`);
|
|
138
|
+
}
|
|
139
|
+
return problems;
|
|
140
|
+
}
|
|
141
|
+
/** Run the catalog. Each case gets its own session from the host's seam. */
|
|
142
|
+
export async function runDispatchIdempotencyConformance(host) {
|
|
143
|
+
const passed = [];
|
|
144
|
+
const warned = [];
|
|
145
|
+
const skipped = [];
|
|
146
|
+
for (const name of CASES) {
|
|
147
|
+
if (!host.openSession) {
|
|
148
|
+
skipped.push({ name, detail: 'the host supplies no openSession seam, so no session declares the kit action' });
|
|
149
|
+
continue;
|
|
150
|
+
}
|
|
151
|
+
const session = await host.openSession();
|
|
152
|
+
try {
|
|
153
|
+
const problems = await runCase(name, host, session);
|
|
154
|
+
if (problems.length === 0)
|
|
155
|
+
passed.push({ name, detail: 'one gesture: one consumed event and one ledger row' });
|
|
156
|
+
else
|
|
157
|
+
warned.push({ name, detail: problems.join('; ') });
|
|
158
|
+
}
|
|
159
|
+
catch (err) {
|
|
160
|
+
// An unbound tool skips the case. Any other throw (the driver's, or the
|
|
161
|
+
// server's through it) is the case not holding, named, and the run goes
|
|
162
|
+
// on: one case's transport failure grades that case only.
|
|
163
|
+
if (err instanceof Unbound)
|
|
164
|
+
skipped.push({ name, detail: err.message });
|
|
165
|
+
else
|
|
166
|
+
warned.push({ name, detail: `the case threw: ${err instanceof Error ? err.message : String(err)}` });
|
|
167
|
+
}
|
|
168
|
+
finally {
|
|
169
|
+
await session.close();
|
|
170
|
+
}
|
|
171
|
+
}
|
|
172
|
+
return { passed, warned, failed: [], skipped };
|
|
173
|
+
}
|
package/dist/fixtures/index.d.ts
CHANGED
|
@@ -20,7 +20,7 @@
|
|
|
20
20
|
*
|
|
21
21
|
* ## Current coverage
|
|
22
22
|
*
|
|
23
|
-
*
|
|
23
|
+
* Seven WebSocket sub-modules with content (13 fixtures). Additional sub-modules
|
|
24
24
|
* (`data-contract`, `tool-name-uniqueness`)
|
|
25
25
|
* are reserved for additive fixture expansion and not yet
|
|
26
26
|
* materialized — adding fixtures there is a kit minor version.
|
|
@@ -90,8 +90,8 @@
|
|
|
90
90
|
* false fail: the runner dispatches every setup directive BEFORE it
|
|
91
91
|
* opens the WebSocket and subscribes, and `emit-envelope` fans out to
|
|
92
92
|
* live subscribers only — a fresh, `fromSeq`-less subscribe replays
|
|
93
|
-
* nothing on declared channels (SPEC §12.2.1 invariant 1:
|
|
94
|
-
*
|
|
93
|
+
* nothing on declared channels (SPEC §12.2.1 invariant 1: a fresh
|
|
94
|
+
* subscribe replays no agent-declared channel), so the setup-time emission is
|
|
95
95
|
* gone before the runner can observe it. The fixture's
|
|
96
96
|
* `inputEnvelope` cannot carry the emission either: an input envelope
|
|
97
97
|
* is by contract a Client→Server wire frame the runner sends on the
|
|
@@ -102,11 +102,29 @@
|
|
|
102
102
|
* driver bound to `tools/call`. The `resources/read` driver above is
|
|
103
103
|
* NOT that driver. Declared here rather than papered over with a
|
|
104
104
|
* fixture that fails conformant servers on sequencing alone.
|
|
105
|
+
*
|
|
106
|
+
* A KNOWN-RESERVED channel is different, and that case IS authored
|
|
107
|
+
* (`reserved-channel-replay`, ggui#1526): a fresh subscribe SHOULD
|
|
108
|
+
* replay its retained envelopes after the ack (SPEC §12.2.1, the
|
|
109
|
+
* ggui#1521 erratum), so the setup-time emission is observable as that
|
|
110
|
+
* replay. It is graded at `level: 'should'`: a server that declines it
|
|
111
|
+
* is warned, never failed.
|
|
112
|
+
*
|
|
113
|
+
* ## What is NOT here — the client side of invariant 3
|
|
114
|
+
*
|
|
115
|
+
* SPEC §12.2.1 invariant 3 is a CLIENT obligation: after a fresh ack,
|
|
116
|
+
* apply every frame, including those at `seq ≤ streamSeq`, and dedupe
|
|
117
|
+
* against the highest `seq` applied, never against `streamSeq`. The kit
|
|
118
|
+
* has no driver for a live-channel CLIENT (the runner IS the client
|
|
119
|
+
* here, and the host-helper kit grades MCP Apps host helpers, not a
|
|
120
|
+
* view's channel), so it cannot grade that obligation. Declared as a
|
|
121
|
+
* gap, not implied as covered.
|
|
105
122
|
*/
|
|
106
123
|
export { bootstrapProtocolFixtures } from './bootstrap-protocol/index.js';
|
|
107
124
|
export { consumeBufferFixtures } from './consume-buffer/index.js';
|
|
108
125
|
export { hostContextFixtures } from './host-context/index.js';
|
|
109
126
|
export { reservedChannelAuthorityFixtures } from './reserved-channel-authority/index.js';
|
|
127
|
+
export { reservedChannelReplayFixtures } from './reserved-channel-replay/index.js';
|
|
110
128
|
export { schemaVersionHandshakeFixtures } from './schema-version-handshake/index.js';
|
|
111
129
|
export { subscribeAppScopeFixtures } from './subscribe-app-scope/index.js';
|
|
112
130
|
import type { TestCase } from '../types.js';
|
|
@@ -116,7 +134,7 @@ import type { TestCase } from '../types.js';
|
|
|
116
134
|
* Extensibly-closed via `(string & {})` so additive sub-modules
|
|
117
135
|
* register without an API break.
|
|
118
136
|
*/
|
|
119
|
-
export type ContractSlug = 'bootstrap-protocol' | 'consume-buffer' | 'host-context' | 'reserved-channel-authority' | 'schema-version-handshake' | 'subscribe-app-scope' | (string & {});
|
|
137
|
+
export type ContractSlug = 'bootstrap-protocol' | 'consume-buffer' | 'host-context' | 'reserved-channel-authority' | 'reserved-channel-replay' | 'schema-version-handshake' | 'subscribe-app-scope' | (string & {});
|
|
120
138
|
/**
|
|
121
139
|
* Every fixture the kit ships, grouped by the contract it exercises.
|
|
122
140
|
* Order within each group matches the sub-module's `export` order.
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/fixtures/index.ts"],"names":[],"mappings":"AAAA
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/fixtures/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAyHG;AAEH,OAAO,EAAE,yBAAyB,EAAE,MAAM,+BAA+B,CAAC;AAC1E,OAAO,EAAE,qBAAqB,EAAE,MAAM,2BAA2B,CAAC;AAClE,OAAO,EAAE,mBAAmB,EAAE,MAAM,yBAAyB,CAAC;AAC9D,OAAO,EAAE,gCAAgC,EAAE,MAAM,uCAAuC,CAAC;AACzF,OAAO,EAAE,6BAA6B,EAAE,MAAM,oCAAoC,CAAC;AACnF,OAAO,EAAE,8BAA8B,EAAE,MAAM,qCAAqC,CAAC;AACrF,OAAO,EAAE,yBAAyB,EAAE,MAAM,gCAAgC,CAAC;AAE3E,OAAO,KAAK,EAAE,QAAQ,EAAE,MAAM,aAAa,CAAC;AAS5C;;;;;GAKG;AACH,MAAM,MAAM,YAAY,GACpB,oBAAoB,GACpB,gBAAgB,GAChB,cAAc,GACd,4BAA4B,GAC5B,yBAAyB,GACzB,0BAA0B,GAC1B,qBAAqB,GACrB,CAAC,MAAM,GAAG,EAAE,CAAC,CAAC;AAElB;;;;;;;GAOG;AACH,eAAO,MAAM,kBAAkB,EAAE,QAAQ,CAAC,MAAM,CAAC,YAAY,EAAE,SAAS,QAAQ,EAAE,CAAC,CAQlF,CAAC;AAEF;;;;GAIG;AACH,eAAO,MAAM,WAAW,EAAE,SAAS,QAAQ,EAGwB,CAAC"}
|