@openwop/openwop-conformance 1.138.1 → 1.139.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/coverage.md +1 -1
- package/package.json +1 -1
- package/schemas/CORPUS-STAMP.json +2 -2
- package/src/global-setup.ts +169 -0
- package/src/lib/fork-availability.test.ts +69 -0
- package/src/lib/fork-availability.ts +61 -0
- package/src/lib/global-setup.test.ts +76 -0
- package/src/scenarios/a2a-task-roundtrip.test.ts +66 -10
- package/src/scenarios/agent-channel-dispatch.test.ts +3 -3
- package/src/scenarios/replay-fanout-suppression.test.ts +39 -6
- package/src/scenarios/replay-fork-arbitrary.test.ts +9 -3
- package/src/scenarios/replay-fork.test.ts +10 -2
- package/src/scenarios/replay-side-effect-suppression.test.ts +3 -2
- package/src/scenarios/replayDeterminism.test.ts +8 -6
- package/src/scenarios/webhook-signed-delivery.test.ts +53 -7
- package/vitest.config.ts +7 -0
package/README.md
CHANGED
|
@@ -78,7 +78,7 @@ Run `npm run test` for normal CI cadence; `npm run test:strict` when claiming fu
|
|
|
78
78
|
| `OPENWOP_OTEL_COLLECTOR_GRPC=true` | Boots the parallel OTLP/gRPC collector alongside the HTTP one (h2c HTTP/2 on a separate port). Shares the same `spans()` + `metrics()` store; spans captured over either transport surface in `getCollector().spans()`. Requires `OPENWOP_OTEL_COLLECTOR=true`. Same `--no-file-parallelism` requirement applies. |
|
|
79
79
|
| `OPENWOP_OTEL_COLLECTOR_GRPC_PORT=4317` | Bind the OTLP/gRPC collector on a specific port (default `4317`, OTLP/gRPC convention). The host MUST be configured with `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:<port>` AND `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`. |
|
|
80
80
|
| `OPENWOP_OTEL_COLLECTOR_PORT=14318` | Bind the OTel collector on a specific port (default `4318`). The host MUST be configured with `OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:<port>`. |
|
|
81
|
-
| `OPENWOP_WEBHOOK_ALLOW_PRIVATE=true` |
|
|
81
|
+
| `OPENWOP_WEBHOOK_ALLOW_PRIVATE=true` | Relaxes the webhook egress guard for the loopback test receiver used by `webhook-signed-delivery.test.ts`, `webhook-negative.test.ts`, and `replay-fanout-suppression.test.ts`. **The receiver is `http://127.0.0.1:{port}/`, which `webhooks.md` forbids three separate times, and the opt-in MUST relax all three to be witnessable:** (1) the **scheme** check — §"SSRF protection" bullet 1 rejects non-`https://`, and this gate fires FIRST on a host that validates scheme before address; (2) the **registration-time** address check — §"SSRF protection"; and (3) **delivery-time re-resolution** — §"Delivery-time egress validation (RFC 0093)". Gates 2 and 3 are independent MUSTs at different layers, so an opt-in reaching only one layer **cannot** produce a witness: delivery-only leaves registration returning `400 webhook_url_rejected`, registration-only leaves the dispatcher refusing to connect. A host whose opt-in reaches one layer is **not** non-conformant — it cannot witness these scenarios, which is a property of the test posture and not of its webhook signing. The scheme gate went unstated here until 2026-08-25; a tier-2 host validating scheme-then-address was blocked by a gate the documented contract never named. Relaxing gates 2 and 3 is **test-only posture** — see `SECURITY/threat-model-secret-leakage.md` §4.9 for why a registration-time relaxation is the more dangerous of the two. When the host rejects, the scenario records `blocked` (RFC 0148 §A), not a pass. |
|
|
82
82
|
| `OPENWOP_MCP_FAKE_SERVER=true` | Boots the synthetic MCP peer for `mcp-tool-roundtrip.test.ts`. |
|
|
83
83
|
| `OPENWOP_MCP_REAL_SERVER_URL=<base-url>` | Points the MCP wire-shape probe at a real MCP server. The probe POSTs JSON-RPC and reads a single-JSON response — matches MCP's `streamable-http` transport in single-response mode. **Does NOT support** stdio transport (which is what most `modelcontextprotocol/servers` references default to) or SSE-streamed responses; an operator collecting interop evidence today runs a custom `StreamableHTTPServerTransport`-style server that returns a single JSON body per request. Adding SSE-frame parsing is tracked in `docs/PROTOCOL-GAP-CLOSURE-PLAN.md` Track 6. Assertions relax to shape-only. When both this and `OPENWOP_MCP_FAKE_SERVER` are set, the real URL wins. Phase 3 T3.4 interop-evidence path. |
|
|
84
84
|
| `OPENWOP_A2A_FAKE_PEER=true` | Boots the synthetic A2A peer for `a2a-task-roundtrip.test.ts`. |
|
package/coverage.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# OpenWOP Conformance Coverage Map
|
|
2
2
|
|
|
3
|
-
> **Status: Living document. Updated 2026-
|
|
3
|
+
> **Status: Living document. Updated 2026-08-25.** This map connects the current scenario files to the protocol surfaces they protect and records the remaining gaps from the protocol deep dive. Scenario names are source-of-truth file names under `conformance/src/scenarios/`.
|
|
4
4
|
|
|
5
5
|
> **Shape grade vs behavior grade.** Some optional-profile scenarios validate **capability shape** (the host's discovery advertisement is well-formed) without yet exercising **behavior** (the host actually implements the profile end-to-end). The "Current grade" column reflects shape; see §"Capability-gated scenarios: shape vs behavior" below for the dual-grade view and the `OPENWOP_REQUIRE_BEHAVIOR=true` strict-mode runner flag.
|
|
6
6
|
|
package/package.json
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
2
|
"_comment": "Provenance of this vendored schemas/ copy. See conformance/README.md \u00a7\"Resolving the contract\". Compare against the stamp in your installed @openwop/openwop-conformance to detect a stale hand-copied contract.",
|
|
3
|
-
"suiteVersion": "1.
|
|
4
|
-
"corpusCommit": "
|
|
3
|
+
"suiteVersion": "1.139.0",
|
|
4
|
+
"corpusCommit": "f34a41ac4c1737ca8af34d0f7f17876268f58e69"
|
|
5
5
|
}
|
|
@@ -0,0 +1,169 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Run-end disposition summary (RFC 0148 §A, 2026-08-25).
|
|
3
|
+
*
|
|
4
|
+
* ## The gap this closes
|
|
5
|
+
*
|
|
6
|
+
* Every scenario file already records ONE honest disposition when it
|
|
7
|
+
* finishes (`setup.ts`'s `afterAll` → `resolveFileRecord`): a file that
|
|
8
|
+
* returned early with zero assertions is `blocked` / `skipped` /
|
|
9
|
+
* `inapplicable` with a reason, never a pass. That machinery is correct.
|
|
10
|
+
*
|
|
11
|
+
* It was **published on one invocation out of two**. The worker appends
|
|
12
|
+
* each recording to `OPENWOP_LEDGER_PATH`, and the ONLY writer of that
|
|
13
|
+
* variable was `cli.ts` inside `runCertify`. Under a plain `vitest run`
|
|
14
|
+
* — what `README.md` documents for host operators, and what host
|
|
15
|
+
* implementers actually run — the worker computed `blocked`, held it in
|
|
16
|
+
* memory, and discarded it at process exit. The single surviving artifact
|
|
17
|
+
* was vitest's `1 passed`.
|
|
18
|
+
*
|
|
19
|
+
* So the suite computed an honest disposition on every run and published
|
|
20
|
+
* it on one. Two host implementers independently read the console line as
|
|
21
|
+
* coverage and reported the suite as claiming a pass it had internally
|
|
22
|
+
* classified `blocked` — correctly, given the only artifact they had.
|
|
23
|
+
* RFC 0158 §"Operator preconditions are declared, not hidden" requires a
|
|
24
|
+
* blocked row be surfaced "never silently skipped or reported as a pass";
|
|
25
|
+
* the bundle honored that and the console contradicted it.
|
|
26
|
+
*
|
|
27
|
+
* ## What this does
|
|
28
|
+
*
|
|
29
|
+
* When `OPENWOP_LEDGER_PATH` is unset, point it at a temp file so the
|
|
30
|
+
* workers record there anyway, then read it at run end and print a
|
|
31
|
+
* summary. `--certify` sets the variable itself, so this defers to it
|
|
32
|
+
* completely and never interferes with bundle generation.
|
|
33
|
+
*
|
|
34
|
+
* The summary is deliberately NOT a pass/fail signal — it does not change
|
|
35
|
+
* the exit code. `vitest`'s exit code answers "did any assertion fail";
|
|
36
|
+
* this answers "what did the run actually witness", and those are
|
|
37
|
+
* different questions. Conflating them is what produced the gap.
|
|
38
|
+
*
|
|
39
|
+
* Set `OPENWOP_DISPOSITION_SUMMARY=false` to silence it.
|
|
40
|
+
*/
|
|
41
|
+
|
|
42
|
+
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
|
|
43
|
+
import { join } from 'node:path';
|
|
44
|
+
import { tmpdir } from 'node:os';
|
|
45
|
+
|
|
46
|
+
/** Dispositions that are NOT a witnessed execution — the rows worth printing. */
|
|
47
|
+
const NON_EXECUTING = new Set(['blocked', 'skipped', 'inapplicable']);
|
|
48
|
+
|
|
49
|
+
/** Cap on printed rows; the overflow is always announced, never silent. */
|
|
50
|
+
const MAX_ROWS = 40;
|
|
51
|
+
|
|
52
|
+
// Import the REAL entry type rather than restating its shape. A hand-written
|
|
53
|
+
// structural guess here read `entry.id` — a key the ledger has never written
|
|
54
|
+
// (it is `requirementId`) — and the `(unnamed)` fallback made the mismatch
|
|
55
|
+
// look like missing data instead of a wrong reader. A type-only import erases
|
|
56
|
+
// at runtime, so this costs nothing and makes the compiler the oracle: rename
|
|
57
|
+
// a ledger field and this file stops building instead of silently degrading.
|
|
58
|
+
import type { LedgerEntry } from './lib/requirement-ledger.js';
|
|
59
|
+
|
|
60
|
+
/** What a JSONL line parses to before validation — every field may be absent. */
|
|
61
|
+
type LedgerLine = Partial<Record<keyof LedgerEntry, unknown>>;
|
|
62
|
+
|
|
63
|
+
let ownedDir: string | null = null;
|
|
64
|
+
let ledgerPath: string | null = null;
|
|
65
|
+
|
|
66
|
+
export function setup(): void {
|
|
67
|
+
if (process.env['OPENWOP_DISPOSITION_SUMMARY'] === 'false') return;
|
|
68
|
+
// `--certify` already routes the ledger to its own report dir and reads it
|
|
69
|
+
// there. Never take it over — this exists only for the path that had no
|
|
70
|
+
// reader at all.
|
|
71
|
+
if ((process.env['OPENWOP_LEDGER_PATH'] ?? '') !== '') return;
|
|
72
|
+
ownedDir = mkdtempSync(join(tmpdir(), 'owp-dispositions-'));
|
|
73
|
+
ledgerPath = join(ownedDir, 'requirement-ledger.jsonl');
|
|
74
|
+
process.env['OPENWOP_LEDGER_PATH'] = ledgerPath;
|
|
75
|
+
}
|
|
76
|
+
|
|
77
|
+
/**
|
|
78
|
+
* Fold raw JSONL into the printable summary. PURE — no I/O, no env — so
|
|
79
|
+
* `global-setup.test.ts` can pin it without a vitest subprocess or a host.
|
|
80
|
+
* Returns `null` when there is nothing to say.
|
|
81
|
+
*/
|
|
82
|
+
export function summarise(raw: string): string | null {
|
|
83
|
+
const counts = new Map<string, number>();
|
|
84
|
+
const rows: Array<{ id: string; disposition: string; detail: string }> = [];
|
|
85
|
+
for (const line of raw.split('\n')) {
|
|
86
|
+
if (line.trim() === '') continue;
|
|
87
|
+
let entry: LedgerLine;
|
|
88
|
+
try {
|
|
89
|
+
entry = JSON.parse(line) as LedgerLine;
|
|
90
|
+
} catch {
|
|
91
|
+
continue;
|
|
92
|
+
}
|
|
93
|
+
const disposition = typeof entry.disposition === 'string' ? entry.disposition : 'unknown';
|
|
94
|
+
const id = typeof entry.requirementId === 'string' ? entry.requirementId : '(unnamed)';
|
|
95
|
+
const detail = typeof entry.detail === 'string' ? entry.detail : '';
|
|
96
|
+
counts.set(disposition, (counts.get(disposition) ?? 0) + 1);
|
|
97
|
+
if (NON_EXECUTING.has(disposition)) {
|
|
98
|
+
rows.push({ id, disposition, detail });
|
|
99
|
+
} else if (disposition === 'executed-pass' && entry.assertionCount === 0) {
|
|
100
|
+
// The other shape RFC 0148 §A rejects: a pass that asserted nothing.
|
|
101
|
+
rows.push({ id, disposition: 'executed-pass (0 assertions)', detail });
|
|
102
|
+
}
|
|
103
|
+
}
|
|
104
|
+
if (counts.size === 0) return null;
|
|
105
|
+
const total = [...counts.values()].reduce((a, b) => a + b, 0);
|
|
106
|
+
const order = ['executed-pass', 'executed-fail', 'blocked', 'skipped', 'inapplicable'];
|
|
107
|
+
const summary = [...order, ...[...counts.keys()].filter((k) => !order.includes(k))]
|
|
108
|
+
.filter((k) => counts.has(k))
|
|
109
|
+
.map((k) => `${k} ${counts.get(k)}`)
|
|
110
|
+
.join(' \u00b7 ');
|
|
111
|
+
const out: string[] = [
|
|
112
|
+
'',
|
|
113
|
+
`[openwop-conformance] RFC 0148 \u00a7A dispositions \u2014 ${total} requirement(s) recorded`,
|
|
114
|
+
` ${summary}`,
|
|
115
|
+
];
|
|
116
|
+
if (rows.length > 0) {
|
|
117
|
+
out.push(
|
|
118
|
+
` ${rows.length} requirement(s) did NOT witness. vitest reports these as passes because`,
|
|
119
|
+
' no assertion failed; that is a test outcome, not a conformance disposition.',
|
|
120
|
+
);
|
|
121
|
+
for (const r of rows.slice(0, MAX_ROWS)) {
|
|
122
|
+
const detail = r.detail.length > 120 ? `${r.detail.slice(0, 117)}...` : r.detail;
|
|
123
|
+
out.push(` ${r.disposition.padEnd(28)} ${r.id}${detail === '' ? '' : `\n ${detail}`}`);
|
|
124
|
+
}
|
|
125
|
+
if (rows.length > MAX_ROWS) {
|
|
126
|
+
// Never truncate silently: a capped list that does not say it was capped
|
|
127
|
+
// reads as a complete one, which is the same defect this whole summary
|
|
128
|
+
// exists to remove.
|
|
129
|
+
out.push(` ... and ${rows.length - MAX_ROWS} more not listed (run with --certify for the full bundle)`);
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
out.push('');
|
|
133
|
+
return out.join('\n');
|
|
134
|
+
}
|
|
135
|
+
|
|
136
|
+
export function teardown(): void {
|
|
137
|
+
if (ledgerPath === null || ownedDir === null) return;
|
|
138
|
+
// We set the variable; unset it so a subsequent in-process run re-derives.
|
|
139
|
+
delete process.env['OPENWOP_LEDGER_PATH'];
|
|
140
|
+
let raw = '';
|
|
141
|
+
try {
|
|
142
|
+
raw = readFileSync(ledgerPath, 'utf8');
|
|
143
|
+
} catch {
|
|
144
|
+
// No file means no worker ever recorded \u2014 nothing to summarise, and a
|
|
145
|
+
// missing summary must never be mistaken for "nothing was blocked", so
|
|
146
|
+
// say so rather than printing an empty clean bill.
|
|
147
|
+
process.stderr.write(
|
|
148
|
+
'\n[openwop-conformance] no disposition ledger was written \u2014 the run recorded nothing, ' +
|
|
149
|
+
'which is NOT the same as every scenario having witnessed its requirement.\n',
|
|
150
|
+
);
|
|
151
|
+
cleanup();
|
|
152
|
+
return;
|
|
153
|
+
}
|
|
154
|
+
const text = summarise(raw);
|
|
155
|
+
if (text !== null) process.stderr.write(text);
|
|
156
|
+
cleanup();
|
|
157
|
+
}
|
|
158
|
+
|
|
159
|
+
function cleanup(): void {
|
|
160
|
+
if (ownedDir !== null) {
|
|
161
|
+
try {
|
|
162
|
+
rmSync(ownedDir, { recursive: true, force: true });
|
|
163
|
+
} catch {
|
|
164
|
+
/* best effort */
|
|
165
|
+
}
|
|
166
|
+
}
|
|
167
|
+
ownedDir = null;
|
|
168
|
+
ledgerPath = null;
|
|
169
|
+
}
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Pins `forkDeclined` (RFC 0148 §A).
|
|
3
|
+
*
|
|
4
|
+
* The risk in a helper like this is not that it misses a status — that shows up
|
|
5
|
+
* as a red test somebody investigates. It is that it swallows one: every status
|
|
6
|
+
* it accepts turns an assertion into a skip, silently, and a suite that skips
|
|
7
|
+
* is indistinguishable from a suite that passes unless something records why.
|
|
8
|
+
* So the load-bearing cases here are the NEGATIVE ones.
|
|
9
|
+
*/
|
|
10
|
+
|
|
11
|
+
import { describe, expect, it, beforeEach } from 'vitest';
|
|
12
|
+
import { forkDeclined } from './fork-availability.js';
|
|
13
|
+
import { resetSoftSkips, softSkipDisposition } from './soft-skip.js';
|
|
14
|
+
|
|
15
|
+
describe('forkDeclined', () => {
|
|
16
|
+
beforeEach(() => resetSoftSkips());
|
|
17
|
+
|
|
18
|
+
it('treats 404 and 403 the same as 501 — route absent is not weaker than route-declines', () => {
|
|
19
|
+
// The asymmetry this fixes: every replay scenario handled 501 and none
|
|
20
|
+
// handled 404. `501` means "I know this route and decline"; `404` means
|
|
21
|
+
// "there is no such route" — strictly less implemented. The suite treated
|
|
22
|
+
// the weaker signal as a skip and the stronger one as a defect, so a host
|
|
23
|
+
// had to implement the route in order to say it had not implemented it.
|
|
24
|
+
for (const status of [404, 403, 501]) {
|
|
25
|
+
expect(forkDeclined(status, 'leg'), `status ${status} means the fork did not happen`).toBe(true);
|
|
26
|
+
}
|
|
27
|
+
});
|
|
28
|
+
|
|
29
|
+
it('does NOT swallow any other status — a 500 is still a defect, not a skip', () => {
|
|
30
|
+
// If this ever goes green for one of these, a real failure has become a
|
|
31
|
+
// silent skip and no assertion downstream will ever run again.
|
|
32
|
+
for (const status of [200, 201, 202, 400, 409, 422, 500, 502, 503]) {
|
|
33
|
+
expect(forkDeclined(status, 'leg'), `status ${status} must reach the assertion`).toBe(false);
|
|
34
|
+
}
|
|
35
|
+
});
|
|
36
|
+
|
|
37
|
+
it('records WHY, so the ledger row is not an unclassified return', () => {
|
|
38
|
+
// Two of the eleven original call sites were a bare `return`. A bare return
|
|
39
|
+
// in a file whose OTHER tests assert is invisible: the file records
|
|
40
|
+
// `executed-pass` and the leg that never ran leaves no trace.
|
|
41
|
+
forkDeclined(404, 'fanout-suppression replay fork');
|
|
42
|
+
const noted = softSkipDisposition('fork-availability.test.ts');
|
|
43
|
+
expect(noted?.kind).toBe('blocked');
|
|
44
|
+
expect(noted?.reason).toContain('404');
|
|
45
|
+
expect(noted?.reason).toContain('fanout-suppression replay fork');
|
|
46
|
+
expect(noted?.reason).toContain('not mounted');
|
|
47
|
+
});
|
|
48
|
+
|
|
49
|
+
it('records nothing when the status is not a decline', () => {
|
|
50
|
+
forkDeclined(201, 'leg');
|
|
51
|
+
expect(softSkipDisposition('fork-availability.test.ts')).toBeNull();
|
|
52
|
+
});
|
|
53
|
+
|
|
54
|
+
it('names the leg, so one file with several forks stays attributable', () => {
|
|
55
|
+
forkDeclined(501, 'branch fork');
|
|
56
|
+
expect(softSkipDisposition('fork-availability.test.ts')?.reason).toContain('branch fork');
|
|
57
|
+
});
|
|
58
|
+
|
|
59
|
+
it('distinguishes the three declines in its reason text', () => {
|
|
60
|
+
forkDeclined(404, 'a');
|
|
61
|
+
const a = softSkipDisposition('fork-availability.test.ts')?.reason ?? '';
|
|
62
|
+
resetSoftSkips();
|
|
63
|
+
forkDeclined(501, 'a');
|
|
64
|
+
const b = softSkipDisposition('fork-availability.test.ts')?.reason ?? '';
|
|
65
|
+
// `blocked` alone does not tell a bundle reader whether the host lacks the
|
|
66
|
+
// route or declines the range; the reason must.
|
|
67
|
+
expect(a).not.toBe(b);
|
|
68
|
+
});
|
|
69
|
+
});
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* "The fork seam did not run" — one rule, one place (RFC 0148 §A).
|
|
3
|
+
*
|
|
4
|
+
* `POST /v1/runs/{runId}:fork` can decline in more than one way, and eleven
|
|
5
|
+
* call sites across six scenarios each decided for themselves which ways
|
|
6
|
+
* counted:
|
|
7
|
+
*
|
|
8
|
+
* - **501** — every site handled this. "Advertised but not implemented for
|
|
9
|
+
* this range", per the suite convention their comments cite.
|
|
10
|
+
* - **404 / 403** — *no* site in the replay family handled it. A host that
|
|
11
|
+
* never mounts the route hard-failed `expect(status).toBe(201)`.
|
|
12
|
+
*
|
|
13
|
+
* That asymmetry is backwards. `501` means "I know this route and decline";
|
|
14
|
+
* `404` means "there is no such route" — strictly less implemented, and the
|
|
15
|
+
* suite treated the weaker signal as a skip and the stronger one as a defect.
|
|
16
|
+
* The postgres reference host 404s, so `replay-fanout-suppression` was red in
|
|
17
|
+
* CI on every run while the other four replay scenarios skipped past the same
|
|
18
|
+
* seam for unrelated fixture reasons. **The suite required a host to implement
|
|
19
|
+
* the route in order to say it had not implemented the route.**
|
|
20
|
+
*
|
|
21
|
+
* Recording matters as much as the predicate. Two of those eleven sites were a
|
|
22
|
+
* bare `return` with no note: `replay-fork.test.ts` reported "6 tests | 6
|
|
23
|
+
* skipped" against a host whose fork surface is entirely unmeasured, and the
|
|
24
|
+
* ledger had nothing to say about why. §A resolves an unclassified return to
|
|
25
|
+
* `blocked`, but only when the file records nothing at all — a bare return in a
|
|
26
|
+
* file whose *other* tests assert is invisible.
|
|
27
|
+
*
|
|
28
|
+
* Disposition is `blocked`, not `inapplicable`: the obligation applies to this
|
|
29
|
+
* host, it simply could not be witnessed. `inapplicable` would claim the host
|
|
30
|
+
* is outside the requirement's scope, which nothing here establishes — these
|
|
31
|
+
* scenarios carry no capability gate, so the suite does not actually know
|
|
32
|
+
* whether the host advertises fork. Saying `blocked` is the claim the evidence
|
|
33
|
+
* supports.
|
|
34
|
+
*/
|
|
35
|
+
|
|
36
|
+
import { softSkip } from './soft-skip.js';
|
|
37
|
+
|
|
38
|
+
/** Statuses that mean "the fork did not happen", with what each one tells us. */
|
|
39
|
+
const DECLINED: ReadonlyMap<number, string> = new Map([
|
|
40
|
+
[404, 'the route is not mounted on this host'],
|
|
41
|
+
[403, 'the caller is not permitted to fork (route present, access refused)'],
|
|
42
|
+
[501, 'the host knows the route and declines this range as not implemented'],
|
|
43
|
+
]);
|
|
44
|
+
|
|
45
|
+
/**
|
|
46
|
+
* True when `status` means the fork seam did not run — and records WHY for the
|
|
47
|
+
* RFC 0148 §A ledger as a side effect, so a caller can `if (forkDeclined(...))
|
|
48
|
+
* return;` without leaving an unclassified return behind.
|
|
49
|
+
*
|
|
50
|
+
* `where` names the leg, so a bundle reader can tell which of several forks in
|
|
51
|
+
* one file declined.
|
|
52
|
+
*/
|
|
53
|
+
export function forkDeclined(status: number, where: string): boolean {
|
|
54
|
+
const why = DECLINED.get(status);
|
|
55
|
+
if (why === undefined) return false;
|
|
56
|
+
softSkip(
|
|
57
|
+
'blocked',
|
|
58
|
+
`${where}: POST /v1/runs/{runId}:fork returned ${status} — ${why}, so the behaviour this requirement is stated over never occurred and its absence below would prove nothing`,
|
|
59
|
+
);
|
|
60
|
+
return true;
|
|
61
|
+
}
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Pins the run-end disposition summary (`src/global-setup.ts`, RFC 0148 §A).
|
|
3
|
+
*
|
|
4
|
+
* The defect this guards is not "the summary is ugly" — it is that the suite
|
|
5
|
+
* computed an honest disposition on every run and PUBLISHED it on one. Two
|
|
6
|
+
* host implementers read vitest's `1 passed` as coverage for a requirement the
|
|
7
|
+
* suite had internally classified `blocked`, because the classification had no
|
|
8
|
+
* reader outside `--certify`.
|
|
9
|
+
*
|
|
10
|
+
* Server-free and pure: `summarise` takes the raw JSONL and returns the text.
|
|
11
|
+
*/
|
|
12
|
+
|
|
13
|
+
import { describe, expect, it } from 'vitest';
|
|
14
|
+
import { summarise } from '../global-setup.js';
|
|
15
|
+
|
|
16
|
+
const line = (o: Record<string, unknown>): string => JSON.stringify(o);
|
|
17
|
+
|
|
18
|
+
describe('global-setup: RFC 0148 §A disposition summary', () => {
|
|
19
|
+
it('reads the ledger key the ledger actually writes (`requirementId`, not `id`)', () => {
|
|
20
|
+
// Regression: the first draft of this reader keyed on `entry.id`, a field
|
|
21
|
+
// `recordRequirement` has never written, and fell back to `(unnamed)`.
|
|
22
|
+
// The fallback made a WRONG READER look like MISSING DATA — the same
|
|
23
|
+
// failure shape the summary exists to expose, reproduced in the exposer.
|
|
24
|
+
const out = summarise(
|
|
25
|
+
line({ requirementId: 'openwop.scenario.webhook-signed-delivery', disposition: 'blocked', detail: 'guard rejected loopback' }),
|
|
26
|
+
);
|
|
27
|
+
expect(out).toContain('openwop.scenario.webhook-signed-delivery');
|
|
28
|
+
expect(out).not.toContain('(unnamed)');
|
|
29
|
+
});
|
|
30
|
+
|
|
31
|
+
it('lists blocked / skipped / inapplicable as NOT witnessed, with the reason', () => {
|
|
32
|
+
const out = summarise(
|
|
33
|
+
[
|
|
34
|
+
line({ requirementId: 'a', disposition: 'blocked', detail: 'seam absent' }),
|
|
35
|
+
line({ requirementId: 'b', disposition: 'skipped', detail: 'operator opted out' }),
|
|
36
|
+
line({ requirementId: 'c', disposition: 'inapplicable', detail: 'profile not advertised' }),
|
|
37
|
+
].join('\n'),
|
|
38
|
+
);
|
|
39
|
+
expect(out).toContain('did NOT witness');
|
|
40
|
+
for (const s of ['seam absent', 'operator opted out', 'profile not advertised']) {
|
|
41
|
+
expect(out).toContain(s);
|
|
42
|
+
}
|
|
43
|
+
});
|
|
44
|
+
|
|
45
|
+
it('flags an executed-pass that asserted nothing — the other RFC 0148 §A vacuity shape', () => {
|
|
46
|
+
const out = summarise(line({ requirementId: 'vacuous', disposition: 'executed-pass', assertionCount: 0 }));
|
|
47
|
+
expect(out).toContain('executed-pass (0 assertions)');
|
|
48
|
+
expect(out).toContain('vacuous');
|
|
49
|
+
});
|
|
50
|
+
|
|
51
|
+
it('does NOT flag a real executed-pass', () => {
|
|
52
|
+
const out = summarise(line({ requirementId: 'real', disposition: 'executed-pass', assertionCount: 12 }));
|
|
53
|
+
expect(out).toContain('executed-pass 1');
|
|
54
|
+
expect(out).not.toContain('did NOT witness');
|
|
55
|
+
});
|
|
56
|
+
|
|
57
|
+
it('announces truncation rather than capping silently', () => {
|
|
58
|
+
// A capped list that does not say it was capped reads as a complete one.
|
|
59
|
+
const many = Array.from({ length: 45 }, (_, i) =>
|
|
60
|
+
line({ requirementId: `r${i}`, disposition: 'blocked', detail: 'why' }),
|
|
61
|
+
).join('\n');
|
|
62
|
+
const out = summarise(many);
|
|
63
|
+
expect(out).toContain('5 more not listed');
|
|
64
|
+
});
|
|
65
|
+
|
|
66
|
+
it('returns null on an empty ledger so teardown prints nothing', () => {
|
|
67
|
+
expect(summarise('')).toBeNull();
|
|
68
|
+
expect(summarise('\n\n')).toBeNull();
|
|
69
|
+
});
|
|
70
|
+
|
|
71
|
+
it('skips unparseable lines without discarding the rest of the run', () => {
|
|
72
|
+
const out = summarise(['{ not json', line({ requirementId: 'ok', disposition: 'blocked', detail: 'r' })].join('\n'));
|
|
73
|
+
expect(out).toContain('ok');
|
|
74
|
+
expect(out).toContain('blocked 1');
|
|
75
|
+
});
|
|
76
|
+
});
|
|
@@ -47,6 +47,7 @@ import { pollUntilTerminal, pollUntilStatus } from '../lib/polling.js';
|
|
|
47
47
|
import { SCHEMAS_DIR } from '../lib/paths.js';
|
|
48
48
|
import { behaviorGate } from '../lib/behavior-gate.js';
|
|
49
49
|
import { readCapabilityFamily } from '../lib/discovery-capabilities.js';
|
|
50
|
+
import { seamAbsent } from '../lib/soft-skip.js';
|
|
50
51
|
|
|
51
52
|
/**
|
|
52
53
|
* Callback-shaped: the host issues A2A JSON-RPC calls to the suite's fake peer.
|
|
@@ -403,21 +404,76 @@ describe.skipIf(HTTP_SKIP)('a2a-task-roundtrip: durable tasks/get after disconne
|
|
|
403
404
|
});
|
|
404
405
|
});
|
|
405
406
|
|
|
406
|
-
|
|
407
|
-
|
|
407
|
+
/**
|
|
408
|
+
* Push-config SSRF, TWO-SIDED (RFC 0100 §4, a2a-integration.md §D.6).
|
|
409
|
+
*
|
|
410
|
+
* The RFC 0093 webhook-egress guard has two arms — `webhooks.md`
|
|
411
|
+
* §"SSRF protection" rejects non-`https://` protocols AND private/loopback/
|
|
412
|
+
* link-local/ULA/metadata addresses — and a host may implement either one
|
|
413
|
+
* alone.
|
|
414
|
+
*
|
|
415
|
+
* Until 2026-08-25 this file probed with a single `http://10.0.0.5/push`,
|
|
416
|
+
* which **violates both arms at once**. Either arm alone refuses it, so a
|
|
417
|
+
* `>= 400` witnessed *that something refused* and never *which guard ran*.
|
|
418
|
+
* A host with only the address arm passed; so did a host with only the
|
|
419
|
+
* scheme arm; so would a host that refused every push URL for an unrelated
|
|
420
|
+
* reason. The assertion was real and the conclusion drawn from it was not.
|
|
421
|
+
*
|
|
422
|
+
* The probes below isolate one arm each: `https` at a private address can
|
|
423
|
+
* only be refused by the address arm, and `http` at a public host can only
|
|
424
|
+
* be refused by the scheme arm. Two legs is the minimum that distinguishes
|
|
425
|
+
* them — the same reason a negative control is not optional.
|
|
426
|
+
*
|
|
427
|
+
* The scheme leg is new, and the obligation it checks was previously stated
|
|
428
|
+
* only by reference (every prior wording abbreviated the guard to its
|
|
429
|
+
* address arm). `COMPATIBILITY.md` §3 records the Class 3 classification: a
|
|
430
|
+
* host accepting a plaintext push target was never conforming. An
|
|
431
|
+
* implementer reddened by this leg is reading a clarification, not a new
|
|
432
|
+
* requirement — the failure message says so.
|
|
433
|
+
*/
|
|
434
|
+
describe.skipIf(HTTP_SKIP)('a2a-task-roundtrip: push-config SSRF, two-sided (gated on a2a.pushNotifications; RFC 0100)', () => {
|
|
435
|
+
async function registerPush(url: string): Promise<{ status: number } | null> {
|
|
408
436
|
const a2a = await readCapabilityFamily<{ pushNotifications?: boolean }>('a2a');
|
|
409
|
-
if (!behaviorGate('a2a.pushNotifications', a2a?.pushNotifications === true)) return;
|
|
437
|
+
if (!behaviorGate('a2a.pushNotifications', a2a?.pushNotifications === true)) return null;
|
|
438
|
+
const res = await driver.post('/v1/host/sample/a2a/tasks/push-config', { taskId: 'run_x', url });
|
|
439
|
+
if (res.status === 404 || res.status === 403) {
|
|
440
|
+
// Previously a bare `return`, invisible to the RFC 0148 §A ledger because
|
|
441
|
+
// this file's other tests assert — so the file recorded `executed-pass`
|
|
442
|
+
// while this leg had witnessed nothing. Say why instead.
|
|
443
|
+
return seamAbsent(
|
|
444
|
+
'a2a-push-egress-ssrf — the `/v1/host/sample/a2a/tasks/push-config` seam is not mounted (404/403), so neither guard arm is observable',
|
|
445
|
+
) ?? null;
|
|
446
|
+
}
|
|
447
|
+
return { status: res.status };
|
|
448
|
+
}
|
|
449
|
+
|
|
450
|
+
it('ADDRESS arm: an https pushConfig.url at a private address is refused', async () => {
|
|
451
|
+
// `https` on purpose: this probe is refusable ONLY by the address arm, so a
|
|
452
|
+
// host that implements the scheme arm alone cannot pass it by accident.
|
|
453
|
+
const res = await registerPush('https://10.0.0.5/push');
|
|
454
|
+
if (res === null) return;
|
|
455
|
+
expect(
|
|
456
|
+
res.status >= 400,
|
|
457
|
+
driver.describe(
|
|
458
|
+
'a2a-integration.md §D.6 (address arm)',
|
|
459
|
+
'a2a-push-egress-ssrf — a pushConfig.url at a private/loopback address MUST be refused before any push, even over https',
|
|
460
|
+
),
|
|
461
|
+
).toBe(true);
|
|
462
|
+
});
|
|
410
463
|
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
if (res.status === 404 || res.status === 403) return; // seam unwired — soft-skip
|
|
464
|
+
it('SCHEME arm: an http pushConfig.url at a public host is refused', async () => {
|
|
465
|
+
// Public hostname on purpose: refusable ONLY by the scheme arm.
|
|
466
|
+
const res = await registerPush('http://push.example.com/push');
|
|
467
|
+
if (res === null) return;
|
|
416
468
|
expect(
|
|
417
469
|
res.status >= 400,
|
|
418
470
|
driver.describe(
|
|
419
|
-
'a2a-integration.md §
|
|
420
|
-
'a2a-push-egress-ssrf — a
|
|
471
|
+
'a2a-integration.md §D.6 (scheme arm)',
|
|
472
|
+
'a2a-push-egress-ssrf — a plaintext `http://` pushConfig.url MUST be refused before any push. '
|
|
473
|
+
+ 'The RFC 0093 webhook-egress guard is the `webhooks.md` §"SSRF protection" list IN FULL, whose first entry is '
|
|
474
|
+
+ '"Non-`https://` protocols". Every prior wording of this requirement abbreviated the guard to its address arm; '
|
|
475
|
+
+ 'COMPATIBILITY.md §3 records this as a Class 3 clarification, so a host failing here was never conforming rather '
|
|
476
|
+
+ 'than newly non-conforming.',
|
|
421
477
|
),
|
|
422
478
|
).toBe(true);
|
|
423
479
|
});
|
|
@@ -38,6 +38,7 @@
|
|
|
38
38
|
|
|
39
39
|
import { describe, it, expect } from 'vitest';
|
|
40
40
|
import { driver } from '../lib/driver.js';
|
|
41
|
+
import { forkDeclined } from '../lib/fork-availability.js';
|
|
41
42
|
import { pollUntilTerminal } from '../lib/polling.js';
|
|
42
43
|
import { behaviorGate } from '../lib/behavior-gate.js';
|
|
43
44
|
import { isFixtureAdvertised } from '../lib/fixtures.js';
|
|
@@ -149,8 +150,7 @@ describe.skipIf(HTTP_SKIP)('agent-channel-dispatch (RFC 0082 §B): production ru
|
|
|
149
150
|
`/v1/runs/${encodeURIComponent(sourceRunId)}:fork`,
|
|
150
151
|
{ fromSeq: 0, mode: 'replay' },
|
|
151
152
|
);
|
|
152
|
-
if (fork1.status
|
|
153
|
-
// replay advertised but not implemented for this run — skip-equivalent.
|
|
153
|
+
if (forkDeclined(fork1.status, 'channel-dispatch replay fork 1')) {
|
|
154
154
|
ctx.skip();
|
|
155
155
|
return;
|
|
156
156
|
}
|
|
@@ -213,7 +213,7 @@ describe.skipIf(HTTP_SKIP)('agent-channel-dispatch (RFC 0082 §B): production ru
|
|
|
213
213
|
`/v1/runs/${encodeURIComponent(sourceRunId)}:fork`,
|
|
214
214
|
{ fromSeq: 0, mode: 'replay' },
|
|
215
215
|
);
|
|
216
|
-
if (fork2.status
|
|
216
|
+
if (forkDeclined(fork2.status, 'channel-dispatch replay fork 2')) {
|
|
217
217
|
ctx.skip();
|
|
218
218
|
return;
|
|
219
219
|
}
|
|
@@ -72,7 +72,8 @@ import { afterEach, describe, expect, it } from 'vitest';
|
|
|
72
72
|
import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http';
|
|
73
73
|
import { softSkip } from '../lib/soft-skip.js';
|
|
74
74
|
import { driver } from '../lib/driver.js';
|
|
75
|
-
import {
|
|
75
|
+
import { forkDeclined } from '../lib/fork-availability.js';
|
|
76
|
+
import { discoveryFamilies, readCapabilityFamily } from '../lib/discovery-capabilities.js';
|
|
76
77
|
import { pollUntilTerminal, scaledTimeoutMs } from '../lib/polling.js';
|
|
77
78
|
import { isFixtureAdvertised } from '../lib/fixtures.js';
|
|
78
79
|
import { discoverOwnedTenant } from '../lib/webhook-receiver.js';
|
|
@@ -213,18 +214,50 @@ describe('replay-fanout-suppression: a replay fork MUST NOT fan out re-emitted e
|
|
|
213
214
|
),
|
|
214
215
|
).toBeGreaterThan(0);
|
|
215
216
|
|
|
217
|
+
// ── CAPABILITY GATE (added 2026-08-25) ──────────────────────────────────
|
|
218
|
+
// This is the ONLY scenario in the replay family that never checked whether
|
|
219
|
+
// the host advertises replay before forking — every sibling reads
|
|
220
|
+
// `replay.supported` first. That omission is why it was the one scenario
|
|
221
|
+
// reaching the fork seam on a host that does not implement it, and why it
|
|
222
|
+
// hard-failed `expected 404 to be 201` on every CI run of `main` while the
|
|
223
|
+
// siblings quietly returned at their capability check.
|
|
224
|
+
//
|
|
225
|
+
// `inapplicable`, not `blocked`: a host that does not advertise replay is
|
|
226
|
+
// outside this MUST NOT's scope entirely, and `blocked` would claim the
|
|
227
|
+
// requirement applies but could not be witnessed — a stronger claim than
|
|
228
|
+
// the evidence supports. Recorded explicitly because LEG 1 above asserted,
|
|
229
|
+
// so the file-level disposition would otherwise be `executed-pass`.
|
|
230
|
+
const replayCap = await readCapabilityFamily<{ supported?: boolean; modes?: unknown }>('replay');
|
|
231
|
+
if (replayCap?.supported !== true) {
|
|
232
|
+
recordRequirement(
|
|
233
|
+
REQUIREMENT_ID,
|
|
234
|
+
'inapplicable',
|
|
235
|
+
'host does not advertise `replay.supported: true`, so a replay fork cannot occur and this MUST NOT '
|
|
236
|
+
+ 'has nothing to constrain on this host',
|
|
237
|
+
);
|
|
238
|
+
ctx.skip();
|
|
239
|
+
return;
|
|
240
|
+
}
|
|
241
|
+
|
|
216
242
|
// ── LEG 2 — THE MUST NOT. A replay fork re-emits; it must not deliver. ───
|
|
217
243
|
const replay = await driver.post(`/v1/runs/${encodeURIComponent(sourceRunId)}:fork`, {
|
|
218
244
|
mode: 'replay',
|
|
219
245
|
});
|
|
220
|
-
if (replay.status
|
|
221
|
-
//
|
|
222
|
-
//
|
|
223
|
-
//
|
|
246
|
+
if (forkDeclined(replay.status, 'fanout-suppression replay fork')) {
|
|
247
|
+
// Leg 1 already asserted, so the FILE is `executed-pass` — but the MUST
|
|
248
|
+
// NOT was never exercised, and that is what this row must say. The
|
|
249
|
+
// explicit record wins over the file-level one (setup.ts).
|
|
250
|
+
//
|
|
251
|
+
// 404 and 403 were NOT handled here until 2026-08-25, only 501. The
|
|
252
|
+
// postgres reference host 404s this route, so this was the one scenario
|
|
253
|
+
// in the replay family that actually reached the seam — and it hard-
|
|
254
|
+
// failed `expected 404 to be 201` on every CI run of `main`, absorbed by
|
|
255
|
+
// the 85% pass-rate floor. The suite required a host to implement the
|
|
256
|
+
// route in order to say it had not implemented the route.
|
|
224
257
|
recordRequirement(
|
|
225
258
|
REQUIREMENT_ID,
|
|
226
259
|
'blocked',
|
|
227
|
-
|
|
260
|
+
`replay fork returned ${replay.status} — the re-emission this requirement is stated over never happened, `
|
|
228
261
|
+ 'so the absence of deliveries below would prove nothing',
|
|
229
262
|
);
|
|
230
263
|
ctx.skip();
|
|
@@ -30,6 +30,8 @@
|
|
|
30
30
|
|
|
31
31
|
import { describe, it, expect } from 'vitest';
|
|
32
32
|
import { driver } from '../lib/driver.js';
|
|
33
|
+
import { softSkip } from '../lib/soft-skip.js';
|
|
34
|
+
import { forkDeclined } from '../lib/fork-availability.js';
|
|
33
35
|
import { pollUntilTerminal } from '../lib/polling.js';
|
|
34
36
|
import { isFixtureAdvertised } from '../lib/fixtures.js';
|
|
35
37
|
|
|
@@ -122,6 +124,7 @@ describe.skipIf(SKIP_NO_MULTI)(
|
|
|
122
124
|
it('mid-fromSeq replay fork produces a new run that reaches `completed`', async (ctx) => {
|
|
123
125
|
const replay = await fetchReplayCapability();
|
|
124
126
|
if (replay?.supported !== true) {
|
|
127
|
+
softSkip('inapplicable', "host does not advertise `replay.supported: true` — the replay contract does not apply to it");
|
|
125
128
|
ctx.skip();
|
|
126
129
|
return;
|
|
127
130
|
}
|
|
@@ -129,6 +132,7 @@ describe.skipIf(SKIP_NO_MULTI)(
|
|
|
129
132
|
? replay.modes.filter((m): m is string => typeof m === 'string')
|
|
130
133
|
: [];
|
|
131
134
|
if (!modes.includes('replay')) {
|
|
135
|
+
softSkip('inapplicable', "host does not advertise the `replay` fork mode — this leg's rule has no path to apply");
|
|
132
136
|
ctx.skip();
|
|
133
137
|
return;
|
|
134
138
|
}
|
|
@@ -140,6 +144,7 @@ describe.skipIf(SKIP_NO_MULTI)(
|
|
|
140
144
|
// Fixture's wire shape doesn't expose node.completed(b) with a
|
|
141
145
|
// numeric sequence — skip rather than fail. Conformant hosts
|
|
142
146
|
// with the standard event shape will hit the assertions below.
|
|
147
|
+
softSkip('blocked', "the advertised fixture's wire shape exposes no numeric sequence for node.completed(b), so there is no mid-run point to fork from");
|
|
143
148
|
ctx.skip();
|
|
144
149
|
return;
|
|
145
150
|
}
|
|
@@ -149,7 +154,7 @@ describe.skipIf(SKIP_NO_MULTI)(
|
|
|
149
154
|
{ fromSeq, mode: 'replay' },
|
|
150
155
|
);
|
|
151
156
|
|
|
152
|
-
if (fork.status
|
|
157
|
+
if (forkDeclined(fork.status, 'arbitrary-event fork')) {
|
|
153
158
|
ctx.skip();
|
|
154
159
|
return;
|
|
155
160
|
}
|
|
@@ -226,7 +231,7 @@ describe.skipIf(SKIP_NO_MULTI)(
|
|
|
226
231
|
`/v1/runs/${encodeURIComponent(sourceRunId)}:fork`,
|
|
227
232
|
{ fromSeq, mode: 'replay' },
|
|
228
233
|
);
|
|
229
|
-
if (fork1.status
|
|
234
|
+
if (forkDeclined(fork1.status, 'arbitrary-event fork 1')) {
|
|
230
235
|
ctx.skip();
|
|
231
236
|
return;
|
|
232
237
|
}
|
|
@@ -238,7 +243,7 @@ describe.skipIf(SKIP_NO_MULTI)(
|
|
|
238
243
|
`/v1/runs/${encodeURIComponent(sourceRunId)}:fork`,
|
|
239
244
|
{ fromSeq, mode: 'replay' },
|
|
240
245
|
);
|
|
241
|
-
if (fork2.status
|
|
246
|
+
if (forkDeclined(fork2.status, 'arbitrary-event fork 2')) {
|
|
242
247
|
ctx.skip();
|
|
243
248
|
return;
|
|
244
249
|
}
|
|
@@ -294,6 +299,7 @@ describe.skipIf(SKIP_NO_MULTI)(
|
|
|
294
299
|
it('mid-fromSeq branch fork with empty overlay produces a new run that reaches `completed`', async (ctx) => {
|
|
295
300
|
const replay = await fetchReplayCapability();
|
|
296
301
|
if (replay?.supported !== true) {
|
|
302
|
+
softSkip('inapplicable', "host does not advertise `replay.supported: true` — the replay contract does not apply to it");
|
|
297
303
|
ctx.skip();
|
|
298
304
|
return;
|
|
299
305
|
}
|
|
@@ -25,6 +25,8 @@
|
|
|
25
25
|
|
|
26
26
|
import { describe, it, expect } from 'vitest';
|
|
27
27
|
import { driver } from '../lib/driver.js';
|
|
28
|
+
import { softSkip } from '../lib/soft-skip.js';
|
|
29
|
+
import { forkDeclined } from '../lib/fork-availability.js';
|
|
28
30
|
import { pollUntilTerminal } from '../lib/polling.js';
|
|
29
31
|
import { isFixtureAdvertised } from '../lib/fixtures.js';
|
|
30
32
|
|
|
@@ -60,6 +62,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay: fork from fromSeq=0 in replay mode', () =
|
|
|
60
62
|
// Visible skip — earlier this was a silent `return` that
|
|
61
63
|
// collapsed to a vacuous pass and made it impossible to tell
|
|
62
64
|
// unexercised tests apart from honest passes.
|
|
65
|
+
softSkip('inapplicable', "host does not advertise the `replay` fork mode — this leg's rule has no path to apply");
|
|
63
66
|
ctx.skip();
|
|
64
67
|
return;
|
|
65
68
|
}
|
|
@@ -70,7 +73,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay: fork from fromSeq=0 in replay mode', () =
|
|
|
70
73
|
{ fromSeq: 0, mode: 'replay' },
|
|
71
74
|
);
|
|
72
75
|
|
|
73
|
-
if (fork.status
|
|
76
|
+
if (forkDeclined(fork.status, 'replay fork')) return;
|
|
74
77
|
expect(fork.status, driver.describe(
|
|
75
78
|
'rest-endpoints.md POST /v1/runs/{runId}:fork',
|
|
76
79
|
'fork MUST return 201 on accepted replay',
|
|
@@ -101,6 +104,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay: fork from fromSeq=0 in branch mode with e
|
|
|
101
104
|
it('produces a new run that reaches terminal `completed`', async (ctx) => {
|
|
102
105
|
const modes = await fetchReplayModes();
|
|
103
106
|
if (!modes.includes('branch')) {
|
|
107
|
+
softSkip('inapplicable', "host does not advertise the `branch` fork mode — this leg's rule has no path to apply");
|
|
104
108
|
ctx.skip();
|
|
105
109
|
return;
|
|
106
110
|
}
|
|
@@ -111,7 +115,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay: fork from fromSeq=0 in branch mode with e
|
|
|
111
115
|
{ fromSeq: 0, mode: 'branch', runOptionsOverlay: {} },
|
|
112
116
|
);
|
|
113
117
|
|
|
114
|
-
if (fork.status
|
|
118
|
+
if (forkDeclined(fork.status, 'branch fork')) return;
|
|
115
119
|
expect(fork.status, driver.describe(
|
|
116
120
|
'rest-endpoints.md POST /v1/runs/{runId}:fork',
|
|
117
121
|
'branch fork MUST return 201',
|
|
@@ -137,6 +141,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay: validation errors', () => {
|
|
|
137
141
|
it('rejects negative fromSeq with 400', async (ctx) => {
|
|
138
142
|
const modes = await fetchReplayModes();
|
|
139
143
|
if (modes.length === 0) {
|
|
144
|
+
softSkip('inapplicable', "host advertises no usable fork mode for this leg");
|
|
140
145
|
ctx.skip();
|
|
141
146
|
return;
|
|
142
147
|
}
|
|
@@ -155,6 +160,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay: validation errors', () => {
|
|
|
155
160
|
it('rejects fromSeq beyond source event log length with 422', async (ctx) => {
|
|
156
161
|
const modes = await fetchReplayModes();
|
|
157
162
|
if (modes.length === 0) {
|
|
163
|
+
softSkip('inapplicable', "host advertises no usable fork mode for this leg");
|
|
158
164
|
ctx.skip();
|
|
159
165
|
return;
|
|
160
166
|
}
|
|
@@ -179,6 +185,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay: validation errors', () => {
|
|
|
179
185
|
// → 400) only applies on hosts that advertise the `replay` mode.
|
|
180
186
|
// A `branch`-only host has no path to even attempt the request.
|
|
181
187
|
// Visible skip rather than silent vacuous pass.
|
|
188
|
+
softSkip('inapplicable', "host does not advertise the `replay` fork mode — the runOptionsOverlay rejection rule only applies to hosts that do");
|
|
182
189
|
ctx.skip();
|
|
183
190
|
return;
|
|
184
191
|
}
|
|
@@ -200,6 +207,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay: validation errors', () => {
|
|
|
200
207
|
it('rejects fork on a non-existent run with 404', async (ctx) => {
|
|
201
208
|
const modes = await fetchReplayModes();
|
|
202
209
|
if (modes.length === 0) {
|
|
210
|
+
softSkip('inapplicable', "host advertises no usable fork mode for this leg");
|
|
203
211
|
ctx.skip();
|
|
204
212
|
return;
|
|
205
213
|
}
|
|
@@ -48,6 +48,7 @@ import { describe, it, expect } from 'vitest';
|
|
|
48
48
|
import { recordRequirement } from '../lib/requirement-ledger.js';
|
|
49
49
|
import { requirementIdForScenario } from '../lib/requirement-registry.js';
|
|
50
50
|
import { driver } from '../lib/driver.js';
|
|
51
|
+
import { forkDeclined } from '../lib/fork-availability.js';
|
|
51
52
|
import { pollUntilTerminal } from '../lib/polling.js';
|
|
52
53
|
import { isFixtureAdvertised } from '../lib/fixtures.js';
|
|
53
54
|
import { readFileSync } from 'node:fs';
|
|
@@ -173,8 +174,8 @@ describe.skipIf(SKIP_NO_FIXTURE)('replay-side-effect-suppression: a replay does
|
|
|
173
174
|
const fork = await driver.post(`/v1/runs/${encodeURIComponent(sourceRunId)}:fork`, {
|
|
174
175
|
mode: 'replay',
|
|
175
176
|
});
|
|
176
|
-
if (fork.status
|
|
177
|
-
ctx.skip();
|
|
177
|
+
if (forkDeclined(fork.status, 'side-effect-suppression replay fork')) {
|
|
178
|
+
ctx.skip();
|
|
178
179
|
return;
|
|
179
180
|
}
|
|
180
181
|
expect(fork.status, 'fork should be accepted').toBe(201);
|
|
@@ -18,6 +18,8 @@
|
|
|
18
18
|
|
|
19
19
|
import { describe, it, expect } from 'vitest';
|
|
20
20
|
import { driver } from '../lib/driver.js';
|
|
21
|
+
import { softSkip } from '../lib/soft-skip.js';
|
|
22
|
+
import { forkDeclined } from '../lib/fork-availability.js';
|
|
21
23
|
import { pollUntilTerminal } from '../lib/polling.js';
|
|
22
24
|
import { isFixtureAdvertised } from '../lib/fixtures.js';
|
|
23
25
|
|
|
@@ -69,7 +71,7 @@ function structuralShape(events: readonly RawEvent[]): Array<{ type: unknown; no
|
|
|
69
71
|
describe('replay-determinism: openwop-replay-fork profile gate', () => {
|
|
70
72
|
it('host advertising replay.supported MUST also advertise replay.modes', async () => {
|
|
71
73
|
const replay = await fetchReplayCapability();
|
|
72
|
-
if (replay === null || replay.supported !== true) return
|
|
74
|
+
if (replay === null || replay.supported !== true) return softSkip('inapplicable', 'host does not advertise `replay.supported: true` — the replay contract does not apply to it');
|
|
73
75
|
|
|
74
76
|
expect(Array.isArray(replay.modes), driver.describe(
|
|
75
77
|
'spec/v1/replay.md',
|
|
@@ -91,8 +93,8 @@ describe.skipIf(SKIP_NO_NOOP)('replay-determinism: same fromSeq + same workflow
|
|
|
91
93
|
'two replay forks of the same point produce structurally-identical event lists',
|
|
92
94
|
async () => {
|
|
93
95
|
const replay = await fetchReplayCapability();
|
|
94
|
-
if (replay === null || replay.supported !== true) return
|
|
95
|
-
if (!Array.isArray(replay.modes) || !replay.modes.includes('replay')) return
|
|
96
|
+
if (replay === null || replay.supported !== true) return softSkip('inapplicable', 'host does not advertise `replay.supported: true` — the replay contract does not apply to it');
|
|
97
|
+
if (!Array.isArray(replay.modes) || !replay.modes.includes('replay')) return softSkip('inapplicable', 'host advertises replay but not the `replay` mode — this leg is out of scope for it');
|
|
96
98
|
|
|
97
99
|
// Phase 1: complete an original run.
|
|
98
100
|
const create = await driver.post('/v1/runs', { workflowId: NOOP_WORKFLOW_ID });
|
|
@@ -105,7 +107,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay-determinism: same fromSeq + same workflow
|
|
|
105
107
|
mode: 'replay',
|
|
106
108
|
fromSeq: 0,
|
|
107
109
|
});
|
|
108
|
-
if (fork1.status
|
|
110
|
+
if (forkDeclined(fork1.status, 'determinism fork 1')) return;
|
|
109
111
|
expect(fork1.status, driver.describe(
|
|
110
112
|
'spec/v1/replay.md',
|
|
111
113
|
'POST /v1/runs/{runId}:fork with mode=replay MUST return 201',
|
|
@@ -118,7 +120,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay-determinism: same fromSeq + same workflow
|
|
|
118
120
|
mode: 'replay',
|
|
119
121
|
fromSeq: 0,
|
|
120
122
|
});
|
|
121
|
-
if (fork2.status
|
|
123
|
+
if (forkDeclined(fork2.status, 'determinism fork 2')) return;
|
|
122
124
|
expect(fork2.status).toBe(201);
|
|
123
125
|
const fork2Id = (fork2.json as { runId: string }).runId;
|
|
124
126
|
await pollUntilTerminal(fork2Id, { timeoutMs: 10_000 });
|
|
@@ -152,7 +154,7 @@ describe.skipIf(SKIP_NO_NOOP)('replay-determinism: same fromSeq + same workflow
|
|
|
152
154
|
describe.skipIf(SKIP_NO_NOOP)('replay-determinism: branch-mode is permitted to diverge', () => {
|
|
153
155
|
it('branch mode does NOT need to produce identical event sequences (negative-control)', async () => {
|
|
154
156
|
const replay = await fetchReplayCapability();
|
|
155
|
-
if (replay === null || replay.supported !== true) return;
|
|
157
|
+
if (replay === null || replay.supported !== true) return softSkip('inapplicable', 'host does not advertise `replay.supported: true` — the replay contract does not apply to it');
|
|
156
158
|
if (!Array.isArray(replay.modes) || !replay.modes.includes('branch')) return;
|
|
157
159
|
|
|
158
160
|
// Self-test on the spec interpretation: branch and replay are
|
|
@@ -16,13 +16,59 @@
|
|
|
16
16
|
* Capability-gated: skips when the host does not advertise
|
|
17
17
|
* `capabilities.webhooks.supported = true`.
|
|
18
18
|
*
|
|
19
|
-
* Operator contract
|
|
20
|
-
*
|
|
21
|
-
*
|
|
22
|
-
*
|
|
23
|
-
* `OPENWOP_WEBHOOK_ALLOW_PRIVATE=true`
|
|
24
|
-
*
|
|
25
|
-
*
|
|
19
|
+
* Operator contract — THREE gates, not one (clarified 2026-08-25).
|
|
20
|
+
*
|
|
21
|
+
* The test receiver is `http://127.0.0.1:{port}/`, and `webhooks.md`
|
|
22
|
+
* forbids it three separate times. A host honoring
|
|
23
|
+
* `OPENWOP_WEBHOOK_ALLOW_PRIVATE=true` (or an equivalent opt-in) MUST
|
|
24
|
+
* relax ALL THREE for this scenario to be witnessable:
|
|
25
|
+
*
|
|
26
|
+
* 1. **Scheme.** §"SSRF protection" bullet 1 rejects non-`https://`
|
|
27
|
+
* protocols, and §"Register" says `url` MUST be `https://`. The
|
|
28
|
+
* receiver is plain `http`. This gate is the one the flag's
|
|
29
|
+
* description omitted until 2026-08-25, and it fires FIRST on a
|
|
30
|
+
* host that validates scheme before address.
|
|
31
|
+
* 2. **Registration-time address check.** §"SSRF protection": the
|
|
32
|
+
* server MUST validate subscription URLs at registration time and
|
|
33
|
+
* reject loopback / RFC1918 / link-local / ULA / metadata.
|
|
34
|
+
* 3. **Delivery-time re-resolution.** §"Delivery-time egress
|
|
35
|
+
* validation (RFC 0093)": the dispatcher MUST re-resolve at
|
|
36
|
+
* delivery time and validate every resolved address against the
|
|
37
|
+
* same ranges.
|
|
38
|
+
*
|
|
39
|
+
* Gates 2 and 3 are INDEPENDENT MUSTs at different layers, so "which
|
|
40
|
+
* layer must the opt-in reach" is not a matter of taste: a relaxation
|
|
41
|
+
* reaching only one layer cannot produce a witness. Delivery-only leaves
|
|
42
|
+
* registration returning `400 webhook_url_rejected` (observed on a tier-2
|
|
43
|
+
* host, 2026-08-25); registration-only leaves the dispatcher re-resolving
|
|
44
|
+
* `127.0.0.1` and refusing to connect. A host whose opt-in reaches one
|
|
45
|
+
* layer is not non-conformant — it simply cannot witness this scenario,
|
|
46
|
+
* and that is a property of the test posture, not of its webhook signing.
|
|
47
|
+
*
|
|
48
|
+
* What that costs is more than a checkmark, and it is worth stating plainly
|
|
49
|
+
* because the cost is invisible from inside such a host. This scenario is
|
|
50
|
+
* typically the ONLY automated oracle a host has on its delivery header
|
|
51
|
+
* NAMES: a host-local delivery test almost always compares a hand-written
|
|
52
|
+
* literal in the test to a hand-written literal in the implementation, which
|
|
53
|
+
* is a mirror — typo both and it stays green — while the HMAC assertion
|
|
54
|
+
* beside it recomputes from `node:crypto` and is real. So the names go
|
|
55
|
+
* unchecked exactly where the bytes are checked well. All three reference
|
|
56
|
+
* hosts emitted invented header names for 16 days (openwop-examples#22) and
|
|
57
|
+
* the runs that would have caught it were the ones this scenario could not
|
|
58
|
+
* reach. A host that runs it is not thereby careful; it is thereby measured.
|
|
59
|
+
*
|
|
60
|
+
* Relaxing gates 2 and 3 is test-only posture. See
|
|
61
|
+
* `SECURITY/threat-model-secret-leakage.md` §4.9 for why a
|
|
62
|
+
* registration-time relaxation is the more dangerous of the two: it
|
|
63
|
+
* writes a durable subscription row that survives the flag being turned
|
|
64
|
+
* back off.
|
|
65
|
+
*
|
|
66
|
+
* When the host rejects with `400 webhook_url_rejected`, this scenario
|
|
67
|
+
* records `blocked` (RFC 0148 §A) — NOT a pass. Under a plain
|
|
68
|
+
* `vitest run` the console still prints "1 passed", because that is
|
|
69
|
+
* vitest reporting test outcomes rather than conformance dispositions;
|
|
70
|
+
* the run-end disposition summary (`src/global-setup.ts`) is where the
|
|
71
|
+
* `blocked` becomes visible without `--certify`.
|
|
26
72
|
*
|
|
27
73
|
* @see spec/v1/webhooks.md §"Signature scheme"
|
|
28
74
|
*/
|
package/vitest.config.ts
CHANGED
|
@@ -17,5 +17,12 @@ export default defineConfig({
|
|
|
17
17
|
// populates `lib/fixtures.ts` so `describe.skipIf(...)` predicates
|
|
18
18
|
// see the cached set when scenarios register their tests.
|
|
19
19
|
setupFiles: ['src/setup.ts'],
|
|
20
|
+
// RFC 0148 §A run-end disposition summary. `setup.ts` records an honest
|
|
21
|
+
// disposition per scenario file on EVERY run; before this, only
|
|
22
|
+
// `--certify` set OPENWOP_LEDGER_PATH, so on a plain `vitest run` the
|
|
23
|
+
// recording was computed and then discarded, leaving "N passed" as the
|
|
24
|
+
// sole artifact for rows the suite had classified `blocked`. This routes
|
|
25
|
+
// the ledger to a temp file and prints what did not witness.
|
|
26
|
+
globalSetup: ['src/global-setup.ts'],
|
|
20
27
|
},
|
|
21
28
|
});
|