@openwop/openwop-conformance 2.0.0-rc.6 → 2.0.0-rc.60
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +8 -3
- package/dist/cli.js +24 -34
- package/dist/lib/requirement-registry.js +69 -0
- package/dist/lib/scenario-disposition.js +84 -11
- package/dist/lib/seams.js +42 -0
- package/dist/lib/soft-skip.js +39 -8
- package/dist/spec-artifacts.lock.json +2 -2
- package/package.json +2 -2
- package/requirement-aliases.json +4 -1
- package/requirements.json +1124 -51
- package/scenario-majors.json +60 -3
- package/schemas/CORPUS-STAMP.json +39 -36
- package/src/cli.ts +23 -30
- package/src/global-setup.ts +11 -0
- package/src/lib/corpus-stamp.ts +24 -2
- package/src/lib/fixtures.ts +31 -0
- package/src/lib/requirement-registry.ts +68 -0
- package/src/lib/scenario-disposition.ts +88 -9
- package/src/lib/soft-skip.ts +42 -7
- package/src/lib/sse.ts +8 -0
- package/src/scenarios/era-key-stamped-v1.test.ts +156 -0
- package/src/scenarios/pause-resume.test.ts +159 -75
- package/src/scenarios/v2-advertised-fixtures-exist.test.ts +25 -43
- package/src/scenarios/v2-advertised-path-space-served.test.ts +165 -0
- package/src/scenarios/v2-coherence-not-in-bundle.test.ts +11 -2
- package/src/scenarios/v2-created-run-readable.test.ts +98 -0
- package/src/scenarios/v2-dual-stack-negotiation.test.ts +29 -1
- package/src/scenarios/v2-effect-seam-manifest.test.ts +9 -53
- package/src/scenarios/v2-effect-seam-no-refire.test.ts +88 -0
- package/src/scenarios/v2-era-2-append-vocabulary.test.ts +16 -2
- package/src/scenarios/v2-era-stamp-universal.test.ts +1 -1
- package/src/scenarios/v2-interrupt-token-scheme.test.ts +1 -1
- package/src/scenarios/v2-malformed-body-envelope.test.ts +79 -0
- package/src/scenarios/v2-run-annotation-not-event.test.ts +71 -0
- package/src/scenarios/v2-run-bulk-cancel.test.ts +91 -0
- package/src/scenarios/v2-run-cancel.test.ts +97 -0
- package/src/scenarios/v2-run-completed-outputs.test.ts +93 -0
- package/src/scenarios/v2-run-diff-identical.test.ts +75 -0
- package/src/scenarios/v2-run-fork-prefix.test.ts +140 -0
- package/src/scenarios/v2-run-fork-refusals.test.ts +70 -0
- package/src/scenarios/v2-run-options-limits.test.ts +69 -0
- package/src/scenarios/v2-run-pause-resume.test.ts +117 -0
- package/src/scenarios/v2-run-snapshot-etag.test.ts +57 -0
- package/src/scenarios/v2-sse-last-event-id.test.ts +93 -0
- package/src/scenarios/v2-stream-mode-refusal.test.ts +118 -0
- package/src/scenarios/v2-stream-sse-projection.test.ts +79 -0
- package/src/scenarios/v2-version-header-honored.test.ts +124 -0
- package/src/scenarios/version-negotiation.test.ts +25 -3
- package/src/setup.ts +104 -50
package/README.md
CHANGED
|
@@ -3,7 +3,12 @@
|
|
|
3
3
|
**openwop is an open, wire-level protocol for multi-agent workflow orchestration** — a single contract for runs in which LLM agents, deterministic tools, sub-workflows, and human reviewers collaborate, with durable suspend / resume, replay, version negotiation, and observability owned by the protocol itself. This package is the black-box conformance suite: point it at any OpenWOP-compliant server (your own or a third party's) and it issues real HTTP requests against the spec'd endpoints and asserts that responses match.
|
|
4
4
|
|
|
5
5
|
```bash
|
|
6
|
-
|
|
6
|
+
# Install BOTH packages at the SAME explicit version — the suite declares
|
|
7
|
+
# @openwop/spec-artifacts as an exact-pinned peer, and `npm i --legacy-peer-deps`
|
|
8
|
+
# on the suite alone does not pull it (a host measured "corpus stamp MISMATCH —
|
|
9
|
+
# missing @openwop/spec-artifacts" on 2026-09-05). Pre-release 2.x is on the
|
|
10
|
+
# `next` dist-tag; pin the version, never the tag (runbook §0.2b).
|
|
11
|
+
npm install @openwop/openwop-conformance@2.0.0-rc.57 @openwop/spec-artifacts@2.0.0-rc.57
|
|
7
12
|
# or run without install:
|
|
8
13
|
npx @openwop/openwop-conformance --base-url https://api.example.com --api-key hk_test_...
|
|
9
14
|
```
|
|
@@ -112,7 +117,7 @@ Exit code is non-zero on any failed assertion. `--certify` distinguishes: `0`
|
|
|
112
117
|
|
|
113
118
|
## What's Covered
|
|
114
119
|
|
|
115
|
-
The current suite has
|
|
120
|
+
The current suite has 516 scenario files under `src/scenarios/`.
|
|
116
121
|
- 2026-09-03 (suite `1.157.0 -> 1.158.0`, gap G17): NEW `idempotency-concurrent-claim.test.ts` — drives the new `host-sample-test-seams.md` §25 concurrent duplicate-delivery seam for the RFC 0150 §B / `idempotency.md` §"Concurrent duplicates (Layer 2)" atomic-claim MUST, which is unconditional and had no witness of any kind. Asserts every executor mints the SAME `logicalInvocationId` **before** asserting `delivered === 1` — without the identity check a host passes by minting different ids and never colliding, one effect because nothing raced. Not profile-gated and so not opt-out-able (the obligation is unconditional); an unmounted seam records `blocked`, which is not certifiable. Graduates `layer2-invocation-claim-atomic` reference-impl -> protocol.
|
|
117
122
|
- 2026-08-19 (suite `1.137.0 → 1.138.0`): NEW `durability-poison-exhaustion.test.ts` — RFC 0158 §C.8, the FIRST row of that RFC's conformance table to land. Asserts what `failure-path.test.ts` cannot: not just that deterministically failing work reaches terminal, but that attempts STOP — counted on the log, re-counted after a scaled quiet window, asserted unchanged. A host still redelivering records more. Seam-gated on the existing event-log seam (`blocked` = unobservable, not unmet) and outside every profile floor.
|
|
118
123
|
- 2026-08-19 (suite `1.136.15 → 1.137.0`): NEW `replay-fanout-suppression.test.ts` — capability-gated on `webhooks.supported`, **outside every profile floor**; witnesses `replay.md` §"Host-initiated fan-out is an external effect", which was the largest normative MUST NOT on the replay surface with no scenario and no SECURITY invariant. Three legs in ONE `it` against ONE receiver and ONE subscription — a positive control, the MUST NOT, and a `branch` boundary leg — because "no delivery arrived" passes identically when delivery never worked, so absence is asserted only after presence is proven on that exact wiring. A host with an SSRF guard correctly refuses the loopback receiver and records `blocked`: **unobservable, not unmet.**
|
|
@@ -457,7 +462,7 @@ Server-required (added in 1.7.0):
|
|
|
457
462
|
| ------------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
458
463
|
| **Redaction** | [`capabilities.md`](../spec/v1/capabilities.md) §"Secrets" + NFR-7 + §"aiProviders" | Vendor-neutral assertions that the server doesn't leak secret material. Three scenario groups: (a) discovery shape contract — `secrets` + `aiProviders` advertisements are well-formed regardless of `secrets.supported`; when `supported === true`, scopes MUST be non-empty + `resolution === 'host-managed'`; `byok ⊆ supported`. (b) bearer-token redaction — invalid Bearer canary in `Authorization` header is not echoed in the 401 response body. (c) credentialRef echo control — gated on `secrets.supported === true`; canary planted in `configurable.ai.credentialRef` MUST NOT appear in any RunEvent payload (poll-based capture; transport-agnostic). Uses runtime-built canary fixtures (`lib/canaries.ts`) that defeat static secret scanners. 6 scenarios. |
|
|
459
464
|
|
|
460
|
-
Current source tree:
|
|
465
|
+
Current source tree: 516 scenario files. Use [`coverage.md`](./coverage.md) for current grade/gap tracking.
|
|
461
466
|
|
|
462
467
|
## Remaining Gaps
|
|
463
468
|
|
package/dist/cli.js
CHANGED
|
@@ -46,7 +46,7 @@ import { deriveRequirementDispositions } from './lib/scenario-disposition.js';
|
|
|
46
46
|
import { scrubEvidence, evidenceSecretsFromEnv, verifyBundleV2 } from './lib/certification-bundle-verify.js';
|
|
47
47
|
import { publicKeyFromPrivate, signBundleV3, verifierSign, verifyBundleV3, witnessDigest } from './lib/certification-bundle-v3.js';
|
|
48
48
|
import { deriveProfiles, isCoreStandard, agentPlatformStatus, DEPRECATED_PROFILE_ALIASES, PROFILE_FLOOR_SCENARIOS, } from './lib/profiles.js';
|
|
49
|
-
import { setV2ProfileFloors } from './lib/requirement-registry.js';
|
|
49
|
+
import { setV2ProfileFloors, v2ProfileFloorFiles } from './lib/requirement-registry.js';
|
|
50
50
|
function parseArgs(argv) {
|
|
51
51
|
let baseUrl;
|
|
52
52
|
let apiKey;
|
|
@@ -354,37 +354,8 @@ function claimedProfilesForV2(doc, conformanceRoot) {
|
|
|
354
354
|
* fail a host for the corpus's own backlog.
|
|
355
355
|
*/
|
|
356
356
|
function v2ProfileFloors(conformanceRoot) {
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
return {};
|
|
360
|
-
let known;
|
|
361
|
-
try {
|
|
362
|
-
known = new Set(Object.keys(JSON.parse(readFileSync(resolvePath(conformanceRoot, 'scenario-majors.json'), 'utf8')).majors));
|
|
363
|
-
}
|
|
364
|
-
catch {
|
|
365
|
-
known = new Set();
|
|
366
|
-
}
|
|
367
|
-
let registry;
|
|
368
|
-
try {
|
|
369
|
-
registry = JSON.parse(readFileSync(registryPath, 'utf8'));
|
|
370
|
-
}
|
|
371
|
-
catch {
|
|
372
|
-
return {};
|
|
373
|
-
}
|
|
374
|
-
const out = {};
|
|
375
|
-
for (const p of registry.profiles ?? []) {
|
|
376
|
-
if (typeof p.id !== 'string')
|
|
377
|
-
continue;
|
|
378
|
-
const raw = Array.isArray(p.floorScenarios) ? p.floorScenarios.map(String) : [];
|
|
379
|
-
const files = [];
|
|
380
|
-
for (const entry of raw) {
|
|
381
|
-
const name = entry.startsWith('planned:') ? `v2-${entry.slice('planned:'.length)}.test.ts` : entry.endsWith('.test.ts') ? entry : `${entry}.test.ts`;
|
|
382
|
-
if (known.size === 0 || known.has(name))
|
|
383
|
-
files.push(name);
|
|
384
|
-
}
|
|
385
|
-
out[p.id] = files;
|
|
386
|
-
}
|
|
387
|
-
return out;
|
|
357
|
+
// One derivation, shared with the ledger's floor-file set (requirement-registry.ts).
|
|
358
|
+
return v2ProfileFloorFiles(conformanceRoot);
|
|
388
359
|
}
|
|
389
360
|
/**
|
|
390
361
|
* Reduce a vitest JSON report into a per-scenario-file terminal state, keyed by
|
|
@@ -459,6 +430,16 @@ async function resolveTargetMajor(args, baseUrl, apiKey, conformanceRoot) {
|
|
|
459
430
|
// Say so, or an operator reads a v1 run as the host's only option.
|
|
460
431
|
if (major === 1 && dualStack)
|
|
461
432
|
source += ' (the host also serves 2.x — pass --target-major 2 to measure it)';
|
|
433
|
+
// The RUNNER's own process must see the target major, not only the vitest
|
|
434
|
+
// child (rc.55). `requirementIdForFile()` decides "is this file a floor?"
|
|
435
|
+
// through `targetMajor()` = process.env; until rc.55 `--target-major 2` set
|
|
436
|
+
// the env on the child alone, so the worker recorded every v2 floor file
|
|
437
|
+
// under `openwop.floor.<stem>` while the runner looked it up as
|
|
438
|
+
// `openwop.scenario.<stem>`, found nothing, emitted a report-derived pass
|
|
439
|
+
// with no assertion count, and its own verifier rejected the bundle as a
|
|
440
|
+
// `vacuous-pass` emitter defect (exit 2, nothing written). The child env is
|
|
441
|
+
// spread from process.env after this, so one assignment covers both.
|
|
442
|
+
process.env['OPENWOP_TARGET_MAJOR'] = String(major);
|
|
462
443
|
const manifest = JSON.parse(readFileSync(resolvePath(conformanceRoot, 'scenario-majors.json'), 'utf8'));
|
|
463
444
|
const files = Object.entries(manifest.majors).filter(([, m]) => m.includes(major)).map(([f]) => `src/scenarios/${f}`);
|
|
464
445
|
return { major, files, source };
|
|
@@ -636,15 +617,24 @@ async function runCertify(args, baseUrl, apiKey) {
|
|
|
636
617
|
const doc3 = document;
|
|
637
618
|
const protocolVersions = Array.isArray(doc3['protocolVersions']) ? doc3['protocolVersions'] : [String(doc3['protocolVersion'] ?? '')];
|
|
638
619
|
const preferredVersion = typeof doc3['preferredVersion'] === 'string' ? doc3['preferredVersion'] : (protocolVersions[0] ?? '');
|
|
639
|
-
// witnessCount: executed-pass rows on the profile's floor
|
|
620
|
+
// witnessCount: witnessed executed-pass rows on the profile's floor. At major
|
|
621
|
+
// 2 the verdict already counted them against the declaration's floor
|
|
622
|
+
// (`witnessedPasses`); the v1 hand table below knows no v2 file and printed
|
|
623
|
+
// 0 for every v2 profile until rc.45 — the third unjoined floor site.
|
|
624
|
+
const verdictFor = (profile) => derived.verdicts.find((v) => v.profile === profile);
|
|
640
625
|
const witnessCountFor = (profile) => {
|
|
626
|
+
if (target.major === 2)
|
|
627
|
+
return verdictFor(profile)?.witnessedPasses ?? 0;
|
|
641
628
|
const floor = PROFILE_FLOOR_SCENARIOS[profile];
|
|
642
629
|
if (!floor)
|
|
643
630
|
return 0;
|
|
644
631
|
const onFloor = (scenario) => floor.required.includes(scenario) || (floor.requiredAnyPrefix ?? []).some((pre) => scenario.startsWith(pre));
|
|
645
632
|
return derived.requirements.filter((r) => r.disposition === 'executed-pass' && onFloor(r.scenarioId)).length;
|
|
646
633
|
};
|
|
647
|
-
|
|
634
|
+
// `certified` IS the verdict (RFC 0148 §A; RFC 0168 §E.1 adds the bundle-wide
|
|
635
|
+
// blocked rule). Until rc.45 it was `!notHeld && !rejected && blocked === 0`
|
|
636
|
+
// and never read `certifiable` — an empty v2 floor certified on no evidence.
|
|
637
|
+
const claimed3 = claimedProfiles.filter((p) => !(p in DEPRECATED_PROFILE_ALIASES)).map((p) => ({ id: p, evidenceTier: args.evidenceTier, witnessCount: witnessCountFor(p), certified: (verdictFor(p)?.certifiable ?? false) && !notHeld.has(p) && !rejectedProfiles.some((v) => v.profile === p) && totals3.blocked === 0 }));
|
|
648
638
|
let relaxations;
|
|
649
639
|
if (process.env['OPENWOP_HOST_RELAXATIONS']) {
|
|
650
640
|
try {
|
|
@@ -21,6 +21,9 @@
|
|
|
21
21
|
* resolves an unrecorded requirement to `blocked`.
|
|
22
22
|
*/
|
|
23
23
|
import { PROFILE_FLOOR_SCENARIOS } from './profiles.js';
|
|
24
|
+
import { createRequire } from 'node:module';
|
|
25
|
+
import { dirname, resolve as resolvePath } from 'node:path';
|
|
26
|
+
import { existsSync, readFileSync } from 'node:fs';
|
|
24
27
|
/** `runs-lifecycle.test.ts` → `openwop.floor.runs-lifecycle`. */
|
|
25
28
|
export function requirementIdForScenario(scenarioFile) {
|
|
26
29
|
return `openwop.floor.${scenarioFile.replace(/\.test\.ts$/, '')}`;
|
|
@@ -90,6 +93,16 @@ let v2Floors = null;
|
|
|
90
93
|
export function setV2ProfileFloors(floors) {
|
|
91
94
|
v2Floors = floors;
|
|
92
95
|
}
|
|
96
|
+
/**
|
|
97
|
+
* True while a major-2 floor map is installed. The v1 hand table
|
|
98
|
+
* (`PROFILE_FLOOR_SCENARIOS`) MUST NOT be consulted for any per-profile fact
|
|
99
|
+
* (`discoveryOnly`, `runtimeDerived`, the floor itself, the witness count) while
|
|
100
|
+
* this is true — every one of those reads is a floor site, and rc.41 found two
|
|
101
|
+
* of four unjoined; rc.45 found the other two.
|
|
102
|
+
*/
|
|
103
|
+
export function v2FloorsActive() {
|
|
104
|
+
return v2Floors !== null;
|
|
105
|
+
}
|
|
93
106
|
export function requirementsFor(profile, document) {
|
|
94
107
|
if (v2Floors !== null) {
|
|
95
108
|
const files = v2Floors[profile];
|
|
@@ -128,3 +141,59 @@ export function allRequirements() {
|
|
|
128
141
|
}
|
|
129
142
|
return [...ids].sort();
|
|
130
143
|
}
|
|
144
|
+
/**
|
|
145
|
+
* The MAJOR-2 floors, derived from `spec/v2/profiles.json` (itself generated
|
|
146
|
+
* from `spec/v2/declaration.json`), keyed by profile id and resolved to
|
|
147
|
+
* scenario file names. A `planned:<stem>` entry names `v2-<stem>.test.ts`; an
|
|
148
|
+
* entry is kept only if `scenario-majors.json` knows the file.
|
|
149
|
+
*
|
|
150
|
+
* This function exists in ONE place on purpose. Until 2026-09-05 the CLI
|
|
151
|
+
* derived these floors privately for `--certify` while the ledger decided
|
|
152
|
+
* "is this file a floor?" from PROFILE_FLOOR_SCENARIOS — the v1 hand table,
|
|
153
|
+
* which knows no v2 file. So a v2 floor file was MINTED as
|
|
154
|
+
* `openwop.scenario.v2-…` and LOOKED UP at certify time as
|
|
155
|
+
* `openwop.floor.v2-…`: 101 executed-pass rows, `witnessCount: 0` on both
|
|
156
|
+
* claimed profiles, and `REJECTING — openwop-discovery-core: unclassified` on
|
|
157
|
+
* a tier-1 host's first production bundle. Two sources of the same fact, one
|
|
158
|
+
* of them stale, and the join between them silent.
|
|
159
|
+
*/
|
|
160
|
+
export function v2ProfileFloorFiles(conformanceRoot) {
|
|
161
|
+
const candidates = [];
|
|
162
|
+
try {
|
|
163
|
+
const req = createRequire(resolvePath(conformanceRoot, 'package.json'));
|
|
164
|
+
candidates.push(resolvePath(dirname(req.resolve('@openwop/spec-artifacts/package.json')), 'spec', 'v2', 'profiles.json'));
|
|
165
|
+
}
|
|
166
|
+
catch { /* not installed as a package; the repo-layout candidates below */ }
|
|
167
|
+
candidates.push(resolvePath(conformanceRoot, 'spec', 'v2', 'profiles.json'), resolvePath(conformanceRoot, '..', 'spec', 'v2', 'profiles.json'));
|
|
168
|
+
const registryPath = candidates.find((c) => existsSync(c));
|
|
169
|
+
if (registryPath === undefined)
|
|
170
|
+
return {};
|
|
171
|
+
let known;
|
|
172
|
+
try {
|
|
173
|
+
known = new Set(Object.keys(JSON.parse(readFileSync(resolvePath(conformanceRoot, 'scenario-majors.json'), 'utf8')).majors));
|
|
174
|
+
}
|
|
175
|
+
catch {
|
|
176
|
+
known = new Set();
|
|
177
|
+
}
|
|
178
|
+
let registry;
|
|
179
|
+
try {
|
|
180
|
+
registry = JSON.parse(readFileSync(registryPath, 'utf8'));
|
|
181
|
+
}
|
|
182
|
+
catch {
|
|
183
|
+
return {};
|
|
184
|
+
}
|
|
185
|
+
const out = {};
|
|
186
|
+
for (const p of registry.profiles ?? []) {
|
|
187
|
+
if (typeof p.id !== 'string')
|
|
188
|
+
continue;
|
|
189
|
+
const raw = Array.isArray(p.floorScenarios) ? p.floorScenarios.map(String) : [];
|
|
190
|
+
const files = [];
|
|
191
|
+
for (const entry of raw) {
|
|
192
|
+
const name = entry.startsWith('planned:') ? `v2-${entry.slice('planned:'.length)}.test.ts` : entry.endsWith('.test.ts') ? entry : `${entry}.test.ts`;
|
|
193
|
+
if (known.size === 0 || known.has(name))
|
|
194
|
+
files.push(name);
|
|
195
|
+
}
|
|
196
|
+
out[p.id] = files;
|
|
197
|
+
}
|
|
198
|
+
return out;
|
|
199
|
+
}
|
|
@@ -27,7 +27,10 @@
|
|
|
27
27
|
*/
|
|
28
28
|
import { scenarioFileOfItId } from './requirement-ids.js';
|
|
29
29
|
import { PROFILE_FLOOR_SCENARIOS } from './profiles.js';
|
|
30
|
-
import {
|
|
30
|
+
import { targetMajor } from './seams.js';
|
|
31
|
+
import { PKG_ROOT_PATH } from './paths.js';
|
|
32
|
+
import { v2ProfileFloorFiles } from './requirement-registry.js';
|
|
33
|
+
import { requirementIdForScenario, requirementIdForPrefix, requirementsFor, v2FloorsActive } from './requirement-registry.js';
|
|
31
34
|
import { UNCLASSIFIED_RETURN_DETAIL } from './soft-skip.js';
|
|
32
35
|
import { SPEC_COHERENCE_SCENARIOS, SPEC_COHERENCE_DETAIL } from './spec-coherence.js';
|
|
33
36
|
import { CERTIFIABLE } from './requirement-ledger.js';
|
|
@@ -41,6 +44,22 @@ export function floorScenarioFiles() {
|
|
|
41
44
|
for (const f of c.required)
|
|
42
45
|
out.add(f);
|
|
43
46
|
}
|
|
47
|
+
// At target major 2 the floors come from the declaration, not the v1 hand
|
|
48
|
+
// table. The ledger and --certify MUST agree on this set, or a floor file is
|
|
49
|
+
// minted `openwop.scenario.*` and looked up as `openwop.floor.*` — which is
|
|
50
|
+
// exactly what refused a tier-1 host's first production bundle over 101
|
|
51
|
+
// executed-pass rows (see v2ProfileFloorFiles).
|
|
52
|
+
//
|
|
53
|
+
// rc.55: the runner installs the v2 floor map (`setV2ProfileFloors`) but
|
|
54
|
+
// reads `targetMajor()` from ITS OWN process.env, which `--target-major 2`
|
|
55
|
+
// never set — so the worker (env set) minted `openwop.floor.v2-…` and the
|
|
56
|
+
// runner (env unset) looked up `openwop.scenario.v2-…`. Either signal is
|
|
57
|
+
// the same fact; honour both so the two halves cannot disagree again.
|
|
58
|
+
if (targetMajor() === 2 || v2FloorsActive()) {
|
|
59
|
+
for (const files of Object.values(v2ProfileFloorFiles(PKG_ROOT_PATH)))
|
|
60
|
+
for (const f of files)
|
|
61
|
+
out.add(f);
|
|
62
|
+
}
|
|
44
63
|
return out;
|
|
45
64
|
}
|
|
46
65
|
/** The requirement id a scenario FILE records under: the §A floor id when the
|
|
@@ -58,6 +77,38 @@ export function requirementIdForFile(basename) {
|
|
|
58
77
|
* the thing that passed.
|
|
59
78
|
*/
|
|
60
79
|
export const PARTIAL_WITNESS_PREFIX = 'partial-witness: ';
|
|
80
|
+
/**
|
|
81
|
+
* The per-`it` record (RFC 0148 §A at test granularity), as `setup.ts`
|
|
82
|
+
* computes it in `afterEach`. Pure so a lib test can pin it:
|
|
83
|
+
* - fail ⇒ executed-fail (detail = the first error message)
|
|
84
|
+
* - pass with ≥ 1 assertion ⇒ executed-pass
|
|
85
|
+
* - a behaviorGate entry journaled during the test ⇒ that gate's disposition
|
|
86
|
+
* - pass with 0 assertions ⇒ the softSkip note written DURING THIS TEST
|
|
87
|
+
* (`inapplicable` / `skipped` / `blocked`, worst-first),
|
|
88
|
+
* else `blocked` + the unclassified-return marker
|
|
89
|
+
* - vitest skip (ctx.skip / it.skip) ⇒ the note written before the skip, else `skipped`
|
|
90
|
+
*
|
|
91
|
+
* rc.56: the fourth line is new. Until then a zero-assertion pass consulted the
|
|
92
|
+
* journal only, so a leg that returned `softSkip('inapplicable', …)` was
|
|
93
|
+
* recorded `blocked / unclassified return` at `it` granularity while its file
|
|
94
|
+
* row (which does read the notes — `resolveFileRecord`) was `inapplicable`.
|
|
95
|
+
* A bundle with any `blocked` row does not certify (RFC 0168 §E.1), so the
|
|
96
|
+
* dishonest per-`it` rows denied certification to every profile on a host that
|
|
97
|
+
* simply did not advertise the gated surface.
|
|
98
|
+
*/
|
|
99
|
+
export function resolveItRecord(state, assertionCalls, gate, noted, firstError) {
|
|
100
|
+
if (state === 'fail')
|
|
101
|
+
return { disposition: 'executed-fail', detail: `the test executed and failed: ${(firstError ?? 'no message').slice(0, 300)}` };
|
|
102
|
+
if (state === 'pass' && assertionCalls > 0)
|
|
103
|
+
return { disposition: 'executed-pass' };
|
|
104
|
+
if (gate !== undefined)
|
|
105
|
+
return { disposition: gate.disposition, detail: gate.detail ?? `${gate.disposition} (gate recorded no reason)` };
|
|
106
|
+
if (noted !== null)
|
|
107
|
+
return { disposition: noted.kind, detail: noted.reason };
|
|
108
|
+
if (state === 'pass')
|
|
109
|
+
return { disposition: 'blocked', detail: 'unclassified return: the test passed with zero assertions and recorded no reason — RFC 0148 §A resolves it to blocked, never to a pass' };
|
|
110
|
+
return { disposition: 'skipped', detail: 'vitest skipped the test (ctx.skip / it.skip) without a recorded gate reason' };
|
|
111
|
+
}
|
|
61
112
|
/** Worker half: fold a file's per-test states (+ any gate-recorded reason) into
|
|
62
113
|
* the ONE disposition the file records. */
|
|
63
114
|
export function fileDisposition(states, gateReason, assertionCount) {
|
|
@@ -207,11 +258,18 @@ document) {
|
|
|
207
258
|
rows.push(row);
|
|
208
259
|
perFile.set(file, row);
|
|
209
260
|
}
|
|
210
|
-
// Prefix requirements: derived from the matching files.
|
|
261
|
+
// Prefix requirements: derived from the matching files. These are the v1 hand
|
|
262
|
+
// table's `requiredAnyPrefix` groups (`interrupt-`); at major 2 the floors are
|
|
263
|
+
// the declaration's and have no prefix groups — and the v1 `interrupt-*` files
|
|
264
|
+
// never run at major 2, so until rc.45 every major-2 bundle carried one
|
|
265
|
+
// `openwop.floor.any.interrupt-` row recorded `blocked` ("no interrupt-*
|
|
266
|
+
// scenario ran"), which by RFC 0168 §E.1 denied certification to every
|
|
267
|
+
// profile on every major-2 bundle. The fifth unjoined floor site.
|
|
211
268
|
const prefixIds = new Set();
|
|
212
|
-
|
|
213
|
-
for (const
|
|
214
|
-
|
|
269
|
+
if (!v2FloorsActive())
|
|
270
|
+
for (const floor of Object.values(PROFILE_FLOOR_SCENARIOS))
|
|
271
|
+
for (const p of floor.requiredAnyPrefix ?? [])
|
|
272
|
+
prefixIds.add(p);
|
|
215
273
|
for (const prefix of [...prefixIds].sort()) {
|
|
216
274
|
const matching = [...perFile.entries()].filter(([f]) => f.startsWith(prefix)).map(([, r]) => r);
|
|
217
275
|
const id = requirementIdForPrefix(prefix);
|
|
@@ -280,12 +338,18 @@ document) {
|
|
|
280
338
|
if (ids === null) {
|
|
281
339
|
const floor = PROFILE_FLOOR_SCENARIOS[profile];
|
|
282
340
|
const why = floor === undefined ? `(no floor defined for ${profile})` : `(discovery-conditional floor for ${profile} is unevaluable without the discovery document)`;
|
|
283
|
-
verdicts.push({ profile, unclassified: [], blocking: [why], certifiable: false, runtimeDerived: false, held: false });
|
|
341
|
+
verdicts.push({ profile, unclassified: [], blocking: [why], certifiable: false, runtimeDerived: false, held: false, witnessedPasses: 0 });
|
|
284
342
|
continue;
|
|
285
343
|
}
|
|
286
344
|
const unclassified = [];
|
|
287
345
|
const blocking = [];
|
|
288
346
|
let witnessedPasses = 0;
|
|
347
|
+
// At major 2 the floor is the declaration's (`v2ProfileFloorFiles`), and
|
|
348
|
+
// the v1 hand table says nothing about these profiles: not their floor, not
|
|
349
|
+
// `discoveryOnly`, not `runtimeDerived`. Reading it here was the fourth
|
|
350
|
+
// unjoined floor site — `openwop-discovery-core` is `discoveryOnly` in v1
|
|
351
|
+
// terms, so at major 2 it certified whatever its v2 floor file said.
|
|
352
|
+
const atMajor2 = v2FloorsActive();
|
|
289
353
|
for (const id of ids) {
|
|
290
354
|
const r = rowById.get(id);
|
|
291
355
|
const fromLedger = byId.has(id) || (r !== undefined && r.scenarioId.endsWith('*'));
|
|
@@ -307,25 +371,34 @@ document) {
|
|
|
307
371
|
if (r === undefined || silent || vacuous)
|
|
308
372
|
unclassified.push(id);
|
|
309
373
|
// Unclassified always blocks: a requirement nobody recorded cannot certify.
|
|
310
|
-
|
|
374
|
+
// A `skipped` floor row at major 2 is an opt-in the host withheld — the
|
|
375
|
+
// suite was not allowed to look. That is not evidence the requirement
|
|
376
|
+
// holds; it blocks the floor (a v1 floor keeps the CERTIFIABLE reading).
|
|
377
|
+
if (r === undefined || silent || vacuous || !CERTIFIABLE.includes(r.disposition) || (atMajor2 && r.disposition === 'skipped'))
|
|
311
378
|
blocking.push(id);
|
|
312
379
|
}
|
|
313
380
|
// discoveryOnly floors have ids.length === 0 and certify by design here (the
|
|
314
381
|
// requirement-ledger's verifyProfileRequirements is stricter; the runner
|
|
315
|
-
// consults PROFILE_FLOOR_SCENARIOS.discoveryOnly separately).
|
|
316
|
-
|
|
317
|
-
const
|
|
382
|
+
// consults PROFILE_FLOOR_SCENARIOS.discoveryOnly separately). Neither flag
|
|
383
|
+
// exists at major 2.
|
|
384
|
+
const discoveryOnly = !atMajor2 && PROFILE_FLOOR_SCENARIOS[profile]?.discoveryOnly === true;
|
|
385
|
+
const runtimeDerived = !atMajor2 && PROFILE_FLOOR_SCENARIOS[profile]?.runtimeDerived === true;
|
|
318
386
|
// A runtime-derived profile is HELD only when every floor row is a witnessed
|
|
319
387
|
// pass ("derivable from which scenarios pass" — profiles.md). Anything else
|
|
320
388
|
// means the host does not hold it: not a rejection, not a blocked claim.
|
|
321
389
|
const held = ids.length > 0 && witnessedPasses === ids.length;
|
|
390
|
+
// Major 2: a floor certifies only on a witnessed pass. `inapplicable` rows
|
|
391
|
+
// are honest per file (capabilities.md §2 requires the omission) but a floor
|
|
392
|
+
// that is inapplicable end to end has witnessed nothing and certifies nothing.
|
|
393
|
+
const certifiableAt2 = ids.length > 0 && blocking.length === 0 && witnessedPasses >= 1;
|
|
322
394
|
verdicts.push({
|
|
323
395
|
profile,
|
|
324
396
|
unclassified: runtimeDerived && !held ? [] : unclassified,
|
|
325
397
|
blocking: runtimeDerived && !held ? ids.filter((id) => rowById.get(id)?.disposition !== 'executed-pass' || (rowById.get(id)?.assertionCount ?? 0) === 0) : blocking,
|
|
326
|
-
certifiable: runtimeDerived ? held : discoveryOnly || (ids.length > 0 && blocking.length === 0),
|
|
398
|
+
certifiable: atMajor2 ? certifiableAt2 : runtimeDerived ? held : discoveryOnly || (ids.length > 0 && blocking.length === 0),
|
|
327
399
|
runtimeDerived,
|
|
328
400
|
held,
|
|
401
|
+
witnessedPasses,
|
|
329
402
|
});
|
|
330
403
|
}
|
|
331
404
|
return { requirements: rows, totals, verdicts, rejectUnclassified: verdicts.some((v) => v.unclassified.length > 0), ledgerPresent };
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Suite 2.0.0 — the seams profile (RFC 0168 §C; `openwop-conformance-seams-v2`).
|
|
3
|
+
*
|
|
4
|
+
* v1 scenarios reach the host-sample seams at `/v1/host/sample/…`,
|
|
5
|
+
* `/v1/host/workspace/files…` and `/v1/packs-test/…`. In v2 those operations
|
|
6
|
+
* leave the canonical API for `api/seams-v2.yaml` under `/conformance/seams/…`
|
|
7
|
+
* (RFC 0168 §C.2). Under target major 2 the driver rewrites the v1 seam path
|
|
8
|
+
* to its v2 address, so the 165 scenarios that name a seam keep working
|
|
9
|
+
* without a sweep; the sweep to `seamPath()` is the P3-F polish.
|
|
10
|
+
*
|
|
11
|
+
* How a v2 host advertises the profile: RFC 0168 §C.1 forbids a `testSeams`
|
|
12
|
+
* capability flag and RFC 0169 §C.1 removes the root `profiles[]`; the
|
|
13
|
+
* reconciliation (recorded against RFC 0168 at its flip) is the
|
|
14
|
+
* `conformance` METADATA key (RFC 0169 §A.1a): `conformance.seamsProfile:
|
|
15
|
+
* "openwop-conformance-seams-v2"`. Metadata, not a capability family.
|
|
16
|
+
*/
|
|
17
|
+
export const SEAMS_PROFILE_ID = 'openwop-conformance-seams-v2';
|
|
18
|
+
export const SEAMS_PREFIX = '/conformance/seams';
|
|
19
|
+
const V1_SEAM_PREFIXES = [
|
|
20
|
+
[/^\/v1\/host\/sample\//, `${SEAMS_PREFIX}/sample/`],
|
|
21
|
+
[/^\/v1\/host\/workspace\/files/, `${SEAMS_PREFIX}/workspace/files`],
|
|
22
|
+
[/^\/v1\/packs-test\//, `${SEAMS_PREFIX}/packs-test/`],
|
|
23
|
+
];
|
|
24
|
+
/** The v2 address of a v1 seam path; a non-seam path is returned unchanged. */
|
|
25
|
+
export function seamPath(v1Path) {
|
|
26
|
+
for (const [re, to] of V1_SEAM_PREFIXES)
|
|
27
|
+
if (re.test(v1Path))
|
|
28
|
+
return v1Path.replace(re, to);
|
|
29
|
+
return v1Path;
|
|
30
|
+
}
|
|
31
|
+
export function isSeamPath(path) {
|
|
32
|
+
return V1_SEAM_PREFIXES.some(([re]) => re.test(path)) || path.startsWith(`${SEAMS_PREFIX}/`);
|
|
33
|
+
}
|
|
34
|
+
/** Whether a (v2) discovery document advertises the seams profile. */
|
|
35
|
+
export function seamsProfileAdvertised(doc) {
|
|
36
|
+
const conf = doc?.['conformance'];
|
|
37
|
+
return typeof conf === 'object' && conf !== null && conf['seamsProfile'] === SEAMS_PROFILE_ID;
|
|
38
|
+
}
|
|
39
|
+
/** The target major the runner selected (RFC 0168 §D.3); 1 when unset. */
|
|
40
|
+
export function targetMajor() {
|
|
41
|
+
return process.env['OPENWOP_TARGET_MAJOR'] === '2' ? 2 : 1;
|
|
42
|
+
}
|
package/dist/lib/soft-skip.js
CHANGED
|
@@ -22,12 +22,24 @@
|
|
|
22
22
|
* resolves it to blocked" — and stays UNCLASSIFIED for certification (a floor
|
|
23
23
|
* row with that disposition still rejects), so the honest bundle row and the
|
|
24
24
|
* pressure to say why both survive.
|
|
25
|
+
*
|
|
26
|
+
* rc.56: every note also carries a sequence number, so the per-`it` row can
|
|
27
|
+
* read the notes written DURING ITS OWN TEST (`softSkipMark()` at test start,
|
|
28
|
+
* `softSkipDispositionSince(file, mark)` at test end). Until rc.56 the
|
|
29
|
+
* per-`it` row consulted only the journal's `behaviorGate` entries, never a
|
|
30
|
+
* softSkip note, so a leg that returned `softSkip('inapplicable', 'a2a facet
|
|
31
|
+
* not advertised')` was recorded `blocked / unclassified return` at `it`
|
|
32
|
+
* granularity while its file row was honestly `inapplicable` — and a bundle
|
|
33
|
+
* with any `blocked` row does not certify (RFC 0168 §E.1). Forty-five such
|
|
34
|
+
* rows on a host that simply does not advertise A2A/MCP denied certification
|
|
35
|
+
* to every profile it claimed.
|
|
25
36
|
*/
|
|
26
37
|
import { expect } from 'vitest';
|
|
27
38
|
import { basename } from 'node:path';
|
|
28
39
|
/** Detail marker the runner writes for a zero-assertion file that noted nothing. */
|
|
29
40
|
export const UNCLASSIFIED_RETURN_DETAIL = 'every test returned early with zero assertions and no recorded reason — unclassified return; RFC 0148 §A resolves it to blocked (add softSkip(kind, reason) at the early return)';
|
|
30
41
|
const notes = new Map();
|
|
42
|
+
let seq = 0;
|
|
31
43
|
function currentFile() {
|
|
32
44
|
try {
|
|
33
45
|
const p = expect.getState().testPath;
|
|
@@ -43,8 +55,9 @@ export function softSkip(kind, reason) {
|
|
|
43
55
|
if (file === null)
|
|
44
56
|
return undefined;
|
|
45
57
|
const arr = notes.get(file) ?? [];
|
|
46
|
-
|
|
47
|
-
|
|
58
|
+
// Every call is recorded with its own sequence number so a per-test window
|
|
59
|
+
// sees it; the file-level join de-duplicates identical (kind, reason) pairs.
|
|
60
|
+
arr.push({ kind, reason, seq: ++seq });
|
|
48
61
|
notes.set(file, arr);
|
|
49
62
|
return undefined;
|
|
50
63
|
}
|
|
@@ -62,6 +75,17 @@ export function seamAbsent(reason) {
|
|
|
62
75
|
return softSkip('blocked', reason);
|
|
63
76
|
}
|
|
64
77
|
const RANK = { blocked: 0, skipped: 1, inapplicable: 2 };
|
|
78
|
+
function fold(arr) {
|
|
79
|
+
if (arr.length === 0)
|
|
80
|
+
return null;
|
|
81
|
+
const uniq = [];
|
|
82
|
+
for (const n of arr)
|
|
83
|
+
if (!uniq.some((u) => u.kind === n.kind && u.reason === n.reason))
|
|
84
|
+
uniq.push(n);
|
|
85
|
+
const kind = [...uniq].sort((a, b) => RANK[a.kind] - RANK[b.kind])[0].kind;
|
|
86
|
+
const reason = uniq.map((n) => (uniq.length > 1 ? `[${n.kind}] ${n.reason}` : n.reason)).join('; ');
|
|
87
|
+
return { kind, reason };
|
|
88
|
+
}
|
|
65
89
|
/**
|
|
66
90
|
* The noted disposition for a file, worst-first when mixed (`blocked` beats
|
|
67
91
|
* `skipped` beats `inapplicable` — a file that could not check one thing is
|
|
@@ -69,14 +93,21 @@ const RANK = { blocked: 0, skipped: 1, inapplicable: 2 };
|
|
|
69
93
|
* the reasons joined. `null` when nothing was noted.
|
|
70
94
|
*/
|
|
71
95
|
export function softSkipDisposition(file) {
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
96
|
+
return fold(notes.get(file) ?? []);
|
|
97
|
+
}
|
|
98
|
+
/** A position in the note sequence; pass it to `softSkipDispositionSince`. */
|
|
99
|
+
export function softSkipMark() {
|
|
100
|
+
return seq;
|
|
101
|
+
}
|
|
102
|
+
/**
|
|
103
|
+
* The noted disposition for a file counting only notes written AFTER `mark`
|
|
104
|
+
* — the notes of the test that is ending. Same fold as the file rule.
|
|
105
|
+
*/
|
|
106
|
+
export function softSkipDispositionSince(file, mark) {
|
|
107
|
+
return fold((notes.get(file) ?? []).filter((n) => n.seq > mark));
|
|
78
108
|
}
|
|
79
109
|
/** Test hook. */
|
|
80
110
|
export function resetSoftSkips() {
|
|
81
111
|
notes.clear();
|
|
112
|
+
seq = 0;
|
|
82
113
|
}
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
2
|
"package": "@openwop/spec-artifacts",
|
|
3
|
-
"version": "2.0.0-rc.
|
|
4
|
-
"stampSha256": "
|
|
3
|
+
"version": "2.0.0-rc.60",
|
|
4
|
+
"stampSha256": "a8d45fba2fd662735347d91b9dda9e16486b50896a866a54b679a7a0f7129cae"
|
|
5
5
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@openwop/openwop-conformance",
|
|
3
|
-
"version": "2.0.0-rc.
|
|
3
|
+
"version": "2.0.0-rc.60",
|
|
4
4
|
"description": "Production-ready black-box conformance suite for OpenWOP v1.0 compliant servers.",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -56,6 +56,6 @@
|
|
|
56
56
|
"@openwop/spec-artifacts": "file:../spec-artifacts"
|
|
57
57
|
},
|
|
58
58
|
"peerDependencies": {
|
|
59
|
-
"@openwop/spec-artifacts": "2.0.0-rc.
|
|
59
|
+
"@openwop/spec-artifacts": "2.0.0-rc.60"
|
|
60
60
|
}
|
|
61
61
|
}
|
package/requirement-aliases.json
CHANGED
|
@@ -1,4 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"$comment": "old id → new id. Add a row when a test title is reworded so bundles citing the old id still resolve.",
|
|
3
|
-
"aliases": {
|
|
3
|
+
"aliases": {
|
|
4
|
+
"openwop.it.v2-run-cancel.cancelling-a-terminal-run-answers-409-run-terminal-or-200-idempotently-the-prose": "openwop.it.v2-run-cancel.cancelling-a-terminal-run-is-refused-409-run-terminal",
|
|
5
|
+
"openwop.it.pause-resume.pause-on-an-already-paused-run-is-a-no-op-200-202-idempotent": "openwop.it.pause-resume.pause-on-an-already-paused-run-returns-409-with-details-runstatus-paused-unless"
|
|
6
|
+
}
|
|
4
7
|
}
|