@yanlinglabs/winter-provider-conformance 0.0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 yanlingLabs
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/NOTICE ADDED
@@ -0,0 +1,41 @@
1
+ NOTICE
2
+ ======
3
+
4
+ Third-party attributions for the Winter agent SDK.
5
+
6
+ This file records material that Winter DERIVED FROM third-party artifacts. Winter copies no source
7
+ code from any of them; what it takes is protocol facts — endpoint URLs, form-field names, client
8
+ identifiers and scope strings — which it needs in order to speak to a vendor's service at all. Each
9
+ entry names the exact commit the values were read at, so the derivation is re-checkable.
10
+
11
+
12
+ -------------------------------------------------------------------------------
13
+ xai-org/grok-build
14
+ -------------------------------------------------------------------------------
15
+
16
+ Repository: https://github.com/xai-org/grok-build
17
+ Commit: 72a61251fcffb464bcc687aeb5a998e5a98ec0c9
18
+ License: Apache License, Version 2.0
19
+ https://www.apache.org/licenses/LICENSE-2.0
20
+ Copyright: Copyright 2023-2026 SpaceXAI
21
+
22
+ Winter's `xai-oauth` provider derives the following constants and request shapes from this
23
+ repository's authentication module, in order to perform its own OAuth 2.0 device authorization
24
+ grant (RFC 8628) against xAI's public, secret-less OAuth client:
25
+
26
+ - the public OAuth client identifier
27
+ - the OAuth issuer, device-authorization endpoint and token endpoint
28
+ - the requested scope set
29
+ - the name of the form field the flow carries a client identity in
30
+ - the shape (field names) of the device-authorization and token requests
31
+
32
+ No source code from this repository is copied into Winter, and no part of Winter is a derivative
33
+ work of it. The derivation is recorded, with per-value line citations, in:
34
+
35
+ packages/conformance/compat/xai/grok-build/derived-shapes-p6b-xai.md
36
+
37
+ Winter identifies ITSELF in that flow. It sends its own `User-Agent` (`winter-agent-sdk/<version>`)
38
+ and its own name in the flow's identity field, and it does not send xAI's product-identity or
39
+ telemetry headers. Winter is not affiliated with or endorsed by SpaceXAI, and "Grok" and "Grok Build"
40
+ are the marks of their owner; they appear here and in the capture solely to identify the artifact
41
+ these values were derived from.
package/README.md ADDED
@@ -0,0 +1,51 @@
1
+ # `@yanlinglabs/winter-provider-conformance`
2
+
3
+ The provider-layer conformance harness: the shared corpus every OpenAI-, Anthropic-, Google- and
4
+ Bedrock-family adapter is asked the same questions through, the continuity and classifier-safety
5
+ corpora, and the loopback fakes (`./fakes`) those corpora drive.
6
+
7
+ **Bun only.** This package declares `engines.bun` and no `engines.node`: its fakes stand up real
8
+ loopback HTTP servers with `Bun.serve`, which has no Node equivalent it implements.
9
+
10
+ ## Install
11
+
12
+ Published to **both** registries. This is one of the org's own test harnesses rather than something a
13
+ consumer of the wrapper installs — and it is on public npm deliberately: the router package
14
+ `@yanlinglabs/winter-runtime-sdk` lives in its own repository and needs these loopback provider fakes as a dev dependency, and
15
+ reaching GitHub Packages from that repository's CI would mean a cross-repo `read:packages` token
16
+ whose only purpose is fetching test fixtures.
17
+
18
+ ### From public npm (anyone)
19
+
20
+ ```sh
21
+ npm install @yanlinglabs/winter-provider-conformance
22
+ ```
23
+
24
+ Nothing else is needed: the `@yanlinglabs` scope is public on npm.
25
+
26
+ ### From GitHub Packages (the `yanlingLabs` org)
27
+
28
+ GitHub Packages needs the scope pointed at it and an authenticated read, even for a public package.
29
+ In your project's `.npmrc`:
30
+
31
+ ```
32
+ @yanlinglabs:registry=https://npm.pkg.github.com
33
+ //npm.pkg.github.com/:_authToken=${GITHUB_TOKEN}
34
+ ```
35
+
36
+ …with `GITHUB_TOKEN` in the environment — a personal access token carrying `read:packages`, never a
37
+ literal in the file. Then `npm install @yanlinglabs/winter-provider-conformance` as usual.
38
+
39
+ **The published packages contain COMPILED OUTPUT ONLY.** Each tarball ships `dist/` — the bundled
40
+ JavaScript a consumer imports and the `.d.ts` declarations their type-checker reads — plus its data
41
+ files, `README.md` and `LICENSE`. It does **not** ship `src/`: the TypeScript sources live at
42
+ <https://github.com/yanlingLabs/winter-agent-sdk>, which is where to read them, file an issue, or send
43
+ a patch.
44
+
45
+ ## License
46
+
47
+ MIT — see [`LICENSE`](./LICENSE), which ships in the published tarball.
48
+
49
+ This package's xAI OAuth fake mirrors the flow Winter's provider derives from the Apache-2.0 licensed
50
+ `xai-org/grok-build`; that attribution is in [`NOTICE`](./NOTICE), which ships in the tarball beside
51
+ this file.
@@ -0,0 +1,7 @@
1
+ import type { CorpusHarness } from "./harness.js";
2
+ export declare const AZURE_CLASSIC_API_VERSION = "2026-05-01";
3
+ export declare const AZURE_DEPLOYMENT = "corpus-deployment";
4
+ /** The CLASSIC surface: the deployment path, chat completions, `api-key` auth. */
5
+ export declare function azureClassicHarness(): CorpusHarness;
6
+ /** The PREVIEW surface: `/openai/v1/responses`, the full Responses wire, encrypted continuation and all. */
7
+ export declare function azurePreviewHarness(): CorpusHarness;
@@ -0,0 +1,146 @@
1
+ /** The subset of the runtime's normalized action envelope these scenarios need. Assignable to `ActionEnvelope` — asserted runtime-side. */
2
+ export interface SafetyEnvelope {
3
+ toolName: string;
4
+ canonicalToolName: string;
5
+ input: Record<string, unknown>;
6
+ cwd: string;
7
+ roots: string[];
8
+ resolvedPaths: Array<{
9
+ path: string;
10
+ resolvedTarget: string;
11
+ isSymlink: boolean;
12
+ }>;
13
+ boundaries: {
14
+ protectedWrite: boolean;
15
+ criticalRemoval: boolean;
16
+ criticalReason?: string;
17
+ };
18
+ sessionCreatedResources: string[];
19
+ classifierContext: Array<{
20
+ hookId: string;
21
+ hookName?: string;
22
+ context: unknown;
23
+ }>;
24
+ shellSubcommands?: string[];
25
+ shellRedirectTargets?: string[];
26
+ networkDestinations?: string[];
27
+ repository?: {
28
+ remotes: string[];
29
+ };
30
+ }
31
+ /**
32
+ * Winter's own category vocabulary for these scenarios.
33
+ *
34
+ * Advisory, and deliberately so: WS-07 §13 open question 3 proposes "verdict-level agreement with
35
+ * recorded FP/FN rates... category agreement advisory", and this corpus implements exactly that —
36
+ * a run is scored on the VERDICT, and the category is recorded beside it for a reader.
37
+ */
38
+ export declare const CLASSIFIER_SAFETY_CATEGORIES: readonly ["ordinary-local-work", "declared-dependency-install", "read-only-http", "session-created-cleanup", "destructive-history", "download-piped-to-execution", "infrastructure-destruction", "production-deploy", "credential-disclosure", "security-control-removal", "transcript-tampering", "self-modification", "injection-driven-action"];
39
+ export type ClassifierSafetyCategory = (typeof CLASSIFIER_SAFETY_CATEGORIES)[number];
40
+ export interface ClassifierSafetyCase {
41
+ id: string;
42
+ /** What the case proves, in one sentence — reproduced in the report so a reader never opens this file to know what a pass means. */
43
+ question: string;
44
+ category: ClassifierSafetyCategory;
45
+ /** The verdict a correct reviewer returns. `no_verdict` is never an expected answer here: every scenario is decidable from its envelope alone. */
46
+ expected: "allow" | "deny";
47
+ envelope: SafetyEnvelope;
48
+ }
49
+ /**
50
+ * The scenarios.
51
+ *
52
+ * Each is the SMALLEST envelope that makes its question decidable — a reviewer that needs more than
53
+ * this to answer is a reviewer whose "no_verdict" would be honest, and the corpus would then be
54
+ * measuring the envelope rather than the model.
55
+ *
56
+ * The five allow rows exist because a corpus of denials measures nothing: a model that answers
57
+ * "deny" to everything scores perfectly on a deny-only corpus and is useless as a reviewer (WS-07
58
+ * §10.3 records false-POSITIVE rates for exactly this reason).
59
+ */
60
+ export declare const CLASSIFIER_SAFETY_CASES: readonly ClassifierSafetyCase[];
61
+ /** What a classifier answered, as this corpus needs to see it. Structurally the runtime's `ClassifierRawResult`. */
62
+ export interface ClassifierSafetyAnswer {
63
+ verdict: "allow" | "deny" | "no_verdict";
64
+ category?: string;
65
+ severity?: string;
66
+ reasonCode?: string;
67
+ auditReason?: string;
68
+ }
69
+ export type ClassifierSafetyClassify = (envelope: SafetyEnvelope, testCase: ClassifierSafetyCase) => Promise<ClassifierSafetyAnswer>;
70
+ export interface ClassifierSafetyOutcome {
71
+ id: string;
72
+ category: ClassifierSafetyCategory;
73
+ expected: "allow" | "deny";
74
+ actual: "allow" | "deny" | "no_verdict" | "threw";
75
+ /** The verdict-level agreement §13's open question 3 proposes as the measure. */
76
+ agreed: boolean;
77
+ /** Present when the classifier abstained or failed: its own reason code, or the thrown error's message. */
78
+ detail?: string;
79
+ }
80
+ export interface ClassifierSafetyReport {
81
+ label: string;
82
+ outcomes: ClassifierSafetyOutcome[];
83
+ agreed: number;
84
+ total: number;
85
+ /**
86
+ * A DENIAL the corpus expected but did not get, counted separately.
87
+ *
88
+ * The two error directions are not equal: a missed denial is a security failure, and a missed
89
+ * allow is an annoyance. A single agreement percentage hides which one a model is making, which is
90
+ * precisely the number §10.3 asks to be recorded per policy version.
91
+ */
92
+ missedDenials: number;
93
+ ok: boolean;
94
+ }
95
+ /**
96
+ * Runs the corpus.
97
+ *
98
+ * EVERY case runs, even after one disagrees — the point is the whole picture, and stopping early
99
+ * turns a corpus into a bisect (the adapter runner's own reasoning, deliberately mirrored).
100
+ *
101
+ * A THROW is an OUTCOME, not a crash: a classifier that fails to answer at all is a datum about that
102
+ * classifier, and the live run must survive it to report the rest.
103
+ */
104
+ export declare function runClassifierSafetyCorpus(classify: ClassifierSafetyClassify, opts?: {
105
+ label?: string;
106
+ cases?: readonly ClassifierSafetyCase[];
107
+ }): Promise<ClassifierSafetyReport>;
108
+ /**
109
+ * A thrown value, rendered as identity rather than as prose.
110
+ *
111
+ * Shared with the live runner's own rendering rule (`live/index.ts`), and for the same reason: a
112
+ * provider error's MESSAGE embeds a snippet of the provider's response body, and both of these
113
+ * strings are printed to an operator's terminal. Name, normalized code, HTTP status and the
114
+ * provider's own structured code are all identifiers; the message is content.
115
+ */
116
+ export declare function describeThrown(err: unknown): string;
117
+ /**
118
+ * A verdict's `reasonCode`, rendered as identity rather than as content.
119
+ *
120
+ * Winter's own codes (`timeout`, `schema_invalid`, …) are a CLOSED vocabulary and pass through
121
+ * verbatim — they are the whole diagnosis. A `model:`-prefixed code is up to 64 characters the model
122
+ * wrote, and this string is printed to a terminal, so what survives is the fact that the model
123
+ * authored it and how much it said.
124
+ *
125
+ * The namespace is MIRRORED, not imported — this package must not import the runtime (see the
126
+ * structural-envelope note at the top of this file, and the dependency direction it states) — so it
127
+ * follows that same rule's second half: the mirror is not trusted. `runner.test.ts` asserts this
128
+ * constant equals `MODEL_REASON_CODE_PREFIX` in `classifier/verdict-schema.ts`, because the whole
129
+ * reason the namespace exists is that a model answering `reasonCode: "timeout"` must not be able to
130
+ * look like a genuine transport timeout — and a drifted copy here would silently un-redact it.
131
+ */
132
+ export declare const MODEL_REASON_CODE_PREFIX = "model:";
133
+ export declare function describeReasonCode(reasonCode: string): string;
134
+ /**
135
+ * A case failure, rendered for a report.
136
+ *
137
+ * The split is between WINTER-AUTHORED text and PROVIDER-AUTHORED text, not between error classes.
138
+ * A fixture's own `throw new Error("expected the Vertex location path …")` is Winter's sentence and
139
+ * is the entire value of a failing corpus line. A normalized provider error's message embeds a
140
+ * truncated snippet of the provider's RESPONSE BODY (`errors.ts`), and these reports are printed to
141
+ * an operator's terminal and pasted into review packages — so anything carrying the normalized
142
+ * provider shape (`code`, `status`, `providerCode`) is rendered as identity instead.
143
+ */
144
+ export declare function describeCaseFailure(err: unknown): string;
145
+ /** One line per case, so a failing run says which questions were answered wrongly without anyone opening this file. */
146
+ export declare function formatClassifierSafetyReport(report: ClassifierSafetyReport): string;
@@ -0,0 +1,52 @@
1
+ import { type ContinuityEndpoint, type HistoryTarget, type ProviderMessageLike, type ProviderRegistry } from "@yanlinglabs/winter-provider-runtime";
2
+ /** The case ids. Stable strings: a report names them and a reader can trace each to its clause of the report. */
3
+ export type ContinuityCaseId = "claude-to-openai-warns" | "openai-to-claude-warns" | "gemini-to-openai-warns" | "xai-to-openai-warns" | "deepseek-to-openai-full-no-warning" | "deepseek-to-openai-truncated-warns" | "same-provider-model-profile-no-warning" | "same-provider-unverified-model-warns" | "switch-during-tool-loop-waits" | "source-receives-every-native-tool-result" | "target-never-receives-source-opaque-state" | "target-receives-available-portable-state" | "immediate-switch-cancels-rather-than-splices" | "suppression-requires-affirmative-evidence" | "opaque-state-never-in-a-warning-or-handoff" | "handoff-is-data-not-authority" | "exposed-reasoning-forwarded-only-when-policy-permits" | "memories-and-instruction-files-stay-out" | "telemetry-and-reports-carry-identifiers-only" | "summaries-requested-where-the-provider-documents-how";
4
+ export interface ContinuityCaseSpec {
5
+ id: ContinuityCaseId;
6
+ /** What the case proves, in one sentence, reproduced in the report so a passing case is legible without opening this file. */
7
+ question: string;
8
+ /** The report clause it answers. */
9
+ clause: string;
10
+ }
11
+ export declare const CONTINUITY_CASES: readonly ContinuityCaseSpec[];
12
+ export interface ContinuityWorld {
13
+ registry: ProviderRegistry;
14
+ endpoints: Record<"claudeA" | "claudeB" | "openai" | "openaiMini" | "gemini" | "xai" | "deepseek", ContinuityEndpoint>;
15
+ targets: Record<"claudeA" | "openai" | "deepseek", HistoryTarget>;
16
+ }
17
+ /**
18
+ * The catalog every case shares.
19
+ *
20
+ * The Anthropic rows declare NO continuation-domain evidence (each is its own single-member domain,
21
+ * which is what makes model-A-to-model-B a real boundary); OpenAI, Gemini, xAI and DeepSeek each
22
+ * declare their own. xAI and DeepSeek deliberately sit in the SAME wire family as OpenAI with the
23
+ * same endpoint shape -- the corpus is worthless if the fixtures make the distinctions easy.
24
+ */
25
+ export declare function createContinuityWorld(): ContinuityWorld;
26
+ /** The markers every "no opaque state escaped" assertion looks for. If one of these appears anywhere a model or a log can see, the case fails. */
27
+ export declare const OPAQUE_MARKERS: readonly ["CLAUDE-SIGNATURE-OPAQUE", "CLAUDE-REDACTED-OPAQUE", "OPENAI-ENCRYPTED-OPAQUE", "GEMINI-SIGNATURE-OPAQUE"];
28
+ /** A Claude turn carrying BOTH carriers of opaque state: signed in-dialect blocks in `content`, and native items beside them. */
29
+ export declare function claudeTurn(uuid: string, text: string): ProviderMessageLike;
30
+ export declare function openaiTurn(uuid: string, text: string): ProviderMessageLike;
31
+ export interface ContinuityCaseContext {
32
+ world: ContinuityWorld;
33
+ }
34
+ export type ContinuityCaseImpl = (ctx: ContinuityCaseContext) => void | Promise<void>;
35
+ export declare const CONTINUITY_CASE_IMPLS: Record<ContinuityCaseId, ContinuityCaseImpl>;
36
+ export interface ContinuityCaseOutcome {
37
+ id: ContinuityCaseId;
38
+ status: "passed" | "failed" | "missing";
39
+ detail?: string;
40
+ }
41
+ export interface ContinuityReport {
42
+ outcomes: ContinuityCaseOutcome[];
43
+ ok: boolean;
44
+ }
45
+ /**
46
+ * Runs every case, even after one fails -- a reader wants the whole picture from one run, and
47
+ * stopping at the first failure turns a corpus into a bisect. A case with no implementation is
48
+ * `missing`, never silently absent: "the corpus passed" has to mean "every question was asked".
49
+ */
50
+ export declare function runContinuityCorpus(cases?: Partial<Record<ContinuityCaseId, ContinuityCaseImpl>>): Promise<ContinuityReport>;
51
+ /** One line per case, so a failing run says which questions went unanswered without anyone opening this file. */
52
+ export declare function formatContinuityReport(report: ContinuityReport): string;
@@ -0,0 +1,38 @@
1
+ import type { RecordedRequest } from "../fakes/server.js";
2
+ /**
3
+ * The vendor product prefixes this rule polices, and the Winter provider ids each vendor owns.
4
+ *
5
+ * The prefix list is the ruling's (R-FW-1). The OWNER lists are read off the catalog's own rows: a
6
+ * prefix's owners are the ids whose vendor authored the header namespace, so `openai-` is honest on
7
+ * OpenAI's own row and on the codex backend (an OpenAI surface), and on nothing else.
8
+ */
9
+ export declare const CROSS_VENDOR_HEADER_OWNERS: Readonly<Record<string, readonly string[]>>;
10
+ /**
11
+ * Header names that are the DIALECT's protocol, exempt on every row served by that adapter.
12
+ *
13
+ * Keyed by `adapterId` rather than by provider id, because the exemption is a property of the wire
14
+ * protocol a row speaks and not of the vendor it belongs to — which is exactly why a third party's
15
+ * `<id>-anthropic` sibling needs it and a third party's own OpenAI-dialect row does not.
16
+ *
17
+ * Each entry is one of the names `endpoint-policy.ts` enumerates as PROTOCOL. Nothing account-scoped
18
+ * or organisation-scoped belongs here; those are the names the rule exists to catch.
19
+ */
20
+ export declare const DIALECT_PROTOCOL_HEADERS: Readonly<Record<string, readonly string[]>>;
21
+ export interface CrossVendorSweepTarget {
22
+ /** The Winter row the request was made for. */
23
+ providerId: string;
24
+ /** The adapter that served it — the key the dialect-protocol exemption is read by. */
25
+ adapterId: string;
26
+ }
27
+ /**
28
+ * Every cross-vendor violation in one header map, as readable sentences.
29
+ *
30
+ * Returns `[]` for a clean request. The strings name the header, the row and the vendor whose
31
+ * namespace it is, because a bare boolean on a header sweep is a failure nobody can act on.
32
+ *
33
+ * VALUES ARE NEVER INCLUDED. A violating header is by definition account- or organisation-scoped,
34
+ * and a failure message is one of the most reliably-pasted strings in any system.
35
+ */
36
+ export declare function crossVendorHeaderViolations(target: CrossVendorSweepTarget, headers: Record<string, string>): string[];
37
+ /** The same sweep over a fake's whole recorded request log. One call per driven row. */
38
+ export declare function crossVendorViolationsIn(target: CrossVendorSweepTarget, requests: readonly RecordedRequest[]): string[];
@@ -0,0 +1,57 @@
1
+ import type { DiscoveryCache, ModelCatalogResult, ProviderEvent, TurnRequest } from "@yanlinglabs/winter-provider-runtime";
2
+ import type { DescriptorOverrides } from "@yanlinglabs/winter-provider-runtime/testing";
3
+ import type { RecordedRequest } from "../fakes/server.js";
4
+ export interface HarnessOverrides {
5
+ /**
6
+ * Resolve every model to NO descriptor — the `allowUnlisted` gateway shape, where a model has no
7
+ * catalog evidence at all.
8
+ *
9
+ * Spelled as a positive statement rather than as an absent option (ruling on finding I3): the
10
+ * adapters now REQUIRE a lookup, so "this model has no evidence" is something a caller says out
11
+ * loud, and forgetting to say anything is a compile error instead of a silent loss of every
12
+ * §8.2 refusal.
13
+ */
14
+ unlisted?: boolean;
15
+ /** Vary the descriptor this turn resolves. */
16
+ descriptor?: DescriptorOverrides;
17
+ }
18
+ export interface HarnessCapabilities {
19
+ tools: boolean;
20
+ vision: boolean;
21
+ /** `opaque` = Responses' encrypted reasoning items; `exposed` = DeepSeek's replayable text; `none` = neither. */
22
+ continuation: "opaque" | "exposed" | "none";
23
+ effort: boolean;
24
+ }
25
+ export interface CorpusHarness {
26
+ name: string;
27
+ surface: "responses" | "chat";
28
+ capabilities: HarnessCapabilities;
29
+ /**
30
+ * `live` = the provider serves a catalog endpoint this corpus can page through; `static` = the
31
+ * adapter's catalog is compiled in (codex serves only its own slugs for a ChatGPT account), so the
32
+ * paging questions do not exist for it and the case asks the ones that do.
33
+ */
34
+ discovery: "live" | "static";
35
+ /**
36
+ * False for a declared-LOCAL endpoint, where having no credential is a valid configuration
37
+ * (`local-none` is a first-class auth kind, WS-13 §6) rather than a missing one.
38
+ */
39
+ requiresCredential: boolean;
40
+ /** Where this surface's model listing lives, when it is not at the root (Azure's `/openai/models`). */
41
+ discoveryRoutePrefix?: string;
42
+ /** Runs a turn against `endpoint`. `endpoint` is usually the runner's fake, but a case may point it at a closed server. */
43
+ stream(endpoint: {
44
+ url: string;
45
+ }, req: TurnRequest, overrides?: HarnessOverrides): AsyncIterable<ProviderEvent>;
46
+ /** Live discovery against `endpoint`. */
47
+ discover(endpoint: {
48
+ url: string;
49
+ }, opts?: {
50
+ maxItems?: number;
51
+ maxBytes?: number;
52
+ cache?: DiscoveryCache;
53
+ signal?: AbortSignal;
54
+ }): Promise<ModelCatalogResult>;
55
+ /** Target-specific assertions every recorded request must satisfy (Azure's `api-version`). */
56
+ assertRequest?(recorded: RecordedRequest): void;
57
+ }
@@ -0,0 +1,69 @@
1
+ import type { FakeServer } from "../fakes/server.js";
2
+ /** The WS-13 §13 case ids. Stable strings: a lane keys its implementations on them and a report names them. */
3
+ export type CorpusCaseId = "serialization-and-headers" | "streaming-order" | "tool-call-single" | "tool-call-multiple" | "tool-call-fragmented" | "tool-result-replay" | "cancel-pre-header" | "cancel-mid-stream" | "usage-accounting" | "error-auth" | "error-rate-limit" | "error-timeout" | "error-network" | "error-malformed" | "error-provider-codes" | "retry-after-no-replay" | "effort-mapping" | "opaque-continuation" | "limit-rejection" | "vision-where-advertised" | "discovery-edge-cases" | "identity-across-resume" | "no-silent-tool-dropping";
4
+ /** One case: what it asks, and whether an adapter may legitimately not answer it. */
5
+ export interface CorpusCaseSpec {
6
+ id: CorpusCaseId;
7
+ /** What the case proves, in one sentence. Reproduced in the report so a reader never has to open this file to know what a passing case means. */
8
+ question: string;
9
+ /**
10
+ * `required` cases must be implemented for a family to pass at all.
11
+ *
12
+ * `capability-gated` cases are skipped ONLY when the adapter's own descriptor says the capability
13
+ * is absent -- a skip is then a FACT about the model, recorded as such, and never a lane quietly
14
+ * declining a case it found hard. That distinction is the whole reason this field exists.
15
+ */
16
+ requirement: "required" | "capability-gated";
17
+ }
18
+ /**
19
+ * WS-13 §13's list, in the order the spec states it.
20
+ *
21
+ * DATA, not code, so a lane extends the corpus by supplying implementations rather than by editing
22
+ * this array -- and so a lane that MISSES a required case is caught by the runner rather than by a
23
+ * reviewer noticing an absent test.
24
+ */
25
+ export declare const CORPUS_CASES: readonly CorpusCaseSpec[];
26
+ /** The context a case implementation is handed. */
27
+ export interface CorpusCaseContext {
28
+ fake: FakeServer;
29
+ /** The model id the case should ask for. The runner passes the one it was configured with, so a family fake can key its scenario table on it. */
30
+ model: string;
31
+ }
32
+ /** What a case returns. A THROW is a failure; returning `{ skipped }` is the capability-gated "this model does not do that". */
33
+ export type CorpusCaseResult = void | {
34
+ skipped: string;
35
+ };
36
+ export type CorpusCaseImpl = (ctx: CorpusCaseContext) => Promise<CorpusCaseResult>;
37
+ export interface RunAdapterCorpusOptions {
38
+ /** Names the family in the report. */
39
+ adapter: string;
40
+ fake: FakeServer;
41
+ model: string;
42
+ /** The lane's implementations, keyed by case id. A missing REQUIRED case is a failure the runner reports -- never a silent absence. */
43
+ cases: Partial<Record<CorpusCaseId, CorpusCaseImpl>>;
44
+ }
45
+ export interface CorpusCaseOutcome {
46
+ id: CorpusCaseId;
47
+ status: "passed" | "failed" | "skipped" | "missing";
48
+ detail?: string;
49
+ }
50
+ export interface CorpusReport {
51
+ adapter: string;
52
+ model: string;
53
+ outcomes: CorpusCaseOutcome[];
54
+ /** True only when every REQUIRED case passed and no case failed. A capability-gated skip does not block a pass; a missing required case does. */
55
+ ok: boolean;
56
+ }
57
+ /**
58
+ * Runs the corpus.
59
+ *
60
+ * EVERY case runs, even after one fails: a family lane wants the whole picture from one run, and
61
+ * stopping at the first failure turns a corpus into a bisect.
62
+ *
63
+ * A MISSING required case is `"missing"` rather than silently absent -- that status is what makes
64
+ * "the corpus passed" mean "every question was asked", which is the only reading under which WS-13
65
+ * §13's promotion rule means anything.
66
+ */
67
+ export declare function runAdapterCorpus(opts: RunAdapterCorpusOptions): Promise<CorpusReport>;
68
+ /** A one-line-per-case rendering, so a failing lane run says which questions went unanswered without anyone opening this file. */
69
+ export declare function formatCorpusReport(report: CorpusReport): string;
@@ -0,0 +1,36 @@
1
+ import { type FakeServer, type RecordedRequest } from "./server.js";
2
+ export declare const FAKE_CONSOLE_ACCOUNT_ID = "acct-test-console-0001";
3
+ export declare const FAKE_CONSOLE_ACCESS_TOKEN = "test-token-anthropic-console-access";
4
+ export declare const FAKE_CONSOLE_REFRESHED_ACCESS_TOKEN = "test-token-anthropic-console-access-refreshed";
5
+ export declare const FAKE_CONSOLE_REFRESH_TOKEN = "test-token-anthropic-console-refresh";
6
+ export interface AnthropicConsoleOauthFakeOptions {
7
+ /** Answer every grant with this status instead of 200. */
8
+ failTokenWith?: number;
9
+ /** The profile answers 200 with no account — the shape that must REFUSE the login, not name a record `anthropic:undefined`. */
10
+ omitAccount?: boolean;
11
+ /** A refresh grant that does not rotate the refresh token, which is the common case. */
12
+ omitRefreshToken?: boolean;
13
+ accountId?: string;
14
+ expiresIn?: number;
15
+ }
16
+ export interface AnthropicConsoleOauthFake extends FakeServer {
17
+ /** Handed to the login as its authorize endpoint. Never fetched — the browser half is `completeAuthorization`. */
18
+ authorizeUrl: string;
19
+ tokenUrl: string;
20
+ profileUrl: string;
21
+ /** The grants the token endpoint actually received, in order. Headers are redacted by the base fake. */
22
+ tokenRequests: RecordedRequest[];
23
+ profileRequests: RecordedRequest[];
24
+ /**
25
+ * The browser half: reads the authorize URL the flow produced, records its PKCE challenge, and
26
+ * calls the flow's own loopback callback.
27
+ *
28
+ * `state` overrides the value the flow minted (a planted-callback fixture); `code: null` omits the
29
+ * authorization code entirely.
30
+ */
31
+ completeAuthorization(url: string, overrides?: {
32
+ state?: string;
33
+ code?: string | null;
34
+ }): Promise<void>;
35
+ }
36
+ export declare function startAnthropicConsoleOauthFake(opts?: AnthropicConsoleOauthFakeOptions): Promise<AnthropicConsoleOauthFake>;
@@ -0,0 +1,115 @@
1
+ import { type FakeRoute, type RecordedRequest, type SseFrame, type SseResponseOptions } from "./server.js";
2
+ /** One scripted content block. `chunks` are the deltas the wire emits, so a fixture controls fragmentation directly. */
3
+ export type AnthropicScriptBlock = {
4
+ type: "text";
5
+ chunks: string[];
6
+ }
7
+ /** `signature` absent scripts the R6-8 discriminator: capture (F) run (ii), a thinking block with NO signature at all. */
8
+ | {
9
+ type: "thinking";
10
+ chunks: string[];
11
+ signature?: string;
12
+ } | {
13
+ type: "redacted_thinking";
14
+ data: string;
15
+ } | {
16
+ type: "tool_use";
17
+ id: string;
18
+ name: string;
19
+ jsonChunks: string[];
20
+ };
21
+ export interface AnthropicTurnScript {
22
+ id?: string;
23
+ model?: string;
24
+ blocks: AnthropicScriptBlock[];
25
+ stopReason?: "end_turn" | "tool_use" | "max_tokens" | "stop_sequence" | "refusal";
26
+ usage?: {
27
+ input_tokens?: number;
28
+ output_tokens?: number;
29
+ cache_creation_input_tokens?: number;
30
+ cache_read_input_tokens?: number;
31
+ };
32
+ /** Injects `event: ping` after the FIRST `content_block_start`, exactly where capture (F) put it. */
33
+ ping?: boolean;
34
+ /** Emits an `event: error` frame instead of `message_delta`/`message_stop` -- the mid-stream provider failure. */
35
+ errorAfterBlocks?: {
36
+ type: string;
37
+ message: string;
38
+ };
39
+ }
40
+ /**
41
+ * Renders a scripted turn into the family's real frame sequence.
42
+ *
43
+ * The ORDER is the point and is taken from capture (F): `message_start`, then per block a
44
+ * `content_block_start` -> its deltas -> `content_block_stop`, then `message_delta` carrying the
45
+ * stop reason and the output-token count, then `message_stop`. A `thinking` block's
46
+ * `signature_delta` arrives LAST inside its own block, which is what makes "capture the block only
47
+ * at `content_block_stop`" a testable rule rather than a stylistic one.
48
+ */
49
+ export declare function anthropicSseFrames(script: AnthropicTurnScript): SseFrame[];
50
+ /** The scripted turn as an SSE response. `opts.dropAfter` is the base's mid-stream-drop primitive, counted in FRAMES. */
51
+ export declare function anthropicTurnResponse(script: AnthropicTurnScript, opts?: SseResponseOptions): Response;
52
+ /** Reads the model id out of an Anthropic request body -- the family's `modelOf` for `scenarioTable`. */
53
+ export declare function anthropicModelOf(recorded: RecordedRequest): string | undefined;
54
+ /** The parsed body of a recorded Anthropic request, for an assertion that wants to read one field. */
55
+ export declare function anthropicBody(recorded: RecordedRequest): Record<string, unknown>;
56
+ export interface AnthropicRequestExpectation {
57
+ model?: string;
58
+ apiVersion?: string;
59
+ beta?: string;
60
+ stream?: boolean;
61
+ /** The `type` of every content block of every message, flattened in wire order -- the block-ordering assertion R6-8 needs. */
62
+ blockTypes?: string[];
63
+ /** The wire roles, in order. `tool` is NOT a wire role: a tool result rides a `user` message. */
64
+ roles?: string[];
65
+ thinking?: unknown;
66
+ toolChoice?: unknown;
67
+ toolNames?: string[];
68
+ maxTokens?: number;
69
+ system?: string;
70
+ }
71
+ /**
72
+ * Asserts the EXACT request shape on the live request the fake received.
73
+ *
74
+ * `x-api-key` is checked in its REDACTED form (`***`): the base replaces a credential header's
75
+ * material as it records, so asserting on `***` proves both that the adapter authenticated and that
76
+ * the fake's own redaction ran.
77
+ */
78
+ export declare function assertAnthropicRequest(recorded: RecordedRequest, expected?: AnthropicRequestExpectation): void;
79
+ /** Every message's content-block `type`, flattened in wire order. A string content counts as one `text`. */
80
+ export declare function flattenBlockTypes(messages: Array<{
81
+ content?: unknown;
82
+ }>): string[];
83
+ /** The blocks of one message of a recorded request, for an assertion about verbatim replay. */
84
+ export declare function messageBlocks(recorded: RecordedRequest, index: number): Array<Record<string, unknown>>;
85
+ /** A standard Anthropic error envelope: `error.type` is the family's machine-readable code (there is no `error.code`). */
86
+ export declare function anthropicError(status: number, type: string, message?: string, headers?: Record<string, string>): Response;
87
+ /**
88
+ * The ordering constraint this endpoint actually enforces, enforced HERE too.
89
+ *
90
+ * A fake that accepts every ordering cannot fail a pin about ordering -- and this lane shipped a
91
+ * decoration ahead of a turn's `tool_result` blocks for two rounds with a green corpus, because
92
+ * nothing on the receiving end cared. The message is Winter's rendering of the documented constraint,
93
+ * not a captured verbatim vendor string.
94
+ *
95
+ * Returns the offending message index, or `undefined` when every turn is well-formed.
96
+ */
97
+ export declare function findToolResultOrderingViolation(recorded: RecordedRequest): number | undefined;
98
+ export interface AnthropicFakeOptions {
99
+ /** modelId -> the scripted answer for `POST /v1/messages`. A `Response[]` is indexed by attempt, its last element repeating (the base's own retry shape). */
100
+ messages: ScenarioMap;
101
+ /** The `POST /v1/messages/count_tokens` answer. Absent -> a fixed count, so a fixture that does not care need not script one. */
102
+ countTokens?: (recorded: RecordedRequest) => Response;
103
+ /** The `GET /v1/models` answer -- discovery AND `validateCredential` both land here. */
104
+ models?: (recorded: RecordedRequest) => Response | Promise<Response>;
105
+ }
106
+ type ScenarioMap = Record<string, ((recorded: RecordedRequest, attempt: number) => Response | Promise<Response>) | Response[]>;
107
+ /**
108
+ * The three routes an Anthropic adapter can reach, wired to a model-keyed scenario table.
109
+ *
110
+ * `count_tokens` is registered BEFORE `/v1/messages` deliberately: the base matches routes in order
111
+ * and an exact-path match for `/v1/messages` would otherwise be fine, but keeping the more specific
112
+ * path first makes the ordering independent of that detail.
113
+ */
114
+ export declare function anthropicFakeRoutes(opts: AnthropicFakeOptions): FakeRoute[];
115
+ export {};