@workglow/test-contract 0.6.1

package/README.md

@@ -0,0 +1,232 @@
+# `@workglow/test-contract`
+
+Parameterized test suites that exercise an interface contract against
+every adapter that implements it. Each suite exports one function:
+
+    export function runXxxConformance(opts: { factory, capabilities, ... }): void;
+
+An adapter writes a thin caller that supplies a factory and capability flags;
+all behavioral assertions are inherited.
+
+    import { runTabularStorageContract } from "@workglow/test-contract/tabular-storage";
+
+    runTabularStorageContract({
+      name: "MyTabularStorage",
+      createStorage: async () => new MyTabularStorage(CompoundSchema, CompoundPrimaryKeyNames),
+      capabilities: { supportsSubscriptions: false, supportsVectorColumns: false,
+                      supportsTransactions: true, supportsQuery: true },
+    });
+
+## Why a separate package
+
+An adapter that implements one of these interfaces is usually not in this
+repository. `@workglow/test` — the 441 concrete tests — is `private: true` and
+404s on npm, so while the suites lived inside it, every implementation
+downstream inherited exactly zero assertions and re-derived the contract by
+hand. This package is the suites and nothing else, published so they can be
+taken.
+
+Its dependency surface is deliberately small, and split one subpath per
+contract: a tabular-storage adapter installs `@workglow/storage` and `vitest`,
+and never loads `@workglow/ai` or `@workglow/browser-control` — those are
+optional peers that only the suites needing them pull in.
+
+Concrete test files (`*.test.ts`) stay in `packages/test/src/test/`. The
+boundary is what makes the pattern obvious: everything here is reusable and
+nothing here runs on its own.
+
+Two pre-existing parameterized suites live with their concrete callers and
+stay in place — moving them is unnecessary churn:
+
+- `packages/test/src/test/job-queue/genericJobQueueTests.ts`
+- `packages/test/src/test/storage-tabular/genericTabularStorageTests.ts`
+
+Treat both as additional examples of the pattern.
+
+## Conventions
+
+1.  **Entrypoint shape.**
+
+        export function runXxxConformance(opts: {
+          readonly name: string;
+          readonly skip?: boolean;
+          readonly timeout: number;
+          readonly factory: () => Promise<{ register, dispose, inspect }>;
+          readonly capabilities: Record<string, boolean>;
+          // ...contract-specific fields
+        }): void;
+
+    Defines a single top-level `describe.skipIf(opts.skip)`.
+
+2.  **Factory shape.** `factory()` returns a fresh handle per top-level
+    `beforeAll`. The handle exposes:
+    - `register()` — install the provider/storage/queue and any model records.
+    - `dispose()` — release resources; called in `afterAll`.
+    - `inspect()` — optional whitebox handle for assertions that need to
+      observe internal state (session maps, disposable refs). Adapters that
+      don't expose internals return `{}`; assertions skip with a logged
+      warning instead of passing silently.
+
+3.  **Capability flags drive `describe.skipIf(!cap)` blocks.** Never silently
+    skip on missing capability without a flag — the absence of a flag
+    indicates a contract gap, not a permitted variation.
+
+4.  **Live-API tests honor existing preload + retry/timeout settings.** Do
+    not introduce new env vars from a contract suite.
+
+5.  **Adapter shims are short.** A new adapter joining a contract suite
+    should be ~30 lines: imports, factory, capability flags, model IDs.
+
+6.  **`dispose()` must be idempotent.** Conformance suites may call dispose
+    multiple times (once for the dispose assertion, once in `afterAll`).
+    Adapters whose underlying resource doesn't natively support repeated
+    dispose should guard with a flag.
+
+### Factory shape variants
+
+The `register/dispose/inspect` factory documented above is one of two
+legitimate shapes — used when an adapter is a long-lived global registration
+(e.g. an AI provider). For contracts whose subject is heavyweight but
+per-test state (e.g. browser contexts), prefer a `create/dispose` factory
+where each top-level block instantiates its own subject:
+
+    factory: () => Promise<{
+      create: () => Promise<TSubject>;
+      dispose: (subject: TSubject) => Promise<void>;
+    }>
+
+The principle is the same: a fresh handle per block, with no shared state
+that block N can leak into block N+1. The methods on the handle are
+contract-specific.
+
+## Worker-proxy pattern
+
+Workers cross a `postMessage` boundary; the inline AiProvider conformance
+assertions must also hold when a provider is registered via
+`register({ worker })`. Each worker-capable adapter's shim invokes the suite
+**three times**:
+
+1. `runAiProviderConformance({ name: "<Adapter> (inline)", factory: inlineFactory, ... })`
+2. `runAiProviderConformance({ name: "<Adapter> (worker)", factory: workerFactory, ... })`
+3. `runWorkerProxyBoundary({ name: "<Adapter>", factory: workerFactory, ... })`
+
+The worker factory's `inspect()` returns `{}` — workers are opaque by
+design. The inherited session-reuse and dispose blocks skip with their
+existing logged-warning behavior; the boundary block adds three
+worker-only assertions (dispose terminates worker, worker-side throw
+surfaces with stack, postMessage handles concurrent streams independently).
+
+Capability flags:
+
+- `browserOnly: true` — entire boundary block emits a single skipped test.
+  Used for TF-MediaPipe until browser test infra arrives.
+- `errorPropagation: false` — relaxes the throw-surfaces assertion to skip
+  the stack-frame check, asserting only a non-empty message.
+
+## Available suites
+
+| Contract                                | Suite                                             | Adapters                                                                                              |
+| --------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
+| `AiProvider`                            | `@workglow/test-contract/ai-provider`             | Anthropic, OpenAI, Gemini, Ollama, HF Inference, HF Transformers, LlamaCpp                            |
+| `IMigrationRunner`                      | `@workglow/test-contract/storage-migrations`      | Postgres, SQLite, IndexedDB                                                                           |
+| Tabular schema migrations               | `@workglow/test-contract/tabular-migrations`      | InMemory, IndexedDB, Postgres, SQLite, FsFolder                                                       |
+| `ITabularStorage`                       | `@workglow/test-contract/tabular-storage`         | InMemory, SharedInMemory, IndexedDB, Postgres, SQLite, DuckDB, Supabase, Cached, Telemetry, FsFolder  |
+| `ITabularStorage.join`                  | `@workglow/test-contract/tabular-storage`         | InMemory, SharedInMemory, IndexedDB, Postgres, SQLite, DuckDB, Supabase, Cached, Telemetry, HttpProxy |
+| `ITabularStorage` (rest of the surface) | `test/storage-tabular/genericTabularStorageTests` | InMemory, IndexedDB, Postgres, SQLite, Supabase, FsFolder, HuggingFace                                |
+| `IQueueStorage` + `IRateLimiterStorage` | `test/job-queue/genericJobQueueTests`             | InMemory, IndexedDB, Postgres, SQLite, Supabase                                                       |
+| `IVectorStorage`                        | `@workglow/test-contract/vector-storage`          | InMemory, SQLite, Postgres, IndexedDB, Scoped, Telemetry                                              |
+| `IEntitlementProfile`                   | `@workglow/test-contract/entitlement-profile`     | Browser, Desktop, Server, Custom                                                                      |
+| `IBrowserContext`                       | `@workglow/test-contract/browser-context`         | Mock, Playwright, BunWebView, Electron                                                                |
+| `IHumanConnector`                       | `@workglow/test-contract/human-connector`         | Mock, Mock (no followUp), McpElicitation, Prompt, Ink, RunEvent                                       |
+| `IWebSearchProvider`                    | `@workglow/test-contract/web-search`              | Brave, Tavily, SearXNG, Anthropic, OpenAI, OpenRouter, Gemini                                         |
+| Worker-proxy parity                     | `@workglow/test-contract/worker-proxy`            | _harness only — no adapters wired yet_                                                                |
+
+## Billing failures: skipped on CI, failed locally
+
+Live provider suites run against real accounts, so "we ran out of money" is a
+condition every one of them can hit. `@workglow/test-contract/credit-exhausted-skip` detects
+it — 402s, `insufficient_quota`, `insufficient_credits`, DeepSeek's
+`Insufficient Balance`, Anthropic's credit-balance error — and the `it` exported
+from that module (which every conformance assertion imports) decides what to do
+with it:
+
+| where         | behavior | why                                                                                                                                            |
+| ------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
+| CI            | skip     | nobody watching a build can top an account up, and one exhausted key would turn every provider suite red for a change that touched no provider |
+| anywhere else | fail     | the developer running the suite IS the person who can act on it                                                                                |
+
+`CI` / `GITHUB_ACTIONS` select the branch; `WORKGLOW_CREDIT_EXHAUSTED_SKIP=1`
+forces the skip locally and `=0` forces the failure on CI.
+
+The detector reads the **message** as well as the numeric status, and that is
+load-bearing rather than belt-and-braces: `classifyProviderError` rebuilds a
+provider error as a `PermanentJobError` carrying the message and neither
+`status` nor `cause`, so by the time a test body catches a DeepSeek 402 the
+number survives only as the `402 Insufficient Balance` text the OpenAI SDK put
+in the message. A status-only detector reads that as an ordinary permanent
+failure — which is exactly how DeepSeek kept failing suites while every other
+provider skipped. Message matching for a bare `402` is anchored to an
+HTTP-shaped position (line start, or after a summary line's colon) so prose
+that merely contains the number is not scavenged as a status.
+
+Rate limits (429 + `rate_limit_exceeded`) are deliberately NOT this: they are
+transient and the retry policy handles them.
+
+## How to add a new contract suite
+
+1. Pick a contract surface (an interface or abstract base class).
+2. Enumerate the behavioral invariants the contract implies but that aren't
+   currently asserted in any concrete test.
+3. Decide the capability matrix — which assertions are universal, which
+   are opt-in.
+4. Create `packages/test-contract/src/<contract-name>/` with `types.ts`,
+   `fixtures.ts`, `run<Contract>Conformance.ts`, and per-assertion files
+   under `assertions/`.
+5. Add `src/<contract-name>.ts` re-exporting all of it, and an
+   `exports["./<contract-name>"]` entry plus a `build-js` entrypoint in
+   `package.json` — a suite nobody can import is the problem this package
+   exists to fix.
+6. Write one shim caller per adapter under
+   `packages/test/src/test/<contract-name>/<Adapter>_Generic.integration.test.ts`.
+7. Add a row to the table above.
+
+## Two strategies, one contract
+
+`runTabularJoinContract` is the one suite here whose subject has two
+implementations of a single semantics — a pushed-down `JOIN` statement when
+both tables share a connection, an application-side hash join otherwise — with
+the planner, not the caller, choosing between them. That makes a disagreement
+between them a wrong answer nothing reports, which changes with where the
+right-hand storage happens to live.
+
+So it runs three blocks rather than one. The behavioural cases run on whichever
+path the pair takes. `join bounded left read` runs only on the hash path and
+asserts the half of the docstring the pushdown gets from the database for free:
+that a bounded join stops reading the left side early, and only when the joined
+rows are already in their final order. `join strategy parity` runs only where
+both paths are reachable, runs the same spec through each, and asserts they
+agree — over specs derived from the fields of `JoinSpec`, so an option added
+there fails to compile until it declares what the two strategies owe each
+other.
+
+What the two are held to depends on what the spec asked for, because `join`
+promises `orderBy`, `limit` and `offset` apply to the joined rows and never what
+order an unordered join arrives in. An unordered join is compared as a set; an
+unordered join that is also windowed is compared by row count and membership.
+Holding either to a shared order would assert the storage engine's row layout,
+which SQLite and Postgres happen to share and DuckDB, being columnar, does not.
+
+## Roadmap
+
+Future contract suites in priority order:
+
+1. Storage extensions (subscribeToChanges ordering, vector-dimension format,
+   putBulk round-trip count, deleteSearch streaming) — additions to the
+   existing `genericTabularStorageTests.ts`.
+2. Worker-proxy contract — harness shipped; per-adapter wiring deferred to a
+   follow-up PR (vitest-Node `Worker` polyfill + per-adapter
+   `WorkerManager` unregister-on-dispose required before HFT/LlamaCpp can
+   register inline + worker in the same test file).
+3. `IBrowserContext` — Playwright / Electron / BunWebView / CDP backends.
+4. `EntitlementProfile` — desktop / web / server profiles.
+5. `IHumanConnector` — IN PROGRESS — `MockHumanConnector` + `McpElicitationConnector`. App / Electron adapters add their own shim when introduced.

package/dist/ai-provider/assertions/capabilityHonesty.d.ts

@@ -0,0 +1,7 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts } from "../types";
+export declare function capabilityHonestyBlock(opts: AiProviderConformanceOpts): void;

package/dist/ai-provider/assertions/dispose.d.ts

@@ -0,0 +1,7 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts, ConformanceHandle } from "../types";
+export declare function disposeBlock(opts: AiProviderConformanceOpts, getHandle: () => ConformanceHandle): void;

package/dist/ai-provider/assertions/inferAdvertisesRegistered.d.ts

@@ -0,0 +1,15 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+/**
+ * A registered run-fn capability that `inferCapabilities` never returns is
+ * unreachable for any model whose record was populated by inference — the
+ * path the catalog exists to serve.
+ *
+ * The inferred sets are unioned across the provider's fixtures, so this
+ * answers "is this capability reachable at all" and says nothing about any one
+ * model; {@link assertInferServesInferred} is the per-model direction.
+ */
+export declare function assertInferAdvertisesRegistered(name: string, registered: readonly (readonly string[])[], inferred: readonly (readonly string[])[]): void;

package/dist/ai-provider/assertions/inferServesInferred.d.ts

@@ -0,0 +1,20 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+/** One fixture model id paired with the capabilities `inferCapabilities` returns for it. */
+export interface InferredForModel {
+    readonly id: string;
+    readonly capabilities: readonly string[];
+}
+/**
+ * The per-model direction of {@link assertInferAdvertisesRegistered}: that one
+ * unions the inferred sets across a provider's fixtures, which answers "is this
+ * capability reachable at all" and passes a branch that advertises a capability
+ * no run-fn can serve *for that model*. Such a model clears
+ * `modelMeetsRequires` during selection and fails inside the provider instead,
+ * with the capability message it would have gotten replaced by whatever the
+ * provider throws.
+ */
+export declare function assertInferServesInferred(name: string, registered: readonly (readonly string[])[], inferred: readonly InferredForModel[]): void;

package/dist/ai-provider/assertions/pricingMatchesModality.d.ts

@@ -0,0 +1,36 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { ModelPricing } from "@workglow/ai";
+/** One fixture model, its inferred capabilities, and the card its provider returns. */
+export interface PricedModel {
+    readonly id: string;
+    readonly capabilities: readonly string[];
+    readonly pricing: ModelPricing | undefined;
+}
+/**
+ * A provider must not quote a per-token rate for a model its own inference says
+ * is not billed per token.
+ *
+ * This is the direction that catches a real fabricated number, and it is a hard
+ * assertion because there is no legitimate case for it: `grok-2-image-1212`
+ * resolved `grok-2`'s per-1M-token card through the pricing table's substring
+ * walk, and `gemini-2.5-flash-image-preview` took `gemini-2.5-flash`'s the same
+ * way. Both are image models by the provider's own catalogue.
+ *
+ * A free card is exempt: `input: 0` on a local model says "this costs nothing",
+ * which is true whatever the billing unit, and is how every local provider
+ * answers.
+ */
+export declare function assertPricingMatchesModality(name: string, models: readonly PricedModel[], namedByTable?: readonly string[]): void;
+/**
+ * The weaker direction: a token-billed model SHOULD carry a card.
+ *
+ * A ratchet rather than an assertion, deliberately. Real gaps exist today —
+ * `gemini-embedding-001` among them — and a hard assert would redden the build
+ * on a known one and get disabled, which costs more than the gap. Recording the
+ * set means it cannot grow while it shrinks deliberately.
+ */
+export declare function assertPricedGapDoesNotGrow(name: string, models: readonly PricedModel[], knownUnpriced: readonly string[]): void;

package/dist/ai-provider/assertions/registryCoverage.d.ts

@@ -0,0 +1,13 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts } from "../types";
+/**
+ * Capability-set conformance: every model in `opts.models` must be backed by
+ * a registry entry whose `serves` superset covers the model's declared
+ * `capabilities`. Each capability is also probed individually to confirm the
+ * provider can dispatch the bare capability via `getRunFnFor`.
+ */
+export declare function registryCoverageBlock(opts: AiProviderConformanceOpts): void;

package/dist/ai-provider/assertions/sessionReuse.d.ts

@@ -0,0 +1,7 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts, ConformanceFixture, ConformanceHandle } from "../types";
+export declare function sessionReuseBlock(opts: AiProviderConformanceOpts, fixture: ConformanceFixture, getHandle: () => ConformanceHandle): void;

package/dist/ai-provider/assertions/signalHonoring.d.ts

@@ -0,0 +1,7 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts, ConformanceFixture } from "../types";
+export declare function signalHonoringBlock(opts: AiProviderConformanceOpts, fixture: ConformanceFixture): void;

package/dist/ai-provider/assertions/structuredGeneration.d.ts

@@ -0,0 +1,7 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts, ConformanceFixture } from "../types";
+export declare function structuredGenerationBlock(opts: AiProviderConformanceOpts, fixture: ConformanceFixture): void;

package/dist/ai-provider/assertions/textGenerationSmoke.d.ts

@@ -0,0 +1,7 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts, ConformanceFixture } from "../types";
+export declare function textGenerationSmokeBlock(opts: AiProviderConformanceOpts, fixture: ConformanceFixture): void;

package/dist/ai-provider/assertions/toolCallAccumulator.d.ts

@@ -0,0 +1,7 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts, ConformanceFixture } from "../types";
+export declare function toolCallAccumulatorBlock(opts: AiProviderConformanceOpts, fixture: ConformanceFixture): void;

package/dist/ai-provider/assertions/toolCallMultiTurn.d.ts

@@ -0,0 +1,7 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts, ConformanceFixture } from "../types";
+export declare function toolCallMultiTurnBlock(opts: AiProviderConformanceOpts, fixture: ConformanceFixture): void;

package/dist/ai-provider/assertions/usageNormalization.d.ts

@@ -0,0 +1,61 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { Usage } from "@workglow/ai";
+/**
+ * Shared contract for every provider's usage normalization.
+ *
+ * Each provider reads token counts off a different terminal frame, but they all
+ * have to land in {@link Usage} under one rule: **`undefined` means the provider
+ * did not report this figure, and never means zero.** Collapsing "not reported"
+ * to `0` silently understates spend — a model that billed no cached tokens and
+ * a model that says nothing about caching are different facts.
+ *
+ * A provider registers its mapper here rather than re-litigating the rule in its
+ * own suite, so a new provider inherits the same guarantees.
+ */
+/** The full set of normalized counters, used to check untouched slots. */
+declare const USAGE_COUNTER_FIELDS: readonly ["input", "output", "cached", "cacheWrite", "reasoning", "total"];
+type UsageCounterField = (typeof USAGE_COUNTER_FIELDS)[number];
+export interface UsageNormalizationCase {
+    /** What this frame represents, e.g. "a cache-hit completion". */
+    readonly name: string;
+    /** The provider-shaped payload handed to the mapper. */
+    readonly frame: unknown;
+    /** The exact expected result — compared with `toEqual`, so extra keys fail. */
+    readonly expected: Usage | undefined;
+}
+export interface UsageNormalizationOpts {
+    /** Provider name, used in test titles. */
+    readonly provider: string;
+    /** The provider's own mapper from a raw frame to {@link Usage}. */
+    readonly mapUsage: (raw: unknown) => Usage | undefined;
+    /** Provider-specific frames and their expected mappings. */
+    readonly cases: readonly UsageNormalizationCase[];
+    /**
+     * A minimal real frame reporting only `sparseReports`. Every counter outside
+     * that set must come back `undefined` — this is the zero-vs-absent guard.
+     */
+    readonly sparseFrame: unknown;
+    /** Counters the sparse frame actually reports. */
+    readonly sparseReports: readonly UsageCounterField[];
+    /**
+     * A frame in which the provider genuinely reports `0` for `sparseReports`.
+     * Proves the mapper does not discard a real zero along with the absent ones.
+     */
+    readonly zeroFrame: unknown;
+}
+/** Assert a mapped {@link Usage} carries only numbers and `undefined`. */
+export declare function assertUsageShape(usage: Usage): void;
+/**
+ * Assert the field relationships every provider's mapper must produce.
+ *
+ * The sum check only runs when the provider states a `total` — it is the
+ * provider's own arithmetic, so it is the one place the normalization can be
+ * checked against an external authority rather than against itself.
+ */
+export declare function assertUsageDisjointness(usage: Usage): void;
+export declare function usageNormalizationBlock(opts: UsageNormalizationOpts): void;
+export {};

package/dist/ai-provider/fixtures.d.ts

@@ -0,0 +1,8 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { ConformanceFixture } from "./types";
+export declare const DEFAULT_CONFORMANCE_FIXTURE: ConformanceFixture;
+export declare function resolveFixture(override: Partial<ConformanceFixture> | undefined): ConformanceFixture;

package/dist/ai-provider/runAiProviderConformance.d.ts

@@ -0,0 +1,7 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { AiProviderConformanceOpts } from "./types";
+export declare function runAiProviderConformance(opts: AiProviderConformanceOpts): void;

package/dist/ai-provider/types.d.ts

@@ -0,0 +1,75 @@
+/**
+ * @license
+ * Copyright 2025 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+import type { JsonSchema } from "@workglow/util/schema";
+export interface AiProviderConformanceOpts {
+    readonly name: string;
+    readonly skip?: boolean;
+    readonly timeout: number;
+    readonly factory: () => Promise<ConformanceHandle>;
+    readonly capabilities: AiProviderCapabilities;
+    readonly models: AiProviderConformanceModels;
+    readonly fixture?: Partial<ConformanceFixture>;
+    /**
+     * Names of conformance assertions that are currently broken in this
+     * adapter. Each named assertion is wrapped in `it.fails` instead of `it`.
+     * Remove the entry once the adapter bug is fixed.
+     *
+     * Known names:
+     *   "signal.nonStreaming"
+     *   "signal.midStream"
+     *   "session.reuse"
+     */
+    readonly expectedFailures?: ReadonlyArray<string>;
+}
+export interface ConformanceHandle {
+    readonly register: () => Promise<void>;
+    readonly dispose: () => Promise<void>;
+    readonly inspect: () => ProviderInspectionHandle;
+    /**
+     * Optional hook the conformance harness calls before the session-reuse
+     * block to release transient state (e.g. cached chat sessions) that prior
+     * blocks may have accumulated. Implementations must NOT tear down models
+     * or contexts — only short-lived per-call resources.
+     */
+    readonly releaseTransients?: () => Promise<void>;
+}
+export interface AiProviderCapabilities {
+    readonly streaming: boolean;
+    readonly tools: boolean;
+    readonly structured: boolean;
+    readonly embeddings: boolean;
+    readonly sessions: boolean;
+    readonly abortMidStream: boolean;
+}
+export interface AiProviderConformanceModels {
+    readonly textGeneration?: string;
+    readonly toolCalling?: string;
+    readonly structured?: string;
+    readonly embeddings?: string;
+}
+export interface ProviderInspectionHandle {
+    readonly sessionMap?: ReadonlyMap<string, unknown>;
+    readonly disposables?: ReadonlyArray<{
+        readonly alive: boolean;
+    }>;
+}
+export interface ConformanceFixture {
+    readonly textPrompt: string;
+    readonly weatherTool: {
+        readonly name: string;
+        readonly description: string;
+        readonly inputSchema: JsonSchema;
+    };
+    readonly weatherToolPrompt: string;
+    readonly multiTurnTranscript: ReadonlyArray<{
+        readonly role: "user" | "assistant" | "tool";
+        readonly text: string;
+    }>;
+    readonly structuredSchema: JsonSchema;
+    readonly structuredPrompt: string;
+    readonly maxTokens: number;
+    readonly abortGraceMs: number;
+}

package/dist/ai-provider.d.ts

@@ -0,0 +1,25 @@
+/**
+ * @license
+ * Copyright 2026 Steven Roussey <sroussey@gmail.com>
+ * SPDX-License-Identifier: Apache-2.0
+ */
+/**
+ * Conformance suite for an AI provider adapter: what it advertises it can do
+ * is what it serves, over both the inline and worker registrations.
+ */
+export * from "./ai-provider/fixtures";
+export * from "./ai-provider/runAiProviderConformance";
+export * from "./ai-provider/types";
+export * from "./ai-provider/assertions/capabilityHonesty";
+export * from "./ai-provider/assertions/dispose";
+export * from "./ai-provider/assertions/inferAdvertisesRegistered";
+export * from "./ai-provider/assertions/inferServesInferred";
+export * from "./ai-provider/assertions/pricingMatchesModality";
+export * from "./ai-provider/assertions/registryCoverage";
+export * from "./ai-provider/assertions/sessionReuse";
+export * from "./ai-provider/assertions/signalHonoring";
+export * from "./ai-provider/assertions/structuredGeneration";
+export * from "./ai-provider/assertions/textGenerationSmoke";
+export * from "./ai-provider/assertions/toolCallAccumulator";
+export * from "./ai-provider/assertions/toolCallMultiTurn";
+export * from "./ai-provider/assertions/usageNormalization";