streamtrust 0.2.0 → 0.3.0

package/README.md

@@ -4,9 +4,13 @@
 
 **[View StreamTrust on npm](https://www.npmjs.com/package/streamtrust)**
 
-Adapter-driven CI checks for streaming applications: compare observed output with authoritative state, exercise cancellation, and verify retry identity.
+StreamTrust is a CI check for streaming applications such as AI chat. Its reusable checks compare what was displayed with what the application authoritatively saved and report disagreements as failed checks. It does not replace Playwright: Playwright drives the browser, and the bundled presets use it inside their adapters.
 
-**Status:** `0.2.0`, an early 0.x release for independent teams. APIs and report formats may change before 1.0. The CLI is dependency-free, ESM, and requires Node.js 22+. MIT licensed. Browser and database libraries belong to your adapter and are not installed by StreamTrust.
+- **Three outcomes, kept apart.** "The app failed" (FAIL), "this capability is not supported" (UNSUPPORTED) and "we could not determine whether it failed" (HARNESS_ERROR) are reported separately, and harness errors exit with their own code.
+- **Repeated runs.** Each scenario runs N times and the report shows every repetition, because cancellation and retry bugs are often intermittent.
+- **Your contract, written down.** An adapter implements your app's start, stream, cancel, retry and saved-state operations; StreamTrust invokes the adapter scenarios and writes JSON, JUnit, Markdown and HTML reports.
+
+**Status:** `0.3.0`, an early 0.x release for independent teams. APIs and report formats may change before 1.0. The CLI is dependency-free, ESM, and requires Node.js 22+. MIT licensed. Browser and database libraries belong to your adapter and are not installed by StreamTrust.
 
 ## Watch the 66-second demo
 
@@ -24,13 +28,19 @@ npx streamtrust init
 npx streamtrust verify --adapter ./streamtrust/adapter.mjs
 ```
 
-For reproducible CI, pin `streamtrust@0.2.0` and commit your lockfile. The generated adapter initially exercises the bundled demonstration; map it to your application before using its results as application evidence.
+For reproducible CI, pin `streamtrust@0.3.0` and commit your lockfile. The generated adapter initially exercises the bundled demonstration; map it to your application before using its results as application evidence.
 
 ## LibreChat integration
 
 A LibreChat preset now exercises persistence, native Stop, regeneration and error-retry through its browser UI and disposable MongoDB. It uses reusable `streamtrust/checks` assertions and summary-only evidence; no chat text, screenshots or application logs are included in its reports.
 
-**Available in StreamTrust 0.2.0.** Install from npm using the [LibreChat quick start and CI guide](examples/librechat/README.md). It requires Node 24 and optional Playwright/MongoDB dependencies. An [integration workflow](.github/workflows/librechat.yml) exercises the packed CLI against the pinned upstream version. The default `init` demonstration remains available.
+**Available since StreamTrust 0.2.0.** Install from npm using the [LibreChat quick start and CI guide](examples/librechat/README.md). It requires Node 24 and optional Playwright/MongoDB dependencies. An [integration workflow](.github/workflows/librechat.yml) exercises the packed CLI against the pinned upstream version. The default `init` demonstration remains available.
+
+## Vercel AI SDK integration
+
+A `vercel-ai` preset runs a minimal Next.js chat built on the AI SDK (`useChat`, `streamText`) in production mode with a synthetic OpenAI-compatible provider, so it needs no API key. It covers the same four scenarios through the browser and the app's saved state, and can be pointed at your own built AI SDK app after you adjust its selectors and saved-state read. It is not an official Vercel integration.
+
+**Available in StreamTrust 0.3.0.** Install from npm and generate it with `npx streamtrust init streamtrust-vercel-ai --preset vercel-ai`. See the [Vercel AI SDK quick start](examples/vercel-ai/README.md) for what it does and does not verify. It requires Node 22+ and Playwright in your verification project. An [integration workflow](.github/workflows/vercel-ai.yml) exercises the packed CLI against the reference app.
 
 ## Run from source
 
@@ -50,7 +60,7 @@ Build the tarball with `npm pack` in this repository.
 Put the supplied tarball in your project's `vendor/` directory, then:
 
 ```sh
-npm install --save-dev ./vendor/streamtrust-0.2.0.tgz
+npm install --save-dev ./vendor/streamtrust-0.3.0.tgz
 npx streamtrust init
 npx streamtrust verify --adapter ./streamtrust/adapter.mjs
 ```
@@ -146,7 +156,7 @@ Add StreamTrust as a test step in your existing CI. For GitHub Actions, follow t
 1. Install StreamTrust and create the starting adapter:
 
    ```sh
-   npm install --save-dev --save-exact streamtrust@0.2.0
+   npm install --save-dev --save-exact streamtrust@0.3.0
    npx streamtrust init
    ```
 
@@ -229,6 +239,6 @@ Use a unique output directory per concurrent job. The CLI takes an exclusive loc
 
 ## Scope and attribution
 
-StreamTrust checks what an adapter asserts. It does not prove exactly-once external effects, AI factuality, automatic recovery, production incident rates or adoption. The HTTP demo is synthetic. Separate development experiments have tested pinned Vercel Chatbot and LibreChat applications; the LibreChat adapter is now available as an npm preset with optional dependencies. The Vercel experiment remains separate.
+StreamTrust checks what an adapter asserts. It does not prove exactly-once external effects, AI factuality, automatic recovery, production incident rates or adoption. The HTTP demo is synthetic. Separate development experiments have tested pinned Vercel Chatbot and LibreChat applications; the LibreChat adapter is now available as an npm preset with optional dependencies. The `vercel-ai` preset targets its own minimal reference app, not the Vercel Chatbot template; that experiment remains separate.
 
 Local preparation: `npm pack` in this package directory runs the package tests and creates the tarball. The public repository runs tests and a clean tarball installation on Node.js 22 and 24. This release is published interactively; npm build provenance is not configured.

package/examples/librechat/README.md

@@ -4,11 +4,11 @@ This preset runs a **built local LibreChat checkout**, native browser UI and rea
 
 ## Installation
 
-Use Node.js 24 and npm 11. In a separate verification project, install StreamTrust 0.2.0 from npm:
+Use Node.js 24 and npm 11. In a separate verification project, install StreamTrust 0.3.0 from npm:
 
 ```sh
 npm init -y
-npm install --save-dev --save-exact streamtrust@0.2.0
+npm install --save-dev --save-exact streamtrust@0.3.0
 npm install --save-dev --save-exact playwright@1.64.1 mongodb@6.20.0 mongodb-memory-server@11.3.0
 npx playwright install chromium
 npx streamtrust init streamtrust-librechat --preset librechat

package/examples/vercel-ai/README.md

@@ -0,0 +1,108 @@
+# Vercel AI SDK CI integration
+
+This preset runs a **minimal Next.js chat built on the AI SDK** (`ai`, `@ai-sdk/react` `useChat`, `streamText`) in production mode, drives its UI with Playwright, and compares what the browser showed with what the app saved. A synthetic OpenAI-compatible HTTP provider supplies deterministic text, so no model key is needed. Four scenarios cover saved answers, Stop, regeneration branches, and retry after a provider failure. This is not an official Vercel integration and is not endorsed by Vercel.
+
+## Quick start (no API key)
+
+Use Node.js 22 or newer (AI SDK 7 requires it). This preset is available in StreamTrust 0.3.0. In a separate verification project:
+
+```sh
+npm init -y
+npm install --save-dev --save-exact streamtrust@0.3.0 playwright@1.64.1
+npx playwright install chromium
+npx streamtrust init streamtrust-vercel-ai --preset vercel-ai
+node streamtrust-vercel-ai/setup.mjs
+npx streamtrust verify --adapter streamtrust-vercel-ai/adapter.mjs \
+  --output reports/vercel-ai --repetitions 3
+```
+
+`setup.mjs` installs and builds `streamtrust-vercel-ai/reference-app` (Next.js 16, AI SDK 7, pinned exact versions). The bundled `npm-shrinkwrap.json` pins transitive dependencies and setup uses `npm ci`. Commit the generated preset including this lockfile, and ignore `reference-app/node_modules/` and `reference-app/.next/`. Build and verification time depend on the machine. A cold install needs network access for npm and the Chromium download.
+
+The verifier starts the built app on a loopback-only ephemeral port with a fresh temporary chat store, starts the provider and a headless browser, and tears all of them down afterwards. It does not forward your environment to the app (only `PATH`, `HOME` and temp-directory variables), and refuses an app directory containing `.env`, `.env.local`, `.env.production` or `.env.production.local`, because Next.js would load those files.
+
+## What each scenario checks
+
+The adapter maps observations onto the shared `streamtrust/checks` lifecycle evaluator:
+
+| Scenario | Action | Key assertions |
+|---|---|---|
+| `persistence` | Send, let the provider finish | Exactly one saved answer equals the streamed text; it survives reload; reload does not call the model again |
+| `cancellation` | Click Stop after the first chunk | The provider connection is aborted before completion; the partial answer is saved as unfinished; a late completion cannot be delivered |
+| `regeneration` | Finish, then click Regenerate | The new answer is saved as a sibling of the original (same parent, distinct id); the original is kept; only the new answer is shown after reload |
+| `error-retry` | Provider drops mid-stream, reload, Regenerate | The failure is saved and still visible after reload; the retry is saved as a complete answer; the failed partial is no longer shown |
+
+The reference app is a fixture for validating the verifier, not evidence of an upstream defect. Negative controls and their observed results are recorded in the release notes.
+
+Two semantics are specific to the AI SDK and worth knowing:
+
+- **Stop is a client-side fetch abort.** There is no server stop endpoint, so `cancel.server-acknowledged` means "the server persisted the aborted state", observed in the store, not a response to a stop request.
+- **Regeneration is checked as branching.** The reference app keeps the old answer as a sibling. Many AI SDK apps instead save the client's `messages` array in `onFinish`, which drops the old answer on regenerate. Against such an app the regeneration branch checks fail by design. If replacing history is your intended behavior, return `{ checks: [], evidence: {}, unsupported: 'Regenerate replaces history' }` for that scenario instead of weakening the check.
+
+## Verify your own AI SDK app
+
+The reference app is a demonstration: **its passing says nothing about your application.** To test your app, point the adapter at a built checkout:
+
+```sh
+export STREAMTRUST_VERCEL_AI_ROOT=/path/to/your/next-app
+# Build your isolated checkout using its usual build command.
+npx streamtrust verify --adapter streamtrust-vercel-ai/adapter.mjs --output reports/vercel-ai
+```
+
+Then edit `adapter.mjs` where your app differs. Expect HARNESS_ERROR, not FAIL, until the mapping matches:
+
+1. **Provider wiring.** The adapter passes `MODEL_BASE_URL`, `MODEL_API_KEY` and `MODEL_NAME` to the app. Your route must create its model from those, for example `createOpenAICompatible({ name, baseURL: process.env.MODEL_BASE_URL, apiKey: process.env.MODEL_API_KEY })`, at least in test builds. Without this the app calls a real provider or fails to start.
+2. **Authoritative read.** `saved(id)` reads the reference app's JSON store from `CHAT_STORE_DIR`. Replace it with a read of your database, mapping each assistant message to `{ id, parentId, text, status }` where status is `complete`, `streaming`, `aborted` or `error`.
+3. **UI selectors.** The adapter types into the textbox named `Message` and presses Enter, then uses the buttons named `Stop` and `Regenerate`, the `data-testid="messages"` container, the URL `/chat/<id>`, and the failure text `The model provider failed.`. Change them to match your UI.
+4. **Other services.** Start any database, auth or migrations your app needs in the adapter factory, and register their cleanup.
+
+Optional settings:
+
+| Variable | Purpose |
+|---|---|
+| `STREAMTRUST_VERCEL_AI_ROOT` | Built app directory; defaults to the bundled `reference-app` |
+| `STREAMTRUST_VERCEL_AI_PORT` | Fixed loopback port; defaults to an ephemeral port |
+| `STREAMTRUST_BROWSER_CHANNEL` | Installed browser channel such as `chrome`; defaults to Playwright Chromium |
+| `STREAMTRUST_VERCEL_AI_DEBUG` | `1` prints raw adapter errors and app stderr to the console (never into reports) |
+
+## CI
+
+Run the installation steps once, then commit your verification project package.json, lockfile, and generated preset. Use a job with `permissions: { contents: read }`, `timeout-minutes: 20`, and no production secrets. These steps run the bundled reference app; adapt the adapter and build step to verify your own application.
+
+```yaml
+- uses: actions/checkout@v4
+- uses: actions/setup-node@v4
+  with:
+    node-version: '22'
+    cache: npm
+- run: npm ci
+- run: npx --no-install playwright install --with-deps chromium
+- run: node streamtrust-vercel-ai/setup.mjs
+- name: Verify AI SDK app
+  run: npx --no-install streamtrust verify --adapter streamtrust-vercel-ai/adapter.mjs --output reports/vercel-ai --repetitions 3
+- uses: actions/upload-artifact@v4
+  if: always()
+  with:
+    name: vercel-ai-verification
+    path: |
+      reports/vercel-ai/report.json
+      reports/vercel-ai/report.md
+      reports/vercel-ai/junit.xml
+      reports/vercel-ai/index.html
+    retention-days: 7
+```
+
+Exit 0 means all declared checks passed; 1 means failed or unsupported checks; 2 means setup, timeout or harness error. Keep the exit status: do not use `continue-on-error`.
+
+## Evidence and limits
+
+Reports use `summary-only-v1`: stable check names, booleans, message counts and provider-call counts. Chat text, message ids, URLs and file paths stay in memory. Reports replace browser and app exceptions with fixed messages even in debug mode. Debug mode prints raw errors to the console; do not enable it in CI that retains logs. This is data minimization, not a sandbox: the adapter and the app under test are trusted code.
+
+What this does **not** verify: real model behavior or output quality, multiple server instances, resumable streams (`resume: true`), server crashes or redeploys mid-stream, tool calls, the AI SDK's own correctness in general, or production reliability. The model is synthetic, and the reference app's file store is a stand-in for a database. The fixture has no authentication or concurrency control and must not be deployed as a production chat app. A single matching saved answer is not proof of exactly-once side effects.
+
+## Release validation
+
+For 0.3.0, a clean tarball consumer on macOS with Node 22.22.3 and Chromium 156.0.8078.4 passed all four scenarios with three repetitions (12 executions, 99 assertions). The 17 package tests passed on Node 22 and 24. The locked fixture dependencies reported no known vulnerabilities in `npm audit` at review time.
+
+Three isolated mutations were each detected as FAIL with exit 1: removing request-abort propagation (`cancel.provider-aborted`), deleting the original regeneration branch (`retry.original-branch-preserved`), and skipping persistence for failed outcomes (`error.saved-before-retry`). These are deliberately broken copies of our fixture, not bugs attributed to Vercel or another team's app. Run `node test/vercel-ai-controls.mjs /path/to/consumer preset-directory` from the source repository to reproduce them against a disposable generated preset; the script restores the route and rebuilds afterwards.
+
+The [Linux GitHub Actions run](https://github.com/santoshsibyala/streamtrust/actions/runs/40817581908) also passed the clean install/build, all 12 baseline executions, all three negative controls, and report artifact upload. Successful report checks found none of the tested synthetic text, local-path, or key markers; this is not a guarantee against arbitrary sensitive data in customized adapters.

package/examples/vercel-ai/adapter.mjs

@@ -0,0 +1,333 @@
+import { assertIsolatedCheckout, normalizeAssistant } from './config.mjs';
+import { existsSync } from 'node:fs';
+import { evaluateLifecycle } from 'streamtrust/checks';
+import { createRequire } from 'node:module';
+import { spawn } from 'node:child_process';
+import { mkdtemp, readFile, rm } from 'node:fs/promises';
+import { tmpdir } from 'node:os';
+import { createServer } from 'node:net';
+import { join, resolve } from 'node:path';
+import { createHash, randomUUID } from 'node:crypto';
+import { fileURLToPath } from 'node:url';
+
+import { startProvider } from './provider.mjs';
+
+// Visible text the app renders for a failed generation; change it for your app.
+const FAILED = 'The model provider failed.';
+const debug = process.env.STREAMTRUST_VERCEL_AI_DEBUG === '1';
+export async function until(check, label, ms = 20000) {
+  const end = Date.now() + ms;
+  while (Date.now() < end) {
+    if (await check()) return;
+    await new Promise((r) => setTimeout(r, 100));
+  }
+  throw new Error(`Timeout: ${label}`);
+}
+let phase = 'preflight';
+async function createAdapter() {
+  if (Number(process.versions.node.split('.')[0]) < 22)
+    throw new Error('vercel-ai preset requires Node.js 22+');
+  const root = resolve(
+    process.env.STREAMTRUST_VERCEL_AI_ROOT ||
+      fileURLToPath(new URL('./reference-app', import.meta.url)),
+  );
+  assertIsolatedCheckout(root);
+  if (!existsSync(resolve(root, '.next/BUILD_ID')))
+    throw new Error('App is not built; run setup.mjs first');
+  const configPort = Number(process.env.STREAMTRUST_VERCEL_AI_PORT || 0);
+  if (!Number.isInteger(configPort) || configPort < 0 || configPort > 65535)
+    throw new Error('Invalid app port');
+  const port =
+    configPort ||
+    (await new Promise((ok, fail) => {
+      const server = createServer();
+      server.once('error', fail);
+      server.listen(0, '127.0.0.1', () => {
+        const value = server.address().port;
+        server.close(() => ok(value));
+      });
+    }));
+  const baseUrl = `http://127.0.0.1:${port}`;
+  const require = createRequire(resolve(process.cwd(), 'package.json'));
+  const { chromium } = require('playwright');
+  const appRequire = createRequire(resolve(root, 'package.json'));
+  const lockPath = ['npm-shrinkwrap.json', 'package-lock.json'].map((name) => resolve(root, name)).find(existsSync);
+  const versions = Object.fromEntries(
+    ['ai', 'next', '@ai-sdk/react'].map((name) => {
+      try {
+        return [name, appRequire(`${name}/package.json`).version];
+      } catch {
+        return [name, 'unknown'];
+      }
+    }),
+  );
+  const cleanups = [];
+  let cleaned = false;
+  const cleanup = async () => {
+    if (cleaned) return;
+    cleaned = true;
+    let cleanupFailed = false;
+    for (const close of cleanups.reverse()) {
+      try {
+        await close();
+      } catch {
+        cleanupFailed = true;
+      }
+    }
+    if (cleanupFailed) throw new Error('Vercel AI preset cleanup failed');
+  };
+  try {
+    const provider = await startProvider();
+    cleanups.push(() => provider.close());
+    phase = 'disposable store';
+    const store = await mkdtemp(join(tmpdir(), 'streamtrust-vercel-ai-'));
+    cleanups.unshift(() => rm(store, { recursive: true, force: true }));
+    const runtime = process.execPath;
+    const env = {
+      // Do not forward provider keys, database URLs or application secrets.
+      ...Object.fromEntries(
+        ['PATH', 'HOME', 'TMPDIR', 'TEMP', 'SystemRoot']
+          .filter((k) => process.env[k])
+          .map((k) => [k, process.env[k]]),
+      ),
+      NODE_ENV: 'production',
+      NEXT_TELEMETRY_DISABLED: '1',
+      CHAT_STORE_DIR: store,
+      MODEL_BASE_URL: provider.url,
+      MODEL_API_KEY: 'fixture-not-a-real-key',
+      MODEL_NAME: 'fixture-model',
+    };
+    phase = 'application startup';
+    const app = spawn(
+      runtime,
+      [
+        appRequire.resolve('next/dist/bin/next'),
+        'start',
+        '--hostname',
+        '127.0.0.1',
+        '--port',
+        String(port),
+      ],
+      {
+        cwd: root,
+        env,
+        stdio: ['ignore', 'ignore', debug ? 'inherit' : 'ignore'],
+      },
+    );
+    let appError = false;
+    app.on('error', () => {
+      appError = true;
+    });
+    cleanups.push(async () => {
+      if (!appError && app.exitCode === null && app.signalCode === null) {
+        const stopped = new Promise((r) => app.once('exit', () => r()));
+        app.kill('SIGTERM');
+        const timer = setTimeout(() => app.kill('SIGKILL'), 10000);
+        await stopped;
+        clearTimeout(timer);
+      }
+    });
+    phase = 'browser startup';
+    const browser = await chromium.launch(
+      !process.env.STREAMTRUST_BROWSER_CHANNEL
+        ? { headless: true }
+        : { channel: process.env.STREAMTRUST_BROWSER_CHANNEL, headless: true },
+    );
+    cleanups.push(() => browser.close());
+    phase = 'application readiness';
+    await until(
+      async () => {
+        if (appError || app.exitCode !== null)
+          throw new Error('App startup failed; check the build and port');
+        try {
+          return (await fetch(baseUrl, { signal: AbortSignal.timeout(2000) }))
+            .ok;
+        } catch {
+          return false;
+        }
+      },
+      'app ready',
+      60000,
+    );
+    const context = await browser.newContext();
+    // Authoritative read. For your app, replace this with a query against its database.
+    async function saved(id) {
+      try {
+        return JSON.parse(await readFile(join(store, `${id}.json`), 'utf8'))
+          .messages;
+      } catch (error) {
+        if (error.code === 'ENOENT') return [];
+        throw error;
+      }
+    }
+    const assistants = async (id) =>
+      (await saved(id)).filter((m) => m.role === 'assistant');
+    const observed = async (check) => {
+      const deadline = Date.now() + 15000;
+      while (Date.now() < deadline) {
+        if (await check()) return true;
+        await new Promise((r) => setTimeout(r, 100));
+      }
+      return false;
+    };
+    const visible = (p) => p.getByTestId('messages').innerText();
+    const stopButton = (p) =>
+      p.getByRole('button', { name: 'Stop', exact: true });
+    const settled = (p, label) =>
+      until(async () => !(await stopButton(p).count()), label);
+    const reload = async (p) => {
+      await p.reload();
+      await p.getByRole('textbox', { name: 'Message' }).waitFor();
+    };
+    async function waitGeneration(p, index) {
+      await until(
+        () => provider.generations.length > index,
+        'provider request',
+      );
+      const g = provider.generations[index];
+      await until(
+        async () => (await visible(p)).includes(g.prefix),
+        'first rendered chunk',
+      );
+      return g;
+    }
+    return {
+      name: 'Vercel AI SDK reference app',
+      scenarios: ['persistence', 'cancellation', 'regeneration', 'error-retry'],
+      async metadata() {
+        return {
+          evidencePolicy: 'summary-only-v1',
+          checkVersion: 1,
+          app: process.env.STREAMTRUST_VERCEL_AI_ROOT
+            ? 'configured application'
+            : 'bundled reference app',
+          versions,
+          node: process.version,
+          browser: browser.version(),
+          lockSha256: lockPath
+            ? createHash('sha256')
+                .update(await readFile(lockPath))
+                .digest('hex')
+            : 'none',
+          scope:
+            'Real Next.js production server, useChat UI, AI SDK route and the app-owned store. Only the OpenAI-compatible model responses are synthetic. Single server process; no multi-instance, resumable-stream, server-crash or production coverage. Stop is a client fetch abort: there is no server stop endpoint, so acknowledgement means the server persisted the aborted state.',
+        };
+      },
+      async run(scenario, { repetition }) {
+        const p = await context.newPage();
+        let first;
+        let second;
+        try {
+          const id = randomUUID();
+          await p.goto(`${baseUrl}/chat/${id}`);
+          const input = p.getByRole('textbox', { name: 'Message' });
+          await input.waitFor();
+          const before = provider.generations.length;
+          await input.fill(
+            `StreamTrust ${scenario} ${repetition}. Write two sentences.`,
+          );
+          await input.press('Enter');
+          first = await waitGeneration(p, before);
+          const cancellation = {};
+          const failure = {};
+          let expected = first.prefix + first.suffix;
+          if (scenario === 'cancellation') {
+            await stopButton(p).click();
+            await observed(() => first.closed);
+            expected = first.prefix;
+            cancellation.acknowledged = await observed(async () =>
+              (await assistants(id)).some((m) => m.status === 'aborted'),
+            );
+            cancellation.providerClosed = first.closed;
+            cancellation.providerFinished = first.finished;
+            cancellation.lateCompletionRejected = !provider.finish(first);
+          } else if (scenario === 'error-retry') {
+            provider.fail(first);
+          } else provider.finish(first);
+          await settled(p, 'generation settles');
+          if (scenario === 'error-retry') {
+            failure.saved = await observed(async () =>
+              (await assistants(id)).some((m) => m.status === 'error'),
+            );
+            await reload(p);
+            failure.visibleAfterReload = await observed(async () =>
+              (await visible(p)).includes(FAILED),
+            );
+          }
+          if (scenario === 'regeneration' || scenario === 'error-retry') {
+            const nextIndex = provider.generations.length;
+            await p
+              .getByRole('button', { name: 'Regenerate', exact: true })
+              .click();
+            second = await waitGeneration(p, nextIndex);
+            provider.finish(second);
+            expected = second.prefix + second.suffix;
+            await settled(p, 'retry settles');
+          }
+          await observed(async () =>
+            (await assistants(id)).some(
+              (m) =>
+                m.text.trim() === expected.trim() && m.status !== 'streaming',
+            ),
+          );
+          const durable = await assistants(id);
+          const beforeReload = await visible(p);
+          const callsBeforeReload = provider.generations.length;
+          await reload(p);
+          await observed(async () => (await visible(p)).includes(expected));
+          const reloaded = await visible(p);
+          return evaluateLifecycle({
+            scenario,
+            expected,
+            original: first.prefix + first.suffix,
+            previousPrefix: first.prefix,
+            beforeReload,
+            reloaded,
+            messages: durable.map(normalizeAssistant),
+            callsBeforeReload,
+            callsAfterReload: provider.generations.length,
+            cancellation:
+              scenario === 'cancellation' ? cancellation : undefined,
+            failure: scenario === 'error-retry' ? failure : undefined,
+          });
+        } finally {
+          if (first && !first.closed) provider.finish(first);
+          if (second && !second.closed) provider.finish(second);
+          await p.close();
+        }
+      },
+      close: cleanup,
+    };
+  } catch (error) {
+    await cleanup();
+    throw error;
+  }
+}
+
+// Browser/app exceptions can contain prompts, URLs and file paths; reports get fixed messages.
+// STREAMTRUST_VERCEL_AI_DEBUG=1 prints raw errors and app stderr to the console, never to reports.
+const withheld = (message) => (error) => {
+  if (debug) console.error(error);
+  throw new Error(message);
+};
+export default async function safeFactory() {
+  phase = 'preflight';
+  const adapter = await createAdapter().catch((error) =>
+    withheld(
+      `Vercel AI preset setup failed during ${phase}. Check Node 22+, Playwright, a built app without .env files, and the port. Raw details withheld.`,
+    )(error),
+  );
+  const run = adapter.run.bind(adapter);
+  adapter.run = (...args) =>
+    run(...args).catch(
+      withheld(
+        'Vercel AI scenario could not complete (details withheld by summary-only policy)',
+      ),
+    );
+  const metadata = adapter.metadata.bind(adapter);
+  adapter.metadata = () =>
+    metadata().catch(
+      withheld('Vercel AI metadata unavailable (details withheld)'),
+    );
+  return adapter;
+}

package/examples/vercel-ai/config.mjs

@@ -0,0 +1,17 @@
+import { existsSync } from 'node:fs';
+import { resolve } from 'node:path';
+
+export function assertIsolatedCheckout(root) {
+  // Next.js production builds and servers load all four of these files.
+  if (['.env', '.env.local', '.env.production', '.env.production.local']
+    .some((name) => existsSync(resolve(root, name))))
+    throw new Error('Use an app checkout without environment files');
+}
+
+export function normalizeAssistant(m) {
+  if (!['complete', 'streaming', 'aborted', 'error'].includes(m.status))
+    throw new Error('Unknown saved assistant status');
+  return { id: m.id, parentId: m.parentId, text: m.text, assistant: true,
+    unfinished: m.status === 'aborted' || m.status === 'streaming',
+    error: m.status === 'error' };
+}

package/examples/vercel-ai/provider.mjs

@@ -0,0 +1,79 @@
+import { createServer } from 'node:http';
+
+export async function startProvider() {
+  const generations = [];
+  const chunk = (g, delta, finish = null) =>
+    g.response.write(
+      `data: ${JSON.stringify({ id: `chatcmpl-fixture-${g.id}`, object: 'chat.completion.chunk', created: 1, model: 'fixture-model', choices: [{ index: 0, delta: { content: delta }, finish_reason: finish }] })}\n\n`,
+    );
+  const server = createServer(async (req, res) => {
+    let body = '';
+    for await (const part of req) body += part;
+    if (req.method !== 'POST') {
+      res.setHeader('content-type', 'application/json');
+      res.end(
+        JSON.stringify({ data: [{ id: 'fixture-model', object: 'model' }] }),
+      );
+      return;
+    }
+    const input = JSON.parse(body);
+    if (!input.stream) {
+      res.setHeader('content-type', 'application/json');
+      res.end(
+        JSON.stringify({
+          choices: [
+            {
+              message: { role: 'assistant', content: 'Fixture conversation' },
+              finish_reason: 'stop',
+            },
+          ],
+        }),
+      );
+      return;
+    }
+    const id = generations.length + 1;
+    const g = {
+      id,
+      prefix: `Response ${id} begins.`,
+      suffix: ` Response ${id} completes.`,
+      closed: false,
+      finished: false,
+      failed: false,
+      path: req.url,
+      response: res,
+    };
+    generations.push(g);
+    res.on('close', () => {
+      g.closed = true;
+    });
+    res.writeHead(200, {
+      'content-type': 'text/event-stream',
+      'cache-control': 'no-cache',
+    });
+    chunk(g, g.prefix);
+  });
+  await new Promise((r) => server.listen(0, '127.0.0.1', r));
+  return {
+    url: `http://127.0.0.1:${server.address().port}/v1`,
+    generations,
+    finish(g) {
+      if (g.closed) return false;
+      chunk(g, g.suffix);
+      chunk(g, '', 'stop');
+      g.finished = true;
+      g.response.end('data: [DONE]\n\n');
+      return true;
+    },
+    fail(g) {
+      g.failed = true;
+      g.response.destroy();
+    },
+    observations() {
+      return generations.map(({ response, ...g }) => g);
+    },
+    async close() {
+      server.closeAllConnections();
+      await new Promise((r) => server.close(() => r()));
+    },
+  };
+}