streamtrust 0.1.1 → 0.3.0

package/README.md

@@ -1,8 +1,24 @@
 # StreamTrust CLI
 
-Adapter-driven CI checks for streaming applications: compare observed output with authoritative state, exercise cancellation, and verify retry identity.
+[![npm version](https://img.shields.io/npm/v/streamtrust.svg)](https://www.npmjs.com/package/streamtrust)
 
-**Status:** `0.1.1`, an early 0.x release for independent teams. APIs and report formats may change before 1.0. The CLI is dependency-free, ESM, and requires Node.js 22+. MIT licensed. Browser and database libraries belong to your adapter and are not installed by StreamTrust.
+**[View StreamTrust on npm](https://www.npmjs.com/package/streamtrust)**
+
+StreamTrust is a CI check for streaming applications such as AI chat. Its reusable checks compare what was displayed with what the application authoritatively saved and report disagreements as failed checks. It does not replace Playwright: Playwright drives the browser, and the bundled presets use it inside their adapters.
+
+- **Three outcomes, kept apart.** "The app failed" (FAIL), "this capability is not supported" (UNSUPPORTED) and "we could not determine whether it failed" (HARNESS_ERROR) are reported separately, and harness errors exit with their own code.
+- **Repeated runs.** Each scenario runs N times and the report shows every repetition, because cancellation and retry bugs are often intermittent.
+- **Your contract, written down.** An adapter implements your app's start, stream, cancel, retry and saved-state operations; StreamTrust invokes the adapter scenarios and writes JSON, JUnit, Markdown and HTML reports.
+
+**Status:** `0.3.0`, an early 0.x release for independent teams. APIs and report formats may change before 1.0. The CLI is dependency-free, ESM, and requires Node.js 22+. MIT licensed. Browser and database libraries belong to your adapter and are not installed by StreamTrust.
+
+## Watch the 66-second demo
+
+[![Open the narrated StreamTrust demo: persistence, cancellation, retry, and CI diagnostics](docs/media/streamtrust-demo-poster.png)](https://santoshsibyala.github.io/streamtrust/)
+
+**[Watch the narrated demo in your browser](https://santoshsibyala.github.io/streamtrust/)** · [Read the transcript](docs/media/streamtrust-demo-transcript.txt) · [Download subtitles](docs/media/streamtrust-demo-0.1.1-brian.srt) · [Silent version](https://raw.githubusercontent.com/santoshsibyala/streamtrust/main/docs/media/streamtrust-demo-0.1.1.mp4)
+
+A 66-second walkthrough narrated with ElevenLabs’ Brian voice (synthetic English narration), on-screen captions, and an optional English subtitle track: install StreamTrust, run the example checks, inspect the diagnostic report, and see a deliberate save failure return a nonzero CI exit code. Recorded with version 0.1.1. The bundled application and injected failure are demonstrations; connect an adapter to your own application to verify its behavior. The browser player includes playback controls, captions, and an MP4 download option.
 
 ## Install
 
@@ -12,7 +28,19 @@ npx streamtrust init
 npx streamtrust verify --adapter ./streamtrust/adapter.mjs
 ```
 
-For reproducible CI, pin `streamtrust@0.1.1` and commit your lockfile. The generated adapter initially exercises the bundled demonstration; map it to your application before using its results as application evidence.
+For reproducible CI, pin `streamtrust@0.3.0` and commit your lockfile. The generated adapter initially exercises the bundled demonstration; map it to your application before using its results as application evidence.
+
+## LibreChat integration
+
+A LibreChat preset now exercises persistence, native Stop, regeneration and error-retry through its browser UI and disposable MongoDB. It uses reusable `streamtrust/checks` assertions and summary-only evidence; no chat text, screenshots or application logs are included in its reports.
+
+**Available since StreamTrust 0.2.0.** Install from npm using the [LibreChat quick start and CI guide](examples/librechat/README.md). It requires Node 24 and optional Playwright/MongoDB dependencies. An [integration workflow](.github/workflows/librechat.yml) exercises the packed CLI against the pinned upstream version. The default `init` demonstration remains available.
+
+## Vercel AI SDK integration
+
+A `vercel-ai` preset runs a minimal Next.js chat built on the AI SDK (`useChat`, `streamText`) in production mode with a synthetic OpenAI-compatible provider, so it needs no API key. It covers the same four scenarios through the browser and the app's saved state, and can be pointed at your own built AI SDK app after you adjust its selectors and saved-state read. It is not an official Vercel integration.
+
+**Available in StreamTrust 0.3.0.** Install from npm and generate it with `npx streamtrust init streamtrust-vercel-ai --preset vercel-ai`. See the [Vercel AI SDK quick start](examples/vercel-ai/README.md) for what it does and does not verify. It requires Node 22+ and Playwright in your verification project. An [integration workflow](.github/workflows/vercel-ai.yml) exercises the packed CLI against the reference app.
 
 ## Run from source
 
@@ -32,7 +60,7 @@ Build the tarball with `npm pack` in this repository.
 Put the supplied tarball in your project's `vendor/` directory, then:
 
 ```sh
-npm install --save-dev ./vendor/streamtrust-0.1.1.tgz
+npm install --save-dev ./vendor/streamtrust-0.3.0.tgz
 npx streamtrust init
 npx streamtrust verify --adapter ./streamtrust/adapter.mjs
 ```
@@ -123,34 +151,82 @@ node --import tsx ./node_modules/streamtrust/bin/streamtrust.js verify \
 
 ## CI
 
-Install the pinned version and commit package.json and package-lock.json so CI can use `npm ci`. If using a locally built tarball instead, keep it in `vendor/` so the committed file dependency can be resolved. The example workflow assumes the application under test is already running; add its start/wait commands before verification. Omit the base URL only when intentionally testing the bundled demonstration.
+Add StreamTrust as a test step in your existing CI. For GitHub Actions, follow these steps:
+
+1. Install StreamTrust and create the starting adapter:
+
+   ```sh
+   npm install --save-dev --save-exact streamtrust@0.3.0
+   npx streamtrust init
+   ```
+
+2. Adapt `streamtrust/adapter.mjs` to your application's native start, stream, cancel, retry, and saved-state operations. **The generated adapter initially tests a bundled demo. Setting a base URL alone does not adapt it to arbitrary application APIs.** See [Connect your application](#connect-your-application) above.
+3. Commit the adapter, `package.json`, and `package-lock.json`. If the adapter imports browser or database libraries, add those dependencies and their required CI setup too.
+4. Save the workflow below as `.github/workflows/streamtrust.yml`, or copy its relevant steps into an existing job.
+
+**Replace `npm run start:test`, port `3000`, and `/health` with your application's test-server command and readiness endpoint.** These are placeholders, not commands or routes supplied by StreamTrust. Add any required build, database, migration, fixture, or browser-install steps before starting your app. Use a disposable test environment because the checks create and modify application state.
 
 ```yaml
 name: Streaming application verification
 on: [push, pull_request]
+permissions:
+  contents: read
 jobs:
   verify:
     runs-on: ubuntu-latest
     timeout-minutes: 10
     steps:
       - uses: actions/checkout@v4
+
       - uses: actions/setup-node@v4
         with:
           node-version: 22
           cache: npm
+
       - run: npm ci
-      # Start your test application and wait for readiness here.
-      - run: npx --no-install streamtrust verify --adapter ./streamtrust/adapter.mjs --output reports/streamtrust
+
+      # Add your application's build and test-service setup here.
+      - name: Start application and wait for readiness
+        shell: bash
+        run: |
+          npm run start:test > app.log 2>&1 &
+          app_pid=$!
+          for attempt in {1..30}; do
+            if ! kill -0 "$app_pid" 2>/dev/null; then
+              cat app.log
+              exit 1
+            fi
+            if curl --fail --silent --max-time 2 http://localhost:3000/health; then
+              exit 0
+            fi
+            sleep 2
+          done
+          cat app.log
+          echo "Application did not become ready in time."
+          exit 1
+
+      - name: Verify streaming behavior
+        run: |
+          npx --no-install streamtrust verify \
+            --adapter ./streamtrust/adapter.mjs \
+            --output ./reports/streamtrust
         env:
-          STREAMTRUST_BASE_URL: http://127.0.0.1:8080
-          STREAMTRUST_API_TOKEN: ${{ secrets.TEST_APP_TOKEN }}
-      - uses: actions/upload-artifact@v4
+          STREAMTRUST_BASE_URL: http://localhost:3000
+
+      - name: Save diagnostic reports
         if: always()
+        uses: actions/upload-artifact@v4
         with:
-          name: streamtrust-evidence
+          name: streamtrust-reports
           path: reports/streamtrust/
 ```
 
+If your application is already running in the job, omit the startup step and point your adapter at its test URL. For the bundled HTTP adapter's optional bearer authentication, add `STREAMTRUST_API_TOKEN: ${{ secrets.TEST_APP_TOKEN }}` to the verification step's `env`, and configure that secret in GitHub. Fork pull requests generally do not receive repository secrets; use a suitable unauthenticated test fixture or an explicitly trusted workflow for those runs. Never put credentials in the adapter or uploaded evidence.
+
+A failed assertion or harness error fails the verification step. The report-upload step runs even after a failure, so teams can inspect diagnostics when they were generated; startup failures may have no StreamTrust report. To block merging, configure the repository's branch protection or ruleset to require this workflow's `verify` check. StreamTrust does not configure merge rules itself.
+
+If you use a locally built tarball instead of npm, keep it in `vendor/` and commit the corresponding file dependency and lockfile so `npm ci` can resolve it.
+
 | Exit | Meaning |
 |---|---|
 | 0 | Every declared scenario and assertion passed. |
@@ -163,6 +239,6 @@ Use a unique output directory per concurrent job. The CLI takes an exclusive loc
 
 ## Scope and attribution
 
-StreamTrust checks what an adapter asserts. It does not prove exactly-once external effects, AI factuality, automatic recovery, production incident rates or adoption. The HTTP demo is synthetic. Separate development experiments have tested pinned Vercel Chatbot and LibreChat applications; their adapters and dependencies are not bundled here.
+StreamTrust checks what an adapter asserts. It does not prove exactly-once external effects, AI factuality, automatic recovery, production incident rates or adoption. The HTTP demo is synthetic. Separate development experiments have tested pinned Vercel Chatbot and LibreChat applications; the LibreChat adapter is now available as an npm preset with optional dependencies. The `vercel-ai` preset targets its own minimal reference app, not the Vercel Chatbot template; that experiment remains separate.
 
 Local preparation: `npm pack` in this package directory runs the package tests and creates the tarball. The public repository runs tests and a clean tarball installation on Node.js 22 and 24. This release is published interactively; npm build provenance is not configured.

package/examples/librechat/README.md

@@ -0,0 +1,91 @@
+# LibreChat CI integration
+
+This preset runs a **built local LibreChat checkout**, native browser UI and real disposable MongoDB. A controlled OpenAI-compatible HTTP provider supplies synthetic text. No live model key or existing database is used. Four scenarios cover saved answers, Stop, regeneration branches, and retry after a provider disconnect. This is not an official LibreChat integration.
+
+## Installation
+
+Use Node.js 24 and npm 11. In a separate verification project, install StreamTrust 0.3.0 from npm:
+
+```sh
+npm init -y
+npm install --save-dev --save-exact streamtrust@0.3.0
+npm install --save-dev --save-exact playwright@1.64.1 mongodb@6.20.0 mongodb-memory-server@11.3.0
+npx playwright install chromium
+npx streamtrust init streamtrust-librechat --preset librechat
+```
+
+Use Node.js 24 and npm 11 for this preset. Commit the generated adapter and your package lock. The core CLI still has no runtime dependencies; browser/database dependencies belong to the verification project. Browser, MongoDB binary and LibreChat installation require network access and disk space. The first build takes several minutes; this is not a five-minute cold installation.
+
+## Run the supported upstream version
+
+```sh
+export STREAMTRUST_LIBRECHAT_ROOT="$PWD/.cache/librechat"
+node streamtrust-librechat/setup.mjs
+npx streamtrust verify --adapter streamtrust-librechat/adapter.mjs \
+  --output reports/librechat --repetitions 3 --timeout-ms 600000
+```
+
+The setup script clones and builds commit `a378657583567310c841d652b0f6b00c83e2bd6b`. It refuses to reset an existing or dirty checkout. The verifier starts its own loopback-only server, chooses an available port, creates a synthetic test user, and tears down its browser, provider and disposable database. The preset refuses checkouts containing `.env`, `.env.local` or `.env.production`. Do not place production configuration in this checkout.
+
+To test your team's committed changes, point `STREAMTRUST_LIBRECHAT_ROOT` at your **built, clean checkout** and set `STREAMTRUST_LIBRECHAT_COMMIT` to its full SHA. Skip setup if already built. This executes the actual application under test; it does not copy a test app into your repository. Compatibility with other commits is not guaranteed: changed selectors/routes/schema can cause HARNESS_ERROR rather than a product failure.
+
+Optional settings:
+
+| Variable | Purpose |
+|---|---|
+| `STREAMTRUST_LIBRECHAT_ROOT` | Required path to the built checkout |
+| `STREAMTRUST_LIBRECHAT_COMMIT` | Expected full SHA; defaults to the tested upstream commit |
+| `STREAMTRUST_LIBRECHAT_PORT` | Optional fixed loopback port; defaults to an ephemeral port |
+| `STREAMTRUST_BROWSER_CHANNEL` | Optional installed browser channel, such as `chrome`; default is Playwright Chromium |
+
+## CI
+
+In your verification project, install once and commit package.json/package-lock.json and the generated preset. Add these steps to a Node 24 GitHub Actions job:
+
+```yaml
+- uses: actions/checkout@v4
+- uses: actions/setup-node@v4
+  with:
+    node-version: '24'
+    cache: npm
+- run: npm ci
+- run: npx --no-install playwright install --with-deps chromium
+- name: Build isolated LibreChat checkout
+  run: node streamtrust-librechat/setup.mjs
+  env:
+    STREAMTRUST_LIBRECHAT_ROOT: ${{ runner.temp }}/librechat
+- name: Verify LibreChat
+  run: npx --no-install streamtrust verify --adapter streamtrust-librechat/adapter.mjs --output reports/librechat --repetitions 3 --timeout-ms 600000
+  env:
+    STREAMTRUST_LIBRECHAT_ROOT: ${{ runner.temp }}/librechat
+- uses: actions/upload-artifact@v4
+  if: always()
+  with:
+    name: librechat-verification
+    path: |
+      reports/librechat/report.json
+      reports/librechat/report.md
+      reports/librechat/junit.xml
+      reports/librechat/index.html
+    retention-days: 7
+```
+
+Use a fresh report directory for each job. Restrict workflow permissions to `contents: read`, set a 30-minute job timeout, and use normal pull-request workflows without production secrets. To verify your fork, replace the build step with checkout/build of the intended commit and pass its SHA to the verifier.
+
+Exit 0 means all declared checks passed; 1 means failed or unsupported checks; 2 means setup, timeout, or harness error. Keep the exit status: do not use `continue-on-error`. Inspect reports before calling a failure an upstream defect.
+
+## Evidence and limits
+
+`summary-only-v1` returns stable check identifiers, booleans, message counts and provider-call counts. Raw chat text, database records, user identifiers, network URLs and tokens remain in memory and are not returned as report evidence. The preset does not write screenshots, browser traces or application logs. Browser/database exceptions are replaced with fixed errors. App subprocess output is discarded. Upload only the four report files, not the entire working directory.
+
+This is data minimization, not a universal PII detector or sandbox. Other adapters, changes to this adapter, setup/build logs and third-party processes are outside that guarantee. The core verifier still runs trusted local code. The preset deliberately does not forward application environment variables, but LibreChat can load files in its checkout: use an isolated checkout with no production configuration.
+
+The model provider is synthetic. The test uses one server and an in-memory stream store: no Redis, real model behavior, database crash, worker kill, multi-replica or production incident-rate coverage. A single matching saved answer is not proof of exactly-once external side effects. Regeneration is a distinct sibling branch, not reuse of a request idempotency key.
+
+The exported `streamtrust/checks` evaluator can be reused by other adapters. Map observations to `LifecycleObservation`; the evaluator validates required types, computes checks, and returns summary-only evidence. Missing or malformed observations throw, becoming HARNESS_ERROR rather than a green result.
+
+## Validation
+
+Local validation used macOS arm64, Node 24.22.1, Playwright 1.64.1 Chromium 156.0.8078.4, MongoDB 8.2.6, and the pinned LibreChat commit. A clean tarball consumer passed all four scenarios with three repetitions (12 executions, 99 assertions). A checkout containing an environment file was rejected with exit 2. Success and error artifact checks found only the four intended files and no tested secret/chat-text markers. The package suite has 13 tests, including negative controls for shared checks and stale-screenshot exclusion.
+
+The [Linux GitHub Actions run](https://github.com/santoshsibyala/streamtrust/actions/runs/40451851475) also passed a cold clone/install/build, clean tarball installation, all 12 scenario executions, and upload of exactly the four report files. The verified implementation commit is `b1b2b27a19b512c6abce164502901d1716bf86a3`. Passing results do not establish an upstream defect or production reliability.