@vanillaskyai/video 0.10.22 → 0.11.0

package/docs/development.md

@@ -1,23 +1,24 @@
-# Develop the chat experience
+# Development
 
-Run `npm ci --no-audit`, then `npm run dev:chat`. The localhost HMR surface renders the actual SDK `VideoChat`, templates and server handler from source. It never loads provider credentials. Select an intent and fault condition in the development toolbar; use the chat's normal media-mode controls. The prompt box remains the real product UI.
+Use Node 22+ and the locked npm version. Run `npm ci` once in your worktree.
 
-Offline answers are deterministic for explanation, story, comedy, imagination, practical steps and golf. They use local waterfall footage and locally synthesized speech matching each displayed opening, body and ending. This harness lets you hear complete narration while checking loading, timing, controls and recovery without AI credits. The waterfall remains generic fixture footage; it cannot prove generated answer quality or visual relevance. See [spoken fixture provenance](https://github.com/VanillaSkyAi/video/blob/main/dev/chat/speech/README.md) for transcripts and regeneration commands. Conditions cover ready, delayed footage, missing media, decode failure, speech failure, exhausted video allowance and request throttling. Browser speech may be used in the speech-failure condition.
+```bash
+npm run dev:chat
+```
 
-The toolbar labels source and fixture identity. It separates the first body surface from the first decoded moving-footage frame; the former is a renderer paint opportunity and can precede decode. Its bounded safe phase log records browser request/stream arrival phases, speech response completion, first speech, footage and buffer pauses. No prompts, narration, scene IDs or provider bodies are retained. Media start/end/skip reasons come from the handler’s separate host-only `onDiagnostic` callback, shown in the local terminal for offline fixtures; live hosts own that callback themselves. Stream arrival is not the server’s exact authorship timestamp. Local fixtures never fetch external footage or invoke a paid model. Mode boundaries, provider deadlines and host admission remain covered by their dedicated server and host suites.
+The local chat harness imports source directly for HMR. Keep media fixtures:
+they exercise real decoder, readiness, audio-clock, and replay boundaries.
+There is no catalog generation or registry synchronization step.
 
-## Optional live host
+For an ordinary edit, run the affected test files, lint, and typecheck. Add a
+focused regression for a behavior change. Run browser tests when playback or UI
+behavior changes; use the packed-consumer gates when exports, starter code, or
+public examples change. Do not run the complete release matrix repeatedly during
+the edit loop.
 
-Set `VANILLASKY_CHAT_LIVE_ENDPOINT` to an application-owned video-chat endpoint before starting the harness. The toolbar then offers **Connect live endpoint (uses allowance)**. It stays offline until that explicit click. Live fetches include host cookies, subject to browser cookie policy. The host must allow credentialed requests from the localhost origin and provide its normal authorization; credentials and allowances stay with that application. This tool does not proxy secrets, reset limits, or retry generated answers. Capabilities and welcome requests may run as soon as you connect. Request only the bounded live examples needed to evaluate actual quality.
+`npm run verify:release` is the final candidate gate: it builds one artifact and
+reuses it for clean-room consumers. `verify:package` already verifies public API
+declarations/runtime boundaries; `verify:api` is a targeted shortcut, not an
+additional full-release pass.
 
-## Fast checks
-
-`npm run check:chat` runs the harness/unit recovery cases, checks harness TypeScript and runs one Chromium recording with short footage looping under actual audio. Install Chromium once with `npx playwright install chromium`. The harness smoke additionally records submit-to-visible-chapter paint opportunity with a 200 ms warm-UI target. This is a browser animation-frame opportunity, not a physical display measurement. The target for the complete command is under a minute on a warm machine; the command reports its measured duration. It does not replace release checks or claim browser-wide/live-provider coverage.
-
-## One candidate for release verification
-
-`npm run verify:release` runs registry, lint, type, unit, acceptance, size, production dependency audit, API, packed-consumer, onboarding, provider and all existing browser checks. It builds and packs once into ignored `artifacts/chat-candidate/`, then sends the exact tarball and integrity to every consumer verifier. Each verifier still uses its independent clean install. Browser and provider compatibility remain independent gates; no paid API calls are enabled by this command.
-
-`candidate.json` records version, source commit, dirty-tree status, SHA-256 and npm integrity. A local candidate may include uncommitted work; it is never called a published artifact. `VANILLASKY_CANDIDATE_DIR` selects a different output directory. The command verifies but never publishes, merges or deploys. CI additionally retains the existing Node and React version checks.
-
-For one targeted consumer check, run `node scripts/chat-candidate.mjs`, then provide `VANILLASKY_PACKED_TARBALL`, `VANILLASKY_EXPECTED_INTEGRITY` and `VANILLASKY_EXPECTED_SHA256` from that manifest to an existing verifier. Do not repack between checks of the same candidate.
+[Architecture](architecture.md) · [Contributing](https://github.com/VanillaSkyAi/video/blob/main/CONTRIBUTING.md)

package/docs/errors.md

@@ -11,9 +11,9 @@ be produced. Never put provider details or stack traces in the interface.
 | Failure | Result |
 | --- | --- |
 | Invalid planner scene | Skip the part and accept later valid scenes |
-| Generated footage | Try stock media, matching completed footage, then a safe template |
-| Stock lookup or candidate | Continue with another candidate or template |
-| Template renderer | Isolate the failed scene with a safe visual |
+| Generated footage | Use a narrated chapter; never silently switch to stock |
+| Stock lookup or candidate | Continue with another stock candidate or narrated chapter |
+| Scene renderer | Isolate the failed scene with a safe visual |
 | Missing narration | Continue with scene text |
 | Generated speech | Use browser voice |
 | Late stream failure | Keep playable opening and completed scenes |
@@ -50,7 +50,7 @@ Automated regression and acceptance tests use mocked providers. See
 
 ## Slow optional providers
 
-Generated video has a 15-second deadline per lookup. Stock media, generated
+Generated video starts with a 15-second preparation budget; later deadlines account for the scene's position. Stock media, generated
 speech preparation, and fallback scene narration have 3-second deadlines.
 A deadline uses the same safe fallback as a failed provider; completed scenes
 and scene order are preserved. Providers receive cancellation, and late results

package/docs/getting-started.md

@@ -1,115 +1,96 @@
-[← Documentation home](../README.md) · [Next: Provider integration →](provider-integration.md)
-
 # Getting started
 
-The fastest VanillaSky integration is the complete, general-purpose video chat.
-It starts with a template introduction and browser voice. Add the video adapter
-and its server key for generated footage, or choose Pexels for stock footage.
-Without a media provider, the answer retains narration and subtitles with an
-chapter template. Generated speech is optional.
-
-## Create the app
-
-Start in an empty folder:
+This is the canonical quickstart. Create the default chat in an empty directory:
 
 ```bash
 npx @vanillaskyai/video init
 ```
 
-Init installs the exact SDK version that ran it, creates a small application
-shell, installs baseline dependencies, and runs doctor automatically. Optional
-speech and video packages are not installed. If installation is interrupted,
-rerun the same init command to finish setup.
-It does not copy VanillaSky's template tree. The important generated files are:
-
-| File | Your application owns |
-| --- | --- |
-| `src/main.tsx` | The mount point for the SDK-owned chat |
-| `server.ts` | Provider choices and callbacks |
-| `stock.ts` | Optional stock search policy |
-| `vite.config.ts` | Local UI and `/api/video-chat` endpoint |
-| `.env.local` | Ignored server-only credentials |
-
-The generated browser entry is intentionally tiny:
-
-```tsx
-import { StrictMode } from "react";
-import { createRoot } from "react-dom/client";
-import { VideoChat } from "@vanillaskyai/video/react";
-import "@vanillaskyai/video/video-chat.css";
-
-createRoot(document.getElementById("root")!).render(
-  <StrictMode><VideoChat /></StrictMode>,
-);
-```
-
-The SDK owns the responsive interface, conversation state, suggestions, voice
-input, narration pacing, streaming player, and packaged templates. Your shell
-stays responsible for providers, keys, authorization, limits, storage,
-branding, and product copy.
-
-## Add the one required key
-
-Add the text-provider key to the generated, ignored `.env.local`:
-
-```dotenv
-ANTHROPIC_API_KEY=
-```
-
-Fill the value locally; do not expose it through a client-prefixed environment
-variable. Then inspect the setup without calling any provider:
+Init creates an editable Vite app, installs its dependencies, and runs doctor.
+The default text adapter uses the optional Vercel AI SDK. Add `ANTHROPIC_API_KEY`
+to the generated, ignored `.env.local`, then:
 
 ```bash
 npx vanillasky doctor
+npm run dev
 ```
 
-The base experience reports `templates + browser voice`. `ANTHROPIC_API_KEY`
-is the only required key. Doctor checks setup locally without calling providers
-and reports names and readiness, never values.
+Open the reported localhost URL. Ask a question, let its spoken answer finish,
+then ask a follow-up. Doctor reports key names and readiness, never key values.
+Rerun init if installation was interrupted. It preserves keys and installed
+provider adapters, and refuses conflicting scaffold files rather than replacing
+your code.
 
-## Add optional capabilities
+For the native alternative, start a new directory with
+`npx @vanillaskyai/video init --native` and set `GEMINI_API_KEY` instead.
+Its editable Gemini REST callbacks need no `ai`, `@ai-sdk/anthropic`, or other
+model SDK. You can replace either text adapter with your own implementation.
+Rerunning ordinary init keeps the selected adapter.
 
-Install only the capability you want:
+## Application files
 
-```bash
-# Generated speech: installs the xAI adapter
-npx vanillasky providers add speech
+- `src/main.tsx` mounts `<VideoChat />` and imports the scoped stylesheet.
+- `server.ts` connects one `createVideoChatHandler` to app-owned providers.
+- `providers.ts` holds optional provider wiring.
+- `providers/text.ts` supplies streaming planning and small text tasks.
+- `vite.config.ts` serves the client and `/api/video-chat`.
+- `.env.local` holds server-only keys; never commit it.
 
-# Generated video and transcription: installs the FAL adapter
-npx vanillasky providers add video
-```
+The SDK owns the interface, shot planning, streaming, subtitles, voice timing,
+and footage/chapter renderers. It does not copy a template tree into your app.
 
-Add `XAI_API_KEY` for speech or `FAL_KEY` for video and transcription to
-`.env.local`, then restart the dev server. Stock media needs only
-`PEXELS_API_KEY`; no extra package or command is needed. The client stays the
-same. Rerun an interrupted provider command to finish its installation, then
-check readiness with `npx vanillasky doctor`.
+## Add footage and voice
 
-## Run and verify
+Add `PEXELS_API_KEY` for stock video without another package. For optional
+generated video or speech, run the corresponding setup command:
 
 ```bash
-npm run dev
+npx vanillasky providers add video fal
+npx vanillasky providers add speech
+npx vanillasky providers add transcription
+npx vanillasky doctor
 ```
 
-Open the reported localhost URL. Try one explanatory question and one unrelated
-creative request in the same conversation. Confirm each response starts with a
-spoken hook from the response stream before the full plan is complete, holds its opening until the first
-scene is ready, reaches its final frame, and leaves the composer ready for
-another turn. With stock media enabled, click a welcome or follow-up card and
-confirm its footage carries directly into that opening. With generated video
-enabled, confirm that moving shots develop a complete answer with narration
-and subtitles, and that the first shot continues the hook without repeating it.
-Check that no finished clip freezes while narration continues.
-
-The generated local authorization accepts localhost only. Replace it with your
-real session check, rate limits, and usage policy before deploying.
-
-## Continue
-
-- Change providers or add media capabilities in [Provider integration](provider-integration.md).
-- Brand or reshape the default experience in [Customization](customization.md).
-- Use a fully custom UI with the headless chat hook described in
-  [Provider integration](provider-integration.md#custom-interface).
-- Copy and edit a visual only when needed in [Custom templates](custom-templates.md).
-- Apply production authorization and key handling from
-  [Production](production.md) and [Security](security.md).
+Choose `fal`, `google`, `runway`, or `custom` after `video`; omitting the name
+selects fal. These install app-owned source, not core vendor dependencies.
+The fal and Runway references use `FAL_KEY` and `RUNWAY_API_KEY`; Google uses
+`GEMINI_API_KEY`. `custom` leaves a callback skeleton for any vendor.
+
+Before enabling generated video, configure `providers/video-delivery.ts` with
+your storage. Its example accepts your `VIDEO_UPLOAD_URL` and
+`VIDEO_STORAGE_TOKEN`; that upload service is not supplied by VanillaSky.
+See [provider integration](provider-integration.md#video-delivery-and-cancellation)
+for the callback contract. Provider credentials alone do not enable the
+starter's generated-video capability.
+
+Speech and transcription are independent upgrades. Speech installs the optional
+xAI/AI SDK adapter and uses `XAI_API_KEY`; Whisper transcription uses `FAL_KEY`
+without selecting fal for video. Doctor inspects the selected configuration
+locally; it never tests a paid generation.
+
+Configure keys locally and restart the server. Generated speech is
+optional; browser speech remains the default. Without usable footage, chapter
+scenes retain the narration and subtitles. See [provider integration](provider-integration.md)
+and [media and voice](media-and-audio.md) for callback and storage responsibilities.
+
+To use an existing assistant rather than ask the planner to answer directly,
+add [`resolveAnswer`](provider-integration.md#use-an-existing-assistant) to
+`server.ts`. The UI does not change.
+
+## Verify the result
+
+Use the real browser with normal motion enabled. Check one complete response
+and a follow-up, pause/mute, visible error recovery, and the console.
+If generated video is ready, verify that the footage actually moves and that
+the final spoken sentence completes. A passing mocked test does not prove
+provider latency or video quality.
+Google/Runway job APIs can take minutes; the SDK cannot turn that wait into
+real-time footage. Narration aims to finish 0.8 seconds before each clip ends;
+if a bounded rewrite cannot fit it, chapter recovery preserves the full speech.
+
+Before exposing the endpoint publicly, add application authentication, request
+limits, media policy, and spending controls. The local starter is not a
+production authorization policy. See [production](production.md) and
+[security](security.md).
+
+[Documentation home](../README.md)

package/docs/media-and-audio.md

@@ -1,4 +1,4 @@
-[← Documentation home](../README.md) · [Previous: Customization](customization.md) · [Next: Custom templates →](custom-templates.md)
+[← Documentation home](../README.md) · [Previous: Customization](customization.md) · [Next: Customization →](customization.md)
 
 # Media, voice, and audio
 
@@ -46,8 +46,7 @@ createVideoChatHandler({
 The planner emits a short semantic keyword, not a URL. The callback returns an
 application-approved image or video URL, and the SDK validates it before it
 reaches a scene. Return `null` when no licensed, safe, relevant asset exists;
-default chat displays an authored chapter while retaining the spoken answer. An explicit custom `templates` registry retains its structured planner,
-validation, and fallback contracts.
+default chat displays an authored chapter while retaining the spoken answer.
 
 For Pexels, keep `PEXELS_API_KEY` on the server, enforce a deadline, filter for
 suitable renditions, and return only validated Pexels asset domains. Licensing,
@@ -85,7 +84,7 @@ Pexels retains literal footage queries; these instructions cannot restyle stock
 assets. The opening chapter keeps its existing appearance. Automatic direction
 is generation-time guidance, not a new persisted style field; saved media keeps
 its rendered appearance and existing caller-supplied style persistence is
-unchanged. Custom template planning keeps its existing behavior.
+unchanged.
 
 ## Generated shots
 
@@ -135,8 +134,11 @@ owns the provider key, model, spend, generated bytes, retention, and delivery.
 Set `generatedClipDurationSec` to the duration your video adapter actually
 requests (default 5, supported range 2–20 seconds). Keep it aligned with provider
 settings and host spending limits. The planner fits natural spoken beats to that
-budget; measured audio determines the final scene timing. Silent footage may
-loop until the finite scene ends. Audible footage is not looped.
+budget with a 0.8-second tail. One bounded rewrite may shorten an oversized
+beat before generation; otherwise its complete narration stays on a chapter.
+Measured audio and actual decoded footage are checked before playback. Normal
+footage plays once at native speed; overruns recover to a chapter without
+cutting off narration or buying another clip.
 
 `generateVideoTimeoutMs` sets the first-shot preparation budget (default 15 seconds).
 Later deadlines account for their position in the answer rather than restarting
@@ -185,4 +187,4 @@ interaction before audible playback on many devices.
   while narration continues.
 - Return a safe fallback instead of leaving the response waiting forever.
 
-[← Documentation home](../README.md) · [Previous: Customization](customization.md) · [Next: Custom templates →](custom-templates.md)
+[← Documentation home](../README.md) · [Previous: Customization](customization.md) · [Next: Customization →](customization.md)

package/docs/performance.md

@@ -39,10 +39,15 @@ playback. It is not the time a speech request finishes. Custom voices must call
 `onStart()` when sound begins; otherwise this measurement remains unavailable.
 Muted responses do not produce a speech-start event.
 
-`stall` measures a wait after the player reaches the end of its available scenes
-while more scenes are being prepared. A completed interval includes `durationMs`.
-Deliberate pauses are excluded. This does not measure network buffering inside
-an image or video, because those assets do not control the scene clock.
+`stall` measures a completed wait interval with `durationMs` and a fixed reason:
+`scene-generation`, `speech`, or `media-decoding`. Deliberate pauses are excluded.
+Custom player owners must not pause voice for a `speech` wait.
+
+`scene-duration` compares prepared speech and clip seconds and records chapter
+recovery. `media-playback` reports the actual decoded clip duration, scene duration
+and observed repeat count. `buffer` samples ready buffered seconds across the
+active and next mounted media elements, not total downloaded bytes. The local
+development chat displays these measurements without retaining response content.
 
 Elapsed times start at prompt submission and can include user pauses. Compare
 unpaused runs with the same mode and voice configuration. Treat missing speech
@@ -75,6 +80,6 @@ chapter scenes. Neither callback measures the immediate opening template;
 measure that surface separately when checking submit-to-template latency.
 
 AI and Pexels modes remain separate. Missing, late, or unplayable footage uses
-the authored chapter and complete narration. Silent clips loop for the finite
-narrated scene. See [the local chat harness](development.md) for fixture timing
+the authored chapter and complete narration. Normal clips play once at native
+speed; overlong narration recovers to a chapter. See [the local chat harness](development.md) for fixture timing
 and explicit live-provider checks.

package/docs/persistence.md

@@ -1,4 +1,4 @@
-[← Documentation home](../README.md) · [Previous: Core concepts](concepts.md) · [Next: Streaming protocol →](streaming-protocol.md)
+[← Documentation home](../README.md) · [Previous: Core concepts](architecture.md) · [Next: Streaming protocol →](reference/protocol.md)
 
 # Persistence and replay
 
@@ -11,7 +11,7 @@ completed conversation history. The default `VideoChat` keeps history in memory;
 use the headless hook when the application needs durable storage.
 
 Every stored video has `schemaVersion: "0.2"`. This storage version is separate
-from streaming protocol `0.5`. The 0.1 policy supports the current storage
+from streaming protocol `0.6`. The SDK supports the current storage
 schema only: there are no compatibility aliases or implicit coercions.
 
 ## Load at the storage boundary
@@ -73,8 +73,4 @@ normal authenticated storage and a cryptographic integrity mechanism when
 those properties are required.
 
 Saved replay makes zero generation endpoint or model-provider requests. It is
-not necessarily zero network traffic: audio, images, videos, fonts, and
-customer-owned renderers may make separate media network requests.
-
-When a saved video uses project-owned templates, provide the matching browser
-registry: `<VideoPlayer video={savedVideo} templates={templates} />`.
+not necessarily zero network traffic: audio, images, videos, and fonts may make separate media network requests.

package/docs/production.md

@@ -17,7 +17,7 @@ Mount `<VideoChat />` or `useVideoChat` against that boundary.
 - Forward cancellation to every text, speech, media, transcription, and video provider.
 
 The handler rejects unknown templates and fields, invalid variables, unsafe
-media, and fabricated quote-template content before a scene reaches the
+media before a scene reaches the
 player. Read the [security guide](security.md) for the complete controls.
 
 ## Cinematic direction and providers
@@ -25,9 +25,7 @@ player. Read the [security guide](security.md) for the complete controls.
 Default chat shows an immediate chapter while speech and selected footage prepare.
 Configure `generateVideo` for AI mode and `searchMedia` for Pexels mode. Neither
 mode calls the other footage source. Missing or late footage uses the authored
-chapter with complete narration. Explicit custom template registries keep their
-existing composition and fallback contracts.
-A stock candidate must match the subject, action and permitted crop. Return
+chapter with complete narration. A stock candidate must match the subject, action and permitted crop. Return
 `null` for uncertainty rather than broadening an essential detail.
 
 Use explicit provider deadlines. Generated video should use idempotency keys
@@ -45,8 +43,7 @@ cards may carry a prepared opening so its speech starts without a model round tr
 
 Do not wait for the complete plan before showing the first validated scene.
 Preload upcoming assets and keep the current visual if the next one is late.
-When generated footage is useful, reserve the first shot in the opening object so generation
-can begin while the planner streams later scenes.
+The answer brief reserves the ending while body shots stream; do not wait for every clip before sending ready scenes.
 
 ## Data and privacy
 
@@ -90,15 +87,7 @@ small, explicitly gated real-provider smoke test before a release.
 
 In CI, build one clean consumer from the packed SDK artifact. This catches
 missing exports, server/browser boundary leaks, code-generation drift, and
-dependency-resolution problems that workspace tests miss. If the application
-owns copied templates, also run:
-
-```bash
-npx vanillasky templates sync --check
-npm run build
-npm test
-```
-
+dependency-resolution problems that workspace tests miss.
 ## Deployment checklist
 
 - [ ] Keys exist only in the server secret store.

package/docs/prompt-and-input.md

@@ -98,8 +98,7 @@ produces safe diagnostics. A media failure does not delete valid narration.
 
 AI mode generates footage within the host allowance. Pexels mode searches
 stock without calling the video generator. Each authored beat includes a short
-chapter title; missing footage becomes that chapter with its complete narration. Handlers configured with an explicit custom `templates` registry continue to
-support the trusted catalog and its existing structured planner contract.
+chapter title; missing footage becomes that chapter with its complete narration.
 
 ## Grounding