@vanillaskyai/video 0.7.1 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (89) hide show
  1. package/CHANGELOG.md +63 -0
  2. package/PUBLIC-API.md +83 -79
  3. package/README.md +20 -15
  4. package/dist/assets/vanillasky-logo.svg +151 -0
  5. package/dist/{builtin-server-KB7FKF6A.js → builtin-server-CUYPIDE7.js} +2 -2
  6. package/dist/check-runtime.js +1 -1
  7. package/dist/{chunk-2PYO6VAC.js → chunk-2DOLF4QN.js} +1 -43
  8. package/dist/{chunk-GAIOHGNR.js → chunk-2OKB27TG.js} +42 -45
  9. package/dist/{chunk-K746NUIP.js → chunk-2USVB3CY.js} +80 -29
  10. package/dist/{chunk-RE4IMWJR.js → chunk-3VRMKFM7.js} +87 -14
  11. package/dist/{chunk-JNN3EYVP.js → chunk-4UNKZEAT.js} +46 -46
  12. package/dist/{chunk-2XT4MZ76.js → chunk-APQKY6P3.js} +36 -3
  13. package/dist/{chunk-LLPMVDJZ.js → chunk-L7QGGKJ2.js} +69 -38
  14. package/dist/{chunk-RGF452LL.js → chunk-OWBT45EN.js} +1 -1
  15. package/dist/{chunk-LKBZX7GV.js → chunk-PP3MCPN4.js} +4 -4
  16. package/dist/chunk-ZD2VTUYR.js +28 -0
  17. package/dist/cli.js +131 -24
  18. package/dist/{compose-video-PD6LKKRK.js → compose-video-2NFH4DZY.js} +3 -3
  19. package/dist/{events-B6qS1Lsb.d.ts → events-BQ6z264n.d.ts} +1 -1
  20. package/dist/index.d.ts +5 -67
  21. package/dist/index.js +2 -4
  22. package/dist/{kit-8g46H2RZ.d.ts → kit-tQnCPMOM.d.ts} +1 -1
  23. package/dist/react.d.ts +33 -53
  24. package/dist/react.js +982 -648
  25. package/dist/server.d.ts +11 -13
  26. package/dist/server.js +170 -81
  27. package/dist/template-catalog.js +1 -1
  28. package/dist/templates.d.ts +3 -3
  29. package/dist/test.d.ts +2 -2
  30. package/dist/test.js +3 -3
  31. package/dist/{text-stream-ROTNGSD4.js → text-stream-54LXABJW.js} +1 -1
  32. package/dist/{types-2wHBqtg8.d.ts → types-CEh9eUFu.d.ts} +1 -1
  33. package/dist/{types-DrABlRa7.d.ts → types-DWFO6qca.d.ts} +1 -1
  34. package/docs/agent-integration.md +9 -3
  35. package/docs/architecture.md +8 -10
  36. package/docs/concepts.md +16 -40
  37. package/docs/custom-templates.md +27 -23
  38. package/docs/customization.md +23 -55
  39. package/docs/errors.md +58 -68
  40. package/docs/getting-started.md +21 -14
  41. package/docs/immersive-interface.md +87 -0
  42. package/docs/media-and-audio.md +2 -2
  43. package/docs/performance.md +69 -0
  44. package/docs/persistence.md +12 -27
  45. package/docs/production.md +2 -2
  46. package/docs/provider-integration.md +11 -5
  47. package/docs/reference/design-system.html +115 -0
  48. package/docs/reference/provider-adapters.md +37 -89
  49. package/docs/security.md +1 -1
  50. package/docs/streaming-protocol.md +2 -2
  51. package/examples/custom-template/README.md +6 -6
  52. package/package.json +6 -7
  53. package/registry/items/barChart.json +2 -2
  54. package/registry/items/bigNumber.json +2 -2
  55. package/registry/items/brandMessage.json +2 -2
  56. package/registry/items/cardList.json +2 -2
  57. package/registry/items/codeEditor.json +2 -2
  58. package/registry/items/ctaLogo.json +2 -2
  59. package/registry/items/ctaMedia.json +2 -2
  60. package/registry/items/incomingCall.json +2 -2
  61. package/registry/items/media.json +2 -2
  62. package/registry/items/milestone.json +2 -2
  63. package/registry/items/notification.json +2 -2
  64. package/registry/items/phoneMockup.json +2 -2
  65. package/registry/items/problemSolution.json +2 -2
  66. package/registry/items/progressRing.json +2 -2
  67. package/registry/items/promptInput.json +2 -2
  68. package/registry/items/reaction.json +2 -2
  69. package/registry/items/reviewStack.json +2 -2
  70. package/registry/items/steps.json +2 -2
  71. package/registry/items/terminal.json +2 -2
  72. package/registry/items/testimonial.json +2 -2
  73. package/registry/items/tripleStats.json +2 -2
  74. package/registry/items/tweet.json +2 -2
  75. package/registry/items/webMockup.json +2 -2
  76. package/starters/video-chat/.env.example +4 -2
  77. package/starters/video-chat/README.md +17 -5
  78. package/starters/video-chat/index.html +2 -1
  79. package/starters/video-chat/package.json +2 -4
  80. package/starters/video-chat/providers/speech.ts +20 -0
  81. package/starters/video-chat/providers/video.ts +66 -0
  82. package/starters/video-chat/providers.ts +3 -0
  83. package/starters/video-chat/server.ts +4 -80
  84. package/starters/video-chat/stock.ts +114 -99
  85. package/styles/video-chat.css +650 -246
  86. package/dist/chunk-SKRGRKHY.js +0 -142
  87. package/dist/state-DZcKuS32.d.ts +0 -3
  88. package/docs/branding-and-personalization.md +0 -87
  89. package/examples/custom-template/supplied-media.tsx +0 -92
@@ -24,11 +24,11 @@ validation, streaming, and player.
24
24
  | Location | Purpose |
25
25
  | --- | --- |
26
26
  | `src/server/create-video-chat-handler.ts` | Complete chat endpoint and capability boundary |
27
- | `src/server/create-video-handler.ts` | Lower-level video composition and provider adapter boundary |
27
+ | `src/server/create-video-handler.ts` | Internal composition and provider adapter boundary |
28
28
  | `src/server/prompts/` | System and user prompts sent to the app-owned model |
29
29
  | `src/server/model/` | Converts provider text deltas into typed video plan parts |
30
30
  | `src/protocol/` | Shared request, event, validation, checksum, and SSE contract |
31
- | `src/player/` | Browser stream client, `useVideo`, timeline, and React player |
31
+ | `src/player/` | Internal stream client, timeline, and React player |
32
32
  | `src/video-chat/` | Default `VideoChat` interface and headless conversation/session engine |
33
33
  | `src/visual-system/catalog/` | Template metadata, schemas, loading, and planner catalog |
34
34
  | `src/visual-system/scene-templates/` | Complete scenes the model may select |
@@ -36,7 +36,7 @@ validation, streaming, and player.
36
36
  | `src/visual-system/backgrounds/` | Standalone background renderers |
37
37
  | `src/visual-system/motion/` | Animation functions and timing behavior |
38
38
  | `src/visual-system/theme/` | Color and design tokens |
39
- | `src/cli/` | `vanillasky init` and `doctor`, plus `vanillasky templates create`, `add`, `sync`, `check`, `list`, and `describe` |
39
+ | `src/cli/` | `vanillasky init`, `doctor`, and `providers add`, plus `vanillasky templates create`, `add`, `sync`, `check`, `list`, and `describe` |
40
40
  | `registry/items/` | Generated distributable copies installed into customer projects |
41
41
  | `src/index.ts`, `src/server.ts`, `src/react.ts`, `src/templates.ts`, `src/template-catalog.ts`, `src/test.ts`, `styles/video-chat.css` | The six small code entry points and one scoped stylesheet |
42
42
 
@@ -61,13 +61,13 @@ prompt/validation registry in `vanillasky/server.ts`.
61
61
  3. `createVideoChatHandler(...)` owns chat actions, suggestions, narration,
62
62
  capability discovery, and generated-video budgets. The application supplies
63
63
  provider callbacks and policy.
64
- 4. Video answers flow into `createVideoHandler(...)`, which calls the
64
+ 4. Video answers flow into the internal composition pipeline, which calls the
65
65
  application's `streamText` adapter. This is where Anthropic, OpenAI, or
66
66
  another text model is connected.
67
67
  5. The system prompt combines the opening and composition rules with the trusted template
68
68
  catalog, including generated metadata for customer-owned templates. The user
69
- prompt serializes the factual input, instructions,
70
- personalization, brand, and approved media.
69
+ prompt serializes the prompt, completed conversation, instructions,
70
+ brand, and approved media.
71
71
  6. The model streams one host-consumed opening object followed by NDJSON plan
72
72
  parts. The server emits the opening event, then parses and validates complete
73
73
  scenes before emitting them.
@@ -75,8 +75,7 @@ prompt/validation registry in `vanillasky/server.ts`.
75
75
  loops or holds until its voice ends and the first playable scene is ready;
76
76
  the planned playback and narration then take over in one cut.
77
77
 
78
- The lower-level `useVideo` and `VideoPlayer` path skips chat actions and remains
79
- available for explicit one-shot video response integrations.
78
+ `VideoPlayer` also replays completed chat videos parsed at the storage boundary.
80
79
 
81
80
  ## Where to change common behavior
82
81
 
@@ -93,7 +92,6 @@ available for explicit one-shot video response integrations.
93
92
 
94
93
  ## Public vocabulary
95
94
 
96
- Use `VideoChat`, `useVideoChat`, `createVideoChatHandler`, `Video`, `VideoInput`,
97
- `useVideo`, and `VideoPlayer` for the product API. Use “video response” when
95
+ Use `VideoChat`, `useVideoChat`, `createVideoChatHandler`, `Video`, and `VideoPlayer` for the product API. Use “video response” when
98
96
  describing the lifecycle or output category. Use “motion” only for animation
99
97
  behavior inside the visual system.
package/docs/concepts.md CHANGED
@@ -29,25 +29,13 @@ or export pipeline when an encoded file is required.
29
29
 
30
30
  ## Input
31
31
 
32
- `VideoInput` is the factual and creative boundary:
33
-
34
- - `input`: raw source material such as a roadmap update, launch brief, article,
35
- curated set of articles, metrics, events, notes, or an AI answer. It may be
36
- short, but the planner is designed to distill larger sources into a concise
37
- video rather than represent every fact;
38
- - `knowledgeMode`: `input-only` by default, or `general` when stable model
39
- knowledge may supplement the request;
40
- - `instructions`: optional creative direction that cannot override facts;
41
- - `opening`: optional custom copy for the deterministic opening; omission uses
42
- `Creating your video...`, while `false` lets the host render loading UI
43
- without adding an opening scene to the video;
44
- - `personalization`: application-defined fields such as name, role, account,
45
- period, goal, or onboarding partner;
46
- - `brand`: an optional background preset plus name, logo, font, surfaces, and
47
- advanced exact tokens;
48
- - `suppliedMedia`: approved images or videos with a role and description;
49
- - `audio`: omit for automatic selection, pass `{ src }`, or use `false` for silence;
50
- - `orientation` and `maxDurationSec`: composition constraints.
32
+ The viewer sends a prompt with bounded completed conversation turns. The server
33
+ adds trusted application `instructions`, template capabilities, and the selected
34
+ visual mode. `VideoChat` options control brand, style, orientation, and custom
35
+ templates. Exact facts belong in the authorized prompt or conversation; secrets
36
+ and provider configuration stay on the server.
37
+
38
+ See [Prompt and conversation input](prompt-and-input.md) for the request contract.
51
39
 
52
40
  ## Template kit
53
41
 
@@ -72,20 +60,10 @@ another streaming text model.
72
60
  The planner does not create public event IDs, checksums, or final
73
61
  snapshots. The runtime owns those guarantees.
74
62
 
75
- `createVideoHandler` asks the planner to emit one early
76
- `scene.add` with `placement: "closer"` after the first playable body scene.
77
- That placement is planner-only: the runtime holds the validated closer,
78
- reserves its readable duration while later body scenes stream, and appends it
79
- last without persisting `placement` into the completed `Video`.
80
-
81
- | Boundary | Type | Owner |
82
- | --- | --- | --- |
83
- | Planner parts | Internal validated plan data | Server/provider adapter |
84
- | Values read from `response.stream` | `VideoEvent` | SDK runtime |
85
- | Terminal editable result | `Video` via `response.result` | SDK reducer |
86
-
87
- A planner must never yield `VideoEvent` envelopes. It yields plan
88
- parts; the runtime validates them and creates the public event metadata.
63
+ The internal composition pipeline can reserve a validated closing scene and
64
+ append it after the body. Planner parts are internal data; applications connect
65
+ text callbacks to `createVideoChatHandler` and let the SDK own event envelopes,
66
+ sequence IDs, and final snapshots.
89
67
 
90
68
  ## Event stream
91
69
 
@@ -93,21 +71,19 @@ The runtime emits ordered protocol events:
93
71
 
94
72
  ```text
95
73
  response.start
96
- audio.set optional, before any scene
97
- scene.add supplied opening
74
+ data.video-chat-opening short spoken hook
98
75
  scene.add generated body
99
76
  scene.add generated body
100
77
  scene.add reserved closer
101
78
  response.complete exact terminal snapshot
102
79
  ```
103
80
 
104
- Network transport uses SSE. Direct in-process integrations expose the same
105
- events as an async iterable.
81
+ Network transport uses SSE. Test utilities can simulate protocol events in process.
106
82
 
107
83
  ## Playback and buffering
108
84
 
109
85
  The player renders the first committed scene and continues through the known
110
- timeline. Audio can loop during a generation gap. A media-bearing scene should
86
+ timeline. The chat holds its current visual during a generation gap. A media-bearing scene should
111
87
  not be committed until its asset is ready; start generated content with a
112
88
  typography-led scene so useful playback does not wait on media lookup.
113
89
 
@@ -124,8 +100,8 @@ complete storage contract.
124
100
  ## Completion snapshot
125
101
 
126
102
  `response.complete` contains the exact reduced `Video` plus a
127
- checksum. Persist the snapshot when you need replay, resume, export, analytics,
128
- or an editing handoff. The event log can also be persisted and revalidated.
103
+ checksum. The chat retains the completed video on its turn for application-owned replay
104
+ and storage. A partial playable answer may complete with non-fatal warnings.
129
105
 
130
106
  Read the complete [Protocol 0.5](reference/protocol.md) and the separate
131
107
  [persisted Video 0.1 contract](persistence.md).
@@ -1,4 +1,4 @@
1
- [← Documentation home](../README.md) · [Previous: Branding and personalization](branding-and-personalization.md) · [Next: Motion and effects →](motion-and-effects.md)
1
+ [← Documentation home](../README.md) · [Previous: Customization](customization.md) · [Next: Motion and effects →](motion-and-effects.md)
2
2
 
3
3
  # Custom templates
4
4
 
@@ -190,7 +190,6 @@ Start from the generated file or one of the packaged references:
190
190
  | --- | --- |
191
191
  | One headline and supporting line | [Minimal text](../examples/custom-template/minimal-text.tsx) |
192
192
  | One exact metric and its change | [Structured data](../examples/custom-template/structured-data.tsx) |
193
- | An application-supplied image | [Supplied media](../examples/custom-template/supplied-media.tsx) |
194
193
 
195
194
  The [reference comparison](../examples/custom-template/README.md) explains when
196
195
  to choose each. Copy a file into `vanillasky/templates/`, change its ID and
@@ -213,18 +212,27 @@ selection, and validator. Import it in the route that connects your model:
213
212
 
214
213
  ```ts
215
214
  // src/video-route.ts
216
- import { streamText } from "ai";
217
- import { createVideoHandler } from "@vanillaskyai/video/server";
215
+ import { generateText, streamText } from "ai";
216
+ import { createVideoChatHandler } from "@vanillaskyai/video/server";
218
217
  import { videoModel } from "./video-model";
219
218
  import { templates } from "../vanillasky/server";
220
219
 
221
- export const handleVideo = createVideoHandler({
220
+ export const handleVideoChat = createVideoChatHandler({
222
221
  templates,
223
222
  authorize: (request) => {
224
223
  if (process.env.VANILLASKY_LOCAL_DEMO !== "1") return false;
225
224
  const hostname = new URL(request.url).hostname;
226
225
  return hostname === "localhost" || hostname === "127.0.0.1";
227
226
  },
227
+ generateText: async ({ systemPrompt, userPrompt, signal }) => {
228
+ const result = await generateText({
229
+ model: videoModel,
230
+ system: systemPrompt,
231
+ prompt: userPrompt,
232
+ abortSignal: signal,
233
+ });
234
+ return result.text;
235
+ },
228
236
  streamText: ({ systemPrompt, userPrompt, signal }) => streamText({
229
237
  model: videoModel,
230
238
  system: systemPrompt,
@@ -242,19 +250,13 @@ check before deployment, as shown in
242
250
  Use the browser registry for generation and playback:
243
251
 
244
252
  ```tsx
245
- // src/video-composer.tsx
246
- import { VideoPlayer, useVideo } from "@vanillaskyai/video/react";
253
+ // src/video-chat.tsx
254
+ import { VideoChat } from "@vanillaskyai/video/react";
255
+ import "@vanillaskyai/video/video-chat.css";
247
256
  import { templates } from "../vanillasky";
248
257
 
249
- export function VideoComposer() {
250
- const video = useVideo({ templates });
251
-
252
- return <>
253
- <button onClick={() => void video.generate({ input: "Grounded answer" })}>
254
- Generate
255
- </button>
256
- <VideoPlayer {...video.playerProps} />
257
- </>;
258
+ export function App() {
259
+ return <VideoChat options={{ templates }} />;
258
260
  }
259
261
  ```
260
262
 
@@ -311,8 +313,7 @@ export function TemplatePreview() {
311
313
 
312
314
  Change the saved variables to each named example, inspect both orientations,
313
315
  and play the scene to inspect its full progress range. This is also the
314
- production replay path for a completed video saved from
315
- `await video.generate(...)`.
316
+ production replay path for a completed chat turn saved from `chat.turns`.
316
317
 
317
318
  ## Schema and grounding
318
319
 
@@ -330,14 +331,17 @@ Useful formats add grounding behavior:
330
331
  - `grounded-stat` marks numeric statistical evidence for the planner; it does
331
332
  not compare the value against raw input at runtime;
332
333
  - `grounded-quote` requires the quote to exist verbatim in the input;
333
- - `supplied-image` requires an image URL listed in `VideoInput.suppliedMedia`;
334
- - `uri` applies the supplied-media or server URL policy to an approved URL;
334
+ - `uri` validates approved media URLs restored by the server;
335
335
  - `stock-media-keyword` is only for hosts that resolve stock media before a
336
336
  scene is committed.
337
337
 
338
338
  Templates whose core proof needs a real statistic can add
339
- `"x-vanillasky": { "requiresStat": true }`. See the structured-data and
340
- supplied-media references for complete examples.
339
+ `"x-vanillasky": { "requiresStat": true }`. See the structured-data reference for a complete example.
340
+
341
+ For media-backed chat scenes, copy the built-in `media` template and keep its
342
+ `mediaKeyword`, `mediaUrl`, and `mediaType` contract. Configure `searchMedia` on
343
+ the chat handler so the server resolves semantic searches into approved assets.
344
+ The chat request does not accept a separate supplied-media input list.
341
345
 
342
346
  The automatic `opening` uses the built-in `media` variables (`texts` and
343
347
  `mediaType: "gradient"`). If you replace `media` while using automatic
@@ -353,5 +357,5 @@ trusted template and fills its declared schema; it never writes executable UI
353
357
  code.
354
358
 
355
359
  For the exact request path and source locations, see
356
- [Architecture](architecture.md). For media policies and supplied-media input,
360
+ [Architecture](architecture.md). For media policies and provider callbacks,
357
361
  see [Media and audio](media-and-audio.md).
@@ -4,9 +4,11 @@
4
4
 
5
5
  ## Video chat interface
6
6
 
7
- The default `VideoChat` interface is already responsive and accessible. Pass a
8
- custom welcome heading and a root class when the application needs its own copy
9
- or chrome colors:
7
+ The default `VideoChat` uses an immersive video canvas with a floating
8
+ conversation field, on-video subtitles, and a single dark settings/history
9
+ treatment. See [interface behavior](immersive-interface.md) and the
10
+ [component reference](reference/design-system.html). Pass a custom welcome
11
+ heading and a root class when the application needs its own copy or chrome colors:
10
12
 
11
13
  ```tsx
12
14
  <VideoChat
@@ -22,21 +24,25 @@ that instance:
22
24
 
23
25
  ```css
24
26
  .acme-chat {
25
- --vs-accent: #0057ff;
26
- --vs-accent-deep: #003fb8;
27
+ --vs-media-glass: rgb(18 24 40 / 80%);
28
+ --vs-media-text: #f8f8fc;
27
29
  --vs-voice: #e11d74;
28
30
  --vs-font: "Inter", sans-serif;
29
- --vs-radius-lg: 1rem;
30
31
  }
31
32
  ```
32
33
 
34
+ The built-in navigation carries the VanillaSky logo. `welcomeTitle` changes
35
+ the welcome heading; it does not replace the navigation logo. `options.brand`
36
+ styles generated video content independently of the surrounding controls.
37
+
33
38
  Use `options` for the endpoint, templates, orientation, visual brand, request
34
39
  headers, and an optional custom voice. Provider capabilities are discovered
35
40
  from the server. Use `useVideoChat()` only when the application needs to own the
36
41
  entire interface.
37
42
 
38
- The sections below customize lower-level, one-shot video composition. They are
39
- not required for the default chat.
43
+ Pass the following visual settings through `VideoChat` or `useVideoChat` options.
44
+ Keep viewer context in the prompt and completed conversation turns; use the
45
+ server handler’s `instructions` for trusted product guidance.
40
46
 
41
47
  ## Background and semantic brand
42
48
 
@@ -96,35 +102,17 @@ These are defaults, not generated CSS. A validated scene may still select a
96
102
  more appropriate text or background treatment when its trusted template allows
97
103
  it.
98
104
 
99
- ## Personalization
100
-
101
- `personalization` accepts JSON-safe application fields. Include only values the
102
- viewer is allowed to see. The system prompt treats them as context, not as
103
- instructions.
104
-
105
- Good fields include `firstName`, `role`, `accountName`, `period`, `goal`,
106
- `locale`, and `onboardingPartner`. Keep facts in `input` as well when they must
107
- appear in the story.
108
-
109
105
  ## Opening
110
106
 
111
- `VideoPlayer` automatically shows a brand-colored generation cover until the
112
- first validated scene arrives. The cover is player state, not video content: it
113
- is never written to the event log, replay, or export.
114
-
115
- The opening is deterministic and should not wait for an LLM or remote media.
116
- Omit it to use `Creating your video...`, or supply one concise custom sentence:
107
+ The planner streams a short spoken hook before the scenes. The chat holds that
108
+ opening until its speech finishes and the first scene is ready. A selected
109
+ suggestion can start with its prewritten opening and already-loaded media:
117
110
 
118
111
  ```ts
119
- opening: "Your Q2 customer impact recap is ready."
112
+ await chat.ask(card.prompt, { opening: card.opening, openingMedia: card.media });
120
113
  ```
121
114
 
122
- Pass `opening: false` to omit the persisted opening and show application-owned
123
- loading UI until the first generated scene arrives.
124
-
125
- Use `opening` only for a genuine opening that should remain in the completed
126
- response. VanillaSky infers the scene ID, `media` template, gradient variables,
127
- and three-second timing. Keep generic loading state in the host UI instead.
115
+ Openings and scene narration share the chat voice and pause/mute controls.
128
116
 
129
117
  ## Aspect ratio and responsive layout
130
118
 
@@ -138,31 +126,11 @@ should display landscape on desktop and portrait on mobile without changing the
138
126
  saved response, pass `orientation="auto"` to `VideoPlayer`; it responds
139
127
  to its container width. See [responsive orientation](responsive-orientation.md).
140
128
 
141
- ## Supplied media
142
-
143
- Provide approved media with semantic descriptions:
144
-
145
- ```ts
146
- suppliedMedia: [{
147
- id: "product-dashboard",
148
- url: "https://cdn.example.com/dashboard.png",
149
- type: "image",
150
- description: "Activation dashboard after the Q2 release",
151
- role: "product",
152
- }]
153
- ```
154
-
155
- Resolve searched media before generation and pass the approved result through
156
- `suppliedMedia`. See [media providers](media-and-audio.md#media-providers).
157
-
158
- ## Soundtrack audio
129
+ ## Media and voice
159
130
 
160
- Pass `audio: { src }` for a specific track, omit `audio` to let the server choose
161
- synchronously from a preloaded catalog, or pass `audio: false` for silence.
162
- VanillaSky infers deterministic duration, volume, beat, and fade-out metadata.
163
- The soundtrack should continue across visual generation gaps and finish with
164
- the final scene. `VideoChat` separately owns narration, TTS integration, browser
165
- voice fallback, and speech synchronization for conversational responses.
131
+ Configure `searchMedia`, `generateVideo`, and `generateSpeech` on the server
132
+ handler. They progressively enhance the same chat; failed optional providers
133
+ fall back to templates or browser voice. See [Media and voice](media-and-audio.md).
166
134
 
167
135
  ## Custom templates
168
136
 
package/docs/errors.md CHANGED
@@ -2,71 +2,61 @@
2
2
 
3
3
  # Errors and recovery
4
4
 
5
- Render errors in normal application UI outside the video. Never show stack traces, provider details, raw deltas, or protocol diagnostics inside a scene.
6
-
7
- | Boundary | Example codes | Host response |
8
- | --- | --- | --- |
9
- | HTTP | `unauthorized`, `origin_forbidden`, `body_too_large` | Fix session/policy; do not retry blindly |
10
- | Input | `invalid_json`, `invalid_request`, `secret_field` | Correct client request |
11
- | Protocol | invalid sequence/checksum/scene | Stop the stream and record a safe diagnostic |
12
- | Generation | `generation_failed` | Hold current scene, end cleanly, offer retry |
13
- | Generated part | `invalid_generated_part` | Keep accepted scenes; inspect the private server log |
14
- | Resume | `invalid_cursor`, `resume_unavailable` | Restart only before visible output or from valid storage |
15
-
16
- ```tsx
17
- import { VideoError } from "@vanillaskyai/video/react";
18
-
19
- try {
20
- const completed = await video.generate({ input });
21
- await saveVideo(completed);
22
- } catch (error) {
23
- if (error instanceof VideoError) {
24
- logSafeFailure({
25
- code: error.code,
26
- status: error.status,
27
- requestId: error.requestId,
28
- runId: error.runId,
29
- });
30
- }
31
- }
32
- ```
33
-
34
- `generate()` resolves only for `response.complete`. A terminal
35
- `response.error` or `response.abort` rejects it. The hook still retains the
36
- latest validated `video.video`, and its player stream still receives terminal
37
- events, so an application may keep already accepted scenes visible while it
38
- offers a retry. If the planner already supplied a validated reserved closer,
39
- the runtime appends it to that playable terminal snapshot before reporting a
40
- late provider failure; it never invents or rewrites closer copy during recovery.
41
-
42
- Both a server `response.abort` and an explicit `video.abort(reason)` keep
43
- `video.status` at `aborted`; `video.error` contains the same safe typed abort
44
- reported by the rejected `generate()` promise.
45
-
46
- `VideoError` preserves only actionable public context: the server's safe
47
- `code` and `message`, HTTP `status` when available, `requestId`, `runId`, and
48
- `recoverable`. Do not replace those fields with raw provider errors.
49
-
50
- Abort replaced compositions and client disconnects. Retry provider throttles or transient failures only within an explicit request/time budget and only before visible output unless you have validated resume storage. Never silently restart after the viewer has begun watching.
51
-
52
- Log safe codes with request/run IDs and latency. Keep private provider messages in server logs through `onError`; redact source, personalization, keys, tokens, and signed URLs.
53
-
54
- Live acceptance artifacts use stable failure categories: `network`,
55
- `authentication`, `model_not_found`, `rate_limit`, `provider`, `planner_parse`,
56
- `scene_validation`, or `unknown`. Before diagnosing the SDK, probe the provider
57
- with one minimal request using the same server-only credential and model. Check
58
- DNS/proxy access, credential validity, model availability, account/rate limits,
59
- and timeout settings in that order. Never copy raw provider payloads, streamed
60
- deltas, prompts, or credentials into browser errors or CI artifacts.
61
-
62
- Server handlers drop invalid generated parts by default. Each rejection calls
63
- `onError` with the full internal reason, while the browser receives only the
64
- recoverable `invalid_generated_part` diagnostic. Set `invalidPartBehavior: "fail"`
65
- only when fail-fast generation is an intentional compatibility requirement.
66
-
67
- `onComplete` runs exactly once only after `response.complete`. Terminal errors,
68
- client disconnects, explicit aborts, and host timeouts do not call it;
69
- `onError` receives internal failures, while cancellation remains a safe
70
- `response.abort`. Abort and timeout signals always propagate to `streamText`.
71
- The host owns timeout construction and retry policy. Do not retry invisibly
72
- after any scene has reached the viewer.
5
+ `VideoChat` preserves the opening and accepted scenes when an optional scene,
6
+ provider, or stream fails. It recovers silently and keeps the answer playable. A fatal error is shown only when no playable response can
7
+ be produced. Never put provider details or stack traces in the interface.
8
+
9
+ ## Recovery by boundary
10
+
11
+ | Failure | Result |
12
+ | --- | --- |
13
+ | Invalid planner scene | Skip the part and accept later valid scenes |
14
+ | Generated footage | Try stock media, matching completed footage, then a safe template |
15
+ | Stock lookup or candidate | Continue with another candidate or template |
16
+ | Template renderer | Isolate the failed scene with a safe visual |
17
+ | Missing narration | Continue with scene text |
18
+ | Generated speech | Use browser voice |
19
+ | Late stream failure | Keep playable opening and completed scenes |
20
+ | Unauthorized or invalid request without playable output | Show a safe error |
21
+
22
+ Keep non-fatal `chat.warnings` as application diagnostics; they are also
23
+ retained on the corresponding `VideoChatTurn.warnings`. Render `chat.error`
24
+ when the response cannot play, and keep `chat.playerProps` mounted during recovery.
25
+ Check `turn.completed` before persisting it as completed conversation context;
26
+ a cancelled or partial visible turn is not automatically a completed turn.
27
+
28
+ `VideoError` exposes actionable public fields: `code`, `message`, optional HTTP
29
+ `status`, `requestId`, `runId`, and `recoverable`. Log only safe codes and IDs
30
+ from the client. The server's `onError` observer receives internal diagnostics;
31
+ redact credentials, source data, provider payloads, and signed URLs before logging.
32
+ Observer failures are isolated from response generation.
33
+
34
+ ## Cancellation and retries
35
+
36
+ `chat.cancel()` cancels the current work while preserving available output.
37
+ Replacing a prompt aborts its old providers and speech. The host must forward
38
+ `signal` to every provider callback and own deadlines, quotas, and retry budgets.
39
+ The chat retries once only before playback. Never silently restart generation
40
+ after the viewer has begun watching, and do not replay paid generation requests
41
+ without a deliberate idempotency and spend policy.
42
+
43
+ The server drops invalid generated parts by default. `invalidPartBehavior: "fail"`
44
+ is an explicit strict policy; ordinary chat should retain the resilient default.
45
+ `onComplete` runs after a `response.complete`, including a recovered playable
46
+ response. Fatal errors, disconnects, and explicit aborts do not call it.
47
+
48
+ Automated regression and acceptance tests use mocked providers. See
49
+ [Testing](testing.md) and the [chat acceptance gate](https://github.com/VanillaSkyAi/video/blob/main/docs/maintainers/acceptance.md).
50
+
51
+ ## Slow optional providers
52
+
53
+ Generated video has a 15-second deadline per lookup. Stock media, generated
54
+ speech preparation, and fallback scene narration have 3-second deadlines.
55
+ A deadline uses the same safe fallback as a failed provider; completed scenes
56
+ and scene order are preserved. Providers receive cancellation, and late results
57
+ are ignored even when a provider does not cooperate. These initial limits bound
58
+ waiting; they are not claims about measured live-provider performance.
59
+
60
+ A completed turn and its saved video are available before follow-up suggestions.
61
+ Suggestions load separately and may be omitted after a short deadline. Starting
62
+ another turn or cancelling discards outstanding suggestions.
@@ -4,7 +4,8 @@
4
4
 
5
5
  The fastest VanillaSky integration is the complete, general-purpose video chat.
6
6
  It starts with packaged templates and browser voice, then turns on optional
7
- speech, stock, transcription, and generated video when their server keys exist.
7
+ speech and generated video when you install their adapters and add server keys.
8
+ Stock media needs only its server key.
8
9
 
9
10
  ## Create the app
10
11
 
@@ -15,7 +16,9 @@ npx @vanillaskyai/video init
15
16
  ```
16
17
 
17
18
  Init installs the exact SDK version that ran it, creates a small application
18
- shell, and installs its provider dependencies.
19
+ shell, installs baseline dependencies, and runs doctor automatically. Optional
20
+ speech and video packages are not installed. If installation is interrupted,
21
+ rerun the same init command to finish setup.
19
22
  It does not copy VanillaSky's template tree. The important generated files are:
20
23
 
21
24
  | File | Your application owns |
@@ -59,23 +62,27 @@ variable. Then inspect the setup without calling any provider:
59
62
  npx vanillasky doctor
60
63
  ```
61
64
 
62
- The base experience reports `templates + browser voice`. A ready text key makes
63
- the chat answer. Optional keys progressively add capabilities:
65
+ The base experience reports `templates + browser voice`. `ANTHROPIC_API_KEY`
66
+ is the only required key. Doctor checks setup locally without calling providers
67
+ and reports names and readiness, never values.
64
68
 
65
- ```dotenv
66
- # Optional generated speech
67
- XAI_API_KEY=
69
+ ## Add optional capabilities
70
+
71
+ Install only the capability you want:
68
72
 
69
- # Optional generated video and voice transcription
70
- FAL_KEY=
73
+ ```bash
74
+ # Generated speech: installs the xAI adapter
75
+ npx vanillasky providers add speech
71
76
 
72
- # Optional stock media
73
- PEXELS_API_KEY=
77
+ # Generated video and transcription: installs the FAL adapter
78
+ npx vanillasky providers add video
74
79
  ```
75
80
 
76
- Doctor reports only key names and readiness, never values. Adding or removing
77
- an optional key changes the available modes after a server restart; the client
78
- does not need to change.
81
+ Add `XAI_API_KEY` for speech or `FAL_KEY` for video and transcription to
82
+ `.env.local`, then restart the dev server. Stock media needs only
83
+ `PEXELS_API_KEY`; no extra package or command is needed. The client stays the
84
+ same. Rerun an interrupted provider command to finish its installation, then
85
+ check readiness with `npx vanillasky doctor`.
79
86
 
80
87
  ## Run and verify
81
88