@vanillaskyai/video 0.7.1 → 0.8.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +63 -0
- package/PUBLIC-API.md +83 -79
- package/README.md +20 -15
- package/dist/assets/vanillasky-logo.svg +151 -0
- package/dist/{builtin-server-KB7FKF6A.js → builtin-server-CUYPIDE7.js} +2 -2
- package/dist/check-runtime.js +1 -1
- package/dist/{chunk-2PYO6VAC.js → chunk-2DOLF4QN.js} +1 -43
- package/dist/{chunk-GAIOHGNR.js → chunk-2OKB27TG.js} +42 -45
- package/dist/{chunk-K746NUIP.js → chunk-2USVB3CY.js} +80 -29
- package/dist/{chunk-RE4IMWJR.js → chunk-3VRMKFM7.js} +87 -14
- package/dist/{chunk-JNN3EYVP.js → chunk-4UNKZEAT.js} +46 -46
- package/dist/{chunk-2XT4MZ76.js → chunk-APQKY6P3.js} +36 -3
- package/dist/{chunk-LLPMVDJZ.js → chunk-L7QGGKJ2.js} +69 -38
- package/dist/{chunk-RGF452LL.js → chunk-OWBT45EN.js} +1 -1
- package/dist/{chunk-LKBZX7GV.js → chunk-PP3MCPN4.js} +4 -4
- package/dist/chunk-ZD2VTUYR.js +28 -0
- package/dist/cli.js +131 -24
- package/dist/{compose-video-PD6LKKRK.js → compose-video-2NFH4DZY.js} +3 -3
- package/dist/{events-B6qS1Lsb.d.ts → events-BQ6z264n.d.ts} +1 -1
- package/dist/index.d.ts +5 -67
- package/dist/index.js +2 -4
- package/dist/{kit-8g46H2RZ.d.ts → kit-tQnCPMOM.d.ts} +1 -1
- package/dist/react.d.ts +33 -53
- package/dist/react.js +982 -648
- package/dist/server.d.ts +11 -13
- package/dist/server.js +170 -81
- package/dist/template-catalog.js +1 -1
- package/dist/templates.d.ts +3 -3
- package/dist/test.d.ts +2 -2
- package/dist/test.js +3 -3
- package/dist/{text-stream-ROTNGSD4.js → text-stream-54LXABJW.js} +1 -1
- package/dist/{types-2wHBqtg8.d.ts → types-CEh9eUFu.d.ts} +1 -1
- package/dist/{types-DrABlRa7.d.ts → types-DWFO6qca.d.ts} +1 -1
- package/docs/agent-integration.md +9 -3
- package/docs/architecture.md +8 -10
- package/docs/concepts.md +16 -40
- package/docs/custom-templates.md +27 -23
- package/docs/customization.md +23 -55
- package/docs/errors.md +58 -68
- package/docs/getting-started.md +21 -14
- package/docs/immersive-interface.md +87 -0
- package/docs/media-and-audio.md +2 -2
- package/docs/performance.md +69 -0
- package/docs/persistence.md +12 -27
- package/docs/production.md +2 -2
- package/docs/provider-integration.md +11 -5
- package/docs/reference/design-system.html +115 -0
- package/docs/reference/provider-adapters.md +37 -89
- package/docs/security.md +1 -1
- package/docs/streaming-protocol.md +2 -2
- package/examples/custom-template/README.md +6 -6
- package/package.json +6 -7
- package/registry/items/barChart.json +2 -2
- package/registry/items/bigNumber.json +2 -2
- package/registry/items/brandMessage.json +2 -2
- package/registry/items/cardList.json +2 -2
- package/registry/items/codeEditor.json +2 -2
- package/registry/items/ctaLogo.json +2 -2
- package/registry/items/ctaMedia.json +2 -2
- package/registry/items/incomingCall.json +2 -2
- package/registry/items/media.json +2 -2
- package/registry/items/milestone.json +2 -2
- package/registry/items/notification.json +2 -2
- package/registry/items/phoneMockup.json +2 -2
- package/registry/items/problemSolution.json +2 -2
- package/registry/items/progressRing.json +2 -2
- package/registry/items/promptInput.json +2 -2
- package/registry/items/reaction.json +2 -2
- package/registry/items/reviewStack.json +2 -2
- package/registry/items/steps.json +2 -2
- package/registry/items/terminal.json +2 -2
- package/registry/items/testimonial.json +2 -2
- package/registry/items/tripleStats.json +2 -2
- package/registry/items/tweet.json +2 -2
- package/registry/items/webMockup.json +2 -2
- package/starters/video-chat/.env.example +4 -2
- package/starters/video-chat/README.md +17 -5
- package/starters/video-chat/index.html +2 -1
- package/starters/video-chat/package.json +2 -4
- package/starters/video-chat/providers/speech.ts +20 -0
- package/starters/video-chat/providers/video.ts +66 -0
- package/starters/video-chat/providers.ts +3 -0
- package/starters/video-chat/server.ts +4 -80
- package/starters/video-chat/stock.ts +114 -99
- package/styles/video-chat.css +650 -246
- package/dist/chunk-SKRGRKHY.js +0 -142
- package/dist/state-DZcKuS32.d.ts +0 -3
- package/docs/branding-and-personalization.md +0 -87
- package/examples/custom-template/supplied-media.tsx +0 -92
package/docs/architecture.md
CHANGED
|
@@ -24,11 +24,11 @@ validation, streaming, and player.
|
|
|
24
24
|
| Location | Purpose |
|
|
25
25
|
| --- | --- |
|
|
26
26
|
| `src/server/create-video-chat-handler.ts` | Complete chat endpoint and capability boundary |
|
|
27
|
-
| `src/server/create-video-handler.ts` |
|
|
27
|
+
| `src/server/create-video-handler.ts` | Internal composition and provider adapter boundary |
|
|
28
28
|
| `src/server/prompts/` | System and user prompts sent to the app-owned model |
|
|
29
29
|
| `src/server/model/` | Converts provider text deltas into typed video plan parts |
|
|
30
30
|
| `src/protocol/` | Shared request, event, validation, checksum, and SSE contract |
|
|
31
|
-
| `src/player/` |
|
|
31
|
+
| `src/player/` | Internal stream client, timeline, and React player |
|
|
32
32
|
| `src/video-chat/` | Default `VideoChat` interface and headless conversation/session engine |
|
|
33
33
|
| `src/visual-system/catalog/` | Template metadata, schemas, loading, and planner catalog |
|
|
34
34
|
| `src/visual-system/scene-templates/` | Complete scenes the model may select |
|
|
@@ -36,7 +36,7 @@ validation, streaming, and player.
|
|
|
36
36
|
| `src/visual-system/backgrounds/` | Standalone background renderers |
|
|
37
37
|
| `src/visual-system/motion/` | Animation functions and timing behavior |
|
|
38
38
|
| `src/visual-system/theme/` | Color and design tokens |
|
|
39
|
-
| `src/cli/` | `vanillasky init` and `
|
|
39
|
+
| `src/cli/` | `vanillasky init`, `doctor`, and `providers add`, plus `vanillasky templates create`, `add`, `sync`, `check`, `list`, and `describe` |
|
|
40
40
|
| `registry/items/` | Generated distributable copies installed into customer projects |
|
|
41
41
|
| `src/index.ts`, `src/server.ts`, `src/react.ts`, `src/templates.ts`, `src/template-catalog.ts`, `src/test.ts`, `styles/video-chat.css` | The six small code entry points and one scoped stylesheet |
|
|
42
42
|
|
|
@@ -61,13 +61,13 @@ prompt/validation registry in `vanillasky/server.ts`.
|
|
|
61
61
|
3. `createVideoChatHandler(...)` owns chat actions, suggestions, narration,
|
|
62
62
|
capability discovery, and generated-video budgets. The application supplies
|
|
63
63
|
provider callbacks and policy.
|
|
64
|
-
4. Video answers flow into
|
|
64
|
+
4. Video answers flow into the internal composition pipeline, which calls the
|
|
65
65
|
application's `streamText` adapter. This is where Anthropic, OpenAI, or
|
|
66
66
|
another text model is connected.
|
|
67
67
|
5. The system prompt combines the opening and composition rules with the trusted template
|
|
68
68
|
catalog, including generated metadata for customer-owned templates. The user
|
|
69
|
-
prompt serializes the
|
|
70
|
-
|
|
69
|
+
prompt serializes the prompt, completed conversation, instructions,
|
|
70
|
+
brand, and approved media.
|
|
71
71
|
6. The model streams one host-consumed opening object followed by NDJSON plan
|
|
72
72
|
parts. The server emits the opening event, then parses and validates complete
|
|
73
73
|
scenes before emitting them.
|
|
@@ -75,8 +75,7 @@ prompt/validation registry in `vanillasky/server.ts`.
|
|
|
75
75
|
loops or holds until its voice ends and the first playable scene is ready;
|
|
76
76
|
the planned playback and narration then take over in one cut.
|
|
77
77
|
|
|
78
|
-
|
|
79
|
-
available for explicit one-shot video response integrations.
|
|
78
|
+
`VideoPlayer` also replays completed chat videos parsed at the storage boundary.
|
|
80
79
|
|
|
81
80
|
## Where to change common behavior
|
|
82
81
|
|
|
@@ -93,7 +92,6 @@ available for explicit one-shot video response integrations.
|
|
|
93
92
|
|
|
94
93
|
## Public vocabulary
|
|
95
94
|
|
|
96
|
-
Use `VideoChat`, `useVideoChat`, `createVideoChatHandler`, `Video`, `
|
|
97
|
-
`useVideo`, and `VideoPlayer` for the product API. Use “video response” when
|
|
95
|
+
Use `VideoChat`, `useVideoChat`, `createVideoChatHandler`, `Video`, and `VideoPlayer` for the product API. Use “video response” when
|
|
98
96
|
describing the lifecycle or output category. Use “motion” only for animation
|
|
99
97
|
behavior inside the visual system.
|
package/docs/concepts.md
CHANGED
|
@@ -29,25 +29,13 @@ or export pipeline when an encoded file is required.
|
|
|
29
29
|
|
|
30
30
|
## Input
|
|
31
31
|
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
knowledge may supplement the request;
|
|
40
|
-
- `instructions`: optional creative direction that cannot override facts;
|
|
41
|
-
- `opening`: optional custom copy for the deterministic opening; omission uses
|
|
42
|
-
`Creating your video...`, while `false` lets the host render loading UI
|
|
43
|
-
without adding an opening scene to the video;
|
|
44
|
-
- `personalization`: application-defined fields such as name, role, account,
|
|
45
|
-
period, goal, or onboarding partner;
|
|
46
|
-
- `brand`: an optional background preset plus name, logo, font, surfaces, and
|
|
47
|
-
advanced exact tokens;
|
|
48
|
-
- `suppliedMedia`: approved images or videos with a role and description;
|
|
49
|
-
- `audio`: omit for automatic selection, pass `{ src }`, or use `false` for silence;
|
|
50
|
-
- `orientation` and `maxDurationSec`: composition constraints.
|
|
32
|
+
The viewer sends a prompt with bounded completed conversation turns. The server
|
|
33
|
+
adds trusted application `instructions`, template capabilities, and the selected
|
|
34
|
+
visual mode. `VideoChat` options control brand, style, orientation, and custom
|
|
35
|
+
templates. Exact facts belong in the authorized prompt or conversation; secrets
|
|
36
|
+
and provider configuration stay on the server.
|
|
37
|
+
|
|
38
|
+
See [Prompt and conversation input](prompt-and-input.md) for the request contract.
|
|
51
39
|
|
|
52
40
|
## Template kit
|
|
53
41
|
|
|
@@ -72,20 +60,10 @@ another streaming text model.
|
|
|
72
60
|
The planner does not create public event IDs, checksums, or final
|
|
73
61
|
snapshots. The runtime owns those guarantees.
|
|
74
62
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
last without persisting `placement` into the completed `Video`.
|
|
80
|
-
|
|
81
|
-
| Boundary | Type | Owner |
|
|
82
|
-
| --- | --- | --- |
|
|
83
|
-
| Planner parts | Internal validated plan data | Server/provider adapter |
|
|
84
|
-
| Values read from `response.stream` | `VideoEvent` | SDK runtime |
|
|
85
|
-
| Terminal editable result | `Video` via `response.result` | SDK reducer |
|
|
86
|
-
|
|
87
|
-
A planner must never yield `VideoEvent` envelopes. It yields plan
|
|
88
|
-
parts; the runtime validates them and creates the public event metadata.
|
|
63
|
+
The internal composition pipeline can reserve a validated closing scene and
|
|
64
|
+
append it after the body. Planner parts are internal data; applications connect
|
|
65
|
+
text callbacks to `createVideoChatHandler` and let the SDK own event envelopes,
|
|
66
|
+
sequence IDs, and final snapshots.
|
|
89
67
|
|
|
90
68
|
## Event stream
|
|
91
69
|
|
|
@@ -93,21 +71,19 @@ The runtime emits ordered protocol events:
|
|
|
93
71
|
|
|
94
72
|
```text
|
|
95
73
|
response.start
|
|
96
|
-
|
|
97
|
-
scene.add supplied opening
|
|
74
|
+
data.video-chat-opening short spoken hook
|
|
98
75
|
scene.add generated body
|
|
99
76
|
scene.add generated body
|
|
100
77
|
scene.add reserved closer
|
|
101
78
|
response.complete exact terminal snapshot
|
|
102
79
|
```
|
|
103
80
|
|
|
104
|
-
Network transport uses SSE.
|
|
105
|
-
events as an async iterable.
|
|
81
|
+
Network transport uses SSE. Test utilities can simulate protocol events in process.
|
|
106
82
|
|
|
107
83
|
## Playback and buffering
|
|
108
84
|
|
|
109
85
|
The player renders the first committed scene and continues through the known
|
|
110
|
-
timeline.
|
|
86
|
+
timeline. The chat holds its current visual during a generation gap. A media-bearing scene should
|
|
111
87
|
not be committed until its asset is ready; start generated content with a
|
|
112
88
|
typography-led scene so useful playback does not wait on media lookup.
|
|
113
89
|
|
|
@@ -124,8 +100,8 @@ complete storage contract.
|
|
|
124
100
|
## Completion snapshot
|
|
125
101
|
|
|
126
102
|
`response.complete` contains the exact reduced `Video` plus a
|
|
127
|
-
checksum.
|
|
128
|
-
|
|
103
|
+
checksum. The chat retains the completed video on its turn for application-owned replay
|
|
104
|
+
and storage. A partial playable answer may complete with non-fatal warnings.
|
|
129
105
|
|
|
130
106
|
Read the complete [Protocol 0.5](reference/protocol.md) and the separate
|
|
131
107
|
[persisted Video 0.1 contract](persistence.md).
|
package/docs/custom-templates.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
[← Documentation home](../README.md) · [Previous:
|
|
1
|
+
[← Documentation home](../README.md) · [Previous: Customization](customization.md) · [Next: Motion and effects →](motion-and-effects.md)
|
|
2
2
|
|
|
3
3
|
# Custom templates
|
|
4
4
|
|
|
@@ -190,7 +190,6 @@ Start from the generated file or one of the packaged references:
|
|
|
190
190
|
| --- | --- |
|
|
191
191
|
| One headline and supporting line | [Minimal text](../examples/custom-template/minimal-text.tsx) |
|
|
192
192
|
| One exact metric and its change | [Structured data](../examples/custom-template/structured-data.tsx) |
|
|
193
|
-
| An application-supplied image | [Supplied media](../examples/custom-template/supplied-media.tsx) |
|
|
194
193
|
|
|
195
194
|
The [reference comparison](../examples/custom-template/README.md) explains when
|
|
196
195
|
to choose each. Copy a file into `vanillasky/templates/`, change its ID and
|
|
@@ -213,18 +212,27 @@ selection, and validator. Import it in the route that connects your model:
|
|
|
213
212
|
|
|
214
213
|
```ts
|
|
215
214
|
// src/video-route.ts
|
|
216
|
-
import { streamText } from "ai";
|
|
217
|
-
import {
|
|
215
|
+
import { generateText, streamText } from "ai";
|
|
216
|
+
import { createVideoChatHandler } from "@vanillaskyai/video/server";
|
|
218
217
|
import { videoModel } from "./video-model";
|
|
219
218
|
import { templates } from "../vanillasky/server";
|
|
220
219
|
|
|
221
|
-
export const
|
|
220
|
+
export const handleVideoChat = createVideoChatHandler({
|
|
222
221
|
templates,
|
|
223
222
|
authorize: (request) => {
|
|
224
223
|
if (process.env.VANILLASKY_LOCAL_DEMO !== "1") return false;
|
|
225
224
|
const hostname = new URL(request.url).hostname;
|
|
226
225
|
return hostname === "localhost" || hostname === "127.0.0.1";
|
|
227
226
|
},
|
|
227
|
+
generateText: async ({ systemPrompt, userPrompt, signal }) => {
|
|
228
|
+
const result = await generateText({
|
|
229
|
+
model: videoModel,
|
|
230
|
+
system: systemPrompt,
|
|
231
|
+
prompt: userPrompt,
|
|
232
|
+
abortSignal: signal,
|
|
233
|
+
});
|
|
234
|
+
return result.text;
|
|
235
|
+
},
|
|
228
236
|
streamText: ({ systemPrompt, userPrompt, signal }) => streamText({
|
|
229
237
|
model: videoModel,
|
|
230
238
|
system: systemPrompt,
|
|
@@ -242,19 +250,13 @@ check before deployment, as shown in
|
|
|
242
250
|
Use the browser registry for generation and playback:
|
|
243
251
|
|
|
244
252
|
```tsx
|
|
245
|
-
// src/video-
|
|
246
|
-
import {
|
|
253
|
+
// src/video-chat.tsx
|
|
254
|
+
import { VideoChat } from "@vanillaskyai/video/react";
|
|
255
|
+
import "@vanillaskyai/video/video-chat.css";
|
|
247
256
|
import { templates } from "../vanillasky";
|
|
248
257
|
|
|
249
|
-
export function
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
return <>
|
|
253
|
-
<button onClick={() => void video.generate({ input: "Grounded answer" })}>
|
|
254
|
-
Generate
|
|
255
|
-
</button>
|
|
256
|
-
<VideoPlayer {...video.playerProps} />
|
|
257
|
-
</>;
|
|
258
|
+
export function App() {
|
|
259
|
+
return <VideoChat options={{ templates }} />;
|
|
258
260
|
}
|
|
259
261
|
```
|
|
260
262
|
|
|
@@ -311,8 +313,7 @@ export function TemplatePreview() {
|
|
|
311
313
|
|
|
312
314
|
Change the saved variables to each named example, inspect both orientations,
|
|
313
315
|
and play the scene to inspect its full progress range. This is also the
|
|
314
|
-
production replay path for a completed
|
|
315
|
-
`await video.generate(...)`.
|
|
316
|
+
production replay path for a completed chat turn saved from `chat.turns`.
|
|
316
317
|
|
|
317
318
|
## Schema and grounding
|
|
318
319
|
|
|
@@ -330,14 +331,17 @@ Useful formats add grounding behavior:
|
|
|
330
331
|
- `grounded-stat` marks numeric statistical evidence for the planner; it does
|
|
331
332
|
not compare the value against raw input at runtime;
|
|
332
333
|
- `grounded-quote` requires the quote to exist verbatim in the input;
|
|
333
|
-
- `
|
|
334
|
-
- `uri` applies the supplied-media or server URL policy to an approved URL;
|
|
334
|
+
- `uri` validates approved media URLs restored by the server;
|
|
335
335
|
- `stock-media-keyword` is only for hosts that resolve stock media before a
|
|
336
336
|
scene is committed.
|
|
337
337
|
|
|
338
338
|
Templates whose core proof needs a real statistic can add
|
|
339
|
-
`"x-vanillasky": { "requiresStat": true }`. See the structured-data
|
|
340
|
-
|
|
339
|
+
`"x-vanillasky": { "requiresStat": true }`. See the structured-data reference for a complete example.
|
|
340
|
+
|
|
341
|
+
For media-backed chat scenes, copy the built-in `media` template and keep its
|
|
342
|
+
`mediaKeyword`, `mediaUrl`, and `mediaType` contract. Configure `searchMedia` on
|
|
343
|
+
the chat handler so the server resolves semantic searches into approved assets.
|
|
344
|
+
The chat request does not accept a separate supplied-media input list.
|
|
341
345
|
|
|
342
346
|
The automatic `opening` uses the built-in `media` variables (`texts` and
|
|
343
347
|
`mediaType: "gradient"`). If you replace `media` while using automatic
|
|
@@ -353,5 +357,5 @@ trusted template and fills its declared schema; it never writes executable UI
|
|
|
353
357
|
code.
|
|
354
358
|
|
|
355
359
|
For the exact request path and source locations, see
|
|
356
|
-
[Architecture](architecture.md). For media policies and
|
|
360
|
+
[Architecture](architecture.md). For media policies and provider callbacks,
|
|
357
361
|
see [Media and audio](media-and-audio.md).
|
package/docs/customization.md
CHANGED
|
@@ -4,9 +4,11 @@
|
|
|
4
4
|
|
|
5
5
|
## Video chat interface
|
|
6
6
|
|
|
7
|
-
The default `VideoChat`
|
|
8
|
-
|
|
9
|
-
|
|
7
|
+
The default `VideoChat` uses an immersive video canvas with a floating
|
|
8
|
+
conversation field, on-video subtitles, and a single dark settings/history
|
|
9
|
+
treatment. See [interface behavior](immersive-interface.md) and the
|
|
10
|
+
[component reference](reference/design-system.html). Pass a custom welcome
|
|
11
|
+
heading and a root class when the application needs its own copy or chrome colors:
|
|
10
12
|
|
|
11
13
|
```tsx
|
|
12
14
|
<VideoChat
|
|
@@ -22,21 +24,25 @@ that instance:
|
|
|
22
24
|
|
|
23
25
|
```css
|
|
24
26
|
.acme-chat {
|
|
25
|
-
--vs-
|
|
26
|
-
--vs-
|
|
27
|
+
--vs-media-glass: rgb(18 24 40 / 80%);
|
|
28
|
+
--vs-media-text: #f8f8fc;
|
|
27
29
|
--vs-voice: #e11d74;
|
|
28
30
|
--vs-font: "Inter", sans-serif;
|
|
29
|
-
--vs-radius-lg: 1rem;
|
|
30
31
|
}
|
|
31
32
|
```
|
|
32
33
|
|
|
34
|
+
The built-in navigation carries the VanillaSky logo. `welcomeTitle` changes
|
|
35
|
+
the welcome heading; it does not replace the navigation logo. `options.brand`
|
|
36
|
+
styles generated video content independently of the surrounding controls.
|
|
37
|
+
|
|
33
38
|
Use `options` for the endpoint, templates, orientation, visual brand, request
|
|
34
39
|
headers, and an optional custom voice. Provider capabilities are discovered
|
|
35
40
|
from the server. Use `useVideoChat()` only when the application needs to own the
|
|
36
41
|
entire interface.
|
|
37
42
|
|
|
38
|
-
|
|
39
|
-
|
|
43
|
+
Pass the following visual settings through `VideoChat` or `useVideoChat` options.
|
|
44
|
+
Keep viewer context in the prompt and completed conversation turns; use the
|
|
45
|
+
server handler’s `instructions` for trusted product guidance.
|
|
40
46
|
|
|
41
47
|
## Background and semantic brand
|
|
42
48
|
|
|
@@ -96,35 +102,17 @@ These are defaults, not generated CSS. A validated scene may still select a
|
|
|
96
102
|
more appropriate text or background treatment when its trusted template allows
|
|
97
103
|
it.
|
|
98
104
|
|
|
99
|
-
## Personalization
|
|
100
|
-
|
|
101
|
-
`personalization` accepts JSON-safe application fields. Include only values the
|
|
102
|
-
viewer is allowed to see. The system prompt treats them as context, not as
|
|
103
|
-
instructions.
|
|
104
|
-
|
|
105
|
-
Good fields include `firstName`, `role`, `accountName`, `period`, `goal`,
|
|
106
|
-
`locale`, and `onboardingPartner`. Keep facts in `input` as well when they must
|
|
107
|
-
appear in the story.
|
|
108
|
-
|
|
109
105
|
## Opening
|
|
110
106
|
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
The opening is deterministic and should not wait for an LLM or remote media.
|
|
116
|
-
Omit it to use `Creating your video...`, or supply one concise custom sentence:
|
|
107
|
+
The planner streams a short spoken hook before the scenes. The chat holds that
|
|
108
|
+
opening until its speech finishes and the first scene is ready. A selected
|
|
109
|
+
suggestion can start with its prewritten opening and already-loaded media:
|
|
117
110
|
|
|
118
111
|
```ts
|
|
119
|
-
|
|
112
|
+
await chat.ask(card.prompt, { opening: card.opening, openingMedia: card.media });
|
|
120
113
|
```
|
|
121
114
|
|
|
122
|
-
|
|
123
|
-
loading UI until the first generated scene arrives.
|
|
124
|
-
|
|
125
|
-
Use `opening` only for a genuine opening that should remain in the completed
|
|
126
|
-
response. VanillaSky infers the scene ID, `media` template, gradient variables,
|
|
127
|
-
and three-second timing. Keep generic loading state in the host UI instead.
|
|
115
|
+
Openings and scene narration share the chat voice and pause/mute controls.
|
|
128
116
|
|
|
129
117
|
## Aspect ratio and responsive layout
|
|
130
118
|
|
|
@@ -138,31 +126,11 @@ should display landscape on desktop and portrait on mobile without changing the
|
|
|
138
126
|
saved response, pass `orientation="auto"` to `VideoPlayer`; it responds
|
|
139
127
|
to its container width. See [responsive orientation](responsive-orientation.md).
|
|
140
128
|
|
|
141
|
-
##
|
|
142
|
-
|
|
143
|
-
Provide approved media with semantic descriptions:
|
|
144
|
-
|
|
145
|
-
```ts
|
|
146
|
-
suppliedMedia: [{
|
|
147
|
-
id: "product-dashboard",
|
|
148
|
-
url: "https://cdn.example.com/dashboard.png",
|
|
149
|
-
type: "image",
|
|
150
|
-
description: "Activation dashboard after the Q2 release",
|
|
151
|
-
role: "product",
|
|
152
|
-
}]
|
|
153
|
-
```
|
|
154
|
-
|
|
155
|
-
Resolve searched media before generation and pass the approved result through
|
|
156
|
-
`suppliedMedia`. See [media providers](media-and-audio.md#media-providers).
|
|
157
|
-
|
|
158
|
-
## Soundtrack audio
|
|
129
|
+
## Media and voice
|
|
159
130
|
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
The soundtrack should continue across visual generation gaps and finish with
|
|
164
|
-
the final scene. `VideoChat` separately owns narration, TTS integration, browser
|
|
165
|
-
voice fallback, and speech synchronization for conversational responses.
|
|
131
|
+
Configure `searchMedia`, `generateVideo`, and `generateSpeech` on the server
|
|
132
|
+
handler. They progressively enhance the same chat; failed optional providers
|
|
133
|
+
fall back to templates or browser voice. See [Media and voice](media-and-audio.md).
|
|
166
134
|
|
|
167
135
|
## Custom templates
|
|
168
136
|
|
package/docs/errors.md
CHANGED
|
@@ -2,71 +2,61 @@
|
|
|
2
2
|
|
|
3
3
|
# Errors and recovery
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
|
12
|
-
|
|
|
13
|
-
|
|
|
14
|
-
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
`onError` with the full internal reason, while the browser receives only the
|
|
64
|
-
recoverable `invalid_generated_part` diagnostic. Set `invalidPartBehavior: "fail"`
|
|
65
|
-
only when fail-fast generation is an intentional compatibility requirement.
|
|
66
|
-
|
|
67
|
-
`onComplete` runs exactly once only after `response.complete`. Terminal errors,
|
|
68
|
-
client disconnects, explicit aborts, and host timeouts do not call it;
|
|
69
|
-
`onError` receives internal failures, while cancellation remains a safe
|
|
70
|
-
`response.abort`. Abort and timeout signals always propagate to `streamText`.
|
|
71
|
-
The host owns timeout construction and retry policy. Do not retry invisibly
|
|
72
|
-
after any scene has reached the viewer.
|
|
5
|
+
`VideoChat` preserves the opening and accepted scenes when an optional scene,
|
|
6
|
+
provider, or stream fails. It recovers silently and keeps the answer playable. A fatal error is shown only when no playable response can
|
|
7
|
+
be produced. Never put provider details or stack traces in the interface.
|
|
8
|
+
|
|
9
|
+
## Recovery by boundary
|
|
10
|
+
|
|
11
|
+
| Failure | Result |
|
|
12
|
+
| --- | --- |
|
|
13
|
+
| Invalid planner scene | Skip the part and accept later valid scenes |
|
|
14
|
+
| Generated footage | Try stock media, matching completed footage, then a safe template |
|
|
15
|
+
| Stock lookup or candidate | Continue with another candidate or template |
|
|
16
|
+
| Template renderer | Isolate the failed scene with a safe visual |
|
|
17
|
+
| Missing narration | Continue with scene text |
|
|
18
|
+
| Generated speech | Use browser voice |
|
|
19
|
+
| Late stream failure | Keep playable opening and completed scenes |
|
|
20
|
+
| Unauthorized or invalid request without playable output | Show a safe error |
|
|
21
|
+
|
|
22
|
+
Keep non-fatal `chat.warnings` as application diagnostics; they are also
|
|
23
|
+
retained on the corresponding `VideoChatTurn.warnings`. Render `chat.error`
|
|
24
|
+
when the response cannot play, and keep `chat.playerProps` mounted during recovery.
|
|
25
|
+
Check `turn.completed` before persisting it as completed conversation context;
|
|
26
|
+
a cancelled or partial visible turn is not automatically a completed turn.
|
|
27
|
+
|
|
28
|
+
`VideoError` exposes actionable public fields: `code`, `message`, optional HTTP
|
|
29
|
+
`status`, `requestId`, `runId`, and `recoverable`. Log only safe codes and IDs
|
|
30
|
+
from the client. The server's `onError` observer receives internal diagnostics;
|
|
31
|
+
redact credentials, source data, provider payloads, and signed URLs before logging.
|
|
32
|
+
Observer failures are isolated from response generation.
|
|
33
|
+
|
|
34
|
+
## Cancellation and retries
|
|
35
|
+
|
|
36
|
+
`chat.cancel()` cancels the current work while preserving available output.
|
|
37
|
+
Replacing a prompt aborts its old providers and speech. The host must forward
|
|
38
|
+
`signal` to every provider callback and own deadlines, quotas, and retry budgets.
|
|
39
|
+
The chat retries once only before playback. Never silently restart generation
|
|
40
|
+
after the viewer has begun watching, and do not replay paid generation requests
|
|
41
|
+
without a deliberate idempotency and spend policy.
|
|
42
|
+
|
|
43
|
+
The server drops invalid generated parts by default. `invalidPartBehavior: "fail"`
|
|
44
|
+
is an explicit strict policy; ordinary chat should retain the resilient default.
|
|
45
|
+
`onComplete` runs after a `response.complete`, including a recovered playable
|
|
46
|
+
response. Fatal errors, disconnects, and explicit aborts do not call it.
|
|
47
|
+
|
|
48
|
+
Automated regression and acceptance tests use mocked providers. See
|
|
49
|
+
[Testing](testing.md) and the [chat acceptance gate](https://github.com/VanillaSkyAi/video/blob/main/docs/maintainers/acceptance.md).
|
|
50
|
+
|
|
51
|
+
## Slow optional providers
|
|
52
|
+
|
|
53
|
+
Generated video has a 15-second deadline per lookup. Stock media, generated
|
|
54
|
+
speech preparation, and fallback scene narration have 3-second deadlines.
|
|
55
|
+
A deadline uses the same safe fallback as a failed provider; completed scenes
|
|
56
|
+
and scene order are preserved. Providers receive cancellation, and late results
|
|
57
|
+
are ignored even when a provider does not cooperate. These initial limits bound
|
|
58
|
+
waiting; they are not claims about measured live-provider performance.
|
|
59
|
+
|
|
60
|
+
A completed turn and its saved video are available before follow-up suggestions.
|
|
61
|
+
Suggestions load separately and may be omitted after a short deadline. Starting
|
|
62
|
+
another turn or cancelling discards outstanding suggestions.
|
package/docs/getting-started.md
CHANGED
|
@@ -4,7 +4,8 @@
|
|
|
4
4
|
|
|
5
5
|
The fastest VanillaSky integration is the complete, general-purpose video chat.
|
|
6
6
|
It starts with packaged templates and browser voice, then turns on optional
|
|
7
|
-
speech
|
|
7
|
+
speech and generated video when you install their adapters and add server keys.
|
|
8
|
+
Stock media needs only its server key.
|
|
8
9
|
|
|
9
10
|
## Create the app
|
|
10
11
|
|
|
@@ -15,7 +16,9 @@ npx @vanillaskyai/video init
|
|
|
15
16
|
```
|
|
16
17
|
|
|
17
18
|
Init installs the exact SDK version that ran it, creates a small application
|
|
18
|
-
shell, and
|
|
19
|
+
shell, installs baseline dependencies, and runs doctor automatically. Optional
|
|
20
|
+
speech and video packages are not installed. If installation is interrupted,
|
|
21
|
+
rerun the same init command to finish setup.
|
|
19
22
|
It does not copy VanillaSky's template tree. The important generated files are:
|
|
20
23
|
|
|
21
24
|
| File | Your application owns |
|
|
@@ -59,23 +62,27 @@ variable. Then inspect the setup without calling any provider:
|
|
|
59
62
|
npx vanillasky doctor
|
|
60
63
|
```
|
|
61
64
|
|
|
62
|
-
The base experience reports `templates + browser voice`.
|
|
63
|
-
the
|
|
65
|
+
The base experience reports `templates + browser voice`. `ANTHROPIC_API_KEY`
|
|
66
|
+
is the only required key. Doctor checks setup locally without calling providers
|
|
67
|
+
and reports names and readiness, never values.
|
|
64
68
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
69
|
+
## Add optional capabilities
|
|
70
|
+
|
|
71
|
+
Install only the capability you want:
|
|
68
72
|
|
|
69
|
-
|
|
70
|
-
|
|
73
|
+
```bash
|
|
74
|
+
# Generated speech: installs the xAI adapter
|
|
75
|
+
npx vanillasky providers add speech
|
|
71
76
|
|
|
72
|
-
#
|
|
73
|
-
|
|
77
|
+
# Generated video and transcription: installs the FAL adapter
|
|
78
|
+
npx vanillasky providers add video
|
|
74
79
|
```
|
|
75
80
|
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
81
|
+
Add `XAI_API_KEY` for speech or `FAL_KEY` for video and transcription to
|
|
82
|
+
`.env.local`, then restart the dev server. Stock media needs only
|
|
83
|
+
`PEXELS_API_KEY`; no extra package or command is needed. The client stays the
|
|
84
|
+
same. Rerun an interrupted provider command to finish its installation, then
|
|
85
|
+
check readiness with `npx vanillasky doctor`.
|
|
79
86
|
|
|
80
87
|
## Run and verify
|
|
81
88
|
|