@vanillaskyai/video 0.10.23 → 0.11.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (127) hide show
  1. package/CHANGELOG.md +61 -0
  2. package/PUBLIC-API.md +45 -438
  3. package/README.md +53 -97
  4. package/dist/builtin-metadata-OT6V7TB4.js +8 -0
  5. package/dist/{chapter-title-2JVDU62E.js → chapter-title-2RCXX7SN.js} +1 -2
  6. package/dist/{chunk-PDQFQIQW.js → chunk-3EQ6PVWL.js} +1 -1
  7. package/dist/chunk-4G4JBMCM.js +37 -0
  8. package/dist/{chunk-7AA2JWHZ.js → chunk-5DOQTIMD.js} +6 -6
  9. package/dist/chunk-666HGTVZ.js +41 -0
  10. package/dist/{chunk-7M56IUUX.js → chunk-6Z3ID54H.js} +0 -18
  11. package/dist/{chunk-5BSJLT6H.js → chunk-E7FHBLAC.js} +16 -29
  12. package/dist/chunk-HIQM4J3U.js +30 -0
  13. package/dist/chunk-ISWQ6T5X.js +36 -0
  14. package/dist/{chunk-224QNWRA.js → chunk-K5J7ESRO.js} +1 -10
  15. package/dist/chunk-KMVRBUL5.js +50 -0
  16. package/dist/chunk-NM4CXXZY.js +25 -0
  17. package/dist/{chunk-44WND2VP.js → chunk-V2CP7PVY.js} +19 -63
  18. package/dist/{chunk-3O7OMMMF.js → chunk-WZESNEPT.js} +0 -1
  19. package/dist/{chunk-RXTN2CW6.js → chunk-Z3DLSLAJ.js} +1 -1
  20. package/dist/cinema-media-YZ2UANJG.js +365 -0
  21. package/dist/cli.js +154 -1748
  22. package/dist/{compose-video-BH5K6XWT.js → compose-video-H2YUQDUY.js} +4 -4
  23. package/dist/{events-B4YCc4vc.d.ts → events-BZAfl0Dm.d.ts} +1 -1
  24. package/dist/index.d.ts +2 -2
  25. package/dist/index.js +4 -6
  26. package/dist/preload-media-23JXUFCF.js +77 -0
  27. package/dist/react.d.ts +39 -15
  28. package/dist/react.js +1350 -739
  29. package/dist/scene-validation-SGLLLFY5.js +9 -0
  30. package/dist/server.d.ts +67 -117
  31. package/dist/server.js +262 -614
  32. package/dist/test.d.ts +2 -2
  33. package/dist/test.js +21 -21
  34. package/dist/{text-stream-LD364KBI.js → text-stream-SOVLYR2L.js} +2 -2
  35. package/dist/{types-BqB8zC9u.d.ts → types-CG-kPI81.d.ts} +3 -1
  36. package/dist/{types-DdZw4GRQ.d.ts → types-ht-Zw3Wv.d.ts} +1 -1
  37. package/docs/agent-integration.md +19 -64
  38. package/docs/architecture.md +80 -106
  39. package/docs/customization.md +49 -66
  40. package/docs/development.md +18 -17
  41. package/docs/errors.md +4 -4
  42. package/docs/getting-started.md +74 -93
  43. package/docs/media-and-audio.md +9 -7
  44. package/docs/performance.md +11 -6
  45. package/docs/persistence.md +3 -7
  46. package/docs/production.md +4 -15
  47. package/docs/prompt-and-input.md +1 -2
  48. package/docs/provider-integration.md +134 -181
  49. package/docs/reference/protocol.md +5 -14
  50. package/docs/reference/provider-adapters.md +21 -13
  51. package/docs/security.md +1 -14
  52. package/docs/testing.md +56 -123
  53. package/package.json +4 -25
  54. package/starters/video-chat/.env.example +12 -4
  55. package/starters/video-chat/.env.native.example +13 -0
  56. package/starters/video-chat/README.md +72 -11
  57. package/starters/video-chat/package.json +1 -1
  58. package/starters/video-chat/providers/text-native.ts +94 -0
  59. package/starters/video-chat/providers/text.ts +19 -0
  60. package/starters/video-chat/providers/transcription.ts +32 -0
  61. package/starters/video-chat/providers/video-custom.ts +45 -0
  62. package/starters/video-chat/providers/video-delivery.ts +57 -0
  63. package/starters/video-chat/providers/video-google.ts +48 -0
  64. package/starters/video-chat/providers/video-job.ts +103 -0
  65. package/starters/video-chat/providers/video-runway.ts +45 -0
  66. package/starters/video-chat/providers/video.ts +39 -63
  67. package/starters/video-chat/server.ts +3 -29
  68. package/starters/video-chat/vite.config.ts +2 -8
  69. package/styles/video-chat.css +8 -105
  70. package/dist/builtin-server-W4PLJUZ5.js +0 -8
  71. package/dist/catalog-types-WTbLP6Jh.d.ts +0 -78
  72. package/dist/check-runtime.d.ts +0 -15
  73. package/dist/check-runtime.js +0 -96
  74. package/dist/chunk-2E6T633S.js +0 -27
  75. package/dist/chunk-4YM2M62S.js +0 -13
  76. package/dist/chunk-4ZJLPHBV.js +0 -684
  77. package/dist/chunk-5JBMYQP6.js +0 -156
  78. package/dist/chunk-73NTSFFI.js +0 -81
  79. package/dist/chunk-EGVQODKU.js +0 -83
  80. package/dist/chunk-HFVNAPHZ.js +0 -35
  81. package/dist/chunk-IIN5M5HW.js +0 -697
  82. package/dist/chunk-IQMYK5DX.js +0 -133
  83. package/dist/chunk-IR44XKBI.js +0 -46
  84. package/dist/chunk-JKVOBTRO.js +0 -145
  85. package/dist/chunk-LVM5Q2DL.js +0 -68
  86. package/dist/chunk-M4QTEJTK.js +0 -709
  87. package/dist/chunk-QSBDB4J2.js +0 -16
  88. package/dist/chunk-R3XAOMKP.js +0 -29
  89. package/dist/chunk-SPVTJH3F.js +0 -24
  90. package/dist/chunk-YAHT3LST.js +0 -663
  91. package/dist/chunk-ZD2VTUYR.js +0 -28
  92. package/dist/cinema-media-HYCDG65Z.js +0 -11
  93. package/dist/comparison-XXAP4S4J.js +0 -36
  94. package/dist/editorial-timeline-VUQC6KPA.js +0 -42
  95. package/dist/key-figure-4TJK7HLT.js +0 -29
  96. package/dist/kit-DrRpdn0p.d.ts +0 -79
  97. package/dist/mobile-message-FIHAO6XC.js +0 -49
  98. package/dist/preload-media-LJXWKTWG.js +0 -57
  99. package/dist/quote-GRPIYKQJ.js +0 -32
  100. package/dist/system-prompt-AG26KJAA.js +0 -12
  101. package/dist/template-catalog.d.ts +0 -400
  102. package/dist/template-catalog.js +0 -6
  103. package/dist/templates.d.ts +0 -25
  104. package/dist/templates.js +0 -102
  105. package/dist/validate-4OUD2CLX.js +0 -10
  106. package/docs/concepts.md +0 -107
  107. package/docs/custom-templates.md +0 -346
  108. package/docs/immersive-interface.md +0 -86
  109. package/docs/motion-and-effects.md +0 -109
  110. package/docs/reference/design-system.html +0 -125
  111. package/docs/responsive-orientation.md +0 -37
  112. package/docs/streaming-protocol.md +0 -15
  113. package/examples/custom-template/README.md +0 -19
  114. package/examples/custom-template/minimal-text.tsx +0 -82
  115. package/examples/custom-template/structured-data.tsx +0 -104
  116. package/registry/items/backgrounds.json +0 -70
  117. package/registry/items/chapterTitle.json +0 -80
  118. package/registry/items/cinemaMedia.json +0 -136
  119. package/registry/items/comparison.json +0 -168
  120. package/registry/items/editorialTimeline.json +0 -172
  121. package/registry/items/keyFigure.json +0 -162
  122. package/registry/items/mobileMessage.json +0 -153
  123. package/registry/items/motion.json +0 -45
  124. package/registry/items/quote.json +0 -161
  125. package/registry/items/template-context.json +0 -31
  126. package/registry/items/theme.json +0 -47
  127. package/registry/items/typography.json +0 -47
package/dist/test.d.ts CHANGED
@@ -1,5 +1,5 @@
1
- import { e as VideoScene, f as VIDEO_PROTOCOL_VERSION, b as VideoOrientation, V as VideoStyle, c as Video, d as VideoAudio, j as VideoInput } from './types-DdZw4GRQ.js';
2
- import { V as VideoWarning, a as VideoFinishReason } from './events-B4YCc4vc.js';
1
+ import { e as VideoScene, f as VIDEO_PROTOCOL_VERSION, V as VideoOrientation, a as VideoStyle, b as Video, d as VideoAudio, j as VideoInput } from './types-ht-Zw3Wv.js';
2
+ import { V as VideoWarning, a as VideoFinishReason } from './events-BZAfl0Dm.js';
3
3
 
4
4
  type MockVideoStreamPart = {
5
5
  type: "scene.add";
package/dist/test.js CHANGED
@@ -25,8 +25,8 @@ var portrait = {
25
25
  type: "scene.add",
26
26
  scene: {
27
27
  id: "portrait-summary",
28
- templateId: "mobileMessage",
29
- variables: { app: "VanillaSky", message: "Grounded product update" },
28
+ templateId: "chapterTitle",
29
+ variables: { title: "Grounded product update" },
30
30
  timing: { fixedDuration: 4 }
31
31
  }
32
32
  },
@@ -43,8 +43,8 @@ var landscape = {
43
43
  type: "scene.add",
44
44
  scene: {
45
45
  id: "landscape-metric",
46
- templateId: "keyFigure",
47
- variables: { value: "42 million", label: "Revenue" },
46
+ templateId: "chapterTitle",
47
+ variables: { title: "Revenue reached 42 million" },
48
48
  timing: { fixedDuration: 4 }
49
49
  }
50
50
  },
@@ -60,8 +60,8 @@ var truncated = [
60
60
  type: "scene.add",
61
61
  scene: {
62
62
  id: "truncated-partial",
63
- templateId: "mobileMessage",
64
- variables: { app: "VanillaSky", message: "Playable partial result" },
63
+ templateId: "chapterTitle",
64
+ variables: { title: "Playable partial result" },
65
65
  timing: { fixedDuration: 4 }
66
66
  }
67
67
  },
@@ -81,8 +81,8 @@ var invalidScene = [
81
81
  type: "scene.add",
82
82
  scene: {
83
83
  id: "valid-after-invalid",
84
- templateId: "mobileMessage",
85
- variables: { app: "VanillaSky", message: "Recovered after invalid content" },
84
+ templateId: "chapterTitle",
85
+ variables: { title: "Recovered after invalid content" },
86
86
  timing: { fixedDuration: 4 }
87
87
  }
88
88
  },
@@ -99,8 +99,8 @@ var contentFilter = [
99
99
  type: "scene.add",
100
100
  scene: {
101
101
  id: "content-filter-partial",
102
- templateId: "mobileMessage",
103
- variables: { app: "VanillaSky", message: "Safe partial result" },
102
+ templateId: "chapterTitle",
103
+ variables: { title: "Safe partial result" },
104
104
  timing: { fixedDuration: 4 }
105
105
  }
106
106
  },
@@ -111,8 +111,8 @@ var abort = [
111
111
  type: "scene.add",
112
112
  scene: {
113
113
  id: "abort-partial",
114
- templateId: "mobileMessage",
115
- variables: { app: "VanillaSky", message: "Partial result before cancellation" },
114
+ templateId: "chapterTitle",
115
+ variables: { title: "Partial result before cancellation" },
116
116
  timing: { fixedDuration: 4 }
117
117
  }
118
118
  },
@@ -123,8 +123,8 @@ var timeout = [
123
123
  type: "scene.add",
124
124
  scene: {
125
125
  id: "timeout-partial",
126
- templateId: "mobileMessage",
127
- variables: { app: "VanillaSky", message: "Partial result before timeout" },
126
+ templateId: "chapterTitle",
127
+ variables: { title: "Partial result before timeout" },
128
128
  timing: { fixedDuration: 4 }
129
129
  }
130
130
  },
@@ -233,10 +233,10 @@ async function* simulateVideoStream(parts, options = {}) {
233
233
  if (timeoutMs != null && (!Number.isFinite(timeoutMs) || timeoutMs < 0)) {
234
234
  throw new Error("Simulation timeoutMs must be a non-negative finite number");
235
235
  }
236
- const { createVideo } = await import("./compose-video-BH5K6XWT.js");
237
- const { createTextDeltaVideoPlanner } = await import("./text-stream-LD364KBI.js");
238
- const { BUILTIN_SERVER_TEMPLATE_KIT } = await import("./builtin-server-W4PLJUZ5.js");
239
- const { createTemplateSceneValidator } = await import("./validate-4OUD2CLX.js");
236
+ const { createVideo } = await import("./compose-video-H2YUQDUY.js");
237
+ const { createTextDeltaVideoPlanner } = await import("./text-stream-SOVLYR2L.js");
238
+ const { SCENE_DEFINITIONS, getBuiltinSceneDefinition } = await import("./builtin-metadata-OT6V7TB4.js");
239
+ const { validateBuiltinScene } = await import("./scene-validation-SGLLLFY5.js");
240
240
  const controller = new AbortController();
241
241
  const forwardAbort = () => controller.abort(options.signal?.reason);
242
242
  if (options.signal?.aborted) forwardAbort();
@@ -248,9 +248,9 @@ async function* simulateVideoStream(parts, options = {}) {
248
248
  const run = createVideo(input, {
249
249
  requestId: options.requestId ?? DEFAULT_REQUEST_ID,
250
250
  runId: options.runId ?? DEFAULT_RUN_ID,
251
- capabilities: cloneValue(BUILTIN_SERVER_TEMPLATE_KIT.capabilities),
252
- validateScene: createTemplateSceneValidator({ kit: BUILTIN_SERVER_TEMPLATE_KIT }),
253
- getTemplatePacing: (templateId) => BUILTIN_SERVER_TEMPLATE_KIT.getTemplateMetadata(templateId),
251
+ capabilities: { templates: SCENE_DEFINITIONS.map((scene) => scene.id) },
252
+ validateScene: validateBuiltinScene,
253
+ getTemplatePacing: getBuiltinSceneDefinition,
254
254
  invalidPartBehavior: options.invalidPartBehavior ?? "drop",
255
255
  signal: controller.signal,
256
256
  generate
@@ -1,8 +1,8 @@
1
1
  import {
2
2
  createTextDeltaVideoPlanner
3
- } from "./chunk-PDQFQIQW.js";
3
+ } from "./chunk-3EQ6PVWL.js";
4
4
  import "./chunk-E7CL7UPB.js";
5
- import "./chunk-7M56IUUX.js";
5
+ import "./chunk-6Z3ID54H.js";
6
6
  import "./chunk-7NZOMTAL.js";
7
7
  export {
8
8
  createTextDeltaVideoPlanner
@@ -15,6 +15,8 @@ interface VideoChatMedia {
15
15
  url: string;
16
16
  type: "image" | "video";
17
17
  posterUrl?: string;
18
+ /** Actual media duration, not a narration estimate. */
19
+ durationSec?: number;
18
20
  }
19
21
  interface VideoChatSuggestion {
20
22
  prompt: string;
@@ -43,4 +45,4 @@ interface VideoChatWelcomeOptions {
43
45
  prompts?: readonly VideoChatWelcomePrompt[];
44
46
  }
45
47
 
46
- export type { VideoChatMode as V, VideoChatAskOptions as a, VideoChatMedia as b, VideoChatSuggestion as c, VideoChatCapabilities as d, VideoChatWelcome as e, VideoChatWelcomeOptions as f, VideoChatConversationTurn as g, VideoChatWelcomePrompt as h };
48
+ export type { VideoChatConversationTurn as V, VideoChatMode as a, VideoChatWelcomeOptions as b, VideoChatCapabilities as c, VideoChatWelcomePrompt as d, VideoChatAskOptions as e, VideoChatMedia as f, VideoChatSuggestion as g, VideoChatWelcome as h };
@@ -191,4 +191,4 @@ interface VideoSnapshotRetention {
191
191
  suppliedMediaUrls?: boolean;
192
192
  }
193
193
 
194
- export { type VideoStyle as V, type VideoCapabilities as a, type VideoOrientation as b, type Video as c, type VideoAudio as d, type VideoScene as e, VIDEO_PROTOCOL_VERSION as f, type VideoTemplatePacing as g, type VideoNarrationGroup as h, type VideoStyleOptions as i, type VideoInput as j, type VideoGenerationContext as k, type VideoPlanner as l, type VideoRequest as m, type VideoSceneValidator as n, type VideoSnapshotRetention as o, type VideoResumeCursor as p, type VideoSceneValidationContext as q };
194
+ export { type VideoOrientation as V, type VideoStyle as a, type Video as b, type VideoCapabilities as c, type VideoAudio as d, type VideoScene as e, VIDEO_PROTOCOL_VERSION as f, type VideoTemplatePacing as g, type VideoNarrationGroup as h, type VideoStyleOptions as i, type VideoInput as j, type VideoPlanner as k, type VideoRequest as l, type VideoSceneValidator as m, type VideoSnapshotRetention as n, type VideoResumeCursor as o, type VideoGenerationContext as p };
@@ -1,71 +1,26 @@
1
- [← Documentation home](../README.md) · [Next: Getting started →](getting-started.md)
2
-
3
1
  # Agent integration
4
2
 
5
- Coding agents should build the same canonical video chat as a human developer.
6
- Install the repository skill when the agent supports skills:
7
-
8
- ```bash
9
- npx skills add VanillaSkyAi/video@vanillasky
10
- ```
11
-
12
- Then prompt: `Use $vanillasky to set up and verify a general-purpose video chat in this project.`
13
-
14
- ## Set up the canonical chat
15
-
16
- The skill follows this public path in an empty npm project:
17
-
18
- ```bash
19
- npx @vanillaskyai/video init
20
- npx vanillasky doctor
21
- npm run dev
22
- ```
23
-
24
- Init generates the thin application-owned shell around the SDK's complete
25
- `VideoChat` experience and runs doctor automatically. The baseline uses an immediate introduction and browser voice, and installs
26
- no optional provider packages. Add the video adapter for moving video answers;
27
- without a media provider, narration and subtitles remain available.
28
- `ANTHROPIC_API_KEY` is the only required key. Rerun init after an interrupted
29
- installation.
30
-
31
- If doctor reports a missing key, add the named key to the ignored `.env.local`
32
- yourself. Never read or print a secret value, and never paste one into the chat.
33
- Doctor is the safe capability interface: it reports names and readiness only.
34
-
35
- For requested upgrades, run `npx vanillasky providers add speech` for xAI speech
36
- or `npx vanillasky providers add video` for FAL video and transcription. Have the
37
- developer add `XAI_API_KEY` or `FAL_KEY` locally, then restart the server. Stock
38
- media needs only `PEXELS_API_KEY`. The client does not change.
39
-
40
- ## Verify, do not merely scaffold
41
-
42
- The agent should leave the dev server running and open its reported localhost
43
- URL in a real browser. Browser automation should use normal motion
44
- (`reducedMotion: "no-preference"`) so playback advances.
45
-
46
- 1. Confirm the welcome screen, suggestions, composer, and voice control load
47
- without console or page errors.
48
- 2. Ask one explanatory question and confirm the answer streams, speaks, reaches
49
- its final frame, and returns to a usable composer.
50
- 3. Ask one unrelated creative question in the same conversation and confirm it
51
- produces a distinct response.
52
- 4. If doctor reports generated video ready, verify moving footage throughout
53
- one complete answer, including its ending, without editing client code.
54
- 5. Check failed network responses before reporting success.
3
+ Build the supported chat path first. The optional integration skill is installed
4
+ with `npx skills add VanillaSkyAi/video@vanillasky`.
55
5
 
56
- The handoff should name the localhost URL and ready capabilities. It should
57
- never include credential values.
6
+ In an empty app, run `npx @vanillaskyai/video init`, use
7
+ `npx vanillasky doctor` to identify missing setup, and start `npm run dev`.
8
+ For an existing app, keep its framework and connect `VideoChat` to one
9
+ `createVideoChatHandler` using the [provider guide](provider-integration.md).
10
+ Do not copy SDK internals or invent a second generation client.
58
11
 
59
- ## Keep ownership clear
12
+ The host adds credentials to ignored server-only environment files. Never read,
13
+ print, screenshot, or send secret values. Doctor exposes key names only.
60
14
 
61
- VanillaSky owns the chat flow, planning prompts, templates, validation,
62
- streaming, voice timing, and player. The application owns provider choices,
63
- keys, authentication, persistence, branding, copy, and policy.
15
+ Verify in a real browser: a complete answer, its ending, a follow-up, pause/mute,
16
+ and console/network failures. Test moving footage when a video adapter is
17
+ configured. Report the localhost URL and ready capabilities, not just scaffold
18
+ completion.
64
19
 
65
- Customize generated application files only after the default chat passes in the
66
- browser. Do not inspect SDK internals, copy its template tree, or invent a
67
- parallel integration path. Source-own a template with
68
- `npx vanillasky templates ...` only when the product actually needs to edit it.
20
+ The SDK owns chat, shot planning, streaming, voice timing, and the two renderers.
21
+ The application owns providers, keys, auth, persistence, copy, and spending.
22
+ Use `useVideoChat` only when the host needs a custom UI; use `parseVideo` and
23
+ `VideoPlayer` for saved responses. Renderer plugins and template authoring are
24
+ not supported.
69
25
 
70
- The root README and [Getting started](getting-started.md) are the canonical
71
- human references. Evaluation-only package checks remain in the maintainer docs.
26
+ [Getting started](getting-started.md) · [Documentation home](../README.md)
@@ -1,118 +1,92 @@
1
1
  # Architecture
2
2
 
3
- VanillaSky turns context into a visual response. Your application and its AI
4
- decide what matters; VanillaSky turns that decision into a grounded, validated,
5
- embedded video that starts playing while it is still being composed.
3
+ The product is one video-chat pipeline: a chapter introduction, then narrated
4
+ footage, with chapter fallback when footage is unavailable. The application owns
5
+ providers and policy. There is no public template builder or template registry.
6
6
 
7
- ## The shortest mental model
7
+ ## Follow one response
8
8
 
9
9
  ```text
10
- user message + conversation context
11
- app-owned model
12
- trusted scene plan
13
- validation and streaming protocol
14
- synchronized voice-and-video response
10
+ VideoChat / useVideoChat
11
+ createVideoChatHandler (host policy, capabilities)
12
+ optional resolveAnswer (application's completed answer)
13
+ createChatShotPlanner (answer brief → ordered shots)
14
+ concurrent footage jobs + early preparation announcements
15
+ → createVideo (validated protocol events)
16
+ → browser speech/media preparation → ordered player + narration
15
17
  ```
16
18
 
17
- VanillaSky does not choose your model, retrieve private application context, or
18
- own your product policy. It supplies the complete default chat, video-planning
19
- prompts, trusted visual vocabulary, conversation and narration lifecycle,
20
- validation, streaming, and player.
19
+ Without `resolveAnswer`, shot planning begins directly from the request and
20
+ conversation. With it, the handler waits for the application's completed answer
21
+ before planning; that answer becomes the sole factual source. The hook is not
22
+ a stream of partially written assistant tokens.
21
23
 
22
- ## Start here
24
+ Scenes can start before the full video plan is ready. Preparation overlaps;
25
+ early media announcements let the browser warm upcoming assets even while an
26
+ earlier shot is pending. Public scene order stays deterministic. The final Video is replayable data;
27
+ saved replay does not request another generation.
23
28
 
24
- Six files, in this order, are enough to hold the whole system in your head.
25
- About 2,600 lines total, and two of them are most of it.
29
+ Narration is estimated against each clip with a 0.8-second tail. An oversized
30
+ beat gets at most one short rewrite before footage is requested. A failed or
31
+ still-oversized rewrite preserves the original speech in a chapter instead of
32
+ spending on unusable footage. Measured speech remains authoritative during
33
+ playback; video plays once and unexpected overruns recover to a chapter. Estimation is
34
+ not a guarantee that every voice/language finishes inside its clip.
26
35
 
27
- 1. `src/protocol/types.ts` the `Video` shape. Everything else exists to
28
- produce, validate, transport, or play this one object.
29
- 2. `src/server/prompts/system-prompt.ts` — what the model is actually asked
30
- for. The product's behavior is mostly here, not in the code around it.
31
- 3. `src/server/create-video-chat-handler.ts` — the one endpoint. Where an
32
- application's providers and policy attach.
33
- 4. `src/protocol/events.ts` — the wire contract between server and browser.
34
- 5. `src/video-chat/use-video-chat.ts` — the client lifecycle, as a reducer.
35
- Read `reducer` first and the hook second.
36
- 6. `src/visual-system/scene-templates/quote.tsx` — one complete scene, small
37
- enough to read in a sitting. Every other template has this shape.
36
+ ## Where to work
38
37
 
39
- To watch it run instead, `npm run dev:chat` renders the real `VideoChat` from
40
- source against fixtures, with no provider credentials and no spend.
41
-
42
- ## Repository map
43
-
44
- | Location | Purpose |
38
+ | Change | Source |
45
39
  | --- | --- |
46
- | `src/server/create-video-chat-handler.ts` | Complete chat endpoint and capability boundary |
47
- | `src/server/create-video-handler.ts` | Internal composition and provider adapter boundary |
48
- | `src/server/prompts/` | System and user prompts sent to the app-owned model |
49
- | `src/server/model/` | Converts provider text deltas into typed video plan parts |
50
- | `src/protocol/` | Shared request, event, validation, checksum, and SSE contract |
51
- | `src/player/` | Internal stream client, timeline, and React player |
52
- | `src/video-chat/` | Default `VideoChat` interface and headless conversation/session engine |
53
- | `src/visual-system/catalog/` | Template metadata, schemas, loading, and planner catalog |
54
- | `src/visual-system/scene-templates/` | Complete scenes the model may select |
55
- | `src/visual-system/backgrounds/` | Standalone background renderers |
56
- | `src/visual-system/motion/` | Animation functions and timing behavior |
57
- | `src/visual-system/theme/` | Color and design tokens |
58
- | `src/visual-system/typography/` | Text fitting, formatting, and kinetic type lifecycles |
59
- | `src/cli/` | `vanillasky init`, `doctor`, and `providers add`, plus `vanillasky templates create`, `add`, `sync`, `check`, `list`, and `describe` |
60
- | `registry/items/` | Generated distributable copies installed into customer projects |
61
- | `src/index.ts`, `src/server.ts`, `src/react.ts`, `src/templates.ts`, `src/template-catalog.ts`, `src/test.ts`, `styles/video-chat.css` | The six small code entry points and one scoped stylesheet |
62
-
63
- The source of truth for built-in visuals is `src/visual-system`. The JSON files
64
- in `registry/items` are distribution artifacts, kept flat so the CLI can address
65
- every installable item by a stable name. Their `meta.vanillasky.layer` field is
66
- `template` for a complete scene the model may select, or `lib` for shared
67
- support code a template imports. Run `npm run registry:sync` after changing
68
- canonical visual source.
69
-
70
- Customer applications do not edit those internal locations. Their source of
71
- truth is one file per visual under `vanillasky/templates/`; `vanillasky templates sync`
72
- derives a browser registry in `vanillasky/index.ts` and a React-free
73
- prompt/validation registry in `vanillasky/server.ts`.
74
-
75
- ## Request flow
76
-
77
- 1. `VideoChat` uses `useVideoChat` to send the user's message. A clicked
78
- suggestion can start its prewritten hook and existing media immediately.
79
- 2. For a typed prompt, the model emits a short hook first and then continues
80
- planning scenes in the same response stream. Its stock-search keyword is
81
- resolved separately while the hook is spoken.
82
- 3. `createVideoChatHandler(...)` owns chat actions, suggestions, narration,
83
- capability discovery, and generated-video budgets. The application supplies
84
- provider callbacks and policy.
85
- 4. Video answers flow into the internal composition pipeline, which calls the
86
- application's `streamText` adapter. This is where Anthropic, OpenAI, or
87
- another text model is connected.
88
- 5. The system prompt combines the opening and composition rules with the trusted template
89
- catalog, including generated metadata for customer-owned templates. The user
90
- prompt serializes the prompt, completed conversation, instructions,
91
- style, and approved media.
92
- 6. The model streams one host-consumed opening object followed by NDJSON plan
93
- parts. The server emits the opening event, then parses and validates complete
94
- scenes before emitting them.
95
- 7. The browser reduces those events into a deterministic `Video`. The opening
96
- loops or holds until its voice ends and the first playable scene is ready;
97
- the planned playback and narration then take over in one cut.
98
-
99
- `VideoPlayer` also replays completed chat videos parsed at the storage boundary.
100
-
101
- ## Where to change common behavior
102
-
103
- - Model connection: `src/server/create-video-handler.ts`
104
- - Base planning rules: `src/server/prompts/system-prompt.ts`
105
- - Per-request context formatting: `src/server/prompts/user-prompt.ts`
106
- - Template-specific prompt catalog: `src/visual-system/catalog/prompt.ts`
107
- - Wire contract: `src/protocol/types.ts` and `src/protocol/events.ts`
108
- - Scene rendering: `src/visual-system/scene-templates/`
109
- - Background effects: `src/visual-system/backgrounds/` and
110
- `src/visual-system/scene-templates/background-effect.ts`
111
- - Text effects: `src/visual-system/scene-templates/text-archetypes.ts`
112
- - Gradients and design tokens: `src/visual-system/theme/`
113
-
114
- ## Public vocabulary
115
-
116
- Use `VideoChat`, `useVideoChat`, `createVideoChatHandler`, `Video`, and `VideoPlayer` for the product API. Use “video response” when
117
- describing the lifecycle or output category. Use “motion” only for animation
118
- behavior inside the visual system.
40
+ | Chat interface and lifecycle | `src/video-chat/` |
41
+ | HTTP admission, methods, CORS and bounded body reading | `src/server/video-chat-http.ts` |
42
+ | Response orchestration, host answer and provider callbacks | `src/server/create-video-chat-handler.ts` |
43
+ | Default planning instructions | `src/server/video-chat-prompts.ts` |
44
+ | Answer brief and shot planning | `src/server/chat-shot-planner.ts` |
45
+ | Validated composition and completion | `src/server/compose-video.ts` |
46
+ | Wire contract and reduction | `src/protocol/` |
47
+ | Turn-owned speech queue and media preparation | `src/video-chat/scene-preparation.ts` |
48
+ | Media readiness, timeline and narration | `src/player/` |
49
+ | Shared speech/clip budget | `src/protocol/clip-budget.ts` |
50
+ | Footage and chapter rendering | `src/visual-system/scene-templates/` |
51
+ | CLI setup and doctor | `src/cli/` |
52
+ | Application-owned provider examples | `starters/video-chat/` |
53
+ | Local real-chat harness | `dev/chat/` |
54
+ | Public entry points | `src/index.ts`, `src/server.ts`, `src/react.ts`, `src/test.ts` |
55
+ | Scoped UI styles and packaged fonts | `styles/` |
56
+
57
+ The root and test entry points are React-free. Server imports must not pull in
58
+ React; browser entries must not pull in Node or provider libraries.
59
+ [PUBLIC-API.md](../PUBLIC-API.md) and the packed API checks enforce that boundary.
60
+
61
+ ## Prompts and ownership
62
+
63
+ The handler creates the real chat prompt from the opening, shot, pacing, and
64
+ visual-direction rules. Application `instructions` add product guidance;
65
+ the separate user prompt carries the request and bounded conversation.
66
+ The model emits structured directions, never React, HTML, or executable code.
67
+ Approved media URLs enter only through server callbacks.
68
+
69
+ Providers can return an async text iterable directly or an AI SDK-shaped result.
70
+ No particular host, model SDK, database, or storage vendor is required.
71
+ Authentication, spending limits, licensing, and persistent media delivery belong
72
+ to the application.
73
+
74
+ The optional `resolveAnswer({ prompt, conversation, signal })` accepts one
75
+ completed, nonempty string of at most 32,000 characters. It has a fixed,
76
+ abortable 30-second bound; invalid output or failure returns an explicit error,
77
+ not a newly invented answer. The video planner still needs a text callback to
78
+ turn that source into shots. See [integration](provider-integration.md).
79
+
80
+ The generated-video callback receives `requestedDurationSec`, `shotDirection`
81
+ and an absolute `deadlineAt`, alongside orientation, look and cancellation.
82
+ Its result can report `durationSec`. Model selection, supported duration and
83
+ resolution, concurrency, submission/polling and durable delivery stay in the
84
+ adapter. The SDK does not own provider jobs or a storage service.
85
+
86
+ Progressive scene delivery cannot remove a vendor's generation delay. A
87
+ minutes-long job API remains minutes-long even when the next scene is prepared
88
+ early. Scope is the beta npm SDK, its documented React/browser boundaries and
89
+ best-effort integration support—not a hosted video service or vendor uptime SLA.
90
+
91
+ See [development](development.md) for the fast edit loop and
92
+ [testing](testing.md) for deterministic public helpers.
@@ -1,25 +1,24 @@
1
- [← Documentation home](../README.md)
2
-
3
1
  # Customization
4
2
 
5
- ## Video chat interface
6
-
7
- The default `VideoChat` uses an immersive video canvas with a floating
8
- conversation field, on-video subtitles, and a single dark settings/history
9
- treatment. See [interface behavior](immersive-interface.md) and the
10
- [component reference](reference/design-system.html). Pass a custom welcome
11
- heading and a root class when the application needs its own copy or chrome colors:
3
+ The default interface places footage behind a floating conversation field,
4
+ with on-video subtitles, pause/mute controls, suggestions, settings, and history.
5
+ Keep that interface unless your application needs to own it.
12
6
 
13
7
  ```tsx
14
8
  <VideoChat
15
9
  className="acme-chat"
16
10
  welcomeTitle={<>Ask Acme<br />See the answer</>}
11
+ branding={{
12
+ name: "Acme",
13
+ logo: <img src="/acme-logo.svg" alt="" width={120} height={30} />,
14
+ homeUrl: "/app",
15
+ showDeveloperLinks: false,
16
+ }}
17
17
  />
18
18
  ```
19
19
 
20
- Override the scoped custom properties after importing
21
- `@vanillaskyai/video/video-chat.css`; the selectors and values stay local to
22
- that instance:
20
+ Import `@vanillaskyai/video/video-chat.css` once. Override its scoped variables
21
+ on the instance class, not global element selectors:
23
22
 
24
23
  ```css
25
24
  .acme-chat {
@@ -30,72 +29,56 @@ that instance:
30
29
  }
31
30
  ```
32
31
 
33
- The built-in navigation carries the VanillaSky logo. `welcomeTitle` changes
34
- the welcome heading; it does not replace the navigation logo. Graphic scenes
35
- use fixed black backgrounds and white/neutral system typography.
32
+ `welcomeTitle` changes the heading. `branding` changes only the navigation
33
+ identity: `name` supplies the accessible home-link label and becomes the visible
34
+ wordmark when `logo` is omitted. Size an app-owned image or React logo explicitly
35
+ to fit the header (about 120–144px wide and 30–36px high). Keep it non-interactive:
36
+ the surrounding home link already handles navigation.
37
+
38
+ Omit `branding` for the unchanged VanillaSky logo and interface. Omit `homeUrl`
39
+ even with custom branding to keep the existing Home behavior: at `/`, an ordinary
40
+ click starts a new session; elsewhere it navigates to `/`. An explicit `homeUrl`
41
+ navigates normally and never resets the current session first. Only root-relative
42
+ paths and HTTP(S) links without embedded credentials are accepted; other values
43
+ fall back to the existing Home behavior.
36
44
 
37
- Use `options` for the endpoint, templates, orientation, request
38
- headers, and an optional custom voice. Provider capabilities are discovered
39
- from the server. Use `useVideoChat()` only when the application needs to own the
40
- entire interface.
45
+ `showDeveloperLinks` defaults to true. It controls the SDK's Docs/About/GitHub
46
+ section in Settings, not an About page for your application. This is a small UI
47
+ option, not a theme system; it does not affect narration, providers or history.
41
48
 
42
- Pass the following visual settings through `VideoChat` or `useVideoChat` options.
43
- Keep viewer context in the prompt and completed conversation turns; use the
44
- server handler’s `instructions` for trusted product guidance.
49
+ The chapter
50
+ introduction uses black with neutral system typography. The supported scenes
51
+ are chapters and footage; there is no renderer-extension or brand-kit API.
45
52
 
46
- ## Cinematic visual direction
53
+ Use `options` for endpoint, request headers, orientation, and voice.
54
+ Capabilities are discovered from the server. `useVideoChat` provides the same
55
+ conversation/playback lifecycle for application-owned controls.
47
56
 
48
- Graphics use black backgrounds and white/neutral typography. Full-bleed media
49
- and Reach out can show naturally colored footage; the other six templates
50
- explain with their own composition and motion. There is no brand-kit option.
57
+ ## Visual direction
51
58
 
52
- The host can provide a shared `generatedLook` description for media preparation.
53
- Custom source-owned templates can define their own visual language in code.
54
- Do not rely on old global brand, text-effect, or gradient controls to restyle
55
- the seven cinematic templates.
59
+ Use server `instructions` for trusted audience, tone, or domain guidance.
60
+ A shared `style.generatedLook` can guide generated footage; the planner carries
61
+ consistent response-specific subjects and setting into each shot.
62
+ [Media and voice](media-and-audio.md) explains the look and adapter contract.
63
+ Style prompts cannot restyle stock assets.
56
64
 
57
- ## Opening
65
+ ## Opening and layout
58
66
 
59
- The planner streams a short spoken hook before the scenes. The chat holds that
60
- opening until its speech finishes and the first scene is ready. A selected
61
- suggestion can start with its prewritten opening and already-loaded media:
67
+ A selected suggestion can start its prepared opening immediately:
62
68
 
63
69
  ```ts
64
70
  await chat.ask(card.prompt, { opening: card.opening, openingMedia: card.media });
65
71
  ```
66
72
 
67
- Openings and scene narration share the chat voice and pause/mute controls.
68
-
69
- ## Aspect ratio and responsive layout
70
-
71
- The player is responsive by default: it fills its container width. Templates
72
- and copy must work at either aspect ratio; orientation is not an AI-planning
73
- input and must not influence the selected templates or wording.
74
-
75
- `portrait` reserves a 9:16 response/export frame and `landscape` reserves 16:9.
76
- This input setting remains stable in the completed config. For an embed that
77
- should display landscape on desktop and portrait on mobile without changing the
78
- saved response, pass `orientation="auto"` to `VideoPlayer`; it responds
79
- to its container width. See [responsive orientation](responsive-orientation.md).
80
-
81
- ## Media and voice
82
-
83
- Configure `searchMedia`, `generateVideo`, and `generateSpeech` on the server
84
- handler. They progressively enhance the same chat; failed optional providers
85
- fall back to templates or browser voice. See [Media and voice](media-and-audio.md).
86
-
87
- ## Custom templates
73
+ Typed prompts receive their opening from the same model stream as the answer.
74
+ The opening holds until its narration completes and the first scene is ready.
88
75
 
89
- The built-in catalog needs no setup. Only source-owned templates need the
90
- optional local TSX compiler; install it once with `npm install --save-dev tsx`.
91
- Then use `npx vanillasky templates create <id>` for an original one-file template or
92
- `npx vanillasky templates add <builtin>` to copy a close built-in. Edit the owned file,
93
- run `npx vanillasky templates sync`, then run `npx vanillasky templates check` before committing.
94
- Pass the generated registry to the server and browser; project-owned IDs
95
- replace matching built-ins and new IDs extend the catalog.
76
+ `portrait` reserves a 9:16 response frame; `landscape` reserves 16:9.
77
+ The saved orientation stays stable. For responsive display without changing the
78
+ saved response, `<VideoPlayer orientation="auto" />` follows container width.
79
+ Keep the player in a container with a usable width and height.
96
80
 
97
- The model sees selection guidance and a schema, not component source. It chooses
98
- a trusted template and fills validated variables. Never evaluate model-authored
99
- React, HTML, CSS, or JavaScript on the live path.
81
+ History in `VideoChat` is in memory. Durable storage belongs to the host;
82
+ see [persistence](persistence.md).
100
83
 
101
- See [custom templates](custom-templates.md).
84
+ [Documentation home](../README.md)