@vanillaskyai/video 0.5.8 → 0.7.0

package/docs/production.md

@@ -2,121 +2,110 @@
 
 # Production guide
 
-Use this checklist before serving a video response to customers.
+Use one `createVideoChatHandler` endpoint for capabilities, welcome content,
+video responses, speech, transcription, stock search, and follow-up prompts.
+Mount `<VideoChat />` or `useVideoChat` against that boundary.
 
 ## Server boundary
 
-- Keep provider keys, the system prompt, and tools on the server.
-- Authenticate the user and tenant before reading the request body.
-- Set an explicit origin allowlist. CORS is not authentication.
-- Apply per-user and per-tenant request, token, and concurrency limits.
-- Set route, model, media, and export timeouts with cancellation propagation.
-- Bound raw input size, media count, scene count, and maximum duration.
-
-Use `createVideoHandler()` for validated SSE and read
-[the security guide](security.md) for the mandatory controls.
-The handler automatically configures scene validation: it rejects unknown
-templates and fields, missing required variables, non-supplied media URLs, and
-fabricated quote-template content before a scene reaches the player.
-Applications with a custom stream adapter may authorize additional final URLs
-with `allowMediaUrl`; the callback validates URLs and does not resolve them.
-Invalid generated parts are dropped by default, after `onError` receives their
-private reason. Accepted scenes continue streaming with a safe recoverable
-diagnostic. Use `invalidPartBehavior: "fail"` only for deliberate fail-fast behavior.
+- Keep provider keys, planner prompts, media tools, and raw errors on the server.
+- Replace the generated localhost authorization with a real user and tenant check.
+- Authenticate before reading the request body.
+- Set an explicit origin allowlist; CORS is not authentication.
+- Apply per-user and per-tenant request, token, concurrency, and spend limits.
+- Bound prompt size, conversation turns, audio bytes, scene count, and duration.
+- Forward cancellation to every text, speech, media, transcription, and video provider.
 
-## Data and privacy
+The handler rejects unknown templates and fields, invalid variables, unsafe
+media, and fabricated quote-template content before a scene reaches the
+player. Read the [security guide](security.md) for the complete controls.
 
-- Send the provider only the source and personalization required for the story.
-- Do not log raw prompts, personalization, authorization headers, signed URLs,
-  or provider deltas.
-- Record request IDs, model ID, duration, event counts, safe error codes, token
-  usage, and acceptance metrics.
-- Review your provider's retention settings and data-processing terms.
-- Keep signed asset URLs valid for the expected viewing and replay window.
-
-## Reliability
-
-- Emit the supplied opening and selected audio before starting model work.
-- Abort provider work when the client disconnects.
-- Persist terminal snapshots for replay and export, then validate loaded values
-  with `parseVideo` before use.
-- Persist event logs when resume is required; validate them with
-  validate stored event logs before replay.
-- Treat the [persistence contract](persistence.md) as the storage boundary;
-  database, tenant policy, object storage, deletion, and URL expiry remain
-  host-owned.
-- Use idempotency keys around durable generation requests.
-- Retry only before visible output or from a validated resume point. Do not
-  silently restart a response after the viewer has begun watching.
+## Visual modes and providers
 
-## Failure experience
+Keep templates available on every deployment. They are the reliable fallback
+when stock, speech, or generated-video providers are missing or late. Add
+`searchMedia` for approved stock footage and `generateVideo` for the separate
+full AI video mode. Do not expose a partial generated-video mode.
+
+Use explicit provider deadlines. Generated video should use idempotency keys
+and `maxRetries: 0` so one visible action cannot silently create several
+billable clips. Validate returned URLs, media types, byte sizes, duration, and
+licensing before use.
 
-- Keep private provider details in `onError`; send only safe typed errors.
-- Do not display stack traces or protocol diagnostics inside the video.
-- Hold the current scene and continue audio through a recoverable generation
-  gap.
-- End cleanly on a terminal error; never append blank media after completion.
-- Provide a normal application retry control outside the player.
+## Fast first response
 
-## Media and audio
+The planner's first streamed object supplies the spoken hook and media keyword.
+Start speech immediately, resolve stock in parallel, and keep the opening
+playing until the first narrated scene and its media are ready. Welcome cards
+should carry a prepared hook and preloaded footage so they can start without a
+model round trip.
 
-- Start generated playback with an asset-free scene.
-- Preload media for the next scene and commit it only when ready.
-- Resolve media before generation and pass approved results through
-  `suppliedMedia`. Stock-keyword resolution is not part of the 0.1 handler.
-- Use customer-approved media domains and enforce type/byte limits.
-- Select soundtrack audio from an already-loaded catalog; declare a positive
-  fade-out. Narration and speech synchronization are application-owned.
-- Respect browser autoplay rules and provide an explicit unmute control.
+Do not wait for the complete plan before showing the first validated scene.
+Preload upcoming assets and keep the current visual if the next one is late.
+For full AI video, reserve the first shot in the opening object so generation
+can begin while the planner streams later scenes.
 
-## Observability
+## Data and privacy
+
+- Send only the prompt, bounded conversation, and context needed for the answer.
+- Do not log authorization headers, secret values, raw prompts, signed URLs, or provider deltas.
+- Review provider retention and data-processing terms.
+- Keep stored media URLs valid for the expected replay window.
+- Validate persisted `Video` values with `parseVideo` before replay.
 
-Track at minimum:
+## Failure experience
 
-- time to supplied opening;
-- time to first generated scene;
-- time to complete plan;
-- seconds of future content buffered;
-- planner and protocol error codes;
-- media readiness failures;
-- completion and abandonment rate;
-- provider model and token usage.
+- Keep private diagnostics in `onError`; expose only safe typed errors.
+- Treat late media as a fallback case, not a reason to freeze playback.
+- Keep the current scene and its narration during recoverable generation gaps.
+- End cleanly on a terminal error and show an application retry control.
+- Retry only before visible output or from an explicit validated resume point.
 
-Use the repository acceptance harness in smoke tests. The defaults
-require an opening within 250 ms, a generated scene within 15 seconds,
-completion within 30 seconds, resolved media, audio before the opening, three
-body scenes, three distinct templates, and a human quality score of 80.
+## Measure the stages
 
-## Testing
+Record safe server timing and quality fields rather than one opaque total:
 
-Use the React-free deterministic helpers in [Test integrations](testing.md) for
-Vitest, in-process streaming, and route-handler tests without a live provider.
+- hook received;
+- speech ready;
+- first scene planned;
+- first scene media ready;
+- first frame painted;
+- plan complete;
+- accepted and rejected scene counts;
+- provider model, finish reason, token usage, and media failures.
 
-Test the public path at its HTTP boundary. Pass a deterministic `streamText`
-generator to `createVideoHandler`, call the route with grounded input, and
-assert the validated terminal result through `useVideo`. Keep separate
-component tests for each trusted template.
+Use `onComplete` for server-side cost and quality reporting and `onWarning` for
+bounded recoverable issues. Never send provider-native metadata to the browser
+unless the application has deliberately enabled and secured it.
 
-In CI, build and test the application. If the project owns copied templates,
-also verify that its registry is current:
+## Test the shipped path
+
+Test the route with deterministic `streamText` and `generateText` callbacks.
+Exercise `capabilities`, `welcome`, and `response`, then assert the rendered
+conversation through `VideoChat`. Keep provider-adapter tests keyless and run a
+small, explicitly gated real-provider smoke test before a release.
+
+In CI, build one clean consumer from the packed SDK artifact. This catches
+missing exports, server/browser boundary leaks, code-generation drift, and
+dependency-resolution problems that workspace tests miss. If the application
+owns copied templates, also run:
 
 ```bash
-npx vanillasky sync --check
+npx vanillasky templates sync --check
 npm run build
 npm test
 ```
 
-Before release, build one clean consumer from the packed SDK tarball. This
-catches missing exports, React/server boundary leaks, code-generation drift,
-and dependency-resolution problems that workspace tests cannot detect.
-
 ## Deployment checklist
 
-- [ ] Provider keys exist only in the server secret store.
+- [ ] Keys exist only in the server secret store.
 - [ ] Authentication, tenant policy, rate limits, and origin allowlist are live.
-- [ ] Cancellation, timeouts, and safe errors are tested.
-- [ ] Every installed template renders in both orientations.
-- [ ] Chromium, Firefox, WebKit, React, and Node compatibility checks pass.
-- [ ] A real provider run passes latency and human quality review.
-- [ ] Final snapshots replay exactly and export through the configured adapter.
-- [ ] Package and application dependency audits meet your severity policy.
+- [ ] Cancellation, timeouts, fallbacks, and safe errors are tested.
+- [ ] Template mode completes when every optional provider is unavailable.
+- [ ] Full mode appears only when generated video is configured.
+- [ ] Both orientations render and narration stays synchronized.
+- [ ] A packed-artifact consumer and deterministic browser chat pass.
+- [ ] One bounded real-provider run meets the product's latency and quality target.
+- [ ] Stored responses replay exactly and application-owned export is verified.
+
+[← Documentation home](../README.md) · [Previous: Errors and recovery](errors.md)

package/docs/prompt-and-input.md

@@ -1,113 +1,63 @@
 [← Documentation home](../README.md) · [Next: Generate your first video →](getting-started.md)
 
-# Prompt and input
+# Prompt and conversation input
 
-VanillaSky separates application truth from visual direction. You provide the
-facts and your server-side model. VanillaSky supplies the planner contract that
-turns those facts into a finite sequence of trusted scenes.
+VanillaSky turns the same things people ask an AI chat into spoken video
+answers. The SDK owns the video-planning prompt and conversation formatting;
+the application owns the model, product guidance, authentication, and data.
 
-## The four layers
+## What the viewer sends
 
-### 1. VanillaSky system prompt
+`<VideoChat />` and `useVideoChat` send the current prompt plus a bounded set of
+earlier turns to one `/api/video-chat` endpoint. Prompts can ask for an
+explanation, story, recommendation, ad, recap, or any other general-purpose AI
+response. Do not put provider keys, private policy, or unrelated personal data
+in the conversation.
 
-`createVideoHandler` generates `systemPrompt` from the trusted template
-registry. It tells the model:
-
-- which templates and variables exist;
-- how to emit the typed plan;
-- how to order and time complete scenes;
-- how to stay inside the factual and media boundaries;
-- how to finish a finite response.
-
-Normal applications should not build, copy, or expose this prompt in browser
-code. Pass it unchanged to the provider adapter.
-
-For maintainers, the base system rules live in
-`src/server/prompts/system-prompt.ts`, request-context formatting lives in
-`src/server/prompts/user-prompt.ts`, and template-specific catalog guidance
-lives in `src/visual-system/catalog/prompt.ts`. See the
-[architecture map](architecture.md) for the complete request flow.
-
-### 2. Application instructions
-
-Use `instructions` for optional presentation direction:
+When using the custom hook, ask in plain language:
 
 ```ts
-video.generate({
-  input: "Account alerts launched today. They refresh every 15 minutes.",
-  instructions: "Make the launch feel direct and energetic. End with adoption.",
-});
+await chat.ask("Pitch a playful ad for a coffee mug that never spills");
 ```
 
-Instructions can influence selection, emphasis, ordering, tone, and pacing.
-They never change `knowledgeMode`, authorize a new media URL, or weaken the
-event and validation contract.
-
-For durable product-wide direction, use the server handler's `basePrompt`.
-Keep per-request creative direction in `instructions`.
-
-### 3. Input and knowledge mode
-
-`input` is required. With the default `knowledgeMode: "input-only"`, it is the
-complete factual source for the video. It may be plain text or a serialized
-structured object:
+A selected welcome or follow-up card can also include a prepared opening line
+and already-loaded media:
 
 ```ts
-video.generate({
-  input: "Activation increased from 41% to 58% after guided onboarding.",
+await chat.ask(card.prompt, {
+  opening: card.opening,
+  openingMedia: card.media,
 });
 ```
 
-```ts
-video.generate({
-  input: JSON.stringify({
-    period: "Q2",
-    activation: { previous: 41, current: 58 },
-    cause: "guided onboarding",
-  }),
-});
-```
+That path starts immediately. A typed prompt instead receives its short spoken
+hook and media keyword from the beginning of the planner stream.
 
-Include exact numbers, quote wording, attribution, names, dates, and product
-facts that may appear on screen. Do not place secrets, provider keys, or hidden
-policy in input.
+## Application guidance
 
-For a chat question or a request to develop content, opt in explicitly:
+Use the server handler's `instructions` option for durable product direction:
 
 ```ts
-video.generate({
-  input: "How can a small team improve customer onboarding?",
-  knowledgeMode: "general",
+createVideoChatHandler({
+  authorize: verifySession,
+  streamText: planWithYourModel,
+  generateText: runSmallTextTask,
+  instructions: [
+    "Speak like a warm, concise creative partner.",
+    "Prefer concrete examples over abstract explanations.",
+  ].join(" "),
 });
 ```
 
-General mode permits stable model knowledge. The generated system prompt still
-forbids invented citations, quotations, URLs, personal details, live facts,
-guarantees, and precise claims that require a source. Claims already present in
-`input` remain authoritative.
-
-`personalization`, `brand`, and `suppliedMedia` are separate structured context.
-They do not replace the source material.
-
-### 4. Streamed plan
-
-Your provider streams text deltas. VanillaSky decodes them into typed planner
-parts, validates each complete scene, and emits deterministic protocol events.
-The model never returns React, HTML, CSS, or executable JavaScript.
-
-Every generated `scene.add` is complete and passes validation before the player
-sees it. Emitted scenes are immutable.
-
-The standard planner contract requires one `scene.add` with
-`placement: "closer"` immediately after the first playable body scene. The
-model writes short grounded conclusion copy: a supplied action when one
-exists, otherwise a declarative payoff that answers the story's “so what.” The
-runtime holds the closer and emits it last, so a long body plan cannot displace
-an ending that was already generated.
+This can define a character, audience, subject area, tone, or answer style. It
+does not change the protocol, authorize media, or weaken validation. Keep the
+viewer prompt separate from these trusted server-side instructions.
 
-## What reaches the LLM
+## What reaches the model
 
-The provider adapter receives:
+`createVideoChatHandler` builds the trusted template catalog, video rules,
+conversation context, and application guidance. Your provider adapter receives
+two complete strings:
 
 ```ts
 streamText: ({ systemPrompt, userPrompt, signal }) => streamText({
@@ -115,94 +65,54 @@ streamText: ({ systemPrompt, userPrompt, signal }) => streamText({
   system: systemPrompt,
   prompt: userPrompt,
   abortSignal: signal,
-})
-```
-
-`userPrompt` is assembled by VanillaSky from:
-
-- orientation and maximum duration;
-- whether a deterministic opening scene already exists or the host is waiting
-  for the first generated scene;
-- raw `input`;
-- creative `instructions`;
-- personalization;
-- descriptions and opaque references for approved supplied media;
-- brand context.
-
-Provider credentials and application authentication never belong in either
-prompt. Original supplied-media URLs and data URIs also remain outside the
-model prompt; the server restores an exact SDK-issued opaque reference only
-after provider output has been parsed.
-
-Generated template values are validated exactly. VanillaSky does not silently
-truncate factual labels to satisfy a layout schema because truncation can drop
-qualifiers or change meaning. A rejected generated scene contributes to
-`rejectedSceneCount`; hosts can use the completion quality fields to retry with
-a stronger model or revised source.
-
-## Input examples
-
-### Product update
-
-```ts
-video.generate({
-  input: "Account alerts launched on August 16. They refresh every 15 minutes, filter by segment, and are available to all plans.",
 });
 ```
 
-### Metrics recap
+Pass both strings unchanged. The system prompt describes the installed
+templates, schema limits, pacing, narration, opening contract, and safe media
+fields. The user prompt contains the current request, bounded prior turns,
+orientation, visual mode, and whether an opening was already spoken.
 
-```ts
-video.generate({
-  input: JSON.stringify({
-    period: "Q2",
-    customerConversations: 142,
-    escalationsResolved: "96%",
-    improvementsLaunched: 4,
-  }),
-  personalization: { role: "Product leader", focus: "activation" },
-});
-```
-
-### Grounded review
-
-```ts
-video.generate({
-  input: 'Review by Maya Chen, VP Product: "Setup took minutes, not weeks." Rating: 5/5.',
-});
-```
+Provider credentials and raw media URLs never belong in either prompt. Media
+callbacks restore approved URLs on the server only after the model's structured
+output has been parsed.
 
-Every visible quote must occur in the input exactly. Attribution and ratings
-should be explicit.
+## One stream, one answer
 
-### Article or long source
+The planner first emits a 6–9 word spoken hook and a media keyword, then keeps
+streaming complete scenes. It is not a separate hook call followed by a second
+planning call. This keeps the opening consistent with the scenes that follow
+and lets the first playable scene arrive without waiting for the complete plan.
 
-Pass the source text and let the planner select the most decision-relevant
-takeaways that fit the duration. The planner summarizes; it should not attempt
-to place every paragraph on screen.
+Every `scene.add` is validated before the browser receives it. The model never
+returns React, HTML, CSS, or executable JavaScript. Accepted scenes are
+immutable; invalid scenes are reported through safe warnings and omitted.
 
-## Grounding and media safety
+In template mode, the planner can choose any trusted template and request stock
+media through `searchMedia`. In full mode, it plans a complete generated-video
+answer and `generateVideo` resolves every visual beat. There is no mixed
+"generate a few clips" mode.
 
-- Numeric templates require real numbers present in the source.
-- Every grounded quote value must occur in the source, not merely one quote in a scene.
-- Screenshot fields require an exact supplied image.
-- Media URLs must be supplied or approved by the server's `allowMediaUrl` policy.
-- Patches are validated after merging with the existing scene.
+## Grounding
 
-These checks happen at runtime. Prompt instructions improve model behavior but
-are not treated as a security boundary.
+General chat permits stable model knowledge, but it still forbids invented
+citations, quotations, URLs, personal details, live facts, and guarantees. If
+exact numbers, names, dates, or wording matter, include them in the prompt or
+conversation. Use retrieval in the application before calling VanillaSky when
+the answer depends on private or current data.
 
-## Debugging
+## Debugging weak answers
 
-If the plan is rejected or the result is weak, inspect the boundary in this order:
+Check these boundaries in order:
 
-1. **Input:** Does it contain the exact facts, numbers, quotes, and attribution?
-2. **Instructions:** Are they presentation guidance rather than new claims?
-3. **Provider adapter:** Does it pass both prompts unchanged and stream only text deltas?
-4. **Finish reason:** Did the provider report `length`, a content filter, or an execution error?
-5. **Template fit:** Does the trusted registry contain a suitable visual for the requested story?
+1. Does the prompt contain the exact facts the answer needs?
+2. Is `instructions` concise product guidance rather than extra source data?
+3. Does the provider pass `systemPrompt` and `userPrompt` unchanged?
+4. Is extended reasoning delaying the first streamed object?
+5. Do `onWarning` and `onComplete` show rejected scenes or a length limit?
+6. Does the trusted registry contain a suitable template for the requested answer?
 
-Log request IDs, provider finish reasons, and terminal SDK status. Do not log
-secrets or expose raw provider diagnostics inside the video.
+Log request IDs, safe warning codes, provider finish reasons, model IDs, and
+token usage. Never log credentials or expose raw provider errors in the video.
 
 [← Documentation home](../README.md) · [Next: Generate your first video →](getting-started.md)