@vanillaskyai/video 0.10.23 → 0.11.1

package/docs/provider-integration.md

@@ -1,197 +1,172 @@
-[← Documentation home](../README.md) · [Previous: Getting started](getting-started.md) · [Next: Customization →](customization.md)
-
 # Provider integration
 
-Start from the generated chat so provider work stays confined to the
-application-owned server:
-
-```bash
-npx @vanillaskyai/video init
-npx vanillasky doctor
-npm run dev
-```
-
-Init runs doctor automatically. The generated `server.ts` starts with one
-`ANTHROPIC_API_KEY`, a template introduction, and browser voice; it installs no
-optional speech or video packages.
+Follow [Getting started](getting-started.md) for setup. Keep one `VideoChat`
+client and one `createVideoChatHandler` endpoint. Models, credentials,
+authentication, retrieval, tools, storage and spending belong to the application;
+the SDK owns shot planning, validation, streaming and playback.
 
-Use `npx vanillasky providers add speech` to install xAI speech, or
-`npx vanillasky providers add video` to install FAL video and transcription.
-Add `XAI_API_KEY` or `FAL_KEY` to `.env.local` and restart the server. Stock media
-needs only `PEXELS_API_KEY`, with no extra installation. These app-owned
-adapters advertise their capabilities automatically; `src/main.tsx` does not
-change. Rerun an interrupted setup command to finish installation.
+The generated `providers/text.ts` is an editable Vercel AI SDK example.
+`init --native` instead installs native Gemini REST callbacks with no `ai` or
+`@ai-sdk/anthropic` dependency. Neither integration is required by the core.
+Any native API, vendor SDK or application service can implement the callbacks.
 
-For the full chat experience, mount one `createVideoChatHandler` and keep every
-provider choice in its callbacks:
+## Text callbacks
 
-```ts
-import "server-only";
-import { anthropic } from "@ai-sdk/anthropic";
-import { generateText, streamText } from "ai";
-import { createVideoChatHandler } from "@vanillaskyai/video/server";
+`streamText` receives the SDK's `systemPrompt`, `userPrompt` and `signal`.
+Return an `AsyncIterable<string>` directly, or an AI SDK-shaped object with a
+`textStream`. The stream contains the model's answer brief and shot records,
+not React components or pre-generated MP4 bytes.
 
-export const handler = createVideoChatHandler({
-  authorize: verifySession,
-  streamText: ({ systemPrompt, userPrompt, signal }) => streamText({
-    model: anthropic(process.env.TEXT_MODEL ?? "claude-sonnet-5"),
-    system: systemPrompt,
-    prompt: userPrompt,
-    abortSignal: signal,
-  }),
-  generateText: async ({ systemPrompt, userPrompt, maxOutputTokens, signal }) => {
-    const result = await generateText({
-      model: anthropic(process.env.TEXT_MODEL ?? "claude-sonnet-5"),
-      system: systemPrompt,
-      prompt: userPrompt,
-      maxOutputTokens,
-      abortSignal: signal,
-    });
-    return result.text;
-  },
-});
-```
-
-That one text provider gives the browser templated video responses and local
-browser speech. Supplying `generateSpeech`, `transcribe`, `searchMedia`, or
-`generateVideo` enables those capabilities automatically. The callbacks are
-structural and provider-neutral; their SDKs and credentials remain application
-dependencies and never enter the browser bundle.
-
-The planner emits a short spoken opening, then continues the answer in the same
-stream. The default UI shows a chapter immediately. A welcome card can carry a
-prewritten `opening`, whose narration starts without waiting for the model.
-Each authored body beat prepares speech and selected footage together. Playback
-starts after its contiguous preparation cushion, without requiring the entire
-plan or a second model call for narration.
-
-The matching complete React interface is one component and one scoped style
-import:
-
-```tsx
-import { VideoChat } from "@vanillaskyai/video/react";
-import "@vanillaskyai/video/video-chat.css";
-
-export function App() {
-  return <VideoChat />;
-}
-```
+`generateText` handles bounded helper tasks. Honor its `systemPrompt`,
+`userPrompt`, `maxOutputTokens` and `signal`; return a string. If you dispatch by
+`task`, handle `narration`, `narration-rewrite` and `suggestions`. The rewrite
+task is an intentional part of the speech/clip budget, not a second answer.
 
-## Custom interface
+Model IDs, reasoning effort, sampling, prompt caching and provider timeouts
+remain in the adapter. Measure first usable scene and answer quality, not only
+time to first token. Keep credentials and raw provider metadata server-side.
+See the [adapter reference](reference/provider-adapters.md) for lower-level
+text stream and completion fields.
 
-For a custom interface, use `useVideoChat` and render its `turns`, `welcome`,
-`suggestions`, `caption`, and `status`; the hook owns their network and playback
-lifecycle. Pass a selected card through
-`chat.ask(card.prompt, { opening: card.opening })` to start its hook immediately.
-Custom interfaces can still pass and render `openingMedia`; the default UI uses
-the chapter. Typed prompts receive their authored opening through the stream.
+## Use an existing assistant
 
-Any AI SDK `LanguageModel` works in both `streamText` and `generateText`. Keep
-selection in one server-only module when an application supports several text
-providers. The chat route and React component stay unchanged; only the model
-passed to those callbacks changes. The AI SDK result can be returned directly:
-its text stream, finish reason, usage, warnings, and response metadata match the
-structural callback contract. See the
-[provider adapter reference](reference/provider-adapters.md) for native provider
-alternatives.
-
-## Planning effort and reasoning modes
-
-Planning is a structured emit against a trusted catalog, not a reasoning task.
-Where a provider exposes a reasoning or effort control, a host that wants a
-video to start quickly should turn extended reasoning off and keep effort low
-to moderate. The default matters: several current models reason by default, and
-that reasoning happens before the first plan part is emitted, so it is added
-directly to time to first generated scene.
-
-With the Vercel AI SDK and a current Anthropic model, that is one option object:
+Add `resolveAnswer` when your application already decides what the assistant
+should say. For example, in your generated `server.ts`:
 
 ```ts
-streamText: ({ systemPrompt, userPrompt, signal }) => streamText({
-  model,
-  system: systemPrompt,
-  prompt: userPrompt,
-  abortSignal: signal,
-  providerOptions: {
-    anthropic: { thinking: { type: "disabled" }, effort: "medium" },
-  },
-}),
-```
-
-Reasoning settings can substantially affect startup latency. Measure them with
-your installed catalog and representative requests. Compare first-scene timing,
-rejected scenes and factual accuracy; the fastest token stream is not useful if
-its scenes cannot be rendered. Keep these settings in the provider adapter.
-
-VanillaSky never sets these controls. Provider selection, sampling parameters,
-and credentials stay with the application.
-
-## Completion and usage
-
-Use `onComplete` for server-side cost and completion measurement:
+import { createVideoChatHandler } from "@vanillaskyai/video/server";
+import { textProvider } from "./providers/text";
+import { providers } from "./providers";
+import { answerQuestion } from "./assistant";
+import { verifySession } from "./auth";
 
-```ts
-createVideoChatHandler({
+export const handleVideoChat = createVideoChatHandler({
   authorize: verifySession,
-  streamText: ({ systemPrompt, userPrompt, signal }) => streamText({
-    model,
-    system: systemPrompt,
-    prompt: userPrompt,
-    abortSignal: signal,
-  }),
-  generateText: runSmallTextTask,
-  onWarning: (warning) => logSafeWarning(warning.code, warning.category),
-  onComplete: (summary) => recordGeneration({
-    finishReason: summary.finishReason,
-    usage: summary.usage,
-    requestedModelId: summary.requestedModelId,
-    resolvedModelId: summary.resolvedModelId,
-    totalDurationMs: summary.totalDurationMs,
-  }),
-  onError: (error) => recordPrivateFailure(error),
+  ...textProvider,
+  ...providers,
+  resolveAnswer: ({ prompt, conversation, signal }) =>
+    answerQuestion({ prompt, conversation, signal }),
 });
 ```
 
-`onComplete` fires once only after `response.complete`. It does not fire for a
-terminal error, abort, disconnect, or timeout. Callback failures are isolated
-from the event stream. Normalized token usage and model IDs remain server-only;
-they never enter SSE or the persisted `Video`. Set `includeRawProviderData:
-true` only when the host deliberately needs bounded provider-native usage and
-metadata and has an appropriate retention policy.
-
-`acceptedSceneCount`, `rejectedSceneCount`, and `timeToFirstSceneMs` describe
-model-generated scene additions; the streamed opening hook is not counted.
-Their sum is the proposed scene count. `videoDurationSec` is the duration
-actually committed. These fields provide a server-side quality signal without
-exposing model metadata in the browser.
-Warnings include the same bounded typed warnings emitted to the client.
-`plan_incomplete` identifies a playable partial response whose planner reported
-a length limit; applications should show that result as incomplete and may
-offer a bounded retry with a larger output or duration budget.
-`plan_missing_closer` identifies a playable answer that ended without its
-explicit final scene. For non-interactive evaluation, define an application
-threshold and retry a bounded number of times. Keep the best accepted result
-rather than treating `finishReason: "stop"` alone as a quality score.
-
-The generated system prompt includes the selected trusted-template catalog and
-is intentionally substantial. It is stable for the same SDK version, template
-kit, media policy, and base prompt. Record input-token usage, keep the selected
-kit no broader than the product needs, and enable provider-side prompt caching
-where the chosen provider/model supports it. VanillaSky does not assume one
-provider's cache controls in its provider-neutral adapter. Default chat streams an answer brief and shot directions rather than the template catalog; use provider-reported token
-usage as the authoritative measurement rather than a character estimate.
-
-Provider finish reasons `error` and `tool-calls` are terminal failures.
-`length` and `content-filter` may complete with already accepted scenes; a
-truncation before the first generated scene fails instead of returning an empty
-success. The request signal is forwarded to the provider. Configure route and
-provider timeouts with that signal, and keep retries host-owned and within the
-same explicit request budget.
-
-## Product-level planner guidance
-
-`createVideoChatHandler` constructs the planner prompt from the trusted template
-registry. Normal integrations do not build prompts or capabilities. Use the
-handler's `instructions` option for durable product-level direction such as a
-character, audience, domain, or answer style. The current user prompt and
-bounded prior turns are supplied separately by the SDK.
+`answerQuestion` and `verifySession` above are functions supplied by **your app**.
+Keep tenant lookup, retrieval, tool execution and answer policy there. The
+callback receives the validated prompt, bounded prior conversation and an
+abort signal. It must return one completed, nonempty string of at most 32,000
+characters after trimming—not an event stream, agent object or partial answer.
+
+The SDK waits up to 30 seconds and forwards cancellation. A failed, empty,
+oversized or timed-out answer returns HTTP 502 with `answer_unavailable`;
+request cancellation returns `aborted`. It does not silently replace your
+assistant's failure with a different answer. Provider work that ignores the
+signal may continue, so your assistant adapter must honor it.
+
+The completed answer becomes the planner's sole factual source in input-only
+mode. Planning still uses `streamText` to present that source as video; it is
+not a second retrieval/tool run or a fact-checking guarantee. The video plan
+streams after the answer has completed. Without `resolveAnswer`, the normal
+planner answers from the prompt and conversation directly.
+
+## Video, speech and transcription
+
+`generateVideo`, `searchMedia`, `generateSpeech` and `transcribe` are separate
+optional callbacks. Their availability advertises the relevant capabilities;
+changing providers does not require a new React interface.
+
+`providers add video fal`, `google` or `runway` copies an app-owned REST reference
+into `providers/video.ts`. `custom` supplies a callback skeleton. These names
+are onboarding examples, not a core allowlist. Keep any other SDK dependency
+inside your application's adapter.
+
+Setup never overwrites an edited adapter. To switch an existing vendor, edit
+`providers/video.ts` and the app manifest's `vanillasky.videoVendor` deliberately;
+keep duration, timeout, concurrency and delivery settings together. No client
+edit or SDK release is needed. Doctor follows that selected configuration.
+
+The video callback receives the visual query plus `requestedDurationSec`,
+`shotDirection`, orientation, `generatedLook`, `signal`, and an absolute
+epoch-millisecond `deadlineAt`. Return browser-safe media such as
+`{ type: "video", url, durationSec }`; report the actual delivered duration when
+known. Keep `generatedClipDurationSec`, `mediaConcurrency` and
+`generateVideoTimeoutMs` aligned with the adapter's model, supported duration,
+resolution and account limits. A model-name change alone is not always enough.
+
+The planner targets speech ending at least 0.8 seconds before each clip ends.
+First-pass writing leaves additional headroom: a five-second clip targets six
+ordinary words and one distinct idea, while repair can use up to eight words
+when needed for meaning. These are authoring guides, not guarantees from a
+text or voice model; the duration checks remain authoritative.
+Compact numeric measurements get a conservative expansion estimate. Authoring
+and repair request spoken numbers and units so short notation cannot conceal
+long speech; the SDK does not translate or alter the provider's spoken text.
+An oversized beat gets at most one bounded `narration-rewrite` call before
+footage generation. If the rewrite fails or still cannot fit, no video job is
+submitted for that beat: its complete original narration plays over a chapter.
+Rewrites are instructed to preserve facts and qualifications; applications
+should still evaluate meaning and timing with their actual models and voices.
+Measured speech can overrun the estimate; playback recovers to a chapter
+rather than looping or cutting off the sentence. Requested duration constrains
+the paid submission; a valid returned duration describes the footage actually
+available for playback. The mounted decoder also checks its physical duration.
+Without reported duration, generated footage keeps its requested budget.
+
+Stock search has its own bounded lookup deadline and no generated-video duration
+cap. Return `durationSec` when known: the SDK selects footage first, then checks
+the spoken beat against that duration. Unknown stock duration is checked by the
+mounted decoder, not replaced with an unrelated video vendor's clip setting.
+Neither path submits another video job to make narration fit.
+
+`onDiagnostic` includes a `narration-rewrite` phase with elapsed work time, clip
+budget and a fixed `rewritten`, `empty`, `oversized`, `timeout`, `provider-error`
+or `cancelled` reason. It never includes the original or rewritten text. Keep
+normal rewrite latency within its 2.5-second bound; shortening the first-pass
+plan avoids that additional call in the common path.
+
+Speech setup uses the optional xAI/AI SDK adapter. Transcription setup uses
+Whisper via fal REST independently of the selected video vendor. Stock footage
+uses `searchMedia`; AI-video mode never silently calls stock, and stock mode
+does not spend on generated video.
+
+## Video delivery and cancellation
+
+Direct references require an app-owned delivery callback. The starter's
+`providers/video-delivery.ts` receives downloaded video bytes, job ID, duration
+and signal. Replace it with your storage code, or implement its example upload
+endpoint: `PUT` MP4 bytes to `VIDEO_UPLOAD_URL` with `VIDEO_STORAGE_TOKEN`, then
+return a public HTTPS `{ url }`. VanillaSky does not host that endpoint.
+
+Google video downloads require a private API key. Keep it server-side, strip
+credentials when following off-origin redirects, and never pass the private
+provider URI to the player. Copy temporary vendor outputs into storage with an
+appropriate replay lifetime. Deliver H.264 MP4 with the moov atom first,
+Content-Type/Content-Length, byte ranges and CORS. Access rules, retention and
+deletion remain application responsibilities.
+
+The references submit each paid job once, retain its ID and poll to a deadline.
+Persist the ID through the helper's `onSubmitted` callback if jobs must survive
+server restarts. Never automatically resubmit an ambiguous network failure;
+check the provider dashboard first. Cancellation is best effort for fal and
+Runway. Gemini Veo has no documented cancellation operation: accepted work may
+finish and be billed even after local polling stops. No cancellation promises
+a refund.
+
+Initial policies are fal: 5s/480P, concurrency 3, 120s deadline; Google:
+6s/720p, concurrency 2, 360s; Runway: 5s/720p, concurrency 2, 180s. These are
+editable settings, not measured latency guarantees. Google and Runway may take
+minutes. Early media preparation enables progressive scene playback, not
+real-time streaming from a provider that only returns completed jobs.
+
+## Completion and integration scope
+
+Use `onComplete` for successful server-side completion and usage summaries;
+it does not fire for terminal failure or cancellation. Use safe diagnostics
+and `onError` for failures without exposing raw provider payloads to browsers.
+Stored completed responses can be validated with `parseVideo` and replayed
+without another generation.
+
+This is a beta npm SDK with [best-effort support](../SUPPORT.md), not a hosted
+generation service or a guarantee for every vendor/model combination. Offline
+fixtures establish callback contracts; live model availability, output quality,
+cost and latency require an explicitly budgeted check in your own account.
+See [production](production.md) before exposing the endpoint publicly and
+[customization](customization.md) for branding or application-owned controls.

package/docs/reference/protocol.md

@@ -75,20 +75,11 @@ planning uses an internal answer brief and shot descriptions. The runtime
 translates them into footage scenes and completes the answer at stream end,
 without asking the model for lifecycle commands.
 
-Handlers with an explicit custom `templates` registry keep the existing
-composition planner contract. A server-only planner emits
-validated `scene.add` or `plan.complete` parts. The runtime assigns sequences,
-IDs, terminal snapshots, and checksums. Generated HTML, React, JavaScript, CSS, component
-source, audio events, protocol envelopes, and unknown part types are rejected.
-
-A planner may add `placement: "closer"` to exactly one `scene.add`. The
-standard handler requires that closer by default, holds it outside the public
-event stream while body scenes continue, and commits it as the final scene.
-Only templates advertised for `jobs:[ask]` or `jobs:[payoff]` qualify. The
-placement marker is not part of `VideoScene` and never enters a replay
-snapshot. If the planner completes without a valid closer, the handler emits
-`plan_missing_closer` and uses `finishReason: "other"`; provider `length` and
-`content-filter` reasons remain unchanged.
+The runtime owns validated scene additions, sequence IDs, terminal snapshots,
+and checksums. The planned ending is reserved and committed after the body.
+If no valid ending is available, the response reports a safe incomplete-plan
+warning. Generated HTML, React, JavaScript, CSS, protocol envelopes, and unknown
+planning parts are rejected.
 
 ## Resume
 

package/docs/reference/provider-adapters.md

@@ -5,11 +5,11 @@ read [Media and voice](../media-and-audio.md).
 
 VanillaSky deliberately does not depend on a model provider or AI framework.
 Your server owns the model and credentials; `createVideoChatHandler` accepts
-`streamText` and `generateText` callbacks. The recommended adapter is the [AI SDK](https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-text),
+`streamText` and `generateText` callbacks. One optional adapter is the [AI SDK](https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-text),
 which gives the application one `LanguageModel` interface across official,
 community, AI Gateway, OpenAI-compatible, and custom providers.
 
-## Recommended: AI SDK
+## Optional: AI SDK
 
 Install the AI SDK plus the provider package your application chooses:
 
@@ -81,15 +81,17 @@ complete.
 ## Native or self-hosted providers
 
 The same chat contract supports a native provider without an AI SDK dependency.
+`npx @vanillaskyai/video init --native` creates an editable Gemini REST example.
 Return an `AsyncIterable<string>` from `streamText`, or an object with
 `textStream` and optional completion metadata. Implement `generateText` for the
-small welcome, suggestion, and fallback narration tasks and return its text.
+small welcome, suggestion, fallback narration, and bounded `narration-rewrite`
+tasks and return its text.
 Infer each callback from `VideoChatHandlerOptions` so the adapter stays aligned
 with the public contract.
 
 Forward the supplied signal, preserve both prompt strings, and keep provider
-errors and credentials on the server. Retry only within an explicit time and
-spend budget before output is visible. The chat does not expose a durable
+errors and credentials on the server. Do not automatically resubmit ambiguous
+paid requests. The chat does not expose a durable
 stream-reconnect contract.
 
 ## Generated-video budget
@@ -105,18 +107,24 @@ never consumes the generated-video allowance. Retries inside your
 provider callback can incur additional charges; bound those separately. Never
 copy an untrusted request value into this application-owned option.
 
-Use `generateVideoTimeoutMs` for slower providers (integer `1`–`120000`, default
+Use `generateVideoTimeoutMs` for slower providers (integer `1`–`600000`, default
 `15000`). Poll queued jobs only until the supplied signal aborts; submit once,
 record the provider job identifier, and cancel that job best-effort on abort.
 A disconnected browser or timed-out request does not prove the job was free.
 Reserve host-owned quotas before submission and retain uncertain attempts.
-Keep the host request deadline and client `timeoutMs` longer than the provider
-deadline. The SDK does not resume a completed queued job into a finished answer.
-
-For a limited offering, `VideoChat` accepts `generatedVideoLabel` and
-`generatedVideoDescription` to explain the generated-video choice in Settings.
-Set `showRecoveryNotice` to opt into a brief dismissible message when generated
-visuals fall back. These presentation props do not enforce limits or enable
+Keep the host request deadline longer than the provider deadline. The default
+client `timeoutMs` is `660000`; an explicit shorter override remains authoritative.
+The SDK does not resume a completed queued job into a finished answer.
+
+The video callback receives `requestedDurationSec`, `shotDirection`, `deadlineAt`
+(epoch milliseconds), and `signal`. Return `durationSec` when known. Keep model,
+duration, resolution, concurrency and timeout together in your editable adapter.
+These longer deadlines accommodate queued providers; they do not make generation
+real-time. The SDK streams ready scenes and prepares upcoming media progressively.
+
+Set `VideoChat`'s `showRecoveryNotice` to opt into a brief dismissible message
+when generated visuals fall back. Use an application-owned interface for custom
+plan labels or controls. Presentation does not enforce spending limits or enable
 provider capabilities; the server remains authoritative.
 
 ## Application retrieval

package/docs/security.md

@@ -1,4 +1,4 @@
-[← Documentation home](../README.md) · [Previous: Streaming protocol](streaming-protocol.md) · [Next: Errors and recovery →](errors.md)
+[← Documentation home](../README.md) · [Previous: Streaming protocol](reference/protocol.md) · [Next: Errors and recovery →](errors.md)
 
 # Security
 
@@ -14,19 +14,6 @@ The SDK validates protocol shape; your application still owns identity, authoriz
 - Allowlist browser origins; CORS is not authentication.
 - Bound request bytes, media count, scene count, duration, tokens, concurrency, and cost.
 - Keep provider keys, system prompts, tools, signed-URL credentials, and admin tokens server-side.
-- Use the same generated template registry on the server and in React; the handler infers validation.
-- Treat every project-owned template as trusted application build code and
-  review it before using the CLI. Normal `vanillasky templates list`,
-  `vanillasky templates describe`, `vanillasky templates add`, `vanillasky templates sync`, and
-  `vanillasky templates check` commands execute project template modules locally. This
-  includes `vanillasky templates add` previews with `--dry-run` or `--diff`, because the
-  CLI must derive the proposed browser and server registries. Resource and
-  environment boundaries reduce accidental damage but are not a portable
-  JavaScript sandbox.
-- Use `--builtin` with `list` or `describe` when you need the packaged catalog
-  only. That view does not execute project template modules. Project template
-  execution requires macOS, Linux, or WSL because Windows cannot provide the
-  process-group cleanup guarantee used by these commands.
 - Restrict media domains, types, dimensions, bytes, redirects, and fetch timeouts.
 - Propagate cancellation and use timeouts for provider, media, persistence, and export work.
 - Return safe typed errors while logging private causes only in protected observability.