@vanillaskyai/video 0.10.1 → 0.10.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/README.md +12 -7
- package/dist/check-runtime.js +2 -2
- package/dist/{chunk-6KZGF63O.js → chunk-3U2F7C7C.js} +53 -34
- package/dist/{chunk-5C6HZNHY.js → chunk-LJR5VLEI.js} +3 -2
- package/dist/{chunk-77VY4O7A.js → chunk-SI6CFFAA.js} +57 -10
- package/dist/{chunk-KHLO5OHT.js → chunk-UCXQORDZ.js} +1 -1
- package/dist/{chunk-VQH3JTQC.js → chunk-VVNZCY2U.js} +1 -1
- package/dist/{chunk-TVIU23OM.js → chunk-YZZSWGC3.js} +8 -3
- package/dist/{cinema-media-APDJS7SC.js → cinema-media-4JUYKPQI.js} +4 -4
- package/dist/{comparison-ZVE2DE7U.js → comparison-3QLMUC5V.js} +4 -4
- package/dist/{editorial-timeline-YIXXVJUV.js → editorial-timeline-IVE2P25N.js} +4 -4
- package/dist/{key-figure-I2C37FWV.js → key-figure-APD56LCO.js} +4 -4
- package/dist/{mobile-message-JSGBU6EX.js → mobile-message-WLK3WY5B.js} +4 -4
- package/dist/{quote-XH54G3FS.js → quote-Z6YET5BO.js} +4 -4
- package/dist/react.js +52 -19
- package/dist/{scene-video-backdrop-PCTARAAJ.js → scene-video-backdrop-OWUXPXHZ.js} +2 -2
- package/dist/server.js +333 -276
- package/docs/agent-integration.md +5 -4
- package/docs/getting-started.md +7 -5
- package/docs/media-and-audio.md +23 -17
- package/docs/production.md +7 -4
- package/docs/prompt-and-input.md +24 -15
- package/docs/provider-integration.md +2 -2
- package/docs/reference/protocol.md +7 -1
- package/docs/reference/provider-adapters.md +5 -3
- package/docs/testing.md +7 -2
- package/package.json +1 -1
- package/registry/items/backgrounds.json +2 -2
- package/registry/items/cinemaMedia.json +1 -1
- package/registry/items/comparison.json +1 -1
- package/registry/items/editorialTimeline.json +1 -1
- package/registry/items/keyFigure.json +1 -1
- package/registry/items/mobileMessage.json +1 -1
- package/registry/items/quote.json +1 -1
- package/starters/video-chat/README.md +18 -17
- package/starters/video-chat/package.json +1 -1
- package/starters/video-chat/providers/video.ts +5 -4
|
@@ -22,8 +22,9 @@ npm run dev
|
|
|
22
22
|
```
|
|
23
23
|
|
|
24
24
|
Init generates the thin application-owned shell around the SDK's complete
|
|
25
|
-
`VideoChat` experience and runs doctor automatically. The
|
|
26
|
-
|
|
25
|
+
`VideoChat` experience and runs doctor automatically. The baseline uses an immediate introduction and browser voice, and installs
|
|
26
|
+
no optional provider packages. Add the video adapter for moving video answers;
|
|
27
|
+
without a media provider, narration and subtitles remain available.
|
|
27
28
|
`ANTHROPIC_API_KEY` is the only required key. Rerun init after an interrupted
|
|
28
29
|
installation.
|
|
29
30
|
|
|
@@ -48,8 +49,8 @@ URL in a real browser. Browser automation should use normal motion
|
|
|
48
49
|
its final frame, and returns to a usable composer.
|
|
49
50
|
3. Ask one unrelated creative question in the same conversation and confirm it
|
|
50
51
|
produces a distinct response.
|
|
51
|
-
4. If doctor reports generated video ready,
|
|
52
|
-
|
|
52
|
+
4. If doctor reports generated video ready, verify moving footage throughout
|
|
53
|
+
one complete answer, including its ending, without editing client code.
|
|
53
54
|
5. Check failed network responses before reporting success.
|
|
54
55
|
|
|
55
56
|
The handoff should name the localhost URL and ready capabilities. It should
|
package/docs/getting-started.md
CHANGED
|
@@ -3,9 +3,10 @@
|
|
|
3
3
|
# Getting started
|
|
4
4
|
|
|
5
5
|
The fastest VanillaSky integration is the complete, general-purpose video chat.
|
|
6
|
-
It starts with
|
|
7
|
-
|
|
8
|
-
|
|
6
|
+
It starts with a template introduction and browser voice. Add the video adapter
|
|
7
|
+
and its server key for generated footage, plus reviewed stock for fallback.
|
|
8
|
+
Without a media provider, the answer retains narration and subtitles with an
|
|
9
|
+
unavailable-visual state. Generated speech is optional.
|
|
9
10
|
|
|
10
11
|
## Create the app
|
|
11
12
|
|
|
@@ -96,8 +97,9 @@ spoken hook from the response stream before the full plan is complete, holds its
|
|
|
96
97
|
scene is ready, reaches its final frame, and leaves the composer ready for
|
|
97
98
|
another turn. With stock media enabled, click a welcome or follow-up card and
|
|
98
99
|
confirm its footage carries directly into that opening. With generated video
|
|
99
|
-
enabled, confirm that
|
|
100
|
-
|
|
100
|
+
enabled, confirm that moving shots develop a complete answer with narration
|
|
101
|
+
and subtitles, and that the first shot continues the hook without repeating it.
|
|
102
|
+
Check that no finished clip freezes while narration continues.
|
|
101
103
|
|
|
102
104
|
The generated local authorization accepts localhost only. Replace it with your
|
|
103
105
|
real session check, rate limits, and usage policy before deploying.
|
package/docs/media-and-audio.md
CHANGED
|
@@ -6,22 +6,23 @@ VanillaSky keeps provider choice in the application. The SDK defines small
|
|
|
6
6
|
server callbacks, advertises only the capabilities you configure, and keeps
|
|
7
7
|
all credentials out of React and the browser bundle.
|
|
8
8
|
|
|
9
|
-
##
|
|
9
|
+
## AI-first video answers
|
|
10
10
|
|
|
11
|
-
The
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
11
|
+
The default chat shows an immediate template introduction while footage prepares,
|
|
12
|
+
then moving video with narration and subtitles. The planner describes visible
|
|
13
|
+
subjects and actions, adapting its direction to the request's intent. The runtime
|
|
14
|
+
attempts generated footage first and uses relevant stock when generation is
|
|
15
|
+
unavailable, denied by the host budget, or fails. It does not select body cards.
|
|
15
16
|
|
|
16
|
-
The
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
17
|
+
The seven packaged templates remain available to custom compositions. Default
|
|
18
|
+
chat body shots use `cinemaMedia` as the footage renderer, without extra headlines.
|
|
19
|
+
If both media providers miss, the answer retains its narration and subtitles
|
|
20
|
+
with an explicit unavailable-visual state rather than a text-slide replacement.
|
|
20
21
|
|
|
21
22
|
## Reviewed stock
|
|
22
23
|
|
|
23
|
-
Add `searchMedia`
|
|
24
|
-
cards
|
|
24
|
+
Add `searchMedia` for relevant fallback footage, welcome covers, and follow-up
|
|
25
|
+
cards:
|
|
25
26
|
|
|
26
27
|
```ts
|
|
27
28
|
createVideoChatHandler({
|
|
@@ -45,14 +46,14 @@ createVideoChatHandler({
|
|
|
45
46
|
The planner emits a short semantic keyword, not a URL. The callback returns an
|
|
46
47
|
application-approved image or video URL, and the SDK validates it before it
|
|
47
48
|
reaches a scene. Return `null` when no licensed, safe, relevant asset exists;
|
|
48
|
-
|
|
49
|
-
|
|
49
|
+
default chat retains the spoken answer when footage is unavailable. An explicit custom `templates` registry retains its structured planner,
|
|
50
|
+
validation, and fallback contracts.
|
|
50
51
|
|
|
51
52
|
For Pexels, keep `PEXELS_API_KEY` on the server, enforce a deadline, filter for
|
|
52
53
|
orientation, and return only validated Pexels asset domains. Licensing,
|
|
53
54
|
attribution, caching, MIME checks, and byte limits remain application-owned.
|
|
54
55
|
|
|
55
|
-
##
|
|
56
|
+
## Generated shots
|
|
56
57
|
|
|
57
58
|
Add `generateVideo` to enable generated shots within cinematic responses. It receives the planned visual
|
|
58
59
|
subject plus the generated look so every clip can follow the same direction:
|
|
@@ -75,6 +76,8 @@ createVideoChatHandler({
|
|
|
75
76
|
orientation,
|
|
76
77
|
requestId,
|
|
77
78
|
sceneId: scene?.id,
|
|
79
|
+
shotDirection: typeof scene?.variables.shotDirection === "string"
|
|
80
|
+
? scene.variables.shotDirection : undefined,
|
|
78
81
|
signal,
|
|
79
82
|
maxRetries: 0,
|
|
80
83
|
});
|
|
@@ -83,8 +86,9 @@ createVideoChatHandler({
|
|
|
83
86
|
});
|
|
84
87
|
```
|
|
85
88
|
|
|
86
|
-
|
|
87
|
-
model
|
|
89
|
+
One model stream supplies an answer brief and shot descriptions. The runtime
|
|
90
|
+
prepares ordered shots while the model continues planning; the first and later
|
|
91
|
+
body shots follow the same path. Use `requestId` and `scene.id` as
|
|
88
92
|
an idempotency key, because generated clips are billable. Keep `maxRetries: 0`
|
|
89
93
|
inside provider calls, honour `signal`, and make retries an explicit product
|
|
90
94
|
decision with a known budget.
|
|
@@ -129,7 +133,9 @@ interaction before audible playback on many devices.
|
|
|
129
133
|
- Keep every provider key in server-only environment variables.
|
|
130
134
|
- Never let a planner return arbitrary final media URLs.
|
|
131
135
|
- Bound query length, response size, duration, concurrency, and generated spend.
|
|
132
|
-
- Preload
|
|
136
|
+
- Preload upcoming footage and pause narration honestly when usable media is late.
|
|
137
|
+
- Plan enough moving footage for the spoken beat; do not hold a finished frame
|
|
138
|
+
while narration continues.
|
|
133
139
|
- Return a safe fallback instead of leaving the response waiting forever.
|
|
134
140
|
|
|
135
141
|
[← Documentation home](../README.md) · [Previous: Customization](customization.md) · [Next: Custom templates →](custom-templates.md)
|
package/docs/production.md
CHANGED
|
@@ -22,10 +22,13 @@ player. Read the [security guide](security.md) for the complete controls.
|
|
|
22
22
|
|
|
23
23
|
## Cinematic direction and providers
|
|
24
24
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
25
|
+
Default chat uses a template introduction during preparation, then generated
|
|
26
|
+
footage with narration and subtitles. Add `generateVideo` for these shots and
|
|
27
|
+
`searchMedia` for relevant stock fallback. The runtime selects media within the
|
|
28
|
+
host generation budget; the model supplies narration and visible actions.
|
|
29
|
+
If both sources miss, valid narration remains available with an explicit visual
|
|
30
|
+
unavailability state. Explicit custom template registries keep their existing
|
|
31
|
+
composition and fallback contracts.
|
|
29
32
|
A stock candidate must match the subject, action and permitted crop. Return
|
|
30
33
|
`null` for uncertainty rather than broadening an essential detail.
|
|
31
34
|
|
package/docs/prompt-and-input.md
CHANGED
|
@@ -55,7 +55,7 @@ viewer prompt separate from these trusted server-side instructions.
|
|
|
55
55
|
|
|
56
56
|
## What reaches the model
|
|
57
57
|
|
|
58
|
-
`createVideoChatHandler` builds the
|
|
58
|
+
`createVideoChatHandler` builds the shot-planning instructions, video rules,
|
|
59
59
|
conversation context, and application guidance. Your provider adapter receives
|
|
60
60
|
two complete strings:
|
|
61
61
|
|
|
@@ -68,9 +68,9 @@ streamText: ({ systemPrompt, userPrompt, signal }) => streamText({
|
|
|
68
68
|
});
|
|
69
69
|
```
|
|
70
70
|
|
|
71
|
-
Pass both strings unchanged. The system prompt describes the
|
|
72
|
-
|
|
73
|
-
|
|
71
|
+
Pass both strings unchanged. The system prompt describes the internal answer
|
|
72
|
+
brief and shot format, pacing, narration, opening contract, and safe media
|
|
73
|
+
directions. The user prompt contains the current request, bounded prior turns,
|
|
74
74
|
orientation, visual mode, and whether an opening was already spoken.
|
|
75
75
|
|
|
76
76
|
Provider credentials and raw media URLs never belong in either prompt. Media
|
|
@@ -79,19 +79,27 @@ output has been parsed.
|
|
|
79
79
|
|
|
80
80
|
## One stream, one answer
|
|
81
81
|
|
|
82
|
-
The planner
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
82
|
+
The planner streams an answer brief containing the opening, creative direction,
|
|
83
|
+
and ending, followed by narrated shot descriptions. One model request serves
|
|
84
|
+
both suggested and typed prompts; a suggestion may already supply its opening.
|
|
85
|
+
There is no separate classification or first-shot planning request.
|
|
86
|
+
|
|
87
|
+
The runtime assigns scene IDs, uses the footage renderer, resolves media, and
|
|
88
|
+
finalizes the response when planning ends. The model does not choose body
|
|
89
|
+
layouts, media providers, or lifecycle events. Narration and visible action are
|
|
90
|
+
planned together: explanations show mechanisms, stories develop consequences,
|
|
91
|
+
comedy times its reveal, imaginative requests depict their invented world, and
|
|
92
|
+
practical answers demonstrate usable steps. These are directions, not fixed
|
|
93
|
+
scene counts or one universal story structure.
|
|
86
94
|
|
|
87
95
|
Every `scene.add` is validated before the browser receives it. The model never
|
|
88
|
-
returns React, HTML, CSS, or executable JavaScript.
|
|
89
|
-
|
|
96
|
+
returns React, HTML, CSS, or executable JavaScript. Invalid planning content
|
|
97
|
+
produces safe diagnostics. A media failure does not delete valid narration.
|
|
90
98
|
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
99
|
+
Generated footage is preferred within the host's allowance. Relevant stock is
|
|
100
|
+
fallback. If neither is available, retain narration and subtitles and report
|
|
101
|
+
unavailable visuals. Handlers configured with an explicit custom `templates` registry continue to
|
|
102
|
+
support the trusted catalog and its existing structured planner contract.
|
|
95
103
|
|
|
96
104
|
## Grounding
|
|
97
105
|
|
|
@@ -110,7 +118,8 @@ Check these boundaries in order:
|
|
|
110
118
|
3. Does the provider pass `systemPrompt` and `userPrompt` unchanged?
|
|
111
119
|
4. Is extended reasoning delaying the first streamed object?
|
|
112
120
|
5. Do `onWarning` and `onComplete` show rejected scenes or a length limit?
|
|
113
|
-
6.
|
|
121
|
+
6. Do the planned actions develop the answer, and does its ending resolve the request?
|
|
122
|
+
7. Does resolved footage actually play for the spoken duration?
|
|
114
123
|
|
|
115
124
|
Log request IDs, safe warning codes, provider finish reasons, model IDs, and
|
|
116
125
|
token usage. Never log credentials or expose raw provider errors in the video.
|
|
@@ -12,7 +12,7 @@ npm run dev
|
|
|
12
12
|
```
|
|
13
13
|
|
|
14
14
|
Init runs doctor automatically. The generated `server.ts` starts with one
|
|
15
|
-
`ANTHROPIC_API_KEY`,
|
|
15
|
+
`ANTHROPIC_API_KEY`, a template introduction, and browser voice; it installs no
|
|
16
16
|
optional speech or video packages.
|
|
17
17
|
|
|
18
18
|
Use `npx vanillasky providers add speech` to install xAI speech, or
|
|
@@ -180,7 +180,7 @@ is intentionally substantial. It is stable for the same SDK version, template
|
|
|
180
180
|
kit, media policy, and base prompt. Record input-token usage, keep the selected
|
|
181
181
|
kit no broader than the product needs, and enable provider-side prompt caching
|
|
182
182
|
where the chosen provider/model supports it. VanillaSky does not assume one
|
|
183
|
-
provider's cache controls in its provider-neutral adapter.
|
|
183
|
+
provider's cache controls in its provider-neutral adapter. Default chat streams an answer brief and shot directions rather than the template catalog; use provider-reported token
|
|
184
184
|
usage as the authoritative measurement rather than a character estimate.
|
|
185
185
|
|
|
186
186
|
Provider finish reasons `error` and `tool-calls` are terminal failures.
|
|
@@ -70,7 +70,13 @@ change core video state.
|
|
|
70
70
|
|
|
71
71
|
## Planning boundary
|
|
72
72
|
|
|
73
|
-
LLMs do not emit public protocol envelopes.
|
|
73
|
+
LLMs do not emit public protocol envelopes. Default `createVideoChatHandler`
|
|
74
|
+
planning uses an internal answer brief and shot descriptions. The runtime
|
|
75
|
+
translates them into footage scenes and completes the answer at stream end,
|
|
76
|
+
without asking the model for lifecycle commands.
|
|
77
|
+
|
|
78
|
+
Handlers with an explicit custom `templates` registry keep the existing
|
|
79
|
+
composition planner contract. A server-only planner emits
|
|
74
80
|
validated `scene.add` or `plan.complete` parts. The runtime assigns sequences,
|
|
75
81
|
IDs, terminal snapshots, and checksums. Generated HTML, React, JavaScript, CSS, component
|
|
76
82
|
source, audio events, protocol envelopes, and unknown part types are rejected.
|
|
@@ -69,7 +69,9 @@ VanillaSky or its protocol. The same boundary also accepts a self-hosted model
|
|
|
69
69
|
or a provider-native async text stream when it is not represented in the AI
|
|
70
70
|
SDK. VanillaSky has no model allowlist.
|
|
71
71
|
|
|
72
|
-
The provider must emit NDJSON text
|
|
72
|
+
The provider must emit NDJSON text matching the supplied prompt: one complete
|
|
73
|
+
object per line. Default chat planning uses an answer brief and shot descriptions;
|
|
74
|
+
the SDK translates them into validated scene events.
|
|
73
75
|
The SDK buffers arbitrary text chunks until a newline, parses the completed
|
|
74
76
|
object, validates it, and only then forwards it to the motion runtime. Do not
|
|
75
77
|
replace that per-line validator with whole-response structured output: motion
|
|
@@ -93,8 +95,8 @@ stream-reconnect contract.
|
|
|
93
95
|
## Generated-video budget
|
|
94
96
|
|
|
95
97
|
Set `maxGeneratedVideos` on `createVideoChatHandler` to a nonnegative safe integer
|
|
96
|
-
(default `5`). This is a per-response attempt limit
|
|
97
|
-
|
|
98
|
+
(default `5`). This is a per-response generation attempt limit, including
|
|
99
|
+
failures. Default chat attempts generated footage before relevant stock. Use `0` to keep the configured full mode on
|
|
98
100
|
stock footage. Stock lookup is independent of this limit. Retries inside your
|
|
99
101
|
provider callback can incur additional charges; bound those separately. Never
|
|
100
102
|
copy an untrusted request value into this application-owned option.
|
package/docs/testing.md
CHANGED
|
@@ -7,12 +7,15 @@ no network request, and needs no model key.
|
|
|
7
7
|
## Test the chat route with Vitest
|
|
8
8
|
|
|
9
9
|
Pass `createMockVideoPlanner()` to the same `createVideoChatHandler` used by the
|
|
10
|
-
application
|
|
10
|
+
application, using an explicit template registry for the structured composition
|
|
11
|
+
fixtures. Default AI-first chat tests should instead return an authored answer
|
|
12
|
+
brief and shot descriptions matching the supplied planning prompt. A standard
|
|
13
|
+
`Request` exercises parsing, validation, pacing, the
|
|
11
14
|
opening extension, and SSE without starting an HTTP server.
|
|
12
15
|
|
|
13
16
|
```ts
|
|
14
17
|
import { describe, expect, it } from "vitest";
|
|
15
|
-
import { createVideoChatHandler } from "@vanillaskyai/video/server";
|
|
18
|
+
import { createVideoChatHandler, createServerTemplateRegistry } from "@vanillaskyai/video/server";
|
|
16
19
|
import { createMockVideoPlanner } from "@vanillaskyai/video/test";
|
|
17
20
|
|
|
18
21
|
describe("POST /api/video-chat", () => {
|
|
@@ -20,6 +23,7 @@ describe("POST /api/video-chat", () => {
|
|
|
20
23
|
const handle = createVideoChatHandler({
|
|
21
24
|
authorize: "none", // Only acceptable because this handler stays in process.
|
|
22
25
|
heartbeatMs: false,
|
|
26
|
+
templates: createServerTemplateRegistry({ templates: [] }),
|
|
23
27
|
streamText: createMockVideoPlanner(),
|
|
24
28
|
generateText: async ({ task }) => task === "suggestions"
|
|
25
29
|
? JSON.stringify({ suggestions: [] })
|
|
@@ -43,6 +47,7 @@ describe("POST /api/video-chat", () => {
|
|
|
43
47
|
|
|
44
48
|
expect(response.status).toBe(200);
|
|
45
49
|
expect(body).toContain('"type":"data.video-chat-opening"');
|
|
50
|
+
expect(body).toContain('"type":"scene.add"');
|
|
46
51
|
expect(body).toContain('"type":"response.complete"');
|
|
47
52
|
});
|
|
48
53
|
});
|
package/package.json
CHANGED
|
@@ -28,7 +28,7 @@
|
|
|
28
28
|
"path": "src/visual-system/scene-templates/external-video-backdrop.tsx",
|
|
29
29
|
"type": "registry:lib",
|
|
30
30
|
"target": "vanillasky/scene-templates/external-video-backdrop.tsx",
|
|
31
|
-
"content": "import React from \"react\";\n\n// Source-owned templates may live in a consumer's tree while VideoFrame comes\n// from the package. Both copies must observe the same internal context or the\n// consumer template would mount a second video over the player-owned plane,\n// and would never inherit the player's native media audio state.\nexport type ExternalVideoBackdropMode = false | \"pending\" | \"ready\" | \"fallback\";\n\ninterface BackdropContextValue {\n mode: ExternalVideoBackdropMode;\n audioMuted: boolean;\n audioVolume: number;\n}\n\nconst DEFAULT: BackdropContextValue = { mode: false, audioMuted: true, audioVolume: 1 };\n\nconst sharedContext = globalThis as typeof globalThis & {\n __vanillaskyVideoBackdropContext?: React.Context<BackdropContextValue>;\n};\nconst BackdropContext = sharedContext.__vanillaskyVideoBackdropContext\n ??= React.createContext<BackdropContextValue>(DEFAULT);\n\nexport function ExternalVideoBackdropProvider({\n mode,\n audioMuted = true,\n audioVolume = 1,\n children,\n}: {\n mode: ExternalVideoBackdropMode;\n audioMuted?: boolean;\n audioVolume?: number;\n children: React.ReactNode;\n}) {\n const value = React.useMemo(\n () => ({ mode, audioMuted, audioVolume }),\n [mode, audioMuted, audioVolume],\n );\n return (\n <BackdropContext.Provider value={value}>\n {children}\n </BackdropContext.Provider>\n );\n}\n\nexport function useExternalVideoBackdrop(): ExternalVideoBackdropMode {\n return React.useContext(BackdropContext).mode;\n}\n\nexport function useMediaAudio(): { muted: boolean; volume: number } {\n const { audioMuted, audioVolume } = React.useContext(BackdropContext);\n return { muted: audioMuted, volume: audioVolume };\n}\n"
|
|
31
|
+
"content": "import React from \"react\";\n\n// Source-owned templates may live in a consumer's tree while VideoFrame comes\n// from the package. Both copies must observe the same internal context or the\n// consumer template would mount a second video over the player-owned plane,\n// and would never inherit the player's native media audio state.\nexport type ExternalVideoBackdropMode = false | \"pending\" | \"ready\" | \"fallback\";\n\ninterface BackdropContextValue {\n mode: ExternalVideoBackdropMode;\n audioMuted: boolean;\n audioVolume: number;\n preparingNarration?: boolean;\n}\n\nconst DEFAULT: BackdropContextValue = { mode: false, audioMuted: true, audioVolume: 1 };\n\nconst sharedContext = globalThis as typeof globalThis & {\n __vanillaskyVideoBackdropContext?: React.Context<BackdropContextValue>;\n};\nconst BackdropContext = sharedContext.__vanillaskyVideoBackdropContext\n ??= React.createContext<BackdropContextValue>(DEFAULT);\n\nexport function ExternalVideoBackdropProvider({\n mode,\n audioMuted = true,\n audioVolume = 1,\n preparingNarration = false,\n children,\n}: {\n mode: ExternalVideoBackdropMode;\n audioMuted?: boolean;\n audioVolume?: number;\n preparingNarration?: boolean;\n children: React.ReactNode;\n}) {\n const value = React.useMemo(\n () => ({ mode, audioMuted, audioVolume, preparingNarration }),\n [mode, audioMuted, audioVolume, preparingNarration],\n );\n return (\n <BackdropContext.Provider value={value}>\n {children}\n </BackdropContext.Provider>\n );\n}\n\nexport function useExternalVideoBackdrop(): ExternalVideoBackdropMode {\n return React.useContext(BackdropContext).mode;\n}\n\nexport function useMediaAudio(): { muted: boolean; volume: number } {\n const { audioMuted, audioVolume } = React.useContext(BackdropContext);\n return { muted: audioMuted, volume: audioVolume };\n}\n\n/** Internal first-frame priming state; an explicit viewer pause never sets it. */\nexport function useNarrationPreroll(): boolean {\n return React.useContext(BackdropContext).preparingNarration === true;\n}\n"
|
|
32
32
|
},
|
|
33
33
|
{
|
|
34
34
|
"path": "src/visual-system/scene-templates/media-position.ts",
|
|
@@ -40,7 +40,7 @@
|
|
|
40
40
|
"path": "src/visual-system/scene-templates/scene-video-backdrop.tsx",
|
|
41
41
|
"type": "registry:lib",
|
|
42
42
|
"target": "vanillasky/scene-templates/scene-video-backdrop.tsx",
|
|
43
|
-
"content": "import React, { useEffect, useRef, useState } from \"react\";\nimport { getBackgroundTransform } from \"../backgrounds\";\nimport { useMediaAudio } from \"./external-video-backdrop\";\nimport { resolveMediaPosition } from \"./media-position\";\n\nexport interface SceneVideoBackdropProps {\n mediaUrl: string;\n mediaPoster?: string;\n mediaPosition?: string;\n backgroundEffect?: string;\n progress: number;\n beatIntensity?: number;\n isPlaying: boolean;\n muted?: boolean;\n volume?: number;\n playbackId?: string;\n retainPoster?: boolean;\n persistent?: boolean;\n preparedPoster?: {\n presentationKey: string;\n mediaPoster: string;\n mediaPosition: string;\n backgroundEffect?: string;\n /** Existing global transition progress. On decoder-constrained Safari,\n * this fades the decoded incoming still above the outgoing video before\n * the single video element changes source. */\n opacity?: number;\n };\n onReady?: () => void;\n onError?: () => void;\n}\n\nexport const SceneVideoBackdrop: React.FC<SceneVideoBackdropProps> = ({\n mediaUrl,\n mediaPoster,\n mediaPosition = \"center\",\n backgroundEffect,\n progress,\n beatIntensity = 0,\n isPlaying,\n muted,\n volume,\n playbackId = mediaUrl,\n retainPoster = false,\n persistent = false,\n preparedPoster,\n onReady,\n onError,\n}) => {\n const inheritedAudio = useMediaAudio();\n const resolvedMuted = muted ?? inheritedAudio.muted;\n const resolvedVolume = volume ?? inheritedAudio.volume;\n const resolvedPosition = resolveMediaPosition(mediaPosition);\n const bgTransform = getBackgroundTransform(backgroundEffect, progress, beatIntensity);\n const [decodedVideoUrl, setDecodedVideoUrl] = useState<string>();\n const videoRef = useRef<HTMLVideoElement>(null);\n const startedVideoUrl = useRef<string | undefined>(undefined);\n const startedPlaybackId = useRef<string | undefined>(undefined);\n const videoPresentationKey = `${playbackId}\\0${mediaUrl}`;\n\n useEffect(() => {\n const video = videoRef.current;\n if (!video) return;\n // React Strict Mode rehearses setup → cleanup → setup in development.\n // The cleanup deliberately releases the decoder, so the repeated setup\n // must restore the declarative source before the playback effect runs.\n if (video.getAttribute(\"src\") !== mediaUrl) {\n video.setAttribute(\"src\", mediaUrl);\n video.load();\n }\n return () => {\n video.pause();\n video.removeAttribute(\"src\");\n video.load();\n startedVideoUrl.current = undefined;\n startedPlaybackId.current = undefined;\n };\n }, [mediaUrl]);\n\n useEffect(() => {\n const video = videoRef.current;\n if (video) video.volume = resolvedVolume;\n }, [resolvedVolume]);\n\n useEffect(() => {\n const video = videoRef.current;\n if (!video) return;\n if (!isPlaying) {\n video.pause();\n return;\n }\n if (startedPlaybackId.current === playbackId) {\n if (
|
|
43
|
+
"content": "import React, { useCallback, useEffect, useRef, useState } from \"react\";\nimport { getBackgroundTransform } from \"../backgrounds\";\nimport { useMediaAudio, useNarrationPreroll } from \"./external-video-backdrop\";\nimport { resolveMediaPosition } from \"./media-position\";\n\nexport interface SceneVideoBackdropProps {\n mediaUrl: string;\n mediaPoster?: string;\n mediaPosition?: string;\n backgroundEffect?: string;\n progress: number;\n /** Narration-led visible duration; muted or pitch-preserving footage may be gently retimed. */\n sceneDuration?: number;\n /** Internal player-owned decoder priming, distinct from viewer pause. */\n preparingNarration?: boolean;\n beatIntensity?: number;\n isPlaying: boolean;\n muted?: boolean;\n volume?: number;\n playbackId?: string;\n retainPoster?: boolean;\n persistent?: boolean;\n preparedPoster?: {\n presentationKey: string;\n mediaPoster: string;\n mediaPosition: string;\n backgroundEffect?: string;\n /** Existing global transition progress. On decoder-constrained Safari,\n * this fades the decoded incoming still above the outgoing video before\n * the single video element changes source. */\n opacity?: number;\n };\n onReady?: () => void;\n onError?: () => void;\n}\n\nexport const SceneVideoBackdrop: React.FC<SceneVideoBackdropProps> = ({\n mediaUrl,\n mediaPoster,\n mediaPosition = \"center\",\n backgroundEffect,\n progress,\n sceneDuration,\n preparingNarration = false,\n beatIntensity = 0,\n isPlaying,\n muted,\n volume,\n playbackId = mediaUrl,\n retainPoster = false,\n persistent = false,\n preparedPoster,\n onReady,\n onError,\n}) => {\n const inheritedAudio = useMediaAudio();\n const inheritedPreroll = useNarrationPreroll();\n const rewindPreroll = preparingNarration || inheritedPreroll;\n const resolvedMuted = muted ?? inheritedAudio.muted;\n const resolvedVolume = volume ?? inheritedAudio.volume;\n const resolvedPosition = resolveMediaPosition(mediaPosition);\n const bgTransform = getBackgroundTransform(backgroundEffect, progress, beatIntensity);\n const [decodedVideoUrl, setDecodedVideoUrl] = useState<string>();\n const [waitingKey, setWaitingKey] = useState<string>();\n const [exhaustedKey, setExhaustedKey] = useState<string>();\n const continuityReplay = useRef<string | undefined>(undefined);\n const videoRef = useRef<HTMLVideoElement>(null);\n const startedVideoUrl = useRef<string | undefined>(undefined);\n const startedPlaybackId = useRef<string | undefined>(undefined);\n const videoPresentationKey = `${playbackId}\\0${mediaUrl}`;\n\n const presentationRef = useRef({ key: videoPresentationKey, playing: isPlaying });\n presentationRef.current = { key: videoPresentationKey, playing: isPlaying };\n const unavailable = () => {\n if (presentationRef.current.key === videoPresentationKey && presentationRef.current.playing) {\n setExhaustedKey(videoPresentationKey);\n onError?.();\n }\n };\n const fitDuration = useCallback((video: HTMLVideoElement) => {\n // Allow a small decode-to-speech onset margin without changing narration.\n video.playbackRate = (resolvedMuted || video.preservesPitch === true) && sceneDuration && Number.isFinite(video.duration) && video.duration > 0\n ? Math.max(.75, Math.min(1, video.duration / (sceneDuration + .2))) : 1;\n }, [resolvedMuted, sceneDuration]);\n useEffect(() => {\n if (videoRef.current) fitDuration(videoRef.current);\n }, [fitDuration]);\n const continueMotion = (video: HTMLVideoElement) => {\n if (!isPlaying) return;\n // The planner supplies short shots as the normal coverage. A single replay\n // bridges exceptional speech overrun/late delivery; never loop indefinitely.\n if (!resolvedMuted || continuityReplay.current === videoPresentationKey) {\n setExhaustedKey(videoPresentationKey);\n return;\n }\n continuityReplay.current = videoPresentationKey;\n video.currentTime = 0;\n void video.play().catch(unavailable);\n };\n\n useEffect(() => {\n const video = videoRef.current;\n if (!video) return;\n // React Strict Mode rehearses setup → cleanup → setup in development.\n // The cleanup deliberately releases the decoder, so the repeated setup\n // must restore the declarative source before the playback effect runs.\n if (video.getAttribute(\"src\") !== mediaUrl) {\n video.setAttribute(\"src\", mediaUrl);\n video.load();\n }\n return () => {\n video.pause();\n video.removeAttribute(\"src\");\n video.load();\n startedVideoUrl.current = undefined;\n startedPlaybackId.current = undefined;\n };\n }, [mediaUrl]);\n\n useEffect(() => {\n const video = videoRef.current;\n if (video) video.volume = resolvedVolume;\n }, [resolvedVolume]);\n\n useEffect(() => {\n const video = videoRef.current;\n if (!video) return;\n if (!isPlaying) {\n video.pause();\n if (rewindPreroll && video.currentTime > 0) video.currentTime = 0;\n return;\n }\n if (startedPlaybackId.current === playbackId) {\n if (video.ended) continueMotion(video);\n else void video.play().catch(unavailable);\n return;\n }\n const changingSource = startedVideoUrl.current !== undefined && startedVideoUrl.current !== mediaUrl;\n fitDuration(video);\n if (!changingSource && video.currentTime > 0) video.currentTime = 0;\n video.play().catch(unavailable);\n startedVideoUrl.current = mediaUrl;\n startedPlaybackId.current = playbackId;\n }, [isPlaying, mediaUrl, playbackId, rewindPreroll]);\n\n const mediaStyle: React.CSSProperties = {\n position: \"absolute\",\n inset: 0,\n width: \"100%\",\n height: \"100%\",\n objectFit: \"cover\",\n objectPosition: resolvedPosition,\n transform: bgTransform.transform,\n transformOrigin: bgTransform.transformOrigin,\n zIndex: persistent ? 1 : undefined,\n };\n const preparedPosition = preparedPoster\n ? resolveMediaPosition(preparedPoster.mediaPosition)\n : resolvedPosition;\n const preparedTransform = getBackgroundTransform(preparedPoster?.backgroundEffect, 0, 0);\n const posterPlanes = [\n ...(persistent && mediaPoster ? [{\n presentationKey: videoPresentationKey,\n mediaPoster,\n mediaPosition: resolvedPosition,\n transform: bgTransform.transform,\n transformOrigin: bgTransform.transformOrigin,\n opacity: 1,\n zIndex: 0,\n role: \"current\",\n }] : []),\n ...(preparedPoster && preparedPoster.presentationKey !== videoPresentationKey ? [{\n presentationKey: preparedPoster.presentationKey,\n mediaPoster: preparedPoster.mediaPoster,\n mediaPosition: preparedPosition,\n transform: preparedTransform.transform,\n transformOrigin: preparedTransform.transformOrigin,\n opacity: preparedPoster.opacity ?? 0,\n zIndex: 2,\n role: \"prepared\",\n }] : []),\n ];\n\n return (\n <>\n {posterPlanes.map((posterPlane) => (\n <img\n key={posterPlane.presentationKey}\n src={posterPlane.mediaPoster}\n alt=\"\"\n aria-hidden=\"true\"\n draggable={false}\n data-video-poster-plane={posterPlane.role}\n data-video-poster-visible={posterPlane.opacity > 0 ? \"true\" : \"false\"}\n style={{\n position: \"absolute\",\n inset: 0,\n width: \"100%\",\n height: \"100%\",\n objectFit: \"cover\",\n objectPosition: posterPlane.mediaPosition,\n transform: posterPlane.transform,\n transformOrigin: posterPlane.transformOrigin,\n zIndex: posterPlane.zIndex,\n opacity: posterPlane.opacity,\n pointerEvents: \"none\",\n }}\n />\n ))}\n {(exhaustedKey === videoPresentationKey || waitingKey === videoPresentationKey) && <div\n role=\"status\" data-media-continuity={exhaustedKey === videoPresentationKey ? \"exhausted\" : \"waiting\"}\n style={{ position: \"absolute\", inset: 0, zIndex: 3, background: \"#000\", color: \"#bbb\", display: \"grid\", placeContent: \"center\", font: \"14px system-ui\" }}\n >{exhaustedKey === videoPresentationKey ? \"Visual unavailable\" : \"Loading visual\"}</div>}\n <video\n ref={videoRef}\n src={mediaUrl}\n poster={retainPoster || decodedVideoUrl !== mediaUrl ? mediaPoster || undefined : undefined}\n muted={resolvedMuted}\n loop={false}\n playsInline\n preload=\"auto\"\n onLoadedMetadata={event => fitDuration(event.currentTarget)}\n onEnded={event => continueMotion(event.currentTarget)}\n onWaiting={() => { if (isPlaying) setWaitingKey(videoPresentationKey); }}\n onPlaying={() => setWaitingKey(undefined)}\n onLoadedData={(event) => {\n const video = event.currentTarget;\n const markPresented = () => {\n if (!video.isConnected) return;\n onReady?.();\n if (!retainPoster) setDecodedVideoUrl(mediaUrl);\n };\n if (video.requestVideoFrameCallback) {\n video.requestVideoFrameCallback(markPresented);\n return;\n }\n markPresented();\n }}\n onError={onError}\n data-media-position={mediaPosition}\n data-video-backdrop={persistent ? \"persistent\" : \"scene\"}\n style={{ ...mediaStyle, visibility: exhaustedKey === videoPresentationKey || waitingKey === videoPresentationKey ? \"hidden\" : undefined }}\n />\n </>\n );\n};\n"
|
|
44
44
|
},
|
|
45
45
|
{
|
|
46
46
|
"path": "src/visual-system/scene-templates/color-utils.ts",
|
|
@@ -29,7 +29,7 @@
|
|
|
29
29
|
"path": "src/visual-system/scene-templates/scene-background.tsx",
|
|
30
30
|
"type": "registry:component",
|
|
31
31
|
"target": "vanillasky/scene-templates/scene-background.tsx",
|
|
32
|
-
"content": "/**\n * Shared photo/video backdrop for built-in and customer-owned templates.\n * Media is host-resolved; this component never searches or generates assets.\n * The base is fixed black. Photos and videos cover it when available, and\n * optional scrims appear only once the media can paint. The cinematic built-ins\n * request no scrim; custom templates can choose a treatment and text anchor.\n * The retained internal \"gradient\" media sentinel selects the black base and\n * ignores mediaUrl. It is not a built-in authoring mode or brand-color control.\n * Video playback can be owned by the player's persistent external backdrop.\n */\n\nimport React, { useEffect, useState } from \"react\";\nimport {\n hasSceneMedia,\n resolveMediaType,\n type ResolvedMediaType,\n} from \"./media-source\";\nimport type { TemplateStyle } from \"../template-context\";\nimport { BrandGradientOverlay } from \"../backgrounds\";\nimport { getBackgroundTransform } from \"../backgrounds\";\nimport { useExternalVideoBackdrop } from \"./external-video-backdrop\";\nimport { resolveMediaPosition, type MediaPosition } from \"./media-position\";\nimport { SceneVideoBackdrop } from \"./scene-video-backdrop\";\n\nexport { hasSceneMedia, resolveMediaType };\nexport { resolveMediaPosition } from \"./media-position\";\nexport { SceneVideoBackdrop } from \"./scene-video-backdrop\";\nexport type { ResolvedMediaType };\n\nexport type { MediaPosition };\nexport type MediaTreatment = \"none\" | \"subtle\" | \"cinematic\" | \"text-safe\";\n\nexport function resolveMediaTreatment(value: string): MediaTreatment {\n return value === \"none\" || value === \"subtle\" || value === \"text-safe\" ? value : \"cinematic\";\n}\n\nexport interface MediaTreatmentLayer {\n id: \"vignette\" | \"center-scrim\" | \"bottom-scrim\";\n background: string;\n style?: React.CSSProperties;\n}\n\n/**\n * Where the template puts its type. The scrim is shaped to the copy, not to\n * the frame: darkening picture the type never touches costs contrast in the\n * photo and buys no legibility. \"full\" is the conservative default for\n * templates that have not declared an anchor.\n */\nexport type MediaTextAnchor = \"center\" | \"bottom\" | \"full\";\n\n/**\n * Smoothstep-sampled alpha stops between `start`% and `end`% of the gradient\n * box, held at full strength before `start` and after `end`.\n *\n * A two-stop `rgba(0,0,0,a) → transparent` scrim ramps alpha linearly, so it\n * ends with a constant slope. Lateral inhibition in the eye amplifies that\n * slope discontinuity into a visible band — the grey bar cutting across the\n * frame that makes an overlay read as an overlay. Smoothstep flattens the\n * curve at both ends, so the scrim holds where the type sits and then leaves\n * without an edge: the same peak coverage over the copy, noticeably less of\n * the picture spent getting there.\n */\nconst SCRIM_STOP_COUNT = 7;\n\nfunction smoothstep(t: number): number {\n return t * t * (3 - 2 * t);\n}\n\nfunction easedStops(\n peakAlpha: number,\n start: number,\n end: number,\n direction: \"fade-out\" | \"fade-in\",\n): string {\n const alphaAt = (t: number): string => {\n const eased = direction === \"fade-out\" ? 1 - smoothstep(t) : smoothstep(t);\n return `rgba(0,0,0,${Number((peakAlpha * eased).toFixed(3))})`;\n };\n const stops: string[] = [];\n if (start > 0) stops.push(`${alphaAt(0)} 0%`);\n for (let i = 0; i < SCRIM_STOP_COUNT; i += 1) {\n const t = i / (SCRIM_STOP_COUNT - 1);\n const position = Number((start + (end - start) * t).toFixed(2));\n stops.push(`${alphaAt(t)} ${position}%`);\n }\n if (end < 100) stops.push(`${alphaAt(1)} 100%`);\n return stops.join(\", \");\n}\n\n/**\n * Export-safe contrast recipes. Overlays only: SVG capture cannot rely on CSS\n * filters, so a blur-behind-text plate is off the table.\n *\n * The scrims deliberately stop short of solving legibility on their own. A\n * uniform darkening strong enough to carry white type over a blown-out sky\n * needs roughly 0.8 alpha — at that point the photo is a texture, not a\n * picture. The cheaper half of the job belongs to the type: a per-glyph halo\n * (MEDIA_TEXT_SHADOW) buys local contrast exactly where it is needed and\n * costs the image nothing. Scrim for the plate, halo for the glyph.\n */\nexport function getMediaTreatmentLayers(\n value: string,\n anchor: MediaTextAnchor = \"full\",\n): MediaTreatmentLayer[] {\n const treatment = resolveMediaTreatment(value);\n // Nothing over the picture at all. Every other treatment exists to carry\n // type across a photograph; where the scene has no type - a generated clip\n // that is the whole point of the beat - even a vignette is something the\n // viewer did not ask to look through.\n if (treatment === \"none\") return [];\n const vignette: MediaTreatmentLayer = {\n id: \"vignette\",\n background:\n treatment === \"subtle\"\n ? `radial-gradient(ellipse at center, ${easedStops(0.28, 45, 100, \"fade-in\")})`\n : `radial-gradient(ellipse at center, ${easedStops(0.72, 32, 100, \"fade-in\")})`,\n };\n if (treatment === \"subtle\") return [vignette];\n\n const textSafe = treatment === \"text-safe\";\n const layers: MediaTreatmentLayer[] = [vignette];\n\n if (anchor !== \"bottom\") {\n layers.push({\n id: \"center-scrim\",\n background: textSafe\n ? `radial-gradient(ellipse 92% 58% at 50% 50%, ${easedStops(0.46, 34, 90, \"fade-out\")})`\n : `radial-gradient(ellipse 88% 52% at 50% 50%, ${easedStops(0.26, 30, 88, \"fade-out\")})`,\n });\n }\n\n if (anchor !== \"center\") {\n layers.push({\n id: \"bottom-scrim\",\n background: `linear-gradient(to top, ${easedStops(textSafe ? 0.64 : 0.5, 8, 100, \"fade-out\")})`,\n style: { top: \"55%\" },\n });\n }\n\n return layers;\n}\n\n/**\n * Whether the backdrop is actually painting, which is what decides if a scrim\n * is earned. \"pending\" is a browser-only state: static and export renders\n * never run effects and never wait on a network, so they start (and stay)\n * ready and their output is unchanged.\n */\ntype MediaPaintState = \"pending\" | \"ready\" | \"failed\";\n\nfunction initialMediaPaint(\n wantsMedia: boolean,\n resolved: ResolvedMediaType,\n mediaUrl: string,\n mediaPoster: string | undefined,\n): MediaPaintState {\n if (typeof window === \"undefined\") return \"ready\";\n if (!wantsMedia) return \"ready\";\n // A poster paints the video's frame immediately, so the scene is already\n // showing footage even though the stream is still decoding.\n if (resolved === \"video\") return mediaPoster ? \"ready\" : \"pending\";\n if (typeof Image === \"undefined\") return \"ready\";\n // Preloaded or browser-cached media decodes synchronously. Reporting it\n // ready on the first render keeps the common mid-playback case free of a\n // black-then-photo flicker.\n const cached = new Image();\n cached.src = mediaUrl;\n return cached.complete && cached.naturalWidth > 0 ? \"ready\" : \"pending\";\n}\n\nexport function getMediaBackgroundProps(variables: Record<string, unknown>) {\n return {\n mediaUrl: String(variables.mediaUrl || \"\"),\n mediaType: String(variables.mediaType || \"auto\"),\n mediaPoster: String(variables.mediaPoster || \"\"),\n mediaPosition: String(variables.mediaPosition || \"center\"),\n mediaTreatment: String(variables.mediaTreatment || \"cinematic\"),\n };\n}\n\nexport interface SceneBackgroundProps {\n style: TemplateStyle;\n progress: number;\n sceneDuration?: number;\n width: number;\n height: number;\n mediaUrl?: string;\n mediaType?: string;\n /** Still image URL shown while the <video> backdrop decodes its first\n * frame. Without it the element renders transparent during the\n * decode window and the black base shows through. */\n mediaPoster?: string;\n /** Cover-crop focal anchor. Keeps the important edge/subject visible. */\n mediaPosition?: string;\n /** Overlay recipe: subtle, cinematic, or stronger text-safe contrast. */\n mediaTreatment?: string;\n /** Where this template's copy sits, so the scrim is shaped to the type\n * instead of to the frame. Defaults to \"full\" (scrim both the middle and\n * the lower third) for templates that have not declared an anchor. */\n textAnchor?: MediaTextAnchor;\n /** Background motion effect (drift / pulse / Ken Burns). Applied to the photo/video. */\n backgroundEffect?: string;\n /** Retained seed input for customer-owned backdrop compositions. */\n seed?: number | string;\n /** Pause video when preview is paused. Defaults to true (export path). */\n isPlaying?: boolean;\n beatIntensity?: number;\n}\n\nexport const SceneBackground: React.FC<SceneBackgroundProps> = ({\n style,\n progress,\n sceneDuration,\n width: _width, // accepted for symmetry; not currently used in render\n height: _height,\n mediaUrl = \"\",\n mediaType = \"auto\",\n mediaPoster,\n mediaPosition = \"center\",\n mediaTreatment = \"cinematic\",\n textAnchor = \"full\",\n backgroundEffect,\n seed,\n isPlaying = true,\n beatIntensity = 0,\n}) => {\n void _width;\n void _height;\n const resolved = resolveMediaType(mediaType, mediaUrl);\n const wantsMedia = resolved !== \"gradient\" && !!mediaUrl;\n const externalVideoBackdrop = useExternalVideoBackdrop();\n const hasExternalVideoBackdrop = externalVideoBackdrop !== false && resolved === \"video\";\n const externalVideoFailed = externalVideoBackdrop === \"fallback\" && resolved === \"video\";\n const externalVideoReady = externalVideoBackdrop === \"ready\" && resolved === \"video\";\n\n // Apply picture and scrim together. A loading or failed asset keeps the\n // fixed black base instead of painting contrast treatment over empty media.\n const [mediaPaint, setMediaPaint] = useState<MediaPaintState>(() =>\n initialMediaPaint(wantsMedia, resolved, mediaUrl, mediaPoster),\n );\n useEffect(() => {\n setMediaPaint(initialMediaPaint(wantsMedia, resolved, mediaUrl, mediaPoster));\n // Video reports its own paint through onLoadedData / onError below.\n if (!wantsMedia || resolved !== \"photo\") return;\n if (typeof Image === \"undefined\") return;\n let cancelled = false;\n const probe = new Image();\n probe.onload = () => {\n if (!cancelled) setMediaPaint(\"ready\");\n };\n probe.onerror = () => {\n if (!cancelled) setMediaPaint(\"failed\");\n };\n probe.src = mediaUrl;\n if (probe.complete) setMediaPaint(probe.naturalWidth > 0 ? \"ready\" : \"failed\");\n return () => {\n cancelled = true;\n probe.onload = null;\n probe.onerror = null;\n };\n }, [mediaUrl, mediaPoster, resolved, wantsMedia]);\n\n // The element stays mounted while pending — that is what loads it. Only a\n // confirmed failure takes it back out.\n const showMedia = wantsMedia && mediaPaint !== \"failed\";\n const showTreatment = wantsMedia && mediaPaint === \"ready\";\n const resolvedPosition = resolveMediaPosition(mediaPosition);\n const resolvedTreatment = resolveMediaTreatment(mediaTreatment);\n const treatmentLayers = getMediaTreatmentLayers(resolvedTreatment, textAnchor);\n\n const gradSeed =\n typeof seed === \"number\"\n ? seed\n : typeof seed === \"string\"\n ? seed.split(\"\").reduce((acc, c) => acc + c.charCodeAt(0), 0)\n : 0;\n\n const bgTransform = getBackgroundTransform(\n backgroundEffect,\n progress,\n beatIntensity,\n );\n\n return (\n <>\n {(!hasExternalVideoBackdrop || externalVideoFailed) && (\n <BrandGradientOverlay\n style={style}\n progress={progress}\n sceneDuration={sceneDuration}\n seed={gradSeed}\n />\n )}\n\n {showMedia && !hasExternalVideoBackdrop &&\n (resolved === \"video\" ? (\n <SceneVideoBackdrop\n mediaUrl={mediaUrl}\n mediaPoster={mediaPoster}\n mediaPosition={mediaPosition}\n backgroundEffect={backgroundEffect}\n progress={progress}\n beatIntensity={beatIntensity}\n isPlaying={isPlaying}\n onReady={() => setMediaPaint(\"ready\")}\n onError={() => setMediaPaint(\"failed\")}\n />\n ) : (\n <img\n src={mediaUrl}\n alt=\"\"\n aria-hidden=\"true\"\n draggable={false}\n data-media-position={mediaPosition}\n style={{\n position: \"absolute\",\n inset: 0,\n transform: bgTransform.transform,\n transformOrigin: bgTransform.transformOrigin,\n width: \"100%\",\n height: \"100%\",\n objectFit: \"cover\",\n objectPosition: resolvedPosition,\n }}\n />\n ))}\n\n {(hasExternalVideoBackdrop ? externalVideoReady : showTreatment) && !externalVideoFailed &&\n treatmentLayers.map((layer) => (\n <div\n key={layer.id}\n data-media-treatment={resolvedTreatment}\n data-media-overlay={layer.id}\n style={{\n position: \"absolute\",\n inset: 0,\n background: layer.background,\n pointerEvents: \"none\",\n ...layer.style,\n }}\n />\n ))}\n </>\n );\n};\n"
|
|
32
|
+
"content": "/**\n * Shared photo/video backdrop for built-in and customer-owned templates.\n * Media is host-resolved; this component never searches or generates assets.\n * The base is fixed black. Photos and videos cover it when available, and\n * optional scrims appear only once the media can paint. The cinematic built-ins\n * request no scrim; custom templates can choose a treatment and text anchor.\n * The retained internal \"gradient\" media sentinel selects the black base and\n * ignores mediaUrl. It is not a built-in authoring mode or brand-color control.\n * Video playback can be owned by the player's persistent external backdrop.\n */\n\nimport React, { useEffect, useState } from \"react\";\nimport {\n hasSceneMedia,\n resolveMediaType,\n type ResolvedMediaType,\n} from \"./media-source\";\nimport type { TemplateStyle } from \"../template-context\";\nimport { BrandGradientOverlay } from \"../backgrounds\";\nimport { getBackgroundTransform } from \"../backgrounds\";\nimport { useExternalVideoBackdrop } from \"./external-video-backdrop\";\nimport { resolveMediaPosition, type MediaPosition } from \"./media-position\";\nimport { SceneVideoBackdrop } from \"./scene-video-backdrop\";\n\nexport { hasSceneMedia, resolveMediaType };\nexport { resolveMediaPosition } from \"./media-position\";\nexport { SceneVideoBackdrop } from \"./scene-video-backdrop\";\nexport type { ResolvedMediaType };\n\nexport type { MediaPosition };\nexport type MediaTreatment = \"none\" | \"subtle\" | \"cinematic\" | \"text-safe\";\n\nexport function resolveMediaTreatment(value: string): MediaTreatment {\n return value === \"none\" || value === \"subtle\" || value === \"text-safe\" ? value : \"cinematic\";\n}\n\nexport interface MediaTreatmentLayer {\n id: \"vignette\" | \"center-scrim\" | \"bottom-scrim\";\n background: string;\n style?: React.CSSProperties;\n}\n\n/**\n * Where the template puts its type. The scrim is shaped to the copy, not to\n * the frame: darkening picture the type never touches costs contrast in the\n * photo and buys no legibility. \"full\" is the conservative default for\n * templates that have not declared an anchor.\n */\nexport type MediaTextAnchor = \"center\" | \"bottom\" | \"full\";\n\n/**\n * Smoothstep-sampled alpha stops between `start`% and `end`% of the gradient\n * box, held at full strength before `start` and after `end`.\n *\n * A two-stop `rgba(0,0,0,a) → transparent` scrim ramps alpha linearly, so it\n * ends with a constant slope. Lateral inhibition in the eye amplifies that\n * slope discontinuity into a visible band — the grey bar cutting across the\n * frame that makes an overlay read as an overlay. Smoothstep flattens the\n * curve at both ends, so the scrim holds where the type sits and then leaves\n * without an edge: the same peak coverage over the copy, noticeably less of\n * the picture spent getting there.\n */\nconst SCRIM_STOP_COUNT = 7;\n\nfunction smoothstep(t: number): number {\n return t * t * (3 - 2 * t);\n}\n\nfunction easedStops(\n peakAlpha: number,\n start: number,\n end: number,\n direction: \"fade-out\" | \"fade-in\",\n): string {\n const alphaAt = (t: number): string => {\n const eased = direction === \"fade-out\" ? 1 - smoothstep(t) : smoothstep(t);\n return `rgba(0,0,0,${Number((peakAlpha * eased).toFixed(3))})`;\n };\n const stops: string[] = [];\n if (start > 0) stops.push(`${alphaAt(0)} 0%`);\n for (let i = 0; i < SCRIM_STOP_COUNT; i += 1) {\n const t = i / (SCRIM_STOP_COUNT - 1);\n const position = Number((start + (end - start) * t).toFixed(2));\n stops.push(`${alphaAt(t)} ${position}%`);\n }\n if (end < 100) stops.push(`${alphaAt(1)} 100%`);\n return stops.join(\", \");\n}\n\n/**\n * Export-safe contrast recipes. Overlays only: SVG capture cannot rely on CSS\n * filters, so a blur-behind-text plate is off the table.\n *\n * The scrims deliberately stop short of solving legibility on their own. A\n * uniform darkening strong enough to carry white type over a blown-out sky\n * needs roughly 0.8 alpha — at that point the photo is a texture, not a\n * picture. The cheaper half of the job belongs to the type: a per-glyph halo\n * (MEDIA_TEXT_SHADOW) buys local contrast exactly where it is needed and\n * costs the image nothing. Scrim for the plate, halo for the glyph.\n */\nexport function getMediaTreatmentLayers(\n value: string,\n anchor: MediaTextAnchor = \"full\",\n): MediaTreatmentLayer[] {\n const treatment = resolveMediaTreatment(value);\n // Nothing over the picture at all. Every other treatment exists to carry\n // type across a photograph; where the scene has no type - a generated clip\n // that is the whole point of the beat - even a vignette is something the\n // viewer did not ask to look through.\n if (treatment === \"none\") return [];\n const vignette: MediaTreatmentLayer = {\n id: \"vignette\",\n background:\n treatment === \"subtle\"\n ? `radial-gradient(ellipse at center, ${easedStops(0.28, 45, 100, \"fade-in\")})`\n : `radial-gradient(ellipse at center, ${easedStops(0.72, 32, 100, \"fade-in\")})`,\n };\n if (treatment === \"subtle\") return [vignette];\n\n const textSafe = treatment === \"text-safe\";\n const layers: MediaTreatmentLayer[] = [vignette];\n\n if (anchor !== \"bottom\") {\n layers.push({\n id: \"center-scrim\",\n background: textSafe\n ? `radial-gradient(ellipse 92% 58% at 50% 50%, ${easedStops(0.46, 34, 90, \"fade-out\")})`\n : `radial-gradient(ellipse 88% 52% at 50% 50%, ${easedStops(0.26, 30, 88, \"fade-out\")})`,\n });\n }\n\n if (anchor !== \"center\") {\n layers.push({\n id: \"bottom-scrim\",\n background: `linear-gradient(to top, ${easedStops(textSafe ? 0.64 : 0.5, 8, 100, \"fade-out\")})`,\n style: { top: \"55%\" },\n });\n }\n\n return layers;\n}\n\n/**\n * Whether the backdrop is actually painting, which is what decides if a scrim\n * is earned. \"pending\" is a browser-only state: static and export renders\n * never run effects and never wait on a network, so they start (and stay)\n * ready and their output is unchanged.\n */\ntype MediaPaintState = \"pending\" | \"ready\" | \"failed\";\n\nfunction initialMediaPaint(\n wantsMedia: boolean,\n resolved: ResolvedMediaType,\n mediaUrl: string,\n mediaPoster: string | undefined,\n): MediaPaintState {\n if (typeof window === \"undefined\") return \"ready\";\n if (!wantsMedia) return \"ready\";\n // A poster paints the video's frame immediately, so the scene is already\n // showing footage even though the stream is still decoding.\n if (resolved === \"video\") return mediaPoster ? \"ready\" : \"pending\";\n if (typeof Image === \"undefined\") return \"ready\";\n // Preloaded or browser-cached media decodes synchronously. Reporting it\n // ready on the first render keeps the common mid-playback case free of a\n // black-then-photo flicker.\n const cached = new Image();\n cached.src = mediaUrl;\n return cached.complete && cached.naturalWidth > 0 ? \"ready\" : \"pending\";\n}\n\nexport function getMediaBackgroundProps(variables: Record<string, unknown>) {\n return {\n mediaUrl: String(variables.mediaUrl || \"\"),\n mediaType: String(variables.mediaType || \"auto\"),\n mediaPoster: String(variables.mediaPoster || \"\"),\n mediaPosition: String(variables.mediaPosition || \"center\"),\n mediaTreatment: String(variables.mediaTreatment || \"cinematic\"),\n };\n}\n\nexport interface SceneBackgroundProps {\n style: TemplateStyle;\n progress: number;\n sceneDuration?: number;\n width: number;\n height: number;\n mediaUrl?: string;\n mediaType?: string;\n /** Still image URL shown while the <video> backdrop decodes its first\n * frame. Without it the element renders transparent during the\n * decode window and the black base shows through. */\n mediaPoster?: string;\n /** Cover-crop focal anchor. Keeps the important edge/subject visible. */\n mediaPosition?: string;\n /** Overlay recipe: subtle, cinematic, or stronger text-safe contrast. */\n mediaTreatment?: string;\n /** Where this template's copy sits, so the scrim is shaped to the type\n * instead of to the frame. Defaults to \"full\" (scrim both the middle and\n * the lower third) for templates that have not declared an anchor. */\n textAnchor?: MediaTextAnchor;\n /** Background motion effect (drift / pulse / Ken Burns). Applied to the photo/video. */\n backgroundEffect?: string;\n /** Retained seed input for customer-owned backdrop compositions. */\n seed?: number | string;\n /** Pause video when preview is paused. Defaults to true (export path). */\n isPlaying?: boolean;\n beatIntensity?: number;\n}\n\nexport const SceneBackground: React.FC<SceneBackgroundProps> = ({\n style,\n progress,\n sceneDuration,\n width: _width, // accepted for symmetry; not currently used in render\n height: _height,\n mediaUrl = \"\",\n mediaType = \"auto\",\n mediaPoster,\n mediaPosition = \"center\",\n mediaTreatment = \"cinematic\",\n textAnchor = \"full\",\n backgroundEffect,\n seed,\n isPlaying = true,\n beatIntensity = 0,\n}) => {\n void _width;\n void _height;\n const resolved = resolveMediaType(mediaType, mediaUrl);\n const wantsMedia = resolved !== \"gradient\" && !!mediaUrl;\n const externalVideoBackdrop = useExternalVideoBackdrop();\n const hasExternalVideoBackdrop = externalVideoBackdrop !== false && resolved === \"video\";\n const externalVideoFailed = externalVideoBackdrop === \"fallback\" && resolved === \"video\";\n const externalVideoReady = externalVideoBackdrop === \"ready\" && resolved === \"video\";\n\n // Apply picture and scrim together. A loading or failed asset keeps the\n // fixed black base instead of painting contrast treatment over empty media.\n const [mediaPaint, setMediaPaint] = useState<MediaPaintState>(() =>\n initialMediaPaint(wantsMedia, resolved, mediaUrl, mediaPoster),\n );\n useEffect(() => {\n setMediaPaint(initialMediaPaint(wantsMedia, resolved, mediaUrl, mediaPoster));\n // Video reports its own paint through onLoadedData / onError below.\n if (!wantsMedia || resolved !== \"photo\") return;\n if (typeof Image === \"undefined\") return;\n let cancelled = false;\n const probe = new Image();\n probe.onload = () => {\n if (!cancelled) setMediaPaint(\"ready\");\n };\n probe.onerror = () => {\n if (!cancelled) setMediaPaint(\"failed\");\n };\n probe.src = mediaUrl;\n if (probe.complete) setMediaPaint(probe.naturalWidth > 0 ? \"ready\" : \"failed\");\n return () => {\n cancelled = true;\n probe.onload = null;\n probe.onerror = null;\n };\n }, [mediaUrl, mediaPoster, resolved, wantsMedia]);\n\n // The element stays mounted while pending — that is what loads it. Only a\n // confirmed failure takes it back out.\n const showMedia = wantsMedia && mediaPaint !== \"failed\";\n const showTreatment = wantsMedia && mediaPaint === \"ready\";\n const resolvedPosition = resolveMediaPosition(mediaPosition);\n const resolvedTreatment = resolveMediaTreatment(mediaTreatment);\n const treatmentLayers = getMediaTreatmentLayers(resolvedTreatment, textAnchor);\n\n const gradSeed =\n typeof seed === \"number\"\n ? seed\n : typeof seed === \"string\"\n ? seed.split(\"\").reduce((acc, c) => acc + c.charCodeAt(0), 0)\n : 0;\n\n const bgTransform = getBackgroundTransform(\n backgroundEffect,\n progress,\n beatIntensity,\n );\n\n return (\n <>\n {(!hasExternalVideoBackdrop || externalVideoFailed) && (\n <BrandGradientOverlay\n style={style}\n progress={progress}\n sceneDuration={sceneDuration}\n seed={gradSeed}\n />\n )}\n\n {showMedia && !hasExternalVideoBackdrop &&\n (resolved === \"video\" ? (\n <SceneVideoBackdrop\n mediaUrl={mediaUrl}\n mediaPoster={mediaPoster}\n mediaPosition={mediaPosition}\n backgroundEffect={backgroundEffect}\n progress={progress}\n sceneDuration={sceneDuration}\n beatIntensity={beatIntensity}\n isPlaying={isPlaying}\n onReady={() => setMediaPaint(\"ready\")}\n onError={() => setMediaPaint(\"failed\")}\n />\n ) : (\n <img\n src={mediaUrl}\n alt=\"\"\n aria-hidden=\"true\"\n draggable={false}\n data-media-position={mediaPosition}\n style={{\n position: \"absolute\",\n inset: 0,\n transform: bgTransform.transform,\n transformOrigin: bgTransform.transformOrigin,\n width: \"100%\",\n height: \"100%\",\n objectFit: \"cover\",\n objectPosition: resolvedPosition,\n }}\n />\n ))}\n\n {(hasExternalVideoBackdrop ? externalVideoReady : showTreatment) && !externalVideoFailed &&\n treatmentLayers.map((layer) => (\n <div\n key={layer.id}\n data-media-treatment={resolvedTreatment}\n data-media-overlay={layer.id}\n style={{\n position: \"absolute\",\n inset: 0,\n background: layer.background,\n pointerEvents: \"none\",\n ...layer.style,\n }}\n />\n ))}\n </>\n );\n};\n"
|
|
33
33
|
},
|
|
34
34
|
{
|
|
35
35
|
"path": "src/visual-system/scene-templates/media-source.ts",
|
|
@@ -42,7 +42,7 @@
|
|
|
42
42
|
"path": "src/visual-system/scene-templates/scene-background.tsx",
|
|
43
43
|
"type": "registry:component",
|
|
44
44
|
"target": "vanillasky/scene-templates/scene-background.tsx",
|
|
45
|
-
"content": "/**\n * Shared photo/video backdrop for built-in and customer-owned templates.\n * Media is host-resolved; this component never searches or generates assets.\n * The base is fixed black. Photos and videos cover it when available, and\n * optional scrims appear only once the media can paint. The cinematic built-ins\n * request no scrim; custom templates can choose a treatment and text anchor.\n * The retained internal \"gradient\" media sentinel selects the black base and\n * ignores mediaUrl. It is not a built-in authoring mode or brand-color control.\n * Video playback can be owned by the player's persistent external backdrop.\n */\n\nimport React, { useEffect, useState } from \"react\";\nimport {\n hasSceneMedia,\n resolveMediaType,\n type ResolvedMediaType,\n} from \"./media-source\";\nimport type { TemplateStyle } from \"../template-context\";\nimport { BrandGradientOverlay } from \"../backgrounds\";\nimport { getBackgroundTransform } from \"../backgrounds\";\nimport { useExternalVideoBackdrop } from \"./external-video-backdrop\";\nimport { resolveMediaPosition, type MediaPosition } from \"./media-position\";\nimport { SceneVideoBackdrop } from \"./scene-video-backdrop\";\n\nexport { hasSceneMedia, resolveMediaType };\nexport { resolveMediaPosition } from \"./media-position\";\nexport { SceneVideoBackdrop } from \"./scene-video-backdrop\";\nexport type { ResolvedMediaType };\n\nexport type { MediaPosition };\nexport type MediaTreatment = \"none\" | \"subtle\" | \"cinematic\" | \"text-safe\";\n\nexport function resolveMediaTreatment(value: string): MediaTreatment {\n return value === \"none\" || value === \"subtle\" || value === \"text-safe\" ? value : \"cinematic\";\n}\n\nexport interface MediaTreatmentLayer {\n id: \"vignette\" | \"center-scrim\" | \"bottom-scrim\";\n background: string;\n style?: React.CSSProperties;\n}\n\n/**\n * Where the template puts its type. The scrim is shaped to the copy, not to\n * the frame: darkening picture the type never touches costs contrast in the\n * photo and buys no legibility. \"full\" is the conservative default for\n * templates that have not declared an anchor.\n */\nexport type MediaTextAnchor = \"center\" | \"bottom\" | \"full\";\n\n/**\n * Smoothstep-sampled alpha stops between `start`% and `end`% of the gradient\n * box, held at full strength before `start` and after `end`.\n *\n * A two-stop `rgba(0,0,0,a) → transparent` scrim ramps alpha linearly, so it\n * ends with a constant slope. Lateral inhibition in the eye amplifies that\n * slope discontinuity into a visible band — the grey bar cutting across the\n * frame that makes an overlay read as an overlay. Smoothstep flattens the\n * curve at both ends, so the scrim holds where the type sits and then leaves\n * without an edge: the same peak coverage over the copy, noticeably less of\n * the picture spent getting there.\n */\nconst SCRIM_STOP_COUNT = 7;\n\nfunction smoothstep(t: number): number {\n return t * t * (3 - 2 * t);\n}\n\nfunction easedStops(\n peakAlpha: number,\n start: number,\n end: number,\n direction: \"fade-out\" | \"fade-in\",\n): string {\n const alphaAt = (t: number): string => {\n const eased = direction === \"fade-out\" ? 1 - smoothstep(t) : smoothstep(t);\n return `rgba(0,0,0,${Number((peakAlpha * eased).toFixed(3))})`;\n };\n const stops: string[] = [];\n if (start > 0) stops.push(`${alphaAt(0)} 0%`);\n for (let i = 0; i < SCRIM_STOP_COUNT; i += 1) {\n const t = i / (SCRIM_STOP_COUNT - 1);\n const position = Number((start + (end - start) * t).toFixed(2));\n stops.push(`${alphaAt(t)} ${position}%`);\n }\n if (end < 100) stops.push(`${alphaAt(1)} 100%`);\n return stops.join(\", \");\n}\n\n/**\n * Export-safe contrast recipes. Overlays only: SVG capture cannot rely on CSS\n * filters, so a blur-behind-text plate is off the table.\n *\n * The scrims deliberately stop short of solving legibility on their own. A\n * uniform darkening strong enough to carry white type over a blown-out sky\n * needs roughly 0.8 alpha — at that point the photo is a texture, not a\n * picture. The cheaper half of the job belongs to the type: a per-glyph halo\n * (MEDIA_TEXT_SHADOW) buys local contrast exactly where it is needed and\n * costs the image nothing. Scrim for the plate, halo for the glyph.\n */\nexport function getMediaTreatmentLayers(\n value: string,\n anchor: MediaTextAnchor = \"full\",\n): MediaTreatmentLayer[] {\n const treatment = resolveMediaTreatment(value);\n // Nothing over the picture at all. Every other treatment exists to carry\n // type across a photograph; where the scene has no type - a generated clip\n // that is the whole point of the beat - even a vignette is something the\n // viewer did not ask to look through.\n if (treatment === \"none\") return [];\n const vignette: MediaTreatmentLayer = {\n id: \"vignette\",\n background:\n treatment === \"subtle\"\n ? `radial-gradient(ellipse at center, ${easedStops(0.28, 45, 100, \"fade-in\")})`\n : `radial-gradient(ellipse at center, ${easedStops(0.72, 32, 100, \"fade-in\")})`,\n };\n if (treatment === \"subtle\") return [vignette];\n\n const textSafe = treatment === \"text-safe\";\n const layers: MediaTreatmentLayer[] = [vignette];\n\n if (anchor !== \"bottom\") {\n layers.push({\n id: \"center-scrim\",\n background: textSafe\n ? `radial-gradient(ellipse 92% 58% at 50% 50%, ${easedStops(0.46, 34, 90, \"fade-out\")})`\n : `radial-gradient(ellipse 88% 52% at 50% 50%, ${easedStops(0.26, 30, 88, \"fade-out\")})`,\n });\n }\n\n if (anchor !== \"center\") {\n layers.push({\n id: \"bottom-scrim\",\n background: `linear-gradient(to top, ${easedStops(textSafe ? 0.64 : 0.5, 8, 100, \"fade-out\")})`,\n style: { top: \"55%\" },\n });\n }\n\n return layers;\n}\n\n/**\n * Whether the backdrop is actually painting, which is what decides if a scrim\n * is earned. \"pending\" is a browser-only state: static and export renders\n * never run effects and never wait on a network, so they start (and stay)\n * ready and their output is unchanged.\n */\ntype MediaPaintState = \"pending\" | \"ready\" | \"failed\";\n\nfunction initialMediaPaint(\n wantsMedia: boolean,\n resolved: ResolvedMediaType,\n mediaUrl: string,\n mediaPoster: string | undefined,\n): MediaPaintState {\n if (typeof window === \"undefined\") return \"ready\";\n if (!wantsMedia) return \"ready\";\n // A poster paints the video's frame immediately, so the scene is already\n // showing footage even though the stream is still decoding.\n if (resolved === \"video\") return mediaPoster ? \"ready\" : \"pending\";\n if (typeof Image === \"undefined\") return \"ready\";\n // Preloaded or browser-cached media decodes synchronously. Reporting it\n // ready on the first render keeps the common mid-playback case free of a\n // black-then-photo flicker.\n const cached = new Image();\n cached.src = mediaUrl;\n return cached.complete && cached.naturalWidth > 0 ? \"ready\" : \"pending\";\n}\n\nexport function getMediaBackgroundProps(variables: Record<string, unknown>) {\n return {\n mediaUrl: String(variables.mediaUrl || \"\"),\n mediaType: String(variables.mediaType || \"auto\"),\n mediaPoster: String(variables.mediaPoster || \"\"),\n mediaPosition: String(variables.mediaPosition || \"center\"),\n mediaTreatment: String(variables.mediaTreatment || \"cinematic\"),\n };\n}\n\nexport interface SceneBackgroundProps {\n style: TemplateStyle;\n progress: number;\n sceneDuration?: number;\n width: number;\n height: number;\n mediaUrl?: string;\n mediaType?: string;\n /** Still image URL shown while the <video> backdrop decodes its first\n * frame. Without it the element renders transparent during the\n * decode window and the black base shows through. */\n mediaPoster?: string;\n /** Cover-crop focal anchor. Keeps the important edge/subject visible. */\n mediaPosition?: string;\n /** Overlay recipe: subtle, cinematic, or stronger text-safe contrast. */\n mediaTreatment?: string;\n /** Where this template's copy sits, so the scrim is shaped to the type\n * instead of to the frame. Defaults to \"full\" (scrim both the middle and\n * the lower third) for templates that have not declared an anchor. */\n textAnchor?: MediaTextAnchor;\n /** Background motion effect (drift / pulse / Ken Burns). Applied to the photo/video. */\n backgroundEffect?: string;\n /** Retained seed input for customer-owned backdrop compositions. */\n seed?: number | string;\n /** Pause video when preview is paused. Defaults to true (export path). */\n isPlaying?: boolean;\n beatIntensity?: number;\n}\n\nexport const SceneBackground: React.FC<SceneBackgroundProps> = ({\n style,\n progress,\n sceneDuration,\n width: _width, // accepted for symmetry; not currently used in render\n height: _height,\n mediaUrl = \"\",\n mediaType = \"auto\",\n mediaPoster,\n mediaPosition = \"center\",\n mediaTreatment = \"cinematic\",\n textAnchor = \"full\",\n backgroundEffect,\n seed,\n isPlaying = true,\n beatIntensity = 0,\n}) => {\n void _width;\n void _height;\n const resolved = resolveMediaType(mediaType, mediaUrl);\n const wantsMedia = resolved !== \"gradient\" && !!mediaUrl;\n const externalVideoBackdrop = useExternalVideoBackdrop();\n const hasExternalVideoBackdrop = externalVideoBackdrop !== false && resolved === \"video\";\n const externalVideoFailed = externalVideoBackdrop === \"fallback\" && resolved === \"video\";\n const externalVideoReady = externalVideoBackdrop === \"ready\" && resolved === \"video\";\n\n // Apply picture and scrim together. A loading or failed asset keeps the\n // fixed black base instead of painting contrast treatment over empty media.\n const [mediaPaint, setMediaPaint] = useState<MediaPaintState>(() =>\n initialMediaPaint(wantsMedia, resolved, mediaUrl, mediaPoster),\n );\n useEffect(() => {\n setMediaPaint(initialMediaPaint(wantsMedia, resolved, mediaUrl, mediaPoster));\n // Video reports its own paint through onLoadedData / onError below.\n if (!wantsMedia || resolved !== \"photo\") return;\n if (typeof Image === \"undefined\") return;\n let cancelled = false;\n const probe = new Image();\n probe.onload = () => {\n if (!cancelled) setMediaPaint(\"ready\");\n };\n probe.onerror = () => {\n if (!cancelled) setMediaPaint(\"failed\");\n };\n probe.src = mediaUrl;\n if (probe.complete) setMediaPaint(probe.naturalWidth > 0 ? \"ready\" : \"failed\");\n return () => {\n cancelled = true;\n probe.onload = null;\n probe.onerror = null;\n };\n }, [mediaUrl, mediaPoster, resolved, wantsMedia]);\n\n // The element stays mounted while pending — that is what loads it. Only a\n // confirmed failure takes it back out.\n const showMedia = wantsMedia && mediaPaint !== \"failed\";\n const showTreatment = wantsMedia && mediaPaint === \"ready\";\n const resolvedPosition = resolveMediaPosition(mediaPosition);\n const resolvedTreatment = resolveMediaTreatment(mediaTreatment);\n const treatmentLayers = getMediaTreatmentLayers(resolvedTreatment, textAnchor);\n\n const gradSeed =\n typeof seed === \"number\"\n ? seed\n : typeof seed === \"string\"\n ? seed.split(\"\").reduce((acc, c) => acc + c.charCodeAt(0), 0)\n : 0;\n\n const bgTransform = getBackgroundTransform(\n backgroundEffect,\n progress,\n beatIntensity,\n );\n\n return (\n <>\n {(!hasExternalVideoBackdrop || externalVideoFailed) && (\n <BrandGradientOverlay\n style={style}\n progress={progress}\n sceneDuration={sceneDuration}\n seed={gradSeed}\n />\n )}\n\n {showMedia && !hasExternalVideoBackdrop &&\n (resolved === \"video\" ? (\n <SceneVideoBackdrop\n mediaUrl={mediaUrl}\n mediaPoster={mediaPoster}\n mediaPosition={mediaPosition}\n backgroundEffect={backgroundEffect}\n progress={progress}\n beatIntensity={beatIntensity}\n isPlaying={isPlaying}\n onReady={() => setMediaPaint(\"ready\")}\n onError={() => setMediaPaint(\"failed\")}\n />\n ) : (\n <img\n src={mediaUrl}\n alt=\"\"\n aria-hidden=\"true\"\n draggable={false}\n data-media-position={mediaPosition}\n style={{\n position: \"absolute\",\n inset: 0,\n transform: bgTransform.transform,\n transformOrigin: bgTransform.transformOrigin,\n width: \"100%\",\n height: \"100%\",\n objectFit: \"cover\",\n objectPosition: resolvedPosition,\n }}\n />\n ))}\n\n {(hasExternalVideoBackdrop ? externalVideoReady : showTreatment) && !externalVideoFailed &&\n treatmentLayers.map((layer) => (\n <div\n key={layer.id}\n data-media-treatment={resolvedTreatment}\n data-media-overlay={layer.id}\n style={{\n position: \"absolute\",\n inset: 0,\n background: layer.background,\n pointerEvents: \"none\",\n ...layer.style,\n }}\n />\n ))}\n </>\n );\n};\n"
|
|
45
|
+
"content": "/**\n * Shared photo/video backdrop for built-in and customer-owned templates.\n * Media is host-resolved; this component never searches or generates assets.\n * The base is fixed black. Photos and videos cover it when available, and\n * optional scrims appear only once the media can paint. The cinematic built-ins\n * request no scrim; custom templates can choose a treatment and text anchor.\n * The retained internal \"gradient\" media sentinel selects the black base and\n * ignores mediaUrl. It is not a built-in authoring mode or brand-color control.\n * Video playback can be owned by the player's persistent external backdrop.\n */\n\nimport React, { useEffect, useState } from \"react\";\nimport {\n hasSceneMedia,\n resolveMediaType,\n type ResolvedMediaType,\n} from \"./media-source\";\nimport type { TemplateStyle } from \"../template-context\";\nimport { BrandGradientOverlay } from \"../backgrounds\";\nimport { getBackgroundTransform } from \"../backgrounds\";\nimport { useExternalVideoBackdrop } from \"./external-video-backdrop\";\nimport { resolveMediaPosition, type MediaPosition } from \"./media-position\";\nimport { SceneVideoBackdrop } from \"./scene-video-backdrop\";\n\nexport { hasSceneMedia, resolveMediaType };\nexport { resolveMediaPosition } from \"./media-position\";\nexport { SceneVideoBackdrop } from \"./scene-video-backdrop\";\nexport type { ResolvedMediaType };\n\nexport type { MediaPosition };\nexport type MediaTreatment = \"none\" | \"subtle\" | \"cinematic\" | \"text-safe\";\n\nexport function resolveMediaTreatment(value: string): MediaTreatment {\n return value === \"none\" || value === \"subtle\" || value === \"text-safe\" ? value : \"cinematic\";\n}\n\nexport interface MediaTreatmentLayer {\n id: \"vignette\" | \"center-scrim\" | \"bottom-scrim\";\n background: string;\n style?: React.CSSProperties;\n}\n\n/**\n * Where the template puts its type. The scrim is shaped to the copy, not to\n * the frame: darkening picture the type never touches costs contrast in the\n * photo and buys no legibility. \"full\" is the conservative default for\n * templates that have not declared an anchor.\n */\nexport type MediaTextAnchor = \"center\" | \"bottom\" | \"full\";\n\n/**\n * Smoothstep-sampled alpha stops between `start`% and `end`% of the gradient\n * box, held at full strength before `start` and after `end`.\n *\n * A two-stop `rgba(0,0,0,a) → transparent` scrim ramps alpha linearly, so it\n * ends with a constant slope. Lateral inhibition in the eye amplifies that\n * slope discontinuity into a visible band — the grey bar cutting across the\n * frame that makes an overlay read as an overlay. Smoothstep flattens the\n * curve at both ends, so the scrim holds where the type sits and then leaves\n * without an edge: the same peak coverage over the copy, noticeably less of\n * the picture spent getting there.\n */\nconst SCRIM_STOP_COUNT = 7;\n\nfunction smoothstep(t: number): number {\n return t * t * (3 - 2 * t);\n}\n\nfunction easedStops(\n peakAlpha: number,\n start: number,\n end: number,\n direction: \"fade-out\" | \"fade-in\",\n): string {\n const alphaAt = (t: number): string => {\n const eased = direction === \"fade-out\" ? 1 - smoothstep(t) : smoothstep(t);\n return `rgba(0,0,0,${Number((peakAlpha * eased).toFixed(3))})`;\n };\n const stops: string[] = [];\n if (start > 0) stops.push(`${alphaAt(0)} 0%`);\n for (let i = 0; i < SCRIM_STOP_COUNT; i += 1) {\n const t = i / (SCRIM_STOP_COUNT - 1);\n const position = Number((start + (end - start) * t).toFixed(2));\n stops.push(`${alphaAt(t)} ${position}%`);\n }\n if (end < 100) stops.push(`${alphaAt(1)} 100%`);\n return stops.join(\", \");\n}\n\n/**\n * Export-safe contrast recipes. Overlays only: SVG capture cannot rely on CSS\n * filters, so a blur-behind-text plate is off the table.\n *\n * The scrims deliberately stop short of solving legibility on their own. A\n * uniform darkening strong enough to carry white type over a blown-out sky\n * needs roughly 0.8 alpha — at that point the photo is a texture, not a\n * picture. The cheaper half of the job belongs to the type: a per-glyph halo\n * (MEDIA_TEXT_SHADOW) buys local contrast exactly where it is needed and\n * costs the image nothing. Scrim for the plate, halo for the glyph.\n */\nexport function getMediaTreatmentLayers(\n value: string,\n anchor: MediaTextAnchor = \"full\",\n): MediaTreatmentLayer[] {\n const treatment = resolveMediaTreatment(value);\n // Nothing over the picture at all. Every other treatment exists to carry\n // type across a photograph; where the scene has no type - a generated clip\n // that is the whole point of the beat - even a vignette is something the\n // viewer did not ask to look through.\n if (treatment === \"none\") return [];\n const vignette: MediaTreatmentLayer = {\n id: \"vignette\",\n background:\n treatment === \"subtle\"\n ? `radial-gradient(ellipse at center, ${easedStops(0.28, 45, 100, \"fade-in\")})`\n : `radial-gradient(ellipse at center, ${easedStops(0.72, 32, 100, \"fade-in\")})`,\n };\n if (treatment === \"subtle\") return [vignette];\n\n const textSafe = treatment === \"text-safe\";\n const layers: MediaTreatmentLayer[] = [vignette];\n\n if (anchor !== \"bottom\") {\n layers.push({\n id: \"center-scrim\",\n background: textSafe\n ? `radial-gradient(ellipse 92% 58% at 50% 50%, ${easedStops(0.46, 34, 90, \"fade-out\")})`\n : `radial-gradient(ellipse 88% 52% at 50% 50%, ${easedStops(0.26, 30, 88, \"fade-out\")})`,\n });\n }\n\n if (anchor !== \"center\") {\n layers.push({\n id: \"bottom-scrim\",\n background: `linear-gradient(to top, ${easedStops(textSafe ? 0.64 : 0.5, 8, 100, \"fade-out\")})`,\n style: { top: \"55%\" },\n });\n }\n\n return layers;\n}\n\n/**\n * Whether the backdrop is actually painting, which is what decides if a scrim\n * is earned. \"pending\" is a browser-only state: static and export renders\n * never run effects and never wait on a network, so they start (and stay)\n * ready and their output is unchanged.\n */\ntype MediaPaintState = \"pending\" | \"ready\" | \"failed\";\n\nfunction initialMediaPaint(\n wantsMedia: boolean,\n resolved: ResolvedMediaType,\n mediaUrl: string,\n mediaPoster: string | undefined,\n): MediaPaintState {\n if (typeof window === \"undefined\") return \"ready\";\n if (!wantsMedia) return \"ready\";\n // A poster paints the video's frame immediately, so the scene is already\n // showing footage even though the stream is still decoding.\n if (resolved === \"video\") return mediaPoster ? \"ready\" : \"pending\";\n if (typeof Image === \"undefined\") return \"ready\";\n // Preloaded or browser-cached media decodes synchronously. Reporting it\n // ready on the first render keeps the common mid-playback case free of a\n // black-then-photo flicker.\n const cached = new Image();\n cached.src = mediaUrl;\n return cached.complete && cached.naturalWidth > 0 ? \"ready\" : \"pending\";\n}\n\nexport function getMediaBackgroundProps(variables: Record<string, unknown>) {\n return {\n mediaUrl: String(variables.mediaUrl || \"\"),\n mediaType: String(variables.mediaType || \"auto\"),\n mediaPoster: String(variables.mediaPoster || \"\"),\n mediaPosition: String(variables.mediaPosition || \"center\"),\n mediaTreatment: String(variables.mediaTreatment || \"cinematic\"),\n };\n}\n\nexport interface SceneBackgroundProps {\n style: TemplateStyle;\n progress: number;\n sceneDuration?: number;\n width: number;\n height: number;\n mediaUrl?: string;\n mediaType?: string;\n /** Still image URL shown while the <video> backdrop decodes its first\n * frame. Without it the element renders transparent during the\n * decode window and the black base shows through. */\n mediaPoster?: string;\n /** Cover-crop focal anchor. Keeps the important edge/subject visible. */\n mediaPosition?: string;\n /** Overlay recipe: subtle, cinematic, or stronger text-safe contrast. */\n mediaTreatment?: string;\n /** Where this template's copy sits, so the scrim is shaped to the type\n * instead of to the frame. Defaults to \"full\" (scrim both the middle and\n * the lower third) for templates that have not declared an anchor. */\n textAnchor?: MediaTextAnchor;\n /** Background motion effect (drift / pulse / Ken Burns). Applied to the photo/video. */\n backgroundEffect?: string;\n /** Retained seed input for customer-owned backdrop compositions. */\n seed?: number | string;\n /** Pause video when preview is paused. Defaults to true (export path). */\n isPlaying?: boolean;\n beatIntensity?: number;\n}\n\nexport const SceneBackground: React.FC<SceneBackgroundProps> = ({\n style,\n progress,\n sceneDuration,\n width: _width, // accepted for symmetry; not currently used in render\n height: _height,\n mediaUrl = \"\",\n mediaType = \"auto\",\n mediaPoster,\n mediaPosition = \"center\",\n mediaTreatment = \"cinematic\",\n textAnchor = \"full\",\n backgroundEffect,\n seed,\n isPlaying = true,\n beatIntensity = 0,\n}) => {\n void _width;\n void _height;\n const resolved = resolveMediaType(mediaType, mediaUrl);\n const wantsMedia = resolved !== \"gradient\" && !!mediaUrl;\n const externalVideoBackdrop = useExternalVideoBackdrop();\n const hasExternalVideoBackdrop = externalVideoBackdrop !== false && resolved === \"video\";\n const externalVideoFailed = externalVideoBackdrop === \"fallback\" && resolved === \"video\";\n const externalVideoReady = externalVideoBackdrop === \"ready\" && resolved === \"video\";\n\n // Apply picture and scrim together. A loading or failed asset keeps the\n // fixed black base instead of painting contrast treatment over empty media.\n const [mediaPaint, setMediaPaint] = useState<MediaPaintState>(() =>\n initialMediaPaint(wantsMedia, resolved, mediaUrl, mediaPoster),\n );\n useEffect(() => {\n setMediaPaint(initialMediaPaint(wantsMedia, resolved, mediaUrl, mediaPoster));\n // Video reports its own paint through onLoadedData / onError below.\n if (!wantsMedia || resolved !== \"photo\") return;\n if (typeof Image === \"undefined\") return;\n let cancelled = false;\n const probe = new Image();\n probe.onload = () => {\n if (!cancelled) setMediaPaint(\"ready\");\n };\n probe.onerror = () => {\n if (!cancelled) setMediaPaint(\"failed\");\n };\n probe.src = mediaUrl;\n if (probe.complete) setMediaPaint(probe.naturalWidth > 0 ? \"ready\" : \"failed\");\n return () => {\n cancelled = true;\n probe.onload = null;\n probe.onerror = null;\n };\n }, [mediaUrl, mediaPoster, resolved, wantsMedia]);\n\n // The element stays mounted while pending — that is what loads it. Only a\n // confirmed failure takes it back out.\n const showMedia = wantsMedia && mediaPaint !== \"failed\";\n const showTreatment = wantsMedia && mediaPaint === \"ready\";\n const resolvedPosition = resolveMediaPosition(mediaPosition);\n const resolvedTreatment = resolveMediaTreatment(mediaTreatment);\n const treatmentLayers = getMediaTreatmentLayers(resolvedTreatment, textAnchor);\n\n const gradSeed =\n typeof seed === \"number\"\n ? seed\n : typeof seed === \"string\"\n ? seed.split(\"\").reduce((acc, c) => acc + c.charCodeAt(0), 0)\n : 0;\n\n const bgTransform = getBackgroundTransform(\n backgroundEffect,\n progress,\n beatIntensity,\n );\n\n return (\n <>\n {(!hasExternalVideoBackdrop || externalVideoFailed) && (\n <BrandGradientOverlay\n style={style}\n progress={progress}\n sceneDuration={sceneDuration}\n seed={gradSeed}\n />\n )}\n\n {showMedia && !hasExternalVideoBackdrop &&\n (resolved === \"video\" ? (\n <SceneVideoBackdrop\n mediaUrl={mediaUrl}\n mediaPoster={mediaPoster}\n mediaPosition={mediaPosition}\n backgroundEffect={backgroundEffect}\n progress={progress}\n sceneDuration={sceneDuration}\n beatIntensity={beatIntensity}\n isPlaying={isPlaying}\n onReady={() => setMediaPaint(\"ready\")}\n onError={() => setMediaPaint(\"failed\")}\n />\n ) : (\n <img\n src={mediaUrl}\n alt=\"\"\n aria-hidden=\"true\"\n draggable={false}\n data-media-position={mediaPosition}\n style={{\n position: \"absolute\",\n inset: 0,\n transform: bgTransform.transform,\n transformOrigin: bgTransform.transformOrigin,\n width: \"100%\",\n height: \"100%\",\n objectFit: \"cover\",\n objectPosition: resolvedPosition,\n }}\n />\n ))}\n\n {(hasExternalVideoBackdrop ? externalVideoReady : showTreatment) && !externalVideoFailed &&\n treatmentLayers.map((layer) => (\n <div\n key={layer.id}\n data-media-treatment={resolvedTreatment}\n data-media-overlay={layer.id}\n style={{\n position: \"absolute\",\n inset: 0,\n background: layer.background,\n pointerEvents: \"none\",\n ...layer.style,\n }}\n />\n ))}\n </>\n );\n};\n"
|
|
46
46
|
},
|
|
47
47
|
{
|
|
48
48
|
"path": "src/visual-system/scene-templates/media-source.ts",
|