@sentry/junior-agent-browser 0.248.0 → 0.250.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -20,7 +20,7 @@ The package includes three skills:
|
|
|
20
20
|
|
|
21
21
|
- `/agent-browser` for general browser navigation, interaction, extraction, and capture
|
|
22
22
|
- `/visual-web-qa` for evidence-driven verification of frontend, docs, layout, theme, loading, animation, and interaction changes
|
|
23
|
-
- `/demo-video` for edited
|
|
23
|
+
- `/demo-video` for edited demos, walkthroughs, and explainers with browser captures, optional narration and music, and Remotion
|
|
24
24
|
|
|
25
25
|
For example:
|
|
26
26
|
|
package/package.json
CHANGED
|
@@ -1,18 +1,25 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: demo-video
|
|
3
|
-
description: Creates and revises
|
|
3
|
+
description: Creates and revises edited videos with Remotion, such as product demos, narrated screen walkthroughs, animated explainers, and short presentations built from browser captures, with optional narration, captions, and music. Use when asked to make, animate, polish, or update a video or presentation of an app or shipped work. Do not use for a raw browser recording, screenshot-only visual QA, or audio-only text-to-speech.
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Demo Video
|
|
7
7
|
|
|
8
|
-
Make the
|
|
8
|
+
Make the subject the story. For a product, show what people can do, why it matters, and what the team built. Use personality in the delivery, not as the story. Do not introduce a revision as a new take or talk about the narrator's voice unless asked.
|
|
9
9
|
|
|
10
|
-
Open
|
|
10
|
+
| Open when you need to... | Read |
|
|
11
|
+
| ----------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
|
|
12
|
+
| set up Remotion, measure clips, mix music, render, check an export, or see revision examples | [references/rendering.md](references/rendering.md) |
|
|
13
|
+
| time motion to the narration: word alignment, camera moves, highlights, cursor, captions, transitions, and stills | [references/motion.md](references/motion.md) |
|
|
14
|
+
|
|
15
|
+
## 0. Pick The Model Profile
|
|
16
|
+
|
|
17
|
+
Composition, motion timing, and frame review are implementation work. When `handoff` is available and a listed profile fits implementation better than the active one, switch before step 1. If the requester names a model or profile, use it when it is available. Do not hand off only to summarize finished work.
|
|
11
18
|
|
|
12
19
|
## 1. Resolve The Brief And Reuse Work
|
|
13
20
|
|
|
14
21
|
- Read the current request and prior decisions. Keep the agreed audience, length, tone, credits, and destination. Ask only for a missing decision that blocks work.
|
|
15
|
-
- Find the existing source, assets, and attachments before rebuilding.
|
|
22
|
+
- Find the existing source, assets, and attachments before rebuilding. Sandbox files can disappear between turns. Look for an earlier source archive in the conversation and load it. If no source exists, say this will be a fresh composition.
|
|
16
23
|
- Inspect the target UI and relevant repository history before writing claims about shipped behavior. Use the repository skill for code access. Do not present an unmerged feature as live.
|
|
17
24
|
- Draft a short scene list with one point per scene. Match each claim to a capture or verified source. Name collaborators when requested and supported.
|
|
18
25
|
- Separate product capabilities from actions taken during this task. Describing reminder emails does not authorize sending one.
|
|
@@ -24,34 +31,37 @@ Load `agent-browser` for browser commands. Use `visual-web-qa` when the request
|
|
|
24
31
|
- Prefer a reachable local or preview page for the requested revision. Use an authorized session for private pages; do not bypass sign-in.
|
|
25
32
|
- If private data prevents safe capture, use the real UI with isolated synthetic fixtures. Label that data in the video. Do not make a lookalike and call it the app.
|
|
26
33
|
- Keep the fixture server local. Prevent it from sending writes, email, or uploads to production. Do not reuse customer submissions or secrets.
|
|
27
|
-
- Capture
|
|
34
|
+
- Capture each state the narration mentions: open panels, dark theme, player cards, and mobile views. Add small synthetic fixtures when a state needs data. Use full-page captures when the camera must scroll.
|
|
35
|
+
- Record element boxes from the DOM at the capture viewport. Highlights and cursor targets use these boxes; see the motion reference.
|
|
28
36
|
- Use screenshots for stable pages. Use short browser recordings when an interaction or motion is part of the story. Stop recordings after the required flow.
|
|
29
37
|
- Wait for expected content and fonts. Open the captures before using them. Check for loading states, clipping, mismatched fixture counts, and sensitive content.
|
|
30
38
|
|
|
31
39
|
For a raw interaction recording without an edited presentation, use `agent-browser` alone. Do not add a Remotion project by default.
|
|
32
40
|
|
|
33
|
-
## 3. Prepare Narration
|
|
41
|
+
## 3. Prepare Narration And Music
|
|
34
42
|
|
|
35
43
|
- Write spoken text separately from scene layout and delivery directions. Keep the opening about the work. Put credits and operational capabilities where they support the story.
|
|
36
44
|
- For requested speech, load the available TTS skill and use its credential, cost, and retry rules. Use the agreed voice preset. Do not silently replace a missing provider with another voice.
|
|
37
45
|
- Generate one clip per scene to make revisions cheap. Reuse unchanged clips and generate changed text to fresh paths. A visual-only edit needs no new speech request.
|
|
38
|
-
- Measure each clip. Derive scene frames from
|
|
39
|
-
- Keep captions tied to the final script.
|
|
46
|
+
- Measure each clip. Align its words to get word times. Derive scene frames from the last spoken word plus explicit lead and tail time. Recompute later scene offsets after a clip changes.
|
|
47
|
+
- Keep captions tied to the final script and the word times. If alignment is unavailable, say that caption timing is approximate. Audio-energy avatar motion is not lip sync.
|
|
48
|
+
- Music is optional. Use only a track that the user supplies or that has a clear license for this use. Do not generate, download, or imitate music without a stated source and license. Keep music under the narration; see the rendering reference.
|
|
40
49
|
|
|
41
50
|
## 4. Compose And Render
|
|
42
51
|
|
|
43
52
|
- Use a separate artifact directory unless the user asked to change the app. Keep composition source, local assets, scene data, and outputs together.
|
|
44
|
-
- Reuse an existing Remotion project and lockfile. For a new project, follow the
|
|
45
|
-
- Match the app's type, colors, and approved assets. Keep UI text readable.
|
|
53
|
+
- Reuse an existing Remotion project and lockfile. For a new project, follow the rendering reference. Browser runtime setup remains owned by the plugin; report a missing runtime instead of repairing it here.
|
|
54
|
+
- Match the app's type, colors, and approved assets. Keep UI text readable.
|
|
55
|
+
- Time every move to a spoken word: camera, highlight, cursor click, caption, and on-screen text. Show one focal point at a time. Use pans, zooms, short wipes, and springs instead of expensive effects. Label any illustration that is not a real screen.
|
|
46
56
|
- Bundle images, fonts, and audio locally. Drive motion from frame numbers, not wall-clock timers or network calls during render.
|
|
47
|
-
- Render stills for each
|
|
57
|
+
- Render stills for each scene and transition before a full video. Fix crops, collisions, and empty states, then render again. Use a short frame range to diagnose failures.
|
|
48
58
|
- Set export resolution and audio bitrate for the destination early. Keep the source at full composition size. A smaller delivery copy need not change the scene design.
|
|
49
59
|
|
|
50
60
|
## 5. Check And Deliver
|
|
51
61
|
|
|
52
|
-
- Decode the full exported video and audio. Inspect frames from every scene, including transitions, first/last frames, captions, and any changed crop. Check durations, missing audio, peaks, and final-frame truncation.
|
|
62
|
+
- Decode the full exported video and audio. Inspect frames from every scene, including transitions, first/last frames, captions, and any changed crop. Check durations, missing audio, loudness, peaks, and final-frame truncation.
|
|
53
63
|
- Listen when audio review is available. Decode and level checks do not prove pronunciation, timing, or listening quality; report unchecked limits.
|
|
54
64
|
- Check the actual file size before attachment. If too large, re-encode the existing master with a smaller delivery profile. Recheck that file. Do not regenerate speech or publish a public URL to work around the limit.
|
|
55
65
|
- Inspect outputs and logs after a timeout before repeating a render. Never infer success from a file's existence alone.
|
|
56
66
|
- Deliver the video with `sendFiles`. Include standalone synthetic narration and editable source when requested or already agreed. Exclude dependencies, caches, secrets, and unused revisions from the source archive.
|
|
57
|
-
- Report successful attachments, duration, checks, and remaining limits. Record capture provenance and rerender commands in the artifact README. Do not say the app or production changed when only the video changed.
|
|
67
|
+
- Report successful attachments, duration, checks, and remaining limits. Record capture provenance, alignment, music license, and rerender commands in the artifact README. Do not say the app or production changed when only the video changed.
|
|
@@ -2,19 +2,23 @@
|
|
|
2
2
|
|
|
3
3
|
## Source Inventory
|
|
4
4
|
|
|
5
|
-
Reviewed 2026-09-29. Official docs and repository contracts are authoritative for syntax and ownership. The completed workflow is evidence of one working setup, not a performance guarantee.
|
|
6
|
-
|
|
7
|
-
| Source | Confidence | Contribution and limits
|
|
8
|
-
| --------------------------------------------------------------------------------- | -------------------------- |
|
|
9
|
-
| [Agent Skills specification](https://agentskills.io/specification) | High | Frontmatter, directory names, and relative bundled files.
|
|
10
|
-
| `packages/junior-agent-browser/plugin.yaml`, package manifest, and sibling skills | High | Browser ownership and packaged skill discovery. Keep runtime setup in the manifest.
|
|
11
|
-
| `packages/junior-tts/skills/tts/SKILL.md` | High | Speech workflow, fresh paths, provider ownership, and paid-request limits. Reference rather than duplicate its credential policy.
|
|
12
|
-
| `policies/agent-steering.md` | High | One owner per rule. No extra system prompt or core tool changes.
|
|
13
|
-
| [Remotion render CLI](https://www.remotion.dev/docs/cli/render) | High | Composition selection, codec, frame range, scale, concurrency, audio bitrate, and overwrite controls. Check flags against the installed version.
|
|
14
|
-
| [Remotion performance](https://www.remotion.dev/docs/performance) | High | Concurrency tradeoffs, local assets, resolution, and expensive GPU effects. No universal fastest setting.
|
|
15
|
-
| [Remotion versions](https://www.remotion.dev/docs/version-mismatch) | High | Exact, matching Remotion package versions.
|
|
16
|
-
| [Remotion licensing](https://www.remotion.dev/docs/license) | High | Commercial use has license conditions; no free-use assumption.
|
|
17
|
-
|
|
|
5
|
+
Reviewed 2026-09-29; updated 2026-10-15. Official docs and repository contracts are authoritative for syntax and ownership. The completed workflow is evidence of one working setup, not a performance guarantee.
|
|
6
|
+
|
|
7
|
+
| Source | Confidence | Contribution and limits |
|
|
8
|
+
| --------------------------------------------------------------------------------- | -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
9
|
+
| [Agent Skills specification](https://agentskills.io/specification) | High | Frontmatter, directory names, and relative bundled files. |
|
|
10
|
+
| `packages/junior-agent-browser/plugin.yaml`, package manifest, and sibling skills | High | Browser ownership and packaged skill discovery. Keep runtime setup in the manifest. |
|
|
11
|
+
| `packages/junior-tts/skills/tts/SKILL.md` | High | Speech workflow, fresh paths, provider ownership, and paid-request limits. Reference rather than duplicate its credential policy. |
|
|
12
|
+
| `policies/agent-steering.md` | High | One owner per rule. No extra system prompt or core tool changes. |
|
|
13
|
+
| [Remotion render CLI](https://www.remotion.dev/docs/cli/render) | High | Composition selection, codec, frame range, scale, concurrency, audio bitrate, and overwrite controls. Check flags against the installed version. |
|
|
14
|
+
| [Remotion performance](https://www.remotion.dev/docs/performance) | High | Concurrency tradeoffs, local assets, resolution, and expensive GPU effects. No universal fastest setting. |
|
|
15
|
+
| [Remotion versions](https://www.remotion.dev/docs/version-mismatch) | High | Exact, matching Remotion package versions. |
|
|
16
|
+
| [Remotion licensing](https://www.remotion.dev/docs/license) | High | Commercial use has license conditions; no free-use assumption. |
|
|
17
|
+
| Animation polish pass on the same presentation, September 2026 | High for observed outcomes | Word alignment with faster-whisper `base.en`; cue-driven camera, rings, cursor, and captions; DOM-measured highlight boxes; extra fixture states; covering wipes; persistent presenter layer; batch stills via the Node API; 10,485,760-byte attachment limit and a 10.3 MB 1080p `-tune animation` delivery copy. The user rated the result "great". |
|
|
18
|
+
| [faster-whisper](https://github.com/SYSTRAN/faster-whisper) | High | CPU word timestamps with `word_timestamps=True`. An `initial_prompt` with the script caused repeated text in one run; `condition_on_previous_text=False` without a prompt matched the script. |
|
|
19
|
+
| [Remotion Node renderer](https://www.remotion.dev/docs/renderer) | High | `bundle`, `selectComposition`, and `renderStill` for many stills from one bundle. |
|
|
20
|
+
| [Remotion audio volume](https://www.remotion.dev/docs/audio/volume) | High | Frame-based `volume` callback for music fades and ducking. Music itself is not sourced here. |
|
|
21
|
+
| Completed browser-to-Remotion presentation and user revisions, September 2026 | High for observed outcomes | Real React UI with local fixtures; measured WAV clips; Remotion 4.0.354; FFmpeg decode and frame checks; successful MP4, WAV, and source delivery. Generalized here without transcript or identity data. |
|
|
18
22
|
|
|
19
23
|
Documentation is linked and summarized, not copied as a bundled upstream manual. No third-party media, project-specific avatar, sample app data, or generated speech is included.
|
|
20
24
|
|
|
@@ -27,11 +31,16 @@ Documentation is linked and summarized, not copied as a bundled upstream manual.
|
|
|
27
31
|
- **Adopted:** inspect output after timeouts; check actual attachment bytes. One render completed despite a tool timeout. A large master needed a smaller delivery copy.
|
|
28
32
|
- **Adopted:** standard-library WAV measurement fallback. FFmpeg was available while FFprobe was not. No extra audio package was needed.
|
|
29
33
|
- **Rejected:** shipping the one-off composition, fixtures, assets, or hardcoded browser path. Those are project choices, not reusable skill policy.
|
|
30
|
-
- **
|
|
34
|
+
- **Adopted:** hand off to an implementation profile for composition. The requester asked for the stronger profile, and the polish pass ran on it. The skill names no model, because profiles change.
|
|
35
|
+
- **Adopted:** ASR word alignment, cue lookup by phrase, and DOM-measured boxes. These replaced word-count captions and guessed coordinates, which caused cut phrases and misplaced highlights.
|
|
36
|
+
- **Adopted:** still-grid review with a defect table. The listed defects were found and fixed during the polish pass.
|
|
37
|
+
- **Adopted, conditional:** music only from supplied or licensed tracks, ducked under narration. The user asked for music guidance; no music was used in the observed runs, so the mix values are a starting point, not a measured result.
|
|
38
|
+
- **Rejected:** the project's brand colors, layout, avatar behavior, and scene copy. Those are project choices.
|
|
39
|
+
- **Deferred:** a Remotion plugin, installed render dependencies, generator scripts, a shared animation component library, music generation, and phoneme-level avatar animation. No recurring need for that scope was established.
|
|
31
40
|
|
|
32
41
|
Coverage includes preconditions (brief, source, tools, license), ordered flow (capture through delivery), safety (local fixtures and no production actions), failures (auth, missing tools, timeout, size), revision examples, and verified output limits. All runtime reference files are routed directly from `SKILL.md`.
|
|
33
42
|
|
|
34
|
-
Portability: pnpm, Node, Remotion, an installed browser, and FFmpeg are explicit. TTS is optional and delegated to the installed skill. Python is only a conditional WAV fallback. No host path, provider token, or model name is required by this skill.
|
|
43
|
+
Portability: pnpm, Node, Remotion, an installed browser, and FFmpeg are explicit. TTS is optional and delegated to the installed skill. Python is only a conditional WAV fallback. No host path, provider token, or model name is required by this skill. `faster-whisper` is optional; without it, timing falls back to measured durations and is reported as approximate.
|
|
35
44
|
|
|
36
45
|
## Precision And Validation Notes
|
|
37
46
|
|
|
@@ -2,18 +2,22 @@
|
|
|
2
2
|
|
|
3
3
|
## Intent And Scope
|
|
4
4
|
|
|
5
|
-
Create or revise an edited
|
|
5
|
+
Create or revise an edited video from real UI captures, optional narration, optional licensed music, and a Remotion composition. Typical outputs are product demos, narrated screen walkthroughs, animated explainers, and short presentations. Keep the story about the subject and the team's work. Preserve useful source and assets across revisions.
|
|
6
6
|
|
|
7
|
-
This is a workflow-process skill with a reference-backed shape. The runtime checklist owns the flow.
|
|
7
|
+
This is a workflow-process skill with a reference-backed shape. The runtime checklist owns the flow. `references/rendering.md` holds setup, audio, render, and export commands with revision examples. `references/motion.md` holds narration-timed animation patterns. No generator script, fixed design template, new package, or runtime dependency is required.
|
|
8
8
|
|
|
9
|
-
Raw browser recordings belong to `agent-browser`. Visual correctness checks belong to `visual-web-qa`. Audio-only requests belong to the available TTS skill. Deployment, event operations, email sending,
|
|
9
|
+
Raw browser recordings belong to `agent-browser`. Visual correctness checks belong to `visual-web-qa`. Audio-only requests belong to the available TTS skill. Model selection belongs to the runtime `handoff` tool; this skill only says when to use it. Deployment, event operations, email sending, voice cloning, and music generation are outside scope.
|
|
10
10
|
|
|
11
11
|
## Contract
|
|
12
12
|
|
|
13
13
|
- Resolve the brief, source, and existing artifacts before generating assets.
|
|
14
14
|
- Verify product claims and preserve requested credits.
|
|
15
15
|
- Capture the real UI safely; label synthetic fixtures and narration.
|
|
16
|
-
-
|
|
16
|
+
- Hand off to an implementation profile before composition when one fits better.
|
|
17
|
+
- Measure speech, align words, reuse unchanged clips, and keep scene offsets current.
|
|
18
|
+
- Time camera, highlights, cursor, captions, and on-screen text to spoken words.
|
|
19
|
+
- Use only user-supplied or licensed music, and keep it under the narration.
|
|
20
|
+
- Review still grids of every scene and transition before the full render.
|
|
17
21
|
- Check the delivered export, not only its source or master.
|
|
18
22
|
- Deliver the requested video and agreed supporting artifacts with honest limits.
|
|
19
23
|
- Leave the app and production unchanged unless a separate request permits changes.
|
|
@@ -26,6 +30,8 @@ Should trigger:
|
|
|
26
30
|
- “Use Remotion to present what we shipped.”
|
|
27
31
|
- “Update this demo video with the new UI and voiceover.”
|
|
28
32
|
- “Add a collaborator credit and the reminder feature to our presentation.”
|
|
33
|
+
- “Polish the animation in this video.”
|
|
34
|
+
- “Make a short animated explainer of this feature with a voiceover and some music.”
|
|
29
35
|
|
|
30
36
|
Should not trigger:
|
|
31
37
|
|
|
@@ -33,6 +39,7 @@ Should not trigger:
|
|
|
33
39
|
- “Take a mobile screenshot and check the spacing.”
|
|
34
40
|
- “Read this paragraph aloud.”
|
|
35
41
|
- “Schedule a demo and send an email reminder.”
|
|
42
|
+
- “Compose a background track for our video.”
|
|
36
43
|
|
|
37
44
|
## Evidence And Validation
|
|
38
45
|
|
|
@@ -42,6 +49,6 @@ Validate frontmatter, bundled links, plugin packaging, and the trigger boundary.
|
|
|
42
49
|
|
|
43
50
|
## Limits And Maintenance
|
|
44
51
|
|
|
45
|
-
Remotion and
|
|
52
|
+
Remotion, FFmpeg, and `faster-whisper` are not provisioned by this skill. Speech needs the appropriate enabled plugin or supplied audio. Music needs a user-supplied or licensed track. Commercial Remotion use needs an appropriate license. ASR word times are close but not exact; captions without alignment remain approximate. Audio-energy motion is not lip sync.
|
|
46
53
|
|
|
47
|
-
Update commands after incompatible Remotion changes. Keep tool setup and credentials in their owning plugins. Extend the existing
|
|
54
|
+
Update commands after incompatible Remotion changes. Keep tool setup and credentials in their owning plugins. Extend the existing references only for a demonstrated failure or recurring task. Keep brand choices, project compositions, and assets out of this skill; do not turn it into a general video platform.
|
|
@@ -0,0 +1,199 @@
|
|
|
1
|
+
# Time Motion To The Narration
|
|
2
|
+
|
|
3
|
+
Use this reference after the clips are measured. Every move must answer a spoken word. Keep one focal point on screen at a time.
|
|
4
|
+
|
|
5
|
+
## Align Words
|
|
6
|
+
|
|
7
|
+
Word times turn the script into cues for the camera, highlights, and captions. Use the narration WAV files, not the compressed export.
|
|
8
|
+
|
|
9
|
+
When Python, network access, and a package install are permitted, `faster-whisper` works on CPU. The first run downloads a model. If the install is not permitted, use sentence pauses from audio levels and say that timing is approximate.
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
python3 -m pip install --user faster-whisper
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
```python
|
|
16
|
+
# align.py: transcribe each clip with word times, then map them onto the script text.
|
|
17
|
+
import difflib, json, re
|
|
18
|
+
from faster_whisper import WhisperModel
|
|
19
|
+
|
|
20
|
+
model = WhisperModel("base.en", device="cpu", compute_type="int8")
|
|
21
|
+
norm = lambda w: re.sub(r"[^a-z0-9']", "", w.lower())
|
|
22
|
+
scenes = json.load(open("src/scenes.json")) # [{"id", "text", "audio"}]
|
|
23
|
+
out = {}
|
|
24
|
+
for scene in scenes:
|
|
25
|
+
segments, _ = model.transcribe(f"public/{scene['audio']}", word_timestamps=True,
|
|
26
|
+
language="en", condition_on_previous_text=False)
|
|
27
|
+
heard = [(w.word.strip(), w.start, w.end) for s in segments for w in s.words]
|
|
28
|
+
script = scene["text"].split()
|
|
29
|
+
times = [None] * len(script)
|
|
30
|
+
ops = difflib.SequenceMatcher(None, [norm(w) for w in script], [norm(w[0]) for w in heard], autojunk=False)
|
|
31
|
+
for tag, i1, i2, j1, j2 in ops.get_opcodes():
|
|
32
|
+
if tag == "equal":
|
|
33
|
+
for k in range(i2 - i1):
|
|
34
|
+
times[i1 + k] = heard[j1 + k][1:]
|
|
35
|
+
elif tag == "replace":
|
|
36
|
+
start, end = heard[j1][1], heard[j2 - 1][2]
|
|
37
|
+
step = (end - start) / (i2 - i1)
|
|
38
|
+
for k in range(i2 - i1):
|
|
39
|
+
times[i1 + k] = (start + k * step, start + (k + 1) * step)
|
|
40
|
+
for i, t in enumerate(times):
|
|
41
|
+
times[i] = t or ((times[i - 1][1] if i else 0), (times[i - 1][1] if i else 0) + 0.2)
|
|
42
|
+
out[scene["id"]] = [[w, round(a, 2), round(b, 2)] for w, (a, b) in zip(script, times)]
|
|
43
|
+
json.dump(out, open("src/words.json", "w"))
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
- Keep the script words, not the transcript words. The transcript can mishear a word; the caption must show the script.
|
|
47
|
+
- Do not pass the script as `initial_prompt`. In one run it made the model repeat sentences that were not spoken.
|
|
48
|
+
- Compare the transcript with the script. A large mismatch can mean a pronunciation problem. Report it; do not hide it.
|
|
49
|
+
- Set scene length from the last word end, not the WAV length. Clips often end with silence.
|
|
50
|
+
|
|
51
|
+
## Cue Helper
|
|
52
|
+
|
|
53
|
+
Look up a phrase and return its scene frame. Fail loudly when the phrase is missing, so a script edit cannot silently move a cue.
|
|
54
|
+
|
|
55
|
+
```tsx
|
|
56
|
+
export function cue(id: string, phrase: string, nth = 0): number {
|
|
57
|
+
const words = sceneWords(id); // [[word, start, end], ...]
|
|
58
|
+
const target = phrase.split(/\s+/).map(norm);
|
|
59
|
+
let seen = 0;
|
|
60
|
+
for (let i = 0; i <= words.length - target.length; i++) {
|
|
61
|
+
if (target.every((t, j) => norm(words[i + j][0]) === t) && seen++ === nth) {
|
|
62
|
+
return Math.round(lead(id) + words[i][1] * FPS);
|
|
63
|
+
}
|
|
64
|
+
}
|
|
65
|
+
throw new Error(`cue not found: ${id} "${phrase}"`);
|
|
66
|
+
}
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Write cues as `cue('02', 'open the screening')`, not frame numbers. A cue is the frame where the word starts. Offset small amounts (`- 6`, `+ 10`) for short motion, such as a highlight or a caption, so it arrives just before or as the word lands. For motion that takes longer, start it earlier by its duration. See the camera example below.
|
|
70
|
+
|
|
71
|
+
## Guided Camera
|
|
72
|
+
|
|
73
|
+
Show a capture inside a window. Pan and zoom it with keyframes in capture pixels.
|
|
74
|
+
|
|
75
|
+
```tsx
|
|
76
|
+
type CamKey = { f: number; cx: number; cy: number; w: number; d?: number };
|
|
77
|
+
|
|
78
|
+
export function camera(
|
|
79
|
+
keys: CamKey[],
|
|
80
|
+
f: number,
|
|
81
|
+
img: [number, number],
|
|
82
|
+
view: [number, number],
|
|
83
|
+
) {
|
|
84
|
+
let { cx, cy, w } = keys[0];
|
|
85
|
+
for (const k of keys.slice(1)) {
|
|
86
|
+
const t = interpolate(f, [k.f, k.f + (k.d ?? 26)], [0, 1], {
|
|
87
|
+
extrapolateLeft: "clamp",
|
|
88
|
+
extrapolateRight: "clamp",
|
|
89
|
+
easing: Easing.bezier(0.65, 0, 0.35, 1),
|
|
90
|
+
});
|
|
91
|
+
cx += (k.cx - cx) * t;
|
|
92
|
+
cy += (k.cy - cy) * t;
|
|
93
|
+
w += (k.w - w) * t;
|
|
94
|
+
}
|
|
95
|
+
const s = view[0] / w;
|
|
96
|
+
const fit = (v: number, vis: number, size: number) =>
|
|
97
|
+
size <= vis ? size / 2 : Math.min(Math.max(v, vis / 2), size - vis / 2);
|
|
98
|
+
cx = fit(cx, view[0] / s, img[0]);
|
|
99
|
+
cy = fit(cy, view[1] / s, img[1]);
|
|
100
|
+
return { s, x: view[0] / 2 - cx * s, y: view[1] / 2 - cy * s };
|
|
101
|
+
}
|
|
102
|
+
// Render: <Img style={{transformOrigin: '0 0', transform: `translate(${cam.x}px, ${cam.y}px) scale(${cam.s})`}} />
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
- `f` is the frame where a move starts. `d` is its length (default 26 frames). The move ends at `f + d`. To make the camera arrive on a word, start the move one duration before the cue: `{ f: cue('02', 'open the screening') - 26, cx, cy, w }`. If you set `f` to the cue, the camera arrives `d` frames after the word.
|
|
106
|
+
- `w` is the visible capture width. A smaller `w` zooms in. Stay at or above about 70% of the capture width; stronger zooms blur 1× captures.
|
|
107
|
+
- Put frame edges between page sections. Do not cut a heading or card title through the middle.
|
|
108
|
+
- Return to a wider view when the narration summarizes.
|
|
109
|
+
- Two states of the same page (for example idle and playing) can share one camera. Cross-fade the second capture on top.
|
|
110
|
+
|
|
111
|
+
## Highlights And Cursor
|
|
112
|
+
|
|
113
|
+
Get element boxes from the live page at the capture viewport. Do not guess them from the image.
|
|
114
|
+
|
|
115
|
+
```bash
|
|
116
|
+
agent-browser --session cap eval "JSON.stringify([...document.querySelectorAll('button')]
|
|
117
|
+
.filter(b => b.innerText.trim() === 'Open the screening')
|
|
118
|
+
.map(b => { const r = b.getBoundingClientRect(); return [r.x + scrollX, r.y + scrollY, r.width, r.height].map(Math.round); }))"
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
- Map a box to the screen with the camera: `screenX = x * cam.s + cam.x`. Draw rings and the cursor in screen space so stroke width and label size stay constant during zooms.
|
|
122
|
+
- Show a ring when its word starts. Remove it before the next focal point. Add a short label only when the narration does not already say it.
|
|
123
|
+
- Move labels below or to the other side near window edges and near other rings.
|
|
124
|
+
- Make the cursor arrive at the target on the cue. Show a short ripple for a click. Hide the cursor when the scene stops pointing.
|
|
125
|
+
- Fake typing is acceptable over a real input when the narration describes filling the form. Cover the placeholder with the input's own background and font.
|
|
126
|
+
|
|
127
|
+
## Captions
|
|
128
|
+
|
|
129
|
+
- Make phrases from word times. End a phrase at `.`, `!`, `?`, or `:`, at a comma after four or more words, at a pause of 0.45 s or more, or at eight words.
|
|
130
|
+
- Show a phrase from its first word start until the next phrase starts. Keep it for about 0.6 s after the last phrase.
|
|
131
|
+
- Dim words that are not spoken yet. Mark the current word with an accent color.
|
|
132
|
+
- Keep captions in a fixed band below the UI window. Do not cover the focal point.
|
|
133
|
+
|
|
134
|
+
## Layout, Type, And Continuity
|
|
135
|
+
|
|
136
|
+
- Put a short title and a beat list beside the window. Add each beat on its cue and mark the current one. Beats summarize; captions carry the full words.
|
|
137
|
+
- Animate title words with per-word springs, a few frames apart. Keep the end state static and readable.
|
|
138
|
+
- Put persistent elements, such as a presenter avatar, progress bar, and transitions, above the scene sequences in one layer. Keyframe them across global frames, so they continue across cuts.
|
|
139
|
+
- For audio-reactive motion, precompute one energy value per frame per clip. Smooth it over three frames. Keep the amplitude small.
|
|
140
|
+
- Label an illustration on screen, for example a chat exchange that shows how a workflow runs. Base its content on documented behavior. Do not show real people saying things they did not say.
|
|
141
|
+
|
|
142
|
+
## Transitions
|
|
143
|
+
|
|
144
|
+
Use a short wipe that fully covers the frame at the cut. Change the scene under the cover.
|
|
145
|
+
|
|
146
|
+
```tsx
|
|
147
|
+
// Scene boundary b. A skewed panel moves from right to left over 2 * W frames.
|
|
148
|
+
const t = interpolate(f, [b - W, b + W], [0, 1], { ...clamp, easing });
|
|
149
|
+
const x = interpolate(t, [0, 1], [width + 160, -(panelWidth + 160)]);
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
- Check that the panel covers the full frame, including the skew, on frame `b`. Render stills at `b - 1`, `b`, and `b + 1`.
|
|
153
|
+
- Start the next narration a few frames after `b`. End the previous narration before `b - W`.
|
|
154
|
+
- Do not cross-fade two text-heavy layouts in the same place. Fade the first out, then bring the second in.
|
|
155
|
+
|
|
156
|
+
## Review Stills In Batches
|
|
157
|
+
|
|
158
|
+
`remotion still` bundles on each call. To check many frames, bundle once with the Node API:
|
|
159
|
+
|
|
160
|
+
```js
|
|
161
|
+
import { bundle } from "@remotion/bundler";
|
|
162
|
+
import { renderStill, selectComposition } from "@remotion/renderer";
|
|
163
|
+
|
|
164
|
+
const serveUrl = await bundle({
|
|
165
|
+
entryPoint: "src/index.tsx",
|
|
166
|
+
publicDir: "public",
|
|
167
|
+
});
|
|
168
|
+
const composition = await selectComposition({
|
|
169
|
+
serveUrl,
|
|
170
|
+
id: "Demo",
|
|
171
|
+
browserExecutable,
|
|
172
|
+
});
|
|
173
|
+
for (const frame of frames) {
|
|
174
|
+
await renderStill({
|
|
175
|
+
composition,
|
|
176
|
+
serveUrl,
|
|
177
|
+
frame,
|
|
178
|
+
output: `out/stills/f${frame}.png`,
|
|
179
|
+
browserExecutable,
|
|
180
|
+
scale: 0.5,
|
|
181
|
+
});
|
|
182
|
+
}
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
With pnpm, import these packages from the installed dependency paths when they are not direct dependencies. Combine stills into a grid with FFmpeg `xstack` and inspect them.
|
|
186
|
+
|
|
187
|
+
Check each grid for these defects:
|
|
188
|
+
|
|
189
|
+
| Defect | Fix |
|
|
190
|
+
| ------------------------------------------- | -------------------------------------------------------------- |
|
|
191
|
+
| Heading or card text cut by the window edge | Move `cy`, or widen `w`, so the edge falls between sections |
|
|
192
|
+
| Two ring labels overlap | Put one label below or on the other side |
|
|
193
|
+
| Empty window while the narration talks | Start the first item on an earlier cue |
|
|
194
|
+
| Overlay content scrolls into a header | Clip the scrolling area below the header |
|
|
195
|
+
| Text overlaps during a scene change | Fade out before the next layout comes in |
|
|
196
|
+
| A moving element covers a title | Reduce the move, or place the element outside the title column |
|
|
197
|
+
| A ring or label is cut by the window | Anchor the label to the other side, or move the camera |
|
|
198
|
+
|
|
199
|
+
Then render the full video. A 1920 × 1080, 30 fps, 103-second composition rendered in about four minutes at concurrency 4 on a 4-vCPU Sandbox. Measure your own sample before you set concurrency.
|
|
@@ -59,6 +59,36 @@ startFrame = sum(previous sceneFrames)
|
|
|
59
59
|
|
|
60
60
|
Place audio at `leadFrames` within the scene. Ensure fades do not cut off speech. Caption times use the same audio offset. Never keep an old duration after replacing a clip.
|
|
61
61
|
|
|
62
|
+
With word times from the motion reference, use the last word end instead of `audioSeconds`. Clips often end with silence; this keeps pacing tight. Keep the tail longer than the trailing silence, so the next clip does not overlap.
|
|
63
|
+
|
|
64
|
+
## Mix Music
|
|
65
|
+
|
|
66
|
+
Add music only from a user-supplied or licensed track. Record the source and license in the artifact README.
|
|
67
|
+
|
|
68
|
+
In Remotion, put the track in one `Audio` element over the whole composition. Lower it while narration plays:
|
|
69
|
+
|
|
70
|
+
```tsx
|
|
71
|
+
<Audio
|
|
72
|
+
src={staticFile("music.mp3")}
|
|
73
|
+
volume={(f) => {
|
|
74
|
+
const fadeIn = interpolate(f, [0, 30], [0, 1], clamp);
|
|
75
|
+
const fadeOut = interpolate(f, [total - 45, total], [1, 0], clamp);
|
|
76
|
+
const ducked = speaking(f) ? 0.12 : 0.3; // speaking(f) comes from scene starts and word times
|
|
77
|
+
return ducked * fadeIn * fadeOut;
|
|
78
|
+
}}
|
|
79
|
+
/>
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Smooth the duck over about 8 frames so it does not pump. Loop or trim the track to the composition length. Do not cut it in the middle of a phrase at the end.
|
|
83
|
+
|
|
84
|
+
After render, measure loudness. Speech-led videos for chat and laptops work at about −16 to −18 LUFS integrated, with true peaks below −1 dBFS:
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
ffmpeg -hide_banner -i out/demo.mp4 -af ebur128=peak=true -f null - 2>&1 | grep -A14 Summary
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
If music is too loud under speech, lower the duck level. Do not compress the narration to compete with it.
|
|
91
|
+
|
|
62
92
|
## Preview, Then Render
|
|
63
93
|
|
|
64
94
|
Run from the artifact project. Replace the entry point, composition ID, frame, and output names as needed. Use fresh output names. Add `--browser-executable="$BROWSER"` only after resolving that path.
|
|
@@ -94,12 +124,14 @@ ffmpeg -v error -ss 12 -i out/demo.mp4 -frames:v 1 -update 1 out/frame.png
|
|
|
94
124
|
|
|
95
125
|
Extract and inspect representative frames from all scenes and changed transitions. Keep captions outside the UI crop. Do not treat a contact sheet as proof that motion or speech timing is correct.
|
|
96
126
|
|
|
97
|
-
Check the destination's current limit and the file's byte size. Do not assume the upload limit from a previous conversation still applies. If necessary, re-encode the existing master:
|
|
127
|
+
Check the destination's current limit and the file's byte size. Do not assume the upload limit from a previous conversation still applies. A rejected `sendFiles` call reports the limit; one run saw 10,485,760 bytes. If necessary, re-encode the existing master:
|
|
98
128
|
|
|
99
129
|
```bash
|
|
100
130
|
ffmpeg -n -i out/demo.mp4 -vf scale=1280:720 -c:v libx264 -preset medium -crf 28 -c:a aac -b:a 96k -movflags +faststart out/demo-delivery.mp4
|
|
101
131
|
```
|
|
102
132
|
|
|
133
|
+
For flat UI and motion graphics, `-tune animation` keeps text sharp at lower bitrates. In one run, a 103-second 1080p master became 10.3 MB with `-preset slower -crf 29 -tune animation` and 112k AAC, with readable UI text. Try that before you drop to 720p.
|
|
134
|
+
|
|
103
135
|
Measure the new file and check its text readability and decoding. If it still exceeds the limit, reduce resolution or bitrate while preserving usable UI text. Keep the original master. For standalone narration, join the original lossless clips with the same scene gaps; do not decode the compressed MP4 audio when source WAV files are available.
|
|
104
136
|
|
|
105
137
|
## Revision Examples
|
|
@@ -108,4 +140,6 @@ Measure the new file and check its text readability and decoding. If it still ex
|
|
|
108
140
|
- **Private app:** “Use our dashboard in a demo.” Sign-in blocks access. Use an authorized session or the real local UI with isolated, labeled synthetic fixtures. State that this is not a production-data recording. Do not copy private submissions or alter production to make the capture work.
|
|
109
141
|
- **Story correction:** “This is about the product, not your new voice.” Bad: regenerate everything and announce “take three.” Correct: remove voice and take references from spoken and on-screen text, replace only changed clips, recompute timing, and keep useful captures.
|
|
110
142
|
- **Add credit and operations:** “Name my collaborator and explain reminders.” Verify the name and capability. Add the credit and a short workflow explanation, including required confirmation. Do not send a reminder just to illustrate it.
|
|
143
|
+
- **Polish pass:** “Make the animation better.” Keep the approved script and narration. Align words, add the missing UI states, and key camera moves, highlights, and captions to cues. Review still grids and fix defects before the full render. No new speech request is needed.
|
|
144
|
+
- **Source gone:** the Sandbox no longer has last turn's project. Load the source archive that was attached earlier. Rebuild only what is missing.
|
|
111
145
|
- **Upload rejected:** a valid MP4 exceeds the attachment limit. Keep its narration and composition. Make a smaller delivery copy, check it, and resend that file rather than repeating paid generation.
|