dsh-remotion 0.8.0 → 0.8.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (102) hide show
  1. package/CHANGELOG.md +12 -1
  2. package/README.en.md +3 -1
  3. package/README.md +3 -1
  4. package/docs/USAGE.en.md +3 -3
  5. package/docs/USAGE.md +3 -3
  6. package/lib/client.js +74 -54
  7. package/lib/media-templates.d.ts +3 -3
  8. package/lib/media-templates.js +1 -1
  9. package/package.json +2 -2
  10. package/scripts/sync-skills.mjs +3 -2
  11. package/skills/UPSTREAM.json +297 -0
  12. package/skills/remotion-best-practices/SKILL.md +58 -4
  13. package/skills/remotion-best-practices/remotion-captions/REFERENCE.md +7 -1
  14. package/skills/remotion-best-practices/remotion-captions/display-captions.md +126 -147
  15. package/skills/remotion-best-practices/remotion-captions/import-srt-captions.md +3 -3
  16. package/skills/remotion-best-practices/remotion-captions/transcribe-captions.md +146 -44
  17. package/skills/remotion-best-practices/remotion-create/REFERENCE.md +16 -26
  18. package/skills/remotion-best-practices/remotion-docs/REFERENCE.md +1 -1
  19. package/skills/remotion-best-practices/remotion-interactivity/REFERENCE.md +70 -53
  20. package/skills/remotion-best-practices/remotion-interactivity/inline-effects.md +42 -0
  21. package/skills/remotion-best-practices/remotion-interactivity/timeline-structure.md +215 -0
  22. package/skills/remotion-best-practices/remotion-maps/REFERENCE.md +1 -1
  23. package/skills/remotion-best-practices/remotion-maps/techniques/mapbox/TECHNIQUE.md +3 -3
  24. package/skills/remotion-best-practices/remotion-maps/techniques/maplibre/TECHNIQUE.md +3 -3
  25. package/skills/remotion-best-practices/remotion-markup/REFERENCE.md +83 -113
  26. package/skills/remotion-best-practices/remotion-markup/arabic-text.md +88 -0
  27. package/skills/remotion-best-practices/remotion-markup/audio.md +35 -14
  28. package/skills/remotion-best-practices/remotion-markup/color-correction.md +190 -0
  29. package/skills/remotion-best-practices/remotion-markup/compositions.md +89 -34
  30. package/skills/remotion-best-practices/remotion-markup/connected-compositions.md +55 -69
  31. package/skills/remotion-best-practices/remotion-markup/cropping.md +2 -0
  32. package/skills/remotion-best-practices/remotion-markup/effects.md +34 -5
  33. package/skills/remotion-best-practices/remotion-markup/embedding-videos.md +20 -13
  34. package/skills/remotion-best-practices/remotion-markup/ffmpeg.md +2 -2
  35. package/skills/remotion-best-practices/remotion-markup/html-in-canvas.md +5 -27
  36. package/skills/remotion-best-practices/remotion-markup/images.md +5 -7
  37. package/skills/remotion-best-practices/remotion-markup/light-leaks.md +2 -4
  38. package/skills/remotion-best-practices/remotion-markup/motion-blur.md +2 -2
  39. package/skills/remotion-best-practices/remotion-markup/multi-scene-video.md +19 -79
  40. package/skills/remotion-best-practices/remotion-markup/remotion-maps/REFERENCE.md +1 -1
  41. package/skills/remotion-best-practices/remotion-markup/remotion-maps/techniques/mapbox/TECHNIQUE.md +3 -3
  42. package/skills/remotion-best-practices/remotion-markup/remotion-maps/techniques/maplibre/TECHNIQUE.md +3 -3
  43. package/skills/remotion-best-practices/remotion-markup/sequencing.md +21 -162
  44. package/skills/remotion-best-practices/remotion-markup/sfx.md +2 -2
  45. package/skills/remotion-best-practices/remotion-markup/silence-detection.md +2 -2
  46. package/skills/remotion-best-practices/remotion-markup/timing-props.md +90 -0
  47. package/skills/remotion-best-practices/remotion-markup/transitions.md +121 -74
  48. package/skills/remotion-best-practices/remotion-markup/video-editing.md +127 -75
  49. package/skills/remotion-best-practices/remotion-markup/voiceover.md +6 -2
  50. package/skills/remotion-best-practices/remotion-multimedia/REFERENCE.md +1 -1
  51. package/skills/remotion-best-practices/remotion-multimedia/get-audio-duration.md +8 -3
  52. package/skills/remotion-best-practices/remotion-multimedia/get-video-dimensions.md +9 -3
  53. package/skills/remotion-best-practices/remotion-multimedia/get-video-duration.md +6 -3
  54. package/skills/remotion-best-practices/remotion-render/REFERENCE.md +1 -1
  55. package/skills/remotion-best-practices/remotion-saas/REFERENCE.md +1 -1
  56. package/skills/remotion-best-practices/remotion-studio/REFERENCE.md +11 -3
  57. package/skills/remotion-best-practices/remotion-upgrade/REFERENCE.md +1 -1
  58. package/skills/remotion-captions/SKILL.md +11 -1
  59. package/skills/remotion-captions/display-captions.md +126 -147
  60. package/skills/remotion-captions/import-srt-captions.md +3 -3
  61. package/skills/remotion-captions/transcribe-captions.md +146 -44
  62. package/skills/remotion-create/SKILL.md +20 -26
  63. package/skills/remotion-docs/SKILL.md +5 -1
  64. package/skills/remotion-interactivity/SKILL.md +74 -53
  65. package/skills/remotion-interactivity/inline-effects.md +42 -0
  66. package/skills/remotion-interactivity/timeline-structure.md +215 -0
  67. package/skills/remotion-maps/SKILL.md +5 -1
  68. package/skills/remotion-maps/techniques/mapbox/TECHNIQUE.md +3 -3
  69. package/skills/remotion-maps/techniques/maplibre/TECHNIQUE.md +3 -3
  70. package/skills/remotion-markup/SKILL.md +87 -113
  71. package/skills/remotion-markup/arabic-text.md +88 -0
  72. package/skills/remotion-markup/audio.md +35 -14
  73. package/skills/remotion-markup/color-correction.md +190 -0
  74. package/skills/remotion-markup/compositions.md +89 -34
  75. package/skills/remotion-markup/connected-compositions.md +55 -69
  76. package/skills/remotion-markup/cropping.md +2 -0
  77. package/skills/remotion-markup/effects.md +34 -5
  78. package/skills/remotion-markup/embedding-videos.md +20 -13
  79. package/skills/remotion-markup/ffmpeg.md +2 -2
  80. package/skills/remotion-markup/html-in-canvas.md +5 -27
  81. package/skills/remotion-markup/images.md +5 -7
  82. package/skills/remotion-markup/light-leaks.md +2 -4
  83. package/skills/remotion-markup/motion-blur.md +2 -2
  84. package/skills/remotion-markup/multi-scene-video.md +19 -79
  85. package/skills/remotion-markup/remotion-maps/REFERENCE.md +1 -1
  86. package/skills/remotion-markup/remotion-maps/techniques/mapbox/TECHNIQUE.md +3 -3
  87. package/skills/remotion-markup/remotion-maps/techniques/maplibre/TECHNIQUE.md +3 -3
  88. package/skills/remotion-markup/sequencing.md +21 -162
  89. package/skills/remotion-markup/sfx.md +2 -2
  90. package/skills/remotion-markup/silence-detection.md +2 -2
  91. package/skills/remotion-markup/timing-props.md +90 -0
  92. package/skills/remotion-markup/transitions.md +121 -74
  93. package/skills/remotion-markup/video-editing.md +127 -75
  94. package/skills/remotion-markup/voiceover.md +6 -2
  95. package/skills/remotion-multimedia/SKILL.md +5 -1
  96. package/skills/remotion-multimedia/get-audio-duration.md +8 -3
  97. package/skills/remotion-multimedia/get-video-dimensions.md +9 -3
  98. package/skills/remotion-multimedia/get-video-duration.md +6 -3
  99. package/skills/remotion-render/SKILL.md +5 -1
  100. package/skills/remotion-saas/SKILL.md +5 -1
  101. package/skills/remotion-studio/SKILL.md +15 -3
  102. package/skills/remotion-upgrade/SKILL.md +5 -1
@@ -1,190 +1,169 @@
1
1
  ---
2
2
  name: display-captions
3
- description: Displaying captions in Remotion with TikTok-style pages and word highlighting
3
+ description: Displaying captions in Remotion using the Basic Captions element
4
4
  metadata:
5
- tags: captions, subtitles, display, tiktok, highlight
5
+ tags: captions, subtitles, display, element
6
6
  ---
7
7
 
8
8
  # Displaying captions in Remotion
9
9
 
10
- This guide explains how to display captions in Remotion, assuming you already have captions in the [`Caption`](https://www.remotion.dev/docs/captions/caption) format.
10
+ This guide explains how to display captions in Remotion, assuming you already have captions in the [`Caption`](https://www.remotion.dev/docs/captions/caption.md) format.
11
11
 
12
12
  ## Prerequisites
13
13
 
14
14
  Read [Transcribing audio](transcribe-captions.md) for how to generate captions.
15
15
 
16
- First, the [`@remotion/captions`](https://www.remotion.dev/docs/captions) package needs to be installed.
16
+ First, the [`@remotion/captions`](https://www.remotion.dev/docs/captions.md) package needs to be installed.
17
17
  If it is not installed, use the following command:
18
18
 
19
19
  ```bash
20
- npx remotion add @remotion/captions
20
+ npx remotion add @remotion/captions # If project uses npm
21
+ bunx remotion add @remotion/captions # If project uses bun
22
+ yarn remotion add @remotion/captions # If project uses yarn
23
+ pnpm exec remotion add @remotion/captions # If project uses pnpm
21
24
  ```
22
25
 
23
- ## Fetching captions
26
+ ## Adding the Basic Captions element
24
27
 
25
- First, fetch your captions JSON file. Use [`useDelayRender()`](https://www.remotion.dev/docs/use-delay-render) to hold the render until the captions are loaded:
28
+ Use the [Basic Captions](https://www.remotion.dev/elements/captions/basic-captions) element to display captions.
26
29
 
27
- ```tsx
28
- import { useState, useEffect, useCallback } from "react";
29
- import { AbsoluteFill, staticFile, useDelayRender } from "remotion";
30
- import type { Caption } from "@remotion/captions";
30
+ Alternatives are available at [Remotion Captions Elements](https://www.remotion.dev/elements/captions), but use Basic Captions unless there are more precise specifications.
31
31
 
32
- export const MyComponent: React.FC = () => {
33
- const [captions, setCaptions] = useState<Caption[] | null>(null);
34
- const { delayRender, continueRender, cancelRender } = useDelayRender();
35
- const [handle] = useState(() => delayRender());
32
+ Fetch its source code from https://www.remotion.dev/elements/captions/basic-captions.md and copy the `basic-captions.tsx` file into the project unchanged.
36
33
 
37
- const fetchCaptions = useCallback(async () => {
38
- try {
39
- // Assuming captions.json is in the public/ folder.
40
- const response = await fetch(staticFile("captions123.json"));
41
- const data = await response.json();
42
- setCaptions(data);
43
- continueRender(handle);
44
- } catch (e) {
45
- cancelRender(e);
46
- }
47
- }, [continueRender, cancelRender, handle]);
34
+ ## Displaying captions alongside video content
48
35
 
49
- useEffect(() => {
50
- fetchCaptions();
51
- }, [fetchCaptions]);
36
+ Keep each clip's video and captions inside the same `<Sequence>` or `<TransitionSeries.Sequence>`.
37
+ Put the clip's `trimBefore`, `durationInFrames`, and `playbackRate` on that shared parent so trimming or changing the speed affects both.
38
+ For independently positioned clips, put `from` on the shared `<Sequence>` too.
52
39
 
53
- if (!captions) {
54
- return null;
55
- }
40
+ Give every captioned clip its own group and inline caption array; do not combine captions from multiple clips into one overlay outside the scene timeline.
41
+ Moving, reordering, trimming, or deleting the group then affects its video and captions together.
56
42
 
57
- return (
58
- <AbsoluteFill>
59
- {/* Render captions here */}
60
- </AbsoluteFill>
61
- );
62
- };
63
- ```
64
-
65
- ## Creating pages
66
-
67
- Use `createTikTokStyleCaptions()` to group captions into pages. The `combineTokensWithinMilliseconds` option controls how many words appear at once:
68
-
69
- ```tsx
70
- import { useMemo } from "react";
71
- import { createTikTokStyleCaptions } from "@remotion/captions";
72
- import type { Caption } from "@remotion/captions";
43
+ Putting captions inside a scene is insufficient if `trimBefore` or `playbackRate` is still applied only to `<Video>`: its sibling captions do not inherit those props. Apply them to the shared parent instead.
73
44
 
74
- // How often captions should switch (in milliseconds)
75
- // Higher values = more words per page
76
- // Lower values = fewer words (more word-by-word)
77
- const SWITCH_CAPTIONS_EVERY_MS = 1200;
45
+ The Caption editor writes back to the array literal on the selected `<BasicCaptions>` source node, so inline the captions directly in its `captions` prop.
46
+ Passing a variable or component prop such as `captions={captions}` leaves no source array for the editor to update.
47
+ If the captions were transcribed to a JSON file, copy its contents into the prop.
78
48
 
79
- const { pages } = useMemo(() => {
80
- return createTikTokStyleCaptions({
81
- captions,
82
- combineTokensWithinMilliseconds: SWITCH_CAPTIONS_EVERY_MS,
83
- });
84
- }, [captions]);
85
- ```
86
-
87
- If a caption has `pageBreakAfter: true`, the current page ends after that caption and the next caption starts a new page.
88
-
89
- ## Rendering with Sequences
49
+ Give the caption area a width and position it over the video. In this
50
+ 1920px-wide composition, the 900px caption area is centered by translating it
51
+ `(1920 - 900) / 2 = 510` pixels from the left:
90
52
 
91
- Map over the pages and render each one in a `<Sequence>`. Calculate the start frame and duration from the page timing:
53
+ ```tsx title="src/MyComposition.tsx"
54
+ import { Video } from "@remotion/media";
55
+ import { TransitionSeries } from "@remotion/transitions";
56
+ import { staticFile, useVideoConfig } from "remotion";
57
+ import { BasicCaptions } from "./basic-captions";
92
58
 
93
- ```tsx
94
- import { Sequence, useVideoConfig, AbsoluteFill } from "remotion";
95
- import type { TikTokPage } from "@remotion/captions";
96
-
97
- const CaptionedContent: React.FC = () => {
59
+ export const MyComposition: React.FC = () => {
98
60
  const { fps } = useVideoConfig();
99
61
 
100
62
  return (
101
- <AbsoluteFill>
102
- {pages.map((page, index) => {
103
- const nextPage = pages[index + 1] ?? null;
104
- const startFrame = (page.startMs / 1000) * fps;
105
- const endFrame = Math.min(
106
- nextPage ? (nextPage.startMs / 1000) * fps : Infinity,
107
- startFrame + (SWITCH_CAPTIONS_EVERY_MS / 1000) * fps,
108
- );
109
- const durationInFrames = endFrame - startFrame;
110
-
111
- if (durationInFrames <= 0) {
112
- return null;
113
- }
114
-
115
- return (
116
- <Sequence
117
- key={index}
118
- from={startFrame}
119
- durationInFrames={durationInFrames}
120
- >
121
- <CaptionPage page={page} />
122
- </Sequence>
123
- );
124
- })}
125
- </AbsoluteFill>
63
+ <TransitionSeries>
64
+ <TransitionSeries.Sequence
65
+ name="Opening"
66
+ trimBefore={4 * fps}
67
+ durationInFrames={20 * fps}
68
+ >
69
+ <Video
70
+ name="Opening footage"
71
+ src={staticFile("opening.mp4")}
72
+ objectFit="cover"
73
+ style={{ width: "100%", height: "100%" }}
74
+ />
75
+ <BasicCaptions
76
+ name="Opening captions"
77
+ width={900}
78
+ style={{
79
+ position: "absolute",
80
+ bottom: 120,
81
+ translate: "510px 0px",
82
+ }}
83
+ captions={[
84
+ {
85
+ text: "Hello",
86
+ startMs: 5000,
87
+ endMs: 5400,
88
+ timestampMs: 5200,
89
+ confidence: null,
90
+ },
91
+ {
92
+ text: " world",
93
+ startMs: 5400,
94
+ endMs: 5900,
95
+ timestampMs: 5650,
96
+ confidence: null,
97
+ },
98
+ ]}
99
+ />
100
+ </TransitionSeries.Sequence>
101
+ <TransitionSeries.Sequence
102
+ name="Next clip"
103
+ trimBefore={1 * fps}
104
+ durationInFrames={5 * fps}
105
+ >
106
+ <Video
107
+ name="Next footage"
108
+ src={staticFile("next.mp4")}
109
+ objectFit="cover"
110
+ style={{ width: "100%", height: "100%" }}
111
+ />
112
+ <BasicCaptions
113
+ name="Next captions"
114
+ width={900}
115
+ style={{
116
+ position: "absolute",
117
+ bottom: 120,
118
+ translate: "510px 0px",
119
+ }}
120
+ captions={[
121
+ {
122
+ text: "Next",
123
+ startMs: 2000,
124
+ endMs: 2400,
125
+ timestampMs: 2200,
126
+ confidence: null,
127
+ },
128
+ {
129
+ text: " clip",
130
+ startMs: 2400,
131
+ endMs: 2900,
132
+ timestampMs: 2650,
133
+ confidence: null,
134
+ },
135
+ ]}
136
+ />
137
+ </TransitionSeries.Sequence>
138
+ </TransitionSeries>
126
139
  );
127
140
  };
128
141
  ```
129
142
 
130
- ## White-space preservation
131
-
132
- The captions are whitespace sensitive. You should include spaces in the `text` field before each word. Use `whiteSpace: "pre"` to preserve the whitespace in the captions.
133
-
134
- ## Separate component for captions
135
-
136
- Put captioning logic in a separate component.
137
- Make a new file for it.
138
-
139
- ## Word highlighting
140
-
141
- A caption page contains `tokens` which you can use to highlight the currently spoken word:
142
-
143
- ```tsx
144
- import { AbsoluteFill, useCurrentFrame, useVideoConfig } from "remotion";
145
- import type { TikTokPage } from "@remotion/captions";
146
-
147
- const HIGHLIGHT_COLOR = "#39E508";
148
-
149
- const CaptionPage: React.FC<{ page: TikTokPage }> = ({ page }) => {
150
- const frame = useCurrentFrame();
151
- const { fps } = useVideoConfig();
152
-
153
- // Current time relative to the start of the sequence
154
- const currentTimeMs = (frame / fps) * 1000;
155
- // Convert to absolute time by adding the page start
156
- const absoluteTimeMs = page.startMs + currentTimeMs;
143
+ ```tsx title="src/Root.tsx"
144
+ import { Composition } from "remotion";
145
+ import { MyComposition } from "./MyComposition";
157
146
 
147
+ export const RemotionRoot = () => {
158
148
  return (
159
- <AbsoluteFill style={{ justifyContent: "center", alignItems: "center" }}>
160
- <div style={{ fontSize: 80, fontWeight: "bold", whiteSpace: "pre" }}>
161
- {page.tokens.map((token, tokenIndex) => {
162
- const isActive =
163
- token.fromMs <= absoluteTimeMs && token.toMs > absoluteTimeMs;
164
-
165
- return (
166
- <span
167
- key={`${token.fromMs}-${tokenIndex}`}
168
- style={{ color: isActive ? HIGHLIGHT_COLOR : "white" }}
169
- >
170
- {token.text}
171
- </span>
172
- );
173
- })}
174
- </div>
175
- </AbsoluteFill>
149
+ <Composition
150
+ id="MyComposition"
151
+ component={MyComposition}
152
+ durationInFrames={750}
153
+ fps={30}
154
+ width={1920}
155
+ height={1080}
156
+ />
176
157
  );
177
158
  };
178
159
  ```
179
160
 
180
- ## Display captions alongside video content
161
+ To change a captioned clip's speed, set `playbackRate` on its shared sequence.
162
+ `durationInFrames` selects a range of source frames; the group occupies `durationInFrames / playbackRate` frames in its parent timeline.
163
+ Update the composition duration when changing the total timeline length.
164
+
165
+ If a caption has `pageBreakAfter: true`, the current page ends after that caption and the next caption starts a new page.
181
166
 
182
- By default, put the captions alongside the video content, so the captions are in sync.
183
- For each video, make a new captions JSON file.
167
+ ## White-space preservation
184
168
 
185
- ```tsx
186
- <AbsoluteFill>
187
- <Video src={staticFile("video.mp4")} />
188
- <CaptionPage page={page} />
189
- </AbsoluteFill>
190
- ```
169
+ The captions are whitespace sensitive. You should include spaces in the `text` field before each word.
@@ -29,7 +29,7 @@ Use `staticFile()` to reference an `.srt` file in your `public` folder, then fet
29
29
 
30
30
  ```tsx
31
31
  import { useState, useEffect, useCallback } from "react";
32
- import { AbsoluteFill, staticFile, useDelayRender } from "remotion";
32
+ import { staticFile, useDelayRender } from "remotion";
33
33
  import { parseSrt } from "@remotion/captions";
34
34
  import type { Caption } from "@remotion/captions";
35
35
 
@@ -59,9 +59,9 @@ export const MyComponent: React.FC = () => {
59
59
  }
60
60
 
61
61
  return (
62
- <AbsoluteFill>
62
+ <>
63
63
  {/* Use captions here */}
64
- </AbsoluteFill>
64
+ </>
65
65
  );
66
66
  };
67
67
  ```
@@ -1,70 +1,172 @@
1
- ---
2
- name: transcribe-captions
3
- description: Transcribing audio to generate captions in Remotion
4
- metadata:
5
- tags: captions, transcribe, whisper, audio, speech-to-text
6
- ---
7
-
8
1
  # Transcribing audio
9
2
 
10
- To transcribe audio to generate captions in Remotion, you can use the [`transcribe()`](https://www.remotion.dev/docs/install-whisper-cpp/transcribe) function from the [`@remotion/install-whisper-cpp`](https://www.remotion.dev/docs/install-whisper-cpp) package.
3
+ To transcribe audio to generate captions in Remotion, use the [`transcribe()`](https://www.remotion.dev/docs/whisper-webgpu/transcribe.md) function from the [`@remotion/whisper-webgpu`](https://www.remotion.dev/docs/whisper-webgpu.md) package.
4
+ It runs Whisper locally on the GPU and works in both Node.js and the browser.
11
5
 
12
6
  ## Prerequisites
13
7
 
14
- First, the @remotion/install-whisper-cpp package needs to be installed.
15
- If it is not installed, use the following command:
8
+ Install the required packages if they are not installed:
16
9
 
17
10
  ```bash
18
- npx remotion add @remotion/install-whisper-cpp
11
+ npx remotion add @remotion/whisper-webgpu @huggingface/transformers mediabunny @mediabunny/server # If project uses npm
12
+ bunx remotion add @remotion/whisper-webgpu @huggingface/transformers mediabunny @mediabunny/server # If project uses bun
13
+ yarn remotion add @remotion/whisper-webgpu @huggingface/transformers mediabunny @mediabunny/server # If project uses yarn
14
+ pnpm exec remotion add @remotion/whisper-webgpu @huggingface/transformers mediabunny @mediabunny/server # If project uses pnpm
19
15
  ```
20
16
 
17
+ A compatible GPU is required. ONNX Runtime does not work on Linux arm64.
18
+
21
19
  ## Transcribing
22
20
 
23
- Make a Node.js script to download Whisper.cpp and a model, and transcribe the audio.
21
+ Make a Node.js script that decodes the audio to a 16kHz mono waveform, downloads a model, and transcribes it.
24
22
 
25
23
  ```ts
26
- import path from "path";
24
+ import { registerMediabunnyServer } from "@mediabunny/server";
27
25
  import {
26
+ WHISPER_WEBGPU_SAMPLE_RATE,
27
+ canUseWhisperWebGpu,
28
28
  downloadWhisperModel,
29
- installWhisperCpp,
30
- transcribe,
29
+ loadWhisperModel,
31
30
  toCaptions,
32
- } from "@remotion/install-whisper-cpp";
33
- import fs from "fs";
34
-
35
- const to = path.join(process.cwd(), "whisper.cpp");
36
-
37
- await installWhisperCpp({
38
- to,
39
- version: "1.5.5",
31
+ transcribe,
32
+ } from "@remotion/whisper-webgpu";
33
+ import {
34
+ ALL_FORMATS,
35
+ Conversion,
36
+ FilePathSource,
37
+ Input,
38
+ NullTarget,
39
+ Output,
40
+ WavOutputFormat,
41
+ } from "mediabunny";
42
+ import { writeFile } from "node:fs/promises";
43
+
44
+ registerMediabunnyServer();
45
+
46
+ const support = await canUseWhisperWebGpu();
47
+ if (!support.supported) {
48
+ throw new Error(support.detailedReason);
49
+ }
50
+
51
+ type WaveformChunk = {
52
+ startFrame: number;
53
+ waveform: Float32Array;
54
+ };
55
+ const chunks: WaveformChunk[] = [];
56
+
57
+ using input = new Input({
58
+ formats: ALL_FORMATS,
59
+ source: new FilePathSource("public/video123.mp4"),
40
60
  });
41
-
42
- await downloadWhisperModel({
43
- model: "medium.en",
44
- folder: to,
61
+ const audioTrack = await input.getPrimaryAudioTrack();
62
+ if (audioTrack === null) {
63
+ throw new Error("The media does not contain an audio track.");
64
+ }
65
+
66
+ const conversion = await Conversion.init({
67
+ input,
68
+ output: new Output({
69
+ format: new WavOutputFormat(),
70
+ target: new NullTarget(),
71
+ }),
72
+ video: { discard: true },
73
+ audio: (track) => {
74
+ if (track.id !== audioTrack.id) {
75
+ return { discard: true };
76
+ }
77
+
78
+ return {
79
+ codec: "pcm-f32",
80
+ forceTranscode: true,
81
+ numberOfChannels: 1,
82
+ sampleFormat: "f32",
83
+ sampleRate: WHISPER_WEBGPU_SAMPLE_RATE,
84
+ process: (sample) => {
85
+ const waveform = new Float32Array(
86
+ sample.allocationSize({ format: "f32", planeIndex: 0 }) /
87
+ Float32Array.BYTES_PER_ELEMENT,
88
+ );
89
+ sample.copyTo(waveform, { format: "f32", planeIndex: 0 });
90
+ chunks.push({
91
+ startFrame: Math.round(sample.timestamp * WHISPER_WEBGPU_SAMPLE_RATE),
92
+ waveform,
93
+ });
94
+ return sample;
95
+ },
96
+ };
97
+ },
45
98
  });
46
99
 
47
- // Convert the audio to a 16KHz wav file first if needed:
48
- // import {execSync} from 'child_process';
49
- // execSync('ffmpeg -i /path/to/audio.mp4 -ar 16000 /path/to/audio.wav -y');
100
+ if (!conversion.isValid) {
101
+ throw new Error("The audio track cannot be decoded.");
102
+ }
103
+
104
+ await conversion.execute();
105
+
106
+ const waveformLength = chunks.reduce(
107
+ (max, chunk) => Math.max(max, chunk.startFrame + chunk.waveform.length),
108
+ 0,
109
+ );
110
+ const channelWaveform = new Float32Array(waveformLength);
111
+ for (const chunk of chunks) {
112
+ const destinationStart = Math.max(0, chunk.startFrame);
113
+ const sourceStart = Math.max(0, -chunk.startFrame);
114
+ const availableLength = Math.min(
115
+ chunk.waveform.length - sourceStart,
116
+ channelWaveform.length - destinationStart,
117
+ );
118
+
119
+ if (availableLength > 0) {
120
+ channelWaveform.set(
121
+ chunk.waveform.subarray(sourceStart, sourceStart + availableLength),
122
+ destinationStart,
123
+ );
124
+ }
125
+ }
126
+
127
+ const model = "small.en";
128
+ await downloadWhisperModel({ model });
129
+ await using modelHandle = await loadWhisperModel({ model });
130
+ const transcription = await transcribe({ channelWaveform, model });
131
+ const { captions } = toCaptions({ whisperWebGpuOutput: transcription });
132
+
133
+ // Write it to a file so the captions can be inlined into the Remotion code
134
+ await writeFile("captions123.json", JSON.stringify(captions, null, 2));
135
+ ```
136
+
137
+ ## Choosing a model
50
138
 
51
- const whisperCppOutput = await transcribe({
52
- model: "medium.en",
53
- whisperPath: to,
54
- whisperCppVersion: "1.5.5",
55
- inputPath: "/path/to/audio123.wav",
56
- tokenLevelTimestamps: true,
57
- });
139
+ `small.en` is the recommended default for English.
140
+ For other languages, use a multilingual model such as `small` and pass the `language` option to `transcribe()` - automatic language detection is not supported.
141
+ See [`getAvailableModels()`](https://www.remotion.dev/docs/whisper-webgpu/get-available-models.md) for all models.
58
142
 
59
- // Optional: Apply our recommended postprocessing
60
- const { captions } = toCaptions({
61
- whisperCppOutput,
62
- });
143
+ ## Transcribing in the browser
63
144
 
64
- // Write it to the public/ folder so it can be fetched from Remotion
65
- fs.writeFileSync("captions123.json", JSON.stringify(captions, null, 2));
145
+ In the browser, use [`resampleTo16Khz()`](https://www.remotion.dev/docs/whisper-webgpu/resample-to-16khz.md) to get the waveform from a `File` instead of using Mediabunny:
146
+
147
+ ```ts
148
+ import {
149
+ downloadWhisperModel,
150
+ resampleTo16Khz,
151
+ toCaptions,
152
+ transcribe,
153
+ } from "@remotion/whisper-webgpu";
154
+
155
+ export const transcribeFile = async (file: File) => {
156
+ await downloadWhisperModel({ model: "small.en" });
157
+ const channelWaveform = await resampleTo16Khz({ file });
158
+ const transcription = await transcribe({
159
+ channelWaveform,
160
+ model: "small.en",
161
+ });
162
+
163
+ const { captions } = toCaptions({ whisperWebGpuOutput: transcription });
164
+ return captions;
165
+ };
66
166
  ```
67
167
 
68
- Transcribe each clip individually and create multiple JSON files.
168
+ Transcribe each original source video or audio file individually.
169
+ Keep the resulting caption timestamps relative to the beginning of that source file, including portions currently trimmed out of the edit.
170
+ Do not add composition offsets or flatten multiple transcripts into composition-wide timestamps.
69
171
 
70
172
  See [Displaying captions](display-captions.md) for how to display the captions in Remotion.
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: remotion-create
3
3
  description: Create a new Remotion video
4
- version: 4.0.529
4
+ version: 4.0.539
5
5
  ---
6
6
 
7
7
  These are instructions for making a new Remotion project and composition.
@@ -32,21 +32,34 @@ npm i
32
32
  If the current folder contains meaningful contents and no project already exists, scaffold into a new subfolder.
33
33
  Replace `my-video` with a suitable project name.
34
34
 
35
-
36
35
  ```bash
37
36
  npx create-video@latest --yes --blank --no-tailwind my-video
38
37
  cd my-video
39
38
  npm i
40
39
  ```
41
40
 
41
+ ## Open the preview before building the video
42
+
43
+ As soon as the project can run, open [Remotion Studio](../remotion-studio/REFERENCE.md) in the browser before writing or changing the composition. For a new project, do this immediately after scaffolding and installing dependencies. For an existing project, do it before editing the video. Keep Studio running while you work so the user can see changes as they appear and steer the result.
44
+
42
45
  ## Designing a video
43
46
 
44
47
  Keep the scaffold and add React Markup.
45
48
  Follow [Remotion React Markup Best Practices](../remotion-markup/REFERENCE.md) and [Video Layout Rules](video-layout.md) for video-first layout and text sizing guidance.
46
49
 
50
+ Before writing the root that registers `<Composition>` or `<Still>` elements,
51
+ follow [Compositions and stills](../remotion-markup/compositions.md).
52
+
53
+ ## Is this a timeline of clips?
54
+
55
+ If the video arranges multiple video or audio clips on a timeline, follow
56
+ [Video editing](../remotion-markup/video-editing.md) before writing the
57
+ timeline. Give every clip that should be edited independently its own authored
58
+ JSX node.
59
+
47
60
  ## Is this a multi-scene video?
48
61
 
49
- If this is a video with multiple subsequence videos, follow guidance at [Multi-scene videos](../remotion-markup/multi-scene-video.md).
62
+ If this is a video with multiple subsequent scenes, follow guidance at [Multi-scene videos](../remotion-markup/multi-scene-video.md).
50
63
 
51
64
  ## Interactivity Best Practices
52
65
 
@@ -56,29 +69,6 @@ By structuring the React Markup following [Remotion Interactivity Best Practices
56
69
 
57
70
  If Tailwind is requested, see [tailwind.md](tailwind.md) for using TailwindCSS in Remotion.
58
71
 
59
- ## Open the preview
60
-
61
- Start the preview server after building the composition:
62
-
63
- ```bash
64
- npx remotion studio --no-open
65
- ```
66
-
67
- This will start a long-running process and print the server URL for the preview.
68
- If the server is already started, it will print the URL.
69
- If an in-harness browser is available, open it there.
70
- You can visit a specific composition by navigating to `/[composition-id]`, for example `http://localhost:3000/MapAnimation`.
71
-
72
- ## Render the video
73
-
74
- Only render if the user explicitly asks for it.
75
-
76
- ```
77
- npx remotion render
78
- ```
79
-
80
- For more options, see [Rendering](../remotion-render/REFERENCE.md).
81
-
82
72
  ## Follow-up
83
73
 
84
74
  The video creation process has finished.
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  name: remotion-docs
3
3
  description: Search Remotion documentation
4
- version: 4.0.529
4
+ version: 4.0.539
5
5
  ---
6
6
 
7
7
  This skill teaches you how to discover and read current Remotion documentation.