dsh-remotion 0.8.0 → 0.8.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -1
- package/README.en.md +3 -1
- package/README.md +3 -1
- package/docs/USAGE.en.md +3 -3
- package/docs/USAGE.md +3 -3
- package/lib/client.js +74 -54
- package/lib/media-templates.d.ts +3 -3
- package/lib/media-templates.js +1 -1
- package/package.json +2 -2
- package/scripts/sync-skills.mjs +3 -2
- package/skills/UPSTREAM.json +297 -0
- package/skills/remotion-best-practices/SKILL.md +58 -4
- package/skills/remotion-best-practices/remotion-captions/REFERENCE.md +7 -1
- package/skills/remotion-best-practices/remotion-captions/display-captions.md +126 -147
- package/skills/remotion-best-practices/remotion-captions/import-srt-captions.md +3 -3
- package/skills/remotion-best-practices/remotion-captions/transcribe-captions.md +146 -44
- package/skills/remotion-best-practices/remotion-create/REFERENCE.md +16 -26
- package/skills/remotion-best-practices/remotion-docs/REFERENCE.md +1 -1
- package/skills/remotion-best-practices/remotion-interactivity/REFERENCE.md +70 -53
- package/skills/remotion-best-practices/remotion-interactivity/inline-effects.md +42 -0
- package/skills/remotion-best-practices/remotion-interactivity/timeline-structure.md +215 -0
- package/skills/remotion-best-practices/remotion-maps/REFERENCE.md +1 -1
- package/skills/remotion-best-practices/remotion-maps/techniques/mapbox/TECHNIQUE.md +3 -3
- package/skills/remotion-best-practices/remotion-maps/techniques/maplibre/TECHNIQUE.md +3 -3
- package/skills/remotion-best-practices/remotion-markup/REFERENCE.md +83 -113
- package/skills/remotion-best-practices/remotion-markup/arabic-text.md +88 -0
- package/skills/remotion-best-practices/remotion-markup/audio.md +35 -14
- package/skills/remotion-best-practices/remotion-markup/color-correction.md +190 -0
- package/skills/remotion-best-practices/remotion-markup/compositions.md +89 -34
- package/skills/remotion-best-practices/remotion-markup/connected-compositions.md +55 -69
- package/skills/remotion-best-practices/remotion-markup/cropping.md +2 -0
- package/skills/remotion-best-practices/remotion-markup/effects.md +34 -5
- package/skills/remotion-best-practices/remotion-markup/embedding-videos.md +20 -13
- package/skills/remotion-best-practices/remotion-markup/ffmpeg.md +2 -2
- package/skills/remotion-best-practices/remotion-markup/html-in-canvas.md +5 -27
- package/skills/remotion-best-practices/remotion-markup/images.md +5 -7
- package/skills/remotion-best-practices/remotion-markup/light-leaks.md +2 -4
- package/skills/remotion-best-practices/remotion-markup/motion-blur.md +2 -2
- package/skills/remotion-best-practices/remotion-markup/multi-scene-video.md +19 -79
- package/skills/remotion-best-practices/remotion-markup/remotion-maps/REFERENCE.md +1 -1
- package/skills/remotion-best-practices/remotion-markup/remotion-maps/techniques/mapbox/TECHNIQUE.md +3 -3
- package/skills/remotion-best-practices/remotion-markup/remotion-maps/techniques/maplibre/TECHNIQUE.md +3 -3
- package/skills/remotion-best-practices/remotion-markup/sequencing.md +21 -162
- package/skills/remotion-best-practices/remotion-markup/sfx.md +2 -2
- package/skills/remotion-best-practices/remotion-markup/silence-detection.md +2 -2
- package/skills/remotion-best-practices/remotion-markup/timing-props.md +90 -0
- package/skills/remotion-best-practices/remotion-markup/transitions.md +121 -74
- package/skills/remotion-best-practices/remotion-markup/video-editing.md +127 -75
- package/skills/remotion-best-practices/remotion-markup/voiceover.md +6 -2
- package/skills/remotion-best-practices/remotion-multimedia/REFERENCE.md +1 -1
- package/skills/remotion-best-practices/remotion-multimedia/get-audio-duration.md +8 -3
- package/skills/remotion-best-practices/remotion-multimedia/get-video-dimensions.md +9 -3
- package/skills/remotion-best-practices/remotion-multimedia/get-video-duration.md +6 -3
- package/skills/remotion-best-practices/remotion-render/REFERENCE.md +1 -1
- package/skills/remotion-best-practices/remotion-saas/REFERENCE.md +1 -1
- package/skills/remotion-best-practices/remotion-studio/REFERENCE.md +11 -3
- package/skills/remotion-best-practices/remotion-upgrade/REFERENCE.md +1 -1
- package/skills/remotion-captions/SKILL.md +11 -1
- package/skills/remotion-captions/display-captions.md +126 -147
- package/skills/remotion-captions/import-srt-captions.md +3 -3
- package/skills/remotion-captions/transcribe-captions.md +146 -44
- package/skills/remotion-create/SKILL.md +20 -26
- package/skills/remotion-docs/SKILL.md +5 -1
- package/skills/remotion-interactivity/SKILL.md +74 -53
- package/skills/remotion-interactivity/inline-effects.md +42 -0
- package/skills/remotion-interactivity/timeline-structure.md +215 -0
- package/skills/remotion-maps/SKILL.md +5 -1
- package/skills/remotion-maps/techniques/mapbox/TECHNIQUE.md +3 -3
- package/skills/remotion-maps/techniques/maplibre/TECHNIQUE.md +3 -3
- package/skills/remotion-markup/SKILL.md +87 -113
- package/skills/remotion-markup/arabic-text.md +88 -0
- package/skills/remotion-markup/audio.md +35 -14
- package/skills/remotion-markup/color-correction.md +190 -0
- package/skills/remotion-markup/compositions.md +89 -34
- package/skills/remotion-markup/connected-compositions.md +55 -69
- package/skills/remotion-markup/cropping.md +2 -0
- package/skills/remotion-markup/effects.md +34 -5
- package/skills/remotion-markup/embedding-videos.md +20 -13
- package/skills/remotion-markup/ffmpeg.md +2 -2
- package/skills/remotion-markup/html-in-canvas.md +5 -27
- package/skills/remotion-markup/images.md +5 -7
- package/skills/remotion-markup/light-leaks.md +2 -4
- package/skills/remotion-markup/motion-blur.md +2 -2
- package/skills/remotion-markup/multi-scene-video.md +19 -79
- package/skills/remotion-markup/remotion-maps/REFERENCE.md +1 -1
- package/skills/remotion-markup/remotion-maps/techniques/mapbox/TECHNIQUE.md +3 -3
- package/skills/remotion-markup/remotion-maps/techniques/maplibre/TECHNIQUE.md +3 -3
- package/skills/remotion-markup/sequencing.md +21 -162
- package/skills/remotion-markup/sfx.md +2 -2
- package/skills/remotion-markup/silence-detection.md +2 -2
- package/skills/remotion-markup/timing-props.md +90 -0
- package/skills/remotion-markup/transitions.md +121 -74
- package/skills/remotion-markup/video-editing.md +127 -75
- package/skills/remotion-markup/voiceover.md +6 -2
- package/skills/remotion-multimedia/SKILL.md +5 -1
- package/skills/remotion-multimedia/get-audio-duration.md +8 -3
- package/skills/remotion-multimedia/get-video-dimensions.md +9 -3
- package/skills/remotion-multimedia/get-video-duration.md +6 -3
- package/skills/remotion-render/SKILL.md +5 -1
- package/skills/remotion-saas/SKILL.md +5 -1
- package/skills/remotion-studio/SKILL.md +15 -3
- package/skills/remotion-upgrade/SKILL.md +5 -1
|
@@ -1,190 +1,169 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: display-captions
|
|
3
|
-
description: Displaying captions in Remotion
|
|
3
|
+
description: Displaying captions in Remotion using the Basic Captions element
|
|
4
4
|
metadata:
|
|
5
|
-
tags: captions, subtitles, display,
|
|
5
|
+
tags: captions, subtitles, display, element
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
# Displaying captions in Remotion
|
|
9
9
|
|
|
10
|
-
This guide explains how to display captions in Remotion, assuming you already have captions in the [`Caption`](https://www.remotion.dev/docs/captions/caption) format.
|
|
10
|
+
This guide explains how to display captions in Remotion, assuming you already have captions in the [`Caption`](https://www.remotion.dev/docs/captions/caption.md) format.
|
|
11
11
|
|
|
12
12
|
## Prerequisites
|
|
13
13
|
|
|
14
14
|
Read [Transcribing audio](transcribe-captions.md) for how to generate captions.
|
|
15
15
|
|
|
16
|
-
First, the [`@remotion/captions`](https://www.remotion.dev/docs/captions) package needs to be installed.
|
|
16
|
+
First, the [`@remotion/captions`](https://www.remotion.dev/docs/captions.md) package needs to be installed.
|
|
17
17
|
If it is not installed, use the following command:
|
|
18
18
|
|
|
19
19
|
```bash
|
|
20
|
-
npx remotion add @remotion/captions
|
|
20
|
+
npx remotion add @remotion/captions # If project uses npm
|
|
21
|
+
bunx remotion add @remotion/captions # If project uses bun
|
|
22
|
+
yarn remotion add @remotion/captions # If project uses yarn
|
|
23
|
+
pnpm exec remotion add @remotion/captions # If project uses pnpm
|
|
21
24
|
```
|
|
22
25
|
|
|
23
|
-
##
|
|
26
|
+
## Adding the Basic Captions element
|
|
24
27
|
|
|
25
|
-
|
|
28
|
+
Use the [Basic Captions](https://www.remotion.dev/elements/captions/basic-captions) element to display captions.
|
|
26
29
|
|
|
27
|
-
|
|
28
|
-
import { useState, useEffect, useCallback } from "react";
|
|
29
|
-
import { AbsoluteFill, staticFile, useDelayRender } from "remotion";
|
|
30
|
-
import type { Caption } from "@remotion/captions";
|
|
30
|
+
Alternatives are available at [Remotion Captions Elements](https://www.remotion.dev/elements/captions), but use Basic Captions unless there are more precise specifications.
|
|
31
31
|
|
|
32
|
-
|
|
33
|
-
const [captions, setCaptions] = useState<Caption[] | null>(null);
|
|
34
|
-
const { delayRender, continueRender, cancelRender } = useDelayRender();
|
|
35
|
-
const [handle] = useState(() => delayRender());
|
|
32
|
+
Fetch its source code from https://www.remotion.dev/elements/captions/basic-captions.md and copy the `basic-captions.tsx` file into the project unchanged.
|
|
36
33
|
|
|
37
|
-
|
|
38
|
-
try {
|
|
39
|
-
// Assuming captions.json is in the public/ folder.
|
|
40
|
-
const response = await fetch(staticFile("captions123.json"));
|
|
41
|
-
const data = await response.json();
|
|
42
|
-
setCaptions(data);
|
|
43
|
-
continueRender(handle);
|
|
44
|
-
} catch (e) {
|
|
45
|
-
cancelRender(e);
|
|
46
|
-
}
|
|
47
|
-
}, [continueRender, cancelRender, handle]);
|
|
34
|
+
## Displaying captions alongside video content
|
|
48
35
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
36
|
+
Keep each clip's video and captions inside the same `<Sequence>` or `<TransitionSeries.Sequence>`.
|
|
37
|
+
Put the clip's `trimBefore`, `durationInFrames`, and `playbackRate` on that shared parent so trimming or changing the speed affects both.
|
|
38
|
+
For independently positioned clips, put `from` on the shared `<Sequence>` too.
|
|
52
39
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
}
|
|
40
|
+
Give every captioned clip its own group and inline caption array; do not combine captions from multiple clips into one overlay outside the scene timeline.
|
|
41
|
+
Moving, reordering, trimming, or deleting the group then affects its video and captions together.
|
|
56
42
|
|
|
57
|
-
|
|
58
|
-
<AbsoluteFill>
|
|
59
|
-
{/* Render captions here */}
|
|
60
|
-
</AbsoluteFill>
|
|
61
|
-
);
|
|
62
|
-
};
|
|
63
|
-
```
|
|
64
|
-
|
|
65
|
-
## Creating pages
|
|
66
|
-
|
|
67
|
-
Use `createTikTokStyleCaptions()` to group captions into pages. The `combineTokensWithinMilliseconds` option controls how many words appear at once:
|
|
68
|
-
|
|
69
|
-
```tsx
|
|
70
|
-
import { useMemo } from "react";
|
|
71
|
-
import { createTikTokStyleCaptions } from "@remotion/captions";
|
|
72
|
-
import type { Caption } from "@remotion/captions";
|
|
43
|
+
Putting captions inside a scene is insufficient if `trimBefore` or `playbackRate` is still applied only to `<Video>`: its sibling captions do not inherit those props. Apply them to the shared parent instead.
|
|
73
44
|
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
const SWITCH_CAPTIONS_EVERY_MS = 1200;
|
|
45
|
+
The Caption editor writes back to the array literal on the selected `<BasicCaptions>` source node, so inline the captions directly in its `captions` prop.
|
|
46
|
+
Passing a variable or component prop such as `captions={captions}` leaves no source array for the editor to update.
|
|
47
|
+
If the captions were transcribed to a JSON file, copy its contents into the prop.
|
|
78
48
|
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
combineTokensWithinMilliseconds: SWITCH_CAPTIONS_EVERY_MS,
|
|
83
|
-
});
|
|
84
|
-
}, [captions]);
|
|
85
|
-
```
|
|
86
|
-
|
|
87
|
-
If a caption has `pageBreakAfter: true`, the current page ends after that caption and the next caption starts a new page.
|
|
88
|
-
|
|
89
|
-
## Rendering with Sequences
|
|
49
|
+
Give the caption area a width and position it over the video. In this
|
|
50
|
+
1920px-wide composition, the 900px caption area is centered by translating it
|
|
51
|
+
`(1920 - 900) / 2 = 510` pixels from the left:
|
|
90
52
|
|
|
91
|
-
|
|
53
|
+
```tsx title="src/MyComposition.tsx"
|
|
54
|
+
import { Video } from "@remotion/media";
|
|
55
|
+
import { TransitionSeries } from "@remotion/transitions";
|
|
56
|
+
import { staticFile, useVideoConfig } from "remotion";
|
|
57
|
+
import { BasicCaptions } from "./basic-captions";
|
|
92
58
|
|
|
93
|
-
|
|
94
|
-
import { Sequence, useVideoConfig, AbsoluteFill } from "remotion";
|
|
95
|
-
import type { TikTokPage } from "@remotion/captions";
|
|
96
|
-
|
|
97
|
-
const CaptionedContent: React.FC = () => {
|
|
59
|
+
export const MyComposition: React.FC = () => {
|
|
98
60
|
const { fps } = useVideoConfig();
|
|
99
61
|
|
|
100
62
|
return (
|
|
101
|
-
<
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
63
|
+
<TransitionSeries>
|
|
64
|
+
<TransitionSeries.Sequence
|
|
65
|
+
name="Opening"
|
|
66
|
+
trimBefore={4 * fps}
|
|
67
|
+
durationInFrames={20 * fps}
|
|
68
|
+
>
|
|
69
|
+
<Video
|
|
70
|
+
name="Opening footage"
|
|
71
|
+
src={staticFile("opening.mp4")}
|
|
72
|
+
objectFit="cover"
|
|
73
|
+
style={{ width: "100%", height: "100%" }}
|
|
74
|
+
/>
|
|
75
|
+
<BasicCaptions
|
|
76
|
+
name="Opening captions"
|
|
77
|
+
width={900}
|
|
78
|
+
style={{
|
|
79
|
+
position: "absolute",
|
|
80
|
+
bottom: 120,
|
|
81
|
+
translate: "510px 0px",
|
|
82
|
+
}}
|
|
83
|
+
captions={[
|
|
84
|
+
{
|
|
85
|
+
text: "Hello",
|
|
86
|
+
startMs: 5000,
|
|
87
|
+
endMs: 5400,
|
|
88
|
+
timestampMs: 5200,
|
|
89
|
+
confidence: null,
|
|
90
|
+
},
|
|
91
|
+
{
|
|
92
|
+
text: " world",
|
|
93
|
+
startMs: 5400,
|
|
94
|
+
endMs: 5900,
|
|
95
|
+
timestampMs: 5650,
|
|
96
|
+
confidence: null,
|
|
97
|
+
},
|
|
98
|
+
]}
|
|
99
|
+
/>
|
|
100
|
+
</TransitionSeries.Sequence>
|
|
101
|
+
<TransitionSeries.Sequence
|
|
102
|
+
name="Next clip"
|
|
103
|
+
trimBefore={1 * fps}
|
|
104
|
+
durationInFrames={5 * fps}
|
|
105
|
+
>
|
|
106
|
+
<Video
|
|
107
|
+
name="Next footage"
|
|
108
|
+
src={staticFile("next.mp4")}
|
|
109
|
+
objectFit="cover"
|
|
110
|
+
style={{ width: "100%", height: "100%" }}
|
|
111
|
+
/>
|
|
112
|
+
<BasicCaptions
|
|
113
|
+
name="Next captions"
|
|
114
|
+
width={900}
|
|
115
|
+
style={{
|
|
116
|
+
position: "absolute",
|
|
117
|
+
bottom: 120,
|
|
118
|
+
translate: "510px 0px",
|
|
119
|
+
}}
|
|
120
|
+
captions={[
|
|
121
|
+
{
|
|
122
|
+
text: "Next",
|
|
123
|
+
startMs: 2000,
|
|
124
|
+
endMs: 2400,
|
|
125
|
+
timestampMs: 2200,
|
|
126
|
+
confidence: null,
|
|
127
|
+
},
|
|
128
|
+
{
|
|
129
|
+
text: " clip",
|
|
130
|
+
startMs: 2400,
|
|
131
|
+
endMs: 2900,
|
|
132
|
+
timestampMs: 2650,
|
|
133
|
+
confidence: null,
|
|
134
|
+
},
|
|
135
|
+
]}
|
|
136
|
+
/>
|
|
137
|
+
</TransitionSeries.Sequence>
|
|
138
|
+
</TransitionSeries>
|
|
126
139
|
);
|
|
127
140
|
};
|
|
128
141
|
```
|
|
129
142
|
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
## Separate component for captions
|
|
135
|
-
|
|
136
|
-
Put captioning logic in a separate component.
|
|
137
|
-
Make a new file for it.
|
|
138
|
-
|
|
139
|
-
## Word highlighting
|
|
140
|
-
|
|
141
|
-
A caption page contains `tokens` which you can use to highlight the currently spoken word:
|
|
142
|
-
|
|
143
|
-
```tsx
|
|
144
|
-
import { AbsoluteFill, useCurrentFrame, useVideoConfig } from "remotion";
|
|
145
|
-
import type { TikTokPage } from "@remotion/captions";
|
|
146
|
-
|
|
147
|
-
const HIGHLIGHT_COLOR = "#39E508";
|
|
148
|
-
|
|
149
|
-
const CaptionPage: React.FC<{ page: TikTokPage }> = ({ page }) => {
|
|
150
|
-
const frame = useCurrentFrame();
|
|
151
|
-
const { fps } = useVideoConfig();
|
|
152
|
-
|
|
153
|
-
// Current time relative to the start of the sequence
|
|
154
|
-
const currentTimeMs = (frame / fps) * 1000;
|
|
155
|
-
// Convert to absolute time by adding the page start
|
|
156
|
-
const absoluteTimeMs = page.startMs + currentTimeMs;
|
|
143
|
+
```tsx title="src/Root.tsx"
|
|
144
|
+
import { Composition } from "remotion";
|
|
145
|
+
import { MyComposition } from "./MyComposition";
|
|
157
146
|
|
|
147
|
+
export const RemotionRoot = () => {
|
|
158
148
|
return (
|
|
159
|
-
<
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
key={`${token.fromMs}-${tokenIndex}`}
|
|
168
|
-
style={{ color: isActive ? HIGHLIGHT_COLOR : "white" }}
|
|
169
|
-
>
|
|
170
|
-
{token.text}
|
|
171
|
-
</span>
|
|
172
|
-
);
|
|
173
|
-
})}
|
|
174
|
-
</div>
|
|
175
|
-
</AbsoluteFill>
|
|
149
|
+
<Composition
|
|
150
|
+
id="MyComposition"
|
|
151
|
+
component={MyComposition}
|
|
152
|
+
durationInFrames={750}
|
|
153
|
+
fps={30}
|
|
154
|
+
width={1920}
|
|
155
|
+
height={1080}
|
|
156
|
+
/>
|
|
176
157
|
);
|
|
177
158
|
};
|
|
178
159
|
```
|
|
179
160
|
|
|
180
|
-
|
|
161
|
+
To change a captioned clip's speed, set `playbackRate` on its shared sequence.
|
|
162
|
+
`durationInFrames` selects a range of source frames; the group occupies `durationInFrames / playbackRate` frames in its parent timeline.
|
|
163
|
+
Update the composition duration when changing the total timeline length.
|
|
164
|
+
|
|
165
|
+
If a caption has `pageBreakAfter: true`, the current page ends after that caption and the next caption starts a new page.
|
|
181
166
|
|
|
182
|
-
|
|
183
|
-
For each video, make a new captions JSON file.
|
|
167
|
+
## White-space preservation
|
|
184
168
|
|
|
185
|
-
|
|
186
|
-
<AbsoluteFill>
|
|
187
|
-
<Video src={staticFile("video.mp4")} />
|
|
188
|
-
<CaptionPage page={page} />
|
|
189
|
-
</AbsoluteFill>
|
|
190
|
-
```
|
|
169
|
+
The captions are whitespace sensitive. You should include spaces in the `text` field before each word.
|
|
@@ -29,7 +29,7 @@ Use `staticFile()` to reference an `.srt` file in your `public` folder, then fet
|
|
|
29
29
|
|
|
30
30
|
```tsx
|
|
31
31
|
import { useState, useEffect, useCallback } from "react";
|
|
32
|
-
import {
|
|
32
|
+
import { staticFile, useDelayRender } from "remotion";
|
|
33
33
|
import { parseSrt } from "@remotion/captions";
|
|
34
34
|
import type { Caption } from "@remotion/captions";
|
|
35
35
|
|
|
@@ -59,9 +59,9 @@ export const MyComponent: React.FC = () => {
|
|
|
59
59
|
}
|
|
60
60
|
|
|
61
61
|
return (
|
|
62
|
-
|
|
62
|
+
<>
|
|
63
63
|
{/* Use captions here */}
|
|
64
|
-
|
|
64
|
+
</>
|
|
65
65
|
);
|
|
66
66
|
};
|
|
67
67
|
```
|
|
@@ -1,70 +1,172 @@
|
|
|
1
|
-
---
|
|
2
|
-
name: transcribe-captions
|
|
3
|
-
description: Transcribing audio to generate captions in Remotion
|
|
4
|
-
metadata:
|
|
5
|
-
tags: captions, transcribe, whisper, audio, speech-to-text
|
|
6
|
-
---
|
|
7
|
-
|
|
8
1
|
# Transcribing audio
|
|
9
2
|
|
|
10
|
-
To transcribe audio to generate captions in Remotion,
|
|
3
|
+
To transcribe audio to generate captions in Remotion, use the [`transcribe()`](https://www.remotion.dev/docs/whisper-webgpu/transcribe.md) function from the [`@remotion/whisper-webgpu`](https://www.remotion.dev/docs/whisper-webgpu.md) package.
|
|
4
|
+
It runs Whisper locally on the GPU and works in both Node.js and the browser.
|
|
11
5
|
|
|
12
6
|
## Prerequisites
|
|
13
7
|
|
|
14
|
-
|
|
15
|
-
If it is not installed, use the following command:
|
|
8
|
+
Install the required packages if they are not installed:
|
|
16
9
|
|
|
17
10
|
```bash
|
|
18
|
-
npx remotion add @remotion/
|
|
11
|
+
npx remotion add @remotion/whisper-webgpu @huggingface/transformers mediabunny @mediabunny/server # If project uses npm
|
|
12
|
+
bunx remotion add @remotion/whisper-webgpu @huggingface/transformers mediabunny @mediabunny/server # If project uses bun
|
|
13
|
+
yarn remotion add @remotion/whisper-webgpu @huggingface/transformers mediabunny @mediabunny/server # If project uses yarn
|
|
14
|
+
pnpm exec remotion add @remotion/whisper-webgpu @huggingface/transformers mediabunny @mediabunny/server # If project uses pnpm
|
|
19
15
|
```
|
|
20
16
|
|
|
17
|
+
A compatible GPU is required. ONNX Runtime does not work on Linux arm64.
|
|
18
|
+
|
|
21
19
|
## Transcribing
|
|
22
20
|
|
|
23
|
-
Make a Node.js script to
|
|
21
|
+
Make a Node.js script that decodes the audio to a 16kHz mono waveform, downloads a model, and transcribes it.
|
|
24
22
|
|
|
25
23
|
```ts
|
|
26
|
-
import
|
|
24
|
+
import { registerMediabunnyServer } from "@mediabunny/server";
|
|
27
25
|
import {
|
|
26
|
+
WHISPER_WEBGPU_SAMPLE_RATE,
|
|
27
|
+
canUseWhisperWebGpu,
|
|
28
28
|
downloadWhisperModel,
|
|
29
|
-
|
|
30
|
-
transcribe,
|
|
29
|
+
loadWhisperModel,
|
|
31
30
|
toCaptions,
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
31
|
+
transcribe,
|
|
32
|
+
} from "@remotion/whisper-webgpu";
|
|
33
|
+
import {
|
|
34
|
+
ALL_FORMATS,
|
|
35
|
+
Conversion,
|
|
36
|
+
FilePathSource,
|
|
37
|
+
Input,
|
|
38
|
+
NullTarget,
|
|
39
|
+
Output,
|
|
40
|
+
WavOutputFormat,
|
|
41
|
+
} from "mediabunny";
|
|
42
|
+
import { writeFile } from "node:fs/promises";
|
|
43
|
+
|
|
44
|
+
registerMediabunnyServer();
|
|
45
|
+
|
|
46
|
+
const support = await canUseWhisperWebGpu();
|
|
47
|
+
if (!support.supported) {
|
|
48
|
+
throw new Error(support.detailedReason);
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
type WaveformChunk = {
|
|
52
|
+
startFrame: number;
|
|
53
|
+
waveform: Float32Array;
|
|
54
|
+
};
|
|
55
|
+
const chunks: WaveformChunk[] = [];
|
|
56
|
+
|
|
57
|
+
using input = new Input({
|
|
58
|
+
formats: ALL_FORMATS,
|
|
59
|
+
source: new FilePathSource("public/video123.mp4"),
|
|
40
60
|
});
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
61
|
+
const audioTrack = await input.getPrimaryAudioTrack();
|
|
62
|
+
if (audioTrack === null) {
|
|
63
|
+
throw new Error("The media does not contain an audio track.");
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
const conversion = await Conversion.init({
|
|
67
|
+
input,
|
|
68
|
+
output: new Output({
|
|
69
|
+
format: new WavOutputFormat(),
|
|
70
|
+
target: new NullTarget(),
|
|
71
|
+
}),
|
|
72
|
+
video: { discard: true },
|
|
73
|
+
audio: (track) => {
|
|
74
|
+
if (track.id !== audioTrack.id) {
|
|
75
|
+
return { discard: true };
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
return {
|
|
79
|
+
codec: "pcm-f32",
|
|
80
|
+
forceTranscode: true,
|
|
81
|
+
numberOfChannels: 1,
|
|
82
|
+
sampleFormat: "f32",
|
|
83
|
+
sampleRate: WHISPER_WEBGPU_SAMPLE_RATE,
|
|
84
|
+
process: (sample) => {
|
|
85
|
+
const waveform = new Float32Array(
|
|
86
|
+
sample.allocationSize({ format: "f32", planeIndex: 0 }) /
|
|
87
|
+
Float32Array.BYTES_PER_ELEMENT,
|
|
88
|
+
);
|
|
89
|
+
sample.copyTo(waveform, { format: "f32", planeIndex: 0 });
|
|
90
|
+
chunks.push({
|
|
91
|
+
startFrame: Math.round(sample.timestamp * WHISPER_WEBGPU_SAMPLE_RATE),
|
|
92
|
+
waveform,
|
|
93
|
+
});
|
|
94
|
+
return sample;
|
|
95
|
+
},
|
|
96
|
+
};
|
|
97
|
+
},
|
|
45
98
|
});
|
|
46
99
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
100
|
+
if (!conversion.isValid) {
|
|
101
|
+
throw new Error("The audio track cannot be decoded.");
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
await conversion.execute();
|
|
105
|
+
|
|
106
|
+
const waveformLength = chunks.reduce(
|
|
107
|
+
(max, chunk) => Math.max(max, chunk.startFrame + chunk.waveform.length),
|
|
108
|
+
0,
|
|
109
|
+
);
|
|
110
|
+
const channelWaveform = new Float32Array(waveformLength);
|
|
111
|
+
for (const chunk of chunks) {
|
|
112
|
+
const destinationStart = Math.max(0, chunk.startFrame);
|
|
113
|
+
const sourceStart = Math.max(0, -chunk.startFrame);
|
|
114
|
+
const availableLength = Math.min(
|
|
115
|
+
chunk.waveform.length - sourceStart,
|
|
116
|
+
channelWaveform.length - destinationStart,
|
|
117
|
+
);
|
|
118
|
+
|
|
119
|
+
if (availableLength > 0) {
|
|
120
|
+
channelWaveform.set(
|
|
121
|
+
chunk.waveform.subarray(sourceStart, sourceStart + availableLength),
|
|
122
|
+
destinationStart,
|
|
123
|
+
);
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
const model = "small.en";
|
|
128
|
+
await downloadWhisperModel({ model });
|
|
129
|
+
await using modelHandle = await loadWhisperModel({ model });
|
|
130
|
+
const transcription = await transcribe({ channelWaveform, model });
|
|
131
|
+
const { captions } = toCaptions({ whisperWebGpuOutput: transcription });
|
|
132
|
+
|
|
133
|
+
// Write it to a file so the captions can be inlined into the Remotion code
|
|
134
|
+
await writeFile("captions123.json", JSON.stringify(captions, null, 2));
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
## Choosing a model
|
|
50
138
|
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
whisperCppVersion: "1.5.5",
|
|
55
|
-
inputPath: "/path/to/audio123.wav",
|
|
56
|
-
tokenLevelTimestamps: true,
|
|
57
|
-
});
|
|
139
|
+
`small.en` is the recommended default for English.
|
|
140
|
+
For other languages, use a multilingual model such as `small` and pass the `language` option to `transcribe()` - automatic language detection is not supported.
|
|
141
|
+
See [`getAvailableModels()`](https://www.remotion.dev/docs/whisper-webgpu/get-available-models.md) for all models.
|
|
58
142
|
|
|
59
|
-
|
|
60
|
-
const { captions } = toCaptions({
|
|
61
|
-
whisperCppOutput,
|
|
62
|
-
});
|
|
143
|
+
## Transcribing in the browser
|
|
63
144
|
|
|
64
|
-
|
|
65
|
-
|
|
145
|
+
In the browser, use [`resampleTo16Khz()`](https://www.remotion.dev/docs/whisper-webgpu/resample-to-16khz.md) to get the waveform from a `File` instead of using Mediabunny:
|
|
146
|
+
|
|
147
|
+
```ts
|
|
148
|
+
import {
|
|
149
|
+
downloadWhisperModel,
|
|
150
|
+
resampleTo16Khz,
|
|
151
|
+
toCaptions,
|
|
152
|
+
transcribe,
|
|
153
|
+
} from "@remotion/whisper-webgpu";
|
|
154
|
+
|
|
155
|
+
export const transcribeFile = async (file: File) => {
|
|
156
|
+
await downloadWhisperModel({ model: "small.en" });
|
|
157
|
+
const channelWaveform = await resampleTo16Khz({ file });
|
|
158
|
+
const transcription = await transcribe({
|
|
159
|
+
channelWaveform,
|
|
160
|
+
model: "small.en",
|
|
161
|
+
});
|
|
162
|
+
|
|
163
|
+
const { captions } = toCaptions({ whisperWebGpuOutput: transcription });
|
|
164
|
+
return captions;
|
|
165
|
+
};
|
|
66
166
|
```
|
|
67
167
|
|
|
68
|
-
Transcribe each
|
|
168
|
+
Transcribe each original source video or audio file individually.
|
|
169
|
+
Keep the resulting caption timestamps relative to the beginning of that source file, including portions currently trimmed out of the edit.
|
|
170
|
+
Do not add composition offsets or flatten multiple transcripts into composition-wide timestamps.
|
|
69
171
|
|
|
70
172
|
See [Displaying captions](display-captions.md) for how to display the captions in Remotion.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: remotion-create
|
|
3
3
|
description: Create a new Remotion video
|
|
4
|
-
version: 4.0.
|
|
4
|
+
version: 4.0.539
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
These are instructions for making a new Remotion project and composition.
|
|
@@ -32,21 +32,34 @@ npm i
|
|
|
32
32
|
If the current folder contains meaningful contents and no project already exists, scaffold into a new subfolder.
|
|
33
33
|
Replace `my-video` with a suitable project name.
|
|
34
34
|
|
|
35
|
-
|
|
36
35
|
```bash
|
|
37
36
|
npx create-video@latest --yes --blank --no-tailwind my-video
|
|
38
37
|
cd my-video
|
|
39
38
|
npm i
|
|
40
39
|
```
|
|
41
40
|
|
|
41
|
+
## Open the preview before building the video
|
|
42
|
+
|
|
43
|
+
As soon as the project can run, open [Remotion Studio](../remotion-studio/REFERENCE.md) in the browser before writing or changing the composition. For a new project, do this immediately after scaffolding and installing dependencies. For an existing project, do it before editing the video. Keep Studio running while you work so the user can see changes as they appear and steer the result.
|
|
44
|
+
|
|
42
45
|
## Designing a video
|
|
43
46
|
|
|
44
47
|
Keep the scaffold and add React Markup.
|
|
45
48
|
Follow [Remotion React Markup Best Practices](../remotion-markup/REFERENCE.md) and [Video Layout Rules](video-layout.md) for video-first layout and text sizing guidance.
|
|
46
49
|
|
|
50
|
+
Before writing the root that registers `<Composition>` or `<Still>` elements,
|
|
51
|
+
follow [Compositions and stills](../remotion-markup/compositions.md).
|
|
52
|
+
|
|
53
|
+
## Is this a timeline of clips?
|
|
54
|
+
|
|
55
|
+
If the video arranges multiple video or audio clips on a timeline, follow
|
|
56
|
+
[Video editing](../remotion-markup/video-editing.md) before writing the
|
|
57
|
+
timeline. Give every clip that should be edited independently its own authored
|
|
58
|
+
JSX node.
|
|
59
|
+
|
|
47
60
|
## Is this a multi-scene video?
|
|
48
61
|
|
|
49
|
-
If this is a video with multiple
|
|
62
|
+
If this is a video with multiple subsequent scenes, follow guidance at [Multi-scene videos](../remotion-markup/multi-scene-video.md).
|
|
50
63
|
|
|
51
64
|
## Interactivity Best Practices
|
|
52
65
|
|
|
@@ -56,29 +69,6 @@ By structuring the React Markup following [Remotion Interactivity Best Practices
|
|
|
56
69
|
|
|
57
70
|
If Tailwind is requested, see [tailwind.md](tailwind.md) for using TailwindCSS in Remotion.
|
|
58
71
|
|
|
59
|
-
## Open the preview
|
|
60
|
-
|
|
61
|
-
Start the preview server after building the composition:
|
|
62
|
-
|
|
63
|
-
```bash
|
|
64
|
-
npx remotion studio --no-open
|
|
65
|
-
```
|
|
66
|
-
|
|
67
|
-
This will start a long-running process and print the server URL for the preview.
|
|
68
|
-
If the server is already started, it will print the URL.
|
|
69
|
-
If an in-harness browser is available, open it there.
|
|
70
|
-
You can visit a specific composition by navigating to `/[composition-id]`, for example `http://localhost:3000/MapAnimation`.
|
|
71
|
-
|
|
72
|
-
## Render the video
|
|
73
|
-
|
|
74
|
-
Only render if the user explicitly asks for it.
|
|
75
|
-
|
|
76
|
-
```
|
|
77
|
-
npx remotion render
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
For more options, see [Rendering](../remotion-render/REFERENCE.md).
|
|
81
|
-
|
|
82
72
|
## Follow-up
|
|
83
73
|
|
|
84
74
|
The video creation process has finished.
|