takomi 2.5.1 → 2.5.4

package/assets/.agent/skills/remotion/rules/silence-detection.md

@@ -0,0 +1,71 @@
+---
+name: silence-detection
+description: Adaptive silence detection for video/audio files using FFmpeg loudnorm and silencedetect
+metadata:
+  tags: silence, detection, trimming, ffmpeg, loudnorm, audio
+---
+
+# Adaptive Silence Detection
+
+Detect silent segments in video or audio files.
+
+Requires FFmpeg — see [ffmpeg.md](./ffmpeg.md) for how to invoke it in Remotion projects.
+
+## Step 1: Measure loudness with `loudnorm`
+
+Use the `loudnorm` filter in JSON mode to get the EBU R128 integrated loudness and gating threshold for each file:
+
+```bash
+npx remotion ffmpeg -i public/video.mov -map 0:a -af loudnorm=print_format=json -f null /dev/null
+```
+
+As output you will get:
+- `input_i`: Integrated loudness (dB) — the overall perceived volume
+- `input_thresh`: EBU R128 gating threshold (dB) — the level below which audio is considered too quiet to count toward loudness measurement
+
+## Step 2: Detect silences using adaptive threshold
+
+Pass the `input_thresh` value from step 1 as the `noise` parameter to `silencedetect`:
+
+```bash
+npx remotion ffmpeg -i public/video.mov -map 0:a -af "silencedetect=noise=${THRESH}dB:d=0.5" -f null /dev/null
+```
+
+Parameters:
+- `noise`: The threshold below which audio is considered silent. Use `input_thresh` from step 1.
+- `d`: Minimum silence duration in seconds. `0.5` is a good default.
+
+## Interpreting the output
+
+The filter outputs pairs of `silence_start` and `silence_end` timestamps:
+
+```
+[silencedetect] silence_start: 0
+[silencedetect] silence_end: 2.241021 | silence_duration: 2.241021
+[silencedetect] silence_start: 38.77425
+[silencedetect] silence_end: 39.619604 | silence_duration: 0.845354
+```
+
+## Identifying leading and trailing silence
+
+- **Leading silence**: Consecutive silence segments starting at or near 0. If the first `silence_start` is > 0.5s, there is no leading silence.
+- **Trailing silence**: The last silence segment that extends to (or near) the end of the file. Compare the last `silence_end` with the file's total duration.
+
+When multiple silences are nearly contiguous at the start or end (gap < 0.2s), treat them as a single leading/trailing silence block.
+
+## Using with Remotion's `<Video>` component
+
+Apply the detected trim points using `trimBefore` and `trimAfter` (values are in frames):
+
+```tsx
+import { Video } from "@remotion/media";
+import { staticFile, useVideoConfig } from "remotion";
+
+const { fps } = useVideoConfig();
+
+<Video
+  src={staticFile("video.mov")}
+  trimBefore={Math.floor(leadingEnd * fps)}
+  trimAfter={Math.ceil(trailingStart * fps)}
+/>
+```

package/assets/.agent/skills/remotion/rules/subtitles.md

@@ -0,0 +1,36 @@
+---
+name: subtitles
+description: subtitles and caption rules
+metadata:
+  tags: subtitles, captions, remotion, json
+---
+
+All captions must be processed in JSON. The captions must use the `Caption` type which is the following:
+
+```ts
+import type { Caption } from "@remotion/captions";
+```
+
+This is the definition:
+
+```ts
+type Caption = {
+  text: string;
+  startMs: number;
+  endMs: number;
+  timestampMs: number | null;
+  confidence: number | null;
+};
+```
+
+## Generating captions
+
+To transcribe video and audio files to generate captions, load the [./transcribe-captions.md](./transcribe-captions.md) file for more instructions.
+
+## Displaying captions
+
+To display captions in your video, load the [./display-captions.md](./display-captions.md) file for more instructions.
+
+## Importing captions
+
+To import captions from a .srt file, load the [./import-srt-captions.md](./import-srt-captions.md) file for more instructions.

package/assets/.agent/skills/remotion/rules/tailwind.md

@@ -6,6 +6,6 @@ metadata:
 
 You can and should use TailwindCSS in Remotion, if TailwindCSS is installed in the project.
 
-Don't use `transition-*` or `animate-*` classes - always animate using the `useCurrentFrame()` hook.  
+Don't use `transition-*` or `animate-*` classes - always animate using the `useCurrentFrame()` hook.
 
-Tailwind must be installed and enabled first in a Remotion project - fetch  https://www.remotion.dev/docs/tailwind using WebFetch for instructions.
+Tailwind must be installed and enabled first in a Remotion project - fetch https://www.remotion.dev/docs/tailwind using WebFetch for instructions.

package/assets/.agent/skills/remotion/rules/timing.md

@@ -1,14 +1,16 @@
 ---
 name: timing
-description: Interpolation curves in Remotion - linear, easing, spring animations
+description: Interpolation and timing in Remotion—prefer interpolate with Bézier easing; springs as a specialized option
 metadata:
-  tags: spring, bounce, easing, interpolation
+  tags: easing, bezier, interpolation, spring, timing
 ---
 
+Drive motion with `interpolate()` over an explicit frame range. Prefer `interpolate()` over `spring()` unless the user explicitly asks for physics-based motion. To customize timing, use **`Easing.bezier`**. The four parameters are the same as CSS `cubic-bezier(x1, y1, x2, y2)`.
+
 A simple linear interpolation is done using the `interpolate` function.
 
 ```ts title="Going from 0 to 1 over 100 frames"
-import {interpolate} from 'remotion';
+import { interpolate } from "remotion";
 
 const opacity = interpolate(frame, [0, 100], [0, 1]);
 ```
@@ -18,162 +20,143 @@ Here is how they can be clamped:
 
 ```ts title="Going from 0 to 1 over 100 frames with extrapolation"
 const opacity = interpolate(frame, [0, 100], [0, 1], {
-  extrapolateRight: 'clamp',
-  extrapolateLeft: 'clamp',
+  extrapolateRight: "clamp",
+  extrapolateLeft: "clamp",
 });
 ```
 
-## Spring animations
-
-Spring animations have a more natural motion.  
-They go from 0 to 1 over time.
-
-```ts title="Spring animation from 0 to 1 over 100 frames"
-import {spring, useCurrentFrame, useVideoConfig} from 'remotion';
+## Studio-editable animation patterns
 
-const frame = useCurrentFrame();
-const {fps} = useVideoConfig();
+When an animation should be editable in Remotion Studio, keep the `interpolate()` call directly in the `style` prop and prefer individual CSS transform properties:
 
-const scale = spring({
-  frame,
-  fps,
-});
+```tsx
+style={{
+  scale: interpolate(frame, [0, 100], [0, 1]),
+  translate: interpolate(frame, [0, 100], ["0px 0px", "100px 100px"]),
+  rotate: interpolate(frame, [0, 100], ["20deg", "90deg"]),
+}}
 ```
 
-### Physical properties
-
-The default configuration is: `mass: 1, damping: 10, stiffness: 100`.  
-This leads to the animation having a bit of bounce before it settles.
+Avoid extracting the value and composing a `transform` string:
 
-The config can be overwritten like this:
+```tsx
+const scale = interpolate(frame, [0, 100], [0, 1]);
 
-```ts
-const scale = spring({
-  frame,
-  fps,
-  config: {damping: 200},
-});
+style={{
+  transform: `scale(${scale})`,
+}}
 ```
 
-The recommended configuration for a natural motion without a bounce is: `{ damping: 200 }`.
+Use `transform` strings only when individual CSS transform properties do not cover the effect, such as `skew()`, `perspective()`, or order-sensitive multi-transform chains.
 
-Here are some common configurations:
+## Bézier easing
 
-```tsx
-const smooth = {damping: 200}; // Smooth, no bounce (subtle reveals)
-const snappy = {damping: 20, stiffness: 200}; // Snappy, minimal bounce (UI elements)
-const bouncy = {damping: 8}; // Bouncy entrance (playful animations)
-const heavy = {damping: 15, stiffness: 80, mass: 2}; // Heavy, slow, small bounce
-```
+Use `Easing.bezier(x1, y1, x2, y2)` inside the `interpolate` options object. The curve is identical in spirit to CSS animations and transitions, which helps when you are stealing timing from the web or from a designer’s spec.
 
-### Delay
-
-The animation starts immediately by default.  
-Use the `delay` parameter to delay the animation by a number of frames.
+```ts
+import { interpolate, Easing } from "remotion";
 
-```tsx
-const entrance = spring({
-  frame: frame - ENTRANCE_DELAY,
-  fps,
-  delay: 20,
+const opacity = interpolate(frame, [0, 60], [0, 1], {
+  easing: Easing.bezier(0.16, 1, 0.3, 1),
+  extrapolateLeft: "clamp",
+  extrapolateRight: "clamp",
 });
 ```
 
-### Duration
+### Examples (copy-paste curves)
 
-A `spring()` has a natural duration based on the physical properties.  
-To stretch the animation to a specific duration, use the `durationInFrames` parameter.
+**1. Crisp UI entrance (strong ease-out, no overshoot)** — slows nicely into the rest value; similar to many system “deceleration” curves.
 
 ```tsx
-const spring = spring({
-  frame,
-  fps,
-  durationInFrames: 40,
+const enter = interpolate(frame, [0, 45], [0, 1], {
+  easing: Easing.bezier(0.16, 1, 0.3, 1),
+  extrapolateLeft: "clamp",
+  extrapolateRight: "clamp",
 });
 ```
 
-### Combining spring() with interpolate()
-
-Map spring output (0-1) to custom ranges:
+**2. Editorial / slow fade (balanced ease-in-out)** — symmetric acceleration and deceleration over a hold-friendly move.
 
 ```tsx
-const springProgress = spring({
-  frame,
-  fps,
+const progress = interpolate(frame, [0, 90], [0, 1], {
+  easing: Easing.bezier(0.45, 0, 0.55, 1),
+  extrapolateLeft: "clamp",
+  extrapolateRight: "clamp",
 });
-
-// Map to rotation
-const rotation = interpolate(springProgress, [0, 1], [0, 360]);
-
-<div style={{rotate: rotation + 'deg'}} />;
 ```
 
-### Adding springs
-
-Springs return just numbers, so math can be performed:
+**3. Playful overshoot (control point y > 1)** — a little past the target then settles; use sparingly for emphasis.
 
 ```tsx
-const frame = useCurrentFrame();
-const {fps, durationInFrames} = useVideoConfig();
-
-const inAnimation = spring({
-  frame,
-  fps,
-});
-const outAnimation = spring({
-  frame,
-  fps,
-  durationInFrames: 1 * fps,
-  delay: durationInFrames - 1 * fps,
+const pop = interpolate(frame, [0, 30], [0, 1], {
+  easing: Easing.bezier(0.34, 1.56, 0.64, 1),
+  extrapolateLeft: "clamp",
+  extrapolateRight: "clamp",
 });
-
-const scale = inAnimation - outAnimation;
 ```
 
-## Easing
+## Preset easings (`Easing.in` / `Easing.out` / named curves)
 
-Easing can be added to the `interpolate` function:
+Easing can be added to the `interpolate` function without a custom cubic:
 
 ```ts
-import {interpolate, Easing} from 'remotion';
+import { interpolate, Easing } from "remotion";
 
 const value1 = interpolate(frame, [0, 100], [0, 1], {
-  easing: Easing.inOut(Easing.quad),
-  extrapolateLeft: 'clamp',
-  extrapolateRight: 'clamp',
+  easing: Easing.inOut(Easing.cubic),
+  extrapolateLeft: "clamp",
+  extrapolateRight: "clamp",
 });
 ```
 
 The default easing is `Easing.linear`.  
-There are various other convexities:
+Convexities:
 
-- `Easing.in` for starting slow and accelerating
-- `Easing.out` for starting fast and slowing down
+- `Easing.in` — starting slow and accelerating
+- `Easing.out` — starting fast and slowing down
 - `Easing.inOut`
 
-and curves (sorted from most linear to most curved):
+Named curves (from most linear to most curved):
 
 - `Easing.quad`
+- `Easing.cubic` (good default when you do not need a custom cubic)
 - `Easing.sin`
 - `Easing.exp`
 - `Easing.circle`
 
-Convexities and curves need be combined for an easing function:
+### Easing direction for enter/exit animations
 
-```ts
-const value1 = interpolate(frame, [0, 100], [0, 1], {
-  easing: Easing.inOut(Easing.quad),
-  extrapolateLeft: 'clamp',
-  extrapolateRight: 'clamp',
-});
-```
+Use `Easing.out` for enter animations (starts fast, decelerates into place) and `Easing.in` for exit animations (starts slow, accelerates away). This feels natural because elements arrive with momentum and leave with gravity. When you need a specific curve from design, prefer a single `Easing.bezier(...)` instead of stacking presets.
 
-Cubic bezier curves are also supported:
+## Composing interpolations
 
-```ts
-const value1 = interpolate(frame, [0, 100], [0, 1], {
-  easing: Easing.bezier(0.8, 0.22, 0.96, 0.65),
-  extrapolateLeft: 'clamp',
-  extrapolateRight: 'clamp',
-});
+When multiple properties share the same timing and do not need Studio keyframe editing (e.g. a slide-in panel and a video shift), avoid duplicating the full interpolation for each property. Instead, create a single normalized progress value (0 to 1) and derive each property from it:
+
+```tsx
+const slideIn = interpolate(
+  frame,
+  [slideInStart, slideInStart + slideInDuration],
+  [0, 1],
+  {
+    easing: Easing.bezier(0.22, 1, 0.36, 1),
+    extrapolateLeft: "clamp",
+    extrapolateRight: "clamp",
+  },
+);
+const slideOut = interpolate(
+  frame,
+  [slideOutStart, slideOutStart + slideOutDuration],
+  [0, 1],
+  { easing: Easing.in(Easing.cubic), extrapolateLeft: "clamp", extrapolateRight: "clamp" },
+);
+const progress = slideIn - slideOut;
+
+// Derive multiple properties from the same progress
+const overlayX = interpolate(progress, [0, 1], [100, 0]);
+const videoX = interpolate(progress, [0, 1], [0, -20]);
+const opacity = interpolate(progress, [0, 1], [0, 1]);
 ```
+
+The key idea: separate **timing** (when and how fast) from **mapping** (what values to animate between).
+
+If the values should be visually keyframed in Studio, prefer inline `interpolate()` calls in the relevant style props, even if it duplicates the timing.

package/assets/.agent/skills/remotion/rules/transcribe-captions.md

@@ -7,13 +7,64 @@ metadata:
 
 # Transcribing audio
 
-Remotion provides several built-in options for transcribing audio to generate captions:
+To transcribe audio to generate captions in Remotion, you can use the [`transcribe()`](https://www.remotion.dev/docs/install-whisper-cpp/transcribe) function from the [`@remotion/install-whisper-cpp`](https://www.remotion.dev/docs/install-whisper-cpp) package.
 
-- `@remotion/install-whisper-cpp` - Transcribe locally on a server using Whisper.cpp. Fast and free, but requires server infrastructure.
-  https://remotion.dev/docs/install-whisper-cpp
+## Prerequisites
 
-- `@remotion/whisper-web` - Transcribe in the browser using WebAssembly. No server needed and free, but slower due to WASM overhead.
-  https://remotion.dev/docs/whisper-web
+First, the @remotion/install-whisper-cpp package needs to be installed.
+If it is not installed, use the following command:
 
-- `@remotion/openai-whisper` - Use OpenAI Whisper API for cloud-based transcription. Fast and no server needed, but requires payment.
-  https://remotion.dev/docs/openai-whisper/openai-whisper-api-to-captions
+```bash
+npx remotion add @remotion/install-whisper-cpp
+```
+
+## Transcribing
+
+Make a Node.js script to download Whisper.cpp and a model, and transcribe the audio.
+
+```ts
+import path from "path";
+import {
+  downloadWhisperModel,
+  installWhisperCpp,
+  transcribe,
+  toCaptions,
+} from "@remotion/install-whisper-cpp";
+import fs from "fs";
+
+const to = path.join(process.cwd(), "whisper.cpp");
+
+await installWhisperCpp({
+  to,
+  version: "1.5.5",
+});
+
+await downloadWhisperModel({
+  model: "medium.en",
+  folder: to,
+});
+
+// Convert the audio to a 16KHz wav file first if needed:
+// import {execSync} from 'child_process';
+// execSync('ffmpeg -i /path/to/audio.mp4 -ar 16000 /path/to/audio.wav -y');
+
+const whisperCppOutput = await transcribe({
+  model: "medium.en",
+  whisperPath: to,
+  whisperCppVersion: "1.5.5",
+  inputPath: "/path/to/audio123.wav",
+  tokenLevelTimestamps: true,
+});
+
+// Optional: Apply our recommended postprocessing
+const { captions } = toCaptions({
+  whisperCppOutput,
+});
+
+// Write it to the public/ folder so it can be fetched from Remotion
+fs.writeFileSync("captions123.json", JSON.stringify(captions, null, 2));
+```
+
+Transcribe each clip individually and create multiple JSON files.
+
+See [Displaying captions](display-captions.md) for how to display the captions in Remotion.