takomi 2.1.41 → 2.1.42

package/assets/.agent/skills/remotion-to-hyperframes/references/escape-hatch.md

@@ -0,0 +1,115 @@
+# When to bow out: the runtime interop pattern
+
+Some Remotion compositions can't be translated cleanly. The skill should
+recognize them upfront and recommend the **runtime interop pattern** from
+[PR #214](https://github.com/heygen-com/hyperframes/pull/214) instead of
+producing broken HTML.
+
+## When to recommend interop
+
+Run `scripts/lint_source.py` first. If it returns any blocker, recommend
+interop. The blockers are:
+
+| Rule                        | What it catches                                                |
+| --------------------------- | -------------------------------------------------------------- |
+| `r2hf/use-state`            | useState driving animation                                     |
+| `r2hf/use-reducer`          | useReducer driving animation                                   |
+| `r2hf/use-effect-deps`      | useEffect/useLayoutEffect with non-empty deps (side effects)   |
+| `r2hf/async-metadata`       | calculateMetadata returns a Promise                            |
+| `r2hf/third-party-react-ui` | Imports from MUI, Chakra, Mantine, antd, shadcn, Radix, NextUI |
+
+Each of these breaks the seek-driven, deterministic-frame model that HF
+relies on. Translating them produces silently-wrong output.
+
+## What the interop pattern actually does
+
+Per [PR #214](https://github.com/heygen-com/hyperframes/pull/214), the
+runtime adapter:
+
+1. Bundles the user's Remotion code with React + `@remotion/player` via esbuild.
+2. Mounts a Remotion `<Player>` inside an HF composition's HTML.
+3. Pauses the player on mount.
+4. Registers the player on `window.__hfRemotion` with `seekTo(frame)`,
+   `pause()`, `durationInFrames`, `fps`.
+5. HF's render loop seeks the player frame-by-frame via `seekTo(frame)`.
+
+Result: Remotion's React tree renders at HF's deterministic frame ticks.
+Custom hooks, useState, useEffect, MUI components — all work because
+Remotion's React reconciler is doing the rendering.
+
+## The recommendation message
+
+When the skill detects a blocker, output something like:
+
+> The Remotion source uses `useState` (and others), which can't be
+> translated to HF's seek-driven HTML model. The recommended path is the
+> **runtime interop pattern**: bundle your Remotion code with `@remotion/player`
+> and let HF drive it frame-by-frame.
+>
+> See https://github.com/heygen-com/hyperframes/pull/214 for the full
+> implementation. Quick summary:
+>
+> 1. Bundle `entry.tsx` with esbuild: `npx esbuild entry.tsx --bundle --outfile=dist/bundle.js --format=iife --jsx=automatic`
+> 2. Mount the Player and register on `window.__hfRemotion`:
+>
+>    ```tsx
+>    const playerRef = useRef<PlayerRef>(null);
+>    useEffect(() => {
+>      playerRef.current?.pause();
+>      window.__hfRemotion = window.__hfRemotion || [];
+>      window.__hfRemotion.push({
+>        seekTo: (f) => playerRef.current?.seekTo(f),
+>        pause: () => playerRef.current?.pause(),
+>        durationInFrames,
+>        fps,
+>      });
+>    }, []);
+>    ```
+>
+> 3. Reference the bundle from your HF `index.html` and render normally:
+>    `<script src="dist/bundle.js"></script>`
+
+## The lint output already includes recommendations
+
+`lint_source.py` emits a `recommendation` field per finding. Surface those
+verbatim — they're tuned per blocker rule:
+
+```json
+{
+  "rule": "r2hf/use-state",
+  "message": "useState detected — Remotion compositions that drive animation via React state are not deterministic frame-capture targets in HyperFrames",
+  "recommendation": "Use the runtime interop pattern from PR #214 instead of attempting a translation"
+}
+```
+
+## When NOT to bow out: warnings only
+
+Some patterns produce warnings, not blockers — translate after dropping
+the wrappers:
+
+| Rule                      | Action                                                              |
+| ------------------------- | ------------------------------------------------------------------- |
+| `r2hf/lambda-import`      | drop the `@remotion/lambda` config; HF runs single-machine, log gap |
+| `r2hf/delay-render`       | drop the call; HF handles asset readiness                           |
+| `r2hf/use-callback`       | drop the wrapper, inline the function                               |
+| `r2hf/use-memo`           | drop the wrapper, compute inline                                    |
+| `r2hf/custom-hook` (pure) | inline the hook body if it's a derivation of `useCurrentFrame`      |
+| `r2hf/static-file`        | replace `staticFile("x")` with `"assets/x"`                         |
+| `r2hf/interpolate-colors` | translate to GSAP color tween (see [timing.md](timing.md))          |
+
+These are documented in T4 cases 05–07.
+
+`r2hf/lambda-import` is a warning — not a blocker — because Lambda
+configuration is orthogonal to the rendered composition. Translating an
+otherwise-clean Remotion comp shouldn't fail just because the author also
+configured AWS Lambda for distributed rendering. The skill drops the
+`@remotion/lambda` imports and `renderMediaOnLambda(...)` calls in step 3
+(Generate) and writes a `TRANSLATION_NOTES.md` entry so the user knows to
+set up HF rendering separately.
+
+## When the source has BOTH blockers AND warnings
+
+Bow out. The presence of a single blocker means the skill shouldn't
+attempt translation — even if the rest of the composition is clean.
+The user should use interop for the whole thing OR refactor the
+blocker patterns out of their Remotion source first.

package/assets/.agent/skills/remotion-to-hyperframes/references/eval.md

@@ -0,0 +1,140 @@
+# Eval: how to validate a translation end-to-end
+
+Every translation should be measured. The skill ships three scripts and
+a tiered test corpus that, together, gate translation quality.
+
+## The three scripts
+
+| Script                   | Input                       | Output                                                              |
+| ------------------------ | --------------------------- | ------------------------------------------------------------------- |
+| `scripts/lint_source.py` | Remotion source dir or file | JSON findings + exit code (0 clean, 1 has blockers)                 |
+| `scripts/render_diff.sh` | two MP4 paths               | per-frame SSIM + JSON summary (`mean`, `min`, `p05`, `p95`, `pass`) |
+| `scripts/frame_strip.sh` | two MP4 paths               | side-by-side comparison strip PNG for visual debugging              |
+
+Run them in this order: **lint → render → diff → (if fail) strip**.
+
+## Per-fixture flow
+
+```bash
+# 1. Lint the source — blockers mean stop
+python3 ../../scripts/lint_source.py ./remotion-src/src/
+
+# 2. Generate any binary assets (T2+T3 only)
+[ -f setup.sh ] && ./setup.sh
+
+# 3. Render Remotion baseline
+cd remotion-src && npm install && npm run render
+# -> remotion-src/out/baseline.mp4
+
+# 4. Render HF translation
+cd .. && node ../../../packages/cli/dist/cli.js render hf-src/ --output hf.mp4
+# -> hf.mp4
+
+# 5. SSIM diff
+../../scripts/render_diff.sh ./remotion-src/out/baseline.mp4 ./hf.mp4 ./diff
+# -> diff/summary.json
+
+# 6. If diff fails, generate frame strip for visual inspection
+../../scripts/frame_strip.sh ./remotion-src/out/baseline.mp4 ./hf.mp4 ./strip 8
+# -> strip/strip.png
+```
+
+## Reading `diff/summary.json`
+
+```json
+{
+  "frame_count": 90,
+  "mean": 0.974,
+  "min": 0.972,
+  "max": 0.999,
+  "p05": 0.972,
+  "p95": 0.983,
+  "threshold": 0.95,
+  "pass": true
+}
+```
+
+| Field         | What it tells you                                                           |
+| ------------- | --------------------------------------------------------------------------- |
+| `mean`        | average SSIM across all frames; the headline number                         |
+| `min`         | worst frame; below threshold means at least one frame is structurally wrong |
+| `p05` / `p95` | 5th / 95th percentile — most frames sit between these                       |
+| `threshold`   | from `R2HF_SSIM_THRESHOLD` env var (default 0.85)                           |
+| `pass`        | whether `mean >= threshold`                                                 |
+
+## Validated tier thresholds
+
+Calibrated against actual Remotion + HF renders:
+
+| Tier | Composition shape                           | Mean  | Threshold | Margin |
+| ---- | ------------------------------------------- | ----- | --------- | ------ |
+| T1   | single-element fade-in                      | 0.974 | 0.95      | +0.022 |
+| T2   | multi-scene + spring + audio + image        | 0.985 | 0.95      | +0.016 |
+| T3   | data-driven, custom subcomponents, count-up | 0.953 | 0.90      | +0.038 |
+
+Each fixture's `expected.json` carries:
+
+- `ssim_threshold` — the gate for `pass`
+- `validation` — the actual measured numbers from the calibration run
+- `translation_notes` — what's lossy and why
+
+## Critical: encoder config
+
+Both Remotion and HF must output the same pixel format for SSIM to be
+meaningful. Remotion's default JPEG output writes `yuvj420p` (full-range);
+HF outputs `yuv420p` (limited-range). The mismatch costs ~0.05 SSIM.
+
+Every fixture's `remotion.config.ts` sets:
+
+```ts
+Config.setVideoImageFormat("png");
+Config.setColorSpace("bt709");
+```
+
+If the user's source doesn't have these, add them in the translation
+step — otherwise the diff measures encoder differences, not translation
+fidelity.
+
+## What the noise floor looks like
+
+The dominant non-translation noise is **system font fallback divergence**.
+Remotion's bundled Chromium and HF's `chrome-headless-shell` interpret
+`font-weight: 800` differently when there's no real font installed:
+
+- Remotion HELLO at 160px: medium-weight stroke
+- HF HELLO at 160px: heavy-weight stroke
+
+This costs ~0.025 mean SSIM. Visible in T1's frame strip.
+[fonts.md](fonts.md) covers how to mitigate (use Inter, load explicit
+Google Fonts).
+
+## Threshold rule of thumb
+
+Set the threshold ~0.02 below measured `p05`:
+
+- Real translation regressions drop mean by 0.05+ — caught.
+- Encoder/font drift between CI runs is bounded at ~0.01 — not caught.
+
+If a calibration run's measured mean is far above your initial threshold
+guess, _don't_ tighten the threshold to fit. Leave headroom — fixtures
+re-rendered on different hardware will drift.
+
+## When the diff fails
+
+1. **Look at `frame_strip.sh` output first.** A side-by-side strip at 6–10
+   evenly-spaced timestamps shows whether the failure is structural
+   (wrong scene durations, missing element) or cosmetic (different font
+   weight, slight timing skew).
+2. **Check `diff/ssim.log`.** Per-frame SSIM tells you _which_ frames
+   failed. Cluster of bad frames in the middle of a scene = animation
+   problem; bad frames at scene boundaries = sequencing problem.
+3. **Re-read the relevant reference.** [timing.md](timing.md) for
+   spring/easing issues, [sequencing.md](sequencing.md) for scene
+   boundary issues, [media.md](media.md) for asset loading issues.
+
+## CI integration
+
+The fixtures are not yet wired into CI (`packages/producer/tests/` runs
+inside Docker; the skill corpus needs the same). PR 7 of the stack adds
+the orchestrator that runs all four tiers and emits an aggregated pass
+report. For now, evaluate by hand per fixture.

package/assets/.agent/skills/remotion-to-hyperframes/references/fonts.md

@@ -0,0 +1,112 @@
+# Font translation
+
+Fonts are the dominant non-translation noise floor. Same `font-weight: 800`
+renders perceptibly bolder on HF's `chrome-headless-shell` than on
+Remotion's bundled Chromium when there's no real font installed. Validation
+showed this costs ~0.025 mean SSIM at the noise floor.
+
+## Pattern: `@remotion/google-fonts/<Family>`
+
+```tsx
+import { loadFont } from "@remotion/google-fonts/Inter";
+loadFont("normal", { weights: ["400", "800"] });
+```
+
+Translate to a `<link>` tag in `<head>`:
+
+```html
+<head>
+  <link rel="preconnect" href="https://fonts.googleapis.com" />
+  <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
+  <link
+    href="https://fonts.googleapis.com/css2?family=Inter:wght@400;800&display=swap"
+    rel="stylesheet"
+  />
+  <style>
+    body {
+      font-family: Inter, sans-serif;
+    }
+  </style>
+</head>
+```
+
+Pull the family name and weights from the import path and `loadFont`
+arguments. HF's compiler inlines the Google Fonts CSS at render time, so
+you don't pay a network round-trip per render.
+
+## Pattern: local fonts via `@font-face`
+
+```tsx
+import { Font } from "remotion";
+
+Font.loadFont("/MyFont.woff2", "MyFont");
+```
+
+Translate to a `@font-face` rule:
+
+```html
+<style>
+  @font-face {
+    font-family: "MyFont";
+    src: url("assets/MyFont.woff2") format("woff2");
+    font-weight: 400;
+    font-style: normal;
+  }
+</style>
+```
+
+Copy the font file into `hf-src/assets/` next to the HTML.
+
+## Pattern: system font fallback (no font load)
+
+```tsx
+<div style={{ fontFamily: "Helvetica, Arial, sans-serif" }}>...</div>
+```
+
+Same string in HF — but be aware: on Linux without a real Helvetica
+installed (typical CI environment), Remotion and HF fall back to
+_different_ sans-serif system fonts because they bundle different
+Chromium versions. This is the noise floor: ~0.025 mean SSIM cost,
+visible as different stroke widths at large font weights (800+).
+
+If matching the Remotion render exactly matters for a specific
+fixture, load the same font explicitly — don't rely on system
+fallback.
+
+## When in doubt: use Inter
+
+Inter renders identically across Chromium versions and is free.
+Translate any "system sans-serif" Remotion comp to Inter when you
+need to minimize font drift in the validation harness.
+
+## Font loading and `delayRender`
+
+Remotion uses `delayRender()` to defer the first frame until fonts
+load. HF's compiler inlines Google Fonts at compile time and waits
+on `@font-face` readiness via the Frame Adapter pattern — the
+`delayRender` call drops in translation. See [media.md](media.md).
+
+## Multi-weight loading
+
+When Remotion loads multiple weights:
+
+```tsx
+loadFont("normal", { weights: ["400", "500", "700", "800"] });
+```
+
+Inline all weights in the Google Fonts URL:
+
+```
+?family=Inter:wght@400;500;700;800&display=swap
+```
+
+Translation rule: enumerate every distinct `font-weight` value
+that appears in the composition's CSS (`font-weight: 800` →
+weight 800 must be loaded). If the Remotion source loads weights
+that aren't actually used, drop them.
+
+## Font subsetting
+
+Remotion's `loadFont` doesn't subset; HF's compiler doesn't either
+(yet). Don't try to optimize this in translation — it's lossless to
+keep the same weight set as the Remotion source.

package/assets/.agent/skills/remotion-to-hyperframes/references/limitations.md

@@ -0,0 +1,136 @@
+# Translation limitations
+
+What the skill explicitly cannot translate, separated from the
+blocker-list (which is enforced by `lint_source.py`). These are
+_known_ gaps — surface them to the user as translation notes
+when translating the surrounding composition.
+
+## React patterns the skill refuses
+
+See [escape-hatch.md](escape-hatch.md). Any of these triggers
+a bow-out:
+
+- `useState`, `useReducer` driving animation
+- `useEffect` / `useLayoutEffect` with non-empty deps (side effects)
+- async `calculateMetadata`
+- Third-party React UI libraries (MUI, Chakra, Mantine, antd, shadcn, Radix, NextUI)
+
+`@remotion/lambda` is no longer in this list — it's a warning, not a
+blocker, because Lambda config is orthogonal to composition rendering.
+The skill drops the imports and `renderMediaOnLambda(...)` calls and
+writes a `TRANSLATION_NOTES.md` entry. See
+[escape-hatch.md](escape-hatch.md).
+
+## Patterns that work with caveats
+
+### Volume ramps on `<Audio>`
+
+Remotion accepts a function for `volume`:
+
+```tsx
+<Audio src={...} volume={(f) => interpolate(f, [0, 30], [0, 1])} />
+```
+
+HF supports static `data-volume` only. Translation: bake the ramp into
+the audio file at translation time using `ffmpeg afade`, OR drop the
+ramp with a note. The dropped-ramp path produces audibly different
+output but visually-identical video, so SSIM passes — just flag it.
+
+### `<Loop>` with stateful children
+
+```tsx
+<Loop durationInFrames={30}>
+  <CounterThatIncrementsViaUseRef />
+</Loop>
+```
+
+Loop with `repeat: -1` works for _visual_ repetition. If the looped
+child has cross-iteration state (a counter, a randomness seed), HF
+won't reproduce it identically per iteration. Bow out unless the
+child is fully deterministic per-iteration.
+
+### Remotion's `<Img>` with crossOrigin
+
+```tsx
+<Img src="https://other-domain.com/x.png" crossOrigin="anonymous" />
+```
+
+HF's renderer doesn't enforce CORS the same way Remotion does. Most
+public images work; private images served with auth headers won't.
+If the source uses `crossOrigin="use-credentials"`, the asset needs
+to be downloaded and inlined at translation time.
+
+### Custom `presentation` in `<TransitionSeries>`
+
+```tsx
+const customPresentation: PresentationComponent = ({ children, presentationProgress }) => {
+  return <div style={{ filter: `blur(${(1 - presentationProgress) * 20}px)` }}>{children}</div>;
+};
+```
+
+Pure presentations (transform/filter/opacity computed from progress)
+translate to GSAP tweens cleanly. Presentations that read
+`useCurrentFrame()` internally or have stateful children don't —
+bow out.
+
+### Code-split components (`React.lazy`)
+
+```tsx
+const HeavyChart = React.lazy(() => import("./HeavyChart"));
+```
+
+`React.lazy` is async and doesn't fit the deterministic-render model.
+Translate to a regular import; the resulting HF composition will
+just include all the code upfront.
+
+## Patterns that always work
+
+- `<AbsoluteFill>` and `<Sequence>` (any nesting)
+- `useCurrentFrame()` derivations: `interpolate`, `spring`, `Easing`,
+  `interpolateColors`, manual math
+- `<Audio>`, `<Video>`, `<Img>`, `<IFrame>` with simple props
+- `staticFile()` references
+- Custom React subcomponents that are pure functions of props
+- Custom hooks that are pure derivations of `useCurrentFrame`
+- `@remotion/lottie` (translates to HF's Lottie adapter)
+- `@remotion/google-fonts/<Family>` (translates to `<link>` or `@font-face`)
+- Sync `calculateMetadata` (resolved at translation time)
+- `<TransitionSeries>` with built-in presentations (`fade`, `slide`,
+  `wipe`, `clockWipe`, `flip`, `iris`)
+
+## What the skill never tries to translate
+
+These are out-of-scope by design:
+
+- **HDR rendering** — HF supports HDR but Remotion doesn't, so there's
+  nothing to translate from.
+- **Variable frame rate** — both tools assume constant fps.
+- **Multi-composition `<Composition>` lists** — translate one at a
+  time. The skill prompts the user to choose which composition.
+- **Remotion Studio props panel** — visual prop editing in HF Studio
+  needs different infrastructure; out of scope.
+
+## Reporting gaps to the user
+
+When translation produces _something_ but the something has gaps, write
+a `TRANSLATION_NOTES.md` next to the output:
+
+```markdown
+# Translation notes
+
+The following Remotion patterns were translated with caveats:
+
+- `<Audio volume={(f) => ...}>` (line 15): volume ramp dropped — added
+  static `data-volume="0.5"`. To preserve the ramp, run
+  `ffmpeg -i music.wav -af "afade=t=in:st=0:d=1" music.faded.wav` and
+  swap the source file.
+- `<HeavyChart>` (line 30): translated as inline HTML. The original
+  React.lazy boundary was dropped — bundle size unchanged because HF
+  serves a single HTML file.
+
+If any of these caveats matter, consider the runtime interop pattern
+instead.
+```
+
+This file is also generated by the skill alongside the HF output, not
+held in the corpus.

package/assets/.agent/skills/remotion-to-hyperframes/references/lottie.md

@@ -0,0 +1,121 @@
+# Lottie translation: @remotion/lottie → HF lottie adapter
+
+Lottie animations are a clean translation case — HF has a built-in
+[Lottie adapter](https://github.com/heygen-com/hyperframes/blob/main/packages/core/src/runtime/adapters/lottie.ts)
+that supports both `lottie-web` and `@lottiefiles/dotlottie-web`. The
+adapter auto-discovers animations registered on `window.__hfLottie`
+and seeks them per-frame via `goToAndStop`.
+
+## Pattern
+
+```tsx
+import { Lottie } from "@remotion/lottie";
+import animationData from "./hello.json";
+
+export const MyComp = () => (
+  <AbsoluteFill>
+    <Lottie animationData={animationData} loop={false} />
+  </AbsoluteFill>
+);
+```
+
+Translates to:
+
+```html
+<div id="stage" ...>
+  <div id="lottie-anim" style="width:100%;height:100%"></div>
+  <script src="https://cdnjs.cloudflare.com/ajax/libs/bodymovin/5.12.2/lottie.min.js"></script>
+  <script>
+    const anim = lottie.loadAnimation({
+      container: document.getElementById("lottie-anim"),
+      renderer: "svg",
+      loop: false,
+      autoplay: false,
+      path: "assets/hello.json",
+    });
+    window.__hfLottie = window.__hfLottie || [];
+    window.__hfLottie.push(anim);
+  </script>
+</div>
+```
+
+Key differences from a typical Lottie embed:
+
+- `autoplay: false` — HF drives playback by seeking
+- `loop: false` typically (unless Remotion's `loop={true}`)
+- `window.__hfLottie.push(anim)` is what hooks the animation into HF's
+  per-frame seek
+
+## Asset handling
+
+Remotion bundles the animation JSON via webpack import. HF needs the JSON
+on disk under `assets/` and references it via path:
+
+1. Copy `hello.json` from the Remotion project into `hf-src/assets/`.
+2. Reference as `path: "assets/hello.json"` in `loadAnimation`.
+
+For dotlottie (binary) format, swap in `@lottiefiles/dotlottie-web`:
+
+```html
+<script src="https://unpkg.com/@lottiefiles/dotlottie-web"></script>
+<canvas id="anim" style="width:100%;height:100%"></canvas>
+<script>
+  const player = new DotLottie({
+    canvas: document.getElementById("anim"),
+    src: "assets/hello.lottie",
+    autoplay: false,
+  });
+  window.__hfLottie = window.__hfLottie || [];
+  window.__hfLottie.push(player);
+</script>
+```
+
+The HF adapter handles both player APIs (it duck-types `goToAndStop`
+vs `setCurrentRawFrameValue` / `seek`).
+
+## Multiple Lottie animations
+
+Multiple `<Lottie>` instances in one composition work — push each one
+onto `window.__hfLottie` and the adapter will seek all of them in sync:
+
+```js
+window.__hfLottie.push(anim1);
+window.__hfLottie.push(anim2);
+window.__hfLottie.push(anim3);
+```
+
+## Lottie source isn't actually translation-blocking
+
+Lottie animations encode their own deterministic timeline. They're the
+_easiest_ part of a Remotion composition to translate because the
+animation logic is already self-contained — neither Remotion nor HF
+"animate" them, both just seek them. Translation cost is near-zero.
+
+## After Effects → Lottie limitations
+
+Lottie supports a subset of After Effects features. Expressions, most
+Effects (drop shadow, color overlay), all blend modes beyond Normal/Add/
+Multiply, luma mattes, and most 3D parameters are not supported. If the
+Remotion composition uses a Lottie file that depends on these, the
+animation will break in BOTH Remotion and HF — this isn't a translation
+problem, it's a Lottie limitation. See
+[airbnb/lottie/after-effects.md](https://github.com/airbnb/lottie/blob/master/after-effects.md)
+for the full supported feature list.
+
+## Loop behavior
+
+Remotion's `loop={true}` plays the animation continuously. Translate to
+the player option only after checking the generated frames. The HF
+adapter seeks absolute composition time; it does not add modulo looping
+or playback-rate scaling on top of the player. For exact repeating
+cycles or non-default playback rates, bake the timing into the Lottie
+asset or author an explicit timeline around the Lottie layer and verify
+the rendered output.
+
+## Performance note
+
+Per the [Lottie adapter](https://github.com/heygen-com/hyperframes/blob/main/packages/core/src/runtime/adapters/lottie.ts)
+docs: lottie-web's `goToAndStop(time, isFrame=false)` takes time in ms;
+the adapter passes `time * 1000` for precision. This is more accurate
+than passing frame numbers (especially for animations whose internal
+fps doesn't match the HF render fps).