dsh-hyperframes 0.2.0 → 0.3.0

package/skills/remotion-to-hyperframes/references/media.md

@@ -0,0 +1,149 @@
+# Media translation: Audio, Video, Img, IFrame, staticFile
+
+## Asset paths
+
+Remotion's `staticFile("x.png")` resolves to the project's `public/` directory.
+HF uses relative paths from the composition's `index.html`, conventionally
+`assets/`:
+
+```tsx
+<Img src={staticFile("logo.png")} />
+```
+
+```html
+<img src="assets/logo.png" />
+```
+
+When translating, copy the asset from `remotion-src/public/x` to
+`hf-src/assets/x`. Multiple files can be batched with a setup script;
+see T2's `setup.sh` for an example pattern.
+
+## `<Audio>`
+
+```tsx
+<Audio src={staticFile("music.wav")} volume={0.5} />
+```
+
+```html
+<audio
+  data-start="0"
+  data-duration="6"
+  data-track-index="2"
+  data-volume="0.5"
+  src="assets/music.wav"
+></audio>
+```
+
+`data-start` and `data-duration` are required — the runtime needs them to
+schedule the audio. Default to the composition's full duration if Remotion
+didn't specify trim.
+
+### Volume ramps
+
+```tsx
+<Audio src={staticFile("music.wav")} volume={(f) => interpolate(f, [0, 30], [0, 1])} />
+```
+
+HF supports static `data-volume` only for now. Volume ramps need to be
+applied to the audio file at translation time (with ffmpeg `afade`) or the
+ramp is dropped with a translation note.
+
+### Trim / playbackRate
+
+```tsx
+<Audio src={staticFile("music.wav")} startFrom={60} endAt={180} playbackRate={1.5} />
+```
+
+```html
+<audio
+  data-start="0"
+  data-duration="<resolved from trim>"
+  data-trim-start="2"
+  data-trim-end="6"
+  data-playback-rate="1.5"
+  src="assets/music.wav"
+></audio>
+```
+
+`startFrom` / `endAt` are frame indexes; convert to seconds.
+
+## `<Video>` and `<OffthreadVideo>`
+
+```tsx
+<Video src={staticFile("intro.mp4")} muted playsInline />
+<OffthreadVideo src={staticFile("intro.mp4")} muted />
+```
+
+```html
+<video
+  muted
+  playsinline
+  data-start="0"
+  data-duration="5"
+  data-track-index="0"
+  src="assets/intro.mp4"
+></video>
+```
+
+`<OffthreadVideo>` is a Remotion-specific optimization for headless
+rendering. HF runs in headless Chrome already, so the off-thread variant
+collapses to a regular `<video>`.
+
+`muted` and `playsinline` are required for the runtime to autoplay
+(browser policy). Always emit them.
+
+## `<Img>`
+
+```tsx
+<Img src={staticFile("logo.png")} style={{ width: 200, height: 200 }} />
+```
+
+```html
+<img src="assets/logo.png" style="width: 200px; height: 200px;" />
+```
+
+Width/height get rounded to integer px. If the original style has
+animated dimensions, the GSAP tween animates them — see [timing.md](timing.md).
+
+## `<IFrame>`
+
+```tsx
+<IFrame src="https://example.com" />
+```
+
+```html
+<iframe src="https://example.com"></iframe>
+```
+
+When HF detects a nested iframe in a composition, it auto-falls back to
+**screenshot mode** rather than the deterministic BeginFrame mode. This
+costs render performance but produces visibly-correct output. See
+[hyperframes-vs-remotion.mdx](https://github.com/heygen-com/hyperframes/blob/main/docs/guides/hyperframes-vs-remotion.mdx)
+for details.
+
+## `delayRender()` / `continueRender()`
+
+```tsx
+const handle = delayRender();
+useEffect(() => {
+  loadAsset().then(() => continueRender(handle));
+}, []);
+```
+
+Drop. HF waits on asset readiness via the [Frame Adapter pattern](https://hyperframes.heygen.com/concepts/frame-adapters)
+— images, videos, fonts, and Lottie animations all signal load
+completion natively. There's nothing to do at the application level.
+
+## When the asset isn't a file
+
+If Remotion's media source is a Buffer, dataURL, or URL.createObjectURL,
+the asset doesn't exist on disk and can't be copied via setup.sh. Two
+options:
+
+1. Materialize the asset at translation time — write the buffer to a file
+   in `hf-src/assets/`.
+2. Embed as a data URL directly in the HTML (`src="data:image/png;base64,..."`)
+   for small assets (< 100 KB).
+
+For audio/video Buffers, option 1 is preferred — base64-encoded media
+bloats the HTML and slows the renderer.

package/skills/remotion-to-hyperframes/references/parameters.md

@@ -0,0 +1,167 @@
+# Parameter translation: Zod schemas, defaultProps, calculateMetadata
+
+How a typed Remotion `<Composition schema={...} defaultProps={...} />`
+turns into a parameterized HF composition.
+
+## Sync calculateMetadata (translatable)
+
+```tsx
+<Composition
+  id="MyVideo"
+  component={MyVideo}
+  schema={z.object({ title: z.string(), duration: z.number() })}
+  defaultProps={{ title: "Hello", duration: 90 }}
+  calculateMetadata={({ props }) => ({
+    durationInFrames: props.duration,
+    fps: 30,
+  })}
+/>
+```
+
+When `calculateMetadata` is synchronous and only uses `props`, **resolve
+it at translation time** — call it with `defaultProps` (or whatever the
+caller specifies) and write the concrete result into the HTML:
+
+```html
+<div
+  id="stage"
+  data-composition-id="MyVideo"
+  data-start="0"
+  data-duration="3"          <!-- 90/30 -->
+  data-fps="30"
+  data-title="Hello"
+></div>
+```
+
+The `data-title` attribute carries the value through. Code that originally
+read `props.title` reads `document.getElementById("stage").dataset.title`
+in HF.
+
+## Async calculateMetadata (NOT translatable)
+
+```tsx
+<Composition
+  calculateMetadata={async ({ props }) => {
+    const res = await fetch(...);
+    return { durationInFrames: res.duration };
+  }}
+/>
+```
+
+**Refuse + interop**. HF needs composition metadata up-front to seed the
+HTML. Resolving network calls at translation time defeats the purpose
+of having dynamic metadata. See [escape-hatch.md](escape-hatch.md).
+
+The lint rule `r2hf/async-metadata` catches this. T4 case 03 tests it.
+
+## Default props
+
+```tsx
+defaultProps={{
+  title: "Hello",
+  subtitle: "World",
+  count: 42,
+}}
+```
+
+Translate to `data-*` attributes on the root `#stage` div:
+
+```html
+<div id="stage" data-title="Hello" data-subtitle="World" data-count="42">...</div>
+```
+
+Convention: `propName` → `data-prop-name` (kebab-case). Inside the GSAP
+script, read via `document.getElementById("stage").dataset.propName`.
+
+## Nested object / array props
+
+```tsx
+defaultProps={{
+  stats: [
+    { label: "Stars", value: 1247, color: "#fbbf24" },
+    { label: "Forks", value: 312, color: "#60a5fa" },
+  ],
+}}
+```
+
+Don't try to encode the array as a JSON `data-` attribute — HF's runtime
+doesn't parse those. Materialize the array as **repeated HTML markup**:
+
+```html
+<div id="scene-stats">
+  <div class="stat-card" data-stat-index="0" data-stat-value="1247" style="--card-color:#fbbf24">
+    <div class="number">0</div>
+    <div class="label">Stars</div>
+  </div>
+  <div class="stat-card" data-stat-index="1" data-stat-value="312" style="--card-color:#60a5fa">
+    <div class="number">0</div>
+    <div class="label">Forks</div>
+  </div>
+</div>
+```
+
+The component template (`StatCard.tsx`) becomes the markup template;
+each instance gets its scalar props rendered as `data-*` and CSS
+custom properties.
+
+Validated in T3 — three StatCards reused with different props,
+mean SSIM 0.953.
+
+## Numeric props that need typed parsing
+
+`document.getElementById("stage").dataset.count` is a string. Convert at
+read time:
+
+```js
+const count = Number(stage.dataset.count);
+```
+
+Or inline values directly into the GSAP script when the data is known
+at translation time and doesn't need to vary per render.
+
+## Boolean props
+
+```tsx
+defaultProps={{ darkMode: true }}
+```
+
+Two conventions:
+
+- `data-dark-mode="true"` — read as string, compare `=== "true"`
+- `data-dark-mode` (presence/absence) — `<div data-dark-mode>` for true, omit for false
+
+Pick one and be consistent. The presence/absence form is HTML-idiomatic
+and pairs well with CSS attribute selectors:
+
+```css
+[data-dark-mode] .scene {
+  background: #000;
+}
+```
+
+## Zod runtime validation
+
+Remotion's `schema` validates props at composition load. HF doesn't have
+an equivalent — by the time the HTML is in the renderer, the schema is
+already gone.
+
+Validate at translation time instead. If the user passes invalid data,
+fail with a translation error before emitting HTML. This matches Zod's
+"fail loud" intent without requiring the runtime dependency.
+
+## When the composition uses props for computed prop derivation
+
+```tsx
+const Composition: React.FC<Props> = ({ stats }) => {
+  const total = stats.reduce((acc, s) => acc + s.value, 0);
+  return <div>{total}</div>;
+};
+```
+
+Compute the derived value at translation time and bake it into the HTML
+or a `data-` attribute. Don't try to express the computation in JS in the
+HF composition — that adds runtime overhead and makes the HTML stateful
+in ways that complicate human editing.
+
+If the derivation is non-trivial (involves the array itself, not just
+scalars), materialize it as static text in the HTML.

package/skills/remotion-to-hyperframes/references/sequencing.md

@@ -0,0 +1,198 @@
+# Sequencing translation: Sequence, Series, Composition root
+
+How Remotion's nested `Sequence` tree maps to HF's flat `data-start` /
+`data-duration` markup with a single paused GSAP timeline.
+
+## The core idea
+
+Remotion's `<Sequence from={F} durationInFrames={D}>` is a coordinate
+transform: it shifts `useCurrentFrame()` by `F` and clips the child
+component to the window `[F, F+D]`. HF doesn't have a per-element
+"current frame" — there's a single composition seek time and the
+runtime hides/shows elements based on their `data-start` / `data-duration`.
+
+Result: the nested tree flattens into a list of siblings on the same
+parent, each with their own time window.
+
+## `<Composition>` → root `#stage`
+
+```tsx
+<Composition
+  id="MyVideo"
+  component={MyVideo}
+  durationInFrames={300}
+  fps={30}
+  width={1280}
+  height={720}
+/>
+```
+
+```html
+<div
+  id="stage"
+  data-composition-id="MyVideo"
+  data-start="0"
+  data-duration="10"      <!-- 300/30 -->
+  data-fps="30"
+  data-width="1280"
+  data-height="720"
+>
+  <!-- composition content -->
+</div>
+```
+
+`data-start="0"` is required on `#stage` (the runtime needs it to anchor
+playback; missing it triggers a lint warning).
+
+## `<AbsoluteFill>` → positioned div
+
+```tsx
+<AbsoluteFill style={{ backgroundColor: "#0a0a0a" }}>...children...</AbsoluteFill>
+```
+
+```html
+<div style="position:absolute;inset:0;background-color:#0a0a0a;">...children...</div>
+```
+
+`AbsoluteFill` is just a styled div in Remotion. Translate to a div with
+`position:absolute; inset:0` and copy through any other style props.
+
+## `<Sequence>` → time-windowed div
+
+```tsx
+<Sequence from={0} durationInFrames={90}>
+  <TitleCard />
+</Sequence>
+```
+
+```html
+<div data-start="0" data-duration="3" data-track-index="0">
+  <!-- TitleCard children inlined -->
+</div>
+```
+
+Convert frames to seconds: `from/fps`, `durationInFrames/fps`. Pick a
+`data-track-index` per parallel rendering layer (background = 0,
+overlays = 1, audio = 2, etc.). Sequential scenes can share an index.
+
+## Nested `<Sequence>` flattens
+
+Remotion adds offsets when sequences nest:
+
+```tsx
+<Sequence from={60} durationInFrames={120}>
+  <Sequence from={30} durationInFrames={60}>
+    <ImageScene />
+  </Sequence>
+</Sequence>
+```
+
+The inner sequence's effective window is `[60+30, 60+30+60] = [90, 150]`.
+
+Translate by computing the sum and emitting one HF div with the resolved
+window:
+
+```html
+<div data-start="3" data-duration="2" data-track-index="0">
+  <!-- ImageScene children -->
+</div>
+```
+
+## `<Series>` → siblings with sequential offsets
+
+```tsx
+<Series>
+  <Series.Sequence durationInFrames={60}>
+    <A />
+  </Series.Sequence>
+  <Series.Sequence durationInFrames={120}>
+    <B />
+  </Series.Sequence>
+  <Series.Sequence durationInFrames={90}>
+    <C />
+  </Series.Sequence>
+</Series>
+```
+
+Each `Sequence.Sequence` lives in the next time slot. Emit siblings
+with `data-start` accumulating:
+
+```html
+<div data-start="0" data-duration="2" data-track-index="0">A</div>
+<div data-start="2" data-duration="4" data-track-index="0">B</div>
+<div data-start="6" data-duration="3" data-track-index="0">C</div>
+```
+
+## Crossfading scene boundaries
+
+Remotion `<Sequence>` shows/hides at hard boundaries by default. HF does
+the same — but if your composition needs a smooth fade between scenes,
+you have to drive opacity explicitly with GSAP at the boundary:
+
+```js
+const tl = gsap.timeline({ paused: true });
+tl.set(scene1, { opacity: 1 }, 0);
+tl.set(scene1, { opacity: 0 }, 2); // hard cut at 2s
+tl.set(scene2, { opacity: 1 }, 2);
+```
+
+For a 0.5 s crossfade:
+
+```js
+tl.to(scene1, { opacity: 0, duration: 0.5 }, 1.5);
+tl.to(scene2, { opacity: 1, duration: 0.5 }, 1.5);
+```
+
+For Remotion `<TransitionSeries>` translations see [transitions.md](transitions.md).
+
+## `<Loop>`
+
+```tsx
+<Loop durationInFrames={30}>
+  <Spinner />
+</Loop>
+```
+
+HF doesn't have a `<Loop>` primitive. Translate it to a bounded GSAP timeline using
+the time available at its insertion point:
+
+```js
+const cycleDuration = 1;
+const availableDuration = compositionDuration - 3;
+const repeat = Math.max(0, Math.floor(availableDuration / cycleDuration) - 1);
+const spinTl = gsap.timeline({ paused: true, repeat, repeatRefresh: false });
+spinTl.to(spinner, { rotate: 360, duration: 1.0, ease: "none" });
+// Embed in the main composition timeline at the right offset:
+mainTl.add(spinTl, 3);
+```
+
+This is fragile — Remotion's `<Loop>` resets internal state every iteration,
+which GSAP repeat does too, but if the looped child has its own animation,
+you need to be careful that GSAP's `repeatRefresh` is on or off as needed.
+The finite count is required because HyperFrames seeks a bounded composition frame-by-frame.
+
+## `<Freeze>`
+
+```tsx
+<Freeze frame={30}>
+  <Animated />
+</Freeze>
+```
+
+Drop the wrapper. `<Freeze>` pins `useCurrentFrame()` at a constant for
+the children — but in HF, the children's animation is already driven by
+explicit GSAP tweens, so freeze translates to "don't tween this element".
+
+## Multiple parallel tracks
+
+When you have a background video + overlay text + audio playing
+simultaneously, use distinct `data-track-index` values:
+
+```html
+<div data-track-index="0">background video</div>
+<div data-track-index="1">overlay text</div>
+<audio data-track-index="2" ...></audio>
+```
+
+The runtime picks track ordering from the index. See [media.md](media.md)
+for media-specific track conventions.

package/skills/remotion-to-hyperframes/references/timing.md

@@ -0,0 +1,165 @@
+# Timing translation: interpolate, spring, easing
+
+The single highest-leverage reference. Easings and timings are what readers
+notice; getting them wrong costs more SSIM than any other translation choice.
+Empirically validated against tiers T1–T3.
+
+## Conversion: frames → seconds
+
+HF's timeline is in seconds. Remotion is frame-based. Always:
+
+```
+time_seconds = frame / fps
+```
+
+So at fps=30:
+
+- frame 15 → 0.5 s
+- frame 30 → 1.0 s
+- frame 90 → 3.0 s
+
+Do this conversion once when translating, not at runtime.
+
+## interpolate — linear
+
+```tsx
+const opacity = interpolate(frame, [0, 30], [0, 1], { extrapolateRight: "clamp" });
+```
+
+Translates to:
+
+```js
+gsap.to(target, { opacity: 1, duration: 1.0, ease: "none" }, 0);
+// fromTo if the property starts at 0 and CSS doesn't already set it
+gsap.fromTo(target, { opacity: 0 }, { opacity: 1, duration: 1.0, ease: "none" }, 0);
+```
+
+`ease: "none"` matches Remotion's default linear interpolation. CSS sets the
+`from` value if your initial state is in CSS; otherwise use `fromTo`.
+
+`extrapolateLeft`/`extrapolateRight` defaults to `"extend"` in Remotion but
+`"clamp"` is what the agent will see most often. GSAP doesn't extend — values
+hold at the start and end of the tween. So for `clamp`, GSAP matches; for
+`extend`, you'd need to extend the input range manually before emitting.
+
+## interpolate — multi-segment
+
+```tsx
+const opacity = interpolate(frame, [0, 15, 75, 90], [0, 1, 1, 0]);
+```
+
+Three keyframed tweens at offsets `[0]/fps`, `[1]/fps`, `[2]/fps`:
+
+```js
+const tl = gsap.timeline({ paused: true });
+tl.to(target, { opacity: 1, duration: 0.5, ease: "none" }, 0);
+tl.to(target, { opacity: 1, duration: 2.0, ease: "none" }, 0.5);
+tl.to(target, { opacity: 0, duration: 0.5, ease: "none" }, 2.5);
+```
+
+Validated in T1 — mean SSIM 0.974 against Remotion baseline.
+
+## spring → GSAP back.out
+
+Remotion's `spring()` is the most lossy translation. The mapping is approximate
+but close enough that real-world compositions hold ≥ 0.92 SSIM (T2: 0.985, T3: 0.953).
+
+| Remotion `spring` config                          | GSAP equivalent                                      | Validated in                     |
+| ------------------------------------------------- | ---------------------------------------------------- | -------------------------------- |
+| `{damping: 12, stiffness: 100, mass: 1}` (snappy) | `back.out(1.4)` over ~0.7 s                          | T2, T3 (TitleScene)              |
+| `{damping: 14, stiffness: 90, mass: 1}` (calmer)  | `back.out(1.2)` over ~0.7 s                          | T3 (StatCard)                    |
+| `{damping: 8, stiffness: 200}` (very bouncy)      | `back.out(2.0)` or `elastic.out(1, 0.5)` over ~0.6 s | not validated; budget ~0.05 SSIM |
+| `{overshootClamping: true}`                       | `power3.out` over ~0.6 s (no overshoot)              | not validated                    |
+
+**Rule of thumb**: `back.out(N)` overshoot ratio ≈ `(stiffness / damping^2) * 1.4`. For
+`damping:12, stiffness:100` that gives `1.4 * 100/144 = 0.97`, which is close to
+the validated 1.4 (the formula is rough; tune by visual). Default duration is
+~0.7 s for the typical config.
+
+When the spring's `delay`/`from`/`to` are non-default, scale the duration
+proportionally.
+
+## interpolate with custom easing
+
+```tsx
+import { Easing } from "remotion";
+interpolate(frame, [0, 30], [0, 1], { easing: Easing.out(Easing.cubic) });
+```
+
+| Remotion                     | GSAP                                                                                   |
+| ---------------------------- | -------------------------------------------------------------------------------------- |
+| `Easing.in(Easing.linear)`   | `ease: "none"`                                                                         |
+| `Easing.out(Easing.cubic)`   | `ease: "power3.out"`                                                                   |
+| `Easing.inOut(Easing.cubic)` | `ease: "power3.inOut"`                                                                 |
+| `Easing.out(Easing.poly(N))` | `ease: "power<N>.out"` (N=2 quad, 3 cubic, 4 quart, 5 quint)                           |
+| `Easing.bezier(a,b,c,d)`     | `CustomEase.create("c", "M0,0 C${a},${b} ${c},${d} 1,1")` (requires CustomEase plugin) |
+| `Easing.elastic(bounciness)` | `ease: "elastic.out(${bounciness}, 0.3)"`                                              |
+| `Easing.bounce`              | `ease: "bounce.out"`                                                                   |
+| `Easing.back(overshoot)`     | `ease: "back.out(${overshoot * 1.7})"` (Remotion's overshoot scale differs)            |
+
+## interpolate driving non-numeric properties
+
+```tsx
+const color = interpolateColors(frame, [0, 30], ["#ff0000", "#0000ff"]);
+```
+
+GSAP does color tweens natively:
+
+```js
+gsap.to(target, { color: "#0000ff", duration: 1.0, ease: "none" }, 0);
+```
+
+Same for `backgroundColor`, `borderColor`. The `from` value is read from CSS
+or the inline style.
+
+## Custom count-up / number tweens
+
+When Remotion uses a frame-driven number ramp (`Math.round(value * eased)`):
+
+```tsx
+const t = interpolate(frame, [0, 45], [0, 1]);
+const eased = 1 - (1 - t) ** 3; // cubic ease-out
+const value = Math.round(target * eased);
+return <div>{value.toLocaleString()}</div>;
+```
+
+GSAP equivalent — tween a counter object, write `textContent` on update:
+
+```js
+const counter = { v: 0 };
+tl.to(
+  counter,
+  {
+    v: target,
+    duration: 1.5,
+    ease: "power3.out",
+    onUpdate: () => {
+      el.textContent = Math.round(counter.v).toLocaleString();
+    },
+  },
+  0,
+);
+```
+
+`power3.out` matches `1 - (1-t)^3` exactly. Validated in T3 (mean SSIM 0.953).
+Per-frame digit mismatches occur on sub-frame timing offsets but final values
+converge — no SSIM impact above the noise floor.
+
+## Stagger via per-instance prop
+
+When custom subcomponents take a `delayInFrames` prop:
+
+```tsx
+<StatCard delayInFrames={i * 12} value={...} />
+```
+
+Translate to GSAP timeline offsets:
+
+```js
+cards.forEach((card, i) => {
+  const start = base + i * (12 / fps); // i * 0.4s at fps=30
+  tl.to(card, { ... }, start);
+});
+```
+
+Validated in T3 — three StatCards staggered at 0.0/0.4/0.8 s.