@autoaiai/aicut 0.1.7 → 0.1.11

package/skills/remotion-ads/references/html-in-canvas.md

@@ -0,0 +1,555 @@
+---
+title: HTML in Canvas
+description: Render live HTML/DOM into a canvas inside a Remotion composition, with optional WebGL2 fragment shaders for per-frame visual effects
+section: composition
+priority: medium
+tags: [html-in-canvas, webgl, shader, dom, effect]
+---
+
+# HTML in Canvas
+
+Remotion 4.0.455+ ships an experimental `<HtmlInCanvas>` component that renders
+arbitrary DOM into a canvas every frame. Combined with a 2D context filter or
+a WebGL2 fragment shader, you can apply effects (progressive blur, vintage
+film, chromatic aberration, vignette, …) to live HTML content that would
+otherwise be impossible with CSS alone.
+
+This is most useful for:
+
+- Animated post-process effects on real DOM (rich type, gradients, layout)
+- Capturing a website into a Reel with a phone-frame mockup + scroll animation
+- Cinematic transitions where the HTML content also animates per frame
+- Browser-window mockups in product demos
+
+> **Reference implementation:** the official Remotion examples repo at
+> `github.com/remotion-dev/html-in-canvas` ships several demo compositions
+> (`ArticleHighlight`, `Vintage`, etc.) — recommended starting point.
+
+---
+
+## Status & Compatibility
+
+- **Remotion ≥ 4.0.455** required.
+- API surface relies on **experimental Chrome features**:
+  - `ctx.drawElementImage(elementImage, x, y)` (CanvasRenderingContext2D)
+  - `gl.texElementImage2D(target, level, internalformat, format, type, source)` (WebGL2)
+- These are gated behind `chrome://flags/#canvas-draw-element` in stock Chrome
+  (Canary 149+). **Remotion's headless Chromium already has the flag enabled
+  internally**, so server-side renders work without any user setup.
+- The Chrome team can change or remove the API. Pin a Remotion version known to
+  work and re-test before upgrading.
+- Cannot be used inside `@remotion/web-renderer` (client-side bundle).
+- Nesting `<HtmlInCanvas>` inside another `<HtmlInCanvas>` is not supported.
+
+---
+
+## Imports
+
+```tsx
+import {
+  HtmlInCanvas,
+  type HtmlInCanvasOnInit,
+  type HtmlInCanvasOnPaint,
+} from "remotion";
+```
+
+---
+
+## Component API
+
+```tsx
+<HtmlInCanvas
+  width={width}
+  height={height}
+  onInit={onInit}      // optional; called once per mount, returns cleanup
+  onPaint={onPaint}    // called every frame
+>
+  {/* Any DOM tree — fonts, gradients, box-shadows, transforms, Img, Video */}
+</HtmlInCanvas>
+```
+
+### Callback signatures
+
+```ts
+type HtmlInCanvasOnPaint = (payload: {
+  canvas: HTMLCanvasElement;
+  element: HTMLElement;
+  elementImage: CanvasImageSource;
+}) => void;
+
+type HtmlInCanvasOnInit = (payload: {
+  canvas: HTMLCanvasElement;
+}) => void | (() => void);
+```
+
+`onPaint` runs every render frame. `elementImage` is a special source the
+experimental Chrome API understands — pass it to `ctx.drawElementImage` or
+`gl.texElementImage2D`. **It is not a regular `HTMLImageElement` — `ctx.drawImage`
+will throw.**
+
+`onInit` is the place to allocate GPU resources (WebGL program, textures, VAO);
+return a cleanup function that frees them.
+
+---
+
+## TypeScript Workaround
+
+`drawElementImage` and `texElementImage2D` are not yet in `@types/dom` /
+`@types/webgl2`. Cast the call site to silence the error:
+
+```ts
+(ctx as unknown as {
+  drawElementImage: (img: unknown, x: number, y: number) => void;
+}).drawElementImage(elementImage, 0, 0);
+```
+
+```ts
+(gl as unknown as {
+  texElementImage2D: (
+    target: number,
+    level: number,
+    internalformat: number,
+    format: number,
+    type: number,
+    source: unknown
+  ) => void;
+}).texElementImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, elementImage);
+```
+
+---
+
+## Pattern 1 — Plain 2D Context (Simple Effects)
+
+Best for blur reveals, single drop-shadow, or stitching DOM into a procedurally
+drawn background. No WebGL needed.
+
+```tsx
+import {
+  AbsoluteFill,
+  HtmlInCanvas,
+  type HtmlInCanvasOnPaint,
+  interpolate,
+  useCurrentFrame,
+  useVideoConfig,
+} from "remotion";
+
+const Card: React.FC = () => (
+  <div style={{
+    width: 720, height: 880, padding: 60, background: "#fff",
+    borderRadius: 36, fontFamily: "Inter, system-ui, sans-serif",
+  }}>
+    {/* Anything you can render with HTML/CSS works here */}
+  </div>
+);
+
+export const Demo: React.FC = () => {
+  const frame = useCurrentFrame();
+  const { width, height } = useVideoConfig();
+
+  const blurAmount = interpolate(frame, [0, 28], [32, 0], {
+    extrapolateLeft: "clamp", extrapolateRight: "clamp",
+  });
+
+  const onPaint: HtmlInCanvasOnPaint = ({ canvas, elementImage }) => {
+    const ctx = canvas.getContext("2d");
+    if (!ctx) return;
+    ctx.reset();
+
+    // Draw a procedural background straight to the canvas.
+    const bg = ctx.createLinearGradient(0, 0, 0, canvas.height);
+    bg.addColorStop(0, "#0F1B2D");
+    bg.addColorStop(1, "#3B6B9A");
+    ctx.fillStyle = bg;
+    ctx.fillRect(0, 0, canvas.width, canvas.height);
+
+    // Apply per-frame filter to the live HTML snapshot.
+    ctx.filter = `blur(${blurAmount}px)`;
+    (ctx as unknown as {
+      drawElementImage: (img: unknown, x: number, y: number) => void;
+    }).drawElementImage(elementImage, 100, 100);
+  };
+
+  return (
+    <AbsoluteFill>
+      <HtmlInCanvas width={width} height={height} onPaint={onPaint}>
+        <Card />
+      </HtmlInCanvas>
+    </AbsoluteFill>
+  );
+};
+```
+
+**Key tools available on the 2D context:**
+
+| Tool | Use |
+|------|-----|
+| `ctx.filter = 'blur(Npx)'` | Per-frame blur amount |
+| `ctx.fillStyle` / `ctx.createLinearGradient` | Procedural backdrops |
+| `ctx.translate / scale / rotate` | Per-frame transform of the HTML snapshot |
+| `ctx.globalAlpha` | Fade in/out |
+| `ctx.globalCompositeOperation` | Blend modes (overlay, multiply, …) |
+
+---
+
+## Pattern 2 — WebGL2 Fragment Shader (Heavy Effects)
+
+For Gaussian blur, chromatic aberration, vignette, grain, dust, scratches —
+anything that needs per-pixel math — use a fragment shader. The DOM snapshot
+is uploaded as a 2D texture each frame via `texElementImage2D`.
+
+### Shape of the GL module
+
+A typical `gl.ts` exports `initGl` and `paintGl`. `initGl` is called from
+`HtmlInCanvas` `onInit`, `paintGl` is called from `onPaint`.
+
+```ts
+// gl.ts
+import type { HtmlInCanvasOnInit, HtmlInCanvasOnPaint } from "remotion";
+
+type ElementImage = Parameters<HtmlInCanvasOnPaint>[0]["elementImage"];
+type Canvas = Parameters<HtmlInCanvasOnInit>[0]["canvas"];
+
+export type GlState = {
+  gl: WebGL2RenderingContext;
+  program: WebGLProgram;
+  texture: WebGLTexture;
+  vao: WebGLVertexArrayObject;
+  /* ...uniform locations */
+};
+
+const VS = `#version 300 es
+in vec2 a_pos; in vec2 a_uv; out vec2 v_uv;
+void main() { gl_Position = vec4(a_pos, 0.0, 1.0); v_uv = a_uv; }`;
+
+const FS = `#version 300 es
+precision highp float;
+uniform sampler2D u_tex;
+in vec2 v_uv;
+out vec4 o;
+void main() {
+  // ...your effect here
+  o = texture(u_tex, v_uv);
+}`;
+
+const QUAD = new Float32Array([
+  -1, -1, 0, 0,   1, -1, 1, 0,   -1, 1, 0, 1,
+   1, -1, 1, 0,  -1,  1, 0, 1,    1, 1, 1, 1,
+]);
+
+export const initGl = (canvas: Canvas) => {
+  const gl = canvas.getContext("webgl2", {
+    alpha: true, premultipliedAlpha: true, antialias: false,
+  }) as WebGL2RenderingContext | null;
+  if (!gl) throw new Error("WebGL2 unavailable");
+
+  gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true);
+  // ...compile + link program, allocate texture, VAO, buffer
+  // (see Remotion examples repo for complete reference impl)
+
+  return { gpu /* ...GlState */, cleanup: () => { /* delete program/texture/vao/buffer */ } };
+};
+
+export const paintGl = (gpu: GlState, elementImage: ElementImage, params: { /* uniforms */ }) => {
+  const { gl } = gpu;
+  gl.viewport(0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight);
+  gl.clear(gl.COLOR_BUFFER_BIT);
+  gl.useProgram(gpu.program);
+
+  gl.activeTexture(gl.TEXTURE0);
+  gl.bindTexture(gl.TEXTURE_2D, gpu.texture);
+
+  // Upload the live DOM snapshot as a texture (every frame).
+  (gl as unknown as {
+    texElementImage2D: (
+      t: number, l: number, ifmt: number, fmt: number, type: number, src: unknown
+    ) => void;
+  }).texElementImage2D(gl.TEXTURE_2D, 0, gl.RGBA, gl.RGBA, gl.UNSIGNED_BYTE, elementImage);
+
+  // ...set uniforms, bind VAO, drawArrays
+};
+```
+
+### Wiring it into a composition
+
+```tsx
+import { useCallback, useRef } from "react";
+import { HtmlInCanvas, type HtmlInCanvasOnInit, type HtmlInCanvasOnPaint } from "remotion";
+import { initGl, paintGl, type GlState } from "./gl";
+
+export const Composition: React.FC = () => {
+  const gpuRef = useRef<GlState | null>(null);
+
+  const onInit: HtmlInCanvasOnInit = useCallback(({ canvas }) => {
+    const { gpu, cleanup } = initGl(canvas);
+    gpuRef.current = gpu;
+    return () => { cleanup(); gpuRef.current = null; };
+  }, []);
+
+  const onPaint: HtmlInCanvasOnPaint = useCallback(({ elementImage }) => {
+    if (gpuRef.current) paintGl(gpuRef.current, elementImage, /* uniforms */ {});
+  }, []);
+
+  return (
+    <HtmlInCanvas width={1920} height={1080} onInit={onInit} onPaint={onPaint}>
+      {/* Your DOM tree */}
+    </HtmlInCanvas>
+  );
+};
+```
+
+---
+
+## Effect Recipes
+
+The Remotion examples repo (`github.com/remotion-dev/html-in-canvas/tree/main/src`)
+has full reference shaders for these. Common building blocks:
+
+### Progressive vertical blur (sharp band, blurry edges)
+
+```glsl
+float edgeY = abs(v_uv.y - 0.5) * 2.0;          // 0 at center, 1 at top/bottom
+float blurMix = pow(smoothstep(0.035, 0.62, edgeY), 1.06);
+float sigmaPx = blurMix * 4.25 + 0.05;          // 9×9 Gaussian kernel
+// ...sample neighborhood with gauss1(x, sigmaPx) weights
+vec3 color = mix(sharp.rgb, blurred, blurMix);
+```
+
+### Chromatic aberration (rim leak)
+
+```glsl
+vec2 fromCenter = uv - 0.5;
+float r = length(fromCenter);
+vec2 dir = r > 0.0001 ? normalize(fromCenter) : vec2(0.0);
+float ca = 0.0024 * pow(r * 1.4, 1.6);          // edge-weighted
+color.r = texture(u_tex, uv + dir * ca).r;
+color.g = texture(u_tex, uv).g;
+color.b = texture(u_tex, uv - dir * ca).b;
+```
+
+### Per-pixel grain quantized to 24fps
+
+```glsl
+float frameIdx = floor(u_time * 24.0);          // discrete film steps
+float g = hash(gl_FragCoord.xy + vec2(frameIdx * 17.31, frameIdx * 7.91));
+color += (g - 0.5) * u_grain;
+```
+
+### Gate weave (subtle frame jitter)
+
+```glsl
+vec2 weave = vec2(
+  sin(u_time * 11.0) * 0.0015 + sin(u_time * 3.7) * 0.0009,
+  cos(u_time *  8.3) * 0.0013 + cos(u_time * 2.4) * 0.0007
+);
+uv += weave;                                     // sample-coord offset
+```
+
+### Vignette + exposure flicker
+
+```glsl
+float vig = smoothstep(1.05, 0.30, r * 1.25);
+color *= mix(1.0, vig, u_vignette);
+float flicker = 1.0 + 0.035 * sin(u_time * 7.0) + 0.018 * sin(u_time * 19.0 + 1.3);
+color *= flicker;
+```
+
+---
+
+## Pattern 3 — Page-to-Video (Captured Site Tour)
+
+To make a Reel that scrolls through an existing website, capture the page once
+and animate `translateY` on the screenshot inside a phone-frame mockup. This
+does **not** require `HtmlInCanvas` at all — a regular `<Img>` and `transform`
+is enough. Listed here because it's a frequent companion pattern.
+
+### Step 1 — Capture with Playwright
+
+```js
+// scripts/capture-page.mjs
+import { chromium } from "playwright";
+import fs from "node:fs";
+
+const URL = process.env.URL || "http://localhost:4000";
+const OUT = "public/tour/landing-fullpage.png";
+
+fs.mkdirSync("public/tour", { recursive: true });
+
+const browser = await chromium.launch();
+const ctx = await browser.newContext({
+  viewport: { width: 414, height: 896 }, deviceScaleFactor: 2, isMobile: true,
+});
+const page = await ctx.newPage();
+await page.goto(URL, { waitUntil: "networkidle" });
+
+// Trigger lazy-loaded images / IntersectionObservers.
+await page.evaluate(async () => {
+  const total = document.documentElement.scrollHeight;
+  for (let y = 0; y <= total; y += 400) {
+    window.scrollTo(0, y);
+    await new Promise((r) => setTimeout(r, 80));
+  }
+  window.scrollTo(0, 0);
+});
+await page.waitForTimeout(2500);
+
+await page.screenshot({ path: OUT, fullPage: true });
+
+// Extract real <section> positions so the tour can land on them precisely.
+const stops = await page.evaluate(() => {
+  const fullH = document.documentElement.scrollHeight;
+  return Array.from(document.querySelectorAll("section")).map((el, i) => {
+    const r = el.getBoundingClientRect();
+    const heading = el.querySelector("h1, h2");
+    return {
+      index: i,
+      label: (heading?.textContent ?? `Section ${i + 1}`).trim().slice(0, 80),
+      scrollPercent: (r.top + window.scrollY) / fullH,
+    };
+  });
+});
+fs.writeFileSync("public/tour/stops.json", JSON.stringify({ stops }, null, 2));
+
+await browser.close();
+```
+
+### Step 2 — Stop-and-go scroll timeline
+
+Don't use a continuous `interpolate(frame, [0, durationInFrames], [0, 1])` —
+it feels monotonous. Instead, build a piecewise timeline that **dwells** at
+each section and travels quickly between them.
+
+```ts
+const STOPS = [
+  { scroll: 0.000, caption: "Hero",     dwell: 36 },
+  { scroll: 0.085, caption: "Trust",    dwell: 30 },
+  { scroll: 0.357, caption: "Reviews",  dwell: 28 },
+  { scroll: 0.657, caption: "Stats",    dwell: 28 },
+  { scroll: 1.000, caption: "CTA",      dwell: 36 },
+] as const;
+
+const TRAVEL = 14; // frames to glide between stops
+
+// Build keyframe arrays for interpolate()
+const inputs: number[] = [];
+const outputs: number[] = [];
+const captionStart: number[] = [];
+let t = 0;
+for (let i = 0; i < STOPS.length; i++) {
+  captionStart.push(t);
+  inputs.push(t);            outputs.push(STOPS[i].scroll);
+  t += STOPS[i].dwell;
+  inputs.push(t);            outputs.push(STOPS[i].scroll);
+  if (i < STOPS.length - 1) t += TRAVEL;
+}
+
+const scrollProgress = interpolate(frame, inputs, outputs, {
+  extrapolateLeft: "clamp",
+  extrapolateRight: "clamp",
+  easing: Easing.inOut(Easing.cubic),
+});
+```
+
+Use the real section positions extracted by your capture script — eyeballing
+percentages produces stops that land between sections.
+
+### Step 3 — Phone mockup with translateY
+
+```tsx
+const SHOT_W = 414;
+const SHOT_H = /* extracted from capture */;
+const PHONE_W = 640;
+const PHONE_H = (PHONE_W * 896) / 414;
+const scaledH = (SHOT_H * PHONE_W) / SHOT_W;
+const headroomPx = PHONE_H * 0.05;
+const sectionTopPx = scrollProgress * scaledH;
+const maxScrollPx = scaledH - PHONE_H;
+const translateY = Math.max(-maxScrollPx, Math.min(0, -(sectionTopPx - headroomPx)));
+
+<div style={{ width: PHONE_W, height: PHONE_H, overflow: "hidden", borderRadius: 40 }}>
+  <Img
+    src={staticFile("tour/landing-fullpage.png")}
+    style={{ width: PHONE_W, height: scaledH, transform: `translateY(${translateY}px)` }}
+  />
+</div>
+```
+
+`scrollProgress = sectionTop / fullPageHeight` — this is the *real fractional
+position* of a section, not "0=top, 1=fully scrolled". The clamp + headroom
+keep the section heading slightly below the phone notch.
+
+---
+
+## Render Commands
+
+```bash
+# Plain 2D context — no special flags needed
+npx remotion render Composition out.mp4 --codec=h264 --crf=18
+
+# WebGL2 — needs ANGLE for the headless GL stack
+npx remotion render Composition out.mp4 --codec=h264 --crf=18 --gl=angle
+```
+
+---
+
+## Gotchas
+
+- **CSS animations / framer-motion inside `<HtmlInCanvas>` do not animate.**
+  The whole DOM is rasterized to an `elementImage` once per frame from the
+  *current* React tree. To animate, drive state from `useCurrentFrame()` and
+  re-render — same pattern as any Remotion composition.
+
+- **`ctx.drawImage(elementImage, …)` throws.** Use `ctx.drawElementImage`. The
+  type system won't help — the cast above is necessary.
+
+- **Multiple `TransitionSeries.Transition` wrapping `<HtmlInCanvas>` content
+  can produce blank frames** on the second/third sequence. The exact cause is
+  unclear (likely a snapshot pipeline interaction), but working around with
+  fewer sequences or a single transition is reliable. Test with `npx remotion
+  studio` first.
+
+- **WebGL2 + parallel render workers can crash with `Page crashed!`** on macOS.
+  Symptom: `TargetCloseError` after a few frames. Fix: render with
+  `--concurrency=1`. Slower but stable. Likely VRAM contention from the
+  per-frame `texElementImage2D` upload.
+
+- **Remotion's headless Chromium has the experimental flag enabled** even
+  though stock Chrome requires `chrome://flags/#canvas-draw-element`. Local
+  studio preview also works. End users opening the bundled output don't need
+  the flag — they're watching a rasterized MP4.
+
+- **Cleanup in `onInit` matters.** Return a disposer that deletes the program,
+  texture, VAO, and buffer. Otherwise hot-reload during studio dev leaks GPU
+  memory.
+
+- **Premultiplied alpha + `gl.UNPACK_FLIP_Y_WEBGL = true`** matches what
+  `texElementImage2D` produces. Forgetting the Y-flip renders the page
+  upside-down.
+
+---
+
+## When NOT to Use HtmlInCanvas
+
+- For straight scrolling-screenshot product tours, plain `<Img>` + `translateY`
+  is simpler and faster (Pattern 3 above).
+- For static graphics that don't need per-frame post-processing, just use
+  Remotion's normal compositing.
+- For rendering text with a single shadow or gradient — CSS does that already.
+
+Use `HtmlInCanvas` when you specifically need a per-frame canvas-side effect
+(blur, shader, procedural compositing) on top of live HTML.
+
+---
+
+## Reference Examples
+
+- **`ArticleHighlight`** — progressive vertical blur + animated highlighter
+  sweep with `mix-blend-mode: multiply` over a styled article.
+  https://github.com/remotion-dev/html-in-canvas/tree/main/src/ArticleHighlight
+
+- **`Vintage`** — full vintage-film treatment (gate weave, chromatic
+  aberration, grain, scratches, dust, light leaks, vignette, flicker) over
+  slide-transitioned title cards.
+  https://github.com/remotion-dev/html-in-canvas/tree/main/src/Vintage
+
+Both ship a `gl.ts` with the WebGL2 setup and a `index.tsx` with the
+composition wiring. Treat them as canonical reference implementations.