@depths/waves 0.1.0

package/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2025 Depths AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

package/README.md

@@ -0,0 +1,435 @@
+# @depths/waves
+
+`@depths/waves` is a TypeScript-first library for rendering videos from a JSON “intermediate representation” (IR).
+
+The intended workflow is:
+
+1. An LLM (or a human) produces JSON that conforms to a schema (`VideoIRSchema`).
+2. The IR is validated (schema + semantics).
+3. The IR is rendered to an output video file (MP4 by default) using Remotion.
+
+This project is designed around explicitness:
+
+- You only render components that have been explicitly registered in a component registry.
+- Component props are validated with Zod before rendering.
+- Scene timing is validated for common authoring mistakes (gaps, overlaps, duration mismatches, children exceeding parent bounds).
+
+## What you get (v0.1.0)
+
+- A stable IR format (currently `version: "1.0"`) for videos composed of scenes and components.
+- A component registry (`ComponentRegistry`) that maps `type` strings to React components + Zod props schemas + metadata.
+- A validator (`IRValidator`) with schema + semantic checks.
+- A rendering engine (`WavesEngine`) that bundles a Remotion project on the fly and renders to a media file.
+- Built-in primitives: `Scene`, `Text`, `Audio`.
+- Utilities to generate JSON Schemas for LLM prompting from registered component props.
+
+## Why this exists
+
+Many “LLM → video” pipelines fail for one of these reasons:
+
+- The LLM output is not reliably validated and fails at render time.
+- The rendering system has hidden magic (implicit component inference, brittle conventions).
+- The model lacks a precise schema + component catalog to target.
+
+Waves solves this by:
+
+- Giving the model a JSON schema to follow.
+- Enforcing strict component registration and props validation.
+- Providing deterministic, testable semantic rules around timing.
+
+## Installation
+
+```bash
+npm i @depths/waves
+```
+
+Peer dependencies (must be installed by the consumer):
+
+```bash
+npm i react remotion
+```
+
+Rendering prerequisites:
+
+- `ffmpeg` must be installed and available on the PATH.
+- A Chromium browser must be available to `@remotion/renderer` (Remotion’s renderer runs headless Chromium).
+
+## CLI / Agent workflow
+
+Waves ships a `waves` CLI so a locally running AI agent (or any terminal user) can:
+
+1) fetch a system prompt + JSON Schemas (the authoring contract)
+2) write a starter IR file
+3) validate the IR
+4) render an MP4 from the IR
+
+Install locally (recommended):
+
+```bash
+npm i -D @depths/waves
+```
+
+Run the CLI via the local bin:
+
+```bash
+npx waves --help
+```
+
+Or run without installing (one-off):
+
+```bash
+npx -y -p @depths/waves waves --help
+```
+
+### 1) Get the agent prompt + schemas
+
+Machine-readable payload (recommended for agents):
+
+```bash
+npx waves prompt --format json --out ./waves-prompt.json
+```
+
+Human-readable system prompt:
+
+```bash
+npx waves prompt --format text --out ./waves-system-prompt.txt
+```
+
+Schemas only:
+
+```bash
+npx waves schema --kind all --pretty --out ./waves-schemas.json
+```
+
+If you register custom components at module import time, include them with repeatable `--register`:
+
+```bash
+npx waves prompt --format json --register ./src/register-waves-components.ts
+```
+
+### 2) Write IR JSON
+
+```bash
+npx waves write-ir --template basic --pretty --out ./video.ir.json
+```
+
+### 3) Validate IR JSON
+
+```bash
+npx waves validate --in ./video.ir.json
+```
+
+Structured validation result:
+
+```bash
+npx waves validate --in ./video.ir.json --format json
+```
+
+### 4) Render MP4
+
+```bash
+npx waves render --in ./video.ir.json --out ./output.mp4 --codec h264 --crf 28 --concurrency 1
+```
+
+If your IR references `"/assets/..."` paths, pass `--publicDir` and ensure the files exist at `${publicDir}/assets/...`:
+
+```bash
+npx waves render --in ./video.ir.json --out ./output.mp4 --publicDir ./public
+```
+
+### Exit codes
+
+- `0`: success
+- `1`: usage error (invalid flags/command)
+- `2`: validation failure (invalid JSON or IR validation errors)
+- `3`: render failure
+- `4`: I/O error (missing/unreadable/unwritable files)
+- `5`: internal error (bug)
+
+## Quickstart (one function)
+
+```ts
+import { renderVideo } from '@depths/waves';
+
+await renderVideo(
+  {
+    version: '1.0',
+    video: { id: 'my-video', width: 1920, height: 1080, fps: 30, durationInFrames: 90 },
+    scenes: [
+      {
+        id: 'scene-1',
+        type: 'Scene',
+        timing: { from: 0, durationInFrames: 90 },
+        props: { background: { type: 'color', value: '#000000' } },
+        children: [
+          {
+            id: 'title',
+            type: 'Text',
+            timing: { from: 0, durationInFrames: 90 },
+            props: { content: 'Hello Waves', fontSize: 72, animation: 'fade' }
+          }
+        ]
+      }
+    ]
+  },
+  { outputPath: './output.mp4' }
+);
+```
+
+## Core concepts
+
+### 1) IR (Intermediate Representation)
+
+The IR is plain JSON. At minimum, it includes:
+
+- `version`: currently `"1.0"`
+- `video`: dimensions, fps, total duration
+- `scenes`: an array of `Scene` components
+
+All timing is expressed in frames.
+
+### 2) Timing model
+
+- Every component has a `timing` object: `{ from, durationInFrames }`
+- Scenes are expected to be sequential:
+  - Scene 0 starts at frame 0
+  - Scene N starts exactly where Scene N-1 ends
+  - Sum of scene durations must equal `video.durationInFrames`
+- Child components (inside a `Scene`) must fit within the scene duration.
+
+### 3) Component registry
+
+Waves never renders arbitrary `type` strings. A component `type` must be registered:
+
+- `type`: a string identifier, e.g. `"Text"`
+- `component`: a React component to render
+- `propsSchema`: a Zod schema used to validate and coerce defaults
+- `metadata`: descriptions and examples useful for LLM prompting
+
+### 4) Rendering engine
+
+Rendering is done through Remotion:
+
+1. Validate IR
+2. Generate a temporary Remotion entry point
+3. Bundle using `@remotion/bundler`
+4. Select the composition using `@remotion/renderer`
+5. Render the media to `outputPath`
+
+Important limitation (v0.1.0):
+
+- `WavesEngine` currently requires using `globalRegistry` for rendering, because the generated Remotion bundle needs to register the same components at bundle-time.
+
+## API reference
+
+### `VideoIRSchema`
+
+Use this to validate IR generated by an LLM:
+
+```ts
+import { VideoIRSchema } from '@depths/waves';
+
+const parsed = VideoIRSchema.safeParse(maybeJson);
+if (!parsed.success) {
+  // parsed.error.issues
+}
+```
+
+### `IRValidator`
+
+Runs both schema validation and semantic checks:
+
+```ts
+import { IRValidator } from '@depths/waves';
+
+const validator = new IRValidator();
+const result = validator.validate(maybeJson);
+if (!result.success) {
+  // result.errors: { path: string[]; message: string; code: string }[]
+}
+```
+
+Semantic checks include:
+
+- Scene duration sum matches `video.durationInFrames` (`DURATION_MISMATCH`)
+- Scene `from` frames are sequential (`TIMING_GAP_OR_OVERLAP`)
+- Child components do not exceed their parent duration (`COMPONENT_EXCEEDS_PARENT`)
+
+### `ComponentRegistry` / `globalRegistry`
+
+```ts
+import { ComponentRegistry, globalRegistry } from '@depths/waves';
+```
+
+Typically you will use `globalRegistry` so that the rendering bundle can register and render the same types.
+
+### `registerBuiltInComponents()`
+
+Registers built-in primitives (`Scene`, `Text`, `Audio`) into `globalRegistry`.
+
+```ts
+import { registerBuiltInComponents } from '@depths/waves';
+
+registerBuiltInComponents();
+```
+
+This function is idempotent: calling it multiple times does not double-register types.
+
+### `renderVideo()`
+
+Convenience wrapper:
+
+- Registers built-ins
+- Constructs `WavesEngine(globalRegistry, new IRValidator())`
+- Calls `engine.render()`
+
+```ts
+import { renderVideo } from '@depths/waves';
+
+await renderVideo(ir, { outputPath: './out.mp4' });
+```
+
+### `WavesEngine`
+
+Lower-level control:
+
+```ts
+import { WavesEngine, globalRegistry, IRValidator, registerBuiltInComponents } from '@depths/waves';
+
+registerBuiltInComponents();
+const engine = new WavesEngine(globalRegistry, new IRValidator());
+await engine.render(ir, {
+  outputPath: './out.mp4',
+  codec: 'h264',
+  crf: 18,
+  concurrency: 4
+});
+```
+
+Options:
+
+- `outputPath` (required): output file path
+- `codec`: one of `'h264' | 'h265' | 'vp8' | 'vp9'` (default: `h264`)
+- `crf`: quality parameter (lower is higher quality)
+- `concurrency`: number or percentage string supported by Remotion renderer
+- `publicDir`: optional directory for Remotion `staticFile()` assets
+
+Advanced / internal:
+
+- `rootDir`: temp directory root (defaults to `process.cwd()`)
+- `registrationModules`: module specifiers to import into the Remotion bundle before rendering (useful if you register custom components at module import time)
+
+## Asset handling
+
+Waves supports two kinds of asset references in the IR:
+
+1. Remote URLs: `https://...` or `http://...`
+2. Absolute “public” paths: `/assets/foo.png`
+
+For absolute paths:
+
+- `Scene`/`Audio` resolve them using Remotion’s `staticFile()` which expects a path relative to the `publicDir`.
+- Example: `src: "/assets/a.png"` becomes `staticFile("assets/a.png")`.
+
+If you reference `"/assets/..."`, pass `publicDir` to `renderVideo()` / `engine.render()` and make sure the file exists at:
+
+```
+${publicDir}/assets/...
+```
+
+## Built-in components (v0.1.0)
+
+### `Scene`
+
+Container component for a segment of the video.
+
+Props:
+
+- `background` (required):
+  - `{ type: "color", value: "#RRGGBB" }`
+  - `{ type: "image", value: "/assets/bg.png" | "https://..." }`
+  - `{ type: "video", value: "/assets/bg.mp4" | "https://..." }`
+
+Children:
+
+- Any registered components; typically `Text` and `Audio`.
+
+### `Text`
+
+Animated text overlay.
+
+Props:
+
+- `content` (required)
+- `fontSize` (default `48`)
+- `color` (default `#FFFFFF`)
+- `position` (default `"center"`) — `"top" | "center" | "bottom" | "left" | "right"`
+- `animation` (default `"fade"`) — `"none" | "fade" | "slide" | "zoom"`
+
+### `Audio`
+
+Audio playback. Supports remote URLs or `staticFile()` assets.
+
+Props:
+
+- `src` (required)
+- `volume` (default `1`)
+- `startFrom` (default `0`) — trims from the beginning in frames
+- `fadeIn` (default `0`) — fade-in duration in frames
+- `fadeOut` (default `0`) — fade-out duration in frames
+
+Notes:
+
+- `fadeIn`/`fadeOut` are implemented via a `volume(frame)` curve.
+- The engine passes an internal prop `__wavesDurationInFrames` so `fadeOut` can compute its end.
+
+## LLM integration
+
+### Generating a prompt schema for the model
+
+```ts
+import { globalRegistry, registerBuiltInComponents } from '@depths/waves';
+
+registerBuiltInComponents();
+const componentCatalog = globalRegistry.getJSONSchemaForLLM();
+```
+
+This returns an object keyed by component type, including:
+
+- A JSON Schema describing the component’s props
+- The component’s metadata (description, examples, etc.)
+
+### Typical prompt rules (recommended)
+
+When you prompt a model, include rules like:
+
+- All timing is in frames (`fps` defaults to 30)
+- Total scene durations must equal `video.durationInFrames`
+- Scene start frames must be sequential with no gaps/overlaps
+- Asset paths are either full URLs or absolute paths like `/assets/foo.png`
+
+## Troubleshooting
+
+### Rendering fails with browser/Chromium errors
+
+Remotion renders using headless Chromium. Ensure a compatible Chromium is available to `@remotion/renderer`.
+
+### Rendering fails with ffmpeg errors
+
+Install `ffmpeg` and ensure it is on the PATH.
+
+### Unknown component type
+
+If the engine throws “Unknown component type”:
+
+- Ensure you called `registerBuiltInComponents()`
+- If it’s a custom component, ensure it is registered into `globalRegistry`
+
+### Props validation errors
+
+Props are validated using the registered Zod schema. Defaults are applied by Zod on parse.
+
+## License notes
+
+- This package is MIT licensed (see `LICENSE`).
+- Remotion has separate licensing terms; ensure you comply with Remotion’s license for your use case.