frameyard 0.4.0 → 0.6.0

package/docs/adr/0012-render-timelines-with-segmented-lossless-clip-intermediates.md

@@ -0,0 +1,31 @@
+# Render timelines with segmented lossless clip intermediates
+
+**Status:** Accepted
+**Date:** 2026-10-31
+
+Timeline rendering will be segmented and parallel. For each segment, Chromium renders the active composition and nested-timeline clips to lossless FFV1 intermediates; FFmpeg decodes media, applies the clip's fit and static transform, composites the segment, encodes H.264, and the ordered segments are concatenated without re-encoding. Audio is mixed once for the whole timeline and muxed at the end. Preview remains a live Chromium timeline host with a bounded pool of active clips; parity is guaranteed at clip capture, not at the final composite.
+
+The production target is a representative 60-minute 1920×1080 30 fps timeline with up to three active layers, a roughly 50/50 composition/media split by layer-seconds, 60% one layer, 30% two layers, and 10% three layers. The measured 10-second segment (two compositions, video, image) captured in 10.7 s, produced 35.2 MB of FFV1, and composited plus encoded in 6.1 s. A continuous three-layer composition-heavy 60-second gate took 52.4 s, and a continuous 50/50 full-frame video-overlay gate took 58.5 s; both extrapolate to about 1× real time and are documented as worst-case behavior, not the target workload. The representative 60-second gate — 50/50 layer-seconds, 60/30/10 occupancy, eight segments, four workers, and a 1080p video fast path — took 29.39 s and produced exactly 1800 frames at 30 fps, extrapolating to about 29.4 minutes for 60 minutes. The full 60-minute gate (2026-11-01) rendered a composition-only stress timeline of 108,000 frames, 120 composition clips, 360 segments, and four workers in 2,120.9 s (35.35 minutes). The output was exactly 108,000 frames / 3,600.000 s at 1920×1080, 30 fps, with scratch cleaned. That gate is stricter than the 50/50 target mix because every second is an HTML composition capture; it exceeds 30 minutes by 18%. The target is therefore met for the representative 50/50 workload, with little margin, while composition-heavy timelines run at roughly 1.7× real time.
+
+The 1.8 TB USB HDD provides enough scratch capacity; a raw-frame cache would be roughly 670–900 GB, which does not fit, which is why the design uses compressed lossless FFV1 rather than raw frames or PNG sequences.
+
+## Considered Options
+
+- **Live browser host (ADR 0011).** Rejected: deterministic and pixel-exact, but 138.8 ms per 1920×1080 frame with four layers versus 33.6 ms for a direct composition capture. It does not meet the long-timeline target.
+- **Skia + FFmpeg streaming compositor.** Rejected for 0.6: it avoids intermediates and preserves capture pixels, but adds a large compositor dependency and a custom frame pipeline. The FFmpeg + FFV1 path already meets the target without it. Skia remains an option if future effects or blend modes outgrow FFmpeg filters.
+- **FFmpeg streaming through pipes.** Rejected: avoids intermediates and dependencies, but requires a fragile cross-platform multi-pipe/filter-graph coordination for a variable set of active layers.
+- **Pre-render every clip for the whole timeline.** Rejected: for 60 minutes the intermediates can be tens to hundreds of gigabytes and the write/read time alone can exceed the render budget. Segmentation bounds peak scratch and lets workers run in parallel.
+- **Lossy intermediates.** Rejected as the default: an extra lossy encode breaks pixel parity at clip capture. FFV1 keeps the captured frames exact; only the final H.264 encode is lossy, as it already is for composition renders.
+
+## Consequences
+
+- Composition and nested-timeline clips are captured with Chromium; media clips are decoded directly by FFmpeg. The timeline compositor is FFmpeg, not the browser.
+- Composition clips that are transparent must be captured with a transparent browser background (`Emulation.setDefaultBackgroundColorOverride` with alpha 0); otherwise the top-level canvas produces a white background. FFV1 uses a pixel format that preserves alpha.
+- Segments are bounded by active-layer changes or a maximum duration (10 s in the benchmark) so each segment has a fixed filter graph; segments are encoded with identical settings and concatenated with `-c copy`.
+- Full-frame video clips with no transform or opacity that already match the project format and whose segment boundaries fall on source keyframes may be stream-copied instead of re-encoded. Other media is scaled/transformed/re-encoded. The representative gate's 29.4-minute projection includes this fast path and excludes any source-transcode time.
+- Timeline workers are bounded (8 by default on the reference machine) and scratch is configurable and cleaned on success and failure. A 60-minute render should not depend on the system `/tmp` tmpfs.
+- Nested timelines are rendered recursively to segment intermediates and treated as one layer, so parent opacity and transform apply to the grouped result. Cycles are rejected statically.
+- Audio is mixed globally rather than per segment, avoiding boundary clicks and keeping clip trims/gains exact.
+- `check` and `still` use the clip capture + timeline composite path for the requested frame; the long-form render path is not used to produce a single still.
+- Preview uses a live Chromium timeline host with a small active-clip pool. Scaling, alpha, and video color conversion may differ from the final FFmpeg composite; the reproducibility guarantee is at clip capture.
+- The 60-second, three-layer, audio-enabled benchmark is the gate before production implementation. A continuous three-layer composition-heavy run measured 52.4 s for 60 s, and a continuous 50/50 full-frame video-overlay run measured 58.5 s; both are about 1× real time and are documented as worst-case. The representative 50/50, 60/30/10 occupancy gate measured 29.39 s for 60 s with four workers and a 1080p media fast path, extrapolating to about 29.4 minutes for 60 minutes with exact 1800-frame output. The full 60-minute, composition-only gate completed in 35.35 minutes with exact 108,000-frame output; composition-heavy performance remains above the 30-minute target. If stricter composition-heavy performance is required, the next levers are a faster lossless intermediate codec, persistent capture sessions, and encoder preset/thread tuning.

package/docs/v0.1-design.md

@@ -2,154 +2,196 @@
 
 **Status:** Accepted
 **Initial acceptance:** 2026-10-27
-**Last design revision:** 2026-10-30
+**Last design revision:** 2026-10-31
 
-This document records the decisions confirmed during design sessions. It is the living product and architecture baseline; later changes revise this document explicitly rather than silently broaden the scope. The original v0.1 baseline has been revised through 0.4. Architecture decisions that changed the baseline are recorded in `docs/adr/`.
+This document records the decisions confirmed during design sessions. It is the living product and architecture baseline; later changes revise this document explicitly rather than silently broaden the scope. The original v0.1 baseline has been revised through 0.6. Architecture decisions that changed the baseline are recorded in `docs/adr/`.
 
 ## 1. Product contract
 
-Frameyard is a local-first, open-source framework for frontend developers and creative coders who build short, parameterized videos with HTML, CSS, and JavaScript. AI agents can perform the same workflows on a user's behalf through an official skill. Its differentiator is web-native authoring combined with deterministic, frame-accurate output—not a React-centric composition model or a feature-for-feature clone of an existing tool.
+Frameyard is a local-first, open-source framework for frontend developers and creative coders who build parameterized videos with HTML, CSS, and JavaScript. It separates reusable, prop-parameterized compositions from the timelines that assemble compositions and media into one video. AI agents can perform the same workflows on a user's behalf through an official skill.
 
-The initial use cases are product demos, social clips, animated diagrams, data visualizations, karaoke-style subtitles, and personalized videos. Optional local speech synthesis and transcription produce assets for these workflows. Professional long-form editing is outside the target audience.
+Its differentiator is web-native authoring combined with deterministic, frame-accurate output and an explicit timeline model—not a React-centric composition model or a feature-for-feature clone of an existing NLE.
+
+The target use cases are product demos, social clips, animated diagrams, data visualizations, karaoke-style subtitles, personalized videos, and long-form edits up to roughly 60 minutes. Professional collaborative editing, cloud production, and broadcast workflows remain outside the target audience.
 
 The core success criterion is an end-to-end example in which a developer can:
 
-1. Create a reusable 30-second, 1920×1080 HTML composition.
-2. Animate it with HTML, CSS, SVG, and, where needed, a seek-safe controller for Canvas 2D and WebGL.
+1. Create reusable, 1920×1080 HTML compositions with fixed metadata.
+2. Animate them with HTML, CSS, SVG, and, where needed, a seek-safe controller for Canvas 2D and WebGL.
 3. Supply JSON props and synchronized audio.
-4. Preview and scrub it interactively.
-5. Render it reproducibly to MP4 without dropped frames or timing drift.
+4. Assemble compositions and local media into a timeline with tracks, clips, trims, layers, and nested timelines.
+5. Preview and scrub the timeline interactively.
+6. Render it reproducibly to MP4 without dropped frames or timing drift.
+7. Render a 60-minute timeline with up to three active layers within the long-form performance target.
 
-The agent-enabled success criterion additionally requires an agent using the official skill to create and preserve a valid source project, optionally produce local audio and transcript assets, validate and inspect the composition, render it, and report all resulting artifact paths without concealing validation failures or skipped visual review.
+The agent-enabled success criterion additionally requires an agent using the official skill to create and preserve a valid source project, optionally produce local audio and transcript assets, validate and inspect the timeline and its compositions, render it, and report all resulting artifact paths without concealing validation failures or skipped visual review.
 
 ## 2. Project and authoring model
 
+### Project
+
 - Projects use ordinary HTML, CSS, static assets, and native JavaScript controllers, either inline in a composition document or in external modules.
-- `video.config.js` explicitly lists `.html` composition entry points. There is no directory-scanning discovery convention.
-- A project may contain multiple compositions, each with a unique ID.
+- `video.config.js` explicitly maps project IDs to source files. There is no directory scanning. It has three optional maps:
+  - `format`: the project's fixed `width`, `height`, and `fps`.
+  - `compositions`: IDs mapped to `.html` composition documents.
+  - `timelines`: IDs mapped to JSON timeline documents.
+- `format` is required when a project has timelines. Every timeline uses it. A composition used in a timeline must share the project frame rate exactly; a different size is fitted and transformed by the clip. A standalone composition render keeps using the composition's own metadata.
+- Composition and timeline IDs share one namespace so `render <id>`, `check <id>`, and `still <id>` are unambiguous. The config rejects duplicates.
+- A project may contain any number of compositions and timelines.
+
+### Composition
+
 - A composition is a full HTML document with one `<fy-composition>` root. The root declares fixed `width`, `height`, `fps`, and `duration-in-frames` metadata. Props cannot change this metadata.
-- Default props are a JSON block, `<script type="application/json" data-fy="props">`. A render's JSON props merge shallowly over the defaults, are validated as JSON, deep-frozen, and fixed for the render.
+- Default props are a JSON block, `<script type="application/json" data-fy="props">`. A load's JSON props merge shallowly over the defaults, are validated as JSON, deep-frozen, and fixed for that load.
 - Props bind into the document through explicit `data-fy-*` attributes: `data-fy-text`, `data-fy-attr-*`, `data-fy-class-*`, `data-fy-if`, and `data-fy-repeat`. Primitive top-level props are also exposed as `--fy-prop-<name>` CSS custom properties. Bindings apply once per load; props are immutable.
 - CSS and the Web Animations API are the animation language. `data-fy-anim-start`, `data-fy-anim-duration`, and `data-fy-anim-easing` schedule one animation per element in composition frames.
-- An optional composition controller is declared with `<script type="text/fy-controller">`. It is either inline source or an external module referenced by `src`; a composition has exactly one controller. It may export `mount({ root, props })`, `update({ frame, props, root })`, `dispose()`, and `validateProps(props)`; at least one is required, and `mount` and `update` are optional for validation-only controllers. It has no metadata, default props, audio, or transcripts. Inline keeps small compositions in one file; external is for reuse, tests, and complex logic.
-- Inline controllers are the default for small, single-use controllers. Extract to an external module when the controller grows, is reused, needs separate imports or tests, or drives Canvas 2D, WebGL, or karaoke.
-- The controller is the only executable code in a composition document. Other executable `<script>` elements and inline `on*` handlers are rejected. Libraries are imported as ES modules from the controller.
+- An optional composition controller is declared with `<script type="text/fy-controller">`. It is either inline source or an external module referenced by `src`; a composition has exactly one controller. It may export `mount`, `update`, `dispose`, and `validateProps`; at least one is required. The controller is the only executable code in the document.
 - `fy` is the authoring namespace: `fy-composition`, `fy-transcript`, `data-fy-*`, `--fy-*`, and `text/fy-controller`. `frameyard` remains the product, package, CLI, config, manifest, server-route, import-specifier, and error-code namespace.
-- User controllers are native JavaScript. The framework itself is authored in TypeScript and ships declarations, but 0.4 does not transpile user TypeScript.
+- User controllers are native JavaScript. The framework itself is authored in TypeScript and ships declarations, but 0.6 does not transpile user TypeScript.
+
+### Timeline
+
+- A timeline is a versioned JSON document. It references compositions and nested timelines by project ID and media clips by project-relative path.
+- A timeline has a project-fixed format, an ordered list of tracks, and a list of clips. It may declare a `background` color and an optional `durationInFrames`.
+- Tracks are `video` or `audio`. Video tracks composite from bottom to top; audio tracks mix. Each clip belongs to exactly one track. Clips on the same track cannot overlap; clips on different tracks can. Touching endpoints are allowed. Transitions are deferred, so overlap within a track would have ambiguous precedence.
+- A clip has an ID, a track, a source, a timeline start, a source in-point, a duration, and clip-specific presentation settings. Source in-point, duration, and timeline start are integer project frames.
+- Clip sources are:
+  - a composition, identified by its project ID;
+  - a nested timeline, identified by its project ID;
+  - a local video, image, or audio asset.
+- A composition clip fixes props for that instance and may trim the composition's source range. The same composition may appear multiple times with different props, source ranges, transforms, and audio settings.
+- A nested timeline clip is a visual and audio source. It uses the project format, trims with the same source in-point and duration, contributes its audio, and is rendered as a group so parent opacity and transform apply to the whole nested result. Cycles are rejected statically. Timeline clips do not override props of compositions inside the nested timeline in 0.6.
+- Media clips use local project files only. Video and audio durations, dimensions, and audio presence are probed before frame production. Images are static and require an explicit duration. A video or audio clip that exceeds its source range is an error; an omitted duration uses the remaining source.
+- Visual clips support a static transform: fit policy (`contain`, `cover`, `stretch`), an optional explicit `width`/`height` box (both fields together, mutually exclusive with `scale`) or a `scale` multiplier, pixel offset, rotation, and opacity. `fit` applies inside the box; without one, the project frame is the box. No keyframes, blend modes, masks, filters, or other effects in 0.6.
+- Visual clips contribute their audio to the timeline mix unless muted: composition audio, embedded video audio, and nested timeline audio. Audio-only clips live on audio tracks and may source audio or video files. Per-clip and per-audio-track gain and mute are supported. There are no linked A/V pairs; a J/L cut is expressed by muting a video clip and placing a separate audio clip with a different source in-point.
+- A timeline's duration is derived from the latest clip end. An explicit `durationInFrames` may extend it but not cut clips. A timeline with no clips is invalid; an audio-only timeline is valid and renders black video with audio.
+- A timeline background is transparent by default. The final video composites it over black; a nested timeline remains transparent so lower layers can show through. A composition that paints no background is transparent and may be used as an overlay.
+- The preview provides a read-only timeline view: tracks, clips, playhead, program monitor, and diagnostics. It is not a visual editor.
 
 ## 3. Deterministic execution
 
-The only sources of animation state are the integer frame number and immutable props. Composition frame rate establishes the mapping between frames and time.
+The timeline frame is the master clock. A composition's frame is derived from the clip's timeline start, source in-point, and duration. The only sources of a composition's animation state remain the integer composition frame and its immutable props.
 
-- The runtime owns the frame clock and deterministic randomness. The controller imports `random("stable:key")` from `frameyard/runtime`.
-- CSS `@keyframes` and Web Animations API animations are paused and positioned from the shared frame clock. `data-fy-anim-*` attributes schedule one animation per element in composition frames; the runtime maps frames to milliseconds and sets the corresponding animation delay and duration.
-- The runtime sets `--fy-frame`, `--fy-time`, `--fy-progress`, `--fy-fps`, `--fy-duration`, and `--fy-duration-frames` on the composition root every seek. Authors use them for simple computed CSS values.
+- The runtime owns the frame clock and deterministic randomness. A composition controller imports `random("stable:key")` from `frameyard/runtime`.
+- CSS `@keyframes` and Web Animations API animations are paused and positioned from the shared composition frame clock. `data-fy-anim-*` schedules one animation per element in composition frames; the runtime maps frames to milliseconds.
+- The runtime sets `--fy-frame`, `--fy-time`, `--fy-progress`, `--fy-fps`, `--fy-duration`, and `--fy-duration-frames` on the composition root every seek.
 - JavaScript animation runs in the controller's `update()`, which must be synchronous and seek-safe. It cannot depend on previously visited frames.
 - Declarative bindings apply once per load, before the controller mounts. Controller-created DOM is the controller's responsibility.
 - Independent `requestAnimationFrame` loops, timers, CSS transitions, wall-clock reads, and unseeded randomness are not supported animation drivers.
-- Development mode provides best-effort diagnostics for forbidden clocks and randomness in the controller; it does not claim to police every third-party library perfectly.
-- Assets, remote data, images, fonts, generated audio, and transcripts are resolved during an explicit loading phase before frame production. Network requests are blocked during frame capture.
+- Timeline clips use static transforms. Timeline-level animation, transitions, effects, and keyframes are not part of 0.6.
+- Preview and production seek composition clips explicitly. The timeline compositor is deterministic given the captured clip frames, media frames, clip settings, and segment boundaries.
+- Assets, remote data, images, fonts, generated audio, transcripts, and media are resolved before frame production. For a segmented timeline render, each segment's composition and nested-timeline clips are loaded and settled before that segment's frames are captured; no frame production depends on a new network request. Network is blocked during capture.
 - AI model inference is never part of the loading phase, controller lifecycle, or frame production.
 - The baseline supports DOM, CSS, SVG, Canvas 2D, WebGL, and frame-seeked embedded video. Canvas 2D and WebGL are drawn from the current frame by the controller.
-- Embedded video and audio are declared in the composition HTML with `data-fy-*` attributes. The runtime loads local files during the loading phase, pauses them, and seeks video from the frame-derived source time before capture. An unmuted video contributes its audio track to the offline mix unless the composition opts out.
+- Embedded video and audio inside compositions are declared with `data-fy-*` attributes. The runtime loads local files, pauses them, and seeks video from the frame-derived source time before capture. An unmuted video contributes its audio track to the timeline mix unless the composition opts out or the clip mutes it.
+- Transparent compositions are captured with an alpha-preserving browser background. Without that step, a top-level Chromium capture paints a white canvas and overlays become white rectangles.
 - WebGL 1 and WebGL 2 canvases are captured through the normal screenshot path. Production sessions pin software WebGL so pixel output does not depend on the host GPU.
 - Compositions are trusted local developer code. Frameyard does not claim to render untrusted compositions safely.
 
 ### Reproducibility guarantee
 
-For the same composition revision, props, frame, assets, and pinned runtime/browser environment, Frameyard guarantees identical visual state. It does not promise bit-identical MP4 bytes, even across runs on the same machine; encoded output depends on the FFmpeg build, encoder threading, and runtime environment.
+For the same timeline or composition revision, props, assets, segment plan, and frame under a pinned runtime/browser/FFmpeg environment, the captured composition frames and the deterministic timeline composite are identical. The final MP4 is not promised to be bit-identical: encoded bytes depend on the encoder and its runtime.
 
-Generated assets are covered once frozen. Frameyard does not promise that repeating speech synthesis or transcription produces identical artifacts across different binaries, models, parameters, or hardware.
+Preview and production share composition behavior and capture composition frames the same way, but the timeline preview composites layers in Chromium while production composites them with FFmpeg. Small differences in scaling, alpha, and video color conversion are expected and do not violate clip-capture parity.
 
 ## 4. Preview and production rendering
 
-Preview and production execute the same composition document and browser host runtime. They differ only in which component drives the frame clock and whether frames are captured.
+Preview and production execute the same composition documents and host runtime. They differ in which component drives the frame clock, which compositor combines timeline layers, and whether frames are captured.
 
 The preview provides:
 
-- Composition selection
-- JSON props editing
-- Play/pause and frame-accurate scrubbing
-- Current frame and time display
-- Audio controls
+- Composition preview: composition selection, JSON props editing, play/pause, frame-accurate scrubbing, frame-by-frame stepping with buttons or arrow keys, one-second jumps with buttons or Shift+arrow keys, current frame and time, audio controls, and diagnostics.
+- Timeline preview: timeline selection, a read-only track/clip view, playhead, program monitor, frame-by-frame stepping with buttons or arrow keys, one-second jumps with buttons or Shift+arrow keys, audio controls, and diagnostics.
+- A bounded pool of active composition and nested-timeline clips. It loads clips around the playhead instead of preloading a whole long timeline.
 
-It is a developer inspection tool, not a visual composition editor.
+The preview server serves each composition HTML document at its project path and injects the framework import map and host runtime. It serves timeline documents and a timeline preview host. The timeline preview host drives active composition/nested-timeline clips, seeks them, and exposes a timeline API to the preview. Preview audio is mixed by the preview page from the resolved timeline audio clips.
 
-The preview server serves each composition HTML document at its project path and injects the framework import map and a host module into composition entries. The host reads the composition metadata, applies bindings, imports the controller from its `src` or from the inline block's virtual module URL, mounts it, and exposes a small `window.__fy` API. Preview loads the document in a same-origin iframe and drives that API. The iframe is not a security sandbox; compositions are trusted local code. The server runs the static composition validation described in §6 before serving a composition to the preview.
+Production composition rendering is unchanged: one Chromium session per composition, explicit frame seeks, PNG capture, and FFmpeg encoding.
 
-Production rendering:
+Production timeline rendering is segmented and parallel:
 
-1. Uses a pinned, Playwright-managed Chromium through the Chrome DevTools Protocol.
-2. Navigates directly to the composition document, waits for the host, and loads the render's props.
-3. Seeks explicitly to each requested frame.
-4. Waits for composition readiness, embedded media decode, and browser paint.
-5. Captures the frame.
-6. Pipes frames in timeline order to FFmpeg for encoding and audio muxing.
+1. Run static timeline validation and resolve the render plan.
+2. Split the timeline into segments at active-layer changes or a maximum segment duration. Each segment has a fixed set of visual layers and a fixed filter graph.
+3. In parallel workers, capture the active composition and nested-timeline clip frames for each segment with Chromium and encode them to lossless FFV1 intermediates. Transparent clips are captured with an alpha-preserving background.
+4. Decode media clips, apply trims, fit, static transforms, and opacity, composite the segment with FFmpeg, and encode H.264.
+5. Join the ordered segments with a stream copy. Segment boundaries never change frame order or timing.
+6. Mix the timeline audio once with FFmpeg and mux it into the final output.
+7. Publish the output atomically; delete scratch intermediates on success and failure.
 
-`check` and `still` use the same host and capture path. `check` evaluates every frame without capturing or encoding. `still` captures one frame through the production path.
+The default timeline worker count is 8, configurable up to the machine's capacity. Scratch is a configurable directory with a bounded budget; it must not depend on `/tmp` alone. The reference long-form target is a representative 60-minute 1920×1080 30 fps timeline with a 50/50 composition/media split by layer-seconds, 60% one active layer, 30% two, and 10% three, rendered in 30 minutes or less on the reference machine. The 10-second benchmark measured 10.7 s for two composition captures, 35.2 MB of FFV1, and 6.1 s to composite and encode. Continuous three-layer stress runs measured 52.4 s and 58.5 s for 60 s of timeline, about 1× real time, and are documented as worst-case rather than the target workload. The representative gate measured 29.39 s for 60 s with four workers, exact 1800-frame output, and a 1080p media fast path, extrapolating to about 29.4 minutes for 60 minutes. The full 60-minute composition-only gate rendered 108,000 frames in 35.35 minutes with exact 108,000-frame / 3,600 s output; it is stricter than the 50/50 target mix because every frame is an HTML composition capture. The fast path copies full-frame, untransformed video whose source matches the project format and has keyframes at segment boundaries; media that needs scaling or transforms is re-encoded and adds cost not included in that projection.
 
-The renderer is sequential by default and has no frame cache. Optional `--concurrency <n>` rendering, with `n` from 1 to 16, opens up to `n` independent browser sessions, each loading the same composition document and props from the same preview server. Workers seek and capture assigned frames, while a single ordered writer pipes them to FFmpeg with bounded lookahead. Parallelism therefore changes scheduling only: every frame remains derivable from its integer frame number and immutable props, and output order is unchanged. The default stays at one session for predictable resource use, and the upper bound limits accidental browser-process and memory growth. Simple scenes target real-time preview, but there is no production rendering-speed guarantee; the CLI reports progress, current frame, and elapsed time.
+The renderer uses a bounded segment plan, a worker pool, and a scratch budget. It never publishes a partial MP4, and it cleans scratch on success and failure. Segment boundaries preserve timeline frame order and timing; the final audio mix is muxed once.
+
+`check` and `still` use the same clip capture and timeline composite path for a requested frame. `check` evaluates every timeline frame; on long timelines this is a validation cost, not a render. `still` captures one frame as PNG without encoding a video.
 
 ## 5. Audio and output
 
-Audio is a declarative timeline of `<audio>` elements. Each element uses `data-fy-start` (composition frame), `data-fy-trim` (seconds from the source start), optional `data-fy-duration` (seconds to use), and `data-fy-volume` (0-1). Preview synchronizes this timeline in the browser; production mixes it offline with FFmpeg rather than treating live `<audio>` or Web Audio playback as the production source of truth. Unmuted embedded videos contribute their own audio tracks to the same mix, placed by the video's frame and trim attributes. Muting a video or setting `data-fy-audio="off"` keeps it visual-only.
+A timeline has one audio mix. It combines:
+
+- composition audio clips declared inside composition clips, shifted and trimmed by the clip's source range;
+- embedded video audio from composition clips and media video clips, unless muted;
+- timeline audio clips on audio tracks;
+- audio from nested timeline clips, recursively.
+
+Per-clip gain and mute, per-audio-track gain and mute, and composition-level volume apply. Audio fades are deferred. The mix is produced once with FFmpeg and muxed into the final MP4. Preview mixes the same resolved clips in the browser for monitoring; it is not the production source of truth.
 
-Audio, video, and transcript `src` values resolve from the project root. The controller `src` and ordinary HTML asset URLs (`img`, stylesheets) resolve relative to the composition document.
+Audio, video, and transcript `src` values resolve from the project root. The controller `src` and ordinary HTML asset URLs resolve relative to the composition document.
 
-Generated speech behaves like any other audio clip. Frameyard does not automatically time-stretch it, change composition duration, or regenerate it to fit. `check` reports clip durations and overrun warnings so users and agents can revise text, timing, or composition deliberately.
+Generated speech behaves like any other audio clip. Frameyard does not automatically time-stretch it, change timeline duration, or regenerate it to fit. `check` reports clip durations and overrun warnings so users and agents can revise text, timing, or the timeline deliberately.
 
-0.4 produces portable MP4 files with H.264 video and AAC audio. It exposes only a small set of validated encoding options, not arbitrary FFmpeg arguments. Single-frame PNG output through `still` is a debugging artifact, not general image-sequence support.
+0.6 produces portable MP4 files with H.264 video and AAC audio. It exposes only a small set of validated encoding options, not arbitrary FFmpeg arguments. Single-frame PNG output through `still` is a debugging artifact, not general image-sequence support.
 
-A render fails immediately on an uncaught exception, unhandled rejection, required-asset failure, or invalid composition output. Diagnostics identify the composition and, where applicable, the frame. The CLI exits nonzero, cleans temporary files, and never reports a partial video as successful. Existing output files are not overwritten unless explicitly requested.
+A render fails immediately on an uncaught exception, unhandled rejection, required-asset failure, invalid composition output, segment encoding failure, or scratch failure. Diagnostics identify the timeline, clip, segment, and, where applicable, the frame. The CLI exits nonzero, cleans temporary files, and never reports a partial video as successful. Existing output files are not overwritten unless explicitly requested.
 
 ## 6. CLI, tooling, and package boundaries
 
 Project and composition commands are:
 
-- `init`: scaffold a minimal HTML composition project, including an agent guide (`AGENTS.md`) and a private `package.json` with pinned `dev`, `check`, `render`, and `still` scripts
-- `dev`: start the preview
-- `render <composition-id> --props props.json --out video.mp4`: render a composition, optionally with `--concurrency <n>` from 1 to 16
-- `check <composition-id>`: run static composition validation, then validate configuration, props, assets, audio, and transcript requirements, load the composition, and evaluate every frame without capturing or encoding
-- `still <composition-id> --frame <n>`: capture one requested frame as PNG through the production rendering path
-- `doctor`: inspect core requirements and optional integrations without repairing or downloading anything
+- `init`: scaffold a timeline-first project with a composition, a one-clip timeline, a project format, an agent guide, a `.gitignore` for `out/`, and a private `package.json` with pinned `dev`, `check`, `render`, and `still` scripts.
+- `dev`: start the preview with composition and timeline modes.
+- `render <id> [--out video.mp4]`: render a timeline or a composition. Timeline renders accept `--workers <n>` and `--scratch <dir>`; composition renders accept `--concurrency <n>` from 1 to 16. The default output is `out/<id>.mp4`.
+- `check <id>`: run static validation, then validate configuration, props, assets, media, audio, and transcript requirements, load the target, and evaluate every frame without capturing or encoding.
+- `still <id> --frame <n>`: capture one requested frame as PNG through the clip capture and timeline composite path.
+- `doctor`: inspect core requirements and optional integrations without repairing or downloading anything.
 
-`check`, `still`, and `render` run a Node-side static composition validation before launching Chromium. It validates metadata, the composition root, the controller declaration, default props, binding attributes and paths, animation attributes, scripts and event handlers, unsupported visual elements, and audio/transcript references. Diagnostics carry stable codes and optional source locations; `check` exits nonzero on static errors without starting a browser. `dev` runs the same validation on load and surfaces its diagnostics in the preview.
+`check`, `still`, and `render` run a Node-side static validation before launching Chromium. For a timeline it validates the JSON document, tracks, clips, references, nested timelines, cycles, frame-rate compatibility, overlaps, source bounds, media, transforms, audio settings, props, and every referenced composition. Static validation and the composition host read the composition grammar through one browser-safe document reader. Timeline diagnostics carry stable codes and optional source locations or JSON paths, and composition diagnostics from inside a timeline carry the clip and track context.
 
 Local asset-production commands are `tts` and `transcribe`. Every non-interactive command supports `--json` with stable result and error schemas. Structured results go to stdout; human diagnostics and inference progress go to stderr.
 
-The built-in server serves native ES modules and static assets, injects an import map and composition host into composition entries, and serves an inline controller through a virtual module URL under the composition's directory. It performs no dependency pre-bundling. Third-party libraries must initially be browser-ready ES modules referenced explicitly. Running through the CLI server is required; direct `file://` execution is unsupported for deterministic preview or rendering. A composition document can still be opened directly as static HTML/CSS.
+The built-in server serves native ES modules, static assets, composition entries, timeline documents, and the timeline preview host. It injects an import map and composition host into composition entries and an inline controller through a virtual module URL. It performs no dependency pre-bundling. Third-party libraries must initially be browser-ready ES modules referenced explicitly. Running through the CLI server is required; direct `file://` execution is unsupported for deterministic preview or rendering.
 
-Frameyard uses one repository and one publishable npm package with three internal layers:
+Frameyard uses one repository and one publishable npm package with four internal layers:
 
-1. Shared composition contracts and HTML binding/metadata parsing
-2. Browser host runtime
-3. Node-based CLI and renderer
+1. Shared composition and timeline contracts and HTML/JSON parsing
+2. Browser composition host and timeline preview host
+3. Node-based static validation and segmented timeline renderer
+4. Node-based CLI, asset production, and doctor
 
-These layers have clean internal boundaries but no public plugin API. FFmpeg is discovered on the user's PATH and checked for required capabilities, with actionable installation and version diagnostics. The project supports the current active Node.js LTS; Linux is the reference and CI platform, while macOS and Windows remain best-effort until covered by real rendering tests.
+These layers have clean internal boundaries but no public plugin API. FFmpeg is required for encoding, probing, media decoding, audio mixing, and timeline compositing. Skia is not required in 0.6. The project supports the current active Node.js LTS; Linux is the reference and CI platform, while macOS and Windows remain best-effort until covered by real rendering tests.
 
 ## 7. Official agent skill
 
-Frameyard ships an optional, host-neutral agent skill with a `SKILL.md`, supporting references, and thin scripts. The npm package includes skill files matching its API version, and documentation explains host-specific installation or referencing. 0.4 does not add a skill-installer command.
+Frameyard ships an optional, host-neutral agent skill with a `SKILL.md`, supporting references, and thin scripts. The npm package includes skill files matching its API version, and documentation explains host-specific installation or referencing. 0.6 does not add a skill-installer command.
 
 The skill:
 
-- Uses the normal project format, HTML composition contract, optional controller, and CLI available to humans.
+- Uses the normal project format, HTML composition contract, JSON timeline contract, optional controller, and CLI available to humans.
 - Does not introduce an agent-only runtime, MCP server, hosted control plane, or separate rendering API.
 - Triggers for Frameyard authoring and operation rather than claiming to be a general-purpose media-generation skill.
 - Uses Frameyard's audio wrappers instead of invoking `qwen-tts` or `whisper-cli` directly.
-- Includes references for deterministic HTML authoring, bindings, animation scheduling, controller rules, static validation, structured diagnostics, common authoring mistakes, audio assets, subtitle and karaoke recipes, JSON command results, and failure recovery.
+- Includes references for deterministic HTML authoring, timeline JSON authoring, tracks, clips, source ranges, props, media, nested timelines, static transforms, audio, validation, diagnostics, common mistakes, subtitle and karaoke recipes, JSON command results, long-form worker/scratch settings, and failure recovery.
 - Invokes speech synthesis or transcription only when requested or explicitly approved as part of the brief.
 - Does not overwrite or regenerate existing generated assets without permission.
 - Runs `check`, inspects representative `still` images when the host supports image viewing, and discloses when visual inspection was impossible.
 - Does not claim success when checks fail or rendering is partial.
 
-A successful agent workflow preserves and reports the source project, props, generated assets, provenance manifest, validation results, and final video path. The user must be able to inspect, modify, and reproduce the result rather than receiving only an opaque MP4.
+A successful agent workflow preserves and reports the source project, timeline document, props, generated assets, provenance manifest, validation results, and final video path. The user must be able to inspect, modify, and reproduce the result rather than receiving only an opaque MP4.
 
 ## 8. Local speech synthesis
 
 The `tts` command integrates [`qwentts.cpp`](https://github.com/ServeurpersoCom/qwentts.cpp) through its external `qwen-tts` executable. It does not link the native library or require the upstream server.
 
-0.4 supports CustomVoice mode only:
+0.6 supports CustomVoice mode only:
 
 - A named speaker is required.
 - An optional style instruction and language may be supplied.
@@ -175,7 +217,7 @@ The canonical artifact is a versioned Frameyard JSON document containing:
 - Timed segments
 - Optional word timings
 
-All timestamps are integer milliseconds relative to the source audio, not composition frame numbers. The controller converts them using the audio clip's start frame, source trim, and composition frame rate.
+All timestamps are integer milliseconds relative to the source audio, not timeline or composition frame numbers. A controller converts them using the audio clip's start frame, source trim, and the composition frame rate.
 
 Compositions declare required transcript assets with `<fy-transcript>` elements:
 
@@ -187,11 +229,11 @@ Compositions declare required transcript assets with `<fy-transcript>` elements:
 
 SRT and VTT are derived exports, not the canonical contract. Word-level karaoke requires actual word timings and fails validation when they are unavailable. Frameyard never fabricates word timings by interpolating segment timestamps and never executes Whisper's generated bash/FFmpeg karaoke script.
 
-The official skill documents seek-safe subtitle and karaoke recipes. Compositions retain responsibility for visual presentation; 0.4 has no built-in subtitle renderer and does not automatically burn captions into videos. Karaoke presentation uses the optional controller.
+The official skill documents seek-safe subtitle and karaoke recipes. Compositions retain responsibility for visual presentation; 0.6 has no built-in subtitle renderer and does not automatically burn captions into videos. Karaoke presentation uses the optional controller.
 
 ## 10. Generated assets and inference operation
 
-Speech synthesis and transcription are separate asset-production operations that complete before composition loading and rendering. Their outputs are ordinary local project assets. Once those assets are frozen, ordinary authoring, preview, checking, and rendering require neither AI executable nor model.
+Speech synthesis and transcription are separate asset-production operations that complete before a timeline or composition is loaded or rendered. Their outputs are ordinary local project assets. Once those assets are frozen, ordinary authoring, preview, checking, and rendering require neither AI executable nor model.
 
 Both integrations are optional and local-only, with no cloud fallback. Network activity needed to acquire tools or models is a separate explicit setup operation, never a hidden side effect of generation or rendering.
 
@@ -214,19 +256,21 @@ Tests prioritize public behavior and end-to-end evidence over implementation-onl
 
 - HTML compositions validate metadata, default props, bindings, conditionals, repeats, and controller declarations, including inline/external parity and the exactly-one-controller rule.
 - Static composition validation catches metadata, root/controller count, binding, animation, script, element, and asset-reference errors before Chromium.
-- Diagnostics carry stable codes and source locations; JSON results expose the composition entry, controller declaration, and top-level diagnostics.
-- Sequential, individual, random-order, and parallel frame renders produce equivalent images.
-- Determinism checks compare captured pixels or stills; encoded MP4 bytes are not a reproducibility oracle.
-- Preview and renderer use the same composition document and host runtime.
-- CSS, controller JavaScript, SVG, and Canvas animations seek correctly.
+- Timeline JSON validation catches schema, track, clip, reference, cycle, frame-rate, overlap, bounds, media, transform, audio, and props errors before Chromium.
+- Diagnostics carry stable codes and source locations or JSON paths; JSON results expose the composition or timeline entry, clips, tracks, and top-level diagnostics.
+- Sequential, individual, random-order, and parallel composition renders produce equivalent images; timeline segment workers produce ordered output and identical captured composition frames.
+- Determinism checks compare captured composition pixels or stills. The final H.264 bytes and the FFmpeg composite versus Chromium preview are not reproducibility oracles.
+- Preview and renderer use the same composition document and host behavior; timeline preview uses the same active clip hosts.
+- CSS, controller JavaScript, SVG, Canvas, WebGL, embedded video, and audio frames seek correctly.
 - Frame-scheduled `data-fy-anim-*` animations and frame-state CSS variables produce the expected frames.
-- Embedded video and audio frames seek deterministically, including under parallel capture, and their audio appears in the offline mix.
-- WebGL canvases produce deterministic stills and rendered frames under the pinned software renderer.
-- Executable scripts other than the inline or external controller, inline event handlers, `javascript:` URLs, and unsupported visual elements are rejected.
-- Asset and runtime failures produce nonzero CLI exits.
-- Rendered MP4 files have the expected dimensions, frame count, duration, and audio stream.
-- Golden-frame tests cover representative visual behavior.
-- `check` exercises every frame, and `still` follows the production capture path.
+- Timeline capture preserves alpha for transparent compositions; FFV1 intermediates decode pixel-identically to the captured frames.
+- Timeline segment encoding, ordered concatenation, global audio mix, and scratch cleanup produce the expected MP4 dimensions, frame count, duration, and audio stream.
+- Nested timelines preserve group opacity and transform semantics; cycles fail static validation.
+- Media clips trim and fit deterministically; images require durations; source overruns fail.
+- Props overrides for timeline composition clips merge over clip props and are reflected in the render.
+- `check` exercises every frame; `still` follows the clip capture and timeline composite path.
+- The 10-second and 60-second long-form benchmarks record capture, FFV1, composite, encode, disk, worker, and audio numbers. Continuous three-layer runs measured 52.4 s and 58.5 s for 60 s of timeline and are documented as worst-case (~1× real time). The representative 50/50, 60/30/10 occupancy gate measured 29.39 s for 60 s with four workers and a 1080p media fast path, extrapolating to about 29.4 minutes for 60 minutes with exact 1800-frame output. The full 60-minute composition-only gate rendered 108,000 frames in 35.35 minutes with exact output and clean scratch, documenting composition-heavy performance above 30 minutes.
+- Golden-frame tests cover representative composition and timeline visual behavior.
 - Normal CI uses fake `qwen-tts` and `whisper-cli` executables and small fixtures to verify arguments, artifacts, manifests, cancellation, and errors without model downloads or GPU hardware.
 - Real-model integration tests are explicitly opt-in and use documented upstream revisions.
 - Agent-workflow evaluations verify source preservation, provenance, validation, still inspection behavior, failure reporting, and final renders.
@@ -235,7 +279,14 @@ Tests prioritize public behavior and end-to-end evidence over implementation-onl
 
 These are scope boundaries, not permanent rejections:
 
-- Visual editing, hosted rendering, collaboration, and asset marketplaces
+- Transitions, effects, keyframes, blend modes, masks, and audio fades
+- Full visual timeline editing, drag-and-drop, clip handles, and undo/redo
+- Timeline-level variables and prop overrides for compositions inside nested timelines
+- Track solo and video-track visibility
+- Per-timeline formats and frame-rate conversion/retiming
+- Persistent clip caches; the segmented renderer cleans its intermediates
+- Hardware encoders, GPU compositing, and non-H.264 output
+- Hosted rendering, collaboration, and asset marketplaces
 - Cloud generative-AI services, text-to-video generation, and image generation
 - `qwentts.cpp` reference-audio voice cloning and VoiceDesign
 - Automatic long-text chunking, streaming speech synthesis, and alternative generated-audio formats
@@ -247,17 +298,16 @@ These are scope boundaries, not permanent rejections:
 - Agent-only runtimes, MCP servers, hosted agent control planes, and host-specific skill installers
 - React or other UI-framework adapters in the core
 - Public plugin APIs and a broad integration ecosystem
-- Dynamic composition metadata derived from props
+- Dynamic composition or timeline metadata derived from props
 - Live streams, remote media, and other visual sources that cannot be deterministically seeked
 - Dynamic per-frame asset loading or AI inference
-- Frame caches
 - Batch still capture; agents can run `still` once per representative frame
 - Additional final-video formats, general image-sequence output, and arbitrary FFmpeg argument passthrough
 - Built-in TypeScript transpilation and npm dependency optimization
 - Rendering untrusted compositions
 - Bit-identical cross-platform encoded output or AI regeneration
-- A minimum production rendering-speed guarantee
+- A minimum production rendering-speed guarantee for compositions or timelines
 
 ## 13. Release posture
 
-Frameyard is MIT-licensed. Initial releases use `0.x` versioning, and composition, configuration, CLI, JSON-result, transcript, and skill-contract changes are documented explicitly until those contracts stabilize. 0.4.0 is a breaking authoring-model release: HTML documents replace JavaScript composition modules, and the `fy` namespace replaces `data-frameyard-*` authoring attributes. See [ADR 0007](./adr/0007-make-html-documents-the-composition-unit.md) and the [0.4 migration guide](./migration-0.4.md).
+Frameyard is MIT-licensed. Initial releases use `0.x` versioning, and composition, timeline, configuration, CLI, JSON-result, transcript, and skill-contract changes are documented explicitly until those contracts stabilize. 0.6.0 adds timelines, project formats, segmented timeline rendering, and a timeline-first scaffold without breaking the existing HTML composition authoring contract. See [ADR 0010](./adr/0010-make-timelines-the-render-unit.md), [ADR 0011](./adr/0011-compose-timeline-frames-in-a-live-browser-host.md), and [ADR 0012](./adr/0012-render-timelines-with-segmented-lossless-clip-intermediates.md).

package/examples/demo/README.md

@@ -10,15 +10,22 @@ A 30-second, 1920×1080 HTML composition that exercises the current core:
 - frame-state CSS variables for the progress bar
 
 The composition lives in `compositions/demo.html`; the optional controller is
-`compositions/demo.controller.js`.
+`compositions/demo.controller.js`. `timelines/demo.timeline.json` wraps the
+composition as a clip in a 1920×1080, 30 fps timeline.
 
 From the repository root:
 
 ```sh
 pnpm build
-node dist/cli.js check demo --project examples/demo
-node dist/cli.js still demo --frame 240 --project examples/demo --out demo-still.png
-node dist/cli.js render demo --project examples/demo --props examples/demo/props/demo.json --out demo.mp4
+node dist/cli.js check demo-timeline --project examples/demo
+node dist/cli.js still demo-timeline --frame 240 --project examples/demo --out demo-still.png
+node dist/cli.js render demo-timeline --project examples/demo --props examples/demo/props/demo.json
+```
+
+The composition remains renderable on its own:
+
+```sh
+node dist/cli.js render demo --project examples/demo --props examples/demo/props/demo.json
 ```
 
 Start the interactive preview with:

package/examples/demo/timelines/demo.timeline.json

@@ -0,0 +1,12 @@
+{
+  "version": 1,
+  "tracks": [{ "id": "v1", "kind": "video" }],
+  "clips": [
+    {
+      "id": "demo",
+      "track": "v1",
+      "source": { "kind": "composition", "id": "demo" },
+      "start": 0
+    }
+  ]
+}

package/examples/demo/video.config.js

@@ -1,5 +1,9 @@
 export default {
+  format: { width: 1920, height: 1080, fps: 30 },
   compositions: {
     demo: "./compositions/demo.html",
   },
+  timelines: {
+    "demo-timeline": "./timelines/demo.timeline.json",
+  },
 };