visualfries 0.1.10120 → 0.3.2

package/docs/BROWSER_EXPORT_SPIKE.md

@@ -0,0 +1,55 @@
+# Browser WebGL + MediaBunny export spike
+
+This spike adds a separate deterministic browser export path. Existing interactive preview and local/cloud FFmpeg rendering are unchanged.
+
+MediaBunny is pinned to `1.49.0` for this spike. The newer release available during implementation was less than 14 days old.
+
+## Architecture and next slices
+
+Current scene transitions are schema data only; existing agent transitions are full-frame overlay components. This milestone therefore keeps normal preview untouched and proves a separate browser export boundary: server-mode Pixi scene canvases → curated WebGL transition canvas → MediaBunny encode/mux. Next reviewable slices are: wire the curated adapter into a frame-index transition manager at the renderer's pre-present seam, adapt MediaBunny decoded samples to the deterministic media provider for real browser video inputs, then add an OPFS/File System Access target for long exports. None requires replacing the FFmpeg/cf-worker fallback.
+
+## What it proves
+
+- VisualFries/Pixi renders two animated 1080x1920 scenes at exact `frameIndex / 30` timestamps.
+- A curated gl-transitions-compatible WebGL adapter composites `from`, `to`, `progress`, `ratio`, and whitelisted parameters. It ships `fade` and `radial-wipe`; shader compile/link failure falls back to fade.
+- The final WebGL canvas is fed directly to MediaBunny `CanvasSource`. No PNG/JPEG serialization and no websocket frame transport occurs.
+- MediaBunny encodes AVC/H.264 video and AAC audio and muxes MP4. The in-browser audio mix starts at timestamp zero.
+- Every `CanvasSource.add()` is awaited, propagating encoder/writer backpressure. The 10-second demo uses `BufferTarget`; larger integrations can pass a positional `StreamTarget` writable that honors each chunk's `position`.
+
+## Run interactively
+
+```bash
+pnpm dev
+```
+
+Open `http://localhost:5173/browser-export-spike`, confirm the AVC + AAC capability report, then click **Render 10-second MP4**.
+
+## Produce the checked artifact
+
+```bash
+node scripts/render-browser-export-spike.mjs
+```
+
+Default output: `artifacts/browser-export-spike/demo.mp4`.
+
+Inspect it:
+
+```bash
+ffprobe -v error -show_streams -show_format artifacts/browser-export-spike/demo.mp4
+ffmpeg -i artifacts/browser-export-spike/demo.mp4 -vf "select='eq(n,119)+eq(n,120)+eq(n,135)+eq(n,149)+eq(n,150)'" -vsync 0 artifacts/browser-export-spike/frame-%02d.png
+```
+
+Expected: 300 frames, 10 seconds, 1080x1920 AVC video, AAC stereo audio with `start_time=0`, no blank transition samples, and zero shader fallbacks on a compatible WebGL implementation.
+
+## Verified artifact (2026-07-11)
+
+- H.264, 1080x1920, 30fps, 300 decoded frames, video duration `10.000000`
+- AAC stereo, 48kHz, audio `start_time=0.000000` (AAC padding extends its reported duration to `10.069333`)
+- First 100ms audio RMS approximately `-34.6 dB`, confirming immediate non-silent audio
+- Transition-window sampled average luma ranged `38.49..50.77`; visual frame inspection found no blank/white frame
+- Four motion samples produced four different decoded-frame hashes
+- Runtime shader fallback count: `0`
+
+## Capability and fallback contract
+
+The browser path probes the exact requested dimensions and AVC/AAC configuration before work starts. It never silently changes codecs. If unavailable, the structured result names `deterministic-local-ffmpeg`, preserving the existing VisualFries/cf-worker fallback. This milestone does not promise mobile support or arbitrary user GLSL.

package/docs/DETERMINISTIC_RENDERER.md

@@ -0,0 +1,83 @@
+# VisualFries Deterministic Renderer
+
+VisualFries should own final rendering as a local, first-class capability.
+
+## Decision
+
+Final output for scenes containing `VIDEO` or `GIF` media must not rely on native browser media seek. Browser media seek is acceptable for preview and QA only.
+
+The local deterministic renderer is the source-of-truth path:
+
+```text
+scene.json
+  -> collect VIDEO/GIF requirements
+  -> predecode media to exact frame images/manifests
+  -> browser compositor receives exact frame image for each scene frame
+  -> encode frames with ffmpeg
+  -> mix/mux audio with ffmpeg
+  -> mp4
+```
+
+ContentFries cloud workers should eventually become hosted execution wrappers around this VisualFries renderer. They own queues, uploads, progress, retries, chunking, URL signing, auth-specific asset resolution, and deployment concerns, but the render semantics should live in VisualFries.
+
+## Preview Contract
+
+Preview means cheap and fast:
+
+- sampled QA frames,
+- low-resolution or short-range renders,
+- layout inspection,
+- rough browser media playback when explicitly requested.
+
+Preview is not a slower copy of final render. If a preview takes longer than final output, it is not a useful preview.
+
+## Current Migration State
+
+This package now exposes agent-safe render planning:
+
+```ts
+import { resolveAgentRenderPlan, requiresDeterministicRender } from 'visualfries/agent';
+
+const plan = resolveAgentRenderPlan(scene, { mode: 'final' });
+```
+
+The CLI blocks final `VIDEO`/`GIF` renders unless the caller explicitly chooses preview or the legacy escape hatch:
+
+```bash
+visualfries render scene.json --render-mode preview --output preview.mp4
+visualfries render scene.json --allow-browser-media-final --output unsafe-experiment.mp4
+```
+
+The package also includes the first local predecode core:
+
+```ts
+import { prepareLocalDeterministicMedia } from 'visualfries/agent';
+```
+
+`prepareLocalDeterministicMedia()` resolves `VIDEO`/`GIF` components, computes active frame windows, extracts exact image sequences with local `ffmpeg`, and returns a `visualfries-provider-predecoded` frame manifest. The CLI can inject that manifest into the render page as a deterministic media provider.
+
+The CLI also supports image-pipe final encoding:
+
+```bash
+visualfries render scene.json --stream-encode --image-format jpg --output out.mp4
+```
+
+With `--stream-encode`, rendered frames are written directly into local `ffmpeg` through `image2pipe` instead of first creating a temporary frame sequence. If audio is present, the CLI writes a silent video first and then muxes audio with `-c:v copy`.
+
+Audio mixing is local-first as well. The CLI now plans audio from active `VIDEO`/`AUDIO` components, `audioTracks`, and `settings.audio`, prepares each source to 48k stereo FLAC, applies timeline delay/volume, mixes with `amix`, trims to the rendered range, and muxes the mixed track into the final MP4.
+
+The same implementation is available as an importable local render boundary:
+
+```ts
+import { renderSceneLocally } from 'visualfries/agent';
+```
+
+Hosted workers should call this boundary instead of reimplementing frame extraction, browser composition, stream encoding, or audio mixing. The scene passed into this boundary must already contain public, local, or otherwise render-accessible media URLs. URL signing, signed URL refresh, and private asset authorization are wrapper responsibilities, not VisualFries responsibilities. VisualFries must not know how to authenticate ContentFries private assets; if media URLs are not render-accessible, it should fail with useful diagnostics. The package must stay light: `ffmpeg`, Playwright/Chromium, and hosted queue/upload infrastructure are runtime dependencies of the caller, not bundled VisualFries assets.
+
+The local renderer owns trim-aware final output. When `scene.settings.trimZones` intersects the requested contiguous frame window, VisualFries renders only the kept frame ranges and concatenates them through the stream encoder. Audio mixing uses the same kept scene-time ranges so video duration and audio duration stay aligned.
+
+The same boundary also supports exact single-frame output with `framesOnly: true` and explicit server renderer selection through `serverRendererMode: "canvas" | "webgl"`. Hosted wrappers should pass these options through instead of maintaining separate single-frame or renderer-mode semantics.
+
+Frame image quality accepts either browser form (`0..1`) or percent form (`0..100`) and is normalized before the browser render starts. This keeps wrappers from failing later with opaque frame extraction errors when they pass values such as `88`.
+
+The next migration slice is making the ContentFries worker call the VisualFries core instead of carrying its own duplicate render semantics.

package/docs/NEXT_RELEASE.md

@@ -0,0 +1,28 @@
+# Next Release Notes (Unreleased)
+
+Recommended semver: **0.2.0**. The schemas remain additive/loadable, but confirmed ignored behavior now evaluates and can change pixels. This deserves a pre-1.0 minor release rather than pretending it is a patch-only change.
+
+## Added
+
+- Typed component capability registry exposed through agent exports and `catalog --component <TYPE> --capabilities --json`.
+- Warning-by-default runtime-support diagnostics and `validate --strict-runtime-support` for agents/CI.
+- Runtime-backed `explain --component <id> --frame <n>` computed target state.
+- Shared deterministic Pixi animation target for IMAGE, VIDEO, GIF, regular/progress SHAPE, COLOR, and GRADIENT.
+- Real COLOR and GRADIENT canvas-to-Pixi render paths.
+- Native TEXT pixel padding, active word/line scale, and solid highlight palette cycling.
+
+## Behavior fixes / migration notes
+
+- Pixi animation x/y now mean placement-relative translation offsets, matching TEXT and existing swipe/drift helpers. Scenes authored against the short-lived absolute-center IMAGE bridge should subtract the component center from old x/y values.
+- `addAgentTransitions` now emits native SHAPE rectangles instead of background-only empty TEXT.
+- `visible: false` is preserved by component state and shared visual wrappers.
+- Active word wins when active word and active line are both enabled; runtime validation reports the conflict.
+- `clipColor` is diagnosed before permissive parsing can silently discard it. Strict runtime validation rejects it.
+
+## Still limited
+
+- Transform origin is fixed-center for Pixi visuals and not authorable across renderers.
+- Client HTML TEXT stacking versus final Pixi TEXT remains an architectural parity gap.
+- Native GIF timing and VIDEO/GIF loop/playback-rate parity are not guarantees yet.
+- Gradient highlight palettes, clipColor semantics, generic effects, and advanced setup/anchor behavior remain unsupported.
+- Radial gradient shape/position uses a centered max-radius fallback.

package/docs/PROPOSED_VISUALFRIES_SKILL.md

@@ -0,0 +1,68 @@
+# Proposed VisualFries Skill Update (Review Only)
+
+This is a repo-local proposal. It does not modify `/Users/lukasgregor/.agents/skills/visualfries/SKILL.md`.
+
+Apply these sections to the installed skill only after review/approval.
+
+## Replace the agent rule
+
+```text
+inspect inputs and actual runtime capability
+-> choose native semantic components
+-> author scene JSON
+-> validate + inspect structure
+-> frame-QA entry, settled, exit, and deterministic reseeks
+-> final deterministic render
+```
+
+## Add: Semantic component policy
+
+- Visible/editable typography defaults to native `TEXT`, including metrics, labels, badges, verdicts, cards, hooks, and CTAs.
+- Transcript-timed typography uses `SUBTITLES`.
+- A TEXT badge owns its background/gradient/radius/alignment/outline/shadow/animation when those treatments are runtime-supported.
+- Do not create a separate `SHAPE` merely as an ordinary text background. SHAPE is independent geometry, decoration, progress, mask-like layout, or transition content.
+- Transition covers use native animated `SHAPE`; do not use empty TEXT as geometry.
+- Use `IMAGE` for real image assets: photos, screenshots, logos, illustrations, or supplied raster/vector artwork.
+- Raster/SVG typography is an explicit fallback. Record which native limitation requires it and frame-QA it.
+- Production-plan overlays default to native TEXT. Use `renderAs: "SVG"` only for a documented compatibility fallback.
+
+## Add: Runtime truth
+
+- Validation is not rendering proof.
+- Confirmed visual targets: `TEXT`, `SUBTITLES`, `IMAGE`, `VIDEO`, `GIF`, `SHAPE`, `COLOR`, and `GRADIENT`. AUDIO is nonvisual.
+- Pixi animation `x`/`y` are placement-relative offsets. Pivot is fixed center. Supported properties: x, y, opacity, rotation (degrees), scale, scaleX, scaleY.
+- `clipColor` is unsupported and strict runtime validation rejects it.
+- `appearance.text.padding` supports pixels; absent element-target padding preserves the legacy `0.22em` fallback.
+- Active word/line scale and solid `highlightColors` cycling work. Active word wins when both word and line modes are enabled.
+- Gradient text should use wrapper-target background when a card background is also needed.
+- Browser HTML TEXT and final Pixi-rasterized TEXT can stack differently; verify risky cross-type compositions in final mode.
+
+## Replace QA guidance
+
+For every animated component:
+
+1. Render a pre-entry frame.
+2. Render first-active or mid-entry frame.
+3. Render settled frame.
+4. Render exit frame when applicable.
+5. Seek away and back to the same frame; compare state/pixels.
+
+Uniform three-frame QA is not enough for short or late events. Prefer exact production-plan `qa.framesAt` or explicit `--from-frame`/`--to-frame` windows.
+
+```bash
+visualfries doctor --json
+visualfries catalog --component TEXT --capabilities --json
+visualfries validate scene.json --strict-runtime-support
+visualfries inspect scene.json --json
+visualfries explain scene.json --component <id> --frame <n> --json
+visualfries parity scene.json --frames <csv> --output qa/parity --json
+visualfries render scene.json --frames-only --from-frame 5 --to-frame 20 --output qa/entry
+visualfries render scene.json --frames-only --from-frame 35 --to-frame 36 --output qa/settled
+visualfries render scene.json --stream-encode --output out.mp4
+```
+
+## Replace the current limitation section
+
+- Cookbook depth is not the only remaining gap.
+- Open engine gaps include authorable transform origin, browser/final TEXT stacking parity, clipColor semantics, gradient highlight palettes, generic effects, VIDEO/GIF playback parity, advanced animation anchors/setup, and calibrated preview/final parity metrics.
+- Link the installed skill to `docs/AUTHORING_BEST_PRACTICES.md` in the package/repository.

package/docs/examples/agent-cues.json

@@ -0,0 +1,49 @@
+{
+  "broll": [
+    {
+      "url": "./assets/profile-scroll.mp4",
+      "start": 0.2,
+      "end": 1.4,
+      "type": "VIDEO",
+      "fade": 0.18
+    },
+    {
+      "url": "./assets/chart.png",
+      "start": 1.4,
+      "end": 2.6,
+      "type": "IMAGE",
+      "motion": "slow-zoom-in"
+    }
+  ],
+  "overlays": [
+    {
+      "text": "LOVE THIS \ud83d\ude0d",
+      "start": 0.4,
+      "end": 1.1,
+      "style": "hook-punch"
+    },
+    {
+      "text": "NECK \ud83e\udd2f",
+      "start": 1.1,
+      "end": 1.7,
+      "style": "shock-word"
+    },
+    {
+      "text": "your move",
+      "start": 2.1,
+      "end": 2.7,
+      "style": "cta-card"
+    }
+  ],
+  "transitions": [
+    {
+      "time": 1.4,
+      "style": "dip-to-black"
+    },
+    {
+      "time": 2.3,
+      "style": "swipe-left",
+      "color": "#04483D"
+    }
+  ]
+}

package/docs/examples/agent-transcript.json

@@ -0,0 +1,23 @@
+{
+	"segments": [
+		{
+			"id": "s1",
+			"text": "Your content is free therapy for people who will never buy.",
+			"start": 0,
+			"end": 2.8,
+			"words": [
+				{ "text": "Your", "start": 0, "end": 0.2 },
+				{ "text": "content", "start": 0.2, "end": 0.55 },
+				{ "text": "is", "start": 0.55, "end": 0.7 },
+				{ "text": "free", "start": 0.7, "end": 1.05 },
+				{ "text": "therapy", "start": 1.05, "end": 1.5 },
+				{ "text": "for", "start": 1.5, "end": 1.68 },
+				{ "text": "people", "start": 1.68, "end": 2.0 },
+				{ "text": "who", "start": 2.0, "end": 2.15 },
+				{ "text": "will", "start": 2.15, "end": 2.32 },
+				{ "text": "never", "start": 2.32, "end": 2.55 },
+				{ "text": "buy.", "start": 2.55, "end": 2.8 }
+			]
+		}
+	]
+}

package/docs/examples/agent-transcript.srt

@@ -0,0 +1,7 @@
+1
+00:00:00,000 --> 00:00:01,200
+The best content asset is not always a post.
+
+2
+00:00:01,200 --> 00:00:02,800
+Sometimes it is the boring problem behind it.

package/docs/examples/agent-transcript.vtt

@@ -0,0 +1,7 @@
+WEBVTT
+
+00:00:00.000 --> 00:00:01.200
+The best content asset is not always a post.
+
+00:00:01.200 --> 00:00:02.800
+Sometimes it is the boring problem behind it.

package/docs/examples/image-animation.json

@@ -0,0 +1,87 @@
+{
+	"id": "image-animation-runtime-proof",
+	"version": "2.0",
+	"name": "IMAGE animation runtime proof",
+	"settings": {
+		"width": 1080,
+		"height": 1920,
+		"duration": 3,
+		"fps": 30,
+		"backgroundColor": "#111827"
+	},
+	"assets": [
+		{
+			"id": "proof-card-asset",
+			"type": "IMAGE",
+			"url": "data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='720' height='480'%3E%3Crect width='720' height='480' rx='48' fill='%23F5D547'/%3E%3Ccircle cx='360' cy='240' r='96' fill='%23101820'/%3E%3C/svg%3E"
+		}
+	],
+	"layers": [
+		{
+			"id": "image-layer",
+			"name": "Animated image",
+			"order": 10,
+			"visible": true,
+			"muted": false,
+			"components": [
+				{
+					"id": "proof-card",
+					"name": "Animated proof card asset",
+					"type": "IMAGE",
+					"source": {
+						"assetId": "proof-card-asset",
+						"url": "data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='720' height='480'%3E%3Crect width='720' height='480' rx='48' fill='%23F5D547'/%3E%3Ccircle cx='360' cy='240' r='96' fill='%23101820'/%3E%3C/svg%3E"
+					},
+					"timeline": { "startAt": 0.5, "endAt": 3 },
+					"visible": true,
+					"order": 0,
+					"appearance": {
+						"x": 180,
+						"y": 720,
+						"width": 720,
+						"height": 480,
+						"opacity": 0.9,
+						"rotation": 3,
+						"scaleX": 1,
+						"scaleY": 1
+					},
+					"animations": {
+						"enabled": true,
+						"list": [
+							{
+								"id": "proof-card-enter",
+								"name": "Image enter",
+								"startAt": 0.15,
+								"enabled": true,
+								"animation": {
+									"id": "proof-card-enter-preset",
+									"duration": 0.6,
+									"timeline": [
+										{
+											"tweens": [
+												{
+													"method": "from",
+													"vars": {
+														"duration": 0.6,
+														"x": -360,
+														"opacity": 0,
+														"scale": 0.82,
+														"rotation": -10,
+														"ease": "power3.out"
+													}
+												}
+											]
+										}
+									]
+								}
+							}
+						]
+					},
+					"effects": { "enabled": true, "map": {} }
+				}
+			]
+		}
+	],
+	"transitions": [],
+	"audioTracks": []
+}

package/docs/examples/native-text-metric-badge.json

@@ -0,0 +1,109 @@
+{
+	"id": "native-text-metric-badge",
+	"version": "2.0",
+	"name": "Native TEXT metric badge",
+	"settings": {
+		"width": 1080,
+		"height": 1920,
+		"duration": 3,
+		"fps": 30,
+		"backgroundColor": "#151A22"
+	},
+	"assets": [],
+	"layers": [
+		{
+			"id": "metric-layer",
+			"name": "Semantic metric typography",
+			"order": 10,
+			"visible": true,
+			"muted": false,
+			"components": [
+				{
+					"id": "views-badge",
+					"name": "881 VIEWS",
+					"type": "TEXT",
+					"text": "881 VIEWS",
+					"timeline": { "startAt": 0.2, "endAt": 3 },
+					"visible": true,
+					"order": 0,
+					"appearance": {
+						"x": 90,
+						"y": 820,
+						"width": 900,
+						"height": 220,
+						"opacity": 1,
+						"rotation": -2,
+						"scaleX": 1,
+						"scaleY": 1,
+						"horizontalAlign": "center",
+						"verticalAlign": "center",
+						"background": {
+							"enabled": true,
+							"color": "#F5D547",
+							"target": "wrapper",
+							"radius": 32
+						},
+						"text": {
+							"fontFamily": "Montserrat",
+							"fontSize": { "value": 116, "unit": "px" },
+							"fontWeight": "900",
+							"lineHeight": { "value": 1, "unit": "em" },
+							"color": "#101820",
+							"textAlign": "center",
+							"textTransform": "uppercase",
+							"outline": {
+								"enabled": true,
+								"color": "#FFFFFF",
+								"size": 2,
+								"opacity": 1,
+								"style": "solid"
+							},
+							"shadow": {
+								"enabled": true,
+								"color": "#000000",
+								"blur": 16,
+								"offsetX": 0,
+								"offsetY": 8,
+								"opacity": 0.5
+							}
+						}
+					},
+					"animations": {
+						"enabled": true,
+						"list": [
+							{
+								"id": "views-badge-enter",
+								"name": "Badge enter",
+								"startAt": 0.1,
+								"enabled": true,
+								"animation": {
+									"id": "views-badge-enter-preset",
+									"duration": 0.5,
+									"timeline": [
+										{
+											"tweens": [
+												{
+													"method": "from",
+													"vars": {
+														"duration": 0.5,
+														"opacity": 0,
+														"scale": 0.82,
+														"y": 70,
+														"ease": "back.out(1.8)"
+													}
+												}
+											]
+										}
+									]
+								}
+							}
+						]
+					},
+					"effects": { "enabled": true, "map": {} }
+				}
+			]
+		}
+	],
+	"transitions": [],
+	"audioTracks": []
+}