@omnimedia/omnitool 1.1.0-1 → 1.1.0-100

package/README.md

@@ -1,120 +1,604 @@
 
 # 🚧 Work In Progress
 
-> **Note:** Omni Tools is under development. Expect breaking changes, evolving APIs, and experimental features.
+> **Note:** Omnitool is under development. Expect breaking changes, evolving APIs, and experimental features.
 
 ---
 
-# 🎬 Omnitool
+## ✅ What this library is
 
-> Code-first video editing toolkit behind [Omniclip](https://omniclip.app) — build timelines, render videos, and automate workflows.
+- API for building and rendering timelines in the browser
+- Uses WebCodecs, Web Workers, and the File System Access API for export
+- usage is currently JavaScript/TypeScript only, cli soon
 
----
+## 🚀 Install
 
-## 🧱 Modular by Design
+```bash
+npm i @omnimedia/omnitool
+```
 
-Omni Tools is a collection of composable utilities for working with Omniclip timelines — via code, JSON, or CLI.
+## 📦 Quick Start
 
-- ✅ Define timelines in JSON or TypeScript
-- ✅ Automate rendering with CLI tools
-- ✅ Ideal for scripting, CI/CD, and AI-generated workflows
+#### Declaring the timeline
 
----
+```ts
+import {Driver, Omni, Datafile} from "@omnimedia/omnitool"
+
+const driver = await Driver.setup()
+const omni = new Omni(driver)
+
+const {clip} = await omni.load({
+	clip: Datafile.make(file) // file is a File or Blob
+})
+
+const timeline = omni.timeline(o => {
+	const caption = o.text("Hello world", {
+		duration: 1500,
+		styles: {fill: "white", fontSize: 48}
+	})
+	const xfade = o.transition.fade(500)
+	const softened = o.filter.blur.make({strength: 8, quality: 4})
+
+	const visual = o.sequence(
+		o.stack(
+			o.video(clip, {start: 0, duration: 3000, filterIds: [softened.id]}),
+			caption
+		),
+		xfade,
+		o.video(clip, {start: 5000, duration: 2500}),
+		o.gap(400)
+	)
+
+	return o.stack(
+		visual,
+		o.audio(clip, {start: 5000, duration: 2500})
+	)
+})
+```
 
-## 🚀 Quick Start
+Declarative helper style (no explicit `o` in timeline declarations):
 
-### Install
+```ts
+import {
+	Driver, Omni, Datafile,
+	timeline, sequence, stack, video, audio, text, gap, transition, filter
+} from "@omnimedia/omnitool"
+
+const driver = await Driver.setup()
+const omni = new Omni(driver)
+const {clip} = await omni.load({clip: Datafile.make(file)})
+
+const timeline = timeline(
+	stack(
+		sequence(
+			stack(
+				filter.blur(
+					video(clip, {start: 0, duration: 3000}),
+					{strength: 8, quality: 4}
+				),
+				text("Hello world", {duration: 1500}),
+			),
+			transition.fade(500),
+			video(clip, {start: 5000, duration: 2500}),
+			gap(400),
+		),
+		audio(clip, {start: 5000, duration: 2500})
+	)
+)
+```
 
-```bash
-npm install @omni/tools
+## 💬 Captions
+
+Captions render transcript data as timed, styled text.
+The transcript can come from anywhere, as long as it follows the structure.
+
+```ts
+const transcript = {
+	text: "Hello world. This is a caption.",
+	chunks: [
+		{text: "Hello", timestamp: [0, 0.4]},
+		{text: "world.", timestamp: [0.4, 1.2]},
+		{text: "This", timestamp: [1.3, 1.6]},
+		{text: "is", timestamp: [1.6, 1.8]},
+		{text: "a", timestamp: [1.8, 1.9]},
+		{text: "caption.", timestamp: [1.9, 2.6]},
+	],
+}
+
+const timeline = omni.timeline(o => {
+	const video = o.video(clip, {duration: 3000})
+	return o.captions(video, transcript)
+})
 ```
 
----
+Use a caption preset to pick a built-in caption style:
+
+```ts
+const timeline = omni.timeline(o => {
+	const video = o.video(clip, {duration: 3000})
+	return o.captions.presets.default(video, transcript)
+})
+```
+
+or do your own styled captions:
+
+```ts
+const timeline = omni.timeline(o => {
+	const video = o.video(clip, {duration: 3000})
+	return o.captions(video, transcript, {
+		styles: {
+			fontFamily: "Inter",
+			fontSize: 64,
+			fill: "#fff7d6",
+			stroke: {color: "#111111", width: 8},
+			align: "center",
+		},
+	})
+})
+```
+
+Use omnitool's built in speech-to-text with default model:
+
+```ts
+import {makeTranscriber, defaultTranscriberSpec} from "@omnimedia/omnitool"
+
+// uses onnx-community/whisper-tiny_timestamped
+const transcriber = await makeTranscriber({
+	driver,
+	spec: defaultTranscriberSpec(),
+	workerUrl: new URL("/features/speech/transcribe/worker.bundle.min.js", import.meta.url),
+	onLoading: loading => console.log("loading", loading),
+})
+
+const transcript = await transcriber.transcribe({
+	source: file,
+	language: "english",
+	onReport: report => console.log("report", report),
+	onTranscription: text => console.log("transcribing", text),
+})
+
+const timeline = omni.timeline(o => {
+	const video = o.video(clip)
+	return o.captions(video, transcript)
+})
+```
+
+Load a custom speech-to-text model:
+
+```ts
+const transcriber = await makeTranscriber({
+	driver,
+	spec: {
+	  model: "onnx-community/whisper-tiny_timestamped",
+	  dtype: "q4",
+	  device: "wasm",
+	  chunkLength: 20,
+	  strideLength: 3
+	},
+	workerUrl: new URL("/features/speech/transcribe/worker.bundle.min.js", import.meta.url),
+	onLoading: loading => console.log("loading", loading),
+})
+```
+
+> [!IMPORTANT]
+> Use a Transformers.js-compatible speech-to-text model, for example `onnx-community/*_timestamped`.
+> The model must support word-level timestamps because captions use `return_timestamps: "word"`.
+> `device` and `dtype` are passed to Transformers.js and depend on your runtime/model.
+> Browser usage commonly uses `"wasm"` or `"webgpu"`. `"webgpu"` for speed, `"wasm"` for more device support
+> `workerUrl` depends on where you host the worker bundle.
+
+`o.captions(video, transcript, options)` creates captions for a video or audio.
+`o.captions` uses `captionPresets.default` preset.
+The generated caption id is stored on the source item as `captionId`.
+use `o.captions.presets` to choose from available pre-styled captions.
+pass `styles` in options to override preset styles.
+transcript chunk timestamps are in seconds.
 
-## 📦 Example (Programmatic Timeline)
+Update caption options after creation:
 
 ```ts
-import { subtitle, crossfade, sequence, stack, video } from "@omni/tools"
+const caption = o.captions.make(transcript, {maxChars: 42})
+const style = o.textStyle({fill: "#00ff00"})
+o.set(caption.id, {
+	maxChars: 32,
+	styleId: style.id,
+})
+```
+
+Caption options:
+`styles` - sets styles, it overrides the preset's styles.
+`start` - transcript time where captions begin, in milliseconds.
+`duration` - caption layer duration, in milliseconds.
+`maxChars` - maximum characters in one generated caption line.
+`maxDuration` - maximum duration of one generated caption line, in milliseconds.
+`maxSilence` - maximum silence allowed inside one caption; longer pauses start a new caption, in milliseconds.
+
+import `captionPresets` to list available caption looks.
 
-const watermark = subtitle("omniclip")
-const xfade = crossfade(500)
+## 🧼 Background Remover
+
+```ts
+import {makeBgRemover, defaultBgRemoverSpec} from "@omnimedia/omnitool"
+
+const bgRemover = await makeBgRemover({
+	spec: defaultBgRemoverSpec(),
+	workerUrl: new URL("/features/bg-remover/worker.bundle.min.js", import.meta.url),
+	onLoading: loading => console.log("loading", loading),
+})
+
+const outputFrame = await bgRemover.remove({frame: inputFrame})
+
+bgRemover.dispose()
+```
 
-const timeline = sequence(
-  video("opening-credits.mp4"),
-  xfade,
-  stack(
-    video("skateboarding.mp4"),
-    watermark
-  ),
-  xfade,
-  stack(
-    video("biking.mp4"),
-    watermark
-  )
+Use it on image or while decoding a video:
+
+```ts
+const bgRemover = await makeBgRemover({
+	spec: defaultBgRemoverSpec(),
+	workerUrl: new URL("/features/bg-remover/worker.bundle.min.js", import.meta.url),
+	onLoading: loading => console.log("loading", loading),
+})
+
+const video = driver.decodeVideo({
+	source: file,
+	onFrame: frame => bgRemover.remove({frame}),
+})
+```
+
+Notes:
+- `defaultBgRemoverSpec()` uses `Xenova/modnet` on `webgpu` with `auto` dtype.
+- `model`, `device`, and `dtype` are passed to Transformers.js and depend on browser/runtime support.
+- Call `dispose()` when the remover is no longer needed.
+
+## 🎛 Filters
+
+Filter application:
+
+```ts
+const timeline = omni.timeline(o =>
+	o.stack(
+		o.filter.blur(
+			o.video(clip, {duration: 3000}),
+			{strength: 8, quality: 4}
+		),
+		o.filter.glow(
+			o.text("Hello world", {duration: 3000}),
+			{distance: 12, outerStrength: 2, color: "#ffffff"}
+		)
+	)
 )
-	
 ```
 
----
+Reusable filter items:
 
-## 🧩 Timeline Format (Omni Timeline Format)
+```ts
+const timeline = omni.timeline(o => {
+	const blur = o.filter.blur.make({strength: 8, quality: 4})
+
+	return o.stack(
+		o.video(clip, {duration: 3000, filterIds: [blur.id]}),
+		o.text("Caption", {duration: 3000, styles: {fill: "white"}})
+	)
+})
+```
 
-Every timeline is defined as a graph:
+Filter metadata for UI:
 
-```json
-{
-  "format": "omni-timeline@1",
-  "root": "root-1",
-  "items": [
-    ["root-1", ["sequence", { "children": ["video-1", "stack-1"] }]],
-    ["video-1", ["video", { ... }]],
-    ["stack-1", ["stack", { "children": ["text-1", "audio-1"] }]],
-    ["text-1", ["text", { ... }]],
-    ["audio-1", ["audio", { ... }]]
-  ]
+```ts
+import {
+	filters
+} from "@omnimedia/omnitool"
+
+const available = Object.entries(filters)
+const blur = filters.blur
+const schema = blur.schema
+```
+
+## 🧭 Spatial Transforms
+
+```ts
+const timeline = omni.timeline(o => {
+	const move = o.spatial(o.transform({
+		position: [120, 40],
+		scale: [0.6, 0.6],
+		rotation: 12
+	}))
+
+	const title = o.text("Lower third", {
+		duration: 2000,
+		styles: {fill: "white", fontSize: 36}
+	})
+	o.set(title.id, {spatialId: move.id})
+
+	return o.stack(
+		o.video(clip, {duration: 4000}),
+		title
+	)
+})
+```
+
+Animations:
+
+```ts
+const timeline = omni.timeline(o => {
+	const fadeIn = o.animate.opacity.make("easeIn", [
+		[0, 0],
+		[700, 1],
+	])
+	const slideOut = o.animate.transform.make("linear", [
+		[0, o.transform({position: [0, 0]})],
+		[1000, o.transform({position: [400, 0]})],
+	])
+
+	const title = o.text("Lower third", {
+		duration: 2000,
+		styles: {fill: "white", fontSize: 36}
+	})
+	o.set(title.id, {animationIds: [fadeIn.id, slideOut.id]})
+
+	return o.stack(
+		o.video(clip, {duration: 4000}),
+		title
+	)
+})
+```
+
+Built-in transform animations:
+
+```ts
+const animOut = {
+	duration: 500,
+	offset: item.duration - 500,
 }
+const slideIn = o.animate.presets.slideIn.make()
+const slideOut = o.animate.presets.slideOut.make(animOut)
+const spinIn = o.animate.presets.spinIn.make()
+const spinOut = o.animate.presets.spinOut.make(animOut)
+const zoomIn = o.animate.presets.zoomIn.make()
+const zoomOut = o.animate.presets.zoomOut.make(animOut)
+const bounceIn = o.animate.presets.bounceIn.make()
+const bounceOut = o.animate.presets.bounceOut.make(animOut)
 ```
 
-Each item is a `[id, item]` pair. Items can reference others by ID, forming a directed graph of reusable, composable blocks.
+Built-in scalar animations:
 
----
+```ts
+const fadeIn = o.animate.presets.fadeIn.make()
+const fadeOut = o.animate.presets.fadeOut.make(animOut)
+```
 
-## 🖥 CLI Usage
+Animation application:
 
-```bash
-# Build a timeline (manually or via AI)
-omnitool build-template promo.json
+```ts
+const timeline = omni.timeline(o => {
+	const title = o.animate.opacity(
+		o.text("Lower third", {
+			duration: 2000,
+			styles: {fill: "white", fontSize: 36},
+		}),
+		"easeIn",
+		[
+			[0, 0],
+			[700, 1],
+		]
+	)
+
+	return o.stack(
+		o.video(clip, {duration: 4000}),
+		title
+	)
+})
+```
 
-# Validate structure
-omnitool validate promo.json
+Reusable animation:
 
-# Render video
-omnitool export promo.json --output final.mp4
+```ts
+const timeline = omni.timeline(o => {
+	const fadeIn = o.animate.opacity.make("easeIn", [
+		[0, 0],
+		[700, 1],
+	])
+
+	const title = o.text("Lower third", {
+		duration: 2000,
+		styles: {fill: "white", fontSize: 36},
+	})
+	o.set(title.id, {animationIds: [fadeIn.id]})
+
+	return o.stack(
+		o.video(clip, {duration: 4000}),
+		title
+	)
+})
+```
 
-# Batch render
-omnitool batch-export ./projects/* --output-dir ./exports
+Animation metadata:
+
+```ts
+import {animatableProperties, animationPresets} from "@omnimedia/omnitool"
+
+Object.entries(animatableProperties).forEach(([property, meta]) => {
+	console.log(property, meta.type, meta.defaultTerp, meta.channels)
+	// transform transform linear [...]
+	// opacity scalar linear [...]
+})
+
+Object.entries(animationPresets).forEach(([preset, meta]) => {
+	console.log(preset, meta.type, meta.label, meta.defaults)
+	// slideIn motion Slide in {...}
+	// slideOut motion Slide out {...}
+	// spinIn motion Spin in {...}
+	// spinOut motion Spin out {...}
+	// zoomIn motion Zoom in {...}
+	// zoomOut motion Zoom out {...}
+	// bounceIn motion Bounce in {...}
+	// bounceOut motion Bounce out {...}
+	// fadeIn scalar Fade in {...}
+	// fadeOut scalar Fade out {...}
+})
 ```
 
----
+Animatable properties describe what can be keyframed, such as `transform` and `opacity`.
+Animation presets describe built-in recipes, such as `slideIn` and `fadeIn`.
+Use `animationPresets` to list available recipes, and `o.animate.presets` to create animation items.
+Use `o.animate` to create or apply animation items.
 
-## ✅ Use Cases
+Preset options:
+- `duration` sets the animation duration, defaulting to `700`.
+- `offset` shifts generated keyframes in item-local time.
+  Useful for out animations: `item.duration - 500` starts `slideOut` 500ms before the item ends.
+- `from` sets the start value, like opacity `0` or position `[-400, 0]`.
+- `to` sets the end value, like opacity `1` or position `[0, 0]`.
+- `terp` sets interpolation, defaulting to the preset's `terp`.
 
-- Render videos from scripts, templates, or AI
-- Build and test timelines without opening the UI
-- Generate video pipelines from code or prompts
+Utils:
 
----
+```ts
+import {resolveScalarAnimation, resolveTransformAnimation} from "@omnimedia/omnitool"
+
+const transform = resolveTransformAnimation(localTime, transformAnimation)
+const opacity = resolveScalarAnimation(localTime, opacityAnimation)
+```
 
-## 📘 More to come
+`resolveTransformAnimation` resolves an animated transform at the given local time.
+`resolveScalarAnimation` resolves an animated scalar value at the given local time.
+`localTime` is time relative to the item being resolved.
+`clamp` is the default and currently only extrapolation mode, holding the first or last keyframe value outside the authored range.
 
-- `omnitool preview` – headless timeline viewer
-- `omnitool optimize` – auto-fit timeline elements
-- `omnitool ai` – native prompt-to-timeline generation
+## 🔀 Transitions
 
----
+Transitions are sequence items placed between two visual items.
+
+```ts
+const timeline = omni.timeline(o =>
+	o.sequence(
+		o.video(firstClip, {duration: 3000}),
+		o.transition.fade(700),
+		o.video(secondClip, {duration: 3000}),
+	)
+)
+```
+
+Use the exported transition registry to build UI:
+
+```ts
+import {transitions} from "@omnimedia/omnitool"
+
+const options = Object.entries(transitions).map(([key, transition]) => ({
+	key,
+	label: transition.label,
+}))
+```
 
-## 🧠 Learn More
+Worker URL notes:
+- `Driver.setup()` defaults to `/node_modules/@omnimedia/omnitool/x/driver/driver.worker.bundle.min.js`.
+- If you serve the worker from a different location, pass `workerUrl`:
 
-Omni Tools powers AI agents, programmatic editing, and upcoming new version of omniclip video editor.
+```ts
+const workerUrl = new URL(
+	"/path/to/driver.worker.bundle.min.js",
+	window.location.href
+)
+const driver = await Driver.setup({workerUrl})
+```
+
+## ▶️ Playback
+
+```ts
+const player = await omni.playback(timeline)
+
+document.body.appendChild(player.canvas)
+await player.play()
+player.playbackRate = 0.5 // slow motion
+player.playbackRate = -1 // reverse
+```
+
+Notes:
+- Call `await player.update(timeline)` if you update the timeline.
+- `playbackRate` supports slower, faster, and reverse visual playback. Audio currently plays only at `1`.
+
+## 📤 Export
+
+```ts
+await omni.render(timeline, framerate)
+```
+
+## 🧩 Timeline Format (TimelineFile)
+
+All durations and timestamps are in milliseconds.
+
+```json
+{
+	"format": "timeline",
+	"info": "https://omniclip.app/",
+	"version": 0,
+	"rootId": 123,
+	"items": [
+		{"id": 123, "kind": 0, "childrenIds": [456, 789]},
+		{"id": 456, "kind": 2, "mediaHash": "...", "start": 0, "duration": 3000},
+		{"id": 789, "kind": 4, "content": "Hello", "duration": 1500}
+	]
+}
+```
+
+Timeline items:
+- 0 `Sequence`
+- 1 `Stack`
+- 2 `Video`
+- 3 `Audio`
+- 4 `Text`
+- 5 `Gap`
+- 6 `Spatial`
+- 7 `Animation`
+- 8 `Transition`
+- 9 `TextStyle`
+- 10 `Filter`
+- 11 `Caption`
+
+## 🗺️ Roadmap
+- CLI commands:
+
+```bash
+# build a reusable template from a timeline
+omnitool build-template promo.json
+# validate a timeline file
+omnitool validate promo.json
+# export a timeline to a video file
+omnitool export promo.json --output final.mp4
+# batch export multiple timelines
+omnitool batch-export ./projects/* --output-dir ./exports
+# headless timeline viewer
+omnitool preview promo.json
+# auto-fit timeline elements
+omnitool optimize promo.json
+# prompt-to-timeline generation
+omnitool ai "make a 15s promo for tea"
+```
+
+- smooth seeking
+- keyframes
+- custom filters, likely via driver-side registration with timeline sugar such as:
+
+```ts
+// Register custom filter
+driver.registerFilter({
+	type: "vhs",
+	make: params => new Filter(/* ... */),
+	schema: {
+		intensity: {type: "number", min: 0, max: 1, default: 0.5},
+		scanlines: {type: "boolean", default: true},
+	},
+})
+
+// Use custom filter
+const timeline = omni.timeline(o =>
+	o.filter.custom(
+		"vhs",
+		o.video(clip, {duration: 3000}),
+		{intensity: 0.8}
+	)
+)
+```
 
-→ [Visit Omniclip](https://omniclip.app)
+- server-side, not just browsers