@sequio/skill 0.1.0 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/llms.txt +8 -4
- package/package.json +1 -1
- package/skills/sequio/SKILL.md +51 -19
- package/skills/sequio/references/api.md +38 -6
- package/skills/sequio/references/recipes.md +260 -14
package/llms.txt
CHANGED
|
@@ -5,8 +5,10 @@
|
|
|
5
5
|
> clock; the SDK owns the low-level runtime — decode, composite, audio, export.
|
|
6
6
|
> Compositions are ordinary imperative TypeScript (you `new` the engine's own
|
|
7
7
|
> classes), so the same code previews in a browser, exports in a browser, and
|
|
8
|
-
> renders on a server.
|
|
9
|
-
>
|
|
8
|
+
> renders on a server. sequio does not render HTML/CSS — there is no DOM/JSX or
|
|
9
|
+
> declarative-template layer; you build the object graph in code. Persistence,
|
|
10
|
+
> schema, undo, collaboration and UI are out of scope — they belong to the layer
|
|
11
|
+
> above.
|
|
10
12
|
|
|
11
13
|
Key facts an assistant should hold before writing sequio code:
|
|
12
14
|
|
|
@@ -23,14 +25,16 @@ Key facts an assistant should hold before writing sequio code:
|
|
|
23
25
|
- `sequio render <file>` encodes to video (pure-Node WebGPU; needs a GPU or Mesa
|
|
24
26
|
lavapipe). `sequio frame <file> --time <sec>` exports a single frame as a PNG
|
|
25
27
|
through the same render core — the fast way to visually check a composition
|
|
26
|
-
before a full render. `sequio
|
|
28
|
+
before a full render. `sequio audio <file>` exports only the audio mix to an
|
|
29
|
+
audio-only file (mp3 default; also m4a/wav/ogg/webm). `sequio preview <file>
|
|
30
|
+
--watch` serves a live preview.
|
|
27
31
|
|
|
28
32
|
## Docs
|
|
29
33
|
|
|
30
34
|
- [Architecture & the five contracts](https://github.com/slightc/sequio/blob/main/docs/architecture.md): modules, invariants, and the public-surface table.
|
|
31
35
|
- [Runtime](https://github.com/slightc/sequio/blob/main/docs/runtime.md): compile + link + run multi-file TS/JS into a `Composer`; `defineComposition`, `loadAsset`, externals.
|
|
32
36
|
- [CLI](https://github.com/slightc/sequio/blob/main/docs/cli.md): `sequio render` / `sequio preview --watch`, `--scale`, fonts, local/network media.
|
|
33
|
-
- [Server-side rendering](https://github.com/slightc/sequio/blob/main/docs/server-side-rendering.md):
|
|
37
|
+
- [Server-side rendering](https://github.com/slightc/sequio/blob/main/docs/server-side-rendering.md): re-run a composition's own code (a `RuntimeBundle`, via `renderBundleToFile`) on a server through either render route (headless Chrome / pure-Node WebGPU) — no spec to serialize or keep in sync.
|
|
34
38
|
- [Text animation](https://github.com/slightc/sequio/blob/main/docs/text-animation.md): clip/text animators, split motion, the GSAP binding.
|
|
35
39
|
- [Contributor / agent guide (AGENT.md)](https://github.com/slightc/sequio/blob/main/AGENT.md): conventions, monorepo layout, working agreement.
|
|
36
40
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@sequio/skill",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.2",
|
|
4
4
|
"description": "An installable AI Agent Skill (SKILL.md) + llms.txt that teach an AI assistant how to use sequio — build a Track/Clip/Effect object graph, preview, and export/render video.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
package/skills/sequio/SKILL.md
CHANGED
|
@@ -6,7 +6,9 @@ description: >-
|
|
|
6
6
|
video/motion-graphics programmatically — creating a timeline of tracks and
|
|
7
7
|
clips (text, image, video, shapes), animating them (keyframes or GSAP),
|
|
8
8
|
applying effects/transitions, mixing audio, and exporting to MP4/WebM or
|
|
9
|
-
rendering server-side.
|
|
9
|
+
rendering server-side. sequio does NOT render HTML/CSS — compositions are
|
|
10
|
+
imperative TypeScript that `new`s engine classes, not a declarative DOM/JSX
|
|
11
|
+
framework. Triggers: "sequio", "video editor SDK", "compose a
|
|
10
12
|
video", "timeline / track / clip", "render an mp4 from code", "TextClip",
|
|
11
13
|
"Compositor", "defineComposition".
|
|
12
14
|
---
|
|
@@ -28,7 +30,7 @@ Pick the package that matches the task before writing code:
|
|
|
28
30
|
|---|---|---|
|
|
29
31
|
| Drive an object graph in an app you control | **engine** directly | `@sequio/engine` |
|
|
30
32
|
| Author a self-contained composition file that previews **and** renders | **runtime** authoring API | `@sequio/engine` + `@sequio/runtime` |
|
|
31
|
-
| Turn a composition file into an `.mp4`, a single-frame PNG, or a live preview from a terminal | **cli** | `sequio render` / `sequio frame` / `sequio preview` |
|
|
33
|
+
| Turn a composition file into an `.mp4`, a single-frame PNG, or a live preview from a terminal | **cli** | `sequio check` / `sequio render` / `sequio frame` / `sequio preview` |
|
|
32
34
|
| Render a composition to a file on a server (no browser) | **server** Route B | `@sequio/server/route-b` |
|
|
33
35
|
|
|
34
36
|
For most "make me a video from code" tasks, author a **composition file** (runtime
|
|
@@ -98,33 +100,53 @@ export default defineComposition(async () => {
|
|
|
98
100
|
Then, from the workspace root:
|
|
99
101
|
|
|
100
102
|
```bash
|
|
103
|
+
sequio check composition.ts # static validation — no GPU, offline (the pre-flight lint)
|
|
101
104
|
sequio preview composition.ts --watch # live in-browser preview, reloads on edit
|
|
102
105
|
sequio frame composition.ts --time 2 --out shot.png # export ONE frame at t=2s as a PNG (fast visual check)
|
|
106
|
+
sequio audio composition.ts --out track.mp3 # export ONLY the audio mix (mp3 default; also m4a/wav/ogg/webm)
|
|
103
107
|
sequio render composition.ts --out out.mp4 # encode the whole thing to video (pure-Node WebGPU)
|
|
104
108
|
sequio render composition.ts --out out.mp4 --scale 2 # 2× resolution (sharp text/edges)
|
|
105
109
|
```
|
|
106
110
|
|
|
107
|
-
`sequio render` and `sequio
|
|
111
|
+
`sequio render`, `sequio frame` and `sequio audio` need a WebGPU host — a real GPU or the Mesa
|
|
108
112
|
**lavapipe** software driver (`apt install mesa-vulkan-drivers`; then
|
|
109
113
|
`export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/lvp_icd.json`). Without one they
|
|
110
114
|
throw a clear error.
|
|
111
115
|
|
|
112
|
-
###
|
|
116
|
+
### Verify loop: `check` → `frame` → `render`
|
|
113
117
|
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
frame and look at it:
|
|
118
|
+
Tighten the loop from coarsest-and-cheapest to full render. As an agent editing
|
|
119
|
+
code, run them in this order:
|
|
117
120
|
|
|
118
121
|
```bash
|
|
119
|
-
sequio
|
|
122
|
+
sequio check composition.ts # 1. offline lint — no GPU, no browser, fast
|
|
123
|
+
sequio frame composition.ts --time 2.5 --out /tmp/check.png # 2. render ONE frame, then open/view the PNG
|
|
124
|
+
sequio render composition.ts --out out.mp4 # 3. encode the whole video
|
|
120
125
|
```
|
|
121
126
|
|
|
122
|
-
`
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
127
|
+
**1. `check` — does the code build a legal object graph?** It compiles + links +
|
|
128
|
+
runs the builder with a **null renderer** (no WebGPU) and walks the graph for the
|
|
129
|
+
mistakes that are cheap to catch offline: illegal clip times (`end ≤ start`), a
|
|
130
|
+
keyframe outside its clip's interval (dead keyframe), a `TextClip.fontFamily` no
|
|
131
|
+
`fonts.load(...)` registered (silent system-font fallback — breaks
|
|
132
|
+
preview↔render), a transition whose two clips don't overlap, an `anchor` outside
|
|
133
|
+
`0..1`, a missing local `loadAsset` file. Exits non-zero on any error;
|
|
134
|
+
`--json` emits a machine-readable `Diagnostic[]`. A green `check` doesn't prove
|
|
135
|
+
the picture is right — but a **red `check` means don't bother rendering yet**.
|
|
136
|
+
It's GPU-free and offline, so run it after every edit.
|
|
137
|
+
|
|
138
|
+
**2. `frame` — does the frame *look* right?** `frame` runs the **same render
|
|
139
|
+
core** as `render` (contract #3), so the PNG is exactly what the video would
|
|
140
|
+
contain at that instant — the way to confirm layout, position, and color.
|
|
141
|
+
`--time` is clamped to `[0, duration]`; `--scale N` renders at N× like `render`.
|
|
142
|
+
|
|
143
|
+
**3. `render`** once `check` is green and a couple of `frame`s look right.
|
|
144
|
+
|
|
145
|
+
> `check` deliberately does **not** decode media, hit the network, or compile
|
|
146
|
+
> shaders — those need a browser/GPU and are what `frame`/`render` are for. A
|
|
147
|
+
> composition that fetches a remote image or decodes video *in its builder* gets
|
|
148
|
+
> a `B4` **warning** from `check` (the code compiled fine; verify the picture
|
|
149
|
+
> with `frame`), not a failure.
|
|
128
150
|
|
|
129
151
|
## Core building blocks
|
|
130
152
|
|
|
@@ -153,8 +175,13 @@ eyeball → then `sequio render` once it looks right.
|
|
|
153
175
|
clip.animator = gsapClipAnimator(gsap, (tl, o) => tl.from(o, { y: -60, alpha: 0, ease: 'back.out(1.7)' }));
|
|
154
176
|
```
|
|
155
177
|
The binding seeks a **paused** timeline, keeping `render(t)` pure.
|
|
156
|
-
- **Text motion** — `TextClip.split` + `
|
|
157
|
-
animate per line/word/char (e.g. a staggered drop-in).
|
|
178
|
+
- **Text motion** — `TextClip.split` + `TextClip.textAnimator` (`StaggerTextAnimator`
|
|
179
|
+
/ `gsapTextAnimator`) animate per line/word/char (e.g. a staggered drop-in).
|
|
180
|
+
- **Bring your own** — the four seams are subclass/implement points: subclass
|
|
181
|
+
`Effect` / `Transition` (or an engine effect) and implement `ClipAnimator`
|
|
182
|
+
(`clip.animator`) / `TextAnimator` (`textClip.textAnimator`). Build on the
|
|
183
|
+
engine's own classes and no `pixi.js` is needed — same in preview and render.
|
|
184
|
+
See recipe 7b and `packages/cli/example/custom-fx/`.
|
|
158
185
|
|
|
159
186
|
## Media, fonts, effects, audio, export — quick pointers
|
|
160
187
|
|
|
@@ -165,11 +192,16 @@ eyeball → then `sequio render` once it looks right.
|
|
|
165
192
|
`TextClip`. Load an explicit web font so preview and Node render match — system
|
|
166
193
|
defaults differ per platform.
|
|
167
194
|
- **Effects/Transitions**: `ColorEffect`, `BlurEffect`, `BulgeEffect`,
|
|
168
|
-
`CrossfadeTransition`, etc. (some — chroma/LUT/wipe — are still TODO).
|
|
195
|
+
`CrossfadeTransition`, etc. (some — chroma/LUT/wipe — are still TODO). Attach an
|
|
196
|
+
effect with `clip.effects.push(fx)`; bind a transition with
|
|
197
|
+
`track.addTransition(new T(frames).between(a, b))` over the clips' overlap.
|
|
198
|
+
Roll your own by subclassing — see recipe 7b.
|
|
169
199
|
- **Audio**: `AudioTrack` + `AudioSource`; `AudioEngine` mixes (Web Audio live,
|
|
170
200
|
OfflineAudioContext for export).
|
|
171
|
-
- **Export in an app** (not the CLI): use `Exporter` from `@sequio/engine
|
|
172
|
-
|
|
201
|
+
- **Export in an app** (not the CLI): use `Exporter` from `@sequio/engine`
|
|
202
|
+
(`export(...)` → video, `exportFrame(t, ...)` → one still image, `exportAudio(...)`
|
|
203
|
+
→ audio-only file), or a runtime `Composer`'s `composer.export(...)`. Audio-only
|
|
204
|
+
on the CLI is `sequio audio composition.ts` (recipe 11).
|
|
173
205
|
|
|
174
206
|
## Before you finish
|
|
175
207
|
|
|
@@ -19,13 +19,16 @@ Times are **seconds** at the API boundary, quantized to frames internally via
|
|
|
19
19
|
- `Transform2D` — a clip's spatial state: `position`, `scale`, `rotation`,
|
|
20
20
|
`anchor`, `alpha` (each an `AnimatableProperty`).
|
|
21
21
|
- `AnimatableProperty` / `Keyframe` — `.setStatic(value)` or
|
|
22
|
-
`.setKeyframes([{ time, value, easing? }, ...])`.
|
|
22
|
+
`.setKeyframes([{ time, value, easing? }, ...])`; `.valueAt(t)`, `.animated`,
|
|
23
|
+
`.keyframeTimes` (sorted keyframe times, for static validation).
|
|
23
24
|
- Easings: `linear`, `hold`, `cubicBezier`, `easeInQuad`, `easeOutQuad`,
|
|
24
25
|
`easeInOutQuad`, `easeInCubic`, `easeOutCubic`, `easeInOutCubic`. Type `Easing`.
|
|
25
26
|
- Clip/text animators: `ClipAnimator`, `TextAnimator`, `StaggerTextAnimator`,
|
|
26
27
|
`TweenAnimator`, `IDENTITY_SAMPLE`, `lerpSample`, `AnimationSample`, plus
|
|
27
28
|
`TextPart` / `TextSplit` / `StaggerOrder` / `StaggerTextOptions` /
|
|
28
|
-
`TweenAnimatorOptions`.
|
|
29
|
+
`TweenAnimatorOptions`. `ClipAnimator` (`sampleAt`) and `TextAnimator`
|
|
30
|
+
(`sampleForPart`) are **extension seams** — implement your own for custom motion
|
|
31
|
+
(assign to `clip.animator` / `textClip.textAnimator`); see recipe 7b.
|
|
29
32
|
- GSAP binding (engine has no gsap dep): `gsapClipAnimator`, `gsapTextAnimator`,
|
|
30
33
|
`identityTarget`; types `GsapLike`, `GsapTimelineLike`, `GsapTarget`.
|
|
31
34
|
- Text layout: `computeTextParts`, `MeasureWidth`.
|
|
@@ -51,6 +54,8 @@ Times are **seconds** at the API boundary, quantized to frames internally via
|
|
|
51
54
|
- `FontManager`, `fonts` (singleton), `buildGoogleCss2Url`; types `FontSpec`,
|
|
52
55
|
`GoogleFontSpec`.
|
|
53
56
|
- `fonts.load({ family, src })` before rendering a `TextClip` in that family.
|
|
57
|
+
- `fonts.families()` — the family names a load has been requested for (used by
|
|
58
|
+
`sequio check` to catch a `TextClip` referencing an unregistered font).
|
|
54
59
|
|
|
55
60
|
## Compositor graph
|
|
56
61
|
|
|
@@ -58,10 +63,28 @@ Times are **seconds** at the API boundary, quantized to frames internally via
|
|
|
58
63
|
fps?, background?, timebase?, createRenderer?, resolution? })`, then
|
|
59
64
|
`await init()`, `addTrack(track)`, `renderPreview(t)`.
|
|
60
65
|
- Tracks: `Track`, `VisualTrack` (stacks by `.zIndex`), `AudioTrack`.
|
|
61
|
-
|
|
62
|
-
`
|
|
63
|
-
`ShapeKind`.
|
|
66
|
+
`TextClip`, `ShapeClip`, `GroupClip`. Types: `TextStyleLike`, `TextStrokeLike`,
|
|
67
|
+
`ShapeSpec`, `ShapeKind`, `MaskSpec`.
|
|
64
68
|
- Every clip has `.start` / `.end` (seconds) and `.transform` (`Transform2D`).
|
|
69
|
+
- Time-remap on any clip: `.speed` (default `1`; time-line seconds → `speed`×
|
|
70
|
+
source seconds — variable-speed + pitch shift for audio) and `.reversed`
|
|
71
|
+
(default `false`; 倒放 — plays the same source window backwards. Video feeds a
|
|
72
|
+
decreasing source time to the decoder (decoded in GOP batches so reverse isn't
|
|
73
|
+
O(GOP²)); audio plays a reversed copy of the buffer, since Web Audio has no
|
|
74
|
+
negative playback rate). `clip.sourceTimeAt(t)` is the resulting source time —
|
|
75
|
+
the single mapping both decode-prep and render use, so `speed`/`reversed` set
|
|
76
|
+
on the clip apply everywhere.
|
|
77
|
+
- `TextClip` style (`TextStyleLike`): `text`, `fontFamily`, `fontSize`
|
|
78
|
+
(animatable), `fill`, plus pass-throughs `fontWeight`, `fontStyle`
|
|
79
|
+
(`'italic'`), `letterSpacing`, `align`, `lineHeight` and `stroke`
|
|
80
|
+
(`TextStrokeLike` `{ color, width }` → hollow / outlined type). Weight and
|
|
81
|
+
letter-spacing feed measurement too, so split-text layout stays correct.
|
|
82
|
+
- `VisualClip.maskShape` (`MaskSpec`): clip the content to a rounded-rect
|
|
83
|
+
(`kind:'rect'`, big `radius` → arch/stadium) or `ellipse` (circle crop), sized
|
|
84
|
+
explicitly (`width`/`height`, optional `x`/`y`, `inset`). Apply it to a
|
|
85
|
+
container-backed clip — a `GroupClip` wrapping the image — since a Sprite
|
|
86
|
+
cannot be masked by its own child; lay the content out from `(0,0)` in the
|
|
87
|
+
group.
|
|
65
88
|
|
|
66
89
|
## Effects & transitions
|
|
67
90
|
|
|
@@ -72,6 +95,10 @@ Times are **seconds** at the API boundary, quantized to frames internally via
|
|
|
72
95
|
- Warp math: `Mat3`, `Quad`, `Vec2`, `UNIT_QUAD`, `squareToQuad`, `invert3x3`,
|
|
73
96
|
`applyHomography`, `perspectiveSampleMatrix`, `bulgeSourceUv`.
|
|
74
97
|
- Transitions: `Transition`, `CrossfadeTransition`, `crossfadeAlpha`.
|
|
98
|
+
- Extend: subclass `Effect` (or an engine effect) for a custom filter/param, and
|
|
99
|
+
`Transition` (or `CrossfadeTransition`) for a custom mix — no `pixi.js` needed if
|
|
100
|
+
you build on the engine's own classes. Attach: `clip.effects.push(fx)`,
|
|
101
|
+
`track.addTransition(new T(frames).between(a, b))`. See recipe 7b.
|
|
75
102
|
- Status: color/blur/warp + crossfade done; chroma/LUT/wipe TODO.
|
|
76
103
|
|
|
77
104
|
## Audio
|
|
@@ -82,9 +109,14 @@ Times are **seconds** at the API boundary, quantized to frames internally via
|
|
|
82
109
|
|
|
83
110
|
## Export
|
|
84
111
|
|
|
85
|
-
- `Exporter` (`ExportOptions`, `ExportFrameOptions`),
|
|
112
|
+
- `Exporter` (`ExportOptions`, `ExportFrameOptions`, `AudioExportOptions`),
|
|
113
|
+
`ExportCancelledError`. `Exporter.export` → video, `exportFrame` → one still
|
|
114
|
+
image, `exportAudio` → audio-only file (the `AudioEngine` offline mix).
|
|
86
115
|
- `ExportSink`, `ResolvedExportOptions`, `exportFrameTimes`,
|
|
87
116
|
`MediabunnyExportSink`.
|
|
117
|
+
- Audio-only: `AudioExportSink`, `AudioExportFormat`
|
|
118
|
+
(`'m4a' | 'mp3' | 'wav' | 'ogg' | 'webm'`), `ResolvedAudioExportOptions`,
|
|
119
|
+
`MediabunnyAudioExportSink`.
|
|
88
120
|
|
|
89
121
|
## Re-exported PixiJS types (type-only)
|
|
90
122
|
|
|
@@ -76,13 +76,16 @@ it directly — the engine declares only the structural `GsapLike` types.
|
|
|
76
76
|
import { StaggerTextAnimator } from '@sequio/engine';
|
|
77
77
|
|
|
78
78
|
title.split = 'char'; // split into per-character parts
|
|
79
|
-
title.
|
|
80
|
-
|
|
81
|
-
|
|
79
|
+
title.textAnimator = new StaggerTextAnimator({
|
|
80
|
+
from: { y: -40, alpha: 0 }, // each part starts 40px up + invisible
|
|
81
|
+
stagger: 0.04, // delay between consecutive parts (s)
|
|
82
|
+
duration: 0.5, // per-part animation length (s)
|
|
82
83
|
});
|
|
83
84
|
```
|
|
84
85
|
|
|
85
|
-
|
|
86
|
+
Split text uses `textAnimator` (a `TextAnimator`); a whole-clip animator uses
|
|
87
|
+
`animator` (a `ClipAnimator`) — don't mix the two fields. For gsap-driven per-part
|
|
88
|
+
motion use `gsapTextAnimator(gsap, split, builder)`.
|
|
86
89
|
|
|
87
90
|
## 5. Video and image clips
|
|
88
91
|
|
|
@@ -127,24 +130,97 @@ const title = new TextClip({ text: 'sequio', fontFamily: 'Poppins', fontSize: 72
|
|
|
127
130
|
```ts
|
|
128
131
|
import { BlurEffect, CrossfadeTransition } from '@sequio/engine';
|
|
129
132
|
|
|
130
|
-
|
|
131
|
-
//
|
|
132
|
-
|
|
133
|
+
const blur = new BlurEffect();
|
|
134
|
+
blur.strength.setStatic(8); // strength is an AnimatableProperty (keyframe it too)
|
|
135
|
+
clip.effects.push(blur); // `effects` is an array
|
|
136
|
+
|
|
137
|
+
// A crossfade between two OVERLAPPING clips on one track — their overlap is the
|
|
138
|
+
// transition window (bind order is direction: from → to):
|
|
139
|
+
track.addTransition(new CrossfadeTransition(15).between(clipA, clipB)); // 15 frames
|
|
133
140
|
```
|
|
134
141
|
|
|
135
|
-
(
|
|
136
|
-
|
|
142
|
+
(chroma/LUT/wipe are not yet built — check `references/api.md` and the classes in
|
|
143
|
+
`packages/engine/src/effects/`. To author your own, see the next recipe.)
|
|
137
144
|
|
|
138
|
-
##
|
|
145
|
+
## 7b. Bring your own — custom effect · transition · animation
|
|
146
|
+
|
|
147
|
+
All four extension points are plain subclasses/implementations imported from
|
|
148
|
+
`@sequio/engine` only. Build on the engine's own classes (or keep the animators
|
|
149
|
+
pure math) and you need **no `pixi.js`**, so they render the same in preview
|
|
150
|
+
(WebGL) and Node render (WebGPU). Full worked demo: `packages/cli/example/custom-fx/`.
|
|
151
|
+
|
|
152
|
+
```ts
|
|
153
|
+
// ── a custom EFFECT — subclass an engine effect, reuse its built-in filter ──
|
|
154
|
+
import { BlurEffect, AnimatableProperty } from '@sequio/engine';
|
|
155
|
+
class FocusPull extends BlurEffect {
|
|
156
|
+
readonly focus = new AnimatableProperty<number>(0); // 0 sharp → 1 blurred
|
|
157
|
+
constructor(readonly maxBlur = 28) { super(); }
|
|
158
|
+
override updateAt(t: number) {
|
|
159
|
+
this.strength.setStatic(this.focus.valueAt(t) * this.maxBlur);
|
|
160
|
+
super.updateAt(t); // let BlurEffect push it into the filter
|
|
161
|
+
}
|
|
162
|
+
}
|
|
163
|
+
const fx = new FocusPull(30);
|
|
164
|
+
fx.focus.setKeyframes([{ time: 0, value: 1 }, { time: 0.8, value: 0 }]);
|
|
165
|
+
clip.effects.push(fx);
|
|
166
|
+
|
|
167
|
+
// ── a custom TRANSITION — subclass CrossfadeTransition, reshape the curve ──
|
|
168
|
+
import { CrossfadeTransition, easeInOutCubic } from '@sequio/engine';
|
|
169
|
+
class EasedCrossfade extends CrossfadeTransition {
|
|
170
|
+
override progressAt(t: number) { return easeInOutCubic(super.progressAt(t)); }
|
|
171
|
+
}
|
|
172
|
+
track.addTransition(new EasedCrossfade(30).between(clipA, clipB));
|
|
173
|
+
|
|
174
|
+
// ── a custom whole-clip ANIMATION — implement ClipAnimator (pure sampleAt) ──
|
|
175
|
+
import type { AnimationSample, ClipAnimator } from '@sequio/engine';
|
|
176
|
+
class OrbitAnimator implements ClipAnimator {
|
|
177
|
+
constructor(private r = 90, private period = 2.6) {}
|
|
178
|
+
sampleAt(localT: number): AnimationSample { // sampled at t - clip.start
|
|
179
|
+
const a = (localT / this.period) * Math.PI * 2;
|
|
180
|
+
return { x: Math.cos(a) * this.r, y: Math.sin(a) * this.r, rotation: a };
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
clip.animator = new OrbitAnimator();
|
|
184
|
+
|
|
185
|
+
// ── a custom TEXT ANIMATION — implement TextAnimator (per split part) ──
|
|
186
|
+
import type { AnimationSample, TextAnimator, TextPart } from '@sequio/engine';
|
|
187
|
+
import { easeOutCubic } from '@sequio/engine';
|
|
188
|
+
class DropInText implements TextAnimator {
|
|
189
|
+
constructor(private stagger = 0.06, private dur = 0.5, private drop = 70) {}
|
|
190
|
+
sampleForPart(part: TextPart, localT: number): AnimationSample {
|
|
191
|
+
const k = (localT - part.index * this.stagger) / this.dur;
|
|
192
|
+
const p = k <= 0 ? 0 : k >= 1 ? 1 : easeOutCubic(k);
|
|
193
|
+
return { y: -this.drop * (1 - p), alpha: p };
|
|
194
|
+
}
|
|
195
|
+
}
|
|
196
|
+
title.split = 'char';
|
|
197
|
+
title.textAnimator = new DropInText();
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
`sampleAt` / `sampleForPart` / `updateAt(t)` / `render(t)` must be **pure functions
|
|
201
|
+
of time** (contract #2) — no wall-clock, no dependence on the previous frame — so
|
|
202
|
+
export stays reproducible. An `AnimationSample` composes over the clip's base
|
|
203
|
+
transform: `x`/`y`/`rotation` add, `scaleX`/`scaleY`/`alpha` multiply.
|
|
139
204
|
|
|
140
|
-
|
|
205
|
+
## 8. Export from your own app (not the CLI)
|
|
141
206
|
|
|
142
207
|
- Runtime `Composer`: `const blob = await composer.export({ format: 'mp4' });`
|
|
143
|
-
- Engine `Exporter` directly against a `Compositor
|
|
208
|
+
- Engine `Exporter` directly against a `Compositor` + `AudioEngine` — one object,
|
|
209
|
+
three exits (all reuse the same render/audio core, contract #3):
|
|
144
210
|
```ts
|
|
145
211
|
import { Exporter } from '@sequio/engine';
|
|
146
|
-
const exporter = new Exporter(compositor,
|
|
147
|
-
|
|
212
|
+
const exporter = new Exporter(compositor, audioEngine);
|
|
213
|
+
|
|
214
|
+
// Video: FixedStep loop, awaits prepare — no dropped frames.
|
|
215
|
+
const movie = await exporter.export({ fps: 30, container: 'mp4' });
|
|
216
|
+
|
|
217
|
+
// One still image at a time (no fps boundary needed).
|
|
218
|
+
const still = await exporter.exportFrame(2.5, { type: 'image/png' });
|
|
219
|
+
|
|
220
|
+
// Audio only — just the AudioEngine offline mix, no frames rendered, no GPU.
|
|
221
|
+
const track = await exporter.exportAudio({ format: 'mp3' });
|
|
222
|
+
// format: 'mp3' (default) | 'm4a' | 'wav' | 'ogg' | 'webm'
|
|
223
|
+
// codec defaults per format (mp3 / aac / pcm-s16 / opus); bitrate, range, sampleRate optional
|
|
148
224
|
```
|
|
149
225
|
|
|
150
226
|
## 9. Server-side render (no browser)
|
|
@@ -181,6 +257,176 @@ few representative times → look at the PNGs → only `sequio render` once it's
|
|
|
181
257
|
Programmatically it's `renderBundleFrameToFile(bundle, { out, time, scale })` from
|
|
182
258
|
`@sequio/server/route-b`, or `runFrame(file, { out, time, scale })` from `@sequio/cli`.
|
|
183
259
|
|
|
260
|
+
## 11. Export just the audio (no video)
|
|
261
|
+
|
|
262
|
+
Encode only the composition's audio track — the same `AudioEngine` offline mix the
|
|
263
|
+
video export muxes — to an audio-only file:
|
|
264
|
+
|
|
265
|
+
```bash
|
|
266
|
+
sequio audio composition.ts --out track.mp3 # default format: mp3
|
|
267
|
+
sequio audio composition.ts --format m4a --out track.m4a # or m4a / wav / ogg / webm
|
|
268
|
+
sequio audio composition.ts --out track.wav --bitrate 192000
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
- `--format` is `mp3` (default) `| m4a | wav | ogg | webm`; when omitted it's
|
|
272
|
+
inferred from `--out`'s extension. `--bitrate` is ignored for `wav` (PCM).
|
|
273
|
+
- Same Route B path as `render`, so it needs a WebGPU host (GPU or Mesa lavapipe).
|
|
274
|
+
- Programmatically it's `exportBundleAudioToFile(bundle, { out, format, bitrate })`
|
|
275
|
+
from `@sequio/server/route-b`, or `runAudio(file, { out, format, bitrate })` from
|
|
276
|
+
`@sequio/cli`; in-app it's `Exporter.exportAudio(...)` (recipe 8).
|
|
277
|
+
|
|
278
|
+
## 12. Bring your own spec (JSON → object graph)
|
|
279
|
+
|
|
280
|
+
sequio ships **no** persistent schema — an imperative `defineComposition` module
|
|
281
|
+
is the only portable format the SDK blesses (the server re-runs that code via a
|
|
282
|
+
`RuntimeBundle`, contract #3). If your app already has a document model — a Y.Doc,
|
|
283
|
+
a database row, an editor's own JSON — **you own the schema**, and mapping it onto
|
|
284
|
+
the engine is a ~30-line builder. This is a **pattern, not an API**: copy it and
|
|
285
|
+
bend it to your data; don't look for an official spec type to import.
|
|
286
|
+
|
|
287
|
+
The mapping has four moving parts, all mechanical:
|
|
288
|
+
|
|
289
|
+
1. **keyframes** → `prop.setStatic(v)` for a constant, `prop.setKeyframes([...])` to animate.
|
|
290
|
+
2. **easing** → a name→function lookup table (easings are just exported functions).
|
|
291
|
+
3. **fonts** → `await fonts.load(...)` up front, before any `TextClip` uses the family.
|
|
292
|
+
4. **effects** → a `type` switch that `new`s the engine effect and writes its params.
|
|
293
|
+
|
|
294
|
+
```ts
|
|
295
|
+
import {
|
|
296
|
+
Compositor, VisualTrack, TextClip, ShapeClip, ImageClip, ImageSource,
|
|
297
|
+
BlurEffect, ColorEffect, fonts,
|
|
298
|
+
linear, easeOutCubic, easeInOutCubic,
|
|
299
|
+
type Easing, type Effect,
|
|
300
|
+
} from '@sequio/engine';
|
|
301
|
+
import { defineComposition, loadAsset } from '@sequio/runtime';
|
|
302
|
+
|
|
303
|
+
// ── YOUR schema (you define this — here's a minimal one) ─────────────────────
|
|
304
|
+
type Prop<T> = T | { keyframes: Array<{ time: number; value: T; easing?: string }> };
|
|
305
|
+
interface MySpec {
|
|
306
|
+
width: number; height: number; fps: number; background?: number;
|
|
307
|
+
fonts?: { family: string; src: string }[];
|
|
308
|
+
clips: Array<{
|
|
309
|
+
type: 'text' | 'shape' | 'image';
|
|
310
|
+
start: number; end: number;
|
|
311
|
+
position?: Prop<[number, number]>;
|
|
312
|
+
opacity?: Prop<number>;
|
|
313
|
+
text?: string; fontFamily?: string; fill?: number; // text
|
|
314
|
+
width?: number; height?: number; src?: string; // shape / image
|
|
315
|
+
effects?: Array<{ type: 'blur' | 'color'; strength?: number; brightness?: number }>;
|
|
316
|
+
}>;
|
|
317
|
+
}
|
|
318
|
+
|
|
319
|
+
// ── the fixed lookup tables ──────────────────────────────────────────────────
|
|
320
|
+
const EASINGS: Record<string, Easing> = { linear, easeOutCubic, easeInOutCubic };
|
|
321
|
+
|
|
322
|
+
function applyProp<T>(p: { setStatic(v: T): void; setKeyframes(k: any[]): void }, s: Prop<T> | undefined) {
|
|
323
|
+
if (s === undefined) return;
|
|
324
|
+
if (typeof s === 'object' && s !== null && 'keyframes' in s)
|
|
325
|
+
p.setKeyframes(s.keyframes.map((k) => ({ ...k, easing: k.easing ? EASINGS[k.easing] : undefined })));
|
|
326
|
+
else p.setStatic(s as T);
|
|
327
|
+
}
|
|
328
|
+
|
|
329
|
+
function buildEffect(e: { type: string; strength?: number; brightness?: number }): Effect {
|
|
330
|
+
if (e.type === 'blur') { const fx = new BlurEffect(); if (e.strength != null) fx.strength.setStatic(e.strength); return fx; }
|
|
331
|
+
const fx = new ColorEffect(); if (e.brightness != null) fx.brightness.setStatic(e.brightness); return fx;
|
|
332
|
+
}
|
|
333
|
+
|
|
334
|
+
// ── the builder: MySpec → Composer ───────────────────────────────────────────
|
|
335
|
+
export function fromSpec(spec: MySpec) {
|
|
336
|
+
return defineComposition(async () => {
|
|
337
|
+
const c = new Compositor({ width: spec.width, height: spec.height, fps: spec.fps, background: spec.background });
|
|
338
|
+
await c.init();
|
|
339
|
+
for (const f of spec.fonts ?? []) await fonts.load(f); // fonts BEFORE clips
|
|
340
|
+
|
|
341
|
+
const track = new VisualTrack();
|
|
342
|
+
for (const cs of spec.clips) {
|
|
343
|
+
let clip;
|
|
344
|
+
if (cs.type === 'text') clip = new TextClip({ text: cs.text!, fontFamily: cs.fontFamily, fill: cs.fill });
|
|
345
|
+
else if (cs.type === 'shape') clip = new ShapeClip({ kind: 'rect', width: cs.width!, height: cs.height!, fill: cs.fill ?? 0xffffff });
|
|
346
|
+
else clip = new ImageClip(new ImageSource({ src: cs.src! }));
|
|
347
|
+
clip.start = cs.start; clip.end = cs.end;
|
|
348
|
+
applyProp(clip.transform.position, cs.position);
|
|
349
|
+
applyProp(clip.opacity, cs.opacity);
|
|
350
|
+
for (const e of cs.effects ?? []) clip.effects.push(buildEffect(e));
|
|
351
|
+
track.add(clip);
|
|
352
|
+
}
|
|
353
|
+
c.addTrack(track);
|
|
354
|
+
return { compositor: c };
|
|
355
|
+
});
|
|
356
|
+
}
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
Feed it your JSON and you get a `Composer` that previews, exports and
|
|
360
|
+
server-renders like any other composition:
|
|
361
|
+
|
|
362
|
+
```ts
|
|
363
|
+
export default fromSpec(JSON.parse(await loadAsset('./timeline.json')));
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
**Why the SDK doesn't ship this for you:** any real product already has a document
|
|
367
|
+
model (undo, collaboration, its own field names); an official spec would just be a
|
|
368
|
+
second schema to fight. The fuller reference implementation — video/audio clips,
|
|
369
|
+
`sourceIn`/`sourceOut`, blend modes, a global grade — lives in
|
|
370
|
+
`packages/server/src/timeline.ts` (the `TimelineSpec` type + `buildTimeline`). Read
|
|
371
|
+
it for the complete pattern, then write your own.
|
|
372
|
+
|
|
373
|
+
**Trust boundary:** a `RuntimeBundle` is *code* — re-running it executes arbitrary
|
|
374
|
+
TS/JS on your server. Serve bundles only from trusted (first-party) sources. For a
|
|
375
|
+
multi-tenant or public-facing service, accept *data* (a tenant's JSON + your
|
|
376
|
+
builder above), never their code, and allowlist any media `src` URL your builder
|
|
377
|
+
fetches. See `docs/server-side-rendering.md` § 信任边界与安全模型.
|
|
378
|
+
|
|
379
|
+
## Outlined / echoed / italic display text
|
|
380
|
+
|
|
381
|
+
`TextStyleLike` passes weight, italic, letter-spacing and a stroke straight into
|
|
382
|
+
Pixi's text style — so one family covers bold / hollow / italic cuts:
|
|
383
|
+
|
|
384
|
+
```ts
|
|
385
|
+
// Hollow outlined word: pale fill + a coloured stroke.
|
|
386
|
+
new TextClip({ text: 'SUPER', fontFamily: 'Oswald', fontSize: 150, fill: 0xf7cfcf,
|
|
387
|
+
fontWeight: '700', letterSpacing: 2, stroke: { color: 0xa5120f, width: 4 } });
|
|
388
|
+
|
|
389
|
+
// Bold italic call-to-action.
|
|
390
|
+
new TextClip({ text: 'MAKE IT YOURS', fontFamily: 'Oswald', fontSize: 96,
|
|
391
|
+
fill: 0xa5120f, fontWeight: '700', fontStyle: 'italic' });
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
An "echo" stack (fading after-image) is N copies with decreasing opacity:
|
|
395
|
+
|
|
396
|
+
```ts
|
|
397
|
+
const g = new GroupClip();
|
|
398
|
+
for (let i = 0; i < 4; i++) {
|
|
399
|
+
const t = new TextClip({ text: 'PROMO', fontFamily: 'Oswald', fontSize: 150, fill: 0xffffff, fontWeight: '700' });
|
|
400
|
+
t.start = 0; t.end = 3;
|
|
401
|
+
t.transform.anchor.setStatic([0, 0]);
|
|
402
|
+
t.transform.position.setStatic([0, i * 205]);
|
|
403
|
+
t.opacity.setStatic(Math.pow(0.7, i)); // each copy fainter
|
|
404
|
+
g.add(t);
|
|
405
|
+
}
|
|
406
|
+
```
|
|
407
|
+
|
|
408
|
+
## Arch / circle-cropped photo (`maskShape`)
|
|
409
|
+
|
|
410
|
+
Clip an image to a rounded-rect (arch) or ellipse. Mask a **`GroupClip`** that
|
|
411
|
+
wraps the image — a Sprite can't be masked by its own child — and lay the image
|
|
412
|
+
out from `(0, 0)` in the group, sized to fill the mask:
|
|
413
|
+
|
|
414
|
+
```ts
|
|
415
|
+
const src = new ImageSource({ src: 'https://…?w=760&h=1500&fit=crop' }); // pre-crop to region aspect
|
|
416
|
+
const meta = await src.load();
|
|
417
|
+
const img = new ImageClip(src);
|
|
418
|
+
img.start = 0; img.end = 3;
|
|
419
|
+
img.transform.anchor.setStatic([0, 0]);
|
|
420
|
+
img.transform.scale.setStatic([760 / meta.width, 1500 / meta.height]);
|
|
421
|
+
|
|
422
|
+
const arch = new GroupClip();
|
|
423
|
+
arch.start = 0; arch.end = 3;
|
|
424
|
+
arch.transform.anchor.setStatic([0, 0]);
|
|
425
|
+
arch.transform.position.setStatic([170, -60]); // where the arch sits
|
|
426
|
+
arch.maskShape = { kind: 'rect', width: 760, height: 1500, radius: 380 }; // big radius → arch
|
|
427
|
+
arch.add(img);
|
|
428
|
+
```
|
|
429
|
+
|
|
184
430
|
## Gotchas
|
|
185
431
|
|
|
186
432
|
- Call `await compositor.init()` before adding tracks.
|