@algorisys/tinyfly 0.101.0 → 0.103.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (59) hide show
  1. package/README.md +17 -6
  2. package/bin/tinyfly.mjs +9 -3
  3. package/docs/DEPLOYMENT.md +2 -0
  4. package/docs/api-reference.md +61 -0
  5. package/docs/dance.md +150 -1
  6. package/docs/editor-guide.md +11 -4
  7. package/docs/examples.md +22 -1
  8. package/docs/getting-started.md +1 -1
  9. package/docs/gsap-compat.md +1 -1
  10. package/docs/llm-guide.md +38 -3
  11. package/docs/maps.md +2 -2
  12. package/docs/scene-3d.md +138 -6
  13. package/docs/teaching.md +1 -1
  14. package/docs/video-rendering.md +44 -1
  15. package/lib/addons/adapters.js +168 -123
  16. package/lib/addons/characters.js +5802 -5506
  17. package/lib/addons/scene-3d-webgl.js +30 -30
  18. package/lib/addons/scene-3d.js +650 -341
  19. package/lib/browser/tinyfly-scene-3d-webgl.iife.js +2 -2
  20. package/lib/browser/tinyfly-scene-3d.iife.js +2 -2
  21. package/lib/browser/tinyfly.iife.js +14 -14
  22. package/lib/browser/tinyfly.js +5798 -5467
  23. package/lib/browser/tinyfly.umd.js +14 -14
  24. package/lib/engine/tinyfly-engine.js +1207 -1172
  25. package/lib/engine/tinyfly-engine.umd.cjs +2 -2
  26. package/lib/headless/headless.js +150 -6
  27. package/lib/types/adapters/canvas/framing.d.ts +27 -0
  28. package/lib/types/adapters/canvas/index.d.ts +1 -0
  29. package/lib/types/characters/capabilities.d.ts +1 -0
  30. package/lib/types/characters/character-script-3d.d.ts +34 -4
  31. package/lib/types/characters/dance-circle.d.ts +59 -0
  32. package/lib/types/characters/dance-contact.d.ts +52 -0
  33. package/lib/types/characters/human-dance.d.ts +86 -0
  34. package/lib/types/characters/index.d.ts +5 -1
  35. package/lib/types/characters/rig/reach.d.ts +6 -3
  36. package/lib/types/characters/staging-3d.d.ts +56 -0
  37. package/lib/types/engine/authoring/framing.d.ts +48 -0
  38. package/lib/types/engine/authoring/index.d.ts +1 -0
  39. package/lib/types/headless/audio-mix.d.ts +40 -0
  40. package/lib/types/headless/ffmpeg-args.d.ts +2 -0
  41. package/lib/types/headless/frame-renderer.d.ts +7 -1
  42. package/lib/types/headless/index.d.ts +1 -0
  43. package/lib/types/headless/node-audio.d.ts +19 -0
  44. package/lib/types/headless/node-video.d.ts +2 -0
  45. package/lib/types/headless/video-scene.d.ts +12 -0
  46. package/lib/types/scene-3d/index.d.ts +3 -0
  47. package/lib/types/scene-3d/lighting.d.ts +58 -0
  48. package/lib/types/scene-3d/resolve-scene.d.ts +5 -1
  49. package/lib/types/scene-3d/scene-3d-adapter.d.ts +2 -0
  50. package/lib/types/scene-3d/scene-types.d.ts +14 -2
  51. package/lib/types/scene-3d/shots.d.ts +162 -0
  52. package/lib/types/scene-3d/software-renderer.d.ts +40 -0
  53. package/llms-full.txt +474 -23
  54. package/llms.txt +29 -29
  55. package/package.json +1 -1
  56. package/skills/tinyfly/SKILL.md +32 -1
  57. package/lib/types/characters/polyline.d.ts +0 -12
  58. package/lib/types/characters/sketch.d.ts +0 -54
  59. package/lib/types/compat/gsap/playhead-crossings.d.ts +0 -47
package/docs/scene-3d.md CHANGED
@@ -95,8 +95,11 @@ for a neon glow: `{ color: '#000000', emissive: '#ff2bd6', shading: 'unlit' }`.
95
95
  `roughness` and `metalness` are for the realistic look (three.js, a later
96
96
  milestone); the stylized renderers ignore them, so one material serves both.
97
97
 
98
- The scene can have `background` and `fog: { color, near, far }` (metres from
99
- the camera). A scene with no lights gets a soft default (ambient plus a key
98
+ The scene can have `background` (a colour, or a sky as a vertical gradient
99
+ `{ top, bottom }`), `fog: { color, near, far }` (metres from the camera) and
100
+ `ground` (the height blob shadows lie at, default 0). A mesh with `shadow:
101
+ true` (or `{ opacity, ground }`) gets a soft blob shadow on the ground under
102
+ it, which spreads and fades as the mesh lifts off. A scene with no lights gets a soft default (ambient plus a key
100
103
  light from upper right). Outlines fade into the fog too. `nearFade: { from, to }` fades geometry
101
104
  nearer the camera than `to` metres, gone at `from`, so a dive passes through
102
105
  things as if they were glass (also an option of `resolveScene3D`).
@@ -129,6 +132,85 @@ so instead of picking a winner.
129
132
  the group. `orbitPosition(target, yaw, pitch, distance)` and
130
133
  `dollyPosition(eye, target, amount)` place cameras for keyframes.
131
134
 
135
+ ## Shots and cameras
136
+
137
+ A film is a shot list, as data, over the same scene and timeline:
138
+
139
+ ```js
140
+ import { shotTracks, shotMarkers, transitionAt, withActiveCamera, cameraMoveTracks, cameraHandheldTracks } from '@algorisys/tinyfly/scene-3d'
141
+ import { drawTransition, drawLetterbox } from '@algorisys/tinyfly/adapters'
142
+
143
+ const shots = [
144
+ { id: 'establish', start: 0, end: 4000, camera: 'wide', label: 'Dolly in' },
145
+ { id: 'orbit', start: 4000, end: 7500, camera: 'orbiter', transition: { kind: 'crossfade', duration: 700 } },
146
+ { id: 'close', start: 7500, end: 10000, camera: 'close', transition: { kind: 'dipToBlack', duration: 600 } },
147
+ ]
148
+ const tracks = [
149
+ ...shotTracks(shots, 'set'), // the activeCamera track: a cut is one keyframe
150
+ ...cameraMoveTracks({ scene: 'set', camera: 'wide', start: 0, end: 4000,
151
+ from: { target: [0, 0.8, 0], yaw: 20, pitch: 18, distance: 11 }, to: { distance: 7 } }), // a dolly
152
+ ...cameraHandheldTracks([], { scene: 'set', camera: 'close', start: 7500, end: 10000,
153
+ base: { position: [1.2, 1.4, 3.6], lookAt: [0, 0.75, 1.3] }, seed: 'close' }),
154
+ ]
155
+ ```
156
+
157
+ - **Shots** are `{ id, start, end, camera, label?, transition? }`;
158
+ `checkShots` names overlaps, bad ends and a transition on the first shot;
159
+ `shotAt(shots, time)` is the shot playing; `shotMarkers` gives the
160
+ timeline's markers (stills and captions step through them).
161
+ - **Transitions** (`cut`, `crossfade`, `dipToBlack`, `wipe` with a
162
+ `direction`) start at the shot's start and last `duration` ms. Any frame can
163
+ ask `transitionAt(shots, time)` which two shots to draw and how far along
164
+ (0..1); `drawTransition(ctx, transition, { width, height, draw(ctx,
165
+ cameraId) })` draws both and blends them (a crossfade mixes, a dip covers
166
+ the cut, a wipe sweeps the new shot in), with `withActiveCamera(values,
167
+ sceneId, cameraId)` to draw the shot going out. Pure and frame-addressable:
168
+ the film renders the same in any order.
169
+ - **Camera moves** are presets that write `position`, `lookAt` (and `fov`)
170
+ tracks on the camera: `cameraMoveTracks({ from, to, easing, every })` with
171
+ poses as an orbit round a target (`yaw`, `pitch`, `distance`), or a fixed
172
+ `position` with a moving `lookAt`, or a `fov`: change `yaw` for an orbit,
173
+ `distance` for a dolly, `pitch` and `distance` for a crane, `lookAt` for a
174
+ pan or tilt, `fov` for a zoom, or any mix. `cameraFollowTracks({ subjectAt,
175
+ offset, lag })` keeps a camera on a moving subject. `cameraHandheldTracks(
176
+ tracks, { amount, aim, rate, seed })` adds a deterministic handheld shake
177
+ onto a move's tracks or a still camera's `base`.
178
+ - **Aspect and safe areas**: `drawLetterbox(ctx, canvas, '2.39:1')` paints
179
+ the bars (in a video's `overlay`), `frameRect`, `frameSize` and `safeAreas`
180
+ are the geometry, `drawSafeAreas` draws the action and title guides for
181
+ stills. `FRAME_ASPECTS` has 16:9, 2.39:1, 4:3, 1:1, 9:16 and 4:5.
182
+
183
+ `examples/headless-video/film-shots.mjs` is a three-shot film with a dolly, a
184
+ crossfade into an orbit, a dip to black into a handheld close-up, letterboxed.
185
+
186
+ ## Sets and light for mood
187
+
188
+ A set is objects like any other; what sets the mood is its light, its sky and
189
+ its fog, and those come as presets:
190
+
191
+ ```js
192
+ import { lightingPreset, lightingTracks, ground } from '@algorisys/tinyfly/scene-3d'
193
+
194
+ const light = lightingPreset('dusk') // or 'day', 'night', 'stage'
195
+ const floor = ground({ size: 20 })
196
+ const scene = {
197
+ id: 'set', camera: 'cam', background: light.background, fog: light.fog,
198
+ materials: { ...floor.materials, red: { color: '#ef4444', shading: 'toon' } },
199
+ objects: [camera, ...light.objects, ...floor.objects, { id: 'box', kind: 'mesh', geometry: { type: 'box', size: [1, 1, 1] }, material: 'red', position: [0, 0.5, 0], shadow: true }],
200
+ }
201
+ // Day turning to night over four seconds: tracks on the same lights.
202
+ const tracks = lightingTracks('day', 'night', { scene: 'set', start: 0, end: 4000 })
203
+ ```
204
+
205
+ Every preset uses the same light ids (`key`, `fill`, `sky`, and `spot` on
206
+ the stage), so `lightingTracks(from, to, { scene, start, end })` keys each
207
+ light's `color`, `intensity`, `position` and `target` from one mood to the
208
+ other (a spot only one mood has fades in or out); the sky and fog are the
209
+ scene's to set per shot. `ground({ size, color, segments, y })` is a floor
210
+ on layer −1 with its material; props from the catalog (trees, houses,
211
+ furniture) are the set pieces, placed by preset name (see Props). Blob
212
+ shadows under meshes are `shadow: true`; projected shadows are planned.
213
+
132
214
  ## Lines and trails
133
215
 
134
216
  Two more kinds of object draw a band that faces the camera, as wide as
@@ -174,7 +256,7 @@ front of the meshes. For trails on a plain 2D canvas, see `drawTrail` in
174
256
  loadScene3D(json): LoadedScene3D // validates and builds meshes once (throws with every problem)
175
257
  validateScene3D(json, tracks?): string[] // problems as sentences, empty when fine
176
258
  resolveScene3D(loaded, values, { width, height, time?, valuesAt? }): ResolvedScene3D // pure: matrices, camera, lights, sorted triangles
177
- drawScene3D(ctx, loaded, values, { width, height, time?, valuesAt? | stateAt? }) // resolve and draw, in one call
259
+ drawScene3D(ctx, loaded, values, { width, height, time?, valuesAt? | stateAt?, depthBuffer? }) // resolve and draw, in one call
178
260
  new Scene3DAdapter(new Canvas2DRenderer(ctx), { width, height, valuesAt? }) // registerScene, applyState(state), render()
179
261
  ```
180
262
 
@@ -244,7 +326,8 @@ A character faces +z before any turn and stands on its object's ground: its
244
326
  lowest point at the object's y, so at y 0 it stands on a floor at y 0. Its
245
327
  pose fields are tracks on its object, next to `x`, `rotateY` and the rest:
246
328
  `stage/tum` `arm.right.spread`, `stage/tum` `turn`. Every dance and flip
247
- plays in 3D: `stickToHuman(danceFrame(style, beat))` gives the pose, and
329
+ plays in 3D: `const frame = danceFrame(style, beat)` followed by
330
+ `stickToHuman(frame.pose, frame.hands)` gives the pose, and
248
331
  `mirrorHumanPose()` its mirror image for a partner. Its `turn`, `roll`,
249
332
  `lift` and ground contact behave as they do front-on; seen front-on through a
250
333
  flat camera, a character is exactly the 2D figure.
@@ -252,6 +335,25 @@ flat camera, a character is exactly the 2D figure.
252
335
  **Walking in metres.** `characterScript3D(id, beats, { scene, position, heading, height })`
253
336
  scripts a character in world metres, as `propScript3D` does props:
254
337
 
338
+ It also accepts `{ do: 'spin', turns: 2, for: 1500 }` (signed full turns,
339
+ linear angular motion) and `{ do: 'dance', dance: 'garba', bpm: 120, beats: 8 }`.
340
+ `face` still takes the shortest turn and can use `for` to set its duration.
341
+ An explicit duration is consumed even when no turn is needed. `spin`
342
+ defaults to one revolution over 1000 ms; negative or fractional `turns`
343
+ are allowed. A native `dance` stays in place, using the style's tempo and
344
+ beat count by default. With only `for`, beats follow the BPM; with both
345
+ `beats` and `for`, those beats are fitted into that duration.
346
+ For circle formations with planted toes and independent facing/spins use
347
+ `danceCircleTracks(['stage/a', 'stage/b'], { radius: 1.8, beatsPerTurn: 32,
348
+ beats: 32, bpm: 120 })`. These write ordinary world-position, `rotateY` and
349
+ native pose tracks. For nonzero `pose.roll`, heading uses pose `turn`
350
+ (quarter-turns) with `rotateY: 0`, preserving native yaw before roll and
351
+ grounding. Supply roll when baking and apply all returned tracks together.
352
+ Use the same character build and base pose for baking and rendering; omitted
353
+ base/rest fields are not reset. Partner stick contacts require `meetDandiya`
354
+ and drawing its returned segments separately.
355
+ See [Native character dances](dance.md#native-character-dances-circles-and-spins).
356
+
255
357
  ```js
256
358
  import { characterScript3D } from '@algorisys/tinyfly/characters'
257
359
 
@@ -272,6 +374,21 @@ phase) with the distance, one stride per cycle, so the feet stay planted, plus
272
374
  the stepping pose as it draws, on the pose it holds: a wave carries on while it
273
375
  walks. Objects take `gait` and `walking` as fields too.
274
376
 
377
+ **Speaking and looking.** `{ do: 'say', say: 'A pot of water!' }` lip-syncs
378
+ the line on the character's mouth (the same `lipSyncOver` the 2D scripts use)
379
+ with a talking head, and lasts as long as the line (`speechLength(text)`, or
380
+ `for`). `{ do: 'look', look: [x, z] }` turns the head and eyes toward a point
381
+ at eye height (`[x, y, z]` for any height, `'camera'` for the viewer with
382
+ `options.camera`); `lookFields(from, at)` is the maths. To keep looking at
383
+ something that moves, `characterLookAtTracks(tracks, { scene, character,
384
+ at: { object: 'didi' } | { camera: 'cam' } | [x, y, z] | (time) => [x, y, z],
385
+ start, end, objects })` keys `head.turn`, `head.nod`, `lookX` and `lookY`
386
+ from where both are at every moment; `objectPositionAt(tracks, target,
387
+ base)` is the sampler it uses. For several figures, `stageSpots({ count,
388
+ shape: 'line' | 'arc' | 'circle' | 'cluster', spacing, centre })` gives
389
+ positions and headings that do not overlap: a line across, an arc facing its
390
+ middle, a circle facing in, or a seeded loose group.
391
+
275
392
  **Riding.** `propRide3D({ prop, propId, propTracks, scene, placement, anchor, riderId, pose, start, end, mount?, dismount? })`
276
393
  keeps a rider's hips on a prop's seat anchor (a horse's `saddle`, a bike's or
277
394
  a cart's `seat`) as the prop moves, turns, bobs with its stride and pitches,
@@ -440,8 +557,9 @@ renderer.render(resolveScene3D(scene, values, { width: canvas.width, height: can
440
557
  trails, go on `overlay`, a 2D canvas laid over the GL one (in front of the meshes);
441
558
  solid characters are meshes and sort properly.
442
559
 
443
- It runs on a page or in a Worker (OffscreenCanvas). Node has no WebGL2, so
444
- headless video uses the Canvas 2D renderer. The cross-browser checks
560
+ It runs on a page or in a Worker (OffscreenCanvas). Node has no WebGL2;
561
+ headless video uses the Canvas 2D renderer, or its software depth buffer
562
+ (below) when shapes cut through each other. The cross-browser checks
445
563
  (`npm run e2e -- --check scene-3d-webgl`) compile it in Chromium, Firefox and
446
564
  WebKit, compare its silhouettes with the Canvas 2D renderer's and check the
447
565
  depth buffer.
@@ -457,5 +575,19 @@ modest, or use a layer. Each triangle has one colour, so smooth shading is
457
575
  smoothed per triangle, not per pixel; more segments look rounder. The WebGL2
458
576
  renderer (above) has a depth buffer and per-pixel light.
459
577
 
578
+ **A software depth buffer** draws right where the painter's order cannot:
579
+ `drawScene3D(ctx, scene, values, { …, depthBuffer: true })`, or
580
+ `new Scene3DAdapter(new SoftwareRenderer(ctx), …)`. Every triangle is
581
+ rasterised into a pixel buffer with a depth per pixel (`rasterizeResolvedScene`
582
+ gives the bytes and depths without a canvas), so crossing beams, huge floors
583
+ and long meshes need no `layer` or segments; see-through meshes blend over
584
+ what is behind them, and ink is depth-tested so an outline never shows
585
+ through what is in front. Each triangle keeps its one shaded colour and its
586
+ fog, edges are hard (no anti-aliasing), and objects drawn whole (characters
587
+ in their pen looks, lines, trails) go on top of the meshes, as on the WebGL2
588
+ overlay. It costs about three times the painter's time and works anywhere
589
+ the Canvas 2D renderer does. Layers keep their meaning: a lower layer is
590
+ drawn first and a higher one paints over it whatever the depths.
591
+
460
592
  Rendering speed in Node: a 1280×720 scene of a few thousand triangles renders
461
593
  at a few frames a second; parallel rendering for long films is planned.
package/docs/teaching.md CHANGED
@@ -25,7 +25,7 @@ path for all of that:
25
25
  </figure>
26
26
 
27
27
  <!-- once, anywhere on the page (a site-wide footer is fine) -->
28
- <script src="https://cdn.jsdelivr.net/gh/algorisys-oss/tinyfly@v0.101.0/cdn/tinyfly-embed.iife.js" data-tinyfly-auto></script>
28
+ <script src="https://cdn.jsdelivr.net/gh/algorisys-oss/tinyfly@v0.103.0/cdn/tinyfly-embed.iife.js" data-tinyfly-auto></script>
29
29
  ```
30
30
 
31
31
  Every `[data-tinyfly-embed]` mounts when the page is ready, with no per-post
@@ -14,6 +14,7 @@ npx tinyfly video scene.mjs --stills stills/ # one PNG per caption line (els
14
14
  npx tinyfly video scene.mjs --stills stills/ --frames 0,79,88 # stills at chosen frames (or --times 0,2640 in ms)
15
15
  npx tinyfly video scene.mjs --loop-check # does it loop? (exit 1 when not)
16
16
  npx tinyfly video scene.mjs --scale 0.5 --fps 12 # quick preview
17
+ npx tinyfly video scene.mjs --codec prores # a ProRes 422 HQ .mov for editing elsewhere
17
18
  ```
18
19
 
19
20
  Runnable scenes are in [`examples/headless-video/`](../examples/headless-video/):
@@ -142,6 +143,38 @@ Scene modules may use top-level `await`, or default-export an async function.
142
143
  `assembleNarration(plan, clips, sampleRate)` and `encodeWav(samples, sampleRate)`
143
144
  are the pure steps underneath, for audio you generate yourself.
144
145
 
146
+ ### A sound mix
147
+
148
+ A film's sound is layers: a music bed, the narration, effects at the moments
149
+ things happen, with the music ducked under the voice. `renderMix` decodes
150
+ every file with ffmpeg, mixes them and writes one WAV for the scene's
151
+ `audio`:
152
+
153
+ ```js
154
+ import { renderMix } from '@algorisys/tinyfly/headless'
155
+
156
+ const audio = await renderMix(
157
+ [
158
+ { id: 'bed', file: 'music/bed.mp3', loop: true, gain: 0.6, fadeOut: 1500, duck: { under: 'voice', amount: 0.7 } },
159
+ { id: 'voice', file: 'build/narration.wav' },
160
+ { file: 'sfx/tap.wav', at: [1200, 2650, 4100], gain: 0.8 }, // a sound effect at three moments
161
+ { file: 'sfx/thud.wav', at: landings.map((t) => t + 40) }, // or at every landing a script reports
162
+ ],
163
+ { output: 'build/mix.wav', duration: END, baseDir: here }
164
+ )
165
+ export default { /* … */ audio }
166
+ ```
167
+
168
+ Each layer has `start` (ms) or `at` (times for a sound effect), `gain`,
169
+ `fadeIn` and `fadeOut`, `loop` for a bed shorter than the film, and `duck:
170
+ { under, amount, attack, release }` to make way for another layer while it
171
+ sounds. The whole mix is soft-limited so peaks never clip. `mixAudio(layers,
172
+ { sampleRate, duration, master })` is the pure step on samples. For cuts and
173
+ dances on the music's beat, `detectTempo(samples, sampleRate)` finds the
174
+ grid; `snapShotsToBeats(shots, grid)` moves each cut to the nearest beat,
175
+ and `danceTracks(target, { bpm: grid.bpm, start: nextBeat(grid, t) })`
176
+ dances to it.
177
+
145
178
  ## Characters
146
179
 
147
180
  `@algorisys/tinyfly/characters` has a poseable stick figure. A pose is a set of
@@ -798,6 +831,12 @@ draw(ctx, frame) {
798
831
  stay put while the camera moves.
799
832
  - `drawingRate: 12` holds the timeline's poses for two frames of 24 fps ("on
800
833
  twos"), as hand-drawn animation is timed.
834
+ - `motionBlur: { samples: 4, shutter: 0.5 }` makes each frame the average of
835
+ four drawings spread over the shutter's open time (half the frame, a 180°
836
+ shutter): fast moves streak as a camera's do. Deterministic, `samples`
837
+ times the work.
838
+ - `supersample: 2` draws at twice the size and scales down: smoother edges
839
+ on thin lines and small text (four times the work).
801
840
 
802
841
  See [acting.md](acting.md) for camera shots, beat scripts and the rest of the
803
842
  cartoon acting tools.
@@ -822,6 +861,7 @@ await renderVideo(scene, {
822
861
  output: 'out.mp4',
823
862
  baseDir: '.', // relative audio and font paths resolve against this
824
863
  scale: 1, crf: 20, preset: 'medium',
864
+ codec: 'h264', // or 'prores' for a .mov
825
865
  onProgress: (done, total) => {},
826
866
  })
827
867
  await renderStills(scene, { dir: 'stills' })
@@ -878,5 +918,8 @@ room) was ported call for call to Canvas 2D and rendered at 1920×1080, 24 fps,
878
918
  rendering) and `ffmpeg`. Rendering defines `globalThis.Path2D` from the canvas
879
919
  package when it is missing, because path targets use it.
880
920
  - Output is H.264 MP4 (`yuv420p`, odd sizes padded to even), with AAC audio when
881
- `audio` is set; video stops at the shorter of picture and sound.
921
+ `audio` is set; video stops at the shorter of picture and sound. `codec:
922
+ 'prores'` (or `--codec prores`) writes ProRes 422 HQ (10-bit, PCM audio) in
923
+ a `.mov` for an editor; any size works, 1080p and 4K included; `renderStills`
924
+ gives a PNG sequence.
882
925
  - `image` targets need images loaded with the canvas package's `loadImage`.