kasaneru 0.0.0-stage → 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,18 @@
1
+ # Changelog
2
+
3
+ All notable changes to `kasaneru` are documented in this file.
4
+
5
+ ## [0.0.1] - 2026-11-21
6
+
7
+ First public release.
8
+
9
+ ### Added
10
+
11
+ - CLI: `render`, `still`, `check`, `inspect`, `sheet`, `srt`, `captions`, `probe`, `envelope`.
12
+ - Manifest layers: `video`, `scene`, `adjust`, `captions`, plus `audio` and `master`.
13
+ - Rendering backends: `browser` (HTML strings, React, in-page scripts such as three.js) and `flutter`.
14
+ - Helper modules for scripts that build manifests: `kasaneru/anim`, `kasaneru/media`, `kasaneru/subtitles`, `kasaneru/srt`.
15
+
16
+ ### Security
17
+
18
+ - The local file server used by `scenes.page` scenes serves only files under the manifest's `assets` folder, and only to requests addressed to its own `127.0.0.1` address.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 K9i
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.ja.md ADDED
@@ -0,0 +1,274 @@
1
+ # kasaneru
2
+
3
+ [English](README.md)
4
+
5
+ manifest(JSON)から動画を書き出すツール。名前は「重ねる」から。層を重ねて動画を作る。
6
+
7
+ - 動画・音声の切り貼りと合成は `ffmpeg`(外部コマンド)
8
+ - 文字・図形・3Dは「描画バックエンド」が画像にして渡す。同じ絵になるコマは1回だけ描く
9
+ - 描画バックエンドは2種類。どちらも画面は出さない
10
+ - `browser`: 手元のChromeをPlaywright経由で動かして撮る。HTML文字列、React、ページ内スクリプト(three.jsなど)を描ける
11
+ - `flutter`: `flutter test` の中でウィジェットを `RepaintBoundary.toImage` で取り出す
12
+
13
+ 動作を確かめているのはmacOSだけ。WindowsとLinuxでは動かしていない。
14
+
15
+ ## 使い方
16
+
17
+ npmから入れる(`npx kasaneru` でそのまま実行もできる):
18
+
19
+ ```bash
20
+ npm install kasaneru
21
+ npx kasaneru check manifest.json
22
+ npx kasaneru render manifest.json -o out.mp4
23
+ ```
24
+
25
+ 例はnpmのパッケージに入っていない。試すときはリポジトリをクローンする:
26
+
27
+ ```bash
28
+ git clone https://github.com/K9i-0/kasaneru.git
29
+ cd kasaneru && npm install
30
+ node examples/basic/make-assets.mjs # 例の素材をffmpegで作る(外部の素材は不要)
31
+ node bin/kasaneru.mjs check examples/basic/manifest.json
32
+ node bin/kasaneru.mjs render examples/basic/manifest.json -o out/basic.mp4
33
+ node bin/kasaneru.mjs still examples/basic/manifest.json --frame 100 -o out/basic_100.png
34
+ node bin/kasaneru.mjs inspect out/basic.mp4
35
+ ```
36
+
37
+ | コマンド | 内容 |
38
+ |---|---|
39
+ | `check <manifest>` | 書き出さずに、manifestの誤りを場所つきで全部挙げる(JSON。誤りがあれば終了コード1) |
40
+ | `render <manifest> -o <mp4>` | 書き出す。先に `check` と同じ検証をして、誤りがあれば書き出さない |
41
+ | `still <manifest> --frame N -o <png>` | 1コマだけ書き出す。その1コマに要る絵しか描かないので速い |
42
+ | `inspect <動画>` | 尺・コマ数・音量と、先頭コマが暗くないか・音が振り切れていないかを報告する |
43
+ | `inspect <動画> --deep` | 上に加えて、1コマだけ壊れた絵(真っ白・崩れ)を探す(`suspectFrames`)。先頭と最後のコマは片側としか比べられないので、隣と大きく違うときに弱い注意として挙げる(`edgeFrames`。カットとは区別できないので、見て確かめる) |
44
+ | `sheet <動画> -o <png>` | 確認用のコマ一覧を作る。`--frames 0,120,240`、`--every N`、`--manifest <manifest>`(クリップや字幕の切り替わりの前後を自動で選ぶ)、`--cols N`(列数、既定は4) |
45
+ | `srt <manifest> -o <srt>` | 字幕をSRTにする(`captions` の層から。`--key props.captions` で場所を指定もできる) |
46
+ | `captions <字幕ファイル> --fps N` | SRT・ASSを `[{from, to, text}]`(コマ数)のJSONにする |
47
+ | `probe <素材> [--fps N]` | 素材の長さ(コマ数)・大きさ・音声の有無を調べる |
48
+ | `envelope <音声> --fps N` | コマごとの音量を配列で出す(口パクなど) |
49
+
50
+ `render` の主な指定:
51
+
52
+ - `--backend browser|flutter|none`(既定 `browser`。`none` は動画層だけ)
53
+ - 出力の形式は拡張子で決まる: `.mp4`(H.264)、`.mov`(ProRes 422 HQ)、`.webm`(VP9)、`.gif`
54
+ - `--frames A-B` は一部のコマだけ書き出す(確認用)
55
+ - `--props <ファイルかJSON>` と `--set 場所=値` は、manifestの `props` を差し替える(同じmanifestから文言違いなどの変種を作る)
56
+ - `--concurrency N` は同時に描くページ数(`browser`、既定4)
57
+ - `--encoder h264_videotoolbox` はmacのハードウェアエンコーダを使う
58
+ - `--crf N` は、既定のエンコーダでの `.mp4` と `.webm`(VP9の範囲に換算する)の画質(既定は18。小さいほど高画質でファイルが大きい)
59
+ - `--every-frame` は、同じ絵を1回だけ描くのをやめて全コマ分の絵を描く(確認用)
60
+ - `--audio-bitrate 256k` は音声のビットレートを変える(既定は `.mp4` が192k、`.webm` が160k。`.mov` は無圧縮なので効かない)
61
+ - `--allow-missing-assets` は、シーンの素材が読めなくても書き出す(下の「読めない素材」)
62
+ - `--keep-work` は作業用フォルダ(描いた絵とffmpegのフィルタ)を消さずに残す。`--work <dir>` で置き場所を指定できる(既定は出力の隣の `<出力名>.work`)
63
+
64
+ 結果は、所要時間の内訳と注意点をJSONで標準出力へ返す。
65
+
66
+ ### 読めない素材
67
+
68
+ `browser` で描くシーンの画像が読めないとき(ファイルがない・壊れている・読み込みが30秒で終わらない)は、絵を全部描いたあと、動画を作る前に失敗にする(終了コード1)。読めなかったURL・最初に気づいた層とコマ・何枚の絵で起きたかを標準エラーへ出す。
69
+
70
+ - 見ているもの: `<img>`(読み込み後の `naturalWidth` が0)と、ページが出した読み込みの失敗(CSSの背景画像、SVGの `<image>`、フォントなど)。`scenes.page` のシーンは、`ctx.assets` から読もうとしてなかったファイル
71
+ - `--allow-missing-assets` を付けると失敗にせず、結果のJSONの `warnings` に同じ内容を入れて書き出す
72
+ - 待つ長さは環境変数 `KASANERU_IMAGE_TIMEOUT_MS` で変えられる。一度時間切れした画像は、次の絵からは待たずに失敗として数える
73
+ - 読み込みの途中で取り消されたもの(次のコマで要らなくなった背景画像など)は、失敗に数えない
74
+ - `<img>` 以外は読み込みを待たずに撮っている。手元のファイルはすぐ読めるので実害は出ていないが、背景画像が撮影に間に合わない・失敗の報告が間に合わないことはありうる。確実に待たせたい画像は `<img>` で置く
75
+ - `scenes.page` のシーンで起きたページ内のエラー(`pageerror`)は、失敗にはせず `warnings` に入れる
76
+
77
+ ### 必要なものと注意
78
+
79
+ 必要なもの: Node 22、ffmpeg、Google Chrome(`browser`)、Flutter(`flutter`)。テストは `npm test`。
80
+
81
+ **信頼できるmanifestだけを描画すること。** manifestはシーンのファイル(`scenes.browser` `scenes.page` `scenes.flutter`)を指し、kasaneruはそれを実行する(手元のNodeのコードとして、ブラウザの中で、または `flutter test` の中で)。他人のmanifestを描画すると、その人のコードが自分の権限で動く。素材のパスも、manifestが指す場所からそのまま読む。
82
+
83
+ `flutter` は、生成したエントリポイントとビルドのキャッシュをkasaneru自身の `flutter_backend/` フォルダに書く。そのため、kasaneruを入れたフォルダに書き込める必要がある(ローカルの `npm install` なら書ける。root所有のシステム全体へのインストールでは書けない)。
84
+
85
+ ブラウザは環境変数 `KASANERU_BROWSER` で変えられる(実行ファイルのパスか、`chrome-beta` `msedge` `chromium` などの名前)。
86
+
87
+ ## manifest
88
+
89
+ 時刻はすべてコンポジションのfpsでのコマ数(整数)で書く。パスはmanifestの場所からの相対。
90
+
91
+ | 項目 | 内容 |
92
+ |---|---|
93
+ | `composition` | `width` `height` `fps` `durationInFrames`。`durationInFrames` を省くと、いちばん遅く終わるクリップ・音声に合わせる |
94
+ | `background` | 背景色(`#RRGGBB`) |
95
+ | `layers` | 下から順に重ねる層。`video` / `scene` / `captions` / `adjust` |
96
+ | `audio` | 音声トラックの配列 |
97
+ | `master` | 音声の仕上げ。`limiter`(上限、0〜1)と `loudnorm`(`i` `tp` `lra`) |
98
+ | `scenes` | シーンの場所。`browser`(HTML/React)、`page`(ページ内スクリプト)、`flutter`(Dartファイル) |
99
+ | `assets` | `page` のシーンから読む素材のフォルダ(省略時はmanifestのあるフォルダ)。ページに渡すのはこのフォルダの下のファイルだけで、外のファイルを読もうとすると「読めない素材」として失敗する |
100
+ | `fonts` | 読み込むフォントファイル(`family` `file`、必要なら `weight` `style`)。システムにないフォントを使うとき |
101
+ | `props` | シーンへ渡す値 |
102
+
103
+ ### video層
104
+
105
+ `clips` にクリップを並べる。クリップが1つなら、層に直接 `src` などを書いてもよい。動画のほか、静止画(png・jpg・webp)と単色も置ける。透過つきの素材(透過PNGなど)は、透明な所で下の層が見える(題字を動画に重ねるなど)。
106
+
107
+ 層に `"sequence": true` を書くと、クリップを順に並べる(`from` を自動で決め、長さは素材から読む)。`overlap` で前後を重ねられる(フェードと組み合わせるとクロスフェード)。
108
+
109
+ | クリップの指定 | 意味 |
110
+ |---|---|
111
+ | `src` | 動画か静止画 |
112
+ | `color` | `src` の代わりに書くと、その色で塗った絵になる(暗転・帯・色かぶせ) |
113
+ | `from` `durationInFrames` | 出す位置と長さ |
114
+ | `trimBefore` | 素材の頭を何コマ飛ばすか |
115
+ | `playbackRate` | 速度(2で倍速、0.5で半分) |
116
+ | `loop` | `true` で素材全体を繰り返す。数値なら、その長さ(コマ数)を繰り返す |
117
+ | `freeze` | 素材のそのコマで止める |
118
+ | `reverse` | 逆再生 |
119
+ | `crop` | 素材の一部を切り出す `{x, y, width, height}`(素材のpx)。`x` `y` はキーフレームにできる(縦動画化のパンなど) |
120
+ | `shake` | 揺れ `{amplitude, speed, from, durationInFrames, seed, decay}`。乱数を使わないので、何度書き出しても同じ絵になる |
121
+ | `fadeIn` `fadeOut` | 透明度のフェード(コマ数)。重なるクリップに使えばクロスフェードになる |
122
+ | `audioFadeIn` `audioFadeOut` | クリップ自身の音声のフェード(コマ数) |
123
+ | `volume` | 書くと、その動画の音声も同じ位置で鳴らす(数値かキーフレーム)。書かなければ無音。キーフレームのコマとモノラルの扱いは `audio` と同じ(クリップ先頭からではなく、動画全体のコマ番号) |
124
+
125
+ 次の指定は、層に書けば全クリップに、クリップに書けばそのクリップだけに効く。
126
+
127
+ | 指定 | 意味 |
128
+ |---|---|
129
+ | `fit` | `cover`(はみ出しを切る、既定)/ `fill`(引き伸ばす)/ `contain`(全体を収め、余白は透明) |
130
+ | `align` | `cover` で切るときの寄せ位置 `[横, 縦]`(0〜1、既定は中央) |
131
+ | `flip` | 反転。`h`(左右)/ `v`(上下)/ `hv` |
132
+ | `filter` | CSSの `filter` と同じ書き方。`brightness()` `contrast()` `saturate()` `grayscale()` `sepia()` `invert()` `hue-rotate()` `blur()` `opacity()` を順にかける |
133
+ | `opacity` | 不透明度。数値か `{keyframes, easing}` |
134
+ | `blur` | ぼかし(CSSのblurと同じ単位) |
135
+ | `rotate` | 回転(度、時計回り)。数値か `{keyframes, easing}`。箱の中心で回り、はみ出した分は切れる |
136
+ | `radius` `mask` | 角丸(px)。`mask` は `"ellipse"` か白黒の画像(白い所が残る) |
137
+ | `chromaKey` | 指定した色を透明にする `{color, similarity, blend}`(グリーンバックなど) |
138
+ | `corners` | 四隅(左上・右上・左下・右下)の行き先を全体の座標で指定して、絵をはめ込む(画面の差し替えなど)。`{keyframes: [[コマ, 四隅], ...]}` で動かせる |
139
+ | `blend` | 下の絵との合成モード。CSSの `mix-blend-mode` と同じ名前(`multiply` `screen` `overlay` `darken` `lighten` `difference` `exclusion` `hard-light` `soft-light` `color-dodge` `color-burn` `plus-lighter`) |
140
+ | `scale` `origin` | 拡大。`scale` は数値か `{keyframes, easing}`、`origin` は基準点 `[横, 縦]`(0〜1) |
141
+ | `offsetX` `offsetY` | 位置のずらし。数値か `{keyframes, easing}`(揺れ・パン。キーフレームはクリップ先頭からのコマ数) |
142
+
143
+ 層の `x` `y` `width` `height` は、クリップを置く箱の位置と大きさ(既定は全面)。
144
+
145
+ ### scene層
146
+
147
+ | 指定 | 意味 |
148
+ |---|---|
149
+ | `id` `scene` | 層の名前と、描くシーンの名前 |
150
+ | `region` | その範囲だけを描く(動く帯などを小さく描く) |
151
+ | `canvas` と `x` `y` | 好きな大きさの絵を描いて置く。`x` `y` をキーフレームにするとffmpegが絵を動かす |
152
+ | `opaque` | 背景色ごと不透明に描き、JPEGで撮る(写真や3Dの多い層が約10倍速い) |
153
+ | `page` | ページ内スクリプトで描く(three.jsなど) |
154
+ | `blend` | 下の絵との合成モード(video層と同じ名前) |
155
+
156
+ ### adjust層(調整レイヤー)
157
+
158
+ ここまでに重ねた絵に、まとめて加工をかける。パネルの背景ぼかし、全体の色調整、VHS風の質感など。
159
+
160
+ | 指定 | 意味 |
161
+ |---|---|
162
+ | `filter` | CSSの `filter` と同じ書き方(`blur()` と色のフィルタ) |
163
+ | `vignette` `grain` | 周辺を暗くする強さ、粒子の強さ(どちらも0〜1)。`seed` で粒子の模様を変えられる |
164
+ | `region` `radius` | かける範囲(省くと全体)と、その角丸 |
165
+ | `from` `durationInFrames` | かける区間(省くと全体) |
166
+
167
+ ### captions層
168
+
169
+ シーンを書かずに字幕を重ねる(`browser` で描く)。
170
+
171
+ | 指定 | 意味 |
172
+ |---|---|
173
+ | `src` | 字幕ファイル(`.srt` / `.ass` / `.ssa`)。装飾タグは外す |
174
+ | `items` | `src` の代わりに `[{from, to, text}]`(コマ数)を直接書く |
175
+ | `style` | `fontFamily` `fontSize` `fontWeight` `color` `strokeColor` `strokeWidth` `lineHeight` `letterSpacing` `align` `left` `right` `top` `bottom` `background` `padding` `borderRadius` `shadow` |
176
+
177
+ 話者ごとの色分けなど凝った字幕は、`scene` 層で書く(字幕ファイルは `kasaneru/subtitles` の `readSubtitles` と `toFrames` で読める)。
178
+
179
+ ### audio
180
+
181
+ | 指定 | 意味 |
182
+ |---|---|
183
+ | `src` | 音声か、音声の入った動画 |
184
+ | `from` `durationInFrames` `trimBefore` | 鳴らす位置、長さ、頭を飛ばす量 |
185
+ | `playbackRate` `loop` `reverse` | 速度(音程は保つ)、繰り返し、逆再生 |
186
+ | `pitch` | 音程を半音単位で変える(長さは保つ) |
187
+ | `group` `duck` | `duck: {by: グループ名, to, attack, release, lead, tail}` で、そのグループのトラックが鳴っている間だけ音量を下げる(BGMを声の下へ) |
188
+ | `volume` | 数値か `[[コマ, 値], ...]`(`{keyframes, easing}` でも書ける)。キーフレームの間は直線でつながるので、段差は直前のコマに元の値を置く |
189
+ | `fadeIn` `fadeOut` | フェード(コマ数) |
190
+
191
+ - **音量のキーフレームのコマは、動画全体のコマ番号**(トラックの頭からではない)。`from: 30` のトラックに `[[0, 0], [60, 1]]` と書くと、鳴り始めの30コマ目ですでに0.5になっている。video層の `offsetX` などのキーフレームは「クリップ先頭から」なので、数え方が違う
192
+ - キーフレームは何点あってもよい(コマごとに1点でも書き出せる)。直線上に並ぶ点は自動で省く
193
+ - **モノラルの素材は、左右へ1倍ずつ入る。** ffmpegの既定の変換(`-ac 2`、左右へ0.707倍ずつ)より3dB大きい。モノラルの声とステレオのBGMを混ぜると、ffmpegの既定の変換で混ぜたときより声だけが3dB大きくなる(実測で全体が -15.6 LUFS → -12.6 LUFS)。ffmpegの既定に合わせるなら、モノラルのトラックの `volume` に0.707を掛ける(キーフレームなら各値に掛ける)
194
+
195
+ キーフレームは `{"keyframes": [[コマ, 値], ...], "easing": "linear"}`。`easing` は `linear` `inCubic` `outCubic` `inOutCubic`。
196
+
197
+ ## シーンの書き方
198
+
199
+ - **HTML/React(`scenes.browser`)**: `render(frame, props)` がHTML文字列を返す。`key(frame, props)` は省略でき、その場合は返したHTMLが同じコマを同じ絵として扱う。`.tsx` はそのまま読める(Reactは `renderToStaticMarkup` でHTMLにする)
200
+ - **フック**: 1コマごとに1回描くだけなので、状態と副作用のフックは働かない(`useState` は初期値のまま、`useEffect` は走らない)。`useContext` `useMemo` のように描く間に値を返すだけのものは使える。フレームをContextで配るコードは、`render` の中でProviderに包めば描ける
201
+ - **Reactは1つだけ**: シーンの `react` と `react-dom` は、シーンを置いた場所の `node_modules` から探し、なければkasaneruのものを使う。片方だけがシーン側にあるなど、`react` と `react-dom` が別々の場所から読まれるとReactが2つになり、Contextが壊れる(Providerの値が届かない)。両方とも同じ `node_modules` から読まれるようにする
202
+ - **素材のURL**: ページは `file://` で開き、相対パスの基準(`<base>`)はシーンのファイルがあるフォルダ。`images/a.png` のような相対パスか `file://` のURLで書く。`/images/a.png` のように `/` で始めるとディスクの根を指すので読めない(「読めない素材」として失敗になる)
203
+ - **ページ内スクリプト(`scenes.page`)**: `setup(ctx)` を1回、`draw(frame)` をコマごとに呼ぶ。素材は `ctx.assets`(manifestの `assets` のURL)から読む
204
+ - **Flutter(`scenes.flutter`)**: 作品側のDartファイルが `final Map<String, KasaneruScene> scenes` を公開する。各シーンは `key` と `build(frame, props)`(Widgetを返す)を持つ。`package:kasaneru_flutter/scenes.dart` と `anim.dart` を読み込める
205
+ - アニメーション用の関数は `kasaneru/anim`(`interpolate` `spring` `settle` `random` `noise` `typewriter` `clamp` と各種イージング)。乱数は種から決まるので、書き出すたびに絵が変わらない。シーンをどこに置いても `import {spring} from 'kasaneru/anim'` で読める。Flutterは `package:kasaneru_flutter/anim.dart`
206
+
207
+ ## manifestを作るスクリプトから使える道具
208
+
209
+ | 読み込み | 内容 |
210
+ |---|---|
211
+ | `kasaneru/media` | `probeMedia(file, fps)`(長さ・大きさ)、`envelope(file, fps)`(コマごとの音量) |
212
+ | `kasaneru/subtitles` | `readSubtitles(file)`、`toFrames(items, fps)` |
213
+ | `kasaneru/srt` | `toSrt(items, fps)` |
214
+ | `kasaneru/anim` | シーンと同じアニメーション関数 |
215
+
216
+ ## 構成
217
+
218
+ | 場所 | 内容 |
219
+ |---|---|
220
+ | `bin/kasaneru.mjs` | CLI |
221
+ | `src/ffmpeg.mjs` | manifestからffmpegのフィルタを組み立てる |
222
+ | `src/validate.mjs` `src/resolve.mjs` | manifestの検証、省略や便利な書き方(順に並べる・長さの自動計算・ダッキング)の具体化 |
223
+ | `src/inspect.mjs` `src/sheet.mjs` `src/media.mjs` `src/srt.mjs` | 動画の点検、コマ一覧、素材を調べる道具、SRT書き出し |
224
+ | `src/cssfilter.mjs` | CSSのfilterの書き方を読み、色の変換を3D LUTにする |
225
+ | `src/subtitles.mjs` `src/captions.mjs` | 字幕ファイルの読み込み、字幕の層 |
226
+ | `src/anim.mjs` | フレームから値を作る関数 |
227
+ | `src/backends/browser.mjs` | HTML/Reactのシーンを撮る |
228
+ | `src/backends/browser_page.mjs` | ページ内スクリプトのシーンを撮る |
229
+ | `src/backends/launch.mjs` `src/backends/deps.mjs` | ブラウザの起動(`KASANERU_BROWSER`)、シーンがimportするkasaneru同梱パッケージの探し場所 |
230
+ | `src/backends/flutter.mjs` と `flutter_backend/` | Flutterで描く |
231
+ | `test/` | 自動テスト。コマ番号を明るさに埋め込んだ素材で、出力の各コマの中身を判定する |
232
+ | `.github/workflows/` | pushごとのテスト(`test.yml`)と、`v*` タグのpushでのnpm公開・GitHub Release作成(`release.yml`) |
233
+ | `examples/basic/` | 外部の素材なしで動く例(クロスフェード・フィルタ・回転・角丸・合成モード・字幕) |
234
+
235
+ ## 描画バックエンドの違い
236
+
237
+ - **browser**: HTML・React・ページ内スクリプト(three.jsのリグ付きGLB、影など)を描ける。フレームをContextで配るReactのコードは、`render` の中でProviderに包めば描ける
238
+ - **flutter**: 字幕・テロップなど、ウィジェットで書いた画面を描ける。速度は `browser` と同程度
239
+ - **flutterの3D**: `flutter test --enable-impeller --enable-flutter-gpu` でGPUは使えたが、`flutter_scene` のシェーダーが読み込めず描けなかった("Failed to unpack shader")。アプリとして起動する方式は試していない
240
+
241
+ ## 落とし穴(実装で踏んだもの)
242
+
243
+ - **ffmpegの `overlay`**
244
+ - 絵の切り替わりがまばらな入力を重ねると、下の層が1コマ遅れることがある。`fps` で一定間隔のコマ列にしてから重ねる
245
+ - 位置の式で `n` を使うとずれる。`t`(時刻)から計算する
246
+ - `format=yuv420` は位置が2px単位。動く層だけ `yuv444` で重ねる
247
+ - `format=gbrp` と `alpha=premultiplied` の組み合わせは暗く合成される
248
+ - ちょうどの長さに切ったクリップは最終コマが落ちることがある。1コマ長く切り、表示区間は `enable` で決める
249
+ - **ffmpegの `scale`**: 既定の丸めでは、色行列の変換(BT.601→BT.709)で輝度が2段暗くなる。`flags=accurate_rnd+full_chroma_int` を必ず付ける
250
+ - **`setpts` で開始位置をずらすとき**: 秒へ丸めた値を足すと、切り捨てで1コマ早まることがある。コマ数から式で計算して `round()` する
251
+ - **concat入力**: 画像ごとに `option framerate` を書くと表示時刻がコマ単位で正確になる
252
+ - **ffmpegの式の入れ子**: キーフレームを先頭から順に `if` の入れ子にすると、100点ほどで式の深さの上限に当たる("Missing ')' or too many args")。区間を半分ずつに分けて入れ子にする
253
+ - **ブラウザの撮影**
254
+ - 読めなかった画像は `img.decode()` の失敗では数えられない。失敗済みの画像は `complete` のまま残るので、次のコマから待つ対象に入らない。待ったあとの `naturalWidth` で判定する
255
+ - PNGは写真の多い絵だと1枚60ms以上かかり、並列にしても速くならない。CDPの `Page.captureScreenshot` をJPEGか `optimizeForSpeed` 付きPNGで呼ぶ
256
+ - CDPで背景色の指定を送ると画面の大きさの指定が外れることがある。大きさも送り直して実際の値を確かめる
257
+ - HTMLを丸ごと差し替えると画像を毎コマ読み直す。変わった属性と文字だけ書き換える
258
+ - **`flutter test`**: フォントの明示読み込み(`.ttc` も読める)、画面サイズとDPRの指定、`debugDisableShadows = false`、`runAsync` の中での `toImage`、CSSの行送りに合わせる `TextLeadingDistribution.even`
259
+
260
+ ## 未実装・未検証
261
+
262
+ - 回転の基準点(今は箱の中心だけ)、縦横別の拡大、3D変形(四隅を指定する透視合成はできる)
263
+ - 色フィルタ・ぼかしのキーフレーム(透明度・拡大・回転・位置・切り出し・四隅はキーフレームにできる)
264
+ - クリップ同士のワイプなどの切り替え効果(フェードと位置のキーフレームで作る)
265
+ - 透過つきの出力(`.mov` / `.webm` は不透明)
266
+ - `captions` 層のFlutter対応、話者ごとのスタイル
267
+ - モノラルの素材を0.707倍で入れる切り替え(今は `volume` に手で掛ける)
268
+ - `<img>` 以外の素材(CSSの背景画像など)の読み込みを待ってから撮ること、Flutterバックエンドでの読めない素材の検出
269
+ - `--frames` での書き出しは、範囲外の動画層も一度合成してから切っている(絵は範囲分しか描かない)
270
+ - Windows・Linux(macOSでだけ動かしている)
271
+
272
+ ## ライセンス
273
+
274
+ MIT(`LICENSE`)。依存は playwright-core(Apache-2.0)、esbuild・react・react-dom・three(MIT)。ffmpeg・Chrome・Flutterは同梱せず、入っているものを呼ぶ。
package/README.md CHANGED
@@ -1,3 +1,274 @@
1
- # Temporary Holding Version
1
+ # kasaneru
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ [日本語](README.ja.md)
4
+
5
+ A tool that renders video from a manifest (JSON). kasaneru (重ねる) means "to layer" in Japanese: you build a video by stacking layers.
6
+
7
+ - Cutting, arranging, and compositing video and audio is done by `ffmpeg` (an external command)
8
+ - Text, shapes, and 3D are turned into images by a "rendering backend". Frames that produce the same picture are drawn only once
9
+ - There are two rendering backends. Neither opens a window
10
+ - `browser`: drives your local Chrome through Playwright and captures it. Draws HTML strings, React, and in-page scripts (three.js and the like)
11
+ - `flutter`: extracts widgets with `RepaintBoundary.toImage` inside `flutter test`
12
+
13
+ It has only been verified on macOS. It has not been run on Windows or Linux.
14
+
15
+ ## Usage
16
+
17
+ Install from npm (or run it with `npx kasaneru`):
18
+
19
+ ```bash
20
+ npm install kasaneru
21
+ npx kasaneru check manifest.json
22
+ npx kasaneru render manifest.json -o out.mp4
23
+ ```
24
+
25
+ The example is not in the npm package. To try it, clone the repository:
26
+
27
+ ```bash
28
+ git clone https://github.com/K9i-0/kasaneru.git
29
+ cd kasaneru && npm install
30
+ node examples/basic/make-assets.mjs # generate the example's assets with ffmpeg (no external assets needed)
31
+ node bin/kasaneru.mjs check examples/basic/manifest.json
32
+ node bin/kasaneru.mjs render examples/basic/manifest.json -o out/basic.mp4
33
+ node bin/kasaneru.mjs still examples/basic/manifest.json --frame 100 -o out/basic_100.png
34
+ node bin/kasaneru.mjs inspect out/basic.mp4
35
+ ```
36
+
37
+ | Command | What it does |
38
+ |---|---|
39
+ | `check <manifest>` | Lists every error in the manifest, with its location, without rendering (JSON; exit code 1 if there are errors) |
40
+ | `render <manifest> -o <mp4>` | Renders. Runs the same validation as `check` first and does not render if there are errors |
41
+ | `still <manifest> --frame N -o <png>` | Renders a single frame. Fast, because it draws only the pictures that frame needs |
42
+ | `inspect <video>` | Reports duration, frame count, and volume, and whether the first frame is dark or the audio is clipping |
43
+ | `inspect <video> --deep` | In addition to the above, looks for single frames with a broken picture (blank white, garbled) (`suspectFrames`). The first and last frames can only be compared with one neighbor, so when they differ greatly from it they are listed as a weak warning (`edgeFrames`; this cannot be told apart from a cut, so check by eye) |
44
+ | `sheet <video> -o <png>` | Makes a contact sheet of frames for review. `--frames 0,120,240`, `--every N`, `--manifest <manifest>` (automatically picks frames just before and after clip and caption changes), `--cols N` (columns, default 4) |
45
+ | `srt <manifest> -o <srt>` | Writes the captions as SRT (from `captions` layers; `--key props.captions` can point at another location) |
46
+ | `captions <subtitle file> --fps N` | Converts SRT/ASS to JSON `[{from, to, text}]` (in frames) |
47
+ | `probe <asset> [--fps N]` | Reports an asset's length (in frames), dimensions, and whether it has audio |
48
+ | `envelope <audio> --fps N` | Outputs the per-frame volume as an array (for lip sync and the like) |
49
+
50
+ Main options for `render`:
51
+
52
+ - `--backend browser|flutter|none` (default `browser`; `none` renders video layers only)
53
+ - The output format is determined by the extension: `.mp4` (H.264), `.mov` (ProRes 422 HQ), `.webm` (VP9), `.gif`
54
+ - `--frames A-B` renders only part of the frames (for review)
55
+ - `--props <file or JSON>` and `--set path=value` override the manifest's `props` (to make variants, such as different wording, from the same manifest)
56
+ - `--concurrency N` is the number of pages drawn at once (`browser`, default 4)
57
+ - `--encoder h264_videotoolbox` uses the Mac hardware encoder
58
+ - `--crf N` sets the quality for `.mp4` with the default encoder and for `.webm`, where it is scaled to VP9's range (default 18; lower is higher quality and larger)
59
+ - `--every-frame` draws a picture for every frame instead of drawing identical pictures only once (for checking)
60
+ - `--audio-bitrate 256k` changes the audio bitrate (default is 192k for `.mp4` and 160k for `.webm`; it has no effect on `.mov`, which is uncompressed)
61
+ - `--allow-missing-assets` renders even if a scene's assets cannot be loaded (see "Missing assets" below)
62
+ - `--keep-work` keeps the working folder (the drawn pictures and the ffmpeg filter) instead of deleting it. `--work <dir>` chooses where it goes (default: next to the output, `<output name>.work`)
63
+
64
+ The result, a timing breakdown plus warnings, is written to standard output as JSON.
65
+
66
+ ### Missing assets
67
+
68
+ When an image in a scene drawn by `browser` cannot be loaded (the file does not exist, is corrupt, or does not finish loading within 30 seconds), the render fails (exit code 1) after all pictures have been drawn and before the video is made. The URL that could not be loaded, the layer and frame where it was first noticed, and the number of pictures affected are written to standard error.
69
+
70
+ - What is checked: `<img>` (a `naturalWidth` of 0 after loading) and load failures reported by the page (CSS background images, SVG `<image>`, fonts, and so on). For `scenes.page` scenes, files requested from `ctx.assets` that did not exist
71
+ - With `--allow-missing-assets`, the render does not fail; the same information goes into `warnings` in the result JSON and the video is rendered
72
+ - The wait time can be changed with the environment variable `KASANERU_IMAGE_TIMEOUT_MS`. Once an image has timed out, later pictures count it as a failure without waiting again
73
+ - Loads cancelled partway (such as a background image no longer needed in the next frame) are not counted as failures
74
+ - Anything other than `<img>` is captured without waiting for it to load. Local files load immediately, so this has caused no real problems, but a background image may miss the capture, or a failure report may arrive too late. Use `<img>` for images that must be waited for
75
+ - In-page errors in `scenes.page` scenes (`pageerror`) do not fail the render; they go into `warnings`
76
+
77
+ ### Requirements and cautions
78
+
79
+ Requirements: Node 22, ffmpeg, Google Chrome (`browser`), Flutter (`flutter`). Run the tests with `npm test`.
80
+
81
+ **Only render manifests you trust.** A manifest names scene files (`scenes.browser`, `scenes.page`, `scenes.flutter`), and kasaneru runs them: on your machine as Node code, in the browser, or inside `flutter test`. Rendering someone else's manifest runs their code with your permissions. Asset paths are also read from wherever the manifest points.
82
+
83
+ The `flutter` backend writes its generated entry point and build cache into kasaneru's own `flutter_backend/` folder, so the folder kasaneru is installed in must be writable (a local `npm install` is; a system-wide install owned by root is not).
84
+
85
+ The browser can be changed with the environment variable `KASANERU_BROWSER` (the path to an executable, or a name such as `chrome-beta`, `msedge`, or `chromium`).
86
+
87
+ ## manifest
88
+
89
+ All times are written as frame counts (integers) at the composition's fps. Paths are relative to the manifest's location.
90
+
91
+ | Field | Description |
92
+ |---|---|
93
+ | `composition` | `width` `height` `fps` `durationInFrames`. If `durationInFrames` is omitted, it is set to the end of the latest-ending clip or audio track |
94
+ | `background` | Background color (`#RRGGBB`) |
95
+ | `layers` | Layers, stacked from the bottom up. `video` / `scene` / `captions` / `adjust` |
96
+ | `audio` | Array of audio tracks |
97
+ | `master` | Audio mastering. `limiter` (ceiling, 0–1) and `loudnorm` (`i` `tp` `lra`) |
98
+ | `scenes` | Where the scenes are. `browser` (HTML/React), `page` (in-page script), `flutter` (Dart file) |
99
+ | `assets` | Folder of assets loaded by `page` scenes (default: the manifest's folder). Only files under this folder are served to the page; a request outside it fails as a missing asset |
100
+ | `fonts` | Font files to load (`family` `file`, plus `weight` `style` if needed). For fonts not installed on the system |
101
+ | `props` | Values passed to scenes |
102
+
103
+ ### video layer
104
+
105
+ List clips in `clips`. If there is only one clip, `src` and the rest can be written directly on the layer. Besides video, still images (png, jpg, webp) and solid colors can be placed. With assets that have transparency (such as transparent PNGs), the layers below show through the transparent areas (for example, a title overlaid on video).
106
+
107
+ Setting `"sequence": true` on a layer lines the clips up one after another (`from` is set automatically and lengths are read from the assets). `overlap` makes neighboring clips overlap (combined with fades, this gives a crossfade).
108
+
109
+ | Clip option | Meaning |
110
+ |---|---|
111
+ | `src` | A video or still image |
112
+ | `color` | Written instead of `src`, gives a picture filled with that color (blackouts, bars, color overlays) |
113
+ | `from` `durationInFrames` | Where it appears and for how long |
114
+ | `trimBefore` | How many frames to skip at the start of the asset |
115
+ | `playbackRate` | Speed (2 is double speed, 0.5 is half) |
116
+ | `loop` | `true` repeats the whole asset. A number repeats that length (in frames) |
117
+ | `freeze` | Holds on that frame of the asset |
118
+ | `reverse` | Plays in reverse |
119
+ | `crop` | Cuts out part of the asset `{x, y, width, height}` (in asset px). `x` and `y` can be keyframed (such as panning when converting to vertical video) |
120
+ | `shake` | Shake `{amplitude, speed, from, durationInFrames, seed, decay}`. No random numbers are used, so every render produces the same picture |
121
+ | `fadeIn` `fadeOut` | Opacity fades (in frames). Used on overlapping clips, they give a crossfade |
122
+ | `audioFadeIn` `audioFadeOut` | Fades for the clip's own audio (in frames) |
123
+ | `volume` | When set, the video's audio also plays at the same position (a number or keyframes). When not set, the clip is silent. Keyframe frames and mono handling are the same as for `audio` (frame numbers of the whole video, not counted from the start of the clip) |
124
+
125
+ The following options apply to every clip when written on the layer, or to a single clip when written on that clip.
126
+
127
+ | Option | Meaning |
128
+ |---|---|
129
+ | `fit` | `cover` (crops the overflow, default) / `fill` (stretches) / `contain` (fits the whole picture; the margins are transparent) |
130
+ | `align` | Anchor position when `cover` crops, `[horizontal, vertical]` (0–1, default is center) |
131
+ | `flip` | Flip. `h` (horizontal) / `v` (vertical) / `hv` |
132
+ | `filter` | Same syntax as CSS `filter`. Applies `brightness()` `contrast()` `saturate()` `grayscale()` `sepia()` `invert()` `hue-rotate()` `blur()` `opacity()` in order |
133
+ | `opacity` | Opacity. A number or `{keyframes, easing}` |
134
+ | `blur` | Blur (same unit as CSS blur) |
135
+ | `rotate` | Rotation (degrees, clockwise). A number or `{keyframes, easing}`. Rotates around the center of the box; anything that sticks out is cropped |
136
+ | `radius` `mask` | Rounded corners (px). `mask` is `"ellipse"` or a black-and-white image (white areas are kept) |
137
+ | `chromaKey` | Makes the given color transparent `{color, similarity, blend}` (green screen and the like) |
138
+ | `corners` | Fits the picture into a quad by giving the destinations of its four corners (top-left, top-right, bottom-left, bottom-right) in composition coordinates (screen replacement and the like). Can be animated with `{keyframes: [[frame, corners], ...]}` |
139
+ | `blend` | Blend mode with the picture below. Same names as CSS `mix-blend-mode` (`multiply` `screen` `overlay` `darken` `lighten` `difference` `exclusion` `hard-light` `soft-light` `color-dodge` `color-burn` `plus-lighter`) |
140
+ | `scale` `origin` | Scaling. `scale` is a number or `{keyframes, easing}`; `origin` is the anchor point `[horizontal, vertical]` (0–1) |
141
+ | `offsetX` `offsetY` | Position offset. A number or `{keyframes, easing}` (shakes, pans; keyframes are counted in frames from the start of the clip) |
142
+
143
+ The layer's `x` `y` `width` `height` are the position and size of the box the clips are placed in (default is the full frame).
144
+
145
+ ### scene layer
146
+
147
+ | Option | Meaning |
148
+ |---|---|
149
+ | `id` `scene` | The layer's name, and the name of the scene to draw |
150
+ | `region` | Draws only that area (to draw a moving bar or similar at a small size) |
151
+ | `canvas` and `x` `y` | Draws a picture of any size and places it. If `x` `y` are keyframed, ffmpeg moves the picture |
152
+ | `opaque` | Draws opaque, background color included, and captures as JPEG (about 10x faster for layers heavy in photos or 3D) |
153
+ | `page` | Draws with an in-page script (three.js and the like) |
154
+ | `blend` | Blend mode with the picture below (same names as the video layer) |
155
+
156
+ ### adjust layer (adjustment layer)
157
+
158
+ Applies processing to everything stacked so far, all at once. Background blur behind a panel, overall color adjustment, a VHS-like texture, and so on.
159
+
160
+ | Option | Meaning |
161
+ |---|---|
162
+ | `filter` | Same syntax as CSS `filter` (`blur()` and the color filters) |
163
+ | `vignette` `grain` | Strength of edge darkening and strength of grain (both 0–1). `seed` changes the grain pattern |
164
+ | `region` `radius` | The area to affect (the whole frame if omitted) and its corner radius |
165
+ | `from` `durationInFrames` | The span to affect (the whole duration if omitted) |
166
+
167
+ ### captions layer
168
+
169
+ Overlays captions without writing a scene (drawn by `browser`).
170
+
171
+ | Option | Meaning |
172
+ |---|---|
173
+ | `src` | Subtitle file (`.srt` / `.ass` / `.ssa`). Styling tags are stripped |
174
+ | `items` | Instead of `src`, write `[{from, to, text}]` (in frames) directly |
175
+ | `style` | `fontFamily` `fontSize` `fontWeight` `color` `strokeColor` `strokeWidth` `lineHeight` `letterSpacing` `align` `left` `right` `top` `bottom` `background` `padding` `borderRadius` `shadow` |
176
+
177
+ For more elaborate captions, such as per-speaker colors, write them in a `scene` layer (subtitle files can be read with `readSubtitles` and `toFrames` from `kasaneru/subtitles`).
178
+
179
+ ### audio
180
+
181
+ | Option | Meaning |
182
+ |---|---|
183
+ | `src` | Audio, or a video that contains audio |
184
+ | `from` `durationInFrames` `trimBefore` | Where it plays, for how long, and how much to skip at the start |
185
+ | `playbackRate` `loop` `reverse` | Speed (pitch is preserved), repeat, reverse playback |
186
+ | `pitch` | Changes pitch in semitones (length is preserved) |
187
+ | `group` `duck` | `duck: {by: group name, to, attack, release, lead, tail}` lowers the volume only while tracks in that group are playing (BGM under voice) |
188
+ | `volume` | A number or `[[frame, value], ...]` (`{keyframes, easing}` also works). Keyframes are joined by straight lines, so for a step, put the old value on the frame just before |
189
+ | `fadeIn` `fadeOut` | Fades (in frames) |
190
+
191
+ - **Volume keyframe frames are frame numbers of the whole video** (not counted from the start of the track). With `[[0, 0], [60, 1]]` on a track with `from: 30`, the volume is already 0.5 at frame 30, when it starts playing. Keyframes such as `offsetX` on video layers are counted "from the start of the clip", so the two count differently
192
+ - Any number of keyframes is fine (even one per frame renders). Points that lie on a straight line are dropped automatically
193
+ - **Mono assets go to left and right at 1x each.** This is 3 dB louder than ffmpeg's default conversion (`-ac 2`, 0.707x to each side). When you mix mono voice with stereo BGM, only the voice comes out 3 dB louder than it would with ffmpeg's default conversion (measured: overall -15.6 LUFS → -12.6 LUFS). To match the ffmpeg default, multiply the mono track's `volume` by 0.707 (for keyframes, multiply each value)
194
+
195
+ Keyframes are `{"keyframes": [[frame, value], ...], "easing": "linear"}`. `easing` is one of `linear` `inCubic` `outCubic` `inOutCubic`.
196
+
197
+ ## Writing scenes
198
+
199
+ - **HTML/React (`scenes.browser`)**: `render(frame, props)` returns an HTML string. `key(frame, props)` is optional; without it, frames that return the same HTML are treated as the same picture. `.tsx` files load as they are (React is turned into HTML with `renderToStaticMarkup`)
200
+ - **Hooks**: each frame is drawn just once, so state and effect hooks do not work (`useState` stays at its initial value, `useEffect` does not run). Hooks that only return a value during rendering, such as `useContext` and `useMemo`, can be used. Code that distributes the frame through Context works if you wrap it in a Provider inside `render`
201
+ - **Only one React**: a scene's `react` and `react-dom` are looked up in the `node_modules` next to where the scene lives, falling back to kasaneru's own. If `react` and `react-dom` are loaded from different places (for example, only one of them exists on the scene side), there are two Reacts and Context breaks (the Provider's value does not arrive). Make sure both are loaded from the same `node_modules`
202
+ - **Asset URLs**: the page is opened as `file://`, and the base for relative paths (`<base>`) is the folder containing the scene file. Write relative paths such as `images/a.png`, or `file://` URLs. A path that starts with `/`, such as `/images/a.png`, points at the root of the disk and cannot be loaded (it fails as a "missing asset")
203
+ - **In-page script (`scenes.page`)**: `setup(ctx)` is called once and `draw(frame)` for every frame. Assets are loaded from `ctx.assets` (the URL of the manifest's `assets`)
204
+ - **Flutter (`scenes.flutter`)**: a Dart file on the project side exposes `final Map<String, KasaneruScene> scenes`. Each scene has a `key` and a `build(frame, props)` (returning a Widget). `package:kasaneru_flutter/scenes.dart` and `anim.dart` can be imported
205
+ - Animation functions are in `kasaneru/anim` (`interpolate` `spring` `settle` `random` `noise` `typewriter` `clamp` and various easings). Random numbers are determined by a seed, so the picture does not change from render to render. Wherever a scene is placed, `import {spring} from 'kasaneru/anim'` works. For Flutter, use `package:kasaneru_flutter/anim.dart`
206
+
207
+ ## Tools for scripts that build manifests
208
+
209
+ | Import | Contents |
210
+ |---|---|
211
+ | `kasaneru/media` | `probeMedia(file, fps)` (length, dimensions), `envelope(file, fps)` (per-frame volume) |
212
+ | `kasaneru/subtitles` | `readSubtitles(file)`, `toFrames(items, fps)` |
213
+ | `kasaneru/srt` | `toSrt(items, fps)` |
214
+ | `kasaneru/anim` | The same animation functions as in scenes |
215
+
216
+ ## Layout
217
+
218
+ | Location | Contents |
219
+ |---|---|
220
+ | `bin/kasaneru.mjs` | CLI |
221
+ | `src/ffmpeg.mjs` | Builds the ffmpeg filter from the manifest |
222
+ | `src/validate.mjs` `src/resolve.mjs` | Manifest validation; resolving omissions and shorthands (sequencing, automatic length calculation, ducking) into concrete values |
223
+ | `src/inspect.mjs` `src/sheet.mjs` `src/media.mjs` `src/srt.mjs` | Video inspection, contact sheets, asset probing tools, SRT output |
224
+ | `src/cssfilter.mjs` | Parses CSS filter syntax and turns the color transforms into a 3D LUT |
225
+ | `src/subtitles.mjs` `src/captions.mjs` | Subtitle file loading, the captions layer |
226
+ | `src/anim.mjs` | Functions that produce values from the frame |
227
+ | `src/backends/browser.mjs` | Captures HTML/React scenes |
228
+ | `src/backends/browser_page.mjs` | Captures in-page script scenes |
229
+ | `src/backends/launch.mjs` `src/backends/deps.mjs` | Launches the browser (`KASANERU_BROWSER`); where to find the packages bundled with kasaneru for scenes that import them |
230
+ | `src/backends/flutter.mjs` and `flutter_backend/` | Draws with Flutter |
231
+ | `test/` | Automated tests. Uses assets with the frame number embedded in their brightness to check the content of each output frame |
232
+ | `.github/workflows/` | Tests on every push (`test.yml`); npm publish and GitHub Release when a `v*` tag is pushed (`release.yml`) |
233
+ | `examples/basic/` | An example that runs without external assets (crossfade, filters, rotation, rounded corners, blend modes, captions) |
234
+
235
+ ## Differences between rendering backends
236
+
237
+ - **browser**: draws HTML, React, and in-page scripts (rigged GLBs in three.js, shadows, and so on). React code that distributes the frame through Context can be drawn by wrapping it in a Provider inside `render`
238
+ - **flutter**: draws screens written with widgets, such as captions and on-screen titles. Speed is about the same as `browser`
239
+ - **3D in flutter**: the GPU was usable with `flutter test --enable-impeller --enable-flutter-gpu`, but the `flutter_scene` shaders failed to load and nothing could be drawn ("Failed to unpack shader"). Launching as an app has not been tried
240
+
241
+ ## Pitfalls (ones hit during implementation)
242
+
243
+ - **ffmpeg `overlay`**
244
+ - When overlaying an input whose pictures change only sparsely, the layer below can lag by one frame. Convert it to a constant-interval frame sequence with `fps` before overlaying
245
+ - Using `n` in the position expression causes drift. Compute from `t` (time)
246
+ - With `format=yuv420`, positions move in 2 px steps. Overlay only the moving layers in `yuv444`
247
+ - The combination of `format=gbrp` and `alpha=premultiplied` composites too dark
248
+ - A clip cut to exactly its length can lose its last frame. Cut it one frame longer and set the visible span with `enable`
249
+ - **ffmpeg `scale`**: with the default rounding, the color matrix conversion (BT.601→BT.709) darkens luma by 2 levels. Always add `flags=accurate_rnd+full_chroma_int`
250
+ - **Shifting the start position with `setpts`**: adding a value rounded to seconds can come out one frame early because of truncation. Compute it in an expression from the frame count and `round()` it
251
+ - **concat input**: writing `option framerate` for each image makes display times frame-accurate
252
+ - **Nesting in ffmpeg expressions**: nesting `if` for keyframes in order from the first one hits the expression depth limit at around 100 points ("Missing ')' or too many args"). Split the span in halves and nest those instead
253
+ - **Browser capture**
254
+ - Images that failed to load cannot be counted by `img.decode()` failures. A failed image stays `complete`, so from the next frame on it is no longer among the things waited for. Judge by `naturalWidth` after waiting
255
+ - PNG takes 60 ms or more per image for photo-heavy pictures, and parallelism does not make it faster. Call CDP's `Page.captureScreenshot` with JPEG, or with PNG plus `optimizeForSpeed`
256
+ - Sending a background color override over CDP can drop the viewport size override. Send the size again as well and check the actual value
257
+ - Replacing the whole HTML reloads the images on every frame. Rewrite only the attributes and text that changed
258
+ - **`flutter test`**: explicit font loading (`.ttc` works too), setting the screen size and DPR, `debugDisableShadows = false`, calling `toImage` inside `runAsync`, and `TextLeadingDistribution.even` to match CSS line spacing
259
+
260
+ ## Not implemented / not verified
261
+
262
+ - A rotation origin (currently only the center of the box), separate horizontal and vertical scaling, 3D transforms (perspective compositing by specifying four corners is possible)
263
+ - Keyframes for color filters and blur (opacity, scale, rotation, position, crop, and corners can be keyframed)
264
+ - Transition effects between clips, such as wipes (build them from fades and position keyframes)
265
+ - Output with transparency (`.mov` / `.webm` are opaque)
266
+ - Flutter support for the `captions` layer, per-speaker styles
267
+ - A switch to bring mono assets in at 0.707x (for now, multiply `volume` by hand)
268
+ - Waiting for assets other than `<img>` (such as CSS background images) to load before capturing; detecting missing assets in the Flutter backend
269
+ - `--frames` still composites video layers outside the range once and trims afterwards (pictures are drawn only for the range)
270
+ - Windows and Linux (it has only been run on macOS)
271
+
272
+ ## License
273
+
274
+ MIT (`LICENSE`). Dependencies are playwright-core (Apache-2.0) and esbuild, react, react-dom, and three (MIT). ffmpeg, Chrome, and Flutter are not bundled; kasaneru calls whatever is installed.