@grundyjs/canvas-take 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/README.md +90 -5
- package/dist/aac-PROVENANCE.md +171 -0
- package/dist/aac-licenses/COPYING.LGPLv2.1 +502 -0
- package/dist/aac-licenses/LICENSE.md +127 -0
- package/dist/aac-licenses/MPL-2.0.txt +373 -0
- package/dist/aac-licenses/bridge.c +191 -0
- package/dist/gif.d.ts +6 -0
- package/dist/gif.js +16 -2
- package/dist/index.d.ts +8 -1
- package/dist/index.js +90 -6
- package/dist/mediabunny-LICENSE.txt +373 -0
- package/dist/video-aac-worker.d.ts +1 -0
- package/dist/video-aac-worker.js +3356 -0
- package/dist/video-aac-worker.js.map +7 -0
- package/dist/video-aac.d.ts +8 -0
- package/dist/video-aac.js +174 -0
- package/dist/video-mp4.d.ts +3 -0
- package/dist/video-mp4.js +38 -0
- package/dist/video-mp4.js.map +7 -0
- package/dist/video-webcodecs.d.ts +3 -0
- package/dist/video-webcodecs.js +149 -0
- package/dist/video.d.ts +215 -0
- package/dist/video.js +503 -0
- package/docs/drafts/offline-video-api.d.ts +2 -0
- package/docs/drafts/offline-video-example.ts +38 -0
- package/docs/drafts/tsconfig.json +12 -0
- package/docs/gif-benchmark.md +129 -0
- package/docs/gif-encoder.md +6 -0
- package/docs/offline-aac.md +87 -0
- package/docs/offline-video-adapters.md +84 -0
- package/docs/offline-video-api.md +154 -0
- package/docs/video-recording.md +55 -0
- package/package.json +48 -7
- package/ru/README.md +166 -6
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
# GIF adapter benchmark
|
|
2
|
+
|
|
3
|
+
Install dependencies and Chromium (`npm ci`, `npx playwright install chromium`), then run:
|
|
4
|
+
|
|
5
|
+
```sh
|
|
6
|
+
npm run benchmark:gif
|
|
7
|
+
# Short local check: two resolutions, two durations, one repeat
|
|
8
|
+
npm run benchmark:gif -- --quick
|
|
9
|
+
# Compare opacity scenes before/after the opaque-pixel optimization
|
|
10
|
+
npm run benchmark:gif -- --compare-opacity
|
|
11
|
+
# Smoke check of the comparison (one repeat, 360p and 720p)
|
|
12
|
+
npm run benchmark:gif -- --compare-opacity --quick
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The full matrix uses 640×360, 1280×720 and 1920×1080; 3, 10 and 30 seconds; 20 fps; three repeats for each adapter. Both adapters use gifenc with 128 colors and `quality: 'high'`. The renderer generates identical opaque flat-graphics frames from the frame index, without random values, fonts, external images or wall-clock animation. Full output SHA-256 hashes must match across adapters and repeats for each case. Pixel/timing correctness is covered separately by the browser tests.
|
|
16
|
+
|
|
17
|
+
`--compare-opacity` uses three-second clips at the same three resolutions, three repeats, both adapters and two core versions (108 exports). It compares the core from Git commit `2f71f3c` (before the optimization) with the current built core, sharing the current adapters and renderer. The baseline requires that commit in local Git history; its TypeScript is transpiled in memory, without changing the checkout. Before/after order alternates alongside adapter order. Every scene/core/adapter receives an excluded warm-up. Output hashes must match across both versions, both adapters and all repeats within each scene/resolution/duration. `comparison.md` adds paired medians and percentage changes to the raw JSON/CSV and full summary.
|
|
18
|
+
|
|
19
|
+
The opacity scenes are `opaque` (solid background and moving bars), `mixed` (an opaque background on one third of the canvas, transparent gaps, and alternating opaque/half-alpha bars), and `gradient` (an animated color gradient with alpha from 0.15 to 0.85, without any opaque pixels). The canvas is cleared before each frame, preventing alpha accumulation. A separate, untimed inspection of the first frame asserts the expected alpha categories and records their pixel counts in metadata. The default exporter background is white for all scenes. Scene timings include drawing; compare versions within a scene rather than interpreting differences between scenes as pure encoding costs.
|
|
20
|
+
|
|
21
|
+
Runs are sequential in headless Chromium, after an excluded one-second warm-up for each adapter. The first adapter alternates between pairs. Each export gets a fresh encoder/Worker. Startup, rendering, canvas readback, compositing, encoding and finalization are included in export time; module loading, hashing and transferring measurements to Node are excluded. Frame p95 includes rendering plus awaited addFrame. Finalization time is also retained in JSON/CSV.
|
|
22
|
+
|
|
23
|
+
An independent 16 ms timer records excess main-thread scheduling delay during each export. The p95 and maximum help compare responsiveness, but are not measurements of input latency. Worker export still reads and composites pixels on the main thread. A faster export is not necessarily more responsive. Memory consumption and cancellation latency are **not measured** by this benchmark; JS heap alone would omit Worker/native buffers and would not be a fair memory comparison.
|
|
24
|
+
|
|
25
|
+
Timestamped `benchmark-results/` directories contain raw per-frame/timer samples and environment information in JSON, per-run CSV, and a Markdown table of medians. Partial JSON survives a later failure; a failing or timed-out case stops the command with a nonzero exit code. Each case has a five-minute timeout. Generated results are ignored by Git. A quick run is a smoke measurement, not a statistically robust comparison.
|
|
26
|
+
|
|
27
|
+
Run on an otherwise idle machine, with stable power settings, and retain the environment metadata when comparing results. This synthetic scene is not representative of photos, noisy imagery or every algorithm visualization. No performance thresholds are imposed in shared CI: hardware contention would make them misleading. Run this command explicitly when evaluating adapter changes.
|
|
28
|
+
|
|
29
|
+
## Baseline: 2026-10-28
|
|
30
|
+
|
|
31
|
+
All 54 exports completed: nine configurations, two adapters, three repeats. All six GIFs for each configuration were byte-identical (SHA-256). Environment: AMD FX-8350, Windows 10 build 19045, Chromium 156.0.8078.4, Node 22.17.0, CanvasTake 0.4.1 at `2fcabcc557ce6cbd10e342045221a9787c1b61c0`, with the benchmark files uncommitted. The run started on October 28 in Europe/Moscow (October 27 UTC).
|
|
32
|
+
|
|
33
|
+
Export times below are medians of three runs. Timer lag columns are medians of the three per-run p95 values, not pooled percentiles. Negative time change means Worker took less time.
|
|
34
|
+
|
|
35
|
+
| Resolution | GIF duration (s) | Sync export (s) | Worker export (s) | Worker time change | Sync timer lag p95 (ms) | Worker timer lag p95 (ms) |
|
|
36
|
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
37
|
+
| 640×360 | 3 | 1.26 | 1.09 | -13.5% | 8.0 | 2.2 |
|
|
38
|
+
| 640×360 | 10 | 4.28 | 3.51 | -18.0% | 7.3 | 2.4 |
|
|
39
|
+
| 640×360 | 30 | 13.00 | 10.56 | -18.8% | 8.0 | 3.1 |
|
|
40
|
+
| 1280×720 | 3 | 4.12 | 3.95 | -4.3% | 66.6 | 25.2 |
|
|
41
|
+
| 1280×720 | 10 | 13.85 | 13.32 | -3.8% | 58.1 | 27.2 |
|
|
42
|
+
| 1280×720 | 30 | 41.97 | 39.95 | -4.8% | 62.9 | 27.4 |
|
|
43
|
+
| 1920×1080 | 3 | 8.98 | 9.07 | +0.9% | 155.4 | 62.0 |
|
|
44
|
+
| 1920×1080 | 10 | 29.98 | 30.76 | +2.6% | 144.1 | 63.0 |
|
|
45
|
+
| 1920×1080 | 30 | 92.94 | 93.54 | +0.6% | 158.1 | 63.6 |
|
|
46
|
+
|
|
47
|
+
On this machine and scene, Worker reduced export time by 13–19% at 360p and 4–5% at 720p. At 1080p, export time was similar, with Worker taking 0.6–2.6% longer. Worker reduced timer lag in every configuration, although 1080p still showed around 62–64 ms p95 lag. Three repeats do not establish statistical significance for small differences; these figures are a local baseline, not general performance guarantees.
|
|
48
|
+
|
|
49
|
+
## Opaque-pixel blending optimization: 2026-11-01
|
|
50
|
+
|
|
51
|
+
The exporter now skips background blending when a pixel's alpha is exactly 255. Other alpha values keep the existing composition and rounding. Snapshot ownership and the cancellation yield are unchanged.
|
|
52
|
+
|
|
53
|
+
A fresh before/after comparison used the core from commit `2f71f3c` and the optimized core, served alongside the same adapters and deterministic renderer. The environment and encoding settings were the same as above. Each core/adapter received an excluded warm-up; exports ran sequentially with alternating before/after order and three repeats. This focused matrix covers three-second GIFs at all three resolutions plus ten-second GIFs at 1080p (48 exports); it does not rerun the full 30-second matrix. All 48 output hashes matched the historical baseline. An exploratory run overlapped browser tests and is excluded from this table; the reported run started after all browser tests ended.
|
|
54
|
+
|
|
55
|
+
| Resolution | GIF duration (s) | Adapter | Before (s) | After (s) | Time reduction | Timer lag p95 before (ms) | Timer lag p95 after (ms) |
|
|
56
|
+
| --- | ---: | --- | ---: | ---: | ---: | ---: | ---: |
|
|
57
|
+
| 640×360 | 3 | Sync | 1.28 | 1.02 | 20.6% | 17.4 | 8.9 |
|
|
58
|
+
| 640×360 | 3 | Worker | 1.05 | 0.80 | 23.8% | 1.8 | 2.6 |
|
|
59
|
+
| 1280×720 | 3 | Sync | 3.94 | 3.06 | 22.5% | 55.5 | 43.7 |
|
|
60
|
+
| 1280×720 | 3 | Worker | 3.88 | 3.00 | 22.7% | 24.4 | 10.3 |
|
|
61
|
+
| 1920×1080 | 3 | Sync | 8.79 | 6.96 | 20.7% | 164.8 | 161.0 |
|
|
62
|
+
| 1920×1080 | 3 | Worker | 8.99 | 6.83 | 24.0% | 63.1 | 27.4 |
|
|
63
|
+
| 1920×1080 | 10 | Sync | 29.72 | 22.51 | 24.3% | 150.7 | 105.9 |
|
|
64
|
+
| 1920×1080 | 10 | Worker | 30.04 | 23.65 | 21.3% | 63.0 | 29.2 |
|
|
65
|
+
|
|
66
|
+
Values are medians across repeats (timer columns use per-run p95); reductions use unrounded timings. Export time decreased by 21–24% on this opaque scene. At 1080p, Worker timer lag p95 fell from about 63 ms to 27–29 ms. Lag did not improve uniformly: the 360p Worker value rose from 1.8 to 2.6 ms. These are measurements on one machine, with only three repeats; they do not establish a universal speedup or responsiveness guarantee. Scenes containing translucent pixels still perform blending for those pixels.
|
|
67
|
+
|
|
68
|
+
## Opacity scene comparison: 2026-11-01
|
|
69
|
+
|
|
70
|
+
Command: `npm run benchmark:gif -- --compare-opacity`. All 108 exports completed, with byte-identical results within each scene/resolution across both core versions, adapters and all three repeats. This is a fresh comparison against the pre-optimization core, not a comparison with historical wall times. Environment: AMD FX-8350, Windows 10 build 19045, Chromium 156.0.8078.4, Node 22.17.0; current core at `6930a09`, baseline at `2f71f3c`; benchmark changes were uncommitted.
|
|
71
|
+
|
|
72
|
+
At 1080p, the first mixed frame contained 45.4% opaque, 40.4% fully transparent and 14.2% partially transparent pixels. The opaque scene was 100% opaque; the gradient was 100% partially transparent. These proportions describe the inspected first frame, not every animated frame.
|
|
73
|
+
|
|
74
|
+
Times are medians; lag values are medians of per-run p95. Positive time reduction means faster; negative means slower.
|
|
75
|
+
|
|
76
|
+
| Scene | Resolution | Seconds | Adapter | Before ms | After ms | Time reduction | Lag p95 before ms | Lag p95 after ms |
|
|
77
|
+
| --- | --- | ---: | --- | ---: | ---: | ---: | ---: | ---: |
|
|
78
|
+
| opaque | 640×360 | 3 | sync | 1406.5 | 1063.4 | 24.4% | 10.1 | 5.3 |
|
|
79
|
+
| opaque | 640×360 | 3 | worker | 1260.2 | 933.5 | 25.9% | 4.9 | 2.8 |
|
|
80
|
+
| opaque | 1280×720 | 3 | sync | 4584.4 | 3369.7 | 26.5% | 75.0 | 46.5 |
|
|
81
|
+
| opaque | 1280×720 | 3 | worker | 4561.3 | 3227.2 | 29.2% | 35.2 | 13.9 |
|
|
82
|
+
| opaque | 1920×1080 | 3 | sync | 9966.9 | 7116.1 | 28.6% | 163.3 | 158.6 |
|
|
83
|
+
| opaque | 1920×1080 | 3 | worker | 10156.1 | 7261.3 | 28.5% | 80.5 | 33.1 |
|
|
84
|
+
| mixed | 640×360 | 3 | sync | 1399.7 | 1252.5 | 10.5% | 10.0 | 7.8 |
|
|
85
|
+
| mixed | 640×360 | 3 | worker | 1217.2 | 1060.0 | 12.9% | 4.6 | 2.6 |
|
|
86
|
+
| mixed | 1280×720 | 3 | sync | 4604.9 | 4080.2 | 11.4% | 71.2 | 66.0 |
|
|
87
|
+
| mixed | 1280×720 | 3 | worker | 4596.3 | 3955.7 | 13.9% | 31.9 | 22.4 |
|
|
88
|
+
| mixed | 1920×1080 | 3 | sync | 10239.6 | 8883.4 | 13.2% | 199.3 | 191.7 |
|
|
89
|
+
| mixed | 1920×1080 | 3 | worker | 10599.5 | 9097.6 | 14.2% | 79.0 | 53.7 |
|
|
90
|
+
| gradient | 640×360 | 3 | sync | 1865.1 | 1929.0 | -3.4% | 20.9 | 21.1 |
|
|
91
|
+
| gradient | 640×360 | 3 | worker | 1805.4 | 1678.0 | 7.1% | 12.9 | 11.4 |
|
|
92
|
+
| gradient | 1280×720 | 3 | sync | 6168.2 | 6256.3 | -1.4% | 95.4 | 103.2 |
|
|
93
|
+
| gradient | 1280×720 | 3 | worker | 6168.6 | 6112.9 | 0.9% | 53.0 | 52.8 |
|
|
94
|
+
| gradient | 1920×1080 | 3 | sync | 13678.2 | 14551.0 | -6.4% | 279.2 | 306.8 |
|
|
95
|
+
| gradient | 1920×1080 | 3 | worker | 14454.1 | 15027.8 | -4.0% | 125.8 | 128.9 |
|
|
96
|
+
|
|
97
|
+
Opaque scenes took 24–29% less time and mixed-alpha scenes 10–14% less time in this run. Gradients showed no consistent benefit: the synchronous adapter was 1.4–6.4% slower, while Worker varied from 7.1% faster to 4.0% slower. At 1080p, both gradient medians regressed (Sync 6.4%, Worker 4.0%); this observed cost must not be described as a universal speedup. With three repeats and noticeable run-to-run variation, the experiment does not establish statistical significance or isolate the extra branch from runtime/system noise. More gradient repetitions and profiling would be appropriate before claiming a stable regression magnitude.
|
|
98
|
+
|
|
99
|
+
At 1080p, Worker timer lag p95 improved from 80.5 to 33.1 ms for opaque frames and from 79.0 to 53.7 ms for mixed alpha, but increased from 125.8 to 128.9 ms for gradients. The optimization benefits pixels that can skip blending; it does not eliminate readback, copying, drawing or encoding work.
|
|
100
|
+
|
|
101
|
+
## Gradient regression investigation: 2026-11-01
|
|
102
|
+
|
|
103
|
+
The earlier 4–6% end-to-end gradient slowdown did **not** reproduce in a larger, warmed-up experiment. A small overhead of the added condition was measurable in an isolated loop, so the conclusion is not that the condition is free: the original export-level slowdown was not a stable estimate of its cost.
|
|
104
|
+
|
|
105
|
+
Reproduce the two experiments sequentially, without other benchmarks or tests running:
|
|
106
|
+
|
|
107
|
+
```sh
|
|
108
|
+
npm run benchmark:gif -- --gradient-study
|
|
109
|
+
node scripts/benchmark-gradient-loop.mjs
|
|
110
|
+
# Run for each output directory printed by the commands above:
|
|
111
|
+
node scripts/analyze-gif-gradient.mjs benchmark-results/<directory>
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
The export study uses 12 paired repeats per adapter (48 measured exports), 1920×1080, 60 frames/three seconds, 20 fps, 128 colors and high quality. Each core/adapter first completes a full-size, full-duration excluded warm-up. Adapter order and before/after order are independently balanced in four-repeat blocks. All output hashes matched. `--gradient-study` implies comparison against `2f71f3c` and cannot be combined with `--quick`.
|
|
115
|
+
|
|
116
|
+
The loop probe extracts the actual old/new loops from source, asserts that their only difference is the opaque-pixel condition, and runs them as separate functions on six gradient frames. It includes a third function with identical baseline code as an A/A control. After warm-up, all six execution-order permutations are repeated six times (36 blocks, six frames per sample). Rendering, copying, scheduling and encoding are excluded from the loop timer; output bytes are checked before measurement. The standalone functions do not reproduce the JIT context of the production async exporter, so their overhead cannot be directly assigned to total export time.
|
|
117
|
+
|
|
118
|
+
Environment: AMD FX-8350, Windows 10 build 19045, Chromium 156.0.8078.4, Node 22.17.0. Current production core: `971789a` (unchanged from the optimization); benchmark tooling was uncommitted. No outliers were removed. Effects below are geometric means of paired ratios; positive means slower. Intervals use 20,000 paired bootstrap resamples with fixed seed 1729. Adjacent runs can be correlated, so these intervals describe this session and are not universal guarantees.
|
|
119
|
+
|
|
120
|
+
| Measurement | Pairs | Before median | After median | Paired change | Approximate 95% interval | Slower pairs |
|
|
121
|
+
| --- | ---: | ---: | ---: | ---: | --- | ---: |
|
|
122
|
+
| Full export, Sync | 12 | 13.154 s | 12.994 s | -1.20% | -1.74% to -0.68% | 1/12 |
|
|
123
|
+
| Full export, Worker | 12 | 13.292 s | 13.210 s | +0.17% | -0.68% to +1.13% | 6/12 |
|
|
124
|
+
| Extracted blending loop | 36 | 65.10 ms/frame | 66.68 ms/frame | +2.60% | +2.09% to +3.13% | 34/36 |
|
|
125
|
+
| Identical-code loop control | 36 | 65.10 ms/frame | 65.19 ms/frame | approximately 0% | -0.43% to +0.40% | 18/36 |
|
|
126
|
+
|
|
127
|
+
The loop slowdown appears with either ordering (+2.79% when baseline runs first, +2.42% when optimized runs first), while the identical-code control remains near zero. This supports a real small cost of the added condition in this isolated experiment. For full exports, Sync remains faster under either order (-0.71% / -1.69%); Worker crosses zero (-0.18% / +0.52%). The earlier 4–6% export regression is therefore best treated as run-to-run variability and measurement conditions, rather than a demonstrated repeatable regression of that size. The experiment does not identify whether the earlier variation came from CPU load, warm-up, JIT, GC or another source, nor prove zero overhead on other engines or hardware.
|
|
128
|
+
|
|
129
|
+
Raw measurements and individual pairs are saved locally in `results.json` and `analysis.json`; `analysis.md` includes the paired intervals and exact two-sided sign test. The library implementation and package version were not changed by this investigation.
|
package/docs/gif-encoder.md
CHANGED
|
@@ -33,6 +33,12 @@ The controller's optional `onProgress` observer works with every encoder. It rep
|
|
|
33
33
|
|
|
34
34
|
## Verification
|
|
35
35
|
|
|
36
|
+
The browser suite also serves the real `examples/gif.html`. It triggers all three limit messages, corrects settings and retries, downloads and independently decodes the 800×450 GIF (121 frames, 8 seconds including the final hold), checks stale preview/download removal after a new failure, and cancels with blank/unlimited fields. These checks run in Chromium and Firefox and wait for observed Workers to close.
|
|
37
|
+
|
|
38
|
+
GIF limits introduced in 0.5.0 (`maxFramePixels`, `maxFrames`, `maxDurationMs`) are enforced by the controller before pixel readback. Bounds are inclusive, duration uses per-frame 10 ms rounding, and playback repeats do not consume the duration budget. Invalid values fail before encoder creation; an exceeded limit makes the export terminally failed and disposes its encoder. See the English/Russian README for the API and failure semantics. Tests exercise exact bounds, rounded delays, failed progress counters, no readback/submission of rejected frames, listener cleanup, one-time disposal, real Worker termination and a fresh independently decoded export for both adapters. Browser checks cover frame/duration failures and restart in Chromium/Firefox; Node checks additionally cover pixel-limit failure during Worker startup.
|
|
39
|
+
|
|
40
|
+
Run `npx playwright install chromium firefox` once, then `npm run test:browser`. The browser suite serves built modules with `worker-src 'self'`, observes actual Worker creation/closure, exports through a real canvas and downloads the GIF for independent decoding with gifuct-js. It checks every pixel, dimensions, frame count/durations, repeat count and progress events. An open-ended frame producer keeps the cancellation case active until the test clicks Cancel after observing a completed frame; no machine-speed-dependent delay is used. A second export on the same page must create and close a fresh Worker and produce the expected GIF. CI installs both browsers and retains the HTML report, GIF outputs, and failure screenshots/traces for seven days. These tests exercise direct module loading; the separate Next.js job covers bundler compilation.
|
|
41
|
+
|
|
36
42
|
`npm run test:next` packs the local library and builds an isolated Next.js 16.3.0 / React 19.2.8 consumer with Turbopack. These are the site's installed versions at the time this fixture was added. Its client component uses the default Worker URL, export progress and cancellation under `worker-src 'self'`. The temporary directory is printed and retained for browser checks with the printed `npm start` command. A build pass alone does not establish browser execution or cancellation; verify those separately. This check does not modify the website checkout or publish a package. CI runs the build as a separate job.
|
|
37
43
|
|
|
38
44
|
Tests use gifuct-js as an independent parser/LZW decoder, checking full pixel patches, dimensions, frame delays, long final holds, loop extension values, alpha composition and palette limits. Lifecycle tests cover pre-abort, abort/dispose during pending frame/finalization calls, late rejection, codec exceptions, tainted canvas, resize, empty exports, duplicate operations and exactly-once adapter disposal. Releasing references makes buffers eligible for garbage collection; tests do not claim a particular GC schedule.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
# Offline AAC-LC in MP4 (since 0.5.0)
|
|
2
|
+
|
|
3
|
+
Use the optional AAC factory with the existing controller, H.264 provider and MP4 writer:
|
|
4
|
+
|
|
5
|
+
```js
|
|
6
|
+
import { createOfflineVideoExporter } from '@grundyjs/canvas-take/video';
|
|
7
|
+
import { createWebCodecsVideoEncoder } from '@grundyjs/canvas-take/video/webcodecs';
|
|
8
|
+
import { createAacAudioEncoder } from '@grundyjs/canvas-take/video/aac';
|
|
9
|
+
import { createMp4Muxer } from '@grundyjs/canvas-take/video/mp4';
|
|
10
|
+
|
|
11
|
+
const exporter = await createOfflineVideoExporter({
|
|
12
|
+
video: { encoder: createWebCodecsVideoEncoder(), config: videoConfig },
|
|
13
|
+
audio: { encoder: createAacAudioEncoder(), config: {
|
|
14
|
+
codec: 'mp4a.40.2', bitstreamFormat: 'aac',
|
|
15
|
+
sampleRate: narration.sampleRate, numberOfChannels: narration.channels.length,
|
|
16
|
+
bitrate: 192_000
|
|
17
|
+
}, source: narration, endPolicy: 'trim-or-pad' },
|
|
18
|
+
muxer: createMp4Muxer(), signal,
|
|
19
|
+
maxAudioInputBytes: 64_000_000, maxOutputBytes: 64_000_000
|
|
20
|
+
});
|
|
21
|
+
try {
|
|
22
|
+
for (const frame of timeline) {
|
|
23
|
+
drawFrame(frame);
|
|
24
|
+
await exporter.addFrame(canvas, { timestampUs: frame.startUs, durationUs: frame.durationUs });
|
|
25
|
+
}
|
|
26
|
+
const result = await exporter.finish();
|
|
27
|
+
// result.audioPresentationSamples counts audible input, excluding codec priming/padding.
|
|
28
|
+
} finally {
|
|
29
|
+
exporter.dispose();
|
|
30
|
+
}
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
`narration` is `PreparedAudio`: planar Float32 PCM, finite values in [-1, 1], equal channel lengths. The caller supplies the canvas, timeline, drawing function and [Baseline AVC config](offline-video-adapters.md). Audio uses 44,100 or 48,000 Hz with one or two channels and AAC-LC/raw `aac` framing. Other rates, channel layouts, profiles or ADTS framing are explicitly unsupported. Bitrate is an integer from 8,000 to 512,000 bps. There is no resampling, mixing, live audio capture, automatic encoder substitution or native AAC fallback. Omit `audio` to retain video-only export.
|
|
34
|
+
|
|
35
|
+
`error` requires source sample count to match `round(videoDurationUs * sampleRate / 1e6)`; `trim-or-pad` explicitly discards extra input or adds PCM silence. Codec padding is separate and is never counted as accepted source samples. The [full typed narrated example](drafts/offline-video-example.ts) uses the real providers. The [browser example](../examples/offline-video.html) offers an optional generated AAC tone, cancellation, preview and download.
|
|
36
|
+
|
|
37
|
+
## Backend, deployment and resource ownership
|
|
38
|
+
|
|
39
|
+
`createAacAudioEncoder({ workerUrl?, timeoutMs? })` implements `AudioEncoderFactory`. A dedicated module Worker loads the pinned **@mediabunny/aac-encoder 1.65.0 libavcodec WASM build**. CanvasTake uses that package's published WASM ABI directly; it does not globally register a Mediabunny encoder or create a second hidden container. This makes the audio encoder and MP4 writer independently replaceable, and avoids guessing native WebCodecs AAC latency. The source algorithm remains FFmpeg's existing AAC encoder.
|
|
40
|
+
|
|
41
|
+
The default Worker URL is relative to `dist/video-aac.js`: `./video-aac-worker.js`. Copy both assets together when serving unbundled files. With a bundler, copy the `@grundyjs/canvas-take/video/aac-worker` asset and pass its emitted URL as `workerUrl`; do not evaluate the worker entry on the main thread. The bundled Worker contains its WASM, requiring no additional binary fetch, Blob Worker, SharedArrayBuffer or cross-origin isolation. Browser WebAssembly/SIMD support and permission to load the module Worker are required. A strict CSP must permit the worker asset and WebAssembly compilation (for example, `worker-src 'self'` and the appropriate `script-src 'wasm-unsafe-eval'` policy). Native H.264 availability is checked separately.
|
|
42
|
+
|
|
43
|
+
The source PCM is copied by the controller. The adapter accumulates one 1,024-sample interleaved frame; only owned copies are transferred to the Worker. One command is in flight at a time. Raw input, encoded response bytes and the serial output sink have finite budgets; an undersized `maxPendingBytes` rejects rather than waiting forever. The WASM heap, retained source PCM, muxer output pages and final Blob are separate allocations, so these work budgets are not a total heap limit.
|
|
44
|
+
|
|
45
|
+
The default per-command timeout is 30 seconds, including startup; `timeoutMs` must be an integer in 1..2147483647. Worker errors, malformed responses and timeouts fail the export. Abort/dispose immediately terminate the Worker, reject outstanding commands/output waits and suppress late messages. Successful finalization also terminates the Worker. A fresh exporter can run after cancellation, size failure or initialization error; caller PCM remains usable.
|
|
46
|
+
|
|
47
|
+
## Encoder delay, end padding and MP4 representation
|
|
48
|
+
|
|
49
|
+
The adapter reads **sample-unit PTS and durations from libavcodec AVPackets**. A negative first packet position defines leading discard; final packet end minus accepted PCM count defines trailing discard. It checks contiguous 1,024-sample packet intervals and a valid delay instead of assuming that all AAC backends have the same latency. In this pinned backend, independent decoding measured 1,024 leading samples at both supported rates. For `N` accepted samples, trailing padding is `(1024 - N % 1024) % 1024`.
|
|
50
|
+
|
|
51
|
+
Packet timestamp/duration conversion uses absolute integer sample boundaries. The MP4 writer recovers sample positions, retains one bounded final AAC packet until the audio end event, writes negative first PTS as an **edit list** that removes leading priming, and shortens the final MP4 sample duration to the exact presentation endpoint. It preserves the compressed AAC payload needed for decoder overlap. It validates leading/trailing accounting and the controller's final sample count before returning the Blob.
|
|
52
|
+
|
|
53
|
+
The audio `mdhd`/`stts` tables use the sample rate: media duration is `leading + N`. The edit's media start is `leading`; thus the sample-table presentation is exactly `N`. The movie and edit segment duration use 57,600 Hz ticks, while video remains quantized from absolute microsecond boundaries. Edit duration differs by at most one movie tick (17.36 µs), with at most one audio sample of end quantization in a player. This does not accumulate as drift over the recording.
|
|
54
|
+
|
|
55
|
+
**Raw decoder output is not the presentation sample count.** FFmpeg 6's MP4 demuxer applies the leading edit, but its AAC-to-raw-PCM path returns the complete final coded frame. For 50,003 accepted samples, a direct decode returns 50,176 samples: the last 173 are beyond MP4 presentation. Ignoring edit lists returns 51,200 samples, including the 1,024 leading samples as well. When exporting raw PCM from such a decoder, apply the container endpoint. The tests independently read MP4 tables, validate that endpoint against the source count, and apply it with `atrim=end_sample=<container count>`; they never use an exporter result or hard-coded expected length to choose the trim. Do not infer support for every third-party player's gapless handling from these tests.
|
|
56
|
+
|
|
57
|
+
## Independent verification
|
|
58
|
+
|
|
59
|
+
`test/browser/offline-audio.spec.mjs` exports actual files in Chromium and Firefox. A separate FFmpeg executable decodes every packet, while a small reader independently checks `mdhd`, `stts` and `elst`:
|
|
60
|
+
|
|
61
|
+
- 44.1 kHz mono / 50,003 samples and 48 kHz stereo / 120,017 samples; exact table-derived presentation counts and both channels' first/last signal windows.
|
|
62
|
+
- Counts 1, 1,023, 1,024 and 1,025 around codec-frame boundaries, plus explicit source trim and silence padding.
|
|
63
|
+
- Band-limited noise correlation: zero-sample lag after the priming edit, 1,024-sample lag when the edit is ignored. Tests require correlation above 0.9.
|
|
64
|
+
- Four synchronized flashes/tones over 137,003 samples with deliberately delayed rendering. Full native H.264/AAC playback must reach `ended`. A/V marker offset must stay below 3 ms, with less than 2 ms spread. Ignoring the priming edit is the negative control and must produce more than 15 ms of offset.
|
|
65
|
+
- Worker closure on cancellation/initialization/output-size failures, preserved PCM and a successful retry.
|
|
66
|
+
|
|
67
|
+
Node tests additionally cover startup/encode/flush/blocked-output cancellation, timeouts, invalid packets, absent delay, configuration/budget errors and idempotent disposal. Artifacts include MP4s, sample-count tables and A/V metrics under `test-results/`; the existing browser CI artifact includes them. FFmpeg used for verification is independent of the pinned WASM encoder build.
|
|
68
|
+
|
|
69
|
+
Local verification on 2026-11-02: 60 Node tests passed; the complete one-worker browser run passed 44 tests and skipped four existing MediaRecorder MP4/audio cases unavailable in Firefox. All ten new AAC cases passed in both browsers. The four corrected A/V marker offsets were 0 ms at the detector's 1 ms resolution, with 0 ms measured spread; the ignored-edit negative control measured about 21 ms on every marker. These are measurements of the fixtures, not a universal timing guarantee for other decoders.
|
|
70
|
+
|
|
71
|
+
| Fixture | Presentation samples | Leading discard | Trailing discard | Raw decode after leading edit |
|
|
72
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
73
|
+
| 44.1 kHz mono | 50,003 | 1,024 | 173 | 50,176 |
|
|
74
|
+
| 48 kHz stereo | 120,017 | 1,024 | 815 | 120,832 |
|
|
75
|
+
| 48 kHz A/V markers | 137,003 | 1,024 | 213 | 137,216 |
|
|
76
|
+
|
|
77
|
+
Both browsers produced the same sample accounting. Correlation of the first and last signal windows in each non-padded channel found zero-sample lag. The MP4 movie/edit duration remains quantized as described above.
|
|
78
|
+
|
|
79
|
+
The 0.5.0 release candidate was also installed from its tarball and served using only its installed assets (`npm run test:release-video`). A real portrait 1080×1920 H.264 Baseline/AAC MP4 contained 300 decoded frames over exactly 10 seconds and 480,000 presentation samples at 48 kHz. Native playback reached `ended`; independent decoding and a PNG frame confirmed dimensions. Priming was 1,024 samples and padding 256; four flash/tone markers at 0.5, 3, 6 and 9 seconds had 0 ms measured offset at 1 ms detector resolution. The local fixture file was 498,885 bytes; compressed bytes can vary by native encoder/backend.
|
|
80
|
+
|
|
81
|
+
## Distribution and source
|
|
82
|
+
|
|
83
|
+
The AAC Worker includes a size-reduced **runtime FFmpeg AAC encoder**, unlike the separate `ffmpeg-static` test executable. This code is loaded only by the optional AAC Worker; the controller, WebCodecs video provider and MP4 writer do not load it. The pinned npm package is a dev dependency used to build the self-contained runtime, not a required dependency of consumers.
|
|
84
|
+
|
|
85
|
+
The package ships `dist/aac-licenses/` (MPL-2.0 wrapper license, FFmpeg LGPL-2.1-or-later license, upstream license notes and `bridge.c`), `dist/aac-PROVENANCE.md` (exact FFmpeg/Emscripten revisions and rebuild commands), and `dist/video-aac-worker.js.map` with the adapter and glue source. The build only converts the published glue's CommonJS export wrapper to ESM; it does not modify the WASM. The unmodified FFmpeg source revision is [60a2818f76dcd1fee3155b1791304af104a0fc1b](https://github.com/FFmpeg/FFmpeg/tree/60a2818f76dcd1fee3155b1791304af104a0fc1b); the bridge/build instructions are also in the [upstream AAC package](https://github.com/Vanilagy/mediabunny/tree/main/packages/aac-encoder). Applications redistributing modified runtime assets should preserve those licenses, source information and rebuild/replacement access.
|
|
86
|
+
|
|
87
|
+
Primary references: [AAC WebCodecs registration and priming](https://www.w3.org/TR/webcodecs-aac-codec-registration/), [AAC encoder extension](https://mediabunny.dev/guide/extensions/aac-encoder), [Apple AAC delay explanation](https://developer.apple.com/documentation/quicktime-file-format/background_aac_encoding). WebCodecs native AAC support alone is not treated as proof of accurate priming.
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# Offline H.264 / AAC / MP4 adapters (since 0.5.0)
|
|
2
|
+
|
|
3
|
+
The offline pipeline exports **H.264 video with optional AAC-LC audio**. The snippet below is video-only; see the [AAC guide](offline-aac.md) for prepared narration, delay, trimming, sample counts and runtime deployment. Import the controller and providers separately:
|
|
4
|
+
|
|
5
|
+
```js
|
|
6
|
+
import { createOfflineVideoExporter } from '@grundyjs/canvas-take/video';
|
|
7
|
+
import { createWebCodecsVideoEncoder } from '@grundyjs/canvas-take/video/webcodecs';
|
|
8
|
+
import { createMp4Muxer } from '@grundyjs/canvas-take/video/mp4';
|
|
9
|
+
|
|
10
|
+
const exporter = await createOfflineVideoExporter({
|
|
11
|
+
video: { encoder: createWebCodecsVideoEncoder(), config: {
|
|
12
|
+
codec: 'avc1.42E028', bitstreamFormat: 'avc', width: 1080, height: 1920,
|
|
13
|
+
bitrate: 8_000_000, frameRateHint: { numerator: 30, denominator: 1 }
|
|
14
|
+
} },
|
|
15
|
+
muxer: createMp4Muxer(), signal,
|
|
16
|
+
maxFramePixels: 1080 * 1920, maxFrames: 300, maxDurationUs: 10_000_000,
|
|
17
|
+
maxOutputBytes: 64_000_000
|
|
18
|
+
});
|
|
19
|
+
try {
|
|
20
|
+
const boundary = i => Math.round(i * 1e6 / 30);
|
|
21
|
+
for (let i = 0; i < 300; i++) {
|
|
22
|
+
drawFrame(i);
|
|
23
|
+
await exporter.addFrame(canvas, {
|
|
24
|
+
timestampUs: boundary(i), durationUs: boundary(i + 1) - boundary(i)
|
|
25
|
+
});
|
|
26
|
+
}
|
|
27
|
+
const result = await exporter.finish(); // video/mp4, H.264, no audio track
|
|
28
|
+
} finally {
|
|
29
|
+
exporter.dispose();
|
|
30
|
+
}
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
The caller supplies `canvas`, `drawFrame` and `signal`. Native codec support depends on the browser, OS and configuration. Use localhost or HTTPS; initialization checks the exact configuration and rejects unsupported choices. It never silently selects another codec or container. The controller still accepts other independently implemented factories.
|
|
34
|
+
|
|
35
|
+
For a runnable preview/download UI, build, run `node examples/server.cjs`, and open
|
|
36
|
+
`http://127.0.0.1:3012/canvas-take/examples/offline-video.html`. It renders 120 portrait frames, then holds the last frame until five seconds; an optional generated AAC tone covers the same timeline. Cancellation releases the session; the next export creates fresh providers. Starting another export clears and revokes the previous preview URL.
|
|
37
|
+
|
|
38
|
+
## WebCodecs video profile
|
|
39
|
+
|
|
40
|
+
`createWebCodecsVideoEncoder()` implements `VideoEncoderFactory`. This initial profile accepts explicit H.264 Baseline codec strings (`avc1.42xxxx`), AVC packet framing, positive even dimensions and an explicit bitrate. High/Main profiles, Annex B, other codecs, alpha and HDR are rejected. The AAC audio factory is a separate optional import. A `checkSupport` success is a native capability check, not a guarantee that later allocation/output succeeds; incompatible decoder descriptions fail during export.
|
|
41
|
+
|
|
42
|
+
The provider uses native **quality** mode, which does not permit frame dropping. Baseline excludes B pictures, and the adapter checks the returned avcC profile as well as packet timestamps/keys. It drains the native encoder after each input and waits for output consumption before accepting another frame. This intentionally favors bounded memory and deterministic timing over maximum encoding throughput. This native drain does not end the adapter track: only the adapter's final `flush()` emits `end`. It creates one temporary VideoFrame per input and closes it immediately after native encode submission.
|
|
43
|
+
|
|
44
|
+
Duration comes from the input interval, including variable duration and a long final hold; missing or differently rounded native chunk duration cannot shorten the interval. Dropped/duplicate/reordered output, a missing requested key frame, incompatible decoder configuration or absent supported color metadata fails the export. Asynchronous native callbacks are bridged to an owned, bounded output queue. Oversized packets/configuration fail before entering an unbounded queue. Cancellation closes the native encoder, rejects blocked work and suppresses late callbacks.
|
|
45
|
+
|
|
46
|
+
The input snapshot is opaque sRGB RGBA8. Its native VideoFrame declares BT.709 primaries, sRGB transfer and RGB/full range. The returned decoder metadata must describe BT.709 primaries/matrix with BT.709 or sRGB transfer and an explicit range flag. These **actual output** fields and the avcC bytes are passed to the MP4 writer; the adapter does not invent a common range/transfer for different browsers.
|
|
47
|
+
|
|
48
|
+
## MP4 writer and timeline precision
|
|
49
|
+
|
|
50
|
+
`createMp4Muxer()` implements `MuxerFactory`, using bundled Mediabunny 1.65.0. It accepts Baseline AVC video and optional AAC-LC audio at 44.1/48 kHz, mono/stereo. Audio descriptions and sample-unit accounting pass through the [AAC protocol](offline-aac.md); unsupported formats are rejected. Decoder descriptions, packet bytes and actual color metadata pass through to the container. The result is a regular MP4 with `moov` after `mdat`, rather than fragmented MP4 or an in-memory fast-start layout. No reencoding occurs in the writer.
|
|
51
|
+
|
|
52
|
+
The pinned writer uses a **57,600 Hz movie/track timescale** without a `frameRate` track option. A frame-rate hint configures encoding only; it must not snap variable intervals in the muxer. The adapter rounds **absolute** microsecond boundaries to ticks and subtracts adjacent tick positions. This preserves the final hold and prevents accumulated rounding drift at fractional rates. Timeline endpoints differ from requested time by at most half a tick (about 8.68 µs), while individual intervals can differ by up to one tick (about 17.36 µs). Positive intervals that collapse to zero ticks, or intervals exceeding the MP4 sample-duration field, are rejected. Result `durationUs` remains the requested timeline; container time is the quantized value.
|
|
53
|
+
|
|
54
|
+
Encoded payload is counted against `maxOutputBytes`; actual header/media writes also check their end positions. Stream writes reject on overflow before the adapter retains those bytes. Seekable 64 KiB pages support metadata patching without building an ever-growing contiguous JS array. The final Blob copies the output, and pages are released after finalization or disposal. This is an output-size limit, not a cap on browser/native heap use: page slack, sample tables, muxer staging, native allocations and the Blob copy also use memory. Finalization checks frame count and endpoint against the controller summary.
|
|
55
|
+
|
|
56
|
+
## Verification
|
|
57
|
+
|
|
58
|
+
`test/browser/offline-video-real.spec.mjs` runs both providers on real canvas frames in Chromium and Firefox. It checks:
|
|
59
|
+
|
|
60
|
+
- Portrait dimensions, H.264 without audio, distinct decoded colors and full-file playback to `ended`.
|
|
61
|
+
- Independently demuxed FFmpeg PTS, DTS, sample durations and every decoded frame, including the long final hold.
|
|
62
|
+
- Exact MP4 movie/track header duration and a single video track, using a small independent box reader.
|
|
63
|
+
- Sixty frames at 30000/1001 with absolute rounded microsecond boundaries plus a final hold; fast and artificially delayed rendering produce the same file timeline.
|
|
64
|
+
- Cancellation after actual packet output, MP4 size failure, native codec closure and successful creation of a fresh export.
|
|
65
|
+
|
|
66
|
+
Node tests cover unavailable/unsupported configurations, native failures, malformed/reordered output, callback queue bounds, blocked-output cancellation, temporary VideoFrame closure, header size limits and intervals too small for the container timebase. Packed-consumer checks verify separate public imports/types without requiring a native encoder in Node. Chromium support is required in CI; other browsers report a precise skip if the requested native configuration is unavailable. On the local Windows verification (2026-11-02), all eight real-pipeline/example tests passed in Chromium and Firefox.
|
|
67
|
+
|
|
68
|
+
Both browsers produced these independently measured timelines:
|
|
69
|
+
|
|
70
|
+
| Scenario | Frames | Requested duration (µs) | MP4 duration (ticks at 57,600 Hz) | Endpoint error (µs) |
|
|
71
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
72
|
+
| Variable intervals + final hold | 5 | 900,125 | 51,847 | −3.4722 |
|
|
73
|
+
| 30000/1001 + final hold, fast | 61 | 2,502,001 | 144,115 | −4.4722 |
|
|
74
|
+
| Same timeline, delayed rendering | 61 | 2,502,001 | 144,115 | −4.4722 |
|
|
75
|
+
|
|
76
|
+
Before AAC integration, the Node suite passed 56 tests. That complete browser run with one worker passed 34 tests and skipped four older MediaRecorder MP4/audio cases unavailable in Firefox. An earlier two-worker run completed all new checks but hit a Firefox `Browser.removeBrowserContext` teardown error after a successful GIF export; the complete single-worker rerun passed without library changes. Local MP4s and inspection JSON are saved under `test-results/`; CI collects them through the existing browser artifact upload.
|
|
77
|
+
|
|
78
|
+
The independent FFmpeg executable remains a development/test dependency. The new optional AAC Worker includes a separate pinned libavcodec WASM encoder at runtime; see its [licenses, provenance and sample-count verification](offline-aac.md). Native video bytes may vary across backends; timeline equality and decodability are the assertions. Real AAC/MP4 priming, padding and A/V are now tested. WebM and higher-throughput scheduling remain future work.
|
|
79
|
+
|
|
80
|
+
## Distribution and sources
|
|
81
|
+
|
|
82
|
+
The controller and WebCodecs entry import no third-party muxer. Only `/video/mp4` includes Mediabunny. There is no runtime npm dependency; the pinned dev dependency is bundled for direct browser use. The package includes its MPL-2.0 license in `dist/mediabunny-LICENSE.txt` and the bundled modules' complete source text in `dist/video-mp4.js.map` (`sourcesContent`). CanvasTake's own sources remain MIT. Mediabunny source is available in the [upstream repository](https://github.com/Vanilagy/mediabunny) and the pinned npm source distribution. Its legacy DOM declarations conflict with TypeScript 6's built-in declarations, so the repository build skips dependency declaration checking; project code stays strict and the packed consumer independently typechecks the public declarations.
|
|
83
|
+
|
|
84
|
+
Relevant primary references: [WebCodecs quality mode](https://www.w3.org/TR/webcodecs/#latency-mode), [AVC framing/decoder descriptions](https://www.w3.org/TR/webcodecs-avc-codec-registration/), [Mediabunny encoded packet sources](https://mediabunny.dev/api/EncodedVideoPacketSource), [MP4 output options](https://mediabunny.dev/guide/output-formats#mp4).
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
# Offline video API and adapter protocol
|
|
2
|
+
|
|
3
|
+
Status: controller and real H.264/AAC adapters available in 0.5.0 (2026-11-02). `src/video.ts` and `/video` implement this protocol. Separate `/video/webcodecs`, `/video/aac` and `/video/mp4` providers export actual H.264 Baseline / AAC-LC MP4; see [adapter usage, limits and verification](offline-video-adapters.md). The [contract reference](drafts/offline-video-api.d.ts) re-exports the canonical declarations; the [narrated usage example](drafts/offline-video-example.ts) uses real factories, with caller-supplied canvas/rendering/PCM. This is an API specification, not the personal development roadmap.
|
|
4
|
+
|
|
5
|
+
## Goal and scope
|
|
6
|
+
|
|
7
|
+
Export a supplied timeline to a complete video file, with optional prepared audio, regardless of rendering/encoding speed. The first integration target is portrait MP4 H.264/AAC for Shorts; WebM VP8/Opus follows using the same controller. Codec and container implementations are separate, caller-selected adapters. There is no implicit codec choice, real-time capture, audio device or MediaRecorder dependency in the offline controller.
|
|
8
|
+
|
|
9
|
+
The existing recorder and GIF APIs keep their own lifecycles. `/video` is the controller entry point and bundles no codec, muxer, FFmpeg or Worker implementation. `/video/webcodecs` supplies a native video encoder factory; `/video/mp4` separately bundles Mediabunny 1.65.0 as an MP4 writer with optional AAC-LC audio. `/video/aac` loads a dedicated libavcodec WASM Worker, with packet-derived priming and padding. These remain optional, independently replaceable factories; see the [AAC guide](offline-aac.md).
|
|
10
|
+
|
|
11
|
+
The first version accepts fixed-size 2D HTMLCanvasElement/OffscreenCanvas frames and one prepared PCM audio track. Native VideoFrame input, multiple tracks, mixing, resampling, arbitrary gaps, B-frame reordering, streaming file sinks and HDR are deferred. This bounds the first implementation without tying the adapter interfaces to WebCodecs objects.
|
|
12
|
+
|
|
13
|
+
## Main API
|
|
14
|
+
|
|
15
|
+
```ts
|
|
16
|
+
const exporter = await createOfflineVideoExporter({
|
|
17
|
+
video: { encoder: myVideoEncoderFactory, config: videoConfig },
|
|
18
|
+
audio: { encoder: myAudioEncoderFactory, config: audioConfig,
|
|
19
|
+
source: preparedNarration, endPolicy: 'trim-or-pad' }, // optional
|
|
20
|
+
muxer: myMp4MuxerFactory,
|
|
21
|
+
signal,
|
|
22
|
+
maxFramePixels: 1080 * 1920,
|
|
23
|
+
maxFrames: 300,
|
|
24
|
+
maxDurationUs: 10_000_000
|
|
25
|
+
});
|
|
26
|
+
try {
|
|
27
|
+
for (let i = 0; i < 300; i++) {
|
|
28
|
+
draw(i);
|
|
29
|
+
const start = Math.round(i * 1_000_000 / 30);
|
|
30
|
+
const end = Math.round((i + 1) * 1_000_000 / 30);
|
|
31
|
+
await exporter.addFrame(canvas, { timestampUs: start, durationUs: end - start });
|
|
32
|
+
}
|
|
33
|
+
const result = await exporter.finish();
|
|
34
|
+
// result: blob, MIME, extension, actual codec strings, durationUs, frame/sample counts.
|
|
35
|
+
} finally {
|
|
36
|
+
exporter.dispose();
|
|
37
|
+
}
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Initialization is asynchronous because support checks, Worker startup or codec configuration may be asynchronous. `addFrame` returns a promise but snapshots the canvas before its first asynchronous yield, as the GIF API does. Await each operation before the next one. There is no `start`, `pause`, `resume`, timer-driven duration limit or automatic finish: the producer controls time by submitting intervals and finishes explicitly. A UI may simply stop feeding frames without adding time to the output.
|
|
41
|
+
|
|
42
|
+
One exporter owns one file. `finish()` drains and finalizes; repeated calls return the same promise/result, including while finalizing. It rejects for zero frames. Once finish is requested, adding frames rejects. Concurrent addFrame calls or finish during an addFrame reject without cancelling the accepted operation. A failed/aborted/disposed session cannot be restarted; a fresh session is required.
|
|
43
|
+
|
|
44
|
+
## Timeline rules
|
|
45
|
+
|
|
46
|
+
- All `*Us` fields are integer microseconds; input timestamps are nonnegative safe integers, durations are positive safe integers, and every sum must stay safe. GIF's 10 ms rounding is not used.
|
|
47
|
+
- The first video frame starts at zero; later timestamps must equal the preceding frame's end. Reject gaps, overlaps, resizes or invalid sums before reading pixels. No implicit black frames, trimming or re-timing.
|
|
48
|
+
- Variable frame durations are part of the contract. `frameRateHint` is an encoder hint, not a source of timestamps. A long final hold is the last frame's explicit duration; adapters unable to preserve it must reject rather than invent a duration. A future CFR helper can explicitly render repeated frames instead.
|
|
49
|
+
- For fractional rates use absolute rounded boundaries, e.g. `round(i * 1e6 * denominator / numerator)`, then subtract adjacent boundaries. Validate the arithmetic; do not repeatedly add a rounded frame duration.
|
|
50
|
+
- Requested presentation duration is the final video frame's end. Muxers rescale absolute boundaries to container ticks and derive their deltas, avoiding cumulative rounding. Container duration must agree within one chosen container tick, with at most one audio sample of end quantization. Distinct positive frame intervals must remain positive after rescaling; if a container timescale cannot represent them, fail rather than collapse them. Encoder priming and padding must not create visible startup/tail material.
|
|
51
|
+
- Runtime speed and CPU load may change completion time and compressed bytes, but not the supplied presentation timeline. Bit-identical encoding across machines or hardware codecs is not promised.
|
|
52
|
+
|
|
53
|
+
Input frames are snapshotted as owned sRGB RGBA8, composited against `background` (default white), with fixed configured dimensions. No implicit resize, rotation, alpha preservation or color-profile conversion beyond the declared SDR policy. The encoder describes its actual output color metadata; the muxer must preserve it. The first MP4 adapter profile must verify SDR color handling, even dimensions where required, and decoder compatibility rather than silently padding/cropping.
|
|
54
|
+
|
|
55
|
+
## Prepared audio and synchronization
|
|
56
|
+
|
|
57
|
+
`PreparedAudio` is planar Float32 PCM with equal, nonempty channel lengths and finite samples in [-1, 1]. SharedArrayBuffer-backed or detached PCM input is rejected. The sample rate/channel count must match the explicit audio config; resampling/mixing/normalization belongs to the caller or a future separate helper. The controller validates and copies the entire source synchronously before the first initialization await. Later caller mutations cannot change the export. Live MediaStreamTrack and AudioContext inputs are not accepted.
|
|
58
|
+
|
|
59
|
+
The first sample starts at timeline zero. A block's exact position is `sampleOffset / sampleRate`; its microsecond timestamp is derived from that absolute sample index, never accumulated from rounded block lengths. The controller owns interleaving: after each accepted video interval it feeds audio up to that interval's end in bounded blocks. Callers do not need to enqueue the whole audio track first or coordinate two producers. The last audio sample count is `round(videoEndUs * sampleRate / 1e6)`; safe arithmetic is required.
|
|
60
|
+
|
|
61
|
+
The implementation uses integer arithmetic for sample/timestamp rescaling, blocks of at most 2,048 samples, and periodic task yields during long audio intervals. A requested audio track must contain at least one presentation sample by finish.
|
|
62
|
+
|
|
63
|
+
`endPolicy` is mandatory when audio is present:
|
|
64
|
+
|
|
65
|
+
| Policy | Behavior |
|
|
66
|
+
| --- | --- |
|
|
67
|
+
| `error` | Reject if the prepared sample count differs from the sample-quantized video duration. Short audio fails as soon as the next frame would exceed it; extra audio fails at finish. |
|
|
68
|
+
| `trim-or-pad` | Discard source samples beyond the video endpoint, or append PCM silence if narration is shorter. These changes are explicit, not inferred from codec availability. |
|
|
69
|
+
|
|
70
|
+
There is no delayed-track offset in the first version; prepare leading silence externally. Audio encoders report the presentation sample count and leading/trailing discard counts in their end event. The muxer must express those trims in the container and align the first audible input sample with zero. Raw packet timestamps can be signed to represent codec delay; they must not be independently rebased per track. The controller validates that presentation sample counts match its submitted timeline.
|
|
71
|
+
|
|
72
|
+
An AAC adapter must establish reliable priming/padding behavior before claiming support. If the chosen backend cannot report or otherwise verifiably account for it, that combination is unsupported for this contract. Do not guess zero delay. The implemented AAC adapter uses a pinned libavcodec WASM backend with sample-unit packet positions and independent decoding tests. It does not infer delay from native WebCodecs AAC capability checks.
|
|
73
|
+
|
|
74
|
+
## Encoder and muxer contracts
|
|
75
|
+
|
|
76
|
+
See the declaration file for all types. Video and audio factories are separate, so an application can combine different implementations. Every successful `create` returns a fresh exclusively owned instance. A supplied factory is caller-owned and is never disposed; its created instances are exporter-owned.
|
|
77
|
+
|
|
78
|
+
`checkSupport` checks the exact codec string, packet framing, dimensions, rate/channel configuration and applicable protocol guarantees, returning either supported or an explicit reason. The muxer checks the requested combination before instances are allocated. Replacing AAC with Opus or MP4 with WebM is never automatic. A check is not a reservation: resource allocation or later encoding may still fail. Native WebCodecs capability checks alone do not establish container compatibility or priming accuracy.
|
|
79
|
+
|
|
80
|
+
Initialization order is: validate/snapshot inputs and options; check all providers; create muxer; create video encoder; create audio encoder if requested. The muxer instance exists before encoder callbacks can produce output. No encoder may emit media until its first input is submitted; initial config/error callbacks are allowed. If any create fails, clean up all previously created instances. Factories must clean up their own partially constructed instances when rejecting.
|
|
81
|
+
|
|
82
|
+
Each encoder gets an `EncoderContext` containing an internal AbortSignal, queue budgets and `CodecOutputSink`. Its event sequence per track is:
|
|
83
|
+
|
|
84
|
+
```text
|
|
85
|
+
config → packet* → end
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
- Config contains the actual codec/framing, decoder description bytes and dimensions or audio parameters. Equivalent config repeats are allowed; incompatible midstream changes fail. Config must precede packets requiring it, including when the first config arrives only with the first encoded output.
|
|
89
|
+
- Packets carry owned bytes, track, presentation timestamp, decode timestamp, duration and key/delta type. The first video packet must be a key frame. Initial support requires one access unit per input video frame, no dropped frames, and PTS=DTS with monotonically increasing video timestamps. The explicit decode timestamp field reserves a clear future extension; v1 rejects reordering rather than inventing DTS from frame rate or sorting compressed packets.
|
|
90
|
+
- A first encoder profile must establish no-reordering behavior, not infer the absence of B-frames from callback order alone. If native output omits duration, the adapter recovers it from the corresponding submitted frame interval rather than guessing from frameRateHint.
|
|
91
|
+
- Audio packets may include declared priming/padding outside the presentation interval; discard counts describe it. Packet time and duration must agree with the encoder's sample accounting. Do not conflate encoded packet duration with audible content duration.
|
|
92
|
+
- `encodeFrame`/`encodeAudio` resolve after consuming or independently retaining the input, and honor queue backpressure; they need not have emitted a packet yet. Input buffers cannot be retained after resolve unless independently copied/transferred under adapter ownership. A core-provided frame/block is exclusive, so a Worker adapter may transfer it; retained PCM source planes must never be transferred.
|
|
93
|
+
- `flush()` is only an end-of-track operation. It emits every delayed output and exactly one end event, then waits for all output writes to settle. It must not wait for a future frame that the controller can no longer submit.
|
|
94
|
+
- Late codec/Worker errors call `output.fail(error)`, even when no public operation is pending. That immediately terminally fails the exporter; a later operation must not hang waiting for a lost callback.
|
|
95
|
+
|
|
96
|
+
`CodecOutputSink.write` returns a promise after serial muxer consumption. Buffers are borrowed until that promise settles: the controller copies a view before deferring work, or otherwise pins its storage, before the adapter can reuse it. A WebCodecs output callback cannot simply return an async promise and expect the browser to await it. The adapter must bridge callbacks to an owned bounded queue, drain it independently, and include those writes in flush/error handling. Do not drop output or wait inside a native callback for the producer to advance.
|
|
97
|
+
|
|
98
|
+
The controller serializes muxer writes across both tracks. Arrival order across tracks is not guaranteed; the muxer owns interleaving and rescaling. `write()` must consume/copy or buffer within its budget and return without waiting for a future track packet. This prevents an audio/video interleaving deadlock. If an adapter cannot operate within a supplied bound it rejects or fails, never buffers without limit.
|
|
99
|
+
|
|
100
|
+
The audio/video input scheduler advances in short bounded audio blocks up to the current video interval, with video and audio encoder progress independent of output draining. Empty output from an encode call is valid. Delayed codecs may need later inputs to produce output, so accepting input and waiting for a matching packet must remain separate. An encoder whose minimum look-ahead cannot fit the supplied budgets must reject that configuration rather than wait forever for an input the budget prevents submitting.
|
|
101
|
+
|
|
102
|
+
## Finish, cancellation and ownership
|
|
103
|
+
|
|
104
|
+
Finish validates the final timeline/audio policy, supplies any final padded/trimmed audio block, flushes both encoders while continuing output draining, waits for both end events and all serial muxer writes, then calls `muxer.finalize(timeline)`. An implementation may flush encoders concurrently; it must keep the consumer running and preserve per-track order. Finalize receives duration, input video-frame count and presentation audio-sample count, and returns one complete Blob. Validate format/size and return immutable result metadata, then dispose owned instances.
|
|
105
|
+
|
|
106
|
+
The result reports the requested presentation duration and actual codecs/container; it must never use elapsed wall time or a captured startup offset. A format mismatch is an error, not a renamed extension. `finalize` has no implicit transcode step.
|
|
107
|
+
|
|
108
|
+
The internal signal is aborted on caller cancellation, disposal or terminal failure. Pending creation, encode, flush and finalization promises race cancellation. Abort settles the public operation promptly with the caller's signal reason (or AbortError); dispose uses AbortError. `dispose()` is synchronous and idempotent. Each acquired instance is disposed once on success/failure/cancellation. Cleanup exceptions do not replace the primary failure.
|
|
109
|
+
|
|
110
|
+
A factory resolving after cancellation must have its late instance disposed immediately. Suppress late packets/progress and attach rejection handlers to detached work. No partial Blob is returned. Native operations already executing may take time to release internal resources; prompt promise rejection is not a guarantee of instantaneous native cancellation. Caller canvases, PCM arrays and factory objects remain usable. Every temporary canvas snapshot, transferred buffer, codec object, Worker and muxer buffer created for the session has an explicit owner.
|
|
111
|
+
|
|
112
|
+
## State, progress, limits and errors
|
|
113
|
+
|
|
114
|
+
| State | Allowed operation |
|
|
115
|
+
| --- | --- |
|
|
116
|
+
| Before async creation resolves | Caller can abort via signal; no exporter exists yet |
|
|
117
|
+
| `ready` | addFrame, finish, dispose |
|
|
118
|
+
| `encoding` | dispose/abort; another addFrame or finish rejects |
|
|
119
|
+
| `finalizing` | repeat finish returns the same promise; dispose/abort |
|
|
120
|
+
| `finished`, `failed`, `aborted` | dispose; finished also permits repeat finish |
|
|
121
|
+
| `disposed` | repeat dispose only |
|
|
122
|
+
|
|
123
|
+
Progress phases are initializing, encoding, finalizing and terminal outcomes. Frozen snapshots report framesAccepted, current video interval end, audioSamplesAccepted, packets successfully consumed by the muxer and encoded payload bytes. Accepted input does not imply encoding/container finalization is complete. Audio progress counts presentation input, excluding codec padding. `totalFrames` is a positive safe-integer display hint, not a limit. `outputBytes` is known only at successful finish in v1. No initial percentage/ETA or finalizing-at-100% guarantee is invented. Observer exceptions are ignored; signal cancellation from an observer is honored.
|
|
124
|
+
|
|
125
|
+
Limits are positive safe integers, snapshotted at creation. maxFramePixels, maxFrames and maxDurationUs are inclusive; exceeding one rejects before reading/submitting the rejected frame and terminally cleans up. There is no automatic finish at a limit. maxAudioInputBytes validates the source before copying. maxOutputBytes is passed to the muxer and checked against the finalized Blob; a muxer checks its actual growing output when possible, not a guessed compression ratio. It is an output-size bound, not an exact heap bound.
|
|
126
|
+
|
|
127
|
+
Default maxPendingInputs is 4 per encoder; default maxPendingBytes is 16 MiB per adapter and for the controller's pending output queue. This bounds outstanding work, not total resident memory. Native codec allocations, the retained PCM snapshot and the growing in-memory output are separate. Output retained by a muxer is charged to maxOutputBytes, while not-yet-consumed/interleaving packets count against maxPendingBytes. Caller-raised limits remain finite safe integers. Limits other than those queue defaults are opt-in. A packet exceeding the queue-byte budget fails rather than waiting forever for space that cannot exist.
|
|
128
|
+
|
|
129
|
+
The controller additionally caps submitted video intervals whose packets have not arrived at maxPendingInputs. When that look-ahead bound is reached, the next frame rejects before readback instead of waiting for an output that might require that very input. Adapters should hold input acceptance while independently drainable work is pending, or require a larger declared look-ahead bound. Audio codec internal input limits remain the audio adapter's responsibility; the controller submits one bounded audio block at a time. The pending output queue charges payload/description bytes plus a 64-byte reservation per event, bounding even empty configuration/end events. A raw RGBA snapshot must itself fit maxPendingBytes before readback. These are conservative work budgets, not native heap accounting.
|
|
130
|
+
|
|
131
|
+
Invalid arguments use TypeError/RangeError; exceeded limits use RangeError naming the option. As creation is asynchronous, invalid initialization arguments reject its promise. Initialization support failures use NotSupportedError with the failing codec/container and reason. Operational failures identify encoder/muxer/protocol origin and preserve their cause; a concrete exported error class is deferred until implementation. Validation/support failures before allocation leave no instances; mid-session failure is terminal. State misuse rejects without cancelling a valid pending operation. Any first failure wins over cleanup and late callbacks.
|
|
132
|
+
|
|
133
|
+
## First adapter profile and acceptance criteria
|
|
134
|
+
|
|
135
|
+
The first target uses caller-selected H.264 (`avc` framing), AAC LC (`aac` framing) and an MP4 muxer. Decoder descriptions must be carried to the muxer, not reconstructed from MIME. Browser-specific resource support, no-reordering output, AAC priming and sample trimming must be independently verified. A preset may return these providers/configs later, but the controller always accepts replacements. A WebM/VP8/Opus combination exercises the same protocol and trim accounting rather than special branches in the controller.
|
|
136
|
+
|
|
137
|
+
Implementation acceptance checks:
|
|
138
|
+
|
|
139
|
+
1. Fake delayed encoders/muxers validate zero-output encode calls, final flush output, late config, asynchronous failures, no drops, per-track order, serial muxing, queue caps and no deadlocks.
|
|
140
|
+
2. Invalid timestamps, gaps/overlaps, dimension changes, fractional-rate rounding, every inclusive limit and rejection-before-readback are covered. Adding artificial render/encode delays changes wall time but not file sample counts or timestamps.
|
|
141
|
+
3. Partial initialization, abort before/during every await, late factory resolutions and abort during finalization release each acquired instance once and settle pending operations. Source PCM/canvas survive success/failure/transfer/disposal; observers cannot corrupt counters.
|
|
142
|
+
4. Real files are independently demuxed/decoded: first presentation at zero, exact frame count and supplied frame intervals, final hold, requested container/codecs, presentation audio samples, priming/tail trims, A/V markers, and duration within container/sample quantization. Decode full file including the last chunk; include negative offset/drift controls.
|
|
143
|
+
5. Silent video, strict narration mismatch, explicit trimming/padding, 44.1/48 kHz sample boundaries and unsupported AAC are explicit cases. Unsupported formats never silently switch. Full-file/container timing is checked directly, unlike real-time recorder tests that must account for capture startup material.
|
|
144
|
+
6. Importing the controller loads no codec/muxer and touches no browser globals. A packed TypeScript consumer, direct browser loading, Worker ownership and a bundled consumer are verified once actual adapters exist.
|
|
145
|
+
|
|
146
|
+
The controller/protocol is covered by `test/offline-video.test.mjs`, including delayed output, sample-clock timing, explicit trim/pad, bounded queues, protocol failures, byte transfer, late factory resolution and cancellation at all asynchronous stages. Browser fixtures exercise actual canvas readback and disposal in Chromium/Firefox. In addition, `test/browser/offline-video-real.spec.mjs` independently demuxes and decodes real silent H.264/MP4 files using FFmpeg, checks variable/fractional timing, final hold, colors, container duration, native cancellation and retry. The packed consumer checks all four public entries without browser globals. `test/browser/offline-audio.spec.mjs` checks real AAC in MP4, table-derived presentation counts, independent full-packet decoding, priming/padding, correlation, A/V markers, negative edit-list controls, playback and Worker cleanup. Raw PCM decoder padding and movie-tick quantization are documented in the [AAC guide](offline-aac.md).
|
|
147
|
+
|
|
148
|
+
## Typecheck and references
|
|
149
|
+
|
|
150
|
+
```sh
|
|
151
|
+
node node_modules/typescript/bin/tsc -p docs/drafts/tsconfig.json
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
The proposal uses microsecond packet time and explicit output draining in line with [WebCodecs](https://www.w3.org/TR/webcodecs/). AVC packet framing/decoder description and AAC framing/priming require codec-specific handling; see the [AVC registration](https://www.w3.org/TR/webcodecs-avc-codec-registration/) and [AAC registration](https://www.w3.org/TR/webcodecs-aac-codec-registration/). These sources specify codec interfaces, not a universally supported MP4 export pipeline. The packet ordering, memory budgets and public controller behavior above are CanvasTake design decisions.
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Video recording verification
|
|
2
|
+
|
|
3
|
+
The pause/resume and active-time limit API introduced in 0.5.0 is described in the English and Russian README. `examples/video.html` demonstrates manual controls, automatic completion, a generated audio track, preview and download.
|
|
4
|
+
|
|
5
|
+
## Automated checks
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
npm test
|
|
9
|
+
npm run test:package
|
|
10
|
+
npm run test:browser
|
|
11
|
+
# Focused synchronization check:
|
|
12
|
+
npx playwright test test/browser/video-sync.spec.mjs --workers=1
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Node tests use controlled clocks and a MediaRecorder fixture to verify active-time accounting across multiple pauses, stopping while paused, repeated/pending operations, error/disposal/limit races, long timer segmentation, option validation and preservation of caller-owned audio tracks. The packed consumer checks the public types.
|
|
16
|
+
|
|
17
|
+
Browser tests record actual WebM VP8/Opus and MP4 H.264/AAC when supported. They play recordings through the browser, check decoded audio and dimensions, verify that a pause longer than the budget does not stop recording, stop manually from a pause and complete automatically after resuming. Unsupported explicit codecs are skipped rather than substituted. Separate checks exercise the actual video example. Existing GIF tests run alongside these checks.
|
|
18
|
+
|
|
19
|
+
On Linux CI, a PulseAudio null sink supplies a real virtual audio output for Firefox's audio engine. Without an output, Web Audio may never start its clock, affecting both the library and direct native control. The test preflight requires `AudioContext.currentTime` to advance within five seconds and fails with an audio-output diagnostic otherwise; audio samples and synchronization assertions are not mocked or skipped.
|
|
20
|
+
|
|
21
|
+
## A/V synchronization after pauses
|
|
22
|
+
|
|
23
|
+
`video-sync.spec.mjs` records three active segments separated by two 600 ms pauses. Each segment contains two 180 ms white flashes and 1 kHz tone pulses driven from the same AudioContext clock, giving six marker pairs. The canvas is 160×90 at a target 30 fps. The producer schedules new pulses only within active segments; pausing the recorder does not pause the caller's audio clock or drawing.
|
|
24
|
+
|
|
25
|
+
The same producer is recorded through CanvasTake and through an independent, direct MediaRecorder control (including the same timeslice). Input flash scheduling lag is retained too. The test-only helper uses the pinned `ffmpeg-static` development dependency. It is not imported or bundled by the library. Installing development dependencies downloads its platform binary. FFmpeg decodes the completed file without modifying it, preserving container timestamps with `-copyts`:
|
|
26
|
+
|
|
27
|
+
- Video: one gray sample per decoded frame plus its presentation timestamp; a transition above 200 detects a flash.
|
|
28
|
+
- Audio: mono float32 at 48 kHz, plus decoded-frame presentation timestamps and sample counts. Five-ms RMS windows above 0.05 detect tone starts. Each onset is mapped back to the audio frame's timestamp, retaining the audio stream's initial offset.
|
|
29
|
+
|
|
30
|
+
Comparing PCM sample indices alone with video media times is insufficient: a PCM-only browser decoder can discard the initial audio timestamp. The independent decoder keeps both streams on the same container timeline and also verifies the actual H.264/AAC or VP8/Opus codecs.
|
|
31
|
+
|
|
32
|
+
Every file must contain all six marker pairs, with video-minus-audio onset offset spread at most 80 ms across the recording. The producer records each marker's expected active-time position from the audio clock, excluding both pauses. The relative positions of decoded audio and video markers must each match those active-time intervals within 150 ms; leaving the 1.2 seconds of pauses in the file fails. From the first marker to the decoded end, each stream must match the expected remaining active time within 300 ms. Audio/video endpoints must be within 250 ms of each other. These are end-to-end tolerances, including canvas scheduling, frame cadence and encoder behavior; they are not a sample-accurate synchronization guarantee.
|
|
33
|
+
|
|
34
|
+
A CI WebM exposed an initial video frame at PTS 0 followed by audio starting at 2,018 ms, before the active-time interval measured from the native start event. Comparing end PTS directly to active duration was therefore incorrect. Marker-relative intervals retain meaningful duration/pause checks without assuming a common zero origin or removing samples. Raw PTS and startup lead relative to the first scheduled marker are saved in metrics. The active-time budget does not trim initial encoder/startup material or impose a strict whole-file duration cap.
|
|
35
|
+
|
|
36
|
+
Absolute offset is reported separately, including a diagnostic `absoluteOffsetWithin120Ms` flag. An initial parallel run exceeded that diagnostic for Chromium MP4 (133 ms maximum, 38 ms spread); this observation is retained rather than dismissed by retrying. A constant capture/encoder startup offset is different from drift introduced by pausing. The regression assertion concerns drift; absolute offsets and the direct native control are retained for investigation. Passing this test does not promise an absolute offset under 120 ms on every machine.
|
|
37
|
+
|
|
38
|
+
A Chromium/WebM negative control remuxes a copy with audio deliberately delayed by 300 ms, using stream copy rather than re-encoding. Its absolute-offset diagnostic must exceed 120 ms, and the detector must recover the injected shift within 15 ms. A constant injected delay does not increase drift; this control verifies timestamp-aware offset measurement rather than the drift assertion. The original recording is untouched.
|
|
39
|
+
|
|
40
|
+
Recordings and `sync-metrics.json` (per-marker timestamps and offsets, codec names, stream endpoints and active/wall durations) are retained in Playwright outputs/report attachments. CI uploads those artifacts with the browser report. Codec support and observed precision can vary by browser, machine and load.
|
|
41
|
+
|
|
42
|
+
## Observed local run: 2026-11-02
|
|
43
|
+
|
|
44
|
+
Windows 10, AMD FX-8350; Chromium and Firefox installed by Playwright 1.64.1. This is one run with two workers, not a universal latency guarantee.
|
|
45
|
+
|
|
46
|
+
| Browser | Recorder | Codecs | Max absolute offset (ms) | Offset spread (ms) | Max input draw lag (ms) |
|
|
47
|
+
| --- | --- | --- | ---: | ---: | ---: |
|
|
48
|
+
| Chromium | library | h264/aac | 13.9 | 24.7 | 19.3 |
|
|
49
|
+
| Chromium | library | vp8/opus | 31 | 23.5 | 19.3 |
|
|
50
|
+
| Firefox | library | vp8/opus | 32 | 50 | 11.3 |
|
|
51
|
+
| Chromium | native | h264/aac | 23 | 31.7 | 8.7 |
|
|
52
|
+
| Chromium | native | vp8/opus | 34.5 | 21 | 11.3 |
|
|
53
|
+
| Firefox | native | vp8/opus | 29 | 35 | 10.7 |
|
|
54
|
+
|
|
55
|
+
The native control uses the same input producer and requested format. Similar values do not identify the cause of every startup offset, and the earlier 133 ms MP4 diagnostic remains an observed limitation. No growing offset beyond the 80 ms regression bound was found after the two pauses.
|