longpipe 0.0.12 → 0.0.13

package/README.md

@@ -33,6 +33,7 @@ const inputStream = await navigator.mediaDevices.getUserMedia({ video: true, aud
 
 const pipeline = new EffectsPipeline(inputStream, {
   background: 'blur',
+  audio: 'denoise'
 })
 
 videoEl.srcObject = pipeline.stream   // available immediately
@@ -98,13 +99,13 @@ new EffectsPipeline(stream, { preset: 'balanced' })   // medium model
 new EffectsPipeline(stream, { preset: 'quality' })    // xl model
 ```
 
-`'auto'` runs a microbenchmark at init and picks the largest preset that fits the per-frame budget on the current device. While `'auto'` is in effect an adaptive controller polls FPS / model time and swaps preset up or down as conditions change. Explicit preset choices (`'fast'`, `'balanced'`, `'quality'`, or a manual config) are always respected and never auto-overridden.
+`'auto'` runs a microbenchmark at init and picks the largest preset that fits the per-frame budget on the current device, **topping out at `large`** — `xl` is the explicit opt-in flagship (`'quality'` or a manual config). While `'auto'` is in effect an adaptive controller polls FPS / model time and swaps preset up or down as conditions change. Explicit preset choices (`'fast'`, `'balanced'`, `'quality'`, or a manual config) are always respected and never auto-overridden.
 
 You can also pass a manual config:
 
 ```ts
 new EffectsPipeline(stream, {
-  preset: { model: 'large', dtype: 'f32', resolution: { w: 640, h: 360 }, skipFrames: 0 },
+  preset: { model: 'large', dtype: 'f32', resolution: { w: 640, h: 400 }, skipFrames: 0 },
 })
 ```
 
@@ -177,7 +178,7 @@ pipeline.destroy()
 
 ## Self-hosting weights
 
-By default Longpipe fetches model weights from `https://cdn.longpipe.dev/models/v/0.0.4/`. You can browse the available files, sizes, and SHA-256 hashes at [cdn.longpipe.dev/models/v/0.0.4/index.html](https://cdn.longpipe.dev/models/v/0.0.4/index.html) (machine-readable list at [manifest.json](https://cdn.longpipe.dev/models/v/0.0.4/manifest.json)).
+By default Longpipe fetches model weights from `https://cdn.longpipe.dev/models/v/0.0.5/`. You can browse the available files, sizes, and SHA-256 hashes at [cdn.longpipe.dev/models/v/0.0.5/index.html](https://cdn.longpipe.dev/models/v/0.0.5/index.html) (machine-readable list at [manifest.json](https://cdn.longpipe.dev/models/v/0.0.5/manifest.json)).
 
 To serve them yourself, mirror the files under any base URL with the same `model_${name}.bin` naming convention and pass it via `weightsBaseUrl`:
 
@@ -193,22 +194,51 @@ Works on Chromium (Chrome, Edge), Firefox, and Safari (desktop and iOS). WebGPU
 
 
 
+## Face touch-up
+
+UV-space skin smoothing (the "beautify" filter). It runs **in parallel with any background effect** — both features share one encoder pass per frame, and the retouched frame feeds the background compositor.
+
+```ts
+new EffectsPipeline(stream, {
+  background: 'blur',
+  touchup: { strength: 0.6 },   // presence enables it
+})
+
+await pipeline.setTouchup({ strength: 0.8, style: 'bilateral' })   // live update
+await pipeline.setTouchup(null)                                    // disable
+```
+
+```ts
+interface TouchupOptions {
+  strength?: number   // 0..1 — blend of smoothed skin over original (default 0.6)
+  amount?:   number   // smoothing radius (default 8)
+  detail?:   number   // high-frequency (pore texture) keep — freq-sep only (default 0.35)
+  style?:    'freq-sep' | 'bilateral'   // default 'freq-sep'
+}
+```
+
+How it works: a 5-point face-keypoint head rides the matting encoder (no second encoder pass); its heatmaps are decoded on the GPU into a face box; a 478-point landmark model runs on the face crop; the face is unwrapped into a pose-independent UV atlas, smoothed there, and composited back through a per-region weight mask. Eyes, brows, and lips are protected and stay pixel-exact; `'freq-sep'` smooths while keeping pore-level texture, `'bilateral'` is a stronger edge-preserving filter. The whole chain is GPU-resident — no per-frame readback.
+
+On first enable the landmark weights (`model_landmark_mesh.bin`, ~2.6 MB) and two small static assets are fetched from `weightsBaseUrl`, then cached; parameter updates are instant. With no face in frame the effect passes through cleanly.
+
+Current limitation: single face — with multiple people in frame, detection can jump between faces. Multi-face support is on the roadmap.
+
 ## How it works
 
 Two layers:
 
-- **Model** (`src/model/`) — an EfficientNet-Lite encoder (lite0; lite4 on the `xl` tier) with a U-Net decoder, plus a lightweight U-Net *wrapper* that sharpens the matte at higher resolution and a temporal ConvGRU that smooths it across frames. Written as TypeScript op classes — each layer is a class; weights load as binary tensors at init; the backend (WebGPU or WebGL2) is injected at construction. BatchNorm is fused into conv weights at export — there is no BN op at inference.
+- **Model** (`src/model/`) — a shared EfficientNet-Lite encoder (lite0; lite3 on the `xl` tier) feeding multiple heads: a U-Net matting decoder (sharpened at higher resolution by a lightweight U-Net *wrapper*), a 5-point face-keypoint head (drives touch-up, and auto-reframe later), and a small optical-flow head that provides temporal stability — the renderer warps the previous outputs along the predicted flow and blends them through a flow-gated stabilizer, so the matte is steady without any recurrent state in the network. A separate 478-point landmark model runs on the detected face crop. Written as TypeScript op classes — each layer is a class; weights load as binary tensors at init; the backend (WebGPU or WebGL2) is injected at construction. BatchNorm is fused into conv weights at export — there is no BN op at inference.
 - **Pipeline** (`src/pipeline/`) — capability detection, per-browser frame transport selection, worker spawn, audio passthrough, autotune, and the adaptive controller. Designed to absorb the browser/codec/canvas plumbing complexity so consumers don't have to. 
 
 Five trained presets cover the hardware range. "Resolution" is the model's working (canvas) resolution; the encoder runs at a lower internal resolution and the U-Net wrapper refines back up to this.
 
 | Preset | Resolution | Encoder      | Decoder  | Skip frames |
 |--------|------------|--------------|----------|-------------|
-| xl     | 1280×720   | full (lite4) | 2× ch    | 0           |
-| large  | 640×360    | full (lite0) | standard | 0           |
-| medium | 512×288    | full (lite0) | standard | 1           |
-| small  | 384×216    | small        | standard | 1           |
-| xs     | 384×216    | small        | standard | 2           |
+| xl     | 1280×768   | full (lite3) | 2× ch    | 0           |
+| large  | 640×400    | full (lite0) | standard | 0           |
+| medium | 512×320    | full (lite0) | standard | 1           |
+| small  | 384×224    | small        | standard | 1           |
+| xs     | 384×240    | small        | standard | 2           |
 
 `Skip frames` is how many input frames the model sits out between runs — the compositor still renders every frame using the most recent alpha matte. Autotune picks one of these at init based on a microbenchmark of the actual network on the actual device.
 
@@ -221,7 +251,10 @@ Training scripts, fixture generation, and the weight export pipeline are not yet
 
 - [x] Background segmentation / virtual backgrounds
 - [x] Background noise removal (audio, separate pipeline)
-- [ ] Face Landmarks / Touchup / AR Effects
+- [x] Face landmarks + touch-up
+- [ ] Multi-face support
+- [ ] Auto-reframe
+- [ ] AR effects
 - [ ] Lighting correction