longpipe 0.0.3 → 0.0.4

package/README.md

@@ -1,6 +1,11 @@
-# Longpipe 🐲
+# Longpipe 🐉
+
+Fast, high quality virtual effects in the browser
+
+
+<img width="640" height="360" alt="0087" src="https://github.com/user-attachments/assets/525f983c-6b3d-43ff-aedf-b1203e2955c1" />
+
 
-Hardware-accelerated real-time AI media processing in the browser.
 
 Try the live demo: [longpipe.dev/demo](https://longpipe.dev/demo)
 
@@ -8,15 +13,10 @@ Try the live demo: [longpipe.dev/demo](https://longpipe.dev/demo)
 
 ## Features
 
-- Real-time portrait matting / virtual backgrounds in the browser
-- WebGPU compute shaders with a WebGL2 fragment-shader fallback
-- Worker-by-default — all model and render work runs off the main thread
-- Background options: blur, image, video, solid color, or none (passthrough)
-- Six trained model presets covering hardware from MacBook Pro down to Chromebook
-- Auto-tuned at init: picks the best preset for the current device, plus the optimal frame transport for the current browser
-- Adaptive at runtime: swaps to a smaller preset when FPS drops, larger when there's headroom
-- Single-call API: `new EffectsPipeline(inputStream, { background })` returns a `MediaStream`
-- Audio passthrough by default
+- **Support across browsers:** Works on every browser, simplifying dozens of browser inconsistencies in one simple API.
+- **Performance:** Longpipe was built from the ground up to work as well on 10-year-old netbooks as it does on the latest MacBook Pro.
+- **Adaptivity:** Longpipe has several model variants (xl to xs), auto-selecting to provide the best quality while maintaining 30 fps.
+- Built by the founder of [Vectorly](https://www.crunchbase.com/organization/vectorly), a commercial effects SDK acquired in 2021. A ground-up redesign for the WebGPU era, with years of production lessons baked in.
 
 ## Install
 
@@ -41,6 +41,25 @@ await pipeline.ready                  // optional — resolves once the effect i
 
 `pipeline.stream` is wired synchronously and emits the unprocessed input until the model is ready (~1–3s on cold start, depending on hardware), so the consumer sees live video the whole time.
 
+
+## Models
+
+Longpipe uses custom-trained models built with an EfficientNetLite encoder, and a UNet style decoder, breaking it down into 7 different variations/performance presets, which vary number of channels, encoder size as well as input size. Even at 0.0.1, the first version of Longpipe (across all variants) have much higher segmentation accuracy than alternative open source models like Mediapipe and Bodypix, while also having much better real-world performance due to implmenting the model as pure WebGPU/WebGL shaders and using a zero-copy fully GPU pipeline.
+
+### Quality
+Using average alpha pixel error on the P3M-10K valdation dataset (499 landscape images), all variants of Longpipe surpassed mediapipe in both MAE and IoU.
+
+<img width="640" alt="lp-quality" src="https://github.com/user-attachments/assets/c7044937-95af-496f-893a-a05a50d7c914" />
+
+### Performance
+With the pure GPU zero copy pipeline, Longpipe achieves better real world performance than mediapipe using much larger models.
+
+<img width="640" alt="lp-speed" src="https://github.com/user-attachments/assets/ce658b8d-dc62-4d6b-90f8-cd62dfef2a59" />
+
+
+
+
+
 ## Backgrounds
 
 The `background` option takes a wide set of inputs and normalizes them internally:
@@ -132,23 +151,7 @@ new EffectsPipeline(stream, {
 
 Works on Chromium (Chrome, Edge), Firefox, and Safari (desktop and iOS). WebGPU is used when available; WebGL2 is the fallback. Longpipe picks the optimal video frame transport for each browser internally — `MediaStreamTrackProcessor`, `transferControlToOffscreen` + `captureStream`, or an `ImageBitmap` shuttle as universal fallback — all invisible to the caller.
 
-## Problem
-
-If you want real-time AI media processing in the browser — virtual backgrounds, background noise removal, face filters — one of the few open-source options is [MediaPipe](https://github.com/google-ai-edge/mediapipe). MediaPipe ships some great open-source models, but they all run in WebAssembly, and using them effectively still requires a fair amount of hardware-accelerated pre- and post-processing on top.
-
-![MediaPipe pipeline diagram](https://github.com/user-attachments/assets/62932072-c5b2-445e-b5d8-a2bd5bb72920)
-
-In 2021, I built an SDK to implement [hardware-accelerated neural networks directly in WebGL](https://medium.com/vectorly/building-a-more-efficient-background-segmentation-model-than-google-74ecd17392d5) that proved much more efficient than the MediaPipe implementation. It was popular, but it was a commercial SDK — the company was acquired and the technology was never opened up.
 
-In 2022, Google Meet adopted a similar approach with their own [hardware-accelerated networks](https://research.google/blog/high-definition-segmentation-in-google-meet/), but that was never open-sourced either.
-
-As of late 2025, MediaPipe is still the state of the art for open-source real-time browser ML, and nothing better has come along.
-
-## Solution
-
-With WebGPU shipping and WebNN on the way, it is now possible to build efficient implementations of popular real-time media-processing features — background segmentation, audio filtering — by combining hardware-accelerated neural networks in WebGPU/WebNN with efficient pre- and post-processing, so developers can plug in features like virtual backgrounds and noise removal without worrying about the details.
-
-Longpipe is that SDK.
 
 ## How it works
 
@@ -174,12 +177,6 @@ Six trained presets cover the hardware range:
 
 Training scripts, fixture generation, and the weight export pipeline are not yet documented here — coming soon.
 
-## Acknowledgements
-
-- [EfficientNet-Lite](https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite) for the backbone architecture.
-- [Google Meet HD Segmentation research](https://research.google/blog/high-definition-segmentation-in-google-meet/) for the architectural choices that hold up on accelerators.
-- [WebGPU](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API) for making compute-heavy work in the browser realistic.
-- [Mediabunny](https://mediabunny.dev/) — a colleague's project and a model for what an open-source media SDK with sponsorship-backed development can look like.
 
 ## Roadmap
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "longpipe",
-  "version": "0.0.3",
+  "version": "0.0.4",
   "description": "Open-source WebGPU/WebGL2 video effects SDK — virtual backgrounds, background blur, portrait matting in real time in the browser.",
   "type": "module",
   "sideEffects": false,