chromapakz 0.2.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (69) hide show
  1. chromapakz-0.2.0/.cursor/rules/chromapakz.md +41 -0
  2. chromapakz-0.2.0/.github/workflows/ci.yml +82 -0
  3. chromapakz-0.2.0/.github/workflows/release.yml +52 -0
  4. chromapakz-0.2.0/.gitignore +27 -0
  5. chromapakz-0.2.0/CMakeLists.txt +27 -0
  6. chromapakz-0.2.0/LICENSE +21 -0
  7. chromapakz-0.2.0/PKG-INFO +236 -0
  8. chromapakz-0.2.0/README.md +224 -0
  9. chromapakz-0.2.0/demo/index.html +312 -0
  10. chromapakz-0.2.0/docs/API.md +110 -0
  11. chromapakz-0.2.0/docs/EVALUATION.md +261 -0
  12. chromapakz-0.2.0/docs/FORMAT.md +42 -0
  13. chromapakz-0.2.0/docs/RELEASING.md +47 -0
  14. chromapakz-0.2.0/docs/logo.png +0 -0
  15. chromapakz-0.2.0/docs/rate-distortion.svg +60 -0
  16. chromapakz-0.2.0/examples/tum_fr1desk.py +52 -0
  17. chromapakz-0.2.0/experiments/webcodecs-lossless/headless.html +249 -0
  18. chromapakz-0.2.0/experiments/webcodecs-lossless/index.html +160 -0
  19. chromapakz-0.2.0/experiments/webcodecs-lossless/png16-test.mjs +44 -0
  20. chromapakz-0.2.0/experiments/webcodecs-lossless/probe.js +211 -0
  21. chromapakz-0.2.0/experiments/webcodecs-lossless/run.mjs +172 -0
  22. chromapakz-0.2.0/experiments/webcodecs-lossless/smoke-demo.mjs +49 -0
  23. chromapakz-0.2.0/native/build.sh +11 -0
  24. chromapakz-0.2.0/native/chromapakz.cpp +595 -0
  25. chromapakz-0.2.0/native/chromapakz.h +43 -0
  26. chromapakz-0.2.0/native/dccli.cpp +117 -0
  27. chromapakz-0.2.0/native/wasm/build-wasm.sh +72 -0
  28. chromapakz-0.2.0/native/wasm/dc_vp9.cpp +196 -0
  29. chromapakz-0.2.0/native/wasm/dc_vp9.h +61 -0
  30. chromapakz-0.2.0/native/wasm/gen-decode-ref.mjs +32 -0
  31. chromapakz-0.2.0/package-lock.json +1192 -0
  32. chromapakz-0.2.0/package.json +49 -0
  33. chromapakz-0.2.0/pyproject.toml +40 -0
  34. chromapakz-0.2.0/python/benchmark_codecs.py +102 -0
  35. chromapakz-0.2.0/python/chromapakz/__init__.py +236 -0
  36. chromapakz-0.2.0/python/ingest.py +227 -0
  37. chromapakz-0.2.0/python/make_synthetic_rgbd.py +70 -0
  38. chromapakz-0.2.0/python/plot_rd.py +121 -0
  39. chromapakz-0.2.0/python/webm_inspect.py +73 -0
  40. chromapakz-0.2.0/scripts/install-libvpx.sh +47 -0
  41. chromapakz-0.2.0/src/backend/decode-ref.js +8 -0
  42. chromapakz-0.2.0/src/backend/probe.js +104 -0
  43. chromapakz-0.2.0/src/backend/select.js +20 -0
  44. chromapakz-0.2.0/src/backend/wasm/decode.js +65 -0
  45. chromapakz-0.2.0/src/backend/wasm/encode.js +55 -0
  46. chromapakz-0.2.0/src/backend/wasm/vp9-decode.js +2 -0
  47. chromapakz-0.2.0/src/backend/wasm/vp9-decode.wasm +0 -0
  48. chromapakz-0.2.0/src/backend/wasm/vp9-encode.js +2 -0
  49. chromapakz-0.2.0/src/backend/wasm/vp9-encode.wasm +0 -0
  50. chromapakz-0.2.0/src/backend/webcodecs.js +87 -0
  51. chromapakz-0.2.0/src/chromapakz-core.js +50 -0
  52. chromapakz-0.2.0/src/chromapakz.js +383 -0
  53. chromapakz-0.2.0/src/signals.js +164 -0
  54. chromapakz-0.2.0/src/webm.js +281 -0
  55. chromapakz-0.2.0/tests/browser/fallback.html +41 -0
  56. chromapakz-0.2.0/tests/browser/run.mjs +82 -0
  57. chromapakz-0.2.0/tests/cross_interop.py +29 -0
  58. chromapakz-0.2.0/tests/ffmpeg_interop.py +39 -0
  59. chromapakz-0.2.0/tests/fixtures/regen_stream.mjs +29 -0
  60. chromapakz-0.2.0/tests/fixtures/stream.webm +0 -0
  61. chromapakz-0.2.0/tests/fixtures/stream_depth.u16 +0 -0
  62. chromapakz-0.2.0/tests/js_metadata_v2.mjs +27 -0
  63. chromapakz-0.2.0/tests/js_quant.mjs +48 -0
  64. chromapakz-0.2.0/tests/js_signals.mjs +35 -0
  65. chromapakz-0.2.0/tests/js_wasm_roundtrip.mjs +52 -0
  66. chromapakz-0.2.0/tests/roundtrip.py +39 -0
  67. chromapakz-0.2.0/tests/stream_interop.py +29 -0
  68. chromapakz-0.2.0/tests/webm_stream.mjs +55 -0
  69. chromapakz-0.2.0/vite.config.js +32 -0
@@ -0,0 +1,41 @@
1
+ # ChromaPakZ
2
+
3
+ Lossless RGB + arbitrary **uint16 signals** in one WebM. Three implementations share v2 `signals[]` metadata only.
4
+
5
+ Format: [`docs/FORMAT.md`](../docs/FORMAT.md). API: [`docs/API.md`](../docs/API.md).
6
+
7
+ ## Signals
8
+
9
+ - Metadata **v2** requires `signals: [{ id, tracks, scheme, quant?, … }]`.
10
+ - Each lossless signal = two VP9 tracks (`signal-{id}-hi/lo`). RGB = track 1 when present.
11
+ - `inverse-depth` quant for float depth; `quant: null` for raw uint16 (object IDs, etc.).
12
+ - **No** top-level `depth` field, v1 metadata, or depth-only sugar APIs.
13
+
14
+ ### Browser
15
+
16
+ ```javascript
17
+ createEncoder({ W, H, signals: [{ id: 'depth', near, far }, { id: 'objectId' }] });
18
+ addFrame({ rgb, signals: { depth: { float }, objectId: { u16 } } });
19
+ ```
20
+
21
+ ### Python / C++
22
+
23
+ ```python
24
+ cz.encode({"depth": u16, "objectId": u16}, specs={"depth": cz.inverse_depth_spec(near, far)}, rgb=rgba)
25
+ cz.decode_signal(data, "depth")
26
+ ```
27
+
28
+ ## Tests
29
+
30
+ | Command | What |
31
+ |---|---|
32
+ | `node tests/js_metadata_v2.mjs` | Rejects v1 / empty metadata |
33
+ | `python tests/cross_interop.py` | v2-only metadata contract |
34
+ | `./build/dccli selftest` | C++ bit-exact |
35
+ | `node run.mjs multisignal` | Browser depth + objectId |
36
+
37
+ ## Conventions
38
+
39
+ - Keep `src/signals.js` and native metadata builders in sync.
40
+ - VP9 lossless luma: full range always.
41
+ - `createEncoder` requires `signals[]`; depth is just a signal id.
@@ -0,0 +1,82 @@
1
+ name: ci
2
+
3
+ on:
4
+ push:
5
+ branches: [main]
6
+ pull_request:
7
+
8
+ concurrency:
9
+ group: ci-${{ github.ref }}
10
+ cancel-in-progress: true
11
+
12
+ jobs:
13
+ native-python:
14
+ name: build + test (${{ matrix.os }})
15
+ strategy:
16
+ fail-fast: false
17
+ matrix:
18
+ os: [ubuntu-latest, macos-latest]
19
+ runs-on: ${{ matrix.os }}
20
+ steps:
21
+ - uses: actions/checkout@v4
22
+
23
+ - name: Install libvpx + tools (Linux)
24
+ if: runner.os == 'Linux'
25
+ run: sudo apt-get update && sudo apt-get install -y libvpx-dev pkg-config cmake ninja-build ffmpeg
26
+
27
+ - name: Install libvpx + tools (macOS)
28
+ if: runner.os == 'macOS'
29
+ run: brew install libvpx pkg-config cmake ninja ffmpeg
30
+
31
+ - uses: actions/setup-python@v5
32
+ with:
33
+ python-version: "3.12"
34
+
35
+ - name: CMake build + C++ self-test (bit-exact)
36
+ run: |
37
+ cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
38
+ cmake --build build -j
39
+ ./build/dccli selftest
40
+
41
+ - name: pip install (compiles via CMake) + Python round-trip
42
+ run: |
43
+ python -m pip install --upgrade pip
44
+ python -m pip install numpy
45
+ python -m pip install . --no-build-isolation || python -m pip install .
46
+ python tests/roundtrip.py
47
+ python tests/cross_interop.py
48
+ python tests/stream_interop.py
49
+
50
+ - name: ffmpeg decode-interop (full-range / bit-exact regression guard)
51
+ run: python tests/ffmpeg_interop.py
52
+
53
+ browser:
54
+ name: in-browser VP9 lossless (Chromium)
55
+ runs-on: ubuntu-latest
56
+ steps:
57
+ - uses: actions/checkout@v4
58
+ - name: Install libvpx
59
+ run: sudo apt-get update && sudo apt-get install -y libvpx-dev pkg-config cmake ninja-build
60
+ - uses: actions/setup-node@v4
61
+ with:
62
+ node-version: "20"
63
+ - name: Run the headless WebCodecs probe (asserts bit-exact encode->decode)
64
+ working-directory: experiments/webcodecs-lossless
65
+ run: |
66
+ npm init -y >/dev/null 2>&1
67
+ npm i -D playwright
68
+ npx playwright install --with-deps chromium
69
+ node ../../tests/js_quant.mjs
70
+ node ../../tests/js_signals.mjs
71
+ node ../../tests/js_metadata_v2.mjs
72
+ node ../../tests/webm_stream.mjs
73
+ BROWSER=chromium node run.mjs single 256 | tee out.txt
74
+ grep -q "EXACT" out.txt
75
+ BROWSER=chromium node run.mjs streaming 256 12 | tee stream.txt
76
+ grep -q "YES ✓" stream.txt
77
+ BROWSER=chromium node run.mjs network 256 8 | tee network.txt
78
+ grep -q "YES ✓" network.txt
79
+ BROWSER=chromium node run.mjs multisignal 128 6 | tee multi.txt
80
+ grep -q "YES ✓" multi.txt
81
+ node smoke-demo.mjs | tee demo.txt
82
+ grep -q "bit-exact" demo.txt
@@ -0,0 +1,52 @@
1
+ name: release
2
+
3
+ # Cut a GitHub Release (tag like v0.1.0) to build wheels + sdist and publish to PyPI
4
+ # via Trusted Publishing (OIDC — no API token stored). See docs/RELEASING.md for one-time setup.
5
+ on:
6
+ release:
7
+ types: [published]
8
+ workflow_dispatch: {} # manual runs build artifacts without publishing
9
+
10
+ jobs:
11
+ wheels:
12
+ name: wheels (${{ matrix.os }})
13
+ strategy:
14
+ fail-fast: false
15
+ matrix:
16
+ os: [ubuntu-latest, macos-latest]
17
+ runs-on: ${{ matrix.os }}
18
+ steps:
19
+ - uses: actions/checkout@v4
20
+ - name: Build wheels
21
+ uses: pypa/cibuildwheel@v2.21
22
+ # libvpx provisioning + test command come from [tool.cibuildwheel] in pyproject.toml
23
+ - uses: actions/upload-artifact@v4
24
+ with:
25
+ name: wheels-${{ matrix.os }}
26
+ path: wheelhouse/*.whl
27
+
28
+ sdist:
29
+ runs-on: ubuntu-latest
30
+ steps:
31
+ - uses: actions/checkout@v4
32
+ - run: pipx run build --sdist
33
+ - uses: actions/upload-artifact@v4
34
+ with:
35
+ name: sdist
36
+ path: dist/*.tar.gz
37
+
38
+ publish:
39
+ name: publish to PyPI
40
+ needs: [wheels, sdist]
41
+ if: github.event_name == 'release'
42
+ runs-on: ubuntu-latest
43
+ environment: pypi # configure a GitHub Environment named "pypi"
44
+ permissions:
45
+ id-token: write # required for Trusted Publishing (OIDC)
46
+ steps:
47
+ - uses: actions/download-artifact@v4
48
+ with:
49
+ path: dist
50
+ merge-multiple: true
51
+ - uses: pypa/gh-action-pypi-publish@release/v1
52
+ # No password/token: PyPI Trusted Publishing authenticates via the OIDC token above.
@@ -0,0 +1,27 @@
1
+ # build artifacts
2
+ node_modules/
3
+ __pycache__/
4
+ build/
5
+ dist/
6
+ wheelhouse/
7
+ *.egg-info/
8
+ native/libchromapakz.*
9
+ native/_core.*
10
+ native/dccli
11
+ native/*.o
12
+ python/chromapakz/_core.*
13
+ python/chromapakz/*.dylib
14
+ python/chromapakz/*.so
15
+
16
+ # test / sample data
17
+ *.webm
18
+ *.u16
19
+ *.npz
20
+ *.rgba
21
+ _png16.png
22
+ demo/preview.png
23
+ experiments/**/package*.json
24
+
25
+ # …but keep the committed regression fixtures (browser-streamed interop golden file)
26
+ !tests/fixtures/stream.webm
27
+ !tests/fixtures/stream_depth.u16
@@ -0,0 +1,27 @@
1
+ cmake_minimum_required(VERSION 3.15)
2
+ project(chromapakz LANGUAGES CXX)
3
+
4
+ set(CMAKE_CXX_STANDARD 17)
5
+ set(CMAKE_CXX_STANDARD_REQUIRED ON)
6
+ if(NOT CMAKE_BUILD_TYPE)
7
+ set(CMAKE_BUILD_TYPE Release)
8
+ endif()
9
+
10
+ # Royalty-free VP9 via libvpx (BSD). pkg-config provides include/lib flags.
11
+ find_package(PkgConfig REQUIRED)
12
+ pkg_check_modules(VPX REQUIRED IMPORTED_TARGET vpx)
13
+
14
+ # Native core shared library. Output named "_core" (no lib prefix) so the Python
15
+ # package's ctypes loader finds chromapakz/_core.{so,dylib}.
16
+ add_library(_core SHARED native/chromapakz.cpp)
17
+ set_target_properties(_core PROPERTIES PREFIX "" OUTPUT_NAME "_core")
18
+ target_include_directories(_core PRIVATE native)
19
+ target_link_libraries(_core PRIVATE PkgConfig::VPX)
20
+
21
+ # Command-line tool (dev/testing; not shipped in the Python wheel).
22
+ add_executable(dccli native/dccli.cpp native/chromapakz.cpp)
23
+ target_include_directories(dccli PRIVATE native)
24
+ target_link_libraries(dccli PRIVATE PkgConfig::VPX)
25
+
26
+ # scikit-build-core installs the core lib into the wheel's package directory.
27
+ install(TARGETS _core LIBRARY DESTINATION chromapakz RUNTIME DESTINATION chromapakz)
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Kevin Blackburn-Matzen
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,236 @@
1
+ Metadata-Version: 2.1
2
+ Name: chromapakz
3
+ Version: 0.2.0
4
+ Summary: Lossless RGB + bit-exact auxiliary signals (depth, object IDs, …) in one WebM (VP9)
5
+ Keywords: depth,rgbd,video,codec,vp9,lossless,webm,signals,object-id
6
+ Author: kmatzen
7
+ License: MIT
8
+ Project-URL: Homepage, https://github.com/kmatzen/ChromaPakZ
9
+ Requires-Python: >=3.9
10
+ Requires-Dist: numpy>=1.20
11
+ Description-Content-Type: text/markdown
12
+
13
+ # ChromaPakZ
14
+
15
+ <p align="center"><img src="docs/logo.png" alt="ChromaPakZ — lossless RGBD video encoder" width="680"></p>
16
+
17
+ **A lossless RGBD video codec** (クロマパックZ): one ordinary `.webm` that carries an 8-bit **RGB** track
18
+ alongside **bit-exact 16-bit auxiliary signals** — depth, object IDs, packed normals, or any other `W×H`
19
+ `uint16` plane — in sync. It is built so that
20
+
21
+ - a **legacy player shows plain RGB** — the depth rides in extra tracks a normal player ignores;
22
+ - it uses only **royalty-free** codecs (VP9 / libvpx, BSD) — no GPL encoder, no patent pool;
23
+ - it runs **in the browser via WebCodecs** — **no WASM on Chromium**, with a small libvpx-WASM fallback for engines whose native path isn't bit-exact; and
24
+ - depth is packed with one **reversible map**, not the range-slice bookkeeping of older schemes;
25
+ - **multiple lossless uint16 signals** (depth, object IDs, …) share one container in sync.
26
+
27
+ It's a clean-room redo of an older MP4/x264 approach (RGB as YUV, plus 16-bit depth sliced into several
28
+ lossless-10-bit ranges). That design worked but had three thorns: x264 is GPL, the range-slicing was
29
+ fiddly, and the browser always needed a WASM codec. ChromaPakZ removes the first two outright; for the
30
+ third, Chromium runs end-to-end on WebCodecs with **no WASM**, and a small libvpx-WASM build is kept only
31
+ as a per-operation fallback for engines whose native path isn't bit-exact (Firefox, Safari).
32
+
33
+ The same format is implemented three times — **browser (WebCodecs)**, **C++ (libvpx)**, and **Python** —
34
+ and a file written by any one decodes bit-exactly in the others.
35
+
36
+ ## Quickstart
37
+
38
+ ```sh
39
+ # Browser demo — encode→file→decode→view, entirely in-page (no WASM on Chromium)
40
+ python3 -m http.server 8000 # from the repo root, then open http://localhost:8000/demo/
41
+
42
+ # Python / C++ — the native libvpx core is compiled from source, so install the build
43
+ # prerequisites first: libvpx (dev headers), pkg-config, CMake, and a C++17 compiler.
44
+ # macOS: brew install libvpx pkg-config cmake ninja
45
+ # Debian: sudo apt-get install libvpx-dev pkg-config cmake ninja-build g++
46
+ pip install . # pip compiles the native core via CMake and bundles it
47
+ python -c "import chromapakz as cz; print(cz.inverse_depth_spec(0.3, 9.0))"
48
+ # cz.encode({"depth": u16}, specs={"depth": cz.inverse_depth_spec(near, far)}, rgb=rgba)
49
+
50
+ # Browser JS API — streaming encode/decode (see docs/API.md)
51
+ # createEncoder({ signals: [{ id:'depth', near, far }, { id:'objectId' }] })
52
+ # createDecoder(bytes).readFrame() -> { rgb, signals: { depth: { u16 }, objectId: { u16 } } }
53
+
54
+ # C++ / CLI
55
+ cmake -S . -B build && cmake --build build -j # or: native/build.sh
56
+ ./build/dccli selftest
57
+ ./build/dccli decodesignal clip.webm depth depth.u16
58
+ ```
59
+
60
+ ## How it works
61
+
62
+ | Layer | Choice |
63
+ |---|---|
64
+ | **Container** | WebM / Matroska, multi-track. RGB is track 1, so any player shows it; depth tracks are ignored by players that don't know them. A Duration, a Cues index, and ~1 s RGB keyframes make it **seekable** in `<video>` (depth stays single-keyframe — it isn't what `<video>` plays). |
65
+ | **RGB track** | 8-bit VP9, YUV 4:2:0, BT.709 full-range — a normal, viewable video stream. |
66
+ | **Lossless signals** | Each signal: optional quant (e.g. inverse-depth for float depth) → **uint16** → **triangle-fold 8+8** → two **VP9 lossless** tracks. Add object IDs, labels, etc. as additional signal pairs. |
67
+ | **Metadata** | v2 `signals[]` only — each signal (`id`, tracks, scheme, quant). |
68
+
69
+ **Inverse-depth quantization** spends precision where it matters (near surfaces), matching how stereo/ToF
70
+ sensors behave. Float can't be stored losslessly in 16 bits, so this quantization *is* the format's defined
71
+ precision boundary; everything below it is bit-exact.
72
+
73
+ **Triangle-fold** is the key trick. The naive low byte `d & 0xFF` is a sawtooth — a hard `255→0` cliff
74
+ every 256 levels — and those manufactured edges wreck any spatial predictor (this is exactly why the old
75
+ design needed range slices). Reflecting every other segment (`lo = (high&1) ? 255-lo : lo`) turns it into a
76
+ continuous triangle wave with no cliffs, so VP9's own predictor works. It's range-slicing collapsed into one
77
+ reversible map, with nothing to manage.
78
+
79
+ **Full color range is signaled** in the bitstream (`VP9E_SET_COLOR_RANGE`), so a range-honouring decoder
80
+ returns the packed luma unscaled instead of applying a limited-range conversion that would corrupt depth.
81
+
82
+ ### Why these choices (measured, not assumed — Chromium 148)
83
+
84
+ WebCodecs has no "lossless" switch, so every claim here is a measurement from
85
+ `experiments/webcodecs-lossless`:
86
+
87
+ - **VP9 at QP 0 is bit-exact through WebCodecs; AV1 is not** (AV1 `quantizer:0` drifts by up to ~257). So
88
+ VP9 carries depth; AV1 is fine only for the lossy RGB track.
89
+ - **Triangle-fold beats a naive byte-split by ~13%**, and **inter-coding cuts another ~52%** (and stays
90
+ bit-exact across the GOP) — most of what looks like incompressible LSB noise is actually static fold
91
+ structure that temporal prediction removes.
92
+ - **8+8 beats high-bit-depth.** 10-bit VP9 encode *is* available in browsers, but a 10+6 split is ~4%
93
+ *worse* than 8+8 and narrows browser reach, so 8+8 wins on both counts.
94
+
95
+ [`docs/EVALUATION.md`](docs/EVALUATION.md) is the full due-diligence record: every codec/container/packing
96
+ alternative considered, the constraint that eliminates each, a head-to-head benchmark (ChromaPakZ beats
97
+ FFV1, PNG-16 and x264 on the same 16-bit depth, beats x265/HEVC at matched 11-bit precision, and lands
98
+ within 1–2% of LZMA), cited licensing/browser
99
+ facts, and a sensitivity analysis of when a different choice would win.
100
+
101
+ ## What it costs
102
+
103
+ Lossless 16-bit depth of a real sensor is **noise-bound**: the low bits are largely sensor noise, and
104
+ lossless coding must preserve every bit of it. On **real Kinect data** (TUM RGB-D `fr1/desk`, 30 frames at
105
+ 640×480, 78% valid):
106
+
107
+ | track | bits / pixel |
108
+ |---|---|
109
+ | RGB | 0.19 |
110
+ | depth (hi + lo) | 0.50 + 4.35 |
111
+ | **total** | **5.04** |
112
+
113
+ — depth round-tripped **bit-exact**. Reproduce with `examples/tum_fr1desk.py` (see its header for the
114
+ one-line dataset fetch).
115
+
116
+ The one knob that moves this is the **quantization precision** vs the sensor's noise floor. Spreading depth
117
+ over all 65,535 codes makes one step far finer than the noise, so the codec faithfully archives randomness.
118
+ Coarsening the grid to match the noise collapses the cost — without losing real signal. The sweep below is
119
+ measured on the synthetic benchmark clip (`make_synthetic_rgbd.py`, range ≈0.9–7.8 m) — a separate clip from
120
+ the TUM numbers above:
121
+
122
+ | effective bits | depth precision at 7.8 m | depth bpp |
123
+ |---|---|---|
124
+ | 16 (default) | 0.9 mm per step | 13.2 |
125
+ | 12 | 14 mm per step | 9.7 |
126
+ | 11 | 28 mm per step | 8.1 |
127
+ | 10 | 56 mm per step | 6.9 |
128
+
129
+ (Reproduce the bpp column with `python python/benchmark_codecs.py`.) `levels` is a first-class,
130
+ metadata-stored parameter (default 65536 = full 16-bit) shared by all three implementations, so
131
+ reduced-precision files reconstruct identically everywhere. Set it with `ingest.py --depth-bits N` or the
132
+ `levels=` argument.
133
+
134
+ ### Codec rate-distortion
135
+
136
+ This is a separate axis from precision: how faithfully the *codec* carries whatever quantized depth you
137
+ give it. PSNR here is the encode→decode path measured against the source codes.
138
+
139
+ ![ChromaPakZ codec rate-distortion](docs/rate-distortion.svg)
140
+
141
+ The lossless codecs all sit on the **∞-dB band** — they reproduce depth exactly and differ only in size,
142
+ where ChromaPakZ (VP9) is smallest, just under FFV1, with PNG-16 well behind. The blue curve is ChromaPakZ's
143
+ own near-lossless option (sweeping the VP9 quantizer trades fidelity for size), but the default operating
144
+ point is **QP 0, bit-exact**. Regenerate with `python python/plot_rd.py`.
145
+
146
+ > **A note on ffmpeg.** *Decoding* ChromaPakZ files with ffmpeg (or any conformant VP9 decoder) is
147
+ > bit-exact. But *encode* with ChromaPakZ, not the ffmpeg CLI: `ffmpeg -c:v libvpx-vp9 -lossless 1` is
148
+ > lossless yet **~3× larger** (≈39 vs ≈13 bpp) — same library, far worse coding decisions, and no flag
149
+ > tested closes the gap. `python/plot_rd.py` therefore uses the real WebCodecs encoder for the VP9 numbers.
150
+
151
+ ## Cross-language implementations
152
+
153
+ All three read and write the identical `.webm`, verified bit-exact in every direction (browser ⇄ C++ ⇄
154
+ Python), and produce standard files — `ffprobe` reports `matroska,webm` with one RGB stream plus two VP9
155
+ streams per lossless signal, and ffmpeg decodes track 0 as plain RGB when present.
156
+
157
+ Format schema: [`docs/FORMAT.md`](docs/FORMAT.md). API: [`docs/API.md`](docs/API.md).
158
+
159
+ | Surface | Codec | Build |
160
+ |---|---|---|
161
+ | **Browser** | WebCodecs VP9 | none — `src/chromapakz.js`, `src/signals.js`, `src/webm.js`. Multi-signal streaming API. |
162
+ | **C++** | libvpx VP9 | CMake → `build/_core` + `dccli` (`dc_encode_multi`, `dc_decode_signal`) |
163
+ | **Python** | ctypes → C++ | `pip install .` — `encode()`, `decode()`, `parse_metadata()` |
164
+
165
+ ```sh
166
+ ./build/dccli encodergbd rgb.rgba depth.u16 W H N fps near far kbps out.webm
167
+ ./build/dccli decodesignal clip.webm objectId ids.u16
168
+ ./build/dccli decodergb clip.webm rgb.rgba
169
+ ```
170
+
171
+ ## Real-data ingestion (`python/`)
172
+
173
+ - **`ingest.py`** — load depth (`.exr` / `.npy` / `.npz` / 16-bit PNG·TIFF / raw) and optional RGB (image
174
+ sequence, video via ffmpeg, or array), auto-derive inverse-depth `near`/`far` from percentiles, encode,
175
+ and report real per-track bpp. Invalid pixels (`<=0`/NaN) map to code 0.
176
+ `python ingest.py --depth 'd_*.exr' --rgb 'rgb_*.png' -o clip.webm --report --verify`
177
+ - **`make_synthetic_rgbd.py`** — a realistic RGBD generator (smooth surfaces, depth edges, disparity-domain
178
+ noise, occlusion shadows, dropout holes) for when you don't have a sensor handy.
179
+ - **`webm_inspect.py`** — pure-Python EBML parser for the per-track byte breakdown.
180
+
181
+ ## How it relates to RealSense / Kinect
182
+
183
+ Depth-camera ecosystems already split into two camps; ChromaPakZ takes the best of both.
184
+
185
+ - **Intel RealSense** colorizes 16-bit depth into an RGB image (Hue, ~10.5 effective bits) and encodes that
186
+ with a stock H.264/H.265 codec. Great for streaming and reuse of hardware codecs, but **lossy** — unfit
187
+ for ground-truth or archival depth.
188
+ - **Kinect / RGBD datasets** store depth raw or as 16-bit PNG. Azure Kinect even records to **Matroska**
189
+ with a 16-bit depth track (lossless via per-frame PNG); TUM RGB-D, NYU and ScanNet use 16-bit PNG
190
+ sequences. Bit-exact, but **intra-only and large** — no temporal compression.
191
+
192
+ | | RealSense colorize | Kinect / PNG | **ChromaPakZ** |
193
+ |---|---|---|---|
194
+ | bit-exact 16-bit depth | ✗ (lossy) | ✓ | **✓** |
195
+ | RGB plays in any legacy player | ✓ | — | **✓** |
196
+ | inter-frame (temporal) compression | ✓ (lossy) | ✗ | **✓ (lossless)** |
197
+ | royalty-free, browser-native (no WASM on Chromium) | — | — | **✓** |
198
+
199
+ That Azure Kinect already chose Matroska — WebM's basis — is telling. ChromaPakZ differs by *compressing*
200
+ depth losslessly (VP9 + triangle-fold, inter-coded) rather than storing raw or intra PNG, and by running in
201
+ the browser. Sources:
202
+ [RealSense colorized depth](https://dev.intelrealsense.com/docs/depth-image-compression-by-colorization-for-intel-realsense-depth-cameras),
203
+ [Azure Kinect record format](https://learn.microsoft.com/en-us/azure/kinect-dk/record-file-format).
204
+
205
+ ## Repository layout
206
+
207
+ ```
208
+ src/ chromapakz.js, signals.js, webm.js, chromapakz-core.js
209
+ native/ chromapakz.{h,cpp}, dccli.cpp
210
+ python/ chromapakz/ (pip package), ingest.py, make_synthetic_rgbd.py
211
+ demo/ index.html in-browser encode→decode→view
212
+ examples/ tum_fr1desk.py
213
+ experiments/ webcodecs-lossless/ run.mjs, smoke-demo.mjs, headless tests
214
+ docs/ FORMAT.md, API.md, EVALUATION.md, RELEASING.md
215
+ tests/ roundtrip.py, cross_interop.py, stream_interop.py, ffmpeg_interop.py, js_*.mjs
216
+ ```
217
+
218
+ CI builds and tests on Linux + macOS and runs the in-browser VP9-lossless probe in headless Chromium;
219
+ `docs/RELEASING.md` covers wheels and PyPI publishing. The full design rationale and benchmarks are in
220
+ [`docs/EVALUATION.md`](docs/EVALUATION.md).
221
+
222
+ ## Status & limitations
223
+
224
+ Working end-to-end and verified across all three implementations. Honest caveats:
225
+
226
+ - **Browser support is engine-specific** (measured, [`EVALUATION.md` §11](docs/EVALUATION.md)): native
227
+ WebCodecs lossless *encode* is Chromium-only today (WebKit lacks WebCodecs' quantizer mode; Firefox's
228
+ QP 0 isn't lossless); native lossless *decode* works on Chromium and WebKit/Safari, while Firefox
229
+ decodes VP9 to color-converted BGRX. Where native can't be trusted, the library transparently falls
230
+ back to a bundled **libvpx-WASM** codec, chosen *per operation* by a cached runtime probe — so a
231
+ decode-only browser (e.g. Safari) downloads only `vp9-decode.wasm` and never the larger encoder, and
232
+ vice-versa. Force it with `backend: 'webcodecs' | 'wasm'` (default `'auto'`); see [`docs/API.md`](docs/API.md).
233
+ These are Playwright engine builds — reconfirm on shipping browsers before hard claims.
234
+ - **"Royalty-free"** reflects the AOMedia/Google position on VP9; Sisvel operates pools that dispute it.
235
+ - An **auto precision picker** (estimate the sensor noise floor to choose `--depth-bits`) is future work.
236
+ - **Network byte streaming** is supported via `onChunk` on encode and `createDecoder()` + `push()`/`finish()` on decode. See [`docs/API.md`](docs/API.md).