chromapakz 0.2.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- chromapakz-0.2.0/.cursor/rules/chromapakz.md +41 -0
- chromapakz-0.2.0/.github/workflows/ci.yml +82 -0
- chromapakz-0.2.0/.github/workflows/release.yml +52 -0
- chromapakz-0.2.0/.gitignore +27 -0
- chromapakz-0.2.0/CMakeLists.txt +27 -0
- chromapakz-0.2.0/LICENSE +21 -0
- chromapakz-0.2.0/PKG-INFO +236 -0
- chromapakz-0.2.0/README.md +224 -0
- chromapakz-0.2.0/demo/index.html +312 -0
- chromapakz-0.2.0/docs/API.md +110 -0
- chromapakz-0.2.0/docs/EVALUATION.md +261 -0
- chromapakz-0.2.0/docs/FORMAT.md +42 -0
- chromapakz-0.2.0/docs/RELEASING.md +47 -0
- chromapakz-0.2.0/docs/logo.png +0 -0
- chromapakz-0.2.0/docs/rate-distortion.svg +60 -0
- chromapakz-0.2.0/examples/tum_fr1desk.py +52 -0
- chromapakz-0.2.0/experiments/webcodecs-lossless/headless.html +249 -0
- chromapakz-0.2.0/experiments/webcodecs-lossless/index.html +160 -0
- chromapakz-0.2.0/experiments/webcodecs-lossless/png16-test.mjs +44 -0
- chromapakz-0.2.0/experiments/webcodecs-lossless/probe.js +211 -0
- chromapakz-0.2.0/experiments/webcodecs-lossless/run.mjs +172 -0
- chromapakz-0.2.0/experiments/webcodecs-lossless/smoke-demo.mjs +49 -0
- chromapakz-0.2.0/native/build.sh +11 -0
- chromapakz-0.2.0/native/chromapakz.cpp +595 -0
- chromapakz-0.2.0/native/chromapakz.h +43 -0
- chromapakz-0.2.0/native/dccli.cpp +117 -0
- chromapakz-0.2.0/native/wasm/build-wasm.sh +72 -0
- chromapakz-0.2.0/native/wasm/dc_vp9.cpp +196 -0
- chromapakz-0.2.0/native/wasm/dc_vp9.h +61 -0
- chromapakz-0.2.0/native/wasm/gen-decode-ref.mjs +32 -0
- chromapakz-0.2.0/package-lock.json +1192 -0
- chromapakz-0.2.0/package.json +49 -0
- chromapakz-0.2.0/pyproject.toml +40 -0
- chromapakz-0.2.0/python/benchmark_codecs.py +102 -0
- chromapakz-0.2.0/python/chromapakz/__init__.py +236 -0
- chromapakz-0.2.0/python/ingest.py +227 -0
- chromapakz-0.2.0/python/make_synthetic_rgbd.py +70 -0
- chromapakz-0.2.0/python/plot_rd.py +121 -0
- chromapakz-0.2.0/python/webm_inspect.py +73 -0
- chromapakz-0.2.0/scripts/install-libvpx.sh +47 -0
- chromapakz-0.2.0/src/backend/decode-ref.js +8 -0
- chromapakz-0.2.0/src/backend/probe.js +104 -0
- chromapakz-0.2.0/src/backend/select.js +20 -0
- chromapakz-0.2.0/src/backend/wasm/decode.js +65 -0
- chromapakz-0.2.0/src/backend/wasm/encode.js +55 -0
- chromapakz-0.2.0/src/backend/wasm/vp9-decode.js +2 -0
- chromapakz-0.2.0/src/backend/wasm/vp9-decode.wasm +0 -0
- chromapakz-0.2.0/src/backend/wasm/vp9-encode.js +2 -0
- chromapakz-0.2.0/src/backend/wasm/vp9-encode.wasm +0 -0
- chromapakz-0.2.0/src/backend/webcodecs.js +87 -0
- chromapakz-0.2.0/src/chromapakz-core.js +50 -0
- chromapakz-0.2.0/src/chromapakz.js +383 -0
- chromapakz-0.2.0/src/signals.js +164 -0
- chromapakz-0.2.0/src/webm.js +281 -0
- chromapakz-0.2.0/tests/browser/fallback.html +41 -0
- chromapakz-0.2.0/tests/browser/run.mjs +82 -0
- chromapakz-0.2.0/tests/cross_interop.py +29 -0
- chromapakz-0.2.0/tests/ffmpeg_interop.py +39 -0
- chromapakz-0.2.0/tests/fixtures/regen_stream.mjs +29 -0
- chromapakz-0.2.0/tests/fixtures/stream.webm +0 -0
- chromapakz-0.2.0/tests/fixtures/stream_depth.u16 +0 -0
- chromapakz-0.2.0/tests/js_metadata_v2.mjs +27 -0
- chromapakz-0.2.0/tests/js_quant.mjs +48 -0
- chromapakz-0.2.0/tests/js_signals.mjs +35 -0
- chromapakz-0.2.0/tests/js_wasm_roundtrip.mjs +52 -0
- chromapakz-0.2.0/tests/roundtrip.py +39 -0
- chromapakz-0.2.0/tests/stream_interop.py +29 -0
- chromapakz-0.2.0/tests/webm_stream.mjs +55 -0
- chromapakz-0.2.0/vite.config.js +32 -0
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# ChromaPakZ
|
|
2
|
+
|
|
3
|
+
Lossless RGB + arbitrary **uint16 signals** in one WebM. Three implementations share v2 `signals[]` metadata only.
|
|
4
|
+
|
|
5
|
+
Format: [`docs/FORMAT.md`](../docs/FORMAT.md). API: [`docs/API.md`](../docs/API.md).
|
|
6
|
+
|
|
7
|
+
## Signals
|
|
8
|
+
|
|
9
|
+
- Metadata **v2** requires `signals: [{ id, tracks, scheme, quant?, … }]`.
|
|
10
|
+
- Each lossless signal = two VP9 tracks (`signal-{id}-hi/lo`). RGB = track 1 when present.
|
|
11
|
+
- `inverse-depth` quant for float depth; `quant: null` for raw uint16 (object IDs, etc.).
|
|
12
|
+
- **No** top-level `depth` field, v1 metadata, or depth-only sugar APIs.
|
|
13
|
+
|
|
14
|
+
### Browser
|
|
15
|
+
|
|
16
|
+
```javascript
|
|
17
|
+
createEncoder({ W, H, signals: [{ id: 'depth', near, far }, { id: 'objectId' }] });
|
|
18
|
+
addFrame({ rgb, signals: { depth: { float }, objectId: { u16 } } });
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
### Python / C++
|
|
22
|
+
|
|
23
|
+
```python
|
|
24
|
+
cz.encode({"depth": u16, "objectId": u16}, specs={"depth": cz.inverse_depth_spec(near, far)}, rgb=rgba)
|
|
25
|
+
cz.decode_signal(data, "depth")
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Tests
|
|
29
|
+
|
|
30
|
+
| Command | What |
|
|
31
|
+
|---|---|
|
|
32
|
+
| `node tests/js_metadata_v2.mjs` | Rejects v1 / empty metadata |
|
|
33
|
+
| `python tests/cross_interop.py` | v2-only metadata contract |
|
|
34
|
+
| `./build/dccli selftest` | C++ bit-exact |
|
|
35
|
+
| `node run.mjs multisignal` | Browser depth + objectId |
|
|
36
|
+
|
|
37
|
+
## Conventions
|
|
38
|
+
|
|
39
|
+
- Keep `src/signals.js` and native metadata builders in sync.
|
|
40
|
+
- VP9 lossless luma: full range always.
|
|
41
|
+
- `createEncoder` requires `signals[]`; depth is just a signal id.
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
name: ci
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
|
|
8
|
+
concurrency:
|
|
9
|
+
group: ci-${{ github.ref }}
|
|
10
|
+
cancel-in-progress: true
|
|
11
|
+
|
|
12
|
+
jobs:
|
|
13
|
+
native-python:
|
|
14
|
+
name: build + test (${{ matrix.os }})
|
|
15
|
+
strategy:
|
|
16
|
+
fail-fast: false
|
|
17
|
+
matrix:
|
|
18
|
+
os: [ubuntu-latest, macos-latest]
|
|
19
|
+
runs-on: ${{ matrix.os }}
|
|
20
|
+
steps:
|
|
21
|
+
- uses: actions/checkout@v4
|
|
22
|
+
|
|
23
|
+
- name: Install libvpx + tools (Linux)
|
|
24
|
+
if: runner.os == 'Linux'
|
|
25
|
+
run: sudo apt-get update && sudo apt-get install -y libvpx-dev pkg-config cmake ninja-build ffmpeg
|
|
26
|
+
|
|
27
|
+
- name: Install libvpx + tools (macOS)
|
|
28
|
+
if: runner.os == 'macOS'
|
|
29
|
+
run: brew install libvpx pkg-config cmake ninja ffmpeg
|
|
30
|
+
|
|
31
|
+
- uses: actions/setup-python@v5
|
|
32
|
+
with:
|
|
33
|
+
python-version: "3.12"
|
|
34
|
+
|
|
35
|
+
- name: CMake build + C++ self-test (bit-exact)
|
|
36
|
+
run: |
|
|
37
|
+
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
|
|
38
|
+
cmake --build build -j
|
|
39
|
+
./build/dccli selftest
|
|
40
|
+
|
|
41
|
+
- name: pip install (compiles via CMake) + Python round-trip
|
|
42
|
+
run: |
|
|
43
|
+
python -m pip install --upgrade pip
|
|
44
|
+
python -m pip install numpy
|
|
45
|
+
python -m pip install . --no-build-isolation || python -m pip install .
|
|
46
|
+
python tests/roundtrip.py
|
|
47
|
+
python tests/cross_interop.py
|
|
48
|
+
python tests/stream_interop.py
|
|
49
|
+
|
|
50
|
+
- name: ffmpeg decode-interop (full-range / bit-exact regression guard)
|
|
51
|
+
run: python tests/ffmpeg_interop.py
|
|
52
|
+
|
|
53
|
+
browser:
|
|
54
|
+
name: in-browser VP9 lossless (Chromium)
|
|
55
|
+
runs-on: ubuntu-latest
|
|
56
|
+
steps:
|
|
57
|
+
- uses: actions/checkout@v4
|
|
58
|
+
- name: Install libvpx
|
|
59
|
+
run: sudo apt-get update && sudo apt-get install -y libvpx-dev pkg-config cmake ninja-build
|
|
60
|
+
- uses: actions/setup-node@v4
|
|
61
|
+
with:
|
|
62
|
+
node-version: "20"
|
|
63
|
+
- name: Run the headless WebCodecs probe (asserts bit-exact encode->decode)
|
|
64
|
+
working-directory: experiments/webcodecs-lossless
|
|
65
|
+
run: |
|
|
66
|
+
npm init -y >/dev/null 2>&1
|
|
67
|
+
npm i -D playwright
|
|
68
|
+
npx playwright install --with-deps chromium
|
|
69
|
+
node ../../tests/js_quant.mjs
|
|
70
|
+
node ../../tests/js_signals.mjs
|
|
71
|
+
node ../../tests/js_metadata_v2.mjs
|
|
72
|
+
node ../../tests/webm_stream.mjs
|
|
73
|
+
BROWSER=chromium node run.mjs single 256 | tee out.txt
|
|
74
|
+
grep -q "EXACT" out.txt
|
|
75
|
+
BROWSER=chromium node run.mjs streaming 256 12 | tee stream.txt
|
|
76
|
+
grep -q "YES ✓" stream.txt
|
|
77
|
+
BROWSER=chromium node run.mjs network 256 8 | tee network.txt
|
|
78
|
+
grep -q "YES ✓" network.txt
|
|
79
|
+
BROWSER=chromium node run.mjs multisignal 128 6 | tee multi.txt
|
|
80
|
+
grep -q "YES ✓" multi.txt
|
|
81
|
+
node smoke-demo.mjs | tee demo.txt
|
|
82
|
+
grep -q "bit-exact" demo.txt
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
name: release
|
|
2
|
+
|
|
3
|
+
# Cut a GitHub Release (tag like v0.1.0) to build wheels + sdist and publish to PyPI
|
|
4
|
+
# via Trusted Publishing (OIDC — no API token stored). See docs/RELEASING.md for one-time setup.
|
|
5
|
+
on:
|
|
6
|
+
release:
|
|
7
|
+
types: [published]
|
|
8
|
+
workflow_dispatch: {} # manual runs build artifacts without publishing
|
|
9
|
+
|
|
10
|
+
jobs:
|
|
11
|
+
wheels:
|
|
12
|
+
name: wheels (${{ matrix.os }})
|
|
13
|
+
strategy:
|
|
14
|
+
fail-fast: false
|
|
15
|
+
matrix:
|
|
16
|
+
os: [ubuntu-latest, macos-latest]
|
|
17
|
+
runs-on: ${{ matrix.os }}
|
|
18
|
+
steps:
|
|
19
|
+
- uses: actions/checkout@v4
|
|
20
|
+
- name: Build wheels
|
|
21
|
+
uses: pypa/cibuildwheel@v2.21
|
|
22
|
+
# libvpx provisioning + test command come from [tool.cibuildwheel] in pyproject.toml
|
|
23
|
+
- uses: actions/upload-artifact@v4
|
|
24
|
+
with:
|
|
25
|
+
name: wheels-${{ matrix.os }}
|
|
26
|
+
path: wheelhouse/*.whl
|
|
27
|
+
|
|
28
|
+
sdist:
|
|
29
|
+
runs-on: ubuntu-latest
|
|
30
|
+
steps:
|
|
31
|
+
- uses: actions/checkout@v4
|
|
32
|
+
- run: pipx run build --sdist
|
|
33
|
+
- uses: actions/upload-artifact@v4
|
|
34
|
+
with:
|
|
35
|
+
name: sdist
|
|
36
|
+
path: dist/*.tar.gz
|
|
37
|
+
|
|
38
|
+
publish:
|
|
39
|
+
name: publish to PyPI
|
|
40
|
+
needs: [wheels, sdist]
|
|
41
|
+
if: github.event_name == 'release'
|
|
42
|
+
runs-on: ubuntu-latest
|
|
43
|
+
environment: pypi # configure a GitHub Environment named "pypi"
|
|
44
|
+
permissions:
|
|
45
|
+
id-token: write # required for Trusted Publishing (OIDC)
|
|
46
|
+
steps:
|
|
47
|
+
- uses: actions/download-artifact@v4
|
|
48
|
+
with:
|
|
49
|
+
path: dist
|
|
50
|
+
merge-multiple: true
|
|
51
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
52
|
+
# No password/token: PyPI Trusted Publishing authenticates via the OIDC token above.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# build artifacts
|
|
2
|
+
node_modules/
|
|
3
|
+
__pycache__/
|
|
4
|
+
build/
|
|
5
|
+
dist/
|
|
6
|
+
wheelhouse/
|
|
7
|
+
*.egg-info/
|
|
8
|
+
native/libchromapakz.*
|
|
9
|
+
native/_core.*
|
|
10
|
+
native/dccli
|
|
11
|
+
native/*.o
|
|
12
|
+
python/chromapakz/_core.*
|
|
13
|
+
python/chromapakz/*.dylib
|
|
14
|
+
python/chromapakz/*.so
|
|
15
|
+
|
|
16
|
+
# test / sample data
|
|
17
|
+
*.webm
|
|
18
|
+
*.u16
|
|
19
|
+
*.npz
|
|
20
|
+
*.rgba
|
|
21
|
+
_png16.png
|
|
22
|
+
demo/preview.png
|
|
23
|
+
experiments/**/package*.json
|
|
24
|
+
|
|
25
|
+
# …but keep the committed regression fixtures (browser-streamed interop golden file)
|
|
26
|
+
!tests/fixtures/stream.webm
|
|
27
|
+
!tests/fixtures/stream_depth.u16
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
cmake_minimum_required(VERSION 3.15)
|
|
2
|
+
project(chromapakz LANGUAGES CXX)
|
|
3
|
+
|
|
4
|
+
set(CMAKE_CXX_STANDARD 17)
|
|
5
|
+
set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
|
6
|
+
if(NOT CMAKE_BUILD_TYPE)
|
|
7
|
+
set(CMAKE_BUILD_TYPE Release)
|
|
8
|
+
endif()
|
|
9
|
+
|
|
10
|
+
# Royalty-free VP9 via libvpx (BSD). pkg-config provides include/lib flags.
|
|
11
|
+
find_package(PkgConfig REQUIRED)
|
|
12
|
+
pkg_check_modules(VPX REQUIRED IMPORTED_TARGET vpx)
|
|
13
|
+
|
|
14
|
+
# Native core shared library. Output named "_core" (no lib prefix) so the Python
|
|
15
|
+
# package's ctypes loader finds chromapakz/_core.{so,dylib}.
|
|
16
|
+
add_library(_core SHARED native/chromapakz.cpp)
|
|
17
|
+
set_target_properties(_core PROPERTIES PREFIX "" OUTPUT_NAME "_core")
|
|
18
|
+
target_include_directories(_core PRIVATE native)
|
|
19
|
+
target_link_libraries(_core PRIVATE PkgConfig::VPX)
|
|
20
|
+
|
|
21
|
+
# Command-line tool (dev/testing; not shipped in the Python wheel).
|
|
22
|
+
add_executable(dccli native/dccli.cpp native/chromapakz.cpp)
|
|
23
|
+
target_include_directories(dccli PRIVATE native)
|
|
24
|
+
target_link_libraries(dccli PRIVATE PkgConfig::VPX)
|
|
25
|
+
|
|
26
|
+
# scikit-build-core installs the core lib into the wheel's package directory.
|
|
27
|
+
install(TARGETS _core LIBRARY DESTINATION chromapakz RUNTIME DESTINATION chromapakz)
|
chromapakz-0.2.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Kevin Blackburn-Matzen
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,236 @@
|
|
|
1
|
+
Metadata-Version: 2.1
|
|
2
|
+
Name: chromapakz
|
|
3
|
+
Version: 0.2.0
|
|
4
|
+
Summary: Lossless RGB + bit-exact auxiliary signals (depth, object IDs, …) in one WebM (VP9)
|
|
5
|
+
Keywords: depth,rgbd,video,codec,vp9,lossless,webm,signals,object-id
|
|
6
|
+
Author: kmatzen
|
|
7
|
+
License: MIT
|
|
8
|
+
Project-URL: Homepage, https://github.com/kmatzen/ChromaPakZ
|
|
9
|
+
Requires-Python: >=3.9
|
|
10
|
+
Requires-Dist: numpy>=1.20
|
|
11
|
+
Description-Content-Type: text/markdown
|
|
12
|
+
|
|
13
|
+
# ChromaPakZ
|
|
14
|
+
|
|
15
|
+
<p align="center"><img src="docs/logo.png" alt="ChromaPakZ — lossless RGBD video encoder" width="680"></p>
|
|
16
|
+
|
|
17
|
+
**A lossless RGBD video codec** (クロマパックZ): one ordinary `.webm` that carries an 8-bit **RGB** track
|
|
18
|
+
alongside **bit-exact 16-bit auxiliary signals** — depth, object IDs, packed normals, or any other `W×H`
|
|
19
|
+
`uint16` plane — in sync. It is built so that
|
|
20
|
+
|
|
21
|
+
- a **legacy player shows plain RGB** — the depth rides in extra tracks a normal player ignores;
|
|
22
|
+
- it uses only **royalty-free** codecs (VP9 / libvpx, BSD) — no GPL encoder, no patent pool;
|
|
23
|
+
- it runs **in the browser via WebCodecs** — **no WASM on Chromium**, with a small libvpx-WASM fallback for engines whose native path isn't bit-exact; and
|
|
24
|
+
- depth is packed with one **reversible map**, not the range-slice bookkeeping of older schemes;
|
|
25
|
+
- **multiple lossless uint16 signals** (depth, object IDs, …) share one container in sync.
|
|
26
|
+
|
|
27
|
+
It's a clean-room redo of an older MP4/x264 approach (RGB as YUV, plus 16-bit depth sliced into several
|
|
28
|
+
lossless-10-bit ranges). That design worked but had three thorns: x264 is GPL, the range-slicing was
|
|
29
|
+
fiddly, and the browser always needed a WASM codec. ChromaPakZ removes the first two outright; for the
|
|
30
|
+
third, Chromium runs end-to-end on WebCodecs with **no WASM**, and a small libvpx-WASM build is kept only
|
|
31
|
+
as a per-operation fallback for engines whose native path isn't bit-exact (Firefox, Safari).
|
|
32
|
+
|
|
33
|
+
The same format is implemented three times — **browser (WebCodecs)**, **C++ (libvpx)**, and **Python** —
|
|
34
|
+
and a file written by any one decodes bit-exactly in the others.
|
|
35
|
+
|
|
36
|
+
## Quickstart
|
|
37
|
+
|
|
38
|
+
```sh
|
|
39
|
+
# Browser demo — encode→file→decode→view, entirely in-page (no WASM on Chromium)
|
|
40
|
+
python3 -m http.server 8000 # from the repo root, then open http://localhost:8000/demo/
|
|
41
|
+
|
|
42
|
+
# Python / C++ — the native libvpx core is compiled from source, so install the build
|
|
43
|
+
# prerequisites first: libvpx (dev headers), pkg-config, CMake, and a C++17 compiler.
|
|
44
|
+
# macOS: brew install libvpx pkg-config cmake ninja
|
|
45
|
+
# Debian: sudo apt-get install libvpx-dev pkg-config cmake ninja-build g++
|
|
46
|
+
pip install . # pip compiles the native core via CMake and bundles it
|
|
47
|
+
python -c "import chromapakz as cz; print(cz.inverse_depth_spec(0.3, 9.0))"
|
|
48
|
+
# cz.encode({"depth": u16}, specs={"depth": cz.inverse_depth_spec(near, far)}, rgb=rgba)
|
|
49
|
+
|
|
50
|
+
# Browser JS API — streaming encode/decode (see docs/API.md)
|
|
51
|
+
# createEncoder({ signals: [{ id:'depth', near, far }, { id:'objectId' }] })
|
|
52
|
+
# createDecoder(bytes).readFrame() -> { rgb, signals: { depth: { u16 }, objectId: { u16 } } }
|
|
53
|
+
|
|
54
|
+
# C++ / CLI
|
|
55
|
+
cmake -S . -B build && cmake --build build -j # or: native/build.sh
|
|
56
|
+
./build/dccli selftest
|
|
57
|
+
./build/dccli decodesignal clip.webm depth depth.u16
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
## How it works
|
|
61
|
+
|
|
62
|
+
| Layer | Choice |
|
|
63
|
+
|---|---|
|
|
64
|
+
| **Container** | WebM / Matroska, multi-track. RGB is track 1, so any player shows it; depth tracks are ignored by players that don't know them. A Duration, a Cues index, and ~1 s RGB keyframes make it **seekable** in `<video>` (depth stays single-keyframe — it isn't what `<video>` plays). |
|
|
65
|
+
| **RGB track** | 8-bit VP9, YUV 4:2:0, BT.709 full-range — a normal, viewable video stream. |
|
|
66
|
+
| **Lossless signals** | Each signal: optional quant (e.g. inverse-depth for float depth) → **uint16** → **triangle-fold 8+8** → two **VP9 lossless** tracks. Add object IDs, labels, etc. as additional signal pairs. |
|
|
67
|
+
| **Metadata** | v2 `signals[]` only — each signal (`id`, tracks, scheme, quant). |
|
|
68
|
+
|
|
69
|
+
**Inverse-depth quantization** spends precision where it matters (near surfaces), matching how stereo/ToF
|
|
70
|
+
sensors behave. Float can't be stored losslessly in 16 bits, so this quantization *is* the format's defined
|
|
71
|
+
precision boundary; everything below it is bit-exact.
|
|
72
|
+
|
|
73
|
+
**Triangle-fold** is the key trick. The naive low byte `d & 0xFF` is a sawtooth — a hard `255→0` cliff
|
|
74
|
+
every 256 levels — and those manufactured edges wreck any spatial predictor (this is exactly why the old
|
|
75
|
+
design needed range slices). Reflecting every other segment (`lo = (high&1) ? 255-lo : lo`) turns it into a
|
|
76
|
+
continuous triangle wave with no cliffs, so VP9's own predictor works. It's range-slicing collapsed into one
|
|
77
|
+
reversible map, with nothing to manage.
|
|
78
|
+
|
|
79
|
+
**Full color range is signaled** in the bitstream (`VP9E_SET_COLOR_RANGE`), so a range-honouring decoder
|
|
80
|
+
returns the packed luma unscaled instead of applying a limited-range conversion that would corrupt depth.
|
|
81
|
+
|
|
82
|
+
### Why these choices (measured, not assumed — Chromium 148)
|
|
83
|
+
|
|
84
|
+
WebCodecs has no "lossless" switch, so every claim here is a measurement from
|
|
85
|
+
`experiments/webcodecs-lossless`:
|
|
86
|
+
|
|
87
|
+
- **VP9 at QP 0 is bit-exact through WebCodecs; AV1 is not** (AV1 `quantizer:0` drifts by up to ~257). So
|
|
88
|
+
VP9 carries depth; AV1 is fine only for the lossy RGB track.
|
|
89
|
+
- **Triangle-fold beats a naive byte-split by ~13%**, and **inter-coding cuts another ~52%** (and stays
|
|
90
|
+
bit-exact across the GOP) — most of what looks like incompressible LSB noise is actually static fold
|
|
91
|
+
structure that temporal prediction removes.
|
|
92
|
+
- **8+8 beats high-bit-depth.** 10-bit VP9 encode *is* available in browsers, but a 10+6 split is ~4%
|
|
93
|
+
*worse* than 8+8 and narrows browser reach, so 8+8 wins on both counts.
|
|
94
|
+
|
|
95
|
+
[`docs/EVALUATION.md`](docs/EVALUATION.md) is the full due-diligence record: every codec/container/packing
|
|
96
|
+
alternative considered, the constraint that eliminates each, a head-to-head benchmark (ChromaPakZ beats
|
|
97
|
+
FFV1, PNG-16 and x264 on the same 16-bit depth, beats x265/HEVC at matched 11-bit precision, and lands
|
|
98
|
+
within 1–2% of LZMA), cited licensing/browser
|
|
99
|
+
facts, and a sensitivity analysis of when a different choice would win.
|
|
100
|
+
|
|
101
|
+
## What it costs
|
|
102
|
+
|
|
103
|
+
Lossless 16-bit depth of a real sensor is **noise-bound**: the low bits are largely sensor noise, and
|
|
104
|
+
lossless coding must preserve every bit of it. On **real Kinect data** (TUM RGB-D `fr1/desk`, 30 frames at
|
|
105
|
+
640×480, 78% valid):
|
|
106
|
+
|
|
107
|
+
| track | bits / pixel |
|
|
108
|
+
|---|---|
|
|
109
|
+
| RGB | 0.19 |
|
|
110
|
+
| depth (hi + lo) | 0.50 + 4.35 |
|
|
111
|
+
| **total** | **5.04** |
|
|
112
|
+
|
|
113
|
+
— depth round-tripped **bit-exact**. Reproduce with `examples/tum_fr1desk.py` (see its header for the
|
|
114
|
+
one-line dataset fetch).
|
|
115
|
+
|
|
116
|
+
The one knob that moves this is the **quantization precision** vs the sensor's noise floor. Spreading depth
|
|
117
|
+
over all 65,535 codes makes one step far finer than the noise, so the codec faithfully archives randomness.
|
|
118
|
+
Coarsening the grid to match the noise collapses the cost — without losing real signal. The sweep below is
|
|
119
|
+
measured on the synthetic benchmark clip (`make_synthetic_rgbd.py`, range ≈0.9–7.8 m) — a separate clip from
|
|
120
|
+
the TUM numbers above:
|
|
121
|
+
|
|
122
|
+
| effective bits | depth precision at 7.8 m | depth bpp |
|
|
123
|
+
|---|---|---|
|
|
124
|
+
| 16 (default) | 0.9 mm per step | 13.2 |
|
|
125
|
+
| 12 | 14 mm per step | 9.7 |
|
|
126
|
+
| 11 | 28 mm per step | 8.1 |
|
|
127
|
+
| 10 | 56 mm per step | 6.9 |
|
|
128
|
+
|
|
129
|
+
(Reproduce the bpp column with `python python/benchmark_codecs.py`.) `levels` is a first-class,
|
|
130
|
+
metadata-stored parameter (default 65536 = full 16-bit) shared by all three implementations, so
|
|
131
|
+
reduced-precision files reconstruct identically everywhere. Set it with `ingest.py --depth-bits N` or the
|
|
132
|
+
`levels=` argument.
|
|
133
|
+
|
|
134
|
+
### Codec rate-distortion
|
|
135
|
+
|
|
136
|
+
This is a separate axis from precision: how faithfully the *codec* carries whatever quantized depth you
|
|
137
|
+
give it. PSNR here is the encode→decode path measured against the source codes.
|
|
138
|
+
|
|
139
|
+

|
|
140
|
+
|
|
141
|
+
The lossless codecs all sit on the **∞-dB band** — they reproduce depth exactly and differ only in size,
|
|
142
|
+
where ChromaPakZ (VP9) is smallest, just under FFV1, with PNG-16 well behind. The blue curve is ChromaPakZ's
|
|
143
|
+
own near-lossless option (sweeping the VP9 quantizer trades fidelity for size), but the default operating
|
|
144
|
+
point is **QP 0, bit-exact**. Regenerate with `python python/plot_rd.py`.
|
|
145
|
+
|
|
146
|
+
> **A note on ffmpeg.** *Decoding* ChromaPakZ files with ffmpeg (or any conformant VP9 decoder) is
|
|
147
|
+
> bit-exact. But *encode* with ChromaPakZ, not the ffmpeg CLI: `ffmpeg -c:v libvpx-vp9 -lossless 1` is
|
|
148
|
+
> lossless yet **~3× larger** (≈39 vs ≈13 bpp) — same library, far worse coding decisions, and no flag
|
|
149
|
+
> tested closes the gap. `python/plot_rd.py` therefore uses the real WebCodecs encoder for the VP9 numbers.
|
|
150
|
+
|
|
151
|
+
## Cross-language implementations
|
|
152
|
+
|
|
153
|
+
All three read and write the identical `.webm`, verified bit-exact in every direction (browser ⇄ C++ ⇄
|
|
154
|
+
Python), and produce standard files — `ffprobe` reports `matroska,webm` with one RGB stream plus two VP9
|
|
155
|
+
streams per lossless signal, and ffmpeg decodes track 0 as plain RGB when present.
|
|
156
|
+
|
|
157
|
+
Format schema: [`docs/FORMAT.md`](docs/FORMAT.md). API: [`docs/API.md`](docs/API.md).
|
|
158
|
+
|
|
159
|
+
| Surface | Codec | Build |
|
|
160
|
+
|---|---|---|
|
|
161
|
+
| **Browser** | WebCodecs VP9 | none — `src/chromapakz.js`, `src/signals.js`, `src/webm.js`. Multi-signal streaming API. |
|
|
162
|
+
| **C++** | libvpx VP9 | CMake → `build/_core` + `dccli` (`dc_encode_multi`, `dc_decode_signal`) |
|
|
163
|
+
| **Python** | ctypes → C++ | `pip install .` — `encode()`, `decode()`, `parse_metadata()` |
|
|
164
|
+
|
|
165
|
+
```sh
|
|
166
|
+
./build/dccli encodergbd rgb.rgba depth.u16 W H N fps near far kbps out.webm
|
|
167
|
+
./build/dccli decodesignal clip.webm objectId ids.u16
|
|
168
|
+
./build/dccli decodergb clip.webm rgb.rgba
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
## Real-data ingestion (`python/`)
|
|
172
|
+
|
|
173
|
+
- **`ingest.py`** — load depth (`.exr` / `.npy` / `.npz` / 16-bit PNG·TIFF / raw) and optional RGB (image
|
|
174
|
+
sequence, video via ffmpeg, or array), auto-derive inverse-depth `near`/`far` from percentiles, encode,
|
|
175
|
+
and report real per-track bpp. Invalid pixels (`<=0`/NaN) map to code 0.
|
|
176
|
+
`python ingest.py --depth 'd_*.exr' --rgb 'rgb_*.png' -o clip.webm --report --verify`
|
|
177
|
+
- **`make_synthetic_rgbd.py`** — a realistic RGBD generator (smooth surfaces, depth edges, disparity-domain
|
|
178
|
+
noise, occlusion shadows, dropout holes) for when you don't have a sensor handy.
|
|
179
|
+
- **`webm_inspect.py`** — pure-Python EBML parser for the per-track byte breakdown.
|
|
180
|
+
|
|
181
|
+
## How it relates to RealSense / Kinect
|
|
182
|
+
|
|
183
|
+
Depth-camera ecosystems already split into two camps; ChromaPakZ takes the best of both.
|
|
184
|
+
|
|
185
|
+
- **Intel RealSense** colorizes 16-bit depth into an RGB image (Hue, ~10.5 effective bits) and encodes that
|
|
186
|
+
with a stock H.264/H.265 codec. Great for streaming and reuse of hardware codecs, but **lossy** — unfit
|
|
187
|
+
for ground-truth or archival depth.
|
|
188
|
+
- **Kinect / RGBD datasets** store depth raw or as 16-bit PNG. Azure Kinect even records to **Matroska**
|
|
189
|
+
with a 16-bit depth track (lossless via per-frame PNG); TUM RGB-D, NYU and ScanNet use 16-bit PNG
|
|
190
|
+
sequences. Bit-exact, but **intra-only and large** — no temporal compression.
|
|
191
|
+
|
|
192
|
+
| | RealSense colorize | Kinect / PNG | **ChromaPakZ** |
|
|
193
|
+
|---|---|---|---|
|
|
194
|
+
| bit-exact 16-bit depth | ✗ (lossy) | ✓ | **✓** |
|
|
195
|
+
| RGB plays in any legacy player | ✓ | — | **✓** |
|
|
196
|
+
| inter-frame (temporal) compression | ✓ (lossy) | ✗ | **✓ (lossless)** |
|
|
197
|
+
| royalty-free, browser-native (no WASM on Chromium) | — | — | **✓** |
|
|
198
|
+
|
|
199
|
+
That Azure Kinect already chose Matroska — WebM's basis — is telling. ChromaPakZ differs by *compressing*
|
|
200
|
+
depth losslessly (VP9 + triangle-fold, inter-coded) rather than storing raw or intra PNG, and by running in
|
|
201
|
+
the browser. Sources:
|
|
202
|
+
[RealSense colorized depth](https://dev.intelrealsense.com/docs/depth-image-compression-by-colorization-for-intel-realsense-depth-cameras),
|
|
203
|
+
[Azure Kinect record format](https://learn.microsoft.com/en-us/azure/kinect-dk/record-file-format).
|
|
204
|
+
|
|
205
|
+
## Repository layout
|
|
206
|
+
|
|
207
|
+
```
|
|
208
|
+
src/ chromapakz.js, signals.js, webm.js, chromapakz-core.js
|
|
209
|
+
native/ chromapakz.{h,cpp}, dccli.cpp
|
|
210
|
+
python/ chromapakz/ (pip package), ingest.py, make_synthetic_rgbd.py
|
|
211
|
+
demo/ index.html in-browser encode→decode→view
|
|
212
|
+
examples/ tum_fr1desk.py
|
|
213
|
+
experiments/ webcodecs-lossless/ run.mjs, smoke-demo.mjs, headless tests
|
|
214
|
+
docs/ FORMAT.md, API.md, EVALUATION.md, RELEASING.md
|
|
215
|
+
tests/ roundtrip.py, cross_interop.py, stream_interop.py, ffmpeg_interop.py, js_*.mjs
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
CI builds and tests on Linux + macOS and runs the in-browser VP9-lossless probe in headless Chromium;
|
|
219
|
+
`docs/RELEASING.md` covers wheels and PyPI publishing. The full design rationale and benchmarks are in
|
|
220
|
+
[`docs/EVALUATION.md`](docs/EVALUATION.md).
|
|
221
|
+
|
|
222
|
+
## Status & limitations
|
|
223
|
+
|
|
224
|
+
Working end-to-end and verified across all three implementations. Honest caveats:
|
|
225
|
+
|
|
226
|
+
- **Browser support is engine-specific** (measured, [`EVALUATION.md` §11](docs/EVALUATION.md)): native
|
|
227
|
+
WebCodecs lossless *encode* is Chromium-only today (WebKit lacks WebCodecs' quantizer mode; Firefox's
|
|
228
|
+
QP 0 isn't lossless); native lossless *decode* works on Chromium and WebKit/Safari, while Firefox
|
|
229
|
+
decodes VP9 to color-converted BGRX. Where native can't be trusted, the library transparently falls
|
|
230
|
+
back to a bundled **libvpx-WASM** codec, chosen *per operation* by a cached runtime probe — so a
|
|
231
|
+
decode-only browser (e.g. Safari) downloads only `vp9-decode.wasm` and never the larger encoder, and
|
|
232
|
+
vice-versa. Force it with `backend: 'webcodecs' | 'wasm'` (default `'auto'`); see [`docs/API.md`](docs/API.md).
|
|
233
|
+
These are Playwright engine builds — reconfirm on shipping browsers before hard claims.
|
|
234
|
+
- **"Royalty-free"** reflects the AOMedia/Google position on VP9; Sisvel operates pools that dispute it.
|
|
235
|
+
- An **auto precision picker** (estimate the sensor noise floor to choose `--depth-bits`) is future work.
|
|
236
|
+
- **Network byte streaming** is supported via `onChunk` on encode and `createDecoder()` + `push()`/`finish()` on decode. See [`docs/API.md`](docs/API.md).
|