opencodecs 0.2.0__tar.gz → 0.3.1__tar.gz

{opencodecs-0.2.0 → opencodecs-0.3.1}/CHANGES.rst

@@ -10,6 +10,146 @@ Versions follow the same ``YYYY.M.D`` cadence as upstream when we
 publish; the entries below cluster work by date rather than by
 release because most of it has shipped continuously to ``main``.
 
+0.3.1 (2026-09-28)
+------------------
+
+A CZI read scheduling fix, and a clear refusal for the files that cannot
+stack. Figures compare against 0.3.0 on the same machines.
+
+**CZI whole-stack reads hand out work by bytes**
+
+``read`` derived its batch size from the worker count, giving each worker
+``ceil(n/workers)`` consecutive sub-blocks. That makes the read wait on
+that many sub-blocks even when most workers have already finished: nine
+8 MiB sub-blocks over eight workers is two rounds, and measured 15.1 ms
+against 7.9 ms for nine one-sub-block tasks. Tasks are now sized by
+output bytes, about 8 MiB each, so a large sub-block gets a task to
+itself and small ones ride together. That keeps the reason batching
+existed, which is that a 0.7 MiB tile cannot pay for its own future,
+without paying the rounding. ``_READ_MAX_WORKERS`` goes 8 to 32: the old
+value was measured against the dispatch above, where extra workers could
+not help. With byte-sized tasks, 20 to 32 measured fastest on both a
+20-core arm64 Mac and a 128-core x86-64 Linux host, and past 32 the Linux
+host gives it back to memory-bandwidth contention. A semaphore keeps the
+resolved worker count a real bound when the byte sizing yields more tasks
+than that.
+
+Eight files of 2000 x 2000 uint16 ZSTDHDR sub-blocks, read back to back:
+22.1 to about 15 ms per file on macOS, which moves a whole-stack read
+from a little behind aicspylibczi to about 1.3x ahead of it.
+
+**A mixed sub-block CZI says so, instead of failing inside the codec**
+
+``read`` stacks every sub-block into one array, and sized that array from
+the first sub-block, which is wrong for any file whose sub-blocks differ:
+a pyramidal slide scan interleaves down-scaled sub-blocks with
+full-resolution ones, and a mosaic can clip its edge tiles. The decode
+destination check then failed with an element count the caller never
+chose, naming neither the file's shape mix nor a way forward. It now
+refuses up front, listing the shapes it cannot stack and the APIs that do
+work (``CziPyramidReader.read_region``, ``entries_at_level``,
+``read_tile`` and ``iter_tiles``), and the new ``CziReader.is_uniform``
+lets a caller branch before asking. Decoding was never at fault: JPEG XR
+sub-blocks are pixel-exact against aicspylibczi.
+
+0.3.0 (2026-09-27)
+------------------
+
+Mostly a speed release. Figures below compare against 0.2.0 unless they
+say otherwise; the README's Performance section has the full table for a
+Mac and a Linux workstation.
+
+**Shared bounded pipeline across readers and writers**
+
+A common layer in ``opencodecs.core`` now carries the work that each
+format used to do its own way: ordered, byte-bounded parallel
+scheduling (``core.pipeline.map_bounded``) with a shared worker
+budget, per-worker scratch buffers, caller-owned output buffers and
+sinks for the native codecs, stateful streaming iterators for seven
+byte codecs, native PNG row sessions, selective range sources, and
+opt-in bit-exact verification of written segments. Each piece of work
+is recorded, with its measured limits, in ``pipeline_catalog.toml``
+and checked by ``ci/check_pipeline_catalog.py``.
+
+**Zarr, TIFF and blosc2 under many threads**
+
+- Zarr regional reads can run in parallel, with ``num_workers`` on
+  ``read_region`` or ``OmeZarrArray``; the default stays serial. Chunks
+  are grouped into tasks of about 2 MiB on a persistent pool, chunk files
+  are read with os-level calls, and plain zstd chunks decompress straight
+  into the output. A 4096 x 4096 array in 256 x 256 chunks, 0.2.0's serial
+  read against eight workers: zstd 72.9 to 19.6 ms on Linux (50.1 to
+  14.9 ms on macOS), zlib 184 to 38.8 ms (80.5 to 25.9 ms).
+- Zarr v2 ``blosc`` chunks decode through the native blosc2 extension
+  instead of numcodecs, whose binding serializes calls across threads:
+  the default serial read goes 37.5 to 20.0 ms on macOS and 35.1 to
+  29.4 ms on Linux.
+- Fix: blosc2 encode selected its compressor through process-global
+  state, so concurrent encodes could use another thread's compressor.
+  Encode, decode and partial decode now each use a context of their own,
+  which also stops them queuing on blosc2's global mutex: from eight
+  threads, 64 chunks of 1 MiB decode 6.2 to 6.5x and encode 6.6 to 7.1x
+  faster. Chunks written with item size 1 and shuffle carry slightly
+  different header flags; old and new chunks decode in both versions.
+- TIFF tiles with the horizontal predictor decode into a writable array,
+  so the predictor no longer copies each tile first (1.1 to 1.2x on a
+  tiled 4096 x 4096 deflate image).
+
+**CZI: faster reads and writes, and real slides read correctly**
+
+- JPEG XR sub-blocks (compression 4, used by most Zeiss slide scans) now
+  decode through a new optional ``_jpegxr`` extension over jxrlib, built
+  when jxrlib is installed. Every tile of the Axioscan corpus slide and its
+  whole 20684 x 32751 level 0 match czifile exactly. The wheels build
+  jxrlib from source (conda-forge on Windows) and link it statically.
+- Fix: overlapping mosaic tiles composed in directory order. Zen writes
+  tiles out of mosaic-index order, and libCZI and czifile draw the higher
+  mosaic index on top; each level now composes in that order, which on the
+  corpus slide changes 8.9 million overlap pixels at level 0.
+- Fix: ``CziPyramidReader`` took sub-block starts as level pixels.
+  Zen stores them in full-resolution slide coordinates, often far from
+  zero and negative, so a real slide reported every level as ``(N, 0)``.
+  Levels now count from the slide origin (``reader.origin``, matching
+  czifile) and divide positions by the level's scale.
+- Sub-blocks decode straight into their destination. Whole-stack
+  ``read`` accepts ``out=`` (``numpy.memmap`` included). zstd
+  decompression and the byte unshuffle run as one native call, which
+  let several threads decode small tiles at once: a stack read with eight
+  workers goes 12.0 to 3.9 ms on Linux (2.5 to 1.7 ms on macOS).
+- Regional reads use a lazily built spatial index and write each tile
+  straight into the output; overlapping tiles still resolve to the
+  later one in the directory. ``PyramidReader.read_regions`` decodes
+  the union of a batch of boxes once, and ``decoded_cache_bytes`` adds
+  an opt-in decoded-tile cache.
+- Opening is faster: the directory parses in about a third of the
+  time and the pyramid reader no longer rescans it per level (180 ms to
+  3 ms on a 12k-sub-block slide).
+- ``read_regions`` on CZI decodes each tile shared by several boxes once,
+  straight into every box's output: 50 disjoint crops take 5.1 ms on
+  Linux against 29.8 ms one call at a time in 0.2.0 (3.1 against 17.2 ms
+  on macOS).
+- The writer shuffles natively, emits sub-blocks as parts without
+  copying unverified payloads, and ``write_many`` compresses frames on
+  workers while keeping the file byte-identical to sequential writes.
+  Eight 2048 x 2048 frames with zstd: 189 to 89 ms one frame at a time on
+  Linux and 41 ms with ``write_many`` (82 to 54 and 20 ms on macOS).
+- ``CziWriter(background_encode=True)`` compresses frames of 1 MiB or
+  more on a background thread while the caller prepares the next one.
+  Opt-in: with work between frames it wrote 1.44 to 1.46x faster, with
+  frames ready and nothing to overlap it was 0.87 to 0.96x.
+
+**Build and tests**
+
+- Every wheel now carries the same 40 compiled extensions, the 39 of
+  0.2.0 plus ``_jpegxr``, and ``ci/check_wheel_contents.py`` requires it.
+- SZ3 3.3.2, since upstream deleted the 3.3.1 tag.
+- The DICOMweb client is tested over HTTP against a local server
+  (``tests/_dicomweb_server.py``) serving a synthetic study written by
+  reference encoders, instead of a public demo server.
+- Fix: the pipeline catalog check accepted a rooted evidence path such
+  as ``/etc/passwd`` on Windows, where ``Path.is_absolute()`` needs a
+  drive letter.
+
 0.2.0 (2026-09-09)
 ------------------
 
@@ -1029,9 +1169,9 @@ streaming write-out without changing their code.
 * Locally validated on a Windows VM (MSVC 14.44 — same as CI) and
   a Linux x86_64 host: ``bash bench/build_codec_libs.sh
   --only=mozjpeg`` followed by ``setup.py build_ext --inplace``
-  produces ``_mozjpeg.{pyd,so}`` cleanly. Per the new lesson in
-  CLAUDE.md, this iteration was validated end-to-end through the
-  same script CI runs, with the same env vars, before any push.
+  produces ``_mozjpeg.{pyd,so}`` cleanly. This iteration was validated
+  end-to-end through the same script CI runs, with the same env vars,
+  before any push.
 
 **Post-publish wheel coverage check (CI hardening)**
 

{opencodecs-0.2.0 → opencodecs-0.3.1}/MANIFEST.in

@@ -21,6 +21,7 @@ recursive-include src/opencodecs *.pyx *.pxd *.h *.hpp
 include src/opencodecs/codecs/b2nd_helpers.c
 include src/opencodecs/codecs/isal_shim.c
 include src/opencodecs/codecs/webp_shim.c
+include src/opencodecs/codecs/jpegxr_shim.c
 include src/opencodecs/codecs/openjph_shim.cpp
 recursive-include 3rdparty *.c *.h *.hpp *.md *.rst *.txt *.toml *.json
 

{opencodecs-0.2.0/src/opencodecs.egg-info → opencodecs-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: opencodecs
-Version: 0.2.0
+Version: 0.3.1
 Summary: Streaming, network-aware image codecs for scientific imaging (prototype: JPEG XL)
 Author: Kevin Cutler
 License-Expression: BSD-3-Clause
@@ -297,9 +297,14 @@ with open_remote_hdf5("https://bucket.s3.amazonaws.com/big.h5") as f:
     arr = f["img"][:1024, :1024]                          # all from cache
 ```
 
-`czi` decodes types 0 (uncompressed) and 6 (ZSTDHDR) — the entire
-modern Zen archive. JPEG-XR sub-blocks (rare in 2022+ output) raise
-`NotImplementedError`. The reader exposes `metadata_bytes` and
+`czi` decodes compression types 0 (uncompressed), 5 (zstd), 6 (ZSTDHDR)
+and 4 (JPEG XR). JPEG XR, which most Zeiss slide scans use, needs the
+optional `_jpegxr` extension, built when jxrlib is installed (Homebrew
+`jxrlib`, conda-forge `jxrlib`, Debian/Ubuntu `libjxr-dev`). The wheels
+carry it from 0.3.0 on, statically linked on every platform. A JPEG XR
+sub-block without it raises `CziError`. Pyramid levels are placed from the slide origin and scale, and
+overlapping mosaic tiles compose with the higher mosaic index on top, as
+libCZI and czifile do. The reader exposes `metadata_bytes` and
 `metadata_xml` as lazy zero-copy accessors.
 
 ### zarr v3 codecs
@@ -320,8 +325,66 @@ z = zarr.create_array(
 
 ## Performance
 
+### 0.3.0 against 0.2.0
+
+0.3.0 is mostly a speed release. Each row below ran both versions in fresh
+processes on the same inputs, alternating which went first, and counts only
+where both returned identical pixels. Medians in milliseconds, on a 20-core
+Apple silicon Mac and a 64-core x86-64 Linux workstation.
+
+| Workload | Mac 0.2.0 | Mac 0.3.0 | Mac speedup | Linux 0.2.0 | Linux 0.3.0 | Linux speedup |
+|---|---:|---:|---:|---:|---:|---:|
+| CZI | | | | | | |
+| Open a whole-slide file | 77.6 | 27.9 | 2.8x | 88.4 | 48.7 | 1.8x |
+| Open it and read the first crop | 255 | 39.1 | 6.5x | 267 | 58.7 | 4.6x |
+| Build the pyramid reader | 174 | 3.4 | 51.1x | 164 | 4.7 | 35.0x |
+| Read a 20 x 20 tile mosaic | 23.9 | 10.1 | 2.4x | 55.8 | 18.4 | 3.0x |
+| Small crops of the mosaic | 69.5 | 26.6 | 2.6x | 121 | 54.7 | 2.2x |
+| 50 disjoint crops, one call each | 17.2 | 6.7 | 2.6x | 29.8 | 11.3 | 2.6x |
+| 50 disjoint crops, `read_regions` | 17.2 | 3.1 | 5.5x | 29.8 | 5.1 | 5.9x |
+| Stack read, 8 workers | 2.5 | 1.7 | 1.5x | 12.0 | 3.9 | 3.1x |
+| Write 8 frames, zstd | 82.1 | 53.9 | 1.5x | 189 | 89.3 | 2.1x |
+| Write 8 frames, zstd, `write_many` | 82.1 | 19.9 | 4.1x | 189 | 40.9 | 4.6x |
+| Write 8 frames, uncompressed | 28.9 | 10.1 | 2.8x | 226 | 38.3 | 5.9x |
+| Zarr, 4096 x 4096 uint16 in 256 x 256 chunks, whole array | | | | | | |
+| v2 zstd, default | 50.1 | 45.6 | 1.1x | 72.9 | 62.2 | 1.2x |
+| v2 zstd, `num_workers=8` | 50.1 | 14.9 | 3.4x | 72.9 | 19.6 | 3.7x |
+| v2 zlib, `num_workers=8` | 80.5 | 25.9 | 3.1x | 184 | 38.8 | 4.7x |
+| v2 blosc, default | 37.5 | 20.0 | 1.9x | 35.1 | 29.4 | 1.2x |
+| v3 zstd, `num_workers=8` | 51.3 | 14.6 | 3.5x | 62.1 | 19.4 | 3.2x |
+| blosc2 from 8 threads, 64 chunks of 1 MiB | | | | | | |
+| Decode | 25.4 | 3.9 | 6.5x | 38.0 | 6.1 | 6.2x |
+| Encode | 32.9 | 5.0 | 6.6x | 47.9 | 6.8 | 7.1x |
+| TIFF | | | | | | |
+| Tiled, deflate with horizontal predictor, 4096 x 4096 | 20.5 | 17.5 | 1.2x | 25.4 | 23.4 | 1.1x |
+
+Rows that name `num_workers=8`, `read_regions` or `write_many` use options
+new in 0.3.0; they are compared with the only way 0.2.0 could do the same
+work. Zarr regional reads stay serial unless you pass `num_workers`
+(to `read_region` or `OmeZarrArray`).
+
+Where it comes from:
+
+- **CZI**: the directory parses with precompiled structs, pyramid levels
+  are grouped once instead of per level, and each tile is decompressed,
+  unshuffled and written into the output in one native call that releases
+  the GIL once. Separate calls made threads queue for the GIL between
+  steps. Regional reads use a spatial index and decode only the tiles a
+  box touches; `read_regions` decodes a tile shared by several boxes once.
+- **Zarr**: parallel regional reads are new, with consecutive chunks
+  batched into tasks of about 2 MiB, plain zstd chunks decompressed
+  straight into the output, and v2 blosc chunks decoded through the native
+  blosc2 extension rather than numcodecs, whose binding serializes calls
+  across threads.
+- **blosc2**: each call now has its own compression context. The old path
+  selected the compressor through process-global state behind a global
+  mutex, which serialized every thread and could hand one thread another's
+  compressor (a correctness bug, also fixed).
+
+### Across releases
+
 Headline numbers from the latest bench run (`bench/run_benchmarks.py
---fast`, macOS M1 Ultra, vs `imagecodecs` / `tifffile` / `ndstorage`):
+--fast`, Apple silicon Mac, vs `imagecodecs` / `tifffile` / `ndstorage`):
 
 | Workload | opencodecs | reference | ratio |
 |---|---:|---:|---:|
@@ -354,7 +417,7 @@ to 8× wall-clock).
 Scientific microscopy CZI (66 MB, 14 sub-blocks of 2000×2000 uint16,
 ZSTDHDR), single-file warm cache:
 
-| Reader        | Mac M3 | Threadripper x86_64 |
+| Reader        | Apple silicon Mac | x86-64 Linux |
 |---------------|-------:|--------------------:|
 | czifile (Python ref) | 148 ms | 414 ms       |
 | aicspylibczi (C++)   |  17 ms | 140 ms       |
@@ -396,9 +459,12 @@ caps = tomllib.load(open("capabilities.toml", "rb"))["codec"]
 [c["name"] for c in caps if c["pyramid"]]   # codecs with a pyramid reader
 ```
 
-The manifest covers the codec registry, so the reader-only backends
-(Imaris, OME-Zarr, NDTiff, N5) are not in it; they are listed in the
-tables above and reached through `open_pyramid` / their own classes.
+The runtime manifest covers the codec registry. The companion
+[pipeline catalog](pipeline_catalog.toml) also covers direct adapters such as
+Imaris, OME-Zarr, NDTiff and N5, and maps shared optimization work across all
+registered codecs. See the [implementation plan](docs/pipeline_optimization_plan.md)
+for priorities, source evidence, correctness prerequisites and acceptance tests.
+Run `python ci/check_pipeline_catalog.py report` for the current worklist.
 
 ### Codec registry
 
@@ -543,10 +609,12 @@ prefix = jxl.thumbnail_bytes("scan.jxl")
 pip install opencodecs
 ```
 
-Wheels are published for CPython 3.10–3.13 on macOS (arm64),
-Linux (x86_64 + aarch64), and Windows (amd64). Each wheel
-bundles libjxl, libavif, libheif, libwebp, libdeflate,
-c-blosc2, and friends — no system dependencies needed.
+Wheels are published for CPython 3.10 to 3.13 on macOS 15+ (arm64),
+Linux (x86_64 + aarch64, manylinux_2_28), and Windows (amd64). Every
+platform ships the same 39 compiled extensions, bundling libjxl,
+libavif, libheif, libwebp, libdeflate, c-blosc2, CharLS and friends,
+so no system dependencies are needed. Intel Macs and macOS releases
+older than 15 install from the sdist instead.
 
 For a source install, system development headers, or to build a
 tuned local libjxl, see [INSTALL.md](INSTALL.md). Wheel publishing
@@ -569,29 +637,41 @@ build).
 
 ## Status
 
-- **v0.1.1** on PyPI (May 2026). Core API stable; **1066 tests passing**
-  on Mac M1 Ultra + Linux x86_64/aarch64 + Windows VM
-- Native readers + writers for the common scientific containers
-  (TIFF, BigTIFF, OME-TIFF, CZI, NDTiff, HDF5, JXL, FITS,
-  OME-Zarr v2 + v3 sharded)
-- Cross-platform bench coverage: Mac arm64 (canonical), Windows 11 LTSC
-  (libvirt VM), Linux x86_64 (Threadripper-class)
+- **v0.3.0** on PyPI (September 2026). Every wheel carries the same 40
+  compiled extensions, and `ci/check_wheel_contents.py` fails the
+  release build if one goes missing.
+- About 4,000 tests locally, including a 40-dataset conformance corpus;
+  CI runs the corpus-independent suite on macOS, Linux and Windows for
+  Python 3.10 and 3.13.
+- Native readers and writers for the common scientific containers
+  (TIFF, BigTIFF, OME-TIFF, CZI, NDTiff, HDF5, JXL, FITS, OME-Zarr v2 +
+  v3 sharded), and pyramid readers for TIFF/COG/SVS, OME-Zarr, CZI,
+  Imaris, DICOM VL Whole Slide Microscopy and Olympus VSI/ETS
+- Shared source adapters support paths, buffers, file objects and remote URLs.
+  Indexed readers use checked range requests, covering caches and adaptive
+  read-ahead; whole-stream codecs retain their documented eager input paths.
+- Multi-frame AVIF, animated WebP and GIF decode to a frame stack; HEIF
+  exposes every top-level image
 - Compression backend auto-detect (libdeflate → zlib-ng-compat → stdlib)
-- Cloud I/O primitives (`HTTPDataSource` with covering-cache + adaptive
-  read-ahead) wired into TIFF / HDF5 / DICOMweb / CZI / FITS / Zarr v3
-  readers
 - `tifffile_patch` opt-in shim reroutes tifffile's codec dispatch through
   opencodecs for users who want only a partial swap
+- `capabilities.toml` records what each codec actually supports, checked
+  against the built extensions, with no open gaps
 
 Deferred work (see [`docs/TODO_DEFERRED.md`](docs/TODO_DEFERRED.md)):
 
-- **Windows wheels currently miss `_sz3`, `_pcodec`, `_sperr`, `_brunsli`**
-  — toolchain mismatch (conda's bash picks GCC over MSVC for CMake);
-  v0.1.2 will restore them. macOS + Linux wheels have the full set.
-- CCITT Fax3/Fax4 encode — legacy fax; zero scientific users
-- JPEG-XR — abandoned format outside niche DICOM
-- libspng `filter_sum` SIMD — off the bench-tracked workload (`h2h_png_4mp_rgb`
-  is at 1.14× already); filter-bound PNG-encode users could see another 2-3×
+- CCITT Fax3/Fax4 encode: legacy fax, zero scientific users
+- JPEG XR encode (decode exists for CZI; see above)
+
+GIF and JPEG XL destination writers now drain encoded frame output before close.
+The shared pipeline also supplies bounded independent-piece scheduling, seven
+stateful byte-codec iterators, native PNG row sessions, selective native sources,
+and checked caller-owned destinations.
+
+Shared streaming, buffering, and overlap mechanisms and remaining integration
+work are documented in [the pipeline catalog](docs/shared_pipeline_capabilities.md).
+
+See [CHANGES.rst](CHANGES.rst) for release history.
 
 ## License
 

{opencodecs-0.2.0 → opencodecs-0.3.1}/README.md

@@ -245,9 +245,14 @@ with open_remote_hdf5("https://bucket.s3.amazonaws.com/big.h5") as f:
     arr = f["img"][:1024, :1024]                          # all from cache
 ```
 
-`czi` decodes types 0 (uncompressed) and 6 (ZSTDHDR) — the entire
-modern Zen archive. JPEG-XR sub-blocks (rare in 2022+ output) raise
-`NotImplementedError`. The reader exposes `metadata_bytes` and
+`czi` decodes compression types 0 (uncompressed), 5 (zstd), 6 (ZSTDHDR)
+and 4 (JPEG XR). JPEG XR, which most Zeiss slide scans use, needs the
+optional `_jpegxr` extension, built when jxrlib is installed (Homebrew
+`jxrlib`, conda-forge `jxrlib`, Debian/Ubuntu `libjxr-dev`). The wheels
+carry it from 0.3.0 on, statically linked on every platform. A JPEG XR
+sub-block without it raises `CziError`. Pyramid levels are placed from the slide origin and scale, and
+overlapping mosaic tiles compose with the higher mosaic index on top, as
+libCZI and czifile do. The reader exposes `metadata_bytes` and
 `metadata_xml` as lazy zero-copy accessors.
 
 ### zarr v3 codecs
@@ -268,8 +273,66 @@ z = zarr.create_array(
 
 ## Performance
 
+### 0.3.0 against 0.2.0
+
+0.3.0 is mostly a speed release. Each row below ran both versions in fresh
+processes on the same inputs, alternating which went first, and counts only
+where both returned identical pixels. Medians in milliseconds, on a 20-core
+Apple silicon Mac and a 64-core x86-64 Linux workstation.
+
+| Workload | Mac 0.2.0 | Mac 0.3.0 | Mac speedup | Linux 0.2.0 | Linux 0.3.0 | Linux speedup |
+|---|---:|---:|---:|---:|---:|---:|
+| CZI | | | | | | |
+| Open a whole-slide file | 77.6 | 27.9 | 2.8x | 88.4 | 48.7 | 1.8x |
+| Open it and read the first crop | 255 | 39.1 | 6.5x | 267 | 58.7 | 4.6x |
+| Build the pyramid reader | 174 | 3.4 | 51.1x | 164 | 4.7 | 35.0x |
+| Read a 20 x 20 tile mosaic | 23.9 | 10.1 | 2.4x | 55.8 | 18.4 | 3.0x |
+| Small crops of the mosaic | 69.5 | 26.6 | 2.6x | 121 | 54.7 | 2.2x |
+| 50 disjoint crops, one call each | 17.2 | 6.7 | 2.6x | 29.8 | 11.3 | 2.6x |
+| 50 disjoint crops, `read_regions` | 17.2 | 3.1 | 5.5x | 29.8 | 5.1 | 5.9x |
+| Stack read, 8 workers | 2.5 | 1.7 | 1.5x | 12.0 | 3.9 | 3.1x |
+| Write 8 frames, zstd | 82.1 | 53.9 | 1.5x | 189 | 89.3 | 2.1x |
+| Write 8 frames, zstd, `write_many` | 82.1 | 19.9 | 4.1x | 189 | 40.9 | 4.6x |
+| Write 8 frames, uncompressed | 28.9 | 10.1 | 2.8x | 226 | 38.3 | 5.9x |
+| Zarr, 4096 x 4096 uint16 in 256 x 256 chunks, whole array | | | | | | |
+| v2 zstd, default | 50.1 | 45.6 | 1.1x | 72.9 | 62.2 | 1.2x |
+| v2 zstd, `num_workers=8` | 50.1 | 14.9 | 3.4x | 72.9 | 19.6 | 3.7x |
+| v2 zlib, `num_workers=8` | 80.5 | 25.9 | 3.1x | 184 | 38.8 | 4.7x |
+| v2 blosc, default | 37.5 | 20.0 | 1.9x | 35.1 | 29.4 | 1.2x |
+| v3 zstd, `num_workers=8` | 51.3 | 14.6 | 3.5x | 62.1 | 19.4 | 3.2x |
+| blosc2 from 8 threads, 64 chunks of 1 MiB | | | | | | |
+| Decode | 25.4 | 3.9 | 6.5x | 38.0 | 6.1 | 6.2x |
+| Encode | 32.9 | 5.0 | 6.6x | 47.9 | 6.8 | 7.1x |
+| TIFF | | | | | | |
+| Tiled, deflate with horizontal predictor, 4096 x 4096 | 20.5 | 17.5 | 1.2x | 25.4 | 23.4 | 1.1x |
+
+Rows that name `num_workers=8`, `read_regions` or `write_many` use options
+new in 0.3.0; they are compared with the only way 0.2.0 could do the same
+work. Zarr regional reads stay serial unless you pass `num_workers`
+(to `read_region` or `OmeZarrArray`).
+
+Where it comes from:
+
+- **CZI**: the directory parses with precompiled structs, pyramid levels
+  are grouped once instead of per level, and each tile is decompressed,
+  unshuffled and written into the output in one native call that releases
+  the GIL once. Separate calls made threads queue for the GIL between
+  steps. Regional reads use a spatial index and decode only the tiles a
+  box touches; `read_regions` decodes a tile shared by several boxes once.
+- **Zarr**: parallel regional reads are new, with consecutive chunks
+  batched into tasks of about 2 MiB, plain zstd chunks decompressed
+  straight into the output, and v2 blosc chunks decoded through the native
+  blosc2 extension rather than numcodecs, whose binding serializes calls
+  across threads.
+- **blosc2**: each call now has its own compression context. The old path
+  selected the compressor through process-global state behind a global
+  mutex, which serialized every thread and could hand one thread another's
+  compressor (a correctness bug, also fixed).
+
+### Across releases
+
 Headline numbers from the latest bench run (`bench/run_benchmarks.py
---fast`, macOS M1 Ultra, vs `imagecodecs` / `tifffile` / `ndstorage`):
+--fast`, Apple silicon Mac, vs `imagecodecs` / `tifffile` / `ndstorage`):
 
 | Workload | opencodecs | reference | ratio |
 |---|---:|---:|---:|
@@ -302,7 +365,7 @@ to 8× wall-clock).
 Scientific microscopy CZI (66 MB, 14 sub-blocks of 2000×2000 uint16,
 ZSTDHDR), single-file warm cache:
 
-| Reader        | Mac M3 | Threadripper x86_64 |
+| Reader        | Apple silicon Mac | x86-64 Linux |
 |---------------|-------:|--------------------:|
 | czifile (Python ref) | 148 ms | 414 ms       |
 | aicspylibczi (C++)   |  17 ms | 140 ms       |
@@ -344,9 +407,12 @@ caps = tomllib.load(open("capabilities.toml", "rb"))["codec"]
 [c["name"] for c in caps if c["pyramid"]]   # codecs with a pyramid reader
 ```
 
-The manifest covers the codec registry, so the reader-only backends
-(Imaris, OME-Zarr, NDTiff, N5) are not in it; they are listed in the
-tables above and reached through `open_pyramid` / their own classes.
+The runtime manifest covers the codec registry. The companion
+[pipeline catalog](pipeline_catalog.toml) also covers direct adapters such as
+Imaris, OME-Zarr, NDTiff and N5, and maps shared optimization work across all
+registered codecs. See the [implementation plan](docs/pipeline_optimization_plan.md)
+for priorities, source evidence, correctness prerequisites and acceptance tests.
+Run `python ci/check_pipeline_catalog.py report` for the current worklist.
 
 ### Codec registry
 
@@ -491,10 +557,12 @@ prefix = jxl.thumbnail_bytes("scan.jxl")
 pip install opencodecs
 ```
 
-Wheels are published for CPython 3.10–3.13 on macOS (arm64),
-Linux (x86_64 + aarch64), and Windows (amd64). Each wheel
-bundles libjxl, libavif, libheif, libwebp, libdeflate,
-c-blosc2, and friends — no system dependencies needed.
+Wheels are published for CPython 3.10 to 3.13 on macOS 15+ (arm64),
+Linux (x86_64 + aarch64, manylinux_2_28), and Windows (amd64). Every
+platform ships the same 39 compiled extensions, bundling libjxl,
+libavif, libheif, libwebp, libdeflate, c-blosc2, CharLS and friends,
+so no system dependencies are needed. Intel Macs and macOS releases
+older than 15 install from the sdist instead.
 
 For a source install, system development headers, or to build a
 tuned local libjxl, see [INSTALL.md](INSTALL.md). Wheel publishing
@@ -517,29 +585,41 @@ build).
 
 ## Status
 
-- **v0.1.1** on PyPI (May 2026). Core API stable; **1066 tests passing**
-  on Mac M1 Ultra + Linux x86_64/aarch64 + Windows VM
-- Native readers + writers for the common scientific containers
-  (TIFF, BigTIFF, OME-TIFF, CZI, NDTiff, HDF5, JXL, FITS,
-  OME-Zarr v2 + v3 sharded)
-- Cross-platform bench coverage: Mac arm64 (canonical), Windows 11 LTSC
-  (libvirt VM), Linux x86_64 (Threadripper-class)
+- **v0.3.0** on PyPI (September 2026). Every wheel carries the same 40
+  compiled extensions, and `ci/check_wheel_contents.py` fails the
+  release build if one goes missing.
+- About 4,000 tests locally, including a 40-dataset conformance corpus;
+  CI runs the corpus-independent suite on macOS, Linux and Windows for
+  Python 3.10 and 3.13.
+- Native readers and writers for the common scientific containers
+  (TIFF, BigTIFF, OME-TIFF, CZI, NDTiff, HDF5, JXL, FITS, OME-Zarr v2 +
+  v3 sharded), and pyramid readers for TIFF/COG/SVS, OME-Zarr, CZI,
+  Imaris, DICOM VL Whole Slide Microscopy and Olympus VSI/ETS
+- Shared source adapters support paths, buffers, file objects and remote URLs.
+  Indexed readers use checked range requests, covering caches and adaptive
+  read-ahead; whole-stream codecs retain their documented eager input paths.
+- Multi-frame AVIF, animated WebP and GIF decode to a frame stack; HEIF
+  exposes every top-level image
 - Compression backend auto-detect (libdeflate → zlib-ng-compat → stdlib)
-- Cloud I/O primitives (`HTTPDataSource` with covering-cache + adaptive
-  read-ahead) wired into TIFF / HDF5 / DICOMweb / CZI / FITS / Zarr v3
-  readers
 - `tifffile_patch` opt-in shim reroutes tifffile's codec dispatch through
   opencodecs for users who want only a partial swap
+- `capabilities.toml` records what each codec actually supports, checked
+  against the built extensions, with no open gaps
 
 Deferred work (see [`docs/TODO_DEFERRED.md`](docs/TODO_DEFERRED.md)):
 
-- **Windows wheels currently miss `_sz3`, `_pcodec`, `_sperr`, `_brunsli`**
-  — toolchain mismatch (conda's bash picks GCC over MSVC for CMake);
-  v0.1.2 will restore them. macOS + Linux wheels have the full set.
-- CCITT Fax3/Fax4 encode — legacy fax; zero scientific users
-- JPEG-XR — abandoned format outside niche DICOM
-- libspng `filter_sum` SIMD — off the bench-tracked workload (`h2h_png_4mp_rgb`
-  is at 1.14× already); filter-bound PNG-encode users could see another 2-3×
+- CCITT Fax3/Fax4 encode: legacy fax, zero scientific users
+- JPEG XR encode (decode exists for CZI; see above)
+
+GIF and JPEG XL destination writers now drain encoded frame output before close.
+The shared pipeline also supplies bounded independent-piece scheduling, seven
+stateful byte-codec iterators, native PNG row sessions, selective native sources,
+and checked caller-owned destinations.
+
+Shared streaming, buffering, and overlap mechanisms and remaining integration
+work are documented in [the pipeline catalog](docs/shared_pipeline_capabilities.md).
+
+See [CHANGES.rst](CHANGES.rst) for release history.
 
 ## License
 

{opencodecs-0.2.0 → opencodecs-0.3.1}/THIRD-PARTY.md

@@ -165,7 +165,7 @@ each library.
 | libaec | 1.1.6 | BSD-2-Clause |
 | lerc | 4.1.0 | Apache-2.0 |
 | zfp | 1.0.1 | BSD-3-Clause |
-| SZ3 | 3.3.1 | BSD-3-Clause |
+| SZ3 | 3.3.2 | BSD-3-Clause |
 | SPERR | 0.8.5 | Apache-2.0 |
 | pcodec | 1.0.2 | Apache-2.0 |
 | brunsli | master | MIT |

{opencodecs-0.2.0 → opencodecs-0.3.1}/pyproject.toml

@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "opencodecs"
-version = "0.2.0"
+version = "0.3.1"
 description = "Streaming, network-aware image codecs for scientific imaging (prototype: JPEG XL)"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -194,7 +194,7 @@ before-all = [
     # bench/build_codec_libs.sh into the cached /cibw-jxl-prefix mount.
     # Skips encoders (aom, x265) — wheel ships decode-only AVIF/HEIF to
     # keep build < 12 min cold + avoid GPL x265.
-    "bash -c 'PATH=$HOME/.cargo/bin:$PATH bash bench/build_codec_libs.sh --only=libjpeg-turbo,c-blosc2,libde265,dav1d,libavif,libheif,libaec,lerc,zfp,SZ3,pcodec,SPERR,brunsli,mozjpeg,libultrahdr,openjph,CharLS'",
+    "bash -c 'PATH=$HOME/.cargo/bin:$PATH bash bench/build_codec_libs.sh --only=libjpeg-turbo,c-blosc2,libde265,dav1d,libavif,libheif,libaec,lerc,zfp,SZ3,pcodec,SPERR,brunsli,mozjpeg,libultrahdr,openjph,CharLS,jxrlib'",
     "bash bench/build_libjxl.sh",
 ]
 # Don't set OPENCODECS_*_PREFIX here — build_wheels.yml's
@@ -215,7 +215,7 @@ before-all = [
     # only, not libturbojpeg, so the _mozjpeg probe couldn't find what it
     # needed. Source-build via build_codec_libs.sh produces both libs.
     "brew install jpeg-xl libavif libheif webp libpng jpeg-turbo openjpeg zstd brotli c-blosc2 lz4 libdeflate libaec liblerc zfp rust cmake ninja libultrahdr giflib snappy charls isa-l",
-    "bash bench/build_codec_libs.sh --only=SZ3,pcodec,SPERR,brunsli,mozjpeg,openjph",
+    "bash bench/build_codec_libs.sh --only=SZ3,pcodec,SPERR,brunsli,mozjpeg,openjph,jxrlib",
 ]
 # macos-latest is now macOS 15 (Sequoia) on arm64. Homebrew formulas are
 # compiled for 15.x, so any wheel target lower than 15 fails the delocate