opencodecs 0.2.0__tar.gz → 0.3.0__tar.gz

{opencodecs-0.2.0 → opencodecs-0.3.0}/CHANGES.rst

@@ -10,6 +10,104 @@ Versions follow the same ``YYYY.M.D`` cadence as upstream when we
 publish; the entries below cluster work by date rather than by
 release because most of it has shipped continuously to ``main``.
 
+0.3.0 (2026-09-27)
+------------------
+
+Mostly a speed release. Figures below compare against 0.2.0 unless they
+say otherwise; the README's Performance section has the full table for a
+Mac and a Linux workstation.
+
+**Shared bounded pipeline across readers and writers**
+
+A common layer in ``opencodecs.core`` now carries the work that each
+format used to do its own way: ordered, byte-bounded parallel
+scheduling (``core.pipeline.map_bounded``) with a shared worker
+budget, per-worker scratch buffers, caller-owned output buffers and
+sinks for the native codecs, stateful streaming iterators for seven
+byte codecs, native PNG row sessions, selective range sources, and
+opt-in bit-exact verification of written segments. Each piece of work
+is recorded, with its measured limits, in ``pipeline_catalog.toml``
+and checked by ``ci/check_pipeline_catalog.py``.
+
+**Zarr, TIFF and blosc2 under many threads**
+
+- Zarr regional reads can run in parallel, with ``num_workers`` on
+  ``read_region`` or ``OmeZarrArray``; the default stays serial. Chunks
+  are grouped into tasks of about 2 MiB on a persistent pool, chunk files
+  are read with os-level calls, and plain zstd chunks decompress straight
+  into the output. A 4096 x 4096 array in 256 x 256 chunks, 0.2.0's serial
+  read against eight workers: zstd 72.9 to 19.6 ms on Linux (50.1 to
+  14.9 ms on macOS), zlib 184 to 38.8 ms (80.5 to 25.9 ms).
+- Zarr v2 ``blosc`` chunks decode through the native blosc2 extension
+  instead of numcodecs, whose binding serializes calls across threads:
+  the default serial read goes 37.5 to 20.0 ms on macOS and 35.1 to
+  29.4 ms on Linux.
+- Fix: blosc2 encode selected its compressor through process-global
+  state, so concurrent encodes could use another thread's compressor.
+  Encode, decode and partial decode now each use a context of their own,
+  which also stops them queuing on blosc2's global mutex: from eight
+  threads, 64 chunks of 1 MiB decode 6.2 to 6.5x and encode 6.6 to 7.1x
+  faster. Chunks written with item size 1 and shuffle carry slightly
+  different header flags; old and new chunks decode in both versions.
+- TIFF tiles with the horizontal predictor decode into a writable array,
+  so the predictor no longer copies each tile first (1.1 to 1.2x on a
+  tiled 4096 x 4096 deflate image).
+
+**CZI: faster reads and writes, and real slides read correctly**
+
+- JPEG XR sub-blocks (compression 4, used by most Zeiss slide scans) now
+  decode through a new optional ``_jpegxr`` extension over jxrlib, built
+  when jxrlib is installed. Every tile of the Axioscan corpus slide and its
+  whole 20684 x 32751 level 0 match czifile exactly. The wheels build
+  jxrlib from source (conda-forge on Windows) and link it statically.
+- Fix: overlapping mosaic tiles composed in directory order. Zen writes
+  tiles out of mosaic-index order, and libCZI and czifile draw the higher
+  mosaic index on top; each level now composes in that order, which on the
+  corpus slide changes 8.9 million overlap pixels at level 0.
+- Fix: ``CziPyramidReader`` took sub-block starts as level pixels.
+  Zen stores them in full-resolution slide coordinates, often far from
+  zero and negative, so a real slide reported every level as ``(N, 0)``.
+  Levels now count from the slide origin (``reader.origin``, matching
+  czifile) and divide positions by the level's scale.
+- Sub-blocks decode straight into their destination. Whole-stack
+  ``read`` accepts ``out=`` (``numpy.memmap`` included). zstd
+  decompression and the byte unshuffle run as one native call, which
+  let several threads decode small tiles at once: a stack read with eight
+  workers goes 12.0 to 3.9 ms on Linux (2.5 to 1.7 ms on macOS).
+- Regional reads use a lazily built spatial index and write each tile
+  straight into the output; overlapping tiles still resolve to the
+  later one in the directory. ``PyramidReader.read_regions`` decodes
+  the union of a batch of boxes once, and ``decoded_cache_bytes`` adds
+  an opt-in decoded-tile cache.
+- Opening is faster: the directory parses in about a third of the
+  time and the pyramid reader no longer rescans it per level (180 ms to
+  3 ms on a 12k-sub-block slide).
+- ``read_regions`` on CZI decodes each tile shared by several boxes once,
+  straight into every box's output: 50 disjoint crops take 5.1 ms on
+  Linux against 29.8 ms one call at a time in 0.2.0 (3.1 against 17.2 ms
+  on macOS).
+- The writer shuffles natively, emits sub-blocks as parts without
+  copying unverified payloads, and ``write_many`` compresses frames on
+  workers while keeping the file byte-identical to sequential writes.
+  Eight 2048 x 2048 frames with zstd: 189 to 89 ms one frame at a time on
+  Linux and 41 ms with ``write_many`` (82 to 54 and 20 ms on macOS).
+- ``CziWriter(background_encode=True)`` compresses frames of 1 MiB or
+  more on a background thread while the caller prepares the next one.
+  Opt-in: with work between frames it wrote 1.44 to 1.46x faster, with
+  frames ready and nothing to overlap it was 0.87 to 0.96x.
+
+**Build and tests**
+
+- Every wheel now carries the same 40 compiled extensions, the 39 of
+  0.2.0 plus ``_jpegxr``, and ``ci/check_wheel_contents.py`` requires it.
+- SZ3 3.3.2, since upstream deleted the 3.3.1 tag.
+- The DICOMweb client is tested over HTTP against a local server
+  (``tests/_dicomweb_server.py``) serving a synthetic study written by
+  reference encoders, instead of a public demo server.
+- Fix: the pipeline catalog check accepted a rooted evidence path such
+  as ``/etc/passwd`` on Windows, where ``Path.is_absolute()`` needs a
+  drive letter.
+
 0.2.0 (2026-09-09)
 ------------------
 
@@ -1029,9 +1127,9 @@ streaming write-out without changing their code.
 * Locally validated on a Windows VM (MSVC 14.44 — same as CI) and
   a Linux x86_64 host: ``bash bench/build_codec_libs.sh
   --only=mozjpeg`` followed by ``setup.py build_ext --inplace``
-  produces ``_mozjpeg.{pyd,so}`` cleanly. Per the new lesson in
-  CLAUDE.md, this iteration was validated end-to-end through the
-  same script CI runs, with the same env vars, before any push.
+  produces ``_mozjpeg.{pyd,so}`` cleanly. This iteration was validated
+  end-to-end through the same script CI runs, with the same env vars,
+  before any push.
 
 **Post-publish wheel coverage check (CI hardening)**
 

{opencodecs-0.2.0 → opencodecs-0.3.0}/MANIFEST.in

@@ -21,6 +21,7 @@ recursive-include src/opencodecs *.pyx *.pxd *.h *.hpp
 include src/opencodecs/codecs/b2nd_helpers.c
 include src/opencodecs/codecs/isal_shim.c
 include src/opencodecs/codecs/webp_shim.c
+include src/opencodecs/codecs/jpegxr_shim.c
 include src/opencodecs/codecs/openjph_shim.cpp
 recursive-include 3rdparty *.c *.h *.hpp *.md *.rst *.txt *.toml *.json
 

{opencodecs-0.2.0/src/opencodecs.egg-info → opencodecs-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: opencodecs
-Version: 0.2.0
+Version: 0.3.0
 Summary: Streaming, network-aware image codecs for scientific imaging (prototype: JPEG XL)
 Author: Kevin Cutler
 License-Expression: BSD-3-Clause
@@ -297,9 +297,14 @@ with open_remote_hdf5("https://bucket.s3.amazonaws.com/big.h5") as f:
     arr = f["img"][:1024, :1024]                          # all from cache
 ```
 
-`czi` decodes types 0 (uncompressed) and 6 (ZSTDHDR) — the entire
-modern Zen archive. JPEG-XR sub-blocks (rare in 2022+ output) raise
-`NotImplementedError`. The reader exposes `metadata_bytes` and
+`czi` decodes compression types 0 (uncompressed), 5 (zstd), 6 (ZSTDHDR)
+and 4 (JPEG XR). JPEG XR, which most Zeiss slide scans use, needs the
+optional `_jpegxr` extension, built when jxrlib is installed (Homebrew
+`jxrlib`, conda-forge `jxrlib`, Debian/Ubuntu `libjxr-dev`). The wheels
+carry it from 0.3.0 on, statically linked on every platform. A JPEG XR
+sub-block without it raises `CziError`. Pyramid levels are placed from the slide origin and scale, and
+overlapping mosaic tiles compose with the higher mosaic index on top, as
+libCZI and czifile do. The reader exposes `metadata_bytes` and
 `metadata_xml` as lazy zero-copy accessors.
 
 ### zarr v3 codecs
@@ -320,8 +325,66 @@ z = zarr.create_array(
 
 ## Performance
 
+### 0.3.0 against 0.2.0
+
+0.3.0 is mostly a speed release. Each row below ran both versions in fresh
+processes on the same inputs, alternating which went first, and counts only
+where both returned identical pixels. Medians in milliseconds, on a 20-core
+Apple silicon Mac and a 64-core x86-64 Linux workstation.
+
+| Workload | Mac 0.2.0 | Mac 0.3.0 | Mac speedup | Linux 0.2.0 | Linux 0.3.0 | Linux speedup |
+|---|---:|---:|---:|---:|---:|---:|
+| CZI | | | | | | |
+| Open a whole-slide file | 77.6 | 27.9 | 2.8x | 88.4 | 48.7 | 1.8x |
+| Open it and read the first crop | 255 | 39.1 | 6.5x | 267 | 58.7 | 4.6x |
+| Build the pyramid reader | 174 | 3.4 | 51.1x | 164 | 4.7 | 35.0x |
+| Read a 20 x 20 tile mosaic | 23.9 | 10.1 | 2.4x | 55.8 | 18.4 | 3.0x |
+| Small crops of the mosaic | 69.5 | 26.6 | 2.6x | 121 | 54.7 | 2.2x |
+| 50 disjoint crops, one call each | 17.2 | 6.7 | 2.6x | 29.8 | 11.3 | 2.6x |
+| 50 disjoint crops, `read_regions` | 17.2 | 3.1 | 5.5x | 29.8 | 5.1 | 5.9x |
+| Stack read, 8 workers | 2.5 | 1.7 | 1.5x | 12.0 | 3.9 | 3.1x |
+| Write 8 frames, zstd | 82.1 | 53.9 | 1.5x | 189 | 89.3 | 2.1x |
+| Write 8 frames, zstd, `write_many` | 82.1 | 19.9 | 4.1x | 189 | 40.9 | 4.6x |
+| Write 8 frames, uncompressed | 28.9 | 10.1 | 2.8x | 226 | 38.3 | 5.9x |
+| Zarr, 4096 x 4096 uint16 in 256 x 256 chunks, whole array | | | | | | |
+| v2 zstd, default | 50.1 | 45.6 | 1.1x | 72.9 | 62.2 | 1.2x |
+| v2 zstd, `num_workers=8` | 50.1 | 14.9 | 3.4x | 72.9 | 19.6 | 3.7x |
+| v2 zlib, `num_workers=8` | 80.5 | 25.9 | 3.1x | 184 | 38.8 | 4.7x |
+| v2 blosc, default | 37.5 | 20.0 | 1.9x | 35.1 | 29.4 | 1.2x |
+| v3 zstd, `num_workers=8` | 51.3 | 14.6 | 3.5x | 62.1 | 19.4 | 3.2x |
+| blosc2 from 8 threads, 64 chunks of 1 MiB | | | | | | |
+| Decode | 25.4 | 3.9 | 6.5x | 38.0 | 6.1 | 6.2x |
+| Encode | 32.9 | 5.0 | 6.6x | 47.9 | 6.8 | 7.1x |
+| TIFF | | | | | | |
+| Tiled, deflate with horizontal predictor, 4096 x 4096 | 20.5 | 17.5 | 1.2x | 25.4 | 23.4 | 1.1x |
+
+Rows that name `num_workers=8`, `read_regions` or `write_many` use options
+new in 0.3.0; they are compared with the only way 0.2.0 could do the same
+work. Zarr regional reads stay serial unless you pass `num_workers`
+(to `read_region` or `OmeZarrArray`).
+
+Where it comes from:
+
+- **CZI**: the directory parses with precompiled structs, pyramid levels
+  are grouped once instead of per level, and each tile is decompressed,
+  unshuffled and written into the output in one native call that releases
+  the GIL once. Separate calls made threads queue for the GIL between
+  steps. Regional reads use a spatial index and decode only the tiles a
+  box touches; `read_regions` decodes a tile shared by several boxes once.
+- **Zarr**: parallel regional reads are new, with consecutive chunks
+  batched into tasks of about 2 MiB, plain zstd chunks decompressed
+  straight into the output, and v2 blosc chunks decoded through the native
+  blosc2 extension rather than numcodecs, whose binding serializes calls
+  across threads.
+- **blosc2**: each call now has its own compression context. The old path
+  selected the compressor through process-global state behind a global
+  mutex, which serialized every thread and could hand one thread another's
+  compressor (a correctness bug, also fixed).
+
+### Across releases
+
 Headline numbers from the latest bench run (`bench/run_benchmarks.py
---fast`, macOS M1 Ultra, vs `imagecodecs` / `tifffile` / `ndstorage`):
+--fast`, Apple silicon Mac, vs `imagecodecs` / `tifffile` / `ndstorage`):
 
 | Workload | opencodecs | reference | ratio |
 |---|---:|---:|---:|
@@ -354,7 +417,7 @@ to 8× wall-clock).
 Scientific microscopy CZI (66 MB, 14 sub-blocks of 2000×2000 uint16,
 ZSTDHDR), single-file warm cache:
 
-| Reader        | Mac M3 | Threadripper x86_64 |
+| Reader        | Apple silicon Mac | x86-64 Linux |
 |---------------|-------:|--------------------:|
 | czifile (Python ref) | 148 ms | 414 ms       |
 | aicspylibczi (C++)   |  17 ms | 140 ms       |
@@ -396,9 +459,12 @@ caps = tomllib.load(open("capabilities.toml", "rb"))["codec"]
 [c["name"] for c in caps if c["pyramid"]]   # codecs with a pyramid reader
 ```
 
-The manifest covers the codec registry, so the reader-only backends
-(Imaris, OME-Zarr, NDTiff, N5) are not in it; they are listed in the
-tables above and reached through `open_pyramid` / their own classes.
+The runtime manifest covers the codec registry. The companion
+[pipeline catalog](pipeline_catalog.toml) also covers direct adapters such as
+Imaris, OME-Zarr, NDTiff and N5, and maps shared optimization work across all
+registered codecs. See the [implementation plan](docs/pipeline_optimization_plan.md)
+for priorities, source evidence, correctness prerequisites and acceptance tests.
+Run `python ci/check_pipeline_catalog.py report` for the current worklist.
 
 ### Codec registry
 
@@ -543,10 +609,12 @@ prefix = jxl.thumbnail_bytes("scan.jxl")
 pip install opencodecs
 ```
 
-Wheels are published for CPython 3.10–3.13 on macOS (arm64),
-Linux (x86_64 + aarch64), and Windows (amd64). Each wheel
-bundles libjxl, libavif, libheif, libwebp, libdeflate,
-c-blosc2, and friends — no system dependencies needed.
+Wheels are published for CPython 3.10 to 3.13 on macOS 15+ (arm64),
+Linux (x86_64 + aarch64, manylinux_2_28), and Windows (amd64). Every
+platform ships the same 39 compiled extensions, bundling libjxl,
+libavif, libheif, libwebp, libdeflate, c-blosc2, CharLS and friends,
+so no system dependencies are needed. Intel Macs and macOS releases
+older than 15 install from the sdist instead.
 
 For a source install, system development headers, or to build a
 tuned local libjxl, see [INSTALL.md](INSTALL.md). Wheel publishing
@@ -569,29 +637,41 @@ build).
 
 ## Status
 
-- **v0.1.1** on PyPI (May 2026). Core API stable; **1066 tests passing**
-  on Mac M1 Ultra + Linux x86_64/aarch64 + Windows VM
-- Native readers + writers for the common scientific containers
-  (TIFF, BigTIFF, OME-TIFF, CZI, NDTiff, HDF5, JXL, FITS,
-  OME-Zarr v2 + v3 sharded)
-- Cross-platform bench coverage: Mac arm64 (canonical), Windows 11 LTSC
-  (libvirt VM), Linux x86_64 (Threadripper-class)
+- **v0.3.0** on PyPI (September 2026). Every wheel carries the same 40
+  compiled extensions, and `ci/check_wheel_contents.py` fails the
+  release build if one goes missing.
+- About 4,000 tests locally, including a 40-dataset conformance corpus;
+  CI runs the corpus-independent suite on macOS, Linux and Windows for
+  Python 3.10 and 3.13.
+- Native readers and writers for the common scientific containers
+  (TIFF, BigTIFF, OME-TIFF, CZI, NDTiff, HDF5, JXL, FITS, OME-Zarr v2 +
+  v3 sharded), and pyramid readers for TIFF/COG/SVS, OME-Zarr, CZI,
+  Imaris, DICOM VL Whole Slide Microscopy and Olympus VSI/ETS
+- Shared source adapters support paths, buffers, file objects and remote URLs.
+  Indexed readers use checked range requests, covering caches and adaptive
+  read-ahead; whole-stream codecs retain their documented eager input paths.
+- Multi-frame AVIF, animated WebP and GIF decode to a frame stack; HEIF
+  exposes every top-level image
 - Compression backend auto-detect (libdeflate → zlib-ng-compat → stdlib)
-- Cloud I/O primitives (`HTTPDataSource` with covering-cache + adaptive
-  read-ahead) wired into TIFF / HDF5 / DICOMweb / CZI / FITS / Zarr v3
-  readers
 - `tifffile_patch` opt-in shim reroutes tifffile's codec dispatch through
   opencodecs for users who want only a partial swap
+- `capabilities.toml` records what each codec actually supports, checked
+  against the built extensions, with no open gaps
 
 Deferred work (see [`docs/TODO_DEFERRED.md`](docs/TODO_DEFERRED.md)):
 
-- **Windows wheels currently miss `_sz3`, `_pcodec`, `_sperr`, `_brunsli`**
-  — toolchain mismatch (conda's bash picks GCC over MSVC for CMake);
-  v0.1.2 will restore them. macOS + Linux wheels have the full set.
-- CCITT Fax3/Fax4 encode — legacy fax; zero scientific users
-- JPEG-XR — abandoned format outside niche DICOM
-- libspng `filter_sum` SIMD — off the bench-tracked workload (`h2h_png_4mp_rgb`
-  is at 1.14× already); filter-bound PNG-encode users could see another 2-3×
+- CCITT Fax3/Fax4 encode: legacy fax, zero scientific users
+- JPEG XR encode (decode exists for CZI; see above)
+
+GIF and JPEG XL destination writers now drain encoded frame output before close.
+The shared pipeline also supplies bounded independent-piece scheduling, seven
+stateful byte-codec iterators, native PNG row sessions, selective native sources,
+and checked caller-owned destinations.
+
+Shared streaming, buffering, and overlap mechanisms and remaining integration
+work are documented in [the pipeline catalog](docs/shared_pipeline_capabilities.md).
+
+See [CHANGES.rst](CHANGES.rst) for release history.
 
 ## License
 

{opencodecs-0.2.0 → opencodecs-0.3.0}/README.md

@@ -245,9 +245,14 @@ with open_remote_hdf5("https://bucket.s3.amazonaws.com/big.h5") as f:
     arr = f["img"][:1024, :1024]                          # all from cache
 ```
 
-`czi` decodes types 0 (uncompressed) and 6 (ZSTDHDR) — the entire
-modern Zen archive. JPEG-XR sub-blocks (rare in 2022+ output) raise
-`NotImplementedError`. The reader exposes `metadata_bytes` and
+`czi` decodes compression types 0 (uncompressed), 5 (zstd), 6 (ZSTDHDR)
+and 4 (JPEG XR). JPEG XR, which most Zeiss slide scans use, needs the
+optional `_jpegxr` extension, built when jxrlib is installed (Homebrew
+`jxrlib`, conda-forge `jxrlib`, Debian/Ubuntu `libjxr-dev`). The wheels
+carry it from 0.3.0 on, statically linked on every platform. A JPEG XR
+sub-block without it raises `CziError`. Pyramid levels are placed from the slide origin and scale, and
+overlapping mosaic tiles compose with the higher mosaic index on top, as
+libCZI and czifile do. The reader exposes `metadata_bytes` and
 `metadata_xml` as lazy zero-copy accessors.
 
 ### zarr v3 codecs
@@ -268,8 +273,66 @@ z = zarr.create_array(
 
 ## Performance
 
+### 0.3.0 against 0.2.0
+
+0.3.0 is mostly a speed release. Each row below ran both versions in fresh
+processes on the same inputs, alternating which went first, and counts only
+where both returned identical pixels. Medians in milliseconds, on a 20-core
+Apple silicon Mac and a 64-core x86-64 Linux workstation.
+
+| Workload | Mac 0.2.0 | Mac 0.3.0 | Mac speedup | Linux 0.2.0 | Linux 0.3.0 | Linux speedup |
+|---|---:|---:|---:|---:|---:|---:|
+| CZI | | | | | | |
+| Open a whole-slide file | 77.6 | 27.9 | 2.8x | 88.4 | 48.7 | 1.8x |
+| Open it and read the first crop | 255 | 39.1 | 6.5x | 267 | 58.7 | 4.6x |
+| Build the pyramid reader | 174 | 3.4 | 51.1x | 164 | 4.7 | 35.0x |
+| Read a 20 x 20 tile mosaic | 23.9 | 10.1 | 2.4x | 55.8 | 18.4 | 3.0x |
+| Small crops of the mosaic | 69.5 | 26.6 | 2.6x | 121 | 54.7 | 2.2x |
+| 50 disjoint crops, one call each | 17.2 | 6.7 | 2.6x | 29.8 | 11.3 | 2.6x |
+| 50 disjoint crops, `read_regions` | 17.2 | 3.1 | 5.5x | 29.8 | 5.1 | 5.9x |
+| Stack read, 8 workers | 2.5 | 1.7 | 1.5x | 12.0 | 3.9 | 3.1x |
+| Write 8 frames, zstd | 82.1 | 53.9 | 1.5x | 189 | 89.3 | 2.1x |
+| Write 8 frames, zstd, `write_many` | 82.1 | 19.9 | 4.1x | 189 | 40.9 | 4.6x |
+| Write 8 frames, uncompressed | 28.9 | 10.1 | 2.8x | 226 | 38.3 | 5.9x |
+| Zarr, 4096 x 4096 uint16 in 256 x 256 chunks, whole array | | | | | | |
+| v2 zstd, default | 50.1 | 45.6 | 1.1x | 72.9 | 62.2 | 1.2x |
+| v2 zstd, `num_workers=8` | 50.1 | 14.9 | 3.4x | 72.9 | 19.6 | 3.7x |
+| v2 zlib, `num_workers=8` | 80.5 | 25.9 | 3.1x | 184 | 38.8 | 4.7x |
+| v2 blosc, default | 37.5 | 20.0 | 1.9x | 35.1 | 29.4 | 1.2x |
+| v3 zstd, `num_workers=8` | 51.3 | 14.6 | 3.5x | 62.1 | 19.4 | 3.2x |
+| blosc2 from 8 threads, 64 chunks of 1 MiB | | | | | | |
+| Decode | 25.4 | 3.9 | 6.5x | 38.0 | 6.1 | 6.2x |
+| Encode | 32.9 | 5.0 | 6.6x | 47.9 | 6.8 | 7.1x |
+| TIFF | | | | | | |
+| Tiled, deflate with horizontal predictor, 4096 x 4096 | 20.5 | 17.5 | 1.2x | 25.4 | 23.4 | 1.1x |
+
+Rows that name `num_workers=8`, `read_regions` or `write_many` use options
+new in 0.3.0; they are compared with the only way 0.2.0 could do the same
+work. Zarr regional reads stay serial unless you pass `num_workers`
+(to `read_region` or `OmeZarrArray`).
+
+Where it comes from:
+
+- **CZI**: the directory parses with precompiled structs, pyramid levels
+  are grouped once instead of per level, and each tile is decompressed,
+  unshuffled and written into the output in one native call that releases
+  the GIL once. Separate calls made threads queue for the GIL between
+  steps. Regional reads use a spatial index and decode only the tiles a
+  box touches; `read_regions` decodes a tile shared by several boxes once.
+- **Zarr**: parallel regional reads are new, with consecutive chunks
+  batched into tasks of about 2 MiB, plain zstd chunks decompressed
+  straight into the output, and v2 blosc chunks decoded through the native
+  blosc2 extension rather than numcodecs, whose binding serializes calls
+  across threads.
+- **blosc2**: each call now has its own compression context. The old path
+  selected the compressor through process-global state behind a global
+  mutex, which serialized every thread and could hand one thread another's
+  compressor (a correctness bug, also fixed).
+
+### Across releases
+
 Headline numbers from the latest bench run (`bench/run_benchmarks.py
---fast`, macOS M1 Ultra, vs `imagecodecs` / `tifffile` / `ndstorage`):
+--fast`, Apple silicon Mac, vs `imagecodecs` / `tifffile` / `ndstorage`):
 
 | Workload | opencodecs | reference | ratio |
 |---|---:|---:|---:|
@@ -302,7 +365,7 @@ to 8× wall-clock).
 Scientific microscopy CZI (66 MB, 14 sub-blocks of 2000×2000 uint16,
 ZSTDHDR), single-file warm cache:
 
-| Reader        | Mac M3 | Threadripper x86_64 |
+| Reader        | Apple silicon Mac | x86-64 Linux |
 |---------------|-------:|--------------------:|
 | czifile (Python ref) | 148 ms | 414 ms       |
 | aicspylibczi (C++)   |  17 ms | 140 ms       |
@@ -344,9 +407,12 @@ caps = tomllib.load(open("capabilities.toml", "rb"))["codec"]
 [c["name"] for c in caps if c["pyramid"]]   # codecs with a pyramid reader
 ```
 
-The manifest covers the codec registry, so the reader-only backends
-(Imaris, OME-Zarr, NDTiff, N5) are not in it; they are listed in the
-tables above and reached through `open_pyramid` / their own classes.
+The runtime manifest covers the codec registry. The companion
+[pipeline catalog](pipeline_catalog.toml) also covers direct adapters such as
+Imaris, OME-Zarr, NDTiff and N5, and maps shared optimization work across all
+registered codecs. See the [implementation plan](docs/pipeline_optimization_plan.md)
+for priorities, source evidence, correctness prerequisites and acceptance tests.
+Run `python ci/check_pipeline_catalog.py report` for the current worklist.
 
 ### Codec registry
 
@@ -491,10 +557,12 @@ prefix = jxl.thumbnail_bytes("scan.jxl")
 pip install opencodecs
 ```
 
-Wheels are published for CPython 3.10–3.13 on macOS (arm64),
-Linux (x86_64 + aarch64), and Windows (amd64). Each wheel
-bundles libjxl, libavif, libheif, libwebp, libdeflate,
-c-blosc2, and friends — no system dependencies needed.
+Wheels are published for CPython 3.10 to 3.13 on macOS 15+ (arm64),
+Linux (x86_64 + aarch64, manylinux_2_28), and Windows (amd64). Every
+platform ships the same 39 compiled extensions, bundling libjxl,
+libavif, libheif, libwebp, libdeflate, c-blosc2, CharLS and friends,
+so no system dependencies are needed. Intel Macs and macOS releases
+older than 15 install from the sdist instead.
 
 For a source install, system development headers, or to build a
 tuned local libjxl, see [INSTALL.md](INSTALL.md). Wheel publishing
@@ -517,29 +585,41 @@ build).
 
 ## Status
 
-- **v0.1.1** on PyPI (May 2026). Core API stable; **1066 tests passing**
-  on Mac M1 Ultra + Linux x86_64/aarch64 + Windows VM
-- Native readers + writers for the common scientific containers
-  (TIFF, BigTIFF, OME-TIFF, CZI, NDTiff, HDF5, JXL, FITS,
-  OME-Zarr v2 + v3 sharded)
-- Cross-platform bench coverage: Mac arm64 (canonical), Windows 11 LTSC
-  (libvirt VM), Linux x86_64 (Threadripper-class)
+- **v0.3.0** on PyPI (September 2026). Every wheel carries the same 40
+  compiled extensions, and `ci/check_wheel_contents.py` fails the
+  release build if one goes missing.
+- About 4,000 tests locally, including a 40-dataset conformance corpus;
+  CI runs the corpus-independent suite on macOS, Linux and Windows for
+  Python 3.10 and 3.13.
+- Native readers and writers for the common scientific containers
+  (TIFF, BigTIFF, OME-TIFF, CZI, NDTiff, HDF5, JXL, FITS, OME-Zarr v2 +
+  v3 sharded), and pyramid readers for TIFF/COG/SVS, OME-Zarr, CZI,
+  Imaris, DICOM VL Whole Slide Microscopy and Olympus VSI/ETS
+- Shared source adapters support paths, buffers, file objects and remote URLs.
+  Indexed readers use checked range requests, covering caches and adaptive
+  read-ahead; whole-stream codecs retain their documented eager input paths.
+- Multi-frame AVIF, animated WebP and GIF decode to a frame stack; HEIF
+  exposes every top-level image
 - Compression backend auto-detect (libdeflate → zlib-ng-compat → stdlib)
-- Cloud I/O primitives (`HTTPDataSource` with covering-cache + adaptive
-  read-ahead) wired into TIFF / HDF5 / DICOMweb / CZI / FITS / Zarr v3
-  readers
 - `tifffile_patch` opt-in shim reroutes tifffile's codec dispatch through
   opencodecs for users who want only a partial swap
+- `capabilities.toml` records what each codec actually supports, checked
+  against the built extensions, with no open gaps
 
 Deferred work (see [`docs/TODO_DEFERRED.md`](docs/TODO_DEFERRED.md)):
 
-- **Windows wheels currently miss `_sz3`, `_pcodec`, `_sperr`, `_brunsli`**
-  — toolchain mismatch (conda's bash picks GCC over MSVC for CMake);
-  v0.1.2 will restore them. macOS + Linux wheels have the full set.
-- CCITT Fax3/Fax4 encode — legacy fax; zero scientific users
-- JPEG-XR — abandoned format outside niche DICOM
-- libspng `filter_sum` SIMD — off the bench-tracked workload (`h2h_png_4mp_rgb`
-  is at 1.14× already); filter-bound PNG-encode users could see another 2-3×
+- CCITT Fax3/Fax4 encode: legacy fax, zero scientific users
+- JPEG XR encode (decode exists for CZI; see above)
+
+GIF and JPEG XL destination writers now drain encoded frame output before close.
+The shared pipeline also supplies bounded independent-piece scheduling, seven
+stateful byte-codec iterators, native PNG row sessions, selective native sources,
+and checked caller-owned destinations.
+
+Shared streaming, buffering, and overlap mechanisms and remaining integration
+work are documented in [the pipeline catalog](docs/shared_pipeline_capabilities.md).
+
+See [CHANGES.rst](CHANGES.rst) for release history.
 
 ## License
 

{opencodecs-0.2.0 → opencodecs-0.3.0}/THIRD-PARTY.md

@@ -165,7 +165,7 @@ each library.
 | libaec | 1.1.6 | BSD-2-Clause |
 | lerc | 4.1.0 | Apache-2.0 |
 | zfp | 1.0.1 | BSD-3-Clause |
-| SZ3 | 3.3.1 | BSD-3-Clause |
+| SZ3 | 3.3.2 | BSD-3-Clause |
 | SPERR | 0.8.5 | Apache-2.0 |
 | pcodec | 1.0.2 | Apache-2.0 |
 | brunsli | master | MIT |

{opencodecs-0.2.0 → opencodecs-0.3.0}/pyproject.toml

@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "opencodecs"
-version = "0.2.0"
+version = "0.3.0"
 description = "Streaming, network-aware image codecs for scientific imaging (prototype: JPEG XL)"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -194,7 +194,7 @@ before-all = [
     # bench/build_codec_libs.sh into the cached /cibw-jxl-prefix mount.
     # Skips encoders (aom, x265) — wheel ships decode-only AVIF/HEIF to
     # keep build < 12 min cold + avoid GPL x265.
-    "bash -c 'PATH=$HOME/.cargo/bin:$PATH bash bench/build_codec_libs.sh --only=libjpeg-turbo,c-blosc2,libde265,dav1d,libavif,libheif,libaec,lerc,zfp,SZ3,pcodec,SPERR,brunsli,mozjpeg,libultrahdr,openjph,CharLS'",
+    "bash -c 'PATH=$HOME/.cargo/bin:$PATH bash bench/build_codec_libs.sh --only=libjpeg-turbo,c-blosc2,libde265,dav1d,libavif,libheif,libaec,lerc,zfp,SZ3,pcodec,SPERR,brunsli,mozjpeg,libultrahdr,openjph,CharLS,jxrlib'",
     "bash bench/build_libjxl.sh",
 ]
 # Don't set OPENCODECS_*_PREFIX here — build_wheels.yml's
@@ -215,7 +215,7 @@ before-all = [
     # only, not libturbojpeg, so the _mozjpeg probe couldn't find what it
     # needed. Source-build via build_codec_libs.sh produces both libs.
     "brew install jpeg-xl libavif libheif webp libpng jpeg-turbo openjpeg zstd brotli c-blosc2 lz4 libdeflate libaec liblerc zfp rust cmake ninja libultrahdr giflib snappy charls isa-l",
-    "bash bench/build_codec_libs.sh --only=SZ3,pcodec,SPERR,brunsli,mozjpeg,openjph",
+    "bash bench/build_codec_libs.sh --only=SZ3,pcodec,SPERR,brunsli,mozjpeg,openjph,jxrlib",
 ]
 # macos-latest is now macOS 15 (Sequoia) on arm64. Homebrew formulas are
 # compiled for 15.x, so any wheel target lower than 15 fails the delocate

{opencodecs-0.2.0 → opencodecs-0.3.0}/setup.py

@@ -1362,6 +1362,10 @@ extensions = [
     Extension(
         name="opencodecs.codecs._zstd",
         sources=["src/opencodecs/codecs/_zstd.pyx"],
+        # The fused decode calls run the byte-plane loop in byteplanes.h,
+        # whose restrict pointers let clang vectorize it; gcc also needs
+        # -O3, as conda's default -O2 leaves it scalar (see _bytetools).
+        extra_compile_args=["-O3"] if sys.platform != "win32" else ["/O2"],
         include_dirs=[
             str(PKG_CODECS),
             *_resolve_include_dirs("zstd.h"),
@@ -1661,6 +1665,32 @@ extensions = [
         define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
         language="c",
     ),
+    # JPEG XR decode via jxrlib, for CZI sub-blocks with compression 4
+    # (most Zeiss whole-slide scans). jxrlib installs its headers under
+    # include/jxrlib/ and needs __ANSI__ off Windows; the shim declares
+    # the in-memory stream constructor its public headers leave out.
+    Extension(
+        name="opencodecs.codecs._jpegxr",
+        sources=[
+            "src/opencodecs/codecs/_jpegxr.pyx",
+            "src/opencodecs/codecs/jpegxr_shim.c",
+        ],
+        include_dirs=[
+            str(PKG_CODECS),
+            numpy.get_include(),
+            *[str(Path(d) / "jxrlib")
+              for d in _resolve_include_dirs("jxrlib/JXRGlue.h")],
+        ],
+        library_dirs=_lib_dirs_for_probes(),
+        # conda-forge on Windows ships static libjxrglue.lib / libjpegxr.lib.
+        libraries=[_libname("jxrglue", "libjxrglue"), _libname("jpegxr", "libjpegxr")],
+        define_macros=[
+            ("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION"),
+            *([] if sys.platform == "win32" else [("__ANSI__", "1")]),
+            ("DISABLE_PERF_MEASUREMENT", "1"),
+        ],
+        language="c",
+    ),
     # JPEG-2000 via OpenJPEG. The header lives in a versioned subdir on
     # most platforms (openjpeg-2.5/), but conda-forge installs it directly
     # at <prefix>/Library/include/openjpeg.h. Probe both layouts.
@@ -1721,6 +1751,13 @@ extensions = [
         name="opencodecs.codecs._bytetools",
         sources=["src/opencodecs/codecs/_bytetools.pyx"],
         include_dirs=[str(PKG_CODECS)],
+        # Byte-plane shuffles are strided byte loops that only pay when the
+        # compiler vectorizes them. conda's default CFLAGS build at -O2,
+        # where gcc's cheap-cost vectorizer leaves them scalar: measured on
+        # x86-64 Linux, the native encode was 1.4-1.7x SLOWER than a
+        # NumPy transpose at -O2. clang vectorizes at -O2 (arm64 macOS
+        # measured 6-10x faster than NumPy), so this only moves gcc.
+        extra_compile_args=["-O3"] if sys.platform != "win32" else ["/O2"],
         language="c",
     ),
     # Native TIFF IFD walker — pure-Python parsing logic in Cython for
@@ -2104,6 +2141,7 @@ _REQUIRED_HEADERS = {
     "opencodecs.codecs._uhdr":   ("ultrahdr_api.h",),
     "opencodecs.codecs._isal":   ("isa-l/igzip_lib.h",),
     "opencodecs.codecs._webp":   ("webp/encode.h",),
+    "opencodecs.codecs._jpegxr": ("jxrlib/JXRGlue.h",),
     "opencodecs.codecs._jpeg2k": ("openjpeg-2.5/openjpeg.h", "openjpeg-2.4/openjpeg.h"),
     "opencodecs.codecs._avif":   ("avif/avif.h",),
     "opencodecs.codecs._heif":   ("libheif/heif.h",),

{opencodecs-0.2.0 → opencodecs-0.3.0}/src/opencodecs/__init__.py

@@ -342,4 +342,4 @@ __all__ = [
     "CziWriter", "CziPyramidWriter",
 ]
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"

{opencodecs-0.2.0 → opencodecs-0.3.0}/src/opencodecs/_aec_codec.py

@@ -29,6 +29,7 @@ from typing import Any
 import numpy as np
 
 from .core.codec import Codec
+from .core.buffers import byte_output
 from .core._io_helpers import read_src as _read_src, write_dest as _write_dest
 from .core._optional_backend import import_or_stubs
 
@@ -126,8 +127,10 @@ class AecCodec(Codec):
         )
         return _write_dest(out, dest)
 
-    def decode(self, src: Any, **opts) -> bytes:
-        return _aec_decode(_read_src(src))
+    def decode(self, src: Any, *, out=None, **opts) -> bytes | memoryview:
+        if out is None:
+            return _aec_decode(_read_src(src))
+        return _aec_decode(_read_src(src), out=byte_output(out))
 
 
 __all__ = ["AecCodec"]