opencodecs 0.5.0__tar.gz → 0.7.0__tar.gz

{opencodecs-0.5.0 → opencodecs-0.7.0}/CHANGES.rst

@@ -10,6 +10,135 @@ Versions follow the same ``YYYY.M.D`` cadence as upstream when we
 publish; the entries below cluster work by date rather than by
 release because most of it has shipped continuously to ``main``.
 
+0.7.0 (2026-10-03)
+------------------
+
+``czi_recompress`` takes ``recompress_when``, a predicate over the reader's
+``CziSubBlockEntry``. Sub-blocks it accepts are decoded and re-encoded as
+before; sub-blocks it rejects are carried across BYTE FOR BYTE, keeping their
+original compression. The default stays ``None``, which re-encodes everything,
+so existing callers are unaffected. The returned summary gains
+``recompressed`` and ``carried`` counts.
+
+This exists because a sub-block's compression is not a detail to be
+normalized away. Re-encoding one that is already compressed is wasted work,
+and re-encoding a LOSSY one is destructive: the pixels are decoded from a
+lossy encoding and written back losslessly, so the file grows and the
+artifacts become permanent. Measured on a 505 MB Axioscan slide scan of 481
+JPEG XR sub-blocks:
+
+====================================  ========  ==========  ============
+strategy                              time      output      vs source
+====================================  ========  ==========  ============
+carry JPEG XR across (predicate)      0.36 s    489 MB      0.97x
+re-encode everything (no predicate)   12.6 s    1309 MB     2.59x
+====================================  ========  ==========  ============
+
+So the predicate is not a speed knob. Without it that file more than doubles
+in size, takes 33x longer, and loses the option of ever recovering the
+original encoding. This is what ZEISS's ``czicompress`` expresses with
+``--strategy``, and the three strategies map directly::
+
+    uncompressed        lambda e: e.compression == 0
+    uncompressed+zstd   lambda e: e.compression in (0, 5, 6)
+    all                 None
+
+Supporting this needed a writer that can emit a sub-block it cannot itself
+produce. ``_build_subblock`` gained ``encoded=``, which writes the payload
+exactly as given and records the compression code as-is, with no encode step
+and no look at the pixels. That is the only path by which a JPEG XR sub-block
+can reach a file this writer wrote, and it is why carrying one across cannot
+alter it. Verification is refused on that path: nothing was encoded, so there
+is nothing to check. ``verify=True`` therefore applies only to the sub-blocks
+that were re-encoded.
+
+0.6.0 (2026-10-03)
+------------------
+
+``czi_recompress`` takes an optional ``should_continue``: a zero-argument
+callable polled once per sub-block, before that sub-block is read. When it
+returns false the rewrite raises the new ``CziCancelled`` and the partial
+output is removed. Existing callers are unaffected; the default is ``None``.
+
+It is a plain callable rather than a ``threading.Event`` so a caller can pass
+a deadline, a signal flag or a cancellation token just as easily, and nothing
+in the writer needs to know which. ``CziCancelled`` subclasses
+``CziWriterError``, so code already catching that keeps working while code
+that cares can tell "the user stopped it" from "the file is broken" without
+matching on message text.
+
+Why it exists: a caller replacing a subprocess (ZEISS's ``czicompress``) with
+this function loses the ability to kill a process mid-file, and a Cancel
+button that only takes effect between files is a visible regression on a
+slide scan. Polling per sub-block takes effect within roughly ``workers``
+sub-blocks, which on a 481-sub-block scan is sub-second.
+
+An interrupted rewrite now removes ``dst`` on any exception, not only
+cancellation -- a failed verification or a ``KeyboardInterrupt`` too. A CZI
+missing sub-blocks still has a valid header and directory, so readers accept
+it and the loss is silent; truncated-but-plausible is worse than absent.
+
+CZI reading
+~~~~~~~~~~~
+
+- ``CziReader(path, index=...)`` reads pixels from a payload index (offsets,
+  sizes, compressions, plane shape, pixel type) without reading the file
+  header, the directory or any sub-block header, and ``payload_index()``
+  returns one from an open reader. A caller that stores the index with its
+  own metadata reads planes with no metadata round trips over a network
+  share.
+- ``read(indices=[...])`` reads chosen sub-blocks in parallel into one
+  stack. ``attachments()`` and ``read_attachment(name)`` return embedded
+  files (Thumbnail, TimeStamps, Label ...) exactly as stored.
+- Files on disk are read with positional reads instead of a memory map: a
+  cold plane off a network share was paged in a cluster at a time, and one
+  exact-range read (split into concurrent pieces of about 1 MiB when only a
+  few planes are read) fetches it in a fraction of the time. The file
+  header is fetched when first needed, the directory comes in one request,
+  and a sub-block's header and payload in one. The directory and metadata
+  are read on first use, so a corrupt directory now raises ``CziError`` on
+  first use rather than in the constructor.
+- Uncompressed stacks are read on several threads. They were split into 8
+  MiB tasks, the size zstd reads use, so nine 2 MiB planes made three
+  tasks, which the worker policy ran on one thread; they now go one plane
+  per 2 MiB task, and two tasks are enough to share. Warm, that stack reads
+  1.63x faster than a reader copying from an mmap on 128-core Linux, where
+  it had been 0.59x as fast (1.51x and 0.88x on a 20-core Mac). Slide
+  regions that need two or three tiles decode them in parallel for the same
+  reason.
+- JPEG XR (compression 4, most slide scans) decodes through the copy of
+  jxrlib maintained inside ZEISS's libCZI (BSD-2-Clause, Microsoft), which
+  ZEISS reworked for speed. No compiler flags brought the upstream 2019.10.9
+  release within 10% of aicspylibczi on x86-64 Linux; with libCZI's copy a
+  slide region read inside one tile takes 0.99x aicspylibczi's time there
+  and is 1.08x faster on a 20-core Mac (upstream: 0.90x and 0.93x). The
+  wheels build it static, from a pinned libCZI commit, with ``NDEBUG``
+  (upstream's Makefile left 68 ``assert()`` calls in the decode loop) and
+  link-time optimization. A jxrlib built by ``bench/build_codec_libs.sh``
+  is now linked by path, so a system jxrlib earlier on the library path
+  can no longer replace it. Upstream jxrlib (Windows wheels, distribution
+  packages) still works.
+- The hi/lo byte unshuffle of zstd sub-blocks uses a byte loop for 2-byte
+  pixels under GCC and Clang, which GCC vectorizes better: 1.46x faster for
+  a 1000 x 1000 plane and 1.11x for 2000 x 2000 on x86-64 Linux. Clang ties;
+  MSVC keeps the word form.
+- Payload buffers are shared and reused, and the read pool holds at most
+  the 32 threads one read uses (it was twice the CPU count).
+
+TIFF reading
+~~~~~~~~~~~~
+
+- A multi-page TIFF decodes each page straight into its slice of one
+  output, instead of decoding pages separately and stacking them, a second
+  copy of the whole stack. A stack stored as plain pixels, end to end in
+  the file (what writers produce for a contiguous series), is read as one
+  span by parallel positioned reads: a 23-page 2000 x 2000 uint16 stack in
+  10.8 ms where tifffile takes 33.8 ms (128-core Linux, warm); page by page
+  it took 59.6 ms. ``TiffPage.asarray`` takes ``out=``.
+- Decode workers are sized by whichever asks for more: 1 MiB of output each,
+  or 256 KiB of compressed input each. Output alone held a deflate image
+  with 6.8 MB of compressed data in 31 strips to 7 workers.
+
 0.5.0 (2026-10-01)
 ------------------
 

{opencodecs-0.5.0/src/opencodecs.egg-info → opencodecs-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: opencodecs
-Version: 0.5.0
+Version: 0.7.0
 Summary: Streaming, network-aware image codecs for scientific imaging (prototype: JPEG XL)
 Author: Kevin Cutler
 License-Expression: BSD-3-Clause

{opencodecs-0.5.0 → opencodecs-0.7.0}/pyproject.toml

@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "opencodecs"
-version = "0.5.0"
+version = "0.7.0"
 description = "Streaming, network-aware image codecs for scientific imaging (prototype: JPEG XL)"
 readme = "README.md"
 requires-python = ">=3.10"

{opencodecs-0.5.0 → opencodecs-0.7.0}/setup.py

@@ -450,6 +450,13 @@ _PROBE_PREFIXES: list[Path] = [
 # ~/Library/Caches/opencodecs/<lib>/ via bench/build_codec_libs.sh.
 # Probe these so the generic header/lib search picks them up.
 _OC_USER_CACHE = _user_cache()
+
+# jxrlib as bench/build_codec_libs.sh builds it: static libraries and the
+# headers behind include/jxrlib/JXRGlue.h, in the per-user cache prefix.
+_JXR_BUILT = _OC_USER_CACHE / "libs"
+if not all((_JXR_BUILT / f).is_file() for f in (
+        "lib/libjpegxr.a", "lib/libjxrglue.a", "include/jxrlib/JXRGlue.h")):
+    _JXR_BUILT = None
 for _libdir in (
     "sz3", "pcodec", "sperr", "brunsli", "lerc", "zstd", "brotli", "giflib",
     # zfp: brew's bottle is built without -march tuning and is ~17%
@@ -1412,6 +1419,11 @@ extensions = [
     # (most Zeiss whole-slide scans). jxrlib installs its headers under
     # include/jxrlib/ and needs __ANSI__ off Windows; the shim declares
     # the in-memory stream constructor its public headers leave out.
+    # The copy bench/build_codec_libs.sh builds (libCZI's, static) is
+    # linked by path, with its headers first: by -l, a system jxrlib in
+    # an earlier library directory (Homebrew's) won the link while the
+    # headers came from the build, leaving the extension calling names
+    # nothing defines.
     Extension(
         name="opencodecs.codecs._jpegxr",
         sources=[
@@ -1421,17 +1433,25 @@ extensions = [
         include_dirs=[
             str(PKG_CODECS),
             numpy.get_include(),
+            *([str(_JXR_BUILT / "include" / "jxrlib")] if _JXR_BUILT else []),
             *[str(Path(d) / "jxrlib")
               for d in _resolve_include_dirs("jxrlib/JXRGlue.h")],
         ],
-        library_dirs=_lib_dirs_for_probes(),
+        library_dirs=[] if _JXR_BUILT else _lib_dirs_for_probes(),
         # conda-forge on Windows ships static libjxrglue.lib / libjpegxr.lib.
-        libraries=[_lib_name("jxrglue", "libjxrglue"), _lib_name("jpegxr", "libjpegxr")],
+        libraries=[] if _JXR_BUILT else [_lib_name("jxrglue", "libjxrglue"),
+                                         _lib_name("jpegxr", "libjpegxr")],
+        extra_objects=[str(_JXR_BUILT / "lib" / "libjxrglue.a"),
+                       str(_JXR_BUILT / "lib" / "libjpegxr.a")] if _JXR_BUILT else [],
         define_macros=[
             ("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION"),
             *([] if _WINDOWS else [("__ANSI__", "1")]),
             ("DISABLE_PERF_MEASUREMENT", "1"),
         ],
+        # bench/build_codec_libs.sh builds jxrlib with -flto; linking with it
+        # too lets the decoder inline across the library (37.7 against
+        # 39.4 ms per slide tile). Harmless with a system jxrlib.
+        extra_link_args=[] if _WINDOWS else ["-flto"],
         language="c",
     ),
     # JPEG-2000 via OpenJPEG. The header lives in a versioned subdir on

{opencodecs-0.5.0 → opencodecs-0.7.0}/src/opencodecs/__init__.py

@@ -70,7 +70,7 @@ from ._fits import FitsStream, FitsHDU, imread as fits_imread
 from ._rgbe import encode as rgbe_encode, decode as rgbe_decode, \
     imread as rgbe_imread, imwrite as rgbe_imwrite
 from ._czi_reader import CziPyramidReader
-from ._czi_writer import (CziWriter, CziPyramidWriter,
+from ._czi_writer import (CziWriter, CziPyramidWriter, CziCancelled,
                           czi_recompress, subblock_dims)
 from ._tiff_pyramid import TiffPyramidReader
 from ._tiff_http import HTTPDataSource, FileDataSource
@@ -345,7 +345,8 @@ __all__ = [
     "FitsStream", "FitsHDU", "fits_imread",
     "rgbe_encode", "rgbe_decode", "rgbe_imread", "rgbe_imwrite",
     "CziPyramidReader",
-    "CziWriter", "CziPyramidWriter", "czi_recompress", "subblock_dims",
+    "CziWriter", "CziPyramidWriter", "CziCancelled",
+    "czi_recompress", "subblock_dims",
 ]
 
-__version__ = "0.5.0"
+__version__ = "0.7.0"