large-image-source-zarr 1.28.2.dev12__tar.gz → 1.30.7.dev12__tar.gz

{large-image-source-zarr-1.28.2.dev12 → large_image_source_zarr-1.30.7.dev12}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: large-image-source-zarr
-Version: 1.28.2.dev12
+Version: 1.30.7.dev12
 Summary: A OME Zarr tilesource for large_image.
 Home-page: https://github.com/girder/large_image
 Author: Kitware, Inc.
@@ -15,13 +15,16 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.6
+Description-Content-Type: text/x-rst
 License-File: LICENSE
-Requires-Dist: large-image>=1.28.2.dev12
-Requires-Dist: zarr
-Requires-Dist: imagecodecs-numcodecs
+Requires-Dist: large-image>=1.30.7.dev12
+Requires-Dist: zarr<3
+Requires-Dist: numcodecs
+Requires-Dist: imagecodecs-numcodecs!=2024.9.22
 Provides-Extra: girder
-Requires-Dist: girder-large-image>=1.28.2.dev12; extra == "girder"
+Requires-Dist: girder-large-image>=1.30.7.dev12; extra == "girder"
 Provides-Extra: all
 Requires-Dist: large-image-converter; extra == "all"
 

{large-image-source-zarr-1.28.2.dev12 → large_image_source_zarr-1.30.7.dev12}/README.rst

@@ -15,7 +15,7 @@ Large Image
    :target: https://codecov.io/github/girder/large_image?branch=master
    :alt: codecov.io
 
-.. |doi-badge| image:: https://img.shields.io/badge/DOI-10.5281%2Fzenodo.4723355-blue
+.. |doi-badge| image:: https://img.shields.io/badge/DOI-10.5281%2Fzenodo.4723355-blue.svg
    :target: https://zenodo.org/badge/latestdoi/45569214
 
 .. |pypi-badge| image:: https://img.shields.io/pypi/v/large-image.svg?logo=python&logoColor=white
@@ -39,7 +39,7 @@ Highlights
 Installation
 ------------
 
-In addition to installing the ``large-image`` package, you'll need at least one tile source (a ``large-image-source-xxx`` package).   You can install everything from the main project with one of these commands:
+In addition to installing the base ``large-image`` package, you'll need at least one tile source which corresponds to your target file format(s) (a ``large-image-source-xxx`` package).   You can install everything from the main project with one of these commands:
 
 Pip
 ~~~
@@ -52,7 +52,7 @@ Install all tile sources on linux::
 
     pip install large-image[all] --find-links https://girder.github.io/large_image_wheels
 
-Install all tile sources and all Girder plugins on linux::
+When using large-image with an instance of `Girder`_, install all tile sources and all Girder plugins on linux::
 
     pip install large-image[all] girder-large-image-annotation[tasks] --find-links https://girder.github.io/large_image_wheels
 
@@ -60,8 +60,9 @@ Install all tile sources and all Girder plugins on linux::
 Conda
 ~~~~~
 
-Conda makes dependency management a bit easier if not on Linux. Some of the source modules are available on conda-forge. You can install the following::
+Conda makes dependency management a bit easier if not on Linux. The base module, converter module, and two of the source modules are available on conda-forge. You can install the following::
 
+    conda install -c conda-forge large-image
     conda install -c conda-forge large-image-source-gdal
     conda install -c conda-forge large-image-source-tiff
     conda install -c conda-forge large-image-converter
@@ -118,7 +119,7 @@ Large Image consists of several Python modules designed to work together.  These
 
   - ``large-image-source-deepzoom``: A tile source for reading Deepzoom tiles.
 
-  - ``large-image-source-dicom``: A tile source for reading DICOM WSI images.
+  - ``large-image-source-dicom``: A tile source for reading DICOM Whole Slide Images (WSI).
 
   - ``large-image-source-gdal``: A tile source for reading geotiff files via GDAL.  This handles source data with more complex transforms than the mapnik tile source.
 
@@ -128,62 +129,41 @@ Large Image consists of several Python modules designed to work together.  These
 
   - ``large-image-source-nd2``: A tile source for reading nd2 (NIS Element) images.
 
-  - ``large-image-source-ometiff``: A tile source using the tiff library that can handle some multi-frame OMETiff files.
+  - ``large-image-source-ometiff``: A tile source using the tiff library that can handle most multi-frame OMETiff files that are compliant with the specification.
 
   - ``large-image-source-openjpeg``: A tile source using the Glymur library to read jp2 (JPEG 2000) files.
 
   - ``large-image-source-openslide``: A tile source using the OpenSlide library.  This works with svs, ndpi, Mirax, tiff, vms, and other file formats.
 
-  - ``large-image-source-pil``: A tile source for small images via the Python Imaging Library (Pillow).
+  - ``large-image-source-pil``: A tile source for small images via the Python Imaging Library (Pillow). By default, the maximum size is 4096, but the maximum size can be configured.
 
   - ``large-image-source-tiff``: A tile source for reading pyramidal tiff files in common compression formats.
 
   - ``large-image-source-tifffile``: A tile source using the tifffile library that can handle a wide variety of tiff-like files.
 
-  - ``large-image-source-vips``: A tile source for reading any files handled by libvips.  This also can be used for writing tiled images from numpy arrays.
+  - ``large-image-source-vips``: A tile source for reading any files handled by libvips.  This also can be used for writing tiled images from numpy arrays (up to 4 dimensions).
 
-  - ``large-image-source-zarr``: A tile source using the zarr library that can handle OME-Zarr (OME-NGFF) files as well as some other zarr files.
+  - ``large-image-source-zarr``: A tile source using the zarr library that can handle OME-Zarr (OME-NGFF) files as well as some other zarr files. This can also be used for writing N-dimensional tiled images from numpy arrays. Written images can be saved as any supported format.
 
   - ``large-image-source-test``: A tile source that generates test tiles, including a simple fractal pattern.  Useful for testing extreme zoom levels.
 
-  - ``large-image-source-dummy``: A tile source that does nothing.
+  - ``large-image-source-dummy``: A tile source that does nothing. This is an absolutely minimal implementation of a tile source used for testing. If you want to create a custom tile source, start with this implementation.
 
-  Most tile sources can be used with girder-large-image.  You can specific an extras_require of ``girder`` to include ``girder-large-image`` with the source.
 
-- As a Girder plugin:
+As a `Girder`_ plugin, ``large-image`` adds end points to access all of the image formats it can read both to get metadata and to act as a tile server.
+In the Girder UI, ``large-image`` shows images on item pages, and can show thumbnails in item lists when browsing folders.
+There is also cache management to balance memory use and speed of response in Girder when ``large-image`` is used as a tile server.
 
-  - ``girder-large-image``: Large Image as a Girder_ 3.x plugin.
-    You can specify extras_require of ``tasks`` to install a Girder Worker task that can convert otherwise unreadable images to pyramidal tiff files.
+Most tile sources can be used with Girder Large Image.  You can specify an extras_require of ``girder`` to install the following packages:
 
-  - ``girder-large-image-annotation``: Annotations for large images as a Girder_ 3.x plugin.
+  - ``girder-large-image``: Large Image as a Girder 3.x plugin.
+    You can install ``large-image[tasks]`` to install a Girder Worker task that can convert otherwise unreadable images to pyramidal tiff files.
+
+  - ``girder-large-image-annotation``: Adds models to the Girder database for supporting annotating large images.  These annotations can be rendered on images. Annotations can include polygons, points, image overlays, and other types. Each annotation can have a label and metadata.
 
   - ``large-image-tasks``: A utility for running the converter via Girder Worker.
     You can specify an extras_require of ``girder`` to include modules needed to work with the Girder remote worker or ``worker`` to include modules needed on the remote side of the Girder remote worker.  If neither is specified, some conversion tasks can be run using Girder local jobs.
 
 
-Developer Installation
-----------------------
-
-To install all packages from source, clone the repository::
-
-    git clone https://github.com/girder/large_image.git
-    cd large_image
-
-Install all packages and dependencies::
-
-    pip install -e . -r requirements-dev.txt
-
-If you aren't developing with Girder 3, you can skip installing those components.  Use ``requirements-dev-core.txt`` instead of ``requirements-dev.txt``::
-
-    pip install -e . -r requirements-dev-core.txt
-
-
-Tile source prerequisites
-=========================
-
-Many tile sources have complex prerequisites.  These can be installed directly using your system's package manager or from some prebuilt Python wheels for Linux.  The prebuilt wheels are not official packages, but they can be used by instructing pip to use them by preference::
-
-    pip install -e . -r requirements-dev.txt --find-links https://girder.github.io/large_image_wheels
-
 
-.. _Girder: https://github.com/girder/girder
+.. _Girder: https://girder.readthedocs.io/en/latest/

{large-image-source-zarr-1.28.2.dev12 → large_image_source_zarr-1.30.7.dev12}/large_image_source_zarr/__init__.py

@@ -1,16 +1,17 @@
 import math
+import multiprocessing
 import os
 import shutil
 import tempfile
 import threading
 import uuid
+import warnings
 from importlib.metadata import PackageNotFoundError
 from importlib.metadata import version as _importlib_version
 from pathlib import Path
 
 import numpy as np
 import packaging.version
-import zarr
 
 import large_image
 from large_image.cache_util import LruCacheMetaclass, methodcache
@@ -27,6 +28,26 @@ except PackageNotFoundError:
     pass
 
 
+zarr = None
+
+
+def _lazyImport():
+    """
+    Import the zarr module.  This is done when needed rather than in the module
+    initialization because it is slow.
+    """
+    global zarr
+
+    if zarr is None:
+        try:
+            import zarr
+
+            warnings.filterwarnings('ignore', category=FutureWarning, module='.*zarr.*')
+        except ImportError:
+            msg = 'zarr module not found.'
+            raise TileSourceError(msg)
+
+
 class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
     """
     Provides tile access to files that the zarr library can read.
@@ -66,6 +87,7 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         """
         super().__init__(path, **kwargs)
 
+        _lazyImport()
         if str(path).startswith(NEW_IMAGE_PATH_FLAG):
             self._initNew(path, **kwargs)
         else:
@@ -75,6 +97,10 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
     def _initOpen(self, **kwargs):
         self._largeImagePath = str(self._getLargeImagePath())
         self._zarr = None
+        self._editable = False
+        self._frameValues = None
+        self._frameAxes = None
+        self._frameUnits = None
         if not os.path.isfile(self._largeImagePath) and '//:' not in self._largeImagePath:
             raise TileSourceFileNotFoundError(self._largeImagePath) from None
         try:
@@ -105,11 +131,12 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         """
         Initialize the tile class for creating a new image.
         """
-        self._tempdir = tempfile.TemporaryDirectory(path)
-        self._zarr_store = zarr.DirectoryStore(self._tempdir.name)
-        self._zarr = zarr.open(self._zarr_store, mode='w')
-        # Make unpickleable
-        self._unpickleable = True
+        self._tempdir = Path(tempfile.gettempdir(), path)
+        self._created = False
+        if not self._tempdir.exists():
+            self._created = True
+        self._zarr_store = zarr.DirectoryStore(str(self._tempdir))
+        self._zarr = zarr.open(self._zarr_store, mode='a')
         self._largeImagePath = None
         self._dims = {}
         self.sizeX = self.sizeY = self.levels = 0
@@ -118,11 +145,24 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         self._output = None
         self._editable = True
         self._bandRanges = None
-        self._addLock = threading.RLock()
+        self._threadLock = threading.RLock()
+        self._processLock = multiprocessing.Lock()
         self._framecount = 0
         self._mm_x = 0
         self._mm_y = 0
+        self._channelNames = []
+        self._channelColors = []
+        self._imageDescription = None
         self._levels = []
+        self._associatedImages = {}
+        self._frameValues = None
+        self._frameAxes = None
+        self._frameUnits = None
+        if not self._created:
+            try:
+                self._validateZarr()
+            except Exception:
+                pass
 
     def __del__(self):
         if not hasattr(self, '_derivedSource'):
@@ -131,7 +171,8 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
             except Exception:
                 pass
             try:
-                self._tempdir.cleanup()
+                if self._created:
+                    shutil.rmtree(self._tempdir)
             except Exception:
                 pass
 
@@ -188,10 +229,10 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
             tuple that is used to find the maximum size array, preferring ome
             arrays, then total pixels, then channels.  'is_ome' is a boolean.
             'series' is a list of the found groups and arrays that match the
-            best criteria.  'axes' and 'channels' are from the best array.
-            'associated' is a list of all groups and arrays that might be
-            associated images.  These have to be culled for the actual groups
-            used in the series.
+            best criteria.  'axes', 'axes_values', 'axes_units', and 'channels'
+            are from the best array. 'associated' is a list of all groups and
+            arrays that might be associated images.  These have to be culled
+            for the actual groups used in the series.
         """
         attrs = group.attrs.asdict() if group is not None else {}
         min_version = packaging.version.Version('0.4')
@@ -203,9 +244,19 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
             all(packaging.version.Version(m['version']) >= min_version
                 for m in attrs['multiscales'] if 'version' in m))
         channels = None
+        axes_values = None
+        axes_units = None
         if is_ome:
             axes = {axis['name']: idx for idx, axis in enumerate(
                 attrs['multiscales'][0]['axes'])}
+            axes_values = {
+                axis['name']: axis.get('values')
+                for axis in attrs['multiscales'][0]['axes']
+            }
+            axes_units = {
+                axis['name']: axis.get('unit')
+                for axis in attrs['multiscales'][0]['axes']
+            }
             if isinstance(attrs['omero'].get('channels'), list):
                 channels = [channel['label'] for channel in attrs['omero']['channels']]
                 if all(channel.startswith('Channel ') for channel in channels):
@@ -224,6 +275,8 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
             results['series'] = [(group, arr)]
             results['is_ome'] = is_ome
             results['axes'] = axes
+            results['axes_values'] = axes_values
+            results['axes_units'] = axes_units
             results['channels'] = channels
         elif check == results['best']:
             results['series'].append((group, arr))
@@ -303,11 +356,53 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         except Exception:
             pass
 
+    def _readFrameValues(self, found, baseArray):
+        """
+        Populate frame values and frame units from image metadata.
+        """
+        axes_values = found.get('axes_values')
+        axes_units = found.get('axes_units')
+        if isinstance(axes_values, dict):
+            self._frameAxes = [
+                a for a, i in sorted(self._axes.items(), key=lambda x: x[1])
+                if axes_values.get(a) is not None
+            ]
+            self._frameUnits = {k: axes_units.get(k) for k in self.frameAxes if k in axes_units}
+            self._frameValues = None
+            frame_values_shape = [baseArray.shape[self._axes[a]] for a in self.frameAxes]
+            frame_values_shape.append(len(frame_values_shape))
+            frame_values = np.empty(frame_values_shape, dtype=object)
+            all_frame_specs = self.getMetadata().get('frames')
+            for axis, values in axes_values.items():
+                if axis in self.frameAxes:
+                    slicing = [slice(None) for i in range(len(frame_values_shape))]
+                    axis_index = self.frameAxes.index(axis)
+                    slicing[-1] = axis_index
+                    if len(values) == frame_values_shape[axis_index]:
+                        # uniform values have same length as axis
+                        for i, value in enumerate(values):
+                            slicing[axis_index] = i
+                            frame_values[tuple(slicing)] = value
+                    elif len(values) == self._framecount:
+                        # non-uniform values have a value for every frame
+                        for i, value in enumerate(values):
+                            if all_frame_specs and len(all_frame_specs) > i:
+                                for k, j in all_frame_specs[i].items():
+                                    if 'Index' in k:
+                                        name = k.replace('Index', '').lower()
+                                        if name:
+                                            slicing[self._frameAxes.index(name)] = j
+                            frame_values[tuple(slicing)] = value
+            if frame_values.size > 0:
+                self._frameValues = frame_values
+
     def _validateZarr(self):
         """
         Validate that we can read tiles from the zarr parent group in
         self._zarr.  Set up the appropriate class variables.
         """
+        if self._editable and hasattr(self, '_axes'):
+            self._writeInternalMetadata()
         found = self._scanZarrGroup(self._zarr)
         if found['best'] is None:
             msg = 'No data array that can be used.'
@@ -315,13 +410,19 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         self._series = found['series']
         baseGroup, baseArray = self._series[0]
         self._is_ome = found['is_ome']
-        self._axes = {k.lower(): v for k, v in found['axes'].items() if baseArray.shape[v] > 1}
+        self._axes = {
+            k.lower(): v
+            for k, v in found['axes'].items()
+            if baseArray.shape[v] > 1 or k in found.get('axes_values', {})
+        }
         if len(self._series) > 1 and 'xy' in self._axes:
             msg = 'Conflicting xy axis data.'
             raise TileSourceError(msg)
         self._channels = found['channels']
-        self._associatedImages = [
-            (g, a) for g, a in found['associated'] if not any(g is gb for gb, _ in self._series)]
+        self._associatedImages = {
+            g.name.replace('/', ''): (g, a)
+            for g, a in found['associated'] if not any(g is gb for gb, _ in self._series)
+        }
         self.sizeX = baseArray.shape[self._axes['x']]
         self.sizeY = baseArray.shape[self._axes['y']]
         self.tileWidth = (
@@ -339,7 +440,7 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         #     self.tileWidth = self.tileHeight = baseArray.chunks[self._axes['x']]
         self.levels = int(max(1, math.ceil(math.log(max(
             self.sizeX / self.tileWidth, self.sizeY / self.tileHeight)) / math.log(2)) + 1))
-        self._dtype = baseArray.dtype
+        self._dtype = np.dtype(baseArray.dtype)
         self._bandCount = 1
         if ('c' in self._axes and 's' not in self._axes and not self._channels and
                 baseArray.shape[self._axes.get('c')] in {1, 3, 4}):
@@ -352,8 +453,10 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         stride = 1
         self._strides = {}
         self._axisCounts = {}
-        for _, k in sorted((-'tzc'.index(k) if k in 'tzc' else 1, k)
-                           for k in self._axes if k not in 'xys'):
+        for _, k in sorted(
+            (-self._axes.get(k, 'tzc'.index(k) if k in 'tzc' else -1), k)
+            for k in self._axes if k not in 'xys'
+        ):
             self._strides[k] = stride
             self._axisCounts[k] = baseArray.shape[self._axes[k]]
             stride *= baseArray.shape[self._axes[k]]
@@ -362,6 +465,7 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
             self._axisCounts['xy'] = len(self._series)
             stride *= len(self._series)
         self._framecount = stride
+        self._readFrameValues(found, baseArray)
 
     def _nonemptyLevelsList(self, frame=0):
         """
@@ -413,6 +517,16 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
                 for axis in self._strides:
                     frame['Index' + axis.upper()] = (
                         idx // self._strides[axis]) % self._axisCounts[axis]
+                if self.frameValues is not None and self.frameAxes is not None:
+                    current_frame_slice = tuple(
+                        frame['Index' + axis.upper()] for axis in self.frameAxes
+                    )
+                    current_frame_values = self.frameValues[current_frame_slice]
+                    for i, axis in enumerate(self.frameAxes):
+                        value = current_frame_values[i]
+                        # ensure that values are returned as native python types
+                        native_value = getattr(value, 'tolist', lambda v=value: v)()
+                        frame['Value' + axis.upper()] = native_value
                 frames.append(frame)
             self._addMetadataFrameInformation(result, getattr(self, '_channels', None))
         return result
@@ -425,6 +539,8 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
 
         :returns: a dictionary of data or None.
         """
+        if self._editable:
+            self._writeInternalMetadata()
         result = {}
         result['zarr'] = {
             'base': self._zarr.attrs.asdict(),
@@ -441,7 +557,7 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
 
         :return: the list of image keys.
         """
-        return [f'image_{idx}' for idx in range(len(self._associatedImages))]
+        return list(self._associatedImages.keys())
 
     def _getAssociatedImage(self, imageKey):
         """
@@ -450,15 +566,9 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         :param imageKey: the key of the associated image.
         :return: the image in PIL format or None.
         """
-        if not imageKey.startswith('image_'):
-            return
-        try:
-            idx = int(imageKey[6:])
-        except Exception:
-            return
-        if idx < 0 or idx >= len(self._associatedImages):
-            return
-        group, arr = self._associatedImages[idx]
+        if imageKey not in self._associatedImages:
+            return None
+        group, arr = self._associatedImages[imageKey]
         axes = self._getGeneralAxes(arr)
         trans = [idx for idx in range(len(arr.shape))
                  if idx not in axes.values()] + [axes['y'], axes['x']]
@@ -490,7 +600,7 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         y1 //= scale
         step //= scale
         if step > 2 ** self._maxSkippedLevels:
-            tile = self._getTileFromEmptyLevel(x, y, z, **kwargs)
+            tile, _format = self._getTileFromEmptyLevel(x, y, z, **kwargs)
             tile = large_image.tilesource.base._imageToNumpy(tile)[0]
         else:
             idx = [slice(None) for _ in arr.shape]
@@ -542,6 +652,50 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
 
         return tile, mask, placement, axes
 
+    def _updateFrameValues(self, frame_values, placement, axes, new_axes, new_dims):
+        self._frameAxes = [
+            a for a in axes
+            if a in frame_values or
+            (self.frameAxes is not None and a in self.frameAxes)
+        ]
+        frames_shape = [new_dims[a] for a in self.frameAxes]
+        frames_shape.append(len(frames_shape))
+        if self.frameValues is None:
+            self._frameValues = np.empty(frames_shape, dtype=object)
+        elif self.frameValues.shape != frames_shape:
+            if len(new_axes):
+                for i in new_axes.values():
+                    self._frameValues = np.expand_dims(self._frameValues, axis=i)
+            frame_padding = [
+                (0, s - self.frameValues.shape[i])
+                for i, s in enumerate(frames_shape)
+            ]
+            frame_padding[-1] = (0, 0)
+            self._frameValues = np.pad(self._frameValues, frame_padding)
+            for i in new_axes.values():
+                self._frameValues = np.insert(
+                    self._frameValues, i, 0, axis=len(frames_shape) - 1,
+                )
+        current_frame_slice = tuple(placement.get(a) for a in self.frameAxes)
+        for i, k in enumerate(self.frameAxes):
+            self.frameValues[(*current_frame_slice, i)] = frame_values.get(k)
+
+    def _resizeImage(self, arr, new_shape, new_axes, chunking):
+        if new_shape != arr.shape:
+            if len(new_axes):
+                for i in new_axes.values():
+                    arr = np.expand_dims(arr, axis=i)
+                arr = np.pad(
+                    arr,
+                    [(0, s - arr.shape[i]) for i, s in enumerate(new_shape)],
+                )
+                new_arr = zarr.empty(new_shape, chunks=chunking, dtype=arr.dtype)
+                new_arr[:] = arr[:]
+                arr = new_arr
+            else:
+                arr.resize(*new_shape)
+        return arr
+
     def addTile(self, tile, x=0, y=0, mask=None, axes=None, **kwargs):
         """
         Add a numpy or image tile to the image, expanding the image as needed
@@ -561,21 +715,32 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         :param kwargs: start locations for any additional axes.  Note that
             ``level`` is a reserved word and not permitted for an axis name.
         """
-        # TODO: improve band bookkeeping
-
         self._checkEditable()
+        try:
+            # read any info written by other processes
+            self._validateZarr()
+        except TileSourceError:
+            pass
+        updateMetadata = False
         store_path = str(kwargs.pop('level', 0))
         placement = {
             'x': x,
             'y': y,
-            **kwargs,
+            **{k: v for k, v in kwargs.items() if not k.endswith('_value')},
+        }
+        frame_values = {
+            k.replace('_value', ''): v for k, v in kwargs.items()
+            if k not in placement
         }
         tile, mask, placement, axes = self._validateNewTile(tile, mask, placement, axes)
 
-        with self._addLock:
+        with self._threadLock and self._processLock:
+            old_axes = self._axes if hasattr(self, '_axes') else {}
             self._axes = {k: i for i, k in enumerate(axes)}
+            new_axes = {k: i for k, i in self._axes.items() if k not in old_axes}
             new_dims = {
                 a: max(
+                    self._axisCounts.get(a, 0) if hasattr(self, '_axisCounts') else 0,
                     self._dims.get(store_path, {}).get(a, 0),
                     placement.get(a, 0) + tile.shape[i],
                 )
@@ -587,6 +752,11 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
                 for i, a in enumerate(axes)
             ])
 
+            if len(frame_values.keys()) > 0:
+                # update self.frameValues
+                updateMetadata = True
+                self._updateFrameValues(frame_values, placement, axes, new_axes, new_dims)
+
             current_arrays = dict(self._zarr.arrays())
             if store_path == '0':
                 # if writing to base data, invalidate generated levels
@@ -603,14 +773,18 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
                 ])
             else:
                 arr = current_arrays[store_path]
-                arr.resize(*tuple(new_dims.values()))
-                if arr.chunks[-1] != new_dims.get('s'):
-                    # rechunk if length of samples axis changes
+                new_shape = tuple(
+                    max(v, arr.shape[old_axes[k]] if k in old_axes else 0)
+                    for k, v in new_dims.items()
+                )
+                if arr.chunks[-1] != new_dims.get('s') or len(new_axes):
+                    # rechunk if length of samples axis changed or any new axis added
                     chunking = tuple([
                         self._tileSize if a in ['x', 'y'] else
                         new_dims.get('s') if a == 's' else 1
                         for a in axes
                     ])
+                arr = self._resizeImage(arr, new_shape, new_axes, chunking)
 
             if mask is not None:
                 arr[placement_slices] = np.where(mask, tile, arr[placement_slices])
@@ -625,25 +799,13 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
                     path=store_path,
                 )
 
-            # If base data changed, update large_image attributes and OME metadata
+            # If base data changed, update large_image attributes
             if store_path == '0':
-                self._zarr.attrs.update({
-                    'multiscales': [{
-                        'version': '0.5-dev',
-                        'axes': [{
-                            'name': a,
-                            'type': 'space' if a in ['x', 'y'] else 'other',
-                        } for a in axes],
-                        'datasets': [{'path': 0}],
-                    }],
-                    'omero': {'version': '0.5-dev'},
-                })
-
-                self._dtype = tile.dtype
+                self._dtype = np.dtype(tile.dtype)
                 self._bandCount = new_dims.get(axes[-1])  # last axis is assumed to be bands
                 self.sizeX = new_dims.get('x')
                 self.sizeY = new_dims.get('y')
-                self._framecount = np.prod([
+                self._framecount = math.prod([
                     length
                     for axis, length in new_dims.items()
                     if axis in axes[:-3]
@@ -652,6 +814,157 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
                 self._levels = None
                 self.levels = int(max(1, math.ceil(math.log(max(
                     self.sizeX / self.tileWidth, self.sizeY / self.tileHeight)) / math.log(2)) + 1))
+                updateMetadata = True
+        if updateMetadata:
+            self._writeInternalMetadata()
+
+    def addAssociatedImage(self, image, imageKey=None):
+        """
+        Add an associated image to this source.
+
+        :param image: a numpy array, PIL Image, or a binary string
+            with an image.  The numpy array can have 2 or 3 dimensions.
+        """
+        data, _ = _imageToNumpy(image)
+        with self._threadLock and self._processLock:
+            if imageKey is None:
+                # Each associated image should be in its own group
+                num_existing = len(self.getAssociatedImagesList())
+                imageKey = f'image_{num_existing + 1}'
+            group = self._zarr.require_group(imageKey)
+            arr = zarr.array(
+                data,
+                store=self._zarr_store,
+                path=f'{imageKey}/image',
+            )
+            self._associatedImages[imageKey] = (group, arr)
+
+    def _getAxisInternalMetadata(self, axis_name):
+        """
+        Get the metadata structure describing an axis in the image.
+        This will be written to the image metadata.
+
+        :param axis_name: a string corresponding to an axis in the image
+        :returns: a dictionary describing the target axis
+        """
+        axis_metadata = {'name': axis_name}
+        if axis_name in ['x', 'y']:
+            axis_metadata['type'] = 'space'
+            axis_metadata['unit'] = 'millimeter'
+        elif axis_name in ['s', 'c']:
+            axis_metadata['type'] = 'channel'
+        if self.frameAxes is not None:
+            axis_index = self.frameAxes.index(axis_name) if axis_name in self.frameAxes else None
+            if axis_index is not None and self.frameValues is not None:
+                all_frame_values = self.frameValues[..., axis_index]
+                split = np.split(
+                    all_frame_values,
+                    all_frame_values.shape[axis_index],
+                    axis=axis_index,
+                )
+                uniform = all(len(np.unique(
+                    # convert all values to strings for mixed type comparison with np.unique
+                    (a[np.not_equal(a, None)]).astype(str),
+                )) == 1 for a in split)
+                if uniform:
+                    values = [a.flat[0] for a in split]
+                else:
+                    values = all_frame_values.flatten().tolist()
+                axis_metadata['values'] = values
+            unit = self.frameUnits.get(axis_name) if self.frameUnits is not None else None
+            if unit is not None:
+                axis_metadata['unit'] = unit
+        return axis_metadata
+
+    def _writeInternalMetadata(self):
+        """
+        Write metadata to Zarr attributes.
+        Metadata structure adheres to OME schema from https://ngff.openmicroscopy.org/latest/
+        """
+        self._checkEditable()
+        with self._threadLock and self._processLock:
+            name = str(self._tempdir.name).split('/')[-1]
+            arrays = dict(self._zarr.arrays())
+            channel_axis = self._axes.get('c', self._axes.get('s'))
+            datasets = []
+            axes = []
+            channels = []
+            rdefs = {'model': 'color' if len(self._channelColors) else 'greyscale'}
+            sorted_axes = [a[0] for a in sorted(self._axes.items(), key=lambda item: item[1])]
+            for arr_name in arrays:
+                level = int(arr_name)
+                scale = [1.0 for a in sorted_axes]
+                scale[self._axes.get('x')] = (self._mm_x or 0) * (2 ** level)
+                scale[self._axes.get('y')] = (self._mm_y or 0) * (2 ** level)
+                dataset_metadata = {
+                    'path': arr_name,
+                    'coordinateTransformations': [{
+                        'type': 'scale',
+                        'scale': scale,
+                    }],
+                }
+                datasets.append(dataset_metadata)
+            for a in sorted_axes:
+                if a == 't':
+                    rdefs['defaultT'] = 0
+                elif a == 'z':
+                    rdefs['defaultZ'] = 0
+                axes.append(self._getAxisInternalMetadata(a))
+            if channel_axis is not None and len(arrays) > 0:
+                base_array = list(arrays.values())[0]
+                base_shape = base_array.shape
+                for c in range(base_shape[channel_axis]):
+                    channel_metadata = {
+                        'active': True,
+                        'coefficient': 1,
+                        'color': 'FFFFFF',
+                        'family': 'linear',
+                        'inverted': False,
+                        'label': f'Band {c + 1}',
+                    }
+                    # slicing = tuple(
+                    #     slice(None)
+                    #     if k != ('c' if 'c' in self._axes else 's')
+                    #     else c
+                    #     for k, v in self._axes.items()
+                    # )
+                    # channel_data = base_array[slicing]
+                    # channel_min = np.min(channel_data)
+                    # channel_max = np.max(channel_data)
+                    # channel_metadata['window'] = {
+                    #     'end': channel_max,
+                    #     'max': channel_max,
+                    #     'min': channel_min,
+                    #     'start': channel_min,
+                    # }
+                    if len(self._channelNames) > c:
+                        channel_metadata['label'] = self._channelNames[c]
+                    if len(self._channelColors) > c:
+                        channel_metadata['color'] = self._channelColors[c]
+                    channels.append(channel_metadata)
+            # Guidelines from https://ngff.openmicroscopy.org/latest/
+            self._zarr.attrs.update({
+                'multiscales': [{
+                    'version': '0.5',
+                    'name': name,
+                    'axes': axes,
+                    'datasets': datasets,
+                    'metadata': {
+                        'description': self._imageDescription or '',
+                        'kwargs': {
+                            'multichannel': (channel_axis is not None),
+                        },
+                    },
+                }],
+                'omero': {
+                    'id': 1,
+                    'version': '0.5',
+                    'name': name,
+                    'channels': channels,
+                    'rdefs': rdefs,
+                },
+                'bioformats2raw.layout': 3,
+            })
 
     @property
     def crop(self):
@@ -678,6 +991,102 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
             raise TileSourceError(msg)
         self._crop = (x, y, w, h)
 
+    @property
+    def mm_x(self):
+        return self._mm_x
+
+    @mm_x.setter
+    def mm_x(self, value):
+        self._checkEditable()
+        value = float(value) if value is not None else None
+        if value is not None and value <= 0:
+            msg = 'mm_x must be positive or None'
+            raise TileSourceError(msg)
+        self._mm_x = value
+
+    @property
+    def mm_y(self):
+        return self._mm_y
+
+    @mm_y.setter
+    def mm_y(self, value):
+        self._checkEditable()
+        value = float(value) if value is not None else None
+        if value is not None and value <= 0:
+            msg = 'mm_y must be positive or None'
+            raise TileSourceError(msg)
+        self._mm_y = value
+
+    @property
+    def imageDescription(self):
+        return self._imageDescription
+
+    @imageDescription.setter
+    def imageDescription(self, description):
+        self._checkEditable()
+        self._imageDescription = description
+
+    @property
+    def channelNames(self):
+        return self._channelNames
+
+    @channelNames.setter
+    def channelNames(self, names):
+        self._checkEditable()
+        self._channelNames = names
+
+    @property
+    def channelColors(self):
+        return self._channelColors
+
+    @channelColors.setter
+    def channelColors(self, colors):
+        self._checkEditable()
+        self._channelColors = colors
+
+    @property
+    def frameAxes(self):
+        return self._frameAxes
+
+    @frameAxes.setter
+    def frameAxes(self, axes):
+        self._checkEditable()
+        self._frameAxes = axes
+        self._writeInternalMetadata()
+
+    @property
+    def frameUnits(self):
+        return self._frameUnits
+
+    @frameUnits.setter
+    def frameUnits(self, units):
+        self._checkEditable()
+        if self.frameAxes is None:
+            err = 'frameAxes must be set first with a list of frame axis names.'
+            raise ValueError(err)
+        if not isinstance(units, dict) or not all(
+            k in self.frameAxes for k in units.keys()
+        ):
+            err = 'frameUnits must be a dictionary with keys that exist in frameAxes.'
+            raise ValueError(err)
+        self._frameUnits = units
+
+    @property
+    def frameValues(self):
+        return self._frameValues
+
+    @frameValues.setter
+    def frameValues(self, a):
+        self._checkEditable()
+        if self.frameAxes is None:
+            err = 'frameAxes must be set first with a list of frame axis names.'
+            raise ValueError(err)
+        if len(a.shape) != len(self.frameAxes) + 1:
+            err = f'frameValues must have {len(self.frameAxes) + 1} dimensions.'
+            raise ValueError(err)
+        self._frameValues = a
+        self._writeInternalMetadata()
+
     def _generateDownsampledLevels(self, resample_method):
         self._checkEditable()
         current_arrays = dict(self._zarr.arrays())
@@ -701,6 +1110,7 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
             width=4096 + tile_overlap['x'],
             height=4096 + tile_overlap['y'],
         )
+        sorted_axes = [a[0] for a in sorted(self._axes.items(), key=lambda item: item[1])]
         for level in range(1, self.levels):
             scale_factor = 2 ** level
             iterator_output = dict(
@@ -735,7 +1145,7 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
                         x=x,
                         y=y,
                         **frame_position,
-                        axes=list(self._axes.keys()),
+                        axes=sorted_axes,
                         level=level,
                     )
             self._validateZarr()  # refresh self._levels before continuing
@@ -747,6 +1157,7 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         alpha=True,
         overwriteAllowed=True,
         resample=None,
+        **converterParams,
     ):
         """
         Output the current image to a file.
@@ -759,6 +1170,8 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         :param resample: one of the ``ResampleMethod`` enum values.  Defaults
             to ``NP_NEAREST`` for lossless and non-uint8 data and to
             ``PIL_LANCZOS`` for lossy uint8 data.
+        :param converterParams: options to pass to the large_image_converter if
+            the output is not a zarr variant.
         """
         if os.path.exists(path):
             if overwriteAllowed:
@@ -795,6 +1208,8 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
                         **frame_position,
                     )
 
+        source._validateZarr()
+
         if suffix in ['.zarr', '.db', '.sqlite', '.zip']:
             if resample is None:
                 resample = (
@@ -803,9 +1218,10 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
                     else ResampleMethod.NP_NEAREST
                 )
             source._generateDownsampledLevels(resample)
+            source._writeInternalMetadata()  # rewrite with new level datasets
 
             if suffix == '.zarr':
-                shutil.copytree(source._tempdir.name, path)
+                shutil.copytree(str(source._tempdir), path)
             elif suffix in ['.db', '.sqlite']:
                 sqlite_store = zarr.SQLiteStore(path)
                 zarr.copy_store(source._zarr_store, sqlite_store, if_exists='replace')
@@ -818,10 +1234,11 @@ class ZarrFileTileSource(FileTileSource, metaclass=LruCacheMetaclass):
         else:
             from large_image_converter import convert
 
-            attrs_path = Path(source._tempdir.name) / '.zattrs'
+            attrs_path = source._tempdir / '.zattrs'
             params = {}
             if lossy and self.dtype == np.uint8:
                 params['compression'] = 'jpeg'
+            params.update(converterParams)
             convert(str(attrs_path), path, overwrite=overwriteAllowed, **params)
 
 

{large-image-source-zarr-1.28.2.dev12 → large_image_source_zarr-1.30.7.dev12}/large_image_source_zarr.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: large-image-source-zarr
-Version: 1.28.2.dev12
+Version: 1.30.7.dev12
 Summary: A OME Zarr tilesource for large_image.
 Home-page: https://github.com/girder/large_image
 Author: Kitware, Inc.
@@ -15,13 +15,16 @@ Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Requires-Python: >=3.6
+Description-Content-Type: text/x-rst
 License-File: LICENSE
-Requires-Dist: large-image>=1.28.2.dev12
-Requires-Dist: zarr
-Requires-Dist: imagecodecs-numcodecs
+Requires-Dist: large-image>=1.30.7.dev12
+Requires-Dist: zarr<3
+Requires-Dist: numcodecs
+Requires-Dist: imagecodecs-numcodecs!=2024.9.22
 Provides-Extra: girder
-Requires-Dist: girder-large-image>=1.28.2.dev12; extra == "girder"
+Requires-Dist: girder-large-image>=1.30.7.dev12; extra == "girder"
 Provides-Extra: all
 Requires-Dist: large-image-converter; extra == "all"
 

large_image_source_zarr-1.30.7.dev12/large_image_source_zarr.egg-info/requires.txt

@@ -0,0 +1,10 @@
+large-image>=1.30.7.dev12
+zarr<3
+numcodecs
+imagecodecs-numcodecs!=2024.9.22
+
+[all]
+large-image-converter
+
+[girder]
+girder-large-image>=1.30.7.dev12

{large-image-source-zarr-1.28.2.dev12 → large_image_source_zarr-1.30.7.dev12}/setup.py

@@ -19,8 +19,7 @@ def prerelease_local_scheme(version):
 
     if os.getenv('CIRCLE_BRANCH') in ('master', ):
         return ''
-    else:
-        return get_local_node_and_date(version)
+    return get_local_node_and_date(version)
 
 
 try:
@@ -37,6 +36,7 @@ setup(
                      'fallback_version': '0.0.0'},
     description=description,
     long_description=long_description,
+    long_description_content_type='text/x-rst',
     license='Apache Software License 2.0',
     author='Kitware, Inc.',
     author_email='kitware@kitware.com',
@@ -49,13 +49,16 @@ setup(
         'Programming Language :: Python :: 3.10',
         'Programming Language :: Python :: 3.11',
         'Programming Language :: Python :: 3.12',
+        'Programming Language :: Python :: 3.13',
     ],
     install_requires=[
         f'large-image{limit_version}',
-        'zarr',
-        # I am uncertain why this is required, since numcodecs is required by
-        # zarr; but without it some jpeg encoded data cannot be read
-        'imagecodecs-numcodecs',
+        # zarr 3 removes the SQLiteStore
+        'zarr<3',
+        # numcodecs had been required by zarr, but now needs to be asked for
+        'numcodecs',
+        # Without imagecodecs-numcodecs, some jpeg encoded data cannot be read
+        'imagecodecs-numcodecs!=2024.9.22',
     ],
     extras_require={
         'girder': f'girder-large-image{limit_version}',

large-image-source-zarr-1.28.2.dev12/large_image_source_zarr.egg-info/requires.txt

@@ -1,9 +0,0 @@
-large-image>=1.28.2.dev12
-zarr
-imagecodecs-numcodecs
-
-[all]
-large-image-converter
-
-[girder]
-girder-large-image>=1.28.2.dev12