micro-reader 0.0.3__tar.gz → 0.0.4__tar.gz

{micro_reader-0.0.3 → micro_reader-0.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: micro-reader
-Version: 0.0.3
+Version: 0.0.4
 Summary: Low-level, memory-bounded pixel reading for microscopy formats: read exactly the rectangle you ask for.
 Project-URL: Homepage, https://github.com/bugraoezdemir/micro-reader
 Project-URL: Source, https://github.com/bugraoezdemir/micro-reader
@@ -54,7 +54,7 @@ Description-Content-Type: text/markdown
 
 **Memory-bounded pixel reader for microscopy images.**
 
-`micro-reader` reads regions of large microscopy files without loading the entire image into memory. It provides a common interface across TIFF, CZI, ND2, Leica, OIR, Imaris, MRC, and common raster formats.
+`micro-reader` reads regions of large microscopy files without loading the entire image into memory. It provides a common interface across TIFF, CZI, ND2, Leica, OIR, OIB / OIF, Olympus VSI, Imaris, DeltaVision, MRC, common raster formats, OME companion file sets, and high-content screening plates (Harmony, MetaXpress, IN Cell, CellVoyager).
 
 > **Status: pre-alpha (0.0.x)**
 >
@@ -213,6 +213,31 @@ resolution
 
 Only dimensions actually present in a file appear in an image's index.
 
+### Files made of several files
+
+Some formats always come as several files: an OIR file with its continuation files (`name_00001`), a Micro-Manager stack split over `..._MMStack_Pos0.ome.tif`, `..._MMStack_Pos0_1.ome.tif`, an NDTiff folder, an Olympus OIF with its `.oif.files` folder, an Olympus VSI with its `_name_` folder, a DeltaVision image with its softWoRx `.log`, a plate's side files (Harmony `Assaylayout/` and `FFC_Profile/`, CellVoyager settings files, MetaXpress `_thumb` previews). Opening any of these files opens the whole image:
+
+```python
+f = micro_reader.open("cells_R3D.dv.log")
+f.path     # the main file: "cells_R3D.dv"
+f.opened   # what was opened: "cells_R3D.dv.log"
+```
+
+You can also open the folder. A folder opens the image it belongs to (an NDTiff dataset folder, a Harmony or CellVoyager measurement folder), or the one image it holds. A folder with several images raises an error that lists them.
+
+To sort many files at once, for example everything under a folder you want to convert, `group_files` says which files make one image:
+
+```python
+g = micro_reader.group_files("experiment/")
+for group in g.groups:
+    group.main      # the file to open: "slide.vsi", "plate/Index.idx.xml"
+    group.parts     # the other files of a multi-file image
+    group.members   # for a plate or companion index: the files its images come from
+g.singles           # every other file, as given
+```
+
+A file that a plate or companion index names is listed only in that index's group. When the main file of a multi-file image is missing, its group has `main=None`.
+
 ## Axes
 
 An image array uses these axes:
@@ -406,7 +431,7 @@ img_scene0_level2 = f.select(scene=0, resolution=2)[0]
 Supported pyramids include:
 
 * **Imaris**: all stored levels
-* **TIFF**: OME-TIFF and other TIFF pyramids such as SVS and NDPI
+* **TIFF**: OME-TIFF and whole-slide pyramids (SVS, NDPI, SCN, BIF, QPTIFF, Philips)
 * **CZI**: stored pyramids of whole scenes
 
 Other formats expose a single resolution level.
@@ -459,6 +484,12 @@ f = micro_reader.open("plate.czi", plates=True)
 
 Missing plane files are read as zeros, with a `MissingImagesWarning`. A field without any of its files is left out.
 
+A plane file whose size or pixel type differs from what the index describes raises an error when it is read. To find such files before reading anything, check them all at once:
+
+```python
+f.check_members()   # opens every plane file once; returns how many were checked
+```
+
 When the channels of a field have different z sampling, `image.parts` gives one image per group of channels that share it.
 
 ## Metadata
@@ -524,13 +555,16 @@ The position is omitted when it cannot be determined reliably.
 
 | Format               | Notes                                                 |
 | -------------------- | ----------------------------------------------------- |
-| **TIFF / BigTIFF**   | TIFF, OME-TIFF, ImageJ, LSM, STK and TIFF pyramids    |
+| **TIFF / BigTIFF**   | TIFF, OME-TIFF, ImageJ, LSM, STK, Micro-Manager, whole-slide TIFFs |
 | **OME companion**    | `.companion.ome` sets of OME-TIFF files               |
 | **CZI**              | Zeiss CZI, including mosaic tiles and stitched scenes |
 | **ND2**              | Nikon ND2, including legacy JPEG 2000 ND2             |
 | **Leica**            | LIF, LOF, LIFEXT, XLIF, XLEF, XLCF                    |
 | **OIR**              | Olympus / Evident FluoView                            |
+| **OIB / OIF**        | Olympus FluoView FV1000 / FV1200 / FV3000             |
+| **VSI**              | Olympus / Evident cellSens and VS120 / VS200 slide scanners |
 | **Imaris**           | `.ims`, with the `[ims]` extra                        |
+| **DeltaVision**      | `.dv`, `.r3d` (softWoRx, OMX, Priism)                 |
 | **MRC**              | Common MRC modes and extended headers                 |
 | **Plates (HCS)**     | Harmony, MetaXpress, IN Cell and CellVoyager plates   |
 | **JPEG / PNG / BMP** | Common raster image formats                           |
@@ -549,6 +583,14 @@ Supported TIFF formats include:
 * ImageJ
 * Zeiss LSM
 * MetaMorph STK
+* Micro-Manager stacks (MMStack and NDTiff), also when one acquisition is spread over several files
+* Whole-slide TIFFs: Aperio SVS, Hamamatsu NDPI, Leica SCN, Ventana BIF, PerkinElmer / Akoya QPTIFF and Philips
+
+Micro-Manager stage positions are scenes. NDTiff channel names and pixel sizes come from the dataset's index and frame metadata.
+
+A whole-slide file gives the slide first, then the other pictures it stores as further scenes, as Bio-Formats lists them: `label` and `macro` (overview), and for some formats `thumbnail` (QPTIFF) or `probability` (BIF tissue map). Pixel sizes, positions and channel names come from each vendor's own metadata.
+
+Ventana BIF scans store overlapping tiles. They are composed as Roche's description of the format places them. Files from scanners other than the VENTANA DP 200 are not supported.
 
 Compressed TIFF data is decoded only for the strips or tiles needed by the requested region.
 
@@ -580,7 +622,15 @@ An OME companion file (`.companion.ome`) describes images stored in separate TIF
 f = micro_reader.open("data.companion.ome")
 ```
 
-Opening one of the TIFF files on its own reads only that file, with its pixel sizes, channels and position taken from the companion. If the companion is missing, the file is read without pixel sizes, with a `CompanionWarning`.
+Opening one of the TIFF files on its own reads only that file, with its pixel sizes, channels and position taken from the companion. If the companion is missing, the file is read without pixel sizes, with a `CompanionWarning`. A TIFF whose metadata lives in another OME-TIFF (instead of a `.companion.ome`) takes it from that file in the same way.
+
+Some OME-TIFF sets keep the full OME-XML in their TIFF files instead of a companion. Opening one of those files reads the whole set. To read only the planes stored in that file, with the set's metadata:
+
+```python
+f = micro_reader.open("set_t001.ome.tif", multifile=False)
+```
+
+Micro-Manager stacks always open whole, also with `multifile=False`: they split their files by size, so one file does not hold a complete part of the image.
 
 ### CZI
 
@@ -643,6 +693,18 @@ These are represented using the common image/index model.
 
 Olympus / Evident FluoView OIR files are supported, including companion files and spectral scans.
 
+### Olympus OIB and OIF
+
+Older FluoView files come in two forms with the same content: an OIB is one file, an OIF is a `.oif` settings file plus a `.oif.files` folder with one TIFF per plane. Both open as one `t c z y x` image, with pixel sizes, z and time steps, channel names, colours and wavelengths. Opening any file in the `.oif.files` folder opens the OIF.
+
+### Olympus VSI
+
+A cellSens VSI is a `.vsi` file plus a `_<name>_` folder that holds the pixels (`stack1/frame_t.ets`, `stack10000/frame_t.ets`, ...). The images are listed as Bio-Formats lists them: one per scanned area, named as in cellSens, with the slide label (`label`) and overview (`overview`) when the scanner took them, and the slide photo (`macro`) last. Each scanned area keeps its resolution levels.
+
+Images are `t c z y x` (`t c z y x s` for brightfield RGB scans), with pixel sizes, z and time steps, and each channel's name, colour and emission wavelength. Stage positions are not read.
+
+Opening the `_<name>_` folder, a `stack` folder or a file in it opens the `.vsi`. An `.ets` file whose `.vsi` is missing is read alone, with a warning, when it is a plain 2-D image; its size is then rounded up to whole tiles.
+
 ### Imaris
 
 Install Imaris support with:
@@ -655,6 +717,12 @@ Imaris resolution level 0 is exposed at the actual image size, with Imaris chunk
 
 Chunked compression is decoded in parallel.
 
+### DeltaVision
+
+DeltaVision files (`.dv`, `.r3d`, `.r3d_d3d`, also deconvolved and projected ones) are recognised by their content, whatever their extension. The image keeps the file's own order of time, waves and z (for example `ctzyx`).
+
+Pixel sizes, and each channel's emission and excitation wavelengths, are read from the file. Channel names come from the softWoRx acquisition log next to it (`<file>.log`); a deconvolved or projected file takes the names from the log of the raw file it was made from. Stage positions are not read.
+
 ### MRC
 
 Supported MRC modes include:

{micro_reader-0.0.3 → micro_reader-0.0.4}/README.md

@@ -2,7 +2,7 @@
 
 **Memory-bounded pixel reader for microscopy images.**
 
-`micro-reader` reads regions of large microscopy files without loading the entire image into memory. It provides a common interface across TIFF, CZI, ND2, Leica, OIR, Imaris, MRC, and common raster formats.
+`micro-reader` reads regions of large microscopy files without loading the entire image into memory. It provides a common interface across TIFF, CZI, ND2, Leica, OIR, OIB / OIF, Olympus VSI, Imaris, DeltaVision, MRC, common raster formats, OME companion file sets, and high-content screening plates (Harmony, MetaXpress, IN Cell, CellVoyager).
 
 > **Status: pre-alpha (0.0.x)**
 >
@@ -161,6 +161,31 @@ resolution
 
 Only dimensions actually present in a file appear in an image's index.
 
+### Files made of several files
+
+Some formats always come as several files: an OIR file with its continuation files (`name_00001`), a Micro-Manager stack split over `..._MMStack_Pos0.ome.tif`, `..._MMStack_Pos0_1.ome.tif`, an NDTiff folder, an Olympus OIF with its `.oif.files` folder, an Olympus VSI with its `_name_` folder, a DeltaVision image with its softWoRx `.log`, a plate's side files (Harmony `Assaylayout/` and `FFC_Profile/`, CellVoyager settings files, MetaXpress `_thumb` previews). Opening any of these files opens the whole image:
+
+```python
+f = micro_reader.open("cells_R3D.dv.log")
+f.path     # the main file: "cells_R3D.dv"
+f.opened   # what was opened: "cells_R3D.dv.log"
+```
+
+You can also open the folder. A folder opens the image it belongs to (an NDTiff dataset folder, a Harmony or CellVoyager measurement folder), or the one image it holds. A folder with several images raises an error that lists them.
+
+To sort many files at once, for example everything under a folder you want to convert, `group_files` says which files make one image:
+
+```python
+g = micro_reader.group_files("experiment/")
+for group in g.groups:
+    group.main      # the file to open: "slide.vsi", "plate/Index.idx.xml"
+    group.parts     # the other files of a multi-file image
+    group.members   # for a plate or companion index: the files its images come from
+g.singles           # every other file, as given
+```
+
+A file that a plate or companion index names is listed only in that index's group. When the main file of a multi-file image is missing, its group has `main=None`.
+
 ## Axes
 
 An image array uses these axes:
@@ -354,7 +379,7 @@ img_scene0_level2 = f.select(scene=0, resolution=2)[0]
 Supported pyramids include:
 
 * **Imaris**: all stored levels
-* **TIFF**: OME-TIFF and other TIFF pyramids such as SVS and NDPI
+* **TIFF**: OME-TIFF and whole-slide pyramids (SVS, NDPI, SCN, BIF, QPTIFF, Philips)
 * **CZI**: stored pyramids of whole scenes
 
 Other formats expose a single resolution level.
@@ -407,6 +432,12 @@ f = micro_reader.open("plate.czi", plates=True)
 
 Missing plane files are read as zeros, with a `MissingImagesWarning`. A field without any of its files is left out.
 
+A plane file whose size or pixel type differs from what the index describes raises an error when it is read. To find such files before reading anything, check them all at once:
+
+```python
+f.check_members()   # opens every plane file once; returns how many were checked
+```
+
 When the channels of a field have different z sampling, `image.parts` gives one image per group of channels that share it.
 
 ## Metadata
@@ -472,13 +503,16 @@ The position is omitted when it cannot be determined reliably.
 
 | Format               | Notes                                                 |
 | -------------------- | ----------------------------------------------------- |
-| **TIFF / BigTIFF**   | TIFF, OME-TIFF, ImageJ, LSM, STK and TIFF pyramids    |
+| **TIFF / BigTIFF**   | TIFF, OME-TIFF, ImageJ, LSM, STK, Micro-Manager, whole-slide TIFFs |
 | **OME companion**    | `.companion.ome` sets of OME-TIFF files               |
 | **CZI**              | Zeiss CZI, including mosaic tiles and stitched scenes |
 | **ND2**              | Nikon ND2, including legacy JPEG 2000 ND2             |
 | **Leica**            | LIF, LOF, LIFEXT, XLIF, XLEF, XLCF                    |
 | **OIR**              | Olympus / Evident FluoView                            |
+| **OIB / OIF**        | Olympus FluoView FV1000 / FV1200 / FV3000             |
+| **VSI**              | Olympus / Evident cellSens and VS120 / VS200 slide scanners |
 | **Imaris**           | `.ims`, with the `[ims]` extra                        |
+| **DeltaVision**      | `.dv`, `.r3d` (softWoRx, OMX, Priism)                 |
 | **MRC**              | Common MRC modes and extended headers                 |
 | **Plates (HCS)**     | Harmony, MetaXpress, IN Cell and CellVoyager plates   |
 | **JPEG / PNG / BMP** | Common raster image formats                           |
@@ -497,6 +531,14 @@ Supported TIFF formats include:
 * ImageJ
 * Zeiss LSM
 * MetaMorph STK
+* Micro-Manager stacks (MMStack and NDTiff), also when one acquisition is spread over several files
+* Whole-slide TIFFs: Aperio SVS, Hamamatsu NDPI, Leica SCN, Ventana BIF, PerkinElmer / Akoya QPTIFF and Philips
+
+Micro-Manager stage positions are scenes. NDTiff channel names and pixel sizes come from the dataset's index and frame metadata.
+
+A whole-slide file gives the slide first, then the other pictures it stores as further scenes, as Bio-Formats lists them: `label` and `macro` (overview), and for some formats `thumbnail` (QPTIFF) or `probability` (BIF tissue map). Pixel sizes, positions and channel names come from each vendor's own metadata.
+
+Ventana BIF scans store overlapping tiles. They are composed as Roche's description of the format places them. Files from scanners other than the VENTANA DP 200 are not supported.
 
 Compressed TIFF data is decoded only for the strips or tiles needed by the requested region.
 
@@ -528,7 +570,15 @@ An OME companion file (`.companion.ome`) describes images stored in separate TIF
 f = micro_reader.open("data.companion.ome")
 ```
 
-Opening one of the TIFF files on its own reads only that file, with its pixel sizes, channels and position taken from the companion. If the companion is missing, the file is read without pixel sizes, with a `CompanionWarning`.
+Opening one of the TIFF files on its own reads only that file, with its pixel sizes, channels and position taken from the companion. If the companion is missing, the file is read without pixel sizes, with a `CompanionWarning`. A TIFF whose metadata lives in another OME-TIFF (instead of a `.companion.ome`) takes it from that file in the same way.
+
+Some OME-TIFF sets keep the full OME-XML in their TIFF files instead of a companion. Opening one of those files reads the whole set. To read only the planes stored in that file, with the set's metadata:
+
+```python
+f = micro_reader.open("set_t001.ome.tif", multifile=False)
+```
+
+Micro-Manager stacks always open whole, also with `multifile=False`: they split their files by size, so one file does not hold a complete part of the image.
 
 ### CZI
 
@@ -591,6 +641,18 @@ These are represented using the common image/index model.
 
 Olympus / Evident FluoView OIR files are supported, including companion files and spectral scans.
 
+### Olympus OIB and OIF
+
+Older FluoView files come in two forms with the same content: an OIB is one file, an OIF is a `.oif` settings file plus a `.oif.files` folder with one TIFF per plane. Both open as one `t c z y x` image, with pixel sizes, z and time steps, channel names, colours and wavelengths. Opening any file in the `.oif.files` folder opens the OIF.
+
+### Olympus VSI
+
+A cellSens VSI is a `.vsi` file plus a `_<name>_` folder that holds the pixels (`stack1/frame_t.ets`, `stack10000/frame_t.ets`, ...). The images are listed as Bio-Formats lists them: one per scanned area, named as in cellSens, with the slide label (`label`) and overview (`overview`) when the scanner took them, and the slide photo (`macro`) last. Each scanned area keeps its resolution levels.
+
+Images are `t c z y x` (`t c z y x s` for brightfield RGB scans), with pixel sizes, z and time steps, and each channel's name, colour and emission wavelength. Stage positions are not read.
+
+Opening the `_<name>_` folder, a `stack` folder or a file in it opens the `.vsi`. An `.ets` file whose `.vsi` is missing is read alone, with a warning, when it is a plain 2-D image; its size is then rounded up to whole tiles.
+
 ### Imaris
 
 Install Imaris support with:
@@ -603,6 +665,12 @@ Imaris resolution level 0 is exposed at the actual image size, with Imaris chunk
 
 Chunked compression is decoded in parallel.
 
+### DeltaVision
+
+DeltaVision files (`.dv`, `.r3d`, `.r3d_d3d`, also deconvolved and projected ones) are recognised by their content, whatever their extension. The image keeps the file's own order of time, waves and z (for example `ctzyx`).
+
+Pixel sizes, and each channel's emission and excitation wavelengths, are read from the file. Channel names come from the softWoRx acquisition log next to it (`<file>.log`); a deconvolved or projected file takes the names from the log of the raw file it was made from. Stage positions are not read.
+
 ### MRC
 
 Supported MRC modes include:

{micro_reader-0.0.3 → micro_reader-0.0.4}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "micro-reader"
-version = "0.0.3"
+version = "0.0.4"
 description = "Low-level, memory-bounded pixel reading for microscopy formats: read exactly the rectangle you ask for."
 readme = "README.md"
 requires-python = ">=3.11"

{micro_reader-0.0.3 → micro_reader-0.0.4}/src/micro_reader/__init__.py

@@ -7,22 +7,23 @@
 from ._base import INDEX_NAMES, PLATE_INDICES, SAMPLED_INDICES, File, Image, ReadStats
 from ._metadata import Channel, Metadata, Quantity
 from ._plate import Acquisition, Field, Plate, Well
-from ._errors import (CompanionWarning, CorruptFileError, MicroReaderError,
+from ._errors import (CompanionWarning, CompositeWarning, CorruptFileError, MicroReaderError,
                       MissingImagesWarning, UnsupportedFormatError)
 from ._memory import MemoryLimitError, memory_limit, set_memory_limit
 from ._io import cache_limit, cached_bytes, set_cache_limit
 from ._pool import decode_threads, set_decode_threads
 from ._registry import BACKENDS, open  # noqa: A004 - the public entry point
+from ._groups import FileGroup, Grouping, group_files
 from .backends.czi import DamagedFileWarning
 
-__version__ = "0.0.3"
+__version__ = "0.0.4"
 
 __all__ = [
-    "open", "File", "Image", "INDEX_NAMES", "PLATE_INDICES", "SAMPLED_INDICES", "ReadStats",
+    "open", "group_files", "FileGroup", "Grouping", "File", "Image", "INDEX_NAMES", "PLATE_INDICES", "SAMPLED_INDICES", "ReadStats",
     "BACKENDS", "Metadata", "Channel", "Quantity", "Plate", "Well", "Field", "Acquisition",
     "decode_threads", "set_decode_threads", "memory_limit", "set_memory_limit",
     "cache_limit", "set_cache_limit", "cached_bytes",
     "MicroReaderError", "UnsupportedFormatError", "CorruptFileError", "MemoryLimitError", "DamagedFileWarning",
-    "MissingImagesWarning", "CompanionWarning",
+    "MissingImagesWarning", "CompanionWarning", "CompositeWarning",
     "__version__",
 ]

{micro_reader-0.0.3 → micro_reader-0.0.4}/src/micro_reader/_base.py

@@ -247,9 +247,17 @@ SAMPLED_INDICES = ("illumination", "phase", "block", "emission", "excitation", "
                    "sequence")
 
 
+def _counted(result, bad: list) -> int:
+    """One member checked (``File.check_members``): *result* is 1, or the
+    text saying why it does not fit (kept in *bad*)."""
+    if result != 1:
+        bad.append(result)
+    return 1
+
+
 class File:
     """An opened file: its images, the backend that reads them (``backend``:
-    "tiff", "czi", "nd2", "lif", "oir", "ims", "mrc", "raster") and the
+    "tiff", "czi", "nd2", "lif", "oir", "oif", "vsi", "ims", "dv", "mrc", "raster") and the
     format variant it is (``format``: e.g. "ome-tiff", "imagej", "lsm",
     "nd2-legacy", "lof", "xlef", "png").  Every format opens as a File; a
     backend's own class is an implementation detail."""
@@ -257,7 +265,10 @@ class File:
     backend: str = "base"
 
     def __init__(self, path: str, images: Sequence[Image], plates: Sequence = ()):
+        #: the file read: for a composite format its main file (docs/COMPOSITE.md)
         self.path = str(path)
+        #: the path ``open()`` was given: a part of a composite, a folder, or ``path``
+        self.opened = self.path
         self.images = list(images)
         #: the plates the images belong to (``_plate.Plate``), like ``images``
         #: a list; empty for a file that is not a plate (docs/HCS.md)
@@ -323,6 +334,46 @@ class File:
         return [image for image in self.images
                 if all(image.index.get(name) == value for name, value in index.items())]
 
+    def _member_files(self) -> set:
+        """The other files this file's images are read from -- a file set's
+        members -- for ``group_files``.  Empty for a file that reads only itself."""
+        out = set()
+        for image in self.images:
+            image = getattr(image, "source", image)
+            for each in [image] + [p for p in image.parts if p is not image]:
+                paths = getattr(each, "_member_paths", None)
+                if paths is not None:
+                    out.update(paths())
+        out.discard(self.path)
+        return out
+
+    def check_members(self) -> int:
+        """For a file set (a plate, an OME companion set): open every member
+        file once and check that it is the plane its index describes (size,
+        type, one plane).  Reading makes the same check plane by plane, at no
+        cost; this finds a file that does not fit before anything is read, at
+        the cost of opening every member.  Returns the number of members
+        checked (0 for a file that is not a file set); raises
+        ``CorruptFileError`` naming the members that do not fit."""
+        from . import _pool
+        from ._errors import CorruptFileError
+        seen, bad = set(), []
+
+        def tasks():                        # lazy: a plate may have millions of members
+            for image in self.images:
+                image = getattr(image, "source", image)        # open(axes=...) wraps it
+                for each in [image] + [p for p in image.parts if p is not image]:
+                    check = getattr(each, "_check_members", None)
+                    if check is not None:
+                        for task in check(seen):
+                            yield lambda task=task: _counted(task(), bad)
+        checked = _pool.run(tasks())
+        if bad:
+            shown = "; ".join(bad[:5]) + (f"; ... ({len(bad)} in all)" if len(bad) > 5 else "")
+            raise CorruptFileError(f"{self.path!r}: {len(bad)} of {checked} member files do "
+                                   f"not fit the index: {shown}")
+        return checked
+
     def close(self) -> None:
         for image in self.images:
             image.close()

micro_reader-0.0.4/src/micro_reader/_cfb.py

@@ -0,0 +1,200 @@
+"""Compound File Binary (OLE2) containers: Olympus OIB, later Zeiss ZVI.
+
+Written from Microsoft's published specification [MS-CFB] (Compound File
+Binary File Format); oiffile's ``CompoundFile`` (Gohlke, BSD-3) was a
+reference.  A container is a little file system: fixed-size sectors, a FAT
+chaining them, a directory of storages (folders) and streams (files); streams
+smaller than the mini-stream cutoff live in 64-byte mini sectors inside the
+root entry's stream.
+
+Only what reading needs: the directory (stream paths ``Storage/name``), small
+streams read whole, and for a large stream its *runs* -- (offset in the
+stream, offset in the file, length) of each stretch of consecutive sectors --
+so that any byte range of the stream is read with positioned reads, never the
+whole stream.
+"""
+from __future__ import annotations
+
+import bisect
+import builtins
+import struct
+
+from ._errors import CorruptFileError
+
+MAGIC = b"\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1"
+FREESECT, ENDOFCHAIN, FATSECT, DIFSECT = 0xFFFFFFFF, 0xFFFFFFFE, 0xFFFFFFFD, 0xFFFFFFFC
+MAXREGSECT = 0xFFFFFFFA
+NOSTREAM = 0xFFFFFFFF
+STORAGE, STREAM, ROOT = 1, 2, 5
+#: directory entries are 128 bytes
+DIR_ENTRY = struct.Struct("<64sHBBIII16sIQQIQ")
+
+
+class Stream:
+    """A stream of the container: *size* bytes, read through *runs*
+    (stream offset, file offset, length), or held in memory (*data*:
+    streams in the mini stream, which are small)."""
+
+    def __init__(self, path: str, size: int, runs=None, data: bytes = None, on_disk: str = None):
+        self.path, self.size, self.runs, self.data = path, size, runs, data
+        #: the compound file the stream is in
+        self.path_on_disk = on_disk
+        self._starts = [r[0] for r in runs] if runs else []
+
+    def read_all(self) -> bytes:
+        """The whole stream (for small ones: settings, text)."""
+        if self.data is not None:
+            return self.data
+        out = bytearray()
+        with builtins.open(self.path_on_disk, "rb") as fh:
+            for at, n in self.pieces(0, self.size):
+                fh.seek(at)
+                out += fh.read(n)
+        return bytes(out)
+
+    def pieces(self, offset: int, size: int):
+        """(file offset, length) pieces that hold bytes offset:offset+size."""
+        if offset < 0 or offset + size > self.size:
+            raise CorruptFileError(f"{self.path!r}: bytes {offset}:{offset + size} of a "
+                                   f"{self.size}-byte stream")
+        k = bisect.bisect_right(self._starts, offset) - 1
+        end = offset + size
+        while offset < end:
+            start, at, length = self.runs[k]
+            take = min(end, start + length) - offset
+            yield at + offset - start, take
+            offset += take
+            k += 1
+
+
+class CompoundFile:
+    """The directory of compound file *path*: ``streams`` maps each stream's
+    path (``Storage00001/s_C001Z001.tif``, ``/`` between levels) to a
+    ``Stream``."""
+
+    def __init__(self, path: str):
+        self.path = path
+        with builtins.open(path, "rb") as fh:
+            head = fh.read(512)
+            if head[:8] != MAGIC:
+                raise CorruptFileError(f"{path!r} is no compound (OLE) file")
+            (_minor, major, byte_order, sector_shift, mini_shift, n_fat, first_dir, self.cutoff,
+             first_minifat, n_minifat, first_difat, n_difat) = self._header(head)
+            #: version 3 files keep only the low 32 bits of a stream size meaningful
+            self._size_mask = 0xFFFFFFFF if major == 3 else (1 << 64) - 1
+            if byte_order != 0xFFFE:
+                raise CorruptFileError(f"{path!r}: compound file byte order {byte_order:#x}")
+            self.sector = 1 << sector_shift
+            self.mini = 1 << mini_shift
+            fh.seek(0, 2)
+            self._file_size = fh.tell()
+            difat = list(struct.unpack_from("<109I", head, 76))
+            sect = first_difat
+            for _ in range(n_difat):
+                if sect >= MAXREGSECT:
+                    break
+                data = self._sector(fh, sect)
+                values = struct.unpack(f"<{self.sector // 4}I", data)
+                difat.extend(values[:-1])
+                sect = values[-1]
+            fat_sectors = [s for s in difat[:n_fat] if s < MAXREGSECT]
+            fat = []
+            for s in fat_sectors:
+                fat.extend(struct.unpack(f"<{self.sector // 4}I", self._sector(fh, s)))
+            self._fat = fat
+            entries = []
+            for chunk in self._chain_data(fh, first_dir):
+                for k in range(0, len(chunk), 128):
+                    entries.append(DIR_ENTRY.unpack_from(chunk, k))
+            minifat = []
+            if n_minifat and first_minifat < MAXREGSECT:
+                for chunk in self._chain_data(fh, first_minifat):
+                    minifat.extend(struct.unpack(f"<{len(chunk) // 4}I", chunk))
+            self._minifat = minifat
+            root = entries[0]
+            root_size = root[12] & self._size_mask
+            mini_stream = b"".join(self._chain_data(fh, root[11]))[:root_size] if root_size else b""
+            self.streams = {}
+            self._walk(fh, entries, entries[0][6], "", mini_stream, set())
+
+    @staticmethod
+    def _header(head: bytes):
+        minor, major, byte_order, sector_shift, mini_shift = struct.unpack_from("<HHHHH", head, 24)
+        n_fat, first_dir = struct.unpack_from("<II", head, 44)
+        cutoff, first_minifat, n_minifat, first_difat, n_difat = struct.unpack_from(
+            "<IIIII", head, 56)
+        return (minor, major, byte_order, sector_shift, mini_shift, n_fat, first_dir, cutoff,
+                first_minifat, n_minifat, first_difat, n_difat)
+
+    def _sector(self, fh, sect: int) -> bytes:
+        fh.seek((sect + 1) * self.sector)
+        data = fh.read(self.sector)
+        if len(data) != self.sector:
+            raise CorruptFileError(f"{self.path!r}: sector {sect} beyond the end of the file")
+        return data
+
+    def _chain(self, start: int) -> list:
+        chain, sect, seen = [], start, set()
+        while sect < MAXREGSECT:
+            if sect in seen or sect >= len(self._fat):
+                raise CorruptFileError(f"{self.path!r}: broken sector chain at {sect}")
+            seen.add(sect)
+            chain.append(sect)
+            sect = self._fat[sect]
+        return chain
+
+    def _chain_data(self, fh, start: int):
+        for sect in self._chain(start):
+            yield self._sector(fh, sect)
+
+    def _runs(self, start: int, size: int) -> list:
+        """(stream offset, file offset, length) of each stretch of consecutive sectors."""
+        runs, offset = [], 0
+        for sect in self._chain(start):
+            at = (sect + 1) * self.sector
+            length = min(self.sector, size - offset)
+            if length <= 0:
+                break
+            if runs and runs[-1][1] + runs[-1][2] == at:
+                s, a, n = runs[-1]
+                runs[-1] = (s, a, n + length)
+            else:
+                runs.append((offset, at, length))
+            offset += length
+        if offset < size:
+            raise CorruptFileError(f"{self.path!r}: a stream's sectors end before its size")
+        return runs
+
+    def _mini_data(self, mini_stream: bytes, start: int, size: int) -> bytes:
+        out, sect, seen = bytearray(), start, set()
+        while sect < MAXREGSECT and len(out) < size:
+            if sect in seen or sect >= len(self._minifat):
+                raise CorruptFileError(f"{self.path!r}: broken mini sector chain at {sect}")
+            seen.add(sect)
+            out += mini_stream[sect * self.mini:(sect + 1) * self.mini]
+            sect = self._minifat[sect]
+        return bytes(out[:size])
+
+    def _walk(self, fh, entries, sid: int, prefix: str, mini_stream: bytes, seen: set):
+        """Every entry of the red-black tree at *sid* (siblings left / right,
+        a storage's children below it)."""
+        stack = [sid]
+        while stack:
+            sid = stack.pop()
+            if sid == NOSTREAM or sid in seen or sid >= len(entries):
+                continue
+            seen.add(sid)
+            (raw, name_len, kind, _color, left, right, child, _clsid, _state, _times, _times2,
+             start, size) = entries[sid]
+            size &= self._size_mask
+            stack += [left, right]
+            name = raw[:max(0, name_len - 2)].decode("utf-16-le", "replace")
+            path = prefix + name
+            if kind == STORAGE:
+                self._walk(fh, entries, child, path + "/", mini_stream, seen)
+            elif kind == STREAM:
+                if size < self.cutoff:
+                    self.streams[path] = Stream(path, size, data=self._mini_data(mini_stream, start, size))
+                else:
+                    self.streams[path] = Stream(path, size, runs=self._runs(start, size),
+                                               on_disk=self.path)

{micro_reader-0.0.3 → micro_reader-0.0.4}/src/micro_reader/_errors.py

@@ -19,6 +19,12 @@ class MissingImagesWarning(UserWarning):
     (docs/HCS.md)."""
 
 
+class CompositeWarning(UserWarning):
+    """A part of a composite format was read on its own because its main file
+    is missing, or a part could not be told apart from another file
+    (docs/COMPOSITE.md)."""
+
+
 class CompanionWarning(UserWarning):
     """An OME companion set's metadata came from somewhere other than the file
     opened, or a member path had to be resolved differently than written