patchworks 3.2.0__tar.gz → 3.3.0__tar.gz

{patchworks-3.2.0 → patchworks-3.3.0}/.github/workflows/test.yml

@@ -32,11 +32,9 @@ jobs:
         run: pytest -q --doctest-modules src
 
   other-platforms:
-    # Not yet verified on macOS or Windows (no CI ran there before), so this
-    # job reports without failing the build; promote it into the matrix
-    # above once it is green.
+    # Green on macOS and Windows since 2026-10-07: a failure there now fails
+    # the build like any other.
     runs-on: ${{ matrix.os }}
-    continue-on-error: true
     strategy:
       fail-fast: false
       matrix:

{patchworks-3.2.0 → patchworks-3.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: patchworks
-Version: 3.2.0
+Version: 3.3.0
 Summary: Tiled processing of arbitrarily large images with globally consistent labels
 Project-URL: Homepage, https://github.com/imcf/patchworks
 Project-URL: Issues, https://github.com/imcf/patchworks/issues
@@ -102,7 +102,7 @@ Provides-Extra: workflow
 Requires-Dist: openpyxl; extra == 'workflow'
 Requires-Dist: pandas>=2.0; extra == 'workflow'
 Requires-Dist: scikit-image; extra == 'workflow'
-Requires-Dist: snakemake-executor-plugin-slurm; extra == 'workflow'
+Requires-Dist: snakemake-executor-plugin-slurm>=2.2; extra == 'workflow'
 Requires-Dist: snakemake>=8; extra == 'workflow'
 Description-Content-Type: text/markdown
 

{patchworks-3.2.0 → patchworks-3.3.0}/docs/examples/dog.md

@@ -54,6 +54,24 @@ directly to the DoG image — start near the DoG's typical peak value on a
 known-positive region and adjust from there; there's no auto (Otsu-style)
 option, since the DoG image isn't bimodal the way a raw intensity image is.
 
+## Thin, oblique objects: `connectivity`
+
+The thresholded voxels are joined into objects across shared **faces** by
+default. A cilium lying obliquely is a staircase of voxels touching only
+along edges or at corners, and comes out as a row of fragments.
+`connectivity=2` also joins voxels sharing an edge, `connectivity=3` (3-D)
+also a corner:
+
+```python
+fn = dog_label_fn(low_sigma=1.0, high_sigma=3.0, threshold=0.02, connectivity=3)
+```
+
+The merge has to join tiles the same way, or objects crossing a tile
+boundary diagonally are split there: the workflow does this by itself; with
+the API, pass the same value to `merge_tile_labels(..., connectivity=3)`.
+The merged result is then exactly that of labelling the whole image at once,
+whatever the tile size.
+
 ## GPU
 
 ```python

{patchworks-3.2.0 → patchworks-3.3.0}/docs/guide/cli.md

@@ -25,8 +25,18 @@ patchworks tables scan.zarr --relate nuclei:cells
 
 # Look at the likely mistakes one by one and fix them (napari)
 patchworks review scan.zarr --expect cells:nuclei=1
+
+# Bring a store written by an older patchworks up to the OME-Zarr spec
+patchworks fix-metadata scan.zarr
 ```
 
+`patchworks fix-metadata` changes metadata only, in place: it gives every
+OME-Zarr 0.5 array its `dimension_names` (required by 0.5; versions before
+this one wrote none, which strict readers such as `ome-zarr-models` refuse),
+and drops surplus coarse levels from a label image with more pyramid levels
+than its image (the spec requires the same number). Running it twice
+changes nothing the second time.
+
 `patchworks review` without a window: `--summary` (counts and error
 estimate), `--export DIR --format csv|xlsx|parquet` (corrected tables),
 `--workbooks DIR` (relation workbooks), `--write-labels NAME` (a label image

{patchworks-3.2.0 → patchworks-3.3.0}/docs/guide/membrane_cells.md

@@ -68,6 +68,7 @@ gets closed by the partitioning, where Cellpose would merge across it.
 PlantSeg is on conda-forge only, so it has its own pixi environment:
 
 ```bash
+export CONDA_OVERRIDE_CUDA=12.0   # login node without a GPU: see below
 pixi install -e plantseg
 pixi run -e plantseg plantseg-fetch generic_confocal_3D_unet   # once, with internet
 pixi run -e plantseg multi-slurm
@@ -162,8 +163,37 @@ pixi run -e plantseg multi-plantseg-dry                        # check the plan
 pixi run -e plantseg multi-plantseg-slurm                      # submit
 ```
 
-The `plantseg` environment is the default one plus PlantSeg, so the
-Cellpose run comes from it too. Afterwards, `nuclei_to_cyto.xlsx` gives
+Your own multi config with a PlantSeg segmentation in it runs the same
+way, from the same environment:
+
+```bash
+pixi run -e plantseg multi-slurm --config /path/to/my_multi.yaml
+```
+
+!!! note "`Virtual package '__cuda' does not match` on a login node"
+
+    The `plantseg` environment holds a CUDA build of PyTorch, and pixi
+    refuses to install or run an environment needing CUDA on a machine with
+    no GPU driver -- a login node. Only the GPU jobs need CUDA, and they
+    start the environment's Python directly, without pixi. Tell pixi on the
+    login node that a driver is there:
+
+    ```bash
+    export CONDA_OVERRIDE_CUDA=12.0   # any 12.x; put it in ~/.bashrc
+    ```
+
+    It must be set for every `pixi run -e plantseg ...` there, not only for
+    the install. Check the GPU nodes' driver supports CUDA 12 (`nvidia-smi`
+    in a GPU job prints "CUDA Version: 12.x" or later).
+
+The `-e plantseg` matters: every job runs in the environment the command
+was started from, and the default one has no PlantSeg. `run_multi` checks
+this before converting anything and says which environment to use (as it
+does for `denoise:`, which needs `-e careamics`).
+
+The `plantseg` environment is the default one plus PlantSeg and cupy, so
+the Cellpose run comes from it too, as does a cilia config with the DoG
+plugin's `use_gpu: true` (cupy) and deconvolution. Afterwards, `nuclei_to_cyto.xlsx` gives
 each nucleus its cell, and `pixi run -e viewer review <work_dir>/image.zarr`
 lists any cell not holding exactly one nucleus.
 

{patchworks-3.2.0 → patchworks-3.3.0}/docs/guide/review.md

@@ -223,9 +223,13 @@ image.zarr/labels/cilia_labels/
 So it travels with the labels, including in a zip bundle, and it is
 replaced whenever the labels are. A table computed from older labels is
 recognised and ignored, never shown against the wrong segmentation.
-Measuring costs one read of the labels, after the merge. Add intensity
-columns with `table_channels: [0, 2]`, or turn tables off with
-`object_table: false` (see the [workflow config](snakemake.md)).
+The table costs no extra read of the labels: each segment job measures
+its tiles' objects as it writes them, and the merge adds those sums up per
+merged object (an object cut by tile boundaries gets exactly the values it
+would have measured whole). Intensity columns (`table_channels: [0, 2]`)
+need the image, so with them the merged labels are measured once instead.
+Turn tables off with `object_table: false` (see the
+[workflow config](snakemake.md)).
 
 For a store that has none, for example a run made before tables existed,
 or labels from elsewhere:

{patchworks-3.2.0 → patchworks-3.3.0}/docs/guide/snakemake.md

@@ -363,8 +363,12 @@ shard_labels: false            # true → also reshard label level 0 after the
     one blob). Set either, both, or neither (`null`, the default, disables
     each). Both run once on the **fully merged** image — not per tile, where
     an object crossing a tile boundary would look smaller or larger than it
-    really is. Runs after `merge` and before the pyramid is built, so every
-    pyramid level reflects the filtered result, and needs `image.zarr` to
+    really is. The sizes come from what the segment jobs measured per tile,
+    added up per merged object, and the filter is applied in the merge's
+    own relabelling pass, so it costs no pass of its own over the image
+    (runs whose tiles were segmented by an older version filter the merged
+    labels in a separate pass instead). Every pyramid level reflects the
+    filtered result. It needs `image.zarr` to
     carry a pixel size (the same calibration deconvolution's voxel sizes and
     Cellpose's `anisotropy` are derived from — see the tip below); an
     uncalibrated store raises rather than silently skipping the filter. See
@@ -650,6 +654,14 @@ running the workflow **twice with two configs against the same `work_dir`**
 never collides: each run gets its own private subdirectory, and both reuse
 the *same* already-converted `image.zarr` (conversion never re-runs).
 
+`labels.done` is what makes Snakemake call a segmentation finished, while
+the labels themselves are in `image.zarr/labels/<label_name>`. Deleting only
+the latter leaves the run "done" with nothing re-making it: `run_multi`
+refuses that up front and names the `work_dir/<label_name>` folder to remove
+for a fresh segmentation. It also refuses a relation naming a label image
+that no listed config makes and the store does not hold, and skips (with a
+message) a relation whose labels are still missing once everything has run.
+
 Most of what those configs contain is identical — the input, the `work_dir`,
 the tiling, everything `convert` reads. Put it in **one** shared file and let
 each config carry only what actually differs. Snakemake merges several
@@ -810,9 +822,10 @@ results/image.zarr/labels/cyto_labels/
     rule, so it doesn't get a `log:` directive for free. Standalone (or
     under plain `multi`), it writes to `<work_dir>/logs/relate.log`
     (override with `relate.py --log`), the same tee-to-file-and-stdout
-    behaviour as the other steps. Under `multi-slurm`, where each pair is
-    its own concurrent job, `run_multi.py` points each one at its own file
-    instead — `<work_dir>/logs/relate/<a>_to_<b>.log` — so concurrent pairs
+    behaviour as the other steps. Under `multi-slurm`, where each parent's
+    relations are one concurrent job, `run_multi.py` points each job at its
+    own file instead — `<work_dir>/logs/relate/to_<parent>.log`, or
+    `<a>_to_<b>.log` for a parent with a single child — so concurrent jobs
     don't interleave into one log; check there instead of scrolling back
     through `srun`'s live output.
 
@@ -856,6 +869,19 @@ pixi run multi        # run locally
 pixi run multi-slurm  # submit every segmentation to SLURM
 ```
 
+They read `config/multi.yaml` unless given another with `--config`:
+
+```bash
+pixi run multi-slurm --config /data/run42/multi.yaml
+pixi run multi-dry --config my_multi.yaml   # relative to where you run pixi
+```
+
+A relative `--config` is looked for in the directory you run `pixi` from,
+then in `workflow/`. The `common:` and `segmentations:` paths inside it are
+looked for next to the multi config first, then in `workflow/` -- so a run's
+configs can live together in a folder of their own, beside the data, while
+the shipped `config/multi.yaml` keeps working as it is.
+
 Before anything is submitted, the script checks that every listed config
 shares one `work_dir` (so `label_relations` has a single `image.zarr` to read
 both label groups from), that `tile_shape` and `level` are identical (so the
@@ -893,7 +919,7 @@ abort the others; you get a per-config status and a non-zero exit.
     # config/multi.yaml
     relate:
       qos: "1day"
-      time: 720        # minutes, per pair; must stay under that QOS's MaxWall
+      time: 720        # minutes, per job (one per parent); under the QOS's MaxWall
     ```
 
     A `--relate-*` flag overrides the block for that one key; anything the
@@ -902,18 +928,23 @@ abort the others; you get a per-config status and a non-zero exit.
     default you meant to replace. Under plain `multi` (no `--profile`), relations
     still run locally, in-process, one after another, as before.
 
-    Each pair logs its shape, chunk count and object count before it starts,
-    then a progress line roughly once a minute (`label_relations: 412/3,600
-    (11%) after 7m, ~55m left`), so a long relation is distinguishable from a
-    hung one in `logs/relate/<a>_to_<b>.log`.
-
-    Because every pair gets its own job, one running long no longer starves
-    the others out of a shared time budget, and a pair that gets killed no
-    longer takes an already-finished sibling's workbook down with it.
-    `relate.py` also skips a pair whose `.xlsx` is already newer than both
-    labels' merge marker, so **re-running the exact same `multi-slurm`
-    command only recomputes what's still missing or stale** — delete a
-    specific `.xlsx` yourself to force just that one to recompute.
+    The relations are grouped by parent: one job reads each parent label
+    image once for all its children (e.g. nuclei, cilia and the other cell
+    segmentation against `cyto_labels`), and only the chunks some child has
+    labels in. Each job logs its images' shape, chunking and object counts
+    before it starts, then a progress line roughly once a minute
+    (`label_relations: 412/3,600 (11%) after 7m, ~55m left`), so a long
+    relation is distinguishable from a hung one in `logs/relate/`.
+
+    Because each parent gets its own job, one running long does not starve
+    the others out of a shared time budget. `relate.py` skips a pair whose
+    workbook (the `.xlsx`, or the two `.csv` files a sheet too long for
+    Excel is written as) is already newer than both labels' merge marker,
+    and rewrites a missing workbook from the object tables when they
+    already hold the relation for the current labels, so **re-running the
+    exact same `multi-slurm` command only recomputes what's still missing
+    or stale** — delete a specific workbook to force just that one. The
+    bundle is likewise left alone when nothing in the store changed.
 
 !!! tip "After a killed run"
     Snakemake only releases its lock on a clean exit, so a run that was killed

{patchworks-3.2.0 → patchworks-3.3.0}/pyproject.toml

@@ -119,7 +119,8 @@ napari = [
 # workbook (per-object + per-container sheets) from the object tables.
 workflow = [
     "snakemake>=8",
-    "snakemake-executor-plugin-slurm",
+    # >=2.2: profile/slurm sets slurm-jobname-prefix, unknown before 2.2
+    "snakemake-executor-plugin-slurm>=2.2",
     "openpyxl",
     "pandas>=2.0",
     # method: "threshold" and the nuclei-seeded watershed plugin

{patchworks-3.2.0 → patchworks-3.3.0}/src/patchworks/_distributed.py

@@ -13,6 +13,7 @@ from __future__ import annotations
 
 import itertools
 import logging
+import os
 from pathlib import Path
 from typing import Callable, Sequence, Union
 
@@ -124,6 +125,7 @@ def create_stage(
     *,
     component: str = "staged",
     dtype=np.int32,
+    zarr_format: int | None = None,
 ) -> str:
     """Create the empty (zero-filled) shared stage store for tiled writes.
 
@@ -140,19 +142,27 @@ def create_stage(
     dtype : data-type, optional
         Label dtype (default ``int32``). Tiles write local labels; the merge's
         first pass renumbers them to a compact global range that fits int32.
+    zarr_format : int, optional
+        2 or 3. Give the image store's own format when staging straight into
+        its ``labels/<name>`` group: a v3 group inside a v2 (OME-Zarr 0.4)
+        store is something no reader can open. Default: zarr's.
 
     Returns
     -------
     str
         The stage store path.
     """
-    root = zarr.open_group(str(stage_path), mode="w")
+    root = zarr.open_group(
+        str(stage_path),
+        mode="w",
+        **({"zarr_format": zarr_format} if zarr_format else {}),
+    )
     root.create_array(
         name=component,
         shape=shape,
         chunks=tile_shape,
         dtype=dtype,
-        **zarr_compressor_kwargs(),
+        **zarr_compressor_kwargs(root.metadata.zarr_format),
     )
     return str(stage_path)
 
@@ -168,6 +178,7 @@ def stage_tile(
     component: str = "staged",
     channel_axis: int | None = None,
     halo_dir: Union[str, Path, None] = None,
+    parts_dir: Union[str, Path, None] = None,
 ) -> int:
     """Run *fn* on a single tile and write it into the shared stage store.
 
@@ -208,6 +219,13 @@ def stage_tile(
         (keys ``"<axis>+"``/``"<axis>-"``), labelled with the same ``1..n``
         ids as the staged core. An object seen only in the halo is 0 there:
         it has no id in this tile. ``None`` (default) keeps nothing.
+    parts_dir : str or Path, optional
+        Also measure the tile's objects as written (size, centroid, spread,
+        bounding box; :func:`patchworks._tables.tile_partial`) into
+        ``<parts_dir>/<index>.npz``, keyed by the staged ``1..n`` ids. The
+        merge combines them into the object table without reading the
+        labels again (``merge_tile_labels(..., parts_dir=...)``). A tile
+        without labels writes none.
 
     Returns
     -------
@@ -272,9 +290,29 @@ def stage_tile(
     n_labels = int(trimmed.max())
     dst = zarr.open_group(str(stage_path), mode="r+")[component]
     dst[sl] = trimmed.astype(dst.dtype)
+    if parts_dir is not None and n_labels:
+        _save_partial(trimmed, sl, parts_dir, index)
     return n_labels
 
 
+def _save_partial(trimmed, sl, parts_dir, index) -> None:
+    """Write one tile's per-object sums to ``<parts_dir>/<index>.npz``.
+
+    Through a temporary name and a rename, so a job killed mid-write leaves
+    no truncated file for the merge to trust.
+    """
+    from ._tables import tile_partial
+
+    part = tile_partial(trimmed, [s.start for s in sl])
+    if part is None:
+        return
+    parts_dir = Path(parts_dir)
+    parts_dir.mkdir(parents=True, exist_ok=True)
+    tmp = parts_dir / f".{int(index)}.tmp.npz"
+    np.savez(tmp, **part)
+    os.replace(tmp, parts_dir / f"{int(index)}.npz")
+
+
 def _core_lut(core: np.ndarray, max_id: int) -> np.ndarray:
     """LUT renumbering *core*'s ids to ``1..n`` (as ``relabel_sequential``);
     ids absent from the core map to 0."""