patchworks 0.10.0__tar.gz → 0.11.0__tar.gz

{patchworks-0.10.0 → patchworks-0.11.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: patchworks
-Version: 0.10.0
+Version: 0.11.0
 Summary: Tiled processing of arbitrarily large images with globally consistent labels
 Project-URL: Homepage, https://github.com/imcf/patchworks
 Project-URL: Issues, https://github.com/imcf/patchworks/issues
@@ -63,6 +63,9 @@ Requires-Dist: psutil; extra == 'io'
 Requires-Dist: tqdm; extra == 'io'
 Provides-Extra: napari
 Requires-Dist: napari[all]; extra == 'napari'
+Provides-Extra: workflow
+Requires-Dist: snakemake-executor-plugin-slurm; extra == 'workflow'
+Requires-Dist: snakemake>=8; extra == 'workflow'
 Description-Content-Type: text/markdown
 
 # patchworks

patchworks-0.11.0/docs/guide/snakemake.md

@@ -0,0 +1,213 @@
+# Cluster workflow (Snakemake + SLURM)
+
+`tile_process` runs every tile **serially on one GPU**. For a large 3-D image
+that can be days. The bundled Snakemake workflow instead submits **one GPU job
+per tile**, so with *N* GPUs the segmentation is ~*N*× faster. This page walks
+through running it from scratch.
+
+```text
+convert ──▶ prepare (checkpoint) ──▶ segment {tile}  ──▶ merge
+                                     one GPU SLURM job per tile
+```
+
+## 1. Get the workflow
+
+The workflow lives in the `workflow/` directory of the patchworks repository
+(it is not shipped inside the pip package — it is a set of Snakemake files you
+run):
+
+```bash
+git clone https://github.com/imcf/patchworks
+cd patchworks/workflow
+```
+
+## 2. Install the dependencies
+
+You need patchworks with the workflow + reader + segmentation extras, in the
+environment Snakemake will use:
+
+```bash
+pip install "patchworks[workflow,cellpose,imaris,bioio]"
+```
+
+- `workflow` → Snakemake + the SLURM executor plugin
+- `cellpose` → the segmentation model
+- `imaris` / `bioio` → read your input format (`.ims`, `.czi`, `.lif`, …)
+
+On a cluster, do this inside a conda/venv that the compute nodes can see, or let
+each rule activate a conda env (see *Conda*, below).
+
+## 3. Configure the run
+
+Copy and edit `config/config.yaml`. Every field:
+
+```yaml
+# input / output
+input: "/data/scan.ims"        # .ims/.czi/.lif/.nd2/ome-tiff/.zarr
+work_dir: "/scratch/results"   # everything is written here
+
+# conversion (input → pyramidal OME-ZARR)
+reuse_pyramid: true            # .ims: copy its own pyramid (fast)
+convert_chunks: null           # null → bounded auto chunks; or [c,z,y,x]
+shard: false                   # true → pack chunks into shards (fewer files)
+
+# tiling
+channel: 0                     # channel to segment (null = keep all)
+level: 0                       # pyramid level (0 = full resolution)
+tile_shape: "auto"             # "auto", or e.g. [16, 1024, 1024] (zyx)
+overlap: 30                    # halo ≈ one object diameter
+skip_empty: true               # skip background tiles
+empty_threshold: null          # null → Otsu
+
+# segmentation
+method: "cellpose"             # "cellpose" (GPU) or "threshold" (no GPU)
+label_name: "cellpose"         # name under image.zarr/labels/
+cellpose:
+  model: "cyto3"
+  diameter: 30
+  do_3D: true
+  gpu: true
+  # extra model.eval() kwargs, e.g. flow_threshold: 0.4
+
+# label pyramid
+pyramid_levels: 5
+pyramid_downscale: 2
+sequential_labels: true        # renumber labels to a contiguous 1..N
+```
+
+!!! tip "Tile size vs runtime"
+    `tile_shape: "auto"` sizes each tile to your GPU's VRAM. Smaller tiles =
+    more (faster) jobs; very large 3-D tiles are slow. Keep `do_3D: false` (2-D
+    per slice) if your objects segment fine per slice — it is much faster.
+
+## 4. Dry-run (always do this first)
+
+Check the plan without running anything:
+
+```bash
+python -m snakemake -s Snakefile --configfile config/config.yaml -n -p
+```
+
+You should see `convert`, `prepare`, and a note that the **checkpoint** will add
+the `segment` jobs after `prepare` runs. (The number of segment jobs is only
+known after `prepare` decides which tiles are non-empty.)
+
+## 5a. Run locally (single machine)
+
+```bash
+python -m snakemake -s Snakefile --configfile config/config.yaml --cores 8
+```
+
+Tiles run on the local machine (one at a time on the GPU). Good for a small
+image or a smoke test.
+
+## 5b. Run on SLURM (one GPU job per tile)
+
+Edit `profile/slurm/config.yaml` for **your** cluster — partitions, account,
+and the GPU request:
+
+```yaml
+executor: slurm
+jobs: 64                       # max concurrent SLURM jobs ≈ GPUs you can grab
+default-resources:
+  slurm_partition: "cpu"       # your CPU partition
+  # slurm_account: "my_account"
+  mem_mb: 16000
+  cpus_per_task: 4
+  runtime: 60
+set-resources:
+  segment:                     # the GPU step
+    slurm_partition: "gpu"     # your GPU partition
+    slurm_extra: "'--gres=gpu:1'"
+    mem_mb: 32000
+    runtime: 120
+  merge:
+    mem_mb: 128000
+    runtime: 240
+```
+
+Then launch (from a login node — Snakemake submits and watches the jobs):
+
+```bash
+python -m snakemake --workflow-profile profile/slurm \
+                    --configfile config/config.yaml
+```
+
+Snakemake submits `convert`, then `prepare`, then **one `segment` job per
+non-empty tile** (up to `jobs:` at once → that many GPUs in parallel), then
+`merge`. Raise `jobs:` to use more GPUs.
+
+!!! note "GPU request flag"
+    Clusters differ. `--gres=gpu:1` is common; some need `--gpus=1` or a
+    specific gres name (`--gres=gpu:a100:1`). Put whatever `sbatch` flag your
+    cluster needs in `slurm_extra`.
+
+## 6. Monitor
+
+- **Snakemake** prints each job as it submits/finishes and a `X of Y steps`
+  counter.
+- **SLURM**: `squeue --me` shows your queued/running jobs (`smk-segment`, …);
+  logs land where your profile/cluster sends them.
+- **patchworks** logs (`processing tile k/N`, ETA) are inside each job's stdout.
+
+## 7. Output
+
+Everything is under `work_dir`:
+
+```text
+results/
+  image.zarr/                 # converted, pyramidal OME-ZARR
+  image.zarr/labels/<name>/   # the segmentation (multi-scale, calibrated)
+```
+
+The labels live **inside** the image store. View image + labels together:
+
+```python
+from patchworks.plugins.napari import view_in_napari
+view_in_napari("/scratch/results/image.zarr")   # auto-loads the labels
+```
+
+## 8. Re-running and resuming
+
+Snakemake is resumable — if jobs fail or you cancel, just relaunch the same
+command and it picks up only the missing tiles. To force a clean rerun, delete
+`work_dir` (or the relevant outputs).
+
+## Conda (optional)
+
+To have each rule run in a named conda env instead of the active one, add
+`--use-conda` and point the rules at an env; or activate your env in a SLURM
+prologue. The simplest path is a single shared env that the compute nodes see.
+
+## Troubleshooting
+
+| Symptom | Fix |
+|---------|-----|
+| `snakemake: command not found` | use `python -m snakemake` |
+| Segment jobs pend forever | wrong `slurm_partition`/`slurm_extra` GPU flag for your cluster |
+| `cellpose is not installed` in a job | the job's env lacks `patchworks[cellpose]` |
+| Reading the input fails | install the matching reader (`patchworks[imaris]`/`[bioio]` + a `bioio-*`) |
+| Out of GPU memory | smaller `tile_shape`, or `do_3D: false` |
+| Very slow | confirm GPU is used (`nvidia-smi`); try 2-D or a lower `level` |
+
+## How it works (for the curious)
+
+The rule scripts are thin wrappers over patchworks' public API, so you can build
+the same per-tile distribution yourself:
+
+```python
+from patchworks import (
+    load_ome_zarr, spatial_tiles, create_stage, stage_tile, merge_tile_labels
+)
+from patchworks.plugins.ome_zarr import write_labels
+
+img = load_ome_zarr("image.zarr", channel=0)
+tiles = spatial_tiles(img.shape, tile_shape=(16, 1024, 1024))
+create_stage("stage.zarr", img.shape, (16, 1024, 1024))
+# (distribute these across jobs:)
+for i in range(len(tiles)):
+    stage_tile(img, my_fn, "stage.zarr", i, tile_shape=(16, 1024, 1024), overlap=30)
+merged = merge_tile_labels("stage.zarr", input_component="staged",
+                           write_to="merged.zarr", sequential_labels=True)
+write_labels("image.zarr", merged, name="cells")
+```

{patchworks-0.10.0 → patchworks-0.11.0}/mkdocs.yml

@@ -39,6 +39,7 @@ nav:
   - Empty tile skipping: guide/skip_empty.md
   - GPU & distributed: guide/gpu_distributed.md
   - Performance & memory: guide/performance.md
+  - Cluster workflow (Snakemake): guide/snakemake.md
   - OME-ZARR & napari: guide/ome_zarr_napari.md
   - Pitfalls: guide/pitfalls.md
 - Examples:

{patchworks-0.10.0 → patchworks-0.11.0}/pyproject.toml

@@ -58,6 +58,8 @@ bioio = [
 imaris = ["imaris-ims-file-reader"]
 # napari enables the interactive viewer plugin.
 napari = ["napari[all]"]
+# workflow runs the Snakemake pipeline (per-tile SLURM jobs across GPUs).
+workflow = ["snakemake>=8", "snakemake-executor-plugin-slurm"]
 dev = ["pytest", "pytest-cov", "scikit-image", "psutil", "tqdm"]
 docs = ["mkdocs-material>=9.0", "mkdocstrings[python]>=0.24"]
 all = [

{patchworks-0.10.0 → patchworks-0.11.0}/src/patchworks/__init__.py

@@ -32,6 +32,7 @@ from importlib.metadata import version as _pkg_version
 from ._chunks import auto_overlap, auto_tile_shape, auto_tile_shape_cellpose
 from ._cluster import make_local_cluster
 from ._core import tile_process
+from ._distributed import create_stage, spatial_tiles, stage_tile
 from ._io import estimate_empty_tiles, load_ome_zarr
 from ._merge import merge_tile_labels
 from ._relabel import relabel_sequential_array, relabel_sequential_zarr
@@ -51,4 +52,7 @@ __all__ = [
     "make_local_cluster",
     "relabel_sequential_array",
     "relabel_sequential_zarr",
+    "spatial_tiles",
+    "create_stage",
+    "stage_tile",
 ]

patchworks-0.11.0/src/patchworks/_distributed.py

@@ -0,0 +1,140 @@
+"""Per-tile building blocks for distributed processing.
+
+``tile_process`` runs every tile and merges in one process. To spread tiles
+across separate jobs (e.g. one SLURM GPU job per tile) you need to process a
+*single* tile independently and merge later. These helpers expose exactly that:
+:func:`spatial_tiles` enumerates the tiles, :func:`create_stage` makes the
+shared output store, and :func:`stage_tile` runs ``fn`` on one tile and writes
+it into that store. Stitch the result with
+:func:`patchworks.merge_tile_labels` (or ``zarr_native_merge``).
+"""
+
+from __future__ import annotations
+
+import itertools
+from pathlib import Path
+from typing import Callable, Union
+
+import numpy as np
+import zarr
+
+
+def spatial_tiles(
+    shape: tuple[int, ...], tile_shape: tuple[int, ...]
+) -> list[tuple[slice, ...]]:
+    """Enumerate the tiles covering *shape*, in row-major order.
+
+    Parameters
+    ----------
+    shape : tuple of int
+        Spatial array shape.
+    tile_shape : tuple of int
+        Tile shape.
+
+    Returns
+    -------
+    list of tuple of slice
+        One slice tuple per tile (the same order ``estimate_empty_tiles``'s
+        ``occupancy`` grid uses when ravelled).
+    """
+    grids = [range(0, s, t) for s, t in zip(shape, tile_shape)]
+    return [
+        tuple(
+            slice(o, min(o + t, s))
+            for o, t, s in zip(starts, tile_shape, shape)
+        )
+        for starts in itertools.product(*grids)
+    ]
+
+
+def create_stage(
+    stage_path: Union[str, Path],
+    shape: tuple[int, ...],
+    tile_shape: tuple[int, ...],
+    *,
+    component: str = "staged",
+    dtype=np.int32,
+) -> str:
+    """Create the empty (zero-filled) shared stage store for tiled writes.
+
+    Parameters
+    ----------
+    stage_path : str or Path
+        Destination ``.zarr`` store.
+    shape : tuple of int
+        Full (spatial) array shape.
+    tile_shape : tuple of int
+        Chunk = tile shape (one chunk per tile, so jobs write disjoint files).
+    component : str, optional
+        Array name inside the store (default ``"staged"``).
+    dtype : data-type, optional
+        Label dtype (default ``int32``).
+
+    Returns
+    -------
+    str
+        The stage store path.
+    """
+    root = zarr.open_group(str(stage_path), mode="w")
+    root.create_array(
+        name=component, shape=shape, chunks=tile_shape, dtype=dtype
+    )
+    return str(stage_path)
+
+
+def stage_tile(
+    image,
+    fn: Callable[[np.ndarray], np.ndarray],
+    stage_path: Union[str, Path],
+    index: int,
+    *,
+    tile_shape: tuple[int, ...],
+    overlap: int = 0,
+    component: str = "staged",
+) -> int:
+    """Run *fn* on a single tile and write it into the shared stage store.
+
+    Reads the tile (expanded by *overlap* on every side for boundary context),
+    runs *fn*, trims the halo back off, and writes the result to the tile's
+    disjoint chunk of ``stage_path/component`` — so many of these can run
+    concurrently (one per job) without conflicts.
+
+    Parameters
+    ----------
+    image : array-like
+        The full image (dask/zarr/NumPy), indexable by slices.
+    fn : callable
+        ``(ndarray) -> ndarray`` returning integer labels of the same shape.
+    stage_path : str or Path
+        Stage store created by :func:`create_stage`.
+    index : int
+        Tile index into :func:`spatial_tiles`.
+    tile_shape : tuple of int
+        Tile shape (must match the stage store's chunks).
+    overlap : int, optional
+        Halo added on every side before calling *fn*.
+    component : str, optional
+        Array name inside the stage store.
+
+    Returns
+    -------
+    int
+        The processed tile *index*.
+    """
+    shape = image.shape
+    sl = spatial_tiles(shape, tile_shape)[index]
+    expanded, trims = [], []
+    for s, dim in zip(sl, shape):
+        lo = max(0, s.start - overlap)
+        hi = min(dim, s.stop + overlap)
+        expanded.append(slice(lo, hi))
+        trims.append((s.start - lo, hi - s.stop))
+    block = np.asarray(image[tuple(expanded)])
+    out = np.asarray(fn(block))
+    sel = tuple(
+        slice(left, out.shape[i] - right)
+        for i, (left, right) in enumerate(trims)
+    )
+    dst = zarr.open_group(str(stage_path), mode="r+")[component]
+    dst[sl] = out[sel].astype(dst.dtype)
+    return index

patchworks-0.11.0/tests/test_distributed.py

@@ -0,0 +1,45 @@
+"""Tests for the per-tile distributed building blocks."""
+
+import numpy as np
+
+from patchworks import (
+    create_stage,
+    merge_tile_labels,
+    spatial_tiles,
+    stage_tile,
+)
+
+
+def _fn(tile):
+    from skimage.measure import label
+
+    return label(tile > 0).astype("int32")
+
+
+def test_spatial_tiles_cover():
+    """Tiles tile the array exactly, in row-major order."""
+    tiles = spatial_tiles((4, 5), (2, 2))
+    assert len(tiles) == 2 * 3  # ceil(4/2) * ceil(5/2)
+    assert tiles[0] == (slice(0, 2), slice(0, 2))
+    assert tiles[-1] == (slice(2, 4), slice(4, 5))  # clipped last tile
+
+
+def test_stage_then_merge_stitches_boundary(tmp_path):
+    """Per-tile staging + merge reproduces a cross-boundary single object."""
+    img = np.zeros((16, 32), "uint16")
+    img[4:12, 8:24] = 500  # block straddling the x=16 tile boundary
+
+    stage = str(tmp_path / "stage.zarr")
+    tile = (16, 16)
+    create_stage(stage, img.shape, tile)
+    for i in range(len(spatial_tiles(img.shape, tile))):
+        stage_tile(img, _fn, stage, i, tile_shape=tile, overlap=4)
+
+    merged = merge_tile_labels(
+        stage,
+        write_to=str(tmp_path / "out.zarr"),
+        input_component="staged",
+        sequential_labels=True,
+    ).compute()
+    ids = np.unique(merged[merged > 0])
+    assert ids.size == 1, f"object split into {ids.size} labels"

patchworks-0.11.0/workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L2ltYWdlLnphcnI=

@@ -0,0 +1 @@
+{"rule": "convert", "input": [], "log": [], "shellcmd": null, "params": [], "code": "        \"../scripts/convert.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 7968324357465, "starttime": 1782285191.3091571, "endtime": 1782285191.3090107, "incomplete": false, "external_jobid": null, "input_checksums": {}}

patchworks-0.11.0/workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L2xhYmVscy5kb25l

@@ -0,0 +1 @@
+{"rule": "merge", "input": ["/tmp/toy/out/seg/0.done", "/tmp/toy/out/seg/1.done", "/tmp/toy/out/seg/3.done", "/tmp/toy/out/seg/4.done"], "log": [], "shellcmd": null, "params": [], "code": "        \"../scripts/merge.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 7968324177309, "starttime": 1782285198.9120235, "endtime": 1782285198.9118745, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/toy/out/seg/0.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/toy/out/seg/1.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/toy/out/seg/3.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", "/tmp/toy/out/seg/4.done": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}}

patchworks-0.11.0/workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3N0YWdlLnphcnI=

@@ -0,0 +1 @@
+{"rule": "prepare", "input": ["/tmp/toy/out/image.zarr"], "log": [], "shellcmd": null, "params": [], "code": "        \"../scripts/prepare_tiles.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 7968324357329, "starttime": 1782285193.0925362, "endtime": 1782285193.0922585, "incomplete": false, "external_jobid": null, "input_checksums": {}}

patchworks-0.11.0/workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3NlZy80LmRvbmU=

@@ -0,0 +1 @@
+{"rule": "segment", "input": ["/tmp/toy/out/image.zarr", "/tmp/toy/out/stage.zarr", "/tmp/toy/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": "        \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 7968316803128, "starttime": 1782285194.9512916, "endtime": 1782285194.9498596, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/toy/out/tiles.json": "sha256:a7ba7de7a100b9eed2f69158d779580251e48da2c4bdb722e28c4d9a79abb411"}}

patchworks-0.11.0/workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3NlZy8wLmRvbmU=

@@ -0,0 +1 @@
+{"rule": "segment", "input": ["/tmp/toy/out/image.zarr", "/tmp/toy/out/stage.zarr", "/tmp/toy/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": "        \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 7968316911209, "starttime": 1782285194.9791708, "endtime": 1782285194.9778593, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/toy/out/tiles.json": "sha256:a7ba7de7a100b9eed2f69158d779580251e48da2c4bdb722e28c4d9a79abb411"}}

patchworks-0.11.0/workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3NlZy8xLmRvbmU=

@@ -0,0 +1 @@
+{"rule": "segment", "input": ["/tmp/toy/out/image.zarr", "/tmp/toy/out/stage.zarr", "/tmp/toy/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": "        \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 7968316911362, "starttime": 1782285196.8732357, "endtime": 1782285196.871846, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/toy/out/tiles.json": "sha256:a7ba7de7a100b9eed2f69158d779580251e48da2c4bdb722e28c4d9a79abb411"}}

patchworks-0.11.0/workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3NlZy8zLmRvbmU=

@@ -0,0 +1 @@
+{"rule": "segment", "input": ["/tmp/toy/out/image.zarr", "/tmp/toy/out/stage.zarr", "/tmp/toy/out/tiles.json"], "log": [], "shellcmd": null, "params": [], "code": "        \"../scripts/segment_tile.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 7968316911515, "starttime": 1782285196.9122667, "endtime": 1782285196.9108458, "incomplete": false, "external_jobid": null, "input_checksums": {"/tmp/toy/out/tiles.json": "sha256:a7ba7de7a100b9eed2f69158d779580251e48da2c4bdb722e28c4d9a79abb411"}}

patchworks-0.11.0/workflow/.snakemake/metadata/L3RtcC90b3kvb3V0L3RpbGVzLmpzb24=

@@ -0,0 +1 @@
+{"rule": "prepare", "input": ["/tmp/toy/out/image.zarr"], "log": [], "shellcmd": null, "params": [], "code": "        \"../scripts/prepare_tiles.py\"\n", "record_format_version": 6, "conda_env": null, "container_img_url": null, "software_stack_hash": "d41d8cd98f00b204e9800998ecf8427e", "job_hash": 7968324357329, "starttime": 1782285193.0925362, "endtime": 1782285193.0908728, "incomplete": false, "external_jobid": null, "input_checksums": {}}

patchworks-0.11.0/workflow/README.md

@@ -0,0 +1,95 @@
+# patchworks Snakemake workflow
+
+A SLURM-ready pipeline that segments an arbitrarily large image and spreads the
+expensive Cellpose step across **many GPUs** — one tile per SLURM job.
+
+> **Full step-by-step guide:**
+> <https://imcf.one/patchworks/guide/snakemake/> — install, configure every
+> field, dry-run, local vs SLURM, monitoring, outputs and troubleshooting.
+
+```text
+convert ──▶ prepare (checkpoint) ──▶ segment {tile}  ──▶ merge
+                                     one GPU job/tile
+```
+
+## Why
+
+`tile_process` runs tiles serially on a single GPU. This workflow instead
+submits **one GPU job per (non-empty) tile**, so N GPUs cut the wall-time ~N×.
+Each job writes a disjoint chunk of a shared stage store; a final CPU job
+stitches labels across tile boundaries and writes them back into the image as a
+calibrated, multi-scale `labels/` group.
+
+## Steps
+
+1. **convert** — input (`.ims`, `.czi`, `.lif`, `.nd2`, OME-TIFF, `.zarr`) → a
+   pyramidal OME-ZARR (`image.zarr`).
+2. **prepare** (checkpoint) — plan tiles, skip background tiles, create the
+   empty `stage.zarr`, and list the tiles to segment.
+3. **segment {tile}** — one GPU job per tile: read tile + halo, run Cellpose,
+   trim, write to the stage. Scattered across the cluster.
+4. **merge** — zarr-native boundary stitch + renumber, written into
+   `image.zarr/labels/<name>/`.
+
+## Install
+
+```bash
+pip install "patchworks[workflow,cellpose,imaris,bioio]"
+# workflow → snakemake + the SLURM executor plugin
+```
+
+## Configure
+
+Edit `config/config.yaml` (input, output dir, channel, tile shape, Cellpose
+model/diameter/`do_3D`, …) and `profile/slurm/config.yaml` (partitions,
+account, GPU request).
+
+## Run
+
+```bash
+# locally (single machine)
+snakemake --cores 8 --configfile config/config.yaml
+
+# on SLURM — one GPU job per tile, up to `jobs:` in parallel
+snakemake --workflow-profile profile/slurm --configfile config/config.yaml
+```
+
+The GPU request lives in `profile/slurm/config.yaml` under
+`set-resources: segment:` (`--gres=gpu:1`). Raise `jobs:` to use more GPUs at
+once.
+
+## Output
+
+`<work_dir>/image.zarr` — the image plus `labels/<label_name>/` (multi-scale,
+calibrated). Open it directly:
+
+```python
+from patchworks.plugins.napari import view_in_napari
+view_in_napari("<work_dir>/image.zarr")   # auto-loads the labels
+```
+
+## Layout
+
+```text
+workflow/
+  Snakefile            # includes the rule files below
+  rules/               # convert.smk, segment.smk, merge.smk, common.smk
+  scripts/             # thin wrappers over patchworks' public API
+  config/config.yaml
+  profile/slurm/config.yaml
+```
+
+The rule scripts are intentionally thin — the work is done by patchworks'
+public API (`spatial_tiles`, `create_stage`, `stage_tile`, `merge_tile_labels`,
+`write_labels`), so the same per-tile distribution is available from your own
+code too.
+
+## Notes
+
+- Tiles overlap on read (halo) but write **disjoint** regions, so the per-tile
+  jobs are safe to run concurrently.
+- Background tiles are skipped (`skip_empty`), so only occupied tiles become
+  jobs.
+- `method:` selects `cellpose` (default) or a simple `threshold` (no GPU —
+  handy for testing or quick masks).
+- For very large stores, set `shard: true` in the config to cut the file count.

patchworks-0.11.0/workflow/Snakefile

@@ -0,0 +1,24 @@
+# patchworks Snakemake workflow.
+#
+#   convert ──▶ prepare (checkpoint) ──▶ segment {tile}  ──▶ merge
+#                                        one GPU job per tile
+#
+# Run locally:
+#   snakemake --cores 8 --configfile config/config.yaml
+# Run on SLURM (one GPU job per tile, many GPUs in parallel):
+#   snakemake --workflow-profile profile/slurm --configfile config/config.yaml
+
+import json
+from pathlib import Path
+
+configfile: "config/config.yaml"
+
+include: "rules/common.smk"
+include: "rules/convert.smk"
+include: "rules/segment.smk"
+include: "rules/merge.smk"
+
+
+rule all:
+    input:
+        f"{WORK}/labels.done",

patchworks-0.11.0/workflow/config/config.yaml

@@ -0,0 +1,42 @@
+# patchworks Snakemake workflow — configuration.
+# Copy and edit, then run:  snakemake --workflow-profile profile/slurm
+#
+# The expensive Cellpose step is split into one SLURM job per tile, so many
+# GPUs run in parallel; the merge is a single CPU job.
+
+# ---- input / output ---------------------------------------------------------
+input: "/path/to/scan.ims" # .ims, .czi, .lif, .nd2, ome-tiff, or an .zarr
+work_dir: "/path/to/results" # everything is written under here
+# Final outputs (under work_dir):
+#   image.zarr            — converted, pyramidal OME-ZARR
+#   image.zarr/labels/<name>/  — the segmentation (multi-scale), in the image
+
+# ---- conversion -------------------------------------------------------------
+reuse_pyramid: true # for .ims: copy its own pyramid (fast); else rebuild
+convert_chunks: null # null → patchworks' bounded auto chunks; or [c,z,y,x]
+shard: false # true → pack chunks into shards (zarr v3; fewer files)
+
+# ---- tiling -----------------------------------------------------------------
+channel: 0 # channel to segment
+level: 0 # pyramid level to segment (0 = full res)
+tile_shape: "auto" # "auto", or e.g. [16, 1024, 1024] (zyx)
+overlap: 30 # halo (≈ one object diameter)
+skip_empty: true # skip background tiles
+empty_threshold: null # null → Otsu; or a number
+
+# ---- segmentation -----------------------------------------------------------
+method: "cellpose" # "cellpose" (GPU) or "threshold" (simple, no GPU; testing)
+label_name: "cellpose"
+cellpose:
+  model: "cyto3"
+  diameter: 30
+  do_3D: true
+  gpu: true
+  # extra kwargs forwarded to model.eval(), e.g.:
+  # flow_threshold: 0.4
+  # cellprob_threshold: 0.0
+
+# ---- pyramid for the labels -------------------------------------------------
+pyramid_levels: 5
+pyramid_downscale: 2
+sequential_labels: true # renumber merged labels to a contiguous 1..N range

patchworks-0.11.0/workflow/profile/slurm/config.yaml

@@ -0,0 +1,40 @@
+# Snakemake 8+ SLURM profile for the patchworks workflow.
+# Needs: pip install "patchworks[workflow]"  (snakemake + slurm executor plugin)
+# Run:   snakemake --workflow-profile profile/slurm --configfile config/config.yaml
+#
+# Edit the partitions / account / GPU request for your cluster.
+
+executor: slurm
+jobs: 64 # max concurrent SLURM jobs (≈ how many GPUs you can grab at once)
+latency-wait: 60
+rerun-incomplete: true
+printshellcmds: true
+keep-going: true
+
+# Defaults for every rule (CPU jobs).
+default-resources:
+  slurm_partition: "cpu"
+  # slurm_account: "my_account"
+  mem_mb: 16000
+  cpus_per_task: 4
+  runtime: 60 # minutes
+
+# Per-rule overrides.
+set-resources:
+  convert:
+    mem_mb: 64000
+    cpus_per_task: 8
+    runtime: 360
+  prepare:
+    mem_mb: 32000
+    runtime: 120
+  segment: # one GPU per tile — this is what spreads Cellpose across GPUs
+    slurm_partition: "gpu"
+    slurm_extra: "'--gres=gpu:1'"
+    mem_mb: 32000
+    cpus_per_task: 4
+    runtime: 120
+  merge:
+    mem_mb: 128000
+    cpus_per_task: 8
+    runtime: 240

patchworks-0.11.0/workflow/rules/common.smk

@@ -0,0 +1,13 @@
+# Shared paths and helpers for the patchworks workflow.
+
+WORK = config["work_dir"]
+IMAGE = f"{WORK}/image.zarr"
+TILES = f"{WORK}/tiles.json"
+STAGE = f"{WORK}/stage.zarr"
+
+
+def occupied_done(wildcards):
+    """Per-tile markers for the occupied tiles (resolved after the checkpoint)."""
+    tiles = checkpoints.prepare.get().output.tiles
+    occupied = json.loads(Path(tiles).read_text())["occupied"]
+    return [f"{WORK}/seg/{i}.done" for i in occupied]

patchworks-0.11.0/workflow/rules/convert.smk

@@ -0,0 +1,7 @@
+# Convert the input to a pyramidal OME-ZARR.
+
+rule convert:
+    output:
+        directory(IMAGE),
+    script:
+        "../scripts/convert.py"

patchworks-0.11.0/workflow/rules/merge.smk

@@ -0,0 +1,9 @@
+# Stitch labels across tile boundaries and write them into the image.
+
+rule merge:
+    input:
+        occupied_done,
+    output:
+        touch(f"{WORK}/labels.done"),
+    script:
+        "../scripts/merge.py"

patchworks-0.11.0/workflow/rules/segment.smk

@@ -0,0 +1,22 @@
+# Plan tiles (checkpoint) and segment each tile on a GPU.
+
+checkpoint prepare:
+    input:
+        IMAGE,
+    output:
+        tiles=TILES,
+        stage=directory(STAGE),
+    script:
+        "../scripts/prepare_tiles.py"
+
+
+rule segment:
+    """Segment one tile on a GPU and write it into the stage store."""
+    input:
+        tiles=TILES,
+        stage=STAGE,
+        image=IMAGE,
+    output:
+        f"{WORK}/seg/{{index}}.done",
+    script:
+        "../scripts/segment_tile.py"

patchworks-0.11.0/workflow/scripts/_pw.py

@@ -0,0 +1,113 @@
+"""Config glue for the patchworks Snakemake workflow.
+
+The heavy lifting lives in patchworks' public API
+(``spatial_tiles``/``create_stage``/``stage_tile``/``merge_tile_labels``);
+these helpers only turn the Snakemake config into the right arguments.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+from patchworks import load_ome_zarr
+
+
+def open_image(work_dir, channel, level):
+    """Open the converted image for segmentation.
+
+    Parameters
+    ----------
+    work_dir : str or Path
+        Workflow output directory containing ``image.zarr``.
+    channel : int or None
+        Channel to select.
+    level : int
+        Pyramid level to read.
+
+    Returns
+    -------
+    da.Array
+        The (lazy) image array.
+    """
+    return load_ome_zarr(
+        str(Path(work_dir) / "image.zarr"), channel=channel, level=level
+    )
+
+
+def stage_path(work_dir):
+    """Path of the staged-labels store.
+
+    Parameters
+    ----------
+    work_dir : str or Path
+        Workflow output directory.
+
+    Returns
+    -------
+    str
+        ``<work_dir>/stage.zarr``.
+    """
+    return str(Path(work_dir) / "stage.zarr")
+
+
+def load_tiles_json(path):
+    """Load the tile manifest written by ``prepare_tiles.py``.
+
+    Parameters
+    ----------
+    path : str or Path
+        Path to ``tiles.json``.
+
+    Returns
+    -------
+    dict
+        The manifest (``tile_shape``, ``overlap``, ``occupied`` indices, …).
+    """
+    return json.loads(Path(path).read_text())
+
+
+def build_fn(cfg):
+    """Build the per-tile segmentation function from the config.
+
+    Parameters
+    ----------
+    cfg : dict
+        Snakemake config. ``method`` selects ``"cellpose"`` (default) or a
+        simple ``"threshold"`` (handy for testing / no-GPU runs).
+
+    Returns
+    -------
+    callable
+        ``(ndarray) -> ndarray`` returning integer labels.
+    """
+    method = cfg.get("method", "cellpose")
+    if method == "threshold":
+
+        def fn(tile):
+            from skimage.filters import threshold_otsu
+            from skimage.measure import label
+
+            thr = threshold_otsu(tile) if tile.max() > tile.min() else 0
+            return label(tile > thr).astype("int32")
+
+        return fn
+
+    if method == "cellpose":
+        from patchworks.plugins.cellpose import cellpose_fn
+
+        cp = cfg["cellpose"]
+        extra = {
+            k: v
+            for k, v in cp.items()
+            if k not in ("model", "diameter", "do_3D", "gpu")
+        }
+        return cellpose_fn(
+            cp.get("model", "cyto3"),
+            gpu=cp.get("gpu", True),
+            diameter=cp.get("diameter"),
+            do_3D=cp.get("do_3D", False),
+            **extra,
+        )
+
+    raise ValueError(f"unknown segmentation method: {method!r}")

patchworks-0.11.0/workflow/scripts/convert.py

@@ -0,0 +1,15 @@
+"""Snakemake script: convert the input to a pyramidal OME-ZARR."""
+
+from patchworks.plugins.ome_zarr import to_ome_zarr
+
+cfg = snakemake.config  # noqa: F821  (injected by Snakemake)
+chunks = cfg.get("convert_chunks")
+to_ome_zarr(
+    cfg["input"],
+    snakemake.output[0],  # noqa: F821
+    chunks=tuple(chunks) if chunks else None,
+    shard=bool(cfg.get("shard", False)),
+    reuse_pyramid=bool(cfg.get("reuse_pyramid", False)),
+    progress=False,
+    overwrite=True,
+)

patchworks-0.11.0/workflow/scripts/merge.py

@@ -0,0 +1,40 @@
+"""Snakemake script: merge the staged tiles into one labelled OME-ZARR.
+
+Runs patchworks' zarr-native boundary merge (stitches labels across tile
+boundaries, optionally renumbers them) and writes the result back into the
+image store under ``labels/<name>/`` as a calibrated, multi-scale pyramid.
+"""
+
+import shutil
+from pathlib import Path
+
+from patchworks import merge_tile_labels
+from patchworks.plugins.ome_zarr import write_labels
+
+from _pw import stage_path
+
+cfg = snakemake.config  # noqa: F821
+work_dir = cfg["work_dir"]
+image_store = str(Path(work_dir) / "image.zarr")
+merged_store = str(Path(work_dir) / "_merged.zarr")
+
+merged = merge_tile_labels(
+    stage_path(work_dir),
+    write_to=merged_store,
+    input_component="staged",
+    sequential_labels=cfg.get("sequential_labels", True),
+    progress=False,
+)
+group = write_labels(
+    image_store,
+    merged,
+    name=cfg.get("label_name", "labels"),
+    n_levels=int(cfg.get("pyramid_levels", 5)),
+    downscale=int(cfg.get("pyramid_downscale", 2)),
+    overwrite=True,
+)
+
+shutil.rmtree(merged_store, ignore_errors=True)
+shutil.rmtree(stage_path(work_dir), ignore_errors=True)
+print(f"[patchworks] labels written to {group}")
+open(snakemake.output[0], "w").close()  # noqa: F821

patchworks-0.11.0/workflow/scripts/prepare_tiles.py

@@ -0,0 +1,56 @@
+"""Snakemake script: plan tiles, create the empty stage store, list work."""
+
+import json
+from functools import partial
+from pathlib import Path
+
+from patchworks import (
+    auto_tile_shape_cellpose,
+    create_stage,
+    estimate_empty_tiles,
+    spatial_tiles,
+)
+
+from _pw import open_image, stage_path
+
+cfg = snakemake.config  # noqa: F821
+work_dir = cfg["work_dir"]
+image = open_image(work_dir, cfg["channel"], cfg["level"])
+
+ts = cfg.get("tile_shape", "auto")
+if ts == "auto":
+    cp = cfg["cellpose"]
+    tile_shape = tuple(
+        partial(
+            auto_tile_shape_cellpose,
+            do_3D=cp.get("do_3D", False),
+            use_gpu=cp.get("gpu", True),
+            diameter=cp.get("diameter"),
+        )(image.shape, image.dtype)
+    )
+else:
+    tile_shape = tuple(ts)
+
+tiles = spatial_tiles(image.shape, tile_shape)
+occupied = list(range(len(tiles)))
+if cfg.get("skip_empty", True):
+    info = estimate_empty_tiles(
+        image, tile_shape, threshold=cfg.get("empty_threshold")
+    )
+    occ = info["occupancy"].ravel()  # row-major, matches spatial_tiles
+    occupied = [i for i in range(len(tiles)) if occ[i]]
+
+create_stage(stage_path(work_dir), image.shape, tile_shape)
+
+Path(work_dir, "tiles.json").write_text(
+    json.dumps(
+        {
+            "tile_shape": list(tile_shape),
+            "overlap": int(cfg.get("overlap", 0)),
+            "n_tiles": len(tiles),
+            "occupied": occupied,
+        },
+        indent=2,
+    )
+)
+print(f"[patchworks] {len(occupied)}/{len(tiles)} tiles to segment")

patchworks-0.11.0/workflow/scripts/segment_tile.py

@@ -0,0 +1,28 @@
+"""Snakemake script: segment ONE tile and write it to the shared stage.
+
+Scattered over tile indices, so each tile is its own SLURM job and many GPUs
+run in parallel. Each job writes a disjoint chunk of the stage store.
+"""
+
+from patchworks import stage_tile
+
+from _pw import build_fn, load_tiles_json, open_image, stage_path
+
+cfg = snakemake.config  # noqa: F821
+index = int(snakemake.wildcards.index)  # noqa: F821
+work_dir = cfg["work_dir"]
+
+manifest = load_tiles_json(snakemake.input.tiles)  # noqa: F821
+image = open_image(work_dir, cfg["channel"], cfg["level"])
+
+stage_tile(
+    image,
+    build_fn(cfg),
+    stage_path(work_dir),
+    index,
+    tile_shape=tuple(manifest["tile_shape"]),
+    overlap=int(manifest["overlap"]),
+)
+
+open(snakemake.output[0], "w").close()  # noqa: F821
+print(f"[patchworks] segmented tile {index}")