patchworks 1.4.3__tar.gz → 2.0.0__tar.gz

{patchworks-1.4.3 → patchworks-2.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: patchworks
-Version: 1.4.3
+Version: 2.0.0
 Summary: Tiled processing of arbitrarily large images with globally consistent labels
 Project-URL: Homepage, https://github.com/imcf/patchworks
 Project-URL: Issues, https://github.com/imcf/patchworks/issues
@@ -376,9 +376,12 @@ tiles where the dask-image approach stalls.
 | ------------------------------ | ----------------------------------------- | ------------------------------------------------------------- |
 | In-process Dask client         | `FutureCancelledError: lost dependencies` | Detected at startup, raises immediately with fix instructions |
 | 3-4× fn recompute during merge | Cellpose runs 3× per tile                 | Staging writes labels once, merge reads from disk             |
-| O(n²) sequential relabelling   | Graph construction hangs at 1000+ tiles   | Linear post-pass O(voxels) via `np.unique` + LUT              |
+| O(n²) sequential relabelling   | Graph construction hangs at 1000+ tiles   | Folded into the merge's own LUT — no extra pass over the volume |
 | Wrong overlap boundary         | Output shape mismatch                     | Always uses `boundary="none"`                                 |
 | Persisting large arrays        | Worker OOM                                | Never persists; keeps dask graph lazy and streams             |
+| Sizing work to the whole node  | Job OOM-killed on a shared cluster node   | Reads the SLURM/cgroup allocation, not `os.cpu_count()`       |
+| Rechunking a pyramid level     | Threaded scheduler stockpiles intermediates | Levels stream one source chunk per task, bounded by construction |
+| Isotropic halo on flat tiles   | 5× the voxels read and segmented, then trimmed | `overlap` takes one width per axis                       |
 
 ---
 

{patchworks-1.4.3 → patchworks-2.0.0}/README.md

@@ -286,9 +286,12 @@ tiles where the dask-image approach stalls.
 | ------------------------------ | ----------------------------------------- | ------------------------------------------------------------- |
 | In-process Dask client         | `FutureCancelledError: lost dependencies` | Detected at startup, raises immediately with fix instructions |
 | 3-4× fn recompute during merge | Cellpose runs 3× per tile                 | Staging writes labels once, merge reads from disk             |
-| O(n²) sequential relabelling   | Graph construction hangs at 1000+ tiles   | Linear post-pass O(voxels) via `np.unique` + LUT              |
+| O(n²) sequential relabelling   | Graph construction hangs at 1000+ tiles   | Folded into the merge's own LUT — no extra pass over the volume |
 | Wrong overlap boundary         | Output shape mismatch                     | Always uses `boundary="none"`                                 |
 | Persisting large arrays        | Worker OOM                                | Never persists; keeps dask graph lazy and streams             |
+| Sizing work to the whole node  | Job OOM-killed on a shared cluster node   | Reads the SLURM/cgroup allocation, not `os.cpu_count()`       |
+| Rechunking a pyramid level     | Threaded scheduler stockpiles intermediates | Levels stream one source chunk per task, bounded by construction |
+| Isotropic halo on flat tiles   | 5× the voxels read and segmented, then trimmed | `overlap` takes one width per axis                       |
 
 ---
 

patchworks-2.0.0/docs/api/chunks.md

@@ -0,0 +1,19 @@
+# Tile sizing
+
+::: patchworks.auto_tile_shape
+
+::: patchworks.auto_tile_shape_cellpose
+
+::: patchworks.auto_overlap
+
+::: patchworks.normalize_overlap
+
+## Cluster resource detection
+
+On a shared node the machine's core count and free RAM say nothing about what
+this job was granted. These read the allocation instead, and everything that
+sizes a worker pool goes through them.
+
+::: patchworks.cpu_allocation
+
+::: patchworks.safe_worker_count

patchworks-2.0.0/docs/api/io.md

@@ -0,0 +1,19 @@
+# I/O helpers
+
+::: patchworks.load_ome_zarr
+
+## Deciding which tiles hold signal
+
+`estimate_empty_tiles` is a fast **preview** — it samples a centred window per
+tile, so it can miss signal at a tile's edge. `build_occupancy_map` +
+`tile_occupancy` are **exact**: a brick maximum exceeds the threshold exactly
+when some voxel in that brick does. Use the latter pair when the result is
+used as a skip list. See [Skipping empty tiles](../guide/skip_empty.md).
+
+::: patchworks.estimate_empty_tiles
+
+::: patchworks.build_occupancy_map
+
+::: patchworks.tile_occupancy
+
+::: patchworks.auto_empty_threshold

{patchworks-1.4.3 → patchworks-2.0.0}/docs/api/plugins/ome_zarr.md

@@ -24,3 +24,15 @@ resolution, matching anisotropic microscopy stacks.
 ## register_labels
 
 ::: patchworks.plugins.ome_zarr.register_labels
+
+## NGFF metadata layout
+
+NGFF 0.4 is defined over zarr v2 and puts its keys at the top level; 0.5 is
+the zarr-v3 revision and nests them under `ome`. patchworks writes whichever
+matches the store, and reads both.
+
+::: patchworks.plugins.ome_zarr.ngff_version
+
+::: patchworks.plugins.ome_zarr.read_ngff_attr
+
+::: patchworks.plugins.ome_zarr.write_ngff_attrs

{patchworks-1.4.3 → patchworks-2.0.0}/docs/guide/custom_segmentation.md

@@ -118,6 +118,18 @@ def segment(tile: np.ndarray, prob_thresh: float = 0.5) -> np.ndarray:
 
 Using a GPU? Just let your framework see it — nothing extra needed.
 
+!!! tip "You do not need to modify patchworks"
+    `method: "custom"` imports any `(tile) -> labels` callable, so a new
+    method is a module of your own and a config block — nothing in the
+    package changes. It then inherits everything the pipeline does: empty
+    tiles are skipped, tiles are batched per job, labels are stitched across
+    boundaries and renumbered, and the result is written as a calibrated
+    pyramid.
+
+    (The `KNOWN_METHODS` list in `workflow/scripts/_pw.py` is only for
+    *built-in* shortcuts like `"cellpose"`. Adding to it is for methods that
+    ship with patchworks, not for your own.)
+
 ## Test it before you submit
 
 Run your function on one real tile first — it catches shape/dtype bugs in
@@ -148,6 +160,9 @@ custom:
   module: "my_seg"        # import name
   function: "segment"     # default is "segment"
   kwargs:                 # optional — forwarded as segment(tile, **kwargs)
+  #                         checked against your function's signature during
+  #                         `prepare`, so a typo fails on a cheap CPU job
+  #                         rather than in the first GPU job hours later
     sigma: 1.5
 ```
 

{patchworks-1.4.3 → patchworks-2.0.0}/docs/guide/gpu_distributed.md

@@ -73,8 +73,16 @@ FutureCancelledError: lost dependencies
 ## GPU memory sizing
 
 When `use_gpu=True`, patchworks queries free GPU VRAM via `nvidia-ml-py`
-(install: `pip install "patchworks[gpu]"`). Tile size is set so each tile
-uses at most half the available VRAM.
+(install: `pip install "patchworks[gpu]"`) and keeps 20% of it as headroom,
+because `info.free` is a point-in-time reading of a device you usually do not
+own outright. `auto_tile_shape` then sizes each tile to at most half of that
+budget; `auto_tile_shape_cellpose` uses Cellpose's own memory model instead
+(roughly 20× the raw tile bytes, plus ~2 GiB for the model).
+
+The device is resolved from `CUDA_VISIBLE_DEVICES`. This matters on
+multi-GPU nodes: NVML enumerates **every** GPU regardless of `--gres=gpu:1`,
+so querying index 0 unconditionally would read a different card's free memory
+than the one your job was granted.
 
 Without `nvidia-ml-py`, a conservative 8 GiB default is used with a warning.
 Install it for accurate sizing on large-VRAM cards (A100, H100):
@@ -82,3 +90,18 @@ Install it for accurate sizing on large-VRAM cards (A100, H100):
 ```bash
 pip install "patchworks[gpu]"
 ```
+
+!!! tip "Sizing for a GPU you can't see"
+    The Snakemake workflow plans tiles in `prepare`, which runs on a CPU
+    node, so no GPU can be queried there. Set `gpu_memory_gb` in the config
+    to the segment GPU's VRAM (e.g. `24` for an RTX 4090) instead of relying
+    on the fallback.
+
+## Surviving a shared GPU
+
+An out-of-memory error on a shared device is often transient — a co-tenant
+job grew, not a tile that doesn't fit. patchworks retries on the GPU with a
+backoff rather than falling back to the CPU, where a single tile can take
+over an hour. Before each backoff it releases its own device memory
+(including any cached Cellpose model), since holding that is exactly what
+starves the other job. Both torch and cupy OOM errors are recognised.

{patchworks-1.4.3 → patchworks-2.0.0}/docs/guide/merging.md

@@ -28,9 +28,9 @@ This is the same approach used by
 
 ### Step 1: stage
 
-Each tile's labels are written to a temporary zarr once. This is critical:
-without staging, any downstream operation that reads the label array re-runs
-your segmentation function. The merge internally reads labels multiple times.
+Each tile's labels are written to zarr once. This is critical: without
+staging, any downstream operation that reads the label array re-runs your
+segmentation function. The merge internally reads labels multiple times.
 
 ```text
 tile_process calls fn once per tile → staged zarr
@@ -38,14 +38,37 @@ tile_process calls fn once per tile → staged zarr
                          merge reads from staged zarr (no fn calls)
 ```
 
-### Step 2: boundary scan
+The Snakemake workflow goes further and stages **directly into**
+`image.zarr/labels/<name>/0`, then has the merge rewrite that array in place
+— saving a whole extra write of the volume plus the scratch store's disk. It
+falls back to a separate store when the tile is larger than the label chunk
+cap, since in place the chunking cannot be changed and level 0 has to stay
+pageable for a viewer.
+
+### Step 2: make the ids globally unique
+
+Tiles write local `1..n`, which collide, so the boundary scan could not
+otherwise tell two different objects apart. If each tile's label **count** is
+known, this is just an exclusive cumulative sum — global id is
+`offset[tile] + local`, computed in `O(n_tiles)` with no read of the volume
+at all. `stage_tile` returns that count for exactly this purpose; pass the
+counts as `label_counts=`.
+
+Without counts, the merge falls back to streaming every chunk and renumbering
+it in place — correct, but a full read **and write** of the volume.
+
+### Step 3: boundary scan
 
 Only the two voxels on either side of each tile boundary are read. For any
-pair of touching non-zero labels `(a, b)`, they must be the same object.
+pair of touching non-zero labels `(a, b)`, they must be the same object. The
+per-tile offsets are applied here, on the fly.
 
-I/O cost: `O(n_boundaries × face_area)`, not `O(full_volume)`.
+I/O cost: `O(n_boundaries × face_area)`, not `O(full_volume)`. The columns
+are read in parallel, and a boundary next to a chunk that holds no labels is
+skipped outright — a pair needs a non-zero label on *both* sides, so it could
+never produce one.
 
-### Step 3: connected components
+### Step 4: connected components
 
 scipy sparse connected components on the touching pairs produces a relabeling
 lookup table. All labels that transitively touch each other are mapped to the
@@ -53,12 +76,38 @@ same canonical label.
 
 Cost: `O(n_touching_pairs)`.
 
-### Step 4: parallel relabel
+With `sequential_labels=True` the contiguous renumbering is folded into this
+same LUT. Because the id domain is dense by construction, the surviving ids
+are exactly the distinct LUT values — a `np.unique` over an array the length
+of the object count, with no scan of the volume.
+
+### Step 5: parallel relabel
 
 The LUT is applied to every tile in parallel via `multiprocessing.Pool`. The
 LUT is shared via process initializer to avoid re-pickling it for every chunk
 (LUTs can be hundreds of MB for dense label volumes).
 
+Chunks whose tile wrote no labels are skipped entirely — not read, and not
+written. Zarr never materialises an unwritten chunk and reads it back as the
+fill value, so background regions cost neither I/O nor disk. On a sparse
+image that is most of the volume.
+
+When the merge's output *is* its input, this pass rewrites the array in
+place. That is safe because the boundary scan (step 3) has already finished,
+so nothing still needs the original ids.
+
+Because an in-place merge destroys its own input, it records how far it got
+on the array itself, and refuses to guess on a re-run:
+
+| State found | What happens |
+| --- | --- |
+| nothing recorded | fresh tile-local ids — merge normally |
+| `running` | a previous attempt died mid-relabel, so the array is part local and part global. **Refuses**: re-segment to rebuild it. |
+| `done` | already merged — a no-op, so a failure *after* the relabel (the pyramid, say) can simply be retried |
+
+Without that, a second pass would add each tile's offset to ids that are
+already global, which can land two unrelated objects on the same id.
+
 ## Using the merge step standalone
 
 You can call the merge step directly on any existing label array or zarr:
@@ -86,9 +135,10 @@ merged = merge_tile_labels(
 
 ## Sequential label numbering
 
-By default, merged labels are globally unique but may be **gappy**
-(block-encoded IDs like 1, 2, 500001, 500002, …). This is fine for
-counting, `regionprops`, and measurement — the IDs just aren't consecutive.
+By default, merged labels are globally unique but may be **gappy** — boundary
+merging fuses ids, leaving holes where the absorbed ones were. This is fine
+for counting, `regionprops`, and measurement — the IDs just aren't
+consecutive.
 
 For contiguous 1..N numbering, use `sequential_labels=True`:
 
@@ -96,7 +146,9 @@ For contiguous 1..N numbering, use `sequential_labels=True`:
 tile_process("image.zarr", fn, write_to="labels.zarr", sequential_labels=True)
 ```
 
-This runs a cheap linear post-pass: `np.unique` + lookup-table remap, O(voxels).
+This is free: it composes into the relabel LUT the merge already applies, so
+it costs a `np.unique` over the object count rather than another pass over
+the volume.
 
 !!! warning "Do not use dask's built-in sequential relabel"
     `dask_image.ndmeasure.merge_labels_across_chunk_boundaries` has a

{patchworks-1.4.3 → patchworks-2.0.0}/docs/guide/ome_zarr_napari.md

@@ -22,9 +22,23 @@ natively. Pass `write_to="other.zarr"` to instead write a separate
 single-resolution label store, or `output_component="cells"` to name the label
 image.
 
-The label pyramid is built lazily (`da.to_zarr`, streamed chunk by chunk), so
-it stays OOM-safe even for terabyte volumes. Control it with `pyramid_levels`
-and `pyramid_downscale`.
+The label pyramid is streamed chunk by chunk, so it stays OOM-safe even for
+terabyte volumes. Control it with `pyramid_levels` and `pyramid_downscale`.
+
+!!! note "NGFF version"
+    The metadata layout follows the zarr format being written: **NGFF 0.5**
+    (keys nested under an `ome` attribute) on zarr v3, **0.4** (keys at the
+    top level) on v2. Writing v3 data with 0.4's top-level layout — which
+    earlier versions did — matches neither revision, and a strict 0.5 reader
+    finds nothing there.
+
+    Reading accepts **both** layouts, so stores written by any patchworks
+    version still load. Use `read_ngff_attr()` if you need to inspect the
+    metadata yourself rather than indexing `attrs["multiscales"]` directly.
+
+    patchworks' own hints (`n_objects`, `sequential_labels`) are not NGFF
+    keys, so they stay at the top level where a consumer finds them without
+    knowing the layout.
 
 ## Why a pyramid?
 
@@ -107,6 +121,21 @@ input — see [Cluster usage](snakemake.md).
     to_ome_zarr("scan.ims", "scan.zarr", reuse_pyramid=True)
     ```
 
+### Axis names
+
+For a file, axes come from the reader's own dimension metadata (bioio's
+`dims.order`, Imaris, or the `sequence_pattern` groups) — not guessed. Only
+non-spatial singleton axes are dropped, so a genuine `z=1` survives.
+
+For a **bare array** there is nothing to read, so axes are inferred from the
+number of dimensions (`yx`, `zyx`, `czyx`, `tczyx`). That is unambiguous up to
+3-D; from 4-D the leading axis could be channel or time, and patchworks logs a
+warning saying what it guessed. Pass `axes=` to settle it:
+
+```python
+to_ome_zarr(arr, "out.zarr", axes="tzyx")   # not the czyx it would assume
+```
+
 ### Pixel calibration
 
 The physical voxel size is read from the input — bioio's `physical_pixel_sizes`,

{patchworks-1.4.3 → patchworks-2.0.0}/docs/guide/performance.md

@@ -3,10 +3,10 @@
 `tile_process` is built so a run **adapts to whatever machine it lands on** and
 can't run out of RAM/VRAM or freeze the box — without you tuning anything.
 
-## Automatic, machine-aware concurrency
+## Automatic, allocation-aware concurrency
 
 The staging step (running your `fn` once per tile to a temp store) and the
-merge step are sized to the host automatically:
+merge step are sized automatically:
 
 - **GPU** (`use_gpu=True`) → **one tile at a time**, so concurrent evaluations
   can never exhaust VRAM.
@@ -14,8 +14,16 @@ merge step are sized to the host automatically:
   from the tile size), and always **leaving one core free** so the machine
   stays responsive — it never pins every core.
 
-The RAM figure is read live via `psutil`; without it, a conservative default is
-used instead of guessing high.
+"Available" means available **to this process**, not to the machine. On a
+shared cluster node those differ wildly — a 32-core, 128 GB job on a 128-core,
+512 GB node would otherwise size itself for the whole box and get OOM-killed
+while its own accounting said it had room. patchworks takes the smallest of
+`SLURM_MEM_PER_NODE`, `SLURM_MEM_PER_CPU × cpus`, the cgroup limit (the one
+that actually triggers the kill) and `psutil`'s free RAM, and reads the core
+count from `SLURM_CPUS_PER_TASK` or the process' CPU affinity mask.
+
+Without any of those signals, a conservative default is used instead of
+guessing high.
 
 ## Live progress dashboard (GPU runs)
 

{patchworks-1.4.3 → patchworks-2.0.0}/docs/guide/skip_empty.md

@@ -37,7 +37,9 @@ tile_process(
 )
 ```
 
-## How `estimate_empty_tiles` works
+## Two ways to decide
+
+### `estimate_empty_tiles` — a fast preview
 
 For each tile in the grid, only a small centred **sample window** is read
 (default: 24×256×256 voxels). If the maximum value in that window exceeds
@@ -47,11 +49,50 @@ This is **bounded I/O**: the total data read is `n_tiles × sample_window`,
 not the full image. For a 2200-tile image with the default window, this reads
 ≈ 30 MB instead of 250 GB — and it runs in seconds.
 
-!!! warning "Approximate"
-    `estimate_empty_tiles` inspects only the tile centre. Signal confined to
-    a tile's edge can be missed. The actual `tile_process` run always inspects
-    the **full tile** max inline — so no objects are ever dropped in the real
-    run, only in the preview.
+!!! warning "Approximate — a preview, not a skip list"
+    Only the tile centre is inspected. On a `(16, 1024, 1024)` tile the
+    default window covers **6.25% of the tile's area**, so an object in the
+    outer ring is invisible. `tile_process` re-tests the **full tile** max
+    inline, so nothing is dropped in a real run — but anything that uses this
+    result *as* the skip list would drop those tiles for good.
+
+### `build_occupancy_map` + `tile_occupancy` — exact
+
+Reduces every `block`-sized brick of the image to its **maximum** and stores
+that beside `image.zarr` (about 1/16384 of the image at the default 128 px
+block). `tile_occupancy` then reduces the map over each tile's full footprint.
+
+```python
+from patchworks import (
+    auto_empty_threshold, block_for_tile, build_occupancy_map, tile_occupancy
+)
+
+# Size the block from the tile, or every tile over-covers the same block and
+# tests occupied — correct, but useless as a skip list.
+build_occupancy_map("image.zarr", block=block_for_tile(TILE))
+info = tile_occupancy(
+    "image.zarr", TILE, channel=0, threshold=auto_empty_threshold(img, 0, 0)
+)
+```
+
+The map is stored **beside** the image (`image.occupancy.zarr`), not inside
+it: it is not an NGFF array, and a zarr hierarchy containing one cannot be
+walked. It is rebuilt automatically if an existing map was built at a
+different block.
+
+This is **exact, not approximate**: `block_max > threshold` is true exactly
+when some voxel in that block exceeds the threshold, so comparing pooled
+maxima against a threshold derived from raw voxels answers the same question
+as scanning every voxel. Max-pooling cannot lose a bright voxel.
+
+The map is built once and shared by every segmentation reading that image, so
+a three-config run pays for one pooling pass instead of three sampling passes.
+This is what the Snakemake workflow uses, and it builds the map on first use
+for stores converted before the map existed.
+
+Skipping a tile pays off twice over: it is never segmented, **and** the merge
+skips its chunk too — neither reading it nor writing zeros over it, so the
+background never reaches disk at all.
 
 ## Threshold selection
 

{patchworks-1.4.3 → patchworks-2.0.0}/docs/guide/snakemake.md

@@ -57,9 +57,10 @@ channel: 0                     # channel to segment (null = keep all)
 level: 0                       # pyramid level (0 = full resolution)
 tile_shape: "auto"             # "auto", or e.g. [16, 1024, 1024] (zyx)
 gpu_memory_gb: null            # for "auto" on SLURM: your segment GPU's VRAM
-overlap: 30                    # halo ≈ one object diameter
+overlap: [4, 30, 30]           # halo ≈ one object diameter; scalar or [z,y,x]
 skip_empty: true               # skip background tiles
 empty_threshold: null          # null → Otsu
+tiles_per_job: 4               # tiles per SLURM job (sequential, shared model)
 
 # segmentation
 method: "cellpose"             # "cellpose" (GPU), "threshold" (no GPU), "custom"
@@ -102,6 +103,29 @@ sequential_labels: true        # renumber labels to a contiguous 1..N
     8 GiB. Harmless, but to size for the real GPU set `gpu_memory_gb:` to its
     VRAM (e.g. `24`, `40`, `80`) — or just set `tile_shape` explicitly.
 
+    With `do_3D: true`, `"auto"` pins z to the **full** extent: no z tiling,
+    no z-boundary stitching, and Cellpose sees each object's whole depth. An
+    explicit z (like `[16, 1024, 1024]`) tiles in z instead. `prepare` logs
+    which regime it picked.
+
+!!! tip "Use a per-axis `overlap`"
+    A scalar halo is applied to every axis. On a `[16, 1024, 1024]` tile,
+    `overlap: 30` reads `76 × 1084 × 1084` to keep `16 × 1024 × 1024` — 5.3×
+    the voxels it uses, nearly all of it wasted z. `[4, 30, 30]` brings that
+    to ~1.7×, i.e. roughly **3× less GPU time for identical results**.
+    `prepare` logs the amplification and refuses a halo wider than the tile.
+
+!!! tip "Batch tiles per job with `tiles_per_job`"
+    Each job pays CUDA init plus a Cellpose weight load — tens of seconds —
+    before its first voxel. `tiles_per_job: N` segments N tiles sequentially
+    in one process, sharing one loaded model, which is most of the win on
+    fast tiles.
+
+    Tiles in a batch run **sequentially**, so job wall time is roughly
+    `N × per-tile time` and must stay inside the profile's `runtime`/QOS
+    ceiling. A retry re-runs the whole batch. Measure one tile with
+    `seff <jobid>` and size N from that; `1` restores one job per tile.
+
 ## 4. Dry-run (always do this first)
 
 Check the plan without running anything:
@@ -126,7 +150,7 @@ image or a smoke test. `--rerun-triggers mtime` re-runs only steps whose output
 is missing/stale — so upgrading patchworks won't redo the conversion (the SLURM
 profile sets this for you).
 
-## 5b. Run on SLURM (one GPU job per tile)
+## 5b. Run on SLURM (one GPU job per batch of tiles)
 
 Edit `profile/slurm/config.yaml` for **your** cluster — partitions, account,
 and the GPU request:
@@ -140,14 +164,15 @@ default-resources:
   mem_mb: 16000
   cpus_per_task: 4
   runtime: 60
+retries: 2                     # resubmit a failed job, asking for more memory
 set-resources:
   segment:                     # the GPU step
     slurm_partition: "gpu"     # your GPU partition
     slurm_extra: "'--gres=gpu:1'"
-    mem_mb: 32000
+    mem_mb: "attempt * 32000"  # grows on each retry
     runtime: 120
   merge:
-    mem_mb: 128000
+    mem_mb: "attempt * 64000"
     runtime: 240
 ```
 
@@ -159,8 +184,14 @@ python -m snakemake --workflow-profile profile/slurm \
 ```
 
 Snakemake submits `convert`, then `prepare`, then **one `segment` job per
-non-empty tile** (up to `jobs:` at once → that many GPUs in parallel), then
-`merge`. Raise `jobs:` to use more GPUs.
+batch of `tiles_per_job` non-empty tiles** (up to `jobs:` at once → that many
+GPUs in parallel), then `merge`. Raise `jobs:` to use more GPUs.
+
+!!! tip "Sizing memory"
+    Every step now sizes its own worker counts from what SLURM actually
+    granted (`SLURM_CPUS_PER_TASK`, `SLURM_MEM_PER_*`, the cgroup limit)
+    rather than the node's totals, and `merge` logs the budget it detected.
+    Compare that line against `seff <jobid>` when tuning `mem_mb`.
 
 !!! note "GPU request flag"
     Clusters differ. `--gres=gpu:1` is common; some need `--gpus=1` or a
@@ -211,8 +242,8 @@ result. Keep `mtime` and reruns happen only when an output is missing or stale.
 
 ## Running two segmentations (e.g. nuclei + cytoplasm)
 
-Every path the workflow writes — `tiles.json`, `stage.zarr`, per-tile `seg/`,
-the cached model, `labels.done` — lives under `work_dir/<label_name>/`, so
+Every path the workflow writes — `tiles.json`, per-batch `seg/` markers, the
+cached model, `labels.done` — lives under `work_dir/<label_name>/`, so
 running the workflow **twice with two configs against the same `work_dir`**
 never collides: each run gets its own private subdirectory, and both reuse
 the *same* already-converted `image.zarr` (conversion never re-runs).
@@ -243,18 +274,22 @@ cellpose:
   do_3D: true
 ```
 
-Run them one after another (or as two independent SLURM submissions, even
-concurrently — they touch disjoint files):
+Run them as two independent SLURM submissions — they touch disjoint files, so
+they can run concurrently. Give each its own `--directory`, because
+Snakemake's lock lives in the working directory, not in the config:
 
 ```bash
-snakemake --workflow-profile profile/slurm --configfile config/config_nuclei.yaml
-snakemake --workflow-profile profile/slurm --configfile config/config_cyto.yaml
+snakemake --workflow-profile profile/slurm --configfile config/config_nuclei.yaml \
+          --directory /scratch/results/nuclei_labels/.snakemake
+snakemake --workflow-profile profile/slurm --configfile config/config_cyto.yaml \
+          --directory /scratch/results/cyto_labels/.snakemake
 ```
 
 !!! tip "One command for several segmentations + relations"
     `config/multi.yaml` lists any number of segmentation configs plus which
-    pairs to relate afterward; `pixi run multi` (or `multi-slurm`) runs them
-    in order and writes a CSV per pair — see *One command: multiple
+    pairs to relate afterward; `pixi run multi` (or `multi-slurm`) converts
+    once, then runs every config **concurrently** with its own state
+    directory, and writes a workbook per pair — see *One command: multiple
     segmentations + relations* below for the config format.
 
 Both land side by side in the same store:
@@ -276,10 +311,10 @@ workflow's own automation is below.
 
 ### One command: multiple segmentations + relations
 
-`scripts/run_multi.py` (wired up as `pixi run multi`) sequences the above
-manually: run every segmentation config listed, then compute + save every
-configured relation — one command instead of juggling several `snakemake`
-calls and a separate Python step.
+`scripts/run_multi.py` (wired up as `pixi run multi`) drives the above: it
+converts once, runs every segmentation config **concurrently**, then computes
+and saves every configured relation — one command instead of juggling several
+`snakemake` calls and a separate Python step.
 
 ```yaml
 # config/multi.yaml
@@ -299,10 +334,22 @@ pixi run multi        # run locally
 pixi run multi-slurm  # submit every segmentation to SLURM
 ```
 
-Every listed segmentation config must share the same `work_dir` (so
-`label_relations` has one `image.zarr` to read both label groups from) — the
-script checks this and errors out otherwise. `relations` is optional; omit it
-to just chain segmentations without a relation step.
+Before anything is submitted, the script checks that every listed config
+shares one `work_dir` (so `label_relations` has a single `image.zarr` to read
+both label groups from), that `tile_shape` and `level` are identical (so the
+label arrays share a chunk layout), and that `label_name` is unique — two
+configs sharing one would silently overwrite each other's outputs. `relations`
+is optional; omit it to just run segmentations without a relation step.
+
+The configs then run at the same time, each with its own state directory, so
+the GPU partition stays busy instead of idling through every config's
+`prepare` and multi-hour `merge` in turn. A config that fails does **not**
+abort the others; you get a per-config status and a non-zero exit.
+
+!!! warning "`jobs:` is per config"
+    The profile's `jobs:` caps one Snakemake process. Running three configs
+    concurrently can therefore have 3× that many jobs in flight — lower it if
+    that would exceed your cluster quota.
 
 Each `output:` is an Excel workbook (`openpyxl`, part of the `workflow`
 extra) with two sheets:
@@ -422,13 +469,34 @@ from patchworks import (
 )
 from patchworks.plugins.ome_zarr import write_labels
 
+TILE = (16, 1024, 1024)
 img = load_ome_zarr("image.zarr", channel=0)
-tiles = spatial_tiles(img.shape, tile_shape=(16, 1024, 1024))
-create_stage("stage.zarr", img.shape, (16, 1024, 1024))
-# (distribute these across jobs:)
+tiles = spatial_tiles(img.shape, tile_shape=TILE)
+create_stage("stage.zarr", img.shape, TILE)
+
+# (distribute these across jobs:) each returns how many labels it wrote
+counts = {}
 for i in range(len(tiles)):
-    stage_tile(img, my_fn, "stage.zarr", i, tile_shape=(16, 1024, 1024), overlap=30)
+    counts[i] = stage_tile(
+        img, my_fn, "stage.zarr", i, tile_shape=TILE, overlap=[4, 30, 30]
+    )
+
+# Handing those counts to the merge lets it derive each tile's global id range
+# by a cumulative sum, instead of streaming the whole store to renumber it.
 merged = merge_tile_labels("stage.zarr", input_component="staged",
-                           write_to="merged.zarr", sequential_labels=True)
+                           write_to="merged.zarr", sequential_labels=True,
+                           label_counts=counts)
 write_labels("image.zarr", merged, name="cells")
 ```
+
+The workflow goes two steps further. It stages **into**
+`image.zarr/labels/cells/0` in the first place, then has the merge rewrite
+that array **in place** (`write_to=` and `input_component=`/
+`output_component=` all pointing at it) and calls `register_labels` to add
+the pyramid. That removes both the scratch store and the full-volume copy
+`write_labels` would otherwise do — three passes over the label volume become
+two.
+
+It falls back to the separate store above when the tile is larger than the
+label chunk cap, because in place the chunking cannot be changed and level 0
+has to stay pageable for a viewer.