patchworks 2.1.2__tar.gz → 2.3.0__tar.gz

{patchworks-2.1.2 → patchworks-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: patchworks
-Version: 2.1.2
+Version: 2.3.0
 Summary: Tiled processing of arbitrarily large images with globally consistent labels
 Project-URL: Homepage, https://github.com/imcf/patchworks
 Project-URL: Issues, https://github.com/imcf/patchworks/issues

{patchworks-2.1.2 → patchworks-2.3.0}/cliff.toml

@@ -31,6 +31,8 @@ commit_preprocessors = [
   { pattern = "(\\w+(?:\\([^)]*\\))?!?:\\s+)[^\\w\\s]+\\s*", replace = "$1" },
 ]
 commit_parsers = [
+  # Merge commits are noise in a changelog — the merged commits are already listed.
+  { message = "^Merge ", skip = true },
   { message = "^feat", group = "✨ Features" },
   { message = "^fix", group = "🐛 Bug Fixes" },
   { message = "^perf", group = "⚡ Performance" },

{patchworks-2.1.2 → patchworks-2.3.0}/docs/guide/label_relations.md

@@ -24,7 +24,7 @@ table[2]
 `table` only contains matched `a` labels (nuclei with at least one
 overlapping voxel in `cells`) — unmatched labels and full per-`b` coverage
 need a bit more bookkeeping (the [cluster workflow's `run_multi.py`
-script](snakemake.md#one-command-multiple-segmentations--relations) does
+script](snakemake.md#one-command-multiple-segmentations-relations) does
 this for you and writes it as a two-sheet workbook).
 
 Save it as a table yourself:
@@ -41,4 +41,4 @@ with open("nuclei_to_cell.csv", "w", newline="") as f:
 
 On the cluster, producing the two label stores in the first place is a
 matter of running the workflow twice against the same `work_dir` — see
-[Running two segmentations](snakemake.md#running-two-segmentations-eg-nuclei--cytoplasm).
+[Running two segmentations](snakemake.md#running-two-segmentations-eg-nuclei-cytoplasm).

{patchworks-2.1.2 → patchworks-2.3.0}/docs/guide/measurements.md

@@ -10,7 +10,7 @@ is built for this — its "Measure" dock widget computes area/centroid/intensity
 stats directly off a Labels layer's dask/zarr-backed array, out-of-core, and
 scales with chunk count rather than object count. It's the best fit for
 measuring *every* object in a store this size, not just a cropped region —
-see [View image + labels in napari](ome_zarr_napari.md#view-image--labels-in-napari).
+see [View image + labels in napari](ome_zarr_napari.md#view-image-labels-in-napari).
 Bundled in `patchworks[napari]`.
 
 For interactively inspecting individual cells by clicking in the viewer (not

{patchworks-2.1.2 → patchworks-2.3.0}/docs/guide/snakemake.md

@@ -187,6 +187,27 @@ Snakemake submits `convert`, then `prepare`, then **one `segment` job per
 batch of `tiles_per_job` non-empty tiles** (up to `jobs:` at once → that many
 GPUs in parallel), then `merge`. Raise `jobs:` to use more GPUs.
 
+!!! tip "Recognisable job names in `squeue`"
+    The SLURM executor names every job after its run UUID and **rejects** a
+    `--job-name` in `slurm_extra`, so by default `squeue` shows nothing you
+    can identify. A prefix is the supported lever, and it goes first in the
+    name (`<prefix>_<uuid>`) — the part a queue listing truncates to:
+
+    ```yaml
+    slurm-jobname-prefix: patchworks    # already in the shipped profile
+    ```
+
+    `run_multi` overrides it per config, so a three-way run shows
+    `pw-convert`, then `pw-nuclei_labels` / `pw-cyto_labels` /
+    `pw-cilia_labels` — telling the concurrent runs apart at a glance:
+
+    ```bash
+    squeue -u $USER -o '%.18i %.24j %.8T %.10M'
+    ```
+
+    Alphanumerics, underscores and hyphens only, 50 characters max; an
+    invalid prefix fails the run, so `label_name` is sanitised before use.
+
 !!! tip "Sizing memory"
     Every step now sizes its own worker counts from what SLURM actually
     granted (`SLURM_CPUS_PER_TASK`, `SLURM_MEM_PER_*`, the cgroup limit)
@@ -248,13 +269,26 @@ running the workflow **twice with two configs against the same `work_dir`**
 never collides: each run gets its own private subdirectory, and both reuse
 the *same* already-converted `image.zarr` (conversion never re-runs).
 
+Most of what those configs contain is identical — the input, the `work_dir`,
+the tiling, everything `convert` reads. Put it in **one** shared file and let
+each config carry only what actually differs. Snakemake merges several
+`--configfile` values in order, with the later one winning:
+
 ```yaml
-# config/config_nuclei.yaml
+# config/common.yaml — shared by every segmentation
 input: "/data/scan.ims"
 work_dir: "/scratch/results"
+tile_shape: [16, 1024, 1024]
+shard: false                   # true → far fewer files, same chunks
+tiles_per_job: 4
+```
+
+```yaml
+# config/config_nuclei.yaml — only the differences
 label_name: "nuclei_labels"
 channel: 1                # nuclear stain channel
-tile_shape: [16, 1024, 1024]
+overlap: [4, 30, 30]
+method: "cellpose"
 cellpose:
   model: "nuclei"
   diameter: 15
@@ -262,12 +296,11 @@ cellpose:
 ```
 
 ```yaml
-# config/config_cyto.yaml
-input: "/data/scan.ims"
-work_dir: "/scratch/results"   # same work_dir — image.zarr is reused
+# config/config_cyto.yaml — only the differences
 label_name: "cyto_labels"
 channel: 0                # cytoplasm/membrane channel
-tile_shape: [16, 1024, 1024]   # keep this identical across configs — see below
+overlap: [4, 30, 30]
+method: "cellpose"
 cellpose:
   model: "cyto3"
   diameter: 30
@@ -279,12 +312,23 @@ they can run concurrently. Give each its own `--directory`, because
 Snakemake's lock lives in the working directory, not in the config:
 
 ```bash
-snakemake --workflow-profile profile/slurm --configfile config/config_nuclei.yaml \
-          --directory /scratch/results/nuclei_labels/.snakemake
-snakemake --workflow-profile profile/slurm --configfile config/config_cyto.yaml \
-          --directory /scratch/results/cyto_labels/.snakemake
+snakemake --workflow-profile profile/slurm --configfile config/common.yaml config/config_nuclei.yaml --directory /scratch/results/nuclei_labels/.snakemake
+```
+
+```bash
+snakemake --workflow-profile profile/slurm --configfile config/common.yaml config/config_cyto.yaml --directory /scratch/results/cyto_labels/.snakemake
 ```
 
+!!! warning "Conversion settings belong in the shared file"
+    `convert` runs **once**, from the first config only. A `shard`, `input` or
+    `pyramid_levels` set on the second config is therefore never read, and
+    nothing logs that it was dropped. `run_multi` refuses to start when those
+    keys disagree across configs and tells you which one — but if you drive
+    the configs by hand, keep them in `common.yaml`.
+
+    Splitting the configs is optional: a self-contained config still works,
+    and `common:` can simply be left out of `multi.yaml`.
+
 !!! tip "One command for several segmentations + relations"
     `config/multi.yaml` lists any number of segmentation configs plus which
     pairs to relate afterward; `pixi run multi` (or `multi-slurm`) converts
@@ -318,6 +362,8 @@ and saves every configured relation — one command instead of juggling several
 
 ```yaml
 # config/multi.yaml
+common: config/common.yaml # shared settings, merged under each config below
+
 segmentations:
   - config/config_nuclei.yaml
   - config/config_cyto.yaml
@@ -328,6 +374,10 @@ relations:
     output: nuclei_to_cyto.xlsx # written into work_dir
 ```
 
+`common:` is optional — leave it out and each config must be self-contained,
+as before. With it, changing the input path or turning on `shard` is a
+one-line edit in one file instead of the same edit repeated per config.
+
 ```bash
 pixi run multi-dry    # dry-run every segmentation config (skips relations)
 pixi run multi        # run locally

patchworks-2.3.0/tests/test_run_multi.py

@@ -0,0 +1,123 @@
+"""Tests for the multi-config driver's SLURM-facing behaviour."""
+
+import re
+import sys
+from pathlib import Path
+
+sys.path.insert(
+    0, str(Path(__file__).resolve().parents[1] / "workflow" / "scripts")
+)
+
+import pytest  # noqa: E402
+import yaml  # noqa: E402
+
+from run_multi import (  # noqa: E402
+    _CONVERT_KEYS,
+    _snakemake_cmd,
+    _validate_configs,
+    slurm_jobname_prefix,
+)
+
+# The SLURM executor's own rule (snakemake_executor_plugin_slurm): it raises a
+# WorkflowError and aborts the whole run if the prefix does not match.
+_EXECUTOR_RULE = re.compile(r"^[A-Za-z0-9_-]{1,50}$")
+
+
+def test_jobname_prefix_satisfies_the_executor():
+    """Whatever a label_name contains, the prefix must stay submittable.
+
+    The executor names jobs after a UUID and refuses a --job-name override, so
+    this prefix is the only thing that makes squeue readable -- and an invalid
+    one fails the run rather than degrading.
+    """
+    for label in ("nuclei_labels", "cyto_labels", "convert", "a"):
+        assert _EXECUTOR_RULE.match(slurm_jobname_prefix(label))
+
+    # Characters a label might plausibly pick up are sanitised, not passed on.
+    assert _EXECUTOR_RULE.match(slurm_jobname_prefix("cilia/v2 (test)"))
+    assert _EXECUTOR_RULE.match(slurm_jobname_prefix("run 1: nuclei"))
+    # And an over-long label is truncated to the executor's 50-char limit.
+    assert _EXECUTOR_RULE.match(slurm_jobname_prefix("x" * 200))
+
+
+def test_jobname_prefix_keeps_the_label_readable():
+    """The label must lead, since that is what a queue listing truncates to."""
+    assert slurm_jobname_prefix("nuclei_labels") == "pw-nuclei_labels"
+    assert slurm_jobname_prefix("convert") == "pw-convert"
+
+
+def test_common_configfile_is_merged_under_the_per_config_one():
+    """Snakemake merges --configfile values in order, later winning.
+
+    That ordering is the whole mechanism: shared settings come from common.yaml
+    and the per-config file overrides only what differs. Swap the two and every
+    config would silently get the shared defaults instead of its own channel.
+    """
+    cmd = _snakemake_cmd(
+        Path("config/config_nuclei.yaml"),
+        workflow_dir=Path("workflow"),
+        profile=None,
+        cores=8,
+        dry_run=False,
+        common=Path("config/common.yaml"),
+    )
+    i = cmd.index("--configfile")
+    assert cmd[i + 1].endswith("common.yaml")
+    assert cmd[i + 2].endswith("config_nuclei.yaml")
+
+    # Without a common file the invocation is unchanged: one configfile, so
+    # a self-contained config keeps working exactly as before.
+    plain = _snakemake_cmd(
+        Path("config/config_nuclei.yaml"),
+        workflow_dir=Path("workflow"),
+        profile=None,
+        cores=8,
+        dry_run=False,
+    )
+    j = plain.index("--configfile")
+    assert plain[j + 1].endswith("config_nuclei.yaml")
+    assert not plain[j + 2].endswith(".yaml")
+
+
+def test_convert_keys_must_agree_across_configs():
+    """`convert` runs once from the first config, so a later one is ignored.
+
+    Setting shard on the second config and watching a million files appear
+    anyway is invisible without this check -- there is no log line saying the
+    value was dropped, because nothing ever read it.
+    """
+    paths = [Path("a.yaml"), Path("b.yaml")]
+    base = {"work_dir": "/w", "tile_shape": [16, 512, 512], "level": 0}
+    good = [
+        {**base, "label_name": "a", "shard": True},
+        {**base, "label_name": "b", "shard": True},
+    ]
+    assert _validate_configs(paths, good) == "/w"
+
+    bad = [
+        {**base, "label_name": "a", "shard": True},
+        {**base, "label_name": "b", "shard": False},
+    ]
+    # It reports every problem and exits, rather than raising, so that a
+    # mistake costs one readable message instead of a traceback.
+    with pytest.raises(SystemExit):
+        _validate_configs(paths, bad)
+
+
+def test_shipped_multi_configs_are_consistent():
+    """The shipped example must satisfy its own validator.
+
+    It is the thing users copy, so a config set that run_multi would refuse to
+    start is worse than no example at all.
+    """
+    cfg_dir = Path(__file__).resolve().parents[1] / "workflow" / "config"
+    multi = yaml.safe_load((cfg_dir / "multi.yaml").read_text())
+    common = yaml.safe_load((cfg_dir.parent / multi["common"]).read_text())
+    paths = [cfg_dir.parent / p for p in multi["segmentations"]]
+    cfgs = [{**common, **yaml.safe_load(p.read_text())} for p in paths]
+
+    assert _validate_configs(paths, cfgs) == common["work_dir"]
+    # Every key convert reads comes from the shared file, not a per-config one.
+    for path in paths:
+        own = yaml.safe_load(path.read_text())
+        assert not set(own) & set(_CONVERT_KEYS), path.name

patchworks-2.3.0/workflow/config/common.yaml

@@ -0,0 +1,42 @@
+# Settings shared by every segmentation in config/multi.yaml.
+#
+# `common:` in multi.yaml points here, and run_multi passes it as the first of
+# two --configfile values. Snakemake merges them in order with the later
+# winning, so a per-config file only carries what actually differs from this
+# one — the channel it reads, the method it uses, its label_name.
+#
+# Everything `convert` reads MUST live here rather than in a per-config file:
+# the conversion runs once, up front, from the first config only, so a `shard`
+# or `input` set on the second config would be silently ignored. run_multi
+# refuses to start if those keys disagree across configs.
+
+input: "/path/to/scan.ims"
+work_dir: "/path/to/results"
+
+# --- conversion (read once, in phase A, from the first config only) ---
+reuse_pyramid: false
+convert_chunks: null
+# true → pack chunks into shards: same chunking and same memory, far fewer
+# files. Worth turning on for anything large; convert warns when a store is
+# heading past ~200,000 chunks, which a shared filesystem will not enjoy.
+shard: false
+
+# --- label pyramid (read by merge, i.e. once per config) ---
+# Shared here for convenience, but a per-config file may override these: each
+# segmentation builds its own label pyramid.
+pyramid_levels: 5
+pyramid_downscale: 2
+
+# --- tiling (must match across configs so label arrays stay comparable) ---
+level: 0
+tile_shape: [16, 1024, 1024]
+gpu_memory_gb: null
+skip_empty: true
+empty_threshold: null
+# Tiles per SLURM job; they run sequentially and share one model load. Raise
+# once you know a tile's runtime -- job wall time is ~N x per-tile time.
+tiles_per_job: 4
+
+# --- merge ---
+sequential_labels: true
+merge_workers: null

{patchworks-2.1.2 → patchworks-2.3.0}/workflow/config/config_cilia.yaml

@@ -1,37 +1,24 @@
 # Example segmentation config: cilia channel, via method: "custom" ->
 # patchworks.plugins.dog.segment (deconvolution + difference-of-Gaussians).
 # Cilia are thin/small structures a cell-body model like Cellpose isn't
-# shaped for. Paired with config_cyto.yaml / config_nuclei.yaml via
-# config/multi.yaml to relate cilia -> containing cell / nucleus. Same
-# work_dir and tile_shape as those two so patchworks.label_relations() can
-# compare the label arrays chunk-for-chunk.
+# shaped for. Related to cyto/nuclei via config/multi.yaml to map each
+# cilium to its containing cell / nucleus.
+#
+# Only what differs from config/common.yaml — the input, work_dir, tiling and
+# conversion settings all live there and are merged in ahead of this file.
 #
 # pycudadecon is CUDA-only, so this segment job needs a GPU too — same
 # profile/slurm as the Cellpose configs already covers it.
 
-input: "/path/to/scan.ims"
-work_dir: "/path/to/results"
-
-reuse_pyramid: false
-convert_chunks: null
-shard: false
-
 channel: 2 # cilia marker channel
-level: 0
-tile_shape: [16, 1024, 1024] # keep identical to config_cyto.yaml/config_nuclei.yaml
-gpu_memory_gb: null
+
 # Per-axis halo [z, y, x], covering the PSF support (decon) + the DoG's
 # high_sigma. Lateral 30 px at dxdata 0.1 = 3 um; at dzdata 0.2 that is 15
 # z-planes -- almost the whole 16-plane tile. Deconvolution genuinely wants a
-# deeper z tile than this: if cilia quality matters, raise tile_shape's z for
-# all three configs together (label_relations needs them identical) rather
-# than pushing the halo up against the tile depth.
+# deeper z tile than this: if cilia quality matters, raise tile_shape's z in
+# common.yaml (label_relations needs it identical across configs) rather than
+# pushing the halo up against the tile depth.
 overlap: [8, 30, 30]
-skip_empty: true
-# Tiles per SLURM job; they run sequentially and share one decon/GPU context.
-# Raise once you know a tile's runtime -- job wall time is ~N x per-tile time.
-tiles_per_job: 4
-empty_threshold: null
 
 method: "custom"
 # dilate: 2       # optional: pixels to grow labels by after segmentation
@@ -58,8 +45,3 @@ custom:
       wavelength: 525
       na: 1.4
       nimm: 1.515
-
-pyramid_levels: 5
-pyramid_downscale: 2
-sequential_labels: true
-merge_workers: null

patchworks-2.3.0/workflow/config/config_cyto.yaml

@@ -0,0 +1,22 @@
+# Example segmentation config: cytoplasm/membrane channel.
+#
+# Only what differs from config/common.yaml — the input, work_dir, tiling and
+# conversion settings all live there and are merged in ahead of this file.
+# Run it via config/multi.yaml, or on its own with both files:
+#
+#   snakemake -s Snakefile --configfile config/common.yaml config/config_cyto.yaml
+
+channel: 0 # cytoplasm/membrane channel
+
+# Per-axis halo [z, y, x]. A scalar 30 would expand a [16, 1024, 1024] tile to
+# 76 x 1084 x 1084 = 5.3x the voxels it keeps, nearly all of it wasted z. The
+# z-halo only needs to cover one cell in z, not one cell in x/y.
+overlap: [4, 30, 30]
+
+method: "cellpose"
+label_name: "cyto_labels"
+cellpose:
+  model: "cyto3"
+  diameter: 30
+  do_3D: true
+  gpu: true

patchworks-2.3.0/workflow/config/config_nuclei.yaml

@@ -0,0 +1,22 @@
+# Example segmentation config: nuclei channel.
+#
+# Only what differs from config/common.yaml — the input, work_dir, tiling and
+# conversion settings all live there and are merged in ahead of this file.
+# Run it via config/multi.yaml, or on its own with both files:
+#
+#   snakemake -s Snakefile --configfile config/common.yaml config/config_nuclei.yaml
+
+channel: 1 # nuclear stain channel
+
+# Per-axis halo [z, y, x]. A scalar 30 would expand a [16, 1024, 1024] tile to
+# 76 x 1084 x 1084 = 5.3x the voxels it keeps, nearly all of it wasted z. The
+# z-halo only needs to cover one nucleus in z, not one nucleus in x/y.
+overlap: [4, 30, 30]
+
+method: "cellpose"
+label_name: "nuclei_labels"
+cellpose:
+  model: "nuclei"
+  diameter: 15
+  do_3D: true
+  gpu: true

{patchworks-2.1.2 → patchworks-2.3.0}/workflow/config/multi.yaml

@@ -13,6 +13,12 @@
 # `relations` (there's no automatic "chain"; list every pair explicitly, e.g.
 # nuclei->cyto, nuclei->membrane, cyto->membrane).
 
+# Settings every segmentation shares — the input, the work_dir, the tiling,
+# everything `convert` reads. Merged in ahead of each config below, so a
+# per-config file carries only what differs. Optional: drop this line and each
+# config can be self-contained again.
+common: config/common.yaml
+
 segmentations:
 - config/config_nuclei.yaml
 - config/config_cyto.yaml

{patchworks-2.1.2 → patchworks-2.3.0}/workflow/profile/slurm/config.yaml

@@ -5,6 +5,14 @@
 # Edit the partitions / account / GPU request for your cluster.
 
 executor: slurm
+# The SLURM executor names every job after its run UUID and rejects a
+# --job-name in slurm_extra, so `squeue` shows nothing recognisable without
+# this. The prefix goes first in the name (<prefix>_<uuid>), which is the part
+# that survives truncation in a queue listing.
+# Alphanumerics, underscores and hyphens only; 50 characters max.
+# run_multi overrides this per config, so a three-way run shows pw-convert,
+# then pw-nuclei_labels / pw-cyto_labels / pw-cilia_labels.
+slurm-jobname-prefix: patchworks
 # Max concurrent SLURM jobs Snakemake will have in flight at once. This is a
 # submission-side cap, not a cluster limit — raising it just lets more
 # "segment" tiles queue/run together; scicore's scheduler still enforces your

{patchworks-2.1.2 → patchworks-2.3.0}/workflow/scripts/run_multi.py

@@ -26,6 +26,7 @@ overlap, including b-objects with zero matches).
 from __future__ import annotations
 
 import argparse
+import re
 import subprocess
 import sys
 from pathlib import Path
@@ -47,25 +48,39 @@ def _snakemake_cmd(
     state_dir: Path | None = None,
     targets: list[str] | None = None,
     extra: list[str] | None = None,
+    jobname_prefix: str | None = None,
+    common: Path | None = None,
 ) -> list[str]:
     """Build one snakemake invocation.
 
     Every path is absolutised because ``--directory`` moves the working
     directory: each config needs its own ``.snakemake`` state directory, or
     concurrent runs would contend for the same ``.snakemake/locks/``.
+
+    *common*, when given, is passed as the first of two ``--configfile``
+    values. Snakemake merges them in order with the later winning, so the
+    settings every config shares -- the input, the work_dir, everything
+    ``convert`` reads -- live in one file and the per-config file carries only
+    what actually differs.
     """
+    configfiles = [str(configfile.resolve())]
+    if common is not None:
+        configfiles.insert(0, str(common.resolve()))
     cmd = [
         "snakemake",
         "-s",
         str(workflow_dir / "Snakefile"),
         "--configfile",
-        str(configfile.resolve()),
+        *configfiles,
     ]
     if state_dir is not None:
         state_dir.mkdir(parents=True, exist_ok=True)
         cmd += ["--directory", str(state_dir.resolve())]
     if profile:
         cmd += ["--workflow-profile", str((workflow_dir / profile).resolve())]
+        if jobname_prefix:
+            # A SLURM-executor setting, so only valid alongside the profile.
+            cmd += ["--slurm-jobname-prefix", jobname_prefix]
     else:
         cmd += ["--cores", str(cores), "--rerun-triggers", "mtime"]
     if dry_run:
@@ -79,11 +94,50 @@ def _snakemake_cmd(
     return cmd
 
 
+def slurm_jobname_prefix(label: str) -> str:
+    """Sanitise *label* into a SLURM job-name prefix the executor accepts.
+
+    The SLURM executor names every job after its run UUID and refuses a
+    ``--job-name`` in ``slurm_extra``, so a prefix is the only way to get
+    something recognisable into ``squeue``. It becomes ``<prefix>_<uuid>``,
+    which puts the readable part first -- the part that survives truncation
+    in a queue listing.
+
+    The executor requires alphanumerics, underscores and hyphens only, at
+    most 50 characters, and rejects the whole run otherwise.
+
+    Examples
+    --------
+    >>> slurm_jobname_prefix("nuclei_labels")
+    'pw-nuclei_labels'
+    >>> slurm_jobname_prefix("cilia/v2 (test)")
+    'pw-cilia-v2--test-'
+    """
+    safe = re.sub(r"[^A-Za-z0-9_-]", "-", label)
+    return f"pw-{safe}"[:50]
+
+
 def _run(cmd: list[str], workflow_dir: Path) -> int:
     print(f"[run_multi] $ {' '.join(cmd)}", flush=True)
     return subprocess.run(cmd, cwd=workflow_dir).returncode
 
 
+# Exactly the config keys scripts/convert.py reads -- keep the two in step.
+# Conversion runs once, in phase A, from the first config, so these have to
+# agree across all of them or the disagreement is invisible.
+#
+# Deliberately NOT here: pyramid_levels / pyramid_downscale. Those are read by
+# merge.py, which runs once per config and builds that config's own label
+# pyramid, so they may legitimately differ.
+_CONVERT_KEYS = (
+    "input",
+    "sequence_pattern",
+    "convert_chunks",
+    "shard",
+    "reuse_pyramid",
+)
+
+
 def _validate_configs(paths: list[Path], cfgs: list[dict]) -> str:
     """Check the cross-config invariants before anything is submitted.
 
@@ -117,6 +171,20 @@ def _validate_configs(paths: list[Path], cfgs: list[dict]) -> str:
                 f"share a chunk layout; got {_spread(key)}"
             )
 
+    # Phase A converts once, from the first config. Anything `convert` reads
+    # out of a later config is therefore silently ignored -- someone setting
+    # `shard: true` on the second config and watching a million files appear
+    # anyway has no way to see why. Refuse instead, and point at common.yaml.
+    for key in _CONVERT_KEYS:
+        values = {repr(cfg.get(key)) for cfg in cfgs}
+        if len(values) != 1:
+            problems.append(
+                f"{key} affects `convert`, which runs once from the first "
+                f"config, so the other values would be silently ignored; got "
+                f"{_spread(key)}. Put the settings every config shares in one "
+                f"file and point `common:` in multi.yaml at it."
+            )
+
     for path, cfg in zip(paths, cfgs):
         source = str(cfg.get("input", ""))
         if any(ch in source for ch in "*?[") and not cfg.get(
@@ -193,7 +261,15 @@ def main() -> None:
     seg_config_paths = [
         _resolve(workflow_dir, c) for c in multi_cfg["segmentations"]
     ]
-    seg_cfgs = [_load_yaml(p) for p in seg_config_paths]
+    # Optional shared config: Snakemake merges --configfile values in order,
+    # so `common` holds what every segmentation agrees on and each per-config
+    # file overrides only what differs. Validation has to see the same merged
+    # view Snakemake will, or it would report a missing work_dir that is
+    # simply defined one file over.
+    common_path = multi_cfg.get("common")
+    common_path = _resolve(workflow_dir, common_path) if common_path else None
+    common_cfg = _load_yaml(common_path) if common_path else {}
+    seg_cfgs = [{**common_cfg, **_load_yaml(p)} for p in seg_config_paths]
     work_dir = _validate_configs(seg_config_paths, seg_cfgs)
     image_store = f"{work_dir}/image.zarr"
 
@@ -217,6 +293,7 @@ def main() -> None:
                     dry_run=False,
                     state_dir=state_dir,
                     extra=["--unlock"],
+                    common=common_path,
                 ),
                 workflow_dir,
             )
@@ -235,6 +312,8 @@ def main() -> None:
             dry_run=args.dry_run,
             state_dir=Path(work_dir) / ".snakemake_convert",
             targets=[f"{image_store}/zarr.json"],
+            jobname_prefix=slurm_jobname_prefix("convert"),
+            common=common_path,
         ),
         workflow_dir,
     )
@@ -287,6 +366,9 @@ def main() -> None:
             cores=args.cores,
             dry_run=args.dry_run,
             state_dir=Path(cfg["work_dir"]) / cfg["label_name"] / ".snakemake",
+            # Names the config in squeue, so concurrent runs are tellable apart.
+            jobname_prefix=slurm_jobname_prefix(cfg["label_name"]),
+            common=common_path,
         )
         print(f"[run_multi] $ {' '.join(cmd)}", flush=True)
         procs.append((cfg_path.name, subprocess.Popen(cmd, cwd=workflow_dir)))

patchworks-2.1.2/workflow/config/config_cyto.yaml

@@ -1,39 +0,0 @@
-# Example segmentation config: cytoplasm/membrane channel.
-# Paired with config_nuclei.yaml via config/multi.yaml — see
-# docs/guide/snakemake.md "Running two segmentations". Same work_dir as
-# config_nuclei.yaml, and the same tile_shape so patchworks.label_relations()
-# can compare the two label arrays chunk-for-chunk.
-
-input: "/path/to/scan.ims"
-work_dir: "/path/to/results"
-
-reuse_pyramid: false
-convert_chunks: null
-shard: false
-
-channel: 0 # cytoplasm/membrane channel
-level: 0
-tile_shape: [16, 1024, 1024] # keep identical to config_nuclei.yaml
-gpu_memory_gb: null
-# Per-axis halo [z, y, x]. A scalar 30 would expand a [16, 1024, 1024] tile to
-# 76 x 1084 x 1084 = 5.3x the voxels it keeps, nearly all of it wasted z. The
-# z-halo only needs to cover one cell in z, not one cell in x/y.
-overlap: [4, 30, 30]
-skip_empty: true
-# Tiles per SLURM job; they run sequentially and share one model load. Raise
-# once you know a tile's runtime -- job wall time is ~N x per-tile time.
-tiles_per_job: 4
-empty_threshold: null
-
-method: "cellpose"
-label_name: "cyto_labels"
-cellpose:
-  model: "cyto3"
-  diameter: 30
-  do_3D: true
-  gpu: true
-
-pyramid_levels: 5
-pyramid_downscale: 2
-sequential_labels: true
-merge_workers: null

patchworks-2.1.2/workflow/config/config_nuclei.yaml

@@ -1,39 +0,0 @@
-# Example segmentation config: nuclei channel.
-# Paired with config_cyto.yaml via config/multi.yaml — see
-# docs/guide/snakemake.md "Running two segmentations". Both configs share
-# work_dir (and thus image.zarr) with config_cyto.yaml, but keep tile_shape
-# identical across the two so patchworks.label_relations() can compare them.
-
-input: "/path/to/scan.ims"
-work_dir: "/path/to/results"
-
-reuse_pyramid: false
-convert_chunks: null
-shard: false
-
-channel: 1 # nuclear stain channel
-level: 0
-tile_shape: [16, 1024, 1024] # keep identical to config_cyto.yaml
-gpu_memory_gb: null
-# Per-axis halo [z, y, x]. A scalar 30 would expand a [16, 1024, 1024] tile to
-# 76 x 1084 x 1084 = 5.3x the voxels it keeps, nearly all of it wasted z. The
-# z-halo only needs to cover one nucleus in z, not one nucleus in x/y.
-overlap: [4, 30, 30]
-skip_empty: true
-# Tiles per SLURM job; they run sequentially and share one model load. Raise
-# once you know a tile's runtime -- job wall time is ~N x per-tile time.
-tiles_per_job: 4
-empty_threshold: null
-
-method: "cellpose"
-label_name: "nuclei_labels"
-cellpose:
-  model: "nuclei"
-  diameter: 15
-  do_3D: true
-  gpu: true
-
-pyramid_levels: 5
-pyramid_downscale: 2
-sequential_labels: true
-merge_workers: null