patchworks 2.6.10__tar.gz → 2.6.11__tar.gz

{patchworks-2.6.10 → patchworks-2.6.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: patchworks
-Version: 2.6.10
+Version: 2.6.11
 Summary: Tiled processing of arbitrarily large images with globally consistent labels
 Project-URL: Homepage, https://github.com/imcf/patchworks
 Project-URL: Issues, https://github.com/imcf/patchworks/issues

{patchworks-2.6.10 → patchworks-2.6.11}/docs/examples/dog.md

@@ -102,6 +102,21 @@ decon_kwargs=dict(psf=psf, dxpsf=0.05, dzpsf=0.1, wavelength=525, ...)
     so edge tiles keep enough context (a plain intensity/threshold halo is
     too thin).
 
+!!! note "cudaDecon can return a smaller volume than it was given"
+    It rounds each axis down to an FFT-efficient length — e.g. a
+    `(32, 1084, 1084)` tile comes back `(32, 1080, 1080)`, because
+    `1080 = 2³·3³·5` while `1084 = 4·271` — and trims the excess off the
+    **high end**, leaving voxel `(0, 0, 0)` where it was. patchworks restores
+    the input shape before the DoG step (one label per input voxel is
+    required) anchored at that origin, and logs a WARNING with both shapes.
+
+    Anchoring matters: restoring it *centred* instead moves every voxel by
+    `excess // 2` — a 2 px y/x shift for the tile above, identical on every
+    tile. That is invisible on a cell tens of voxels wide and obvious on a
+    cilium a few voxels wide, which is how it was eventually caught. If the
+    logged difference is more than a few voxels, the PSF or the voxel sizes
+    are wrong.
+
 ## Growing the labels afterwards
 
 DoG spots/threads are often thin — grow each label by a few pixels with

{patchworks-2.6.10 → patchworks-2.6.11}/docs/guide/snakemake.md

@@ -444,10 +444,14 @@ results/image.zarr/labels/cyto_labels/
 
 !!! tip "The relate step's log"
     Unlike `prepare`/`segment`/`merge`, the relate step isn't a Snakemake
-    rule, so it doesn't get a `log:` directive for free. It writes its own
-    log to `<work_dir>/logs/relate.log` (override with `relate.py --log`),
-    the same tee-to-file-and-stdout behaviour as the other steps — check
-    there instead of scrolling back through `srun`'s live output.
+    rule, so it doesn't get a `log:` directive for free. Standalone (or
+    under plain `multi`), it writes to `<work_dir>/logs/relate.log`
+    (override with `relate.py --log`), the same tee-to-file-and-stdout
+    behaviour as the other steps. Under `multi-slurm`, where each pair is
+    its own concurrent job, `run_multi.py` points each one at its own file
+    instead — `<work_dir>/logs/relate/<a>_to_<b>.log` — so concurrent pairs
+    don't interleave into one log; check there instead of scrolling back
+    through `srun`'s live output.
 
 See [Relating labels across segmentations](label_relations.md) for what
 `label_relations()` returns and how to save it yourself — the cluster
@@ -498,15 +502,29 @@ abort the others; you get a per-config status and a non-zero exit.
 
 !!! tip "The relate step runs on the cluster too, under `multi-slurm`"
     `label_relations()` streams every chunk of two full-resolution label
-    volumes — real CPU/IO work, not orchestration. Under `multi-slurm` it is
-    submitted as its own `srun` job (`scripts/relate.py`) instead of running
-    in the driver process on the login node, the same fix already applied to
-    the occupancy map. Tune its allocation with `--relate-partition`,
-    `--relate-mem`, `--relate-cpus` and `--relate-time` (defaults: `scicore`,
-    `32G`, `8`, `180` minutes) — these are wide-margin guesses, not measured
-    numbers, so raise them for a very large or very object-dense pair. Under
-    plain `multi` (no `--profile`), it still runs locally, in-process, as
-    before.
+    volumes — real CPU/IO work, not orchestration. Under `multi-slurm`,
+    **each pair in `relations:` is submitted as its own `srun` job**
+    (`scripts/relate.py`) instead of running in the driver process on the
+    login node, the same fix already applied to the occupancy map. Tune the
+    allocation with `--relate-partition`, `--relate-mem`, `--relate-cpus`
+    and `--relate-time` (defaults: `scicore`, `32G`, `8`, `180` minutes,
+    **per relation**) — these are wide-margin guesses, not measured numbers,
+    so raise them for a very large or very object-dense pair; a pair needing
+    a chunk-layout rechunk first (see below) is the usual reason one runs
+    long. Set `--relate-qos` if your account's default QOS for the partition
+    caps the wall time below `--relate-time` — `srun` fails immediately with
+    `QOSMaxWallDurationPerJobLimit` when that happens; `sacctmgr -p show
+    assoc user=$USER` and `sacctmgr -p show qos` list what's available and
+    each one's `MaxWall`. Under plain `multi` (no `--profile`), relations
+    still run locally, in-process, one after another, as before.
+
+    Because every pair gets its own job, one running long no longer starves
+    the others out of a shared time budget, and a pair that gets killed no
+    longer takes an already-finished sibling's workbook down with it.
+    `relate.py` also skips a pair whose `.xlsx` is already newer than both
+    labels' merge marker, so **re-running the exact same `multi-slurm`
+    command only recomputes what's still missing or stale** — delete a
+    specific `.xlsx` yourself to force just that one to recompute.
 
 !!! tip "After a killed run"
     Snakemake only releases its lock on a clean exit, so a run that was killed

{patchworks-2.6.10 → patchworks-2.6.11}/src/patchworks/plugins/dog.py

@@ -239,18 +239,23 @@ def _run(block: np.ndarray, dog_dict: dict[str, Any]) -> np.ndarray:
 
 
 def _restore_shape(arr: np.ndarray, shape: tuple[int, ...]) -> np.ndarray:
-    """Centre *arr* back into an array of *shape*, cropping or edge-padding.
+    """Restore *arr* to *shape*, anchored at the origin, cropping or padding.
 
     Deconvolution must not change the field of view: patchworks writes the
     result into a destination slice derived from the tile's geometry, so one
     label per input voxel is required.
 
-    Centring is the right correction for a symmetric crop, which is what
-    apodisation produces. The discrepancies observed are small (a voxel in z,
-    a few in x/y) and land inside the halo, which is discarded anyway -- so
-    the labels that survive the trim are unaffected. It is logged at WARNING
-    with the exact shapes so a larger, non-symmetric crop cannot pass
-    silently.
+    The alignment is **origin-anchored**, not centred: cudaDecon rounds each
+    axis down to an FFT-efficient length (e.g. 1084 -> 1080, since
+    1080 = 2**3 * 3**3 * 5 while 1084 = 4 * 271) and trims the excess off the
+    high end, leaving voxel (0, 0, 0) where it was. Re-centring content that
+    was never centred shifts every voxel by ``excess // 2`` -- measured at 2
+    px in y and x on a real (32, 1084, 1084) tile, in the same direction on
+    every tile. That is invisible on a cell tens of voxels across and glaring
+    on a cilium a few voxels across, which is exactly how it was found.
+
+    A mismatch is still logged at WARNING with the exact shapes, since a
+    large one means the PSF or voxel sizes are wrong.
 
     Parameters
     ----------
@@ -271,22 +276,22 @@ def _restore_shape(arr: np.ndarray, shape: tuple[int, ...]) -> np.ndarray:
     (14, 1024)
     """
     logger.warning(
-        "deconvolution returned %s for a %s input; re-centring to the input "
-        "shape. patchworks needs one label per input voxel. A large or "
-        "asymmetric difference here would shift labels -- check the PSF and "
-        "voxel sizes if this is more than a few voxels.",
+        "deconvolution returned %s for a %s input; restoring the input shape "
+        "from the origin. patchworks needs one label per input voxel. A large "
+        "difference here means the PSF or voxel sizes are wrong -- check them "
+        "if this is more than a few voxels.",
         arr.shape,
         shape,
     )
     # Crop first, so an axis that grew is handled before padding the rest.
+    # Both keep voxel 0 where it is: cudaDecon trims off the high end (see the
+    # docstring), so the low corner is the one landmark known to be unmoved.
     crop = tuple(
-        slice((a - s) // 2, (a - s) // 2 + s) if a > s else slice(None)
-        for a, s in zip(arr.shape, shape)
+        slice(0, s) if a > s else slice(None) for a, s in zip(arr.shape, shape)
     )
     arr = arr[crop]
     pad = tuple(
-        ((s - a) // 2, s - a - (s - a) // 2) if a < s else (0, 0)
-        for a, s in zip(arr.shape, shape)
+        (0, s - a) if a < s else (0, 0) for a, s in zip(arr.shape, shape)
     )
     if any(lo or hi for lo, hi in pad):
         arr = np.pad(arr, pad, mode="edge")

{patchworks-2.6.10 → patchworks-2.6.11}/tests/test_dog.py

@@ -109,20 +109,32 @@ def test_explicit_decon_kwargs_win_over_the_calibration(monkeypatch):
     assert captured["dzpsf"] == 0.2
 
 
-def test_restore_shape_recentres_a_cropped_decon():
+def test_restore_shape_anchors_a_cropped_decon_at_the_origin():
     """cudaDecon can hand back a smaller volume than it was given.
 
-    Observed on a real edge tile: (14, 1024, 1024) in, (13, 1020, 1020) out.
-    patchworks needs one label per input voxel, so the field of view has to be
-    restored before the DoG step.
+    Observed on a real tile: (32, 1084, 1084) in, (32, 1080, 1080) out --
+    each axis rounded down to an FFT-efficient length, with the excess taken
+    off the high end. Restoring it *centred* (what this used to do) moved
+    every voxel by excess // 2, measured as a 2 px y/x shift on real data:
+    invisible on a cell, glaring on a cilium a few voxels across.
     """
     from patchworks.plugins.dog import _restore_shape
 
     arr = np.arange(13 * 1020 * 1020, dtype="float32").reshape(13, 1020, 1020)
     out = _restore_shape(arr, (14, 1024, 1024))
     assert out.shape == (14, 1024, 1024)
-    # The original content is preserved, centred, not resampled.
-    assert np.array_equal(out[0:13, 2:1022, 2:1022], arr)
+    # Content keeps its original indices -- voxel 0 stays voxel 0.
+    assert np.array_equal(out[0:13, 0:1020, 0:1020], arr)
+
+
+def test_restore_shape_crops_from_the_high_end():
+    """The mirror case: an axis that came back too long keeps its low corner."""
+    from patchworks.plugins.dog import _restore_shape
+
+    arr = np.arange(6 * 12, dtype="float32").reshape(6, 12)
+    out = _restore_shape(arr, (4, 8))
+    assert out.shape == (4, 8)
+    assert np.array_equal(out, arr[0:4, 0:8])
 
 
 def test_restore_shape_handles_growth_and_exact_fit():

{patchworks-2.6.10 → patchworks-2.6.11}/tests/test_run_multi.py

@@ -1,7 +1,10 @@
 """Tests for the multi-config driver's SLURM-facing behaviour."""
 
+import json
+import os
 import re
 import sys
+import time
 from pathlib import Path
 
 sys.path.insert(
@@ -15,6 +18,7 @@ import yaml  # noqa: E402
 
 from run_multi import (  # noqa: E402
     _CONVERT_KEYS,
+    _relate_cmd,
     _snakemake_cmd,
     _validate_configs,
     slurm_jobname_prefix,
@@ -176,6 +180,85 @@ def test_relate_is_submitted_via_slurm_under_profile():
     assert "label_relations(" not in src
 
 
+def _relate_kwargs(**overrides):
+    kwargs = dict(
+        work_dir="/w",
+        image_store="/w/image.zarr",
+        workflow_dir=Path("/workflow"),
+        relate_partition="scicore",
+        relate_mem="32G",
+        relate_cpus=8,
+        relate_time=180,
+        relate_qos=None,
+    )
+    kwargs.update(overrides)
+    return kwargs
+
+
+def test_relate_cmd_is_one_job_per_relation_pair():
+    """A killed shared job used to lose every relation still queued behind
+
+    the one that was running -- one srun per pair means a slow or failing
+    pair can no longer starve, or take down, its siblings' time budget.
+    """
+    relations = [
+        {"a": "nuclei_labels", "b": "cyto_labels", "output": "n2c.xlsx"},
+        {"a": "cilia_labels", "b": "cyto_labels", "output": "c2c.xlsx"},
+    ]
+    cmds = [_relate_cmd(rel, **_relate_kwargs()) for rel in relations]
+
+    assert len(cmds) == 2
+    for cmd, rel in zip(cmds, relations):
+        assert cmd[0] == "srun"
+        assert "--relations" in cmd
+        # Each job's payload is *only* its own pair, not the whole list.
+        payload = json.loads(cmd[cmd.index("--relations") + 1])
+        assert payload == [rel]
+
+
+def test_relate_cmd_job_name_identifies_the_pair():
+    cmd = _relate_cmd(
+        {"a": "cilia_labels", "b": "cyto_labels"}, **_relate_kwargs()
+    )
+    name = cmd[cmd.index("--job-name") + 1]
+    assert _EXECUTOR_RULE.match(name)
+    assert "cilia_labels" in name
+    assert "cyto_labels" in name
+
+
+def test_relate_cmd_gives_each_pair_its_own_log():
+    """Concurrent per-pair jobs sharing one relate.log would interleave --
+
+    each pair's --log must be a distinct file, or the whole point of
+    splitting the log the way segment/<batch>.log already does is lost.
+    """
+    cmds = [
+        _relate_cmd(rel, **_relate_kwargs())
+        for rel in (
+            {"a": "nuclei_labels", "b": "cyto_labels"},
+            {"a": "cilia_labels", "b": "cyto_labels"},
+        )
+    ]
+    logs = [cmd[cmd.index("--log") + 1] for cmd in cmds]
+    assert len(set(logs)) == 2
+    assert all(log.startswith("/w/logs/relate/") for log in logs)
+
+
+def test_relate_cmd_omits_qos_by_default():
+    cmd = _relate_cmd({"a": "a", "b": "b"}, **_relate_kwargs())
+    assert "--qos" not in cmd
+
+
+def test_relate_cmd_passes_qos_when_set():
+    """A default QOS whose MaxWall is shorter than --relate-time is exactly
+
+    what killed a real run (QOSMaxWallDurationPerJobLimit) -- --relate-qos
+    lets a longer one be requested explicitly instead of guessed at.
+    """
+    cmd = _relate_cmd({"a": "a", "b": "b"}, **_relate_kwargs(relate_qos="1day"))
+    assert cmd[cmd.index("--qos") + 1] == "1day"
+
+
 def test_relate_script_has_the_real_bookkeeping():
     """relate.py must be the actual implementation, not a stub.
 
@@ -285,6 +368,88 @@ def test_relate_rechunks_mismatched_label_arrays(tmp_path):
     assert rows[2] == (None, 0, 0)  # label 2 touches nothing in b
 
 
+def test_relation_up_to_date_missing_output_is_false(tmp_path):
+    from relate import _relation_up_to_date
+
+    assert not _relation_up_to_date(
+        str(tmp_path), "a", "b", tmp_path / "nope.xlsx"
+    )
+
+
+def test_relation_up_to_date_missing_marker_is_false(tmp_path):
+    """No labels.done for a label means its state can't be judged -- treat
+
+    that as "recompute", not as "trust the existing workbook".
+    """
+    from relate import _relation_up_to_date
+
+    out = tmp_path / "rel.xlsx"
+    out.write_text("x")
+    assert not _relation_up_to_date(str(tmp_path), "a", "b", out)
+
+
+def test_relation_up_to_date_true_only_when_newer_than_both_markers(
+    tmp_path,
+):
+    from relate import _relation_up_to_date
+
+    (tmp_path / "a").mkdir()
+    (tmp_path / "b").mkdir()
+    (tmp_path / "a" / "labels.done").touch()
+    (tmp_path / "b" / "labels.done").touch()
+    out = tmp_path / "rel.xlsx"
+    out.write_text("x")
+
+    now = time.time()
+    os.utime(tmp_path / "a" / "labels.done", (now, now))
+    os.utime(tmp_path / "b" / "labels.done", (now, now))
+
+    # older than both markers -> stale
+    os.utime(out, (now - 10, now - 10))
+    assert not _relation_up_to_date(str(tmp_path), "a", "b", out)
+
+    # newer than both markers -> up to date
+    os.utime(out, (now + 10, now + 10))
+    assert _relation_up_to_date(str(tmp_path), "a", "b", out)
+
+    # b re-merged after the workbook was written -> stale again
+    os.utime(tmp_path / "b" / "labels.done", (now + 20, now + 20))
+    assert not _relation_up_to_date(str(tmp_path), "a", "b", out)
+
+
+def test_relate_skips_a_relation_whose_workbook_is_up_to_date(tmp_path):
+    """A retry must not recompute what already finished -- only what a
+
+    shared, now-split-per-pair job left missing after a partial failure.
+    Proven by planting sentinel content no real relation would produce: if
+    run_relations() recomputed anyway, the sentinel would be gone.
+    """
+    from relate import run_relations
+
+    work_dir = tmp_path / "work"
+    work_dir.mkdir()
+    for name in ("a_labels", "b_labels"):
+        (work_dir / name).mkdir()
+        (work_dir / name / "labels.done").touch()
+
+    out_path = work_dir / "rel.xlsx"
+    sentinel = openpyxl.Workbook()
+    sentinel.active.append(["sentinel"])
+    sentinel.save(out_path)
+    os.utime(out_path, (time.time() + 60, time.time() + 60))
+
+    # No image_store/labels at all -- if this weren't skipped, run_relations
+    # would raise trying to open them, not just produce the wrong content.
+    run_relations(
+        str(work_dir),
+        str(tmp_path / "image.zarr"),
+        [{"a": "a_labels", "b": "b_labels", "output": "rel.xlsx"}],
+    )
+
+    wb = openpyxl.load_workbook(out_path)
+    assert wb.active["A1"].value == "sentinel"
+
+
 def test_mixed_nuclei_channel_auto_passes_validation():
     """A channel-count mismatch under `tile_shape: "auto"` is no longer
 

{patchworks-2.6.10 → patchworks-2.6.11}/workflow/scripts/relate.py

@@ -54,11 +54,43 @@ def _label_ids(image_store: str, name: str) -> list[int]:
     return sorted(int(x) for x in da.unique(arr[arr > 0]).compute())
 
 
+def _relation_up_to_date(
+    work_dir: str, a_name: str, b_name: str, out_path: Path
+) -> bool:
+    """Whether *out_path* already reflects the current *a_name*/*b_name* labels.
+
+    Mirrors the workflow's existing "delete to force" convention (see
+    ``image.zarr`` not being reconverted once it exists): a relation is
+    considered current when its workbook is newer than both labels'
+    ``labels.done`` merge marker, and stale (or never computed) otherwise.
+    Missing markers -- a label group written before merge started recording
+    one, or a nonstandard ``image_store`` layout -- are treated as "unknown,
+    recompute" rather than raise, since staleness can't be judged without them.
+    """
+    if not out_path.exists():
+        return False
+    try:
+        out_mtime = out_path.stat().st_mtime
+        for name in (a_name, b_name):
+            marker = Path(work_dir) / name / "labels.done"
+            if not marker.exists() or marker.stat().st_mtime > out_mtime:
+                return False
+    except OSError:
+        return False
+    return True
+
+
 def run_relations(
     work_dir: str, image_store: str, relations: list[dict]
 ) -> None:
     """Compute and write every configured relation pair as an .xlsx workbook.
 
+    A relation already reflected by an up-to-date workbook (see
+    :func:`_relation_up_to_date`) is skipped rather than recomputed -- so
+    retrying a partially-failed run (this step has no Snakemake rule of its
+    own to track that for it) only redoes what's actually missing or stale.
+    Delete the ``.xlsx`` yourself to force a specific relation to recompute.
+
     Parameters
     ----------
     work_dir : str
@@ -81,6 +113,13 @@ def run_relations(
         out_path = Path(work_dir) / rel.get(
             "output", f"{a_name}_to_{b_name}.xlsx"
         )
+        if _relation_up_to_date(work_dir, a_name, b_name, out_path):
+            print(
+                f"[relate] {out_path} is already up to date with "
+                f"{a_name}/{b_name}; skipping",
+                flush=True,
+            )
+            continue
         print(f"[relate] relating {a_name} -> {b_name} …", flush=True)
         a = da.from_zarr(image_store, component=f"labels/{a_name}/0")
         b = da.from_zarr(image_store, component=f"labels/{b_name}/0")

{patchworks-2.6.10 → patchworks-2.6.11}/workflow/scripts/run_multi.py

@@ -129,6 +129,11 @@ def slurm_jobname_prefix(label: str) -> str:
     return f"pw-{safe}"[:50]
 
 
+def _safe_filename(name: str) -> str:
+    """Sanitise a label name for use as (part of) a log filename."""
+    return re.sub(r"[^A-Za-z0-9_-]", "-", name)
+
+
 def _test_email(cfg: dict) -> int:
     """Send one test notification and report the outcome. Returns an exit code.
 
@@ -186,6 +191,67 @@ def _test_email(cfg: dict) -> int:
     return 1
 
 
+def _relate_cmd(
+    rel: dict,
+    *,
+    work_dir: str,
+    image_store: str,
+    workflow_dir: Path,
+    relate_partition: str,
+    relate_mem: str,
+    relate_cpus: int,
+    relate_time: int,
+    relate_qos: str | None,
+) -> list[str]:
+    """Build one ``srun`` invocation of ``relate.py`` for a single relation pair.
+
+    One job per pair, not one job for the whole ``relations:`` list: a
+    single shared ``srun`` time budget lets a slow pair (e.g. one needing a
+    chunk-layout rechunk first) starve the others out of a fixed
+    ``--relate-time``, and a kill that way loses everything not yet written
+    even though earlier pairs already finished. Separate jobs also run
+    concurrently instead of one after another, and ``relate.py`` itself
+    skips a pair whose workbook is already up to date, so retrying with the
+    same relations only redoes what actually failed.
+    """
+    a_name, b_name = rel["a"], rel["b"]
+    cmd = [
+        "srun",
+        "--partition",
+        relate_partition,
+    ]
+    if relate_qos:
+        cmd += ["--qos", relate_qos]
+    cmd += [
+        "--mem",
+        relate_mem,
+        "--cpus-per-task",
+        str(relate_cpus),
+        "--time",
+        str(relate_time),
+        "--job-name",
+        slurm_jobname_prefix(f"relate-{a_name}-to-{b_name}"),
+        sys.executable,
+        str(workflow_dir / "scripts" / "relate.py"),
+        "--work-dir",
+        work_dir,
+        "--image-store",
+        image_store,
+        "--relations",
+        json.dumps([rel]),
+        # Concurrent per-pair jobs sharing the default <work_dir>/logs/
+        # relate.log would interleave -- give each pair its own file.
+        "--log",
+        str(
+            Path(work_dir)
+            / "logs"
+            / "relate"
+            / f"{_safe_filename(a_name)}_to_{_safe_filename(b_name)}.log"
+        ),
+    ]
+    return cmd
+
+
 def _run(cmd: list[str], workflow_dir: Path) -> int:
     print(f"[run_multi] $ {' '.join(cmd)}", flush=True)
     return subprocess.run(cmd, cwd=workflow_dir).returncode
@@ -448,7 +514,22 @@ def main() -> None:
         "--relate-time",
         type=int,
         default=180,
-        help="srun --time in minutes for the relate step under --profile (default: 180)",
+        help=(
+            "srun --time in minutes for the relate step under --profile "
+            "(default: 180). Each relation pair is its own SLURM job, so "
+            "this bounds one relation, not the whole relations: list."
+        ),
+    )
+    parser.add_argument(
+        "--relate-qos",
+        default=None,
+        help=(
+            "srun --qos for the relate step under --profile. Omit to let "
+            "SLURM pick your account's default QOS for the partition -- set "
+            "this explicitly if that default's MaxWall is shorter than "
+            "--relate-time (sacctmgr -p show assoc/qos shows what's "
+            "available, e.g. '1day', '1week')."
+        ),
     )
     args = parser.parse_args()
 
@@ -603,38 +684,52 @@ def main() -> None:
         # Real CPU/IO work -- tens of thousands of zarr chunk reads for a
         # full-resolution label volume -- not orchestration, so (like the
         # occupancy map) it does not belong in this driver process on the
-        # login node. Submit it as its own job instead.
-        cmd = [
-            "srun",
-            "--partition",
-            args.relate_partition,
-            "--mem",
-            args.relate_mem,
-            "--cpus-per-task",
-            str(args.relate_cpus),
-            "--time",
-            str(args.relate_time),
-            "--job-name",
-            "pw-relate",
-            sys.executable,
-            str(workflow_dir / "scripts" / "relate.py"),
-            "--work-dir",
-            work_dir,
-            "--image-store",
-            image_store,
-            "--relations",
-            json.dumps(relations),
-        ]
-        rc = _run(cmd, workflow_dir)
-        if rc != 0:
+        # login node. Submit it as its own job.
+        #
+        # One job *per relation pair*, not one job for the whole list: a
+        # single shared srun budget lets a slow pair (e.g. one needing a
+        # chunk-layout rechunk first) starve the others' time out of a fixed
+        # --relate-time, and killed that way loses everything not yet
+        # written even though earlier pairs already finished. Separate jobs
+        # also run concurrently rather than one after another, and
+        # relate.py itself now skips a pair whose workbook is already
+        # up to date, so retrying this exact command only redoes what
+        # actually failed.
+        procs = []
+        for rel in relations:
+            cmd = _relate_cmd(
+                rel,
+                work_dir=work_dir,
+                image_store=image_store,
+                workflow_dir=workflow_dir,
+                relate_partition=args.relate_partition,
+                relate_mem=args.relate_mem,
+                relate_cpus=args.relate_cpus,
+                relate_time=args.relate_time,
+                relate_qos=args.relate_qos,
+            )
+            print(f"[run_multi] $ {' '.join(cmd)}", flush=True)
+            procs.append(
+                (
+                    f"{rel['a']} -> {rel['b']}",
+                    subprocess.Popen(cmd, cwd=workflow_dir),
+                )
+            )
+
+        failed = [name for name, p in procs if p.wait() != 0]
+        for name, p in procs:
+            status = "FAILED" if p.returncode else "ok"
+            print(f"[run_multi] relate {name}: {status}", flush=True)
+        if failed:
             print(
-                f"[run_multi] ERROR: relate step failed (exit {rc}). "
-                "Segmentations already succeeded -- only the relation "
-                "workbook(s) are missing. Re-run with the same --config to "
-                "retry just this step.",
+                f"[run_multi] ERROR: {len(failed)} relation(s) failed: "
+                f"{', '.join(failed)}. Segmentations already succeeded, and "
+                "any relation that did finish is written -- re-run with the "
+                "same --config to retry just the ones still missing (already "
+                "up-to-date workbooks are skipped, not recomputed).",
                 file=sys.stderr,
             )
-            sys.exit(rc)
+            sys.exit(1)
         return
 
     from relate import run_relations