patchworks 2.3.0__tar.gz → 2.5.0__tar.gz

{patchworks-2.3.0 → patchworks-2.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: patchworks
-Version: 2.3.0
+Version: 2.5.0
 Summary: Tiled processing of arbitrarily large images with globally consistent labels
 Project-URL: Homepage, https://github.com/imcf/patchworks
 Project-URL: Issues, https://github.com/imcf/patchworks/issues

{patchworks-2.3.0 → patchworks-2.5.0}/cliff.toml

@@ -32,7 +32,10 @@ commit_preprocessors = [
 ]
 commit_parsers = [
   # Merge commits are noise in a changelog — the merged commits are already listed.
+  # Anchored to the message start so a commit that merely mentions "merge dev
+  # ... main" in its own text isn't swept up too.
   { message = "^Merge ", skip = true },
+  { message = "(?i)^chore:\\s*merge dev.*main", skip = true },
   { message = "^feat", group = "✨ Features" },
   { message = "^fix", group = "🐛 Bug Fixes" },
   { message = "^perf", group = "⚡ Performance" },

{patchworks-2.3.0 → patchworks-2.5.0}/docs/examples/dog.md

@@ -147,11 +147,13 @@ example above — and the segment job deconvolves each tile with
 `pycudadecon` before running the DoG detector:
 
 ```yaml
-# config/config_cilia.yaml (excerpt)
+# config/config_cilia.yaml (excerpt) — only what differs from common.yaml,
+# which supplies the input, work_dir, tile_shape and skip_empty
 channel: 2
-tile_shape: [16, 1024, 1024]
-overlap: 30 # cover the PSF support (decon) + the DoG's high_sigma
-skip_empty: true
+# Per-axis halo [z, y, x], covering the PSF support (decon) + the DoG's
+# high_sigma. A scalar 30 would expand a [16, 1024, 1024] tile to 5.3x the
+# voxels it keeps, nearly all of it wasted z.
+overlap: [8, 30, 30]
 
 method: "custom"
 label_name: "cilia_labels"
@@ -185,11 +187,11 @@ in `dxdata`/`dxpsf` from X/Y and `dzdata`/`dzpsf` from Z. Set any of them in
     of the calibration in its config. If the store is uncalibrated the
     workflow says so and passes nothing.
 
-Run it exactly like a Cellpose config:
+Run it exactly like a Cellpose config — the shared settings come from
+`config/common.yaml`, merged in ahead of this one:
 
 ```bash
-python -m snakemake --workflow-profile profile/slurm \
-                    --configfile config/config_cilia.yaml
+python -m snakemake --workflow-profile profile/slurm --configfile config/common.yaml config/config_cilia.yaml
 ```
 
 Checklist specific to this config:
@@ -206,10 +208,13 @@ Checklist specific to this config:
   a thin intensity/threshold halo isn't enough once deconvolution is in the
   loop.
 - **`skip_empty`:** the `prepare` rule (`workflow/scripts/prepare_tiles.py`)
-  calls `estimate_empty_tiles()` before submitting any `segment` jobs,
-  regardless of `method`, so cilia/DoG runs skip background tiles exactly
-  like Cellpose runs — no extra config needed beyond `skip_empty: true`
-  (the default).
+  builds a max-pooled occupancy map and reduces it over each tile's **full**
+  footprint (`build_occupancy_map` + `tile_occupancy`) before submitting any
+  `segment` jobs, regardless of `method`. Cilia are small and often sit near
+  a tile's edge, which is precisely where the older centred-window preview
+  could miss them — this decides every tile exactly. No extra config needed
+  beyond `skip_empty: true` (the default), and the map is built once and
+  shared by every config against that store.
 - Run alongside `config_cyto.yaml`/`config_nuclei.yaml` via `config/multi.yaml`
   to also get the cilia→cell/nucleus relation — see *Relating cilia to their
   cell*, below.

{patchworks-2.3.0 → patchworks-2.5.0}/docs/guide/performance.md

@@ -74,6 +74,9 @@ extra read-back of the staged data.
 
 - `tile_shape="auto"` sizes tiles to free RAM (or VRAM with `use_gpu=True`).
 - `skip_empty=True` with `estimate_empty_tiles()` skips background tiles.
+  That preview samples a centred window, so use `build_occupancy_map()` +
+  `tile_occupancy()` when the result becomes an authoritative skip list —
+  they are exact, and it is what the Snakemake workflow uses.
 - A Dask **distributed** cluster (`make_local_cluster`) parallelises across
   workers/GPUs; patchworks then defers concurrency to the cluster.
 

{patchworks-2.3.0 → patchworks-2.5.0}/docs/guide/snakemake.md

@@ -442,6 +442,46 @@ cyto_labels` and `cilia_labels -> nuclei_labels`) so you can use whichever
 fits a given dataset. See `config/config_cilia.yaml`. Its deconvolution step
 needs `pip install "patchworks[dog]"` in the segment jobs' environment.
 
+## Email notifications
+
+Set an address and the workflow mails you when the long steps finish or fail:
+
+```yaml
+# config/common.yaml (or config/config.yaml for a single-config run)
+notify_email: "you@unibas.ch"
+notify_events: ["finish", "error"] # any of: start, finish, error
+```
+
+Leave `notify_email` empty (the default) and nothing is sent.
+
+Per-job mail is **SLURM's own** `--mail-type`, not a message sent from inside
+the job. That matters: the controller sends it, so it still arrives when a job
+is OOM-killed or cancelled by the scheduler — exactly the failures worth
+hearing about, and exactly the ones a notification sent from within the job
+would miss.
+
+It is applied to the long single-job steps only — `convert`, `occupancy` and
+`merge`. `segment` is deliberately excluded: there is one job per tile batch,
+so a thousand-tile run would mean hundreds of messages.
+
+On top of that, the workflow itself sends:
+
+| When | Mail |
+| --- | --- |
+| The run fails | subject `[patchworks] FAILED: <label_name>`, with the last 40 lines of the failing step's log — usually the traceback itself |
+| The run succeeds | subject `[patchworks] done: <label_name>`, with the output label path |
+
+These cover what SLURM cannot: a local run with no scheduler at all, and
+failures where the useful content is the Python traceback rather than an exit
+code.
+
+!!! note "Delivery is best-effort, by design"
+    A notification can never fail a run. If no local `sendmail` exists and no
+    SMTP server answers on localhost, the failure is logged as a warning and
+    the pipeline carries on — a six-hour segmentation that worked must not be
+    reported as failed because a mail host was down. If you get the warning
+    but no mail, ask your cluster admins which relay host to use.
+
 ## Measurements
 
 See [Measurements](measurements.md) for computing area/centroid/intensity
@@ -520,7 +560,8 @@ prologue. The simplest path is a single shared env that the compute nodes see.
 | `cellpose is not installed` in a job | the job's env lacks `patchworks[cellpose]` |
 | Reading the input fails | install the matching reader (`patchworks[imaris]`/`[bioio]` + a `bioio-*`) |
 | Out of GPU memory | smaller `tile_shape`, or `do_3D: false` |
-| A job fails with an empty SLURM log | read `logs/segment/<index>.log` (per tile) or `logs/steps.log` — the real traceback is there |
+| A job fails with an empty SLURM log | read the step's own log — `logs/convert.log`, `logs/prepare.log`, `logs/segment/<batch>.log`, `logs/merge.log` — the real traceback is there |
+| A long step looks hung | every step logs progress (`… 4,200/8,064 (52%) after 31m, ~28m left`) roughly once a minute; `tail -f` the step's log above |
 | Very slow | confirm GPU is used (`nvidia-smi`); try 2-D or a lower `level` |
 
 ## How it works (for the curious)

{patchworks-2.3.0 → patchworks-2.5.0}/src/patchworks/_merge.py

@@ -23,7 +23,6 @@ import logging
 import os
 import tempfile
 from concurrent.futures import ThreadPoolExecutor
-from contextlib import nullcontext as _nullcontext
 from itertools import product as _iproduct
 from multiprocessing import Pool as _Pool
 from pathlib import Path
@@ -35,11 +34,7 @@ import zarr
 
 from ._chunks import cpu_allocation
 from ._io import zarr_compressor_kwargs
-
-try:
-    from tqdm.auto import tqdm as _tqdm
-except ImportError:
-    _tqdm = None
+from ._progress import track
 
 logger = logging.getLogger(__name__)
 
@@ -165,6 +160,7 @@ def _scan_touching_pairs(
     label_offsets: "np.ndarray | None" = None,
     n_workers: int = 1,
     has_labels: "np.ndarray | None" = None,
+    progress: bool = False,
 ) -> np.ndarray:
     """Scan chunk-boundary slabs; return (N, 2) int64 array of touching pairs.
 
@@ -259,12 +255,26 @@ def _scan_touching_pairs(
 
     nw = max(1, min(n_workers, len(tasks)))
     if nw <= 1:
-        results = [_one(t) for t in tasks]
+        results = list(
+            track(
+                (_one(t) for t in tasks),
+                "scan boundaries",
+                len(tasks),
+                enabled=progress,
+            )
+        )
     else:
         # Reads and decompression release the GIL, so threads scale here and
         # nothing has to be pickled across processes.
         with ThreadPoolExecutor(max_workers=nw) as pool:
-            results = list(pool.map(_one, tasks))
+            results = list(
+                track(
+                    pool.map(_one, tasks),
+                    "scan boundaries",
+                    len(tasks),
+                    enabled=progress,
+                )
+            )
 
     all_pairs = [r for r in results if r is not None]
     if not all_pairs:
@@ -609,6 +619,7 @@ def zarr_native_merge(
         label_offsets=offsets,
         n_workers=n_workers,
         has_labels=has_labels,
+        progress=show_progress,
     )
     logger.info(
         "zarr_native_merge: %d touching pairs → building LUT", len(pairs)
@@ -743,9 +754,9 @@ def zarr_native_merge(
             _init_worker(
                 lut_path, staged_path, staged_component, out_path, out_component
             )
-            it: Any = tasks
-            if show_progress and _tqdm is not None:
-                it = _tqdm(it, total=n_chunks, desc="relabel chunks")
+            it: Any = track(
+                tasks, "relabel chunks", n_chunks, enabled=show_progress
+            )
             for task in it:
                 _relabel_chunk_worker(task)
         else:
@@ -760,9 +771,12 @@ def zarr_native_merge(
                     out_component,
                 ),
             ) as pool:
-                it = pool.imap_unordered(_relabel_chunk_worker, tasks)
-                if show_progress and _tqdm is not None:
-                    it = _tqdm(it, total=n_chunks, desc="relabel chunks")
+                it = track(
+                    pool.imap_unordered(_relabel_chunk_worker, tasks),
+                    "relabel chunks",
+                    n_chunks,
+                    enabled=show_progress,
+                )
                 for _ in it:
                     pass
     finally:
@@ -926,9 +940,10 @@ def merge_tile_labels(
         stage_path = os.path.join(_base, "_pws_stage.zarr")
 
         import dask
-        from dask.diagnostics import ProgressBar
 
-        ctx = ProgressBar() if progress else _nullcontext()
+        from ._progress import dask_progress
+
+        ctx = dask_progress("stage tiles", progress)
         logger.info("Staging per-tile labels to %s …", stage_path)
         with ctx:
             dask.compute(

patchworks-2.5.0/src/patchworks/_notify.py

@@ -0,0 +1,157 @@
+"""Best-effort email notification.
+
+Used for the events SLURM cannot report on its own: the end of a whole
+multi-config run, and failures in a local (non-SLURM) run. Per-job
+start/finish mail on a cluster is left to SLURM's own ``--mail-type``, which
+is delivered by the controller and does not depend on a compute node being
+able to reach an MTA.
+
+Every function here is best-effort by design: a notification that cannot be
+delivered must never fail a pipeline that otherwise succeeded, and must never
+turn a real error into a confusing one about email.
+"""
+
+from __future__ import annotations
+
+import logging
+import shutil
+import socket
+import subprocess
+from email.message import EmailMessage
+from pathlib import Path
+from typing import Union
+
+logger = logging.getLogger(__name__)
+
+# Lines of a failing step's log to quote in an error mail. Enough to carry a
+# traceback, short enough that the mail stays readable on a phone.
+LOG_TAIL_LINES = 40
+
+
+def log_tail(path: Union[str, Path], lines: int = LOG_TAIL_LINES) -> str:
+    """Return the last *lines* of a log file, or a note if unreadable."""
+    try:
+        content = Path(path).read_text(errors="replace").splitlines()
+    except OSError as exc:
+        return f"(could not read {path}: {exc})"
+    if not content:
+        return f"({path} is empty)"
+    return "\n".join(content[-lines:])
+
+
+def send(
+    to: Union[str, None],
+    subject: str,
+    body: str,
+    *,
+    sender: Union[str, None] = None,
+) -> bool:
+    """Send *body* to *to*, returning whether it went out.
+
+    Tries a local ``sendmail`` first (what a cluster node normally has), then
+    an SMTP server on localhost. Never raises.
+
+    Parameters
+    ----------
+    to : str or None
+        Recipient. ``None`` or empty disables the notification entirely, which
+        is the default state -- no address configured, no mail, no error.
+    subject : str
+        Subject line.
+    body : str
+        Plain-text body.
+    sender : str, optional
+        From address. Defaults to ``patchworks@<hostname>``.
+
+    Returns
+    -------
+    bool
+        True when the message was handed to a transport.
+    """
+    if not to:
+        return False
+
+    msg = EmailMessage()
+    msg["To"] = to
+    msg["From"] = sender or f"patchworks@{socket.getfqdn()}"
+    msg["Subject"] = subject
+    msg.set_content(body)
+
+    sendmail = shutil.which("sendmail") or shutil.which(
+        "sendmail", path="/usr/sbin:/usr/lib"
+    )
+    if sendmail:
+        try:
+            subprocess.run(
+                [sendmail, "-t", "-oi"],
+                input=msg.as_bytes(),
+                check=True,
+                capture_output=True,
+                timeout=30,
+            )
+            return True
+        except (subprocess.SubprocessError, OSError) as exc:
+            logger.warning(
+                "sendmail failed (%s); trying SMTP on localhost", exc
+            )
+
+    try:
+        import smtplib
+
+        with smtplib.SMTP("localhost", timeout=30) as smtp:
+            smtp.send_message(msg)
+        return True
+    except (OSError, Exception) as exc:  # noqa: BLE001 - never fail the run
+        logger.warning(
+            "could not send notification to %s (%s). The run itself is "
+            "unaffected; set notify_email to '' to silence this.",
+            to,
+            exc,
+        )
+        return False
+
+
+def slurm_mail_extra(
+    email: Union[str, None], events: Union[list, tuple, None] = None
+) -> str:
+    """Build the ``slurm_extra`` fragment for per-job mail.
+
+    SLURM's own ``--mail-type`` is used rather than sending from inside the
+    job: the controller delivers it, so it still arrives when the job is
+    killed by the OOM reaper or the scheduler -- exactly the cases worth
+    hearing about, and exactly the ones an in-job notification misses.
+
+    Parameters
+    ----------
+    email : str or None
+        Recipient; empty/None yields an empty string (no mail configured).
+    events : sequence of str, optional
+        Any of ``"start"``, ``"finish"``, ``"error"``. Defaults to finish and
+        error -- a BEGIN mail per job is rarely worth the inbox.
+
+    Returns
+    -------
+    str
+        Something like ``--mail-type=END,FAIL --mail-user=me@example.org``,
+        or ``""`` when no address is configured.
+
+    Examples
+    --------
+    >>> slurm_mail_extra("me@example.org", ["error"])
+    '--mail-type=FAIL --mail-user=me@example.org'
+    >>> slurm_mail_extra(None)
+    ''
+    """
+    if not email:
+        return ""
+    mapping = {"start": "BEGIN", "finish": "END", "error": "FAIL"}
+    chosen = list(events) if events else ["finish", "error"]
+    unknown = sorted(set(chosen) - set(mapping))
+    if unknown:
+        raise ValueError(
+            f"unknown notify_events {unknown}; use any of "
+            f"{sorted(mapping)} (they map to SLURM's BEGIN/END/FAIL)"
+        )
+    # Keep SLURM's own order, not the config's, so the string is stable.
+    types = [mapping[k] for k in ("start", "finish", "error") if k in chosen]
+    return f"--mail-type={','.join(types)} --mail-user={email}"

{patchworks-2.3.0 → patchworks-2.5.0}/src/patchworks/_occupancy.py

@@ -36,6 +36,7 @@ import numpy as np
 import zarr
 
 from ._chunks import cpu_allocation
+from ._progress import track
 
 logger = logging.getLogger(__name__)
 
@@ -271,13 +272,19 @@ def build_occupancy_map(
     try:
         n_workers = max(1, min(cpu_allocation(), len(regions)))
         if n_workers <= 1:
-            for starts in regions:
-                _one(starts)
+            for _ in track(
+                (_one(starts) for starts in regions),
+                "occupancy map",
+                len(regions),
+            ):
+                pass
         else:
             # Reads and decompression release the GIL, so threads are enough
             # and there is no worker payload to pickle.
             with ThreadPoolExecutor(max_workers=n_workers) as pool:
-                for _ in pool.map(_one, regions):
+                for _ in track(
+                    pool.map(_one, regions), "occupancy map", len(regions)
+                ):
                     pass
         dst.attrs["block"] = list(block)
         dst.attrs["level"] = int(level)

patchworks-2.5.0/src/patchworks/_progress.py

@@ -0,0 +1,184 @@
+"""Progress reporting that survives being written to a log file.
+
+Every long step in the workflow runs unattended in a batch job, where the
+output is read hours later out of a file. A redrawing bar (dask's
+``ProgressBar``, ``tqdm``) collapses into one enormous unreadable line there,
+so the default here is periodic log records instead -- and a bar only when
+someone is actually watching a terminal.
+
+The interval matters more than the precision: the job runs for hours, and the
+question being answered is "is this working or hung?", not "exactly how far".
+"""
+
+from __future__ import annotations
+
+import logging
+import sys
+import time
+from contextlib import nullcontext
+from typing import Iterable, Iterator, TypeVar
+
+logger = logging.getLogger(__name__)
+
+T = TypeVar("T")
+
+# Often enough to tell progress from a hang, rare enough that a six-hour job
+# leaves a log you can still read.
+PROGRESS_INTERVAL_S = 60.0
+
+
+def is_interactive() -> bool:
+    """True when someone is plausibly watching a terminal."""
+    return bool(getattr(sys.stderr, "isatty", lambda: False)())
+
+
+def format_eta(done: int, total: int, elapsed: float) -> str:
+    """Rough remaining time from a linear extrapolation.
+
+    Parameters
+    ----------
+    done, total : int
+        Units finished and expected.
+    elapsed : float
+        Seconds spent so far.
+
+    Returns
+    -------
+    str
+        A short human-readable duration, or ``"?"`` when it cannot be
+        estimated yet.
+    """
+    if done <= 0 or done >= total:
+        return "?"
+    left = elapsed / done * (total - done)
+    if left < 90:
+        return f"{left:.0f}s"
+    if left < 5400:
+        return f"{left / 60:.0f}m"
+    return f"{left / 3600:.1f}h"
+
+
+def log_progress(label: str, done: int, total: int, started: float) -> None:
+    """Emit one progress line.
+
+    Parameters
+    ----------
+    label : str
+        What is being worked on, e.g. ``"image.zarr/0"``.
+    done, total : int
+        Units finished and expected.
+    started : float
+        ``time.monotonic()`` when the work began.
+    """
+    elapsed = time.monotonic() - started
+    logger.info(
+        "%s: %s/%s (%.0f%%) after %.0fm, ~%s left",
+        label,
+        f"{done:,}",
+        f"{total:,}",
+        100.0 * done / max(1, total),
+        elapsed / 60,
+        format_eta(done, total, elapsed),
+    )
+
+
+def track(
+    iterable: Iterable[T],
+    label: str,
+    total: int,
+    *,
+    enabled: bool = True,
+) -> Iterator[T]:
+    """Yield from *iterable*, reporting progress as it goes.
+
+    Uses ``tqdm`` when attached to a terminal and periodic log lines
+    otherwise, so the same call is right in a notebook and in a SLURM job.
+
+    Parameters
+    ----------
+    iterable : iterable
+        The work to iterate. Consumed lazily, so this is safe over
+        ``imap_unordered``.
+    label : str
+        Description of the work.
+    total : int
+        Expected number of items, used for the percentage and the ETA.
+    enabled : bool, optional
+        Set ``False`` to pass items straight through. Default ``True``.
+
+    Yields
+    ------
+    object
+        The items of *iterable*, unchanged.
+    """
+    if not enabled:
+        yield from iterable
+        return
+
+    if is_interactive():
+        try:
+            from tqdm.auto import tqdm
+
+            yield from tqdm(iterable, total=total, desc=label)
+            return
+        except ImportError:
+            pass
+
+    started = time.monotonic()
+    last = started
+    done = 0
+    logger.info("%s: starting (%s items)", label, f"{total:,}")
+    for item in iterable:
+        yield item
+        done += 1
+        now = time.monotonic()
+        if now - last >= PROGRESS_INTERVAL_S:
+            last = now
+            log_progress(label, done, total, started)
+    log_progress(label, done, total, started)
+
+
+def dask_progress(label: str, enabled: bool = True):
+    """Progress context manager for a dask computation.
+
+    Returns
+    -------
+    contextmanager
+        ``ProgressBar`` on a terminal, a periodically-logging callback
+        otherwise, or a no-op when *enabled* is false.
+    """
+    if not enabled:
+        return nullcontext()
+
+    logger.info("writing %s …", label)
+    if is_interactive():
+        from dask.diagnostics import ProgressBar
+
+        return ProgressBar()
+
+    from dask.callbacks import Callback
+
+    class _LogProgress(Callback):
+        """Count finished dask tasks and log every PROGRESS_INTERVAL_S."""
+
+        def _start_state(self, dsk, state):
+            self._total = sum(
+                len(state[k])
+                for k in ("ready", "waiting", "running", "finished")
+            )
+            self._done = 0
+            self._t0 = time.monotonic()
+            self._last = self._t0
+
+        def _posttask(self, key, result, dsk, state, worker_id):
+            self._done += 1
+            now = time.monotonic()
+            if now - self._last >= PROGRESS_INTERVAL_S:
+                self._last = now
+                log_progress(label, self._done, self._total, self._t0)
+
+        def _finish(self, dsk, state, errored):
+            if not errored:
+                log_progress(label, self._total, self._total, self._t0)
+
+    return _LogProgress()