stepup-queue 1.1.0__tar.gz → 2.0.0rc1__tar.gz

stepup_queue-2.0.0rc1/CLAUDE.md

@@ -0,0 +1,126 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+StepUp Queue is a StepUp Core extension that integrates SLURM job scheduler workflows. It allows
+StepUp workflows to submit SLURM jobs, wait for them, and resume from existing jobs after restarts
+— making long-running HPC workflows resumable across interrupted sessions.
+
+The related `stepup-core` repo is at `../stepup-core` and on GitHub.
+
+## Development Environment
+
+Uses [uv](https://docs.astral.sh/uv/) for environment management:
+
+```bash
+uv sync --extra dev
+pre-commit install
+direnv allow   # activates .venv and sets env vars from .envrc
+```
+
+The `.envrc` sets `STEPUP_DEBUG=1`, `STEPUP_BUILD_DURATION=0`, and `STEPUP_SYNC_RPC_TIMEOUT=30`.
+Without `direnv`, prefix commands with `uv run`.
+
+## Common Commands
+
+```bash
+# Run all tests (parallel by default via pytest-xdist, quite fast)
+pytest -vv
+
+# Run all linters
+pre-commit run --all
+
+# Docs live preview
+mkdocs serve
+```
+
+## Architecture
+
+### Package layout
+
+```text
+stepup/queue/
+  api.py         — Public Python API: sbatch() for use in plan.py files
+  sbatch.py      — sq-sbatch-and-wait CLI: submits, waits, polls, caches sacct output
+  log.py         — slurmjob.log format (version 2): read/write/validate
+  utils.py       — SLURM state sets, parse_sbatch(), search_jobs()
+  canceljobs.py  — stepup canceljobs subcommand
+  removejobs.py  — stepup removejobs subcommand
+```
+
+### How it fits into StepUp
+
+`stepup.queue.api.sbatch()` is called from a user's `plan.py`. It calls
+`stepup.core.api.run()` to register the `sq-sbatch-and-wait` step with StepUp Core.
+When StepUp executes that step, `sq-sbatch-and-wait` (entry point for `stepup/queue/sbatch.py`)
+runs in the working directory of the job.
+
+### Job lifecycle and files
+
+Every SLURM job lives in its own working directory. The conventions are:
+
+- `slurmjob{ext}` — the user-written job script (must be executable, must have shebang)
+- `slurmjob.log` — StepUp Queue's log (volatile; tracks submission + SLURM state history)
+- `slurmjob.out` / `slurmjob.err` — SLURM stdout/stderr (declared as `out`)
+- `slurmjob.ret` — exit code written by wrapper script (declared as `out`)
+
+`slurmjob.log` is declared as a `vol` (volatile) file in StepUp, not `out`, so it is not
+treated as reproducible output. It contains: a version header, an input digest (SHA-256 of
+all step inputs), and timestamped status lines (`Submitted <jobid>[;cluster]`, then SLURM states).
+
+### Idempotent submit-and-wait
+
+`submit_once_and_wait()` in `sbatch.py` is the core function:
+
+1. Reads `slurmjob.log` and checks the stored input digest against `STEPUP_STEP_INP_DIGEST`.
+2. If no log exists → submits a new job via `sbatch --parsable`.
+3. If log exists with a matching digest → resumes waiting for the existing job.
+4. If digest mismatch → behaviour depends on `onchange` policy (`raise`/`resubmit`/`ignore`).
+5. Polls status via `sacct`, using a **shared on-disk cache** at
+   `.stepup/queue/sbatch_wait_sacct[.cluster].out` with `fcntl.LOCK_EX` to avoid
+   hammering SLURM when many jobs run in parallel.
+
+### sacct caching
+
+`cached_run()` in `sbatch.py` manages the shared `sacct` cache. All concurrent `sq-sbatch-and-wait`
+processes share a single cached file per cluster; only one process calls `sacct` at a time (via
+`fcntl` lock). The cache file has a fixed-length header (`v1 datetime=... returncode=...`).
+
+### Entry points
+
+- `sq-sbatch-and-wait` — CLI that wraps `sbatch()` → `submit_once_and_wait()`
+- `stepup canceljobs` — registered as `stepup.tools` entry point; cancels running SLURM jobs
+  by reading `slurmjob.log` files recursively
+- `stepup removejobs` — registered as `stepup.tools` entry point; removes directories of failed jobs
+
+### Key environment variables
+
+| Variable | Default | Purpose |
+| --- | --- | --- |
+| `STEPUP_SBATCH_CACHE_TIMEOUT` | 30 | Seconds between sacct calls |
+| `STEPUP_SBATCH_POLLING_MIN/MAX` | 10/20 | Random polling interval (seconds) |
+| `STEPUP_SBATCH_RETRY_NUM` | 5 | sbatch retry attempts on transient failure |
+| `STEPUP_SBATCH_RETRY_DELAY_MIN/MAX` | 60/120 | Retry delay range (seconds) |
+| `STEPUP_SACCT_START_TIME` | now-7days | `-S` argument passed to sacct |
+| `STEPUP_SBATCH_UNLISTED_TIMEOUT` | 600 | Seconds before unlisted job is declared failed |
+| `STEPUP_QUEUE_ONCHANGE` | raise | Default `onchange` policy |
+
+### Linting
+
+Ruff with `line-length = 100`, targeting Python 3.11+. The `ruff.lint` section in
+`pyproject.toml` selects many rule sets; several `PLR` (complexity) rules are deliberately
+disabled. Imports are sorted with `stepup` as a known-first-party package.
+
+### Testing
+
+`pytest` is configured with `-n auto --dist worksteal -W error` — all warnings are errors,
+tests run in parallel. The `conftest.py` provides only a `path_tmp` fixture wrapping `tmpdir`.
+Tests are pure unit tests; no SLURM cluster is required.
+
+## Release Process
+
+1. Update `docs/changelog.md` with the new version.
+2. Commit and tag: `git tag vX.Y.Z`.
+3. Push with tags: `git push origin main --tags` (triggers PyPI GitHub Action).

{stepup_queue-1.1.0/stepup_queue.egg-info → stepup_queue-2.0.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: stepup-queue
-Version: 1.1.0
+Version: 2.0.0rc1
 Summary: StepUp Queue integrates queued jobs into a StepUp workflow.
 Author-email: Toon Verstraelen <toon.verstraelen@ugent.be>
 License-Expression: GPL-3.0-or-later
@@ -25,7 +25,8 @@ Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: path>=16.14.0
-Requires-Dist: stepup<4.0.0,>=3.2.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: stepup<5.0.0a1,>=4.0.0rc3
 Provides-Extra: dev
 Requires-Dist: psutil; extra == "dev"
 Requires-Dist: pytest; extra == "dev"

{stepup_queue-1.1.0 → stepup_queue-2.0.0rc1}/pyproject.toml

@@ -29,7 +29,8 @@ classifiers = [
 dependencies = [
     # Ensure changes to these dependencies are reflected in .github/requirements-old.txt
     "path>=16.14.0",
-    "stepup>=3.2.0,<4.0.0",
+    "rich>=13.0.0",
+    "stepup>=4.0.0rc3,<5.0.0a1",
 ]
 dynamic = ["version"]
 
@@ -52,8 +53,8 @@ Issues = "https://github.com/reproducible-reporting/stepup-queue/issues"
 Source = "https://github.com/reproducible-reporting/stepup-queue/"
 Changelog = "https://reproducible-reporting.github.io/stepup-queue/changelog/"
 
-[project.entry-points."stepup.actions"]
-sbatch = "stepup.queue.actions:sbatch"
+[project.scripts]
+sq-sbatch-and-wait = "stepup.queue.sbatch:sbatch"
 
 [project.entry-points."stepup.tools"]
 canceljobs = "stepup.queue.canceljobs:canceljobs_subcommand"

{stepup_queue-1.1.0 → stepup_queue-2.0.0rc1}/stepup/queue/__init__.py

@@ -1,5 +1,5 @@
 # StepUp Queue integrates queued jobs into a StepUp workflow.
-# © 2025 Toon Verstraelen
+# Copyright 2025-2026 Toon Verstraelen
 #
 # This file is part of StepUp Queue.
 #

{stepup_queue-1.1.0 → stepup_queue-2.0.0rc1}/stepup/queue/api.py

@@ -1,5 +1,5 @@
 # StepUp Queue integrates queued jobs into a StepUp workflow.
-# © 2025 Toon Verstraelen
+# Copyright 2025-2026 Toon Verstraelen
 #
 # This file is part of StepUp Queue.
 #
@@ -22,25 +22,24 @@
 import shlex
 from collections.abc import Collection
 
-from stepup.core.api import step
-from stepup.core.utils import string_to_list
+from stepup.core.api import run
+from stepup.core.path import StrPath, coerce_paths
 
 __all__ = ("sbatch",)
 
 
 def sbatch(
-    workdir: str,
+    workdir: StrPath,
     *,
     ext: str = ".sh",
     rc: str | None = None,
-    inp: Collection[str] | str = (),
+    inp: Collection[StrPath] | StrPath = (),
     env: Collection[str] | str = (),
-    out: Collection[str] | str = (),
-    vol: Collection[str] | str = (),
+    out: Collection[StrPath] | StrPath = (),
+    vol: Collection[StrPath] | StrPath = (),
     onchange: str | None = None,
     optional: bool = False,
-    pool: str | None = None,
-    block: bool = False,
+    resources: dict[str, int] | str | None = None,
 ):
     """Submit a SLURM job script.
 
@@ -60,8 +59,7 @@ def sbatch(
     If submitted, the step will wait until the job is finished.
     If already finished, the step will essentially be a no-op.
 
-    See `step()` documentation in StepUp Core for all optional arguments.
-    and the return value.
+    See `run()` documentation in StepUp Core for all optional arguments and return value.
     Note that the `inp`, `out` and `vol` arguments are extended
     with the files mentioned above and that any additional files you specify
     are interpreted relative to the working directory.
@@ -90,23 +88,22 @@ def sbatch(
         ext = f".{ext}"
     if ext in [".log", ".out", ".err", ".ret"]:
         raise ValueError(f"Invalid extension {ext}. The extension must not be .log, .out or .err.")
-    action = "sbatch"
+    cmd = "sq-sbatch-and-wait"
     if ext != ".sh":
-        action += f" {ext}"
+        cmd += f" {ext}"
     if rc is not None:
-        action += f" --rc={shlex.quote(rc)}"
+        cmd += f" --rc={shlex.quote(rc)}"
     if onchange is not None:
         if onchange not in ["raise", "resubmit", "ignore"]:
             raise ValueError(f"Invalid onchange policy {onchange}.")
-        action += f" --onchange={onchange}"
-    return step(
-        action,
-        inp=[f"slurmjob{ext}", *string_to_list(inp)],
+        cmd += f" --onchange={onchange}"
+    return run(
+        cmd,
+        inp=[f"slurmjob{ext}", *coerce_paths(inp)],
         env=env,
-        out=["slurmjob.out", "slurmjob.err", "slurmjob.ret", *string_to_list(out)],
-        vol=["slurmjob.log", *string_to_list(vol)],
+        out=["slurmjob.out", "slurmjob.err", "slurmjob.ret", *coerce_paths(out)],
+        vol=["slurmjob.log", *coerce_paths(vol)],
         workdir=workdir,
         optional=optional,
-        pool=pool,
-        block=block,
+        resources=resources,
     )

{stepup_queue-1.1.0 → stepup_queue-2.0.0rc1}/stepup/queue/canceljobs.py

@@ -1,5 +1,5 @@
 # StepUp Queue integrates queued jobs into a StepUp workflow.
-# © 2025 Toon Verstraelen
+# Copyright 2025-2026 Toon Verstraelen
 #
 # This file is part of StepUp Queue.
 #
@@ -22,21 +22,30 @@
 import argparse
 import subprocess
 import sys
+from collections.abc import Callable
 
 from path import Path
+from rich.console import Console
 
-from .sbatch import DONE_STATES, parse_sbatch, read_log, read_status
-from .utils import search_jobs
+from stepup.core.config import ConfigLoader
+
+from .log import read_jobid_cluster_status
+from .utils import DONE_STATES, search_jobs
 
 
 def canceljobs_tool(args: argparse.Namespace):
     """Iterate over all slurmjob.log files, read the SLURM job IDs, and cancel them."""
+    console = Console(highlight=False)
+    if not args.commit:
+        console.print("[yellow]# Note: No jobs are actually cancelled.[/]")
+        console.print("[yellow]# Use the --commit option to execute the cancellations.[/]")
+
     jobs = {}
-    for path_log in search_jobs(args.paths, verbose=True):
+    for path_log in search_jobs(args.paths, console):
         try:
             job_id, cluster, status = read_jobid_cluster_status(path_log)
         except ValueError as e:
-            print(f"# WARNING: Could not read job ID from {path_log}: {e}")
+            console.print(f"[red]# WARNING: Could not read job ID from {path_log}: {e}[/]")
             continue
         if args.all or status not in DONE_STATES:
             jobs.setdefault(cluster, []).append((job_id, path_log, status))
@@ -56,39 +65,21 @@ def canceljobs_tool(args: argparse.Namespace):
                 command_args.extend(str(job_id) for job_id, _, _ in cancel_jobs)
 
                 # Using subprocess.run for better control and error handling
-                print(" ".join(command_args))
+                print_cancel_command(
+                    console, [job_id for job_id, _, _ in cancel_jobs], cluster, None
+                )
                 result = subprocess.run(command_args, check=False)
                 all_good &= result.returncode == 0
         else:
             for job_id, path_log, status in cluster_jobs:
-                command = "scancel"
-                if cluster is not None:
-                    command += f" -M {cluster}"
-                command += f" {job_id}  # {path_log} {status}"
-                print(command)
+                print_cancel_command(console, [job_id], cluster, f"{path_log} {status}")
     if not all_good:
-        print("Some jobs could not be cancelled. See messages above.")
+        console.print("[red]Some jobs could not be cancelled. See messages above.[/]")
         sys.exit(1)
 
 
-def read_jobid_cluster_status(path_log: str) -> tuple[int, str | None, str | None]:
-    """Read the job ID, cluster, and job status from the job log file."""
-    lines = read_log(path_log, False)
-    if len(lines) < 1:
-        raise ValueError(f"Incomplete file: {path_log}.")
-    words = lines[0].split()
-    if len(words) != 3:
-        raise ValueError(f"Could not read job ID from first status line: {lines[0]}")
-    _, status, job_id_cluster = words
-    if status != "Submitted":
-        raise ValueError(f"No 'Submitted' on first status line: {lines[0]}")
-    job_id, cluster = parse_sbatch(job_id_cluster)
-    status = read_status(lines[-1:])[1]
-    return job_id, cluster, status
-
-
-def canceljobs_subcommand(subparser: argparse.ArgumentParser) -> callable:
-    parser = subparser.add_parser(
+def canceljobs_subcommand(subparsers, loader: ConfigLoader) -> Callable:
+    parser = subparsers.add_parser(
         "canceljobs",
         help="Cancel running jobs in the current StepUp workflow.",
     )
@@ -114,4 +105,18 @@ def canceljobs_subcommand(subparser: argparse.ArgumentParser) -> callable:
         default=False,
         help="Select all jobs, including the ones that seem to be done already.",
     )
+    loader.patch_parser(parser)
     return canceljobs_tool
+
+
+def print_cancel_command(
+    console: Console, job_ids: list[int], cluster: str | None, comment: str | None
+) -> str:
+    """Print the job cancellation command."""
+    parts = ["[green]scancel[/]"]
+    if cluster is not None:
+        parts.append(f"[cyan]-M {cluster}[/]")
+    parts.extend(str(job_id) for job_id in job_ids)
+    if comment is not None:
+        parts.append(f" [bright_black]# {comment}[/]")
+    console.print(" ".join(parts))

stepup_queue-2.0.0rc1/stepup/queue/log.py

@@ -0,0 +1,121 @@
+# StepUp Queue integrates queued jobs into a StepUp workflow.
+# Copyright 2025-2026 Toon Verstraelen
+#
+# This file is part of StepUp Queue.
+#
+# StepUp Queue is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 3
+# of the License, or (at your option) any later version.
+#
+# StepUp Queue is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, see <http://www.gnu.org/licenses/>
+#
+# --
+"""The job log file format and utilities to read and write it."""
+
+from datetime import datetime
+
+from path import Path
+
+from .utils import parse_sbatch
+
+__all__ = (
+    "FIRST_LINE",
+    "InpDigestError",
+    "init_log",
+    "log_status",
+    "read_jobid_cluster_status",
+    "read_log",
+    "read_status",
+)
+
+FIRST_LINE = "StepUp Queue sbatch wait log format version 2"
+
+
+class InpDigestError(ValueError):
+    """The input digest in the log file does not match the one in the environment."""
+
+
+def init_log(path_log: str, inp_digest: str):
+    """Initialize a new log file."""
+    with open(path_log, "w") as fh:
+        print(FIRST_LINE, file=fh)
+        print(inp_digest, file=fh)
+
+
+def log_status(path_log: Path, status: str):
+    """Write a status to the log."""
+    dt = datetime.now().isoformat()
+    with open(path_log, "a") as f:
+        line = f"{dt} {status}"
+        f.write(f"{line}\n")
+
+
+def read_jobid_cluster_status(path_log: str) -> tuple[int, str | None, str | None]:
+    """Read the job ID, cluster, and job status from the job log file."""
+    lines = read_log(path_log, None)
+    if len(lines) < 1:
+        raise ValueError(f"Incomplete file: {path_log}.")
+    words = lines[0].split()
+    if len(words) != 3:
+        raise ValueError(f"Could not read job ID from first status line: {lines[0]}")
+    _, status, job_id_cluster = words
+    if status != "Submitted":
+        raise ValueError(f"No 'Submitted' on first status line: {lines[0]}")
+    job_id, cluster = parse_sbatch(job_id_cluster)
+    status = read_status(lines[-1:])[1]
+    return job_id, cluster, status
+
+
+def read_log(path_log: str, expected_inp_digest: str | None = None) -> list[str]:
+    """Read lines from a previously created log file."""
+    lines = []
+    with open(path_log) as f:
+        try:
+            check_log_version(next(f).strip())
+        except StopIteration as exc:
+            raise ValueError("Existing log file is empty.") from exc
+        try:
+            actual_inp_digest = next(f).strip()
+        except StopIteration as exc:
+            raise ValueError("Existing log file has no input digest.") from exc
+        if expected_inp_digest is not None:
+            check_log_inp_digest(actual_inp_digest, expected_inp_digest)
+        for line in f:
+            line = line.strip()
+            lines.append(line)
+    return lines
+
+
+def check_log_version(line: str):
+    """Validate the log version, abort if there is a mismatch."""
+    if line != FIRST_LINE:
+        raise ValueError(
+            f"The first line of the log is wrong. Expected: '{FIRST_LINE}' Found: '{line}'"
+        )
+
+
+def check_log_inp_digest(actual: str, expected: str):
+    """Validate the log input digest, abort if there is a mismatch."""
+    if actual != expected:
+        raise InpDigestError(
+            "The second line of the log contains the wrong input digest.\n"
+            f"Actual:   {actual}\nExpected: {expected}\n"
+        )
+
+
+def read_status(lines: list[str]) -> tuple[float | None, str | None]:
+    """Read a status from the log file."""
+    if len(lines) == 0:
+        return None, None
+    line = lines.pop(0)
+    words = line.split(maxsplit=1)
+    if len(words) != 2:
+        raise ValueError(f"Expected a status in log but found line '{line}'.")
+    return datetime.fromisoformat(words[0]).timestamp(), words[1].strip()

{stepup_queue-1.1.0 → stepup_queue-2.0.0rc1}/stepup/queue/removejobs.py

@@ -1,5 +1,5 @@
 # StepUp Queue integrates queued jobs into a StepUp workflow.
-# © 2025 Toon Verstraelen
+# Copyright 2025-2026 Toon Verstraelen
 #
 # This file is part of StepUp Queue.
 #
@@ -21,10 +21,14 @@
 
 import argparse
 import shutil
+from collections.abc import Callable
 
 from path import Path
+from rich.console import Console
 
-from .sbatch import read_log, read_status
+from stepup.core.config import ConfigLoader
+
+from .log import read_log, read_status
 from .utils import search_jobs
 
 FAILED_STATES = {
@@ -45,31 +49,36 @@ FAILED_STATES = {
 
 def removejobs_tool(args: argparse.Namespace):
     """Iterate over all slurmjob.log files and remove their parent job directories."""
+    console = Console(highlight=False)
+    if not args.commit:
+        console.print("[yellow]# Note: No job directories are actually removed.[/]")
+        console.print("[yellow]# Use the --commit option to execute the removals.[/]")
+
     jobs = []
-    for path_log in search_jobs(args.paths, verbose=True):
+    for path_log in search_jobs(args.paths, console):
         try:
             status = read_last_status(path_log)
         except ValueError as e:
-            print(f"Warning: Could not read job status from {path_log}: {e}")
+            console.print(f"[red]# WARNING: Could not read job status from {path_log}: {e}[/]")
             status = None
         if args.all or status in FAILED_STATES:
             jobs.append((path_log, status))
 
     for path_log, status in jobs:
-        command = f"rm -rf {path_log.parent}  # state={status}"
-        print(command)
+        command = f"[cyan]rm -rf[/] {path_log.parent}  [bright_black]# state={status}[/]"
+        console.print(command)
         if args.commit:
             shutil.rmtree(path_log.parent)
 
 
 def read_last_status(path_log: str) -> str | None:
     """Read the last job status from the job log file."""
-    lines = read_log(path_log, False)
+    lines = read_log(path_log, None)
     return read_status(lines[-1:])[1]
 
 
-def removejobs_subcommand(subparser: argparse.ArgumentParser) -> callable:
-    parser = subparser.add_parser(
+def removejobs_subcommand(subparsers, loader: ConfigLoader) -> Callable:
+    parser = subparsers.add_parser(
         "removejobs",
         help="Remove directories of failed (and optionally all completed) jobs "
         "in the current StepUp workflow.",
@@ -96,4 +105,5 @@ def removejobs_subcommand(subparser: argparse.ArgumentParser) -> callable:
         default=False,
         help="Remove all jobs, not only failed jobs.",
     )
+    loader.patch_parser(parser)
     return removejobs_tool