stepup-queue 1.1.1__tar.gz → 2.0.0rc1__tar.gz

stepup_queue-2.0.0rc1/CLAUDE.md

@@ -0,0 +1,126 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+StepUp Queue is a StepUp Core extension that integrates SLURM job scheduler workflows. It allows
+StepUp workflows to submit SLURM jobs, wait for them, and resume from existing jobs after restarts
+— making long-running HPC workflows resumable across interrupted sessions.
+
+The related `stepup-core` repo is at `../stepup-core` and on GitHub.
+
+## Development Environment
+
+Uses [uv](https://docs.astral.sh/uv/) for environment management:
+
+```bash
+uv sync --extra dev
+pre-commit install
+direnv allow   # activates .venv and sets env vars from .envrc
+```
+
+The `.envrc` sets `STEPUP_DEBUG=1`, `STEPUP_BUILD_DURATION=0`, and `STEPUP_SYNC_RPC_TIMEOUT=30`.
+Without `direnv`, prefix commands with `uv run`.
+
+## Common Commands
+
+```bash
+# Run all tests (parallel by default via pytest-xdist, quite fast)
+pytest -vv
+
+# Run all linters
+pre-commit run --all
+
+# Docs live preview
+mkdocs serve
+```
+
+## Architecture
+
+### Package layout
+
+```text
+stepup/queue/
+  api.py         — Public Python API: sbatch() for use in plan.py files
+  sbatch.py      — sq-sbatch-and-wait CLI: submits, waits, polls, caches sacct output
+  log.py         — slurmjob.log format (version 2): read/write/validate
+  utils.py       — SLURM state sets, parse_sbatch(), search_jobs()
+  canceljobs.py  — stepup canceljobs subcommand
+  removejobs.py  — stepup removejobs subcommand
+```
+
+### How it fits into StepUp
+
+`stepup.queue.api.sbatch()` is called from a user's `plan.py`. It calls
+`stepup.core.api.run()` to register the `sq-sbatch-and-wait` step with StepUp Core.
+When StepUp executes that step, `sq-sbatch-and-wait` (entry point for `stepup/queue/sbatch.py`)
+runs in the working directory of the job.
+
+### Job lifecycle and files
+
+Every SLURM job lives in its own working directory. The conventions are:
+
+- `slurmjob{ext}` — the user-written job script (must be executable, must have shebang)
+- `slurmjob.log` — StepUp Queue's log (volatile; tracks submission + SLURM state history)
+- `slurmjob.out` / `slurmjob.err` — SLURM stdout/stderr (declared as `out`)
+- `slurmjob.ret` — exit code written by wrapper script (declared as `out`)
+
+`slurmjob.log` is declared as a `vol` (volatile) file in StepUp, not `out`, so it is not
+treated as reproducible output. It contains: a version header, an input digest (SHA-256 of
+all step inputs), and timestamped status lines (`Submitted <jobid>[;cluster]`, then SLURM states).
+
+### Idempotent submit-and-wait
+
+`submit_once_and_wait()` in `sbatch.py` is the core function:
+
+1. Reads `slurmjob.log` and checks the stored input digest against `STEPUP_STEP_INP_DIGEST`.
+2. If no log exists → submits a new job via `sbatch --parsable`.
+3. If log exists with a matching digest → resumes waiting for the existing job.
+4. If digest mismatch → behaviour depends on `onchange` policy (`raise`/`resubmit`/`ignore`).
+5. Polls status via `sacct`, using a **shared on-disk cache** at
+   `.stepup/queue/sbatch_wait_sacct[.cluster].out` with `fcntl.LOCK_EX` to avoid
+   hammering SLURM when many jobs run in parallel.
+
+### sacct caching
+
+`cached_run()` in `sbatch.py` manages the shared `sacct` cache. All concurrent `sq-sbatch-and-wait`
+processes share a single cached file per cluster; only one process calls `sacct` at a time (via
+`fcntl` lock). The cache file has a fixed-length header (`v1 datetime=... returncode=...`).
+
+### Entry points
+
+- `sq-sbatch-and-wait` — CLI that wraps `sbatch()` → `submit_once_and_wait()`
+- `stepup canceljobs` — registered as `stepup.tools` entry point; cancels running SLURM jobs
+  by reading `slurmjob.log` files recursively
+- `stepup removejobs` — registered as `stepup.tools` entry point; removes directories of failed jobs
+
+### Key environment variables
+
+| Variable | Default | Purpose |
+| --- | --- | --- |
+| `STEPUP_SBATCH_CACHE_TIMEOUT` | 30 | Seconds between sacct calls |
+| `STEPUP_SBATCH_POLLING_MIN/MAX` | 10/20 | Random polling interval (seconds) |
+| `STEPUP_SBATCH_RETRY_NUM` | 5 | sbatch retry attempts on transient failure |
+| `STEPUP_SBATCH_RETRY_DELAY_MIN/MAX` | 60/120 | Retry delay range (seconds) |
+| `STEPUP_SACCT_START_TIME` | now-7days | `-S` argument passed to sacct |
+| `STEPUP_SBATCH_UNLISTED_TIMEOUT` | 600 | Seconds before unlisted job is declared failed |
+| `STEPUP_QUEUE_ONCHANGE` | raise | Default `onchange` policy |
+
+### Linting
+
+Ruff with `line-length = 100`, targeting Python 3.11+. The `ruff.lint` section in
+`pyproject.toml` selects many rule sets; several `PLR` (complexity) rules are deliberately
+disabled. Imports are sorted with `stepup` as a known-first-party package.
+
+### Testing
+
+`pytest` is configured with `-n auto --dist worksteal -W error` — all warnings are errors,
+tests run in parallel. The `conftest.py` provides only a `path_tmp` fixture wrapping `tmpdir`.
+Tests are pure unit tests; no SLURM cluster is required.
+
+## Release Process
+
+1. Update `docs/changelog.md` with the new version.
+2. Commit and tag: `git tag vX.Y.Z`.
+3. Push with tags: `git push origin main --tags` (triggers PyPI GitHub Action).

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: stepup-queue
-Version: 1.1.1
+Version: 2.0.0rc1
 Summary: StepUp Queue integrates queued jobs into a StepUp workflow.
 Author-email: Toon Verstraelen <toon.verstraelen@ugent.be>
 License-Expression: GPL-3.0-or-later
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: path>=16.14.0
 Requires-Dist: rich>=13.0.0
-Requires-Dist: stepup<4.0.0,>=3.2.0
+Requires-Dist: stepup<5.0.0a1,>=4.0.0rc3
 Provides-Extra: dev
 Requires-Dist: psutil; extra == "dev"
 Requires-Dist: pytest; extra == "dev"

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/pyproject.toml

@@ -30,7 +30,7 @@ dependencies = [
     # Ensure changes to these dependencies are reflected in .github/requirements-old.txt
     "path>=16.14.0",
     "rich>=13.0.0",
-    "stepup>=3.2.0,<4.0.0",
+    "stepup>=4.0.0rc3,<5.0.0a1",
 ]
 dynamic = ["version"]
 
@@ -53,8 +53,8 @@ Issues = "https://github.com/reproducible-reporting/stepup-queue/issues"
 Source = "https://github.com/reproducible-reporting/stepup-queue/"
 Changelog = "https://reproducible-reporting.github.io/stepup-queue/changelog/"
 
-[project.entry-points."stepup.actions"]
-sbatch = "stepup.queue.actions:sbatch"
+[project.scripts]
+sq-sbatch-and-wait = "stepup.queue.sbatch:sbatch"
 
 [project.entry-points."stepup.tools"]
 canceljobs = "stepup.queue.canceljobs:canceljobs_subcommand"

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/stepup/queue/api.py

@@ -22,25 +22,24 @@
 import shlex
 from collections.abc import Collection
 
-from stepup.core.api import step
-from stepup.core.utils import string_to_list
+from stepup.core.api import run
+from stepup.core.path import StrPath, coerce_paths
 
 __all__ = ("sbatch",)
 
 
 def sbatch(
-    workdir: str,
+    workdir: StrPath,
     *,
     ext: str = ".sh",
     rc: str | None = None,
-    inp: Collection[str] | str = (),
+    inp: Collection[StrPath] | StrPath = (),
     env: Collection[str] | str = (),
-    out: Collection[str] | str = (),
-    vol: Collection[str] | str = (),
+    out: Collection[StrPath] | StrPath = (),
+    vol: Collection[StrPath] | StrPath = (),
     onchange: str | None = None,
     optional: bool = False,
-    pool: str | None = None,
-    block: bool = False,
+    resources: dict[str, int] | str | None = None,
 ):
     """Submit a SLURM job script.
 
@@ -60,8 +59,7 @@ def sbatch(
     If submitted, the step will wait until the job is finished.
     If already finished, the step will essentially be a no-op.
 
-    See `step()` documentation in StepUp Core for all optional arguments.
-    and the return value.
+    See `run()` documentation in StepUp Core for all optional arguments and return value.
     Note that the `inp`, `out` and `vol` arguments are extended
     with the files mentioned above and that any additional files you specify
     are interpreted relative to the working directory.
@@ -90,23 +88,22 @@ def sbatch(
         ext = f".{ext}"
     if ext in [".log", ".out", ".err", ".ret"]:
         raise ValueError(f"Invalid extension {ext}. The extension must not be .log, .out or .err.")
-    action = "sbatch"
+    cmd = "sq-sbatch-and-wait"
     if ext != ".sh":
-        action += f" {ext}"
+        cmd += f" {ext}"
     if rc is not None:
-        action += f" --rc={shlex.quote(rc)}"
+        cmd += f" --rc={shlex.quote(rc)}"
     if onchange is not None:
         if onchange not in ["raise", "resubmit", "ignore"]:
             raise ValueError(f"Invalid onchange policy {onchange}.")
-        action += f" --onchange={onchange}"
-    return step(
-        action,
-        inp=[f"slurmjob{ext}", *string_to_list(inp)],
+        cmd += f" --onchange={onchange}"
+    return run(
+        cmd,
+        inp=[f"slurmjob{ext}", *coerce_paths(inp)],
         env=env,
-        out=["slurmjob.out", "slurmjob.err", "slurmjob.ret", *string_to_list(out)],
-        vol=["slurmjob.log", *string_to_list(vol)],
+        out=["slurmjob.out", "slurmjob.err", "slurmjob.ret", *coerce_paths(out)],
+        vol=["slurmjob.log", *coerce_paths(vol)],
         workdir=workdir,
         optional=optional,
-        pool=pool,
-        block=block,
+        resources=resources,
     )

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/stepup/queue/canceljobs.py

@@ -22,12 +22,15 @@
 import argparse
 import subprocess
 import sys
+from collections.abc import Callable
 
 from path import Path
 from rich.console import Console
 
-from .sbatch import DONE_STATES, parse_sbatch, read_log, read_status
-from .utils import search_jobs
+from stepup.core.config import ConfigLoader
+
+from .log import read_jobid_cluster_status
+from .utils import DONE_STATES, search_jobs
 
 
 def canceljobs_tool(args: argparse.Namespace):
@@ -75,24 +78,8 @@ def canceljobs_tool(args: argparse.Namespace):
         sys.exit(1)
 
 
-def read_jobid_cluster_status(path_log: str) -> tuple[int, str | None, str | None]:
-    """Read the job ID, cluster, and job status from the job log file."""
-    lines = read_log(path_log, None)
-    if len(lines) < 1:
-        raise ValueError(f"Incomplete file: {path_log}.")
-    words = lines[0].split()
-    if len(words) != 3:
-        raise ValueError(f"Could not read job ID from first status line: {lines[0]}")
-    _, status, job_id_cluster = words
-    if status != "Submitted":
-        raise ValueError(f"No 'Submitted' on first status line: {lines[0]}")
-    job_id, cluster = parse_sbatch(job_id_cluster)
-    status = read_status(lines[-1:])[1]
-    return job_id, cluster, status
-
-
-def canceljobs_subcommand(subparser: argparse.ArgumentParser) -> callable:
-    parser = subparser.add_parser(
+def canceljobs_subcommand(subparsers, loader: ConfigLoader) -> Callable:
+    parser = subparsers.add_parser(
         "canceljobs",
         help="Cancel running jobs in the current StepUp workflow.",
     )
@@ -118,6 +105,7 @@ def canceljobs_subcommand(subparser: argparse.ArgumentParser) -> callable:
         default=False,
         help="Select all jobs, including the ones that seem to be done already.",
     )
+    loader.patch_parser(parser)
     return canceljobs_tool
 
 

stepup_queue-2.0.0rc1/stepup/queue/log.py

@@ -0,0 +1,121 @@
+# StepUp Queue integrates queued jobs into a StepUp workflow.
+# Copyright 2025-2026 Toon Verstraelen
+#
+# This file is part of StepUp Queue.
+#
+# StepUp Queue is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version 3
+# of the License, or (at your option) any later version.
+#
+# StepUp Queue is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, see <http://www.gnu.org/licenses/>
+#
+# --
+"""The job log file format and utilities to read and write it."""
+
+from datetime import datetime
+
+from path import Path
+
+from .utils import parse_sbatch
+
+__all__ = (
+    "FIRST_LINE",
+    "InpDigestError",
+    "init_log",
+    "log_status",
+    "read_jobid_cluster_status",
+    "read_log",
+    "read_status",
+)
+
+FIRST_LINE = "StepUp Queue sbatch wait log format version 2"
+
+
+class InpDigestError(ValueError):
+    """The input digest in the log file does not match the one in the environment."""
+
+
+def init_log(path_log: str, inp_digest: str):
+    """Initialize a new log file."""
+    with open(path_log, "w") as fh:
+        print(FIRST_LINE, file=fh)
+        print(inp_digest, file=fh)
+
+
+def log_status(path_log: Path, status: str):
+    """Write a status to the log."""
+    dt = datetime.now().isoformat()
+    with open(path_log, "a") as f:
+        line = f"{dt} {status}"
+        f.write(f"{line}\n")
+
+
+def read_jobid_cluster_status(path_log: str) -> tuple[int, str | None, str | None]:
+    """Read the job ID, cluster, and job status from the job log file."""
+    lines = read_log(path_log, None)
+    if len(lines) < 1:
+        raise ValueError(f"Incomplete file: {path_log}.")
+    words = lines[0].split()
+    if len(words) != 3:
+        raise ValueError(f"Could not read job ID from first status line: {lines[0]}")
+    _, status, job_id_cluster = words
+    if status != "Submitted":
+        raise ValueError(f"No 'Submitted' on first status line: {lines[0]}")
+    job_id, cluster = parse_sbatch(job_id_cluster)
+    status = read_status(lines[-1:])[1]
+    return job_id, cluster, status
+
+
+def read_log(path_log: str, expected_inp_digest: str | None = None) -> list[str]:
+    """Read lines from a previously created log file."""
+    lines = []
+    with open(path_log) as f:
+        try:
+            check_log_version(next(f).strip())
+        except StopIteration as exc:
+            raise ValueError("Existing log file is empty.") from exc
+        try:
+            actual_inp_digest = next(f).strip()
+        except StopIteration as exc:
+            raise ValueError("Existing log file has no input digest.") from exc
+        if expected_inp_digest is not None:
+            check_log_inp_digest(actual_inp_digest, expected_inp_digest)
+        for line in f:
+            line = line.strip()
+            lines.append(line)
+    return lines
+
+
+def check_log_version(line: str):
+    """Validate the log version, abort if there is a mismatch."""
+    if line != FIRST_LINE:
+        raise ValueError(
+            f"The first line of the log is wrong. Expected: '{FIRST_LINE}' Found: '{line}'"
+        )
+
+
+def check_log_inp_digest(actual: str, expected: str):
+    """Validate the log input digest, abort if there is a mismatch."""
+    if actual != expected:
+        raise InpDigestError(
+            "The second line of the log contains the wrong input digest.\n"
+            f"Actual:   {actual}\nExpected: {expected}\n"
+        )
+
+
+def read_status(lines: list[str]) -> tuple[float | None, str | None]:
+    """Read a status from the log file."""
+    if len(lines) == 0:
+        return None, None
+    line = lines.pop(0)
+    words = line.split(maxsplit=1)
+    if len(words) != 2:
+        raise ValueError(f"Expected a status in log but found line '{line}'.")
+    return datetime.fromisoformat(words[0]).timestamp(), words[1].strip()

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/stepup/queue/removejobs.py

@@ -21,11 +21,14 @@
 
 import argparse
 import shutil
+from collections.abc import Callable
 
 from path import Path
 from rich.console import Console
 
-from .sbatch import read_log, read_status
+from stepup.core.config import ConfigLoader
+
+from .log import read_log, read_status
 from .utils import search_jobs
 
 FAILED_STATES = {
@@ -74,8 +77,8 @@ def read_last_status(path_log: str) -> str | None:
     return read_status(lines[-1:])[1]
 
 
-def removejobs_subcommand(subparser: argparse.ArgumentParser) -> callable:
-    parser = subparser.add_parser(
+def removejobs_subcommand(subparsers, loader: ConfigLoader) -> Callable:
+    parser = subparsers.add_parser(
         "removejobs",
         help="Remove directories of failed (and optionally all completed) jobs "
         "in the current StepUp workflow.",
@@ -102,4 +105,5 @@ def removejobs_subcommand(subparser: argparse.ArgumentParser) -> callable:
         default=False,
         help="Remove all jobs, not only failed jobs.",
     )
+    loader.patch_parser(parser)
     return removejobs_tool

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/stepup/queue/sbatch.py

@@ -19,18 +19,31 @@
 # --
 """An sbatch wrapper to submit only on the first call, and to wait until a job has finished."""
 
+import argparse
 import fcntl
 import os
 import random
 import re
+import shlex
+import subprocess
+import sys
 import time
 from datetime import datetime
 
 from path import Path
 
-from stepup.core.worker import WorkThread
+from stepup.core.extapi import record_subprocess, run_subprocess
+
+from .log import (
+    InpDigestError,
+    init_log,
+    log_status,
+    read_jobid_cluster_status,
+    read_log,
+    read_status,
+)
+from .utils import DONE_STATES, KNOWN_JOB_STATES, parse_sbatch
 
-FIRST_LINE = "StepUp Queue sbatch wait log format version 2"
 SBATCH_RETRY_NUM = int(os.getenv("STEPUP_SBATCH_RETRY_NUM", "5"))
 SBATCH_RETRY_DELAY_MIN = int(os.getenv("STEPUP_SBATCH_RETRY_DELAY_MIN", "60"))
 SBATCH_RETRY_DELAY_MAX = int(os.getenv("STEPUP_SBATCH_RETRY_DELAY_MAX", "120"))
@@ -42,17 +55,14 @@ UNLISTED_TIMEOUT = int(os.getenv("STEPUP_SBATCH_UNLISTED_TIMEOUT", "600"))
 
 
 def submit_once_and_wait(
-    work_thread: WorkThread,
     job_ext: str,
     sbatch_rc: str | None = None,
     validate_inp_digest: bool = True,
-) -> int:
+):
     """Submit a job and wait for it to complete. When called a second time, just wait.
 
     Parameters
     ----------
-    work_thread
-        The work thread to use for launching the subprocesses.
     job_ext
         The file extension of the job script to be submitted.
     sbatch_rc
@@ -61,12 +71,6 @@ def submit_once_and_wait(
     validate_inp_digest
         If False, the input digest is not checked.
         This is useful when the job script is modified but the changes are harmless.
-
-    Returns
-    -------
-    returncode
-        The return code of the job.
-        0 if successful, 1 if the job failed.
     """
     inp_digest = os.getenv("STEPUP_STEP_INP_DIGEST")
     if inp_digest is None:
@@ -85,9 +89,9 @@ def submit_once_and_wait(
     if status is None:
         # A new job must be submitted.
         submit_time = time.time()
-        sbatch_stdout = submit_job(work_thread, job_ext, sbatch_rc)
+        sbatch_stdout = submit_job(job_ext, sbatch_rc)
         # Create a new log file after submitting the job.
-        _init_log(path_log, inp_digest)
+        init_log(path_log, inp_digest)
         log_status(path_log, f"Submitted {sbatch_stdout}")
         rndsleep()
     else:
@@ -107,134 +111,43 @@ def submit_once_and_wait(
     # Here, we take a random sleep time, by default between 30 and 60 seconds to play nice.
     status = "UNDEFINED"
     done = False
+    first = True
     while not done:
-        status, done = _read_or_poll_status(
-            work_thread, submit_time, jobid, cluster, previous_lines, path_log, status
+        status, done, called = _read_or_poll_status(
+            submit_time, jobid, cluster, previous_lines, path_log, status, first
         )
+        if called:
+            first = False
 
     if status == "COMPLETED":
         # Get the return code from the job
         with open("slurmjob.ret") as fh:
             returncode = fh.read().strip()
         try:
-            return int(returncode)
+            returncode = int(returncode)
         except ValueError as exc:
             raise ValueError(
                 f"Could not parse return code from slurmjob.ret. Got '{returncode}'"
             ) from exc
-    raise RuntimeError(f"Job ended with status '{status}'.")
-
-
-def read_log(path_log: str, expected_inp_digest: str | None = None) -> list[str]:
-    """Read lines from a previously created log file."""
-    lines = []
-    with open(path_log) as f:
-        try:
-            check_log_version(next(f).strip())
-        except StopIteration as exc:
-            raise ValueError("Existing log file is empty.") from exc
-        try:
-            actual_inp_digest = next(f).strip()
-        except StopIteration as exc:
-            raise ValueError("Existing log file has no input digest.") from exc
-        if expected_inp_digest is not None:
-            check_log_inp_digest(actual_inp_digest, expected_inp_digest)
-        for line in f:
-            line = line.strip()
-            lines.append(line)
-    return lines
-
-
-def check_log_version(line: str):
-    """Validate the log version, abort if there is a mismatch."""
-    if line != FIRST_LINE:
-        raise ValueError(
-            f"The first line of the log is wrong. Expected: '{FIRST_LINE}' Found: '{line}'"
-        )
-
-
-def _init_log(path_log: str, inp_digest: str):
-    """Initialize a new log file."""
-    with open(path_log, "w") as fh:
-        print(FIRST_LINE, file=fh)
-        print(inp_digest, file=fh)
-
-
-# From: https://slurm.schedmd.com/job_state_codes.html
-KNOWN_JOB_STATES = {
-    # -- Job states
-    # done
-    "BOOT_FAIL",
-    "CANCELLED",
-    "COMPLETED",
-    "DEADLINE",
-    "FAILED",
-    "NODE_FAIL",
-    "OUT_OF_MEMORY",
-    "PREEMPTED",
-    "TIMEOUT",
-    # waiting or running
-    "PENDING",
-    "RUNNING",
-    "SUSPENDED",
-    # -- Job flags
-    # done
-    "LAUNCH_FAILED",
-    "RECONFIG_FAIL",
-    "REVOKED",
-    "STOPPED",
-    # waiting or running
-    "COMPLETING",
-    "CONFIGURING",
-    "EXPEDITING",
-    "POWER_UP_NODE",
-    "REQUEUED",
-    "REQUEUE_FED",
-    "REQUEUE_HOLD",
-    "RESIZING",
-    "RESV_DEL_HOLD",
-    "SIGNALING",
-    "SPECIAL_EXIT",
-    "STAGE_OUT",
-    "UPDATE_DB",
-    # -- Specific to this script
-    # to be ignored (same as waiting or running), must not be logged
-    "invalid",
-    "unlisted",
-}
-
-DONE_STATES = {
-    "BOOT_FAIL",
-    "CANCELLED",
-    "COMPLETED",
-    "DEADLINE",
-    "FAILED",
-    "NODE_FAIL",
-    "OUT_OF_MEMORY",
-    "PREEMPTED",
-    "TIMEOUT",
-    "LAUNCH_FAILED",
-    "RECONFIG_FAIL",
-    "REVOKED",
-    "STOPPED",
-}
+        if returncode != 0:
+            raise RuntimeError(f"Job ended with return code {returncode}.")
+    else:
+        raise RuntimeError(f"Job ended with status '{status}'.")
 
 
 def _read_or_poll_status(
-    work_thread: WorkThread,
     submit_time: float,
     jobid: int,
     cluster: str,
     previous_lines: list[str],
     path_log: str,
     last_status: str,
-) -> tuple[str, bool]:
+    first: bool,
+) -> tuple[str, bool, bool]:
     """One polling iteration. Before polling, previous lines from the log are parsed.
 
     Parameters
     ----------
-    work_thread
-        The work thread to use for launching the sacct command.
     submit_time
         The timestamp when the job was submitted.
     jobid
@@ -248,6 +161,8 @@ def _read_or_poll_status(
     last_status
         The status from the previous iteration.
         If the status does not change, nothing is added to the log file.
+    first
+        True if this is the first call to _read_or_poll_status in this process.
 
     Returns
     -------
@@ -255,14 +170,17 @@ def _read_or_poll_status(
         The status result obtained by polling the scheduler.
     done
         True when the waiting is over.
+    called
+        True if the scheduler was polled, False if the status was obtained from the log.
     """
     # First try to replay previously logged states
+    called = False
     _, status = read_status(previous_lines)
     if status is None:
         # All previously logged states are processed.
         # Call sacct and parse its response.
         rndsleep()
-        _, status = get_status(work_thread, jobid, cluster)
+        _, status, called = get_status(jobid, cluster, first)
         # Log only if the status changed, and is not invalid or unlisted.
         # These two statuses are (potentially) transient and should not be logged.
         if status != last_status and status not in ["invalid", "unlisted"]:
@@ -277,31 +195,7 @@ def _read_or_poll_status(
         # This prevents an infinite loop if the job ID was wrong or purged.
         done = True
 
-    return status, done
-
-
-class InpDigestError(ValueError):
-    """The input digest in the log file does not match the one in the environment."""
-
-
-def check_log_inp_digest(actual: str, expected: str):
-    """Validate the log input digest, abort if there is a mismatch."""
-    if actual != expected:
-        raise InpDigestError(
-            "The second line of the log contains the wrong input digest.\n"
-            f"Actual:   {actual}\nExpected: {expected}\n"
-        )
-
-
-def read_status(lines: list[str]) -> tuple[float | None, str | None]:
-    """Read a status from the log file."""
-    if len(lines) == 0:
-        return None, None
-    line = lines.pop(0)
-    words = line.split(maxsplit=1)
-    if len(words) != 2:
-        raise ValueError(f"Expected a status in log but found line '{line}'.")
-    return datetime.fromisoformat(words[0]).timestamp(), words[1].strip()
+    return status, done, called
 
 
 def rndsleep():
@@ -333,7 +227,7 @@ UNSUPPORTED_DIRECTIVES = [
 ]
 
 
-def submit_job(work_thread: WorkThread, job_ext: str, sbatch_rc: str | None = None) -> str:
+def submit_job(job_ext: str, sbatch_rc: str | None = None) -> str:
     """Submit a job with sbatch."""
     # Verify that the job script is executable.
     path_job = f"slurmjob{job_ext}"
@@ -364,50 +258,37 @@ def submit_job(work_thread: WorkThread, job_ext: str, sbatch_rc: str | None = No
         sbatch_header = "\n".join(sbatch_header)
 
     command = "sbatch --parsable -o slurmjob.out -e slurmjob.err"
+    shell = False
     if sbatch_rc is not None:
         command = f"{sbatch_rc} < /dev/null && {command}"
+        shell = True
     stdin = JOB_SCRIPT_WRAPPER.format(sbatch_header=sbatch_header, job_script=path_job)
     for _ in range(SBATCH_RETRY_NUM):
-        returncode, stdout, stderr = work_thread.runsh(command, stdin=stdin)
-        if returncode == 0:
-            return stdout.strip()
-        if not (stderr is None or stderr == ""):
-            print(stderr)
+        cp = run_subprocess(command, stdin=stdin, check=False, shell=shell)
+        if cp.returncode == 0:
+            return cp.stdout.strip()
+        if not (cp.stderr is None or cp.stderr == ""):
+            sys.stderr.write(cp.stderr)
         delay = random.randint(SBATCH_RETRY_DELAY_MIN, SBATCH_RETRY_DELAY_MAX)
-        print(f"sbatch failed with return code {returncode}. Retrying in {delay} seconds.")
+        print(
+            f"sbatch failed with return code {cp.returncode}. Retrying in {delay} seconds.",
+            file=sys.stderr,
+        )
         time.sleep(delay)
     raise RuntimeError(f"sbatch failed {SBATCH_RETRY_NUM} times. Giving up.")
 
 
-def log_status(path_log: Path, status: str):
-    """Write a status to the log."""
-    dt = datetime.now().isoformat()
-    with open(path_log, "a") as f:
-        line = f"{dt} {status}"
-        f.write(f"{line}\n")
-
-
-def parse_sbatch(stdout: str) -> tuple[int, str | None]:
-    """Parse the 'parsable' output of sbatch."""
-    words = stdout.split(";")
-    if len(words) == 1:
-        return int(words[0]), None
-    if len(words) == 2:
-        return int(words[0]), words[1]
-    raise ValueError(f"Cannot parse sbatch output: {stdout}")
-
-
-def get_status(work_thread: WorkThread, jobid: int, cluster: str | None) -> tuple[float, str]:
+def get_status(jobid: int, cluster: str | None, first: bool) -> tuple[float, str, bool]:
     """Load cached sacct output or run sacct if outdated.
 
     Parameters
     ----------
-    work_thread
-        The work thread to use for launching the sacct command.
     jobid
         The job to wait for.
     cluster
         The cluster to which the job was submitted.
+    first
+        True if this is the first call to get_status in this process.
 
     Returns
     -------
@@ -417,6 +298,8 @@ def get_status(work_thread: WorkThread, jobid: int, cluster: str | None) -> tupl
         A status reported by sacct,
         or `invalid` if sacct failed (retry sacct later),
         or `unlisted` if the job is not found (probably ended long ago).
+    called
+        True if sacct was called, False if the status was obtained from the cache.
     """
     # Load cached output or run again
     command = f"sacct -o 'jobid,state' -PXn -S {SACCT_START}"
@@ -426,27 +309,27 @@ def get_status(work_thread: WorkThread, jobid: int, cluster: str | None) -> tupl
     else:
         command += f" --cluster={cluster}"
         path_out /= f"sbatch_wait_sacct.{cluster}.out"
-    status_time, sacct_out, returncode = cached_run(work_thread, command, path_out, CACHE_TIMEOUT)
+    status_time, sacct_out, returncode, called = cached_run(command, path_out, CACHE_TIMEOUT, first)
     if returncode != 0:
-        return status_time, "invalid"
-    return status_time, parse_sacct_out(sacct_out, jobid)
+        return status_time, "invalid", called
+    return status_time, parse_sacct_out(sacct_out, jobid), called
 
 
 def cached_run(
-    work_thread: WorkThread, command: str, path_out: Path, cache_timeout
-) -> tuple[float, str, int]:
+    command: str, path_out: Path, cache_timeout: float, first: bool
+) -> tuple[float, str, int, bool]:
     """Execute a command if its previous output is outdated.
 
     Parameters
     ----------
-    work_thread
-        The work thread to use for launching the command.
     command
         Command to run if the cached output is outdated.
     path_out
         The path where the output is cached.
     cache_timeout
         The waiting time between two actual calls.
+    first
+        True if this is the first call to cached_run in this process.
 
     Returns
     -------
@@ -456,6 +339,8 @@ def cached_run(
         The output of the file, either new or cached.
     returncode
         The return code of the (cached) command.
+    called
+        True if the command was executed, False if the output was read from the cache.
 
     Notes
     -----
@@ -472,19 +357,26 @@ def cached_run(
         header = fh.read(CACHE_HEADER_LENGTH)
         cache_time, returncode = parse_cache_header(header)
         if cache_time is None or time.time() > cache_time + cache_timeout:
-            returncode, stdout, _ = work_thread.runsh(command)
+            cp = subprocess.run(shlex.split(command), capture_output=True, text=True, check=False)
+            if first:
+                # Only the first call is recorded to avoid duplicate entries in StepUp's metadata.
+                # Note that the recording of subprocesses is intended to be informative,
+                # not authoritative.
+                record_subprocess(
+                    f"{command}  # first call only", cp.returncode, workdir=os.getcwd()
+                )
             # Go the the beginning of the file before truncating.
             # (Possibly related to issue with zero bytes at start of file.)
             fh.seek(0)
             fh.truncate(0)
             cache_time = time.time()
-            header = make_cache_header(cache_time, returncode)
+            header = make_cache_header(cache_time, cp.returncode)
             fh.write(header)
-            fh.write(stdout)
+            fh.write(cp.stdout)
             fh.flush()
             os.fsync(fh.fileno())
-            return cache_time, stdout, returncode
-        return cache_time, fh.read(), returncode
+            return cache_time, cp.stdout, cp.returncode, True
+        return cache_time, fh.read(), returncode, False
 
 
 def make_cache_header(cache_time: float, returncode: int):
@@ -542,3 +434,35 @@ def parse_sacct_out(sacct_out: str, jobid: int) -> str:
     except (ValueError, IndexError):
         return "invalid"
     return "unlisted"
+
+
+def sbatch():
+    """Submit a job and wait for it to complete. When called a second time, just wait."""
+    parser = argparse.ArgumentParser()
+    parser.add_argument("ext", nargs="?", default=".sh")
+    parser.add_argument("--rc", default=None)
+    default_onchange = os.getenv("STEPUP_QUEUE_ONCHANGE", "raise")
+    parser.add_argument(
+        "--onchange", default=default_onchange, choices=["raise", "resubmit", "ignore"]
+    )
+    args = parser.parse_args()
+
+    if args.onchange == "resubmit":
+        try:
+            submit_once_and_wait(args.ext, args.rc)
+            return
+        except InpDigestError:
+            pass
+        # Cancel running job (if any), clean log and resubmit
+        path_log = Path("slurmjob.log")
+        job_id, cluster, _ = read_jobid_cluster_status(path_log)
+        if cluster is None:
+            run_subprocess(f"scancel {job_id}")
+        else:
+            run_subprocess(f"scancel -M {cluster} {job_id}")
+        path_log.remove_p()
+    submit_once_and_wait(args.ext, args.rc, args.onchange != "ignore")
+
+
+if __name__ == "__main__":
+    sbatch()

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/stepup/queue/utils.py

@@ -24,7 +24,72 @@ from itertools import chain
 from path import Path
 from rich.console import Console
 
-__all__ = ("search_jobs",)
+__all__ = (
+    "DONE_STATES",
+    "KNOWN_JOB_STATES",
+    "parse_sbatch",
+    "search_jobs",
+)
+
+
+# From: https://slurm.schedmd.com/job_state_codes.html
+KNOWN_JOB_STATES = {
+    # -- Job states
+    # done
+    "BOOT_FAIL",
+    "CANCELLED",
+    "COMPLETED",
+    "DEADLINE",
+    "FAILED",
+    "NODE_FAIL",
+    "OUT_OF_MEMORY",
+    "PREEMPTED",
+    "TIMEOUT",
+    # waiting or running
+    "PENDING",
+    "RUNNING",
+    "SUSPENDED",
+    # -- Job flags
+    # done
+    "LAUNCH_FAILED",
+    "RECONFIG_FAIL",
+    "REVOKED",
+    "STOPPED",
+    # waiting or running
+    "COMPLETING",
+    "CONFIGURING",
+    "EXPEDITING",
+    "POWER_UP_NODE",
+    "REQUEUED",
+    "REQUEUE_FED",
+    "REQUEUE_HOLD",
+    "RESIZING",
+    "RESV_DEL_HOLD",
+    "SIGNALING",
+    "SPECIAL_EXIT",
+    "STAGE_OUT",
+    "UPDATE_DB",
+    # -- Specific to this script
+    # to be ignored (same as waiting or running), must not be logged
+    "invalid",
+    "unlisted",
+}
+
+DONE_STATES = {
+    "BOOT_FAIL",
+    "CANCELLED",
+    "COMPLETED",
+    "DEADLINE",
+    "FAILED",
+    "NODE_FAIL",
+    "OUT_OF_MEMORY",
+    "PREEMPTED",
+    "TIMEOUT",
+    "LAUNCH_FAILED",
+    "RECONFIG_FAIL",
+    "REVOKED",
+    "STOPPED",
+}
 
 
 def search_jobs(paths: list[Path], console: Console | None = None) -> list[Path]:
@@ -57,3 +122,13 @@ def search_jobs(paths: list[Path], console: Console | None = None) -> list[Path]
             if path_log.is_file():
                 paths_log.add(path_log)
     return sorted(paths_log)
+
+
+def parse_sbatch(stdout: str) -> tuple[int, str | None]:
+    """Parse the 'parsable' output of sbatch."""
+    words = stdout.split(";")
+    if len(words) == 1:
+        return int(words[0]), None
+    if len(words) == 2:
+        return int(words[0]), words[1]
+    raise ValueError(f"Cannot parse sbatch output: {stdout}")

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/stepup_queue.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: stepup-queue
-Version: 1.1.1
+Version: 2.0.0rc1
 Summary: StepUp Queue integrates queued jobs into a StepUp workflow.
 Author-email: Toon Verstraelen <toon.verstraelen@ugent.be>
 License-Expression: GPL-3.0-or-later
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: path>=16.14.0
 Requires-Dist: rich>=13.0.0
-Requires-Dist: stepup<4.0.0,>=3.2.0
+Requires-Dist: stepup<5.0.0a1,>=4.0.0rc3
 Provides-Extra: dev
 Requires-Dist: psutil; extra == "dev"
 Requires-Dist: pytest; extra == "dev"

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/stepup_queue.egg-info/SOURCES.txt

@@ -1,11 +1,12 @@
+CLAUDE.md
 LICENSE
 MANIFEST.in
 README.md
 pyproject.toml
 stepup/queue/__init__.py
-stepup/queue/actions.py
 stepup/queue/api.py
 stepup/queue/canceljobs.py
+stepup/queue/log.py
 stepup/queue/removejobs.py
 stepup/queue/sbatch.py
 stepup/queue/utils.py
@@ -14,4 +15,6 @@ stepup_queue.egg-info/SOURCES.txt
 stepup_queue.egg-info/dependency_links.txt
 stepup_queue.egg-info/entry_points.txt
 stepup_queue.egg-info/requires.txt
+stepup_queue.egg-info/scm_file_list.json
+stepup_queue.egg-info/scm_version.json
 stepup_queue.egg-info/top_level.txt

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/stepup_queue.egg-info/entry_points.txt

@@ -1,5 +1,5 @@
-[stepup.actions]
-sbatch = stepup.queue.actions:sbatch
+[console_scripts]
+sq-sbatch-and-wait = stepup.queue.sbatch:sbatch
 
 [stepup.tools]
 canceljobs = stepup.queue.canceljobs:canceljobs_subcommand

{stepup_queue-1.1.1 → stepup_queue-2.0.0rc1}/stepup_queue.egg-info/requires.txt

@@ -1,6 +1,6 @@
 path>=16.14.0
 rich>=13.0.0
-stepup<4.0.0,>=3.2.0
+stepup<5.0.0a1,>=4.0.0rc3
 
 [dev]
 psutil

stepup_queue-2.0.0rc1/stepup_queue.egg-info/scm_file_list.json

@@ -0,0 +1,51 @@
+{
+  "files": [
+    ".pre-commit-config.yaml",
+    "README.md",
+    "LICENSE",
+    "pyproject.toml",
+    "mkdocs.yaml",
+    "CLAUDE.md",
+    "MANIFEST.in",
+    ".editorconfig",
+    ".gitignore",
+    ".markdownlint-cli2.jsonc",
+    "docs/development.md",
+    "docs/installation.md",
+    "docs/license.md",
+    "docs/changelog.md",
+    "docs/usage.md",
+    "docs/logo.svg",
+    "docs/stepup.queue.api.md",
+    "docs/index.md",
+    "docs/examples/slurm-perpetual/README.md",
+    "docs/examples/slurm-perpetual/workflow.sh",
+    "docs/examples/slurm-perpetual/plan.py",
+    "docs/examples/slurm-perpetual/.gitignore",
+    "docs/examples/slurm-perpetual/step1/slurmjob.sh",
+    "docs/examples/slurm-perpetual/step2/slurmjob.sh",
+    "docs/examples/slurm-basic/README.md",
+    "docs/examples/slurm-basic/dynamic-template.sh",
+    "docs/examples/slurm-basic/plan.py",
+    "docs/examples/slurm-basic/.gitignore",
+    "docs/examples/slurm-basic/pass/slurmjob.py",
+    "docs/examples/slurm-basic/fail/slurmjob.sh",
+    "overrides/main.html",
+    "stepup/queue/__init__.py",
+    "stepup/queue/sbatch.py",
+    "stepup/queue/utils.py",
+    "stepup/queue/canceljobs.py",
+    "stepup/queue/api.py",
+    "stepup/queue/log.py",
+    "stepup/queue/removejobs.py",
+    "tests/test_utils.py",
+    "tests/test_log.py",
+    "tests/test_sbatch.py",
+    "tests/conftest.py",
+    ".github/requirements-old.txt",
+    ".github/scripts/extract-notes.sh",
+    ".github/workflows/pytest.yaml",
+    ".github/workflows/release.yaml",
+    ".github/workflows/mkdocs.yaml"
+  ]
+}

stepup_queue-2.0.0rc1/stepup_queue.egg-info/scm_version.json

@@ -0,0 +1,8 @@
+{
+  "tag": "2.0.0rc1",
+  "distance": 0,
+  "node": "g45b2ffc75f29698bb6f50b69e85b2ea4233a3f27",
+  "dirty": false,
+  "branch": "HEAD",
+  "node_date": "2026-06-28"
+}

stepup_queue-1.1.1/stepup/queue/actions.py

@@ -1,57 +0,0 @@
-# StepUp Queue integrates queued jobs into a StepUp workflow.
-# Copyright 2025-2026 Toon Verstraelen
-#
-# This file is part of StepUp Queue.
-#
-# StepUp Queue is free software; you can redistribute it and/or
-# modify it under the terms of the GNU General Public License
-# as published by the Free Software Foundation; either version 3
-# of the License, or (at your option) any later version.
-#
-# StepUp Queue is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, see <http://www.gnu.org/licenses/>
-#
-# --
-"""StepUp Queue package."""
-
-import argparse
-import contextlib
-import os
-import shlex
-
-from path import Path
-
-from stepup.core.worker import WorkThread
-
-from .canceljobs import read_jobid_cluster_status
-from .sbatch import InpDigestError, submit_once_and_wait
-
-
-def sbatch(argstr: str, work_thread: WorkThread) -> int:
-    # Use argparse to parse the argstr
-    parser = argparse.ArgumentParser()
-    parser.add_argument("ext", nargs="?", default=".sh")
-    parser.add_argument("--rc", default=None)
-    default_onchange = os.getenv("STEPUP_QUEUE_ONCHANGE", "raise")
-    parser.add_argument(
-        "--onchange", default=default_onchange, choices=["raise", "resubmit", "ignore"]
-    )
-    args = parser.parse_args(shlex.split(argstr))
-
-    if args.onchange == "resubmit":
-        with contextlib.suppress(InpDigestError):
-            return submit_once_and_wait(work_thread, args.ext, args.rc)
-        # Cancel running job (if any), clean log and resubmit
-        path_log = Path("slurmjob.log")
-        job_id, cluster, _ = read_jobid_cluster_status(path_log)
-        if cluster is None:
-            work_thread.runsh(f"scancel {job_id}")
-        else:
-            work_thread.runsh(f"scancel -M {cluster} {job_id}")
-        path_log.remove_p()
-    return submit_once_and_wait(work_thread, args.ext, args.rc, args.onchange != "ignore")