py-cluster-api 0.7.0__tar.gz → 0.8.1__tar.gz

{py_cluster_api-0.7.0 → py_cluster_api-0.8.1}/CLAUDE.md

@@ -2,7 +2,7 @@
 
 Generic Python library for submitting and monitoring jobs on HPC clusters. Wraps scheduler CLIs (bsub/bjobs/bkill) behind an async executor abstraction with an active polling monitor that fires callbacks on job completion. Inspired by dask-jobqueue's script templating and Nextflow's portable config profiles, but unlike dask-jobqueue, this library actively polls the scheduler rather than relying on workers phoning home.
 
-Key capabilities beyond submit/poll/cancel: `reconnect()` rediscovers running jobs after a process restart (requires `job_name_prefix`), and `cancel_by_name()` kills jobs by name pattern (LSF only).
+Key capabilities beyond submit/poll/cancel: `reconnect()` rediscovers running jobs after a process restart (requires `job_name_prefix`), and `cancel_by_name()` kills jobs by name pattern (LSF only). `depends_on` on `submit()`/`submit_array()` chains jobs (native `bsub -w ... -ti` on LSF; poll-driven deferred start on Local) with auto-cancel of dependents when an upstream job fails.
 
 Founding principles: async-only API, executors are thin wrappers around scheduler CLIs, all state lives in `JobRecord` dataclasses tracked in-process, monitoring is poll-based via `bjobs -json`, and configuration uses Nextflow-style YAML profiles.
 

{py_cluster_api-0.7.0 → py_cluster_api-0.8.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: py-cluster-api
-Version: 0.7.0
+Version: 0.8.1
 Summary: Generic Python library for running jobs on HPC clusters
 Project-URL: Homepage, https://github.com/JaneliaSciComp/py-cluster-api
 Project-URL: Repository, https://github.com/JaneliaSciComp/py-cluster-api
@@ -71,6 +71,9 @@ A Python library for submitting and monitoring jobs on HPC clusters. Supports ru
 - **Zombie detection** — jobs that disappear from the scheduler are marked as failed
 - **YAML config with profiles** — Nextflow-style config with per-environment profiles
 - **Callback chaining** — register `on_success`, `on_failure`, or `on_exit` handlers on any job
+- **Job dependencies** — chain jobs with `depends_on`; dependents are auto-cancelled if an upstream job fails
+- **Environment control** — `inherit_env` and `login_shell` submit options control what environment a job runs with
+- **Reconnect & external tracking** — `reconnect()` rediscovers jobs after a restart; `track()` seeds a job from an external store (e.g. a database) into the executor
 
 ## Installation
 
@@ -136,6 +139,24 @@ async def run_array():
 
 The array index environment variable depends on the executor: LSF uses `$LSB_JOBINDEX`, while the local executor uses `$ARRAY_INDEX`.
 
+### Job Dependencies
+
+Chain jobs so each stage starts only after the previous one succeeds:
+
+```python
+convert = await executor.submit("convert.sh input.tif", name="convert-s0")
+pyramid = await executor.submit(
+    "build_pyramid.sh", name="pyramid",
+    depends_on=[convert],   # JobRecords or raw job-id strings
+)
+```
+
+`depends_on` accepts multiple jobs (fan-in). On LSF this uses native
+`bsub -w "done(...)" -ti`, so chains keep running if your process exits,
+and LSF terminates dependents whose upstream failed — no jobs stuck
+pending forever. When a dependent is cancelled this way,
+`record.metadata["dependency_failed"]` lists the failed upstream job ids.
+
 ### Reconnecting After Restart
 
 If your process crashes or restarts, `reconnect()` rediscovers running jobs from the scheduler and resumes tracking them. Requires `job_name_prefix` to be set in config.
@@ -156,6 +177,28 @@ async def resume():
     await monitor.stop()
 ```
 
+### Environment Control
+
+By default a submitted job inherits the submitting process's environment plus whatever `env` you pass. Set `inherit_env=False` to start from a minimal environment (only `env` and scheduler-provided variables), or `login_shell=True` to run under a login shell so the target user's own profile builds `PATH`, modules, conda, etc.:
+
+```python
+job = await executor.submit(
+    command="my_tool.sh",
+    name="my-job",
+    inherit_env=False,
+    login_shell=True,
+    env={"MY_VAR": "1"},
+)
+```
+
+### Tracking Jobs From External State
+
+`track()` seeds the executor with a job ID from a persistent store (e.g. a database) without re-submitting it, so a later `poll()` or `cancel()` can act on jobs submitted by a previous process:
+
+```python
+job = executor.track(job_id="12345", status=JobStatus.RUNNING)
+```
+
 ### Local Testing
 
 ```python

{py_cluster_api-0.7.0 → py_cluster_api-0.8.1}/README.md

@@ -19,6 +19,9 @@ A Python library for submitting and monitoring jobs on HPC clusters. Supports ru
 - **Zombie detection** — jobs that disappear from the scheduler are marked as failed
 - **YAML config with profiles** — Nextflow-style config with per-environment profiles
 - **Callback chaining** — register `on_success`, `on_failure`, or `on_exit` handlers on any job
+- **Job dependencies** — chain jobs with `depends_on`; dependents are auto-cancelled if an upstream job fails
+- **Environment control** — `inherit_env` and `login_shell` submit options control what environment a job runs with
+- **Reconnect & external tracking** — `reconnect()` rediscovers jobs after a restart; `track()` seeds a job from an external store (e.g. a database) into the executor
 
 ## Installation
 
@@ -84,6 +87,24 @@ async def run_array():
 
 The array index environment variable depends on the executor: LSF uses `$LSB_JOBINDEX`, while the local executor uses `$ARRAY_INDEX`.
 
+### Job Dependencies
+
+Chain jobs so each stage starts only after the previous one succeeds:
+
+```python
+convert = await executor.submit("convert.sh input.tif", name="convert-s0")
+pyramid = await executor.submit(
+    "build_pyramid.sh", name="pyramid",
+    depends_on=[convert],   # JobRecords or raw job-id strings
+)
+```
+
+`depends_on` accepts multiple jobs (fan-in). On LSF this uses native
+`bsub -w "done(...)" -ti`, so chains keep running if your process exits,
+and LSF terminates dependents whose upstream failed — no jobs stuck
+pending forever. When a dependent is cancelled this way,
+`record.metadata["dependency_failed"]` lists the failed upstream job ids.
+
 ### Reconnecting After Restart
 
 If your process crashes or restarts, `reconnect()` rediscovers running jobs from the scheduler and resumes tracking them. Requires `job_name_prefix` to be set in config.
@@ -104,6 +125,28 @@ async def resume():
     await monitor.stop()
 ```
 
+### Environment Control
+
+By default a submitted job inherits the submitting process's environment plus whatever `env` you pass. Set `inherit_env=False` to start from a minimal environment (only `env` and scheduler-provided variables), or `login_shell=True` to run under a login shell so the target user's own profile builds `PATH`, modules, conda, etc.:
+
+```python
+job = await executor.submit(
+    command="my_tool.sh",
+    name="my-job",
+    inherit_env=False,
+    login_shell=True,
+    env={"MY_VAR": "1"},
+)
+```
+
+### Tracking Jobs From External State
+
+`track()` seeds the executor with a job ID from a persistent store (e.g. a database) without re-submitting it, so a later `poll()` or `cancel()` can act on jobs submitted by a previous process:
+
+```python
+job = executor.track(job_id="12345", status=JobStatus.RUNNING)
+```
+
 ### Local Testing
 
 ```python

{py_cluster_api-0.7.0 → py_cluster_api-0.8.1}/cluster_api/_types.py

@@ -98,6 +98,7 @@ class JobRecord:
     start_time: datetime | None = None
     finish_time: datetime | None = None
     array_elements: dict[int, ArrayElement] = field(default_factory=dict)
+    depends_on: list[str] = field(default_factory=list)
     _callbacks: list[tuple[JobExitCondition, Callable]] = field(
         default_factory=list, repr=False
     )

{py_cluster_api-0.7.0 → py_cluster_api-0.8.1}/cluster_api/core.py

@@ -7,6 +7,7 @@ import asyncio
 import logging
 import os
 import re
+from collections.abc import Sequence
 from datetime import datetime, timezone
 from typing import Any
 
@@ -73,6 +74,15 @@ class Executor(abc.ABC):
 
     # --- Submission ---
 
+    @staticmethod
+    def _resolve_dependency_ids(
+        depends_on: Sequence[JobRecord | str] | None,
+    ) -> list[str]:
+        """Resolve JobRecords or raw id strings to a list of job ids."""
+        if not depends_on:
+            return []
+        return [d.job_id if isinstance(d, JobRecord) else str(d) for d in depends_on]
+
     async def submit(
         self,
         command: str,
@@ -84,6 +94,7 @@ class Executor(abc.ABC):
         metadata: dict[str, Any] | None = None,
         inherit_env: bool = True,
         login_shell: bool = False,
+        depends_on: Sequence[JobRecord | str] | None = None,
     ) -> JobRecord:
         """Submit a job to the scheduler.
 
@@ -95,14 +106,20 @@ class Executor(abc.ABC):
             login_shell: When True, the job runs under a login shell so the
                 target user's own profile builds the environment (PATH,
                 modules, conda), as if they had SSH'd to the node.
+            depends_on: Jobs that must finish successfully before this job
+                starts. Accepts JobRecords or raw job-id strings. If any
+                dependency fails or is killed, this job is cancelled instead
+                of pending forever.
         """
         resources = resources or ResourceSpec()
         full_name = _sanitize_job_name(f"{self._prefix}-{name}" if self._prefix else name)
+        dep_ids = self._resolve_dependency_ids(depends_on)
 
         job_id, script_path = await self._submit_job(
             command, full_name, resources, prologue, epilogue, env,
             cwd=resources.work_dir,
             inherit_env=inherit_env, login_shell=login_shell,
+            depends_on=dep_ids or None,
         )
 
         record = JobRecord(
@@ -113,6 +130,7 @@ class Executor(abc.ABC):
             resources=resources,
             script_path=script_path,
             metadata=metadata or {},
+            depends_on=dep_ids,
             _last_seen=datetime.now(timezone.utc),
         )
         self._jobs[job_id] = record
@@ -132,15 +150,18 @@ class Executor(abc.ABC):
         max_concurrent: int | None = None,
         inherit_env: bool = True,
         login_shell: bool = False,
+        depends_on: Sequence[JobRecord | str] | None = None,
     ) -> JobRecord:
         """Submit a job array to the scheduler."""
         resources = resources or ResourceSpec()
         full_name = _sanitize_job_name(f"{self._prefix}-{name}" if self._prefix else name)
+        dep_ids = self._resolve_dependency_ids(depends_on)
 
         job_id, script_path = await self._submit_array_job(
             command, full_name, array_range, resources, prologue, epilogue,
             env, max_concurrent, cwd=resources.work_dir,
             inherit_env=inherit_env, login_shell=login_shell,
+            depends_on=dep_ids or None,
         )
 
         meta = {**(metadata or {}), "array_range": array_range}
@@ -155,6 +176,7 @@ class Executor(abc.ABC):
             resources=resources,
             script_path=script_path,
             metadata=meta,
+            depends_on=dep_ids,
             _last_seen=datetime.now(timezone.utc),
         )
         self._jobs[job_id] = record
@@ -177,6 +199,7 @@ class Executor(abc.ABC):
         cwd: str | None = None,
         inherit_env: bool = True,
         login_shell: bool = False,
+        depends_on: list[str] | None = None,
     ) -> tuple[str, str | None]:
         """Submit a single job.
 
@@ -199,11 +222,13 @@ class Executor(abc.ABC):
         cwd: str | None = None,
         inherit_env: bool = True,
         login_shell: bool = False,
+        depends_on: list[str] | None = None,
     ) -> tuple[str, str | None]:
         """Submit an array job. Override in subclasses."""
         return await self._submit_job(
             command, name, resources, prologue, epilogue, env, cwd=cwd,
             inherit_env=inherit_env, login_shell=login_shell,
+            depends_on=depends_on,
         )
 
     def _job_id_from_submit_output(self, out: str) -> str:
@@ -336,6 +361,22 @@ class Executor(abc.ABC):
             record = self._jobs[parent_id]
             record.status = record.compute_array_status()
 
+        # Annotate dependency-driven failures so callbacks can distinguish
+        # "upstream broke" from "my job broke".
+        for record in self._jobs.values():
+            if (
+                record.depends_on
+                and record.status in (JobStatus.FAILED, JobStatus.KILLED)
+                and "dependency_failed" not in record.metadata
+            ):
+                culprits = [
+                    d for d in record.depends_on
+                    if (dep := self._jobs.get(d)) is not None
+                    and dep.status in (JobStatus.FAILED, JobStatus.KILLED)
+                ]
+                if culprits:
+                    record.metadata["dependency_failed"] = culprits
+
         return {jid: r.status for jid, r in self._jobs.items()}
 
     # --- Subprocess helper ---

{py_cluster_api-0.7.0 → py_cluster_api-0.8.1}/cluster_api/executors/local.py

@@ -6,6 +6,8 @@ import asyncio
 import itertools
 import logging
 import os
+import signal
+import time
 from datetime import datetime, timezone
 from pathlib import Path
 from typing import Any
@@ -13,6 +15,7 @@ from typing import Any
 from .._types import ArrayElement, JobStatus, ResourceSpec
 from ..config import ClusterConfig
 from ..core import Executor, _ARRAY_ELEMENT_RE
+from ..exceptions import ClusterAPIError
 from ..script import render_script, write_script
 
 logger = logging.getLogger(__name__)
@@ -48,6 +51,7 @@ class LocalExecutor(Executor):
         self._open_fds: dict[str, tuple[int, int]] = {}
         self._next_id = 1
         self._script_counter = itertools.count(1)
+        self._waiting: dict[str, dict] = {}
 
     def build_header(
         self, name: str, resources: ResourceSpec | None = None
@@ -55,6 +59,14 @@ class LocalExecutor(Executor):
         """Local executor doesn't need scheduler directives."""
         return [f"# LOCAL Job: {name}"]
 
+    @staticmethod
+    def _job_env(env: dict[str, str] | None, inherit_env: bool) -> dict[str, str]:
+        return {**os.environ, **(env or {})} if inherit_env else _baseline_env(env)
+
+    @staticmethod
+    def _bash_cmd(script_path: str, login_shell: bool) -> list[str]:
+        return ["bash", "-l", script_path] if login_shell else ["bash", script_path]
+
     async def _submit_job(
         self,
         command: str,
@@ -67,23 +79,42 @@ class LocalExecutor(Executor):
         cwd: str | None = None,
         inherit_env: bool = True,
         login_shell: bool = False,
+        depends_on: list[str] | None = None,
     ) -> tuple[str, str | None]:
-        """Render script, write to disk, run as a background subprocess."""
+        """Render script, write to disk, run as a background subprocess.
+
+        The job_id is the subprocess PID, which doubles as its process-group
+        id (the child is spawned with ``start_new_session=True``). This makes
+        the id a durable OS handle: a fresh executor in a later process can
+        cancel the job's whole tree by PID alone, without the in-memory
+        ``_processes`` entry (see :meth:`cancel`).
+        """
         header = self.build_header(name, resources)
         script = render_script(self.config, command, header, prologue, epilogue)
-        script_path = write_script(resources.work_dir, script, name, next(self._script_counter))
-
-        job_id = str(self._next_id)
-        self._next_id += 1
-
-        if inherit_env:
-            full_env = {**os.environ, **(env or {})}
-        else:
-            full_env = _baseline_env(env)
-        bash_cmd = ["bash", "-l", script_path] if login_shell else ["bash", script_path]
-
-        # Write stdout/stderr directly to log files for real-time access
-        stdout_dest, stderr_dest = self._open_output_files(resources, job_id=job_id)
+        token = next(self._script_counter)
+        script_path = write_script(resources.work_dir, script, name, token)
+
+        if depends_on:
+            job_id = f"waiting-{self._next_id}"
+            self._next_id += 1
+            self._waiting[job_id] = {
+                "script_path": script_path,
+                "resources": resources,
+                "env": env,
+                "cwd": cwd,
+                "inherit_env": inherit_env,
+                "login_shell": login_shell,
+                "array_range": None,
+            }
+            return job_id, script_path
+
+        full_env = self._job_env(env, inherit_env)
+        bash_cmd = self._bash_cmd(script_path, login_shell)
+
+        # Open logs before spawn (they become the child's stdout/stderr). The
+        # PID isn't known yet, so key them by the script token, then re-key to
+        # the PID once we have it.
+        stdout_dest, stderr_dest = self._open_output_files(resources, job_id=str(token))
 
         proc = await asyncio.create_subprocess_exec(
             *bash_cmd,
@@ -91,8 +122,18 @@ class LocalExecutor(Executor):
             stderr=stderr_dest,
             env=full_env,
             cwd=cwd,
+            start_new_session=True,
         )
 
+        job_id = str(proc.pid)
+        self._open_fds[job_id] = self._open_fds.pop(str(token))
+        # Default log files were named by the pre-spawn token; re-point them at
+        # the PID so the documented stdout.{job_id}.log resolves. The open fds
+        # keep writing to the same inode across the rename.
+        if not resources.stdout_path:
+            base = Path(resources.work_dir)
+            for stream in ("stdout", "stderr"):
+                (base / f"{stream}.{token}.log").rename(base / f"{stream}.{job_id}.log")
         self._processes[job_id] = proc
         return job_id, script_path
 
@@ -110,6 +151,7 @@ class LocalExecutor(Executor):
         cwd: str | None = None,
         inherit_env: bool = True,
         login_shell: bool = False,
+        depends_on: list[str] | None = None,
     ) -> tuple[str, str | None]:
         """Spawn one subprocess per array element with ARRAY_INDEX env var."""
         if max_concurrent is not None:
@@ -122,14 +164,30 @@ class LocalExecutor(Executor):
         script = render_script(self.config, command, header, prologue, epilogue)
         script_path = write_script(resources.work_dir, script, name, next(self._script_counter))
 
-        job_id = str(self._next_id)
+        if depends_on:
+            job_id = f"waiting-{self._next_id}"
+            self._next_id += 1
+            self._waiting[job_id] = {
+                "script_path": script_path,
+                "resources": resources,
+                "env": env,
+                "cwd": cwd,
+                "inherit_env": inherit_env,
+                "login_shell": login_shell,
+                "array_range": array_range,
+            }
+            return job_id, script_path
+
+        # Array jobs spawn N processes, so no single PID identifies the job.
+        # Use a synthetic, deliberately non-numeric id: it can't be mistaken
+        # for a process-group id by the stateless cancel path (which would
+        # otherwise int() it), so an array is only ever cancelled in-process
+        # via its element handles.
+        job_id = f"array-{self._next_id}"
         self._next_id += 1
 
-        if inherit_env:
-            full_env = {**os.environ, **(env or {})}
-        else:
-            full_env = _baseline_env(env)
-        bash_cmd = ["bash", "-l", script_path] if login_shell else ["bash", script_path]
+        full_env = self._job_env(env, inherit_env)
+        bash_cmd = self._bash_cmd(script_path, login_shell)
 
         for index in range(array_range[0], array_range[1] + 1):
             element_env = {**full_env, "ARRAY_INDEX": str(index)}
@@ -147,6 +205,44 @@ class LocalExecutor(Executor):
 
         return job_id, script_path
 
+    async def _spawn_deferred(self, job_id: str, entry: dict) -> None:
+        """Spawn a parked job whose dependencies are now satisfied.
+
+        The job keeps its synthetic ``waiting-N`` id (ids never change once
+        returned), so the PID-based stateless cancel path does not apply —
+        same trade-off as local array jobs.
+        """
+        resources = entry["resources"]
+        full_env = self._job_env(entry["env"], entry["inherit_env"])
+        bash_cmd = self._bash_cmd(entry["script_path"], entry["login_shell"])
+
+        if entry["array_range"] is None:
+            stdout_dest, stderr_dest = self._open_output_files(resources, job_id=job_id)
+            proc = await asyncio.create_subprocess_exec(
+                *bash_cmd,
+                stdout=stdout_dest,
+                stderr=stderr_dest,
+                env=full_env,
+                cwd=entry["cwd"],
+                start_new_session=True,
+            )
+            self._processes[job_id] = proc
+        else:
+            start, end = entry["array_range"]
+            for index in range(start, end + 1):
+                element_env = {**full_env, "ARRAY_INDEX": str(index)}
+                stdout_dest, stderr_dest = self._open_output_files(
+                    resources, job_id=job_id, element_index=index,
+                )
+                proc = await asyncio.create_subprocess_exec(
+                    *bash_cmd,
+                    stdout=stdout_dest,
+                    stderr=stderr_dest,
+                    env=element_env,
+                    cwd=entry["cwd"],
+                )
+                self._processes[f"{job_id}[{index}]"] = proc
+
     def _build_status_args(self) -> list[str]:
         # Not used for local executor; poll() is overridden
         return []
@@ -223,11 +319,48 @@ class LocalExecutor(Executor):
             if record.is_terminal:
                 record.finish_time = datetime.now(timezone.utc)
 
+        # --- Waiting (dependency-deferred) jobs ---
+        for job_id in list(self._waiting):
+            record = self._jobs.get(job_id)
+            if record is None or record.is_terminal:
+                self._waiting.pop(job_id)
+                continue
+            dep_records = [self._jobs.get(d) for d in record.depends_on]
+            failed = [
+                d for d, r in zip(record.depends_on, dep_records)
+                if r is None or r.status in {JobStatus.FAILED, JobStatus.KILLED}
+            ]
+            now = datetime.now(timezone.utc)
+            if failed:
+                # An untracked dep (None) can never be satisfied — treat as failed.
+                self._waiting.pop(job_id)
+                record.status = JobStatus.KILLED
+                record.metadata["dependency_failed"] = failed
+                record.finish_time = now
+            elif all(r.status == JobStatus.DONE for r in dep_records):
+                entry = self._waiting.pop(job_id)
+                await self._spawn_deferred(job_id, entry)
+                record._last_seen = now
+            else:
+                record._last_seen = now  # keep zombie detection at bay while parked
+
         return {jid: r.status for jid, r in self._jobs.items()}
 
     async def cancel(self, job_id: str, *, done: bool = False) -> None:
-        """Terminate a local subprocess (or all element processes for an array job)."""
-        # Collect all live processes for this job (single + array elements)
+        """Terminate a local job and its whole process tree.
+
+        Fast path: if this executor submitted the job (its ``asyncio``
+        subprocess handle is still in ``_processes``), signal that handle
+        directly. Stateless path: otherwise ``job_id`` is treated as the
+        job's process-group id (== leader PID from submit) and the entire
+        group is killed by PID — so a fresh executor in a later process can
+        cancel a job it never submitted.
+        """
+        # A parked (dependency-deferred) job has no process yet; discard its
+        # entry so a later poll can't spawn it after cancellation.
+        deferred = self._waiting.pop(job_id, None)
+
+        # Fast path: live in-memory handles for this job (single + array elements).
         live: list[tuple[str, asyncio.subprocess.Process]] = []
         proc = self._processes.get(job_id)
         if proc and proc.returncode is None:
@@ -237,10 +370,10 @@ class LocalExecutor(Executor):
             if key.startswith(prefix) and proc.returncode is None:
                 live.append((key, proc))
 
-        # Send SIGTERM to all, then wait concurrently
-        for _key, p in live:
-            p.terminate()
         if live:
+            # Send SIGTERM to all, then wait concurrently
+            for _key, p in live:
+                p.terminate()
             tasks = [asyncio.ensure_future(p.wait()) for _key, p in live]
             _, pending = await asyncio.wait(tasks, timeout=5.0)
             # SIGKILL any that didn't exit in time
@@ -250,9 +383,11 @@ class LocalExecutor(Executor):
             # Reap the killed processes
             if pending:
                 await asyncio.wait(pending, timeout=5.0)
-
-        for key, _p in live:
-            self._close_output_files(key)
+            for key, _p in live:
+                self._close_output_files(key)
+        elif deferred is None and job_id not in self._processes:
+            # Stateless path: no handle for this job — kill by process group.
+            await self._terminate_group(job_id)
 
         target_status = JobStatus.DONE if done else JobStatus.KILLED
         if job_id in self._jobs:
@@ -263,6 +398,59 @@ class LocalExecutor(Executor):
                     elem.status = target_status
         logger.info("Cancelled local job %s (done=%s)", job_id, done)
 
+    async def _terminate_group(
+        self, job_id: str, grace_seconds: float = 3.0
+    ) -> None:
+        """Kill a job's process group (SIGTERM, then SIGKILL) by PID.
+
+        ``job_id`` is the process-group id (the leader PID from submit, which
+        used ``start_new_session=True``). Killing the group reaches the launcher
+        bash *and* its workload — bash does not forward SIGTERM to its child, so
+        signalling the leader alone would orphan the real job.
+
+        Raises ClusterAPIError if anything in the group outlives SIGKILL.
+        """
+        try:
+            pgid = int(job_id)
+        except ValueError:
+            # Non-PID id (e.g. an array counter) — nothing to kill statelessly.
+            logger.warning("Cannot cancel local job %s: no live handle", job_id)
+            return
+
+        def present() -> bool:
+            # True while the group has any member — including zombies not yet
+            # reaped. After SIGKILL a dead process lingers as a zombie until its
+            # parent (init, once the submitter exited) reaps it, so we poll
+            # briefly rather than trusting the first check.
+            try:
+                os.killpg(pgid, 0)
+                return True
+            except ProcessLookupError:
+                return False
+            except PermissionError:
+                # Group exists but is owned by someone else — treat as alive.
+                return True
+
+        # ponytail: no PID-reuse guard — the group id could have been recycled
+        # onto an unrelated process. This matches the prior job.pid behavior;
+        # add a start-time check here if reuse ever bites in practice.
+        if not present():
+            return
+        os.killpg(pgid, signal.SIGTERM)
+        deadline = time.monotonic() + grace_seconds
+        while present() and time.monotonic() < deadline:
+            await asyncio.sleep(0.1)
+        if present():
+            os.killpg(pgid, signal.SIGKILL)
+            # SIGKILL is immediate, but reaping the resulting zombies isn't.
+            deadline = time.monotonic() + 2.0
+            while present() and time.monotonic() < deadline:
+                await asyncio.sleep(0.05)
+        if present():
+            raise ClusterAPIError(
+                f"Local job {job_id} survived SIGKILL; may still be running"
+            )
+
     def _open_output_files(
         self,
         resources: ResourceSpec,