py-cluster-api 0.6.0__tar.gz → 0.8.0__tar.gz

{py_cluster_api-0.6.0 → py_cluster_api-0.8.0}/.github/workflows/ci.yml

@@ -23,7 +23,7 @@ jobs:
     - name: Set up Pixi
       uses: prefix-dev/setup-pixi@v0.9.0
       with:
-        pixi-version: v0.65.0
+        pixi-version: v0.70.2
         cache: true
 
     - name: Lint

{py_cluster_api-0.6.0 → py_cluster_api-0.8.0}/CLAUDE.md

@@ -2,7 +2,7 @@
 
 Generic Python library for submitting and monitoring jobs on HPC clusters. Wraps scheduler CLIs (bsub/bjobs/bkill) behind an async executor abstraction with an active polling monitor that fires callbacks on job completion. Inspired by dask-jobqueue's script templating and Nextflow's portable config profiles, but unlike dask-jobqueue, this library actively polls the scheduler rather than relying on workers phoning home.
 
-Key capabilities beyond submit/poll/cancel: `reconnect()` rediscovers running jobs after a process restart (requires `job_name_prefix`), and `cancel_by_name()` kills jobs by name pattern (LSF only).
+Key capabilities beyond submit/poll/cancel: `reconnect()` rediscovers running jobs after a process restart (requires `job_name_prefix`), and `cancel_by_name()` kills jobs by name pattern (LSF only). `depends_on` on `submit()`/`submit_array()` chains jobs (native `bsub -w ... -ti` on LSF; poll-driven deferred start on Local) with auto-cancel of dependents when an upstream job fails.
 
 Founding principles: async-only API, executors are thin wrappers around scheduler CLIs, all state lives in `JobRecord` dataclasses tracked in-process, monitoring is poll-based via `bjobs -json`, and configuration uses Nextflow-style YAML profiles.
 

{py_cluster_api-0.6.0 → py_cluster_api-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: py-cluster-api
-Version: 0.6.0
+Version: 0.8.0
 Summary: Generic Python library for running jobs on HPC clusters
 Project-URL: Homepage, https://github.com/JaneliaSciComp/py-cluster-api
 Project-URL: Repository, https://github.com/JaneliaSciComp/py-cluster-api
@@ -71,6 +71,9 @@ A Python library for submitting and monitoring jobs on HPC clusters. Supports ru
 - **Zombie detection** — jobs that disappear from the scheduler are marked as failed
 - **YAML config with profiles** — Nextflow-style config with per-environment profiles
 - **Callback chaining** — register `on_success`, `on_failure`, or `on_exit` handlers on any job
+- **Job dependencies** — chain jobs with `depends_on`; dependents are auto-cancelled if an upstream job fails
+- **Environment control** — `inherit_env` and `login_shell` submit options control what environment a job runs with
+- **Reconnect & external tracking** — `reconnect()` rediscovers jobs after a restart; `track()` seeds a job from an external store (e.g. a database) into the executor
 
 ## Installation
 
@@ -136,6 +139,24 @@ async def run_array():
 
 The array index environment variable depends on the executor: LSF uses `$LSB_JOBINDEX`, while the local executor uses `$ARRAY_INDEX`.
 
+### Job Dependencies
+
+Chain jobs so each stage starts only after the previous one succeeds:
+
+```python
+convert = await executor.submit("convert.sh input.tif", name="convert-s0")
+pyramid = await executor.submit(
+    "build_pyramid.sh", name="pyramid",
+    depends_on=[convert],   # JobRecords or raw job-id strings
+)
+```
+
+`depends_on` accepts multiple jobs (fan-in). On LSF this uses native
+`bsub -w "done(...)" -ti`, so chains keep running if your process exits,
+and LSF terminates dependents whose upstream failed — no jobs stuck
+pending forever. When a dependent is cancelled this way,
+`record.metadata["dependency_failed"]` lists the failed upstream job ids.
+
 ### Reconnecting After Restart
 
 If your process crashes or restarts, `reconnect()` rediscovers running jobs from the scheduler and resumes tracking them. Requires `job_name_prefix` to be set in config.
@@ -156,6 +177,28 @@ async def resume():
     await monitor.stop()
 ```
 
+### Environment Control
+
+By default a submitted job inherits the submitting process's environment plus whatever `env` you pass. Set `inherit_env=False` to start from a minimal environment (only `env` and scheduler-provided variables), or `login_shell=True` to run under a login shell so the target user's own profile builds `PATH`, modules, conda, etc.:
+
+```python
+job = await executor.submit(
+    command="my_tool.sh",
+    name="my-job",
+    inherit_env=False,
+    login_shell=True,
+    env={"MY_VAR": "1"},
+)
+```
+
+### Tracking Jobs From External State
+
+`track()` seeds the executor with a job ID from a persistent store (e.g. a database) without re-submitting it, so a later `poll()` or `cancel()` can act on jobs submitted by a previous process:
+
+```python
+job = executor.track(job_id="12345", status=JobStatus.RUNNING)
+```
+
 ### Local Testing
 
 ```python

{py_cluster_api-0.6.0 → py_cluster_api-0.8.0}/README.md

@@ -19,6 +19,9 @@ A Python library for submitting and monitoring jobs on HPC clusters. Supports ru
 - **Zombie detection** — jobs that disappear from the scheduler are marked as failed
 - **YAML config with profiles** — Nextflow-style config with per-environment profiles
 - **Callback chaining** — register `on_success`, `on_failure`, or `on_exit` handlers on any job
+- **Job dependencies** — chain jobs with `depends_on`; dependents are auto-cancelled if an upstream job fails
+- **Environment control** — `inherit_env` and `login_shell` submit options control what environment a job runs with
+- **Reconnect & external tracking** — `reconnect()` rediscovers jobs after a restart; `track()` seeds a job from an external store (e.g. a database) into the executor
 
 ## Installation
 
@@ -84,6 +87,24 @@ async def run_array():
 
 The array index environment variable depends on the executor: LSF uses `$LSB_JOBINDEX`, while the local executor uses `$ARRAY_INDEX`.
 
+### Job Dependencies
+
+Chain jobs so each stage starts only after the previous one succeeds:
+
+```python
+convert = await executor.submit("convert.sh input.tif", name="convert-s0")
+pyramid = await executor.submit(
+    "build_pyramid.sh", name="pyramid",
+    depends_on=[convert],   # JobRecords or raw job-id strings
+)
+```
+
+`depends_on` accepts multiple jobs (fan-in). On LSF this uses native
+`bsub -w "done(...)" -ti`, so chains keep running if your process exits,
+and LSF terminates dependents whose upstream failed — no jobs stuck
+pending forever. When a dependent is cancelled this way,
+`record.metadata["dependency_failed"]` lists the failed upstream job ids.
+
 ### Reconnecting After Restart
 
 If your process crashes or restarts, `reconnect()` rediscovers running jobs from the scheduler and resumes tracking them. Requires `job_name_prefix` to be set in config.
@@ -104,6 +125,28 @@ async def resume():
     await monitor.stop()
 ```
 
+### Environment Control
+
+By default a submitted job inherits the submitting process's environment plus whatever `env` you pass. Set `inherit_env=False` to start from a minimal environment (only `env` and scheduler-provided variables), or `login_shell=True` to run under a login shell so the target user's own profile builds `PATH`, modules, conda, etc.:
+
+```python
+job = await executor.submit(
+    command="my_tool.sh",
+    name="my-job",
+    inherit_env=False,
+    login_shell=True,
+    env={"MY_VAR": "1"},
+)
+```
+
+### Tracking Jobs From External State
+
+`track()` seeds the executor with a job ID from a persistent store (e.g. a database) without re-submitting it, so a later `poll()` or `cancel()` can act on jobs submitted by a previous process:
+
+```python
+job = executor.track(job_id="12345", status=JobStatus.RUNNING)
+```
+
 ### Local Testing
 
 ```python

{py_cluster_api-0.6.0 → py_cluster_api-0.8.0}/cluster_api/_types.py

@@ -98,6 +98,7 @@ class JobRecord:
     start_time: datetime | None = None
     finish_time: datetime | None = None
     array_elements: dict[int, ArrayElement] = field(default_factory=dict)
+    depends_on: list[str] = field(default_factory=list)
     _callbacks: list[tuple[JobExitCondition, Callable]] = field(
         default_factory=list, repr=False
     )

{py_cluster_api-0.6.0 → py_cluster_api-0.8.0}/cluster_api/core.py

@@ -7,6 +7,7 @@ import asyncio
 import logging
 import os
 import re
+from collections.abc import Sequence
 from datetime import datetime, timezone
 from typing import Any
 
@@ -73,6 +74,15 @@ class Executor(abc.ABC):
 
     # --- Submission ---
 
+    @staticmethod
+    def _resolve_dependency_ids(
+        depends_on: Sequence[JobRecord | str] | None,
+    ) -> list[str]:
+        """Resolve JobRecords or raw id strings to a list of job ids."""
+        if not depends_on:
+            return []
+        return [d.job_id if isinstance(d, JobRecord) else str(d) for d in depends_on]
+
     async def submit(
         self,
         command: str,
@@ -82,14 +92,34 @@ class Executor(abc.ABC):
         epilogue: list[str] | None = None,
         env: dict[str, str] | None = None,
         metadata: dict[str, Any] | None = None,
+        inherit_env: bool = True,
+        login_shell: bool = False,
+        depends_on: Sequence[JobRecord | str] | None = None,
     ) -> JobRecord:
-        """Submit a job to the scheduler."""
+        """Submit a job to the scheduler.
+
+        Args:
+            inherit_env: When True (default), the job inherits the submitting
+                process's environment. When False, the job starts from a
+                minimal environment: only *env* plus scheduler-provided
+                variables reach the job.
+            login_shell: When True, the job runs under a login shell so the
+                target user's own profile builds the environment (PATH,
+                modules, conda), as if they had SSH'd to the node.
+            depends_on: Jobs that must finish successfully before this job
+                starts. Accepts JobRecords or raw job-id strings. If any
+                dependency fails or is killed, this job is cancelled instead
+                of pending forever.
+        """
         resources = resources or ResourceSpec()
         full_name = _sanitize_job_name(f"{self._prefix}-{name}" if self._prefix else name)
+        dep_ids = self._resolve_dependency_ids(depends_on)
 
         job_id, script_path = await self._submit_job(
             command, full_name, resources, prologue, epilogue, env,
             cwd=resources.work_dir,
+            inherit_env=inherit_env, login_shell=login_shell,
+            depends_on=dep_ids or None,
         )
 
         record = JobRecord(
@@ -100,6 +130,7 @@ class Executor(abc.ABC):
             resources=resources,
             script_path=script_path,
             metadata=metadata or {},
+            depends_on=dep_ids,
             _last_seen=datetime.now(timezone.utc),
         )
         self._jobs[job_id] = record
@@ -117,14 +148,20 @@ class Executor(abc.ABC):
         env: dict[str, str] | None = None,
         metadata: dict[str, Any] | None = None,
         max_concurrent: int | None = None,
+        inherit_env: bool = True,
+        login_shell: bool = False,
+        depends_on: Sequence[JobRecord | str] | None = None,
     ) -> JobRecord:
         """Submit a job array to the scheduler."""
         resources = resources or ResourceSpec()
         full_name = _sanitize_job_name(f"{self._prefix}-{name}" if self._prefix else name)
+        dep_ids = self._resolve_dependency_ids(depends_on)
 
         job_id, script_path = await self._submit_array_job(
             command, full_name, array_range, resources, prologue, epilogue,
             env, max_concurrent, cwd=resources.work_dir,
+            inherit_env=inherit_env, login_shell=login_shell,
+            depends_on=dep_ids or None,
         )
 
         meta = {**(metadata or {}), "array_range": array_range}
@@ -139,6 +176,7 @@ class Executor(abc.ABC):
             resources=resources,
             script_path=script_path,
             metadata=meta,
+            depends_on=dep_ids,
             _last_seen=datetime.now(timezone.utc),
         )
         self._jobs[job_id] = record
@@ -159,6 +197,9 @@ class Executor(abc.ABC):
         env: dict[str, str] | None = None,
         *,
         cwd: str | None = None,
+        inherit_env: bool = True,
+        login_shell: bool = False,
+        depends_on: list[str] | None = None,
     ) -> tuple[str, str | None]:
         """Submit a single job.
 
@@ -179,10 +220,15 @@ class Executor(abc.ABC):
         max_concurrent: int | None = None,
         *,
         cwd: str | None = None,
+        inherit_env: bool = True,
+        login_shell: bool = False,
+        depends_on: list[str] | None = None,
     ) -> tuple[str, str | None]:
         """Submit an array job. Override in subclasses."""
         return await self._submit_job(
             command, name, resources, prologue, epilogue, env, cwd=cwd,
+            inherit_env=inherit_env, login_shell=login_shell,
+            depends_on=depends_on,
         )
 
     def _job_id_from_submit_output(self, out: str) -> str:
@@ -315,6 +361,22 @@ class Executor(abc.ABC):
             record = self._jobs[parent_id]
             record.status = record.compute_array_status()
 
+        # Annotate dependency-driven failures so callbacks can distinguish
+        # "upstream broke" from "my job broke".
+        for record in self._jobs.values():
+            if (
+                record.depends_on
+                and record.status in (JobStatus.FAILED, JobStatus.KILLED)
+                and "dependency_failed" not in record.metadata
+            ):
+                culprits = [
+                    d for d in record.depends_on
+                    if (dep := self._jobs.get(d)) is not None
+                    and dep.status in (JobStatus.FAILED, JobStatus.KILLED)
+                ]
+                if culprits:
+                    record.metadata["dependency_failed"] = culprits
+
         return {jid: r.status for jid, r in self._jobs.items()}
 
     # --- Subprocess helper ---