libpipe 0.7__tar.gz → 0.9__tar.gz

libpipe-0.9/PKG-INFO

@@ -0,0 +1,85 @@
+Metadata-Version: 2.4
+Name: libpipe
+Version: 0.9
+Summary: Pipeline orchestration and node management utilities
+License-Expression: GPL-3.0-or-later
+License-File: LICENSE
+Author: Florent Mertens
+Author-email: florent.mertens@gmail.com
+Requires-Python: >=3.10,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Provides-Extra: dev
+Provides-Extra: docs
+Requires-Dist: asyncssh (>=2.0)
+Requires-Dist: click (>=8.0)
+Requires-Dist: myst-parser (>=2) ; extra == "docs"
+Requires-Dist: progressbar2 (>=3.40)
+Requires-Dist: pydata-sphinx-theme (>=0.15) ; extra == "docs"
+Requires-Dist: pytest (>=8.0) ; extra == "dev"
+Requires-Dist: pytest-cov (>=5.0) ; extra == "dev"
+Requires-Dist: ruff (==0.16.1) ; extra == "dev"
+Requires-Dist: sphinx (>=7) ; extra == "docs"
+Requires-Dist: sphinx-click (>=6) ; extra == "docs"
+Requires-Dist: toml (>=0.10)
+Project-URL: Documentation, https://libpipe.readthedocs.io
+Project-URL: Homepage, https://gitlab.com/flomertens/libpipe
+Project-URL: Issues, https://gitlab.com/flomertens/libpipe/-/issues
+Project-URL: Repository, https://gitlab.com/flomertens/libpipe
+Description-Content-Type: text/markdown
+
+Libpipe
+=======
+
+Common pipeline framework and library
+
+Installation
+------------
+
+libpipe can be installed via pip:
+
+    $ pip install libpipe
+
+and requires Python 3.10 or higher.
+
+Worker pools on a Slurm cluster
+-------------------------------
+
+A `WorkerPool` runs its tasks locally, on other hosts over SSH, or through Slurm with the
+host `slurm`: each task is then started with `srun`, as a job of its own when the pool runs
+on a login node, as a job step inside an allocation (`salloc`, `sbatch`).
+
+    pool = worker.WorkerPool(['slurm'], name='Imaging', max_tasks_per_worker=8,
+                             slurm_options='-p regular --mem=200G --time=24:00:00')
+
+`max_tasks_per_worker` is the number of jobs in flight. The srun options (partition,
+account, resources) come from `slurm_options`, or from the `LIBPIPE_SLURM_OPTIONS`
+environment variable; `max_time` becomes the job time limit when the options set none.
+Host constraints (`run_on_host`) do not apply: Slurm places the tasks, so the data must be
+on a file system shared with the compute nodes.
+
+With `slurm_nodes`, a pool on a login node requests one allocation of up to that many nodes
+when its first task starts, runs each task as a job step in it (`srun --jobid --exact`), and
+releases it at the end, or after 10 minutes without step. Slurm packs the steps on the nodes
+from their CPUs and memory, given by the category of the task; a step starts when the
+allocation has its CPUs free, so that no srun waits polling the controller:
+
+    pool = worker.WorkerPool(['slurm'], name='Cubes', slurm_nodes=4,
+                             resources={'heavy': {'cpus': 12, 'mem': '40GB'}})
+    pool.add('pstool gen_vis_cube ...', category='small', label='gen_vis_cube')
+
+The categories (`DEFAULT_RESOURCES`) are `small` (1 CPU, 2 GB), `medium` (4, 8 GB),
+`high_mem` (2, 16 GB), `cpu` (12, 8 GB) and `heavy` (12, 40 GB, the default), the others capped
+at `heavy`; a step also gets the matching `OMP_NUM_THREADS`. The options apply to the jobs and
+the allocation, not to the steps. Inside an allocation started by hand, the tasks are job
+steps with these resources as well.
+
+Slurm does not necessarily limit the memory of a job step: the memory of a category must stay
+above the peak memory of its tasks. The local and Slurm pools measure each task (wall time,
+CPU time, peak RSS, in its log file), and `WorkerPoolReport.usage_lines()` summarises them per
+task type (`label`).
+

libpipe-0.9/README.md

@@ -0,0 +1,50 @@
+Libpipe
+=======
+
+Common pipeline framework and library
+
+Installation
+------------
+
+libpipe can be installed via pip:
+
+    $ pip install libpipe
+
+and requires Python 3.10 or higher.
+
+Worker pools on a Slurm cluster
+-------------------------------
+
+A `WorkerPool` runs its tasks locally, on other hosts over SSH, or through Slurm with the
+host `slurm`: each task is then started with `srun`, as a job of its own when the pool runs
+on a login node, as a job step inside an allocation (`salloc`, `sbatch`).
+
+    pool = worker.WorkerPool(['slurm'], name='Imaging', max_tasks_per_worker=8,
+                             slurm_options='-p regular --mem=200G --time=24:00:00')
+
+`max_tasks_per_worker` is the number of jobs in flight. The srun options (partition,
+account, resources) come from `slurm_options`, or from the `LIBPIPE_SLURM_OPTIONS`
+environment variable; `max_time` becomes the job time limit when the options set none.
+Host constraints (`run_on_host`) do not apply: Slurm places the tasks, so the data must be
+on a file system shared with the compute nodes.
+
+With `slurm_nodes`, a pool on a login node requests one allocation of up to that many nodes
+when its first task starts, runs each task as a job step in it (`srun --jobid --exact`), and
+releases it at the end, or after 10 minutes without step. Slurm packs the steps on the nodes
+from their CPUs and memory, given by the category of the task; a step starts when the
+allocation has its CPUs free, so that no srun waits polling the controller:
+
+    pool = worker.WorkerPool(['slurm'], name='Cubes', slurm_nodes=4,
+                             resources={'heavy': {'cpus': 12, 'mem': '40GB'}})
+    pool.add('pstool gen_vis_cube ...', category='small', label='gen_vis_cube')
+
+The categories (`DEFAULT_RESOURCES`) are `small` (1 CPU, 2 GB), `medium` (4, 8 GB),
+`high_mem` (2, 16 GB), `cpu` (12, 8 GB) and `heavy` (12, 40 GB, the default), the others capped
+at `heavy`; a step also gets the matching `OMP_NUM_THREADS`. The options apply to the jobs and
+the allocation, not to the steps. Inside an allocation started by hand, the tasks are job
+steps with these resources as well.
+
+Slurm does not necessarily limit the memory of a job step: the memory of a category must stay
+above the peak memory of its tasks. The local and Slurm pools measure each task (wall time,
+CPU time, peak RSS, in its log file), and `WorkerPoolReport.usage_lines()` summarises them per
+task type (`label`).

libpipe-0.9/libpipe/taskrun.py

@@ -0,0 +1,25 @@
+'''Run a command, then report its wall time, CPU time and peak memory on stderr, for the task
+accounting of WorkerPool.
+
+Usage: python taskrun.py COMMAND [ARGS...]'''
+
+import resource
+import subprocess
+import sys
+import time
+
+MARKER = '[libpipe] usage '
+
+
+def main():
+    start = time.monotonic()
+    returncode = subprocess.call(sys.argv[1:])
+    # Of the waited descendants: peak RSS of the largest one, CPU time of all
+    usage = resource.getrusage(resource.RUSAGE_CHILDREN)
+    print(f'{MARKER}wall={time.monotonic() - start:.1f} cpu={usage.ru_utime + usage.ru_stime:.1f} '
+          f'maxrss_kb={usage.ru_maxrss}', file=sys.stderr, flush=True)
+    sys.exit(returncode if returncode >= 0 else 128 - returncode)
+
+
+if __name__ == '__main__':
+    main()