executorlib 0.0.11__tar.gz → 0.2.0__tar.gz

{executorlib-0.0.11/executorlib.egg-info → executorlib-0.2.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.1
 Name: executorlib
-Version: 0.0.11
-Summary: Scale serial and MPI-parallel python functions over hundreds of compute nodes all from within a jupyter notebook or serial python process.
+Version: 0.2.0
+Summary: Up-scale python functions for high performance computing (HPC) with executorlib.
 Author-email: Jan Janssen <janssen@lanl.gov>
 License: BSD 3-Clause License
         
@@ -36,7 +36,7 @@ License: BSD 3-Clause License
 Project-URL: Homepage, https://github.com/pyiron/executorlib
 Project-URL: Documentation, https://executorlib.readthedocs.io
 Project-URL: Repository, https://github.com/pyiron/executorlib
-Keywords: pyiron
+Keywords: high performance computing,hpc,task scheduler,slurm,flux-framework,executor
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Topic :: Scientific/Engineering :: Physics
 Classifier: License :: OSI Approved :: BSD License
@@ -63,9 +63,9 @@ Requires-Dist: networkx<=3.4.2,>=2.8.8; extra == "graphnotebook"
 Requires-Dist: ipython<=8.32.0,>=7.33.0; extra == "graphnotebook"
 Provides-Extra: mpi
 Requires-Dist: mpi4py<=4.0.1,>=3.1.4; extra == "mpi"
-Provides-Extra: submission
-Requires-Dist: pysqa==0.2.3; extra == "submission"
-Requires-Dist: h5py<=3.12.1,>=3.6.0; extra == "submission"
+Provides-Extra: cluster
+Requires-Dist: pysqa==0.2.3; extra == "cluster"
+Requires-Dist: h5py<=3.12.1,>=3.6.0; extra == "cluster"
 Provides-Extra: all
 Requires-Dist: mpi4py<=4.0.1,>=3.1.4; extra == "all"
 Requires-Dist: pysqa==0.2.3; extra == "all"
@@ -75,8 +75,8 @@ Requires-Dist: networkx<=3.4.2,>=2.8.8; extra == "all"
 Requires-Dist: ipython<=8.32.0,>=7.33.0; extra == "all"
 
 # executorlib
-[![Unittests](https://github.com/pyiron/executorlib/actions/workflows/unittest-openmpi.yml/badge.svg)](https://github.com/pyiron/executorlib/actions/workflows/unittest-openmpi.yml)
-[![Coverage Status](https://coveralls.io/repos/github/pyiron/executorlib/badge.svg?branch=main)](https://coveralls.io/github/pyiron/executorlib?branch=main)
+[![Pipeline](https://github.com/pyiron/executorlib/actions/workflows/pipeline.yml/badge.svg)](https://github.com/pyiron/executorlib/actions/workflows/pipeline.yml)
+[![codecov](https://codecov.io/gh/pyiron/executorlib/graph/badge.svg?token=KFIO3R08H3)](https://codecov.io/gh/pyiron/executorlib)
 [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/pyiron/executorlib/HEAD?labpath=notebooks%2Fexamples.ipynb)
 
 Up-scale python functions for high performance computing (HPC) with executorlib. 
@@ -99,17 +99,17 @@ with the [ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futu
 [ThreadPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) for parallel 
 execution of Python functions on a single computer. executorlib extends this functionality to distribute Python 
 functions over multiple computers within a high performance computing (HPC) cluster. This can be either achieved by 
-submitting each function as individual job to the HPC job scheduler - [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html) - 
-or by requesting a compute allocation of multiple nodes and then distribute the Python functions within this - allocation -
-[HPC Allocation Mode](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html). Finally, to accelerate the 
-development process executorlib also provides a - [Local Mode](https://executorlib.readthedocs.io/en/latest/1-local.html) - 
-to use the executorlib functionality on a single workstation for testing. Starting with the [Local Mode](https://executorlib.readthedocs.io/en/latest/1-local.html) 
-set by setting the backend parameter to local - `backend="local"`:
+submitting each function as individual job to the HPC job scheduler with an [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html) - 
+or by requesting a job from the HPC cluster and then distribute the Python functions within this job with an
+[HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html). Finally, to accelerate the 
+development process executorlib also provides a [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html) - 
+to use the executorlib functionality on a laptop, workstation or single compute node for testing. Starting with the 
+[Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html):
 ```python
-from executorlib import Executor
+from executorlib import SingleNodeExecutor
 
 
-with Executor(backend="local") as exe:
+with SingleNodeExecutor() as exe:
     future_lst = [exe.submit(sum, [i, i]) for i in range(1, 5)]
     print([f.result() for f in future_lst])
 ```
@@ -117,7 +117,7 @@ In the same way executorlib can also execute Python functions which use addition
 CPU cores, CPU threads or GPUs. For example if the Python function internally uses the Message Passing Interface (MPI) 
 via the [mpi4py](https://mpi4py.readthedocs.io) Python libary: 
 ```python
-from executorlib import Executor
+from executorlib import SingleNodeExecutor
 
 
 def calc(i):
@@ -128,7 +128,7 @@ def calc(i):
     return i, size, rank
 
 
-with Executor(backend="local") as exe:
+with SingleNodeExecutor() as exe:
     fs = exe.submit(calc, 3, resource_dict={"cores": 2})
     print(fs.result())
 ```
@@ -142,11 +142,11 @@ This flexibility to assign computing resources on a per-function-call basis simp
 Only the part of the Python functions which benefit from parallel execution are implemented as MPI parallel Python 
 funtions, while the rest of the program remains serial. 
 
-The same function can be submitted to the [SLURM](https://slurm.schedmd.com) queuing by just changing the `backend` 
-parameter to `slurm_submission`. The rest of the example remains the same, which highlights how executorlib accelerates
-the rapid prototyping and up-scaling of HPC Python programs. 
+The same function can be submitted to the [SLURM](https://slurm.schedmd.com) job scheduler by replacing the 
+`SingleNodeExecutor` with the `SlurmClusterExecutor`.  The rest of the example remains the same, which highlights how 
+executorlib accelerates the rapid prototyping and up-scaling of HPC Python programs. 
 ```python
-from executorlib import Executor
+from executorlib import SlurmClusterExecutor
 
 
 def calc(i):
@@ -157,21 +157,21 @@ def calc(i):
     return i, size, rank
 
 
-with Executor(backend="slurm_submission") as exe:
+with SlurmClusterExecutor() as exe:
     fs = exe.submit(calc, 3, resource_dict={"cores": 2})
     print(fs.result())
 ```
 In this case the [Python simple queuing system adapter (pysqa)](https://pysqa.readthedocs.io) is used to submit the 
 `calc()` function to the [SLURM](https://slurm.schedmd.com) job scheduler and request an allocation with two CPU cores 
-for the execution of the function - [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html). In the background the [sbatch](https://slurm.schedmd.com/sbatch.html) 
+for the execution of the function - [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html). In the background the [sbatch](https://slurm.schedmd.com/sbatch.html) 
 command is used to request the allocation to execute the Python function. 
 
-Within a given [SLURM](https://slurm.schedmd.com) allocation executorlib can also be used to assign a subset of the 
+Within a given [SLURM](https://slurm.schedmd.com) job executorlib can also be used to assign a subset of the 
 available computing resources to execute a given Python function. In terms of the [SLURM](https://slurm.schedmd.com) 
 commands, this functionality internally uses the [srun](https://slurm.schedmd.com/srun.html) command to receive a subset
 of the resources of a given queuing system allocation. 
 ```python
-from executorlib import Executor
+from executorlib import SlurmJobExecutor
 
 
 def calc(i):
@@ -182,7 +182,7 @@ def calc(i):
     return i, size, rank
 
 
-with Executor(backend="slurm_allocation") as exe:
+with SlurmJobExecutor() as exe:
     fs = exe.submit(calc, 3, resource_dict={"cores": 2})
     print(fs.result())
 ```
@@ -192,7 +192,7 @@ In addition, to support for [SLURM](https://slurm.schedmd.com) executorlib also
 to address the needs for the up-coming generation of Exascale computers. Still even on traditional HPC clusters the 
 hierarchical approach of the [flux](http://flux-framework.org) is beneficial to distribute hundreds of tasks within a
 given allocation. Even when [SLURM](https://slurm.schedmd.com) is used as primary job scheduler of your HPC, it is 
-recommended to use [SLURM with flux](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html#slurm-with-flux) 
+recommended to use [SLURM with flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux) 
 as hierarchical job scheduler within the allocations. 
 
 ## Documentation
@@ -200,21 +200,21 @@ as hierarchical job scheduler within the allocations.
   * [Minimal](https://executorlib.readthedocs.io/en/latest/installation.html#minimal)
   * [MPI Support](https://executorlib.readthedocs.io/en/latest/installation.html#mpi-support)
   * [Caching](https://executorlib.readthedocs.io/en/latest/installation.html#caching)
-  * [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-submission-mode)
-  * [HPC Allocation Mode](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-allocation-mode)
+  * [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-cluster-executor)
+  * [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-job-executor)
   * [Visualisation](https://executorlib.readthedocs.io/en/latest/installation.html#visualisation)
   * [For Developers](https://executorlib.readthedocs.io/en/latest/installation.html#for-developers)
-* [Local Mode](https://executorlib.readthedocs.io/en/latest/1-local.html)
-  * [Basic Functionality](https://executorlib.readthedocs.io/en/latest/1-local.html#basic-functionality)
-  * [Parallel Functions](https://executorlib.readthedocs.io/en/latest/1-local.html#parallel-functions)
-  * [Performance Optimization](https://executorlib.readthedocs.io/en/latest/1-local.html#performance-optimization)
-* [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html)
-  * [SLURM](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html#slurm)
-  * [Flux](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html#flux)
-* [HPC Allocation Mode](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html)
-  * [SLURM](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html#slurm)
-  * [SLURM with Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html#slurm-with-flux)
-  * [Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html#flux)
+* [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html)
+  * [Basic Functionality](https://executorlib.readthedocs.io/en/latest/1-single-node.html#basic-functionality)
+  * [Parallel Functions](https://executorlib.readthedocs.io/en/latest/1-single-node.html#parallel-functions)
+  * [Performance Optimization](https://executorlib.readthedocs.io/en/latest/1-single-node.html#performance-optimization)
+* [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html)
+  * [SLURM](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#slurm)
+  * [Flux](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#flux)
+* [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html)
+  * [SLURM](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm)
+  * [SLURM with Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux)
+  * [Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#flux)
 * [Trouble Shooting](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html)
   * [Filesystem Usage](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#filesystem-usage)
   * [Firewall Issues](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#firewall-issues)

{executorlib-0.0.11 → executorlib-0.2.0}/README.md

@@ -1,6 +1,6 @@
 # executorlib
-[![Unittests](https://github.com/pyiron/executorlib/actions/workflows/unittest-openmpi.yml/badge.svg)](https://github.com/pyiron/executorlib/actions/workflows/unittest-openmpi.yml)
-[![Coverage Status](https://coveralls.io/repos/github/pyiron/executorlib/badge.svg?branch=main)](https://coveralls.io/github/pyiron/executorlib?branch=main)
+[![Pipeline](https://github.com/pyiron/executorlib/actions/workflows/pipeline.yml/badge.svg)](https://github.com/pyiron/executorlib/actions/workflows/pipeline.yml)
+[![codecov](https://codecov.io/gh/pyiron/executorlib/graph/badge.svg?token=KFIO3R08H3)](https://codecov.io/gh/pyiron/executorlib)
 [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/pyiron/executorlib/HEAD?labpath=notebooks%2Fexamples.ipynb)
 
 Up-scale python functions for high performance computing (HPC) with executorlib. 
@@ -23,17 +23,17 @@ with the [ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futu
 [ThreadPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) for parallel 
 execution of Python functions on a single computer. executorlib extends this functionality to distribute Python 
 functions over multiple computers within a high performance computing (HPC) cluster. This can be either achieved by 
-submitting each function as individual job to the HPC job scheduler - [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html) - 
-or by requesting a compute allocation of multiple nodes and then distribute the Python functions within this - allocation -
-[HPC Allocation Mode](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html). Finally, to accelerate the 
-development process executorlib also provides a - [Local Mode](https://executorlib.readthedocs.io/en/latest/1-local.html) - 
-to use the executorlib functionality on a single workstation for testing. Starting with the [Local Mode](https://executorlib.readthedocs.io/en/latest/1-local.html) 
-set by setting the backend parameter to local - `backend="local"`:
+submitting each function as individual job to the HPC job scheduler with an [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html) - 
+or by requesting a job from the HPC cluster and then distribute the Python functions within this job with an
+[HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html). Finally, to accelerate the 
+development process executorlib also provides a [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html) - 
+to use the executorlib functionality on a laptop, workstation or single compute node for testing. Starting with the 
+[Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html):
 ```python
-from executorlib import Executor
+from executorlib import SingleNodeExecutor
 
 
-with Executor(backend="local") as exe:
+with SingleNodeExecutor() as exe:
     future_lst = [exe.submit(sum, [i, i]) for i in range(1, 5)]
     print([f.result() for f in future_lst])
 ```
@@ -41,7 +41,7 @@ In the same way executorlib can also execute Python functions which use addition
 CPU cores, CPU threads or GPUs. For example if the Python function internally uses the Message Passing Interface (MPI) 
 via the [mpi4py](https://mpi4py.readthedocs.io) Python libary: 
 ```python
-from executorlib import Executor
+from executorlib import SingleNodeExecutor
 
 
 def calc(i):
@@ -52,7 +52,7 @@ def calc(i):
     return i, size, rank
 
 
-with Executor(backend="local") as exe:
+with SingleNodeExecutor() as exe:
     fs = exe.submit(calc, 3, resource_dict={"cores": 2})
     print(fs.result())
 ```
@@ -66,11 +66,11 @@ This flexibility to assign computing resources on a per-function-call basis simp
 Only the part of the Python functions which benefit from parallel execution are implemented as MPI parallel Python 
 funtions, while the rest of the program remains serial. 
 
-The same function can be submitted to the [SLURM](https://slurm.schedmd.com) queuing by just changing the `backend` 
-parameter to `slurm_submission`. The rest of the example remains the same, which highlights how executorlib accelerates
-the rapid prototyping and up-scaling of HPC Python programs. 
+The same function can be submitted to the [SLURM](https://slurm.schedmd.com) job scheduler by replacing the 
+`SingleNodeExecutor` with the `SlurmClusterExecutor`.  The rest of the example remains the same, which highlights how 
+executorlib accelerates the rapid prototyping and up-scaling of HPC Python programs. 
 ```python
-from executorlib import Executor
+from executorlib import SlurmClusterExecutor
 
 
 def calc(i):
@@ -81,21 +81,21 @@ def calc(i):
     return i, size, rank
 
 
-with Executor(backend="slurm_submission") as exe:
+with SlurmClusterExecutor() as exe:
     fs = exe.submit(calc, 3, resource_dict={"cores": 2})
     print(fs.result())
 ```
 In this case the [Python simple queuing system adapter (pysqa)](https://pysqa.readthedocs.io) is used to submit the 
 `calc()` function to the [SLURM](https://slurm.schedmd.com) job scheduler and request an allocation with two CPU cores 
-for the execution of the function - [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html). In the background the [sbatch](https://slurm.schedmd.com/sbatch.html) 
+for the execution of the function - [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html). In the background the [sbatch](https://slurm.schedmd.com/sbatch.html) 
 command is used to request the allocation to execute the Python function. 
 
-Within a given [SLURM](https://slurm.schedmd.com) allocation executorlib can also be used to assign a subset of the 
+Within a given [SLURM](https://slurm.schedmd.com) job executorlib can also be used to assign a subset of the 
 available computing resources to execute a given Python function. In terms of the [SLURM](https://slurm.schedmd.com) 
 commands, this functionality internally uses the [srun](https://slurm.schedmd.com/srun.html) command to receive a subset
 of the resources of a given queuing system allocation. 
 ```python
-from executorlib import Executor
+from executorlib import SlurmJobExecutor
 
 
 def calc(i):
@@ -106,7 +106,7 @@ def calc(i):
     return i, size, rank
 
 
-with Executor(backend="slurm_allocation") as exe:
+with SlurmJobExecutor() as exe:
     fs = exe.submit(calc, 3, resource_dict={"cores": 2})
     print(fs.result())
 ```
@@ -116,7 +116,7 @@ In addition, to support for [SLURM](https://slurm.schedmd.com) executorlib also
 to address the needs for the up-coming generation of Exascale computers. Still even on traditional HPC clusters the 
 hierarchical approach of the [flux](http://flux-framework.org) is beneficial to distribute hundreds of tasks within a
 given allocation. Even when [SLURM](https://slurm.schedmd.com) is used as primary job scheduler of your HPC, it is 
-recommended to use [SLURM with flux](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html#slurm-with-flux) 
+recommended to use [SLURM with flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux) 
 as hierarchical job scheduler within the allocations. 
 
 ## Documentation
@@ -124,21 +124,21 @@ as hierarchical job scheduler within the allocations.
   * [Minimal](https://executorlib.readthedocs.io/en/latest/installation.html#minimal)
   * [MPI Support](https://executorlib.readthedocs.io/en/latest/installation.html#mpi-support)
   * [Caching](https://executorlib.readthedocs.io/en/latest/installation.html#caching)
-  * [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-submission-mode)
-  * [HPC Allocation Mode](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-allocation-mode)
+  * [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-cluster-executor)
+  * [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-job-executor)
   * [Visualisation](https://executorlib.readthedocs.io/en/latest/installation.html#visualisation)
   * [For Developers](https://executorlib.readthedocs.io/en/latest/installation.html#for-developers)
-* [Local Mode](https://executorlib.readthedocs.io/en/latest/1-local.html)
-  * [Basic Functionality](https://executorlib.readthedocs.io/en/latest/1-local.html#basic-functionality)
-  * [Parallel Functions](https://executorlib.readthedocs.io/en/latest/1-local.html#parallel-functions)
-  * [Performance Optimization](https://executorlib.readthedocs.io/en/latest/1-local.html#performance-optimization)
-* [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html)
-  * [SLURM](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html#slurm)
-  * [Flux](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html#flux)
-* [HPC Allocation Mode](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html)
-  * [SLURM](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html#slurm)
-  * [SLURM with Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html#slurm-with-flux)
-  * [Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html#flux)
+* [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html)
+  * [Basic Functionality](https://executorlib.readthedocs.io/en/latest/1-single-node.html#basic-functionality)
+  * [Parallel Functions](https://executorlib.readthedocs.io/en/latest/1-single-node.html#parallel-functions)
+  * [Performance Optimization](https://executorlib.readthedocs.io/en/latest/1-single-node.html#performance-optimization)
+* [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html)
+  * [SLURM](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#slurm)
+  * [Flux](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#flux)
+* [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html)
+  * [SLURM](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm)
+  * [SLURM with Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux)
+  * [Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#flux)
 * [Trouble Shooting](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html)
   * [Filesystem Usage](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#filesystem-usage)
   * [Firewall Issues](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#firewall-issues)

executorlib-0.2.0/executorlib/__init__.py

@@ -0,0 +1,19 @@
+from executorlib._version import get_versions as _get_versions
+from executorlib.interfaces.flux import (
+    FluxClusterExecutor,
+    FluxJobExecutor,
+)
+from executorlib.interfaces.single import SingleNodeExecutor
+from executorlib.interfaces.slurm import (
+    SlurmClusterExecutor,
+    SlurmJobExecutor,
+)
+
+__version__ = _get_versions()["version"]
+__all__: list = [
+    "FluxJobExecutor",
+    "FluxClusterExecutor",
+    "SingleNodeExecutor",
+    "SlurmJobExecutor",
+    "SlurmClusterExecutor",
+]

{executorlib-0.0.11 → executorlib-0.2.0}/executorlib/_version.py

@@ -8,11 +8,11 @@ import json
 
 version_json = '''
 {
- "date": "2025-02-03T20:15:36+0100",
+ "date": "2025-02-11T12:47:28+0100",
  "dirty": true,
  "error": null,
- "full-revisionid": "11d44cc6f9a0f096fb8b95e72c6b28f3b1994146",
- "version": "0.0.11"
+ "full-revisionid": "0ffd31288952f78be3b0c810eac1890759634d35",
+ "version": "0.2.0"
 }
 '''  # END VERSION_JSON
 

{executorlib-0.0.11 → executorlib-0.2.0}/executorlib/interactive/flux.py

@@ -29,6 +29,8 @@ class FluxPythonSpawner(BaseSpawner):
         cores (int, optional): The number of cores. Defaults to 1.
         threads_per_core (int, optional): The number of threads per base. Defaults to 1.
         gpus_per_core (int, optional): The number of GPUs per base. Defaults to 0.
+        num_nodes (int, optional): The number of compute nodes to use for executing the task. Defaults to None.
+        exclusive (bool): Whether to exclusively reserve the compute nodes, or allow sharing compute notes. Defaults to False.
         openmpi_oversubscribe (bool, optional): Whether to oversubscribe. Defaults to False.
         flux_executor (flux.job.FluxExecutor, optional): The FluxExecutor instance. Defaults to None.
         flux_executor_pmi_mode (str, optional): The PMI option. Defaults to None.
@@ -42,6 +44,8 @@ class FluxPythonSpawner(BaseSpawner):
         cores: int = 1,
         threads_per_core: int = 1,
         gpus_per_core: int = 0,
+        num_nodes: Optional[int] = None,
+        exclusive: bool = False,
         openmpi_oversubscribe: bool = False,
         flux_executor: Optional[flux.job.FluxExecutor] = None,
         flux_executor_pmi_mode: Optional[str] = None,
@@ -55,6 +59,8 @@ class FluxPythonSpawner(BaseSpawner):
         )
         self._threads_per_core = threads_per_core
         self._gpus_per_core = gpus_per_core
+        self._num_nodes = num_nodes
+        self._exclusive = exclusive
         self._flux_executor = flux_executor
         self._flux_executor_pmi_mode = flux_executor_pmi_mode
         self._flux_executor_nesting = flux_executor_nesting
@@ -85,8 +91,8 @@ class FluxPythonSpawner(BaseSpawner):
                 num_tasks=self._cores,
                 cores_per_task=self._threads_per_core,
                 gpus_per_task=self._gpus_per_core,
-                num_nodes=None,
-                exclusive=False,
+                num_nodes=self._num_nodes,
+                exclusive=self._exclusive,
             )
         else:
             jobspec = flux.job.JobspecV1.from_nest_command(
@@ -94,8 +100,8 @@ class FluxPythonSpawner(BaseSpawner):
                 num_slots=self._cores,
                 cores_per_slot=self._threads_per_core,
                 gpus_per_slot=self._gpus_per_core,
-                num_nodes=None,
-                exclusive=False,
+                num_nodes=self._num_nodes,
+                exclusive=self._exclusive,
             )
         jobspec.environment = dict(os.environ)
         if self._flux_executor_pmi_mode is not None:

{executorlib-0.0.11 → executorlib-0.2.0}/executorlib/interactive/shared.py

@@ -483,6 +483,8 @@ def _update_futures_in_input(args: tuple, kwargs: dict) -> tuple[tuple, dict]:
             return arg.result()
         elif isinstance(arg, list):
             return [get_result(arg=el) for el in arg]
+        elif isinstance(arg, dict):
+            return {k: get_result(arg=v) for k, v in arg.items()}
         else:
             return arg
 
@@ -510,6 +512,8 @@ def _get_future_objects_from_input(task_dict: dict):
                 future_lst.append(el)
             elif isinstance(el, list):
                 find_future_in_list(lst=el)
+            elif isinstance(el, dict):
+                find_future_in_list(lst=el.values())
 
     find_future_in_list(lst=task_dict["args"])
     find_future_in_list(lst=task_dict["kwargs"].values())

{executorlib-0.0.11 → executorlib-0.2.0}/executorlib/interactive/slurm.py

@@ -27,6 +27,8 @@ class SrunSpawner(SubprocessSpawner):
         cores: int = 1,
         threads_per_core: int = 1,
         gpus_per_core: int = 0,
+        num_nodes: Optional[int] = None,
+        exclusive: bool = False,
         openmpi_oversubscribe: bool = False,
         slurm_cmd_args: Optional[list[str]] = None,
     ):
@@ -38,6 +40,8 @@ class SrunSpawner(SubprocessSpawner):
             cores (int, optional): The number of cores to use. Defaults to 1.
             threads_per_core (int, optional): The number of threads per core. Defaults to 1.
             gpus_per_core (int, optional): The number of GPUs per core. Defaults to 0.
+            num_nodes (int, optional): The number of compute nodes to use for executing the task. Defaults to None.
+            exclusive (bool): Whether to exclusively reserve the compute nodes, or allow sharing compute notes. Defaults to False.
             openmpi_oversubscribe (bool, optional): Whether to oversubscribe the cores. Defaults to False.
             slurm_cmd_args (list[str], optional): Additional command line arguments. Defaults to [].
         """
@@ -49,6 +53,8 @@ class SrunSpawner(SubprocessSpawner):
         )
         self._gpus_per_core = gpus_per_core
         self._slurm_cmd_args = slurm_cmd_args
+        self._num_nodes = num_nodes
+        self._exclusive = exclusive
 
     def generate_command(self, command_lst: list[str]) -> list[str]:
         """
@@ -65,6 +71,8 @@ class SrunSpawner(SubprocessSpawner):
             cwd=self._cwd,
             threads_per_core=self._threads_per_core,
             gpus_per_core=self._gpus_per_core,
+            num_nodes=self._num_nodes,
+            exclusive=self._exclusive,
             openmpi_oversubscribe=self._openmpi_oversubscribe,
             slurm_cmd_args=self._slurm_cmd_args,
         )
@@ -78,6 +86,8 @@ def generate_slurm_command(
     cwd: Optional[str],
     threads_per_core: int = 1,
     gpus_per_core: int = 0,
+    num_nodes: Optional[int] = None,
+    exclusive: bool = False,
     openmpi_oversubscribe: bool = False,
     slurm_cmd_args: Optional[list[str]] = None,
 ) -> list[str]:
@@ -89,6 +99,8 @@ def generate_slurm_command(
         cwd (str): The current working directory.
         threads_per_core (int, optional): The number of threads per core. Defaults to 1.
         gpus_per_core (int, optional): The number of GPUs per core. Defaults to 0.
+        num_nodes (int, optional): The number of compute nodes to use for executing the task. Defaults to None.
+        exclusive (bool): Whether to exclusively reserve the compute nodes, or allow sharing compute notes. Defaults to False.
         openmpi_oversubscribe (bool, optional): Whether to oversubscribe the cores. Defaults to False.
         slurm_cmd_args (list[str], optional): Additional command line arguments. Defaults to [].
 
@@ -98,10 +110,14 @@ def generate_slurm_command(
     command_prepend_lst = [SLURM_COMMAND, "-n", str(cores)]
     if cwd is not None:
         command_prepend_lst += ["-D", cwd]
+    if num_nodes is not None:
+        command_prepend_lst += ["-N", str(num_nodes)]
     if threads_per_core > 1:
-        command_prepend_lst += ["--cpus-per-task" + str(threads_per_core)]
+        command_prepend_lst += ["--cpus-per-task=" + str(threads_per_core)]
     if gpus_per_core > 0:
         command_prepend_lst += ["--gpus-per-task=" + str(gpus_per_core)]
+    if exclusive:
+        command_prepend_lst += ["--exact"]
     if openmpi_oversubscribe:
         command_prepend_lst += ["--oversubscribe"]
     if slurm_cmd_args is not None and len(slurm_cmd_args) > 0: