executorlib 1.9.3__tar.gz → 1.10.0__tar.gz

{executorlib-1.9.3 → executorlib-1.10.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: executorlib
-Version: 1.9.3
+Version: 1.10.0
 Summary: Up-scale python functions for high performance computing (HPC) with executorlib.
 Project-URL: Homepage, https://github.com/pyiron/executorlib
 Project-URL: Documentation, https://executorlib.readthedocs.io
@@ -36,7 +36,7 @@ License: BSD 3-Clause License
         OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
         OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 License-File: LICENSE
-Keywords: executor,flux-framework,high performance computing,hpc,slurm,task scheduler
+Keywords: concurrent.futures,distributed computing,executor,flux-framework,gpu,high performance computing,hpc,mpi4py,parallel computing,python,scientific computing,slurm,task scheduler
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Science/Research
 Classifier: License :: OSI Approved :: BSD License
@@ -52,16 +52,16 @@ Requires-Dist: pyzmq<=27.1.0,>=25.0.0
 Provides-Extra: all
 Requires-Dist: h5py<=3.16.0,>=3.6.0; extra == 'all'
 Requires-Dist: ipython<=9.9.0,>=7.33.0; extra == 'all'
-Requires-Dist: mpi4py<=4.1.1,>=3.1.4; extra == 'all'
+Requires-Dist: mpi4py<=4.1.2,>=3.1.4; extra == 'all'
 Requires-Dist: networkx<=3.6.1,>=2.8.8; extra == 'all'
 Requires-Dist: pydantic<=2.13.4,>=2.5.3; extra == 'all'
 Requires-Dist: pygraphviz<=1.14,>=1.10; extra == 'all'
-Requires-Dist: pysqa==0.4.1; extra == 'all'
+Requires-Dist: pysqa==0.4.2; extra == 'all'
 Provides-Extra: cache
 Requires-Dist: h5py<=3.16.0,>=3.6.0; extra == 'cache'
 Provides-Extra: cluster
 Requires-Dist: h5py<=3.16.0,>=3.6.0; extra == 'cluster'
-Requires-Dist: pysqa==0.4.1; extra == 'cluster'
+Requires-Dist: pysqa==0.4.2; extra == 'cluster'
 Provides-Extra: graph
 Requires-Dist: networkx<=3.6.1,>=2.8.8; extra == 'graph'
 Requires-Dist: pygraphviz<=1.14,>=1.10; extra == 'graph'
@@ -70,7 +70,7 @@ Requires-Dist: ipython<=9.9.0,>=7.33.0; extra == 'graphnotebook'
 Requires-Dist: networkx<=3.6.1,>=2.8.8; extra == 'graphnotebook'
 Requires-Dist: pygraphviz<=1.14,>=1.10; extra == 'graphnotebook'
 Provides-Extra: mpi
-Requires-Dist: mpi4py<=4.1.1,>=3.1.4; extra == 'mpi'
+Requires-Dist: mpi4py<=4.1.2,>=3.1.4; extra == 'mpi'
 Provides-Extra: validation
 Requires-Dist: pydantic<=2.13.4,>=2.5.3; extra == 'validation'
 Description-Content-Type: text/markdown
@@ -96,6 +96,26 @@ Up-scale python functions for high performance computing (HPC) with executorlib.
   machine learning pipelines and simulation workflows executorlib provides optional caching of intermediate results for 
   iterative development in interactive environments like jupyter notebooks.
 
+## Why executorlib?
+executorlib is the lightest path to take *existing* Python functions and scale them across high performance computing
+(HPC) nodes — with per-function-call resource control and native [SLURM](https://slurm.schedmd.com) and
+[flux](http://flux-framework.org) integration — without rewriting your code into a new paradigm. It extends the standard
+library [Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects) you already
+know, rather than asking you to adopt a new data, actor, or workflow model.
+
+| | executorlib | [Futures](https://docs.python.org/3/library/concurrent.futures.html) | [Dask](https://www.dask.org) | [Parsl](https://parsl-project.org) | [Ray](https://www.ray.io) |
+|---|---|---|---|---|---|
+| Drop-in `Executor` API | ✅ | ✅ | ⚠️ | ⚠️  | ❌ |
+| Per-call resource assignment | ✅ | ❌ | ⚠️ | ✅ | ✅ |
+| Native HPC scheduler (SLURM/flux) | ✅ | ❌ | ⚠️ | ✅ | ⚠️ |
+| MPI-parallel functions | ✅ | ❌ | ⚠️ | ⚠️ | ⚠️ |
+| Caching of results | ✅ | ❌ | ⚠️ | ✅ | ❌ |
+| Setup / learning overhead | Low | Very low | Medium | Medium | Medium |
+
+✅ first-class · ⚠️ possible via add-on/config · ❌ not supported. See the full
+[comparison: when to use which](https://executorlib.readthedocs.io/en/latest/comparison.html) for honest guidance on
+when another tool is the better fit.
+
 ## Examples
 The Python standard library provides the [Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects)
 with the [ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) and the 
@@ -198,6 +218,25 @@ given allocation. Even when [SLURM](https://slurm.schedmd.com) is used as primar
 recommended to use [SLURM with flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux) 
 as hierarchical job scheduler within the allocations. 
 
+## Which Executor should I use?
+executorlib provides five `Executor` classes. They all share the same `submit()` / `map()` interface and only differ in
+*where* the Python functions are executed and *how* the resources are requested. A common workflow is to develop and 
+test with the `SingleNodeExecutor` on a laptop and then switch to one of the HPC executors by changing only the class 
+name:
+
+| Executor | Where it runs | Scheduler command | Best for |
+|---|---|---|---|
+| [`SingleNodeExecutor`](https://executorlib.readthedocs.io/en/latest/1-single-node.html) | laptop, workstation or single compute node | `subprocess` | developing and testing a workflow |
+| [`SlurmClusterExecutor`](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#slurm) | HPC login node | `sbatch` (one job per function) | long-running functions that should outlive the Python session |
+| [`SlurmJobExecutor`](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm) | inside a SLURM allocation | `srun` (job steps) | many functions within one existing allocation |
+| [`FluxClusterExecutor`](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#flux) | HPC login node or Flux instance | `flux submit` | long-running functions; disconnecting and reconnecting |
+| [`FluxJobExecutor`](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#flux) | inside a Flux allocation | `flux run` | high-throughput execution of many short functions |
+
+The **Cluster** executors submit each Python function as an individual job and communicate via the file system, so the
+Python process which created the executor can be closed and the results reloaded later. The **Job** executors run inside
+an existing allocation and communicate via sockets, which has lower overhead and is the better choice for many short
+function calls.
+
 ## Documentation
 * [Installation](https://executorlib.readthedocs.io/en/latest/installation.html)
   * [Minimal](https://executorlib.readthedocs.io/en/latest/installation.html#minimal)
@@ -207,6 +246,13 @@ as hierarchical job scheduler within the allocations.
   * [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-job-executor)
   * [Visualisation](https://executorlib.readthedocs.io/en/latest/installation.html#visualisation)
   * [For Developers](https://executorlib.readthedocs.io/en/latest/installation.html#for-developers)
+* [Comparison](https://executorlib.readthedocs.io/en/latest/comparison.html#comparison)
+  * [At a glace](https://executorlib.readthedocs.io/en/latest/comparison.html#at-a-glance)
+  * [Concurrent futures](https://executorlib.readthedocs.io/en/latest/comparison.html#concurrent-futures-the-python-standard-library)
+  * [Dask](https://executorlib.readthedocs.io/en/latest/comparison.html#dask)
+  * [Parsl](https://executorlib.readthedocs.io/en/latest/comparison.html#parsl)
+  * [Ray](https://executorlib.readthedocs.io/en/latest/comparison.html#ray)
+  * [Choose executorlib when](https://executorlib.readthedocs.io/en/latest/comparison.html#choose-executorlib-when)
 * [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html)
   * [Basic Functionality](https://executorlib.readthedocs.io/en/latest/1-single-node.html#basic-functionality)
   * [Parallel Functions](https://executorlib.readthedocs.io/en/latest/1-single-node.html#parallel-functions)
@@ -216,6 +262,9 @@ as hierarchical job scheduler within the allocations.
 * [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html)
   * [SLURM](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#slurm)
   * [Flux](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#flux)
+  * [Disconnecting and Reconnecting](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#disconnecting-and-reconnecting)
+  * [Combine both](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#combine-both)
+  * [Cleaning Cache](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#cleaning-cache)
 * [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html)
   * [SLURM](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm)
   * [SLURM with Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux)
@@ -223,6 +272,12 @@ as hierarchical job scheduler within the allocations.
 * [Application](https://executorlib.readthedocs.io/en/latest/application.html)
   * [GPAW](https://executorlib.readthedocs.io/en/latest/4-1-gpaw.html)
   * [Quantum Espresso](https://executorlib.readthedocs.io/en/latest/4-2-quantum-espresso.html)
+* [Integration](https://executorlib.readthedocs.io/en/latest/coupling.html)
+  * [emcee](https://executorlib.readthedocs.io/en/latest/coupling.html#emcee)
+  * [pipefunc](https://executorlib.readthedocs.io/en/latest/coupling.html#pipefunc)
+  * [omp4py](https://executorlib.readthedocs.io/en/latest/coupling.html#omp4py)
+  * [pylammpsmpi](https://executorlib.readthedocs.io/en/latest/coupling.html#pylammpsmpi)
+  * [General Pattern](https://executorlib.readthedocs.io/en/latest/coupling.html#general-pattern)
 * [Trouble Shooting](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html)
   * [Filesystem Usage](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#filesystem-usage)
   * [Firewall Issues](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#firewall-issues)

{executorlib-1.9.3 → executorlib-1.10.0}/README.md

@@ -19,6 +19,26 @@ Up-scale python functions for high performance computing (HPC) with executorlib.
   machine learning pipelines and simulation workflows executorlib provides optional caching of intermediate results for 
   iterative development in interactive environments like jupyter notebooks.
 
+## Why executorlib?
+executorlib is the lightest path to take *existing* Python functions and scale them across high performance computing
+(HPC) nodes — with per-function-call resource control and native [SLURM](https://slurm.schedmd.com) and
+[flux](http://flux-framework.org) integration — without rewriting your code into a new paradigm. It extends the standard
+library [Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects) you already
+know, rather than asking you to adopt a new data, actor, or workflow model.
+
+| | executorlib | [Futures](https://docs.python.org/3/library/concurrent.futures.html) | [Dask](https://www.dask.org) | [Parsl](https://parsl-project.org) | [Ray](https://www.ray.io) |
+|---|---|---|---|---|---|
+| Drop-in `Executor` API | ✅ | ✅ | ⚠️ | ⚠️  | ❌ |
+| Per-call resource assignment | ✅ | ❌ | ⚠️ | ✅ | ✅ |
+| Native HPC scheduler (SLURM/flux) | ✅ | ❌ | ⚠️ | ✅ | ⚠️ |
+| MPI-parallel functions | ✅ | ❌ | ⚠️ | ⚠️ | ⚠️ |
+| Caching of results | ✅ | ❌ | ⚠️ | ✅ | ❌ |
+| Setup / learning overhead | Low | Very low | Medium | Medium | Medium |
+
+✅ first-class · ⚠️ possible via add-on/config · ❌ not supported. See the full
+[comparison: when to use which](https://executorlib.readthedocs.io/en/latest/comparison.html) for honest guidance on
+when another tool is the better fit.
+
 ## Examples
 The Python standard library provides the [Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects)
 with the [ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) and the 
@@ -121,6 +141,25 @@ given allocation. Even when [SLURM](https://slurm.schedmd.com) is used as primar
 recommended to use [SLURM with flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux) 
 as hierarchical job scheduler within the allocations. 
 
+## Which Executor should I use?
+executorlib provides five `Executor` classes. They all share the same `submit()` / `map()` interface and only differ in
+*where* the Python functions are executed and *how* the resources are requested. A common workflow is to develop and 
+test with the `SingleNodeExecutor` on a laptop and then switch to one of the HPC executors by changing only the class 
+name:
+
+| Executor | Where it runs | Scheduler command | Best for |
+|---|---|---|---|
+| [`SingleNodeExecutor`](https://executorlib.readthedocs.io/en/latest/1-single-node.html) | laptop, workstation or single compute node | `subprocess` | developing and testing a workflow |
+| [`SlurmClusterExecutor`](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#slurm) | HPC login node | `sbatch` (one job per function) | long-running functions that should outlive the Python session |
+| [`SlurmJobExecutor`](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm) | inside a SLURM allocation | `srun` (job steps) | many functions within one existing allocation |
+| [`FluxClusterExecutor`](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#flux) | HPC login node or Flux instance | `flux submit` | long-running functions; disconnecting and reconnecting |
+| [`FluxJobExecutor`](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#flux) | inside a Flux allocation | `flux run` | high-throughput execution of many short functions |
+
+The **Cluster** executors submit each Python function as an individual job and communicate via the file system, so the
+Python process which created the executor can be closed and the results reloaded later. The **Job** executors run inside
+an existing allocation and communicate via sockets, which has lower overhead and is the better choice for many short
+function calls.
+
 ## Documentation
 * [Installation](https://executorlib.readthedocs.io/en/latest/installation.html)
   * [Minimal](https://executorlib.readthedocs.io/en/latest/installation.html#minimal)
@@ -130,6 +169,13 @@ as hierarchical job scheduler within the allocations.
   * [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-job-executor)
   * [Visualisation](https://executorlib.readthedocs.io/en/latest/installation.html#visualisation)
   * [For Developers](https://executorlib.readthedocs.io/en/latest/installation.html#for-developers)
+* [Comparison](https://executorlib.readthedocs.io/en/latest/comparison.html#comparison)
+  * [At a glace](https://executorlib.readthedocs.io/en/latest/comparison.html#at-a-glance)
+  * [Concurrent futures](https://executorlib.readthedocs.io/en/latest/comparison.html#concurrent-futures-the-python-standard-library)
+  * [Dask](https://executorlib.readthedocs.io/en/latest/comparison.html#dask)
+  * [Parsl](https://executorlib.readthedocs.io/en/latest/comparison.html#parsl)
+  * [Ray](https://executorlib.readthedocs.io/en/latest/comparison.html#ray)
+  * [Choose executorlib when](https://executorlib.readthedocs.io/en/latest/comparison.html#choose-executorlib-when)
 * [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html)
   * [Basic Functionality](https://executorlib.readthedocs.io/en/latest/1-single-node.html#basic-functionality)
   * [Parallel Functions](https://executorlib.readthedocs.io/en/latest/1-single-node.html#parallel-functions)
@@ -139,6 +185,9 @@ as hierarchical job scheduler within the allocations.
 * [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html)
   * [SLURM](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#slurm)
   * [Flux](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#flux)
+  * [Disconnecting and Reconnecting](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#disconnecting-and-reconnecting)
+  * [Combine both](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#combine-both)
+  * [Cleaning Cache](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#cleaning-cache)
 * [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html)
   * [SLURM](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm)
   * [SLURM with Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux)
@@ -146,6 +195,12 @@ as hierarchical job scheduler within the allocations.
 * [Application](https://executorlib.readthedocs.io/en/latest/application.html)
   * [GPAW](https://executorlib.readthedocs.io/en/latest/4-1-gpaw.html)
   * [Quantum Espresso](https://executorlib.readthedocs.io/en/latest/4-2-quantum-espresso.html)
+* [Integration](https://executorlib.readthedocs.io/en/latest/coupling.html)
+  * [emcee](https://executorlib.readthedocs.io/en/latest/coupling.html#emcee)
+  * [pipefunc](https://executorlib.readthedocs.io/en/latest/coupling.html#pipefunc)
+  * [omp4py](https://executorlib.readthedocs.io/en/latest/coupling.html#omp4py)
+  * [pylammpsmpi](https://executorlib.readthedocs.io/en/latest/coupling.html#pylammpsmpi)
+  * [General Pattern](https://executorlib.readthedocs.io/en/latest/coupling.html#general-pattern)
 * [Trouble Shooting](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html)
   * [Filesystem Usage](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#filesystem-usage)
   * [Firewall Issues](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#firewall-issues)

{executorlib-1.9.3 → executorlib-1.10.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [build-system]
 requires = [
-    "hatchling>=1.27.0,<=1.29.0",
+    "hatchling>=1.27.0,<=1.30.1",
     "hatch-vcs==0.5.0",
     "cloudpickle>=2.0.0,<=3.1.2",
     "pyzmq>=25.0.0,<=27.1.0",
@@ -15,7 +15,7 @@ authors = [
 ]
 readme = "README.md"
 license = { file = "LICENSE" }
-keywords = ["high performance computing", "hpc", "task scheduler", "slurm", "flux-framework", "executor"]
+keywords = ["high performance computing", "hpc", "task scheduler", "slurm", "flux-framework", "executor", "python", "parallel computing", "distributed computing", "scientific computing", "concurrent.futures", "mpi4py", "gpu"]
 requires-python = ">3.9, <3.14"
 classifiers = [
     "Development Status :: 5 - Production/Stable",
@@ -50,17 +50,17 @@ graphnotebook = [
     "networkx>=2.8.8,<=3.6.1",
     "ipython>=7.33.0,<=9.9.0",
 ]
-mpi = ["mpi4py>=3.1.4,<=4.1.1"]
+mpi = ["mpi4py>=3.1.4,<=4.1.2"]
 cluster = [
-    "pysqa==0.4.1",
+    "pysqa==0.4.2",
     "h5py>=3.6.0,<=3.16.0",
 ]
 validation = [
     "pydantic>=2.5.3,<=2.13.4",
 ]
 all = [
-    "mpi4py>=3.1.4,<=4.1.1",
-    "pysqa==0.4.1",
+    "mpi4py>=3.1.4,<=4.1.2",
+    "pysqa==0.4.2",
     "h5py>=3.6.0,<=3.16.0",
     "pydantic>=2.5.3,<=2.13.4",
     "pygraphviz>=1.10,<=1.14",

executorlib-1.10.0/src/executorlib/_version.py

@@ -0,0 +1,24 @@
+# file generated by vcs-versioning
+# don't change, don't track in version control
+from __future__ import annotations
+
+__all__ = [
+    "__version__",
+    "__version_tuple__",
+    "version",
+    "version_tuple",
+    "__commit_id__",
+    "commit_id",
+]
+
+version: str
+__version__: str
+__version_tuple__: tuple[int | str, ...]
+version_tuple: tuple[int | str, ...]
+commit_id: str | None
+__commit_id__: str | None
+
+__version__ = version = '1.10.0'
+__version_tuple__ = version_tuple = (1, 10, 0)
+
+__commit_id__ = commit_id = None

{executorlib-1.9.3 → executorlib-1.10.0}/src/executorlib/executor/base.py

@@ -25,10 +25,22 @@ class BaseExecutor(FutureExecutor, ABC):
 
     @property
     def max_workers(self) -> Optional[int]:
+        """
+        Return the number of parallel workers configured for this executor.
+
+        Returns:
+            Optional[int]: The maximum number of parallel workers, or None if unconstrained.
+        """
         return self._task_scheduler.max_workers
 
     @max_workers.setter
     def max_workers(self, max_workers: int):
+        """
+        Set the number of parallel workers on the underlying task scheduler.
+
+        Args:
+            max_workers (int): New maximum number of parallel workers.
+        """
         self._task_scheduler.max_workers = max_workers
 
     @property

executorlib-1.10.0/src/executorlib/standalone/batched.py

@@ -0,0 +1,39 @@
+from concurrent.futures import Future
+
+
+def batched_futures(
+    lst: list[Future], nested_skip_lst: list[Future[list]], n: int
+) -> tuple[bool, list[Future]]:
+    """
+    Batch n completed future objects. If the number of completed futures is smaller than n and the end of the batch is
+    not reached yet, then an empty list is returned. If n future objects are done, which are not included in the skip_set
+    then they are returned as batch.
+
+    Args:
+        lst (list): list of all future objects
+        nested_skip_lst (list): list of future objects, which contain the list of future objects ids which should be skipped for the batch
+        n (int): batch size
+
+    Returns:
+        list: results of the batched futures
+    """
+    skip_set = {fid for f in nested_skip_lst for fid in f.result()}
+
+    done_lst = []
+    failed_lst = []
+    n_expected = min(n, len(lst) - len(skip_set))
+    for v in lst:
+        if id(v) not in skip_set and v.done():
+            if v.exception() is not None:
+                failed_lst.append(v)
+            elif id(v) not in skip_set and v.done():
+                done_lst.append(v)
+                if len(done_lst) == n_expected:
+                    return True, done_lst
+    if (len(lst) - len(skip_set)) == len(failed_lst):
+        return (
+            False,
+            failed_lst[:n_expected],
+        )  # raise the exception only after all futures have failed
+    else:
+        return True, []

{executorlib-1.9.3 → executorlib-1.10.0}/src/executorlib/standalone/command.py

@@ -153,7 +153,7 @@ def generate_slurm_command(
         command_prepend_lst += ["--mpi=" + pmi_mode]
     if num_nodes is not None:
         command_prepend_lst += ["-N", str(num_nodes)]
-    if threads_per_core > 1:
+    if threads_per_core >= 1:
         command_prepend_lst += ["--cpus-per-task=" + str(threads_per_core)]
     if gpus_per_core > 0:
         command_prepend_lst += ["--gpus-per-task=" + str(gpus_per_core)]

{executorlib-1.9.3 → executorlib-1.10.0}/src/executorlib/standalone/hdf.py

@@ -1,3 +1,4 @@
+import contextlib
 import os
 from concurrent.futures import Future
 from time import sleep
@@ -34,10 +35,14 @@ def dump(file_name: Optional[str], data_dict: dict) -> None:
         with h5py.File(file_name_abs, "a") as fname:
             for data_key, data_value in data_dict.items():
                 if data_key in group_dict:
-                    fname.create_dataset(
-                        name="/" + group_dict[data_key],
-                        data=np.void(cloudpickle.dumps(data_value)),
-                    )
+                    with contextlib.suppress(ValueError):
+                        fname.create_dataset(
+                            name="/" + group_dict[data_key],
+                            data=np.frombuffer(
+                                cloudpickle.dumps(data_value), dtype=np.uint8
+                            ),
+                            compression="gzip",
+                        )
 
 
 def load(file_name: str) -> dict:
@@ -53,26 +58,26 @@ def load(file_name: str) -> dict:
     with h5py.File(file_name, "r") as hdf:
         data_dict = {}
         if "function" in hdf:
-            data_dict["fn"] = cloudpickle.loads(np.void(hdf["/function"]))
+            data_dict["fn"] = cloudpickle.loads(hdf["/function"][()].tobytes())
         else:
             raise TypeError("Function not found in HDF5 file.")
         if "input_args" in hdf:
-            data_dict["args"] = cloudpickle.loads(np.void(hdf["/input_args"]))
+            data_dict["args"] = cloudpickle.loads(hdf["/input_args"][()].tobytes())
         else:
             data_dict["args"] = ()
         if "input_kwargs" in hdf:
-            data_dict["kwargs"] = cloudpickle.loads(np.void(hdf["/input_kwargs"]))
+            data_dict["kwargs"] = cloudpickle.loads(hdf["/input_kwargs"][()].tobytes())
         else:
             data_dict["kwargs"] = {}
         if "resource_dict" in hdf:
             data_dict["resource_dict"] = cloudpickle.loads(
-                np.void(hdf["/resource_dict"])
+                hdf["/resource_dict"][()].tobytes()
             )
         else:
             data_dict["resource_dict"] = {}
         if "error_log_file" in hdf:
             data_dict["error_log_file"] = cloudpickle.loads(
-                np.void(hdf["/error_log_file"])
+                hdf["/error_log_file"][()].tobytes()
             )
         return data_dict
 
@@ -91,9 +96,9 @@ def get_output(file_name: str) -> tuple[bool, bool, Any]:
     def get_output_helper(file_name: str) -> tuple[bool, bool, Any]:
         with h5py.File(file_name, "r") as hdf:
             if "output" in hdf:
-                return True, True, cloudpickle.loads(np.void(hdf["/output"]))
+                return True, True, cloudpickle.loads(hdf["/output"][()].tobytes())
             elif "error" in hdf:
-                return True, False, cloudpickle.loads(np.void(hdf["/error"]))
+                return True, False, cloudpickle.loads(hdf["/error"][()].tobytes())
             else:
                 return False, False, None
 
@@ -120,7 +125,7 @@ def get_runtime(file_name: str) -> float:
     """
     with h5py.File(file_name, "r") as hdf:
         if "runtime" in hdf:
-            return cloudpickle.loads(np.void(hdf["/runtime"]))
+            return cloudpickle.loads(hdf["/runtime"][()].tobytes())
         else:
             return 0.0
 
@@ -138,7 +143,7 @@ def get_queue_id(file_name: Optional[str]) -> Optional[int]:
     if file_name is not None and os.path.exists(file_name):
         with h5py.File(file_name, "r") as hdf:
             if "queue_id" in hdf:
-                return cloudpickle.loads(np.void(hdf["/queue_id"]))
+                return cloudpickle.loads(hdf["/queue_id"][()].tobytes())
     return None
 
 
@@ -171,7 +176,11 @@ def get_cache_files(cache_directory: str) -> list[str]:
     file_lst = []
     cache_directory_abs = os.path.abspath(cache_directory)
     for dirpath, _, filenames in os.walk(cache_directory_abs):
-        file_lst += [os.path.join(dirpath, f) for f in filenames if f.endswith("_o.h5")]
+        file_lst += [
+            os.path.join(dirpath, f)
+            for f in filenames
+            if f.endswith("_o.h5") or f.endswith("_i.h5")
+        ]
     return file_lst
 
 
@@ -224,7 +233,7 @@ def _get_content_of_file(file_name: str) -> dict:
     """
     with h5py.File(file_name, "r") as hdf:
         return {
-            key: cloudpickle.loads(np.void(hdf["/" + key]))
+            key: cloudpickle.loads(hdf["/" + key][()].tobytes())
             for key in group_dict.values()
             if key in hdf
         }

{executorlib-1.9.3 → executorlib-1.10.0}/src/executorlib/standalone/inputcheck.py

@@ -141,6 +141,16 @@ def check_init_function(
 def check_max_workers_and_cores(
     max_workers: Optional[int], max_cores: Optional[int]
 ) -> None:
+    """
+    Check that neither max_workers nor max_cores is set when using a pysqa-based backend.
+
+    Args:
+        max_workers (int, optional): Maximum number of workers.
+        max_cores (int, optional): Maximum number of cores.
+
+    Raises:
+        ValueError: If max_workers or max_cores is not None.
+    """
     if max_workers is not None:
         raise ValueError(
             "The number of workers cannot be controlled with the pysqa based backend."
@@ -152,6 +162,15 @@ def check_max_workers_and_cores(
 
 
 def check_hostname_localhost(hostname_localhost: Optional[bool]) -> None:
+    """
+    Check that hostname_localhost is not set when using a pysqa-based backend.
+
+    Args:
+        hostname_localhost (bool, optional): Flag to use localhost for ZMQ connections.
+
+    Raises:
+        ValueError: If hostname_localhost is not None.
+    """
     if hostname_localhost is not None:
         raise ValueError(
             "The option to connect to hosts based on their hostname is not available with the pysqa based backend."
@@ -159,6 +178,16 @@ def check_hostname_localhost(hostname_localhost: Optional[bool]) -> None:
 
 
 def check_restart_limit(restart_limit: int, block_allocation: bool = True) -> None:
+    """
+    Check that restart_limit is only used together with block_allocation.
+
+    Args:
+        restart_limit (int): Maximum number of times a worker process may be restarted.
+        block_allocation (bool): Whether block allocation is enabled. Defaults to True.
+
+    Raises:
+        ValueError: If restart_limit is non-zero and block_allocation is False.
+    """
     if not block_allocation and restart_limit != 0:
         raise ValueError(
             "The option to specify a restart limit for worker processes is only available with block_allocation=True."
@@ -166,6 +195,15 @@ def check_restart_limit(restart_limit: int, block_allocation: bool = True) -> No
 
 
 def check_pmi_mode(pmi_mode: Optional[str]) -> None:
+    """
+    Check that pmi_mode is not set on a local workstation without SLURM or flux.
+
+    Args:
+        pmi_mode (str, optional): PMI interface name (e.g. "pmix", "pmi1", "pmi2").
+
+    Raises:
+        ValueError: If pmi_mode is not None.
+    """
     if pmi_mode is not None:
         raise ValueError(
             "The option to specify the pmi mode is not available on a local workstation, it requires SLURM or flux."
@@ -199,7 +237,22 @@ def validate_number_of_cores(
     set_local_cores: bool = False,
 ) -> int:
     """
-    Validate the number of cores and return the appropriate value.
+    Validate the number of cores and return the number of workers to use.
+
+    Precedence: max_cores / cores_per_worker > max_workers > CPU count (with warning).
+
+    Args:
+        max_cores (int, optional): Total number of cores available.
+        max_workers (int, optional): Explicit number of parallel workers.
+        cores_per_worker (int, optional): Number of cores allocated to each worker. Defaults to 1.
+        set_local_cores (bool): When True, fall back to the local CPU count if neither
+            max_cores nor max_workers is given instead of raising an error. Defaults to False.
+
+    Returns:
+        int: Number of parallel workers to use.
+
+    Raises:
+        ValueError: If neither max_cores nor max_workers is set and set_local_cores is False.
     """
     if max_cores is not None and max_workers is None and cores_per_worker is not None:
         return int(max_cores / cores_per_worker)

{executorlib-1.9.3 → executorlib-1.10.0}/src/executorlib/standalone/interactive/arguments.py

@@ -27,10 +27,21 @@ def get_future_objects_from_input(args: tuple, kwargs: dict):
 
     find_future_in_list(lst=args)
     find_future_in_list(lst=kwargs.values())
-    boolean_flag = len([future for future in future_lst if future.done()]) == len(
-        future_lst
-    )
-    return future_lst, boolean_flag
+
+    return future_lst
+
+
+def check_list_of_futures_is_done(future_lst: list[Future]) -> bool:
+    """
+    Check if all future objects in the list of future objects are done
+
+    Args:
+        future_lst (list): list of future objects
+
+    Returns:
+        bool: True if all future objects in the list of future objects are done, False otherwise
+    """
+    return len([future for future in future_lst if future.done()]) == len(future_lst)
 
 
 def get_exception_lst(future_lst: list[Future]) -> list:

{executorlib-1.9.3 → executorlib-1.10.0}/src/executorlib/standalone/interactive/communication.py

@@ -240,6 +240,9 @@ def interface_connect(host: str, port: str) -> tuple[zmq.Context, zmq.Socket]:
     Args:
         host (str): hostname of the host running the SocketInterface instance to connect to.
         port (str): port on the host the SocketInterface instance is running on.
+
+    Returns:
+        Tuple[zmq.Context, zmq.Socket]: The ZMQ context and the connected PAIR socket.
     """
     context = zmq.Context()
     socket = context.socket(zmq.PAIR)
@@ -265,6 +268,9 @@ def interface_receive(socket: Optional[zmq.Socket]) -> dict:
 
     Args:
         socket (zmq.Socket): socket for the connection
+
+    Returns:
+        dict: Deserialized dictionary received from the socket, or an empty dict if socket is None.
     """
     if socket is not None:
         return cloudpickle.loads(socket.recv())