executorlib 1.9.3__tar.gz → 1.9.4__tar.gz

{executorlib-1.9.3 → executorlib-1.9.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: executorlib
-Version: 1.9.3
+Version: 1.9.4
 Summary: Up-scale python functions for high performance computing (HPC) with executorlib.
 Project-URL: Homepage, https://github.com/pyiron/executorlib
 Project-URL: Documentation, https://executorlib.readthedocs.io
@@ -36,7 +36,7 @@ License: BSD 3-Clause License
         OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
         OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 License-File: LICENSE
-Keywords: executor,flux-framework,high performance computing,hpc,slurm,task scheduler
+Keywords: concurrent.futures,distributed computing,executor,flux-framework,gpu,high performance computing,hpc,mpi4py,parallel computing,python,scientific computing,slurm,task scheduler
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Science/Research
 Classifier: License :: OSI Approved :: BSD License
@@ -52,16 +52,16 @@ Requires-Dist: pyzmq<=27.1.0,>=25.0.0
 Provides-Extra: all
 Requires-Dist: h5py<=3.16.0,>=3.6.0; extra == 'all'
 Requires-Dist: ipython<=9.9.0,>=7.33.0; extra == 'all'
-Requires-Dist: mpi4py<=4.1.1,>=3.1.4; extra == 'all'
+Requires-Dist: mpi4py<=4.1.2,>=3.1.4; extra == 'all'
 Requires-Dist: networkx<=3.6.1,>=2.8.8; extra == 'all'
 Requires-Dist: pydantic<=2.13.4,>=2.5.3; extra == 'all'
 Requires-Dist: pygraphviz<=1.14,>=1.10; extra == 'all'
-Requires-Dist: pysqa==0.4.1; extra == 'all'
+Requires-Dist: pysqa==0.4.2; extra == 'all'
 Provides-Extra: cache
 Requires-Dist: h5py<=3.16.0,>=3.6.0; extra == 'cache'
 Provides-Extra: cluster
 Requires-Dist: h5py<=3.16.0,>=3.6.0; extra == 'cluster'
-Requires-Dist: pysqa==0.4.1; extra == 'cluster'
+Requires-Dist: pysqa==0.4.2; extra == 'cluster'
 Provides-Extra: graph
 Requires-Dist: networkx<=3.6.1,>=2.8.8; extra == 'graph'
 Requires-Dist: pygraphviz<=1.14,>=1.10; extra == 'graph'
@@ -70,7 +70,7 @@ Requires-Dist: ipython<=9.9.0,>=7.33.0; extra == 'graphnotebook'
 Requires-Dist: networkx<=3.6.1,>=2.8.8; extra == 'graphnotebook'
 Requires-Dist: pygraphviz<=1.14,>=1.10; extra == 'graphnotebook'
 Provides-Extra: mpi
-Requires-Dist: mpi4py<=4.1.1,>=3.1.4; extra == 'mpi'
+Requires-Dist: mpi4py<=4.1.2,>=3.1.4; extra == 'mpi'
 Provides-Extra: validation
 Requires-Dist: pydantic<=2.13.4,>=2.5.3; extra == 'validation'
 Description-Content-Type: text/markdown
@@ -96,6 +96,26 @@ Up-scale python functions for high performance computing (HPC) with executorlib.
   machine learning pipelines and simulation workflows executorlib provides optional caching of intermediate results for 
   iterative development in interactive environments like jupyter notebooks.
 
+## Why executorlib?
+executorlib is the lightest path to take *existing* Python functions and scale them across high performance computing
+(HPC) nodes — with per-function-call resource control and native [SLURM](https://slurm.schedmd.com) and
+[flux](http://flux-framework.org) integration — without rewriting your code into a new paradigm. It extends the standard
+library [Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects) you already
+know, rather than asking you to adopt a new data, actor, or workflow model.
+
+| | executorlib | [Futures](https://docs.python.org/3/library/concurrent.futures.html) | [Dask](https://www.dask.org) | [Parsl](https://parsl-project.org) | [Ray](https://www.ray.io) |
+|---|---|---|---|---|---|
+| Drop-in `Executor` API | ✅ | ✅ | ⚠️ | ⚠️  | ❌ |
+| Per-call resource assignment | ✅ | ❌ | ⚠️ | ✅ | ✅ |
+| Native HPC scheduler (SLURM/flux) | ✅ | ❌ | ⚠️ | ✅ | ⚠️ |
+| MPI-parallel functions | ✅ | ❌ | ⚠️ | ⚠️ | ⚠️ |
+| Caching of results | ✅ | ❌ | ⚠️ | ✅ | ❌ |
+| Setup / learning overhead | Low | Very low | Medium | Medium | Medium |
+
+✅ first-class · ⚠️ possible via add-on/config · ❌ not supported. See the full
+[comparison: when to use which](https://executorlib.readthedocs.io/en/latest/comparison.html) for honest guidance on
+when another tool is the better fit.
+
 ## Examples
 The Python standard library provides the [Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects)
 with the [ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) and the 
@@ -207,6 +227,13 @@ as hierarchical job scheduler within the allocations.
   * [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-job-executor)
   * [Visualisation](https://executorlib.readthedocs.io/en/latest/installation.html#visualisation)
   * [For Developers](https://executorlib.readthedocs.io/en/latest/installation.html#for-developers)
+* [Comparison](https://executorlib.readthedocs.io/en/latest/comparison.html#comparison)
+  * [At a glace](https://executorlib.readthedocs.io/en/latest/comparison.html#at-a-glance)
+  * [Concurrent futures](https://executorlib.readthedocs.io/en/latest/comparison.html#concurrent-futures-the-python-standard-library)
+  * [Dask](https://executorlib.readthedocs.io/en/latest/comparison.html#dask)
+  * [Parsl](https://executorlib.readthedocs.io/en/latest/comparison.html#parsl)
+  * [Ray](https://executorlib.readthedocs.io/en/latest/comparison.html#ray)
+  * [Choose executorlib when](https://executorlib.readthedocs.io/en/latest/comparison.html#choose-executorlib-when)
 * [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html)
   * [Basic Functionality](https://executorlib.readthedocs.io/en/latest/1-single-node.html#basic-functionality)
   * [Parallel Functions](https://executorlib.readthedocs.io/en/latest/1-single-node.html#parallel-functions)

{executorlib-1.9.3 → executorlib-1.9.4}/README.md

@@ -19,6 +19,26 @@ Up-scale python functions for high performance computing (HPC) with executorlib.
   machine learning pipelines and simulation workflows executorlib provides optional caching of intermediate results for 
   iterative development in interactive environments like jupyter notebooks.
 
+## Why executorlib?
+executorlib is the lightest path to take *existing* Python functions and scale them across high performance computing
+(HPC) nodes — with per-function-call resource control and native [SLURM](https://slurm.schedmd.com) and
+[flux](http://flux-framework.org) integration — without rewriting your code into a new paradigm. It extends the standard
+library [Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects) you already
+know, rather than asking you to adopt a new data, actor, or workflow model.
+
+| | executorlib | [Futures](https://docs.python.org/3/library/concurrent.futures.html) | [Dask](https://www.dask.org) | [Parsl](https://parsl-project.org) | [Ray](https://www.ray.io) |
+|---|---|---|---|---|---|
+| Drop-in `Executor` API | ✅ | ✅ | ⚠️ | ⚠️  | ❌ |
+| Per-call resource assignment | ✅ | ❌ | ⚠️ | ✅ | ✅ |
+| Native HPC scheduler (SLURM/flux) | ✅ | ❌ | ⚠️ | ✅ | ⚠️ |
+| MPI-parallel functions | ✅ | ❌ | ⚠️ | ⚠️ | ⚠️ |
+| Caching of results | ✅ | ❌ | ⚠️ | ✅ | ❌ |
+| Setup / learning overhead | Low | Very low | Medium | Medium | Medium |
+
+✅ first-class · ⚠️ possible via add-on/config · ❌ not supported. See the full
+[comparison: when to use which](https://executorlib.readthedocs.io/en/latest/comparison.html) for honest guidance on
+when another tool is the better fit.
+
 ## Examples
 The Python standard library provides the [Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects)
 with the [ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) and the 
@@ -130,6 +150,13 @@ as hierarchical job scheduler within the allocations.
   * [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-job-executor)
   * [Visualisation](https://executorlib.readthedocs.io/en/latest/installation.html#visualisation)
   * [For Developers](https://executorlib.readthedocs.io/en/latest/installation.html#for-developers)
+* [Comparison](https://executorlib.readthedocs.io/en/latest/comparison.html#comparison)
+  * [At a glace](https://executorlib.readthedocs.io/en/latest/comparison.html#at-a-glance)
+  * [Concurrent futures](https://executorlib.readthedocs.io/en/latest/comparison.html#concurrent-futures-the-python-standard-library)
+  * [Dask](https://executorlib.readthedocs.io/en/latest/comparison.html#dask)
+  * [Parsl](https://executorlib.readthedocs.io/en/latest/comparison.html#parsl)
+  * [Ray](https://executorlib.readthedocs.io/en/latest/comparison.html#ray)
+  * [Choose executorlib when](https://executorlib.readthedocs.io/en/latest/comparison.html#choose-executorlib-when)
 * [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html)
   * [Basic Functionality](https://executorlib.readthedocs.io/en/latest/1-single-node.html#basic-functionality)
   * [Parallel Functions](https://executorlib.readthedocs.io/en/latest/1-single-node.html#parallel-functions)

{executorlib-1.9.3 → executorlib-1.9.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [build-system]
 requires = [
-    "hatchling>=1.27.0,<=1.29.0",
+    "hatchling>=1.27.0,<=1.30.1",
     "hatch-vcs==0.5.0",
     "cloudpickle>=2.0.0,<=3.1.2",
     "pyzmq>=25.0.0,<=27.1.0",
@@ -15,7 +15,7 @@ authors = [
 ]
 readme = "README.md"
 license = { file = "LICENSE" }
-keywords = ["high performance computing", "hpc", "task scheduler", "slurm", "flux-framework", "executor"]
+keywords = ["high performance computing", "hpc", "task scheduler", "slurm", "flux-framework", "executor", "python", "parallel computing", "distributed computing", "scientific computing", "concurrent.futures", "mpi4py", "gpu"]
 requires-python = ">3.9, <3.14"
 classifiers = [
     "Development Status :: 5 - Production/Stable",
@@ -50,17 +50,17 @@ graphnotebook = [
     "networkx>=2.8.8,<=3.6.1",
     "ipython>=7.33.0,<=9.9.0",
 ]
-mpi = ["mpi4py>=3.1.4,<=4.1.1"]
+mpi = ["mpi4py>=3.1.4,<=4.1.2"]
 cluster = [
-    "pysqa==0.4.1",
+    "pysqa==0.4.2",
     "h5py>=3.6.0,<=3.16.0",
 ]
 validation = [
     "pydantic>=2.5.3,<=2.13.4",
 ]
 all = [
-    "mpi4py>=3.1.4,<=4.1.1",
-    "pysqa==0.4.1",
+    "mpi4py>=3.1.4,<=4.1.2",
+    "pysqa==0.4.2",
     "h5py>=3.6.0,<=3.16.0",
     "pydantic>=2.5.3,<=2.13.4",
     "pygraphviz>=1.10,<=1.14",

executorlib-1.9.4/src/executorlib/_version.py

@@ -0,0 +1,24 @@
+# file generated by vcs-versioning
+# don't change, don't track in version control
+from __future__ import annotations
+
+__all__ = [
+    "__version__",
+    "__version_tuple__",
+    "version",
+    "version_tuple",
+    "__commit_id__",
+    "commit_id",
+]
+
+version: str
+__version__: str
+__version_tuple__: tuple[int | str, ...]
+version_tuple: tuple[int | str, ...]
+commit_id: str | None
+__commit_id__: str | None
+
+__version__ = version = '1.9.4'
+__version_tuple__ = version_tuple = (1, 9, 4)
+
+__commit_id__ = commit_id = None

{executorlib-1.9.3 → executorlib-1.9.4}/src/executorlib/executor/base.py

@@ -25,10 +25,22 @@ class BaseExecutor(FutureExecutor, ABC):
 
     @property
     def max_workers(self) -> Optional[int]:
+        """
+        Return the number of parallel workers configured for this executor.
+
+        Returns:
+            Optional[int]: The maximum number of parallel workers, or None if unconstrained.
+        """
         return self._task_scheduler.max_workers
 
     @max_workers.setter
     def max_workers(self, max_workers: int):
+        """
+        Set the number of parallel workers on the underlying task scheduler.
+
+        Args:
+            max_workers (int): New maximum number of parallel workers.
+        """
         self._task_scheduler.max_workers = max_workers
 
     @property

executorlib-1.9.4/src/executorlib/standalone/batched.py

@@ -0,0 +1,39 @@
+from concurrent.futures import Future
+
+
+def batched_futures(
+    lst: list[Future], nested_skip_lst: list[Future[list]], n: int
+) -> tuple[bool, list[Future]]:
+    """
+    Batch n completed future objects. If the number of completed futures is smaller than n and the end of the batch is
+    not reached yet, then an empty list is returned. If n future objects are done, which are not included in the skip_set
+    then they are returned as batch.
+
+    Args:
+        lst (list): list of all future objects
+        nested_skip_lst (list): list of future objects, which contain the list of future objects ids which should be skipped for the batch
+        n (int): batch size
+
+    Returns:
+        list: results of the batched futures
+    """
+    skip_set = {fid for f in nested_skip_lst for fid in f.result()}
+
+    done_lst = []
+    failed_lst = []
+    n_expected = min(n, len(lst) - len(skip_set))
+    for v in lst:
+        if id(v) not in skip_set and v.done():
+            if v.exception() is not None:
+                failed_lst.append(v)
+            elif id(v) not in skip_set and v.done():
+                done_lst.append(v)
+                if len(done_lst) == n_expected:
+                    return True, done_lst
+    if (len(lst) - len(skip_set)) == len(failed_lst):
+        return (
+            False,
+            failed_lst[:n_expected],
+        )  # raise the exception only after all futures have failed
+    else:
+        return True, []

{executorlib-1.9.3 → executorlib-1.9.4}/src/executorlib/standalone/command.py

@@ -153,7 +153,7 @@ def generate_slurm_command(
         command_prepend_lst += ["--mpi=" + pmi_mode]
     if num_nodes is not None:
         command_prepend_lst += ["-N", str(num_nodes)]
-    if threads_per_core > 1:
+    if threads_per_core >= 1:
         command_prepend_lst += ["--cpus-per-task=" + str(threads_per_core)]
     if gpus_per_core > 0:
         command_prepend_lst += ["--gpus-per-task=" + str(gpus_per_core)]

{executorlib-1.9.3 → executorlib-1.9.4}/src/executorlib/standalone/hdf.py

@@ -1,3 +1,4 @@
+import contextlib
 import os
 from concurrent.futures import Future
 from time import sleep
@@ -34,10 +35,11 @@ def dump(file_name: Optional[str], data_dict: dict) -> None:
         with h5py.File(file_name_abs, "a") as fname:
             for data_key, data_value in data_dict.items():
                 if data_key in group_dict:
-                    fname.create_dataset(
-                        name="/" + group_dict[data_key],
-                        data=np.void(cloudpickle.dumps(data_value)),
-                    )
+                    with contextlib.suppress(ValueError):
+                        fname.create_dataset(
+                            name="/" + group_dict[data_key],
+                            data=np.void(cloudpickle.dumps(data_value)),
+                        )
 
 
 def load(file_name: str) -> dict:
@@ -171,7 +173,11 @@ def get_cache_files(cache_directory: str) -> list[str]:
     file_lst = []
     cache_directory_abs = os.path.abspath(cache_directory)
     for dirpath, _, filenames in os.walk(cache_directory_abs):
-        file_lst += [os.path.join(dirpath, f) for f in filenames if f.endswith("_o.h5")]
+        file_lst += [
+            os.path.join(dirpath, f)
+            for f in filenames
+            if f.endswith("_o.h5") or f.endswith("_i.h5")
+        ]
     return file_lst
 
 

{executorlib-1.9.3 → executorlib-1.9.4}/src/executorlib/standalone/inputcheck.py

@@ -141,6 +141,16 @@ def check_init_function(
 def check_max_workers_and_cores(
     max_workers: Optional[int], max_cores: Optional[int]
 ) -> None:
+    """
+    Check that neither max_workers nor max_cores is set when using a pysqa-based backend.
+
+    Args:
+        max_workers (int, optional): Maximum number of workers.
+        max_cores (int, optional): Maximum number of cores.
+
+    Raises:
+        ValueError: If max_workers or max_cores is not None.
+    """
     if max_workers is not None:
         raise ValueError(
             "The number of workers cannot be controlled with the pysqa based backend."
@@ -152,6 +162,15 @@ def check_max_workers_and_cores(
 
 
 def check_hostname_localhost(hostname_localhost: Optional[bool]) -> None:
+    """
+    Check that hostname_localhost is not set when using a pysqa-based backend.
+
+    Args:
+        hostname_localhost (bool, optional): Flag to use localhost for ZMQ connections.
+
+    Raises:
+        ValueError: If hostname_localhost is not None.
+    """
     if hostname_localhost is not None:
         raise ValueError(
             "The option to connect to hosts based on their hostname is not available with the pysqa based backend."
@@ -159,6 +178,16 @@ def check_hostname_localhost(hostname_localhost: Optional[bool]) -> None:
 
 
 def check_restart_limit(restart_limit: int, block_allocation: bool = True) -> None:
+    """
+    Check that restart_limit is only used together with block_allocation.
+
+    Args:
+        restart_limit (int): Maximum number of times a worker process may be restarted.
+        block_allocation (bool): Whether block allocation is enabled. Defaults to True.
+
+    Raises:
+        ValueError: If restart_limit is non-zero and block_allocation is False.
+    """
     if not block_allocation and restart_limit != 0:
         raise ValueError(
             "The option to specify a restart limit for worker processes is only available with block_allocation=True."
@@ -166,6 +195,15 @@ def check_restart_limit(restart_limit: int, block_allocation: bool = True) -> No
 
 
 def check_pmi_mode(pmi_mode: Optional[str]) -> None:
+    """
+    Check that pmi_mode is not set on a local workstation without SLURM or flux.
+
+    Args:
+        pmi_mode (str, optional): PMI interface name (e.g. "pmix", "pmi1", "pmi2").
+
+    Raises:
+        ValueError: If pmi_mode is not None.
+    """
     if pmi_mode is not None:
         raise ValueError(
             "The option to specify the pmi mode is not available on a local workstation, it requires SLURM or flux."
@@ -199,7 +237,22 @@ def validate_number_of_cores(
     set_local_cores: bool = False,
 ) -> int:
     """
-    Validate the number of cores and return the appropriate value.
+    Validate the number of cores and return the number of workers to use.
+
+    Precedence: max_cores / cores_per_worker > max_workers > CPU count (with warning).
+
+    Args:
+        max_cores (int, optional): Total number of cores available.
+        max_workers (int, optional): Explicit number of parallel workers.
+        cores_per_worker (int, optional): Number of cores allocated to each worker. Defaults to 1.
+        set_local_cores (bool): When True, fall back to the local CPU count if neither
+            max_cores nor max_workers is given instead of raising an error. Defaults to False.
+
+    Returns:
+        int: Number of parallel workers to use.
+
+    Raises:
+        ValueError: If neither max_cores nor max_workers is set and set_local_cores is False.
     """
     if max_cores is not None and max_workers is None and cores_per_worker is not None:
         return int(max_cores / cores_per_worker)

{executorlib-1.9.3 → executorlib-1.9.4}/src/executorlib/standalone/interactive/arguments.py

@@ -27,10 +27,21 @@ def get_future_objects_from_input(args: tuple, kwargs: dict):
 
     find_future_in_list(lst=args)
     find_future_in_list(lst=kwargs.values())
-    boolean_flag = len([future for future in future_lst if future.done()]) == len(
-        future_lst
-    )
-    return future_lst, boolean_flag
+
+    return future_lst
+
+
+def check_list_of_futures_is_done(future_lst: list[Future]) -> bool:
+    """
+    Check if all future objects in the list of future objects are done
+
+    Args:
+        future_lst (list): list of future objects
+
+    Returns:
+        bool: True if all future objects in the list of future objects are done, False otherwise
+    """
+    return len([future for future in future_lst if future.done()]) == len(future_lst)
 
 
 def get_exception_lst(future_lst: list[Future]) -> list:

{executorlib-1.9.3 → executorlib-1.9.4}/src/executorlib/standalone/interactive/communication.py

@@ -240,6 +240,9 @@ def interface_connect(host: str, port: str) -> tuple[zmq.Context, zmq.Socket]:
     Args:
         host (str): hostname of the host running the SocketInterface instance to connect to.
         port (str): port on the host the SocketInterface instance is running on.
+
+    Returns:
+        Tuple[zmq.Context, zmq.Socket]: The ZMQ context and the connected PAIR socket.
     """
     context = zmq.Context()
     socket = context.socket(zmq.PAIR)
@@ -265,6 +268,9 @@ def interface_receive(socket: Optional[zmq.Socket]) -> dict:
 
     Args:
         socket (zmq.Socket): socket for the connection
+
+    Returns:
+        dict: Deserialized dictionary received from the socket, or an empty dict if socket is None.
     """
     if socket is not None:
         return cloudpickle.loads(socket.recv())

{executorlib-1.9.3 → executorlib-1.9.4}/src/executorlib/standalone/select.py

@@ -3,21 +3,51 @@ from typing import Any, Optional
 
 
 class FutureSelector(Future):
+    """
+    A Future wrapper that returns a single element from a result that is a tuple, list, or dict.
+
+    This is used by split_future() and get_item_from_future() to give callers an individual
+    Future-like object for each output of a function that returns a collection.
+
+    Args:
+        future (Future): The underlying future whose result is a collection.
+        selector (int | str): Index (for sequences) or key (for mappings) used to extract
+            the desired element from the result.
+    """
+
     def __init__(self, future: Future, selector: int | str):
+        """
+        Args:
+            future (Future): The underlying future whose result is a collection.
+            selector (int | str): Index or key used to extract an element from the result.
+        """
         self._future = future
         self._selector = selector
         super().__init__()
 
     def __getattr__(self, attr: str) -> Any:
+        """Delegate attribute access to the wrapped future."""
         return getattr(self._future, attr)
 
     def __setattr__(self, name: str, value: Any):
+        """Set attributes on the wrapped future, except for the two private instance variables."""
         if name in ["_future", "_selector"]:
             super().__setattr__(name, value)
         else:
             setattr(self._future, name, value)
 
     def result(self, timeout: Optional[float] = None) -> Any:
+        """
+        Return the selected element from the underlying future's result.
+
+        Args:
+            timeout (float, optional): Maximum seconds to wait for the result. Defaults to None
+                (wait indefinitely).
+
+        Returns:
+            Any: The element at position/key ``selector`` in the underlying result, or None if
+                the underlying result is None.
+        """
         result = self._future.result(timeout=timeout)
         if result is not None:
             return result[self._selector]

{executorlib-1.9.3 → executorlib-1.9.4}/src/executorlib/standalone/serialize.py

@@ -102,6 +102,18 @@ def _get_hash(binary: bytes) -> str:
 
 
 def _get_function_name(fn: Callable) -> str:
+    """
+    Return a human-readable name for a callable.
+
+    For regular functions and methods the ``__name__`` attribute is used. For callable objects
+    (classes implementing ``__call__``) the class name is extracted from ``__class__``.
+
+    Args:
+        fn (Callable): The callable whose name should be determined.
+
+    Returns:
+        str: A short name string suitable for use as part of a cache key prefix.
+    """
     if hasattr(fn, "__name__"):
         return fn.__name__
     else:

executorlib-1.9.4/src/executorlib/standalone/validate.py

@@ -0,0 +1,117 @@
+import warnings
+from typing import Optional
+
+try:
+    from pydantic import BaseModel, Extra
+
+    HAS_PYDANTIC = True
+except ImportError:
+    from dataclasses import dataclass
+
+    BaseModel = object
+    Extra = None
+    HAS_PYDANTIC = False
+
+
+class ResourceDictValidation(BaseModel):
+    """
+    Pydantic (or dataclass fallback) model for validating resource dictionaries passed to task submissions.
+
+    Attributes:
+        cores (int, optional): Number of MPI cores to be used for each function call.
+        threads_per_core (int, optional): Number of OpenMP threads to be used for each function call.
+        gpus_per_core (int, optional): Number of GPUs per worker.
+        cwd (str, optional): Current working directory where the parallel python task is executed.
+        cache_key (str, optional): External cache key to identify tasks on the file system.
+        cache_directory (str, optional): The directory to store cache files.
+        num_nodes (int, optional): Number of compute nodes used for the evaluation.
+        exclusive (bool, optional): Reserve exclusive access to selected compute nodes.
+        error_log_file (str, optional): Path to the error log file.
+        run_time_max (int, optional): Maximum allowed execution time in seconds.
+        priority (int, optional): Queuing system priority for the task.
+        slurm_cmd_args (list[str], optional): Additional command line arguments for the srun call.
+        submission_template (str, optional): Template for queuing system job submission scripts.
+    """
+
+    cores: Optional[int] = None
+    threads_per_core: Optional[int] = None
+    gpus_per_core: Optional[int] = None
+    cwd: Optional[str] = None
+    cache_key: Optional[str] = None
+    cache_directory: Optional[str] = None
+    num_nodes: Optional[int] = None
+    exclusive: Optional[bool] = None
+    error_log_file: Optional[str] = None
+    run_time_max: Optional[int] = None
+    priority: Optional[int] = None
+    slurm_cmd_args: Optional[list[str]] = None
+    submission_template: Optional[str] = None
+
+    if HAS_PYDANTIC:
+
+        class Config:
+            extra = Extra.forbid
+
+
+if not HAS_PYDANTIC:
+    ResourceDictValidation = dataclass(ResourceDictValidation)  # type: ignore
+
+
+def _get_accepted_keys(class_type) -> list[str]:
+    """
+    Return a list of accepted field names from a Pydantic model or dataclass.
+
+    Args:
+        class_type: A Pydantic BaseModel subclass or a dataclass.
+
+    Returns:
+        list[str]: Field names declared on the class.
+
+    Raises:
+        TypeError: If the class type is neither a Pydantic model nor a dataclass.
+    """
+    if hasattr(class_type, "model_fields"):
+        return list(class_type.model_fields.keys())
+    elif hasattr(class_type, "__dataclass_fields__"):
+        return list(class_type.__dataclass_fields__.keys())
+    raise TypeError("Unsupported class type for validation")
+
+
+def validate_resource_dict(resource_dict: dict) -> None:
+    """
+    Validate a resource dictionary against the declared fields of ResourceDictValidation.
+
+    Args:
+        resource_dict (dict): Dictionary of resource requirements to validate. Unknown keys raise
+            a validation error when pydantic is available.
+
+    Raises:
+        ValidationError: If any key/value pair violates the ResourceDictValidation schema.
+    """
+    _ = ResourceDictValidation(**resource_dict)
+
+
+def validate_resource_dict_with_optional_keys(resource_dict: dict) -> None:
+    """
+    Validate a resource dictionary, allowing unknown keys with a warning instead of an error.
+
+    Known keys are validated against ResourceDictValidation. Unknown keys are collected and
+    emitted as a UserWarning so callers can detect unsupported options without hard failure.
+
+    Args:
+        resource_dict (dict): Dictionary of resource requirements to validate.
+
+    Raises:
+        ValidationError: If any known key/value pair violates the ResourceDictValidation schema.
+    """
+    accepted_keys = _get_accepted_keys(class_type=ResourceDictValidation)
+    optional_lst = [key for key in resource_dict if key not in accepted_keys]
+    validate_dict = {
+        key: value for key, value in resource_dict.items() if key in accepted_keys
+    }
+    _ = ResourceDictValidation(**validate_dict)
+    if len(optional_lst) > 0:
+        warnings.warn(
+            f"The following keys are not recognized and cannot be validated: {optional_lst}",
+            stacklevel=2,
+        )