matensemble 0.2.1__tar.gz → 0.3.3__tar.gz

{matensemble-0.2.1 → matensemble-0.3.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: matensemble
-Version: 0.2.1
+Version: 0.3.3
 Summary: An adaptive and highly asynchronous ensemble simulation workflow manager MatEnsemble (https://github.com/Q-CAD/MatEnsemble) built jointly on top of the hierarchical graph based scheduler FLUX and concurrent-futures infrastructure of python
 Author: Soumendu Bagchi, Kaleb Duchesneau
 Author-email: Soumendu Bagchi <soumendubagchi@gmail.com>, Kaleb Duchesneau <kalebduchesneau@gmail.com>
@@ -32,17 +32,17 @@ Description-Content-Type: text/markdown
 
 # MatEnsemble
 
-MatEnsemble is a Python library for **high-throughput workflows** on HPC systems. You define a directed acyclic graph (DAG) of tasks—**Python callables** or **executable commands**—and MatEnsemble submits work through **[Flux](https://flux-framework.readthedocs.io/)**, tracks completions, **adapts** scheduling to free CPUs and GPUs, and writes structured logs and per-chore output directories.
+MatEnsemble is a Python library for **high-throughput workflows** on HPC systems. You define a directed acyclic graph (DAG) of chores—**Python callables** or **executable commands**—and MatEnsemble submits work through **[Flux](https://flux-framework.readthedocs.io/)**, tracks completions, **adapts** scheduling to free CPUs and GPUs, and writes structured logs and per-chore output directories.
 
 An optional in-tree **dynopro** stack supports streaming dynamics and on-the-fly analysis for advanced materials simulation workflows.
 
 ## Features
 
 - **DAG-based workflows** with dependencies via deferred return values (`OutputReference`)
-- **Adaptive scheduling** that back-fills the allocation as tasks finish (with a non-adaptive mode when you need it)
+- **Adaptive scheduling** that back-fills the allocation as chores finish (with a non-adaptive available)
 - **Two chore types**: Python chores (remotely unpickled and executed by `matensemble.runtime_worker`) and argv-style **executable** chores
 - **Resource requests**: tasks, cores per task, GPUs per task, optional MPI (`pmi2`) via Flux
-- **Observability**: `status.json`, `matensemble_workflow.log`, per-chore `stdout` / `stderr`, pickle and JSON result artifacts; optional **web dashboard** (FastAPI on port 8000)
+- **Observability**: `status.json`, `matensemble_workflow.log`, per-chore `stdout` / `stderr`, pickle and JSON result artifacts; optional **web dashboard**
 
 <p align="center">
   <img src="images/Cap_1_adaptive_task_management.png" alt="Adaptive task management" width="620" />
@@ -52,41 +52,24 @@ An optional in-tree **dynopro** stack supports streaming dynamics and on-the-fly
   <img src="images/Cap_2_dynopro.png" alt="On-the-fly dynamics and analysis" width="620" />
 </p>
 
-## Documentation
-
-Documentation (overview, architecture, tutorials, API reference):
-
-**[matensemble.readthedocs.io](https://matensemble.readthedocs.io/en/latest/)**
 
 ## Installation
 
-### Containers (recommended on many clusters)
-
-OCI images are published to GitHub Container Registry, for example:
+OCI images are published to GitHub Container Registry
 
 `ghcr.io/freddude2004/matensemble:baseline-vX.Y.Z`
 
 See the [container packages](https://github.com/FredDude2004/MatEnsemble/pkgs/container/matensemble) and the [Quick start](https://matensemble.readthedocs.io/en/latest/quickstart.html) in the docs for Apptainer/Singularity and site-specific notes.
 
-### Development install
-
-From a clone of this repository:
+### Anaconda
 
-```bash
-uv sync
-uv sync --group dev   # optional: docs and pytest tooling
-uv run pytest
-```
-
-Or with pip:
+You can build a Conda environment with MatEnsemble and dependencies installed using the environment.yaml file.
 
 ```bash
-pip install -e ".[flux]"
+conda env create -f environment.yaml
 ```
 
-Site-specific Conda-style environment files live under `scripts/` (for example `scripts/baseline/environment.yaml`, `scripts/frontier/`, `scripts/perlmuter/`). Align Python with **3.12+** and Flux with your center’s modules.
-
-## Quick example
+## Example
 
 ```python
 from matensemble.pipeline import Pipeline

{matensemble-0.2.1 → matensemble-0.3.3}/README.md

@@ -9,17 +9,17 @@
 
 # MatEnsemble
 
-MatEnsemble is a Python library for **high-throughput workflows** on HPC systems. You define a directed acyclic graph (DAG) of tasks—**Python callables** or **executable commands**—and MatEnsemble submits work through **[Flux](https://flux-framework.readthedocs.io/)**, tracks completions, **adapts** scheduling to free CPUs and GPUs, and writes structured logs and per-chore output directories.
+MatEnsemble is a Python library for **high-throughput workflows** on HPC systems. You define a directed acyclic graph (DAG) of chores—**Python callables** or **executable commands**—and MatEnsemble submits work through **[Flux](https://flux-framework.readthedocs.io/)**, tracks completions, **adapts** scheduling to free CPUs and GPUs, and writes structured logs and per-chore output directories.
 
 An optional in-tree **dynopro** stack supports streaming dynamics and on-the-fly analysis for advanced materials simulation workflows.
 
 ## Features
 
 - **DAG-based workflows** with dependencies via deferred return values (`OutputReference`)
-- **Adaptive scheduling** that back-fills the allocation as tasks finish (with a non-adaptive mode when you need it)
+- **Adaptive scheduling** that back-fills the allocation as chores finish (with a non-adaptive available)
 - **Two chore types**: Python chores (remotely unpickled and executed by `matensemble.runtime_worker`) and argv-style **executable** chores
 - **Resource requests**: tasks, cores per task, GPUs per task, optional MPI (`pmi2`) via Flux
-- **Observability**: `status.json`, `matensemble_workflow.log`, per-chore `stdout` / `stderr`, pickle and JSON result artifacts; optional **web dashboard** (FastAPI on port 8000)
+- **Observability**: `status.json`, `matensemble_workflow.log`, per-chore `stdout` / `stderr`, pickle and JSON result artifacts; optional **web dashboard**
 
 <p align="center">
   <img src="images/Cap_1_adaptive_task_management.png" alt="Adaptive task management" width="620" />
@@ -29,41 +29,24 @@ An optional in-tree **dynopro** stack supports streaming dynamics and on-the-fly
   <img src="images/Cap_2_dynopro.png" alt="On-the-fly dynamics and analysis" width="620" />
 </p>
 
-## Documentation
-
-Documentation (overview, architecture, tutorials, API reference):
-
-**[matensemble.readthedocs.io](https://matensemble.readthedocs.io/en/latest/)**
 
 ## Installation
 
-### Containers (recommended on many clusters)
-
-OCI images are published to GitHub Container Registry, for example:
+OCI images are published to GitHub Container Registry
 
 `ghcr.io/freddude2004/matensemble:baseline-vX.Y.Z`
 
 See the [container packages](https://github.com/FredDude2004/MatEnsemble/pkgs/container/matensemble) and the [Quick start](https://matensemble.readthedocs.io/en/latest/quickstart.html) in the docs for Apptainer/Singularity and site-specific notes.
 
-### Development install
-
-From a clone of this repository:
+### Anaconda
 
-```bash
-uv sync
-uv sync --group dev   # optional: docs and pytest tooling
-uv run pytest
-```
-
-Or with pip:
+You can build a Conda environment with MatEnsemble and dependencies installed using the environment.yaml file.
 
 ```bash
-pip install -e ".[flux]"
+conda env create -f environment.yaml
 ```
 
-Site-specific Conda-style environment files live under `scripts/` (for example `scripts/baseline/environment.yaml`, `scripts/frontier/`, `scripts/perlmuter/`). Align Python with **3.12+** and Flux with your center’s modules.
-
-## Quick example
+## Example
 
 ```python
 from matensemble.pipeline import Pipeline

{matensemble-0.2.1 → matensemble-0.3.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "matensemble"
-version = "0.2.1"
+version = "0.3.3"
 description = "An adaptive and highly asynchronous ensemble simulation workflow manager MatEnsemble (https://github.com/Q-CAD/MatEnsemble) built jointly on top of the hierarchical graph based scheduler FLUX and concurrent-futures infrastructure of python"
 readme = "README.md"
 license-files = ["LICENSE"]

{matensemble-0.2.1 → matensemble-0.3.3}/src/matensemble/TODO.md

@@ -149,11 +149,11 @@
 - [x] Update README.md
 - [x] Create multi architecture builds for baseline image
 - [x] Run release script
+- [x] Try the MPI Hello examples in perlmutter image and see if they work
 
 ## --- Create new strategy to enable autonomous workflows ---
 - [x] Update the OutputReference objects to have the ability to get the results
 - [x] Create a method in the Pipeline to be able to get the results of all chores
-- [x] Make a strategy that can take in chore and does processing which spawns a new chore
 - [x] Figure out how to spawn a new chore
 - [x] Change to only use cloudpickle and only pickle the actual function once
 - [x] Change the chore objects to not store the function
@@ -162,9 +162,14 @@
 - [x] Add a set of OutputReference objects in the pipeline
 - [x] Connect the added chores back to the pipeline somehow
 - [x] Add a method in pipeline where you can get the results of all of your chores
+- [x] Make the Pipeline.submit() function asynchronous
+- [x] Factor out the common pieces of the FutureProcessingStrategy into the base class
+- [x] Add a UserStrategy class
+- [x] Make a strategy that can take in chore and does processing which spawns a new chore
+- [x] Make a method in the pipeline that constructs a UserStrategy
 - [x] Ping Neil about MPICH
 - [x] Convert Scaffold into PowerPoint Presentation
-- [x] Email coordinatior about length of presentation and audience
+- [x] Email coordinator about length of presentation and audience
 
 ## === AFTER EVERYTHING ABOVE IS DONE AND STABLE ===
 

{matensemble-0.2.1 → matensemble-0.3.3}/src/matensemble/chore.py

@@ -11,6 +11,23 @@ from matensemble.model import ChoreType, Resources
 from matensemble.utils import _json_safe
 
 
+class ChoreSpec:
+    """
+    The specification of a :obj:`Chore`
+
+    Holds the arguments, key-word
+    arguments and the name of the chore that you want those arguments to be
+    passed to. This is class is used by the user when creating a UserStrategy
+    that does processing on completed chores and can spawn new chores.
+    """
+
+    def __init__(self, args, kwargs, qualname, resources: Resources) -> None:
+        self.args = args
+        self.kwargs = kwargs
+        self.qualname = qualname
+        self.resources = resources
+
+
 class Chore:
     """
     A :obj:`Chore` is what MatEnsemble is built around. :obj:`Job`'s can have two
@@ -28,13 +45,11 @@ class Chore:
     def __init__(
         self,
         id: str,
+        workdir: Path,
         command: str | list[str],
-        chore_type: ChoreType,
+        chore_type: ChoreType | int,
         resources: Resources,
-        workdir: Path,
-        func_module: str | None = None,
-        func_qualname: str | None = None,
-        serialized_callable: bytes | None = None,
+        chore_qualname: str | None = None,
         deps: tuple[str, ...] = (),
         args: tuple = (),
         kwargs: dict | None = None,
@@ -76,21 +91,12 @@ class Chore:
             shlex.split(command) if isinstance(command, str) else list(command)
         )
 
-        if chore_type == ChoreType.PYTHON:
-            if serialized_callable is None and not (func_module and func_qualname):
-                raise ValueError(
-                    "Python chores require either serialized_callable or func_module+func_qualname"
-                )
-
         self.chore_type = chore_type
         self.resources = resources
         self.workdir = workdir.resolve()
-        self.spec_path = self.workdir / "chore.pkl"
+        self.spec_path = self.workdir / "chore.pickle"
 
-        self.func_module = func_module
-        self.func_qualname = func_qualname
-        self.serialized_callable = serialized_callable
-        
+        self.chore_qualname = chore_qualname
         self.deps = deps
         self.args = args
         self.kwargs = {} if kwargs is None else kwargs
@@ -111,22 +117,19 @@ class Chore:
                 "env": _json_safe(self.resources.env),
                 "inherit_env": self.resources.inherit_env,
             },
-            "spec_file": str(self.spec_path),
-            "func_module": self.func_module,
-            "func_qualname": self.func_qualname,
-            "has_serialized_callable": self.serialized_callable is not None,
+            "chore_qualname": self.chore_qualname,
             "deps": list(self.deps),
             "args": _json_safe(self.args),
             "kwargs": _json_safe(self.kwargs),
         }
 
-    def _write_debug_json(self) -> None:
+    def _write_metadata(self) -> None:
         """
         The :obj:`Chore` is pickled at runtime to be used later on, but it is also
         written as json for debugging.
         """
 
-        debug_file = self.spec_path.parent / "chore.json"
+        debug_file = self.spec_path.parent / "metadata.json"
         debug_file.parent.mkdir(parents=True, exist_ok=True)
         with debug_file.open("w") as f:
             json.dump(self._to_debug_dict(), f, indent=2)

{matensemble-0.2.1 → matensemble-0.3.3}/src/matensemble/fluxlet.py

@@ -148,7 +148,7 @@ class Fluxlet:
             jobspec.environment = base_env
 
             # helpful for debugging
-            chore._write_debug_json()
+            chore._write_metadata()
 
             # only set this if you truly want every chore to span a fixed node count
             if nnodes is not None:

{matensemble-0.2.1 → matensemble-0.3.3}/src/matensemble/manager.py

@@ -485,6 +485,69 @@ class FluxManager:
                 and len(self._blocked) == 0
             )
 
+    def _add_chore(self, chore: Chore) -> bool:
+        """
+        Add a UserStrategy spawned chore to the queue.
+
+        Returns
+        -------
+        bool
+            True if *chore* was admitted to the manager, False if it was rejected.
+        """
+
+        if not self._chore_fits_allocation(chore):
+            self._record_failure(chore.id, reason="chore_exceeds_allocation")
+            self._logger.error(
+                "CHORE INVALID: chore=%s requires more resources than the allocation can provide",
+                chore.id,
+            )
+            self._fail_dependents(chore.id)
+            return False
+
+        if chore.id in self._chores_by_id:
+            self._logger.error(
+                "CHORE DUPLICATE: chore=%s already exists, rejecting spawn",
+                chore.id,
+            )
+            return False
+
+        for dep in chore.deps:
+            if dep not in self._chores_by_id:
+                self._record_failure(
+                    chore.id, reason="unknown_dependency", upstream=dep
+                )
+                self._logger.error(
+                    "CHORE INVALID: chore=%s has unknown dependency %s",
+                    chore.id,
+                    dep,
+                )
+                return False
+            if self._has_failed(dep):
+                self._record_failure(chore.id, reason="dependency_failed", upstream=dep)
+                self._logger.error(
+                    "CHORE SKIPPED: chore=%s because dependency %s already failed",
+                    chore.id,
+                    dep,
+                )
+                return False
+
+        self._chores_by_id[chore.id] = chore
+        self._dependents.setdefault(chore.id, [])
+
+        remaining = sum(1 for dep in chore.deps if dep not in self._completed_chores)
+        self._remaining_deps[chore.id] = remaining
+
+        for dep in chore.deps:
+            self._dependents.setdefault(dep, []).append(chore.id)
+
+        if remaining == 0:
+            self._ready.appendleft(chore.id)
+            self._blocked.discard(chore.id)
+        else:
+            self._blocked.add(chore.id)
+
+        return True
+
     def run(
         self,
         buffer_time: float = 1.0,

{matensemble-0.2.1 → matensemble-0.3.3}/src/matensemble/model.py

@@ -18,14 +18,22 @@ class OutputReference:
         """
         Return the deserialized result of the referenced chore as a string.
         """
-        # TODO: Make sure that this file exists and add some exception handling
-        dep_result = self.workdir / "result.pkl"
+
+        dep_result = self.workdir / "result.pickle"
         try:
             with dep_result.open("rb") as f:
                 return str(pickle.load(f))
         except Exception as e:
             return f"Error: Could not open result of chore: {self.chore_id} becuase of the following exception: {e}"
 
+    def result(self):
+        dep_result = self.workdir / "result.pickle"
+        try:
+            with dep_result.open("rb") as f:
+                return pickle.load(f)
+        except Exception as e:
+            return f"Error: Could not open result of chore: {self.chore_id} becuase of the following exception: {e}"
+
 
 @dataclass
 class Resources: