ensemble-launcher 0.1.0__tar.gz

ensemble_launcher-0.1.0/.github/workflows/pages.yml

@@ -0,0 +1,42 @@
+name: Deploy docs to GitHub Pages
+
+on:
+  push:
+    branches: [main, stage_release]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: "pages"
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install MkDocs
+        run: pip install mkdocs-material
+      - name: Build site
+        run: mkdocs build --config-file mkdocs.yml
+        working-directory: docs
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: docs/site
+
+  deploy:
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - id: deployment
+        uses: actions/deploy-pages@v4

ensemble_launcher-0.1.0/.github/workflows/tests.yml

@@ -0,0 +1,39 @@
+name: Tests
+
+on:
+  push:
+    branches: [main, stage_release]
+  pull_request:
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  core-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install OpenMPI
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y openmpi-bin libopenmpi-dev
+
+      - name: Install package
+        run: pip install -e ".[dev-core]"
+
+      - name: Run core tests
+        working-directory: ensemble_launcher/tests
+        timeout-minutes: 15
+        run: pytest -m core --timeout=300 -v --tb=short -W ignore::pytest.PytestUnraisableExceptionWarning --ignore=test_mcp.py --ignore=test_vllm.py

ensemble_launcher-0.1.0/.gitignore

@@ -0,0 +1,12 @@
+**/__pycache__/
+ckpt_*/
+mcp_*/
+logs/
+*_status.json
+*.mpiexec_tmp/
+profiles/
+build/
+dist/
+*.egg-info/
+docs/site/
+

ensemble_launcher-0.1.0/CLAUDE.md

@@ -0,0 +1,104 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Install & Setup
+
+```bash
+# Editable install with dev dependencies
+python3 -m pip install -e ".[dev]"
+
+# Optional extras
+python3 -m pip install -e ".[mcp]"       # mcp + paramiko
+python3 -m pip install -e ".[dragonhpc]" # DragonHPC backend
+```
+
+The CLI entry point is `el` (`ensemble_launcher/cli.py`).
+
+## Running Tests
+
+Tests live in `ensemble_launcher/tests/`. They must be run from that directory because several tests import from `utils.py` via a relative import:
+
+```bash
+cd ensemble_launcher/tests
+pytest                                   # all tests
+pytest test_ensemble_launcher.py         # end-to-end launcher tests
+pytest test_async_master.py              # async master/worker tests
+pytest test_cluster.py                   # cluster-mode tests
+pytest test_mcp.py                       # MCP interface tests
+pytest test_ensemble_launcher.py::test_el_run  # single test
+```
+
+Tests require `pytest-asyncio` (already in `install_requires`). Async tests are decorated with `@pytest.mark.asyncio`.
+
+## Architecture Overview
+
+### Core abstraction layers (bottom-up)
+
+1. **Executors** (`ensemble_launcher/executors/`) — launch subprocesses or coroutines. Registry pattern: `executor_registry.create_executor(name)`. Key executors:
+   - `async_processpool` — `AsyncProcessPoolExecutor` (default for serial tasks)
+   - `async_mpi` — `AsyncMPIExecutor` (MPI tasks)
+   - `multiprocessing`, `mpi`, `dragon` — legacy sync executors
+
+2. **Communication** (`ensemble_launcher/comm/`) — message passing between nodes. Two async backends:
+   - `async_zmq` — `AsyncZMQComm` (preferred for multi-node)
+   - `multiprocessing` — `MPComm` (single-node only)
+   Messages are typed dataclasses in `comm/messages.py` (`Task`, `Result`, `ResultBatch`, `Status`, `Action`, `NodeUpdate`, `TaskUpdate`, `HeartBeat`, `Stop`).
+
+3. **Scheduler** (`ensemble_launcher/scheduler/`) — assigns tasks to worker nodes. `WorkerScheduler` wraps a `LocalClusterResource` and a pluggable `ChildrenPolicy`. The default policy is `greedy_children_policy`. Custom policies can be loaded at runtime via env vars `EL_EXTERNAL_POLICY_MODULE` / `EL_EXTERNAL_POLICY_PATH`.
+
+4. **Orchestrator** (`ensemble_launcher/orchestrator/`) — the master/worker tree:
+   - `AsyncMaster` — manages a layer of children (sub-masters or workers); the root is always named `"main"`.
+   - `AsyncWorker` — leaf node that executes tasks using a task executor.
+   - `AsyncWorkStealingMaster` / `AsyncWorkStealingWorker` — work-stealing variant (enabled via `LauncherConfig.enable_workstealing`).
+   - `ClusterClient` — connects to a running cluster via checkpoint directory to submit tasks and retrieve `concurrent.futures.Future`s.
+
+5. **EnsembleLauncher** (`ensemble_launcher/ensemble_launcher.py`) — top-level entry point. Reads JSON config or a dict of `Task` objects, auto-configures `LauncherConfig` if not provided, builds the orchestrator tree, and exposes `run()` (blocking) / `start()` + `stop()` (non-blocking cluster mode).
+
+### Node naming convention
+
+Orchestrator nodes follow a hierarchical naming scheme:
+- `main` — root master
+- `main.w0`, `main.w1` — workers directly under root (nlevels=1)
+- `main.m0`, `main.m1` — sub-masters (nlevels=2)
+- `main.m0.w0` — worker under sub-master 0
+
+`ClusterClient(node_id="global")` auto-resolves to the root master by reading checkpoints.
+
+### Key configuration types (`ensemble_launcher/config.py`)
+
+`SystemConfig` — describes one node: `ncpus`, `ngpus`, `cpus` (list of IDs), `gpus` (list of IDs or strings for overloading).
+
+`LauncherConfig` — controls the entire orchestration:
+- `comm_name`: `"async_zmq"` only (sync backends removed)
+- `task_executor_name`: `"async_processpool"` | `"async_mpi"` | list for mixed workloads
+- `child_executor_name`: executor used to launch sub-master/worker processes
+- `nlevels`: hierarchy depth (0=worker only, 1=master+workers, 2=master+sub-masters+workers)
+- `cluster`: enables long-lived cluster mode + `ClusterClient` API
+- `checkpoint_dir`: where the cluster writes its ZMQ address for clients to discover
+- `enable_workstealing`: switches to `AsyncWorkStealingMaster`
+- `req_res`: enables ACK-based guaranteed delivery at the comm layer (default `True`)
+- `send_retries`: number of retry attempts on ACK timeout, only when `req_res=True` (default `10`)
+- `send_timeout`: per-attempt ACK timeout in seconds, only when `req_res=True` (default `1.0`)
+
+### Cluster / MCP mode
+
+When `cluster=True`, the orchestrator runs as a background service. `ClusterClient` reads the checkpoint directory to find the ZMQ address and submit tasks dynamically.
+
+`ensemble_launcher/mcp/Interface` wraps FastMCP and connects to a running cluster:
+- `@interface.tool` — single-task MCP tool
+- `@interface.ensemble_tool` — batch ensemble MCP tool
+SSH tunnel helpers for HPC login→compute node are in `ensemble_launcher/mcp/utils.py`.
+
+### Checkpointing & Profiling
+
+- Checkpoints written to `checkpoint_dir/` (ZMQ addresses for cluster discovery)
+- Logs written to `logs/master-*.log` and `logs/worker-*.log`
+- Profiling: set `LauncherConfig(profile="perfetto")` → outputs `profiles/*_perfetto.json` and `profiles/*_stats.json`
+
+### Auto-configuration logic
+
+When `launcher_config=None`, `EnsembleLauncher.__init__` auto-selects:
+- If all tasks have `nnodes*ppn == 1` → `async_processpool`; otherwise `async_mpi`
+- 1 node → `nlevels=0`, `async_zmq`
+- 2–64 nodes → `nlevels=1`; 65–2048 → `nlevels=2`; 2048+ → `nlevels=3`

ensemble_launcher-0.1.0/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2025, UChicago Argonne, LLC
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

ensemble_launcher-0.1.0/PKG-INFO

@@ -0,0 +1,94 @@
+Metadata-Version: 2.4
+Name: ensemble_launcher
+Version: 0.1.0
+Summary: A lightweight, scalable tool for launching and orchestrating task ensembles across HPC clusters.
+Author: Hari Tummalapalli
+License: BSD-3-Clause
+Project-URL: Homepage, https://github.com/argonne-lcf/ensemble_launcher
+Project-URL: Documentation, https://argonne-lcf.github.io/ensemble_launcher
+Project-URL: Repository, https://github.com/argonne-lcf/ensemble_launcher
+Project-URL: Issues, https://github.com/argonne-lcf/ensemble_launcher/issues
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: System :: Distributed Computing
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: cloudpickle
+Requires-Dist: pydantic
+Requires-Dist: pyzmq
+Requires-Dist: loky
+Requires-Dist: typer
+Provides-Extra: hpc
+Requires-Dist: mpi4py; extra == "hpc"
+Provides-Extra: mcp
+Requires-Dist: mcp; extra == "mcp"
+Requires-Dist: paramiko; extra == "mcp"
+Provides-Extra: inference
+Requires-Dist: vllm; extra == "inference"
+Requires-Dist: uvloop; extra == "inference"
+Requires-Dist: uvicorn; extra == "inference"
+Requires-Dist: fastapi; extra == "inference"
+Provides-Extra: docs
+Requires-Dist: mkdocs-material; extra == "docs"
+Provides-Extra: dev-core
+Requires-Dist: pytest; extra == "dev-core"
+Requires-Dist: pytest-timeout; extra == "dev-core"
+Requires-Dist: pytest-asyncio; extra == "dev-core"
+Requires-Dist: ensemble_launcher[hpc]; extra == "dev-core"
+Provides-Extra: dev-extensions
+Requires-Dist: ensemble_launcher[dev-core,inference,mcp]; extra == "dev-extensions"
+Provides-Extra: all
+Requires-Dist: ensemble_launcher[hpc,inference,mcp]; extra == "all"
+Dynamic: license-file
+
+# Ensemble Launcher
+
+A lightweight, scalable tool for launching and orchestrating task ensembles across HPC clusters with intelligent resource management and hierarchical execution.
+
+[Documentation](https://argonne-lcf.github.io/ensemble_launcher/) | [GitHub](https://github.com/argonne-lcf/ensemble_launcher)
+
+## Features
+
+- **Scalability** -- Hierarchical master-worker architecture tested from 1 to 2048+ nodes
+- **Flexible Execution** -- Support for serial, MPI, and mixed workloads with Python callables or shell commands
+- **Co-Scheduling** -- Run heterogeneous tasks (different node counts, GPU requirements) in a single ensemble
+- **Custom Scheduling Policies** -- Pluggable policy system with built-in bin-packing, split, and FIFO strategies, or write your own
+- **Actors** -- Distributed actor model with async/await communication over ZMQ for long-lived stateful services
+- **Inference** -- Actor-based vLLM wrappers for offline, online, and multi-node LLM serving on HPC clusters
+
+## Quick Example
+
+```python
+from ensemble_launcher import EnsembleLauncher
+
+el = EnsembleLauncher("config.json")
+results = el.run()
+```
+
+```bash
+# Or use the CLI
+el start my_ensemble.json
+```
+
+## Acknowledgments
+
+This work was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357.
+
+## Citation
+
+```bibtex
+@article{tummalapalli2026overcoming,
+  title={Overcoming Orchestration Bottlenecks at Exascale: A Decentralized, Policy-Driven Approach for Sim-AI Ensembles},
+  author={Tummalapalli, Harikrishna and Simpson, Christine M and Balin, Riccardo and Morozov, Vitali A and Pham, Thang D and Keceli, Murat and Uram, Thomas D},
+  journal={arXiv preprint arXiv:2607.12211},
+  year={2026}
+}
+```

ensemble_launcher-0.1.0/README.md

@@ -0,0 +1,43 @@
+# Ensemble Launcher
+
+A lightweight, scalable tool for launching and orchestrating task ensembles across HPC clusters with intelligent resource management and hierarchical execution.
+
+[Documentation](https://argonne-lcf.github.io/ensemble_launcher/) | [GitHub](https://github.com/argonne-lcf/ensemble_launcher)
+
+## Features
+
+- **Scalability** -- Hierarchical master-worker architecture tested from 1 to 2048+ nodes
+- **Flexible Execution** -- Support for serial, MPI, and mixed workloads with Python callables or shell commands
+- **Co-Scheduling** -- Run heterogeneous tasks (different node counts, GPU requirements) in a single ensemble
+- **Custom Scheduling Policies** -- Pluggable policy system with built-in bin-packing, split, and FIFO strategies, or write your own
+- **Actors** -- Distributed actor model with async/await communication over ZMQ for long-lived stateful services
+- **Inference** -- Actor-based vLLM wrappers for offline, online, and multi-node LLM serving on HPC clusters
+
+## Quick Example
+
+```python
+from ensemble_launcher import EnsembleLauncher
+
+el = EnsembleLauncher("config.json")
+results = el.run()
+```
+
+```bash
+# Or use the CLI
+el start my_ensemble.json
+```
+
+## Acknowledgments
+
+This work was supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357.
+
+## Citation
+
+```bibtex
+@article{tummalapalli2026overcoming,
+  title={Overcoming Orchestration Bottlenecks at Exascale: A Decentralized, Policy-Driven Approach for Sim-AI Ensembles},
+  author={Tummalapalli, Harikrishna and Simpson, Christine M and Balin, Riccardo and Morozov, Vitali A and Pham, Thang D and Keceli, Murat and Uram, Thomas D},
+  journal={arXiv preprint arXiv:2607.12211},
+  year={2026}
+}
+```