runtime-experiment-harness 0.2.0__tar.gz

runtime_experiment_harness-0.2.0/.gitignore

@@ -0,0 +1,9 @@
+__pycache__/
+*.pyc
+plots/
+experiment_results.json
+.venv
+dist/
+build/
+*.egg-info/
+.pytest_cache/

runtime_experiment_harness-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Git Machine
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

runtime_experiment_harness-0.2.0/PKG-INFO

@@ -0,0 +1,152 @@
+Metadata-Version: 2.4
+Name: runtime-experiment-harness
+Version: 0.2.0
+Summary: Runtime experiment harness for vLLM / guidellm benchmarking on NVLink GPU clusters.
+Project-URL: Homepage, https://github.com/GitM-Labs/runtime-experiment-harness
+Project-URL: Repository, https://github.com/GitM-Labs/runtime-experiment-harness
+Project-URL: Issues, https://github.com/GitM-Labs/runtime-experiment-harness
+Author-email: Adit Chawdhary <adit@gitmachine.ai>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: benchmark,gpu,guidellm,inference,llm,nvlink,vllm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: GPU :: NVIDIA CUDA
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Benchmark
+Requires-Python: >=3.10
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Provides-Extra: gpu
+Requires-Dist: guidellm; extra == 'gpu'
+Requires-Dist: vllm; extra == 'gpu'
+Provides-Extra: plot
+Requires-Dist: plotly>=5.20.0; extra == 'plot'
+Description-Content-Type: text/markdown
+
+# Runtime Experiment Harness
+
+A Python experiment harness for benchmarking vLLM with guidellm on NVIDIA
+H100 / NVLink clusters. Ships as the `rex` command.
+
+## Install
+
+```bash
+pip install runtime-experiment-harness            # harness only
+pip install "runtime-experiment-harness[plot]"    # + Plotly HTML result plots
+pip install "runtime-experiment-harness[gpu]"     # + vllm and guidellm (CUDA hosts only)
+```
+
+`vllm` and `guidellm` are intentionally optional. They are large, CUDA-specific,
+and will not build on a machine without an NVIDIA toolchain — making them
+required would break `pip install` on a laptop. The harness invokes both as
+subprocesses, so install the `gpu` extra on the cluster (or let `rex run
+--install` fetch them at runtime).
+
+## Usage
+
+```bash
+rex init                        # write a starter experiments.yaml
+rex check                       # verify the host and provision it
+rex run                         # provision, then run every experiment
+```
+
+`rex check` prepares the box in five steps and stops at the first failure:
+
+1. Verify CUDA >= 13.0 (vLLM dropped everything older)
+2. Print GPU count and the `nvidia-smi` NVLink topology matrix
+3. Install `torch`, `vllm`, `guidellm`, `huggingface-hub`, `plotly`
+4. Create the HF cache, `experiments/`, and `/workspace/guidellm_reports`
+5. Point `HF_HOME` at the cache
+
+`rex run` performs the same five steps, then runs the sweep. Useful flags on
+both:
+
+```bash
+rex check --no-install                    # verify and set up, skip pip
+rex check --hf-home /mnt/models           # default is /workspace/hf_hub
+rex run --config sweep.yaml --output-dir ./results
+rex run --experiments-dir /workspace/experiments
+```
+
+`python -m runtime_harness` works identically to `rex` if the console script is
+not on your PATH.
+
+## Results
+
+Each guidellm run is written per-experiment as:
+
+```text
+experiments/<experiment-id>/<timestamp>_<experiment-id>.json
+```
+
+where `<experiment-id>` is `<name>-gpu<N>` — so one manifest entry swept across
+2, 4, and 8 GPUs produces three directories. Timestamps are UTC and
+filename-safe (`20260803T142305Z`); colons are avoided because they are illegal
+in filenames on macOS and Windows and need quoting in every shell.
+
+guidellm's own JSON report is written to `/workspace/guidellm_reports` as
+`<timestamp>_<experiment-id>_guidellm.json`, and a copy is placed beside the
+harness record so each run directory is self-contained. A manifest that sets
+`--output kind=json,path=...` explicitly on its `guidellm_command` keeps that
+path — an explicit config value beats the default.
+
+## vLLM startup
+
+`vllm_args` from the manifest are passed through verbatim, with
+`--tensor-parallel-size` and `--enable-expert-parallel` filled in from the
+`tp`/`ep` fields when not already present. The harness then polls
+`http://localhost:<port>/health` until the server actually accepts traffic
+rather than sleeping a fixed interval — a cold HF cache can take many minutes to
+load a large checkpoint. `--port` is read from `vllm_args` (default 8000). If
+the server dies or fails to come up within 15 minutes, that experiment is
+recorded as failed and the sweep moves on.
+
+A combined `experiment_results.json` and a `plots/` folder also land in the
+working directory (override with `--output-dir`).
+
+### `HF_HOME` and your shell
+
+`rex` exports `HF_HOME` into its own process, so every vLLM and guidellm
+subprocess it launches inherits the cache location. A child process cannot
+change its parent's environment, so this does **not** persist into your shell —
+`rex` prints the matching `export` line if you want it there too.
+
+## Requirements
+
+- Python 3.10+
+- CUDA 13.0 or newer — vLLM has dropped support for anything older, so `rex
+  check` and `rex run` both refuse to proceed below that and tell you what was
+  detected.
+- NVIDIA drivers with `nvidia-smi` on `PATH`
+
+## Config
+
+`experiments.yaml` defines the sweep: `gpu_counts` to iterate over, then one
+entry per experiment with model, prompt, and EP/DP/TP/PP parallelism settings.
+`vllm_args` and `guidellm_args` are passed through verbatim, so any flag the
+underlying tools accept works without changes here. `rex init` writes a
+worked example to start from.
+
+## Development
+
+```bash
+pip install -e ".[dev,plot]"
+pytest
+```
+
+## Release
+
+```bash
+python -m build
+python -m twine check dist/*
+python -m twine upload dist/*
+```

runtime_experiment_harness-0.2.0/README.md

@@ -0,0 +1,118 @@
+# Runtime Experiment Harness
+
+A Python experiment harness for benchmarking vLLM with guidellm on NVIDIA
+H100 / NVLink clusters. Ships as the `rex` command.
+
+## Install
+
+```bash
+pip install runtime-experiment-harness            # harness only
+pip install "runtime-experiment-harness[plot]"    # + Plotly HTML result plots
+pip install "runtime-experiment-harness[gpu]"     # + vllm and guidellm (CUDA hosts only)
+```
+
+`vllm` and `guidellm` are intentionally optional. They are large, CUDA-specific,
+and will not build on a machine without an NVIDIA toolchain — making them
+required would break `pip install` on a laptop. The harness invokes both as
+subprocesses, so install the `gpu` extra on the cluster (or let `rex run
+--install` fetch them at runtime).
+
+## Usage
+
+```bash
+rex init                        # write a starter experiments.yaml
+rex check                       # verify the host and provision it
+rex run                         # provision, then run every experiment
+```
+
+`rex check` prepares the box in five steps and stops at the first failure:
+
+1. Verify CUDA >= 13.0 (vLLM dropped everything older)
+2. Print GPU count and the `nvidia-smi` NVLink topology matrix
+3. Install `torch`, `vllm`, `guidellm`, `huggingface-hub`, `plotly`
+4. Create the HF cache, `experiments/`, and `/workspace/guidellm_reports`
+5. Point `HF_HOME` at the cache
+
+`rex run` performs the same five steps, then runs the sweep. Useful flags on
+both:
+
+```bash
+rex check --no-install                    # verify and set up, skip pip
+rex check --hf-home /mnt/models           # default is /workspace/hf_hub
+rex run --config sweep.yaml --output-dir ./results
+rex run --experiments-dir /workspace/experiments
+```
+
+`python -m runtime_harness` works identically to `rex` if the console script is
+not on your PATH.
+
+## Results
+
+Each guidellm run is written per-experiment as:
+
+```text
+experiments/<experiment-id>/<timestamp>_<experiment-id>.json
+```
+
+where `<experiment-id>` is `<name>-gpu<N>` — so one manifest entry swept across
+2, 4, and 8 GPUs produces three directories. Timestamps are UTC and
+filename-safe (`20260803T142305Z`); colons are avoided because they are illegal
+in filenames on macOS and Windows and need quoting in every shell.
+
+guidellm's own JSON report is written to `/workspace/guidellm_reports` as
+`<timestamp>_<experiment-id>_guidellm.json`, and a copy is placed beside the
+harness record so each run directory is self-contained. A manifest that sets
+`--output kind=json,path=...` explicitly on its `guidellm_command` keeps that
+path — an explicit config value beats the default.
+
+## vLLM startup
+
+`vllm_args` from the manifest are passed through verbatim, with
+`--tensor-parallel-size` and `--enable-expert-parallel` filled in from the
+`tp`/`ep` fields when not already present. The harness then polls
+`http://localhost:<port>/health` until the server actually accepts traffic
+rather than sleeping a fixed interval — a cold HF cache can take many minutes to
+load a large checkpoint. `--port` is read from `vllm_args` (default 8000). If
+the server dies or fails to come up within 15 minutes, that experiment is
+recorded as failed and the sweep moves on.
+
+A combined `experiment_results.json` and a `plots/` folder also land in the
+working directory (override with `--output-dir`).
+
+### `HF_HOME` and your shell
+
+`rex` exports `HF_HOME` into its own process, so every vLLM and guidellm
+subprocess it launches inherits the cache location. A child process cannot
+change its parent's environment, so this does **not** persist into your shell —
+`rex` prints the matching `export` line if you want it there too.
+
+## Requirements
+
+- Python 3.10+
+- CUDA 13.0 or newer — vLLM has dropped support for anything older, so `rex
+  check` and `rex run` both refuse to proceed below that and tell you what was
+  detected.
+- NVIDIA drivers with `nvidia-smi` on `PATH`
+
+## Config
+
+`experiments.yaml` defines the sweep: `gpu_counts` to iterate over, then one
+entry per experiment with model, prompt, and EP/DP/TP/PP parallelism settings.
+`vllm_args` and `guidellm_args` are passed through verbatim, so any flag the
+underlying tools accept works without changes here. `rex init` writes a
+worked example to start from.
+
+## Development
+
+```bash
+pip install -e ".[dev,plot]"
+pytest
+```
+
+## Release
+
+```bash
+python -m build
+python -m twine check dist/*
+python -m twine upload dist/*
+```

runtime_experiment_harness-0.2.0/pyproject.toml

@@ -0,0 +1,59 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "runtime-experiment-harness"
+dynamic = ["version"]
+description = "Runtime experiment harness for vLLM / guidellm benchmarking on NVLink GPU clusters."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "Adit Chawdhary", email = "adit@gitmachine.ai" }]
+keywords = ["vllm", "guidellm", "benchmark", "gpu", "nvlink", "llm", "inference"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Environment :: GPU :: NVIDIA CUDA",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+    "Topic :: System :: Benchmark",
+]
+
+# vllm and guidellm are deliberately NOT core dependencies: they are large,
+# CUDA-specific, and unbuildable on non-NVIDIA hosts, which would make a plain
+# `pip install` fail on any laptop. Both are invoked as subprocesses, so the
+# harness imports fine without them. Install via the `gpu` extra on the cluster.
+dependencies = [
+    "pyyaml>=6.0",
+    "rich>=13.0.0",
+]
+
+[project.optional-dependencies]
+plot = ["plotly>=5.20.0"]
+gpu = ["vllm", "guidellm"]
+dev = ["pytest>=7.0", "build", "twine"]
+
+[project.scripts]
+rex = "runtime_harness.cli:main"
+
+[project.urls]
+Homepage = "https://github.com/GitM-Labs/runtime-experiment-harness"
+Repository = "https://github.com/GitM-Labs/runtime-experiment-harness"
+Issues = "https://github.com/GitM-Labs/runtime-experiment-harness"
+
+[tool.hatch.version]
+path = "runtime_harness/__init__.py"
+
+[tool.hatch.build.targets.wheel]
+packages = ["runtime_harness"]
+
+[tool.hatch.build.targets.sdist]
+include = ["runtime_harness", "tests", "README.md", "LICENSE"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

runtime_experiment_harness-0.2.0/runtime_harness/__init__.py

@@ -0,0 +1,5 @@
+"""Runtime experiment harness for vLLM / guidellm on NVLink GPU clusters."""
+
+__version__ = "0.2.0"
+
+__all__ = ["__version__"]

runtime_experiment_harness-0.2.0/runtime_harness/__main__.py

@@ -0,0 +1,8 @@
+"""Allow `python -m runtime_harness` alongside the `rex` console script."""
+
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

runtime_experiment_harness-0.2.0/runtime_harness/banner.py

@@ -0,0 +1,149 @@
+"""Startup banner for runtime.
+
+Usage:
+    from banner import banner
+    print(banner())                 # auto-detects colour support
+    print(banner("runtime", subtitle="v0.4.1"))
+
+Run directly to preview:  python banner.py
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from collections.abc import Sequence
+
+# --- 5x5 block font -------------------------------------------------------
+# '#' = filled cell, '.' = empty. Add glyphs here to support more names.
+
+FONT: dict[str, tuple[str, ...]] = {
+    "R": ("####.", "#...#", "####.", "#..#.", "#...#"),
+    "U": ("#...#", "#...#", "#...#", "#...#", ".###."),
+    "N": ("#...#", "##..#", "#.#.#", "#..##", "#...#"),
+    "T": ("#####", "..#..", "..#..", "..#..", "..#.."),
+    "I": ("#####", "..#..", "..#..", "..#..", "#####"),
+    "M": ("#...#", "##.##", "#.#.#", "#...#", "#...#"),
+    "E": ("#####", "#....", "####.", "#....", "#####"),
+    "A": (".###.", "#...#", "#####", "#...#", "#...#"),
+    "C": (".####", "#....", "#....", "#....", ".####"),
+    "D": ("####.", "#...#", "#...#", "#...#", "####."),
+    "O": (".###.", "#...#", "#...#", "#...#", ".###."),
+    "S": (".####", "#....", ".###.", "....#", "####."),
+    "L": ("#....", "#....", "#....", "#....", "#####"),
+    "P": ("####.", "#...#", "####.", "#....", "#...."),
+    "Y": ("#...#", ".#.#.", "..#..", "..#..", "..#.."),
+    "X": ("#...#", ".#.#.", "..#..", ".#.#.", "#...#"),
+    "H": ("#...#", "#...#", "#####", "#...#", "#...#"),
+    "V": ("#...#", "#...#", "#...#", ".#.#.", "..#.."),
+    "G": (".####", "#....", "#..##", "#...#", ".###."),
+    "B": ("####.", "#...#", "####.", "#...#", "####."),
+    "F": ("#####", "#....", "####.", "#....", "#...."),
+    " ": (".....", ".....", ".....", ".....", "....."),
+}
+
+BLOCK = "\u25a6"  # ▦
+HEIGHT = 5
+
+# Vertical gradient, top row brightest. Truecolor RGB.
+GRADIENT = [(0xE6, 0xF7, 0xFF), (0x9A, 0xD9, 0xF5),
+            (0x5C, 0xB3, 0xE8), (0x2E, 0x86, 0xC8), (0x1B, 0x5A, 0x9E)]
+
+DIM = "\u001b[2m"
+RESET = "\u001b[0m"
+
+
+def supports_color(stream=None) -> bool:
+    """True when it is safe to emit ANSI escapes."""
+    stream = stream or sys.stdout
+    if os.environ.get("NO_COLOR") is not None:
+        return False
+    if os.environ.get("FORCE_COLOR"):
+        return True
+    if not hasattr(stream, "isatty") or not stream.isatty():
+        return False
+    return os.environ.get("TERM", "") != "dumb"
+
+
+def _rows(text: str, cell: str, gap: int) -> list[str]:
+    glyphs = []
+    for ch in text.upper():
+        try:
+            glyphs.append(FONT[ch])
+        except KeyError:
+            raise ValueError(f"no glyph for {ch!r}; add one to FONT") from None
+
+    spacer = " " * gap
+    rows = []
+    for y in range(HEIGHT):
+        parts = ["".join(cell if c == "#" else " " * len(cell) for c in g[y])
+                 for g in glyphs]
+        rows.append(spacer.join(parts).rstrip())
+    return rows
+
+
+def banner(
+    text: str = "runtime",
+    subtitle: str | Sequence[str] | None = None,
+    *,
+    color: bool | None = None,
+    cell: str = BLOCK * 2,
+    gap: int = 1,
+    indent: int = 2,
+) -> str:
+    """Render `text` as a block banner.
+
+    color=None auto-detects; pass True/False to force.
+    cell is the glyph used per filled pixel (two blocks keeps the
+    aspect ratio close to square in most terminals).
+    """
+    rows = _rows(text, cell, gap)
+    pad = " " * indent
+    use_color = supports_color() if color is None else color
+
+    out = []
+    for i, row in enumerate(rows):
+        if use_color:
+            r, g, b = GRADIENT[i % len(GRADIENT)]
+            out.append(f"{pad}\u001b[38;2;{r};{g};{b}m{row}{RESET}")
+        else:
+            out.append(pad + row)
+
+    if subtitle:
+        lines = [subtitle] if isinstance(subtitle, str) else list(subtitle)
+        width = max(len(r) for r in rows)
+        out.append("")
+        for line in lines:
+            centered = line.center(width).rstrip()
+            out.append(f"{pad}{DIM}{centered}{RESET}" if use_color else pad + centered)
+
+    return "\n".join(out)
+
+
+def harness_banner(
+    version: str = "",
+    model: str | None = None,
+    *,
+    color: bool | None = None,
+) -> str:
+    """The artwork shown when the harness starts.
+
+    Renders REX in the block font over a dim strapline. `model` is included only
+    when known, so `rex check` (which has no manifest) stays uncluttered.
+    """
+    strapline = "runtime experiment harness"
+    if version:
+        strapline += f"  \u00b7  v{version}"
+        # strapline += "vLLM  \u00b7  guidellm  \u00b7  NVLink fabric"
+
+    subtitles = [strapline]
+    if model:
+        subtitles.append(model)
+
+    return banner("RUNTIME", subtitle=subtitles, color=color) 
+
+
+if __name__ == "__main__":
+    print()
+    print(harness_banner("0.2.0", color=True))
+    print()