runtime-experiment-harness 0.2.0__py3-none-any.whl

runtime_experiment_harness-0.2.0.dist-info/METADATA

@@ -0,0 +1,152 @@
+Metadata-Version: 2.4
+Name: runtime-experiment-harness
+Version: 0.2.0
+Summary: Runtime experiment harness for vLLM / guidellm benchmarking on NVLink GPU clusters.
+Project-URL: Homepage, https://github.com/GitM-Labs/runtime-experiment-harness
+Project-URL: Repository, https://github.com/GitM-Labs/runtime-experiment-harness
+Project-URL: Issues, https://github.com/GitM-Labs/runtime-experiment-harness
+Author-email: Adit Chawdhary <adit@gitmachine.ai>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: benchmark,gpu,guidellm,inference,llm,nvlink,vllm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: GPU :: NVIDIA CUDA
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Benchmark
+Requires-Python: >=3.10
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Provides-Extra: gpu
+Requires-Dist: guidellm; extra == 'gpu'
+Requires-Dist: vllm; extra == 'gpu'
+Provides-Extra: plot
+Requires-Dist: plotly>=5.20.0; extra == 'plot'
+Description-Content-Type: text/markdown
+
+# Runtime Experiment Harness
+
+A Python experiment harness for benchmarking vLLM with guidellm on NVIDIA
+H100 / NVLink clusters. Ships as the `rex` command.
+
+## Install
+
+```bash
+pip install runtime-experiment-harness            # harness only
+pip install "runtime-experiment-harness[plot]"    # + Plotly HTML result plots
+pip install "runtime-experiment-harness[gpu]"     # + vllm and guidellm (CUDA hosts only)
+```
+
+`vllm` and `guidellm` are intentionally optional. They are large, CUDA-specific,
+and will not build on a machine without an NVIDIA toolchain — making them
+required would break `pip install` on a laptop. The harness invokes both as
+subprocesses, so install the `gpu` extra on the cluster (or let `rex run
+--install` fetch them at runtime).
+
+## Usage
+
+```bash
+rex init                        # write a starter experiments.yaml
+rex check                       # verify the host and provision it
+rex run                         # provision, then run every experiment
+```
+
+`rex check` prepares the box in five steps and stops at the first failure:
+
+1. Verify CUDA >= 13.0 (vLLM dropped everything older)
+2. Print GPU count and the `nvidia-smi` NVLink topology matrix
+3. Install `torch`, `vllm`, `guidellm`, `huggingface-hub`, `plotly`
+4. Create the HF cache, `experiments/`, and `/workspace/guidellm_reports`
+5. Point `HF_HOME` at the cache
+
+`rex run` performs the same five steps, then runs the sweep. Useful flags on
+both:
+
+```bash
+rex check --no-install                    # verify and set up, skip pip
+rex check --hf-home /mnt/models           # default is /workspace/hf_hub
+rex run --config sweep.yaml --output-dir ./results
+rex run --experiments-dir /workspace/experiments
+```
+
+`python -m runtime_harness` works identically to `rex` if the console script is
+not on your PATH.
+
+## Results
+
+Each guidellm run is written per-experiment as:
+
+```text
+experiments/<experiment-id>/<timestamp>_<experiment-id>.json
+```
+
+where `<experiment-id>` is `<name>-gpu<N>` — so one manifest entry swept across
+2, 4, and 8 GPUs produces three directories. Timestamps are UTC and
+filename-safe (`20260803T142305Z`); colons are avoided because they are illegal
+in filenames on macOS and Windows and need quoting in every shell.
+
+guidellm's own JSON report is written to `/workspace/guidellm_reports` as
+`<timestamp>_<experiment-id>_guidellm.json`, and a copy is placed beside the
+harness record so each run directory is self-contained. A manifest that sets
+`--output kind=json,path=...` explicitly on its `guidellm_command` keeps that
+path — an explicit config value beats the default.
+
+## vLLM startup
+
+`vllm_args` from the manifest are passed through verbatim, with
+`--tensor-parallel-size` and `--enable-expert-parallel` filled in from the
+`tp`/`ep` fields when not already present. The harness then polls
+`http://localhost:<port>/health` until the server actually accepts traffic
+rather than sleeping a fixed interval — a cold HF cache can take many minutes to
+load a large checkpoint. `--port` is read from `vllm_args` (default 8000). If
+the server dies or fails to come up within 15 minutes, that experiment is
+recorded as failed and the sweep moves on.
+
+A combined `experiment_results.json` and a `plots/` folder also land in the
+working directory (override with `--output-dir`).
+
+### `HF_HOME` and your shell
+
+`rex` exports `HF_HOME` into its own process, so every vLLM and guidellm
+subprocess it launches inherits the cache location. A child process cannot
+change its parent's environment, so this does **not** persist into your shell —
+`rex` prints the matching `export` line if you want it there too.
+
+## Requirements
+
+- Python 3.10+
+- CUDA 13.0 or newer — vLLM has dropped support for anything older, so `rex
+  check` and `rex run` both refuse to proceed below that and tell you what was
+  detected.
+- NVIDIA drivers with `nvidia-smi` on `PATH`
+
+## Config
+
+`experiments.yaml` defines the sweep: `gpu_counts` to iterate over, then one
+entry per experiment with model, prompt, and EP/DP/TP/PP parallelism settings.
+`vllm_args` and `guidellm_args` are passed through verbatim, so any flag the
+underlying tools accept works without changes here. `rex init` writes a
+worked example to start from.
+
+## Development
+
+```bash
+pip install -e ".[dev,plot]"
+pytest
+```
+
+## Release
+
+```bash
+python -m build
+python -m twine check dist/*
+python -m twine upload dist/*
+```

runtime_experiment_harness-0.2.0.dist-info/RECORD

@@ -0,0 +1,12 @@
+runtime_harness/__init__.py,sha256=3MibRUzONEGjlgt-VeUdg_8y8ApaR7fXmu52rg_Dg3M,127
+runtime_harness/__main__.py,sha256=ho1jLf3h0btspnOT2K7MJ_qLwZfxuiOSUdkSBDlbxn0,160
+runtime_harness/banner.py,sha256=5pNLuORAQaMQwgPViFSnGe9zNENROVl1f48xBJoyACs,4776
+runtime_harness/checks.py,sha256=GfkH3IwCayL7CBwpoaB8goLeAaL7ZrxNWmbTtZ8sItQ,6256
+runtime_harness/cli.py,sha256=6OJq3BBStoNXNn8b5urnZjVew7GDFdAr9_82HxmC2GQ,4418
+runtime_harness/experiments.py,sha256=mLGaSc4T_7yaQzuuVo2KhBXPWxse1zT5DAujFW1s9cY,14875
+runtime_harness/data/experiments.yaml,sha256=5wUnwTOWpf5D3cER-Q8PVCI1GCjDh1QdaUlFx8AJZRc,1364
+runtime_experiment_harness-0.2.0.dist-info/METADATA,sha256=s5sfelF1buP7-zskSPRTVMkr__GYwyzCSCXf7_bZV0k,5800
+runtime_experiment_harness-0.2.0.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
+runtime_experiment_harness-0.2.0.dist-info/entry_points.txt,sha256=yvdKhAFQdEjd62ECc_fU9pKG9GfBPLWRtehRuJZ9ulQ,49
+runtime_experiment_harness-0.2.0.dist-info/licenses/LICENSE,sha256=XDYIWg_2DNTrf0H3TfKpFzg77iPVhdJBuOkgQoLPnTk,1068
+runtime_experiment_harness-0.2.0.dist-info/RECORD,,

runtime_experiment_harness-0.2.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.31.0
+Root-Is-Purelib: true
+Tag: py3-none-any

runtime_experiment_harness-0.2.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+rex = runtime_harness.cli:main

runtime_experiment_harness-0.2.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Git Machine
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

runtime_harness/__init__.py

@@ -0,0 +1,5 @@
+"""Runtime experiment harness for vLLM / guidellm on NVLink GPU clusters."""
+
+__version__ = "0.2.0"
+
+__all__ = ["__version__"]

runtime_harness/__main__.py

@@ -0,0 +1,8 @@
+"""Allow `python -m runtime_harness` alongside the `rex` console script."""
+
+import sys
+
+from .cli import main
+
+if __name__ == "__main__":
+    sys.exit(main())

runtime_harness/banner.py

@@ -0,0 +1,149 @@
+"""Startup banner for runtime.
+
+Usage:
+    from banner import banner
+    print(banner())                 # auto-detects colour support
+    print(banner("runtime", subtitle="v0.4.1"))
+
+Run directly to preview:  python banner.py
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from collections.abc import Sequence
+
+# --- 5x5 block font -------------------------------------------------------
+# '#' = filled cell, '.' = empty. Add glyphs here to support more names.
+
+FONT: dict[str, tuple[str, ...]] = {
+    "R": ("####.", "#...#", "####.", "#..#.", "#...#"),
+    "U": ("#...#", "#...#", "#...#", "#...#", ".###."),
+    "N": ("#...#", "##..#", "#.#.#", "#..##", "#...#"),
+    "T": ("#####", "..#..", "..#..", "..#..", "..#.."),
+    "I": ("#####", "..#..", "..#..", "..#..", "#####"),
+    "M": ("#...#", "##.##", "#.#.#", "#...#", "#...#"),
+    "E": ("#####", "#....", "####.", "#....", "#####"),
+    "A": (".###.", "#...#", "#####", "#...#", "#...#"),
+    "C": (".####", "#....", "#....", "#....", ".####"),
+    "D": ("####.", "#...#", "#...#", "#...#", "####."),
+    "O": (".###.", "#...#", "#...#", "#...#", ".###."),
+    "S": (".####", "#....", ".###.", "....#", "####."),
+    "L": ("#....", "#....", "#....", "#....", "#####"),
+    "P": ("####.", "#...#", "####.", "#....", "#...."),
+    "Y": ("#...#", ".#.#.", "..#..", "..#..", "..#.."),
+    "X": ("#...#", ".#.#.", "..#..", ".#.#.", "#...#"),
+    "H": ("#...#", "#...#", "#####", "#...#", "#...#"),
+    "V": ("#...#", "#...#", "#...#", ".#.#.", "..#.."),
+    "G": (".####", "#....", "#..##", "#...#", ".###."),
+    "B": ("####.", "#...#", "####.", "#...#", "####."),
+    "F": ("#####", "#....", "####.", "#....", "#...."),
+    " ": (".....", ".....", ".....", ".....", "....."),
+}
+
+BLOCK = "\u25a6"  # ▦
+HEIGHT = 5
+
+# Vertical gradient, top row brightest. Truecolor RGB.
+GRADIENT = [(0xE6, 0xF7, 0xFF), (0x9A, 0xD9, 0xF5),
+            (0x5C, 0xB3, 0xE8), (0x2E, 0x86, 0xC8), (0x1B, 0x5A, 0x9E)]
+
+DIM = "\u001b[2m"
+RESET = "\u001b[0m"
+
+
+def supports_color(stream=None) -> bool:
+    """True when it is safe to emit ANSI escapes."""
+    stream = stream or sys.stdout
+    if os.environ.get("NO_COLOR") is not None:
+        return False
+    if os.environ.get("FORCE_COLOR"):
+        return True
+    if not hasattr(stream, "isatty") or not stream.isatty():
+        return False
+    return os.environ.get("TERM", "") != "dumb"
+
+
+def _rows(text: str, cell: str, gap: int) -> list[str]:
+    glyphs = []
+    for ch in text.upper():
+        try:
+            glyphs.append(FONT[ch])
+        except KeyError:
+            raise ValueError(f"no glyph for {ch!r}; add one to FONT") from None
+
+    spacer = " " * gap
+    rows = []
+    for y in range(HEIGHT):
+        parts = ["".join(cell if c == "#" else " " * len(cell) for c in g[y])
+                 for g in glyphs]
+        rows.append(spacer.join(parts).rstrip())
+    return rows
+
+
+def banner(
+    text: str = "runtime",
+    subtitle: str | Sequence[str] | None = None,
+    *,
+    color: bool | None = None,
+    cell: str = BLOCK * 2,
+    gap: int = 1,
+    indent: int = 2,
+) -> str:
+    """Render `text` as a block banner.
+
+    color=None auto-detects; pass True/False to force.
+    cell is the glyph used per filled pixel (two blocks keeps the
+    aspect ratio close to square in most terminals).
+    """
+    rows = _rows(text, cell, gap)
+    pad = " " * indent
+    use_color = supports_color() if color is None else color
+
+    out = []
+    for i, row in enumerate(rows):
+        if use_color:
+            r, g, b = GRADIENT[i % len(GRADIENT)]
+            out.append(f"{pad}\u001b[38;2;{r};{g};{b}m{row}{RESET}")
+        else:
+            out.append(pad + row)
+
+    if subtitle:
+        lines = [subtitle] if isinstance(subtitle, str) else list(subtitle)
+        width = max(len(r) for r in rows)
+        out.append("")
+        for line in lines:
+            centered = line.center(width).rstrip()
+            out.append(f"{pad}{DIM}{centered}{RESET}" if use_color else pad + centered)
+
+    return "\n".join(out)
+
+
+def harness_banner(
+    version: str = "",
+    model: str | None = None,
+    *,
+    color: bool | None = None,
+) -> str:
+    """The artwork shown when the harness starts.
+
+    Renders REX in the block font over a dim strapline. `model` is included only
+    when known, so `rex check` (which has no manifest) stays uncluttered.
+    """
+    strapline = "runtime experiment harness"
+    if version:
+        strapline += f"  \u00b7  v{version}"
+        # strapline += "vLLM  \u00b7  guidellm  \u00b7  NVLink fabric"
+
+    subtitles = [strapline]
+    if model:
+        subtitles.append(model)
+
+    return banner("RUNTIME", subtitle=subtitles, color=color) 
+
+
+if __name__ == "__main__":
+    print()
+    print(harness_banner("0.2.0", color=True))
+    print()

runtime_harness/checks.py

@@ -0,0 +1,168 @@
+"""Preflight checks for the GPU host: CUDA version and NVLink topology."""
+
+from __future__ import annotations
+
+import os
+import re
+import shlex
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+console = Console()
+
+# vLLM dropped support for CUDA older than this.
+MIN_CUDA_VERSION = (13, 0)
+
+# Model weights are large; keep them on the roomy workspace volume rather than
+# the default ~/.cache/huggingface, which is often a small root disk.
+DEFAULT_HF_HOME = Path("/workspace/hf_hub")
+DEFAULT_EXPERIMENTS_DIR = Path("experiments")
+DEFAULT_REPORTS_DIR = Path("/workspace/guidellm_reports")
+
+# torch is listed alongside vllm so pip resolves them together -- vllm pins an
+# exact torch build, and installing torch first would just get it replaced.
+RUNTIME_PACKAGES = ["torch", "vllm", "guidellm", "huggingface-hub", "plotly"]
+
+
+def format_version(version: tuple[int, int]) -> str:
+    return ".".join(str(part) for part in version)
+
+
+def run_command(command, capture_output=False, check=True, env=None):
+    console.log(f"[blue]Running command:[/blue] {command}")
+    process = subprocess.run(
+        shlex.split(command),
+        capture_output=capture_output,
+        text=True,
+        check=check,
+        env=env,
+    )
+    return process.stdout if capture_output else None
+
+
+def ensure_dependencies(packages=None):
+    console.rule("Installing runtime dependencies")
+    packages = list(packages) if packages is not None else RUNTIME_PACKAGES
+    command = f"{sys.executable} -m pip install {' '.join(packages)}"
+    run_command(command)
+    console.log("[green]Dependencies installed successfully[/green]")
+
+
+def prepare_workspace(hf_home=None, experiments_dir=None, reports_dir=None):
+    """Create the HF cache, experiments, and report directories; point HF at the cache.
+
+    The environment variables are exported into this process, so every vLLM and
+    guidellm subprocess the harness launches inherits them. They do not leak back
+    into the calling shell -- see the printed export line for that.
+    """
+    hf_home = Path(hf_home) if hf_home is not None else DEFAULT_HF_HOME
+    experiments_dir = Path(experiments_dir) if experiments_dir is not None else DEFAULT_EXPERIMENTS_DIR
+    reports_dir = Path(reports_dir) if reports_dir is not None else DEFAULT_REPORTS_DIR
+
+    console.rule("Preparing workspace")
+
+    flags = {"hf_home": "--hf-home", "experiments": "--experiments-dir", "reports": "--reports-dir"}
+    for label, directory in (
+        ("hf_home", hf_home),
+        ("experiments", experiments_dir),
+        ("reports", reports_dir),
+    ):
+        try:
+            directory.mkdir(parents=True, exist_ok=True)
+        except OSError as exc:
+            raise RuntimeError(
+                f"Could not create {directory}: {exc}. "
+                f"Pass {flags[label]} to use a writable location."
+            ) from exc
+
+    os.environ["HF_HOME"] = str(hf_home)
+    os.environ.setdefault("HF_HUB_ENABLE_HF_TRANSFER", "1")
+
+    console.print(f"[bold]HF_HOME:[/bold] {hf_home}")
+    console.print(f"[bold]Experiments directory:[/bold] {experiments_dir.resolve()}")
+    console.print(f"[bold]guidellm reports:[/bold] {reports_dir}")
+    console.print(f"[dim]For your shell too:  export HF_HOME={hf_home}[/dim]")
+
+    return hf_home, experiments_dir, reports_dir
+
+
+def parse_cuda_version(smi_output: str):
+    """Return the CUDA version reported by nvidia-smi as an (major, minor) tuple."""
+    match = re.search(r"CUDA Version:\s*([0-9]+)\.([0-9]+)", smi_output)
+    if match is None:
+        return None
+    return int(match.group(1)), int(match.group(2))
+
+
+def ensure_cuda_supported():
+    """Fail fast when the CUDA runtime is older than vLLM supports."""
+    if shutil.which("nvidia-smi") is None:
+        raise FileNotFoundError("nvidia-smi not found in PATH. NVIDIA drivers must be installed.")
+
+    smi_output = run_command("nvidia-smi", capture_output=True) or ""
+    version = parse_cuda_version(smi_output)
+    minimum = format_version(MIN_CUDA_VERSION)
+
+    if version is None:
+        raise RuntimeError(
+            f"Could not determine the CUDA version from nvidia-smi output. vLLM requires CUDA >= {minimum}."
+        )
+
+    detected = format_version(version)
+    if version < MIN_CUDA_VERSION:
+        raise RuntimeError(
+            f"CUDA {detected} detected, but vLLM requires CUDA >= {minimum}. "
+            "Upgrade the NVIDIA driver / CUDA toolkit before running experiments."
+        )
+
+    console.print(f"[bold]CUDA version:[/bold] {detected}")
+    return version
+
+
+def parse_gpu_topology():
+    if shutil.which("nvidia-smi") is None:
+        raise FileNotFoundError("nvidia-smi not found in PATH. NVIDIA drivers must be installed.")
+
+    topo_raw = run_command("nvidia-smi topo --matrix", capture_output=True) or ""
+    gpu_count = 0
+    nvlink_matrix = []
+
+    for line in topo_raw.splitlines():
+        if line.startswith("GPU") and "CPU" not in line:
+            parts = re.split(r"\s+", line.strip())
+            if parts:
+                gpu_count += 1
+                nvlink_matrix.append(parts[1:gpu_count + 1])
+
+    nvlink_available = any("NV" in cell.upper() for row in nvlink_matrix for cell in row)
+    return gpu_count, nvlink_available, topo_raw
+
+
+def run_preflight(install=True, hf_home=None, experiments_dir=None, reports_dir=None):
+    """Verify the host and provision it for experiments.
+
+    1. CUDA >= 13.0 (vLLM dropped everything older)
+    2. GPU count and NVLink topology matrix
+    3. torch / vllm / guidellm / huggingface-hub installed
+    4. HF cache, experiments, and guidellm report directories created
+    5. HF_HOME pointed at the cache
+
+    Raises on the first unmet requirement.
+    """
+    ensure_cuda_supported()
+
+    gpu_count, nvlink_available, topo_raw = parse_gpu_topology()
+    console.print(f"[bold]Detected GPUs:[/bold] {gpu_count}")
+    console.print(f"[bold]NVLink available:[/bold] {nvlink_available}")
+    console.print("[bold]Topology matrix:[/bold]\n" + topo_raw)
+
+    if install:
+        ensure_dependencies()
+
+    hf_home, experiments_dir, reports_dir = prepare_workspace(hf_home, experiments_dir, reports_dir)
+
+    return gpu_count, nvlink_available, topo_raw, hf_home, experiments_dir, reports_dir

runtime_harness/cli.py

@@ -0,0 +1,142 @@
+"""Command line entry point for the `rex` runtime experiment harness."""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import shutil
+import sys
+from pathlib import Path
+
+from rich.console import Console
+
+from . import __version__
+from .banner import harness_banner
+from .checks import DEFAULT_EXPERIMENTS_DIR, DEFAULT_HF_HOME, DEFAULT_REPORTS_DIR, run_preflight
+from .experiments import BUNDLED_CONFIG, DEFAULT_CONFIG_NAME, run_all_experiments
+
+console = Console()
+
+
+def cmd_check(args) -> int:
+    # Printed directly, not via rich: the banner already carries its own ANSI
+    # colour, and rich would re-parse those escapes as literal text.
+    print(harness_banner(__version__))
+    run_preflight(
+        install=not args.no_install,
+        hf_home=args.hf_home,
+        experiments_dir=args.experiments_dir,
+        reports_dir=args.reports_dir,
+    )
+    console.print("[bold green]Host ready.[/bold green] Run [bold]rex run[/bold] to start experiments.")
+    return 0
+
+
+def cmd_run(args) -> int:
+    asyncio.run(
+        run_all_experiments(
+            args.config,
+            install=not args.no_install,
+            output_dir=args.output_dir,
+            hf_home=args.hf_home,
+            experiments_dir=args.experiments_dir,
+            reports_dir=args.reports_dir,
+        )
+    )
+    return 0
+
+
+def cmd_init(args) -> int:
+    destination = args.path
+    if destination.exists() and not args.force:
+        console.print(
+            f"[yellow]{destination} already exists. Pass --force to overwrite.[/yellow]"
+        )
+        return 1
+    shutil.copyfile(BUNDLED_CONFIG, destination)
+    console.print(f"[green]Wrote starter manifest to {destination}[/green]")
+    return 0
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="rex",
+        description="Runtime experiment harness for vLLM / guidellm on NVLink GPU clusters.",
+    )
+    parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    def add_setup_flags(sub):
+        sub.add_argument(
+            "--hf-home",
+            type=Path,
+            default=None,
+            help=f"Hugging Face cache directory (default: {DEFAULT_HF_HOME}).",
+        )
+        sub.add_argument(
+            "--experiments-dir",
+            type=Path,
+            default=None,
+            help=f"Directory for per-experiment results (default: ./{DEFAULT_EXPERIMENTS_DIR}).",
+        )
+        sub.add_argument(
+            "--reports-dir",
+            type=Path,
+            default=None,
+            help=f"Directory for guidellm JSON reports (default: {DEFAULT_REPORTS_DIR}).",
+        )
+        sub.add_argument(
+            "--no-install",
+            action="store_true",
+            help="Skip installing torch/vllm/guidellm/huggingface-hub; verify and set up only.",
+        )
+
+    check = subparsers.add_parser(
+        "check",
+        help="Verify CUDA and topology, install runtime deps, and prepare the workspace.",
+    )
+    add_setup_flags(check)
+    check.set_defaults(func=cmd_check)
+
+    run = subparsers.add_parser("run", help="Run the experiment suite from a YAML manifest.")
+    run.add_argument(
+        "--config",
+        type=Path,
+        default=Path(DEFAULT_CONFIG_NAME),
+        help=f"Path to the experiments manifest (default: ./{DEFAULT_CONFIG_NAME}).",
+    )
+    run.add_argument(
+        "--output-dir",
+        type=Path,
+        default=None,
+        help="Directory for plots and experiment_results.json (default: current directory).",
+    )
+    add_setup_flags(run)
+    run.set_defaults(func=cmd_run)
+
+    init = subparsers.add_parser("init", help="Write a starter experiments.yaml.")
+    init.add_argument(
+        "path",
+        type=Path,
+        nargs="?",
+        default=Path(DEFAULT_CONFIG_NAME),
+        help=f"Where to write the manifest (default: ./{DEFAULT_CONFIG_NAME}).",
+    )
+    init.add_argument("--force", action="store_true", help="Overwrite an existing file.")
+    init.set_defaults(func=cmd_init)
+
+    return parser
+
+
+def main(argv=None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    try:
+        return args.func(args)
+    except Exception as exc:
+        console.print(f"[bold red]Experiment harness failed:[/bold red] {exc}")
+        return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())

runtime_harness/data/experiments.yaml

@@ -0,0 +1,48 @@
+gpu_counts:
+  - 2
+
+experiments:
+  - name: "qwen-3.6-35b-a3b-sxm"
+    model: "Qwen/Qwen3.6-35B-A3B-FP8"
+    prompt: "Summarize the following text: The NVIDIA H100 SXM platform with NVLink provides extremely high bandwidth for large-scale multi-GPU inference."
+    batch_size: 4
+    sequence_length: 2048
+    num_steps: 40
+    ep: 1
+    dp: 1
+    tp: 2
+    pp: 1
+    vllm_args:
+      - "--trust-remote-code"
+      - "--enable-expert-parallel"
+      - "--tensor-parallel-size"
+      - "2"
+      - "--gpu-memory-utilization"
+      - "0.95"
+      - "--max-num-batched-tokens"
+      - "8192"
+      - "--max-num-seqs"
+      - "256"
+      - "--max-model-len"
+      - "auto"
+      - "--enable-auto-tool-choice"
+      - "--tool-call-parser"
+      - "qwen3_xml"
+      - "--reasoning-parser"
+      - "qwen3"
+      - "--mm-encoder-tp-mode"
+      - "data"
+    guidellm_command:
+      - "run"
+      - "--backend"
+      - "kind=openai_http,target=http://localhost:8000/v1,model=Qwen/Qwen3.6-35B-A3B-FP8,max_tokens=1024"
+      - "--profile"
+      - "kind=concurrent,streams=16,warmup=0.1,cooldown=0.1"
+      - "--constraint"
+      - "kind=max_duration,seconds=300"
+      - "--constraint"
+      - "kind=over_saturation"
+      - "--data"
+      - "kind=synthetic_text,prompt_tokens=16384,output_tokens=1024"
+      - "--output"
+      - "kind=json,path=/workspace/tp2-ep-prof-run.json"

runtime_harness/experiments.py

@@ -0,0 +1,424 @@
+"""Experiment orchestration: launch vLLM, drive guidellm, collect results."""
+
+from __future__ import annotations
+
+import asyncio
+import dataclasses
+import json
+import os
+import re
+import shlex
+import shutil
+import subprocess
+import sys
+import urllib.request
+from datetime import datetime, timezone
+from pathlib import Path
+
+try:
+    import plotly.graph_objects as go
+except ModuleNotFoundError:  # pragma: no cover - handled when optional plotting dependency is absent.
+    go = None
+
+import yaml
+from rich.console import Console
+
+from . import __version__
+from .banner import harness_banner
+from .checks import DEFAULT_EXPERIMENTS_DIR, run_preflight
+
+console = Console()
+
+# Config and outputs resolve against the invocation directory, never the install
+# location -- site-packages is often read-only and is never the user's workspace.
+DEFAULT_CONFIG_NAME = "experiments.yaml"
+BUNDLED_CONFIG = Path(__file__).parent / "data" / DEFAULT_CONFIG_NAME
+
+
+def utc_timestamp() -> str:
+    """Filename-safe UTC stamp: 20260803T142305Z.
+
+    Deliberately not literal `date:time` -- colons are illegal in filenames on
+    macOS and Windows and need quoting in every shell command that touches them.
+    """
+    return datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+
+
+def experiment_id(experiment: "Experiment", gpu_count: int) -> str:
+    """Stable identifier for one (experiment, GPU count) pair in the sweep."""
+    return f"{experiment.name}-gpu{gpu_count}"
+
+
+def save_experiment_result(result: dict, experiments_dir: Path, exp_id: str, timestamp=None) -> Path:
+    """Write one guidellm result to experiments/<id>/<timestamp>_<id>.json.
+
+    If guidellm was configured to emit its own JSON report (via
+    `--output kind=json,path=...`), that artifact is copied in alongside so the
+    run directory is self-contained.
+    """
+    timestamp = timestamp or utc_timestamp()
+    run_dir = Path(experiments_dir) / exp_id
+    run_dir.mkdir(parents=True, exist_ok=True)
+
+    output_file = run_dir / f"{timestamp}_{exp_id}.json"
+    payload = dict(result)
+    payload["experiment_id"] = exp_id
+    payload["timestamp"] = timestamp
+    payload["latency"] = parse_latency_from_output(result.get("stdout", ""))
+
+    with output_file.open("w", encoding="utf-8") as handle:
+        json.dump(payload, handle, indent=2)
+
+    guidellm_report = result.get("guidellm_output_path")
+    if guidellm_report and Path(guidellm_report).is_file():
+        shutil.copyfile(guidellm_report, run_dir / f"{timestamp}_{exp_id}_guidellm.json")
+
+    console.log(f"[green]Saved {exp_id} result to {output_file}[/green]")
+    return output_file
+
+
+def guidellm_output_path(experiment: "Experiment"):
+    """Extract the report path guidellm was told to write, if any."""
+    for arg in list(experiment.guidellm_command) + list(experiment.guidellm_args):
+        match = re.search(r"kind=json,\s*path=([^,\s]+)", arg)
+        if match:
+            return match.group(1)
+    return None
+
+
+@dataclasses.dataclass
+class Experiment:
+    name: str
+    model: str
+    prompt: str
+    batch_size: int
+    sequence_length: int
+    num_steps: int
+    ep: int
+    dp: int
+    tp: int
+    pp: int
+    vllm_args: list[str] = dataclasses.field(default_factory=list)
+    guidellm_args: list[str] = dataclasses.field(default_factory=list)
+    guidellm_command: list[str] = dataclasses.field(default_factory=list)
+    metadata: dict = dataclasses.field(default_factory=dict)
+
+
+def normalize_arg_list(raw):
+    if raw is None:
+        return []
+    if isinstance(raw, str):
+        return shlex.split(raw)
+    if isinstance(raw, list):
+        return [str(item) for item in raw]
+    raise ValueError("Argument lists must be strings or lists.")
+
+
+def load_experiments(config_path: Path):
+    with config_path.open("r", encoding="utf-8") as handle:
+        config = yaml.safe_load(handle)
+
+    if not isinstance(config, dict):
+        raise ValueError("Configuration file must contain a YAML mapping.")
+
+    gpu_counts = config.get("gpu_counts", [2, 4, 8])
+    experiments = []
+    for raw in config.get("experiments", []):
+        experiment = Experiment(
+            name=raw["name"],
+            model=raw["model"],
+            prompt=raw["prompt"],
+            batch_size=int(raw.get("batch_size", 1)),
+            sequence_length=int(raw.get("sequence_length", 512)),
+            num_steps=int(raw.get("num_steps", 10)),
+            ep=int(raw.get("ep", 1)),
+            dp=int(raw.get("dp", 1)),
+            tp=int(raw.get("tp", 1)),
+            pp=int(raw.get("pp", 1)),
+            vllm_args=normalize_arg_list(raw.get("vllm_args")),
+            guidellm_args=normalize_arg_list(raw.get("guidellm_args")),
+            guidellm_command=normalize_arg_list(raw.get("guidellm_command")),
+            metadata=raw.get("metadata", {}),
+        )
+        experiments.append(experiment)
+
+    return gpu_counts, experiments
+
+
+def build_vllm_command(experiment: Experiment, gpu_count: int):
+    command = [sys.executable, "-m", "vllm", "serve", experiment.model]
+
+    if experiment.tp and not any(arg.startswith("--tensor-parallel-size") for arg in experiment.vllm_args):
+        command += ["--tensor-parallel-size", str(experiment.tp)]
+
+    if experiment.ep > 1 and "--enable-expert-parallel" not in experiment.vllm_args:
+        command.append("--enable-expert-parallel")
+
+    command += experiment.vllm_args
+    return command
+
+
+def vllm_port(experiment: Experiment, default: int = 8000) -> int:
+    """Port the manifest asked vLLM to serve on, so readiness polls the right place."""
+    args = experiment.vllm_args
+    for index, arg in enumerate(args):
+        if arg == "--port" and index + 1 < len(args):
+            return int(args[index + 1])
+        if arg.startswith("--port="):
+            return int(arg.split("=", 1)[1])
+    return default
+
+
+def server_is_ready(port: int, timeout: float = 2.0) -> bool:
+    url = f"http://localhost:{port}/health"
+    try:
+        with urllib.request.urlopen(url, timeout=timeout) as response:
+            return 200 <= response.status < 300
+    except Exception:
+        return False
+
+
+def build_guidellm_command(experiment: Experiment):
+    if experiment.guidellm_command:
+        return [sys.executable, "-m", "guidellm"] + experiment.guidellm_command
+
+    command = [
+        sys.executable,
+        "-m",
+        "guidellm",
+        "run",
+        "--name",
+        experiment.name,
+        "--model",
+        experiment.model,
+        "--prompt",
+        experiment.prompt,
+        "--batch-size",
+        str(experiment.batch_size),
+        "--sequence-length",
+        str(experiment.sequence_length),
+        "--num-steps",
+        str(experiment.num_steps),
+    ]
+    command += experiment.guidellm_args
+    return command
+
+
+async def start_vllm_server(experiment: Experiment, gpu_count: int, startup_timeout: float = 900.0):
+    """Launch vLLM with the manifest's parameters and wait until it serves traffic.
+
+    Loading a large checkpoint can take many minutes on a cold HF cache, so this
+    polls /health until ready rather than assuming a fixed startup delay.
+    """
+    console.rule(f"Starting vLLM server: {experiment.name} on {gpu_count} GPUs")
+    command = build_vllm_command(experiment, gpu_count)
+    port = vllm_port(experiment)
+    env = os.environ.copy()
+    env["CUDA_VISIBLE_DEVICES"] = ",".join(str(i) for i in range(gpu_count))
+
+    console.log(f"[blue]vLLM command:[/blue] {shlex.join(command)}")
+
+    process = subprocess.Popen(
+        command,
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.PIPE,
+        text=True,
+        env=env,
+    )
+
+    def failure(stderr: str):
+        return {
+            "name": experiment.name,
+            "engine": "vllm",
+            "gpu_count": gpu_count,
+            "port": port,
+            "success": False,
+            "stdout": "",
+            "stderr": stderr,
+        }, None
+
+    deadline = asyncio.get_running_loop().time() + startup_timeout
+    while True:
+        if process.poll() is not None:
+            stderr = process.stderr.read() if process.stderr else ""
+            console.log(f"[red]vLLM server exited during startup for {experiment.name}[/red]")
+            console.log(stderr)
+            return failure(stderr)
+
+        if await asyncio.to_thread(server_is_ready, port):
+            break
+
+        if asyncio.get_running_loop().time() >= deadline:
+            console.log(
+                f"[red]vLLM server did not become ready within {startup_timeout:.0f}s "
+                f"for {experiment.name}[/red]"
+            )
+            process.terminate()
+            return failure(f"Timed out waiting for http://localhost:{port}/health")
+
+        await asyncio.sleep(3)
+
+    console.log(f"[green]vLLM server ready for {experiment.name} on port {port}[/green]")
+    return {
+        "name": experiment.name,
+        "engine": "vllm",
+        "gpu_count": gpu_count,
+        "port": port,
+        "success": True,
+        "stdout": "",
+        "stderr": "",
+    }, process
+
+
+def run_guidellm(experiment: Experiment, reports_dir=None, exp_id=None, timestamp=None):
+    """Run guidellm, directing its JSON report into the reports directory.
+
+    A manifest that already specifies `--output kind=json,path=...` is left
+    alone -- an explicit path in the config beats our default.
+    """
+    console.rule(f"Running guidellm: {experiment.name}")
+    command = build_guidellm_command(experiment)
+
+    report_path = guidellm_output_path(experiment)
+    if report_path is None and reports_dir is not None:
+        exp_id = exp_id or experiment.name
+        timestamp = timestamp or utc_timestamp()
+        report_path = str(Path(reports_dir) / f"{timestamp}_{exp_id}_guidellm.json")
+        Path(report_path).parent.mkdir(parents=True, exist_ok=True)
+        command += ["--output", f"kind=json,path={report_path}"]
+
+    process = subprocess.run(command, capture_output=True, text=True)
+    success = process.returncode == 0
+    if not success:
+        console.log(f"[red]guidellm failed for {experiment.name}[/red]")
+        console.log(process.stderr)
+    return {
+        "name": experiment.name,
+        "engine": "guidellm",
+        "success": success,
+        "stdout": process.stdout,
+        "stderr": process.stderr,
+        "guidellm_output_path": report_path,
+    }
+
+
+def parse_latency_from_output(output: str):
+    match = re.search(r"latency[:=]\s*([0-9]+(?:\.[0-9]+)?)", output, re.IGNORECASE)
+    if match:
+        return float(match.group(1))
+    return None
+
+
+async def plot_results_async(results: list[dict], output_dir: Path):
+    if go is None:
+        console.log("[yellow]Plotly is not installed; skipping HTML result plotting.[/yellow]")
+        return None
+
+    output_dir.mkdir(parents=True, exist_ok=True)
+    output_file = output_dir / f"results_{len(results)}.html"
+
+    fig = go.Figure()
+    for result in results:
+        latency = parse_latency_from_output(result.get("stdout", ""))
+        score = result.get("metadata", {}).get("score")
+        label = f"{result['name']}:{result['engine']}"
+        if latency is not None:
+            fig.add_trace(go.Bar(name=label, x=[label], y=[latency], marker_color="#1f77b4"))
+        elif score is not None:
+            fig.add_trace(go.Bar(name=label, x=[label], y=[score], marker_color="#ff7f0e"))
+        else:
+            fig.add_trace(go.Bar(name=label, x=[label], y=[0], marker_color="#2ca02c"))
+
+    fig.update_layout(
+        title="Experiment Results",
+        xaxis_title="Experiment",
+        yaxis_title="Metric",
+        barmode="group",
+        template="plotly_dark",
+    )
+
+    await asyncio.to_thread(fig.write_html, str(output_file), include_plotlyjs="cdn")
+    console.log(f"[green]Saved plot to {output_file}[/green]")
+    return output_file
+
+
+def render_runtime_banner(model: str | None = None, version: str | None = None) -> str:
+    """Startup artwork. Delegates to the block-font renderer in banner.py."""
+    return harness_banner(version or __version__, model)
+
+
+async def run_all_experiments(
+    config_path: Path,
+    install: bool = True,
+    output_dir: Path | None = None,
+    hf_home: Path | None = None,
+    experiments_dir: Path | None = None,
+    reports_dir: Path | None = None,
+):
+    output_dir = Path(output_dir) if output_dir is not None else Path.cwd()
+
+    # See cmd_check: the banner carries its own ANSI colour, so bypass rich.
+    print(render_runtime_banner())
+
+    if not config_path.exists():
+        raise FileNotFoundError(
+            f"No experiment manifest at {config_path}. Run 'rex init' to write a starter manifest."
+        )
+
+    gpu_count, _nvlink, _topo, _hf_home, experiments_dir, reports_dir = run_preflight(
+        install=install,
+        hf_home=hf_home,
+        experiments_dir=experiments_dir if experiments_dir is not None else output_dir / DEFAULT_EXPERIMENTS_DIR,
+        reports_dir=reports_dir,
+    )
+
+    gpu_counts, experiments = load_experiments(config_path)
+    results = []
+    pending_plots = []
+
+    desired_gpu_counts = [count for count in gpu_counts if count <= gpu_count]
+    if not desired_gpu_counts:
+        raise ValueError(f"No supported GPU counts found for this machine. Detected {gpu_count}.")
+
+    for exp in experiments:
+        for count in desired_gpu_counts:
+            exp_id = experiment_id(exp, count)
+            timestamp = utc_timestamp()
+            vllm_result, server_process = await start_vllm_server(exp, count)
+            results.append(vllm_result)
+
+            if vllm_result["success"]:
+                guidellm_result = run_guidellm(exp, reports_dir, exp_id, timestamp)
+                results.append(guidellm_result)
+            else:
+                guidellm_result = {
+                    "name": exp.name,
+                    "engine": "guidellm",
+                    "success": False,
+                    "stdout": "",
+                    "stderr": "vLLM server failed to start.",
+                }
+                results.append(guidellm_result)
+
+            guidellm_result["gpu_count"] = count
+            save_experiment_result(guidellm_result, experiments_dir, exp_id, timestamp)
+
+            if server_process is not None and server_process.poll() is None:
+                server_process.terminate()
+                try:
+                    server_process.wait(timeout=10)
+                except subprocess.TimeoutExpired:
+                    server_process.kill()
+
+            task = asyncio.create_task(plot_results_async(results.copy(), output_dir / "plots"))
+            pending_plots.append(task)
+
+    if pending_plots:
+        await asyncio.gather(*pending_plots)
+
+    output_json = output_dir / "experiment_results.json"
+    output_json.parent.mkdir(parents=True, exist_ok=True)
+    with output_json.open("w", encoding="utf-8") as handle:
+        json.dump(results, handle, indent=2)
+    console.log(f"[green]Saved results to {output_json}[/green]")
+    return results