vllm-optimizer 0.1.0a11__tar.gz

vllm_optimizer-0.1.0a11/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 vLLM Optimizer contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vllm_optimizer-0.1.0a11/PKG-INFO

@@ -0,0 +1,224 @@
+Metadata-Version: 2.4
+Name: vllm-optimizer
+Version: 0.1.0a11
+Summary: Benchmark-driven configuration optimization for vLLM inference servers
+License-Expression: MIT
+Project-URL: Repository, https://github.com/brtydse100/vllm-optimizer
+Project-URL: Issues, https://github.com/brtydse100/vllm-optimizer/issues
+Keywords: vllm,ai,llm,guidellm,benchmark,optimization,inference
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: Optuna<5,>=4.0
+Requires-Dist: PyYAML>=6.0
+Provides-Extra: runtime
+Requires-Dist: vllm<0.29,>=0.28; extra == "runtime"
+Requires-Dist: guidellm<0.8,>=0.7.3; extra == "runtime"
+Provides-Extra: test
+Requires-Dist: pytest<9,>=8; extra == "test"
+Dynamic: license-file
+
+# vLLM Optimizer
+
+vLLM Optimizer is a local-first benchmarking and optimization tool for vLLM
+serving configurations. Users define the parameters and workloads they care
+about; the `vllm-opt` CLI manages the server lifecycle, runs repeatable benchmarks,
+explores the search space, and reports which configurations performed best.
+
+vLLM Optimizer is alpha software targeting Linux with NVIDIA GPUs and
+Python 3.11–3.12. Install `vllm-optimizer`, import `vllm_optimizer`, and run
+`vllm-opt`. The former `vtune` aliases remain available for one release cycle.
+
+This independent community project is not affiliated with the vLLM project.
+
+**[Documentation](https://brtydse100.github.io/vllm-optimizer/)** ·
+**[Quick start](https://brtydse100.github.io/vllm-optimizer/getting-started/)** ·
+**[PyPI](https://pypi.org/project/vllm-optimizer/)**
+
+The current code is verified with vLLM 0.28.0 and GuideLLM 0.7.3 on WSL2
+with an RTX 3080. That host required
+  `VLLM_USE_V2_MODEL_RUNNER: "0"` because UVA was unavailable and
+  `VLLM_USE_FLASHINFER_SAMPLER: "0"` because the CUDA compiler toolkit was
+  not installed. Native Linux systems may not require these settings.
+
+Other combinations may work but are not yet verified.
+
+Each new trial stores a typed `execution` assignment in its trial result and
+manifest. a5/a6 runs may lack it. Reports show only statistics supplied by the
+benchmark backend; a7 offline regeneration corrects derived a6 summaries in a
+new destination without changing the source run.
+
+The published `py3-none-any` wheel installs on Linux and Windows. Configuration
+validation and stored-result inspection work on Windows, but starting an
+experiment is supported only on Linux because vLLM has no native Windows
+runtime.
+
+## Installation
+
+Choose the installation that matches what you want to do:
+
+| Goal | Command | Platform |
+| --- | --- | --- |
+| Run complete experiments | `pip install "vllm-optimizer[runtime]"` | Linux/WSL with NVIDIA GPU |
+| Read configs, results, and reports | `pip install vllm-optimizer` | Linux, Windows, or macOS |
+
+The core package intentionally does not install GPU frameworks. The `runtime`
+extra adds vLLM and GuideLLM, which select large PyTorch/CUDA dependencies for
+the machine. See the [installation guide](https://brtydse100.github.io/vllm-optimizer/installation/)
+for virtual environments, CUDA guidance, and verification commands.
+
+## Quick start
+
+Create and activate a Python 3.11 or 3.12 virtual environment on Linux or WSL,
+then install the complete experiment runtime:
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install "vllm-optimizer[runtime]"
+vllm --help
+guidellm --help
+```
+
+Create `experiment.yaml`:
+
+```yaml
+experiment:
+  name: first-run
+server:
+  model: /models/opt-125m
+  gpu-memory-utilization: 0.8
+tune:
+  max-num-seqs:
+    values: [8, 16]
+benchmark:
+  engine: guidellm  # Default. Use vllm for `vllm bench serve`.
+  runs:
+    - name: throughput
+      profile:
+        kind: throughput
+        max_concurrency: 16
+      constraints:
+        - kind: max_requests
+          count: 10
+      data:
+        - kind: synthetic_text
+          prompt_tokens: 32
+          output_tokens: 16
+optimization:
+  maximize: output_tokens_per_second
+  sampler: tpe
+  trials: 2
+timeouts:
+  benchmark: 20m
+```
+
+Run it:
+
+```bash
+vllm-opt --config experiment.yaml
+```
+
+The short form is `vllm-opt -c experiment.yaml`. The command validates the file,
+runs the experiment, persists results, and generates its exports and report.
+The `vllm-opt` CLI binds vLLM to `127.0.0.1` by default. Set `server.host` explicitly
+only when the benchmark server must be reachable from another host.
+
+Fixed vLLM flags go directly under `server`; tunable flags use top-level
+`tune`. Fixed and tunable environment variables use `env` and `tune_env`.
+See the [configuration guide](https://brtydse100.github.io/vllm-optimizer/configuration/)
+for categorical, boolean, integer-range, float-range, list, and environment
+examples. The [complete YAML](https://brtydse100.github.io/vllm-optimizer/full-example/)
+and [benchmark guide](https://brtydse100.github.io/vllm-optimizer/benchmarking/) show
+every supported control with copyable examples.
+
+To use vLLM's native benchmark, set `benchmark.engine: vllm`. Its `args`
+map directly to `vllm bench serve` flags; the `vllm-opt` CLI supplies the model, server
+address, and JSON output path:
+
+```yaml
+benchmark:
+  engine: vllm
+  runs:
+    - name: throughput
+      args:
+        dataset-name: random
+        random-input-len: 32
+        random-output-len: 16
+        num-prompts: 100
+        request-rate: inf
+        max-concurrency: 16
+```
+
+Interactive terminal output uses color and remains concise by default. Set the
+standard `NO_COLOR` environment variable to disable color. To stream server and
+benchmark logs:
+
+```bash
+vllm-opt --config experiment.yaml --verbose
+```
+
+The persistent equivalent uses GuideLLM's logging level names:
+
+```yaml
+logging:
+  level: DEBUG
+```
+
+Supported levels are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`.
+Full per-trial log files are always saved. `--verbose` overrides the configured
+level with `DEBUG` for that invocation.
+
+Retry one or more selected trials into a new immutable linked run:
+
+```bash
+vllm-opt retry --run runs/EXPERIMENT/RUN_ID \
+  --trial trial-0001 --trial trial-0004
+```
+
+The source run is never modified.
+
+Display every stored vLLM and GuideLLM command for a trial without executing
+anything:
+
+```bash
+vllm-opt reproduce --run runs/EXPERIMENT/RUN_ID --trial trial-0001
+```
+
+Each completed run also contains a self-contained `report.html` decision
+dashboard with the best observed configuration, per-benchmark elapsed time,
+average/median/P99 latency, baseline comparison, score history,
+throughput/latency tradeoff, metric definitions, and observed parameter effects.
+
+Random and TPE runs never execute the same resolved configuration twice. If
+`optimization.trials` exceeds the unique search space, the `vllm-opt` CLI warns and runs
+every unique configuration once.
+
+Multiple independent trials can run on explicitly assigned, non-overlapping
+GPU sets and ports. A sequential or tensor-parallel server receives port 8000
+unless `server.port` overrides it; local-parallel trials use their configured
+port range. Sequential execution remains the default. See
+[parallel trials](https://brtydse100.github.io/vllm-optimizer/parallel-trials/) for the
+YAML and measurement caveats.
+
+## Product documents
+
+- [First MVP specification](docs/MVP_SPEC.md)
+- [Future implementation roadmap](docs/ROADMAP.md)
+- [Architecture overview and early sketch](docs/ARCHITECTURE.md)
+- [Editable Draw.io architecture diagram](docs/vllm-optimizer-architecture.drawio)
+- [Contributor guide](CONTRIBUTING.md)
+- [Release notes](CHANGELOG.md)
+
+The MVP specification defines the first releasable version and its acceptance
+criteria. The roadmap describes capabilities that should be designed for now
+but implemented after the core experiment loop is reliable.
+
+vLLM Optimizer is available under the [MIT License](LICENSE).

vllm_optimizer-0.1.0a11/README.md

@@ -0,0 +1,197 @@
+# vLLM Optimizer
+
+vLLM Optimizer is a local-first benchmarking and optimization tool for vLLM
+serving configurations. Users define the parameters and workloads they care
+about; the `vllm-opt` CLI manages the server lifecycle, runs repeatable benchmarks,
+explores the search space, and reports which configurations performed best.
+
+vLLM Optimizer is alpha software targeting Linux with NVIDIA GPUs and
+Python 3.11–3.12. Install `vllm-optimizer`, import `vllm_optimizer`, and run
+`vllm-opt`. The former `vtune` aliases remain available for one release cycle.
+
+This independent community project is not affiliated with the vLLM project.
+
+**[Documentation](https://brtydse100.github.io/vllm-optimizer/)** ·
+**[Quick start](https://brtydse100.github.io/vllm-optimizer/getting-started/)** ·
+**[PyPI](https://pypi.org/project/vllm-optimizer/)**
+
+The current code is verified with vLLM 0.28.0 and GuideLLM 0.7.3 on WSL2
+with an RTX 3080. That host required
+  `VLLM_USE_V2_MODEL_RUNNER: "0"` because UVA was unavailable and
+  `VLLM_USE_FLASHINFER_SAMPLER: "0"` because the CUDA compiler toolkit was
+  not installed. Native Linux systems may not require these settings.
+
+Other combinations may work but are not yet verified.
+
+Each new trial stores a typed `execution` assignment in its trial result and
+manifest. a5/a6 runs may lack it. Reports show only statistics supplied by the
+benchmark backend; a7 offline regeneration corrects derived a6 summaries in a
+new destination without changing the source run.
+
+The published `py3-none-any` wheel installs on Linux and Windows. Configuration
+validation and stored-result inspection work on Windows, but starting an
+experiment is supported only on Linux because vLLM has no native Windows
+runtime.
+
+## Installation
+
+Choose the installation that matches what you want to do:
+
+| Goal | Command | Platform |
+| --- | --- | --- |
+| Run complete experiments | `pip install "vllm-optimizer[runtime]"` | Linux/WSL with NVIDIA GPU |
+| Read configs, results, and reports | `pip install vllm-optimizer` | Linux, Windows, or macOS |
+
+The core package intentionally does not install GPU frameworks. The `runtime`
+extra adds vLLM and GuideLLM, which select large PyTorch/CUDA dependencies for
+the machine. See the [installation guide](https://brtydse100.github.io/vllm-optimizer/installation/)
+for virtual environments, CUDA guidance, and verification commands.
+
+## Quick start
+
+Create and activate a Python 3.11 or 3.12 virtual environment on Linux or WSL,
+then install the complete experiment runtime:
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install "vllm-optimizer[runtime]"
+vllm --help
+guidellm --help
+```
+
+Create `experiment.yaml`:
+
+```yaml
+experiment:
+  name: first-run
+server:
+  model: /models/opt-125m
+  gpu-memory-utilization: 0.8
+tune:
+  max-num-seqs:
+    values: [8, 16]
+benchmark:
+  engine: guidellm  # Default. Use vllm for `vllm bench serve`.
+  runs:
+    - name: throughput
+      profile:
+        kind: throughput
+        max_concurrency: 16
+      constraints:
+        - kind: max_requests
+          count: 10
+      data:
+        - kind: synthetic_text
+          prompt_tokens: 32
+          output_tokens: 16
+optimization:
+  maximize: output_tokens_per_second
+  sampler: tpe
+  trials: 2
+timeouts:
+  benchmark: 20m
+```
+
+Run it:
+
+```bash
+vllm-opt --config experiment.yaml
+```
+
+The short form is `vllm-opt -c experiment.yaml`. The command validates the file,
+runs the experiment, persists results, and generates its exports and report.
+The `vllm-opt` CLI binds vLLM to `127.0.0.1` by default. Set `server.host` explicitly
+only when the benchmark server must be reachable from another host.
+
+Fixed vLLM flags go directly under `server`; tunable flags use top-level
+`tune`. Fixed and tunable environment variables use `env` and `tune_env`.
+See the [configuration guide](https://brtydse100.github.io/vllm-optimizer/configuration/)
+for categorical, boolean, integer-range, float-range, list, and environment
+examples. The [complete YAML](https://brtydse100.github.io/vllm-optimizer/full-example/)
+and [benchmark guide](https://brtydse100.github.io/vllm-optimizer/benchmarking/) show
+every supported control with copyable examples.
+
+To use vLLM's native benchmark, set `benchmark.engine: vllm`. Its `args`
+map directly to `vllm bench serve` flags; the `vllm-opt` CLI supplies the model, server
+address, and JSON output path:
+
+```yaml
+benchmark:
+  engine: vllm
+  runs:
+    - name: throughput
+      args:
+        dataset-name: random
+        random-input-len: 32
+        random-output-len: 16
+        num-prompts: 100
+        request-rate: inf
+        max-concurrency: 16
+```
+
+Interactive terminal output uses color and remains concise by default. Set the
+standard `NO_COLOR` environment variable to disable color. To stream server and
+benchmark logs:
+
+```bash
+vllm-opt --config experiment.yaml --verbose
+```
+
+The persistent equivalent uses GuideLLM's logging level names:
+
+```yaml
+logging:
+  level: DEBUG
+```
+
+Supported levels are `DEBUG`, `INFO`, `WARNING`, `ERROR`, and `CRITICAL`.
+Full per-trial log files are always saved. `--verbose` overrides the configured
+level with `DEBUG` for that invocation.
+
+Retry one or more selected trials into a new immutable linked run:
+
+```bash
+vllm-opt retry --run runs/EXPERIMENT/RUN_ID \
+  --trial trial-0001 --trial trial-0004
+```
+
+The source run is never modified.
+
+Display every stored vLLM and GuideLLM command for a trial without executing
+anything:
+
+```bash
+vllm-opt reproduce --run runs/EXPERIMENT/RUN_ID --trial trial-0001
+```
+
+Each completed run also contains a self-contained `report.html` decision
+dashboard with the best observed configuration, per-benchmark elapsed time,
+average/median/P99 latency, baseline comparison, score history,
+throughput/latency tradeoff, metric definitions, and observed parameter effects.
+
+Random and TPE runs never execute the same resolved configuration twice. If
+`optimization.trials` exceeds the unique search space, the `vllm-opt` CLI warns and runs
+every unique configuration once.
+
+Multiple independent trials can run on explicitly assigned, non-overlapping
+GPU sets and ports. A sequential or tensor-parallel server receives port 8000
+unless `server.port` overrides it; local-parallel trials use their configured
+port range. Sequential execution remains the default. See
+[parallel trials](https://brtydse100.github.io/vllm-optimizer/parallel-trials/) for the
+YAML and measurement caveats.
+
+## Product documents
+
+- [First MVP specification](docs/MVP_SPEC.md)
+- [Future implementation roadmap](docs/ROADMAP.md)
+- [Architecture overview and early sketch](docs/ARCHITECTURE.md)
+- [Editable Draw.io architecture diagram](docs/vllm-optimizer-architecture.drawio)
+- [Contributor guide](CONTRIBUTING.md)
+- [Release notes](CHANGELOG.md)
+
+The MVP specification defines the first releasable version and its acceptance
+criteria. The roadmap describes capabilities that should be designed for now
+but implemented after the core experiment loop is reliable.
+
+vLLM Optimizer is available under the [MIT License](LICENSE).

vllm_optimizer-0.1.0a11/pyproject.toml

@@ -0,0 +1,38 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "vllm-optimizer"
+version = "0.1.0a11"
+description = "Benchmark-driven configuration optimization for vLLM inference servers"
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+license-files = ["LICENSE"]
+keywords = ["vllm", "ai", "llm", "guidellm", "benchmark", "optimization", "inference"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Environment :: Console",
+  "Operating System :: POSIX :: Linux",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Typing :: Typed",
+]
+dependencies = ["Optuna>=4.0,<5", "PyYAML>=6.0"]
+
+[project.optional-dependencies]
+runtime = ["vllm>=0.28,<0.29", "guidellm>=0.7.3,<0.8"]
+test = ["pytest>=8,<9"]
+
+[project.urls]
+Repository = "https://github.com/brtydse100/vllm-optimizer"
+Issues = "https://github.com/brtydse100/vllm-optimizer/issues"
+
+[project.scripts]
+vllm-opt = "vllm_optimizer.cli:main"
+vtune = "vllm_optimizer.cli:legacy_main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

vllm_optimizer-0.1.0a11/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

vllm_optimizer-0.1.0a11/src/vllm_optimizer/__init__.py

@@ -0,0 +1,7 @@
+"""vLLM Optimizer package."""
+
+__version__ = "0.1.0a11"
+
+from .orchestrator import Orchestrator, RunOutcome
+
+__all__ = ["Orchestrator", "RunOutcome"]

vllm_optimizer-0.1.0a11/src/vllm_optimizer/benchmarks/__init__.py

@@ -0,0 +1,13 @@
+"""Benchmark backend adapters."""
+
+from .configuration import (
+    configured_engine, configured_min_repeats, configured_repeats,
+    configured_runs, configured_warmup_repeats,
+)
+from .guidellm import GuideLLMPlan, build_plan, parse_result
+
+__all__ = [
+    "GuideLLMPlan", "build_plan", "configured_engine", "configured_min_repeats",
+    "configured_repeats", "configured_runs", "configured_warmup_repeats",
+    "parse_result",
+]

vllm_optimizer-0.1.0a11/src/vllm_optimizer/benchmarks/configuration.py

@@ -0,0 +1,84 @@
+"""Shared benchmark configuration validation."""
+
+from collections.abc import Mapping
+import re
+
+from vllm_optimizer.config.models import VTuneConfig
+
+_RUN_NAME = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]*$")
+_ENGINES = {"guidellm", "vllm"}
+
+
+def configured_engine(config: VTuneConfig) -> str:
+    value = config.benchmark.get("engine", "guidellm")
+    if not isinstance(value, str) or value not in _ENGINES:
+        raise ValueError("benchmark.engine must be 'guidellm' or 'vllm'")
+    return value
+
+
+def configured_repeats(config: VTuneConfig) -> int:
+    value = config.benchmark.get("repeats", 1)
+    if isinstance(value, bool) or not isinstance(value, int) or value < 1:
+        raise ValueError("benchmark.repeats must be a positive integer")
+    return value
+
+
+def configured_warmup_repeats(config: VTuneConfig) -> int:
+    value = config.benchmark.get("warmup_repeats", 0)
+    if isinstance(value, bool) or not isinstance(value, int) or value < 0:
+        raise ValueError("benchmark.warmup_repeats must be a non-negative integer")
+    return value
+
+
+def configured_min_repeats(config: VTuneConfig) -> int:
+    value = config.benchmark.get("min_repeats", 1)
+    if isinstance(value, bool) or not isinstance(value, int) or value < 1:
+        raise ValueError("benchmark.min_repeats must be a positive integer")
+    if value > configured_repeats(config):
+        raise ValueError("benchmark.min_repeats cannot exceed benchmark.repeats")
+    return value
+
+
+def configured_runs(config: VTuneConfig) -> tuple[Mapping[str, object], ...]:
+    unknown = set(config.benchmark) - {
+        "engine", "runs", "repeats", "warmup_repeats", "min_repeats",
+    }
+    if unknown:
+        raise ValueError(f"Unsupported benchmark setting(s): {', '.join(sorted(unknown))}")
+    values = config.benchmark.get("runs")
+    if not isinstance(values, list) or not values:
+        raise ValueError("'benchmark.runs' must be a non-empty list")
+    engine, names = configured_engine(config), set()
+    runs: list[Mapping[str, object]] = []
+    for index, value in enumerate(values):
+        run = _mapping(value, f"run {index}")
+        name = run.get("name")
+        if not isinstance(name, str) or not _RUN_NAME.fullmatch(name):
+            raise ValueError("benchmark run names must use letters, numbers, '_' or '-'")
+        if name in names:
+            raise ValueError(f"duplicate benchmark run name: {name}")
+        names.add(name)
+        _validate_run(engine, run, index, name)
+        runs.append(run)
+    return tuple(runs)
+
+
+def _validate_run(engine: str, run: Mapping[str, object], index: int, name: str) -> None:
+    allowed = ({"name", "request_format", "profile", "constraints", "data"}
+               if engine == "guidellm" else {"name", "args"})
+    if unknown := set(run) - allowed:
+        raise ValueError(
+            f"Unsupported setting(s) in benchmark run {index}: {', '.join(sorted(unknown))}"
+        )
+    if engine == "guidellm":
+        data = run.get("data")
+        if not isinstance(data, list) or len(data) != 1:
+            raise ValueError(f"benchmark run '{name}' must configure exactly one dataset")
+    else:
+        _mapping(run.get("args", {}), f"vllm benchmark run '{name}' args")
+
+
+def _mapping(value: object, label: str) -> Mapping[str, object]:
+    if not isinstance(value, dict) or any(not isinstance(key, str) for key in value):
+        raise ValueError(f"{label} must be an object")
+    return value