vllm-benchmark 0.1.0__tar.gz

vllm_benchmark-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 viniciusCSreis
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vllm_benchmark-0.1.0/MANIFEST.in

@@ -0,0 +1,5 @@
+include LICENSE README.md RESULT_FORMAT.md PUBLISHING.md
+recursive-include examples *.py *.yaml *.md
+recursive-include tests *.py
+prune examples/minimal/results
+global-exclude __pycache__ *.py[cod]

vllm_benchmark-0.1.0/PKG-INFO

@@ -0,0 +1,363 @@
+Metadata-Version: 2.4
+Name: vllm-benchmark
+Version: 0.1.0
+Summary: Run script-defined vLLM benchmarks through Docker Compose or Modal.
+Author: viniciusCSreis
+License-Expression: MIT
+Project-URL: Repository, https://github.com/viniciusCSreis/vllm_benchmark
+Project-URL: Issues, https://github.com/viniciusCSreis/vllm_benchmark/issues
+Keywords: vllm,benchmark,docker,modal,inference
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: System :: Benchmark
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic<3,>=2.12
+Requires-Dist: PyYAML>=6
+Requires-Dist: requests>=2.26
+Requires-Dist: prometheus-client>=0.18
+Requires-Dist: matplotlib>=3.7
+Provides-Extra: modal
+Requires-Dist: modal<2,>=1.6.2; extra == "modal"
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == "dev"
+Requires-Dist: twine>=6; extra == "dev"
+Requires-Dist: ruff>=0.11; extra == "dev"
+Dynamic: license-file
+
+# vllm_bench
+
+Run user-defined vLLM benchmarks against existing service URLs or with services
+managed through Docker Compose or Modal. Benchmark
+scripts and the runner execute on your machine; startup scripts execute in
+containers. The same Python library supplies arguments and service addresses to
+both. The provider owns service infrastructure; the runner and reports are shared.
+
+## Install
+
+Requires Python 3.11 or newer on the host and in service images. Local runs
+require Docker Compose v2; Modal runs require a Modal account and the optional
+SDK. Once published on PyPI:
+
+```bash
+python -m pip install vllm-benchmark
+# Or, with Modal support:
+python -m pip install 'vllm-benchmark[modal]'
+```
+
+For development, clone this repository and run `python -m pip install -e '.[dev]'`.
+No vLLM installation or sibling repository is required on the host unless your
+own benchmark scripts need it. Project service images supply their own runtime
+dependencies, such as PyTorch, vLLM, or FAISS.
+
+The local provider mounts the installed `vllm_bench` package read-only into every
+service container. Modal copies it into the remote image. Providers override
+startup to run a wrapper that launches your configured script, samples GPUs,
+and handles shutdown. Images need no CLI installation or special entrypoint;
+no packages are installed at service startup. Service scripts can use
+`from vllm_bench import bench_util` without an initialization call.
+
+Try the [minimal example](examples/minimal/README.md) with an ordinary Python
+image and no model download:
+
+```bash
+cd examples/minimal
+vllm_bench run_scenarios --benchmark example --scenario baseline
+```
+
+## Commands and selection
+
+A project supplies `.vllm_bench/configs/base.yaml` plus environment configs:
+`local_cpu.yaml`, `local_cpu_arm.yaml`, and `modal.yaml`. The default environment
+is `local_cpu`; choose another with `--config`. Environment selection is separate
+from benchmark and scenario selection.
+
+Both `up` and `run_scenarios` require `--benchmark` and `--scenario`. The plural aliases
+`--benchmarks` and `--scenarios` work too. Each accepts one or more names or `'*'`
+on its own. Quote the wildcard so your shell does not expand it into filenames.
+
+```bash
+# Every scenario for one benchmark:
+vllm_bench run_scenarios --config local_cpu --benchmark example --scenarios '*'
+# A named scenario across all benchmarks:
+vllm_bench run_scenarios --benchmark '*' --scenario baseline
+# Every benchmark and its own scenarios:
+vllm_bench run_scenarios --benchmark '*' --scenario '*'
+# Explicit subsets:
+vllm_bench run_scenarios --benchmark example other --scenario baseline distributed
+# Start one environment and follow its logs:
+vllm_bench up --config local_cpu_arm --benchmark example --scenario baseline
+vllm_bench down
+vllm_bench report results/local-comparison/run001
+```
+
+A named scenario must exist in every selected benchmark. Unknown names fail
+before starting containers, stopping an active environment, or saving results.
+A scenario wildcard uses each benchmark's own scenarios; they need not match.
+`run_scenarios` creates fresh services for every selected benchmark/scenario/repeat and
+then generates reports. The former `run` command is now named `run_scenarios`.
+
+`up` must resolve to exactly one benchmark/scenario pair. It only attaches to an
+active environment when provider, image, benchmark, scenario, and arguments match.
+It streams logs;
+Ctrl+C stops services and saves output. Use `up -d` to leave it running, then
+`down` to stop it. `down` uses saved state, and `report` uses its result directory,
+so neither needs benchmark/scenario selectors or an environment config.
+
+`run_scenarios` stops an existing environment only after validating the complete selection.
+Failures retain partial results and continue; Ctrl+C stops the experiment.
+Cleanup failures preserve recovery state for `down` and stop further scheduling.
+Reports can be regenerated offline.
+
+`--current-dir /path/to/project` works before or after any command and sets the
+root for config lookup, scripts, snapshots, runtime state, and results. Relative
+report paths also resolve from this root. `--config` works before or after `up`
+or `run_scenarios`; it accepts an environment name or YAML path.
+
+## Run against existing services
+
+`run_only` executes exactly one benchmark/scenario once, regardless of `runs`.
+It does not start, stop, reconfigure, or restart services, and leaves any active
+`up` environment and its state file untouched. It uses the same benchmark executor
+and report format as `run_scenarios`. Managed runs use the startup/readiness
+routine shared with `up` and the provider shutdown used by `down` after each trial,
+including failed trials.
+
+```bash
+vllm_bench run_only --config local_cpu \
+  --benchmark example --scenario baseline \
+  --service-url leader=https://example.com:8000
+```
+
+Repeat `--service-url NAME=URL` for each required service. Names match the config's
+`services` keys. Supply every service selected by the scenario and benchmark
+service dependencies; unknown, duplicate, and missing names fail before execution.
+URLs must be HTTP(S) base URLs, including any reverse-proxy prefix. Each supplied
+service group describes one external endpoint; `replicas` controls managed services
+only. The client uses the supplied URLs for health checks, metrics, and benchmark
+requests. `internal=True` returns the same URLs in this mode, so services that call
+each other must also be able to reach those addresses.
+
+Only benchmark scripts need to exist locally. An external-only config can omit
+`base_image` and service `startup_script`; `up` and `run_scenarios` still require
+those fields. Set `global_args` and `service_args` to match the models already
+served. Selecting a scenario records its arguments; it does not change the remote
+server's coordinator or model. The old `run` command has been renamed to
+`run_scenarios`.
+
+Results are saved under `results/external-<name>/runNNN/`, with one trial,
+endpoint URLs, the client context, logs, metrics, and reports. External runs do not
+upload a project snapshot, transfer inputs, or collect server logs/GPU samples.
+When the client and servers share a filesystem, `get_service_path()` returns the
+local path unchanged. For cloud deployments, make inputs available to the servers
+first; `--dataset-path /server/path` passes that value as the benchmark's
+`dataset_path` argument. Your benchmark adapter decides how to use it.
+
+After health checks pass, `up` prints a copyable `run_only` command before the
+endpoint and log details, including the selected config, project, benchmark,
+scenario, and service URLs. You can start an environment with `up -d`, then run
+that command as often as needed before stopping it with `down`. Supply dataset paths
+visible to that existing environment when it cannot access the client's paths.
+
+## Configuration
+
+`base.yaml` defines shared services and scenarios. Benchmarks reference scenario names. For example:
+
+```yaml
+name: comparison
+services:
+  leader:
+    role: leader
+    startup_script: scripts/start_leader.py
+    health_port: 8000
+    metrics_path: /metrics
+  worker:
+    role: worker
+    startup_script: scripts/start_worker.py
+    health_port: 8000
+    metrics_path: /metrics
+scenarios:
+  baseline:
+    args:
+      coordinator: single_worker
+    services: [leader]
+  distributed:
+    args:
+      coordinator: round_robin
+    services: [leader, worker]
+benchmarks:
+  example:
+    script: scripts/benchmark.py
+    script_args:
+      dataset_percentage: 0.1
+    scenarios: [baseline, distributed]
+runs: 1
+startup_timeout: 1800
+benchmark_timeout: 3600
+sample_interval: 1
+```
+
+`local_cpu.yaml` selects the local image and model:
+
+```yaml
+provider: local
+base_image: my-project:cpu
+global_args:
+  model: Qwen/Qwen2.5-0.5B-Instruct
+```
+
+`local_cpu_arm.yaml` uses the same structure with an ARM64 CPU image.
+An environment config automatically merges over its sibling `base.yaml`.
+Mappings merge recursively; child lists and scalar values replace base values.
+`extends: other_config` can inherit another environment before applying overrides.
+Saved runs contain the resolved environment, shared scenario definitions, and
+selected benchmarks with their selected scenario references. There are no separate benchmark YAML files or `benchmark_scenarios` field.
+
+Scenarios are defined once in the top-level `scenarios` mapping. Each benchmark
+selects a nonempty list of unique scenario names. Every scenario declares a
+nonempty list of known service names. A benchmark may also declare `services: [icp]` to add dependencies to every selected scenario. The CLI combines both
+service lists, removes overlaps, and expands each service to its replica count.
+Scenario and benchmark arguments are arbitrary project data.
+Names use lowercase letters, digits, underscores, or hyphens, starting with a
+letter. Script paths are relative to the project root.
+
+Optional settings:
+
+- Service `replicas`: number of instances; defaults to one.
+- Service `gpus: all` or `gpus: ["0"]`: local NVIDIA GPU allocation. Omit for CPU.
+- Service `environment`: extra string environment values.
+- Service `service_args`: arguments that override `global_args` for that service.
+  Startup scripts use `bench_util.get_service_args()` to read the merged values;
+  clients can use `bench_util.get_service_args("cascade")` for a named service.
+  Unspecified arguments inherit global values; `get_global_args()` stays unchanged.
+- Service `health_path`: defaults to `/health`; omit `metrics_path` if unavailable.
+- `skip_volume_mount: true`: use project code already inside the image.
+- `image_workdir`: project directory for that mode; defaults to `/workspace/project`.
+
+Startup scripts bind to `0.0.0.0` on their assigned service port and run in the
+foreground. Returning stops the service. Images must support the selected
+hardware; a CUDA-only vLLM build cannot automatically run on CPU.
+
+## Modal
+
+Install `python -m pip install 'vllm-benchmark[modal]'` and authenticate with
+`modal setup`. Define `.vllm_bench/configs/modal.yaml`:
+
+```yaml
+provider: modal
+base_image: your-registry/benchmark:cuda
+global_args:
+  model: Qwen/Qwen2.5-0.5B-Instruct
+modal:
+  app: vllm-bench
+  gpu: T4:1
+  cpu: 8
+  memory: 16384
+  timeout: 14400
+  cache_volume: vllm-bench-model-cache
+```
+
+```bash
+vllm_bench run_scenarios --config modal --benchmark example --scenario '*'
+vllm_bench up --config modal --benchmark example --scenario baseline
+vllm_bench down
+```
+
+The registry image contains project dependencies; Modal copies the installed CLI
+library into it without installing packages at startup. Each trial uses one
+sandbox in the named Modal App. Services share its CPU/GPU allocation and use
+separate ports. Service `gpus` settings apply to local Compose; `modal.gpu` sets
+the remote allocation. Memory is in MiB, and timeout is the sandbox lifetime in
+seconds. The model-cache volume mounts at `/mnt/vllm-bench-cache`; the provider
+points `HF_HOME`, `VLLM_CACHE_ROOT`, and `XDG_CACHE_HOME` there. This keeps the
+mount separate from cache files already baked into the service image. Pin the registry image digest for reproducible runs: Modal can cache
+mutable tags after the registry image changes.
+
+Project snapshots and client-generated dataset files are uploaded. Benchmark
+scripts, metrics collection, and reports execute locally using HTTPS tunnels.
+Use `bench_util` for addresses, assigned ports, and file paths. Shutdown downloads
+service output and logs. If the sandbox exits early, available combined logs are
+saved as `modal.log`; service files may no longer be available.
+
+## Script library
+
+```python
+from vllm_bench import bench_util
+
+model = bench_util.get_global_arg("model")
+options = bench_util.get_service_args()
+coordinator = bench_util.get_scenario_arg("coordinator")
+fraction = bench_util.get_benchmark_arg("dataset_percentage", 1.0)
+repeat = bench_util.current_run
+benchmark = bench_util.current_benchmark
+output = bench_util.get_output_dir()
+```
+
+Other helpers:
+
+- `get_project_dir()` returns the project root in the calling environment.
+- `get_service()` returns the calling startup's name, role, replica, and URL;
+  it returns `None` in client benchmark scripts.
+- `get_leader_url()` and `get_worker_urls()` choose the correct addresses for
+  the calling environment. Disabled workers yield an empty list.
+- `get_service_url("icp")` returns a named service. For a replicated group, use
+  its individual name such as `worker-1`.
+- `get_service_url("icp", internal=True)` returns the container address when
+  constructing a benchmark payload that the leader will execute.
+- `get_service_path(local_path)` translates project/trial paths in a client
+  benchmark to paths readable inside the containers.
+- `get_gpu_count()`, `is_gpu_available()`, and `is_cpu()` inspect the calling
+  environment using `nvidia-smi` without importing PyTorch.
+
+Benchmark scripts write `response.json` in `get_output_dir()`. Arbitrary JSON
+is preserved. When present, a numeric `latency_sec` and a `results` list provide
+additional summary fields; process runtime is measured independently.
+
+## Files and infrastructure
+
+The CLI library is mounted read-only at `/opt/vllm-bench-lib/vllm_bench`,
+independently of the project mount and even with `skip_volume_mount: true`.
+Keep the host installation available while containers run. Compose uses
+`python3 -m vllm_bench.service` as its startup override and prepends the library
+location to the configured service `PYTHONPATH`.
+
+The project is mounted read-only at `/workspace/project`. Each trial directory
+is mounted writable at `/run/vllm-bench/trial`; `get_output_dir()` selects the
+client trial directory or a replica's service subdirectory. Docker logs and GPU
+samples are saved under `services/<group>-<replica>/`.
+
+The generated Compose file, resolved per-service contexts, and provider IDs are
+saved under the trial's `.provider/`. The active state is recorded under the
+project's `.vllm_bench/state.json`. Host ports bind to `127.0.0.1` and container addresses
+use Compose DNS. A named model-cache volume persists after `down`.
+
+By default, each run also saves a ZIP of nonignored project files. Results,
+environments, native binaries, caches, `.env` files, and generated wheels are
+excluded. The live local project mount reflects subsequent edits; keep files
+unchanged during a run when comparing its saved snapshot.
+
+See [RESULT_FORMAT.md](RESULT_FORMAT.md) for the saved contract and reports.
+
+## Development checks
+
+```bash
+python -m pip install -e '.[dev]'
+ruff check .
+python -m unittest discover -s tests
+python tests/compose_smoke.py --image python:3.12-slim
+```
+
+The opt-in Compose check launches HTTP stand-ins with Prometheus metrics. It
+checks lifecycle, endpoints, mounting, scenarios, collection, and reports without
+loading any model or allocating cloud resources.
+
+## Publishing and license
+
+See [PUBLISHING.md](PUBLISHING.md) for GitHub setup, package checks, and PyPI
+Trusted Publishing. The project uses the [MIT license](LICENSE).

vllm_benchmark-0.1.0/PUBLISHING.md

@@ -0,0 +1,96 @@
+# Publishing vllm-benchmark
+
+This directory is an independent Git repository. The distribution is named
+`vllm-benchmark`, the Python package and command are `vllm_bench`, and the source
+repository is `viniciusCSreis/vllm_benchmark`. The project uses the MIT license.
+No sibling checkout is needed to build, test, install, or publish it.
+
+## Create the public repository
+
+Create an empty public GitHub repository named `vllm_benchmark` under
+`viniciusCSreis`. Do not initialize it with a README or license; these files
+already exist locally. From this directory:
+
+```bash
+git add .
+git commit -m "Prepare standalone benchmark package"
+git remote add origin https://github.com/viniciusCSreis/vllm_benchmark.git
+git push -u origin HEAD
+```
+
+If `origin` already exists, inspect `git remote -v` and use `git remote set-url`
+instead of adding it again. Update `[project.urls]` in `pyproject.toml` if you
+choose another owner or repository name.
+
+## Check a release locally
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install -e '.[dev]'
+ruff check .
+python -m unittest discover -s tests
+python tests/compose_smoke.py --image python:3.12-slim
+python -m build
+python -m twine check dist/*
+```
+
+The Compose smoke check requires Docker and launches ordinary Python containers;
+it does not load a model or allocate cloud GPUs. The build creates a wheel and
+source distribution in `dist/`. Use an empty `dist/` for each release so older
+versions are not uploaded with it.
+
+The version is defined only in `vllm_bench/__init__.py`; the initial release is
+`0.1.0`. Change it before every later release. PyPI does not allow reusing an
+uploaded version's filenames.
+
+## Configure PyPI Trusted Publishing
+
+1. Create a PyPI account, verify your email, and enable two-factor authentication.
+2. In GitHub repository Settings, create an Actions environment named `pypi`.
+3. Open <https://pypi.org/manage/account/publishing/> and add a pending GitHub
+   publisher with these exact values:
+
+   | Field | Value |
+   | --- | --- |
+   | PyPI project name | `vllm-benchmark` |
+   | Owner | `viniciusCSreis` |
+   | Repository | `vllm_benchmark` |
+   | Workflow filename | `publish.yml` |
+   | Environment | `pypi` |
+
+The package name had no public PyPI project during preparation; PyPI still must
+accept the name when you register/publish. If it is unavailable, choose another
+`[project].name` and update this guide, install commands, and the workflow URL.
+The import name can remain `vllm_bench`.
+
+See PyPI's [pending publisher instructions](https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/)
+and [GitHub Actions publishing guide](https://docs.pypi.org/trusted-publishers/using-a-publisher/).
+The workflow authenticates with GitHub OIDC, so no PyPI API-token secret is needed.
+
+## Publish the release
+
+Wait for CI to pass, then create and push a version tag:
+
+```bash
+git tag v0.1.0
+git push origin v0.1.0
+```
+
+On GitHub, create and publish a Release using that existing tag. Publishing the
+release triggers `.github/workflows/publish.yml`; pushing a tag alone does not.
+The workflow runs checks, verifies the tag matches the package version, builds
+both distributions, and uploads them to PyPI through the `pypi` environment.
+If you configured environment reviewers, approve the waiting publish job.
+
+After the workflow succeeds, check installation in a fresh virtual environment:
+
+```bash
+python3 -m venv /tmp/vllm-benchmark-check
+/tmp/vllm-benchmark-check/bin/python -m pip install vllm-benchmark
+/tmp/vllm-benchmark-check/bin/vllm_bench --version
+```
+
+Users install Modal support with `python -m pip install 'vllm-benchmark[modal]'`.
+For later releases, update the version, commit and push, then repeat with the
+matching `vX.Y.Z` tag and GitHub Release.