benchmaker 0.1.0__tar.gz

benchmaker-0.1.0/PKG-INFO

@@ -0,0 +1,214 @@
+Metadata-Version: 2.4
+Name: benchmaker
+Version: 0.1.0
+Summary: Async HTTP benchmarking utility with pluggable workloads and load models.
+Author: Xiaozhe Yao
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: aiohttp>=3.9
+Requires-Dist: click>=8.1
+Requires-Dist: pyyaml>=6.0
+Provides-Extra: rich
+Requires-Dist: rich>=13; extra == "rich"
+Provides-Extra: hf
+Requires-Dist: datasets>=2.18; extra == "hf"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+
+# bench-maker
+
+Async HTTP benchmarking with pluggable workload-types (protocols), workloads
+(datasets), load models, hooks, and optional periodic monitors.
+
+```text
++--------+   item   +---------------+   request   +-----------+   +---------+
+|workload|--------->| workload-type |------------>| pre-hooks |-->| aiohttp |
+|(dataset|          | (protocol)    |             +-----------+   +---------+
+| / log) |          | make_request  |                                 |
++--------+          | make_sample   |              +------------+     v
+   ^                +---------------+              | post-hooks |<----+
+   |                                               +------------+
+   +-- load model decides WHEN to fire ----+              v
+                                           |        +----------+
+              monitors run alongside ------+------->| metrics  |
+              (Prometheus, NVML, ...)               | aggregator|
+                                                    +----------+
+```
+
+## Install
+
+```bash
+pip install -e .
+pip install -e .[dev]   # for tests
+```
+
+This installs the `benchmaker` Python package and the `bench-maker` CLI.
+
+## 30-second tour
+
+```python
+import asyncio
+from benchmaker import BenchConfig, BenchRunner, ConstantRPS, HttpWorkloadType
+
+async def main():
+    cfg = BenchConfig(
+        workload_type=HttpWorkloadType(url="https://httpbin.org/get"),
+        load=ConstantRPS(rps=50, duration_s=10),
+    )
+    result = await BenchRunner(cfg).run()
+    print(result.summary)
+
+asyncio.run(main())
+```
+
+Or via the CLI:
+
+```bash
+bench-maker quick --url https://httpbin.org/get --rate poisson:50 --duration 10s
+```
+
+## Walkthrough: benchmarking an LLM endpoint with ShareGPT
+
+A realistic LLM benchmark needs a real prompt distribution.
+[ShareGPT V3](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered)
+is a common choice — multi-turn human/assistant conversations scraped from real
+ChatGPT users. A cleaned, benchmark-ready copy is published at
+[`researchcomputer/llmsys-bench`](https://huggingface.co/datasets/researchcomputer/llmsys-bench)
+(`split="sharegpt"`), with one row per conversation:
+
+```json
+{"id": "...", "messages": [{"role": "user", "content": "..."},
+                           {"role": "assistant", "content": "..."},
+                           {"role": "user", "content": "..."}]}
+```
+
+`messages` is the only content field — it's everything a chat benchmark needs.
+Each row is truncated to end on a **user** turn, so it's a valid generation
+request: the server completes the final assistant reply given the prior
+history. Short source conversations collapse to a single user turn (a plain
+single-turn prompt); longer ones carry multi-turn context.
+
+### Load it directly from the Hub
+
+Pull the published split and feed each row's `messages` list straight into the
+chat workload-type (`pip install -e .[hf]`):
+
+```python
+import asyncio
+from datasets import load_dataset
+from benchmaker import (
+    BenchConfig, BenchRunner, OpenAIChatWorkloadType,
+    IterableWorkload, parse_rate_spec,
+)
+
+async def main():
+    ds = load_dataset("researchcomputer/llmsys-bench", split="sharegpt")
+    cfg = BenchConfig(
+        workload_type=OpenAIChatWorkloadType(
+            url="http://localhost:8000/v1/chat/completions",
+            model="meta-llama/Llama-3.1-8B-Instruct",
+            max_tokens=256,
+        ),
+        workload=IterableWorkload(row["messages"] for row in ds),
+        load=parse_rate_spec("poisson:8", duration_s=60),
+        timeout_s=600,
+    )
+    result = await BenchRunner(cfg).run()
+    print(result.summary)
+
+asyncio.run(main())
+```
+
+`OpenAIChatWorkloadType` receives the message list as-is, so single-turn rows
+send one user message and multi-turn rows replay the full history before the
+server generates the final assistant turn. TTFT, inter-token latency, and
+tokens/sec are captured the same way in both cases. URL / model / API key can
+also come from `.env` via `OpenAIChatWorkloadType.from_env(...)`.
+
+### Rebuild or customize it yourself
+
+The published split is produced by `tools/prepare_sharegpt.py`, which downloads
+the upstream JSON once into `.local/` (gitignored) and converts it to the JSONL
+shape above. Run it when you want a subset, different filtering, or a refresh:
+
+```bash
+# Defaults: .local/sharegpt_v3_raw.json  ->  .local/sharegpt_v3.jsonl
+python tools/prepare_sharegpt.py
+
+# A quick subset for smoke tests:
+python tools/prepare_sharegpt.py --max-items 2000
+```
+
+The raw download is ~700 MB. Use `--min-chars` / `--max-chars` to drop empty or
+pathologically long conversations (measured over total message content per
+row). Point any workload at the local file with `JsonlWorkload(path=...,
+field="messages")`, or on the CLI:
+
+```bash
+bench-maker llm \
+    --url   http://localhost:8000/v1/chat/completions \
+    --model meta-llama/Llama-3.1-8B-Instruct \
+    --prompts-jsonl .local/sharegpt_v3.jsonl \
+    --prompt-field  messages \
+    --max-tokens 256 \
+    --rate poisson:8 --duration 60s \
+    --out-dir ./runs --label dataset=sharegpt
+```
+
+To re-publish after regenerating, `tools/upload_sharegpt_hf.py` pushes the
+JSONL back to the Hub (needs a write token).
+
+## Documentation
+
+Full docs live in [`docs/`](docs/):
+
+- [Quickstart](docs/quickstart.md)
+- [Concepts](docs/concepts.md) — WorkloadType, Workload, LoadModel, Monitor
+- [Load models](docs/load-models.md) — rate-spec syntax, open vs closed loop
+- [Workloads & workload-types](docs/workloads.md) — built-ins and custom subclasses
+- [Hooks](docs/hooks.md) — pre/post request processing
+- [Monitors](docs/monitors.md) — vLLM `/metrics`, GPU telemetry, custom samplers
+- [Metrics & output](docs/metrics.md) — summary structure, JSONL dumps
+- [Correctness / accuracy eval](docs/eval.md) — grade responses against references
+- [CLI & YAML reference](docs/cli-and-yaml.md)
+- [ShareGPT benchmark](docs/sharegpt-benchmark.md) — self-contained end-to-end walkthrough
+
+## Examples
+
+Under [`examples/`](examples/):
+
+- `simple_get.py`         — minimal library usage
+- `custom_hooks.py`       — request signing + response parsing
+- `llm_chat.py`           — OpenAI-compatible LLM endpoint with streaming
+- `vllm_with_monitor.py`  — LLM benchmark with concurrent vLLM `/metrics` scrape
+- `sandbox_exec.py`       — Flash Sandbox `/exec` latency benchmark
+- `sandbox_lifecycle.py`  — full create → exec → delete cold-start benchmark
+- `llm_eval.py`           — LLM benchmark + accuracy grading (exact/regex/judge)
+- `gsm8k_eval.py`         — GSM8K from HuggingFace + integer-match scorer
+- `config.yaml`           — generic HTTP YAML config
+- `config_llm.yaml`       — LLM YAML config with a Prometheus monitor
+
+Helper scripts under [`tools/`](tools/):
+
+- `prepare_sharegpt.py`   — fetch ShareGPT V3 and convert to a generic JSONL
+- `upload_sharegpt_hf.py` — push the converted JSONL to the HF Hub (write token)
+- `start_local_llm.sh`    — example local SGLang launch command
+
+## Project layout
+
+```
+benchmaker/          # library code
+entrypoints/         # CLI (bench-maker)
+examples/            # runnable examples
+tools/               # one-off helper scripts (dataset prep, etc.)
+tests/               # pytest smoke tests
+docs/                # reference docs
+```
+
+## Run the tests
+
+```bash
+pytest -q
+```

benchmaker-0.1.0/README.md

@@ -0,0 +1,195 @@
+# bench-maker
+
+Async HTTP benchmarking with pluggable workload-types (protocols), workloads
+(datasets), load models, hooks, and optional periodic monitors.
+
+```text
++--------+   item   +---------------+   request   +-----------+   +---------+
+|workload|--------->| workload-type |------------>| pre-hooks |-->| aiohttp |
+|(dataset|          | (protocol)    |             +-----------+   +---------+
+| / log) |          | make_request  |                                 |
++--------+          | make_sample   |              +------------+     v
+   ^                +---------------+              | post-hooks |<----+
+   |                                               +------------+
+   +-- load model decides WHEN to fire ----+              v
+                                           |        +----------+
+              monitors run alongside ------+------->| metrics  |
+              (Prometheus, NVML, ...)               | aggregator|
+                                                    +----------+
+```
+
+## Install
+
+```bash
+pip install -e .
+pip install -e .[dev]   # for tests
+```
+
+This installs the `benchmaker` Python package and the `bench-maker` CLI.
+
+## 30-second tour
+
+```python
+import asyncio
+from benchmaker import BenchConfig, BenchRunner, ConstantRPS, HttpWorkloadType
+
+async def main():
+    cfg = BenchConfig(
+        workload_type=HttpWorkloadType(url="https://httpbin.org/get"),
+        load=ConstantRPS(rps=50, duration_s=10),
+    )
+    result = await BenchRunner(cfg).run()
+    print(result.summary)
+
+asyncio.run(main())
+```
+
+Or via the CLI:
+
+```bash
+bench-maker quick --url https://httpbin.org/get --rate poisson:50 --duration 10s
+```
+
+## Walkthrough: benchmarking an LLM endpoint with ShareGPT
+
+A realistic LLM benchmark needs a real prompt distribution.
+[ShareGPT V3](https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered)
+is a common choice — multi-turn human/assistant conversations scraped from real
+ChatGPT users. A cleaned, benchmark-ready copy is published at
+[`researchcomputer/llmsys-bench`](https://huggingface.co/datasets/researchcomputer/llmsys-bench)
+(`split="sharegpt"`), with one row per conversation:
+
+```json
+{"id": "...", "messages": [{"role": "user", "content": "..."},
+                           {"role": "assistant", "content": "..."},
+                           {"role": "user", "content": "..."}]}
+```
+
+`messages` is the only content field — it's everything a chat benchmark needs.
+Each row is truncated to end on a **user** turn, so it's a valid generation
+request: the server completes the final assistant reply given the prior
+history. Short source conversations collapse to a single user turn (a plain
+single-turn prompt); longer ones carry multi-turn context.
+
+### Load it directly from the Hub
+
+Pull the published split and feed each row's `messages` list straight into the
+chat workload-type (`pip install -e .[hf]`):
+
+```python
+import asyncio
+from datasets import load_dataset
+from benchmaker import (
+    BenchConfig, BenchRunner, OpenAIChatWorkloadType,
+    IterableWorkload, parse_rate_spec,
+)
+
+async def main():
+    ds = load_dataset("researchcomputer/llmsys-bench", split="sharegpt")
+    cfg = BenchConfig(
+        workload_type=OpenAIChatWorkloadType(
+            url="http://localhost:8000/v1/chat/completions",
+            model="meta-llama/Llama-3.1-8B-Instruct",
+            max_tokens=256,
+        ),
+        workload=IterableWorkload(row["messages"] for row in ds),
+        load=parse_rate_spec("poisson:8", duration_s=60),
+        timeout_s=600,
+    )
+    result = await BenchRunner(cfg).run()
+    print(result.summary)
+
+asyncio.run(main())
+```
+
+`OpenAIChatWorkloadType` receives the message list as-is, so single-turn rows
+send one user message and multi-turn rows replay the full history before the
+server generates the final assistant turn. TTFT, inter-token latency, and
+tokens/sec are captured the same way in both cases. URL / model / API key can
+also come from `.env` via `OpenAIChatWorkloadType.from_env(...)`.
+
+### Rebuild or customize it yourself
+
+The published split is produced by `tools/prepare_sharegpt.py`, which downloads
+the upstream JSON once into `.local/` (gitignored) and converts it to the JSONL
+shape above. Run it when you want a subset, different filtering, or a refresh:
+
+```bash
+# Defaults: .local/sharegpt_v3_raw.json  ->  .local/sharegpt_v3.jsonl
+python tools/prepare_sharegpt.py
+
+# A quick subset for smoke tests:
+python tools/prepare_sharegpt.py --max-items 2000
+```
+
+The raw download is ~700 MB. Use `--min-chars` / `--max-chars` to drop empty or
+pathologically long conversations (measured over total message content per
+row). Point any workload at the local file with `JsonlWorkload(path=...,
+field="messages")`, or on the CLI:
+
+```bash
+bench-maker llm \
+    --url   http://localhost:8000/v1/chat/completions \
+    --model meta-llama/Llama-3.1-8B-Instruct \
+    --prompts-jsonl .local/sharegpt_v3.jsonl \
+    --prompt-field  messages \
+    --max-tokens 256 \
+    --rate poisson:8 --duration 60s \
+    --out-dir ./runs --label dataset=sharegpt
+```
+
+To re-publish after regenerating, `tools/upload_sharegpt_hf.py` pushes the
+JSONL back to the Hub (needs a write token).
+
+## Documentation
+
+Full docs live in [`docs/`](docs/):
+
+- [Quickstart](docs/quickstart.md)
+- [Concepts](docs/concepts.md) — WorkloadType, Workload, LoadModel, Monitor
+- [Load models](docs/load-models.md) — rate-spec syntax, open vs closed loop
+- [Workloads & workload-types](docs/workloads.md) — built-ins and custom subclasses
+- [Hooks](docs/hooks.md) — pre/post request processing
+- [Monitors](docs/monitors.md) — vLLM `/metrics`, GPU telemetry, custom samplers
+- [Metrics & output](docs/metrics.md) — summary structure, JSONL dumps
+- [Correctness / accuracy eval](docs/eval.md) — grade responses against references
+- [CLI & YAML reference](docs/cli-and-yaml.md)
+- [ShareGPT benchmark](docs/sharegpt-benchmark.md) — self-contained end-to-end walkthrough
+
+## Examples
+
+Under [`examples/`](examples/):
+
+- `simple_get.py`         — minimal library usage
+- `custom_hooks.py`       — request signing + response parsing
+- `llm_chat.py`           — OpenAI-compatible LLM endpoint with streaming
+- `vllm_with_monitor.py`  — LLM benchmark with concurrent vLLM `/metrics` scrape
+- `sandbox_exec.py`       — Flash Sandbox `/exec` latency benchmark
+- `sandbox_lifecycle.py`  — full create → exec → delete cold-start benchmark
+- `llm_eval.py`           — LLM benchmark + accuracy grading (exact/regex/judge)
+- `gsm8k_eval.py`         — GSM8K from HuggingFace + integer-match scorer
+- `config.yaml`           — generic HTTP YAML config
+- `config_llm.yaml`       — LLM YAML config with a Prometheus monitor
+
+Helper scripts under [`tools/`](tools/):
+
+- `prepare_sharegpt.py`   — fetch ShareGPT V3 and convert to a generic JSONL
+- `upload_sharegpt_hf.py` — push the converted JSONL to the HF Hub (write token)
+- `start_local_llm.sh`    — example local SGLang launch command
+
+## Project layout
+
+```
+benchmaker/          # library code
+entrypoints/         # CLI (bench-maker)
+examples/            # runnable examples
+tools/               # one-off helper scripts (dataset prep, etc.)
+tests/               # pytest smoke tests
+docs/                # reference docs
+```
+
+## Run the tests
+
+```bash
+pytest -q
+```

benchmaker-0.1.0/benchmaker/__init__.py

@@ -0,0 +1,152 @@
+"""benchmaker: async HTTP benchmarking with pluggable workload-types + workloads (datasets)."""
+
+from benchmaker.types import (
+    Request,
+    Response,
+    Sample,
+    PreRequestHook,
+    PostResponseHook,
+)
+from benchmaker.workloads.base import WorkloadType
+from benchmaker.workloads.datasets import (
+    Workload,
+    StaticWorkload,
+    JsonlWorkload,
+    CallableWorkload,
+    IterableWorkload,
+)
+from benchmaker.workloads.http import HttpWorkloadType
+from benchmaker.workloads.llm import OpenAIChatWorkloadType
+from benchmaker.workloads.sandbox import SandboxWorkloadType
+from benchmaker.workloads.hf import HFDatasetWorkload
+from benchmaker.workloads.agent import (
+    Agent,
+    AgentContext,
+    AgentResult,
+    AgentWorkloadType,
+    CallableAgent,
+)
+from benchmaker.workloads.eval import (
+    EvalWorkloadType,
+    Scorer,
+    correctness_hook,
+    extract_openai_text,
+    extract_raw_text,
+    extract_text,
+    exact_match,
+    contains,
+    regex_match,
+    json_valid,
+    multiple_choice,
+    judge_llm,
+    openai_chat_judge,
+)
+from benchmaker.load import (
+    LoadModel,
+    ConstantRPS,
+    PoissonRPS,
+    ClosedLoop,
+    Sweep,
+    Ramp,
+    parse_rate_spec,
+)
+from benchmaker.env import interpolate, load_dotenv
+from benchmaker.monitors import (
+    Monitor,
+    FunctionMonitor,
+    PrometheusMonitor,
+    parse_prometheus,
+)
+from benchmaker.runner import BenchRunner, BenchConfig, BenchResult
+from benchmaker.trace import (
+    ReplayWorkloadType,
+    TracePacedLoad,
+    TraceRecorder,
+    TraceWorkload,
+    load_trace,
+)
+from benchmaker.bundle import (
+    BUNDLE_VERSION,
+    RunMeta,
+    default_run_id,
+    is_bundle_dir,
+    iter_jsonl,
+    read_bundle,
+    write_bundle,
+)
+
+__all__ = [
+    "Request",
+    "Response",
+    "Sample",
+    "PreRequestHook",
+    "PostResponseHook",
+    # workload-types (protocols)
+    "WorkloadType",
+    "HttpWorkloadType",
+    "OpenAIChatWorkloadType",
+    "SandboxWorkloadType",
+    "HFDatasetWorkload",
+    # agent workload (pluggable user-defined agents)
+    "Agent",
+    "AgentContext",
+    "AgentResult",
+    "AgentWorkloadType",
+    "CallableAgent",
+    # eval / correctness
+    "EvalWorkloadType",
+    "Scorer",
+    "correctness_hook",
+    "extract_openai_text",
+    "extract_raw_text",
+    "extract_text",
+    "exact_match",
+    "contains",
+    "regex_match",
+    "json_valid",
+    "multiple_choice",
+    "judge_llm",
+    "openai_chat_judge",
+    # workloads (datasets / input sources)
+    "Workload",
+    "StaticWorkload",
+    "JsonlWorkload",
+    "CallableWorkload",
+    "IterableWorkload",
+    # load models
+    "LoadModel",
+    "ConstantRPS",
+    "PoissonRPS",
+    "ClosedLoop",
+    "Sweep",
+    "Ramp",
+    "parse_rate_spec",
+    # monitors
+    "Monitor",
+    "FunctionMonitor",
+    "PrometheusMonitor",
+    "parse_prometheus",
+    # env
+    "load_dotenv",
+    "interpolate",
+    # runner
+    "BenchRunner",
+    "BenchConfig",
+    "BenchResult",
+    # trace: record & replay
+    "TraceRecorder",
+    "ReplayWorkloadType",
+    "TraceWorkload",
+    "TracePacedLoad",
+    "load_trace",
+    # bundle / output layout
+    "BUNDLE_VERSION",
+    "RunMeta",
+    "default_run_id",
+    "is_bundle_dir",
+    "iter_jsonl",
+    "read_bundle",
+    "write_bundle",
+]
+
+__version__ = "0.1.0"