batchbench 0.1.0__tar.gz

batchbench-0.1.0/MANIFEST.in

@@ -0,0 +1,2 @@
+include README.md
+recursive-include src/batchbench/bin *

batchbench-0.1.0/PKG-INFO

@@ -0,0 +1,80 @@
+Metadata-Version: 2.4
+Name: batchbench
+Version: 0.1.0
+Summary: Offline and online benchmarking utilities for large language model workloads
+Author: BatchBench Contributors
+License: Apache-2.0
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Provides-Extra: generate
+Requires-Dist: transformers>=4.39.0; extra == "generate"
+Provides-Extra: offline
+Requires-Dist: vllm>=0.4.0; extra == "offline"
+
+# BatchBench
+
+BatchBench bundles three benchmarking utilities behind installable Python entrypoints:
+
+- `batchbench.generate` produces JSONL request corpora with controllable prefix overlap and approximate token counts.
+- `batchbench.offline` drives an offline vLLM workload to record prompt and generation throughput.
+- `batchbench.online` launches the packaged Rust binary that fans requests out to OpenAI-compatible endpoints in parallel.
+
+## Installation
+
+```bash
+pip install batchbench
+```
+
+Optional extras install tool-specific dependencies:
+
+```bash
+pip install "batchbench[generate]"   # adds transformers for prompt sizing
+pip install "batchbench[offline]"    # adds vllm for the offline benchmark
+```
+
+## Generating Requests
+
+```bash
+batchbench.generate \
+  --count 100 \
+  --prefix-overlap 0.3 \
+  --approx-input-tokens 512 \
+  --tokenizer-model gpt-3.5-turbo \
+  --output data
+```
+
+Each row in the resulting JSONL file has a `text` field. The filename embeds run metadata (count, tokens, prefix, tokenizer) to keep runs distinct.
+
+## Offline Benchmarking
+
+The offline harness requires vLLM and a compatible model checkpoint.
+
+```bash
+batchbench.offline \
+  --model facebook/opt-125m \
+  --num_reqs 2048 \
+  --icl 1024 \
+  --ocl 1
+```
+
+The command prints prompt/generation throughput statistics and writes the sampled history to `vllm_throughput_history.csv` (configurable via `--throughput_csv`).
+
+## Online Benchmarking
+
+`batchbench.online` wraps the Rust executable that used to live under `rust-bench/`. The binary ships inside the wheel, so Cargo is not required on the host.
+
+```bash
+batchbench.online \
+  --jsonl data/requests.jsonl \
+  --model gpt-4o-mini \
+  --host https://api.openai.com \
+  --endpoint /v1/chat/completions \
+  --users 8 \
+  --requests-per-user 1
+```
+
+Provide an API key via `--api-key` or the environment variable named by `--api-key-env` (defaults to `OPENAI_API_KEY`).
+
+## Development Notes
+
+The project now follows a `src/` layout. Run `pip install -e .[generate,offline]` during development to work against the editable package. The Rust binary can be rebuilt with `cargo build --release` inside `rust-bench/`; copy the resulting executable to `src/batchbench/bin/` if you need to refresh it.

batchbench-0.1.0/README.md

@@ -0,0 +1,67 @@
+# BatchBench
+
+BatchBench bundles three benchmarking utilities behind installable Python entrypoints:
+
+- `batchbench.generate` produces JSONL request corpora with controllable prefix overlap and approximate token counts.
+- `batchbench.offline` drives an offline vLLM workload to record prompt and generation throughput.
+- `batchbench.online` launches the packaged Rust binary that fans requests out to OpenAI-compatible endpoints in parallel.
+
+## Installation
+
+```bash
+pip install batchbench
+```
+
+Optional extras install tool-specific dependencies:
+
+```bash
+pip install "batchbench[generate]"   # adds transformers for prompt sizing
+pip install "batchbench[offline]"    # adds vllm for the offline benchmark
+```
+
+## Generating Requests
+
+```bash
+batchbench.generate \
+  --count 100 \
+  --prefix-overlap 0.3 \
+  --approx-input-tokens 512 \
+  --tokenizer-model gpt-3.5-turbo \
+  --output data
+```
+
+Each row in the resulting JSONL file has a `text` field. The filename embeds run metadata (count, tokens, prefix, tokenizer) to keep runs distinct.
+
+## Offline Benchmarking
+
+The offline harness requires vLLM and a compatible model checkpoint.
+
+```bash
+batchbench.offline \
+  --model facebook/opt-125m \
+  --num_reqs 2048 \
+  --icl 1024 \
+  --ocl 1
+```
+
+The command prints prompt/generation throughput statistics and writes the sampled history to `vllm_throughput_history.csv` (configurable via `--throughput_csv`).
+
+## Online Benchmarking
+
+`batchbench.online` wraps the Rust executable that used to live under `rust-bench/`. The binary ships inside the wheel, so Cargo is not required on the host.
+
+```bash
+batchbench.online \
+  --jsonl data/requests.jsonl \
+  --model gpt-4o-mini \
+  --host https://api.openai.com \
+  --endpoint /v1/chat/completions \
+  --users 8 \
+  --requests-per-user 1
+```
+
+Provide an API key via `--api-key` or the environment variable named by `--api-key-env` (defaults to `OPENAI_API_KEY`).
+
+## Development Notes
+
+The project now follows a `src/` layout. Run `pip install -e .[generate,offline]` during development to work against the editable package. The Rust binary can be rebuilt with `cargo build --release` inside `rust-bench/`; copy the resulting executable to `src/batchbench/bin/` if you need to refresh it.

batchbench-0.1.0/pyproject.toml

@@ -0,0 +1,33 @@
+[build-system]
+requires = ["setuptools>=69", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "batchbench"
+version = "0.1.0"
+description = "Offline and online benchmarking utilities for large language model workloads"
+readme = "README.md"
+authors = [
+    {name = "BatchBench Contributors"}
+]
+license = {text = "Apache-2.0"}
+requires-python = ">=3.9"
+dependencies = []
+
+[project.optional-dependencies]
+generate = ["transformers>=4.39.0"]
+offline = ["vllm>=0.4.0"]
+
+[project.scripts]
+'batchbench.generate' = "batchbench.generate:main"
+'batchbench.offline' = "batchbench.offline:main"
+'batchbench.online' = "batchbench.online:main"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+"batchbench" = ["bin/batchbench"]

batchbench-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

batchbench-0.1.0/src/batchbench/__init__.py

@@ -0,0 +1,22 @@
+"""batchbench brings offline and online benchmarking utilities under one roof."""
+
+from __future__ import annotations
+
+from importlib import resources
+
+try:  # pragma: no cover
+    from importlib.metadata import PackageNotFoundError, version
+except ImportError:  # pragma: no cover
+    from importlib_metadata import PackageNotFoundError, version  # type: ignore
+
+
+def package_version() -> str:
+    """Return the installed package version or a placeholder when run from source."""
+    try:
+        return version("batchbench")
+    except PackageNotFoundError:
+        return "0.0.0"
+
+
+__all__ = ["package_version", "resources"]
+__version__ = package_version()

batchbench-0.1.0/src/batchbench/generate.py

@@ -0,0 +1,256 @@
+"""Utilities and CLI entrypoint for generating batchbench request payloads."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import random
+import re
+from pathlib import Path
+from typing import Any, List
+
+DEFAULT_PREFIX_TEXT = (
+    "In this experiment we explore the capability of large language models "
+    "to adapt their narrative based on subtle contextual variations. The "
+    "following prompt requests creative output across a range of scenarios."
+)
+
+
+def load_tokenizer(model_name: str, token: str | None = None):
+    """Load a Hugging Face tokenizer and configure pad token if needed."""
+    try:
+        from transformers import AutoTokenizer
+    except ImportError as exc:  # pragma: no cover
+        raise SystemExit(
+            "Approximate token sizing requires the 'transformers' package. "
+            "Install it with `pip install transformers`."
+        ) from exc
+
+    load_kwargs = {"use_fast": True}
+    if token:
+        load_kwargs["token"] = token
+    try:
+        tokenizer = AutoTokenizer.from_pretrained(model_name, **load_kwargs)
+    except Exception as exc:  # pragma: no cover
+        raise SystemExit(f"Failed to load tokenizer '{model_name}': {exc}") from exc
+
+    if tokenizer.pad_token is None and tokenizer.eos_token is not None:
+        tokenizer.pad_token = tokenizer.eos_token
+
+    return tokenizer
+
+
+def assemble_prompts(
+    count: int,
+    prefix_overlap: float,
+    *,
+    target_tokens: int | None = None,
+    tokenizer: Any | None = None,
+    tolerance: int = 5,
+) -> List[str]:
+    """Create prompt strings whose prefixes overlap by the requested fraction."""
+    if tokenizer is None:
+        raise ValueError("A tokenizer is required to detokenize random token ids.")
+
+    prompts: List[str] = []
+    rng = random.Random()
+    sequence_lengths: List[int] = []
+    for _ in range(count):
+        sequence_length = 1
+        if target_tokens and target_tokens > 0:
+            lower = max(1, target_tokens - (tolerance if tolerance else 0))
+            upper = target_tokens + (tolerance if tolerance else 0)
+            if lower > upper:
+                lower = upper
+            sequence_length = rng.randint(lower, upper) if lower != upper else lower
+        sequence_lengths.append(sequence_length)
+
+    prefix_ratio = max(0.0, min(prefix_overlap, 1.0))
+    min_length = min(sequence_lengths) if sequence_lengths else 0
+    prefix_length = int(math.floor(min_length * prefix_ratio)) if min_length else 0
+    if prefix_ratio > 0.0 and prefix_length == 0 and min_length > 0:
+        prefix_length = 1
+
+    prefix_ids = (
+        [rng.randint(1, 10000) for _ in range(prefix_length)] if prefix_length else []
+    )
+
+    for seq_length in sequence_lengths:
+        token_ids = [rng.randint(1, 10000) for _ in range(seq_length)]
+        unique_ids = token_ids[prefix_length:]
+        final_ids = prefix_ids + unique_ids
+
+        prompt_text = tokenizer.decode(
+            final_ids,
+            skip_special_tokens=True,
+            clean_up_tokenization_spaces=True,
+        )
+
+        if not prompt_text.strip():
+            try:
+                tokens = tokenizer.convert_ids_to_tokens(final_ids)
+                prompt_text = " ".join(tokens).strip()
+            except Exception:
+                prompt_text = " ".join(str(tid) for tid in final_ids)
+
+        prompts.append(prompt_text)
+
+    return prompts
+
+
+def parse_args(argv: List[str] | None = None) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--count",
+        "-n",
+        type=int,
+        default=10,
+        help="Number of requests to generate (default: 10)",
+    )
+    parser.add_argument(
+        "--prefix-overlap",
+        type=float,
+        default=0.0,
+        help="Fraction of tokens shared as a prefix across requests (0.0-1.0, default: 0.0)",
+    )
+    parser.add_argument(
+        "--output",
+        "-o",
+        type=Path,
+        default=Path("outputs"),
+        help=(
+            "Destination directory or base filename. Metadata (count, prefix, token "
+            "target, tokenizer) is appended automatically."
+        ),
+    )
+    parser.add_argument(
+        "--approx-input-tokens",
+        type=int,
+        default=0,
+        help="Approximate number of tokens each prompt should contain (default: 0 = no adjustment)",
+    )
+    parser.add_argument(
+        "--tokenizer-model",
+        default=None,
+        help=(
+            "Hugging Face tokenizer identifier to use when approximating token counts. "
+            "Defaults to 'gpt2' when not specified."
+        ),
+    )
+    parser.add_argument(
+        "--token-tolerance",
+        type=int,
+        default=None,
+        help="Acceptable +/- token tolerance when approximating lengths (default: max(5, 5%% of target))",
+    )
+    parser.add_argument(
+        "--huggingface-token",
+        default=None,
+        help=(
+            "Personal access token for Hugging Face Hub (optional). If omitted, the "
+            "generator checks HUGGINGFACE_TOKEN and HUGGING_FACE_HUB_TOKEN env vars."
+        ),
+    )
+    return parser.parse_args(argv)
+
+
+def resolve_tolerance(target_tokens: int, explicit: int | None) -> int:
+    if target_tokens <= 0:
+        return 0
+    if explicit is not None and explicit >= 0:
+        return explicit
+    return max(5, int(target_tokens * 0.05))
+
+
+def sanitize_component(value: str | None) -> str:
+    if not value:
+        return "none"
+    cleaned = re.sub(r"[^0-9A-Za-z._-]+", "-", value).strip("-._")
+    return cleaned or "none"
+
+
+def format_prefix(prefix_overlap: float) -> str:
+    return f"{prefix_overlap:.2f}".replace(".", "p")
+
+
+def build_output_path(
+    base_path: Path,
+    *,
+    count: int,
+    prefix_overlap: float,
+    target_tokens: int | None,
+    tokenizer_label: str,
+) -> Path:
+    tokens_label = str(target_tokens) if target_tokens and target_tokens > 0 else "none"
+    prefix_label = format_prefix(prefix_overlap)
+    tokenizer_component = sanitize_component(tokenizer_label)
+    metadata_suffix = (
+        f"count-{count}_tokens-{tokens_label}_prefix-{prefix_label}_tokenizer-{tokenizer_component}"
+    )
+
+    if base_path.suffix == ".jsonl" and not base_path.is_dir():
+        directory = base_path.parent if base_path.parent else Path(".")
+        stem = sanitize_component(base_path.stem) or "requests"
+        filename = f"{stem}_{metadata_suffix}.jsonl"
+    else:
+        directory = base_path
+        filename = f"requests_{metadata_suffix}.jsonl"
+
+    directory.mkdir(parents=True, exist_ok=True)
+    output_path = directory / filename
+    if output_path.exists():
+        raise SystemExit(
+            f"Refusing to overwrite existing file {output_path}. Delete it or choose a different output directory."
+        )
+    return output_path
+
+
+def main(argv: List[str] | None = None) -> int:
+    args = parse_args(argv)
+
+    hf_token = (
+        args.huggingface_token
+        or os.getenv("HUGGINGFACE_TOKEN")
+        or os.getenv("HUGGING_FACE_HUB_TOKEN")
+    )
+
+    target_tokens = args.approx_input_tokens if args.approx_input_tokens > 0 else None
+    tokenizer_label = args.tokenizer_model or "gpt2"
+    tokenizer = load_tokenizer(tokenizer_label, hf_token)
+    tolerance = (
+        resolve_tolerance(target_tokens, args.token_tolerance) if target_tokens else 0
+    )
+
+    prompts = assemble_prompts(
+        count=args.count,
+        prefix_overlap=args.prefix_overlap,
+        target_tokens=target_tokens,
+        tokenizer=tokenizer,
+        tolerance=tolerance,
+    )
+
+    output_path = build_output_path(
+        args.output,
+        count=args.count,
+        prefix_overlap=args.prefix_overlap,
+        target_tokens=target_tokens,
+        tokenizer_label=tokenizer_label,
+    )
+
+    with output_path.open("w", encoding="utf-8") as handle:
+        for prompt in prompts:
+            json.dump({"text": prompt}, handle)
+            handle.write("\n")
+
+    print(
+        f"Wrote {len(prompts)} request prompts to {output_path} "
+        f"with prefix overlap {args.prefix_overlap:.2f}."
+    )
+
+    return 0
+
+
+if __name__ == "__main__":  # pragma: no cover
+    raise SystemExit(main())

batchbench-0.1.0/src/batchbench/offline.py

@@ -0,0 +1,210 @@
+"""Offline vLLM benchmarking CLI entrypoint."""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import logging
+import os
+import re
+from contextlib import contextmanager
+from random import randint
+from statistics import mean, pstdev
+from typing import Dict, Iterable, List, Optional
+
+def _load_vllm():
+    try:
+        from vllm import LLM, SamplingParams  # type: ignore
+    except ImportError as exc:  # pragma: no cover
+        raise SystemExit(
+            "vLLM is required for batchbench.offline. Install with `pip install batchbench[offline]`."
+        ) from exc
+
+    # Ensure vLLM keeps emitting throughput stats while we run.
+    os.environ.setdefault("VLLM_LOG_STATS_INTERVAL", "1")
+    return LLM, SamplingParams
+
+
+class VLLMThroughputCollector(logging.Handler):
+    """Logging handler that captures vLLM throughput stats from INFO logs."""
+
+    def __init__(self):
+        super().__init__(level=logging.INFO)
+        self.prompt_tps: List[float] = []
+        self.gen_tps: List[float] = []
+        self.TP_LINE = re.compile(
+            r"Avg prompt throughput:\s*([0-9.]+)\s*tokens/s,\s*"
+            r"Avg generation throughput:\s*([0-9.]+)\s*tokens/s"
+        )
+
+    def emit(self, record: logging.LogRecord) -> None:  # type: ignore[override]
+        try:
+            msg = record.getMessage()
+        except Exception:
+            return
+        match = self.TP_LINE.search(msg)
+        if match:
+            self.prompt_tps.append(float(match.group(1)))
+            self.gen_tps.append(float(match.group(2)))
+
+    def summary(self) -> Optional[Dict[str, float]]:
+        if not self.prompt_tps or not self.gen_tps:
+            return None
+        return {
+            "prompt_avg": mean(self.prompt_tps),
+            "prompt_std": pstdev(self.prompt_tps),
+            "gen_avg": mean(self.gen_tps),
+            "gen_std": pstdev(self.gen_tps),
+        }
+
+    def save_csv(self, path: str = "vllm_throughput_history.csv") -> str:
+        """Persist the captured throughput history for later inspection."""
+        with open(path, "w", newline="") as handle:
+            writer = csv.writer(handle)
+            writer.writerow(["prompt_tps", "gen_tps"])
+            for prompt_tps, gen_tps in zip(self.prompt_tps, self.gen_tps):
+                writer.writerow([prompt_tps, gen_tps])
+        return path
+
+
+@contextmanager
+def capture_vllm_throughput() -> Iterable[VLLMThroughputCollector]:
+    """Attach the collector to the vLLM logger while the workload runs."""
+    logger = logging.getLogger("vllm")
+    collector = VLLMThroughputCollector()
+    logger.addHandler(collector)
+    if logger.level > logging.INFO:
+        logger.setLevel(logging.INFO)
+    try:
+        yield collector
+    finally:
+        logger.removeHandler(collector)
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description=(
+            "Run the metrics workload while allowing the model and vLLM options "
+            "to be configured from the command line."
+        )
+    )
+    parser.add_argument(
+        "--model",
+        default="facebook/opt-125m",
+        help="Model identifier or path to load with vLLM."
+    )
+    parser.add_argument(
+        "--num_reqs",
+        type=int,
+        default=2048,
+        help="Number of synthetic prompts to generate."
+    )
+    parser.add_argument(
+        "--icl",
+        type=int,
+        default=1024,
+        help="Input context length (tokens per prompt)."
+    )
+    parser.add_argument(
+        "--ocl",
+        type=int,
+        default=1,
+        help="Output context length (max tokens generated per request)."
+    )
+    parser.add_argument(
+        "--throughput_csv",
+        default="vllm_throughput_history.csv",
+        help="Where to persist the throughput history CSV."
+    )
+    parser.add_argument(
+        "--tensor_parallel_size",
+        type=int,
+        default=1,
+        help="Tensor parallel world size for vLLM initialisation."
+    )
+    parser.add_argument(
+        "--pipeline_parallel_size",
+        type=int,
+        default=1,
+        help="Pipeline parallel world size for vLLM initialisation."
+    )
+    parser.add_argument(
+        "--max_num_batched_tokens",
+        type=int,
+        default=512,
+        help="Maximum tokens per batch when pre-filling prompts."
+    )
+    return parser
+
+
+def validate_args(args: argparse.Namespace) -> None:
+    if args.num_reqs < 1:
+        raise ValueError("num_reqs must be >= 1")
+    if args.icl < 1:
+        raise ValueError("icl must be >= 1")
+    if args.ocl < 0:
+        raise ValueError("ocl must be >= 0")
+    if args.tensor_parallel_size < 1:
+        raise ValueError("tensor_parallel_size must be >= 1")
+    if args.pipeline_parallel_size < 1:
+        raise ValueError("pipeline_parallel_size must be >= 1")
+    if args.max_num_batched_tokens < 1:
+        raise ValueError("max_num_batched_tokens must be >= 1")
+
+
+def main(argv: List[str] | None = None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+
+    try:
+        validate_args(args)
+    except ValueError as exc:
+        parser.error(str(exc))
+
+    LLM, SamplingParams = _load_vllm()
+
+    llm_kwargs = {
+        "model": args.model,
+        "disable_log_stats": False,
+        "enable_chunked_prefill": True,
+        "tensor_parallel_size": args.tensor_parallel_size,
+        "pipeline_parallel_size": args.pipeline_parallel_size,
+        "max_num_batched_tokens": args.max_num_batched_tokens,
+    }
+
+    sampling_params = SamplingParams(
+        temperature=0.8,
+        top_k=20,
+        max_tokens=args.ocl,
+        min_tokens=args.ocl,
+    )
+
+    llm = LLM(**llm_kwargs)
+
+    tokenizer = llm.get_tokenizer()
+
+    tokenized_prompts = [
+        [randint(1, 10000) for _ in range(args.icl)]
+        for _ in range(args.num_reqs)
+    ]
+    prompts = tokenizer.batch_decode(tokenized_prompts)
+
+    with capture_vllm_throughput() as collector:
+        llm.generate(prompts, sampling_params, use_tqdm=False)
+        stats = collector.summary()
+        csv_path = collector.save_csv(args.throughput_csv)
+
+    if stats is None:
+        print("No throughput lines were captured. Make sure vLLM is emitting stats logs.")
+    else:
+        print(
+            f"Prompt TPS  mean={stats['prompt_avg']:.2f} +/- {stats['prompt_std']:.2f}\n"
+            f"Gen TPS     mean={stats['gen_avg']:.2f} +/- {stats['gen_std']:.2f}\n"
+            f"History CSV written to: {csv_path}"
+        )
+
+    return 0
+
+
+if __name__ == "__main__":  # pragma: no cover
+    raise SystemExit(main())

batchbench-0.1.0/src/batchbench/online.py

@@ -0,0 +1,38 @@
+"""Wrapper that launches the bundled Rust-powered online benchmark."""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+
+from importlib import resources
+
+
+BINARY_NAME = "batchbench"
+
+
+def main(argv: list[str] | None = None) -> int:
+    args = argv if argv is not None else sys.argv[1:]
+
+    resource = resources.files(__package__).joinpath("bin", BINARY_NAME)
+    if not resource.is_file():  # pragma: no cover
+        raise SystemExit(
+            "Bundled online benchmarking binary not found. "
+            "Please reinstall batchbench or report an issue."
+        )
+
+    env = os.environ.copy()
+
+    with resources.as_file(resource) as binary_path:
+        command = [str(binary_path), *args]
+        try:
+            completed = subprocess.run(command, check=False, env=env)
+        except FileNotFoundError as exc:  # pragma: no cover
+            raise SystemExit(f"Failed to execute bundled binary: {exc}") from exc
+
+    return int(completed.returncode)
+
+
+if __name__ == "__main__":  # pragma: no cover
+    raise SystemExit(main())

batchbench-0.1.0/src/batchbench.egg-info/PKG-INFO

@@ -0,0 +1,80 @@
+Metadata-Version: 2.4
+Name: batchbench
+Version: 0.1.0
+Summary: Offline and online benchmarking utilities for large language model workloads
+Author: BatchBench Contributors
+License: Apache-2.0
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Provides-Extra: generate
+Requires-Dist: transformers>=4.39.0; extra == "generate"
+Provides-Extra: offline
+Requires-Dist: vllm>=0.4.0; extra == "offline"
+
+# BatchBench
+
+BatchBench bundles three benchmarking utilities behind installable Python entrypoints:
+
+- `batchbench.generate` produces JSONL request corpora with controllable prefix overlap and approximate token counts.
+- `batchbench.offline` drives an offline vLLM workload to record prompt and generation throughput.
+- `batchbench.online` launches the packaged Rust binary that fans requests out to OpenAI-compatible endpoints in parallel.
+
+## Installation
+
+```bash
+pip install batchbench
+```
+
+Optional extras install tool-specific dependencies:
+
+```bash
+pip install "batchbench[generate]"   # adds transformers for prompt sizing
+pip install "batchbench[offline]"    # adds vllm for the offline benchmark
+```
+
+## Generating Requests
+
+```bash
+batchbench.generate \
+  --count 100 \
+  --prefix-overlap 0.3 \
+  --approx-input-tokens 512 \
+  --tokenizer-model gpt-3.5-turbo \
+  --output data
+```
+
+Each row in the resulting JSONL file has a `text` field. The filename embeds run metadata (count, tokens, prefix, tokenizer) to keep runs distinct.
+
+## Offline Benchmarking
+
+The offline harness requires vLLM and a compatible model checkpoint.
+
+```bash
+batchbench.offline \
+  --model facebook/opt-125m \
+  --num_reqs 2048 \
+  --icl 1024 \
+  --ocl 1
+```
+
+The command prints prompt/generation throughput statistics and writes the sampled history to `vllm_throughput_history.csv` (configurable via `--throughput_csv`).
+
+## Online Benchmarking
+
+`batchbench.online` wraps the Rust executable that used to live under `rust-bench/`. The binary ships inside the wheel, so Cargo is not required on the host.
+
+```bash
+batchbench.online \
+  --jsonl data/requests.jsonl \
+  --model gpt-4o-mini \
+  --host https://api.openai.com \
+  --endpoint /v1/chat/completions \
+  --users 8 \
+  --requests-per-user 1
+```
+
+Provide an API key via `--api-key` or the environment variable named by `--api-key-env` (defaults to `OPENAI_API_KEY`).
+
+## Development Notes
+
+The project now follows a `src/` layout. Run `pip install -e .[generate,offline]` during development to work against the editable package. The Rust binary can be rebuilt with `cargo build --release` inside `rust-bench/`; copy the resulting executable to `src/batchbench/bin/` if you need to refresh it.

batchbench-0.1.0/src/batchbench.egg-info/SOURCES.txt

@@ -0,0 +1,14 @@
+MANIFEST.in
+README.md
+pyproject.toml
+src/batchbench/__init__.py
+src/batchbench/generate.py
+src/batchbench/offline.py
+src/batchbench/online.py
+src/batchbench.egg-info/PKG-INFO
+src/batchbench.egg-info/SOURCES.txt
+src/batchbench.egg-info/dependency_links.txt
+src/batchbench.egg-info/entry_points.txt
+src/batchbench.egg-info/requires.txt
+src/batchbench.egg-info/top_level.txt
+src/batchbench/bin/batchbench

batchbench-0.1.0/src/batchbench.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

batchbench-0.1.0/src/batchbench.egg-info/entry_points.txt

@@ -0,0 +1,4 @@
+[console_scripts]
+batchbench.generate = batchbench.generate:main
+batchbench.offline = batchbench.offline:main
+batchbench.online = batchbench.online:main

batchbench-0.1.0/src/batchbench.egg-info/requires.txt

@@ -0,0 +1,6 @@
+
+[generate]
+transformers>=4.39.0
+
+[offline]
+vllm>=0.4.0

batchbench-0.1.0/src/batchbench.egg-info/top_level.txt

@@ -0,0 +1 @@
+batchbench