polyserve 0.1.0__tar.gz

polyserve-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Aagam Sachin Bothara
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

polyserve-0.1.0/PKG-INFO

@@ -0,0 +1,87 @@
+Metadata-Version: 2.4
+Name: polyserve
+Version: 0.1.0
+Summary: Autotuner for LLM serving: measures vLLM, SGLang and llama.cpp settings on your GPU and your traffic, then serves the fastest
+Author-email: Aagam Sachin Bothara <mweeb19@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/Aagam-Bothara/polyserve
+Project-URL: Documentation, https://github.com/Aagam-Bothara/polyserve/blob/main/docs/usage.md
+Project-URL: Benchmarks, https://github.com/Aagam-Bothara/polyserve/blob/main/docs/benchmarks.md
+Project-URL: Issues, https://github.com/Aagam-Bothara/polyserve/issues
+Keywords: llm,inference,vllm,llama.cpp,serving,autotuning
+Classifier: Development Status :: 3 - Alpha
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: <3.14,>=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.0
+Requires-Dist: typer>=0.12
+Requires-Dist: rich>=13.0
+Requires-Dist: httpx>=0.27
+Requires-Dist: fastapi>=0.110
+Requires-Dist: uvicorn>=0.29
+Requires-Dist: psutil>=5.9
+Requires-Dist: huggingface_hub>=0.23
+Provides-Extra: nvml
+Requires-Dist: nvidia-ml-py>=12.535.133; extra == "nvml"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: ruff==0.15.0; extra == "dev"
+Dynamic: license-file
+
+# PolyServe
+
+[![tests](https://github.com/Aagam-Bothara/polyserve/actions/workflows/tests.yml/badge.svg)](https://github.com/Aagam-Bothara/polyserve/actions/workflows/tests.yml)
+
+**PolyServe is an autotuner for LLM serving: it finds the fastest configuration for your GPU and your traffic, then serves it.** Give it a model and, ideally, a sample of your prompts. It knows which settings each engine offers on each card (vLLM, SGLang and llama.cpp; weight precision, batch size, KV-cache type, speculative decoding and more), drops every configuration that will not fit in memory, measures the rest on your prompts, and serves the fastest one that meets your latency target behind an OpenAI-compatible API. What pays is what it tries and how it measures, not the order it tries things in. The winning settings changed from card to card and from one set of prompts to another, so PolyServe's pick beat a fixed rule of thumb (fp8 weights and KV cache, a short context, batch 256) by 15–97% on the three cards where both ran; but random sampling of the same settings, given the same time, did as well as PolyServe's staged order on two of those cards and better on the third. Calibration measures speed and latency; what quantization costs in answer quality is checked separately.
+
+## Results
+
+Measured on rented GPUs against stock `vllm serve` and SGLang defaults:
+
+- **The pick holds on prompts it never saw.** Llama 3.1 8B, calibrated on 300 Dolly-15k prompts and measured on 300 others: **+93%** on an A40, **+59%** on an A100, **+71%** on an RTX 4090 over stock SGLang (stock vLLM does not start there), and **+24%** on an H100 over stock vLLM with fp8 weights.
+- **Your traffic decides what pays.** On one A40 with Qwen2.5-3B the pick changed with the prompts: suffix decoding on Dolly prompts (**+122%** on held-out prompts), a draft model on news-article extraction (+66.5%), an fp8 KV cache on real chat (+10%).
+- **Calibration pays for itself within hours.** It took 22–56 minutes per model and workload, repaid by 0.5–2.6 hours of busy serving on held-out prompts. Measuring the busiest load first skips 29–45% of that time without changing a pick.
+
+Every result, with methods, ablations, quality checks and limits: [docs/benchmarks.md](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/benchmarks.md).
+
+## Quickstart
+
+```bash
+pip install git+https://github.com/Aagam-Bothara/polyserve.git   # not on PyPI yet
+pip install "vllm==0.29.0"     # driver 580+; older drivers: "vllm==0.11.0" "transformers>=4.56,<5"
+polyserve serve Qwen/Qwen2.5-3B-Instruct --workload chat
+curl localhost:8000/v1/chat/completions -H 'content-type: application/json' \
+  -d '{"model":"Qwen/Qwen2.5-3B-Instruct","messages":[{"role":"user","content":"hi"}]}'
+```
+
+The first launch calibrates and caches the result per machine, model, objective and workload; later launches serve at once. To tune for your own traffic, pass a sample of it: `--workload-file prompts.jsonl`, one prompt per line. Optional: `pip install arctic-inference==0.1.1` adds suffix decoding; SGLang can live in its own environment (set `SGLANG_PYTHON` to its `python`); llama.cpp needs `llama-server` built with CUDA, on `PATH` or in `LLAMA_SERVER`.
+
+With Docker (built on the official vLLM image):
+
+```bash
+docker build -t polyserve .
+docker run --gpus all --ipc=host -p 8000:8000 -v ~/.cache/huggingface:/root/.cache/huggingface \
+  -v ~/.polyserve:/root/.polyserve polyserve serve Qwen/Qwen2.5-3B-Instruct
+```
+
+### Supported hardware
+
+Linux, Python 3.10–3.13. NVIDIA GPUs of compute capability 7.5 or newer run vLLM, SGLang and llama.cpp (measured on an A40, A100, H100 NVL, L4 and RTX 4090); older NVIDIA GPUs and x86 CPUs run llama.cpp. vLLM-CPU and pre-Turing GPUs have never been benchmarked ([details](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/benchmarks.md#hardware-and-engines-measured)).
+
+## Learn more
+
+- [docs/benchmarks.md](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/benchmarks.md): every result, at a glance and in full, with methods, ablations, quality, limits and what is still unmeasured.
+- [docs/usage.md](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/usage.md): how calibration works, workloads, objectives, every search option and the CLI.
+- [docs/writeup.md](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/writeup.md): the design of the memory planner, calibration and the predictor, what the evidence does and does not support, and the roadmap.
+- [benchmarks/strategies/SUMMARY.md](https://github.com/Aagam-Bothara/polyserve/blob/main/benchmarks/strategies/SUMMARY.md): every table, regenerated from the raw JSON.
+- [CONTRIBUTING.md](https://github.com/Aagam-Bothara/polyserve/blob/main/CONTRIBUTING.md): development setup, tests and adding a backend.
+
+MIT licensed.

polyserve-0.1.0/README.md

@@ -0,0 +1,49 @@
+# PolyServe
+
+[![tests](https://github.com/Aagam-Bothara/polyserve/actions/workflows/tests.yml/badge.svg)](https://github.com/Aagam-Bothara/polyserve/actions/workflows/tests.yml)
+
+**PolyServe is an autotuner for LLM serving: it finds the fastest configuration for your GPU and your traffic, then serves it.** Give it a model and, ideally, a sample of your prompts. It knows which settings each engine offers on each card (vLLM, SGLang and llama.cpp; weight precision, batch size, KV-cache type, speculative decoding and more), drops every configuration that will not fit in memory, measures the rest on your prompts, and serves the fastest one that meets your latency target behind an OpenAI-compatible API. What pays is what it tries and how it measures, not the order it tries things in. The winning settings changed from card to card and from one set of prompts to another, so PolyServe's pick beat a fixed rule of thumb (fp8 weights and KV cache, a short context, batch 256) by 15–97% on the three cards where both ran; but random sampling of the same settings, given the same time, did as well as PolyServe's staged order on two of those cards and better on the third. Calibration measures speed and latency; what quantization costs in answer quality is checked separately.
+
+## Results
+
+Measured on rented GPUs against stock `vllm serve` and SGLang defaults:
+
+- **The pick holds on prompts it never saw.** Llama 3.1 8B, calibrated on 300 Dolly-15k prompts and measured on 300 others: **+93%** on an A40, **+59%** on an A100, **+71%** on an RTX 4090 over stock SGLang (stock vLLM does not start there), and **+24%** on an H100 over stock vLLM with fp8 weights.
+- **Your traffic decides what pays.** On one A40 with Qwen2.5-3B the pick changed with the prompts: suffix decoding on Dolly prompts (**+122%** on held-out prompts), a draft model on news-article extraction (+66.5%), an fp8 KV cache on real chat (+10%).
+- **Calibration pays for itself within hours.** It took 22–56 minutes per model and workload, repaid by 0.5–2.6 hours of busy serving on held-out prompts. Measuring the busiest load first skips 29–45% of that time without changing a pick.
+
+Every result, with methods, ablations, quality checks and limits: [docs/benchmarks.md](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/benchmarks.md).
+
+## Quickstart
+
+```bash
+pip install git+https://github.com/Aagam-Bothara/polyserve.git   # not on PyPI yet
+pip install "vllm==0.29.0"     # driver 580+; older drivers: "vllm==0.11.0" "transformers>=4.56,<5"
+polyserve serve Qwen/Qwen2.5-3B-Instruct --workload chat
+curl localhost:8000/v1/chat/completions -H 'content-type: application/json' \
+  -d '{"model":"Qwen/Qwen2.5-3B-Instruct","messages":[{"role":"user","content":"hi"}]}'
+```
+
+The first launch calibrates and caches the result per machine, model, objective and workload; later launches serve at once. To tune for your own traffic, pass a sample of it: `--workload-file prompts.jsonl`, one prompt per line. Optional: `pip install arctic-inference==0.1.1` adds suffix decoding; SGLang can live in its own environment (set `SGLANG_PYTHON` to its `python`); llama.cpp needs `llama-server` built with CUDA, on `PATH` or in `LLAMA_SERVER`.
+
+With Docker (built on the official vLLM image):
+
+```bash
+docker build -t polyserve .
+docker run --gpus all --ipc=host -p 8000:8000 -v ~/.cache/huggingface:/root/.cache/huggingface \
+  -v ~/.polyserve:/root/.polyserve polyserve serve Qwen/Qwen2.5-3B-Instruct
+```
+
+### Supported hardware
+
+Linux, Python 3.10–3.13. NVIDIA GPUs of compute capability 7.5 or newer run vLLM, SGLang and llama.cpp (measured on an A40, A100, H100 NVL, L4 and RTX 4090); older NVIDIA GPUs and x86 CPUs run llama.cpp. vLLM-CPU and pre-Turing GPUs have never been benchmarked ([details](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/benchmarks.md#hardware-and-engines-measured)).
+
+## Learn more
+
+- [docs/benchmarks.md](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/benchmarks.md): every result, at a glance and in full, with methods, ablations, quality, limits and what is still unmeasured.
+- [docs/usage.md](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/usage.md): how calibration works, workloads, objectives, every search option and the CLI.
+- [docs/writeup.md](https://github.com/Aagam-Bothara/polyserve/blob/main/docs/writeup.md): the design of the memory planner, calibration and the predictor, what the evidence does and does not support, and the roadmap.
+- [benchmarks/strategies/SUMMARY.md](https://github.com/Aagam-Bothara/polyserve/blob/main/benchmarks/strategies/SUMMARY.md): every table, regenerated from the raw JSON.
+- [CONTRIBUTING.md](https://github.com/Aagam-Bothara/polyserve/blob/main/CONTRIBUTING.md): development setup, tests and adding a backend.
+
+MIT licensed.

polyserve-0.1.0/polyserve/__init__.py

@@ -0,0 +1,6 @@
+"""PolyServe: an autotuner for LLM serving.
+
+probe -> select backends -> prepare model -> plan memory -> calibrate -> cache -> serve
+"""
+
+__version__ = "0.1.0"

polyserve-0.1.0/polyserve/backends/__init__.py

@@ -0,0 +1,36 @@
+"""Backend registry. New hardware = new class here, no core changes."""
+
+from __future__ import annotations
+
+from typing import Dict
+
+from polyserve.backends.base import Backend, BaseBackend, LaunchSpec, LlmtraceHooks, Process, free_port
+
+
+def registry() -> Dict[str, BaseBackend]:
+    from polyserve.backends.llamacpp import LlamaCppCpuBackend, LlamaCppCudaBackend
+    from polyserve.backends.sglang import SglangBackend
+    from polyserve.backends.vllm import VllmBackend
+    from polyserve.backends.vllm_cpu import VllmCpuBackend
+
+    backends = [VllmBackend(), SglangBackend(), LlamaCppCudaBackend(), LlamaCppCpuBackend(), VllmCpuBackend()]
+    return {b.name: b for b in backends}
+
+
+def get_backend(name: str) -> BaseBackend:
+    reg = registry()
+    if name not in reg:
+        raise KeyError(f"unknown backend {name!r}; known: {sorted(reg)}")
+    return reg[name]
+
+
+__all__ = [
+    "Backend",
+    "BaseBackend",
+    "LaunchSpec",
+    "LlmtraceHooks",
+    "Process",
+    "free_port",
+    "registry",
+    "get_backend",
+]

polyserve-0.1.0/polyserve/backends/base.py

@@ -0,0 +1,314 @@
+"""Backend interface (spec section 3) plus the subprocess wrapper every backend uses."""
+
+from __future__ import annotations
+
+import logging
+import os
+import signal
+import socket
+import subprocess
+import sys
+import time
+from dataclasses import dataclass, field
+from pathlib import Path
+from typing import Dict, List, Optional, Protocol, Tuple, runtime_checkable
+
+import httpx
+
+from polyserve.memory import MemoryModel
+from polyserve.models import Config, HardwareDescriptor, ModelSpec, PreparedModel
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class LlmtraceHooks:
+    """How the calibration driver should talk to (and measure) this backend."""
+
+    completions_path: str = "/v1/completions"
+    health_path: str = "/health"
+    models_path: str = "/v1/models"
+    stream_usage: bool = True  # backend reports usage in the final stream chunk when asked
+    model_name: Optional[str] = None  # value to put in the "model" field of requests
+    tokenizer_id: Optional[str] = None  # HF id whose tokenizer counts tokens when usage is absent
+    gpu_ids: List[int] = field(default_factory=list)  # NVML indices llmtrace should sample
+    process_memory: bool = False  # sample RSS of the backend process (CPU backends)
+
+
+@dataclass
+class LaunchSpec:
+    args: List[str]
+    env: Dict[str, str] = field(default_factory=dict)
+    cwd: Optional[str] = None
+
+
+class Process:
+    """A launched backend server: subprocess + readiness probe + logs."""
+
+    def __init__(self, spec: LaunchSpec, port: int, health_url: str, log_path: Optional[Path] = None):
+        self.spec = spec
+        self.port = port
+        self.health_url = health_url
+        self.log_path = log_path
+        self._proc: Optional[subprocess.Popen] = None
+        self._log_fh = None
+        self.started_at: Optional[float] = None
+        self.ready_at: Optional[float] = None
+
+    @property
+    def pid(self) -> Optional[int]:
+        return self._proc.pid if self._proc else None
+
+    def start(self) -> "Process":
+        env = dict(os.environ)
+        env.update(self.spec.env)
+        if self.log_path:
+            self.log_path.parent.mkdir(parents=True, exist_ok=True)
+            self._log_fh = open(self.log_path, "ab")
+            stdout = self._log_fh
+        else:
+            stdout = subprocess.DEVNULL
+        logger.info("launch: %s", " ".join(self.spec.args))
+        kwargs = {}
+        if sys.platform != "win32":
+            kwargs["start_new_session"] = True  # own process group so we can kill children
+        self._proc = subprocess.Popen(
+            self.spec.args,
+            env=env,
+            cwd=self.spec.cwd,
+            stdout=stdout,
+            stderr=subprocess.STDOUT,
+            stdin=subprocess.DEVNULL,
+            **kwargs,
+        )
+        self.started_at = time.monotonic()
+        return self
+
+    def alive(self) -> bool:
+        return self._proc is not None and self._proc.poll() is None
+
+    def returncode(self) -> Optional[int]:
+        return self._proc.poll() if self._proc else None
+
+    def wait_ready(self, timeout: float = 600.0, poll: float = 1.0) -> bool:
+        deadline = time.monotonic() + timeout
+        with httpx.Client(timeout=5.0) as client:
+            while time.monotonic() < deadline:
+                if not self.alive():
+                    logger.error("backend exited during startup (rc=%s); see %s", self.returncode(), self.log_path)
+                    return False
+                try:
+                    r = client.get(self.health_url)
+                    if r.status_code < 500:
+                        self.ready_at = time.monotonic()
+                        return True
+                except httpx.HTTPError:
+                    pass
+                time.sleep(poll)
+        logger.error("backend not ready after %.0fs; see %s", timeout, self.log_path)
+        return False
+
+    def stop(self, grace: float = 15.0) -> None:
+        if self._proc is None:
+            return
+        if self.alive():
+            try:
+                if sys.platform != "win32":
+                    os.killpg(self._proc.pid, signal.SIGTERM)  # own session: group id == server pid
+                else:
+                    self._proc.terminate()
+            except Exception:
+                pass
+            try:
+                self._proc.wait(timeout=grace)
+            except subprocess.TimeoutExpired:
+                try:
+                    if sys.platform != "win32":
+                        os.killpg(self._proc.pid, signal.SIGKILL)
+                    else:
+                        self._proc.kill()
+                except Exception:
+                    pass
+                self._proc.wait(timeout=10)
+        if sys.platform != "win32":
+            # The server can be gone while its children are not: a vLLM API server that died leaves
+            # VLLM::EngineCore holding the GPU, and every later trial then sees less memory. The
+            # children share the server's process group, so reap whatever is left of it.
+            try:
+                os.killpg(self._proc.pid, signal.SIGKILL)
+            except OSError:
+                pass
+        if self._log_fh:
+            self._log_fh.close()
+            self._log_fh = None
+
+    def tail_log(self, n: int = 40) -> str:
+        if not self.log_path or not self.log_path.exists():
+            return ""
+        try:
+            lines = self.log_path.read_text(errors="replace").splitlines()
+            return "\n".join(lines[-n:])
+        except OSError:
+            return ""
+
+
+CTX_GRID = (2048, 4096, 8192, 16384, 32768, 65536, 131072)
+CTX_GRID_WIDTH = 3  # sizes tried per calibration: the smallest that fits the workload and the next two
+
+
+def ctx_grid(max_pos: int, min_ctx: int = 0) -> List[int]:
+    """Context lengths to try: the three smallest standard sizes that fit the model and hold the workload.
+
+    Returns [] if the model cannot hold `min_ctx` at all.
+    """
+    if min_ctx > max_pos:
+        return []
+    grid = [c for c in CTX_GRID if min_ctx <= c <= max_pos]
+    if not grid:
+        # Workload needs more than the largest standard size that fits: use the model max.
+        grid = [max_pos]
+    return grid[:CTX_GRID_WIDTH]
+
+
+def render_extra(extra: Dict[str, object], skip: Tuple[str, ...] = ()) -> List[str]:
+    """Extra launch flags. `True` renders as a bare flag; `False` and None are dropped."""
+    out: List[str] = []
+    for k, v in extra.items():
+        if k in skip or v is None or v is False:
+            continue
+        flag = f"--{k.replace('_', '-')}"
+        out += [flag] if v is True else [flag, str(v)]
+    return out
+
+
+def free_port(preferred: Optional[int] = None) -> int:
+    if preferred:
+        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+            try:
+                s.bind(("127.0.0.1", preferred))
+                return preferred
+            except OSError:
+                pass
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        s.bind(("127.0.0.1", 0))
+        return s.getsockname()[1]
+
+
+@runtime_checkable
+class Backend(Protocol):
+    name: str
+
+    def available(self, hw: HardwareDescriptor) -> bool: ...
+
+    def supports(self, hw: HardwareDescriptor, model: ModelSpec) -> bool: ...
+
+    def prepare(self, model: ModelSpec, hw: HardwareDescriptor, quants: Optional[List[str]] = None) -> PreparedModel: ...
+
+    def memory_model(self, hw: HardwareDescriptor) -> MemoryModel: ...
+
+    def estimate_memory(self, cfg: Config, model: PreparedModel, hw: HardwareDescriptor) -> int: ...
+
+    def candidate_configs(self, hw: HardwareDescriptor, model: PreparedModel, min_ctx: int = 0) -> List[Config]: ...
+
+    def launch_spec(self, cfg: Config, model: PreparedModel, port: int) -> LaunchSpec: ...
+
+    def launch(self, cfg: Config, model: PreparedModel, port: int, log_path: Optional[Path] = None) -> Process: ...
+
+    def workload_hooks(self, hw: HardwareDescriptor, model: PreparedModel) -> LlmtraceHooks: ...
+
+    def default_config(self, hw: HardwareDescriptor, model: PreparedModel, min_ctx: int = 0) -> Config: ...
+
+    def version(self, hw: HardwareDescriptor) -> Optional[str]: ...
+
+
+class BaseBackend:
+    """Shared plumbing. Concrete backends override the abstract-ish methods."""
+
+    name: str = "base"
+    runtime_workspace_bytes: int = 1024 * 1024 * 1024
+
+    def available(self, hw: HardwareDescriptor) -> bool:
+        return hw.backend_available(self.name)
+
+    def version(self, hw: HardwareDescriptor) -> Optional[str]:
+        return hw.backend_version(self.name)
+
+    def memory_model(self, hw: HardwareDescriptor) -> MemoryModel:
+        raise NotImplementedError
+
+    def calibrated_memory(self, hw: HardwareDescriptor, kv_tokens_fn, device: str) -> MemoryModel:
+        """MemoryModel using this machine's fitted workspace/margin when `polyserve memory-report --apply` ran."""
+        from polyserve.hardware import hardware_hash
+        from polyserve.memcal import margin_override, workspace_override
+        from polyserve.memory import DEFAULT_MARGIN_FRACTION
+
+        hh = hardware_hash(hw)
+        ws = workspace_override(hh, self.name)
+        mf = margin_override(hh, self.name)
+        return MemoryModel(
+            runtime_workspace=ws if ws is not None else self.runtime_workspace_bytes,
+            kv_tokens_fn=kv_tokens_fn,
+            device=device,
+            margin_fraction=mf if mf is not None else DEFAULT_MARGIN_FRACTION,
+            calibrated=ws is not None or mf is not None,
+        )
+
+    def estimate_memory(self, cfg: Config, model: PreparedModel, hw: HardwareDescriptor) -> int:
+        from polyserve.memory import estimate
+
+        return estimate(hw, model, cfg, self.memory_model(hw)).total
+
+    def materialize(self, model: PreparedModel, quants: List[str]) -> PreparedModel:
+        """Download / convert weights for the quants the planner kept. Default: nothing to do."""
+        return model
+
+    def prefill_variants(self, cfg: Config) -> List[Config]:
+        """Configs differing from `cfg` only in the prefill knob. Default: this backend has none."""
+        return []
+
+    def disagg_launch_spec(self, cfg: Config, model: PreparedModel, port: int, role: str,
+                           kv_transfer_config: dict, gpu_index: int, side_channel_port: int) -> LaunchSpec:
+        """Launch one engine of a disaggregated prefill/decode pair."""
+        raise NotImplementedError(f"{self.name} does not support disaggregated prefill/decode")
+
+    # ---- optional search dimensions (default: this backend offers none)
+
+    supports_tp: bool = False
+
+    def supported_quants(self, hw: HardwareDescriptor) -> List[str]:
+        """Every weight precision this backend could run on `hw` (what --quant filters)."""
+        return []
+
+    def kv_dtypes(self, hw: HardwareDescriptor) -> List[str]:
+        """Quantized KV-cache types worth trying on `hw`."""
+        return []
+
+    def batch_ladder(self) -> Tuple[int, ...]:
+        """Batch sizes, smallest first, that a smaller KV cache may let the search step up to."""
+        return ()
+
+    def prefix_variants(self, cfg: Config) -> List[Config]:
+        """Prefix-cache settings to try when the workload's prompts share a prefix."""
+        return []
+
+    def spec_variants(self, cfg: Config, model: PreparedModel) -> List[Config]:
+        """Speculative-decoding settings to try."""
+        return []
+
+    def replica_launch_spec(self, cfg: Config, model: PreparedModel, port: int, gpu_index: int) -> LaunchSpec:
+        """One full engine pinned to one GPU, for the replicas layout."""
+        spec = self.launch_spec(cfg, model, port)
+        spec.env = {**spec.env, "CUDA_VISIBLE_DEVICES": str(gpu_index)}
+        return spec
+
+    def launch_spec(self, cfg: Config, model: PreparedModel, port: int) -> LaunchSpec:
+        raise NotImplementedError
+
+    health_path: str = "/health"
+
+    def workload_hooks(self, hw: HardwareDescriptor, model: PreparedModel) -> LlmtraceHooks:
+        return LlmtraceHooks(gpu_ids=[hw.gpu.index] if hw.gpu else [])
+
+    def launch(self, cfg: Config, model: PreparedModel, port: int, log_path: Optional[Path] = None) -> Process:
+        spec = self.launch_spec(cfg, model, port)
+        return Process(spec, port, f"http://127.0.0.1:{port}{self.health_path}", log_path=log_path).start()