rag-wright 0.5.1__tar.gz → 0.7.0__tar.gz

{rag_wright-0.5.1 → rag_wright-0.7.0}/.claude/skills/qwen-vllm-modal/SKILL.md

@@ -18,7 +18,7 @@ This skill is the operational checklist that ties them together.
 
 ## The LOCKED production config (ADR-0110) — do not re-litigate
 **Config B: `Qwen/Qwen3.8-27B-FP8` weights + `--kv-cache-dtype fp8`, on 1× A100-80GB, TP=1, `--max-model-len 16384`, `--gpu-memory-utilization 0.95`.**
-- Config B (FP8 weights) is the **locked deploy** — 52.71× concurrency @16K (ADR-0110 rounds it to 53×), ~23% higher throughput, ~27 GB weights;
+- Config B (FP8 weights) is the **locked deploy** — 52.71× concurrency @16K (ADR-0110 rounds it to 53×), ~23% higher throughput, ~31 GB weights (30.9 GB on disk, 28.9 GiB in VRAM);
   accuracy measured **identical to bf16** on the real compliance workload (no FP8 penalty).
 - **Config A** (`Qwen/Qwen3.8-27B`, **bf16 weights** + FP8 KV) is the **documented FALLBACK only** — 25.47× concurrency,
   "zero weight-loss" — use it *only* if a later clean check ever shows an FP8-weight regression. Don't default to A.
@@ -29,8 +29,8 @@ This skill is the operational checklist that ties them together.
 **From the engine (PS-19): deploy, then use the server like any model.** `ModelServerSpec`'s defaults ARE the
 locked config below (FP8 weights + FP8 KV, one A100-80GB, TP 1, 16K, 0.95, served `Qwen/Qwen3.8-27B`, hermes +
 qwen3 parsers, image builder 2025.06); state only the concurrency target, which ADR-0110 leaves to you.
-`adeploy_model_server` runs the deploy below with those settings, waits for `/health` (with progress; a cold start is
-~8 min), and registers a model profile with the server's own endpoint, so no `VLLM_BASE_URL` is needed:
+`adeploy_model_server` runs the deploy below with those settings, waits for `/health` (with progress; a restore from
+the snapshot is seconds, a snapshot-building boot 4-9 min, see Cold start below), and registers a model profile with the server's own endpoint, so no `VLLM_BASE_URL` is needed:
 
 ```python
 from rag_wright.api import EngineConfig, ModelServerSpec, adeploy_model_server, stop_model_server
@@ -48,19 +48,21 @@ different servers (and URLs), for example one per tenant.
 
 **The exact working deploy (this succeeded):**
 ```
-MODAL_IMAGE_BUILDER_VERSION=2025.06 \
+VLLM_API_KEY=<a long random string> MODAL_IMAGE_BUILDER_VERSION=2025.06 \
   APP_NAME=rw-qwen3-modal MODEL=Qwen/Qwen3.8-27B-FP8 KV_CACHE_DTYPE=fp8 \
   SERVED_NAME=Qwen/Qwen3.8-27B TOOL_PARSER=hermes REASONING_PARSER=qwen3 \
   GPU=A100-80GB:1 TP=1 MAX_LEN=16384 GPU_UTIL=0.95 MAX_NUM_SEQS=<target> \
   uv run --no-sync modal deploy scripts/modal_qwen3_vllm_server.py
 ```
 (From a product repo, deploy the copy in this skill's directory: `.claude/skills/qwen-vllm-modal/scripts/modal_qwen3_vllm_server.py` once the skills are linked.)
-Then wire the engine: `.env` `VLLM_BASE_URL=https://<workspace>--rw-qwen3-modal-serve.modal.run/v1`,
-`VLLM_API_KEY=rw-vllm-dev-key`; the profile `qwen3.8-27b-modal` routes there. **Stop billing when done:**
+The script refuses to deploy without `VLLM_API_KEY`: Modal's web endpoint is public and unauthenticated, so vLLM's
+`--api-key` is the only gate on a billing A100, and there is no default key. Then wire the engine: `.env`
+`VLLM_BASE_URL=https://<workspace>--rw-qwen3-modal-serve.modal.run/v1` and the same `VLLM_API_KEY`; the profile
+`qwen3.8-27b-modal` routes there (the seam also refuses an unset key). **Stop billing when done:**
 `uv run --no-sync modal app stop rw-qwen3-modal --yes` (A100 is expensive).
 
 **The engine does NOT reach Modal by default.** Every role's default is `_PRODUCT_LLM = "qwen3.8-27b-modal-or"`,
-the same model routed through OpenRouter (the Modal cold start is unsolved). To send calls to the Modal server, set
+the same model routed through OpenRouter (a Modal cold start can still take minutes when Modal builds a snapshot). To send calls to the Modal server, set
 `RAG_MODEL_ALL=qwen3.8-27b-modal` (all roles), a per-role `RAG_MODEL_<ROLE>`, or `EngineConfig.models`. For bulk
 closed-vocab labeling, the reasoning-off profile `qwen3.8-27b-modal-nothink` exists (opt-in; see the reasoning
 caveat below). `VISION_OCR` now defaults to the product LLM too, but image inputs through the Modal vLLM server are
@@ -96,17 +98,31 @@ the OFFICIAL prebuilt vLLM image** rather than pip-building vLLM:
    reasoning calls take ~50–60 s; the seam's default 60 s timeout kills the ones just over the line → **retry pileup →
    requests pile past `max_inputs` → instant `APIConnectionError` (0.0 s) cascade**. FIX: the seam reads
    `RAG_STRUCTURED_TIMEOUT_S` (default 60) — raise it (e.g. `150`) for reasoning-ON bulk work. Server logs confirm the
-   calls actually complete 200 OK (~50 s); it was the client giving up.
+   calls actually complete 200 OK (~50 s); it was the client giving up. The total per call (all tries) follows it;
+   `RAG_MODEL_DEADLINE_S` overrides that (ADR-0133). Reasoning loops end at the profile's 8,192-token structured cap,
+   not at the deadline; keep temperature 0 (Qwen's recommended sampling did not remove them).
 3. **Pick client concurrency empirically, below the ceiling.** 16 was stable; 20 (== `max-num-seqs`) sat at the exact
    ceiling and risked queueing. `Running: N, Waiting: 0` + zero cancellations = healthy.
 4. **Reasoning ON vs OFF for bulk classification:** reasoning-OFF is ~30× faster but **changes the labels**
    (over-keeps / different distribution) — do NOT swap it in for work that must match the reasoning-teacher's gold
    (e.g. distillation silver). Keep reasoning ON when fidelity to the gold teacher matters; accept the cost.
+5. **Never expose vLLM's port directly (PS-20).** Sleep mode needs `VLLM_SERVER_DEV_MODE=1`, which adds
+   unauthenticated `/sleep`, `/wake_up` and `/collective_rpc` routes (`--api-key` guards only `/v1`, `/v2`,
+   `/inference`, `/cohere`). The script binds vLLM to 127.0.0.1:8001 and serves `rw_gate.py` on the public port,
+   forwarding only `/v1/*` and `/health`; keep `rw_gate.py` next to the script.
 
 ## Cold start & cost
-- Cold start ~440–480 s (CPU-bound engine init + a 51-graph CUDA-graph capture + first-time weight download), NOT
-  weight I/O — no load trick alone fixes it. GPU-snapshot / sleep-mode is a DEFERRED todo (the one attempt failed
-  because it was CPU-only; GPU retry not done). Run long deploys detached; poll `/health` for readiness.
+- **The server is a class restored from a Modal GPU memory snapshot (PS-20, ADR-0132).** A boot without a snapshot
+  starts vLLM with `--enable-sleep-mode`, warms it, sleeps it at level 1 (weights to host RAM, KV discarded) and
+  Modal snapshots the container; a restore only wakes vLLM (~3.5 s) and is ready ~13 s after its container starts.
+  Modal builds the first few snapshots after a deploy (4-9 min boots; 3-4 in our runs) and now and then another
+  (a new host, a preemption): budget for an occasional slow cold start. Redeploying keeps the snapshots; **`modal app
+  stop` discards them**. Snapshots need TP=1 (the script boots without one at TP>1).
+- Without a snapshot a cold start is ~2-4 min on Config B (114-245 s measured): about 30 s of imports in EACH of
+  vLLM's two processes (transformers' `importlib.metadata.packages_distributions()` stats every installed file),
+  a weight read of 12-154 s from the Volume (9P), then compile-cache load + CUDA-graph capture. Measured and NOT
+  worth adopting (ADR-0109 addendum): `--load-format runai_streamer`, precompiled `.pyc`, `HF_HUB_OFFLINE=1` (it
+  changes vLLM's compile-cache key every container, so each start recompiles). Poll `/health` for readiness.
 - **A100 is billed while up — `modal app stop <app> --yes` the moment you're done.** Verify with `modal app list`
   (state `stopped`, 0 tasks) + endpoint 404.
 - **Do NOT warm the GPU before the consumer is ready, and do NOT trust auto-scaledown to protect billing** (burned

rag_wright-0.7.0/.claude/skills/qwen-vllm-modal/scripts/modal_qwen3_vllm_server.py

@@ -0,0 +1,164 @@
+"""Qwen3.8-27B as an OpenAI-compatible vLLM web server on Modal A100-80GB. Two uses, one script:
+
+(1) QUANTIZATION ACCURACY EVAL (ADR-0110): serve one config under a stable served-name ("qwen3-eval").
+  MODEL=Qwen/Qwen3.8-27B     KV_CACHE_DTYPE=auto uv run --no-sync modal deploy scripts/modal_qwen3_vllm_server.py
+  MODEL=Qwen/Qwen3.8-27B-FP8 KV_CACHE_DTYPE=fp8  uv run --no-sync modal deploy scripts/modal_qwen3_vllm_server.py
+
+(2) 0048 part-2 JOINT RUN -- the UNIFIED endpoint both the product's agent-llm (tool-calling) and the engine
+    drive: Config B (FP8), ALL capabilities on (tool calls + reasoning + guided decoding), served under the
+    name the engine profile `qwen3.8-27b-modal` expects ("Qwen/Qwen3.8-27B"), sized for the measured ~16 peak
+    concurrency of unbounded reasoning outputs (2 GPUs to keep decode from saturating).
+  APP_NAME=rw-qwen3-modal MODEL=Qwen/Qwen3.8-27B-FP8 KV_CACHE_DTYPE=fp8 \
+    SERVED_NAME=Qwen/Qwen3.8-27B TOOL_PARSER=hermes REASONING_PARSER=qwen3 \
+    GPU=A100-80GB:2 TP=2 MAX_NUM_SEQS=32 \
+    uv run --no-sync modal deploy scripts/modal_qwen3_vllm_server.py
+  uv run --no-sync modal app stop <APP_NAME>   # tear down (stop billing)
+
+URL: https://<workspace>--<APP_NAME>-serve.modal.run  (+ /v1 for the OpenAI base). Weights + the serve-path
+torch.compile cache reuse the persistent Volumes (fast warm cold-start where a matching cache exists).
+
+Cold start (PS-20, ADR-0132): at TP=1 the server is restored from a Modal GPU memory snapshot. A boot without one
+starts vLLM, warms it, puts it to sleep (weights parked in host RAM) and Modal snapshots the container; a restore
+only wakes vLLM (~13 s from container start). Modal builds the first 2-3 snapshots per GPU type, and now and then
+another, on ordinary boots; stopping the app discards them. vLLM listens on 127.0.0.1 only: sleep mode needs its
+unauthenticated dev routes, so the public port is `rw_gate.py`, which forwards only /v1/* and /health.
+"""
+
+import os
+import subprocess
+import time
+import urllib.request
+from pathlib import Path
+
+import modal
+
+MODEL = os.environ.get("MODEL", "Qwen/Qwen3.8-27B")          # ...-FP8 for the quantized-weights config
+KV_DTYPE = os.environ.get("KV_CACHE_DTYPE", "auto")          # "auto" (bf16 KV, the reference) | "fp8"
+GPU = os.environ.get("GPU", "A100-80GB:1")
+TP = int(os.environ.get("TP", "1"))
+MAX_LEN = int(os.environ.get("MAX_LEN", "16384"))            # the real workload's context (9K+ prompts)
+GPU_UTIL = os.environ.get("GPU_UTIL", "0.95")
+MAX_NUM_SEQS = os.environ.get("MAX_NUM_SEQS", "256")         # concurrency ceiling; size to the eval's ~16 peak
+SERVED = os.environ.get("SERVED_NAME", "qwen3-eval")         # served-model-name clients ask for (profile match)
+TOOL_PARSER = os.environ.get("TOOL_PARSER", "")             # "" off | "hermes" -> tool calling (the product agent)
+REASONING_PARSER = os.environ.get("REASONING_PARSER", "")   # "" off | "qwen3" -> reasoning coexists with a tool call
+APP_NAME = os.environ.get("APP_NAME", "rw-qwen3-vllm-eval")  # distinct app (=distinct URL) per purpose
+API_KEY = os.environ.get("VLLM_API_KEY")  # vLLM's --api-key: the only gate on the server's public URL
+if not API_KEY:  # no default key (PS-21): a default would be public, and would gate a billing GPU
+    raise SystemExit("set VLLM_API_KEY to the server's API key (a long random string) before deploying")
+SNAPSHOT = TP == 1  # Modal: GPU memory snapshots do not work across several GPUs; TP>1 boots without one
+VLLM_PORT, PUBLIC_PORT = 8001, 8000  # vLLM on loopback; the gate on the public port
+GATE = "/root/rw_gate.py"
+
+HF_CACHE = "/root/.cache/huggingface"
+VLLM_CACHE = "/root/.cache/vllm"
+hf_vol = modal.Volume.from_name("rw-hf-cache", create_if_missing=True)
+vllm_cache_vol = modal.Volume.from_name("rw-vllm-serve-compile-cache", create_if_missing=True)
+
+app = modal.App(APP_NAME)
+vllm_image = (
+    # Base off the OFFICIAL prebuilt vLLM image: vllm + torch + xformers + flashinfer already installed as matched
+    # wheels, so a fresh build on a clean account compiles NOTHING (the xformers source-build / resolver failures
+    # only happen when pip/uv try to assemble that stack themselves). Latest/unpinned by design — `:latest` tracks
+    # newest vLLM, which is all we need since we only consume it. add_python omitted (the image ships Python).
+    # The vLLM image ships `python3` but no `python`; Modal's OWN pip bootstrap runs right after FROM (before any
+    # later layer) as `python -m pip …` -> exit 127. setup_dockerfile_commands runs BEFORE that bootstrap, so the
+    # symlink is in place when Modal bootstraps.
+    modal.Image.from_registry(
+        "vllm/vllm-openai:latest",
+        setup_dockerfile_commands=["RUN ln -sf $(command -v python3) /usr/local/bin/python"],
+    )
+    .entrypoint([])  # the image's ENTRYPOINT (the api_server) would hijack Modal's runtime; clear it
+    .pip_install("hf_transfer")  # enable fast HF weight downloads (HF_HUB_ENABLE_HF_TRANSFER below)
+    # Bake the runtime knobs into the image env: a deployed function reads the CONTAINER's env, not the deploying
+    # shell's, so an override must live here to reach the container (same lesson as the bench script). Xet disabled
+    # so a fresh weight download's open log handle can't block a Volume commit.
+    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1", "HF_HUB_DISABLE_XET": "1", "HF_HOME": HF_CACHE,
+          "MODEL": MODEL, "KV_CACHE_DTYPE": KV_DTYPE, "TP": str(TP), "MAX_LEN": str(MAX_LEN),
+          "GPU_UTIL": GPU_UTIL, "MAX_NUM_SEQS": str(MAX_NUM_SEQS), "SERVED_NAME": SERVED,
+          "TOOL_PARSER": TOOL_PARSER, "REASONING_PARSER": REASONING_PARSER, "VLLM_API_KEY": API_KEY,
+          # Sleep mode's /sleep and /wake_up exist only in dev mode; they stay on loopback (see the gate).
+          "VLLM_SERVER_DEV_MODE": "1", "RW_VLLM_PORT": str(VLLM_PORT), "RW_PUBLIC_PORT": str(PUBLIC_PORT),
+          "TORCHINDUCTOR_COMPILE_THREADS": "1"})  # Modal: torch.compile can otherwise break snapshot creation
+    .add_local_file(Path(__file__).resolve().parent / "rw_gate.py", GATE)  # resolve(): scripts/ holds a symlink
+)
+
+
+def _vllm(path: str, method: str = "GET", body: bytes | None = None, timeout: float = 600) -> int:
+    request = urllib.request.Request(f"http://127.0.0.1:{VLLM_PORT}{path}", data=body, method=method, headers={
+        "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"})
+    with urllib.request.urlopen(request, timeout=timeout) as response:
+        return response.status
+
+
+def _wait_healthy(proc: subprocess.Popen) -> None:
+    while True:
+        if proc.poll() is not None:
+            raise RuntimeError(f"vLLM exited with {proc.returncode}")
+        try:
+            if _vllm("/health", timeout=5) == 200:
+                return
+        except OSError:
+            pass
+        time.sleep(1)
+
+
+@app.cls(image=vllm_image, gpu=GPU, volumes={HF_CACHE: hf_vol, VLLM_CACHE: vllm_cache_vol},
+         timeout=3600, max_containers=1, scaledown_window=600,
+         memory=65536,  # host RAM for the sleeping weights (~29 GiB for Config B), which the snapshot captures
+         enable_memory_snapshot=SNAPSHOT, experimental_options={"enable_gpu_snapshot": SNAPSHOT})
+# Without this Modal feeds the container ONE HTTP request at a time (Running:1 + "cancellation signal" floods,
+# /health blocked) — vLLM's continuous batching never gets fed. Let many requests hit the one container; vLLM
+# batches them internally up to --max-num-seqs.
+@modal.concurrent(max_inputs=int(MAX_NUM_SEQS))
+class QwenServer:
+    """vLLM's OpenAI-compatible server for the configured Qwen config. Reuses the persisted weights + serve-path
+    torch.compile cache; compiled (production-faithful), optional FP8 KV, optional tool-calling + reasoning parsers so
+    ONE endpoint serves free-text, guided/JSON, tool calls, and reasoning together."""
+
+    @modal.enter(snap=True)
+    def start(self) -> None:
+        """Start vLLM, warm it, and put it to sleep: the snapshot is taken after this (when SNAPSHOT is on)."""
+        hf_vol.reload()
+        vllm_cache_vol.reload()
+        args = [
+            "vllm", "serve", MODEL, "--host", "127.0.0.1", "--port", str(VLLM_PORT), "--served-model-name", SERVED,
+            "--tensor-parallel-size", str(TP), "--dtype", "bfloat16", "--max-model-len", str(MAX_LEN),
+            "--gpu-memory-utilization", GPU_UTIL, "--max-num-seqs", str(MAX_NUM_SEQS),
+            "--api-key", API_KEY, "--trust-remote-code", "--enable-sleep-mode",
+        ]
+        if KV_DTYPE != "auto":
+            args += ["--kv-cache-dtype", KV_DTYPE]
+        if TOOL_PARSER:  # the product's deep-agent orchestrator is a tool-caller (RuleWright Q1/Q3)
+            args += ["--enable-auto-tool-choice", "--tool-call-parser", TOOL_PARSER]
+        if REASONING_PARSER:  # so reasoning coexists with a forced tool/schema instead of colliding
+            args += ["--reasoning-parser", REASONING_PARSER]
+        print(f"[qwen] serving {MODEL} as '{SERVED}' | TP={TP} max_len={MAX_LEN} util={GPU_UTIL} kv={KV_DTYPE} "
+              f"max_seqs={MAX_NUM_SEQS} tool={TOOL_PARSER or '-'} reasoning={REASONING_PARSER or '-'} "
+              f"snapshot={SNAPSHOT}", flush=True)
+        self.vllm = subprocess.Popen(args)
+        _wait_healthy(self.vllm)
+        warm = (b'{"model": "%s", "messages": [{"role": "user", "content": "Say OK."}], "max_tokens": 8, '
+                b'"chat_template_kwargs": {"enable_thinking": false}}' % SERVED.encode())
+        for _ in range(2):
+            _vllm("/v1/chat/completions", "POST", warm)
+        _vllm("/sleep?level=1", "POST", b"")  # level 1: weights to host RAM (in the snapshot), KV cache discarded
+        print("[qwen] warmed and asleep; snapshot next", flush=True)
+
+    @modal.enter(snap=False)
+    def wake(self) -> None:
+        """After a restore (or straight after `start` without one): wake vLLM, then open the public port."""
+        _vllm("/wake_up", "POST", b"")
+        _wait_healthy(self.vllm)
+        self.gate = subprocess.Popen(["python", GATE])
+        print("[qwen] awake; gate open", flush=True)
+
+    @modal.web_server(port=PUBLIC_PORT, startup_timeout=1200, label=f"{APP_NAME}-serve")
+    def serve(self) -> None:
+        """The public URL: the gate, which forwards /v1/* and /health to vLLM."""
+
+    @modal.exit()
+    def stop(self) -> None:
+        for proc in (getattr(self, "gate", None), getattr(self, "vllm", None)):
+            if proc:
+                proc.terminate()

rag_wright-0.7.0/.claude/skills/qwen-vllm-modal/scripts/rw_gate.py

@@ -0,0 +1,58 @@
+"""The model server's public front (PS-20): forwards only the OpenAI API (`/v1/...`) and `/health` to vLLM.
+
+vLLM's `--api-key` guards only `/v1`, `/v2`, `/inference` and `/cohere`. Sleep mode, which the GPU snapshot needs,
+requires `VLLM_SERVER_DEV_MODE=1`, and that adds unauthenticated `/sleep`, `/wake_up` and `/collective_rpc` routes. So
+vLLM listens on 127.0.0.1 only and this gate is what the public URL reaches. It runs inside the server container
+(`modal_qwen3_vllm_server.py` copies it into the image); responses stream through as they arrive.
+"""
+import os
+
+UPSTREAM = f"http://127.0.0.1:{os.environ.get('RW_VLLM_PORT', '8001')}"
+_HOP_BY_HOP = {"connection", "keep-alive", "transfer-encoding", "upgrade", "proxy-connection", "te", "trailer",
+               "host", "content-length"}
+
+
+def allowed(path: str) -> bool:
+    """The routes the public URL may reach: the OpenAI API and the health check."""
+    return path == "/health" or path.startswith("/v1/")
+
+
+async def _forward(request):
+    from aiohttp import web
+
+    if not allowed(request.path):
+        return web.json_response({"error": "Not Found"}, status=404)
+    headers = {k: v for k, v in request.headers.items() if k.lower() not in _HOP_BY_HOP}
+    async with request.app["session"].request(request.method, UPSTREAM + request.path_qs, headers=headers,
+                                              data=await request.read()) as upstream:
+        response = web.StreamResponse(status=upstream.status, headers={
+            k: v for k, v in upstream.headers.items() if k.lower() not in _HOP_BY_HOP})
+        await response.prepare(request)
+        async for chunk in upstream.content.iter_any():  # streamed completions pass through as they arrive
+            await response.write(chunk)
+        await response.write_eof()
+        return response
+
+
+async def _app():
+    from aiohttp import ClientSession, ClientTimeout, TCPConnector, web
+
+    app = web.Application(client_max_size=64 * 1024 * 1024)
+
+    async def open_session(app):
+        app["session"] = ClientSession(connector=TCPConnector(limit=0), timeout=ClientTimeout(total=None))
+
+    async def close_session(app):
+        await app["session"].close()
+
+    app.on_startup.append(open_session)
+    app.on_cleanup.append(close_session)
+    app.router.add_route("*", "/{tail:.*}", _forward)
+    return app
+
+
+if __name__ == "__main__":
+    from aiohttp import web
+
+    web.run_app(_app(), host=os.environ.get("RW_GATE_HOST", "0.0.0.0"),
+                port=int(os.environ.get("RW_PUBLIC_PORT", "8000")), print=None)

{rag_wright-0.5.1/src/rag_wright/.agents → rag_wright-0.7.0/.claude}/skills/using-the-rag-wright-engine/SKILL.md

@@ -141,7 +141,11 @@ Work the domain-adaptation guide, grounding each engine call (step 2) and writin
 `server = await adeploy_model_server(ModelServerSpec(name="<app>", max_num_seqs=<concurrency target>))` (the locked
 setup of the `qwen-vllm-modal` skill; needs `rag-wright[modal]`, a Modal login and `VLLM_API_KEY` in `.env`), then
 `EngineConfig(models={"general": server.model_id, ...})`. Other processes reconnect with `amodel_server(spec)`;
-call `await_model_server(server)` before traffic (cold start is minutes); `stop_model_server(name)` stops billing.
+call `await_model_server(server, timeout_s=<startup budget>)` before traffic (cold start is minutes; the default
+waits 25 min); `stop_model_server(name)` stops billing. `amodel_server_status(spec)` looks without waking it (deployed,
+running, healthy, weights cached). There is no default `VLLM_API_KEY`. Count concurrency end to end: `aingest` runs
+`min(document_concurrency, documents) x extract_concurrency` calls per run, parallel runs add up, and the sum must
+stay below `max_num_seqs` (`aingest` warns when its own peak exceeds it).
 Full walk-through: `docs/configuration.md`, "A self-hosted model server".
 
 **Using the decision model (Jev).** Register `jev_decision` (from `engine_capabilities()`) and set

rag_wright-0.7.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.7.0"
+}

{rag_wright-0.5.1 → rag_wright-0.7.0}/CHANGELOG.md

@@ -4,6 +4,25 @@ All notable changes to RAG_Wright are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.7.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.6.0...0.7.0) (2026-10-11)
+
+
+### Features
+
+* **PS-20:** the model server cold-starts from a Modal GPU memory snapshot behind a public-route gate (ADR-0132) ([1305237](https://github.com/fzaidi-dai/RAG_Wright/commit/13052378c57f31d126d281cda49c614ca9e7207b))
+* **PS-22:** the model-call deadline follows the configured timeout; a profile caps thinking models' structured output (ADR-0133) ([d327813](https://github.com/fzaidi-dai/RAG_Wright/commit/d3278130aaae10e7af3c326e503612337e7336cb))
+
+## [0.6.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.5.1...0.6.0) (2026-10-10)
+
+
+### ⚠ BREAKING CHANGES
+
+* **PS-21:** a self-hosted server call or deploy with VLLM_API_KEY (or the profile's api_key_env) unset now raises instead of using the default key "rw-vllm-dev-key".
+
+### Bug Fixes
+
+* **PS-21:** no default model-server key; concurrency ceiling, status and clearer errors (ADR-0131) ([a73a4fd](https://github.com/fzaidi-dai/RAG_Wright/commit/a73a4fd9595a9f8d70d95d393147ba10a1f1718b))
+
 ## [0.5.1](https://github.com/fzaidi-dai/RAG_Wright/compare/0.5.0...0.5.1) (2026-10-10)
 
 

{rag_wright-0.5.1 → rag_wright-0.7.0}/CLAUDE.md

@@ -55,10 +55,10 @@ The failure mode to avoid, which has happened before: a command fails with "modu
 
 ## Start of every session
 
-> **CURRENT STATE (2026-10-09): the public-surface workstream is COMPLETE and released as `rag-wright` 0.5.0** (ledger
+> **CURRENT STATE (2026-10-11): the public-surface workstream is COMPLETE and released as `rag-wright` 0.6.0** (ledger
 > `docs/specs/public-surface/plan.md`, ADR-0128: two public tiers, `rag_wright.api` for a product's seam and
 > `rag_wright.pack_sdk` for a pack's code, enforced by import contracts and a domain-vocabulary guard). Next:
-> RuleWright moves from `docs/specs/public-surface/rulewright-migration-0.3.0.md` then `rulewright-migration-0.4.0.md` and `-0.5.0.md`; PS-R4 (the unit-labelling eval) is done, unreleased. Before it,
+> RuleWright moves from `docs/specs/public-surface/rulewright-migration-0.3.0.md` then `rulewright-migration-0.4.0.md`, `-0.5.0.md` and `-0.6.0.md`; PS-R4 (the unit-labelling eval) is done, unreleased. Before it,
 > the **ingestion-hooks** workstream (ADR-0124) is
 > COMPLETE and released as **`rag-wright` 0.2.0** (PyPI + GitHub tag/release `0.2.0`; release process
 > `docs/releasing.md`): the generic `build_ingestion` with hooks and domain-neutral defaults, the neutral default

{rag_wright-0.5.1 → rag_wright-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rag-wright
-Version: 0.5.1
+Version: 0.7.0
 Summary: A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime.
 Project-URL: Homepage, https://github.com/fzaidi-dai/RAG_Wright
 Project-URL: Repository, https://github.com/fzaidi-dai/RAG_Wright

{rag_wright-0.5.1 → rag_wright-0.7.0}/docs/adr/0109-vllm-cold-start-reduction.md

@@ -45,6 +45,44 @@ Plausible target after the snapshot is built: **~10–20s cold restart**, well u
 - **Fallback if the alpha proves unstable:** the §4 supporting trims alone do NOT reach sub-60s (per the reframe), so the fallback for a hard sub-60s SLA reverts to keep-one-warm (the ruled-out cost) until the snapshot path matures.
 - No engine code changes here; this is deployment/serving infrastructure for the product substrate (ADR-0039). This ADR sets direction and diagnosis; a follow-up will record the prototype's measured result.
 
+## Addendum (2026-10-11): `runai_streamer` measured on Modal, not adopted
+
+The §4 trim `--load-format runai_streamer` was measured on the locked Config B (ADR-0110: `Qwen/Qwen3.8-27B-FP8`,
+fp8 KV, 1×A100-80GB, TP=1, 16K) with the weights cached on the `rw-hf-cache` Volume and the AOT compile cache hit in
+every run. It is already in the `vllm/vllm-openai` image (vLLM's Dockerfile installs `runai-model-streamer`), and it
+streams from the Hugging Face cache when given a repo id. Seven cold starts, timed from container start:
+
+| Loader | Weight read (s) | "Model loading" (s) | Container start → ready (s) |
+|---|---|---|---|
+| default | 40.6 / 26.5 / 11.6 | 46.8 / 36.8 / 17.0 | 146 / 245 / 114 |
+| `runai_streamer`, concurrency 16 | 30 / 31 / 56 | 36.9 / 61.9 / 64.8 | 154 / 167 / 185 |
+| `runai_streamer`, concurrency 32 | 45 | 57.0 | 218 |
+
+**Decision: keep the default loader.** Means: weight read 26 s (default) against 39 s (streamer at 16); start to
+ready 168 s against 169 s. On a Modal Volume the read rate swings 0.76 to 2.7 GB/s with the default loader alone,
+depending on whether the files are already cached, which is larger than any loader effect; the streamer's
+published gains (2.4× on local NVMe, up to 6× from object storage) do not transfer. The streamer's "Model loading"
+also carries ~30 s beyond its read. The weight read is 10-30% of a cold start; the remainder is process start
+(~70-120 s before the load begins) and engine init (~30-50 s), which confirms the reframe above: the cold-start
+work is the snapshot path, not the loader.
+
+Two corrections to figures above. Config B's weights are **30.9 GB on disk (28.9 GiB in VRAM)**, not ~27 GB. The
+current cold start with weights and compile cache warm is **~2-4 minutes** (114-245 s here), not the ~481 s
+measured for the bf16 model in September.
+
+## Addendum (2026-10-11): where the non-snapshot cold start goes, and the outcome
+
+Timed from the server function's start in three default-loader cold starts (146 / 245 / 114 s): the API server
+process reaching its arguments 30 / 80 / 31 s; the engine process initialised 31 / 70 / 29 s later; model loading
+47 / 37 / 17 s; compile-cache load and CUDA-graph capture 31 / 45 / 31 s. The first two are imports: on a CPU
+container the API server's import takes 37 s cold and 28-29 s warm, the same with every module precompiled to
+bytecode (so not compilation). The largest single cost is transformers' `import_utils`, whose import calls
+`importlib.metadata.packages_distributions()` (it stats every file of every installed package) and reads its own
+model modules (21,413 `stat` calls in one `import transformers`). `HF_HUB_OFFLINE=1` removes the Hub lookups but
+resolves the model to a local snapshot path, which changed vLLM's compile-cache key in each container (a 73-104 s
+recompile per start), so it is not adopted. **Outcome: the snapshot path (decision 2-3 above) is adopted; see
+ADR-0132.**
+
 ### Sources
 - Modal GPU snapshots (alpha, what's captured): https://modal.com/blog/gpu-mem-snapshots ; caveats (multi-GPU incompat, storage-bound caveat, torch.compile): https://modal.com/docs/guide/memory-snapshots ; vLLM example: https://modal.com/docs/examples/lfm_snapshot , https://modal.com/docs/examples/gpu_snapshot
 - Cold-start is CPU-bound; Tensorizer fastest: https://arxiv.org/abs/2606.07362 ; 35B breakdown: https://route179.dev/2026/06/30/optimizing-vllm-cold-start-with-model-streaming-and-compile-caching/
@@ -52,3 +90,4 @@ Plausible target after the snapshot is built: **~10–20s cold restart**, well u
 - vLLM loaders: runai_streamer https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/ ; Tensorizer https://docs.vllm.ai/en/v0.10.1/models/extensions/tensorizer.html
 - vLLM CUDA graphs / capture cost: https://docs.vllm.ai/en/stable/design/cuda_graphs/
 - Modal Volume 1–2 GB/s, keep-warm knobs: https://modal.com/docs/guide/high-performance-llm-inference , https://modal.com/docs/guide/cold-start , https://modal.com/docs/guide/scale
+- vLLM loaders (cont.): runai_streamer source https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/model_loader/runai_streamer_loader.py ; route179 35B breakdown (2.4× on NVMe) above

rag_wright-0.7.0/docs/adr/0131-model-server-hardening-after-rulewright-review.md

@@ -0,0 +1,34 @@
+# ADR-0131: No default server key anywhere; a model's concurrency ceiling travels with its id
+
+**Status:** accepted · **Date:** 2026-10-11 · **Related:** ADR-0130 (deploy the model server from the engine),
+ADR-0110 (`max_num_seqs` is the concurrency target), PS-21 (RuleWright's model-server review)
+
+## Context
+
+RuleWright's first use of the model-server API found five gaps. The deploy script (and six dev scripts) fell back to
+a key committed in the repo when `VLLM_API_KEY` was unset, and Modal's web endpoint is public and unauthenticated, so
+the raw `modal deploy` path the skill documents put a billing A100 behind a published key; the seam sent the same
+default. A product could not tell whether its load fits the server, because the engine's own ingestion fan-out
+multiplies with the product's and nothing compared it to `max_num_seqs`. Cache state, a not-deployed error and the
+startup timeout were also unclear.
+
+## Decision
+
+1. No default key. Every Modal vLLM server script raises on an unset `VLLM_API_KEY` (and bakes it into the image
+   env, which the container reads); the seam's `vllm` backend raises instead of sending a default. A test fails if
+   the old default string reappears under `src/`, `scripts/` or `docs/*.md`. Breaking, approved.
+2. `ModelProfile.max_concurrency` (optional; approved schema change): the most requests the model's server batches
+   at once. A deployed server registers its `max_num_seqs` there. `aingest` warns (a `RuntimeWarning` and a progress
+   line) when `min(document_concurrency, documents) x extract_concurrency` exceeds the ceiling of any model the
+   workspace resolves. It cannot see a product's other parallel runs; the docs give the arithmetic.
+3. `amodel_server_status(spec)`: deployed, running, healthy, weights cached. It asks `/health` only of a running
+   container (a request to a scaled-down server starts a cold start) and reads the weights Volume (`rw-hf-cache`,
+   pinned to the script by a test).
+4. `ModelServerNotDeployed` (a `RuntimeError`) replaces Modal's raw `NotFoundError`; `astop_model_server` is the async
+   stop.
+
+## Consequences
+
+- A hand deploy or a client without `VLLM_API_KEY` now fails with a message naming the variable.
+- The ingestion warning covers the generic pipeline's extractor fan-out; other callers (a pack's own concurrency,
+  queries) are covered by the documented arithmetic, not a check.

rag_wright-0.7.0/docs/adr/0132-model-server-cold-start-from-a-gpu-snapshot.md

@@ -0,0 +1,45 @@
+# ADR-0132: The model server cold-starts from a Modal GPU memory snapshot, behind a public-route gate
+
+**Status:** accepted · **Date:** 2026-10-11 · **Related:** ADR-0109 (cold-start reduction: direction and the
+measurements behind it), ADR-0110 (Config B), ADR-0130/0131 (the model-server API), PS-20
+
+## Context
+
+A cold start of the self-hosted Qwen server (Config B on one A100) took 114-245 s, sometimes more, every time a
+scaled-down server got a request. Measured on 2026-10-11 (ADR-0109 addenda): about 30 s of imports in each of vLLM's
+two processes, a 12-154 s weight read from the Volume, then compile-cache load and CUDA-graph capture. The levers
+that do not remove startup were measured and rejected: the Run:ai streamer, precompiled bytecode, and
+`HF_HUB_OFFLINE=1` (it changes vLLM's compile-cache key in every container, so each start recompiles).
+
+## Decision
+
+1. **The server is a Modal class restored from a GPU memory snapshot** (`enable_memory_snapshot` +
+   `experimental_options={"enable_gpu_snapshot": True}`, alpha). `@modal.enter(snap=True)` starts vLLM with
+   `--enable-sleep-mode`, warms it with two requests and sleeps it at level 1 (weights to host RAM, ~29 GiB, KV
+   discarded; the class requests 64 GiB of memory); Modal snapshots the container. `@modal.enter(snap=False)` wakes
+   vLLM. `TORCHINDUCTOR_COMPILE_THREADS=1` as Modal requires. Snapshots only at TP=1 (Modal: incompatible with
+   several GPUs); TP>1 boots without one.
+2. **vLLM listens on 127.0.0.1 only; the public port is `rw_gate.py`**, which forwards `/v1/*` and `/health` and
+   returns 404 for everything else. Sleep mode needs `VLLM_SERVER_DEV_MODE=1`, which adds unauthenticated `/sleep`,
+   `/wake_up` and `/collective_rpc` routes, and vLLM's `--api-key` guards only `/v1`, `/v2`, `/inference` and
+   `/cohere` (v0.30.0 `serve/middleware/authenticate.py`). Modal's published vLLM snapshot examples expose those
+   routes publicly; we do not.
+3. **The URL does not change**: `@modal.web_server(..., label=f"{APP_NAME}-serve")` keeps
+   `https://<workspace>--<app>-serve.modal.run`. The engine looks the server up as
+   `modal.Cls.from_name(name, "QwenServer")().serve` (the URL and the container count; Modal's public
+   `Function.from_name` refuses a class's `<Class>.*` name). The public API is unchanged.
+
+## Consequences
+
+- **A restore is ready ~13 s after its container starts** (wake ~3.5 s); 21-67 s end to end in our runs, including
+  getting a container. One restore took ~112 s (Modal loading the snapshot slowly).
+- **Modal decides per container whether to restore or build**, and a build is a full boot plus the snapshot: 4-9
+  minutes. It built 3-4 snapshots after each deploy in our runs (one after a GPU preemption) and another later in
+  one run. So a scaled-to-zero server usually answers in seconds and occasionally makes a request wait minutes; a
+  product that cannot accept that keeps the server warm during working hours.
+- **Redeploying keeps the snapshots; stopping the app discards them**, so the next deploy pays the builds again.
+- **Alpha-feature risk** stays (Modal labels GPU snapshots alpha). If a restore fails, Modal boots the container in
+  full, which is the pre-snapshot behaviour.
+- Live gate (engine API only): deploy, a question metered on the server's model id, status while up, public
+  `/sleep` and `/collective_rpc` 404, three cold starts, a redeploy then a restored cold start (67 s), stop, status
+  not deployed. Prototype runs: restores in 4 of 11 cold starts across two deployments, all after the first builds.

rag_wright-0.7.0/docs/adr/0133-model-call-deadline-follows-the-timeout-and-a-thinking-cap.md

@@ -0,0 +1,40 @@
+# ADR-0133: The model-call deadline follows the configured timeout, and a thinking model's structured output is capped by its profile
+
+**Status:** accepted · **Date:** 2026-10-11 · **Related:** ADR-0056/0057 (one bounded retry layer; the async
+wall-clock deadline), ADR-0110 (the timeout trap on reasoning-on calls), PS-22
+
+## Context
+
+The 2026-10-11 sampling A/B (ContractNLI, 150 pairs, thinking on, Qwen on our A100) found two things:
+
+1. The async seam cancelled every logical call at a fixed 180 s (`_MODEL_DEADLINE_S`), whatever
+   `RAG_STRUCTURED_TIMEOUT_S` said. The docs and the qwen skill tell people to raise that per-try timeout for
+   reasoning-on work, and past 180 s in total it did nothing. A loop and a slow healthy call both ended as
+   `ModelCallTimeout`, and the ContractNLI eval scored that as "neutral".
+2. Reasoning loops are real and random: one pair ran to 8,192 tokens in 1 of 3 tries at Qwen's recommended
+   sampling, and correct verdicts took up to ~7k tokens. Qwen's recommended sampling did not remove loops (greedy
+   stays the default).
+
+## Decision
+
+1. **`model_deadline_s()`** is `RAG_MODEL_DEADLINE_S` when set; otherwise the per-try timeout times the tries plus
+   the largest backoffs, never under 180 s. Every async seam call runs under it, and `ModelCallTimeout` names both
+   settings.
+2. **`ModelProfile.structured_max_tokens`** is the default `max_tokens` for a forced structured call (a caller's own
+   value wins). The four thinking-on Qwen profiles set 8,192, so a loop ends at `finish_reason=length`, a parse
+   failure the bounded retry handles, rather than at the deadline. A server deployed from the engine inherits it.
+3. **The cap never fails a call that worked without it.** vLLM rejects a request whose prompt plus `max_tokens`
+   exceeds the context with a 400 naming `input_tokens` (`vllm/renderers/params.py`), so that error is answered by
+   sending the call once more without the cap. A caller's own `max_tokens` keeps today's behaviour (no retry).
+4. `eval/contractnli_judge.py` reports failed calls as errors, never as a verdict.
+
+Rejected: vLLM's `thinking_token_budget` (needs `--reasoning-config` on the server and is not supported by vLLM's V2
+model runner yet, and it is vLLM-only), and higher sampling temperature as a loop fix (measured, above).
+
+## Consequences
+
+- Raising `RAG_STRUCTURED_TIMEOUT_S` now raises the total budget too; the default is ~182 s (was 180 s).
+- A looping thinking call costs at most 8,192 output tokens per try instead of running until cancelled.
+- Free-text calls are not capped by the profile (thinking is off on that path in every Qwen profile).
+- RuleWright: callers that turn a failed call into a result should treat `ModelCallTimeout` as an error, not a
+  verdict (`rulewright-migration-0.7.0.md`).

{rag_wright-0.5.1 → rag_wright-0.7.0}/docs/adr/README.md

@@ -1,7 +1,7 @@
 # Architecture Decision Records — index
 
-128 ADR files (0001–0119, plus 0120 — the former duplicate `0015`, renumbered 2026-10-05 to repair the collision —
-and 0121–0130). ADRs are an
+131 ADR files (0001–0119, plus 0120 — the former duplicate `0015`, renumbered 2026-10-05 to repair the collision —
+and 0121–0133). ADRs are an
 immutable, cross-referenced decision log: nothing here is moved or renumbered — legacy records carry a `Status:`
 banner pointing to what replaced them. This index groups them so a newcomer isn't drowned.
 
@@ -186,4 +186,7 @@ framing that still appears in older ADRs and the root `SPEC.md`/`plan.md`.
 0127 Retire the OKF code ·
 0128 Two public tiers: `rag_wright.api` for products, `rag_wright.pack_sdk` for packs ·
 0129 The shipped Laya temperatures stay as fitted; scikit-learn floor at 1.9.1 ·
-0130 A product deploys the self-hosted model server from the engine and uses it like any model
+0130 A product deploys the self-hosted model server from the engine and uses it like any model ·
+0131 No default server key anywhere; a model's concurrency ceiling travels with its id ·
+0132 The model server cold-starts from a Modal GPU memory snapshot, behind a public-route gate ·
+0133 The model-call deadline follows the configured timeout; a thinking model's structured output is capped by its profile