rag-wright 0.6.0__tar.gz → 0.8.0__tar.gz

{rag_wright-0.6.0/src/rag_wright/.agents → rag_wright-0.8.0/.claude}/skills/building-an-ingestion-capability/SKILL.md

@@ -136,6 +136,11 @@ network-bound work inside it; the engine runs up to `IngestionTuning.extract_con
 
 - **Hidden sheets** are ingested by default; `IngestSource(include_hidden_sheets=False)` skips them (listed in
   `DocumentReport.skipped_hidden_sheets`).
+- **Blank rows inside a table**: the parse cuts a sheet's table at every blank row; the engine glues a piece back
+  onto the table above when its first row is data (its number or date columns agree, or it is formatted like the
+  data rather than the header), so no record is read as a header. A text-only table with no formatting cannot be
+  decided from the file: it stays apart and `evaluate_ingestion` lists it in `warnings` (format the header row, or
+  remove the blank rows).
 - **Exact cell values**: `table_rows(parse_document(...))` returns every data row as a `TableRow` (`columns`,
   `values`, `cell(name)`), read from the parse's cell grid, whole even when chunking split the table. A per-row unit
   carries its row in `Unit.table_row`, so the extractor reads cells, not re-parsed text.

{rag_wright-0.6.0 → rag_wright-0.8.0}/.claude/skills/creating-evals/SKILL.md

@@ -6,8 +6,10 @@ description: >-
   starting a new domain (right after capabilities are defined), when adding or changing a capability, or when A/B-ing
   alternatives (e.g. a trained classifier vs a System-1 decision model). Covers gold-set design + reliability,
   per-capability-KIND metrics (extraction / classification / retrieval / graph / generation / judgment), the
-  gate-vs-diagnostic split, building the gold cheaply, an isolated executable harness, and optional Langfuse
-  Datasets/Experiments/Scores automation. An eval is the executable acceptance criterion; training data IS an eval.
+  gate-vs-diagnostic split, building the gold cheaply, and running it with the engine's eval toolkit
+  (`load_gold_set` from an expert's own files, one `score_*` per kind, `arun_eval` with repeats and errors kept apart,
+  an `EvalReport` a backend can return, optional Langfuse datasets and runs). An eval is the executable acceptance
+  criterion; training data IS an eval.
 ---
 
 # Creating evals (eval-first / TDD)
@@ -19,6 +21,13 @@ can A/B alternatives and catch regressions forever. Corollary observed repeatedl
 data you build for a classifier/decision model IS an eval** (a labeled gold set + a metric) — so building the eval
 first also gives you the data design for free.
 
+**The toolkit (on `rag_wright.api`).** `load_gold_set` turns an expert's CSV / XLSX / JSONL into a `GoldSet`;
+one scorer per capability kind (section 2) turns predictions into a `ScorerResult` (the gate, its thresholds,
+diagnostics that are never gated); `arun_eval` runs your capability over the gold set (concurrently, `[eval] i/N`
+progress, repeats, a failed call kept as an error) and returns an `EvalReport` -- JSON-ready, every item's outputs
+kept, so `rescore_eval` can score it again later without calling anything. Use it rather than hand-writing a
+harness; section 6 is a complete example.
+
 Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`) are in the engine repository: read them there or on GitHub, at the tag matching your installed engine. The `eval/` and `scripts/` files named as patterns are the engine's own evals: read them as
 worked examples; your evals live in your repo.
 
@@ -50,12 +59,30 @@ worked examples; your evals live in your repo.
 - **Separate a tuning set from a held-out set.** Label the held-out set BEFORE any prompt runs on it, tune only on
   the tuning set, and report both numbers (ADR-0122 boundary residue prompt: 94.5% held-out vs 96.2% tuning). A
   number measured only on the set you tuned on is not a result.
+- **Gold from your experts' own files.** Domain experts label in spreadsheets: `load_gold_set(path_or_upload,
+  GoldMapping(id=..., input=..., expected=..., group=...), name=..., version=...)` reads them as written -- the header
+  found under a title, blank rows skipped, values kept exactly, each item's place recorded -- and refuses anything
+  that would make the gold wrong (a missing column, a row without its id, a duplicate id) with where it is. Map
+  `group` to the source document so splits stay leakage-safe. A gold set is `local_only` by default: client data
+  never leaves the machine.
 - **Gold stays local when the corpus is restrictively licensed** (CUAD/ACORD-derived gold lives under the
   gitignored `data/eval/`); commit the builder and the scorer, never the data.
 
 ## 2. Pick the metric by capability KIND, and split GATE vs DIAGNOSTIC
 Always set ONE pass/fail **gate** (from the acceptance criterion) and report **diagnostics** alongside (never gate
-on a diagnostic).
+on a diagnostic). Each kind has its scorer on `rag_wright.api`, checked against the engine evals named below:
+
+| kind | scorer | gate |
+|---|---|---|
+| classification / typed decision | `score_classification` | the weakest class's recall@k (+ top-1 accuracy via `top1_floor`) |
+| retrieval | `score_retrieval` | mean recall@k at a relevance grade; nDCG reported |
+| extraction | `score_extraction` | recall and precision, apart |
+| judgment / verdict | `score_judgment` | precision and recall of the actionable verdict |
+| generation / QA | `score_generation` | citation recall, correct abstention, groundedness |
+| graph / relational | `score_relational` | recall over the answer set |
+| ingestion structure / unit labelling | `score_ingestion` | `evaluate_ingestion`'s checks (+ label accuracy) |
+| candidate-then-choose | `score_candidate_choice` | candidate coverage, value-level recall and precision |
+
 - **Extraction** (section → records, clauses, requirements): recall / precision / F1 of extracted items vs gold,
   reported SEPARATELY (under- vs over-extraction are different failures). Watch over-extraction on non-operative
   input (definitions) and under-extraction on long input.
@@ -84,7 +111,7 @@ on a diagnostic).
   scores mean what they say), not one accuracy number. Pattern: `eval/semantic_judge_gold.py` (ING-9: decision
   model 95.3% vs LLM 90.1% on 192 blind hand-labelled cases).
 - **Candidate-then-choose pipelines** (a generator proposes, a model picks): measure the generator's **coverage**
-  separately, because it caps recall (ING-9b residual values: candidate coverage 95%; value-level recall 0.83 /
+  separately, because it caps recall (ING-9b residual values: candidate coverage 0.93, 188/203 recomputed from the saved outputs; value-level recall 0.83 /
   precision 0.88 vs the LLM's 0.60 / 0.79. Pattern: `eval/residual_decision_gold.py`). **Count is not accuracy**:
   emitting more values is not a win without precision.
 - **Ingestion structure**: `evaluate_ingestion` (in `rag_wright.api`) is the packaged structural eval of the
@@ -111,42 +138,87 @@ on a diagnostic).
 - **Generate hard cases** (near-boundary positives/negatives) to stress the exact confusions — TRAIN-ONLY; the
   gold test stays real. (Full data playbook: the `setfit` skill Phase 1/4 + `classifier-opportunity-analysis`.)
 
-## 4. Make it an executable, isolated harness
-- **Isolate the capability under test**: inject it (a seam / `extract_override` / an injected `retrieve`) so the
-  eval measures ONE capability, not the whole pipeline. Invoke production code THROUGH the capability layer
-  (`ainvoke_subgraph`/`ainvoke_model`), never a hand-built copy. Score the **shipped request** (the exact
-  prompt/question builder production uses, e.g. `judge_request`), not a copy of it re-typed in the eval.
-- **Repeatability for non-deterministic models.** Run each case several times on identical input; report how many
-  answers flip and how far the scores sit from the threshold. Flips cluster near the threshold and usually mean
-  the QUESTION is ambiguous: fix the question, do not vote it away (Jev flipped 5/181 answers at scores 0.47-0.56;
-  temperature/seed did not help, majority voting barely helped, rewriting the ambiguous question did. The ADR-0122
-  boundary residue prompt went from 94-96% with flips to 460/461, zero flips across 3 calls). Pattern:
-  `eval/boundary_residue_gold.py`.
+## 4. Run it: `arun_eval`
+`arun_eval(gold, task, scorers, *, repeats, concurrency, progress)` is the harness. It runs `task` (your coroutine,
+or a plain function, which runs off the event loop) on every item, `repeats` times, `concurrency` at a time, streams
+`[eval] i/N` lines to `progress`, keeps a call that raises as an ERROR in that item's slot (never dropped, never an
+answer), scores each repeat, and passes a scorer only if it passes on every repeat (its gate is the worst repeat;
+`per_repeat_gate` lists each). `run_eval` is the synchronous form; inside a web backend, await `arun_eval` as a
+background job. What is still yours:
+- **Isolate the capability under test**: the task calls ONE capability, injected (a seam / `extract_override` / an
+  injected `retrieve`), so the eval measures it, not the whole pipeline. Invoke production code THROUGH the
+  capability layer (`ainvoke_subgraph`/`ainvoke_model`), never a hand-built copy. Score the **shipped request** (the
+  exact prompt/question builder production uses, e.g. `judge_request`), not a copy of it re-typed in the eval.
+- **Repeatability for non-deterministic models.** Use `repeats` and read `report.flips`; flips cluster near the
+  threshold and usually mean the QUESTION is ambiguous: fix the question, do not vote it away (Jev flipped 5/181
+  answers at scores 0.47-0.56; temperature/seed did not help, majority voting barely helped, rewriting the ambiguous
+  question did. The ADR-0122 boundary residue prompt went from 94-96% with flips to 460/461, zero flips across 3
+  calls). Pattern: `eval/boundary_residue_gold.py`.
 - **Cache keys include the prompt and the method.** Any decision or extraction cache must key on the prompt text
   and the method (decision model vs LLM, and which model), so an eval never reuses outputs a different prompt or
   method produced (ADR-0122 ING-4d / ADR-0040 ING-9b: the residue decision cache and the clause cache).
-- **Score from a RESULT ARTIFACT (JSON), not stdout scraping** (scraping truncates and silently drops rows).
-- **Parallelize** model/LLM calls (async + semaphore) — same cost, far less wall-clock; order-preserving so it
-  stays deterministic.
-- **Env-selected** so the SAME harness runs local (dev) or on Modal (full corpus + GPU).
+- **Keep the report.** `report.model_dump_json()` holds every item's outputs; save it next to the gold version.
+  `rescore_eval(report, gold, scorers)` re-scores it with new thresholds, a new scorer or a corrected gold, free. An
+  eval kept only as aggregate numbers can never be checked again.
 - **Pre-flight paid bulk** (>~50 paid calls): print the exact count + cost and wait (`warn-before-bulk` rule).
-- Stream `X/N` progress + actively monitor any run over ~30s (never launch-and-forget).
 - **Hermetic tests never touch the network**: `tests/conftest.py` fails a test that resolves a non-local host unless
-  it carries a live marker (`model`, `store`, ...). Live evals run as `uv run python -u eval/<name>.py` (outside
-  pytest) or carry a live marker.
-
-## 5. Langfuse — optional eval automation (we already use it for tracing)
-Langfuse has a first-class eval stack we are NOT yet using (we use it only for spans/usage today): a **Dataset**
-(items = input + optional expected output) → a **Task** (your capability) run over the dataset as an **Experiment
-Run** → **Evaluators** (deterministic checks or LLM-as-judge) producing **Scores** (numeric/categorical/boolean),
-all linked to traces. Reach for it when you want **tracked, re-runnable, dashboarded** evals + regression tracking
-across capability versions; a local JSON harness is enough for a quick one-off gate. Keep the gold BUILDER + a
-committed snapshot in the repo (reproducibility); push the items to the Langfuse dataset so runs and scores land
-next to the traces you already collect. Ground the exact Dataset/Experiment API before wiring
-(https://langfuse.com/docs/evaluation/concepts, https://langfuse.com/docs/datasets/overview); confirm the installed
-SDK version's surface, not prose.
-
-## 6. The TDD loop
+  it carries a live marker (`model`, `store`, ...). Live evals run as scripts (outside pytest) or carry a live
+  marker.
+
+## 5. Langfuse -- optional: datasets, runs and scores next to the traces
+`arun_eval(..., langfuse=True)` (or `RAG_EVAL_BACKEND=langfuse`; the keys alone send nothing) runs the same eval on
+Langfuse: the gold set becomes a dataset (items upserted; items no longer in the gold set archived; the version is
+the server's latest timestamp, recorded as `report.dataset_version`), each repeat a dataset run (`report.run_urls`),
+and each scorer's gates and pass/fail are pushed as run scores. The report is the same, scored locally from the same
+outputs. With engine tracing on (`RAG_TRACE_LEVEL`), every model call the task makes nests under its item's trace,
+on both backends. What was verified against Langfuse 4.15.1 (self-hosted), and what to know:
+- **`local_only` is refused before anything is sent.** Only a self-hosted Langfuse should ever receive client data;
+  set `local_only=False` only for data that may leave the machine.
+- **Scoring runs in your process**, not in Langfuse: its code evaluators are a 2-second, standard-library sandbox
+  with no network, so they cannot run your scorers.
+- **The SDK's own `run_experiment` drops an item whose task raises, and blocks a running event loop**; `arun_eval`
+  handles both (an error is an error in its slot; the run is off the loop). Call `arun_eval`, not the SDK directly.
+- **Reading observations back**: the v2 list endpoint is Cloud-only until Langfuse v4 ships in the open-source
+  release, and the v1 endpoint times out on a trace-only query: bound every read by time.
+- Langfuse's managed LLM-as-a-judge and decision-model evaluators, and its UI prompt experiments, test a prompt in
+  Langfuse, not your pipeline: out of scope for this toolkit (a product may still use them).
+
+## 6. A complete example
+An expert's labelled workbook, your capability run through the capability layer, two repeats, the report kept and
+scored again later. `my_judge` stands for your capability's call (e.g. through `ainvoke_model`); the rest runs as
+written.
+
+```python
+import asyncio
+import functools
+from pathlib import Path
+
+from rag_wright.api import (EvalReport, GoldMapping, arun_eval, load_gold_set, rescore_eval,
+                            score_judgment)
+
+GOLD_FILE = "gold/nli_expert_labels.xlsx"  # the expert's own file
+REPORT = "eval_runs/nli_judge.json"
+
+mapping = GoldMapping(id="ID", input=["Premise", "Hypothesis"], expected="Label", group="Contract")
+gold = load_gold_set(GOLD_FILE, mapping, name="nli-judge", version="2026-10-12")
+
+
+async def judge(item):
+    return await my_judge(item.input["Premise"], item.input["Hypothesis"])
+
+
+gate = functools.partial(score_judgment, actionable="contradiction", precision=0.8, recall=0.7)
+report = asyncio.run(arun_eval(gold, judge, [gate], repeats=2, concurrency=8, progress=print))
+Path(REPORT).parent.mkdir(parents=True, exist_ok=True)
+Path(REPORT).write_text(report.model_dump_json(indent=1), encoding="utf-8")
+print(report.passed, report.scores[0].gate, "errors", report.errors, "flips", report.flips)
+
+stricter = functools.partial(score_judgment, actionable="contradiction", precision=0.9, recall=0.7)
+saved = EvalReport.model_validate_json(Path(REPORT).read_text(encoding="utf-8"))
+print("at 0.9 precision:", rescore_eval(saved, gold, [stricter]).passed)
+```
+
+## 7. The TDD loop
 1. Write the eval from the contract + a handful of real gold items → it fails (red) on the unbuilt/weak capability.
 2. Implement the minimum to pass the gate (green).
 3. A/B alternatives (rule / trained classifier / decision model / LLM) on the SAME gold; select on gate + diagnostics + cost + calibration.
@@ -158,7 +230,11 @@ SDK version's surface, not prose.
 - **Gating on a diagnostic** (e.g. nDCG) — report it, don't threshold it.
 - **No reliability check on a subjective gold** — you can't read a number off a gold whose own labels disagree.
 - **Eval not isolated** to the capability (measures the whole pipeline) — you can't attribute a regression.
-- **Scraping stdout** for metrics — write + read a JSON artifact.
+- **Scraping stdout** for metrics — keep and read the `EvalReport` JSON.
+- **A failed call scored as an answer** (a timeout read as "neutral", an error read as "abstained") — it is an error;
+  `arun_eval` keeps it apart (ADR-0133).
+- **Only aggregates kept** — an eval whose per-item outputs were not saved can never be re-scored or checked; keep
+  the report.
 - **Faking a win by shrinking the sample / picking easy items** — real, representative, honest.
 
 ## Hand-off / where this sits

{rag_wright-0.6.0 → rag_wright-0.8.0}/.claude/skills/qwen-vllm-modal/SKILL.md

@@ -16,9 +16,14 @@ ADR-0110 and the other `docs/` paths are in the engine repository: read them the
 matching your installed engine.
 This skill is the operational checklist that ties them together.
 
+**Which route.** A product deploys and reaches the server through the engine API, `adeploy_model_server` (the
+section after the locked config). The raw `modal deploy` of the bundled script further down is the engine
+developer's route, and what the API runs underneath. **When to stop it** is one rule, in "Cold start & cost": deploy
+once, let it scale to zero between uses, never stop it between runs.
+
 ## The LOCKED production config (ADR-0110) — do not re-litigate
 **Config B: `Qwen/Qwen3.8-27B-FP8` weights + `--kv-cache-dtype fp8`, on 1× A100-80GB, TP=1, `--max-model-len 16384`, `--gpu-memory-utilization 0.95`.**
-- Config B (FP8 weights) is the **locked deploy** — 52.71× concurrency @16K (ADR-0110 rounds it to 53×), ~23% higher throughput, ~27 GB weights;
+- Config B (FP8 weights) is the **locked deploy** — 52.71× concurrency @16K (ADR-0110 rounds it to 53×), ~23% higher throughput, ~31 GB weights (30.9 GB on disk, 28.9 GiB in VRAM);
   accuracy measured **identical to bf16** on the real compliance workload (no FP8 penalty).
 - **Config A** (`Qwen/Qwen3.8-27B`, **bf16 weights** + FP8 KV) is the **documented FALLBACK only** — 25.47× concurrency,
   "zero weight-loss" — use it *only* if a later clean check ever shows an FP8-weight regression. Don't default to A.
@@ -29,8 +34,8 @@ This skill is the operational checklist that ties them together.
 **From the engine (PS-19): deploy, then use the server like any model.** `ModelServerSpec`'s defaults ARE the
 locked config below (FP8 weights + FP8 KV, one A100-80GB, TP 1, 16K, 0.95, served `Qwen/Qwen3.8-27B`, hermes +
 qwen3 parsers, image builder 2025.06); state only the concurrency target, which ADR-0110 leaves to you.
-`adeploy_model_server` runs the deploy below with those settings, waits for `/health` (with progress; a cold start is
-~8 min), and registers a model profile with the server's own endpoint, so no `VLLM_BASE_URL` is needed:
+`adeploy_model_server` runs the deploy below with those settings, waits for `/health` (with progress; a restore from
+the snapshot is seconds, a snapshot-building boot 4-9 min, see Cold start below), and registers a model profile with the server's own endpoint, so no `VLLM_BASE_URL` is needed:
 
 ```python
 from rag_wright.api import EngineConfig, ModelServerSpec, adeploy_model_server, stop_model_server
@@ -38,7 +43,7 @@ from rag_wright.api import EngineConfig, ModelServerSpec, adeploy_model_server,
 server = await adeploy_model_server(ModelServerSpec(name="acme-qwen", max_num_seqs=32))  # key from VLLM_API_KEY
 config = EngineConfig(store=..., models={"general": server.model_id})   # or any model= argument
 ...
-stop_model_server("acme-qwen")                                            # stop billing
+stop_model_server("acme-qwen")  # only to retire it: stopping discards the snapshots (see "Cold start & cost")
 ```
 
 `amodel_server(spec)` reconnects to a server deployed earlier (a restart, another process) without redeploying.
@@ -46,7 +51,7 @@ The API key comes from the variable `ModelServerSpec.api_key_env` names (default
 key, so set a real one. Needs `rag-wright[modal]` and a Modal login. One app per server name: different names give
 different servers (and URLs), for example one per tenant.
 
-**The exact working deploy (this succeeded):**
+**The engine developer's route: the raw deploy (this succeeded):**
 ```
 VLLM_API_KEY=<a long random string> MODAL_IMAGE_BUILDER_VERSION=2025.06 \
   APP_NAME=rw-qwen3-modal MODEL=Qwen/Qwen3.8-27B-FP8 KV_CACHE_DTYPE=fp8 \
@@ -58,11 +63,10 @@ VLLM_API_KEY=<a long random string> MODAL_IMAGE_BUILDER_VERSION=2025.06 \
 The script refuses to deploy without `VLLM_API_KEY`: Modal's web endpoint is public and unauthenticated, so vLLM's
 `--api-key` is the only gate on a billing A100, and there is no default key. Then wire the engine: `.env`
 `VLLM_BASE_URL=https://<workspace>--rw-qwen3-modal-serve.modal.run/v1` and the same `VLLM_API_KEY`; the profile
-`qwen3.8-27b-modal` routes there (the seam also refuses an unset key). **Stop billing when done:**
-`uv run --no-sync modal app stop rw-qwen3-modal --yes` (A100 is expensive).
+`qwen3.8-27b-modal` routes there (the seam also refuses an unset key). When to stop it: "Cold start & cost" below.
 
 **The engine does NOT reach Modal by default.** Every role's default is `_PRODUCT_LLM = "qwen3.8-27b-modal-or"`,
-the same model routed through OpenRouter (the Modal cold start is unsolved). To send calls to the Modal server, set
+the same model routed through OpenRouter (a Modal cold start can still take minutes when Modal builds a snapshot). To send calls to the Modal server, set
 `RAG_MODEL_ALL=qwen3.8-27b-modal` (all roles), a per-role `RAG_MODEL_<ROLE>`, or `EngineConfig.models`. For bulk
 closed-vocab labeling, the reasoning-off profile `qwen3.8-27b-modal-nothink` exists (opt-in; see the reasoning
 caveat below). `VISION_OCR` now defaults to the product LLM too, but image inputs through the Modal vLLM server are
@@ -98,25 +102,47 @@ the OFFICIAL prebuilt vLLM image** rather than pip-building vLLM:
    reasoning calls take ~50–60 s; the seam's default 60 s timeout kills the ones just over the line → **retry pileup →
    requests pile past `max_inputs` → instant `APIConnectionError` (0.0 s) cascade**. FIX: the seam reads
    `RAG_STRUCTURED_TIMEOUT_S` (default 60) — raise it (e.g. `150`) for reasoning-ON bulk work. Server logs confirm the
-   calls actually complete 200 OK (~50 s); it was the client giving up.
+   calls actually complete 200 OK (~50 s); it was the client giving up. The total per call (all tries) follows it;
+   `RAG_MODEL_DEADLINE_S` overrides that (ADR-0133). Reasoning loops end at the profile's 8,192-token structured cap,
+   not at the deadline; keep temperature 0 (Qwen's recommended sampling did not remove them).
 3. **Pick client concurrency empirically, below the ceiling.** 16 was stable; 20 (== `max-num-seqs`) sat at the exact
    ceiling and risked queueing. `Running: N, Waiting: 0` + zero cancellations = healthy.
 4. **Reasoning ON vs OFF for bulk classification:** reasoning-OFF is ~30× faster but **changes the labels**
    (over-keeps / different distribution) — do NOT swap it in for work that must match the reasoning-teacher's gold
    (e.g. distillation silver). Keep reasoning ON when fidelity to the gold teacher matters; accept the cost.
+5. **Never expose vLLM's port directly (PS-20).** Sleep mode needs `VLLM_SERVER_DEV_MODE=1`, which adds
+   unauthenticated `/sleep`, `/wake_up` and `/collective_rpc` routes (`--api-key` guards only `/v1`, `/v2`,
+   `/inference`, `/cohere`). The script binds vLLM to 127.0.0.1:8001 and serves `rw_gate.py` on the public port,
+   forwarding only `/v1/*` and `/health`; keep `rw_gate.py` next to the script.
 
 ## Cold start & cost
-- Cold start ~440–480 s (CPU-bound engine init + a 51-graph CUDA-graph capture + first-time weight download), NOT
-  weight I/O — no load trick alone fixes it. GPU-snapshot / sleep-mode is a DEFERRED todo (the one attempt failed
-  because it was CPU-only; GPU retry not done). Run long deploys detached; poll `/health` for readiness.
-- **A100 is billed while up — `modal app stop <app> --yes` the moment you're done.** Verify with `modal app list`
-  (state `stopped`, 0 tasks) + endpoint 404.
-- **Do NOT warm the GPU before the consumer is ready, and do NOT trust auto-scaledown to protect billing** (burned
-  2026-10-04: a smoke-warmed A100 sat idle ~20 min while unrelated work ran, because "scaledown_window will handle
-  it" was assumed — it did not, fast enough). RULE: build the bulk/label/eval consumer FIRST (while the app is
-  stopped/cold), THEN warm → smoke → run → **`modal app stop` immediately after**, in one continuous go. If you warm
-  only to prove stand-up and the consumer is not next, STOP it right after the smoke. Treat the explicit stop as
-  mandatory, not the scaledown as sufficient; verify stopped with `modal app list`.
+- **The server is a class restored from a Modal GPU memory snapshot (PS-20, ADR-0132).** A boot without a snapshot
+  starts vLLM with `--enable-sleep-mode`, warms it, sleeps it at level 1 (weights to host RAM, KV discarded) and
+  Modal snapshots the container; a restore only wakes vLLM (~3.5 s) and is ready ~13 s after its container starts.
+  Modal builds the first few snapshots after a deploy (4-9 min boots; 3-4 in our runs) and now and then another
+  (a new host, a preemption): budget for an occasional slow cold start. Redeploying keeps the snapshots; **`modal app
+  stop` discards them**. Snapshots need TP=1 (the script boots without one at TP>1).
+- Without a snapshot a cold start is ~2-4 min on Config B (114-245 s measured): about 30 s of imports in EACH of
+  vLLM's two processes (transformers' `importlib.metadata.packages_distributions()` stats every installed file),
+  a weight read of 12-154 s from the Volume (9P), then compile-cache load + CUDA-graph capture. Measured and NOT
+  worth adopting (ADR-0109 addendum): `--load-format runai_streamer`, precompiled `.pyc`, `HF_HUB_OFFLINE=1` (it
+  changes vLLM's compile-cache key every container, so each start recompiles). Poll `/health` for readiness.
+- **When to stop it -- one rule (EV-6; it used to contradict the line above).** The A100 is billed only while a
+  container runs; the server scales to zero after 10 idle minutes (`scaledown_window=600`, no warm minimum) and then
+  costs nothing and KEEPS its snapshots, so the next request is back in ~13 s. `modal app stop` saves at most those
+  10 minutes and makes the next deploy pay a 9-15 min snapshot-building boot (2026-10-12: a stop after a failed smoke
+  run, then a redeploy, cost a 911 s boot).
+  1. **Deploy once per server name**; reuse the name (a new name builds new snapshots).
+  2. **Build the consumer first** (the bulk / label / eval script, its smoke test, its progress logging) while no
+     container runs, then warm, smoke and run in one go. Do not warm a GPU before its consumer is ready (2026-10-04:
+     a smoke-warmed A100 sat idle ~20 min while unrelated work ran).
+  3. **Between runs, and when you are done, let it scale to zero.** Never stop it between runs; a script's error
+     path must not stop it either when a fix-and-rerun is likely.
+  4. **Check it reached zero**: `modal app list` shows the app `deployed` with 0 tasks about 10 minutes after the
+     last call. If tasks remain past the window, find out why (that is the 2026-10-04 burn) and stop it.
+  5. **Stop (`modal app stop <app> --yes`, or `stop_model_server`) only to retire a server**: you will not use it
+     for a long time and do not want its endpoint up, or you are replacing its deployment. Expect the next deploy to
+     rebuild the snapshots.
 
 ## The meta-lesson
 Config + recipe live in **ADR-0110 + the script + this skill**. Before any Qwen/Modal work: read them, don't

rag_wright-0.8.0/.claude/skills/qwen-vllm-modal/scripts/modal_qwen3_vllm_server.py

@@ -0,0 +1,164 @@
+"""Qwen3.8-27B as an OpenAI-compatible vLLM web server on Modal A100-80GB. Two uses, one script:
+
+(1) QUANTIZATION ACCURACY EVAL (ADR-0110): serve one config under a stable served-name ("qwen3-eval").
+  MODEL=Qwen/Qwen3.8-27B     KV_CACHE_DTYPE=auto uv run --no-sync modal deploy scripts/modal_qwen3_vllm_server.py
+  MODEL=Qwen/Qwen3.8-27B-FP8 KV_CACHE_DTYPE=fp8  uv run --no-sync modal deploy scripts/modal_qwen3_vllm_server.py
+
+(2) 0048 part-2 JOINT RUN -- the UNIFIED endpoint both the product's agent-llm (tool-calling) and the engine
+    drive: Config B (FP8), ALL capabilities on (tool calls + reasoning + guided decoding), served under the
+    name the engine profile `qwen3.8-27b-modal` expects ("Qwen/Qwen3.8-27B"), sized for the measured ~16 peak
+    concurrency of unbounded reasoning outputs (2 GPUs to keep decode from saturating).
+  APP_NAME=rw-qwen3-modal MODEL=Qwen/Qwen3.8-27B-FP8 KV_CACHE_DTYPE=fp8 \
+    SERVED_NAME=Qwen/Qwen3.8-27B TOOL_PARSER=hermes REASONING_PARSER=qwen3 \
+    GPU=A100-80GB:2 TP=2 MAX_NUM_SEQS=32 \
+    uv run --no-sync modal deploy scripts/modal_qwen3_vllm_server.py
+  uv run --no-sync modal app stop <APP_NAME>   # tear down (stop billing)
+
+URL: https://<workspace>--<APP_NAME>-serve.modal.run  (+ /v1 for the OpenAI base). Weights + the serve-path
+torch.compile cache reuse the persistent Volumes (fast warm cold-start where a matching cache exists).
+
+Cold start (PS-20, ADR-0132): at TP=1 the server is restored from a Modal GPU memory snapshot. A boot without one
+starts vLLM, warms it, puts it to sleep (weights parked in host RAM) and Modal snapshots the container; a restore
+only wakes vLLM (~13 s from container start). Modal builds the first 2-3 snapshots per GPU type, and now and then
+another, on ordinary boots; stopping the app discards them. vLLM listens on 127.0.0.1 only: sleep mode needs its
+unauthenticated dev routes, so the public port is `rw_gate.py`, which forwards only /v1/* and /health.
+"""
+
+import os
+import subprocess
+import time
+import urllib.request
+from pathlib import Path
+
+import modal
+
+MODEL = os.environ.get("MODEL", "Qwen/Qwen3.8-27B")          # ...-FP8 for the quantized-weights config
+KV_DTYPE = os.environ.get("KV_CACHE_DTYPE", "auto")          # "auto" (bf16 KV, the reference) | "fp8"
+GPU = os.environ.get("GPU", "A100-80GB:1")
+TP = int(os.environ.get("TP", "1"))
+MAX_LEN = int(os.environ.get("MAX_LEN", "16384"))            # the real workload's context (9K+ prompts)
+GPU_UTIL = os.environ.get("GPU_UTIL", "0.95")
+MAX_NUM_SEQS = os.environ.get("MAX_NUM_SEQS", "256")         # concurrency ceiling; size to the eval's ~16 peak
+SERVED = os.environ.get("SERVED_NAME", "qwen3-eval")         # served-model-name clients ask for (profile match)
+TOOL_PARSER = os.environ.get("TOOL_PARSER", "")             # "" off | "hermes" -> tool calling (the product agent)
+REASONING_PARSER = os.environ.get("REASONING_PARSER", "")   # "" off | "qwen3" -> reasoning coexists with a tool call
+APP_NAME = os.environ.get("APP_NAME", "rw-qwen3-vllm-eval")  # distinct app (=distinct URL) per purpose
+API_KEY = os.environ.get("VLLM_API_KEY")  # vLLM's --api-key: the only gate on the server's public URL
+if not API_KEY:  # no default key (PS-21): a default would be public, and would gate a billing GPU
+    raise SystemExit("set VLLM_API_KEY to the server's API key (a long random string) before deploying")
+SNAPSHOT = TP == 1  # Modal: GPU memory snapshots do not work across several GPUs; TP>1 boots without one
+VLLM_PORT, PUBLIC_PORT = 8001, 8000  # vLLM on loopback; the gate on the public port
+GATE = "/root/rw_gate.py"
+
+HF_CACHE = "/root/.cache/huggingface"
+VLLM_CACHE = "/root/.cache/vllm"
+hf_vol = modal.Volume.from_name("rw-hf-cache", create_if_missing=True)
+vllm_cache_vol = modal.Volume.from_name("rw-vllm-serve-compile-cache", create_if_missing=True)
+
+app = modal.App(APP_NAME)
+vllm_image = (
+    # Base off the OFFICIAL prebuilt vLLM image: vllm + torch + xformers + flashinfer already installed as matched
+    # wheels, so a fresh build on a clean account compiles NOTHING (the xformers source-build / resolver failures
+    # only happen when pip/uv try to assemble that stack themselves). Latest/unpinned by design — `:latest` tracks
+    # newest vLLM, which is all we need since we only consume it. add_python omitted (the image ships Python).
+    # The vLLM image ships `python3` but no `python`; Modal's OWN pip bootstrap runs right after FROM (before any
+    # later layer) as `python -m pip …` -> exit 127. setup_dockerfile_commands runs BEFORE that bootstrap, so the
+    # symlink is in place when Modal bootstraps.
+    modal.Image.from_registry(
+        "vllm/vllm-openai:latest",
+        setup_dockerfile_commands=["RUN ln -sf $(command -v python3) /usr/local/bin/python"],
+    )
+    .entrypoint([])  # the image's ENTRYPOINT (the api_server) would hijack Modal's runtime; clear it
+    .pip_install("hf_transfer")  # enable fast HF weight downloads (HF_HUB_ENABLE_HF_TRANSFER below)
+    # Bake the runtime knobs into the image env: a deployed function reads the CONTAINER's env, not the deploying
+    # shell's, so an override must live here to reach the container (same lesson as the bench script). Xet disabled
+    # so a fresh weight download's open log handle can't block a Volume commit.
+    .env({"HF_HUB_ENABLE_HF_TRANSFER": "1", "HF_HUB_DISABLE_XET": "1", "HF_HOME": HF_CACHE,
+          "MODEL": MODEL, "KV_CACHE_DTYPE": KV_DTYPE, "TP": str(TP), "MAX_LEN": str(MAX_LEN),
+          "GPU_UTIL": GPU_UTIL, "MAX_NUM_SEQS": str(MAX_NUM_SEQS), "SERVED_NAME": SERVED,
+          "TOOL_PARSER": TOOL_PARSER, "REASONING_PARSER": REASONING_PARSER, "VLLM_API_KEY": API_KEY,
+          # Sleep mode's /sleep and /wake_up exist only in dev mode; they stay on loopback (see the gate).
+          "VLLM_SERVER_DEV_MODE": "1", "RW_VLLM_PORT": str(VLLM_PORT), "RW_PUBLIC_PORT": str(PUBLIC_PORT),
+          "TORCHINDUCTOR_COMPILE_THREADS": "1"})  # Modal: torch.compile can otherwise break snapshot creation
+    .add_local_file(Path(__file__).resolve().parent / "rw_gate.py", GATE)  # resolve(): scripts/ holds a symlink
+)
+
+
+def _vllm(path: str, method: str = "GET", body: bytes | None = None, timeout: float = 600) -> int:
+    request = urllib.request.Request(f"http://127.0.0.1:{VLLM_PORT}{path}", data=body, method=method, headers={
+        "Authorization": f"Bearer {API_KEY}", "Content-Type": "application/json"})
+    with urllib.request.urlopen(request, timeout=timeout) as response:
+        return response.status
+
+
+def _wait_healthy(proc: subprocess.Popen) -> None:
+    while True:
+        if proc.poll() is not None:
+            raise RuntimeError(f"vLLM exited with {proc.returncode}")
+        try:
+            if _vllm("/health", timeout=5) == 200:
+                return
+        except OSError:
+            pass
+        time.sleep(1)
+
+
+@app.cls(image=vllm_image, gpu=GPU, volumes={HF_CACHE: hf_vol, VLLM_CACHE: vllm_cache_vol},
+         timeout=3600, max_containers=1, scaledown_window=600,
+         memory=65536,  # host RAM for the sleeping weights (~29 GiB for Config B), which the snapshot captures
+         enable_memory_snapshot=SNAPSHOT, experimental_options={"enable_gpu_snapshot": SNAPSHOT})
+# Without this Modal feeds the container ONE HTTP request at a time (Running:1 + "cancellation signal" floods,
+# /health blocked) — vLLM's continuous batching never gets fed. Let many requests hit the one container; vLLM
+# batches them internally up to --max-num-seqs.
+@modal.concurrent(max_inputs=int(MAX_NUM_SEQS))
+class QwenServer:
+    """vLLM's OpenAI-compatible server for the configured Qwen config. Reuses the persisted weights + serve-path
+    torch.compile cache; compiled (production-faithful), optional FP8 KV, optional tool-calling + reasoning parsers so
+    ONE endpoint serves free-text, guided/JSON, tool calls, and reasoning together."""
+
+    @modal.enter(snap=True)
+    def start(self) -> None:
+        """Start vLLM, warm it, and put it to sleep: the snapshot is taken after this (when SNAPSHOT is on)."""
+        hf_vol.reload()
+        vllm_cache_vol.reload()
+        args = [
+            "vllm", "serve", MODEL, "--host", "127.0.0.1", "--port", str(VLLM_PORT), "--served-model-name", SERVED,
+            "--tensor-parallel-size", str(TP), "--dtype", "bfloat16", "--max-model-len", str(MAX_LEN),
+            "--gpu-memory-utilization", GPU_UTIL, "--max-num-seqs", str(MAX_NUM_SEQS),
+            "--api-key", API_KEY, "--trust-remote-code", "--enable-sleep-mode",
+        ]
+        if KV_DTYPE != "auto":
+            args += ["--kv-cache-dtype", KV_DTYPE]
+        if TOOL_PARSER:  # the product's deep-agent orchestrator is a tool-caller (RuleWright Q1/Q3)
+            args += ["--enable-auto-tool-choice", "--tool-call-parser", TOOL_PARSER]
+        if REASONING_PARSER:  # so reasoning coexists with a forced tool/schema instead of colliding
+            args += ["--reasoning-parser", REASONING_PARSER]
+        print(f"[qwen] serving {MODEL} as '{SERVED}' | TP={TP} max_len={MAX_LEN} util={GPU_UTIL} kv={KV_DTYPE} "
+              f"max_seqs={MAX_NUM_SEQS} tool={TOOL_PARSER or '-'} reasoning={REASONING_PARSER or '-'} "
+              f"snapshot={SNAPSHOT}", flush=True)
+        self.vllm = subprocess.Popen(args)
+        _wait_healthy(self.vllm)
+        warm = (b'{"model": "%s", "messages": [{"role": "user", "content": "Say OK."}], "max_tokens": 8, '
+                b'"chat_template_kwargs": {"enable_thinking": false}}' % SERVED.encode())
+        for _ in range(2):
+            _vllm("/v1/chat/completions", "POST", warm)
+        _vllm("/sleep?level=1", "POST", b"")  # level 1: weights to host RAM (in the snapshot), KV cache discarded
+        print("[qwen] warmed and asleep; snapshot next", flush=True)
+
+    @modal.enter(snap=False)
+    def wake(self) -> None:
+        """After a restore (or straight after `start` without one): wake vLLM, then open the public port."""
+        _vllm("/wake_up", "POST", b"")
+        _wait_healthy(self.vllm)
+        self.gate = subprocess.Popen(["python", GATE])
+        print("[qwen] awake; gate open", flush=True)
+
+    @modal.web_server(port=PUBLIC_PORT, startup_timeout=1200, label=f"{APP_NAME}-serve")
+    def serve(self) -> None:
+        """The public URL: the gate, which forwards /v1/* and /health to vLLM."""
+
+    @modal.exit()
+    def stop(self) -> None:
+        for proc in (getattr(self, "gate", None), getattr(self, "vllm", None)):
+            if proc:
+                proc.terminate()

rag_wright-0.8.0/.claude/skills/qwen-vllm-modal/scripts/rw_gate.py

@@ -0,0 +1,58 @@
+"""The model server's public front (PS-20): forwards only the OpenAI API (`/v1/...`) and `/health` to vLLM.
+
+vLLM's `--api-key` guards only `/v1`, `/v2`, `/inference` and `/cohere`. Sleep mode, which the GPU snapshot needs,
+requires `VLLM_SERVER_DEV_MODE=1`, and that adds unauthenticated `/sleep`, `/wake_up` and `/collective_rpc` routes. So
+vLLM listens on 127.0.0.1 only and this gate is what the public URL reaches. It runs inside the server container
+(`modal_qwen3_vllm_server.py` copies it into the image); responses stream through as they arrive.
+"""
+import os
+
+UPSTREAM = f"http://127.0.0.1:{os.environ.get('RW_VLLM_PORT', '8001')}"
+_HOP_BY_HOP = {"connection", "keep-alive", "transfer-encoding", "upgrade", "proxy-connection", "te", "trailer",
+               "host", "content-length"}
+
+
+def allowed(path: str) -> bool:
+    """The routes the public URL may reach: the OpenAI API and the health check."""
+    return path == "/health" or path.startswith("/v1/")
+
+
+async def _forward(request):
+    from aiohttp import web
+
+    if not allowed(request.path):
+        return web.json_response({"error": "Not Found"}, status=404)
+    headers = {k: v for k, v in request.headers.items() if k.lower() not in _HOP_BY_HOP}
+    async with request.app["session"].request(request.method, UPSTREAM + request.path_qs, headers=headers,
+                                              data=await request.read()) as upstream:
+        response = web.StreamResponse(status=upstream.status, headers={
+            k: v for k, v in upstream.headers.items() if k.lower() not in _HOP_BY_HOP})
+        await response.prepare(request)
+        async for chunk in upstream.content.iter_any():  # streamed completions pass through as they arrive
+            await response.write(chunk)
+        await response.write_eof()
+        return response
+
+
+async def _app():
+    from aiohttp import ClientSession, ClientTimeout, TCPConnector, web
+
+    app = web.Application(client_max_size=64 * 1024 * 1024)
+
+    async def open_session(app):
+        app["session"] = ClientSession(connector=TCPConnector(limit=0), timeout=ClientTimeout(total=None))
+
+    async def close_session(app):
+        await app["session"].close()
+
+    app.on_startup.append(open_session)
+    app.on_cleanup.append(close_session)
+    app.router.add_route("*", "/{tail:.*}", _forward)
+    return app
+
+
+if __name__ == "__main__":
+    from aiohttp import web
+
+    web.run_app(_app(), host=os.environ.get("RW_GATE_HOST", "0.0.0.0"),
+                port=int(os.environ.get("RW_PUBLIC_PORT", "8000")), print=None)

{rag_wright-0.6.0/src/rag_wright/.agents → rag_wright-0.8.0/.claude}/skills/using-the-rag-wright-engine/SKILL.md

@@ -157,7 +157,24 @@ unset. Check both before trusting a run's numbers.
 Read the engine's **reference pack** (`load_reference_pack()`) as a worked template — but the contract/compliance
 domain is only an example; your pack and capabilities are your own.
 
-## 5. Keep the engine current
+## 5. Evals behind your product's backend
+Domain experts own the gold; your UI lets them upload it, run it and compare runs. The engine gives you the pieces
+(the `creating-evals` skill has the method and a complete example):
+- **Upload a gold set**: `aload_gold_set(IngestSource(data=uploaded_bytes, name=filename), GoldMapping(...),
+  name=..., version=...)` -- CSV / XLSX / JSONL as the expert wrote it; a `GoldImportError` says where the file is
+  wrong, which you show as is. Client data stays `local_only` (the default).
+- **Run it as a background job**, never inside the request: start `arun_eval(gold, task, scorers, repeats=...,
+  progress=...)` as a task you keep a reference to, return a job id at once, and let the UI poll. `progress` receives
+  `[eval] i/N <item> r<k> ok|error` lines; `evaluate_ingestion`'s async form `aevaluate_ingestion` works the same
+  way for ingestion structure. The synchronous `run_eval` raises inside a running event loop.
+- **Return the `EvalReport`** (`report.model_dump(mode="json")`): `passed`, each scorer's gate against its
+  thresholds and its diagnostics, errors and flips kept apart, every item's outputs.
+- **Keep run history in your store**: save each report with its gold version; `rescore_eval` scores a saved report
+  again (new thresholds, a corrected gold) without calling anything. With `langfuse=True` the runs also appear in
+  Langfuse with their links (`report.run_urls`), for teams that use it.
+- **The task is your capability through the capability layer**, so the eval measures what production runs.
+
+## 6. Keep the engine current
 
 The engine releases in batches (release-please; the engine's `docs/releasing.md`), so you never chase every
 engine PR:
@@ -185,7 +202,8 @@ mkdir -p .claude/skills && for d in "$SKILLS"/*/; do ln -sfn "${d%/}" ".claude/s
 ```
 
 Re-run it after every engine upgrade (a symlink follows the installed version, but a newly added skill needs a new
-link). The Addy-Osmani spec-driven/TDD/planning skills come from their own source, as the product template says.
+link). Link these, and any product-building skills another repo writes for product developers, into this repo's
+`.claude/skills/` only, never into the global Claude Code skills, where they would apply in every repo on the machine. The Addy-Osmani spec-driven/TDD/planning skills come from their own source, as the product template says.
 
 ## Anti-patterns
 

rag_wright-0.8.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.8.0"
+}