rag-wright 0.7.0__tar.gz → 0.8.0__tar.gz

{rag_wright-0.7.0/src/rag_wright/.agents → rag_wright-0.8.0/.claude}/skills/building-an-ingestion-capability/SKILL.md

@@ -136,6 +136,11 @@ network-bound work inside it; the engine runs up to `IngestionTuning.extract_con
 
 - **Hidden sheets** are ingested by default; `IngestSource(include_hidden_sheets=False)` skips them (listed in
   `DocumentReport.skipped_hidden_sheets`).
+- **Blank rows inside a table**: the parse cuts a sheet's table at every blank row; the engine glues a piece back
+  onto the table above when its first row is data (its number or date columns agree, or it is formatted like the
+  data rather than the header), so no record is read as a header. A text-only table with no formatting cannot be
+  decided from the file: it stays apart and `evaluate_ingestion` lists it in `warnings` (format the header row, or
+  remove the blank rows).
 - **Exact cell values**: `table_rows(parse_document(...))` returns every data row as a `TableRow` (`columns`,
   `values`, `cell(name)`), read from the parse's cell grid, whole even when chunking split the table. A per-row unit
   carries its row in `Unit.table_row`, so the extractor reads cells, not re-parsed text.

{rag_wright-0.7.0 → rag_wright-0.8.0}/.claude/skills/creating-evals/SKILL.md

@@ -6,8 +6,10 @@ description: >-
   starting a new domain (right after capabilities are defined), when adding or changing a capability, or when A/B-ing
   alternatives (e.g. a trained classifier vs a System-1 decision model). Covers gold-set design + reliability,
   per-capability-KIND metrics (extraction / classification / retrieval / graph / generation / judgment), the
-  gate-vs-diagnostic split, building the gold cheaply, an isolated executable harness, and optional Langfuse
-  Datasets/Experiments/Scores automation. An eval is the executable acceptance criterion; training data IS an eval.
+  gate-vs-diagnostic split, building the gold cheaply, and running it with the engine's eval toolkit
+  (`load_gold_set` from an expert's own files, one `score_*` per kind, `arun_eval` with repeats and errors kept apart,
+  an `EvalReport` a backend can return, optional Langfuse datasets and runs). An eval is the executable acceptance
+  criterion; training data IS an eval.
 ---
 
 # Creating evals (eval-first / TDD)
@@ -19,6 +21,13 @@ can A/B alternatives and catch regressions forever. Corollary observed repeatedl
 data you build for a classifier/decision model IS an eval** (a labeled gold set + a metric) — so building the eval
 first also gives you the data design for free.
 
+**The toolkit (on `rag_wright.api`).** `load_gold_set` turns an expert's CSV / XLSX / JSONL into a `GoldSet`;
+one scorer per capability kind (section 2) turns predictions into a `ScorerResult` (the gate, its thresholds,
+diagnostics that are never gated); `arun_eval` runs your capability over the gold set (concurrently, `[eval] i/N`
+progress, repeats, a failed call kept as an error) and returns an `EvalReport` -- JSON-ready, every item's outputs
+kept, so `rescore_eval` can score it again later without calling anything. Use it rather than hand-writing a
+harness; section 6 is a complete example.
+
 Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`) are in the engine repository: read them there or on GitHub, at the tag matching your installed engine. The `eval/` and `scripts/` files named as patterns are the engine's own evals: read them as
 worked examples; your evals live in your repo.
 
@@ -50,12 +59,30 @@ worked examples; your evals live in your repo.
 - **Separate a tuning set from a held-out set.** Label the held-out set BEFORE any prompt runs on it, tune only on
   the tuning set, and report both numbers (ADR-0122 boundary residue prompt: 94.5% held-out vs 96.2% tuning). A
   number measured only on the set you tuned on is not a result.
+- **Gold from your experts' own files.** Domain experts label in spreadsheets: `load_gold_set(path_or_upload,
+  GoldMapping(id=..., input=..., expected=..., group=...), name=..., version=...)` reads them as written -- the header
+  found under a title, blank rows skipped, values kept exactly, each item's place recorded -- and refuses anything
+  that would make the gold wrong (a missing column, a row without its id, a duplicate id) with where it is. Map
+  `group` to the source document so splits stay leakage-safe. A gold set is `local_only` by default: client data
+  never leaves the machine.
 - **Gold stays local when the corpus is restrictively licensed** (CUAD/ACORD-derived gold lives under the
   gitignored `data/eval/`); commit the builder and the scorer, never the data.
 
 ## 2. Pick the metric by capability KIND, and split GATE vs DIAGNOSTIC
 Always set ONE pass/fail **gate** (from the acceptance criterion) and report **diagnostics** alongside (never gate
-on a diagnostic).
+on a diagnostic). Each kind has its scorer on `rag_wright.api`, checked against the engine evals named below:
+
+| kind | scorer | gate |
+|---|---|---|
+| classification / typed decision | `score_classification` | the weakest class's recall@k (+ top-1 accuracy via `top1_floor`) |
+| retrieval | `score_retrieval` | mean recall@k at a relevance grade; nDCG reported |
+| extraction | `score_extraction` | recall and precision, apart |
+| judgment / verdict | `score_judgment` | precision and recall of the actionable verdict |
+| generation / QA | `score_generation` | citation recall, correct abstention, groundedness |
+| graph / relational | `score_relational` | recall over the answer set |
+| ingestion structure / unit labelling | `score_ingestion` | `evaluate_ingestion`'s checks (+ label accuracy) |
+| candidate-then-choose | `score_candidate_choice` | candidate coverage, value-level recall and precision |
+
 - **Extraction** (section → records, clauses, requirements): recall / precision / F1 of extracted items vs gold,
   reported SEPARATELY (under- vs over-extraction are different failures). Watch over-extraction on non-operative
   input (definitions) and under-extraction on long input.
@@ -84,7 +111,7 @@ on a diagnostic).
   scores mean what they say), not one accuracy number. Pattern: `eval/semantic_judge_gold.py` (ING-9: decision
   model 95.3% vs LLM 90.1% on 192 blind hand-labelled cases).
 - **Candidate-then-choose pipelines** (a generator proposes, a model picks): measure the generator's **coverage**
-  separately, because it caps recall (ING-9b residual values: candidate coverage 95%; value-level recall 0.83 /
+  separately, because it caps recall (ING-9b residual values: candidate coverage 0.93, 188/203 recomputed from the saved outputs; value-level recall 0.83 /
   precision 0.88 vs the LLM's 0.60 / 0.79. Pattern: `eval/residual_decision_gold.py`). **Count is not accuracy**:
   emitting more values is not a win without precision.
 - **Ingestion structure**: `evaluate_ingestion` (in `rag_wright.api`) is the packaged structural eval of the
@@ -111,42 +138,87 @@ on a diagnostic).
 - **Generate hard cases** (near-boundary positives/negatives) to stress the exact confusions — TRAIN-ONLY; the
   gold test stays real. (Full data playbook: the `setfit` skill Phase 1/4 + `classifier-opportunity-analysis`.)
 
-## 4. Make it an executable, isolated harness
-- **Isolate the capability under test**: inject it (a seam / `extract_override` / an injected `retrieve`) so the
-  eval measures ONE capability, not the whole pipeline. Invoke production code THROUGH the capability layer
-  (`ainvoke_subgraph`/`ainvoke_model`), never a hand-built copy. Score the **shipped request** (the exact
-  prompt/question builder production uses, e.g. `judge_request`), not a copy of it re-typed in the eval.
-- **Repeatability for non-deterministic models.** Run each case several times on identical input; report how many
-  answers flip and how far the scores sit from the threshold. Flips cluster near the threshold and usually mean
-  the QUESTION is ambiguous: fix the question, do not vote it away (Jev flipped 5/181 answers at scores 0.47-0.56;
-  temperature/seed did not help, majority voting barely helped, rewriting the ambiguous question did. The ADR-0122
-  boundary residue prompt went from 94-96% with flips to 460/461, zero flips across 3 calls). Pattern:
-  `eval/boundary_residue_gold.py`.
+## 4. Run it: `arun_eval`
+`arun_eval(gold, task, scorers, *, repeats, concurrency, progress)` is the harness. It runs `task` (your coroutine,
+or a plain function, which runs off the event loop) on every item, `repeats` times, `concurrency` at a time, streams
+`[eval] i/N` lines to `progress`, keeps a call that raises as an ERROR in that item's slot (never dropped, never an
+answer), scores each repeat, and passes a scorer only if it passes on every repeat (its gate is the worst repeat;
+`per_repeat_gate` lists each). `run_eval` is the synchronous form; inside a web backend, await `arun_eval` as a
+background job. What is still yours:
+- **Isolate the capability under test**: the task calls ONE capability, injected (a seam / `extract_override` / an
+  injected `retrieve`), so the eval measures it, not the whole pipeline. Invoke production code THROUGH the
+  capability layer (`ainvoke_subgraph`/`ainvoke_model`), never a hand-built copy. Score the **shipped request** (the
+  exact prompt/question builder production uses, e.g. `judge_request`), not a copy of it re-typed in the eval.
+- **Repeatability for non-deterministic models.** Use `repeats` and read `report.flips`; flips cluster near the
+  threshold and usually mean the QUESTION is ambiguous: fix the question, do not vote it away (Jev flipped 5/181
+  answers at scores 0.47-0.56; temperature/seed did not help, majority voting barely helped, rewriting the ambiguous
+  question did. The ADR-0122 boundary residue prompt went from 94-96% with flips to 460/461, zero flips across 3
+  calls). Pattern: `eval/boundary_residue_gold.py`.
 - **Cache keys include the prompt and the method.** Any decision or extraction cache must key on the prompt text
   and the method (decision model vs LLM, and which model), so an eval never reuses outputs a different prompt or
   method produced (ADR-0122 ING-4d / ADR-0040 ING-9b: the residue decision cache and the clause cache).
-- **Score from a RESULT ARTIFACT (JSON), not stdout scraping** (scraping truncates and silently drops rows).
-- **Parallelize** model/LLM calls (async + semaphore) — same cost, far less wall-clock; order-preserving so it
-  stays deterministic.
-- **Env-selected** so the SAME harness runs local (dev) or on Modal (full corpus + GPU).
+- **Keep the report.** `report.model_dump_json()` holds every item's outputs; save it next to the gold version.
+  `rescore_eval(report, gold, scorers)` re-scores it with new thresholds, a new scorer or a corrected gold, free. An
+  eval kept only as aggregate numbers can never be checked again.
 - **Pre-flight paid bulk** (>~50 paid calls): print the exact count + cost and wait (`warn-before-bulk` rule).
-- Stream `X/N` progress + actively monitor any run over ~30s (never launch-and-forget).
 - **Hermetic tests never touch the network**: `tests/conftest.py` fails a test that resolves a non-local host unless
-  it carries a live marker (`model`, `store`, ...). Live evals run as `uv run python -u eval/<name>.py` (outside
-  pytest) or carry a live marker.
-
-## 5. Langfuse — optional eval automation (we already use it for tracing)
-Langfuse has a first-class eval stack we are NOT yet using (we use it only for spans/usage today): a **Dataset**
-(items = input + optional expected output) → a **Task** (your capability) run over the dataset as an **Experiment
-Run** → **Evaluators** (deterministic checks or LLM-as-judge) producing **Scores** (numeric/categorical/boolean),
-all linked to traces. Reach for it when you want **tracked, re-runnable, dashboarded** evals + regression tracking
-across capability versions; a local JSON harness is enough for a quick one-off gate. Keep the gold BUILDER + a
-committed snapshot in the repo (reproducibility); push the items to the Langfuse dataset so runs and scores land
-next to the traces you already collect. Ground the exact Dataset/Experiment API before wiring
-(https://langfuse.com/docs/evaluation/concepts, https://langfuse.com/docs/datasets/overview); confirm the installed
-SDK version's surface, not prose.
-
-## 6. The TDD loop
+  it carries a live marker (`model`, `store`, ...). Live evals run as scripts (outside pytest) or carry a live
+  marker.
+
+## 5. Langfuse -- optional: datasets, runs and scores next to the traces
+`arun_eval(..., langfuse=True)` (or `RAG_EVAL_BACKEND=langfuse`; the keys alone send nothing) runs the same eval on
+Langfuse: the gold set becomes a dataset (items upserted; items no longer in the gold set archived; the version is
+the server's latest timestamp, recorded as `report.dataset_version`), each repeat a dataset run (`report.run_urls`),
+and each scorer's gates and pass/fail are pushed as run scores. The report is the same, scored locally from the same
+outputs. With engine tracing on (`RAG_TRACE_LEVEL`), every model call the task makes nests under its item's trace,
+on both backends. What was verified against Langfuse 4.15.1 (self-hosted), and what to know:
+- **`local_only` is refused before anything is sent.** Only a self-hosted Langfuse should ever receive client data;
+  set `local_only=False` only for data that may leave the machine.
+- **Scoring runs in your process**, not in Langfuse: its code evaluators are a 2-second, standard-library sandbox
+  with no network, so they cannot run your scorers.
+- **The SDK's own `run_experiment` drops an item whose task raises, and blocks a running event loop**; `arun_eval`
+  handles both (an error is an error in its slot; the run is off the loop). Call `arun_eval`, not the SDK directly.
+- **Reading observations back**: the v2 list endpoint is Cloud-only until Langfuse v4 ships in the open-source
+  release, and the v1 endpoint times out on a trace-only query: bound every read by time.
+- Langfuse's managed LLM-as-a-judge and decision-model evaluators, and its UI prompt experiments, test a prompt in
+  Langfuse, not your pipeline: out of scope for this toolkit (a product may still use them).
+
+## 6. A complete example
+An expert's labelled workbook, your capability run through the capability layer, two repeats, the report kept and
+scored again later. `my_judge` stands for your capability's call (e.g. through `ainvoke_model`); the rest runs as
+written.
+
+```python
+import asyncio
+import functools
+from pathlib import Path
+
+from rag_wright.api import (EvalReport, GoldMapping, arun_eval, load_gold_set, rescore_eval,
+                            score_judgment)
+
+GOLD_FILE = "gold/nli_expert_labels.xlsx"  # the expert's own file
+REPORT = "eval_runs/nli_judge.json"
+
+mapping = GoldMapping(id="ID", input=["Premise", "Hypothesis"], expected="Label", group="Contract")
+gold = load_gold_set(GOLD_FILE, mapping, name="nli-judge", version="2026-10-12")
+
+
+async def judge(item):
+    return await my_judge(item.input["Premise"], item.input["Hypothesis"])
+
+
+gate = functools.partial(score_judgment, actionable="contradiction", precision=0.8, recall=0.7)
+report = asyncio.run(arun_eval(gold, judge, [gate], repeats=2, concurrency=8, progress=print))
+Path(REPORT).parent.mkdir(parents=True, exist_ok=True)
+Path(REPORT).write_text(report.model_dump_json(indent=1), encoding="utf-8")
+print(report.passed, report.scores[0].gate, "errors", report.errors, "flips", report.flips)
+
+stricter = functools.partial(score_judgment, actionable="contradiction", precision=0.9, recall=0.7)
+saved = EvalReport.model_validate_json(Path(REPORT).read_text(encoding="utf-8"))
+print("at 0.9 precision:", rescore_eval(saved, gold, [stricter]).passed)
+```
+
+## 7. The TDD loop
 1. Write the eval from the contract + a handful of real gold items → it fails (red) on the unbuilt/weak capability.
 2. Implement the minimum to pass the gate (green).
 3. A/B alternatives (rule / trained classifier / decision model / LLM) on the SAME gold; select on gate + diagnostics + cost + calibration.
@@ -158,7 +230,11 @@ SDK version's surface, not prose.
 - **Gating on a diagnostic** (e.g. nDCG) — report it, don't threshold it.
 - **No reliability check on a subjective gold** — you can't read a number off a gold whose own labels disagree.
 - **Eval not isolated** to the capability (measures the whole pipeline) — you can't attribute a regression.
-- **Scraping stdout** for metrics — write + read a JSON artifact.
+- **Scraping stdout** for metrics — keep and read the `EvalReport` JSON.
+- **A failed call scored as an answer** (a timeout read as "neutral", an error read as "abstained") — it is an error;
+  `arun_eval` keeps it apart (ADR-0133).
+- **Only aggregates kept** — an eval whose per-item outputs were not saved can never be re-scored or checked; keep
+  the report.
 - **Faking a win by shrinking the sample / picking easy items** — real, representative, honest.
 
 ## Hand-off / where this sits

{rag_wright-0.7.0 → rag_wright-0.8.0}/.claude/skills/qwen-vllm-modal/SKILL.md

@@ -16,6 +16,11 @@ ADR-0110 and the other `docs/` paths are in the engine repository: read them the
 matching your installed engine.
 This skill is the operational checklist that ties them together.
 
+**Which route.** A product deploys and reaches the server through the engine API, `adeploy_model_server` (the
+section after the locked config). The raw `modal deploy` of the bundled script further down is the engine
+developer's route, and what the API runs underneath. **When to stop it** is one rule, in "Cold start & cost": deploy
+once, let it scale to zero between uses, never stop it between runs.
+
 ## The LOCKED production config (ADR-0110) — do not re-litigate
 **Config B: `Qwen/Qwen3.8-27B-FP8` weights + `--kv-cache-dtype fp8`, on 1× A100-80GB, TP=1, `--max-model-len 16384`, `--gpu-memory-utilization 0.95`.**
 - Config B (FP8 weights) is the **locked deploy** — 52.71× concurrency @16K (ADR-0110 rounds it to 53×), ~23% higher throughput, ~31 GB weights (30.9 GB on disk, 28.9 GiB in VRAM);
@@ -38,7 +43,7 @@ from rag_wright.api import EngineConfig, ModelServerSpec, adeploy_model_server,
 server = await adeploy_model_server(ModelServerSpec(name="acme-qwen", max_num_seqs=32))  # key from VLLM_API_KEY
 config = EngineConfig(store=..., models={"general": server.model_id})   # or any model= argument
 ...
-stop_model_server("acme-qwen")                                            # stop billing
+stop_model_server("acme-qwen")  # only to retire it: stopping discards the snapshots (see "Cold start & cost")
 ```
 
 `amodel_server(spec)` reconnects to a server deployed earlier (a restart, another process) without redeploying.
@@ -46,7 +51,7 @@ The API key comes from the variable `ModelServerSpec.api_key_env` names (default
 key, so set a real one. Needs `rag-wright[modal]` and a Modal login. One app per server name: different names give
 different servers (and URLs), for example one per tenant.
 
-**The exact working deploy (this succeeded):**
+**The engine developer's route: the raw deploy (this succeeded):**
 ```
 VLLM_API_KEY=<a long random string> MODAL_IMAGE_BUILDER_VERSION=2025.06 \
   APP_NAME=rw-qwen3-modal MODEL=Qwen/Qwen3.8-27B-FP8 KV_CACHE_DTYPE=fp8 \
@@ -58,8 +63,7 @@ VLLM_API_KEY=<a long random string> MODAL_IMAGE_BUILDER_VERSION=2025.06 \
 The script refuses to deploy without `VLLM_API_KEY`: Modal's web endpoint is public and unauthenticated, so vLLM's
 `--api-key` is the only gate on a billing A100, and there is no default key. Then wire the engine: `.env`
 `VLLM_BASE_URL=https://<workspace>--rw-qwen3-modal-serve.modal.run/v1` and the same `VLLM_API_KEY`; the profile
-`qwen3.8-27b-modal` routes there (the seam also refuses an unset key). **Stop billing when done:**
-`uv run --no-sync modal app stop rw-qwen3-modal --yes` (A100 is expensive).
+`qwen3.8-27b-modal` routes there (the seam also refuses an unset key). When to stop it: "Cold start & cost" below.
 
 **The engine does NOT reach Modal by default.** Every role's default is `_PRODUCT_LLM = "qwen3.8-27b-modal-or"`,
 the same model routed through OpenRouter (a Modal cold start can still take minutes when Modal builds a snapshot). To send calls to the Modal server, set
@@ -123,14 +127,22 @@ the OFFICIAL prebuilt vLLM image** rather than pip-building vLLM:
   a weight read of 12-154 s from the Volume (9P), then compile-cache load + CUDA-graph capture. Measured and NOT
   worth adopting (ADR-0109 addendum): `--load-format runai_streamer`, precompiled `.pyc`, `HF_HUB_OFFLINE=1` (it
   changes vLLM's compile-cache key every container, so each start recompiles). Poll `/health` for readiness.
-- **A100 is billed while up — `modal app stop <app> --yes` the moment you're done.** Verify with `modal app list`
-  (state `stopped`, 0 tasks) + endpoint 404.
-- **Do NOT warm the GPU before the consumer is ready, and do NOT trust auto-scaledown to protect billing** (burned
-  2026-10-04: a smoke-warmed A100 sat idle ~20 min while unrelated work ran, because "scaledown_window will handle
-  it" was assumed — it did not, fast enough). RULE: build the bulk/label/eval consumer FIRST (while the app is
-  stopped/cold), THEN warm → smoke → run → **`modal app stop` immediately after**, in one continuous go. If you warm
-  only to prove stand-up and the consumer is not next, STOP it right after the smoke. Treat the explicit stop as
-  mandatory, not the scaledown as sufficient; verify stopped with `modal app list`.
+- **When to stop it -- one rule (EV-6; it used to contradict the line above).** The A100 is billed only while a
+  container runs; the server scales to zero after 10 idle minutes (`scaledown_window=600`, no warm minimum) and then
+  costs nothing and KEEPS its snapshots, so the next request is back in ~13 s. `modal app stop` saves at most those
+  10 minutes and makes the next deploy pay a 9-15 min snapshot-building boot (2026-10-12: a stop after a failed smoke
+  run, then a redeploy, cost a 911 s boot).
+  1. **Deploy once per server name**; reuse the name (a new name builds new snapshots).
+  2. **Build the consumer first** (the bulk / label / eval script, its smoke test, its progress logging) while no
+     container runs, then warm, smoke and run in one go. Do not warm a GPU before its consumer is ready (2026-10-04:
+     a smoke-warmed A100 sat idle ~20 min while unrelated work ran).
+  3. **Between runs, and when you are done, let it scale to zero.** Never stop it between runs; a script's error
+     path must not stop it either when a fix-and-rerun is likely.
+  4. **Check it reached zero**: `modal app list` shows the app `deployed` with 0 tasks about 10 minutes after the
+     last call. If tasks remain past the window, find out why (that is the 2026-10-04 burn) and stop it.
+  5. **Stop (`modal app stop <app> --yes`, or `stop_model_server`) only to retire a server**: you will not use it
+     for a long time and do not want its endpoint up, or you are replacing its deployment. Expect the next deploy to
+     rebuild the snapshots.
 
 ## The meta-lesson
 Config + recipe live in **ADR-0110 + the script + this skill**. Before any Qwen/Modal work: read them, don't

{rag_wright-0.7.0/src/rag_wright/.agents → rag_wright-0.8.0/.claude}/skills/using-the-rag-wright-engine/SKILL.md

@@ -157,7 +157,24 @@ unset. Check both before trusting a run's numbers.
 Read the engine's **reference pack** (`load_reference_pack()`) as a worked template — but the contract/compliance
 domain is only an example; your pack and capabilities are your own.
 
-## 5. Keep the engine current
+## 5. Evals behind your product's backend
+Domain experts own the gold; your UI lets them upload it, run it and compare runs. The engine gives you the pieces
+(the `creating-evals` skill has the method and a complete example):
+- **Upload a gold set**: `aload_gold_set(IngestSource(data=uploaded_bytes, name=filename), GoldMapping(...),
+  name=..., version=...)` -- CSV / XLSX / JSONL as the expert wrote it; a `GoldImportError` says where the file is
+  wrong, which you show as is. Client data stays `local_only` (the default).
+- **Run it as a background job**, never inside the request: start `arun_eval(gold, task, scorers, repeats=...,
+  progress=...)` as a task you keep a reference to, return a job id at once, and let the UI poll. `progress` receives
+  `[eval] i/N <item> r<k> ok|error` lines; `evaluate_ingestion`'s async form `aevaluate_ingestion` works the same
+  way for ingestion structure. The synchronous `run_eval` raises inside a running event loop.
+- **Return the `EvalReport`** (`report.model_dump(mode="json")`): `passed`, each scorer's gate against its
+  thresholds and its diagnostics, errors and flips kept apart, every item's outputs.
+- **Keep run history in your store**: save each report with its gold version; `rescore_eval` scores a saved report
+  again (new thresholds, a corrected gold) without calling anything. With `langfuse=True` the runs also appear in
+  Langfuse with their links (`report.run_urls`), for teams that use it.
+- **The task is your capability through the capability layer**, so the eval measures what production runs.
+
+## 6. Keep the engine current
 
 The engine releases in batches (release-please; the engine's `docs/releasing.md`), so you never chase every
 engine PR:
@@ -185,7 +202,8 @@ mkdir -p .claude/skills && for d in "$SKILLS"/*/; do ln -sfn "${d%/}" ".claude/s
 ```
 
 Re-run it after every engine upgrade (a symlink follows the installed version, but a newly added skill needs a new
-link). The Addy-Osmani spec-driven/TDD/planning skills come from their own source, as the product template says.
+link). Link these, and any product-building skills another repo writes for product developers, into this repo's
+`.claude/skills/` only, never into the global Claude Code skills, where they would apply in every repo on the machine. The Addy-Osmani spec-driven/TDD/planning skills come from their own source, as the product template says.
 
 ## Anti-patterns
 

rag_wright-0.8.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.8.0"
+}

{rag_wright-0.7.0 → rag_wright-0.8.0}/CHANGELOG.md

@@ -4,6 +4,30 @@ All notable changes to RAG_Wright are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.8.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.7.0...0.8.0) (2026-10-12)
+
+
+### Features
+
+* **EV-1:** aevaluate_ingestion for a web backend, with progress and a JSON-ready result ([7e6377a](https://github.com/fzaidi-dai/RAG_Wright/commit/7e6377a8994764c11f08dcf8cdb47633e6f4660c))
+* **EV-2:** gold-set and report contracts, and gold import from CSV, XLSX and JSONL ([4e72d36](https://github.com/fzaidi-dai/RAG_Wright/commit/4e72d368a24045df2a09f0fc3a221c15985cbf73))
+* **EV-3a:** classification and candidate-then-choose scorers, with parity on saved engine outputs ([977f9a8](https://github.com/fzaidi-dai/RAG_Wright/commit/977f9a8aa59b932d4cc1325bb93ffc6edf4b36a1))
+* **EV-3b:** retrieval and relational scorers, with parity on the ACORD bar and the EDGAR graph leg ([b27b86f](https://github.com/fzaidi-dai/RAG_Wright/commit/b27b86f90d79c68f173f1842b99201a73e634870))
+* **EV-3c:** extraction and judgment scorers, with parity on party extraction, ContractNLI and the semantic judge ([f1fc319](https://github.com/fzaidi-dai/RAG_Wright/commit/f1fc31973dd1d65d20d07578d7d41d6b6e91565f))
+* **EV-3d:** generation and ingestion scorers; all eight scorer kinds done ([b082938](https://github.com/fzaidi-dai/RAG_Wright/commit/b082938db3bc42d81584840d63bc79f0a0ec1d65))
+* **EV-4:** the eval runner (arun_eval, run_eval, rescore_eval) ([a1c0512](https://github.com/fzaidi-dai/RAG_Wright/commit/a1c0512b24fdef245869c854494b42ab8040e5b3))
+* **EV-5:** the optional Langfuse backend; model calls nest under their eval item ([fbbc729](https://github.com/fzaidi-dai/RAG_Wright/commit/fbbc72911733392837bae52ac105ab80d2a922a2))
+* **PS-27:** a missing database is created visibly, or refused with create=False ([2e404f2](https://github.com/fzaidi-dai/RAG_Wright/commit/2e404f2bd44bacf278c2314e6ccf412e72b336c9))
+* **PS-31:** rag_wright.api.pdf_page_count, a locked PDF page count for products ([bf83f57](https://github.com/fzaidi-dai/RAG_Wright/commit/bf83f57905f8c9480c2f7890633b7ba3e474cd09))
+
+
+### Bug Fixes
+
+* **PS-24:** async model calls work on every event loop in a process (ADR-0135) ([ebd72b2](https://github.com/fzaidi-dai/RAG_Wright/commit/ebd72b2b8b962e20dadb48158d2fb429430038e3))
+* **PS-26:** spreadsheet tables are glued back across blank rows, so no record is read as a header (ADR-0136) ([d0f31ac](https://github.com/fzaidi-dai/RAG_Wright/commit/d0f31ac7628a9aabd079630f18fc86a8da617cfb))
+* **PS-28:** a generated answer states the evidence's support, and "none" is an abstention (ADR-0137) ([cb98642](https://github.com/fzaidi-dai/RAG_Wright/commit/cb9864236d076aea7bf31e09cab18a8eb4575dc2))
+* **PS-29:** PDF parses run one at a time and the engine reaches pdfium through one locked door (ADR-0138) ([f7a17ad](https://github.com/fzaidi-dai/RAG_Wright/commit/f7a17ad7645d0bd269f955e51059a431ce5bad17))
+
 ## [0.7.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.6.0...0.7.0) (2026-10-11)
 
 

{rag_wright-0.7.0 → rag_wright-0.8.0}/CLAUDE.md

@@ -55,10 +55,19 @@ The failure mode to avoid, which has happened before: a command fails with "modu
 
 ## Start of every session
 
-> **CURRENT STATE (2026-10-11): the public-surface workstream is COMPLETE and released as `rag-wright` 0.6.0** (ledger
+> **CURRENT STATE (2026-10-12): the evals workstream is COMPLETE; `rag-wright` 0.8.0 is ready to release** (ledger
+> `docs/specs/evals/plan.md`, EV-0..EV-6; EV-7 optional): gold sets from experts' files, a scorer per capability kind,
+> the runner `arun_eval` with an optional Langfuse backend, `EvalReport`, `aevaluate_ingestion`; the reference is
+> `docs/evals.md`. With it in 0.8.0: PS-24 (async calls on every event loop, ADR-0135), PS-25 (the release PR bumps
+> `uv.lock`), PS-26 (spreadsheet tables glued across blank rows, ADR-0136), PS-27 (a missing database created
+> visibly, or refused with `create=False`), PS-28 (answers state their support; "none" abstains, ADR-0137), PS-29 and
+> PS-31 (one PDF parse at a time and one locked door to pdfium, `pdf_page_count`; RuleWright issue 0052, ADR-0138).
+> RuleWright's guide: `docs/specs/public-surface/rulewright-migration-0.8.0.md`. Open: PS-30 (a process-pool parse,
+> later). Before it:
+> **The public-surface workstream is COMPLETE and released as `rag-wright` 0.7.0** (ledger
 > `docs/specs/public-surface/plan.md`, ADR-0128: two public tiers, `rag_wright.api` for a product's seam and
 > `rag_wright.pack_sdk` for a pack's code, enforced by import contracts and a domain-vocabulary guard). Next:
-> RuleWright moves from `docs/specs/public-surface/rulewright-migration-0.3.0.md` then `rulewright-migration-0.4.0.md`, `-0.5.0.md` and `-0.6.0.md`; PS-R4 (the unit-labelling eval) is done, unreleased. Before it,
+> RuleWright moves from `docs/specs/public-surface/rulewright-migration-0.3.0.md` then `rulewright-migration-0.4.0.md`, `-0.5.0.md`, `-0.6.0.md` and `-0.7.0.md`; PS-R4 (the unit-labelling eval) is done, unreleased. 0.7.0 adds PS-20 (the model server cold-starts from a GPU snapshot, ADR-0132) and PS-22 (the model-call deadline follows the timeout; a token cap on thinking models' structured calls, ADR-0133). Before it,
 > the **ingestion-hooks** workstream (ADR-0124) is
 > COMPLETE and released as **`rag-wright` 0.2.0** (PyPI + GitHub tag/release `0.2.0`; release process
 > `docs/releasing.md`): the generic `build_ingestion` with hooks and domain-neutral defaults, the neutral default
@@ -188,6 +197,7 @@ Commits are how the memory above becomes durable. Git history is the parallel re
 - `building-an-ingestion-capability` (`.claude/skills/building-an-ingestion-capability/SKILL.md`), when a domain needs its documents ingested with `build_ingestion`: deciding the unit, the extractor and optional hooks, the pack record types, tuning with `evaluate_ingestion`, spreadsheets and embedded files (its example is executed by `tests/skills/test_ingestion_skill_example.py`).
 - `using-the-rag-wright-engine`, the CONSUMER-side playbook (for a product repo building on the engine) — kept here, version-matched to the engine; a product pulls it into its own `.claude/skills/`.
 - The engine-authored skills above live once in `src/rag_wright/.agents/skills/` and ship in the wheel (PS-10); this repo's `.claude/skills/` entries are symlinks to them, so edit the files there. A product links them from its installed engine (`docs/installation.md`). `tests/arch/test_shipped_skills.py` keeps them packaged, linked and readable from a product repo.
+- Skill placement (ADR-0134): a skill written for product developers goes in `src/rag_wright/.agents/skills/` and is linked into a repo's `.claude/skills/` only, never into the global `~/.claude/skills` (that is for the developer's own way-of-working skills); a skill only for working on this repo is a real folder in `.claude/skills/` and is not shipped.
 - `Graphify`, for the `framework` and `project` graphs.
 - The LangChain docs MCP server (`https://docs.langchain.com/mcp`), for understanding LangGraph and Deep Agents concepts. It explains; it does not confirm. Grounding still resolves against the code graph.
 

{rag_wright-0.7.0 → rag_wright-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rag-wright
-Version: 0.7.0
+Version: 0.8.0
 Summary: A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime.
 Project-URL: Homepage, https://github.com/fzaidi-dai/RAG_Wright
 Project-URL: Repository, https://github.com/fzaidi-dai/RAG_Wright
@@ -34,6 +34,7 @@ Requires-Dist: langgraph>=1.2.11
 Requires-Dist: laya>=0.3.22
 Requires-Dist: mcp>=1.28.1
 Requires-Dist: olefile>=0.47
+Requires-Dist: openpyxl>=3.1
 Requires-Dist: pillow>=12.3.0
 Requires-Dist: pydantic>=2.0
 Requires-Dist: pypdfium2>=5.11
@@ -112,6 +113,8 @@ The whole doc set is under [`docs/`](docs/). Pick your path:
 - **Building a product on the engine?** The [domain-adaptation guide](docs/domain-adaptation/) walks the whole
   sequence (author a `.ttl` pack → build & register capabilities → eval-first → ingest → entity resolution → the
   product seam); a coding agent should drive it with the `using-the-rag-wright-engine` skill.
+- **Measuring a capability?** [Evals](docs/evals.md): gold sets from your experts' files, a scorer per capability
+  kind, a runner with repeats, a report your backend can return, optional Langfuse runs.
 - **Going deep?** [Architecture](docs/architecture.md), the generated [API reference](docs/api/), the
   [Reference pack](docs/reference-pack.md), and the [ADR index](docs/adr/README.md).
 - **Releases?** See the [CHANGELOG](CHANGELOG.md) and [how releases work](docs/releasing.md) (batched via

{rag_wright-0.7.0 → rag_wright-0.8.0}/README.md

@@ -57,6 +57,8 @@ The whole doc set is under [`docs/`](docs/). Pick your path:
 - **Building a product on the engine?** The [domain-adaptation guide](docs/domain-adaptation/) walks the whole
   sequence (author a `.ttl` pack → build & register capabilities → eval-first → ingest → entity resolution → the
   product seam); a coding agent should drive it with the `using-the-rag-wright-engine` skill.
+- **Measuring a capability?** [Evals](docs/evals.md): gold sets from your experts' files, a scorer per capability
+  kind, a runner with repeats, a report your backend can return, optional Langfuse runs.
 - **Going deep?** [Architecture](docs/architecture.md), the generated [API reference](docs/api/), the
   [Reference pack](docs/reference-pack.md), and the [ADR index](docs/adr/README.md).
 - **Releases?** See the [CHANGELOG](CHANGELOG.md) and [how releases work](docs/releasing.md) (batched via

rag_wright-0.8.0/docs/adr/0134-skills-for-product-developers-are-linked-per-project-never-globally.md

@@ -0,0 +1,39 @@
+# ADR-0134: Skills for product developers are linked into each product repo, never installed globally
+
+**Status:** accepted · **Date:** 2026-10-12 · **Related:** PS-10 (the engine's skills ship in the wheel),
+ADR-0052 (engine/product split)
+
+## Context
+
+Two kinds of repo now write skills for people building a product on the engine. The engine writes skills about using
+the engine (`using-the-rag-wright-engine`, `creating-evals`, ...). A product may write generic product-building
+skills (UI, backend, tenancy, workspaces, the agent harness) that a later product can reuse. A skill linked into
+`~/.claude/skills` applies in every repo on the machine: an agent working on the engine, or on an unrelated project, then sees product-building advice that is not meant for it, and a globally linked skill
+follows one checkout rather than the version a given product depends on. The product-starter templates said "link
+the shared generic skills from their source" without saying where to link them.
+
+## Decision
+
+Skills live in one of three places, by who they are for:
+
+1. **The developer's own way of working** (TDD, planning, debugging, ADRs) may be installed in `~/.claude/skills`.
+   Nothing from the engine or a product repo is linked there.
+2. **Skills for working on one repo only** are real folders in that repo's `.claude/skills/` and are never shipped.
+3. **Skills for product developers** live in a source folder in the repo that writes them, and each product links them
+   into its own `.claude/skills/` (gitignored), never globally:
+   - the engine's: `src/rag_wright/.agents/skills/`, shipped in the wheel (PS-10). A product links them from its
+     installed engine (the copy that matches the version it runs) or, while co-developing, from an engine checkout;
+     both are links into the product's own `.claude/skills/`;
+   - a product's own: wherever that product keeps them, outside the engine's concern.
+
+The engine's docs, templates and skills that show how to link skills only ever link into the product repo's
+`.claude/skills/`; `tests/arch/test_shipped_skills.py` fails on a command in them that links into
+`~/.claude/skills`.
+
+## Consequences
+
+- An engine session sees the engine's skills through this repo's `.claude/skills/` links only; a session in an
+  unrelated repo does not see them.
+- A product re-runs its link step after an engine upgrade (as before) and after pulling a new generic skill.
+- The engine governs only its own skills, docs and templates; where a product keeps its skills is the product's
+  decision, and the engine's templates name the slot (`{{PRODUCT_DEV_SKILL_SOURCES}}`) without naming a folder.

rag_wright-0.8.0/docs/adr/0135-the-seam-gives-each-event-loop-its-own-async-http-client.md

@@ -0,0 +1,29 @@
+# ADR-0135: The model seam gives each event loop its own async HTTP client
+
+**Status:** accepted · **Date:** 2026-10-12 · **Related:** ADR-0056/0057 (the bounded retry and the async
+deadline), EV-0 finding 5 (`docs/specs/evals/plan.md`), PS-24
+
+## Context
+
+langchain_openai's `ChatOpenAI` takes its async HTTP client at construction, and by default shares one cached
+client per base URL and timeout across the whole process. httpx's pooled connections belong to the event loop that
+opened them. The seam builds runnables once and calls them later, possibly on another loop: Langfuse's
+`run_experiment` runs every experiment on a fresh loop, and a web backend has its own loop beside a worker's. EV-0
+saw the first calls of the second run fail at once with `APIConnectionError` (`Event loop is closed` underneath);
+the seam's retry recovered them, at the cost of a retry, and a burst at the start of a run could use up the
+retries.
+
+## Decision
+
+`build_model` passes `ChatOpenAI` an `httpx.AsyncClient` (`_LoopLocalAsyncClient`) that sends each request through
+a client of the running event loop. That client is built the way langchain_openai builds its default (its own
+builder and socket options), kept per loop in a weak map so it goes with the loop, and the wrapper is cached per
+base URL and timeout, so calls on one loop still share their connections. A caller's own `http_async_client` wins,
+and with `OPENAI_PROXY` set langchain_openai keeps building its proxied client.
+
+## Consequences
+
+- Async calls work on every event loop in a process, with no retry spent; synchronous calls are unchanged.
+- The seam now imports two private langchain_openai helpers (`_build_async_httpx_client`,
+  `_resolve_socket_options`), so the per-loop client matches the default exactly; a langchain_openai upgrade that
+  moves them fails at import, and the tests in `tests/models/test_seam_event_loops.py` show it.

rag_wright-0.8.0/docs/adr/0136-spreadsheet-tables-are-glued-back-across-blank-rows.md

@@ -0,0 +1,38 @@
+# ADR-0136: A spreadsheet table cut at its blank rows is glued back together after the parse
+
+**Status:** accepted · **Date:** 2026-10-12 · **Related:** ADR-0124 (ING-4a hidden sheets, ING-6 embedded files,
+ING-7 `table_rows`), EV-2 (gold import), PS-26
+
+## Context
+
+docling cuts a sheet into separate tables at every blank row (its Excel backend's `gap_tolerance` defaults to 0),
+and the whole engine reads each table's first row as its header: `table_rows`, the compact table text, the segmenter,
+record units and the row-span map. A database sheet with blank rows between blocks therefore lost the first record
+after each blank row, and the rows after it were keyed under that record's values (a 5-record fixture kept 3).
+Raising `gap_tolerance` is not a fix: it also merges a title block and a different table below into one table.
+
+## Decision
+
+After the parse (`capabilities/sheet_tables.py`), every table piece directly below another in the same columns,
+with only blank rows between, is decided by its first row:
+
+- **data, glued on**: the number or date columns agree (every data row above has a number or date there, the
+  header does not, and the first row does), or, with no such columns, the formatting does (the header is bold or
+  filled differently from the data rows, read with openpyxl, and the first row is formatted like the data);
+- **header, left apart**: the same header repeated, a header-formatted row, text in a number column, a table above
+  whose top row does not name every column its rows use (a form's section label), or a first row that does not fill
+  the columns the rows above fill;
+- **unsure, left apart and warned**: a text-only table with no formatting difference, which the file alone cannot
+  decide. `evaluate_ingestion` lists it in a new `warnings` list (never a failure).
+
+A glued table keeps each block's sheet rectangle as another `prov` entry, so an embedded file anchored on a sheet
+cell still maps to its table row (`_table_at` walks the blocks). The merge runs on a fresh parse and once on a parse
+cached before it (the ING-4a pattern), so no cache key changes. `openpyxl` is declared (it was installed through
+docling).
+
+## Consequences
+
+- A database sheet's records survive its blank rows; titles, forms and different tables are parsed as before.
+- On the five TexWright workbooks (measured locally, structure only): nothing glued, 63 tables and 596 rows as
+  before, 2 warnings (one-column lists separated by a blank row).
+- A sheet shape none of the rules covers is added as a new case when a customer's file shows it.

rag_wright-0.8.0/docs/adr/0137-generated-answers-state-their-support-and-none-is-an-abstention.md

@@ -0,0 +1,34 @@
+# ADR-0137: A generated answer states the evidence's support every time, and "none" is an abstention
+
+**Status:** accepted · **Date:** 2026-10-12 · **Related:** ADR-0045 (free text with light tags), PREC-1a
+(`answer_kind`), EV-3d (`score_generation`), PS-28
+
+## Context
+
+The answer generator offered `<abstain/>` as an optional alternative output. Asked about something the evidence
+does not contain, the model wrote "the provided evidence does not contain X" inside `<answer>`, cited evidence, and
+the parser labelled it `answered` (EV-3d, live: 20 of 20 questions asked of contracts that lack the clause). A
+product showed an answered, cited result for something that is not there.
+
+## Decision
+
+1. The tag instructions require `<support>full|partial|none</support>` on every response, judged by meaning rather
+   than labels (text that serves the purpose asked about answers it even when the question's wording never
+   appears). `none` is for when nothing in the evidence answers any part of the question, including a reply that
+   the evidence has no such provision; it cites nothing.
+2. The parser maps `none` to an abstention that keeps the model's sentence as its reason (scrubbed of ids like any
+   prose the reader sees), `partial` to `partial` and `full` to `answered`. A response without the tag parses as
+   before (`<partial/>`, else answered).
+3. The `generation` skill says the same. No phrase matching on the answer text (English-only, and it would catch
+   real answers such as "the agreement does not allow assignment").
+
+Rejected: treating `none` with citations as `partial` (measured: false answers back to 8 of 20).
+
+## Consequences
+
+- Measured once on the EV-3d set (20 answerable, 20 not): false answers 20/20 -> 0/20; false abstentions 0/20 ->
+  4/20, all hard cases (gold answers starting mid-sentence, a party named only in a header); citation recall
+  0.946 -> 0.758, the old figure inflated by "does not contain" answers whose citations happened to hold the gold.
+  Generation varies near the abstain boundary: one run, not a distribution.
+- A product sees an abstention, with the model's one-sentence reason in `answer`, where it used to see an answered
+  "not found".