rag-wright 0.7.0__tar.gz → 0.9.0__tar.gz

{rag_wright-0.7.0 → rag_wright-0.9.0}/.claude/skills/authoring-a-capability/SKILL.md

@@ -37,9 +37,13 @@ MCP) follows automatically for invokable kinds; 4b (a bespoke MCP server) is opt
 1. **Implementation** — the real code, in that kind's home (see per-kind below).
 2. **The invoke factory + `impl_ref`** (invokable kinds: subgraph/model) — write a co-located
    `async def ainvoke(resources, inputs)` (subgraph) / `def <name>(resources, inputs)` (model) in the capability's
-   own module. A subgraph factory builds over the opaque `WorkspaceHandle` (`resources._store`,
-   `resources._embedder`, `resources.model_id(role)`), never env; a model factory is store-independent and ignores
-   `resources`. The manifest's `impl_ref="module:attr"` points to it. The invoker imports
+   own module. A subgraph factory builds over the opaque `WorkspaceHandle`, never env: its store access with
+   `pack_store(resources, YourStore)` (your pack's store extension over the workspace's store) and its models with
+   `resources.model_id(role)`; its query embedder with `workspace_embedder(resources)` (`rag_wright.pack_sdk`, built
+   from the workspace's embedding profile) and its options with `resources.config`; a model factory is
+   store-independent and ignores `resources`. A builder that takes the store as a keyword goes through `pack_store`
+   too: `pack_store(resources, lambda store: production_x(store=store, ...))`. The handle's underscored attributes
+   are engine-internal; the reference pack's factories show the public form. The manifest's `impl_ref="module:attr"` points to it. The invoker imports
    it LAZILY and calls it — there is **NO central adapter dict** (EP-CORE-2). A plain function/agent_skill/mcp_tool
    declares no `impl_ref`.
 3. **ARD manifest + register it** — a `CapabilityManifest(slug, kind, display_name, description,
@@ -57,8 +61,9 @@ MCP) follows automatically for invokable kinds; 4b (a bespoke MCP server) is opt
    A downstream product can register without touching the canonical set (`register_capability` does not check
    slugs), BUT `manifests.author()` rejects a non-canonical slug, so a product that publishes ARD JSON must call
    `register_canonical_slugs` for its slugs first.
-   Publish to `~/.air/registry` with `uv run python scripts/publish_manifests.py`; that script publishes only the
-   engine's reference pack (it calls `load_reference_pack()`), so a product publishes its own with `publish_all`.
+   Publish the ARD JSON (`urn:air`) of what is registered with `publish(slug, root=...)` or `publish_all(root=...)`
+   from `rag_wright.api` (the root defaults to `ARD_REGISTRY_ROOT`, else `~/.air/registry`); a slug must be canonical
+   (`register_canonical_slugs`, above). The engine's `scripts/publish_manifests.py` does this for its reference pack.
    Callable kinds get `ResponseBounds` (defaulted); `agent_skill` must NOT declare bounds (loaded, not called).
 4. **Invocable + MCP for free** — once registered with an `impl_ref`, the capability is callable as
    `ainvoke_subgraph(slug, inputs, resources=ws)` / `invoke_model(slug, inputs, resources=ws)` (or
@@ -79,7 +84,7 @@ MCP) follows automatically for invokable kinds; 4b (a bespoke MCP server) is opt
 - **Invoke:** the co-located `async def ainvoke(resources, inputs)` factory (impl_ref target) builds + awaits the graph.
 - Retry/dead-letter come from the graph scaffold, not the invoker. `CapabilityManifest` has no contract field (only
   the internal `CapabilityRegistry.register(contract=...)` takes one); where the typed I/O must be declared, set
-  `capability_interface` on the manifest.
+  `capability_interface` on the manifest (`CapabilityInterface`, from `rag_wright.pack_sdk`).
 
 ### function — a plain, typed callable
 - **Home:** `capabilities/<slug>.py` (generic) or `packs/<pack>/capabilities/<slug>.py` (domain). A deterministic or model-backed callable with a Pydantic in/out contract.
@@ -103,7 +108,9 @@ MCP) follows automatically for invokable kinds; 4b (a bespoke MCP server) is opt
 - **Home:** `skills/<slug>/SKILL.md` (generic) or `packs/<pack>/skills/<slug>/SKILL.md` (domain, e.g.
   `packs/compliance/skills/compliance_judgment/SKILL.md`) + the agent runtime (e.g. `skills/rlm/`). It is LOADED (progressive
   disclosure), not called: no `ResponseBounds`. Declare `requires=(...)` for a closure over other skills and
-  `skill_runtime` for its intrinsic runtime.
+  `skill_runtime` (`SkillRuntime`, from `rag_wright.pack_sdk`) for its intrinsic runtime.
+- A product's agent (a Deep Agents harness) is not a capability kind yet: an `agent` kind is a later engine workstream
+  (PS-33, engine ADR-0139). Build agents in the product, on `agent_chat_model` for their model.
 
 ### mcp_tool — a capability exposed over MCP
 - A distinct ARD identity (`<slug>_mcp`) for the same underlying capability exposed as a cross-agent MCP tool.
@@ -129,7 +136,7 @@ add it to `_RESERVED_WITHOUT_MANIFEST` with a one-line reason. The engine's `tes
 pack, so these tests see its slugs.
 
 **The import boundary (engine repo).** `pyproject.toml` `[tool.importlinter]` has three forbidden contracts: every
-generic engine package (`source_modules`: `rag_wright.api`, `capabilities`, `contracts`, `corpus`, `ingestion`,
+generic engine package (`source_modules`: `rag_wright.api`, `capabilities`, `contracts`, `corpus`, `evals`, `ingestion`,
 `models`, `ontology`, `pack_sdk`, `skills`, `spans`, `store`, `subgraphs`, `util`) must never import
 `rag_wright.packs`; pack code (`rag_wright.packs`) imports only `rag_wright.api` and `rag_wright.pack_sdk` (plus
 itself), never another engine package directly; and `rag_wright.packs.contracts` must never import

{rag_wright-0.7.0 → rag_wright-0.9.0}/.claude/skills/building-an-ingestion-capability/SKILL.md

@@ -136,6 +136,11 @@ network-bound work inside it; the engine runs up to `IngestionTuning.extract_con
 
 - **Hidden sheets** are ingested by default; `IngestSource(include_hidden_sheets=False)` skips them (listed in
   `DocumentReport.skipped_hidden_sheets`).
+- **Blank rows inside a table**: the parse cuts a sheet's table at every blank row; the engine glues a piece back
+  onto the table above when its first row is data (its number or date columns agree, or it is formatted like the
+  data rather than the header), so no record is read as a header. A text-only table with no formatting cannot be
+  decided from the file: it stays apart and `evaluate_ingestion` lists it in `warnings` (format the header row, or
+  remove the blank rows).
 - **Exact cell values**: `table_rows(parse_document(...))` returns every data row as a `TableRow` (`columns`,
   `values`, `cell(name)`), read from the parse's cell grid, whole even when chunking split the table. A per-row unit
   carries its row in `Unit.table_row`, so the extractor reads cells, not re-parsed text.
@@ -144,12 +149,32 @@ network-bound work inside it; the engine runs up to `IngestionTuning.extract_con
   to its parent, and an `AttachedTo` edge from the child to the table-row span it belongs to, with a confidence and
   the identifier evidence (`IngestionTuning.identifier` sets what counts as an identifier). The report lists them in
   `DocumentReport.children`, `links` and `unmapped_links`.
+- **Record table or form, decided per table.** In `auto` mode a table is a database (one unit per row) when its
+  header has `RecordTableRule.min_header_cols`+ named columns, `distinct_ratio`+ of them distinct, and either a serial
+  first column in `serial_ratio`+ of the rows or `min_cols`+ columns; anything else stays whole (a form). A test log
+  or results sheet judged wrongly is fixed with `IngestionTuning(record_table=RecordTableRule(...))`, or per source
+  with `IngestSource(table_mode="record"|"block")`. Check the decision on your samples with
+  `evaluate_ingestion(..., table_labels=[{"pattern": <regex on the header row>, "label": "record"|"block"}])`
+  (first match wins): a block table split per row, or a wide record table not split, is a failure, and tables no
+  pattern matched are listed in `unlabelled_tables`.
+- **How an embedded file finds its row.** An identifier is a token of 4+ characters containing a digit that appears
+  on at most `IdentifierRule.max_rows` table rows and in at most `max_files` embedded files (defaults 3 and 3; set
+  them with `IngestionTuning(identifier=IdentifierRule(...))`). A file whose identifiers match no row, or too many,
+  stays a child document with no `AttachedTo` edge and is listed in `unmapped_links`: for a workbook with dozens of
+  embedded lab reports, check `links` against what you expect before trusting the graph.
+- **Nesting.** A file embedded in an embedded file is ingested; anything deeper is listed in
+  `DocumentReport.embedded_skipped`, not walked.
+- **Stable ids.** `IngestSource(doc_id=...)` gives a document your own id (an upload's record id, say), so a re-ingest
+  of the same file lands on the same `Document`.
 
 ## 6. Verify (definition of done)
 
-1. `evaluate_ingestion` passes on a representative sample of YOUR documents (not a hand-picked easy one). If your
-   units carry labels, its `labels` report on your gold meets the bar you set, with the confusions understood.
-2. A live ingest of that sample: no dead letters, `extraction_failures` explained, records cite spans
+1. `evaluate_ingestion` passes on a representative sample of YOUR documents (not a hand-picked easy one), with
+   `table_labels` covering every table type you have. If your units carry labels, its `labels` report on your gold
+   meets the bar you set, with the confusions understood. Inside a web backend use `aevaluate_ingestion` (with
+   `progress`) as a background job.
+2. A live ingest of that sample, in a disposable database (`with temporary_workspace(config, prefix="trial") as ws:`),
+   no dead letters, `extraction_failures` explained, records cite spans
    (`kg_read(ws, "<your type>", fields=["span_id"])`), and `span_positions(ws, doc_id)` gives the citation positions.
 3. A hermetic test of your extractor on fixed units (no model calls in the default suite; the engine's test network
    guard fails any unmarked test that reaches the network).

{rag_wright-0.7.0 → rag_wright-0.9.0}/.claude/skills/creating-evals/SKILL.md

@@ -6,8 +6,10 @@ description: >-
   starting a new domain (right after capabilities are defined), when adding or changing a capability, or when A/B-ing
   alternatives (e.g. a trained classifier vs a System-1 decision model). Covers gold-set design + reliability,
   per-capability-KIND metrics (extraction / classification / retrieval / graph / generation / judgment), the
-  gate-vs-diagnostic split, building the gold cheaply, an isolated executable harness, and optional Langfuse
-  Datasets/Experiments/Scores automation. An eval is the executable acceptance criterion; training data IS an eval.
+  gate-vs-diagnostic split, building the gold cheaply, and running it with the engine's eval toolkit
+  (`load_gold_set` from an expert's own files, one `score_*` per kind, `arun_eval` with repeats and errors kept apart,
+  an `EvalReport` a backend can return, optional Langfuse datasets and runs). An eval is the executable acceptance
+  criterion; training data IS an eval.
 ---
 
 # Creating evals (eval-first / TDD)
@@ -19,6 +21,13 @@ can A/B alternatives and catch regressions forever. Corollary observed repeatedl
 data you build for a classifier/decision model IS an eval** (a labeled gold set + a metric) — so building the eval
 first also gives you the data design for free.
 
+**The toolkit (on `rag_wright.api`).** `load_gold_set` turns an expert's CSV / XLSX / JSONL into a `GoldSet`;
+one scorer per capability kind (section 2) turns predictions into a `ScorerResult` (the gate, its thresholds,
+diagnostics that are never gated); `arun_eval` runs your capability over the gold set (concurrently, `[eval] i/N`
+progress, repeats, a failed call kept as an error) and returns an `EvalReport` -- JSON-ready, every item's outputs
+kept, so `rescore_eval` can score it again later without calling anything. Use it rather than hand-writing a
+harness; section 6 is a complete example.
+
 Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`) are in the engine repository: read them there or on GitHub, at the tag matching your installed engine. The `eval/` and `scripts/` files named as patterns are the engine's own evals: read them as
 worked examples; your evals live in your repo.
 
@@ -50,12 +59,32 @@ worked examples; your evals live in your repo.
 - **Separate a tuning set from a held-out set.** Label the held-out set BEFORE any prompt runs on it, tune only on
   the tuning set, and report both numbers (ADR-0122 boundary residue prompt: 94.5% held-out vs 96.2% tuning). A
   number measured only on the set you tuned on is not a result.
+- **Gold from your experts' own files.** Domain experts label in spreadsheets: `load_gold_set(path_or_upload,
+  GoldMapping(id=..., input=..., expected=..., group=...), name=..., version=...)` reads them as written -- the header
+  found under a title, blank rows skipped, values kept exactly, each item's place recorded -- and refuses anything
+  that would make the gold wrong (a missing column, a row without its id, a duplicate id) with where it is. Map
+  `group` to the source document so splits stay leakage-safe. A workbook with several sheets: name one with
+  `sheet=`. Say how the gold was made with `provenance=` (`hand`, `silver` or `generated`; or a per-row column through
+  `GoldMapping(provenance=...)`), so silver is never conflated with gold. A gold set is `local_only` by default:
+  client data never leaves the machine.
 - **Gold stays local when the corpus is restrictively licensed** (CUAD/ACORD-derived gold lives under the
   gitignored `data/eval/`); commit the builder and the scorer, never the data.
 
 ## 2. Pick the metric by capability KIND, and split GATE vs DIAGNOSTIC
 Always set ONE pass/fail **gate** (from the acceptance criterion) and report **diagnostics** alongside (never gate
-on a diagnostic).
+on a diagnostic). Each kind has its scorer on `rag_wright.api`, checked against the engine evals named below:
+
+| kind | scorer | gate |
+|---|---|---|
+| classification / typed decision | `score_classification` | the weakest class's recall@k (+ top-1 accuracy via `top1_floor`) |
+| retrieval | `score_retrieval` | mean recall@k at a relevance grade; nDCG reported |
+| extraction | `score_extraction` | recall and precision, apart |
+| judgment / verdict | `score_judgment` | precision and recall of the actionable verdict |
+| generation / QA | `score_generation` | citation recall, correct abstention, groundedness |
+| graph / relational | `score_relational` | recall over the answer set |
+| ingestion structure / unit labelling | `score_ingestion` | `evaluate_ingestion`'s checks (+ label accuracy) |
+| candidate-then-choose | `score_candidate_choice` | candidate coverage, value-level recall and precision |
+
 - **Extraction** (section → records, clauses, requirements): recall / precision / F1 of extracted items vs gold,
   reported SEPARATELY (under- vs over-extraction are different failures). Watch over-extraction on non-operative
   input (definitions) and under-extraction on long input.
@@ -84,7 +113,7 @@ on a diagnostic).
   scores mean what they say), not one accuracy number. Pattern: `eval/semantic_judge_gold.py` (ING-9: decision
   model 95.3% vs LLM 90.1% on 192 blind hand-labelled cases).
 - **Candidate-then-choose pipelines** (a generator proposes, a model picks): measure the generator's **coverage**
-  separately, because it caps recall (ING-9b residual values: candidate coverage 95%; value-level recall 0.83 /
+  separately, because it caps recall (ING-9b residual values: candidate coverage 0.93, 188/203 recomputed from the saved outputs; value-level recall 0.83 /
   precision 0.88 vs the LLM's 0.60 / 0.79. Pattern: `eval/residual_decision_gold.py`). **Count is not accuracy**:
   emitting more values is not a win without precision.
 - **Ingestion structure**: `evaluate_ingestion` (in `rag_wright.api`) is the packaged structural eval of the
@@ -111,42 +140,87 @@ on a diagnostic).
 - **Generate hard cases** (near-boundary positives/negatives) to stress the exact confusions — TRAIN-ONLY; the
   gold test stays real. (Full data playbook: the `setfit` skill Phase 1/4 + `classifier-opportunity-analysis`.)
 
-## 4. Make it an executable, isolated harness
-- **Isolate the capability under test**: inject it (a seam / `extract_override` / an injected `retrieve`) so the
-  eval measures ONE capability, not the whole pipeline. Invoke production code THROUGH the capability layer
-  (`ainvoke_subgraph`/`ainvoke_model`), never a hand-built copy. Score the **shipped request** (the exact
-  prompt/question builder production uses, e.g. `judge_request`), not a copy of it re-typed in the eval.
-- **Repeatability for non-deterministic models.** Run each case several times on identical input; report how many
-  answers flip and how far the scores sit from the threshold. Flips cluster near the threshold and usually mean
-  the QUESTION is ambiguous: fix the question, do not vote it away (Jev flipped 5/181 answers at scores 0.47-0.56;
-  temperature/seed did not help, majority voting barely helped, rewriting the ambiguous question did. The ADR-0122
-  boundary residue prompt went from 94-96% with flips to 460/461, zero flips across 3 calls). Pattern:
-  `eval/boundary_residue_gold.py`.
+## 4. Run it: `arun_eval`
+`arun_eval(gold, task, scorers, *, repeats, concurrency, progress)` is the harness. It runs `task` (your coroutine,
+or a plain function, which runs off the event loop) on every item, `repeats` times, `concurrency` at a time, streams
+`[eval] i/N` lines to `progress`, keeps a call that raises as an ERROR in that item's slot (never dropped, never an
+answer), scores each repeat, and passes a scorer only if it passes on every repeat (its gate is the worst repeat;
+`per_repeat_gate` lists each). `run_eval` is the synchronous form; inside a web backend, await `arun_eval` as a
+background job. What is still yours:
+- **Isolate the capability under test**: the task calls ONE capability, injected (a seam, such as the reference pack's
+  `extract_override` or an injected `retrieve`), so the eval measures it, not the whole pipeline. Invoke production code THROUGH the
+  capability layer (`ainvoke_subgraph`/`ainvoke_model`), never a hand-built copy. Score the **shipped request** (the
+  exact prompt/question builder production uses, e.g. the reference pack's `judge_request`), not a copy of it re-typed in the eval.
+- **Repeatability for non-deterministic models.** Use `repeats` and read `report.flips`; flips cluster near the
+  threshold and usually mean the QUESTION is ambiguous: fix the question, do not vote it away (Jev flipped 5/181
+  answers at scores 0.47-0.56; temperature/seed did not help, majority voting barely helped, rewriting the ambiguous
+  question did. The ADR-0122 boundary residue prompt went from 94-96% with flips to 460/461, zero flips across 3
+  calls). Pattern: `eval/boundary_residue_gold.py`.
 - **Cache keys include the prompt and the method.** Any decision or extraction cache must key on the prompt text
   and the method (decision model vs LLM, and which model), so an eval never reuses outputs a different prompt or
   method produced (ADR-0122 ING-4d / ADR-0040 ING-9b: the residue decision cache and the clause cache).
-- **Score from a RESULT ARTIFACT (JSON), not stdout scraping** (scraping truncates and silently drops rows).
-- **Parallelize** model/LLM calls (async + semaphore) — same cost, far less wall-clock; order-preserving so it
-  stays deterministic.
-- **Env-selected** so the SAME harness runs local (dev) or on Modal (full corpus + GPU).
-- **Pre-flight paid bulk** (>~50 paid calls): print the exact count + cost and wait (`warn-before-bulk` rule).
-- Stream `X/N` progress + actively monitor any run over ~30s (never launch-and-forget).
+- **Keep the report.** `report.model_dump_json()` holds every item's outputs; save it next to the gold version.
+  `rescore_eval(report, gold, scorers)` re-scores it with new thresholds, a new scorer or a corrected gold, free. An
+  eval kept only as aggregate numbers can never be checked again.
+- **Pre-flight paid bulk** (>~50 paid calls): print the exact count + cost and wait for approval before launching.
 - **Hermetic tests never touch the network**: `tests/conftest.py` fails a test that resolves a non-local host unless
-  it carries a live marker (`model`, `store`, ...). Live evals run as `uv run python -u eval/<name>.py` (outside
-  pytest) or carry a live marker.
-
-## 5. Langfuse — optional eval automation (we already use it for tracing)
-Langfuse has a first-class eval stack we are NOT yet using (we use it only for spans/usage today): a **Dataset**
-(items = input + optional expected output) → a **Task** (your capability) run over the dataset as an **Experiment
-Run** → **Evaluators** (deterministic checks or LLM-as-judge) producing **Scores** (numeric/categorical/boolean),
-all linked to traces. Reach for it when you want **tracked, re-runnable, dashboarded** evals + regression tracking
-across capability versions; a local JSON harness is enough for a quick one-off gate. Keep the gold BUILDER + a
-committed snapshot in the repo (reproducibility); push the items to the Langfuse dataset so runs and scores land
-next to the traces you already collect. Ground the exact Dataset/Experiment API before wiring
-(https://langfuse.com/docs/evaluation/concepts, https://langfuse.com/docs/datasets/overview); confirm the installed
-SDK version's surface, not prose.
-
-## 6. The TDD loop
+  it carries a live marker (`model`, `store`, ...). Live evals run as scripts (outside pytest) or carry a live
+  marker.
+
+## 5. Langfuse -- optional: datasets, runs and scores next to the traces
+`arun_eval(..., langfuse=True)` (or `RAG_EVAL_BACKEND=langfuse`; the keys alone send nothing) runs the same eval on
+Langfuse: the gold set becomes a dataset (items upserted; items no longer in the gold set archived; the version is
+the server's latest timestamp, recorded as `report.dataset_version`), each repeat a dataset run (`report.run_urls`),
+and each scorer's gates and pass/fail are pushed as run scores. The report is the same, scored locally from the same
+outputs. With engine tracing on (`RAG_TRACE_LEVEL`), every model call the task makes nests under its item's trace,
+on both backends. What was verified against Langfuse 4.15.1 (self-hosted), and what to know:
+- **`local_only` is refused before anything is sent.** Only a self-hosted Langfuse should ever receive client data;
+  set `local_only=False` only for data that may leave the machine.
+- **Scoring runs in your process**, not in Langfuse: its code evaluators are a 2-second, standard-library sandbox
+  with no network, so they cannot run your scorers.
+- **The SDK's own `run_experiment` drops an item whose task raises, and blocks a running event loop**; `arun_eval`
+  handles both (an error is an error in its slot; the run is off the loop). Call `arun_eval`, not the SDK directly.
+- **Reading observations back**: the v2 list endpoint is Cloud-only until Langfuse v4 ships in the open-source
+  release, and the v1 endpoint times out on a trace-only query: bound every read by time.
+- Langfuse's managed LLM-as-a-judge and decision-model evaluators, and its UI prompt experiments, test a prompt in
+  Langfuse, not your pipeline: out of scope for this toolkit (a product may still use them).
+
+## 6. A complete example
+An expert's labelled workbook, your capability run through the capability layer, two repeats, the report kept and
+scored again later. `my_judge` stands for your capability's call (e.g. through `ainvoke_model`); the rest runs as
+written.
+
+```python
+import asyncio
+import functools
+from pathlib import Path
+
+from rag_wright.api import (EvalReport, GoldMapping, arun_eval, load_gold_set, rescore_eval,
+                            score_judgment)
+
+GOLD_FILE = "gold/nli_expert_labels.xlsx"  # the expert's own file
+REPORT = "eval_runs/nli_judge.json"
+
+mapping = GoldMapping(id="ID", input=["Premise", "Hypothesis"], expected="Label", group="Contract")
+gold = load_gold_set(GOLD_FILE, mapping, name="nli-judge", version="2026-10-12")
+
+
+async def judge(item):
+    return await my_judge(item.input["Premise"], item.input["Hypothesis"])
+
+
+gate = functools.partial(score_judgment, actionable="contradiction", precision=0.8, recall=0.7)
+report = asyncio.run(arun_eval(gold, judge, [gate], repeats=2, concurrency=8, progress=print))
+Path(REPORT).parent.mkdir(parents=True, exist_ok=True)
+Path(REPORT).write_text(report.model_dump_json(indent=1), encoding="utf-8")
+print(report.passed, report.scores[0].gate, "errors", report.errors, "flips", report.flips)
+
+stricter = functools.partial(score_judgment, actionable="contradiction", precision=0.9, recall=0.7)
+saved = EvalReport.model_validate_json(Path(REPORT).read_text(encoding="utf-8"))
+print("at 0.9 precision:", rescore_eval(saved, gold, [stricter]).passed)
+```
+
+## 7. The TDD loop
 1. Write the eval from the contract + a handful of real gold items → it fails (red) on the unbuilt/weak capability.
 2. Implement the minimum to pass the gate (green).
 3. A/B alternatives (rule / trained classifier / decision model / LLM) on the SAME gold; select on gate + diagnostics + cost + calibration.
@@ -158,7 +232,11 @@ SDK version's surface, not prose.
 - **Gating on a diagnostic** (e.g. nDCG) — report it, don't threshold it.
 - **No reliability check on a subjective gold** — you can't read a number off a gold whose own labels disagree.
 - **Eval not isolated** to the capability (measures the whole pipeline) — you can't attribute a regression.
-- **Scraping stdout** for metrics — write + read a JSON artifact.
+- **Scraping stdout** for metrics — keep and read the `EvalReport` JSON.
+- **A failed call scored as an answer** (a timeout read as "neutral", an error read as "abstained") — it is an error;
+  `arun_eval` keeps it apart (ADR-0133).
+- **Only aggregates kept** — an eval whose per-item outputs were not saved can never be re-scored or checked; keep
+  the report.
 - **Faking a win by shrinking the sample / picking easy items** — real, representative, honest.
 
 ## Hand-off / where this sits

{rag_wright-0.7.0 → rag_wright-0.9.0}/.claude/skills/laya/SKILL.md

@@ -25,9 +25,10 @@ you can fine-tune. In RAG_Wright a decision model is wired as a `kind="model"` c
 
 **Swapping Jev for Laya is config, but not free.** `DECISION_PROFILES` has only Jev entries (`jev-1.13`, the
 default, and `jev-latest`). A Laya swap needs (1) a server that speaks the Decisions API (POST
-`{model, state, questions}`, return `{answers: {<id>: {type, noul|choice|score}}}`), and (2) a `DecisionModelProfile`
-entry in `DECISION_PROFILES` with its `endpoint`, `served_model_id` and `api_key_env` (an unknown id falls back to
-the OpenRouter endpoint). Then select it with `RAG_DECISION_MODEL=<profile id>`;
+`{model, state, questions}`, return `{answers: {<id>: {type, noul|choice|score}}}`), and (2) its profile, registered
+at startup from your product: `register_decision_profile(DecisionModelProfile(model_id="laya-onprem",
+served_model_id=..., endpoint=..., api_key_env=...))`, both on `rag_wright.api` (an unregistered id falls back to the
+OpenRouter endpoint). Then select it with `RAG_DECISION_MODEL=<profile id>` (or the `jev_decision` input `model`);
 `RAG_JEV_TIMEOUT_S` overrides the request timeout.
 
 **Batch the questions.** One call carries one `state` and many keyed questions, so ask everything about a unit in
@@ -68,7 +69,7 @@ server you swap in.
 ## Data — the JSONL schema + criteria (the differentiator)
 One JSONL line per case (`research/scripts/finetune_single_device.py` reads this exact shape):
 ```
-{"state": "<clause text>",
+{"state": "<unit text>",
  "questions": {"<dim>": {"type":"choice","instructions":"<the question>","criteria":{"<value>":"<description>", ...}}},
  "gold": {"<dim>": {"probabilities": {"<value>": <p>, ...}, "label": "<value>"}}}
 ```
@@ -78,7 +79,7 @@ One JSONL line per case (`research/scripts/finetune_single_device.py` reads this
 - **`gold` is a DISTRIBUTION (soft targets), not a hard label** — RLCD trains on the teacher's probability per option.
   One-hot (`{gold:1.0, other:0.0}`) works and is the pragmatic start; true soft targets (teacher per-option probs)
   are the design-intended enhancement.
-- Reuse your existing **contract-disjoint, symmetric splits** so floors are directly comparable to SetFit. Silver
+- Reuse your existing **document-disjoint, symmetric splits** so floors are directly comparable to SetFit. Silver
   goes in TRAIN only; val/test stay gold.
 
 ## Fine-tune — single T4, RLCD, calibration built in
@@ -116,11 +117,12 @@ One JSONL line per case (`research/scripts/finetune_single_device.py` reads this
   the model from the engine (never a hardcoded/stale id; bulk teacher labeling on Modal Qwen, not OpenRouter), load
   `.env` by EXPLICIT path from an out-of-repo script, SMOKE one item before the fan-out, and stream X/N to a log you
   actively monitor. Those exact mistakes cost runs on 2026-10-04.
-- **Modal harness = reuse the [[setfit]] hardened pattern:** `.uv_pip_install("laya")` + `.add_local_file` the
+- **Modal harness = follow the [[setfit]] hardened pattern** (the pattern described there; that harness's code is
+  not shipped): `.uv_pip_install("laya")` + `.add_local_file` the
   finetune script; launcher BLOCKS on `.get()` per spawn (no spawn-and-return), stamps + verifies a `data_sha`,
   writes a manifest, streams X/N; snapshot keepers server-side to a `/checkpoints/<name>` path (a small copy fn) —
   the laya volume has no built-in snapshot, add one. Never lose a fine-tune.
-- **ACCOUNT CONTAINER CAP = 10 (fzaidi2014).** Spawning more than 10 fine-tunes at once does NOT run them all —
+- **ACCOUNT CONTAINER CAP (10 on the engine authors' account; check your workspace's).** Spawning more than 10 fine-tunes at once does NOT run them all —
   Modal queues the rest and runs ~10 at a time (correct, but ~N/10 waves of wall-clock, and a "why only 10 running?"
   surprise). Size a `groupbake`/dimbatch fan-out to ≤10 in flight (chunk into waves + gather between), or state the
   wave count honestly. This is a DIFFERENT knob from the vLLM `max_containers=1` + `@modal.concurrent` batching in the

{rag_wright-0.7.0/src/rag_wright/.agents → rag_wright-0.9.0/.claude}/skills/qwen-vllm-modal/SKILL.md

@@ -16,6 +16,11 @@ ADR-0110 and the other `docs/` paths are in the engine repository: read them the
 matching your installed engine.
 This skill is the operational checklist that ties them together.
 
+**Which route.** A product deploys and reaches the server through the engine API, `adeploy_model_server` (the
+section after the locked config). The raw `modal deploy` of the bundled script further down is the engine
+developer's route, and what the API runs underneath. **When to stop it** is one rule, in "Cold start & cost": deploy
+once, let it scale to zero between uses, never stop it between runs.
+
 ## The LOCKED production config (ADR-0110) — do not re-litigate
 **Config B: `Qwen/Qwen3.8-27B-FP8` weights + `--kv-cache-dtype fp8`, on 1× A100-80GB, TP=1, `--max-model-len 16384`, `--gpu-memory-utilization 0.95`.**
 - Config B (FP8 weights) is the **locked deploy** — 52.71× concurrency @16K (ADR-0110 rounds it to 53×), ~23% higher throughput, ~31 GB weights (30.9 GB on disk, 28.9 GiB in VRAM);
@@ -38,15 +43,21 @@ from rag_wright.api import EngineConfig, ModelServerSpec, adeploy_model_server,
 server = await adeploy_model_server(ModelServerSpec(name="acme-qwen", max_num_seqs=32))  # key from VLLM_API_KEY
 config = EngineConfig(store=..., models={"general": server.model_id})   # or any model= argument
 ...
-stop_model_server("acme-qwen")                                            # stop billing
+stop_model_server("acme-qwen")  # only to retire it: stopping discards the snapshots (see "Cold start & cost")
 ```
 
-`amodel_server(spec)` reconnects to a server deployed earlier (a restart, another process) without redeploying.
+`server.model_id` is `qwen3.8-27b-modal@<name>`: a copy of the `qwen3.8-27b-modal` profile pointed at this server
+(reasoning on for structured calls and agent turns, off for free text). Use that id, never `VLLM_BASE_URL`. A
+reasoning-off structured pick (bulk closed-vocab labeling) has no `@<name>` variant: point `VLLM_BASE_URL` at
+`server.base_url` and use the `qwen3.8-27b-modal-nothink` profile.
+`amodel_server(spec)` reconnects to a server deployed earlier (a restart, another process) without redeploying;
+`await_model_server(server, timeout_s=...)` waits for readiness; `amodel_server_status(spec)` looks without waking it
+(`deployed`, `running` (a container bills), `healthy`, `weights_cached`); `astop_model_server(name)` is the async stop.
 The API key comes from the variable `ModelServerSpec.api_key_env` names (default `VLLM_API_KEY`); there is no default
 key, so set a real one. Needs `rag-wright[modal]` and a Modal login. One app per server name: different names give
 different servers (and URLs), for example one per tenant.
 
-**The exact working deploy (this succeeded):**
+**The engine developer's route: the raw deploy (this succeeded):**
 ```
 VLLM_API_KEY=<a long random string> MODAL_IMAGE_BUILDER_VERSION=2025.06 \
   APP_NAME=rw-qwen3-modal MODEL=Qwen/Qwen3.8-27B-FP8 KV_CACHE_DTYPE=fp8 \
@@ -58,12 +69,12 @@ VLLM_API_KEY=<a long random string> MODAL_IMAGE_BUILDER_VERSION=2025.06 \
 The script refuses to deploy without `VLLM_API_KEY`: Modal's web endpoint is public and unauthenticated, so vLLM's
 `--api-key` is the only gate on a billing A100, and there is no default key. Then wire the engine: `.env`
 `VLLM_BASE_URL=https://<workspace>--rw-qwen3-modal-serve.modal.run/v1` and the same `VLLM_API_KEY`; the profile
-`qwen3.8-27b-modal` routes there (the seam also refuses an unset key). **Stop billing when done:**
-`uv run --no-sync modal app stop rw-qwen3-modal --yes` (A100 is expensive).
+`qwen3.8-27b-modal` routes there (the seam also refuses an unset key). When to stop it: "Cold start & cost" below.
 
 **The engine does NOT reach Modal by default.** Every role's default is `_PRODUCT_LLM = "qwen3.8-27b-modal-or"`,
-the same model routed through OpenRouter (a Modal cold start can still take minutes when Modal builds a snapshot). To send calls to the Modal server, set
-`RAG_MODEL_ALL=qwen3.8-27b-modal` (all roles), a per-role `RAG_MODEL_<ROLE>`, or `EngineConfig.models`. For bulk
+the same model routed through OpenRouter (a Modal cold start can still take minutes when Modal builds a snapshot). To send calls to the Modal server, put
+`server.model_id` in `EngineConfig.models` (the API route), or on the raw route set `RAG_MODEL_ALL=qwen3.8-27b-modal`
+(all roles) or a per-role `RAG_MODEL_<ROLE>`. For bulk
 closed-vocab labeling, the reasoning-off profile `qwen3.8-27b-modal-nothink` exists (opt-in; see the reasoning
 caveat below). `VISION_OCR` now defaults to the product LLM too, but image inputs through the Modal vLLM server are
 UNVERIFIED (the serve script passes no multimodal flags): confirm an image call works before pointing
@@ -123,14 +134,22 @@ the OFFICIAL prebuilt vLLM image** rather than pip-building vLLM:
   a weight read of 12-154 s from the Volume (9P), then compile-cache load + CUDA-graph capture. Measured and NOT
   worth adopting (ADR-0109 addendum): `--load-format runai_streamer`, precompiled `.pyc`, `HF_HUB_OFFLINE=1` (it
   changes vLLM's compile-cache key every container, so each start recompiles). Poll `/health` for readiness.
-- **A100 is billed while up — `modal app stop <app> --yes` the moment you're done.** Verify with `modal app list`
-  (state `stopped`, 0 tasks) + endpoint 404.
-- **Do NOT warm the GPU before the consumer is ready, and do NOT trust auto-scaledown to protect billing** (burned
-  2026-10-04: a smoke-warmed A100 sat idle ~20 min while unrelated work ran, because "scaledown_window will handle
-  it" was assumed — it did not, fast enough). RULE: build the bulk/label/eval consumer FIRST (while the app is
-  stopped/cold), THEN warm → smoke → run → **`modal app stop` immediately after**, in one continuous go. If you warm
-  only to prove stand-up and the consumer is not next, STOP it right after the smoke. Treat the explicit stop as
-  mandatory, not the scaledown as sufficient; verify stopped with `modal app list`.
+- **When to stop it -- one rule (EV-6; it used to contradict the line above).** The A100 is billed only while a
+  container runs; the server scales to zero after 10 idle minutes (`scaledown_window=600`, no warm minimum) and then
+  costs nothing and KEEPS its snapshots, so the next request is back in ~13 s. `modal app stop` saves at most those
+  10 minutes and makes the next deploy pay a snapshot-building boot (usually 4-9 min; 15 min seen once) (2026-10-12: a stop after a failed smoke
+  run, then a redeploy, cost a 911 s boot).
+  1. **Deploy once per server name**; reuse the name (a new name builds new snapshots).
+  2. **Build the consumer first** (the bulk / label / eval script, its smoke test, its progress logging) while no
+     container runs, then warm, smoke and run in one go. Do not warm a GPU before its consumer is ready (2026-10-04:
+     a smoke-warmed A100 sat idle ~20 min while unrelated work ran).
+  3. **Between runs, and when you are done, let it scale to zero.** Never stop it between runs; a script's error
+     path must not stop it either when a fix-and-rerun is likely.
+  4. **Check it reached zero**: `amodel_server_status(spec).running` is False (or `modal app list` shows the app
+     `deployed` with 0 tasks) about 10 minutes after the last call. If tasks remain past the window, find out why (that is the 2026-10-04 burn) and stop it.
+  5. **Stop (`modal app stop <app> --yes`, or `stop_model_server`) only to retire a server**: you will not use it
+     for a long time and do not want its endpoint up, or you are replacing its deployment. Expect the next deploy to
+     rebuild the snapshots.
 
 ## The meta-lesson
 Config + recipe live in **ADR-0110 + the script + this skill**. Before any Qwen/Modal work: read them, don't

{rag_wright-0.7.0/src/rag_wright/.agents → rag_wright-0.9.0/.claude}/skills/setfit/SKILL.md

@@ -219,11 +219,14 @@ after the 21-classifier runs — each one cost real time and each is avoidable w
 order before you fan out any bulk job.
 
 1. **Resolve the model from the engine, never hardcode a model id.** A hardcoded id rots: `ibm-granite/granite-4.1-8b`
-   now 404s on OpenRouter ("No endpoints found"). Use the engine resolver (`models.profiles.model_for(ModelRole.<ROLE>)`,
+   now 404s on OpenRouter ("No endpoints found"). Use the engine resolver (`rag_wright.pack_sdk.model_for(ModelRole.<ROLE>)`,
    or `ws.model_id(role)` on a workspace; the model-profile seam) so the id and backend come from config, and **confirm it with a one-call smoke before any
    bulk run**. Bulk teacher labeling runs on **self-hosted Qwen on Modal** (`qwen-vllm-modal` skill + ADR-0110), NOT
-   OpenRouter (flat-GPU = the warn-before-bulk rule); reasoning ON for label fidelity; client concurrency ≤16; raise
-   the structured timeout for reasoning-ON bulk; `modal app stop` when done.
+   OpenRouter (a flat GPU rate, not a per-call bill: before more than ~50 paid calls, print the exact count and cost
+   and wait for approval); reasoning ON for label fidelity; client concurrency below the server's `max_num_seqs`
+   (counting every concurrent run); raise the structured timeout for reasoning-ON bulk; between runs let the server
+   scale to zero, and stop it only to retire it (`qwen-vllm-modal`, "Cold start & cost": a stop discards its
+   snapshots).
 2. **Load `.env` by EXPLICIT path in any script run from outside the repo root** (a scratchpad / one-off script).
    `load_dotenv()` with no path searches the CALLING SCRIPT's directory, so a script in `/tmp/...` silently misses
    the project `.env` → `KeyError` or wrong creds. Pass the repo `.env` path (`load_dotenv("/abs/repo/.env")`) or
@@ -257,7 +260,7 @@ Concrete tool gotchas from a real run; following them saves real time.
   per-run progress log on the volume. A run whose app shows `stopped / 0 tasks` seconds after launch did NOT train —
   confirm via the platform task count and the registry, not by assuming.
 - **Mind the ACCOUNT CONTAINER CAP when sizing the fan-out.** A workspace has a max concurrent-container count (10 on
-  the fzaidi2014 account). Spawning more than that does NOT run them all at once — the excess QUEUES and runs in waves
+  the engine authors' account; check your own workspace's limit). Spawning more than that does NOT run them all at once — the excess QUEUES and runs in waves
   (correct + no double-spend, but ~N/cap× the wall-clock, and only `cap` show as running). Size the wave to ≤ the cap
   (chunk spawns + gather between waves) or report the wave count honestly. Distinct from any in-container concurrency.
 - **When multiple entrypoints exist, name the one to run** (`script.py::entrypoint`) and use the FULL script path;