rag-wright 0.1.0__tar.gz → 0.2.0__tar.gz

rag_wright-0.2.0/.claude/skills/authoring-a-capability/SKILL.md

@@ -0,0 +1,145 @@
+---
+name: authoring-a-capability
+description: >-
+  How to author a new RAG_Wright engine capability of any kind (subgraph, function, model, agent_skill, mcp_tool)
+  so it is registered, ARD-discoverable, and invokable by name through the engine API. Use it whenever you add a
+  new capability or a new-domain product/graph needs one: it gives the shared registration + ARD + invocation
+  contract (the four surfaces + the definition of done), the per-kind implementation specifics, and the
+  conformance guardrail that keeps the catalog honest. Grounded against the real code; keep it in step with it.
+---
+
+# Authoring a capability
+
+A **capability** is a named, ARD-registered unit of engine behavior (FR-C). Every capability has a `kind`
+(`ard.py::EntryKind`): `subgraph | function | model | agent_skill | mcp_tool` (`dagster_asset` is reserved). The
+contract below is the SAME for every kind; only the implementation differs. Capabilities compose — a subgraph calls
+functions/models; a product invokes a capability by name through `rag_wright.api`.
+
+**Ground every call before writing it** (CLAUDE.md library rule). The authoritative sources this skill summarizes:
+`capabilities/registry.py` (the canonical-slug set, `register_canonical_slugs`, the internal
+`CapabilityRegistry.register`), `capabilities/manifests.py` (`CapabilityManifest`; `_ENGINE_SPECS`, the engine's
+7 generic manifests returned by `engine_capabilities()`; `MANIFEST_SPECS`, the runtime catalog; `register_capability`,
+`load_pack(module)`), the reference pack's manifests (`CONTRACT_SPECS` / `COMPLIANCE_SPECS` in `rag_wright.packs.{contracts,compliance}.pack`), `capabilities/ard.py` (`EntryKind`, `MEDIA_TYPE_BY_KIND`,
+`CALLABLE_KINDS`), `scripts/publish_manifests.py`, `api/invoke.py` + `capabilities/invoke.py::capability_impl` (the adapter-free impl_ref invoker + drift guard),
+`api/mcp.py` (generic MCP exposure). The guardrail test is `tests/capabilities/test_authoring_contract.py`.
+Paths here are relative to `src/rag_wright/` unless they start with `scripts/` or `tests/`. A product imports the
+authoring surface from `rag_wright.api`: `CapabilityManifest`, `register_capability`, `load_pack`,
+`engine_capabilities`, `register_canonical_slugs`, `canonical_capability_slugs`, `load_reference_pack`,
+`reference_pack` (the same objects as in `capabilities/manifests.py` and `capabilities/registry.py`).
+
+## The four surfaces (the definition of done)
+
+A finished capability touches these: 1 (implementation) is for EVERY kind; 2 (the invoke factory + `impl_ref`) is
+for invokable kinds (`subgraph`/`model`); 3 (manifest + register) is for every discoverable kind; 4 (invocable +
+MCP) follows automatically for invokable kinds; 4b (a bespoke MCP server) is optional.
+
+1. **Implementation** — the real code, in that kind's home (see per-kind below).
+2. **The invoke factory + `impl_ref`** (invokable kinds: subgraph/model) — write a co-located
+   `async def ainvoke(resources, inputs)` (subgraph) / `def <name>(resources, inputs)` (model) in the capability's
+   own module. A subgraph factory builds over the opaque `WorkspaceHandle` (`resources._store`,
+   `resources._embedder`, `resources.model_id(role)`), never env; a model factory is store-independent and ignores
+   `resources`. The manifest's `impl_ref="module:attr"` points to it. The invoker imports
+   it LAZILY and calls it — there is **NO central adapter dict** (EP-CORE-2). A plain function/agent_skill/mcp_tool
+   declares no `impl_ref`.
+3. **ARD manifest + register it** — a `CapabilityManifest(slug, kind, display_name, description,
+   representative_queries=(2-5…), tags=…, impl_ref=…)`. **The catalog ships EMPTY (EP-CORE-3):** call
+   `register_capability(manifest)` at runtime to add it (a product registers its own; the engine's reference pack is
+   opt-in via `load_reference_pack()`). `representative_queries` is the field ARD discovery ranks on — write real,
+   specific queries. For the ENGINE's reference pack, the manifest is committed in its pack's `pack.py` (`CONTRACT_SPECS` / `COMPLIANCE_SPECS`) and
+   the slug in its `*_CAPABILITY_SLUGS` (added to the registry by `register_canonical_slugs` when the pack loads);
+   a GENERIC engine capability's manifest is in `manifests.py::_ENGINE_SPECS` and its slug in
+   `registry.ENGINE_CAPABILITY_SLUGS`. Engine capabilities (`jev_decision`, `generation`, ...) are NOT in the
+   catalog until registered too: register them from `engine_capabilities()` when your pack uses them.
+   **Packs:** a pack is a module exposing `register()`, which calls `register_canonical_slugs(...)` for its slugs and
+   then `register_capability(m)` per manifest (plus any engine capabilities it builds on); load it with
+   `load_pack("<module>")`. `load_reference_pack()` is just `load_pack("rag_wright.packs.compliance.pack")` (compliance registers contracts first).
+   A downstream product can register without touching the canonical set (`register_capability` does not check
+   slugs), BUT `manifests.author()` rejects a non-canonical slug, so a product that publishes ARD JSON must call
+   `register_canonical_slugs` for its slugs first.
+   Publish to `~/.air/registry` with `uv run python scripts/publish_manifests.py`; that script publishes only the
+   engine's reference pack (it calls `load_reference_pack()`), so a product publishes its own with `publish_all`.
+   Callable kinds get `ResponseBounds` (defaulted); `agent_skill` must NOT declare bounds (loaded, not called).
+4. **Invocable + MCP for free** — once registered with an `impl_ref`, the capability is callable as
+   `ainvoke_subgraph(slug, inputs, resources=ws)` / `invoke_model(slug, inputs, resources=ws)` (or
+   `ainvoke_model(...)`, all on `rag_wright.api`). The invoker first checks the slug is in the catalog with the
+   kind that invoker serves (`KeyError` / `ValueError` otherwise), then imports the `impl_ref` via
+   `capabilities.invoke.capability_impl` — AND exposable over MCP (surface 4b), with **zero engine edits**.
+
+4b. **MCP exposure** (optional) — any invokable capability is already an MCP tool with zero extra code via
+   `api/mcp.py::build_capability_mcp(slug, resources=ws)` (EP-RT-2). Write a bespoke FastMCP server only
+   when you want a CURATED, typed tool signature instead of the generic opaque-`inputs` surface.
+
+## Per-kind specifics
+
+### subgraph — a compiled LangGraph `StateGraph`
+- **Home:** a generic engine subgraph in `subgraphs/<slug>.py`; a domain subgraph in its pack, `packs/<pack>/subgraphs/<slug>.py`
+  (the reference pack: `packs/contracts/subgraphs/typed_property_retrieval.py`). A `production_<slug>(*, store, ...) -> CompiledGraph` builder: `g = StateGraph(_State)`,
+  add nodes/edges with `START`/`END`, `return g.compile()`. Nodes call functions/models (compose).
+- **Invoke:** the co-located `async def ainvoke(resources, inputs)` factory (impl_ref target) builds + awaits the graph.
+- Retry/dead-letter come from the graph scaffold, not the invoker. `CapabilityManifest` has no contract field (only
+  the internal `CapabilityRegistry.register(contract=...)` takes one); where the typed I/O must be declared, set
+  `capability_interface` on the manifest.
+
+### function — a plain, typed callable
+- **Home:** `capabilities/<slug>.py` (generic) or `packs/<pack>/capabilities/<slug>.py` (domain). A deterministic or model-backed callable with a Pydantic in/out contract.
+- Invoker adapters for `function` are not wired yet (EP-API-2c); until then functions are composed inside
+  subgraphs, not invoked standalone through the API. Still register + manifest it.
+
+### model — a trained checkpoint behind a seam
+- **Home:** a GENERIC engine model capability lives in `capabilities/` (e.g. `capabilities/jev_decision.py`); a
+  domain model (e.g. the reference pack's SetFit clause classifier and 29-dim property fleet) lives with its pack,
+  in the reference pack (`rag_wright.packs.contracts.spans.model_capabilities`) for the engine's worked example, or in
+  your product repo. Load the checkpoint ONCE and cache it (the fleet is heavy).
+- Serve behind the existing seam/adapter so nothing upstream changes (to FIND where a model cap belongs, use the
+  `classifier-opportunity-analysis` skill; to BUILD/train + checkpoint + serve it, the `setfit` skill). The impl_ref
+  factory is `def <slug>(resources, inputs)` for a SYNC impl (CPU-bound local inference — a classifier/XGBoost
+  checkpoint; `resources` ignored) or `async def <slug>(resources, inputs)` for an ASYNC impl (I/O-bound — an
+  LLM-backed model cap calling OpenRouter / a local vLLM client). `invoke_model` runs a sync impl and REFUSES an
+  async one; `ainvoke_model` (EP-API-7) off-loads a sync impl with `asyncio.to_thread` and awaits an async impl
+  directly, with an optional `sem` for fan-out backpressure.
+
+### agent_skill — authored SKILL.md + the Deep Agents runtime
+- **Home:** `skills/<slug>/SKILL.md` (generic) or `packs/<pack>/skills/<slug>/SKILL.md` (domain, e.g.
+  `packs/compliance/skills/compliance_judgment/SKILL.md`) + the agent runtime (e.g. `skills/rlm/`). It is LOADED (progressive
+  disclosure), not called: no `ResponseBounds`. Declare `requires=(...)` for a closure over other skills and
+  `skill_runtime` for its intrinsic runtime.
+
+### mcp_tool — a capability exposed over MCP
+- A distinct ARD identity (`<slug>_mcp`) for the same underlying capability exposed as a cross-agent MCP tool.
+  Prefer the generic `build_capability_mcp` (surface 4b); author a bespoke server only for a curated typed
+  signature (the reference pack's are in `packs/contracts/mcp/` and `packs/compliance/mcp/`, e.g.
+  `packs/compliance/mcp/compliance_server.py`). Bind the store server-side (issue 0035) — the tool never takes a tenant/store argument.
+
+## Verify (the guardrail)
+
+Run `uv run pytest tests/capabilities/test_authoring_contract.py tests/capabilities/test_manifests.py
+tests/capabilities/test_registry.py tests/arch/test_import_contracts.py` after authoring. It pins the contract this
+skill teaches: no manifest under a non-canonical slug; the reserved-without-manifest set is a fixed allowlist (so
+adding a slug but forgetting its manifest FAILS here); every manifest kind is a real ARD kind; every cap that declares
+an `impl_ref` is a canonical slug with a manifest of the matching kind. The guardrail does not import the
+`impl_ref` itself: add a test of your own that resolves it with `capability_impl(slug)` and calls it (the reference
+pack's is `tests/spans/test_model_capabilities.py`). If you deliberately add a reserved/internal slug (no manifest),
+add it to `_RESERVED_WITHOUT_MANIFEST` with a one-line reason. The engine's `tests/conftest.py` loads the reference
+pack, so these tests see its slugs.
+
+**The import boundary (engine repo).** `pyproject.toml` `[tool.importlinter]` has two forbidden contracts: every
+generic engine package (`source_modules`: `rag_wright.api`, `capabilities`, `contracts`, `corpus`, `ingestion`,
+`models`, `okf`, `ontology`, `skills`, `spans`, `store`, `subgraphs`, `util`) must never import `rag_wright.packs`;
+and `rag_wright.packs.contracts` must never import `rag_wright.packs.compliance`. A new generic top-level package
+goes in the first contract's `source_modules`; a new module inside an existing package or inside `rag_wright.packs`
+needs no edit. `tests/arch/test_import_contracts.py` enforces both.
+
+**The network guard.** `tests/conftest.py` fails any test that resolves a non-local host unless it carries a live
+marker (`model`, `store`, `parse`, `embed`, `rerank`, `ner`, `fleet`). Mock model and HTTP calls in unit tests, or
+mark the test live.
+
+## Common mistakes
+
+- Adding the slug but forgetting the manifest (slug becomes silently un-discoverable) — the guardrail catches it.
+- An impl_ref factory that reaches env/globals instead of the `WorkspaceHandle` — breaks multi-workspace use; build
+  everything from `resources`.
+- A heavy import at the top of an impl_ref factory module that something light imports (a pack's `register()`
+  module, `rag_wright.api`): it inflates the light index; import inside the factory body.
+- Declaring `response_bounds` on an `agent_skill` (constructing that `CapabilityManifest` raises `ValueError`: the
+  bounds apply only to callable kinds, and a skill is loaded, not called).
+- Inventing a kind. If a capability fits none of the five, flag it — do not force-fit.

rag_wright-0.2.0/.claude/skills/building-an-ingestion-capability/SKILL.md

@@ -0,0 +1,149 @@
+---
+name: building-an-ingestion-capability
+description: >-
+  How to build a NEW domain's ingestion on the RAG_Wright engine with `build_ingestion`: decide what one extraction
+  UNIT is in your documents, write the one required hook (the extractor) and only the optional hooks you need,
+  declare the record types in your pack `.ttl`, tune the default hooks with `evaluate_ingestion` on your own sample
+  documents, and handle spreadsheets, tables and embedded files. Use it when a product or domain pack needs to turn
+  its documents into a cited knowledge graph, or when an ingestion result looks wrong (units too big, tables split,
+  records uncited). Grounded in ADR-0124 and the generated API reference (`docs/api/`).
+---
+
+# Building an ingestion capability
+
+The engine owns the ingestion MECHANISM; a domain supplies one function, the extractor, plus any optional hook whose
+default does not fit. Everything named here is imported from `rag_wright.api` (the generated `docs/api/README.md` has
+every signature). The decision record is ADR-0124 (`docs/adr/0124-generic-ingestion-builder-and-hooks.md`).
+
+## 1. Who owns which stage
+
+| stage | owner | default (engine) | override with |
+|---|---|---|---|
+| parse (PDF, Office, spreadsheets incl. hidden sheets, HTML, Markdown; embedded files) | engine | docling, content-hash cached | (none) |
+| chunk | engine | structural boundaries; a model refines only an over-cap section | `chunk_model=` |
+| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` |
+| tag spans (soft tags) | domain, optional | none | `span_tagger=` |
+| index spans (embed + store, page/bbox provenance) | engine | the workspace's ingest embedder | `embedder=` |
+| group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
+| extract records from a unit | **domain (required)** | (none) | the `extractor` argument |
+| write records | engine default | `kg_write` | `writer=` |
+| per-document follow-up (e.g. an entity graph) | domain, optional | none | `document_hook=` |
+| `Document` node, embedded children (`EmbeddedIn` / `AttachedTo`), progress, dead-lettering | engine | always on | (none) |
+
+Every hook's output is checked by the engine, default or override alike: `check_tiling` (spans tile the chunk text,
+ids `<chunk_id>#<index>`), `check_units` (known spans, each once, in order), `check_extraction` (provenance, below).
+A unit whose extraction fails is recorded in the report and skipped; a document that fails is dead-lettered; the run
+goes on.
+
+## 2. Decide what ONE unit is (before writing code)
+
+The unit is the text one extractor call reads. Get it right first; every other choice follows.
+
+- Ask: "one record in my domain comes from ... ?" A section under a heading (a report, a policy) -> the default
+  grouper already does this. One table row (a register, a test log) -> the default treats a DATABASE-style table
+  (named, distinct header columns plus a serial first column or many columns) as one unit per row, and sets
+  `Unit.table_row` with exact cell values; force it per document with `IngestSource(table_mode="record")`, or keep
+  tables whole with `"block"`. A form (fields of ONE record) -> one unit (`auto` keeps a form grid whole).
+- Units are capped at `IngestionTuning.max_unit_chars` (default 6000); a split table repeats its header row in each
+  continuation unit.
+- If your documents mark units in a way layout does not show (a numbering scheme, a domain heading convention),
+  override `unit_grouper=` or pass a `boundary_decider=` (candidate line texts -> "starts a new unit?" per text) to
+  settle the lines the default grouper is unsure of. Domain conventions belong in your pack, never in the engine.
+
+## 3. Declare your record types in the pack `.ttl`
+
+The store creates only the types a pack declares (ADR-0066: schema lives in the ontology, not in code). A record type
+needs `span_id` and `confidence` properties to carry provenance:
+
+```turtle
+@prefix eng: <https://ragwright.local/ontology/engine#> .
+@prefix my:  <https://example.org/my-domain#> .
+
+my:SectionNode a eng:KgVertexType ; eng:vertexName "Section" ;
+    eng:kgProperty "section_id:STRING", "title:STRING", "span_id:STRING", "confidence:STRING" ;
+    eng:uniqueIndexOn "section_id" .
+```
+
+Point the workspace at it with `EngineConfig(pack="<path to your .ttl>")`; `open_workspace` then creates these types
+on top of the neutral engine types. Writing a type the pack does not declare fails.
+
+## 4. Write the extractor, tune, ingest
+
+The extractor turns one `Unit` into a `UnitExtraction` of `KgNode` / `KgEdge` records. Provenance rule
+(`check_extraction`): a fact node or edge carries `span_id` (a span of THIS unit, e.g. `unit.anchor.span_id`) and
+`confidence` (`EXTRACTED`, `INFERRED` or `AMBIGUOUS`); nodes without `span_id` are shared vocabulary; a non-empty
+extraction must cite at least once. Run `evaluate_ingestion` on your OWN sample documents before trusting the
+defaults; it needs no store and makes no model calls.
+
+```python
+import asyncio
+import os
+
+from rag_wright.api import (
+    EngineConfig, IngestionTuning, IngestSource, KgNode, StoreConfig, UnitExtraction, build_ingestion,
+    evaluate_ingestion, open_workspace,
+)
+
+PACK_TTL = "my_domain/pack.ttl"
+SAMPLES = ["samples/report.pdf"]
+CORPUS = "my_domain"
+CACHE = "data/cache/my_domain"
+
+
+async def extract(unit, *, source_doc_id):
+    """One unit -> its records, each citing a span of the unit."""
+    title = unit.text.strip().splitlines()[0][:200]
+    return UnitExtraction(nodes=[KgNode("Section", "section_id", {
+        "section_id": f"{source_doc_id}:{unit.index}", "title": title,
+        "span_id": unit.anchor.span_id, "confidence": "EXTRACTED"})])
+
+
+tuning = IngestionTuning(max_unit_chars=6000)
+evaluation = evaluate_ingestion(SAMPLES, cache_dir=CACHE, tuning=tuning)
+print("evaluation passed:", evaluation.passed, evaluation.failures)
+
+ws = open_workspace(EngineConfig(store=StoreConfig(
+    host=os.environ["ARCADEDB_HOST"], port=os.environ["ARCADEDB_PORT"],
+    user=os.environ["ARCADEDB_USER"], password=os.environ["ARCADEDB_PASSWORD"]), pack=PACK_TTL), corpus=CORPUS)
+pipeline = build_ingestion(extract, tuning=tuning)
+report = asyncio.run(pipeline.aingest(ws, [IngestSource(path=p) for p in SAMPLES], cache_dir=CACHE))
+print(report.succeeded, "ingested,", report.failed, "dead-lettered")
+for doc in report.documents:
+    print(doc.doc_id, doc.units, "units,", doc.records, "records,", doc.extraction_failures)
+```
+
+Iterate on `evaluation.failures` (and the `DocumentEvaluation` measures: `tables_whole`, `headings_start_units`,
+`coverage`, `cap_ok`, ...) by changing `IngestionTuning` before you override a hook. Keep `extract` async and
+network-bound work inside it; the engine runs up to `IngestionTuning.extract_concurrency` units at once.
+
+## 5. Spreadsheets, tables and embedded files
+
+- **Hidden sheets** are ingested by default; `IngestSource(include_hidden_sheets=False)` skips them (listed in
+  `DocumentReport.skipped_hidden_sheets`).
+- **Exact cell values**: `table_rows(parse_document(...))` returns every data row as a `TableRow` (`columns`,
+  `values`, `cell(name)`), read from the parse's cell grid, whole even when chunking split the table. A per-row unit
+  carries its row in `Unit.table_row`, so the extractor reads cells, not re-parsed text.
+- **Embedded files and PDF attachments** (an Office package's embedded workbook or PDF, a PDF's attached files) are
+  ingested as CHILD documents through the same pipeline: each gets its own `Document` node and an `EmbeddedIn` edge
+  to its parent, and an `AttachedTo` edge from the child to the table-row span it belongs to, with a confidence and
+  the identifier evidence (`IngestionTuning.identifier` sets what counts as an identifier). The report lists them in
+  `DocumentReport.children`, `links` and `unmapped_links`.
+
+## 6. Verify (definition of done)
+
+1. `evaluate_ingestion` passes on a representative sample of YOUR documents (not a hand-picked easy one).
+2. A live ingest of that sample: no dead letters, `extraction_failures` explained, records cite spans
+   (`kg_read(ws, "<your type>", fields=["span_id"])`), and `span_positions(ws, doc_id)` gives the citation positions.
+3. A hermetic test of your extractor on fixed units (no model calls in the default suite; the engine's test network
+   guard fails any unmarked test that reaches the network).
+4. If the extractor calls a model, meter it: run the ingest inside `measure_usage()` and check the call count and
+   cost per document before a bulk run.
+
+## Anti-patterns
+
+- Putting domain rules (section words, abbreviations, value lists) in Python: they belong in your pack `.ttl`.
+- Overriding the segmenter or grouper before `evaluate_ingestion` shows the default fails on your documents.
+- An extractor that returns records without `span_id`, or cites a span outside its unit (the run reports it as an
+  extraction failure, and the records are not written).
+- Re-parsing table text with regexes when `Unit.table_row` / `table_rows` already give the exact cells.
+- Importing engine internals (`rag_wright.ingestion.*`, `rag_wright.store.*`): everything here is on `rag_wright.api`.

rag_wright-0.2.0/.claude/skills/classifier-opportunity-analysis/SKILL.md

@@ -0,0 +1,188 @@
+---
+name: classifier-opportunity-analysis
+description: >-
+  Structured guide for ANALYZING a domain's ingestion + retrieval pipeline to find where an LLM call can be
+  replaced by a deterministic rule, a trained classifier, or a routing decision. Use it BEFORE building or
+  refactoring a domain pack, when an LLM is doing per-unit work that multiplies over a document, or when
+  onboarding a new domain — it is the identification/decision step upstream of `setfit` (which BUILDS the
+  classifier) and `authoring-a-capability` (which REGISTERS it as a capability). It captures the recipe applied
+  twice (contracts, then compliance) so the next domain is mapped the same way instead of re-derived.
+---
+
+# Finding classifier / routing opportunities in a pipeline
+
+This is an **analysis** skill: its output is a decision — an *opportunity list / decomposition plan*, not code and
+not a trained model. It answers "where in this domain's ingestion and retrieval does a classifier, a routing
+decision, or a deterministic rule belong, and where must the LLM stay?" Build what it identifies with the `setfit`
+skill, serve the teacher with `qwen-vllm-modal`, and register each result as a capability with
+`authoring-a-capability`.
+
+## The core pattern (why this works, and why we have done it twice)
+
+A per-unit "do everything" LLM extraction is almost never one decision. It is a **bundle of separable decisions**
+wearing one prompt. Decomposed, most of the bundle is not LLM-shaped work:
+
+- **boundaries** (where does a unit start / is this span worth extracting) are usually structural → deterministic;
+- **closed-vocab tags** (the type, the role, the dimension values) are classification → a classifier, or a
+  deterministic cue-rule when the cues are enumerable;
+- only the **genuinely open part** (numbers, free text, synthesis) needs an LLM, and then only **one residual call
+  per unit**.
+
+Worked precedent in this engine:
+
+| | Contracts (decomposed) | Compliance (the CIC arc) |
+|---|---|---|
+| Unit boundaries | deterministic (section numbering / headings) + a decision-model residue decider (one batched `noul` call for the uncertain lines; ADR-0122) | deterministic operative-rule spans + one `jev_decision` call per span for the operative gate (the default `extraction_backend="jev"`) |
+| Closed-vocab tags | the 21-dim / 29-dim classifier fleet | deontic cue-rule + actor / claim-types in that same per-span Jev call |
+| Open / numeric field | deterministic candidate spans + one Jev `choice` call per provision with candidates (none without); the LLM only as fallback (`RAG_RESIDUAL_EXTRACTOR=llm`). Value recall / precision 0.83 / 0.88 vs the LLM's 0.60 / 0.79 | applicability + evidence standard: one gated residual LLM call, only for a rule with a conditional or evidence cue (most skip it); the `"docling"` fallback keeps one LLM extraction per section |
+| Extraction judge | the Layer-3 judge on Jev: one batched call per provision, 95.3% vs the LLM judge's 90.1% (192 blind hand-labelled cases); `RAG_SEMANTIC_JUDGE=llm` opts out | (none) |
+| Text of the record | verbatim span | verbatim span (was a paraphrase) |
+| Structure / graph extraction | once per document (parties), keep the LLM | once per document, keep the LLM |
+
+Cost shape (live, 131 provisions of one contract): 215 Jev calls, 0 LLM calls, $0.013, against 277 calls (276 of them
+LLM) and $0.277 on the LLM path (ADR-0040, ING-9 / ING-9b addendum). One batched call per unit is the shape to aim for.
+
+The pattern is domain-independent. What changes per domain is the vocabulary and the document structure — which is
+exactly what the phases below make you look at.
+
+## Phase A — Map the pipeline as per-unit decisions
+
+Enumerate every LLM call and, for each, its **unit** and how the unit COUNT scales:
+
+- per **document** (parse, a party/graph-structure pass) — count ≈ corpus size; cheap per doc.
+- per **section / chunk / segment / span** — count scales with **document length**. A 100-page document is
+  thousands of these. **This is where the cost lives and where decomposition pays.**
+- per **query** / per **candidate** / per **(claim, requirement) pair** — query-time; count ≈ traffic, usually a
+  few per request (see Phase D).
+
+Two things to separate immediately:
+
+1. **Structure / connection extraction** (entities, parties, graph edges — "who/what is in this document and how is
+   it connected") is a **once-per-document** pass. It is NOT the per-unit cost; leave the LLM there. Do not mistake
+   it for the thing to decompose. (In this engine that is the docling-graph pass; it runs once per contract and once
+   per regulation.)
+2. **Per-unit semantic tagging** (what IS this unit, what are its typed properties) is the multiplying cost. This is
+   the target.
+
+## Phase B — Classify each decision by its shape, then pick the mechanism
+
+For every per-unit decision, name its shape. The shape dictates the mechanism, in this order of preference (cheapest
+and most robust first):
+
+1. **Boundary / segmentation** — "does a new unit start here?", "is this span operative / extractable?" →
+   **deterministic structure first** (numbering, enumeration `(a)(b)`, headings, list markers). Add a small
+   **binary classifier** only for the residue where structure is ambiguous (the analog of a `is_extractable_span`
+   model). Rarely needs an LLM.
+2. **Closed-vocab with an enumerable cue list** — a value the ontology can map from a fixed set of trigger phrases
+   (e.g. a deontic type from "must / shall / may not") → a **deterministic cue-rule, NO ML**. Do this before
+   training anything; it is free and exact.
+3. **Single-label routing** — one of a closed set, no clean cue list → a **classifier**.
+4. **Multi-label soft-tagging** — several of a closed set, used as *guidance* not a gate → a **soft-tag classifier**
+   (top-k). The most forgiving shape; a modest-accuracy model is still useful because wrong extra tags are cheap.
+5. **Verbatim vs generated text** — if the record just needs the unit's text, extract the **verbatim span**
+   (deterministic) rather than a generated paraphrase. Drops a generative LLM step and is more faithful for
+   citation. Keep a paraphrase only if a human-readable restatement is a real requirement.
+6. **Open / numeric / free-text / synthesis**: no closed set. Often still not LLM work: **propose candidates
+   deterministically** (the numbers, amounts, durations in the unit), then a **decision model labels each
+   candidate's role** in one batched call per unit. Author the roles and their one-line criteria in the pack `.ttl`
+   (the reference pack's `cbr:ResidualRole` + `cbr:decisionCriterion`). Keep the LLM, as **one residual call per
+   unit**, only for values no candidate generator can propose.
+7. **Pair / entailment / verdict**: rerank a candidate against a query, or a judge verdict over a (subject, rule)
+   pair → a **decision model** is a strong candidate (a verdict over a closed label set IS a classification), then
+   a **cross-encoder / NLI classifier**. The ingestion extraction judge moved to a decision model (ING-9). Often
+   query-time; see Phase D.
+
+## Phase C — What to look for in the documents themselves
+
+Signals that make decomposition **feasible** (push toward rules + classifiers):
+
+- explicit **numbering / enumeration / heading** structure → deterministic boundaries;
+- a **closed, ontology-authored vocabulary** for the typed fields → classifiers + cue-rules;
+- **repeated template structure** across documents → stable features;
+- **enumerable linguistic cues** (deontic verbs, defined terms, standard phrasings) → cue-rules.
+
+Signals that **resist** it (keep the LLM): genuinely open / unbounded values, cross-document or multi-hop
+reasoning, long-prose synthesis, values that depend on interpretation rather than surface form.
+
+## Phase D — Ingestion vs retrieval: where the win actually is
+
+- **Ingestion** per-unit work on long documents multiplies into thousands of calls. This is the **biggest, do-first**
+  opportunity. The whole Phase B decomposition applies.
+- **Retrieval / query time** is typically a **few calls per request** (classify the query, extract its constraints,
+  rerank, judge). These ARE classifier/routing shapes (closed-set query routing, a pair-classifier reranker or
+  judge), but the volume is low and you often want the LLM's rationale or synthesis. It is legitimate to **live with
+  the LLM at query time** (as this engine does for the contract and compliance judges) and still decompose
+  ingestion fully. State this as a deliberate choice per decision; do not reflexively de-LLM query time.
+
+## Phase E — Soft-tag vs hard-gate: decide the role before committing
+
+The same classifier is safe or dangerous depending on how its output is used:
+
+- a **soft tag** that only augments / hints → safe even at modest accuracy; wrong extra tags are cheap.
+- a **hard gate** that drops, blocks, or routes irreversibly → needs high accuracy AND a safe fallback.
+
+Decide the role first. Keep a **graceful-degrade path**: a classifier abstention or a persistent rule-miss should
+fall back to the residual LLM (or to an explicit "ambiguous"), never to a silent wrong answer.
+
+## Phase F — Make it ttl-driven, and know what transfers across domains
+
+Author the closed vocabulary and the cue lists in the **ontology (`.ttl`), never in Python** (the engine's
+knowledge-in-the-ontology rule). Then:
+
+- the **deterministic mechanism** (structure split + cue-rule + verbatim extraction) is domain-generic and
+  **transfers to any pack for free** — it reads whatever vocab/cues the pack authors;
+- a **trained classifier is vocabulary-specific** — a new-vocabulary domain pack trains **its own**. "Reuse across
+  products" therefore means the *same mechanism + per-pack models*, not one model everywhere.
+- A pack that only re-routes an existing vocabulary at query time (an override overlay) is NOT a new-vocabulary pack
+  and needs no new ingestion classifier.
+
+## The output: the opportunity list
+
+Produce a decomposition plan, not prose:
+
+1. **Headline the single biggest per-unit LLM cost** (the multiplying ingestion pass).
+2. For **each decision** give: its unit, its shape (Phase B), the chosen mechanism (deterministic / cue-rule /
+   classifier / residual-LLM / verbatim), and its role (soft-tag vs gate).
+3. Separate an **ingestion bucket** (do first) from a **query-time bucket** (decide case by case; often live with
+   the LLM).
+4. Note the **ttl + per-pack** generality (what transfers, what each pack re-trains).
+5. **Map each decision onto an ingestion hook** (`build_ingestion`, ADR-0124): a boundary decision →
+   `BoundaryDecider` / `UnitGrouper`; closed-vocab tags → `SpanTagger`; residual values and the judge → `Extractor`.
+6. Exclude anything that is not actually a per-unit cost (once-per-document structure extraction) and anything
+   already settled (a decision an existing cue-rule covers).
+
+## Hand-off (what to do with the opportunities)
+
+- **Deterministic rule / cue-rule / span-split** → plain code in an ingestion hook (`Segmenter`, `UnitGrouper`,
+  `SpanTagger`, passed to `build_ingestion`). NOT a capability; it is mechanism, and the knowledge it reads lives in the `.ttl`.
+- **System-1 decision model (NO training)** → for a closed-set decision (yes/no, choice, score), A/B a decision
+  model — **Jev** (managed, OpenRouter Decisions API, zero/few-shot, calibrated) or **Laya** (open, fine-tuned) —
+  BEFORE committing to a trained classifier. It often wins when data is scarce or label-ambiguous, or when you need
+  calibrated uncertainty to route/gate (measured: RAG_Wright CIC-1c — Jev zero-shot 0.92 vs a trained SetFit 0.82;
+  ADR-0119). See `setfit` Phase 0.5 (the decision-vs-train A/B) and the `laya` skill; wire it as a `jev_decision`-style
+  model capability with a `DecisionModelProfile`. **Evaluate this first; it may remove the need to train at all.**
+  The decision path runs only when `jev_decision` is registered (from `engine_capabilities()`) AND
+  `OPENROUTER_API_KEY` is set; otherwise the reference pack's decision paths degrade silently (to the deterministic
+  rule or the LLM), so check both before reading a number.
+- **Trained classifier** → build it with the **`setfit`** skill (framing, symmetric leakage-safe eval, per-class
+  floor, soft-tag/top-k, rare-class curation, checkpointing). Serve the teacher / bulk-labeler with **`qwen-vllm-modal`**.
+  Then register it as a capability with **`authoring-a-capability`**: a `kind="model"` capability with an
+  `impl_ref` factory `def <slug>(resources, inputs)` over a cached checkpoint, invoked by name through the engine
+  API: from your ingestion hook, call `ainvoke_model(slug, inputs, resources=ws)`, **routed THROUGH the
+  capability layer, never hand-constructed around it.** A model impl may be sync (a classifier / XGBoost — run
+  off-loop by `ainvoke_model`) or async (an LLM-backed cap — awaited by `ainvoke_model`).
+
+## Anti-patterns (from the real sessions — do not repeat)
+
+- **Letting classifier training cost/time decide whether an opportunity exists.** Identify opportunities by the
+  *pattern* (per-unit closed-set decision on a long document); training ROI is a separate, later question.
+- **Mistaking once-per-document structure extraction for the per-unit cost.** It is cheap; leave the LLM.
+- **Claiming a judge/verdict step "can't classify."** A verdict over a closed label set (compliant / violation /
+  needs-review, relevant / not) IS a classification; it is a legitimate pair-classifier candidate (usually
+  query-time).
+- **Hardcoding the vocabulary or cues in Python.** They belong in the `.ttl`; the mechanism reads them.
+- **Training a classifier where an enumerable cue-rule already settles the decision** (the deontic-type case).
+- **Forcing a hard gate where a soft tag suffices** — it imposes an accuracy bar you did not need.
+- **Reflexively de-LLM'ing query time** — low volume + wanted rationale often make the LLM the right call there.
+- **A router/cascade of specialists** — it multiplies errors (router acc × specialist acc). A flat classifier +
+  multi-tag usually beats it (see `setfit`).

rag_wright-0.2.0/.claude/skills/creating-evals/SKILL.md

@@ -0,0 +1,146 @@
+---
+name: creating-evals
+description: >-
+  Domain-agnostic recipe for creating EVALS for engine/product capabilities, eval-first (TDD): write the eval as
+  soon as a capability is DEFINED (its contract + acceptance criterion), before it is implemented. Use it when
+  starting a new domain (right after capabilities are defined), when adding or changing a capability, or when A/B-ing
+  alternatives (e.g. a trained classifier vs a System-1 decision model). Covers gold-set design + reliability,
+  per-capability-KIND metrics (extraction / classification / retrieval / graph / generation / judgment), the
+  gate-vs-diagnostic split, building the gold cheaply, an isolated executable harness, and optional Langfuse
+  Datasets/Experiments/Scores automation. An eval is the executable acceptance criterion; training data IS an eval.
+---
+
+# Creating evals (eval-first / TDD)
+
+An eval is the **executable acceptance criterion** for a capability. Write it **as soon as the capability is
+DEFINED** — its contract (typed in/out) and its acceptance criterion exist — **before it is implemented**. This is
+TDD at the capability level: the eval fails (red) on the unbuilt/weak capability, you implement to green, then you
+can A/B alternatives and catch regressions forever. Corollary observed repeatedly in this engine: **the training
+data you build for a classifier/decision model IS an eval** (a labeled gold set + a metric) — so building the eval
+first also gives you the data design for free.
+
+## When to use
+- **Starting a new domain**: the FIRST build step after capabilities are defined (step 5 of the domain-adaptation guide, `docs/domain-adaptation/README.md`) — write
+  each capability's eval before/while you implement it.
+- Adding or changing a capability, or tuning a threshold/prompt/model.
+- **A/B-ing alternatives** on one capability (a deterministic rule vs a trained classifier vs a System-1 decision
+  model vs an LLM) — the eval is the neutral judge; select on the metric.
+
+## 1. Design the gold set (the foundation — get this right FIRST)
+- **Real, in-domain items + expected outputs/labels.** Small but REPRESENTATIVE; never the easy cases only.
+- **Reproducible + pinned + gitignored.** The gold is a generated artifact: pin its source snapshot + the selected
+  ids so it rebuilds identically; gitignore the data, commit the BUILDER. (Pattern: `eval/build_golden.py`.)
+- **Leakage-safe + balanced.** Split by the natural grouping unit (document / source / record), not by row.
+  For classification, a SYMMETRIC per-class test and a per-class FLOOR (not overall accuracy — it hides dead
+  classes). "k-shot" = k per class.
+- **The gold is the CEILING — measure its RELIABILITY.** For subjective/ambiguous labels, get a second
+  independent labeling and report inter-annotator (or inter-pass) agreement; adjudicate the disagreements and
+  document the calls. A model cannot beat the gold's own consistency, and label ambiguity in the gold shows up as a
+  classifier ceiling you cannot train past (CIC-1c: a two-pass gold agreed at 0.905 and a trained SetFit capped
+  ~0.82 — diagnose the gold before blaming the model). If you have no human expert yet, a documented rubric + a
+  two-pass consensus is
+  the honest proxy — say so.
+- **Validate silver against a small BLIND hand-labelled sample before treating it as gold.** Label the sample
+  without seeing any model output, then measure silver-vs-hand agreement. If it is low, the hand-labelled sample IS
+  the gold. (ING-9: silver judge labels derived from classifier test sets agreed only 64% with hand labels; the
+  decision rested on 192 blind hand-labelled cases. Pattern: `eval/semantic_judge_gold.py --score-blind`.)
+- **Separate a tuning set from a held-out set.** Label the held-out set BEFORE any prompt runs on it, tune only on
+  the tuning set, and report both numbers (ADR-0122 boundary residue prompt: 94.5% held-out vs 96.2% tuning). A
+  number measured only on the set you tuned on is not a result.
+- **Gold stays local when the corpus is restrictively licensed** (CUAD/ACORD-derived gold lives under the
+  gitignored `data/eval/`); commit the builder and the scorer, never the data.
+
+## 2. Pick the metric by capability KIND, and split GATE vs DIAGNOSTIC
+Always set ONE pass/fail **gate** (from the acceptance criterion) and report **diagnostics** alongside (never gate
+on a diagnostic).
+- **Extraction** (section → records, clauses, requirements): recall / precision / F1 of extracted items vs gold,
+  reported SEPARATELY (under- vs over-extraction are different failures). Watch over-extraction on non-operative
+  input (definitions) and under-extraction on long input.
+- **Classification / typed decision** (incl. SetFit, Laya, **Jev**): per-class recall + the per-class **floor**;
+  symmetric eval; top-k recall for multi-label (reported against tags-per-item). Prefer a soft-tag/calibrated
+  metric when the output routes rather than hard-gates.
+- **Retrieval**: recall@k (binary, relevant = grade ≥ a floor) as the GATE; nDCG@k (graded, exp gain) as a
+  DIAGNOSTIC (do NOT threshold nDCG). Isolate the retrieval legs (score corpus-ids before rehydration). Pattern:
+  `eval/acord_retrieval.py`.
+- **Graph / relational**: recall over the answer SET (reachability/traversal), k large enough to cover the set;
+  node ids match gold by construction. Pattern: `eval/relational_eval.py`.
+- **Generation / QA**: citation-recall / groundedness / correct-abstention; LLM-as-judge for free-text, but anchor
+  with deterministic checks (a cited span must exist). Generation is non-deterministic near the abstain boundary —
+  measure over several runs.
+- **Judgment / compliance verdicts**: report PRECISION and RECALL of the actionable class (e.g. violation)
+  SEPARATELY (alert-fatigue vs missed), and break out by provenance (real vs constructed). Pattern:
+  `scripts/eval_compliance_gold.py`. For a judge that accepts or refutes other outputs, report the **error-catch
+  rate** (wrong outputs refuted), the **false-refute rate** (correct outputs refuted) and **calibration** (do its
+  scores mean what they say), not one accuracy number. Pattern: `eval/semantic_judge_gold.py` (ING-9: decision
+  model 95.3% vs LLM 90.1% on 192 blind hand-labelled cases).
+- **Candidate-then-choose pipelines** (a generator proposes, a model picks): measure the generator's **coverage**
+  separately, because it caps recall (ING-9b residual values: candidate coverage 95%; value-level recall 0.83 /
+  precision 0.88 vs the LLM's 0.60 / 0.79. Pattern: `eval/residual_decision_gold.py`). **Count is not accuracy**:
+  emitting more values is not a win without precision.
+- **Ingestion structure**: `evaluate_ingestion` (in `rag_wright.api`) is the packaged structural eval of the
+  ingestion hooks (tiling, table-row integrity, layout respect, coverage) on your own sample documents. Gate
+  patterns: `eval/segmenter_eval.py`, `eval/unit_grouper_eval.py`.
+
+## 3. Build the gold cheaply (without faking it)
+- **Silver bootstrapping**: a higher-capability teacher (LLM, or a decision model) labels candidate items; CURATE
+  with reject-rules; mark silver, never conflate with gold. (Teacher labeling runs on the flat-GPU substrate per
+  `qwen-vllm-modal`, not ad-hoc paid calls.)
+- **Real public datasets** where they exist (e.g. labeled corpora for the task) — add to TRAIN; keep the TEST
+  in-corpus + human-adjudicated so it stays an honest transfer test.
+- **Generate hard cases** (near-boundary positives/negatives) to stress the exact confusions — TRAIN-ONLY; the
+  gold test stays real. (Full data playbook: the `setfit` skill Phase 1/4 + `classifier-opportunity-analysis`.)
+
+## 4. Make it an executable, isolated harness
+- **Isolate the capability under test**: inject it (a seam / `extract_override` / an injected `retrieve`) so the
+  eval measures ONE capability, not the whole pipeline. Invoke production code THROUGH the capability layer
+  (`ainvoke_subgraph`/`ainvoke_model`), never a hand-built copy. Score the **shipped request** (the exact
+  prompt/question builder production uses, e.g. `judge_request`), not a copy of it re-typed in the eval.
+- **Repeatability for non-deterministic models.** Run each case several times on identical input; report how many
+  answers flip and how far the scores sit from the threshold. Flips cluster near the threshold and usually mean
+  the QUESTION is ambiguous: fix the question, do not vote it away (Jev flipped 5/181 answers at scores 0.47-0.56;
+  temperature/seed did not help, majority voting barely helped, rewriting the ambiguous question did. The ADR-0122
+  boundary residue prompt went from 94-96% with flips to 460/461, zero flips across 3 calls). Pattern:
+  `eval/boundary_residue_gold.py`.
+- **Cache keys include the prompt and the method.** Any decision or extraction cache must key on the prompt text
+  and the method (decision model vs LLM, and which model), so an eval never reuses outputs a different prompt or
+  method produced (ADR-0122 ING-4d / ADR-0040 ING-9b: the residue decision cache and the clause cache).
+- **Score from a RESULT ARTIFACT (JSON), not stdout scraping** (scraping truncates and silently drops rows).
+- **Parallelize** model/LLM calls (async + semaphore) — same cost, far less wall-clock; order-preserving so it
+  stays deterministic.
+- **Env-selected** so the SAME harness runs local (dev) or on Modal (full corpus + GPU).
+- **Pre-flight paid bulk** (>~50 paid calls): print the exact count + cost and wait (`warn-before-bulk` rule).
+- Stream `X/N` progress + actively monitor any run over ~30s (never launch-and-forget).
+- **Hermetic tests never touch the network**: `tests/conftest.py` fails a test that resolves a non-local host unless
+  it carries a live marker (`model`, `store`, ...). Live evals run as `uv run python -u eval/<name>.py` (outside
+  pytest) or carry a live marker.
+
+## 5. Langfuse — optional eval automation (we already use it for tracing)
+Langfuse has a first-class eval stack we are NOT yet using (we use it only for spans/usage today): a **Dataset**
+(items = input + optional expected output) → a **Task** (your capability) run over the dataset as an **Experiment
+Run** → **Evaluators** (deterministic checks or LLM-as-judge) producing **Scores** (numeric/categorical/boolean),
+all linked to traces. Reach for it when you want **tracked, re-runnable, dashboarded** evals + regression tracking
+across capability versions; a local JSON harness is enough for a quick one-off gate. Keep the gold BUILDER + a
+committed snapshot in the repo (reproducibility); push the items to the Langfuse dataset so runs and scores land
+next to the traces you already collect. Ground the exact Dataset/Experiment API before wiring
+(https://langfuse.com/docs/evaluation/concepts, https://langfuse.com/docs/datasets/overview); confirm the installed
+SDK version's surface, not prose.
+
+## 6. The TDD loop
+1. Write the eval from the contract + a handful of real gold items → it fails (red) on the unbuilt/weak capability.
+2. Implement the minimum to pass the gate (green).
+3. A/B alternatives (rule / trained classifier / decision model / LLM) on the SAME gold; select on gate + diagnostics + cost + calibration.
+4. Keep the eval; it is the regression guard (the Beyonce rule — if you shipped it, it has an eval).
+
+## Anti-patterns (seen, do not repeat)
+- **Overall accuracy** instead of a per-class floor — hides dead classes.
+- **Testing on teacher/generated data as if it were gold** — measures mimicry, not accuracy; keep the test real + adjudicated.
+- **Gating on a diagnostic** (e.g. nDCG) — report it, don't threshold it.
+- **No reliability check on a subjective gold** — you can't read a number off a gold whose own labels disagree.
+- **Eval not isolated** to the capability (measures the whole pipeline) — you can't attribute a regression.
+- **Scraping stdout** for metrics — write + read a JSON artifact.
+- **Faking a win by shrinking the sample / picking easy items** — real, representative, honest.
+
+## Hand-off / where this sits
+Chain: `classifier-opportunity-analysis` (identify the decision) → **`creating-evals` (THIS — write the eval FIRST)**
+→ build it (a deterministic rule, or `setfit`/`laya`/Jev for a decision, or an LLM) → `authoring-a-capability`
+(register). In the new-domain build sequence, this is the step right after capabilities are defined.