rag-wright 0.1.0__tar.gz

rag_wright-0.1.0/.claude/settings.json

@@ -0,0 +1,14 @@
+{
+  "hooks": {
+    "SessionStart": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "bash \"$CLAUDE_PROJECT_DIR/scripts/graph_status.sh\""
+          }
+        ]
+      }
+    ]
+  }
+}

rag_wright-0.1.0/.claude/skills/authoring-a-capability/SKILL.md

@@ -0,0 +1,106 @@
+---
+name: authoring-a-capability
+description: >-
+  How to author a new RAG_Wright engine capability of any kind (subgraph, function, model, agent_skill, mcp_tool)
+  so it is registered, ARD-discoverable, and invokable by name through the engine API. Use it whenever you add a
+  new capability or a new-domain product/graph needs one: it gives the shared registration + ARD + invocation
+  contract (the four surfaces + the definition of done), the per-kind implementation specifics, and the
+  conformance guardrail that keeps the catalog honest. Grounded against the real code; keep it in step with it.
+---
+
+# Authoring a capability
+
+A **capability** is a named, ARD-registered unit of engine behavior (FR-C). Every capability has a `kind`
+(`ard.py::EntryKind`): `subgraph | function | model | agent_skill | mcp_tool` (`dagster_asset` is reserved). The
+contract below is the SAME for every kind; only the implementation differs. Capabilities compose — a subgraph calls
+functions/models; a product invokes a capability by name through `rag_wright.api`.
+
+**Ground every call before writing it** (CLAUDE.md library rule). The authoritative sources this skill summarizes:
+`capabilities/registry.py` (the canonical-slug set + `CapabilityRegistry.register`), `capabilities/manifests.py`
+(`CapabilityManifest` + `_SPECS`/`MANIFEST_SPECS`), `capabilities/ard.py` (`EntryKind`, `MEDIA_TYPE_BY_KIND`,
+`CALLABLE_KINDS`), `scripts/publish_manifests.py`, `api/invoke.py` + `capabilities/invoke.py::capability_impl` (the adapter-free impl_ref invoker + drift guard),
+`api/mcp.py` (generic MCP exposure). The guardrail test is `tests/capabilities/test_authoring_contract.py`.
+
+## The four surfaces (the definition of done)
+
+A finished capability touches these: 1 (implementation) is for EVERY kind; 2 (the invoke factory + `impl_ref`) is
+for invokable kinds (`subgraph`/`model`); 3 (manifest + register) is for every discoverable kind; 4 (invocable +
+MCP) follows automatically for invokable kinds; 4b (a bespoke MCP server) is optional.
+
+1. **Implementation** — the real code, in that kind's home (see per-kind below).
+2. **The invoke factory + `impl_ref`** (invokable kinds: subgraph/model) — write a co-located
+   `async def ainvoke(resources, inputs)` (subgraph) / `def <name>(resources, inputs)` (model) in the capability's
+   own module (model factories ignore `resources`; they build over the opaque `WorkspaceHandle` — `resources._store`,
+   `resources.model_id(role)`, never env). The manifest's `impl_ref="module:attr"` points to it. The invoker imports
+   it LAZILY and calls it — there is **NO central adapter dict** (EP-CORE-2). A plain function/agent_skill/mcp_tool
+   declares no `impl_ref`.
+3. **ARD manifest + register it** — a `CapabilityManifest(slug, kind, display_name, description,
+   representative_queries=(2-5…), tags=…, impl_ref=…)`. **The catalog ships EMPTY (EP-CORE-3):** call
+   `register_capability(manifest)` at runtime to add it (a product registers its own; the engine's reference pack is
+   opt-in via `load_reference_pack()`). `representative_queries` is the field ARD discovery ranks on — write real,
+   specific queries. For the ENGINE's reference pack, the manifest is committed in `manifests.py::_SPECS` and the
+   slug is in `CANONICAL_CAPABILITY_SLUGS`; a downstream product registers freely (its slugs need not be canonical).
+   Publish to `~/.air/registry` (what GraphWright's store loads) with `uv run python scripts/publish_manifests.py`.
+   Callable kinds get `ResponseBounds` (defaulted); `agent_skill` must NOT declare bounds (loaded, not called).
+4. **Invocable + MCP for free** — once registered with an `impl_ref`, the capability is callable as
+   `ainvoke_subgraph(slug, inputs, resources=ws)` / `invoke_model(slug, inputs, resources=ws)` — the invoker resolves
+   the `impl_ref` via `capabilities.invoke.capability_impl` (the drift guard asserts it resolves to a callable of the
+   declared kind) — AND exposable over MCP (surface 4b), with **zero engine edits**.
+
+4b. **MCP exposure** (optional) — any invokable capability is already an MCP tool with zero extra code via
+   `api/mcp.py::build_capability_mcp(slug, resources=ws)` (EP-RT-2). Write a bespoke `mcp/<slug>_server.py` only
+   when you want a CURATED, typed tool signature instead of the generic opaque-`inputs` surface.
+
+## Per-kind specifics
+
+### subgraph — a compiled LangGraph `StateGraph`
+- **Home:** `subgraphs/<slug>.py`. A `production_<slug>(*, store, ...) -> CompiledGraph` builder: `g = StateGraph(_State)`,
+  add nodes/edges with `START`/`END`, `return g.compile()`. Nodes call functions/models (compose).
+- **Invoke:** the co-located `async def ainvoke(resources, inputs)` factory (impl_ref target) builds + awaits the graph.
+- Retry/dead-letter come from the graph scaffold, not the invoker. The output contract is the registered `contract`.
+
+### function — a plain, typed callable
+- **Home:** `capabilities/<slug>.py`. A deterministic or model-backed callable with a Pydantic in/out contract.
+- Invoker adapters for `function` are not wired yet (EP-API-2c); until then functions are composed inside
+  subgraphs, not invoked standalone through the API. Still register + manifest it.
+
+### model — a trained checkpoint behind a seam
+- **Home:** `spans/` or `capabilities/` wrapping the checkpoint (e.g. the SetFit clause classifier; the 29-dim
+  property fleet via `spans/property_extractor.py`). Load the checkpoint ONCE and cache it (the fleet is heavy) —
+  see `spans/model_capabilities.py::_dim_registry`.
+- Serve behind the existing seam/adapter so nothing upstream changes (to FIND where a model cap belongs, use the
+  `classifier-opportunity-analysis` skill; to BUILD/train + checkpoint + serve it, the `setfit` skill). The impl_ref
+  factory is `def <slug>(resources, inputs)` for a SYNC impl (CPU-bound local inference — a classifier/XGBoost
+  checkpoint; `resources` ignored) or `async def <slug>(resources, inputs)` for an ASYNC impl (I/O-bound — an
+  LLM-backed model cap calling OpenRouter / a local vLLM client). `invoke_model` runs a sync impl and REFUSES an
+  async one; `ainvoke_model` (EP-API-7) off-loads a sync impl with `asyncio.to_thread` and awaits an async impl
+  directly, with an optional `sem` for fan-out backpressure.
+
+### agent_skill — authored SKILL.md + the Deep Agents runtime
+- **Home:** `skills/<slug>/SKILL.md` + the agent runtime (e.g. `skills/rlm/`). It is LOADED (progressive
+  disclosure), not called: no `ResponseBounds`. Declare `requires=(...)` for a closure over other skills and
+  `skill_runtime` for its intrinsic runtime.
+
+### mcp_tool — a capability exposed over MCP
+- A distinct ARD identity (`<slug>_mcp`) for the same underlying capability exposed as a cross-agent MCP tool.
+  Prefer the generic `build_capability_mcp` (surface 4b); author a bespoke `mcp/<slug>_server.py` only for a
+  curated typed signature. Bind the store server-side (issue 0035) — the tool never takes a tenant/store argument.
+
+## Verify (the guardrail)
+
+Run `uv run pytest tests/capabilities/test_authoring_contract.py tests/capabilities/test_manifests.py
+tests/capabilities/test_registry.py` after authoring. It pins the contract this skill teaches: no manifest under a
+non-canonical slug; the reserved-without-manifest set is a fixed allowlist (so adding a slug but forgetting its
+manifest FAILS here); every manifest kind is a real ARD kind; every invokable cap's impl_ref resolves to a callable whose
+manifest declares the matching kind. If you deliberately add a reserved/internal slug (no manifest), add it to
+`_RESERVED_WITHOUT_MANIFEST` with a one-line reason.
+
+## Common mistakes
+
+- Adding the slug but forgetting the manifest (slug becomes silently un-discoverable) — the guardrail catches it.
+- An impl_ref factory that reaches env/globals instead of the `WorkspaceHandle` — breaks multi-workspace use; build
+  everything from `h`.
+- A non-lazy import at the top of an adapter — inflates the light index / `import rag_wright.api`; import inside
+  the adapter body.
+- Declaring `response_bounds` on an `agent_skill`, or omitting the output `contract` on registration.
+- Inventing a kind. If a capability fits none of the five, flag it — do not force-fit.

rag_wright-0.1.0/.claude/skills/classifier-opportunity-analysis/SKILL.md

@@ -0,0 +1,175 @@
+---
+name: classifier-opportunity-analysis
+description: >-
+  Structured guide for ANALYZING a domain's ingestion + retrieval pipeline to find where an LLM call can be
+  replaced by a deterministic rule, a trained classifier, or a routing decision. Use it BEFORE building or
+  refactoring a domain pack, when an LLM is doing per-unit work that multiplies over a document, or when
+  onboarding a new domain — it is the identification/decision step upstream of `setfit` (which BUILDS the
+  classifier) and `authoring-a-capability` (which REGISTERS it as a capability). It captures the recipe applied
+  twice (contracts, then compliance) so the next domain is mapped the same way instead of re-derived.
+---
+
+# Finding classifier / routing opportunities in a pipeline
+
+This is an **analysis** skill: its output is a decision — an *opportunity list / decomposition plan*, not code and
+not a trained model. It answers "where in this domain's ingestion and retrieval does a classifier, a routing
+decision, or a deterministic rule belong, and where must the LLM stay?" Build what it identifies with the `setfit`
+skill, serve the teacher with `qwen-vllm-modal`, and register each result as a capability with
+`authoring-a-capability`.
+
+## The core pattern (why this works, and why we have done it twice)
+
+A per-unit "do everything" LLM extraction is almost never one decision. It is a **bundle of separable decisions**
+wearing one prompt. Decomposed, most of the bundle is not LLM-shaped work:
+
+- **boundaries** (where does a unit start / is this span worth extracting) are usually structural → deterministic;
+- **closed-vocab tags** (the type, the role, the dimension values) are classification → a classifier, or a
+  deterministic cue-rule when the cues are enumerable;
+- only the **genuinely open part** (numbers, free text, synthesis) needs an LLM, and then only **one residual call
+  per unit**.
+
+Worked precedent in this engine:
+
+| | Contracts (decomposed) | Compliance (the CIC arc) |
+|---|---|---|
+| Unit boundaries | deterministic (section numbering / headings) + a soft type classifier | deterministic sub-section split + a span-level operative-cue gate |
+| Closed-vocab tags | the 21-dim / 29-dim classifier fleet | deontic cue-rule + actor / claim-type / applicability classifiers |
+| Open / numeric field | 1 residual LLM call / provision (7 fields) | 1 residual LLM call / section (evidence standard) |
+| Text of the record | verbatim span | verbatim span (was a paraphrase) |
+| Structure / graph extraction | once per document (parties), keep the LLM | once per document, keep the LLM |
+
+The pattern is domain-independent. What changes per domain is the vocabulary and the document structure — which is
+exactly what the phases below make you look at.
+
+## Phase A — Map the pipeline as per-unit decisions
+
+Enumerate every LLM call and, for each, its **unit** and how the unit COUNT scales:
+
+- per **document** (parse, a party/graph-structure pass) — count ≈ corpus size; cheap per doc.
+- per **section / chunk / segment / span** — count scales with **document length**. A 100-page document is
+  thousands of these. **This is where the cost lives and where decomposition pays.**
+- per **query** / per **candidate** / per **(claim, requirement) pair** — query-time; count ≈ traffic, usually a
+  few per request (see Phase D).
+
+Two things to separate immediately:
+
+1. **Structure / connection extraction** (entities, parties, graph edges — "who/what is in this document and how is
+   it connected") is a **once-per-document** pass. It is NOT the per-unit cost; leave the LLM there. Do not mistake
+   it for the thing to decompose. (In this engine that is the docling-graph pass; it runs once per contract and once
+   per regulation.)
+2. **Per-unit semantic tagging** (what IS this unit, what are its typed properties) is the multiplying cost. This is
+   the target.
+
+## Phase B — Classify each decision by its shape, then pick the mechanism
+
+For every per-unit decision, name its shape. The shape dictates the mechanism, in this order of preference (cheapest
+and most robust first):
+
+1. **Boundary / segmentation** — "does a new unit start here?", "is this span operative / extractable?" →
+   **deterministic structure first** (numbering, enumeration `(a)(b)`, headings, list markers). Add a small
+   **binary classifier** only for the residue where structure is ambiguous (the analog of a `is_extractable_span`
+   model). Rarely needs an LLM.
+2. **Closed-vocab with an enumerable cue list** — a value the ontology can map from a fixed set of trigger phrases
+   (e.g. a deontic type from "must / shall / may not") → a **deterministic cue-rule, NO ML**. Do this before
+   training anything; it is free and exact.
+3. **Single-label routing** — one of a closed set, no clean cue list → a **classifier**.
+4. **Multi-label soft-tagging** — several of a closed set, used as *guidance* not a gate → a **soft-tag classifier**
+   (top-k). The most forgiving shape; a modest-accuracy model is still useful because wrong extra tags are cheap.
+5. **Verbatim vs generated text** — if the record just needs the unit's text, extract the **verbatim span**
+   (deterministic) rather than a generated paraphrase. Drops a generative LLM step and is more faithful for
+   citation. Keep a paraphrase only if a human-readable restatement is a real requirement.
+6. **Open / numeric / free-text / synthesis** — no closed set → keep the LLM, but reduce it to **one residual call
+   per unit** carrying only the fields that genuinely need it.
+7. **Pair / entailment** — rerank a candidate against a query, or a judge verdict over a (subject, rule) pair → a
+   **cross-encoder / NLI classifier** is a candidate (a verdict over a closed label set IS a classification). Often
+   query-time; see Phase D.
+
+## Phase C — What to look for in the documents themselves
+
+Signals that make decomposition **feasible** (push toward rules + classifiers):
+
+- explicit **numbering / enumeration / heading** structure → deterministic boundaries;
+- a **closed, ontology-authored vocabulary** for the typed fields → classifiers + cue-rules;
+- **repeated template structure** across documents → stable features;
+- **enumerable linguistic cues** (deontic verbs, defined terms, standard phrasings) → cue-rules.
+
+Signals that **resist** it (keep the LLM): genuinely open / unbounded values, cross-document or multi-hop
+reasoning, long-prose synthesis, values that depend on interpretation rather than surface form.
+
+## Phase D — Ingestion vs retrieval: where the win actually is
+
+- **Ingestion** per-unit work on long documents multiplies into thousands of calls. This is the **biggest, do-first**
+  opportunity. The whole Phase B decomposition applies.
+- **Retrieval / query time** is typically a **few calls per request** (classify the query, extract its constraints,
+  rerank, judge). These ARE classifier/routing shapes (closed-set query routing, a pair-classifier reranker or
+  judge), but the volume is low and you often want the LLM's rationale or synthesis. It is legitimate to **live with
+  the LLM at query time** (as this engine does for the contract and compliance judges) and still decompose
+  ingestion fully. State this as a deliberate choice per decision; do not reflexively de-LLM query time.
+
+## Phase E — Soft-tag vs hard-gate: decide the role before committing
+
+The same classifier is safe or dangerous depending on how its output is used:
+
+- a **soft tag** that only augments / hints → safe even at modest accuracy; wrong extra tags are cheap.
+- a **hard gate** that drops, blocks, or routes irreversibly → needs high accuracy AND a safe fallback.
+
+Decide the role first. Keep a **graceful-degrade path**: a classifier abstention or a persistent rule-miss should
+fall back to the residual LLM (or to an explicit "ambiguous"), never to a silent wrong answer.
+
+## Phase F — Make it ttl-driven, and know what transfers across domains
+
+Author the closed vocabulary and the cue lists in the **ontology (`.ttl`), never in Python** (the engine's
+knowledge-in-the-ontology rule). Then:
+
+- the **deterministic mechanism** (structure split + cue-rule + verbatim extraction) is domain-generic and
+  **transfers to any pack for free** — it reads whatever vocab/cues the pack authors;
+- a **trained classifier is vocabulary-specific** — a new-vocabulary domain pack trains **its own**. "Reuse across
+  products" therefore means the *same mechanism + per-pack models*, not one model everywhere.
+- A pack that only re-routes an existing vocabulary at query time (an override overlay) is NOT a new-vocabulary pack
+  and needs no new ingestion classifier.
+
+## The output: the opportunity list
+
+Produce a decomposition plan, not prose:
+
+1. **Headline the single biggest per-unit LLM cost** (the multiplying ingestion pass).
+2. For **each decision** give: its unit, its shape (Phase B), the chosen mechanism (deterministic / cue-rule /
+   classifier / residual-LLM / verbatim), and its role (soft-tag vs gate).
+3. Separate an **ingestion bucket** (do first) from a **query-time bucket** (decide case by case; often live with
+   the LLM).
+4. Note the **ttl + per-pack** generality (what transfers, what each pack re-trains).
+5. Exclude anything that is not actually a per-unit cost (once-per-document structure extraction) and anything
+   already settled (a decision an existing cue-rule covers).
+
+## Hand-off (what to do with the opportunities)
+
+- **Deterministic rule / cue-rule / span-split** → plain code inside the ingestion subgraph. NOT a capability; it is
+  mechanism, and the knowledge it reads lives in the `.ttl`.
+- **System-1 decision model (NO training)** → for a closed-set decision (yes/no, choice, score), A/B a decision
+  model — **Jev** (managed, OpenRouter Decisions API, zero/few-shot, calibrated) or **Laya** (open, fine-tuned) —
+  BEFORE committing to a trained classifier. It often wins when data is scarce or label-ambiguous, or when you need
+  calibrated uncertainty to route/gate (measured: RAG_Wright CIC-1c — Jev zero-shot 0.92 vs a trained SetFit 0.82;
+  ADR-0119). See `setfit` Phase 0.5 (the decision-vs-train A/B) and the `laya` skill; wire it as a `jev_decision`-style
+  model capability with a `DecisionModelProfile`. **Evaluate this first; it may remove the need to train at all.**
+- **Trained classifier** → build it with the **`setfit`** skill (framing, symmetric leakage-safe eval, per-class
+  floor, soft-tag/top-k, rare-class curation, checkpointing). Serve the teacher / bulk-labeler with **`qwen-vllm-modal`**.
+  Then register it as a capability with **`authoring-a-capability`**: a `kind="model"` capability with an
+  `impl_ref` factory `def <slug>(resources, inputs)` over a cached checkpoint, invoked by name through the engine
+  API (`invoke_model` / `ainvoke_model`) and the pipeline (`dispatch_model` / `adispatch_model`) — **routed THROUGH
+  the capability layer, never hand-constructed around it.** A model impl may be sync (a classifier / XGBoost — run
+  off-loop by `ainvoke_model`) or async (an LLM-backed cap — awaited by `ainvoke_model`).
+
+## Anti-patterns (from the real sessions — do not repeat)
+
+- **Letting classifier training cost/time decide whether an opportunity exists.** Identify opportunities by the
+  *pattern* (per-unit closed-set decision on a long document); training ROI is a separate, later question.
+- **Mistaking once-per-document structure extraction for the per-unit cost.** It is cheap; leave the LLM.
+- **Claiming a judge/verdict step "can't classify."** A verdict over a closed label set (compliant / violation /
+  needs-review, relevant / not) IS a classification; it is a legitimate pair-classifier candidate (usually
+  query-time).
+- **Hardcoding the vocabulary or cues in Python.** They belong in the `.ttl`; the mechanism reads them.
+- **Training a classifier where an enumerable cue-rule already settles the decision** (the deontic-type case).
+- **Forcing a hard gate where a soft tag suffices** — it imposes an accuracy bar you did not need.
+- **Reflexively de-LLM'ing query time** — low volume + wanted rationale often make the LLM the right call there.
+- **A router/cascade of specialists** — it multiplies errors (router acc × specialist acc). A flat classifier +
+  multi-tag usually beats it (see `setfit`).

rag_wright-0.1.0/.claude/skills/creating-evals/SKILL.md

@@ -0,0 +1,114 @@
+---
+name: creating-evals
+description: >-
+  Domain-agnostic recipe for creating EVALS for engine/product capabilities, eval-first (TDD): write the eval as
+  soon as a capability is DEFINED (its contract + acceptance criterion), before it is implemented. Use it when
+  starting a new domain (right after capabilities are defined), when adding or changing a capability, or when A/B-ing
+  alternatives (e.g. a trained classifier vs a System-1 decision model). Covers gold-set design + reliability,
+  per-capability-KIND metrics (extraction / classification / retrieval / graph / generation / judgment), the
+  gate-vs-diagnostic split, building the gold cheaply, an isolated executable harness, and optional Langfuse
+  Datasets/Experiments/Scores automation. An eval is the executable acceptance criterion; training data IS an eval.
+---
+
+# Creating evals (eval-first / TDD)
+
+An eval is the **executable acceptance criterion** for a capability. Write it **as soon as the capability is
+DEFINED** — its contract (typed in/out) and its acceptance criterion exist — **before it is implemented**. This is
+TDD at the capability level: the eval fails (red) on the unbuilt/weak capability, you implement to green, then you
+can A/B alternatives and catch regressions forever. Corollary observed repeatedly in this engine: **the training
+data you build for a classifier/decision model IS an eval** (a labeled gold set + a metric) — so building the eval
+first also gives you the data design for free.
+
+## When to use
+- **Starting a new domain**: the FIRST build step after capabilities are defined (build-sequence step 4.5) — write
+  each capability's eval before/while you implement it.
+- Adding or changing a capability, or tuning a threshold/prompt/model.
+- **A/B-ing alternatives** on one capability (a deterministic rule vs a trained classifier vs a System-1 decision
+  model vs an LLM) — the eval is the neutral judge; select on the metric.
+
+## 1. Design the gold set (the foundation — get this right FIRST)
+- **Real, in-domain items + expected outputs/labels.** Small but REPRESENTATIVE; never the easy cases only.
+- **Reproducible + pinned + gitignored.** The gold is a generated artifact: pin its source snapshot + the selected
+  ids so it rebuilds identically; gitignore the data, commit the BUILDER. (Pattern: `eval/build_golden.py`.)
+- **Leakage-safe + balanced.** Split by the natural grouping unit (document / source / record), not by row.
+  For classification, a SYMMETRIC per-class test and a per-class FLOOR (not overall accuracy — it hides dead
+  classes). "k-shot" = k per class.
+- **The gold is the CEILING — measure its RELIABILITY.** For subjective/ambiguous labels, get a second
+  independent labeling and report inter-annotator (or inter-pass) agreement; adjudicate the disagreements and
+  document the calls. A model cannot beat the gold's own consistency, and label ambiguity in the gold shows up as a
+  classifier ceiling you cannot train past (CIC-1c: a two-pass gold agreed at 0.905 and a trained SetFit capped
+  ~0.82 — diagnose the gold before blaming the model). If you have no human expert yet, a documented rubric + a
+  two-pass consensus is
+  the honest proxy — say so.
+
+## 2. Pick the metric by capability KIND, and split GATE vs DIAGNOSTIC
+Always set ONE pass/fail **gate** (from the acceptance criterion) and report **diagnostics** alongside (never gate
+on a diagnostic).
+- **Extraction** (section → records, clauses, requirements): recall / precision / F1 of extracted items vs gold,
+  reported SEPARATELY (under- vs over-extraction are different failures). Watch over-extraction on non-operative
+  input (definitions) and under-extraction on long input.
+- **Classification / typed decision** (incl. SetFit, Laya, **Jev**): per-class recall + the per-class **floor**;
+  symmetric eval; top-k recall for multi-label (reported against tags-per-item). Prefer a soft-tag/calibrated
+  metric when the output routes rather than hard-gates.
+- **Retrieval**: recall@k (binary, relevant = grade ≥ a floor) as the GATE; nDCG@k (graded, exp gain) as a
+  DIAGNOSTIC (do NOT threshold nDCG). Isolate the retrieval legs (score corpus-ids before rehydration). Pattern:
+  `eval/acord_retrieval.py`.
+- **Graph / relational**: recall over the answer SET (reachability/traversal), k large enough to cover the set;
+  node ids match gold by construction. Pattern: `eval/relational_eval.py`.
+- **Generation / QA**: citation-recall / groundedness / correct-abstention; LLM-as-judge for free-text, but anchor
+  with deterministic checks (a cited span must exist). Generation is non-deterministic near the abstain boundary —
+  measure over several runs.
+- **Judgment / compliance verdicts**: report PRECISION and RECALL of the actionable class (e.g. violation)
+  SEPARATELY (alert-fatigue vs missed), and break out by provenance (real vs constructed). Pattern:
+  `scripts/eval_compliance_gold.py`.
+
+## 3. Build the gold cheaply (without faking it)
+- **Silver bootstrapping**: a higher-capability teacher (LLM, or a decision model) labels candidate items; CURATE
+  with reject-rules; mark silver, never conflate with gold. (Teacher labeling runs on the flat-GPU substrate per
+  `qwen-vllm-modal`, not ad-hoc paid calls.)
+- **Real public datasets** where they exist (e.g. labeled corpora for the task) — add to TRAIN; keep the TEST
+  in-corpus + human-adjudicated so it stays an honest transfer test.
+- **Generate hard cases** (near-boundary positives/negatives) to stress the exact confusions — TRAIN-ONLY; the
+  gold test stays real. (Full data playbook: the `setfit` skill Phase 1/4 + `classifier-opportunity-analysis`.)
+
+## 4. Make it an executable, isolated harness
+- **Isolate the capability under test**: inject it (a seam / `extract_override` / an injected `retrieve`) so the
+  eval measures ONE capability, not the whole pipeline. Invoke production code THROUGH the capability layer
+  (`ainvoke_subgraph`/`ainvoke_model`), never a hand-built copy.
+- **Score from a RESULT ARTIFACT (JSON), not stdout scraping** (scraping truncates and silently drops rows).
+- **Parallelize** model/LLM calls (async + semaphore) — same cost, far less wall-clock; order-preserving so it
+  stays deterministic.
+- **Env-selected** so the SAME harness runs local (dev) or on Modal (full corpus + GPU).
+- **Pre-flight paid bulk** (>~50 paid calls): print the exact count + cost and wait (`warn-before-bulk` rule).
+- Stream `X/N` progress + actively monitor any run over ~30s (never launch-and-forget).
+
+## 5. Langfuse — optional eval automation (we already use it for tracing)
+Langfuse has a first-class eval stack we are NOT yet using (we use it only for spans/usage today): a **Dataset**
+(items = input + optional expected output) → a **Task** (your capability) run over the dataset as an **Experiment
+Run** → **Evaluators** (deterministic checks or LLM-as-judge) producing **Scores** (numeric/categorical/boolean),
+all linked to traces. Reach for it when you want **tracked, re-runnable, dashboarded** evals + regression tracking
+across capability versions; a local JSON harness is enough for a quick one-off gate. Keep the gold BUILDER + a
+committed snapshot in the repo (reproducibility); push the items to the Langfuse dataset so runs and scores land
+next to the traces you already collect. Ground the exact Dataset/Experiment API before wiring
+(https://langfuse.com/docs/evaluation/concepts, https://langfuse.com/docs/datasets/overview); confirm the installed
+SDK version's surface, not prose.
+
+## 6. The TDD loop
+1. Write the eval from the contract + a handful of real gold items → it fails (red) on the unbuilt/weak capability.
+2. Implement the minimum to pass the gate (green).
+3. A/B alternatives (rule / trained classifier / decision model / LLM) on the SAME gold; select on gate + diagnostics + cost + calibration.
+4. Keep the eval; it is the regression guard (the Beyonce rule — if you shipped it, it has an eval).
+
+## Anti-patterns (seen, do not repeat)
+- **Overall accuracy** instead of a per-class floor — hides dead classes.
+- **Testing on teacher/generated data as if it were gold** — measures mimicry, not accuracy; keep the test real + adjudicated.
+- **Gating on a diagnostic** (e.g. nDCG) — report it, don't threshold it.
+- **No reliability check on a subjective gold** — you can't read a number off a gold whose own labels disagree.
+- **Eval not isolated** to the capability (measures the whole pipeline) — you can't attribute a regression.
+- **Scraping stdout** for metrics — write + read a JSON artifact.
+- **Faking a win by shrinking the sample / picking easy items** — real, representative, honest.
+
+## Hand-off / where this sits
+Chain: `classifier-opportunity-analysis` (identify the decision) → **`creating-evals` (THIS — write the eval FIRST)**
+→ build it (a deterministic rule, or `setfit`/`laya`/Jev for a decision, or an LLM) → `authoring-a-capability`
+(register). In the new-domain build sequence, this is the step right after capabilities are defined.

rag_wright-0.1.0/.claude/skills/laya/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: laya
+description: Recipe for fine-tuning and serving a Laya (ModernBERT-large, RL-trained typed-decision) classifier -- the OPEN-weight System-1 decision model -- to replace an LLM decision that a SetFit/encoder classifier PLATEAUS on. Also explains Jev, the MANAGED zero-shot sibling (OpenRouter Decisions API), and when to A/B Jev first (usually) vs fine-tune Laya (on-prem / no managed API). Use when confusable/relational/subjective closed-vocab values won't clear the bar with any encoder backbone. Covers uv install + env-isolation gotchas, the JSONL schema, per-value criteria, single-T4 fine-tune (RLCD), few-shot-in-context, balance, OOM knobs, Modal harness, CPU/MPS/GPU serving, and the jev_decision/DecisionModelProfile wiring (ADR-0119).
+---
+
+# Laya typed-decision classifier recipe
+
+Laya (https://github.com/NandhaKishorM/laya) is a **non-autoregressive "System-1 decision engine"**: a
+ModernBERT-large encoder + an **RL-trained decision head** that makes a typed `choice` / `score` / `noul` in one
+forward pass, with per-option **criteria** (written descriptions) and calibrated confidence. It is the **escalation
+past SetFit**: use it when the decision — not the representation — is the bottleneck.
+
+**Every number below is a DIRECTION from one project, never a target — re-measure on your own data.**
+
+## Laya vs Jev (the managed sibling) — pick the decision model first
+Laya is the **open-weight** System-1 decision model; **Jev** (TypeSafe, https://openrouter.ai, `typesafe/jev-1.13`
+via the OpenRouter **Decisions API**) is the **managed** one — same idea (typed `noul`/`choice`/`score` + calibrated
+probabilities), but **strong ZERO/few-shot with no fine-tuning**, whereas Laya scores near-random zero-shot and MUST
+be fine-tuned. Measured (RAG_Wright CIC-1c, ADR-0119): on the operative-rule gate **Jev zero-shot hit 0.92** where a
+trained SetFit capped ~0.82 and fine-tuned Laya reached ~0.70–0.74. So: **for a closed-set decision, A/B Jev
+(zero-shot, instant) first**; reach for Laya when a **managed API is unacceptable** (on-prem / data-residency) and
+you can fine-tune. In RAG_Wright a decision model is wired as a `kind="model"` capability (`jev_decision`) behind a
+`DecisionModelProfile` (model id / endpoint / thresholds in config, swappable Jev ↔ Laya) — see `setfit` Phase 0.5 +
+`authoring-a-capability`. The decision questions/criteria live in the `.ttl` (ADR-0066), not the capability.
+
+## When to use Laya (vs SetFit)
+- A closed-vocab value **plateaus below the bar with EVERY encoder backbone you try** (we ruled it out with
+  LegalBERT, bge-large, all-mpnet, AND ModernBERT-large as SetFit bodies — all stuck). That means the bottleneck is
+  the *decision* ("who bears the obligation", "which side is favored", "may either party terminate?"), not the
+  embedding. Laya's RL-against-proper-scoring-rules training targets exactly that.
+- Measured proof it's a different mechanism: `termination_right` cleared 0.68 with Laya where four encoders maxed at
+  0.60–0.64. It is NOT magic — it did NOT rescue minority-data-starved or purely-numeric dims (see Limits).
+- Default to SetFit first (cheaper, simpler, CPU-native). Reach for Laya only for the residual confusable/relational
+  dims SetFit can't crack.
+
+## Install — uv only, and beware env bleed
+- **uv, never pip.** Locally `uv add laya`; on Modal `.uv_pip_install("laya", extra_index_url="https://download.pytorch.org/whl/cu124")` for CUDA torch wheels.
+- **The env-bleed gotcha (cost real time):** a bare `uv run --with laya` on a machine with a system Anaconda picked
+  up conda's numpy/scipy/sklearn and crashed (`numpy.core.multiarray failed to import`). FIX: force uv's own managed
+  Python with a cleared path — `PYTHONPATH= PYTHONNOUSERSITE=1 uv run --python 3.11 --with laya python …` — or run
+  inside a proper isolated uv project. Never let a base conda leak in.
+- Deps are compatible with our stack: `transformers>=4.48`, `torch>=2.0`, `huggingface_hub`, Python ≥3.10.
+
+## Checkpoints — pick the ENGLISH base, not multilingual
+- English base = `convaiinnovations/laya` **root** (`laya.load("convaiinnovations/laya")`, subfolder=None) —
+  ModernBERT-large. `subfolder="multilingual"` = mmBERT; `subfolder="typed-decisions"` = a fine-tuned variant.
+- **The fine-tune script defaults `--model-dir` to the MULTILINGUAL subfolder** — you must pass the English root
+  explicitly, e.g. `snapshot_download("convaiinnovations/laya", allow_patterns=["*.json","*.safetensors","encoder/*","tokenizer/*"])` and point `--model-dir` there. A checkpoint dir = `{encoder/, tokenizer/, model.safetensors, rl_agent_config.json}`.
+- Base checkpoints score ~chance zero-shot (~0.36); **all the value is in fine-tuning.** (Zero-shot on our clauses
+  was directionally right but ~0.5 confidence — expected.)
+
+## Data — the JSONL schema + criteria (the differentiator)
+One JSONL line per case (`research/scripts/finetune_single_device.py` reads this exact shape):
+```
+{"state": "<clause text>",
+ "questions": {"<dim>": {"type":"choice","instructions":"<the question>","criteria":{"<value>":"<description>", ...}}},
+ "gold": {"<dim>": {"probabilities": {"<value>": <p>, ...}, "label": "<value>"}}}
+```
+- **`criteria` = per-value written descriptions = the semantic-guidance lever SetFit never had.** Author them from
+  the ontology (`.ttl`); if the ontology is thin (ours had vocab but no per-value defs), author from the value
+  semantics and CONFIRM with a human before training. Wording matters — it directly shapes what the model learns.
+- **`gold` is a DISTRIBUTION (soft targets), not a hard label** — RLCD trains on the teacher's probability per option.
+  One-hot (`{gold:1.0, other:0.0}`) works and is the pragmatic start; true soft targets (teacher per-option probs)
+  are the design-intended enhancement.
+- Reuse your existing **contract-disjoint, symmetric splits** so floors are directly comparable to SetFit. Silver
+  goes in TRAIN only; val/test stay gold.
+
+## Fine-tune — single T4, RLCD, calibration built in
+- `python research/scripts/finetune_single_device.py --data <train.jsonl> --model-dir <english-base> --output-dir <out> --epochs 4` (reproduces the 2×T4 notebook without DDP; CPU/one-GPU).
+- Runs on **one 16 GB GPU (T4)** — ~1–2 h for large data, minutes for our small dims. RLCD = soft-CE + GRPO-style
+  policy gradient on proper scoring rules. Calibration (one temperature per type) is fitted inside the run on a
+  held-out slice; **argmax/accuracy unchanged, only confidence moves** — always fit before gating on confidence.
+- **OOM knob (hit this):** ModernBERT-large (~400M) OOMs a T4/L4 at the default `batch_size=16, max_seq=256`. Clauses
+  are short → set `--batch-size 8 --max-seq 128` (the script/notebook also enables gradient checkpointing). That fit
+  comfortably.
+- **Run preflight FIRST — see the [[setfit]] "Run preflight & monitoring" section.** It is framework-agnostic and
+  applies to Laya exactly as to SetFit, including when you GENERATE the Laya JSONL labels with a teacher: resolve
+  the model from the engine (never a hardcoded/stale id; bulk teacher labeling on Modal Qwen, not OpenRouter), load
+  `.env` by EXPLICIT path from an out-of-repo script, SMOKE one item before the fan-out, and stream X/N to a log you
+  actively monitor. Those exact mistakes cost runs on 2026-10-04.
+- **Modal harness = reuse the [[setfit]] hardened pattern:** `.uv_pip_install("laya")` + `.add_local_file` the
+  finetune script; launcher BLOCKS on `.get()` per spawn (no spawn-and-return), stamps + verifies a `data_sha`,
+  writes a manifest, streams X/N; snapshot keepers server-side to a `/checkpoints/<name>` path (a small copy fn) —
+  the laya volume has no built-in snapshot, add one. Never lose a fine-tune.
+- **ACCOUNT CONTAINER CAP = 10 (fzaidi2014).** Spawning more than 10 fine-tunes at once does NOT run them all —
+  Modal queues the rest and runs ~10 at a time (correct, but ~N/10 waves of wall-clock, and a "why only 10 running?"
+  surprise). Size a `groupbake`/dimbatch fan-out to ≤10 in flight (chunk into waves + gather between), or state the
+  wave count honestly. This is a DIFFERENT knob from the vLLM `max_containers=1` + `@modal.concurrent` batching in the
+  qwen-vllm-modal skill — do not conflate. (Ignored the stated cap once, spawned 29 → 3 waves.)
+
+## Levers that matter (measured, dim-dependent)
+- **Few-shot-in-`state` is a big BUT dim-dependent lever.** Prepend a few labeled exemplars (from TRAIN, per class)
+  to the state. It lifted subjective/relational dims a lot (favorability 0.17→0.50, party_asymmetry 0.18→0.55) and
+  **HURT a numeric dim** (cap_basis 0.40→0.00, collapsed). **A/B few-shot per dim; never assume it helps.**
+- **Balance: don't down-sample to tiny data.** Balancing favorability DOWN to 18/18 (36 rows) + soft targets
+  COLLAPSED it (0.00) — the larger imbalanced set did better (0.50). A starved minority value needs balance-**UP**
+  (mine more minority data), not down-sampling the majority.
+- **The residual failures are minority-data-starvation, not a Laya ceiling** — the closest miss (party_asymmetry
+  0.55) is one minority-silver top-up from the bar. Diagnose starvation before concluding "Laya can't."
+
+## Limits (honest)
+- Laya rescues *decision-limited* confusable dims; it does **not** fix (a) **minority-data-starved** values (needs
+  more data), or (b) **numeric/structural** distinctions (cap_basis "fixed sum vs multiple-of-fees" — encoders and
+  Laya both failed; that belongs on the LLM).
+
+## Serving — device-agnostic, load once
+- `agent = laya.load(<checkpoint>)` — device auto: CUDA → MPS → CPU. **Verified on a Mac (MPS): ~25 s load, ~1.8 s
+  first-call warmup, then ~80–140 ms/call.** Runs on CPU too. So it honors the engine's "use a GPU if present, else
+  CPU" philosophy — same as SetFit and the LLM profiles.
+- **Load once / preload** (`Router(preload=True)`); never load per call. Serve behind the existing extractor seam so
+  nothing upstream changes. The checkpoint is ~820 MB (ModernBERT-large) — heavier than a SetFit body+joblib head;
+  budget the memory when co-loading with SetFit models.
+- Serving location is NOT pinned: single T4, share the LLM's A100, or CPU/Mac — the seam + a device arg decide at
+  runtime, exactly like the model-profile seam for the LLM.
+
+## Reference implementation
+`~/work/laya` (the cloned repo: `research/scripts/finetune_single_device.py`, `docs/finetune.md`, `examples/`) and
+`~/work/clause-classifier-ab/{laya_modal.py, build_laya_data.py, build_laya_refine.py}` (our Modal fine-tune + eval
++ data builders). Re-use the PATTERNS; the criteria, thresholds, and floors are specific to that problem.

rag_wright-0.1.0/.claude/skills/qwen-vllm-modal/SKILL.md

@@ -0,0 +1,96 @@
+---
+name: qwen-vllm-modal
+description: Authoritative recipe for standing up the self-hosted Qwen3.8-27B vLLM server on Modal (the production LLM substrate). READ THIS + ADR-0110 before touching the deploy — it fixes the "we forgot the agreed config and rediscovered every vLLM/Modal gotcha the hard way" failure. Covers the locked config, the image-build fixes, the concurrency + timeout serving fixes, cold start, and cost control.
+---
+
+# Qwen3.8-27B on Modal (vLLM) — the production LLM server
+
+**Read the record, do NOT reconstruct from memory.** The config is decided and the gotchas are known. Rummaging
+half-remembered fragments and iterating live cost a full painful session even though it had all been done, decided,
+and tested before. The two sources of truth:
+1. **ADR-0110** (`docs/adr/0110-fp8-kv-cache-16k-single-a100.md`) — the LOCKED config + why.
+2. **`scripts/modal_qwen3_vllm_server.py`** — the deploy script.
+This skill is the operational checklist that ties them together.
+
+## The LOCKED production config (ADR-0110) — do not re-litigate
+**Config B: `Qwen/Qwen3.8-27B-FP8` weights + `--kv-cache-dtype fp8`, on 1× A100-80GB, TP=1, `--max-model-len 16384`, `--gpu-memory-utilization 0.95`.**
+- Config B (FP8 weights) is the **locked deploy** — 52× concurrency @16K, ~23% higher throughput, ~27 GB weights;
+  accuracy measured **identical to bf16** on the real compliance workload (no FP8 penalty).
+- **Config A** (`Qwen/Qwen3.8-27B`, **bf16 weights** + FP8 KV) is the **documented FALLBACK only** — 25× concurrency,
+  "zero weight-loss" — use it *only* if a later clean check ever shows an FP8-weight regression. Don't default to A.
+- If anyone (including a future you) says "config A is production" — **check ADR-0110 first**; the ADR locks **B**.
+- Served-model-name is `Qwen/Qwen3.8-27B` (what the engine profile `qwen3.8-27b-modal` asks for), even though the
+  weights are the `-FP8` repo.
+
+**The exact working deploy (this succeeded):**
+```
+MODAL_IMAGE_BUILDER_VERSION=2025.06 \
+  APP_NAME=rw-qwen3-modal MODEL=Qwen/Qwen3.8-27B-FP8 KV_CACHE_DTYPE=fp8 \
+  SERVED_NAME=Qwen/Qwen3.8-27B TOOL_PARSER=hermes REASONING_PARSER=qwen3 \
+  GPU=A100-80GB:1 TP=1 MAX_LEN=16384 GPU_UTIL=0.95 MAX_NUM_SEQS=<target> \
+  uv run --no-sync modal deploy scripts/modal_qwen3_vllm_server.py
+```
+Then wire the engine: `.env` `VLLM_BASE_URL=https://<workspace>--rw-qwen3-modal-serve.modal.run/v1`,
+`VLLM_API_KEY=rw-vllm-dev-key`; the profile `qwen3.8-27b-modal` routes there. **Stop billing when done:**
+`uv run --no-sync modal app stop rw-qwen3-modal --yes` (A100 is expensive).
+
+## Image-build gotchas (the painful iterations — all fixed)
+Building a fresh vLLM image on a clean Modal account exposed a chain of failures. **The reliable fix is to base off
+the OFFICIAL prebuilt vLLM image** rather than pip-building vLLM:
+1. `pip_install("vllm")` (unpinned) on `cuda:12.8.1` → **`xformers` source-builds** → `ModuleNotFound: torch`
+   (pip build-isolation). Don't pip-build vLLM.
+2. `uv_pip_install("vllm", extra_index_url=pytorch-cu)` → resolver "no versions of vllm … unsatisfiable" — also fragile.
+3. **WORKING:** `modal.Image.from_registry("vllm/vllm-openai:latest")` — vllm+torch+xformers+flashinfer prebuilt,
+   nothing compiles. It needs two adjustments:
+   - `setup_dockerfile_commands=["RUN ln -sf $(command -v python3) /usr/local/bin/python"]` — the image ships
+     `python3` but not `python`; Modal's **own pip bootstrap runs right after FROM** as `python -m pip` → **exit 127**
+     without the symlink. It must be in `setup_dockerfile_commands` (runs BEFORE the bootstrap), not a later layer.
+   - `.entrypoint([])` — clear the image's ENTRYPOINT (the api_server) so Modal's runtime isn't hijacked.
+4. **Modal LEGACY image builder clobbers the stack**: it installs its old client deps OVER the vLLM image,
+   downgrading **pydantic→v1 and fastapi→old**, so vLLM crashes at startup (`cannot import name 'model_validator'`,
+   then `cannot import name 'Undefined' from pydantic.fields`). Patching pydantic just moves the break to fastapi
+   (whack-a-mole). **FIX = `MODAL_IMAGE_BUILDER_VERSION=2025.06`** (a modern builder whose client stack is pydantic-v2
+   compatible). Valid builder versions: `{2023.12, 2024.04, 2024.10, 2025.06}` — use the newest.
+5. `HF_HUB_DISABLE_XET=1` in the image env — Xet holds an open log handle under the HF cache so `vol.commit()` fails
+   on a fresh weight download.
+
+## Serving gotchas (equally painful — all fixed)
+1. **`@modal.concurrent(max_inputs=N)` is REQUIRED on the web_server function.** Without it Modal feeds the single
+   container ONE request at a time (`Running: 1 reqs` + a flood of "Received a cancellation signal", `/health`
+   blocked) — vLLM's continuous batching is starved. Concurrency comes from **KV-cache batching inside ONE container**
+   (`max_containers=1`), not from more containers. Set `max_inputs` = your `--max-num-seqs`.
+2. **The client structured-call timeout is too tight for reasoning-ON calls under batch.** At `--max-num-seqs` load,
+   reasoning calls take ~50–60 s; the seam's default 60 s timeout kills the ones just over the line → **retry pileup →
+   requests pile past `max_inputs` → instant `APIConnectionError` (0.0 s) cascade**. FIX: the seam reads
+   `RAG_STRUCTURED_TIMEOUT_S` (default 60) — raise it (e.g. `150`) for reasoning-ON bulk work. Server logs confirm the
+   calls actually complete 200 OK (~50 s); it was the client giving up.
+3. **Pick client concurrency empirically, below the ceiling.** 16 was stable; 20 (== `max-num-seqs`) sat at the exact
+   ceiling and risked queueing. `Running: N, Waiting: 0` + zero cancellations = healthy.
+4. **Reasoning ON vs OFF for bulk classification:** reasoning-OFF is ~30× faster but **changes the labels**
+   (over-keeps / different distribution) — do NOT swap it in for work that must match the reasoning-teacher's gold
+   (e.g. distillation silver). Keep reasoning ON when fidelity to the gold teacher matters; accept the cost.
+
+## Cold start & cost
+- Cold start ~440–480 s (CPU-bound engine init + a 51-graph CUDA-graph capture + first-time weight download), NOT
+  weight I/O — no load trick alone fixes it. GPU-snapshot / sleep-mode is a DEFERRED todo (the one attempt failed
+  because it was CPU-only; GPU retry not done). Run long deploys detached; poll `/health` for readiness.
+- **A100 is billed while up — `modal app stop <app> --yes` the moment you're done.** Verify with `modal app list`
+  (state `stopped`, 0 tasks) + endpoint 404.
+- **Do NOT warm the GPU before the consumer is ready, and do NOT trust auto-scaledown to protect billing** (burned
+  2026-10-04: a smoke-warmed A100 sat idle ~20 min while unrelated work ran, because "scaledown_window will handle
+  it" was assumed — it did not, fast enough). RULE: build the bulk/label/eval consumer FIRST (while the app is
+  stopped/cold), THEN warm → smoke → run → **`modal app stop` immediately after**, in one continuous go. If you warm
+  only to prove stand-up and the consumer is not next, STOP it right after the smoke. Treat the explicit stop as
+  mandatory, not the scaledown as sufficient; verify stopped with `modal app list`.
+
+## The meta-lesson
+Config + recipe live in **ADR-0110 + the script + this skill**. Before any Qwen/Modal work: read them, don't
+reconstruct. When something "was working and we changed accounts/rebuilt," expect the image-builder + concurrency +
+timeout trio above — they are the recurring three.
+
+## Client side (when you use this server for bulk labeling/eval)
+The SERVER config is here; the CLIENT preflight (resolve the model from the engine not a hardcoded id, load `.env`
+by explicit path from an out-of-repo script, smoke ONE item before the bulk fan-out, mandatory X/N progress +
+active monitoring) is in the **`setfit` skill's "Run preflight & monitoring"** section. Read it before driving a
+bulk job against this server — those client gotchas (a stale model id, a silently-unloaded `.env`) cost a run each
+on 2026-10-04.