rag-wright 0.2.1__tar.gz → 0.3.0__tar.gz

rag_wright-0.3.0/.claude/skills/authoring-a-capability/SKILL.md

@@ -0,0 +1,153 @@
+---
+name: authoring-a-capability
+description: >-
+  How to author a new RAG_Wright engine capability of any kind (subgraph, function, model, agent_skill, mcp_tool)
+  so it is registered, ARD-discoverable, and invokable by name through the engine API. Use it whenever you add a
+  new capability or a new-domain product/graph needs one: it gives the shared registration + ARD + invocation
+  contract (the four surfaces + the definition of done), the per-kind implementation specifics, and the
+  conformance guardrail that keeps the catalog honest. Grounded against the real code; keep it in step with it.
+---
+
+# Authoring a capability
+
+A **capability** is a named, ARD-registered unit of engine behavior (FR-C). Every capability has a `kind`
+(`ard.py::EntryKind`): `subgraph | function | model | agent_skill | mcp_tool` (`dagster_asset` is reserved). The
+contract below is the SAME for every kind; only the implementation differs. Capabilities compose — a subgraph calls
+functions/models; a product invokes a capability by name through `rag_wright.api`.
+
+**Ground every call before writing it** (CLAUDE.md library rule). The authoritative sources this skill summarizes:
+`capabilities/registry.py` (the canonical-slug set, `register_canonical_slugs`, the internal
+`CapabilityRegistry.register`), `capabilities/manifests.py` (`CapabilityManifest`; `_ENGINE_SPECS`, the engine's
+7 generic manifests returned by `engine_capabilities()`; `MANIFEST_SPECS`, the runtime catalog; `register_capability`,
+`load_pack(module)`), the reference pack's manifests (`CONTRACT_SPECS` / `COMPLIANCE_SPECS` in `rag_wright.packs.{contracts,compliance}.pack`), `capabilities/ard.py` (`EntryKind`, `MEDIA_TYPE_BY_KIND`,
+`CALLABLE_KINDS`), `scripts/publish_manifests.py`, `api/invoke.py` + `capabilities/invoke.py::capability_impl` (the adapter-free impl_ref invoker + drift guard),
+`api/mcp.py` (generic MCP exposure). The guardrail test is `tests/capabilities/test_authoring_contract.py`.
+Paths here are relative to `src/rag_wright/` unless they start with `scripts/` or `tests/`; all of them
+are in the engine repository: read them there or on GitHub, at the tag matching your installed engine. A product imports the
+authoring surface from `rag_wright.api`: `CapabilityManifest`, `register_capability`, `load_pack`,
+`engine_capabilities`, `register_canonical_slugs`, `canonical_capability_slugs`, `load_reference_pack`,
+`reference_pack` (the same objects as in `capabilities/manifests.py` and `capabilities/registry.py`).
+
+## The four surfaces (the definition of done)
+
+A finished capability touches these: 1 (implementation) is for EVERY kind; 2 (the invoke factory + `impl_ref`) is
+for invokable kinds (`subgraph`/`model`); 3 (manifest + register) is for every discoverable kind; 4 (invocable +
+MCP) follows automatically for invokable kinds; 4b (a bespoke MCP server) is optional.
+
+1. **Implementation** — the real code, in that kind's home (see per-kind below).
+2. **The invoke factory + `impl_ref`** (invokable kinds: subgraph/model) — write a co-located
+   `async def ainvoke(resources, inputs)` (subgraph) / `def <name>(resources, inputs)` (model) in the capability's
+   own module. A subgraph factory builds over the opaque `WorkspaceHandle` (`resources._store`,
+   `resources._embedder`, `resources.model_id(role)`), never env; a model factory is store-independent and ignores
+   `resources`. The manifest's `impl_ref="module:attr"` points to it. The invoker imports
+   it LAZILY and calls it — there is **NO central adapter dict** (EP-CORE-2). A plain function/agent_skill/mcp_tool
+   declares no `impl_ref`.
+3. **ARD manifest + register it** — a `CapabilityManifest(slug, kind, display_name, description,
+   representative_queries=(2-5…), tags=…, impl_ref=…)`. **The catalog ships EMPTY (EP-CORE-3):** call
+   `register_capability(manifest)` at runtime to add it (a product registers its own; the engine's reference pack is
+   opt-in via `load_reference_pack()`). `representative_queries` is the field ARD discovery ranks on — write real,
+   specific queries. For the ENGINE's reference pack, the manifest is committed in its pack's `pack.py` (`CONTRACT_SPECS` / `COMPLIANCE_SPECS`) and
+   the slug in its `*_CAPABILITY_SLUGS` (added to the registry by `register_canonical_slugs` when the pack loads);
+   a GENERIC engine capability's manifest is in `manifests.py::_ENGINE_SPECS` and its slug in
+   `registry.ENGINE_CAPABILITY_SLUGS`. Engine capabilities (`jev_decision`, `generation`, ...) are NOT in the
+   catalog until registered too: register them from `engine_capabilities()` when your pack uses them.
+   **Packs:** a pack is a module exposing `register()`, which calls `register_canonical_slugs(...)` for its slugs and
+   then `register_capability(m)` per manifest (plus any engine capabilities it builds on); load it with
+   `load_pack("<module>")`. `load_reference_pack()` is just `load_pack("rag_wright.packs.compliance.pack")` (compliance registers contracts first).
+   A downstream product can register without touching the canonical set (`register_capability` does not check
+   slugs), BUT `manifests.author()` rejects a non-canonical slug, so a product that publishes ARD JSON must call
+   `register_canonical_slugs` for its slugs first.
+   Publish to `~/.air/registry` with `uv run python scripts/publish_manifests.py`; that script publishes only the
+   engine's reference pack (it calls `load_reference_pack()`), so a product publishes its own with `publish_all`.
+   Callable kinds get `ResponseBounds` (defaulted); `agent_skill` must NOT declare bounds (loaded, not called).
+4. **Invocable + MCP for free** — once registered with an `impl_ref`, the capability is callable as
+   `ainvoke_subgraph(slug, inputs, resources=ws)` / `invoke_model(slug, inputs, resources=ws)` (or
+   `ainvoke_model(...)`, all on `rag_wright.api`). The invoker first checks the slug is in the catalog with the
+   kind that invoker serves (`KeyError` / `ValueError` otherwise), then imports the `impl_ref` via
+   `capabilities.invoke.capability_impl` — AND exposable over MCP (surface 4b), with **zero engine edits**.
+
+4b. **MCP exposure** (optional) — any invokable capability is already an MCP tool with zero extra code via
+   `api/mcp.py::build_capability_mcp(slug, resources=ws)` (EP-RT-2). Write a bespoke FastMCP server only
+   when you want a CURATED, typed tool signature instead of the generic opaque-`inputs` surface.
+
+## Per-kind specifics
+
+### subgraph — a compiled LangGraph `StateGraph`
+- **Home:** a generic engine subgraph in `subgraphs/<slug>.py`; a domain subgraph in its pack, `packs/<pack>/subgraphs/<slug>.py`
+  (the reference pack: `packs/contracts/subgraphs/typed_property_retrieval.py`). A `production_<slug>(*, store, ...) -> CompiledGraph` builder: `g = StateGraph(_State)`,
+  add nodes/edges with `START`/`END`, `return g.compile()`. Nodes call functions/models (compose).
+- **Invoke:** the co-located `async def ainvoke(resources, inputs)` factory (impl_ref target) builds + awaits the graph.
+- Retry/dead-letter come from the graph scaffold, not the invoker. `CapabilityManifest` has no contract field (only
+  the internal `CapabilityRegistry.register(contract=...)` takes one); where the typed I/O must be declared, set
+  `capability_interface` on the manifest.
+
+### function — a plain, typed callable
+- **Home:** `capabilities/<slug>.py` (generic) or `packs/<pack>/capabilities/<slug>.py` (domain). A deterministic or model-backed callable with a Pydantic in/out contract.
+- Invoker adapters for `function` are not wired yet (EP-API-2c); until then functions are composed inside
+  subgraphs, not invoked standalone through the API. Still register + manifest it.
+
+### model — a trained checkpoint behind a seam
+- **Home:** a GENERIC engine model capability lives in `capabilities/` (e.g. `capabilities/jev_decision.py`); a
+  domain model (e.g. the reference pack's SetFit clause classifier and 29-dim property fleet) lives with its pack,
+  in the reference pack (`rag_wright.packs.contracts.spans.model_capabilities`) for the engine's worked example, or in
+  your product repo. Load the checkpoint ONCE and cache it (the fleet is heavy).
+- Serve behind the existing seam/adapter so nothing upstream changes (to FIND where a model cap belongs, use the
+  `classifier-opportunity-analysis` skill; to BUILD/train + checkpoint + serve it, the `setfit` skill). The impl_ref
+  factory is `def <slug>(resources, inputs)` for a SYNC impl (CPU-bound local inference — a classifier/XGBoost
+  checkpoint; `resources` ignored) or `async def <slug>(resources, inputs)` for an ASYNC impl (I/O-bound — an
+  LLM-backed model cap calling OpenRouter / a local vLLM client). `invoke_model` runs a sync impl and REFUSES an
+  async one; `ainvoke_model` (EP-API-7) off-loads a sync impl with `asyncio.to_thread` and awaits an async impl
+  directly, with an optional `sem` for fan-out backpressure.
+
+### agent_skill — authored SKILL.md + the Deep Agents runtime
+- **Home:** `skills/<slug>/SKILL.md` (generic) or `packs/<pack>/skills/<slug>/SKILL.md` (domain, e.g.
+  `packs/compliance/skills/compliance_judgment/SKILL.md`) + the agent runtime (e.g. `skills/rlm/`). It is LOADED (progressive
+  disclosure), not called: no `ResponseBounds`. Declare `requires=(...)` for a closure over other skills and
+  `skill_runtime` for its intrinsic runtime.
+
+### mcp_tool — a capability exposed over MCP
+- A distinct ARD identity (`<slug>_mcp`) for the same underlying capability exposed as a cross-agent MCP tool.
+  Prefer the generic `build_capability_mcp` (surface 4b); author a bespoke server only for a curated typed
+  signature (the reference pack's are in `packs/contracts/mcp/` and `packs/compliance/mcp/`, e.g.
+  `packs/compliance/mcp/compliance_server.py`). Bind the store server-side (issue 0035) — the tool never takes a tenant/store argument.
+
+## Verify (the guardrail)
+
+**In a product**, the engine's guardrail tests are not yours to run; write the equivalent for your own pack: load
+it with `load_pack("<your pack module>")` in a test, then for each of your manifests assert the slug is in
+`capability_index()` with the kind you declared, and resolve and call its `impl_ref` (`capability_impl(slug)`, from
+`rag_wright.pack_sdk`) with a small input. The rest of this section is the engine's own guardrail.
+
+**In the engine repo**, run `uv run pytest tests/capabilities/test_authoring_contract.py tests/capabilities/test_manifests.py
+tests/capabilities/test_registry.py tests/arch/test_import_contracts.py` after authoring. It pins the contract this
+skill teaches: no manifest under a non-canonical slug; the reserved-without-manifest set is a fixed allowlist (so
+adding a slug but forgetting its manifest FAILS here); every manifest kind is a real ARD kind; every cap that declares
+an `impl_ref` is a canonical slug with a manifest of the matching kind. The guardrail does not import the
+`impl_ref` itself: add a test of your own that resolves it with `capability_impl(slug)` and calls it (the reference
+pack's is `tests/spans/test_model_capabilities.py`). If you deliberately add a reserved/internal slug (no manifest),
+add it to `_RESERVED_WITHOUT_MANIFEST` with a one-line reason. The engine's `tests/conftest.py` loads the reference
+pack, so these tests see its slugs.
+
+**The import boundary (engine repo).** `pyproject.toml` `[tool.importlinter]` has three forbidden contracts: every
+generic engine package (`source_modules`: `rag_wright.api`, `capabilities`, `contracts`, `corpus`, `ingestion`,
+`models`, `ontology`, `pack_sdk`, `skills`, `spans`, `store`, `subgraphs`, `util`) must never import
+`rag_wright.packs`; pack code (`rag_wright.packs`) imports only `rag_wright.api` and `rag_wright.pack_sdk` (plus
+itself), never another engine package directly; and `rag_wright.packs.contracts` must never import
+`rag_wright.packs.compliance`. A pack capability's code therefore takes its building blocks from those two tiers. A new generic top-level package
+goes in the first contract's `source_modules`; a new module inside an existing package or inside `rag_wright.packs`
+needs no edit. `tests/arch/test_import_contracts.py` enforces all three.
+
+**The network guard.** `tests/conftest.py` fails any test that resolves a non-local host unless it carries a live
+marker (`model`, `store`, `parse`, `embed`, `rerank`, `ner`, `fleet`). Mock model and HTTP calls in unit tests, or
+mark the test live.
+
+## Common mistakes
+
+- Adding the slug but forgetting the manifest (slug becomes silently un-discoverable) — the guardrail catches it.
+- An impl_ref factory that reaches env/globals instead of the `WorkspaceHandle` — breaks multi-workspace use; build
+  everything from `resources`.
+- A heavy import at the top of an impl_ref factory module that something light imports (a pack's `register()`
+  module, `rag_wright.api`): it inflates the light index; import inside the factory body.
+- Declaring `response_bounds` on an `agent_skill` (constructing that `CapabilityManifest` raises `ValueError`: the
+  bounds apply only to callable kinds, and a skill is loaded, not called).
+- Inventing a kind. If a capability fits none of the five, flag it — do not force-fit.

rag_wright-0.3.0/.claude/skills/building-an-ingestion-capability/SKILL.md

@@ -0,0 +1,161 @@
+---
+name: building-an-ingestion-capability
+description: >-
+  How to build a NEW domain's ingestion on the RAG_Wright engine with `build_ingestion`: decide what one extraction
+  UNIT is in your documents, write the one required hook (the extractor) and only the optional hooks you need,
+  declare the record types in your pack `.ttl`, tune the default hooks with `evaluate_ingestion` on your own sample
+  documents, and handle spreadsheets, tables and embedded files. Use it when a product or domain pack needs to turn
+  its documents into a cited knowledge graph, or when an ingestion result looks wrong (units too big, tables split,
+  records uncited). Grounded in ADR-0124 and the generated API reference (`docs/api/`).
+---
+
+# Building an ingestion capability
+
+The engine owns the ingestion MECHANISM; a domain supplies one function, the extractor, plus any optional hook whose
+default does not fit. Everything named here is imported from `rag_wright.api` (the generated `docs/api/README.md` has
+every signature). The decision record is ADR-0124 (`docs/adr/0124-generic-ingestion-builder-and-hooks.md`).
+Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`) are in the engine repository: read them there or on GitHub, at the tag matching your installed engine.
+
+## 1. Who owns which stage
+
+| stage | owner | default (engine) | override with |
+|---|---|---|---|
+| parse (PDF, Office, spreadsheets incl. hidden sheets, HTML, Markdown; embedded files) | engine | docling, content-hash cached | (none) |
+| chunk | engine default, domain may override | structural boundaries; a model refines only an over-cap section (domain-neutral prompt) | `chunk_model=`, `chunk_discoverer=` (e.g. `default_chunk_discoverer(guidance=...)` to say what a coherent unit is in your documents) |
+| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` |
+| tag spans (soft tags) | domain, optional | none | `span_tagger=` (to compare models behind it, see section 5 of `docs/domain-adaptation/classification-and-decision-models.md`) |
+| index spans (embed + store, page/bbox provenance) | engine | the workspace's ingest embedder | `embedder=` |
+| group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
+| choose each unit's representative span (its `anchor` + leading tag) | **domain decision**, optional | the grouper's choice: the first member, often a heading | `unit_representative=` |
+| extract records from a unit | **domain (required)** | (none) | the `extractor` argument |
+| write records | engine default | `kg_write` | `writer=` |
+| per-document follow-up (e.g. an entity graph) | domain, optional | none | `document_hook=` |
+| `Document` node, embedded children (`EmbeddedIn` / `AttachedTo`), progress, dead-lettering | engine | always on | (none) |
+
+Every hook's output is checked by the engine, default or override alike: `check_tiling` (spans tile the chunk text,
+ids `<chunk_id>#<index>`), `check_units` (known spans, each once, in order), `check_extraction` (provenance, below).
+A unit whose extraction fails is recorded in the report and skipped; a document that fails is dead-lettered; the run
+goes on.
+
+## 2. Decide what ONE unit is (before writing code)
+
+The unit is the text one extractor call reads. Get it right first; every other choice follows.
+
+- Ask: "one record in my domain comes from ... ?" A section under a heading (a report, a policy) -> the default
+  grouper already does this. One table row (a register, a test log) -> the default treats a DATABASE-style table
+  (named, distinct header columns plus a serial first column or many columns) as one unit per row, and sets
+  `Unit.table_row` with exact cell values; force it per document with `IngestSource(table_mode="record")`, or keep
+  tables whole with `"block"`. A form (fields of ONE record) -> one unit (`auto` keeps a form grid whole).
+- Units are capped at `IngestionTuning.max_unit_chars` (default 6000); a split table repeats its header row in each
+  continuation unit.
+- A source is a file path or uploaded bytes: `IngestSource(path=...)`, or `IngestSource(data=..., name=...)` (the
+  name's extension picks the format), so an upload from object storage needs no temp file.
+- If your documents mark units in a way layout does not show (a numbering scheme, a domain heading convention),
+  override `unit_grouper=` or pass a `boundary_decider=` (candidate line texts -> "starts a new unit?" per text) to
+  settle the lines the default grouper is unsure of. Domain conventions belong in your pack, never in the engine.
+- **Decide which span REPRESENTS a unit** (`unit_representative=`, a `UnitRepresentative`: the unit's member spans,
+  with their tags -> the one that represents it). The chosen span becomes `unit.anchor` (the citation your records
+  carry, `span_id = unit.anchor.span_id`) and its primary tag leads `unit.tags` (the label your extractor reads).
+  Without it a unit is represented by its first member, which is usually its heading: the weakest text to tag and a
+  poor citation. The reference contracts pack passes `provision_vote` (ADR-0126): the label with the highest
+  probability summed over the non-heading members, cited by the member most confident in it; on 510 CUAD contracts
+  that beat heading-first by about 10 points. Measure your own rule on a gold set of your documents before choosing
+  it (the creating-evals skill).
+
+## 3. Declare your record types in the pack `.ttl`
+
+The store creates only the types a pack declares (ADR-0066: schema lives in the ontology, not in code). A record type
+needs `span_id` and `confidence` properties to carry provenance:
+
+```turtle
+@prefix eng: <https://ragwright.local/ontology/engine#> .
+@prefix my:  <https://example.org/my-domain#> .
+
+my:SectionNode a eng:KgVertexType ; eng:vertexName "Section" ;
+    eng:kgProperty "section_id:STRING", "title:STRING", "span_id:STRING", "confidence:STRING" ;
+    eng:uniqueIndexOn "section_id" .
+```
+
+Point the workspace at it with `EngineConfig(pack="<path to your .ttl>")`; `open_workspace` then creates these types
+on top of the neutral engine types. Writing a type the pack does not declare fails.
+
+## 4. Write the extractor, tune, ingest
+
+The extractor turns one `Unit` into a `UnitExtraction` of `KgNode` / `KgEdge` records. Provenance rule
+(`check_extraction`): a fact node or edge carries `span_id` (a span of THIS unit, e.g. `unit.anchor.span_id`) and
+`confidence` (`EXTRACTED`, `INFERRED` or `AMBIGUOUS`); nodes without `span_id` are shared vocabulary; a non-empty
+extraction must cite at least once. Run `evaluate_ingestion` on your OWN sample documents before trusting the
+defaults; it needs no store and makes no model calls.
+
+```python
+import asyncio
+import os
+
+from rag_wright.api import (
+    EngineConfig, IngestionTuning, IngestSource, KgNode, StoreConfig, UnitExtraction, build_ingestion,
+    evaluate_ingestion, open_workspace,
+)
+
+PACK_TTL = "my_domain/pack.ttl"
+SAMPLES = ["samples/report.pdf"]
+CORPUS = "my_domain"
+CACHE = "data/cache/my_domain"
+
+
+async def extract(unit, *, source_doc_id):
+    """One unit -> its records, each citing a span of the unit."""
+    title = unit.text.strip().splitlines()[0][:200]
+    return UnitExtraction(nodes=[KgNode("Section", "section_id", {
+        "section_id": f"{source_doc_id}:{unit.index}", "title": title,
+        "span_id": unit.anchor.span_id, "confidence": "EXTRACTED"})])
+
+
+tuning = IngestionTuning(max_unit_chars=6000)
+evaluation = evaluate_ingestion(SAMPLES, cache_dir=CACHE, tuning=tuning)
+print("evaluation passed:", evaluation.passed, evaluation.failures)
+
+ws = open_workspace(EngineConfig(store=StoreConfig(
+    host=os.environ["ARCADEDB_HOST"], port=os.environ["ARCADEDB_PORT"],
+    user=os.environ["ARCADEDB_USER"], password=os.environ["ARCADEDB_PASSWORD"]), pack=PACK_TTL), corpus=CORPUS)
+pipeline = build_ingestion(extract, tuning=tuning)
+report = asyncio.run(pipeline.aingest(ws, [IngestSource(path=p) for p in SAMPLES], cache_dir=CACHE))
+print(report.succeeded, "ingested,", report.failed, "dead-lettered")
+for doc in report.documents:
+    print(doc.doc_id, doc.units, "units,", doc.records, "records,", doc.extraction_failures)
+```
+
+Iterate on `evaluation.failures` (and the `DocumentEvaluation` measures: `tables_whole`, `headings_start_units`,
+`coverage`, `cap_ok`, ...) by changing `IngestionTuning` before you override a hook. Keep `extract` async and
+network-bound work inside it; the engine runs up to `IngestionTuning.extract_concurrency` units at once.
+
+## 5. Spreadsheets, tables and embedded files
+
+- **Hidden sheets** are ingested by default; `IngestSource(include_hidden_sheets=False)` skips them (listed in
+  `DocumentReport.skipped_hidden_sheets`).
+- **Exact cell values**: `table_rows(parse_document(...))` returns every data row as a `TableRow` (`columns`,
+  `values`, `cell(name)`), read from the parse's cell grid, whole even when chunking split the table. A per-row unit
+  carries its row in `Unit.table_row`, so the extractor reads cells, not re-parsed text.
+- **Embedded files and PDF attachments** (an Office package's embedded workbook or PDF, a PDF's attached files) are
+  ingested as CHILD documents through the same pipeline: each gets its own `Document` node and an `EmbeddedIn` edge
+  to its parent, and an `AttachedTo` edge from the child to the table-row span it belongs to, with a confidence and
+  the identifier evidence (`IngestionTuning.identifier` sets what counts as an identifier). The report lists them in
+  `DocumentReport.children`, `links` and `unmapped_links`.
+
+## 6. Verify (definition of done)
+
+1. `evaluate_ingestion` passes on a representative sample of YOUR documents (not a hand-picked easy one).
+2. A live ingest of that sample: no dead letters, `extraction_failures` explained, records cite spans
+   (`kg_read(ws, "<your type>", fields=["span_id"])`), and `span_positions(ws, doc_id)` gives the citation positions.
+3. A hermetic test of your extractor on fixed units (no model calls in the default suite; the engine's test network
+   guard fails any unmarked test that reaches the network).
+4. If the extractor calls a model, meter it: run the ingest inside `measure_usage()` and check the call count and
+   cost per document before a bulk run.
+
+## Anti-patterns
+
+- Putting domain rules (section words, abbreviations, value lists) in Python: they belong in your pack `.ttl`.
+- Overriding the segmenter or grouper before `evaluate_ingestion` shows the default fails on your documents.
+- An extractor that returns records without `span_id`, or cites a span outside its unit (the run reports it as an
+  extraction failure, and the records are not written).
+- Re-parsing table text with regexes when `Unit.table_row` / `table_rows` already give the exact cells.
+- Importing engine internals (`rag_wright.ingestion.*`, `rag_wright.store.*`): everything here is on `rag_wright.api`.

rag_wright-0.3.0/.claude/skills/creating-evals/SKILL.md

@@ -0,0 +1,156 @@
+---
+name: creating-evals
+description: >-
+  Domain-agnostic recipe for creating EVALS for engine/product capabilities, eval-first (TDD): write the eval as
+  soon as a capability is DEFINED (its contract + acceptance criterion), before it is implemented. Use it when
+  starting a new domain (right after capabilities are defined), when adding or changing a capability, or when A/B-ing
+  alternatives (e.g. a trained classifier vs a System-1 decision model). Covers gold-set design + reliability,
+  per-capability-KIND metrics (extraction / classification / retrieval / graph / generation / judgment), the
+  gate-vs-diagnostic split, building the gold cheaply, an isolated executable harness, and optional Langfuse
+  Datasets/Experiments/Scores automation. An eval is the executable acceptance criterion; training data IS an eval.
+---
+
+# Creating evals (eval-first / TDD)
+
+An eval is the **executable acceptance criterion** for a capability. Write it **as soon as the capability is
+DEFINED** — its contract (typed in/out) and its acceptance criterion exist — **before it is implemented**. This is
+TDD at the capability level: the eval fails (red) on the unbuilt/weak capability, you implement to green, then you
+can A/B alternatives and catch regressions forever. Corollary observed repeatedly in this engine: **the training
+data you build for a classifier/decision model IS an eval** (a labeled gold set + a metric) — so building the eval
+first also gives you the data design for free.
+
+Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`) are in the engine repository: read them there or on GitHub, at the tag matching your installed engine. The `eval/` and `scripts/` files named as patterns are the engine's own evals: read them as
+worked examples; your evals live in your repo.
+
+## When to use
+- **Starting a new domain**: the FIRST build step after capabilities are defined (step 5 of the domain-adaptation guide, `docs/domain-adaptation/README.md`) — write
+  each capability's eval before/while you implement it.
+- Adding or changing a capability, or tuning a threshold/prompt/model.
+- **A/B-ing alternatives** on one capability (a deterministic rule vs a trained classifier vs a System-1 decision
+  model vs an LLM) — the eval is the neutral judge; select on the metric.
+
+## 1. Design the gold set (the foundation — get this right FIRST)
+- **Real, in-domain items + expected outputs/labels.** Small but REPRESENTATIVE; never the easy cases only.
+- **Reproducible + pinned + gitignored.** The gold is a generated artifact: pin its source snapshot + the selected
+  ids so it rebuilds identically; gitignore the data, commit the BUILDER. (Pattern: `eval/build_golden.py`.)
+- **Leakage-safe + balanced.** Split by the natural grouping unit (document / source / record), not by row.
+  For classification, a SYMMETRIC per-class test and a per-class FLOOR (not overall accuracy — it hides dead
+  classes). "k-shot" = k per class.
+- **The gold is the CEILING — measure its RELIABILITY.** For subjective/ambiguous labels, get a second
+  independent labeling and report inter-annotator (or inter-pass) agreement; adjudicate the disagreements and
+  document the calls. A model cannot beat the gold's own consistency, and label ambiguity in the gold shows up as a
+  classifier ceiling you cannot train past (CIC-1c: a two-pass gold agreed at 0.905 and a trained SetFit capped
+  ~0.82 — diagnose the gold before blaming the model). If you have no human expert yet, a documented rubric + a
+  two-pass consensus is
+  the honest proxy — say so.
+- **Validate silver against a small BLIND hand-labelled sample before treating it as gold.** Label the sample
+  without seeing any model output, then measure silver-vs-hand agreement. If it is low, the hand-labelled sample IS
+  the gold. (ING-9: silver judge labels derived from classifier test sets agreed only 64% with hand labels; the
+  decision rested on 192 blind hand-labelled cases. Pattern: `eval/semantic_judge_gold.py --score-blind`.)
+- **Separate a tuning set from a held-out set.** Label the held-out set BEFORE any prompt runs on it, tune only on
+  the tuning set, and report both numbers (ADR-0122 boundary residue prompt: 94.5% held-out vs 96.2% tuning). A
+  number measured only on the set you tuned on is not a result.
+- **Gold stays local when the corpus is restrictively licensed** (CUAD/ACORD-derived gold lives under the
+  gitignored `data/eval/`); commit the builder and the scorer, never the data.
+
+## 2. Pick the metric by capability KIND, and split GATE vs DIAGNOSTIC
+Always set ONE pass/fail **gate** (from the acceptance criterion) and report **diagnostics** alongside (never gate
+on a diagnostic).
+- **Extraction** (section → records, clauses, requirements): recall / precision / F1 of extracted items vs gold,
+  reported SEPARATELY (under- vs over-extraction are different failures). Watch over-extraction on non-operative
+  input (definitions) and under-extraction on long input.
+- **Classification / typed decision** (incl. SetFit, Laya, **Jev**): per-class recall + the per-class **floor**;
+  symmetric eval; top-k recall for multi-label (reported against tags-per-item). Prefer a soft-tag/calibrated
+  metric when the output routes rather than hard-gates.
+  - **Evaluate it the way its consumers USE it (ADR-0126).** List every consumer of the output first. If anything acts
+    on the PRIMARY label (a link, a record's type, a route), report top-1 accuracy as well as top-k recall: a top-k
+    metric hides a confusable pair whose correct label is always rank 2. Measure on PIPELINE-PRODUCED inputs (what the
+    segmenter really emits: headings, fragments, mixed sentences, the many no-label spans), not only on curated
+    snippets. And before declaring a classifier swap has "no downstream impact", check each consumer of its primary
+    label end to end. (ADR-0114's SetFit clause classifier passed >0.65 top-3 recall on gold snippets; in the pipeline
+    its top-1 confused Cap / Uncapped Liability, which silently broke exception linking.)
+- **Retrieval**: recall@k (binary, relevant = grade ≥ a floor) as the GATE; nDCG@k (graded, exp gain) as a
+  DIAGNOSTIC (do NOT threshold nDCG). Isolate the retrieval legs (score corpus-ids before rehydration). Pattern:
+  `eval/acord_retrieval.py`.
+- **Graph / relational**: recall over the answer SET (reachability/traversal), k large enough to cover the set;
+  node ids match gold by construction. Pattern: `eval/relational_eval.py`.
+- **Generation / QA**: citation-recall / groundedness / correct-abstention; LLM-as-judge for free-text, but anchor
+  with deterministic checks (a cited span must exist). Generation is non-deterministic near the abstain boundary —
+  measure over several runs.
+- **Judgment / compliance verdicts**: report PRECISION and RECALL of the actionable class (e.g. violation)
+  SEPARATELY (alert-fatigue vs missed), and break out by provenance (real vs constructed). Pattern:
+  `scripts/eval_compliance_gold.py`. For a judge that accepts or refutes other outputs, report the **error-catch
+  rate** (wrong outputs refuted), the **false-refute rate** (correct outputs refuted) and **calibration** (do its
+  scores mean what they say), not one accuracy number. Pattern: `eval/semantic_judge_gold.py` (ING-9: decision
+  model 95.3% vs LLM 90.1% on 192 blind hand-labelled cases).
+- **Candidate-then-choose pipelines** (a generator proposes, a model picks): measure the generator's **coverage**
+  separately, because it caps recall (ING-9b residual values: candidate coverage 95%; value-level recall 0.83 /
+  precision 0.88 vs the LLM's 0.60 / 0.79. Pattern: `eval/residual_decision_gold.py`). **Count is not accuracy**:
+  emitting more values is not a win without precision.
+- **Ingestion structure**: `evaluate_ingestion` (in `rag_wright.api`) is the packaged structural eval of the
+  ingestion hooks (tiling, table-row integrity, layout respect, coverage) on your own sample documents. Gate
+  patterns: `eval/segmenter_eval.py`, `eval/unit_grouper_eval.py`.
+
+## 3. Build the gold cheaply (without faking it)
+- **Silver bootstrapping**: a higher-capability teacher (LLM, or a decision model) labels candidate items; CURATE
+  with reject-rules; mark silver, never conflate with gold. (Teacher labeling runs on the flat-GPU substrate per
+  `qwen-vllm-modal`, not ad-hoc paid calls.)
+- **Real public datasets** where they exist (e.g. labeled corpora for the task) — add to TRAIN; keep the TEST
+  in-corpus + human-adjudicated so it stays an honest transfer test.
+- **Generate hard cases** (near-boundary positives/negatives) to stress the exact confusions — TRAIN-ONLY; the
+  gold test stays real. (Full data playbook: the `setfit` skill Phase 1/4 + `classifier-opportunity-analysis`.)
+
+## 4. Make it an executable, isolated harness
+- **Isolate the capability under test**: inject it (a seam / `extract_override` / an injected `retrieve`) so the
+  eval measures ONE capability, not the whole pipeline. Invoke production code THROUGH the capability layer
+  (`ainvoke_subgraph`/`ainvoke_model`), never a hand-built copy. Score the **shipped request** (the exact
+  prompt/question builder production uses, e.g. `judge_request`), not a copy of it re-typed in the eval.
+- **Repeatability for non-deterministic models.** Run each case several times on identical input; report how many
+  answers flip and how far the scores sit from the threshold. Flips cluster near the threshold and usually mean
+  the QUESTION is ambiguous: fix the question, do not vote it away (Jev flipped 5/181 answers at scores 0.47-0.56;
+  temperature/seed did not help, majority voting barely helped, rewriting the ambiguous question did. The ADR-0122
+  boundary residue prompt went from 94-96% with flips to 460/461, zero flips across 3 calls). Pattern:
+  `eval/boundary_residue_gold.py`.
+- **Cache keys include the prompt and the method.** Any decision or extraction cache must key on the prompt text
+  and the method (decision model vs LLM, and which model), so an eval never reuses outputs a different prompt or
+  method produced (ADR-0122 ING-4d / ADR-0040 ING-9b: the residue decision cache and the clause cache).
+- **Score from a RESULT ARTIFACT (JSON), not stdout scraping** (scraping truncates and silently drops rows).
+- **Parallelize** model/LLM calls (async + semaphore) — same cost, far less wall-clock; order-preserving so it
+  stays deterministic.
+- **Env-selected** so the SAME harness runs local (dev) or on Modal (full corpus + GPU).
+- **Pre-flight paid bulk** (>~50 paid calls): print the exact count + cost and wait (`warn-before-bulk` rule).
+- Stream `X/N` progress + actively monitor any run over ~30s (never launch-and-forget).
+- **Hermetic tests never touch the network**: `tests/conftest.py` fails a test that resolves a non-local host unless
+  it carries a live marker (`model`, `store`, ...). Live evals run as `uv run python -u eval/<name>.py` (outside
+  pytest) or carry a live marker.
+
+## 5. Langfuse — optional eval automation (we already use it for tracing)
+Langfuse has a first-class eval stack we are NOT yet using (we use it only for spans/usage today): a **Dataset**
+(items = input + optional expected output) → a **Task** (your capability) run over the dataset as an **Experiment
+Run** → **Evaluators** (deterministic checks or LLM-as-judge) producing **Scores** (numeric/categorical/boolean),
+all linked to traces. Reach for it when you want **tracked, re-runnable, dashboarded** evals + regression tracking
+across capability versions; a local JSON harness is enough for a quick one-off gate. Keep the gold BUILDER + a
+committed snapshot in the repo (reproducibility); push the items to the Langfuse dataset so runs and scores land
+next to the traces you already collect. Ground the exact Dataset/Experiment API before wiring
+(https://langfuse.com/docs/evaluation/concepts, https://langfuse.com/docs/datasets/overview); confirm the installed
+SDK version's surface, not prose.
+
+## 6. The TDD loop
+1. Write the eval from the contract + a handful of real gold items → it fails (red) on the unbuilt/weak capability.
+2. Implement the minimum to pass the gate (green).
+3. A/B alternatives (rule / trained classifier / decision model / LLM) on the SAME gold; select on gate + diagnostics + cost + calibration.
+4. Keep the eval; it is the regression guard (the Beyonce rule — if you shipped it, it has an eval).
+
+## Anti-patterns (seen, do not repeat)
+- **Overall accuracy** instead of a per-class floor — hides dead classes.
+- **Testing on teacher/generated data as if it were gold** — measures mimicry, not accuracy; keep the test real + adjudicated.
+- **Gating on a diagnostic** (e.g. nDCG) — report it, don't threshold it.
+- **No reliability check on a subjective gold** — you can't read a number off a gold whose own labels disagree.
+- **Eval not isolated** to the capability (measures the whole pipeline) — you can't attribute a regression.
+- **Scraping stdout** for metrics — write + read a JSON artifact.
+- **Faking a win by shrinking the sample / picking easy items** — real, representative, honest.
+
+## Hand-off / where this sits
+Chain: `classifier-opportunity-analysis` (identify the decision) → **`creating-evals` (THIS — write the eval FIRST)**
+→ build it (a deterministic rule, or `setfit`/`laya`/Jev for a decision, or an LLM) → `authoring-a-capability`
+(register). In the new-domain build sequence, this is the step right after capabilities are defined.

rag_wright-0.3.0/.claude/skills/laya/SKILL.md

@@ -0,0 +1,136 @@
+---
+name: laya
+description: Recipe for fine-tuning and serving a Laya (ModernBERT-large, RL-trained typed-decision) classifier -- the OPEN-weight System-1 decision model -- to replace an LLM decision that a SetFit/encoder classifier PLATEAUS on. Also explains Jev, the MANAGED zero-shot sibling (OpenRouter Decisions API), and when to A/B Jev first (usually) vs fine-tune Laya (on-prem / no managed API). Use when confusable/relational/subjective closed-vocab values won't clear the bar with any encoder backbone. Covers uv install + env-isolation gotchas, the JSONL schema, per-value criteria, single-T4 fine-tune (RLCD), few-shot-in-context, balance, OOM knobs, Modal harness, CPU/MPS/GPU serving, and the jev_decision/DecisionModelProfile wiring (ADR-0119).
+---
+
+# Laya typed-decision classifier recipe
+
+Laya (https://github.com/NandhaKishorM/laya) is a **non-autoregressive "System-1 decision engine"**: a
+ModernBERT-large encoder + an **RL-trained decision head** that makes a typed `choice` / `score` / `noul` in one
+forward pass, with per-option **criteria** (written descriptions) and calibrated confidence. It is the **escalation
+past SetFit**: use it when the decision — not the representation — is the bottleneck.
+
+**Every number below is a DIRECTION from one project, never a target — re-measure on your own data.**
+
+## Laya vs Jev (the managed sibling) — pick the decision model first
+Laya is the **open-weight** System-1 decision model; **Jev** (TypeSafe, https://openrouter.ai, `typesafe/jev-1.13`
+via the OpenRouter **Decisions API**) is the **managed** one — same idea (typed `noul`/`choice`/`score` + calibrated
+probabilities), but **strong ZERO/few-shot with no fine-tuning**, whereas Laya scores near-random zero-shot and MUST
+be fine-tuned. Measured (RAG_Wright CIC-1c, ADR-0119): on the operative-rule gate **Jev zero-shot hit 0.92** where a
+trained SetFit capped ~0.82 and fine-tuned Laya reached ~0.70–0.74. So: **for a closed-set decision, A/B Jev
+(zero-shot, instant) first**; reach for Laya when a **managed API is unacceptable** (on-prem / data-residency) and
+you can fine-tune. In RAG_Wright a decision model is wired as a `kind="model"` capability (`jev_decision`) behind a
+`DecisionModelProfile` (model id / endpoint / thresholds in config) — see `setfit` Phase 0.5 +
+`authoring-a-capability`. The decision questions/criteria live in the `.ttl` (ADR-0066), not the capability.
+
+**Swapping Jev for Laya is config, but not free.** `DECISION_PROFILES` has only Jev entries (`jev-1.13`, the
+default, and `jev-latest`). A Laya swap needs (1) a server that speaks the Decisions API (POST
+`{model, state, questions}`, return `{answers: {<id>: {type, noul|choice|score}}}`), and (2) a `DecisionModelProfile`
+entry in `DECISION_PROFILES` with its `endpoint`, `served_model_id` and `api_key_env` (an unknown id falls back to
+the OpenRouter endpoint). Then select it with `RAG_DECISION_MODEL=<profile id>`;
+`RAG_JEV_TIMEOUT_S` overrides the request timeout.
+
+**Batch the questions.** One call carries one `state` and many keyed questions, so ask everything about a unit in
+ONE call (the reference pack judges all of a provision's values, and labels all of its residual candidates, in one
+call each). That is the cost shape: one call per unit, not per value.
+
+**Repeatability.** Answers near the threshold can flip across identical calls (Jev: 5/181 flips, all at scores
+0.47-0.56; temperature/seed did not help, majority voting barely helped). Fix the ambiguous question instead: state
+the rubric once, use structural criteria, pass the items without surrounding text. Re-measure flips on any Laya
+server you swap in.
+
+## When to use Laya (vs SetFit)
+- A closed-vocab value **plateaus below the bar with EVERY encoder backbone you try** (we ruled it out with
+  LegalBERT, bge-large, all-mpnet, AND ModernBERT-large as SetFit bodies — all stuck). That means the bottleneck is
+  the *decision* ("who bears the obligation", "which side is favored", "may either party terminate?"), not the
+  embedding. Laya's RL-against-proper-scoring-rules training targets exactly that.
+- Measured proof it's a different mechanism: `termination_right` cleared 0.68 with Laya where four encoders maxed at
+  0.60–0.64. It is NOT magic — it did NOT rescue minority-data-starved or purely-numeric dims (see Limits).
+- Default to SetFit first (cheaper, simpler, CPU-native). Reach for Laya only for the residual confusable/relational
+  dims SetFit can't crack.
+
+## Install — uv only, and beware env bleed
+- **uv, never pip.** Locally `uv add laya`; on Modal `.uv_pip_install("laya", extra_index_url="https://download.pytorch.org/whl/cu124")` for CUDA torch wheels.
+- **The env-bleed gotcha (cost real time):** a bare `uv run --with laya` on a machine with a system Anaconda picked
+  up conda's numpy/scipy/sklearn and crashed (`numpy.core.multiarray failed to import`). FIX: force uv's own managed
+  Python with a cleared path — `PYTHONPATH= PYTHONNOUSERSITE=1 uv run --python 3.11 --with laya python …` — or run
+  inside a proper isolated uv project. Never let a base conda leak in.
+- Deps are compatible with our stack: `transformers>=4.48`, `torch>=2.0`, `huggingface_hub`, Python ≥3.10.
+
+## Checkpoints — pick the ENGLISH base, not multilingual
+- English base = `convaiinnovations/laya` **root** (`laya.load("convaiinnovations/laya")`, subfolder=None) —
+  ModernBERT-large. `subfolder="multilingual"` = mmBERT; `subfolder="typed-decisions"` = a fine-tuned variant.
+- **The fine-tune script defaults `--model-dir` to the MULTILINGUAL subfolder** — you must pass the English root
+  explicitly, e.g. `snapshot_download("convaiinnovations/laya", allow_patterns=["*.json","*.safetensors","encoder/*","tokenizer/*"])` and point `--model-dir` there. A checkpoint dir = `{encoder/, tokenizer/, model.safetensors, rl_agent_config.json}`.
+- Base checkpoints score ~chance zero-shot (~0.36); **all the value is in fine-tuning.** (Zero-shot on our clauses
+  was directionally right but ~0.5 confidence — expected.)
+
+## Data — the JSONL schema + criteria (the differentiator)
+One JSONL line per case (`research/scripts/finetune_single_device.py` reads this exact shape):
+```
+{"state": "<clause text>",
+ "questions": {"<dim>": {"type":"choice","instructions":"<the question>","criteria":{"<value>":"<description>", ...}}},
+ "gold": {"<dim>": {"probabilities": {"<value>": <p>, ...}, "label": "<value>"}}}
+```
+- **`criteria` = per-value written descriptions = the semantic-guidance lever SetFit never had.** Author them from
+  the ontology (`.ttl`); if the ontology is thin (ours had vocab but no per-value defs), author from the value
+  semantics and CONFIRM with a human before training. Wording matters — it directly shapes what the model learns.
+- **`gold` is a DISTRIBUTION (soft targets), not a hard label** — RLCD trains on the teacher's probability per option.
+  One-hot (`{gold:1.0, other:0.0}`) works and is the pragmatic start; true soft targets (teacher per-option probs)
+  are the design-intended enhancement.
+- Reuse your existing **contract-disjoint, symmetric splits** so floors are directly comparable to SetFit. Silver
+  goes in TRAIN only; val/test stay gold.
+
+## Fine-tune — single T4, RLCD, calibration built in
+- From a clone of the laya repo (its own uv project, not your product's env): `uv run python research/scripts/finetune_single_device.py --data <train.jsonl> --model-dir <english-base> --output-dir <out> --epochs 4` (reproduces the 2×T4 notebook without DDP; CPU/one-GPU; flags: `--data`, `--model-dir`, `--output-dir`, `--device`, `--epochs`, `--seed`). Inside the Modal image the harness runs the copied script with the container's own `python`.
+- Runs on **one 16 GB GPU (T4)** — ~1–2 h for large data, minutes for our small dims. RLCD = soft-CE + GRPO-style
+  policy gradient on proper scoring rules. Calibration (one temperature per type) is fitted inside the run on a
+  held-out slice; **argmax/accuracy unchanged, only confidence moves** — always fit before gating on confidence.
+- **OOM knobs:** ModernBERT-large (~400M) is tight on a 16 GB T4/L4. The single-device script has NO batch or
+  sequence flags: it fixes `micro_batch = 8`, `max_tokens_per_batch = 4096`, `max_len = 1024` and `head_max_len = 256`
+  in code, and turns on gradient checkpointing on CUDA. Our short-clause dims ran with it unchanged. If a run still
+  OOMs, lower those values in a copy of the script, or use the repo's `laya_finetune_typed_decisions_mps.py` (under notebooks/), which
+  takes `--micro-batch` and `--grad-accum`.
+- **Run preflight FIRST — see the [[setfit]] "Run preflight & monitoring" section.** It is framework-agnostic and
+  applies to Laya exactly as to SetFit, including when you GENERATE the Laya JSONL labels with a teacher: resolve
+  the model from the engine (never a hardcoded/stale id; bulk teacher labeling on Modal Qwen, not OpenRouter), load
+  `.env` by EXPLICIT path from an out-of-repo script, SMOKE one item before the fan-out, and stream X/N to a log you
+  actively monitor. Those exact mistakes cost runs on 2026-10-04.
+- **Modal harness = reuse the [[setfit]] hardened pattern:** `.uv_pip_install("laya")` + `.add_local_file` the
+  finetune script; launcher BLOCKS on `.get()` per spawn (no spawn-and-return), stamps + verifies a `data_sha`,
+  writes a manifest, streams X/N; snapshot keepers server-side to a `/checkpoints/<name>` path (a small copy fn) —
+  the laya volume has no built-in snapshot, add one. Never lose a fine-tune.
+- **ACCOUNT CONTAINER CAP = 10 (fzaidi2014).** Spawning more than 10 fine-tunes at once does NOT run them all —
+  Modal queues the rest and runs ~10 at a time (correct, but ~N/10 waves of wall-clock, and a "why only 10 running?"
+  surprise). Size a `groupbake`/dimbatch fan-out to ≤10 in flight (chunk into waves + gather between), or state the
+  wave count honestly. This is a DIFFERENT knob from the vLLM `max_containers=1` + `@modal.concurrent` batching in the
+  qwen-vllm-modal skill — do not conflate. (Ignored the stated cap once, spawned 29 → 3 waves.)
+
+## Levers that matter (measured, dim-dependent)
+- **Few-shot-in-`state` is a big BUT dim-dependent lever.** Prepend a few labeled exemplars (from TRAIN, per class)
+  to the state. It lifted subjective/relational dims a lot (favorability 0.17→0.50, party_asymmetry 0.18→0.55) and
+  **HURT a numeric dim** (cap_basis 0.40→0.00, collapsed). **A/B few-shot per dim; never assume it helps.**
+- **Balance: don't down-sample to tiny data.** Balancing favorability DOWN to 18/18 (36 rows) + soft targets
+  COLLAPSED it (0.00) — the larger imbalanced set did better (0.50). A starved minority value needs balance-**UP**
+  (mine more minority data), not down-sampling the majority.
+- **The residual failures are minority-data-starvation, not a Laya ceiling** — the closest miss (party_asymmetry
+  0.55) is one minority-silver top-up from the bar. Diagnose starvation before concluding "Laya can't."
+
+## Limits (honest)
+- Laya rescues *decision-limited* confusable dims; it does **not** fix (a) **minority-data-starved** values (needs
+  more data), or (b) **numeric/structural** distinctions (cap_basis "fixed sum vs multiple-of-fees" — encoders and
+  Laya both failed; that belongs on the LLM).
+
+## Serving — device-agnostic, load once
+- `agent = laya.load(<checkpoint>)` — device auto: CUDA → MPS → CPU. **Verified on a Mac (MPS): ~25 s load, ~1.8 s
+  first-call warmup, then ~80–140 ms/call.** Runs on CPU too. So it honors the engine's "use a GPU if present, else
+  CPU" philosophy — same as SetFit and the LLM profiles.
+- **Load once / preload** (`Router(preload=True)`); never load per call. Serve behind the existing extractor seam so
+  nothing upstream changes. The checkpoint is ~820 MB (ModernBERT-large) — heavier than a SetFit body+joblib head;
+  budget the memory when co-loading with SetFit models.
+- Serving location is NOT pinned: single T4, share the LLM's A100, or CPU/Mac — the seam + a device arg decide at
+  runtime, exactly like the model-profile seam for the LLM.
+
+## Reference implementation
+The laya repo (its `research/scripts/finetune_single_device.py`, its fine-tuning guide and its `examples/`) and
+the engine authors' own Modal fine-tune + eval + data builders (a private working repo, not shipped). Re-use the PATTERNS; the criteria, thresholds, and floors are specific to that problem.