rag-wright 0.13.0__tar.gz → 0.14.1__tar.gz

{rag_wright-0.13.0/src/rag_wright/.agents → rag_wright-0.14.1/.claude}/skills/authoring-a-capability/SKILL.md

@@ -1,9 +1,9 @@
 ---
 name: authoring-a-capability
 description: >-
-  How to author a new RAG_Wright engine capability of any kind (subgraph, function, model, agent_skill, mcp_tool)
-  so it is registered, ARD-discoverable, and invokable by name through the engine API. Use it whenever you add a
-  new capability or a new-domain product/graph needs one: it gives the shared registration + ARD + invocation
+  How to author a new capability on RAG_Wright, in your product's own pack or in the engine, of any kind
+  (subgraph, function, model, agent_skill, mcp_tool), so it is registered, ARD-discoverable, and invokable by name
+  through the engine API. Use it whenever a product (or the engine) adds a capability: it gives the shared registration + ARD + invocation
   contract (the four surfaces + the definition of done), the per-kind implementation specifics, and the
   conformance guardrail that keeps the catalog honest. Grounded against the real code; keep it in step with it.
 ---
@@ -93,6 +93,12 @@ MCP) follows automatically for invokable kinds; 4b (a bespoke MCP server) is opt
   (the reference pack: `packs/contracts/subgraphs/typed_property_retrieval.py`). A `production_<slug>(*, store, ...) -> CompiledGraph` builder: `g = StateGraph(_State)`,
   add nodes/edges with `START`/`END`, `return g.compile()`. Nodes call functions/models (compose).
 - **Invoke:** the co-located `async def ainvoke(resources, inputs)` factory (impl_ref target) builds + awaits the graph.
+- **An ingestion capability already has its graph.** `build_ingestion(...)` compiles one (LGI, ADR-0124): its
+  `production_<slug>` returns `pipeline.graph(ws, cache_dir=...)` (the whole run: each source through the
+  per-document graph parse -> chunk -> segment -> index -> extract -> write -> hooks -> children), and its `ainvoke`
+  runs `await pipeline.aingest(resources, sources, cache_dir=...)`. Do not wrap it in a second `StateGraph`. For a
+  long ingest, pass your own LangGraph checkpointer and a `run_id` to `aingest` so an interrupted run resumes
+  (`building-an-ingestion-capability`).
 - Retry/dead-letter come from the graph scaffold, not the invoker. `CapabilityManifest` has no contract field (only
   the internal `CapabilityRegistry.register(contract=...)` takes one); where the typed I/O must be declared, set
   `capability_interface` on the manifest (`CapabilityInterface`, from `rag_wright.pack_sdk`).

{rag_wright-0.13.0 → rag_wright-0.14.1}/.claude/skills/building-an-ingestion-capability/SKILL.md

@@ -197,6 +197,13 @@ category.
   whose continuation rows sit under its main row needs a `unit_grouper=` that keeps them together, or a
   `document_hook=` that reads the whole table with `table_rows` and writes the grouped record. An entity several units
   mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key.
+- **Resume a long ingest instead of restarting it.** Pass a LangGraph checkpointer (yours, for example a Postgres
+  saver; the engine ships none) and a `run_id` naming the run:
+  `await pipeline.aingest(ws, sources, cache_dir=CACHE, checkpointer=saver, run_id=batch_id)`. If the process stops,
+  call it again with the same sources, `cache_dir`, checkpointer and `run_id`: each document carries on from its last
+  finished stage, and nothing that finished runs again (no repeat extractor calls). A finished `run_id`, or different
+  sources, raise `ValueError`: a new run gets a new id. The graph state is built-in values only, so it works under
+  LangGraph's strict msgpack mode; a tuple inside a record's free-form `props` comes back as a list after a resume.
 - **Re-ingest.** Parsing is cached by content hash, so re-running is cheap. The default writer upserts nodes by their
   key but is not fully idempotent for edges (engine gap G8): when a source is re-run, remove its previous records
   first (`kg_delete` on the provenance your records carry, such as a per-source property) and write again.
@@ -216,7 +223,9 @@ category.
    cost per document before a bulk run.
 5. **Register the pipeline as a capability** (`authoring-a-capability`): a subgraph whose `ainvoke(resources, inputs)`
    builds it with `build_ingestion` over the workspace and runs `aingest`, so your product's ingest workflow, an
-   agent and an eval all invoke it by name.
+   agent and an eval all invoke it by name. The pipeline IS a LangGraph graph: `pipeline.graph(ws, cache_dir=...)`
+   returns the compiled run (each source through the per-document graph), which is what your `production_<slug>`
+   returns; do not wrap it in another graph.
 6. **Field-level accuracy, not only structure**: `evaluate_ingestion` checks units; the records' values are checked
    against gold records (`creating-evals`, extraction scoring over what `kg_read` returns).
 

{rag_wright-0.13.0/src/rag_wright/.agents → rag_wright-0.14.1/.claude}/skills/using-the-rag-wright-engine/SKILL.md

@@ -41,7 +41,8 @@ open-core and domain-neutral; your product brings the domain. Hard rules:
   (`parse_document`/`aparse_document`, `parse_document_bytes`/`aparse_document_bytes` for uploads,
   `source_document`/`table_rows`, `decode_bbox`, `document_of`), and the **ingestion surface**: `build_ingestion`
   (returns an `IngestionPipeline`; `await pipeline.aingest(ws, sources, cache_dir=...)` returns an
-  `IngestionReport`), its hook contracts (`Segmenter`, `SpanTagger`, `UnitGrouper`, `BoundaryDecider`,
+  `IngestionReport`; the pipeline is a LangGraph graph, `pipeline.graph(ws, cache_dir=...)`, and `aingest` with
+  `checkpointer=` and `run_id=` resumes an interrupted run), its hook contracts (`Segmenter`, `SpanTagger`, `UnitGrouper`, `BoundaryDecider`,
   `UnitRepresentative`, `BoundaryDiscoverer` with `default_chunk_discoverer(guidance=...)`, `Extractor`,
   `RecordWriter`, `DocumentHook`, checked by `check_tiling`/`check_units`/`check_extraction`), `IngestionTuning`,
   `IngestSource` (a `path`, or `data` + `name` for bytes; `table_mode`, `include_hidden_sheets`, your own `metadata`

rag_wright-0.14.1/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.14.1"
+}

{rag_wright-0.13.0 → rag_wright-0.14.1}/CHANGELOG.md

@@ -4,6 +4,23 @@ All notable changes to RAG_Wright are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.14.1](https://github.com/fzaidi-dai/RAG_Wright/compare/0.14.0...0.14.1) (2026-10-28)
+
+
+### Bug Fixes
+
+* **PS-49:** a retried ArcadeDB write conflict no longer prints as an error ([32f649f](https://github.com/fzaidi-dai/RAG_Wright/commit/32f649f94ac1b864ee5b22460d3ef401dabbbf6a))
+* **PS-50:** ensure_pack_schema declares a property a pack gains on an existing type ([c27cf40](https://github.com/fzaidi-dai/RAG_Wright/commit/c27cf40dba2512f809ba1ff06affc58d7f25f45f))
+
+## [0.14.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.13.0...0.14.0) (2026-10-17)
+
+
+### Features
+
+* **LGI-1:** each document is ingested through a LangGraph graph of the stages ([24c15df](https://github.com/fzaidi-dai/RAG_Wright/commit/24c15dfc692b463382f18b183c551127e7092c0c))
+* **LGI-2:** the whole ingest run is one LangGraph graph ([e38e9c8](https://github.com/fzaidi-dai/RAG_Wright/commit/e38e9c8733fb0bcb4da51e71e3e7093fbde4b9c3))
+* **LGI-3:** an interrupted ingest resumes from its last finished stage ([9970189](https://github.com/fzaidi-dai/RAG_Wright/commit/9970189d6d5759dc4a1eeaccb35d734cc05e886e))
+
 ## [0.13.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.12.0...0.13.0) (2026-10-15)
 
 

{rag_wright-0.13.0 → rag_wright-0.14.1}/CLAUDE.md

@@ -55,7 +55,17 @@ The failure mode to avoid, which has happened before: a command fails with "modu
 
 ## Start of every session
 
-> **CURRENT STATE (2026-10-15): `rag-wright` 0.11.0 is RELEASED** (PR #13: PS-40 the product-building skills ship,
+> **CURRENT STATE (2026-10-17): the LangGraph ingestion workstream is RELEASED as `rag-wright` 0.14.0** (PR #16; LGI-1..5,
+> `docs/specs/langgraph-ingestion/plan.md`, ADR-0124 amended): `build_ingestion` is a LangGraph graph
+> (`IngestionPipeline.graph`, per-document `document_graph`), `aingest(..., checkpointer=, run_id=)` resumes an
+> interrupted run (built-in state, strict msgpack safe), `IngestionReport.from_dict`; live old-vs-new parity identical;
+> guide `rulewright-migration-0.14.0.md`. Next: PS-48 (harness tests onto the fabric fixture,
+> proposed, not started), EV-7, PS-30, PS-33. Before it,
+> **(2026-10-15): `rag-wright` 0.13.0 is RELEASED** (PR #15): PS-35 `migrate_span_fields` on the API,
+> PS-44 `Range` KG filters (TexWright ER-1), PS-45 `span_search` (G24), PS-46 blank PDF pages never sent to the VLM
+> (TexWright's report; ADR-0062 amended), PS-47 `find_blank_pages_with_text` (audit of pre-fix parse caches); guide
+> `rulewright-migration-0.13.0.md`. Open: harness behaviour tests onto the fabric fixture, EV-7, PS-30, PS-33. Before it,
+> **`rag-wright` 0.11.0 is RELEASED** (PR #13: PS-40 the product-building skills ship,
 > PS-41 the skills audit, PS-42 by-slug registration; RuleWright's guide `docs/specs/public-surface/rulewright-migration-0.11.0.md`).
 > **PS-43 RELEASED as 0.12.0 (2026-10-15, PR #14)**: `worker_spec(definition=)`, `docs/harness.md` (tables generated by
 > `scripts/build_harness_doc.py`), the harness source cleared of RuleWright's vocabulary with a guard; guide

{rag_wright-0.13.0 → rag_wright-0.14.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rag-wright
-Version: 0.13.0
+Version: 0.14.1
 Summary: A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime.
 Project-URL: Homepage, https://github.com/fzaidi-dai/RAG_Wright
 Project-URL: Repository, https://github.com/fzaidi-dai/RAG_Wright

{rag_wright-0.13.0 → rag_wright-0.14.1}/docs/adr/0124-generic-ingestion-builder-and-hooks.md

@@ -253,3 +253,28 @@ writing to change one decision (engine gaps G3/G4).
   (7,297 blocks, 14,059 spans) fingerprint identically before and after; a live Aimmune re-ingest is record-identical
   to ING-8e. `tests/spans/test_segmentation_vocab.py` locks the vocabulary and checks the built patterns decide as the
   original literals did.
+
+## LGI-1 addendum (2026-10-17): each document runs through a LangGraph graph
+
+- **Context.** Products register their `build_ingestion` pipeline as a `subgraph` capability, but the generic path
+  was plain asyncio: no graph existed, so the capability kind's promise (a compiled `StateGraph`) did not hold, and
+  the graph version of ingestion (ADR-0050) lived only in the reference pack.
+- **Decision.** `IngestionPipeline.document_graph(ws, stages)` builds the per-document graph over the same
+  `IngestionStages`: parse -> chunk -> segment -> index -> extract -> write -> document_hook -> document_node ->
+  children, with every stage routing a failure to a `dead_letter` node (never raised). Embedded files run through the
+  same graph. `aingest` drives it, one compiled graph per run. Stages stay sequential per document (index before
+  extract), as before. Plan and ledger: `docs/specs/langgraph-ingestion/plan.md` (LGI-2 the run graph, LGI-3 resume).
+- **Consequences.** No public signature, report or progress line changed. Parity is held by a record captured from
+  the asyncio driver before the change (`tests/fixtures/ingestion/parity.json`, 11 scenarios: reports, store
+  contents, hook calls, progress lines), with the scenario inputs committed because freshly built PDFs and OOXML
+  files carry random ids and timestamps.
+- **LGI-2 (2026-10-17): the run is one graph.** `IngestionPipeline.graph(ws, *, cache_dir)` compiles the whole run:
+  `start` sends each source to `document` (the per-document graph; at most `document_concurrency` in flight, by a
+  semaphore in the node so a direct caller needs no config), `finish` builds the `IngestionReport`. `aingest` runs
+  it. The report's documents are now in source order (they were in completion order, which varied run to run).
+- **LGI-3 (2026-10-17): an interrupted run resumes.** `aingest(..., checkpointer=, run_id=)` checkpoints every stage
+  of every document (embedded children included) on the thread `run_id`; calling it again with the same sources and
+  `run_id` resumes from each document's last finished stage. A finished run, different sources, or only one of the
+  two arguments raise `ValueError`. The graph state holds built-in values only (each node loads the typed values it
+  needs), because LangGraph's strict msgpack mode blocks the engine's types; a tuple in a record's free-form `props`
+  comes back as a list after a resume. `IngestionReport.from_dict` rebuilds the report from `graph()`'s output.

{rag_wright-0.13.0 → rag_wright-0.14.1}/docs/api/README.md

@@ -186,10 +186,28 @@ Built by `build_ingestion`; run with `await pipeline.aingest(ws, sources, cache_
 
 Advanced: the individual stage functions bound to a workspace (its store, ingest embedder and tuning), for a domain that drives the stages itself (the reference contract pipeline does). Most domains only need `aingest`.
 
-#### `IngestionPipeline.aingest(self, ws: 'Any', sources: 'Sequence[Union[str, Path, IngestSource]]', *, cache_dir: 'Union[str, Path]') -> 'IngestionReport'`
+#### `IngestionPipeline.aingest(self, ws: 'Any', sources: 'Sequence[Union[str, Path, IngestSource]]', *, cache_dir: 'Union[str, Path]', checkpointer: 'Any' = None, run_id: 'Optional[str]' = None) -> 'IngestionReport'`
 
 Ingest every source (and its embedded children) into the workspace `ws` (from `open_workspace`). `sources` are file paths or `IngestSource`s (a path, or in-memory bytes with a file name); `cache_dir` holds the content-hash-gated parse and chunk caches (a re-ingest of an unchanged file re-uses them). Documents run concurrently up to `tuning.document_concurrency`; a document that fails is dead-lettered in its `DocumentReport`, never raised. Returns the `IngestionReport`. Every model role resolves through the workspace's `EngineConfig.models` (PS-14).
 
+LGI-3, resuming: pass a LangGraph `checkpointer` (yours, e.g. a Postgres saver) and a `run_id` naming the run. If the process stops mid-run, call `aingest` again with the same sources, `cache_dir`, checkpointer and `run_id`: it carries on from the last finished stage of each document, re-running none that finished. A `run_id` whose run finished raises `ValueError` (start a new run with a new id), as do different sources.
+
+#### `IngestionPipeline.graph(self, ws: 'Any', *, cache_dir: 'Union[str, Path]', checkpointer: 'Any' = None) -> 'Any'`
+
+LGI-2: the whole run as one compiled LangGraph `StateGraph`, bound to the workspace `ws`:
+
+start --(one Send per source)--> document --> finish
+
+`document` runs one source through `document_graph` (embedded files included); at most `tuning.document_concurrency` documents are in flight. The input is `{"sources": [...]}` (file paths or `IngestSource`s); the output's `report` is the `IngestionReport` as plain data (`IngestionReport.from_dict` rebuilds it), its documents in source order. This is what `aingest` runs (without `aingest`'s workspace model scoping: wrap the call in `use_workspace_models(ws)` when the workspace overrides model roles). Progress goes to the pipeline's `progress` callback. With a `checkpointer`, every stage of every document is checkpointed on the thread named by the call's `configurable.thread_id` (see `aingest` for resuming). The state holds only built-in values (dicts, lists, strings, numbers, bytes, dates), so it round-trips through any saver, including LangGraph's strict msgpack mode; a tuple inside a record's free-form `props` comes back from a checkpoint as a list.
+
+#### `IngestionPipeline.document_graph(self, ws: 'Any', stages: 'IngestionStages') -> 'Any'`
+
+LGI-1: the per-document ingestion graph, a compiled LangGraph `StateGraph` over `stages` (bound to `ws`):
+
+parse -> chunk -> segment -> index -> extract -> write -> document_hook -> document_node -> children
+
+A stage that raises routes to `dead_letter`, which ends the document with its `DocumentReport.dead_letter` set (never raised, so one bad document never stops the run). `children` ingests each embedded file through this same graph, then links it to its parent. The input is `{doc_id, source, parent_id, child, depth}`; the output's `reports` is the document's `DocumentReport` followed by its children's.
+
 ### `IngestionReport(documents: 'list[DocumentReport]' = <factory>) -> None`
 
 The result of `IngestionPipeline.aingest`: one `DocumentReport` per document (embedded children included). `failed` counts the dead-lettered documents and `succeeded` the rest (properties).

{rag_wright-0.13.0 → rag_wright-0.14.1}/docs/architecture.md

@@ -30,7 +30,7 @@ Everything a product touches is re-exported from `rag_wright.api`; the packages
 | `api/` | the stable, domain-agnostic public surface: config, workspace, invokers, KG accessors, ids, usage | **yes** (in the enforced source set) |
 | `pack_sdk/` | the pack-author tier (PS-8b): re-exports of the engine building blocks a domain pack's code needs beyond `api` (identifiers, the model seam, the LangGraph scaffold, the store protocol, the generic capabilities a pack composes); a pack imports only `api` and `pack_sdk` | **yes** |
 | `capabilities/` | the capability catalog + ARD runtime — `CapabilityManifest`, `registry`, `manifests`, the adapter-free `invoke` client; plus the generic primitives (graph query, retrieval core, embedding, disambiguation, entity resolution, the decision model) | **yes** |
-| `ingestion/` | the generic ingestion mechanism (ADR-0124): `build_ingestion` + the shared `IngestionStages`, the default layout segmenter and structural unit grouper, `table_rows`, `evaluate_ingestion` | **yes** |
+| `ingestion/` | the generic ingestion mechanism (ADR-0124): `build_ingestion` (a LangGraph graph over the shared `IngestionStages`, resumable), the default layout segmenter and structural unit grouper, `table_rows`, `evaluate_ingestion` | **yes** |
 | `subgraphs/` | the generic LangGraph scaffolding (`scaffold.py`), semantic chunking, graph extraction and observability; domain graphs live in their pack | **yes** |
 | `models/` | the model-profile seam: `ModelRole`, `profiles`, `seam`, `tag_structured`, usage/tracing | **yes** |
 | `store/` | the single ArcadeDB store behind the seam: `arcadedb` (only the engine types and generic reads/writes; a pack's own store methods live in the pack, e.g. `ContractKGStore`, `ComplianceStore`), `seam` (`KgNode`/`KgEdge`/`NOT_NULL`), `chunk_text` | **yes** |
@@ -76,7 +76,9 @@ expensive stages are content-hash gated so re-ingest is cheap. Entity extraction
 attachments become child documents linked by `EmbeddedIn` / `AttachedTo`).
 
 A new domain does not build this pipeline: it calls `build_ingestion(extractor, ...)` (in `rag_wright.ingestion`,
-re-exported from `rag_wright.api`) with its hooks. The engine owns parse, chunk, segment, index, group and write;
+re-exported from `rag_wright.api`) with its hooks. The pipeline is a LangGraph graph (`IngestionPipeline.graph`: one
+run graph sending each source to a per-document graph of the stages), so a product's ingestion capability is a real
+`subgraph`, and a run given a checkpointer and a `run_id` resumes where it stopped. The engine owns parse, chunk, segment, index, group and write;
 the domain supplies the extractor and overrides only the hooks it needs. The default segmenter follows the docling
 layout and the default unit grouper is structural. The reference `contract_ingestion_pipeline` drives the same
 shared `IngestionStages` with legal hooks (a legal segmenter, a clause-function span tagger, a provision grouper

{rag_wright-0.13.0 → rag_wright-0.14.1}/docs/concepts.md

@@ -106,6 +106,12 @@ pipeline = build_ingestion(extractor, segmenter=None, span_tagger=None, unit_gro
 report = await pipeline.aingest(ws, sources, cache_dir="cache/")
 ```
 
+- **It is a LangGraph graph.** `pipeline.graph(ws, cache_dir=...)` is the compiled run: each source goes through the
+  per-document graph (parse, chunk, segment, index, extract, write, the document hook, the `Document` node, then its
+  embedded files through the same graph), at most `document_concurrency` documents at a time; a stage that fails
+  dead-letters its document, never the run. `aingest` runs it, and with `checkpointer=` and `run_id=` an interrupted
+  run resumes from each document's last finished stage.
+
 - **The engine owns the mechanism:** parse (PDF, Office, spreadsheets incl. hidden sheets), chunk, segment each chunk
   into spans that tile it, index the spans (dense + sparse), group spans into extraction **units**, run the
   extractor on each unit concurrently, write, and record one `Document` node per document. Embedded files and PDF

{rag_wright-0.13.0 → rag_wright-0.14.1}/docs/domain-adaptation/ontology-authoring.md

@@ -18,7 +18,12 @@ The store creates the vertex and edge types your pack declares. The generic read
 `rag_wright.ontology.pack_schema.load_kg_schema(path)`, called by the store's `ensure_pack_schema(ttl)`, which
 `ensure_schema()` calls only when a pack is configured (`EngineConfig.pack`). It creates the `eng:KgVertexType`
 vertex types with their properties and unique indexes, and the `eng:KgStructuralEdge` edge types; nothing else in
-the pack is read by the engine. A minimal, domain-neutral example:
+the pack is read by the engine. Each is created when absent, so a pack that later gains a property gets it declared
+on the existing database the next time a process opens the workspace (`open_workspace` runs `ensure_schema`); no
+fresh database is needed. Nothing is altered or dropped: a property whose declared type changed is logged as a
+warning and left as it is (a migration is yours to run), and a numeric property declared over values stored before it
+is logged too, since those values are not converted until a re-ingest rewrites them (a value that does not
+convert, `'195 GSM'` for a `DOUBLE`, fails its write and dead-letters that document). A minimal, domain-neutral example:
 
 ```turtle
 @prefix eng:  <https://ragwright.local/ontology/engine#> .

rag_wright-0.14.1/docs/specs/langgraph-ingestion/plan.md

@@ -0,0 +1,77 @@
+# LangGraph ingestion: plan and ledger (LGI)
+
+Status: COMPLETE 2026-10-17 (LGI-1 to LGI-5); approved 2026-10-17 (decisions: sequential index then extract; run graph + document subgraph; serialization measured in LGI-3). Runs before PS-48 (the harness tests onto the fabric fixture).
+
+## Why
+
+`build_ingestion` (ADR-0124) is the engine's generic ingestion path, and every product's ingestion capability is
+built on it (TexWright's `texwright_ingestion` and `texwright_screen_ingestion`). It is plain asyncio: `aingest`
+runs `IngestionStages` per document with `asyncio.gather` and a semaphore. No LangGraph graph exists on that path.
+Yet the `authoring-a-capability` skill defines the `subgraph` kind as "a compiled LangGraph `StateGraph`", and
+`building-an-ingestion-capability` tells a product to register its pipeline as a `subgraph`. So every product
+registers a "subgraph" that has no graph, and gets none of what the graph gives: durable resume of a long ingest
+from a checkpointer, per-node streaming, and the substrate for a later `interrupt()` review step. The graph
+version (ADR-0050's async ingestion) exists only in the reference pack (`contract_ingestion_pipeline`,
+`async_ingestion`), not on the generic path.
+
+## Goal
+
+`build_ingestion` builds a LangGraph graph, and `aingest` runs it. A product's ingestion capability is then a real
+`subgraph`: its `production_<slug>` returns `pipeline.graph(...)`, its `ainvoke` stays `pipeline.aingest(...)`.
+
+**Nothing breaks.** `build_ingestion`'s parameters, `aingest(ws, sources, *, cache_dir)`, `IngestionReport` /
+`DocumentReport`, the `[ingest] i/N` progress lines, dead-letter semantics and `IngestionStages` (used directly by
+RuleWright's and the reference pack's contract pipelines) are unchanged. Additions only, so a `feat:` (0.14.0).
+
+## Design
+
+```
+run graph  (IngestionPipeline.graph(ws, *, cache_dir, checkpointer=None))
+  START --Send per source--> document (the per-document subgraph) --> END        report = reducer over documents
+                             at most tuning.document_concurrency documents in flight (a semaphore in the node)
+
+document subgraph
+  parse -> chunk -> segment -> index -> extract -> write -> document_hook -> document_node -> children --> END
+     |any stage raises: the document's DocumentReport.dead_letter = "<Type>: <message>" --> END (never raised)
+  children: each embedded file / attachment through the same document subgraph (depth cap 2, as today),
+            then its EmbeddedIn / AttachedTo links
+```
+
+- **The nodes ARE `IngestionStages`.** One implementation of each stage, now with three drivers' worth of reuse:
+  the generic graph, the reference pack's contract graph, and any product driving the stages itself.
+- **Order and semantics match today's driver exactly**: stages sequential per document (index before extract),
+  documents concurrent up to `document_concurrency`, children after their parent, same dead-letter strings, same
+  progress lines. Parity is the acceptance gate, measured, not assumed.
+- **State stays small and serializable.** The parse is not held in state: the `parse` node fills the
+  content-hash parse cache and later nodes reload from it (they already do, through `IngestionStages.parsed`).
+  State holds the chunks, tagged spans, units/extractions and the report fields.
+- **Durable resume is opt-in.** `checkpointer=` (any LangGraph saver; the product supplies its own, e.g. Postgres,
+  as with the harness: ADR-0139, no Postgres extra in the engine) plus a `run_id` used as the thread id. Re-running
+  an interrupted ingest with the same `run_id` skips documents and stages already checkpointed. Without a
+  checkpointer, behaviour is today's.
+- **No retry policy added.** Today a stage failure dead-letters the document; that stays (a `RetryPolicy` would
+  change behaviour and cost). The graph makes it a one-line addition later if a product asks.
+
+## Open questions (decide before LGI-1)
+
+1. **Parallel index and extract after segment?** The reference pack fans them out; on the generic path they are
+   independent (embedding vs the extractor). A wall-clock gain on model-backed extractors, but a change of order
+   in store writes. **Recommendation: keep sequential here (parity), measure the gain in a later task.**
+2. **Run graph or per-document graph only?** The run graph (one `aingest` call over its sources; a "run", named by `run_id`; not a `corpus`, which is the database name) (Send fan-out) makes the whole ingest one
+   compiled graph, so the capability is one graph and a checkpointer covers the run. **Recommendation: both, as
+   designed (the per-document subgraph is a node of the run graph).**
+3. **Checkpoint serialization of the stage values** (pydantic chunks/spans/units, dataclass `KgNode`/`KgEdge`):
+   LangGraph's serializer must round-trip them. Measured in LGI-3; if a type does not round-trip, it is converted
+   at the node boundary, never by changing the public contracts.
+
+## Ledger
+
+| Id | Task | Acceptance (verify) | Status |
+|---|---|---|---|
+| LGI-1 | The per-document subgraph over `IngestionStages` (nodes above, dead-letter route, children through the same subgraph); `aingest` drives it per document, documents still fanned out as today. Parity fixture recorded FIRST from the current driver: `IngestionReport`s and store contents for the existing hermetic ingestion fixtures (text, PDF, spreadsheet with embedded children and links, a failing extractor, a failing document) | the recorded parity fixture equal after the change; the whole existing ingestion suite green unchanged; ADR-0124 amended (and ADR-0050's note that the generic path is not a graph) | done 2026-10-17 (`document_graph`; parity 11/11 scenarios; suite 2297 passed) |
+| LGI-2 | The run graph: `IngestionPipeline.graph(ws, *, cache_dir)` public, Send fan-out per source, at most `document_concurrency` documents in flight (a semaphore inside the `document` node, so a direct `graph()` caller gets it without passing a config), the report reduced from the documents; `aingest` = run the run graph | parity fixture equal; a concurrency test (peak documents in flight = `document_concurrency`); progress lines identical (the `[ingest] start N=` / `i/N` / `done` lines asserted); API docs regenerated | done 2026-10-17 (`graph()`; parity 11/11; source-order report; suite 2304 passed) |
+| LGI-3 | Durable resume: `checkpointer=` and `run_id=` on `graph()` and `aingest`; state round-trips through the saver | hermetic, `InMemorySaver`: an ingest interrupted mid-run (a node raises a non-dead-letter interruption on document 2) resumed with the same `run_id` re-runs no completed stage (extractor and embedder call counts) and yields the same report as an uninterrupted run; without a checkpointer behaviour is unchanged | done 2026-10-17 (`checkpointer=`/`run_id=`; built-in state, strict serde green; suite 2311 passed) |
+| LGI-4 | The capability shape and docs: `authoring-a-capability` (the `subgraph` kind: an ingestion capability's `production_<slug>` returns `pipeline.graph(...)`), `building-an-ingestion-capability` step 5 (resume with the product's checkpointer, the run id), `docs/architecture.md` / `concepts.md` ingestion sections; the product-starter templates if they show the ingest | doc-reference, shipped-skills and ingestion-skill-example guards green; the skill example runs the graph | done 2026-10-17 (both skills, concepts, architecture, the consumer skill and the starter template) |
+| LGI-5 | Live parity and handover: a live `build_ingestion` run on real parses (the PS-47 workbook with an embedded PDF, and a PDF set) before and after: same `IngestionReport`, same store counts per type; a live resume with a real checkpointer; RuleWright and TexWright notes (nothing to change; the new optional resume) | live tests green; the migration notes written | done 2026-10-17 (live old-vs-new identical: 10 documents, 394 spans, 55 records, links; live crash-and-resume green; RuleWright guide 0.14.0) |
+
+Each task is one approval gate; one commit per approved task.

{rag_wright-0.13.0 → rag_wright-0.14.1}/docs/specs/public-surface/plan.md

@@ -120,3 +120,5 @@ in two at the PS-6 gate so the store rewrite is its own visible task. One task,
 | PS-45 | Engine gap G24: a public hybrid span search. `span_search(ws, query, *, k, documents, tag, dense_floor)` and `aspan_search` on `rag_wright.api`, over the span index ingestion fills: the store-side RRF fusion of the dense and sparse legs plus the dense floor (ADR-0104), returning citation-ready `SpanHit`s (text, document, offsets, pages, box, `found_by`); no reranking (the caller composes `BGEReranker`). G24 closed; the guide, the authoring page and the specifying and consumer skills name it | hermetic tests of the composition and the public call; a live test on real ArcadeDB and BGE-M3 (found by meaning, scoped, cited) | done 2026-10-15 |
 | PS-46 | TexWright's 2026-10-15 report: a blank page escalated to the VLM came back as an invented document the parse kept. The scan-quality gate gains `ScanQuality.BLANK` (no OCR text and no ink: `ink_frac`, pixels < 200, at most 1e-4; the heavy faint scan measures ~0.08) and `TieredOCRParser` never escalates or flags a blank page, reporting it in `TieredOCRReport.blank_pages`. The post-check stays a flag, not a filter (a verbatim transcription of a page with few common English words fails it). ADR-0062 amended | hermetic gate and tiered tests (a real 2-page PDF: the VLM gets page 1 only); a live docling parse reports the blank page; the heavy fixture still escalates all 10 pages | done 2026-10-15 |
 | PS-47 | The audit for PS-46: a parse cached before the fix keeps a blank page's invented text (the cache is keyed on content). `find_blank_pages_with_text(cache_dir, sources=(), *, progress)` and `BlankPageText` on `rag_wright.api` list every page blank in the source PDF (no ink, no text layer) with text in its cached parse, and the cache files to delete before re-ingesting. Searches the cache recursively (`aingest` keeps parses in `parsed/`); embedded files are audited from the cache's own copies; top-level sources are passed as paths or `(name, bytes)` | hermetic tests; a live `build_ingestion` of a workbook embedding a blank-tail PDF: clean, then a planted page found with nothing passed, then clean after delete and re-ingest | done 2026-10-15 |
+| PS-49 | TexWright's request (`docs/engine-requests/2026-10-17-concurrent-modification-printed-during-a-clean-ingest.md`, their repo): a clean keeper ingest printed `ArcadeDB Error Response ... ConcurrentModificationException` lines that read as failures. The retry is the engine's (`_retry_on_conflict`, every store write, since ING-5); the line is the client library's own ERROR log, printed before it raises. A log filter drops that line while the retry handles the call; the retry logs INFO `retried N times; the write landed`, or ERROR `gave up after 10 attempts` and raises | three hermetic tests (red before, green after); live 8-thread concurrent writes: 240/240 landed, 130 error lines before the fix, 0 after (62 INFO retried lines) | done 2026-10-17 |
+| PS-50 | TexWright's request (`docs/engine-requests/2026-10-28-ensure_pack_schema-does-not-declare-a-new-property-on-an-existing-type.md`, their repo): `ensure_pack_schema` declared a type's properties only when it created the type. It now creates every declared property the type lacks, on existing types too; a property whose declared type differs is logged and left as it is; a property the pack dropped stays; a non-STRING property declared over values stored before it is logged (they are not converted; a re-ingest rewrites them, and a value that does not convert fails its write) | a live test (red before, green after: added, mismatch reported, nothing dropped, the unconverted-values warning, a second run issues no DDL); live probe of a re-ingest over the new DOUBLE | done 2026-10-28 |

rag_wright-0.14.1/docs/specs/public-surface/rulewright-migration-0.13.0.md

@@ -0,0 +1,34 @@
+# RuleWright: moving to rag-wright 0.13
+
+What 0.13.0 changes for RuleWright (released 2026-10-15: PyPI and GitHub release `0.13.0`, PR #15). Do the 0.12
+guide first. Nothing here is edited from the engine repo, and nothing breaks an existing call: four items add calls
+and one changes how blank PDF pages are parsed.
+
+## 1. Audit your parse cache for invented blank-page text (PS-46, PS-47) — do this one
+
+**What changed.** Before 0.13.0 a blank PDF page (no ink, no text layer) was escalated to the `VISION_OCR` model,
+which invents a page when asked to transcribe an empty one, and the parse kept that text (TexWright, 2026-10-15).
+Now the scan-quality gate returns `ScanQuality.BLANK` for such a page: never escalated, never in
+`ocr_unreadable_pages`, listed in `TieredOCRReport.blank_pages`.
+
+**Why RuleWright is likely affected.** You ingest with an OpenRouter key set (`VISION_OCR` is configured), a blank
+separator page in a born-digital contract has no text layer, and invented English text passes the post-escalation
+check. The parse cache is keyed on content, so the fix does not redo an old parse.
+
+**What to do.** Run `find_blank_pages_with_text(cache_dir, sources, progress=...)` from `rag_wright.api`:
+`cache_dir` is your parse-cache root (searched recursively), `sources` your contract PDFs as paths or
+`(name, bytes)` (a top-level source's bytes are not in the cache; embedded files are found there). For each finding,
+delete its `cache_files` and re-ingest that contract. Tell us how many documents and pages it found.
+
+## 2. New calls, optional
+
+- `migrate_span_fields(config, *, corpus=...)` (PS-35): the pre-0.2.0 `Span` field migration through the public
+  API, replacing the engine's `scripts/migrate_span_fields.py`.
+- `Range(gt=, ge=, lt=, le=)` (PS-44): a `where` value for comparisons evaluated in the store, on every KG call that
+  takes `where`; `kg_read` now accepts `NOT_NULL` too.
+- `span_search(ws, query, ...)` / `aspan_search` returning `SpanHit` (PS-45): the engine's hybrid span search with
+  the dense floor (ADR-0104), citation-ready, no reranking.
+
+## Verify
+
+Your fast and db suites green on `>=0.13.0`; bump the floor. No harness name changed.

rag_wright-0.14.1/docs/specs/public-surface/rulewright-migration-0.14.0.md

@@ -0,0 +1,45 @@
+# RuleWright: moving to rag-wright 0.14
+
+What 0.14.0 changes for RuleWright (released 2026-10-17: PyPI and GitHub release `0.14.0`, PR #16; the LangGraph ingestion workstream, `docs/specs/langgraph-ingestion/plan.md`,
+LGI-1 to LGI-5; ADR-0124 amended). Do the 0.13 guide first. Nothing here is edited from the engine repo, and nothing
+breaks an existing call.
+
+## 1. `build_ingestion` is a LangGraph graph (nothing to change)
+
+`IngestionPipeline.aingest` now runs a compiled LangGraph graph: one run graph sends each source to a per-document
+graph of the stages (parse, chunk, segment, index, extract, write, the document hook, the `Document` node, then the
+embedded files through the same graph). `pipeline.graph(ws, cache_dir=...)` returns it. Arguments, the
+`IngestionReport`, dead-lettering and the progress lines are unchanged: a recorded parity fixture (11 scenarios) and
+a live before/after ingest (7 sources, 10 documents, 394 spans) are identical.
+
+Two small differences:
+
+- `IngestionReport.documents` is in source order (it was completion order, which varied run to run).
+- Hooks and the writer receive equal, freshly built objects between stages (the graph state holds plain values).
+
+**Your contract pipeline is unaffected**: it drives `IngestionStages` directly (`pack_sdk.IngestionStages`), which did
+not change.
+
+## 2. Resuming a long ingest (optional)
+
+`aingest(ws, sources, cache_dir=..., checkpointer=<your LangGraph saver>, run_id="<batch id>")` checkpoints every
+stage of every document. If the process stops, the same call with the same sources and `run_id` carries on from each
+document's last finished stage; no finished stage runs again. A finished `run_id`, or different sources, raise
+`ValueError`. The state is built-in values only, so it works under LangGraph's strict msgpack mode; a tuple in a
+record's free-form `props` comes back as a list after a resume. The engine ships no saver: pass yours (the
+Postgres saver you use for the harness fits).
+
+## 3. The skills that changed
+
+Re-read these if you link them (ADR-0134; from your installed engine):
+
+- **`building-an-ingestion-capability`**: section 6 step 5 (the pipeline IS a LangGraph graph; a capability's
+  `production_<slug>` returns `pipeline.graph(...)`), and a new item, "Resume a long ingest instead of restarting
+  it" (your checkpointer, the `run_id`, what a resume skips, the `ValueError` cases, strict msgpack mode).
+- **`authoring-a-capability`**: the `subgraph` section's new bullet, "An ingestion capability already has its graph".
+- **`using-the-rag-wright-engine`**: the ingestion-surface line names `IngestionPipeline.graph` and
+  `checkpointer=` / `run_id=`.
+
+## Verify
+
+Your fast and db suites green on `>=0.14.0`; bump the floor.

{rag_wright-0.13.0 → rag_wright-0.14.1}/docs/templates/product-starter/CLAUDE.md.template

@@ -79,7 +79,8 @@ engine — a separate repo that depends on the open-core engine, one way.
   `agent_chat_model`, answers (`agenerate_answer`/`ajudge_spans`), the eval toolkit (`aload_gold_set`/`arun_eval`/
   `EvalReport`), the document helpers (`aparse_document_bytes`, `pdf_page_count`),
   `register_capability`/`load_reference_pack`,
-  and the ingestion surface (`build_ingestion` -> `IngestionPipeline.aingest` -> `IngestionReport`,
+  and the ingestion surface (`build_ingestion` -> `IngestionPipeline.aingest` -> `IngestionReport`; the pipeline is a
+  LangGraph graph, `IngestionPipeline.graph`, and `aingest(..., checkpointer=, run_id=)` resumes an interrupted run,
   `IngestionTuning`, `IngestSource`, `evaluate_ingestion`, `table_rows`/`TableRow`, the hook contracts
   `Unit`/`UnitExtraction`/`KgNode`/`KgEdge`, the hook protocols and `DocumentHook`, and the checks
   `check_tiling`/`check_units`/`check_extraction`; `NOT_NULL` for `kg_edges` presence filters) — never engine internals (`ArcadeDBStore`,

{rag_wright-0.13.0 → rag_wright-0.14.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "rag-wright"
-version = "0.13.0"
+version = "0.14.1"
 description = "A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime."
 readme = "README.md"
 requires-python = ">=3.12,<3.13"

{rag_wright-0.13.0/.claude → rag_wright-0.14.1/src/rag_wright/.agents}/skills/authoring-a-capability/SKILL.md

@@ -1,9 +1,9 @@
 ---
 name: authoring-a-capability
 description: >-
-  How to author a new RAG_Wright engine capability of any kind (subgraph, function, model, agent_skill, mcp_tool)
-  so it is registered, ARD-discoverable, and invokable by name through the engine API. Use it whenever you add a
-  new capability or a new-domain product/graph needs one: it gives the shared registration + ARD + invocation
+  How to author a new capability on RAG_Wright, in your product's own pack or in the engine, of any kind
+  (subgraph, function, model, agent_skill, mcp_tool), so it is registered, ARD-discoverable, and invokable by name
+  through the engine API. Use it whenever a product (or the engine) adds a capability: it gives the shared registration + ARD + invocation
   contract (the four surfaces + the definition of done), the per-kind implementation specifics, and the
   conformance guardrail that keeps the catalog honest. Grounded against the real code; keep it in step with it.
 ---
@@ -93,6 +93,12 @@ MCP) follows automatically for invokable kinds; 4b (a bespoke MCP server) is opt
   (the reference pack: `packs/contracts/subgraphs/typed_property_retrieval.py`). A `production_<slug>(*, store, ...) -> CompiledGraph` builder: `g = StateGraph(_State)`,
   add nodes/edges with `START`/`END`, `return g.compile()`. Nodes call functions/models (compose).
 - **Invoke:** the co-located `async def ainvoke(resources, inputs)` factory (impl_ref target) builds + awaits the graph.
+- **An ingestion capability already has its graph.** `build_ingestion(...)` compiles one (LGI, ADR-0124): its
+  `production_<slug>` returns `pipeline.graph(ws, cache_dir=...)` (the whole run: each source through the
+  per-document graph parse -> chunk -> segment -> index -> extract -> write -> hooks -> children), and its `ainvoke`
+  runs `await pipeline.aingest(resources, sources, cache_dir=...)`. Do not wrap it in a second `StateGraph`. For a
+  long ingest, pass your own LangGraph checkpointer and a `run_id` to `aingest` so an interrupted run resumes
+  (`building-an-ingestion-capability`).
 - Retry/dead-letter come from the graph scaffold, not the invoker. `CapabilityManifest` has no contract field (only
   the internal `CapabilityRegistry.register(contract=...)` takes one); where the typed I/O must be declared, set
   `capability_interface` on the manifest (`CapabilityInterface`, from `rag_wright.pack_sdk`).

{rag_wright-0.13.0 → rag_wright-0.14.1}/src/rag_wright/.agents/skills/building-an-ingestion-capability/SKILL.md

@@ -197,6 +197,13 @@ category.
   whose continuation rows sit under its main row needs a `unit_grouper=` that keeps them together, or a
   `document_hook=` that reads the whole table with `table_rows` and writes the grouped record. An entity several units
   mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key.
+- **Resume a long ingest instead of restarting it.** Pass a LangGraph checkpointer (yours, for example a Postgres
+  saver; the engine ships none) and a `run_id` naming the run:
+  `await pipeline.aingest(ws, sources, cache_dir=CACHE, checkpointer=saver, run_id=batch_id)`. If the process stops,
+  call it again with the same sources, `cache_dir`, checkpointer and `run_id`: each document carries on from its last
+  finished stage, and nothing that finished runs again (no repeat extractor calls). A finished `run_id`, or different
+  sources, raise `ValueError`: a new run gets a new id. The graph state is built-in values only, so it works under
+  LangGraph's strict msgpack mode; a tuple inside a record's free-form `props` comes back as a list after a resume.
 - **Re-ingest.** Parsing is cached by content hash, so re-running is cheap. The default writer upserts nodes by their
   key but is not fully idempotent for edges (engine gap G8): when a source is re-run, remove its previous records
   first (`kg_delete` on the provenance your records carry, such as a per-source property) and write again.
@@ -216,7 +223,9 @@ category.
    cost per document before a bulk run.
 5. **Register the pipeline as a capability** (`authoring-a-capability`): a subgraph whose `ainvoke(resources, inputs)`
    builds it with `build_ingestion` over the workspace and runs `aingest`, so your product's ingest workflow, an
-   agent and an eval all invoke it by name.
+   agent and an eval all invoke it by name. The pipeline IS a LangGraph graph: `pipeline.graph(ws, cache_dir=...)`
+   returns the compiled run (each source through the per-document graph), which is what your `production_<slug>`
+   returns; do not wrap it in another graph.
 6. **Field-level accuracy, not only structure**: `evaluate_ingestion` checks units; the records' values are checked
    against gold records (`creating-evals`, extraction scoring over what `kg_read` returns).
 

{rag_wright-0.13.0/.claude → rag_wright-0.14.1/src/rag_wright/.agents}/skills/using-the-rag-wright-engine/SKILL.md

@@ -41,7 +41,8 @@ open-core and domain-neutral; your product brings the domain. Hard rules:
   (`parse_document`/`aparse_document`, `parse_document_bytes`/`aparse_document_bytes` for uploads,
   `source_document`/`table_rows`, `decode_bbox`, `document_of`), and the **ingestion surface**: `build_ingestion`
   (returns an `IngestionPipeline`; `await pipeline.aingest(ws, sources, cache_dir=...)` returns an
-  `IngestionReport`), its hook contracts (`Segmenter`, `SpanTagger`, `UnitGrouper`, `BoundaryDecider`,
+  `IngestionReport`; the pipeline is a LangGraph graph, `pipeline.graph(ws, cache_dir=...)`, and `aingest` with
+  `checkpointer=` and `run_id=` resumes an interrupted run), its hook contracts (`Segmenter`, `SpanTagger`, `UnitGrouper`, `BoundaryDecider`,
   `UnitRepresentative`, `BoundaryDiscoverer` with `default_chunk_discoverer(guidance=...)`, `Extractor`,
   `RecordWriter`, `DocumentHook`, checked by `check_tiling`/`check_units`/`check_extraction`), `IngestionTuning`,
   `IngestSource` (a `path`, or `data` + `name` for bytes; `table_mode`, `include_hidden_sheets`, your own `metadata`