rag-wright 0.16.1__tar.gz → 0.17.1__tar.gz

{rag_wright-0.16.1 → rag_wright-0.17.1}/.claude/skills/building-an-ingestion-capability/SKILL.md

@@ -22,7 +22,7 @@ Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`)
 |---|---|---|---|
 | parse (PDF, Office, spreadsheets incl. hidden sheets, HTML, Markdown, images and scans; embedded files) | engine | docling, content-hash cached; images and scanned pages are OCR'd, and only degraded pages escalate to the vision model (the `VISION_OCR` role) | (none) |
 | chunk | engine default, domain may override | structural boundaries; a model refines only an over-cap section (domain-neutral prompt) | `chunk_model=`, `chunk_discoverer=` (e.g. `default_chunk_discoverer(guidance=...)` to say what a coherent unit is in your documents) |
-| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` |
+| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` (to adjust the default rather than replace it, wrap `pack_sdk.segment_layout`) |
 | tag spans (soft tags) | domain, optional | none | `span_tagger=` (to compare models behind it, see section 5 of `docs/domain-adaptation/classification-and-decision-models.md`) |
 | index spans (embed + store, page/bbox provenance) | engine | the workspace's ingest embedder | `embedder=` |
 | group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
@@ -181,6 +181,9 @@ category.
   `evaluate_ingestion(..., table_labels=[{"pattern": <regex on the header row>, "label": "record"|"block"}])`
   (first match wins): a block table split per row, or a wide record table not split, is a failure, and tables no
   pattern matched are listed in `unlabelled_tables`.
+- **Every row of a record table should reach a unit.** After an ingest, `DocumentReport.unreached_table_rows` lists
+  each record table whose rows did not all reach a unit with their `TableRow` (with the `[ingest] warning:` line);
+  an empty list on your samples is the check. Rows listed there produced no record.
 - **How an embedded file finds its row.** An identifier is a token of 4+ characters containing a digit that appears
   on at most `IdentifierRule.max_rows` table rows and in at most `max_files` embedded files (defaults 3 and 3; set
   them with `IngestionTuning(identifier=IdentifierRule(...))`). A file whose identifiers match no row, or too many,
@@ -196,7 +199,11 @@ category.
 - **One record over several rows, or one entity over many units.** The extractor sees one unit at a time. A record
   whose continuation rows sit under its main row needs a `unit_grouper=` that keeps them together, or a
   `document_hook=` that reads the whole table with `table_rows` and writes the grouped record. An entity several units
-  mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key.
+  mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key; an edge to it
+  emitted from each unit lands once, because the default writer keeps one edge per (type, from node, to node)
+  (PS-63). A `writer=` of your own gets the same with `kg_write(ws, nodes, edges, ensure_edges=True)`; leave the flag
+  off only when parallel edges of one type between the same two nodes are meant to stay (each with its own
+  provenance).
 - **Resume a long ingest instead of restarting it.** Pass a LangGraph checkpointer (yours, for example a Postgres
   saver; the engine ships none) and a `run_id` naming the run:
   `await pipeline.aingest(ws, sources, cache_dir=CACHE, checkpointer=saver, run_id=batch_id)`. If the process stops,
@@ -205,8 +212,9 @@ category.
   sources, raise `ValueError`: a new run gets a new id. The graph state is built-in values only, so it works under
   LangGraph's strict msgpack mode; a tuple inside a record's free-form `props` comes back as a list after a resume.
 - **Re-ingest.** Parsing is cached by content hash, so re-running is cheap. The default writer upserts nodes by their
-  key but is not fully idempotent for edges (engine gap G8): when a source is re-run, remove its previous records
-  first (`kg_delete` on the provenance your records carry, such as a per-source property) and write again.
+  key and writes each edge once, updating its props, so a re-run leaves no duplicates. It does not remove what a
+  source no longer yields: when a re-run should drop records, remove its previous records first (`kg_delete` on the
+  provenance your records carry, such as a per-source property) and write again.
 
 ## 6. Verify (definition of done)
 

rag_wright-0.17.1/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.17.1"
+}

{rag_wright-0.16.1 → rag_wright-0.17.1}/CHANGELOG.md

@@ -4,6 +4,26 @@ All notable changes to RAG_Wright are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.17.1](https://github.com/fzaidi-dai/RAG_Wright/compare/0.17.0...0.17.1) (2026-11-07)
+
+
+### Bug Fixes
+
+* **PS-63:** the default record writer keeps one edge per type and endpoints (G8) ([e77ca91](https://github.com/fzaidi-dai/RAG_Wright/commit/e77ca9151e8cd5a425237f30ada721349eac7bb0))
+
+## [0.17.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.16.1...0.17.0) (2026-11-05)
+
+
+### Features
+
+* **PS-61:** the default segmenter is on pack_sdk, so a pack can wrap it ([bbb179d](https://github.com/fzaidi-dai/RAG_Wright/commit/bbb179d64e5fc905ada2ddc57367cdd2cc2ca5b6))
+* **PS-62:** a record table's rows that reach no unit are reported, never silent ([632fefb](https://github.com/fzaidi-dai/RAG_Wright/commit/632fefb390e11014a0dcb4cdd16956697a92ea58))
+
+
+### Bug Fixes
+
+* **PS-60:** a table over the chunk cap keeps every row's span and TableRow ([acd9ce5](https://github.com/fzaidi-dai/RAG_Wright/commit/acd9ce5917de1daac6ac47ea0a09ca975fcdf556))
+
 ## [0.16.1](https://github.com/fzaidi-dai/RAG_Wright/compare/0.16.0...0.16.1) (2026-11-01)
 
 

{rag_wright-0.16.1 → rag_wright-0.17.1}/CLAUDE.md

@@ -55,7 +55,17 @@ The failure mode to avoid, which has happened before: a command fails with "modu
 
 ## Start of every session
 
-> **CURRENT STATE (2026-10-29): `rag-wright` 0.16.0 is RELEASED** (PR #19; TexWright's requests: PS-54 a `Contains`
+> **CURRENT STATE (2026-11-07): PS-63 done, unreleased** (TexWright's G8 request: the default record writer keeps one
+> edge per type and endpoints, `kg_write(..., ensure_edges=True)`, ADR-0142). Next: PS-48, EV-7, PS-30, PS-33. Before it,
+> **`rag-wright` 0.17.0 is RELEASED** (PR #21; TexWright's request: PS-60 a table over
+> the chunk cap is split between lines with its header repeated, so every row keeps its span and `TableRow` (ADR-0141);
+> PS-61 the default segmenter `segment_layout` is on `pack_sdk`; PS-62 a record table's rows that reach no unit are
+> reported, `DocumentReport.unreached_table_rows`). Next: PS-48, EV-7, PS-30, PS-33. Before it,
+> **`rag-wright` 0.16.1 is RELEASED** (PR #20; RuleWright's two requests and a finding
+> from them: PS-57 a retried contracts-pack write no longer loses a document's clauses, the ArcadeDB conflict budget is
+> configurable (default 20 attempts / 2 s cap) and a racing UPSERT's duplicate key is retried; PS-58 a `RecordWriter`
+> returns the count that landed; PS-59 the store's client keeps keep-alive connections, one session per thread). Next:
+> PS-48, EV-7, PS-30, PS-33. Before it, **`rag-wright` 0.16.0 is RELEASED** (PR #19; TexWright's requests: PS-54 a `Contains`
 > substring `where` value, PS-55 only the BGE models on Metal under one process lock (ADR-0140); PS-56 the BGE reranker
 > scores again on transformers 5). Before it, **`rag-wright` 0.15.0 is RELEASED** (PR #18; PS-51 the frontend skill draws the shell and
 > ships its scaffold, RuleWright's request; PS-52 one workspace embedder under concurrent first use and PS-53 KG calls refuse

{rag_wright-0.16.1 → rag_wright-0.17.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rag-wright
-Version: 0.16.1
+Version: 0.17.1
 Summary: A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime.
 Project-URL: Homepage, https://github.com/fzaidi-dai/RAG_Wright
 Project-URL: Repository, https://github.com/fzaidi-dai/RAG_Wright

rag_wright-0.17.1/docs/adr/0141-the-hard-split-cuts-between-lines-and-repeats-a-tables-header.md

@@ -0,0 +1,38 @@
+# ADR-0141: The cap's hard split cuts between lines and repeats a table's header
+
+**Status:** accepted · **Date:** 2026-11-05 · **Related:** PS-60, TexWright's request
+`2026-11-05-export-the-default-segmenter-on-pack-sdk.md`, ADR-0069 (tables chunked in reading order), ADR-0124
+(ING-2 the default segmenter, ING-7 `table_rows`), ADR-0136 (spreadsheet tables glued across blank rows)
+
+## Context
+
+A chunk is a run of whole parsed items, and the token cap is enforced last: a section over the cap that the
+discoverer cannot subdivide (a single table is one item) is hard-split. The split cut every `token_cap * 4`
+characters, wherever that fell. In a table it fell mid-row, and three things followed. The table item was found
+whole in no chunk, so `chunk_layouts` placed it nowhere, and the unplaced item also stopped every later item in the
+document from getting its parsed layout. Each piece then fell back to the text layout: the first piece still
+opened with the table's header, but a continuation piece began mid-row, read as one paragraph, and gave ONE span to
+all its rows. With no row spans, no unit carried a `TableRow`, so a record extractor never saw those rows, and the
+ingest reported success. TexWright lost 4 of 140 records this way; a 140-record, 76-column synthetic workbook lost
+205 of 280 rows.
+
+## Decision
+
+- The hard split cuts at the last blank line in the second half of the window, else the last line break, else the
+  last space, and only a run with none of them at a fixed length. The whitespace character it cuts at is dropped.
+  Pieces stay strictly under the cap.
+- A cut inside a pipe table repeats the table's header and separator rows at the top of the next piece, so every
+  piece is a well-formed table, each row is whole in one chunk, and the segmenter, the unit grouper's record-table
+  test and the row-to-span map work unchanged on every piece. A header half the cap or longer is not repeated.
+- `chunk_layouts` skips an item that no remaining chunk holds whole, instead of blocking the items after it.
+- A cached chunk manifest holding a piece exactly the cap long with table lines in it can only come from the old
+  cut; it is re-chunked once, with a log line. An old prose split keeps its cached chunks.
+
+## Consequences
+
+- Live, the workbook above yields 280 of 280 row records (75 before), and every span is one row.
+- A document with a section over the cap gets different chunk boundaries, so different `chunk_id`s and spans when
+  re-ingested; the `chunk_id` scheme is unchanged and documents under the cap are untouched. A repeated header
+  appears once per continuation chunk in the canonical document text.
+- Whether a pack may build on the default segmenter (PS-61) and whether a record table's lost rows are reported
+  (PS-62) are separate tasks.

rag_wright-0.17.1/docs/adr/0142-the-default-writer-keeps-one-edge-per-type-and-endpoints.md

@@ -0,0 +1,32 @@
+# ADR-0142: The default record writer keeps one edge per type and endpoints
+
+**Status:** accepted · **Date:** 2026-11-07 · **Related:** PS-63, engine gap G8, TexWright's request
+`2026-10-15-g8-duplicates-edges-within-one-ingest.md`, ADR-0124 (ING-4b `kg_ensure_edges`), issue 0029
+(`outV()`/`inV()` endpoint filters)
+
+## Context
+
+With no `writer=`, `build_ingestion` wrote records with `kg_write`: nodes upserted by key, edges created with a plain
+`CREATE EDGE`. The engine's own guidance has an entity several units mention emitted by its key from each unit, so
+each such unit also emitted the edge to it, and that edge landed once per unit, within one ingest and again on every
+re-ingest. Node counts stayed right, so only a traversal showed it (TexWright: 3 children read back as 6). The
+store's idempotent `kg_ensure_edges` runs a query and a write per edge outside any transaction, and scans the edge
+type for each edge.
+
+## Decision
+
+- `Store.kg_write` takes `ensure_edges` (default `False`). When true, an edge is identified by its type and its two
+  endpoints (direction counts): the batch keeps each edge's last props; each edge is an index-driven `UPDATE` of an
+  existing one, reached through `outE` from its indexed from node, then `CREATE EDGE ... IF NOT EXISTS`, all in the
+  write's one transaction under the same conflict retry. ArcadeDB 26.7 was probed live: `IF NOT EXISTS` holds within
+  one transaction, across transactions and across concurrent writers.
+- The ingestion default writer passes `True`; `rag_wright.api.kg_write` exposes the flag for a domain's own writer.
+- Without the flag `kg_write` is unchanged: a caller whose parallel edges each carry their own provenance (the
+  reference pack's clause-to-value assertions, RuleWright's writer) keeps them.
+
+## Consequences
+
+- An edge several units emit, or a re-ingest writes again, lands once with the last unit's props; G8 is closed.
+- A domain that means parallel edges of one type between the same two nodes, and uses the default writer, now gets
+  one; it passes a `writer=` that calls `kg_write` without the flag.
+- A re-ingest still does not remove records a document no longer yields. `kg_ensure_edges` is unchanged.

{rag_wright-0.16.1 → rag_wright-0.17.1}/docs/adr/README.md

@@ -196,4 +196,6 @@ framing that still appears in older ADRs and the root `SPEC.md`/`plan.md`.
 0137 A generated answer states the evidence's support every time, and "none" is an abstention ·
 0138 PDF parses run one at a time, and the engine reaches pdfium through one locked door ·
 0139 The agent harness substrate is engine scope ·
-0140 One model family on Metal, under one process lock
+0140 One model family on Metal, under one process lock ·
+0141 The cap's hard split cuts between lines and repeats a table's header ·
+0142 The default record writer keeps one edge per type and endpoints

{rag_wright-0.16.1 → rag_wright-0.17.1}/docs/api/README.md

@@ -216,9 +216,9 @@ A stage that raises routes to `dead_letter`, which ends the document with its `D
 
 The result of `IngestionPipeline.aingest`: one `DocumentReport` per document (embedded children included). `failed` counts the dead-lettered documents and `succeeded` the rest (properties).
 
-### `DocumentReport(doc_id: 'str', parent_doc_id: 'Optional[str]' = None, chunks: 'int' = 0, spans: 'int' = 0, units: 'int' = 0, records: 'int' = 0, extraction_failures: 'list[dict]' = <factory>, span_failures: 'list[dict]' = <factory>, skipped_hidden_sheets: 'list[str]' = <factory>, embedded_skipped: 'list[str]' = <factory>, children: 'list[str]' = <factory>, links: 'dict[str, int]' = <factory>, unmapped_links: 'int' = 0, dead_letter: 'Optional[str]' = None) -> None`
+### `DocumentReport(doc_id: 'str', parent_doc_id: 'Optional[str]' = None, chunks: 'int' = 0, spans: 'int' = 0, units: 'int' = 0, records: 'int' = 0, extraction_failures: 'list[dict]' = <factory>, span_failures: 'list[dict]' = <factory>, skipped_hidden_sheets: 'list[str]' = <factory>, embedded_skipped: 'list[str]' = <factory>, children: 'list[str]' = <factory>, links: 'dict[str, int]' = <factory>, unmapped_links: 'int' = 0, unreached_table_rows: 'list[dict]' = <factory>, dead_letter: 'Optional[str]' = None) -> None`
 
-What ingesting one document did. Counts: `chunks`, `spans` (indexed), `units` (extracted), `records` (nodes written). `extraction_failures` (`{unit, anchor, reason}`) / `span_failures` (`{span_id, reason}`): the units / spans whose extraction or indexing failed; the rest of the document still lands. `skipped_hidden_sheets`: hidden spreadsheet sheets left out (`IngestSource.include_hidden_sheets=False`). `children`: the ids of embedded files and PDF attachments ingested as child documents (`parent_doc_id` is set on theirs); `embedded_skipped`: embedded files that could not be ingested. `links`: `AttachedTo` record links written, by confidence; `unmapped_links`: links whose record row could not be found. `dead_letter`: why the whole document failed (None when it landed).
+What ingesting one document did. Counts: `chunks`, `spans` (indexed), `units` (extracted), `records` (nodes written). `extraction_failures` (`{unit, anchor, reason}`) / `span_failures` (`{span_id, reason}`): the units / spans whose extraction or indexing failed; the rest of the document still lands. `skipped_hidden_sheets`: hidden spreadsheet sheets left out (`IngestSource.include_hidden_sheets=False`). `children`: the ids of embedded files and PDF attachments ingested as child documents (`parent_doc_id` is set on theirs); `embedded_skipped`: embedded files that could not be ingested. `links`: `AttachedTo` record links written, by confidence; `unmapped_links`: links whose record row could not be found. `unreached_table_rows` (PS-62): one entry per record table -- a table at least one of whose rows reached a unit carrying its `TableRow` -- whose data rows did not ALL reach one, `{table_ref, sheet, reached, total, missing}` (`missing`: the 1-based data-row indices, at most the first 1,000); the document still lands, so read this, or the `[ingest] warning:` line, to know rows were lost. `dead_letter`: why the whole document failed (None when it landed).
 
 ### `IngestionEvaluation(documents: 'list[DocumentEvaluation]' = <factory>, failures: 'list[str]' = <factory>, unlabelled_tables: 'list[str]' = <factory>, labels: 'Optional[LabelEvaluation]' = None, warnings: 'list[str]' = <factory>) -> None`
 
@@ -430,9 +430,9 @@ Rank the live ARD catalog by semantic match to `query`; return the top `k` (opti
 
 Read typed nodes of `node_type` from the workspace (see `Store.kg_read`). Each `where` value is a scalar (equality), a list (membership; an empty list is scope-to-nothing -> `[]`), `NOT_NULL` (presence), a `Range` (comparisons, on any number of fields: `{"weight": Range(ge=180, le=220), "tensile": Range(gt=400)}`) or a `Contains` (a literal substring, case-insensitive unless `case_sensitive=True`; an empty field never matches: `{"composition": Contains("UHMWPE")}`); all filters are AND-ed and evaluated in the store. `key_range=(field, lo, hi)` is the half-open shorthand (`lo <= field < hi`). Then projection, distinct, order and limit. Every name is checked, never interpolated unchecked (the module docstring): a request-chosen field is safe, but an unknown one matches nothing.
 
-### `kg_write(ws: 'WorkspaceHandle', nodes: 'list', edges: 'Any' = ()) -> 'None'`
+### `kg_write(ws: 'WorkspaceHandle', nodes: 'list', edges: 'Any' = (), *, ensure_edges: 'bool' = False) -> 'None'`
 
-Upsert typed `nodes` + create typed `edges` in one transaction (see `Store.kg_write`). `nodes`/`edges` are `KgNode`/`KgEdge` (from `rag_wright.api`); the store encodes each field per its pack-declared type. A node type the workspace's schema does not declare fails the write.
+Upsert typed `nodes` + create typed `edges` in one transaction (see `Store.kg_write`). `nodes`/`edges` are `KgNode`/`KgEdge` (from `rag_wright.api`); the store encodes each field per its pack-declared type. A node type the workspace's schema does not declare fails the write. `ensure_edges=True` (PS-63) writes at most one edge per (type, from node, to node), the last props winning: what a domain `writer=` passes to match the default writer.
 
 ### `kg_edges(ws: 'WorkspaceHandle', from_type: 'Optional[str]' = None, *, where: 'Optional[dict]' = None, key_range: 'Optional[tuple]' = None, direction: 'str' = 'out', edge_type: 'Optional[str]' = None, edge_where: 'Optional[dict]' = None, target_where: 'Optional[dict]' = None, select: 'dict') -> 'list[dict]'`
 

{rag_wright-0.16.1 → rag_wright-0.17.1}/docs/api/pack_sdk.md

@@ -182,7 +182,7 @@ Group into units (checked), attach table rows, extract each unit concurrently (p
 
 #### `IngestionStages.write(self, sd: 'Any', extractions: 'list') -> 'int'`
 
-Persist the records (the domain's writer, else `kg_write`). Returns the number that landed: the writer's own count when it returns one; else the record nodes handed over, which the default `kg_write` lands in one transaction (it raises rather than land part of them).
+Persist the records (the domain's writer, else `kg_write` with idempotent edges). Returns the number that landed: the writer's own count when it returns one; else the record nodes handed over, which the default `kg_write` lands in one transaction (it raises rather than land part of them).
 
 #### `IngestionStages.write_document_node(self, doc_id: 'str', *, parent_id: 'Optional[str]', filename: 'str', media_type: 'str', sha256: 'str', metadata: 'Optional[dict]' = None) -> 'None'`
 
@@ -492,6 +492,10 @@ ASYNC twin of `chunk_texts` (SEG-2/SEG-5): awaits the discoverer's async boundar
 
 Validate a `documents` scope against `store.known_document_ids()` BEFORE any retrieval spends (mirrors issue 0007's `_validate_sources`). `None` (whole store) is not validated. `[]` (scope-to-nothing) is a valid empty scope, not an error -- the retrieval surfaces short-circuit it. A non-empty list with any id absent from the store raises `UnknownDocumentError`. A store without `known_document_ids` (a minimal fake) is treated as un-validatable and passes through.
 
+### `segment_layout(chunk_id: 'str', text: 'str', layout: 'Sequence[LayoutItem]', *, min_alnum: 'Optional[int]' = None, tuning: 'Optional[IngestionTuning]' = None) -> 'list[Span]'`
+
+Segment one chunk into spans (the engine's default `Segmenter`). The fragment floor is `min_alnum`, else `tuning.min_fragment_alnum`, else the default.
+
 ### `capability_impl(name: 'str') -> 'Callable[..., Any]'`
 
 Resolve a capability name to its invoke factory via the manifest `impl_ref`. Raises `KeyError` if the name is unknown to the ARD catalog, `NotImplementedError` if it declares no `impl_ref` (not invokable by name).

{rag_wright-0.16.1 → rag_wright-0.17.1}/docs/domain-adaptation/_engine-gaps.md

@@ -53,13 +53,16 @@ they are the real "does a new customer benefit?" items. The genuine engine ones
 - **Proposed:** a generic reader in `ontology.pack_schema` (or similar) over an engine-namespaced criterion
   vocabulary.
 
-### G8: The default record writer is not fully idempotent
-- **Where:** with no `writer=`, `build_ingestion` writes with the store's `kg_write`: nodes are upserted by their key,
-  but edges are created (`CREATE EDGE`), so re-ingesting a document duplicates its record edges. The store's
-  idempotent `kg_ensure_edges` (used for `EmbeddedIn` / `AttachedTo`) is not exported from `rag_wright.api`.
-- **Impact:** a domain whose extractor returns edges must pass its own idempotent `writer`, or clear the document's
-  records before a re-ingest.
-- **Proposed:** make the default writer use `kg_ensure_edges`, or export it.
+### G8: The default record writer is not fully idempotent (closed by PS-63)
+- **Where:** with no `writer=`, `build_ingestion` wrote with the store's `kg_write`: nodes were upserted by their key,
+  but edges were created (`CREATE EDGE`).
+- **Impact:** not only a re-ingest. Within ONE ingest of one document, an edge that several units emit (an entity
+  several rows mention, emitted by its key from each unit) landed once per unit, so a traversal over-counted while
+  node counts looked right (TexWright, 2026-10-15).
+- **Closed by PS-63 (ADR-0142):** the default writer calls `kg_write(..., ensure_edges=True)`: at most one edge per
+  (type, from node, to node), within the write, across writes and across concurrent writers, the last props landing,
+  in the same one transaction. A domain `writer=` gets the same with `rag_wright.api.kg_write(ws, nodes, edges,
+  ensure_edges=True)`; without the flag `kg_write` still keeps parallel edges.
 
 ### G9: One domain pack per workspace through the public API
 - **Where:** `EngineConfig.pack` takes one `.ttl`. A second pack's schema needs the store-internal

{rag_wright-0.16.1 → rag_wright-0.17.1}/docs/domain-adaptation/kg-construction.md

@@ -34,6 +34,20 @@ The reference pack's contract pipeline is one instantiation of the same stages (
 legal segmenter, a function tagger, a provision grouper, a decision-model boundary decider, its clause extractor and
 its typed writer. Read it as an example of hooks, not as the engine's definition of ingestion.
 
+A hook you pass replaces the default for the whole pipeline, for every document it ingests. When your documents
+need the default's behaviour in most places and your own rule in a few, build on the default rather than replace it:
+the default segmenter is `rag_wright.pack_sdk.segment_layout` (and the default chunk discoverer
+`StructuralModelFallbackDiscoverer`), so your segmenter can call it and adjust only the spans it gets wrong:
+
+```python
+from rag_wright.pack_sdk import segment_layout
+
+
+def segmenter(chunk_id, text, layout):
+    spans = segment_layout(chunk_id, text, layout)  # the engine's evaluated segmentation
+    return spans  # split or merge here only the spans your domain reads differently; keep them tiling the text
+```
+
 ## The stages
 
 ```
@@ -169,8 +183,10 @@ the run goes on. Every node and edge type your extractor returns must be declare
 
 `aingest` returns an `IngestionReport` (`succeeded`, `failed`, `documents`). Each `DocumentReport` has the counts
 (`chunks`, `spans`, `units`, `records`), `extraction_failures`, `span_failures`, `dead_letter` (the reason, if the
-document failed), `skipped_hidden_sheets`, `embedded_skipped`, `children`, and the `links` / `unmapped_links` of its
-embedded files. Nothing is skipped silently.
+document failed), `skipped_hidden_sheets`, `embedded_skipped`, `children`, the `links` / `unmapped_links` of its
+embedded files, and `unreached_table_rows`: each record table some of whose data rows reached no unit carrying their
+`TableRow`, so produced no record (`reached`, `total` and the `missing` row indices; also printed as an
+`[ingest] warning:` line). Nothing is skipped silently.
 
 ## The two graphs (one store)
 
@@ -205,10 +221,11 @@ ask-first change.
   citation**.
 - **What is cached on a re-ingest:** in `build_ingestion`, parse and chunk are content-hash cached; spans,
   record nodes and the `Document` node are upserted by key; the `EmbeddedIn` / `AttachedTo` edges are ensured (never
-  duplicated). Extraction runs again on every ingest (cache inside your extractor if it is expensive), and the
-  default writer creates record edges rather than ensuring them, so re-ingesting a document whose extractor returns
-  edges duplicates them unless your `writer` is idempotent (engine gap G8). A boundary decider is not cached either
-  (engine gap G6).
+  duplicated). Extraction runs again on every ingest (cache inside your extractor if it is expensive). The
+  default writer ensures record edges too (PS-63): one edge per (type, from node, to node), whether several units
+  emit it or a re-ingest writes it again, with the last props landing; a domain `writer=` gets the same from
+  `kg_write(ws, nodes, edges, ensure_edges=True)`. A re-ingest does not remove records a document no longer yields.
+  A boundary decider is not cached either (engine gap G6).
 
 ## Building your own
 

{rag_wright-0.16.1 → rag_wright-0.17.1}/docs/specs/public-surface/plan.md

@@ -131,4 +131,8 @@ in two at the PS-6 gate so the store rewrite is its own visible task. One task,
 | PS-57 | RuleWright's request (`docs/engine-requests/2026-10-31-a-write-conflict-loses-a-documents-clauses-and-your-pack-carries-the-defect.md`, their repo): at 4 concurrent documents, 2 of 15 reported `ingested` with zero clauses. The contracts pack's `write` node popped the pending extractions (filled by `extract_clauses`) before the store write landed, so the node's retry wrote nothing and reported success; it now reads them and releases them after the write. The conflict budget is reachable: `StoreConfig.conflict_attempts` / `conflict_backoff_cap_s`, else `ARCADEDB_CONFLICT_ATTEMPTS` / `ARCADEDB_CONFLICT_BACKOFF_CAP_S`; default raised 10 attempts / 0.5 s cap to 20 / 2 s. Found by the live probe: two writers creating the same new node get a duplicate key, never retried; `kg_write` and `write_graph` (UPSERT-only) now retry it | pack retry test through the real per-document graph (0/3 clauses landed before, 3/3 after); six store tests; live probe (clause-shaped transactions over shared value nodes, pooled sessions): old budget lost 38/2,400 at 4 and 8 writers, new lost 0/2,400 at the same wall-clock; live store suite 42 passed | done 2026-11-01 |
 | PS-58 | RuleWright's request (`docs/engine-requests/2026-10-31-ingestionstages-write-returns-a-count-derived-from-its-own-input.md`, their repo): `IngestionStages.write` returned the record nodes it was handed, and `RecordWriter` (declared `-> None`) could not say what landed. A writer now returns the number of records in the graph (`Optional[int]`; `None` keeps the engine's count); the default `kg_write` branch keeps the handed-over count, true because that write is one transaction (lands whole or raises). The contracts pack's writer returns the clauses in the graph and its `write` node reports that as `clauses` | three hermetic tests + the PS-57 retry test's count (red before, green after); live `build_ingestion` on two fixtures: default and a writer landing every other record, `records` equals the graph in all four documents | done 2026-11-01 |
 | PS-59 | Found during PS-57: the ArcadeDB client opens a new connection for every request (`requests.post`, no session); at 8 concurrent writers against Docker on macOS about two thirds of writes failed with `RemoteDisconnected`, and with a keep-alive session per thread the same run lost none. Give the store's client a pooled session | live probe at 8 writers: zero connection drops | done 2026-11-01: the store's client (`_PooledClient`) keeps one keep-alive `requests.Session` per thread and `ArcadeDBStore.close()` releases them; live, the same probe lost 374 of 800 transactions to `RemoteDisconnected` unpooled and 0 of 800 at 8 writers and 0 of 1,600 at 16 pooled |
+| PS-60 | TexWright (2026-11-05, `2026-11-05-export-the-default-segmenter-on-pack-sdk.md`): a table over the chunk cap was hard-split at a fixed length, mid-row; the rows after the cut lost their table layout (one span between them, no `TableRow`), and one item `chunk_layouts` could not place blocked every later item's layout. 4 of 140 records lost silently | the cut falls between lines and repeats the table's header in each continuation chunk; an unplaceable item no longer blocks later ones; a manifest cached with the old cut is re-chunked once; live: a 140-record x 76-column workbook yields 140 records | done 2026-11-05 (ADR-0141): live, the workbook (280 rows, two per record) gave 280/280 row records, 75/280 with the old cut |
+| PS-61 | TexWright's question in the same request: the default segmenter `segment_layout` is on neither public tier, so a pack cannot wrap it | export it on `rag_wright.pack_sdk` | done 2026-11-05: `pack_sdk.segment_layout` (the engine's own object); the wrap-the-default pattern in `kg-construction.md` and the ingestion skill; a pack segmenter delegating to it runs through `build_ingestion` |
+| PS-62 | TexWright, same request: the loss was silent (every `check_*` passed) | a record table whose rows do not each reach a unit with its `TableRow` is reported in the `IngestionReport` | done 2026-11-05: `DocumentReport.unreached_table_rows` (`table_ref`, `sheet`, `reached`, `total`, `missing`) and an `[ingest] warning:` line; a block table is not a loss; live, the pre-PS-60 cut on the workbook reports 75 of 280 rows reached (205 missing, matching the records lost), the new cut reports nothing |
+| PS-63 | TexWright, `2026-10-15-g8-duplicates-edges-within-one-ingest.md` (gap G8): the default writer created every edge, so an edge several units emit (an entity several rows mention) landed once per unit, within one ingest and on every re-ingest | the default writer leaves one edge per (type, from node, to node); `api.kg_write` exposes the same; `kg_write` without the flag unchanged | done 2026-11-07: `kg_write(..., ensure_edges=)` (ADR-0142), the default writer passes it, `api.kg_write` exposes it; G8 closed, the ingestion skill and the KG-construction guide updated; live, TexWright's shape (140 children on two rows each, 20 parents) leaves 140 edges on an ingest and a re-ingest (280 then 560 before) |
 | PS-48 | Port the agent harness's substrate behaviour tests from RuleWright onto the engine's fabric fixture (ADR-0139 handover): of RuleWright's 553 harness-touching tests, 314 test the substrate (port), 213 RuleWright's composition (stay), 25 live (1 substrate: OpenRouter cost accounting). Seven gated sub-tasks: **a** shared `tests/harness/fabric.py` + scripting/slow/hanging/failing fake models + `InMemorySaver` helpers, then events, skills, clarify (~23); **b** models and errors (~42); **c** runs, confirm, continuation (~32); **d** decisions and rules (~51); **e** grading (~32); **f** seen and memory (~41); **g** workers, batch, dispatch, tiers, agent wiring, filesystem, interpreter, withhold + the live accounting test (~93, may split). Ambiguous tests split, not moved whole; a ported test failing on engine code is a substrate defect fixed in that sub-task; deadline globals via monkeypatch; no new dependencies. Follow-up (owner RuleWright): retire its substrate duplicates once ported | each sub-task's ported tests green on the fabric fixture | todo (proposed; awaiting go-ahead) |

{rag_wright-0.16.1 → rag_wright-0.17.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "rag-wright"
-version = "0.16.1"
+version = "0.17.1"
 description = "A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime."
 readme = "README.md"
 requires-python = ">=3.12,<3.13"

{rag_wright-0.16.1 → rag_wright-0.17.1}/src/rag_wright/.agents/skills/building-an-ingestion-capability/SKILL.md

@@ -22,7 +22,7 @@ Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`)
 |---|---|---|---|
 | parse (PDF, Office, spreadsheets incl. hidden sheets, HTML, Markdown, images and scans; embedded files) | engine | docling, content-hash cached; images and scanned pages are OCR'd, and only degraded pages escalate to the vision model (the `VISION_OCR` role) | (none) |
 | chunk | engine default, domain may override | structural boundaries; a model refines only an over-cap section (domain-neutral prompt) | `chunk_model=`, `chunk_discoverer=` (e.g. `default_chunk_discoverer(guidance=...)` to say what a coherent unit is in your documents) |
-| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` |
+| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` (to adjust the default rather than replace it, wrap `pack_sdk.segment_layout`) |
 | tag spans (soft tags) | domain, optional | none | `span_tagger=` (to compare models behind it, see section 5 of `docs/domain-adaptation/classification-and-decision-models.md`) |
 | index spans (embed + store, page/bbox provenance) | engine | the workspace's ingest embedder | `embedder=` |
 | group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
@@ -181,6 +181,9 @@ category.
   `evaluate_ingestion(..., table_labels=[{"pattern": <regex on the header row>, "label": "record"|"block"}])`
   (first match wins): a block table split per row, or a wide record table not split, is a failure, and tables no
   pattern matched are listed in `unlabelled_tables`.
+- **Every row of a record table should reach a unit.** After an ingest, `DocumentReport.unreached_table_rows` lists
+  each record table whose rows did not all reach a unit with their `TableRow` (with the `[ingest] warning:` line);
+  an empty list on your samples is the check. Rows listed there produced no record.
 - **How an embedded file finds its row.** An identifier is a token of 4+ characters containing a digit that appears
   on at most `IdentifierRule.max_rows` table rows and in at most `max_files` embedded files (defaults 3 and 3; set
   them with `IngestionTuning(identifier=IdentifierRule(...))`). A file whose identifiers match no row, or too many,
@@ -196,7 +199,11 @@ category.
 - **One record over several rows, or one entity over many units.** The extractor sees one unit at a time. A record
   whose continuation rows sit under its main row needs a `unit_grouper=` that keeps them together, or a
   `document_hook=` that reads the whole table with `table_rows` and writes the grouped record. An entity several units
-  mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key.
+  mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key; an edge to it
+  emitted from each unit lands once, because the default writer keeps one edge per (type, from node, to node)
+  (PS-63). A `writer=` of your own gets the same with `kg_write(ws, nodes, edges, ensure_edges=True)`; leave the flag
+  off only when parallel edges of one type between the same two nodes are meant to stay (each with its own
+  provenance).
 - **Resume a long ingest instead of restarting it.** Pass a LangGraph checkpointer (yours, for example a Postgres
   saver; the engine ships none) and a `run_id` naming the run:
   `await pipeline.aingest(ws, sources, cache_dir=CACHE, checkpointer=saver, run_id=batch_id)`. If the process stops,
@@ -205,8 +212,9 @@ category.
   sources, raise `ValueError`: a new run gets a new id. The graph state is built-in values only, so it works under
   LangGraph's strict msgpack mode; a tuple inside a record's free-form `props` comes back as a list after a resume.
 - **Re-ingest.** Parsing is cached by content hash, so re-running is cheap. The default writer upserts nodes by their
-  key but is not fully idempotent for edges (engine gap G8): when a source is re-run, remove its previous records
-  first (`kg_delete` on the provenance your records carry, such as a per-source property) and write again.
+  key and writes each edge once, updating its props, so a re-run leaves no duplicates. It does not remove what a
+  source no longer yields: when a re-run should drop records, remove its previous records first (`kg_delete` on the
+  provenance your records carry, such as a per-source property) and write again.
 
 ## 6. Verify (definition of done)
 

{rag_wright-0.16.1 → rag_wright-0.17.1}/src/rag_wright/api/kg.py

@@ -62,11 +62,12 @@ def kg_update(ws: WorkspaceHandle, type_name: str, *, set: dict, where: Optional
     return ws._store.kg_update(type_name, set=set, where=where, key_range=key_range)
 
 
-def kg_write(ws: WorkspaceHandle, nodes: list, edges: Any = ()) -> None:
+def kg_write(ws: WorkspaceHandle, nodes: list, edges: Any = (), *, ensure_edges: bool = False) -> None:
     """Upsert typed `nodes` + create typed `edges` in one transaction (see `Store.kg_write`). `nodes`/`edges` are
     `KgNode`/`KgEdge` (from `rag_wright.api`); the store encodes each field per its pack-declared type. A node type
-    the workspace's schema does not declare fails the write."""
-    ws._store.kg_write(nodes, edges)
+    the workspace's schema does not declare fails the write. `ensure_edges=True` (PS-63) writes at most one edge per
+    (type, from node, to node), the last props winning: what a domain `writer=` passes to match the default writer."""
+    ws._store.kg_write(nodes, edges, ensure_edges=ensure_edges)
 
 
 def kg_edges(ws: WorkspaceHandle, from_type: Optional[str] = None, *, where: Optional[dict] = None,

{rag_wright-0.16.1 → rag_wright-0.17.1}/src/rag_wright/capabilities/rlm_chunking.py

@@ -25,6 +25,8 @@ from __future__ import annotations
 
 import asyncio
 import json
+import logging
+import re
 from pathlib import Path
 from typing import Any, Optional, Protocol, runtime_checkable
 
@@ -43,6 +45,8 @@ from rag_wright.models.profiles import ModelRole, model_for
 from rag_wright.models.seam import build_structured
 from rag_wright.skills.rlm.agent import build_rlm_agent, rlm_interpreter_session
 
+_log = logging.getLogger(__name__)
+
 DEFAULT_TOKEN_CAP = 20_000
 DEFAULT_SUMMARY_CONCURRENCY = 8  # in-flight summary calls (backpressure); summaries are network-bound
 _SUMMARIZE_PROMPT = "Summarize this document chunk in one or two sentences, factually:"
@@ -518,10 +522,70 @@ def _estimate_tokens(text: str) -> int:
     return max(1, len(text) // 4)
 
 
+_TABLE_LINE = re.compile(r"[ \t]*\|")
+_TABLE_SEPARATOR = re.compile(r"[ \t]*\|?[ \t:|-]*-[ \t:|-]*\|?[ \t]*")
+
+
 def _hard_split(text: str, token_cap: int) -> list[str]:
-    """Split text that alone exceeds the cap into <= cap-sized pieces (by chars)."""
-    cap_chars = token_cap * 4
-    return [text[i : i + cap_chars] for i in range(0, len(text), cap_chars)]
+    """Split text that alone exceeds the cap into pieces under the cap (by chars).
+
+    PS-60 (TexWright): each cut falls at the last blank line in the window's second half, else the last line break,
+    else the last space, and only a run with none of them is cut at a fixed length; the one whitespace character cut
+    at is dropped. A cut inside a pipe table repeats the table's header and separator rows at the top of the next
+    piece, so every piece is a well-formed table and every row stays whole: the old fixed-length cut fell mid-row,
+    and the rows after it lost their table layout, their own spans and their `TableRow`. Pieces stay strictly under
+    the cap, so a cached manifest holding a cap-length piece can only come from the old cut (`_stale_split`)."""
+    cap = token_cap * 4
+    pieces: list[str] = []
+    rest = text
+    floor = 0  # the carried header's length: a cut must fall after it, or the split would not advance
+    while len(rest) >= cap:
+        window = rest[: cap - 1]
+        blank = window.rfind("\n\n", max(floor, cap // 2))
+        line = window.rfind("\n", floor + 1)
+        space = max(window.rfind(" ", floor + 1), window.rfind("\t", floor + 1))
+        if blank > floor:
+            end, start = blank, blank + 2
+        elif line > floor:
+            end, start = line, line + 1
+        elif space > floor:
+            end, start = space, space + 1
+        else:
+            end = start = cap - 1
+        piece, rest = rest[:end], rest[start:]
+        pieces.append(piece)
+        header = _continued_table_header(piece, rest)
+        if header and len(header) < cap // 2:
+            rest, floor = f"{header}\n{rest}", len(header)
+        else:
+            floor = 0
+    if rest:
+        pieces.append(rest)
+    return pieces
+
+
+def _stale_split(manifest: ChunkManifest) -> bool:
+    """PS-60: whether a cached manifest holds a table cut by the old fixed-length hard split, which made every piece
+    but the last exactly the cap long; `_hard_split` now keeps pieces under it. Only a piece holding table rows
+    counts: an old prose split keeps its cached chunks (re-chunking it would change nothing a reader relies on)."""
+    cap = manifest.token_cap * 4
+    return any(len(c.text) == cap and any(_TABLE_LINE.match(ln) for ln in c.text.split("\n"))
+               for c in manifest.chunks[:-1])
+
+
+def _continued_table_header(piece: str, rest: str) -> str:
+    """The header and separator rows of the pipe table a cut fell inside (`piece` ends in it and `rest` continues
+    it), or "" when the cut is not inside a table or the table's header cannot be told."""
+    if not _TABLE_LINE.match(rest.split("\n", 1)[0]):
+        return ""
+    lines = piece.split("\n")
+    i = len(lines)
+    while i > 0 and _TABLE_LINE.match(lines[i - 1]):
+        i -= 1
+    block = lines[i:]
+    if len(block) >= 2 and _TABLE_SEPARATOR.fullmatch(block[1]):
+        return f"{block[0]}\n{block[1]}"
+    return ""
 
 
 _SEP = "\n\n"
@@ -745,7 +809,10 @@ def _chunk_prepare(parsed: ParsedDocument, cache_dir: Path, discoverer: Optional
     cache_dir.mkdir(parents=True, exist_ok=True)
     manifest_path = cache_dir / f"{parsed.source_doc_id}.{parsed.content_hash[:16]}.chunks.json"
     if manifest_path.exists():  # content-hash gate
-        return ChunkManifest.model_validate_json(manifest_path.read_text(encoding="utf-8")), manifest_path, None, None
+        cached = ChunkManifest.model_validate_json(manifest_path.read_text(encoding="utf-8"))
+        if not _stale_split(cached):
+            return cached, manifest_path, None, None
+        _log.info("re-chunking %s: its cached chunks cut a table mid-row (PS-60)", parsed.source_doc_id)
     document = _as_content_view(load_document(parsed))  # issue 0014: chunk the reading-order body (incl. tables)
     disc = discoverer if discoverer is not None else SeamBoundaryDiscoverer(token_cap=token_cap)
     return None, manifest_path, document, disc