rag-wright 0.16.0__tar.gz → 0.17.0__tar.gz

{rag_wright-0.16.0 → rag_wright-0.17.0}/.claude/skills/building-an-ingestion-capability/SKILL.md

@@ -22,13 +22,13 @@ Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`)
 |---|---|---|---|
 | parse (PDF, Office, spreadsheets incl. hidden sheets, HTML, Markdown, images and scans; embedded files) | engine | docling, content-hash cached; images and scanned pages are OCR'd, and only degraded pages escalate to the vision model (the `VISION_OCR` role) | (none) |
 | chunk | engine default, domain may override | structural boundaries; a model refines only an over-cap section (domain-neutral prompt) | `chunk_model=`, `chunk_discoverer=` (e.g. `default_chunk_discoverer(guidance=...)` to say what a coherent unit is in your documents) |
-| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` |
+| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` (to adjust the default rather than replace it, wrap `pack_sdk.segment_layout`) |
 | tag spans (soft tags) | domain, optional | none | `span_tagger=` (to compare models behind it, see section 5 of `docs/domain-adaptation/classification-and-decision-models.md`) |
 | index spans (embed + store, page/bbox provenance) | engine | the workspace's ingest embedder | `embedder=` |
 | group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
 | choose each unit's representative span (its `anchor` + leading tag) | **domain decision**, optional | the grouper's choice: the first member, often a heading | `unit_representative=` |
 | extract records from a unit | **domain (required)** | (none) | the `extractor` argument |
-| write records | engine default | `kg_write` | `writer=` |
+| write records | engine default | `kg_write` | `writer=` (returns the number of records now in the graph, or `None`) |
 | per-document follow-up (e.g. an entity graph) | domain, optional | none | `document_hook=` |
 | `Document` node, embedded children (`EmbeddedIn` / `AttachedTo`), progress, dead-lettering | engine | always on | (none) |
 
@@ -181,6 +181,9 @@ category.
   `evaluate_ingestion(..., table_labels=[{"pattern": <regex on the header row>, "label": "record"|"block"}])`
   (first match wins): a block table split per row, or a wide record table not split, is a failure, and tables no
   pattern matched are listed in `unlabelled_tables`.
+- **Every row of a record table should reach a unit.** After an ingest, `DocumentReport.unreached_table_rows` lists
+  each record table whose rows did not all reach a unit with their `TableRow` (with the `[ingest] warning:` line);
+  an empty list on your samples is the check. Rows listed there produced no record.
 - **How an embedded file finds its row.** An identifier is a token of 4+ characters containing a digit that appears
   on at most `IdentifierRule.max_rows` table rows and in at most `max_files` embedded files (defaults 3 and 3; set
   them with `IngestionTuning(identifier=IdentifierRule(...))`). A file whose identifiers match no row, or too many,

rag_wright-0.17.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.17.0"
+}

{rag_wright-0.16.0 → rag_wright-0.17.0}/CHANGELOG.md

@@ -4,6 +4,28 @@ All notable changes to RAG_Wright are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.17.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.16.1...0.17.0) (2026-11-05)
+
+
+### Features
+
+* **PS-61:** the default segmenter is on pack_sdk, so a pack can wrap it ([bbb179d](https://github.com/fzaidi-dai/RAG_Wright/commit/bbb179d64e5fc905ada2ddc57367cdd2cc2ca5b6))
+* **PS-62:** a record table's rows that reach no unit are reported, never silent ([632fefb](https://github.com/fzaidi-dai/RAG_Wright/commit/632fefb390e11014a0dcb4cdd16956697a92ea58))
+
+
+### Bug Fixes
+
+* **PS-60:** a table over the chunk cap keeps every row's span and TableRow ([acd9ce5](https://github.com/fzaidi-dai/RAG_Wright/commit/acd9ce5917de1daac6ac47ea0a09ca975fcdf556))
+
+## [0.16.1](https://github.com/fzaidi-dai/RAG_Wright/compare/0.16.0...0.16.1) (2026-11-01)
+
+
+### Bug Fixes
+
+* **PS-57:** a retried write no longer loses a document's clauses; the conflict budget is configurable ([5d209d7](https://github.com/fzaidi-dai/RAG_Wright/commit/5d209d7eb1ab4bad93cd7026e81e79d9b6852fe5))
+* **PS-58:** a record writer reports the count that landed ([a7c384c](https://github.com/fzaidi-dai/RAG_Wright/commit/a7c384cfab59832f9714bac9996f91a29340ed71))
+* **PS-59:** the store's client keeps keep-alive connections, one session per thread ([f644ba3](https://github.com/fzaidi-dai/RAG_Wright/commit/f644ba3f3f37f12102037405b9a58567098358b3))
+
 ## [0.16.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.15.0...0.16.0) (2026-10-28)
 
 

{rag_wright-0.16.0 → rag_wright-0.17.0}/CLAUDE.md

@@ -55,7 +55,13 @@ The failure mode to avoid, which has happened before: a command fails with "modu
 
 ## Start of every session
 
-> **CURRENT STATE (2026-10-28): `rag-wright` 0.15.0 is RELEASED** (PR #18; PS-51 the frontend skill draws the shell and
+> **CURRENT STATE (2026-11-01): `rag-wright` 0.16.1 is RELEASED** (PR #20; RuleWright's two requests and a finding
+> from them: PS-57 a retried contracts-pack write no longer loses a document's clauses, the ArcadeDB conflict budget is
+> configurable (default 20 attempts / 2 s cap) and a racing UPSERT's duplicate key is retried; PS-58 a `RecordWriter`
+> returns the count that landed; PS-59 the store's client keeps keep-alive connections, one session per thread). Next:
+> PS-48, EV-7, PS-30, PS-33. Before it, **`rag-wright` 0.16.0 is RELEASED** (PR #19; TexWright's requests: PS-54 a `Contains`
+> substring `where` value, PS-55 only the BGE models on Metal under one process lock (ADR-0140); PS-56 the BGE reranker
+> scores again on transformers 5). Before it, **`rag-wright` 0.15.0 is RELEASED** (PR #18; PS-51 the frontend skill draws the shell and
 > ships its scaffold, RuleWright's request; PS-52 one workspace embedder under concurrent first use and PS-53 KG calls refuse
 > a name that is not an identifier, TexWright's requests). Before it, **`rag-wright` 0.14.1 is RELEASED** (PR #17; TexWright's two requests: PS-49 a retried
 > ArcadeDB write conflict no longer prints as an error, PS-50 `ensure_pack_schema` declares a property a pack gains on an

{rag_wright-0.16.0 → rag_wright-0.17.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rag-wright
-Version: 0.16.0
+Version: 0.17.0
 Summary: A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime.
 Project-URL: Homepage, https://github.com/fzaidi-dai/RAG_Wright
 Project-URL: Repository, https://github.com/fzaidi-dai/RAG_Wright
@@ -42,6 +42,7 @@ Requires-Dist: pydantic>=2.0
 Requires-Dist: pypdfium2>=5.11
 Requires-Dist: pyshacl>=0.40.1
 Requires-Dist: python-dotenv>=1.2
+Requires-Dist: requests>=2.34.2
 Requires-Dist: scikit-learn>=1.9.1
 Requires-Dist: sentence-transformers>=6.1.0
 Requires-Dist: transformers<5.9,>=5.4

rag_wright-0.17.0/docs/adr/0141-the-hard-split-cuts-between-lines-and-repeats-a-tables-header.md

@@ -0,0 +1,38 @@
+# ADR-0141: The cap's hard split cuts between lines and repeats a table's header
+
+**Status:** accepted · **Date:** 2026-11-05 · **Related:** PS-60, TexWright's request
+`2026-11-05-export-the-default-segmenter-on-pack-sdk.md`, ADR-0069 (tables chunked in reading order), ADR-0124
+(ING-2 the default segmenter, ING-7 `table_rows`), ADR-0136 (spreadsheet tables glued across blank rows)
+
+## Context
+
+A chunk is a run of whole parsed items, and the token cap is enforced last: a section over the cap that the
+discoverer cannot subdivide (a single table is one item) is hard-split. The split cut every `token_cap * 4`
+characters, wherever that fell. In a table it fell mid-row, and three things followed. The table item was found
+whole in no chunk, so `chunk_layouts` placed it nowhere, and the unplaced item also stopped every later item in the
+document from getting its parsed layout. Each piece then fell back to the text layout: the first piece still
+opened with the table's header, but a continuation piece began mid-row, read as one paragraph, and gave ONE span to
+all its rows. With no row spans, no unit carried a `TableRow`, so a record extractor never saw those rows, and the
+ingest reported success. TexWright lost 4 of 140 records this way; a 140-record, 76-column synthetic workbook lost
+205 of 280 rows.
+
+## Decision
+
+- The hard split cuts at the last blank line in the second half of the window, else the last line break, else the
+  last space, and only a run with none of them at a fixed length. The whitespace character it cuts at is dropped.
+  Pieces stay strictly under the cap.
+- A cut inside a pipe table repeats the table's header and separator rows at the top of the next piece, so every
+  piece is a well-formed table, each row is whole in one chunk, and the segmenter, the unit grouper's record-table
+  test and the row-to-span map work unchanged on every piece. A header half the cap or longer is not repeated.
+- `chunk_layouts` skips an item that no remaining chunk holds whole, instead of blocking the items after it.
+- A cached chunk manifest holding a piece exactly the cap long with table lines in it can only come from the old
+  cut; it is re-chunked once, with a log line. An old prose split keeps its cached chunks.
+
+## Consequences
+
+- Live, the workbook above yields 280 of 280 row records (75 before), and every span is one row.
+- A document with a section over the cap gets different chunk boundaries, so different `chunk_id`s and spans when
+  re-ingested; the `chunk_id` scheme is unchanged and documents under the cap are untouched. A repeated header
+  appears once per continuation chunk in the canonical document text.
+- Whether a pack may build on the default segmenter (PS-61) and whether a record table's lost rows are reported
+  (PS-62) are separate tasks.

{rag_wright-0.16.0 → rag_wright-0.17.0}/docs/adr/README.md

@@ -196,4 +196,5 @@ framing that still appears in older ADRs and the root `SPEC.md`/`plan.md`.
 0137 A generated answer states the evidence's support every time, and "none" is an abstention ·
 0138 PDF parses run one at a time, and the engine reaches pdfium through one locked door ·
 0139 The agent harness substrate is engine scope ·
-0140 One model family on Metal, under one process lock
+0140 One model family on Metal, under one process lock ·
+0141 The cap's hard split cuts between lines and repeats a table's header

{rag_wright-0.16.0 → rag_wright-0.17.0}/docs/api/README.md

@@ -8,7 +8,7 @@
 
 The product's view of the engine: the store connection, chosen models (`models`: `ModelRole` value -> model alias), the embedding profile, the `options` catalog (generic `ingest` knobs + each pack's options under `packs`), and `pack`: the path to the domain pack `.ttl` whose KG types `open_workspace` creates (None = only the neutral engine types: Chunk, Entity, Relationship, Mentions, Span, Document, EmbeddedIn, AttachedTo). Defaults just work; override only to trade quality/cost/latency. Implementation details (ArcadeDB, BGE) never cross this boundary.
 
-### `StoreConfig(host: 'str', port: 'str', user: 'str', password: 'str', backend: 'str' = 'arcadedb', protocol: 'str' = 'http') -> None`
+### `StoreConfig(host: 'str', port: 'str', user: 'str', password: 'str', backend: 'str' = 'arcadedb', protocol: 'str' = 'http', conflict_attempts: 'Optional[int]' = None, conflict_backoff_cap_s: 'Optional[float]' = None) -> None`
 
 How to reach the KG/retrieval store. `backend` selects the implementation (only `arcadedb` today; a new backend -- e.g. Neo4j -- is added here, invisibly to the product).
 
@@ -216,9 +216,9 @@ A stage that raises routes to `dead_letter`, which ends the document with its `D
 
 The result of `IngestionPipeline.aingest`: one `DocumentReport` per document (embedded children included). `failed` counts the dead-lettered documents and `succeeded` the rest (properties).
 
-### `DocumentReport(doc_id: 'str', parent_doc_id: 'Optional[str]' = None, chunks: 'int' = 0, spans: 'int' = 0, units: 'int' = 0, records: 'int' = 0, extraction_failures: 'list[dict]' = <factory>, span_failures: 'list[dict]' = <factory>, skipped_hidden_sheets: 'list[str]' = <factory>, embedded_skipped: 'list[str]' = <factory>, children: 'list[str]' = <factory>, links: 'dict[str, int]' = <factory>, unmapped_links: 'int' = 0, dead_letter: 'Optional[str]' = None) -> None`
+### `DocumentReport(doc_id: 'str', parent_doc_id: 'Optional[str]' = None, chunks: 'int' = 0, spans: 'int' = 0, units: 'int' = 0, records: 'int' = 0, extraction_failures: 'list[dict]' = <factory>, span_failures: 'list[dict]' = <factory>, skipped_hidden_sheets: 'list[str]' = <factory>, embedded_skipped: 'list[str]' = <factory>, children: 'list[str]' = <factory>, links: 'dict[str, int]' = <factory>, unmapped_links: 'int' = 0, unreached_table_rows: 'list[dict]' = <factory>, dead_letter: 'Optional[str]' = None) -> None`
 
-What ingesting one document did. Counts: `chunks`, `spans` (indexed), `units` (extracted), `records` (nodes written). `extraction_failures` (`{unit, anchor, reason}`) / `span_failures` (`{span_id, reason}`): the units / spans whose extraction or indexing failed; the rest of the document still lands. `skipped_hidden_sheets`: hidden spreadsheet sheets left out (`IngestSource.include_hidden_sheets=False`). `children`: the ids of embedded files and PDF attachments ingested as child documents (`parent_doc_id` is set on theirs); `embedded_skipped`: embedded files that could not be ingested. `links`: `AttachedTo` record links written, by confidence; `unmapped_links`: links whose record row could not be found. `dead_letter`: why the whole document failed (None when it landed).
+What ingesting one document did. Counts: `chunks`, `spans` (indexed), `units` (extracted), `records` (nodes written). `extraction_failures` (`{unit, anchor, reason}`) / `span_failures` (`{span_id, reason}`): the units / spans whose extraction or indexing failed; the rest of the document still lands. `skipped_hidden_sheets`: hidden spreadsheet sheets left out (`IngestSource.include_hidden_sheets=False`). `children`: the ids of embedded files and PDF attachments ingested as child documents (`parent_doc_id` is set on theirs); `embedded_skipped`: embedded files that could not be ingested. `links`: `AttachedTo` record links written, by confidence; `unmapped_links`: links whose record row could not be found. `unreached_table_rows` (PS-62): one entry per record table -- a table at least one of whose rows reached a unit carrying its `TableRow` -- whose data rows did not ALL reach one, `{table_ref, sheet, reached, total, missing}` (`missing`: the 1-based data-row indices, at most the first 1,000); the document still lands, so read this, or the `[ingest] warning:` line, to know rows were lost. `dead_letter`: why the whole document failed (None when it landed).
 
 ### `IngestionEvaluation(documents: 'list[DocumentEvaluation]' = <factory>, failures: 'list[str]' = <factory>, unlabelled_tables: 'list[str]' = <factory>, labels: 'Optional[LabelEvaluation]' = None, warnings: 'list[str]' = <factory>) -> None`
 
@@ -304,9 +304,9 @@ PS-R3: which member span represents a unit -- a DOMAIN decision. The engine appl
 
 One unit -> the domain's typed records (required; the domain-specific step).
 
-### `RecordWriter: (source_doc_id: 'str', extractions: 'Sequence[UnitExtraction]') -> 'Awaitable[None]'`
+### `RecordWriter: (source_doc_id: 'str', extractions: 'Sequence[UnitExtraction]') -> 'Awaitable[Optional[int]]'`
 
-Optional: persist a document's extractions. The engine default writes them with `kg_write`.
+Optional: persist a document's extractions. The engine default writes them with `kg_write`. Return the number of records now in the graph (written, or found already written); `None` lets the engine count the record nodes it handed over (RuleWright, 2026-10-31: only the writer knows what landed).
 
 ### `BoundaryDiscoverer: (**kwargs)`
 

{rag_wright-0.16.0 → rag_wright-0.17.0}/docs/api/pack_sdk.md

@@ -182,7 +182,7 @@ Group into units (checked), attach table rows, extract each unit concurrently (p
 
 #### `IngestionStages.write(self, sd: 'Any', extractions: 'list') -> 'int'`
 
-Persist the records (the domain's writer, else `kg_write`). Returns the number of record nodes.
+Persist the records (the domain's writer, else `kg_write`). Returns the number that landed: the writer's own count when it returns one; else the record nodes handed over, which the default `kg_write` lands in one transaction (it raises rather than land part of them).
 
 #### `IngestionStages.write_document_node(self, doc_id: 'str', *, parent_id: 'Optional[str]', filename: 'str', media_type: 'str', sha256: 'str', metadata: 'Optional[dict]' = None) -> 'None'`
 
@@ -492,6 +492,10 @@ ASYNC twin of `chunk_texts` (SEG-2/SEG-5): awaits the discoverer's async boundar
 
 Validate a `documents` scope against `store.known_document_ids()` BEFORE any retrieval spends (mirrors issue 0007's `_validate_sources`). `None` (whole store) is not validated. `[]` (scope-to-nothing) is a valid empty scope, not an error -- the retrieval surfaces short-circuit it. A non-empty list with any id absent from the store raises `UnknownDocumentError`. A store without `known_document_ids` (a minimal fake) is treated as un-validatable and passes through.
 
+### `segment_layout(chunk_id: 'str', text: 'str', layout: 'Sequence[LayoutItem]', *, min_alnum: 'Optional[int]' = None, tuning: 'Optional[IngestionTuning]' = None) -> 'list[Span]'`
+
+Segment one chunk into spans (the engine's default `Segmenter`). The fragment floor is `min_alnum`, else `tuning.min_fragment_alnum`, else the default.
+
 ### `capability_impl(name: 'str') -> 'Callable[..., Any]'`
 
 Resolve a capability name to its invoke factory via the manifest `impl_ref`. Raises `KeyError` if the name is unknown to the ARD catalog, `NotImplementedError` if it declares no `impl_ref` (not invokable by name).

{rag_wright-0.16.0 → rag_wright-0.17.0}/docs/configuration.md

@@ -45,6 +45,14 @@ configuration change takes effect without a restart ([`workspaces.md`](workspace
 | `password` | `str` | — | password |
 | `backend` | `str` | `"arcadedb"` | store implementation (only `arcadedb` today) |
 | `protocol` | `str` | `"http"` | `http` locally; `https` for a remote/Modal store |
+| `conflict_attempts` | `int \| None` | `None` | attempts per write when ArcadeDB reports a write conflict (`None`: `ARCADEDB_CONFLICT_ATTEMPTS`, else 20) |
+| `conflict_backoff_cap_s` | `float \| None` | `None` | the longest wait between those attempts, in seconds (`None`: `ARCADEDB_CONFLICT_BACKOFF_CAP_S`, else 2) |
+
+Concurrent writers that touch the same page get a write conflict; the store retries the write (the failed
+transaction is rolled back whole) with a doubling wait from 10 ms up to the cap. Two writers creating the same new
+node by `kg_write` get a duplicate key instead, which is retried the same way. A write that is still conflicting
+after the last attempt raises, about 25 s after the first one with the defaults. Raise the budget when more than
+eight documents write into one graph at once.
 
 `StoreConfig.from_env()` builds one from `ARCADEDB_HOST` (default `localhost`), `ARCADEDB_PORT` (required: no
 default, so an unset variable cannot reach another instance on the machine),
@@ -284,6 +292,7 @@ Prefer config; use env for secrets and for non-API callers. The common ones the
 | variable | purpose |
 |---|---|
 | `ARCADEDB_HOST` / `ARCADEDB_PORT` / `ARCADEDB_USER` / `ARCADEDB_PASSWORD` / `ARCADEDB_DATABASE` / `ARCADEDB_PROTOCOL` | store connection: `StoreConfig.from_env()` reads them (and the scripts do); `open_workspace` itself reads only the `StoreConfig` it is given |
+| `ARCADEDB_CONFLICT_ATTEMPTS` / `ARCADEDB_CONFLICT_BACKOFF_CAP_S` | the write-conflict retry budget when `StoreConfig.conflict_attempts` / `conflict_backoff_cap_s` are unset |
 | `OPENROUTER_API_KEY` / `OPENROUTER_BASE_URL` / `OPENROUTER_PROVIDER` / `OPENROUTER_ALLOW_FALLBACKS` | OpenRouter access + provider routing (`OPENROUTER_SORT` / `OPENROUTER_PROVIDER_ORDER`: the contracts pack's docling-graph extraction only) |
 | `RAG_SERVING` / `VLLM_BASE_URL` / `VLLM_API_KEY` / `STACK_URL` | serving backend (openrouter \| vllm) + self-hosted endpoint |
 | `RAG_MODEL_<ROLE>` (e.g. `RAG_MODEL_GENERAL`, `RAG_MODEL_VISION_OCR`) / `RAG_MODEL_ALL` / `RAG_GRAPH_EXTRACT_MODEL` / `RAG_DECISION_MODEL` | model overrides (one role / all roles / graph extraction / decision-model profile) |

{rag_wright-0.16.0 → rag_wright-0.17.0}/docs/domain-adaptation/kg-construction.md

@@ -34,6 +34,20 @@ The reference pack's contract pipeline is one instantiation of the same stages (
 legal segmenter, a function tagger, a provision grouper, a decision-model boundary decider, its clause extractor and
 its typed writer. Read it as an example of hooks, not as the engine's definition of ingestion.
 
+A hook you pass replaces the default for the whole pipeline, for every document it ingests. When your documents
+need the default's behaviour in most places and your own rule in a few, build on the default rather than replace it:
+the default segmenter is `rag_wright.pack_sdk.segment_layout` (and the default chunk discoverer
+`StructuralModelFallbackDiscoverer`), so your segmenter can call it and adjust only the spans it gets wrong:
+
+```python
+from rag_wright.pack_sdk import segment_layout
+
+
+def segmenter(chunk_id, text, layout):
+    spans = segment_layout(chunk_id, text, layout)  # the engine's evaluated segmentation
+    return spans  # split or merge here only the spans your domain reads differently; keep them tiling the text
+```
+
 ## The stages
 
 ```
@@ -59,7 +73,9 @@ parse → chunk → segment (+ optional tag) → index  (embed each span → the
   A unit holding exactly one data row of a parsed table carries that row as `Unit.table_row` (exact cells).
 - **extract**: your `extractor` turns each unit into **typed records** (`UnitExtraction`: `KgNode`s and `KgEdge`s
   in your pack's schema). This is the domain-specific step.
-- **write**: persist the records (your `writer`, else `kg_write`), then a generic `Document` node per document.
+- **write**: persist the records (your `writer`, else `kg_write`), then a generic `Document` node per document. Your
+  `writer` returns the number of records now in the graph (the report's `records`), or `None` to let the engine count
+  the record nodes it handed over.
 - **entity graph + resolve**: not run by default. Run your entity extraction and resolution in a `document_hook`
   (see [entity resolution](entity-resolution.md)).
 
@@ -167,8 +183,10 @@ the run goes on. Every node and edge type your extractor returns must be declare
 
 `aingest` returns an `IngestionReport` (`succeeded`, `failed`, `documents`). Each `DocumentReport` has the counts
 (`chunks`, `spans`, `units`, `records`), `extraction_failures`, `span_failures`, `dead_letter` (the reason, if the
-document failed), `skipped_hidden_sheets`, `embedded_skipped`, `children`, and the `links` / `unmapped_links` of its
-embedded files. Nothing is skipped silently.
+document failed), `skipped_hidden_sheets`, `embedded_skipped`, `children`, the `links` / `unmapped_links` of its
+embedded files, and `unreached_table_rows`: each record table some of whose data rows reached no unit carrying their
+`TableRow`, so produced no record (`reached`, `total` and the `missing` row indices; also printed as an
+`[ingest] warning:` line). Nothing is skipped silently.
 
 ## The two graphs (one store)
 

{rag_wright-0.16.0 → rag_wright-0.17.0}/docs/specs/public-surface/plan.md

@@ -128,4 +128,10 @@ in two at the PS-6 gate so the store rewrite is its own visible task. One task,
 | PS-54 | TexWright's request (`docs/engine-requests/2026-10-19-substring-match-in-the-kg-filter-grammar.md`, their repo): the KG filter grammar had no substring predicate ("composition contains UHMWPE" over free text). `Contains(text, case_sensitive=False)`, a `where` value beside `Range` on every KG call that takes `where`: a literal `indexOf` test (not `LIKE`, whose `%`, `_` and `?` are wildcards, measured live), case folded on both sides in the store, a field with no value never matches, empty text refused | 8 hermetic tests (red before) + a live test (case both ways, literal wildcards, quotes and backslashes, accents, missing values, with `Range`, `kg_count`, a `kg_edges` traversal filtered at all three points); full hermetic (2353) and live store (42) suites green | done 2026-10-29 |
 | PS-55 | TexWright's request (`docs/engine-requests/2026-10-25-two-model-families-in-one-process-abort-on-metal.md`, their repo): Metal is not safe for concurrent use and the engine put three families on it automatically (BGE-M3, the BGE reranker, Docling's layout/table/RapidOCR models), then ran them on different threads (an ingest embeds while it parses; a server embeds a query while an upload parses), so a process aborted in Metal. RuleWright had measured and pinned it in their own seam (T-7.25/T-7.46), never adopted here. `rag_wright.util.devices` (ADR-0140): only the BGE family on Metal, every call there under one process lock (`METAL_LOCK`); Docling and the pack classifiers on CPU where Metal would be picked (`pack_sdk.torch_device`); explicit device variables win; CUDA/CPU hosts unchanged; `docs/workspaces.md` corrected | red, torch 2.14.1 (TexWright's env, this engine tree): embedder + Docling in two threads, 6/6 child runs died in seconds (5 SIGABRT with their assertion, 1 SIGSEGV; OCR and PDF); torch 2.12.1 never failed. Green, same interpreter: 9/9 runs survived 150s each (OCR, PDF, two embedders). Full hermetic (2361) and live embed/parse/store (60) suites green. 8 hermetic tests (4 behaviour tests red before); Docling output identical CPU vs Metal | done 2026-10-29 |
 | PS-56 | Found during PS-55: `BGEReranker` failed on every call (`XLMRobertaTokenizer has no attribute prepare_for_model`): FlagEmbedding's `FlagAutoReranker.compute_score` calls a tokenizer method transformers 5 removed, so the reference pack's `intra_document_qa` rerank (and RuleWright's copy) and any product composing `BGEReranker` over `span_search` hits were broken. The reranker is now a `sentence_transformers.CrossEncoder` (already a dependency) over the same `bge-reranker-v2-m3`, raw logits, 512 tokens per pair; same `BGEReranker` API, device and lock (PS-55) | red: the live reranker test failed on the tokenizer error. Green: it passes, plus a one-pair and a >512-token case; CrossEncoder's logits equal the model card's plain-transformers recipe (max abs diff 0.0, truncated passage included); embedder + reranker on Metal under the shared lock, torch 2.14.1: 3/3 runs survived 150s, 590 reranks and 586 embeds, none failing (before: the rerank thread died on its first call); full hermetic (2361) green | done 2026-10-29 |
+| PS-57 | RuleWright's request (`docs/engine-requests/2026-10-31-a-write-conflict-loses-a-documents-clauses-and-your-pack-carries-the-defect.md`, their repo): at 4 concurrent documents, 2 of 15 reported `ingested` with zero clauses. The contracts pack's `write` node popped the pending extractions (filled by `extract_clauses`) before the store write landed, so the node's retry wrote nothing and reported success; it now reads them and releases them after the write. The conflict budget is reachable: `StoreConfig.conflict_attempts` / `conflict_backoff_cap_s`, else `ARCADEDB_CONFLICT_ATTEMPTS` / `ARCADEDB_CONFLICT_BACKOFF_CAP_S`; default raised 10 attempts / 0.5 s cap to 20 / 2 s. Found by the live probe: two writers creating the same new node get a duplicate key, never retried; `kg_write` and `write_graph` (UPSERT-only) now retry it | pack retry test through the real per-document graph (0/3 clauses landed before, 3/3 after); six store tests; live probe (clause-shaped transactions over shared value nodes, pooled sessions): old budget lost 38/2,400 at 4 and 8 writers, new lost 0/2,400 at the same wall-clock; live store suite 42 passed | done 2026-11-01 |
+| PS-58 | RuleWright's request (`docs/engine-requests/2026-10-31-ingestionstages-write-returns-a-count-derived-from-its-own-input.md`, their repo): `IngestionStages.write` returned the record nodes it was handed, and `RecordWriter` (declared `-> None`) could not say what landed. A writer now returns the number of records in the graph (`Optional[int]`; `None` keeps the engine's count); the default `kg_write` branch keeps the handed-over count, true because that write is one transaction (lands whole or raises). The contracts pack's writer returns the clauses in the graph and its `write` node reports that as `clauses` | three hermetic tests + the PS-57 retry test's count (red before, green after); live `build_ingestion` on two fixtures: default and a writer landing every other record, `records` equals the graph in all four documents | done 2026-11-01 |
+| PS-59 | Found during PS-57: the ArcadeDB client opens a new connection for every request (`requests.post`, no session); at 8 concurrent writers against Docker on macOS about two thirds of writes failed with `RemoteDisconnected`, and with a keep-alive session per thread the same run lost none. Give the store's client a pooled session | live probe at 8 writers: zero connection drops | done 2026-11-01: the store's client (`_PooledClient`) keeps one keep-alive `requests.Session` per thread and `ArcadeDBStore.close()` releases them; live, the same probe lost 374 of 800 transactions to `RemoteDisconnected` unpooled and 0 of 800 at 8 writers and 0 of 1,600 at 16 pooled |
+| PS-60 | TexWright (2026-11-05, `2026-11-05-export-the-default-segmenter-on-pack-sdk.md`): a table over the chunk cap was hard-split at a fixed length, mid-row; the rows after the cut lost their table layout (one span between them, no `TableRow`), and one item `chunk_layouts` could not place blocked every later item's layout. 4 of 140 records lost silently | the cut falls between lines and repeats the table's header in each continuation chunk; an unplaceable item no longer blocks later ones; a manifest cached with the old cut is re-chunked once; live: a 140-record x 76-column workbook yields 140 records | done 2026-11-05 (ADR-0141): live, the workbook (280 rows, two per record) gave 280/280 row records, 75/280 with the old cut |
+| PS-61 | TexWright's question in the same request: the default segmenter `segment_layout` is on neither public tier, so a pack cannot wrap it | export it on `rag_wright.pack_sdk` | done 2026-11-05: `pack_sdk.segment_layout` (the engine's own object); the wrap-the-default pattern in `kg-construction.md` and the ingestion skill; a pack segmenter delegating to it runs through `build_ingestion` |
+| PS-62 | TexWright, same request: the loss was silent (every `check_*` passed) | a record table whose rows do not each reach a unit with its `TableRow` is reported in the `IngestionReport` | done 2026-11-05: `DocumentReport.unreached_table_rows` (`table_ref`, `sheet`, `reached`, `total`, `missing`) and an `[ingest] warning:` line; a block table is not a loss; live, the pre-PS-60 cut on the workbook reports 75 of 280 rows reached (205 missing, matching the records lost), the new cut reports nothing |
 | PS-48 | Port the agent harness's substrate behaviour tests from RuleWright onto the engine's fabric fixture (ADR-0139 handover): of RuleWright's 553 harness-touching tests, 314 test the substrate (port), 213 RuleWright's composition (stay), 25 live (1 substrate: OpenRouter cost accounting). Seven gated sub-tasks: **a** shared `tests/harness/fabric.py` + scripting/slow/hanging/failing fake models + `InMemorySaver` helpers, then events, skills, clarify (~23); **b** models and errors (~42); **c** runs, confirm, continuation (~32); **d** decisions and rules (~51); **e** grading (~32); **f** seen and memory (~41); **g** workers, batch, dispatch, tiers, agent wiring, filesystem, interpreter, withhold + the live accounting test (~93, may split). Ambiguous tests split, not moved whole; a ported test failing on engine code is a substrate defect fixed in that sub-task; deadline globals via monkeypatch; no new dependencies. Follow-up (owner RuleWright): retire its substrate duplicates once ported | each sub-task's ported tests green on the fabric fixture | todo (proposed; awaiting go-ahead) |

{rag_wright-0.16.0 → rag_wright-0.17.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "rag-wright"
-version = "0.16.0"
+version = "0.17.0"
 description = "A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime."
 readme = "README.md"
 requires-python = ">=3.12,<3.13"
@@ -51,6 +51,7 @@ dependencies = [
     "pypdfium2>=5.11",
     "pyshacl>=0.40.1",
     "python-dotenv>=1.2",
+    "requests>=2.34.2",
     "scikit-learn>=1.9.1",
     "sentence-transformers>=6.1.0",
     "transformers>=5.4,<5.9",

{rag_wright-0.16.0 → rag_wright-0.17.0}/src/rag_wright/.agents/skills/building-an-ingestion-capability/SKILL.md

@@ -22,13 +22,13 @@ Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`)
 |---|---|---|---|
 | parse (PDF, Office, spreadsheets incl. hidden sheets, HTML, Markdown, images and scans; embedded files) | engine | docling, content-hash cached; images and scanned pages are OCR'd, and only degraded pages escalate to the vision model (the `VISION_OCR` role) | (none) |
 | chunk | engine default, domain may override | structural boundaries; a model refines only an over-cap section (domain-neutral prompt) | `chunk_model=`, `chunk_discoverer=` (e.g. `default_chunk_discoverer(guidance=...)` to say what a coherent unit is in your documents) |
-| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` |
+| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` (to adjust the default rather than replace it, wrap `pack_sdk.segment_layout`) |
 | tag spans (soft tags) | domain, optional | none | `span_tagger=` (to compare models behind it, see section 5 of `docs/domain-adaptation/classification-and-decision-models.md`) |
 | index spans (embed + store, page/bbox provenance) | engine | the workspace's ingest embedder | `embedder=` |
 | group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
 | choose each unit's representative span (its `anchor` + leading tag) | **domain decision**, optional | the grouper's choice: the first member, often a heading | `unit_representative=` |
 | extract records from a unit | **domain (required)** | (none) | the `extractor` argument |
-| write records | engine default | `kg_write` | `writer=` |
+| write records | engine default | `kg_write` | `writer=` (returns the number of records now in the graph, or `None`) |
 | per-document follow-up (e.g. an entity graph) | domain, optional | none | `document_hook=` |
 | `Document` node, embedded children (`EmbeddedIn` / `AttachedTo`), progress, dead-lettering | engine | always on | (none) |
 
@@ -181,6 +181,9 @@ category.
   `evaluate_ingestion(..., table_labels=[{"pattern": <regex on the header row>, "label": "record"|"block"}])`
   (first match wins): a block table split per row, or a wide record table not split, is a failure, and tables no
   pattern matched are listed in `unlabelled_tables`.
+- **Every row of a record table should reach a unit.** After an ingest, `DocumentReport.unreached_table_rows` lists
+  each record table whose rows did not all reach a unit with their `TableRow` (with the `[ingest] warning:` line);
+  an empty list on your samples is the check. Rows listed there produced no record.
 - **How an embedded file finds its row.** An identifier is a token of 4+ characters containing a digit that appears
   on at most `IdentifierRule.max_rows` table rows and in at most `max_files` embedded files (defaults 3 and 3; set
   them with `IngestionTuning(identifier=IdentifierRule(...))`). A file whose identifiers match no row, or too many,

{rag_wright-0.16.0 → rag_wright-0.17.0}/src/rag_wright/api/config.py

@@ -23,6 +23,10 @@ class StoreConfig:
     password: str
     backend: str = "arcadedb"
     protocol: str = "http"
+    # RuleWright (2026-10-31): the write-conflict retry budget. None: `ARCADEDB_CONFLICT_ATTEMPTS` /
+    # `ARCADEDB_CONFLICT_BACKOFF_CAP_S`, else 20 attempts with each wait capped at 2 s (about 25 s in all).
+    conflict_attempts: Optional[int] = None
+    conflict_backoff_cap_s: Optional[float] = None
 
     @classmethod
     def from_env(cls) -> "StoreConfig":

{rag_wright-0.16.0 → rag_wright-0.17.0}/src/rag_wright/api/workspace.py

@@ -131,7 +131,9 @@ def _build_store(config: EngineConfig, corpus: str, *, reset: bool, create: bool
     from rag_wright.store.arcadedb import ArcadeDBStore
 
     return ArcadeDBStore.from_config(sc.host, sc.port, sc.user, sc.password, database=corpus,
-                                     protocol=sc.protocol, reset=reset, pack_ttl=config.pack, create=create)
+                                     protocol=sc.protocol, reset=reset, pack_ttl=config.pack, create=create,
+                                     conflict_attempts=sc.conflict_attempts,
+                                     conflict_backoff_cap_s=sc.conflict_backoff_cap_s)
 
 
 def drop_workspace(config: EngineConfig, *, corpus: str) -> bool:

{rag_wright-0.16.0 → rag_wright-0.17.0}/src/rag_wright/capabilities/rlm_chunking.py

@@ -25,6 +25,8 @@ from __future__ import annotations
 
 import asyncio
 import json
+import logging
+import re
 from pathlib import Path
 from typing import Any, Optional, Protocol, runtime_checkable
 
@@ -43,6 +45,8 @@ from rag_wright.models.profiles import ModelRole, model_for
 from rag_wright.models.seam import build_structured
 from rag_wright.skills.rlm.agent import build_rlm_agent, rlm_interpreter_session
 
+_log = logging.getLogger(__name__)
+
 DEFAULT_TOKEN_CAP = 20_000
 DEFAULT_SUMMARY_CONCURRENCY = 8  # in-flight summary calls (backpressure); summaries are network-bound
 _SUMMARIZE_PROMPT = "Summarize this document chunk in one or two sentences, factually:"
@@ -518,10 +522,70 @@ def _estimate_tokens(text: str) -> int:
     return max(1, len(text) // 4)
 
 
+_TABLE_LINE = re.compile(r"[ \t]*\|")
+_TABLE_SEPARATOR = re.compile(r"[ \t]*\|?[ \t:|-]*-[ \t:|-]*\|?[ \t]*")
+
+
 def _hard_split(text: str, token_cap: int) -> list[str]:
-    """Split text that alone exceeds the cap into <= cap-sized pieces (by chars)."""
-    cap_chars = token_cap * 4
-    return [text[i : i + cap_chars] for i in range(0, len(text), cap_chars)]
+    """Split text that alone exceeds the cap into pieces under the cap (by chars).
+
+    PS-60 (TexWright): each cut falls at the last blank line in the window's second half, else the last line break,
+    else the last space, and only a run with none of them is cut at a fixed length; the one whitespace character cut
+    at is dropped. A cut inside a pipe table repeats the table's header and separator rows at the top of the next
+    piece, so every piece is a well-formed table and every row stays whole: the old fixed-length cut fell mid-row,
+    and the rows after it lost their table layout, their own spans and their `TableRow`. Pieces stay strictly under
+    the cap, so a cached manifest holding a cap-length piece can only come from the old cut (`_stale_split`)."""
+    cap = token_cap * 4
+    pieces: list[str] = []
+    rest = text
+    floor = 0  # the carried header's length: a cut must fall after it, or the split would not advance
+    while len(rest) >= cap:
+        window = rest[: cap - 1]
+        blank = window.rfind("\n\n", max(floor, cap // 2))
+        line = window.rfind("\n", floor + 1)
+        space = max(window.rfind(" ", floor + 1), window.rfind("\t", floor + 1))
+        if blank > floor:
+            end, start = blank, blank + 2
+        elif line > floor:
+            end, start = line, line + 1
+        elif space > floor:
+            end, start = space, space + 1
+        else:
+            end = start = cap - 1
+        piece, rest = rest[:end], rest[start:]
+        pieces.append(piece)
+        header = _continued_table_header(piece, rest)
+        if header and len(header) < cap // 2:
+            rest, floor = f"{header}\n{rest}", len(header)
+        else:
+            floor = 0
+    if rest:
+        pieces.append(rest)
+    return pieces
+
+
+def _stale_split(manifest: ChunkManifest) -> bool:
+    """PS-60: whether a cached manifest holds a table cut by the old fixed-length hard split, which made every piece
+    but the last exactly the cap long; `_hard_split` now keeps pieces under it. Only a piece holding table rows
+    counts: an old prose split keeps its cached chunks (re-chunking it would change nothing a reader relies on)."""
+    cap = manifest.token_cap * 4
+    return any(len(c.text) == cap and any(_TABLE_LINE.match(ln) for ln in c.text.split("\n"))
+               for c in manifest.chunks[:-1])
+
+
+def _continued_table_header(piece: str, rest: str) -> str:
+    """The header and separator rows of the pipe table a cut fell inside (`piece` ends in it and `rest` continues
+    it), or "" when the cut is not inside a table or the table's header cannot be told."""
+    if not _TABLE_LINE.match(rest.split("\n", 1)[0]):
+        return ""
+    lines = piece.split("\n")
+    i = len(lines)
+    while i > 0 and _TABLE_LINE.match(lines[i - 1]):
+        i -= 1
+    block = lines[i:]
+    if len(block) >= 2 and _TABLE_SEPARATOR.fullmatch(block[1]):
+        return f"{block[0]}\n{block[1]}"
+    return ""
 
 
 _SEP = "\n\n"
@@ -745,7 +809,10 @@ def _chunk_prepare(parsed: ParsedDocument, cache_dir: Path, discoverer: Optional
     cache_dir.mkdir(parents=True, exist_ok=True)
     manifest_path = cache_dir / f"{parsed.source_doc_id}.{parsed.content_hash[:16]}.chunks.json"
     if manifest_path.exists():  # content-hash gate
-        return ChunkManifest.model_validate_json(manifest_path.read_text(encoding="utf-8")), manifest_path, None, None
+        cached = ChunkManifest.model_validate_json(manifest_path.read_text(encoding="utf-8"))
+        if not _stale_split(cached):
+            return cached, manifest_path, None, None
+        _log.info("re-chunking %s: its cached chunks cut a table mid-row (PS-60)", parsed.source_doc_id)
     document = _as_content_view(load_document(parsed))  # issue 0014: chunk the reading-order body (incl. tables)
     disc = discoverer if discoverer is not None else SeamBoundaryDiscoverer(token_cap=token_cap)
     return None, manifest_path, document, disc

{rag_wright-0.16.0 → rag_wright-0.17.0}/src/rag_wright/contracts/ingestion.py

@@ -196,9 +196,11 @@ class Extractor(Protocol):
 
 
 class RecordWriter(Protocol):
-    """Optional: persist a document's extractions. The engine default writes them with `kg_write`."""
+    """Optional: persist a document's extractions. The engine default writes them with `kg_write`. Return the number of
+    records now in the graph (written, or found already written); `None` lets the engine count the record nodes it
+    handed over (RuleWright, 2026-10-31: only the writer knows what landed)."""
 
-    def __call__(self, source_doc_id: str, extractions: Sequence[UnitExtraction]) -> Awaitable[None]: ...
+    def __call__(self, source_doc_id: str, extractions: Sequence[UnitExtraction]) -> Awaitable[Optional[int]]: ...
 
 
 # --- the engine-enforced checks --------------------------------------------------------------------------------

{rag_wright-0.16.0 → rag_wright-0.17.0}/src/rag_wright/ingestion/builder.py

@@ -55,7 +55,11 @@ class DocumentReport:
     sheets left out (`IngestSource.include_hidden_sheets=False`). `children`: the ids of embedded files and PDF
     attachments ingested as child documents (`parent_doc_id` is set on theirs); `embedded_skipped`: embedded files
     that could not be ingested. `links`: `AttachedTo` record links written, by confidence; `unmapped_links`: links
-    whose record row could not be found. `dead_letter`: why the whole document failed (None when it landed)."""
+    whose record row could not be found. `unreached_table_rows` (PS-62): one entry per record table -- a table at
+    least one of whose rows reached a unit carrying its `TableRow` -- whose data rows did not ALL reach one,
+    `{table_ref, sheet, reached, total, missing}` (`missing`: the 1-based data-row indices, at most the first
+    1,000); the document still lands, so read this, or the `[ingest] warning:` line, to know rows were lost.
+    `dead_letter`: why the whole document failed (None when it landed)."""
 
     doc_id: str
     parent_doc_id: Optional[str] = None
@@ -70,6 +74,7 @@ class DocumentReport:
     children: list[str] = field(default_factory=list)
     links: dict[str, int] = field(default_factory=dict)  # AttachedTo edges written, by confidence
     unmapped_links: int = 0  # record links whose row span could not be found
+    unreached_table_rows: list[dict] = field(default_factory=list)  # PS-62: record tables that lost rows
     dead_letter: Optional[str] = None
 
 
@@ -119,6 +124,7 @@ class ExtractStage:
     extractions: list = field(default_factory=list)
     failures: list[dict] = field(default_factory=list)
     row_spans: dict = field(default_factory=dict)
+    unreached_rows: list[dict] = field(default_factory=list)  # PS-62: `_unreached_rows`
 
 
 class IngestionStages:
@@ -235,6 +241,7 @@ class IngestionStages:
         document = self.document(sd)
         stage.row_spans = _row_spans(document, [t.span for t in tagged])
         _attach_table_rows(document, stage.units, stage.row_spans)
+        stage.unreached_rows = _unreached_rows(document, stage.units)
         sem = asyncio.Semaphore(self.tuning.extract_concurrency)
 
         async def one(unit: Any) -> Optional[UnitExtraction]:
@@ -252,9 +259,13 @@ class IngestionStages:
         return stage
 
     async def write(self, sd: Any, extractions: list) -> int:
-        """Persist the records (the domain's writer, else `kg_write`). Returns the number of record nodes."""
+        """Persist the records (the domain's writer, else `kg_write`). Returns the number that landed: the writer's own
+        count when it returns one; else the record nodes handed over, which the default `kg_write` lands in one
+        transaction (it raises rather than land part of them)."""
         if self.writer is not None:
-            await self.writer(sd.source_doc_id, extractions)
+            landed = await self.writer(sd.source_doc_id, extractions)
+            if landed is not None:
+                return int(landed)
         else:
             nodes = [n for e in extractions for n in e.nodes]
             edges = [ed for e in extractions for ed in e.edges]
@@ -413,6 +424,12 @@ class IngestionPipeline:
                            + (f"DEAD-LETTER: {top.dead_letter}" if top.dead_letter else
                               f"spans={top.spans} units={top.units} records={top.records} "
                               f"children={len(top.children)} failures={len(top.extraction_failures)}"))
+            for d in docs:  # PS-62: a record table's lost rows are said, never only counted
+                for t in d.unreached_table_rows:
+                    where = f"sheet {t['sheet']!r}" if t["sheet"] else f"table {t['table_ref']}"
+                    self._progress(f"[ingest] warning: {d.doc_id}: {where}: {t['reached']} of {t['total']} rows "
+                                   f"reached a unit with their row; the rest produced no record "
+                                   f"(first missing: {t['missing'][:5]})")
             return {"documents": [_dump(list[DocumentReport], docs)]}
 
         def finish(state: _RunState) -> dict:
@@ -518,6 +535,7 @@ class IngestionPipeline:
             rep.chunks, rep.units = len(state["chunks"]), len(extracted.units)
             rep.spans, rep.span_failures = indexed["span_count"], indexed["span_failures"]
             rep.extraction_failures = extracted.failures
+            rep.unreached_table_rows = extracted.unreached_rows
             reports = [rep]
             for j, emb in enumerate(sd.embedded, 1):
                 if depth + 1 > _MAX_CHILD_DEPTH:
@@ -580,13 +598,15 @@ def _load(kind: Any, value: Any) -> Any:
 
 def _dump_extract_stage(stage: ExtractStage) -> dict:
     return {"units": _dump(list[Unit], stage.units), "extractions": _dump(list[UnitExtraction], stage.extractions),
-            "failures": stage.failures, "row_spans": [[ref, row, span] for (ref, row), span in stage.row_spans.items()]}
+            "failures": stage.failures, "row_spans": [[ref, row, span] for (ref, row), span in stage.row_spans.items()],
+            "unreached_rows": stage.unreached_rows}
 
 
 def _load_extract_stage(value: dict) -> ExtractStage:
     return ExtractStage(units=_load(list[Unit], value["units"]),
                         extractions=_load(list[UnitExtraction], value["extractions"]), failures=value["failures"],
-                        row_spans={(ref, row): span for ref, row, span in value["row_spans"]})
+                        row_spans={(ref, row): span for ref, row, span in value["row_spans"]},
+                        unreached_rows=value.get("unreached_rows", []))  # absent from a pre-PS-62 checkpoint
 
 
 class _RunState(TypedDict, total=False):
@@ -710,6 +730,32 @@ def _attach_table_rows(document: Any, units: list, row_spans: dict) -> None:
             unit.table_row = rows.get(keys.pop())
 
 
+_MAX_MISSING_LISTED = 1000
+
+
+def _unreached_rows(document: Any, units: list) -> list[dict]:
+    """PS-62: the record tables whose data rows did not all reach a unit carrying their `TableRow`. A table none of
+    whose rows reached one is a block table (one unit for the whole table) or not a table the grouper split by row,
+    so it is not a loss; a table some of whose rows did is read row by row, and every row it lacks was lost."""
+    from rag_wright.ingestion.tables import rows_of, visible_tables
+
+    reached: dict[str, set[int]] = {}
+    for unit in units:
+        if unit.table_row is not None:
+            reached.setdefault(unit.table_row.table_ref, set()).add(unit.table_row.row_index)
+    out: list[dict] = []
+    for table in visible_tables(document):
+        got = reached.get(table.self_ref)
+        if not got:
+            continue
+        rows = rows_of(table, document)
+        missing = [r.row_index for r in rows if r.row_index not in got]
+        if missing:
+            out.append({"table_ref": table.self_ref, "sheet": rows[0].sheet, "reached": len(rows) - len(missing),
+                        "total": len(rows), "missing": missing[:_MAX_MISSING_LISTED]})
+    return out
+
+
 def _check_model_ceilings(tuning: IngestionTuning, documents: int, progress: Callable[[str], Any]) -> None:
     """PS-21: warn when this ingest's extractor fan-out can exceed a resolved model's server ceiling. The peak is
     `min(document_concurrency, documents) x extract_concurrency`; a model's ceiling is its profile's