rag-wright 0.16.1__tar.gz → 0.17.0__tar.gz

{rag_wright-0.16.1 → rag_wright-0.17.0}/.claude/skills/building-an-ingestion-capability/SKILL.md

@@ -22,7 +22,7 @@ Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`)
 |---|---|---|---|
 | parse (PDF, Office, spreadsheets incl. hidden sheets, HTML, Markdown, images and scans; embedded files) | engine | docling, content-hash cached; images and scanned pages are OCR'd, and only degraded pages escalate to the vision model (the `VISION_OCR` role) | (none) |
 | chunk | engine default, domain may override | structural boundaries; a model refines only an over-cap section (domain-neutral prompt) | `chunk_model=`, `chunk_discoverer=` (e.g. `default_chunk_discoverer(guidance=...)` to say what a coherent unit is in your documents) |
-| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` |
+| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` (to adjust the default rather than replace it, wrap `pack_sdk.segment_layout`) |
 | tag spans (soft tags) | domain, optional | none | `span_tagger=` (to compare models behind it, see section 5 of `docs/domain-adaptation/classification-and-decision-models.md`) |
 | index spans (embed + store, page/bbox provenance) | engine | the workspace's ingest embedder | `embedder=` |
 | group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
@@ -181,6 +181,9 @@ category.
   `evaluate_ingestion(..., table_labels=[{"pattern": <regex on the header row>, "label": "record"|"block"}])`
   (first match wins): a block table split per row, or a wide record table not split, is a failure, and tables no
   pattern matched are listed in `unlabelled_tables`.
+- **Every row of a record table should reach a unit.** After an ingest, `DocumentReport.unreached_table_rows` lists
+  each record table whose rows did not all reach a unit with their `TableRow` (with the `[ingest] warning:` line);
+  an empty list on your samples is the check. Rows listed there produced no record.
 - **How an embedded file finds its row.** An identifier is a token of 4+ characters containing a digit that appears
   on at most `IdentifierRule.max_rows` table rows and in at most `max_files` embedded files (defaults 3 and 3; set
   them with `IngestionTuning(identifier=IdentifierRule(...))`). A file whose identifiers match no row, or too many,

rag_wright-0.17.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.17.0"
+}

{rag_wright-0.16.1 → rag_wright-0.17.0}/CHANGELOG.md

@@ -4,6 +4,19 @@ All notable changes to RAG_Wright are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.17.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.16.1...0.17.0) (2026-11-05)
+
+
+### Features
+
+* **PS-61:** the default segmenter is on pack_sdk, so a pack can wrap it ([bbb179d](https://github.com/fzaidi-dai/RAG_Wright/commit/bbb179d64e5fc905ada2ddc57367cdd2cc2ca5b6))
+* **PS-62:** a record table's rows that reach no unit are reported, never silent ([632fefb](https://github.com/fzaidi-dai/RAG_Wright/commit/632fefb390e11014a0dcb4cdd16956697a92ea58))
+
+
+### Bug Fixes
+
+* **PS-60:** a table over the chunk cap keeps every row's span and TableRow ([acd9ce5](https://github.com/fzaidi-dai/RAG_Wright/commit/acd9ce5917de1daac6ac47ea0a09ca975fcdf556))
+
 ## [0.16.1](https://github.com/fzaidi-dai/RAG_Wright/compare/0.16.0...0.16.1) (2026-11-01)
 
 

{rag_wright-0.16.1 → rag_wright-0.17.0}/CLAUDE.md

@@ -55,7 +55,11 @@ The failure mode to avoid, which has happened before: a command fails with "modu
 
 ## Start of every session
 
-> **CURRENT STATE (2026-10-29): `rag-wright` 0.16.0 is RELEASED** (PR #19; TexWright's requests: PS-54 a `Contains`
+> **CURRENT STATE (2026-11-01): `rag-wright` 0.16.1 is RELEASED** (PR #20; RuleWright's two requests and a finding
+> from them: PS-57 a retried contracts-pack write no longer loses a document's clauses, the ArcadeDB conflict budget is
+> configurable (default 20 attempts / 2 s cap) and a racing UPSERT's duplicate key is retried; PS-58 a `RecordWriter`
+> returns the count that landed; PS-59 the store's client keeps keep-alive connections, one session per thread). Next:
+> PS-48, EV-7, PS-30, PS-33. Before it, **`rag-wright` 0.16.0 is RELEASED** (PR #19; TexWright's requests: PS-54 a `Contains`
 > substring `where` value, PS-55 only the BGE models on Metal under one process lock (ADR-0140); PS-56 the BGE reranker
 > scores again on transformers 5). Before it, **`rag-wright` 0.15.0 is RELEASED** (PR #18; PS-51 the frontend skill draws the shell and
 > ships its scaffold, RuleWright's request; PS-52 one workspace embedder under concurrent first use and PS-53 KG calls refuse

{rag_wright-0.16.1 → rag_wright-0.17.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rag-wright
-Version: 0.16.1
+Version: 0.17.0
 Summary: A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime.
 Project-URL: Homepage, https://github.com/fzaidi-dai/RAG_Wright
 Project-URL: Repository, https://github.com/fzaidi-dai/RAG_Wright

rag_wright-0.17.0/docs/adr/0141-the-hard-split-cuts-between-lines-and-repeats-a-tables-header.md

@@ -0,0 +1,38 @@
+# ADR-0141: The cap's hard split cuts between lines and repeats a table's header
+
+**Status:** accepted · **Date:** 2026-11-05 · **Related:** PS-60, TexWright's request
+`2026-11-05-export-the-default-segmenter-on-pack-sdk.md`, ADR-0069 (tables chunked in reading order), ADR-0124
+(ING-2 the default segmenter, ING-7 `table_rows`), ADR-0136 (spreadsheet tables glued across blank rows)
+
+## Context
+
+A chunk is a run of whole parsed items, and the token cap is enforced last: a section over the cap that the
+discoverer cannot subdivide (a single table is one item) is hard-split. The split cut every `token_cap * 4`
+characters, wherever that fell. In a table it fell mid-row, and three things followed. The table item was found
+whole in no chunk, so `chunk_layouts` placed it nowhere, and the unplaced item also stopped every later item in the
+document from getting its parsed layout. Each piece then fell back to the text layout: the first piece still
+opened with the table's header, but a continuation piece began mid-row, read as one paragraph, and gave ONE span to
+all its rows. With no row spans, no unit carried a `TableRow`, so a record extractor never saw those rows, and the
+ingest reported success. TexWright lost 4 of 140 records this way; a 140-record, 76-column synthetic workbook lost
+205 of 280 rows.
+
+## Decision
+
+- The hard split cuts at the last blank line in the second half of the window, else the last line break, else the
+  last space, and only a run with none of them at a fixed length. The whitespace character it cuts at is dropped.
+  Pieces stay strictly under the cap.
+- A cut inside a pipe table repeats the table's header and separator rows at the top of the next piece, so every
+  piece is a well-formed table, each row is whole in one chunk, and the segmenter, the unit grouper's record-table
+  test and the row-to-span map work unchanged on every piece. A header half the cap or longer is not repeated.
+- `chunk_layouts` skips an item that no remaining chunk holds whole, instead of blocking the items after it.
+- A cached chunk manifest holding a piece exactly the cap long with table lines in it can only come from the old
+  cut; it is re-chunked once, with a log line. An old prose split keeps its cached chunks.
+
+## Consequences
+
+- Live, the workbook above yields 280 of 280 row records (75 before), and every span is one row.
+- A document with a section over the cap gets different chunk boundaries, so different `chunk_id`s and spans when
+  re-ingested; the `chunk_id` scheme is unchanged and documents under the cap are untouched. A repeated header
+  appears once per continuation chunk in the canonical document text.
+- Whether a pack may build on the default segmenter (PS-61) and whether a record table's lost rows are reported
+  (PS-62) are separate tasks.

{rag_wright-0.16.1 → rag_wright-0.17.0}/docs/adr/README.md

@@ -196,4 +196,5 @@ framing that still appears in older ADRs and the root `SPEC.md`/`plan.md`.
 0137 A generated answer states the evidence's support every time, and "none" is an abstention ·
 0138 PDF parses run one at a time, and the engine reaches pdfium through one locked door ·
 0139 The agent harness substrate is engine scope ·
-0140 One model family on Metal, under one process lock
+0140 One model family on Metal, under one process lock ·
+0141 The cap's hard split cuts between lines and repeats a table's header

{rag_wright-0.16.1 → rag_wright-0.17.0}/docs/api/README.md

@@ -216,9 +216,9 @@ A stage that raises routes to `dead_letter`, which ends the document with its `D
 
 The result of `IngestionPipeline.aingest`: one `DocumentReport` per document (embedded children included). `failed` counts the dead-lettered documents and `succeeded` the rest (properties).
 
-### `DocumentReport(doc_id: 'str', parent_doc_id: 'Optional[str]' = None, chunks: 'int' = 0, spans: 'int' = 0, units: 'int' = 0, records: 'int' = 0, extraction_failures: 'list[dict]' = <factory>, span_failures: 'list[dict]' = <factory>, skipped_hidden_sheets: 'list[str]' = <factory>, embedded_skipped: 'list[str]' = <factory>, children: 'list[str]' = <factory>, links: 'dict[str, int]' = <factory>, unmapped_links: 'int' = 0, dead_letter: 'Optional[str]' = None) -> None`
+### `DocumentReport(doc_id: 'str', parent_doc_id: 'Optional[str]' = None, chunks: 'int' = 0, spans: 'int' = 0, units: 'int' = 0, records: 'int' = 0, extraction_failures: 'list[dict]' = <factory>, span_failures: 'list[dict]' = <factory>, skipped_hidden_sheets: 'list[str]' = <factory>, embedded_skipped: 'list[str]' = <factory>, children: 'list[str]' = <factory>, links: 'dict[str, int]' = <factory>, unmapped_links: 'int' = 0, unreached_table_rows: 'list[dict]' = <factory>, dead_letter: 'Optional[str]' = None) -> None`
 
-What ingesting one document did. Counts: `chunks`, `spans` (indexed), `units` (extracted), `records` (nodes written). `extraction_failures` (`{unit, anchor, reason}`) / `span_failures` (`{span_id, reason}`): the units / spans whose extraction or indexing failed; the rest of the document still lands. `skipped_hidden_sheets`: hidden spreadsheet sheets left out (`IngestSource.include_hidden_sheets=False`). `children`: the ids of embedded files and PDF attachments ingested as child documents (`parent_doc_id` is set on theirs); `embedded_skipped`: embedded files that could not be ingested. `links`: `AttachedTo` record links written, by confidence; `unmapped_links`: links whose record row could not be found. `dead_letter`: why the whole document failed (None when it landed).
+What ingesting one document did. Counts: `chunks`, `spans` (indexed), `units` (extracted), `records` (nodes written). `extraction_failures` (`{unit, anchor, reason}`) / `span_failures` (`{span_id, reason}`): the units / spans whose extraction or indexing failed; the rest of the document still lands. `skipped_hidden_sheets`: hidden spreadsheet sheets left out (`IngestSource.include_hidden_sheets=False`). `children`: the ids of embedded files and PDF attachments ingested as child documents (`parent_doc_id` is set on theirs); `embedded_skipped`: embedded files that could not be ingested. `links`: `AttachedTo` record links written, by confidence; `unmapped_links`: links whose record row could not be found. `unreached_table_rows` (PS-62): one entry per record table -- a table at least one of whose rows reached a unit carrying its `TableRow` -- whose data rows did not ALL reach one, `{table_ref, sheet, reached, total, missing}` (`missing`: the 1-based data-row indices, at most the first 1,000); the document still lands, so read this, or the `[ingest] warning:` line, to know rows were lost. `dead_letter`: why the whole document failed (None when it landed).
 
 ### `IngestionEvaluation(documents: 'list[DocumentEvaluation]' = <factory>, failures: 'list[str]' = <factory>, unlabelled_tables: 'list[str]' = <factory>, labels: 'Optional[LabelEvaluation]' = None, warnings: 'list[str]' = <factory>) -> None`
 

{rag_wright-0.16.1 → rag_wright-0.17.0}/docs/api/pack_sdk.md

@@ -492,6 +492,10 @@ ASYNC twin of `chunk_texts` (SEG-2/SEG-5): awaits the discoverer's async boundar
 
 Validate a `documents` scope against `store.known_document_ids()` BEFORE any retrieval spends (mirrors issue 0007's `_validate_sources`). `None` (whole store) is not validated. `[]` (scope-to-nothing) is a valid empty scope, not an error -- the retrieval surfaces short-circuit it. A non-empty list with any id absent from the store raises `UnknownDocumentError`. A store without `known_document_ids` (a minimal fake) is treated as un-validatable and passes through.
 
+### `segment_layout(chunk_id: 'str', text: 'str', layout: 'Sequence[LayoutItem]', *, min_alnum: 'Optional[int]' = None, tuning: 'Optional[IngestionTuning]' = None) -> 'list[Span]'`
+
+Segment one chunk into spans (the engine's default `Segmenter`). The fragment floor is `min_alnum`, else `tuning.min_fragment_alnum`, else the default.
+
 ### `capability_impl(name: 'str') -> 'Callable[..., Any]'`
 
 Resolve a capability name to its invoke factory via the manifest `impl_ref`. Raises `KeyError` if the name is unknown to the ARD catalog, `NotImplementedError` if it declares no `impl_ref` (not invokable by name).

{rag_wright-0.16.1 → rag_wright-0.17.0}/docs/domain-adaptation/kg-construction.md

@@ -34,6 +34,20 @@ The reference pack's contract pipeline is one instantiation of the same stages (
 legal segmenter, a function tagger, a provision grouper, a decision-model boundary decider, its clause extractor and
 its typed writer. Read it as an example of hooks, not as the engine's definition of ingestion.
 
+A hook you pass replaces the default for the whole pipeline, for every document it ingests. When your documents
+need the default's behaviour in most places and your own rule in a few, build on the default rather than replace it:
+the default segmenter is `rag_wright.pack_sdk.segment_layout` (and the default chunk discoverer
+`StructuralModelFallbackDiscoverer`), so your segmenter can call it and adjust only the spans it gets wrong:
+
+```python
+from rag_wright.pack_sdk import segment_layout
+
+
+def segmenter(chunk_id, text, layout):
+    spans = segment_layout(chunk_id, text, layout)  # the engine's evaluated segmentation
+    return spans  # split or merge here only the spans your domain reads differently; keep them tiling the text
+```
+
 ## The stages
 
 ```
@@ -169,8 +183,10 @@ the run goes on. Every node and edge type your extractor returns must be declare
 
 `aingest` returns an `IngestionReport` (`succeeded`, `failed`, `documents`). Each `DocumentReport` has the counts
 (`chunks`, `spans`, `units`, `records`), `extraction_failures`, `span_failures`, `dead_letter` (the reason, if the
-document failed), `skipped_hidden_sheets`, `embedded_skipped`, `children`, and the `links` / `unmapped_links` of its
-embedded files. Nothing is skipped silently.
+document failed), `skipped_hidden_sheets`, `embedded_skipped`, `children`, the `links` / `unmapped_links` of its
+embedded files, and `unreached_table_rows`: each record table some of whose data rows reached no unit carrying their
+`TableRow`, so produced no record (`reached`, `total` and the `missing` row indices; also printed as an
+`[ingest] warning:` line). Nothing is skipped silently.
 
 ## The two graphs (one store)
 

{rag_wright-0.16.1 → rag_wright-0.17.0}/docs/specs/public-surface/plan.md

@@ -131,4 +131,7 @@ in two at the PS-6 gate so the store rewrite is its own visible task. One task,
 | PS-57 | RuleWright's request (`docs/engine-requests/2026-10-31-a-write-conflict-loses-a-documents-clauses-and-your-pack-carries-the-defect.md`, their repo): at 4 concurrent documents, 2 of 15 reported `ingested` with zero clauses. The contracts pack's `write` node popped the pending extractions (filled by `extract_clauses`) before the store write landed, so the node's retry wrote nothing and reported success; it now reads them and releases them after the write. The conflict budget is reachable: `StoreConfig.conflict_attempts` / `conflict_backoff_cap_s`, else `ARCADEDB_CONFLICT_ATTEMPTS` / `ARCADEDB_CONFLICT_BACKOFF_CAP_S`; default raised 10 attempts / 0.5 s cap to 20 / 2 s. Found by the live probe: two writers creating the same new node get a duplicate key, never retried; `kg_write` and `write_graph` (UPSERT-only) now retry it | pack retry test through the real per-document graph (0/3 clauses landed before, 3/3 after); six store tests; live probe (clause-shaped transactions over shared value nodes, pooled sessions): old budget lost 38/2,400 at 4 and 8 writers, new lost 0/2,400 at the same wall-clock; live store suite 42 passed | done 2026-11-01 |
 | PS-58 | RuleWright's request (`docs/engine-requests/2026-10-31-ingestionstages-write-returns-a-count-derived-from-its-own-input.md`, their repo): `IngestionStages.write` returned the record nodes it was handed, and `RecordWriter` (declared `-> None`) could not say what landed. A writer now returns the number of records in the graph (`Optional[int]`; `None` keeps the engine's count); the default `kg_write` branch keeps the handed-over count, true because that write is one transaction (lands whole or raises). The contracts pack's writer returns the clauses in the graph and its `write` node reports that as `clauses` | three hermetic tests + the PS-57 retry test's count (red before, green after); live `build_ingestion` on two fixtures: default and a writer landing every other record, `records` equals the graph in all four documents | done 2026-11-01 |
 | PS-59 | Found during PS-57: the ArcadeDB client opens a new connection for every request (`requests.post`, no session); at 8 concurrent writers against Docker on macOS about two thirds of writes failed with `RemoteDisconnected`, and with a keep-alive session per thread the same run lost none. Give the store's client a pooled session | live probe at 8 writers: zero connection drops | done 2026-11-01: the store's client (`_PooledClient`) keeps one keep-alive `requests.Session` per thread and `ArcadeDBStore.close()` releases them; live, the same probe lost 374 of 800 transactions to `RemoteDisconnected` unpooled and 0 of 800 at 8 writers and 0 of 1,600 at 16 pooled |
+| PS-60 | TexWright (2026-11-05, `2026-11-05-export-the-default-segmenter-on-pack-sdk.md`): a table over the chunk cap was hard-split at a fixed length, mid-row; the rows after the cut lost their table layout (one span between them, no `TableRow`), and one item `chunk_layouts` could not place blocked every later item's layout. 4 of 140 records lost silently | the cut falls between lines and repeats the table's header in each continuation chunk; an unplaceable item no longer blocks later ones; a manifest cached with the old cut is re-chunked once; live: a 140-record x 76-column workbook yields 140 records | done 2026-11-05 (ADR-0141): live, the workbook (280 rows, two per record) gave 280/280 row records, 75/280 with the old cut |
+| PS-61 | TexWright's question in the same request: the default segmenter `segment_layout` is on neither public tier, so a pack cannot wrap it | export it on `rag_wright.pack_sdk` | done 2026-11-05: `pack_sdk.segment_layout` (the engine's own object); the wrap-the-default pattern in `kg-construction.md` and the ingestion skill; a pack segmenter delegating to it runs through `build_ingestion` |
+| PS-62 | TexWright, same request: the loss was silent (every `check_*` passed) | a record table whose rows do not each reach a unit with its `TableRow` is reported in the `IngestionReport` | done 2026-11-05: `DocumentReport.unreached_table_rows` (`table_ref`, `sheet`, `reached`, `total`, `missing`) and an `[ingest] warning:` line; a block table is not a loss; live, the pre-PS-60 cut on the workbook reports 75 of 280 rows reached (205 missing, matching the records lost), the new cut reports nothing |
 | PS-48 | Port the agent harness's substrate behaviour tests from RuleWright onto the engine's fabric fixture (ADR-0139 handover): of RuleWright's 553 harness-touching tests, 314 test the substrate (port), 213 RuleWright's composition (stay), 25 live (1 substrate: OpenRouter cost accounting). Seven gated sub-tasks: **a** shared `tests/harness/fabric.py` + scripting/slow/hanging/failing fake models + `InMemorySaver` helpers, then events, skills, clarify (~23); **b** models and errors (~42); **c** runs, confirm, continuation (~32); **d** decisions and rules (~51); **e** grading (~32); **f** seen and memory (~41); **g** workers, batch, dispatch, tiers, agent wiring, filesystem, interpreter, withhold + the live accounting test (~93, may split). Ambiguous tests split, not moved whole; a ported test failing on engine code is a substrate defect fixed in that sub-task; deadline globals via monkeypatch; no new dependencies. Follow-up (owner RuleWright): retire its substrate duplicates once ported | each sub-task's ported tests green on the fabric fixture | todo (proposed; awaiting go-ahead) |

{rag_wright-0.16.1 → rag_wright-0.17.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "rag-wright"
-version = "0.16.1"
+version = "0.17.0"
 description = "A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime."
 readme = "README.md"
 requires-python = ">=3.12,<3.13"

{rag_wright-0.16.1 → rag_wright-0.17.0}/src/rag_wright/.agents/skills/building-an-ingestion-capability/SKILL.md

@@ -22,7 +22,7 @@ Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`)
 |---|---|---|---|
 | parse (PDF, Office, spreadsheets incl. hidden sheets, HTML, Markdown, images and scans; embedded files) | engine | docling, content-hash cached; images and scanned pages are OCR'd, and only degraded pages escalate to the vision model (the `VISION_OCR` role) | (none) |
 | chunk | engine default, domain may override | structural boundaries; a model refines only an over-cap section (domain-neutral prompt) | `chunk_model=`, `chunk_discoverer=` (e.g. `default_chunk_discoverer(guidance=...)` to say what a coherent unit is in your documents) |
-| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` |
+| segment a chunk into spans | engine default, domain may override | layout-driven: a table row per span, sentences for prose, a heading joins what follows | `segmenter=` (to adjust the default rather than replace it, wrap `pack_sdk.segment_layout`) |
 | tag spans (soft tags) | domain, optional | none | `span_tagger=` (to compare models behind it, see section 5 of `docs/domain-adaptation/classification-and-decision-models.md`) |
 | index spans (embed + store, page/bbox provenance) | engine | the workspace's ingest embedder | `embedder=` |
 | group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
@@ -181,6 +181,9 @@ category.
   `evaluate_ingestion(..., table_labels=[{"pattern": <regex on the header row>, "label": "record"|"block"}])`
   (first match wins): a block table split per row, or a wide record table not split, is a failure, and tables no
   pattern matched are listed in `unlabelled_tables`.
+- **Every row of a record table should reach a unit.** After an ingest, `DocumentReport.unreached_table_rows` lists
+  each record table whose rows did not all reach a unit with their `TableRow` (with the `[ingest] warning:` line);
+  an empty list on your samples is the check. Rows listed there produced no record.
 - **How an embedded file finds its row.** An identifier is a token of 4+ characters containing a digit that appears
   on at most `IdentifierRule.max_rows` table rows and in at most `max_files` embedded files (defaults 3 and 3; set
   them with `IngestionTuning(identifier=IdentifierRule(...))`). A file whose identifiers match no row, or too many,

{rag_wright-0.16.1 → rag_wright-0.17.0}/src/rag_wright/capabilities/rlm_chunking.py

@@ -25,6 +25,8 @@ from __future__ import annotations
 
 import asyncio
 import json
+import logging
+import re
 from pathlib import Path
 from typing import Any, Optional, Protocol, runtime_checkable
 
@@ -43,6 +45,8 @@ from rag_wright.models.profiles import ModelRole, model_for
 from rag_wright.models.seam import build_structured
 from rag_wright.skills.rlm.agent import build_rlm_agent, rlm_interpreter_session
 
+_log = logging.getLogger(__name__)
+
 DEFAULT_TOKEN_CAP = 20_000
 DEFAULT_SUMMARY_CONCURRENCY = 8  # in-flight summary calls (backpressure); summaries are network-bound
 _SUMMARIZE_PROMPT = "Summarize this document chunk in one or two sentences, factually:"
@@ -518,10 +522,70 @@ def _estimate_tokens(text: str) -> int:
     return max(1, len(text) // 4)
 
 
+_TABLE_LINE = re.compile(r"[ \t]*\|")
+_TABLE_SEPARATOR = re.compile(r"[ \t]*\|?[ \t:|-]*-[ \t:|-]*\|?[ \t]*")
+
+
 def _hard_split(text: str, token_cap: int) -> list[str]:
-    """Split text that alone exceeds the cap into <= cap-sized pieces (by chars)."""
-    cap_chars = token_cap * 4
-    return [text[i : i + cap_chars] for i in range(0, len(text), cap_chars)]
+    """Split text that alone exceeds the cap into pieces under the cap (by chars).
+
+    PS-60 (TexWright): each cut falls at the last blank line in the window's second half, else the last line break,
+    else the last space, and only a run with none of them is cut at a fixed length; the one whitespace character cut
+    at is dropped. A cut inside a pipe table repeats the table's header and separator rows at the top of the next
+    piece, so every piece is a well-formed table and every row stays whole: the old fixed-length cut fell mid-row,
+    and the rows after it lost their table layout, their own spans and their `TableRow`. Pieces stay strictly under
+    the cap, so a cached manifest holding a cap-length piece can only come from the old cut (`_stale_split`)."""
+    cap = token_cap * 4
+    pieces: list[str] = []
+    rest = text
+    floor = 0  # the carried header's length: a cut must fall after it, or the split would not advance
+    while len(rest) >= cap:
+        window = rest[: cap - 1]
+        blank = window.rfind("\n\n", max(floor, cap // 2))
+        line = window.rfind("\n", floor + 1)
+        space = max(window.rfind(" ", floor + 1), window.rfind("\t", floor + 1))
+        if blank > floor:
+            end, start = blank, blank + 2
+        elif line > floor:
+            end, start = line, line + 1
+        elif space > floor:
+            end, start = space, space + 1
+        else:
+            end = start = cap - 1
+        piece, rest = rest[:end], rest[start:]
+        pieces.append(piece)
+        header = _continued_table_header(piece, rest)
+        if header and len(header) < cap // 2:
+            rest, floor = f"{header}\n{rest}", len(header)
+        else:
+            floor = 0
+    if rest:
+        pieces.append(rest)
+    return pieces
+
+
+def _stale_split(manifest: ChunkManifest) -> bool:
+    """PS-60: whether a cached manifest holds a table cut by the old fixed-length hard split, which made every piece
+    but the last exactly the cap long; `_hard_split` now keeps pieces under it. Only a piece holding table rows
+    counts: an old prose split keeps its cached chunks (re-chunking it would change nothing a reader relies on)."""
+    cap = manifest.token_cap * 4
+    return any(len(c.text) == cap and any(_TABLE_LINE.match(ln) for ln in c.text.split("\n"))
+               for c in manifest.chunks[:-1])
+
+
+def _continued_table_header(piece: str, rest: str) -> str:
+    """The header and separator rows of the pipe table a cut fell inside (`piece` ends in it and `rest` continues
+    it), or "" when the cut is not inside a table or the table's header cannot be told."""
+    if not _TABLE_LINE.match(rest.split("\n", 1)[0]):
+        return ""
+    lines = piece.split("\n")
+    i = len(lines)
+    while i > 0 and _TABLE_LINE.match(lines[i - 1]):
+        i -= 1
+    block = lines[i:]
+    if len(block) >= 2 and _TABLE_SEPARATOR.fullmatch(block[1]):
+        return f"{block[0]}\n{block[1]}"
+    return ""
 
 
 _SEP = "\n\n"
@@ -745,7 +809,10 @@ def _chunk_prepare(parsed: ParsedDocument, cache_dir: Path, discoverer: Optional
     cache_dir.mkdir(parents=True, exist_ok=True)
     manifest_path = cache_dir / f"{parsed.source_doc_id}.{parsed.content_hash[:16]}.chunks.json"
     if manifest_path.exists():  # content-hash gate
-        return ChunkManifest.model_validate_json(manifest_path.read_text(encoding="utf-8")), manifest_path, None, None
+        cached = ChunkManifest.model_validate_json(manifest_path.read_text(encoding="utf-8"))
+        if not _stale_split(cached):
+            return cached, manifest_path, None, None
+        _log.info("re-chunking %s: its cached chunks cut a table mid-row (PS-60)", parsed.source_doc_id)
     document = _as_content_view(load_document(parsed))  # issue 0014: chunk the reading-order body (incl. tables)
     disc = discoverer if discoverer is not None else SeamBoundaryDiscoverer(token_cap=token_cap)
     return None, manifest_path, document, disc

{rag_wright-0.16.1 → rag_wright-0.17.0}/src/rag_wright/ingestion/builder.py

@@ -55,7 +55,11 @@ class DocumentReport:
     sheets left out (`IngestSource.include_hidden_sheets=False`). `children`: the ids of embedded files and PDF
     attachments ingested as child documents (`parent_doc_id` is set on theirs); `embedded_skipped`: embedded files
     that could not be ingested. `links`: `AttachedTo` record links written, by confidence; `unmapped_links`: links
-    whose record row could not be found. `dead_letter`: why the whole document failed (None when it landed)."""
+    whose record row could not be found. `unreached_table_rows` (PS-62): one entry per record table -- a table at
+    least one of whose rows reached a unit carrying its `TableRow` -- whose data rows did not ALL reach one,
+    `{table_ref, sheet, reached, total, missing}` (`missing`: the 1-based data-row indices, at most the first
+    1,000); the document still lands, so read this, or the `[ingest] warning:` line, to know rows were lost.
+    `dead_letter`: why the whole document failed (None when it landed)."""
 
     doc_id: str
     parent_doc_id: Optional[str] = None
@@ -70,6 +74,7 @@ class DocumentReport:
     children: list[str] = field(default_factory=list)
     links: dict[str, int] = field(default_factory=dict)  # AttachedTo edges written, by confidence
     unmapped_links: int = 0  # record links whose row span could not be found
+    unreached_table_rows: list[dict] = field(default_factory=list)  # PS-62: record tables that lost rows
     dead_letter: Optional[str] = None
 
 
@@ -119,6 +124,7 @@ class ExtractStage:
     extractions: list = field(default_factory=list)
     failures: list[dict] = field(default_factory=list)
     row_spans: dict = field(default_factory=dict)
+    unreached_rows: list[dict] = field(default_factory=list)  # PS-62: `_unreached_rows`
 
 
 class IngestionStages:
@@ -235,6 +241,7 @@ class IngestionStages:
         document = self.document(sd)
         stage.row_spans = _row_spans(document, [t.span for t in tagged])
         _attach_table_rows(document, stage.units, stage.row_spans)
+        stage.unreached_rows = _unreached_rows(document, stage.units)
         sem = asyncio.Semaphore(self.tuning.extract_concurrency)
 
         async def one(unit: Any) -> Optional[UnitExtraction]:
@@ -417,6 +424,12 @@ class IngestionPipeline:
                            + (f"DEAD-LETTER: {top.dead_letter}" if top.dead_letter else
                               f"spans={top.spans} units={top.units} records={top.records} "
                               f"children={len(top.children)} failures={len(top.extraction_failures)}"))
+            for d in docs:  # PS-62: a record table's lost rows are said, never only counted
+                for t in d.unreached_table_rows:
+                    where = f"sheet {t['sheet']!r}" if t["sheet"] else f"table {t['table_ref']}"
+                    self._progress(f"[ingest] warning: {d.doc_id}: {where}: {t['reached']} of {t['total']} rows "
+                                   f"reached a unit with their row; the rest produced no record "
+                                   f"(first missing: {t['missing'][:5]})")
             return {"documents": [_dump(list[DocumentReport], docs)]}
 
         def finish(state: _RunState) -> dict:
@@ -522,6 +535,7 @@ class IngestionPipeline:
             rep.chunks, rep.units = len(state["chunks"]), len(extracted.units)
             rep.spans, rep.span_failures = indexed["span_count"], indexed["span_failures"]
             rep.extraction_failures = extracted.failures
+            rep.unreached_table_rows = extracted.unreached_rows
             reports = [rep]
             for j, emb in enumerate(sd.embedded, 1):
                 if depth + 1 > _MAX_CHILD_DEPTH:
@@ -584,13 +598,15 @@ def _load(kind: Any, value: Any) -> Any:
 
 def _dump_extract_stage(stage: ExtractStage) -> dict:
     return {"units": _dump(list[Unit], stage.units), "extractions": _dump(list[UnitExtraction], stage.extractions),
-            "failures": stage.failures, "row_spans": [[ref, row, span] for (ref, row), span in stage.row_spans.items()]}
+            "failures": stage.failures, "row_spans": [[ref, row, span] for (ref, row), span in stage.row_spans.items()],
+            "unreached_rows": stage.unreached_rows}
 
 
 def _load_extract_stage(value: dict) -> ExtractStage:
     return ExtractStage(units=_load(list[Unit], value["units"]),
                         extractions=_load(list[UnitExtraction], value["extractions"]), failures=value["failures"],
-                        row_spans={(ref, row): span for ref, row, span in value["row_spans"]})
+                        row_spans={(ref, row): span for ref, row, span in value["row_spans"]},
+                        unreached_rows=value.get("unreached_rows", []))  # absent from a pre-PS-62 checkpoint
 
 
 class _RunState(TypedDict, total=False):
@@ -714,6 +730,32 @@ def _attach_table_rows(document: Any, units: list, row_spans: dict) -> None:
             unit.table_row = rows.get(keys.pop())
 
 
+_MAX_MISSING_LISTED = 1000
+
+
+def _unreached_rows(document: Any, units: list) -> list[dict]:
+    """PS-62: the record tables whose data rows did not all reach a unit carrying their `TableRow`. A table none of
+    whose rows reached one is a block table (one unit for the whole table) or not a table the grouper split by row,
+    so it is not a loss; a table some of whose rows did is read row by row, and every row it lacks was lost."""
+    from rag_wright.ingestion.tables import rows_of, visible_tables
+
+    reached: dict[str, set[int]] = {}
+    for unit in units:
+        if unit.table_row is not None:
+            reached.setdefault(unit.table_row.table_ref, set()).add(unit.table_row.row_index)
+    out: list[dict] = []
+    for table in visible_tables(document):
+        got = reached.get(table.self_ref)
+        if not got:
+            continue
+        rows = rows_of(table, document)
+        missing = [r.row_index for r in rows if r.row_index not in got]
+        if missing:
+            out.append({"table_ref": table.self_ref, "sheet": rows[0].sheet, "reached": len(rows) - len(missing),
+                        "total": len(rows), "missing": missing[:_MAX_MISSING_LISTED]})
+    return out
+
+
 def _check_model_ceilings(tuning: IngestionTuning, documents: int, progress: Callable[[str], Any]) -> None:
     """PS-21: warn when this ingest's extractor fan-out can exceed a resolved model's server ceiling. The peak is
     `min(document_concurrency, documents) x extract_concurrency`; a model's ceiling is its profile's

{rag_wright-0.16.1 → rag_wright-0.17.0}/src/rag_wright/ingestion/layout.py

@@ -40,14 +40,17 @@ def chunk_layouts(document: Any, chunk_texts: Sequence[str]) -> list[list[Layout
     items = [(it, t) for it, t in items if t]
     out: list[list[LayoutItem]] = []
     k = 0  # next unplaced item
-    for text in chunk_texts:
+    for c, text in enumerate(chunk_texts):
         layout: list[LayoutItem] = []
         cursor = 0
         while k < len(items):
             item, t = items[k]
             pos = text.find(t, cursor)
             if pos < 0:
-                break  # belongs to a later chunk (or was hard-split: left unlocated)
+                if any(t in later for later in chunk_texts[c + 1:]):
+                    break  # belongs to a later chunk
+                k += 1  # PS-60: no chunk holds it whole (hard-split): skip it, never block the items after it
+                continue  # (its text is laid out from the chunk text by the segmenter)
             layout.append(LayoutItem(kind=layout_kind(item.label), text=t, start=pos, end=pos + len(t),
                                      level=item.level, pages=[item.page] if item.page else []))
             cursor = pos + len(t)

{rag_wright-0.16.1 → rag_wright-0.17.0}/src/rag_wright/pack_sdk/__init__.py

@@ -8,7 +8,8 @@ migration note):
 - `rag_wright.pack_sdk` (this module) -- what a PACK's own code additionally needs to implement capabilities: the
   identifiers and provenance types, the model seam (model resolution, structured output, streaming), the LangGraph
   scaffold, the store protocol, the generic capabilities a pack composes (explicit-model generation and judging,
-  entity resolution, reranking, embedding, parsing, chunking, the ingestion stages), and capability plumbing.
+  entity resolution, reranking, embedding, parsing, chunking, the default segmenter, the ingestion stages), and
+  capability plumbing.
 
 A pack imports from `rag_wright.api` and `rag_wright.pack_sdk` (and its own modules) only; anything else in the
 engine is internal and may change. Everything here is a re-export of the engine's own object (no copies). Where a
@@ -64,6 +65,7 @@ from rag_wright.corpus.document_parser import (
 from rag_wright.corpus.document_parser import aparse_document_bytes as aparse_docling_bytes
 from rag_wright.corpus.document_parser import parse_document_bytes as parse_docling_bytes
 from rag_wright.ingestion.builder import IngestionStages
+from rag_wright.ingestion.segment import segment_layout
 from rag_wright.models.profiles import decision_profile, model_for, profile_for
 from rag_wright.models.seam import (
     ModelCallTimeout,
@@ -114,6 +116,7 @@ __all__ = [
     "INGEST_PARSE_DEADLINE_S", "TieredOCRParser", "parse_docling_bytes", "aparse_docling_bytes",
     "document_to_sections", "LEADING_ENUM", "is_bare_heading", "section_number", "StructuralModelFallbackDiscoverer",
     "achunk_texts", "IngestionStages", "UnknownDocumentError", "validate_documents",
+    "segment_layout",  # PS-61: the default `Segmenter`, so a pack's segmenter can delegate to it and adjust its spans
     # capability plumbing
     "capability_impl", "CapabilityRegistry", "ResponseBounds", "RLM_GRANTED_SUBAGENTS",
     "CapabilityInterface", "SkillRuntime",  # PS-38 (G23): the manifest's typed-interface and skill-runtime fields

{rag_wright-0.16.1 → rag_wright-0.17.0}/tasks.md

@@ -3,7 +3,9 @@
 Phase 2 output. The persistent, cross-session task ledger and shared memory of progress. Derived
 from `plan.md` (Phase 1) and `SPEC.md` v0.1, honoring ADR-0001 (stack) and ADR-0002 (corpus).
 
-> **2026-11-01: PS-57 done, unreleased** (RuleWright's request: a retried contracts-pack write no longer loses a document's clauses; the ArcadeDB conflict budget is configurable, default 20 attempts / 2 s cap; a racing UPSERT's duplicate key is retried). PS-58 done, unreleased (RuleWright's second request: a `RecordWriter` returns the count that landed). PS-59 done, unreleased (the store's client keeps keep-alive connections, one session per thread; `requests` declared). Next: PS-48.
+> **2026-11-05: PS-60 done, unreleased** (TexWright's request: a table over the chunk cap is split between lines with its header repeated, so every row keeps its own span and `TableRow`; ADR-0141). PS-61 done, unreleased (TexWright's question: the default segmenter `segment_layout` is exported on `pack_sdk`, so a pack can wrap it). PS-62 done, unreleased (a record table's rows that reach no unit are reported: `DocumentReport.unreached_table_rows` and an `[ingest] warning:` line). Next: reply to TexWright, then PS-48.
+>
+> **2026-11-01: `rag-wright` 0.16.1 is RELEASED (PR #20: PS-57, PS-58, PS-59).** PS-57 done (RuleWright's request: a retried contracts-pack write no longer loses a document's clauses; the ArcadeDB conflict budget is configurable, default 20 attempts / 2 s cap; a racing UPSERT's duplicate key is retried). PS-58 done (RuleWright's second request: a `RecordWriter` returns the count that landed). PS-59 done (the store's client keeps keep-alive connections, one session per thread; `requests` declared). Next: PS-48.
 >
 > **⇒ RESUME / NEXT UP (2026-10-17, NEWEST): the LangGraph ingestion workstream is RELEASED as `rag-wright` 0.14.0** (PR #16)
 > (`docs/specs/langgraph-ingestion/plan.md`, LGI-1..5; ADR-0124 amended; RuleWright guide

rag_wright-0.17.0/tests/api/test_pack_sdk_segmenter.py

@@ -0,0 +1,42 @@
+"""PS-61 (TexWright, 2026-11-05): the default segmenter is on the pack tier, so a pack can wrap it rather than replace
+it. `build_ingestion(segmenter=)` replaces segmentation for the whole pipeline; a pipeline that ingests both a
+domain's tables and ordinary prose keeps the engine's evaluated segmentation for the prose by delegating to it."""
+from __future__ import annotations
+
+import asyncio
+
+from rag_wright import pack_sdk
+from rag_wright.api import KgNode, UnitExtraction, build_ingestion, check_tiling
+from rag_wright.ingestion.segment import segment_layout
+
+from tests.ingestion.test_builder import FIXTURES, _FakeEmbedder, _ws
+
+
+def test_the_default_segmenter_is_on_the_pack_tier():
+    assert "segment_layout" in pack_sdk.__all__
+    assert pack_sdk.segment_layout is segment_layout  # a re-export of the engine's own object, not a copy
+
+
+def test_a_pack_segmenter_can_delegate_to_the_default_and_adjust_its_spans(tmp_path):
+    seen = []
+
+    def segmenter(chunk_id, text, layout):
+        spans = pack_sdk.segment_layout(chunk_id, text, layout, min_alnum=2)  # the engine's evaluated default
+        seen.append(len(spans))
+        return spans  # (a pack would adjust them here, e.g. split a span its domain reads as two)
+
+    async def extract(unit, *, source_doc_id):
+        return UnitExtraction(nodes=[KgNode("Record", "record_id", {"record_id": f"{source_doc_id}:{unit.index}"})])
+
+    handle = _ws()
+    pipe = build_ingestion(extract, segmenter=segmenter, embedder=_FakeEmbedder(), progress=lambda _l: None)
+    report = asyncio.run(pipe.aingest(handle, [str(FIXTURES / "textile_spec_sheet.md")], cache_dir=tmp_path / "c"))
+    (doc,) = report.documents
+    assert doc.dead_letter is None and seen and doc.spans == sum(seen)
+
+
+def test_the_export_tiles_a_chunk_like_the_builder_default():
+    text = "| Sr# | GSM |\n|---|---|\n| 1 | 380 |\n| 2 | 412 |\n\nA closing note. It has two sentences."
+    spans = pack_sdk.segment_layout("doc:0:h", text, [])
+    check_tiling("doc:0:h", text, spans)
+    assert [s.kind for s in spans] == ["table", "table_row", "table_row", "paragraph", "paragraph"]