rag-wright 0.16.0__tar.gz → 0.16.1__tar.gz

{rag_wright-0.16.0 → rag_wright-0.16.1}/.claude/skills/building-an-ingestion-capability/SKILL.md

@@ -28,7 +28,7 @@ Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`)
 | group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
 | choose each unit's representative span (its `anchor` + leading tag) | **domain decision**, optional | the grouper's choice: the first member, often a heading | `unit_representative=` |
 | extract records from a unit | **domain (required)** | (none) | the `extractor` argument |
-| write records | engine default | `kg_write` | `writer=` |
+| write records | engine default | `kg_write` | `writer=` (returns the number of records now in the graph, or `None`) |
 | per-document follow-up (e.g. an entity graph) | domain, optional | none | `document_hook=` |
 | `Document` node, embedded children (`EmbeddedIn` / `AttachedTo`), progress, dead-lettering | engine | always on | (none) |
 

rag_wright-0.16.1/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.16.1"
+}

{rag_wright-0.16.0 → rag_wright-0.16.1}/CHANGELOG.md

@@ -4,6 +4,15 @@ All notable changes to RAG_Wright are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.16.1](https://github.com/fzaidi-dai/RAG_Wright/compare/0.16.0...0.16.1) (2026-11-01)
+
+
+### Bug Fixes
+
+* **PS-57:** a retried write no longer loses a document's clauses; the conflict budget is configurable ([5d209d7](https://github.com/fzaidi-dai/RAG_Wright/commit/5d209d7eb1ab4bad93cd7026e81e79d9b6852fe5))
+* **PS-58:** a record writer reports the count that landed ([a7c384c](https://github.com/fzaidi-dai/RAG_Wright/commit/a7c384cfab59832f9714bac9996f91a29340ed71))
+* **PS-59:** the store's client keeps keep-alive connections, one session per thread ([f644ba3](https://github.com/fzaidi-dai/RAG_Wright/commit/f644ba3f3f37f12102037405b9a58567098358b3))
+
 ## [0.16.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.15.0...0.16.0) (2026-10-28)
 
 

{rag_wright-0.16.0 → rag_wright-0.16.1}/CLAUDE.md

@@ -55,7 +55,9 @@ The failure mode to avoid, which has happened before: a command fails with "modu
 
 ## Start of every session
 
-> **CURRENT STATE (2026-10-28): `rag-wright` 0.15.0 is RELEASED** (PR #18; PS-51 the frontend skill draws the shell and
+> **CURRENT STATE (2026-10-29): `rag-wright` 0.16.0 is RELEASED** (PR #19; TexWright's requests: PS-54 a `Contains`
+> substring `where` value, PS-55 only the BGE models on Metal under one process lock (ADR-0140); PS-56 the BGE reranker
+> scores again on transformers 5). Before it, **`rag-wright` 0.15.0 is RELEASED** (PR #18; PS-51 the frontend skill draws the shell and
 > ships its scaffold, RuleWright's request; PS-52 one workspace embedder under concurrent first use and PS-53 KG calls refuse
 > a name that is not an identifier, TexWright's requests). Before it, **`rag-wright` 0.14.1 is RELEASED** (PR #17; TexWright's two requests: PS-49 a retried
 > ArcadeDB write conflict no longer prints as an error, PS-50 `ensure_pack_schema` declares a property a pack gains on an

{rag_wright-0.16.0 → rag_wright-0.16.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rag-wright
-Version: 0.16.0
+Version: 0.16.1
 Summary: A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime.
 Project-URL: Homepage, https://github.com/fzaidi-dai/RAG_Wright
 Project-URL: Repository, https://github.com/fzaidi-dai/RAG_Wright
@@ -42,6 +42,7 @@ Requires-Dist: pydantic>=2.0
 Requires-Dist: pypdfium2>=5.11
 Requires-Dist: pyshacl>=0.40.1
 Requires-Dist: python-dotenv>=1.2
+Requires-Dist: requests>=2.34.2
 Requires-Dist: scikit-learn>=1.9.1
 Requires-Dist: sentence-transformers>=6.1.0
 Requires-Dist: transformers<5.9,>=5.4

{rag_wright-0.16.0 → rag_wright-0.16.1}/docs/api/README.md

@@ -8,7 +8,7 @@
 
 The product's view of the engine: the store connection, chosen models (`models`: `ModelRole` value -> model alias), the embedding profile, the `options` catalog (generic `ingest` knobs + each pack's options under `packs`), and `pack`: the path to the domain pack `.ttl` whose KG types `open_workspace` creates (None = only the neutral engine types: Chunk, Entity, Relationship, Mentions, Span, Document, EmbeddedIn, AttachedTo). Defaults just work; override only to trade quality/cost/latency. Implementation details (ArcadeDB, BGE) never cross this boundary.
 
-### `StoreConfig(host: 'str', port: 'str', user: 'str', password: 'str', backend: 'str' = 'arcadedb', protocol: 'str' = 'http') -> None`
+### `StoreConfig(host: 'str', port: 'str', user: 'str', password: 'str', backend: 'str' = 'arcadedb', protocol: 'str' = 'http', conflict_attempts: 'Optional[int]' = None, conflict_backoff_cap_s: 'Optional[float]' = None) -> None`
 
 How to reach the KG/retrieval store. `backend` selects the implementation (only `arcadedb` today; a new backend -- e.g. Neo4j -- is added here, invisibly to the product).
 
@@ -304,9 +304,9 @@ PS-R3: which member span represents a unit -- a DOMAIN decision. The engine appl
 
 One unit -> the domain's typed records (required; the domain-specific step).
 
-### `RecordWriter: (source_doc_id: 'str', extractions: 'Sequence[UnitExtraction]') -> 'Awaitable[None]'`
+### `RecordWriter: (source_doc_id: 'str', extractions: 'Sequence[UnitExtraction]') -> 'Awaitable[Optional[int]]'`
 
-Optional: persist a document's extractions. The engine default writes them with `kg_write`.
+Optional: persist a document's extractions. The engine default writes them with `kg_write`. Return the number of records now in the graph (written, or found already written); `None` lets the engine count the record nodes it handed over (RuleWright, 2026-10-31: only the writer knows what landed).
 
 ### `BoundaryDiscoverer: (**kwargs)`
 

{rag_wright-0.16.0 → rag_wright-0.16.1}/docs/api/pack_sdk.md

@@ -182,7 +182,7 @@ Group into units (checked), attach table rows, extract each unit concurrently (p
 
 #### `IngestionStages.write(self, sd: 'Any', extractions: 'list') -> 'int'`
 
-Persist the records (the domain's writer, else `kg_write`). Returns the number of record nodes.
+Persist the records (the domain's writer, else `kg_write`). Returns the number that landed: the writer's own count when it returns one; else the record nodes handed over, which the default `kg_write` lands in one transaction (it raises rather than land part of them).
 
 #### `IngestionStages.write_document_node(self, doc_id: 'str', *, parent_id: 'Optional[str]', filename: 'str', media_type: 'str', sha256: 'str', metadata: 'Optional[dict]' = None) -> 'None'`
 

{rag_wright-0.16.0 → rag_wright-0.16.1}/docs/configuration.md

@@ -45,6 +45,14 @@ configuration change takes effect without a restart ([`workspaces.md`](workspace
 | `password` | `str` | — | password |
 | `backend` | `str` | `"arcadedb"` | store implementation (only `arcadedb` today) |
 | `protocol` | `str` | `"http"` | `http` locally; `https` for a remote/Modal store |
+| `conflict_attempts` | `int \| None` | `None` | attempts per write when ArcadeDB reports a write conflict (`None`: `ARCADEDB_CONFLICT_ATTEMPTS`, else 20) |
+| `conflict_backoff_cap_s` | `float \| None` | `None` | the longest wait between those attempts, in seconds (`None`: `ARCADEDB_CONFLICT_BACKOFF_CAP_S`, else 2) |
+
+Concurrent writers that touch the same page get a write conflict; the store retries the write (the failed
+transaction is rolled back whole) with a doubling wait from 10 ms up to the cap. Two writers creating the same new
+node by `kg_write` get a duplicate key instead, which is retried the same way. A write that is still conflicting
+after the last attempt raises, about 25 s after the first one with the defaults. Raise the budget when more than
+eight documents write into one graph at once.
 
 `StoreConfig.from_env()` builds one from `ARCADEDB_HOST` (default `localhost`), `ARCADEDB_PORT` (required: no
 default, so an unset variable cannot reach another instance on the machine),
@@ -284,6 +292,7 @@ Prefer config; use env for secrets and for non-API callers. The common ones the
 | variable | purpose |
 |---|---|
 | `ARCADEDB_HOST` / `ARCADEDB_PORT` / `ARCADEDB_USER` / `ARCADEDB_PASSWORD` / `ARCADEDB_DATABASE` / `ARCADEDB_PROTOCOL` | store connection: `StoreConfig.from_env()` reads them (and the scripts do); `open_workspace` itself reads only the `StoreConfig` it is given |
+| `ARCADEDB_CONFLICT_ATTEMPTS` / `ARCADEDB_CONFLICT_BACKOFF_CAP_S` | the write-conflict retry budget when `StoreConfig.conflict_attempts` / `conflict_backoff_cap_s` are unset |
 | `OPENROUTER_API_KEY` / `OPENROUTER_BASE_URL` / `OPENROUTER_PROVIDER` / `OPENROUTER_ALLOW_FALLBACKS` | OpenRouter access + provider routing (`OPENROUTER_SORT` / `OPENROUTER_PROVIDER_ORDER`: the contracts pack's docling-graph extraction only) |
 | `RAG_SERVING` / `VLLM_BASE_URL` / `VLLM_API_KEY` / `STACK_URL` | serving backend (openrouter \| vllm) + self-hosted endpoint |
 | `RAG_MODEL_<ROLE>` (e.g. `RAG_MODEL_GENERAL`, `RAG_MODEL_VISION_OCR`) / `RAG_MODEL_ALL` / `RAG_GRAPH_EXTRACT_MODEL` / `RAG_DECISION_MODEL` | model overrides (one role / all roles / graph extraction / decision-model profile) |

{rag_wright-0.16.0 → rag_wright-0.16.1}/docs/domain-adaptation/kg-construction.md

@@ -59,7 +59,9 @@ parse → chunk → segment (+ optional tag) → index  (embed each span → the
   A unit holding exactly one data row of a parsed table carries that row as `Unit.table_row` (exact cells).
 - **extract**: your `extractor` turns each unit into **typed records** (`UnitExtraction`: `KgNode`s and `KgEdge`s
   in your pack's schema). This is the domain-specific step.
-- **write**: persist the records (your `writer`, else `kg_write`), then a generic `Document` node per document.
+- **write**: persist the records (your `writer`, else `kg_write`), then a generic `Document` node per document. Your
+  `writer` returns the number of records now in the graph (the report's `records`), or `None` to let the engine count
+  the record nodes it handed over.
 - **entity graph + resolve**: not run by default. Run your entity extraction and resolution in a `document_hook`
   (see [entity resolution](entity-resolution.md)).
 

{rag_wright-0.16.0 → rag_wright-0.16.1}/docs/specs/public-surface/plan.md

@@ -128,4 +128,7 @@ in two at the PS-6 gate so the store rewrite is its own visible task. One task,
 | PS-54 | TexWright's request (`docs/engine-requests/2026-10-19-substring-match-in-the-kg-filter-grammar.md`, their repo): the KG filter grammar had no substring predicate ("composition contains UHMWPE" over free text). `Contains(text, case_sensitive=False)`, a `where` value beside `Range` on every KG call that takes `where`: a literal `indexOf` test (not `LIKE`, whose `%`, `_` and `?` are wildcards, measured live), case folded on both sides in the store, a field with no value never matches, empty text refused | 8 hermetic tests (red before) + a live test (case both ways, literal wildcards, quotes and backslashes, accents, missing values, with `Range`, `kg_count`, a `kg_edges` traversal filtered at all three points); full hermetic (2353) and live store (42) suites green | done 2026-10-29 |
 | PS-55 | TexWright's request (`docs/engine-requests/2026-10-25-two-model-families-in-one-process-abort-on-metal.md`, their repo): Metal is not safe for concurrent use and the engine put three families on it automatically (BGE-M3, the BGE reranker, Docling's layout/table/RapidOCR models), then ran them on different threads (an ingest embeds while it parses; a server embeds a query while an upload parses), so a process aborted in Metal. RuleWright had measured and pinned it in their own seam (T-7.25/T-7.46), never adopted here. `rag_wright.util.devices` (ADR-0140): only the BGE family on Metal, every call there under one process lock (`METAL_LOCK`); Docling and the pack classifiers on CPU where Metal would be picked (`pack_sdk.torch_device`); explicit device variables win; CUDA/CPU hosts unchanged; `docs/workspaces.md` corrected | red, torch 2.14.1 (TexWright's env, this engine tree): embedder + Docling in two threads, 6/6 child runs died in seconds (5 SIGABRT with their assertion, 1 SIGSEGV; OCR and PDF); torch 2.12.1 never failed. Green, same interpreter: 9/9 runs survived 150s each (OCR, PDF, two embedders). Full hermetic (2361) and live embed/parse/store (60) suites green. 8 hermetic tests (4 behaviour tests red before); Docling output identical CPU vs Metal | done 2026-10-29 |
 | PS-56 | Found during PS-55: `BGEReranker` failed on every call (`XLMRobertaTokenizer has no attribute prepare_for_model`): FlagEmbedding's `FlagAutoReranker.compute_score` calls a tokenizer method transformers 5 removed, so the reference pack's `intra_document_qa` rerank (and RuleWright's copy) and any product composing `BGEReranker` over `span_search` hits were broken. The reranker is now a `sentence_transformers.CrossEncoder` (already a dependency) over the same `bge-reranker-v2-m3`, raw logits, 512 tokens per pair; same `BGEReranker` API, device and lock (PS-55) | red: the live reranker test failed on the tokenizer error. Green: it passes, plus a one-pair and a >512-token case; CrossEncoder's logits equal the model card's plain-transformers recipe (max abs diff 0.0, truncated passage included); embedder + reranker on Metal under the shared lock, torch 2.14.1: 3/3 runs survived 150s, 590 reranks and 586 embeds, none failing (before: the rerank thread died on its first call); full hermetic (2361) green | done 2026-10-29 |
+| PS-57 | RuleWright's request (`docs/engine-requests/2026-10-31-a-write-conflict-loses-a-documents-clauses-and-your-pack-carries-the-defect.md`, their repo): at 4 concurrent documents, 2 of 15 reported `ingested` with zero clauses. The contracts pack's `write` node popped the pending extractions (filled by `extract_clauses`) before the store write landed, so the node's retry wrote nothing and reported success; it now reads them and releases them after the write. The conflict budget is reachable: `StoreConfig.conflict_attempts` / `conflict_backoff_cap_s`, else `ARCADEDB_CONFLICT_ATTEMPTS` / `ARCADEDB_CONFLICT_BACKOFF_CAP_S`; default raised 10 attempts / 0.5 s cap to 20 / 2 s. Found by the live probe: two writers creating the same new node get a duplicate key, never retried; `kg_write` and `write_graph` (UPSERT-only) now retry it | pack retry test through the real per-document graph (0/3 clauses landed before, 3/3 after); six store tests; live probe (clause-shaped transactions over shared value nodes, pooled sessions): old budget lost 38/2,400 at 4 and 8 writers, new lost 0/2,400 at the same wall-clock; live store suite 42 passed | done 2026-11-01 |
+| PS-58 | RuleWright's request (`docs/engine-requests/2026-10-31-ingestionstages-write-returns-a-count-derived-from-its-own-input.md`, their repo): `IngestionStages.write` returned the record nodes it was handed, and `RecordWriter` (declared `-> None`) could not say what landed. A writer now returns the number of records in the graph (`Optional[int]`; `None` keeps the engine's count); the default `kg_write` branch keeps the handed-over count, true because that write is one transaction (lands whole or raises). The contracts pack's writer returns the clauses in the graph and its `write` node reports that as `clauses` | three hermetic tests + the PS-57 retry test's count (red before, green after); live `build_ingestion` on two fixtures: default and a writer landing every other record, `records` equals the graph in all four documents | done 2026-11-01 |
+| PS-59 | Found during PS-57: the ArcadeDB client opens a new connection for every request (`requests.post`, no session); at 8 concurrent writers against Docker on macOS about two thirds of writes failed with `RemoteDisconnected`, and with a keep-alive session per thread the same run lost none. Give the store's client a pooled session | live probe at 8 writers: zero connection drops | done 2026-11-01: the store's client (`_PooledClient`) keeps one keep-alive `requests.Session` per thread and `ArcadeDBStore.close()` releases them; live, the same probe lost 374 of 800 transactions to `RemoteDisconnected` unpooled and 0 of 800 at 8 writers and 0 of 1,600 at 16 pooled |
 | PS-48 | Port the agent harness's substrate behaviour tests from RuleWright onto the engine's fabric fixture (ADR-0139 handover): of RuleWright's 553 harness-touching tests, 314 test the substrate (port), 213 RuleWright's composition (stay), 25 live (1 substrate: OpenRouter cost accounting). Seven gated sub-tasks: **a** shared `tests/harness/fabric.py` + scripting/slow/hanging/failing fake models + `InMemorySaver` helpers, then events, skills, clarify (~23); **b** models and errors (~42); **c** runs, confirm, continuation (~32); **d** decisions and rules (~51); **e** grading (~32); **f** seen and memory (~41); **g** workers, batch, dispatch, tiers, agent wiring, filesystem, interpreter, withhold + the live accounting test (~93, may split). Ambiguous tests split, not moved whole; a ported test failing on engine code is a substrate defect fixed in that sub-task; deadline globals via monkeypatch; no new dependencies. Follow-up (owner RuleWright): retire its substrate duplicates once ported | each sub-task's ported tests green on the fabric fixture | todo (proposed; awaiting go-ahead) |

{rag_wright-0.16.0 → rag_wright-0.16.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "rag-wright"
-version = "0.16.0"
+version = "0.16.1"
 description = "A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime."
 readme = "README.md"
 requires-python = ">=3.12,<3.13"
@@ -51,6 +51,7 @@ dependencies = [
     "pypdfium2>=5.11",
     "pyshacl>=0.40.1",
     "python-dotenv>=1.2",
+    "requests>=2.34.2",
     "scikit-learn>=1.9.1",
     "sentence-transformers>=6.1.0",
     "transformers>=5.4,<5.9",

{rag_wright-0.16.0 → rag_wright-0.16.1}/src/rag_wright/.agents/skills/building-an-ingestion-capability/SKILL.md

@@ -28,7 +28,7 @@ Repository paths in this skill (`docs/`, `eval/`, `scripts/`, `tests/`, `src/`)
 | group spans into units | engine default, domain may override | a heading starts a unit, a table stays whole (a record table is one unit per row), page furniture dropped, units capped | `unit_grouper=`, `boundary_decider=` |
 | choose each unit's representative span (its `anchor` + leading tag) | **domain decision**, optional | the grouper's choice: the first member, often a heading | `unit_representative=` |
 | extract records from a unit | **domain (required)** | (none) | the `extractor` argument |
-| write records | engine default | `kg_write` | `writer=` |
+| write records | engine default | `kg_write` | `writer=` (returns the number of records now in the graph, or `None`) |
 | per-document follow-up (e.g. an entity graph) | domain, optional | none | `document_hook=` |
 | `Document` node, embedded children (`EmbeddedIn` / `AttachedTo`), progress, dead-lettering | engine | always on | (none) |
 

{rag_wright-0.16.0 → rag_wright-0.16.1}/src/rag_wright/api/config.py

@@ -23,6 +23,10 @@ class StoreConfig:
     password: str
     backend: str = "arcadedb"
     protocol: str = "http"
+    # RuleWright (2026-10-31): the write-conflict retry budget. None: `ARCADEDB_CONFLICT_ATTEMPTS` /
+    # `ARCADEDB_CONFLICT_BACKOFF_CAP_S`, else 20 attempts with each wait capped at 2 s (about 25 s in all).
+    conflict_attempts: Optional[int] = None
+    conflict_backoff_cap_s: Optional[float] = None
 
     @classmethod
     def from_env(cls) -> "StoreConfig":

{rag_wright-0.16.0 → rag_wright-0.16.1}/src/rag_wright/api/workspace.py

@@ -131,7 +131,9 @@ def _build_store(config: EngineConfig, corpus: str, *, reset: bool, create: bool
     from rag_wright.store.arcadedb import ArcadeDBStore
 
     return ArcadeDBStore.from_config(sc.host, sc.port, sc.user, sc.password, database=corpus,
-                                     protocol=sc.protocol, reset=reset, pack_ttl=config.pack, create=create)
+                                     protocol=sc.protocol, reset=reset, pack_ttl=config.pack, create=create,
+                                     conflict_attempts=sc.conflict_attempts,
+                                     conflict_backoff_cap_s=sc.conflict_backoff_cap_s)
 
 
 def drop_workspace(config: EngineConfig, *, corpus: str) -> bool:

{rag_wright-0.16.0 → rag_wright-0.16.1}/src/rag_wright/contracts/ingestion.py

@@ -196,9 +196,11 @@ class Extractor(Protocol):
 
 
 class RecordWriter(Protocol):
-    """Optional: persist a document's extractions. The engine default writes them with `kg_write`."""
+    """Optional: persist a document's extractions. The engine default writes them with `kg_write`. Return the number of
+    records now in the graph (written, or found already written); `None` lets the engine count the record nodes it
+    handed over (RuleWright, 2026-10-31: only the writer knows what landed)."""
 
-    def __call__(self, source_doc_id: str, extractions: Sequence[UnitExtraction]) -> Awaitable[None]: ...
+    def __call__(self, source_doc_id: str, extractions: Sequence[UnitExtraction]) -> Awaitable[Optional[int]]: ...
 
 
 # --- the engine-enforced checks --------------------------------------------------------------------------------

{rag_wright-0.16.0 → rag_wright-0.16.1}/src/rag_wright/ingestion/builder.py

@@ -252,9 +252,13 @@ class IngestionStages:
         return stage
 
     async def write(self, sd: Any, extractions: list) -> int:
-        """Persist the records (the domain's writer, else `kg_write`). Returns the number of record nodes."""
+        """Persist the records (the domain's writer, else `kg_write`). Returns the number that landed: the writer's own
+        count when it returns one; else the record nodes handed over, which the default `kg_write` lands in one
+        transaction (it raises rather than land part of them)."""
         if self.writer is not None:
-            await self.writer(sd.source_doc_id, extractions)
+            landed = await self.writer(sd.source_doc_id, extractions)
+            if landed is not None:
+                return int(landed)
         else:
             nodes = [n for e in extractions for n in e.nodes]
             edges = [ed for e in extractions for ed in e.edges]

{rag_wright-0.16.0 → rag_wright-0.16.1}/src/rag_wright/packs/contracts/subgraphs/contract_ingestion_pipeline.py

@@ -966,13 +966,14 @@ def aproduction_document_ingest(
         nodes, edges = clause_kg_graph(record)
         return UnitExtraction(nodes=nodes, edges=edges)
 
-    async def clause_kg_writer(source_doc_id: str, extractions: list) -> None:
-        def _write() -> None:
+    async def clause_kg_writer(source_doc_id: str, extractions: list) -> int:
+        def _write() -> int:
             for e in extractions:  # the write_clause_kg content-hash gate (idempotent re-ingest)
                 if not ckg.already_written(e.nodes[0].props["clause_id"]):
-                    store.kg_write(e.nodes, e.edges)
+                    store.kg_write(e.nodes, e.edges)  # one transaction per clause: it lands whole or raises
+            return len(extractions)  # every clause is now in the graph, written here or before
 
-        await asyncio.to_thread(_write)
+        return await asyncio.to_thread(_write)
 
     stages = IngestionStages(
         store, extractor=clause_extractor_hook, segmenter=legal_segmenter, span_tagger=function_tagger,
@@ -1017,8 +1018,9 @@ def aproduction_document_ingest(
                 entity_rules=entity_rules)))
 
     async def write_fn(doc: SourceDocument, clause_records: list, resolution: Any) -> dict:
-        await stages.write(doc, extractions_by_doc.pop(doc.source_doc_id, []))
-        chunks_by_doc.pop(doc.source_doc_id, None)
+        # read, not pop: `extract_clauses` fills this map, so a retry of this node must still find it (RuleWright,
+        # 2026-10-31: a popped map made the retry write nothing and report success). Released once the write lands.
+        landed = await stages.write(doc, extractions_by_doc.get(doc.source_doc_id, []))  # the clauses in the graph
 
         def _write() -> dict:
             nodes, edges = resolution
@@ -1030,9 +1032,12 @@ def aproduction_document_ingest(
             stages.write_document_node(doc.source_doc_id, parent_id=None,
                                        filename=doc.metadata.get("raw_title", "") or doc.source_doc_id,
                                        media_type="", sha256=hashlib.sha256(doc.text.encode("utf-8")).hexdigest())
-            return {"clauses": len(clause_records), "entities": len(nodes), "edges": len(edges)}
+            return {"clauses": landed, "entities": len(nodes), "edges": len(edges)}
 
-        return await asyncio.to_thread(_write)
+        written = await asyncio.to_thread(_write)
+        extractions_by_doc.pop(doc.source_doc_id, None)
+        chunks_by_doc.pop(doc.source_doc_id, None)
+        return written
 
     return abuild_document_ingest(
         chunk_fn, segment_fn, clauses_fn, index_fn, graph_fn, resolve_fn, write_fn)

{rag_wright-0.16.0 → rag_wright-0.16.1}/src/rag_wright/store/arcadedb.py

@@ -22,9 +22,12 @@ import logging
 import os
 import random
 import re
+import threading
 import time
+from dataclasses import dataclass, replace
 from typing import Any, Iterable, Optional
 
+import requests
 from arcadedb_python import DatabaseDao, SyncClient
 
 from rag_wright.contracts.chunk import BGE_M3_DENSE_DIM, ChunkRecord, MetadataValue
@@ -70,39 +73,59 @@ _SPAN_SPARSE_INDEX = f"{SPAN_TYPE}[sparse_indices,sparse_weights]"
 # ArcadeDB answers concurrent writes to the same page with a `ConcurrentModificationException` and asks the client to
 # retry; the failed command or transaction is rolled back whole, so a bounded retry with jittered backoff is safe
 # (found by the ING-5 doc audit: unretried, concurrent ingestion silently lost spans and records).
-_CONFLICT_RETRIES = 10
+# RuleWright (2026-10-31): the budget is the store's (`StoreConfig.conflict_attempts` / `conflict_backoff_cap_s`, else
+# `ARCADEDB_CONFLICT_ATTEMPTS` / `ARCADEDB_CONFLICT_BACKOFF_CAP_S`, else these defaults). Measured live with clause-shaped
+# transactions over shared value nodes: 10 attempts capped at 0.5 s (about 2 s of waiting) lost 38 of 2,400 at 4 and 8
+# writers; 20 capped at 2 s lost none, at the same wall-clock (it waits only while a write conflicts).
+_CONFLICT_RETRIES = 20
+_CONFLICT_BACKOFF_CAP_S = 2.0
+
+
+@dataclass(frozen=True)
+class _ConflictBudget:
+    attempts: int = _CONFLICT_RETRIES
+    cap_s: float = _CONFLICT_BACKOFF_CAP_S
+    # Two writers UPSERTing the same NEW key: the loser gets a duplicate key on the unique index, not a page conflict.
+    # Retried only where every node write is an UPSERT (the retry then finds the winner's record), never for a
+    # genuine duplicate.
+    duplicate_key: bool = False
+
+
 # PS-49 (TexWright, 2026-10-17): the client logs every error response at ERROR before raising, so a conflict the retry
 # absorbs printed exactly like a failure. While a retry is handling the call, that line is dropped; the retry says
 # itself what happened (INFO when the write landed after retrying, ERROR when it gave up).
-_RETRYING = contextvars.ContextVar("arcadedb_retrying", default=False)
+_RETRYING: contextvars.ContextVar[Optional[_ConflictBudget]] = contextvars.ContextVar("arcadedb_retrying", default=None)
 
 
 class _RetriedConflictFilter(logging.Filter):
     def filter(self, record: logging.LogRecord) -> bool:
-        return not (_RETRYING.get() and _is_retryable_conflict(Exception(record.getMessage())))
+        budget = _RETRYING.get()
+        return not (budget and _is_retryable_conflict(Exception(record.getMessage()),
+                                                      duplicate_key=budget.duplicate_key))
 
 
 logging.getLogger("arcadedb_python.api.sync").addFilter(_RetriedConflictFilter())
 
 
-def _is_retryable_conflict(exc: BaseException) -> bool:
+def _is_retryable_conflict(exc: BaseException, *, duplicate_key: bool = False) -> bool:
     msg = str(exc)
-    return "ConcurrentModificationException" in msg or "Concurrent modification on page" in msg
+    return ("ConcurrentModificationException" in msg or "Concurrent modification on page" in msg
+            or (duplicate_key and ("DuplicatedKeyException" in msg or "Found duplicate key" in msg)))
 
 
-def _retry_on_conflict(fn, *args, **kwargs):
-    token = _RETRYING.set(True)
+def _retry_on_conflict(fn, *args, budget: _ConflictBudget = _ConflictBudget(), **kwargs):
+    token = _RETRYING.set(budget)
     try:
-        for attempt in range(_CONFLICT_RETRIES):
+        for attempt in range(budget.attempts):
             try:
                 result = fn(*args, **kwargs)
             except Exception as exc:  # noqa: BLE001 - only the retryable conflict is retried; anything else re-raises
-                if not _is_retryable_conflict(exc):
+                if not _is_retryable_conflict(exc, duplicate_key=budget.duplicate_key):
                     raise
-                if attempt == _CONFLICT_RETRIES - 1:
-                    _log.error("ArcadeDB write conflict: gave up after %d attempts: %s", _CONFLICT_RETRIES, exc)
+                if attempt == budget.attempts - 1:
+                    _log.error("ArcadeDB write conflict: gave up after %d attempts: %s", budget.attempts, exc)
                     raise
-                time.sleep(min(0.5, 0.01 * 2 ** attempt) + random.uniform(0, 0.01))
+                time.sleep(min(budget.cap_s, 0.01 * 2 ** attempt) + random.uniform(0, 0.01))
             else:
                 if attempt:
                     _log.info("ArcadeDB write conflict retried %d times; the write landed", attempt)
@@ -312,13 +335,67 @@ def _sql_literal(value: MetadataValue) -> str:
     return _sql_str(str(value))
 
 
+class _PooledClient(SyncClient):
+    """`SyncClient` over keep-alive connections, one `requests.Session` per thread (PS-59).
+
+    `SyncClient.post` / `get` call the bare `requests.post` / `requests.get`, so every request opens and closes a TCP
+    connection. Measured live during PS-57, 8 concurrent writers against Docker Desktop on macOS lost about two thirds
+    of their writes to `RemoteDisconnected`, and none with one keep-alive session per thread. Per thread because a
+    `Session` is not documented as thread-safe and the store is called from `asyncio.to_thread` workers. The two
+    methods are the library's own with the session in place of the module call (arcadedb_python 0.4.0); ArcadeDB
+    keeps a transaction in the `arcadedb-session-id` header and sets no cookies, so a session carries nothing else."""
+
+    def __init__(self, host: str, port: str, protocol: str = "http", **kwargs):
+        self._local = threading.local()
+        self._sessions: list[requests.Session] = []
+        self._sessions_lock = threading.Lock()
+        super().__init__(host, port, protocol, **kwargs)  # its login check already posts through `_session`
+
+    def _session(self) -> requests.Session:
+        session = getattr(self._local, "session", None)
+        if session is None:
+            session = self._local.session = requests.Session()
+            with self._sessions_lock:
+                self._sessions.append(session)
+        return session
+
+    def post(self, endpoint: str, payload: dict, return_headers: bool = False, extra_headers: dict = {}):
+        response = self._session().post(self._get_endpoint(endpoint), data=json.dumps(payload),
+                                        headers={**self.headers, **extra_headers},
+                                        auth=(self.username, self.password))
+        query = payload.get("command") if isinstance(payload, dict) else None
+        return self.subhandler(response, return_headers=return_headers, query=query)
+
+    def get(self, endpoint: str, return_headers: bool = False, extra_headers: dict = {}):
+        response = self._session().get(self._get_endpoint(endpoint), auth=(self.username, self.password),
+                                       headers=extra_headers)
+        return self.subhandler(response, return_headers=return_headers)
+
+    def close(self) -> None:
+        """Close every thread's session. A thread that calls again afterwards opens a new one."""
+        with self._sessions_lock:
+            sessions, self._sessions = self._sessions, []
+            self._local = threading.local()
+        for session in sessions:
+            session.close()
+
+
 class ArcadeDBStore:
     """The default store: schema management over a single ArcadeDB database."""
 
-    def __init__(self, client: SyncClient, database: str, *, pack_ttl: str | None = None) -> None:
+    _conflicts = _ConflictBudget()  # the default write-conflict budget; `__init__` resolves the configured one
+
+    def __init__(self, client: SyncClient, database: str, *, pack_ttl: str | None = None,
+                 conflict_attempts: int | None = None, conflict_backoff_cap_s: float | None = None) -> None:
         self._client = client
         self._database = database
         self._db = DatabaseDao(client, database)
+        # RuleWright (2026-10-31): the write-conflict retry budget (argument, else environment, else the default)
+        attempts = conflict_attempts or int(os.getenv("ARCADEDB_CONFLICT_ATTEMPTS") or _CONFLICT_RETRIES)
+        cap_s = conflict_backoff_cap_s or float(os.getenv("ARCADEDB_CONFLICT_BACKOFF_CAP_S") or _CONFLICT_BACKOFF_CAP_S)
+        if attempts < 1 or cap_s <= 0:
+            raise ValueError(f"the conflict budget needs attempts >= 1 and a backoff cap > 0 (got {attempts}, {cap_s})")
+        self._conflicts = _ConflictBudget(attempts, cap_s)
         # AC-journey / ING-8a: the DOMAIN pack `.ttl` whose KG vertex/edge types `ensure_schema` creates and whose
         # property storage types `kg_write` encodes by. `None` = NO domain pack (the neutral engine schema); a domain
         # points this at its OWN pack ("config + .ttl", no engine edit), or ensures it on use (`ensure_pack_schema`).
@@ -326,7 +403,8 @@ class ArcadeDBStore:
 
     @classmethod
     def from_env(cls, *, database: str | None = None, reset: bool = False,
-                 pack_ttl: str | None = None, create: bool = True) -> "ArcadeDBStore":
+                 pack_ttl: str | None = None, create: bool = True, conflict_attempts: int | None = None,
+                 conflict_backoff_cap_s: float | None = None) -> "ArcadeDBStore":
         """Build a store from `ARCADEDB_*` env, creating the database if absent (logged); `create=False` raises
         `LookupError` instead (PS-27), for a path where the database must already exist.
 
@@ -339,21 +417,25 @@ class ArcadeDBStore:
             os.environ["ARCADEDB_USER"], os.environ["ARCADEDB_PASSWORD"],
             database=database or os.environ["ARCADEDB_DATABASE"],
             protocol=os.getenv("ARCADEDB_PROTOCOL", "http"),  # `https` for the Modal-hosted KG (EC-2)
-            reset=reset, pack_ttl=pack_ttl, create=create)
+            reset=reset, pack_ttl=pack_ttl, create=create, conflict_attempts=conflict_attempts,
+            conflict_backoff_cap_s=conflict_backoff_cap_s)
 
     @classmethod
     def from_config(cls, host: str, port: str, user: str, password: str, *, database: str,
                     protocol: str = "http", reset: bool = False,
-                    pack_ttl: str | None = None, create: bool = True) -> "ArcadeDBStore":
+                    pack_ttl: str | None = None, create: bool = True, conflict_attempts: int | None = None,
+                    conflict_backoff_cap_s: float | None = None) -> "ArcadeDBStore":
         """Build a store from EXPLICIT connection params (EP-API-1: the de-env'd twin of `from_env`, so engine
         config flows as data, not `os.environ`). Creates the database if absent, and logs that it did (a product
         provisions a new corpus this way); `create=False` raises `LookupError` naming the databases that exist,
         and creates nothing (PS-27: a wrong name otherwise yields an empty graph, silently). `reset=True` drops +
         recreates it, so it cannot be combined with `create=False`. `pack_ttl` selects the domain pack schema
-        (None = the neutral engine schema, no domain pack)."""
+        (None = the neutral engine schema, no domain pack). `conflict_attempts` / `conflict_backoff_cap_s` set the
+        write-conflict retry budget (None: `ARCADEDB_CONFLICT_ATTEMPTS` / `ARCADEDB_CONFLICT_BACKOFF_CAP_S`, else 20
+        attempts with waits capped at 2 s)."""
         if reset and not create:
             raise ValueError("reset=True drops and recreates the database; it cannot be combined with create=False")
-        client = SyncClient(host, port, protocol=protocol, username=user, password=password)
+        client = _PooledClient(host, port, protocol=protocol, username=user, password=password)
         if reset and DatabaseDao.exists(client, database):
             DatabaseDao.delete(client, database)
         if not DatabaseDao.exists(client, database):
@@ -362,7 +444,8 @@ class ArcadeDBStore:
                 raise LookupError(f"database {database!r} does not exist (create=False); existing: {existing}")
             DatabaseDao.create(client, database)
             _log.info("created database %r (it did not exist)", database)
-        return cls(client, database, pack_ttl=pack_ttl)
+        return cls(client, database, pack_ttl=pack_ttl, conflict_attempts=conflict_attempts,
+                   conflict_backoff_cap_s=conflict_backoff_cap_s)
 
     @staticmethod
     def drop_database(host: str, port: str, user: str, password: str, *, database: str,
@@ -566,8 +649,10 @@ class ArcadeDBStore:
         return DatabaseDao.exists(self._client, self._database)
 
     def close(self) -> None:
-        # The HTTP client holds no persistent connection to release.
-        pass
+        # PS-59: release the client's keep-alive connections (a client built elsewhere may hold none)
+        close = getattr(self._client, "close", None)
+        if close is not None:
+            close()
 
     # --- write-side (T20) -----------------------------------------------------------------------
 
@@ -779,7 +864,9 @@ class ArcadeDBStore:
                 stmt += " SET " + ", ".join(f"{k} = {_kg_sql(v)}" for k, v in e.props.items())
             statements.append(stmt)
         if statements:
-            _retry_on_conflict(self._db.execute_transaction, statements)
+            # every node is an UPSERT, so a duplicate key here is two writers racing to create one node
+            _retry_on_conflict(self._db.execute_transaction, statements,
+                               budget=replace(self._conflicts, duplicate_key=True))
 
     def kg_ensure_edges(self, edges) -> int:
         """ING-4b (see `Store.kg_ensure_edges`): create each typed edge only if no edge of its type already joins
@@ -1026,7 +1113,9 @@ class ArcadeDBStore:
                 f" confidence = {_sql_str(edge.confidence)}, {_edge_provenance_assignments(edge.chunk_id)}"
             )
         if statements:
-            _retry_on_conflict(self._db.execute_transaction, statements)
+            # every node is an UPSERT, so a duplicate key here is two writers racing to create one node
+            _retry_on_conflict(self._db.execute_transaction, statements,
+                               budget=replace(self._conflicts, duplicate_key=True))
 
     def _mentions_edge_exists(self, chunk_id: str, node_key: str) -> bool:
         """True iff a `Mentions` edge already connects this chunk to this entity (issue 0029 idempotence).
@@ -1200,7 +1289,7 @@ class ArcadeDBStore:
     # --- internals ------------------------------------------------------------------------------
 
     def _command(self, sql: str) -> Any:
-        return _retry_on_conflict(self._db.query, "sql", sql, is_command=True)
+        return _retry_on_conflict(self._db.query, "sql", sql, is_command=True, budget=self._conflicts)
 
     def _query(self, sql: str) -> list[dict]:
         result = self._db.query("sql", sql)

{rag_wright-0.16.0 → rag_wright-0.16.1}/tasks.md

@@ -3,6 +3,8 @@
 Phase 2 output. The persistent, cross-session task ledger and shared memory of progress. Derived
 from `plan.md` (Phase 1) and `SPEC.md` v0.1, honoring ADR-0001 (stack) and ADR-0002 (corpus).
 
+> **2026-11-01: PS-57 done, unreleased** (RuleWright's request: a retried contracts-pack write no longer loses a document's clauses; the ArcadeDB conflict budget is configurable, default 20 attempts / 2 s cap; a racing UPSERT's duplicate key is retried). PS-58 done, unreleased (RuleWright's second request: a `RecordWriter` returns the count that landed). PS-59 done, unreleased (the store's client keeps keep-alive connections, one session per thread; `requests` declared). Next: PS-48.
+>
 > **⇒ RESUME / NEXT UP (2026-10-17, NEWEST): the LangGraph ingestion workstream is RELEASED as `rag-wright` 0.14.0** (PR #16)
 > (`docs/specs/langgraph-ingestion/plan.md`, LGI-1..5; ADR-0124 amended; RuleWright guide
 > `docs/specs/public-surface/rulewright-migration-0.14.0.md`). PS-49 done 2026-10-17 (a retried ArcadeDB write conflict no
@@ -10,7 +12,7 @@ from `plan.md` (Phase 1) and `SPEC.md` v0.1, honoring ADR-0001 (stack) and ADR-0
 > existing type; TexWright's request); both RELEASED as `rag-wright` 0.14.1 (2026-10-28, PR #17). PS-51 done 2026-10-28 (the frontend skill draws the shell and ships
 > its scaffold; RuleWright's request). PS-52 done 2026-10-28 (a workspace builds one embedder under
 > concurrent first use; TexWright's 2026-10-24 request). **The open list, in order:** (1) TexWright's
-> engine requests: 2026-10-21 unescaped KG identifiers (PS-53 done 2026-10-28; PS-51..53 RELEASED as `rag-wright` 0.15.0, 2026-10-28, PR #18), 2026-10-19 substring filter (PS-54 `Contains` done 2026-10-29, unreleased, lands as 0.16.0), 2026-10-25 Metal abort (PS-55 done 2026-10-29: one model family on Metal, ADR-0140; unreleased, lands in 0.16.0; PS-56 done 2026-10-29: the reranker works on transformers 5 again);
+> engine requests: 2026-10-21 unescaped KG identifiers (PS-53 done 2026-10-28; PS-51..53 RELEASED as `rag-wright` 0.15.0, 2026-10-28, PR #18), 2026-10-19 substring filter (PS-54 `Contains` done 2026-10-29), 2026-10-25 Metal abort (PS-55 done 2026-10-29: one model family on Metal, ADR-0140; PS-56 done 2026-10-29: the reranker works on transformers 5 again; PS-54..56 RELEASED as `rag-wright` 0.16.0, 2026-10-29, PR #19);
 > (2) **PS-48**, port the harness substrate tests from RuleWright onto the fabric fixture (ledger row in
 > `docs/specs/public-surface/plan.md`, sub-tasks a..g, awaiting go-ahead); (3) EV-7, PS-30, PS-33. Before it,
 > **(2026-10-14): PS-32 done, the agent harness is `rag_wright.harness`** (third public

rag_wright-0.16.1/tests/ingestion/test_write_count.py

@@ -0,0 +1,39 @@
+"""RuleWright's request (2026-10-31): `IngestionStages.write` reported the records it was OFFERED, and a domain writer
+had no way to say what landed. A writer may now return the number that landed; `None` keeps the engine's count."""
+from __future__ import annotations
+
+import asyncio
+
+from rag_wright.api import KgNode, UnitExtraction
+from rag_wright.pack_sdk import IngestionStages
+from rag_wright.packs.contracts.subgraphs.contract_ingestion_pipeline import SourceDocument
+
+_DOC = SourceDocument(source_doc_id="D1", text="text of D1")
+_EXTRACTIONS = [UnitExtraction(nodes=[KgNode("Rec", "k", {"k": f"r{i}"})]) for i in range(3)]
+
+
+def _stages(writer=None, store=None, tmp_path="."):
+    return IngestionStages(store, extractor=None, writer=writer, cache_dir=tmp_path)
+
+
+def test_the_writer_reports_what_landed(tmp_path):
+    async def writer(doc_id, extractions):
+        return 1  # it landed one of the three
+
+    assert asyncio.run(_stages(writer, tmp_path=tmp_path).write(_DOC, _EXTRACTIONS)) == 1
+
+
+def test_a_writer_that_returns_nothing_keeps_the_engine_count(tmp_path):
+    async def writer(doc_id, extractions):
+        return None
+
+    assert asyncio.run(_stages(writer, tmp_path=tmp_path).write(_DOC, _EXTRACTIONS)) == 3
+
+
+def test_the_default_writer_counts_the_nodes_its_one_transaction_landed(tmp_path):
+    class _Store:
+        def kg_write(self, nodes, edges=()):
+            self.nodes = list(nodes)
+
+    store = _Store()
+    assert asyncio.run(_stages(store=store, tmp_path=tmp_path).write(_DOC, _EXTRACTIONS)) == 3 == len(store.nodes)