rag-wright 0.17.0__tar.gz → 0.17.2__tar.gz

{rag_wright-0.17.0 → rag_wright-0.17.2}/.claude/skills/building-an-ingestion-capability/SKILL.md

@@ -199,7 +199,11 @@ category.
 - **One record over several rows, or one entity over many units.** The extractor sees one unit at a time. A record
   whose continuation rows sit under its main row needs a `unit_grouper=` that keeps them together, or a
   `document_hook=` that reads the whole table with `table_rows` and writes the grouped record. An entity several units
-  mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key.
+  mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key; an edge to it
+  emitted from each unit lands once, because the default writer keeps one edge per (type, from node, to node)
+  (PS-63). A `writer=` of your own gets the same with `kg_write(ws, nodes, edges, ensure_edges=True)`; leave the flag
+  off only when parallel edges of one type between the same two nodes are meant to stay (each with its own
+  provenance).
 - **Resume a long ingest instead of restarting it.** Pass a LangGraph checkpointer (yours, for example a Postgres
   saver; the engine ships none) and a `run_id` naming the run:
   `await pipeline.aingest(ws, sources, cache_dir=CACHE, checkpointer=saver, run_id=batch_id)`. If the process stops,
@@ -208,8 +212,9 @@ category.
   sources, raise `ValueError`: a new run gets a new id. The graph state is built-in values only, so it works under
   LangGraph's strict msgpack mode; a tuple inside a record's free-form `props` comes back as a list after a resume.
 - **Re-ingest.** Parsing is cached by content hash, so re-running is cheap. The default writer upserts nodes by their
-  key but is not fully idempotent for edges (engine gap G8): when a source is re-run, remove its previous records
-  first (`kg_delete` on the provenance your records carry, such as a per-source property) and write again.
+  key and writes each edge once, updating its props, so a re-run leaves no duplicates. It does not remove what a
+  source no longer yields: when a re-run should drop records, remove its previous records first (`kg_delete` on the
+  provenance your records carry, such as a per-source property) and write again.
 
 ## 6. Verify (definition of done)
 

rag_wright-0.17.2/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.17.2"
+}

{rag_wright-0.17.0 → rag_wright-0.17.2}/CHANGELOG.md

@@ -4,6 +4,21 @@ All notable changes to RAG_Wright are documented here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project aims to follow
 [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.17.2](https://github.com/fzaidi-dai/RAG_Wright/compare/0.17.1...0.17.2) (2026-11-11)
+
+
+### Bug Fixes
+
+* **PS-64:** a float is written and compared as a double, so an unchanged DOUBLE settles ([a337890](https://github.com/fzaidi-dai/RAG_Wright/commit/a33789085bb6623890b26fb0e361ed01bb8d5e0b))
+* **PS-65:** one loaded BGE model per process, shared with its one lock ([676d0a5](https://github.com/fzaidi-dai/RAG_Wright/commit/676d0a512ecd4494e8fff2e5f5266958b0a1f2d9))
+
+## [0.17.1](https://github.com/fzaidi-dai/RAG_Wright/compare/0.17.0...0.17.1) (2026-11-07)
+
+
+### Bug Fixes
+
+* **PS-63:** the default record writer keeps one edge per type and endpoints (G8) ([e77ca91](https://github.com/fzaidi-dai/RAG_Wright/commit/e77ca9151e8cd5a425237f30ada721349eac7bb0))
+
 ## [0.17.0](https://github.com/fzaidi-dai/RAG_Wright/compare/0.16.1...0.17.0) (2026-11-05)
 
 

{rag_wright-0.17.0 → rag_wright-0.17.2}/CLAUDE.md

@@ -55,7 +55,15 @@ The failure mode to avoid, which has happened before: a command fails with "modu
 
 ## Start of every session
 
-> **CURRENT STATE (2026-11-01): `rag-wright` 0.16.1 is RELEASED** (PR #20; RuleWright's two requests and a finding
+> **CURRENT STATE (2026-11-11): PS-64 and PS-65 done, unreleased** (TexWright: a float is written and compared as a
+> double, ADR-0143; one loaded BGE model per process with its one lock, ADR-0144). Next: PS-48, EV-7, PS-30, PS-33. Before it,
+> **`rag-wright` 0.17.1 is RELEASED** (PR #22; PS-63, TexWright's G8 request: the default record writer keeps one
+> edge per type and endpoints, `kg_write(..., ensure_edges=True)`, ADR-0142). Next: PS-48, EV-7, PS-30, PS-33. Before it,
+> **`rag-wright` 0.17.0 is RELEASED** (PR #21; TexWright's request: PS-60 a table over
+> the chunk cap is split between lines with its header repeated, so every row keeps its span and `TableRow` (ADR-0141);
+> PS-61 the default segmenter `segment_layout` is on `pack_sdk`; PS-62 a record table's rows that reach no unit are
+> reported, `DocumentReport.unreached_table_rows`). Next: PS-48, EV-7, PS-30, PS-33. Before it,
+> **`rag-wright` 0.16.1 is RELEASED** (PR #20; RuleWright's two requests and a finding
 > from them: PS-57 a retried contracts-pack write no longer loses a document's clauses, the ArcadeDB conflict budget is
 > configurable (default 20 attempts / 2 s cap) and a racing UPSERT's duplicate key is retried; PS-58 a `RecordWriter`
 > returns the count that landed; PS-59 the store's client keeps keep-alive connections, one session per thread). Next:

{rag_wright-0.17.0 → rag_wright-0.17.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: rag-wright
-Version: 0.17.0
+Version: 0.17.2
 Summary: A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime.
 Project-URL: Homepage, https://github.com/fzaidi-dai/RAG_Wright
 Project-URL: Repository, https://github.com/fzaidi-dai/RAG_Wright

rag_wright-0.17.2/docs/adr/0142-the-default-writer-keeps-one-edge-per-type-and-endpoints.md

@@ -0,0 +1,32 @@
+# ADR-0142: The default record writer keeps one edge per type and endpoints
+
+**Status:** accepted · **Date:** 2026-11-07 · **Related:** PS-63, engine gap G8, TexWright's request
+`2026-10-15-g8-duplicates-edges-within-one-ingest.md`, ADR-0124 (ING-4b `kg_ensure_edges`), issue 0029
+(`outV()`/`inV()` endpoint filters)
+
+## Context
+
+With no `writer=`, `build_ingestion` wrote records with `kg_write`: nodes upserted by key, edges created with a plain
+`CREATE EDGE`. The engine's own guidance has an entity several units mention emitted by its key from each unit, so
+each such unit also emitted the edge to it, and that edge landed once per unit, within one ingest and again on every
+re-ingest. Node counts stayed right, so only a traversal showed it (TexWright: 3 children read back as 6). The
+store's idempotent `kg_ensure_edges` runs a query and a write per edge outside any transaction, and scans the edge
+type for each edge.
+
+## Decision
+
+- `Store.kg_write` takes `ensure_edges` (default `False`). When true, an edge is identified by its type and its two
+  endpoints (direction counts): the batch keeps each edge's last props; each edge is an index-driven `UPDATE` of an
+  existing one, reached through `outE` from its indexed from node, then `CREATE EDGE ... IF NOT EXISTS`, all in the
+  write's one transaction under the same conflict retry. ArcadeDB 26.7 was probed live: `IF NOT EXISTS` holds within
+  one transaction, across transactions and across concurrent writers.
+- The ingestion default writer passes `True`; `rag_wright.api.kg_write` exposes the flag for a domain's own writer.
+- Without the flag `kg_write` is unchanged: a caller whose parallel edges each carry their own provenance (the
+  reference pack's clause-to-value assertions, RuleWright's writer) keeps them.
+
+## Consequences
+
+- An edge several units emit, or a re-ingest writes again, lands once with the last unit's props; G8 is closed.
+- A domain that means parallel edges of one type between the same two nodes, and uses the default writer, now gets
+  one; it passes a `writer=` that calls `kg_write` without the flag.
+- A re-ingest still does not remove records a document no longer yields. `kg_ensure_edges` is unchanged.

rag_wright-0.17.2/docs/adr/0143-a-float-is-written-and-compared-as-a-double.md

@@ -0,0 +1,33 @@
+# ADR-0143: A float is written and compared as a double
+
+**Status:** accepted · **Date:** 2026-11-11 · **Related:** PS-64, TexWright's request
+`2026-11-09-kg-update-rewrites-unchanged-doubles.md`, DD-1a/DD-1b (`kg_read` / `kg_write` encoding), ER-1 (`Range`),
+PS-54 (the shared filter grammar)
+
+## Context
+
+The store put a Python float into SQL as `repr(value)`, a bare decimal literal. ArcadeDB 26.7 reads a bare decimal
+literal as a 32-bit FLOAT (probed live). A DOUBLE or DECIMAL column holding a value not exact in binary, such as
+0.9523, therefore never equalled it: `kg_update`'s "only rows that differ" test was always true, so a derive step
+rewrote 50 of 50 rows on every pass; equality, membership and `Range(ge=v, le=v)` filters on that column found 0 of
+50; and a float on a property the pack does not declare, or on an edge, was stored at 32 bits (123456789.12345679
+read back as 123456790.0). No pack in the engine, RuleWright or TexWright declares a FLOAT column; 54 declare DOUBLE.
+
+## Decision
+
+- A float is written and compared as a double literal, `repr(value) + "d"`, which ArcadeDB keeps at 64 bits (probed
+  for exponents, negatives, subnormals and 17 significant digits). Only a field the pack declares `FLOAT` gets the
+  bare literal, since a 32-bit column does not equal a double either.
+- The filter grammar takes the queried type's declared property types, so a filter encodes each value as its
+  column is declared: `kg_read`, `kg_count`, `kg_delete`, `kg_update` and `kg_edges`' start node.
+- On a field the pack does not declare, a float equality or membership filter matches either width,
+  `(f = Xd OR f = X)`, so values stored at 32 bits before this change are still found. `kg_update` on such a field
+  rewrites an old 32-bit value once, as a double, and then settles.
+
+## Consequences
+
+- An unchanged measurement is a no-op for `kg_update`; filters on a DOUBLE find their values; new undeclared floats
+  keep full precision. Live, the 50-row derive step changes 50 then 0 rows (50 then 50 before).
+- Edge-property and target filters in `kg_edges` have no declared types, so they use the either-width form.
+- Out of scope and unchanged: the hybrid-search chunk filter (`hybrid_search(filters=)`, its own literal encoder)
+  and the vector arrays (`ARRAY_OF_FLOATS`, 32-bit by declaration).

rag_wright-0.17.2/docs/adr/0144-one-loaded-bge-model-per-process.md

@@ -0,0 +1,31 @@
+# ADR-0144: One loaded BGE model per process, with its one lock
+
+**Status:** accepted · **Date:** 2026-11-11 · **Related:** PS-65, TexWright's request
+`2026-11-09-the-embedder-model-is-rebuilt-for-every-workspace.md`, issue 0016 (in-place mutation on encode), PS-52
+(the workspace embedder's lazy build), PS-55 / ADR-0140 (`METAL_LOCK`)
+
+## Context
+
+Every `BGEM3Embedder` and `BGEReranker` loaded its own copy of its model: each workspace's query embedder, each
+ingestion pipeline's embedder, and the reference pack's `intra_document_qa`, which builds a reranker per call. Nothing
+chose this; PS-52's lock only makes one handle's lazy build atomic. Live, one process holding three workspaces, an
+ingestion pipeline and two rerankers peaked at 2816 MB, each repeat costing 1 to 1.5 s. The lock a wrapper holds
+was per instance, so two wrappers of one shared model would each lock their own lock and run the model together,
+which issue 0016 shows segfaults.
+
+## Decision
+
+- `rag_wright.util.devices.shared_model(key, device, build)` keeps one loaded model per key -- (kind, model name,
+  precision, device) -- for the life of the process, built once under a guard, and returns it WITH its one lock
+  (`model_lock`: `METAL_LOCK` on Metal, else a lock of its own). Both BGE wrappers take their model and lock from it,
+  so every wrapper of a model serializes on the same lock. An injected `model=` is not shared.
+- The wrappers stay per workspace; only the model and its lock are shared.
+
+## Consequences
+
+- Live, the same process peaks at 1534 MB and every repeat is free; the encodes through two wrappers from 8 threads
+  equal the serial vectors.
+- Off Metal, two tenants' encodes on one model now queue on its lock where each had its own model before. Encoding is
+  a small slice of an ingest and Metal already serialized everything, so memory and load time win. A tenant that needs
+  its own model runs in its own process.
+- A loaded model is never freed before the process exits; dropping a workspace does not release it.

{rag_wright-0.17.0 → rag_wright-0.17.2}/docs/adr/README.md

@@ -197,4 +197,7 @@ framing that still appears in older ADRs and the root `SPEC.md`/`plan.md`.
 0138 PDF parses run one at a time, and the engine reaches pdfium through one locked door ·
 0139 The agent harness substrate is engine scope ·
 0140 One model family on Metal, under one process lock ·
-0141 The cap's hard split cuts between lines and repeats a table's header
+0141 The cap's hard split cuts between lines and repeats a table's header ·
+0142 The default record writer keeps one edge per type and endpoints ·
+0143 A float is written and compared as a double ·
+0144 One loaded BGE model per process, with its one lock

{rag_wright-0.17.0 → rag_wright-0.17.2}/docs/api/README.md

@@ -430,9 +430,9 @@ Rank the live ARD catalog by semantic match to `query`; return the top `k` (opti
 
 Read typed nodes of `node_type` from the workspace (see `Store.kg_read`). Each `where` value is a scalar (equality), a list (membership; an empty list is scope-to-nothing -> `[]`), `NOT_NULL` (presence), a `Range` (comparisons, on any number of fields: `{"weight": Range(ge=180, le=220), "tensile": Range(gt=400)}`) or a `Contains` (a literal substring, case-insensitive unless `case_sensitive=True`; an empty field never matches: `{"composition": Contains("UHMWPE")}`); all filters are AND-ed and evaluated in the store. `key_range=(field, lo, hi)` is the half-open shorthand (`lo <= field < hi`). Then projection, distinct, order and limit. Every name is checked, never interpolated unchecked (the module docstring): a request-chosen field is safe, but an unknown one matches nothing.
 
-### `kg_write(ws: 'WorkspaceHandle', nodes: 'list', edges: 'Any' = ()) -> 'None'`
+### `kg_write(ws: 'WorkspaceHandle', nodes: 'list', edges: 'Any' = (), *, ensure_edges: 'bool' = False) -> 'None'`
 
-Upsert typed `nodes` + create typed `edges` in one transaction (see `Store.kg_write`). `nodes`/`edges` are `KgNode`/`KgEdge` (from `rag_wright.api`); the store encodes each field per its pack-declared type. A node type the workspace's schema does not declare fails the write.
+Upsert typed `nodes` + create typed `edges` in one transaction (see `Store.kg_write`). `nodes`/`edges` are `KgNode`/`KgEdge` (from `rag_wright.api`); the store encodes each field per its pack-declared type. A node type the workspace's schema does not declare fails the write. `ensure_edges=True` (PS-63) writes at most one edge per (type, from node, to node), the last props winning: what a domain `writer=` passes to match the default writer.
 
 ### `kg_edges(ws: 'WorkspaceHandle', from_type: 'Optional[str]' = None, *, where: 'Optional[dict]' = None, key_range: 'Optional[tuple]' = None, direction: 'str' = 'out', edge_type: 'Optional[str]' = None, edge_where: 'Optional[dict]' = None, target_where: 'Optional[dict]' = None, select: 'dict') -> 'list[dict]'`
 

{rag_wright-0.17.0 → rag_wright-0.17.2}/docs/api/pack_sdk.md

@@ -120,7 +120,7 @@ The graph query capability's output: candidate answers as evidence for fusion (F
 
 The real reranker: a `sentence_transformers.CrossEncoder` over `bge-reranker-v2-m3` (PS-56), scoring raw logits (no activation), 512 tokens per pair as before.
 
-THREAD-SAFE (engine issue 0016): a shared reranker is guarded by an instance lock -- defense-in-depth for any concurrent `score` caller (its FlagEmbedding predecessor mutated the model in place on every call). It is in the BGE family: its device follows the embedder's (`EMBED_DEVICE`), and on Metal its lock is the process-wide one the embedders hold (PS-55), so a rerank never runs on Metal beside an embed.
+THREAD-SAFE (engine issue 0016): a shared reranker is guarded by an instance lock -- defense-in-depth for any concurrent `score` caller (its FlagEmbedding predecessor mutated the model in place on every call). It is in the BGE family: its device follows the embedder's (`EMBED_DEVICE`), and on Metal its lock is the process-wide one the embedders hold (PS-55), so a rerank never runs on Metal beside an embed. Like the embedder, the loaded model is shared by every reranker of the same (model, precision, device) in the process, with its one lock (PS-65).
 
 #### `BGEReranker.score(self, query: 'str', passages: 'list[str]') -> 'list[float]'`
 
@@ -182,7 +182,7 @@ Group into units (checked), attach table rows, extract each unit concurrently (p
 
 #### `IngestionStages.write(self, sd: 'Any', extractions: 'list') -> 'int'`
 
-Persist the records (the domain's writer, else `kg_write`). Returns the number that landed: the writer's own count when it returns one; else the record nodes handed over, which the default `kg_write` lands in one transaction (it raises rather than land part of them).
+Persist the records (the domain's writer, else `kg_write` with idempotent edges). Returns the number that landed: the writer's own count when it returns one; else the record nodes handed over, which the default `kg_write` lands in one transaction (it raises rather than land part of them).
 
 #### `IngestionStages.write_document_node(self, doc_id: 'str', *, parent_id: 'Optional[str]', filename: 'str', media_type: 'str', sha256: 'str', metadata: 'Optional[dict]' = None) -> 'None'`
 

{rag_wright-0.17.0 → rag_wright-0.17.2}/docs/domain-adaptation/_engine-gaps.md

@@ -53,13 +53,16 @@ they are the real "does a new customer benefit?" items. The genuine engine ones
 - **Proposed:** a generic reader in `ontology.pack_schema` (or similar) over an engine-namespaced criterion
   vocabulary.
 
-### G8: The default record writer is not fully idempotent
-- **Where:** with no `writer=`, `build_ingestion` writes with the store's `kg_write`: nodes are upserted by their key,
-  but edges are created (`CREATE EDGE`), so re-ingesting a document duplicates its record edges. The store's
-  idempotent `kg_ensure_edges` (used for `EmbeddedIn` / `AttachedTo`) is not exported from `rag_wright.api`.
-- **Impact:** a domain whose extractor returns edges must pass its own idempotent `writer`, or clear the document's
-  records before a re-ingest.
-- **Proposed:** make the default writer use `kg_ensure_edges`, or export it.
+### G8: The default record writer is not fully idempotent (closed by PS-63)
+- **Where:** with no `writer=`, `build_ingestion` wrote with the store's `kg_write`: nodes were upserted by their key,
+  but edges were created (`CREATE EDGE`).
+- **Impact:** not only a re-ingest. Within ONE ingest of one document, an edge that several units emit (an entity
+  several rows mention, emitted by its key from each unit) landed once per unit, so a traversal over-counted while
+  node counts looked right (TexWright, 2026-10-15).
+- **Closed by PS-63 (ADR-0142):** the default writer calls `kg_write(..., ensure_edges=True)`: at most one edge per
+  (type, from node, to node), within the write, across writes and across concurrent writers, the last props landing,
+  in the same one transaction. A domain `writer=` gets the same with `rag_wright.api.kg_write(ws, nodes, edges,
+  ensure_edges=True)`; without the flag `kg_write` still keeps parallel edges.
 
 ### G9: One domain pack per workspace through the public API
 - **Where:** `EngineConfig.pack` takes one `.ttl`. A second pack's schema needs the store-internal

{rag_wright-0.17.0 → rag_wright-0.17.2}/docs/domain-adaptation/kg-construction.md

@@ -221,10 +221,11 @@ ask-first change.
   citation**.
 - **What is cached on a re-ingest:** in `build_ingestion`, parse and chunk are content-hash cached; spans,
   record nodes and the `Document` node are upserted by key; the `EmbeddedIn` / `AttachedTo` edges are ensured (never
-  duplicated). Extraction runs again on every ingest (cache inside your extractor if it is expensive), and the
-  default writer creates record edges rather than ensuring them, so re-ingesting a document whose extractor returns
-  edges duplicates them unless your `writer` is idempotent (engine gap G8). A boundary decider is not cached either
-  (engine gap G6).
+  duplicated). Extraction runs again on every ingest (cache inside your extractor if it is expensive). The
+  default writer ensures record edges too (PS-63): one edge per (type, from node, to node), whether several units
+  emit it or a re-ingest writes it again, with the last props landing; a domain `writer=` gets the same from
+  `kg_write(ws, nodes, edges, ensure_edges=True)`. A re-ingest does not remove records a document no longer yields.
+  A boundary decider is not cached either (engine gap G6).
 
 ## Building your own
 

{rag_wright-0.17.0 → rag_wright-0.17.2}/docs/domain-adaptation/ontology-authoring.md

@@ -46,6 +46,9 @@ ex:LinkedTo a eng:EntityRelationshipType ; rdfs:label "Linked To" .
 
 - **Declare every type your extractor writes.** The store does not create types on write, so a `kg_write` to an
   undeclared vertex or edge type fails.
+- **A measurement is a `DOUBLE`.** A Python float is written as a 64-bit double, so it reads back, filters
+  (equality, membership, `Range`) and compares in `kg_update` exactly; an undeclared float property is stored as a
+  double too. Declare `FLOAT` only for a value you mean to keep at 32 bits.
 - **Record vertices carry provenance.** Declare `span_id:STRING` and `confidence:STRING` on each record type: the
   engine's `check_extraction` requires every `span_id` to be a span of the unit, a non-empty extraction to cite at
   least once, and any `confidence` to be a `ConfidenceTag` (`EXTRACTED`, `INFERRED` or `AMBIGUOUS`)

{rag_wright-0.17.0 → rag_wright-0.17.2}/docs/specs/public-surface/plan.md

@@ -134,4 +134,7 @@ in two at the PS-6 gate so the store rewrite is its own visible task. One task,
 | PS-60 | TexWright (2026-11-05, `2026-11-05-export-the-default-segmenter-on-pack-sdk.md`): a table over the chunk cap was hard-split at a fixed length, mid-row; the rows after the cut lost their table layout (one span between them, no `TableRow`), and one item `chunk_layouts` could not place blocked every later item's layout. 4 of 140 records lost silently | the cut falls between lines and repeats the table's header in each continuation chunk; an unplaceable item no longer blocks later ones; a manifest cached with the old cut is re-chunked once; live: a 140-record x 76-column workbook yields 140 records | done 2026-11-05 (ADR-0141): live, the workbook (280 rows, two per record) gave 280/280 row records, 75/280 with the old cut |
 | PS-61 | TexWright's question in the same request: the default segmenter `segment_layout` is on neither public tier, so a pack cannot wrap it | export it on `rag_wright.pack_sdk` | done 2026-11-05: `pack_sdk.segment_layout` (the engine's own object); the wrap-the-default pattern in `kg-construction.md` and the ingestion skill; a pack segmenter delegating to it runs through `build_ingestion` |
 | PS-62 | TexWright, same request: the loss was silent (every `check_*` passed) | a record table whose rows do not each reach a unit with its `TableRow` is reported in the `IngestionReport` | done 2026-11-05: `DocumentReport.unreached_table_rows` (`table_ref`, `sheet`, `reached`, `total`, `missing`) and an `[ingest] warning:` line; a block table is not a loss; live, the pre-PS-60 cut on the workbook reports 75 of 280 rows reached (205 missing, matching the records lost), the new cut reports nothing |
+| PS-63 | TexWright, `2026-10-15-g8-duplicates-edges-within-one-ingest.md` (gap G8): the default writer created every edge, so an edge several units emit (an entity several rows mention) landed once per unit, within one ingest and on every re-ingest | the default writer leaves one edge per (type, from node, to node); `api.kg_write` exposes the same; `kg_write` without the flag unchanged | done 2026-11-07: `kg_write(..., ensure_edges=)` (ADR-0142), the default writer passes it, `api.kg_write` exposes it; G8 closed, the ingestion skill and the KG-construction guide updated; live, TexWright's shape (140 children on two rows each, 20 parents) leaves 140 edges on an ingest and a re-ingest (280 then 560 before) |
+| PS-64 | TexWright, `2026-11-09-kg-update-rewrites-unchanged-doubles.md`: ArcadeDB reads a bare decimal literal as a 32-bit FLOAT, so `kg_update` rewrote an unchanged DOUBLE, filters on a DOUBLE found nothing and an undeclared float was stored at 32 bits | a float written and compared as a double unless declared FLOAT; filters encode by the declared type; an undeclared field still matches old 32-bit values | done 2026-11-11: `_kg_double`, the filter grammar takes declared types (ADR-0143); live, a 50-row derive step changes 50 then 0 rows (50 then 50 before), equality / membership / `Range` find 50 of 50 (0 before) |
+| PS-65 | TexWright, `2026-11-09-the-embedder-model-is-rebuilt-for-every-workspace.md`: every workspace, ingestion pipeline and `intra_document_qa` call loaded its own BGE model (not a design choice; PS-52's lock is per handle) | one loaded model per (model, precision, device) per process, every wrapper of it on its one lock | done 2026-11-11: `util.devices.shared_model` (ADR-0144) behind `BGEM3Embedder` and `BGEReranker`; `docs/workspaces.md` item 7; live, three workspaces + a pipeline + two rerankers peak at 1534 MB (2816 before), each repeat 0 s (1 to 1.5 s), concurrent encodes through two wrappers equal the serial vectors |
 | PS-48 | Port the agent harness's substrate behaviour tests from RuleWright onto the engine's fabric fixture (ADR-0139 handover): of RuleWright's 553 harness-touching tests, 314 test the substrate (port), 213 RuleWright's composition (stay), 25 live (1 substrate: OpenRouter cost accounting). Seven gated sub-tasks: **a** shared `tests/harness/fabric.py` + scripting/slow/hanging/failing fake models + `InMemorySaver` helpers, then events, skills, clarify (~23); **b** models and errors (~42); **c** runs, confirm, continuation (~32); **d** decisions and rules (~51); **e** grading (~32); **f** seen and memory (~41); **g** workers, batch, dispatch, tiers, agent wiring, filesystem, interpreter, withhold + the live accounting test (~93, may split). Ambiguous tests split, not moved whole; a ported test failing on engine code is a substrate defect fixed in that sub-task; deadline globals via monkeypatch; no new dependencies. Follow-up (owner RuleWright): retire its substrate duplicates once ported | each sub-task's ported tests green on the fabric fixture | todo (proposed; awaiting go-ahead) |

{rag_wright-0.17.0 → rag_wright-0.17.2}/docs/workspaces.md

@@ -67,6 +67,11 @@ torch model takes its device from `pack_sdk.torch_device`, which never picks Met
      but the metadata is silently dropped (an unknown field is ignored), so documents are ingested without it.
 6. **One process, many workspaces, one set of process settings.** Where tenants need a different serving backend,
    decision model, timeouts or classifier weights, run a process per such set.
+7. **One loaded model per process, not per tenant.** The in-process BGE-M3 embedder and BGE reranker are loaded once
+   per (model, precision, device) and shared by every workspace and ingestion pipeline in the process (about 1 GB for
+   the embedder and 0.3 GB for the reranker, whatever the number of tenants). Their calls queue on that model's one
+   lock, so size throughput, not memory, per tenant; a tenant that needs its own model runs in its own process. A
+   loaded model stays until the process exits.
 
 ## Changing a workspace's configuration
 

{rag_wright-0.17.0 → rag_wright-0.17.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "rag-wright"
-version = "0.17.0"
+version = "0.17.2"
 description = "A domain-retargetable, open-core engine for hybrid retrieval + knowledge-graph RAG: parsing, chunking, embedding, hybrid search, reranking, graph extraction, entity resolution, ontology-driven knowledge, and a capability runtime."
 readme = "README.md"
 requires-python = ">=3.12,<3.13"

{rag_wright-0.17.0 → rag_wright-0.17.2}/src/rag_wright/.agents/skills/building-an-ingestion-capability/SKILL.md

@@ -199,7 +199,11 @@ category.
 - **One record over several rows, or one entity over many units.** The extractor sees one unit at a time. A record
   whose continuation rows sit under its main row needs a `unit_grouper=` that keeps them together, or a
   `document_hook=` that reads the whole table with `table_rows` and writes the grouped record. An entity several units
-  mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key.
+  mention (a case its trial rows share) is emitted by its key from each unit and upserted by that key; an edge to it
+  emitted from each unit lands once, because the default writer keeps one edge per (type, from node, to node)
+  (PS-63). A `writer=` of your own gets the same with `kg_write(ws, nodes, edges, ensure_edges=True)`; leave the flag
+  off only when parallel edges of one type between the same two nodes are meant to stay (each with its own
+  provenance).
 - **Resume a long ingest instead of restarting it.** Pass a LangGraph checkpointer (yours, for example a Postgres
   saver; the engine ships none) and a `run_id` naming the run:
   `await pipeline.aingest(ws, sources, cache_dir=CACHE, checkpointer=saver, run_id=batch_id)`. If the process stops,
@@ -208,8 +212,9 @@ category.
   sources, raise `ValueError`: a new run gets a new id. The graph state is built-in values only, so it works under
   LangGraph's strict msgpack mode; a tuple inside a record's free-form `props` comes back as a list after a resume.
 - **Re-ingest.** Parsing is cached by content hash, so re-running is cheap. The default writer upserts nodes by their
-  key but is not fully idempotent for edges (engine gap G8): when a source is re-run, remove its previous records
-  first (`kg_delete` on the provenance your records carry, such as a per-source property) and write again.
+  key and writes each edge once, updating its props, so a re-run leaves no duplicates. It does not remove what a
+  source no longer yields: when a re-run should drop records, remove its previous records first (`kg_delete` on the
+  provenance your records carry, such as a per-source property) and write again.
 
 ## 6. Verify (definition of done)
 

{rag_wright-0.17.0 → rag_wright-0.17.2}/src/rag_wright/api/kg.py

@@ -62,11 +62,12 @@ def kg_update(ws: WorkspaceHandle, type_name: str, *, set: dict, where: Optional
     return ws._store.kg_update(type_name, set=set, where=where, key_range=key_range)
 
 
-def kg_write(ws: WorkspaceHandle, nodes: list, edges: Any = ()) -> None:
+def kg_write(ws: WorkspaceHandle, nodes: list, edges: Any = (), *, ensure_edges: bool = False) -> None:
     """Upsert typed `nodes` + create typed `edges` in one transaction (see `Store.kg_write`). `nodes`/`edges` are
     `KgNode`/`KgEdge` (from `rag_wright.api`); the store encodes each field per its pack-declared type. A node type
-    the workspace's schema does not declare fails the write."""
-    ws._store.kg_write(nodes, edges)
+    the workspace's schema does not declare fails the write. `ensure_edges=True` (PS-63) writes at most one edge per
+    (type, from node, to node), the last props winning: what a domain `writer=` passes to match the default writer."""
+    ws._store.kg_write(nodes, edges, ensure_edges=ensure_edges)
 
 
 def kg_edges(ws: WorkspaceHandle, from_type: Optional[str] = None, *, where: Optional[dict] = None,

{rag_wright-0.17.0 → rag_wright-0.17.2}/src/rag_wright/capabilities/embedding.py

@@ -80,25 +80,32 @@ class BGEM3Embedder:
     driving one shared model swap the same parameter-tensor storage under each other and SEGFAULT the interpreter
     (not a catchable exception). The engine SHARES one `Embedder` across concurrent-document ingestion
     (`run_job(max_concurrency=...)` -> per-document `encode_batch`) and across `embed_chunks`' per-chunk thread
-    fan-out. So every model call is serialized by an instance lock. The lock guards ONLY `self._model.encode`
+    fan-out. So every model call is serialized by the model's lock. The lock guards ONLY `self._model.encode`
     (the numpy results are per-call, thread-local); the serialized section is the CPU/GPU encode, which is a small
     slice of ingest time -- the concurrency worth overlapping is the network-bound extraction, not the encode.
-    On Metal the lock is the process-wide `METAL_LOCK` instead (PS-55): Metal is not safe for concurrent use, and a
-    process can hold several embedders (the query embedder, each workspace's) plus the reranker."""
+    On Metal the lock is the process-wide `METAL_LOCK` instead (PS-55): Metal is not safe for concurrent use.
+
+    ONE MODEL PER PROCESS (PS-65): the loaded model is shared by every embedder of the same (model, precision, device)
+    -- the query embedder, each workspace's, each ingestion pipeline's -- with its one lock (`shared_model`), so a
+    second workspace costs no load time and no memory, and their encodes queue on that lock. An injected `model=`
+    is never shared."""
 
     def __init__(self, model_name: str = "BAAI/bge-m3", *, use_fp16: bool = False,
                  device: str | None = None, batch_size: int = 64, model: object | None = None) -> None:
         self._batch_size = batch_size  # cross-item independent, so batching never changes a vector, only speed
-        from rag_wright.util.devices import model_lock
+        from rag_wright.util.devices import model_lock, shared_model
 
         resolved = _resolve_device(device)
-        self._lock = model_lock(resolved)  # issue 0016 per instance; on Metal one lock per process (PS-55)
-        if model is not None:  # injected (hermetic tests) -- avoids loading the real model
-            self._model = model
+        if model is not None:  # injected (hermetic tests) -- avoids loading the real model; its own lock
+            self._model, self._lock = model, model_lock(resolved)
         else:
-            from FlagEmbedding import BGEM3FlagModel
+            def build() -> object:
+                from FlagEmbedding import BGEM3FlagModel
+
+                return BGEM3FlagModel(model_name, use_fp16=use_fp16, devices=resolved)
 
-            self._model = BGEM3FlagModel(model_name, use_fp16=use_fp16, devices=resolved)
+            # PS-65: one model per (model, precision, device) in the process, with its one lock (issue 0016, PS-55)
+            self._model, self._lock = shared_model(("bge-m3", model_name, use_fp16, resolved), resolved, build)
 
     def encode_dense(self, text: str) -> list[float]:
         with self._lock:  # issue 0016

{rag_wright-0.17.0 → rag_wright-0.17.2}/src/rag_wright/capabilities/reranking.py

@@ -70,25 +70,30 @@ class BGEReranker:
     THREAD-SAFE (engine issue 0016): a shared reranker is guarded by an instance lock -- defense-in-depth for any
     concurrent `score` caller (its FlagEmbedding predecessor mutated the model in place on every call). It is in the
     BGE family: its device follows the embedder's (`EMBED_DEVICE`), and on Metal its lock is the process-wide one
-    the embedders hold (PS-55), so a rerank never runs on Metal beside an embed."""
+    the embedders hold (PS-55), so a rerank never runs on Metal beside an embed. Like the embedder, the loaded model is
+    shared by every reranker of the same (model, precision, device) in the process, with its one lock (PS-65)."""
 
     def __init__(self, model_name: str = DEFAULT_RERANKER_MODEL, *, use_fp16: bool = False,
                  device: str | None = None, model: object | None = None) -> None:
         from rag_wright.capabilities.embedding import _resolve_device
-        from rag_wright.util.devices import model_lock
+        from rag_wright.util.devices import model_lock, shared_model
 
         resolved = _resolve_device(device)
-        self._lock = model_lock(resolved)  # issue 0016 per instance; on Metal one lock per process (PS-55)
-        if model is not None:  # injected (hermetic tests)
-            self._model = model
+        if model is not None:  # injected (hermetic tests); its own lock
+            self._model, self._lock = model, model_lock(resolved)
         else:
-            import torch
-            from sentence_transformers import CrossEncoder
-
-            self._model = CrossEncoder(model_name, device=resolved, max_length=512,
-                                       activation_fn=torch.nn.Identity())  # raw logits, as FlagEmbedding gave
-            if use_fp16 and resolved != "cpu":
-                self._model.model.half()
+            def build() -> object:
+                import torch
+                from sentence_transformers import CrossEncoder
+
+                loaded = CrossEncoder(model_name, device=resolved, max_length=512,
+                                      activation_fn=torch.nn.Identity())  # raw logits, as FlagEmbedding gave
+                if use_fp16 and resolved != "cpu":
+                    loaded.model.half()
+                return loaded
+
+            # PS-65: one model per (model, precision, device) in the process, with its one lock (issue 0016, PS-55)
+            self._model, self._lock = shared_model(("bge-reranker", model_name, use_fp16, resolved), resolved, build)
 
     def score(self, query: str, passages: list[str]) -> list[float]:
         if not passages:

{rag_wright-0.17.0 → rag_wright-0.17.2}/src/rag_wright/ingestion/builder.py

@@ -259,9 +259,9 @@ class IngestionStages:
         return stage
 
     async def write(self, sd: Any, extractions: list) -> int:
-        """Persist the records (the domain's writer, else `kg_write`). Returns the number that landed: the writer's own
-        count when it returns one; else the record nodes handed over, which the default `kg_write` lands in one
-        transaction (it raises rather than land part of them)."""
+        """Persist the records (the domain's writer, else `kg_write` with idempotent edges). Returns the number that
+        landed: the writer's own count when it returns one; else the record nodes handed over, which the default
+        `kg_write` lands in one transaction (it raises rather than land part of them)."""
         if self.writer is not None:
             landed = await self.writer(sd.source_doc_id, extractions)
             if landed is not None:
@@ -269,7 +269,8 @@ class IngestionStages:
         else:
             nodes = [n for e in extractions for n in e.nodes]
             edges = [ed for e in extractions for ed in e.edges]
-            await asyncio.to_thread(self.store.kg_write, nodes, edges)
+            # PS-63 (G8): an edge several units emit (an entity several rows mention), or a re-ingest, lands once
+            await asyncio.to_thread(self.store.kg_write, nodes, edges, ensure_edges=True)
         return sum(len(e.nodes) for e in extractions)
 
     def write_document_node(self, doc_id: str, *, parent_id: Optional[str], filename: str, media_type: str,