@pentatonic-ai/ai-agent-sdk 0.10.19 → 0.10.20

package/dist/index.cjs

@@ -878,7 +878,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
 }
 
 // src/telemetry.js
-var VERSION = "0.10.19";
+var VERSION = "0.10.20";
 var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
 function machineId() {
   const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";

package/dist/index.js

@@ -847,7 +847,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
 }
 
 // src/telemetry.js
-var VERSION = "0.10.19";
+var VERSION = "0.10.20";
 var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
 function machineId() {
   const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pentatonic-ai/ai-agent-sdk",
-  "version": "0.10.19",
+  "version": "0.10.20",
   "description": "TES SDK — LLM observability and lifecycle tracking via Pentatonic Thing Event System. Track token usage, tool calls, and conversations. Manage things through event-sourced lifecycle stages with AI enrichment and vector search.",
   "type": "module",
   "main": "./dist/index.cjs",

package/packages/memory-engine-v2/RFC-decay-and-fusion.md

@@ -1,10 +1,18 @@
 # RFC: the Fusion Drive — v2 memory self-healing (cross-run node fusion + decay)
 
 > **Fusion Drive** = the continuous, arena-scoped background engine that keeps the v2
-> memory graph self-healing: it *fuses* duplicate/near-duplicate nodes from different
-> distillation runs into a single master node (horizontal convergence) and *decays* stale,
-> low-value, and junk nodes out of existence (vertical aging). Named for the drive that
-> does the fusing — the decay pass rides the same engine.
+> memory graph self-healing. It triages every node into one of **three** outcomes:
+> it *fuses* duplicate/near-duplicate nodes from different distillation runs into a single
+> master node (horizontal convergence); it *re-distills* high-value extractions produced by
+> a superseded teacher/prompt — regenerating them from the still-present source event through
+> the current clean teacher (depth refresh); and it *decays* stale, low-value, and junk nodes
+> out of existence (vertical aging). Named for the drive that does the fusing — the re-distill
+> and decay passes ride the same engine.
+>
+> *(Revised 2026-06-22: added Part B′ — re-distillation — as the third triage verb, with the
+> prompt-version-drift trigger. Motivated by the clean-prompt deploy (SDK 0.10.19, #126 +
+> #129) which made "the current teacher is materially better than the one that produced most
+> of the graph" concrete and measurable via `system_prompt_hash`.)*
 
 **Status:** draft / spec — 2026-06-12
 **Builds on:** `RFC-entity-reconciliation.md`, `scripts/entity_resolution_v2.py` (#82),
@@ -139,15 +147,101 @@ sparse backfill.
 
 ---
 
+## Part B′ — Re-distillation: regenerate stale-prompt extractions from source
+
+Fusion (A) needs a *correct counterpart* to converge toward; Decay (B) just *deletes*. But
+the common case after a teacher/prompt upgrade is a **high-value node with no correct
+counterpart yet** — the only extraction that exists is the stale-prompt one. Fusion has
+nothing to fuse to; decay would throw away real information. The cure is the third verb: the
+**source event still exists** (`events` table, 376k rows live), so regenerate the extraction
+by re-running that event through the *current clean teacher*. Fusion converges horizontally,
+decay ages vertically; re-distill refreshes **in depth**.
+
+### B′1. Trigger — prompt-version drift, not raw age
+The defect population is *exactly* the facts/entities whose provenance traces an **old
+`system_prompt_hash`** — `bbdaba6b…` / `f1e0ff55…` / `ef0647c7…` (pre-clean), vs the clean
+`6ccfe70f…` deployed with 0.10.19 (#126 modality/attribution + #129 email-discipline &
+entity-separation). #118 propagated source onto facts, so provenance → the event's
+`distillation_traces.system_prompt_hash` is queryable. **Age is a weak proxy; prompt-version
+selects the defect set directly** — a months-old node the clean teacher would extract
+identically needs nothing; a two-day-old node from the dirty prompt is a defect. Prioritize
+by `salience` (B1) so high-value stale nodes go first.
+
+### B′2. Triage routing — 3-way, by salience × prompt-version
+Per assessed node/event:
+
+| condition | outcome |
+|---|---|
+| stale prompt-hash **+** high salience **+** source event present | **re-distill** (this part) |
+| has a correct newer-teacher counterpart in the arena | **fuse** (Part A) |
+| low salience, junk-born (B2), no corroboration, never accessed | **decay** (Part B) |
+
+### B′3. Mechanism — re-enqueue, don't mutate in place
+Re-distill = re-insert the source `event_id` into `distillation_queue` (`status='pending'`,
+`attempts=0`). The existing **extractor-async** worker claims it, runs the clean teacher,
+writes the new extraction **and a fresh `6ccfe70f` trace**. No new pipeline — it reuses the
+distiller, the combined-demand **autoscaler**, and the trace ledger. (Re-distill is a
+*producer* of queue demand; the autoscaler's student-aware floor already keeps a teacher box
+warm for it — see the deploy notes.)
+
+### B′4. Supersedence — the load-bearing requirement
+The store is **pure-accretion** (the whole motivation of this RFC). A naive re-enqueue makes
+the clean extraction land **beside** the dirty one → it *worsens* fragmentation. So
+re-distill MUST close the loop through Fusion's tombstone machinery — it is **sequenced into
+the Fusion Drive, not bolted on**:
+
+1. Each re-distill is recorded in a `redistill_runs` ledger with its triggering
+   `(event_id, old_prompt_hash)`.
+2. When the clean extraction completes, **Fusion converges old ↔ new for that event** using
+   the teacher-version master signal (A2/A3): the new `6ccfe70f` extraction wins as master;
+   the old extraction's now-orphaned nodes (those whose **only** provenance was this event
+   under the old hash) are tombstoned/repointed via `entity_merges` / `fact_merges`.
+3. Where an old node carries **other live provenance** (multi-event corroboration), only this
+   event's contribution is repointed — **never blind-delete a multi-source node** (the
+   over-merge failure mode: a hotel email wrongly attached to a person must not let one
+   event's repoint nuke an otherwise-corroborated node).
+
+This dependency is hard: **re-distill is unsafe until Fusion's cross-run / teacher-version
+master selection (E3) is live.** Until then a re-distill loop accretes. An interim cheaper
+option (Open Q): explicit **event-scoped supersede** — delete only the facts/entities whose
+provenance set is exactly `{this event}` under the old hash before re-enqueue — covers the
+single-provenance majority without the full fusion adjudicator.
+
+### B′5. Corpus-as-byproduct — one loop, three wins
+Every re-distill emits a clean `6ccfe70f` `distillation_trace`. A prompt-version-drift
+re-distill loop therefore **builds the student retrain corpus while it repairs the graph**
+(`scripts/build_retrain_corpus.py` consumes those traces). It subsumes the one-shot full
+re-distill: gradual, rate-limited, no nuke — graph repair **+** corpus **+** self-healing
+from a single engine. This is the durable answer to "is the corpus building?": it is, as a
+side effect of the gardener.
+
+### B′6. Cadence + cost + safety
+Rolling, rate-limited, autoscaler-aware, off-peak. Budget *N* events/hour against teacher
+capacity; order by `salience × staleness`. **Never big-bang the full backlog** — gradual
+migration is the point. Arena-scoped, dry-run → `--apply`, `redistill_runs` ledger for
+observability and rollback. Same operational shape as fusion/decay/autoscaler.
+
+---
+
 ## Part C — Ordering & how they combine
 
-Per arena, on schedule: **(1) fusion → (2) decay.** Fusion first so a master node absorbs
-its duplicates' provenance/salience *before* decay judges it (else a real node split across
-two weak dupes could wrongly decay out). Then decay ages + evicts the survivors.
+Per arena, on schedule: **(1) triage → re-distill the high-value stale-prompt set (async via
+the queue) → (2) fusion → (3) decay.** Re-distill is enqueued first so that by the time
+fusion runs, the clean counterpart exists for it to crown as master (else fusion has only
+stale renderings to choose between). Fusion then absorbs each master's duplicates'
+provenance/salience *before* decay judges it (else a real node split across two weak dupes
+could wrongly decay out). Then decay ages + evicts the survivors.
+
+*(Re-distill is asynchronous — it completes on the teacher's schedule — so in practice a
+node re-distilled in this pass is fused/decayed in the **next** per-arena pass, once its
+clean trace + extraction have landed. The ledger links the two.)*
 
 **This is what finally cures immortal pollution:**
 - 7B polluted node *with* a correct Qwen3.6 counterpart → **fused**, correct one as master,
   polluted demoted to alias / tombstoned.
+- stale-prompt node, *high-value*, *no* correct counterpart, source event present →
+  **re-distilled** through the clean teacher → new master extraction; old superseded via
+  fusion (B′4). The information is *recovered*, not lost.
 - 7B pure-junk node with *no* correct counterpart (numeric-ID-person, ungrounded) → born-low
   salience + no corroboration + never accessed → **decays out and is evicted**.
 
@@ -165,8 +259,15 @@ reset, but no longer the *only* path).
 - `relationships`: `+ salience REAL`, `+ last_accessed` (already has `weight`,
   `first/last_seen`).
 - new `fact_merges` audit (mirror `entity_merges` incl. `rollback_payload`).
-- new `fusion_runs` + `decay_runs` ledgers for observability.
+- new `fusion_runs` + `decay_runs` + `redistill_runs` ledgers for observability. `redistill_runs`:
+  `(id, arena, event_id, old_prompt_hash, new_prompt_hash, salience_at_trigger, enqueued_at,
+  completed_at, fused_at, mode)` — links a re-distill to its triggering node and to the fusion
+  that superseded the old extraction.
 - `/search` gains a `last_accessed = NOW()` bump on returned nodes (batched).
+- re-distill trigger needs provenance → prompt-version: either denormalize `system_prompt_hash`
+  onto `facts`/`entities` at write time (cheap filter), or join through
+  `distillation_traces(event_id → system_prompt_hash)` on the provenance event ids (no schema
+  change, costlier query). Prefer the join until the trigger volume justifies denormalizing.
 
 ## Part E — Rollout (each flag-gated, arena-scoped, dry-run-first, audited)
 
@@ -176,6 +277,13 @@ reset, but no longer the *only* path).
 3. **Fusion extension** — scored canonical selection (fix typo-crowning) + cross-run
    detection + fact fusion, dry-run → apply.
 4. **Online/continuous** — wire fusion+decay to run after distillation per arena.
+5. **Re-distill loop (Part B′)** — dry-run triage first (count stale-prompt nodes by
+   `system_prompt_hash` × salience bucket to size the work), then a **bounded `--apply` slice**
+   on one curated arena (re-enqueue + verify clean trace + verify fusion supersedes the old
+   extraction), then wire continuous. **Gated on step 3** (Fusion cross-run / teacher-version
+   master selection): until that's live, re-distill must use the interim **event-scoped
+   supersede** (B′4) or it accretes. Ships as `scripts/redistill.py` (dry-run default,
+   `--apply` gate, arena-scoped, `redistill_runs` ledger).
 
 ## Open questions
 - Half-life constants per category — needs a calibration pass against real arenas.
@@ -183,3 +291,9 @@ reset, but no longer the *only* path).
 - Directory authority source for canonical anchoring — HubSpot contacts? a curated table?
 - Interaction with the (still-open) source_id supersede mode — fusion partly subsumes it,
   but explicit supersede is cheaper for known-mutable sources.
+- **Re-distill supersedence before full fusion is live** — is event-scoped supersede (delete
+  only nodes whose provenance set is exactly `{this event}` under the old hash) a safe enough
+  interim, or do we hard-gate the loop on E3? Single-provenance nodes are the majority, but
+  the multi-provenance tail is where the over-merge risk concentrates.
+- **Re-distill prioritization** — pure `salience × staleness`, or weight toward the entities
+  behind known user-visible confabulations (Vickers/Boedecker) first?

package/packages/memory-engine-v2/compat/server.py

@@ -896,6 +896,8 @@ class GraphQueryRequest(BaseModel):
     entity_type: str | None = None
     name: str | None = None             # canonical_name (ILIKE)
     subject: str | None = None          # entity name OR canonical_name (facts.subject_entity)
+    subject_entity_id: str | None = None  # EXACT facts.subject_entity_id — strict, no name bleed
+    object_entity_id: str | None = None   # EXACT facts.object_entity_id
     predicate: str | None = None
     category: str | None = None         # facts.category
     from_name: str | None = None        # relationships.from_entity.canonical_name
@@ -944,9 +946,14 @@ async def list_entities(req: GraphQueryRequest):
 
 @app.post("/facts")
 async def list_facts(req: GraphQueryRequest):
-    """Filter facts by arena + optional category/predicate + optional
-    subject-entity name. Subject filter joins facts → entities via
-    subject_entity_id."""
+    """Filter facts by arena + optional category/predicate + subject.
+
+    PREFER `subject_entity_id` (exact id match) over `subject` (name ILIKE):
+    name matching bleeds one person's facts into another's answer when names
+    collide or fragment (the Will Vickers ⟵ Will Spencer confabulation — a
+    query resolved to one entity must NOT pull a same/similar-named entity's
+    facts). The name path is kept for back-compat callers that haven't resolved
+    an id yet, but entity-id is the strict, bleed-free path."""
     arenas = _resolve_arenas(req)
     conditions = ["f.arena = ANY(%s)"]
     params: list[Any] = [arenas]
@@ -956,7 +963,14 @@ async def list_facts(req: GraphQueryRequest):
     if req.predicate:
         conditions.append("f.predicate ILIKE %s")
         params.append(f"%{req.predicate}%")
-    if req.subject:
+    if req.subject_entity_id:
+        conditions.append("f.subject_entity_id = %s")
+        params.append(req.subject_entity_id)
+    if req.object_entity_id:
+        conditions.append("f.object_entity_id = %s")
+        params.append(req.object_entity_id)
+    # Name path: only when no exact id was given (back-compat / unresolved callers).
+    if req.subject and not req.subject_entity_id:
         conditions.append("EXISTS (SELECT 1 FROM entities e WHERE e.id = f.subject_entity_id AND (e.canonical_name ILIKE %s OR %s = ANY(e.aliases)))")
         params.extend([f"%{req.subject}%", req.subject])
     sql = f"""

package/packages/memory-engine-v2/extractor-async/test_email_alias_guard.py

@@ -0,0 +1,78 @@
+"""Unit tests for the email-alias guard (_email_plausibly_belongs).
+
+Pins the live pollution case (the "Johann Boedecker" node, 2026-06-22): keep the
+person's own addresses; drop the bystander emails (a hotel, newsletters, unrelated
+gmails) the LLM stapled on from co-occurring documents.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+
+import pytest
+
+_THIS = Path(__file__).resolve().parent
+
+
+def _load(name="extractor_async_worker_aliasguard"):
+    spec = importlib.util.spec_from_file_location(name, _THIS / "worker.py")
+    mod = importlib.util.module_from_spec(spec)
+    sys.modules[name] = mod
+    spec.loader.exec_module(mod)
+    return mod
+
+
+try:
+    worker = _load()
+except ImportError as e:
+    pytest.skip(f"extractor-async deps unavailable: {e}", allow_module_level=True)
+
+belongs = lambda n, e: worker._email_plausibly_belongs(n, e)
+
+
+# ── KEEP: the person's own addresses ─────────────────────────────────────
+@pytest.mark.parametrize("email", [
+    "johann@pentatonic.com",
+    "johann.boedecker@pentatonic.com",
+    "boedeckerjohann@gmail.com",
+    "JOHANN@pentatonic.com",          # case-insensitive
+    "jb@pentatonic.com",              # initials
+    "j.boedecker@pentatonic.com",     # surname token
+])
+def test_keeps_owner_emails(email):
+    assert belongs("Johann Boedecker", email) is True
+
+
+# ── DROP: the actual bystander emails found on the live Johann node ──────
+@pytest.mark.parametrize("email", [
+    "reservations.nyc@acehotel.com",
+    "marketingadmin@sustainablebrands.com",
+    "martinvasquez87@gmail.com",
+    "schwaabd@yahoo.de",
+    "cvanderlip@redish.com",
+    "leechihshan33@gmail.com",
+])
+def test_drops_bystander_emails(email):
+    assert belongs("Johann Boedecker", email) is False
+
+
+# ── edges ────────────────────────────────────────────────────────────────
+def test_initials_either_order():
+    assert belongs("Johann Boedecker", "bj@pentatonic.com") is True   # reversed initials
+
+
+def test_no_usable_name_does_not_overfilter():
+    # a bare/empty name has nothing to check against → keep (don't strip)
+    assert belongs("", "anything@x.com") is True
+    assert belongs("J", "anything@x.com") is True  # single letter < 2 → no tokens
+
+
+def test_surname_only_person_keeps_surname_email():
+    assert belongs("Vickers", "will.vickers@vickers-oil.com") is True
+    assert belongs("Vickers", "reservations.nyc@acehotel.com") is False
+
+
+def test_guard_flag_default_on():
+    assert worker.EMAIL_ALIAS_GUARD is True

package/packages/memory-engine-v2/extractor-async/worker.py

@@ -1253,6 +1253,43 @@ def org_node_id_key(entity_type: str, name: str, stamped_domain: str | None) ->
     return name
 
 
+# --------------------------------------------------------------------
+# Email-alias guard — stop bystander emails polluting a person
+# --------------------------------------------------------------------
+# The async LLM pass sometimes emits a PERSON entity whose `email` is a BYSTANDER
+# address co-occurring in the same doc/thread (a hotel booking, a newsletter, an
+# unrelated gmail). _parse_guided_json promotes it into the entity's aliases and
+# upsert_entities then stores + RESOLVES on it — folding strangers' identities
+# (and their facts) onto the person. Measured live (pentatonic-team): a "Johann
+# Boedecker" node carrying reservations.nyc@acehotel.com + unrelated gmails, all
+# from STUDENT-distilled `doc` events. This guard keeps an email alias on a person
+# only when its local-part plausibly relates to the person's name; clear bystanders
+# are dropped BEFORE resolution/storage. Conservative: dropping a genuine but
+# non-name-matching alias is a mild loss; keeping a bystander is a confabulation
+# source. Flag-revertible (EMAIL_ALIAS_GUARD=false). Persons only — org domain
+# stamping is untouched.
+EMAIL_ALIAS_GUARD = _envflag("EMAIL_ALIAS_GUARD", "true")
+_ALIAS_NONALPHA = re.compile(r"[^a-z]")
+_ALIAS_SPLIT = re.compile(r"[^a-z]+")
+
+
+def _email_plausibly_belongs(person_name: str, email: str) -> bool:
+    """True ⇒ keep this email as an alias of `person_name`; False ⇒ drop (clear
+    bystander). Match = a name token appears in the local-part, OR the local-part
+    is the person's initials. Pure + deterministic."""
+    local = email.split("@", 1)[0].lower()
+    local_letters = _ALIAS_NONALPHA.sub("", local)
+    name_tokens = {t for t in _ALIAS_SPLIT.split(person_name.lower()) if len(t) >= 2}
+    if not name_tokens or not local_letters:
+        return True  # nothing to check against — don't over-filter
+    if any(nt in local_letters for nt in name_tokens):
+        return True  # johann@…, johann.boedecker@…, boedeckerjohann@…
+    initials = "".join(t[0] for t in person_name.lower().split() if t[:1].isalpha())
+    if len(initials) >= 2 and local_letters in (initials, initials[::-1]):
+        return True  # jb@… / bj@… for "Johann Boedecker"
+    return False
+
+
 def upsert_entities(
     conn: psycopg.Connection,
     arena: str,
@@ -1345,6 +1382,21 @@ def upsert_entities(
                 continue
             aliases = [a for a in (e.get("aliases") or []) if a]
 
+            # Email-alias guard (persons only): drop bystander emails the LLM
+            # stapled on from a co-occurring doc/thread, BEFORE they reach
+            # resolution or storage. See _email_plausibly_belongs.
+            if EMAIL_ALIAS_GUARD and etype == "person" and aliases:
+                kept = []
+                for a in aliases:
+                    if "@" in a and " " not in a and not _email_plausibly_belongs(name, a):
+                        log.info(
+                            f"alias-guard: dropped bystander email {a!r} from "
+                            f"person {name!r} (arena={arena})"
+                        )
+                        continue
+                    kept.append(a)
+                aliases = kept
+
             # Hard-key stamps for THIS entity, merged onto the node's attributes
             # and (for domain) into the resolution aliases. Adding domain to
             # aliases before forms are computed is deliberate — that's what makes

package/packages/memory-engine-v2/scripts/build_retrain_corpus.py

@@ -0,0 +1,240 @@
+#!/usr/bin/env python3
+"""Build a student retrain corpus from CLEAN teacher gold.
+
+The student (NuExtract-2.0-4B FT) was originally trained on teacher traces
+produced under the *old* distiller prompts (bbdaba / f1e0ff), which had no
+email-discipline or modality rules — so the student learned to (a) promote
+bystander emails into a person's aliases (the Johann/hotel over-merge), (b)
+collapse future/invited roles to established `state` facts, (c) conflate
+"X & Y" into one entity, and (d) mint email-named / generic-infra entities.
+
+This builder draws ONLY from traces produced under the clean prompt (#126
+modality/attribution + #129 email-discipline & entity-separation), whose
+`system_prompt_hash` is the deployed clean hash. Building from old-prompt
+traces would just re-teach the defects, so that is explicitly NOT the default
+(you must pass --allow-dirty-hash to override, and you shouldn't).
+
+A defect filter runs as a SECOND line of defence: even clean-prompt output is
+screened for the known defect signatures and dropped if any survive. Every
+drop is counted by reason so the corpus's cleanliness is auditable (not a
+black box — see the printed report).
+
+INPUT — an NDJSON stream of trace rows, one object per line:
+    {"event_id": "...", "user_prompt": "...", "raw_response": "...",
+     "system_prompt_hash": "..."}
+Produce it from the engine box's org_model DB with row_to_json (the escaping
+that bit us before — \\copy double-escapes, $-quoting gets eaten by the shell —
+is avoided by -At + row_to_json):
+
+    sudo docker exec -i pme2-org-model psql -U pme -d org_model -At -c \\
+      "SELECT row_to_json(t) FROM (
+         SELECT event_id, user_prompt, raw_response, system_prompt_hash
+         FROM distillation_traces
+         WHERE system_prompt_hash = '6ccfe70f1286a131'
+       ) t" > traces.ndjson
+
+OUTPUT — {"input": <per-event block>, "output": <extraction JSON string>}
+JSONL(.gz), the exact shape train_lora.py's load() consumes (it keeps rows
+where both `input` and `output` are truthy, then trains user=input ->
+assistant=output via the NuExtract chat template; no system prompt in the
+pair). The corpus is PER-EVENT while a trace is a 3-event chunk, so each
+trace's user_prompt is split on the `[event K]` markers and matched to
+raw_response[index == K].
+
+Usage:
+    python build_retrain_corpus.py --traces traces.ndjson --out retrain_clean.jsonl.gz
+    zcat traces.ndjson.gz | python build_retrain_corpus.py --traces - --out c.jsonl.gz
+"""
+from __future__ import annotations
+
+import argparse
+import gzip
+import hashlib
+import json
+import re
+import sys
+from collections import Counter
+
+# The clean prompt deployed as SDK 0.10.19 (#126 + #129). Verify against the
+# running extractor-async (worker.SYSTEM_PROMPT_HASH) before a real corpus cut —
+# a prompt edit advances this and old-hash traces must not silently leak in.
+CLEAN_PROMPT_HASH = "6ccfe70f1286a131"
+
+# Generic infra / environment tokens that must never be standalone entities
+# (mirrors the #129 DISTINCT ENTITIES rule — kept in sync by hand).
+INFRA_TOKENS = {
+    "prod", "production", "staging", "stage", "uat", "qa", "dev", "test",
+    "warehouse", "datalake", "data lake", "cluster", "backend", "frontend",
+    "the system", "the platform", "the api", "the database", "the server",
+}
+
+EVENT_BLOCK_RE = re.compile(r"(?=^\[event \d+\])", re.MULTILINE)
+EVENT_IDX_RE = re.compile(r"^\[event (\d+)\]")
+EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$")
+
+
+def _email_plausibly_belongs(person_name: str, email: str) -> bool:
+    """Keep an email on a person only if it plausibly is theirs: a name token
+    appears in the local-part, or the initials match. Same heuristic as the
+    write-side guard (#128) so corpus filtering and runtime agree."""
+    local = email.split("@", 1)[0].lower()
+    local_alnum = re.sub(r"[^a-z0-9]", "", local)
+    tokens = [t for t in re.split(r"\s+", person_name.lower()) if t]
+    if not tokens:
+        return False
+    for t in tokens:
+        t_alnum = re.sub(r"[^a-z0-9]", "", t)
+        if len(t_alnum) >= 3 and t_alnum in local_alnum:
+            return True
+    initials = "".join(t[0] for t in tokens if t)
+    if len(initials) >= 2 and initials in local_alnum:
+        return True
+    return False
+
+
+def _entity_defect(ent: dict) -> str | None:
+    """Return a drop-reason if this entity carries a known defect, else None."""
+    name = (ent.get("name") or "").strip()
+    etype = (ent.get("type") or "").lower()
+    if not name:
+        return "empty_entity_name"
+    if EMAIL_RE.match(name):
+        return "email_as_entity"
+    if name.lower() in INFRA_TOKENS:
+        return "generic_infra_entity"
+    # Conflation: "Acme & Globex" / "Alice and Bob" smuggled into one node.
+    if re.search(r"\s&\s", name) or re.search(r"\b and \b", name.lower()):
+        return "conflated_entity"
+    if etype == "person":
+        for a in ent.get("aliases") or []:
+            if isinstance(a, str) and "@" in a and " " not in a \
+                    and not _email_plausibly_belongs(name, a):
+                return "bystander_email_alias"
+    return None
+
+
+def _output_is_clean(obj: dict) -> str | None:
+    """Screen one per-event extraction object; return a drop-reason or None."""
+    if not isinstance(obj, dict):
+        return "output_not_object"
+    for ent in obj.get("entities") or []:
+        r = _entity_defect(ent)
+        if r:
+            return r
+    return None
+
+
+def split_events(user_prompt: str) -> dict[int, str]:
+    """Split a chunk prompt into {event_index: block_text}."""
+    blocks: dict[int, str] = {}
+    for block in EVENT_BLOCK_RE.split(user_prompt):
+        block = block.rstrip()
+        m = EVENT_IDX_RE.match(block)
+        if m:
+            blocks[int(m.group(1))] = block
+    return blocks
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--traces", required=True,
+                    help="NDJSON trace rows, or '-' for stdin")
+    ap.add_argument("--out", required=True, help="output .jsonl.gz")
+    ap.add_argument("--hash", default=CLEAN_PROMPT_HASH,
+                    help=f"keep only this system_prompt_hash (default {CLEAN_PROMPT_HASH})")
+    ap.add_argument("--allow-dirty-hash", action="store_true",
+                    help="do NOT filter by hash — DANGER: re-teaches old-prompt defects")
+    ap.add_argument("--report", help="optional path for a JSON stats report")
+    args = ap.parse_args()
+
+    fh = sys.stdin if args.traces == "-" else open(args.traces, encoding="utf-8")
+    stats = Counter()
+    seen: set[str] = set()
+    examples: list[dict] = []
+
+    for line in fh:
+        line = line.strip()
+        if not line:
+            continue
+        stats["trace_rows"] += 1
+        try:
+            row = json.loads(line)
+        except json.JSONDecodeError:
+            stats["drop_trace_unparseable"] += 1
+            continue
+
+        if not args.allow_dirty_hash and row.get("system_prompt_hash") != args.hash:
+            stats["drop_wrong_hash"] += 1
+            continue
+
+        raw = row.get("raw_response") or ""
+        try:
+            parsed = json.loads(raw)
+        except json.JSONDecodeError:
+            stats["drop_response_unparseable"] += 1
+            continue
+        # raw_response is either a single per-event object (current trace
+        # format — one row per event) or, for legacy chunked traces, a JSON
+        # array / {"events": [...]} of per-event objects.
+        if isinstance(parsed, dict):
+            objs = parsed.get("events") if isinstance(parsed.get("events"), list) else [parsed]
+        elif isinstance(parsed, list):
+            objs = parsed
+        else:
+            stats["drop_response_shape"] += 1
+            continue
+
+        blocks = split_events(row.get("user_prompt") or "")
+        for obj in objs:
+            if not isinstance(obj, dict):
+                stats["drop_obj_not_object"] += 1
+                continue
+            idx = obj.get("index")
+            block = blocks.get(idx) if idx is not None else None
+            # Single-event trace: one block, one object — match by position
+            # even if the stored index doesn't line up with the marker.
+            if block is None and len(objs) == 1 and len(blocks) == 1:
+                block = next(iter(blocks.values()))
+            if not block:
+                stats["drop_no_matching_block"] += 1
+                continue
+
+            key = hashlib.sha1(block.encode("utf-8")).hexdigest()
+            if key in seen:
+                stats["drop_dup"] += 1
+                continue
+
+            reason = _output_is_clean(obj)
+            if reason:
+                stats[f"drop_{reason}"] += 1
+                continue
+
+            seen.add(key)
+            examples.append({"input": block, "output": json.dumps(obj, ensure_ascii=False)})
+            stats["kept"] += 1
+
+    if args.traces != "-":
+        fh.close()
+
+    with gzip.open(args.out, "wt", encoding="utf-8") as out:
+        for ex in examples:
+            out.write(json.dumps(ex, ensure_ascii=False) + "\n")
+
+    report = {"out": args.out, "hash": (None if args.allow_dirty_hash else args.hash),
+              "stats": dict(sorted(stats.items()))}
+    print(json.dumps(report, indent=2))
+    if args.report:
+        with open(args.report, "w", encoding="utf-8") as rf:
+            json.dump(report, rf, indent=2)
+
+    if stats["kept"] == 0:
+        print("\nWARNING: 0 examples kept. If you targeted the clean hash, the "
+              "clean-prompt teacher has not produced enough gold yet — let it "
+              "accumulate (or run a teacher-only re-distill of a curated event "
+              "slice through the clean prompt), then re-run.", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/packages/memory-engine-v2/scripts/fusion_defrag.py

@@ -0,0 +1,440 @@
+#!/usr/bin/env python3
+"""Fusion de-fragmentation — cluster same-surname PERSON fragments and propose
+merges (RFC-decay-and-fusion A2/A3). DRY-RUN by default.
+
+The deterministic upsert resolver (worker.py) converges same-form / shared-alias
+entities, but it CANNOT safely merge surface-form/nickname variants of one real
+person (e.g. "Will Vickers" / "William Vickers" / "William F. Vickers" / bare
+"Vickers") — they have different normalized names and no shared alias, so they
+fragment (209 "Vickers" nodes observed). Merging them is Fusion's job, and it is
+DESTRUCTIVE (repoints facts/relationships, tombstones losers), so the over-merge
+failure mode (folding two DIFFERENT people, or a person into an org) must be
+designed out. This tool is conservative + dry-run-first; --apply is double-gated.
+
+CLUSTERING (anti-over-merge by construction):
+  - PERSON entities only; never crosses entity_type (so "Vickers Oils" the org is
+    never pulled in).
+  - Same surname token (the --surname scope).
+  - First-name compatibility for the NON-surname tokens: equal, or one an initial
+    of the other (W ↔ William), or one a prefix of the other (Will ⊂ William).
+    Two DISTINCT full first names (Will vs Jane) are INCOMPATIBLE → never merged.
+  - Union-find over compatible NON-bare names → each cluster = one real person.
+  - Bare "<surname>" nodes (no first name) are folded in ONLY when there is
+    exactly ONE non-bare cluster for the surname (unambiguous); otherwise they
+    are left for human review (never used to bridge two distinct people).
+
+CANONICAL (A3 scored master, replaces richest-row-wins):
+  + has email (attributes.email or an email alias)   strongest identity signal
+  + full name (>=2 name tokens)                       a real rendering, not a stub
+  + corroboration (provenance event count)            grounded in more events
+  + fact count                                        the node that holds the picture
+  - bare single-token name                            penalize stub
+  - ID-like (digit ratio > 0.5)                       penalize 7B numeric-id junk
+
+OUTPUT: per cluster — master, losers, why, and the repoint impact (facts +
+relationships that would move onto the master). No DB writes in dry-run
+(the session is forced read-only). --apply would execute via the reviewed
+fusion_drive merge executor + entity_merges audit (NOT enabled here).
+
+Usage:
+    python fusion_defrag.py --arena 'pentatonic-team%' --surname vickers
+    python fusion_defrag.py --arena 'pentatonic-team%' --surname vickers --json out.json
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+import uuid
+from collections import defaultdict
+
+
+def _connect(dsn: str):
+    import psycopg
+    import psycopg.rows
+    return psycopg.connect(dsn, row_factory=psycopg.rows.dict_row)
+
+
+_TOKEN_RE = re.compile(r"[^a-z0-9]+")
+_EMAIL_RE = re.compile(r"^[^@\s]+@[^@\s]+\.[^@\s]+$")
+
+
+def name_tokens(name: str) -> list[str]:
+    return [t for t in _TOKEN_RE.split((name or "").lower()) if t]
+
+
+def digit_ratio(name: str) -> float:
+    s = re.sub(r"\s+", "", name or "")
+    return sum(c.isdigit() for c in s) / len(s) if s else 0.0
+
+
+# Honorifics/titles are NOT first names — strip them so "Herr Johann Boedecker"
+# matches "Johann Boedecker", and a title-only "Herr Boedecker" reduces to a bare
+# surname (held for review, not merged on the title).
+_TITLES = {"herr", "frau", "fr", "dr", "prof", "mr", "mrs", "ms", "miss",
+           "sir", "dame", "mx", "mme", "mlle", "hr"}
+
+
+def first_name_tokens(name: str, surname: str) -> list[str]:
+    """Name tokens minus the surname (first occurrence) and minus honorifics."""
+    toks = name_tokens(name)
+    out, dropped = [], False
+    for t in toks:
+        if not dropped and t == surname:
+            dropped = True
+            continue
+        if t in _TITLES:
+            continue
+        out.append(t)
+    return out
+
+
+def first_names_compatible(a: list[str], b: list[str]) -> bool:
+    """Compatible iff the leading given-name tokens don't CONFLICT. Equal /
+    initial-of / prefix-of are compatible; two distinct full names are not.
+    Empty (bare surname) is handled separately by the caller — NOT here."""
+    if not a or not b:
+        return False  # bare names never auto-bridge via this predicate
+    x, y = a[0], b[0]
+    if x == y:
+        return True
+    # initial ↔ full (w / william)
+    if (len(x) == 1 and y.startswith(x)) or (len(y) == 1 and x.startswith(y)):
+        return True
+    # nickname/prefix (will / william) — require >=3 chars to avoid junk
+    if len(x) >= 3 and y.startswith(x):
+        return True
+    if len(y) >= 3 and x.startswith(y):
+        return True
+    return False
+
+
+class UnionFind:
+    def __init__(self, ids):
+        self.p = {i: i for i in ids}
+
+    def find(self, i):
+        while self.p[i] != i:
+            self.p[i] = self.p[self.p[i]]
+            i = self.p[i]
+        return i
+
+    def union(self, a, b):
+        ra, rb = self.find(a), self.find(b)
+        if ra != rb:
+            self.p[ra] = rb
+
+
+def has_email(ent: dict) -> bool:
+    attrs = ent.get("attributes") or {}
+    if isinstance(attrs, dict) and attrs.get("email"):
+        return True
+    return any(isinstance(a, str) and _EMAIL_RE.match(a) for a in (ent.get("aliases") or []))
+
+
+def master_score(ent: dict, surname: str) -> float:
+    fn = first_name_tokens(ent["canonical_name"], surname)
+    score = 0.0
+    if has_email(ent):
+        score += 3.0
+    if len(name_tokens(ent["canonical_name"])) >= 2:
+        score += 2.0
+    score += min(len(ent.get("provenance_event_ids") or []), 20) * 0.2
+    score += min(ent.get("fact_n", 0), 40) * 0.05
+    if not fn:  # bare surname
+        score -= 3.0
+    if digit_ratio(ent["canonical_name"]) > 0.5:
+        score -= 5.0
+    return score
+
+
+def _union(*lists):
+    seen = {}
+    for lst in lists:
+        for x in lst or []:
+            seen.setdefault(x, None)
+    return list(seen.keys())
+
+
+def apply_cluster(cur, conn, arena: str, master: dict, losers: list[dict]) -> dict:
+    """Fold `losers` into `master` within ONE transaction. Faithful inline of the
+    reviewed fusion_drive _execute_entity_plan + build_entity_merge_plan: repoint
+    facts (subject/object) and relationships (endpoints, summing weight on the
+    post-repoint (from,to,type) collision), accrete aliases+provenance onto the
+    master, write one entity_merges audit row per loser (rollback_payload = full
+    loser row), then delete the losers. Re-validates losers still exist first."""
+    loser_ids = [l["id"] for l in losers]
+    # Load edges/facts touching the losers AND THE MASTER. Including the master is
+    # load-bearing for relationships: a loser edge repointed onto the master can
+    # collide with an edge the master ALREADY has (or another loser's) on the
+    # UNIQUE(arena,from,to,type) key — if we don't see the master's existing edges
+    # in collision detection, the repoint UPDATE hits a duplicate-key violation
+    # (caught the hard way on the first Vickers apply; the txn rolled back clean).
+    # Facts have no such unique key, so master facts are loaded but never repointed
+    # (repoint decisions key on the loser set only).
+    targets = loser_ids + [master["id"]]
+    cur.execute(
+        "SELECT id, subject_entity_id, object_entity_id FROM facts "
+        "WHERE arena = %s AND (subject_entity_id = ANY(%s) OR object_entity_id = ANY(%s))",
+        (arena, targets, targets),
+    )
+    facts = cur.fetchall()
+    cur.execute(
+        "SELECT id, from_entity_id, to_entity_id, relationship_type, weight, "
+        "provenance_event_ids FROM relationships "
+        "WHERE arena = %s AND (from_entity_id = ANY(%s) OR to_entity_id = ANY(%s))",
+        (arena, targets, targets),
+    )
+    rels = cur.fetchall()
+
+    lset = set(loser_ids)
+    aliases = _union(master.get("aliases") or [],
+                     [l["canonical_name"] for l in losers],
+                     *[l.get("aliases") or [] for l in losers])
+    aliases = [a for a in aliases if a != master["canonical_name"]]
+    provenance = _union(master.get("provenance_event_ids") or [],
+                        *[l.get("provenance_event_ids") or [] for l in losers])
+    fact_subj = [f["id"] for f in facts if f["subject_entity_id"] in lset]
+    fact_obj = [f["id"] for f in facts if f["object_entity_id"] in lset]
+
+    def rk(r):
+        frm = master["id"] if r["from_entity_id"] in lset else r["from_entity_id"]
+        to = master["id"] if r["to_entity_id"] in lset else r["to_entity_id"]
+        return (frm, to, r["relationship_type"])
+    by_key, rel_repoints, rel_collisions = {}, [], []
+    for r in rels:
+        touches = r["from_entity_id"] in lset or r["to_entity_id"] in lset
+        key = rk(r)
+        if key in by_key:
+            keep = by_key[key]
+            rel_collisions.append({
+                "keep": keep["id"], "drop": r["id"],
+                "summed_weight": round((keep.get("weight") or 1.0) + (r.get("weight") or 1.0), 4),
+                "provenance": _union(keep.get("provenance_event_ids") or [],
+                                     r.get("provenance_event_ids") or []),
+            })
+        else:
+            by_key[key] = r
+            if touches:
+                rel_repoints.append(r["id"])
+
+    with conn.transaction():
+        live = set()
+        cur.execute("SELECT id FROM entities WHERE id = ANY(%s)", (loser_ids,))
+        live = {r["id"] for r in cur.fetchall()}
+        if live != lset:
+            return {"applied": False, "reason": "stale: some losers already gone"}
+        cur.execute("UPDATE entities SET aliases=%s, provenance_event_ids=%s, last_seen=NOW() "
+                    "WHERE id=%s", (aliases, provenance, master["id"]))
+        for fid in fact_subj:
+            cur.execute("UPDATE facts SET subject_entity_id=%s WHERE id=%s", (master["id"], fid))
+        for fid in fact_obj:
+            cur.execute("UPDATE facts SET object_entity_id=%s WHERE id=%s", (master["id"], fid))
+        # DELETE colliding edges BEFORE repointing — else repointing a "keep" edge
+        # onto the master collides with the not-yet-deleted "drop" on the UNIQUE
+        # (arena,from,to,type) key. Carry each drop's weight+provenance onto its keep.
+        for col in rel_collisions:
+            cur.execute("UPDATE relationships SET weight=%s, provenance_event_ids=%s WHERE id=%s",
+                        (col["summed_weight"], col["provenance"], col["keep"]))
+            cur.execute("DELETE FROM relationships WHERE id=%s", (col["drop"],))
+        for rid in rel_repoints:
+            cur.execute(
+                "UPDATE relationships SET "
+                "from_entity_id = CASE WHEN from_entity_id = ANY(%s) THEN %s ELSE from_entity_id END, "
+                "to_entity_id   = CASE WHEN to_entity_id   = ANY(%s) THEN %s ELSE to_entity_id END "
+                "WHERE id=%s",
+                (loser_ids, master["id"], loser_ids, master["id"], rid))
+        for l in losers:
+            cur.execute(
+                "INSERT INTO entity_merges (id, arena, canonical_id, deprecated_id, "
+                "deprecated_canonical_name, deprecated_aliases, merge_signal, "
+                "facts_repointed, relationships_repointed, merged_by, rollback_payload) "
+                "VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s::jsonb)",
+                ("em_" + uuid.uuid4().hex[:20], arena, master["id"], l["id"],
+                 l["canonical_name"], l.get("aliases") or [], "heuristic",
+                 len(fact_subj) + len(fact_obj), len(rel_repoints), "fusion-defrag",
+                 json.dumps(l, default=str)))
+        cur.execute("DELETE FROM entities WHERE id = ANY(%s)", (loser_ids,))
+    return {"applied": True, "facts_repointed": len(fact_subj) + len(fact_obj),
+            "rels_repointed": len(rel_repoints), "rel_collisions": len(rel_collisions),
+            "tombstoned": len(loser_ids)}
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--arena", required=True, help="arena LIKE filter (REQUIRED)")
+    ap.add_argument("--surname", required=True, help="surname token to scope (e.g. vickers)")
+    ap.add_argument("--pg-dsn", default="", help="Postgres DSN (or PG_DSN env)")
+    ap.add_argument("--json", help="write the proposal as JSON")
+    ap.add_argument("--apply", action="store_true",
+                    help="EXECUTE the merges (default: dry-run). Requires --i-have-a-snapshot.")
+    ap.add_argument("--i-have-a-snapshot", action="store_true",
+                    help="operator asserts a DB/row snapshot exists for rollback (required with --apply)")
+    args = ap.parse_args()
+    if args.apply and not args.i_have_a_snapshot:
+        print("REFUSED: --apply requires --i-have-a-snapshot (take a snapshot first; "
+              "merges repoint facts/rels + tombstone nodes — entity_merges holds rollback "
+              "payloads but a row/DB snapshot is the real safety net).", file=sys.stderr)
+        return 2
+
+    import os
+    dsn = args.pg_dsn or os.environ.get("PG_DSN", "")
+    if not dsn:
+        print("FATAL: no --pg-dsn / PG_DSN", file=sys.stderr)
+        return 2
+    surname = args.surname.lower()
+
+    with _connect(dsn) as conn:
+        with conn.cursor() as cur:
+            if not args.apply:
+                cur.execute("SET default_transaction_read_only = on")  # dry-run safety
+            cur.execute("SET max_parallel_workers_per_gather = 0")
+            # Person fragments whose CANONICAL NAME carries the surname as a real
+            # token. Critically NOT alias-scoped: an @vickers-oil.com email domain
+            # in aliases would otherwise drag in unrelated employees (Paul Vann,
+            # Matt Tooze) who merely WORK at a Vickers company — the over-merge the
+            # first dry-run caught. Email-named stubs (canonical_name has '@') are
+            # also excluded: tokenizing an email is not a safe surname signal (they
+            # converge via the alias-resolution path instead).
+            cur.execute(
+                """
+                SELECT e.id, e.arena, e.canonical_name, e.aliases, e.provenance_event_ids,
+                       e.attributes, e.last_seen,
+                       (SELECT count(*) FROM facts f
+                          WHERE f.provenance_event_ids && e.provenance_event_ids
+                            AND (f.subject_entity_id = e.id OR f.object_entity_id = e.id)) AS fact_n
+                  FROM entities e
+                 WHERE e.arena LIKE %s AND e.entity_type = 'person'
+                   AND position('@' in e.canonical_name) = 0
+                   AND lower(e.canonical_name) ~ %s
+                """,
+                (args.arena, rf"(^|[^a-z]){surname}([^a-z]|$)"),
+            )
+            ents = cur.fetchall()
+            # Confirm the surname is a real NAME token (regex is a coarse guard).
+            ents = [e for e in ents if surname in name_tokens(e["canonical_name"])]
+            print(f"[defrag] arena={args.arena} surname={surname!r}: "
+                  f"{len(ents)} person fragments")
+            if not ents:
+                return 0
+
+            by_id = {e["id"]: e for e in ents}
+            non_bare = [e for e in ents if first_name_tokens(e["canonical_name"], surname)]
+            bare = [e for e in ents if not first_name_tokens(e["canonical_name"], surname)]
+
+            # Union-find over compatible non-bare names.
+            uf = UnionFind([e["id"] for e in non_bare])
+            for i in range(len(non_bare)):
+                for j in range(i + 1, len(non_bare)):
+                    a, b = non_bare[i], non_bare[j]
+                    # Never union across exact arenas — entity id = hash(arena|
+                    # type|name), so cross-arena same-name nodes are genuinely
+                    # different scoped entities; merging them would be wrong.
+                    if a["arena"] != b["arena"]:
+                        continue
+                    if first_names_compatible(
+                        first_name_tokens(a["canonical_name"], surname),
+                        first_name_tokens(b["canonical_name"], surname),
+                    ):
+                        uf.union(a["id"], b["id"])
+            clusters: dict[str, list[dict]] = defaultdict(list)
+            for e in non_bare:
+                clusters[uf.find(e["id"])].append(e)
+
+            # Bare surnames: fold in ONLY if exactly one non-bare cluster exists.
+            bare_note = ""
+            if bare:
+                if len(clusters) == 1:
+                    only = next(iter(clusters))
+                    cl_arena = clusters[only][0]["arena"]
+                    same = [b for b in bare if b["arena"] == cl_arena]
+                    clusters[only].extend(same)
+                    bare_note = f"{len(same)} bare-'{surname}' node(s) folded into the single cluster"
+                else:
+                    bare_note = (f"{len(bare)} bare-'{surname}' node(s) LEFT FOR REVIEW "
+                                 f"({len(clusters)} distinct name-clusters — ambiguous which person)")
+
+            proposals = []
+            for cid, members in clusters.items():
+                if len(members) < 2:
+                    continue
+                master = max(members, key=lambda e: (master_score(e, surname),
+                                                     len(e.get("provenance_event_ids") or []),
+                                                     len(e["canonical_name"])))
+                losers = [e for e in members if e["id"] != master["id"]]
+                loser_ids = [l["id"] for l in losers]
+                # Repoint impact (read-only counts).
+                cur.execute(
+                    "SELECT count(*) AS n FROM facts WHERE arena LIKE %s AND "
+                    "(subject_entity_id = ANY(%s) OR object_entity_id = ANY(%s))",
+                    (args.arena, loser_ids, loser_ids),
+                )
+                facts_repointed = cur.fetchone()["n"]
+                cur.execute(
+                    "SELECT count(*) AS n FROM relationships WHERE arena LIKE %s AND "
+                    "(from_entity_id = ANY(%s) OR to_entity_id = ANY(%s))",
+                    (args.arena, loser_ids, loser_ids),
+                )
+                rels_repointed = cur.fetchone()["n"]
+                proposals.append({
+                    "arena": master["arena"],
+                    "master_row": master,   # full row for apply
+                    "loser_rows": losers,
+                    "master": {"id": master["id"], "name": master["canonical_name"],
+                               "facts": master.get("fact_n", 0),
+                               "prov": len(master.get("provenance_event_ids") or []),
+                               "email": has_email(master),
+                               "score": round(master_score(master, surname), 2)},
+                    "losers": [{"id": l["id"], "name": l["canonical_name"],
+                                "facts": l.get("fact_n", 0),
+                                "prov": len(l.get("provenance_event_ids") or []),
+                                "email": has_email(l)} for l in losers],
+                    "facts_repointed": facts_repointed,
+                    "rels_repointed": rels_repointed,
+                })
+
+            if args.apply and proposals:
+                conn.rollback()  # end the read-only probe txn cleanly before writes
+                print(f"\n[defrag] APPLYING {len(proposals)} cluster(s) — arena-scoped, transactional…")
+                for p in proposals:
+                    p["apply_result"] = apply_cluster(cur, conn, p["arena"],
+                                                       p["master_row"], p["loser_rows"])
+                    print(f"  master={p['master']['name']!r} ({p['arena']}): {p['apply_result']}")
+
+    # ---- report ----
+    mode = "APPLIED" if args.apply else "PROPOSED (dry-run, no writes)"
+    print(f"\n=== {mode} MERGES — surname '{surname}' ===")
+    if bare_note:
+        print(f"  note: {bare_note}")
+    if not proposals:
+        print("  (no multi-node clusters — nothing to merge)")
+    tot_dep = tot_f = tot_r = 0
+    for i, p in enumerate(proposals, 1):
+        m = p["master"]
+        print(f"\n[{i}] MASTER ← {m['name']!r}  ({m['id'][:10]}…) "
+              f"score={m['score']} facts={m['facts']} prov={m['prov']} email={m['email']}")
+        for l in p["losers"]:
+            print(f"      merge: {l['name']!r}  ({l['id'][:10]}…) "
+                  f"facts={l['facts']} prov={l['prov']} email={l['email']}")
+        print(f"      → would repoint {p['facts_repointed']} facts, "
+              f"{p['rels_repointed']} relationships onto the master; "
+              f"{len(p['losers'])} node(s) tombstoned")
+        tot_dep += len(p["losers"]); tot_f += p["facts_repointed"]; tot_r += p["rels_repointed"]
+    tail = ("APPLIED — rollback via entity_merges (merged_by='fusion-defrag') + snapshot."
+            if args.apply else "DRY-RUN — nothing written.")
+    print(f"\nTOTAL: {len(proposals)} cluster(s), {tot_dep} nodes tombstoned, "
+          f"{tot_f} facts + {tot_r} relationships repointed. {tail}")
+    if args.json:
+        with open(args.json, "w") as f:
+            json.dump({"surname": surname, "arena": args.arena, "applied": args.apply,
+                       "bare_note": bare_note,
+                       "proposals": [{k: v for k, v in p.items()
+                                      if k not in ("master_row", "loser_rows")}
+                                     for p in proposals]}, f, indent=2)
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

package/packages/memory-engine-v2/scripts/redistill.py

@@ -0,0 +1,236 @@
+#!/usr/bin/env python3
+"""Prompt-version-drift re-distillation — RFC-decay-and-fusion Part B′.
+
+The third triage verb of the Fusion Drive: regenerate high-value extractions
+that were produced by a *superseded* teacher/prompt by re-running their source
+event through the *current clean* teacher. Fusion converges horizontally, decay
+ages vertically; this refreshes in depth.
+
+TRIGGER (not raw age): an event is stale when `event_distillations` shows it was
+distilled only under a dirty `system_prompt_hash` (pre-clean: bbdaba/f1e0ff/
+ef0647…) and never under the clean one (`6ccfe70f…`, SDK 0.10.19 = #126 + #129).
+
+MECHANISM: re-insert the event into `distillation_queue` (status=pending). The
+existing extractor-async worker + autoscaler do the rest, writing a fresh clean
+extraction AND a clean trace (which `build_retrain_corpus.py` then harvests — one
+loop repairs the graph and grows the retrain corpus).
+
+SUPERSEDENCE (the load-bearing requirement — Part B′4): the store is
+pure-accretion, so a naive re-enqueue makes the clean extraction land *beside*
+the dirty one and WORSENS fragmentation. The durable fix routes supersede
+through Fusion's tombstone machinery (RFC A2/A3), which is not yet live. This
+tool ships the **interim event-scoped supersede** (Part B′4 Open Q): with
+--supersede-facts it deletes only facts whose provenance set is exactly
+{this event} under the old hash — the single-provenance majority, where no
+corroboration is lost — after dumping a rollback payload. Multi-provenance facts
+and ALL entities are left untouched (entity-level fragmentation is fusion/decay's
+job; deleting an entity cascades to relationships and NULLs other facts' FKs).
+
+SAFETY: dry-run by default; --apply gated; --arena REQUIRED (no global runs);
+--limit caps the batch; every destructive op is dumped to a rollback JSONL first;
+a run ledger is written per invocation. Idempotent: skips events that already
+have a pending/claimed queue row or an existing clean-hash distillation.
+
+Usage:
+    # size the work (no writes):
+    python redistill.py --arena 'pentatonic-team%' --dry-run
+    # re-enqueue a bounded slice (non-destructive — accretes until fusion runs):
+    python redistill.py --arena 'pentatonic-team%' --limit 25 --apply
+    # re-enqueue AND interim-supersede single-provenance stale facts:
+    python redistill.py --arena 'pentatonic-team%' --limit 25 --apply --supersede-facts
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from datetime import datetime, timezone
+
+CLEAN_PROMPT_HASH = "6ccfe70f1286a131"  # SDK 0.10.19 (#126 + #129); verify vs worker.SYSTEM_PROMPT_HASH
+
+
+def _connect(dsn: str):
+    import psycopg
+    import psycopg.rows
+    return psycopg.connect(dsn, row_factory=psycopg.rows.dict_row)
+
+
+def triage(cur, arena: str, clean_hash: str, dirty_hashes: list[str] | None, limit: int,
+           order_by_recency: bool = False):
+    """Stale events: distilled under a dirty prompt, not yet re-distilled clean.
+
+    IDEMPOTENCY SIGNAL = `distillation_traces.system_prompt_hash`, NOT
+    `event_distillations`. The teacher ALWAYS writes a trace (worker.py
+    `_insert_trace`, producer=='teacher'), but `event_distillations` is written
+    ONLY when CASCADE_ENABLED (worker.py line ~2325). In teacher-only mode a
+    re-distilled event gets a clean *trace* but its event_distillations row stays
+    stamped at the old student hash — so keying the clean-check off the ledger
+    would re-select the same events forever. Traces are the reliable per-event
+    prompt-version record here.
+
+    Dirty evidence = a dirty-hash trace OR a dirty-hash event_distillations row
+    (covers student-distilled events, which have a ledger row but no trace)."""
+    params: list = [arena, clean_hash]          # arena, clean (no-clean-trace check)
+    dirty_trace = "AND t2.system_prompt_hash <> %s"
+    params_tail_dirty: list = [clean_hash]
+    if dirty_hashes:
+        dirty_trace = "AND t2.system_prompt_hash = ANY(%s)"
+        params_tail_dirty = [dirty_hashes]
+    # Build param order to match placeholders below.
+    params = [arena, clean_hash] + params_tail_dirty + [clean_hash, limit]
+    # ORDER BY received_at forces a full scan+sort of the whole arena (~270k rows
+    # for pentatonic-team) every call — pathological for bulk/loop work. Default
+    # OFF so the planner early-terminates once LIMIT stale rows are found (most
+    # arena rows ARE stale, so it scans ~LIMIT rows). Recency is only a salience
+    # proxy anyway, moot until the salience column lands (RFC Part D). Opt in with
+    # --order-by-recency for a one-off "freshest first" pass.
+    order_clause = "ORDER BY e.received_at DESC" if order_by_recency else ""
+    cur.execute(
+        f"""
+        WITH stale AS (
+          SELECT e.id AS event_id, e.received_at AS recency
+          FROM events e
+          WHERE e.arena LIKE %s
+            AND NOT EXISTS (                          -- not yet re-distilled clean
+              SELECT 1 FROM distillation_traces t
+              WHERE t.event_id = e.id AND t.system_prompt_hash = %s)
+            AND NOT EXISTS (                          -- not already queued (so loops/batches advance)
+              SELECT 1 FROM distillation_queue q
+              WHERE q.event_id = e.id AND q.status IN ('pending','claimed'))
+            AND (
+              EXISTS (SELECT 1 FROM distillation_traces t2     -- dirty teacher trace
+                      WHERE t2.event_id = e.id {dirty_trace})
+              OR EXISTS (SELECT 1 FROM event_distillations d   -- or dirty student/teacher ledger
+                         WHERE d.event_id = e.id
+                           AND d.system_prompt_hash IS NOT NULL
+                           AND d.system_prompt_hash <> %s)
+            )
+          {order_clause}
+          LIMIT %s
+        )
+        SELECT s.event_id,
+               -- `@> ARRAY[id]` (containment) uses idx_facts_provenance (GIN);
+               -- `id = ANY(col)` does NOT and seq-scans facts per row (2 min+ at
+               -- limit 1000). Must stay @> for the bulk/loop path to be viable.
+               (SELECT count(*) FROM facts f
+                  WHERE f.provenance_event_ids @> ARRAY[s.event_id]) AS fact_n,
+               (SELECT count(*) FROM facts f
+                  WHERE f.provenance_event_ids @> ARRAY[s.event_id]
+                    AND cardinality(f.provenance_event_ids) = 1) AS solo_fact_n,
+               EXISTS (SELECT 1 FROM distillation_queue q
+                       WHERE q.event_id = s.event_id
+                         AND q.status IN ('pending','claimed')) AS in_flight
+        FROM stale s
+        ORDER BY fact_n DESC
+        """,
+        params,
+    )
+    return cur.fetchall()
+
+
+def dump_and_delete_solo_facts(cur, event_id: str, rollback_fh) -> int:
+    """Delete facts whose provenance is exactly {event_id}; dump them first."""
+    solo = "provenance_event_ids @> ARRAY[%s] AND cardinality(provenance_event_ids) = 1"
+    cur.execute(f"SELECT * FROM facts WHERE {solo}", (event_id,))
+    rows = cur.fetchall()
+    for r in rows:
+        rollback_fh.write(json.dumps({"op": "delete_fact", "event_id": event_id,
+                                      "row": _jsonable(r)}, ensure_ascii=False) + "\n")
+    if rows:
+        cur.execute(f"DELETE FROM facts WHERE {solo}", (event_id,))
+    return len(rows)
+
+
+def _jsonable(row: dict) -> dict:
+    out = {}
+    for k, v in row.items():
+        out[k] = v.isoformat() if isinstance(v, datetime) else v
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--arena", required=True, help="arena LIKE filter (REQUIRED — scope safety)")
+    ap.add_argument("--clean-hash", default=CLEAN_PROMPT_HASH)
+    ap.add_argument("--dirty-hashes", help="comma-separated hashes to target (default: any != clean)")
+    ap.add_argument("--limit", type=int, default=50, help="max events to re-enqueue (safety cap)")
+    ap.add_argument("--apply", action="store_true", help="actually write (default: dry-run)")
+    ap.add_argument("--supersede-facts", action="store_true",
+                    help="also delete single-provenance stale facts (interim supersede, Part B′4)")
+    ap.add_argument("--order-by-recency", action="store_true",
+                    help="freshest stale events first (forces a full arena scan+sort — slow; off by default)")
+    ap.add_argument("--rollback-dir", default=".", help="dir for rollback + ledger JSONL")
+    ap.add_argument("--pg-dsn", default=os.environ.get("PG_DSN", ""), help="Postgres DSN")
+    args = ap.parse_args()
+
+    if not args.pg_dsn:
+        print("FATAL: no --pg-dsn / PG_DSN", file=sys.stderr)
+        return 2
+    dirty = [h.strip() for h in args.dirty_hashes.split(",")] if args.dirty_hashes else None
+    stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
+
+    with _connect(args.pg_dsn) as conn:
+        with conn.cursor() as cur:
+            # The org-model postgres container runs with the Docker default 64MB
+            # /dev/shm; a parallel-gather over the facts GIN exhausts the dynamic
+            # shared-memory segment ("DiskFull ... shared memory"). Disable
+            # parallel workers for this session — temp spills go to the (ample)
+            # data disk instead of shm.
+            cur.execute("SET max_parallel_workers_per_gather = 0")
+            cands = triage(cur, args.arena, args.clean_hash, dirty, args.limit,
+                           order_by_recency=args.order_by_recency)
+
+            total = len(cands)
+            in_flight = sum(1 for c in cands if c["in_flight"])
+            actionable = [c for c in cands if not c["in_flight"]]
+            solo = sum(c["solo_fact_n"] for c in actionable)
+            multi = sum(c["fact_n"] - c["solo_fact_n"] for c in actionable)
+            print(json.dumps({
+                "mode": "apply" if args.apply else "dry-run",
+                "arena": args.arena, "clean_hash": args.clean_hash,
+                "candidates": total, "in_flight_skipped": in_flight,
+                "actionable": len(actionable),
+                "stale_facts_solo_provenance": solo,
+                "stale_facts_multi_provenance_LEFT_ALONE": multi,
+                "supersede_facts": args.supersede_facts,
+            }, indent=2))
+
+            if not args.apply:
+                print("\n[dry-run] no writes. Re-run with --apply to re-enqueue"
+                      + (" + supersede solo facts." if args.supersede_facts else "."))
+                return 0
+
+            rb_path = os.path.join(args.rollback_dir, f"redistill_rollback_{stamp}.jsonl")
+            ledger_path = os.path.join(args.rollback_dir, f"redistill_runs_{stamp}.jsonl")
+            enq = deleted = 0
+            with open(rb_path, "w", encoding="utf-8") as rb, \
+                    open(ledger_path, "w", encoding="utf-8") as led:
+                for c in actionable:
+                    eid = c["event_id"]
+                    if args.supersede_facts:
+                        deleted += dump_and_delete_solo_facts(cur, eid, rb)
+                    cur.execute(
+                        "INSERT INTO distillation_queue (event_id, status, attempts) "
+                        "VALUES (%s, 'pending', 0)", (eid,))
+                    enq += 1
+                    led.write(json.dumps({
+                        "event_id": eid, "arena": args.arena,
+                        "old_hash": "dirty", "target_hash": args.clean_hash,
+                        "solo_facts_superseded": c["solo_fact_n"] if args.supersede_facts else 0,
+                        "enqueued_at": stamp,
+                    }, ensure_ascii=False) + "\n")
+                conn.commit()
+
+            print(json.dumps({
+                "applied": True, "re_enqueued": enq, "facts_superseded": deleted,
+                "rollback": rb_path, "ledger": ledger_path,
+            }, indent=2))
+            if deleted:
+                print(f"\nROLLBACK: facts are in {rb_path} — re-INSERT them to undo.")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())