@pentatonic-ai/ai-agent-sdk 0.10.18 → 0.10.20

package/dist/index.cjs

@@ -878,7 +878,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
 }
 
 // src/telemetry.js
-var VERSION = "0.10.18";
+var VERSION = "0.10.20";
 var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
 function machineId() {
   const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";

package/dist/index.js

@@ -847,7 +847,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
 }
 
 // src/telemetry.js
-var VERSION = "0.10.18";
+var VERSION = "0.10.20";
 var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
 function machineId() {
   const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pentatonic-ai/ai-agent-sdk",
-  "version": "0.10.18",
+  "version": "0.10.20",
   "description": "TES SDK — LLM observability and lifecycle tracking via Pentatonic Thing Event System. Track token usage, tool calls, and conversations. Manage things through event-sourced lifecycle stages with AI enrichment and vector search.",
   "type": "module",
   "main": "./dist/index.cjs",

package/packages/memory-engine-v2/RFC-decay-and-fusion.md

@@ -1,10 +1,18 @@
 # RFC: the Fusion Drive — v2 memory self-healing (cross-run node fusion + decay)
 
 > **Fusion Drive** = the continuous, arena-scoped background engine that keeps the v2
-> memory graph self-healing: it *fuses* duplicate/near-duplicate nodes from different
-> distillation runs into a single master node (horizontal convergence) and *decays* stale,
-> low-value, and junk nodes out of existence (vertical aging). Named for the drive that
-> does the fusing — the decay pass rides the same engine.
+> memory graph self-healing. It triages every node into one of **three** outcomes:
+> it *fuses* duplicate/near-duplicate nodes from different distillation runs into a single
+> master node (horizontal convergence); it *re-distills* high-value extractions produced by
+> a superseded teacher/prompt — regenerating them from the still-present source event through
+> the current clean teacher (depth refresh); and it *decays* stale, low-value, and junk nodes
+> out of existence (vertical aging). Named for the drive that does the fusing — the re-distill
+> and decay passes ride the same engine.
+>
+> *(Revised 2026-06-22: added Part B′ — re-distillation — as the third triage verb, with the
+> prompt-version-drift trigger. Motivated by the clean-prompt deploy (SDK 0.10.19, #126 +
+> #129) which made "the current teacher is materially better than the one that produced most
+> of the graph" concrete and measurable via `system_prompt_hash`.)*
 
 **Status:** draft / spec — 2026-06-12
 **Builds on:** `RFC-entity-reconciliation.md`, `scripts/entity_resolution_v2.py` (#82),
@@ -139,15 +147,101 @@ sparse backfill.
 
 ---
 
+## Part B′ — Re-distillation: regenerate stale-prompt extractions from source
+
+Fusion (A) needs a *correct counterpart* to converge toward; Decay (B) just *deletes*. But
+the common case after a teacher/prompt upgrade is a **high-value node with no correct
+counterpart yet** — the only extraction that exists is the stale-prompt one. Fusion has
+nothing to fuse to; decay would throw away real information. The cure is the third verb: the
+**source event still exists** (`events` table, 376k rows live), so regenerate the extraction
+by re-running that event through the *current clean teacher*. Fusion converges horizontally,
+decay ages vertically; re-distill refreshes **in depth**.
+
+### B′1. Trigger — prompt-version drift, not raw age
+The defect population is *exactly* the facts/entities whose provenance traces an **old
+`system_prompt_hash`** — `bbdaba6b…` / `f1e0ff55…` / `ef0647c7…` (pre-clean), vs the clean
+`6ccfe70f…` deployed with 0.10.19 (#126 modality/attribution + #129 email-discipline &
+entity-separation). #118 propagated source onto facts, so provenance → the event's
+`distillation_traces.system_prompt_hash` is queryable. **Age is a weak proxy; prompt-version
+selects the defect set directly** — a months-old node the clean teacher would extract
+identically needs nothing; a two-day-old node from the dirty prompt is a defect. Prioritize
+by `salience` (B1) so high-value stale nodes go first.
+
+### B′2. Triage routing — 3-way, by salience × prompt-version
+Per assessed node/event:
+
+| condition | outcome |
+|---|---|
+| stale prompt-hash **+** high salience **+** source event present | **re-distill** (this part) |
+| has a correct newer-teacher counterpart in the arena | **fuse** (Part A) |
+| low salience, junk-born (B2), no corroboration, never accessed | **decay** (Part B) |
+
+### B′3. Mechanism — re-enqueue, don't mutate in place
+Re-distill = re-insert the source `event_id` into `distillation_queue` (`status='pending'`,
+`attempts=0`). The existing **extractor-async** worker claims it, runs the clean teacher,
+writes the new extraction **and a fresh `6ccfe70f` trace**. No new pipeline — it reuses the
+distiller, the combined-demand **autoscaler**, and the trace ledger. (Re-distill is a
+*producer* of queue demand; the autoscaler's student-aware floor already keeps a teacher box
+warm for it — see the deploy notes.)
+
+### B′4. Supersedence — the load-bearing requirement
+The store is **pure-accretion** (the whole motivation of this RFC). A naive re-enqueue makes
+the clean extraction land **beside** the dirty one → it *worsens* fragmentation. So
+re-distill MUST close the loop through Fusion's tombstone machinery — it is **sequenced into
+the Fusion Drive, not bolted on**:
+
+1. Each re-distill is recorded in a `redistill_runs` ledger with its triggering
+   `(event_id, old_prompt_hash)`.
+2. When the clean extraction completes, **Fusion converges old ↔ new for that event** using
+   the teacher-version master signal (A2/A3): the new `6ccfe70f` extraction wins as master;
+   the old extraction's now-orphaned nodes (those whose **only** provenance was this event
+   under the old hash) are tombstoned/repointed via `entity_merges` / `fact_merges`.
+3. Where an old node carries **other live provenance** (multi-event corroboration), only this
+   event's contribution is repointed — **never blind-delete a multi-source node** (the
+   over-merge failure mode: a hotel email wrongly attached to a person must not let one
+   event's repoint nuke an otherwise-corroborated node).
+
+This dependency is hard: **re-distill is unsafe until Fusion's cross-run / teacher-version
+master selection (E3) is live.** Until then a re-distill loop accretes. An interim cheaper
+option (Open Q): explicit **event-scoped supersede** — delete only the facts/entities whose
+provenance set is exactly `{this event}` under the old hash before re-enqueue — covers the
+single-provenance majority without the full fusion adjudicator.
+
+### B′5. Corpus-as-byproduct — one loop, three wins
+Every re-distill emits a clean `6ccfe70f` `distillation_trace`. A prompt-version-drift
+re-distill loop therefore **builds the student retrain corpus while it repairs the graph**
+(`scripts/build_retrain_corpus.py` consumes those traces). It subsumes the one-shot full
+re-distill: gradual, rate-limited, no nuke — graph repair **+** corpus **+** self-healing
+from a single engine. This is the durable answer to "is the corpus building?": it is, as a
+side effect of the gardener.
+
+### B′6. Cadence + cost + safety
+Rolling, rate-limited, autoscaler-aware, off-peak. Budget *N* events/hour against teacher
+capacity; order by `salience × staleness`. **Never big-bang the full backlog** — gradual
+migration is the point. Arena-scoped, dry-run → `--apply`, `redistill_runs` ledger for
+observability and rollback. Same operational shape as fusion/decay/autoscaler.
+
+---
+
 ## Part C — Ordering & how they combine
 
-Per arena, on schedule: **(1) fusion → (2) decay.** Fusion first so a master node absorbs
-its duplicates' provenance/salience *before* decay judges it (else a real node split across
-two weak dupes could wrongly decay out). Then decay ages + evicts the survivors.
+Per arena, on schedule: **(1) triage → re-distill the high-value stale-prompt set (async via
+the queue) → (2) fusion → (3) decay.** Re-distill is enqueued first so that by the time
+fusion runs, the clean counterpart exists for it to crown as master (else fusion has only
+stale renderings to choose between). Fusion then absorbs each master's duplicates'
+provenance/salience *before* decay judges it (else a real node split across two weak dupes
+could wrongly decay out). Then decay ages + evicts the survivors.
+
+*(Re-distill is asynchronous — it completes on the teacher's schedule — so in practice a
+node re-distilled in this pass is fused/decayed in the **next** per-arena pass, once its
+clean trace + extraction have landed. The ledger links the two.)*
 
 **This is what finally cures immortal pollution:**
 - 7B polluted node *with* a correct Qwen3.6 counterpart → **fused**, correct one as master,
   polluted demoted to alias / tombstoned.
+- stale-prompt node, *high-value*, *no* correct counterpart, source event present →
+  **re-distilled** through the clean teacher → new master extraction; old superseded via
+  fusion (B′4). The information is *recovered*, not lost.
 - 7B pure-junk node with *no* correct counterpart (numeric-ID-person, ungrounded) → born-low
   salience + no corroboration + never accessed → **decays out and is evicted**.
 
@@ -165,8 +259,15 @@ reset, but no longer the *only* path).
 - `relationships`: `+ salience REAL`, `+ last_accessed` (already has `weight`,
   `first/last_seen`).
 - new `fact_merges` audit (mirror `entity_merges` incl. `rollback_payload`).
-- new `fusion_runs` + `decay_runs` ledgers for observability.
+- new `fusion_runs` + `decay_runs` + `redistill_runs` ledgers for observability. `redistill_runs`:
+  `(id, arena, event_id, old_prompt_hash, new_prompt_hash, salience_at_trigger, enqueued_at,
+  completed_at, fused_at, mode)` — links a re-distill to its triggering node and to the fusion
+  that superseded the old extraction.
 - `/search` gains a `last_accessed = NOW()` bump on returned nodes (batched).
+- re-distill trigger needs provenance → prompt-version: either denormalize `system_prompt_hash`
+  onto `facts`/`entities` at write time (cheap filter), or join through
+  `distillation_traces(event_id → system_prompt_hash)` on the provenance event ids (no schema
+  change, costlier query). Prefer the join until the trigger volume justifies denormalizing.
 
 ## Part E — Rollout (each flag-gated, arena-scoped, dry-run-first, audited)
 
@@ -176,6 +277,13 @@ reset, but no longer the *only* path).
 3. **Fusion extension** — scored canonical selection (fix typo-crowning) + cross-run
    detection + fact fusion, dry-run → apply.
 4. **Online/continuous** — wire fusion+decay to run after distillation per arena.
+5. **Re-distill loop (Part B′)** — dry-run triage first (count stale-prompt nodes by
+   `system_prompt_hash` × salience bucket to size the work), then a **bounded `--apply` slice**
+   on one curated arena (re-enqueue + verify clean trace + verify fusion supersedes the old
+   extraction), then wire continuous. **Gated on step 3** (Fusion cross-run / teacher-version
+   master selection): until that's live, re-distill must use the interim **event-scoped
+   supersede** (B′4) or it accretes. Ships as `scripts/redistill.py` (dry-run default,
+   `--apply` gate, arena-scoped, `redistill_runs` ledger).
 
 ## Open questions
 - Half-life constants per category — needs a calibration pass against real arenas.
@@ -183,3 +291,9 @@ reset, but no longer the *only* path).
 - Directory authority source for canonical anchoring — HubSpot contacts? a curated table?
 - Interaction with the (still-open) source_id supersede mode — fusion partly subsumes it,
   but explicit supersede is cheaper for known-mutable sources.
+- **Re-distill supersedence before full fusion is live** — is event-scoped supersede (delete
+  only nodes whose provenance set is exactly `{this event}` under the old hash) a safe enough
+  interim, or do we hard-gate the loop on E3? Single-provenance nodes are the majority, but
+  the multi-provenance tail is where the over-merge risk concentrates.
+- **Re-distill prioritization** — pure `salience × staleness`, or weight toward the entities
+  behind known user-visible confabulations (Vickers/Boedecker) first?

package/packages/memory-engine-v2/compat/server.py

@@ -896,6 +896,8 @@ class GraphQueryRequest(BaseModel):
     entity_type: str | None = None
     name: str | None = None             # canonical_name (ILIKE)
     subject: str | None = None          # entity name OR canonical_name (facts.subject_entity)
+    subject_entity_id: str | None = None  # EXACT facts.subject_entity_id — strict, no name bleed
+    object_entity_id: str | None = None   # EXACT facts.object_entity_id
     predicate: str | None = None
     category: str | None = None         # facts.category
     from_name: str | None = None        # relationships.from_entity.canonical_name
@@ -944,9 +946,14 @@ async def list_entities(req: GraphQueryRequest):
 
 @app.post("/facts")
 async def list_facts(req: GraphQueryRequest):
-    """Filter facts by arena + optional category/predicate + optional
-    subject-entity name. Subject filter joins facts → entities via
-    subject_entity_id."""
+    """Filter facts by arena + optional category/predicate + subject.
+
+    PREFER `subject_entity_id` (exact id match) over `subject` (name ILIKE):
+    name matching bleeds one person's facts into another's answer when names
+    collide or fragment (the Will Vickers ⟵ Will Spencer confabulation — a
+    query resolved to one entity must NOT pull a same/similar-named entity's
+    facts). The name path is kept for back-compat callers that haven't resolved
+    an id yet, but entity-id is the strict, bleed-free path."""
     arenas = _resolve_arenas(req)
     conditions = ["f.arena = ANY(%s)"]
     params: list[Any] = [arenas]
@@ -956,7 +963,14 @@ async def list_facts(req: GraphQueryRequest):
     if req.predicate:
         conditions.append("f.predicate ILIKE %s")
         params.append(f"%{req.predicate}%")
-    if req.subject:
+    if req.subject_entity_id:
+        conditions.append("f.subject_entity_id = %s")
+        params.append(req.subject_entity_id)
+    if req.object_entity_id:
+        conditions.append("f.object_entity_id = %s")
+        params.append(req.object_entity_id)
+    # Name path: only when no exact id was given (back-compat / unresolved callers).
+    if req.subject and not req.subject_entity_id:
         conditions.append("EXISTS (SELECT 1 FROM entities e WHERE e.id = f.subject_entity_id AND (e.canonical_name ILIKE %s OR %s = ANY(e.aliases)))")
         params.extend([f"%{req.subject}%", req.subject])
     sql = f"""

package/packages/memory-engine-v2/docs/redistill-execution-plan-2026-06-22.md

@@ -0,0 +1,269 @@
+# Re-distill EXECUTION plan — pentatonic-team, #126 modality fix (2026-06-22)
+
+> **For review (Phil H) before any prod write.** This is the concrete, gated
+> execution plan for deploying the #126 distiller fix and re-distilling the
+> `pentatonic-team` graph. It supersedes the operational detail in
+> `redistill-plan-2026-06-21.md` where live findings differ (scale, deploy
+> mechanism, queue schema). The *why* and the guardrails there still hold.
+>
+> **Nothing here has been run except the outage fix in §1.** The irreversible
+> full delete (§4) is explicitly gated on the pilot audit (§3) + a human go.
+>
+> **⚠️ Three prerequisites added after review (§0) are BLOCKING** — without them
+> the re-distill would (a) re-introduce the same fabrications via the cascade
+> student, (b) leave the vector index inconsistent, and (c) starve every other
+> tenant's live ingest. Read §0 first.
+
+## Why (recap)
+
+A fact-by-fact audit found ~18.7% of distilled `pentatonic-team` facts were
+wrong — dominated by **modality collapse** (future/scheduled/planned content
+asserted as established fact), plus attribution errors and same-name
+conflations. The teacher prompt is fixed in **ai-agent-sdk PR #126** (TENSE &
+MODALITY / ATTRIBUTION FIDELITY / IDENTITY rules, in both `BATCH_SYSTEM_PROMPT`
+and the active `GUIDED_JSON_SYSTEM_PROMPT`). The prompt only governs *future*
+extractions, so the historical graph must be re-distilled to inherit the fix.
+
+## Live findings that change the 2026-06-21 runbook
+
+| Claim in 06-21 runbook | Live reality (verified 2026-06-22) |
+|---|---|
+| "~163k events", "434 facts" | **272,893 events · 483,097 facts · 291,797 rels** for `arena LIKE 'pentatonic-team%'`. The 434 was a 49-entity sample. |
+| Deploy = "the L4 box running extractor-async" | **The distiller runs on the DB box `i-0559922cf59ac6975`** (`pme-prod-us-east-1`), container `pme2-extractor-async`. The `seesa-distiller-bakeoff` box (`i-0d65…`) is **NOT** the host (empty, stopped since 06-10) — earlier handoff was wrong. Models are external: teacher `seesa-distiller-l40s` (`172.31.26.202:8005`, `qwen3.6-27b-fp8`), student `seesa-student-l4` (`172.31.29.121:8005`). |
+| `UPDATE distillation_queue ... WHERE arena LIKE` | **`distillation_queue` has NO `arena` column** (cols: id, event_id, enqueued_at, claimed_by, claimed_at, claim_expires_at, status, attempts, last_error, completed_at). Scope via a join to `events.arena` (see §3/§4). |
+| disk: "prior 30GB root outage — watch disk" | Root is now **485G, 412G free (16%)** — ample. |
+| "Merge #126 and deploy the new distiller" | The running container **may** build from a local copy `/opt/engine-v2/extractor-async/worker.py`, but the engine-deploy path also installs the SDK tarball **from S3 into `node_modules`** and builds from there — these have caused a *week of drift* before (editing a reference-only copy). **§2.0 makes verifying the real build context a hard step, not an assumption.** |
+
+## 0. BLOCKING prerequisites (added 2026-06-22 review)
+
+### 0.1 The re-distill MUST run TEACHER-ONLY (else it re-creates the bug)
+
+`CASCADE_ENABLED=true` is live in prod (the student→teacher cascade, #99). Under
+it, re-enqueued events flow **student-first**: ~75–80% are handled by the
+fine-tuned student, only gated/escalated events reach the teacher. **#126 fixes
+the *teacher* prompt only.** The deployed student is the `f1e0ff`-trained
+fine-tune — it *learned the modality-collapse behaviour from the old teacher* —
+so re-distilling through the live cascade would have the **student re-assert the
+same future-as-fact fabrications on the majority of events**, and the audit
+would not reach `<3%`.
+
+**Therefore, for the entire re-distill (pilot §3 + full §4):**
+
+```bash
+# turn the cascade OFF so every re-enqueued event goes to the #126 teacher
+aws ssm put-parameter --name /pme/prod-us-east-1/CASCADE_ENABLED --value false \
+  --type String --overwrite --region us-east-1
+# then redeploy/restart extractor-async so the env reaches the worker (§2),
+# and CONFIRM in the startup log: "cascade DISABLED" / no "student-primary" line.
+```
+
+Re-enable the cascade (`=true` + restart) only **after** §4 completes and the
+audit passes. While it's off, the teacher fleet carries 100% of distillation —
+factor that into §4.4 fleet sizing.
+
+**Forward note (not part of this run):** deploying #126 advances the teacher
+`prompt_hash`, which strands the live student (trained on `f1e0ff`). Once the
+cascade is re-enabled, watch the random-sample student↔teacher agreement for
+drift; the student will eventually want a refresh on #126 traces. Tracked
+separately from this re-distill.
+
+### 0.2 Vector index (Qdrant) must be reconciled — not just Postgres
+
+Hybrid retrieval is live (Qdrant 1.18.2, `evidence` collection). The §4 DELETE
+removes 483k facts from Postgres but **leaves their vectors orphaned in Qdrant**,
+and re-distilled facts are content-hash-distinct → *new* vectors. Net without
+reconciliation: search returns deleted facts + stale duplicates.
+
+- **Confirm** what the `evidence` collection is keyed on (fact id vs event id)
+  and how a PG fact delete maps to vector points (`vector_provenance` table).
+- **Purge** the arena's old vectors as part of §4.2 (delete the corresponding
+  Qdrant points / `vector_provenance` rows), and **verify** re-distilled facts
+  get re-embedded (the embedder lanes must keep up — `NV_EMBED_URL` interactive
+  + bulk).
+- The §4.1 snapshot + §Rollback are **Postgres-only**; add the vector state (see
+  §4.1 / Rollback below) or a rollback leaves Qdrant inconsistent with restored
+  PG.
+
+### 0.3 Re-enqueue must NOT starve live ingest (all tenants)
+
+`claim_next_batch` orders by `ORDER BY id`. Re-enqueued events are *old* → **low
+ids** → they would be claimed **ahead of** live ingest (high ids) for **every
+tenant**, stalling all forward memory ingest behind a multi-day job. Mitigate by
+**dripping the re-enqueue in batches** (§4.3) rather than flipping all 273k to
+`pending` at once — keep the pending re-distill backlog bounded (e.g. ≤ a few k)
+so live ingest interleaves. Monitor that non-`pentatonic-team` pending age
+doesn't climb.
+
+## 1. Outage already fixed (2026-06-22 ~08:26 UTC)
+
+`pme2-extractor-async` had been SIGKILL'd at 05:03 UTC (`OOMKilled=false` → a
+failed deploy attempt) and silently not draining for ~3.5h while forward ingest
+kept enqueuing — **no alert fired** (see Seesa SEE-189). Fixed: `docker start
+pme2-extractor-async`; reset 60 orphaned claims (`status='claimed' AND
+claim_expires_at < now()` → `pending`); stopped the wrongly-started bakeoff box.
+Verified draining (~60/min, 0 new failures, active claims with future expiry).
+**This restored the OLD prompt** (`prompt_hash=f1e0ff554f708d05`); §2 replaces it.
+
+## 2. Deploy #126 (prompt fix) + disable cascade
+
+SDK version bumped **0.10.18 → 0.10.19** (this PR) and tarball built
+(`ai-agent-sdk-0.10.19.tgz`, bundled `worker.py` carries the 4 fix markers).
+
+0. **Verify the real build context FIRST** (do not assume local-copy — this is
+   the "week of drift" failure mode). On `i-0559…`:
+   `docker inspect pme2-extractor-async --format '{{.Config.Image}}'` and read
+   the `extractor-async` service `build.context` in the *deployed* compose
+   (`/opt/engine-v2/...`). Confirm the file you are about to edit is the one that
+   image actually builds from. If the context resolves into
+   `node_modules/@pentatonic-ai/ai-agent-sdk/...` (the S3-tarball install), edit
+   THAT path (or replace the tarball), not a stray `/opt/engine-v2/extractor-async`
+   copy. Resolve Open-Q #1 (npm/S3 source of truth) here too.
+1. **Diff** (safety — don't silently revert any box-local change): copy the #126
+   `worker.py` to `/tmp/worker.new.py`, then `diff` it against the file the build
+   context actually uses (from step 0). Expect **only** the
+   TENSE/MODALITY/ATTRIBUTION/IDENTITY prompt additions. If other diffs appear,
+   STOP and reconcile.
+2. Replace that worker.py with the #126 version (keep a timestamped backup for
+   the Rollback worker-revert).
+3. **Set `CASCADE_ENABLED=false`** (§0.1) so the re-distill is teacher-only.
+4. `cd /opt/engine-v2 && sudo docker compose up -d --build extractor-async`.
+5. **Verify**: startup log prints a NEW `prompt_hash` (≠ `f1e0ff554f708d05`)
+   AND shows the cascade is OFF (no "cascade ENABLED — student-primary" line);
+   `grep -c MODALITY` in the running container's worker.py > 0; a few fresh
+   completions look sane.
+
+## 3. Pilot re-distill (~100 events) — GATE
+
+1. Pilot set: the known-bad source events (Will Vickers, Catherine Hayes, Johann
+   Boedecker, Katrin) + ~80 random `pentatonic-team` events.
+2. Scoped clear + re-enqueue (note the `events` join — no `arena` on the queue;
+   reset `attempts` so the rows are claim-eligible — eligibility is
+   `attempts < MAX_ATTEMPTS`):
+   ```sql
+   -- clear old facts for the pilot events. NB provenance_event_ids[1] catches
+   -- facts whose FIRST source is a pilot event; a fact corroborated by (but not
+   -- first-seen from) a pilot event is missed — acceptable for a pilot, exact
+   -- scope comes in §4's whole-arena delete.
+   DELETE FROM facts f
+    WHERE f.arena LIKE 'pentatonic-team%'
+      AND f.provenance_event_ids[1] = ANY(:pilot_event_ids);
+   -- re-enqueue (queue has no arena; scope by event_id set; reset attempts)
+   UPDATE distillation_queue
+      SET status='pending', claimed_by=NULL, claimed_at=NULL,
+          claim_expires_at=NULL, attempts=0
+    WHERE event_id = ANY(:pilot_event_ids);
+   ```
+   (Cascade is OFF per §0.1, so these run on the #126 teacher — the pilot
+   actually tests the fix, not the stale student.)
+3. Let the fleet drain. **Audit** the pilot entities with the validation harness:
+   modality/attribution rate should fall toward ~0; "attended/is" → "is scheduled
+   to / plans to" or dropped. Vickers "Board Observer" should be modal/absent.
+4. **🚦 GATE: report the audit; require an explicit human go before §4.**
+
+## 4. Full re-distill (only after the pilot passes + go)
+
+1. **Snapshot** (rollback point), off-box — Postgres AND vector state:
+   ```bash
+   pg_dump ... -t facts -t entities -t relationships -t vector_provenance \
+     --where "arena LIKE 'pentatonic-team%'" > pt_graph_pre_redistill_2026-06-22.sql
+   # also snapshot/record the Qdrant evidence points for the arena (or a full
+   # collection snapshot) so §Rollback can restore the index, not just PG.
+   ```
+2. Clear (keep entities — Fusion + projection sweeps rebuild downstream) +
+   purge the orphaned vectors (§0.2):
+   ```sql
+   DELETE FROM facts         WHERE arena LIKE 'pentatonic-team%';
+   DELETE FROM relationships WHERE arena LIKE 'pentatonic-team%';
+   -- + delete the corresponding Qdrant points and vector_provenance rows for
+   --   the arena (see §0.2 — confirm the keying first).
+   ```
+   Note: entities are retained, so old-prompt **junk entities** the new prompt
+   won't re-create (e.g. the "Pentatonic GmbH" footer affiliation) will persist
+   until Fusion decay/eviction clears them — flag for a post-run entity sweep if
+   the audit still spots them.
+3. Re-enqueue in **bounded batches** (§0.3 — do NOT flip all 273k at once, or
+   live ingest for all tenants starves behind the low-id backlog). First confirm
+   completeness, then drip:
+   ```sql
+   -- completeness check: how many arena events have a queue row vs not?
+   SELECT count(*) FILTER (WHERE q.id IS NOT NULL) AS have_row,
+          count(*) FILTER (WHERE q.id IS NULL)     AS missing
+     FROM events e LEFT JOIN distillation_queue q ON q.event_id = e.id
+    WHERE e.arena LIKE 'pentatonic-team%';
+   -- for events WITH a row: re-enqueue a batch (reset attempts), repeat as the
+   -- pending re-distill backlog drains below a cap (e.g. 3k):
+   UPDATE distillation_queue q
+      SET status='pending', claimed_by=NULL, claimed_at=NULL,
+          claim_expires_at=NULL, attempts=0
+     FROM events e
+    WHERE e.id = q.event_id AND e.arena LIKE 'pentatonic-team%'
+      AND q.event_id IN (:next_batch_event_ids);
+   -- for events MISSING a row (pruned post-done): re-insert from events.
+   INSERT INTO distillation_queue (event_id, status, enqueued_at, attempts)
+   SELECT e.id, 'pending', now(), 0 FROM events e
+    WHERE e.arena LIKE 'pentatonic-team%'
+      AND NOT EXISTS (SELECT 1 FROM distillation_queue q WHERE q.event_id = e.id)
+      AND e.id IN (:next_batch_event_ids);
+   ```
+   **Assert** the total re-enqueued (UPDATE + INSERT) across all batches == the
+   arena event count (~272,893) before declaring the enqueue complete.
+4. **Scale the fleet** for ~273k events, **teacher-only** (cascade off → the
+   teacher carries 100%, not the usual ~25%). Single-worker ~60/min ⇒ ~76h;
+   `extractor-async-2/-3` + `distiller-autoscale.sh` help, BUT **g6e capacity is
+   currently severe** — this week we could not launch *or restart a stopped*
+   g6e. So: do **not** let the autoscaler scale the fleet to zero mid-run (a
+   stopped box may not come back); treat the currently-running L40S boxes as the
+   ceiling and the 76h as optimistic. Monitor `distillation_queue` depth → 0,
+   GPU, disk, and the `failed`/`ReadTimeout` rate (teacher ~20–49s/call).
+
+## 5. Downstream reconciliation
+
+1. **Fusion Drive** re-run for the arena (de-dup; run AFTER extraction settles) —
+   `backfill_entity_reconciliation.py` / `fusion-drive-*.sh`.
+2. **Projection re-fold** (Seesa side): org/person projection sweeps re-fold from
+   the corrected graph — the Seesa **SEE-184** watermark sweep (now live, 03:45
+   UTC) detects the advanced `graphAsOf` and marks projections stale → SEE-168
+   refresh re-folds. Worker D1 read-models inherit the heal.
+3. **Unblocks Seesa SEE-183** — the content-sensitivity retag (forward detection
+   already live) was deliberately held until this re-distill completes, to avoid
+   piling re-emitted events onto the shared queue mid-run.
+4. **Re-enable the cascade** (§0.1): `CASCADE_ENABLED=true` + restart
+   extractor-async; confirm "cascade ENABLED — student-primary" returns. Then
+   watch student↔teacher agreement (the student is now `f1e0ff`-stale vs the
+   #126 teacher — schedule a student refresh if drift shows).
+
+## Validation gate (before declaring done)
+Re-run the audit harness over a fresh 30–50 entity sample → **bad-fact rate
+< 3%** (from ~18.7%). Spot the canonical failures: Vickers "Board Observer"
+(modal/absent), Catherine "will send / to organise" (→ commitments), Matvii
+"Pentatonic GmbH" footer affiliation (gone), Sebastian conflation (split/keyed).
+
+## Rollback
+Restore `facts`/`relationships`/`vector_provenance` for `pentatonic-team%` from
+the §4.1 snapshot AND restore the Qdrant `evidence` points (new-prompt
+extractions are content-hash-distinct, so a restore is clean — but PG and Qdrant
+must be rolled back *together* or search desyncs). Worker revert = redeploy the
+prior worker.py (the §2.2 backup) + rebuild. Cascade: set `CASCADE_ENABLED=true`
+back if it was changed.
+
+## Open questions for Phil
+1. **Durable SDK release / source of truth.** Resolve in §2.0: is the deployed
+   worker built from a local copy, or from the S3/npm SDK install? If the
+   lockfile/`npm install` path is authoritative, a future redeploy reverts #126
+   unless 0.10.19 is published there + the `/opt/engine-v2` pin bumped. (S3
+   `sdk/` has tarballs to 0.10.18; the lockfile root version was stale at 0.10.1
+   — confirm the real install path before relying on either.)
+2. **Fleet sizing** for the **teacher-only** 273k run — given g6e capacity is
+   currently unreliable (can't restart stopped boxes), is the running L40S fleet
+   enough, or do we need reserved/alt-region capacity (or to accept a longer
+   wall-clock)?
+3. ~~Re-enqueue completeness~~ → **resolved in §4.3** (completeness check +
+   INSERT-from-events fallback + count assertion).
+4. Run **off-peak / batched**? §0.3 + §4.3 make it batched to protect live
+   ingest; still worth starting off-peak given the multi-day duration.
+
+## Authorization note (Seesa-side / Claude Code)
+The auto-mode Bash classifier blocks these shared-prod writes (SSM/ec2/psql) even
+with verbal authority; they must run via the `!` prefix or a settings permission
+rule. Read-only SSM queries are fine. The full DELETE (§4) will be kept behind an
+explicit human go regardless of any rule.

package/packages/memory-engine-v2/docs/redistill-plan-2026-06-21.md

@@ -0,0 +1,101 @@
+# Re-distill plan — pentatonic-team, post modality/attribution fix (2026-06-21)
+
+**Why.** A fact-by-fact audit of `org_model` (49 entities, 434 facts) found **~18.7%
+of distilled facts wrong**, dominated by *modality collapse* (future/scheduled/
+planned content asserted as established fact), plus attribution errors (unauthored
+docs → person, attendee → organiser, org activity → person) and a few same-name
+conflations. The teacher prompt is fixed in **ai-agent-sdk PR #126** (TENSE &
+MODALITY / ATTRIBUTION FIDELITY / IDENTITY rules). The prompt only governs
+*future* extractions, so the historical graph must be **re-distilled** to inherit
+the fix.
+
+**Already done (2026-06-21):** a targeted cleanup retracted **229** facts whose
+provenance source event is dated in the future (`emitted_at > now()`) and whose
+category is established (`state/mention/decision`), scoped to `pentatonic-team`.
+Future *commitments* (29) were spared; the frozen `pip-agents` legacy was
+untouched. ~11 "attends standup"-class facts (future *content* but a past-dated
+source event) are NOT deterministically catchable and remain for this re-distill.
+
+---
+
+## Scope & guardrails
+- **Arena: `pentatonic-team%` ONLY.** `org_model` is multi-tenant — `pip-agents`
+  (~246k facts) is LEGACY/frozen and must never be touched. Every statement here
+  carries `WHERE arena LIKE 'pentatonic-team%'`.
+- **Idempotency caveat (load-bearing).** `worker.py` IDs entities/facts/rels by
+  content-hash, so re-running the *same* prompt converges. But the **prompt
+  changed** (`SYSTEM_PROMPT_HASH` is new), so re-extraction yields *different*
+  facts with *different* IDs — the old wrong facts will **coexist** unless
+  cleared. ⇒ the re-distill MUST delete the existing pentatonic-team facts for
+  each re-processed event before/with re-extraction.
+
+## Prerequisites
+1. **Merge PR #126** and **deploy the new distiller** to the extractor fleet
+   (the L4 box running `extractor-async/worker.py`). Confirm the running worker's
+   `SYSTEM_PROMPT_HASH` matches the new prompt (it's logged on the trace line).
+2. **Snapshot** (rollback point):
+   `pg_dump … -t facts -t entities -t relationships --where "arena LIKE 'pentatonic-team%'"`
+   (or a filtered `COPY … TO` per table). Store off-box.
+
+## Procedure
+
+### Stage 0 — pilot (≈100 events incl. the known-bad ones)
+1. Pick a pilot set: the Will Vickers / Catherine Hayes / Johann / Katrin source
+   events + a random ~80 pentatonic-team events.
+2. Delete existing facts for those events (scoped), then re-enqueue:
+   ```sql
+   -- clear old facts for the pilot events
+   DELETE FROM facts f
+    WHERE f.arena LIKE 'pentatonic-team%'
+      AND f.provenance_event_ids[1] = ANY(:pilot_event_ids);
+   -- re-enqueue them
+   UPDATE distillation_queue
+      SET status='pending', claimed_by=NULL, claim_expires_at=NULL
+    WHERE arena LIKE 'pentatonic-team%'
+      AND event_id = ANY(:pilot_event_ids);
+   ```
+3. Let the fleet drain the queue. **Validate**: re-run the audit harness over the
+   pilot entities; the modality/attribution rate should fall toward ~0. Eyeball
+   the Vickers/Katrin facts — "attended/is" should become "is scheduled to /
+   plans to" (or drop).
+
+### Stage 1 — full pentatonic-team re-distill (only if the pilot passes)
+1. Clear pentatonic-team facts (keep entities; Fusion + the projection sweep
+   rebuild downstream). Relationships likewise.
+   ```sql
+   DELETE FROM facts         WHERE arena LIKE 'pentatonic-team%';
+   DELETE FROM relationships WHERE arena LIKE 'pentatonic-team%';
+   ```
+2. Re-enqueue every pentatonic-team event:
+   ```sql
+   UPDATE distillation_queue
+      SET status='pending', claimed_by=NULL, claim_expires_at=NULL
+    WHERE arena LIKE 'pentatonic-team%';
+   ```
+   (If queue rows were pruned post-done, re-insert from `events` for the arena.)
+3. Scale the fleet for the backlog; monitor `distillation_queue` depth until 0.
+
+### Stage 2 — downstream reconciliation
+1. **Fusion Drive** re-run for the arena (consolidate the freshly-extracted
+   nodes/facts) — it's a de-duplicator, so run AFTER extraction settles.
+   `backfill_entity_reconciliation.py` covers the entity side.
+2. **Projection re-fold** (Seesa side): the org/person projection sweeps re-fold
+   from the corrected graph (SEE-168 nightly sweep does this automatically; or
+   trigger manually). The Worker D1 read-models then inherit the heal.
+
+## Validation (gate before declaring done)
+- Re-run the audit harness over a fresh 30–50 entity sample → **bad-fact rate
+  target < 3%** (from ~18.7%).
+- Spot the canonical failures: Vickers "Board Observer" (should be modal/absent),
+  Catherine's "will send / to organise" (→ commitments), Matvii's "Pentatonic
+  GmbH" footer affiliation (gone), Sebastian conflation (split or correctly keyed).
+
+## Rollback
+Restore `facts`/`relationships` for `pentatonic-team%` from the Stage-0 snapshot;
+the new-prompt extractions are content-hash-distinct so a restore is clean.
+
+## Cost / time
+~163k pentatonic-team events × 7B extraction on the L4 fleet. Bounded by fleet
+size × per-batch latency; run off-peak, monitor GPU/disk (prior outage: v2 wrote
+to a 30 GB root — watch disk). Pilot first to estimate throughput before the full
+run.