@pentatonic-ai/ai-agent-sdk 0.10.17 → 0.10.19

package/dist/index.cjs

@@ -878,7 +878,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
 }
 
 // src/telemetry.js
-var VERSION = "0.10.17";
+var VERSION = "0.10.19";
 var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
 function machineId() {
   const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";

package/dist/index.js

@@ -847,7 +847,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
 }
 
 // src/telemetry.js
-var VERSION = "0.10.17";
+var VERSION = "0.10.19";
 var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
 function machineId() {
   const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pentatonic-ai/ai-agent-sdk",
-  "version": "0.10.17",
+  "version": "0.10.19",
   "description": "TES SDK — LLM observability and lifecycle tracking via Pentatonic Thing Event System. Track token usage, tool calls, and conversations. Manage things through event-sourced lifecycle stages with AI enrichment and vector search.",
   "type": "module",
   "main": "./dist/index.cjs",

package/packages/memory-engine-v2/docs/redistill-execution-plan-2026-06-22.md

@@ -0,0 +1,269 @@
+# Re-distill EXECUTION plan — pentatonic-team, #126 modality fix (2026-06-22)
+
+> **For review (Phil H) before any prod write.** This is the concrete, gated
+> execution plan for deploying the #126 distiller fix and re-distilling the
+> `pentatonic-team` graph. It supersedes the operational detail in
+> `redistill-plan-2026-06-21.md` where live findings differ (scale, deploy
+> mechanism, queue schema). The *why* and the guardrails there still hold.
+>
+> **Nothing here has been run except the outage fix in §1.** The irreversible
+> full delete (§4) is explicitly gated on the pilot audit (§3) + a human go.
+>
+> **⚠️ Three prerequisites added after review (§0) are BLOCKING** — without them
+> the re-distill would (a) re-introduce the same fabrications via the cascade
+> student, (b) leave the vector index inconsistent, and (c) starve every other
+> tenant's live ingest. Read §0 first.
+
+## Why (recap)
+
+A fact-by-fact audit found ~18.7% of distilled `pentatonic-team` facts were
+wrong — dominated by **modality collapse** (future/scheduled/planned content
+asserted as established fact), plus attribution errors and same-name
+conflations. The teacher prompt is fixed in **ai-agent-sdk PR #126** (TENSE &
+MODALITY / ATTRIBUTION FIDELITY / IDENTITY rules, in both `BATCH_SYSTEM_PROMPT`
+and the active `GUIDED_JSON_SYSTEM_PROMPT`). The prompt only governs *future*
+extractions, so the historical graph must be re-distilled to inherit the fix.
+
+## Live findings that change the 2026-06-21 runbook
+
+| Claim in 06-21 runbook | Live reality (verified 2026-06-22) |
+|---|---|
+| "~163k events", "434 facts" | **272,893 events · 483,097 facts · 291,797 rels** for `arena LIKE 'pentatonic-team%'`. The 434 was a 49-entity sample. |
+| Deploy = "the L4 box running extractor-async" | **The distiller runs on the DB box `i-0559922cf59ac6975`** (`pme-prod-us-east-1`), container `pme2-extractor-async`. The `seesa-distiller-bakeoff` box (`i-0d65…`) is **NOT** the host (empty, stopped since 06-10) — earlier handoff was wrong. Models are external: teacher `seesa-distiller-l40s` (`172.31.26.202:8005`, `qwen3.6-27b-fp8`), student `seesa-student-l4` (`172.31.29.121:8005`). |
+| `UPDATE distillation_queue ... WHERE arena LIKE` | **`distillation_queue` has NO `arena` column** (cols: id, event_id, enqueued_at, claimed_by, claimed_at, claim_expires_at, status, attempts, last_error, completed_at). Scope via a join to `events.arena` (see §3/§4). |
+| disk: "prior 30GB root outage — watch disk" | Root is now **485G, 412G free (16%)** — ample. |
+| "Merge #126 and deploy the new distiller" | The running container **may** build from a local copy `/opt/engine-v2/extractor-async/worker.py`, but the engine-deploy path also installs the SDK tarball **from S3 into `node_modules`** and builds from there — these have caused a *week of drift* before (editing a reference-only copy). **§2.0 makes verifying the real build context a hard step, not an assumption.** |
+
+## 0. BLOCKING prerequisites (added 2026-06-22 review)
+
+### 0.1 The re-distill MUST run TEACHER-ONLY (else it re-creates the bug)
+
+`CASCADE_ENABLED=true` is live in prod (the student→teacher cascade, #99). Under
+it, re-enqueued events flow **student-first**: ~75–80% are handled by the
+fine-tuned student, only gated/escalated events reach the teacher. **#126 fixes
+the *teacher* prompt only.** The deployed student is the `f1e0ff`-trained
+fine-tune — it *learned the modality-collapse behaviour from the old teacher* —
+so re-distilling through the live cascade would have the **student re-assert the
+same future-as-fact fabrications on the majority of events**, and the audit
+would not reach `<3%`.
+
+**Therefore, for the entire re-distill (pilot §3 + full §4):**
+
+```bash
+# turn the cascade OFF so every re-enqueued event goes to the #126 teacher
+aws ssm put-parameter --name /pme/prod-us-east-1/CASCADE_ENABLED --value false \
+  --type String --overwrite --region us-east-1
+# then redeploy/restart extractor-async so the env reaches the worker (§2),
+# and CONFIRM in the startup log: "cascade DISABLED" / no "student-primary" line.
+```
+
+Re-enable the cascade (`=true` + restart) only **after** §4 completes and the
+audit passes. While it's off, the teacher fleet carries 100% of distillation —
+factor that into §4.4 fleet sizing.
+
+**Forward note (not part of this run):** deploying #126 advances the teacher
+`prompt_hash`, which strands the live student (trained on `f1e0ff`). Once the
+cascade is re-enabled, watch the random-sample student↔teacher agreement for
+drift; the student will eventually want a refresh on #126 traces. Tracked
+separately from this re-distill.
+
+### 0.2 Vector index (Qdrant) must be reconciled — not just Postgres
+
+Hybrid retrieval is live (Qdrant 1.18.2, `evidence` collection). The §4 DELETE
+removes 483k facts from Postgres but **leaves their vectors orphaned in Qdrant**,
+and re-distilled facts are content-hash-distinct → *new* vectors. Net without
+reconciliation: search returns deleted facts + stale duplicates.
+
+- **Confirm** what the `evidence` collection is keyed on (fact id vs event id)
+  and how a PG fact delete maps to vector points (`vector_provenance` table).
+- **Purge** the arena's old vectors as part of §4.2 (delete the corresponding
+  Qdrant points / `vector_provenance` rows), and **verify** re-distilled facts
+  get re-embedded (the embedder lanes must keep up — `NV_EMBED_URL` interactive
+  + bulk).
+- The §4.1 snapshot + §Rollback are **Postgres-only**; add the vector state (see
+  §4.1 / Rollback below) or a rollback leaves Qdrant inconsistent with restored
+  PG.
+
+### 0.3 Re-enqueue must NOT starve live ingest (all tenants)
+
+`claim_next_batch` orders by `ORDER BY id`. Re-enqueued events are *old* → **low
+ids** → they would be claimed **ahead of** live ingest (high ids) for **every
+tenant**, stalling all forward memory ingest behind a multi-day job. Mitigate by
+**dripping the re-enqueue in batches** (§4.3) rather than flipping all 273k to
+`pending` at once — keep the pending re-distill backlog bounded (e.g. ≤ a few k)
+so live ingest interleaves. Monitor that non-`pentatonic-team` pending age
+doesn't climb.
+
+## 1. Outage already fixed (2026-06-22 ~08:26 UTC)
+
+`pme2-extractor-async` had been SIGKILL'd at 05:03 UTC (`OOMKilled=false` → a
+failed deploy attempt) and silently not draining for ~3.5h while forward ingest
+kept enqueuing — **no alert fired** (see Seesa SEE-189). Fixed: `docker start
+pme2-extractor-async`; reset 60 orphaned claims (`status='claimed' AND
+claim_expires_at < now()` → `pending`); stopped the wrongly-started bakeoff box.
+Verified draining (~60/min, 0 new failures, active claims with future expiry).
+**This restored the OLD prompt** (`prompt_hash=f1e0ff554f708d05`); §2 replaces it.
+
+## 2. Deploy #126 (prompt fix) + disable cascade
+
+SDK version bumped **0.10.18 → 0.10.19** (this PR) and tarball built
+(`ai-agent-sdk-0.10.19.tgz`, bundled `worker.py` carries the 4 fix markers).
+
+0. **Verify the real build context FIRST** (do not assume local-copy — this is
+   the "week of drift" failure mode). On `i-0559…`:
+   `docker inspect pme2-extractor-async --format '{{.Config.Image}}'` and read
+   the `extractor-async` service `build.context` in the *deployed* compose
+   (`/opt/engine-v2/...`). Confirm the file you are about to edit is the one that
+   image actually builds from. If the context resolves into
+   `node_modules/@pentatonic-ai/ai-agent-sdk/...` (the S3-tarball install), edit
+   THAT path (or replace the tarball), not a stray `/opt/engine-v2/extractor-async`
+   copy. Resolve Open-Q #1 (npm/S3 source of truth) here too.
+1. **Diff** (safety — don't silently revert any box-local change): copy the #126
+   `worker.py` to `/tmp/worker.new.py`, then `diff` it against the file the build
+   context actually uses (from step 0). Expect **only** the
+   TENSE/MODALITY/ATTRIBUTION/IDENTITY prompt additions. If other diffs appear,
+   STOP and reconcile.
+2. Replace that worker.py with the #126 version (keep a timestamped backup for
+   the Rollback worker-revert).
+3. **Set `CASCADE_ENABLED=false`** (§0.1) so the re-distill is teacher-only.
+4. `cd /opt/engine-v2 && sudo docker compose up -d --build extractor-async`.
+5. **Verify**: startup log prints a NEW `prompt_hash` (≠ `f1e0ff554f708d05`)
+   AND shows the cascade is OFF (no "cascade ENABLED — student-primary" line);
+   `grep -c MODALITY` in the running container's worker.py > 0; a few fresh
+   completions look sane.
+
+## 3. Pilot re-distill (~100 events) — GATE
+
+1. Pilot set: the known-bad source events (Will Vickers, Catherine Hayes, Johann
+   Boedecker, Katrin) + ~80 random `pentatonic-team` events.
+2. Scoped clear + re-enqueue (note the `events` join — no `arena` on the queue;
+   reset `attempts` so the rows are claim-eligible — eligibility is
+   `attempts < MAX_ATTEMPTS`):
+   ```sql
+   -- clear old facts for the pilot events. NB provenance_event_ids[1] catches
+   -- facts whose FIRST source is a pilot event; a fact corroborated by (but not
+   -- first-seen from) a pilot event is missed — acceptable for a pilot, exact
+   -- scope comes in §4's whole-arena delete.
+   DELETE FROM facts f
+    WHERE f.arena LIKE 'pentatonic-team%'
+      AND f.provenance_event_ids[1] = ANY(:pilot_event_ids);
+   -- re-enqueue (queue has no arena; scope by event_id set; reset attempts)
+   UPDATE distillation_queue
+      SET status='pending', claimed_by=NULL, claimed_at=NULL,
+          claim_expires_at=NULL, attempts=0
+    WHERE event_id = ANY(:pilot_event_ids);
+   ```
+   (Cascade is OFF per §0.1, so these run on the #126 teacher — the pilot
+   actually tests the fix, not the stale student.)
+3. Let the fleet drain. **Audit** the pilot entities with the validation harness:
+   modality/attribution rate should fall toward ~0; "attended/is" → "is scheduled
+   to / plans to" or dropped. Vickers "Board Observer" should be modal/absent.
+4. **🚦 GATE: report the audit; require an explicit human go before §4.**
+
+## 4. Full re-distill (only after the pilot passes + go)
+
+1. **Snapshot** (rollback point), off-box — Postgres AND vector state:
+   ```bash
+   pg_dump ... -t facts -t entities -t relationships -t vector_provenance \
+     --where "arena LIKE 'pentatonic-team%'" > pt_graph_pre_redistill_2026-06-22.sql
+   # also snapshot/record the Qdrant evidence points for the arena (or a full
+   # collection snapshot) so §Rollback can restore the index, not just PG.
+   ```
+2. Clear (keep entities — Fusion + projection sweeps rebuild downstream) +
+   purge the orphaned vectors (§0.2):
+   ```sql
+   DELETE FROM facts         WHERE arena LIKE 'pentatonic-team%';
+   DELETE FROM relationships WHERE arena LIKE 'pentatonic-team%';
+   -- + delete the corresponding Qdrant points and vector_provenance rows for
+   --   the arena (see §0.2 — confirm the keying first).
+   ```
+   Note: entities are retained, so old-prompt **junk entities** the new prompt
+   won't re-create (e.g. the "Pentatonic GmbH" footer affiliation) will persist
+   until Fusion decay/eviction clears them — flag for a post-run entity sweep if
+   the audit still spots them.
+3. Re-enqueue in **bounded batches** (§0.3 — do NOT flip all 273k at once, or
+   live ingest for all tenants starves behind the low-id backlog). First confirm
+   completeness, then drip:
+   ```sql
+   -- completeness check: how many arena events have a queue row vs not?
+   SELECT count(*) FILTER (WHERE q.id IS NOT NULL) AS have_row,
+          count(*) FILTER (WHERE q.id IS NULL)     AS missing
+     FROM events e LEFT JOIN distillation_queue q ON q.event_id = e.id
+    WHERE e.arena LIKE 'pentatonic-team%';
+   -- for events WITH a row: re-enqueue a batch (reset attempts), repeat as the
+   -- pending re-distill backlog drains below a cap (e.g. 3k):
+   UPDATE distillation_queue q
+      SET status='pending', claimed_by=NULL, claimed_at=NULL,
+          claim_expires_at=NULL, attempts=0
+     FROM events e
+    WHERE e.id = q.event_id AND e.arena LIKE 'pentatonic-team%'
+      AND q.event_id IN (:next_batch_event_ids);
+   -- for events MISSING a row (pruned post-done): re-insert from events.
+   INSERT INTO distillation_queue (event_id, status, enqueued_at, attempts)
+   SELECT e.id, 'pending', now(), 0 FROM events e
+    WHERE e.arena LIKE 'pentatonic-team%'
+      AND NOT EXISTS (SELECT 1 FROM distillation_queue q WHERE q.event_id = e.id)
+      AND e.id IN (:next_batch_event_ids);
+   ```
+   **Assert** the total re-enqueued (UPDATE + INSERT) across all batches == the
+   arena event count (~272,893) before declaring the enqueue complete.
+4. **Scale the fleet** for ~273k events, **teacher-only** (cascade off → the
+   teacher carries 100%, not the usual ~25%). Single-worker ~60/min ⇒ ~76h;
+   `extractor-async-2/-3` + `distiller-autoscale.sh` help, BUT **g6e capacity is
+   currently severe** — this week we could not launch *or restart a stopped*
+   g6e. So: do **not** let the autoscaler scale the fleet to zero mid-run (a
+   stopped box may not come back); treat the currently-running L40S boxes as the
+   ceiling and the 76h as optimistic. Monitor `distillation_queue` depth → 0,
+   GPU, disk, and the `failed`/`ReadTimeout` rate (teacher ~20–49s/call).
+
+## 5. Downstream reconciliation
+
+1. **Fusion Drive** re-run for the arena (de-dup; run AFTER extraction settles) —
+   `backfill_entity_reconciliation.py` / `fusion-drive-*.sh`.
+2. **Projection re-fold** (Seesa side): org/person projection sweeps re-fold from
+   the corrected graph — the Seesa **SEE-184** watermark sweep (now live, 03:45
+   UTC) detects the advanced `graphAsOf` and marks projections stale → SEE-168
+   refresh re-folds. Worker D1 read-models inherit the heal.
+3. **Unblocks Seesa SEE-183** — the content-sensitivity retag (forward detection
+   already live) was deliberately held until this re-distill completes, to avoid
+   piling re-emitted events onto the shared queue mid-run.
+4. **Re-enable the cascade** (§0.1): `CASCADE_ENABLED=true` + restart
+   extractor-async; confirm "cascade ENABLED — student-primary" returns. Then
+   watch student↔teacher agreement (the student is now `f1e0ff`-stale vs the
+   #126 teacher — schedule a student refresh if drift shows).
+
+## Validation gate (before declaring done)
+Re-run the audit harness over a fresh 30–50 entity sample → **bad-fact rate
+< 3%** (from ~18.7%). Spot the canonical failures: Vickers "Board Observer"
+(modal/absent), Catherine "will send / to organise" (→ commitments), Matvii
+"Pentatonic GmbH" footer affiliation (gone), Sebastian conflation (split/keyed).
+
+## Rollback
+Restore `facts`/`relationships`/`vector_provenance` for `pentatonic-team%` from
+the §4.1 snapshot AND restore the Qdrant `evidence` points (new-prompt
+extractions are content-hash-distinct, so a restore is clean — but PG and Qdrant
+must be rolled back *together* or search desyncs). Worker revert = redeploy the
+prior worker.py (the §2.2 backup) + rebuild. Cascade: set `CASCADE_ENABLED=true`
+back if it was changed.
+
+## Open questions for Phil
+1. **Durable SDK release / source of truth.** Resolve in §2.0: is the deployed
+   worker built from a local copy, or from the S3/npm SDK install? If the
+   lockfile/`npm install` path is authoritative, a future redeploy reverts #126
+   unless 0.10.19 is published there + the `/opt/engine-v2` pin bumped. (S3
+   `sdk/` has tarballs to 0.10.18; the lockfile root version was stale at 0.10.1
+   — confirm the real install path before relying on either.)
+2. **Fleet sizing** for the **teacher-only** 273k run — given g6e capacity is
+   currently unreliable (can't restart stopped boxes), is the running L40S fleet
+   enough, or do we need reserved/alt-region capacity (or to accept a longer
+   wall-clock)?
+3. ~~Re-enqueue completeness~~ → **resolved in §4.3** (completeness check +
+   INSERT-from-events fallback + count assertion).
+4. Run **off-peak / batched**? §0.3 + §4.3 make it batched to protect live
+   ingest; still worth starting off-peak given the multi-day duration.
+
+## Authorization note (Seesa-side / Claude Code)
+The auto-mode Bash classifier blocks these shared-prod writes (SSM/ec2/psql) even
+with verbal authority; they must run via the `!` prefix or a settings permission
+rule. Read-only SSM queries are fine. The full DELETE (§4) will be kept behind an
+explicit human go regardless of any rule.

package/packages/memory-engine-v2/docs/redistill-plan-2026-06-21.md

@@ -0,0 +1,101 @@
+# Re-distill plan — pentatonic-team, post modality/attribution fix (2026-06-21)
+
+**Why.** A fact-by-fact audit of `org_model` (49 entities, 434 facts) found **~18.7%
+of distilled facts wrong**, dominated by *modality collapse* (future/scheduled/
+planned content asserted as established fact), plus attribution errors (unauthored
+docs → person, attendee → organiser, org activity → person) and a few same-name
+conflations. The teacher prompt is fixed in **ai-agent-sdk PR #126** (TENSE &
+MODALITY / ATTRIBUTION FIDELITY / IDENTITY rules). The prompt only governs
+*future* extractions, so the historical graph must be **re-distilled** to inherit
+the fix.
+
+**Already done (2026-06-21):** a targeted cleanup retracted **229** facts whose
+provenance source event is dated in the future (`emitted_at > now()`) and whose
+category is established (`state/mention/decision`), scoped to `pentatonic-team`.
+Future *commitments* (29) were spared; the frozen `pip-agents` legacy was
+untouched. ~11 "attends standup"-class facts (future *content* but a past-dated
+source event) are NOT deterministically catchable and remain for this re-distill.
+
+---
+
+## Scope & guardrails
+- **Arena: `pentatonic-team%` ONLY.** `org_model` is multi-tenant — `pip-agents`
+  (~246k facts) is LEGACY/frozen and must never be touched. Every statement here
+  carries `WHERE arena LIKE 'pentatonic-team%'`.
+- **Idempotency caveat (load-bearing).** `worker.py` IDs entities/facts/rels by
+  content-hash, so re-running the *same* prompt converges. But the **prompt
+  changed** (`SYSTEM_PROMPT_HASH` is new), so re-extraction yields *different*
+  facts with *different* IDs — the old wrong facts will **coexist** unless
+  cleared. ⇒ the re-distill MUST delete the existing pentatonic-team facts for
+  each re-processed event before/with re-extraction.
+
+## Prerequisites
+1. **Merge PR #126** and **deploy the new distiller** to the extractor fleet
+   (the L4 box running `extractor-async/worker.py`). Confirm the running worker's
+   `SYSTEM_PROMPT_HASH` matches the new prompt (it's logged on the trace line).
+2. **Snapshot** (rollback point):
+   `pg_dump … -t facts -t entities -t relationships --where "arena LIKE 'pentatonic-team%'"`
+   (or a filtered `COPY … TO` per table). Store off-box.
+
+## Procedure
+
+### Stage 0 — pilot (≈100 events incl. the known-bad ones)
+1. Pick a pilot set: the Will Vickers / Catherine Hayes / Johann / Katrin source
+   events + a random ~80 pentatonic-team events.
+2. Delete existing facts for those events (scoped), then re-enqueue:
+   ```sql
+   -- clear old facts for the pilot events
+   DELETE FROM facts f
+    WHERE f.arena LIKE 'pentatonic-team%'
+      AND f.provenance_event_ids[1] = ANY(:pilot_event_ids);
+   -- re-enqueue them
+   UPDATE distillation_queue
+      SET status='pending', claimed_by=NULL, claim_expires_at=NULL
+    WHERE arena LIKE 'pentatonic-team%'
+      AND event_id = ANY(:pilot_event_ids);
+   ```
+3. Let the fleet drain the queue. **Validate**: re-run the audit harness over the
+   pilot entities; the modality/attribution rate should fall toward ~0. Eyeball
+   the Vickers/Katrin facts — "attended/is" should become "is scheduled to /
+   plans to" (or drop).
+
+### Stage 1 — full pentatonic-team re-distill (only if the pilot passes)
+1. Clear pentatonic-team facts (keep entities; Fusion + the projection sweep
+   rebuild downstream). Relationships likewise.
+   ```sql
+   DELETE FROM facts         WHERE arena LIKE 'pentatonic-team%';
+   DELETE FROM relationships WHERE arena LIKE 'pentatonic-team%';
+   ```
+2. Re-enqueue every pentatonic-team event:
+   ```sql
+   UPDATE distillation_queue
+      SET status='pending', claimed_by=NULL, claim_expires_at=NULL
+    WHERE arena LIKE 'pentatonic-team%';
+   ```
+   (If queue rows were pruned post-done, re-insert from `events` for the arena.)
+3. Scale the fleet for the backlog; monitor `distillation_queue` depth until 0.
+
+### Stage 2 — downstream reconciliation
+1. **Fusion Drive** re-run for the arena (consolidate the freshly-extracted
+   nodes/facts) — it's a de-duplicator, so run AFTER extraction settles.
+   `backfill_entity_reconciliation.py` covers the entity side.
+2. **Projection re-fold** (Seesa side): the org/person projection sweeps re-fold
+   from the corrected graph (SEE-168 nightly sweep does this automatically; or
+   trigger manually). The Worker D1 read-models then inherit the heal.
+
+## Validation (gate before declaring done)
+- Re-run the audit harness over a fresh 30–50 entity sample → **bad-fact rate
+  target < 3%** (from ~18.7%).
+- Spot the canonical failures: Vickers "Board Observer" (should be modal/absent),
+  Catherine's "will send / to organise" (→ commitments), Matvii's "Pentatonic
+  GmbH" footer affiliation (gone), Sebastian conflation (split or correctly keyed).
+
+## Rollback
+Restore `facts`/`relationships` for `pentatonic-team%` from the Stage-0 snapshot;
+the new-prompt extractions are content-hash-distinct so a restore is clean.
+
+## Cost / time
+~163k pentatonic-team events × 7B extraction on the L4 fleet. Bounded by fleet
+size × per-batch latency; run off-peak, monitor GPU/disk (prior outage: v2 wrote
+to a 30 GB root — watch disk). Pilot first to estimate throughput before the full
+run.

package/packages/memory-engine-v2/extractor-async/extraction_diff.py

@@ -0,0 +1,218 @@
+"""Structured diff between two extractions (student vs teacher gold).
+
+Replaces the crude agreement proxies we'd been quoting (entity-name-exact-match;
+word-Jaccard on whole statements) with a STRUCTURED comparison that:
+
+  - matches entities on fuzzy name + compatible type (not exact lowercased string,
+    which penalised "Acme" vs "Acme Corp" / normalisation variants),
+  - matches facts on their s·p·o STRUCTURE plus the statement as a fallback
+    (not bag-of-words on the statement, which ignored who-did-what),
+  - matches relationships as (from, type, to) triples,
+  - reports precision / recall / F1 PER AXIS, and facts broken down PER CATEGORY
+    (so `decision`/`commitment` agreement is isolated — the cascade's
+    high-value gate question).
+
+Deterministic + stdlib-only (difflib) so it runs offline, in CI, and on any box
+without a GPU. Semantic-embedding matching is a deliberate non-goal here: a
+deterministic structural diff is the defensible, un-game-able baseline (no model
+judging another model's output); an embedding tiebreak can layer on later if the
+fuzzy threshold proves too strict.
+
+Shapes (mirror _parse_guided_json output):
+  entity       = {"name", "type", "aliases"?: [emails]}
+  fact         = {"category", "subject", "predicate", "object"?, "statement"}
+  relationship = {"from", "to", "type"}
+"""
+
+from __future__ import annotations
+
+import re
+from dataclasses import dataclass
+from difflib import SequenceMatcher
+from typing import Any
+
+# Org/legal-form suffixes stripped before name comparison so "Acme" == "Acme Inc".
+_ORG_SUFFIX = re.compile(
+    r"\b(inc|incorporated|ltd|limited|llc|llp|plc|corp|corporation|co|gmbh|ag|sa|"
+    r"sas|bv|nv|pty|group|holdings?|company)\b\.?",
+    re.IGNORECASE,
+)
+_NONWORD = re.compile(r"[^a-z0-9 ]+")
+_WS = re.compile(r"\s+")
+
+
+def normalize_name(s: str | None) -> str:
+    """Lowercase, drop punctuation + org suffixes, collapse whitespace."""
+    if not s:
+        return ""
+    s = s.lower()
+    s = _ORG_SUFFIX.sub(" ", s)
+    s = _NONWORD.sub(" ", s)
+    return _WS.sub(" ", s).strip()
+
+
+def _tokens(s: str) -> set[str]:
+    return set(normalize_name(s).split())
+
+
+def sim(a: str | None, b: str | None) -> float:
+    """Similarity in [0,1]: max of char-level ratio and token-set Jaccard on the
+    normalised strings. The token-set arm rewards word overlap regardless of
+    order/length ("ship the Q3 release" vs "Q3 release will ship"); the
+    char-ratio arm rewards near-identical short strings."""
+    na, nb = normalize_name(a), normalize_name(b)
+    if not na and not nb:
+        return 1.0
+    if not na or not nb:
+        return 0.0
+    if na == nb:
+        return 1.0
+    char = SequenceMatcher(None, na, nb).ratio()
+    ta, tb = set(na.split()), set(nb.split())
+    jac = len(ta & tb) / len(ta | tb) if (ta | tb) else 0.0
+    return max(char, jac)
+
+
+@dataclass
+class PRF:
+    """Precision/recall/F1 for one axis. n_gold/n_pred are the item counts."""
+    n_gold: int
+    n_pred: int
+    matched: int
+
+    @property
+    def precision(self) -> float:
+        if self.n_pred == 0:
+            return 1.0 if self.n_gold == 0 else 0.0
+        return self.matched / self.n_pred
+
+    @property
+    def recall(self) -> float:
+        if self.n_gold == 0:
+            return 1.0 if self.n_pred == 0 else 0.0
+        return self.matched / self.n_gold
+
+    @property
+    def f1(self) -> float:
+        p, r = self.precision, self.recall
+        return 2 * p * r / (p + r) if (p + r) else (1.0 if self.n_gold == 0 and self.n_pred == 0 else 0.0)
+
+    def as_dict(self) -> dict[str, float | int]:
+        return {
+            "n_gold": self.n_gold, "n_pred": self.n_pred, "matched": self.matched,
+            "precision": round(self.precision, 4), "recall": round(self.recall, 4),
+            "f1": round(self.f1, 4),
+        }
+
+
+def _greedy_match(gold: list, pred: list, score_fn, threshold: float) -> int:
+    """Count matched gold items via greedy 1:1 alignment. All (gold, pred) pairs
+    scored, sorted desc, claimed highest-first so each item matches at most once.
+    Deterministic (stable sort, then index tiebreak)."""
+    pairs = []
+    for gi, g in enumerate(gold):
+        for pi, p in enumerate(pred):
+            s = score_fn(g, p)
+            if s >= threshold:
+                pairs.append((-s, gi, pi))
+    pairs.sort()
+    used_g: set[int] = set()
+    used_p: set[int] = set()
+    matched = 0
+    for _, gi, pi in pairs:
+        if gi in used_g or pi in used_p:
+            continue
+        used_g.add(gi)
+        used_p.add(pi)
+        matched += 1
+    return matched
+
+
+# ── per-item scorers ─────────────────────────────────────────────────────
+
+def _entity_score(g: dict, p: dict) -> float:
+    """Name similarity, gated by type compatibility (equal, or either side
+    'other'/missing — the LLMs disagree on type far more than on identity)."""
+    gt = (g.get("type") or "").lower()
+    pt = (p.get("type") or "").lower()
+    if gt and pt and gt != pt and "other" not in (gt, pt):
+        return 0.0
+    return sim(g.get("name"), p.get("name"))
+
+
+def _fact_score(g: dict, p: dict) -> float:
+    """Structural s·p·o similarity, OR statement similarity as a fallback.
+    Structure: mean of subject/predicate/object sims (object absent on both =
+    neutral 1.0 for that term). Take the max of structural and statement so a
+    well-phrased statement still matches even if s/p/o were split differently."""
+    subj = sim(g.get("subject"), p.get("subject"))
+    pred = sim(g.get("predicate"), p.get("predicate"))
+    go, po = g.get("object"), p.get("object")
+    obj = 1.0 if not go and not po else sim(go, po)
+    structural = (subj + pred + obj) / 3
+    statement = sim(g.get("statement"), p.get("statement"))
+    return max(structural, statement)
+
+
+def _rel_score(g: dict, p: dict) -> float:
+    """(from, type, to) triple: mean of the three term sims."""
+    return (sim(g.get("from"), p.get("from"))
+            + sim(g.get("type"), p.get("type"))
+            + sim(g.get("to"), p.get("to"))) / 3
+
+
+# ── public API ─────────────────────────────────────────────────────────────
+
+ENTITY_THRESHOLD = 0.85
+FACT_THRESHOLD = 0.60
+REL_THRESHOLD = 0.60
+
+
+def diff_axis(gold: list[dict], pred: list[dict], kind: str) -> PRF:
+    """Match one axis ('entities' | 'facts' | 'relationships'); return PRF."""
+    scorer, thr = {
+        "entities": (_entity_score, ENTITY_THRESHOLD),
+        "facts": (_fact_score, FACT_THRESHOLD),
+        "relationships": (_rel_score, REL_THRESHOLD),
+    }[kind]
+    matched = _greedy_match(gold, pred, scorer, thr)
+    return PRF(n_gold=len(gold), n_pred=len(pred), matched=matched)
+
+
+def _facts_in(extraction: dict, categories: set[str] | None) -> list[dict]:
+    facts = extraction.get("facts") or []
+    if categories is None:
+        return facts
+    return [f for f in facts if (f.get("category") or "").lower() in categories]
+
+
+def diff_extractions(
+    gold: dict, pred: dict, fact_categories: set[str] | None = None
+) -> dict[str, Any]:
+    """Full structured diff. `gold`/`pred` are extraction dicts. Returns per-axis
+    PRF dicts plus per-fact-category PRF. `fact_categories`, if given, also
+    reports a 'facts_filtered' PRF over just those categories (e.g.
+    {'decision','commitment'} for the high-value-gate question)."""
+    out: dict[str, Any] = {
+        "entities": diff_axis(gold.get("entities") or [], pred.get("entities") or [], "entities").as_dict(),
+        "facts": diff_axis(gold.get("facts") or [], pred.get("facts") or [], "facts").as_dict(),
+        "relationships": diff_axis(
+            gold.get("relationships") or [], pred.get("relationships") or [], "relationships"
+        ).as_dict(),
+    }
+    # per-category fact breakdown
+    cats = {(f.get("category") or "").lower() for f in (gold.get("facts") or [])}
+    cats |= {(f.get("category") or "").lower() for f in (pred.get("facts") or [])}
+    cats.discard("")
+    by_cat: dict[str, Any] = {}
+    for c in sorted(cats):
+        g = _facts_in(gold, {c})
+        p = _facts_in(pred, {c})
+        by_cat[c] = diff_axis(g, p, "facts").as_dict()
+    out["facts_by_category"] = by_cat
+    if fact_categories is not None:
+        g = _facts_in(gold, fact_categories)
+        p = _facts_in(pred, fact_categories)
+        out["facts_filtered"] = diff_axis(g, p, "facts").as_dict()
+        out["facts_filtered_categories"] = sorted(fact_categories)
+    return out

package/packages/memory-engine-v2/extractor-async/test_extraction_diff.py

@@ -0,0 +1,180 @@
+"""Unit tests for the structured-diff agreement metric (extraction_diff).
+
+Pins the behaviours that make it a real metric rather than the old proxies:
+fuzzy/normalised name matching, structural s·p·o fact matching, per-axis P/R/F1,
+per-category fact breakdown, and the high-value filtered view.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+
+import pytest
+
+_THIS = Path(__file__).resolve().parent
+
+
+def _load(name="extraction_diff_mod"):
+    spec = importlib.util.spec_from_file_location(name, _THIS / "extraction_diff.py")
+    mod = importlib.util.module_from_spec(spec)
+    # Register before exec so @dataclass's type resolution (which walks
+    # sys.modules[cls.__module__]) works under the importlib custom-name load
+    # on Python 3.12+/3.14. A normal `import extraction_diff` (CI/prod) is fine.
+    sys.modules[name] = mod
+    spec.loader.exec_module(mod)
+    return mod
+
+
+ed = _load()
+
+
+# ── normalize_name / sim ─────────────────────────────────────────────────
+
+def test_normalize_strips_org_suffix_and_punct():
+    assert ed.normalize_name("Acme Corp.") == "acme"
+    assert ed.normalize_name("Acme, Inc.") == "acme"
+    assert ed.normalize_name("ACME Limited") == "acme"
+
+
+def test_sim_normalisation_variants_are_high():
+    # the exact-string proxy scored these 0; structured sim should be ~1.
+    assert ed.sim("Acme", "Acme Corp") >= 0.99
+    assert ed.sim("Acme Inc.", "ACME") >= 0.99
+
+
+def test_sim_token_overlap_order_invariant():
+    assert ed.sim("ship the Q3 release", "Q3 release will ship") >= 0.6
+
+
+def test_sim_unrelated_low():
+    assert ed.sim("Acme", "Globex") < 0.5
+
+
+def test_sim_both_empty_is_one():
+    assert ed.sim("", "") == 1.0
+    assert ed.sim(None, "x") == 0.0
+
+
+# ── PRF arithmetic ───────────────────────────────────────────────────────
+
+def test_prf_basic():
+    prf = ed.PRF(n_gold=4, n_pred=5, matched=3)
+    assert prf.recall == 0.75
+    assert prf.precision == 0.6
+    assert round(prf.f1, 3) == 0.667
+
+
+def test_prf_empty_both_is_perfect():
+    prf = ed.PRF(n_gold=0, n_pred=0, matched=0)
+    assert prf.precision == 1.0 and prf.recall == 1.0 and prf.f1 == 1.0
+
+
+def test_prf_pred_without_gold_is_zero_precision():
+    prf = ed.PRF(n_gold=0, n_pred=2, matched=0)
+    assert prf.precision == 0.0
+
+
+# ── entity matching ──────────────────────────────────────────────────────
+
+def test_entity_match_fuzzy_name_same_type():
+    g = [{"name": "Acme Corp", "type": "org"}]
+    p = [{"name": "Acme", "type": "org"}]
+    prf = ed.diff_axis(g, p, "entities")
+    assert prf.matched == 1 and prf.f1 == 1.0
+
+
+def test_entity_type_mismatch_blocks_match():
+    g = [{"name": "Apple", "type": "org"}]
+    p = [{"name": "Apple", "type": "person"}]
+    assert ed.diff_axis(g, p, "entities").matched == 0
+
+
+def test_entity_other_type_is_compatible():
+    g = [{"name": "Apple", "type": "org"}]
+    p = [{"name": "Apple", "type": "other"}]
+    assert ed.diff_axis(g, p, "entities").matched == 1
+
+
+def test_entity_greedy_one_to_one():
+    # two golds, one pred → at most one match
+    g = [{"name": "Acme", "type": "org"}, {"name": "Acme", "type": "org"}]
+    p = [{"name": "Acme", "type": "org"}]
+    prf = ed.diff_axis(g, p, "entities")
+    assert prf.matched == 1 and prf.recall == 0.5 and prf.precision == 1.0
+
+
+# ── fact matching (structural vs statement) ──────────────────────────────
+
+def test_fact_match_on_structure_despite_statement_rephrase():
+    g = [{"category": "decision", "subject": "Acme", "predicate": "will renew",
+          "object": "the contract", "statement": "Acme decided to renew the contract for 2027."}]
+    p = [{"category": "decision", "subject": "Acme", "predicate": "renews",
+          "object": "contract", "statement": "The 2027 contract renewal was agreed by Acme."}]
+    prf = ed.diff_axis(g, p, "facts")
+    assert prf.matched == 1
+
+
+def test_fact_match_on_statement_when_spo_split_differs():
+    g = [{"category": "commitment", "subject": "Bob", "predicate": "owns", "object": "migration",
+          "statement": "Bob will lead the migration starting in March."}]
+    p = [{"category": "commitment", "subject": "Bob Chen", "predicate": "leads",
+          "object": "the data migration", "statement": "Bob will lead the migration starting in March."}]
+    assert ed.diff_axis(g, p, "facts").matched == 1
+
+
+def test_fact_unrelated_no_match():
+    g = [{"category": "decision", "subject": "Acme", "predicate": "hired", "object": "a CFO",
+          "statement": "Acme hired a new CFO."}]
+    p = [{"category": "decision", "subject": "Globex", "predicate": "closed", "object": "the Berlin office",
+          "statement": "Globex shut its Berlin office."}]
+    assert ed.diff_axis(g, p, "facts").matched == 0
+
+
+# ── relationships ────────────────────────────────────────────────────────
+
+def test_relationship_triple_match():
+    g = [{"from": "Jane", "to": "Acme Corp", "type": "works at"}]
+    p = [{"from": "Jane", "to": "Acme", "type": "employed by"}]
+    # from + to match strongly; type weaker → mean may dip below threshold
+    prf = ed.diff_axis(g, p, "relationships")
+    assert prf.n_gold == 1 and prf.n_pred == 1
+
+
+# ── full diff + per-category + filtered ──────────────────────────────────
+
+def test_diff_extractions_per_category_and_filtered():
+    gold = {
+        "entities": [{"name": "Acme", "type": "org"}],
+        "facts": [
+            {"category": "decision", "subject": "Acme", "predicate": "will renew",
+             "object": "contract", "statement": "Acme will renew the contract."},
+            {"category": "state", "subject": "Acme", "predicate": "is", "object": "a customer",
+             "statement": "Acme is a customer."},
+        ],
+        "relationships": [],
+    }
+    pred = {
+        "entities": [{"name": "Acme Corp", "type": "org"}],
+        "facts": [
+            {"category": "decision", "subject": "Acme", "predicate": "renews",
+             "object": "the contract", "statement": "Acme renews its contract."},
+        ],
+        "relationships": [],
+    }
+    out = ed.diff_extractions(gold, pred, fact_categories={"decision", "commitment"})
+    assert out["entities"]["matched"] == 1
+    # per-category: decision matched 1/1; state missing (recall 0)
+    assert out["facts_by_category"]["decision"]["matched"] == 1
+    assert out["facts_by_category"]["state"]["recall"] == 0.0
+    # filtered to decision/commitment: 1 gold, 1 pred, 1 matched → perfect
+    assert out["facts_filtered"]["recall"] == 1.0
+    assert out["facts_filtered"]["precision"] == 1.0
+    assert out["facts_filtered_categories"] == ["commitment", "decision"]
+
+
+def test_diff_extractions_empty_both():
+    out = ed.diff_extractions({"entities": [], "facts": [], "relationships": []},
+                              {"entities": [], "facts": [], "relationships": []})
+    assert out["facts"]["f1"] == 1.0

package/packages/memory-engine-v2/extractor-async/test_prompt_rules.py

@@ -0,0 +1,58 @@
+"""Guard tests for the distiller system-prompt content rules.
+
+Pins that the email-discipline + entity-separation rules (this change) and the
+#126 modality/attribution rules are present in BOTH prompt variants — a cheap
+regression guard so a future prompt edit can't silently drop them.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import sys
+from pathlib import Path
+
+import pytest
+
+_THIS = Path(__file__).resolve().parent
+
+
+def _load(name="extractor_async_worker_prompts"):
+    spec = importlib.util.spec_from_file_location(name, _THIS / "worker.py")
+    mod = importlib.util.module_from_spec(spec)
+    sys.modules[name] = mod
+    spec.loader.exec_module(mod)
+    return mod
+
+
+try:
+    worker = _load()
+except ImportError as e:
+    pytest.skip(f"extractor-async deps unavailable: {e}", allow_module_level=True)
+
+PROMPTS = lambda: (worker.BATCH_SYSTEM_PROMPT, worker.GUIDED_JSON_SYSTEM_PROMPT)
+
+
+def test_email_discipline_in_both_prompts():
+    for p in PROMPTS():
+        assert "An email address is NOT a person" in p
+        assert "reservations@" in p           # role/transactional examples present
+        assert "clearly THEIRS" in p           # bystander-attachment ban
+
+
+def test_distinct_entities_rule_in_both_prompts():
+    for p in PROMPTS():
+        assert "DISTINCT ENTITIES" in p
+        assert "Acme & Globex" in p            # conflation-split example
+        assert "warehouse" in p                # generic-token suppression
+
+
+def test_126_rules_not_regressed():
+    for p in PROMPTS():
+        assert "TENSE & MODALITY" in p
+        assert "ATTRIBUTION FIDELITY" in p
+
+
+def test_active_prompt_carries_the_rules_and_fresh_hash():
+    assert "An email address is NOT a person" in worker.ACTIVE_SYSTEM_PROMPT
+    assert "DISTINCT ENTITIES" in worker.ACTIVE_SYSTEM_PROMPT
+    assert len(worker.SYSTEM_PROMPT_HASH) == 16  # hash recomputed off ACTIVE prompt

package/packages/memory-engine-v2/extractor-async/test_queue_attempts.py

@@ -0,0 +1,69 @@
+"""Unit tests for the distillation_queue attempts/retry accounting.
+
+Regression guard for the lease-reclaim bug (gotcha #11): claiming must NOT
+consume the retry budget — only genuine processing failures do — so a worker
+restart (deploy recreating the container) can re-claim stranded in-flight work
+indefinitely instead of stranding it in `claimed` forever. The DB-touching
+claim/release/fail SQL isn't unit-testable here (no DB in this suite), but the
+give-up decision is pure logic, so we pin it.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+import pytest
+
+_THIS = Path(__file__).resolve().parent
+
+
+def _load_worker(name: str = "extractor_async_worker_qa"):
+    spec = importlib.util.spec_from_file_location(name, _THIS / "worker.py")
+    assert spec and spec.loader
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    return mod
+
+
+try:
+    worker = _load_worker()
+except ImportError as e:
+    pytest.skip(f"extractor-async deps unavailable: {e}", allow_module_level=True)
+
+
+def test_attempts_exhausted_gives_exactly_max_genuine_tries(monkeypatch) -> None:
+    """`attempts` is the count of PRIOR genuine failures at claim time. With
+    MAX_ATTEMPTS=3 the sequence is: fail#1 (attempts=0)→retry, fail#2
+    (attempts=1)→retry, fail#3 (attempts=2)→terminal. Exactly 3 tries."""
+    monkeypatch.setattr(worker, "MAX_ATTEMPTS", 3)
+    assert worker._attempts_exhausted(0) is False  # 1st failure → retry
+    assert worker._attempts_exhausted(1) is False  # 2nd failure → retry
+    assert worker._attempts_exhausted(2) is True   # 3rd failure → give up
+    assert worker._attempts_exhausted(3) is True
+
+
+def test_attempts_exhausted_respects_max(monkeypatch) -> None:
+    monkeypatch.setattr(worker, "MAX_ATTEMPTS", 1)
+    assert worker._attempts_exhausted(0) is True   # single try, no retry
+    monkeypatch.setattr(worker, "MAX_ATTEMPTS", 5)
+    assert worker._attempts_exhausted(3) is False
+    assert worker._attempts_exhausted(4) is True
+
+
+def test_claim_sql_does_not_increment_attempts() -> None:
+    """The fix: claiming must not touch `attempts` (only release/fail do). Guard
+    against a regression that reintroduces the increment at claim time. We check
+    the source of claim_next_batch rather than execute it (no DB here)."""
+    import inspect
+    src = inspect.getsource(worker.claim_next_batch)
+    # the claim UPDATE must not bump attempts; the only attempts reference is the
+    # eligibility predicate `attempts < %s`.
+    assert "attempts = attempts + 1" not in src
+    assert "attempts <" in src  # eligibility gate still present
+
+
+def test_release_and_fail_increment_attempts() -> None:
+    import inspect
+    assert "attempts = attempts + 1" in inspect.getsource(worker.release_claim)
+    assert "attempts = attempts + 1" in inspect.getsource(worker.mark_failed)

package/packages/memory-engine-v2/extractor-async/worker.py

@@ -284,6 +284,38 @@ and NEVER drop a field.
   object MAY be an entity name OR a literal string OR `-` if absent.
   statement ≤ 140 characters, a self-contained sentence.
   WORKED EXAMPLE: `FCT|commitment|Timothy Bradley|agreed to|SAFE amendments|Timothy confirmed the SAFE amendments are set (14 May 2026)`
+- TENSE & MODALITY — never record the future or the merely-planned as done:
+  * A SCHEDULED or FUTURE event (a calendar invite, a meeting/call dated \
+later than this event, a recurring-meeting instance) is UPCOMING, not \
+completed. NEVER emit "attended / hosted / met / reported / decided" for a \
+meeting that has not happened — use category=commitment with predicate "is \
+scheduled to" / "plans to".
+  * A PLAN, PROPOSAL, INTENT or next-step ("will", "plans to", "aims to", \
+"to organise", "we'll", "next steps:") is category=commitment, NOT state and \
+NOT a completed act. Keep "will send" as a commitment — never record it as "sent".
+- ATTRIBUTION FIDELITY — the subject must be who the source actually credits:
+  * Attribute a document/deck/message's content to a person ONLY if the source \
+names them as its author or speaker. Do NOT attribute an unauthored doc, agenda \
+or deck to whoever it merely mentions or whoever shared it.
+  * Do NOT promote a meeting ATTENDEE to organiser/host without explicit evidence.
+  * Do NOT attribute an ORGANISATION's activity (deals, intros, pipeline) to an \
+individual person.
+- IDENTITY & EMAILS — do NOT infer a person's employer/affiliation from an email \
+signature or a company's standard footer/boilerplate; affiliation needs an \
+explicit statement in the body. \
+An email address is NOT a person: NEVER emit an entity whose name is an email \
+address, and NEVER treat a role/transactional address (reservations@, bookings@, \
+no-reply@, info@, office@, marketing@, support@, notifications@, admin@) as a \
+person. Attach an email to a person ONLY when it is clearly THEIRS (the author, \
+or a local-part matching their name) — NEVER attach an address that merely \
+co-occurs in the same thread, CC list, or document (a hotel booking, a \
+newsletter, an unrelated contact).
+- DISTINCT ENTITIES — two names joined by "and" / "&" / "/" are TWO separate \
+entities, never one merged node ("Acme & Globex" → emit Acme AND Globex). Do NOT \
+turn a sentence fragment or phrase into an entity, and do NOT mint generic \
+infrastructure / environment tokens (prod, staging, preview, UAT, warehouse, \
+main, PRD, CRM, platform, localhost) as entities — they are not named people, \
+orgs, products, or projects.
 - REL lines have exactly 4 fields: `REL`, from, to, rel_type.
   from and to MUST be entity names declared in THIS event's ENT lines.
   rel_type is a short verb / preposition phrase.
@@ -337,6 +369,38 @@ observation, preference}.
   WORKED EXAMPLE: {"category": "commitment", "subject": "Timothy \
 Bradley", "predicate": "agreed to", "object": "SAFE amendments", \
 "statement": "Timothy confirmed the SAFE amendments are set (14 May 2026)"}
+- TENSE & MODALITY — never record the future or the merely-planned as done:
+  * A SCHEDULED or FUTURE event (a calendar invite, a meeting/call dated \
+later than this event, a recurring-meeting instance) is UPCOMING, not \
+completed. NEVER emit "attended / hosted / met / reported / decided" for a \
+meeting that has not happened — use category=commitment with predicate "is \
+scheduled to" / "plans to".
+  * A PLAN, PROPOSAL, INTENT or next-step ("will", "plans to", "aims to", \
+"to organise", "we'll", "next steps:") is category=commitment, NOT state and \
+NOT a completed act. Keep "will send" as a commitment — never record it as "sent".
+- ATTRIBUTION FIDELITY — the subject must be who the source actually credits:
+  * Attribute a document/deck/message's content to a person ONLY if the source \
+names them as its author or speaker. Do NOT attribute an unauthored doc, agenda \
+or deck to whoever it merely mentions or whoever shared it.
+  * Do NOT promote a meeting ATTENDEE to organiser/host without explicit evidence.
+  * Do NOT attribute an ORGANISATION's activity (deals, intros, pipeline) to an \
+individual person.
+- IDENTITY & EMAILS — do NOT infer a person's employer/affiliation from an email \
+signature or a company's standard footer/boilerplate; affiliation needs an \
+explicit statement in the body. \
+An email address is NOT a person: NEVER emit an entity whose name is an email \
+address, and NEVER treat a role/transactional address (reservations@, bookings@, \
+no-reply@, info@, office@, marketing@, support@, notifications@, admin@) as a \
+person. Attach an email to a person ONLY when it is clearly THEIRS (the author, \
+or a local-part matching their name) — NEVER attach an address that merely \
+co-occurs in the same thread, CC list, or document (a hotel booking, a \
+newsletter, an unrelated contact).
+- DISTINCT ENTITIES — two names joined by "and" / "&" / "/" are TWO separate \
+entities, never one merged node ("Acme & Globex" → emit Acme AND Globex). Do NOT \
+turn a sentence fragment or phrase into an entity, and do NOT mint generic \
+infrastructure / environment tokens (prod, staging, preview, UAT, warehouse, \
+main, PRD, CRM, platform, localhost) as entities — they are not named people, \
+orgs, products, or projects.
 - relationships: "from" and "to" MUST be entity names declared in THIS \
 event's "entities". "type" is a short verb / preposition phrase.
 - HARD CAPS per event: 8 entities, 6 facts, 6 relationships. Pick the \
@@ -1905,8 +1969,16 @@ def claim_next_batch(conn: psycopg.Connection) -> list[dict[str, Any]]:
               status = 'claimed',
               claimed_by = %s,
               claimed_at = NOW(),
-              claim_expires_at = NOW() + (%s || ' seconds')::interval,
-              attempts = attempts + 1
+              claim_expires_at = NOW() + (%s || ' seconds')::interval
+              -- NB: claiming does NOT increment `attempts`. `attempts` counts
+              -- genuine PROCESSING failures (release_claim / mark_failed), not
+              -- claim-grabs. A worker that dies mid-batch (e.g. a deploy
+              -- recreates the container) leaves its rows in `claimed`; the lease
+              -- expires and they are re-claimed here WITHOUT burning the retry
+              -- budget — so restarts can't strand in-flight work. (Pre-fix, the
+              -- increment lived here and ~3 deploys could push a row to
+              -- attempts=MAX, making it forever-ineligible for reclaim AND never
+              -- marked failed → orphaned in `claimed`. See gotcha #11.)
             WHERE id IN (
               SELECT id FROM distillation_queue
               WHERE (
@@ -1943,14 +2015,21 @@ def mark_done(conn: psycopg.Connection, queue_id: int) -> None:
 
 
 def mark_failed(conn: psycopg.Connection, queue_id: int, error: str) -> None:
+    # Terminal genuine-failure path → count the attempt (claiming no longer
+    # does; see claim_next_batch). Leaves the row's `attempts` reflecting the
+    # true number of processing attempts on a failed row.
     with conn.cursor() as cur:
         cur.execute(
-            "UPDATE distillation_queue SET status = 'failed', last_error = %s WHERE id = %s",
+            "UPDATE distillation_queue SET status = 'failed', "
+            "attempts = attempts + 1, last_error = %s WHERE id = %s",
             (error[:1024], queue_id),
         )
 
 
 def release_claim(conn: psycopg.Connection, queue_id: int, error: str) -> None:
+    # Recoverable genuine-failure path (will retry) → count the attempt. This is
+    # where the retry budget is spent — NOT at claim time — so a deploy-induced
+    # reclaim never consumes it.
     with conn.cursor() as cur:
         cur.execute(
             """
@@ -1959,6 +2038,7 @@ def release_claim(conn: psycopg.Connection, queue_id: int, error: str) -> None:
               claimed_by = NULL,
               claimed_at = NULL,
               claim_expires_at = NULL,
+              attempts = attempts + 1,
               last_error = %s
             WHERE id = %s
             """,
@@ -1966,6 +2046,15 @@ def release_claim(conn: psycopg.Connection, queue_id: int, error: str) -> None:
         )
 
 
+def _attempts_exhausted(attempts: int) -> bool:
+    """Whether THIS processing failure should be terminal (mark_failed) rather
+    than retried (release_claim). `attempts` is the row's value at claim time =
+    the count of PRIOR genuine failures (claiming no longer increments it). This
+    failure is attempt #(attempts+1), so we give up once that reaches
+    MAX_ATTEMPTS — giving exactly MAX_ATTEMPTS genuine tries before failing."""
+    return attempts + 1 >= MAX_ATTEMPTS
+
+
 # --------------------------------------------------------------------
 # Main loop
 # --------------------------------------------------------------------
@@ -2142,7 +2231,7 @@ async def _run_teacher(
                 log.warning(
                     f"extraction failed queue_id={queue_id} attempts={attempts}: {err}"
                 )
-                if attempts >= MAX_ATTEMPTS:
+                if _attempts_exhausted(attempts):
                     mark_failed(conn, queue_id, err)
                 else:
                     release_claim(conn, queue_id, err)
@@ -2254,7 +2343,7 @@ def _apply_extraction(
         log.warning(
             f"db upsert failed queue_id={queue_id} attempts={attempts}: {err}"
         )
-        if attempts >= MAX_ATTEMPTS:
+        if _attempts_exhausted(attempts):
             mark_failed(conn, queue_id, err)
         else:
             release_claim(conn, queue_id, err)