@pentatonic-ai/ai-agent-sdk 0.10.15 → 0.10.17

package/dist/index.cjs

@@ -878,7 +878,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
 }
 
 // src/telemetry.js
-var VERSION = "0.10.15";
+var VERSION = "0.10.17";
 var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
 function machineId() {
   const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";

package/dist/index.js

@@ -847,7 +847,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
 }
 
 // src/telemetry.js
-var VERSION = "0.10.15";
+var VERSION = "0.10.17";
 var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
 function machineId() {
   const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pentatonic-ai/ai-agent-sdk",
-  "version": "0.10.15",
+  "version": "0.10.17",
   "description": "TES SDK — LLM observability and lifecycle tracking via Pentatonic Thing Event System. Track token usage, tool calls, and conversations. Manage things through event-sourced lifecycle stages with AI enrichment and vector search.",
   "type": "module",
   "main": "./dist/index.cjs",

package/packages/memory-engine-v2/RFC-student-cascade.md

@@ -0,0 +1,148 @@
+# RFC — Student→Teacher Distillation Cascade (#99)
+
+**Status:** proposed (plan of record) · **Date:** 2026-06-18 · **Owner:** Phil
+
+## Goal
+
+Cut distillation GPU cost/latency by making the cheap fine-tuned **student**
+(`numind/NuExtract-2.0-4B`, full-FT on teacher traces) the **primary** distiller,
+and reaching the expensive **teacher** (Qwen3.6-27B on the L40S fleet) only for a
+**sampled + gated subset**. The teacher's output on that subset both corrects the
+graph and feeds continuous improvement of the student.
+
+This is a model **cascade**, not a replacement: the teacher stays the source of
+truth on the hard/sampled tail; the student carries the bulk.
+
+## What we already have (grounding)
+
+- **Trained student** at `s3://pme-deploy-prod-us-east-1-170649632502/backups/training/nuextract-2.0-4b-ft-final/` (7 GB). Train/eval loss 0.112/0.106.
+- **Quality vs the CURRENT teacher** (`f1e0ff` prompt): median entity-F1 **0.909**, ~**0% invalid JSON**, ~1% email-hallucination, **no drift** despite a teacher-prompt change since training (the student trained on `bbdaba`). So it's deployable without a retrain.
+- **Routing signals are weak.** Single-pass token-confidence and a feature-based verifier (HistGBM, AUC 0.63) only modestly beat random at predicting student↔teacher disagreement. Cheap features don't pinpoint the student's errors. → routing leans on deterministic gates + sampling, not a confidence/verifier gate.
+- **Infra to reuse:** `distillation_queue` + the `extractor-async` consumer pattern; the combined-demand distiller **autoscaler**; the `fusion_queue` consumer pattern; `distillation_traces` with `system_prompt_hash` (teacher-version segmentation); the Fusion Drive (fuzzy self-healing on top).
+- **Caveat carried forward:** the "disagreement rate" we've quoted (~12%) is a crude entity-name-exact-match proxy — entities only, penalizes normalization, teacher-as-gold. A proper structured-diff metric is a dependency for trustworthy monitoring + verifier labels (see Open Questions).
+
+## Architecture (steady state, post-flip)
+
+```
+event → extractor-sync (deterministic provisional write, unchanged) → distillation_queue
+            │
+            ▼
+   STUDENT consumer (cheap, always-on small GPU)
+      • distils the event, writes entities/facts/relationships to the graph
+        tagged producer='student'
+      • computes the escalation decision (below)
+            │
+            └── ESCALATE subset → distillation_queue-teacher (the existing 27B flow,
+                                   autoscaled L40S fleet)
+                                   • teacher distils
+                                   • SUPERSEDES the student's rows for that event
+                                   • writes distillation_traces (gold) → monitoring + retraining
+```
+
+The student is a new consumer; escalation is an enqueue onto the existing teacher
+path. No new scheduler — the autoscaler already scales the teacher fleet on queue
+depth (incl. fusion). The student runs on a cheap always-on GPU (L4/g6 — 4B fits
+in ~8 GB; the teacher L40S fleet stays scale-to-zero for the escalation subset).
+
+## Escalation policy (the "sample out")
+
+Escalate to the teacher iff **any** of:
+
+1. **Deterministic gate-fail** (cheap, high-precision):
+   - student output isn't valid JSON / violates schema, OR
+   - **grounding violation** — student emits an email/entity contradicting the
+     event's structured envelope (the #111 hard-key bag). *This is the gate that
+     catches "confidently wrong about a known fact," which confidence can't.*
+2. **High-value event class** — e.g. `decision`/`commitment` facts, VIP arenas —
+   always teacher (cheap metadata routing, decided pre-student where possible).
+3. **Random sample** (e.g. 3–5%) — *not* a quality lever; this exists to (a)
+   monitor student↔teacher agreement over time (drift), and (b) generate fresh
+   teacher-gold on live traffic for retraining the student + verifier (active
+   learning). The events the student is *unsure* on are the highest-value
+   retraining data.
+4. *(soft)* **verifier score** — verifier-v0 is weak (AUC 0.63); use it only as a
+   low-weight tiebreak to nudge borderline events toward the teacher, never as
+   the primary gate. Revisit if a stronger signal (self-consistency) is built.
+
+Everything else: the student's write stands.
+
+## Supersede-on-escalation (the one genuinely new mechanism)
+
+The store is **pure-accretion** — event identity is `content-hash`, there is **no
+supersede-by-source_id**, and graph upserts only accrete (see
+`pme2-dedup-supersede-semantics`). So when the teacher re-distils an escalated
+event, its output must **replace** the student's rows for that event, not pile on
+top. Options:
+
+- **(A) Producer-tagged supersede (recommended).** Tag every graph row written by
+  distillation with `producer` (`student`/`teacher`) + `event_id` (already in
+  `provenance_event_ids`). On a teacher escalation, in one transaction: delete the
+  `producer='student'` rows whose provenance is that single event **and** that no
+  other event corroborates (don't delete a row a second event also supports —
+  decrement/repoint instead), then write the teacher's rows. Mirrors the Fusion
+  Drive's repoint/audit discipline; reversible via an audit receipt.
+- **(B) Defer-write.** Decide escalation *before* the student writes (gates that
+  don't need the student output: high-value class, random sample), and for those
+  skip the student write entirely — teacher-only. Gate-fails (which need the
+  student output) still need (A). Cuts most supersede churn.
+- Recommended: **B for the pre-decidable escalations + A for gate-fail
+  escalations.** Most escalations (class/random) are pre-decidable → no student
+  write to undo; only the (rarer) gate-fails incur a supersede.
+
+Open: define "no other event corroborates" precisely against the accretion graph;
+reuse Fusion Drive's `entity_merges`/`fact_merges` audit tables for reversibility.
+
+## Rollout sequence
+
+1. **Shadow** (no graph impact): student runs alongside; teacher still does
+   everything; log student-vs-teacher per event → validate on live traffic +
+   accumulate verifier/quality-metric labels. (We've already done a *batch* shadow
+   over recent traces; a brief standing shadow confirms on live flow.)
+2. **Flip to student-primary + sampled teacher** (the diagram above), starting
+   with a **conservative escalation rate** (high random %, broad high-value
+   classes), tighten as monitoring confirms quality.
+3. **Iterate**: retrain student on accumulated teacher-gold (esp. escalated/hard
+   events); rebuild the verifier when a proper metric + more data exist.
+
+Kill switch + dry-run posture mirror the Fusion Drive (a flag to fall back to
+teacher-primary instantly).
+
+## Monitoring & active learning
+
+- **Drift:** the random-sample agreement rate, segmented by `system_prompt_hash`.
+  A teacher-prompt change (like `bbdaba`→`f1e0ff`) shows up as an agreement drop →
+  trigger a student refresh. (The hash segmentation already exists.)
+- **Active learning:** escalated + sampled events with teacher-gold are the next
+  training corpus; the student improves on exactly its weak spots.
+- **Cost model:** worth it iff escalation rate × teacher-cost ≪ teacher-on-
+  everything. Student-on-everything (cheap GPU, always-on) + teacher on ~10–20%
+  beats teacher-on-100% comfortably; the L40S fleet scale-to-zero already assumes
+  bursty teacher load.
+
+## Open questions / risks
+
+- **Disagreement metric.** Replace the entity-name-exact-match proxy with a
+  structured diff (entities w/ type+email, facts as s·p·o, relationships; fuzzy
+  name matching; small independent rubric for semantic equivalence). Dependency
+  for trustworthy monitoring **and** better verifier labels. *(Highest-leverage
+  next build.)*
+- **Prompt-version coupling.** The student is bound to a teacher prompt version
+  (`system_prompt_hash`). Every teacher-prompt change risks staling it → the
+  monitoring must watch the hash and the refresh pipeline must be cheap.
+- **Routing is unsolved.** No cheap signal cleanly predicts student errors yet;
+  self-consistency (K-sample) is the most promising unbuilt option but costs K×.
+  Until then, gates + sampling carry it and we accept the residual tail.
+- **Supersede correctness** in the accretion store (above) — the riskiest piece
+  to get exactly right.
+
+## Build phases (components)
+
+1. **Metric** — structured-diff scorer (offline, no GPU). *Do first.*
+2. **Student service** — always-on cheap-GPU server (4B) + a `distillation_queue`
+   student consumer; producer-tagging on graph writes.
+3. **Escalation + supersede** — gate/class/random logic; supersede-on-escalation
+   (B+A); reuse audit tables.
+4. **Shadow wiring** → **flip** → **monitoring dashboard** (agreement by hash).
+5. **Retrain loop** — periodic student/verifier retrain on accumulated gold.
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)

package/packages/memory-engine-v2/compat/server.py

@@ -928,7 +928,7 @@ async def list_entities(req: GraphQueryRequest):
         params.extend([pattern, pattern])
     sql = f"""
         SELECT id, arena, entity_type, canonical_name, aliases,
-               provenance_event_ids, last_seen
+               provenance_event_ids, attributes, last_seen
           FROM entities
          WHERE {' AND '.join(conditions)}
       ORDER BY last_seen DESC

package/packages/memory-engine-v2/extractor-async/entity_id.py

@@ -55,3 +55,23 @@ def entity_id(arena: str, entity_type: str, canonical_name: str) -> str:
     """
     key = f"{arena}|{entity_type}|{normalize_surface_form(canonical_name)}"
     return "e_" + hashlib.sha256(key.encode()).hexdigest()[:24]
+
+
+def person_id_key(name: str | None, email: str | None) -> str:
+    """The string `entity_id()` should hash to mint a PERSON's node id — the
+    EMAIL (the person's deterministic hard key) when present, else the name.
+
+    Lives HERE, in the byte-identical shared file, BECAUSE both extractors mint
+    person nodes and must agree: the sync pass builds them from the envelope at
+    ingest (it has the email), the async pass builds them from prose (email
+    promoted to an alias). Keying both on the email means the same person mints
+    the SAME id from either pass and in any processing order — converging to one
+    node instead of sync's name-keyed node and async's node racing, or a
+    re-distill re-homing them differently. The name is the fallback for a person
+    with no email (Fusion still merges those fuzzily). `entity_id()` normalises
+    (lowercase + trim) the result, so casing/whitespace variants of an email
+    collapse. (Org keying is the async-only `org_node_id_key`; person keying is
+    cross-pass, so it belongs in this parity-guarded file.)
+    """
+    e = (email or "").strip()
+    return e if e else (name or "")

package/packages/memory-engine-v2/extractor-async/test_cascade.py

@@ -0,0 +1,175 @@
+"""Unit tests for the student→teacher distillation cascade (#99).
+
+Covers the pure decision surface — the JSON-salvage of student output, the
+grounding email set derived from an event, and the escalation gates (parse
+fail, empty, grounding violation, high-value class, random sample, pass). The
+network call (call_student_one) and the DB writes (_apply_extraction,
+_record_distillation) are integration-tested elsewhere; here we pin the routing
+logic that decides student-XOR-teacher.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+from pathlib import Path
+
+import pytest
+
+_THIS = Path(__file__).resolve().parent
+
+
+def _load_worker(name: str = "extractor_async_worker_cascade"):
+    spec = importlib.util.spec_from_file_location(name, _THIS / "worker.py")
+    assert spec and spec.loader
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    return mod
+
+
+try:
+    worker = _load_worker()
+except ImportError as e:
+    pytest.skip(f"extractor-async deps unavailable: {e}", allow_module_level=True)
+
+
+# ----------------------------------------------------------------------
+# _salvage_json_object — the JSON-validity gate's parser
+# ----------------------------------------------------------------------
+
+def test_salvage_plain_object() -> None:
+    assert worker._salvage_json_object('{"entities": []}') == {"entities": []}
+
+
+def test_salvage_strips_code_fence() -> None:
+    txt = '```json\n{"facts": [{"statement": "x"}]}\n```'
+    assert worker._salvage_json_object(txt) == {"facts": [{"statement": "x"}]}
+
+
+def test_salvage_extracts_embedded_object() -> None:
+    txt = 'Sure! Here is the extraction:\n{"entities": [{"name": "Acme"}]}\nDone.'
+    assert worker._salvage_json_object(txt) == {"entities": [{"name": "Acme"}]}
+
+
+def test_salvage_returns_none_on_garbage() -> None:
+    assert worker._salvage_json_object("not json at all") is None
+    assert worker._salvage_json_object("") is None
+    # a bare JSON array is not an event object
+    assert worker._salvage_json_object("[1, 2, 3]") is None
+
+
+# ----------------------------------------------------------------------
+# _event_known_emails — grounding set (content + structured envelope)
+# ----------------------------------------------------------------------
+
+def test_known_emails_from_content_and_envelope() -> None:
+    event = {
+        "content": "Reach me at alice@acme.com tomorrow.",
+        "attributes": {
+            "contact_email": "Bob <bob@acme.com>",
+            "to_emails": ["carol@acme.com", "dave@x.io"],
+            "cc_emails": ["erin@acme.com"],
+        },
+    }
+    known = worker._event_known_emails(event)
+    assert known == {
+        "alice@acme.com", "bob@acme.com", "carol@acme.com",
+        "dave@x.io", "erin@acme.com",
+    }
+
+
+def test_known_emails_lowercased_and_empty_safe() -> None:
+    assert worker._event_known_emails({"content": "FOO@BAR.COM"}) == {"foo@bar.com"}
+    assert worker._event_known_emails({}) == set()
+
+
+# ----------------------------------------------------------------------
+# escalation_decision — the gates (precedence + each trigger)
+# ----------------------------------------------------------------------
+
+def _no_sample(monkeypatch):
+    """Pin the random sample off so the deterministic gates are isolated."""
+    monkeypatch.setattr(worker, "STUDENT_SAMPLE_RATE", 0.0)
+
+
+def test_escalate_on_parse_fail(monkeypatch) -> None:
+    _no_sample(monkeypatch)
+    esc, reason = worker.escalation_decision({"content": ""}, None)
+    assert esc and reason == "student_parse_fail"
+
+
+def test_escalate_on_empty_extraction(monkeypatch) -> None:
+    _no_sample(monkeypatch)
+    empty = {"entities": [], "facts": [], "relationships": []}
+    esc, reason = worker.escalation_decision({"content": ""}, empty)
+    assert esc and reason == "student_empty"
+
+
+def test_escalate_on_grounding_violation(monkeypatch) -> None:
+    _no_sample(monkeypatch)
+    event = {"content": "Met with the vendor.", "attributes": {}}
+    # student invents an email not present anywhere in the event
+    result = {
+        "entities": [{"type": "person", "name": "X", "aliases": ["ghost@evil.com"]}],
+        "facts": [],
+        "relationships": [],
+    }
+    esc, reason = worker.escalation_decision(event, result)
+    assert esc and reason == "grounding_violation"
+
+
+def test_no_escalation_when_email_is_grounded(monkeypatch) -> None:
+    _no_sample(monkeypatch)
+    event = {"content": "ping alice@acme.com", "attributes": {}}
+    result = {
+        "entities": [{"type": "person", "name": "Alice", "aliases": ["alice@acme.com"]}],
+        "facts": [],
+        "relationships": [],
+    }
+    esc, reason = worker.escalation_decision(event, result)
+    assert not esc and reason is None
+
+
+def test_escalate_on_high_value_class(monkeypatch) -> None:
+    _no_sample(monkeypatch)
+    monkeypatch.setattr(worker, "HIGH_VALUE_CATEGORIES", {"decision", "commitment"})
+    event = {"content": "We will ship Friday.", "attributes": {}}
+    result = {
+        "entities": [],
+        "facts": [{"category": "decision", "statement": "ship Friday", "subject": "team"}],
+        "relationships": [],
+    }
+    esc, reason = worker.escalation_decision(event, result)
+    assert esc and reason == "high_value_class"
+
+
+def test_escalate_on_random_sample(monkeypatch) -> None:
+    monkeypatch.setattr(worker, "STUDENT_SAMPLE_RATE", 1.0)  # always sample
+    monkeypatch.setattr(worker, "HIGH_VALUE_CATEGORIES", set())
+    event = {"content": "low value note", "attributes": {}}
+    result = {"entities": [{"type": "other", "name": "thing"}], "facts": [], "relationships": []}
+    esc, reason = worker.escalation_decision(event, result)
+    assert esc and reason == "random_sample"
+
+
+def test_student_handles_clean_low_value_event(monkeypatch) -> None:
+    """The common case: valid, grounded, non-high-value, not sampled → the
+    student's write stands (no escalation)."""
+    _no_sample(monkeypatch)
+    monkeypatch.setattr(worker, "HIGH_VALUE_CATEGORIES", {"decision", "commitment"})
+    event = {"content": "Acme released a new SKU.", "attributes": {}}
+    result = {
+        "entities": [{"type": "org", "name": "Acme"}],
+        "facts": [{"category": "state", "statement": "Acme released a SKU", "subject": "Acme"}],
+        "relationships": [],
+    }
+    esc, reason = worker.escalation_decision(event, result)
+    assert not esc and reason is None
+
+
+# ----------------------------------------------------------------------
+# Flag contract — cascade is a no-op until CASCADE_ENABLED is flipped.
+# ----------------------------------------------------------------------
+
+def test_cascade_default_off() -> None:
+    """Default env ⇒ teacher-only. The flag is the kill switch."""
+    assert worker.CASCADE_ENABLED is False

package/packages/memory-engine-v2/extractor-async/test_fact_source.py

@@ -0,0 +1,62 @@
+"""Tests for fact_source — deriving the source label stamped onto facts.
+
+The contract under test (SoR-drift foundation): prefer the finer
+producer label `attributes.source` (gmail / hubspot / ...) over the
+coarse `source_kind` enum; fall back to `source_kind` when no finer
+label; return None only when neither is present (NULL == source-unknown,
+the pre-009 state). Pure + total: never raises.
+
+Run: pytest packages/memory-engine-v2/extractor-async/test_fact_source.py
+"""
+
+from __future__ import annotations
+
+import pytest
+
+from worker import fact_source
+
+
+class TestFactSource:
+    def test_prefers_finer_attributes_source(self):
+        # attributes.source (hubspot) wins over the coarse source_kind
+        # (system) — this is the CRM-vs-email granularity SoR-drift needs.
+        ev = {"source_kind": "system", "attributes": {"source": "hubspot"}}
+        assert fact_source(ev) == "hubspot"
+
+    def test_email_finer_than_note_kind(self):
+        ev = {"source_kind": "note", "attributes": {"source": "gmail"}}
+        assert fact_source(ev) == "gmail"
+
+    def test_falls_back_to_source_kind_when_no_attribute(self):
+        ev = {"source_kind": "chat", "attributes": {}}
+        assert fact_source(ev) == "chat"
+
+    def test_falls_back_when_attributes_missing(self):
+        ev = {"source_kind": "doc"}
+        assert fact_source(ev) == "doc"
+
+    def test_strips_whitespace(self):
+        ev = {"source_kind": "system", "attributes": {"source": "  slack  "}}
+        assert fact_source(ev) == "slack"
+
+    def test_blank_attribute_falls_through_to_kind(self):
+        # An empty/whitespace `source` must NOT win over a real kind.
+        ev = {"source_kind": "doc", "attributes": {"source": "   "}}
+        assert fact_source(ev) == "doc"
+
+    # --- None cases: source-unknown, column stays NULL ---
+
+    @pytest.mark.parametrize(
+        "ev",
+        [
+            {},
+            {"attributes": {}},
+            {"attributes": {"source": ""}},
+            {"source_kind": "", "attributes": {"source": None}},
+            {"source_kind": None, "attributes": None},
+            # Non-string types must not crash and must not be stamped.
+            {"source_kind": 7, "attributes": {"source": 42}},
+        ],
+    )
+    def test_returns_none_when_no_usable_source(self, ev):
+        assert fact_source(ev) is None