@pentatonic-ai/ai-agent-sdk 0.10.37 → 0.10.39

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/dist/index.cjs CHANGED
@@ -878,7 +878,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
878
878
  }
879
879
 
880
880
  // src/telemetry.js
881
- var VERSION = "0.10.37";
881
+ var VERSION = "0.10.39";
882
882
  var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
883
883
  function machineId() {
884
884
  const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";
package/dist/index.js CHANGED
@@ -847,7 +847,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
847
847
  }
848
848
 
849
849
  // src/telemetry.js
850
- var VERSION = "0.10.37";
850
+ var VERSION = "0.10.39";
851
851
  var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
852
852
  function machineId() {
853
853
  const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@pentatonic-ai/ai-agent-sdk",
3
- "version": "0.10.37",
3
+ "version": "0.10.39",
4
4
  "description": "TES SDK — LLM observability and lifecycle tracking via Pentatonic Thing Event System. Track token usage, tool calls, and conversations. Manage things through event-sourced lifecycle stages with AI enrichment and vector search.",
5
5
  "type": "module",
6
6
  "main": "./dist/index.cjs",
@@ -0,0 +1,497 @@
1
+ # Async Escalation Implementation Plan
2
+
3
+ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
+
5
+ **Goal:** Take the teacher off the ingest critical path — the student writes the confident majority synchronously; escalated events become an `escalated_pending` queue state drained asynchronously by the existing L40S mop-up fleet.
6
+
7
+ **Architecture:** One `distillation_queue`, two consumers. The student-primary consumer (always-on L4) claims `pending`, writes what it's confident on, and marks the rest `escalated_pending` instead of calling the teacher inline. A teacher-mop-up consumer claims `escalated_pending` and writes it once via the teacher vLLM. Deferred first-write ⇒ no supersede. Everything ships dark behind `ASYNC_ESCALATION_ENABLED` (default false) with instant revert to today's synchronous cascade.
8
+
9
+ **Tech Stack:** Python 3.12 (`extractor-async/worker.py`), Postgres (`distillation_queue`, org-model migrations), bash systemd ops (TES `distiller-autoscale.sh`), psycopg.
10
+
11
+ Spec: `packages/memory-engine-v2/RFC-async-escalation.md`.
12
+
13
+ ## Global Constraints
14
+
15
+ - Kill switch `ASYNC_ESCALATION_ENABLED` (env, default `false`) gates ALL new behavior; when false the worker runs today's synchronous inline cascade byte-for-byte.
16
+ - Migration 014 MUST be applied to prod BEFORE the code deploys (org-model migrations are not auto-applied to existing DBs; same discipline as 013).
17
+ - No supersede — escalated events are never written by the student, so the teacher's write is the first and only write. One graph write per event (provenance-count invariant).
18
+ - Reuse existing mechanics verbatim: fair-share claim shape, `FOR UPDATE SKIP LOCKED`, lease/`attempts` cap, `requeue_transient` (0.10.38), `_apply_extraction`, trace inserts.
19
+ - `distillation_queue.status` has a `valid_status` CHECK = `{pending,claimed,done,failed}` — it must be recreated to admit `escalated_pending`.
20
+ - The existing fair-share claim is `WHERE status='pending'` and MUST stay so (it must never claim `escalated_pending` rows).
21
+
22
+ ---
23
+
24
+ ### Task 1: Migration 014 — `escalated_pending` state
25
+
26
+ **Files:**
27
+ - Create: `packages/memory-engine-v2/org-model/migrations/014_escalated_pending.sql`
28
+
29
+ **Interfaces:**
30
+ - Produces: `distillation_queue.status` accepts `'escalated_pending'`; new column `escalation_reason text`; partial index `idx_distillation_escalated`.
31
+
32
+ - [ ] **Step 1: Write the migration**
33
+
34
+ ```sql
35
+ -- 014 — escalated_pending state for async escalation (RFC-async-escalation.md).
36
+ -- The student-primary consumer marks escalated events 'escalated_pending' (no
37
+ -- write) instead of calling the teacher inline; the teacher-mop-up consumer
38
+ -- claims and writes them once. Additive + safe online; apply BEFORE the code.
39
+
40
+ ALTER TABLE distillation_queue
41
+ ADD COLUMN IF NOT EXISTS escalation_reason text;
42
+
43
+ -- Recreate the status CHECK to admit the new value. NOT VALID + VALIDATE avoids
44
+ -- the long ACCESS EXCLUSIVE validation lock on the large prod table.
45
+ ALTER TABLE distillation_queue DROP CONSTRAINT IF EXISTS valid_status;
46
+ ALTER TABLE distillation_queue ADD CONSTRAINT valid_status
47
+ CHECK (status = ANY (ARRAY['pending','claimed','done','failed','escalated_pending']))
48
+ NOT VALID;
49
+ ALTER TABLE distillation_queue VALIDATE CONSTRAINT valid_status;
50
+ ```
51
+
52
+ - [ ] **Step 2: Write the index (separate — CONCURRENTLY can't share a txn)**
53
+
54
+ Append to the same file, but it is run as its own statement by the migration runner (psql runs each statement; CONCURRENTLY must not be inside a transaction block — the schema-manager applies these outside an explicit txn):
55
+
56
+ ```sql
57
+ -- Mop-up claim + autoscaler count. Mirrors idx_distillation_fair_claim shape.
58
+ CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_distillation_escalated
59
+ ON distillation_queue (priority, emit_ts, id) WHERE status = 'escalated_pending';
60
+ ```
61
+
62
+ - [ ] **Step 3: Validate the SQL parses (local dockerized PG if available, else psql --dry via EXPLAIN)**
63
+
64
+ Run: `psql -f packages/memory-engine-v2/org-model/migrations/014_escalated_pending.sql` against a scratch DB with the `distillation_queue` DDL loaded.
65
+ Expected: `ALTER TABLE` ×3, `CREATE INDEX`.
66
+
67
+ - [ ] **Step 4: Commit**
68
+
69
+ ```bash
70
+ git add packages/memory-engine-v2/org-model/migrations/014_escalated_pending.sql
71
+ git commit -m "feat(pme2): migration 014 — escalated_pending queue state"
72
+ ```
73
+
74
+ - [ ] **Step 5: Apply to prod (ops, before any code deploy)**
75
+
76
+ Via SSM on the engine box, run the ALTERs then the CONCURRENTLY index against `org_model`. Verify:
77
+ `SELECT pg_get_constraintdef(oid) FROM pg_constraint WHERE conname='valid_status';` includes `escalated_pending`.
78
+
79
+ ---
80
+
81
+ ### Task 2: `escalation_reason` plumbed into the queue mark
82
+
83
+ **Files:**
84
+ - Modify: `packages/memory-engine-v2/extractor-async/worker.py` (new helper near `mark_failed`/`release_claim`, ~line 2675–2713)
85
+ - Test: `packages/memory-engine-v2/extractor-async/test_async_escalation.py` (create)
86
+
87
+ **Interfaces:**
88
+ - Produces: `mark_escalated_pending(conn, queue_id, reason)` — sets `status='escalated_pending', escalation_reason=reason, claimed_by=NULL, claimed_at=NULL, claim_expires_at=NULL` and does NOT touch `attempts`.
89
+
90
+ - [ ] **Step 1: Write the failing test** (`test_async_escalation.py`)
91
+
92
+ ```python
93
+ import importlib.util, os, sys
94
+ from pathlib import Path
95
+
96
+ EXTRACTOR = Path(__file__).parent
97
+ spec = importlib.util.spec_from_file_location("worker", EXTRACTOR / "worker.py")
98
+
99
+ def test_mark_escalated_pending_sql_shape(monkeypatch):
100
+ # Import worker with deps stubbed enough to load module-level code.
101
+ import worker # noqa: assumes httpx/psycopg importable in CI env
102
+ calls = {}
103
+ class FakeCur:
104
+ def __enter__(self): return self
105
+ def __exit__(self, *a): return False
106
+ def execute(self, sql, params): calls["sql"], calls["params"] = sql, params
107
+ class FakeConn:
108
+ def cursor(self): return FakeCur()
109
+ worker.mark_escalated_pending(FakeConn(), 42, "high_value")
110
+ assert "status = 'escalated_pending'" in calls["sql"]
111
+ assert "escalation_reason = %s" in calls["sql"]
112
+ assert "attempts" not in calls["sql"] # must NOT burn the retry budget
113
+ assert calls["params"] == ("high_value", 42)
114
+ ```
115
+
116
+ - [ ] **Step 2: Run to verify it fails**
117
+
118
+ Run: `cd packages/memory-engine-v2/extractor-async && python -m pytest test_async_escalation.py::test_mark_escalated_pending_sql_shape -v`
119
+ Expected: FAIL — `AttributeError: module 'worker' has no attribute 'mark_escalated_pending'`.
120
+
121
+ - [ ] **Step 3: Implement `mark_escalated_pending`** (after `release_claim`, ~line 2705)
122
+
123
+ ```python
124
+ def mark_escalated_pending(
125
+ conn: psycopg.Connection, queue_id: int, reason: str
126
+ ) -> None:
127
+ """Student ran; a teacher-only pass is owed. Move the row to
128
+ 'escalated_pending' WITHOUT touching `attempts` (the student didn't fail —
129
+ it deferred) and clear the student's claim so the teacher-mop-up consumer
130
+ can pick it up. The student's output is NOT written (deferred-write → the
131
+ teacher's write is the first and only write; no supersede)."""
132
+ with conn.cursor() as cur:
133
+ cur.execute(
134
+ """
135
+ UPDATE distillation_queue SET
136
+ status = 'escalated_pending',
137
+ escalation_reason = %s,
138
+ claimed_by = NULL,
139
+ claimed_at = NULL,
140
+ claim_expires_at = NULL
141
+ WHERE id = %s
142
+ """,
143
+ (reason, queue_id),
144
+ )
145
+ ```
146
+
147
+ - [ ] **Step 4: Run to verify it passes**
148
+
149
+ Run: `python -m pytest test_async_escalation.py::test_mark_escalated_pending_sql_shape -v`
150
+ Expected: PASS.
151
+
152
+ - [ ] **Step 5: Commit**
153
+
154
+ ```bash
155
+ git add packages/memory-engine-v2/extractor-async/worker.py packages/memory-engine-v2/extractor-async/test_async_escalation.py
156
+ git commit -m "feat(pme2): mark_escalated_pending — defer escalated events (no attempt burn)"
157
+ ```
158
+
159
+ ---
160
+
161
+ ### Task 3: Student-primary path — defer escalation behind the flag
162
+
163
+ **Files:**
164
+ - Modify: `packages/memory-engine-v2/extractor-async/worker.py` — config block (~line 105–135) and the cascade escalation section (~line 2884–2960)
165
+ - Test: `packages/memory-engine-v2/extractor-async/test_async_escalation.py`
166
+
167
+ **Interfaces:**
168
+ - Consumes: `mark_escalated_pending` (Task 2), existing `escalation_decision`, `_apply_extraction`, `_insert_trace`.
169
+ - Produces: config flag `ASYNC_ESCALATION_ENABLED: bool`; behavior — when true, escalated items are marked `escalated_pending` and the inline teacher block is skipped.
170
+
171
+ - [ ] **Step 1: Add the flag** (config block, ~line 135, next to `CASCADE_ENABLED`)
172
+
173
+ ```python
174
+ # Async escalation (RFC-async-escalation.md). When true, the student-primary
175
+ # pass does NOT call the teacher inline; escalated events become
176
+ # 'escalated_pending' for the teacher-mop-up consumer. Default false = today's
177
+ # synchronous inline cascade (instant revert).
178
+ ASYNC_ESCALATION_ENABLED = os.environ.get("ASYNC_ESCALATION_ENABLED", "") not in ("", "0", "false")
179
+ ```
180
+
181
+ - [ ] **Step 2: Write the failing test** (control-flow: escalated item is deferred, not sent to the inline teacher, when the flag is on)
182
+
183
+ ```python
184
+ def test_async_defers_escalated(monkeypatch):
185
+ import worker
186
+ monkeypatch.setattr(worker, "ASYNC_ESCALATION_ENABLED", True)
187
+ marked = []
188
+ monkeypatch.setattr(worker, "mark_escalated_pending",
189
+ lambda conn, qid, reason: marked.append((qid, reason)))
190
+ # escalate_items handling should route through mark_escalated_pending and
191
+ # NOT append to the inline teacher batch.
192
+ teacher_batch = worker._route_escalations(
193
+ conn=object(),
194
+ escalate_items=[{"id": 7, "event_id": "e7"}],
195
+ reason_by_qid={7: "high_value"},
196
+ )
197
+ assert teacher_batch == [] # nothing goes to the inline teacher
198
+ assert marked == [(7, "high_value")]
199
+ ```
200
+
201
+ - [ ] **Step 3: Run to verify it fails**
202
+
203
+ Run: `python -m pytest test_async_escalation.py::test_async_defers_escalated -v`
204
+ Expected: FAIL — `_route_escalations` not defined.
205
+
206
+ - [ ] **Step 4: Extract the escalation routing into `_route_escalations` and branch on the flag**
207
+
208
+ Replace the current post-`_student`-loop handling (today: `escalate_items` are passed straight to the inline teacher call after ~line 2933) with a routing helper. Add:
209
+
210
+ ```python
211
+ def _route_escalations(conn, escalate_items, reason_by_qid):
212
+ """Async: mark each escalated item 'escalated_pending' and return an empty
213
+ inline-teacher batch. Sync (flag off): return the items unchanged so the
214
+ caller runs the teacher inline exactly as before."""
215
+ if not ASYNC_ESCALATION_ENABLED:
216
+ return escalate_items
217
+ for item in escalate_items:
218
+ mark_escalated_pending(conn, item["id"], reason_by_qid.get(item["id"], "escalated"))
219
+ return []
220
+ ```
221
+
222
+ Then, in the cascade section, after `escalate_items`/`reason_by_qid` are built (post ~line 2928) and BEFORE the inline teacher runs, funnel through it:
223
+
224
+ ```python
225
+ escalate_items = _route_escalations(conn, escalate_items, reason_by_qid)
226
+ # (existing inline-teacher block below now operates on the possibly-empty
227
+ # escalate_items; when async is on it's [], so the teacher isn't called.)
228
+ ```
229
+
230
+ The student agreement-trace insert (~line 2908) and the non-escalated `_apply_extraction` (~line 2924) are UNCHANGED — they run in both modes.
231
+
232
+ - [ ] **Step 5: Run to verify it passes**
233
+
234
+ Run: `python -m pytest test_async_escalation.py::test_async_defers_escalated -v`
235
+ Expected: PASS.
236
+
237
+ - [ ] **Step 6: Verify sync mode is byte-for-byte unchanged**
238
+
239
+ ```python
240
+ def test_sync_mode_passthrough(monkeypatch):
241
+ import worker
242
+ monkeypatch.setattr(worker, "ASYNC_ESCALATION_ENABLED", False)
243
+ items = [{"id": 1, "event_id": "e1"}]
244
+ assert worker._route_escalations(object(), items, {1: "x"}) is items
245
+ ```
246
+
247
+ Run: `python -m pytest test_async_escalation.py -v`
248
+ Expected: PASS (both).
249
+
250
+ - [ ] **Step 7: Commit**
251
+
252
+ ```bash
253
+ git add packages/memory-engine-v2/extractor-async/worker.py packages/memory-engine-v2/extractor-async/test_async_escalation.py
254
+ git commit -m "feat(pme2): student-primary path defers escalation behind ASYNC_ESCALATION_ENABLED"
255
+ ```
256
+
257
+ ---
258
+
259
+ ### Task 4: Teacher-mop-up consumer — claim & write `escalated_pending`
260
+
261
+ **Files:**
262
+ - Modify: `packages/memory-engine-v2/extractor-async/worker.py` — new `claim_escalated_batch` (near `claim_next_batch`, ~line 2440) and wire into `amain` (~line where the poll loop lives)
263
+ - Test: `packages/memory-engine-v2/extractor-async/test_async_escalation.py`
264
+
265
+ **Interfaces:**
266
+ - Consumes: existing teacher call path (the inline-teacher logic from Task 3's untouched block), `_apply_extraction(producer="teacher")`, `requeue_transient`, `mark_failed`.
267
+ - Produces: `claim_escalated_batch(conn, worker_id, batch_size, claim_ttl, max_attempts) -> list[dict]` — claims `escalated_pending` rows (leased), returns `id, event_id, attempts, transient_attempts, escalation_reason`.
268
+
269
+ - [ ] **Step 1: Write the failing test** (claim SQL targets escalated_pending, leases, respects attempts)
270
+
271
+ ```python
272
+ def test_claim_escalated_sql(monkeypatch):
273
+ import worker
274
+ captured = {}
275
+ class FakeCur:
276
+ def __enter__(self): return self
277
+ def __exit__(self, *a): return False
278
+ def execute(self, sql, params): captured["sql"] = sql
279
+ def fetchall(self): return []
280
+ class FakeConn:
281
+ def cursor(self, **k): return FakeCur()
282
+ worker.claim_escalated_batch(FakeConn(), "w1", 16, 900, 3)
283
+ s = captured["sql"]
284
+ assert "status = 'escalated_pending'" in s
285
+ assert "FOR UPDATE SKIP LOCKED" in s
286
+ assert "claim_expires_at" in s # leased like the main claim
287
+ assert "attempts < %s" in s
288
+ ```
289
+
290
+ - [ ] **Step 2: Run to verify it fails**
291
+
292
+ Run: `python -m pytest test_async_escalation.py::test_claim_escalated_sql -v`
293
+ Expected: FAIL — `claim_escalated_batch` not defined.
294
+
295
+ - [ ] **Step 3: Implement `claim_escalated_batch`** (mirror `claim_next_batch` but simpler — single status, priority/emit_ts order, no student fair-share needed since it's the teacher tier)
296
+
297
+ ```python
298
+ def claim_escalated_batch(conn, worker_id, batch_size, claim_ttl_sec, max_attempts):
299
+ """Teacher-mop-up claim: lease escalated_pending rows (priority, emit_ts, id),
300
+ crash-safe via claim_expires_at. Served by idx_distillation_escalated."""
301
+ with conn.cursor(row_factory=psycopg.rows.dict_row) as cur:
302
+ cur.execute(
303
+ """
304
+ UPDATE distillation_queue SET
305
+ status = 'claimed',
306
+ claimed_by = %s,
307
+ claimed_at = NOW(),
308
+ claim_expires_at = NOW() + (%s || ' seconds')::interval
309
+ WHERE id IN (
310
+ SELECT id FROM distillation_queue
311
+ WHERE status = 'escalated_pending' AND attempts < %s
312
+ ORDER BY COALESCE(priority, 100), emit_ts, id
313
+ LIMIT %s
314
+ FOR UPDATE SKIP LOCKED
315
+ )
316
+ RETURNING id, event_id, attempts, transient_attempts, escalation_reason
317
+ """,
318
+ (worker_id, claim_ttl_sec, max_attempts, batch_size),
319
+ )
320
+ return cur.fetchall()
321
+ ```
322
+
323
+ Note: escalated rows re-claimed after a lease expiry re-enter as `status='claimed'`; on completion they go to `done`. A crashed mop-up leaves them `claimed` → lease expires → re-claimed here (the `status='escalated_pending'` filter misses expired-claimed rows, so ALSO include the expired-claimed re-entry — mirror `claim_next_batch`'s `OR (status='claimed' AND claim_expires_at < NOW())` but scoped to rows whose prior state was escalated). Simplest: the mop-up claim's WHERE becomes `((status='escalated_pending') OR (status='claimed' AND claim_expires_at < NOW() AND escalation_reason IS NOT NULL)) AND attempts < %s`.
324
+
325
+ - [ ] **Step 4: Update the claim to handle expired re-claim** (apply the WHERE from the note above), re-run the test asserting both branches present:
326
+
327
+ ```python
328
+ assert "escalation_reason IS NOT NULL" in s # expired-claimed re-entry scoped to escalated rows
329
+ ```
330
+
331
+ - [ ] **Step 5: Run to verify it passes**
332
+
333
+ Run: `python -m pytest test_async_escalation.py::test_claim_escalated_sql -v`
334
+ Expected: PASS.
335
+
336
+ - [ ] **Step 6: Wire the mop-up loop into `amain`**
337
+
338
+ When `ASYNC_ESCALATION_ENABLED`, the worker alternates: drain `pending` (student-primary) and drain `escalated_pending` (teacher-mop-up) using the SAME teacher processing code path that the inline cascade uses today (call teacher vLLM → `_apply_extraction(producer="teacher")` → teacher `_insert_trace` → `mark_done`; transient errors → `requeue_transient`; exhausted → `mark_failed`). Reuse the existing teacher batch function; feed it `claim_escalated_batch` results instead of the inline `escalate_items`.
339
+
340
+ ```python
341
+ # in the poll loop, when ASYNC_ESCALATION_ENABLED:
342
+ esc = claim_escalated_batch(conn, WORKER_ID, BATCH_SIZE, CLAIM_TTL_SEC, MAX_ATTEMPTS)
343
+ if esc:
344
+ await _process_teacher_batch(conn, http, esc) # the extracted inline-teacher path
345
+ ```
346
+
347
+ (Extract today's inline-teacher block into `_process_teacher_batch(conn, http, items)` so it is shared by the sync cascade and the async mop-up — no logic change, just a callable.)
348
+
349
+ - [ ] **Step 7: Integration test (stubbed LLM)** — one escalated event flows pending→escalated_pending→done, written once by the teacher.
350
+
351
+ ```python
352
+ def test_end_to_end_escalated_written_once(async_pg, stub_teacher):
353
+ # async_pg: fixture with distillation_queue + one pending event that the
354
+ # stubbed student escalates. stub_teacher returns a fixed extraction.
355
+ run_student_pass(async_pg) # marks escalated_pending, no write
356
+ assert queue_status(async_pg, event="e1") == "escalated_pending"
357
+ assert graph_rows(async_pg, "e1") == 0
358
+ run_mopup_pass(async_pg, stub_teacher)
359
+ assert queue_status(async_pg, event="e1") == "done"
360
+ assert graph_rows(async_pg, "e1") == 1 # written exactly once, by teacher
361
+ ```
362
+
363
+ Run: `python -m pytest test_async_escalation.py::test_end_to_end_escalated_written_once -v`
364
+ Expected: PASS.
365
+
366
+ - [ ] **Step 8: Commit**
367
+
368
+ ```bash
369
+ git add packages/memory-engine-v2/extractor-async/worker.py packages/memory-engine-v2/extractor-async/test_async_escalation.py
370
+ git commit -m "feat(pme2): teacher-mop-up consumer drains escalated_pending"
371
+ ```
372
+
373
+ ---
374
+
375
+ ### Task 5: Autoscaler counts `escalated_pending`
376
+
377
+ **Files:**
378
+ - Modify: `thing-event-system/modules/seesa/deploy/ops/distiller-autoscale.sh` (the `PENDING`/`OLDEST_MIN` queries, ~line 57–58)
379
+
380
+ **Interfaces:**
381
+ - Consumes: migration 014 (`escalated_pending` rows exist).
382
+ - Produces: the fleet scales up on escalated backlog (the teacher's actual work under async).
383
+
384
+ - [ ] **Step 1: Update the depth + age queries to include escalated_pending**
385
+
386
+ ```sh
387
+ PENDING=$(PSQL "SELECT count(*) FROM distillation_queue WHERE status IN ('pending','escalated_pending')"); PENDING=${PENDING:-0}
388
+ OLDEST_MIN=$(PSQL "SELECT coalesce(round(extract(epoch from now()-min(enqueued_at))/60)::int,0) FROM distillation_queue WHERE status IN ('pending','escalated_pending')"); OLDEST_MIN=${OLDEST_MIN:-0}
389
+ ```
390
+
391
+ Rationale comment to add: under async escalation the teacher's work is `escalated_pending`; the student drains `pending` on the always-on L4, so the fleet must wake on escalated depth/age. (Pre-async there are no escalated_pending rows, so this is a no-op until the flag flips — safe to ship first.)
392
+
393
+ - [ ] **Step 2: Syntax check + simulate**
394
+
395
+ Run: `bash -n modules/seesa/deploy/ops/distiller-autoscale.sh` → OK. Re-run the desired-count simulation from the age-aware change to confirm unchanged behavior when escalated=0.
396
+
397
+ - [ ] **Step 3: Commit (TES repo)**
398
+
399
+ ```bash
400
+ git add modules/seesa/deploy/ops/distiller-autoscale.sh
401
+ git commit -m "feat(seesa-deploy): autoscaler counts escalated_pending for async escalation"
402
+ ```
403
+
404
+ ---
405
+
406
+ ### Task 6: Redistill produces `escalated_pending` (teacher-only)
407
+
408
+ **Files:**
409
+ - Modify: `packages/memory-engine-v2/scripts/redistill.py` — the enqueue INSERT (~line 219)
410
+ - Test: `packages/memory-engine-v2/scripts/test_redistill.py` (create or extend)
411
+
412
+ **Interfaces:**
413
+ - Consumes: `escalated_pending` state (Task 1).
414
+ - Produces: redistilled events go straight to the teacher (bypass the student), fixing the latent "hits student first" gap.
415
+
416
+ - [ ] **Step 1: Write the failing test** — enqueue writes `status='escalated_pending', escalation_reason='redistill'`.
417
+
418
+ ```python
419
+ def test_redistill_enqueues_teacher_only():
420
+ from redistill import ENQUEUE_SQL # extract the INSERT to a module constant
421
+ assert "escalated_pending" in ENQUEUE_SQL
422
+ assert "'redistill'" in ENQUEUE_SQL
423
+ ```
424
+
425
+ - [ ] **Step 2: Run to verify it fails**
426
+
427
+ Run: `cd packages/memory-engine-v2/scripts && python -m pytest test_redistill.py::test_redistill_enqueues_teacher_only -v`
428
+ Expected: FAIL — `ENQUEUE_SQL` not defined / value mismatch.
429
+
430
+ - [ ] **Step 3: Change the enqueue** (redistill.py ~line 219). Only under async mode (gate on the same env flag so a redistill run against a pre-async engine still works):
431
+
432
+ ```python
433
+ ENQUEUE_SQL = (
434
+ "INSERT INTO distillation_queue (event_id, status, attempts, priority, escalation_reason) "
435
+ "VALUES (%s, 'escalated_pending', 0, 200, 'redistill')"
436
+ if os.environ.get("ASYNC_ESCALATION_ENABLED", "") not in ("", "0", "false")
437
+ else "INSERT INTO distillation_queue (event_id, status, attempts, priority) "
438
+ "VALUES (%s, 'pending', 0, 200)"
439
+ )
440
+ ...
441
+ cur.execute(ENQUEUE_SQL, (eid,))
442
+ ```
443
+
444
+ Keep `--supersede-facts` unchanged — redistill still re-processes already-written events, so it still needs its own supersede (async escalation does not inherit that).
445
+
446
+ - [ ] **Step 4: Run to verify it passes**
447
+
448
+ Run: `python -m pytest test_redistill.py::test_redistill_enqueues_teacher_only -v` (with `ASYNC_ESCALATION_ENABLED=1`)
449
+ Expected: PASS.
450
+
451
+ - [ ] **Step 5: Commit**
452
+
453
+ ```bash
454
+ git add packages/memory-engine-v2/scripts/redistill.py packages/memory-engine-v2/scripts/test_redistill.py
455
+ git commit -m "feat(pme2): redistill enqueues escalated_pending (teacher-only) under async"
456
+ ```
457
+
458
+ ---
459
+
460
+ ### Task 7: Version bump + release + rollout
461
+
462
+ **Files:**
463
+ - Modify: `package.json` (version bump)
464
+
465
+ - [ ] **Step 1: Bump version** (patch bump, e.g. 0.10.39 → next).
466
+
467
+ - [ ] **Step 2: Apply migration 014 to prod** (Task 1 Step 5) and verify the constraint before deploying code.
468
+
469
+ - [ ] **Step 3: Ship dark** — merge + tag; the release chain deploys the worker with `ASYNC_ESCALATION_ENABLED` unset (false). Verify prod behaves identically (sync cascade), `escalated_pending` count = 0.
470
+
471
+ - [ ] **Step 4: Merge the TES autoscaler + pin-bump**, deploy.
472
+
473
+ - [ ] **Step 5: Canary** — set `ASYNC_ESCALATION_ENABLED=true` (SSM env) for the worker; watch: ingest drain, `escalated_pending` depth/oldest-age, teacher-fleet wake, graph-diff vs a control window, per-event provenance-count invariant (exactly one write/event).
474
+
475
+ - [ ] **Step 6: Ramp to 100%**; then enable redistill's teacher-only path.
476
+
477
+ ---
478
+
479
+ ## Self-Review
480
+
481
+ **Spec coverage:**
482
+ - escalated_pending primitive → Task 1 (schema) + Tasks 2–4 (producer/consumer). ✓
483
+ - Two-consumer split → Task 3 (student defers) + Task 4 (mop-up). ✓
484
+ - No supersede (deferred write) → Task 2 (`mark_escalated_pending`, no student write). ✓
485
+ - Kill switch → Task 3 Step 1 flag, gated everywhere. ✓
486
+ - Autoscaler signal → Task 5. ✓
487
+ - Redistill refactor → Task 6. ✓
488
+ - Migration + valid_status CHECK → Task 1. ✓
489
+ - Priority (escalated inherits priority/emit_ts) → Task 4 claim `ORDER BY priority, emit_ts`. ✓
490
+ - Graph-cleaning coexistence → no code change needed (RFC §6); nothing in the plan touches fusion/decay. ✓
491
+ - Rollout/dark-launch → Task 7. ✓
492
+
493
+ **Placeholder scan:** all steps carry concrete SQL/Python/commands. The `_process_teacher_batch` extraction (Task 4 Step 6) references today's inline-teacher block — the implementer extracts it verbatim (no logic change); code shown at call-site.
494
+
495
+ **Type consistency:** `mark_escalated_pending(conn, queue_id, reason)` (Task 2) used identically in Task 3 `_route_escalations`. `claim_escalated_batch` RETURNING columns (`id, event_id, attempts, transient_attempts, escalation_reason`) match the fields the teacher path + `requeue_transient` consume. `ASYNC_ESCALATION_ENABLED` env-string check identical in worker.py and redistill.py.
496
+
497
+ **Open implementation note:** worker.py can't be imported without `httpx`/`psycopg` present; the unit tests assume the CI env has them (the PyPI test job installs the package). If not, gate the import-heavy tests behind an availability check.