@pentatonic-ai/ai-agent-sdk 0.10.38 → 0.10.39
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/index.cjs +1 -1
- package/dist/index.js +1 -1
- package/package.json +1 -1
- package/packages/memory-engine-v2/PLAN-async-escalation.md +497 -0
- package/packages/memory-engine-v2/RFC-async-escalation.md +218 -0
- package/packages/memory-engine-v2/extractor-async/test_async_escalation.py +314 -0
- package/packages/memory-engine-v2/extractor-async/worker.py +173 -9
- package/packages/memory-engine-v2/org-model/migrations/014_escalated_pending.sql +22 -0
- package/packages/memory-engine-v2/scripts/redistill.py +25 -9
- package/packages/memory-engine-v2/scripts/test_redistill.py +81 -0
package/dist/index.cjs
CHANGED
|
@@ -878,7 +878,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
|
|
|
878
878
|
}
|
|
879
879
|
|
|
880
880
|
// src/telemetry.js
|
|
881
|
-
var VERSION = "0.10.
|
|
881
|
+
var VERSION = "0.10.39";
|
|
882
882
|
var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
|
|
883
883
|
function machineId() {
|
|
884
884
|
const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";
|
package/dist/index.js
CHANGED
|
@@ -847,7 +847,7 @@ function fireAndForgetEmit(clientConfig, sessionOpts, messages, result, model) {
|
|
|
847
847
|
}
|
|
848
848
|
|
|
849
849
|
// src/telemetry.js
|
|
850
|
-
var VERSION = "0.10.
|
|
850
|
+
var VERSION = "0.10.39";
|
|
851
851
|
var TELEMETRY_URL = "https://sdk-telemetry.philip-134.workers.dev";
|
|
852
852
|
function machineId() {
|
|
853
853
|
const raw = typeof process !== "undefined" ? `${process.env?.USER || process.env?.USERNAME || "u"}:${process.platform || "x"}:${process.arch || "x"}` : "browser";
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@pentatonic-ai/ai-agent-sdk",
|
|
3
|
-
"version": "0.10.
|
|
3
|
+
"version": "0.10.39",
|
|
4
4
|
"description": "TES SDK — LLM observability and lifecycle tracking via Pentatonic Thing Event System. Track token usage, tool calls, and conversations. Manage things through event-sourced lifecycle stages with AI enrichment and vector search.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./dist/index.cjs",
|
|
@@ -0,0 +1,497 @@
|
|
|
1
|
+
# Async Escalation Implementation Plan
|
|
2
|
+
|
|
3
|
+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
4
|
+
|
|
5
|
+
**Goal:** Take the teacher off the ingest critical path — the student writes the confident majority synchronously; escalated events become an `escalated_pending` queue state drained asynchronously by the existing L40S mop-up fleet.
|
|
6
|
+
|
|
7
|
+
**Architecture:** One `distillation_queue`, two consumers. The student-primary consumer (always-on L4) claims `pending`, writes what it's confident on, and marks the rest `escalated_pending` instead of calling the teacher inline. A teacher-mop-up consumer claims `escalated_pending` and writes it once via the teacher vLLM. Deferred first-write ⇒ no supersede. Everything ships dark behind `ASYNC_ESCALATION_ENABLED` (default false) with instant revert to today's synchronous cascade.
|
|
8
|
+
|
|
9
|
+
**Tech Stack:** Python 3.12 (`extractor-async/worker.py`), Postgres (`distillation_queue`, org-model migrations), bash systemd ops (TES `distiller-autoscale.sh`), psycopg.
|
|
10
|
+
|
|
11
|
+
Spec: `packages/memory-engine-v2/RFC-async-escalation.md`.
|
|
12
|
+
|
|
13
|
+
## Global Constraints
|
|
14
|
+
|
|
15
|
+
- Kill switch `ASYNC_ESCALATION_ENABLED` (env, default `false`) gates ALL new behavior; when false the worker runs today's synchronous inline cascade byte-for-byte.
|
|
16
|
+
- Migration 014 MUST be applied to prod BEFORE the code deploys (org-model migrations are not auto-applied to existing DBs; same discipline as 013).
|
|
17
|
+
- No supersede — escalated events are never written by the student, so the teacher's write is the first and only write. One graph write per event (provenance-count invariant).
|
|
18
|
+
- Reuse existing mechanics verbatim: fair-share claim shape, `FOR UPDATE SKIP LOCKED`, lease/`attempts` cap, `requeue_transient` (0.10.38), `_apply_extraction`, trace inserts.
|
|
19
|
+
- `distillation_queue.status` has a `valid_status` CHECK = `{pending,claimed,done,failed}` — it must be recreated to admit `escalated_pending`.
|
|
20
|
+
- The existing fair-share claim is `WHERE status='pending'` and MUST stay so (it must never claim `escalated_pending` rows).
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
### Task 1: Migration 014 — `escalated_pending` state
|
|
25
|
+
|
|
26
|
+
**Files:**
|
|
27
|
+
- Create: `packages/memory-engine-v2/org-model/migrations/014_escalated_pending.sql`
|
|
28
|
+
|
|
29
|
+
**Interfaces:**
|
|
30
|
+
- Produces: `distillation_queue.status` accepts `'escalated_pending'`; new column `escalation_reason text`; partial index `idx_distillation_escalated`.
|
|
31
|
+
|
|
32
|
+
- [ ] **Step 1: Write the migration**
|
|
33
|
+
|
|
34
|
+
```sql
|
|
35
|
+
-- 014 — escalated_pending state for async escalation (RFC-async-escalation.md).
|
|
36
|
+
-- The student-primary consumer marks escalated events 'escalated_pending' (no
|
|
37
|
+
-- write) instead of calling the teacher inline; the teacher-mop-up consumer
|
|
38
|
+
-- claims and writes them once. Additive + safe online; apply BEFORE the code.
|
|
39
|
+
|
|
40
|
+
ALTER TABLE distillation_queue
|
|
41
|
+
ADD COLUMN IF NOT EXISTS escalation_reason text;
|
|
42
|
+
|
|
43
|
+
-- Recreate the status CHECK to admit the new value. NOT VALID + VALIDATE avoids
|
|
44
|
+
-- the long ACCESS EXCLUSIVE validation lock on the large prod table.
|
|
45
|
+
ALTER TABLE distillation_queue DROP CONSTRAINT IF EXISTS valid_status;
|
|
46
|
+
ALTER TABLE distillation_queue ADD CONSTRAINT valid_status
|
|
47
|
+
CHECK (status = ANY (ARRAY['pending','claimed','done','failed','escalated_pending']))
|
|
48
|
+
NOT VALID;
|
|
49
|
+
ALTER TABLE distillation_queue VALIDATE CONSTRAINT valid_status;
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
- [ ] **Step 2: Write the index (separate — CONCURRENTLY can't share a txn)**
|
|
53
|
+
|
|
54
|
+
Append to the same file, but it is run as its own statement by the migration runner (psql runs each statement; CONCURRENTLY must not be inside a transaction block — the schema-manager applies these outside an explicit txn):
|
|
55
|
+
|
|
56
|
+
```sql
|
|
57
|
+
-- Mop-up claim + autoscaler count. Mirrors idx_distillation_fair_claim shape.
|
|
58
|
+
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_distillation_escalated
|
|
59
|
+
ON distillation_queue (priority, emit_ts, id) WHERE status = 'escalated_pending';
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
- [ ] **Step 3: Validate the SQL parses (local dockerized PG if available, else psql --dry via EXPLAIN)**
|
|
63
|
+
|
|
64
|
+
Run: `psql -f packages/memory-engine-v2/org-model/migrations/014_escalated_pending.sql` against a scratch DB with the `distillation_queue` DDL loaded.
|
|
65
|
+
Expected: `ALTER TABLE` ×3, `CREATE INDEX`.
|
|
66
|
+
|
|
67
|
+
- [ ] **Step 4: Commit**
|
|
68
|
+
|
|
69
|
+
```bash
|
|
70
|
+
git add packages/memory-engine-v2/org-model/migrations/014_escalated_pending.sql
|
|
71
|
+
git commit -m "feat(pme2): migration 014 — escalated_pending queue state"
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
- [ ] **Step 5: Apply to prod (ops, before any code deploy)**
|
|
75
|
+
|
|
76
|
+
Via SSM on the engine box, run the ALTERs then the CONCURRENTLY index against `org_model`. Verify:
|
|
77
|
+
`SELECT pg_get_constraintdef(oid) FROM pg_constraint WHERE conname='valid_status';` includes `escalated_pending`.
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
### Task 2: `escalation_reason` plumbed into the queue mark
|
|
82
|
+
|
|
83
|
+
**Files:**
|
|
84
|
+
- Modify: `packages/memory-engine-v2/extractor-async/worker.py` (new helper near `mark_failed`/`release_claim`, ~line 2675–2713)
|
|
85
|
+
- Test: `packages/memory-engine-v2/extractor-async/test_async_escalation.py` (create)
|
|
86
|
+
|
|
87
|
+
**Interfaces:**
|
|
88
|
+
- Produces: `mark_escalated_pending(conn, queue_id, reason)` — sets `status='escalated_pending', escalation_reason=reason, claimed_by=NULL, claimed_at=NULL, claim_expires_at=NULL` and does NOT touch `attempts`.
|
|
89
|
+
|
|
90
|
+
- [ ] **Step 1: Write the failing test** (`test_async_escalation.py`)
|
|
91
|
+
|
|
92
|
+
```python
|
|
93
|
+
import importlib.util, os, sys
|
|
94
|
+
from pathlib import Path
|
|
95
|
+
|
|
96
|
+
EXTRACTOR = Path(__file__).parent
|
|
97
|
+
spec = importlib.util.spec_from_file_location("worker", EXTRACTOR / "worker.py")
|
|
98
|
+
|
|
99
|
+
def test_mark_escalated_pending_sql_shape(monkeypatch):
|
|
100
|
+
# Import worker with deps stubbed enough to load module-level code.
|
|
101
|
+
import worker # noqa: assumes httpx/psycopg importable in CI env
|
|
102
|
+
calls = {}
|
|
103
|
+
class FakeCur:
|
|
104
|
+
def __enter__(self): return self
|
|
105
|
+
def __exit__(self, *a): return False
|
|
106
|
+
def execute(self, sql, params): calls["sql"], calls["params"] = sql, params
|
|
107
|
+
class FakeConn:
|
|
108
|
+
def cursor(self): return FakeCur()
|
|
109
|
+
worker.mark_escalated_pending(FakeConn(), 42, "high_value")
|
|
110
|
+
assert "status = 'escalated_pending'" in calls["sql"]
|
|
111
|
+
assert "escalation_reason = %s" in calls["sql"]
|
|
112
|
+
assert "attempts" not in calls["sql"] # must NOT burn the retry budget
|
|
113
|
+
assert calls["params"] == ("high_value", 42)
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
- [ ] **Step 2: Run to verify it fails**
|
|
117
|
+
|
|
118
|
+
Run: `cd packages/memory-engine-v2/extractor-async && python -m pytest test_async_escalation.py::test_mark_escalated_pending_sql_shape -v`
|
|
119
|
+
Expected: FAIL — `AttributeError: module 'worker' has no attribute 'mark_escalated_pending'`.
|
|
120
|
+
|
|
121
|
+
- [ ] **Step 3: Implement `mark_escalated_pending`** (after `release_claim`, ~line 2705)
|
|
122
|
+
|
|
123
|
+
```python
|
|
124
|
+
def mark_escalated_pending(
|
|
125
|
+
conn: psycopg.Connection, queue_id: int, reason: str
|
|
126
|
+
) -> None:
|
|
127
|
+
"""Student ran; a teacher-only pass is owed. Move the row to
|
|
128
|
+
'escalated_pending' WITHOUT touching `attempts` (the student didn't fail —
|
|
129
|
+
it deferred) and clear the student's claim so the teacher-mop-up consumer
|
|
130
|
+
can pick it up. The student's output is NOT written (deferred-write → the
|
|
131
|
+
teacher's write is the first and only write; no supersede)."""
|
|
132
|
+
with conn.cursor() as cur:
|
|
133
|
+
cur.execute(
|
|
134
|
+
"""
|
|
135
|
+
UPDATE distillation_queue SET
|
|
136
|
+
status = 'escalated_pending',
|
|
137
|
+
escalation_reason = %s,
|
|
138
|
+
claimed_by = NULL,
|
|
139
|
+
claimed_at = NULL,
|
|
140
|
+
claim_expires_at = NULL
|
|
141
|
+
WHERE id = %s
|
|
142
|
+
""",
|
|
143
|
+
(reason, queue_id),
|
|
144
|
+
)
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
- [ ] **Step 4: Run to verify it passes**
|
|
148
|
+
|
|
149
|
+
Run: `python -m pytest test_async_escalation.py::test_mark_escalated_pending_sql_shape -v`
|
|
150
|
+
Expected: PASS.
|
|
151
|
+
|
|
152
|
+
- [ ] **Step 5: Commit**
|
|
153
|
+
|
|
154
|
+
```bash
|
|
155
|
+
git add packages/memory-engine-v2/extractor-async/worker.py packages/memory-engine-v2/extractor-async/test_async_escalation.py
|
|
156
|
+
git commit -m "feat(pme2): mark_escalated_pending — defer escalated events (no attempt burn)"
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
---
|
|
160
|
+
|
|
161
|
+
### Task 3: Student-primary path — defer escalation behind the flag
|
|
162
|
+
|
|
163
|
+
**Files:**
|
|
164
|
+
- Modify: `packages/memory-engine-v2/extractor-async/worker.py` — config block (~line 105–135) and the cascade escalation section (~line 2884–2960)
|
|
165
|
+
- Test: `packages/memory-engine-v2/extractor-async/test_async_escalation.py`
|
|
166
|
+
|
|
167
|
+
**Interfaces:**
|
|
168
|
+
- Consumes: `mark_escalated_pending` (Task 2), existing `escalation_decision`, `_apply_extraction`, `_insert_trace`.
|
|
169
|
+
- Produces: config flag `ASYNC_ESCALATION_ENABLED: bool`; behavior — when true, escalated items are marked `escalated_pending` and the inline teacher block is skipped.
|
|
170
|
+
|
|
171
|
+
- [ ] **Step 1: Add the flag** (config block, ~line 135, next to `CASCADE_ENABLED`)
|
|
172
|
+
|
|
173
|
+
```python
|
|
174
|
+
# Async escalation (RFC-async-escalation.md). When true, the student-primary
|
|
175
|
+
# pass does NOT call the teacher inline; escalated events become
|
|
176
|
+
# 'escalated_pending' for the teacher-mop-up consumer. Default false = today's
|
|
177
|
+
# synchronous inline cascade (instant revert).
|
|
178
|
+
ASYNC_ESCALATION_ENABLED = os.environ.get("ASYNC_ESCALATION_ENABLED", "") not in ("", "0", "false")
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
- [ ] **Step 2: Write the failing test** (control-flow: escalated item is deferred, not sent to the inline teacher, when the flag is on)
|
|
182
|
+
|
|
183
|
+
```python
|
|
184
|
+
def test_async_defers_escalated(monkeypatch):
|
|
185
|
+
import worker
|
|
186
|
+
monkeypatch.setattr(worker, "ASYNC_ESCALATION_ENABLED", True)
|
|
187
|
+
marked = []
|
|
188
|
+
monkeypatch.setattr(worker, "mark_escalated_pending",
|
|
189
|
+
lambda conn, qid, reason: marked.append((qid, reason)))
|
|
190
|
+
# escalate_items handling should route through mark_escalated_pending and
|
|
191
|
+
# NOT append to the inline teacher batch.
|
|
192
|
+
teacher_batch = worker._route_escalations(
|
|
193
|
+
conn=object(),
|
|
194
|
+
escalate_items=[{"id": 7, "event_id": "e7"}],
|
|
195
|
+
reason_by_qid={7: "high_value"},
|
|
196
|
+
)
|
|
197
|
+
assert teacher_batch == [] # nothing goes to the inline teacher
|
|
198
|
+
assert marked == [(7, "high_value")]
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
- [ ] **Step 3: Run to verify it fails**
|
|
202
|
+
|
|
203
|
+
Run: `python -m pytest test_async_escalation.py::test_async_defers_escalated -v`
|
|
204
|
+
Expected: FAIL — `_route_escalations` not defined.
|
|
205
|
+
|
|
206
|
+
- [ ] **Step 4: Extract the escalation routing into `_route_escalations` and branch on the flag**
|
|
207
|
+
|
|
208
|
+
Replace the current post-`_student`-loop handling (today: `escalate_items` are passed straight to the inline teacher call after ~line 2933) with a routing helper. Add:
|
|
209
|
+
|
|
210
|
+
```python
|
|
211
|
+
def _route_escalations(conn, escalate_items, reason_by_qid):
|
|
212
|
+
"""Async: mark each escalated item 'escalated_pending' and return an empty
|
|
213
|
+
inline-teacher batch. Sync (flag off): return the items unchanged so the
|
|
214
|
+
caller runs the teacher inline exactly as before."""
|
|
215
|
+
if not ASYNC_ESCALATION_ENABLED:
|
|
216
|
+
return escalate_items
|
|
217
|
+
for item in escalate_items:
|
|
218
|
+
mark_escalated_pending(conn, item["id"], reason_by_qid.get(item["id"], "escalated"))
|
|
219
|
+
return []
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
Then, in the cascade section, after `escalate_items`/`reason_by_qid` are built (post ~line 2928) and BEFORE the inline teacher runs, funnel through it:
|
|
223
|
+
|
|
224
|
+
```python
|
|
225
|
+
escalate_items = _route_escalations(conn, escalate_items, reason_by_qid)
|
|
226
|
+
# (existing inline-teacher block below now operates on the possibly-empty
|
|
227
|
+
# escalate_items; when async is on it's [], so the teacher isn't called.)
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
The student agreement-trace insert (~line 2908) and the non-escalated `_apply_extraction` (~line 2924) are UNCHANGED — they run in both modes.
|
|
231
|
+
|
|
232
|
+
- [ ] **Step 5: Run to verify it passes**
|
|
233
|
+
|
|
234
|
+
Run: `python -m pytest test_async_escalation.py::test_async_defers_escalated -v`
|
|
235
|
+
Expected: PASS.
|
|
236
|
+
|
|
237
|
+
- [ ] **Step 6: Verify sync mode is byte-for-byte unchanged**
|
|
238
|
+
|
|
239
|
+
```python
|
|
240
|
+
def test_sync_mode_passthrough(monkeypatch):
|
|
241
|
+
import worker
|
|
242
|
+
monkeypatch.setattr(worker, "ASYNC_ESCALATION_ENABLED", False)
|
|
243
|
+
items = [{"id": 1, "event_id": "e1"}]
|
|
244
|
+
assert worker._route_escalations(object(), items, {1: "x"}) is items
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
Run: `python -m pytest test_async_escalation.py -v`
|
|
248
|
+
Expected: PASS (both).
|
|
249
|
+
|
|
250
|
+
- [ ] **Step 7: Commit**
|
|
251
|
+
|
|
252
|
+
```bash
|
|
253
|
+
git add packages/memory-engine-v2/extractor-async/worker.py packages/memory-engine-v2/extractor-async/test_async_escalation.py
|
|
254
|
+
git commit -m "feat(pme2): student-primary path defers escalation behind ASYNC_ESCALATION_ENABLED"
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
---
|
|
258
|
+
|
|
259
|
+
### Task 4: Teacher-mop-up consumer — claim & write `escalated_pending`
|
|
260
|
+
|
|
261
|
+
**Files:**
|
|
262
|
+
- Modify: `packages/memory-engine-v2/extractor-async/worker.py` — new `claim_escalated_batch` (near `claim_next_batch`, ~line 2440) and wire into `amain` (~line where the poll loop lives)
|
|
263
|
+
- Test: `packages/memory-engine-v2/extractor-async/test_async_escalation.py`
|
|
264
|
+
|
|
265
|
+
**Interfaces:**
|
|
266
|
+
- Consumes: existing teacher call path (the inline-teacher logic from Task 3's untouched block), `_apply_extraction(producer="teacher")`, `requeue_transient`, `mark_failed`.
|
|
267
|
+
- Produces: `claim_escalated_batch(conn, worker_id, batch_size, claim_ttl, max_attempts) -> list[dict]` — claims `escalated_pending` rows (leased), returns `id, event_id, attempts, transient_attempts, escalation_reason`.
|
|
268
|
+
|
|
269
|
+
- [ ] **Step 1: Write the failing test** (claim SQL targets escalated_pending, leases, respects attempts)
|
|
270
|
+
|
|
271
|
+
```python
|
|
272
|
+
def test_claim_escalated_sql(monkeypatch):
|
|
273
|
+
import worker
|
|
274
|
+
captured = {}
|
|
275
|
+
class FakeCur:
|
|
276
|
+
def __enter__(self): return self
|
|
277
|
+
def __exit__(self, *a): return False
|
|
278
|
+
def execute(self, sql, params): captured["sql"] = sql
|
|
279
|
+
def fetchall(self): return []
|
|
280
|
+
class FakeConn:
|
|
281
|
+
def cursor(self, **k): return FakeCur()
|
|
282
|
+
worker.claim_escalated_batch(FakeConn(), "w1", 16, 900, 3)
|
|
283
|
+
s = captured["sql"]
|
|
284
|
+
assert "status = 'escalated_pending'" in s
|
|
285
|
+
assert "FOR UPDATE SKIP LOCKED" in s
|
|
286
|
+
assert "claim_expires_at" in s # leased like the main claim
|
|
287
|
+
assert "attempts < %s" in s
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
- [ ] **Step 2: Run to verify it fails**
|
|
291
|
+
|
|
292
|
+
Run: `python -m pytest test_async_escalation.py::test_claim_escalated_sql -v`
|
|
293
|
+
Expected: FAIL — `claim_escalated_batch` not defined.
|
|
294
|
+
|
|
295
|
+
- [ ] **Step 3: Implement `claim_escalated_batch`** (mirror `claim_next_batch` but simpler — single status, priority/emit_ts order, no student fair-share needed since it's the teacher tier)
|
|
296
|
+
|
|
297
|
+
```python
|
|
298
|
+
def claim_escalated_batch(conn, worker_id, batch_size, claim_ttl_sec, max_attempts):
|
|
299
|
+
"""Teacher-mop-up claim: lease escalated_pending rows (priority, emit_ts, id),
|
|
300
|
+
crash-safe via claim_expires_at. Served by idx_distillation_escalated."""
|
|
301
|
+
with conn.cursor(row_factory=psycopg.rows.dict_row) as cur:
|
|
302
|
+
cur.execute(
|
|
303
|
+
"""
|
|
304
|
+
UPDATE distillation_queue SET
|
|
305
|
+
status = 'claimed',
|
|
306
|
+
claimed_by = %s,
|
|
307
|
+
claimed_at = NOW(),
|
|
308
|
+
claim_expires_at = NOW() + (%s || ' seconds')::interval
|
|
309
|
+
WHERE id IN (
|
|
310
|
+
SELECT id FROM distillation_queue
|
|
311
|
+
WHERE status = 'escalated_pending' AND attempts < %s
|
|
312
|
+
ORDER BY COALESCE(priority, 100), emit_ts, id
|
|
313
|
+
LIMIT %s
|
|
314
|
+
FOR UPDATE SKIP LOCKED
|
|
315
|
+
)
|
|
316
|
+
RETURNING id, event_id, attempts, transient_attempts, escalation_reason
|
|
317
|
+
""",
|
|
318
|
+
(worker_id, claim_ttl_sec, max_attempts, batch_size),
|
|
319
|
+
)
|
|
320
|
+
return cur.fetchall()
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
Note: escalated rows re-claimed after a lease expiry re-enter as `status='claimed'`; on completion they go to `done`. A crashed mop-up leaves them `claimed` → lease expires → re-claimed here (the `status='escalated_pending'` filter misses expired-claimed rows, so ALSO include the expired-claimed re-entry — mirror `claim_next_batch`'s `OR (status='claimed' AND claim_expires_at < NOW())` but scoped to rows whose prior state was escalated). Simplest: the mop-up claim's WHERE becomes `((status='escalated_pending') OR (status='claimed' AND claim_expires_at < NOW() AND escalation_reason IS NOT NULL)) AND attempts < %s`.
|
|
324
|
+
|
|
325
|
+
- [ ] **Step 4: Update the claim to handle expired re-claim** (apply the WHERE from the note above), re-run the test asserting both branches present:
|
|
326
|
+
|
|
327
|
+
```python
|
|
328
|
+
assert "escalation_reason IS NOT NULL" in s # expired-claimed re-entry scoped to escalated rows
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
- [ ] **Step 5: Run to verify it passes**
|
|
332
|
+
|
|
333
|
+
Run: `python -m pytest test_async_escalation.py::test_claim_escalated_sql -v`
|
|
334
|
+
Expected: PASS.
|
|
335
|
+
|
|
336
|
+
- [ ] **Step 6: Wire the mop-up loop into `amain`**
|
|
337
|
+
|
|
338
|
+
When `ASYNC_ESCALATION_ENABLED`, the worker alternates: drain `pending` (student-primary) and drain `escalated_pending` (teacher-mop-up) using the SAME teacher processing code path that the inline cascade uses today (call teacher vLLM → `_apply_extraction(producer="teacher")` → teacher `_insert_trace` → `mark_done`; transient errors → `requeue_transient`; exhausted → `mark_failed`). Reuse the existing teacher batch function; feed it `claim_escalated_batch` results instead of the inline `escalate_items`.
|
|
339
|
+
|
|
340
|
+
```python
|
|
341
|
+
# in the poll loop, when ASYNC_ESCALATION_ENABLED:
|
|
342
|
+
esc = claim_escalated_batch(conn, WORKER_ID, BATCH_SIZE, CLAIM_TTL_SEC, MAX_ATTEMPTS)
|
|
343
|
+
if esc:
|
|
344
|
+
await _process_teacher_batch(conn, http, esc) # the extracted inline-teacher path
|
|
345
|
+
```
|
|
346
|
+
|
|
347
|
+
(Extract today's inline-teacher block into `_process_teacher_batch(conn, http, items)` so it is shared by the sync cascade and the async mop-up — no logic change, just a callable.)
|
|
348
|
+
|
|
349
|
+
- [ ] **Step 7: Integration test (stubbed LLM)** — one escalated event flows pending→escalated_pending→done, written once by the teacher.
|
|
350
|
+
|
|
351
|
+
```python
|
|
352
|
+
def test_end_to_end_escalated_written_once(async_pg, stub_teacher):
|
|
353
|
+
# async_pg: fixture with distillation_queue + one pending event that the
|
|
354
|
+
# stubbed student escalates. stub_teacher returns a fixed extraction.
|
|
355
|
+
run_student_pass(async_pg) # marks escalated_pending, no write
|
|
356
|
+
assert queue_status(async_pg, event="e1") == "escalated_pending"
|
|
357
|
+
assert graph_rows(async_pg, "e1") == 0
|
|
358
|
+
run_mopup_pass(async_pg, stub_teacher)
|
|
359
|
+
assert queue_status(async_pg, event="e1") == "done"
|
|
360
|
+
assert graph_rows(async_pg, "e1") == 1 # written exactly once, by teacher
|
|
361
|
+
```
|
|
362
|
+
|
|
363
|
+
Run: `python -m pytest test_async_escalation.py::test_end_to_end_escalated_written_once -v`
|
|
364
|
+
Expected: PASS.
|
|
365
|
+
|
|
366
|
+
- [ ] **Step 8: Commit**
|
|
367
|
+
|
|
368
|
+
```bash
|
|
369
|
+
git add packages/memory-engine-v2/extractor-async/worker.py packages/memory-engine-v2/extractor-async/test_async_escalation.py
|
|
370
|
+
git commit -m "feat(pme2): teacher-mop-up consumer drains escalated_pending"
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
---
|
|
374
|
+
|
|
375
|
+
### Task 5: Autoscaler counts `escalated_pending`
|
|
376
|
+
|
|
377
|
+
**Files:**
|
|
378
|
+
- Modify: `thing-event-system/modules/seesa/deploy/ops/distiller-autoscale.sh` (the `PENDING`/`OLDEST_MIN` queries, ~line 57–58)
|
|
379
|
+
|
|
380
|
+
**Interfaces:**
|
|
381
|
+
- Consumes: migration 014 (`escalated_pending` rows exist).
|
|
382
|
+
- Produces: the fleet scales up on escalated backlog (the teacher's actual work under async).
|
|
383
|
+
|
|
384
|
+
- [ ] **Step 1: Update the depth + age queries to include escalated_pending**
|
|
385
|
+
|
|
386
|
+
```sh
|
|
387
|
+
PENDING=$(PSQL "SELECT count(*) FROM distillation_queue WHERE status IN ('pending','escalated_pending')"); PENDING=${PENDING:-0}
|
|
388
|
+
OLDEST_MIN=$(PSQL "SELECT coalesce(round(extract(epoch from now()-min(enqueued_at))/60)::int,0) FROM distillation_queue WHERE status IN ('pending','escalated_pending')"); OLDEST_MIN=${OLDEST_MIN:-0}
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
Rationale comment to add: under async escalation the teacher's work is `escalated_pending`; the student drains `pending` on the always-on L4, so the fleet must wake on escalated depth/age. (Pre-async there are no escalated_pending rows, so this is a no-op until the flag flips — safe to ship first.)
|
|
392
|
+
|
|
393
|
+
- [ ] **Step 2: Syntax check + simulate**
|
|
394
|
+
|
|
395
|
+
Run: `bash -n modules/seesa/deploy/ops/distiller-autoscale.sh` → OK. Re-run the desired-count simulation from the age-aware change to confirm unchanged behavior when escalated=0.
|
|
396
|
+
|
|
397
|
+
- [ ] **Step 3: Commit (TES repo)**
|
|
398
|
+
|
|
399
|
+
```bash
|
|
400
|
+
git add modules/seesa/deploy/ops/distiller-autoscale.sh
|
|
401
|
+
git commit -m "feat(seesa-deploy): autoscaler counts escalated_pending for async escalation"
|
|
402
|
+
```
|
|
403
|
+
|
|
404
|
+
---
|
|
405
|
+
|
|
406
|
+
### Task 6: Redistill produces `escalated_pending` (teacher-only)
|
|
407
|
+
|
|
408
|
+
**Files:**
|
|
409
|
+
- Modify: `packages/memory-engine-v2/scripts/redistill.py` — the enqueue INSERT (~line 219)
|
|
410
|
+
- Test: `packages/memory-engine-v2/scripts/test_redistill.py` (create or extend)
|
|
411
|
+
|
|
412
|
+
**Interfaces:**
|
|
413
|
+
- Consumes: `escalated_pending` state (Task 1).
|
|
414
|
+
- Produces: redistilled events go straight to the teacher (bypass the student), fixing the latent "hits student first" gap.
|
|
415
|
+
|
|
416
|
+
- [ ] **Step 1: Write the failing test** — enqueue writes `status='escalated_pending', escalation_reason='redistill'`.
|
|
417
|
+
|
|
418
|
+
```python
|
|
419
|
+
def test_redistill_enqueues_teacher_only():
|
|
420
|
+
from redistill import ENQUEUE_SQL # extract the INSERT to a module constant
|
|
421
|
+
assert "escalated_pending" in ENQUEUE_SQL
|
|
422
|
+
assert "'redistill'" in ENQUEUE_SQL
|
|
423
|
+
```
|
|
424
|
+
|
|
425
|
+
- [ ] **Step 2: Run to verify it fails**
|
|
426
|
+
|
|
427
|
+
Run: `cd packages/memory-engine-v2/scripts && python -m pytest test_redistill.py::test_redistill_enqueues_teacher_only -v`
|
|
428
|
+
Expected: FAIL — `ENQUEUE_SQL` not defined / value mismatch.
|
|
429
|
+
|
|
430
|
+
- [ ] **Step 3: Change the enqueue** (redistill.py ~line 219). Only under async mode (gate on the same env flag so a redistill run against a pre-async engine still works):
|
|
431
|
+
|
|
432
|
+
```python
|
|
433
|
+
ENQUEUE_SQL = (
|
|
434
|
+
"INSERT INTO distillation_queue (event_id, status, attempts, priority, escalation_reason) "
|
|
435
|
+
"VALUES (%s, 'escalated_pending', 0, 200, 'redistill')"
|
|
436
|
+
if os.environ.get("ASYNC_ESCALATION_ENABLED", "") not in ("", "0", "false")
|
|
437
|
+
else "INSERT INTO distillation_queue (event_id, status, attempts, priority) "
|
|
438
|
+
"VALUES (%s, 'pending', 0, 200)"
|
|
439
|
+
)
|
|
440
|
+
...
|
|
441
|
+
cur.execute(ENQUEUE_SQL, (eid,))
|
|
442
|
+
```
|
|
443
|
+
|
|
444
|
+
Keep `--supersede-facts` unchanged — redistill still re-processes already-written events, so it still needs its own supersede (async escalation does not inherit that).
|
|
445
|
+
|
|
446
|
+
- [ ] **Step 4: Run to verify it passes**
|
|
447
|
+
|
|
448
|
+
Run: `python -m pytest test_redistill.py::test_redistill_enqueues_teacher_only -v` (with `ASYNC_ESCALATION_ENABLED=1`)
|
|
449
|
+
Expected: PASS.
|
|
450
|
+
|
|
451
|
+
- [ ] **Step 5: Commit**
|
|
452
|
+
|
|
453
|
+
```bash
|
|
454
|
+
git add packages/memory-engine-v2/scripts/redistill.py packages/memory-engine-v2/scripts/test_redistill.py
|
|
455
|
+
git commit -m "feat(pme2): redistill enqueues escalated_pending (teacher-only) under async"
|
|
456
|
+
```
|
|
457
|
+
|
|
458
|
+
---
|
|
459
|
+
|
|
460
|
+
### Task 7: Version bump + release + rollout
|
|
461
|
+
|
|
462
|
+
**Files:**
|
|
463
|
+
- Modify: `package.json` (version bump)
|
|
464
|
+
|
|
465
|
+
- [ ] **Step 1: Bump version** (patch bump, e.g. 0.10.39 → next).
|
|
466
|
+
|
|
467
|
+
- [ ] **Step 2: Apply migration 014 to prod** (Task 1 Step 5) and verify the constraint before deploying code.
|
|
468
|
+
|
|
469
|
+
- [ ] **Step 3: Ship dark** — merge + tag; the release chain deploys the worker with `ASYNC_ESCALATION_ENABLED` unset (false). Verify prod behaves identically (sync cascade), `escalated_pending` count = 0.
|
|
470
|
+
|
|
471
|
+
- [ ] **Step 4: Merge the TES autoscaler + pin-bump**, deploy.
|
|
472
|
+
|
|
473
|
+
- [ ] **Step 5: Canary** — set `ASYNC_ESCALATION_ENABLED=true` (SSM env) for the worker; watch: ingest drain, `escalated_pending` depth/oldest-age, teacher-fleet wake, graph-diff vs a control window, per-event provenance-count invariant (exactly one write/event).
|
|
474
|
+
|
|
475
|
+
- [ ] **Step 6: Ramp to 100%**; then enable redistill's teacher-only path.
|
|
476
|
+
|
|
477
|
+
---
|
|
478
|
+
|
|
479
|
+
## Self-Review
|
|
480
|
+
|
|
481
|
+
**Spec coverage:**
|
|
482
|
+
- escalated_pending primitive → Task 1 (schema) + Tasks 2–4 (producer/consumer). ✓
|
|
483
|
+
- Two-consumer split → Task 3 (student defers) + Task 4 (mop-up). ✓
|
|
484
|
+
- No supersede (deferred write) → Task 2 (`mark_escalated_pending`, no student write). ✓
|
|
485
|
+
- Kill switch → Task 3 Step 1 flag, gated everywhere. ✓
|
|
486
|
+
- Autoscaler signal → Task 5. ✓
|
|
487
|
+
- Redistill refactor → Task 6. ✓
|
|
488
|
+
- Migration + valid_status CHECK → Task 1. ✓
|
|
489
|
+
- Priority (escalated inherits priority/emit_ts) → Task 4 claim `ORDER BY priority, emit_ts`. ✓
|
|
490
|
+
- Graph-cleaning coexistence → no code change needed (RFC §6); nothing in the plan touches fusion/decay. ✓
|
|
491
|
+
- Rollout/dark-launch → Task 7. ✓
|
|
492
|
+
|
|
493
|
+
**Placeholder scan:** all steps carry concrete SQL/Python/commands. The `_process_teacher_batch` extraction (Task 4 Step 6) references today's inline-teacher block — the implementer extracts it verbatim (no logic change); code shown at call-site.
|
|
494
|
+
|
|
495
|
+
**Type consistency:** `mark_escalated_pending(conn, queue_id, reason)` (Task 2) used identically in Task 3 `_route_escalations`. `claim_escalated_batch` RETURNING columns (`id, event_id, attempts, transient_attempts, escalation_reason`) match the fields the teacher path + `requeue_transient` consume. `ASYNC_ESCALATION_ENABLED` env-string check identical in worker.py and redistill.py.
|
|
496
|
+
|
|
497
|
+
**Open implementation note:** worker.py can't be imported without `httpx`/`psycopg` present; the unit tests assume the CI env has them (the PyPI test job installs the package). If not, gate the import-heavy tests behind an availability check.
|