agent-working-memory 0.7.3 → 0.7.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +25 -2
- package/dist/api/routes.js +1 -1
- package/dist/cli.js +1 -1
- package/dist/coordination/index.d.ts.map +1 -1
- package/dist/coordination/index.js +18 -1
- package/dist/coordination/index.js.map +1 -1
- package/dist/coordination/routes.d.ts.map +1 -1
- package/dist/coordination/routes.js +152 -14
- package/dist/coordination/routes.js.map +1 -1
- package/dist/coordination/schemas.d.ts +18 -1
- package/dist/coordination/schemas.d.ts.map +1 -1
- package/dist/coordination/schemas.js +14 -2
- package/dist/coordination/schemas.js.map +1 -1
- package/dist/core/salience.d.ts +5 -1
- package/dist/core/salience.d.ts.map +1 -1
- package/dist/core/salience.js +95 -2
- package/dist/core/salience.js.map +1 -1
- package/dist/engine/activation.d.ts.map +1 -1
- package/dist/engine/activation.js +3 -1
- package/dist/engine/activation.js.map +1 -1
- package/dist/index.js +1 -1
- package/dist/mcp.js +2 -2
- package/dist/storage/sqlite.d.ts +9 -0
- package/dist/storage/sqlite.d.ts.map +1 -1
- package/dist/storage/sqlite.js +75 -10
- package/dist/storage/sqlite.js.map +1 -1
- package/package.json +57 -57
- package/src/api/routes.ts +723 -723
- package/src/cli.ts +719 -719
- package/src/coordination/index.ts +20 -1
- package/src/coordination/routes.ts +193 -15
- package/src/coordination/schemas.ts +17 -2
- package/src/core/salience.ts +91 -3
- package/src/engine/activation.ts +3 -1
- package/src/index.ts +199 -199
- package/src/mcp.ts +1192 -1192
- package/src/storage/sqlite.ts +77 -10
package/README.md
CHANGED
|
@@ -407,7 +407,30 @@ npm run test:locomo # LoCoMo industry benchmark (28.2%)
|
|
|
407
407
|
|
|
408
408
|
All three ML models run locally via ONNX. No external API calls for retrieval. The entire system is a single SQLite file + a Node.js process.
|
|
409
409
|
|
|
410
|
-
## What's New in v0.7.
|
|
410
|
+
## What's New in v0.7.6
|
|
411
|
+
|
|
412
|
+
- **Recall latency 11-23s → 2.5s end-to-end (~5× faster)** — measurement spike found the slow path was a SQLite query-plan trap, not vector search. The BM25 query `JOIN engrams_fts ON e.rowid + WHERE MATCH + ORDER BY rank LIMIT N` materialized all matching rows (with 1.5KB embedding blobs) before the LIMIT applied. CTE prefilter forces FTS5 LIMIT first, then joins only the top-K rowids. Same SQLite, same data, same results — 567× faster for wide OR queries (3682ms → 6.5ms verified). Also added `getAssociationsForBatch` to replace the per-candidate N+1 in the activation scoring loop. Top-K results are byte-identical to the old query (verified by the equivalence test in `spike/`).
|
|
413
|
+
- **Salience filter — auto-promote verified operational records** — operational batch summaries (e.g., "Submitted 6 events 2026-05-07 — IDs 18969, 18971…") were being discarded at salience 0.14 because BM25 novelty couldn't distinguish "useful new operational record" from "duplicate observation" when topic terminology repeated. New `detectVerifiedFinding()` pattern detector parallel to `detectUserFeedback()`: requires action-verb header (Submitted/Finalized/Completed/Reconciled/Triaged/etc.) plus ≥2 concrete identifiers (ISO date or contextual numeric ID). Matched memories get a 0.45 salience floor (active disposition, not canonical). 7 new tests, 23 salience tests pass.
|
|
414
|
+
|
|
415
|
+
## What's New in v0.7.4
|
|
416
|
+
|
|
417
|
+
- **Channel push telemetry** — new `GET /telemetry/channels` JSON endpoint and Prometheus counters (`coord_channel_push_attempts_total`, `..._delivered_total`, `..._failed_total{reason}`, `..._no_session_total`, `..._fallback_mailbox_total`, `..._session_disconnects_total`). Surfaces real delivery rate so coordination reliability can be measured rather than guessed.
|
|
418
|
+
- **Role-based `/channel/push` addressing** — accepts `{role, workspace, message}` as alternative to `{agentId, message}`. Server resolves role+workspace to most-recently-seen alive agent. Lets workers notify the coordinator without hardcoding its UUID (which changes across coordinator restarts). Enables event-driven worker → coordinator hand-off in place of fragile coordinator self-polling.
|
|
419
|
+
- **`/checkin` writes role on every call** — previously the UPDATE on existing rows preserved a stale role from initial registration; now agents can correct their own role via re-checkin.
|
|
420
|
+
- **`/workers` JOINs channel sessions** — `alive` field is now `recent_pulse OR connected_channel_session`. Stops false-dead duplicate-spawn loops where a busy worker's `/pulse` went stale during long tool sequences while their channel-server stayed reachable.
|
|
421
|
+
- **`cleanupStale` runs on a 5-minute schedule** — was only invoked manually; now zombie agents get marked dead automatically with a 600s threshold (forgiving for long edits).
|
|
422
|
+
- **`user_feedback` salience event type** — new event type with bonus 0.3 (highest of any). Auto-detect heuristic on `memory_write` content matching `^(Robert|Katherine|Nancy|...) (said|verbatim|directed|decided|...)` forces `memoryClass='canonical'` so user-stated decisions can't be discarded by the BM25 novelty floor in populated DBs.
|
|
423
|
+
|
|
424
|
+
### v0.7.3
|
|
425
|
+
|
|
426
|
+
- **Salience filter production tuning** — fixed BM25 novelty floor that was discarding ~17% salience for most writes in 10K+ engram DBs. Quadratic dampening curve (`max(0.05, 1 - topScore²)`); concept-match penalty scoped to last 30 days; floor lowered 0.10 → 0.05.
|
|
427
|
+
- Maintenance scripts for backup pruning + lme/bench database cleanup.
|
|
428
|
+
|
|
429
|
+
### v0.7.2
|
|
430
|
+
|
|
431
|
+
- Workspace recall fix (was returning UUIDs not names in v0.7.1 release).
|
|
432
|
+
|
|
433
|
+
### v0.7.1
|
|
411
434
|
|
|
412
435
|
- **Agent-provided metadata tags** — `memory_write` accepts `project`, `topic`, `source`, `confidence_level`, `session_id`, `intent`. Stored as searchable prefixed tags (`proj=X`, `sid=Z`). Session ID tags alone improved LongMemEval recall 3x.
|
|
413
436
|
- **Dual synthesis** — consolidation creates two types of summary memories: session summaries (tag-based, for perfect recall) and pattern syntheses (cross-session, for novel recall/creative connections).
|
|
@@ -434,7 +457,7 @@ See [CHANGELOG.md](CHANGELOG.md) for full details.
|
|
|
434
457
|
|
|
435
458
|
## Project Status
|
|
436
459
|
|
|
437
|
-
AWM is in active development (v0.7.
|
|
460
|
+
AWM is in active development (v0.7.6). The core memory pipeline, consolidation system, multi-agent coordination, and MCP integration are stable and used daily in production coding workflows.
|
|
438
461
|
|
|
439
462
|
- Core retrieval and consolidation: **stable**
|
|
440
463
|
- MCP tools and Claude Code integration: **stable**
|
package/dist/api/routes.js
CHANGED
package/dist/cli.js
CHANGED
|
@@ -307,7 +307,7 @@ async function exportMemories() {
|
|
|
307
307
|
// Collect unique agents
|
|
308
308
|
const agents = [...new Set(memories.map((m) => m.agent_id))];
|
|
309
309
|
const exportData = {
|
|
310
|
-
version: '0.7.
|
|
310
|
+
version: '0.7.6',
|
|
311
311
|
exported_at: new Date().toISOString(),
|
|
312
312
|
source_db: dbPath,
|
|
313
313
|
agent_filter: agentFilter,
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/coordination/index.ts"],"names":[],"mappings":"AAEA;;;GAGG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,SAAS,CAAC;AAC/C,OAAO,KAAK,QAAQ,MAAM,gBAAgB,CAAC;AAC3C,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,sBAAsB,CAAC;AAMxD,OAAO,EAAkB,KAAK,oBAAoB,EAAE,MAAM,aAAa,CAAC;AAGxE,mBAAmB,YAAY,CAAC;AAChC,OAAO,EAAE,KAAK,oBAAoB,EAAE,KAAK,kBAAkB,EAAE,MAAM,aAAa,CAAC;AACjF,YAAY,EAAE,SAAS,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAQ/D,yEAAyE;AACzE,wBAAgB,WAAW,IAAI,oBAAoB,GAAG,IAAI,CAEzD;AAED,iEAAiE;AACjE,wBAAgB,qBAAqB,IAAI,OAAO,CAG/C;AAED,mGAAmG;AACnG,wBAAgB,gBAAgB,CAAC,GAAG,EAAE,eAAe,EAAE,EAAE,EAAE,QAAQ,CAAC,QAAQ,EAAE,KAAK,CAAC,EAAE,WAAW,GAAG,IAAI,
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/coordination/index.ts"],"names":[],"mappings":"AAEA;;;GAGG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,SAAS,CAAC;AAC/C,OAAO,KAAK,QAAQ,MAAM,gBAAgB,CAAC;AAC3C,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,sBAAsB,CAAC;AAMxD,OAAO,EAAkB,KAAK,oBAAoB,EAAE,MAAM,aAAa,CAAC;AAGxE,mBAAmB,YAAY,CAAC;AAChC,OAAO,EAAE,KAAK,oBAAoB,EAAE,KAAK,kBAAkB,EAAE,MAAM,aAAa,CAAC;AACjF,YAAY,EAAE,SAAS,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAQ/D,yEAAyE;AACzE,wBAAgB,WAAW,IAAI,oBAAoB,GAAG,IAAI,CAEzD;AAED,iEAAiE;AACjE,wBAAgB,qBAAqB,IAAI,OAAO,CAG/C;AAED,mGAAmG;AACnG,wBAAgB,gBAAgB,CAAC,GAAG,EAAE,eAAe,EAAE,EAAE,EAAE,QAAQ,CAAC,QAAQ,EAAE,KAAK,CAAC,EAAE,WAAW,GAAG,IAAI,CAqHvG;AAED,qFAAqF;AACrF,wBAAsB,uBAAuB,IAAI,OAAO,CAAC,IAAI,CAAC,CAI7D"}
|
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
import { ZodError } from 'zod';
|
|
8
8
|
import { initCoordinationTables } from './schema.js';
|
|
9
9
|
import { registerCoordinationRoutes } from './routes.js';
|
|
10
|
-
import { cleanSlate, pruneOldHeartbeats, purgeDeadAgents } from './stale.js';
|
|
10
|
+
import { cleanSlate, pruneOldHeartbeats, purgeDeadAgents, cleanupStale } from './stale.js';
|
|
11
11
|
import { createWriteMutex, needsWriteLock } from './write-mutex.js';
|
|
12
12
|
import { createEventBus } from './events.js';
|
|
13
13
|
import { loadPlugins, teardownPlugins } from './plugin-loader.js';
|
|
@@ -87,6 +87,23 @@ export function initCoordination(app, db, store) {
|
|
|
87
87
|
}
|
|
88
88
|
catch { /* db may be closed */ }
|
|
89
89
|
}, 60 * 60 * 1000));
|
|
90
|
+
// Periodic stale-agent cleanup every 5 min with 600s threshold (10 min idle).
|
|
91
|
+
// Forgiving for long-running edits — workers should pulse every 60s during active
|
|
92
|
+
// work, so 10 min without a pulse is genuinely dead. This catches the
|
|
93
|
+
// "alive but not seeing each other" pattern where workers' processes persist
|
|
94
|
+
// but their heartbeats stop. Without this scheduled, only an explicit
|
|
95
|
+
// POST /stale/cleanup call (made by the coordinator agent on startup) ever
|
|
96
|
+
// fires cleanupStale, leaving zombie agents accumulating between coordinator
|
|
97
|
+
// sessions.
|
|
98
|
+
cleanupIntervals.push(setInterval(() => {
|
|
99
|
+
try {
|
|
100
|
+
const result = cleanupStale(db, 600);
|
|
101
|
+
if (result.cleaned > 0) {
|
|
102
|
+
console.log(` [stale-cleanup] auto-cleaned ${result.stale.length} stale agent(s), ${result.cleaned} resource(s) released`);
|
|
103
|
+
}
|
|
104
|
+
}
|
|
105
|
+
catch { /* db may be closed */ }
|
|
106
|
+
}, 5 * 60 * 1000));
|
|
90
107
|
// Periodic channel liveness probe every 60s — mark unreachable sessions as disconnected
|
|
91
108
|
cleanupIntervals.push(setInterval(async () => {
|
|
92
109
|
try {
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/coordination/index.ts"],"names":[],"mappings":"AAAA,gDAAgD;AAChD,sCAAsC;AACtC;;;GAGG;AAKH,OAAO,EAAE,QAAQ,EAAE,MAAM,KAAK,CAAC;AAC/B,OAAO,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AACrD,OAAO,EAAE,0BAA0B,EAAE,MAAM,aAAa,CAAC;AACzD,OAAO,EAAE,UAAU,EAAE,kBAAkB,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/coordination/index.ts"],"names":[],"mappings":"AAAA,gDAAgD;AAChD,sCAAsC;AACtC;;;GAGG;AAKH,OAAO,EAAE,QAAQ,EAAE,MAAM,KAAK,CAAC;AAC/B,OAAO,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AACrD,OAAO,EAAE,0BAA0B,EAAE,MAAM,aAAa,CAAC;AACzD,OAAO,EAAE,UAAU,EAAE,kBAAkB,EAAE,eAAe,EAAE,YAAY,EAAE,MAAM,YAAY,CAAC;AAC3F,OAAO,EAAE,gBAAgB,EAAE,cAAc,EAAE,MAAM,kBAAkB,CAAC;AACpE,OAAO,EAAE,cAAc,EAA6B,MAAM,aAAa,CAAC;AACxE,OAAO,EAAE,WAAW,EAAE,eAAe,EAAE,MAAM,oBAAoB,CAAC;AAMlE,sDAAsD;AACtD,MAAM,gBAAgB,GAAqB,EAAE,CAAC;AAE9C,iEAAiE;AACjE,IAAI,oBAAoB,GAAgC,IAAI,CAAC;AAE7D,yEAAyE;AACzE,MAAM,UAAU,WAAW;IACzB,OAAO,oBAAoB,CAAC;AAC9B,CAAC;AAED,iEAAiE;AACjE,MAAM,UAAU,qBAAqB;IACnC,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,gBAAgB,CAAC;IACzC,OAAO,GAAG,KAAK,MAAM,IAAI,GAAG,KAAK,GAAG,CAAC;AACvC,CAAC;AAED,mGAAmG;AACnG,MAAM,UAAU,gBAAgB,CAAC,GAAoB,EAAE,EAAqB,EAAE,KAAmB;IAC/F,0CAA0C;IAC1C,sBAAsB,CAAC,EAAE,CAAC,CAAC;IAE3B,gEAAgE;IAChE,UAAU,CAAC,EAAE,CAAC,CAAC;IAEf,mEAAmE;IACnE,GAAG,CAAC,OAAO,CAAC,WAAW,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE;QAChD,MAAM,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC,MAAM,IAAI,EAAE,CAAC;QAC5C,IAAI,+BAA+B,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,kCAAkC,CAAC,IAAI,CAAC,MAAM,CAAC,EAAE,CAAC;YACpG,KAAK,CAAC,MAAM,CAAC,6BAA6B,EAAE,MAAM,CAAC,CAAC;YACpD,KAAK,CAAC,MAAM,CAAC,8BAA8B,EAAE,wCAAwC,CAAC,CAAC;YACvF,KAAK,CAAC,MAAM,CAAC,8BAA8B,EAAE,6BAA6B,CAAC,CAAC;QAC9E,CAAC;QACD,IAAI,OAAO,CAAC,MAAM,KAAK,SAAS,EAAE,CAAC;YACjC,OAAO,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC;QAChC,CAAC;IACH,CAAC,CAAC,CAAC;IAEH,0FAA0F;IAC1F,GAAG,CAAC,OAAO,CAAC,SAAS,EAAE,CAAC,YAAY,EAAE,EAAE;QACtC,IAAI,CAAC,YAAY,CAAC,SAAS,EAAE,CAAC;YAC5B,YAAY,CAAC,SAAS,GAAG,OAAO,CAAC;QACnC,CAAC;IACH,CAAC,CAAC,CAAC;IAEH,uEAAuE;IACvE,0DAA0D;IAC1D,MAAM,UAAU,GAAG,gBAAgB,EAAE,CAAC;IACtC,GAAG,CAAC,OAAO,CAAC,YAAY,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE;QACjD,IAAI,cAAc,CAAC,OAAO,CAAC,MAAM,EAAE,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC;YAChD,MAAM,OAAO,GAAG,MAAM,UAAU,CAAC,OAAO,EAAE,CAAC;YAC3C,gEAAgE;YAChE,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC;QAClC,CAAC;IACH,CAAC,CAAC,CAAC;IAEH,8CAA8C;IAC9C,oBAAoB,GAAG,cAAc,EAAE,CAAC;IAExC,qCAAqC;IACrC,0BAA0B,CAAC,GAAG,EAAE,EAAE,EAAE,KAAK,EAAE,oBAAoB,CAAC,CAAC;IAEjE,qFAAqF;IACrF,GAAG,CAAC,eAAe,CAAC,CAAC,KAAsC,EAAE,QAAQ,EAAE,KAAK,EAAE,EAAE;QAC9E,IAAI,KAAK,YAAY,QAAQ,EAAE,CAAC;YAC9B,OAAO,KAAK,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC;gBAC5B,IAAI,EAAE,IAAI;gBACV,KAAK,EAAE,EAAE,IAAI,EAAE,kBAAkB,EAAE,OAAO,EAAE,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,OAAO,EAAE,MAAM,EAAE,KAAK,CAAC,MAAM,EAAE;aAC5F,CAAC,CAAC;QACL,CAAC;QACD,KAAK,CAAC,MAAM,CAAC,KAAK,CAAC,UAAU,IAAI,GAAG,CAAC,CAAC,IAAI,CAAC;YACzC,IAAI,EAAE,IAAI;YACV,KAAK,EAAE,EAAE,IAAI,EAAE,gBAAgB,EAAE,OAAO,EAAE,KAAK,CAAC,OAAO,EAAE;SAC1D,CAAC,CAAC;IACL,CAAC,CAAC,CAAC;IAEH,sFAAsF;IACtF,gBAAgB,CAAC,IAAI,CACnB,WAAW,CAAC,GAAG,EAAE;QACf,IAAI,CAAC;YAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;QAAC,CAAC;QAAC,MAAM,CAAC,CAAC,sBAAsB,CAAC,CAAC;IAClE,CAAC,EAAE,EAAE,GAAG,EAAE,GAAG,IAAI,CAAC,EAClB,WAAW,CAAC,GAAG,EAAE;QACf,IAAI,CAAC;YAAC,eAAe,CAAC,EAAE,EAAE,EAAE,CAAC,CAAC;QAAC,CAAC;QAAC,MAAM,CAAC,CAAC,sBAAsB,CAAC,CAAC;IACnE,CAAC,EAAE,EAAE,GAAG,EAAE,GAAG,IAAI,CAAC,CACnB,CAAC;IAEF,8EAA8E;IAC9E,kFAAkF;IAClF,sEAAsE;IACtE,6EAA6E;IAC7E,sEAAsE;IACtE,2EAA2E;IAC3E,6EAA6E;IAC7E,YAAY;IACZ,gBAAgB,CAAC,IAAI,CACnB,WAAW,CAAC,GAAG,EAAE;QACf,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,YAAY,CAAC,EAAE,EAAE,GAAG,CAAC,CAAC;YACrC,IAAI,MAAM,CAAC,OAAO,GAAG,CAAC,EAAE,CAAC;gBACvB,OAAO,CAAC,GAAG,CAAC,kCAAkC,MAAM,CAAC,KAAK,CAAC,MAAM,oBAAoB,MAAM,CAAC,OAAO,uBAAuB,CAAC,CAAC;YAC9H,CAAC;QACH,CAAC;QAAC,MAAM,CAAC,CAAC,sBAAsB,CAAC,CAAC;IACpC,CAAC,EAAE,CAAC,GAAG,EAAE,GAAG,IAAI,CAAC,CAClB,CAAC;IAEF,wFAAwF;IACxF,gBAAgB,CAAC,IAAI,CACnB,WAAW,CAAC,KAAK,IAAI,EAAE;QACrB,IAAI,CAAC;YACH,MAAM,QAAQ,GAAG,EAAE,CAAC,OAAO,CACzB,oFAAoF,CACrF,CAAC,GAAG,EAAqD,CAAC;YAE3D,KAAK,MAAM,OAAO,IAAI,QAAQ,EAAE,CAAC;gBAC/B,IAAI,CAAC;oBACH,MAAM,GAAG,GAAG,MAAM,KAAK,CAAC,GAAG,OAAO,CAAC,UAAU,SAAS,EAAE;wBACtD,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,IAAI,CAAC;qBAClC,CAAC,CAAC;oBACH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;wBACZ,EAAE,CAAC,OAAO,CAAC,8EAA8E,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;oBACnH,CAAC;gBACH,CAAC;gBAAC,MAAM,CAAC;oBACP,EAAE,CAAC,OAAO,CAAC,8EAA8E,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,QAAQ,CAAC,CAAC;gBACnH,CAAC;YACH,CAAC;QACH,CAAC;QAAC,MAAM,CAAC,CAAC,sBAAsB,CAAC,CAAC;IACpC,CAAC,EAAE,MAAM,CAAC,CACX,CAAC;IAEF,mEAAmE;IACnE,WAAW,CAAC,EAAE,MAAM,EAAE,oBAAoB,EAAE,EAAE,EAAE,OAAO,EAAE,GAAG,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,GAAG,EAAE,EAAE;QAC5E,OAAO,CAAC,KAAK,CAAC,iCAAiC,EAAG,GAAa,CAAC,OAAO,CAAC,CAAC;IAC3E,CAAC,CAAC,CAAC;IAEH,OAAO,CAAC,GAAG,CAAC,+BAA+B,CAAC,CAAC;AAC/C,CAAC;AAED,qFAAqF;AACrF,MAAM,CAAC,KAAK,UAAU,uBAAuB;IAC3C,KAAK,MAAM,EAAE,IAAI,gBAAgB;QAAE,aAAa,CAAC,EAAE,CAAC,CAAC;IACrD,gBAAgB,CAAC,MAAM,GAAG,CAAC,CAAC;IAC5B,MAAM,eAAe,EAAE,CAAC;AAC1B,CAAC"}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"routes.d.ts","sourceRoot":"","sources":["../../src/coordination/routes.ts"],"names":[],"mappings":"AAEA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,SAAS,CAAC;AAC/C,OAAO,KAAK,QAAQ,MAAM,gBAAgB,CAAC;AAC3C,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,sBAAsB,CAAC;
|
|
1
|
+
{"version":3,"file":"routes.d.ts","sourceRoot":"","sources":["../../src/coordination/routes.ts"],"names":[],"mappings":"AAEA;;;;GAIG;AAEH,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,SAAS,CAAC;AAC/C,OAAO,KAAK,QAAQ,MAAM,gBAAgB,CAAC;AAC3C,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,sBAAsB,CAAC;AA6ExD,wBAAgB,0BAA0B,CAAC,GAAG,EAAE,eAAe,EAAE,EAAE,EAAE,QAAQ,CAAC,QAAQ,EAAE,KAAK,CAAC,EAAE,WAAW,EAAE,QAAQ,CAAC,EAAE,OAAO,aAAa,EAAE,oBAAoB,GAAG,IAAI,CAu9DxK"}
|
|
@@ -16,6 +16,18 @@ function ts() {
|
|
|
16
16
|
function coordLog(msg) {
|
|
17
17
|
console.log(`${ts()} [coord] ${msg}`);
|
|
18
18
|
}
|
|
19
|
+
function createChannelMetrics() {
|
|
20
|
+
return {
|
|
21
|
+
attempts: 0,
|
|
22
|
+
delivered: 0,
|
|
23
|
+
failed_http: 0,
|
|
24
|
+
failed_unreachable: 0,
|
|
25
|
+
no_session: 0,
|
|
26
|
+
fallback_mailbox: 0,
|
|
27
|
+
session_disconnects: 0,
|
|
28
|
+
started_at: Date.now(),
|
|
29
|
+
};
|
|
30
|
+
}
|
|
19
31
|
/**
|
|
20
32
|
* Optional session-token check.
|
|
21
33
|
* If X-Session-Token header is present and doesn't match the stored token → returns false (caller should 403).
|
|
@@ -31,6 +43,8 @@ function sessionTokenOk(db, agentId, req) {
|
|
|
31
43
|
return row.session_token === provided;
|
|
32
44
|
}
|
|
33
45
|
export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
46
|
+
// Channel push telemetry — process-scoped counters. See ChannelMetrics docs above.
|
|
47
|
+
const channelMetrics = createChannelMetrics();
|
|
34
48
|
// Request logging — one line per request with method, url, status, response time
|
|
35
49
|
app.addHook('onRequest', async (request) => {
|
|
36
50
|
request._startTime = Date.now();
|
|
@@ -100,7 +114,12 @@ export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
|
100
114
|
const wasDead = existing.status === 'dead';
|
|
101
115
|
// Issue a fresh token on reconnect; reuse existing token for live heartbeats
|
|
102
116
|
const sessionToken = wasDead ? randomUUID() : (db.prepare(`SELECT session_token FROM coord_agents WHERE id = ?`).get(existing.id).session_token ?? randomUUID());
|
|
103
|
-
|
|
117
|
+
// role IS updated on every checkin — agents know their own role and
|
|
118
|
+
// re-registrations may correct stale role values (e.g., when an old
|
|
119
|
+
// coord_agents row was inserted with role='orchestrator' before the
|
|
120
|
+
// 'coordinator' role was canonical, or when the channel-server's
|
|
121
|
+
// hardcoded role='worker' overwrote a real role).
|
|
122
|
+
db.prepare(`UPDATE coord_agents SET last_seen = datetime('now'), status = CASE WHEN status = 'dead' THEN 'idle' ELSE status END, role = ?, pid = COALESCE(?, pid), capabilities = COALESCE(?, capabilities), workspace = COALESCE(?, workspace), session_token = ? WHERE id = ?`).run(role, pid ?? null, capsJson, workspace ?? null, sessionToken, existing.id);
|
|
104
123
|
const eventType = wasDead ? 'reconnected' : 'heartbeat';
|
|
105
124
|
const detail = wasDead ? `${name} reconnected (was dead)` : `heartbeat from ${name}`;
|
|
106
125
|
db.prepare(`INSERT INTO coord_events (agent_id, event_type, detail) VALUES (?, ?, ?)`).run(existing.id, eventType, detail);
|
|
@@ -392,6 +411,10 @@ export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
|
392
411
|
db.prepare(`UPDATE coord_channel_sessions SET last_push_at = datetime('now'), push_count = push_count + 1 WHERE agent_id = ?`).run(agentId);
|
|
393
412
|
}
|
|
394
413
|
}
|
|
414
|
+
else {
|
|
415
|
+
// Session disappeared between intent record and delivery — race or rapid disconnect
|
|
416
|
+
channelMetrics.no_session++;
|
|
417
|
+
}
|
|
395
418
|
}
|
|
396
419
|
// Bridge context to AWM engrams (outside transaction — engram store has its own DB)
|
|
397
420
|
if (store && context) {
|
|
@@ -979,17 +1002,28 @@ export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
|
979
1002
|
app.get('/workers', async (req, reply) => {
|
|
980
1003
|
const q = workersQuerySchema.safeParse(req.query);
|
|
981
1004
|
const { capability, status: filterStatus, workspace } = q.success ? q.data : { capability: undefined, status: undefined, workspace: undefined };
|
|
1005
|
+
// Join with coord_channel_sessions so the coordinator agent can compute
|
|
1006
|
+
// alive=true for workers that have a connected channel session even when
|
|
1007
|
+
// their /pulse is stale. Without this, /workers under-reports liveness
|
|
1008
|
+
// during long tool-call sequences where the worker is processing but
|
|
1009
|
+
// hasn't called /pulse for >5min — leading to false-positive duplicate
|
|
1010
|
+
// spawns. Channel sessions get probed every 60s (coordination/index.ts:111),
|
|
1011
|
+
// so a stale channel-server.js gets status='disconnected' within 60-120s.
|
|
982
1012
|
let workers = workspace
|
|
983
|
-
? db.prepare(`SELECT id, name, role, status, current_task, capabilities, workspace, last_seen,
|
|
984
|
-
ROUND((julianday('now') - julianday(last_seen)) * 86400) AS seconds_since_seen
|
|
985
|
-
|
|
986
|
-
|
|
987
|
-
|
|
988
|
-
|
|
989
|
-
|
|
990
|
-
|
|
991
|
-
|
|
992
|
-
|
|
1013
|
+
? db.prepare(`SELECT a.id, a.name, a.role, a.status, a.current_task, a.capabilities, a.workspace, a.last_seen,
|
|
1014
|
+
ROUND((julianday('now') - julianday(a.last_seen)) * 86400) AS seconds_since_seen,
|
|
1015
|
+
cs.status AS channel_status, cs.last_push_at AS channel_last_push
|
|
1016
|
+
FROM coord_agents a
|
|
1017
|
+
LEFT JOIN coord_channel_sessions cs ON cs.agent_id = a.id
|
|
1018
|
+
WHERE a.status != 'dead' AND a.role NOT IN ('orchestrator', 'coordinator') AND a.workspace = ?
|
|
1019
|
+
ORDER BY a.name LIMIT 200`).all(workspace)
|
|
1020
|
+
: db.prepare(`SELECT a.id, a.name, a.role, a.status, a.current_task, a.capabilities, a.workspace, a.last_seen,
|
|
1021
|
+
ROUND((julianday('now') - julianday(a.last_seen)) * 86400) AS seconds_since_seen,
|
|
1022
|
+
cs.status AS channel_status, cs.last_push_at AS channel_last_push
|
|
1023
|
+
FROM coord_agents a
|
|
1024
|
+
LEFT JOIN coord_channel_sessions cs ON cs.agent_id = a.id
|
|
1025
|
+
WHERE a.status != 'dead' AND a.role NOT IN ('orchestrator', 'coordinator')
|
|
1026
|
+
ORDER BY a.name LIMIT 200`).all();
|
|
993
1027
|
if (capability) {
|
|
994
1028
|
workers = workers.filter(w => {
|
|
995
1029
|
if (!w.capabilities)
|
|
@@ -1016,7 +1050,14 @@ export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
|
1016
1050
|
workspace: w.workspace,
|
|
1017
1051
|
lastSeen: w.last_seen,
|
|
1018
1052
|
secondsSinceSeen: w.seconds_since_seen,
|
|
1019
|
-
alive
|
|
1053
|
+
// alive = recent /pulse OR connected channel session.
|
|
1054
|
+
// Channel sessions get probed every 60s and marked 'disconnected'
|
|
1055
|
+
// when unreachable, so a connected session is reliable proof of life
|
|
1056
|
+
// even during long tool sequences where the worker hasn't pulsed.
|
|
1057
|
+
// Prevents duplicate worker spawns when /pulse is stale but worker is busy.
|
|
1058
|
+
alive: w.seconds_since_seen < 300 || w.channel_status === 'connected',
|
|
1059
|
+
channelStatus: w.channel_status,
|
|
1060
|
+
channelLastPush: w.channel_last_push,
|
|
1020
1061
|
}));
|
|
1021
1062
|
return reply.send({
|
|
1022
1063
|
count: result.length,
|
|
@@ -1220,6 +1261,26 @@ export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
|
1220
1261
|
lines.push('# HELP coord_uptime_seconds Seconds since first agent registered');
|
|
1221
1262
|
lines.push('# TYPE coord_uptime_seconds gauge');
|
|
1222
1263
|
lines.push(`coord_uptime_seconds ${uptime}`);
|
|
1264
|
+
// ─── Channel push telemetry (process-scoped, reset on restart) ───
|
|
1265
|
+
lines.push('# HELP coord_channel_push_attempts_total Total channel push attempts since coordinator startup');
|
|
1266
|
+
lines.push('# TYPE coord_channel_push_attempts_total counter');
|
|
1267
|
+
lines.push(`coord_channel_push_attempts_total ${channelMetrics.attempts}`);
|
|
1268
|
+
lines.push('# HELP coord_channel_push_delivered_total Successful channel deliveries');
|
|
1269
|
+
lines.push('# TYPE coord_channel_push_delivered_total counter');
|
|
1270
|
+
lines.push(`coord_channel_push_delivered_total ${channelMetrics.delivered}`);
|
|
1271
|
+
lines.push('# HELP coord_channel_push_failed_total Failed channel deliveries by reason');
|
|
1272
|
+
lines.push('# TYPE coord_channel_push_failed_total counter');
|
|
1273
|
+
lines.push(`coord_channel_push_failed_total{reason="http"} ${channelMetrics.failed_http}`);
|
|
1274
|
+
lines.push(`coord_channel_push_failed_total{reason="unreachable"} ${channelMetrics.failed_unreachable}`);
|
|
1275
|
+
lines.push('# HELP coord_channel_no_session_total Push attempts where agent had no connected session');
|
|
1276
|
+
lines.push('# TYPE coord_channel_no_session_total counter');
|
|
1277
|
+
lines.push(`coord_channel_no_session_total ${channelMetrics.no_session}`);
|
|
1278
|
+
lines.push('# HELP coord_channel_fallback_mailbox_total Pushes that fell back to mailbox after delivery failure');
|
|
1279
|
+
lines.push('# TYPE coord_channel_fallback_mailbox_total counter');
|
|
1280
|
+
lines.push(`coord_channel_fallback_mailbox_total ${channelMetrics.fallback_mailbox}`);
|
|
1281
|
+
lines.push('# HELP coord_channel_session_disconnects_total Sessions marked disconnected after delivery failure');
|
|
1282
|
+
lines.push('# TYPE coord_channel_session_disconnects_total counter');
|
|
1283
|
+
lines.push(`coord_channel_session_disconnects_total ${channelMetrics.session_disconnects}`);
|
|
1223
1284
|
return reply.type('text/plain; version=0.0.4; charset=utf-8').send(lines.join('\n') + '\n');
|
|
1224
1285
|
});
|
|
1225
1286
|
// ─── Deep Health ───────────────────────────────────────────────
|
|
@@ -1302,6 +1363,7 @@ export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
|
1302
1363
|
* Returns { delivered, error? }. On connection failure, marks session dead.
|
|
1303
1364
|
*/
|
|
1304
1365
|
async function deliverToChannel(agentId, channelUrl, content, meta) {
|
|
1366
|
+
channelMetrics.attempts++;
|
|
1305
1367
|
try {
|
|
1306
1368
|
const res = await fetch(`${channelUrl}/push`, {
|
|
1307
1369
|
method: 'POST',
|
|
@@ -1310,24 +1372,54 @@ export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
|
1310
1372
|
signal: AbortSignal.timeout(5000),
|
|
1311
1373
|
});
|
|
1312
1374
|
if (!res.ok) {
|
|
1375
|
+
channelMetrics.failed_http++;
|
|
1313
1376
|
return { delivered: false, error: `channel returned ${res.status}` };
|
|
1314
1377
|
}
|
|
1378
|
+
channelMetrics.delivered++;
|
|
1315
1379
|
return { delivered: true };
|
|
1316
1380
|
}
|
|
1317
1381
|
catch (err) {
|
|
1318
1382
|
// Connection refused / timeout → worker process is dead, mark session disconnected
|
|
1383
|
+
channelMetrics.failed_unreachable++;
|
|
1384
|
+
channelMetrics.session_disconnects++;
|
|
1319
1385
|
db.prepare(`UPDATE coord_channel_sessions SET status = 'disconnected' WHERE agent_id = ?`).run(agentId);
|
|
1320
1386
|
const agent = db.prepare(`SELECT name FROM coord_agents WHERE id = ?`).get(agentId);
|
|
1321
1387
|
coordLog(`channel/deliver FAILED → ${agent?.name ?? agentId}: ${err instanceof Error ? err.message : err} — session marked disconnected`);
|
|
1322
1388
|
return { delivered: false, error: `worker unreachable: ${err instanceof Error ? err.message : err}` };
|
|
1323
1389
|
}
|
|
1324
1390
|
}
|
|
1325
|
-
/** POST /channel/push — Push a message to an agent. Tries live delivery first, falls back to mailbox queue.
|
|
1391
|
+
/** POST /channel/push — Push a message to an agent. Tries live delivery first, falls back to mailbox queue.
|
|
1392
|
+
*
|
|
1393
|
+
* Two addressing modes:
|
|
1394
|
+
* - {agentId, message} — direct UUID
|
|
1395
|
+
* - {role, workspace, message} — server resolves to most-recently-seen alive agent
|
|
1396
|
+
* matching role+workspace. Used by workers to notify
|
|
1397
|
+
* coordinator (whose UUID changes across restarts).
|
|
1398
|
+
*/
|
|
1326
1399
|
app.post('/channel/push', async (request, reply) => {
|
|
1327
1400
|
const parsed = channelPushSchema.safeParse(request.body);
|
|
1328
1401
|
if (!parsed.success)
|
|
1329
1402
|
return reply.status(400).send({ error: parsed.error.flatten() });
|
|
1330
|
-
const {
|
|
1403
|
+
const { message } = parsed.data;
|
|
1404
|
+
let { agentId } = parsed.data;
|
|
1405
|
+
// Role-based addressing — resolve to a concrete agentId
|
|
1406
|
+
if (!agentId && parsed.data.role && parsed.data.workspace) {
|
|
1407
|
+
const resolved = db.prepare(`SELECT id FROM coord_agents
|
|
1408
|
+
WHERE role = ? AND workspace = ? AND status != 'dead'
|
|
1409
|
+
ORDER BY last_seen DESC
|
|
1410
|
+
LIMIT 1`).get(parsed.data.role, parsed.data.workspace);
|
|
1411
|
+
if (!resolved) {
|
|
1412
|
+
return reply.status(404).send({
|
|
1413
|
+
error: `No alive agent found for role='${parsed.data.role}' workspace='${parsed.data.workspace}'`,
|
|
1414
|
+
});
|
|
1415
|
+
}
|
|
1416
|
+
agentId = resolved.id;
|
|
1417
|
+
}
|
|
1418
|
+
// Type narrowing — Zod refine guarantees agentId is set by this point,
|
|
1419
|
+
// but TypeScript can't see through the refine. This guard is unreachable
|
|
1420
|
+
// in practice (would have 400'd earlier).
|
|
1421
|
+
if (!agentId)
|
|
1422
|
+
return reply.status(400).send({ error: 'Internal: agentId resolution failed' });
|
|
1331
1423
|
const agent = db.prepare(`SELECT name, workspace FROM coord_agents WHERE id = ?`).get(agentId);
|
|
1332
1424
|
if (!agent)
|
|
1333
1425
|
return reply.status(404).send({ error: 'Agent not found' });
|
|
@@ -1342,6 +1434,11 @@ export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
|
1342
1434
|
return reply.send({ ok: true, delivered: true, channelId: session.channel_id });
|
|
1343
1435
|
}
|
|
1344
1436
|
// Live delivery failed — fall through to mailbox
|
|
1437
|
+
channelMetrics.fallback_mailbox++;
|
|
1438
|
+
}
|
|
1439
|
+
else {
|
|
1440
|
+
// No connected session — push went straight to mailbox
|
|
1441
|
+
channelMetrics.no_session++;
|
|
1345
1442
|
}
|
|
1346
1443
|
// Queue to mailbox (delivered on next /next poll)
|
|
1347
1444
|
db.prepare(`INSERT INTO coord_mailbox (worker_name, workspace, message, source) VALUES (?, ?, ?, 'coordinator')`).run(agent.name, agent.workspace, message);
|
|
@@ -1392,5 +1489,46 @@ export function registerCoordinationRoutes(app, db, store, eventBus) {
|
|
|
1392
1489
|
coordLog(`channel/probe: ${alive} alive, ${dead} dead — dead sessions marked disconnected`);
|
|
1393
1490
|
return reply.send({ probed: results.length, alive, dead, results });
|
|
1394
1491
|
});
|
|
1492
|
+
/**
|
|
1493
|
+
* GET /telemetry/channels — Channel push delivery telemetry.
|
|
1494
|
+
*
|
|
1495
|
+
* Counters reset on coordinator restart (in-process). Use this to answer:
|
|
1496
|
+
* "Are channels reliable enough to depend on, or do we need a polling fallback?"
|
|
1497
|
+
*
|
|
1498
|
+
* Response shape:
|
|
1499
|
+
* {
|
|
1500
|
+
* since: ISO timestamp of when counters started,
|
|
1501
|
+
* uptime_seconds: number,
|
|
1502
|
+
* attempts, delivered, failed_http, failed_unreachable,
|
|
1503
|
+
* no_session, fallback_mailbox, session_disconnects: number,
|
|
1504
|
+
* delivery_rate: 0..1 (delivered / attempts) or null if zero attempts,
|
|
1505
|
+
* per_agent: [{ agent_name, push_count, last_push_at, status }]
|
|
1506
|
+
* }
|
|
1507
|
+
*/
|
|
1508
|
+
app.get('/telemetry/channels', async (_request, reply) => {
|
|
1509
|
+
const perAgent = db.prepare(`
|
|
1510
|
+
SELECT a.name AS agent_name, cs.push_count, cs.last_push_at, cs.status,
|
|
1511
|
+
cs.connected_at
|
|
1512
|
+
FROM coord_channel_sessions cs
|
|
1513
|
+
JOIN coord_agents a ON a.id = cs.agent_id
|
|
1514
|
+
ORDER BY cs.push_count DESC, cs.connected_at DESC
|
|
1515
|
+
`).all();
|
|
1516
|
+
const deliveryRate = channelMetrics.attempts > 0
|
|
1517
|
+
? channelMetrics.delivered / channelMetrics.attempts
|
|
1518
|
+
: null;
|
|
1519
|
+
return reply.send({
|
|
1520
|
+
since: new Date(channelMetrics.started_at).toISOString(),
|
|
1521
|
+
uptime_seconds: Math.round((Date.now() - channelMetrics.started_at) / 1000),
|
|
1522
|
+
attempts: channelMetrics.attempts,
|
|
1523
|
+
delivered: channelMetrics.delivered,
|
|
1524
|
+
failed_http: channelMetrics.failed_http,
|
|
1525
|
+
failed_unreachable: channelMetrics.failed_unreachable,
|
|
1526
|
+
no_session: channelMetrics.no_session,
|
|
1527
|
+
fallback_mailbox: channelMetrics.fallback_mailbox,
|
|
1528
|
+
session_disconnects: channelMetrics.session_disconnects,
|
|
1529
|
+
delivery_rate: deliveryRate,
|
|
1530
|
+
per_agent: perAgent,
|
|
1531
|
+
});
|
|
1532
|
+
});
|
|
1395
1533
|
}
|
|
1396
1534
|
//# sourceMappingURL=routes.js.map
|