engrm 0.4.23 → 0.4.25

package/README.md

@@ -211,7 +211,7 @@ The MCP server exposes tools that supported agents can call directly:
 | `recent_activity` | Inspect what Engrm captured most recently |
 | `memory_stats` | View high-level capture and sync health |
 | `capture_status` | Check whether local hooks are registered and raw prompt/tool chronology is actually being captured |
-| `activity_feed` | Inspect one chronological local feed across prompts, tools, observations, and summaries |
+| `activity_feed` | Inspect one chronological local feed across prompts, tools, chat, handoffs, observations, and summaries |
 | `memory_console` | Show a high-signal local memory console for the current project |
 | `project_memory_index` | Show typed local memory by project, including hot files and recent sessions |
 | `workspace_memory_index` | Show cross-project local memory coverage across the whole workspace |
@@ -222,8 +222,12 @@ The MCP server exposes tools that supported agents can call directly:
 | `recent_sessions` | List recent local sessions to inspect further |
 | `session_story` | Show prompts, tools, observations, and summary for one session |
 | `create_handoff` | Save an explicit syncable handoff so you can resume work on another device |
+| `refresh_handoff` | Refresh the rolling live handoff draft for the current session without creating a new saved handoff |
 | `recent_handoffs` | List recent saved handoffs for the current project or workspace |
 | `load_handoff` | Open a saved handoff as a resume point for a new session |
+| `refresh_chat_recall` | Rehydrate the separate chat lane from a Claude transcript when a long session feels under-captured |
+| `recent_chat` | Inspect the separate synced chat lane without mixing it into durable memory |
+| `search_chat` | Search recent chat recall separately from reusable memory observations |
 | `plugin_catalog` | Inspect Engrm plugin manifests for memory-aware integrations |
 | `save_plugin_memory` | Save reduced plugin output with stable Engrm provenance |
 | `capture_git_diff` | Reduce a git diff into a durable memory object and save it |
@@ -282,15 +286,55 @@ For long-running work across devices, Engrm now has an explicit handoff flow:
 
 - `create_handoff`
   - snapshot the active thread into a syncable handoff message
+- `refresh_handoff`
+  - refresh the rolling live handoff draft for the current session
 - `recent_handoffs`
   - list the latest saved handoffs
 - `load_handoff`
   - reopen a saved handoff as a clear resume point in a new session
 
+Recent handoffs now carry:
+
+- source machine
+- freshness
+- current thread / recent outcomes
+- optional chat snippets when the session is still thin
+
+Rolling handoff drafts:
+
+- are kept as one updatable syncable draft per session
+- refresh during prompt-time and tool-time summary updates
+- let another machine resume live work even before you save a deliberate handoff
+- are refreshed again before Claude compacts, so the active thread survives context compression better
+
+The local workbench now shows handoff split too:
+
+- saved handoffs
+- rolling drafts
+
+`activity_feed` and `session_story` now keep that distinction visible too, so a live rolling draft does not masquerade as a deliberate saved handoff.
+
+When Engrm knows your current device, handoff tools also prefer resume points from another machine before showing the newest local handoff again.
+
 This is the deliberate version of multi-device continuity: useful when you want to move from laptop to home machine without waiting for an end-of-session summary.
 
 The separate chat lane is still kept distinct from durable observations, but it can now sync too, so recent user/assistant conversation is recoverable on another machine without polluting the main memory feed.
 
+For long sessions, Engrm now also supports transcript-backed chat hydration:
+
+- `refresh_chat_recall`
+  - reads the Claude transcript for the current session
+  - fills gaps in the separate chat lane with transcript-backed messages
+  - keeps those rows marked separately from hook-edge chat so recall can prefer the fuller thread
+
+Before Claude compacts, Engrm now also:
+
+- refreshes transcript-backed chat recall for the active session
+- refreshes the rolling handoff draft
+- then injects the preserved thread into the compacted context
+
+So compaction should reduce prompt-window pressure without making the memory layer act like the conversation never happened.
+
 ### Local Memory Inspection
 
 For local testing, Engrm now exposes a small inspection set that lets you see
@@ -314,9 +358,9 @@ What each tool is good for:
 
 - `capture_status` tells you whether prompt/tool hooks are live on this machine
 - `memory_console` gives the quickest project snapshot
-- `activity_feed` shows the merged chronology across prompts, tools, observations, and summaries
+- `activity_feed` shows the merged chronology across prompts, tools, chat, handoffs, observations, and summaries
 - `recent_sessions` helps you pick a session worth opening
-- `session_story` reconstructs one session in detail
+- `session_story` reconstructs one session in detail, including handoffs and chat recall
 - `tool_memory_index` shows which tools and plugins are actually producing durable memory
 - `session_tool_memory` shows which tool calls in one session turned into reusable memory and which did not
 - `project_memory_index` shows typed memory by repo

package/dist/cli.js

@@ -695,6 +695,19 @@ var MIGRATIONS = [
       CREATE INDEX idx_outbox_status ON sync_outbox(status, next_retry_epoch);
       CREATE INDEX idx_outbox_record ON sync_outbox(record_type, record_id);
     `
+  },
+  {
+    version: 17,
+    description: "Track transcript-backed chat messages separately from hook chat",
+    sql: `
+      ALTER TABLE chat_messages ADD COLUMN source_kind TEXT DEFAULT 'hook';
+      ALTER TABLE chat_messages ADD COLUMN transcript_index INTEGER;
+      CREATE INDEX IF NOT EXISTS idx_chat_messages_source_kind
+        ON chat_messages(session_id, source_kind, transcript_index, created_at_epoch DESC, id DESC);
+      CREATE UNIQUE INDEX IF NOT EXISTS idx_chat_messages_session_transcript
+        ON chat_messages(session_id, transcript_index)
+        WHERE transcript_index IS NOT NULL;
+    `
   }
 ];
 function isVecExtensionLoaded(db) {
@@ -765,6 +778,9 @@ function inferLegacySchemaVersion(db) {
   if (syncOutboxSupportsChatMessages(db)) {
     version = Math.max(version, 16);
   }
+  if (columnExists(db, "chat_messages", "source_kind") && columnExists(db, "chat_messages", "transcript_index")) {
+    version = Math.max(version, 17);
+  }
   return version;
 }
 function runMigrations(db) {
@@ -868,9 +884,17 @@ function ensureChatMessageColumns(db) {
     db.exec("ALTER TABLE chat_messages ADD COLUMN remote_source_id TEXT");
   }
   db.exec("CREATE UNIQUE INDEX IF NOT EXISTS idx_chat_messages_remote_source ON chat_messages(remote_source_id) WHERE remote_source_id IS NOT NULL");
+  if (!columnExists(db, "chat_messages", "source_kind")) {
+    db.exec("ALTER TABLE chat_messages ADD COLUMN source_kind TEXT DEFAULT 'hook'");
+  }
+  if (!columnExists(db, "chat_messages", "transcript_index")) {
+    db.exec("ALTER TABLE chat_messages ADD COLUMN transcript_index INTEGER");
+  }
+  db.exec("CREATE INDEX IF NOT EXISTS idx_chat_messages_source_kind ON chat_messages(session_id, source_kind, transcript_index, created_at_epoch DESC, id DESC)");
+  db.exec("CREATE UNIQUE INDEX IF NOT EXISTS idx_chat_messages_session_transcript ON chat_messages(session_id, transcript_index) WHERE transcript_index IS NOT NULL");
   const current = getSchemaVersion(db);
-  if (current < 15) {
-    db.exec("PRAGMA user_version = 15");
+  if (current < 17) {
+    db.exec("PRAGMA user_version = 17");
   }
 }
 function ensureSyncOutboxSupportsChatMessages(db) {
@@ -1155,6 +1179,22 @@ class MemDatabase {
   getObservationById(id) {
     return this.db.query("SELECT * FROM observations WHERE id = ?").get(id) ?? null;
   }
+  updateObservationContent(id, update) {
+    const existing = this.getObservationById(id);
+    if (!existing)
+      return null;
+    const createdAtEpoch = update.created_at_epoch ?? existing.created_at_epoch;
+    const createdAt = new Date(createdAtEpoch * 1000).toISOString();
+    this.db.query(`UPDATE observations
+         SET title = ?, narrative = ?, facts = ?, concepts = ?, created_at = ?, created_at_epoch = ?
+         WHERE id = ?`).run(update.title, update.narrative ?? null, update.facts ?? null, update.concepts ?? null, createdAt, createdAtEpoch, id);
+    this.ftsDelete(existing);
+    const refreshed = this.getObservationById(id);
+    if (!refreshed)
+      return null;
+    this.ftsInsert(refreshed);
+    return refreshed;
+  }
   getObservationsByIds(ids, userId) {
     if (ids.length === 0)
       return [];
@@ -1426,8 +1466,8 @@ class MemDatabase {
     const createdAt = input.created_at_epoch ?? Math.floor(Date.now() / 1000);
     const content = input.content.trim();
     const result = this.db.query(`INSERT INTO chat_messages (
-          session_id, project_id, role, content, user_id, device_id, agent, created_at_epoch, remote_source_id
-        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`).run(input.session_id, input.project_id, input.role, content, input.user_id, input.device_id, input.agent ?? "claude-code", createdAt, input.remote_source_id ?? null);
+          session_id, project_id, role, content, user_id, device_id, agent, created_at_epoch, remote_source_id, source_kind, transcript_index
+        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`).run(input.session_id, input.project_id, input.role, content, input.user_id, input.device_id, input.agent ?? "claude-code", createdAt, input.remote_source_id ?? null, input.source_kind ?? "hook", input.transcript_index ?? null);
     return this.getChatMessageById(Number(result.lastInsertRowid));
   }
   getChatMessageById(id) {
@@ -1439,7 +1479,17 @@ class MemDatabase {
   getSessionChatMessages(sessionId, limit = 50) {
     return this.db.query(`SELECT * FROM chat_messages
          WHERE session_id = ?
-         ORDER BY created_at_epoch ASC, id ASC
+           AND (
+             source_kind = 'transcript'
+             OR NOT EXISTS (
+               SELECT 1 FROM chat_messages t2
+               WHERE t2.session_id = chat_messages.session_id
+                 AND t2.source_kind = 'transcript'
+             )
+           )
+         ORDER BY
+           CASE WHEN transcript_index IS NULL THEN created_at_epoch ELSE transcript_index END ASC,
+           id ASC
          LIMIT ?`).all(sessionId, limit);
   }
   getRecentChatMessages(projectId, limit = 20, userId) {
@@ -1447,11 +1497,27 @@ class MemDatabase {
     if (projectId !== null) {
       return this.db.query(`SELECT * FROM chat_messages
            WHERE project_id = ?${visibilityClause}
+             AND (
+               source_kind = 'transcript'
+               OR NOT EXISTS (
+                 SELECT 1 FROM chat_messages t2
+                 WHERE t2.session_id = chat_messages.session_id
+                   AND t2.source_kind = 'transcript'
+               )
+             )
            ORDER BY created_at_epoch DESC, id DESC
            LIMIT ?`).all(projectId, ...userId ? [userId] : [], limit);
     }
     return this.db.query(`SELECT * FROM chat_messages
          WHERE 1 = 1${visibilityClause}
+           AND (
+             source_kind = 'transcript'
+             OR NOT EXISTS (
+               SELECT 1 FROM chat_messages t2
+               WHERE t2.session_id = chat_messages.session_id
+                 AND t2.source_kind = 'transcript'
+             )
+           )
          ORDER BY created_at_epoch DESC, id DESC
          LIMIT ?`).all(...userId ? [userId] : [], limit);
   }
@@ -1462,14 +1528,33 @@ class MemDatabase {
       return this.db.query(`SELECT * FROM chat_messages
            WHERE project_id = ?
              AND lower(content) LIKE ?${visibilityClause}
+             AND (
+               source_kind = 'transcript'
+               OR NOT EXISTS (
+                 SELECT 1 FROM chat_messages t2
+                 WHERE t2.session_id = chat_messages.session_id
+                   AND t2.source_kind = 'transcript'
+               )
+             )
            ORDER BY created_at_epoch DESC, id DESC
            LIMIT ?`).all(projectId, needle, ...userId ? [userId] : [], limit);
     }
     return this.db.query(`SELECT * FROM chat_messages
          WHERE lower(content) LIKE ?${visibilityClause}
+           AND (
+             source_kind = 'transcript'
+             OR NOT EXISTS (
+               SELECT 1 FROM chat_messages t2
+               WHERE t2.session_id = chat_messages.session_id
+                 AND t2.source_kind = 'transcript'
+             )
+           )
          ORDER BY created_at_epoch DESC, id DESC
          LIMIT ?`).all(needle, ...userId ? [userId] : [], limit);
   }
+  getTranscriptChatMessage(sessionId, transcriptIndex) {
+    return this.db.query("SELECT * FROM chat_messages WHERE session_id = ? AND transcript_index = ?").get(sessionId, transcriptIndex) ?? null;
+  }
   addToOutbox(recordType, recordId) {
     const now = Math.floor(Date.now() / 1000);
     this.db.query(`INSERT INTO sync_outbox (record_type, record_id, created_at_epoch)
@@ -2198,6 +2283,19 @@ var MIGRATIONS2 = [
       CREATE INDEX idx_outbox_status ON sync_outbox(status, next_retry_epoch);
       CREATE INDEX idx_outbox_record ON sync_outbox(record_type, record_id);
     `
+  },
+  {
+    version: 17,
+    description: "Track transcript-backed chat messages separately from hook chat",
+    sql: `
+      ALTER TABLE chat_messages ADD COLUMN source_kind TEXT DEFAULT 'hook';
+      ALTER TABLE chat_messages ADD COLUMN transcript_index INTEGER;
+      CREATE INDEX IF NOT EXISTS idx_chat_messages_source_kind
+        ON chat_messages(session_id, source_kind, transcript_index, created_at_epoch DESC, id DESC);
+      CREATE UNIQUE INDEX IF NOT EXISTS idx_chat_messages_session_transcript
+        ON chat_messages(session_id, transcript_index)
+        WHERE transcript_index IS NOT NULL;
+    `
   }
 ];
 function isVecExtensionLoaded2(db) {

package/dist/hooks/elicitation-result.js

@@ -1528,6 +1528,19 @@ var MIGRATIONS = [
       CREATE INDEX idx_outbox_status ON sync_outbox(status, next_retry_epoch);
       CREATE INDEX idx_outbox_record ON sync_outbox(record_type, record_id);
     `
+  },
+  {
+    version: 17,
+    description: "Track transcript-backed chat messages separately from hook chat",
+    sql: `
+      ALTER TABLE chat_messages ADD COLUMN source_kind TEXT DEFAULT 'hook';
+      ALTER TABLE chat_messages ADD COLUMN transcript_index INTEGER;
+      CREATE INDEX IF NOT EXISTS idx_chat_messages_source_kind
+        ON chat_messages(session_id, source_kind, transcript_index, created_at_epoch DESC, id DESC);
+      CREATE UNIQUE INDEX IF NOT EXISTS idx_chat_messages_session_transcript
+        ON chat_messages(session_id, transcript_index)
+        WHERE transcript_index IS NOT NULL;
+    `
   }
 ];
 function isVecExtensionLoaded(db) {
@@ -1598,6 +1611,9 @@ function inferLegacySchemaVersion(db) {
   if (syncOutboxSupportsChatMessages(db)) {
     version = Math.max(version, 16);
   }
+  if (columnExists(db, "chat_messages", "source_kind") && columnExists(db, "chat_messages", "transcript_index")) {
+    version = Math.max(version, 17);
+  }
   return version;
 }
 function runMigrations(db) {
@@ -1701,9 +1717,17 @@ function ensureChatMessageColumns(db) {
     db.exec("ALTER TABLE chat_messages ADD COLUMN remote_source_id TEXT");
   }
   db.exec("CREATE UNIQUE INDEX IF NOT EXISTS idx_chat_messages_remote_source ON chat_messages(remote_source_id) WHERE remote_source_id IS NOT NULL");
+  if (!columnExists(db, "chat_messages", "source_kind")) {
+    db.exec("ALTER TABLE chat_messages ADD COLUMN source_kind TEXT DEFAULT 'hook'");
+  }
+  if (!columnExists(db, "chat_messages", "transcript_index")) {
+    db.exec("ALTER TABLE chat_messages ADD COLUMN transcript_index INTEGER");
+  }
+  db.exec("CREATE INDEX IF NOT EXISTS idx_chat_messages_source_kind ON chat_messages(session_id, source_kind, transcript_index, created_at_epoch DESC, id DESC)");
+  db.exec("CREATE UNIQUE INDEX IF NOT EXISTS idx_chat_messages_session_transcript ON chat_messages(session_id, transcript_index) WHERE transcript_index IS NOT NULL");
   const current = getSchemaVersion(db);
-  if (current < 15) {
-    db.exec("PRAGMA user_version = 15");
+  if (current < 17) {
+    db.exec("PRAGMA user_version = 17");
   }
 }
 function ensureSyncOutboxSupportsChatMessages(db) {
@@ -1988,6 +2012,22 @@ class MemDatabase {
   getObservationById(id) {
     return this.db.query("SELECT * FROM observations WHERE id = ?").get(id) ?? null;
   }
+  updateObservationContent(id, update) {
+    const existing = this.getObservationById(id);
+    if (!existing)
+      return null;
+    const createdAtEpoch = update.created_at_epoch ?? existing.created_at_epoch;
+    const createdAt = new Date(createdAtEpoch * 1000).toISOString();
+    this.db.query(`UPDATE observations
+         SET title = ?, narrative = ?, facts = ?, concepts = ?, created_at = ?, created_at_epoch = ?
+         WHERE id = ?`).run(update.title, update.narrative ?? null, update.facts ?? null, update.concepts ?? null, createdAt, createdAtEpoch, id);
+    this.ftsDelete(existing);
+    const refreshed = this.getObservationById(id);
+    if (!refreshed)
+      return null;
+    this.ftsInsert(refreshed);
+    return refreshed;
+  }
   getObservationsByIds(ids, userId) {
     if (ids.length === 0)
       return [];
@@ -2259,8 +2299,8 @@ class MemDatabase {
     const createdAt = input.created_at_epoch ?? Math.floor(Date.now() / 1000);
     const content = input.content.trim();
     const result = this.db.query(`INSERT INTO chat_messages (
-          session_id, project_id, role, content, user_id, device_id, agent, created_at_epoch, remote_source_id
-        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)`).run(input.session_id, input.project_id, input.role, content, input.user_id, input.device_id, input.agent ?? "claude-code", createdAt, input.remote_source_id ?? null);
+          session_id, project_id, role, content, user_id, device_id, agent, created_at_epoch, remote_source_id, source_kind, transcript_index
+        ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`).run(input.session_id, input.project_id, input.role, content, input.user_id, input.device_id, input.agent ?? "claude-code", createdAt, input.remote_source_id ?? null, input.source_kind ?? "hook", input.transcript_index ?? null);
     return this.getChatMessageById(Number(result.lastInsertRowid));
   }
   getChatMessageById(id) {
@@ -2272,7 +2312,17 @@ class MemDatabase {
   getSessionChatMessages(sessionId, limit = 50) {
     return this.db.query(`SELECT * FROM chat_messages
          WHERE session_id = ?
-         ORDER BY created_at_epoch ASC, id ASC
+           AND (
+             source_kind = 'transcript'
+             OR NOT EXISTS (
+               SELECT 1 FROM chat_messages t2
+               WHERE t2.session_id = chat_messages.session_id
+                 AND t2.source_kind = 'transcript'
+             )
+           )
+         ORDER BY
+           CASE WHEN transcript_index IS NULL THEN created_at_epoch ELSE transcript_index END ASC,
+           id ASC
          LIMIT ?`).all(sessionId, limit);
   }
   getRecentChatMessages(projectId, limit = 20, userId) {
@@ -2280,11 +2330,27 @@ class MemDatabase {
     if (projectId !== null) {
       return this.db.query(`SELECT * FROM chat_messages
            WHERE project_id = ?${visibilityClause}
+             AND (
+               source_kind = 'transcript'
+               OR NOT EXISTS (
+                 SELECT 1 FROM chat_messages t2
+                 WHERE t2.session_id = chat_messages.session_id
+                   AND t2.source_kind = 'transcript'
+               )
+             )
            ORDER BY created_at_epoch DESC, id DESC
            LIMIT ?`).all(projectId, ...userId ? [userId] : [], limit);
     }
     return this.db.query(`SELECT * FROM chat_messages
          WHERE 1 = 1${visibilityClause}
+           AND (
+             source_kind = 'transcript'
+             OR NOT EXISTS (
+               SELECT 1 FROM chat_messages t2
+               WHERE t2.session_id = chat_messages.session_id
+                 AND t2.source_kind = 'transcript'
+             )
+           )
          ORDER BY created_at_epoch DESC, id DESC
          LIMIT ?`).all(...userId ? [userId] : [], limit);
   }
@@ -2295,14 +2361,33 @@ class MemDatabase {
       return this.db.query(`SELECT * FROM chat_messages
            WHERE project_id = ?
              AND lower(content) LIKE ?${visibilityClause}
+             AND (
+               source_kind = 'transcript'
+               OR NOT EXISTS (
+                 SELECT 1 FROM chat_messages t2
+                 WHERE t2.session_id = chat_messages.session_id
+                   AND t2.source_kind = 'transcript'
+               )
+             )
            ORDER BY created_at_epoch DESC, id DESC
            LIMIT ?`).all(projectId, needle, ...userId ? [userId] : [], limit);
     }
     return this.db.query(`SELECT * FROM chat_messages
          WHERE lower(content) LIKE ?${visibilityClause}
+           AND (
+             source_kind = 'transcript'
+             OR NOT EXISTS (
+               SELECT 1 FROM chat_messages t2
+               WHERE t2.session_id = chat_messages.session_id
+                 AND t2.source_kind = 'transcript'
+             )
+           )
          ORDER BY created_at_epoch DESC, id DESC
          LIMIT ?`).all(needle, ...userId ? [userId] : [], limit);
   }
+  getTranscriptChatMessage(sessionId, transcriptIndex) {
+    return this.db.query("SELECT * FROM chat_messages WHERE session_id = ? AND transcript_index = ?").get(sessionId, transcriptIndex) ?? null;
+  }
   addToOutbox(recordType, recordId) {
     const now = Math.floor(Date.now() / 1000);
     this.db.query(`INSERT INTO sync_outbox (record_type, record_id, created_at_epoch)