opencode-mempalace-persistence 2.0.1 → 2.2.0

package/README.md

@@ -6,8 +6,12 @@ An OpenCode plugin that automatically saves every conversation to MemPalace and
 
 Follows the official MemPalace automation pattern (same as the Claude Code hooks): the plugin decides **when** to save, the model decides **what** to file via the MemPalace MCP tools.
 
+[![npm version](https://img.shields.io/npm/v/opencode-mempalace-persistence.svg)](https://www.npmjs.com/package/opencode-mempalace-persistence)
+[![npm downloads](https://img.shields.io/npm/dm/opencode-mempalace-persistence.svg)](https://www.npmjs.com/package/opencode-mempalace-persistence)
 [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
+![Demo: a decision filed on Monday is recalled verbatim by a different session on Thursday — memory outlives sessions, not just compaction](demo.gif)
+
 ---
 
 ## How it works in 3 seconds
@@ -170,8 +174,7 @@ You ask a question
   → Model files topics/decisions/quotes via MCP tools, then answers
 
 The model responds
-  → Plugin detects the response is complete
-  → Saves the conversation to MemPalace (flat export, no hardcoded wings)
+  → Once the turn completes, the next idle/exit/startup mines it to MemPalace (flat export, no hardcoded wings)
   → Model records new KG facts via MCP tools (only when something new emerged)
 
 Session goes idle / process exits
@@ -190,7 +193,7 @@ Next time you ask
 
 ## What gets saved
 
-Every turn (question + answer) is saved as a drawer in MemPalace. No forced categorization — mining runs with `--mode convos --extract general`, so MemPalace itself classifies content into decisions, preferences, milestones, problems, and emotional context. Exports are grouped one wing per project (official multi-project pattern: `bot-oc` sessions land in wing `bot-oc`, never leaking across projects). The model additionally records KG facts (decisions, milestones, preferences) during conversation and at each checkpoint via MCP tools.
+Every turn (question + answer) is saved as a drawer in MemPalace. Mining runs with `--mode convos` (default `exchange` extraction: one drawer per exchange pair, verbatim, no paraphrasing). Exports are grouped one wing per project (official multi-project pattern: `bot-oc` sessions land in wing `bot-oc`, never leaking across projects). Only completed turns are exported (in-flight replies are revisited by the next sync). The model additionally records KG facts (decisions, milestones, preferences) during conversation and at each checkpoint via MCP tools.
 
 ### Backfill existing sessions
 
@@ -218,18 +221,16 @@ The plugin exports everything in the opencode database on the next sync, then re
                  │    ↓                          │
                  │  Model sees context → answers │
                  │    ↓                          │
-  Answer done ──►│  chat.message (count) + session.idle    │
-                  │  Every N msgs / idle / exit:              │
-                  │    ↓                          │
-                  │  Query OpenCode DB            │
-                  │  since last sync              │
-                  │    ↓                          │
-                  │  Export → flat text files     │
-                  │    ↓                          │
-                  │  mempalace mine --mode convos │
-                  │  --extract general (async)    │
-                  │  single serialized call       │
-                  └──────────────────────────────┘
+  Answer done ──►│  chat.message (count) + session.idle   │
+                 │  mine on idle / exit / startup           │
+                 │    ↓                                     │
+                 │  Query OpenCode DB (completed turns)     │
+                 │    ↓                                     │
+                 │  Export → flat text files (0700)         │
+                 │    ↓                                     │
+                 │  mempalace mine --mode convos            │
+                 │  single serialized call                  │
+                 └──────────────────────────────────────────┘
                             │
                             ▼
                  ┌──────────────────────────┐
@@ -258,7 +259,8 @@ The plugin exports everything in the opencode database on the next sync, then re
 | `~/.config/opencode/skills/mempalace-recall/SKILL.md` | Bundled recall skill (copy from `skills/` in this repo) |
 | `~/.mempalace/identity.txt` | Your identity (injected by plugin) |
 | `~/.mempalace/hook_state/opencode_counters.json` | Per-session message counters (checkpoint cadence) |
-| `~/.mempalace/hook_state/hook.log` | Checkpoint / pre-compact event log |
+| `~/.mempalace/hook_state/hook.log` | Checkpoint / pre-compact event log (errors always land here) |
+| `~/.mempalace/oc-sessions/` | Private (0700) export workspace for pending transcripts |
 | `~/.mempalace/config.json` | MemPalace config (palace path) |
 | `~/.mempalace/knowledge_graph.sqlite3` | Knowledge Graph (structured facts) |
 | `~/opencode-memory/` | MemPalace vector DB (all drawers) |

package/dist/index.js

@@ -1,10 +1,9 @@
-import { execSync, exec, spawnSync } from "child_process";
+import { execSync, execFileSync, execFile, spawnSync } from "child_process";
 import { existsSync, readFileSync, writeFileSync, mkdirSync, rmdirSync, unlinkSync, appendFileSync } from "fs";
 import { homedir } from "os";
 import { join } from "path";
 import { createHash } from "crypto";
 const HOME = homedir();
-const VENV_PYTHON = join(HOME, ".local/share/pipx/venvs/mempalace/bin/python3");
 const MEMPALACE_BIN = join(HOME, ".local/bin/mempalace");
 const OPENCODE_DB = join(HOME, ".local/share/opencode/opencode.db");
 const STATE_FILE = join(HOME, ".mempalace/sync_state.json");
@@ -13,8 +12,11 @@ const IDENTITY_FILE = join(HOME, ".mempalace/identity.txt");
 const HOOK_STATE_DIR = join(HOME, ".mempalace/hook_state");
 const COUNTERS_FILE = join(HOOK_STATE_DIR, "opencode_counters.json");
 const HOOK_LOG = join(HOOK_STATE_DIR, "hook.log");
-const OUT_DIR = "/tmp/oc-sessions";
-const TMP_SCRIPT = "/tmp/oc-plugin-query.py";
+// Private sync workspace (0700): transcripts contain conversation text,
+// so they must never sit world-readable in /tmp (see PR #1524 review).
+const SYNC_DIR = join(HOME, ".mempalace/oc-sessions");
+const OUT_DIR = SYNC_DIR;
+const TMP_SCRIPT = join(SYNC_DIR, "oc-plugin-query.py");
 const DEBUG = !!process.env.OPENCODE_MEMPALACE_DEBUG;
 const LOG_FILE = "/tmp/opencode-mempalace.log";
 const MAX_INJECT_CHARS = 900;
@@ -38,6 +40,45 @@ function hookLog(msg) {
     }
     catch { }
 }
+// Errors are never silent: hook.log is always written (unlike the
+// DEBUG-gated log), so a broken pipeline is visible by default.
+function errLog(msg) {
+    log("ERROR: " + msg);
+    hookLog("ERROR: " + msg);
+}
+// Probe for a working Python interpreter at startup instead of hardcoding
+// one installer layout (pipx vs uv tool vs system). runPython only needs
+// stdlib (sqlite3/json), so any python3 works. Priority: explicit env
+// override, legacy pipx venv, uv tool venv, PATH fallback.
+let resolvedPython = undefined;
+function resolvePython() {
+    if (resolvedPython !== undefined)
+        return resolvedPython;
+    const candidates = [
+        process.env.MEMPALACE_PYTHON,
+        join(HOME, ".local/share/pipx/venvs/mempalace/bin/python3"),
+        join(HOME, ".local/share/uv/tools/mempalace/bin/python3"),
+    ].filter((p) => !!p && existsSync(p));
+    if (candidates.length > 0) {
+        resolvedPython = candidates[0];
+    }
+    else {
+        try {
+            execSync("python3 --version", { encoding: "utf-8", timeout: 10000 });
+            resolvedPython = "python3";
+        }
+        catch {
+            resolvedPython = null;
+        }
+    }
+    if (resolvedPython) {
+        log("using python: " + resolvedPython);
+    }
+    else {
+        errLog("no working Python interpreter found (tried MEMPALACE_PYTHON, pipx venv, uv tool venv, PATH python3) — DB export disabled");
+    }
+    return resolvedPython;
+}
 let miningLock = false;
 let lastSyncTs = 0;
 let wakeupDone = false;
@@ -46,14 +87,55 @@ let wakeupDone = false;
 // official Stop hook: the hook decides WHEN, the model decides WHAT).
 let pendingCheckpoint = null;
 function runPython(code) {
-    writeFileSync(TMP_SCRIPT, code);
+    const python = resolvePython();
+    if (!python)
+        throw new Error("no working Python interpreter (see hook.log)");
+    mkdirSync(SYNC_DIR, { recursive: true, mode: 0o700 });
+    writeFileSync(TMP_SCRIPT, code, { mode: 0o600 });
     try {
-        return execSync(`${VENV_PYTHON} ${TMP_SCRIPT}`, { encoding: "utf-8", timeout: 30000 }).trim();
+        // argv array, no shell (see PR #2): paths here are fixed, never user input.
+        return execFileSync(python, [TMP_SCRIPT], { encoding: "utf-8", timeout: 30000 }).trim();
     }
     finally {
-        unlinkSync(TMP_SCRIPT);
+        try {
+            unlinkSync(TMP_SCRIPT);
+        }
+        catch { }
     }
 }
+// Resolve the mempalace CLI without hardcoding one installer layout:
+// explicit env override, PATH lookup (cross-platform, incl. Windows),
+// legacy ~/.local/bin fallback.
+let resolvedBin = undefined;
+function resolveBin() {
+    if (resolvedBin !== undefined)
+        return resolvedBin;
+    const envBin = process.env.MEMPALACE_BIN;
+    if (envBin && existsSync(envBin)) {
+        resolvedBin = envBin;
+    }
+    else {
+        try {
+            const found = execSync(process.platform === "win32" ? "where mempalace" : "command -v mempalace", {
+                encoding: "utf-8", timeout: 10000,
+            }).trim().split(/\r?\n/)[0]?.trim();
+            if (found)
+                resolvedBin = found;
+        }
+        catch { }
+        if (!resolvedBin && existsSync(MEMPALACE_BIN))
+            resolvedBin = MEMPALACE_BIN;
+        if (!resolvedBin)
+            resolvedBin = null;
+    }
+    if (resolvedBin) {
+        log("using mempalace: " + resolvedBin);
+    }
+    else {
+        errLog("mempalace CLI not found (tried MEMPALACE_BIN env, PATH, ~/.local/bin) — search/wake-up/mine disabled");
+    }
+    return resolvedBin;
+}
 function hasText(parts) {
     return parts
         .filter((p) => p?.type === "text" && p?.text?.trim())
@@ -98,8 +180,12 @@ function persistCounters(counters) {
     }
 }
 function mempalaceWakeup() {
+    const bin = resolveBin();
+    if (!bin)
+        return "";
     try {
-        const out = execSync(`${MEMPALACE_BIN} wake-up`, { encoding: "utf-8", timeout: 15000 }).trim();
+        // argv array, no shell.
+        const out = execFileSync(bin, ["wake-up"], { encoding: "utf-8", timeout: 15000 }).trim();
         if (!out)
             return "";
         return out.slice(0, MAX_WAKEUP_CHARS);
@@ -132,8 +218,13 @@ function readIdentity() {
     }
 }
 function mempalaceSearch(query) {
+    const bin = resolveBin();
+    if (!bin)
+        return "";
     try {
-        const out = execSync(`${MEMPALACE_BIN} search "${query.replace(/"/g, '\\"')}" --results ${MAX_SEARCH_RESULTS}`, {
+        // argv array, no shell (see PR #2): the query is raw user message
+        // text, so it must never pass through /bin/sh. No manual escaping needed.
+        const out = execFileSync(bin, ["search", query, "--results", String(MAX_SEARCH_RESULTS)], {
             encoding: "utf-8",
             timeout: 15000,
         }).trim();
@@ -162,7 +253,7 @@ function dbSync() {
         doDbSync();
     }
     catch (e) {
-        log("sync err: " + String(e));
+        errLog("sync err: " + String(e));
     }
 }
 function backfillRequested() {
@@ -197,8 +288,11 @@ print(json.dumps(rows))
     if (!sessionsArr || sessionsArr.length === 0)
         return { wings: new Map(), now: Date.now() };
     const now = Date.now();
+    // Never advance the cursor past an in-flight reply: anything skipped
+    // as incomplete is revisited by the next sync (idle/exit/startup).
+    let cursor = now;
     const wings = new Map();
-    mkdirSync(OUT_DIR, { recursive: true });
+    mkdirSync(OUT_DIR, { recursive: true, mode: 0o700 });
     for (const sess of sessionsArr) {
         const [sessId, title, , directory] = sess;
         const wing = ((directory || "").split("/").filter(Boolean).pop() || "global")
@@ -214,10 +308,20 @@ rows = db.execute("""
   ORDER BY m.time_created
 """).fetchall()
 texts = []
+incomplete = []
 for (mid, mts, mdata_raw) in rows:
     try: mdata = json.loads(mdata_raw)
     except: mdata = {}
     role = mdata.get("role", "unknown")
+    # Completion tracking (see PR #1524 review): the assistant message row
+    # is created when a reply STARTS, parts stream in afterwards, and
+    # finish is set only on completion. Exporting mid-reply would
+    # snapshot partial parts while the cursor advances past the message
+    # timestamp — losing the rest of the reply forever. So assistant
+    # messages without finish are skipped and revisited next sync.
+    if role == "assistant" and not mdata.get("finish"):
+        incomplete.append(mts)
+        continue
     for (pdata_raw,) in db.execute("SELECT data FROM part WHERE message_id = ? ORDER BY time_created", (mid,)).fetchall():
         try:
             pdata = json.loads(pdata_raw)
@@ -225,16 +329,22 @@ for (mid, mts, mdata_raw) in rows:
                 texts.append({"role": role, "text": pdata.get("text").strip(), "ts": mts})
         except: pass
 db.close()
-print(json.dumps(texts))
+print(json.dumps({"texts": texts, "incomplete": incomplete}))
 `);
         let msgList;
+        let incompleteTs = [];
         try {
-            msgList = JSON.parse(msgs);
+            const parsed = JSON.parse(msgs);
+            msgList = parsed.texts;
+            incompleteTs = parsed.incomplete || [];
         }
         catch {
             continue;
         }
-        if (msgList.length < 2)
+        if (incompleteTs.length > 0) {
+            cursor = Math.min(cursor, Math.min(...incompleteTs) - 1);
+        }
+        if (msgList.length < 2 && incompleteTs.length === 0)
             continue;
         const lines = [
             `# ${title || label}`,
@@ -254,24 +364,28 @@ print(json.dumps(texts))
             continue;
         const contentHash = createHash("sha256").update(content).digest("hex").slice(0, 12);
         const wingDir = join(OUT_DIR, wing);
-        mkdirSync(wingDir, { recursive: true });
+        mkdirSync(wingDir, { recursive: true, mode: 0o700 });
         const fname = `sync_${prefix}_${contentHash}.txt`;
-        writeFileSync(join(wingDir, fname), content + "\n");
+        writeFileSync(join(wingDir, fname), content + "\n", { mode: 0o600 });
         if (!wings.has(wing))
             wings.set(wing, []);
         wings.get(wing).push(join(wingDir, fname));
     }
-    return { wings, now };
+    return { wings, now: cursor };
 }
 function markSynced(now) {
     writeFileSync(STATE_FILE, JSON.stringify({ last_sync_ms: now }));
     lastSyncTs = Date.now();
 }
-// Official classification: decisions, preferences, milestones, problems,
-// emotional context. Agent tag keeps opencode-mined drawers attributable.
+// Default `exchange` extraction: one drawer per exchange pair, verbatim,
+// no paraphrasing (see PR #1524 review). Intelligent filing (decisions,
+// KG facts, diary) happens through AI checkpoints, not the miner.
+// Agent tag keeps opencode-mined drawers attributable.
 // One wing per project (official multi-project pattern).
-function mineCommand(wingDir, wing) {
-    return `${MEMPALACE_BIN} mine ${wingDir} --mode convos --extract general --agent opencode --wing ${wing}`;
+// Argv array, no shell (see PR #2): wing names are sanitized, but the
+// spawn path stays shell-free regardless.
+function mineArgs(wingDir, wing) {
+    return ["mine", wingDir, "--mode", "convos", "--agent", "opencode", "--wing", wing];
 }
 function cleanupExport(wings) {
     for (const files of wings.values()) {
@@ -323,13 +437,22 @@ function doDbSync() {
             return;
         }
         const [wing, files] = entries[i];
-        exec(mineCommand(join(OUT_DIR, wing), wing), {
+        // No timeout here by design (see PR #4): Node would kill only the
+        // wrapper shell and orphan the python mine process, which keeps
+        // holding the palace lock while the next mine piles up. miningLock
+        // already serializes concurrent mines; long mines run to completion.
+        const bin = resolveBin();
+        if (!bin) {
+            miningLock = false;
+            errLog("mine skipped: mempalace CLI not found");
+            return;
+        }
+        execFile(bin, mineArgs(join(OUT_DIR, wing), wing), {
             encoding: "utf-8",
-            timeout: 300000,
         }, (err) => {
             if (err) {
                 miningLock = false;
-                log(`mine err (${wing}): ${err.message}`);
+                errLog(`mine err (${wing}): ${err.message}`);
                 return;
             }
             log(`mined wing ${wing} (${files.length} sessions)`);
@@ -339,20 +462,34 @@ function doDbSync() {
     mineNext(0);
 }
 // Best-effort synchronous save for process exit (SIGINT/SIGTERM/exit):
-// only synchronous calls are allowed here.
+// only synchronous calls are allowed here. Bounded by EXIT_BUDGET_MS so
+// shutdown stays fast; miningLock is deliberately ignored here because
+// any in-flight async mine dies with the process — at exit this sync
+// mine takes ownership (see PR #1524 review).
+const EXIT_BUDGET_MS = 45000;
+const EXIT_WING_TIMEOUT_MS = 30000;
 function exitSync() {
     try {
+        const bin = resolveBin();
+        if (!bin)
+            return;
         const { wings, now } = exportNewSessions(getLastSync());
         if (wings.size === 0)
             return;
+        const deadline = Date.now() + EXIT_BUDGET_MS;
         for (const [wing] of wings) {
+            const remaining = deadline - Date.now();
+            if (remaining <= 0) {
+                log("exit save: budget exhausted, rest covered next startup");
+                break;
+            }
             log(`exit save: mining wing ${wing}`);
-            const res = spawnSync(MEMPALACE_BIN, ["mine", join(OUT_DIR, wing), "--mode", "convos", "--extract", "general", "--agent", "opencode", "--wing", wing], {
+            const res = spawnSync(bin, mineArgs(join(OUT_DIR, wing), wing), {
                 encoding: "utf-8",
-                timeout: 60000,
+                timeout: Math.min(EXIT_WING_TIMEOUT_MS, remaining),
             });
             if (res.error || res.status !== 0) {
-                log(`exit mine err (${wing}): ${String(res.error || res.status)}`);
+                errLog(`exit mine err (${wing}): ${String(res.error || res.status)}`);
                 return;
             }
         }
@@ -361,16 +498,19 @@ function exitSync() {
         log("exit save done");
     }
     catch (e) {
-        log("exit save err: " + String(e));
+        errLog("exit save err: " + String(e));
     }
 }
 export default (async () => {
-    mkdirSync(OUT_DIR, { recursive: true });
+    mkdirSync(OUT_DIR, { recursive: true, mode: 0o700 });
     mkdirSync(HOOK_STATE_DIR, { recursive: true });
     const autoInject = isAutoInjectEnabled();
     const identity = readIdentity();
     const interval = saveInterval();
     log(`loaded (autoInjectContext: ${autoInject}, saveInterval: ${interval})`);
+    // Catch anything missed by a previous run (e.g. content skipped when
+    // the exit budget ran out). Fires once per server lifetime.
+    setTimeout(() => dbSync(), 10000);
     // Crash safety: best-effort synchronous save on hard exit.
     // Mirrors the official emergency-save intent (nothing async allowed here).
     let exitHandled = false;
@@ -395,6 +535,10 @@ export default (async () => {
             // Official Save-hook cadence: count human messages per session,
             // persist like ~/.mempalace/hook_state/, arm ONE AI checkpoint
             // per boundary. The model decides WHAT to file.
+            // NOTE: no mine here by design (see PR #1524 review) — mining a
+            // mid-reply snapshot would export partial assistant parts. Mines
+            // run on idle/exit/startup, when turns are complete; the export
+            // additionally skips unfinished replies (finish tracking).
             const counters = loadCounters();
             const c = counters[sessionID] || { humanMsgs: 0, lastCheckpoint: 0 };
             c.humanMsgs += 1;
@@ -403,8 +547,6 @@ export default (async () => {
                 c.lastCheckpoint = boundary;
                 pendingCheckpoint = { sessionID, count: c.humanMsgs };
                 hookLog(`session ${sessionID}: ${c.humanMsgs} human msgs — checkpoint armed`);
-                log("threshold crossed - queue sync");
-                setTimeout(() => dbSync(), 500);
             }
             counters[sessionID] = c;
             persistCounters(counters);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "opencode-mempalace-persistence",
-  "version": "2.0.1",
+  "version": "2.2.0",
   "description": "OpenCode plugin — auto-sync conversations to MemPalace memory in real-time. No forced wings, KG extraction via MCP tools.",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",