opencode-mempalace-persistence 1.0.1 → 1.1.1

package/README.md

@@ -1,8 +1,8 @@
 # opencode-mempalace-persistence
 
-An OpenCode plugin that automatically saves every conversation to MemPalace and uses stored memory to provide better, context-aware responses. Real-time, zero cron, zero external scripts.
+> **Community plugin** — not officially maintained by the MemPalace team. Fully open source, ~120 lines of TypeScript.
 
-> **It just works** — install, use OpenCode, the plugin handles the rest.
+An OpenCode plugin that automatically saves every conversation to MemPalace and uses stored memory to provide better, context-aware responses. Real-time, zero cron, zero external scripts.
 
 [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
 
@@ -14,7 +14,7 @@ An OpenCode plugin that automatically saves every conversation to MemPalace and
 |---|---|
 | Every session starts from scratch | The model knows who you are and what you've done |
 | You repeat context each time | Memory is automatic |
-| No Knowledge Graph | Decisions, milestones, problems tracked |
+| Model starts from scratch each time | Memory persists across sessions |
 
 The model searches MemPalace on every question (via AGENTS.md + MCP), and the plugin saves every response. A perfect feedback loop.
 
@@ -39,11 +39,40 @@ Create `~/.config/opencode/AGENTS.md`:
 ```markdown
 # Memory & Knowledge instructions
 
-Before answering the user, always search your MemPalace memory for relevant context using the MCP tools.
+## CRITICAL: You MUST follow these steps BEFORE every response.
+
+### Step 1 — Search MemPalace
+Call `mempalace_mempalace_search` with the user's question or key topics as query. Get the top 5-10 most relevant memory drawers.
+**This is mandatory. Never skip this step. No exceptions.** Every question — even seemingly trivial ones — may relate to past conversations, projects, or preferences you've forgotten. Searching costs ~nothing; skipping costs continuity.
+
+### Step 2 — Query Knowledge Graph
+Call `mempalace_mempalace_kg_query` for entity "user". Then filter the returned facts — keep only those whose text contains keywords from the user's question, so irrelevant facts are excluded.
+
+### Step 3 — Use the context
+Incorporate relevant memory and facts into your response. If the user's question relates to a past conversation, project, or decision, reference it naturally. If the search returned nothing useful, you are done — do NOT skip steps 1-2 just because you *expect* nothing.
+
+### Naming reminder
+All MemPalace tools use the prefix `mempalace_mempalace_*` (not `mempalace_*`). Examples:
+- `mempalace_mempalace_search` (NOT `mempalace_search`)
+- `mempalace_mempalace_kg_query` (NOT `mempalace_kg_query`)
+- `mempalace_mempalace_add_drawer`
+- `mempalace_mempalace_get_drawer`
+- `mempalace_mempalace_list_wings`
+If you ever catch yourself typing `mempalace_search` or `mempalace_kg_query`, STOP — the correct prefix is `mempalace_mempalace_`.
+
+## Note
 
-1. **Search MemPalace**: Call `mempalace_search` with the user's question or key topics as query. Get the top 5-10 most relevant memory drawers.
-2. **Query Knowledge Graph**: Call `mempalace_kg_query` for entity "user". Then filter the returned facts — keep only those whose text contains keywords from the user's question.
-3. **Use the context**: Incorporate relevant memory and facts into your response, referencing the source naturally when useful.
+The KG stores structured facts like: decision (choices made), milestone (completed tasks), problem (issues encountered), preference (likes/dislikes), emotional (feelings). Use these to understand the user's history and context better.
+
+### Knowledge Graph management
+
+When you discover structured facts during conversation (decisions made, milestones reached, problems encountered, preferences expressed, emotional states), record them in the Knowledge Graph:
+
+- **New facts**: Call `mempalace_mempalace_kg_add` with subject → predicate → object (e.g. `subject="user"`, `predicate="preference"`, `object="prefers TypeScript over Python"`)
+- **Changed facts**: First call `mempalace_mempalace_kg_invalidate` on the old fact, then `mempalace_mempalace_kg_add` for the new one
+- **Retrieval**: Call `mempalace_mempalace_kg_query` for entity "user" to see all known facts
+
+This is optional but recommended — the more facts you record, the better the model understands the user's history and preferences.
 ```
 
 ### 3. Identity (who you are)
@@ -78,8 +107,10 @@ This is loaded automatically at session start via `instructions` in opencode.jso
 ### 5. MemPalace (if not already installed)
 
 ```bash
-# Install
-pipx install mempalace
+# Install (requires mempalace>=3.3.5 for HNSW corruption fix)
+uv tool install "mempalace>=3.3.5"
+# or
+pipx install "mempalace>=3.3.5"
 
 # Create palace
 mempalace init ~/opencode-memory
@@ -102,8 +133,8 @@ You ask a question
 
 The model responds
   → The opencode-mempalace-persistence plugin detects the response is complete
-  → Saves the conversation to MemPalace
-  → Extracts Knowledge Graph facts
+  → Saves the conversation to MemPalace (flat export, no hardcoded wings)
+  → The model may record KG facts via MCP tools (optional, per-session)
 
 Next time you ask
   → The model finds the previous memory → coherent responses
@@ -114,15 +145,7 @@ Next time you ask
 
 ## What gets saved
 
-Every turn (question + answer):
-
-- **Text** categorized by wing (developer, creative, emotions, family, consciousness)
-- **Knowledge Graph**: automatically extracted facts
-  - `decision` → "decided to use TypeScript"
-  - `milestone` → "backend deploy completed"
-  - `problem` → "chromadb ModuleNotFoundError"
-  - `preference` → "prefer Svelte over React"
-  - `emotional` → "frustrated with Docker compose"
+Every turn (question + answer) is saved as a drawer in MemPalace. No forced categorization — MemPalace's own mining handles organization. The model can optionally record KG facts (decisions, milestones, preferences) during conversation via MCP tools.
 
 ---
 
@@ -137,15 +160,13 @@ Every turn (question + answer):
                  │  Query OpenCode DB        │
                  │  (messages since lastSync)│
                  │    ↓                      │
-                 │  Categorize by wing       │
-                 │    ↓                      │
-                 │  Export delta → tmp       │
+                 │  Export sessions → flat   │
+                 │  (no wing subdirs)        │
                  │    ↓                      │
                  │  Save state immediately   │
                  │    ↓                      │
-                 │  mempalace mine (async)   │ ← non-blocking
-                 │    ↓                      │
-                 │  Extract KG facts         │
+                 │  mempalace mine (async)   │
+                 │  single call, serialized  │
                  │    ↓                      │
   Session idle ─►│  session.idle hook        │
                  │  (saves last turn)        │
@@ -155,7 +176,7 @@ Every turn (question + answer):
                  ┌──────────────────────────┐
                  │      MemPalace            │
                  │  ~/opencode-memory/       │
-                 │  Vector DB + KG SQLite    │
+                 │  Vector DB                │
                  └──────────────────────────┘
                             ▲
                             │
@@ -163,7 +184,9 @@ Every turn (question + answer):
                  │  AGENTS.md + MCP          │
                  │  The model searches       │
                  │  MemPalace on every       │
-                 │  question                 │
+                 │  question. Optionally     │
+                 │  records KG facts via     │
+                 │  kg_add / kg_invalidate   │
                  └──────────────────────────┘
 ```
 
@@ -209,6 +232,36 @@ When set, the plugin writes a debug log to `/tmp/opencode-mempalace.log`.
 
 ---
 
+## Recommendations
+
+### Model choice affects memory retrieval reliability
+
+Empirical data from the MemPalace community (Cat-9a diagnostic) shows that the model's tool-use discipline significantly impacts how reliably `mempalace_search` is invoked:
+
+| Model | Skips search | Mean recall |
+|-------|:-:|:-:|
+| gemma4:e4b (4B) | **60%** | 0.417 |
+| qwen3.5:4b (4B, Tau2-tuned) | **13%** | 0.717 |
+
+For reliable read-side memory retrieval, **Qwen 3.5 4B+ or equivalent** is recommended as the minimum orchestrator. Smaller or older models may skip memory search on most questions regardless of AGENTS.md instructions.
+
+### Forced invocation (belt and suspenders)
+
+A plugin-level config flag that injects a mandatory `mempalace_search` directive into the system prompt (on top of AGENTS.md) can recover ~15pp of recall on low-discipline models. This is being evaluated as a future config option — suggestions welcome.
+
+### Complementary: upstream OpenCode source adapter
+
+The MemPalace project has an upstream PR ([#1484](https://github.com/MemPalace/mempalace/pull/1484)) adding `mempalace mine --source opencode` — a pull-based adapter for retrospective ingest of existing OpenCode sessions. This plugin (push, real-time) and the adapter (pull, backfill) are complementary:
+
+| Approach | Direction | Captures |
+|----------|-----------|----------|
+| This plugin | Push | Live conversation turns |
+| `mempalace mine --source opencode` (PR #1484) | Pull | Existing OpenCode session files |
+
+For full coverage: install this plugin for live capture, run `mempalace mine --source opencode` once for backfill, never think about it again.
+
+---
+
 ## License
 
 MIT

package/dist/index.js

@@ -1,5 +1,5 @@
 import { execSync, exec } from "child_process";
-import { existsSync, readFileSync, writeFileSync, mkdirSync, rmSync, rmdirSync, unlinkSync, appendFileSync } from "fs";
+import { existsSync, readFileSync, writeFileSync, mkdirSync, rmdirSync, unlinkSync, appendFileSync } from "fs";
 import { homedir } from "os";
 import { join } from "path";
 import { createHash } from "crypto";
@@ -8,7 +8,6 @@ const VENV_PYTHON = join(HOME, ".local/share/pipx/venvs/mempalace/bin/python3");
 const MEMPALACE_BIN = join(HOME, ".local/bin/mempalace");
 const OPENCODE_DB = join(HOME, ".local/share/opencode/opencode.db");
 const STATE_FILE = join(HOME, ".mempalace/sync_state.json");
-const KG_DB = join(HOME, ".mempalace/knowledge_graph.sqlite3");
 const OUT_DIR = "/tmp/oc-sessions";
 const TMP_SCRIPT = "/tmp/oc-plugin-query.py";
 const DEBUG = !!process.env.OPENCODE_MEMPALACE_DEBUG;
@@ -22,7 +21,7 @@ function log(msg) {
     }
     catch { }
 }
-let pendingSync = false;
+let miningLock = false;
 let lastSyncTs = 0;
 function runPython(code) {
     writeFileSync(TMP_SCRIPT, code);
@@ -33,29 +32,6 @@ function runPython(code) {
         unlinkSync(TMP_SCRIPT);
     }
 }
-const WING_KEYWORDS = {
-    creative: ["eurovision", "music", "art", "film", "game", "festival", "song", "concert"],
-    family: ["family", "kid", "son", "daughter", "wife", "husband", "parent", "mother", "father"],
-    emotions: ["feel", "happy", "sad", "love", "hate", "angry", "fear", "anxiety", "grateful"],
-    consciousness: ["think", "mind", "conscious", "self", "identity", "exist", "meaning"],
-};
-const KG_PATTERNS = {
-    decision: ["ho deciso", "ho scelto", "implementato", "added ", "ho aggiunto", "decided", "chosen", "created", "creato", "modified"],
-    milestone: ["completato", "finito", "rilasciato", "deployato", "ho completato", "successo", "funziona", "completed", "done", "deployed"],
-    problem: ["problema", "bug", "errore", "issue", "fix", "fixato", "risolto", "solved", "fixed", "error"],
-    preference: ["preferisco", "mi piace", "non mi piace", "prefer", "like to", "better"],
-    emotional: ["sentimento", "emozione", "frustrato", "contento", "happy", "sad", "feel", "emotion"],
-};
-function categorize(text) {
-    if (!text)
-        return "developer";
-    const lower = text.toLowerCase();
-    for (const [wing, keywords] of Object.entries(WING_KEYWORDS)) {
-        if (keywords.some((kw) => lower.includes(kw)))
-            return wing;
-    }
-    return "developer";
-}
 function hasText(parts) {
     return parts
         .filter((p) => p?.type === "text" && p?.text?.trim())
@@ -73,9 +49,8 @@ function getLastSync() {
     }
 }
 function dbSync() {
-    if (pendingSync)
+    if (miningLock)
         return;
-    pendingSync = true;
     try {
         doDbSync();
     }
@@ -106,17 +81,15 @@ print(json.dumps(rows))
         sessionsArr = JSON.parse(sessions);
     }
     catch {
-        pendingSync = false;
         return;
     }
-    if (!sessionsArr || sessionsArr.length === 0) {
-        pendingSync = false;
+    if (!sessionsArr || sessionsArr.length === 0)
         return;
-    }
     const now = Date.now();
     const exported = [];
+    mkdirSync(OUT_DIR, { recursive: true });
     for (const sess of sessionsArr) {
-        const [sessId, title, worktree, directory] = sess;
+        const [sessId, title] = sess;
         const label = (title || "").replace(/[^a-zA-Z0-9 _-]/g, "_") || (sessId || "").slice(0, 12);
         const prefix = `${new Date().toISOString().slice(0, 10)}_${label.slice(0, 30)}_${(sessId || "").slice(0, 8)}`;
         const msgs = runPython(`
@@ -158,7 +131,7 @@ print(json.dumps(texts))
         ];
         for (const m of msgList) {
             const ts = m.ts ? new Date(m.ts).toISOString().slice(11, 19) : "";
-            lines.push(`## ${m.role.toUpperCase()} — ${ts}`);
+            lines.push(`## ${m.role.toUpperCase()} \u2014 ${ts}`);
             lines.push("");
             lines.push(m.text);
             lines.push("");
@@ -166,88 +139,38 @@ print(json.dumps(texts))
         const content = lines.join("\n").trim();
         if (!content)
             continue;
-        const wing = categorize(content);
         const contentHash = createHash("sha256").update(content).digest("hex").slice(0, 12);
         const fname = `sync_${prefix}_${contentHash}.txt`;
-        const wingDir = join(OUT_DIR, wing);
-        mkdirSync(wingDir, { recursive: true });
-        writeFileSync(join(wingDir, fname), content + "\n");
-        exported.push({ wing, fname, dir: wingDir });
+        writeFileSync(join(OUT_DIR, fname), content + "\n");
+        exported.push(join(OUT_DIR, fname));
     }
-    if (exported.length === 0) {
-        pendingSync = false;
+    if (exported.length === 0)
         return;
-    }
-    // Save state immediately so next sync doesn't re-process
     writeFileSync(STATE_FILE, JSON.stringify({ last_sync_ms: now }));
     lastSyncTs = Date.now();
-    pendingSync = false;
-    log(`queued ${exported.length} files for mining`);
-    // Mining async (fire-and-forget, non-blocking)
-    for (const { wing, fname, dir: wingDir } of exported) {
-        const filePath = join(wingDir, fname);
-        exec(`${MEMPALACE_BIN} mine ${wingDir} --mode convos --extract general --wing ${wing}`, {
-            encoding: "utf-8",
-            timeout: 120000,
-        }, (err) => {
-            if (err) {
-                log(`mine err ${wing}: ${err.message}`);
-                return;
-            }
-            rmSync(filePath);
-            try {
-                rmdirSync(wingDir);
-            }
-            catch { }
-            log(`mined ${wing}`);
-            // KG extraction after mining
+    miningLock = true;
+    log(`mining ${exported.length} sessions`);
+    exec(`${MEMPALACE_BIN} mine ${OUT_DIR} --mode convos`, {
+        encoding: "utf-8",
+        timeout: 120000,
+    }, (err) => {
+        miningLock = false;
+        if (err) {
+            log(`mine err: ${err.message}`);
+            return;
+        }
+        for (const f of exported) {
             try {
-                const raw = runPython(`
-from mempalace.config import MempalaceConfig
-import chromadb, json, re
-c = MempalaceConfig()
-col = chromadb.PersistentClient(path=c.palace_path).get_collection(c.collection_name)
-d = col.get(limit=5000, include=["metadatas","documents"])
-p = ${JSON.stringify(KG_PATTERNS)}
-r = []
-for i,m,doc in zip(d.get("ids",[]), d.get("metadatas",[]), d.get("documents",[])):
-  if not m or not doc: continue
-  rm = m.get("room","")
-  if rm not in ("decision","milestone","problem","preference","emotional"): continue
-  da = doc.split("Date: ")[1][:10] if "Date: " in doc else ""
-  if not da: continue
-  if not any(pat in doc.lower() for pat in p.get(rm,[])): continue
-  for ln in doc.strip().split("\\n")[:5]:
-    l = ln.strip()
-    if l and not l.startswith("#") and len(l) > 20 and len(re.findall(r"[^a-zA-Z0-9\\s]", l))/len(l) <= 0.3:
-      r.append([da, rm, l[:120]]); break
-print(json.dumps(r))
-`);
-                const nf = JSON.parse(raw) || [];
-                if (nf.length > 0) {
-                    const ex = runPython(`import sqlite3,json;d=sqlite3.connect(${JSON.stringify(KG_DB)});r=d.execute("SELECT predicate,object FROM triples WHERE subject='user'").fetchall();d.close();print(json.dumps(r))`);
-                    const seen = new Set();
-                    try {
-                        for (const [p, o] of JSON.parse(ex))
-                            seen.add(p + "::" + o);
-                    }
-                    catch { }
-                    const ins = [];
-                    for (const [d, t, x] of nf) {
-                        const k = t + "::" + x;
-                        if (seen.has(k))
-                            continue;
-                        seen.add(k);
-                        ins.push(`INSERT OR IGNORE INTO triples(id,subject,predicate,object,valid_from,confidence,extracted_at) VALUES('t_user_${t}_${createHash("sha256").update(x).digest("hex").slice(0, 12)}','user','${t}','${x.replace(/'/g, "''")}','${d}',1.0,datetime('now'))`);
-                    }
-                    if (ins.length > 0) {
-                        runPython(`import sqlite3;db=sqlite3.connect(${JSON.stringify(KG_DB)});[db.execute(s) for s in ${JSON.stringify(ins)}];db.commit();db.close()`);
-                    }
-                }
+                unlinkSync(f);
             }
             catch { }
-        });
-    }
+        }
+        try {
+            rmdirSync(OUT_DIR);
+        }
+        catch { }
+        log("mine done");
+    });
 }
 export default (async () => {
     mkdirSync(OUT_DIR, { recursive: true });

package/package.json

@@ -1,15 +1,23 @@
 {
   "name": "opencode-mempalace-persistence",
-  "version": "1.0.1",
-  "description": "OpenCode plugin — auto-sync conversations to MemPalace memory in real-time with auto-categorization and Knowledge Graph",
+  "version": "1.1.1",
+  "description": "OpenCode plugin — auto-sync conversations to MemPalace memory in real-time. No forced wings, KG extraction via MCP tools.",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
-  "files": ["dist"],
+  "files": [
+    "dist"
+  ],
   "scripts": {
     "build": "tsc",
     "prepublishOnly": "npm run build"
   },
-  "keywords": ["opencode", "mempalace", "memory", "opencode-plugin", "typescript"],
+  "keywords": [
+    "opencode",
+    "mempalace",
+    "memory",
+    "opencode-plugin",
+    "typescript"
+  ],
   "author": "geco",
   "license": "MIT",
   "type": "module",