opencode-memory-pro 1.4.0 → 1.4.1

package/README.md

@@ -42,12 +42,6 @@ opencode plugin opencode-memory-pro
 
 The latest release is **v1.4.0** on [npm](https://www.npmjs.com/package/opencode-memory-pro); source and releases are on [GitHub](https://github.com/tman204-50/opencode-memory-pro).
 
-Remove the old plugin pin at the same time:
-
-```bash
-opencode plugin lancedb-opencode-pro -g   # removes pin (if installed)
-```
-
 ### Getting started
 
 **1. Install and restart OpenCode** — done above. That's it for a baseline
@@ -432,6 +426,7 @@ All tools are auto-registered when the plugin loads. Hybrid recall surfaces
 | `memory_event_cleanup` | Clean up expired effectiveness events (optional archive). |
 | `memory_consolidate` | Merge near-duplicate memories in a scope. |
 | `memory_consolidate_all` | Global duplicate cleanup (daily cron friendly). |
+| `memory_reembed` | Detect/repair an embedding-dimension mismatch (backs up, rebuilds the table, re-embeds every memory). |
 
 **Scoping**
 
@@ -480,15 +475,47 @@ npm run verify      # tests + pack dry-run
 
 CI runs on GitHub Actions (Node 22 + 24) on every push/PR to `main`.
 
-## Migrating from `lancedb-opencode-pro`
-
-Clean-break rename: sidecar is `opencode-memory-pro.json`, env prefix is
-`OPENCODE_MEMORY_PRO_*`. Data is **not** affected — the default storage paths
-are unchanged (`~/.opencode/memory/lancedb` + `~/.opencode/memory/graph.db`),
-so your memories and graph carry over untouched.
-
 ## Changelog
 
+### v1.4.1 (2026-09-06)
+
+New `memory_reembed` tool — detects and repairs embedding-dimension
+mismatches, which previously corrupted the store silently:
+
+- **Root cause**: the `memories` table's `vector` column is an Arrow
+  `FixedSizeList` whose width is fixed forever by the first row ever
+  written. `init()` re-probes the embedder's dimension on every startup but
+  silently discarded that value once a table already existed — nothing ever
+  compared "what the embedder produces now" against "what the table is
+  physically built for." Switching `embedding.provider`/`embedding.model` to
+  a different-dimension model did not error: LanceDB silently coerced
+  mismatched writes into the old fixed-width column (corrupting the vector,
+  not rejecting the write), and every `vectorSearch()` call at the new
+  dimension threw inside `findSimilarVectors`'s catch block, which silently
+  swallowed it — so write-time dedup and `memory_consolidate` silently
+  stopped finding neighbors for anything written after the switch, with zero
+  visible symptom beyond a passive `memory_stats.incompatibleVectors` count.
+- **Detection**: `init()` now reads back the table's actual physical vector
+  width (`getPhysicalVectorDim()`) and compares it to the freshly-probed
+  embedder dimension on every startup, logging a `warn` on mismatch.
+  `getIndexHealth()` (and therefore `memory_stats.index`) now reports
+  `dimensionMismatch`/`expectedDim`/`actualDim`, and `computeDegradedFlags`
+  surfaces an `embedding-dimension-mismatch` flag pointing at the fix.
+- **Repair**: `memory_reembed` (`dryRun` default `true`, `confirm` gate for
+  the actual repair — same pattern as `memory_clear`/`memory_forget`)
+  discovers every scope in the store (a dimension mismatch is table-wide,
+  not scope-scoped), backs up every memory to
+  `<dbPath's parent>/backups/reembed-repair-<ts>.json` (same shape as
+  `memory_export`, written *before* any mutation, always), then drops and
+  recreates the `memories` table at the current embedder's dimension and
+  re-embeds every memory from its stored text under its original id (so
+  entity-graph edges and citation chains keyed by id stay valid).
+- **Tests**: new integration test covers detection on a freshly-created
+  table (no false positive), detection after reopening with a different
+  dimension, and a full repair pass — asserting the physical column width
+  actually changes, every original id/text survives, and post-repair health
+  reports no mismatch.
+
 ### v1.4.0 (2026-09-06)
 
 Dedup correctness overhaul — the write-time duplicate check compared against

package/dist/index.js

@@ -11,7 +11,7 @@ import { requestLLMCapture, isOwnSession } from "./llm.js";
 import { createMemoryTools, createFeedbackTools, createEpisodicTools } from "./tools/index.js";
 import { sweepExpiredMemories } from "./tools/memory.js";
 import { createGraphStore } from "./graph.js";
-const PLUGIN_VERSION = "1.4.0";
+const PLUGIN_VERSION = "1.4.1";
 const SCHEMA_VERSION = 1;
 // Event-driven dedup: run consolidateDuplicates on session.idle (throttled to
 // this interval so chatty sessions aren't re-scanning the store every turn)

package/dist/store.d.ts

@@ -98,7 +98,12 @@ export declare class MemoryStore {
         ftsError?: string;
         vectorRetries?: number;
         ftsRetries?: number;
+        dimensionMismatch: boolean;
+        expectedDim: number | null;
+        actualDim: number | null;
     };
+    getPhysicalVectorDim(): Promise<number | null>;
+    listDistinctScopes(): Promise<string[]>;
     private invalidateScope;
     private getCachedScopes;
     private enforceMaxScopes;

package/dist/store.js

@@ -56,6 +56,13 @@ export class MemoryStore {
         ftsError: "",
         vectorRetries: 0,
         ftsRetries: 0,
+        // DIMENSION_MISMATCH_DETECT: set by init() by comparing the live
+        // embedder's probed dimension against the "vector" column's actual
+        // physical FixedSizeList width (fixed forever once the table's first
+        // row is written). See getPhysicalVectorDim() / repairEmbeddingDimension().
+        dimensionMismatch: false,
+        expectedDim: null,
+        actualDim: null,
     };
     scopeCache = new Map();
     // SCOPE_CACHE_LAZY (1.1.7): per-scope write counter. invalidateScope()
@@ -362,6 +369,35 @@ export class MemoryStore {
         await this.ensureMemoriesTableCompatibility();
         await this.ensureEventTableCompatibility();
         await this.ensureIndexes();
+        // DIMENSION_MISMATCH_DETECT: compare the embedder dimension this
+        // process just probed (vectorDim, the init() argument) against the
+        // table's actual physical column width. They only diverge when
+        // embedding.provider/embedding.model was changed to a different-
+        // output-size model without resetting the store — and when that
+        // happens, LanceDB does NOT reject the mismatched write; it silently
+        // coerces it into the old fixed-width column (corrupting the vector),
+        // and every vectorSearch() call at the new dimension throws (silently
+        // swallowed by findSimilarVectors's catch), so dedup/consolidation
+        // silently stop finding neighbors for anything written after the
+        // switch. See repairEmbeddingDimension() for the fix.
+        try {
+            const physicalDim = await this.getPhysicalVectorDim();
+            this.indexState.expectedDim = physicalDim;
+            this.indexState.actualDim = vectorDim;
+            this.indexState.dimensionMismatch = physicalDim !== null && physicalDim !== vectorDim;
+            if (this.indexState.dimensionMismatch) {
+                log("warn", `[store] Embedding dimension mismatch: the embedder currently produces ` +
+                    `${vectorDim}-dim vectors, but this store's "vector" column is physically fixed ` +
+                    `at ${physicalDim}-dim (set when the table was first created). New memories will ` +
+                    `be written with corrupted vectors and dedup/consolidation will silently stop ` +
+                    `finding neighbors for anything written from now on. Fix: call the memory_reembed ` +
+                    `tool (dryRun:false, confirm:true) to back up and re-embed every memory under the ` +
+                    `current model.`);
+            }
+        }
+        catch (error) {
+            log("debug", `[store] dimension-mismatch check failed: ${error instanceof Error ? error.message : String(error)}`);
+        }
         const retentionDays = this.retentionConfig?.effectivenessEventsDays;
         if (retentionDays !== undefined && retentionDays > 0) {
             await this.cleanupExpiredEvents(undefined, retentionDays);
@@ -1673,8 +1709,33 @@ export class MemoryStore {
             ftsError: this.indexState.ftsError || undefined,
             vectorRetries: this.indexState.vectorRetries,
             ftsRetries: this.indexState.ftsRetries,
+            dimensionMismatch: this.indexState.dimensionMismatch,
+            expectedDim: this.indexState.expectedDim,
+            actualDim: this.indexState.actualDim,
         };
     }
+    // DIMENSION_MISMATCH_DETECT: the "vector" column is an Arrow
+    // FixedSizeList whose width is fixed forever by the first row ever
+    // written to the table (LanceDB/Arrow enforce a uniform width per
+    // column) — NOT by whatever `vectorDim` a later write claims in its
+    // bookkeeping column. Reading it back via table.schema() is the only
+    // reliable way to know the table's true, physical embedding dimension.
+    async getPhysicalVectorDim() {
+        const table = this.requireTable();
+        const schema = await table.schema();
+        const vectorField = schema.fields.find((field) => field.name === "vector");
+        const listSize = vectorField?.type?.listSize;
+        return typeof listSize === "number" ? listSize : null;
+    }
+    // DIMENSION_MISMATCH_REPAIR: a dimension mismatch is a whole-table
+    // structural problem (the physical column width is table-wide, not
+    // scope-scoped), so the repair must span every scope present, not just
+    // the caller's current scope.
+    async listDistinctScopes() {
+        const table = this.requireTable();
+        const rows = await table.query().select(["scope"]).limit(200000).toArray();
+        return [...new Set(rows.map((row) => String(row.scope ?? "")).filter((scope) => scope.length > 0))];
+    }
     invalidateScope(scope) {
         this.scopeVersions.set(scope, (this.scopeVersions.get(scope) ?? 0) + 1);
     }

package/dist/tools/memory.d.ts

@@ -92,6 +92,17 @@ export declare function createMemoryTools(state: ToolRuntimeState): {
             scope?: string | undefined;
         }, context: import("@opencode-ai/plugin").ToolContext): Promise<string>;
     };
+    memory_reembed: {
+        description: string;
+        args: {
+            dryRun: import("zod").ZodDefault<import("zod").ZodOptional<import("zod").ZodBoolean>>;
+            confirm: import("zod").ZodDefault<import("zod").ZodOptional<import("zod").ZodBoolean>>;
+        };
+        execute(args: {
+            dryRun: boolean;
+            confirm: boolean;
+        }): Promise<string>;
+    };
     memory_event_cleanup: {
         description: string;
         args: {

package/dist/tools/memory.js

@@ -32,6 +32,12 @@ function computeDegradedFlags(state, embedderHealth, graphStats) {
     if (state.config?.capture?.llm?.provider && state.config?.capture?.llm?.model && getLlmHealth().status === "error") {
         flags.push("llm-unhealthy: last LLM capture/digest call failed — falling back to heuristics/extractive digests");
     }
+    const idx = state.store?.getIndexHealth?.();
+    if (idx?.dimensionMismatch) {
+        flags.push(`embedding-dimension-mismatch: embedder produces ${idx.actualDim}-dim vectors but the ` +
+            `store is fixed at ${idx.expectedDim}-dim — new writes are being silently corrupted and ` +
+            `dedup/consolidation are silently disabled. Run memory_reembed (dryRun:false, confirm:true) to repair.`);
+    }
     return flags;
 }
 // LLM_CAPTURE (1.1): mode-aware digest builder shared by memory_summarize
@@ -1226,6 +1232,117 @@ ${explanations.join("\n")}`;
                 }, null, 2);
             },
         }),
+        // DIMENSION_MISMATCH_REPAIR: the "vector" column's physical width is
+        // fixed for the whole table (set by the first row ever written), not
+        // per-scope, so this operates on every scope in the store — unlike
+        // every other tool here, it does not take a `scope` argument.
+        // Backs up first (always, even dryRun) so the operation is never
+        // riskier than memory_export followed by memory_import(replace).
+        memory_reembed: tool({
+            description: "Detect (and, with confirm:true, repair) an embedding-dimension mismatch between the " +
+                "configured embedder and the on-disk vector store. A mismatch happens when embedding.provider " +
+                "or embedding.model changed to a different output dimension without resetting the store — " +
+                "LanceDB silently corrupts new writes in that state instead of rejecting them, and dedup/" +
+                "consolidation silently stop finding neighbors. Repair backs up every memory (all scopes) to " +
+                "a JSON file, drops and recreates the memories table at the current embedder's dimension, and " +
+                "re-embeds every memory from its stored text under its original id (graph edges and citation " +
+                "chains keyed by id stay valid).",
+            args: {
+                dryRun: tool.schema.boolean().optional().default(true),
+                confirm: tool.schema.boolean().optional().default(false),
+            },
+            execute: async (args) => {
+                await state.ensureInitialized();
+                if (!state.initialized)
+                    return unavailableMessage(state.config.embedding.provider);
+                const actualDim = await state.embedder.dim();
+                const expectedDim = await state.store.getPhysicalVectorDim();
+                if (expectedDim === null || expectedDim === actualDim) {
+                    return JSON.stringify({
+                        mismatch: false,
+                        actualDim,
+                        message: "No dimension mismatch detected. Nothing to repair.",
+                    }, null, 2);
+                }
+                const scopes = await state.store.listDistinctScopes();
+                const records = await state.store.exportAllRecords(scopes);
+                if (args.dryRun && !args.confirm) {
+                    return JSON.stringify({
+                        mismatch: true,
+                        expectedDim,
+                        actualDim,
+                        scopes,
+                        recordCount: records.length,
+                        message: "Dry run — no changes made. Call again with dryRun:false, confirm:true to " +
+                            "repair (this drops and rebuilds the memories table; a backup is written first).",
+                    }, null, 2);
+                }
+                if (!args.confirm) {
+                    return JSON.stringify({
+                        error: "Set confirm:true to actually repair — this drops and rebuilds the memories " +
+                            "table (like memory_clear/memory_forget, destructive operations require confirm:true).",
+                    }, null, 2);
+                }
+                // BACKUP_ALWAYS_FIRST: same JSON shape as memory_export, so
+                // memory_import can restore from it independently if anything
+                // below fails partway through.
+                const fs = await import("node:fs");
+                const dbDirEnd = state.config.dbPath.lastIndexOf("/");
+                const backupDir = (dbDirEnd > 0 ? state.config.dbPath.slice(0, dbDirEnd) : ".") + "/backups";
+                await fs.promises.mkdir(backupDir, { recursive: true }).catch(() => { });
+                const backupPath = `${backupDir}/reembed-repair-${Date.now()}.json`;
+                await fs.promises.writeFile(backupPath, JSON.stringify({
+                    format: "opencode-memory-pro/backup",
+                    version: 1,
+                    exportedAt: new Date().toISOString(),
+                    provider: state.config.provider,
+                    dbPath: state.config.dbPath,
+                    reason: "pre-reembed-repair-backup",
+                    fromDim: expectedDim,
+                    toDim: actualDim,
+                    scopes,
+                    count: records.length,
+                    memories: records,
+                }, null, 2));
+                await state.store.connection.dropTable("memories");
+                state.store.table = null;
+                await state.store.init(actualDim);
+                let repaired = 0;
+                let failed = 0;
+                const failures = [];
+                for (const record of records) {
+                    try {
+                        const vector = await state.embedder.embed(record.text || "");
+                        await state.store.put({
+                            ...record,
+                            vector,
+                            vectorDim: vector.length,
+                            embeddingModel: state.embedder.model,
+                        });
+                        repaired += 1;
+                    }
+                    catch (error) {
+                        failed += 1;
+                        failures.push({ id: record.id, reason: error instanceof Error ? error.message : String(error) });
+                    }
+                }
+                await state.store.ensureIndexes();
+                return JSON.stringify({
+                    mismatch: true,
+                    repaired,
+                    failed,
+                    failures: failures.slice(0, 10),
+                    fromDim: expectedDim,
+                    toDim: actualDim,
+                    backupPath,
+                    scopes,
+                    message: failed > 0
+                        ? `Repaired ${repaired}/${records.length}. ${failed} failed but remain intact in the ` +
+                            `backup at ${backupPath} — re-run memory_reembed once the embedder issue is fixed.`
+                        : `Repaired all ${repaired} memories at ${actualDim}-dim. Backup retained at ${backupPath}.`,
+                }, null, 2);
+            },
+        }),
         memory_summarize: tool({
             description: "Create digests of old memories (store-level summarization). LLM abstractive digests when capture.mode=llm, offline extractive otherwise. Optionally mark originals 'digested' (replace=true) so only the digest remains in recall.",
             args: {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "opencode-memory-pro",
-  "version": "1.4.0",
+  "version": "1.4.1",
   "description": "LanceDB-backed long-term memory provider for OpenCode — standalone fork of lancedb-opencode-pro with entity graph, lifecycle, and retention",
   "type": "module",
   "main": "dist/index.js",