@danielmarbach/mnemonic-mcp 0.22.0 → 0.23.0

package/CHANGELOG.md

@@ -4,6 +4,20 @@ All notable changes to `mnemonic` will be documented in this file.
 
 The format is loosely based on Keep a Changelog and uses semver-style version headings.
 
+## [0.23.0] - 2026-04-17
+
+### Added
+
+- Lexical rescue now ranks candidates by TF-IDF similarity, improving recall for identifier-heavy and jargon queries without affecting semantic ranking.
+- Recall now boosts notes that explain key decisions and concepts when you ask "why"-style questions, using structural signals like role, connections, and format rather than keyword matching.
+- `run-dogfood-packs.mjs --isolated` copies notes into a temporary workspace for reproducible validation runs without polluting the live vault.
+- Rescue candidates no longer appear when `minSimilarity` is set above the default, so explicit quality filters are respected.
+
+### Changed
+
+- Decision and overview notes now surface more reliably for questions like "why are embeddings gitignored" instead of being outranked by incidental mentions.
+- Lexical rescue now correctly activates when no semantic results are found at all.
+
 ## [0.22.0] - 2026-04-13
 
 ### Added

package/README.md

@@ -17,7 +17,7 @@ For the high-level system map, see [`ARCHITECTURE.md`](ARCHITECTURE.md). For rel
 
 ## Stability
 
-mnemonic is at the inception stage. The storage format (frontmatter schema, vault layout, config structure) is still stabilizing and **may change in breaking ways** between releases. Migrations are provided when possible, but treat your vault as something you can afford to rebuild or re-migrate during this period. Keep an eye on the changelog; mnemonic surfaces pending migrations at startup and `list_migrations` shows pending work per vault after each update.
+The storage format is stable with migration support for any future changes. Keep an eye on the changelog; `list_migrations` shows pending work per vault after each update.
 
 **Scale:** Designed for simplicity and portability — not large-scale knowledge bases.
 
@@ -45,7 +45,7 @@ No code changes required — set `EMBED_MODEL=qwen3-embedding:0.6b` in your envi
 
 ## Setup
 
-### Native (Node.js 18+)
+### Native (Node.js 20+)
 
 ```bash
 npm install
@@ -63,6 +63,8 @@ npm run mcp:local
 
 This rebuilds first, then launches `build/index.js`, so MCP clients always point at the latest source.
 
+For reproducible dogfooding of recency and relationship-navigation behavior, prefer the isolated dogfood runner over the live project vault. The isolated runner copies the current `.mnemonic` notes into a temporary workspace, runs the chosen pack there, and deletes the workspace afterward.
+
 ### Docker
 
 ```bash
@@ -308,7 +310,7 @@ Project identity derives from the **git remote URL**, normalized to a stable slu
 
 `recall` with `cwd` searches both vaults. Project notes get a **+0.15 similarity boost** — a soft signal, not a hard filter — so global memories remain accessible while project context floats to the top.
 
-**Hybrid recall** enhances semantic search with lightweight lexical reranking over note projections. When semantic results are weak, a bounded lexical rescue path scans projections for additional candidates, improving exact-match and partial-query recall without changing the storage model or adding new infrastructure. Lexical scores act as tiebreakers — they cannot overcome a large semantic gap but can reorder close candidates.
+**Hybrid recall** enhances semantic search with lightweight lexical reranking over note projections. When semantic results are weak, a bounded lexical rescue path scans projections for additional candidates, improving exact-match and identifier-heavy recall without changing the storage model or adding new infrastructure. **Canonical explanation promotion** boosts notes that explain key decisions and concepts for "why"-style questions, using structural signals like role, connections, and format rather than keyword matching.
 
 Temporal recall is opt-in via `mode: "temporal"`. It keeps semantic selection first, then enriches only the top matches with compact git-backed history so agents can inspect how a note evolved without turning recall into raw log or diff output.
 
@@ -572,6 +574,14 @@ This keeps early ideation reusable as personal/global knowledge while moving con
 
 mnemonic and Beads address complementary concerns. mnemonic is a **knowledge graph**: it stores notes, relationships between them, and lets agents retrieve relevant context through semantic search. [Beads](https://github.com/steveyegge/beads) is a **task and dependency tracker**: it models work items and their dependencies so agents can determine what is ready to execute next. Both tools can coexist in the same workflow — mnemonic stores knowledge and reasoning while Beads manages execution.
 
+**How does mnemonic differ from Memory Bank MCP?**
+
+mnemonic and Memory Bank MCP both provide persistent memory for agents, but differ in hosting and scope. Memory Bank MCP is a **centralized service** — your memory lives in a remote MCP service and is accessed across projects through that single endpoint. mnemonic is **local-first** — your memories live as plain markdown files on your machine: project-scoped notes in `.mnemonic/` within each repo, and personal notes in a global vault under your home directory. There is no always-on server to configure or depend on; the MCP server spawns on demand per session.
+
+**How does mnemonic differ from Basic Memory?**
+
+Both tools are local-first and use markdown, but with different scoping models. [Basic Memory](https://github.com/basicmachines/basicmemory) maintains a **knowledge base per project** that agents can search and update, with optional cloud sync. mnemonic splits memory into **two distinct vaults**: a global personal vault (`~/mnemonic-vault/`) for cross-project knowledge, and a project-scoped vault (`.mnemonic/`) that travels with the repo and is shared via git. This lets you capture early ideas globally before a repo exists, then migrate only project-relevant notes into the shared vault once collaboration begins.
+
 **What are temporary notes?**
 
 mnemonic distinguishes between two lifecycle states. `temporary` notes capture evolving working-state: hypotheses, in-progress plans, experiment results, draft reasoning. `permanent` notes capture durable knowledge: decisions, root cause explanations, architectural guidance, lessons learned. As an investigation progresses, a cluster of temporary notes is typically `consolidate`d into one or more permanent notes, and the scaffolding is discarded. This two-phase lifecycle keeps exploratory thinking from polluting long-term memory while still giving agents a place to reason incrementally before committing to a conclusion.

package/build/index.js

@@ -14,8 +14,8 @@ import { invalidateActiveProjectCache, getOrBuildVaultEmbeddings, getOrBuildVaul
 import { performance } from "perf_hooks";
 import { filterRelationships, mergeRelationshipsFromNotes, normalizeMergePlanSourceIds, resolveEffectiveConsolidationMode, } from "./consolidate.js";
 import { suggestAutoRelationships } from "./auto-relate.js";
-import { computeRecallMetadataBoost, computeHybridScore, selectRecallResults, applyLexicalReranking } from "./recall.js";
-import { shouldTriggerLexicalRescue, computeLexicalScore, LEXICAL_RESCUE_CANDIDATE_LIMIT, LEXICAL_RESCUE_THRESHOLD, LEXICAL_RESCUE_RESULT_LIMIT, } from "./lexical.js";
+import { computeRecallMetadataBoost, computeHybridScore, selectRecallResults, applyLexicalReranking, applyCanonicalExplanationPromotion, } from "./recall.js";
+import { shouldTriggerLexicalRescue, rankDocumentsByTfIdf, LEXICAL_RESCUE_CANDIDATE_LIMIT, LEXICAL_RESCUE_THRESHOLD, LEXICAL_RESCUE_RESULT_LIMIT, } from "./lexical.js";
 import { getRelationshipPreview } from "./relationships.js";
 import { cleanMarkdown } from "./markdown.js";
 import { MnemonicConfigStore, readVaultSchemaVersion } from "./config.js";
@@ -1757,9 +1757,25 @@ server.registerTool("get_project_memory_policy", {
     };
 });
 // ── Lexical rescue helper ─────────────────────────────────────────────────────
-async function collectLexicalRescueCandidates(vaults, query, project, scope, tags, existingIds) {
+function buildRecallCandidateContext(note) {
+    const metadata = getEffectiveMetadata(note);
+    const relatedCount = note.relatedTo?.length ?? 0;
+    return {
+        metadata,
+        metadataBoost: computeRecallMetadataBoost(metadata),
+        lifecycle: note.lifecycle,
+        relatedCount,
+        connectionDiversity: new Set((note.relatedTo ?? []).map((rel) => rel.type)).size,
+        structureScore: Math.min(0.04, [
+            note.content.includes("## ") ? 0.02 : 0,
+            note.content.includes("- ") || note.content.includes("1. ") ? 0.01 : 0,
+            note.content.length >= 400 ? 0.01 : 0,
+        ].reduce((sum, value) => sum + value, 0)),
+    };
+}
+async function collectLexicalRescueCandidates(vaults, query, project, scope, tags, lifecycle, existingIds) {
     const existingIdSet = new Set(existingIds.map((c) => c.id));
-    const candidates = [];
+    const rescuePool = [];
     for (const vault of vaults) {
         const notes = await vault.storage.listNotes().catch(() => []);
         for (const note of notes) {
@@ -1770,6 +1786,8 @@ async function collectLexicalRescueCandidates(vaults, query, project, scope, tag
                 if (!tags.every((t) => noteTags.has(t)))
                     continue;
             }
+            if (lifecycle && note.lifecycle !== lifecycle)
+                continue;
             const isProjectNote = note.project !== undefined;
             const isCurrentProject = project && note.project === project.id;
             if (scope === "project" && !isCurrentProject)
@@ -1779,24 +1797,39 @@ async function collectLexicalRescueCandidates(vaults, query, project, scope, tag
             const projection = await getOrBuildProjection(vault.storage, note).catch(() => undefined);
             if (!projection)
                 continue;
-            const lexicalScore = computeLexicalScore(query, projection.projectionText);
-            if (lexicalScore < LEXICAL_RESCUE_THRESHOLD)
-                continue;
-            const metadataBoost = computeRecallMetadataBoost(getEffectiveMetadata(note));
-            const boost = (isCurrentProject ? 0.15 : 0) + metadataBoost;
-            candidates.push({
+            rescuePool.push({
                 id: note.id,
-                score: 0,
-                boosted: boost,
                 vault,
                 isCurrentProject: Boolean(isCurrentProject),
-                lexicalScore,
+                projectionText: projection.projectionText,
+                context: buildRecallCandidateContext(note),
             });
-            if (candidates.length >= LEXICAL_RESCUE_CANDIDATE_LIMIT)
-                break;
         }
-        if (candidates.length >= LEXICAL_RESCUE_CANDIDATE_LIMIT)
-            break;
+    }
+    const rankedRescueIds = new Map(rankDocumentsByTfIdf(query, rescuePool.map((candidate) => ({ id: candidate.id, text: candidate.projectionText })), LEXICAL_RESCUE_CANDIDATE_LIMIT).map((candidate) => [candidate.id, candidate.score]));
+    const candidates = [];
+    for (const candidate of rescuePool) {
+        const tfIdfScore = rankedRescueIds.get(candidate.id);
+        if (tfIdfScore === undefined || tfIdfScore <= 0)
+            continue;
+        const lexicalScore = tfIdfScore;
+        if (lexicalScore < LEXICAL_RESCUE_THRESHOLD)
+            continue;
+        const boost = (candidate.isCurrentProject ? 0.15 : 0) + candidate.context.metadataBoost;
+        candidates.push({
+            id: candidate.id,
+            score: lexicalScore,
+            semanticScoreForPromotion: 0,
+            boosted: boost,
+            vault: candidate.vault,
+            isCurrentProject: candidate.isCurrentProject,
+            lexicalScore,
+            lifecycle: candidate.context.lifecycle,
+            relatedCount: candidate.context.relatedCount,
+            connectionDiversity: candidate.context.connectionDiversity,
+            structureScore: candidate.context.structureScore,
+            metadata: candidate.context.metadata,
+        });
     }
     return candidates
         .sort((a, b) => computeHybridScore(b) - computeHybridScore(a))
@@ -1905,9 +1938,21 @@ server.registerTool("recall", {
                 if (isProjectNote)
                     continue;
             }
-            const metadataBoost = computeRecallMetadataBoost(getEffectiveMetadata(note));
-            const boost = (isCurrentProject ? 0.15 : 0) + metadataBoost;
-            scored.push({ id: rec.id, score: rawScore, boosted: rawScore + boost, vault, isCurrentProject: Boolean(isCurrentProject) });
+            const context = buildRecallCandidateContext(note);
+            const boost = (isCurrentProject ? 0.15 : 0) + context.metadataBoost;
+            scored.push({
+                id: rec.id,
+                score: rawScore,
+                semanticScoreForPromotion: rawScore,
+                boosted: rawScore + boost,
+                vault,
+                isCurrentProject: Boolean(isCurrentProject),
+                lifecycle: context.lifecycle,
+                relatedCount: context.relatedCount,
+                connectionDiversity: context.connectionDiversity,
+                structureScore: context.structureScore,
+                metadata: context.metadata,
+            });
         }
     }
     const projectionTexts = new Map();
@@ -1938,14 +1983,19 @@ server.registerTool("recall", {
         }
         return undefined;
     };
+    const strongestSemanticScore = scored.reduce((max, candidate) => max === undefined ? candidate.score : Math.max(max, candidate.score), undefined);
     const reranked = applyLexicalReranking(scored, query, getProjectionText);
-    // Lexical rescue: when semantic results are weak, scan projections for additional candidates
-    const topScore = reranked.length > 0 ? reranked[0].score : undefined;
-    if (shouldTriggerLexicalRescue(topScore, reranked.length)) {
-        const rescueCandidates = await collectLexicalRescueCandidates(vaults, query, project ?? undefined, scope, tags, reranked);
-        reranked.push(...rescueCandidates);
-    }
-    const top = selectRecallResults(reranked, limit, scope);
+    let promoted = applyCanonicalExplanationPromotion(reranked);
+    // Lexical rescue: when semantic results are weak, scan projections for additional candidates.
+    // Skip rescue when the caller set a strict minSimilarity above the default,
+    // because rescue candidates lack genuine semantic backing.
+    const rescueAllowed = minSimilarity <= DEFAULT_MIN_SIMILARITY;
+    if (rescueAllowed && shouldTriggerLexicalRescue(strongestSemanticScore, scored.length)) {
+        const rescueCandidates = await collectLexicalRescueCandidates(vaults, query, project ?? undefined, scope, tags, lifecycle, promoted);
+        promoted.push(...rescueCandidates);
+        promoted = applyCanonicalExplanationPromotion(promoted);
+    }
+    const top = selectRecallResults(promoted, limit, scope);
     if (top.length === 0) {
         const structuredContent = { action: "recalled", query, scope: scope || "all", results: [] };
         return { content: [{ type: "text", text: "No memories found matching that query." }], structuredContent };
@@ -1992,8 +2042,11 @@ server.registerTool("recall", {
             const formattedRelationships = relationships !== undefined
                 ? `\n\n${formatRelationshipPreview(relationships)}`
                 : "";
+            const provenanceLine = provenance || confidence
+                ? `\n**confidence:** ${confidence ?? "medium"}${provenance?.recentlyChanged ? " | **recently changed**" : ""}`
+                : "";
             // Suppress raw related IDs when enriched preview is shown to avoid duplication
-            sections.push(`${formatNote(note, score, relationships === undefined)}${formattedHistory}${formattedRelationships}`);
+            sections.push(`${formatNote(note, score, relationships === undefined)}${provenanceLine}${formattedHistory}${formattedRelationships}`);
             structuredResults.push({
                 id,
                 title: note.title,