@velvetmonkey/flywheel-memory 2.0.75 → 2.0.77

package/README.md

@@ -12,13 +12,13 @@
 [![Scale](https://img.shields.io/badge/scale-100k--line%20files%20%7C%202.5k%20entities-brightgreen.svg)](https://github.com/velvetmonkey/flywheel-memory/blob/main/docs/TESTING.md#performance-benchmarks)
 [![Tests](https://img.shields.io/badge/tests-2,456%20passed-brightgreen.svg)](https://github.com/velvetmonkey/flywheel-memory/blob/main/docs/TESTING.md)
 
-| | Without Flywheel | With Flywheel |
+| | Grep approach | Flywheel |
 |---|---|---|
-| "What's overdue?" | Read every file | Indexed query, <10ms |
-| "What links here?" | grep every file | Pre-indexed backlink graph |
-| "Add a meeting note" | Raw write, no linking | Write + auto-wikilink |
-| "What should I link?" | Manual or grep | Smart scoring + semantic understanding |
-| Token cost | 2,000-250,000 | 50-200 |
+| "What's overdue?" | Grep + read matches (~500-2,000 tokens) | Indexed metadata query (~50-200 tokens) |
+| "What links here?" | Grep for note name (flat list, no graph) | Pre-indexed backlink graph (<10ms) |
+| "Add a meeting note" | Raw write, no linking | Structured write + auto-wikilink |
+| "What should I link?" | Not possible | 10-dimension scoring + semantic search |
+| Hubs, orphans, paths? | Not possible | Pre-indexed graph analysis |
 
 51 tools across 17 categories. 6-line config. Zero cloud dependencies.
 
@@ -35,8 +35,6 @@ Then ask: *"How much have I billed Acme Corp?"*
 
 ## See It Work
 
-![Flywheel demo](https://raw.githubusercontent.com/velvetmonkey/flywheel-memory/main/demos/flywheel-demo.gif)
-
 ### Read: "How much have I billed Acme Corp?"
 
 From the [carter-strategy](https://github.com/velvetmonkey/flywheel-memory/tree/main/demos/carter-strategy/) demo -- a solo consultant with 3 clients, 5 projects, and $27K in invoices.
@@ -44,49 +42,47 @@ From the [carter-strategy](https://github.com/velvetmonkey/flywheel-memory/tree/
 ```
 ❯ How much have I billed Acme Corp?
 
-● flywheel › get_backlinks
-  path: "clients/Acme Corp.md"
-  ← invoices/INV-2025-047.md
-    invoices/INV-2025-048.md
-    projects/Acme Data Migration.md
-    proposals/Acme Analytics Add-on.md
+● flywheel › search
+  query: "Acme Corp"
+  → clients/Acme Corp.md, invoices/INV-2025-047.md,
+    invoices/INV-2025-048.md, projects/Acme Data Migration.md
+
+● read_file("clients/Acme Corp.md")
+  ← total_billed: 156000, rate: $300/hr, status: active
 
-● flywheel › get_note_metadata
-  path: "invoices/INV-2025-047.md"
+● read_file("invoices/INV-2025-047.md")
   ← amount: 15000, status: paid, period: November 2025
 
-● flywheel › get_note_metadata
-  path: "invoices/INV-2025-048.md"
+● read_file("invoices/INV-2025-048.md")
   ← amount: 12000, status: pending, period: December 2025
 
 ┌─ RESULT ──────────────────────────────────────────────┐
-│ Acme Corp Billing: $27,000                            │
+│ Acme Corp: $156K total billed                         │
 │                                                       │
 │   Paid:    $15,000 — Acme Data Migration (Nov 2025)   │
 │   Pending: $12,000 — Acme Data Migration (Dec 2025)   │
 │                                                       │
-│ 4 tool calls · ~160 tokens · 0 files read             │
+│ Also: $35K pending proposal (Analytics Add-on)        │
 └───────────────────────────────────────────────────────┘
 ```
 
-Claude didn't read any files. It navigated the graph: backlinks to find related notes, metadata to extract the numbers.
+Flywheel's indexed search found all Acme-related notes in one call. Claude read the files it needed for billing details. No grepping, no guessing paths.
 
-```
-Same 3 queries without Flywheel: 11,150 tokens (reading files repeatedly)
-Same 3 queries with Flywheel:       300 tokens (querying the index)
-                                     37x savings
-```
+Flywheel's search found all related notes in one call. Without it, Claude would grep for "Acme" and scan every matching file.
+
+The bigger difference isn't just tokens — it's that Flywheel answers structural questions (backlinks, hubs, shortest paths, schema analysis) that file-level access can't answer at all.
 
 ### Write: Auto-wikilinks on every mutation
 
 ```
-❯ Log that I finished the Acme strategy deck
+❯ Log that Stacy Thompson reviewed the API Security Checklist for Acme before the Beta Corp Dashboard kickoff
 
 ● flywheel › vault_add_to_section
   path: "daily-notes/2026-01-04.md"
   section: "Log"
-  content: "finished the [[Acme Corp]] strategy deck"
-                       ↑ auto-linked because Acme Corp.md exists
+  content: "[[Stacy Thompson]] reviewed the [[API Security Checklist]] for [[Acme Corp|Acme]] before the [[Beta Corp Dashboard]] kickoff → [[GlobalBank API Audit]], [[Acme Analytics Add-on]], [[Acme Data Migration]]"
+            ↑ 4 entities auto-linked — "Acme" resolved to Acme Corp via alias
+            → 3 contextual suggestions appended (scored ≥12 via co-occurrence with linked entities)
 ```
 
 Try it yourself: `cd demos/carter-strategy && claude`
@@ -184,9 +180,18 @@ Every query leverages hundreds of accumulated connections. New content auto-link
 
 ### What This Looks Like
 
+```mermaid
+graph LR
+    W[Write] --> A[Auto-link]
+    A --> D[Denser Graph]
+    D --> B[Better Queries]
+    B --> M[More Use]
+    M --> W
+```
+
 ```
-Input:  "Met with Sarah about the data migration"
-Output: "Met with [[Sarah Mitchell]] about the [[Acme Data Migration]]"
+Input:  "Stacy Thompson finished reviewing the API Security Checklist for the Beta Corp Dashboard"
+Output: "[[Stacy Thompson]] finished reviewing the [[API Security Checklist]] for the [[Beta Corp Dashboard]]"
 ```
 
 No manual linking. No broken references. Use compounds into structure, structure compounds into intelligence.
@@ -236,7 +241,7 @@ Measured against a 96-note/61-entity ground truth vault.
 
 - **50-generation stress test** — suggest → accept/reject (85% correct, 15% noise) → mutate vault → rebuild index → repeat. F1 holds steady — the feedback loop doesn't degrade under realistic noise.
 - **7 vault archetypes** — hub-and-spoke, hierarchical, dense-mesh, sparse-orphan, bridge-network, small-world, chaos
-- **13 scoring layers** individually ablated, contribution measured
+- **13 pipeline stages** (10 scoring dimensions + filters + suppression) individually ablated, contribution measured
 - **Regression gate** — CI fails if any mode's F1/precision/recall drops >5pp from baseline
 
 See [docs/TESTING.md](https://github.com/velvetmonkey/flywheel-memory/blob/main/docs/TESTING.md) for full methodology. Auto-generated report: [docs/QUALITY_REPORT.md](https://github.com/velvetmonkey/flywheel-memory/blob/main/docs/QUALITY_REPORT.md).

package/dist/index.js

@@ -1886,24 +1886,42 @@ async function initEmbeddings() {
   if (pipeline) return;
   if (initPromise) return initPromise;
   initPromise = (async () => {
-    try {
-      const transformers = await Function("specifier", "return import(specifier)")("@huggingface/transformers");
-      pipeline = await transformers.pipeline("feature-extraction", activeModelConfig.id, {
-        dtype: "fp32"
-      });
-      if (activeModelConfig.dims === 0) {
-        const probe = await pipeline("test", { pooling: "mean", normalize: true });
-        activeModelConfig.dims = probe.data.length;
-        console.error(`[Semantic] Probed model ${activeModelConfig.id}: ${activeModelConfig.dims} dims`);
-      }
-    } catch (err) {
-      initPromise = null;
-      if (err instanceof Error && (err.message.includes("Cannot find package") || err.message.includes("MODULE_NOT_FOUND") || err.message.includes("Cannot find module") || err.message.includes("ERR_MODULE_NOT_FOUND"))) {
-        throw new Error(
-          "Semantic search requires @huggingface/transformers. Install it with: npm install @huggingface/transformers"
-        );
+    const MAX_RETRIES = 3;
+    const RETRY_DELAYS = [2e3, 5e3, 1e4];
+    for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) {
+      try {
+        const transformers = await Function("specifier", "return import(specifier)")("@huggingface/transformers");
+        console.error(`[Semantic] Loading model ${activeModelConfig.id} (~23MB, cached after first download)...`);
+        pipeline = await transformers.pipeline("feature-extraction", activeModelConfig.id, {
+          dtype: "fp32"
+        });
+        console.error(`[Semantic] Model loaded successfully`);
+        if (activeModelConfig.dims === 0) {
+          const probe = await pipeline("test", { pooling: "mean", normalize: true });
+          activeModelConfig.dims = probe.data.length;
+          console.error(`[Semantic] Probed model ${activeModelConfig.id}: ${activeModelConfig.dims} dims`);
+        }
+        return;
+      } catch (err) {
+        if (err instanceof Error && (err.message.includes("Cannot find package") || err.message.includes("MODULE_NOT_FOUND") || err.message.includes("Cannot find module") || err.message.includes("ERR_MODULE_NOT_FOUND"))) {
+          initPromise = null;
+          throw new Error(
+            "Semantic search requires @huggingface/transformers. Install it with: npm install @huggingface/transformers"
+          );
+        }
+        if (attempt < MAX_RETRIES) {
+          const delay = RETRY_DELAYS[attempt - 1];
+          console.error(`[Semantic] Model load failed (attempt ${attempt}/${MAX_RETRIES}): ${err instanceof Error ? err.message : err}`);
+          console.error(`[Semantic] Retrying in ${delay / 1e3}s...`);
+          await new Promise((resolve2) => setTimeout(resolve2, delay));
+          pipeline = null;
+        } else {
+          console.error(`[Semantic] Model load failed after ${MAX_RETRIES} attempts: ${err instanceof Error ? err.message : err}`);
+          console.error(`[Semantic] Semantic search disabled. Keyword search (BM25) remains available.`);
+          initPromise = null;
+          throw err;
+        }
       }
-      throw err;
     }
   })();
   return initPromise;
@@ -2552,7 +2570,7 @@ function deserializeVaultIndex(data) {
     builtAt: new Date(data.builtAt)
   };
 }
-function loadVaultIndexFromCache(stateDb2, scannedFileCount, maxAgeMs = 24 * 60 * 60 * 1e3, tolerancePercent = 5) {
+function loadVaultIndexFromCache(stateDb2, scannedFileCount, maxAgeMs = 24 * 60 * 60 * 1e3, tolerancePercent = 5, newestFileMtime) {
   const info = getVaultIndexCacheInfo(stateDb2);
   if (!info) {
     console.error("[Flywheel] No cached index found");
@@ -2569,6 +2587,10 @@ function loadVaultIndexFromCache(stateDb2, scannedFileCount, maxAgeMs = 24 * 60
     console.error(`[Flywheel] Cache invalid: too old (${Math.round(age / 1e3 / 60)} minutes)`);
     return null;
   }
+  if (newestFileMtime && newestFileMtime.getTime() > info.builtAt.getTime()) {
+    console.error(`[Flywheel] Cache invalid: files modified since cache was built (newest file: ${newestFileMtime.toISOString()}, cache built: ${info.builtAt.toISOString()})`);
+    return null;
+  }
   const data = loadVaultIndexCache(stateDb2);
   if (!data) {
     console.error("[Flywheel] Failed to load cached index data");
@@ -6126,7 +6148,6 @@ function sortEntitiesByPriority(entities, notePath) {
 }
 function processWikilinks(content, notePath, existingContent) {
   if (!isEntityIndexReady() || !entityIndex) {
-    console.error("[Flywheel:DEBUG] Entity index not ready, entities:", entityIndex?._metadata?.total_entities ?? 0);
     return {
       content,
       linksAdded: 0,
@@ -6134,7 +6155,6 @@ function processWikilinks(content, notePath, existingContent) {
     };
   }
   let entities = getAllEntities(entityIndex);
-  console.error(`[Flywheel:DEBUG] Processing wikilinks with ${entities.length} entities`);
   if (moduleStateDb5) {
     const folder = notePath ? notePath.split("/")[0] : void 0;
     entities = entities.filter((e) => {
@@ -6239,39 +6259,6 @@ function maybeApplyWikilinks(content, skipWikilinks, notePath, existingContent)
   }
   return { content: result.content };
 }
-function getEntityIndexStats() {
-  if (!indexReady || !entityIndex) {
-    return {
-      ready: false,
-      totalEntities: 0,
-      categories: {},
-      error: indexError2?.message
-    };
-  }
-  return {
-    ready: true,
-    totalEntities: entityIndex._metadata.total_entities,
-    categories: {
-      technologies: entityIndex.technologies.length,
-      acronyms: entityIndex.acronyms.length,
-      people: entityIndex.people.length,
-      projects: entityIndex.projects.length,
-      organizations: entityIndex.organizations?.length ?? 0,
-      locations: entityIndex.locations?.length ?? 0,
-      concepts: entityIndex.concepts?.length ?? 0,
-      animals: entityIndex.animals?.length ?? 0,
-      media: entityIndex.media?.length ?? 0,
-      events: entityIndex.events?.length ?? 0,
-      documents: entityIndex.documents?.length ?? 0,
-      vehicles: entityIndex.vehicles?.length ?? 0,
-      health: entityIndex.health?.length ?? 0,
-      finance: entityIndex.finance?.length ?? 0,
-      food: entityIndex.food?.length ?? 0,
-      hobbies: entityIndex.hobbies?.length ?? 0,
-      other: entityIndex.other.length
-    }
-  };
-}
 var SUGGESTION_PATTERN = /→\s*\[\[.+$/;
 var GENERIC_WORDS = /* @__PURE__ */ new Set([
   // Common nouns that appear everywhere and rarely mean anything specific
@@ -13540,12 +13527,6 @@ Example: vault_add_to_section({ path: "daily/2026-02-15.md", section: "Log", con
               trackWikilinkApplications(stateDb2, notePath, linkedEntities);
             }
           }
-          const _debug = {
-            entityCount: getEntityIndexStats().totalEntities,
-            indexReady: getEntityIndexStats().ready,
-            skipWikilinks,
-            wikilinkInfo: wikilinkInfo || "none"
-          };
           let suggestInfo;
           if (suggestOutgoingLinks && !skipWikilinks && processedContent.length >= 100) {
             const result = await suggestRelatedLinks(processedContent, { maxSuggestions, notePath });
@@ -13570,8 +13551,6 @@ Example: vault_add_to_section({ path: "daily/2026-02-15.md", section: "Log", con
             updatedContent,
             message: `Added content to section "${ctx.sectionBoundary.name}" in ${notePath}${createdInfo}`,
             preview,
-            _debug,
-            // Temporary debug field for production troubleshooting
             warnings: validationResult.inputWarnings.length > 0 ? validationResult.inputWarnings : void 0,
             outputIssues: validationResult.outputIssues.length > 0 ? validationResult.outputIssues : void 0,
             normalizationChanges: validationResult.normalizationChanges.length > 0 ? validationResult.normalizationChanges : void 0
@@ -19712,7 +19691,8 @@ async function main() {
       const files = await scanVault(vaultPath);
       const noteCount = files.length;
       serverLog("index", `Found ${noteCount} markdown files`);
-      cachedIndex = loadVaultIndexFromCache(stateDb, noteCount);
+      const newestMtime = files.reduce((max, f) => f.modified > max ? f.modified : max, /* @__PURE__ */ new Date(0));
+      cachedIndex = loadVaultIndexFromCache(stateDb, noteCount, void 0, void 0, newestMtime);
     } catch (err) {
       serverLog("index", `Cache check failed: ${err instanceof Error ? err.message : err}`, "warn");
     }
@@ -19913,30 +19893,45 @@ async function runPostIndexWork(index) {
     if (hasEmbeddingsIndex() && !modelChanged) {
       serverLog("semantic", "Embeddings already built, skipping full scan");
     } else {
-      setEmbeddingsBuilding(true);
-      buildEmbeddingsIndex(vaultPath, (p) => {
-        if (p.current % 100 === 0 || p.current === p.total) {
-          serverLog("semantic", `Embedding ${p.current}/${p.total} notes...`);
-        }
-      }).then(async () => {
-        if (stateDb) {
-          const entities = getAllEntitiesFromDb3(stateDb);
-          if (entities.length > 0) {
-            const entityMap = new Map(entities.map((e) => [e.name, {
-              name: e.name,
-              path: e.path,
-              category: e.category,
-              aliases: e.aliases
-            }]));
-            await buildEntityEmbeddingsIndex(vaultPath, entityMap);
+      const MAX_BUILD_RETRIES = 2;
+      const attemptBuild = async (attempt) => {
+        setEmbeddingsBuilding(true);
+        try {
+          await buildEmbeddingsIndex(vaultPath, (p) => {
+            if (p.current % 100 === 0 || p.current === p.total) {
+              serverLog("semantic", `Embedding ${p.current}/${p.total} notes...`);
+            }
+          });
+          if (stateDb) {
+            const entities = getAllEntitiesFromDb3(stateDb);
+            if (entities.length > 0) {
+              const entityMap = new Map(entities.map((e) => [e.name, {
+                name: e.name,
+                path: e.path,
+                category: e.category,
+                aliases: e.aliases
+              }]));
+              await buildEntityEmbeddingsIndex(vaultPath, entityMap);
+            }
           }
+          loadEntityEmbeddingsToMemory();
+          setEmbeddingsBuildState("complete");
+          serverLog("semantic", "Embeddings ready \u2014 searches now use hybrid ranking");
+        } catch (err) {
+          const msg = err instanceof Error ? err.message : String(err);
+          if (attempt < MAX_BUILD_RETRIES) {
+            const delay = 1e4;
+            serverLog("semantic", `Build failed (attempt ${attempt}/${MAX_BUILD_RETRIES}): ${msg}. Retrying in ${delay / 1e3}s...`, "error");
+            await new Promise((resolve2) => setTimeout(resolve2, delay));
+            return attemptBuild(attempt + 1);
+          }
+          serverLog("semantic", `Embeddings build failed after ${MAX_BUILD_RETRIES} attempts: ${msg}`, "error");
+          serverLog("semantic", "Keyword search (BM25) remains fully available", "error");
+        } finally {
+          setEmbeddingsBuilding(false);
         }
-        loadEntityEmbeddingsToMemory();
-        setEmbeddingsBuildState("complete");
-        serverLog("semantic", "Embeddings ready");
-      }).catch((err) => {
-        serverLog("semantic", `Embeddings build failed: ${err instanceof Error ? err.message : err}`, "error");
-      });
+      };
+      attemptBuild(1);
     }
   } else {
     serverLog("semantic", "Skipping \u2014 FLYWHEEL_SKIP_EMBEDDINGS");

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@velvetmonkey/flywheel-memory",
-  "version": "2.0.75",
+  "version": "2.0.77",
   "description": "MCP server that gives Claude full read/write access to your Obsidian vault. Select from 51 tools for search, backlinks, graph queries, mutations, agent memory, and hybrid semantic search.",
   "type": "module",
   "main": "dist/index.js",
@@ -52,7 +52,7 @@
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.25.1",
-    "@velvetmonkey/vault-core": "^2.0.75",
+    "@velvetmonkey/vault-core": "^2.0.77",
     "better-sqlite3": "^11.0.0",
     "chokidar": "^4.0.0",
     "gray-matter": "^4.0.3",