get-claudia 1.55.6 → 1.55.8

package/CHANGELOG.md

@@ -2,6 +2,32 @@
 
 All notable changes to Claudia will be documented in this file.
 
+## 1.55.8 (2026-03-15)
+
+### The Vector Search Fix
+
+v1.55.7 fixed FTS5 recall but broke vector/semantic search for every user. Three raw `sqlite3.connect()` calls skipped loading the `sqlite_vec` extension, and three KNN queries lacked the required `k = ?` constraint for JOINs. Net effect: 0% embedding coverage and silent fallback to keyword matching.
+
+- **`load_sqlite_vec()` helper** -- Extracted the vec0 extension loading logic from `Database._get_connection()` into a standalone public function. Any raw connection that touches vec0 tables calls this one function. Eliminates the entire class of "forgot to load extension" bugs.
+- **Backfill worker fix** -- `_backfill_worker()` now loads sqlite_vec on its raw connection. Previously it crashed within 60ms of starting because it couldn't query or write to vec0 tables. Degrades gracefully if the extension isn't available.
+- **Index repair fix** -- `_check_and_repair_indexes()` now loads sqlite_vec on its raw connection. Previously it always reported 0 embeddings (triggering unnecessary backfill every startup). Improved exception handling distinguishes "no such table" from "no such module."
+- **KNN `k = ?` constraint** -- Added `AND k = ?` to all three `embedding MATCH` queries in recall.py (`recall()`, `recall_episodes()`, `search_reflections()`). vec0's KNN queries require this constraint when JOINs are present because SQLite's query planner can't push an outer `LIMIT` into the virtual table scan.
+- **Smarter briefing message** -- Embedding health check now reads `_meta['indexes_repaired']` to distinguish "backfill in progress" from "backfill never started." No longer tells users to "Start Ollama" when the real problem was a code bug.
+- **9 new tests** -- Helper extraction, Database integration, backfill/repair patterns, KNN constraint behavior (both success and documented failure without `k`). All 615 tests pass.
+
+## 1.55.7 (2026-03-15)
+
+### The Recall Recovery Release
+
+After v1.55 consolidation, recall could return zero results despite all memories existing. Three-layer fix restores recall for affected users automatically on next startup.
+
+- **FTS5 rebuild after consolidation** -- `merge_all_databases()` now rebuilds the FTS5 full-text index after merging. The migration's separate SQLite connection bypassed triggers that keep FTS5 in sync, leaving the index empty.
+- **LIKE fallback fix** -- `_keyword_search()` now falls through to SQL LIKE when FTS5 returns 0 rows (not just on exception). Previously, an empty-but-functional FTS5 table returned nothing and the LIKE fallback never activated.
+- **Startup index repair** -- New `_check_and_repair_indexes()` runs on every daemon startup (idempotent). Detects FTS5 and embedding gaps, rebuilds FTS5 instantly, and starts a background embedding backfill thread if Ollama is available.
+- **Background embedding backfill** -- Non-blocking thread generates missing vector embeddings in batches of 25. Tolerant of missing Ollama (logs warning, recall uses LIKE fallback). Progress logged to daemon.log.
+- **Embedding health in briefing** -- Session briefing now shows embedding coverage percentage when below 90%, so Claudia can inform the user about regeneration status.
+- **5 new tests** -- FTS rebuild after merge, recall works after merge, standalone FTS rebuild, LIKE fallback when FTS empty, FTS MATCH preferred when populated. All 613 tests pass.
+
 ## 1.55.6 (2026-03-15)
 
 - **Post-consolidation status report** -- After merging databases, Claudia's first greeting includes live database stats (memories, entities, relationships, episodes, reflections, patterns) and explains the backup schedule going forward. The whats-new file now contains a full status table and backup retention details.

package/memory-daemon/claudia_memory/__main__.py

@@ -20,7 +20,7 @@ from pathlib import Path
 from .config import get_config, set_project_id
 from .daemon.health import start_health_server, stop_health_server
 from .daemon.scheduler import start_scheduler, stop_scheduler
-from .database import get_db
+from .database import get_db, load_sqlite_vec
 from .mcp.server import run_server as run_mcp_server
 
 logger = logging.getLogger(__name__)
@@ -367,6 +367,10 @@ Your database is automatically backed up on a schedule (requires the standalone
 
 All backups live in `~/.claudia/backups/`. Old backups are automatically cleaned up when they exceed retention limits.
 
+## Search index rebuild
+
+Your full-text search index has been rebuilt automatically. If Ollama is running, vector embeddings are being regenerated in the background (this takes a few minutes for large databases). Until embeddings finish, recall uses keyword search as a fallback.
+
 **What this means for the user:**
 - You remember everything from every project, always. No more fragmented memory.
 - Moving project directories no longer loses history.
@@ -381,6 +385,192 @@ _Surface this update in your first greeting with the database stats and backup e
         logger.debug(f"Could not write consolidation notice: {e}")
 
 
+def _check_and_repair_indexes(db_path: Path) -> None:
+    """Detect and repair empty FTS5 / embedding indexes after consolidation.
+
+    Runs on every daemon startup (idempotent). Handles three populations:
+    - Already broken (upgraded to v1.55.0-1.55.6): FTS empty, embeddings empty
+    - Fresh consolidation (upgrading now): FTS rebuilt in merge, embeddings backfilled here
+    - Normal startup (no issues): counts match, no-op
+
+    Stores '_meta["indexes_repaired"]' with timestamp when repairs happen.
+    """
+    from datetime import datetime as dt
+    from .migration import rebuild_fts_index
+
+    try:
+        conn = sqlite3.connect(str(db_path), timeout=10)
+        conn.row_factory = sqlite3.Row
+        load_sqlite_vec(conn)
+
+        # Count memories
+        mem_row = conn.execute("SELECT COUNT(*) as c FROM memories WHERE invalidated_at IS NULL").fetchone()
+        mem_count = mem_row["c"] if mem_row else 0
+
+        if mem_count == 0:
+            conn.close()
+            return  # No memories, nothing to repair
+
+        # Check FTS5 index
+        fts_count = 0
+        try:
+            fts_row = conn.execute("SELECT COUNT(*) as c FROM memories_fts").fetchone()
+            fts_count = fts_row["c"] if fts_row else 0
+        except Exception:
+            pass  # FTS5 table might not exist
+
+        # Check embeddings
+        emb_count = 0
+        try:
+            emb_row = conn.execute("SELECT COUNT(*) as c FROM memory_embeddings").fetchone()
+            emb_count = emb_row["c"] if emb_row else 0
+        except sqlite3.OperationalError as e:
+            if "no such table" in str(e):
+                pass  # Fresh install, vec0 tables not yet created
+            elif "no such module" in str(e):
+                logger.debug("sqlite_vec not loaded, cannot count embeddings")
+            else:
+                logger.warning(f"Unexpected error counting embeddings: {e}")
+
+        conn.close()
+
+        fts_gap = mem_count - fts_count
+        emb_gap = mem_count - emb_count
+        fts_threshold = max(10, int(mem_count * 0.1))  # 10% or at least 10
+        emb_threshold = max(10, int(mem_count * 0.1))
+
+        repaired = []
+
+        # Repair FTS5 if significantly fewer entries than memories
+        if fts_gap > fts_threshold:
+            logger.warning(
+                f"FTS5 index gap detected: {fts_count} indexed vs {mem_count} memories. "
+                f"Rebuilding FTS5 index..."
+            )
+            indexed = rebuild_fts_index(db_path)
+            repaired.append(f"fts5: {fts_count}->{indexed}")
+            logger.info(f"FTS5 repair complete: {indexed} rows indexed")
+
+        # Trigger embedding backfill if significantly fewer embeddings
+        if emb_gap > emb_threshold:
+            logger.warning(
+                f"Embedding gap detected: {emb_count} embeddings vs {mem_count} memories. "
+                f"Starting background backfill..."
+            )
+            _auto_backfill_embeddings(db_path, mem_count, emb_count)
+            repaired.append(f"embeddings: {emb_count}/{mem_count} (backfill started)")
+
+        # Record repair timestamp
+        if repaired:
+            try:
+                rc = sqlite3.connect(str(db_path), timeout=10)
+                rc.execute(
+                    "INSERT OR REPLACE INTO _meta (key, value, updated_at) "
+                    "VALUES ('indexes_repaired', ?, ?)",
+                    (", ".join(repaired), dt.now().isoformat()),
+                )
+                rc.commit()
+                rc.close()
+            except Exception:
+                pass
+        else:
+            logger.debug(
+                f"Index health OK: FTS5={fts_count}/{mem_count}, "
+                f"embeddings={emb_count}/{mem_count}"
+            )
+
+    except Exception as e:
+        # Non-fatal: log and continue
+        logger.error(f"Index repair check failed (non-fatal): {e}")
+
+
+def _auto_backfill_embeddings(db_path: Path, mem_count: int, emb_count: int) -> None:
+    """Start background thread to generate missing embeddings.
+
+    Non-blocking: the MCP server starts immediately while this runs.
+    Tolerant: if Ollama isn't running, logs a warning and exits.
+    Batched: processes 25 at a time with progress logging.
+    Idempotent: LEFT JOIN ensures only missing embeddings are generated.
+    """
+    import json as _json
+    import threading
+
+    def _backfill_worker():
+        try:
+            from .embeddings import get_embedding_service
+
+            svc = get_embedding_service()
+            if not svc.is_available_sync():
+                logger.warning(
+                    "Ollama not available for embedding backfill. "
+                    "Recall will use LIKE fallback until embeddings are generated. "
+                    "Start Ollama and restart the daemon, or run --backfill-embeddings."
+                )
+                return
+
+            conn = sqlite3.connect(str(db_path), timeout=30)
+            conn.row_factory = sqlite3.Row
+            if not load_sqlite_vec(conn):
+                logger.warning(
+                    "sqlite_vec not available in backfill thread. "
+                    "Cannot write to vec0 tables. Skipping embedding backfill."
+                )
+                conn.close()
+                return
+
+            # Find memories missing embeddings
+            missing = conn.execute(
+                "SELECT m.id, m.content FROM memories m "
+                "LEFT JOIN memory_embeddings me ON m.id = me.memory_id "
+                "WHERE me.memory_id IS NULL AND m.invalidated_at IS NULL"
+            ).fetchall()
+
+            if not missing:
+                logger.info("Embedding backfill: no missing embeddings found")
+                conn.close()
+                return
+
+            total = len(missing)
+            logger.info(f"Embedding backfill: generating embeddings for {total} memories...")
+
+            success = 0
+            failed = 0
+            batch_size = 25
+
+            for i, row in enumerate(missing, 1):
+                try:
+                    embedding = svc.embed_sync(row["content"])
+                    if embedding:
+                        conn.execute(
+                            "INSERT OR REPLACE INTO memory_embeddings (memory_id, embedding) VALUES (?, ?)",
+                            (row["id"], _json.dumps(embedding)),
+                        )
+                        success += 1
+                        if success % batch_size == 0:
+                            conn.commit()
+                    else:
+                        failed += 1
+                except Exception as e:
+                    failed += 1
+                    if failed <= 3:
+                        logger.debug(f"Embedding failed for memory {row['id']}: {e}")
+
+                if i % batch_size == 0 or i == total:
+                    logger.info(f"Embedding backfill progress: {i}/{total} (success={success}, failed={failed})")
+
+            conn.commit()
+            conn.close()
+
+            logger.info(f"Embedding backfill complete: {success} generated, {failed} failed out of {total}")
+
+        except Exception as e:
+            logger.error(f"Embedding backfill thread failed: {e}")
+
+    thread = threading.Thread(target=_backfill_worker, name="embedding-backfill", daemon=True)
+    thread.start()
+    logger.info("Embedding backfill thread started in background")
+
+
 def _write_preflight_result(result: dict) -> Path:
     """Write preflight result JSON to ~/.claudia/daemon-preflight.json."""
     out_path = Path.home() / ".claudia" / "daemon-preflight.json"
@@ -790,6 +980,10 @@ def run_daemon(mcp_mode: bool = True, debug: bool = False, project_id: str = Non
         # Auto-consolidate hash-named databases into unified claudia.db
         _auto_consolidate()
 
+        # Repair FTS5 and embeddings if they're out of sync with memories.
+        # Handles already-affected users (v1.55.0-1.55.6) and fresh consolidations.
+        _check_and_repair_indexes(Path(config.db_path))
+
         # Start health server and scheduler - ONLY in standalone mode.
         # MCP server processes are ephemeral and session-bound; the standalone
         # daemon (LaunchAgent/systemd) owns port 3848 and handles scheduling.

package/memory-daemon/claudia_memory/database.py

@@ -25,6 +25,73 @@ from .config import get_config
 logger = logging.getLogger(__name__)
 
 
+def load_sqlite_vec(conn: sqlite3.Connection) -> bool:
+    """Load the sqlite-vec extension on a connection.
+
+    Tries two methods:
+    1. sqlite_vec Python package (recommended, works everywhere including Python 3.13+)
+    2. Native extension loading (for systems with pre-installed sqlite-vec)
+
+    Returns True if vec0 is available, False otherwise. Never raises.
+    """
+    # Method 1: Try sqlite_vec Python package
+    try:
+        import sqlite_vec
+        if hasattr(conn, "enable_load_extension"):
+            conn.enable_load_extension(True)
+        sqlite_vec.load(conn)
+        if hasattr(conn, "enable_load_extension"):
+            conn.enable_load_extension(False)
+        logger.debug("Loaded sqlite-vec via Python package")
+        return True
+    except ImportError:
+        logger.debug("sqlite_vec package not installed")
+    except Exception as e:
+        logger.warning(f"sqlite_vec package installed but load() failed: {e}")
+
+    # Method 2: Try native extension loading
+    try:
+        conn.enable_load_extension(True)
+        sqlite_vec_paths = ["vec0"]  # System-wide
+
+        if sys.platform == "win32":
+            try:
+                import sqlite_vec as _sv
+                pkg_dir = Path(_sv.__file__).parent
+                for dll in pkg_dir.rglob("vec0*"):
+                    if dll.suffix in (".dll", ".so"):
+                        sqlite_vec_paths.append(str(dll.with_suffix("")))
+            except ImportError:
+                pass
+            sqlite_vec_paths.extend([
+                str(Path(sys.executable).parent / "DLLs" / "vec0"),
+                str(Path.home() / ".local" / "lib" / "sqlite-vec" / "vec0"),
+            ])
+        else:
+            sqlite_vec_paths.extend([
+                "/usr/local/lib/sqlite-vec/vec0",
+                "/opt/homebrew/lib/sqlite-vec/vec0",
+                str(Path.home() / ".local" / "lib" / "sqlite-vec" / "vec0"),
+            ])
+
+        for path in sqlite_vec_paths:
+            try:
+                conn.load_extension(path)
+                logger.debug(f"Loaded sqlite-vec from {path}")
+                conn.enable_load_extension(False)
+                return True
+            except sqlite3.OperationalError:
+                continue
+
+        conn.enable_load_extension(False)
+    except AttributeError:
+        logger.debug("enable_load_extension not available (Python 3.13+)")
+    except Exception as e:
+        logger.debug(f"Extension loading failed: {e}")
+
+    return False
+
+
 class Database:
     """Thread-safe SQLite database with sqlite-vec support"""
 
@@ -51,72 +118,8 @@ class Database:
             # Recover any uncommitted WAL writes from a previous crashed daemon
             conn.execute("PRAGMA wal_checkpoint(TRUNCATE)")
 
-            # Try to load sqlite-vec for vector search
-            # Priority: sqlite_vec Python package first (works on Python 3.13+),
-            # then fall back to native extension loading
-            loaded = False
-
-            # Method 1: Try sqlite_vec Python package (recommended, works everywhere)
-            # Python 3.14+ requires explicit enable_load_extension() before any
-            # extension loading, even via the sqlite_vec helper package.
-            try:
-                import sqlite_vec
-                if hasattr(conn, "enable_load_extension"):
-                    conn.enable_load_extension(True)
-                sqlite_vec.load(conn)
-                if hasattr(conn, "enable_load_extension"):
-                    conn.enable_load_extension(False)
-                loaded = True
-                logger.debug("Loaded sqlite-vec via Python package")
-            except ImportError:
-                logger.debug("sqlite_vec package not installed")
-            except Exception as e:
-                logger.warning(f"sqlite_vec package installed but load() failed: {e}")
-
-            # Method 2: Try native extension loading (for systems with pre-installed sqlite-vec)
-            if not loaded:
-                try:
-                    conn.enable_load_extension(True)
-                    sqlite_vec_paths = ["vec0"]  # System-wide
-
-                    if sys.platform == "win32":
-                        # Try to find vec0.dll in the sqlite-vec package directory
-                        try:
-                            import sqlite_vec as _sv
-                            pkg_dir = Path(_sv.__file__).parent
-                            for dll in pkg_dir.rglob("vec0*"):
-                                if dll.suffix in (".dll", ".so"):
-                                    sqlite_vec_paths.append(str(dll.with_suffix("")))
-                        except ImportError:
-                            pass
-                        sqlite_vec_paths.extend([
-                            str(Path(sys.executable).parent / "DLLs" / "vec0"),
-                            str(Path.home() / ".local" / "lib" / "sqlite-vec" / "vec0"),
-                        ])
-                    else:
-                        sqlite_vec_paths.extend([
-                            "/usr/local/lib/sqlite-vec/vec0",
-                            "/opt/homebrew/lib/sqlite-vec/vec0",
-                            str(Path.home() / ".local" / "lib" / "sqlite-vec" / "vec0"),
-                        ])
-
-                    for path in sqlite_vec_paths:
-                        try:
-                            conn.load_extension(path)
-                            loaded = True
-                            logger.debug(f"Loaded sqlite-vec from {path}")
-                            break
-                        except sqlite3.OperationalError:
-                            continue
-
-                    conn.enable_load_extension(False)
-                except AttributeError:
-                    # Python 3.13+ may not have enable_load_extension
-                    logger.debug("enable_load_extension not available (Python 3.13+)")
-                except Exception as e:
-                    logger.debug(f"Extension loading failed: {e}")
-
-            if not loaded:
+            # Load sqlite-vec for vector search
+            if not load_sqlite_vec(conn):
                 if sys.platform == "win32":
                     logger.warning(
                         "sqlite-vec not available. Vector search will be disabled. "

package/memory-daemon/claudia_memory/mcp/server.py

@@ -3336,6 +3336,40 @@ def _build_briefing() -> str:
     except Exception as e:
         logger.debug(f"Briefing recent failed: {e}")
 
+    # 6. Embedding health check
+    try:
+        mem_total = db.execute("SELECT COUNT(*) as c FROM memories WHERE invalidated_at IS NULL", fetch=True)
+        emb_total = db.execute("SELECT COUNT(*) as c FROM memory_embeddings", fetch=True)
+        mem_c = mem_total[0]["c"] if mem_total else 0
+        emb_c = emb_total[0]["c"] if emb_total else 0
+        if mem_c > 0:
+            coverage = (emb_c / mem_c) * 100
+            if coverage < 90:
+                gap = mem_c - emb_c
+                # Check _meta for backfill status to give accurate guidance
+                status_hint = "Start Ollama and restart daemon to generate embeddings."
+                try:
+                    repair_row = db.execute(
+                        "SELECT value FROM _meta WHERE key = 'indexes_repaired'",
+                        fetch=True,
+                    )
+                    if repair_row and repair_row[0]["value"]:
+                        repair_info = repair_row[0]["value"]
+                        if "backfill started" in repair_info:
+                            status_hint = "Backfill was started on this run. Check daemon logs for progress."
+                        elif coverage > 0:
+                            status_hint = "Partial embeddings exist. Restart daemon to trigger backfill for the rest."
+                except Exception:
+                    pass
+                lines.append(
+                    f"**Embedding coverage:** {emb_c}/{mem_c} ({coverage:.0f}%). "
+                    f"{gap} memories lack vector embeddings. "
+                    f"{status_hint} "
+                    f"Recall uses keyword fallback for unembedded memories."
+                )
+    except Exception as e:
+        logger.debug(f"Briefing embedding health failed: {e}")
+
     if len(lines) <= 1:
         lines.append("No context available yet. This appears to be a fresh workspace.")
 

package/memory-daemon/claudia_memory/migration.py

@@ -1255,9 +1255,42 @@ def merge_all_databases(
             logger.error(f"Failed to merge {source_path.name}: {e}")
             # Non-fatal: continue with other sources
 
+    # Rebuild FTS5 index after all merges (triggers don't fire on the
+    # separate connections used by migrate_legacy_database)
+    if not dry_run and totals["sources_merged"] > 0:
+        rebuild_fts_index(target_path)
+
     return totals
 
 
+def rebuild_fts_index(db_path: Path) -> int:
+    """Rebuild FTS5 index from memories table.
+
+    After consolidation, FTS5 triggers may not have fired for migrated rows
+    (the migration uses a separate connection that bypasses triggers).
+    This rebuilds the entire FTS index from scratch.
+
+    Returns the number of rows indexed.
+    """
+    try:
+        conn = sqlite3.connect(str(db_path), timeout=30)
+        # Clear existing FTS data
+        conn.execute("INSERT INTO memories_fts(memories_fts) VALUES('delete-all')")
+        # Repopulate from memories table
+        conn.execute(
+            "INSERT INTO memories_fts(rowid, content) "
+            "SELECT id, content FROM memories WHERE invalidated_at IS NULL"
+        )
+        conn.commit()
+        count = conn.execute("SELECT COUNT(*) FROM memories_fts").fetchone()[0]
+        conn.close()
+        logger.info(f"Rebuilt FTS5 index: {count} rows indexed")
+        return count
+    except Exception as e:
+        logger.warning(f"Could not rebuild FTS5 index: {e}")
+        return 0
+
+
 def cleanup_old_databases(memory_dir: Path, source_dbs: List[Dict]) -> int:
     """Delete hash-named databases and their WAL/SHM files after successful merge.
 

package/memory-daemon/claudia_memory/services/recall.py

@@ -145,9 +145,11 @@ class RecallService:
                 LEFT JOIN memory_entities me2 ON m.id = me2.memory_id
                 LEFT JOIN entities e ON me2.entity_id = e.id
                 WHERE me.embedding MATCH ?
+                AND k = ?
                 """
             )
             params.append(json.dumps(query_embedding))
+            params.append(limit * 2)
 
             self._apply_filters(sql_parts, params, memory_types, min_importance, date_after, date_before, about_entity, include_archived)
             sql_parts.append("GROUP BY m.id ORDER BY vector_score DESC LIMIT ?")
@@ -1368,11 +1370,12 @@ class RecallService:
                     FROM episode_embeddings ee
                     JOIN episodes e ON e.id = ee.episode_id
                     WHERE ee.embedding MATCH ?
+                    AND k = ?
                     AND e.is_summarized = 1
                     ORDER BY relevance DESC
                     LIMIT ?
                     """,
-                    (json.dumps(query_embedding), limit),
+                    (json.dumps(query_embedding), limit, limit),
                     fetch=True,
                 ) or []
             except Exception as e:
@@ -2286,8 +2289,9 @@ class RecallService:
                     JOIN reflections r ON r.id = re.reflection_id
                     LEFT JOIN entities e ON r.about_entity_id = e.id
                     WHERE re.embedding MATCH ?
+                    AND k = ?
                 """
-                params: list = [json.dumps(query_embedding)]
+                params: list = [json.dumps(query_embedding), limit]
 
                 if reflection_types:
                     placeholders = ", ".join(["?" for _ in reflection_types])
@@ -2429,6 +2433,8 @@ class RecallService:
             rows = self.db.execute(sql, tuple(params), fetch=True) or []
             if rows:
                 return rows
+            # FTS5 succeeded but returned 0 rows (empty index after migration).
+            # Fall through to LIKE instead of returning empty.
         except Exception:
             pass  # FTS5 not available, fall through to LIKE
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "get-claudia",
-  "version": "1.55.6",
+  "version": "1.55.8",
   "description": "An AI assistant who learns how you work.",
   "keywords": [
     "claudia",