superlocalmemory 2.4.2 → 2.5.0

package/CHANGELOG.md

@@ -16,6 +16,52 @@ SuperLocalMemory V2 - Intelligent local memory system for AI coding assistants.
 
 ---
 
+## [2.5.0] - 2026-02-12
+
+**Release Type:** Major Feature Release — "Your AI Memory Has a Heartbeat"
+**Backward Compatible:** Yes (additive tables, columns, and modules only)
+
+### The Big Picture
+SuperLocalMemory transforms from passive storage to **active coordination layer**. Every memory operation now triggers real-time events. Agents are tracked. Trust is scored. The dashboard is live.
+
+### Added
+- **DbConnectionManager** (`src/db_connection_manager.py`): SQLite WAL mode, 5s busy timeout, thread-local read pool, serialized write queue. Fixes "database is locked" errors when multiple agents write simultaneously
+- **Event Bus** (`src/event_bus.py`): Real-time event broadcasting — every memory write/update/delete fires events to SSE, WebSocket, and Webhook subscribers. Tiered retention (hot 48h, warm 14d, cold 30d)
+- **Subscription Manager** (`src/subscription_manager.py`): Durable + ephemeral event subscriptions with filters (event type, importance, protocol)
+- **Webhook Dispatcher** (`src/webhook_dispatcher.py`): Background HTTP POST delivery with exponential backoff retry (3 attempts)
+- **Agent Registry** (`src/agent_registry.py`): Tracks which AI agents connect — protocol, write/recall counters, last seen, metadata
+- **Provenance Tracker** (`src/provenance_tracker.py`): Tracks who created each memory — created_by, source_protocol, trust_score, provenance_chain
+- **Trust Scorer** (`src/trust_scorer.py`): Bayesian trust signal collection — positive (cross-agent recall, high importance), negative (quick delete, burst write). Silent collection in v2.5, enforcement in v2.6
+- **SSE endpoint** (`/events/stream`): Real-time Server-Sent Events with cross-process DB polling, Last-Event-ID replay, event type filtering
+- **Dashboard: Live Events tab**: Real-time event stream with color-coded types, filter dropdown, stats counters
+- **Dashboard: Agents tab**: Connected agents table with protocol badges, trust scores, write/recall counters, trust overview
+- **4 new database tables**: `memory_events`, `subscriptions`, `agent_registry`, `trust_signals`
+- **4 new columns on `memories`**: `created_by`, `source_protocol`, `trust_score`, `provenance_chain`
+- **28 API endpoints** across 8 modular route files
+- **Architecture document**: `docs/ARCHITECTURE-V2.5.md`
+
+### Changed
+- **`memory_store_v2.py`**: Replaced 15 direct `sqlite3.connect()` calls with `DbConnectionManager` (read pool + write queue). Added EventBus, Provenance, Trust integration. Fully backward compatible — falls back to direct SQLite if new modules unavailable
+- **`mcp_server.py`**: Replaced 9 per-call `MemoryStoreV2(DB_PATH)` instantiations with shared singleton. Added agent registration and provenance tracking for MCP calls
+- **`ui_server.py`**: Refactored from 2008-line monolith to 194-line app shell + 9 route modules in `routes/` directory using FastAPI APIRouter
+- **`ui/app.js`**: Split from 1588-line monolith to 13 modular files in `ui/js/` directory (largest: 230 lines)
+
+### Fixed
+- **"Database is locked" errors**: WAL mode + write serialization eliminates concurrent access failures
+- **Fresh database crashes**: `/api/stats`, `/api/graph`, `/api/clusters` now return graceful empty responses when `graph_nodes`/`graph_edges`/`sessions` tables don't exist yet
+- **Per-call overhead**: MCP tool handlers no longer rebuild TF-IDF vectorizer on every call
+
+### Testing
+- 63 pytest tests (unit, concurrency, regression, security, edge cases, Docker/new-user scenarios)
+- 27 end-to-end tests (all v2.5 components + 50 concurrent writes + reads during writes)
+- 15 fresh-database edge case tests
+- 17 backward compatibility tests against live v2.4.2 database
+- All CLI commands and skill scripts verified
+- Profile isolation verified across 2 profiles
+- Playwright dashboard testing (all 8 tabs, SSE stream, profile switching)
+
+---
+
 ## [2.4.2] - 2026-02-11
 
 **Release Type:** Bug Fix Release

package/README.md

@@ -39,6 +39,27 @@
 
 ---
 
+## NEW: v2.5 — "Your AI Memory Has a Heartbeat"
+
+> **SuperLocalMemory is no longer passive storage — it's a real-time coordination layer.**
+
+| What's New | Why It Matters |
+|------------|----------------|
+| **Real-Time Event Stream** | See every memory operation live in the dashboard — no refresh needed. SSE-powered, cross-process. |
+| **No More "Database Locked"** | WAL mode + serialized write queue. 50 concurrent agents writing? Zero errors. |
+| **Agent Tracking** | Know exactly which AI tool wrote what. Claude, Cursor, Windsurf, CLI — all tracked automatically. |
+| **Trust Scoring** | Bayesian trust signals detect spam, quick-deletes, and cross-agent validation. Silent in v2.5, enforced in v2.6. |
+| **Memory Provenance** | Every memory records who created it, via which protocol, with full derivation lineage. |
+| **Production-Grade Code** | 28 API endpoints across 8 modular route files. 13 modular JS files. 63 pytest tests. |
+
+**Upgrade:** `npm install -g superlocalmemory@latest`
+
+**Dashboard:** `python3 ~/.claude-memory/ui_server.py` then open `http://localhost:8765`
+
+[Architecture Doc](docs/ARCHITECTURE-V2.5.md) | [Full Changelog](CHANGELOG.md)
+
+---
+
 ## Install in One Command
 
 ```bash

package/docs/ARCHITECTURE-V2.5.md

@@ -0,0 +1,190 @@
+# SuperLocalMemory V2.5 Architecture — "Your AI Memory Has a Heartbeat"
+
+**Version:** 2.5.0 | **Date:** February 12, 2026 | **Author:** Varun Pratap Bhardwaj
+
+---
+
+## What Changed in v2.5
+
+SuperLocalMemory transforms from **passive storage** (filing cabinet) to **active coordination layer** (nervous system). Every memory write, update, delete, or recall now triggers real-time events visible across all connected tools.
+
+### Before v2.5 (Passive)
+```
+Claude writes memory → saved to SQLite → done (silent)
+Cursor reads memory → returned → done (silent)
+```
+
+### After v2.5 (Active)
+```
+Claude writes memory → saved to SQLite → Event Bus fires
+                                           ├── SSE → Dashboard shows it live
+                                           ├── Agent registered in registry
+                                           ├── Trust signal recorded
+                                           ├── Provenance tracked
+                                           └── Webhook dispatched (if configured)
+```
+
+---
+
+## System Architecture
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│                    ACCESS LAYER                              │
+│  MCP Server │ CLI │ REST API │ Skills │ Python Import │ A2A  │
+└──────────────────────┬──────────────────────────────────────┘
+                       │
+┌──────────────────────┴──────────────────────────────────────┐
+│                MEMORY STORE V2                               │
+│  ┌─────────────────────────────────────────────────────────┐    │
+│  │  DbConnectionManager (Singleton)                     │    │
+│  │  ├── WAL Mode (concurrent reads + serialized writes) │    │
+│  │  ├── Busy Timeout (5s wait, not fail)                │    │
+│  │  ├── Thread-Local Read Pool                          │    │
+│  │  ├── Serialized Write Queue (threading.Queue)        │    │
+│  │  └── Post-Write Hooks → Event Bus                    │    │
+│  └─────────────────────────────────────────────────────┘    │
+│                                                              │
+│  On every write:                                             │
+│  ├── EventBus.emit() → memory_events table + SSE + WS + WH  │
+│  ├── ProvenanceTracker → created_by, source_protocol columns │
+│  ├── AgentRegistry → agent tracking + write/recall counters  │
+│  └── TrustScorer → signal collection (silent, no enforcement)│
+└──────────────────────────────────────────────────────────────┘
+                       │
+┌──────────────────────┴──────────────────────────────────────┐
+│                    STORAGE LAYER                             │
+│  SQLite (single file: ~/.claude-memory/memory.db)            │
+│                                                              │
+│  Tables:                                                     │
+│  ├── memories (+ provenance columns: created_by,             │
+│  │            source_protocol, trust_score, provenance_chain) │
+│  ├── memory_events (event log with tiered retention)         │
+│  ├── subscriptions (durable + ephemeral event subscriptions) │
+│  ├── agent_registry (connected agents + statistics)          │
+│  ├── trust_signals (trust signal audit trail)                │
+│  ├── graph_nodes, graph_edges, graph_clusters (knowledge graph)│
+│  ├── identity_patterns (learned preferences)                 │
+│  ├── sessions, creator_metadata                              │
+│  └── memories_fts (FTS5 full-text search index)              │
+└──────────────────────────────────────────────────────────────┘
+                       │
+┌──────────────────────┴──────────────────────────────────────┐
+│                    DASHBOARD LAYER                            │
+│  FastAPI UI Server (modular routes)                          │
+│  ├── 8 route modules in routes/ directory                    │
+│  ├── 28 API endpoints                                        │
+│  ├── SSE /events/stream (real-time, cross-process)           │
+│  ├── WebSocket /ws/updates                                   │
+│  ├── 13 modular JS files in ui/js/                           │
+│  └── 8 dashboard tabs (Graph, Memories, Clusters, Patterns,  │
+│       Timeline, Live Events, Agents, Settings)               │
+└──────────────────────────────────────────────────────────────┘
+```
+
+---
+
+## New Components (v2.5)
+
+| Component | File | Purpose |
+|-----------|------|---------|
+| **DbConnectionManager** | `src/db_connection_manager.py` | WAL mode, busy timeout, read pool, write queue, post-write hooks |
+| **EventBus** | `src/event_bus.py` | Event emission, persistence, in-memory buffer, tiered retention |
+| **SubscriptionManager** | `src/subscription_manager.py` | Durable + ephemeral event subscriptions with filters |
+| **WebhookDispatcher** | `src/webhook_dispatcher.py` | Background HTTP POST delivery with retry logic |
+| **AgentRegistry** | `src/agent_registry.py` | Agent tracking, write/recall counters, protocol detection |
+| **ProvenanceTracker** | `src/provenance_tracker.py` | Memory origin tracking (who, how, trust, lineage) |
+| **TrustScorer** | `src/trust_scorer.py` | Bayesian trust scoring with decay, burst detection |
+
+## New Database Tables (v2.5)
+
+| Table | Columns | Purpose |
+|-------|---------|---------|
+| `memory_events` | id, event_type, memory_id, source_agent, source_protocol, payload, importance, tier, created_at | Event log with tiered retention |
+| `subscriptions` | id, subscriber_id, channel, filter, webhook_url, durable, last_event_id | Event subscriptions |
+| `agent_registry` | id, agent_id, agent_name, protocol, first_seen, last_seen, memories_written, memories_recalled, trust_score, metadata | Agent tracking |
+| `trust_signals` | id, agent_id, signal_type, delta, old_score, new_score, context, created_at | Trust audit trail |
+
+## New Columns on `memories` Table (v2.5)
+
+| Column | Type | Default | Purpose |
+|--------|------|---------|---------|
+| `created_by` | TEXT | 'user' | Agent ID that created this memory |
+| `source_protocol` | TEXT | 'cli' | Protocol used (mcp, cli, rest, python, a2a) |
+| `trust_score` | REAL | 1.0 | Trust score at creation time |
+| `provenance_chain` | TEXT | '[]' | JSON derivation lineage |
+
+---
+
+## Event Types
+
+| Event | Trigger |
+|-------|---------|
+| `memory.created` | New memory written |
+| `memory.updated` | Existing memory modified |
+| `memory.deleted` | Memory removed |
+| `memory.recalled` | Memory retrieved by an agent |
+| `graph.updated` | Knowledge graph rebuilt |
+| `pattern.learned` | New pattern detected |
+| `agent.connected` | New agent connects |
+| `agent.disconnected` | Agent disconnects |
+
+## Event Retention Tiers
+
+| Tier | Window | Kept |
+|------|--------|------|
+| Hot | 0-48h | All events |
+| Warm | 2-14d | Importance >= 5 only |
+| Cold | 14-30d | Daily aggregates |
+| Archive | 30d+ | Pruned (stats in pattern_learner) |
+
+---
+
+## Trust Scoring (Silent Collection — v2.5)
+
+All agents start at trust 1.0. Signals adjust the score with a Bayesian decay factor that stabilizes over time.
+
+| Signal | Delta | When |
+|--------|-------|------|
+| `high_importance_write` | +0.015 | Memory with importance >= 7 |
+| `memory_recalled_by_others` | +0.02 | Another agent finds this memory useful |
+| `quick_delete` | -0.03 | Memory deleted within 1 hour |
+| `high_volume_burst` | -0.02 | >20 writes in 5 minutes |
+
+**v2.5:** Silent collection only. **v2.6:** Trust-weighted recall ranking. **v3.0:** Active enforcement.
+
+---
+
+## API Endpoints (28 total)
+
+| Route | Method | Module | Purpose |
+|-------|--------|--------|---------|
+| `/` | GET | ui_server.py | Dashboard |
+| `/health` | GET | ui_server.py | Health check |
+| `/api/memories` | GET | routes/memories.py | List memories |
+| `/api/graph` | GET | routes/memories.py | Knowledge graph data |
+| `/api/search` | POST | routes/memories.py | Semantic search |
+| `/api/clusters` | GET | routes/memories.py | Cluster info |
+| `/api/clusters/{id}` | GET | routes/memories.py | Cluster detail |
+| `/api/stats` | GET | routes/stats.py | System statistics |
+| `/api/timeline` | GET | routes/stats.py | Timeline aggregation |
+| `/api/patterns` | GET | routes/stats.py | Learned patterns |
+| `/api/profiles` | GET | routes/profiles.py | List profiles |
+| `/api/profiles/{n}/switch` | POST | routes/profiles.py | Switch profile |
+| `/api/profiles/create` | POST | routes/profiles.py | Create profile |
+| `/api/profiles/{n}` | DELETE | routes/profiles.py | Delete profile |
+| `/api/export` | GET | routes/data_io.py | Export memories |
+| `/api/import` | POST | routes/data_io.py | Import memories |
+| `/api/backup/*` | GET/POST | routes/backup.py | Backup management |
+| `/events/stream` | GET | routes/events.py | SSE real-time stream |
+| `/api/events` | GET | routes/events.py | Event polling |
+| `/api/events/stats` | GET | routes/events.py | Event statistics |
+| `/api/agents` | GET | routes/agents.py | Agent list |
+| `/api/agents/stats` | GET | routes/agents.py | Agent statistics |
+| `/api/trust/stats` | GET | routes/agents.py | Trust overview |
+| `/api/trust/signals/{id}` | GET | routes/agents.py | Agent trust signals |
+| `/ws/updates` | WS | routes/ws.py | WebSocket |
+
+---
+
+*SuperLocalMemory V2.5 — Created by Varun Pratap Bhardwaj. MIT License.*

package/mcp_server.py

@@ -46,6 +46,14 @@ except ImportError as e:
     print(f"Ensure SuperLocalMemory V2 is installed at {MEMORY_DIR}", file=sys.stderr)
     sys.exit(1)
 
+# Agent Registry + Provenance (v2.5+)
+try:
+    from agent_registry import AgentRegistry
+    from provenance_tracker import ProvenanceTracker
+    PROVENANCE_AVAILABLE = True
+except ImportError:
+    PROVENANCE_AVAILABLE = False
+
 # Parse command line arguments early (needed for port in constructor)
 import argparse as _argparse
 _parser = _argparse.ArgumentParser(add_help=False)
@@ -63,6 +71,80 @@ mcp = FastMCP(
 # Database path
 DB_PATH = MEMORY_DIR / "memory.db"
 
+# ============================================================================
+# Shared singleton instances (v2.5 — fixes per-call instantiation overhead)
+# All MCP tool handlers share one MemoryStoreV2 instance instead of creating
+# a new one per call. This means one ConnectionManager, one TF-IDF vectorizer,
+# one write queue — shared across all concurrent MCP requests.
+# ============================================================================
+
+_store = None
+_graph_engine = None
+_pattern_learner = None
+
+
+def get_store() -> MemoryStoreV2:
+    """Get or create the shared MemoryStoreV2 singleton."""
+    global _store
+    if _store is None:
+        _store = MemoryStoreV2(DB_PATH)
+    return _store
+
+
+def get_graph_engine() -> GraphEngine:
+    """Get or create the shared GraphEngine singleton."""
+    global _graph_engine
+    if _graph_engine is None:
+        _graph_engine = GraphEngine(DB_PATH)
+    return _graph_engine
+
+
+def get_pattern_learner() -> PatternLearner:
+    """Get or create the shared PatternLearner singleton."""
+    global _pattern_learner
+    if _pattern_learner is None:
+        _pattern_learner = PatternLearner(DB_PATH)
+    return _pattern_learner
+
+
+_agent_registry = None
+_provenance_tracker = None
+
+
+def get_agent_registry():
+    """Get shared AgentRegistry singleton (v2.5+). Returns None if unavailable."""
+    global _agent_registry
+    if not PROVENANCE_AVAILABLE:
+        return None
+    if _agent_registry is None:
+        _agent_registry = AgentRegistry.get_instance(DB_PATH)
+    return _agent_registry
+
+
+def get_provenance_tracker():
+    """Get shared ProvenanceTracker singleton (v2.5+). Returns None if unavailable."""
+    global _provenance_tracker
+    if not PROVENANCE_AVAILABLE:
+        return None
+    if _provenance_tracker is None:
+        _provenance_tracker = ProvenanceTracker.get_instance(DB_PATH)
+    return _provenance_tracker
+
+
+def _register_mcp_agent(agent_name: str = "mcp-client"):
+    """Register the calling MCP agent and record activity. Non-blocking."""
+    registry = get_agent_registry()
+    if registry:
+        try:
+            registry.register_agent(
+                agent_id=f"mcp:{agent_name}",
+                agent_name=agent_name,
+                protocol="mcp",
+            )
+        except Exception:
+            pass
+
+
 # ============================================================================
 # MCP TOOLS (Functions callable by AI)
 # ============================================================================
@@ -103,8 +185,11 @@ async def remember(
         remember("JWT auth with refresh tokens", tags="security,auth", importance=8)
     """
     try:
+        # Register MCP agent (v2.5 — agent tracking)
+        _register_mcp_agent()
+
         # Use existing MemoryStoreV2 class (no duplicate logic)
-        store = MemoryStoreV2(DB_PATH)
+        store = get_store()
 
         # Call existing add_memory method
         memory_id = store.add_memory(
@@ -114,6 +199,22 @@ async def remember(
             importance=importance
         )
 
+        # Record provenance (v2.5 — who created this memory)
+        prov = get_provenance_tracker()
+        if prov:
+            try:
+                prov.record_provenance(memory_id, created_by="mcp:client", source_protocol="mcp")
+            except Exception:
+                pass
+
+        # Track write in agent registry
+        registry = get_agent_registry()
+        if registry:
+            try:
+                registry.record_write("mcp:mcp-client")
+            except Exception:
+                pass
+
         # Format response
         preview = content[:100] + "..." if len(content) > 100 else content
 
@@ -174,7 +275,7 @@ async def recall(
     """
     try:
         # Use existing MemoryStoreV2 class
-        store = MemoryStoreV2(DB_PATH)
+        store = get_store()
 
         # Call existing search method
         results = store.search(query, limit=limit)
@@ -223,7 +324,7 @@ async def list_recent(limit: int = 10) -> dict:
     """
     try:
         # Use existing MemoryStoreV2 class
-        store = MemoryStoreV2(DB_PATH)
+        store = get_store()
 
         # Call existing list_all method
         memories = store.list_all(limit=limit)
@@ -263,7 +364,7 @@ async def get_status() -> dict:
     """
     try:
         # Use existing MemoryStoreV2 class
-        store = MemoryStoreV2(DB_PATH)
+        store = get_store()
 
         # Call existing get_stats method
         stats = store.get_stats()
@@ -303,7 +404,7 @@ async def build_graph() -> dict:
     """
     try:
         # Use existing GraphEngine class
-        engine = GraphEngine(DB_PATH)
+        engine = get_graph_engine()
 
         # Call existing build_graph method
         stats = engine.build_graph()
@@ -461,7 +562,7 @@ async def search(query: str) -> dict:
         {"results": [{"id": str, "title": str, "text": str, "url": str}]}
     """
     try:
-        store = MemoryStoreV2(DB_PATH)
+        store = get_store()
         raw_results = store.search(query, limit=20)
 
         results = []
@@ -504,7 +605,7 @@ async def fetch(id: str) -> dict:
         {"id": str, "title": str, "text": str, "url": str, "metadata": dict|null}
     """
     try:
-        store = MemoryStoreV2(DB_PATH)
+        store = get_store()
         mem = store.get_by_id(int(id))
 
         if not mem:
@@ -549,7 +650,7 @@ async def get_recent_memories_resource(limit: str) -> str:
     Usage: memory://recent/10
     """
     try:
-        store = MemoryStoreV2(DB_PATH)
+        store = get_store()
         memories = store.list_all(limit=int(limit))
         return json.dumps(memories, indent=2)
     except Exception as e:
@@ -564,7 +665,7 @@ async def get_stats_resource() -> str:
     Usage: memory://stats
     """
     try:
-        store = MemoryStoreV2(DB_PATH)
+        store = get_store()
         stats = store.get_stats()
         return json.dumps(stats, indent=2)
     except Exception as e:
@@ -579,7 +680,7 @@ async def get_clusters_resource() -> str:
     Usage: memory://graph/clusters
     """
     try:
-        engine = GraphEngine(DB_PATH)
+        engine = get_graph_engine()
         stats = engine.get_stats()
         clusters = stats.get('clusters', [])
         return json.dumps(clusters, indent=2)
@@ -595,7 +696,7 @@ async def get_coding_identity_resource() -> str:
     Usage: memory://patterns/identity
     """
     try:
-        learner = PatternLearner(DB_PATH)
+        learner = get_pattern_learner()
         patterns = learner.get_identity_context(min_confidence=0.5)
         return json.dumps(patterns, indent=2)
     except Exception as e:
@@ -615,7 +716,7 @@ async def coding_identity_prompt() -> str:
     based on learned preferences and patterns.
     """
     try:
-        learner = PatternLearner(DB_PATH)
+        learner = get_pattern_learner()
         patterns = learner.get_identity_context(min_confidence=0.6)
 
         if not patterns:
@@ -656,7 +757,7 @@ async def project_context_prompt(project_name: str) -> str:
         Formatted prompt with relevant project memories
     """
     try:
-        store = MemoryStoreV2(DB_PATH)
+        store = get_store()
 
         # Search for project-related memories
         memories = store.search(f"project:{project_name}", limit=20)
@@ -711,7 +812,7 @@ if __name__ == "__main__":
     # Print startup message to stderr (stdout is used for MCP protocol)
     print("=" * 60, file=sys.stderr)
     print("SuperLocalMemory V2 - MCP Server", file=sys.stderr)
-    print("Version: 2.4.1", file=sys.stderr)
+    print("Version: 2.5.0", file=sys.stderr)
     print("=" * 60, file=sys.stderr)
     print("Created by: Varun Pratap Bhardwaj (Solution Architect)", file=sys.stderr)
     print("Repository: https://github.com/varun369/SuperLocalMemoryV2", file=sys.stderr)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "superlocalmemory",
-  "version": "2.4.2",
+  "version": "2.5.0",
   "description": "Your AI Finally Remembers You - Local-first intelligent memory system for AI assistants. Works with Claude, Cursor, Windsurf, VS Code/Copilot, Codex, and 16+ AI tools. 100% local, zero cloud dependencies.",
   "keywords": [
     "ai-memory",