gnosys-mcp 1.1.1 → 1.3.0

package/README.md

@@ -245,18 +245,29 @@ ANTHROPIC_API_KEY = "your-key-here"
 | `gnosys_ask` | Ask a question, get a synthesized answer with citations |
 | `gnosys_reindex` | Rebuild semantic embeddings from all memories |
 | `gnosys_list` | List memories with optional filters |
+| `gnosys_lens` | Filtered views — combine category, tag, status, confidence, date filters |
 | `gnosys_add` | Add a memory (LLM-structured) |
 | `gnosys_add_structured` | Add with explicit fields (no LLM) |
 | `gnosys_update` | Update frontmatter or content |
 | `gnosys_reinforce` | Signal usefulness of a memory |
 | `gnosys_commit_context` | Extract memories from conversation context |
+| `gnosys_bootstrap` | Batch-import existing markdown documents |
 | `gnosys_import` | Bulk import from CSV, JSON, or JSONL |
 | `gnosys_init` | Initialize a new store |
-| `gnosys_maintain` | Run vault maintenance (decay, dedup, consolidation) |
-| `gnosys_dashboard` | System dashboard (memory count, health, graph, LLM status) |
+| `gnosys_stale` | Find memories not modified within N days |
+| `gnosys_history` | Git-backed version history for a memory |
+| `gnosys_rollback` | Rollback a memory to a previous commit |
+| `gnosys_timeline` | Show when memories were created/modified over time |
+| `gnosys_stats` | Summary statistics for the memory store |
+| `gnosys_links` | Show wikilinks and backlinks for a memory |
+| `gnosys_graph` | Full cross-reference graph across all memories |
+| `gnosys_maintain` | Run vault maintenance (decay, dedup, consolidation, archiving) |
+| `gnosys_dearchive` | Force-dearchive memories from archive back to active |
+| `gnosys_dashboard` | System dashboard (memory count, health, archive, graph, LLM status) |
 | `gnosys_reindex_graph` | Build/rebuild the wikilink graph |
 | `gnosys_stores` | Show active stores |
 | `gnosys_tags` | List tag registry |
+| `gnosys_tags_add` | Add a new tag to the registry |
 
 ---
 
@@ -276,6 +287,7 @@ your-project/
     nvd-cves/
       cve-2024-1234.md
     gnosys.json          # configuration
+    archive.db           # two-tier memory archive (SQLite)
     .config/tags.json    # tag registry
     CHANGELOG.md
     .git/                # auto-versioned
@@ -528,26 +540,111 @@ The `gnosys_maintain` MCP tool lets agents trigger maintenance programmatically
 
 ## Comparison
 
-| Feature | **Gnosys** | NotebookLM | Official MCP Memory |
-|---------|-----------|------------|-------------------|
-| Storage | Markdown files + Git | Google proprietary | JSON file |
-| Transparent/editable | ✅ Plain `.md` files | ❌ Opaque | ✅ But flat JSON |
-| Version history | ✅ Full Git history | ❌ | ❌ |
-| Obsidian vault | ✅ Native | ❌ | ❌ |
-| Bulk import | ✅ CSV/JSON/JSONL | ❌ Manual | ❌ |
-| MCP server | ✅ Native | ❌ | ✅ |
-| CLI | ✅ Full-featured | ❌ | ❌ |
-| Layered stores | ✅ 4 layers | ❌ | ❌ |
-| Wikilinks | ✅ Auto-generated | ❌ | ❌ |
-| Search | Hybrid: FTS5 + semantic + RRF | Proprietary | None |
-| Freeform Q&A | ✅ gnosys_ask with citations | ✅ Built-in | ❌ |
-| Self-hosted | ✅ | ❌ | ✅ |
-| LLM providers | 5 (Anthropic, Ollama, Groq, OpenAI, LM Studio) | Proprietary | No LLM |
-| Wikilink graph | ✅ Persistent JSON graph | ❌ | ❌ |
-| System dashboard | ✅ Pretty CLI + MCP tool | ❌ | ❌ |
-| Auto maintenance | ✅ Decay, dedup, consolidation | ❌ | ❌ |
-| Docker support | ✅ | ❌ | ❌ |
-| Price | Free / MIT | Free tier, then paid | Free |
+Agent memory is a spectrum — from a single markdown file to full knowledge graphs. Here's an honest look at the trade-offs.
+
+| Aspect | Plain Markdown | RAG (Vector DB) | Knowledge Graph | **Gnosys** |
+|--------|---------------|-----------------|-----------------|-----------|
+| **Examples** | CLAUDE.md, .cursorrules | Mem0, LangChain Memory | Graphiti/Zep, Mem0 Graph | — |
+| **Storage** | `.md` files | Embeddings in vector DB | Nodes/edges in graph DB | `.md` files + SQLite index + SQLite archive |
+| **Transparency** | Perfect | Lossy (embeddings) | High (query nodes) | High (readable markdown) |
+| **Version history** | Git native | None built-in | None built-in | Git native |
+| **Keyword search** | Manual / grep | BM25 layer (some) | BM25 layer (some) | FTS5 (built-in) |
+| **Semantic search** | None | Vector similarity | Graph + vectors | Vector + FTS5 hybrid (RRF) |
+| **Relationship traversal** | None | None | Multi-hop graph queries | Wikilinks (manual encoding) |
+| **Automatic extraction** | No | Yes (embeddings) | Yes (entities + edges) | No (explicit structuring) |
+| **Conflict detection** | No | No | Yes (graph rules) | No |
+| **Scale comfort zone** | ~5K memories | 100K+ | 100K+ | 100K+ (two-tier: active `.md` + SQLite archive) |
+| **Setup time** | < 5 min | 30 min – 2 hours | 4 – 8 hours | 15 – 30 min |
+| **Infrastructure** | None | Vector DB + embeddings API | Graph DB + LLM | SQLite (embedded) |
+| **Human editability** | Excellent | Poor (re-embed) | Moderate | Excellent |
+| **MCP integration** | Via skill files | Custom server | Mem0 ships MCP | MCP server (included) |
+| **Obsidian compatible** | Partially | No | No | Yes (full vault) |
+| **Cost** | Free | $0–500+/mo (cloud DB + embeddings) | $250+/mo (Mem0 Pro) or self-host | Free (MIT) |
+| **Memory lifecycle** | Manual cleanup | Manual / TTL | Manual / TTL | Auto-archive + auto-dearchive on cite |
+| **Offline capable** | Yes | Self-hosted only | Self-hosted only | Yes (Ollama/LM Studio) |
+
+### Where others genuinely win
+
+- **Knowledge graphs** (Graphiti, Mem0 Graph) excel at multi-hop reasoning ("Who does Alice report to?") and automatic conflict detection. If your domain has clear entities and relationships — org charts, dependency trees, CRM data — a graph DB is the right tool.
+- **RAG/vector search** handles fuzzy semantic matching without requiring explicit keyword clouds. You don't need to think about relevance fields — the embeddings handle conceptual similarity automatically.
+- **Automatic extraction** in both RAG and graph approaches means the system learns from conversations without you explicitly structuring each fact.
+
+### Where Gnosys wins
+
+- **Zero infrastructure**: No vector DB to deploy, no graph DB to manage. SQLite is embedded.
+- **Full transparency**: Every memory is a readable, editable `.md` file. No opaque embeddings.
+- **Git versioning**: Complete history, rollback, diff, blame. No other memory system versions every write.
+- **Obsidian native**: Browse, edit, graph view, wikilinks — all with your existing Obsidian setup.
+- **Hybrid search without a vector DB**: FTS5 keyword search is built-in. Semantic search is optional (local embeddings via Ollama, no API costs).
+- **Bulk import**: CSV, JSON, JSONL. Turn a dataset into a searchable knowledge base in seconds.
+- **Cost**: Genuinely free. No cloud service, no API costs if using local LLM providers.
+- **Two-tier memory**: Active memories stay lightning-fast as markdown files. Old/low-confidence memories automatically archive to SQLite — and auto-dearchive when needed by search or ask.
+
+### Two-Tier Memory (Active + Archive)
+
+Gnosys uses a two-tier architecture so your current work stays fast while safely growing to 100k+ memories:
+
+**Active layer** — `.gnosys/<category>/*.md` — fast markdown files for day-to-day work.
+
+**Archive layer** — `.gnosys/archive.db` — SQLite database for old or low-confidence memories.
+
+The flow is fully automatic and bidirectional:
+1. `gnosys maintain --auto-apply` moves stale memories (>90 days unreinforced + confidence below 0.3) from active → archive
+2. Every search and ask query checks the archive if active results are insufficient
+3. Archived memories that get cited in an answer are automatically restored to active and reinforced
+
+You can also force-dearchive with `gnosys dearchive "query"` or the `gnosys_dearchive` MCP tool.
+
+Configure thresholds in `gnosys.json`:
+```json
+{
+  "archive": {
+    "maxActiveDays": 90,
+    "minConfidence": 0.3
+  }
+}
+```
+
+### Enterprise Reliability (v1.3.0)
+
+Built for long-running agent orchestrators that call Gnosys hundreds of times per session.
+
+**Always-on recall** — `gnosys_recall` tool + `gnosys://recall` resource — Sub-50ms automatic context injection. No LLM, no embeddings. Three modes: **aggressive** (default, always injects top 3+), **balanced** (relevance threshold), **conservative** (high relevance only). MCP resource with `priority: 1` for hosts that support automatic resource injection. Returns `<gnosys-recall>` blocks or a `<gnosys: no-strong-recall-needed>` heartbeat marker.
+
+```bash
+# CLI — aggressive mode (default)
+gnosys recall "React state management"
+
+# Override mode, host-friendly format
+gnosys recall "React state management" --mode balanced --host
+
+# MCP tool (called by orchestrator before each turn)
+gnosys_recall { query: "React state management", mode: "aggressive" }
+```
+
+Configure in `gnosys.json`:
+```json
+{
+  "recall": {
+    "mode": "aggressive",
+    "minRelevanceScore": 0.65,
+    "maxMemoriesPerTurn": 8,
+    "alwaysInjectTopN": 3
+  }
+}
+```
+
+**Concurrency safety** — Write locking with PID tracking prevents corruption when multiple agents write simultaneously. SQLite databases (archive + embeddings) use WAL mode for concurrent reads during writes. Stale lock detection auto-recovers from crashed processes.
+
+**Audit trail** — Every memory operation (read, write, recall, ask, maintain, archive, dearchive) is logged to `.gnosys/.config/audit.jsonl` with timestamps, durations, and optional traceIds for correlation with your outer orchestrator.
+
+```bash
+gnosys audit --days 7 --operation recall --json
+```
+
+**Deterministic dearchive** — When an LLM-synthesized answer cites archived memories, a three-stage fallback ensures they're always restored: path match → title match → all archive results from context. No memory is left behind even if the LLM output is unpredictable.
+
+**Performance monitoring** — `gnosys dashboard` now includes enterprise performance benchmarks: recall latency, active search latency, and archive search latency, with warnings if recall exceeds the 50ms target.
 
 ---
 
@@ -558,31 +655,47 @@ gnosys --help               # List all commands
 gnosys init                  # Initialize a new store
 gnosys add "raw input"       # Add memory via LLM
 gnosys add-structured ...    # Add memory with explicit fields
+gnosys commit-context "..."  # Extract memories from conversation
+gnosys bootstrap <dir>       # Batch-import existing markdown files
+gnosys import <file> ...     # Bulk import CSV/JSON/JSONL data
 gnosys discover "keywords"   # Find relevant memories (metadata only)
 gnosys search "query"        # Full-text search with snippets
 gnosys hybrid-search "q"     # Hybrid keyword + semantic search
 gnosys semantic-search "q"   # Semantic similarity search
 gnosys ask "question"        # Ask a question, get cited answer
-gnosys reindex               # Build/rebuild semantic embeddings
 gnosys read <path>           # Read a specific memory
 gnosys list                  # List all memories
+gnosys lens                  # Filtered views (category, tag, status, date...)
 gnosys update <path> ...     # Update a memory
 gnosys reinforce <id> ...    # Signal memory usefulness
 gnosys stale                 # Find stale memories
-gnosys commit-context "..."  # Extract memories from conversation
-gnosys import <file> ...     # Bulk import data
+gnosys history <path>        # Git-backed version history
+gnosys rollback <path> <hash>  # Rollback to a previous commit
+gnosys timeline              # Knowledge evolution over time
+gnosys stats                 # Summary statistics
+gnosys links <path>          # Wikilinks and backlinks for a memory
+gnosys graph                 # Full cross-reference graph
+gnosys tags                  # List tag registry
+gnosys tags-add              # Add a new tag
+gnosys reindex               # Build/rebuild semantic embeddings
+gnosys reindex-graph         # Build/rebuild wikilink graph
 gnosys maintain              # Run vault maintenance (dry run by default)
-gnosys maintain --dry-run    # Preview changes without modifying
-gnosys maintain --auto-apply # Apply all maintenance automatically
+gnosys maintain --auto-apply # Apply all maintenance + archiving automatically
+gnosys dearchive "query"     # Force-dearchive memories from archive to active
 gnosys dashboard             # Pretty system dashboard
 gnosys dashboard --json      # Dashboard as JSON
-gnosys reindex-graph         # Build/rebuild wikilink graph
 gnosys config show           # Show SOC configuration
 gnosys config set provider <name>  # Set default provider
 gnosys config set task <task> <provider> <model>  # Route task
 gnosys doctor                # Full system health check (all providers)
-gnosys tags                  # List tag registry
 gnosys stores                # Show active stores
+gnosys recall "query"        # Always-on recall (aggressive mode by default)
+gnosys recall "q" --mode balanced  # Override recall mode
+gnosys recall "q" --host     # Output in <gnosys-recall> host format
+gnosys recall "q" --json     # Recall as JSON for programmatic use
+gnosys audit                 # View audit trail (last 7 days)
+gnosys audit --days 30       # View last 30 days of operations
+gnosys audit --operation ask # Filter by operation type
 gnosys serve                 # Start MCP server (stdio)
 gnosys serve --with-maintenance  # MCP server + maintenance every 6h
 ```
@@ -612,8 +725,12 @@ src/
     hybridSearch.ts # Hybrid search with RRF fusion
     ask.ts          # Freeform Q&A with LLM synthesis + citations
     llm.ts          # LLM abstraction — System of Cognition (5 providers)
-    maintenance.ts  # Auto-maintenance: decay, dedup, consolidation, reinforcement
-    dashboard.ts    # Aggregated system dashboard
+    maintenance.ts  # Auto-maintenance: decay, dedup, consolidation, archiving
+    archive.ts      # Two-tier memory: active ↔ archive (SQLite)
+    recall.ts       # Ultra-fast recall hook for agent orchestrators
+    audit.ts        # Structured JSONL audit logging
+    lock.ts         # File-level write locking + WAL helper
+    dashboard.ts    # Aggregated system dashboard + performance monitoring
     graph.ts        # Persistent wikilink graph (graph.json)
     tags.ts         # Tag registry management
     ingest.ts       # LLM-powered structuring (with retry logic)
@@ -636,22 +753,20 @@ src/
 
 Real numbers from our demo vault (120 memories — 100 USDA foods + 20 NVD CVEs):
 
-| Metric | Gnosys | NotebookLM |
-|--------|--------|------------|
-| Import 100 records | 0.6s (structured) | Manual upload |
-| Cold start (first load) | 0.3s | ~5s (cloud) |
-| Keyword search | <10ms (FTS5) | Cloud-dependent |
-| Hybrid search (keyword + semantic) | ~50ms | N/A |
-| Reindex 120 embeddings | ~8s (first run: model download ~80 MB) | N/A |
-| Maintenance dry-run (120 memories) | ~2s | N/A |
-| Graph reindex (120 memories) | <1s | N/A |
-| Storage per memory | ~1 KB `.md` file | Opaque |
-| Embedding storage | ~0.3 MB for 120 memories | Cloud |
-| LLM providers | 5 (Anthropic, Ollama, Groq, OpenAI, LM Studio) | 1 (Google) |
-| Offline capable | ✅ (Ollama / LM Studio) | ❌ |
-| Test suite | 143 tests, 0 errors | N/A |
-
-All benchmarks on Apple M-series hardware, Node.js 20+. Import speed depends on mode — `structured` bypasses LLM entirely. LLM-enriched imports depend on provider latency.
+| Metric | Result |
+|--------|--------|
+| Import 100 records (structured) | 0.6s |
+| Cold start (first load) | 0.3s |
+| Keyword search (FTS5) | <10ms |
+| Hybrid search (keyword + semantic) | ~50ms |
+| Reindex 120 embeddings | ~8s (first run downloads ~80 MB model) |
+| Maintenance dry-run (120 memories) | ~2s |
+| Graph reindex (120 memories) | <1s |
+| Storage per memory | ~1 KB `.md` file |
+| Embedding storage (120 memories) | ~0.3 MB |
+| Test suite | 143 tests, 0 errors |
+
+All benchmarks on Apple M-series hardware, Node.js 20+. Structured imports bypass LLM entirely. LLM-enriched imports depend on provider latency.
 
 ---
 
@@ -678,14 +793,6 @@ Gnosys is open source (MIT) and actively developed. Here's how to get involved:
 
 ---
 
-## Roadmap
-
-See the [6-phase roadmap](https://gnosys.ai/roadmap) for what's next.
-
-**Have ideas?** [Join the discussion →](https://github.com/proticom/gnosys-mcp/discussions)
-
----
-
 ## License
 
 MIT — [LICENSE](LICENSE)

package/dist/cli.js

@@ -1163,7 +1163,7 @@ program
     }
     const embeddings = new GnosysEmbeddings(storePath);
     const hybridSearch = new GnosysHybridSearch(search, embeddings, resolver, storePath);
-    const ask = new GnosysAsk(hybridSearch, cliConfig);
+    const ask = new GnosysAsk(hybridSearch, cliConfig, resolver, storePath);
     if (!ask.isLLMAvailable) {
         console.error("No LLM provider available. Set ANTHROPIC_API_KEY or switch to Ollama: gnosys config set provider ollama");
         process.exit(1);
@@ -1453,10 +1453,53 @@ program
     console.log("");
     console.log(formatMaintenanceReport(report));
 });
+// ─── gnosys dearchive ───────────────────────────────────────────────────
+program
+    .command("dearchive <query>")
+    .description("Force-dearchive memories matching a query from archive.db back to active")
+    .option("--limit <n>", "Max memories to dearchive", "5")
+    .action(async (query, opts) => {
+    const { GnosysArchive } = await import("./lib/archive.js");
+    const resolver = await getResolver();
+    const stores = resolver.getStores();
+    if (stores.length === 0) {
+        console.error("No Gnosys stores found. Run gnosys init first.");
+        process.exit(1);
+    }
+    const writeTarget = resolver.getWriteTarget();
+    if (!writeTarget) {
+        console.error("No writable store found.");
+        process.exit(1);
+    }
+    const archive = new GnosysArchive(writeTarget.path);
+    if (!archive.isAvailable()) {
+        console.error("Archive not available. Is better-sqlite3 installed?");
+        process.exit(1);
+    }
+    const results = archive.searchArchive(query, parseInt(opts.limit));
+    if (results.length === 0) {
+        console.log(`No archived memories found matching "${query}".`);
+        archive.close();
+        return;
+    }
+    console.log(`Found ${results.length} archived memories matching "${query}":\n`);
+    for (const r of results) {
+        console.log(`  • ${r.title} (${r.id})`);
+    }
+    console.log("");
+    // Dearchive all found
+    const ids = results.map((r) => r.id);
+    const restored = await archive.dearchiveBatch(ids, writeTarget.store);
+    archive.close();
+    console.log(`Dearchived ${restored.length} memories back to active:`);
+    for (const rp of restored) {
+        console.log(`  → ${rp}`);
+    }
+});
 // ─── gnosys doctor ──────────────────────────────────────────────────────
 program
     .command("doctor")
-    .description("Check system health: stores, LLM connectivity, embeddings")
+    .description("Check system health: stores, LLM connectivity, embeddings, archive")
     .action(async () => {
     const resolver = await getResolver();
     const stores = resolver.getStores();
@@ -1474,6 +1517,31 @@ program
         }
     }
     console.log("");
+    // Check archive
+    if (stores.length > 0) {
+        console.log("Archive (Two-Tier Memory):");
+        try {
+            const { GnosysArchive } = await import("./lib/archive.js");
+            const archive = new GnosysArchive(stores[0].path);
+            if (archive.isAvailable()) {
+                const stats = archive.getStats();
+                console.log(`  Archived memories: ${stats.totalArchived}`);
+                if (stats.totalArchived > 0) {
+                    console.log(`  Archive DB size: ${stats.dbSizeMB.toFixed(2)} MB`);
+                    console.log(`  Oldest archived: ${stats.oldestArchived}`);
+                    console.log(`  Newest archived: ${stats.newestArchived}`);
+                }
+                archive.close();
+            }
+            else {
+                console.log("  Not available (better-sqlite3 not installed)");
+            }
+        }
+        catch {
+            console.log("  Not initialized");
+        }
+        console.log("");
+    }
     // Check config — SOC routing
     const cfg = stores.length > 0 ? await loadConfig(stores[0].path) : DEFAULT_CONFIG;
     console.log("System of Cognition (SOC):");
@@ -1616,5 +1684,84 @@ program
     }
     await import("./index.js");
 });
+// ─── gnosys recall ───────────────────────────────────────────────────────
+program
+    .command("recall <query>")
+    .description("Always-on memory recall — injects most relevant memories as context (sub-50ms, no LLM)")
+    .option("--limit <n>", "Max memories to return (default from config)")
+    .option("--mode <mode>", "Recall mode: aggressive | balanced | conservative (default from config)")
+    .option("--trace-id <id>", "Trace ID for audit correlation")
+    .option("--json", "Output raw JSON instead of formatted text")
+    .option("--host", "Output in host-friendly <gnosys-recall> format (default for MCP)")
+    .action(async (query, opts) => {
+    const resolver = new GnosysResolver();
+    await resolver.resolve();
+    const stores = resolver.getStores();
+    if (stores.length === 0) {
+        console.error("No Gnosys stores found. Run 'gnosys init' first.");
+        process.exit(1);
+    }
+    const { recall, formatRecall, formatRecallCLI } = await import("./lib/recall.js");
+    const { initAudit, closeAudit } = await import("./lib/audit.js");
+    const storePath = stores[0].path;
+    initAudit(storePath);
+    // Load config for recall settings
+    const cfg = await loadConfig(storePath);
+    const recallConfig = {
+        ...cfg.recall,
+        ...(opts.mode ? { mode: opts.mode } : {}),
+    };
+    // Build search index
+    const search = new GnosysSearch(storePath);
+    await search.addStoreMemories(stores[0].store);
+    const result = await recall(query, {
+        limit: opts.limit ? parseInt(opts.limit, 10) : undefined,
+        search,
+        resolver,
+        storePath,
+        traceId: opts.traceId,
+        recallConfig,
+    });
+    if (opts.json) {
+        console.log(JSON.stringify(result, null, 2));
+    }
+    else if (opts.host) {
+        console.log(formatRecall(result));
+    }
+    else {
+        console.log(formatRecallCLI(result));
+    }
+    closeAudit();
+});
+// ─── gnosys audit ────────────────────────────────────────────────────────
+program
+    .command("audit")
+    .description("View the structured audit trail of memory operations")
+    .option("--days <n>", "Show entries from the last N days", "7")
+    .option("--operation <op>", "Filter by operation type (read, write, recall, etc.)")
+    .option("--limit <n>", "Max entries to show")
+    .option("--json", "Output raw JSON instead of formatted timeline")
+    .action(async (opts) => {
+    const resolver = new GnosysResolver();
+    await resolver.resolve();
+    const stores = resolver.getStores();
+    if (stores.length === 0) {
+        console.error("No Gnosys stores found. Run 'gnosys init' first.");
+        process.exit(1);
+    }
+    const { readAuditLog, formatAuditTimeline } = await import("./lib/audit.js");
+    const storePath = stores[0].path;
+    const entries = readAuditLog(storePath, {
+        days: parseInt(opts.days, 10),
+        operation: opts.operation,
+        limit: opts.limit ? parseInt(opts.limit, 10) : undefined,
+    });
+    if (opts.json) {
+        console.log(JSON.stringify(entries, null, 2));
+    }
+    else {
+        console.log(formatAuditTimeline(entries));
+    }
+});
 program.parse();
 //# sourceMappingURL=cli.js.map