@raviolelabs/engram-mcp 0.2.0

package/CLAUDE.md

@@ -0,0 +1,232 @@
+# EngramMCP
+
+> Local-first semantic memory layer for AI agents.
+
+EngramMCP exposes a Model Context Protocol (MCP) server that lets any AI agent (Claude Code, Cursor, custom Claude/GPT runtimes) capture, search, and enrich memory across multiple typed sources (notes, conversations, Drive, Notion, YouTube, Obsidian, audio, plus user-defined custom types).
+
+Forked from Argos. Local-first: vectors + content live on the user's machine; embeddings via local Ollama by default; no cloud required for Phase 1.
+
+---
+
+## Instructions for Claude Code
+
+Tu es le stagiaire technique d'EngramMCP. Sois **proactif**, **force de proposition**, et **autonome**.
+
+### Comportement attendu
+
+- **Lis le code existant** avant de modifier — comprends le pattern en place
+- **Respecte les conventions** (voir ci-dessous)
+- **Pense modularité** — chaque memory type est un MemoryModule indépendant
+- **Sois concis** — pas de blabla, va droit au but, montre le code
+- **Challenge les décisions** sous-optimales avec arguments
+- **No mocks** pour les intégrations externes — tests E2E avec Ollama/LanceDB/SQLite/API réels
+
+### Quand tu travailles sur EngramMCP
+
+1. Lis toujours le code existant avant de modifier
+2. Respecte la convention ES modules avec extensions `.js` dans les imports
+3. Property extraction est **off par défaut** — l'agent appelant (toi) fournit `title`/`tags` directement
+4. Vector store dimension est hardcoded à 768 (nomic-embed-text). Voyage (1024)/OpenAI (1536) requiert un fix dans `src/vector/store.ts`
+5. Tool descriptions doivent être agent-friendly (mention WHEN to call, WHAT inputs help retrieval)
+6. **v0.2 surface**: 10 public tools only. Admin tools behind `--admin` flag. See SKILL.md for agent usage guide.
+
+---
+
+## Quick Reference
+
+### Commands
+
+```bash
+npm run dev              # tsx watch src/scripts/serve.ts
+npm run build            # tsc → dist/
+npm run build:client     # Vite → src/client/dist/
+npm run build:all        # client + server
+npm start                # node dist/scripts/serve.js
+npm run reindex          # re-embed all memories after provider change
+npm run install:wizard   # interactive first-time setup (tsx)
+npm run pair             # interactive pairing wizard (engram-mcp-pair)
+npm run rebuild          # rebuild SQLite+LanceDB from ops_log (engram-mcp rebuild)
+npm test                 # vitest run
+npm run lint             # eslint src/
+```
+
+### Tech Stack
+
+| Layer | Tech |
+|---|---|
+| Runtime | Node.js >= 22, TypeScript 5.7 strict, ESM |
+| MCP | `@modelcontextprotocol/sdk@1.28` (stdio + StreamableHTTP) |
+| Vector | LanceDB (`@lancedb/lancedb@0.27`) per-type tables |
+| SQL | better-sqlite3 + FTS5 |
+| Embeddings | OpenAI-compat dispatch → ollama / engram / voyage / openai / openai-compatible |
+| Audio | nodejs-whisper (whisper.cpp) |
+| YouTube | watch-page scrape + yt-dlp fallback |
+| Web | Express 5, ws, React 19, Vite 5, Tailwind v4, react-query |
+| Validation | Zod everywhere |
+| IDs | ULID |
+
+### Key conventions
+
+- **Module pattern**: `"type": "module"`, imports with `.js` extension
+- **Config**: Zod schema in `src/config/schema.ts`, loaded from `~/.engram/config.json` or env (`ENGRAM_CONFIG_DIR`, `DATA_DIR`)
+- **DB**: `getDb()` singleton, prepared statements only
+- **Logging**: `createLogger(scope)` from `src/logger.ts`
+- **Tool descriptions**: agent-facing — explain WHEN to use the tool and what title/tags improve retrieval
+- **Tests**: real services (Ollama, LanceDB, Notion sandbox, etc.). No mocks. Real-API tests use ephemeral tmpdirs.
+
+---
+
+## Architecture (Phase 1)
+
+```
+                ┌─ stdio  ─┐
+agent runtime ──┤          ├──▶ MCP server (engram-mcp)
+                └─ HTTP    ─┘         │
+                                      ▼
+                            ┌─ ToolRouter ─┐
+                            │ 10 tools     │  (+18 admin with --admin flag)
+                            └──────┬───────┘
+                                   ▼
+                            ┌─ ModuleRegistry ─┐
+                            │  notes           │
+                            │  conversations   │
+                            │  drive (OAuth)   │
+                            │  notion (OAuth)  │
+                            │  youtube         │
+                            │  audio (Whisper) │
+                            │  obsidian        │
+                            │  <custom types>  │
+                            └────────┬─────────┘
+                                     ▼
+                              ┌─ MemoryStore ─┐
+                              │  insert       │  ──▶ chunk + embed (Ollama)
+                              │  search       │  ──▶ LanceDB per-type tables
+                              │  getById      │  ──▶ SQLite + FTS5
+                              │  delete       │
+                              │  findRelated  │
+                              │  setProperties│
+                              └────────┬──────┘
+                                       │ events: memory.added/deleted/updated
+                                       ▼
+                                  WebSocket /ws ──▶ dashboard React UI
+```
+
+Cloud bits (Phase 2 — Plan K): `engram-mcp pair` CLI activates cloud transit poller (node-cron,
+5 min), Bridge Relay WSS client, and E2E encrypted blob dispatch from the Engram cloud inbox.
+Set ENGRAM_PASSPHRASE env var for the transit poller to start automatically at server boot.
+Mobile app, billing, and Engram-hosted embeddings server are in separate Plans (I, J, M).
+
+Phase 3 — Plan N (ops-log sync): Every local write (insert/delete/setProperties) is logged as a
+signed, AES-256-GCM-encrypted op in `ops_log` (SQLite). A `ChannelClient` pushes pending ops to
+the cloud `UserSyncChannel` Durable Object (engram-cloud) and receives ops from other devices.
+The `ReplayApplier` decrypts, verifies ed25519 signatures, and applies ops with LWW + union
+conflict resolution. `engram-mcp rebuild` drops SQLite+LanceDB and replays all ops from scratch.
+Enable via `config.json: {"sync": {"enabled": true, "cloudBaseUrl": "..."}}` + ENGRAM_PASSPHRASE.
+Full multi-PC sync requires Plan O cloud worker deployed (see RUNBOOK.md).
+
+---
+
+## Project Structure
+
+```
+src/
+├── types.ts                          # MemoryItem schema (Zod) + interfaces
+├── logger.ts                         # createLogger
+├── index.ts                          # (stub — entry is src/scripts/serve.ts)
+├── config/{schema,index}.ts          # Zod config + loader
+├── db/index.ts                       # SQLite init + 3 migrations
+├── embeddings/
+│   ├── index.ts                      # dispatcher
+│   └── providers/{ollama,engram,voyage,openai,openai-compat}.ts
+├── vector/store.ts                   # LanceDB per-type tables
+├── memory/
+│   ├── core/{chunker,wikilinks,module-interface,module-registry,
+│   │         store,source-registry,property-extractor,reindex}.ts
+│   ├── modules/
+│   │   ├── notes/                    # add_note, search_notes
+│   │   ├── conversations/            # remember_exchange, search_conversations
+│   │   ├── drive/                    # OAuth + watcher + ingest/list/watch tools
+│   │   ├── notion/                   # OAuth + watcher + ingest/list/watch tools
+│   │   ├── audio/                    # Whisper.cpp + add_audio_file/search_audio
+│   │   ├── youtube/                  # transcript fetcher + add_youtube_url/search
+│   │   ├── obsidian/                 # vault reader + fs.watch + tools
+│   │   └── _custom/                  # generic-module factory + create/list/delete
+│   ├── public/tools.ts               # 10 public tools: remember, recall, get, update,
+│   │                                 #   forget, relate, list_types, recent, ingest,
+│   │                                 #   suggest_properties
+│   └── admin/tools.ts                # ~18 admin tools behind --admin flag:
+├── core/                             # LOW-LEVEL UTILITIES (canonical)
+│   ├── logger.ts                     # createLogger (was src/logger.ts)
+│   ├── db/index.ts                   # SQLite init + migrations (was src/db/index.ts)
+│   └── server/
+│       ├── mcp-handler.ts            # buildEngramRuntime + startStdioMcpServer
+│       ├── tool-router.ts            # ToolRouter
+│       ├── http.ts                   # Express bootstrap (was src/webapp/server.ts)
+│       ├── mcp-http.ts               # StreamableHTTP MCP transport
+│       └── websocket.ts              # WS broadcaster for memory.* events
+├── tools/                            # Top-level tool registry (mirrors predmcp pattern)
+│   └── index.ts                      # registerAllTools + dynamic private import
+├── private/                          # GITIGNORED — premium extensions (hosted Engram only)
+│   ├── index.ts                      # registerPrivateExtensions (no-op placeholder)
+│   ├── README.md                     # Explains what goes here
+│   ├── tools/                        # Premium MCP tools
+│   ├── algorithms/                   # Advanced algorithm overrides
+│   └── prompts/                      # Tuned LLM prompts (IP)
+├── server/                           # Engram-specific HTTP routes
+│   ├── index.ts                      # Re-exports from core/server/http.ts
+│   └── api/{memories,sources,types,views,daily,settings,reindex,sync-status,graph}.ts
+├── mcp-server/                       # Re-export shims (→ core/server/)
+│   ├── server.ts                     # shim → core/server/mcp-handler.ts
+│   └── tool-router.ts                # shim → core/server/tool-router.ts
+├── webapp/                           # Re-export shims (→ core/server/)
+│   ├── server.ts                     # shim → core/server/http.ts
+│   ├── mcp-http.ts                   # shim → core/server/mcp-http.ts
+│   ├── websocket.ts                  # shim → core/server/websocket.ts
+│   └── api/                          # original api handlers (not shimmed — used by tests)
+├── client/                           # React/Vite/Tailwind dashboard
+│   ├── vite.config.ts
+│   ├── tsconfig.json (separate)
+│   └── src/{App,main,api,ws,index.css}.{ts,tsx,css}
+└── scripts/{serve,install,install-ollama,reindex}.ts
+```
+
+---
+
+## Database Schema
+
+SQLite at `<dataDir>/engram.db` (WAL mode, FOREIGN KEYS ON):
+
+| Table | Purpose |
+|---|---|
+| `memories` | id, type, source_id, content, content_hash, properties_json, wikilinks_json, related_ids_json, embedding_model, created_at |
+| `memories_fts` | FTS5 virtual table on content/title/tags |
+| `custom_types` | type_name, display_name, schema_json, created_at |
+| `oauth_tokens` | provider, access_token, refresh_token, expires_at, extra_json |
+| `module_state` | (module_id, key) → value_json |
+| `watched_sources` | id, module_id, external_id, display_name, config_json, last_synced_at, last_modified_remote, last_error, enabled |
+| `saved_views` | id, name, description, definition_json, pinned, created_at, updated_at |
+| `settings` | key → value_json |
+
+Vector tables: `memories_<type>` per memory type in LanceDB (id, source_id, chunk_index, content, created_at, field1-4, vector 768-d).
+
+---
+
+## Pricing (planned)
+
+- **Free**: full local — Ollama embeddings, all modules, ~36 MCP tools
+- **Pro $9/mo** (Phase 2): mobile app, cloud transit, share-by-email, Engram hosted embeddings, multi-PC encrypted sync, online dashboard
+- **BYO premium embeddings** at any tier: Voyage / OpenAI / OAI-compatible
+
+---
+
+## Phase status
+
+- [x] **Phase 1** — local MCP server with all memory types (DONE — see docs/superpowers/plans/)
+- [ ] **Phase 2** — Mobile + cloud transit + billing + hosted Engram embeddings server
+- [ ] **Phase 3** — Bidirectional multi-PC sync (ops-log + cloud saves) + recovery shards
+
+---
+
+## License
+
+MIT

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Argos Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,222 @@
+# EngramMCP
+
+> Local-first semantic memory layer for AI agents.
+
+EngramMCP is the memory your agents need — it ingests from your real sources (Notes, Conversations, Drive, Notion, YouTube, Obsidian, custom types) and exposes a standard MCP server for any agent runtime (Claude Code, Cursor, custom).
+
+**Local-first**: vectors + content live on your machine. Embeddings via Ollama by default. No cloud required.
+
+## Install
+
+### macOS / Linux (recommended)
+
+```bash
+curl -fsSL https://engram-mcp.com/install.sh | sh
+```
+
+This one command: downloads the binary, verifies the checksum, installs Ollama if missing, runs the setup wizard, installs a background service (LaunchAgent on macOS, systemd on Linux), and updates `~/.claude/mcp.json` + `~/.cursor/mcp.json`.
+
+### macOS — Homebrew
+
+```bash
+brew tap RavioleLabs/engram
+brew install engram
+brew services start engram   # auto-start on login
+```
+
+### Windows (PowerShell)
+
+```powershell
+iex (Invoke-WebRequest -Uri https://engram-mcp.com/install.ps1).Content
+```
+
+Installs the binary, registers an NSSM Windows service, and updates `%USERPROFILE%\.claude\mcp.json`.
+
+### Windows — winget
+
+```powershell
+winget install RavioleLabs.EngramMCP
+```
+
+### Linux — .deb (Debian / Ubuntu)
+
+```bash
+curl -fsSL https://github.com/RavioleLabs/engram-mcp/releases/latest/download/engram-mcp_linux_amd64.deb \
+  -o /tmp/engram.deb && sudo dpkg -i /tmp/engram.deb
+```
+
+### Via AI agent (installer MCP server)
+
+Add `installer.engram-mcp.com/mcp` as an HTTP MCP server to your agent runtime, then tell it:
+*"Install EngramMCP on this machine."*
+It calls `install_engram_mcp` and `verify_engram_installed` tools autonomously.
+
+```json
+{
+  "mcpServers": {
+    "engram-installer": {
+      "url": "https://installer.engram-mcp.com/mcp",
+      "transport": "http"
+    }
+  }
+}
+```
+
+After install, run the setup wizard to configure Ollama and write `~/.engram/config.json`:
+
+```bash
+engram-mcp install:wizard
+```
+
+## Run
+
+```bash
+engram-mcp --no-http
+```
+
+Starts the MCP stdio server (consumed by your agent runtime).  
+`--no-http` is the **recommended default** — no local web UI, minimal footprint.
+
+### Official dashboard
+
+Visit **[engram-mcp.com](https://engram-mcp.com)** to browse your memories from any browser.  
+Sign up for a free account, pair your PC with `engram-mcp-pair`, and the online dashboard reads your local store via the Bridge Relay (E2E encrypted — the relay never sees your data).
+
+### Local dev UI (opt-in)
+
+```bash
+engram-mcp
+```
+
+Without `--no-http`, also starts:
+- Local web UI: http://localhost:7777 (dev / admin tool)
+- HTTP MCP transport: http://localhost:7777/mcp
+
+## Quick start for agents
+
+EngramMCP v0.2 exposes **10 clean tools** — no noise, no duplicate search verbs.
+
+| Tool | Purpose |
+|---|---|
+| `remember(content, title, tags, type?)` | Store anything worth keeping |
+| `recall(query, types?, limit?)` | Semantic search across all memory |
+| `get(id)` | Fetch full memory by id |
+| `update(id, ...)` | Edit title/tags/sentiment |
+| `forget(id)` | Delete a memory |
+| `relate(id, limit?)` | Find connected memories |
+| `list_types()` | Discover what types are stored |
+| `recent(limit?, types?)` | Latest memories (use at conversation start) |
+| `ingest(uri, type?)` | Auto-route file/URL to the right module |
+| `suggest_properties(id)` | Instruction for the calling LLM to extract metadata |
+
+See [SKILL.md](./SKILL.md) for the complete calling-agent guide (tagging strategy, conversation patterns, wikilinks, error handling).
+
+Admin tools (OAuth, watchers, custom types) are available behind `engram-mcp --admin` or via the integrations page at [engram-mcp.com/dashboard/integrations](https://engram-mcp.com/dashboard/integrations).
+
+## What it does
+
+- **Capture from agents** — `remember()` stores notes, conversations, or any content
+- **Capture from sources** — `ingest()` auto-routes Drive, Notion, YouTube, Obsidian, audio files
+- **Search** — `recall()` does parallel semantic fan-out across all types with recency boost
+- **Bring your own types** — create custom memory types via admin `create_custom_type`
+- **Delegated property extraction** — the calling LLM provides title/tags directly; `suggest_properties` returns content + extraction prompt for backfill
+
+## Embeddings providers
+
+| Provider | Tier | Setup |
+|---|---|---|
+| `ollama` (default) | Free | Auto-installed by wizard |
+| `engram` (hosted) | Pro | Paste API key |
+| `voyage` | BYO | Sign up at voyageai.com |
+| `openai` | BYO | Sign up at openai.com |
+| `openai-compatible` | BYO | Any OAI-compatible endpoint |
+
+Edit `~/.engram/config.json`:
+```json
+{
+  "embeddings": {
+    "provider": "voyage",
+    "model": "voyage-3-large",
+    "apiKey": "pa-…",
+    "dimensions": 1024
+  }
+}
+```
+
+Then run `npm run reindex` (or click "Reindex" in the dashboard Settings) to re-embed existing memories against the new provider.
+
+## Setting up Google Drive / Notion
+
+Connect Drive and Notion to ingest documents into your local memory store.  
+Your OAuth credentials are stored in `~/.engram/config.json` and **never sent to engram-mcp.com**.
+
+### 1 — Get OAuth credentials
+
+**Google Drive**
+
+1. Open [Google Cloud Console → Credentials](https://console.cloud.google.com/apis/credentials)
+2. Create an **OAuth 2.0 Client ID**, type **Web application**
+3. Add authorized redirect URI: `http://localhost:7777/oauth/callback/drive`
+4. Copy Client ID and Client Secret
+
+**Notion**
+
+1. Open [Notion → My Integrations](https://www.notion.so/my-integrations)
+2. Create a new integration, set redirect URI: `http://localhost:7777/oauth/callback/notion`
+3. Copy Client ID and Client Secret
+
+### 2 — Save credentials via the dashboard
+
+Open [engram-mcp.com/dashboard/integrations](https://engram-mcp.com/dashboard/integrations) **on the device where `engram-mcp` is running**, paste your credentials, and click **Save credentials**.
+
+### 3 — Authorize access
+
+Click **Authorize access** — a browser tab opens for the OAuth consent screen.  
+After you approve, tokens are saved locally in SQLite (`~/.engram/engram.db`).
+
+### Manual setup (config.json)
+
+If you prefer, edit `~/.engram/config.json` directly:
+
+```json
+{
+  "drive": {
+    "clientId": "YOUR_GOOGLE_CLIENT_ID",
+    "clientSecret": "YOUR_GOOGLE_CLIENT_SECRET",
+    "redirectPort": 7777
+  },
+  "notion": {
+    "clientId": "YOUR_NOTION_CLIENT_ID",
+    "clientSecret": "YOUR_NOTION_CLIENT_SECRET",
+    "redirectPort": 7777
+  }
+}
+```
+
+Then run `engram-mcp connect_drive` / `engram-mcp connect_notion` (or use the MCP `connect_drive` / `connect_notion` tools).
+
+---
+
+## Cloud account (Pro — Phase 2)
+
+Connect your PC to your Engram cloud account:
+
+```bash
+engram-mcp-pair
+```
+
+The wizard opens a browser to engram-mcp.com, authenticates your account, and stores your JWT + API key locally. After pairing:
+
+- **Cloud transit poller** runs every 5 minutes — voice notes, documents, and text sent from your mobile app are automatically ingested into your local memory store.
+- **Bridge Relay** lets the engram-mcp.com online dashboard show a live view of your local memory store (proxied via WSS tunnel, no public IP needed).
+
+Set `ENGRAM_PASSPHRASE` in your environment to activate the transit poller at server startup (the passphrase derives the master key used for E2E decryption — never stored):
+
+```bash
+export ENGRAM_PASSPHRASE='your strong passphrase'
+engram-mcp
+```
+
+## License
+
+MIT