agent-knowledge 1.0.0

package/docs/ARCHITECTURE.md

@@ -0,0 +1,244 @@
+# Architecture
+
+## Overview
+
+```mermaid
+graph TB
+    Claude[Claude Code] -->|MCP stdio| Server[server.ts]
+    Server --> Knowledge[Knowledge Module]
+    Server --> Session[Session Module]
+
+    Knowledge --> Store[store.ts — CRUD]
+    Knowledge --> KSearch[search.ts — TF-IDF]
+    Knowledge --> Git[git.ts — Sync]
+
+    Store --> Vault[(~/claude-memory)]
+    Git --> Remote[(Git Remote)]
+
+    Session --> Parser[parser.ts — JSONL + Cache]
+    Session --> SSearch[search.ts — TF-IDF Index]
+    Session --> Scopes[scopes.ts — 6 Filters]
+    Session --> Summary[summary.ts]
+
+    Parser --> Transcripts[(~/.claude/projects/*.jsonl)]
+
+    Server --> Dashboard[dashboard.ts — :3423]
+    Dashboard --> HTTP[REST API]
+    Dashboard --> WS[WebSocket]
+    Dashboard --> Watcher[File Watcher]
+    HTTP --> Browser[Browser UI]
+    WS --> Browser
+```
+
+## File Structure
+
+```
+src/
+  index.ts              Entry point — MCP stdio + dashboard auto-start
+  server.ts             10 tool definitions, request routing, error handling
+  dashboard.ts          HTTP + WebSocket server, REST API, file watcher
+  types.ts              KnowledgeConfig interface, getConfig()
+  knowledge/
+    store.ts            CRUD for markdown entries with YAML frontmatter
+    search.ts           TF-IDF search over knowledge entries
+    git.ts              git pull/push/sync with execSync + timeouts
+  sessions/
+    parser.ts           JSONL parsing with mtime-based file cache
+    search.ts           TF-IDF ranked search with 60s global index cache
+    scopes.ts           6 search scopes, post-filters cached index results
+    summary.ts          Topic extraction, tool/file detection
+  search/
+    tfidf.ts            TF-IDF scoring engine (tokenizer, stopwords, index)
+    fuzzy.ts            Levenshtein distance, sliding window matching
+    types.ts            SearchResult, SearchOptions interfaces
+  ui/
+    index.html          Dashboard SPA
+    styles.css           MD3 design tokens (light + dark)
+    app.js              Client-side JS (WebSocket, tabs, rendering)
+```
+
+## Knowledge Module
+
+### store.ts
+
+CRUD for markdown files with YAML frontmatter:
+
+- **parseFrontmatter()** — splits on `---` delimiters, extracts title/tags/updated
+- **listEntries()** — recursively finds `.md` files, skips dot-directories, filters by category/tag
+- **readEntry()** — reads file with path traversal protection (`path.resolve` must start with base dir)
+- **writeEntry()** — validates category against allowed list, ensures directory exists, auto-adds `.md`
+- **deleteEntry()** — removes file with path traversal protection
+
+### git.ts
+
+Wraps `execSync` for git operations with timeouts:
+
+- `gitPull()` — `git pull --rebase --quiet` (15s timeout)
+- `gitPush()` — `git add -A`, conditional commit (checks `git diff --cached --quiet`), push (5s/5s/15s)
+- `gitSync()` — pull then push, returns both results
+
+### search.ts
+
+Builds a TF-IDF index from all knowledge entries, searches with ranking, falls back to regex for exact phrases.
+
+## Session Module
+
+### parser.ts — Mtime Cache
+
+Before parsing a JSONL file, checks `fs.statSync` for mtime. If unchanged since last parse, returns cached result. This avoids re-parsing large transcript files on every search.
+
+```
+parseSessionFile(path)
+  → statSync(path).mtimeMs
+  → if mtime matches cache → return cached entries
+  → else parse JSONL lines → cache with mtime → return
+```
+
+### search.ts — Global TF-IDF Index
+
+Maintains a single TF-IDF index across all sessions with a 60-second TTL:
+
+```
+getOrBuildIndex(projects)
+  → if cache exists AND age < 60s → return cached index
+  → else scan all sessions → parse (using mtime cache) → index all messages → cache → return
+```
+
+Role filtering happens post-search: the index includes all roles, and results are filtered after scoring.
+
+### scopes.ts
+
+Uses the cached search index from `search.ts` (via `searchSessions`), then post-filters by scope patterns:
+
+| Scope       | Filter                                                          |
+| ----------- | --------------------------------------------------------------- |
+| `errors`    | Regex: Error, Exception, failed, crash, ENOENT, TypeError, etc. |
+| `plans`     | Regex: plan, step, phase, strategy, TODO, architecture, etc.    |
+| `configs`   | Regex: config, .env, .json, tsconfig, docker, etc.              |
+| `tools`     | Role filter: tool_use, tool_result messages only                |
+| `files`     | Regex: src/, .ts, .js, created, modified, deleted, etc.         |
+| `decisions` | Regex: decided, chose, because, tradeoff, opted for, etc.       |
+
+### summary.ts
+
+Extracts session summaries:
+
+- **Topics**: user messages filtered to exclude JSON/tool_result/base64/system-reminders
+- **Tools used**: tool names from tool_use entries
+- **Files modified**: file paths detected via regex in tool_result content
+
+## Search Engine
+
+### tfidf.ts
+
+Self-contained TF-IDF implementation:
+
+**Tokenization**: lowercase → split on `[^a-z0-9]+` → remove ~100 English stopwords
+
+**Scoring**:
+
+```
+TF(t, d)  = count(t in d) / total_terms(d)
+IDF(t)    = log(1 + N / docs_containing(t))
+Score(q, d) = sum(TF(t, d) * IDF(t)) for each term t in query q
+```
+
+The `1 +` in IDF ensures single-document results still get a positive score.
+
+### fuzzy.ts
+
+Levenshtein edit distance with two-row DP (O(n\*m) time, O(m) space). Fuzzy matching uses a sliding window of varying size to find approximate substring matches.
+
+## Dashboard
+
+### dashboard.ts
+
+Single HTTP server handles both REST API and static files:
+
+- **Static serving**: resolves UI directory (checks `src/ui/` then `dist/ui/`), serves with MIME detection and CSP headers
+- **REST API**: routes for knowledge CRUD/search, session list/search/recall/get/summary, health
+- **WebSocket**: `ws` library with `noServer` mode, heartbeat every 30s, initial state snapshot on connect
+- **File watcher**: `fs.watch` on UI directory, debounced 200ms, broadcasts `{type: "reload"}` to all WS clients
+
+### UI Architecture
+
+Vanilla JS SPA (no framework, no build step):
+
+- WebSocket connects on load, handles `state` and `reload` messages
+- 4 tabs with lazy data loading
+- `marked` + `DOMPurify` + `highlight.js` for markdown rendering
+- Theme persisted in `localStorage('agent-knowledge-theme')`
+
+## Caching Strategy
+
+```
+Search Request
+    │
+    ▼
+┌─────────────────────┐
+│ TF-IDF index < 60s? │──Yes──► Search cached index (~40ms)
+└─────────────────────┘
+    │ No
+    ▼
+┌─────────────────────┐
+│ Scan session files  │
+│ Check mtime cache   │──► Parse only changed files
+└─────────────────────┘
+    │
+    ▼
+┌─────────────────────┐
+│ Rebuild index       │──► Cache with 60s TTL (~5s cold)
+└─────────────────────┘
+    │
+    ▼
+  Search new index
+```
+
+## Data Flow
+
+### Session Search
+
+```mermaid
+sequenceDiagram
+    participant C as Claude Code
+    participant S as MCP Server
+    participant I as TF-IDF Index
+    participant P as Parser Cache
+    participant F as File System
+
+    C->>S: knowledge_search({ query })
+    S->>I: search(query)
+    alt Index expired
+        I->>F: List .jsonl files
+        loop Each file
+            alt Mtime changed
+                I->>P: parse(file)
+                P->>F: Read JSONL
+                P-->>I: Parsed entries
+            else Mtime unchanged
+                I->>P: getCached(file)
+                P-->>I: Cached entries
+            end
+        end
+        I->>I: Rebuild index
+    end
+    I-->>S: Ranked results
+    S-->>C: SearchResult[]
+```
+
+### Knowledge Write
+
+```mermaid
+sequenceDiagram
+    participant C as Claude Code
+    participant S as MCP Server
+    participant G as Git
+    participant F as File System
+
+    C->>S: knowledge_write({ category, filename, content })
+    S->>G: git pull --rebase
+    S->>F: Write markdown file
+    S->>G: git add -A && commit && push
+    G-->>S: Push result
+    S-->>C: { path, git status }
+```

package/docs/DASHBOARD.md

@@ -0,0 +1,133 @@
+# Dashboard
+
+The dashboard runs at **http://localhost:3423** and auto-starts with the MCP server.
+
+## Tabs
+
+| Tab           | Purpose                                                         |
+| ------------- | --------------------------------------------------------------- |
+| **Knowledge** | Browse knowledge base entries by category                       |
+| **Search**    | TF-IDF ranked search across session transcripts                 |
+| **Sessions**  | Browse and read session conversation logs                       |
+| **Recall**    | Scoped search (errors, plans, configs, tools, files, decisions) |
+
+## Knowledge Tab
+
+Card grid of knowledge entries. Each card shows:
+
+- Category badge with color: projects (blue), people (purple), decisions (orange), workflows (green), notes (yellow)
+- Title and tag pills
+- Last updated date
+
+Category filter chips at the top: All, Projects, People, Decisions, Workflows, Notes.
+
+Click a card to open the side panel with rendered markdown content.
+
+## Search Tab
+
+Full-text search across all session transcripts.
+
+**Controls:**
+
+- Search input with debounce (300ms)
+- Role filter chips: All, User, Assistant
+- Mode toggle: Ranked (TF-IDF) vs Regex
+
+**Results show:**
+
+- Role badge (user/assistant)
+- Project name
+- Relative timestamp
+- Score bar with numeric value
+- Excerpt with highlighted matching terms
+
+Click a result to open the session in the side panel.
+
+## Sessions Tab
+
+Lists all Claude Code sessions with metadata:
+
+- Project name
+- Git branch
+- Message count
+- Date
+- Preview of first user message
+
+Project filter dropdown at the top.
+
+Click a session to open the side panel with the full conversation rendered as chat bubbles.
+
+## Recall Tab
+
+Scoped search that pre-filters results by category:
+
+| Scope       | What it finds                             |
+| ----------- | ----------------------------------------- |
+| `errors`    | Stack traces, exceptions, failed commands |
+| `plans`     | Architecture, TODOs, implementation steps |
+| `configs`   | Settings, env vars, configuration files   |
+| `tools`     | MCP tool calls, CLI commands              |
+| `files`     | File paths, modifications                 |
+| `decisions` | Trade-offs, rationale, choices            |
+
+Results use the same format as the Search tab.
+
+## Side Panel
+
+- Width: 560px, resizable by dragging the left edge
+- Close: X button or press Escape
+- Knowledge entries: rendered as markdown via marked + DOMPurify + highlight.js
+- Sessions: chat bubbles (user = right/accent, assistant = left/surface)
+
+## Theming
+
+- Toggle: sun/moon button in header
+- Persisted in `localStorage('agent-knowledge-theme')`
+- MD3 design tokens matching agent-comm and agent-tasks dashboards
+- CSS custom properties on `:root`, switched via `data-theme` attribute
+
+## Live Reload
+
+File watcher monitors `src/ui/` for `.html`, `.css`, `.js` changes. On change, broadcasts `{type: "reload"}` via WebSocket. Connected browsers auto-refresh.
+
+## Keyboard Shortcuts
+
+| Shortcut        | Action             |
+| --------------- | ------------------ |
+| `/` or `Ctrl+K` | Focus search input |
+| `Escape`        | Close side panel   |
+
+## REST API
+
+| Method | Endpoint                                | Description      |
+| ------ | --------------------------------------- | ---------------- |
+| GET    | `/api/knowledge`                        | List entries     |
+| GET    | `/api/knowledge/search?q=`              | Search knowledge |
+| GET    | `/api/knowledge/:path`                  | Read entry       |
+| GET    | `/api/sessions`                         | List sessions    |
+| GET    | `/api/sessions/search?q=&role=&ranked=` | Search sessions  |
+| GET    | `/api/sessions/recall?scope=&q=`        | Scoped recall    |
+| GET    | `/api/sessions/:id`                     | Read session     |
+| GET    | `/api/sessions/:id/summary`             | Session summary  |
+| GET    | `/health`                               | Health check     |
+
+## WebSocket
+
+Connects to `ws://localhost:3423` on page load.
+
+**State message** (on connect):
+
+```json
+{
+  "type": "state",
+  "knowledge": [...],
+  "sessions": [...],
+  "stats": { "knowledge_entries": 12, "session_count": 247 }
+}
+```
+
+**Reload message** (on file change):
+
+```json
+{ "type": "reload" }
+```

package/docs/SETUP.md

@@ -0,0 +1,178 @@
+# Setup Guide
+
+## Prerequisites
+
+- **Node.js 20+** (ES modules and TypeScript features)
+- **Git** (for knowledge base sync)
+
+```bash
+node --version   # v20.0.0 or later
+git --version
+```
+
+## Installation
+
+```bash
+git clone https://github.com/keshrath/agent-knowledge.git
+cd agent-knowledge
+npm install
+npm run build
+```
+
+For development with auto-rebuild:
+
+```bash
+npm run dev
+```
+
+## Configuration in Claude Code
+
+### Register the MCP server
+
+```bash
+claude mcp add agent-knowledge -s user \
+  -e KNOWLEDGE_MEMORY_DIR="$HOME/claude-memory" \
+  -- node /path/to/agent-knowledge/dist/index.js
+```
+
+### Permissions
+
+Add to `~/.claude/settings.json`:
+
+```json
+{
+  "permissions": {
+    "allow": ["mcp__agent-knowledge__*"]
+  }
+}
+```
+
+### Verify
+
+```bash
+claude mcp list
+# Should show: agent-knowledge ... Connected
+```
+
+## Environment Variables
+
+| Variable               | Default           | Description                       |
+| ---------------------- | ----------------- | --------------------------------- |
+| `KNOWLEDGE_MEMORY_DIR` | `~/claude-memory` | Path to git-synced knowledge base |
+| `CLAUDE_MEMORY_DIR`    | `~/claude-memory` | Alias (backwards compat)          |
+| `CLAUDE_DIR`           | `~/.claude`       | Claude Code data directory        |
+| `KNOWLEDGE_PORT`       | `3423`            | Dashboard HTTP/WebSocket port     |
+
+Set in your shell profile or pass via MCP config:
+
+```bash
+export KNOWLEDGE_MEMORY_DIR="$HOME/claude-memory"
+```
+
+On Windows (PowerShell):
+
+```powershell
+$env:KNOWLEDGE_MEMORY_DIR = "$env:USERPROFILE\claude-memory"
+```
+
+## Knowledge Base Setup
+
+The knowledge base is a git repository with categorized markdown files.
+
+### Clone existing or create new
+
+```bash
+# Clone existing
+git clone https://your-git-host/claude-memory.git ~/claude-memory
+
+# Or create new
+mkdir -p ~/claude-memory && cd ~/claude-memory && git init
+mkdir projects people decisions workflows notes
+git add . && git commit -m "Initialize knowledge base"
+git remote add origin <your-remote-url>
+git push -u origin main
+```
+
+### Directory structure
+
+```
+~/claude-memory/
+  projects/       # Project context, architecture, tech stacks
+  people/         # Team members, contacts, preferences
+  decisions/      # Architecture decisions, trade-offs, rationale
+  workflows/      # Processes, deployment steps, runbooks
+  notes/          # General notes, research, references
+```
+
+### Entry format
+
+```markdown
+---
+title: My Project
+tags: [typescript, react, postgres]
+updated: 2026-03-25
+---
+
+# My Project
+
+Architecture notes, deployment info, etc.
+```
+
+## Dashboard
+
+Auto-starts with the MCP server at **http://localhost:3423**.
+
+4 tabs: Knowledge, Search, Sessions, Recall. Supports light/dark theme.
+
+Live reload: edit files in `src/ui/` and the browser refreshes automatically.
+
+## Multi-Machine Sync
+
+| Operation                             | Git Action                     | When           |
+| ------------------------------------- | ------------------------------ | -------------- |
+| `knowledge_read`, `knowledge_list`    | `git pull --rebase`            | Before reading |
+| `knowledge_write`, `knowledge_delete` | `git add -A && commit && push` | After writing  |
+| `knowledge_sync`                      | `git pull` then `git push`     | Manual trigger |
+
+Ensure git credentials are configured (SSH key or credential helper):
+
+```bash
+cd ~/claude-memory && git pull && git push  # Should work without prompts
+```
+
+## Troubleshooting
+
+### Port already in use
+
+```bash
+# Find the process
+netstat -ano | findstr :3423   # Windows
+lsof -i :3423                  # Linux/macOS
+
+# Use a different port
+export KNOWLEDGE_PORT=3424
+```
+
+### Git authentication failures
+
+Verify credentials work manually:
+
+```bash
+cd ~/claude-memory && git push
+```
+
+Set up SSH keys or a credential helper if prompted.
+
+### MCP server not connecting
+
+1. Check path points to `dist/index.js` (not `src/index.ts`)
+2. Verify Node.js 20+ is on your PATH
+3. Try running manually: `node /path/to/agent-knowledge/dist/index.js`
+
+### No session results
+
+Verify session transcripts exist:
+
+```bash
+ls ~/.claude/projects/  # Should contain project directories with .jsonl files
+```

package/package.json

@@ -0,0 +1,92 @@
+{
+  "name": "agent-knowledge",
+  "version": "1.0.0",
+  "description": "Cross-session memory and recall for AI agents — git-synced knowledge base, hybrid semantic+TF-IDF search, auto-distillation with secrets scrubbing",
+  "type": "module",
+  "main": "dist/index.js",
+  "bin": {
+    "agent-knowledge": "dist/index.js"
+  },
+  "files": [
+    "dist/",
+    "scripts/",
+    "docs/*.md",
+    "README.md",
+    "CHANGELOG.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "build": "tsc && node scripts/copy-ui.js",
+    "start": "node dist/index.js",
+    "dev": "tsc --watch",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "test:coverage": "vitest run --coverage",
+    "lint": "eslint src/ tests/",
+    "lint:fix": "eslint src/ tests/ --fix",
+    "format": "prettier --write .",
+    "format:check": "prettier --check .",
+    "typecheck": "tsc --noEmit",
+    "check": "npm run typecheck && npm run lint && npm run format:check && npm run test",
+    "clean": "rm -rf dist",
+    "prepare": "husky"
+  },
+  "keywords": [
+    "mcp",
+    "agent",
+    "memory",
+    "recall",
+    "session-search",
+    "knowledge-base",
+    "tfidf",
+    "embeddings",
+    "semantic-search",
+    "model-context-protocol",
+    "ai-agents",
+    "claude-code"
+  ],
+  "author": "keshrath",
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/keshrath/agent-knowledge.git"
+  },
+  "bugs": {
+    "url": "https://github.com/keshrath/agent-knowledge/issues"
+  },
+  "homepage": "https://github.com/keshrath/agent-knowledge#readme",
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.0.0",
+    "better-sqlite3": "^11.0.0",
+    "sqlite-vec": "^0.1.0",
+    "ws": "^8.20.0"
+  },
+  "devDependencies": {
+    "@eslint/js": "^10.0.1",
+    "@types/better-sqlite3": "^7.6.0",
+    "@types/node": "^25.5.0",
+    "@types/ws": "^8.18.1",
+    "eslint": "^10.1.0",
+    "husky": "^9.0.0",
+    "lint-staged": "^16.0.0",
+    "prettier": "^3.8.1",
+    "typescript": "^5.7.0",
+    "typescript-eslint": "^8.57.2",
+    "vitest": "^3.0.0"
+  },
+  "optionalDependencies": {
+    "@huggingface/transformers": "^3.0.0"
+  },
+  "lint-staged": {
+    "*.{ts,js}": [
+      "prettier --write",
+      "eslint --fix"
+    ],
+    "*.{json,md,css,html}": [
+      "prettier --write"
+    ]
+  },
+  "engines": {
+    "node": ">=20.11.0"
+  }
+}

package/scripts/copy-ui.js

@@ -0,0 +1,6 @@
+import { cpSync } from 'fs';
+import { fileURLToPath } from 'url';
+import { dirname, join } from 'path';
+
+const root = join(dirname(fileURLToPath(import.meta.url)), '..');
+cpSync(join(root, 'src', 'ui'), join(root, 'dist', 'ui'), { recursive: true });