@a13xu/lucid 1.1.0 → 1.9.0

package/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2026 a13xu
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2026 a13xu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,99 +1,221 @@
-# @a13xu/lucid
-
-Persistent memory for Claude Code agents, backed by **SQLite + FTS5**.
-
-Stores a knowledge graph (entities, relations, observations) with full-text search — indexed queries under 1ms, no JSON files, no linear scans.
-
-## Install
-
-**Requirements:** Node.js 18+
-
-```bash
-# Run directly (no install needed)
-npx @a13xu/lucid
-
-# Or install globally
-npm install -g @a13xu/lucid
-lucid
-```
-
-### Add to Claude Code
-
-```bash
-claude mcp add --transport stdio lucid -- npx -y @a13xu/lucid
-```
-
-Or add manually to `.mcp.json` in your project root:
-
-```json
-{
-  "mcpServers": {
-    "lucid": {
-      "type": "stdio",
-      "command": "npx",
-      "args": ["-y", "@a13xu/lucid"],
-      "env": {
-        "MEMORY_DB_PATH": "/your/project/.claude/memory.db"
-      }
-    }
-  }
-}
-```
-
-Default DB path: `~/.claude/memory.db`
-
-## Why SQLite + FTS5 instead of JSON?
-
-| | JSON file | SQLite + FTS5 |
-|---|---|---|
-| Search | O(n) linear scan | O(log n) indexed |
-| Write | Rewrite entire file | Atomic incremental |
-| Concurrent reads | Lock entire file | WAL mode |
-| Stemming / unicode | Manual | Built-in |
-
-## Tools (6)
-
-| Tool | Description |
-|---|---|
-| `remember` | Store a fact about an entity (project, person, tool, decision…) |
-| `relate` | Create a directed relationship between two entities |
-| `recall` | Full-text search across all memory (FTS5 + LIKE fallback) |
-| `recall_all` | Return the entire knowledge graph with stats |
-| `forget` | Remove an entity and all its relations |
-| `memory_stats` | DB size, WAL status, entity/relation counts |
-
-### Entity types
-`person` · `project` · `decision` · `pattern` · `tool` · `config` · `bug` · `convention`
-
-### Relation types
-`uses` · `depends_on` · `created_by` · `part_of` · `replaced_by` · `conflicts_with` · `tested_by`
-
-## Usage examples
-
-```
-You:    "Remember that this project uses PostgreSQL with Prisma ORM"
-Claude: [calls remember] → Created "PostgreSQL" [tool]
-
-You:    "What do you know about the database?"
-Claude: [calls recall("database PostgreSQL")] → returns entity + observations
-
-You:    "How are the services connected?"
-Claude: [calls recall_all] → returns full knowledge graph
-```
-
-## Debugging
-
-```bash
-echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"capabilities":{},"clientInfo":{"name":"test","version":"1.0"},"protocolVersion":"2024-11-05"}}' \
-  | npx @a13xu/lucid
-```
-
-In Claude Code: run `/mcp` — you should see `lucid` listed with 6 tools.
-
-## Tech stack
-
-- **Runtime:** Node.js 18+, TypeScript, ES modules
-- **MCP SDK:** `@modelcontextprotocol/sdk`
-- **Database:** `better-sqlite3` (synchronous, no async overhead)
-- **Validation:** `zod`
-- **Transport:** stdio
+# @a13xu/lucid
+
+Token-efficient memory, code indexing, and validation for Claude Code agents — backed by **SQLite + FTS5**.
+
+Stores a persistent knowledge graph (entities, relations, observations), indexes source files as compressed binary with change detection, retrieves minimal relevant context via TF-IDF or Qdrant, and validates code for LLM drift patterns.
+
+## Install
+
+**Requirements:** Node.js 18+
+
+```bash
+# Add to Claude Code (no install needed)
+claude mcp add --transport stdio lucid -- npx -y @a13xu/lucid
+```
+
+Or add to `.mcp.json` in your project root:
+
+```json
+{
+  "mcpServers": {
+    "lucid": {
+      "type": "stdio",
+      "command": "npx",
+      "args": ["-y", "@a13xu/lucid"],
+      "env": {
+        "MEMORY_DB_PATH": "/your/project/.claude/memory.db"
+      }
+    }
+  }
+}
+```
+
+Default DB path: `~/.claude/memory.db`
+
+## Quick start
+
+```
+1. "Index this project" → init_project()        → scans CLAUDE.md, package.json, src/**
+2. Write code          → sync_file(path)         → compressed + hashed + diff stored
+3. "What's relevant?"  → get_context("auth flow") → TF-IDF ranked skeletons, ~500 tokens
+4. "What changed?"     → get_recent(hours=2)      → line diffs of recent edits
+5. "Where is X used?"  → grep_code("X")           → matching lines only, ~30 tokens
+6. "What do we know?"  → recall("query")          → knowledge graph search
+```
+
+## Tools (15)
+
+### Memory
+| Tool | Description |
+|---|---|
+| `remember` | Store a fact about an entity (project, person, tool, decision…) |
+| `relate` | Create a directed relationship between two entities |
+| `recall` | Full-text search across all memory (FTS5 + LIKE fallback) |
+| `recall_all` | Return the entire knowledge graph with statistics |
+| `forget` | Remove an entity and all its relations |
+| `memory_stats` | DB size, WAL status, entity/relation counts |
+
+### Code indexing
+| Tool | Description |
+|---|---|
+| `init_project` | Scan project directory recursively and bootstrap knowledge graph. Reads `CLAUDE.md`, `package.json`/`pyproject.toml`, `README.md`, `.mcp.json`, `logic-guardian.yaml`, all source files. Installs a Claude Code hook for auto-sync. |
+| `sync_file` | Index or re-index a single file after writing/editing. Stores compressed binary (zlib-9), skips instantly if SHA-256 hash unchanged. Stores line-level diff from previous version. |
+| `sync_project` | Re-index entire project incrementally. Reports compression ratio. |
+| `grep_code` | Regex search across all indexed files. Decompresses binary on-the-fly, returns only matching lines with context — ~20-50 tokens vs reading full files. |
+
+### Token optimization
+| Tool | Description |
+|---|---|
+| `get_context` | **Smart context retrieval.** Ranks all indexed files by TF-IDF relevance (or Qdrant vector search if `QDRANT_URL` is set), applies recency boost, returns skeletons (imports + signatures only) for large files. Respects `maxContextTokens` budget. |
+| `get_recent` | Return files modified in the last N hours with line-level diffs. |
+
+### Logic Guardian
+| Tool | Description |
+|---|---|
+| `validate_file` | Detect LLM drift patterns in a source file: logic inversions, null propagation, type confusion, copy-paste drift, silent exceptions. Supports Python, JS, TS. |
+| `check_drift` | Analyze a code snippet inline without saving to disk. |
+| `get_checklist` | Return the full 5-pass validation protocol (Logic Trace, Contract Verification, Stupid Mistakes, Integration Sanity, Explain It). |
+
+## Token optimization in depth
+
+### How `get_context` works
+
+```
+query: "auth middleware"
+         ↓
+  1. TF-IDF score all indexed files against query
+     (or Qdrant top-k if QDRANT_URL is set)
+         ↓
+  2. Boost recently-modified files (+0.3 score)
+         ↓
+  3. Apply whitelist dirs filter (if configured)
+         ↓
+  4. For each file within token budget:
+       file < maxTokensPerFile  → return full source
+       file > maxTokensPerFile  → return skeleton only
+                                   (imports + signatures + TODOs)
+                                   + relevant fragments around query terms
+         ↓
+  output: ~500–2000 tokens  vs  5000–20000 for reading full files
+```
+
+### Skeleton pruning (AST-based)
+
+Large files are replaced with their structural skeleton:
+
+```typescript
+// src/middleware/auth.ts [skeleton]
+// Validates JWT tokens and attaches user to request context
+
+import { Request, Response, NextFunction } from "express"
+import { verifyToken } from "../services/jwt.js"
+
+// — exports —
+export function authMiddleware(req: Request, res: Response, next: NextFunction): void { … }
+export function requireRole(role: string): RequestHandler { … }
+export type AuthenticatedRequest = Request & { user: User }
+```
+
+vs reading the full 200-line file.
+
+### Qdrant vector search (optional)
+
+Set env vars to enable semantic search instead of TF-IDF:
+
+```bash
+QDRANT_URL=http://localhost:6333
+QDRANT_API_KEY=your-key          # optional
+OPENAI_API_KEY=sk-...            # for embeddings
+EMBEDDING_MODEL=text-embedding-3-small  # optional
+```
+
+Or in `.mcp.json`:
+```json
+{
+  "mcpServers": {
+    "lucid": {
+      "command": "npx", "args": ["-y", "@a13xu/lucid"],
+      "env": {
+        "QDRANT_URL": "http://localhost:6333",
+        "OPENAI_API_KEY": "sk-..."
+      }
+    }
+  }
+}
+```
+
+Falls back to TF-IDF automatically if Qdrant is unreachable.
+
+### Configuration (`lucid.config.json`)
+
+Create in your project root to customize behavior:
+
+```json
+{
+  "whitelistDirs": ["src", "backend", "api"],
+  "blacklistDirs": ["migrations", "fixtures"],
+  "maxTokensPerFile": 400,
+  "maxContextTokens": 6000,
+  "recentWindowHours": 12
+}
+```
+
+| Key | Default | Description |
+|---|---|---|
+| `whitelistDirs` | — | Only index/return files from these dirs |
+| `blacklistDirs` | — | Extra dirs to skip (merged with built-in skips) |
+| `maxTokensPerFile` | `400` | Files above this get skeleton treatment |
+| `maxContextTokens` | `4000` | Total token budget for `get_context` |
+| `recentWindowHours` | `24` | "Recently touched" threshold |
+
+## Why no vectors by default?
+
+Code has explicit structure — no NLP needed for most queries:
+
+| Need | Approach | Tokens |
+|---|---|---|
+| "Where is X defined?" | `grep_code("export.*X")` | ~30 |
+| "What does auth.ts export?" | `recall("auth.ts")` | ~50 |
+| "What changed recently?" | `get_recent(hours=2)` | ~200 |
+| "Context for this task" | `get_context("auth flow")` | ~500 |
+| "Project conventions?" | `recall("CLAUDE.md conventions")` | ~80 |
+| Read full file | `Read tool` | ~500–2000 |
+
+TF-IDF is fast, deterministic, and requires zero external services. Qdrant is available when you need semantic similarity across large codebases.
+
+## Why SQLite + FTS5?
+
+| | JSON file | SQLite + FTS5 |
+|---|---|---|
+| Search | O(n) linear scan | O(log n) indexed |
+| Write | Rewrite entire file | Atomic incremental |
+| Concurrent reads | Lock entire file | WAL mode |
+| Code storage | Plain text | Compressed BLOB + hash |
+| Change detection | Manual diff | SHA-256 per file |
+| Diff history | None | Line-level diffs per file |
+
+## Entity types
+`person` · `project` · `decision` · `pattern` · `tool` · `config` · `bug` · `convention`
+
+## Relation types
+`uses` · `depends_on` · `created_by` · `part_of` · `replaced_by` · `conflicts_with` · `tested_by`
+
+## Debugging
+
+```bash
+echo '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"capabilities":{},"clientInfo":{"name":"test","version":"1.0"},"protocolVersion":"2024-11-05"}}' \
+  | npx @a13xu/lucid
+```
+
+In Claude Code: run `/mcp` — you should see `lucid` with 15 tools.
+
+## Tech stack
+
+- **Runtime:** Node.js 18+, TypeScript, ES modules
+- **MCP SDK:** `@modelcontextprotocol/sdk`
+- **Database:** `better-sqlite3` (synchronous, WAL mode)
+- **Compression:** Node.js built-in `zlib` (deflate level 9)
+- **Hashing:** SHA-256 via `crypto` (change detection)
+- **Ranking:** TF-IDF (built-in) or Qdrant (optional, via REST)
+- **Validation:** `zod`
+- **Transport:** stdio

package/build/config.d.ts

@@ -0,0 +1,37 @@
+import type { SecurityConfig } from "./security/guard.js";
+export interface LucidConfig {
+    /** Only index/return files from these directories (e.g. ["src", "lib", "backend"]) */
+    whitelistDirs?: string[];
+    /** Additional dirs to skip (merged with built-in SKIP_DIRS) */
+    blacklistDirs?: string[];
+    /** Max estimated tokens to return per file in get_context (default 400) */
+    maxTokensPerFile?: number;
+    /** Total token budget for get_context response (default 4000) */
+    maxContextTokens?: number;
+    /** "Recently touched" = modified within N hours (default 24) */
+    recentWindowHours?: number;
+    /** Security guard configuration */
+    security?: SecurityConfig;
+    /** Optional Qdrant vector search (falls back to TF-IDF if not configured) */
+    qdrant?: {
+        url: string;
+        apiKey?: string;
+        collection?: string;
+        /** Embedding endpoint — must be OpenAI-compatible */
+        embeddingUrl?: string;
+        embeddingApiKey?: string;
+        embeddingModel?: string;
+        vectorDim?: number;
+    };
+}
+export interface Defaults {
+    maxTokensPerFile: number;
+    maxContextTokens: number;
+    recentWindowHours: number;
+}
+export declare const DEFAULTS: Defaults;
+export type ResolvedConfig = Defaults & LucidConfig;
+export declare function loadConfig(projectDir?: string): ResolvedConfig;
+/** Qdrant config from lucid.config.json or env vars */
+export declare function getQdrantConfig(cfg: ResolvedConfig): ResolvedConfig["qdrant"] | null;
+export declare function resetConfigCache(): void;

package/build/config.js

@@ -0,0 +1,45 @@
+import { existsSync, readFileSync } from "fs";
+import { join } from "path";
+export const DEFAULTS = {
+    maxTokensPerFile: 400,
+    maxContextTokens: 4000,
+    recentWindowHours: 24,
+};
+let _cached = null;
+export function loadConfig(projectDir) {
+    if (_cached)
+        return _cached;
+    const dirs = [projectDir, process.cwd()].filter(Boolean);
+    for (const dir of dirs) {
+        const cfgPath = join(dir, "lucid.config.json");
+        if (existsSync(cfgPath)) {
+            try {
+                const raw = JSON.parse(readFileSync(cfgPath, "utf-8"));
+                _cached = { ...DEFAULTS, ...raw };
+                return _cached;
+            }
+            catch { /* malformed — skip */ }
+        }
+    }
+    _cached = { ...DEFAULTS };
+    return _cached;
+}
+/** Qdrant config from lucid.config.json or env vars */
+export function getQdrantConfig(cfg) {
+    // Env vars override config file
+    const url = process.env["QDRANT_URL"] ?? cfg.qdrant?.url;
+    if (!url)
+        return null;
+    return {
+        url,
+        apiKey: process.env["QDRANT_API_KEY"] ?? cfg.qdrant?.apiKey,
+        collection: process.env["QDRANT_COLLECTION"] ?? cfg.qdrant?.collection ?? "lucid",
+        embeddingUrl: process.env["EMBEDDING_URL"] ?? cfg.qdrant?.embeddingUrl ?? "https://api.openai.com/v1/embeddings",
+        embeddingApiKey: process.env["EMBEDDING_API_KEY"] ?? process.env["OPENAI_API_KEY"] ?? cfg.qdrant?.embeddingApiKey,
+        embeddingModel: process.env["EMBEDDING_MODEL"] ?? cfg.qdrant?.embeddingModel ?? "text-embedding-3-small",
+        vectorDim: cfg.qdrant?.vectorDim ?? 1536,
+    };
+}
+export function resetConfigCache() {
+    _cached = null;
+}

package/build/database.d.ts

@@ -3,7 +3,53 @@ import type { EntityRow, RelationRow } from "./types.js";
 export declare function initDatabase(): Database.Database;
 type Stmt<P extends unknown[], R> = Database.Statement<P, R>;
 type WriteStmt<P extends unknown[]> = Database.Statement<P, unknown>;
+export interface FileContentRow {
+    id: number;
+    filepath: string;
+    content: Buffer;
+    content_hash: string;
+    original_size: number;
+    compressed_size: number;
+    language: string;
+    indexed_at: number;
+}
+export interface FileDiffRow {
+    filepath: string;
+    prev_hash: string;
+    diff_text: string;
+    changed_at: number;
+}
+export interface ExperienceRow {
+    id: number;
+    query: string;
+    query_terms: string;
+    context_fps: string;
+    strategy: string;
+    reward: number;
+    feedback: string | null;
+    created_at: number;
+    rewarded_at: number | null;
+}
+export interface FileRewardRow {
+    filepath: string;
+    total_reward: number;
+    use_count: number;
+    last_rewarded: number | null;
+}
 export interface Statements {
+    getFileByPath: Stmt<[string], FileContentRow>;
+    upsertFile: WriteStmt<[string, Buffer, string, number, number, string]>;
+    getAllFiles: Stmt<[], Pick<FileContentRow, "filepath" | "content" | "language" | "content_hash" | "indexed_at">>;
+    getRecentFiles: Stmt<[number], Pick<FileContentRow, "filepath" | "language" | "indexed_at">>;
+    deleteFile: WriteStmt<[string]>;
+    fileStorageStats: Stmt<[], {
+        count: number;
+        total_original: number;
+        total_compressed: number;
+    }>;
+    upsertDiff: WriteStmt<[string, string, string]>;
+    getDiff: Stmt<[string], FileDiffRow>;
+    getRecentDiffs: Stmt<[number], FileDiffRow>;
     getEntityByName: Stmt<[string], EntityRow>;
     insertEntity: WriteStmt<[string, string, string]>;
     updateEntity: WriteStmt<[string, number]>;
@@ -29,6 +75,14 @@ export interface Statements {
         from_name: string;
         to_name: string;
     }>;
+    insertExperience: WriteStmt<[string, string, string, string]>;
+    updateExperienceReward: WriteStmt<[number, string | null, number]>;
+    getExperienceById: Stmt<[number], ExperienceRow>;
+    getTopExperiences: Stmt<[number], ExperienceRow>;
+    searchExperiencesFTS: Stmt<[string, number], ExperienceRow>;
+    upsertFileReward: WriteStmt<[string, number]>;
+    getFileRewards: Stmt<[], FileRewardRow>;
+    getTopFileRewards: Stmt<[number], FileRewardRow>;
 }
 export declare function prepareStatements(db: Database.Database): Statements;
 export {};