@lossless-claude/lcm 0.2.0

package/docs/configuration.md

@@ -0,0 +1,168 @@
+# Configuration guide
+
+## Quick start
+
+Clone the repo and link it with Claude Code's plugin installer:
+
+```bash
+git clone https://github.com/lossless-claude/lcm.git
+cd lcm
+npm install
+npm run build
+claude plugins install --link /path/to/lossless-claude
+```
+
+`claude plugins install --link` handles plugin registration/enabling and slot selection automatically.
+
+Set recommended environment variables:
+
+```bash
+export LCM_FRESH_TAIL_COUNT=32
+export LCM_INCREMENTAL_MAX_DEPTH=-1
+```
+
+Restart Claude Code.
+
+## Tuning guide
+
+### Context threshold
+
+`LCM_CONTEXT_THRESHOLD` (default `0.75`) controls when compaction triggers as a fraction of the model's context window.
+
+- **Lower values** (e.g., 0.5) trigger compaction earlier, keeping context smaller but doing more LLM calls for summarization.
+- **Higher values** (e.g., 0.85) let conversations grow longer before compacting, reducing summarization cost but risking overflow with large model responses.
+
+For most use cases, 0.75 is a good balance.
+
+### Fresh tail count
+
+`LCM_FRESH_TAIL_COUNT` (default `32`) is the number of most recent messages that are never compacted. These raw messages give the model immediate conversational continuity.
+
+- **Smaller values** (e.g., 8–16) save context space for summaries but may lose recent nuance.
+- **Larger values** (e.g., 32–64) give better continuity at the cost of a larger mandatory context floor.
+
+For coding conversations with tool calls (which generate many messages per logical turn), 32 is recommended.
+
+### Leaf fanout
+
+`LCM_LEAF_MIN_FANOUT` (default `8`) is the minimum number of raw messages that must be available outside the fresh tail before a leaf pass runs.
+
+- Lower values create summaries more frequently (more, smaller summaries).
+- Higher values create larger, more comprehensive summaries less often.
+
+### Condensed fanout
+
+`LCM_CONDENSED_MIN_FANOUT` (default `4`) controls how many same-depth summaries accumulate before they're condensed into a higher-level summary.
+
+- Lower values create deeper DAGs with more levels of abstraction.
+- Higher values keep the DAG shallower but with more nodes at each level.
+
+### Incremental max depth
+
+`LCM_INCREMENTAL_MAX_DEPTH` (default `0`) controls whether condensation happens automatically after leaf passes.
+
+- **0** — Only leaf summaries are created incrementally. Condensation only happens during manual `/compact` or overflow.
+- **1** — After each leaf pass, attempt to condense d0 summaries into d1.
+- **2+** — Deeper automatic condensation up to the specified depth.
+- **-1** — Unlimited depth. Condensation cascades as deep as needed after each leaf pass. Recommended for long-running sessions.
+
+### Summary target tokens
+
+`LCM_LEAF_TARGET_TOKENS` (default `1200`) and `LCM_CONDENSED_TARGET_TOKENS` (default `2000`) control the target size of generated summaries.
+
+- Larger targets preserve more detail but consume more context space.
+- Smaller targets are more aggressive, losing detail faster.
+
+The actual summary size depends on the LLM's output; these values are guidelines passed in the prompt's token target instruction.
+
+### Leaf chunk tokens
+
+`LCM_LEAF_CHUNK_TOKENS` (default `20000`) caps the amount of source material per leaf compaction pass.
+
+- Larger chunks create more comprehensive summaries from more material.
+- Smaller chunks create summaries more frequently from less material.
+- This also affects the condensed minimum input threshold (10% of this value).
+
+## Model selection
+
+LCM defaults to `LCM_SUMMARY_PROVIDER=auto`.
+
+- In Claude sessions, `auto` resolves to `claude-process`
+- In Codex sessions, `auto` resolves to `codex-process`
+- If you explicitly set `LCM_SUMMARY_PROVIDER`, that override applies to both CLIs
+
+You can pin a specific summarizer provider and model:
+
+```bash
+# Use a specific provider + model for summarization
+export LCM_SUMMARY_MODEL=anthropic/claude-sonnet-4-20250514
+export LCM_SUMMARY_PROVIDER=anthropic
+```
+
+Valid provider values are:
+
+- `auto`
+- `claude-process`
+- `codex-process`
+- `anthropic`
+- `openai`
+- `disabled`
+
+Using a cheaper or faster model for summarization can reduce costs, but quality matters because poor summaries compound as they are condensed into higher-level nodes.
+
+
+## Database management
+
+The SQLite database lives at `LCM_DATABASE_PATH` (default `~/.claude/lcm.db`). 
+
+### Inspecting the database
+
+```bash
+sqlite3 ~/.claude/lcm.db
+
+# Count conversations
+SELECT COUNT(*) FROM conversations;
+
+# See context items for a conversation
+SELECT * FROM context_items WHERE conversation_id = 1 ORDER BY ordinal;
+
+# Check summary depth distribution
+SELECT depth, COUNT(*) FROM summaries GROUP BY depth;
+
+# Find large summaries
+SELECT summary_id, depth, token_count FROM summaries ORDER BY token_count DESC LIMIT 10;
+```
+
+### Backup
+
+The database is a single file. Back it up with:
+
+```bash
+cp ~/.claude/lcm.db ~/.claude/lcm.db.backup
+```
+
+Or use SQLite's online backup:
+
+```bash
+sqlite3 ~/.claude/lcm.db ".backup ~/.claude/lcm.db.backup"
+```
+
+## Per-agent configuration
+
+In multi-agent Claude Code setups, each agent uses the same LCM database but has its own conversations (keyed by session ID). The plugin config applies globally; per-agent overrides use environment variables set in the agent's config.
+
+## Disabling LCM
+
+To fall back to Claude Code's built-in compaction:
+
+```json
+{
+  "plugins": {
+    "slots": {
+      "contextEngine": "legacy"
+    }
+  }
+}
+```
+
+Or set `LCM_ENABLED=false` to disable the plugin while keeping it registered.

package/docs/fts5.md

@@ -0,0 +1,161 @@
+# Optional: enable FTS5 for fast full-text search
+
+`lossless-claude` works without FTS5 as of the current release. When FTS5 is unavailable in the
+Node runtime that runs the Claude Code gateway, the plugin:
+
+- keeps persisting messages and summaries
+- falls back from `"full_text"` search to a slower `LIKE`-based search
+- loses FTS ranking/snippet quality
+
+If you want native FTS5 search performance and ranking, the **exact Node runtime that runs the
+gateway** must have SQLite FTS5 compiled in.
+
+## Probe the gateway runtime
+
+Run this with the same `node` binary your gateway uses:
+
+```bash
+node --input-type=module - <<'NODE'
+import { DatabaseSync } from 'node:sqlite';
+const db = new DatabaseSync(':memory:');
+const options = db.prepare('pragma compile_options').all().map((row) => row.compile_options);
+
+console.log(options.filter((value) => value.includes('FTS')).join('\n') || 'no fts compile options');
+
+try {
+  db.exec("CREATE VIRTUAL TABLE t USING fts5(content)");
+  console.log("fts5: ok");
+} catch (err) {
+  console.log("fts5: fail");
+  console.log(err instanceof Error ? err.message : String(err));
+}
+NODE
+```
+
+Expected output:
+
+```text
+ENABLE_FTS5
+fts5: ok
+```
+
+If you get `fts5: fail`, build or install an FTS5-capable Node and point the gateway at that runtime.
+
+## Build an FTS5-capable Node on macOS
+
+This workflow was verified with Node `v22.15.0`.
+
+```bash
+cd ~/Projects
+git clone --depth 1 --branch v22.15.0 https://github.com/nodejs/node.git node-fts5
+cd node-fts5
+```
+
+Edit `deps/sqlite/sqlite.gyp` and add `SQLITE_ENABLE_FTS5` to the `defines` list for the `sqlite`
+target:
+
+```diff
+ 'defines': [
+   'SQLITE_DEFAULT_MEMSTATUS=0',
++  'SQLITE_ENABLE_FTS5',
+   'SQLITE_ENABLE_MATH_FUNCTIONS',
+   'SQLITE_ENABLE_SESSION',
+   'SQLITE_ENABLE_PREUPDATE_HOOK'
+ ],
+```
+
+Important:
+
+- patch `deps/sqlite/sqlite.gyp`, not only `node.gyp`
+- `node:sqlite` uses the embedded SQLite built from `deps/sqlite/sqlite.gyp`
+
+Build the runtime:
+
+```bash
+./configure --prefix="$PWD/out-install"
+make -j8 node
+```
+
+Expose the binary under a Node-compatible basename that Claude Code recognizes:
+
+```bash
+mkdir -p ~/Projects/node-fts5/bin
+ln -sfn ~/Projects/node-fts5/out/Release/node ~/Projects/node-fts5/bin/node-22.15.0
+```
+
+Use a basename like `node-22.15.0`, `node`, or `nodejs`. Names like
+`node-v22.15.0-fts5` may not be recognized correctly by Claude Code's CLI/runtime parsing.
+
+Verify the new runtime:
+
+```bash
+~/Projects/node-fts5/bin/node-22.15.0 --version
+~/Projects/node-fts5/bin/node-22.15.0 --input-type=module - <<'NODE'
+import { DatabaseSync } from 'node:sqlite';
+const db = new DatabaseSync(':memory:');
+db.exec("CREATE VIRTUAL TABLE t USING fts5(content)");
+console.log("fts5: ok");
+NODE
+```
+
+## Point the Claude Code gateway at that runtime on macOS
+
+Back up the existing LaunchAgent plist first:
+
+```bash
+cp ~/Library/LaunchAgents/ai.claude.gateway.plist \
+  ~/Library/LaunchAgents/ai.claude.gateway.plist.bak-$(date +%Y%m%d-%H%M%S)
+```
+
+Replace the runtime path, then reload the agent:
+
+```bash
+/usr/libexec/PlistBuddy -c 'Set :ProgramArguments:0 /Users/youruser/Projects/node-fts5/bin/node-22.15.0' \
+  ~/Library/LaunchAgents/ai.claude.gateway.plist
+
+launchctl bootout gui/$UID ~/Library/LaunchAgents/ai.claude.gateway.plist 2>/dev/null || true
+launchctl bootstrap gui/$UID ~/Library/LaunchAgents/ai.claude.gateway.plist
+launchctl kickstart -k gui/$UID/ai.claude.gateway
+```
+
+Verify the live runtime:
+
+```bash
+launchctl print gui/$UID/ai.claude.gateway | sed -n '1,80p'
+```
+
+You should see:
+
+```text
+program = /Users/youruser/Projects/node-fts5/bin/node-22.15.0
+```
+
+## Verify `lossless-claude`
+
+Check the logs:
+
+```bash
+tail -n 60 ~/.claude/logs/gateway.log
+tail -n 60 ~/.claude/logs/gateway.err.log
+```
+
+You want:
+
+- `[gateway] [lcm] Plugin loaded ...`
+- no new `no such module: fts5`
+
+Then force one turn through the gateway and verify the DB fills:
+
+```bash
+/Users/youruser/Projects/node-fts5/bin/node-22.15.0 \
+  /path/to/claude/dist/index.js \
+  agent --session-id fts5-smoke --message 'Reply with exactly: ok' --timeout 60
+
+sqlite3 ~/.claude/lcm.db '
+  select count(*) as conversations from conversations;
+  select count(*) as messages from messages;
+  select count(*) as summaries from summaries;
+'
+```
+
+Those counts should increase after a real turn.

package/docs/hook-protocol.md

@@ -0,0 +1,41 @@
+# Claude Code Hook Protocol — Verified Schemas
+
+## PreCompact Hook
+
+**Status:** ⚠️ Pending verification via probe-precompact.ts
+
+Expected fields (unverified):
+- `session_id` — session identifier
+- `transcript_path` — path to session JSONL transcript (C1: unverified)
+- `cwd` — working directory
+- `hook_event_name` — "PreCompact"
+
+### How to verify (C1)
+
+1. Add to `~/.claude/settings.json` hooks:
+   ```json
+   "PreCompact": [{ "type": "command", "command": "node /path/to/src/hooks/probe-precompact.ts" }]
+   ```
+2. Start Claude Code, trigger compaction (use `/compact` command or fill context)
+3. Check `~/.lossless-claude/precompact-probe.json`
+4. Update this file with verified schema
+
+## SessionStart Hook
+
+**Status:** ⚠️ Pending verification via probe-sessionstart.ts
+
+Expected fields:
+- `session_id`
+- `cwd`
+- `hook_event_name` — "SessionStart"
+- `source` — "startup" | "resume" | "compact" (C3: verify "compact" exists)
+
+### How to verify (C3)
+
+1. Add to `~/.claude/settings.json` hooks:
+   ```json
+   "SessionStart": [{ "type": "command", "command": "node /path/to/src/hooks/probe-sessionstart.ts" }]
+   ```
+2. Trigger compaction, then observe the next SessionStart event
+3. Check `~/.lossless-claude/sessionstart-probe.jsonl` for `source` values
+4. Update this file with verified schema

package/docs/privacy.md

@@ -0,0 +1,101 @@
+# Privacy & Data Handling
+
+lossless-claude stores your conversation history locally to enable memory across sessions. This document explains exactly what is stored, what leaves your machine, and how to control sensitive data.
+
+## What is stored locally
+
+All storage is on your machine:
+
+- **`~/.lossless-claude/projects/{hash}/db.sqlite`** — Conversation messages, summaries, and promoted long-term memory for each project. The hash is a SHA-256 of the project directory path.
+- **`~/.lossless-claude/projects/{hash}/sensitive-patterns.txt`** — Per-project sensitive patterns (if configured).
+- **`~/.lossless-claude/config.json`** — Global configuration including the optional `security.sensitivePatterns` array.
+- **`~/.lossless-claude/daemon.pid`** — Daemon process ID (transient).
+
+No data is sent to any lossless-claude server. There is no telemetry.
+
+## What leaves your machine
+
+lossless-claude is a local runtime. By default, **nothing leaves your machine**.
+
+The exception is the summarizer, which you configure explicitly:
+
+| Summarizer (`llm.provider`) | Data sent externally |
+|-----------------------------|----------------------|
+| `disabled` (default) | Nothing |
+| `claude-process` | Messages sent to Anthropic via the `claude` CLI (your Claude subscription) |
+| `codex-process` | Messages sent to OpenAI via the `codex` CLI (your OpenAI subscription) |
+| `anthropic` | Messages sent to Anthropic API (your API key) |
+| `openai` | Messages sent to OpenAI API (your API key) |
+
+When using an external summarizer, only the text being summarized is sent — not your full history. The summarizer receives a batch of recent messages to compress into a summary.
+
+## Secret redaction
+
+lossless-claude scrubs secrets from message content **before writing to SQLite** and **before sending to the summarizer**. Redaction happens at both write points to ensure secrets are never persisted or transmitted.
+
+### Built-in patterns
+
+These patterns are always active, regardless of configuration:
+
+| Pattern | Example match |
+|---------|--------------|
+| OpenAI secret key | `sk-...` |
+| Anthropic API key | `sk-ant-...` |
+| GitHub personal access token | `ghp_...` |
+| AWS access key ID | `AKIA...` |
+| PEM private key | `-----BEGIN ... KEY-----` |
+| Bearer token | `Authorization: Bearer ...` |
+| Password assignment | `password=...`, `PASSWORD: ...` |
+
+### Project-specific patterns
+
+Add patterns for secrets specific to your project:
+
+```bash
+# Add a pattern (stored in ~/.lossless-claude/projects/{hash}/sensitive-patterns.txt)
+lcm sensitive add "MY_APP_API_KEY_[A-Z0-9]+"
+
+# Add a global pattern (applies to all projects, stored in config.json)
+lcm sensitive add --global "CORP_INTERNAL_TOKEN"
+
+# Test what gets redacted
+lcm sensitive test "token=MY_APP_API_KEY_ABCDEF123"
+# → token=[REDACTED]
+
+# List all active patterns
+lcm sensitive list
+```
+
+Patterns are JavaScript-compatible regular expressions. Use specific patterns (e.g., `MY_SECRET_[A-Z0-9]+`) rather than broad ones (e.g., `MY_.*`) to avoid over-redaction.
+
+## Data retention
+
+Messages and summaries persist until you explicitly remove them:
+
+```bash
+# Remove data for the current project
+lcm sensitive purge --yes
+
+# Remove all lossless-claude data
+lcm uninstall
+```
+
+SQLite database files are stored in `~/.lossless-claude/projects/`. You can delete individual project directories manually to remove their history.
+
+## Verifying your setup
+
+```bash
+lcm doctor
+```
+
+The `Security` section of the doctor output shows:
+- How many built-in patterns are active
+- Whether project-specific patterns are configured
+
+## Summary
+
+- All data is local — SQLite in `~/.lossless-claude/`.
+- External summarizer (optional) receives only the text to be summarized, after scrubbing.
+- Built-in patterns redact common secret formats automatically.
+- Add project-specific patterns with `lcm sensitive add`.
+- Delete your data with `lcm uninstall` or by removing `~/.lossless-claude/`.

package/mcp.mjs

@@ -0,0 +1,17 @@
+#!/usr/bin/env node
+// MCP entrypoint for plugin system — delegates to the built lcm MCP server.
+// Uses import.meta.url to resolve paths relative to this file, so it works
+// regardless of how the plugin cache resolves ${CLAUDE_PLUGIN_ROOT}.
+import { fileURLToPath, pathToFileURL } from "node:url";
+import { join, dirname } from "node:path";
+
+let __dirname = dirname(fileURLToPath(import.meta.url));
+// If we're in .claude-plugin/, go up one level to find dist/
+if (__dirname.endsWith(".claude-plugin")) {
+  __dirname = join(__dirname, "..");
+}
+const serverModule = join(__dirname, "dist", "src", "mcp", "server.js");
+
+// Use file:// URL for cross-platform compatibility (notably Windows)
+const { startMcpServer } = await import(pathToFileURL(serverModule).href);
+await startMcpServer();

package/package.json

@@ -0,0 +1,79 @@
+{
+  "name": "@lossless-claude/lcm",
+  "version": "0.2.0",
+  "description": "Never lose context again. lossless-claude compresses Claude Code sessions into searchable memory — every message preserved, every insight remembered.",
+  "type": "module",
+  "main": "dist/src/memory/index.js",
+  "exports": {
+    ".": "./dist/src/memory/index.js"
+  },
+  "bin": {
+    "lcm": "dist/bin/lcm.js"
+  },
+  "license": "MIT",
+  "author": "Pedro Almeida (https://github.com/ipedro)",
+  "keywords": [
+    "claude-code",
+    "claude-code-plugin",
+    "context-management",
+    "llm",
+    "summarization",
+    "conversation-memory",
+    "dag"
+  ],
+  "files": [
+    "dist/",
+    "mcp.mjs",
+    ".claude-plugin/",
+    "docs/",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "build": "tsc",
+    "postbuild": "mkdir -p dist/src/prompts && cp -r src/prompts/*.yaml dist/src/prompts/ && mkdir -p dist/src/connectors/templates && cp -r src/connectors/templates/. dist/src/connectors/templates/",
+    "prepublishOnly": "npm run build",
+    "changeset": "changeset",
+    "release:verify": "npm test && npm pack --dry-run",
+    "test": "vitest run --dir test",
+    "typecheck": "tsc --noEmit",
+    "version-packages": "changeset version"
+  },
+  "dependencies": {
+    "@modelcontextprotocol/sdk": "^1.12.0",
+    "@sinclair/typebox": "0.34.48",
+    "js-yaml": "^4.1.1"
+  },
+  "devDependencies": {
+    "@anthropic-ai/sdk": "^0.39.0",
+    "@changesets/cli": "^2.30.0",
+    "@types/js-yaml": "^4.0.9",
+    "@types/node": ">=22",
+    "openai": "^6.32.0",
+    "typescript": "^5.7.0",
+    "vitest": "^3.0.0"
+  },
+  "peerDependencies": {
+    "@anthropic-ai/sdk": "^0.39.0",
+    "openai": "^6.32.0"
+  },
+  "peerDependenciesMeta": {
+    "@anthropic-ai/sdk": {
+      "optional": true
+    },
+    "openai": {
+      "optional": true
+    }
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/lossless-claude/lcm.git"
+  },
+  "homepage": "https://github.com/lossless-claude/lcm#readme",
+  "bugs": {
+    "url": "https://github.com/lossless-claude/lcm/issues"
+  }
+}