compendium-mcp 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Compendium contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,295 @@
+# Compendium
+
+MCP server that **minimizes LLM token usage** by compressing, summarizing, filtering, and chunk-referencing large context before it reaches the model.
+
+Built in Rust with the official [`rmcp`](https://crates.io/crates/rmcp) SDK.
+
+## Install via npm / npx
+
+Package name: **`compendium-mcp`** (the npm name `compendium` is taken). CLI bin: `compendium`.
+
+```bash
+npx -y compendium-mcp --help
+# or: npm install -g compendium-mcp
+```
+
+**Cursor / Claude Desktop**
+
+```json
+{
+  "mcpServers": {
+    "compendium": {
+      "command": "npx",
+      "args": ["-y", "compendium-mcp"]
+    }
+  }
+}
+```
+
+Distribution uses **optional platform packages** (`compendium-mcp-darwin-arm64`, …) with a **GitHub Releases download fallback**. See [npm/DISTRIBUTION.md](npm/DISTRIBUTION.md) for local testing, CI release, and publishing.
+
+## Transports
+
+| Mode | Command | Notes |
+|------|---------|-------|
+| **stdio** (default) | `compendium` / `compendium stdio` | Cursor / Claude Desktop |
+| **Streamable HTTP/SSE** | `compendium http [BIND]` | Requires `--features http`. Endpoint: `http://{bind}/mcp` |
+
+Default HTTP bind: `127.0.0.1:8788` (override with arg or `COMPENDIUM_HTTP_BIND`).
+
+## Tools
+
+Single MCP tool: **`compendium`**. Choose the operation with `action`:
+
+| `action` | Purpose | Main fields |
+|----------|---------|-------------|
+| `filter` | Strip ANSI, boilerplate, whitespace; densify JSON; keep/drop regexes | `text`, `filter` |
+| `compress` | Dense representation of text/code/logs | `text`, `compress` |
+| `compress_output` | Domain-aware stdout/stderr scrub (git, cargo, npm, docker, …) | `text`, `output` |
+| `summarize` | Hierarchical summary (conversation / file tree / outline) | `text`, `summarize` |
+| `summarize_smart` | Local-SLM dense summary (heuristic fallback if unset/fails) | `text`, `smart?`, `summarize?` |
+| `filter_relevant` | Query-aware keep of relevant lines (local SLM + heuristic fallback) | `text`, `query`, `smart?` |
+| `prune_history` | Drop filler / compress older chat turns | `text` or `messages`, `prune` |
+| `chunk` | Split into `cmp://` chunks (session-cached) | `text`, `chunk` |
+| `resolve` | Fetch chunk content by id | `id` (+ optional `map` / `text`) |
+| `count_tokens` | Measure tokens | `text` |
+| `stats` | Session savings + latency/bypass/backend telemetry | `reset?` |
+| `cache_store` | Park bulky payload outside the prompt | `text`, `cache` |
+| `cache_get` | Retrieve by key | `key` |
+| `cache_invalidate` | Drop one key or clear cache | `key?` |
+| `sanitize` | Redact secrets + neutralize IPI phrases | `text`, `sanitize?` |
+| `rerank` | BM25-rank candidates / chunks for a query | `query`, `items` or `text` or chunk `map`, `rerank?` |
+
+Optional on most text actions: `sanitize_input: true` scrubs before processing. Soft payloads under `COMPENDIUM_SIGNAL_MIN_CHARS` (default 1000) bypass `compress` / `summarize` / `summarize_smart` unless `force: true`.
+
+`filter` accepts optional `query` (top-level or `filter.query`) for BM25 line keep. `prune_history` supports `prune.strategy: "afm"` (Critical / Thematic / Distant tiers; distant blob cached for `cache_get`).
+
+Example:
+
+```json
+{
+  "action": "filter",
+  "text": "…noisy log…",
+  "filter": { "strip_ansi": true, "keep_patterns": ["ERROR|WARN"] }
+}
+```
+
+Response envelope: `{ "ok": true, "action": "filter", "result_json": "{...}" }`. Parse `result_json` as JSON for the action-specific payload.
+
+## Project layout
+
+```
+package.json / bin/run.js  # npm wrapper for npx compendium-mcp
+npm/                       # platform packages + distribution docs
+.github/workflows/         # release cross-compile + npm publish
+src/
+  main.rs              # CLI: stdio | http
+  lib.rs
+  config.rs            # COMPENDIUM_* env config
+  server.rs            # MCP tool handlers (rmcp macros)
+  http.rs              # Streamable HTTP/SSE (feature = "http")
+  pipeline/
+    tokens.rs          # heuristic or tiktoken BPE (feature = "real-tokens")
+    filter.rs
+    compress.rs
+    summarize.rs
+    smart.rs           # summarize_smart + filter_relevant
+    local_llm.rs       # OpenAI-compatible local SLM client
+    chunk.rs           # chunk + resolve
+    cache.rs           # session key/value cache
+    stats.rs           # session savings counters
+    prune.rs           # conversation history pruning
+    output.rs          # domain-aware compress_output
+tests/
+  integration.rs
+  e2e_smoke.rs         # spawns binary, MCP handshake, all tools
+```
+
+## Build
+
+```bash
+# Default: heuristic tokens + stdio only
+cargo build --release
+
+# Exact BPE token counts (tiktoken-rs)
+cargo build --release --features real-tokens
+
+# Streamable HTTP transport
+cargo build --release --features http
+
+# Everything
+cargo build --release --features real-tokens,http
+```
+
+Binary: `target/release/compendium`
+
+## Configure (Cursor / Claude Desktop)
+
+### Cursor (`~/.cursor/mcp.json` or project `.cursor/mcp.json`)
+
+```json
+{
+  "mcpServers": {
+    "compendium": {
+      "command": "npx",
+      "args": ["-y", "compendium-mcp"],
+      "env": {
+        "RUST_LOG": "compendium=info",
+        "COMPENDIUM_DEFAULT_MAX_TOKENS": "2048",
+        "COMPENDIUM_TOKENIZER": "cl100k_base"
+      }
+    }
+  }
+}
+```
+
+Or point at a local release binary:
+
+```json
+{
+  "mcpServers": {
+    "compendium": {
+      "command": "/absolute/path/to/Compendium/target/release/compendium",
+      "env": {
+        "RUST_LOG": "compendium=info",
+        "COMPENDIUM_DEFAULT_MAX_TOKENS": "2048",
+        "COMPENDIUM_CHARS_PER_TOKEN": "4.0",
+        "COMPENDIUM_TOKENIZER": "cl100k_base"
+      }
+    }
+  }
+}
+```
+
+### Remote / sidecar (HTTP)
+
+```bash
+cargo run --features http -- http 127.0.0.1:8788
+# MCP endpoint: http://127.0.0.1:8788/mcp
+```
+
+Point an MCP streamable-HTTP client at that URL (e.g. `StreamableHttpClientTransport::from_uri`).
+
+## Environment
+
+| Variable | Default | Meaning |
+|----------|---------|---------|
+| `COMPENDIUM_CHARS_PER_TOKEN` | `4.0` | Heuristic chars÷tokens (ignored with `real-tokens`) |
+| `COMPENDIUM_TOKENIZER` | `cl100k_base` | BPE encoding: `cl100k_base` or `o200k_base` (`real-tokens`) |
+| `COMPENDIUM_DEFAULT_MAX_TOKENS` | `2048` | Soft cap for compress |
+| `COMPENDIUM_MAX_BLANK_LINES` | `1` | Blank-line collapse limit |
+| `COMPENDIUM_SIMILARITY_THRESHOLD` | `0.85` | Jaccard line-dedupe threshold |
+| `COMPENDIUM_HTTP_BIND` | `127.0.0.1:8788` | Default HTTP listen address |
+| `COMPENDIUM_LOCAL_LLM_URL` | _(unset)_ | OpenAI-compatible base URL (e.g. `http://127.0.0.1:11434/v1` or `http://127.0.0.1:13305/api/v1`). Enables smart actions. |
+| `COMPENDIUM_LOCAL_LLM_MODEL` | `Qwen3-4B-GGUF` | Model id accepted by the local server |
+| `COMPENDIUM_LOCAL_LLM_API_KEY` | _(unset)_ | Optional bearer token for locked loopback servers |
+| `COMPENDIUM_LOCAL_LLM_TIMEOUT_SECS` | `120` | HTTP timeout (first model load can be slow) |
+| `COMPENDIUM_SIGNAL_MIN_CHARS` | `1000` | Bypass compress/summarize below this length (`0` disables) |
+| `RUST_LOG` | `compendium=info` | Logs on **stderr** only |
+
+## Example tool calls
+
+**Filter noisy terminal output**
+
+```json
+{
+  "name": "compendium_filter",
+  "arguments": {
+    "text": "\u001b[31mERROR\u001b[0m boom\n\n\nINFO ok",
+    "options": {
+      "strip_ansi": true,
+      "keep_patterns": ["ERROR|WARN"]
+    }
+  }
+}
+```
+
+**Compress a large log**
+
+```json
+{
+  "name": "compendium_compress",
+  "arguments": {
+    "text": "...",
+    "options": {
+      "content_type": "log",
+      "max_tokens": 512
+    }
+  }
+}
+```
+
+**Chunk a document into references**
+
+```json
+{
+  "name": "compendium_chunk",
+  "arguments": {
+    "text": "... huge file ...",
+    "options": {
+      "source": "file:///path/to/doc.md",
+      "chunk_tokens": 400,
+      "overlap_tokens": 40
+    }
+  }
+}
+```
+
+Prefer the returned `index_text` in the model context; pull individual chunk contents by id only when needed.
+
+**Query-aware filter (local SLM or heuristic fallback)**
+
+```json
+{
+  "action": "filter_relevant",
+  "text": "... noisy cargo/test log ...",
+  "query": "why did the auth tests fail",
+  "smart": { "max_tokens": 512, "fallback": true }
+}
+```
+
+Without `COMPENDIUM_LOCAL_LLM_URL`, `summarize_smart` / `filter_relevant` automatically use heuristics and set `backend: "heuristic"` plus `fallback_reason` in the result.
+
+## Local small language model
+
+Smart actions call a **loopback-only** OpenAI-compatible chat endpoint — never a cloud API. `COMPENDIUM_LOCAL_LLM_URL` must be `127.0.0.1`, `::1`, or `localhost` (SSRF guard). Requests use `temperature=0` and `seed=0` for stable outputs. Point Compendium at Ollama, Lemonade, or llama.cpp:
+
+```json
+{
+  "mcpServers": {
+    "compendium": {
+      "command": "npx",
+      "args": ["-y", "compendium-mcp"],
+      "env": {
+        "COMPENDIUM_LOCAL_LLM_URL": "http://127.0.0.1:11434/v1",
+        "COMPENDIUM_LOCAL_LLM_MODEL": "qwen2.5:3b"
+      }
+    }
+  }
+}
+```
+
+Lemonade example: `COMPENDIUM_LOCAL_LLM_URL=http://127.0.0.1:13305/api/v1` and `COMPENDIUM_LOCAL_LLM_MODEL=Qwen3-4B-GGUF`.
+
+## Develop / test
+
+```bash
+cargo test
+cargo test --features real-tokens
+cargo test --features http --test http_smoke
+cargo test --test e2e_smoke
+cargo run --features http -- http 127.0.0.1:8788
+```
+
+`e2e_smoke` spawns `CARGO_BIN_EXE_compendium`, completes the MCP initialize handshake over stdio, lists tools, then calls gateway actions. `http_smoke` (requires `--features http`) exercises streamable HTTP in-process.
+
+## Design notes
+
+- **Deterministic by default** — heuristic pipeline needs no network; smart actions only call a configured **local** OpenAI-compatible URL and fall back to heuristics when unset or failing.
+- **Token backends** — fast heuristic by default; opt into exact BPE with `real-tokens`.
+- **Zero stdout pollution** (stdio mode) — tracing goes to stderr so JSON-RPC framing stays clean.
+- **Release profile** — LTO + stripped binary for low footprint.
+
+## License
+
+MIT

package/bin/run.js

@@ -0,0 +1,201 @@
+#!/usr/bin/env node
+'use strict';
+
+/**
+ * Compendium MCP dispatcher.
+ *
+ * Resolves the native binary and exec's it with inherited stdio so Cursor /
+ * Claude Desktop / `npx compendium-mcp` talk JSON-RPC over stdin/stdout.
+ *
+ * Resolution order:
+ *   1. COMPENDIUM_BINARY env
+ *   2. Dev build: <repo>/target/release|debug/compendium(.exe)
+ *   3. optionalDependency platform package (compendium-mcp-<platform>)
+ *   4. Cached GitHub Release download (~/.cache/compendium-mcp/<version>/)
+ */
+
+const { spawn } = require('child_process');
+const fs = require('fs');
+const https = require('https');
+const http = require('http');
+const path = require('path');
+const { pipeline } = require('stream/promises');
+const { createWriteStream } = require('fs');
+
+const {
+  packageVersion,
+  githubRepo,
+  currentPlatform,
+  binaryName,
+  cacheDir,
+  platformKey,
+} = require('../npm/lib/platform');
+
+function existsExecutable(file) {
+  try {
+    fs.accessSync(file, fs.constants.F_OK);
+    // On Windows, X_OK is unreliable; presence is enough.
+    if (process.platform !== 'win32') {
+      fs.accessSync(file, fs.constants.X_OK);
+    }
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+function tryRequirePlatformBinary() {
+  const spec = currentPlatform();
+  try {
+    // optionalDependency installs next to this package when available.
+    const pkgRoot = path.dirname(require.resolve(`${spec.pkg}/package.json`));
+    const candidate = path.join(pkgRoot, 'bin', binaryName(spec));
+    if (existsExecutable(candidate)) return candidate;
+  } catch {
+    // optional dep missing — fall through
+  }
+  return null;
+}
+
+function tryDevBinary() {
+  const repoRoot = path.resolve(__dirname, '..');
+  const name = process.platform === 'win32' ? 'compendium.exe' : 'compendium';
+  const release = path.join(repoRoot, 'target', 'release', name);
+  const debug = path.join(repoRoot, 'target', 'debug', name);
+  if (existsExecutable(release)) return release;
+  if (existsExecutable(debug)) return debug;
+  return null;
+}
+
+function download(url, dest) {
+  return new Promise((resolve, reject) => {
+    const client = url.startsWith('https:') ? https : http;
+    const req = client.get(
+      url,
+      {
+        headers: {
+          'User-Agent': `compendium-mcp/${packageVersion()}`,
+          Accept: 'application/octet-stream',
+        },
+      },
+      (res) => {
+        // Follow one redirect hop (GitHub release assets).
+        if (
+          res.statusCode >= 300 &&
+          res.statusCode < 400 &&
+          res.headers.location
+        ) {
+          res.resume();
+          download(res.headers.location, dest).then(resolve, reject);
+          return;
+        }
+        if (res.statusCode !== 200) {
+          res.resume();
+          reject(
+            new Error(
+              `Download failed (${res.statusCode}) for ${url}. ` +
+                `Ensure release assets exist for ${platformKey()}.`
+            )
+          );
+          return;
+        }
+        const tmp = `${dest}.partial`;
+        const out = createWriteStream(tmp);
+        pipeline(res, out)
+          .then(() => {
+            fs.renameSync(tmp, dest);
+            if (process.platform !== 'win32') {
+              fs.chmodSync(dest, 0o755);
+            }
+            resolve(dest);
+          })
+          .catch((err) => {
+            try {
+              fs.unlinkSync(tmp);
+            } catch {
+              /* ignore */
+            }
+            reject(err);
+          });
+      }
+    );
+    req.on('error', reject);
+  });
+}
+
+async function ensureDownloadedBinary() {
+  const spec = currentPlatform();
+  const version = packageVersion();
+  const dir = cacheDir(version);
+  fs.mkdirSync(dir, { recursive: true });
+  const dest = path.join(dir, binaryName(spec));
+  if (existsExecutable(dest)) return dest;
+
+  const repo = githubRepo();
+  const tag = `v${version}`;
+  const url = `https://github.com/${repo}/releases/download/${tag}/${spec.asset}`;
+
+  process.stderr.write(
+    `[compendium-mcp] Downloading ${spec.asset} (${tag})…\n`
+  );
+  await download(url, dest);
+  process.stderr.write(`[compendium-mcp] Cached at ${dest}\n`);
+  return dest;
+}
+
+async function resolveBinary() {
+  if (process.env.COMPENDIUM_BINARY) {
+    const forced = path.resolve(process.env.COMPENDIUM_BINARY);
+    if (!existsExecutable(forced)) {
+      throw new Error(`COMPENDIUM_BINARY not executable: ${forced}`);
+    }
+    return forced;
+  }
+
+  // Prefer a local Cargo build when developing from this repo so `cargo build
+  // --release` is picked up immediately (optional platform packages can lag).
+  const fromDev = tryDevBinary();
+  if (fromDev) return fromDev;
+
+  const fromOptional = tryRequirePlatformBinary();
+  if (fromOptional) return fromOptional;
+
+  return ensureDownloadedBinary();
+}
+
+async function main() {
+  let binary;
+  try {
+    binary = await resolveBinary();
+  } catch (err) {
+    process.stderr.write(`[compendium-mcp] ${err.message}\n`);
+    process.exit(1);
+  }
+
+  process.stderr.write(`[compendium-mcp] using ${binary}\n`);
+
+  const child = spawn(binary, process.argv.slice(2), {
+    stdio: 'inherit',
+    windowsHide: true,
+  });
+
+  child.on('error', (err) => {
+    process.stderr.write(`[compendium-mcp] failed to spawn ${binary}: ${err.message}\n`);
+    process.exit(1);
+  });
+
+  const forward = (signal) => {
+    if (!child.killed) child.kill(signal);
+  };
+  process.on('SIGINT', () => forward('SIGINT'));
+  process.on('SIGTERM', () => forward('SIGTERM'));
+
+  child.on('exit', (code, signal) => {
+    if (signal) {
+      process.exit(signal === 'SIGINT' ? 130 : 1);
+    }
+    process.exit(code ?? 1);
+  });
+}
+
+main();

package/npm/lib/platform.js

@@ -0,0 +1,115 @@
+'use strict';
+
+/**
+ * Shared platform → package / release-asset mapping for Compendium npm distribution.
+ *
+ * Distribution strategy (hybrid):
+ * 1. Prefer optionalDependency platform packages (esbuild-style) — offline, fast.
+ * 2. Fall back to downloading the matching GitHub Release asset into a local cache.
+ * 3. Dev override: COMPENDIUM_BINARY or ./target/release/compendium.
+ */
+
+const os = require('os');
+const path = require('path');
+
+/** @typedef {{ pkg: string, asset: string, rustTarget: string }} PlatformSpec */
+
+/** @type {Record<string, PlatformSpec>} */
+const PLATFORMS = {
+  'darwin-arm64': {
+    pkg: 'compendium-mcp-darwin-arm64',
+    asset: 'compendium-darwin-arm64',
+    rustTarget: 'aarch64-apple-darwin',
+  },
+  'darwin-x64': {
+    pkg: 'compendium-mcp-darwin-x64',
+    asset: 'compendium-darwin-x64',
+    rustTarget: 'x86_64-apple-darwin',
+  },
+  'linux-x64': {
+    pkg: 'compendium-mcp-linux-x64',
+    asset: 'compendium-linux-x64',
+    rustTarget: 'x86_64-unknown-linux-gnu',
+  },
+  'linux-arm64': {
+    pkg: 'compendium-mcp-linux-arm64',
+    asset: 'compendium-linux-arm64',
+    rustTarget: 'aarch64-unknown-linux-gnu',
+  },
+  'win32-x64': {
+    pkg: 'compendium-mcp-win32-x64',
+    asset: 'compendium-win32-x64.exe',
+    rustTarget: 'x86_64-pc-windows-msvc',
+  },
+};
+
+function packageVersion() {
+  // Keep in sync with root package.json / Cargo.toml at publish time.
+  return require('../../package.json').version;
+}
+
+function githubRepo() {
+  return (
+    process.env.COMPENDIUM_GITHUB_REPO ||
+    process.env.GITHUB_REPOSITORY ||
+    'hocestnonsatis/Compendium'
+  );
+}
+
+/**
+ * Normalize Node's process.platform + process.arch into our key.
+ * @returns {string}
+ */
+function platformKey() {
+  const platform = process.platform;
+  let arch = process.arch;
+  // Rosetta / rare aliases
+  if (arch === 'ia32') arch = 'x64';
+  return `${platform}-${arch}`;
+}
+
+/**
+ * @returns {PlatformSpec}
+ */
+function currentPlatform() {
+  const key = platformKey();
+  const spec = PLATFORMS[key];
+  if (!spec) {
+    const supported = Object.keys(PLATFORMS).join(', ');
+    throw new Error(
+      `Unsupported platform "${key}". Supported: ${supported}.\n` +
+        `Set COMPENDIUM_BINARY to a local build, or open an issue for this target.`
+    );
+  }
+  return spec;
+}
+
+/**
+ * Binary filename inside a platform package / cache.
+ * @param {PlatformSpec} [spec]
+ */
+function binaryName(spec = currentPlatform()) {
+  return process.platform === 'win32' ? 'compendium.exe' : 'compendium';
+}
+
+/**
+ * Cache directory for lazily downloaded binaries.
+ * @param {string} version
+ */
+function cacheDir(version = packageVersion()) {
+  const base =
+    process.env.COMPENDIUM_CACHE_DIR ||
+    process.env.XDG_CACHE_HOME ||
+    path.join(os.homedir(), '.cache');
+  return path.join(base, 'compendium-mcp', version);
+}
+
+module.exports = {
+  PLATFORMS,
+  packageVersion,
+  githubRepo,
+  platformKey,
+  currentPlatform,
+  binaryName,
+  cacheDir,
+};

package/package.json

@@ -0,0 +1,46 @@
+{
+  "name": "compendium-mcp",
+  "version": "0.1.0",
+  "description": "MCP server that compresses, summarizes, and filters context to minimize LLM token usage",
+  "license": "MIT",
+  "keywords": [
+    "mcp",
+    "model-context-protocol",
+    "llm",
+    "tokens",
+    "compression",
+    "cursor",
+    "claude"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/hocestnonsatis/Compendium.git"
+  },
+  "bugs": {
+    "url": "https://github.com/hocestnonsatis/Compendium/issues"
+  },
+  "homepage": "https://github.com/hocestnonsatis/Compendium#readme",
+  "engines": {
+    "node": ">=18"
+  },
+  "bin": {
+    "compendium": "bin/run.js"
+  },
+  "files": [
+    "bin/",
+    "npm/lib/",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "prepack": "node npm/scripts/check-wrapper.js"
+  },
+  "optionalDependencies": {
+    "compendium-mcp-darwin-arm64": "0.1.0",
+    "compendium-mcp-darwin-x64": "0.1.0",
+    "compendium-mcp-linux-x64": "0.1.0",
+    "compendium-mcp-linux-arm64": "0.1.0",
+    "compendium-mcp-win32-x64": "0.1.0"
+  },
+  "preferUnplugged": true
+}