compendium-mcp 0.1.3 → 0.5.0

package/README.md

@@ -80,6 +80,8 @@ Binary packaging details for maintainers: [npm/DISTRIBUTION.md](npm/DISTRIBUTION
 ## Community
 
 - [Contributing](CONTRIBUTING.md)
+- [Changelog](CHANGELOG.md)
+- [Architecture](docs/architecture.md)
 - [Code of Conduct](CODE_OF_CONDUCT.md)
 - [Security policy](SECURITY.md)
 - [Support](SUPPORT.md)
@@ -99,9 +101,9 @@ Single MCP tool: **`compendium`**. Choose the operation with `action`:
 
 | `action` | Purpose | Main fields |
 |----------|---------|-------------|
-| `filter` | Strip ANSI, boilerplate, whitespace; densify JSON; keep/drop regexes | `text`, `filter` |
-| `compress` | Dense representation of text/code/logs | `text`, `compress` |
-| `compress_output` | Domain-aware stdout/stderr scrub (git, cargo, npm, docker, …) | `text`, `output` |
+| `filter` | Strip ANSI, boilerplate, whitespace; densify JSON; keep/drop regexes | `text`, `filter` — not for cargo/npm dumps (`compress_output`) |
+| `compress` | Dense representation of text/code/logs | `text`, `compress` — soft inputs under ~1000 chars bypass unless `force` |
+| `compress_output` | Domain-aware stdout/stderr scrub (git, cargo, npm, docker, …) | `text`, `output` — prefer when CLI domain is known |
 | `summarize` | Hierarchical summary (conversation / file tree / outline) | `text`, `summarize` |
 | `summarize_smart` | Local-SLM dense summary (heuristic fallback if unset/fails) | `text`, `smart?`, `summarize?` |
 | `filter_relevant` | Query-aware keep of relevant lines (local SLM + heuristic fallback) | `text`, `query`, `smart?` |
@@ -109,19 +111,20 @@ Single MCP tool: **`compendium`**. Choose the operation with `action`:
 | `chunk` | Split into `cmp://` chunks (session-cached) | `text`, `chunk` |
 | `resolve` | Fetch chunk content by id | `id` (+ optional `map` / `text`) |
 | `count_tokens` | Measure tokens | `text` |
-| `stats` | Session savings + latency/bypass/backend telemetry | `reset?` |
+| `stats` | Session savings + latency/bypass/backend telemetry | `reset?` — see playbook `stats-debug` |
 | `cache_store` | Park bulky payload outside the prompt | `text`, `cache` |
 | `cache_get` | Retrieve by key | `key` |
 | `cache_invalidate` | Drop one key or clear cache | `key?` |
-| `sanitize` | Redact secrets + neutralize IPI phrases | `text`, `sanitize?` |
-| `rerank` | BM25-rank candidates / chunks for a query | `query`, `items` or `text` or chunk `map`, `rerank?` |
+| `sanitize` | Redact secrets + neutralize IPI phrases | `text`, `sanitize?` — or `sanitize_input` |
+| `rerank` | BM25 (+ optional loopback embeddings + opt-in SLM cross-encoder) rank candidates / chunks | `query`, `items` or `text` or chunk `map`, `rerank?` |
 | `brief` | Scan a workspace; pack a structured starter briefing + cache key | `query`, `brief?` (`root`, caps), optional `text` hint |
-| `catalog` | Short action (+ playbook) ads; prefer before guessing | _(none)_ |
+| `catalog` | Short action (+ playbook) ads; prefer before guessing | _(none)_ — call first when unsure |
 | `help` | Usage notes for one action (default **compressed**; `force: true` → full) | `id`, `force?` |
-| `playbooks` | List bundled token-hygiene playbook ads | _(none)_ |
-| `playbook` | Load one playbook body (sanitized) | `id` |
-| `pack` | Zip text/files into a size-capped archive (cache and/or base64) | `text` or `items`, `pack?` |
-| `unpack` | Unpack zip with caps → chunks (**never runs scripts**) | `key` or base64 `text`, `pack?` |
+| `playbooks` | List playbook ads | _(none)_ |
+| `playbook` | Load one playbook body | `id` |
+| `pack` | Zip text/files into a bounded archive | `text` or `items`, `pack?` |
+| `unpack` | Unpack zip with size caps into chunks (never runs scripts) | `text` or `key`, `pack?` |
+| `llm_status` | Probe configured local LLM (models; `force` = chat ping) | `force?` — when smart/hybrid unexpectedly heuristic |
 
 ### Progressive disclosure (skills)
 
@@ -157,22 +160,26 @@ Response envelope: `{ "ok": true, "action": "filter", "result_json": "{...}" }`.
 
 ```
 assets/                # brand mark (SVG/PNG); baked into MCP icons via data URI
+docs/                  # architecture notes
+examples/              # sample MCP tool-call JSON payloads
+testdata/              # eval fixtures (noisy logs, cargo fail, …)
 src/
   main.rs              # CLI: stdio | http
   lib.rs
   brand.rs             # SEP-973 icons for serverInfo + tool
   config.rs            # COMPENDIUM_* env config
-  server.rs            # MCP tool + resources handlers (rmcp)
+  server/              # MCP tool + resources + action handlers (rmcp)
   http.rs              # Streamable HTTP/SSE (feature = "http")
   pipeline/
+    brief/             # workspace brief (walk / window / pack / synthesize)
     tokens.rs          # heuristic or tiktoken BPE (feature = "real-tokens")
     filter.rs
     compress.rs
     summarize.rs
     smart.rs           # summarize_smart + filter_relevant
-    local_llm.rs       # OpenAI-compatible local SLM client
+    local_llm.rs       # OpenAI-compatible local SLM client (+ embed cache)
     chunk.rs           # chunk + resolve
-    cache.rs           # session key/value cache
+    cache.rs           # session key/value cache (+ optional disk / embed vectors)
     catalog.rs         # action ads + help (progressive disclosure)
     playbook.rs        # bundled / dir playbooks
     pack.rs            # zip pack/unpack with size caps
@@ -183,6 +190,9 @@ playbooks/             # embedded skill-md playbooks
 tests/
   integration.rs
   e2e_smoke.rs         # spawns binary, MCP handshake, tools + resources
+  eval_regression.rs   # B1 heuristic quality + latency smoke
+CHANGELOG.md
+REPORT.md              # design essay + Shipped/Next/Deferred roadmap
 ```
 
 ## Build
@@ -260,6 +270,11 @@ Point an MCP streamable-HTTP client at that URL (e.g. `StreamableHttpClientTrans
 | `COMPENDIUM_HTTP_BIND` | `127.0.0.1:8788` | Default HTTP listen address |
 | `COMPENDIUM_LOCAL_LLM_URL` | _(unset)_ | OpenAI-compatible base URL (e.g. `http://127.0.0.1:11434/v1` or `http://127.0.0.1:13305/api/v1`). Enables smart actions. |
 | `COMPENDIUM_LOCAL_LLM_MODEL` | `Qwen3-4B-GGUF` | Model id on that server (Ollama: e.g. `qwen:latest`) |
+| `COMPENDIUM_LOCAL_EMBED_MODEL` | _(same as chat)_ | Embeddings model for hybrid `rerank` / `brief` (e.g. `nomic-embed-text`) |
+| `COMPENDIUM_HYBRID_ALPHA` | `0.55` | BM25 weight in hybrid score (0–1); remainder is embedding cosine |
+| `COMPENDIUM_RERANK_CROSS_ENCODER` | _(off)_ | When `1`/`true`, `rerank` SLM-rescores top-N after BM25/hybrid |
+| `COMPENDIUM_CROSS_ENCODER_TOP_N` | `16` | Candidates passed to cross-encoder (clamped 4–64) |
+| `COMPENDIUM_AUDIT_PATH` | _(unset)_ | Append-only JSONL audit log (action metadata only; no payloads) |
 | `COMPENDIUM_LOCAL_LLM_API_KEY` | _(unset)_ | Optional bearer token for locked loopback servers |
 | `COMPENDIUM_LOCAL_LLM_TIMEOUT_SECS` | `120` | HTTP timeout (first model load can be slow) |
 | `COMPENDIUM_SIGNAL_MIN_CHARS` | `1000` | Bypass compress/summarize below this length (`0` disables) |
@@ -269,21 +284,23 @@ Point an MCP streamable-HTTP client at that URL (e.g. `StreamableHttpClientTrans
 | `COMPENDIUM_ARCHIVE_MAX_UNCOMPRESSED` | `4194304` | Max total uncompressed bytes for pack/unpack |
 | `COMPENDIUM_ARCHIVE_MAX_FILES` | `50` | Max files per archive |
 | `COMPENDIUM_SKILL_TTL_MS` | `300000` | Soft TTL (ms) on skill `resources/read` responses |
+| `COMPENDIUM_CACHE_DIR` | _(unset)_ | Persist session cache (chunks/cache keys) across restarts; default size cap 64 MiB |
+| `COMPENDIUM_CACHE_MAX_BYTES` | _(unset / 64MiB with dir)_ | Soft cap on total cached payload bytes |
 | `RUST_LOG` | `compendium=info` | Logs on **stderr** only |
 
 ## Example tool calls
 
+All calls use the single tool **`compendium`** with an `action` field.
+
 **Filter noisy terminal output**
 
 ```json
 {
-  "name": "compendium_filter",
-  "arguments": {
-    "text": "\u001b[31mERROR\u001b[0m boom\n\n\nINFO ok",
-    "options": {
-      "strip_ansi": true,
-      "keep_patterns": ["ERROR|WARN"]
-    }
+  "action": "filter",
+  "text": "\u001b[31mERROR\u001b[0m boom\n\n\nINFO ok",
+  "filter": {
+    "strip_ansi": true,
+    "keep_patterns": ["ERROR|WARN"]
   }
 }
 ```
@@ -292,13 +309,11 @@ Point an MCP streamable-HTTP client at that URL (e.g. `StreamableHttpClientTrans
 
 ```json
 {
-  "name": "compendium_compress",
-  "arguments": {
-    "text": "...",
-    "options": {
-      "content_type": "log",
-      "max_tokens": 512
-    }
+  "action": "compress",
+  "text": "...",
+  "compress": {
+    "content_type": "log",
+    "max_tokens": 512
   }
 }
 ```
@@ -307,14 +322,12 @@ Point an MCP streamable-HTTP client at that URL (e.g. `StreamableHttpClientTrans
 
 ```json
 {
-  "name": "compendium_chunk",
-  "arguments": {
-    "text": "... huge file ...",
-    "options": {
-      "source": "file:///path/to/doc.md",
-      "chunk_tokens": 400,
-      "overlap_tokens": 40
-    }
+  "action": "chunk",
+  "text": "... huge file ...",
+  "chunk": {
+    "source": "file:///path/to/doc.md",
+    "chunk_tokens": 400,
+    "overlap_tokens": 40
   }
 }
 ```

package/npm/lib/platform.js

@@ -7,10 +7,13 @@
  * 1. Prefer optionalDependency platform packages (esbuild-style) — offline, fast.
  * 2. Fall back to downloading the matching GitHub Release asset into a local cache.
  * 3. Dev override: COMPENDIUM_BINARY or ./target/release/compendium.
+ * 4. Force key: COMPENDIUM_PLATFORM (e.g. linux-x64-musl).
  */
 
+const fs = require('fs');
 const os = require('os');
 const path = require('path');
+const { execSync } = require('child_process');
 
 /** @typedef {{ pkg: string, asset: string, rustTarget: string }} PlatformSpec */
 
@@ -31,6 +34,11 @@ const PLATFORMS = {
     asset: 'compendium-linux-x64',
     rustTarget: 'x86_64-unknown-linux-gnu',
   },
+  'linux-x64-musl': {
+    pkg: 'compendium-mcp-linux-x64-musl',
+    asset: 'compendium-linux-x64-musl',
+    rustTarget: 'x86_64-unknown-linux-musl',
+  },
   'linux-arm64': {
     pkg: 'compendium-mcp-linux-arm64',
     asset: 'compendium-linux-arm64',
@@ -41,6 +49,11 @@ const PLATFORMS = {
     asset: 'compendium-win32-x64.exe',
     rustTarget: 'x86_64-pc-windows-msvc',
   },
+  'win32-arm64': {
+    pkg: 'compendium-mcp-win32-arm64',
+    asset: 'compendium-win32-arm64.exe',
+    rustTarget: 'aarch64-pc-windows-msvc',
+  },
 };
 
 function packageVersion() {
@@ -56,15 +69,51 @@ function githubRepo() {
   );
 }
 
+/** Best-effort musl detection (Alpine / static libc). Default is glibc. */
+function isLinuxMusl() {
+  if (process.platform !== 'linux') return false;
+  try {
+    if (typeof process.report?.getReport === 'function') {
+      const report = process.report.getReport();
+      if (report?.header?.glibcVersionRuntime) return false;
+    }
+  } catch (_) {
+    /* ignore */
+  }
+  try {
+    if (fs.existsSync('/etc/alpine-release')) return true;
+  } catch (_) {
+    /* ignore */
+  }
+  try {
+    const out = execSync('ldd --version 2>&1 || true', {
+      encoding: 'utf8',
+      stdio: ['ignore', 'pipe', 'pipe'],
+    });
+    if (/musl/i.test(out)) return true;
+  } catch (_) {
+    /* ignore */
+  }
+  return false;
+}
+
 /**
  * Normalize Node's process.platform + process.arch into our key.
+ * Override with COMPENDIUM_PLATFORM when needed (e.g. linux-x64-musl).
  * @returns {string}
  */
 function platformKey() {
+  const forced = (process.env.COMPENDIUM_PLATFORM || '').trim();
+  if (forced) return forced;
+
   const platform = process.platform;
   let arch = process.arch;
   // Rosetta / rare aliases
   if (arch === 'ia32') arch = 'x64';
+
+  if (platform === 'linux' && arch === 'x64' && isLinuxMusl()) {
+    return 'linux-x64-musl';
+  }
   return `${platform}-${arch}`;
 }
 
@@ -78,7 +127,7 @@ function currentPlatform() {
     const supported = Object.keys(PLATFORMS).join(', ');
     throw new Error(
       `Unsupported platform "${key}". Supported: ${supported}.\n` +
-        `Set COMPENDIUM_BINARY to a local build, or open an issue for this target.`
+        `Set COMPENDIUM_BINARY to a local build, COMPENDIUM_PLATFORM to a known key, or open an issue for this target.`
     );
   }
   return spec;
@@ -89,7 +138,9 @@ function currentPlatform() {
  * @param {PlatformSpec} [spec]
  */
 function binaryName(spec = currentPlatform()) {
-  return process.platform === 'win32' ? 'compendium.exe' : 'compendium';
+  return process.platform === 'win32' || spec.asset.endsWith('.exe')
+    ? 'compendium.exe'
+    : 'compendium';
 }
 
 /**
@@ -112,4 +163,5 @@ module.exports = {
   currentPlatform,
   binaryName,
   cacheDir,
+  isLinuxMusl,
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "compendium-mcp",
-  "version": "0.1.3",
+  "version": "0.5.0",
   "description": "MCP server that compresses, summarizes, and filters context to minimize LLM token usage",
   "license": "MIT",
   "keywords": [
@@ -38,11 +38,13 @@
     "prepare": "node npm/scripts/link-bins.js"
   },
   "optionalDependencies": {
-    "compendium-mcp-darwin-arm64": "0.1.3",
-    "compendium-mcp-darwin-x64": "0.1.3",
-    "compendium-mcp-linux-x64": "0.1.3",
-    "compendium-mcp-linux-arm64": "0.1.3",
-    "compendium-mcp-win32-x64": "0.1.3"
+    "compendium-mcp-darwin-arm64": "0.5.0",
+    "compendium-mcp-darwin-x64": "0.5.0",
+    "compendium-mcp-linux-x64": "0.5.0",
+    "compendium-mcp-linux-x64-musl": "0.5.0",
+    "compendium-mcp-linux-arm64": "0.5.0",
+    "compendium-mcp-win32-x64": "0.5.0",
+    "compendium-mcp-win32-arm64": "0.5.0"
   },
   "preferUnplugged": true
 }