mcp-compression-proxy 1.0.2 → 1.1.0

package/README.md

@@ -2,841 +2,409 @@
 
 # 🗜️ MCP Compression Proxy
 
-**Aggregate tools from multiple MCP servers with intelligent LLM-based description compression**
+### More MCP tools. Less context overhead.
+
+Use a large MCP toolset without loading every tool into every prompt. MCP Compression Proxy is a local, open-source gateway with progressive discovery for shell-capable agents and one compatible endpoint for native MCP clients.
 
 [![npm version][npm-version-badge]][npm-package]
 [![npm downloads][npm-downloads-badge]][npm-package]
-[![npm types][npm-types-badge]][npm-package]
-[![License: MIT][license-badge]][license]
-[![Node][node-badge]][nodejs]
-[![MCP][mcp-badge]][mcp-protocol]
-
+[![Node.js 22+][node-badge]][nodejs]
 [![CI][ci-badge]][ci-workflow]
 [![codecov][codecov-badge]][codecov]
-[![CodeQL][codeql-badge]][codeql-workflow]
+[![License: MIT][license-badge]][license-file]
 
-[![GitHub Stars][stars-badge]][stargazers]
-[![GitHub Forks][forks-badge]][network]
-[![GitHub Issues][issues-badge]][repo-issues]
-[![GitHub Last Commit][commit-badge]][commits]
-[![PRs Welcome][prs-badge]][contributing]
-
-[Quick Start](#-quick-start) •
-[Features](#-features) •
-[mcp-cli](#️-mcp-cli-progressive-tool-discovery) •
-[Configuration](#-configuration) •
-[FAQ](#-faq) •
-[Contributing](#-contributing)
+[🚀 Quick start](#-quick-start-progressive-discovery) · [🧭 Choose a mode](#-choose-a-mode) · [🔧 Configuration](#-configuration) · [💖 Support](#-support-the-project)
 
 </div>
 
----
-
-## 📑 Table of Contents
-
-- [What is MCP Compression Proxy?](#what-is-mcp-compression-proxy)
-- [✨ Features](#-features)
-- [🚀 Quick Start](#-quick-start)
-- [🎯 Usage](#-usage)
-- [⌨️ mcp-cli (Progressive Tool Discovery)](#️-mcp-cli-progressive-tool-discovery)
-- [🔧 Configuration](#-configuration)
-- [💡 Best Practices](#-best-practices)
-- [❓ FAQ](#-faq)
-- [🧪 Testing](#-testing)
-- [🤝 Contributing](#-contributing)
-- [💖 Support This Project](#-support-this-project)
-
----
-
-## What is MCP Compression Proxy?
-
-A **Model Context Protocol (MCP) server** that solves two common problems:
+MCP Compression Proxy combines local stdio and remote Streamable HTTP servers behind one configuration. Agents can search for a tool, inspect its schema only when needed, and keep oversized results out of the conversation.
 
-1. **Multi-server aggregation**: Access tools from multiple MCP servers through a single connection
-2. **Context optimization**: Reduce token consumption by 50-80% using intelligent LLM-based description compression
+- 🔍 **Discover tools on demand** with `mcp-cli`.
+- 🔌 **Connect through one MCP endpoint** when native MCP compatibility is required.
+- 📦 **Keep large results local** and read only the relevant portions.
+- ♻️ **Reuse warm backends** and refresh stale or unhealthy connections.
+- 🏠 **Keep control local** without a hosted gateway or control plane.
 
-Instead of connecting to multiple MCP servers separately and consuming thousands of tokens on verbose tool descriptions, MCP Compression Proxy aggregates all your tools and compresses their descriptions intelligently—preserving critical information while removing redundancy.
+> [!IMPORTANT]
+> `mcp-cli` provides the largest context reduction because it defers full tool schemas until an agent requests one. Native proxy mode shortens tool descriptions, but MCP clients still receive each tool's input schema during discovery.
 
-**Perfect for:**
-- Users with many MCP servers (filesystem, GitHub, databases, etc.)
-- AI agents working with limited context windows
-- Anyone wanting to minimize token costs while maximizing tool availability
+## 🧭 Choose a mode
 
-## ✨ Features
+| Your client                   | Start with                  | Context behavior                                                                  |
+| ----------------------------- | --------------------------- | --------------------------------------------------------------------------------- |
+| ⌨️ Shell-capable coding agent | **`mcp-cli` (recommended)** | Search compact summaries, inspect one schema, then call the tool                  |
+| 🔌 Native MCP client          | **`mcp-compression-proxy`** | Connect through one endpoint and use shorter descriptions; schemas remain exposed |
 
-- **🔗 Multi-Server Aggregation** - Access tools from multiple MCP servers through one connection
-- **🤖 LLM-Based Compression** - Intelligent description compression (50-80% token reduction)
-- **💾 Persistent Storage** - Compressed descriptions saved to disk and restored on restart
-- **🎭 Session-Based Expansion** - Independent expansion state per conversation
-- **⚡ Parallel Initialization** - All servers connect in parallel with configurable timeouts
-- **🎯 Selective Expansion** - Compress all tools, expand only what you need
-- **📦 Zero Config** - Works out-of-the-box with sensible defaults
-- **🔥 Standard MCP** - Compatible with any MCP client (Claude Desktop, Cline, etc.)
+Both modes use the same server configuration and support local stdio and remote Streamable HTTP backends.
 
-## 🚀 Quick Start
+## 🚀 Quick start: progressive discovery
 
-### Prerequisites
-
-- **Node.js 22+** installed on your system
-- An **MCP-compatible client** (Claude Desktop, Cline, Continue.dev, etc.)
+Requires Node.js 22 or newer.
 
 ### 1. Install
 
-**Option A: Install from npm** (recommended for most users):
 ```bash
 npm install -g mcp-compression-proxy
 ```
 
-**Option B: Install from source** (for development or latest features):
-```bash
-git clone https://github.com/kdpa-llc/mcp-compression-proxy.git
-cd mcp-compression-proxy
-npm install
-npm run build
-```
-
-### 2. Configure MCP Client
-
-Add to your MCP client configuration file:
+The package installs two commands: `mcp-cli` for progressive discovery and `mcp-compression-proxy` for native MCP clients.
 
-- **macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`
-- **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
+### 2. Add one MCP server
 
-**If installed via npm:**
-```json
-{
-  "mcpServers": {
-    "compression-proxy": {
-      "command": "mcp-compression-proxy",
-      "env": {
-        "LOG_LEVEL": "info"
-      }
-    }
-  }
-}
-```
+This macOS/Linux example gives the filesystem server access to `/tmp`:
 
-**If installed from source:**
-```json
-{
-  "mcpServers": {
-    "compression-proxy": {
-      "command": "node",
-      "args": [
-        "/absolute/path/to/mcp-compression-proxy/dist/index.js"
-      ],
-      "env": {
-        "LOG_LEVEL": "info"
-      }
-    }
-  }
-}
-```
-
-### 3. Configure Servers
-
-Create a JSON configuration file to define which MCP servers to aggregate:
-
-**Option 1: User-level config** (recommended for personal use)
-- Location: `~/.mcp-compression-proxy/servers.json`
-
-**Option 2: Project-level config** (recommended for team projects)
-- Location: `./servers.json` (in the mcp-compression-proxy directory)
-
-**Example configuration:**
-
-```json
+```bash
+mkdir -p "$HOME/.mcp-compression-proxy"
+cat > "$HOME/.mcp-compression-proxy/servers.json" <<'JSON'
 {
   "mcpServers": [
     {
       "name": "filesystem",
       "command": "npx",
-      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"],
-      "enabled": true
-    },
-    {
-      "name": "github",
-      "command": "npx",
-      "args": ["-y", "@modelcontextprotocol/server-github"],
-      "env": {
-        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}"
-      },
-      "enabled": true
+      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
     }
   ]
 }
+JSON
 ```
 
-> **Note:** No rebuild needed! Just edit the JSON file and restart your MCP client.
-
-### 4. Restart Your MCP Client
-
-Restart your MCP client (e.g., Claude Desktop) to load the new configuration. The proxy will automatically connect to all configured MCP servers and make their tools available.
-
-## 🎯 Usage
-
-### Tool Naming
-
-**Proxied tools** from your configured MCP servers use the format `serverName__toolName`:
-- `filesystem__read_file`
-- `filesystem__write_file`
-- `github__create_issue`
-
-**Management tools** (built-in) don't have a prefix and are listed below.
-
-### Management Tools
-
-| Tool | Description |
-|------|-------------|
-| `create_session` | Create a new session for independent tool expansion |
-| `set_session` | Set the active session |
-| `delete_session` | Delete a session |
-| `clear_compressed_tools_cache` | Clear all cached compressed tool descriptions |
-| `get_uncompressed_tools` | Get tools that need compression (batch processing) |
-| `cache_compressed_tools` | Save compressed descriptions to cache (batch processing) |
-| `compress_via_sampling` | Compress automatically using the client's own LLM (requires sampling support) |
-| `expand_tool` | Expand a tool to show full description (session-specific) |
-| `collapse_tool` | Collapse tool back to compressed description |
-| `stats` | Return JSON summary of coverage, cache health, sessions, and per-server tool counts |
-
-Use `stats` from your client (e.g., `mcp-compression-proxy__stats`) to sanity-check coverage. Optional inputs: `serverName` to scope to one backend and `detailLevel` (`summary` or `full`, default `summary`). The response includes coverage %, estimated token savings, cache state, active sessions, and per-server tool counts (respecting your exclude patterns).
+### 3. Find and call a tool
 
-#### Automatic Compression (MCP Sampling)
-
-If your client supports [MCP sampling][mcp-sampling], the proxy can compress
-descriptions by borrowing the client's own LLM — no API key, no second model
-to configure, and no work for you beyond one tool call:
-
-```
-mcp-compression-proxy__compress_via_sampling
+```bash
+mcp-cli doctor
+mcp-cli search file
+mcp-cli info filesystem/list_directory
+mcp-cli call filesystem/list_directory '{"path":"/tmp"}'
 ```
 
-It reads the uncompressed tools, asks the host to rewrite them, caches the
-results, and reports before/after coverage. Call it again for the next batch
-until nothing remains. Batches are sent one at a time, since a host may ask
-you to approve each request.
-
-Support varies by client — Cursor implements sampling; Claude Desktop and
-Cline did not at the time of writing. On a client without it the tool returns
-an error pointing at the manual `get_uncompressed_tools` →
-`cache_compressed_tools` flow, which works everywhere.
+The daemon starts automatically on the first command and keeps backend connections warm. `doctor` validates the configuration and reports backend health.
 
-### Workflow Example
+### 4. Tell your agent how to use it
 
-#### 1. Before Compression
+Add this to your project instructions or `AGENTS.md`:
 
-When you first ask your AI assistant about available tools:
-
-```
-User: What tools do you have?
-
-AI: I have access to these tools:
-- filesystem__read_file: Reads the complete contents of a file at the
-  specified path. The file must exist and be readable. Returns the file
-  contents as text. Supports absolute and relative paths. Maximum file
-  size is 10MB. Will throw an error if the file doesn't exist...
-  [~200 tokens for one tool]
+```text
+Use mcp-cli to access MCP tools. Search before choosing a tool, inspect its
+schema before the first call, and read large outputs with `mcp-cli output`
+instead of loading an entire payload into the conversation.
 ```
 
-#### 2. Enable Compression (One-Time Setup)
+That is the complete progressive-discovery setup. Add more servers to the same `servers.json` file as needed.
 
-Ask your AI assistant to compress the descriptions:
+> 🎉 **You're ready.** Your agent can now discover the right tool, load one schema, and call it without carrying the entire catalog through the conversation.
 
-```
-User: Use the mcp-compression-proxy tools to compress tool descriptions and save model context
+## 🔌 Native MCP client setup
 
-AI: I'll compress the tool descriptions:
-1. Getting all tools via get_uncompressed_tools...
-2. Compressing descriptions intelligently...
-3. Saving compressed versions via cache_compressed_tools...
+Use this mode when a client expects to launch an MCP server directly. Add the proxy to the client's MCP configuration:
 
-Done! Tool descriptions are now compressed and saved to cache.
+```json
+{
+  "mcpServers": {
+    "compression-proxy": {
+      "command": "mcp-compression-proxy",
+      "env": {
+        "LOG_LEVEL": "info"
+      }
+    }
+  }
+}
 ```
 
-#### 3. After Compression
+Restart the client after saving its configuration. The proxy loads backend definitions from `~/.mcp-compression-proxy/servers.json` and exposes their tools as `serverName__toolName`, such as `filesystem__read_file`.
 
-The same request now uses far fewer tokens:
+### Compress descriptions
 
-```
-User: What tools do you have?
+If the client supports [MCP sampling][mcp-sampling], ask it to call:
 
-AI: I have access to these tools:
-- filesystem__read_file: Read file contents (text, max 10MB)
-- filesystem__write_file: Write/overwrite file
-- github__create_issue: Create GitHub issue
-  [~30 tokens for one tool]
+```text
+mcp-compression-proxy__compress_via_sampling
 ```
 
-**Result**: ~70% reduction in tokens for tool listings!
+The proxy asks the client's existing model to shorten a batch of descriptions and saves the results in `~/.mcp-compression-proxy/cache.json`. No separate model provider or API key is required. Sampling is deprecated in the 2026-07-28 MCP specification, so new clients may omit it.
 
-#### 4. Persistent Storage
+If sampling is unavailable, use `mcp-compression-proxy__get_uncompressed_tools` and then `mcp-compression-proxy__cache_compressed_tools`. Updated backend descriptions are detected as stale and queued for compression again.
 
-Compressed descriptions are automatically saved to disk at `~/.mcp-compression-proxy/cache.json` and loaded on server restart. No need to re-compress after restarting!
+## 🧠 Where the context savings come from
 
-**To clear the cache if needed:**
-```bash
-# If installed via npm
-mcp-compression-proxy --clear-cache
+| Access pattern   | Loaded before the task                                     | Loaded when a tool is selected             |
+| ---------------- | ---------------------------------------------------------- | ------------------------------------------ |
+| Eager MCP client | Every advertised name, description, and input schema       | Nothing additional                         |
+| Native proxy     | Names, shorter cached descriptions, and every input schema | Full description on request                |
+| Progressive CLI  | A small command vocabulary and compact search results      | The selected tool's description and schema |
 
-# If installed from source
-node dist/index.js --clear-cache
-```
+Exact savings depend on the number of servers, their schema sizes, and which tools a task uses. The project intentionally does not claim a universal percentage: measure the complete tool definitions in your own stack rather than description text alone.
 
-> **💡 Tip**: After setting up, simply tell your AI: *"Compress the tool descriptions to save context"* and it will handle the rest!
+## ✨ What it handles
 
-## ⌨️ mcp-cli (Progressive Tool Discovery)
+- **One configuration:** Aggregate any number of local stdio and remote Streamable HTTP servers.
+- **Progressive discovery:** Search tools and fetch only the schema needed for the next call.
+- **Large-output control:** Store results above a configurable threshold in private local files, then search or page through them.
+- **Warm, replaceable connections:** Reuse backend processes while draining old generations without interrupting active calls.
+- **Authentication recovery:** Reconnect after configured authentication failures and retry only tools explicitly marked safe.
+- **Declarative call chains:** Run dependent MCP calls with JSON Pointer references and no arbitrary shell or JavaScript execution.
+- **Operational visibility:** Inspect live server state, connection age, active calls, retries, failures, and compression coverage.
+- **Tool policy:** Exclude tools entirely or preserve selected original descriptions with case-insensitive wildcard patterns.
 
-The package also installs `mcp-cli`, a second entry point for agents that
-would rather *shell out* than hold every tool definition in context. Instead
-of loading all tools up front, the agent lists or searches for what it needs
-and pulls the full schema only for the tool it is about to call.
+## 🏗️ How it works
 
-A background daemon keeps warm connections to the backend MCP servers, so
-each command is a short IPC round-trip rather than a fresh server startup.
-
-```bash
-mcp-cli tools                          # List all tools (compressed)
-mcp-cli search <query>                 # Search tools by name/description
-mcp-cli info <server>/<tool>           # Full schema for one tool
-mcp-cli call <server>/<tool> '<json>'  # Execute a tool
-mcp-cli stats                          # Compression statistics
-
-mcp-cli daemon start                   # Start the background daemon
-mcp-cli daemon status                  # Show daemon status
-mcp-cli daemon stop                    # Stop the daemon
+```mermaid
+flowchart TD
+    A["Agent"] --> B{"Access mode"}
+    B --> C["mcp-cli"]
+    B --> D["Native MCP proxy"]
+    C --> E["Configured MCP servers"]
+    D --> E
 ```
 
-The daemon starts automatically on first use. Pass `--no-auto-start` to fail
-fast instead when it is not already running.
+`mcp-cli` uses a local daemon so repeated commands are short IPC round trips. Native clients launch the stdio proxy and see one namespaced MCP tool catalog.
 
-`call` also accepts its JSON argument on stdin, which avoids shell quoting
-problems with large payloads:
+## ⌨️ CLI reference
 
-```bash
-echo '{"path": "/tmp/notes.md"}' | mcp-cli call filesystem/read_file
-```
+| Command                                           | Purpose                                           |
+| ------------------------------------------------- | ------------------------------------------------- |
+| `mcp-cli search <query>`                          | Search tool names and descriptions                |
+| `mcp-cli info <server>/<tool>`                    | Load the full schema for one tool                 |
+| `mcp-cli call <server>/<tool> '<json>'`           | Execute a tool                                    |
+| `mcp-cli tools`                                   | List compact summaries for every available tool   |
+| `mcp-cli output find <id> <query>`                | Search a cached large output                      |
+| `mcp-cli output read <id> [offset] [length\|all]` | Read a bounded page or the remainder of an output |
+| `mcp-cli script '<json>'`                         | Run a declarative sequence of calls               |
+| `mcp-cli stats`                                   | Show server and compression statistics            |
+| `mcp-cli doctor`                                  | Validate configuration and backend health         |
+| `mcp-cli daemon status`                           | Show daemon and connection lifecycle state        |
+| `mcp-cli daemon logs [-n N] [-f]`                 | Read or follow daemon logs                        |
+| `mcp-cli daemon restart`                          | Restart the local daemon                          |
 
-### CLI Configuration
+Pass call JSON on stdin when shell quoting becomes awkward:
 
-The optional `cli` block in `servers.json` tunes daemon behavior:
-
-```json
-{
-  "cli": {
-    "payloadThreshold": 500,
-    "autoStartDaemon": true,
-    "daemonLogLevel": "info"
-  },
-  "mcpServers": []
-}
+```bash
+echo '{"path":"/tmp/notes.md"}' | mcp-cli call filesystem/read_file
 ```
 
-| Field | Type | Default | Description |
-|-------|------|---------|-------------|
-| `payloadThreshold` | number | `500` | Tool output longer than this many characters is written to a temp file and replaced with a reference, keeping large payloads out of the agent's context |
-| `autoStartDaemon` | boolean | `true` | Start the daemon automatically when a command needs it |
-| `daemonLogLevel` | string | `"info"` | Daemon log level (`debug`, `info`, `warn`, `error`) |
-
-Daemon state lives in `~/.mcp-compression-proxy/` (socket, PID file, and
-`daemon.log`), created with `0700` permissions since the control socket
-accepts commands that execute downstream MCP tools.
-
-## 🔧 Configuration
+Pass `--no-auto-start` when a command should fail instead of starting the daemon.
 
-### Server Configuration
+### Large outputs
 
-Create or edit your JSON configuration file at:
-- `~/.mcp-compression-proxy/servers.json` (user-level), or
-- `./servers.json` (project-level)
+Tool results longer than 10,000 characters are stored under `~/.mcp-compression-proxy/payloads/` by default. The CLI returns a payload ID instead of flooding the agent's context:
 
-```json
-{
-  "mcpServers": [
-    {
-      "name": "my-server",
-      "command": "command-to-run",
-      "args": ["arg1", "arg2"],
-      "env": {
-        "ENV_VAR": "value"
-      },
-      "enabled": true
-    }
-  ]
-}
-```
-
-#### Configuration Schema
-
-**Root Level:**
-| Field | Type | Required | Description |
-|-------|------|----------|-------------|
-| `mcpServers` | array | ✅ | Array of server configurations |
-| `excludeTools` | string[] | ❌ | Tool name patterns to exclude from tool list entirely (supports wildcards) |
-| `noCompressTools` | string[] | ❌ | Tool name patterns to never compress - descriptions pass through unchanged (supports wildcards) |
-| `defaultTimeout` | number | ❌ | Default timeout in seconds for all servers (default: 30). Can be overridden per-server. |
-| `inheritEnv` | boolean \| string[] | ❌ | Which of the proxy's environment variables backend servers inherit (default: `true`). Can be overridden per-server. |
-| `compressionFallbackBehavior` | `"original"` \| `"blank"` | ❌ | What to show for a tool that has not been compressed yet (default: `"original"`) |
-
-**Server Configuration:**
-| Field | Type | Required | Description |
-|-------|------|----------|-------------|
-| `name` | string | ✅ | Unique server identifier |
-| `command` | string | ✅ | Command to execute |
-| `args` | string[] | ❌ | Command arguments |
-| `env` | object | ❌ | Environment variables |
-| `inheritEnv` | boolean \| string[] | ❌ | Overrides the root-level `inheritEnv` for this server |
-| `enabled` | boolean | ❌ | Enable/disable server (default: true) |
-| `timeout` | number | ❌ | Server-specific timeout in seconds (overrides `defaultTimeout`) |
-
-#### Environment Variable Expansion
-
-Use `${VAR_NAME}` syntax to reference environment variables:
-
-```json
-{
-  "mcpServers": [
-    {
-      "name": "github",
-      "command": "npx",
-      "args": ["-y", "@modelcontextprotocol/server-github"],
-      "env": {
-        "GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}",
-        "GITHUB_ORG": "${MY_GITHUB_ORG}"
-      }
-    }
-  ]
-}
+```bash
+mcp-cli output find <payload-id> "needle"
+mcp-cli output read <payload-id> 0 10000
+mcp-cli output read <payload-id> 10000 all
 ```
 
-Variables are expanded at runtime from your shell environment.
+The payload directory is mode `0700` and payload files are mode `0600`. Up to 100 entries are retained by the running process; the oldest are evicted first.
 
-Two extra forms are supported:
+### Declarative call chains
 
-| Syntax | Meaning |
-|--------|---------|
-| `${VAR}` | Substitutes `VAR`, or an empty string with a warning if it is not set |
-| `${VAR:-fallback}` | Substitutes `VAR`, or `fallback` when unset or empty |
-| `$${VAR}` | Escapes the expansion — produces the literal text `${VAR}` |
-
-If a `${VAR}` reference cannot be resolved, the proxy logs a warning naming
-the variable. An unset variable becomes an empty string, which downstream
-servers usually report as an authentication failure rather than a config
-error, so check the startup log first when a server rejects valid-looking
-credentials.
-
-#### What Backend Servers Inherit
-
-Backend servers inherit the proxy's full environment by default, so variables
-you exported in your shell are visible to them without being listed in `env`.
-Entries in `env` always take precedence over inherited values.
-
-Narrow this with `inheritEnv` when a server should not see unrelated secrets:
+Scripts can run up to 20 sequential MCP calls. A later step may use a prior JSON value through an [RFC 6901 JSON Pointer](https://www.rfc-editor.org/rfc/rfc6901):
 
 ```json
 {
-  "inheritEnv": ["HOME", "PATH", "LANG"],
-  "mcpServers": [
+  "steps": [
     {
-      "name": "trusted",
-      "command": "npx",
-      "args": ["-y", "@modelcontextprotocol/server-github"],
-      "inheritEnv": true
+      "id": "search",
+      "server": "docs",
+      "tool": "search",
+      "arguments": { "query": "progressive discovery" }
     },
     {
-      "name": "untrusted",
-      "command": "some-third-party-server",
-      "inheritEnv": false,
-      "env": { "API_KEY": "${THIRD_PARTY_KEY}" }
+      "id": "read",
+      "server": "docs",
+      "tool": "read",
+      "arguments": {
+        "url": { "$ref": "search#/results/0/url" }
+      }
     }
   ]
 }
 ```
 
-| Value | Effect |
-|-------|--------|
-| `true` (default) | Pass the proxy's full environment through |
-| `false` | Pass only the stdio transport's safe defaults (`PATH`, `HOME`, `SHELL`, …) |
-| `string[]` | Pass only the named variables, plus the transport's safe defaults |
-
-#### Uncompressed Tool Descriptions
+Scripts stop on the first failed step unless that step sets `continueOnError`. References substitute exact values; they do not transform data.
 
-Before a tool has been compressed, the proxy shows its original description.
-Set `compressionFallbackBehavior` to `"blank"` to show nothing instead, so
-uncompressed tools cost no context while you work through them:
+## 🔧 Configuration
 
-```json
-{
-  "compressionFallbackBehavior": "blank",
-  "mcpServers": []
-}
-```
+The proxy reads both of these files when present:
 
-This affects only tools with no cached compressed description. Compressed
-tools, expanded tools, and anything matching `noCompressTools` are unchanged.
+- `~/.mcp-compression-proxy/servers.json` for user-level servers and defaults.
+- `./servers.json` for project-specific additions and overrides.
 
-#### Server Initialization and Timeouts
+Servers and tool patterns from both files are combined. Project-level scalar settings override user-level settings. Configuration changes are watched and healthy backends remain connected when their definitions have not changed.
 
-The proxy initializes all configured MCP servers in **parallel** before becoming ready. Each server connection is wrapped with a timeout to prevent indefinite hanging:
+### Local and remote servers
 
-- **Default timeout**: 30 seconds (if not specified)
-- **Global timeout**: Set `defaultTimeout` in config to change the default for all servers
-- **Per-server timeout**: Set `timeout` on individual servers to override the default
+Each server must define exactly one transport: `command` for local stdio or `url` for Streamable HTTP.
 
 ```json
 {
-  "defaultTimeout": 60,
   "mcpServers": [
     {
-      "name": "fast-server",
+      "name": "filesystem",
       "command": "npx",
-      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
+      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"],
+      "inheritEnv": false
     },
     {
-      "name": "slow-server",
-      "command": "python",
-      "args": ["slow_mcp_server.py"],
-      "timeout": 120
-    }
-  ]
-}
-```
-
-**Behavior:**
-- All servers initialize in parallel (not sequentially)
-- If a server exceeds its timeout, it's marked as failed but doesn't block other servers
-- The proxy reports ready only after all servers have either connected or timed out
-- This ensures all available tools are loaded before the MCP client can query them
-
-**Why this matters:** Without proper timeout handling, a single hanging server could make the entire proxy unresponsive.
-
-#### Tool Filtering Patterns
-
-**Exclude Tools** - Remove tools from the tool list entirely:
-
-Use the `excludeTools` field to filter out unwanted tools using wildcard patterns (case-insensitive):
-
-```json
-{
-  "mcpServers": [...],
-  "excludeTools": [
-    "github__delete_*",     // Exclude all GitHub delete tools
-    "*__experimental*",     // Exclude all experimental tools
-    "filesystem__write_*",  // Exclude filesystem write tools
-    "set_*"                 // Exclude management tools starting with set_
-  ]
-}
-```
-
-**No-Compress Tools** - Keep tools but never compress their descriptions:
-
-Use the `noCompressTools` field to bypass compression for specific tools (descriptions pass through unchanged):
-
-```json
-{
-  "mcpServers": [...],
-  "noCompressTools": [
-    "filesystem__*",        // Never compress filesystem tool descriptions
-    "*__help",              // Never compress help commands
-    "github__search_*"      // Never compress GitHub search tools
-  ]
-}
-```
-
-**Pattern Examples:**
-- `"serverName__*"` - All tools from specific server
-- `"*__toolPattern*"` - Tools matching pattern from any server
-- `"exact_tool_name"` - Exact tool name match
-
-**Use Cases:**
-- **excludeTools**: Remove dangerous tools, unwanted features, or tools not relevant to your workflow
-- **noCompressTools**: Preserve detailed descriptions for complex tools where compression might lose important information
-
-#### Configuration Aggregation
-
-Both config files are loaded and combined:
-
-1. Load user config (`~/.mcp-compression-proxy/servers.json`)
-2. Load project config (`./servers.json`)
-3. Aggregate servers from both configs
-4. Aggregate exclude and noCompress patterns from both configs
-5. Apply exclude patterns to filter tools
-6. Apply noCompress patterns to bypass compression
-
-This allows:
-- Personal defaults in user config
-- Team/project-specific servers in project config
-- Fine-grained tool filtering with exclude patterns
-- Selective compression bypass with noCompress patterns
-
-### Environment Variables
-
-**For the compression proxy** (set in your MCP client config):
-- `LOG_LEVEL` - Logging level (debug, info, warn, error). Default: `info`
-
-**For MCP servers** (set in `servers.json` using `${VAR_NAME}` syntax):
-- `GITHUB_TOKEN` - GitHub personal access token (if using GitHub MCP server)
-- Any other environment variables required by your configured MCP servers
-
-See [Environment Variable Expansion](#environment-variable-expansion) for details on using variables in your server configuration.
-
-### Command-Line Options
-
-**`--clear-cache`** - Clear the persistent compression cache and exit
-
-```bash
-# If installed via npm
-mcp-compression-proxy --clear-cache
-
-# If installed from source
-node dist/index.js --clear-cache
-```
-
-### Debugging
-
-**1. Enable debug logging** in your MCP client config:
-
-```json
-{
-  "mcpServers": {
-    "compression-proxy": {
-      "command": "mcp-compression-proxy",
-      "env": {
-        "LOG_LEVEL": "debug"
-      }
+      "name": "remote-docs",
+      "url": "https://mcp.example.com/mcp",
+      "headers": {
+        "Authorization": "Bearer ${DOCS_TOKEN}"
+      },
+      "timeout": 30
     }
+  ],
+  "excludeTools": ["*__delete_*", "*__experimental*"],
+  "noCompressTools": ["filesystem__write_file"],
+  "cli": {
+    "payloadThreshold": 10000,
+    "autoStartDaemon": true,
+    "daemonLogLevel": "info"
   }
 }
 ```
 
-**2. View logs** (for Claude Desktop):
-- **macOS**: `~/Library/Logs/Claude/mcp*.log`
-- **Windows**: `%APPDATA%\Claude\Logs\mcp*.log`
-
-**3. Check for common issues:**
-- Ensure all configured MCP servers are accessible and properly configured
-- Verify environment variables are correctly expanded
-- Check that Node.js version is 18 or higher
-
-## 💡 Best Practices
+Remote servers support static headers. The proxy does not perform an interactive OAuth redirect, so use bearer tokens or API-key headers supported by the remote endpoint.
 
-### Good Compression
+Environment references work in `env` and `headers`:
 
-**Preserves**:
-- Core functionality
-- Key parameters
-- Critical constraints
-- Return types
+| Syntax              | Result                                                     |
+| ------------------- | ---------------------------------------------------------- |
+| `${NAME}`           | Use `NAME`; warn and substitute an empty string when unset |
+| `${NAME:-fallback}` | Use `NAME`, or `fallback` when unset or empty              |
+| `$${NAME}`          | Preserve the literal text `${NAME}`                        |
 
-**Removes**:
-- Verbose explanations
-- Redundant phrases
-- Non-critical examples
-- Marketing language
+Local servers inherit the proxy's environment by default. Set `inheritEnv` to `false` for transport-safe defaults only, or provide an array such as `["HOME", "PATH", "LANG"]`. Explicit values in a server's `env` object always win.
 
-### Example
+### Core options
 
-**Original** (42 tokens):
-```
-"Searches for files in the specified directory and its subdirectories using
-glob patterns. Supports wildcards like *, **, and ?. Returns an array of
-matching file paths. Case-sensitive by default."
-```
+| Option                        | Default      | Purpose                                                     |
+| ----------------------------- | ------------ | ----------------------------------------------------------- |
+| `defaultTimeout`              | `30`         | Backend timeout in seconds; overridable per server          |
+| `excludeTools`                | `[]`         | Hide matching `server__tool` names completely               |
+| `noCompressTools`             | `[]`         | Always show original descriptions for matching tools        |
+| `compressionFallbackBehavior` | `"original"` | Show `"original"` or `"blank"` before compression exists    |
+| `inheritEnv`                  | `true`       | Control which environment variables local servers receive   |
+| `softMaxConnectionAgeSeconds` | `3600`       | Lazily replace a connection on its next use after this age  |
+| `hardMaxConnectionAgeSeconds` | `28800`      | Drain a connection at this age and reopen it on demand      |
+| `authErrorPatterns`           | `[]`         | Identify authentication failures in errors or tool results  |
+| `authRetryTools`              | `[]`         | Name tools safe to retry once after authentication recovery |
+| `cli.payloadThreshold`        | `10000`      | Store larger outputs in private local files                 |
+| `cli.autoStartDaemon`         | `true`       | Start the daemon when a CLI command needs it                |
+| `cli.daemonLogLevel`          | `"info"`     | Set `debug`, `info`, `warn`, or `error` logging             |
 
-**Compressed** (12 tokens):
-```
-"Search files by glob pattern (*, **, ?), case-sensitive, returns paths"
-```
+Set either connection age to `0` to disable that policy. Lifecycle and authentication options may also be set per server. Authentication failures always replace the backend generation that produced them, but automatic replay occurs only for names matching `authRetryTools`.
 
-## ❓ FAQ
+See [`servers.json.example`](servers.json.example) for a complete starting point.
 
 <details>
-<summary><strong>Q: What MCP clients are supported?</strong></summary>
-<p>Any MCP-compatible client: Claude Desktop, Cline, Continue.dev, or custom agents.</p>
-</details>
+<summary><strong>🚦 Versioned daemon deployments</strong></summary>
 
-<details>
-<summary><strong>Q: How much context does compression save?</strong></summary>
-<p>Typically 50-80% reduction in token count for tool listings while preserving critical information.</p>
-</details>
+The daemon exposes a contract for a separate stable router to run candidate and active releases on different sockets. Configure each instance with `MCP_DAEMON_SOCKET_PATH`, `MCP_DAEMON_PID_FILE`, `MCP_DAEMON_READY_FILE`, `MCP_DAEMON_LOG_FILE`, and `MCP_DAEMON_RELEASE_ID`.
 
-<details>
-<summary><strong>Q: Do I need to restart after adding servers?</strong></summary>
-<p>Yes, restart your MCP client to load the new configuration. No rebuild needed when using JSON configuration.</p>
-</details>
+`MCP_DAEMON_BASE_DIR` selects the shared state root, while `MCP_PAYLOAD_DIR` can preserve payload IDs across a release cutover. When `active-release.json` exists in that root, `mcp-cli` treats the installation as router-managed and refuses to start a legacy daemon on the stable socket.
 
-<details>
-<summary><strong>Q: Can I use multiple MCP servers?</strong></summary>
-<p>Yes! That's the primary use case. Add as many as you need in your <code>servers.json</code> configuration file.</p>
-</details>
+This allows an external router to canary a candidate, switch new requests atomically, drain calls pinned to the old release, and roll back without terminating in-flight work.
 
-<details>
-<summary><strong>Q: Is compression permanent?</strong></summary>
-<p>Compressed descriptions are persisted to disk at <code>~/.mcp-compression-proxy/cache.json</code> and automatically restored on server restart. Session-based expansions are temporary and reset per session.</p>
 </details>
 
-<details>
-<summary><strong>Q: Where is the compression cache stored?</strong></summary>
-<p>Cache is stored at <code>~/.mcp-compression-proxy/cache.json</code>. Use <code>--clear-cache</code> flag to clear it if needed.</p>
-</details>
+## 🛡️ Security and privacy
 
-<details>
-<summary><strong>Q: Works with local LLMs?</strong></summary>
-<p>Yes! Works with any MCP-compatible setup, including local models.</p>
-</details>
+- The proxy runs locally and does not require a hosted control plane.
+- Tool inputs and outputs go only to the backend servers you configure; remote backends naturally receive calls addressed to them.
+- Large outputs are stored in owner-only files and are retrieved by opaque payload ID, not arbitrary path.
+- The daemon's local control socket can execute downstream MCP tools and is kept inside an owner-only directory.
+- Use `inheritEnv: false` or an allowlist when a third-party local server should not receive unrelated environment variables.
+- MCP Compression Proxy is a transport and lifecycle layer, not a sandbox or authorization boundary. Apply normal trust and permission controls to every backend server.
 
-<details>
-<summary><strong>Q: How do I add a new MCP server?</strong></summary>
-<p>Edit your <code>servers.json</code> configuration file (in <code>~/.mcp-compression-proxy/</code> or project root), add your server config, and restart your MCP client. No rebuild needed.</p>
-</details>
+See [SECURITY.md][security] to report a vulnerability.
 
-**More:** See [CONTRIBUTING.md][contributing], [SECURITY.md][security], [tests/README.md](./tests/README.md)
+## 🩺 Troubleshooting
 
-## 🧪 Testing
-
-Comprehensive test suite included:
+Start with:
 
 ```bash
-npm test                      # Run all tests
-npm run test:unit             # Unit tests only
-npm run test:integration      # Integration tests only
-npm run test:e2e              # End-to-end tests only
-npm run test:e2e:real-llm     # Real LLM integration tests (requires Ollama)
-npm run test:coverage         # Generate coverage report
+mcp-cli doctor
+mcp-cli daemon status
+mcp-cli daemon logs -n 100
 ```
 
-See [tests/README.md](./tests/README.md) for details.
+- After changing daemon-specific settings, run `mcp-cli daemon restart`.
+- Clear saved descriptions with `mcp-compression-proxy --clear-cache`.
+- Native MCP logs go to stderr so stdout remains valid JSON-RPC.
+- A restricted agent sandbox may block the daemon's Unix socket. Grant access to `~/.mcp-compression-proxy/` or run the CLI in the host environment.
 
 ## 🤝 Contributing
 
-Contributions welcome! See [CONTRIBUTING.md][contributing] for guidelines.
-
-Quick start:
-
-1. Fork the repository
-2. Create your feature branch
-3. Make your changes and test
-4. Commit using [Conventional Commits](https://www.conventionalcommits.org/)
-5. Open a Pull Request
-
-Note: This project follows a [Code of Conduct][code-of-conduct].
-
-## 🔗 Complementary Projects
+Found a bug 🐛, have an idea ✨, or want to improve the docs? Contributions are welcome. Read [CONTRIBUTING.md][contributing], follow the [Code of Conduct][code-of-conduct], and run the checks before opening a pull request:
 
-**Maximize your MCP workflow with these complementary tools:**
-
-### [Local Skills MCP][local-skills-mcp]
-
-**Portable, reusable prompt libraries for any MCP client**
-
-While MCP Compression Proxy optimizes your tool descriptions, [Local Skills MCP][local-skills-mcp] provides expert-level prompt instructions that work across any MCP-compatible client.
-
-**Perfect combination:**
-- **MCP Compression Proxy** - Aggregates and compresses tool descriptions (50-80% token reduction)
-- **Local Skills MCP** - Provides expert skills with lazy loading (~50 tokens/skill)
+```bash
+npm install
+npm run typecheck
+npm run lint
+npm test
+```
 
-**Together they enable:**
-- 🎯 Optimized context usage across tools AND prompts
-- 🔄 Portable workflows that work with Claude, Cline, Continue.dev, and more
-- ⚡ Efficient AI interactions with minimal context consumption
-- 🚀 Professional-grade AI agent capabilities
+Additional test guidance is in [`tests/README.md`](tests/README.md).
 
-[Learn more about Local Skills MCP →][local-skills-mcp]
+## 💖 Support the project
 
-## 💖 Support This Project
+Open source grows through the people who try it, share it, and improve it. If MCP Compression Proxy gives your agent some breathing room:
 
-If you find MCP Compression Proxy useful, please consider supporting its development!
+- ⭐ **[Star the repository][stargazers]** so more MCP users can discover it.
+- 🐛 **[Report a bug or suggest an idea][repo-issues]** to help shape the roadmap.
+- 📝 **[Contribute code or documentation][contributing]**—first-time contributors are welcome.
+- 💖 **Sponsor continued development** using any of the options below.
 
 <div align="center">
 
-[![GitHub Sponsors][sponsor-github-badge]][sponsor-github]
-[![Buy Me A Coffee][sponsor-coffee-badge]][sponsor-coffee]
+[![GitHub Stars][stars-badge]][stargazers]
+[![Sponsor on GitHub][sponsor-github-badge]][sponsor-github]
+[![Buy Me a Coffee][sponsor-coffee-badge]][sponsor-coffee]
 [![PayPal][sponsor-paypal-badge]][sponsor-paypal]
 
-</div>
-
-**Ways to support:**
+**Every star, issue, pull request, and contribution helps. Thank you! 🙌**
 
-- ⭐ [Star this repository][stargazers]
-- 💰 Sponsor via the badges above
-- 🐛 [Report bugs and suggest features][repo-issues]
-- 📝 [Contribute code or documentation][contributing]
+</div>
 
 ## 📄 License
 
-MIT License - see [LICENSE][license-file] file. **Copyright © 2025 KDPA**
-
-## 🙏 Acknowledgments
-
-Built with [Model Context Protocol SDK][mcp-sdk]
+[MIT](LICENSE) © 2025 KDPA. Built with the [Model Context Protocol TypeScript SDK][mcp-sdk].
 
 ---
 
 <div align="center">
 
-**[⬆ Back to Top](#-mcp-compression-proxy)**
+[⬆ Back to top](#-mcp-compression-proxy)
 
 Made with ❤️ by KDPA
 
 </div>
 
-<!-- Reference Links -->
-<!-- Badges - Top of README -->
+<!-- Reference links -->
 
 [npm-version-badge]: https://img.shields.io/npm/v/mcp-compression-proxy.svg
 [npm-package]: https://www.npmjs.com/package/mcp-compression-proxy
 [npm-downloads-badge]: https://img.shields.io/npm/dm/mcp-compression-proxy
-[npm-types-badge]: https://img.shields.io/npm/types/mcp-compression-proxy
-[license-badge]: https://img.shields.io/badge/License-MIT-yellow.svg
-[license]: https://opensource.org/licenses/MIT
 [node-badge]: https://img.shields.io/badge/node-%3E%3D22.0.0-brightgreen.svg
 [nodejs]: https://nodejs.org/
-[mcp-badge]: https://img.shields.io/badge/MCP-Compatible-purple.svg
-[mcp-protocol]: https://modelcontextprotocol.io/
-[mcp-sampling]: https://modelcontextprotocol.io/docs/concepts/sampling
-
-<!-- CI/CD Badges -->
-
 [ci-badge]: https://github.com/kdpa-llc/mcp-compression-proxy/actions/workflows/test.yml/badge.svg
 [ci-workflow]: https://github.com/kdpa-llc/mcp-compression-proxy/actions/workflows/test.yml
 [codecov-badge]: https://codecov.io/gh/kdpa-llc/mcp-compression-proxy/branch/main/graph/badge.svg
 [codecov]: https://codecov.io/gh/kdpa-llc/mcp-compression-proxy
-[codeql-badge]: https://github.com/kdpa-llc/mcp-compression-proxy/actions/workflows/codeql.yml/badge.svg
-[codeql-workflow]: https://github.com/kdpa-llc/mcp-compression-proxy/actions/workflows/codeql.yml
-
-<!-- GitHub Badges -->
-
+[license-badge]: https://img.shields.io/badge/License-MIT-yellow.svg
+[license-file]: LICENSE
+[repo]: https://github.com/kdpa-llc/mcp-compression-proxy
 [stars-badge]: https://img.shields.io/github/stars/kdpa-llc/mcp-compression-proxy?style=social
 [stargazers]: https://github.com/kdpa-llc/mcp-compression-proxy/stargazers
-[forks-badge]: https://img.shields.io/github/forks/kdpa-llc/mcp-compression-proxy?style=social
-[network]: https://github.com/kdpa-llc/mcp-compression-proxy/network/members
-[issues-badge]: https://img.shields.io/github/issues/kdpa-llc/mcp-compression-proxy
 [repo-issues]: https://github.com/kdpa-llc/mcp-compression-proxy/issues
-[commit-badge]: https://img.shields.io/github/last-commit/kdpa-llc/mcp-compression-proxy
-[commits]: https://github.com/kdpa-llc/mcp-compression-proxy/commits/main
-[prs-badge]: https://img.shields.io/badge/PRs-welcome-brightgreen.svg
-
-<!-- Repository Links -->
-
-[repo]: https://github.com/kdpa-llc/mcp-compression-proxy
-[pulls]: https://github.com/kdpa-llc/mcp-compression-proxy/pulls
-
-<!-- Documentation Links -->
-
 [contributing]: CONTRIBUTING.md
 [security]: SECURITY.md
 [code-of-conduct]: CODE_OF_CONDUCT.md
-[license-file]: LICENSE
-
-<!-- Sponsorship Links -->
-
 [sponsor-github-badge]: https://img.shields.io/badge/Sponsor-GitHub%20Sponsors-ea4aaa?logo=github
 [sponsor-github]: https://github.com/sponsors/moscaverd
-[sponsor-coffee-badge]: https://img.shields.io/badge/Buy%20Me%20A%20Coffee-support-yellow?logo=buy-me-a-coffee
+[sponsor-coffee-badge]: https://img.shields.io/badge/Buy%20Me%20a%20Coffee-support-yellow?logo=buy-me-a-coffee
 [sponsor-coffee]: https://buymeacoffee.com/moscaverd
 [sponsor-paypal-badge]: https://img.shields.io/badge/PayPal-donate-blue?logo=paypal
 [sponsor-paypal]: https://paypal.me/moscaverd
-
-<!-- External Links -->
-
-[mcp-sdk]: https://github.com/modelcontextprotocol/sdk
-[local-skills-mcp]: https://github.com/kdpa-llc/local-skills-mcp
+[mcp-sdk]: https://github.com/modelcontextprotocol/typescript-sdk
+[mcp-sampling]: https://modelcontextprotocol.io/specification/2026-07-28/client/sampling