openclaw-mcp-router 0.2.6 → 1.1.0

package/.github/pull_request_template.md

@@ -0,0 +1,14 @@
+## Summary
+- 
+
+## Why
+- 
+
+## Validation
+- [ ] `npm test` passes locally
+
+## Documentation Checklist
+- [ ] README updated (if behavior/config/user workflow changed)
+- [ ] docs/ updated to match README and implementation
+- [ ] No README/docs drift introduced
+- [ ] skills/ content updated to match behavior/config changes (if applicable)

package/README.md

@@ -1,207 +1,159 @@
-# openclaw-mcp-router
+# OpenClaw MCP Router 🚀
 
-Dynamic MCP tool router for [OpenClaw](https://openclaw.ai) — semantic search over large MCP tool catalogs to eliminate context bloat.
+OpenClaw MCP Router is an OpenClaw plugin that keeps MCP tool catalogs out of the system prompt until needed.
 
-## The problem
+Instead of injecting every MCP schema up front, it provides two lightweight meta-tools:
 
-Loading all MCP tool schemas upfront burns 55k–134k tokens before your agent processes a single message. With 58 tools across 5 MCP servers, that's ~77k tokens wasted on schemas the agent may never use.
+- `mcp_search` → discover the right tool at runtime
+- `mcp_call` → execute as JSON fallback
 
-## The solution
+This cuts context bloat and improves tool selection quality on large MCP catalogs.
 
-Two tiny tools replace the full schema dump:
+---
 
-- **`mcp_search(query)`** — embed the query via Ollama, search a local LanceDB index, return only matching tool definitions (~8.7k tokens, 95% reduction)
-- **`mcp_call(tool_name, params_json)`** — look up the owning MCP server, execute the call, return the result
+## Why this exists
 
-The agent asks for tools it needs instead of receiving every schema upfront.
+Large MCP catalogs are expensive in prompt space.
 
-## Install
+- **Token waste:** tens of thousands of tokens before first user turn
+- **Reasoning quality loss:** "lost in the middle" on oversized prompts
+- **Higher cost:** more prompt tokens every turn
 
-```sh
-openclaw plugins install openclaw-mcp-router
-```
-
-**Alternative: install from source**
-
-```bash
-git clone https://github.com/lunarmoon26/openclaw-mcp-router.git
-openclaw plugins install ./openclaw-mcp-router
-```
-
-Requires [Ollama](https://ollama.ai) running locally with an embedding model:
+MCP Router applies Anthropic's tool-search pattern so only relevant tools are surfaced when needed.
 
-```sh
-ollama pull embeddinggemma
-ollama serve
-```
-
-## Configuration
-
-Add to `~/.openclaw/openclaw.yml`:
-
-```yaml
-tools:
-  alsoAllow:
-    - mcp_search
-    - mcp_call
-
-plugins:
-  openclaw-mcp-router:
-    enabled: true
-    config:
-      servers:
-        - name: filesystem
-          transport: stdio
-          command: npx
-          args: ["-y", "@modelcontextprotocol/server-filesystem", "/tmp"]
-        - name: github
-          transport: sse
-          url: https://api.githubcopilot.com/mcp/
-      embedding:
-        provider: ollama
-        model: embeddinggemma    # or qwen3-embedding:0.6b, all-minilm
-        url: http://localhost:11434
-      search:
-        topK: 5        # tools returned per search (1–20)
-        minScore: 0.3  # minimum similarity threshold (0–1)
-```
+Refs:
+- Tool search / advanced tool use: <https://www.anthropic.com/engineering/advanced-tool-use>
+- Code execution with MCP: <https://www.anthropic.com/engineering/code-execution-with-mcp>
 
-### Important: `tools.alsoAllow` is required
+---
 
-The plugin registers `mcp_search` and `mcp_call` as **optional tools** (`optional: true`). This means the gateway loads them, but they are **not exposed to agents** unless explicitly allowlisted.
-
-If the plugin is running and `openclaw openclaw-mcp-router stats` shows indexed tools, but your agent can't call `mcp_search` — this is why. Add the allowlist to your config:
-
-```yaml
-# Global — all agents get access
-tools:
-  alsoAllow:
-    - mcp_search
-    - mcp_call
-```
+## Core model
 
-Or scope it to specific agents:
+### 1) Index time (`reindex`)
+- Connect to configured MCP servers
+- List tools
+- Embed tool text
+- Store vectors in LanceDB
+- Register tool→server ownership
+- *(Optional)* generate CLI artifacts via `mcporter generate-cli`
 
-```yaml
-# Per-agent or under agents.defaults
-agents:
-  defaults:
-    tools:
-      alsoAllow:
-        - mcp_search
-        - mcp_call
-```
+### 2) Runtime (`mcp_search`)
+- Semantic search over indexed tools
+- Default schema verbosity is adaptive:
+  - if `mcporter` is available: compact cards by default
+  - if `mcporter` is not available: include JSON params by default
+- Full JSON schema can always be forced with `include_schema=true`
 
-Restart the gateway after changing the config.
+### 3) Execute (`mcp_call`)
+- JSON-based execution path (classic MCP params flow)
 
-## Configuration reference
+---
 
-### `servers[]`
+## CLI-first behavior (new)
 
-| Field | Required | Description |
-|-------|----------|-------------|
-| `name` | yes | Unique server identifier |
-| `transport` | yes | `stdio`, `sse`, or `http` |
-| `command` | stdio only | Executable to run |
-| `args` | stdio only | Arguments array |
-| `env` | no | Extra env vars merged over process.env; supports `${VAR}` expansion |
-| `url` | sse/http only | Server endpoint URL |
-| `timeout` | no | Per-server connect timeout in ms; overrides `indexer.connectTimeout` |
+Router is now optimized for a CLI-first workflow:
 
-### `embedding`
+- Prefer: `mcporter call <server>.<tool> ...`
+- Fallback: `mcp_call(tool_name, params_json)`
 
-| Field | Default | Description |
-|-------|---------|-------------|
-| `provider` | `ollama` | Only Ollama is supported |
-| `model` | `embeddinggemma` | Embedding model name |
-| `url` | `http://localhost:11434` | Ollama base URL (must be localhost) |
+`mcp_search` adapts to environment: compact when mcporter is present, schema-forward when it is not (so agents can drive `mcp_call` reliably).
 
-### `vectorDb`
+---
 
-| Field | Default | Description |
-|-------|---------|-------------|
-| `path` | `~/.openclaw/openclaw-mcp-router/lancedb` | LanceDB database directory |
+## Quick start
 
-### `indexer`
+### Prerequisites
 
-Controls retry behavior and timeouts when connecting to MCP servers at startup. Useful for self-hosted servers (e.g. started via `uvx`) that take time to start up.
+```bash
+ollama pull embeddinggemma
+```
 
-| Field | Default | Description |
-|-------|---------|-------------|
-| `connectTimeout` | `60000` | Per-server default connect timeout in ms |
-| `maxRetries` | `3` | Retry attempts per server (0 = no retry) |
-| `initialRetryDelay` | `2000` | Initial backoff delay in ms |
-| `maxRetryDelay` | `30000` | Max backoff cap in ms |
-| `maxChunkChars` | `500` | Max characters per chunk for long tool descriptions. `0` = disable chunking |
-| `overlapChars` | `100` | Overlap characters between adjacent chunks |
+### Install
 
-Retries use exponential backoff: delays are `initialRetryDelay * 2^(attempt-1)`, capped at `maxRetryDelay`. With defaults, a slow server gets attempts at ~0s, ~2s, ~4s, ~8s before giving up.
+```bash
+openclaw plugins install openclaw-mcp-router
+```
 
-**Chunking:** When a tool description exceeds `maxChunkChars`, it is split into overlapping chunks at semantic boundaries (paragraphs, lines, sentences). Each chunk is stored as a separate vector, and search results are deduplicated so each tool appears once with its best matching score. Short descriptions (the common case) are unaffected.
+### Setup + index
 
-Example for a slow-starting Python server:
+```bash
+openclaw openclaw-mcp-router setup
+openclaw openclaw-mcp-router reindex
+```
 
-```yaml
-plugins:
-  openclaw-mcp-router:
-    config:
-      mcpServers:
-        my-python-server:
-          command: uvx
-          args: ["my-mcp-server"]
-          timeout: 120000  # this server needs 2 minutes to start
-      indexer:
-        maxRetries: 5
-        initialRetryDelay: 3000
+The setup wizard auto-detects whether `mcporter` is installed and suggests a sensible default for `mcp_search` schema verbosity.
+
+---
+
+## Key configuration
+
+In `~/.openclaw/openclaw.json` under `plugins.entries.openclaw-mcp-router.config`:
+
+```json5
+{
+  "search": {
+    "topK": 5,
+    "minScore": 0.3
+    // includeParametersDefault optional:
+    // true  -> always include params
+    // false -> always compact
+    // unset -> auto (based on mcporter availability)
+  },
+  "indexer": {
+    "connectTimeout": 60000,
+    "maxRetries": 3,
+    "initialRetryDelay": 2000,
+    "maxRetryDelay": 30000,
+    "maxChunkChars": 500,
+    "overlapChars": 100,
+    "generateCliArtifacts": false
+  }
+}
 ```
 
-### `search`
+### Notes
 
-| Field | Default | Description |
-|-------|---------|-------------|
-| `topK` | `5` | Max tools returned per search |
-| `minScore` | `0.3` | Minimum similarity score (0–1) |
+- `search.includeParametersDefault` is optional; if omitted, router auto-decides based on mcporter availability.
+- `indexer.generateCliArtifacts=true` enables best-effort per-server `mcporter generate-cli` during reindex.
+- `mcp_call` stays the classic JSON meta-tool (no backend mode flag).
 
-## CLI commands
+---
 
-```sh
-# Re-index all configured MCP servers
-openclaw openclaw-mcp-router reindex
+## Server management
 
-# Show indexed tool count
-openclaw openclaw-mcp-router stats
+```bash
+openclaw openclaw-mcp-router control
+openclaw openclaw-mcp-router list
+openclaw openclaw-mcp-router add <name> <command-or-url> [...]
+openclaw openclaw-mcp-router reindex
 ```
 
-## How it works
-
-1. At gateway startup, the plugin connects to each MCP server in parallel (with retry and configurable timeouts), lists its tools, embeds each description via Ollama, and stores them in a local LanceDB index.
-2. When the agent needs to use an MCP capability, it calls `mcp_search("what I want to do")` to find relevant tools.
-3. The agent then calls `mcp_call("tool_name", '{"param": "value"}')` to execute the chosen tool.
+---
 
-Disabling the plugin (`openclaw plugins disable openclaw-mcp-router`) cancels any in-progress indexing immediately. Re-enabling starts fresh.
+## MCPorter inspiration
 
-## Supported embedding models
+Huge thanks to **@steipete** and [mcporter](https://github.com/steipete/mcporter) for the CLI-first MCP execution model inspiration.
 
-| Model | Dims | Notes |
-|-------|------|-------|
-| `embeddinggemma` | 768 | Good balance, recommended default |
-| `qwen3-embedding:0.6b` | 1024 | Higher quality, larger footprint |
-| `all-minilm` | 384 | Fast and lightweight |
+---
 
-Any Ollama embedding model works — dimensions are detected automatically for unknown models.
+## Documentation
 
-## Background
+- Architecture + flow details: `docs/CLI_FIRST_WORKFLOW.md`
+- Plugin config schema: `openclaw.plugin.json`
+- Skill usage examples: `skills/mcp-router/`
 
-This plugin is a basic implementation of the [Tool Search Tool](https://www.anthropic.com/engineering/advanced-tool-use) pattern from Anthropic's advanced tool use guide. The core idea: instead of injecting all tool schemas into the system prompt, provide a search tool that dynamically surfaces only relevant tools at runtime. Anthropic's benchmarks showed this improved tool selection accuracy from 49% to 74% (Opus 4) and 79.5% to 88.1% (Opus 4.5).
+---
 
-Our approach is intentionally simple — pure vector similarity over tool descriptions. There's plenty of room to improve:
+## Contributing
 
-- **Hybrid search (BM25 + embedding).** Pure embedding search can miss exact keyword matches. Combining sparse retrieval (BM25/TF-IDF) with dense vectors would improve recall, especially for tools with distinctive names like `git_diff` or `kubectl_apply`.
-- **LLM-based reranking.** After the initial vector search returns candidates, a small LLM could rerank them based on the full query context — catching semantic nuances that cosine similarity misses.
-- **Tool use examples.** Indexing not just descriptions but example invocations (input/output pairs) would let the search match against concrete usage patterns, not just what the tool claims to do.
-- **Programmatic tool calling.** Anthropic's guide describes letting the agent compose tool calls inside code blocks rather than pure JSON — reducing context pollution and enabling multi-step tool pipelines.
+PRs welcome — especially around:
+- better reranking
+- hybrid retrieval (vector + lexical)
 
-Contributions welcome if any of these interest you.
+PR hygiene:
+- keep README + docs in sync for every behavior/config/workflow change
+- keep `skills/` guidance in sync when user-facing behavior changes
+- run test suite before opening PR
 
 ## License
 

package/docs/CLI_FIRST_WORKFLOW.md

@@ -0,0 +1,76 @@
+# CLI-First Workflow (MCP Router + MCPorter)
+
+This document explains the intended execution model:
+
+1. Reindex tools + metadata
+2. Search tools with adaptive schema verbosity
+3. Prefer CLI calls when available
+4. Use `mcp_call` as classic JSON fallback
+
+## Reindex behavior
+
+During `openclaw openclaw-mcp-router reindex`:
+
+- Router connects to enabled MCP servers
+- Lists tools
+- Chunks/embeds descriptions
+- Stores vectors + metadata in LanceDB
+
+Optional:
+- If `indexer.generateCliArtifacts=true`, router runs best-effort `mcporter generate-cli` per server.
+- Generation failures do **not** block indexing.
+
+## `mcp_search` behavior
+
+`mcp_search` supports adaptive defaults:
+
+- If `mcporter` is installed: default to compact cards (save tokens)
+- If `mcporter` is not installed: include JSON params by default (so agents can call `mcp_call` reliably)
+
+Overrides:
+- request-level: `include_schema=true|false`
+- config-level: `search.includeParametersDefault=true|false`
+  - if unset, auto mode is used
+
+## `mcp_call` behavior
+
+`mcp_call` remains the original JSON meta-tool path:
+
+- resolve tool owner by registry
+- open MCP connection
+- call tool with `params_json`
+- return content/errors
+
+No backend mode flag is required.
+
+## Recommended config
+
+```json5
+{
+  "plugins": {
+    "entries": {
+      "openclaw-mcp-router": {
+        "enabled": true,
+        "config": {
+          "search": {
+            "topK": 5,
+            "minScore": 0.3
+            // includeParametersDefault optional (true|false)
+          },
+          "indexer": {
+            "generateCliArtifacts": true
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+## Acknowledgement
+
+Special thanks to **@steipete** and **MCPorter**:
+<https://github.com/steipete/mcporter>
+
+Anthropic reference:
+<https://www.anthropic.com/engineering/code-execution-with-mcp>

package/openclaw.plugin.json

@@ -1,7 +1,7 @@
 {
   "id": "openclaw-mcp-router",
   "name": "MCP Router",
-  "description": "Semantic search across MCP tool catalogs. Reduces context bloat from ~77k→8.7k tokens by dynamically surfacing only relevant tools.",
+  "description": "Semantic search across MCP tool catalogs. Reduces context bloat from ~77k\u21928.7k tokens by dynamically surfacing only relevant tools.",
   "configSchema": {
     "type": "object",
     "additionalProperties": false,
@@ -12,36 +12,101 @@
         "additionalProperties": {
           "type": "object",
           "properties": {
-            "command":   { "type": "string" },
-            "args":      { "type": "array", "items": { "type": "string" } },
-            "env":       { "type": "object", "additionalProperties": { "type": "string" } },
-            "url":       { "type": "string" },
-            "serverUrl": { "type": "string" },
-            "type":      { "type": "string", "enum": ["stdio", "sse", "http"] },
-            "headers":   { "type": "object", "additionalProperties": { "type": "string" } },
-            "timeout":   { "type": "number", "description": "Per-server connect timeout in ms; overrides indexer.connectTimeout" }
+            "command": {
+              "type": "string"
+            },
+            "args": {
+              "type": "array",
+              "items": {
+                "type": "string"
+              }
+            },
+            "env": {
+              "type": "object",
+              "additionalProperties": {
+                "type": "string"
+              }
+            },
+            "url": {
+              "type": "string"
+            },
+            "serverUrl": {
+              "type": "string"
+            },
+            "type": {
+              "type": "string",
+              "enum": [
+                "stdio",
+                "sse",
+                "http"
+              ]
+            },
+            "headers": {
+              "type": "object",
+              "additionalProperties": {
+                "type": "string"
+              }
+            },
+            "timeout": {
+              "type": "number",
+              "description": "Per-server connect timeout in ms; overrides indexer.connectTimeout"
+            }
           }
         }
       },
       "mcpServersFile": {
         "type": "string",
-        "description": "Path to a .mcp.json file. Default: ~/.openclaw/.mcp.json"
+        "description": "Path to a .mcp.json file. Default: ~/.openclaw/openclaw-mcp-router/.mcp.json"
       },
       "servers": {
         "type": "array",
         "description": "Legacy server array format. Prefer mcpServers dict.",
         "items": {
           "type": "object",
-          "required": ["name", "transport"],
+          "required": [
+            "name",
+            "transport"
+          ],
           "properties": {
-            "name":      { "type": "string" },
-            "transport": { "type": "string", "enum": ["stdio", "sse", "http"] },
-            "command":   { "type": "string" },
-            "args":      { "type": "array", "items": { "type": "string" } },
-            "env":       { "type": "object", "additionalProperties": { "type": "string" } },
-            "url":       { "type": "string" },
-            "headers":   { "type": "object", "additionalProperties": { "type": "string" } },
-            "timeout":   { "type": "number", "description": "Per-server connect timeout in ms; overrides indexer.connectTimeout" }
+            "name": {
+              "type": "string"
+            },
+            "transport": {
+              "type": "string",
+              "enum": [
+                "stdio",
+                "sse",
+                "http"
+              ]
+            },
+            "command": {
+              "type": "string"
+            },
+            "args": {
+              "type": "array",
+              "items": {
+                "type": "string"
+              }
+            },
+            "env": {
+              "type": "object",
+              "additionalProperties": {
+                "type": "string"
+              }
+            },
+            "url": {
+              "type": "string"
+            },
+            "headers": {
+              "type": "object",
+              "additionalProperties": {
+                "type": "string"
+              }
+            },
+            "timeout": {
+              "type": "number",
+              "description": "Per-server connect timeout in ms; overrides indexer.connectTimeout"
+            }
           }
         }
       },
@@ -49,34 +114,98 @@
         "type": "object",
         "description": "Indexer retry and timeout settings",
         "properties": {
-          "connectTimeout":    { "type": "number", "description": "Per-server default connect timeout in ms (default: 60000)" },
-          "maxRetries":        { "type": "number", "minimum": 0, "description": "Retry attempts per server, 0 = no retry (default: 3)" },
-          "initialRetryDelay": { "type": "number", "description": "Initial backoff delay in ms (default: 2000)" },
-          "maxRetryDelay":     { "type": "number", "description": "Max backoff cap in ms (default: 30000)" },
-          "maxChunkChars":     { "type": "number", "minimum": 0, "description": "Max chars per chunk for long tool descriptions. 0 = disable chunking (default: 500)" },
-          "overlapChars":      { "type": "number", "minimum": 0, "description": "Overlap chars between adjacent chunks (default: 100)" }
+          "connectTimeout": {
+            "type": "number",
+            "description": "Per-server default connect timeout in ms (default: 60000)"
+          },
+          "maxRetries": {
+            "type": "number",
+            "minimum": 0,
+            "description": "Retry attempts per server, 0 = no retry (default: 3)"
+          },
+          "initialRetryDelay": {
+            "type": "number",
+            "description": "Initial backoff delay in ms (default: 2000)"
+          },
+          "maxRetryDelay": {
+            "type": "number",
+            "description": "Max backoff cap in ms (default: 30000)"
+          },
+          "maxChunkChars": {
+            "type": "number",
+            "minimum": 0,
+            "description": "Max chars per chunk for long tool descriptions. 0 = disable chunking (default: 500)"
+          },
+          "overlapChars": {
+            "type": "number",
+            "minimum": 0,
+            "description": "Overlap chars between adjacent chunks (default: 100)"
+          },
+          "generateCliArtifacts": {
+            "type": "boolean",
+            "description": "Generate mcporter CLI artifacts during reindex (default: false)"
+          }
         }
       },
       "embedding": {
         "type": "object",
         "properties": {
-          "provider": { "type": "string", "enum": ["openai", "gemini", "voyage", "mistral", "ollama"] },
-          "model":    { "type": "string" },
-          "baseUrl":  { "type": "string" },
-          "url":      { "type": "string", "description": "Deprecated: use baseUrl. Old Ollama URL (gets /v1 appended)." },
-          "apiKey":   { "type": "string" },
-          "headers":  { "type": "object", "additionalProperties": { "type": "string" } }
+          "provider": {
+            "type": "string",
+            "enum": [
+              "openai",
+              "gemini",
+              "voyage",
+              "mistral",
+              "ollama"
+            ]
+          },
+          "model": {
+            "type": "string"
+          },
+          "baseUrl": {
+            "type": "string"
+          },
+          "url": {
+            "type": "string",
+            "description": "Deprecated: use baseUrl. Old Ollama URL (gets /v1 appended)."
+          },
+          "apiKey": {
+            "type": "string"
+          },
+          "headers": {
+            "type": "object",
+            "additionalProperties": {
+              "type": "string"
+            }
+          }
         }
       },
       "vectorDb": {
         "type": "object",
-        "properties": { "path": { "type": "string" } }
+        "properties": {
+          "path": {
+            "type": "string"
+          }
+        }
       },
       "search": {
         "type": "object",
         "properties": {
-          "topK":     { "type": "number", "minimum": 1, "maximum": 20 },
-          "minScore": { "type": "number", "minimum": 0, "maximum": 1 }
+          "topK": {
+            "type": "number",
+            "minimum": 1,
+            "maximum": 20
+          },
+          "minScore": {
+            "type": "number",
+            "minimum": 0,
+            "maximum": 1
+          },
+          "includeParametersDefault": {
+            "type": "boolean",
+            "description": "Include full input schema by default in mcp_search results. If omitted, defaults automatically based on mcporter availability."
+          }
         }
       }
     }