toolfunnel 0.5.0 → 0.7.0

package/src/extend/add-mcp.md

@@ -1,121 +1,119 @@
-# How to register an upstream MCP (self-extension)
-
-This is the instruction served by `toolfunnel_howto({ topic: "add-mcp" })`. It explains how to connect an
-**upstream MCP server** (a browser-automation MCP, a chat bridge, …) to the aggregator and choose **which of its
-tools to expose downstream** to the CLI. All of this is configured in one persisted file:
-`mcp/expose.json`.
-
-The architecture (see the architecture overview, §1): many upstream MCPs each expose many tools; the
-aggregator connects to them and the **MCP Manager picks which tools to surface** to the CLI as
-*curated-direct* tools — alongside the four lean meta-tools (`toolfunnel_list_tools`,
-`toolfunnel_tool_instructions`, `toolfunnel_run_tool`, `toolfunnel_howto`). The CLI sees a small surface; everything
-else stays reachable through the register.
-
-> Phase note: the live aggregator (`src/mcp/aggregator.js`) is Phase 2. This doc defines the
-> `expose.json` shape so the config is authorable now and the aggregator reads exactly these fields
-> when it lands.
-
----
-
-## 1. `expose.json` — the two blocks
-
-`expose.json` persists (a) the upstream MCP connections and (b) the curated-direct selection drawn
-from them:
-
-```json
-{
-  "version": 1,
-  "upstreams": [
-    {
-      "id": "chat",
-      "transport": "stdio",
-      "command": "node",
-      "args": ["vendor/chat-mcp/server.js"],
-      "env": { "CHAT_DB": "..." },
-      "enabled": true,
-      "description": "Chat bridge MCP — chats, messages, send."
-    },
-    {
-      "id": "browser",
-      "transport": "stdio",
-      "command": "npx",
-      "args": ["@example/browser-mcp"],
-      "enabled": false,
-      "description": "Browser automation MCP."
-    }
-  ],
-  "expose": [
-    {
-      "upstream": "chat",
-      "tool": "list_chats",
-      "as": "chat_list_chats",
-      "category": "chat",
-      "enabled": true
-    },
-    {
-      "upstream": "chat",
-      "tool": "send_message",
-      "as": "chat_send_message",
-      "category": "chat",
-      "enabled": true
-    }
-  ]
-}
-```
-
-### `upstreams[]` — one entry per upstream MCP
-
-| Field | Type | Meaning |
-|---|---|---|
-| `id` | string | Stable key for this upstream. Referenced by `expose[].upstream`. Unique. |
-| `transport` | string | How the aggregator connects. `"stdio"` (spawn `command`+`args`) is the baseline. |
-| `command` / `args` / `env` | string / string[] / object | The process the aggregator spawns for a `stdio` upstream. Paths must point **inside the project's vendor directory** (clone vendored servers in; never reference servers configured elsewhere on the host). |
-| `enabled` | boolean | Whether the aggregator connects to this upstream at startup. |
-| `description` | string | Human-facing note for the MCP Manager UI. |
-
-### `expose[]` — which upstream tools become curated-direct
-
-| Field | Type | Meaning |
-|---|---|---|
-| `upstream` | string | Must match an `upstreams[].id`. |
-| `tool` | string | The upstream tool's own name (as the upstream's `tools/list` reports it). |
-| `as` | string | The downstream name the CLI sees (namespaced so two upstreams can't collide, e.g. `chat_send_message`). Optional — defaults to `<upstream>_<tool>`. |
-| `category` | string | Grouping label, same convention as register entries. |
-| `enabled` | boolean | Whether this tool is actually exposed downstream right now. |
-
-Only the tools listed in `expose[]` with `enabled: true` are surfaced to the CLI as curated-direct
-tools. Everything else on the upstream stays connected but hidden — reachable, if needed, by adding
-an `expose` entry (no upstream reconnect required to *list*, though see §3 on the CLI surface).
-
----
-
-## 2. Steps to register an upstream and expose a tool
-
-1. Add an entry to `upstreams[]` with a unique `id`, the `stdio` `command`/`args`, and
-   `enabled: true`. Vendor the server **inside the project's vendor directory** if it isn't already.
-2. Let the aggregator connect and run the upstream's `tools/list` (the MCP Manager UI surfaces the
-   discovered tools).
-3. For each tool you want hot, add an `expose[]` entry: `{ upstream, tool, as, category, enabled:true }`.
-4. Leave the long tail unexposed — the model can still reach those via the register/meta-tools if you
-   later mirror them, but the *curated-direct* surface stays lean by design (the dial in §1 of the
-   architecture: directly-exposed = the HOT tools; the register meta-tools = the long tail).
-
----
-
-## 3. Curated-direct vs. the register (update cost)
-
-This matters because the two layers refresh differently (see the architecture overview, §1):
-
-- **Curated-direct tools** (what `expose[]` produces) are negotiated with the CLI **at connect time**.
-  Changing the exposed set therefore needs a **reconnect**, or a `notifications/tools/list_changed`
-  message *if the CLI honours it* (VERIFY per host; restart is the safe baseline).
-- **Register tools** (the `toolfunnel_list_tools`/`run` path) are **always live** — the model re-reads the
-  register on every `toolfunnel_list_tools` call, so no reconnect is needed.
-
-Rule of thumb: expose a handful of HOT, high-reliability tools curated-direct; everything else lives
-in the register and is reached through the meta-tools.
-
-> Safety: a curated-direct upstream tool is still ultimately invoked by the model. When you wire
-> upstream execution through the host, route it through the same PreToolUse gate the register's `run`
-> uses (see `toolfunnel_howto({ topic: "add-hook" })`) so the safety invariant holds uniformly. The gate
-> firing on every execution path is a tested invariant, not an assumption.
+# How to register an upstream MCP (self-extension)
+
+This is the instruction served by `toolfunnel_howto({ topic: "add-mcp" })`. It explains how to connect an
+**upstream MCP server** (a browser-automation MCP, a chat bridge, ...) to the aggregator and choose **which of its
+tools to expose downstream** to the CLI. All of this is configured in one persisted file:
+`mcp/expose.json`.
+
+The architecture: many upstream MCPs each expose many tools; the
+aggregator connects to them and the **MCP Manager picks which tools to surface** to the CLI as
+*curated-direct* tools - alongside the four lean meta-tools (`toolfunnel_list_tools`,
+`toolfunnel_tool_instructions`, `toolfunnel_run_tool`, `toolfunnel_howto`). The CLI sees a small surface; everything
+else stays reachable through the register.
+
+
+---
+
+## 1. `expose.json` - the two blocks
+
+`expose.json` persists (a) the upstream MCP connections and (b) the curated-direct selection drawn
+from them:
+
+```json
+{
+  "version": 1,
+  "upstreams": [
+    {
+      "id": "chat",
+      "transport": "stdio",
+      "command": "node",
+      "args": ["vendor/chat-mcp/server.js"],
+      "env": { "CHAT_DB": "..." },
+      "enabled": true,
+      "description": "Chat bridge MCP - chats, messages, send."
+    },
+    {
+      "id": "browser",
+      "transport": "stdio",
+      "command": "npx",
+      "args": ["@example/browser-mcp"],
+      "enabled": false,
+      "description": "Browser automation MCP."
+    }
+  ],
+  "expose": [
+    {
+      "upstream": "chat",
+      "tool": "list_chats",
+      "as": "chat_list_chats",
+      "category": "chat",
+      "enabled": true
+    },
+    {
+      "upstream": "chat",
+      "tool": "send_message",
+      "as": "chat_send_message",
+      "category": "chat",
+      "enabled": true
+    }
+  ]
+}
+```
+
+### `upstreams[]` - one entry per upstream MCP
+
+| Field | Type | Meaning |
+|---|---|---|
+| `id` | string | Stable key for this upstream. Referenced by `expose[].upstream`. Unique. |
+| `transport` | string | How the aggregator connects. `"stdio"` (spawn `command`+`args`) is the baseline. |
+| `command` / `args` / `env` | string / string[] / object | The process the aggregator spawns for a `stdio` upstream. The isolation guard refuses to connect an upstream whose path-shaped **args** (including `--flag=value` values), **cwd**, or code-loading **env** vars (`NODE_OPTIONS`, `NODE_PATH`, and their equivalents) reach outside the config home. (The `command` itself, an executable, is not guarded. Opt-outs, each warned loudly by name: the currently-WRAPPED upstream is exempt; `allowOutsidePaths: true` on the entry exempts its paths; `allowCodeLoadingEnv: true` exempts the env class.) |
+| `enabled` | boolean | Whether the aggregator connects to this upstream at startup. |
+| `description` | string | Human-facing note for the MCP Manager UI. |
+
+### `expose[]` - which upstream tools become curated-direct
+
+| Field | Type | Meaning |
+|---|---|---|
+| `upstream` | string | Must match an `upstreams[].id`. |
+| `tool` | string | The upstream tool's own name (as the upstream's `tools/list` reports it). |
+| `as` | string | The downstream name the CLI sees (namespaced so two upstreams can't collide, e.g. `chat_send_message`). Optional - defaults to `<upstream>_<tool>`. |
+| `category` | string | Grouping label, same convention as register entries. |
+| `enabled` | boolean | Whether this tool is actually exposed downstream right now. |
+
+Only the tools listed in `expose[]` with `enabled: true` are surfaced to the CLI as curated-direct
+tools. Everything else on the upstream stays connected but hidden - reachable, if needed, by adding
+an `expose` entry (no upstream reconnect required to *list*, though see §3 on the CLI surface).
+
+---
+
+## 2. Steps to register an upstream and expose a tool
+
+1. Add an entry to `upstreams[]` with a unique `id`, the `stdio` `command`/`args`, and
+   `enabled: true`. Keep the server's path-shaped args **inside the config home** (vendor the
+   server in if it isn't already) - the isolation guard refuses args that escape it.
+2. Let the aggregator connect and run the upstream's `tools/list` (the MCP Manager UI surfaces the
+   discovered tools).
+3. For each tool you want hot, add an `expose[]` entry: `{ upstream, tool, as, category, enabled:true }`.
+4. Leave the long tail unexposed - the model can still reach those via the register/meta-tools if you
+   later mirror them, but the *curated-direct* surface stays lean by design (the dial in §1 of the
+   architecture: directly-exposed = the HOT tools; the register meta-tools = the long tail).
+
+---
+
+## 3. Curated-direct vs. the register (update cost)
+
+This matters because the two layers refresh differently:
+
+- **Curated-direct tools** (what `expose[]` produces) are negotiated with the CLI **at connect time**.
+  Changing the exposed set therefore needs a **reconnect**, or a `notifications/tools/list_changed`
+  message *if the CLI honours it* (VERIFY per host; restart is the safe baseline).
+- **Register tools** (the `toolfunnel_list_tools`/`run` path) are **always live** - the model re-reads the
+  register on every `toolfunnel_list_tools` call, so no reconnect is needed.
+
+Rule of thumb: expose a handful of HOT, high-reliability tools curated-direct; everything else lives
+in the register and is reached through the meta-tools.
+
+> Safety: a curated-direct upstream tool is still ultimately invoked by the model. When you wire
+> upstream execution through the host, route it through the same PreToolUse gate the register's `run`
+> uses (see `toolfunnel_howto({ topic: "add-hook" })`) so the safety invariant holds uniformly. The gate
+> firing on every execution path is a tested invariant, not an assumption.

package/src/extend/configure.md

@@ -0,0 +1,88 @@
+# How to configure ToolFunnel - config files only, no code
+
+This is the instruction served by `toolfunnel_howto({ topic: "configure" })`. It is the complete
+no-code setup map: every behaviour of the gateway is driven by a small set of JSON files at the
+**config home**. Editing these files (or driving the UI / the `tf_*` management tools, which write
+the same files) is the entire configuration story - there is nothing to program.
+
+## 1. The config home
+
+All files live under one directory, resolved in this order:
+
+1. `--config-dir <dir>` on the command line
+2. the `TOOLFUNNEL_HOME` environment variable
+3. the package/repo root (the default for a git clone)
+
+On first start with a fresh `--config-dir`, the home is seeded with the standard layout. Every
+file below is read FRESH per request - edits are live without a restart, except `toolfunnel.json`
+(read at boot; restart to apply).
+
+## 2. The files
+
+| File | What it controls | Live? |
+|---|---|---|
+| `toolfunnel.json` | Identity + default ports (see §3) | restart |
+| `mcp/expose.json` | Attached upstream MCPs + which of their tools are curated-direct | live |
+| `tools/tools.register.json` | The local tool register (the gateway's own tools) | live |
+| `tools/tools.state.json` | Per-tool overlays: enabled / hot / hidden, and the active wrap (`"passthrough"`). Wrapped example: `{ "passthrough": "acme-notes" }`; cleared: `{}` (the file-edit equivalent of `wrap --off`) | live |
+| `hooks/hooks.manifest.json` | PreToolUse / PostToolUse policy hooks (the gate) | live |
+| `hooks/hooks.state.json` | Per-hook enabled overlay | live |
+| `logs/log.config.json` | Activity log on/off + path | live |
+| `auth/auth.config.json` | OAuth 2.1 resource-server validation (opt-in; needs `toolfunnel install-oauth` once) | live |
+
+Shapes: `expose.json` → `toolfunnel_howto({ topic: "add-mcp" })`; register entries →
+`toolfunnel_howto({ topic: "create-tool" })`; hook entries → `toolfunnel_howto({ topic: "add-hook" })`;
+the wrap field → `toolfunnel_howto({ topic: "wrap" })`.
+
+## 3. `toolfunnel.json` - identity & ports
+
+Every field optional; an absent or empty file IS the default identity. Blank/omitted fields fall
+back per-field (a broken file never stops the gateway starting).
+
+```json
+{
+  "serverName":    "my-mcp",     // what CLIENTS see in the initialize handshake   (default "toolfunnel")
+  "serverVersion": "1.0.0",      // ...its version                                   (default: package version)
+  "clientName":    "my-client",  // what UPSTREAM servers see from the gateway     (default "toolfunnel")
+  "clientVersion": "1.0.0",      // ...its version
+  "httpPort":      9998,         // --http default port                            (default 9998)
+  "uiPort":        9777          // --ui   default port                            (default 9777)
+}
+```
+
+Precedence: CLI flag > this file > built-in default. Under a WRAP on stdio, the wrapped upstream
+is shown the real downstream client's identity automatically - `clientName`/`clientVersion` cover
+funnel mode, HTTP, and the boot connection. The UI's **Settings** tab edits this file.
+
+## 4. Worked example - a complete gateway from five files
+
+A gateway that attaches one upstream, exposes two of its tools, gates everything, logs activity,
+and introduces itself as `acme-tools`:
+
+1. `toolfunnel.json` → `{ "serverName": "acme-tools", "clientName": "acme-tools" }`
+2. `mcp/expose.json` → one `upstreams[]` entry (id, `"transport": "stdio"`, command, args,
+   `"enabled": true`) + two `expose[]` entries (`{ "upstream", "tool", "enabled": true }`)
+3. `hooks/hooks.manifest.json` → one PreToolUse entry matching the tools you want policy on
+4. `logs/log.config.json` → `{ "enabled": true }`
+5. Start it: `node bin/toolfunnel.js` (stdio) or `--http` / `--ui` as needed.
+
+Optional per-upstream flags in `expose.json`: `"legacyPin": true` pins that upstream to the
+legacy MCP protocol permanently (opt-in, warns loudly); `"env"` passes environment variables to
+the spawned server; `"cwd"` sets its working directory; `"timeoutMs"` raises the payload
+timeout for that upstream (default 120000 ms - applies to `tools/call`, `prompts/get` and
+`resources/read`; under a wrap, a tool that reports progress keeps its call alive regardless.
+the separate 10 s handshake/list window detects dead servers, and `"requestTimeoutMs"` raises
+it for a server that needs longer to boot); `"modernOnly": true`
+requires that upstream to speak the modern (2026-07-28) protocol - the connect fails with a
+clear error instead of falling back to legacy (the mirror of `legacyPin`; setting both on one
+upstream is refused). Server side, `"serveLegacy": false` in `toolfunnel.json` makes the
+gateway itself modern-only: legacy clients are refused with a clear error (default: serve
+both eras - only an explicit `false` flips it).
+
+## 5. Where the AI-driven and human paths meet
+
+The UI (all five tabs + Settings), the `tf_*` management tools, and hand-editing these files are
+three views of the SAME state - every write goes through the same validated, atomic store code.
+Pick whichever fits: config files for repeatable/scripted setups, the UI for humans, the
+management tools for in-band agent-driven changes (each documented via
+`toolfunnel_tool_instructions`).