toolfunnel 0.3.0 → 0.5.0

package/docs/configuration.md

@@ -1,402 +0,0 @@
-# Configuration reference
-
-ToolFunnel is driven entirely by plain JSON files on disk — no database, no environment-variable
-configuration of the surface. Everything the gateway exposes, curates, and gates is read from a
-handful of config files. This document is the authoritative, field-by-field reference; it is
-derived from the loaders that actually read each file (`src/tools/registry.js`,
-`src/tools/tool-state.js`, `src/mcp/expose-store.js`, `src/core/hook-loader.js`), not from intent.
-
-## The clean split: `src/` is the engine, the top level is yours
-
-ToolFunnel keeps a deliberate separation between **the engine** and **your configuration**:
-
-- **`src/`** is ToolFunnel's own engine code. You never edit it. It holds the server, the protocol,
-  the transports, the aggregator, the hook engine, and the tool *system* (`src/tools/registry.js`,
-  `src/tools/tool-state.js`, `src/tools/drift.js`). These modules *read* your config — they are not
-  your config.
-- **`tools/`, `mcp/`, and `hooks/`** at the repo root are **your** stuff — plain JSON plus the
-  scripts they reference. This is the only place you author anything:
-
-| Directory | What you put there | Read by (engine) |
-|---|---|---|
-| `tools/` | `tools.register.json`, `tools.state.json`, `scripts/*.js` — your first-party tools | `src/tools/registry.js`, `src/tools/tool-state.js` |
-| `mcp/`   | `expose.json` — your upstream MCP servers + the curated tools drawn from them | `src/mcp/expose-store.js` |
-| `hooks/` | `hooks.manifest.json`, `hooks.state.json`, `scripts/*` — your policy gate | `src/core/hook-loader.js` |
-
-Rule of thumb: if you are editing something under `src/`, stop — that is the machinery. Your
-register, your upstreams, and your gate all live at the top level.
-
-## The files at a glance
-
-| File | Loader | Purpose |
-|---|---|---|
-| `tools/tools.register.json` | `src/tools/registry.js` | The tool register — every first-party tool's metadata + how to invoke it. |
-| `tools/tools.state.json` | `src/tools/tool-state.js` | Per-tool enable/disable + hidden overlay over the register. |
-| `mcp/expose.json` | `src/mcp/expose-store.js` | Upstream MCP connections + the curated-direct tools drawn from them. |
-| `hooks/hooks.manifest.json` | `src/core/hook-loader.js` | The hook manifest — the gate. Each entry is one lifecycle hook. |
-| `hooks/hooks.state.json` | `src/core/hook-loader.js` | Per-hook enabled overlay over the manifest (sibling of the manifest). |
-| `logs/log.config.json` | `src/core/logger.js` | Toggle + path for the optional JSONL audit log (tool runs + gate allow/deny decisions). **Default OFF**; the file is not created until logging is enabled (via the `tf_log` management function or the admin UI Logs tab). Full reference: [`logging.md`](logging.md). |
-| `auth/auth.config.json` | `src/auth/config.js` | Toggle + settings for the optional **OAuth 2.1 resource-server** validation. **Default OFF** (loopback-only, unauthenticated). When enabled, the HTTP transport validates a bearer token on every request. Full reference: [`oauth.md`](oauth.md). |
-
-> **All paths resolve under the repo root.** The server (`src/mcp/server.js`) anchors
-> `ROOT = <repo>` from its own location (`path.resolve(__dirname, '..', '..')`) and joins every
-> config path off it: `tools/tools.register.json`, `tools/tools.state.json`,
-> `hooks/hooks.manifest.json`, `mcp/expose.json`, and the tool scripts root `tools/scripts`.
-> Nothing is read from outside the tree, and script/command targets are path-guarded to stay
-> inside it (the isolation rule).
-
-## Conventions shared by every file
-
-- **JSON only**, parsed once at load. A malformed register, expose store, or manifest throws a
-  clear error at load time — misconfiguration is caught at startup, not at model-call time. The
-  two overlay files (`tools.state.json`, `hooks.state.json`) are the exception: a
-  missing/unreadable/malformed overlay degrades to "no overrides" and never throws.
-- **Atomic writes.** Every write goes to a temp file in the same directory, is fsync'd, then
-  renamed over the target. A crash leaves either the old file or the new file, never a
-  half-written config. Prefer the loader's write methods over hand-editing a live file.
-- **No restart for register/state edits.** The server reads `tools.register.json` and
-  `tools.state.json` fresh on every `toolfunnel_list_tools` call, so a tool added or toggled is
-  visible with no reconnect. (Curated-direct tools from `expose.json` are the exception — see that
-  section.)
-
----
-
-## `tools/tools.register.json` — the tool register
-
-Top-level shape:
-
-```json
-{
-  "version": 1,
-  "description": "Tool register for the gateway — built-in demo scripts.",
-  "tools": [ /* entries */ ]
-}
-```
-
-- `version` — number (defaults to `1` if absent). Schema version of the register.
-- `description` — string (defaults to `""`). Free-text label, preserved across writes.
-- `tools` — array of entries (defaults to `[]`).
-
-### Entry shape
-
-```json
-{
-  "id": "echo",
-  "name": "Echo",
-  "summary": "Return the provided arguments back unchanged.",
-  "category": "demo",
-  "instructions": "Echoes its structured args back in the result. Pass any JSON value as args and the same value is returned.",
-  "invoke": { "type": "script", "path": "scripts/echo.js" }
-}
-```
-
-| Field | Type | Required | Meaning |
-|---|---|---|---|
-| `id` | string | **yes** | Stable unique key. `toolfunnel_run_tool({ name })` / `toolfunnel_tool_instructions({ name })` look up by this (id **or** display name). Duplicate ids are rejected **at load**. |
-| `name` | string | **yes** | Display name shown in `toolfunnel_list_tools` briefs and the manager UI. |
-| `summary` | string | no (→ `""`) | One line. This is *all* a brief carries for the tool — keep it tight so the model can pick from the brief alone. |
-| `category` | string | no (→ `""`) | Grouping label (`demo`, `crypto`, `data`, …). `toolfunnel_list_tools({ category })` filters on an exact match. |
-| `instructions` | string | no (→ `""`) | Full usage doc returned by `toolfunnel_tool_instructions`. The long-tail payload — be complete here, terse in `summary`. |
-| `mode` | string | no (→ inferred) | Execution mode — `gateway` or `reference`. See below. |
-| `invoke` | object | **yes** (unless `mode: "reference"`) | How the tool runs. Two forms below. |
-
-Validation (from `registry.js::validateEntry`): `id` and `name` must be non-empty strings, and
-`mode`, if present, must be `reference` or `gateway`. `invoke` must be present with `invoke.type`
-one of `script` | `shell` — **except** an explicit `mode: "reference"` may omit `invoke` entirely
-(a reference tool runs nothing server-side, so there is nothing to invoke). `summary`, `category`,
-and `instructions` are optional and default to empty strings.
-
-### `mode` — gateway vs reference execution
-
-The optional `mode` field decides **who runs the tool** (from `registry.js::resolveMode` /
-`VALID_MODES`):
-
-- **`gateway`** — ToolFunnel executes the `invoke` **server-side**, through the gated run path
-  (PreToolUse → execute → PostToolUse). This is the original behaviour of every `script`/`shell`
-  tool.
-- **`reference`** — ToolFunnel executes **nothing**. `toolfunnel_run_tool` returns the tool's
-  `instructions` so the **connected AI performs the action in its own environment**. A reference
-  tool needs no `invoke`.
-
-When `mode` is absent it is **inferred backward-compatibly**: a `script`/`shell` invoke ⇒
-`gateway` (so every existing register entry keeps running exactly as before), anything else ⇒
-`reference`. An explicit, valid `mode` always wins over the inference.
-
-### `invoke` — two forms
-
-**Script invoke (the implemented, tested path):**
-
-```json
-{ "type": "script", "path": "scripts/echo.js" }
-```
-
-- `path` points at a file under **`tools/scripts/`**. The runner takes `path.basename(path)` and
-  resolves it under the scripts root (`tools/scripts`), then refuses any result that escapes that
-  directory (defense-in-depth path guard). A `script` invoke needs a non-empty `path` or load
-  fails. In practice `path` is always written as `scripts/<id>.js`.
-- At run, the gateway spawns `node <resolved script>` with the structured args delivered in the
-  environment variable `TOOLFUNNEL_TOOL_ARGS` (JSON-encoded; `null` if no args). The script prints
-  exactly **one JSON line** to stdout and exits `0`. A non-zero exit is surfaced as the tool's
-  failure (the runner never rejects on exit code — only on a spawn failure).
-
-**Shell invoke:**
-
-```json
-{ "type": "shell", "command": "node tools/scripts/echo-upper.js" }
-```
-
-- A `shell` invoke needs a non-empty `command` or load fails.
-- `registry.js` deliberately does **not** spawn a shell. `resolveExecution` returns a deferred
-  descriptor (`{ deferred:true, type:"shell", command, args }`) and hands it to the gated runner,
-  which owns the actual shell spawn — so arbitrary shell execution stays behind the PreToolUse gate
-  and out of the un-gated register module.
-- When the gate allows it, the server runs the `command` **verbatim through the OS shell**
-  (`spawnSync(command, { shell:true })`) with **`cwd` set to the repo root** and
-  `TOOLFUNNEL_TOOL_ARGS` (the JSON args) injected into the environment. There is **no token
-  substitution** — the string is passed to the shell as written, so a relative path such as
-  `tools/scripts/echo-upper.js` resolves against the repo root. Prefer `script` for anything
-  reusable; reach for `shell` only for thin one-liners.
-
-> **Isolation rule.** An `invoke` must only reference scripts/commands that live **inside the
-> gateway's own tree** (`tools/scripts/`). Never point `invoke` at a tool living outside it — copy
-> the logic into `tools/scripts/` first, then point at the clone. The script path-guard enforces
-> this for `script` invokes; for `shell` you are responsible for keeping the command inside the
-> tree.
-
-### Live example (the shipped register)
-
-The default register ships **seven** demo `script` tools — `echo`, `base64`, `hash`, `uuid`,
-`json`, `text-stats`, `danger` — each
-`{ "type": "script", "path": "scripts/<id>.js" }`, with the matching file in `tools/scripts/`.
-
----
-
-## `tools/tools.state.json` — the enable/disable + hidden overlay
-
-A small overlay keyed by tool `id`. The register (`tools.register.json`) holds **all** tools; this
-file decides which are surfaced to clients and which are decluttered from the manager list. The
-shipped default is just `{}` (no overrides — everything on, nothing hidden).
-
-```json
-{
-  "danger":     { "enabled": false },
-  "text-stats": { "hidden": true }
-}
-```
-
-| Per-id field | Default | Axis | Effect |
-|---|---|---|---|
-| `enabled` | **ON** | MCP visibility | A tool is active unless the overlay explicitly says `enabled: false`. Disabled tools are filtered out of `toolfunnel_list_tools` so the client never sees them. |
-| `hidden` | not hidden | manager-list only | `hidden: true` declutters the tool from the **manager** listing only. It is **never** consulted by the MCP server and does not change the client's view. |
-
-Key behaviours (from `tool-state.js`):
-
-- **Default ON.** A tool absent from the overlay — or present without `enabled: false` — is active.
-  Newly-added register tools are live until you untick them, and the file only ever lists the tools
-  you have toggled.
-- **The two axes are independent.** A tool can be active-but-hidden or disabled-but-visible.
-  `enabled` controls what the client sees; `hidden` only declutters the manager view.
-- **Toggles merge.** `setToolEnabled` / `setToolHidden` merge into the existing entry, so flipping
-  one flag preserves the other, then persist atomically.
-- **Tolerant load.** A missing or malformed file loads as `{}` (no overrides) and never throws.
-- **Read fresh.** The server re-reads this overlay on every `toolfunnel_list_tools` call, so UI
-  toggles take effect with no restart.
-
----
-
-## `mcp/expose.json` — upstream MCPs + curated-direct selection
-
-Two blocks: the upstream MCP servers to connect to, and which of their tools to surface downstream
-as *curated-direct* tools. The shipped default is empty
-(`{ "version": 1, "upstreams": [], "expose": [] }`), so the gateway connects to nothing and the
-curated-direct surface is empty until you author entries. An annotated sample lives alongside it at
-`mcp/expose.example.json`.
-
-```json
-{
-  "version": 1,
-  "upstreams": [
-    {
-      "id": "mock",
-      "transport": "stdio",
-      "command": "node",
-      "args": ["mcp/servers/mock-upstream/server.js"],
-      "env": {},
-      "enabled": true,
-      "description": "Bundled demo upstream MCP (zero-dependency stdio): ping, add, echo."
-    }
-  ],
-  "expose": [
-    {
-      "upstream": "mock",
-      "tool": "ping",
-      "as": "mock_ping",
-      "category": "demo",
-      "enabled": true
-    }
-  ]
-}
-```
-
-> The demo upstream that ships with ToolFunnel lives at `mcp/servers/mock-upstream/server.js` and
-> offers `ping`, `add`, and `echo`. A ready-to-copy version of the sample above is at
-> `mcp/expose.example.json`.
-
-### `upstreams[]` — one entry per upstream MCP
-
-| Field | Type | Default | Meaning |
-|---|---|---|---|
-| `id` | string | — (required) | Stable unique key, referenced by `expose[].upstream`. Duplicate ids are rejected at load. |
-| `transport` | string | `"stdio"` | How the aggregator connects. `"stdio"` (spawn `command` + `args`) is the only valid transport today. |
-| `command` | string | `""` | The process to spawn for a `stdio` upstream. **Required and non-empty** for `stdio`. The interpreter slot (`node`, `python`, …) is not path-guarded; the `args` path is. |
-| `args` | string[] | `[]` | Command arguments. A script path here must resolve **inside the gateway root** (the aggregator's isolation guard rejects an arg path that escapes the tree — a relative path like `mcp/servers/mock-upstream/server.js` is resolved against the repo root and stays inside it). |
-| `env` | object | `{}` | Extra environment for the spawned upstream. |
-| `enabled` | boolean | `true` | Whether the aggregator connects to this upstream at startup. |
-| `description` | string | `""` | Human-facing note for the MCP Manager UI. |
-
-### `expose[]` — which upstream tools become curated-direct
-
-| Field | Type | Default | Meaning |
-|---|---|---|---|
-| `upstream` | string | — (required) | Must match an existing `upstreams[].id`. Referential integrity is enforced on write — you cannot expose a tool from an unknown upstream (`addExpose` throws). |
-| `tool` | string | — (required) | The upstream tool's own name, as its `tools/list` reports it. |
-| `as` | string | `<upstream>_<tool>` | The downstream name the client sees — the **rename**. Optional; defaults to the namespaced `<upstream>_<tool>` so two upstreams can't collide. On write the default is materialised into the stored entry, so the saved entry is self-describing. |
-| `category` | string | `""` | Grouping label, same convention as register entries. |
-| `enabled` | boolean | `true` | Whether the tool is actually exposed downstream right now. |
-
-Key behaviours (from `expose-store.js`):
-
-- **Natural key.** `(upstream, tool)` is the key; duplicate pairs are rejected at load.
-- **Cascade delete.** Removing an upstream also removes every `expose[]` entry that references it,
-  so no expose entry can dangle against a missing upstream.
-- **Lenient missing file.** A missing `expose.json` loads as an empty store bound to the path; the
-  file is **not created until the first write** (the server starts clean with no live upstreams). A
-  malformed file, a duplicate upstream id, or a duplicate expose key throws at load.
-- **Reconnect cost.** Unlike register tools, curated-direct tools are negotiated with the client at
-  connect time — changing the exposed set needs a reconnect (or a `tools/list_changed`
-  notification if the host honours it). Register tools reached via the meta-tools are always live.
-
----
-
-## `hooks/hooks.manifest.json` — the hook manifest (the gate)
-
-Every tool execution funnels through the hook engine: a `PreToolUse` hook can block a call before
-it runs (fails closed), and a `PostToolUse` hook observes the result (advisory — it cannot un-run
-the tool). The manifest is the list of those hooks. The shipped default is
-`{ "version": 1, "hooks": [] }` — **an empty `hooks` array is allow-all** (no gate fires).
-
-Top-level shape:
-
-```json
-{
-  "version": 1,
-  "hooksDir": "optional absolute override",
-  "hooks": [ /* entries */ ]
-}
-```
-
-- `version` — manifest schema version.
-- `hooksDir` — optional. The absolute hooks directory. If omitted (the normal case) it is resolved
-  from the manifest's own location — the manifest sits inside `hooks/`, so **that directory is the
-  source of truth** and `${HOOKS_DIR}` expands to the absolute path of `hooks/`.
-- `hooks` — array of hook entries.
-
-### Hook entry shape
-
-```json
-{
-  "id": "pre-tool-use/deny-dangerous",
-  "event": "PreToolUse",
-  "matcher": "Bash|toolfunnel_run_tool",
-  "type": "command",
-  "command": "bash \"${HOOKS_DIR}/scripts/deny-dangerous.sh\"",
-  "script": "scripts/deny-dangerous.sh",
-  "timeout": 5,
-  "enabled": true,
-  "description": "Denies any tool call whose args contain a destructive pattern."
-}
-```
-
-| Field | Type | Meaning |
-|---|---|---|
-| `id` | string | Unique key, convention `"<kebab-event>/<script-basename>"`. The loader and `hooks.state.json` key on this. |
-| `event` | string | One of the six lifecycle events (see below). |
-| `matcher` | string | Regex, anchored as a **full match** against `tool_name` (wrapped `^(?:…)$`). `""`, `undefined`, or `"*"` = always fire. Tool-less events ignore it. A malformed regex fails closed (never matches), never throws. Example: `"Bash\|Write\|Edit"`. |
-| `type` | string | `"command"` — the only type today. |
-| `command` | string | The command line the runner spawns (with `shell: true`). `${HOOKS_DIR}` expands to the absolute `hooks/` path **at load time**. The event payload is written to the command's **stdin** as JSON. |
-| `script` | string | The script path relative to `hooksDir` (e.g. `scripts/deny-dangerous.sh`). Used by the manager's open/edit; `writeScript` refuses any path that resolves outside `hooks/scripts/`. Inline-command hooks (no `script`) are allowed — there is just no file to edit. |
-| `timeout` | number | **Seconds** (the runner multiplies by 1000). Defaults to **60s** if absent or non-positive. The child is killed past this. |
-| `enabled` | boolean | The manifest's **seed** state. The live value is the overlay in `hooks.state.json` — see below. Normalised to a strict boolean at load (an omitted `enabled` is `false`). |
-| `description` | string | What the hook does, for the manager UI. |
-
-One entry **per hook command** — a single settings-style event with N commands becomes N entries.
-
-### The six lifecycle events
-
-`SessionStart`, `UserPromptSubmit`, `PreToolUse`, `PostToolUse`, `Stop`, `PreCompact` (from
-`src/core/events.js`). `PreToolUse` is the only event that can block a tool *before* it runs. The
-matcher applies to the tool-bearing events (`PreToolUse` / `PostToolUse`) and is ignored for the
-tool-less ones (`SessionStart`, `UserPromptSubmit`, `Stop`, `PreCompact`), which always fire.
-
-### How a hook blocks (the two protocols)
-
-The runner (`src/core/hook-runner.js`) spawns the command, pipes the event JSON to stdin, then
-reads the result:
-
-- **Exit-code protocol:** `exit 2` blocks; **stderr is the reason**. `exit 0` succeeds (for
-  `SessionStart` / `UserPromptSubmit`, stdout becomes injected context; otherwise advisory). Any
-  other non-zero code is a non-blocking error.
-- **JSON protocol** (honoured only on `exit 0`, when stdout parses as a JSON object): a `PreToolUse`
-  hook blocks via `hookSpecificOutput.permissionDecision: "deny"` (`"allow"` passes; `"ask"` is
-  non-blocking in the autonomous host). The other blockable events block via top-level
-  `decision: "block"`. `continue: false` stops the whole loop;
-  `hookSpecificOutput.additionalContext` is injected text.
-
-### Activation overlay — `hooks/hooks.state.json`
-
-What *exists* (the `hooks/scripts/` folder + manifest config) is split from what's *on*. The live
-enabled/disabled value lives in a sibling overlay keyed by hook `id`, and **the overlay wins over
-the manifest's seed `enabled`** at load.
-
-`setEnabled(id, bool)` writes the overlay atomically as a flat map:
-
-```json
-{
-  "pre-tool-use/deny-dangerous": true,
-  "user-prompt-submit/inject-context": false
-}
-```
-
-The loader also tolerates a wrapped form on read (forward-compatibility):
-
-```json
-{ "version": 1, "enabled": { "pre-tool-use/deny-dangerous": true } }
-```
-
-Both read the same way; the flat map is what `setEnabled` writes. The overlay is kept separate from
-the auto-detected inventory, so re-scanning `hooks/scripts/` never clobbers your toggles, and
-toggles survive restarts. (`setEnabled` also mirrors the flag back into the manifest — re-collapsing
-`${HOOKS_DIR}` so the on-disk manifest stays portable — but the overlay is the authoritative,
-precedence-winning store.)
-
----
-
-## Path resolution summary
-
-Every config and execution target resolves under the repo root. The server anchors the top-level
-config paths; the hook loader anchors the hooks paths to the manifest's own directory.
-
-| Anchor | Path |
-|---|---|
-| Tool register | `<repo>/tools/tools.register.json` |
-| Tool state overlay | `<repo>/tools/tools.state.json` |
-| Tool scripts root | `<repo>/tools/scripts/` |
-| Expose store | `<repo>/mcp/expose.json` |
-| Hook manifest | `<repo>/hooks/hooks.manifest.json` |
-| Hook state overlay | `<repo>/hooks/hooks.state.json` |
-| Hook scripts root | `<repo>/hooks/scripts/` |
-| Log config | `<repo>/logs/log.config.json` |
-
-Script and command targets are additionally path-guarded to stay inside their respective
-`scripts/` directories — a config entry can never become a path-escape out of the tree. Engine code
-under `src/` reads all of the above; you never edit `src/` to configure the gateway.