@newsails/veil-cli 1.0.1

package/docs/guide/06-tools.md

@@ -0,0 +1,643 @@
+# Built-in Tools
+
+VeilCLI ships 25 built-in tools. Agents access tools through the permission system — see [Permissions](07-permissions.md) for how to allow/deny them.
+
+---
+
+## Tool Discovery — Four-Tier Loading
+
+VeilCLI uses a four-tier model to keep context lean while giving agents access to unlimited custom tools:
+
+| Tier | What | How the LLM sees it | Purpose |
+|------|------|---------------------|---------|
+| 1 | **Built-in tools** | Full schema in `tools[]` on every call | Always callable |
+| 2 | **Custom tools (all)** | Names + one-line descriptions in system prompt | Awareness only |
+| 3 | **`tool_search`** | Full JSON schemas returned as text | Inspect and decide |
+| 4 | **`tool_activate`** | Tool schema added to `tools[]` for the next LLM call | Explicit opt-in to call |
+
+**Workflow for using a custom tool:**
+
+```
+tool_search("query")         → read full schemas, inspect parameters
+tool_activate("chosen_tool") → tool becomes callable in the next iteration
+chosen_tool(args)            → executes normally
+```
+
+`tool_search` is **read-only** — searching does not make any tool callable. The LLM must explicitly call `tool_activate` once it has decided to use a specific custom tool. This prevents speculative searches from polluting the active tool list.
+
+`tool_activate` is pre-loaded as a built-in (Tier 1), so it is always available without any prior activation step.
+
+---
+
+## Tool Summary
+
+| Tool | Category | Description |
+|------|----------|-------------|
+| [`bash`](#bash) | Shell | Execute shell commands |
+| [`read_file`](#read_file) | File I/O | Read file contents |
+| [`write_file`](#write_file) | File I/O | Write/create files |
+| [`edit_file`](#edit_file) | File I/O | Patch files with find-and-replace |
+| [`list_dir`](#list_dir) | File I/O | List directory contents |
+| [`glob`](#glob) | File I/O | Find files by pattern |
+| [`grep`](#grep) | File I/O | Search file contents by regex |
+| [`web_search`](#web_search) | Web | DuckDuckGo search |
+| [`web_fetch`](#web_fetch) | Web | Fetch a URL |
+| [`memory_read`](#memory_read) | Memory | Read agent or project memory |
+| [`memory_write`](#memory_write) | Memory | Append to agent or project memory |
+| [`memory_search`](#memory_search) | Memory | Search memory for relevant content |
+| [`todo_write`](#todo_write) | Productivity | Write a todo list |
+| [`todo_read`](#todo_read) | Productivity | Read the current todo list |
+| [`task_create`](#task_create) | Multi-agent | Create an async task for any agent |
+| [`task_status`](#task_status) | Multi-agent | Check task status |
+| [`task_respond`](#task_respond) | Multi-agent | Unblock a waiting task |
+| [`task_subscribe`](#task_subscribe) | Multi-agent | Subscribe to task completion |
+| [`agent_spawn`](#agent_spawn) | Multi-agent | Start a new chat session with an agent, get sessionId back |
+| [`task_spawn`](#task_spawn) | Multi-agent | Spawn a sub-agent task (sync or async) |
+| [`agent_message`](#agent_message) | Multi-agent | Send a synchronous message to a specific agent session |
+| [`agent_send`](#agent_send) | Multi-agent | Fire-and-forget message to a specific agent session |
+| [`log_write`](#log_write) | Utility | Write a log entry to the task log |
+| [`sleep`](#sleep) | Utility | Wait N seconds |
+| [`tool_search`](#tool_search) | Utility | Search available tools by name or description |
+| [`tool_activate`](#tool_activate) | Utility | Make a custom tool callable (load into active tool list) |
+
+---
+
+## File I/O Tools
+
+### `bash`
+
+Execute a shell command and return its output. The command runs in the workspace directory by default.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `command` | string | ✓ | Shell command to execute |
+| `timeout` | integer | | Timeout in seconds (default: 30) |
+| `workingDir` | string | | Override working directory |
+
+Returns stdout on success, or `Exit code: N\nstderr:\n...` on failure.
+
+```
+"Run: ls -la /tmp"
+→ "total 1024\ndrwxrwxrwt 18 root root..."
+```
+
+---
+
+### `read_file`
+
+Read the contents of a file, with optional line range.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `path` | string | ✓ | File path (absolute or relative to workspace) |
+| `offset` | integer | | 1-indexed line number to start from |
+| `limit` | integer | | Number of lines to read |
+
+Returns file contents with line numbers. If the file is binary, returns a description.
+
+```
+"Read: package.json"
+→ "     1 {\n     2   \"name\": \"veilcli\"..."
+```
+
+---
+
+### `write_file`
+
+Create or overwrite a file.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `path` | string | ✓ | File path to write |
+| `content` | string | ✓ | Content to write |
+
+Creates parent directories if they don't exist.
+
+---
+
+### `edit_file`
+
+Patch a file by replacing an exact string with a new string. Safer than `write_file` for targeted edits.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `path` | string | ✓ | File path |
+| `old_string` | string | ✓ | Exact text to replace (must be unique in the file) |
+| `new_string` | string | ✓ | Replacement text |
+
+Returns an error if `old_string` is not found or appears more than once.
+
+---
+
+### `list_dir`
+
+List files and directories in a path.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `dir` | string | ✓ | Directory path |
+| `recursive` | boolean | | Include subdirectories (default: false) |
+
+Returns a formatted list with file sizes and types.
+
+---
+
+### `glob`
+
+Find files matching a glob pattern.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `pattern` | string | ✓ | Glob pattern (e.g. `**/*.js`, `src/*.ts`) |
+| `cwd` | string | | Base directory (default: workspace root) |
+| `limit` | integer | | Max results (default: 50) |
+
+---
+
+### `grep`
+
+Search file contents with a regex or literal string.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `pattern` | string | ✓ | Search pattern (regex or literal) |
+| `path` | string | ✓ | File or directory to search |
+| `recursive` | boolean | | Search subdirectories (default: false) |
+| `case_sensitive` | boolean | | Case-sensitive match (default: false) |
+| `fixed_strings` | boolean | | Treat pattern as literal string (default: false) |
+| `include` | string | | Glob to filter files (e.g. `*.js`) |
+| `context_lines` | integer | | Lines of context around each match |
+
+Returns matching lines with file path and line number.
+
+---
+
+## Web Tools
+
+### `web_search`
+
+Search the web using DuckDuckGo. No API key required.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `query` | string | ✓ | Search query |
+| `numResults` | integer | | Number of results (default: 5, max: 10) |
+
+Returns a list of results with titles, URLs, and snippets.
+
+---
+
+### `web_fetch`
+
+Fetch the content of a URL.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `url` | string | ✓ | URL to fetch |
+| `maxLength` | integer | | Max characters to return (default: 20000) |
+
+Returns the response text (HTML stripped to readable text for HTML pages).
+
+---
+
+## Memory Tools
+
+### `memory_read`
+
+Read the agent's persistent memory (or project-level memory).
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `scope` | string | | `"agent"` (default) or `"global"` |
+
+Returns the contents of the memory file, or an empty string if no memory exists yet.
+
+---
+
+### `memory_write`
+
+Append content to persistent memory. Entries are timestamped automatically.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `content` | string | ✓ | Content to append |
+| `scope` | string | | `"agent"` (default) or `"global"` |
+
+Memory is stored in `.veil/memory/agents/<name>/MEMORY.md` (agent) or `.veil/memory/MEMORY.md` (global).
+
+---
+
+### `memory_search`
+
+Search memory for content relevant to a query.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `query` | string | ✓ | Search query |
+| `scope` | string | | `"agent"` (default) or `"global"` |
+| `maxResults` | integer | | Max matching entries (default: 5) |
+
+Returns matching memory entries scored by relevance.
+
+---
+
+## Productivity Tools
+
+### `todo_write`
+
+Write a structured todo list for the current task.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `todos` | array | ✓ | Array of todo items |
+
+Each todo item:
+```json
+{
+  "id": "1",
+  "content": "Read the config file",
+  "status": "pending",
+  "priority": "high"
+}
+```
+
+`status`: `"pending"`, `"in_progress"`, `"completed"`  
+`priority`: `"high"`, `"medium"`, `"low"`
+
+Todos are stored in the database and can be read with `todo_read`.
+
+---
+
+### `todo_read`
+
+Read the current todo list for the active task or session.
+
+No parameters.
+
+Returns a formatted list of todos with statuses and priorities.
+
+---
+
+## Multi-Agent Tools
+
+### `task_create`
+
+Create a new task for any agent. Returns immediately with a `taskId` — the task runs asynchronously.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `agent` | string | ✓ | Agent name |
+| `input` | string | ✓ | Task instructions |
+| `priority` | string | | `"high"`, `"normal"`, `"low"` (default: `"normal"`) |
+| `tags` | string[] | | Optional labels |
+| `maxIterations` | integer | | Override max iterations |
+| `maxDurationSeconds` | integer | | Override max duration |
+
+Returns: `{ taskId, status: "pending", agent }`
+
+Use `task_status` to poll for completion.
+
+---
+
+### `task_status`
+
+Check the status and output of a task.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `taskId` | string | ✓ | Task ID from `task_create` or `agent_spawn` |
+
+Returns a JSON object with the full task record (status, output, error, iterations, tokens).
+
+---
+
+### `task_respond`
+
+Send a response to a task that is in `waiting` status (i.e. an agent called `task_respond` internally and is waiting for a reply).
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `taskId` | string | ✓ | Task ID |
+| `message` | string | ✓ | Message to inject into the agent's context |
+
+---
+
+### `task_subscribe`
+
+Create a durable subscription to a task's completion. When the task finishes, the subscriber session is notified. Subscriptions survive server restarts.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `taskId` | string | ✓ | Task ID to subscribe to |
+
+Used by orchestrators to reliably track delegated work.
+
+---
+
+### `agent_spawn`
+
+Start a new chat session with an agent and send the first message. Returns `{ sessionId, response }`. Use the `sessionId` with `agent_message` or `agent_send` for follow-up exchanges in the same conversation.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|--------------|
+| `agent` | string | ✓ | Agent name |
+| `message` | string | ✓ | First message to send |
+| `returnMode` | string | | `"full"` (default) → `{ sessionId, response }`, `"responseOnly"` → response text only |
+
+Returns an error immediately if the agent does not exist.
+
+---
+
+### `task_spawn`
+
+Spawn a sub-agent task with an isolated context. The most powerful multi-agent tool for delegating work.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|--------------|
+| `agent` | string | ✓ | Agent name |
+| `instruction` | string | ✓ | Task instruction. Use `"/skill-name args"` to trigger a skill. |
+| `returnMode` | string | | `"summary"` (default), `"raw"`, or `"lastMessage"` |
+| `wait` | boolean | | `true` = sync (block until done), `false` = async (return taskId immediately). Default: `true` |
+
+**Sync (default)**: blocks until the sub-agent completes, returns its output directly.
+
+**Async** (`wait: false`): returns immediately with `{ taskId }`. Use `task_status` or `task_subscribe` to track completion. Enables parallel fan-out.
+
+Return modes:
+- `"summary"` — returns the agent's final text output
+- `"raw"` — returns `{ taskId, status, output }` as JSON
+- `"lastMessage"` — returns the last message content
+
+---
+
+### `agent_message`
+
+Send a message to a specific agent session and wait for a reply (synchronous).
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|--------------|
+| `agent` | string | ✓ | Agent name |
+| `sessionId` | string | ✓ | Session ID obtained from `agent_spawn` |
+| `message` | string | ✓ | Message content |
+| `timeout` | integer | | Seconds to wait for reply (default: 60, no cap) |
+
+**Routing logic:**
+- If the session's loop is **idle** and mode is `chat` → triggers a new chat turn directly (same as a user message via API)
+- If the session's loop is **active** (mid-processing) → injects the message at the next `drainNonFollowup` checkpoint between tool calls
+- If the loop ends before picking up the message → automatically falls back to `runChat` directly
+
+Returns an error immediately if the agent or session does not exist, or if `sessionId` belongs to a different agent.
+
+---
+
+### `agent_send`
+
+Fire-and-forget message to a specific agent session. Does not wait for a reply.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|--------------|
+| `agent` | string | ✓ | Agent name |
+| `sessionId` | string | ✓ | Session ID obtained from `agent_spawn` |
+| `message` | string | ✓ | Message content |
+| `followup` | boolean | | If `true`, delivered only after the agent finishes its current task (default: `false`) |
+
+**Routing logic:**
+- **Idle chat session** → fires a new chat turn immediately (fire-and-forget)
+- **Active session** → injects into queue; a background watcher ensures the message is never lost — if the loop ends before draining it, `runChat` is triggered automatically
+
+Returns an error immediately if the agent or session does not exist.
+
+---
+
+## Utility Tools
+
+### `log_write`
+
+Write a log entry to the current task's log. Useful for agents to document their progress.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `message` | string | ✓ | Log message |
+| `level` | string | | `"info"` (default), `"warn"`, `"error"` |
+| `metadata` | object | | Optional structured data |
+
+Logs are stored as task events and visible in `GET /tasks/:id/events`.
+
+---
+
+### `sleep`
+
+Wait for a specified number of seconds. Time counts against `maxDurationSeconds` but not against `maxIterations`.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `seconds` | number | ✓ | Seconds to wait (max: 300) |
+
+Useful for polling loops or rate-limiting.
+
+---
+
+### `tool_search`
+
+Search the list of available tools (built-in + any custom tools loaded for the agent). Returns full schemas so the LLM can inspect parameter requirements before deciding which tool to activate. **This is read-only — calling `tool_search` does not make any tool callable.**
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `query` | string | ✓ | Search term (matches name and description) |
+
+Returns an array of matching tool schemas (name, description, input_schema).
+
+---
+
+### `tool_activate`
+
+Make a specific custom tool callable by loading its full schema into the active tool list. Call this after `tool_search` once you have decided to use a tool.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `name` | string | ✓ | Exact tool name to activate (as returned by `tool_search`) |
+
+Returns `{ "activated": "<name>" }` on success, or a not-found message if the tool doesn't exist. After activation the tool is callable in the same turn.
+
+The same whitelist/blocklist rules that apply to built-in tools are enforced at activation time — a blocked tool cannot be activated.
+
+---
+
+## Custom Tools
+
+You can add custom tools to a specific agent by placing `.js` files in `.veil/agents/<name>/tools/`. Each file must export:
+
+```js
+module.exports = {
+  schema: {
+    name: 'my_tool',
+    description: 'Does something useful',
+    input_schema: {
+      type: 'object',
+      properties: {
+        input: { type: 'string', description: 'Input value' }
+      },
+      required: ['input']
+    },
+    timeout: 30,  // seconds
+  },
+  async execute({ input, _cwd, _agent, _sessionId, _taskId, _settings, _remoteMethodExecution }) {
+    // _cwd, _agent, _sessionId, _taskId, _settings, _remoteMethodExecution are injected automatically
+    return `Processed: ${input}`;
+  }
+};
+```
+
+Custom tools are only available to the agent that owns them (not globally).
+
+---
+
+## Remote Method Execution
+
+`_remoteMethodExecution` lets a custom tool **pause and wait for a response from a connected UI**. This is the mechanism for building interactive UI-side tools — for example, a tool that opens a dialog in the dashboard and resumes once the user answers.
+
+### How it works
+
+```
+Tool calls _remoteMethodExecution()
+        │
+        ▼
+Server emits  method.pending  over SSE
+        │
+        ▼
+UI receives event, shows dialog / performs action
+        │
+        ▼
+UI posts result to  POST /remote-methods/:id/result
+        │
+        ▼
+Server broadcasts  method.done  to all SSE clients
+        │
+        ▼
+_remoteMethodExecution() resolves with the result
+        │
+        ▼
+Tool continues execution
+```
+
+Methods live **in memory only** — they do not survive a server restart.
+
+### Tool-side usage
+
+```js
+async execute({ question, _remoteMethodExecution }) {
+  const response = await _remoteMethodExecution({
+    method: 'user-dialog',          // arbitrary string — UI uses this to decide how to render
+    data: { question },             // any JSON-serialisable value the UI needs
+    timeoutMs: 120_000,             // optional, default 120 s; throws on timeout
+  });
+  // response is whatever the UI sent as body.result (any JSON value)
+  return `User answered: ${JSON.stringify(response)}`;
+}
+```
+
+**If no UI is listening** the call will hang until `timeoutMs` elapses, then the tool receives a rejection error. Design your tool to handle this gracefully:
+
+```js
+try {
+  const answer = await _remoteMethodExecution({ method: 'ask_user', data: { prompt } });
+  return `User said: ${answer}`;
+} catch (err) {
+  return `No UI responded in time: ${err.message}`;
+}
+```
+
+### API endpoints
+
+#### `GET /remote-methods`  *(SSE)*
+
+Connect to receive a stream of method events. On connect, all currently-pending methods are replayed so a freshly-connected UI is immediately in sync.
+
+**Event types**
+
+| type | fields | meaning |
+|------|--------|---------|
+| `method.pending` | `id`, `method`, `data`, `createdAt` | A tool is waiting for a result |
+| `method.done` | `id`, `timedOut` | Method was resolved (or timed out) — discard it |
+
+```
+: connected
+
+data: {"type":"method.pending","id":"550e8400-e29b-41d4-a716-446655440000","method":"user-dialog","data":{"question":"Continue?"},"createdAt":"2025-01-01T00:00:00.000Z"}
+
+data: {"type":"method.done","id":"550e8400-e29b-41d4-a716-446655440000","timedOut":false}
+```
+
+Headers required: `Authorization: Bearer <secret>` (same as all other endpoints).
+
+#### `POST /remote-methods/:id/result`
+
+Deliver the UI's response for a specific pending call.
+
+**Body**
+```json
+{ "result": <any JSON value> }
+```
+
+**Responses**
+
+| Status | Meaning |
+|--------|---------|
+| `200 { ok: true, id }` | Result accepted, waiting tool unblocked |
+| `409 { error: "not_found" }` | ID unknown — already resolved, timed out, or invalid |
+
+#### `GET /remote-methods/pending` *(diagnostic)*
+
+Returns `{ pending: ["id1", "id2", ...] }` — the IDs of all currently-waiting calls.
+
+### UI implementation example
+
+```js
+// Connect to the SSE stream
+const es = new EventSource('/remote-methods', {
+  headers: { Authorization: `Bearer ${secret}` }
+});
+
+es.onmessage = async (e) => {
+  const event = JSON.parse(e.data);
+
+  if (event.type === 'method.pending') {
+    if (event.method === 'user-dialog') {
+      // Show your dialog
+      const answer = await showDialog(event.data);
+
+      // Post the result back
+      await fetch(`/remote-methods/${event.id}/result`, {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${secret}` },
+        body: JSON.stringify({ result: answer }),
+      });
+    }
+  }
+
+  if (event.type === 'method.done') {
+    // Optionally close/dismiss any pending dialog for event.id
+    dismissDialog(event.id);
+  }
+};
+```
+
+### Multi-client behaviour
+
+If multiple UIs are connected simultaneously:
+
+- **All** receive `method.pending` — the **first** to POST a result wins.
+- The server immediately broadcasts `method.done` to all remaining clients so they can discard the prompt.
+- A late POST returns `409 not_found` — handle this gracefully on the UI side.
+
+### Timeouts
+
+The default timeout is **120 seconds**. Override per-call:
+
+```js
+await _remoteMethodExecution({ method: 'confirm', data: {}, timeoutMs: 30_000 });
+```
+
+When a call times out, `method.done` with `timedOut: true` is broadcast to all SSE clients.
+
+---