@newsails/veil-cli 1.0.1

package/docs/guide/07-permissions.md

@@ -0,0 +1,236 @@
+# Permissions
+
+VeilCLI uses a layered allow/deny permission system to control which tools an agent can use in each mode.
+
+---
+
+## Overview
+
+When an agent tries to call a tool, the runtime checks a permission chain from most-specific to least-specific:
+
+```
+agent.modes.<mode>.permissions  (most specific — wins)
+    ↓
+agent.modes.<mode>.tools / disallowedTools
+    ↓
+settings.json permissions        (project-wide default)
+    ↓
+built-in defaults                (deny everything not listed)
+```
+
+The **first matching rule wins**.
+
+---
+
+## Permission Fields
+
+### `allow`
+
+An array of tool names that are explicitly permitted. Use `["*"]` as a wildcard to allow all tools.
+
+```json
+"permissions": {
+  "allow": ["read_file", "list_dir", "bash"]
+}
+```
+
+### `deny`
+
+An array of tool names that are explicitly blocked. **`deny` always wins over `allow`** — if a tool appears in both, it is denied.
+
+```json
+"permissions": {
+  "allow": ["*"],
+  "deny": ["bash", "write_file"]
+}
+```
+
+### `ask`
+
+*(Settings-level only — not available in agent.json)*
+
+Tools that require interactive human approval before executing. When a tool in the `ask` list is called, the task pauses (`status: "waiting"`) and the caller must use `POST /tasks/:id/respond` to approve and continue.
+
+```json
+"permissions": {
+  "allow": ["*"],
+  "ask": ["write_file", "edit_file", "bash"]
+}
+```
+
+---
+
+## Where to Set Permissions
+
+### 1. Global default — `settings.json`
+
+Applies to all agents and all modes unless overridden:
+
+```json
+{
+  "permissions": {
+    "allow": ["read_file", "list_dir", "grep", "glob", "web_search"],
+    "deny": [],
+    "ask": ["bash", "write_file"]
+  }
+}
+```
+
+### 2. Per-mode in `agent.json`
+
+Overrides the global default for a specific mode of a specific agent:
+
+```json
+{
+  "modes": {
+    "task": {
+      "enabled": true,
+      "permissions": {
+        "allow": ["read_file", "list_dir", "bash", "write_file"],
+        "deny": []
+      }
+    }
+  }
+}
+```
+
+### 3. Tool whitelists and blacklists in `agent.json`
+
+`tools` and `disallowedTools` provide an alternative syntax:
+
+```json
+{
+  "modes": {
+    "task": {
+      "enabled": true,
+      "tools": ["read_file", "list_dir"],
+      "disallowedTools": ["bash"]
+    }
+  }
+}
+```
+
+`tools` = controls which tools are shown to the LLM (tool list in the API call)  
+`disallowedTools` = hides tools from the LLM  
+`permissions.allow`/`deny` = controls which tools are allowed to *execute* at runtime (checked at call time, supports glob patterns like `"bash(rm *)"`)
+
+You can use `tools`/`disallowedTools`, `permissions`, or both together.
+
+---
+
+## Common Patterns
+
+### Allow everything (development / trusted environment)
+
+```json
+{
+  "permissions": {
+    "allow": ["*"],
+    "deny": [],
+    "ask": []
+  }
+}
+```
+
+### Read-only agent (safe for untrusted tasks)
+
+```json
+{
+  "modes": {
+    "task": {
+      "enabled": true,
+      "permissions": {
+        "allow": ["read_file", "list_dir", "grep", "glob", "web_search", "web_fetch"],
+        "deny": []
+      }
+    }
+  }
+}
+```
+
+### Full access except destructive tools
+
+```json
+{
+  "permissions": {
+    "allow": ["*"],
+    "deny": ["bash", "write_file", "edit_file"]
+  }
+}
+```
+
+### Require approval for write operations
+
+```json
+{
+  "permissions": {
+    "allow": ["*"],
+    "deny": [],
+    "ask": ["write_file", "edit_file", "bash"]
+  }
+}
+```
+
+### Orchestrator agent (multi-agent tools only)
+
+```json
+{
+  "modes": {
+    "task": {
+      "enabled": true,
+      "permissions": {
+        "allow": ["agent_spawn", "task_create", "task_status", "task_subscribe", "log_write"],
+        "deny": []
+      }
+    }
+  }
+}
+```
+
+---
+
+## Permission Denied Behaviour
+
+When a tool is denied, the agent receives a tool result of:
+
+```
+Permission denied for tool "bash"
+```
+
+The agent loop continues — the LLM can choose to try another approach.
+
+---
+
+## Agent Restrictions
+
+In addition to tool permissions, you can restrict which sub-agents an agent is allowed to spawn:
+
+```json
+{
+  "modes": {
+    "task": {
+      "allowedAgents": ["coder", "writer"],
+      "disallowedAgents": ["admin"]
+    }
+  }
+}
+```
+
+`allowedAgents` — whitelist: only these agents can be spawned  
+`disallowedAgents` — blacklist: these agents are blocked
+
+If both are empty, all agents are allowed (subject to the target agent's own `subagent` mode config).
+
+---
+
+## Quick Reference
+
+| Field | Location | Description |
+|-------|----------|-------------|
+| `permissions.allow` | settings.json or agent.json modes | Tools allowed to execute at runtime (or `["*"]`). Supports glob patterns. |
+| `permissions.deny` | settings.json or agent.json modes | Tools blocked from executing at runtime. Wins over allow. Supports glob patterns. |
+| `permissions.ask` | **settings.json only** | Tools requiring human approval before executing |
+| `tools` | agent.json modes | Tools shown to the LLM (whitelist — LLM cannot call tools not in this list) |
+| `disallowedTools` | agent.json modes | Tools hidden from the LLM (blacklist) |
+| `allowedAgents` | agent.json modes | Sub-agents this agent may spawn |
+| `disallowedAgents` | agent.json modes | Sub-agents blocked from spawning |

package/docs/guide/08-memory.md

@@ -0,0 +1,139 @@
+# Memory & Context Compaction
+
+VeilCLI provides two complementary systems for managing long-running agent knowledge: **persistent memory files** for cross-session recall, and **context compaction** for keeping LLM context windows healthy during long tasks.
+
+---
+
+## Persistent Memory
+
+Memory files are plain Markdown files that persist across sessions and tasks. They are injected into the agent's system prompt at the start of each conversation.
+
+### Memory scopes
+
+| Scope | File path | Who writes it | Who reads it |
+|-------|-----------|--------------|--------------|
+| `agent` | `.veil/memory/agents/<name>/MEMORY.md` | The agent itself | Only that agent |
+| `global` | `.veil/memory/MEMORY.md` | Any agent | All agents with memory enabled |
+
+### Writing memory
+
+Agents use the `memory_write` tool:
+
+```
+"Remember that the project uses port 5051"
+→ memory_write({ content: "Project uses port 5051.", scope: "agent" })
+```
+
+Each entry is appended with a date comment:
+```markdown
+<!-- 2025-03-02 -->
+Project uses port 5051.
+```
+
+### Reading memory
+
+- **Automatic injection**: If `memory.enabled` is `true` in the agent config (or settings), the memory file is read and included in the system prompt at session start.
+- **Manual read**: Use `memory_read` to explicitly read memory mid-task.
+- **Search**: Use `memory_search` to find relevant entries by keyword.
+
+### Configuring memory
+
+In `agent.json`:
+```json
+{
+  "memory": {
+    "enabled": true,
+    "maxLines": 300
+  }
+}
+```
+
+In `settings.json` (applies globally to all agents):
+```json
+{
+  "memory": {
+    "enabled": true,
+    "maxLines": 500
+  }
+}
+```
+
+`maxLines` limits how much of the memory file is injected. Oldest entries are truncated first. The full file is preserved on disk; only the injection is capped.
+
+### Memory best practices
+
+- Write focused, factual entries — avoid vague notes like "talked about stuff"
+- Use global memory for facts shared across agents (e.g. project conventions, API endpoints)
+- Use agent memory for agent-specific state (e.g. "already processed file X")
+- Periodically use `memory_search` before writing to avoid duplicates
+
+---
+
+## Context Compaction
+
+LLM context windows are finite. During long tasks with many tool calls, the conversation history can exceed the model's context limit. VeilCLI manages this automatically through **context compaction**.
+
+### How it works
+
+1. **Observation masking**: Tool result messages older than `observationMaskingTurns` turns are replaced with `[observation masked]`. The tool call itself remains, but the full output is hidden. This is the first and cheapest form of compression.
+
+2. **Full compaction**: If the estimated token count exceeds `threshold × model_context_limit`, the runtime:
+   - Sends the current conversation to an LLM (using the `compact` model role if configured, otherwise `main`) with a special prompt asking for a structured summary
+   - Replaces the full message history with the summary plus the most recent turns
+   - Resumes the task with the compressed context
+
+### Configuration
+
+In `settings.json`:
+```json
+{
+  "compaction": {
+    "threshold": 0.8,
+    "observationMaskingTurns": 10
+  }
+}
+```
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `threshold` | `0.8` | Fraction of context window that triggers full compaction (0.1–1.0) |
+| `observationMaskingTurns` | `10` | Keep tool results visible for N most recent turns; mask older ones |
+
+### Using a dedicated compact model
+
+For cost efficiency, use a cheaper/faster model for compaction:
+
+```json
+{
+  "models": {
+    "main": {
+      "base_url": "https://openrouter.ai/api/v1",
+      "api_key": "sk-or-v1-...",
+      "model": "anthropic/claude-3.5-sonnet"
+    },
+    "compact": {
+      "base_url": "https://openrouter.ai/api/v1",
+      "api_key": "sk-or-v1-...",
+      "model": "google/gemini-flash-1.5-8b"
+    }
+  }
+}
+```
+
+### Memory extraction during compaction
+
+Before compacting, VeilCLI extracts important facts from the conversation and writes them to the agent's memory file (if memory is enabled). This means the agent "remembers" key decisions even after the context is compressed.
+
+---
+
+## Memory vs Compaction
+
+| Feature | Memory | Compaction |
+|---------|--------|------------|
+| **Scope** | Cross-session | Within a single session/task |
+| **Trigger** | Agent explicitly calls `memory_write` | Automatic when context is full |
+| **Storage** | Markdown files on disk | Replaced messages in the DB |
+| **Purpose** | Long-term recall across runs | Keep long tasks running smoothly |
+| **Controllable by agent** | Yes | No (automatic) |
+
+Use **memory** to remember things between runs. Rely on **compaction** to handle long-running tasks gracefully without manual intervention.

package/docs/guide/09-multi-agent.md

@@ -0,0 +1,271 @@
+# Multi-Agent Orchestration
+
+VeilCLI supports multiple agents working together. An **orchestrator** agent delegates work to **worker** agents using spawn, messaging, and subscription tools.
+
+---
+
+## Core Tools
+
+| Tool | Mode | Description |
+|------|------|-------------|
+| `agent_spawn` | Sync | Start a **chat session** with an agent — returns `{ sessionId, response }` |
+| `task_spawn` | Sync or async | Spawn a **task** for an agent with an isolated context |
+| `task_create` | Always async | Create a task for any agent (uses task mode) |
+| `task_status` | — | Poll a task's current status and output |
+| `task_subscribe` | — | Durable subscription to task completion |
+| `agent_message` | Sync | Send a message to a **specific session**, wait for reply |
+| `agent_send` | Fire-and-forget | Send a message to a **specific session** without waiting |
+
+> **Key distinction:** `task_spawn` isolates the sub-agent in its own context (task mode). `agent_spawn` opens a conversational session you can continue with `agent_message`. Use `task_spawn` for heavy delegated work; use `agent_spawn` + `agent_message` for interactive back-and-forth.
+
+---
+
+## Pattern 1: Simple Delegation (Sync)
+
+The simplest pattern — orchestrator spawns a task worker and waits for the result:
+
+```
+Orchestrator
+  └► task_spawn(agent="coder", instruction="Analyse src/index.js", wait=true)
+        └► Coder runs in task mode, returns summary
+  ◄─── Result returned inline
+```
+
+**Agent config** (`orchestrator/agent.json`):
+```json
+{
+  "name": "orchestrator",
+  "modes": {
+    "task": {
+      "enabled": true,
+      "permissions": {
+        "allow": ["task_spawn", "task_status", "log_write"]
+      }
+    }
+  }
+}
+```
+
+**How it works**: `task_spawn` with `wait: true` (default) blocks until the sub-agent finishes. The sub-agent runs in `subagent` mode — isolated from the orchestrator's context.
+
+---
+
+## Pattern 2: Parallel Fan-Out (Async)
+
+Spawn multiple task workers simultaneously, then collect results:
+
+```
+Orchestrator
+  ├► task_spawn(agent="researcher", instruction="Find X", wait=false) → taskId_1
+  ├► task_spawn(agent="researcher", instruction="Find Y", wait=false) → taskId_2
+  └► task_spawn(agent="researcher", instruction="Find Z", wait=false) → taskId_3
+        │
+        ├─ poll task_status(taskId_1) until finished
+        ├─ poll task_status(taskId_2) until finished
+        └─ poll task_status(taskId_3) until finished
+              │
+              └► Synthesise all results
+```
+
+**Orchestrator AGENT.md instruction**:
+```markdown
+When given a research topic with multiple subtopics:
+1. Use task_spawn with wait=false to spawn one researcher per subtopic
+2. Save all taskIds
+3. Poll each taskId with task_status until all are finished
+4. Synthesise all outputs into a final report
+```
+
+---
+
+## Pattern 3: Durable Subscriptions
+
+For long-running async tasks, use `task_subscribe` to get notified on completion even if the orchestrator's own session is interrupted:
+
+```
+Orchestrator
+  └► task_create(agent="analyst", input="Long analysis job") → taskId
+  └► task_subscribe(taskId)   ← stored in DB, survives restart
+        │
+        [... time passes, orchestrator may pause ...]
+        │
+        ◄─── Notification injected when analyst finishes
+```
+
+`task_subscribe` writes to the `task_subscriptions` SQLite table. When the target task reaches a terminal state (`finished`/`failed`), the runtime injects a notification into the subscriber's message queue — even if the subscriber session had been paused.
+
+**When to use**: Prefer `task_subscribe` over polling loops for very long tasks or when you need to save iterations.
+
+---
+
+## Pattern 4: Deep Research Pipeline
+
+The full deep research pattern combining fan-out + subscriptions + synthesis:
+
+```
+Research Orchestrator
+  ├─ Decompose topic into N sub-questions
+  ├─ Spawn N researchers async (wait=false) via task_spawn
+  ├─ Subscribe to all N tasks
+  ├─ Sleep or continue other work
+  ├─ Collect results as notifications arrive
+  └─ Call synthesis agent with all results
+```
+
+**Orchestrator task flow** (pseudocode for AGENT.md):
+```markdown
+1. Decompose the research question into 3-5 subtopics
+2. For each subtopic:
+   a. Use task_spawn(wait=false) to spawn a "researcher" agent
+   b. Note the returned taskId
+   c. Use task_subscribe(taskId) for durable tracking
+3. Poll task_status for each taskId until all are finished or failed
+4. Collect all outputs
+5. Use task_spawn to call a "writer" agent with all collected research
+6. Return the final synthesised report
+```
+
+---
+
+## Pattern 5: Hierarchical Delegation
+
+Multi-level hierarchies — orchestrators can spawn sub-orchestrators:
+
+```
+Top-level Orchestrator
+  ├─► Project Manager (sub-orchestrator)
+  │     ├─► Coder agent
+  │     └─► Reviewer agent
+  └─► Quality Checker
+        └─► Tester agent
+```
+
+The `maxSubAgentDepth` setting (default: 5) limits how deep nesting can go, preventing infinite recursion.
+
+---
+
+## Pattern 6: Session-Based Messaging
+
+For back-and-forth exchanges with an agent in the same conversation:
+
+```
+Orchestrator
+  └► agent_spawn(agent="advisor", message="What should I prioritise?") → { sessionId, response }
+  └► agent_message(agent="advisor", sessionId=sessionId, message="What about task B?")
+             ◄─ "Task B should come second because..."
+  └► agent_send(agent="advisor", sessionId=sessionId, message="Thanks, moving on.")
+```
+
+**`agent_spawn`** opens the session and gets the first response. Always required to obtain a `sessionId`.
+
+**`agent_message`** is synchronous — it routes the message to the exact session and waits for a reply.
+- If the session is idle → triggers a new turn directly (same as a user message)
+- If the session is mid-processing → injects at the next tool-call checkpoint
+- Automatically falls back if the message is missed at the last iteration
+
+**`agent_send`** is fire-and-forget — useful for notifications or status updates without blocking the orchestrator.
+
+Both tools require a `sessionId` (from `agent_spawn`) and validate that the `agent` name matches the session. Unknown agents or mismatched sessions return an immediate error instead of silently timing out.
+
+---
+
+## Sub-agent Configuration
+
+Target agents need `subagent` mode enabled to be spawnable via `task_spawn`. If `subagent` mode is not configured, the runtime falls back to `task` mode.
+
+For `agent_spawn` (chat session), agents need `chat` mode enabled.
+
+```json
+{
+  "name": "researcher",
+  "modes": {
+    "chat": {
+      "enabled": true,
+      "maxIterations": 20
+    },
+    "subagent": {
+      "enabled": true,
+      "maxIterations": 15,
+      "maxDurationSeconds": 90,
+      "permissions": {
+        "allow": ["web_search", "web_fetch", "read_file", "memory_write"]
+      }
+    }
+  }
+}
+```
+
+---
+
+## Agent Restrictions
+
+Orchestrators can whitelist which sub-agents they're allowed to spawn:
+
+```json
+{
+  "modes": {
+    "task": {
+      "allowedAgents": ["researcher", "writer", "coder"],
+      "disallowedAgents": ["admin"]
+    }
+  }
+}
+```
+
+---
+
+## Depth Limit
+
+Configure the maximum nesting depth in `settings.json`:
+
+```json
+{
+  "maxSubAgentDepth": 3
+}
+```
+
+When an agent tries to spawn beyond this depth, the spawn is rejected with an error.
+
+---
+
+## Practical Example: Code Review Pipeline
+
+**Agents**: `coordinator`, `coder`, `reviewer`
+
+**Coordinator AGENT.md**:
+```markdown
+You are a code review coordinator.
+
+When given a directory to review:
+1. Use list_dir to find all .js files
+2. For each file, use task_spawn with wait=false to spawn a "coder" agent
+3. Subscribe to all tasks with task_subscribe
+4. Once all analyses are done, use task_spawn to call a "reviewer" agent with the combined analysis
+5. Return the final review report
+```
+
+**Coder `agent.json`**:
+```json
+{
+  "name": "coder",
+  "modes": {
+    "subagent": {
+      "enabled": true,
+      "permissions": { "allow": ["read_file", "grep", "glob"] }
+    }
+  }
+}
+```
+
+**Reviewer `agent.json`**:
+```json
+{
+  "name": "reviewer",
+  "modes": {
+    "subagent": {
+      "enabled": true,
+      "permissions": { "allow": ["read_file", "memory_write"] }
+    }
+  }
+}
+```