@newsails/veil-cli 1.0.1

package/PLAN/05-tools.md

@@ -0,0 +1,211 @@
+# 05 — Tools
+
+## Built-in Tools (24)
+
+| Tool | Category | Description |
+|------|----------|-------------|
+| `bash` | System | Execute shell commands (with permission checks) |
+| `read_file` | Files | Read file contents (with offset/limit for large files) |
+| `write_file` | Files | Write/overwrite entire file |
+| `edit_file` | Files | Search-and-replace patch (str_replace, like Claude Code) |
+| `list_dir` | Files | List directory contents with metadata |
+| `glob` | Files | Find files by glob pattern |
+| `grep` | Files | Search file contents (ripgrep-style) |
+| `web_fetch` | Network | Fetch URL content (HTML → text, JSON, images → base64) |
+| `web_search` | Network | Search the web, return results |
+| `memory_write` | Memory | Write to memory (agent-scoped by default; `scope: "global"` writes to `.veil/memory/MEMORY.md`) |
+| `memory_read` | Memory | Read agent's memory files |
+| `memory_search` | Memory | Search across project-level memory files (`.veil/memory/` + agent-specific). Does NOT search global `~/.veil/memory/`. |
+| `todo_write` | Memory | Create/update structured TODO list (survives compaction) |
+| `todo_read` | Memory | Read current TODO list |
+| `task_create` | Orchestration | Create a new task for any agent |
+| `task_status` | Orchestration | Check task status and output |
+| `task_respond` | Orchestration | Respond to a task waiting for input (task mode only) |
+| `agent_spawn` | Orchestration | Spawn a sub-agent with isolated context (sync or async via `wait` param) |
+| `agent_message` | Orchestration | Send message to another agent, wait for response (sync) |
+| `agent_send` | Orchestration | Send message to another agent, fire-and-forget (async) |
+| `task_subscribe` | Orchestration | Subscribe to a task's completion (for external observers — own spawns/tasks auto-subscribe) |
+| `tool_search` | Meta | Search available tools by name/description, load full schema |
+| `sleep` | Control | Wait for a configurable duration (max 300s). Counts against `maxDurationSeconds` but not `maxIterations`. |
+| `log_write` | Logging | Write entry to task log / push progress event |
+
+### Notes
+- **No `ask_user` tool** — chat mode agents respond naturally, task mode agents use `task_respond` to pause and wait.
+- **`task_respond` only available in task mode** — other modes don't need it.
+- **`task_subscribe`** registers durable interest in a task the agent didn't create. `agent_spawn(wait: false)` and `task_create` auto-subscribe the caller — no need for explicit `task_subscribe` on your own work. Use `task_subscribe` for monitoring agents, daemons, or external observers. Notifications survive session closure (runtime reopens session to deliver).
+- All other tools available in all modes (unless restricted by agent.json `tools`/`disallowedTools`).
+
+### Key Parameters (Orchestration Tools)
+
+Full JSON Schemas defined at implementation time. Key parameters for complex tools:
+
+| Tool | Key Parameters |
+|------|---------------|
+| `agent_spawn` | `agent` (string, required), `instruction` (string, required), `returnMode` ("summary" / "raw" / "lastMessage", default: "summary"), `wait` (boolean, default: true — false for async parallel fan-out) |
+
+**`returnMode` output formats:**
+- `"summary"` — LLM-generated summary of the sub-agent's work (~500 tokens max). Best for orchestrators that need gist only.
+- `"raw"` — Full conversation history (all messages). Can be large. Use for debugging or when caller needs full context.
+- `"lastMessage"` — Sub-agent's final assistant message only. Compact, good for direct answers.
+
+| `agent_message` | `agent` (string, required), `message` (string, required), `timeout` (number, default: 60s) |
+| `agent_send` | `agent` (string, required), `message` (string, required), `followup` (boolean, default: false) |
+| `task_create` | `agent` (string, required), `input` (string, required), `priority` ("high" / "normal" / "low"), `tags` (string[]), `maxIterations` (number), `maxDurationSeconds` (number) |
+| `task_subscribe` | `taskId` (string, required) |
+| `task_status` | `taskId` (string, required) |
+| `task_respond` | `taskId` (string, required), `message` (string, required) |
+| `memory_write` | `content` (string, required), `scope` ("agent" / "global", default: "agent") |
+| `memory_read` | `file` (string, optional — defaults to MEMORY.md) |
+| `memory_search` | `query` (string, required), `scope` ("project" / "agent", default: "project" — searches all project memory) |
+| `sleep` | `seconds` (number, required, max: 300) |
+| `edit_file` | `file` (string, required), `old_string` (string, required), `new_string` (string, required) |
+| `web_fetch` | `url` (string, required), `mode` ("text" / "json" / "base64", default: "text") |
+
+## Custom JS Tools
+
+User-defined tools at three levels:
+- **Agent-level:** `.veil/agents/<agentName>/tools/<toolName>/` — scoped to that agent only
+- **Project-level:** `.veil/tools/<toolName>/` — available to all agents in the project
+- **Global-level:** `~/.veil/tools/<toolName>/` — available to all agents everywhere
+
+```
+<toolName>/
+├── tool.json       ← Manifest (name, description, input_schema, timeout)
+└── index.js        ← Implementation
+```
+
+### tool.json
+```json
+{
+  "name": "send_slack_message",
+  "description": "Sends a message to a Slack channel",
+  "input_schema": {
+    "type": "object",
+    "properties": {
+      "channel": { "type": "string", "description": "Slack channel name" },
+      "message": { "type": "string", "description": "Message to send" }
+    },
+    "required": ["channel", "message"]
+  },
+  "timeout": 30
+}
+```
+
+### index.js
+```javascript
+/** @param {Object} input - Validated against input_schema */
+module.exports = async function(input) {
+  // Execute tool logic
+  return { success: true, messageId: "msg_123" };
+};
+```
+
+### Execution Model
+- **No sandboxing** — tools run in the runtime process via `require()`. User's responsibility.
+- **Input validation** — `ajv` validates args against `input_schema` before execution.
+- **Timeout** — enforced by runtime. Default 30s. Tool killed on timeout.
+- **Errors** — thrown errors caught by runtime, formatted as tool error message for LLM.
+- **Hot reload** — tool re-loaded from disk on each invocation (no caching).
+- **Dependencies** — custom tools can have their own `node_modules` (user runs `npm install` in tool folder).
+
+## MCP Tools
+
+MCP (Model Context Protocol) tools from external MCP servers.
+
+- **Client:** Existing MCP client library (not custom-built)
+- **Transports:** `stdio` and `Streamable HTTP`
+- **Configuration:** In `settings.json` under `mcpServers`
+- **Caching:** Tool descriptions cached, refreshed on server reconnect
+- **Permissions:** MCP tools subject to the same deny/ask/allow permission system. Use `mcp:<server>.<tool>(pattern)` format in permissions (see 09-permissions.md).
+- **Not the primary extension mechanism** — built-in + custom JS tools are core
+
+## Tool Resolution Order
+
+When the LLM requests a tool, resolved in order:
+```
+1. Built-in tools               ← always available
+2. Agent-level custom tools     ← ./.veil/agents/<name>/tools/
+3. Project-level custom tools   ← ./.veil/tools/
+4. Global custom tools          ← ~/.veil/tools/
+5. MCP tools                    ← from configured MCP servers
+6. NOT FOUND → error returned to LLM
+```
+
+Name conflicts: first match wins. Agent tools shadow project tools shadow global tools shadow MCP tools.
+
+## Tool Discovery (Hybrid Tiered Loading)
+
+Context window is finite. Not all tool descriptions should be loaded at once.
+
+- **Tier 1 (always in context):** Built-in tools — name + description + full schema (~24 tools)
+- **Tier 2 (listed, not loaded):** Custom (agent + project + global) + MCP tools — name + one-line description only
+- **Tier 3 (on-demand):** Full schema loaded via `tool_search` when agent needs it
+
+The `tool_search` tool lets agents discover and load tools they don't have full schemas for:
+```
+tool_search("slack") → returns full schema for send_slack_message
+```
+
+## Skills
+
+Skills are **prompt templates** that agents invoke. They are NOT tools — they inject instructions or spawn sub-agent contexts.
+
+Skill resolution order:
+```
+1. ./.veil/agents/<agentName>/skills/   ← agent-specific
+2. ./.veil/skills/                       ← project-level
+3. ~/.veil/skills/                       ← global
+4. NOT FOUND → error
+```
+
+### SKILL.md
+```markdown
+---
+name: review-code
+description: Review code for security and performance issues
+argument-hint: [file-path]
+allowed-tools: bash, read_file, edit_file
+context: fork
+agent: anthropic/claude-3-haiku
+disable-model-invocation: false
+---
+
+Review the following file: $1
+
+Focus on:
+1. Security vulnerabilities
+2. Performance bottlenecks
+3. Code style issues
+```
+
+### Execution Modes
+- **`context: main`** — skill content injected into current conversation
+- **`context: fork`** — skill runs in an isolated sub-agent context (via `agent_spawn`)
+
+### Loading
+- **Default:** Names + descriptions listed in system prompt. Full content loaded on-demand when invoked.
+- **Auto-loaded:** Skills listed in agent.json `autoLoadSkills` array have full content always in system prompt.
+- **Disabled:** When agent.json `skillDiscovery: false`, no skills listed in system prompt. Agent can only use skills explicitly in `autoLoadSkills`.
+- **`disable-model-invocation`:** When `true`, only user/API can trigger this skill — the agent cannot invoke it autonomously. Default: `false`. Use for dangerous skills (e.g., `/deploy-prod`) that require human initiation.
+
+### Skill Invocation via Agent Communication
+When using `agent_spawn`, `agent_message`, or `agent_send`, the instruction can start with `/skill-name` to trigger a specific skill on the target agent:
+```
+agent_spawn(agent: "researcher", instruction: "/deep-analysis What are the security risks?")
+```
+Runtime detects the `/` prefix, looks up the skill on the target agent, and executes it with the rest of the instruction as `$ARGUMENTS`.
+
+### Template Variables
+
+**`$AGENT_FOLDER`** — resolved in **AGENT.md, SKILL.md, and heartbeat files** at load time. Returns the absolute path to the agent's folder. Also available as the `VEIL_AGENT_FOLDER` environment variable in custom tool execution and hooks.
+
+**Skill-only variables** — resolved only when a skill is invoked. They do NOT apply to AGENT.md, agent.json, or heartbeat files.
+
+| Variable | Scope | Description |
+|----------|-------|-------------|
+| `$AGENT_FOLDER` | AGENT.md, SKILL.md, heartbeat | Absolute path to the agent's folder |
+| `$ARGUMENTS` | SKILL.md only | All arguments passed when skill is invoked |
+| `$1`, `$2`, ... | SKILL.md only | Arguments by position |
+| `${VEIL_SESSION_ID}` | SKILL.md only | Current session ID |
+| `${VEIL_TASK_ID}` | SKILL.md only | Current task ID |
+| `${VEIL_AGENT_NAME}` | SKILL.md only | Current agent name |

package/PLAN/06-communication.md

@@ -0,0 +1,243 @@
+# 06 — Communication & Orchestration
+
+## Agent-to-Agent Communication
+
+Five mechanisms, each for a different use case:
+
+### 1. `agent_message` (Sync Direct Message)
+Agent A sends a message to Agent B and **blocks** until B responds.
+
+```
+Agent A calls agent_message(agent: "researcher", message: "What's the status of X?")
+    ↓
+If Agent B is idle:
+    Runtime creates a temporary chat session for Agent B
+    Agent B processes the message (may use tools)
+    Agent B's response returned to Agent A as tool output
+If Agent B is busy:
+    Message queued. Agent A blocks. Message injected into B at next
+    safe injection point (between tool calls). B's response returned.
+```
+
+**Use when:** Quick question/answer. B's response is needed to continue.
+**Timeout:** Configurable. Default 60s.
+
+### 2. `agent_send` (Async Fire-and-Forget)
+Agent A sends a message to Agent B and **continues immediately**. Returns a `message_id`.
+
+```
+Agent A calls agent_send(
+  agent: "notifier",
+  message: "Deploy completed",
+  followup: false           // default: false
+)
+    ↓
+Runtime queues the message for Agent B
+    ↓
+Agent A gets back { messageId: "msg_xxx" } and continues
+    ↓
+If Agent B is idle:
+    Message delivered immediately as a new chat turn
+If Agent B is busy:
+    Message held in queue, injected at next safe point
+    (between tool calls — after tool result, before next LLM call)
+If followup: true:
+    Message only injected after Agent B fully finishes current work
+```
+
+**Use when:** Notifications, status updates, triggering side effects without waiting.
+
+**`followup` behavior per mode:**
+- **Chat mode:** `followup: true` waits until agent's current response is complete (no pending tool calls).
+- **Task mode:** `followup: true` waits until task finishes entirely (status = finished/failed).
+- **Daemon mode:** `followup: true` waits until current tick completes.
+- **Subagent mode:** `followup: true` waits until sub-agent finishes its work.
+
+### 3. `task_create` (Async Task Delegation)
+Agent A creates a **task** for Agent B. The task enters the queue and runs independently.
+
+```
+Agent A calls task_create(agent: "researcher", input: "Research topic X", priority: "high")
+    ↓
+Runtime creates task in SQLite: status = pending
+    ↓
+Agent A gets back { taskId: "task_xxx" } and can:
+  - Continue working (fire-and-forget)
+  - Poll with task_status(taskId)
+  - Set own status to "waiting" until task completes
+```
+
+**Use when:** Long-running work. Parallel execution. Work that should be tracked/queryable.
+
+## Sub-Agent Spawning
+
+### `agent_spawn` (Isolated Context — Sync or Async)
+
+```
+Agent A calls agent_spawn(
+  agent: "explore",
+  instruction: "Find all files related to authentication",
+  returnMode: "summary",   // "summary" | "raw" | "lastMessage"
+  wait: true                // true (default) = sync, false = async
+)
+    ↓
+Runtime creates a new session with clean context:
+  - Sub-agent's AGENT.md + SOUL.md loaded
+  - Instruction injected as user message
+  - Tools inherited from sub-agent's agent.json (`modes.subagent`)
+  - If instruction starts with `/skill-name`, runtime triggers that skill
+  - Runtime creates a task entry in SQLite (trackable via task_status)
+    ↓
+If wait: true (default — synchronous):
+  Sub-agent runs to completion (or timeout)
+  Result returned to Agent A based on returnMode
+If wait: false (asynchronous):
+  Agent A gets back { taskId: "task_xxx" } immediately
+  Runtime auto-subscribes Agent A to this task (durable notification)
+  Sub-agent runs in background using modes.subagent config
+  Agent A can:
+    - Spawn more sub-agents (parallel fan-out)
+    - Poll with task_status(taskId) to get result early
+  When sub-agent finishes:
+    - Result stored as task output (per returnMode)
+    - Notification auto-injected into Agent A's conversation
+```
+
+**Parallel fan-out pattern** (deep research, multi-source investigation):
+```
+// Orchestrator spawns 5 researchers in parallel:
+spawn1 = agent_spawn(agent: "researcher", instruction: "...", wait: false)
+spawn2 = agent_spawn(agent: "researcher", instruction: "...", wait: false)
+spawn3 = agent_spawn(agent: "researcher", instruction: "...", wait: false)
+// Auto-subscribed to all 3 — no explicit task_subscribe needed.
+// Orchestrator continues planning while subagents work...
+// Notifications arrive as each subagent completes.
+// Orchestrator collects results via task_status and synthesizes.
+```
+
+**Fallback:** If the target agent doesn't have `modes.subagent` defined or enabled, `agent_spawn` falls back to chat-like behavior using `modes.chat` config.
+
+**Key properties:**
+- Isolated context window (no parent conversation history)
+- Gets its own task + session in SQLite (queryable, auditable)
+- Token usage aggregated with parent task
+- Max recursion depth: configurable (default 3 — agents can spawn sub-agents that spawn sub-agents)
+- Timeout: inherits parent's remaining time or fixed limit (whichever is shorter)
+- `wait: false` spawns count against `maxConcurrentTasks` limit. When limit is reached, new spawns queue as `pending` until a slot opens (no error, no blocking).
+- **Daemon mode exemption:** Daemon ticks do NOT auto-subscribe. Daemon sessions are ephemeral (per-tick). Use system-wide `triggers` (task.completed, task.failed) instead.
+
+### 4. `task_subscribe` (Durable Callback for External Observers)
+Any agent can subscribe to another task's completion — even tasks it didn't create. When the watched task finishes/fails, the runtime injects a notification into the subscriber's conversation.
+
+**Note:** `agent_spawn(wait: false)` **auto-subscribes** the caller. You do NOT need `task_subscribe` for your own async spawns. `task_subscribe` is for **external observers**: monitoring agents, daemons, or agents that want to watch tasks they didn't create.
+
+```
+// Monitoring agent watches a task it didn't create:
+Agent M calls task_subscribe(taskId: "task_xyz")
+    → registered in SQLite. Agent M continues other work.
+
+... later, task_xyz finishes ...
+
+If Agent M's session is still active:
+    Runtime injects system message at next safe point:
+        "[System] Task task_xyz completed: { output: ... }"
+
+If Agent M's session has ended:
+    Runtime reopens the session and injects the notification.
+    Agent M wakes up, processes the result, and can continue or finish.
+```
+
+**Use when:** Monitoring agent, daemon, or script wants to observe tasks it didn't create.
+**Auto-subscribe:** `agent_spawn(wait: false)` and `task_create` callers are auto-subscribed — no need to call `task_subscribe` for your own work.
+**Durability:** Subscriptions are stored in SQLite and survive session completion. If the subscriber's session has ended, the runtime reopens it to deliver the notification.
+
+**Edge cases:**
+- **Delivery timeout:** If session cannot be reopened within 60s, subscription marked `failed` and logged.
+- **Multiple subscribers:** Each subscriber gets notified independently. No deduplication.
+- **Cleanup:** Delivered subscriptions (`status: delivered`) are purged after 7 days. Failed subscriptions retained for debugging.
+
+### When to Use What
+
+| Mechanism | Sync? | Context | Use Case |
+|-----------|-------|---------|----------|
+| `agent_message` | Yes | Temporary session | Quick Q&A between agents |
+| `agent_send` | No | Queued | Notifications, fire-and-forget |
+| `task_create` | No | New task session (modes.task) | Long work, parallel execution, tracked (auto-subscribes caller) |
+| `task_subscribe` | No | Durable callback | External observer watching tasks it didn't create |
+| `agent_spawn` | Configurable | Isolated sub-session (modes.subagent) | `wait:true` = focused sub-task now; `wait:false` = parallel fan-out (auto-subscribes) |
+
+**`agent_spawn` vs `task_create`:** Both can run async and both auto-subscribe the caller. The difference is the **mode used**: `agent_spawn` uses `modes.subagent` (restricted tools, isolated context, result condensed). `task_create` uses `modes.task` (full task capabilities, formal task lifecycle). Use `agent_spawn(wait: false)` when you want parallel sub-agents with subagent-mode restrictions. Use `task_create` when you want a formal, independently tracked task.
+
+## Task System
+
+### Task Lifecycle
+```
+pending → processing → [waiting] → finished
+                   ↘              ↗
+                    → failed
+                    → canceled
+
+**Transitions from `waiting`:**
+- `waiting` → `processing` — via `task_respond` (human provides input)
+- `waiting` → `failed` — via API cancel or wait timeout (`maxWaitSeconds`, default 3600)
+- `waiting` → `canceled` — via `DELETE /tasks/:id`
+
+**Note:** Tasks in `waiting` status do NOT count against `maxConcurrentTasks` (they're paused, not running).
+```
+
+### Task Fields
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | string | `task_<nanoid>` |
+| `agentName` | string | Which agent runs this task |
+| `status` | enum | pending/processing/waiting/finished/failed/canceled |
+| `priority` | enum | high/normal/low |
+| `parentTaskId` | string? | If created by another task |
+| `input` | object | Prompt + files + context |
+| `output` | object? | Final result (when finished) |
+| `error` | string? | Error message (when failed) |
+| `iterations` | number | Loop iterations executed |
+| `maxIterations` | number | Limit |
+| `maxDurationSeconds` | number | Limit |
+| `tokenUsage` | object | input/output/cache tokens + cost |
+| `tags` | string[] | User-defined tags for filtering |
+| `createdAt` | timestamp | |
+| `startedAt` | timestamp? | |
+| `finishedAt` | timestamp? | |
+
+### Task Dependencies
+Managed by the **orchestrating agent**, not by the runtime. The runtime provides `task_create`, `task_status`, and the agent decides ordering, parallelism, and failure handling.
+
+### Parallel Execution
+Multiple tasks can run concurrently. Configurable concurrency limit in settings.json (`maxConcurrentTasks`, default 5).
+
+## Message Queue
+
+When a message targets a busy agent (mid-loop), it enters the **message queue**.
+
+### Queue Behavior
+- Messages stored in SQLite `agent_messages` table
+- **Normal messages** (`followup: false`): injected at the next **safe injection point** — between tool calls (after tool result returned, before next LLM call)
+- **Followup messages** (`followup: true`): held until agent fully finishes current work (no more tool calls)
+- Injected as system messages: `[Message from <agentName>]: <content>`
+- Multiple queued messages injected in FIFO order
+
+### Safe Injection Points
+```
+LLM responds with tool_calls
+    ↓
+Execute tool call #1
+    ↓
+★ SAFE INJECTION POINT ★  ← drain normal message queue here
+    ↓
+Execute tool call #2
+    ↓
+Loop back to LLM
+    ↓
+LLM responds with no tool_calls (done)
+    ↓
+★ FOLLOWUP INJECTION POINT ★  ← drain followup queue here
+```
+
+### For `agent_message` (sync)
+If target agent is busy, the calling agent **blocks and waits**. The message is queued and injected at the next **safe injection point** (between tool calls). Once the target agent processes it and produces a response, that response is returned to the caller. This is consistent with `agent_message`'s blocking nature — the caller waits, but the target agent doesn't have to fully finish its current work first.

package/PLAN/07-storage.md

@@ -0,0 +1,218 @@
+# 07 — Storage
+
+## Architecture
+
+**SQLite for runtime data.** Single file: `~/.veil/data.db`.
+**Files for definitions.** agent.json, settings.json, AGENT.md — human-editable, versionable.
+**Files for mutable state.** Memory (`.veil/memory/`), heartbeats (`.veil/heartbeats/`) — human-editable, agent-writable.
+
+### Naming Convention
+- **JSON configs:** camelCase (`maxDurationSeconds`, `allowedAgents`)
+- **SQLite columns:** snake_case (`max_duration_seconds`, `allowed_agents`)
+
+The runtime maps between conventions automatically. This is a standard pattern — JSON for human-facing configs, snake_case for SQL.
+
+## SQLite Configuration
+
+```javascript
+db.pragma('journal_mode = WAL');     // Concurrent readers while writer is active
+db.pragma('synchronous = NORMAL');   // Optimal crash safety for WAL mode
+db.pragma('foreign_keys = ON');      // Enforce relational integrity
+```
+
+## Tables (Core)
+
+### sessions
+| Column | Type | Description |
+|--------|------|-------------|
+| id | TEXT PK | `sess_<nanoid>` |
+| agent_name | TEXT NOT NULL | Agent that owns this session |
+| mode | TEXT NOT NULL | `chat` / `task` / `daemon` / `subagent` |
+| instance_folder | TEXT | Project folder path (tracks which directory the session was started from, used to resolve project-level agents/settings). See **Project Isolation** below. |
+| status | TEXT | `active` / `closed` |
+| model | TEXT | Model used |
+| title | TEXT | Auto-generated title |
+| message_count | INTEGER | |
+| token_input | INTEGER | Cumulative input tokens |
+| token_output | INTEGER | Cumulative output tokens |
+| token_cache | INTEGER | Cumulative cache tokens |
+| estimated_cost | REAL | Estimated USD cost |
+| compaction_count | INTEGER | Times compacted |
+| created_at | TEXT | ISO 8601 |
+| updated_at | TEXT | ISO 8601 |
+
+### messages
+| Column | Type | Description |
+|--------|------|-------------|
+| id | INTEGER PK AUTOINCREMENT | |
+| session_id | TEXT FK → sessions | |
+| role | TEXT NOT NULL | `system` / `user` / `assistant` / `tool` |
+| content | TEXT | Text content (may be large) |
+| tool_calls | TEXT | JSON array of tool calls |
+| tool_call_id | TEXT | For tool response messages |
+| token_count | INTEGER | Tokens in this message |
+| created_at | TEXT | ISO 8601 |
+
+### tasks
+| Column | Type | Description |
+|--------|------|-------------|
+| id | TEXT PK | `task_<nanoid>` |
+| session_id | TEXT FK → sessions | |
+| agent_name | TEXT NOT NULL | |
+| status | TEXT NOT NULL | pending/processing/waiting/finished/failed/canceled |
+| priority | TEXT | high/normal/low |
+| parent_task_id | TEXT FK → tasks | |
+| input | TEXT | JSON |
+| output | TEXT | JSON |
+| error | TEXT | |
+| tags | TEXT | JSON array |
+| iterations | INTEGER | |
+| max_iterations | INTEGER | |
+| max_duration_seconds | INTEGER | |
+| token_input | INTEGER | |
+| token_output | INTEGER | |
+| token_cache | INTEGER | |
+| estimated_cost | REAL | |
+| created_at | TEXT | |
+| started_at | TEXT | |
+| finished_at | TEXT | |
+| updated_at | TEXT | |
+
+### task_events
+| Column | Type | Description |
+|--------|------|-------------|
+| id | INTEGER PK AUTOINCREMENT | |
+| task_id | TEXT FK → tasks | |
+| type | TEXT NOT NULL | iteration.start, tool.start, tool.end, status.change, custom |
+| data | TEXT | JSON payload (see schemas below) |
+| created_at | TEXT | ISO 8601 |
+
+**Event `data` schemas:**
+```json
+// type: "iteration.start"
+{ "iteration": 5, "tokensSoFar": 12500 }
+
+// type: "tool.start"
+{ "toolName": "bash", "toolInput": { "command": "npm test" } }
+
+// type: "tool.end"
+{ "toolName": "bash", "durationMs": 1234, "success": true, "outputPreview": "..." }
+
+// type: "status.change"
+{ "from": "processing", "to": "waiting", "reason": "task_respond called" }
+
+// type: "custom" (via log_write tool)
+{ "level": "info", "message": "Completed phase 1", "metadata": { ... } }
+```
+
+### todos
+| Column | Type | Description |
+|--------|------|-------------|
+| session_id | TEXT PK FK → sessions | |
+| items | TEXT NOT NULL | JSON array of {id, content, status, priority} |
+| updated_at | TEXT | |
+
+### agent_messages
+| Column | Type | Description |
+|--------|------|-------------|
+| id | INTEGER PK AUTOINCREMENT | |
+| target_agent | TEXT NOT NULL | Agent this message is queued for |
+| target_session_id | TEXT FK → sessions | Session to inject into (if active) |
+| from_agent | TEXT NOT NULL | Sending agent name |
+| content | TEXT NOT NULL | Message content |
+| followup | INTEGER NOT NULL DEFAULT 0 | 1 = only inject after agent finishes |
+| skill | TEXT | Skill to trigger (if `/skill-name` prefix) |
+| status | TEXT NOT NULL | pending / delivered / expired |
+| correlation_id | TEXT | For sync `agent_message` response matching |
+| response | TEXT | Response content (for sync `agent_message`) |
+| created_at | TEXT | ISO 8601 |
+| delivered_at | TEXT | ISO 8601 |
+
+### task_subscriptions
+| Column | Type | Description |
+|--------|------|-------------|
+| id | INTEGER PK AUTOINCREMENT | |
+| task_id | TEXT FK → tasks NOT NULL | Task being watched |
+| subscriber_session_id | TEXT FK → sessions NOT NULL | Session to notify (reopened if closed) |
+| subscriber_agent | TEXT NOT NULL | Agent that subscribed |
+| status | TEXT NOT NULL | pending / delivered |
+| created_at | TEXT | ISO 8601 |
+| delivered_at | TEXT | ISO 8601 |
+
+### schema_version
+| Column | Type | Description |
+|--------|------|-------------|
+| version | INTEGER PK | Current schema version |
+| applied_at | TEXT | ISO 8601 |
+
+## Indexes
+
+```sql
+CREATE INDEX idx_sessions_agent ON sessions(agent_name);
+CREATE INDEX idx_sessions_status ON sessions(status);
+CREATE INDEX idx_sessions_created ON sessions(created_at);
+CREATE INDEX idx_messages_session ON messages(session_id);
+CREATE INDEX idx_tasks_agent ON tasks(agent_name);
+CREATE INDEX idx_tasks_status ON tasks(status);
+CREATE INDEX idx_tasks_parent ON tasks(parent_task_id);
+CREATE INDEX idx_tasks_created ON tasks(created_at);
+CREATE INDEX idx_task_events_task ON task_events(task_id);
+CREATE INDEX idx_agent_messages_target ON agent_messages(target_agent, status);
+CREATE INDEX idx_task_subscriptions_task ON task_subscriptions(task_id, status);
+```
+
+## Migrations
+
+- Numbered SQL files: `migrations/001-initial.sql`, `002-add-events.sql`, etc.
+- `schema_version` table tracks current version and applied timestamp
+- Additive-only rule: never drop columns, never rename tables
+- Applied in transaction on startup
+
+**All migration SQL uses `IF NOT EXISTS` (idempotent).** Migrations are run on every startup, not only when the database is freshly created. This prevents timing bugs where the DB file exists (SQLite created it) but the schema hasn't been applied yet:
+```sql
+CREATE TABLE IF NOT EXISTS sessions (...);
+CREATE TABLE IF NOT EXISTS messages (...);
+CREATE INDEX IF NOT EXISTS idx_sessions_agent ON sessions(agent_name);
+```
+The `schema_version` table check still determines whether to run a given numbered migration (skip if already at that version), but the SQL itself is always safe to re-run. Never use `CREATE TABLE` without `IF NOT EXISTS`.
+
+## Project Isolation
+
+All sessions go to a single global DB (`~/.veil/data.db`). Project isolation is handled via `instance_folder`:
+
+- **Same agent name, different projects:** Each session records its `instance_folder`. When listing/querying, results are filtered by the current working directory.
+- **Agent resolution:** Runtime uses `instance_folder` to resolve project-level agents first (`./<instance_folder>/.veil/agents/`) before falling back to global (`~/.veil/agents/`).
+- **No cross-project leakage:** An agent in Project A cannot accidentally see sessions from Project B — queries always filter by `instance_folder`.
+
+## Lazy Loading
+
+- **Session metadata** always queryable (SELECT from sessions)
+- **Messages** loaded on-demand (only when session is active or API requests them)
+- **LRU cache** — max 50 active sessions in memory. Evict oldest on overflow.
+
+## Retention & Cleanup
+
+Configurable in settings.json:
+```json
+{
+  "storage": {
+    "retention": {
+      "sessions": { "maxAgeDays": 90, "maxCount": 10000 },
+      "tasks": { "maxAgeDays": 30, "maxCount": 5000 }
+    }
+  }
+}
+```
+
+**Cleanup process:**
+1. Runs on startup and every 24h
+2. Sessions/tasks older than `maxAgeDays` → archive to gzipped JSONL → delete from SQLite
+3. If count exceeds `maxCount` → archive oldest → delete
+4. Archive path: `~/.veil/archive/sessions-2026-03.jsonl.gz`
+
+## Backup
+
+- Hot backup via SQLite `VACUUM INTO` (non-blocking)
+- Triggered before cleanup runs
+- Stored in `~/.veil/backups/data-2026-03-01.db`
+- Keep last 3 backups