@newsails/veil-cli 1.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (199) hide show
  1. package/.veil/agents/analyst/AGENT.md +21 -0
  2. package/.veil/agents/analyst/agent.json +23 -0
  3. package/.veil/agents/assistant/AGENT.md +15 -0
  4. package/.veil/agents/assistant/agent.json +19 -0
  5. package/.veil/agents/coder/AGENT.md +18 -0
  6. package/.veil/agents/coder/agent.json +19 -0
  7. package/.veil/agents/hello/AGENT.md +5 -0
  8. package/.veil/agents/hello/agent.json +13 -0
  9. package/.veil/agents/writer/AGENT.md +12 -0
  10. package/.veil/agents/writer/agent.json +17 -0
  11. package/.veil/memory/MEMORY.md +343 -0
  12. package/.veil/memory/agents/analyst/MEMORY.md +55 -0
  13. package/.veil/memory/agents/hello/MEMORY.md +12 -0
  14. package/.veil/runtime.pid +1 -0
  15. package/.veil/settings.json +10 -0
  16. package/.veil-studio/studio.db +0 -0
  17. package/.veil-studio/studio.db-shm +0 -0
  18. package/.veil-studio/studio.db-wal +0 -0
  19. package/PLAN/01-vision.md +26 -0
  20. package/PLAN/02-tech-stack.md +94 -0
  21. package/PLAN/03-agents.md +232 -0
  22. package/PLAN/04-runtime.md +171 -0
  23. package/PLAN/05-tools.md +211 -0
  24. package/PLAN/06-communication.md +243 -0
  25. package/PLAN/07-storage.md +218 -0
  26. package/PLAN/08-api-cli.md +153 -0
  27. package/PLAN/09-permissions.md +108 -0
  28. package/PLAN/10-ably.md +105 -0
  29. package/PLAN/11-file-formats.md +442 -0
  30. package/PLAN/12-folder-structure.md +205 -0
  31. package/PLAN/13-operations.md +212 -0
  32. package/PLAN/README.md +23 -0
  33. package/README.md +128 -0
  34. package/REPORT.md +174 -0
  35. package/TODO.md +45 -0
  36. package/ai-tests/FRONTEND_PROMPT.md +220 -0
  37. package/ai-tests/Research & Planning.md +814 -0
  38. package/ai-tests/prompt-001-basic-api.md +230 -0
  39. package/ai-tests/prompt-002-basic-flows.md +230 -0
  40. package/ai-tests/prompt-003-agent-behaviors.md +220 -0
  41. package/api/middleware.js +60 -0
  42. package/api/routes/agents.js +193 -0
  43. package/api/routes/chat.js +93 -0
  44. package/api/routes/completions.js +122 -0
  45. package/api/routes/daemons.js +80 -0
  46. package/api/routes/memory.js +169 -0
  47. package/api/routes/models.js +40 -0
  48. package/api/routes/remote-methods.js +74 -0
  49. package/api/routes/sessions.js +208 -0
  50. package/api/routes/settings.js +108 -0
  51. package/api/routes/system.js +50 -0
  52. package/api/routes/tasks.js +270 -0
  53. package/api/server.js +120 -0
  54. package/cli/formatter.js +70 -0
  55. package/cli/index.js +443 -0
  56. package/cli/parser.js +113 -0
  57. package/config/config.json +10 -0
  58. package/config/models.json +6826 -0
  59. package/core/agent.js +329 -0
  60. package/core/cancel.js +38 -0
  61. package/core/compaction.js +176 -0
  62. package/core/events.js +13 -0
  63. package/core/loop.js +564 -0
  64. package/core/memory.js +51 -0
  65. package/core/prompt.js +185 -0
  66. package/core/queue.js +96 -0
  67. package/core/registry.js +291 -0
  68. package/core/remote-methods.js +124 -0
  69. package/core/router.js +386 -0
  70. package/core/running-sessions.js +18 -0
  71. package/docs/api/01-system.md +84 -0
  72. package/docs/api/02-agents.md +374 -0
  73. package/docs/api/03-chat.md +269 -0
  74. package/docs/api/04-tasks.md +470 -0
  75. package/docs/api/05-sessions.md +444 -0
  76. package/docs/api/06-daemons.md +142 -0
  77. package/docs/api/07-memory.md +186 -0
  78. package/docs/api/08-settings.md +133 -0
  79. package/docs/api/09-models.md +119 -0
  80. package/docs/api/09-websocket.md +350 -0
  81. package/docs/api/10-completions.md +134 -0
  82. package/docs/api/README.md +116 -0
  83. package/docs/guide/01-quickstart.md +220 -0
  84. package/docs/guide/02-folder-structure.md +185 -0
  85. package/docs/guide/03-configuration.md +252 -0
  86. package/docs/guide/04-agents.md +267 -0
  87. package/docs/guide/05-cli.md +290 -0
  88. package/docs/guide/06-tools.md +643 -0
  89. package/docs/guide/07-permissions.md +236 -0
  90. package/docs/guide/08-memory.md +139 -0
  91. package/docs/guide/09-multi-agent.md +271 -0
  92. package/docs/guide/10-daemons.md +226 -0
  93. package/docs/guide/README.md +53 -0
  94. package/docs/index.html +623 -0
  95. package/examples/README.md +151 -0
  96. package/examples/agents/assistant/AGENT.md +31 -0
  97. package/examples/agents/assistant/SOUL.md +9 -0
  98. package/examples/agents/assistant/agent.json +74 -0
  99. package/examples/agents/hello/AGENT.md +15 -0
  100. package/examples/agents/hello/agent.json +14 -0
  101. package/examples/agents/monitor/AGENT.md +51 -0
  102. package/examples/agents/monitor/agent.json +33 -0
  103. package/examples/agents/monitor/heartbeats/monitor.md +24 -0
  104. package/examples/agents/orchestrator/AGENT.md +70 -0
  105. package/examples/agents/orchestrator/agent.json +30 -0
  106. package/examples/agents/researcher/AGENT.md +52 -0
  107. package/examples/agents/researcher/agent.json +49 -0
  108. package/examples/agents/researcher/skills/web-research.md +28 -0
  109. package/examples/skills/code-review.md +72 -0
  110. package/examples/skills/summarise.md +59 -0
  111. package/examples/skills/web-research.md +42 -0
  112. package/examples/tools/word-count/index.js +27 -0
  113. package/examples/tools/word-count/tool.json +18 -0
  114. package/infrastructure/database.js +563 -0
  115. package/infrastructure/scheduler.js +122 -0
  116. package/llm/client.js +206 -0
  117. package/migrations/001-initial.sql +121 -0
  118. package/migrations/002-debuggability.sql +13 -0
  119. package/migrations/003-drop-orphaned-columns.sql +72 -0
  120. package/migrations/004-session-message-token-fields.sql +78 -0
  121. package/migrations/005-session-thinking.sql +5 -0
  122. package/package.json +30 -0
  123. package/schemas/agent.json +143 -0
  124. package/schemas/settings.json +111 -0
  125. package/scripts/fetch-models.js +93 -0
  126. package/session-debug-scenario.md +248 -0
  127. package/settings/fields.js +52 -0
  128. package/system-prompts/base-core.md +7 -0
  129. package/system-prompts/environment.md +13 -0
  130. package/system-prompts/reminders/anti-drift.md +6 -0
  131. package/system-prompts/reminders/stall-recovery.md +10 -0
  132. package/system-prompts/safety-rules.md +25 -0
  133. package/system-prompts/task-heuristics.md +27 -0
  134. package/test/client.js +71 -0
  135. package/test/integration/01-health.test.js +25 -0
  136. package/test/integration/02-agents.test.js +80 -0
  137. package/test/integration/03-chat-hello.test.js +48 -0
  138. package/test/integration/04-chat-multiturn.test.js +61 -0
  139. package/test/integration/05-chat-writer.test.js +48 -0
  140. package/test/integration/06-task-basic.test.js +68 -0
  141. package/test/integration/07-task-tools.test.js +74 -0
  142. package/test/integration/08-task-code-analysis.test.js +69 -0
  143. package/test/integration/09-memory-analyst.test.js +63 -0
  144. package/test/integration/10-task-advanced.test.js +85 -0
  145. package/test/integration/11-sessions-advanced.test.js +84 -0
  146. package/test/integration/12-assistant-chat-tools.test.js +75 -0
  147. package/test/integration/13-edge-cases.test.js +99 -0
  148. package/test/integration/14-cancel.test.js +62 -0
  149. package/test/integration/15-debug.test.js +106 -0
  150. package/test/integration/16-memory-api.test.js +83 -0
  151. package/test/integration/17-settings-api.test.js +41 -0
  152. package/test/integration/18-tool-search-activation.test.js +119 -0
  153. package/test/results/.gitkeep +0 -0
  154. package/test/runner.js +206 -0
  155. package/test/smoke.js +216 -0
  156. package/tools/agent_message.js +85 -0
  157. package/tools/agent_send.js +80 -0
  158. package/tools/agent_spawn.js +44 -0
  159. package/tools/bash.js +49 -0
  160. package/tools/edit_file.js +41 -0
  161. package/tools/glob.js +64 -0
  162. package/tools/grep.js +82 -0
  163. package/tools/list_dir.js +63 -0
  164. package/tools/log_write.js +31 -0
  165. package/tools/memory_read.js +38 -0
  166. package/tools/memory_search.js +65 -0
  167. package/tools/memory_write.js +42 -0
  168. package/tools/read_file.js +48 -0
  169. package/tools/sleep.js +22 -0
  170. package/tools/task_create.js +41 -0
  171. package/tools/task_respond.js +37 -0
  172. package/tools/task_spawn.js +64 -0
  173. package/tools/task_status.js +39 -0
  174. package/tools/task_subscribe.js +37 -0
  175. package/tools/todo_read.js +26 -0
  176. package/tools/todo_write.js +38 -0
  177. package/tools/tool_activate.js +24 -0
  178. package/tools/tool_search.js +24 -0
  179. package/tools/web_fetch.js +50 -0
  180. package/tools/web_search.js +52 -0
  181. package/tools/write_file.js +28 -0
  182. package/ui/api.js +190 -0
  183. package/ui/app.js +281 -0
  184. package/ui/index.html +382 -0
  185. package/ui/views/agents.js +377 -0
  186. package/ui/views/chat.js +610 -0
  187. package/ui/views/connection.js +96 -0
  188. package/ui/views/daemons.js +129 -0
  189. package/ui/views/feed.js +194 -0
  190. package/ui/views/memory.js +263 -0
  191. package/ui/views/models.js +146 -0
  192. package/ui/views/sessions.js +314 -0
  193. package/ui/views/settings.js +142 -0
  194. package/ui/views/tasks.js +415 -0
  195. package/utils/context.js +49 -0
  196. package/utils/id.js +16 -0
  197. package/utils/models.js +88 -0
  198. package/utils/paths.js +213 -0
  199. package/utils/settings.js +172 -0
@@ -0,0 +1,94 @@
1
+ # 02 — Tech Stack
2
+
3
+ ## Language
4
+
5
+ **Node.js + JSDoc** (no TypeScript). Plain JavaScript with JSDoc type annotations for IDE support and type checking.
6
+
7
+ > **v2 Note:** TypeScript is strongly recommended for a future version. Most of the function-signature mismatches, missing exports, and field-name inconsistencies discovered during testing would be caught at compile time.
8
+
9
+ ## Dependencies (Minimal Set)
10
+
11
+ | Concern | Package | Justification |
12
+ |---------|---------|---------------|
13
+ | **LLM Client** | native `fetch` (built-in) | Node 18+ built-in. No SDK version drift. Works with any OpenAI-compatible endpoint (OpenRouter, Anthropic, Ollama, LM Studio, etc.) without external dependencies. |
14
+ | **REST Server** | `express` | Simplest, most maintained, vast middleware ecosystem |
15
+ | **CORS** | `cors` | Required by express for cross-origin requests |
16
+ | **Validation** | `ajv` + `ajv-formats` | JSON Schema validator, compiles schemas to fast native functions. `ajv-formats` required for `date-time` and other format validators. |
17
+ | **Database** | `better-sqlite3` | Fastest SQLite for Node.js, synchronous (optimal for local) |
18
+ | **Cron** | `node-cron` | Lightweight, Unix-native cron syntax |
19
+ | **MCP Client** | TBD (existing lib) | Not custom-built, supports stdio + Streamable HTTP |
20
+ | **CLI Parsing** | `util.parseArgs` | Built-in Node.js (>=18.3), zero external deps |
21
+
22
+ **Rejected alternatives:**
23
+ - `openai` SDK — SDK versioning caused `OpenAI is not a constructor` failures (v3 vs v4 API mismatch). Native `fetch` is simpler, portable, and requires no versioning decisions.
24
+ - `nanoid` — was missing from package.json, caused runtime crash. Replaced by `crypto.randomBytes()` from Node's built-in `crypto` module (see `src/utils/id.js`).
25
+ - `fastify` — more complex setup for marginal perf gain
26
+ - `langchain` / `@vercel/ai` — framework lock-in
27
+ - `sqlite3` (async) — slower than sync for local operations
28
+ - `yargs` / `commander` — unnecessary weight when `util.parseArgs` exists
29
+ - `chokidar` — not needed (no file watching for agent discovery)
30
+
31
+ ## LLM Interface
32
+
33
+ Single interface for all providers using native `fetch`:
34
+ ```
35
+ base_url + api_key + model_name
36
+ ```
37
+
38
+ Works with: OpenRouter, Anthropic, OpenAI, Google, Ollama, LM Studio, or any OpenAI-compatible endpoint. No provider-specific code paths. No SDK version to manage.
39
+
40
+ ```javascript
41
+ // src/llm/client.js — fetch-based, no SDK dependency
42
+ async function callLLM({ baseUrl, apiKey, model, messages, tools }) {
43
+ const response = await fetch(`${baseUrl}/chat/completions`, {
44
+ method: 'POST',
45
+ headers: {
46
+ 'Content-Type': 'application/json',
47
+ 'Authorization': `Bearer ${apiKey}`
48
+ },
49
+ body: JSON.stringify({ model, messages, tools })
50
+ });
51
+ if (!response.ok) throw new Error(`LLM error: ${response.status}`);
52
+ return response.json();
53
+ }
54
+ ```
55
+
56
+ ## ID Generation
57
+
58
+ Use `crypto.randomBytes()` from Node's built-in `crypto` module. **Do not use `nanoid`** — it must be installed as an external dependency and was the cause of runtime crashes when omitted from `package.json`.
59
+
60
+ ```javascript
61
+ // src/utils/id.js
62
+ const crypto = require('crypto');
63
+ function generateId(prefix) {
64
+ return `${prefix}${crypto.randomBytes(8).toString('hex')}`;
65
+ }
66
+ // Usage: generateId('sess_') → 'sess_a1b2c3d4e5f6g7h8'
67
+ ```
68
+
69
+ ## Node.js Version
70
+
71
+ Minimum: **Node.js 18.3+** (for native `fetch`, `util.parseArgs`, `node:test`, and `AbortController`).
72
+
73
+ **Enforced two ways:**
74
+
75
+ 1. `package.json` engines field:
76
+ ```json
77
+ { "engines": { "node": ">=18.3.0" } }
78
+ ```
79
+
80
+ 2. Runtime check at startup entry point:
81
+ ```javascript
82
+ const [major] = process.version.slice(1).split('.').map(Number);
83
+ if (major < 18) {
84
+ console.error(`VeilCLI requires Node.js 18+. Current: ${process.version}`);
85
+ process.exit(1);
86
+ }
87
+ ```
88
+
89
+ ## Dependency Discipline
90
+
91
+ - Use **exact versions** in `package.json` (no `^` or `~` for production deps)
92
+ - Run `npm ci` in CI, not `npm install`
93
+ - Audit `package.json` completeness before any commit (ensure all `require()`d packages are listed)
94
+ - All built-in Node.js modules must be listed as comments in `package.json` (`// built-in: crypto, path, fs, util`) to make dependency inventory obvious
@@ -0,0 +1,232 @@
1
+ # 03 — Agent Architecture
2
+
3
+ ## Core Concept
4
+
5
+ An **agent** is an autonomous AI entity defined by files. **How it's invoked** (chat, task, daemon, subagent) is a **mode**, not a type. The same agent can support multiple modes with different tools, skills, and permissions per mode.
6
+
7
+ > **Required location:** Agents MUST be placed in `.veil/agents/<agentName>/`. A directory named `agents/` at the project root is NOT scanned. This is not configurable. Claude Code projects must be imported via `veil import` first.
8
+
9
+ ## Agent Definition (Two Files)
10
+
11
+ Agent folders are **static and read-only** — safe to download from a registry, version in git, or share between projects. All mutable state (memory, heartbeats) lives elsewhere in `.veil/`.
12
+
13
+ ```
14
+ .veil/agents/<agentName>/
15
+ ├── AGENT.md ← Instructions / system prompt (read-only)
16
+ ├── agent.json ← Configuration (modes, model, tools, permissions)
17
+ ├── SOUL.md ← Optional persona and behavioral identity
18
+ ├── skills/ ← Agent-specific skills
19
+ │ └── <skillName>/
20
+ │ ├── SKILL.md
21
+ │ └── scripts/ ← Scripts referenced by the skill
22
+ └── tools/ ← Agent-level custom tools (scoped to this agent only)
23
+ └── <toolName>/
24
+ ├── tool.json
25
+ └── index.js
26
+ ```
27
+
28
+ ### AGENT.md
29
+ Plain markdown. The agent's instructions, rules, and personality. Always loaded into the system prompt. No configuration here — that goes in `agent.json`.
30
+
31
+ ### agent.json
32
+ All configuration. Defines which modes the agent supports and how each mode behaves.
33
+
34
+ > **Validated at load time.** Every `agent.json` is validated against the JSON schema using ajv when the runtime loads an agent. Invalid files cause a startup error with a clear message — the agent will not be available until fixed. This catches field type errors early rather than causing silent failures during LLM calls.
35
+
36
+ ```json
37
+ {
38
+ "name": "researcher",
39
+ "description": "Deep research agent that investigates topics",
40
+ "model": "anthropic/claude-3.5-sonnet",
41
+ "temperature": 0.7,
42
+ "reasoning": "medium",
43
+ "memory": { "enabled": true, "maxLines": 200 },
44
+ "skillDiscovery": true,
45
+ "modes": {
46
+ "chat": {
47
+ "enabled": true,
48
+ "model": "anthropic/claude-3.5-sonnet",
49
+ "temperature": 0.7,
50
+ "tools": [],
51
+ "disallowedTools": [],
52
+ "skills": [],
53
+ "autoLoadSkills": [],
54
+ "mcpServers": [],
55
+ "allowedAgents": [],
56
+ "disallowedAgents": [],
57
+ "permissions": {}
58
+ },
59
+ "task": {
60
+ "enabled": true,
61
+ "model": "anthropic/claude-3.5-sonnet",
62
+ "temperature": 0.5,
63
+ "reasoning": "medium",
64
+ "tools": [],
65
+ "disallowedTools": ["bash"],
66
+ "skills": [],
67
+ "autoLoadSkills": [],
68
+ "mcpServers": [],
69
+ "allowedAgents": [],
70
+ "disallowedAgents": ["deploy-prod"],
71
+ "maxIterations": 50,
72
+ "maxDurationSeconds": 300,
73
+ "permissions": {}
74
+ },
75
+ "subagent": {
76
+ "enabled": true,
77
+ "model": "anthropic/claude-3-haiku",
78
+ "temperature": 0.3,
79
+ "reasoning": "medium",
80
+ "tools": ["web_search", "read_file", "grep"],
81
+ "disallowedTools": [],
82
+ "skills": ["summarize"],
83
+ "autoLoadSkills": [],
84
+ "mcpServers": [],
85
+ "allowedAgents": [],
86
+ "disallowedAgents": [],
87
+ "maxIterations": 20,
88
+ "maxDurationSeconds": 120,
89
+ "permissions": {}
90
+ },
91
+ "daemon": {
92
+ "enabled": false,
93
+ "model": "anthropic/claude-3-haiku",
94
+ "temperature": 0,
95
+ "reasoning": "medium",
96
+ "cron": "*/5 * * * *",
97
+ "triggers": ["task.completed", "task.failed"],
98
+ "conflictPolicy": "skip",
99
+ "alertRouting": ["log", "webhook:https://slack.com/webhook/xxx"],
100
+ "heartbeatFile": "",
101
+ "tools": [],
102
+ "disallowedTools": [],
103
+ "skills": [],
104
+ "autoLoadSkills": [],
105
+ "mcpServers": [],
106
+ "allowedAgents": [],
107
+ "disallowedAgents": [],
108
+ "permissions": {}
109
+ }
110
+ }
111
+ }
112
+ ```
113
+
114
+ **Critical field type rules (these caused real failures during testing):**
115
+ - **`model`** — must be a **string** e.g. `"anthropic/claude-3.5-sonnet"`. NOT an object like `{ "role": "main" }`. Using an object here causes every agent to fail schema validation.
116
+ - **`modes.chat` / `modes.task` / etc.** — must be an **object** with at minimum `{ "enabled": boolean }`. NOT a bare boolean like `true` or `false`. Using a bare boolean here causes every agent to fail schema validation.
117
+
118
+ **Field rules:**
119
+ - **`tools`** — non-empty = whitelist (only these tools). Empty = all available.
120
+ - **`disallowedTools`** — non-empty = blacklist (all tools EXCEPT these). Empty = no exclusions.
121
+ - `tools` and `disallowedTools` are mutually exclusive — setting both non-empty is a validation error. Both empty = all tools allowed (not an error).
122
+ - Same whitelist/blacklist logic for `allowedAgents`/`disallowedAgents`. `skills` and `mcpServers` are whitelist-only (empty = all).
123
+ - **`skillDiscovery`** — when `false`, no skill names/descriptions listed in system prompt. Agent is skill-blind unless given via `autoLoadSkills`. Default: `true`.
124
+ - **`memory.enabled`** — enables persistent memory for this agent. Default: `true`.
125
+ - **`memory.maxLines`** — MEMORY.md line cap before auto-export. Default: 200 (from settings.json). Per-agent override.
126
+ - **`mcpServers`** — whitelist of MCP server names. Empty = all configured servers available.
127
+ - **`allowedAgents`/`disallowedAgents`** — controls which agents this agent can spawn/message. Available in all modes. Empty = all agents.
128
+ - **`heartbeatFile`** — path to heartbeat checklist for daemon mode. Default: `.veil/heartbeats/<agentName>.md`.
129
+ - `autoLoadSkills` = skills whose full content is always in the system prompt (not on-demand)
130
+ - `permissions` = per-mode overrides (empty = inherit from settings.json)
131
+ - `model` at root = default for all modes. Per-mode `model` overrides the root value.
132
+ - `temperature` at root = default for all modes. Per-mode `temperature` overrides the root value.
133
+ - `reasoning` at root = default for all modes. Per-mode `reasoning` overrides the root value.
134
+ - **`alertRouting`** — array of routing targets: `"log"`, `"webhook:<url>"`, `"agent:<name>"`, `"ably:<channel>"`. Single string also accepted.
135
+
136
+ ## `$AGENT_FOLDER` Variable
137
+
138
+ The runtime exposes `$AGENT_FOLDER` as a template variable in AGENT.md, SKILL.md, and heartbeat files. Resolved at load time to the **absolute path** of the agent's folder.
139
+
140
+ This lets agents and skills reference co-located scripts:
141
+ ```markdown
142
+ Use the script at $AGENT_FOLDER/skills/scrape/scripts/url_to_markdown.js to convert URLs to markdown.
143
+ ```
144
+
145
+ Resolved example: `$AGENT_FOLDER` → `/home/user/project/.veil/agents/scraper`
146
+
147
+ Also available as the `VEIL_AGENT_FOLDER` environment variable in:
148
+ - Custom tool execution
149
+ - Skill scripts (referenced in SKILL.md)
150
+ - PreToolUse / PostToolUse hooks
151
+
152
+ **Not available in:** MCP tool execution (MCP servers are external processes with their own environment).
153
+
154
+ ## Example Agents
155
+
156
+ The source tree ships an `examples/` directory with reference agents that:
157
+ - Are guaranteed to pass the JSON schema validator
158
+ - Have correct field types (string model, object modes)
159
+ - Are used directly in smoke tests and documentation
160
+
161
+ Minimal reference: `examples/agents/hello/` — a simple chat-only agent. Always check this when unsure about the correct format before writing a new `agent.json` from scratch.
162
+
163
+ ## Invocation Modes
164
+
165
+ ### Chat Mode
166
+ - **Trigger:** API call `POST /agents/:name/chat`
167
+ - **Behavior:** Synchronous request/response. Stateful via sessions. User sends message, agent responds.
168
+ - **Session:** Created on first message, resumed via session_id.
169
+ - **Completion:** Never "completes" — session stays open until closed.
170
+
171
+ ### Task Mode
172
+ - **Trigger:** API call `POST /agents/:name/task` or CLI `veil run --agent <name>`
173
+ - **Behavior:** Asynchronous. Agent runs autonomously to completion.
174
+ - **Status flow:** `pending` → `processing` → `waiting` → `finished` / `failed` / `canceled`
175
+ - **Completion:** Agent decides it's done (no more tool calls) or hits limits.
176
+ - **Waiting:** Agent can pause and wait for human input via `task_respond`.
177
+
178
+ ### Daemon Mode
179
+ - **Trigger:** Cron schedule or **system-wide** event trigger (task.completed, task.failed, agent.error)
180
+ - **System-wide events:** These fire when ANY task in the system completes/fails — not a specific one. The daemon gets woken up to handle it. This is different from `task_subscribe` (which an already-running agent uses to watch a specific task it created).
181
+ - **Behavior:** Each tick: reads heartbeat file (from `.veil/heartbeats/<agentName>.md` or custom path), decides action, outputs result.
182
+ - **Output contract:** Must output `HEARTBEAT_OK` or `HEARTBEAT_ALERT: <message>`
183
+ - **Alert routing:** Configured in agent.json — log, webhook URL, another agent, Ably channel.
184
+ - **Conflict policy:** What to do when tick fires mid-execution:
185
+ - `skip` — discard this tick, no-op
186
+ - `queue` — add to FIFO queue (max 10 pending ticks, oldest dropped if exceeded)
187
+ - `restart` — cancel running tick immediately, start new one
188
+ - **Auto-start:** All agents with `daemon.enabled: true` start on `veil start`.
189
+
190
+ ### Sub-Agent Mode
191
+ - **Trigger:** Another agent calls `agent_spawn(agent: "<name>")` (sync or async)
192
+ - **Behavior:** Uses `modes.subagent` config for tools/skills/permissions. (`task_create` always uses `modes.task`, not subagent.)
193
+ - **Sync vs Async:** `agent_spawn(wait: true)` blocks until sub-agent completes (default). `agent_spawn(wait: false)` returns a `taskId` immediately — sub-agent runs in background, result retrievable via `task_status`.
194
+ - **Purpose:** An agent may need different behavior when called as a sub-agent vs running independently. For example, a "researcher" agent might have full web_search + bash in chat mode, but only web_search + read_file when spawned as a sub-agent.
195
+ - **Parallel fan-out:** An orchestrator can `agent_spawn(wait: false)` × N to run multiple sub-agents in parallel. Caller is auto-subscribed — notifications arrive as each completes. Results retrieved via `task_status`.
196
+ - **Fallback:** If `modes.subagent` is not defined or not enabled, the runtime falls back to `modes.chat` config. If `modes.chat` is also not enabled, the spawn fails with an error. Fallback emits a warning log.
197
+
198
+ ## Sub-Agents
199
+
200
+ Any agent can be a sub-agent. The `subagent` mode gives fine-grained control over behavior when called by other agents.
201
+
202
+ - **`agent_spawn`** — Sync (`wait: true`, default) or async (`wait: false`). Creates an isolated context window using `modes.subagent`. Sync returns condensed result directly. Async returns `taskId` for later retrieval.
203
+ - **`task_create`** — Always asynchronous. Creates a task in the queue using `modes.task`. Parent can continue or wait.
204
+
205
+ Built-in agent patterns (all just regular agents with specific tool configs):
206
+ - **Explore** — read-only tools (read_file, grep, glob, list_dir, web_search)
207
+ - **Plan** — read-only + todo_write, no execution tools
208
+ - **Execute** — full tool access
209
+
210
+ ## Built-in Hidden Agents
211
+
212
+ Shipped with the runtime, not visible to users:
213
+ - **compact** — Context compaction (uses `compact` model)
214
+ - **title** — Session title generation (uses `title` model)
215
+ - **summary** — Session summary generation (uses `title` model)
216
+
217
+ ## Agent Discovery
218
+
219
+ Agent resolution order (when an agent is needed by name):
220
+
221
+ ```
222
+ 1. ./.veil/agents/<name>/ ← project-local (user owns, never auto-updated)
223
+ 2. ~/.veil/agents/<name>/ ← global (auto-updated if version hash differs)
224
+ 3. Veil.com registry ← remote (if connected, cached locally)
225
+ 4. NOT FOUND → error
226
+ ```
227
+
228
+ **Startup scan:** On `veil start`, the runtime scans agent folders to auto-start daemons (`daemon.enabled: true`) and populate the `GET /agents` list. No file watcher — agents added after startup are discovered on next invocation or restart.
229
+
230
+ Global instructions: `.veil/AGENT.md` only. No fallback to project root. Claude Code projects must be converted via `veil import`.
231
+
232
+ > **Common mistake:** Placing agents in `./agents/` (visible project root directory) instead of `./.veil/agents/`. The runtime only scans `.veil/agents/` and `~/.veil/agents/`. Agents in the wrong location will silently not appear in `GET /agents`.
@@ -0,0 +1,171 @@
1
+ # 04 — Agent Runtime
2
+
3
+ ## Agentic Loop
4
+
5
+ The core execution model. Runs identically for all modes — the mode only determines how the loop is started and how results are returned.
6
+
7
+ ```
8
+ Load agent definition (AGENT.md + agent.json)
9
+
10
+ Compile system prompt (7-layer assembly)
11
+
12
+ Send to LLM (via native fetch — OpenAI-compatible API)
13
+
14
+ LLM responds with text and/or tool_calls
15
+
16
+ ┌─ If tool_calls: ──────────────────────────────┐
17
+ │ For each tool call: │
18
+ │ 1. Drain message queue (inject pending) │
19
+ │ 2. Check permissions (deny/ask/allow) │
20
+ │ 3. Run PreToolUse hook │
21
+ │ 4. Validate args against JSON Schema (ajv) │
22
+ │ 5. Execute tool │
23
+ │ 6. Run PostToolUse hook │
24
+ │ 7. Append tool result to messages │
25
+ │ Loop back to "Send to LLM" │
26
+ └────────────────────────────────────────────┘
27
+ ┌─ If no tool_calls: ───────────────────────────┐
28
+ │ Chat mode: return response to user │
29
+ │ Task mode: check if agent is done or stuck │
30
+ │ Daemon mode: parse HEARTBEAT_OK/ALERT │
31
+ │ Drain followup queue (inject followup msgs) │
32
+ └────────────────────────────────────────────┘
33
+
34
+ Check limits:
35
+ - max_iterations reached? → fail
36
+ - max_duration reached? → fail
37
+ - context > 85% threshold? → extract memories → trigger compaction → continue
38
+ ```
39
+
40
+ **Implementation pattern:** Async generators (`async function*`). Each iteration yields a state event, allowing consumers (API, CLI) to observe progress without coupling.
41
+
42
+ **Function interface discipline:** All core functions use **object-destructuring parameters**, not positional arguments. This prevents signature mismatch bugs (a real source of failures found in testing) and makes call sites self-documenting:
43
+ ```javascript
44
+ // Correct — named params, order-independent:
45
+ async function runChat({ agent, message, sessionId, settings, cwd }) { ... }
46
+
47
+ // Wrong — positional, fragile:
48
+ async function runChat(agent, message, options) { ... }
49
+ ```
50
+ All public functions in `src/core/`, `src/api/`, and `src/cli/` follow this convention. JSDoc `@param` annotations are required on all exported functions.
51
+
52
+ ## Model Delegation (v1)
53
+
54
+ Three model roles. Each can have a different `base_url` + `api_key` + `model`.
55
+
56
+ ```json
57
+ {
58
+ "models": {
59
+ "main": { "model": "anthropic/claude-3.5-sonnet" },
60
+ "compact": { "model": "anthropic/claude-3-haiku" },
61
+ "title": { "model": "anthropic/claude-3-haiku" }
62
+ }
63
+ }
64
+ ```
65
+
66
+ | Role | Used For | Typical Model |
67
+ |------|----------|---------------|
68
+ | `main` | Reasoning, planning, tool decisions, file edits (str_replace) | Claude 3.5 Sonnet, GPT-4o |
69
+ | `compact` | Context compaction / summarization | Claude Haiku, GPT-4o-mini |
70
+ | `title` | Session title, session summary | Cheapest available |
71
+
72
+ If only `main` is configured, all roles fall back to `main`.
73
+
74
+ **LLM calls use native `fetch` directly** — no OpenAI SDK. See `02-tech-stack.md` and `src/llm/client.js`. The `callLLM()` function validates its inputs before calling the API:
75
+ ```javascript
76
+ async function callLLM({ baseUrl, apiKey, model, messages, tools = [] }) {
77
+ if (!Array.isArray(tools)) throw new Error('tools must be an array');
78
+ if (!Array.isArray(messages) || messages.length === 0) throw new Error('messages must be a non-empty array');
79
+ // ... fetch call
80
+ }
81
+ ```
82
+ Failing fast on bad inputs prevents silent failures where tool calls never reach the LLM.
83
+
84
+ **Apply layer deferred to future version.** For v1, `edit_file` uses search-and-replace via the main model (like Claude Code).
85
+
86
+ ## Context Engineering
87
+
88
+ ### Auto-Compaction
89
+ - Triggered when context window usage exceeds **85%** (configurable)
90
+ - Hidden `compact` agent summarizes conversation history (runs using the `compact` model role from settings.json)
91
+ - Preserves: key decisions, unresolved issues, file paths, active TODOs, error patterns
92
+ - Result replaces old messages with a single compacted summary message
93
+ - TODOs from `todo_write` are always included in compacted context
94
+
95
+ ### Pre-Compaction Memory Extraction
96
+ Before compaction runs, the runtime asks the main model:
97
+ > "What key facts, decisions, or learnings from this conversation should be remembered long-term?"
98
+
99
+ - Model returns structured memories → runtime writes them to agent's MEMORY.md via `memory_write`
100
+ - Model can return `NO_MEMORY` to skip
101
+ - Only runs if `memory.enabled: true` in agent.json (default: on)
102
+ - This ensures important context survives compaction even if the agent didn't explicitly `memory_write`
103
+
104
+ ### Context Trimming (applied every iteration, in order)
105
+
106
+ **Execution order:**
107
+ 1. **Observation masking** (first) — Tool outputs beyond 10 turns old are replaced with `[output hidden]` placeholders. Agent's reasoning text is preserved. Configurable via `settings.json` → `compaction.observationMaskingTurns` (default: 10).
108
+ 2. **Tool result clearing** (second) — For messages in turns 1–10 (within masking window), large raw outputs (file reads, bash outputs) are truncated to ~500 chars + `"... [truncated]"`. This keeps recent tool outputs readable but bounded. Messages already masked in step 1 are skipped (already replaced with placeholder).
109
+ 3. **Compaction** (third, conditional) — If context still exceeds 85% threshold after steps 1-2, full compaction triggers (see above).
110
+
111
+ **Why this order:** Masking is the cheapest operation (placeholder replacement). Clearing is slightly more work (truncation). Compaction is expensive (LLM call). Running them in this order minimizes cost while progressively reducing context.
112
+
113
+ ### Token Tracking
114
+ Per session/task:
115
+ - Input tokens, output tokens, cache tokens
116
+ - Estimated cost (based on model pricing config)
117
+ - Iteration count, tool call count
118
+ - Stored in SQLite, queryable via API
119
+
120
+ ## System Prompt Architecture (7 Layers)
121
+
122
+ Assembled at session start. Layers 1-2 are static (optimize for prompt caching). Layers 3-7 are dynamic.
123
+
124
+ | Layer | Content | Type |
125
+ |-------|---------|------|
126
+ | 1. Base Core | Identity, version, mandate | Static, cached |
127
+ | 2. Safety & Rules | Permission rules, instruction hierarchy | Static, cached |
128
+ | 3. Environment | OS, shell, cwd, git status, MCP servers | Dynamic per session |
129
+ | 4. Agent Instructions | AGENT.md + SOUL.md + Memory files | User-defined |
130
+ | 5. Tools & Capabilities | Tool names + descriptions (progressive disclosure) | Conditional |
131
+ | 6. Task Heuristics | "Read before edit", "no over-engineering", mode-specific tips | Conditional |
132
+ | 7. Session Context | Compacted history, active TODOs, task brief | Dynamic |
133
+
134
+ **Token budget:** ~6,400-8,400 tokens target. Hard cap: 12,000 tokens.
135
+
136
+ ### System Reminders (Mid-Conversation Injections)
137
+
138
+ | Reminder | Trigger | Purpose |
139
+ |----------|---------|---------|
140
+ | Anti-drift | Every N iterations | Refocus on original task |
141
+ | Stall recovery | 3+ iterations with no progress | Suggest different approach |
142
+ | Safe-edit | Before write/edit tool | Remind to read first |
143
+ | Plan adherence | When TODO exists | Check progress against plan |
144
+ | Compaction warning | At 75% context | Prepare for imminent compaction |
145
+
146
+ ### Instruction Priority Hierarchy
147
+
148
+ Baked into Layer 2 (Safety & Rules):
149
+ ```
150
+ Safety rules (Layer 2) > User's current message > AGENT.md > SOUL.md > MEMORY.md > System defaults
151
+ ```
152
+
153
+ ## Progress Events
154
+
155
+ The runtime emits progress events during execution. These are:
156
+ - **Written to SQLite** (queryable via `GET /tasks/:id/events`)
157
+ - **Written to stdout** in one-shot mode (`VEIL_PROGRESS={...}`)
158
+ - **Pushed via Ably** for remote consumers
159
+
160
+ **Automatic events** (emitted by runtime, `type: "iteration"`):
161
+ - `iteration.start` — new loop iteration
162
+ - `tool.start` — tool execution beginning
163
+ - `tool.end` — tool execution complete (with duration)
164
+ - `compaction.start` / `compaction.end`
165
+ - `status.change` — task status transition
166
+
167
+ **Agent-driven events** (via `log_write` tool, `type: "custom"`):
168
+ - Custom progress messages, research updates, milestones
169
+ - Format is free-form — agents decide what's useful to report
170
+
171
+ All events include a `type` field (`"iteration"` or `"custom"`) so API consumers can filter/distinguish runtime progress from agent-specific progress.