@newsails/veil-cli 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.veil/agents/analyst/AGENT.md +21 -0
- package/.veil/agents/analyst/agent.json +23 -0
- package/.veil/agents/assistant/AGENT.md +15 -0
- package/.veil/agents/assistant/agent.json +19 -0
- package/.veil/agents/coder/AGENT.md +18 -0
- package/.veil/agents/coder/agent.json +19 -0
- package/.veil/agents/hello/AGENT.md +5 -0
- package/.veil/agents/hello/agent.json +13 -0
- package/.veil/agents/writer/AGENT.md +12 -0
- package/.veil/agents/writer/agent.json +17 -0
- package/.veil/memory/MEMORY.md +343 -0
- package/.veil/memory/agents/analyst/MEMORY.md +55 -0
- package/.veil/memory/agents/hello/MEMORY.md +12 -0
- package/.veil/runtime.pid +1 -0
- package/.veil/settings.json +10 -0
- package/.veil-studio/studio.db +0 -0
- package/.veil-studio/studio.db-shm +0 -0
- package/.veil-studio/studio.db-wal +0 -0
- package/PLAN/01-vision.md +26 -0
- package/PLAN/02-tech-stack.md +94 -0
- package/PLAN/03-agents.md +232 -0
- package/PLAN/04-runtime.md +171 -0
- package/PLAN/05-tools.md +211 -0
- package/PLAN/06-communication.md +243 -0
- package/PLAN/07-storage.md +218 -0
- package/PLAN/08-api-cli.md +153 -0
- package/PLAN/09-permissions.md +108 -0
- package/PLAN/10-ably.md +105 -0
- package/PLAN/11-file-formats.md +442 -0
- package/PLAN/12-folder-structure.md +205 -0
- package/PLAN/13-operations.md +212 -0
- package/PLAN/README.md +23 -0
- package/README.md +128 -0
- package/REPORT.md +174 -0
- package/TODO.md +45 -0
- package/ai-tests/FRONTEND_PROMPT.md +220 -0
- package/ai-tests/Research & Planning.md +814 -0
- package/ai-tests/prompt-001-basic-api.md +230 -0
- package/ai-tests/prompt-002-basic-flows.md +230 -0
- package/ai-tests/prompt-003-agent-behaviors.md +220 -0
- package/api/middleware.js +60 -0
- package/api/routes/agents.js +193 -0
- package/api/routes/chat.js +93 -0
- package/api/routes/completions.js +122 -0
- package/api/routes/daemons.js +80 -0
- package/api/routes/memory.js +169 -0
- package/api/routes/models.js +40 -0
- package/api/routes/remote-methods.js +74 -0
- package/api/routes/sessions.js +208 -0
- package/api/routes/settings.js +108 -0
- package/api/routes/system.js +50 -0
- package/api/routes/tasks.js +270 -0
- package/api/server.js +120 -0
- package/cli/formatter.js +70 -0
- package/cli/index.js +443 -0
- package/cli/parser.js +113 -0
- package/config/config.json +10 -0
- package/config/models.json +6826 -0
- package/core/agent.js +329 -0
- package/core/cancel.js +38 -0
- package/core/compaction.js +176 -0
- package/core/events.js +13 -0
- package/core/loop.js +564 -0
- package/core/memory.js +51 -0
- package/core/prompt.js +185 -0
- package/core/queue.js +96 -0
- package/core/registry.js +291 -0
- package/core/remote-methods.js +124 -0
- package/core/router.js +386 -0
- package/core/running-sessions.js +18 -0
- package/docs/api/01-system.md +84 -0
- package/docs/api/02-agents.md +374 -0
- package/docs/api/03-chat.md +269 -0
- package/docs/api/04-tasks.md +470 -0
- package/docs/api/05-sessions.md +444 -0
- package/docs/api/06-daemons.md +142 -0
- package/docs/api/07-memory.md +186 -0
- package/docs/api/08-settings.md +133 -0
- package/docs/api/09-models.md +119 -0
- package/docs/api/09-websocket.md +350 -0
- package/docs/api/10-completions.md +134 -0
- package/docs/api/README.md +116 -0
- package/docs/guide/01-quickstart.md +220 -0
- package/docs/guide/02-folder-structure.md +185 -0
- package/docs/guide/03-configuration.md +252 -0
- package/docs/guide/04-agents.md +267 -0
- package/docs/guide/05-cli.md +290 -0
- package/docs/guide/06-tools.md +643 -0
- package/docs/guide/07-permissions.md +236 -0
- package/docs/guide/08-memory.md +139 -0
- package/docs/guide/09-multi-agent.md +271 -0
- package/docs/guide/10-daemons.md +226 -0
- package/docs/guide/README.md +53 -0
- package/docs/index.html +623 -0
- package/examples/README.md +151 -0
- package/examples/agents/assistant/AGENT.md +31 -0
- package/examples/agents/assistant/SOUL.md +9 -0
- package/examples/agents/assistant/agent.json +74 -0
- package/examples/agents/hello/AGENT.md +15 -0
- package/examples/agents/hello/agent.json +14 -0
- package/examples/agents/monitor/AGENT.md +51 -0
- package/examples/agents/monitor/agent.json +33 -0
- package/examples/agents/monitor/heartbeats/monitor.md +24 -0
- package/examples/agents/orchestrator/AGENT.md +70 -0
- package/examples/agents/orchestrator/agent.json +30 -0
- package/examples/agents/researcher/AGENT.md +52 -0
- package/examples/agents/researcher/agent.json +49 -0
- package/examples/agents/researcher/skills/web-research.md +28 -0
- package/examples/skills/code-review.md +72 -0
- package/examples/skills/summarise.md +59 -0
- package/examples/skills/web-research.md +42 -0
- package/examples/tools/word-count/index.js +27 -0
- package/examples/tools/word-count/tool.json +18 -0
- package/infrastructure/database.js +563 -0
- package/infrastructure/scheduler.js +122 -0
- package/llm/client.js +206 -0
- package/migrations/001-initial.sql +121 -0
- package/migrations/002-debuggability.sql +13 -0
- package/migrations/003-drop-orphaned-columns.sql +72 -0
- package/migrations/004-session-message-token-fields.sql +78 -0
- package/migrations/005-session-thinking.sql +5 -0
- package/package.json +30 -0
- package/schemas/agent.json +143 -0
- package/schemas/settings.json +111 -0
- package/scripts/fetch-models.js +93 -0
- package/session-debug-scenario.md +248 -0
- package/settings/fields.js +52 -0
- package/system-prompts/base-core.md +7 -0
- package/system-prompts/environment.md +13 -0
- package/system-prompts/reminders/anti-drift.md +6 -0
- package/system-prompts/reminders/stall-recovery.md +10 -0
- package/system-prompts/safety-rules.md +25 -0
- package/system-prompts/task-heuristics.md +27 -0
- package/test/client.js +71 -0
- package/test/integration/01-health.test.js +25 -0
- package/test/integration/02-agents.test.js +80 -0
- package/test/integration/03-chat-hello.test.js +48 -0
- package/test/integration/04-chat-multiturn.test.js +61 -0
- package/test/integration/05-chat-writer.test.js +48 -0
- package/test/integration/06-task-basic.test.js +68 -0
- package/test/integration/07-task-tools.test.js +74 -0
- package/test/integration/08-task-code-analysis.test.js +69 -0
- package/test/integration/09-memory-analyst.test.js +63 -0
- package/test/integration/10-task-advanced.test.js +85 -0
- package/test/integration/11-sessions-advanced.test.js +84 -0
- package/test/integration/12-assistant-chat-tools.test.js +75 -0
- package/test/integration/13-edge-cases.test.js +99 -0
- package/test/integration/14-cancel.test.js +62 -0
- package/test/integration/15-debug.test.js +106 -0
- package/test/integration/16-memory-api.test.js +83 -0
- package/test/integration/17-settings-api.test.js +41 -0
- package/test/integration/18-tool-search-activation.test.js +119 -0
- package/test/results/.gitkeep +0 -0
- package/test/runner.js +206 -0
- package/test/smoke.js +216 -0
- package/tools/agent_message.js +85 -0
- package/tools/agent_send.js +80 -0
- package/tools/agent_spawn.js +44 -0
- package/tools/bash.js +49 -0
- package/tools/edit_file.js +41 -0
- package/tools/glob.js +64 -0
- package/tools/grep.js +82 -0
- package/tools/list_dir.js +63 -0
- package/tools/log_write.js +31 -0
- package/tools/memory_read.js +38 -0
- package/tools/memory_search.js +65 -0
- package/tools/memory_write.js +42 -0
- package/tools/read_file.js +48 -0
- package/tools/sleep.js +22 -0
- package/tools/task_create.js +41 -0
- package/tools/task_respond.js +37 -0
- package/tools/task_spawn.js +64 -0
- package/tools/task_status.js +39 -0
- package/tools/task_subscribe.js +37 -0
- package/tools/todo_read.js +26 -0
- package/tools/todo_write.js +38 -0
- package/tools/tool_activate.js +24 -0
- package/tools/tool_search.js +24 -0
- package/tools/web_fetch.js +50 -0
- package/tools/web_search.js +52 -0
- package/tools/write_file.js +28 -0
- package/ui/api.js +190 -0
- package/ui/app.js +281 -0
- package/ui/index.html +382 -0
- package/ui/views/agents.js +377 -0
- package/ui/views/chat.js +610 -0
- package/ui/views/connection.js +96 -0
- package/ui/views/daemons.js +129 -0
- package/ui/views/feed.js +194 -0
- package/ui/views/memory.js +263 -0
- package/ui/views/models.js +146 -0
- package/ui/views/sessions.js +314 -0
- package/ui/views/settings.js +142 -0
- package/ui/views/tasks.js +415 -0
- package/utils/context.js +49 -0
- package/utils/id.js +16 -0
- package/utils/models.js +88 -0
- package/utils/paths.js +213 -0
- package/utils/settings.js +172 -0
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# 02 — Tech Stack
|
|
2
|
+
|
|
3
|
+
## Language
|
|
4
|
+
|
|
5
|
+
**Node.js + JSDoc** (no TypeScript). Plain JavaScript with JSDoc type annotations for IDE support and type checking.
|
|
6
|
+
|
|
7
|
+
> **v2 Note:** TypeScript is strongly recommended for a future version. Most of the function-signature mismatches, missing exports, and field-name inconsistencies discovered during testing would be caught at compile time.
|
|
8
|
+
|
|
9
|
+
## Dependencies (Minimal Set)
|
|
10
|
+
|
|
11
|
+
| Concern | Package | Justification |
|
|
12
|
+
|---------|---------|---------------|
|
|
13
|
+
| **LLM Client** | native `fetch` (built-in) | Node 18+ built-in. No SDK version drift. Works with any OpenAI-compatible endpoint (OpenRouter, Anthropic, Ollama, LM Studio, etc.) without external dependencies. |
|
|
14
|
+
| **REST Server** | `express` | Simplest, most maintained, vast middleware ecosystem |
|
|
15
|
+
| **CORS** | `cors` | Required by express for cross-origin requests |
|
|
16
|
+
| **Validation** | `ajv` + `ajv-formats` | JSON Schema validator, compiles schemas to fast native functions. `ajv-formats` required for `date-time` and other format validators. |
|
|
17
|
+
| **Database** | `better-sqlite3` | Fastest SQLite for Node.js, synchronous (optimal for local) |
|
|
18
|
+
| **Cron** | `node-cron` | Lightweight, Unix-native cron syntax |
|
|
19
|
+
| **MCP Client** | TBD (existing lib) | Not custom-built, supports stdio + Streamable HTTP |
|
|
20
|
+
| **CLI Parsing** | `util.parseArgs` | Built-in Node.js (>=18.3), zero external deps |
|
|
21
|
+
|
|
22
|
+
**Rejected alternatives:**
|
|
23
|
+
- `openai` SDK — SDK versioning caused `OpenAI is not a constructor` failures (v3 vs v4 API mismatch). Native `fetch` is simpler, portable, and requires no versioning decisions.
|
|
24
|
+
- `nanoid` — was missing from package.json, caused runtime crash. Replaced by `crypto.randomBytes()` from Node's built-in `crypto` module (see `src/utils/id.js`).
|
|
25
|
+
- `fastify` — more complex setup for marginal perf gain
|
|
26
|
+
- `langchain` / `@vercel/ai` — framework lock-in
|
|
27
|
+
- `sqlite3` (async) — slower than sync for local operations
|
|
28
|
+
- `yargs` / `commander` — unnecessary weight when `util.parseArgs` exists
|
|
29
|
+
- `chokidar` — not needed (no file watching for agent discovery)
|
|
30
|
+
|
|
31
|
+
## LLM Interface
|
|
32
|
+
|
|
33
|
+
Single interface for all providers using native `fetch`:
|
|
34
|
+
```
|
|
35
|
+
base_url + api_key + model_name
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Works with: OpenRouter, Anthropic, OpenAI, Google, Ollama, LM Studio, or any OpenAI-compatible endpoint. No provider-specific code paths. No SDK version to manage.
|
|
39
|
+
|
|
40
|
+
```javascript
|
|
41
|
+
// src/llm/client.js — fetch-based, no SDK dependency
|
|
42
|
+
async function callLLM({ baseUrl, apiKey, model, messages, tools }) {
|
|
43
|
+
const response = await fetch(`${baseUrl}/chat/completions`, {
|
|
44
|
+
method: 'POST',
|
|
45
|
+
headers: {
|
|
46
|
+
'Content-Type': 'application/json',
|
|
47
|
+
'Authorization': `Bearer ${apiKey}`
|
|
48
|
+
},
|
|
49
|
+
body: JSON.stringify({ model, messages, tools })
|
|
50
|
+
});
|
|
51
|
+
if (!response.ok) throw new Error(`LLM error: ${response.status}`);
|
|
52
|
+
return response.json();
|
|
53
|
+
}
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## ID Generation
|
|
57
|
+
|
|
58
|
+
Use `crypto.randomBytes()` from Node's built-in `crypto` module. **Do not use `nanoid`** — it must be installed as an external dependency and was the cause of runtime crashes when omitted from `package.json`.
|
|
59
|
+
|
|
60
|
+
```javascript
|
|
61
|
+
// src/utils/id.js
|
|
62
|
+
const crypto = require('crypto');
|
|
63
|
+
function generateId(prefix) {
|
|
64
|
+
return `${prefix}${crypto.randomBytes(8).toString('hex')}`;
|
|
65
|
+
}
|
|
66
|
+
// Usage: generateId('sess_') → 'sess_a1b2c3d4e5f6g7h8'
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
## Node.js Version
|
|
70
|
+
|
|
71
|
+
Minimum: **Node.js 18.3+** (for native `fetch`, `util.parseArgs`, `node:test`, and `AbortController`).
|
|
72
|
+
|
|
73
|
+
**Enforced two ways:**
|
|
74
|
+
|
|
75
|
+
1. `package.json` engines field:
|
|
76
|
+
```json
|
|
77
|
+
{ "engines": { "node": ">=18.3.0" } }
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
2. Runtime check at startup entry point:
|
|
81
|
+
```javascript
|
|
82
|
+
const [major] = process.version.slice(1).split('.').map(Number);
|
|
83
|
+
if (major < 18) {
|
|
84
|
+
console.error(`VeilCLI requires Node.js 18+. Current: ${process.version}`);
|
|
85
|
+
process.exit(1);
|
|
86
|
+
}
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
## Dependency Discipline
|
|
90
|
+
|
|
91
|
+
- Use **exact versions** in `package.json` (no `^` or `~` for production deps)
|
|
92
|
+
- Run `npm ci` in CI, not `npm install`
|
|
93
|
+
- Audit `package.json` completeness before any commit (ensure all `require()`d packages are listed)
|
|
94
|
+
- All built-in Node.js modules must be listed as comments in `package.json` (`// built-in: crypto, path, fs, util`) to make dependency inventory obvious
|
|
@@ -0,0 +1,232 @@
|
|
|
1
|
+
# 03 — Agent Architecture
|
|
2
|
+
|
|
3
|
+
## Core Concept
|
|
4
|
+
|
|
5
|
+
An **agent** is an autonomous AI entity defined by files. **How it's invoked** (chat, task, daemon, subagent) is a **mode**, not a type. The same agent can support multiple modes with different tools, skills, and permissions per mode.
|
|
6
|
+
|
|
7
|
+
> **Required location:** Agents MUST be placed in `.veil/agents/<agentName>/`. A directory named `agents/` at the project root is NOT scanned. This is not configurable. Claude Code projects must be imported via `veil import` first.
|
|
8
|
+
|
|
9
|
+
## Agent Definition (Two Files)
|
|
10
|
+
|
|
11
|
+
Agent folders are **static and read-only** — safe to download from a registry, version in git, or share between projects. All mutable state (memory, heartbeats) lives elsewhere in `.veil/`.
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
.veil/agents/<agentName>/
|
|
15
|
+
├── AGENT.md ← Instructions / system prompt (read-only)
|
|
16
|
+
├── agent.json ← Configuration (modes, model, tools, permissions)
|
|
17
|
+
├── SOUL.md ← Optional persona and behavioral identity
|
|
18
|
+
├── skills/ ← Agent-specific skills
|
|
19
|
+
│ └── <skillName>/
|
|
20
|
+
│ ├── SKILL.md
|
|
21
|
+
│ └── scripts/ ← Scripts referenced by the skill
|
|
22
|
+
└── tools/ ← Agent-level custom tools (scoped to this agent only)
|
|
23
|
+
└── <toolName>/
|
|
24
|
+
├── tool.json
|
|
25
|
+
└── index.js
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
### AGENT.md
|
|
29
|
+
Plain markdown. The agent's instructions, rules, and personality. Always loaded into the system prompt. No configuration here — that goes in `agent.json`.
|
|
30
|
+
|
|
31
|
+
### agent.json
|
|
32
|
+
All configuration. Defines which modes the agent supports and how each mode behaves.
|
|
33
|
+
|
|
34
|
+
> **Validated at load time.** Every `agent.json` is validated against the JSON schema using ajv when the runtime loads an agent. Invalid files cause a startup error with a clear message — the agent will not be available until fixed. This catches field type errors early rather than causing silent failures during LLM calls.
|
|
35
|
+
|
|
36
|
+
```json
|
|
37
|
+
{
|
|
38
|
+
"name": "researcher",
|
|
39
|
+
"description": "Deep research agent that investigates topics",
|
|
40
|
+
"model": "anthropic/claude-3.5-sonnet",
|
|
41
|
+
"temperature": 0.7,
|
|
42
|
+
"reasoning": "medium",
|
|
43
|
+
"memory": { "enabled": true, "maxLines": 200 },
|
|
44
|
+
"skillDiscovery": true,
|
|
45
|
+
"modes": {
|
|
46
|
+
"chat": {
|
|
47
|
+
"enabled": true,
|
|
48
|
+
"model": "anthropic/claude-3.5-sonnet",
|
|
49
|
+
"temperature": 0.7,
|
|
50
|
+
"tools": [],
|
|
51
|
+
"disallowedTools": [],
|
|
52
|
+
"skills": [],
|
|
53
|
+
"autoLoadSkills": [],
|
|
54
|
+
"mcpServers": [],
|
|
55
|
+
"allowedAgents": [],
|
|
56
|
+
"disallowedAgents": [],
|
|
57
|
+
"permissions": {}
|
|
58
|
+
},
|
|
59
|
+
"task": {
|
|
60
|
+
"enabled": true,
|
|
61
|
+
"model": "anthropic/claude-3.5-sonnet",
|
|
62
|
+
"temperature": 0.5,
|
|
63
|
+
"reasoning": "medium",
|
|
64
|
+
"tools": [],
|
|
65
|
+
"disallowedTools": ["bash"],
|
|
66
|
+
"skills": [],
|
|
67
|
+
"autoLoadSkills": [],
|
|
68
|
+
"mcpServers": [],
|
|
69
|
+
"allowedAgents": [],
|
|
70
|
+
"disallowedAgents": ["deploy-prod"],
|
|
71
|
+
"maxIterations": 50,
|
|
72
|
+
"maxDurationSeconds": 300,
|
|
73
|
+
"permissions": {}
|
|
74
|
+
},
|
|
75
|
+
"subagent": {
|
|
76
|
+
"enabled": true,
|
|
77
|
+
"model": "anthropic/claude-3-haiku",
|
|
78
|
+
"temperature": 0.3,
|
|
79
|
+
"reasoning": "medium",
|
|
80
|
+
"tools": ["web_search", "read_file", "grep"],
|
|
81
|
+
"disallowedTools": [],
|
|
82
|
+
"skills": ["summarize"],
|
|
83
|
+
"autoLoadSkills": [],
|
|
84
|
+
"mcpServers": [],
|
|
85
|
+
"allowedAgents": [],
|
|
86
|
+
"disallowedAgents": [],
|
|
87
|
+
"maxIterations": 20,
|
|
88
|
+
"maxDurationSeconds": 120,
|
|
89
|
+
"permissions": {}
|
|
90
|
+
},
|
|
91
|
+
"daemon": {
|
|
92
|
+
"enabled": false,
|
|
93
|
+
"model": "anthropic/claude-3-haiku",
|
|
94
|
+
"temperature": 0,
|
|
95
|
+
"reasoning": "medium",
|
|
96
|
+
"cron": "*/5 * * * *",
|
|
97
|
+
"triggers": ["task.completed", "task.failed"],
|
|
98
|
+
"conflictPolicy": "skip",
|
|
99
|
+
"alertRouting": ["log", "webhook:https://slack.com/webhook/xxx"],
|
|
100
|
+
"heartbeatFile": "",
|
|
101
|
+
"tools": [],
|
|
102
|
+
"disallowedTools": [],
|
|
103
|
+
"skills": [],
|
|
104
|
+
"autoLoadSkills": [],
|
|
105
|
+
"mcpServers": [],
|
|
106
|
+
"allowedAgents": [],
|
|
107
|
+
"disallowedAgents": [],
|
|
108
|
+
"permissions": {}
|
|
109
|
+
}
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
**Critical field type rules (these caused real failures during testing):**
|
|
115
|
+
- **`model`** — must be a **string** e.g. `"anthropic/claude-3.5-sonnet"`. NOT an object like `{ "role": "main" }`. Using an object here causes every agent to fail schema validation.
|
|
116
|
+
- **`modes.chat` / `modes.task` / etc.** — must be an **object** with at minimum `{ "enabled": boolean }`. NOT a bare boolean like `true` or `false`. Using a bare boolean here causes every agent to fail schema validation.
|
|
117
|
+
|
|
118
|
+
**Field rules:**
|
|
119
|
+
- **`tools`** — non-empty = whitelist (only these tools). Empty = all available.
|
|
120
|
+
- **`disallowedTools`** — non-empty = blacklist (all tools EXCEPT these). Empty = no exclusions.
|
|
121
|
+
- `tools` and `disallowedTools` are mutually exclusive — setting both non-empty is a validation error. Both empty = all tools allowed (not an error).
|
|
122
|
+
- Same whitelist/blacklist logic for `allowedAgents`/`disallowedAgents`. `skills` and `mcpServers` are whitelist-only (empty = all).
|
|
123
|
+
- **`skillDiscovery`** — when `false`, no skill names/descriptions listed in system prompt. Agent is skill-blind unless given via `autoLoadSkills`. Default: `true`.
|
|
124
|
+
- **`memory.enabled`** — enables persistent memory for this agent. Default: `true`.
|
|
125
|
+
- **`memory.maxLines`** — MEMORY.md line cap before auto-export. Default: 200 (from settings.json). Per-agent override.
|
|
126
|
+
- **`mcpServers`** — whitelist of MCP server names. Empty = all configured servers available.
|
|
127
|
+
- **`allowedAgents`/`disallowedAgents`** — controls which agents this agent can spawn/message. Available in all modes. Empty = all agents.
|
|
128
|
+
- **`heartbeatFile`** — path to heartbeat checklist for daemon mode. Default: `.veil/heartbeats/<agentName>.md`.
|
|
129
|
+
- `autoLoadSkills` = skills whose full content is always in the system prompt (not on-demand)
|
|
130
|
+
- `permissions` = per-mode overrides (empty = inherit from settings.json)
|
|
131
|
+
- `model` at root = default for all modes. Per-mode `model` overrides the root value.
|
|
132
|
+
- `temperature` at root = default for all modes. Per-mode `temperature` overrides the root value.
|
|
133
|
+
- `reasoning` at root = default for all modes. Per-mode `reasoning` overrides the root value.
|
|
134
|
+
- **`alertRouting`** — array of routing targets: `"log"`, `"webhook:<url>"`, `"agent:<name>"`, `"ably:<channel>"`. Single string also accepted.
|
|
135
|
+
|
|
136
|
+
## `$AGENT_FOLDER` Variable
|
|
137
|
+
|
|
138
|
+
The runtime exposes `$AGENT_FOLDER` as a template variable in AGENT.md, SKILL.md, and heartbeat files. Resolved at load time to the **absolute path** of the agent's folder.
|
|
139
|
+
|
|
140
|
+
This lets agents and skills reference co-located scripts:
|
|
141
|
+
```markdown
|
|
142
|
+
Use the script at $AGENT_FOLDER/skills/scrape/scripts/url_to_markdown.js to convert URLs to markdown.
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
Resolved example: `$AGENT_FOLDER` → `/home/user/project/.veil/agents/scraper`
|
|
146
|
+
|
|
147
|
+
Also available as the `VEIL_AGENT_FOLDER` environment variable in:
|
|
148
|
+
- Custom tool execution
|
|
149
|
+
- Skill scripts (referenced in SKILL.md)
|
|
150
|
+
- PreToolUse / PostToolUse hooks
|
|
151
|
+
|
|
152
|
+
**Not available in:** MCP tool execution (MCP servers are external processes with their own environment).
|
|
153
|
+
|
|
154
|
+
## Example Agents
|
|
155
|
+
|
|
156
|
+
The source tree ships an `examples/` directory with reference agents that:
|
|
157
|
+
- Are guaranteed to pass the JSON schema validator
|
|
158
|
+
- Have correct field types (string model, object modes)
|
|
159
|
+
- Are used directly in smoke tests and documentation
|
|
160
|
+
|
|
161
|
+
Minimal reference: `examples/agents/hello/` — a simple chat-only agent. Always check this when unsure about the correct format before writing a new `agent.json` from scratch.
|
|
162
|
+
|
|
163
|
+
## Invocation Modes
|
|
164
|
+
|
|
165
|
+
### Chat Mode
|
|
166
|
+
- **Trigger:** API call `POST /agents/:name/chat`
|
|
167
|
+
- **Behavior:** Synchronous request/response. Stateful via sessions. User sends message, agent responds.
|
|
168
|
+
- **Session:** Created on first message, resumed via session_id.
|
|
169
|
+
- **Completion:** Never "completes" — session stays open until closed.
|
|
170
|
+
|
|
171
|
+
### Task Mode
|
|
172
|
+
- **Trigger:** API call `POST /agents/:name/task` or CLI `veil run --agent <name>`
|
|
173
|
+
- **Behavior:** Asynchronous. Agent runs autonomously to completion.
|
|
174
|
+
- **Status flow:** `pending` → `processing` → `waiting` → `finished` / `failed` / `canceled`
|
|
175
|
+
- **Completion:** Agent decides it's done (no more tool calls) or hits limits.
|
|
176
|
+
- **Waiting:** Agent can pause and wait for human input via `task_respond`.
|
|
177
|
+
|
|
178
|
+
### Daemon Mode
|
|
179
|
+
- **Trigger:** Cron schedule or **system-wide** event trigger (task.completed, task.failed, agent.error)
|
|
180
|
+
- **System-wide events:** These fire when ANY task in the system completes/fails — not a specific one. The daemon gets woken up to handle it. This is different from `task_subscribe` (which an already-running agent uses to watch a specific task it created).
|
|
181
|
+
- **Behavior:** Each tick: reads heartbeat file (from `.veil/heartbeats/<agentName>.md` or custom path), decides action, outputs result.
|
|
182
|
+
- **Output contract:** Must output `HEARTBEAT_OK` or `HEARTBEAT_ALERT: <message>`
|
|
183
|
+
- **Alert routing:** Configured in agent.json — log, webhook URL, another agent, Ably channel.
|
|
184
|
+
- **Conflict policy:** What to do when tick fires mid-execution:
|
|
185
|
+
- `skip` — discard this tick, no-op
|
|
186
|
+
- `queue` — add to FIFO queue (max 10 pending ticks, oldest dropped if exceeded)
|
|
187
|
+
- `restart` — cancel running tick immediately, start new one
|
|
188
|
+
- **Auto-start:** All agents with `daemon.enabled: true` start on `veil start`.
|
|
189
|
+
|
|
190
|
+
### Sub-Agent Mode
|
|
191
|
+
- **Trigger:** Another agent calls `agent_spawn(agent: "<name>")` (sync or async)
|
|
192
|
+
- **Behavior:** Uses `modes.subagent` config for tools/skills/permissions. (`task_create` always uses `modes.task`, not subagent.)
|
|
193
|
+
- **Sync vs Async:** `agent_spawn(wait: true)` blocks until sub-agent completes (default). `agent_spawn(wait: false)` returns a `taskId` immediately — sub-agent runs in background, result retrievable via `task_status`.
|
|
194
|
+
- **Purpose:** An agent may need different behavior when called as a sub-agent vs running independently. For example, a "researcher" agent might have full web_search + bash in chat mode, but only web_search + read_file when spawned as a sub-agent.
|
|
195
|
+
- **Parallel fan-out:** An orchestrator can `agent_spawn(wait: false)` × N to run multiple sub-agents in parallel. Caller is auto-subscribed — notifications arrive as each completes. Results retrieved via `task_status`.
|
|
196
|
+
- **Fallback:** If `modes.subagent` is not defined or not enabled, the runtime falls back to `modes.chat` config. If `modes.chat` is also not enabled, the spawn fails with an error. Fallback emits a warning log.
|
|
197
|
+
|
|
198
|
+
## Sub-Agents
|
|
199
|
+
|
|
200
|
+
Any agent can be a sub-agent. The `subagent` mode gives fine-grained control over behavior when called by other agents.
|
|
201
|
+
|
|
202
|
+
- **`agent_spawn`** — Sync (`wait: true`, default) or async (`wait: false`). Creates an isolated context window using `modes.subagent`. Sync returns condensed result directly. Async returns `taskId` for later retrieval.
|
|
203
|
+
- **`task_create`** — Always asynchronous. Creates a task in the queue using `modes.task`. Parent can continue or wait.
|
|
204
|
+
|
|
205
|
+
Built-in agent patterns (all just regular agents with specific tool configs):
|
|
206
|
+
- **Explore** — read-only tools (read_file, grep, glob, list_dir, web_search)
|
|
207
|
+
- **Plan** — read-only + todo_write, no execution tools
|
|
208
|
+
- **Execute** — full tool access
|
|
209
|
+
|
|
210
|
+
## Built-in Hidden Agents
|
|
211
|
+
|
|
212
|
+
Shipped with the runtime, not visible to users:
|
|
213
|
+
- **compact** — Context compaction (uses `compact` model)
|
|
214
|
+
- **title** — Session title generation (uses `title` model)
|
|
215
|
+
- **summary** — Session summary generation (uses `title` model)
|
|
216
|
+
|
|
217
|
+
## Agent Discovery
|
|
218
|
+
|
|
219
|
+
Agent resolution order (when an agent is needed by name):
|
|
220
|
+
|
|
221
|
+
```
|
|
222
|
+
1. ./.veil/agents/<name>/ ← project-local (user owns, never auto-updated)
|
|
223
|
+
2. ~/.veil/agents/<name>/ ← global (auto-updated if version hash differs)
|
|
224
|
+
3. Veil.com registry ← remote (if connected, cached locally)
|
|
225
|
+
4. NOT FOUND → error
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
**Startup scan:** On `veil start`, the runtime scans agent folders to auto-start daemons (`daemon.enabled: true`) and populate the `GET /agents` list. No file watcher — agents added after startup are discovered on next invocation or restart.
|
|
229
|
+
|
|
230
|
+
Global instructions: `.veil/AGENT.md` only. No fallback to project root. Claude Code projects must be converted via `veil import`.
|
|
231
|
+
|
|
232
|
+
> **Common mistake:** Placing agents in `./agents/` (visible project root directory) instead of `./.veil/agents/`. The runtime only scans `.veil/agents/` and `~/.veil/agents/`. Agents in the wrong location will silently not appear in `GET /agents`.
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
# 04 — Agent Runtime
|
|
2
|
+
|
|
3
|
+
## Agentic Loop
|
|
4
|
+
|
|
5
|
+
The core execution model. Runs identically for all modes — the mode only determines how the loop is started and how results are returned.
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
Load agent definition (AGENT.md + agent.json)
|
|
9
|
+
↓
|
|
10
|
+
Compile system prompt (7-layer assembly)
|
|
11
|
+
↓
|
|
12
|
+
Send to LLM (via native fetch — OpenAI-compatible API)
|
|
13
|
+
↓
|
|
14
|
+
LLM responds with text and/or tool_calls
|
|
15
|
+
↓
|
|
16
|
+
┌─ If tool_calls: ──────────────────────────────┐
|
|
17
|
+
│ For each tool call: │
|
|
18
|
+
│ 1. Drain message queue (inject pending) │
|
|
19
|
+
│ 2. Check permissions (deny/ask/allow) │
|
|
20
|
+
│ 3. Run PreToolUse hook │
|
|
21
|
+
│ 4. Validate args against JSON Schema (ajv) │
|
|
22
|
+
│ 5. Execute tool │
|
|
23
|
+
│ 6. Run PostToolUse hook │
|
|
24
|
+
│ 7. Append tool result to messages │
|
|
25
|
+
│ Loop back to "Send to LLM" │
|
|
26
|
+
└────────────────────────────────────────────┘
|
|
27
|
+
┌─ If no tool_calls: ───────────────────────────┐
|
|
28
|
+
│ Chat mode: return response to user │
|
|
29
|
+
│ Task mode: check if agent is done or stuck │
|
|
30
|
+
│ Daemon mode: parse HEARTBEAT_OK/ALERT │
|
|
31
|
+
│ Drain followup queue (inject followup msgs) │
|
|
32
|
+
└────────────────────────────────────────────┘
|
|
33
|
+
↓
|
|
34
|
+
Check limits:
|
|
35
|
+
- max_iterations reached? → fail
|
|
36
|
+
- max_duration reached? → fail
|
|
37
|
+
- context > 85% threshold? → extract memories → trigger compaction → continue
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
**Implementation pattern:** Async generators (`async function*`). Each iteration yields a state event, allowing consumers (API, CLI) to observe progress without coupling.
|
|
41
|
+
|
|
42
|
+
**Function interface discipline:** All core functions use **object-destructuring parameters**, not positional arguments. This prevents signature mismatch bugs (a real source of failures found in testing) and makes call sites self-documenting:
|
|
43
|
+
```javascript
|
|
44
|
+
// Correct — named params, order-independent:
|
|
45
|
+
async function runChat({ agent, message, sessionId, settings, cwd }) { ... }
|
|
46
|
+
|
|
47
|
+
// Wrong — positional, fragile:
|
|
48
|
+
async function runChat(agent, message, options) { ... }
|
|
49
|
+
```
|
|
50
|
+
All public functions in `src/core/`, `src/api/`, and `src/cli/` follow this convention. JSDoc `@param` annotations are required on all exported functions.
|
|
51
|
+
|
|
52
|
+
## Model Delegation (v1)
|
|
53
|
+
|
|
54
|
+
Three model roles. Each can have a different `base_url` + `api_key` + `model`.
|
|
55
|
+
|
|
56
|
+
```json
|
|
57
|
+
{
|
|
58
|
+
"models": {
|
|
59
|
+
"main": { "model": "anthropic/claude-3.5-sonnet" },
|
|
60
|
+
"compact": { "model": "anthropic/claude-3-haiku" },
|
|
61
|
+
"title": { "model": "anthropic/claude-3-haiku" }
|
|
62
|
+
}
|
|
63
|
+
}
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
| Role | Used For | Typical Model |
|
|
67
|
+
|------|----------|---------------|
|
|
68
|
+
| `main` | Reasoning, planning, tool decisions, file edits (str_replace) | Claude 3.5 Sonnet, GPT-4o |
|
|
69
|
+
| `compact` | Context compaction / summarization | Claude Haiku, GPT-4o-mini |
|
|
70
|
+
| `title` | Session title, session summary | Cheapest available |
|
|
71
|
+
|
|
72
|
+
If only `main` is configured, all roles fall back to `main`.
|
|
73
|
+
|
|
74
|
+
**LLM calls use native `fetch` directly** — no OpenAI SDK. See `02-tech-stack.md` and `src/llm/client.js`. The `callLLM()` function validates its inputs before calling the API:
|
|
75
|
+
```javascript
|
|
76
|
+
async function callLLM({ baseUrl, apiKey, model, messages, tools = [] }) {
|
|
77
|
+
if (!Array.isArray(tools)) throw new Error('tools must be an array');
|
|
78
|
+
if (!Array.isArray(messages) || messages.length === 0) throw new Error('messages must be a non-empty array');
|
|
79
|
+
// ... fetch call
|
|
80
|
+
}
|
|
81
|
+
```
|
|
82
|
+
Failing fast on bad inputs prevents silent failures where tool calls never reach the LLM.
|
|
83
|
+
|
|
84
|
+
**Apply layer deferred to future version.** For v1, `edit_file` uses search-and-replace via the main model (like Claude Code).
|
|
85
|
+
|
|
86
|
+
## Context Engineering
|
|
87
|
+
|
|
88
|
+
### Auto-Compaction
|
|
89
|
+
- Triggered when context window usage exceeds **85%** (configurable)
|
|
90
|
+
- Hidden `compact` agent summarizes conversation history (runs using the `compact` model role from settings.json)
|
|
91
|
+
- Preserves: key decisions, unresolved issues, file paths, active TODOs, error patterns
|
|
92
|
+
- Result replaces old messages with a single compacted summary message
|
|
93
|
+
- TODOs from `todo_write` are always included in compacted context
|
|
94
|
+
|
|
95
|
+
### Pre-Compaction Memory Extraction
|
|
96
|
+
Before compaction runs, the runtime asks the main model:
|
|
97
|
+
> "What key facts, decisions, or learnings from this conversation should be remembered long-term?"
|
|
98
|
+
|
|
99
|
+
- Model returns structured memories → runtime writes them to agent's MEMORY.md via `memory_write`
|
|
100
|
+
- Model can return `NO_MEMORY` to skip
|
|
101
|
+
- Only runs if `memory.enabled: true` in agent.json (default: on)
|
|
102
|
+
- This ensures important context survives compaction even if the agent didn't explicitly `memory_write`
|
|
103
|
+
|
|
104
|
+
### Context Trimming (applied every iteration, in order)
|
|
105
|
+
|
|
106
|
+
**Execution order:**
|
|
107
|
+
1. **Observation masking** (first) — Tool outputs beyond 10 turns old are replaced with `[output hidden]` placeholders. Agent's reasoning text is preserved. Configurable via `settings.json` → `compaction.observationMaskingTurns` (default: 10).
|
|
108
|
+
2. **Tool result clearing** (second) — For messages in turns 1–10 (within masking window), large raw outputs (file reads, bash outputs) are truncated to ~500 chars + `"... [truncated]"`. This keeps recent tool outputs readable but bounded. Messages already masked in step 1 are skipped (already replaced with placeholder).
|
|
109
|
+
3. **Compaction** (third, conditional) — If context still exceeds 85% threshold after steps 1-2, full compaction triggers (see above).
|
|
110
|
+
|
|
111
|
+
**Why this order:** Masking is the cheapest operation (placeholder replacement). Clearing is slightly more work (truncation). Compaction is expensive (LLM call). Running them in this order minimizes cost while progressively reducing context.
|
|
112
|
+
|
|
113
|
+
### Token Tracking
|
|
114
|
+
Per session/task:
|
|
115
|
+
- Input tokens, output tokens, cache tokens
|
|
116
|
+
- Estimated cost (based on model pricing config)
|
|
117
|
+
- Iteration count, tool call count
|
|
118
|
+
- Stored in SQLite, queryable via API
|
|
119
|
+
|
|
120
|
+
## System Prompt Architecture (7 Layers)
|
|
121
|
+
|
|
122
|
+
Assembled at session start. Layers 1-2 are static (optimize for prompt caching). Layers 3-7 are dynamic.
|
|
123
|
+
|
|
124
|
+
| Layer | Content | Type |
|
|
125
|
+
|-------|---------|------|
|
|
126
|
+
| 1. Base Core | Identity, version, mandate | Static, cached |
|
|
127
|
+
| 2. Safety & Rules | Permission rules, instruction hierarchy | Static, cached |
|
|
128
|
+
| 3. Environment | OS, shell, cwd, git status, MCP servers | Dynamic per session |
|
|
129
|
+
| 4. Agent Instructions | AGENT.md + SOUL.md + Memory files | User-defined |
|
|
130
|
+
| 5. Tools & Capabilities | Tool names + descriptions (progressive disclosure) | Conditional |
|
|
131
|
+
| 6. Task Heuristics | "Read before edit", "no over-engineering", mode-specific tips | Conditional |
|
|
132
|
+
| 7. Session Context | Compacted history, active TODOs, task brief | Dynamic |
|
|
133
|
+
|
|
134
|
+
**Token budget:** ~6,400-8,400 tokens target. Hard cap: 12,000 tokens.
|
|
135
|
+
|
|
136
|
+
### System Reminders (Mid-Conversation Injections)
|
|
137
|
+
|
|
138
|
+
| Reminder | Trigger | Purpose |
|
|
139
|
+
|----------|---------|---------|
|
|
140
|
+
| Anti-drift | Every N iterations | Refocus on original task |
|
|
141
|
+
| Stall recovery | 3+ iterations with no progress | Suggest different approach |
|
|
142
|
+
| Safe-edit | Before write/edit tool | Remind to read first |
|
|
143
|
+
| Plan adherence | When TODO exists | Check progress against plan |
|
|
144
|
+
| Compaction warning | At 75% context | Prepare for imminent compaction |
|
|
145
|
+
|
|
146
|
+
### Instruction Priority Hierarchy
|
|
147
|
+
|
|
148
|
+
Baked into Layer 2 (Safety & Rules):
|
|
149
|
+
```
|
|
150
|
+
Safety rules (Layer 2) > User's current message > AGENT.md > SOUL.md > MEMORY.md > System defaults
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
## Progress Events
|
|
154
|
+
|
|
155
|
+
The runtime emits progress events during execution. These are:
|
|
156
|
+
- **Written to SQLite** (queryable via `GET /tasks/:id/events`)
|
|
157
|
+
- **Written to stdout** in one-shot mode (`VEIL_PROGRESS={...}`)
|
|
158
|
+
- **Pushed via Ably** for remote consumers
|
|
159
|
+
|
|
160
|
+
**Automatic events** (emitted by runtime, `type: "iteration"`):
|
|
161
|
+
- `iteration.start` — new loop iteration
|
|
162
|
+
- `tool.start` — tool execution beginning
|
|
163
|
+
- `tool.end` — tool execution complete (with duration)
|
|
164
|
+
- `compaction.start` / `compaction.end`
|
|
165
|
+
- `status.change` — task status transition
|
|
166
|
+
|
|
167
|
+
**Agent-driven events** (via `log_write` tool, `type: "custom"`):
|
|
168
|
+
- Custom progress messages, research updates, milestones
|
|
169
|
+
- Format is free-form — agents decide what's useful to report
|
|
170
|
+
|
|
171
|
+
All events include a `type` field (`"iteration"` or `"custom"`) so API consumers can filter/distinguish runtime progress from agent-specific progress.
|