@newsails/veil-cli 1.0.1

package/PLAN/11-file-formats.md

@@ -0,0 +1,442 @@
+# 11 — File Formats
+
+## agent.json
+
+Agent configuration. Defines modes, model, tools, permissions.
+
+> **Critical field types — common mistakes:**
+> - `model` must be a **string** (e.g. `"moonshotai/kimi-k2.5"`), NOT an object like `{ "role": "main" }`
+> - Each `modes.*` entry must be an **object** with at minimum `{ "enabled": boolean }`, NOT a bare boolean like `true`
+> - Both of these were real bugs found during testing that caused every agent to fail validation
+
+```json
+{
+  "name": "researcher",
+  "description": "Deep research agent that investigates topics",
+  "model": "anthropic/claude-3.5-sonnet",
+  "temperature": 0.7,
+  "reasoning": "medium",
+  "memory": { "enabled": true, "maxLines": 200 },
+  "skillDiscovery": true,
+  "modes": {
+    "chat": {
+      "enabled": true,
+      "tools": [],
+      "disallowedTools": [],
+      "skills": [],
+      "autoLoadSkills": [],
+      "mcpServers": [],
+      "permissions": {}
+    },
+    "task": {
+      "enabled": true,
+      "tools": [],
+      "disallowedTools": ["bash"],
+      "skills": [],
+      "autoLoadSkills": [],
+      "mcpServers": [],
+      "maxIterations": 50,
+      "maxDurationSeconds": 300,
+      "permissions": {}
+    },
+    "subagent": {
+      "enabled": true,
+      "tools": ["web_search", "read_file", "grep"],
+      "disallowedTools": [],
+      "skills": ["summarize"],
+      "autoLoadSkills": [],
+      "mcpServers": [],
+      "allowedAgents": [],
+      "disallowedAgents": [],
+      "maxIterations": 20,
+      "maxDurationSeconds": 120,
+      "permissions": {}
+    },
+    "daemon": {
+      "enabled": false,
+      "cron": "*/5 * * * *",
+      "triggers": [],
+      "conflictPolicy": "skip",
+      "alertRouting": ["log", "webhook:https://slack.com/webhook/xxx"],
+      "heartbeatFile": "",
+      "tools": [],
+      "disallowedTools": [],
+      "skills": [],
+      "autoLoadSkills": [],
+      "mcpServers": [],
+      "permissions": {}
+    }
+  }
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | string | Agent identifier (overrides folder name) |
+| `description` | string | Human-readable description |
+| `model` | string | Default model for all modes |
+| `temperature` | number | 0.0–1.0 |
+| `reasoning` | string | Enable reasoning mode for this agent (default: "medium"). Options: "low", "medium", "high" |
+| `memory.enabled` | boolean | Enable persistent memory for this agent (default: true) |
+| `memory.maxLines` | number | MEMORY.md line cap before auto-export (default: 200 from settings.json) |
+| `skillDiscovery` | boolean | List skill names in system prompt (default: true). When false, agent is skill-blind. |
+| `modes.chat.enabled` | boolean | Whether chat mode is supported |
+| `modes.task.enabled` | boolean | Whether task mode is supported |
+| `modes.task.maxIterations` | number | Override global limit |
+| `modes.task.maxDurationSeconds` | number | Override global limit |
+| `modes.subagent.enabled` | boolean | Whether sub-agent mode is supported |
+| `modes.subagent.maxIterations` | number | Limit when called as sub-agent |
+| `modes.subagent.maxDurationSeconds` | number | Limit when called as sub-agent |
+| `modes.subagent.allowedAgents` | string[] | Agents this sub-agent can spawn/message (empty = all) |
+| `modes.subagent.disallowedAgents` | string[] | Agents this sub-agent cannot spawn/message |
+| `modes.daemon.enabled` | boolean | Whether daemon mode is supported |
+| `modes.daemon.cron` | string | Cron expression for scheduling |
+| `modes.daemon.triggers` | string[] | System-wide event triggers: `task.completed`, `task.failed`, `agent.error`. See **Daemon Trigger Payload** below. |
+| `modes.daemon.conflictPolicy` | string | skip / queue / restart |
+| `modes.daemon.alertRouting` | string \| string[] | Array of routing targets: `"log"`, `"webhook:<url>"`, `"agent:<name>"`, `"ably:<channel>"`. Single string also accepted. |
+| `modes.daemon.heartbeatFile` | string | Custom path to heartbeat file (default: `.veil/heartbeats/<name>.md`) |
+| `modes.*.tools` | string[] | Tool whitelist (empty = all) |
+| `modes.*.disallowedTools` | string[] | Tool blacklist (mutually exclusive with `tools`) |
+| `modes.*.skills` | string[] | Skill whitelist (empty = all) |
+| `modes.*.autoLoadSkills` | string[] | Skills always loaded in full |
+| `modes.*.mcpServers` | string[] | MCP server whitelist (empty = all configured) |
+| `modes.*.permissions` | object | `{ allow: [], deny: [] }` per-mode overrides |
+
+**Minimal valid example** (guaranteed to pass schema validation):
+```json
+{
+  "name": "hello",
+  "description": "A minimal reference agent",
+  "model": "anthropic/claude-3-haiku",
+  "temperature": 0.5,
+  "reasoning": null,
+  "modes": {
+    "chat": { "enabled": true },
+    "task": { "enabled": false },
+    "subagent": { "enabled": false },
+    "daemon": { "enabled": false }
+  }
+}
+```
+
+This example is the basis for `examples/agents/hello/agent.json` in the source tree — it is validated in CI and used as the reference for documentation.
+
+---
+
+## AGENT.md
+
+Plain markdown. Agent instructions and system prompt content. No configuration — that's in agent.json.
+
+```markdown
+You are a research agent. Your job is to investigate topics thoroughly
+and produce detailed, well-sourced reports.
+
+## Rules
+- Always cite your sources
+- Prefer primary sources over summaries
+- If you can't find reliable information, say so
+```
+
+---
+
+## SOUL.md
+
+Optional. Plain markdown. Persona and behavioral identity. Loaded after AGENT.md.
+
+```markdown
+You speak in a concise, professional tone.
+You never apologize unnecessarily.
+You prefer bullet points over long paragraphs.
+```
+
+---
+
+## Memory Files
+
+Persistent memory lives in `.veil/memory/`, NOT in agent folders (agent folders are static/read-only).
+
+```
+.veil/memory/
+├── MEMORY.md                ← global memory (always loaded, ~200-line cap)
+├── <topic>.md               ← auto-exported global memory files
+└── agents/
+    └── <agentName>/
+        ├── MEMORY.md            ← agent-specific memory (always loaded, ~200-line cap)
+        └── <topic>.md           ← auto-exported agent memory files
+```
+
+**Main MEMORY.md** is always loaded into the system prompt (Layer 4). Capped at ~200 lines (configurable).
+**Auto-export:** When MEMORY.md exceeds the line cap, the compact model moves older entries into a separate `<topic>.md` file.
+**Memory tools:** `memory_write` (append), `memory_read` (read), `memory_search` (search across all memory files).
+**System prompt includes:** Main MEMORY.md content + list of available memory files with descriptions.
+**Pre-compaction:** Before context compaction, runtime asks model to extract key learnings → writes to MEMORY.md (model can return `NO_MEMORY` to skip).
+
+```markdown
+# Memory
+
+## User Preferences
+- Prefers reports in markdown format
+- Works in EST timezone
+
+## Known Facts
+- Main project is called ProjectX
+- Production server is at 192.168.1.100
+```
+
+---
+
+## Heartbeat Files
+
+Daemon checklists live in `.veil/heartbeats/`, NOT in agent folders.
+
+```
+.veil/heartbeats/
+└── <agentName>.md       ← per-agent heartbeat checklist
+```
+
+Plain markdown. Read on each daemon tick. Agent.json `heartbeatFile` field can override the default path.
+
+```markdown
+# Heartbeat Checklist
+
+- Check if any tasks have been in "processing" state for more than 1 hour
+- Monitor ./logs/ folder for new ERROR lines since last check
+- If it's Monday morning, generate a weekly summary
+```
+
+**Output contract:** Agent MUST output `HEARTBEAT_OK` or `HEARTBEAT_ALERT: <message>`.
+
+### Daemon Trigger Payload
+
+When a daemon is triggered by a system event (not cron), it receives context via a system message:
+
+```
+[System] Daemon triggered by event: task.completed
+{
+  "event": "task.completed",
+  "taskId": "task_abc123",
+  "agentName": "researcher",
+  "status": "finished",
+  "output": { ... },
+  "timestamp": "2026-03-01T08:00:00Z"
+}
+```
+
+The daemon can then decide what action to take based on the event payload.
+
+---
+
+## SKILL.md
+
+Markdown with YAML frontmatter. Prompt template invokable by agents.
+
+```markdown
+---
+name: review-code
+description: Review code for security and performance issues
+argument-hint: [file-path]
+allowed-tools: bash, read_file, edit_file
+context: fork
+agent: anthropic/claude-3-haiku
+disable-model-invocation: false
+---
+
+Review the following file: $1
+
+Focus on:
+1. Security vulnerabilities
+2. Performance bottlenecks
+3. Code style issues
+
+Additional context: $ARGUMENTS
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | string | Skill identifier |
+| `description` | string | Human-readable description |
+| `argument-hint` | string | Hint shown for usage |
+| `allowed-tools` | string | Comma-separated tool whitelist |
+| `context` | main / fork | `main` = inject into conversation. `fork` = isolated sub-agent. |
+| `agent` | string | Model to use (fork mode only). Named `agent` for Claude Code heritage; functionally equivalent to `model`. |
+| `disable-model-invocation` | boolean | Only user/API can trigger this skill, not the agent itself (default: false) |
+
+---
+
+## tool.json
+
+Custom tool manifest.
+
+```json
+{
+  "name": "send_slack_message",
+  "description": "Sends a message to a Slack channel",
+  "input_schema": {
+    "type": "object",
+    "properties": {
+      "channel": { "type": "string", "description": "Slack channel name" },
+      "message": { "type": "string", "description": "Message to send" }
+    },
+    "required": ["channel", "message"]
+  },
+  "timeout": 30
+}
+```
+
+---
+
+## config.json (Forkable, Source Code Only)
+
+All branding/naming values. This is a **source code file** inside the VeilCLI repository (`config/config.json`), NOT a user-facing config file. Anyone forking VeilCLI changes only this file to rebrand.
+
+```json
+{
+  "productName": "VeilCli",
+  "productNameShort": "veil",
+  "domain": "veil.com",
+  "configDir": ".veil",
+  "globalConfigDir": "~/.veil",
+  "defaultPort": 5050,
+  "authHeader": "X-Veil-Secret",
+  "tokenPrefix": "kbt_",
+  "sessionIdPrefix": "sess_",
+  "taskIdPrefix": "task_"
+}
+```
+
+---
+
+## auth.json
+
+Local authentication. Gitignored. The `llm` section provides **default** LLM credentials. If `settings.json` also defines `models.main.base_url` / `models.main.api_key`, the settings.json values take precedence per model role.
+
+```json
+{
+  "veil_token": "kbt_xxxxxxxxxxxx",
+  "llm": {
+    "base_url": "https://openrouter.ai/api/v1",
+    "api_key": "sk-or-xxxxxxxxxxxx"
+  }
+}
+```
+
+---
+
+## settings.json
+
+Full configuration. Merged across layers (global → project → local → CLI flags).
+
+> **Location:** Project settings go in `.veil/settings.json` — NOT in the project root as `settings.json`. A common mistake is placing settings at `./settings.json` which is never read by the runtime.
+
+> **Canonical field names:** The LLM credential fields are `base_url` and `api_key` (snake_case). Do NOT use `apiEndpoint`, `apiKey`, or other variants. These names match the fields in `auth.json` and are exported as constants from `src/settings/fields.js` to prevent inconsistency across the codebase.
+
+```json
+{
+  "models": {
+    "main": {
+      "model": "anthropic/claude-3.5-sonnet",
+      "base_url": "",
+      "api_key": ""
+    },
+    "compact": {
+      "model": "anthropic/claude-3-haiku"
+    },
+    "title": {
+      "model": "anthropic/claude-3-haiku"
+    }
+  },
+  "permissions": {
+    "allow": [],
+    "ask": [],
+    "deny": []
+  },
+  "hooks": {
+    "PreToolUse": "",
+    "PostToolUse": ""
+  },
+  "mcpServers": {
+    "filesystem": {
+      "transport": "stdio",
+      "command": "npx",
+      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/project"]
+    },
+    "web-search": {
+      "transport": "http",
+      "url": "http://localhost:3001/mcp"
+    }
+  },
+  "api": {
+    "port": 5050,
+    "secret": ""
+  },
+  "compaction": {
+    "threshold": 0.85,
+    "enabled": true,
+    "observationMaskingTurns": 10
+  },
+  "memory": {
+    "enabled": true,
+    "maxLines": 200,
+    "preCompactionExtraction": true
+  },
+  "storage": {
+    "retention": {
+      "sessions": { "maxAgeDays": 90, "maxCount": 10000 },
+      "tasks": { "maxAgeDays": 30, "maxCount": 5000 }
+    }
+  },
+  "limits": {
+    "maxIterations": 50,
+    "maxDurationSeconds": 300,
+    "maxConcurrentTasks": 5,
+    "maxSubAgentDepth": 3
+  },
+  "retry": {
+    "maxAttempts": 3,
+    "delaySeconds": 5,
+    "onExhausted": "fail"
+  },
+  "ably": {
+    "enabled": false,
+    "instanceId": "",
+    "tokenEndpoint": "https://veil.com/api/v1/ably/token",
+    "requestTimeout": 30000,
+    "maxRequestAge": 60000
+  }
+}
+```
+
+**`onExhausted` valid values:** `"fail"` (default — task fails) or `"wait"` (task pauses in `waiting` status for human retry via `task_respond`).
+
+Settings resolution order (last wins):
+```
+1. VeilCLI hardcoded defaults
+2. ~/.veil/settings.json
+3. ./.veil/settings.json
+4. ./.veil/settings.local.json    ← same format as settings.json, partial overrides merged
+5. CLI flags
+```
+
+`settings.local.json` uses the same schema as `settings.json`. Only the fields present are merged — omitted fields inherit from the previous layer. This file is gitignored and intended for developer-specific overrides (e.g., different API keys, ports).
+
+**Settings field names are defined as constants in `src/settings/fields.js`** and used everywhere in the runtime code. Never hardcode field name strings (`'api_key'`, `'base_url'`) inline — always import from `fields.js`. This prevents the class of bugs where one module reads `settings.apiKey` and another writes `settings.api_key`.
+
+---
+
+## version.json
+
+Present **only** in globally cached agents/tools from the Veil.com registry (`~/.veil/agents/<name>/`, `~/.veil/tools/<name>/`). Project-level agents/tools (`.veil/agents/`, `.veil/tools/`) do NOT have this file — they are user-owned and not registry-managed.
+
+```json
+{
+  "hash": "sha256:a1b2c3d4e5f6...",
+  "version": "1.2.0",
+  "updatedAt": "2026-03-01T08:00:00Z"
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `hash` | string | SHA256 hash of agent/tool contents |
+| `version` | string | Version from Veil.com registry |
+| `updatedAt` | string | ISO 8601 timestamp of last update |

package/PLAN/12-folder-structure.md

@@ -0,0 +1,205 @@
+# 12 — Folder Structure
+
+## Project-Level
+
+```
+./                              ← any folder user runs veil from (clean, no Veil files at root)
+└── .veil/
+    ├── AGENT.md               ← global instructions for all agents in this project
+    ├── settings.json          ← project settings (committed to git)
+    ├── settings.local.json    ← local overrides (gitignored)
+    ├── auth.json              ← local auth override (gitignored)
+    ├── runtime.pid            ← written on server start, deleted on stop
+    ├── agents/                ← agent definitions (STATIC, read-only)
+    │   └── <agentName>/       ← $AGENT_FOLDER resolves to this path
+    │       ├── AGENT.md       ← agent instructions
+    │       ├── agent.json     ← agent config (modes, model, tools)
+    │       ├── SOUL.md        ← optional persona
+    │       ├── skills/
+    │       │   └── <skillName>/
+    │       │       ├── SKILL.md
+    │       │       └── scripts/   ← scripts referenced by the skill
+    │       └── tools/         ← agent-level custom tools
+    │           └── <toolName>/
+    │               ├── tool.json
+    │               └── index.js
+    ├── memory/                ← persistent memory (MUTABLE)
+    │   ├── MEMORY.md          ← global memory (always loaded, ~200-line cap)
+    │   ├── <topic>.md         ← auto-exported global memory files
+    │   └── agents/
+    │       └── <agentName>/
+    │           ├── MEMORY.md  ← agent-specific memory (always loaded, ~200-line cap)
+    │           └── <topic>.md ← auto-exported agent memory files
+    ├── heartbeats/            ← daemon checklists (MUTABLE)
+    │   └── <agentName>.md     ← per-agent heartbeat checklist
+    ├── skills/                ← shared skills available to all agents
+    │   └── <skillName>/
+    │       └── SKILL.md
+    └── tools/                 ← project-level custom tools
+        └── <toolName>/
+            ├── tool.json
+            └── index.js
+```
+
+**Global instructions:** `.veil/AGENT.md` only. No fallback to project root. Claude Code projects must be converted via `veil import`.
+
+> **Common mistake:** Placing agents in `./agents/` (project root) instead of `./.veil/agents/`. The runtime only scans `.veil/agents/`.
+
+## Global
+
+```
+~/.veil/
+├── auth.json                   ← Veil.com token + LLM API config
+├── settings.json               ← global default settings
+├── data.db                     ← SQLite database (all runtime data)
+├── agents/                     ← agents cached from Veil.com (static)
+│   └── <agentName>/
+│       ├── version.json        ← SHA256 hash for registry version checking
+│       └── ...                 ← same structure as project agent
+├── skills/                     ← global shared skills
+│   └── <skillName>/
+│       └── SKILL.md
+├── tools/                      ← global custom tools
+│   └── <toolName>/
+│       ├── tool.json
+│       ├── index.js
+│       └── version.json        ← SHA256 hash for registry version checking
+├── backups/                    ← auto-backups of data.db
+│   └── data-2026-03-01.db
+└── archive/                    ← archived old sessions/tasks (gzipped JSONL)
+    └── sessions-2026-03.jsonl.gz
+```
+
+## Source Code (Runtime)
+
+```
+veilcli/
+├── package.json                ← bin: { "veil": "./src/cli/index.js" }  ← MUST point here
+├── config/
+│   └── config.json             ← forkable branding/naming
+├── src/
+│   ├── core/                   ← Pure business logic (transport agnostic)
+│   │   ├── loop.js             ← Agentic loop (async generator)
+│   │   ├── router.js           ← Multi-model delegation (IntelligenceRouter)
+│   │   ├── registry.js         ← Tool registration + ajv compilation
+│   │   ├── prompt.js           ← System prompt assembly (7 layers)
+│   │   ├── compaction.js       ← Context compaction logic
+│   │   ├── agent.js            ← Agent loader + discovery
+│   │   ├── memory.js           ← Memory management (read/write/search/export)
+│   │   └── queue.js            ← Agent message queue
+│   ├── api/                    ← REST-specific transport
+│   │   ├── routes.js
+│   │   ├── middleware.js
+│   │   └── controllers.js
+│   ├── cli/
+│   │   ├── index.js            ← CLI entry point (bin target)
+│   │   ├── parser.js
+│   │   └── formatter.js
+│   ├── llm/
+│   │   └── client.js           ← Native fetch-based LLM client (no SDK)
+│   ├── settings/
+│   │   └── fields.js           ← Settings field name constants (single source of truth)
+│   ├── utils/
+│   │   ├── context.js          ← CWD singleton — set once at startup, read everywhere
+│   │   ├── id.js               ← generateId() via crypto.randomBytes (no nanoid)
+│   │   └── paths.js            ← Path helpers: getProjectAgentsDir(cwd), etc.
+│   ├── tools/                  ← Built-in tool implementations
+│   │   ├── bash.js
+│   │   ├── read-file.js
+│   │   ├── write-file.js
+│   │   └── ...
+│   ├── infrastructure/         ← External resource management
+│   │   ├── database.js         ← SQLite setup, migrations, queries
+│   │   ├── scheduler.js        ← Cron scheduling for daemons
+│   │   ├── ably.js             ← Ably bridge
+│   │   └── mcp.js              ← MCP client wrapper
+│   └── system-prompts/         ← System prompt fragments
+│       ├── base-core.md
+│       ├── safety-rules.md
+│       ├── environment.md
+│       ├── task-heuristics.md
+│       └── reminders/
+│           ├── anti-drift.md
+│           ├── stall-recovery.md
+│           └── ...
+├── migrations/                 ← SQLite migration files (all use IF NOT EXISTS)
+│   ├── 001-initial.sql
+│   └── ...
+├── examples/                   ← Reference agents guaranteed to pass schema validation
+│   └── agents/
+│       └── hello/              ← Minimal chat-only agent (used in smoke test)
+│           ├── AGENT.md
+│           └── agent.json
+└── test/
+    ├── unit/
+    ├── api/
+    └── smoke.js                ← Fast startup + health + one chat call
+```
+
+### Critical Implementation Rules
+
+**1. CWD singleton (`src/utils/context.js`):**
+The server's working directory is set **once** at startup (in `src/cli/index.js`) and stored in a singleton. All routes, agents, and tools read from `context.getCwd()`. **Never call `process.cwd()` inside `src/core/`, `src/api/`, `src/tools/`, or `src/infrastructure/`.** Only `src/cli/index.js` may call `process.cwd()` (at launch time).
+
+```javascript
+// src/utils/context.js
+let _cwd = null;
+module.exports = {
+  setCwd(cwd) { _cwd = cwd; },
+  getCwd() {
+    if (!_cwd) throw new Error('Context not initialized — call setCwd() at startup');
+    return _cwd;
+  }
+};
+// Usage in routes:
+const { getCwd } = require('../utils/context');
+const agentsDir = paths.getProjectAgentsDir(getCwd());
+```
+
+This also eliminates the circular dependency that arises when routes import from `server.js` to get `cwd` and `server.js` imports routes.
+
+**2. Path helpers (`src/utils/paths.js`):**
+All path construction goes through named helpers. **Do not construct `.veil/` paths inline.**
+
+```javascript
+// Correct — explicit, clear:
+const agentsDir = paths.getProjectAgentsDir(cwd);  // → cwd/.veil/agents
+
+// Wrong — was getAgentsDir(cwd) which returned cwd/agents (missing .veil):
+const agentsDir = paths.getAgentsDir(cwd); // REMOVED — ambiguous name
+```
+
+Key functions in `paths.js`:
+- `getProjectConfigDir(cwd)` → `cwd/.veil/`
+- `getProjectAgentsDir(cwd)` → `cwd/.veil/agents/`
+- `getProjectSettingsPath(cwd)` → `cwd/.veil/settings.json`
+- `getGlobalAgentsDir()` → `~/.veil/agents/`
+- `getAgentDir(cwd, agentName)` → full path for a named agent
+
+**3. Settings field names (`src/settings/fields.js`):**
+```javascript
+// src/settings/fields.js — single source of truth
+module.exports = {
+  MODEL_BASE_URL: 'base_url',
+  MODEL_API_KEY: 'api_key',
+  MODEL_NAME: 'model',
+  // ... all other field names
+};
+// Never hardcode 'api_key' or 'base_url' as strings in any other file.
+```
+
+**4. `package.json` bin path:**
+```json
+{ "bin": { "veil": "./src/cli/index.js" } }
+```
+The root `index.js` does NOT exist. The CLI entry is always `src/cli/index.js`.
+
+**5. Version reading:**
+Do NOT use `require('../../package.json')` inside deeply-nested route files — the relative depth varies and causes `Cannot find module` errors. Instead, read version at startup and pass it via `context.js` or a `constants.js` file:
+```javascript
+// Set once at startup in src/cli/index.js:
+const { version } = require('../package.json'); // correct relative path from cli/
+context.setVersion(version);
+// In routes:
+const version = context.getVersion();
+```