@newsails/veil-cli 1.0.1

package/ai-tests/Research & Planning.md

@@ -0,0 +1,814 @@
+# VeilCLI Test Engine — Research & Planning Document
+*Prepared for handoff to local dev agent with full reasoning context*
+
+---
+
+## 📋 What This Document Is
+
+This document is the output of a full research session between the project owner and an AI research assistant. Its purpose is to give a **local dev agent full context** to implement the VeilCLI Test Engine — without needing to re-ask any questions already answered here.
+
+The dev agent's job: **read the VeilCLI codebase, then use this document to build the test engine.** Every section contains both the *what* and the *why/how we thought of it*.
+
+---
+
+## 🧠 Core Understanding — Read This First
+
+### What the test engine is
+A **coded test suite** that lives inside the VeilCLI repo as its own independent runnable package. It is:
+- Run via a single entry point with a CLI interface (`node test-engine.js` or similar)
+- Able to filter by group (`--group basic-api`, `--group flows`, etc.)
+- Self-contained: auto-spins a fresh isolated workspace + VeilCLI server, runs all tests, tears everything down
+- Tests are hardcoded scenarios written in code — you add new ones by coding them
+
+### Why it exists
+The project owner had basic scripts that returned PASS but real-world usage revealed broken behavior. The failures were **not** at the HTTP level — status codes were fine. The failures were in **agent runtime behavior**: tools appearing to succeed but producing wrong/empty results, agent loops breaking silently, memory not persisting correctly, inter-agent communication losing data in transit.
+
+### The fundamental verification philosophy
+> **If a script can verify it deterministically → use a script. Only use an AI judge when a script would produce a false positive.**
+
+Examples of when AI judge is needed vs. not:
+- ✅ Script: Did `POST /agents` return 201 with an `id` field? → Script
+- ✅ Script: Did the task events show `tool.start` for `read_file`? → Script
+- ✅ Script: Does `GET /sessions/:id/messages` return token counts per message? → Script
+- ⚠️ AI Judge: Agent A was supposed to call Agent B with a meaningful instruction — the tool fired (HTTP 200, taskId returned) but was the actual message content semantically valid or empty/garbled? → AI Judge
+- ⚠️ AI Judge: Agent was given a task requiring it to read a file and report its contents — did it actually report content from the file, or did it say "I cannot read files"? (We are not testing model hallucination — we are testing that VeilCLI's tool pipeline delivered the file content to the agent correctly) → AI Judge
+- ⚠️ AI Judge: Agent completed a multi-step agentic flow — did it actually complete a coherent start → tool loop → end cycle, or did it stall/give up halfway? → AI Judge
+
+### What is NOT being tested
+- LLM response quality / intelligence / accuracy (the model is assumed capable — using Kimi K2.5)
+- Model hallucination
+- External service reliability (OpenRouter, web search availability)
+
+### AI Judge setup
+- Separate external model — NOT the same VeilCLI runtime being tested (avoids circular testing)
+- Called via direct API (not through VeilCLI's `/completions`) 
+- Used sparingly — only where explicitly decided per test
+- Returns a structured verdict: PASS / FAIL + reasoning string
+
+---
+
+## 🏗️ Engine Architecture
+
+### Project location
+Inside the VeilCLI repo, as its own package:
+```
+VeilCLI/
+└── test-engine/
+    ├── package.json
+    ├── index.js              ← CLI entry point
+    ├── runner.js             ← test runner (serial within group, parallel across groups)
+    ├── workspace.js          ← workspace + server lifecycle manager
+    ├── client.js             ← HTTP test client (thin wrapper around fetch)
+    ├── assert.js             ← assertion helpers (standard + deep + AI judge)
+    ├── reporter.js           ← console output + failure artifact preservation
+    ├── fixtures/             ← reusable agent configs, memory seeds, tool files
+    └── tests/
+        ├── basic-api/
+        ├── flows/
+        ├── agent-behaviors/
+        ├── tool-coverage/
+        ├── debuggability/
+        ├── cli/
+        └── ai-judged/
+```
+
+### Workspace lifecycle
+```
+START RUN
+  → Create fresh temp folder: .veil-test-{timestamp}/
+  → Copy auth.json from project's .veil/auth.json
+  → Write settings.json (test-specific: low timeouts, test secret)
+  → Start VeilCLI server (child_process), wait for /health to respond
+  → Run all test groups
+  → On failure: preserve entire workspace + HTTP logs + AI judge transcripts
+  → On success: delete workspace
+  → Stop server
+END RUN
+```
+
+### Verification depth — per test, not global
+Each test explicitly defines what it checks:
+- **Level 1**: HTTP status + response shape
+- **Level 2**: Level 1 + specific field values
+- **Level 3**: Level 2 + task events (tool call sequence, parameters)
+- **Level 4**: Level 3 + file system / memory state / DB-visible state via API
+- **Level 5**: Level 4 + AI judge semantic verification
+
+### Console output philosophy
+Enough detail to understand what failed **without opening any files**. On failure, show:
+- Which test failed
+- What was expected vs. what was received
+- Which step in the flow failed (not just final assertion)
+- AI judge reasoning if applicable
+
+---
+
+## 📦 Fixtures
+
+**Reasoning**: Tests need agents, memory files, and custom tools to be pre-defined so scenarios are reproducible and readable. A base set is always present in every workspace run; test-specific additions are layered on top.
+
+The dev agent should look at `examples/` in the VeilCLI repo and the `agent.json` schema to understand what valid agent configs look like, then build fixture templates for:
+
+### Base fixtures (always installed)
+```
+fixtures/
+├── agents/
+│   ├── basic-chat/          ← minimal agent, chat mode only, no tools
+│   ├── task-runner/         ← task mode, has file + memory tools
+│   ├── memory-agent/        ← memory enabled, memory tools whitelisted
+│   ├── tool-tester/         ← all built-in tools whitelisted
+│   ├── restricted-agent/    ← specific tools explicitly denied
+│   ├── orchestrator/        ← can spawn subagents, agent_spawn + task_create
+│   └── worker/              ← subagent mode, spawned by orchestrator
+├── memory/
+│   ├── global-seed.md       ← pre-populated global memory for memory tests
+│   └── agent-seed.md        ← pre-populated agent memory
+├── tools/
+│   └── echo-tool.js         ← custom tool that just echoes input (for custom tool loading tests)
+└── files/
+    ├── sample.txt            ← readable test file for read_file tests
+    ├── sample-dir/           ← directory for list_dir / glob tests
+    └── grep-target.txt       ← file with known content for grep tests
+```
+
+**Dev agent instruction**: Look at the VeilCLI `schemas/` folder for the exact `agent.json` schema fields. Look at `tools/` for the structure of a built-in tool to understand how `echo-tool.js` should be shaped. Look at `examples/` for reference agent definitions.
+
+---
+
+## 🧪 Test Groups & Individual Tests
+
+For each test below, the reasoning explains:
+1. **Why this test exists** (what real failure it catches)
+2. **How to implement it** (what API calls, what to verify)
+3. **Verification level** (script / deep / AI judge)
+
+---
+
+### GROUP 1: Basic API
+*Reasoning: Before testing any behavior, validate that the HTTP surface is wired correctly. These tests have nothing to do with AI — they are pure contract tests. The previous basic scripts likely covered some of these, but may have missed field-level validation.*
+
+---
+
+**TEST 1.1 — Agent CRUD**
+- **Why**: Core API. If agent creation/read/update/delete is broken, nothing else works.
+- **How**:
+  1. `POST /agents` with a valid agent config → expect 201, response has `name` field matching input
+  2. `GET /agents` → expect array containing the created agent
+  3. `GET /agents/:name` → expect full config returned, verify key fields match what was sent
+  4. `PUT /agents/:name` → update a field (e.g. temperature), then GET again and verify it changed
+  5. `DELETE /agents/:name` → expect 200/204, then GET returns 404
+- **Verification**: Level 2 (HTTP + field values)
+- **Dev agent**: Check the exact response shape of each endpoint in `api/routes/` to know which fields to assert on
+
+---
+
+**TEST 1.2 — Settings CRUD**
+- **Why**: Settings affect the whole runtime. If read/write is broken, config changes don't take effect.
+- **How**:
+  1. `GET /settings` → verify response has expected top-level fields, API keys are redacted
+  2. `PUT /settings` with a safe field change (e.g. bump `maxIterations`) → expect 200
+  3. `GET /settings` again → verify the change is reflected
+  4. Test `?level=project` vs `?level=merged` → verify merged includes defaults
+- **Verification**: Level 2
+- **Dev agent**: Look at `utils/settings loader` to understand the merge layers and which fields are safe to mutate in tests
+
+---
+
+**TEST 1.3 — Health and Status**
+- **Why**: Basic liveness. Also `/status` returns counts that other tests can use to validate state changes.
+- **How**:
+  1. `GET /health` → 200, no DB access required (verify it's fast)
+  2. `GET /status` → verify fields: `uptime`, `cwd`, agent count, session count, task count
+  3. Create an agent, re-check `/status` → agent count increased
+- **Verification**: Level 2
+
+---
+
+**TEST 1.4 — Auth / Secret enforcement**
+- **Why**: If secret is set but not enforced, the security feature is broken silently.
+- **How**:
+  1. Start test server with `secret` set in settings
+  2. `GET /agents` without header → expect 401
+  3. `GET /agents` with wrong secret → expect 401
+  4. `GET /agents` with correct `X-Veil-Secret` → expect 200
+- **Verification**: Level 1
+
+---
+
+**TEST 1.5 — Session CRUD**
+- **Why**: Sessions are the backbone of chat continuity. Broken session management = broken chat history.
+- **How**:
+  1. `POST /sessions` pre-create → verify `id` returned
+  2. `GET /sessions` → list contains the session
+  3. `GET /sessions/:id` → verify fields: `agent`, `mode`, `status`, `message_count`
+  4. `POST /sessions/:id/reset` → verify messages cleared (message_count = 0 after)
+  5. `DELETE /sessions/:id` (soft) → verify status changed
+  6. `DELETE /sessions/:id?hard=true` → verify permanent removal
+- **Verification**: Level 2-3
+- **Dev agent**: Check the session model in `infrastructure/` SQLite schema to know exact field names
+
+---
+
+**TEST 1.6 — Models endpoint**
+- **Why**: `/models` is used internally for context size limits and cost calculation. If broken, cost tracking fails silently.
+- **How**:
+  1. `GET /models` → verify `updated_at` field exists, `models` is non-empty array
+  2. Each model has: `id`, `name`, `context_length`, `pricing`
+  3. `GET /models/:provider/:name` → look up a known model, verify fields
+- **Verification**: Level 2
+
+---
+
+### GROUP 2: Basic Flows
+*Reasoning: The project owner's primary failure — scripts passed but real usage broke. These tests simulate what a real new user would do. They are end-to-end flows, not unit tests. They catch integration failures that unit-level API tests miss.*
+
+---
+
+**TEST 2.1 — New user happy path (chat)**
+- **Why**: This is THE most critical regression test. A new user creates one agent and chats. If this breaks, VeilCLI is unusable.
+- **How**:
+  1. Use `basic-chat` fixture agent (already installed in workspace)
+  2. `POST /agents/:name/chat` with a simple message → expect 200, `message` field non-empty, `sessionId` returned
+  3. Send a follow-up using the returned `sessionId` → verify the session continues (message_count increases)
+  4. `GET /sessions/:id/messages` → verify both turns exist with correct roles
+  5. Verify `input_tokens` and `output_tokens` are non-zero on messages
+- **Verification**: Level 3
+- **Dev agent**: Check `api/routes/` for the exact chat response shape. Check how sessionId is passed in subsequent requests.
+
+---
+
+**TEST 2.2 — Async task full lifecycle**
+- **Why**: Task mode is async — the lifecycle (pending → processing → finished) must transition correctly. Previous scripts likely just checked 202 and didn't poll.
+- **How**:
+  1. `POST /agents/task-runner/task` with a simple input → expect 202, `taskId` returned, status `pending`
+  2. Poll `GET /tasks/:id` until status is `finished` or `failed` (with timeout — e.g. 60s)
+  3. Verify final status is `finished`, `output` is non-empty
+  4. `GET /tasks/:id/events` → verify events exist, at minimum one `status.change` event
+  5. Verify `token_input` and `token_output` are non-zero on the task record
+- **Verification**: Level 3
+- **Dev agent**: Check the task polling logic — what's a safe poll interval? Check event types in `api/routes/tasks`.
+
+---
+
+**TEST 2.3 — Chat with SSE streaming**
+- **Why**: SSE mode is a separate code path. If broken, streaming clients (UI) get no response while non-streaming works fine.
+- **How**:
+  1. `POST /agents/:name/chat` with `{ sse: true }` → expect `text/event-stream` content type
+  2. Collect all SSE events until `done` event received
+  3. Verify `chunk` events were received (content streamed)
+  4. Verify `done` event contains final message and `tokenUsage`
+- **Verification**: Level 2-3
+- **Dev agent**: Check how SSE is implemented in the chat route — what events are emitted and in what format.
+
+---
+
+**TEST 2.4 — Session resumption**
+- **Why**: A user closes their client and comes back later. The session must be resumable with history intact.
+- **How**:
+  1. Chat with agent, get `sessionId`
+  2. Send second message with same `sessionId` — verify context is maintained (agent references earlier turn)
+  3. Retrieve `GET /sessions/:id/messages` — verify full history is there
+  4. `POST /sessions/:id/reset` — verify messages cleared but session still exists
+  5. Chat again on same session — verify it works fresh
+- **Verification**: Level 3 (message history) + optionally Level 5 (AI judge: does agent actually reference the earlier message?)
+- **AI Judge note**: Use AI judge only to confirm context continuity — did the agent's response acknowledge the prior conversation? This is hard to script without semantic understanding.
+
+---
+
+### GROUP 3: Agent Behaviors
+*Reasoning: These tests validate that VeilCLI correctly enforces agent configuration — permissions, tool access, mode restrictions. These were identified as a major source of silent failures.*
+
+---
+
+**TEST 3.1 — Tool permission enforcement (deny list)**
+- **Why**: If `disallowedTools` doesn't actually block tool usage, security and behavior boundaries are broken. A script check on HTTP alone won't catch this — the agent might try the tool and get an internal error that still returns HTTP 200.
+- **How**:
+  1. Use `restricted-agent` fixture (has `bash` in `disallowedTools`)
+  2. Give agent a task that explicitly requires using `bash` ("run the command `echo hello`")
+  3. Poll task to completion
+  4. `GET /tasks/:id/events` → verify NO `tool.start` event for `bash` exists
+  5. Verify task either finished (agent worked around it) or failed with appropriate error — NOT silently ran bash
+- **Verification**: Level 3 (event log inspection)
+- **Dev agent**: Check how `disallowedTools` is enforced in `core/agentic loop`. Understand what event is emitted when a tool is blocked.
+
+---
+
+**TEST 3.2 — Tool whitelist enforcement (allow list)**
+- **Why**: Mirror of above. If `tools` whitelist doesn't restrict, agent can use any tool.
+- **How**:
+  1. Create agent with only `read_file` in its tools whitelist
+  2. Give it a task requiring `write_file` 
+  3. Verify `write_file` never appears in task events
+- **Verification**: Level 3
+
+---
+
+**TEST 3.3 — Mode enforcement**
+- **Why**: An agent with `chat.enabled: false` should not be chattable. If mode enforcement is broken, wrong code paths execute.
+- **How**:
+  1. Create agent with `modes.chat.enabled: false`
+  2. `POST /agents/:name/chat` → expect appropriate error response (not 200)
+  3. Dev agent: check what error code/status VeilCLI returns for disabled modes
+- **Verification**: Level 1
+
+---
+
+**TEST 3.4 — maxIterations enforcement**
+- **Why**: Without iteration limits, a broken agentic loop runs forever and burns tokens/cost.
+- **How**:
+  1. Create task with `maxIterations: 2`
+  2. Give agent a task that would normally require many tool calls
+  3. Verify task stops after 2 iterations — check `iterations` field on task record
+  4. Verify `onExhausted: "fail"` → task status is `failed`; `onExhausted: "wait"` → task status is `waiting`
+- **Verification**: Level 3
+
+---
+
+**TEST 3.5 — Agent reload from disk**
+- **Why**: `POST /agents/:name/reload` must actually refresh config. If it doesn't, live config changes never take effect.
+- **How**:
+  1. Create agent, read its config
+  2. Directly modify the `agent.json` file on disk (change temperature)
+  3. `POST /agents/:name/reload`
+  4. `GET /agents/:name` → verify the changed field is now reflected
+- **Verification**: Level 2
+
+---
+
+### GROUP 4: Tool Coverage
+*Reasoning: There are 24 built-in tools. The previous scripts didn't test tool execution — they tested API endpoints. A tool can return HTTP 200 from the API but fail silently inside the agent loop. Each tool test gives an agent a task that REQUIRES that tool, then verifies via task events that the tool was called AND produced a meaningful result.*
+
+*Pattern for each tool test:*
+1. *Create a task that cannot be completed without using the specific tool*
+2. *Poll to completion*
+3. *Check `GET /tasks/:id/events` for `tool.start` + `tool.end` for that tool*
+4. *Check the `tool.end` event result is not an error and has meaningful content*
+5. *For some tools, verify side effects (file created, memory updated, etc.)*
+
+---
+
+**TEST 4.1 — File I/O tools**
+
+`read_file`:
+- Setup: Write a file to workspace with known content (e.g. `"TESTMARKER_XYZ"`)
+- Task: "Read the file at [path] and tell me what's in it"
+- Verify: `tool.end` result contains the file content, NOT an error string
+- AI Judge: Was the agent able to report the file content? (catches case where tool pipeline broke and agent said "I cannot read files")
+
+`write_file`:
+- Task: "Write the text 'hello test' to a file called output.txt in [dir]"
+- Verify: `tool.end` shows success, then actually check the file exists on disk with correct content
+
+`edit_file`:
+- Setup: Write a file with known content
+- Task: "Edit the file at [path], replace 'OLD_TEXT' with 'NEW_TEXT'"
+- Verify: File on disk now contains 'NEW_TEXT', not 'OLD_TEXT'
+
+`list_dir`:
+- Setup: Create a directory with 3 known files
+- Task: "List the files in [dir] and report the filenames"
+- Verify: `tool.end` result contains the expected filenames
+
+`glob`:
+- Setup: Create files with `.txt` and `.js` extensions in a dir
+- Task: "Find all `.txt` files in [dir]"
+- Verify: `tool.end` result matches only `.txt` files
+
+`grep`:
+- Setup: Create a file with known unique string
+- Task: "Search for the pattern 'UNIQUE_GREP_MARKER' in [dir]"
+- Verify: `tool.end` result contains the match with file + line reference
+
+`bash`:
+- Task: "Run the command `echo BASH_MARKER_TEST` and report the output"
+- Verify: `tool.end` result contains `BASH_MARKER_TEST`
+- This is a good deterministic test — the echo output is predictable
+
+---
+
+**TEST 4.2 — Memory tools**
+
+`memory_write` + `memory_read` (same test, sequential):
+- Task 1: "Write a note to your memory: 'MEMORY_MARKER_12345'"
+- Verify: `memory_write` appears in events, `tool.end` success
+- Also verify: `GET /agents/:name/memory/MEMORY.md` via API → file contains the written text
+- Task 2 (new session, same agent): "Read your memory and tell me what notes you have"
+- Verify: `memory_read` in events, AND agent's response contains `MEMORY_MARKER_12345`
+- AI Judge: Did the agent actually retrieve and report from memory, or did it say it has no memory? (Tests persistence across sessions)
+
+`memory_search`:
+- Setup: Pre-seed agent memory with several distinct entries
+- Task: "Search your memory for [specific topic]"
+- Verify: `memory_search` in events, result is relevant entry (not empty)
+
+---
+
+**TEST 4.3 — Todo tools**
+
+`todo_write` + `todo_read`:
+- Task: "Plan the following 3 tasks as todos: [list]" (agent should use todo_write naturally)
+- Verify: `todo_write` appears in events with structured `todos` array in parameters
+- Then: "What are your current todos?" (agent should use todo_read)
+- Verify: `todo_read` in events, response reflects the todos written
+- *Note: todo tools are scoped to the current task — verify this scoping works correctly*
+
+---
+
+**TEST 4.4 — Web tools**
+
+`web_search`:
+- Task: "Search the web for 'VeilCLI test engine'"
+- Verify: `web_search` in events, `tool.end` result is non-empty (not error)
+- *Note: This test may be flaky if DuckDuckGo is rate-limiting — mark it as non-blocking*
+
+`web_fetch`:
+- Task: "Fetch the content of https://example.com and tell me the page title"
+- Verify: `web_fetch` in events, result contains HTML-stripped text, not an error
+
+---
+
+**TEST 4.5 — Utility tools**
+
+`sleep`:
+- Task: "Wait 3 seconds then say done"
+- Verify: `sleep` in events with `seconds: 3`, task took at least 3 seconds
+
+`log_write`:
+- Task: "Write a log entry saying 'TEST_LOG_MARKER'"
+- Verify: `log_write` in events, then `GET /tasks/:id/events` → a `log` event contains `TEST_LOG_MARKER`
+
+`tool_search`:
+- Task: "Search your available tools for memory-related tools"
+- Verify: `tool_search` in events, result is non-empty
+
+---
+
+**TEST 4.6 — Multi-agent tools** *(also covered in Group 5, but event-level here)*
+
+`task_create`:
+- Task: "Create a new task for the worker agent with input 'say hello'"
+- Verify: `task_create` in events, `tool.end` contains a valid `taskId`
+- Then: poll that taskId via API → verify it actually exists and runs
+
+`task_status`:
+- Setup: Create a task programmatically via API
+- Task: "Check the status of task [id]"
+- Verify: `task_status` in events, result contains a valid status string
+
+`task_respond`:
+- Setup: Create a task with `onExhausted: "wait"`, let it reach waiting state
+- Task: Tell an agent "respond to task [id] with 'continue'"
+- Verify: `task_respond` in events, target task resumes
+
+`agent_message`:
+- Task: "Send a message to the worker agent asking it to say hello, wait for the reply"
+- Verify: `agent_message` in events, `tool.end` result is non-empty reply
+- AI Judge: Was the reply from the worker agent meaningful (not empty/error)?
+
+`agent_send`:
+- Task: "Send a fire-and-forget message to the worker agent"
+- Verify: `agent_send` in events, `tool.end` success (not expecting a reply)
+
+`agent_spawn (wait=true)`:
+- Task: "Spawn the worker agent to [do something], wait for its result, then report it"
+- Verify: `agent_spawn` in events with `wait: true`, result is non-empty, task has `parent_task_id` set
+- AI Judge: Did the orchestrator's final response actually incorporate the worker's output?
+
+`agent_spawn (wait=false)`:
+- Task: "Spawn 2 worker agents in parallel to [do something], collect their taskIds"  
+- Verify: Two `agent_spawn` events with `wait: false`, two separate taskIds returned
+
+`task_subscribe`:
+- Task: Create a task, have agent subscribe to it
+- Verify: `task_subscribe` in events, subscription exists in DB (check via API behavior after task completes)
+
+---
+
+**TEST 4.7 — Custom tool loading**
+- **Why**: Custom tools in `.veil/agents/<name>/tools/` must be auto-discovered and available.
+- **How**:
+  1. Install `echo-tool.js` fixture into the test agent's tools folder
+  2. `GET /agents/:name/skills` → verify custom tool appears in the list
+  3. Give agent a task that calls the echo tool
+  4. Verify `tool.start` + `tool.end` in events for the custom tool name
+- **Dev agent**: Look at how custom tools are loaded in `core/agent loader` and what the `schema + execute` export shape must be
+
+---
+
+### GROUP 5: Multi-Agent Flows
+*Reasoning: Multi-agent communication is where the most subtle bugs hide. HTTP 200 is meaningless here — the tool can "succeed" but pass empty, truncated, or malformed data between agents. This is the primary group requiring AI judge.*
+
+---
+
+**TEST 5.1 — Orchestrator spawns worker (sync, wait=true)**
+- **Why**: The full sync delegation pattern. If `agent_spawn(wait=true)` passes an empty instruction or loses the result, the orchestrator gets nothing but no error is raised.
+- **How**:
+  1. Give orchestrator a task: "Spawn the worker agent and ask it to [specific task], then report exactly what it said"
+  2. Poll orchestrator task to completion
+  3. Check events: `agent_spawn` was called with `wait: true` and a non-empty `instruction`
+  4. Check worker task was created with a `parent_task_id` matching orchestrator task
+  5. Check orchestrator's output references the worker's result
+- **Verification**: Level 3 + AI Judge
+- **AI Judge**: "Given this orchestrator output and this worker output, did the orchestrator correctly incorporate the worker's response?"
+
+---
+
+**TEST 5.2 — Parallel fan-out (wait=false)**
+- **Why**: Parallel spawning is a complex pattern. Each `agent_spawn(wait=false)` must return a distinct `taskId` immediately, and both child tasks must actually run.
+- **How**:
+  1. Give orchestrator: "Spawn 2 worker agents in parallel with different tasks, collect their results"
+  2. Verify: Two distinct `agent_spawn` calls in events, each with `wait: false`
+  3. Verify: Two child tasks exist in DB with `parent_task_id` set
+  4. Verify: Both child tasks eventually reach `finished`
+  5. Verify: Orchestrator output references both results
+- **Verification**: Level 3 + AI Judge
+
+---
+
+**TEST 5.3 — Agent messaging (agent_message sync)**
+- **Why**: `agent_message` is synchronous — the caller blocks. If the target agent's response is lost or the call hangs, the calling agent stalls indefinitely.
+- **How**:
+  1. Give orchestrator: "Message the worker agent and ask it what 2+2 is, report the answer"
+  2. Verify: `agent_message` in events, `tool.end` has non-empty result
+  3. Verify: Orchestrator output contains the answer
+- **AI Judge**: Was the answer from the worker agent passed through correctly?
+
+---
+
+**TEST 5.4 — Durable task subscription**
+- **Why**: `task_subscribe` writes to SQLite and survives server restart. If this breaks, agents lose notification of subtask completion.
+- **How**:
+  1. Create a long-running task (use sleep tool)
+  2. Have subscriber agent call `task_subscribe` on it
+  3. Verify subscription exists (via runtime behavior — when task completes, subscriber gets notified)
+  4. Wait for target task to complete
+  5. Verify subscriber agent received notification (check its session events)
+- **Dev agent**: Check `task_subscriptions` table schema and how notifications are injected into subscriber sessions
+
+---
+
+**TEST 5.5 — maxSubAgentDepth enforcement**
+- **Why**: Without depth limits, a rogue agent can spawn infinitely deep chains.
+- **How**:
+  1. Set `maxSubAgentDepth: 2` in test settings
+  2. Create orchestrator that spawns a worker that spawns another worker
+  3. Verify: Third level spawn is rejected with appropriate error
+- **Dev agent**: Check what error/event is emitted when depth limit is exceeded in `core/`
+
+---
+
+### GROUP 6: Debuggability
+*Reasoning: The project owner specifically called this out. VeilCLI claims to provide full observability — token counts, cost, tool traces, conversation history. If these are missing or wrong, developers can't debug agent behavior in production. These tests verify the instrumentation works, not the agent behavior.*
+
+---
+
+**TEST 6.1 — Token tracking per message**
+- **Why**: Token counts must exist on every message for cost accountability.
+- **How**:
+  1. Run a chat conversation (2-3 turns)
+  2. `GET /sessions/:id/messages` → every assistant message has `output_tokens > 0`, every user message has `input_tokens > 0`
+  3. Verify no message has `null` or `0` tokens (unless it's a system message — dev agent: check if system messages are in the message list)
+- **Verification**: Level 2
+
+---
+
+**TEST 6.2 — Token tracking per session**
+- **Why**: Session-level totals must roll up correctly from message-level.
+- **How**:
+  1. After conversation above, `GET /sessions/:id`
+  2. Verify `token_input` and `token_output` are non-zero
+  3. Verify they are >= sum of individual message tokens (system prompt adds overhead)
+- **Verification**: Level 2
+
+---
+
+**TEST 6.3 — Cost calculation**
+- **Why**: Cost must be calculated even when the LLM provider doesn't return it natively (fallback to pricing model).
+- **How**:
+  1. After a conversation, `GET /sessions/:id` → verify `cost` is a non-zero number
+  2. `GET /sessions/:id/messages` → verify individual message `cost` fields exist
+  3. The session cost should approximately equal sum of message costs
+- **Verification**: Level 2
+
+---
+
+**TEST 6.4 — Task event trace completeness**
+- **Why**: Task events must capture the full agentic loop for debugging. If events are missing, you can't trace what happened.
+- **How**:
+  1. Run a task that uses at least 2 different tools
+  2. `GET /tasks/:id/events` → verify:
+     - At least one `status.change` event (pending → processing)
+     - One `tool.start` + `tool.end` pair per tool used
+     - Final `status.change` to `finished`
+  3. Verify `tool.start` events contain tool name and parameters
+  4. Verify `tool.end` events contain result (not empty)
+- **Verification**: Level 3
+
+---
+
+**TEST 6.5 — Context snapshot**
+- **Why**: `GET /tasks/:id/context` must return a valid LLM context snapshot for debugging mid-task.
+- **How**:
+  1. Run a task
+  2. `GET /tasks/:id/context` → verify `messages`, `tools`, `iteration` fields exist
+  3. `messages` array is non-empty, contains at least system + user turn
+  4. `tools` is non-empty array (tool schemas)
+- **Verification**: Level 2
+
+---
+
+**TEST 6.6 — Session message history**
+- **Why**: `GET /sessions/:id/messages` is the primary debugging tool for chat. Pagination must work.
+- **How**:
+  1. Create a chat session with 10+ turns
+  2. `GET /sessions/:id/messages?limit=3&offset=0` → verify exactly 3 messages returned
+  3. `GET /sessions/:id/messages?limit=3&offset=3` → verify next 3
+  4. Verify `role` field is either `user`, `assistant`, or `system` on each
+- **Verification**: Level 2
+
+---
+
+### GROUP 7: CLI Tool
+*Reasoning: VeilCLI has a CLI entry point (start, stop, status, agents, login). The CLI is a different code path than the REST API. If the CLI is broken, new users can't even start the server. These tests invoke the actual CLI binary via shell.*
+
+---
+
+**TEST 7.1 — Server start/stop via CLI**
+- **Why**: `veil start` and `veil stop` are the primary user entry points.
+- **How**:
+  1. Stop the test server (temporarily)
+  2. Run `veil start` in test workspace → wait for it to be ready (`GET /health` responds)
+  3. Run `veil status` → verify output contains running server info
+  4. Run `veil stop` → server goes down (`GET /health` fails)
+- **Dev agent**: Check `cli/` folder for exact command names and expected output format
+
+---
+
+**TEST 7.2 — Agent listing via CLI**
+- **How**:
+  1. With server running, run `veil agents` (or equivalent)
+  2. Verify output contains the test agents created in workspace
+- **Dev agent**: Check CLI command name and output format in `cli/`
+
+---
+
+**TEST 7.3 — CLI chat flow**
+- **Why**: A user should be able to start a chat from the CLI, get a response, and resume it.
+- **How**:
+  1. Run `veil chat <agent-name>` with piped input (or appropriate flag for non-interactive mode)
+  2. Verify response is printed to stdout
+  3. Note the session ID from output
+  4. Re-run with session flag to resume — verify agent acknowledges context
+- **Dev agent**: Check the `cli/` start command and whether there's a chat subcommand or how the CLI initiates chat. Look for non-interactive mode flags for testability.
+
+---
+
+### GROUP 8: Daemon Mode
+*Reasoning: Daemon mode runs on cron schedules. Tests can't wait for actual cron — use `POST /agents/:name/daemon/trigger` to fire immediately.*
+
+---
+
+**TEST 8.1 — Daemon start/stop/trigger**
+- **How**:
+  1. Create daemon agent with a schedule
+  2. `POST /agents/:name/daemon/start` → verify it appears in `GET /daemons`
+  3. `POST /agents/:name/daemon/trigger` → verify it fires (a new task is created)
+  4. Poll that task to completion
+  5. `POST /agents/:name/daemon/stop` → verify removed from `GET /daemons`
+- **Verification**: Level 3
+
+---
+
+**TEST 8.2 — Daemon conflict policy (skip)**
+- **How**:
+  1. Start daemon, trigger it
+  2. While first tick is still running, trigger again
+  3. With `conflictPolicy: "skip"` → second tick should be skipped (no second task created)
+- **Dev agent**: Check how conflict policy is implemented in `infrastructure/cron scheduler`
+
+---
+
+**TEST 8.3 — Daemon reads heartbeat file**
+- **Why**: Daemon agents read from `.veil/heartbeats/<name>.md` each tick for instructions. If this is broken, daemon behavior can't be configured at runtime.
+- **How**:
+  1. Write a specific instruction to the heartbeat file
+  2. Trigger daemon
+  3. AI Judge: Did the daemon's task output reflect the heartbeat instruction?
+- **Verification**: Level 3 + AI Judge
+
+---
+
+### GROUP 9: Memory Persistence
+*Reasoning: Memory is a markdown file on disk, injected into system prompts at session start. The key thing to test: does the agent's memory actually persist across sessions and influence behavior?*
+
+---
+
+**TEST 9.1 — Memory write + API read**
+- **How**:
+  1. Agent writes to memory via `memory_write` tool in a task
+  2. `GET /agents/:name/memory/MEMORY.md` → verify the written content appears
+  3. `PUT /agents/:name/memory/:file` via API → write directly
+  4. `GET /agents/:name/memory/:file` → verify roundtrip
+- **Verification**: Level 2-3
+
+---
+
+**TEST 9.2 — Memory influences next session**
+- **Why**: Memory is injected at session start. If injection is broken, agents don't have memory even if the file exists.
+- **How**:
+  1. Pre-seed agent memory with `"MEMORY_SEED_MARKER_ABC"`
+  2. Start a new chat session
+  3. Ask agent: "What do you remember about yourself?"
+  4. Verify agent mentions the marker
+- **AI Judge**: Did the agent's response demonstrate it had access to the memory content? (This is the canonical AI judge use case — a script can't verify semantic incorporation)
+- **Verification**: Level 5 (AI Judge)
+
+---
+
+**TEST 9.3 — Global vs agent memory scoping**
+- **How**:
+  1. Write to global memory (`PUT /memory/MEMORY.md`)
+  2. Write to agent-specific memory (`PUT /agents/:name/memory/MEMORY.md`)
+  3. Start chat with agent that has memory enabled
+  4. Verify both memories are injected (ask agent about both markers)
+- **AI Judge**: Did the agent demonstrate awareness of both global and agent-specific memory?
+
+---
+
+## 🔧 Implementation Notes for Dev Agent
+
+### How to detect tool usage in events
+Every agentic tool call produces a `tool.start` and `tool.end` event in `GET /tasks/:id/events`. The `tool.start` event contains the tool name and parameters. The `tool.end` event contains the result. **This is the primary verification mechanism for tool tests** — don't try to infer tool usage from the agent's text response.
+
+### Polling pattern for async tasks
+```javascript
+async function pollTask(taskId, timeoutMs = 60000) {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    const task = await client.get(`/tasks/${taskId}`);
+    if (['finished', 'failed', 'canceled'].includes(task.status)) return task;
+    await sleep(2000);
+  }
+  throw new Error(`Task ${taskId} timed out`);
+}
+```
+
+### AI Judge invocation pattern
+```javascript
+// Only use when a script cannot verify semantically
+async function aiJudge(context, criteria) {
+  // Call external model API directly (not VeilCLI /completions)
+  // Prompt: "Given this [context], did [criteria]? Reply TEST_PASS or TEST_FAIL followed by one sentence reasoning."
+  // Parse response: starts with TEST_PASS → pass, TEST_FAIL → fail
+  // Log full prompt + response to failure artifacts always
+}
+```
+
+### On failure: preserve everything
+When any assertion fails:
+- Keep the entire test workspace folder (all agent files, DB, memory)
+- Dump all HTTP requests/responses made during that test
+- Dump AI judge prompts and responses
+- Print workspace path to console so developer can inspect directly
+
+### Test agent design
+- `basic-chat`: chat mode only, no tools, memory disabled — for pure API tests
+- `task-runner`: task mode, tools: `[read_file, write_file, bash, list_dir, glob, grep, sleep, log_write, todo_write, todo_read, tool_search]`
+- `memory-agent`: memory enabled, tools: `[memory_read, memory_write, memory_search]`
+- `orchestrator`: tools: `[agent_spawn, agent_message, agent_send, task_create, task_status, task_respond, task_subscribe]`, allowedAgents: `[worker]`
+- `worker`: subagent mode, tools: `[read_file, write_file, bash, memory_read, memory_write]`
+- `restricted-agent`: all tools denied except `read_file`
+- `daemon-agent`: daemon mode with a schedule, reads heartbeat
+
+### Dev agent: things to check in the codebase before implementing
+1. `api/routes/` — exact request/response shapes for every endpoint (don't guess field names)
+2. `core/agentic loop` — how iterations work, how tools are called, how events are emitted
+3. `infrastructure/` — SQLite schema (field names on tasks, sessions, messages, events)
+4. `cli/` — exact command names and whether non-interactive mode exists for chat
+5. `schemas/agent.json` — exact valid fields for agent configs
+6. `tools/` — each tool's schema to understand what parameters and return values look like
+7. `system-prompts/` — what's injected automatically so you don't duplicate in test agents
+
+---
+
+## ✅ Summary of All Decisions Made in Research Session
+
+| Decision | Value |
+|---|---|
+| Engine type | Test suite + validation runtime |
+| Location | Inside VeilCLI repo, own package |
+| Entry point | CLI (`node test-engine.js`, group filters) |
+| Server lifecycle | Auto-start/stop per run |
+| Workspace | Fresh per run, preserved on failure |
+| Parallelism | Parallel across groups, serial within group |
+| AI Judge model | External model, NOT VeilCLI `/completions` |
+| AI Judge usage | Only when scripts produce false positives |
+| AI Judge trigger | Manually decided per test |
+| Fixture strategy | Reusable base set + test-specific additions |
+| Test authorship | Hardcoded in engine code, add by coding |
+| Verification depth | Per-test, explicitly defined |
+| LLM calls | Real (no mocking), tolerate variability |
+| Reproducibility | 3 identical runs should all pass |
+| Output | Console: enough detail to understand failure without opening files |
+| Failure artifacts | Workspace + HTTP logs + AI judge transcripts |
+| CLI tests | Via shell exec of actual `veil` binary |