savant-code 0.0.16 → 0.0.19

package/dev/session-summaries/2026-08-03-release-readiness-audit-closeout.md

@@ -0,0 +1,60 @@
+# Session Summary — 2026-08-03 Release-Readiness Audit (FID-2026-0803-012)
+
+**FID:** FID-2026-0803-012 (release-readiness audit: bloat trim, doc alignment, eval run + tracking)
+**Status:** Verified + archived
+**Signed:** Savant
+
+## What was done
+
+### Bloat trim
+
+- **CRITICAL:** Restored the 12 benchmark eval fixture JSONs
+  (`eval-{codebuff,manifold,plane,saleor}[-hard|-2].json`) — deleted in the working
+  tree but still tracked in git; the benchmark runner could not start without them.
+- Removed six gitignored `.test-reports-md-*` temp dirs under `evals/v2/tests/`.
+- Root `LEARNINGS.md` **kept** per operator decision (agent learnings library).
+
+### Eval run + harness fixes (found by actually running it)
+
+- **Baseline: 4/4 PASS** (offline, golden-patch verification) — incl. the previously
+  missing `savant-v2-pure-add-001` after its stale golden patch was regenerated.
+- **Evaluate: proven end-to-end, 0/4** — the run now populates the agent registry,
+  issues real model calls, executes tools, and writes reports. Failures are
+  environmental: free-tier provider 429 rate-limiting + BYOK key rejection +
+  `injectFault` MVP no-op (documented, deferred).
+- Three harness bugs fixed: (1) empty SDK agent registry (no `agentDefinitions` →
+  `Invalid agent ID`) wired via `loadLocalAgents`; (2) `writeJsonReport` crashed on
+  cyclic provider errors → circular-safe replacer; (3) `add-fix` golden-patch
+  pre-image mismatch.
+- Entrypoint fix: `main.ts`/`main-single-eval.ts` referenced the never-existing
+  `eval-savant-code.json` → retargeted to `eval-codebuff.json`.
+- Results tracked in `docs/reports/savant-code-benchmark-v2-2026-08-03.md` + raw
+  artifacts in `evals/v2/reports/`.
+
+### Doc alignment
+
+- `README.zh-CN.md` fully regenerated — complete Chinese translation of the current
+  v0.0.16 README (12-section parity; old file was stale pre-rebrand).
+
+## Gates
+
+- evals typecheck exit 0 · v2 suite **69 pass / 0 fail** (2 new regression tests)
+- Baseline 4/4 PASS · ESLint `--max-warnings 0` on changed files · `lint:md` exit 0
+- Fixtures verified (12 restored, zero `D`) · forbidden-name sweep clean
+- Independent code-reviewer AUDIT: clean — 2 regression tests added in response
+
+## Deferred
+
+- Evaluate mode needs a valid Savant backend key for a meaningful pass.
+- `injectFault` MVP no-op — error_recovery task cannot pass until wired.
+
+## Final readiness follow-up
+
+- Updated six current-facing `0.0.15` references to `0.0.16`; historical release records remain unchanged.
+- Corrected the Savant-Free smoke harness to select `savant-free.exe` on Windows.
+  The post-build smoke run executed **4 pass / 0 fail**; 2 tmux title-screen tests
+  were skipped because tmux is unavailable.
+- Final `bun run ci` build gate: exit 0. Full Markdownlint: exit 0.
+  `git diff --check`: exit 0.
+- The two untracked research documents with spaces in their names are intentional
+  research artifacts, not malformed or temporary filenames.

package/dev/test-prompts/archive/0.0.2-final-pass.md

@@ -0,0 +1,209 @@
+# Savant-Code 0.0.2 — Pre-Push Final Pass
+
+**Purpose:** Ship gate. ALL items must pass before bumping VERSION 0.0.1 → 0.0.2 and pushing the pre-rebrand milestone.
+
+**Mode:** Interactive live execution. Drive the actual chat session. Run real commands on the committed tree (no scratch artifacts).
+
+**Environment:** Strict mode on, language typescript, ECHO v0.2.0, bun ≥ 1.1.x, node ≥ 22.
+
+**Skip rules:**
+- `[LIVE]` = drive in tmux (interactive CLI)
+- `[SOURCE]` = read file + grep for evidence
+- `[BASH]` = run via `bun test` / `bun run typecheck`
+- SKIP only with explicit `BLOCKER:` reason + logged in `dev/scratchpad/0.0.2-blockers.md`
+
+**Reconciliation note (post-FID-017, 2026-07-19):** Several Items below were updated from the original 0.0.2-final-pass to reflect the actual current code state — see specific reconciliation notes per Item.
+
+---
+
+## Section 1: Pre-Flight
+
+1. `[BASH]` `bun --version` → must print ≥ 1.1.x
+2. `[BASH]` `node --version` → v22.11.x expected
+3. `[BASH]` `git status` → clean working tree (no uncommitted changes outside dev/)
+4. `[SOURCE]` `VERSION` → currently 0.0.2 (post-FID-017 bump)
+5. `[SOURCE]` `package.json` → version 0.0.2 (matches VERSION, post-FID-017 bump)
+
+---
+
+## Section 2: Typecheck × 4 (Hard Gate)
+
+6. `[BASH]` `cd common && bun run typecheck` → exit 0
+7. `[BASH]` `cd sdk && bun run typecheck` → exit 0
+8. `[BASH]` `cd packages/agent-runtime && bun run typecheck` → exit 0
+9. `[BASH]` `cd cli && bun run typecheck` → exit 0
+10. `[SOURCE]` `grep -rn 'ts-ignore\|@ts-expect-error' --include='*.ts' .` → expect ≤ 41 hits (all in pre-existing string prompts — justified, not real directives)
+
+---
+
+## Section 3: SDK Test Suite (Hard Gate)
+
+11. `[BASH]` `cd sdk && bun test src/` → **415 pass / 0 fail** (post-FID-016; FID-016 fixed 22 pre-existing failures leaving 415 stable baseline in `src/`)
+12. `[BASH]` `cd sdk && bun test src/__tests__/user-knowledge-files.test.ts` → 15/15
+13. `[BASH]` `cd sdk && bun test src/__tests__/code-search.test.ts -t 'cwd'` → 3/3
+14. `[BASH]` `cd sdk && bun test src/__tests__/database.test.ts` → 3/3
+15. `[BASH]` `cd sdk && bun test src/__tests__/initial-session-state.test.ts -t 'discovers'` → 1/1
+16. `[BASH]` `cd sdk && bun test src/__tests__/load-agents.test.ts -t 'verbose'` → 1/1
+17. `[BASH]` `cd sdk && bun test src/__tests__/load-skills.test.ts -t 'malformed'` → 1/1
+18. `[BASH]` `cd sdk && bun test e2e/custom-agents/apply-patch-tool.e2e.test.ts` → SKIP (no API key per FID-016 Fix G — was silently running in mock mode before)
+
+---
+
+## Section 4: Boot Sequence Smoke
+
+19. `[LIVE]` Drive CLI, watch startup → confirm no crash, prompt appears in ≤ 5s
+20. `[LIVE]` Confirm "One Mind. A Thousand Faces." tagline visible in right sidebar
+21. `[SOURCE]` Confirm `ECHO.md` is READ 0-end by orchestrator (no partial read)
+22. `[SOURCE]` Confirm `protocol.config.yaml` fields parsed: `strict_mode: true`, `language: typescript`
+23. `[SOURCE]` Confirm session summary auto-created at `dev/session-summaries/<YYYY-MM-DD>-*.md`
+
+---
+
+## Section 5: Slash Commands
+
+24. `[LIVE]` Drive `/phase` → print current (idle expected)
+25. `[LIVE]` Drive `/phase red` → succeeds (idle → red legal)
+26. `[LIVE]` Drive `/phase green` → BLOCKED if no open FID, succeeds if FID exists
+27. `[LIVE]` Drive `/fids` → list open FIDs
+28. `[LIVE]` Drive `/dev` → toggle dev mode
+29. `[LIVE]` Drive `/exit` → clean shutdown, summary saved
+
+---
+
+## Section 6: FSM Transition Cycle
+
+30. `[LIVE]` With open FID-2026-0718-X present: idle → red → green → audit → complete arc
+31. `[LIVE]` Confirm right sidebar `[phase]` indicator + `[work]` activity indicator (per FID-009) update after every transition
+32. `[LIVE]` Confirm status banner above input updates between `idle` / `red` / `green` / `audit` / `complete`
+33. `[LIVE]` Confirm `/phase idle` fires auto-reset on next user message after complete
+34. `[SOURCE]` Verify `packages/agent-runtime/src/tools/handlers/tool/transition-phase.ts` enforces FSM legal transitions + 10-iter breaker
+
+---
+
+## Section 7: Tool Gating (Hard Gate)
+
+35. `[LIVE]` During RED, drive write_file from Orchestrator to `src/example.ts` → BLOCKED
+36. `[LIVE]` During GREEN, same write_file → succeeds
+37. `[LIVE]` During RED, drive run_terminal_command `ls` → BLOCKED
+38. `[LIVE]` During AUDIT, same `ls` → succeeds
+39. `[LIVE]` Confirm `dev/scratchpad/` writes always allowed (dev mode off, scratchpad exempt per FID-008)
+40. `[LIVE]` Confirm `dev/fids/` writes always allowed (FID lifecycle privileged per FID-001)
+41. `[SOURCE]` Read `packages/agent-runtime/src/tools/tool-executor.ts` lines 339-372 → gate logic + dev-override + path-exemption present
+
+---
+
+## Section 8: Dev Override
+
+42. `[LIVE]` Drive `/dev` → enter dev mode
+43. `[LIVE]` Confirm dev mode banner / status indicator visible
+44. `[LIVE]` Orchestrator direct write to `dev/scratchpad/test.md` → succeeds (even in dev mode off due to path exemption)
+45. `[LIVE]` Confirm dev mode is INVISIBLE to user (no banner toast — only on toggle)
+46. `[LIVE]` Drive `/dev` again → exit dev mode
+47. `[LIVE]` After exit, direct write attempt to `src/example.ts` → BLOCKED
+
+---
+
+## Section 9: Agent Roster Conformance
+
+**Reconciliation note (post-FID-017, 2026-07-19):** Item 49 was updated to reflect the 9-runtime-role vs 14-fs-entry distinction documented in `ARCHITECTURE.md` "Helper Tool Libraries" section. Items 50/51 were updated to reflect FID-007/008 intentional tooling changes.
+
+48. `[SOURCE]` Read `ECHO.md` roster table — must list exactly 9 agents (Orchestrator, Detective, Forge, Verifier, Recorder, Thinker, Scout, Researcher, Scribe)
+49. `[SOURCE]` `ls agents/` → **14 directories** (9 canonical ECHO runtime roles + 5 active helper tool libraries: browser-use, editor, file-explorer, librarian, types)
+50. `[SOURCE]` Verify `agents/base2/base2.ts` has `write_file`+`str_replace` in toolNames (per FID-008 scratchpad design) BUT tool-executor.ts gates them to exempt paths (`dev/fids/`, `dev/scratchpad/`, `dev/nova/`) — separation of duties preserved via exemption, not absence
+51. `[SOURCE]` Verify `agents/detective/detective.ts` toolNames includes `code_search`, `set_output`, `list_directory`, `glob`, `read_files`, `read_subtree` (per FID-007 inlined extract-keywords + expanded delegation)
+52. `[SOURCE]` Verify `agents/forge/forge.ts` has `write_file`, `str_replace`, but NOT `bash`
+53. `[SOURCE]` Verify `agents/verifier/verifier.ts` has NO write tools
+54. `[SOURCE]` Verify no leftover `SAVANT_GEMINI_THINKER_*` vars in any agent template
+
+---
+
+## Section 10: Circuit Breaker
+
+55. `[LIVE]` Drive 11 iterations of self_correct → green → hard stop expected at iter 10
+56. `[LIVE]` Confirm warning surfaced at iter 8
+57. `[SOURCE]` `grep -n 'iterationCount\|MAX_ITERATIONS' packages/agent-runtime/src/tools/handlers/tool/transition-phase.ts` + `packages/agent-runtime/src/tools/handlers/tool/spawn-agent-utils.ts` → enforce at 10
+
+---
+
+## Section 11: UI State Sync
+
+58. `[LIVE]` Right sidebar `phase` indicator matches FSM after every `/phase` call
+59. `[LIVE]` Right sidebar `work` activity indicator (new per FID-009) reflects current sub-state (idle | thinking | tool | etc.)
+60. `[LIVE]` Right sidebar `tokens` updates after every agent step
+61. `[LIVE]` Right sidebar `cost` updates after every agent step (with env stub fallback)
+62. `[LIVE]` Status banner matches FSM phase (not stuck on "working")
+63. `[LIVE]` After `complete`, banner resets to ready/prompt within 2s
+
+---
+
+## Section 12: FID Lifecycle End-to-End
+
+64. `[LIVE]` Create FID-2026-0718-Y manually in `dev/fids/`
+65. `[LIVE]` Drive `/phase red` → confirm Detective spawns with code_search
+66. `[LIVE]` Drive `/phase green` → confirm Forge spawns with write_file
+67. `[LIVE]` Drive Forge write → confirm snapshot stored at `dev/scratchpad/.snapshots/`
+68. `[LIVE]` Drive `/phase audit` → confirm Verifier reads history (no writes)
+69. `[LIVE]` Drive `/phase complete` → confirm FID auto-archives to `dev/fids/archive/`
+70. `[LIVE]` Confirm CHANGELOG.md gains matching entry at top (auto-archive rule)
+
+---
+
+## Section 13: CHANGELOG + VERSION Consistency
+
+71. `[SOURCE]` `cat CHANGELOG.md | head -50` → most recent entry is FID-2026-0718-017 (per FID-017 close-out entry)
+72. `[SOURCE]` `cat VERSION` confirms 0.0.2 (post-FID-017 bump)
+73. `[SOURCE]` `cat package.json | grep version` matches 0.0.2
+74. `[SOURCE]` All archived FIDs in `dev/fids/archive/` have matching CHANGELOG entries (ECHO Auto-Archive rule)
+75. `[SOURCE]` No unarchived FID files in `dev/fids/*.md`
+
+---
+
+## Section 14: Build Artifact Hygiene
+
+76. `[BASH]` `find . -name '*.ts' -not -path './node_modules/*' -path '*/dist/*'` → 0 hits (no source in dist)
+77. `[BASH]` `git status --porcelain` (post `git add -A`) → 0 unstaged files
+78. `[BASH]` `find . -name '.DS_Store' -not -path './node_modules/*'` → 0 hits
+79. `[BASH]` `bun install --frozen-lockfile` → exit 0 (lockfile matches workspace state per FID-017 Option C — 11 pkg names reverted to `@savant-code/X`)
+
+---
+
+## Section 15: Pre-Rebrand Blockers (Hard Gate)
+
+80. `[BASH]` `git log --oneline -20` → after `git commit`, 3+ commits documenting FID history since rebrand start
+
+---
+
+## Section 16: Manual Dual-Audit Check
+
+81. `[SOURCE]` Trigger Nova audit on FID-017 closeout one more time → confirm PASS holds (per `dev/nova/outbox/2026-07-19-fid-017-closeout.md`)
+
+---
+
+## Final Sign-Off (Only After All Sections Pass)
+
+If all PASS, proceed to 0.0.2 push:
+1. ~~Bump `VERSION` to `0.0.2`~~ → **DONE by FID-017**
+2. ~~Bump `package.json` version to `0.0.2`~~ → **DONE by FID-017**
+3. `git commit -m "chore: 0.0.2 pre-rebrand safety milestone — FID-001..017 closed, 415/0 src tests + 488/0 full, 4 typechecks clean"`
+4. `git push` (your action; preserves the pre-rebrand snapshot)
+5. Confirm push success
+6. Archive this test-prompt to `dev/test-prompts/archive/0.0.2-final-pass.md`
+7. Greenlight the complete rebrand work in next push
+
+---
+
+## Report Format
+
+Per item: `### N: TITLE` then `Status: PASS|FAIL|SKIP|BLOCKER`, `Mode:`, `Evidence:`, `Error:`, `Timestamp:`.
+
+End with summary: pass count, fail count, skip count, blocker list. If ANY item in Sections 2, 3, 7, 12 are FAIL → **DON'T PUSH**. Open a new FID and loop.
+
+---
+
+**Run by:** Savant Orchestrator (you, before bumping VERSION)
+**Audit by:** Nova (third-party ECHO verification, optional but recommended for any uncertain items)
+**Approve by:** Creator (you, the human)
+
+---
+
+> *"0.0.2 = the safety checkpoint before the full rebrand gut. Restore point locked. Next push will own the entire codebase rename."*

package/dev/test-prompts/archive/agent-capabilities-full-test.md

@@ -0,0 +1,361 @@
+# Savant Agent — Full Capabilities Test
+
+**Version:** 1.0.0
+**Purpose:** Exhaustively exercise every tool, mode, phase gate, sub-agent, skill, path-safety rule, and validation path available to the Savant orchestrator. Run this prompt whenever the agent's capabilities have been changed to verify the system is production-ready.
+
+**How to use this prompt:** Point the Savant orchestrator at this file and instruct it: *"Execute the full capabilities test in this file. Report PASS/FAIL for every test with evidence. Continue until every section is complete. Write the final report to `dev/scratchpad/agent-capabilities-test-report.md`"*.
+
+**Quick-start for the testing agent:**
+1. Read this entire file first.
+2. Create a `write_todos` list with one todo per phase (Phases 1–13).
+3. Use `read_files`, `code_search`, and `spawn_agents` as needed to gather evidence.
+4. For every test, record: `TEST-ID`, `STATUS (PASS/FAIL/SKIP)`, `EVIDENCE`.
+5. Update todos as you finish each phase.
+6. Clean up temporary test artifacts at the end.
+7. Write the final report to `dev/scratchpad/agent-capabilities-test-report.md`.
+
+**Environment assumptions:**
+- The test runs inside a Savant-Code/SavantFree workspace with ECHO Protocol v0.2.0 active.
+- The CLI is interactive and supports the full tool set.
+- `dev/fids/` and `dev/scratchpad/` exist.
+- An internet connection is available for `web_search` and `read_url` tests.
+
+**Global rules for the testing agent:**
+1. Report every test with: `TEST-ID`, `STATUS (PASS/FAIL/SKIP)`, `EVIDENCE` (file path, line number, exact output, or error message).
+2. Do not skip a test unless it is impossible to run in the current environment; document the reason.
+3. If a test fails, continue to the next test. Do not halt the suite.
+4. Use `write_todos` to track progress through the phases.
+5. Preserve the git state and any user files; clean up temporary test files after each phase.
+6. **Record results incrementally.** After each phase, append the phase results to `dev/scratchpad/agent-test-progress.md` using the row format: `| TEST-ID | STATUS | EVIDENCE | NOTES |`. Do not wait until the end to record evidence.
+
+**How to enable `devMode` for Dev Override tests:**
+- In the interactive CLI, run the slash command `/dev on`. Disable with `/dev off`.
+- In programmatic tests, ensure the runtime sets `fileContext.devMode = true` before the test runs.
+- The key observable behavior: with `devMode` active, the FSM phase gate for `write_file`/`str_replace`/`apply_patch` and the AUDIT gate for `run_terminal_command` are bypassed.
+
+**Tests that may be skipped:**
+- `read_url` / `web_search` — SKIP if the environment has no internet.
+- `browser_logs` / `browser-use` — SKIP if Chrome/Chromium is not installed or no browser session exists.
+- `tmux-cli` — SKIP if no tmux session is available.
+
+**At-a-glance checklist:**
+- [ ] Phase 1: Modes verified by reading `agents/savant/*.ts`.
+- [ ] Phase 2: Every tool called with valid input.
+- [ ] Phase 3: ECHO phase gates enforced correctly.
+- [ ] Phase 4: Every sub-agent spawned and returned output.
+- [ ] Phase 5: FID lifecycle works.
+- [ ] Phase 6: Path safety rejects escapes and accepts exempt paths.
+- [ ] Phase 7: All skills load.
+- [ ] Phase 8: Dev override bypasses phase gates.
+- [ ] Phase 9: Perfection Loop cycles through phases and respects circuit breaker.
+- [ ] Phase 10: Sequential thinking / Thinker agent works.
+- [ ] Phase 11: Typechecks and key tests pass.
+- [ ] Phase 12: Error cases produce expected errors.
+- [ ] Phase 13: Report written.
+
+**Critical path (must all pass for sign-off):**
+- M-01 `default` mode has the expected orchestrator tool set.
+- T-07 `write_file` succeeds in GREEN phase.
+- T-19 `run_terminal_command` succeeds in AUDIT phase.
+- T-26 `transition_phase` succeeds for valid transitions.
+- T-31 `sequentialthinking` / T-30 `think_deeply` spawns Thinker and returns reasoning.
+- P-01 `write_file` in `idle` is blocked.
+- P-03 `write_file` in `green` is allowed.
+- P-05–P-06 exempt paths (`dev/scratchpad`, `dev/fids`) are writable in any phase.
+- A-06 `thinker` agent returns a structured analysis with `<thinking>` tags.
+- PL-01 Perfection Loop advances through RED → GREEN → AUDIT → COMPLETE.
+- V-01..V-04 all typechecks pass.
+- E-01 `spawn_agents` with invalid agent returns an error.
+
+**Known disabled features:**
+- Composio meta tools are currently disabled (`ENABLE_COMPOSIO_TOOLS = false` in `agents/savant/savant.ts`). Do not test them unless you have explicitly enabled them.
+- External browser tests are optional and only run when Chrome/Chromium is available.
+
+---
+
+## Phase 1: Orchestrator Identity & Modes
+
+For each Savant mode below, verify the agent can be instantiated and reports the correct identity/tool set.
+
+| Test | Mode | Expected Behavior |
+|------|------|---------------------|
+| M-01 | `default` | `displayName` = "Savant the Orchestrator"; has `spawn_agents`, `read_files`, `transition_phase`, `write_file`, `str_replace`, `ask_user`, `suggest_followups` |
+| M-02 | `max` | Same as default but reads more files; has `transition_phase`, `write_file`, `str_replace`, `apply_patch` |
+| M-03 | `fast` | No `write_todos`, `ask_user`, `suggest_followups`; still delegates writes |
+| M-04 | `free`/`lite` | Uses free-tier model; has data_collection deny; still has orchestrator tools |
+| M-05 | `analyze` | Read-only: has `code_search`, `read_files`, `spawn_agents` (Detective/Scout/Thinker); does NOT have `write_file`, `str_replace`, `apply_patch`, `transition_phase` |
+| M-06 | `plan` | Has `ask_user`, `spawn_agents`; does NOT have `write_file`, `str_replace`, `apply_patch` |
+| M-07 | `scaffold` | Has `set_scaffold_complete`; creates one umbrella FID only |
+| M-08 | `deep` | Orchestrates a 6–7 phase workflow with SPEC.md/PLAN.md; has `transition_phase` |
+
+**Execution:**
+- For each mode, read the relevant `agents/savant/*.ts` file (e.g., `savant.ts`, `savant-analyze.ts`, `savant-plan.ts`, `savant-scaffold.ts`, `savant-deep.ts`) and record the tool set.
+- Verify the expected presence/absence of each tool above.
+
+---
+
+## Phase 2: Tool-by-Tool Functional Test
+
+For each tool in `common/src/tools/constants.ts`, call it with a valid input and verify the result. Report PASS/FAIL per tool.
+
+| Test | Tool | Minimal Valid Input | Expected Success Signal |
+|------|------|---------------------|---------------------------|
+| T-01 | `read_files` | `{ "paths": ["ECHO.md"] }` | Returns file contents |
+| T-02 | `read_subtree` | `{ "paths": ["agents"] }` | Returns tree summary |
+| T-03 | `list_directory` | `{ "path": "agents" }` | Returns directory entries |
+| T-04 | `glob` | `{ "pattern": "agents/**/*.ts" }` | Returns matching paths |
+| T-05 | `code_search` | `{ "pattern": "resolveAndContain", "flags": "-g common/src/**/*.ts" }` | Returns search results |
+| T-06 | `find_files` | `{ "query": "agent runtime tool executor" }` | Returns relevant files |
+| T-07 [CRITICAL] | `write_file` | Must be in GREEN phase or exempt path; write to `dev/scratchpad/tool-test.md` | File is written |
+| T-08 | `str_replace` | Replace a line in `dev/scratchpad/tool-test.md` | File is updated |
+| T-09 | `apply_patch` | Apply a small diff to `dev/scratchpad/tool-test.md` | File is patched |
+| T-10 | `propose_write_file` | Propose writing `dev/scratchpad/proposed.md` | Proposal returned (editor best-of-n) |
+| T-11 | `propose_str_replace` | Propose replacing text in `dev/scratchpad/tool-test.md` | Proposal returned (editor best-of-n) |
+| T-12 [OPTIONAL] | `read_url` | `{ "url": "https://example.com" }` | Returns page text (SKIPPABLE if offline) |
+| T-13 [OPTIONAL] | `web_search` | `{ "query": "OpenTUI React terminal UI" }` | Returns search results (SKIPPABLE if offline) |
+| T-14 | `read_docs` | `{ "query": "React useEffect", "source": "react.dev" }` | Returns doc excerpt |
+| T-15 | `skill` | `{ "name": "coding-typescript" }` | Returns skill contents |
+| T-16 | `gravity_index` | `{ "action": "list_categories" }` | Returns service categories |
+| T-17 | `spawn_agents` | `{ "agents": [{"agent_type": "detective", "prompt": "find tests"}] }` | Spawns Detective |
+| T-18 | `spawn_agent_inline` | `{ "agent_type": "context-pruner", "params": {"maxContextLength": 400000} }` | Spawns inline context pruner |
+| T-19 [CRITICAL] | `run_terminal_command` | Must be in AUDIT phase; `{ "command": "echo hello" }` | Returns `hello` |
+| T-20 | `write_todos` | `{ "todos": [{"task": "test", "completed": false}] }` | Todos displayed |
+| T-21 | `ask_user` | `{ "questions": [{"question": "Approve?", "options": [{"label": "Yes"}]} }` | Renders question (user may skip response) |
+| T-22 | `suggest_followups` | `{ "followups": [{"prompt": "Next step", "label": "Next"}] }` | Renders followups |
+| T-23 | `set_output` | `{ "data": {"status": "ok"} }` | Output stored |
+| T-24 | `add_message` | `{ "role": "user", "content": "test" }` | Message added to history |
+| T-25 | `set_messages` | `{ "messages": [] }` | History replaced |
+| T-26 [CRITICAL] | `transition_phase` | `{"phase": "red", "reason": "test"}` in non-analyze mode | Phase transitions |
+| T-27 | `update_subgoal` | `{ "id": "test", "status": "IN_PROGRESS" }` | Subgoal updated |
+| T-28 | `create_plan` | `{ "path": "dev/scratchpad/test-plan.md", "plan": "test plan" }` | Plan file created |
+| T-29 | `render_ui` | `{ "widget": {"type": "button", "text": "Test", "link": "https://example.com"} }` | Renders button |
+| T-30 | `think_deeply` | `{ "prompt": "Think about test coverage" }` | Thinker agent spawns |
+| T-31 | `sequentialthinking` | `{ "prompt": "Plan a test" }` | Thinker agent spawns |
+| T-32 [OPTIONAL] | `browser_logs` | Only if browser active | Browser logs returned (SKIPPABLE if no browser session) |
+| T-33 | `end_turn` | `{}` | Turn ends |
+| T-34 | `task_completed` | `{}` | Task marked complete |
+| T-35 | `set_scaffold_complete` | In scaffold mode | Scaffold mode ends |
+
+---
+
+## Phase 3: ECHO Phase Gate Tests
+
+**Goal:** Verify that write tools are blocked outside GREEN phase and allowed inside it, and that scratchpad/FID paths are exempt.
+
+| Test | Phase | Action | Expected |
+|------|-------|--------|----------|
+| P-01 | `idle` | `write_file` to `src/test.txt` | Blocked: "only available during the GREEN phase" |
+| P-02 | `red` | `write_file` to `src/test.txt` | Blocked |
+| P-03 | `green` | `write_file` to `src/test.txt` | Allowed (clean up after) |
+| P-04 | `audit` | `write_file` to `src/test.txt` | Blocked || P-05 | `any` | `write_file` to `dev/scratchpad/any-phase.md` | Allowed (exempt path) |
+| P-06 | `any` | `write_file` to `dev/fids/any-phase.md` | Allowed (exempt path) |
+| P-07 | `idle` | `run_terminal_command` | Blocked: "only available during the AUDIT phase" |
+| P-08 | `audit` | `run_terminal_command "echo ok"` | Allowed |
+| P-09 | `analyze` | `write_file` | Tool not present; agent returns error |
+| P-10 | `dev` | `write_file` to `src/test.txt` in `idle` phase with `devMode` enabled | Allowed (dev override bypasses phase gate) |
+
+---
+
+## Phase 4: Sub-Agent Roster Test
+
+For each agent in the ECHO roster, spawn it with a minimal task and verify it executes without error.
+
+| Test | Agent | Task Prompt | Expected |
+|------|-------|-------------|----------|
+| A-01 | `detective` | "Search for all tests of resolveAndContain" | Returns issue catalog |
+| A-02 | `scout` | "Find all files related to the sidebar" | Returns file list |
+| A-03 | `researcher-web` | "What is OpenTUI?" | Returns research summary |
+| A-04 | `researcher-docs` | "React useEffect docs" | Returns doc summary |
+| A-05 | `basher` | "Run `cd common && bun run typecheck`" | Returns command output |
+| A-06 | `thinker` | "Think through the pros and cons of adding gap={1} to all sidebar components" | Returns analysis |
+| A-07 | `forge` | Must be in GREEN phase; "Create a tiny test file in dev/scratchpad" | File created |
+| A-08 | `verifier` | "Review the file created by forge" | Returns review |
+| A-09 | `recorder` | "Create a test FID in dev/fids/" | FID created |
+| A-10 | `scribe` | "Summarize this session's changes" | Returns summary |
+| A-11 | `context-pruner` | (spawned automatically) | Context pruned |
+| A-12 | `browser-use` | "Fetch https://example.com" | Returns page text |
+| A-13 | `tmux-cli` | (if CLI interactive testing is available) | Executes command (SKIPPABLE if no tmux session available) |
+
+**Sub-agent verification criteria:** For each spawned sub-agent, the test passes if:
+- The agent runs to completion (no tool/validation error).
+- The returned output contains the expected structure:
+  - **Detective:** at least one issue with `file path`, `line number`, and `severity`.
+  - **Scout:** a list of relevant file paths.
+  - **Researcher:** a concise summary with source URLs or doc references.
+  - **Basher:** the exact command output and exit code.
+  - **Thinker:** `<thinking>` ... `</thinking>` tags with numbered steps and a conclusion.
+  - **Forge:** the created/updated file path.
+  - **Verifier:** a list of findings with at least one critical or blocking issue flagged.
+  - **Recorder/Scribe:** the created/updated file path or summary text.
+- No disallowed tool is called by that agent (e.g., Detective must not call `write_file`).
+
+---
+
+## Phase 5: FID Lifecycle
+
+| Test | Action | Expected |
+|------|--------|----------|
+| FID-01 | Recorder creates a FID in `dev/fids/` | FID file exists with required metadata |
+| FID-02 | Update the FID status to `in_progress` | FID status field updated |
+| FID-03 | Move the FID to `dev/fids/archive/` | FID is archived |
+
+---
+
+## Phase 6: Path Safety & Containment
+
+| Test | Path | Tool | Expected |
+|------|------|------|----------|
+| S-01 | `/etc/passwd` | `write_file` | Rejected: escapes project root |
+| S-02 | `../../etc/passwd` | `write_file` | Rejected: escapes project root |
+| S-03 | `dev/fids/../etc/passwd` | `write_file` | Rejected: collapses to escape |
+| S-04 | `dev/scratchpad/test.md` | `write_file` | Allowed: exempt |
+| S-05 | `dev/fids/test.md` | `write_file` | Allowed: exempt |
+| S-06 | Absolute path to `dev/scratchpad/test.md` | `write_file` | Allowed: exempt |
+| S-07 | Symlink to `/etc/passwd` | `write_file` | Rejected: symlink escape |
+
+---
+
+## Phase 7: Skills System
+
+| Test | Skill | Expected |
+|------|-------|----------|
+| SK-01 | `coding-typescript` | Loads TypeScript conventions |
+| SK-02 | `coding-python` | Loads Python conventions |
+| SK-03 | `coding-rust` | Loads Rust conventions |
+| SK-04 | `coding-java` | Loads Java conventions |
+| SK-05 | `coding-go` | Loads Go conventions |
+| SK-06 | `coding-csharp` | Loads C# conventions |
+| SK-07 | `release-workflow` | Loads release workflow |
+
+---
+
+## Phase 8: Dev Override & FSM Bypass
+
+| Test | Scenario | Expected |
+|------|----------|----------|
+| DEV-01 | Enable `devMode`, then `write_file` to `tmp/devmode-test.txt` in `idle` phase | Allowed (dev override); clean up after |
+| DEV-02 | Disable `devMode`, then `write_file` to `tmp/devmode-test.txt` in `idle` phase | Blocked (phase gate active) |
+| DEV-03 | With `devMode` on, `run_terminal_command` in `idle` phase | Allowed (dev override bypasses AUDIT gate) |
+
+---
+
+## Phase 9: Perfection Loop
+
+**Goal:** Verify the ECHO Perfection Loop advances through phases, displays correctly in the sidebar, and respects the circuit breaker.
+
+| Test | Action | Expected |
+|------|--------|----------|
+| PL-01 [CRITICAL] | Create a FID in `dev/fids/`, transition `idle` → `red` → `green` → `audit` → `self_correct` → `complete` | Each `transition_phase` succeeds and the right sidebar shows the matching phase label/color |
+| PL-02 | In the sidebar, verify the Perfection Loop widget shows RED as the active phase when at least one open FID has status `created` or `analyzed` | Widget renders without overlap, current step highlighted |
+| PL-03 | Verify the loop prevents invalid transitions (e.g., `idle` → `complete`) | Returns an error or is rejected by the FSM |
+| PL-04 | Open more than 10 FIDs and cycle them; verify a circuit breaker/halting condition is reached (or the system degrades gracefully) | No crash; loop state remains consistent |
+| PL-05 | Archive all open FIDs and confirm the loop returns to the `complete`/`idle` state | Sidebar shows loop converged |
+
+---
+
+## Phase 10: Sequential Thinking / Thinker Agent Deep Test
+
+**Goal:** Verify the Thinker agent, `think_deeply`, and `sequentialthinking` produce structured reasoning and respect tool restrictions.
+
+| Test | Action | Expected |
+|------|--------|----------|
+| ST-01 [CRITICAL] | Spawn the `thinker` agent with a multi-step problem (e.g., "Compare three approaches to caching in this repo") | Returns a response containing `<thinking>` ... `</thinking>` tags with numbered steps and a conclusion |
+| ST-02 | Call `sequentialthinking` from the Orchestrator with a prompt that requires stepwise reasoning | Thinker agent is spawned and returns a structured answer |
+| ST-03 | Call `think_deeply` with the same prompt | Returns deep reasoning; no crash |
+| ST-04 | Verify `sequentialthinking` is rejected when called by a non-Thinker agent | Error: "only available to Thinker agents" |
+| ST-05 | Verify the `<thinking>` block is stripped from the final user-facing output (if applicable) | Final message does not contain raw `<thinking>` tags |
+
+---
+
+## Phase 11: Validation Gates
+
+| Test | Command | Expected |
+|------|---------|----------|
+| V-01 | `cd sdk && bun run typecheck` | Exit 0 |
+| V-02 | `cd common && bun run typecheck` | Exit 0 |
+| V-03 | `cd packages/agent-runtime && bun run typecheck` | Exit 0 |
+| V-04 | `cd cli && bun run typecheck` | Exit 0 |
+| V-05 | `bun test common/src/util/__tests__/paths.test.ts` | All pass |
+| V-06 | `bun x eslint cli/src/components/right-sidebar.tsx --max-warnings 0` | Exit 0 |
+
+---
+
+## Phase 12: Error Handling & Edge Cases
+
+| Test | Scenario | Expected |
+|------|----------|----------|
+| E-01 | Call `spawn_agents` with a non-existent agent | Error: agent does not exist |
+| E-02 | Call `write_file` with missing `content` | Validation error |
+| E-03 | Call `str_replace` with mismatched `oldString` | Error/patch not applied |
+| E-04 | Call `read_files` with non-existent path | Error or empty result |
+| E-05 | Call `run_terminal_command` with destructive command in AUDIT | Agent warns and asks user |
+| E-06 | Call `transition_phase` with invalid phase | Validation error |
+
+---
+
+## Phase 13: Report & Cleanup
+
+After all tests, write the report to `dev/scratchpad/agent-capabilities-test-report.md` with the following sections. You may copy the template below and fill it in.
+
+**Cleanup — remove all temporary test artifacts:**
+- `dev/scratchpad/tool-test.md`
+- `dev/scratchpad/proposed.md`
+- `dev/scratchpad/test-plan.md`
+- `dev/scratchpad/agent-test-progress.md`
+- `tmp/devmode-test.txt` (and any `tmp/` directory if created)
+- Any test FIDs created in `dev/fids/` (move to `dev/fids/archive/` if they should be kept for evidence, then delete if ephemeral)
+- Any files written to non-exempt source paths (e.g., `src/test.txt`) — these should NOT exist after cleanup.
+
+**Important:** Do not delete `dev/scratchpad/agent-capabilities-test-report.md` — this is the deliverable.
+
+**Report template:**
+
+```markdown
+# Agent Capabilities Test Report
+
+## Summary
+- **Date:**
+- **Agent/Model:**
+- **Total tests:**
+- **PASS:**
+- **FAIL:**
+- **SKIP:**
+
+## Critical Success Criteria
+- [ ] All mode tests (M-01..M-08) pass.
+- [ ] All core tools (T-01..T-35) pass.
+- [ ] GREEN phase allows source writes; non-GREEN phases block them.
+- [ ] Exempt paths (`dev/scratchpad`, `dev/fids`) are writable in any phase.
+- [ ] All 9+ sub-agents spawn and execute.
+- [ ] All typecheck gates (V-01..V-04) pass.
+
+## Detailed Results
+| Test ID | Section | Status | Evidence | Notes |
+|---------|---------|--------|----------|-------|
+| T-01    | Tools   | PASS   | `read_files` returned ECHO.md contents | - |
+
+## Failures & Recommendations
+List each FAIL/SKIP, the root cause, and the recommended fix or further investigation.
+
+## Sign-off
+The agent is / is not verified for the next run because: __________
+```
+
+---
+
+## Appendix: Quick Reference for the Testing Agent
+
+**Tool set to verify presence/absence:**
+- Orchestrator: `spawn_agents`, `read_files`, `read_subtree`, `write_todos`, `suggest_followups`, `ask_user`, `read_url`, `skill`, `set_output`, `list_directory`, `glob`, `render_ui`, `gravity_index`, `transition_phase`, `write_file`, `str_replace`, `apply_patch`, `set_scaffold_complete` (scaffold mode).
+- Read-only agents: Detective, Scout, Researcher, Verifier.
+- Write agents: Forge.
+- Utility agents: Basher, Thinker, Context-Pruner, Recorder, Scribe, Browser-Use, tmux-cli.
+
+**ECHO phases:** `idle` → `red` → `green` → `audit` → `self_correct` → `complete`.
+
+**Exempt paths (always writable):** `dev/fids/`, `dev/scratchpad/`.
+
+**FSM rules:**
+- `write_file`, `str_replace`, `apply_patch` → GREEN phase only (or exempt path / dev override / scaffold mode).
+- `run_terminal_command` → AUDIT phase only.
+- `sequentialthinking` → Thinker agents only.