khabotcli 1.0.0

package/.OLD/STEP-1/step-1.md

@@ -0,0 +1,1066 @@
+# KhaBotCli — Step 1: Audit, Questions & Recommendations
+
+> **Date:** Feb 27, 2026  
+> **Context:** Full audit of PLAN.md v1.0 against the current AI agent ecosystem (Claude Code v2.1.62, OpenCode, Cursor 2.0, Codex CLI, MCP spec 2025-11-25, Anthropic's context engineering guide).
+
+---
+
+## Table of Contents
+
+1. [Critical Questions (Need Your Answers)](#1-critical-questions)
+2. [Over-Complications to Remove](#2-over-complications-to-remove)
+3. [CLI Behaviour Issues](#3-cli-behaviour-issues)
+4. [Missing Features](#4-missing-features)
+5. [Session Stats & Context Optimizations](#5-session-stats--context-optimizations)
+6. [Built-in Tools Revision](#6-built-in-tools-revision)
+7. [Agent/Skill/Plugin Architecture](#7-agentskillplugin-architecture)
+8. [System Prompts](#8-system-prompts)
+9. [Image Handling](#9-image-handling)
+10. [File/URL Handling](#10-fileurl-handling)
+11. [Agent-to-Agent Communication](#11-agent-to-agent-communication)
+12. [Loading & Memory Optimizations](#12-loading--memory-optimizations)
+13. [MCP Revision](#13-mcp-revision)
+14. [Task System Revision](#14-task-system-revision)
+15. [Daemon Agents Revision](#15-daemon-agents-revision)
+16. [Testing Strategy](#16-testing-strategy)
+17. [Documentation Plan](#17-documentation-plan)
+18. [Forkable Config & Branding](#18-forkable-config--branding)
+19. [Diagrams & Overview](#19-diagrams--overview)
+20. [Schema/JSON Revisions](#20-schemajson-revisions)
+21. [Plan Structure Proposal](#21-plan-structure-proposal)
+22. [Research Items Needed](#22-research-items-needed)
+
+---
+
+## 1. Critical Questions
+
+These questions **must be answered** before the plan can be rewritten. Mark your answers inline or in a separate file.
+
+### Q1. KhaBot.com API Scope
+Since KhaBot.com isn't ready, what minimal API surface should KhaBotCLI expect from it in the future?
+- **(a)** Just token validation (verify `khabot_token` is valid)
+- **(b)** Token validation + agent/skill/tool registry (download pre-built agents)
+- **(c)** Full platform (registry + billing + analytics + team management)
+- **(d)** Other: ___
+
+> **Your answer:** When I said KhaBot.com isn't ready, I meant to make support for some pre-built agents, skills and tools available, witout the need for internet, i mean i dont have to download the file reader tool for example, also KhaBot.com will be edited inline with the KhaBotCLI development, just we should have a plan of whats expected from KhaBot.com in details
+
+### Q2. Target LLM Providers
+- **(a)** OpenRouter only (simplest — one `base_url` + `api_key`)
+- **(b)** OpenRouter + direct provider APIs (Anthropic, OpenAI, Google, etc.)
+- **(c)** OpenRouter + direct APIs + local models (Ollama, LM Studio, etc.)
+
+> **Your answer:** I mean we just do (`base_url` + `api_key`) and user can use any LLM provider, OpenRouter or any other provider, since Ollama (Local LLMs) provides local (`base_url` + `api_key`) that can be used for KhaBotCLI
+
+### Q3. Primary Language
+- **(a)** Node.js / TypeScript (matches Claude Code ecosystem)
+- **(b)** Go (like OpenCode — faster, single binary, no runtime dependency)
+- **(c)** Rust
+- **(d)** Other: ___
+
+> **Your answer:** Node.js + JSdoc (No TypeScript)
+
+### Q4. Interactive TUI Mode
+Current plan is REST API + one-shot only. Do you want an interactive terminal UI?
+- **(a)** No TUI — strictly REST API + one-shot CLI. Consumers build their own UI.
+- **(b)** Basic TUI for debugging/testing (like OpenCode's Bubble Tea UI)
+- **(c)** Full TUI as a primary interface
+
+> **Your answer:** strictly REST API + one-shot CLI + A Good DOC, so we can build independent UIs that uses the Rest API
+
+### Q5. Streaming Responses
+Current plan says "Returns response synchronously (no streaming)." Every competitor streams.
+- **(a)** No streaming — keep synchronous. Consumers poll `/tasks/:id` for progress.
+- **(b)** SSE streaming on API endpoints
+- **(c)** WebSocket streaming
+- **(d)** Both SSE and polling (SSE for real-time consumers, polling as fallback)
+
+> **Your answer:** No streaming, for easy maintenance, and more focus on the needs, plus AIs are getting faster anyway
+
+### Q6. Agent Ownership / Runtime Creation
+- **(a)** Agents are only defined as files by the user (current plan)
+- **(b)** Agents can also be created at runtime via API (e.g., `POST /agents`)
+- **(c)** Agents can create other agents at runtime (self-replication)
+
+> **Your answer:** Agents are only defined as files in ~/.khabot/agents or ./.khabot/agents, if a script/agent/human wants to add a custom agent, it should add it to ./.khabot/agents and run khabotcli from there
+
+### Q7. Forking Scope
+What exactly should be configurable for forks?
+- **(a)** Just branding (domain name, product name, logo URL)
+- **(b)** Branding + default agents/tools/skills
+- **(c)** Branding + defaults + custom API endpoints + custom auth flow
+- **(d)** Full white-label (everything swappable)
+
+> **Your answer:** Basically the "khabot.com" endpoint, the name "KhaBot" and the defaults, the system is the same
+
+### Q8. Image Handling Priority
+- **(a)** Vision only (agent can read/analyze images sent to it)
+- **(b)** Vision + generation (via tool/MCP, e.g., DALL-E, Stable Diffusion)
+- **(c)** Vision + generation + editing (full image pipeline)
+- **(d)** Not a priority for v1
+
+> **Your answer:** Basic Vision + generation (Gen using normal API since Many new models support generating images/audios/videos), no need for third party tools, but MCP already can do all (MCP wont be used, it just to be there for people who want to use it in their custom agents)
+
+### Q9. Browser Automation
+Claude Code now has a `Computer` tool for Chrome browser automation. Does KhaBotCLI need this?
+- **(a)** No — out of scope
+- **(b)** Yes, as a built-in tool
+- **(c)** Yes, but as an optional MCP server (not built-in)
+
+> **Your answer:** No, out of scope, if user wants Browser Automation, they can use an MCP for that
+
+### Q10. Remote Access (Ably vs Alternatives)
+- **(a)** Keep Ably (current plan)
+- **(b)** Replace with WebSocket server built into KhaBotCLI
+- **(c)** Replace with SSE endpoints
+- **(d)** Remove remote access from v1, add later
+- **(e)** Keep Ably but also support direct WebSocket
+
+> **Your answer:** Keep Ably (current plan)
+
+### Q11. Skills vs Commands
+In Claude Code: Skills = reusable prompt templates, Commands = slash-invocable actions. In the current plan, both exist but their distinction is unclear.
+- **(a)** Merge into one concept ("Skills" that can be invoked via slash or by the agent)
+- **(b)** Keep both separate (Skills = agent-invocable, Commands = user-invocable)
+- **(c)** Rename both to something else: ___
+
+> **Your answer:** Skills are meant to be used by the agents, But dont forget that the user can chat with an Agent and use commands in their message
+
+### Q12. How Are Agents Invoked Externally?
+- **(a)** REST API only (current plan)
+- **(b)** REST API + CLI commands (e.g., `khabot chat researcher "what is quantum computing?"`)
+- **(c)** REST API + CLI + Ably/remote
+
+> **Your answer:** REST API + CLI, But for CLI a user can start an agent (one shot mode), and ably will simply expose the REST API for remote access through Ably
+
+---
+
+## 2. Over-Complications to Remove
+
+These items add complexity without proportional value. Recommendation: **remove or simplify**.
+
+### 2.1 Browser OAuth Login (Section 2.4)
+**Problem:** The plan says `khabot login` opens a browser to KhaBot.com OAuth flow. You confirmed KhaBot.com isn't ready and this was hallucinated.  
+**Fix:** Remove browser OAuth entirely. Login = `khabot login --key <token>` only. Store token in `~/.khabot/auth.json`.
+
+> **Your answer:** ACCEPTED
+
+### 2.2 Dual auth.json Locations
+**Problem:** Auth can be at `~/.khabot/auth.json` (global) OR `./.khabot/auth.json` (project). This creates confusion about which token is used.  
+**Fix:** Single location: `~/.khabot/auth.json`. Project-level settings can override the *model* and *permissions*, but auth should be global only.
+
+> **Your answer:** REJECTED, needed for testing
+
+### 2.3 Three Agent Types as Rigid Categories
+**Problem:** Chat, Task, and Daemon are treated as fundamentally different things with separate code paths and endpoints. In reality:
+- A **Chat** agent is just an agent with a persistent session
+- A **Task** agent is just an agent with a job queue entry and a terminal state
+- A **Daemon** agent is just an agent with a cron trigger  
+
+**Fix:** Unify into a single Agent concept. The *type* becomes an *invocation mode*. Any agent can be invoked as chat, task, or daemon. The agent definition doesn't change — only how it's triggered.
+
+> **Your answer:** REJECTED, Chat are meant to interact with the user, Task are meant to receive a single input and loop until completion, Daemon are meant to run in the background and perform actions, with time different optimizations will be applied to each, Like Task agent will have access to Task answering tool, giving that to a chat agent may make confusion, so the developer of an agent should state the type of the agent they want to create
+
+### 2.4 Custom JS Tool Implementations
+**Problem:** `tool.json` + `index.js` custom tools are a security risk (arbitrary code execution) and duplicate what MCP already provides.  
+**Fix:** Deprecate custom JS tools. Use MCP servers for all external tool extensions. Built-in tools are part of KhaBotCLI core. If custom tools are truly needed, they should be MCP servers.
+
+> **Your answer:** REJECTED, Custom Tools need to exist, for testing or custom Agents, MCP only for specific user needs
+
+### 2.5 Individual JSON Files Per Session/Task
+**Problem:** At 50k sessions with individual `meta.json` + `messages.json` files each, the filesystem becomes unmanageable. Directory listing alone becomes slow. No indexing, no querying.  
+**Fix:** Use SQLite (like OpenCode) for session/task storage. Single `~/.khabot/data.db` file. Queryable, indexable, atomic writes, backup-friendly.
+
+> **Your answer:** ACCEPTED
+
+### 2.6 Over-Specified Hooks
+**Problem:** 5 hook types (`PreToolUse`, `PostToolUse`, `beforeBash`, `afterWrite`, `afterResponse`) with inconsistent naming (camelCase vs PascalCase).  
+**Fix:** Standardize to just `PreToolUse` and `PostToolUse` (matching Claude Code). Tool name is available in env vars, so `beforeBash` = `PreToolUse` where `KHABOT_TOOL_NAME=bash`. Add `Notification` hook for agent events.
+
+> **Your answer:** ACCEPTED
+
+---
+
+## 3. CLI Behaviour Issues
+
+Problems with the current CLI design:
+
+### 3.1 Import Mode is Too Rigid
+**Problem:** `khabot import --type=claude-code` assumes a specific source format. What about importing from OpenCode, Cursor, or raw markdown?  
+**Fix:** Make import extensible with adapters. Ship with `claude-code` and `openclaw` adapters. Allow community adapters via MCP or plugins.
+
+> **Your answer:** ACCEPTED
+
+### 3.2 One-Shot Mode Has No Progress Feedback
+**Problem:** `khabot run --agent researcher --input "..."` blocks until completion with no output. For long tasks, the user has no idea what's happening.  
+**Fix:** Stream progress to stderr (iterations, tool calls) while the final result goes to stdout. Or add `--progress` flag.
+
+> **Your answer:** ACCEPTED, but progress should stream to stdout with a special prefix (e.g., `KHABOT_PROGRESS={ ... json object of the progress state ...}`)
+
+### 3.3 No `khabot status` Command
+**Problem:** No way to check what's running without hitting the API.  
+**Fix:** Add `khabot status` — shows running server, active sessions, active tasks, daemon status.
+
+> **Your answer:** ACCEPTED
+
+### 3.4 No `khabot stop` Command
+**Problem:** `POST /shutdown` exists but no CLI equivalent.  
+**Fix:** Add `khabot stop` — sends shutdown signal to running server.
+
+> **Your answer:** ACCEPTED
+
+### 3.5 No Agent Management Commands
+**Problem:** No CLI commands to list/inspect/create agents.  
+**Fix:** Add:
+- `khabot agents list` — list all discovered agents
+- `khabot agents inspect <name>` — show agent details
+- `khabot agents create` — interactive agent creation (like OpenCode's `opencode agent create`)
+
+> **Your answer:** ACCEPTED
+
+---
+
+## 4. Missing Features
+
+### 4.1 Context Compaction System
+**What:** Auto-summarize conversation when approaching context window limit.  
+**How it works in competitors:**
+- **Claude Code:** Hidden compaction agent summarizes message history, preserves architectural decisions + unresolved bugs + implementation details, discards redundant tool outputs. Keeps 5 most recently accessed files.
+- **OpenCode:** Auto-compact at 95% context usage. Creates new session with summary.
+- **Anthropic's recommendation:** Maximize recall first, then iterate precision. Tool result clearing is the lightest compaction.
+
+**Recommendation for KhaBotCLI:**
+- Built-in hidden `compaction` agent (uses cheaper model)
+- Triggers at configurable threshold (default 90%)
+- Summarizes: key decisions, unresolved issues, current task state, file paths
+- Clears: raw tool outputs, redundant messages, resolved issues
+- Configurable in `settings.json`: `{ "compaction": { "threshold": 0.9, "model": "anthropic/claude-3-haiku", "enabled": true } }`
+
+> **Your answer:** ACCEPTED
+
+### 4.2 Session Statistics & Token Tracking
+**What:** Track per-session/task: input tokens, output tokens, cache tokens, cost estimate, duration, iterations, tool calls count.  
+**Why:** Essential for cost management and debugging. No competitor ships without this.  
+**Schema addition:**
+```json
+{
+  "tokenUsage": {
+    "inputTokens": 15420,
+    "outputTokens": 3200,
+    "cacheReadTokens": 8000,
+    "cacheWriteTokens": 2000,
+    "totalTokens": 28620,
+    "estimatedCost": 0.042
+  },
+  "toolCalls": 12,
+  "compactionCount": 1
+}
+```
+
+> **Your answer:** ACCEPTED
+
+### 4.3 Sub-Agent Architecture
+**What:** Agents can spawn lightweight sub-agents with isolated context windows.  
+**How it works in Claude Code:**
+- `Task` tool — spawns a sub-agent for a specific task, returns result
+- `Explore` sub-agent — read-only, for codebase exploration
+- `Plan` sub-agent — analysis only, no modifications
+- Each sub-agent gets a clean context, does extensive work, returns a condensed summary (1,000-2,000 tokens)
+
+**Recommendation:** Built-in sub-agent types:
+- **explore** — read-only codebase exploration (cheaper model)
+- **plan** — analysis and planning without modifications
+- **execute** — full tool access for specific tasks
+- **summarize** — context compaction (cheapest model)
+- Custom sub-agents definable by user
+
+> **Your answer:** ACCEPTED, but in order to make `Explore` agent for example, for chat and to be a sub-agent, there should be two of them, one called `Explore` (type: Chat Agent) and one called `Explore sub-agent` (type: Task Agent), to align with our current infrastructure, unless you give me a better idea.
+
+### 4.4 Structured Note-Taking / TodoWrite
+**What:** Agents maintain a structured TODO list that persists across compactions.  
+**How in Claude Code:** `TodoWrite` tool (2,167 tokens description!) — creates and manages task lists, tracks progress, survives compaction.  
+**Recommendation:** Built-in `todo_write` and `todo_read` tools. TODO stored in session metadata, always included in compacted context.
+
+> **Your answer:** ACCEPTED
+
+### 4.5 ToolSearch / Lazy Tool Loading
+**What:** Not all tools are loaded into context at startup. Agent discovers tools when needed.  
+**How in Claude Code:** `ToolSearch` tool — agent can search for available tools by name/description. Tools are loaded into context only when found and needed.  
+**Why:** Reduces context pollution. With 50+ MCP tools, loading all tool descriptions into system prompt wastes context.  
+**Recommendation:** Built-in `tool_search` tool. Tool descriptions loaded on-demand. Only actively-used tool descriptions kept in context.
+
+> **Your answer:** ACCEPTED
+
+### 4.6 Sleep / Wait Tool
+**What:** Agent can pause execution and wait (e.g., for a build to complete, for a file to change).  
+**How in Claude Code:** `Sleep` tool — waits with early wake capability on user input.  
+**Recommendation:** Built-in `sleep` tool with configurable timeout and wake conditions.
+
+> **Your answer:** ACCEPTED
+
+### 4.7 LSP Integration
+**What:** Language Server Protocol for code intelligence (go-to-definition, find-references, diagnostics).  
+**How in OpenCode:** LSP servers per language, integrated with AI tools. Diagnostics available as a tool.  
+**Recommendation:** Optional LSP support via `settings.json`. Built-in `diagnostics` tool that queries LSP for errors/warnings.
+
+> **Your answer:** ACCEPTED
+
+### 4.8 WebSearch Tool
+**What:** Agent can search the web (not just fetch a URL).  
+**How in Claude Code:** Separate `WebSearch` and `WebFetch` tools.  
+**Recommendation:** Built-in `web_search` tool (uses configurable search API) + existing `web_fetch`.
+
+> **Your answer:** ACCEPTED
+
+---
+
+## 5. Session Stats & Context Optimizations
+
+### Current State (PLAN.md)
+- **No token tracking** — no visibility into costs
+- **No compaction** — sessions grow until they hit context limit, then fail
+- **No tool result clearing** — raw tool outputs accumulate forever
+- **No sub-agent isolation** — one context window for everything
+
+### Required Optimizations
+
+| Optimization | Priority | Description |
+|-------------|----------|-------------|
+| **Token tracking** | P0 | Count input/output/cache tokens per LLM call, aggregate per session |
+| **Auto compaction** | P0 | Summarize at 90% threshold, use cheaper model |
+| **Tool result clearing** | P0 | Clear old tool results after N turns (keep summary only) |
+| **Sub-agent isolation** | P1 | Spawn sub-agents with clean context, return condensed results |
+| **Structured notes** | P1 | TODO/NOTES persisted outside context, survive compaction |
+| **Lazy tool loading** | P1 | ToolSearch pattern, don't load all tool descriptions upfront |
+| **Session indexing** | P1 | SQLite index for fast session lookup without loading messages |
+| **Message pagination** | P2 | API returns messages paginated, not all at once |
+| **Context budget display** | P2 | Show remaining context % in session stats |
+
+---
+
+## 6. Built-in Tools Revision
+
+### Current Plan (14 tools)
+```
+bash, read_file, write_file, edit_file, list_directory, glob,
+web_fetch, memory_write, memory_read, task_create, task_status,
+task_respond, session_spawn, session_message, log_write
+```
+
+### Claude Code v2.1.62 (18 built-in tools)
+```
+Bash, ReadFile, Write, Edit, Glob, Grep, WebFetch, WebSearch,
+NotebookEdit, LSP, Computer, Sleep, AskUserQuestion,
+TodoWrite, Task, TaskCreate, Skill, EnterPlanMode, ExitPlanMode,
+ToolSearch, TeammateTool, SendMessageTool, TeamDelete, EnterWorktree
+```
+
+### OpenCode (12 tools)
+```
+bash, view, write, edit, glob, grep, ls, fetch, sourcegraph,
+diagnostics, agent, context_window
+```
+
+### Recommended KhaBotCLI Built-in Tools
+
+| Tool | Category | Description | Model |
+|------|----------|-------------|-------|
+| `bash` | System | Execute shell commands (with permissions) | Main |
+| `read_file` | Files | Read file contents (with offset/limit) | Main |
+| `write_file` | Files | Write/overwrite file | Cheap (apply) |
+| `edit_file` | Files | Patch specific lines via search-replace | Cheap (apply) |
+| `list_dir` | Files | List directory contents | Main |
+| `glob` | Files | Find files by pattern | Main |
+| `grep` | Files | Search file contents (ripgrep) | Main |
+| `web_fetch` | Network | Fetch URL content | Main |
+| `web_search` | Network | Search the web | Main |
+| `memory_write` | Memory | Write to agent's MEMORY.md | Main |
+| `memory_read` | Memory | Read agent's MEMORY.md | Main |
+| `todo_write` | Memory | Create/update structured TODO list | Main |
+| `todo_read` | Memory | Read current TODO list | Main |
+| `task_create` | Orchestration | Create a new task for any agent | Main |
+| `task_status` | Orchestration | Check task status | Main |
+| `task_respond` | Orchestration | Respond to a waiting task | Main |
+| `agent_spawn` | Orchestration | Spawn a sub-agent with isolated context | Main |
+| `agent_message` | Orchestration | Send message to another agent directly | Main |
+| `tool_search` | Meta | Search available tools by name/description | Cheap |
+| `sleep` | Control | Wait with configurable timeout | — |
+| `ask_user` | Control | Ask the user a question (with options) | Main |
+| `log_write` | Logging | Write entry to task log | Main |
+
+**Key changes from current plan:**
+- **Added:** `grep`, `web_search`, `todo_write`, `todo_read`, `agent_spawn`, `agent_message`, `tool_search`, `sleep`, `ask_user`
+- **Removed:** `session_spawn`, `session_message` (replaced by `agent_spawn`, `agent_message`)
+- **Revised:** `write_file` and `edit_file` can use cheaper "apply" model for the actual file operation
+
+> **Your answer:** ACCEPTED, but no need for `ask_user` tool, We dont have a chat interface, if its a Task Agent it will add a comment to the Task and stop, if its a chat agent it will simply send a message back
+
+### Cheaper Model Delegation (Apply Layer)
+
+Modern AI coding agents use a two-model approach:
+1. **Main model** (e.g., Claude 3.5 Sonnet) — reasoning, planning, deciding what to change
+2. **Apply model** (e.g., Claude Haiku, Morph Fast Apply) — executing the actual file edit
+
+**How it works:**
+- Main model outputs edit intent: "Add error handling to function X in file Y"
+- Apply model receives: instruction + current file content + edit snippet → outputs merged file
+- 10x cheaper, 5x faster for file operations
+
+**Recommendation:** Configurable in settings:
+```json
+{
+  "models": {
+    "main": "anthropic/claude-3.5-sonnet",
+    "apply": "anthropic/claude-3-haiku",
+    "compact": "anthropic/claude-3-haiku",
+    "title": "anthropic/claude-3-haiku"
+  }
+}
+```
+
+> **Your answer:** ACCEPTED
+
+---
+
+## 7. Agent/Skill/Plugin Architecture
+
+### Current Plan Issues
+- Agent types (Chat/Task/Daemon) are too rigid
+- Skills are loaded all at once, no auto-import/remove
+- No sub-agent concept
+- No agent creation at runtime
+- No per-agent model/temperature/tools/permissions
+
+> **Your answer:** WRONG, I have explained above the agent types (Chat/Task/Daemon) and any agent can be a sub-agent, YES Skills should be loaded on demand by agents, unless configured for auto-import, Agents creation at runtime is already supported, a script/user can add a folder for a new agent then simply call it with the API/CLI, since the KhaBotCLI should check for the agent at each run (to check for existance or an update), YES per-agent model/temperature/tools/permissions should be supported
+
+### Claude Code's Architecture (Feb 2026)
+
+**Agent hierarchy:**
+- **Primary agents** — user-facing (Build, Plan)
+- **Sub-agents** — invoked by primary agents (Explore, Task, General)
+- **Hidden system agents** — automatic (Compaction, Title, Summary)
+- **Agent Teams** — multi-agent coordination via TeammateTool/SendMessageTool
+
+**Skills:**
+- Invoked via `Skill` tool
+- Can specify `context: fork` for isolated execution
+- Can specify different model per skill
+- Template variables for arguments
+
+**Key insight:** Skills in Claude Code are "auto-imported" — the agent is told about available skills in the system prompt, but the actual skill content is only loaded when invoked. Skills can be auto-removed from context after execution (fork mode).
+
+### Recommended Architecture for KhaBotCLI
+
+```
+Agent Definition (AGENT.md + frontmatter)
+├── mode: primary | subagent | hidden | all
+├── model: override model for this agent
+├── temperature: 0.0-1.0
+├── steps: max iterations before forced summary
+├── tools: { write: true, bash: false, ... }
+├── permissions: { edit: "ask", bash: { "*": "deny", "npm test": "allow" } }
+└── hidden: true/false (hidden from user, system-only)
+
+Built-in Agents (shipped with KhaBotCLI):
+├── build (primary) — full tool access, default agent
+├── plan (primary) — read-only, analysis and planning
+├── explore (subagent) — fast codebase exploration, read-only
+├── execute (subagent) — full access for delegated tasks
+├── compact (hidden) — context compaction
+├── title (hidden) — session title generation
+└── summary (hidden) — session summary generation
+
+Skills:
+├── Loaded on-demand (names listed in system prompt, content loaded on invoke)
+├── Fork mode = isolated sub-agent context
+├── Main mode = injected into current context
+└── Auto-removed from context after fork execution
+```
+
+---
+
+## 8. System Prompts
+
+### Current Plan Issue
+The plan has zero mention of system prompts beyond "AGENT.md content is loaded." No structure, no conditional fragments, no best practices.
+
+### Claude Code's System Prompt Architecture (110+ fragments)
+
+Claude Code assembles its system prompt from:
+1. **Base system prompt** — core identity, capabilities, safety rules (~3,000 tokens)
+2. **Task-specific fragments** — "avoid over-engineering", "read before modifying", "no premature abstractions", etc. (~30-50 tokens each)
+3. **Tool descriptions** — each built-in tool has its own prompt (100-2,000 tokens each)
+4. **Agent-specific prompts** — different for Explore, Plan, Task sub-agents
+5. **Conditional fragments** — added based on environment (git status, LSP available, hooks configured)
+6. **System reminders** — ~40 mid-conversation reminders injected at specific points
+7. **User's CLAUDE.md** — loaded as part of system prompt
+8. **Utility prompts** — compaction, title generation, summary, etc.
+
+### Recommended for KhaBotCLI
+
+**System prompt assembly order:**
+```
+1. Base system prompt (identity, safety, core rules)
+2. Environment info (OS, shell, cwd, git status)
+3. Agent-specific prompt (AGENT.md content)
+4. Persona (SOUL.md if exists)
+5. Memory (MEMORY.md if exists)
+6. Available tools (names + short descriptions only)
+7. Available skills (names + descriptions only)
+8. Available agents (names + descriptions only)
+9. Task-specific fragments (conditional)
+10. Permission rules summary
+11. Session context (compacted history if resuming)
+```
+
+**Source of system prompts:**
+- **Piebald-AI/claude-code-system-prompts** — open source, all 110+ fragments with token counts
+- **OpenCode source code** — agent prompts in Go source
+- **OpenClaw** — agent prompts
+
+**Action needed:** Create a `./system-prompts/` directory in KhaBotCLI with modular prompt fragments. See [RESEARCH-TODO/deep-research-system-prompts.md].
+
+---
+
+## 9. Image Handling
+
+### Current Plan
+No mention of images whatsoever.
+
+### What's Needed
+
+| Capability | How | Priority |
+|-----------|-----|----------|
+| **Image reading (vision)** | Send images as base64 data URLs in messages. Supported by Claude, GPT-4V, Gemini. | P1 |
+| **Image from URL** | Fetch image from URL, convert to base64, include in context | P1 |
+| **Image from file** | Read local image file, convert to base64, include in context | P1 |
+| **Image generation** | Via MCP tool (DALL-E, Stable Diffusion MCP servers) | P2 |
+| **Image editing** | Via MCP tool | P3 |
+| **Screenshot capture** | Via `Computer` tool or `bash` + screenshot utility | P3 |
+
+**Message format with images:**
+```json
+{
+  "role": "user",
+  "content": [
+    { "type": "text", "text": "What's in this image?" },
+    { "type": "image", "source": {
+      "type": "base64",
+      "media_type": "image/png",
+      "data": "iVBOR..."
+    }}
+  ]
+}
+```
+
+**Task input with images:**
+```json
+{
+  "prompt": "Analyze this screenshot",
+  "files": ["./screenshot.png", "https://example.com/image.jpg"],
+  "images": [
+    { "type": "url", "url": "https://example.com/image.jpg" },
+    { "type": "file", "path": "./screenshot.png" },
+    { "type": "data_url", "data": "data:image/png;base64,iVBOR..." }
+  ]
+}
+```
+
+---
+
+## 10. File/URL Handling
+
+### Current Plan Issue
+Task input `files` field only supports local file paths. Since KhaBotCLI is API-driven, it should support:
+
+| Type | Format | Example |
+|------|--------|---------|
+| Local file | `./path/to/file` | `./src/auth.js` |
+| Absolute path | `/path/to/file` | `/home/user/project/src/auth.js` |
+| HTTP URL | `https://...` | `https://raw.githubusercontent.com/...` |
+| Data URL | `data:...` | `data:text/plain;base64,SGVsbG8=` |
+| File URL | `file:///...` | `file:///home/user/project/src/auth.js` |
+
+**Implementation:** File resolver that:
+1. Detects type from prefix
+2. Fetches content (local read, HTTP GET, base64 decode)
+3. Returns unified `{ path, content, mimeType, size }` object
+4. Respects permissions (deny reading outside project for security)
+
+---
+
+## 11. Agent-to-Agent Communication
+
+### Current Plan
+"Agents do NOT communicate directly. The Task Queue is the message bus." — This is a bottleneck.
+
+### Claude Code's Approach (Feb 2026)
+- **TeammateTool** (1,642 tokens) — manage teams and coordinate teammates in a swarm
+- **SendMessageTool** (1,241 tokens) — send messages to teammates and handle protocol requests/responses
+- **TeamDelete** — remove team members
+- Direct messaging between agents without going through task queue
+
+### Recommended for KhaBotCLI
+
+**Two communication patterns:**
+
+1. **Task-based (async)** — Current plan's approach. Good for: fire-and-forget, long-running delegations.
+   - Agent A creates task for Agent B
+   - Agent A continues or waits
+   - Agent B completes task, result stored
+
+2. **Direct messaging (sync)** — New. Good for: quick queries, coordination, real-time collaboration.
+   - Agent A sends message to Agent B via `agent_message` tool
+   - Agent B receives message in its context
+   - Agent B responds directly
+   - No task queue overhead
+
+**Built-in tools:**
+- `agent_message` — send direct message to another agent, get response
+- `agent_spawn` — spawn a sub-agent with specific task and isolated context
+- `task_create` — create async task (existing)
+
+---
+
+## 12. Loading & Memory Optimizations
+
+### Current Plan Issue
+"Session remains in `~/.khabot/sessions/` indefinitely (never auto-deleted, user manages cleanup)"  
+At 50k sessions × (meta.json + messages.json) = 100k files. This will be extremely slow.
+
+### Recommended Optimizations
+
+| Optimization | Description |
+|-------------|-------------|
+| **SQLite storage** | Single `data.db` for all sessions, tasks, messages. Indexed by session ID, agent name, status, date. |
+| **Lazy loading** | Only load session metadata into memory. Load messages on-demand when session is accessed. |
+| **Active session limit** | Keep max N sessions in memory (configurable, default 50). LRU eviction. |
+| **Finished task cleanup** | Completed/failed tasks older than N days → archived (compressed) or deleted. Configurable retention. |
+| **Message pagination** | API returns messages paginated (default 50 per page). |
+| **Index on startup** | On startup, only scan SQLite index — don't load all session data. |
+| **Compacted message storage** | After compaction, store compacted summary as a single message, delete originals from DB. |
+| **Memory-mapped tool results** | Large tool results (file contents, bash output) stored as references, not inline in message history. |
+
+### Settings Addition
+```json
+{
+  "storage": {
+    "engine": "sqlite",
+    "path": "~/.khabot/data.db",
+    "retention": {
+      "sessions": { "maxAge": "90d", "maxCount": 10000 },
+      "tasks": { "maxAge": "30d", "maxCount": 5000 },
+      "onLimit": "archive"
+    },
+    "memory": {
+      "maxActiveSessions": 50,
+      "maxActiveTasksPerAgent": 10
+    }
+  }
+}
+```
+
+---
+
+## 13. MCP Revision
+
+> **Your answer:** Why not just use a famous MCP client library?
+
+### Current Plan Issues
+- Only mentions `stdio` transport
+- No mention of Streamable HTTP (replaced SSE in March 2025)
+- No authorization support
+- No Elicitation support
+- No mention of MCP Resources or Prompts (only Tools)
+
+### MCP Spec 2025-11-25 Features to Support
+
+| Feature | Description | Priority |
+|---------|-------------|----------|
+| **stdio transport** | Local MCP servers via stdin/stdout | P0 (already planned) |
+| **Streamable HTTP** | Remote MCP servers via HTTP (replaced SSE) | P1 |
+| **Tools** | MCP server provides tools to agents | P0 (already planned) |
+| **Resources** | MCP server provides data/context to agents | P1 |
+| **Prompts** | MCP server provides prompt templates | P2 |
+| **Elicitation** | MCP server can ask user for input | P2 |
+| **Authorization** | OAuth-based auth for remote MCP servers | P2 |
+| **Sampling** | MCP server can request LLM completions | P3 |
+
+### Updated MCP Config
+```json
+{
+  "mcpServers": {
+    "github": {
+      "transport": "stdio",
+      "command": "npx",
+      "args": ["-y", "@modelcontextprotocol/server-github"],
+      "env": { "GITHUB_TOKEN": "" }
+    },
+    "remote-db": {
+      "transport": "streamable-http",
+      "url": "https://mcp.example.com/db",
+      "authorization": {
+        "type": "bearer",
+        "token": "${MCP_DB_TOKEN}"
+      }
+    }
+  }
+}
+```
+
+---
+
+## 14. Task System Revision
+
+### Current Plan Issues
+- No parallel task execution awareness
+- No task dependencies / DAG
+- No task priority
+- No task timeout per-task (only global)
+- No task cancellation propagation (cancel parent → cancel children)
+
+> **Your answer:** Parallel execution/timeouts SHOULD be supported, task dependencies/priority/cancellation i think they should be managed by the Agent who created it right?
+
+### Recommended Updates
+
+| Feature | Description |
+|---------|-------------|
+| **Task priority** | `high`, `normal`, `low` — affects queue ordering |
+| **Task dependencies** | `dependsOn: ["task_xxx"]` — wait for dependencies before starting |
+| **Parent-child tasks** | `parentTaskId` field — sub-tasks linked to parent |
+| **Cancellation propagation** | Cancel parent → cancel all running children |
+| **Per-task timeout** | Override in task creation request |
+| **Parallel execution** | Configurable max concurrent tasks per agent and globally |
+| **Task retry** | Per-task retry config (override global) |
+| **Task tags** | Arbitrary tags for filtering/grouping |
+
+### Updated task.json
+```json
+{
+  "id": "task_xxx",
+  "agentName": "researcher",
+  "status": "processing",
+  "priority": "normal",
+  "parentTaskId": null,
+  "dependsOn": [],
+  "tags": ["research", "quantum"],
+  "input": { "prompt": "...", "files": [], "images": [], "context": {} },
+  "output": null,
+  "tokenUsage": { "inputTokens": 0, "outputTokens": 0, "estimatedCost": 0 },
+  "iterations": 5,
+  "maxIterations": 50,
+  "timeoutSeconds": 300,
+  "createdAt": "...",
+  "startedAt": "...",
+  "finishedAt": null
+}
+```
+
+---
+
+## 15. Daemon Agents Revision
+
+> **Your answer:** I already gave you context on this
+
+### Current Plan Issues
+- Only cron-based triggers
+- `HEARTBEAT_OK` / `HEARTBEAT_ALERT` is simplistic
+- No event-driven triggers
+- No integration with task system events
+
+### Recommended Updates (Aligned with OpenClaw)
+
+| Feature | Description |
+|---------|-------------|
+| **Cron trigger** | Keep existing: `"interval": "*/30 * * * *"` |
+| **Event trigger** | React to system events: `task.completed`, `task.failed`, `session.created`, `agent.error` |
+| **Webhook trigger** | External webhook hits API → triggers daemon |
+| **File watch trigger** | Watch specific files/directories for changes |
+| **Chained triggers** | One daemon can trigger another |
+| **Output routing** | Route alerts to: log, webhook URL, another agent, or Ably channel |
+| **Health metrics** | Track: uptime, tick count, error count, last success/failure |
+
+### Updated HEARTBEAT.md Schema
+```yaml
+---
+triggers:
+  - type: cron
+    interval: "*/30 * * * *"
+  - type: event
+    events: ["task.failed", "agent.error"]
+  - type: file_watch
+    paths: ["./logs/*.log"]
+output:
+  onAlert: 
+    - type: webhook
+      url: "https://hooks.slack.com/..."
+    - type: agent
+      name: "incident-responder"
+  onOk: ignore
+---
+
+# Heartbeat Checklist
+...
+```
+
+---
+
+## 16. Testing Strategy
+
+> **Your answer:** Should be revised after we set on a clear direction
+
+### Current Plan
+No testing mentioned at all.
+
+### Recommended Testing Layers
+
+| Layer | What | How |
+|-------|------|-----|
+| **Unit tests** | Individual tools, file resolver, permission evaluator, settings merger | Standard test framework (Jest/Vitest for TS, Go test for Go) |
+| **Integration tests** | Agent lifecycle, session creation → message → compaction → resume | Test against mock LLM (record/replay responses) |
+| **Agent tests** | Built-in agents produce correct outputs for known inputs | LLM-as-judge scoring rubric |
+| **MCP tests** | MCP server discovery, tool registration, tool execution | Mock MCP servers |
+| **API tests** | All REST endpoints return correct responses | HTTP test client |
+| **Performance tests** | Startup time, 1000 sessions load time, concurrent task handling | Benchmark suite |
+| **E2E tests** | Full scenario: start server → create agent → run task → check output | Scripted test scenario |
+
+### CI Pipeline
+```
+1. Lint + type check
+2. Unit tests
+3. Integration tests (mock LLM)
+4. Build
+5. E2E tests (against built binary, with mock LLM)
+6. Performance benchmarks (compare against baseline)
+7. Package + publish (on tag)
+```
+
+### Test Script Structure
+```
+tests/
+├── unit/
+│   ├── tools/
+│   ├── permissions/
+│   ├── settings/
+│   └── storage/
+├── integration/
+│   ├── agent-lifecycle/
+│   ├── session-management/
+│   ├── task-queue/
+│   └── mcp/
+├── e2e/
+│   ├── scenarios/
+│   └── fixtures/
+├── performance/
+│   └── benchmarks/
+└── fixtures/
+    ├── mock-llm-responses/
+    └── test-agents/
+```
+
+---
+
+## 17. Documentation Plan
+
+> **Your answer:** Should be revised after we set on a clear direction
+
+### Current Plan
+No documentation mentioned.
+
+### Recommended Documentation Structure
+
+```
+docs/
+├── getting-started.md          — Quick start guide
+├── installation.md             — Install methods (npm, binary, source)
+├── configuration.md            — All settings explained
+├── agents/
+│   ├── creating-agents.md      — How to create custom agents
+│   ├── built-in-agents.md      — Built-in agents reference
+│   ├── sub-agents.md           — Sub-agent patterns
+│   └── skills.md               — Skills/commands reference
+├── tools/
+│   ├── built-in-tools.md       — All built-in tools reference
+│   └── mcp-tools.md            — Using MCP servers as tools
+├── api/
+│   ├── reference.md            — Full API reference
+│   └── examples.md             — API usage examples
+├── concepts/
+│   ├── context-management.md   — Compaction, notes, sub-agents
+│   ├── permissions.md          — Permission system
+│   ├── sessions-tasks.md       — Session/task lifecycle
+│   └── daemon-agents.md        — Daemon patterns
+├── guides/
+│   ├── migration-from-claude.md
+│   ├── forking.md              — How to fork KhaBotCLI
+│   └── testing-agents.md       — How to test your agents
+└── contributing.md             — How to contribute
+```
+
+**Auto-generation:** JSON schemas → API docs, tool descriptions → tool reference, agent frontmatter → agent reference.
+
+---
+
+## 18. Forkable Config & Branding
+
+> **Your answer:** Should be revised after we set on a clear direction
+
+### Current Plan Issue
+Domain "KhaBot.com" is hardcoded throughout. No config for forking.
+
+### Recommended Structure
+
+```
+config/
+├── branding.json              — All brand-specific values
+├── defaults/
+│   ├── agents/                — Default agents shipped with the product
+│   │   ├── build/
+│   │   │   └── AGENT.md
+│   │   ├── plan/
+│   │   │   └── AGENT.md
+│   │   └── explore/
+│   │       └── AGENT.md
+│   ├── skills/                — Default skills
+│   ├── tools/                 — Default tool configs
+│   └── system-prompts/        — System prompt fragments
+└── schemas/                   — JSON schemas for validation
+    ├── settings.schema.json
+    ├── agent.schema.json
+    ├── task.schema.json
+    └── session.schema.json
+```
+
+**branding.json:**
+```json
+{
+  "productName": "KhaBotCli",
+  "productNameShort": "khabot",
+  "domain": "khabot.com",
+  "apiBaseUrl": "https://api.khabot.com",
+  "configDir": ".khabot",
+  "globalConfigDir": "~/.khabot",
+  "schemaBaseUrl": "https://khabot.com/schemas",
+  "defaultPort": 5050,
+  "authHeader": "X-KhaBot-Secret",
+  "tokenPrefix": "kbt_",
+  "sessionIdPrefix": "sess_",
+  "taskIdPrefix": "task_",
+  "logo": "🤖",
+  "description": "Locally-running autonomous agent runtime"
+}
+```
+
+**Every reference** to "KhaBot", "khabot.com", ".khabot", "kbt_", "X-KhaBot-Secret", etc. should use values from `branding.json`.
+
+---
+
+## 19. Diagrams & Overview
+
+The following diagrams need to be created (as Mermaid or ASCII in the plan):
+
+### 19.1 Architecture Overview
+```
+┌─────────────────────────────────────────────────┐
+│                   KhaBotCLI                      │
+├─────────────┬──────────────┬────────────────────┤
+│  CLI Layer  │  API Layer   │  Ably/Remote Layer │
+├─────────────┴──────────────┴────────────────────┤
+│                 Agent Runtime                    │
+│  ┌─────────┐ ┌──────────┐ ┌──────────────────┐ │
+│  │ Session  │ │  Task    │ │  Daemon Manager  │ │
+│  │ Manager  │ │  Queue   │ │  (cron/events)   │ │
+│  └────┬─────┘ └────┬─────┘ └────────┬─────────┘ │
+│       └─────────────┼────────────────┘           │
+│              ┌──────┴──────┐                     │
+│              │  Agent Core │                     │
+│              │  (LLM loop) │                     │
+│              └──────┬──────┘                     │
+│       ┌─────────────┼─────────────┐              │
+│  ┌────┴────┐  ┌─────┴─────┐ ┌────┴────┐        │
+│  │ Built-in│  │    MCP    │ │ Context │        │
+│  │  Tools  │  │  Servers  │ │ Manager │        │
+│  └─────────┘  └───────────┘ └─────────┘        │
+├──────────────────────────────────────────────────┤
+│              Storage Layer (SQLite)              │
+│  Sessions │ Tasks │ Messages │ Agent Index       │
+└──────────────────────────────────────────────────┘
+```
+
+### 19.2 Agent Lifecycle, Session Lifecycle, Task Lifecycle, Data Flow
+→ To be created in `PLAN/OVERVIEW-DIAGRAM.md`
+
+---
+
+## 20. Schema/JSON Revisions
+
+| Schema | Changes Needed |
+|--------|---------------|
+| **settings.json** | Add: `models` (main/apply/compact/title), `compaction`, `storage`, `branding` reference. Update: `mcpServers` (add transport type). Simplify: `hooks` (just PreToolUse/PostToolUse). |
+| **auth.json** | Simplify to global-only. Add: `providers` object for multi-provider support. |
+| **AGENT.md frontmatter** | Add: `mode`, `steps`, `temperature`, `tools`, `permissions`, `hidden`. |
+| **Session (in SQLite)** | Add: `tokenUsage`, `compactionCount`, `parentSessionId`, `subAgentOf`. |
+| **Task (in SQLite)** | Add: `tokenUsage`, `priority`, `parentTaskId`, `dependsOn`, `tags`, per-task timeout. |
+| **tool.json** | Deprecate. Replace with MCP server configs. |
+| **HEARTBEAT.md** | Add: `triggers` (cron/event/webhook/file_watch), `output` routing config. |
+
+---
+
+## 21. Plan Structure Proposal
+
+The current `PLAN.md` is a single 1,176-line file. Proposed split into `./PLAN/` folder:
+
+```
+PLAN/
+├── 00-overview.md              — Vision, identity, philosophy, architecture diagram
+├── 01-cli-modes.md             — CLI commands and modes of operation
+├── 02-folder-structure.md      — Project and global folder structure
+├── 03-agents.md                — Agent types, lifecycle, sub-agents, built-in agents
+├── 04-tools.md                 — Built-in tools, MCP tools, apply layer, tool search
+├── 05-context-management.md    — Compaction, notes, token tracking, sub-agent isolation
+├── 06-sessions-tasks.md        — Session/task lifecycle, storage, stats, optimization
+├── 07-system-prompts.md        — System prompt architecture, fragments, assembly
+├── 08-api.md                   — REST API specification
+├── 09-mcp.md                   — MCP support (stdio, Streamable HTTP, resources, prompts)
+├── 10-permissions-hooks.md     — Permission system, hooks
+├── 11-daemon-agents.md         — Daemon triggers, health, output routing
+├── 12-agent-communication.md   — Task-based + direct messaging patterns
+├── 13-images-files.md          — Image handling, file/URL resolution
+├── 14-config-branding.md       — Forkable config, branding.json
+├── 15-testing.md               — Testing strategy, CI pipeline
+├── 16-documentation.md         — Docs structure, auto-generation
+├── 17-local-defaults.md        — Pre-packaged agents, skills, tools, system prompts
+├── 18-schemas.md               — All JSON schemas (settings, agent, task, session)
+├── 19-compatibility.md         — Claude Code, OpenClaw import/compatibility
+├── 20-failure-recovery.md      — Error handling, retry, startup recovery
+└── OVERVIEW-DIAGRAM.md         — All architecture diagrams (Mermaid)
+```
+
+---
+
+## 22. Research Items Needed
+
+These items require deeper investigation. Files have been created in `./RESEARCH-TODO/` and `./SCOPED-QUESTIONS/`.
+
+### RESEARCH-TODO (for Deep Research Agent)
+1. **Context engineering deep dive** — compaction algorithms, optimal prompts, when to compact vs note-take vs spawn sub-agent
+2. **Agent orchestration patterns** — detailed comparison with recommendations for KhaBotCLI
+3. **Testing AI agents** — scoring rubrics, LLM-as-judge, regression detection
+4. **Apply layer / cheaper models** — Morph, speculative editing, search-replace vs diff strategies
+5. **System prompt best practices** — modular architecture, conditional fragments, token budgets
+
+### SCOPED-QUESTIONS (for context-free AI chat)
+1. Agent types unification (Chat/Task/Daemon → invocation modes?)
+2. Session storage at scale (files vs SQLite vs hybrid?)
+3. Skill auto-loading patterns
+4. System prompt modular architecture
+5. MCP-only vs custom tool scripts
+
+---
+
+## Next Steps
+
+1. **You answer the 12 questions** in Section 1
+2. **You send RESEARCH-TODO files** to your Deep Research Agent
+3. **You send SCOPED-QUESTIONS files** to your scoped AI chat
+4. **I receive answers** and proceed to Step 2: Schema/JSON revision + Plan rewrite