khabotcli 1.0.0

package/PLAN/09-permissions.md

@@ -0,0 +1,108 @@
+# 09 — Permissions & Hooks
+
+## Permission System
+
+Adopted from Claude Code. Evaluated in order: **deny → ask → allow**.
+
+### Permission Format
+```
+tool_name(pattern)
+
+Examples:
+  bash(npm run *)          ← bash commands starting with "npm run"
+  read_file(**)            ← read any file recursively
+  write_file(./src/**)     ← write anywhere in src/
+  edit_file(./src/**)      ← edit anywhere in src/
+  web_fetch(https://api.*) ← fetch from api.* domains
+```
+
+**Tool names are snake_case** — must match exactly. All built-in tools use snake_case (`bash`, `read_file`, `write_file`, `web_fetch`, etc.).
+
+### MCP Tool Permissions
+
+MCP tools use the format `mcp:<server>.<tool>`:
+```
+  mcp:filesystem.read_file(**)     ← MCP filesystem server's read_file
+  mcp:database.query(SELECT *)     ← MCP database server's query tool
+  mcp:*.*(**)                       ← all MCP tools (wildcard)
+```
+
+### Defaults
+- **Always denied:** `read_file(./.khabot/auth.json)`, `read_file(~/.khabot/auth.json)`
+- **Interactive mode (chat):** default `ask` — agent requests permission, user approves
+- **Non-interactive mode (task, daemon, subagent, one-shot):** default `deny` — must be explicitly allowed
+
+### Configuration Layers
+
+Permissions resolved in order (last wins):
+```
+1. Built-in defaults (auth.json always denied)
+2. ~/.khabot/settings.json → permissions
+3. ./.khabot/settings.json → permissions
+4. ./.khabot/settings.local.json → permissions
+5. agent.json → modes.<mode>.permissions   ← per-agent, per-mode overrides
+```
+
+### Interaction with `disallowedTools`
+
+The permission system and `disallowedTools` in agent.json are complementary:
+- **`disallowedTools`** — removes the tool entirely from the agent's toolset (LLM never sees it)
+- **Permissions** — controls whether a visible tool can actually execute (deny/ask/allow)
+
+Resolution order: `disallowedTools` checked first (tool removed) → then permissions checked (tool allowed/denied).
+
+Per-agent, per-mode permissions in agent.json support `allow` and `deny` only. The `ask` level is only available in settings.json because it requires interactive user approval, which is a global behavior — not agent-specific.
+
+Example:
+```json
+{
+  "modes": {
+    "task": {
+      "permissions": {
+        "allow": ["bash(npm run *)", "read_file(**)", "write_file(./src/**)"],
+        "deny": ["bash(rm *)"]
+      }
+    }
+  }
+}
+```
+
+---
+
+## Hooks
+
+Two hooks only: **PreToolUse** and **PostToolUse**. Shell commands defined in settings.json.
+
+### Configuration
+```json
+{
+  "hooks": {
+    "PreToolUse": "node .khabot/hooks/pre-tool.js",
+    "PostToolUse": "node .khabot/hooks/post-tool.js"
+  }
+}
+```
+
+### Environment Variables
+Hooks receive context via environment variables:
+
+| Variable | Description |
+|----------|-------------|
+| `KHABOT_TOOL_NAME` | Tool being called (e.g., `bash`, `write_file`) |
+| `KHABOT_TOOL_INPUT` | JSON-encoded tool input |
+| `KHABOT_TOOL_OUTPUT` | JSON-encoded tool output (PostToolUse only) |
+| `KHABOT_AGENT_NAME` | Agent calling the tool |
+| `KHABOT_SESSION_ID` | Current session ID |
+| `KHABOT_TASK_ID` | Current task ID (if in task mode) |
+| `KHABOT_MODE` | Current mode: chat/task/daemon/subagent |
+| `KHABOT_AGENT_FOLDER` | Absolute path to agent's folder |
+
+### Behavior
+- **PreToolUse:** Exit 0 → allow tool execution. Exit 1 → block (tool returns error to LLM).
+- **PostToolUse:** Exit 0 → continue. Exit 1 → log warning (tool already executed).
+- **Timeout:** 10s default. Hook killed on timeout, tool proceeds.
+
+### Use Cases
+- `PreToolUse` where `KHABOT_TOOL_NAME=bash` → validate bash commands before execution
+- `PostToolUse` where `KHABOT_TOOL_NAME=write_file` → trigger lint/format after file write
+- `PreToolUse` → log all tool calls to external audit system

package/PLAN/10-ably.md

@@ -0,0 +1,105 @@
+# 10 — Ably Remote Access
+
+## Overview
+
+When enabled, KhaBotCLI subscribes to an Ably channel and relays REST API calls over it. This allows remote consumers (web app, mobile app, KhaBot.com dashboard) to interact with a local KhaBotCLI instance without port forwarding.
+
+```
+[Remote Client] → [KhaBot.com] → [Ably Channel] → [KhaBotCLI (local)]
+                                                          ↓
+                                                    REST API handler
+                                                          ↓
+                                                    [Ably Channel] → [KhaBot.com] → [Remote Client]
+```
+
+## Channel Architecture
+
+**One channel per KhaBotCLI instance** (not per agent or session).
+
+- Channel name: `khabot:instance:<instanceId>`
+- `instanceId` assigned by KhaBot.com on registration
+- Namespace `khabot:` allows applying Ably channel rules globally
+
+**Message name routing** on the single channel:
+- `api:request` — remote client → KhaBotCLI
+- `api:response` — KhaBotCLI → remote client
+- `task:progress` — KhaBotCLI pushes task progress events
+- `system:ping` / `system:pong` — health checks
+
+## Message Format (RPC over Pub/Sub)
+
+Request/response via **correlation IDs** (`requestId`).
+
+**Request:**
+```json
+{
+  "requestId": "req_a1b2c3d4",
+  "method": "POST",
+  "path": "/agents/researcher/task",
+  "headers": { "X-KhaBot-Secret": "mysecret" },
+  "body": { "input": "Research topic X" },
+  "timestamp": 1740793200000,
+  "timeout": 30000
+}
+```
+
+**Response:**
+```json
+{
+  "requestId": "req_a1b2c3d4",
+  "status": 200,
+  "body": { "taskId": "task_xyz789", "status": "pending" },
+  "timestamp": 1740793201500,
+  "duration": 1500
+}
+```
+
+## Authentication Flow
+
+1. KhaBotCLI starts with `ably.enabled: true`
+2. Creates Ably Realtime client with `authCallback`
+3. `authCallback` calls KhaBot.com: `POST /api/v1/ably/token` with `khabot_token`
+4. KhaBot.com validates token, issues scoped Ably TokenRequest (1h TTL)
+5. Ably SDK handles automatic token refresh before expiry
+6. KhaBotCLI enters presence on its channel to signal "online"
+
+**Remote clients never connect to Ably directly.** KhaBot.com relays on their behalf.
+
+## Reliability
+
+| Offline Duration | Ably State | Behavior |
+|-----------------|------------|----------|
+| 0–2 min | `disconnected` | Auto-reconnect. Missed messages replayed. |
+| >2 min | `suspended` | Messages during gap are lost. Re-enter presence on reconnect. |
+| Extended | `failed` | Auth token expired. Log error, await manual reconnect. |
+
+**Stale request handling:** Requests with `timestamp` older than `timeout` are skipped — remote client already timed out.
+
+## Limits
+
+- **Message size:** 64 KiB. Responses >60 KiB truncated with `{ "truncated": true, "fetchUrl": "/messages/msg_xyz/full" }`.
+- **Rate:** 50 msg/s per channel (sufficient for KhaBotCLI's use case).
+- **Latency overhead:** ~100-300ms round-trip. Negligible vs LLM call time.
+
+## Security
+
+Even though KhaBotCLI trusts KhaBot.com:
+- Validate incoming message schema (required fields, method whitelist)
+- Local `X-KhaBot-Secret` still required in relayed requests
+- Track recent `requestId`s to reject duplicates
+- Permission system still applies to all tool calls via Ably
+- Optional: AES-256-CBC per-channel encryption for high-security deployments
+
+## Settings
+
+```json
+{
+  "ably": {
+    "enabled": false,
+    "instanceId": "",
+    "tokenEndpoint": "https://khabot.com/api/v1/ably/token",
+    "requestTimeout": 30000,
+    "maxRequestAge": 60000
+  }
+}
+```

package/PLAN/11-file-formats.md

@@ -0,0 +1,442 @@
+# 11 — File Formats
+
+## agent.json
+
+Agent configuration. Defines modes, model, tools, permissions.
+
+> **Critical field types — common mistakes:**
+> - `model` must be a **string** (e.g. `"moonshotai/kimi-k2.5"`), NOT an object like `{ "role": "main" }`
+> - Each `modes.*` entry must be an **object** with at minimum `{ "enabled": boolean }`, NOT a bare boolean like `true`
+> - Both of these were real bugs found during testing that caused every agent to fail validation
+
+```json
+{
+  "name": "researcher",
+  "description": "Deep research agent that investigates topics",
+  "model": "anthropic/claude-3.5-sonnet",
+  "temperature": 0.7,
+  "reasoning": "medium",
+  "memory": { "enabled": true, "maxLines": 200 },
+  "skillDiscovery": true,
+  "modes": {
+    "chat": {
+      "enabled": true,
+      "tools": [],
+      "disallowedTools": [],
+      "skills": [],
+      "autoLoadSkills": [],
+      "mcpServers": [],
+      "permissions": {}
+    },
+    "task": {
+      "enabled": true,
+      "tools": [],
+      "disallowedTools": ["bash"],
+      "skills": [],
+      "autoLoadSkills": [],
+      "mcpServers": [],
+      "maxIterations": 50,
+      "maxDurationSeconds": 300,
+      "permissions": {}
+    },
+    "subagent": {
+      "enabled": true,
+      "tools": ["web_search", "read_file", "grep"],
+      "disallowedTools": [],
+      "skills": ["summarize"],
+      "autoLoadSkills": [],
+      "mcpServers": [],
+      "allowedAgents": [],
+      "disallowedAgents": [],
+      "maxIterations": 20,
+      "maxDurationSeconds": 120,
+      "permissions": {}
+    },
+    "daemon": {
+      "enabled": false,
+      "cron": "*/5 * * * *",
+      "triggers": [],
+      "conflictPolicy": "skip",
+      "alertRouting": ["log", "webhook:https://slack.com/webhook/xxx"],
+      "heartbeatFile": "",
+      "tools": [],
+      "disallowedTools": [],
+      "skills": [],
+      "autoLoadSkills": [],
+      "mcpServers": [],
+      "permissions": {}
+    }
+  }
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | string | Agent identifier (overrides folder name) |
+| `description` | string | Human-readable description |
+| `model` | string | Default model for all modes |
+| `temperature` | number | 0.0–1.0 |
+| `reasoning` | string | Enable reasoning mode for this agent (default: "medium"). Options: "low", "medium", "high" |
+| `memory.enabled` | boolean | Enable persistent memory for this agent (default: true) |
+| `memory.maxLines` | number | MEMORY.md line cap before auto-export (default: 200 from settings.json) |
+| `skillDiscovery` | boolean | List skill names in system prompt (default: true). When false, agent is skill-blind. |
+| `modes.chat.enabled` | boolean | Whether chat mode is supported |
+| `modes.task.enabled` | boolean | Whether task mode is supported |
+| `modes.task.maxIterations` | number | Override global limit |
+| `modes.task.maxDurationSeconds` | number | Override global limit |
+| `modes.subagent.enabled` | boolean | Whether sub-agent mode is supported |
+| `modes.subagent.maxIterations` | number | Limit when called as sub-agent |
+| `modes.subagent.maxDurationSeconds` | number | Limit when called as sub-agent |
+| `modes.subagent.allowedAgents` | string[] | Agents this sub-agent can spawn/message (empty = all) |
+| `modes.subagent.disallowedAgents` | string[] | Agents this sub-agent cannot spawn/message |
+| `modes.daemon.enabled` | boolean | Whether daemon mode is supported |
+| `modes.daemon.cron` | string | Cron expression for scheduling |
+| `modes.daemon.triggers` | string[] | System-wide event triggers: `task.completed`, `task.failed`, `agent.error`. See **Daemon Trigger Payload** below. |
+| `modes.daemon.conflictPolicy` | string | skip / queue / restart |
+| `modes.daemon.alertRouting` | string \| string[] | Array of routing targets: `"log"`, `"webhook:<url>"`, `"agent:<name>"`, `"ably:<channel>"`. Single string also accepted. |
+| `modes.daemon.heartbeatFile` | string | Custom path to heartbeat file (default: `.khabot/heartbeats/<name>.md`) |
+| `modes.*.tools` | string[] | Tool whitelist (empty = all) |
+| `modes.*.disallowedTools` | string[] | Tool blacklist (mutually exclusive with `tools`) |
+| `modes.*.skills` | string[] | Skill whitelist (empty = all) |
+| `modes.*.autoLoadSkills` | string[] | Skills always loaded in full |
+| `modes.*.mcpServers` | string[] | MCP server whitelist (empty = all configured) |
+| `modes.*.permissions` | object | `{ allow: [], deny: [] }` per-mode overrides |
+
+**Minimal valid example** (guaranteed to pass schema validation):
+```json
+{
+  "name": "hello",
+  "description": "A minimal reference agent",
+  "model": "anthropic/claude-3-haiku",
+  "temperature": 0.5,
+  "reasoning": null,
+  "modes": {
+    "chat": { "enabled": true },
+    "task": { "enabled": false },
+    "subagent": { "enabled": false },
+    "daemon": { "enabled": false }
+  }
+}
+```
+
+This example is the basis for `examples/agents/hello/agent.json` in the source tree — it is validated in CI and used as the reference for documentation.
+
+---
+
+## AGENT.md
+
+Plain markdown. Agent instructions and system prompt content. No configuration — that's in agent.json.
+
+```markdown
+You are a research agent. Your job is to investigate topics thoroughly
+and produce detailed, well-sourced reports.
+
+## Rules
+- Always cite your sources
+- Prefer primary sources over summaries
+- If you can't find reliable information, say so
+```
+
+---
+
+## SOUL.md
+
+Optional. Plain markdown. Persona and behavioral identity. Loaded after AGENT.md.
+
+```markdown
+You speak in a concise, professional tone.
+You never apologize unnecessarily.
+You prefer bullet points over long paragraphs.
+```
+
+---
+
+## Memory Files
+
+Persistent memory lives in `.khabot/memory/`, NOT in agent folders (agent folders are static/read-only).
+
+```
+.khabot/memory/
+├── MEMORY.md                ← global memory (always loaded, ~200-line cap)
+├── <topic>.md               ← auto-exported global memory files
+└── agents/
+    └── <agentName>/
+        ├── MEMORY.md            ← agent-specific memory (always loaded, ~200-line cap)
+        └── <topic>.md           ← auto-exported agent memory files
+```
+
+**Main MEMORY.md** is always loaded into the system prompt (Layer 4). Capped at ~200 lines (configurable).
+**Auto-export:** When MEMORY.md exceeds the line cap, the compact model moves older entries into a separate `<topic>.md` file.
+**Memory tools:** `memory_write` (append), `memory_read` (read), `memory_search` (search across all memory files).
+**System prompt includes:** Main MEMORY.md content + list of available memory files with descriptions.
+**Pre-compaction:** Before context compaction, runtime asks model to extract key learnings → writes to MEMORY.md (model can return `NO_MEMORY` to skip).
+
+```markdown
+# Memory
+
+## User Preferences
+- Prefers reports in markdown format
+- Works in EST timezone
+
+## Known Facts
+- Main project is called ProjectX
+- Production server is at 192.168.1.100
+```
+
+---
+
+## Heartbeat Files
+
+Daemon checklists live in `.khabot/heartbeats/`, NOT in agent folders.
+
+```
+.khabot/heartbeats/
+└── <agentName>.md       ← per-agent heartbeat checklist
+```
+
+Plain markdown. Read on each daemon tick. Agent.json `heartbeatFile` field can override the default path.
+
+```markdown
+# Heartbeat Checklist
+
+- Check if any tasks have been in "processing" state for more than 1 hour
+- Monitor ./logs/ folder for new ERROR lines since last check
+- If it's Monday morning, generate a weekly summary
+```
+
+**Output contract:** Agent MUST output `HEARTBEAT_OK` or `HEARTBEAT_ALERT: <message>`.
+
+### Daemon Trigger Payload
+
+When a daemon is triggered by a system event (not cron), it receives context via a system message:
+
+```
+[System] Daemon triggered by event: task.completed
+{
+  "event": "task.completed",
+  "taskId": "task_abc123",
+  "agentName": "researcher",
+  "status": "finished",
+  "output": { ... },
+  "timestamp": "2026-03-01T08:00:00Z"
+}
+```
+
+The daemon can then decide what action to take based on the event payload.
+
+---
+
+## SKILL.md
+
+Markdown with YAML frontmatter. Prompt template invokable by agents.
+
+```markdown
+---
+name: review-code
+description: Review code for security and performance issues
+argument-hint: [file-path]
+allowed-tools: bash, read_file, edit_file
+context: fork
+agent: anthropic/claude-3-haiku
+disable-model-invocation: false
+---
+
+Review the following file: $1
+
+Focus on:
+1. Security vulnerabilities
+2. Performance bottlenecks
+3. Code style issues
+
+Additional context: $ARGUMENTS
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `name` | string | Skill identifier |
+| `description` | string | Human-readable description |
+| `argument-hint` | string | Hint shown for usage |
+| `allowed-tools` | string | Comma-separated tool whitelist |
+| `context` | main / fork | `main` = inject into conversation. `fork` = isolated sub-agent. |
+| `agent` | string | Model to use (fork mode only). Named `agent` for Claude Code heritage; functionally equivalent to `model`. |
+| `disable-model-invocation` | boolean | Only user/API can trigger this skill, not the agent itself (default: false) |
+
+---
+
+## tool.json
+
+Custom tool manifest.
+
+```json
+{
+  "name": "send_slack_message",
+  "description": "Sends a message to a Slack channel",
+  "input_schema": {
+    "type": "object",
+    "properties": {
+      "channel": { "type": "string", "description": "Slack channel name" },
+      "message": { "type": "string", "description": "Message to send" }
+    },
+    "required": ["channel", "message"]
+  },
+  "timeout": 30
+}
+```
+
+---
+
+## config.json (Forkable, Source Code Only)
+
+All branding/naming values. This is a **source code file** inside the KhaBotCLI repository (`config/config.json`), NOT a user-facing config file. Anyone forking KhaBotCLI changes only this file to rebrand.
+
+```json
+{
+  "productName": "KhaBotCli",
+  "productNameShort": "khabot",
+  "domain": "khabot.com",
+  "configDir": ".khabot",
+  "globalConfigDir": "~/.khabot",
+  "defaultPort": 5050,
+  "authHeader": "X-KhaBot-Secret",
+  "tokenPrefix": "kbt_",
+  "sessionIdPrefix": "sess_",
+  "taskIdPrefix": "task_"
+}
+```
+
+---
+
+## auth.json
+
+Local authentication. Gitignored. The `llm` section provides **default** LLM credentials. If `settings.json` also defines `models.main.base_url` / `models.main.api_key`, the settings.json values take precedence per model role.
+
+```json
+{
+  "khabot_token": "kbt_xxxxxxxxxxxx",
+  "llm": {
+    "base_url": "https://openrouter.ai/api/v1",
+    "api_key": "sk-or-xxxxxxxxxxxx"
+  }
+}
+```
+
+---
+
+## settings.json
+
+Full configuration. Merged across layers (global → project → local → CLI flags).
+
+> **Location:** Project settings go in `.khabot/settings.json` — NOT in the project root as `settings.json`. A common mistake is placing settings at `./settings.json` which is never read by the runtime.
+
+> **Canonical field names:** The LLM credential fields are `base_url` and `api_key` (snake_case). Do NOT use `apiEndpoint`, `apiKey`, or other variants. These names match the fields in `auth.json` and are exported as constants from `src/settings/fields.js` to prevent inconsistency across the codebase.
+
+```json
+{
+  "models": {
+    "main": {
+      "model": "anthropic/claude-3.5-sonnet",
+      "base_url": "",
+      "api_key": ""
+    },
+    "compact": {
+      "model": "anthropic/claude-3-haiku"
+    },
+    "title": {
+      "model": "anthropic/claude-3-haiku"
+    }
+  },
+  "permissions": {
+    "allow": [],
+    "ask": [],
+    "deny": []
+  },
+  "hooks": {
+    "PreToolUse": "",
+    "PostToolUse": ""
+  },
+  "mcpServers": {
+    "filesystem": {
+      "transport": "stdio",
+      "command": "npx",
+      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/project"]
+    },
+    "web-search": {
+      "transport": "http",
+      "url": "http://localhost:3001/mcp"
+    }
+  },
+  "api": {
+    "port": 5050,
+    "secret": ""
+  },
+  "compaction": {
+    "threshold": 0.85,
+    "enabled": true,
+    "observationMaskingTurns": 10
+  },
+  "memory": {
+    "enabled": true,
+    "maxLines": 200,
+    "preCompactionExtraction": true
+  },
+  "storage": {
+    "retention": {
+      "sessions": { "maxAgeDays": 90, "maxCount": 10000 },
+      "tasks": { "maxAgeDays": 30, "maxCount": 5000 }
+    }
+  },
+  "limits": {
+    "maxIterations": 50,
+    "maxDurationSeconds": 300,
+    "maxConcurrentTasks": 5,
+    "maxSubAgentDepth": 3
+  },
+  "retry": {
+    "maxAttempts": 3,
+    "delaySeconds": 5,
+    "onExhausted": "fail"
+  },
+  "ably": {
+    "enabled": false,
+    "instanceId": "",
+    "tokenEndpoint": "https://khabot.com/api/v1/ably/token",
+    "requestTimeout": 30000,
+    "maxRequestAge": 60000
+  }
+}
+```
+
+**`onExhausted` valid values:** `"fail"` (default — task fails) or `"wait"` (task pauses in `waiting` status for human retry via `task_respond`).
+
+Settings resolution order (last wins):
+```
+1. KhaBotCLI hardcoded defaults
+2. ~/.khabot/settings.json
+3. ./.khabot/settings.json
+4. ./.khabot/settings.local.json    ← same format as settings.json, partial overrides merged
+5. CLI flags
+```
+
+`settings.local.json` uses the same schema as `settings.json`. Only the fields present are merged — omitted fields inherit from the previous layer. This file is gitignored and intended for developer-specific overrides (e.g., different API keys, ports).
+
+**Settings field names are defined as constants in `src/settings/fields.js`** and used everywhere in the runtime code. Never hardcode field name strings (`'api_key'`, `'base_url'`) inline — always import from `fields.js`. This prevents the class of bugs where one module reads `settings.apiKey` and another writes `settings.api_key`.
+
+---
+
+## version.json
+
+Present **only** in globally cached agents/tools from the KhaBot.com registry (`~/.khabot/agents/<name>/`, `~/.khabot/tools/<name>/`). Project-level agents/tools (`.khabot/agents/`, `.khabot/tools/`) do NOT have this file — they are user-owned and not registry-managed.
+
+```json
+{
+  "hash": "sha256:a1b2c3d4e5f6...",
+  "version": "1.2.0",
+  "updatedAt": "2026-03-01T08:00:00Z"
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `hash` | string | SHA256 hash of agent/tool contents |
+| `version` | string | Version from KhaBot.com registry |
+| `updatedAt` | string | ISO 8601 timestamp of last update |