khabotcli 1.0.0

package/.OLD/STEP-2/SCOPED-QUESTIONS/scoped-sqlite-schema.md

@@ -0,0 +1,53 @@
+# Scoped Question: SQLite Schema Design for KhaBotCLI
+
+## Context
+KhaBotCLI is a locally-running autonomous agent runtime exposed as a REST API. It manages AI agents (Chat, Task, Daemon types) and needs persistent storage for runtime data.
+
+## Constraints
+- **Engine:** SQLite (single file: `~/.khabot/data.db`)
+- **Language:** Node.js + JSDoc (no TypeScript)
+- **Library:** `better-sqlite3` (synchronous, fast, zero-config)
+- **Config stays in JSON/Markdown files** — SQLite is for runtime data only
+
+## What Needs Storing
+
+### Sessions
+- Each chat message or task execution creates a session
+- Session metadata: id, agent_name, agent_type (chat|task|daemon), instance_folder, status (active|closed), model, message_count, token_usage (input/output/cache), estimated_cost, compaction_count, created_at, updated_at
+
+### Messages
+- Append-only (never update, only INSERT or DELETE during compaction)
+- Fields: session_id, role (system|user|assistant|tool), content (text or JSON for multipart), tool_calls (JSON array), tool_call_id, token_count, created_at
+- Content can be large (full file reads, bash outputs)
+
+### Tasks
+- Fields: id, session_id, agent_name, status (pending|processing|waiting|finished|failed|canceled), priority (high|normal|low), parent_task_id, input (JSON), output (JSON), tags (JSON array), iterations, max_iterations, timeout_seconds, token_usage, estimated_cost, error, created_at, started_at, finished_at, updated_at
+
+### Task Logs
+- Append-only iteration logs
+- Fields: task_id, level (info|warn|error|iteration), content, created_at
+
+### TODO State
+- Per-session TODO lists that survive compaction
+- Fields: session_id, items (JSON array of {id, content, status, priority})
+
+## Questions to Answer
+
+1. **Complete schema** with all tables, columns, types, and constraints
+2. **Indexes** — which queries are most common and need optimization?
+3. **Migration strategy** — numbered SQL files, schema_version table, additive-only rule
+4. **Query patterns** for common API endpoints:
+   - `GET /sessions` (list with filters)
+   - `GET /sessions/:id` (with messages)
+   - `GET /tasks` (list with filters)
+   - `GET /tasks/:id`
+   - `POST /tasks/:id/respond`
+   - Cleanup queries (age-based, count-based)
+5. **Large content handling** — should large tool outputs be stored inline or as references?
+6. **Concurrent access patterns** — multiple agents writing to the same DB simultaneously
+7. **Recommended PRAGMAs** for performance and safety
+8. **Backup strategy** — hot backup approach
+9. **Archive format** — JSONL export for old data
+
+## Starting Point
+See the SQLite schema proposed in `STEP-1/SCOPED-QUESTIONS-ANSWERS/scoped-session-storage.md` — use that as a base and refine it given the full requirements above.

package/.OLD/STEP-2/SCOPED-QUESTIONS/scoped-subagent-lifecycle.md

@@ -0,0 +1,65 @@
+# Scoped Question: Agent Spawn & Sub-Agent Lifecycle
+
+## Context
+KhaBotCLI allows agents to spawn sub-agents via the `agent_spawn` built-in tool. A sub-agent runs with an isolated context window, performs a focused task, and returns a condensed summary to the parent. This is different from `task_create`, which creates an async task in the task queue.
+
+## The Two Spawning Mechanisms
+
+### `agent_spawn` (Sub-Agent — Synchronous, Isolated Context)
+- Parent agent calls `agent_spawn` with: agent name, instruction, optional files/context
+- Runtime creates a new session with a clean context window
+- Sub-agent runs to completion (or timeout)
+- Sub-agent's full output is condensed into ~1k-2k tokens
+- Condensed result returned to parent agent as tool output
+- Parent agent continues with the result in its context
+
+### `task_create` (Task — Asynchronous, Task Queue)
+- Parent agent calls `task_create` with: agent name, input (prompt + files + context)
+- Runtime creates a task entry in SQLite with status: pending
+- Task enters the execution queue
+- Parent can continue (fire-and-forget) or set own status to waiting
+- Parent polls with `task_status` to check completion
+- Task result available via `task_status` output field
+
+## Questions to Answer
+
+1. **Context setup for spawned sub-agent:**
+   - Does it get the parent's AGENT.md? Or a minimal system prompt?
+   - Does it inherit the parent's MEMORY.md?
+   - Does it get the parent's tool access? Or a restricted subset?
+   - Does it get any of the parent's conversation history?
+
+2. **Model selection:**
+   - Does it use the parent's model? Or a cheaper model (the `compact` model)?
+   - Can the parent specify which model the sub-agent should use?
+   - Should the `agent_spawn` tool accept a `model` parameter?
+
+3. **Tool access:**
+   - Which tools does a sub-agent have access to?
+   - Can it spawn its own sub-agents? (recursion limit?)
+   - Can it create tasks? Send messages to other agents?
+   - Should there be a configurable max depth for nested sub-agents?
+
+4. **Timeout and limits:**
+   - How long can a sub-agent run? Parent's remaining time? Fixed limit?
+   - Max iterations for a sub-agent?
+   - What happens when timeout is reached? Force kill? Summarize partial work?
+
+5. **Return format:**
+   - Who condenses the output? The sub-agent itself? A hidden summarize agent?
+   - What token budget for the return summary? (1k-2k tokens recommended by research)
+   - What if the sub-agent fails? What error format is returned to parent?
+
+6. **Session handling:**
+   - Does the sub-agent get its own session in SQLite? Or is it part of the parent's session?
+   - Are sub-agent messages queryable via the API?
+   - Token usage: counted separately or aggregated with parent?
+
+7. **Difference from `task_create`:**
+   - When should an agent use `agent_spawn` vs `task_create`?
+   - Provide a clear decision framework for the system prompt to teach agents when to use which.
+
+8. **Built-in sub-agent patterns:**
+   - The user accepted these built-in agent roles: explore (read-only), plan (analysis), execute (full access), compact (summarize), title (generate title), summary (generate summary)
+   - Should these be actual agents in `.khabot/agents/` or hardcoded in the runtime?
+   - Do they have AGENT.md files? Or are they defined in `system-prompts/agents/`?

package/.OLD/STEP-2/SCOPED-QUESTIONS-ANSWERS/scoped-api-design.md

@@ -0,0 +1,394 @@
+# Scoped Answer: REST API Design — Final Specification
+
+---
+
+## 1. Complete Endpoint List
+
+### Agent Operations
+
+| Method | Path | Request Body | Response | Status Codes |
+|--------|------|-------------|----------|-------------|
+| `GET` | `/agents` | — | `{ data: Agent[], pagination }` | 200 |
+| `GET` | `/agents/:name` | — | `{ data: AgentDetail }` | 200, 404 |
+| `POST` | `/agents/:name/chat` | `{ message, session_id? }` | `{ data: { session_id, response, tool_calls? } }` | 200, 404, 400, 409 |
+| `POST` | `/agents/:name/task` | `{ input, priority?, tags?[] }` | `{ data: { task_id, status: "queued" } }` | 201, 404, 400 |
+| `POST` | `/agents/:name/trigger` | `{ }` | `{ data: { result, triggered_at } }` | 200, 404, 400, 409 |
+
+### Session Operations
+
+| Method | Path | Request Body | Response | Status Codes |
+|--------|------|-------------|----------|-------------|
+| `GET` | `/sessions` | — (query: `agent`, `status`, `after`, `before`, `limit`, `cursor`) | `{ data: Session[], pagination }` | 200 |
+| `GET` | `/sessions/:id` | — (query: `limit`, `cursor` for messages) | `{ data: SessionDetail }` | 200, 404 |
+| `POST` | `/sessions/:id/messages` | `{ message }` | `{ data: { response, tool_calls? } }` | 200, 404, 409 |
+| `DELETE` | `/sessions/:id` | — | `{ data: { id, status: "closed" } }` | 200, 404, 409 |
+
+### Task Operations
+
+| Method | Path | Request Body | Response | Status Codes |
+|--------|------|-------------|----------|-------------|
+| `GET` | `/tasks` | — (query: `agent`, `status`, `priority`, `tags`, `limit`, `cursor`) | `{ data: Task[], pagination }` | 200 |
+| `GET` | `/tasks/:id` | — | `{ data: TaskDetail }` | 200, 404 |
+| `GET` | `/tasks/:id/log` | — (query: `limit`, `cursor`) | `{ data: LogEntry[], pagination }` | 200, 404 |
+| `POST` | `/tasks/:id/respond` | `{ response }` | `{ data: { task_id, status } }` | 200, 404, 409 |
+| `POST` | `/tasks/:id/cancel` | — | `{ data: { task_id, status: "cancelled" } }` | 200, 404, 409 |
+
+### Daemon Operations
+
+| Method | Path | Request Body | Response | Status Codes |
+|--------|------|-------------|----------|-------------|
+| `GET` | `/daemons` | — | `{ data: Daemon[] }` | 200 |
+| `POST` | `/daemons/:name/start` | — | `{ data: { name, status: "running" } }` | 200, 404, 409 |
+| `POST` | `/daemons/:name/stop` | — | `{ data: { name, status: "stopped" } }` | 200, 404, 409 |
+| `POST` | `/daemons/:name/trigger` | — | `{ data: { result, triggered_at } }` | 200, 404, 409 |
+
+### System Operations
+
+| Method | Path | Request Body | Response | Status Codes |
+|--------|------|-------------|----------|-------------|
+| `GET` | `/health` | — | `{ status: "ok", uptime }` | 200 |
+| `GET` | `/status` | — | `{ data: SystemStatus }` | 200 |
+| `POST` | `/shutdown` | — | `{ data: { message: "shutting down" } }` | 202 |
+
+### Debug Operations (dev-mode only)
+
+| Method | Path | Request Body | Response | Status Codes |
+|--------|------|-------------|----------|-------------|
+| `GET` | `/debug/agents/:name/prompt` | — | `{ data: { layers[], assembled } }` | 200, 404, 403 |
+| `GET` | `/debug/agents/:name/tools` | — | `{ data: Tool[] }` | 200, 404, 403 |
+
+---
+
+## 2. Separate Invoke Endpoints (Recommended)
+
+**Recommendation: Separate endpoints per agent type.**
+
+Use `POST /agents/:name/chat`, `POST /agents/:name/task`, and `POST /agents/:name/trigger` — NOT a unified `/agents/:name/invoke`.
+
+**Rationale:**
+
+- **Different request shapes** — Chat needs `{ message, session_id? }`, Task needs `{ input, priority?, tags? }`, Trigger needs nothing or minimal config. Forcing these into one body with a `mode` discriminator adds unnecessary complexity.
+- **Different response shapes** — Chat returns a synchronous response with `session_id`. Task returns a `task_id` with `status: "queued"`. Trigger returns an immediate result. A unified endpoint would need a response union type that makes the API harder to document and consume.
+- **Different status codes** — Task creation returns `201 Created`, Chat returns `200 OK`, Trigger returns `200 OK`. A unified endpoint obscures this semantic difference.
+- **Different validation rules** — Each endpoint can validate strictly against its own schema without needing conditional validation logic.
+- **Self-documenting** — `POST /agents/:name/chat` tells you exactly what it does. `POST /agents/:name/invoke { mode: "chat" }` requires reading the body to understand intent.
+- **Server-side routing is simpler** — No switch/case on mode, just three separate handlers.
+- **Type safety for consumers** — API clients can type each endpoint independently, which is much cleaner than a discriminated union.
+
+The server should still enforce type correctness: calling `POST /agents/:name/chat` on a Task agent returns `400 AGENT_TYPE_MISMATCH`. This provides clear feedback rather than silently accepting a wrong invocation mode.
+
+---
+
+## 3. Pagination Format
+
+**Recommendation: Cursor-based pagination.**
+
+Cursor-based is superior to offset/limit for this use case because:
+- Data is append-heavy (messages, logs, tasks) — offset pagination breaks when new items are inserted during traversal.
+- Cursors are opaque, so the server can change the underlying implementation without breaking clients.
+- More efficient on SQLite — cursor translates to a `WHERE id > ?` which uses the index, whereas `OFFSET N` scans and discards N rows.
+
+### Response Envelope
+
+```json
+{
+  "ok": true,
+  "data": [ ... ],
+  "pagination": {
+    "next_cursor": "abc123",
+    "has_more": true,
+    "limit": 50
+  }
+}
+```
+
+### Request Parameters (query string)
+
+| Param | Type | Default | Description |
+|-------|------|---------|-------------|
+| `cursor` | string | `null` | Opaque cursor from previous response |
+| `limit` | integer | `50` | Items per page, max `200` |
+
+### Rules
+
+- **First request**: omit `cursor` — returns the first page.
+- **Next page**: pass `pagination.next_cursor` as `?cursor=abc123`.
+- **No previous page navigation**: cursor-based is forward-only by design. If the client needs to go back, it caches previous pages locally.
+- **Cursor encoding**: Base64-encoded JSON of `{ id, dir }` internally. Clients treat it as opaque.
+- **Empty result**: `data: []`, `has_more: false`, `next_cursor: null`.
+
+### Error Envelope (for non-2xx responses)
+
+```json
+{
+  "ok": false,
+  "error": {
+    "code": "AGENT_NOT_FOUND",
+    "message": "Agent 'my-agent' does not exist",
+    "details": {}
+  }
+}
+```
+
+The `ok` boolean allows quick checks without inspecting HTTP status. This is especially useful when the API is relayed through Ably where HTTP semantics may be abstracted away.
+
+---
+
+## 4. Error Code Taxonomy
+
+### Error Code Format
+
+All error codes are UPPER_SNAKE_CASE strings. They are stable identifiers for programmatic handling; the `message` field is human-readable and may change.
+
+### General Errors
+
+| Code | HTTP Status | Description |
+|------|------------|-------------|
+| `BAD_REQUEST` | 400 | Malformed JSON, missing required fields |
+| `VALIDATION_ERROR` | 400 | Field-level validation failure (details has field map) |
+| `UNAUTHORIZED` | 401 | Missing or invalid `X-KhaBot-Secret` |
+| `FORBIDDEN` | 403 | Feature disabled (e.g., debug endpoints in prod) |
+| `NOT_FOUND` | 404 | Generic not found |
+| `METHOD_NOT_ALLOWED` | 405 | Wrong HTTP method |
+| `CONFLICT` | 409 | Operation conflicts with current state |
+| `RATE_LIMITED` | 429 | Rate limit exceeded |
+| `INTERNAL_ERROR` | 500 | Unexpected server error |
+
+### Agent Errors
+
+| Code | HTTP Status | Description |
+|------|------------|-------------|
+| `AGENT_NOT_FOUND` | 404 | Agent with given name doesn't exist |
+| `AGENT_TYPE_MISMATCH` | 400 | Wrong endpoint for agent type (e.g., `/chat` on Task agent) |
+| `AGENT_BUSY` | 409 | Agent is currently processing and cannot accept new work |
+| `AGENT_DISABLED` | 409 | Agent exists but is disabled |
+
+### Session Errors
+
+| Code | HTTP Status | Description |
+|------|------------|-------------|
+| `SESSION_NOT_FOUND` | 404 | Session ID doesn't exist |
+| `SESSION_CLOSED` | 409 | Session is already closed, cannot send messages |
+| `SESSION_AGENT_MISMATCH` | 400 | Session doesn't belong to the specified agent |
+
+### Task Errors
+
+| Code | HTTP Status | Description |
+|------|------------|-------------|
+| `TASK_NOT_FOUND` | 404 | Task ID doesn't exist |
+| `TASK_NOT_WAITING` | 409 | Task is not in waiting state, cannot respond |
+| `TASK_ALREADY_DONE` | 409 | Task is completed/cancelled, cannot modify |
+| `TASK_CANCELLED` | 409 | Task was already cancelled |
+
+### Daemon Errors
+
+| Code | HTTP Status | Description |
+|------|------------|-------------|
+| `DAEMON_NOT_FOUND` | 404 | Daemon with given name doesn't exist |
+| `DAEMON_ALREADY_RUNNING` | 409 | Daemon is already started |
+| `DAEMON_ALREADY_STOPPED` | 409 | Daemon is already stopped |
+| `DAEMON_BUSY` | 409 | Daemon is mid-tick, cannot trigger again |
+
+### Tool/Permission Errors
+
+| Code | HTTP Status | Description |
+|------|------------|-------------|
+| `TOOL_NOT_FOUND` | 404 | Tool not registered |
+| `PERMISSION_DENIED` | 403 | Tool call blocked by permission policy |
+| `TOOL_EXECUTION_ERROR` | 500 | Tool ran but threw an error |
+
+### Validation Error Detail Example
+
+```json
+{
+  "ok": false,
+  "error": {
+    "code": "VALIDATION_ERROR",
+    "message": "Request validation failed",
+    "details": {
+      "fields": {
+        "message": "required",
+        "priority": "must be one of: low, normal, high, urgent"
+      }
+    }
+  }
+}
+```
+
+---
+
+## 5. Rate Limiting
+
+**Recommendation: Built-in, simple, optional.**
+
+### Why Built-in
+
+- The API runs on localhost, so there's no external reverse proxy by default.
+- When exposed via Ably, the Ably relay is the only "proxy" — and it may not offer request-level rate limiting.
+- Agent invocations trigger LLM calls which are expensive. Protecting against runaway clients or misbehaving scripts is critical even locally.
+
+### Implementation
+
+- **In-memory sliding window** — no need for Redis. A simple token bucket or fixed window counter per IP (or per API key if auth is enabled).
+- **Configurable** in the runtime config file:
+
+```json
+{
+  "rate_limit": {
+    "enabled": true,
+    "window_seconds": 60,
+    "max_requests": 120
+  }
+}
+```
+
+- **Default**: enabled, 120 requests/minute. Generous enough for normal use, prevents runaway loops.
+- **Per-endpoint overrides** (optional, for v2): allow tighter limits on `/agents/:name/chat` and `/agents/:name/task` since they trigger LLM calls.
+
+### Response Headers
+
+```
+X-RateLimit-Limit: 120
+X-RateLimit-Remaining: 47
+X-RateLimit-Reset: 1709284800
+```
+
+When exceeded → `429` with `RATE_LIMITED` error code and `Retry-After` header.
+
+### Ably Consideration
+
+When requests come via Ably, rate limiting should apply per Ably client ID rather than per IP, since all Ably requests come from the same connection. The Ably relay layer should inject a `X-Ably-Client-Id` header that the rate limiter uses as the key.
+
+---
+
+## 6. Request Validation
+
+**Recommendation: Strict, but with a pragmatic approach.**
+
+### Rules
+
+1. **Missing required fields** → `400 VALIDATION_ERROR` with field-level details.
+2. **Wrong field types** (e.g., string where number expected) → `400 VALIDATION_ERROR`.
+3. **Unknown fields** → **Silently ignore them. Do NOT return 400.**
+
+### Rationale for Ignoring Unknown Fields
+
+- **Forward compatibility** — If a newer client sends a field the server doesn't know about yet, it shouldn't break. This is a core REST API design principle (Postel's law: "Be liberal in what you accept").
+- **Ably relay** — Ably or middleware might inject extra metadata fields. Rejecting unknowns would break the relay.
+- **Less friction** — Clients can evolve independently. Strict unknown-field rejection creates tight coupling.
+
+### Validation Library
+
+Use a schema validation library like `ajv` (JSON Schema) or `zod` — define schemas per endpoint, validate on entry, strip unknown fields before passing to handlers. This keeps validation logic declarative and centralized.
+
+### Content-Type Enforcement
+
+- All `POST`/`PUT`/`PATCH` requests must have `Content-Type: application/json`.
+- Missing or wrong content type → `400 BAD_REQUEST` with message `"Content-Type must be application/json"`.
+- `GET` and `DELETE` requests must not have a body (silently ignored if present).
+
+---
+
+## 7. Ably Compatibility
+
+### Constraints and Design Decisions
+
+Ably acts as a relay — it receives requests remotely and forwards them to the local API. The response goes back through Ably to the remote caller. This imposes several constraints:
+
+### 1. Request/Response Must Be Pure JSON
+
+- No streaming, no chunked transfer encoding — already satisfied by the "no streaming" constraint.
+- No binary payloads in request/response bodies. If file uploads are needed later, use base64 encoding or a reference-based approach.
+
+### 2. Message Size Limits
+
+- Ably has a **default message size limit of 64 KB** (can be increased to 256 KB on some plans).
+- **Responses must be bounded** — This is why pagination is critical. Never return unbounded lists.
+- **Large responses** (e.g., full conversation history) should be paginated with conservative default limits (e.g., 50 messages per page).
+- If a single response might exceed 64 KB (e.g., a very long agent response), consider a `truncated` flag and a follow-up endpoint for the full content.
+
+### 3. No HTTP Headers in Ably Channel Messages
+
+- Ably messages are JSON payloads, not HTTP requests. The Ably relay layer must map:
+  - **Auth**: `X-KhaBot-Secret` → include as a `secret` field in the Ably message envelope.
+  - **Rate limit client ID**: inject from Ably connection metadata.
+- The API response envelope (`{ ok, data, error, pagination }`) carries everything needed — no reliance on HTTP headers for essential data.
+
+### 4. Ably Message Envelope
+
+The Ably relay should use a standard envelope:
+
+```json
+{
+  "request_id": "uuid-v4",
+  "method": "POST",
+  "path": "/agents/my-agent/chat",
+  "body": { "message": "hello", "session_id": "abc" },
+  "secret": "optional-api-key"
+}
+```
+
+Response:
+
+```json
+{
+  "request_id": "uuid-v4",
+  "status": 200,
+  "body": {
+    "ok": true,
+    "data": { ... }
+  }
+}
+```
+
+The `request_id` correlates request/response since Ably is pub/sub, not request/response natively.
+
+### 5. Idempotency
+
+Ably may deliver messages more than once (at-least-once delivery). Critical mutating operations should be idempotent where possible:
+- `POST /agents/:name/task` — include an optional `idempotency_key` in the request body. If the same key is seen again, return the original task instead of creating a duplicate.
+- `POST /tasks/:id/cancel` — cancelling an already-cancelled task returns 200 (idempotent), not 409.
+- `DELETE /sessions/:id` — deleting an already-closed session returns 200.
+
+---
+
+## 8. Versioning
+
+**Recommendation: Unversioned for v1, with a planned migration path.**
+
+### Why Unversioned for v1
+
+- This is a localhost-first tool. The server and client are typically co-located and updated together.
+- Adding `/v1/` prefix from day one is premature — there's no v2 to differentiate from.
+- It adds noise to every endpoint path and every piece of documentation for zero current benefit.
+
+### Migration Path for Later
+
+When a breaking change is needed:
+1. **Preferred**: Use API version via header — `Accept: application/vnd.khabot.v2+json`. The server defaults to the latest version if no header is sent.
+2. **Fallback**: Add path prefix `/v2/` only when header-based negotiation is not practical (e.g., Ably relay doesn't support custom headers easily).
+
+### Practical Safeguard
+
+Include a `X-KhaBot-API-Version` response header in every response from day one:
+
+```
+X-KhaBot-API-Version: 1
+```
+
+This lets clients detect which version they're talking to without requiring a versioned path. When v2 arrives, clients can check this header and adapt.
+
+---
+
+## Summary of Recommendations
+
+| # | Topic | Decision |
+|---|-------|----------|
+| 1 | Endpoints | 22 endpoints across 6 categories (see tables above) |
+| 2 | Invoke pattern | **Separate endpoints** per agent type (`/chat`, `/task`, `/trigger`) |
+| 3 | Pagination | **Cursor-based**, `{ data, pagination: { next_cursor, has_more, limit } }` |
+| 4 | Error codes | **25 error codes** across 6 categories, UPPER_SNAKE_CASE, with `details` for validation |
+| 5 | Rate limiting | **Built-in**, in-memory sliding window, 120 req/min default, configurable |
+| 6 | Validation | **Strict on types/required**, **ignore unknown fields** (Postel's law) |
+| 7 | Ably compat | Pure JSON, respect 64KB limit, request_id correlation, idempotency keys |
+| 8 | Versioning | **Unversioned** for v1, version header in responses, path prefix deferred |