khabotcli 1.0.0

package/.OLD/STEP-1/SCOPED-QUESTIONS-ANSWERS/scoped-mcp-vs-custom-tools.md

@@ -0,0 +1,234 @@
+# Answer: MCP-Only vs Custom Tool Scripts
+
+> **Recommendation: Approach B — MCP-Only**, with a scaffolding layer that makes simple tool creation trivially easy.
+
+---
+
+## 1. Recommendation & Reasoning
+
+**Go MCP-only.** The reasoning is structural, not ideological:
+
+### Why not Custom Scripts (Approach A)
+- **Security liability.** Custom scripts execute arbitrary code with the runtime's full permissions. There's no sandboxing, no controlled interface, no capability negotiation. Every `index.js` is an attack surface.
+- **Proprietary dead-end.** Custom scripts only work inside KhaBotCli. No community, no shared ecosystem, no reusable packages. Every tool written this way is a sunk cost that cannot be exported.
+- **One tool = one script.** The 1:1 mapping between scripts and tools means a Slack integration that needs `send_message`, `list_channels`, and `read_thread` requires three separate script folders. MCP exposes all three from a single server.
+- **No lifecycle management.** Spawning a new process per tool call is expensive. Making scripts long-running requires reinventing exactly what MCP already provides (JSON-RPC, heartbeats, capability negotiation).
+
+### Why not Both (Approach C)
+- **Maintenance burden doubles.** Two discovery paths, two execution engines, two sets of error handling, two documentation sections. For a v1 runtime built by a small team, this is unjustifiable.
+- **User confusion.** "Should I write a custom script or an MCP server?" becomes the first question every new user asks. The answer will always be "MCP unless it's trivially simple," which means custom scripts become a stepping stone that users outgrow — creating migration friction.
+- **Naming conflicts.** If both a custom script and an MCP server expose `send_slack_message`, resolution order creates implicit behavior that's hard to debug. The current PLAN.md resolution order (§6) already has 6 priority levels — adding MCP/script disambiguation makes it worse.
+
+### Why MCP-Only (Approach B) wins
+- **Industry convergence.** Anthropic, OpenAI, Google, Cursor, Windsurf, and dozens of other tools have adopted MCP. The ecosystem of pre-built MCP servers is growing rapidly (GitHub, Slack, PostgreSQL, filesystem, web search, etc.). KhaBotCli users get access to this entire ecosystem for free.
+- **Richer capabilities.** MCP provides Tools + Resources + Prompts. Custom scripts only provide tools. Resources let a Slack MCP server expose channel lists as browsable context. Prompts let it provide pre-built interaction templates. This richness matters for agent quality.
+- **Controlled interface.** MCP servers communicate via JSON-RPC over stdio or HTTP. The runtime controls what data flows in and out. No arbitrary filesystem access, no ambient authority.
+- **Multi-tool bundling.** One MCP server = many tools with shared state. A database MCP server can expose `query`, `schema`, `tables`, `explain` — all sharing a single connection pool.
+- **Dynamic discovery.** MCP servers declare their capabilities at handshake time. No static manifests to keep in sync. Add a capability to the server, and the agent sees it immediately.
+
+---
+
+## 2. Making Simple Tool Creation Easy (The Scaffolding Layer)
+
+The main objection to MCP-only is: *"MCP is too complex for a simple tool."* This is solved with a **scaffolding CLI + lightweight MCP wrapper**, not by maintaining a parallel system.
+
+### 2.1 `khabot tool create` — Interactive Scaffolder
+
+```bash
+$ khabot tool create
+
+? Tool name: send_slack_message
+? Description: Sends a message to a Slack channel
+? Input parameters:
+  - channel (string, required): Slack channel name
+  - message (string, required): Message content
+? Language: javascript
+? Multiple tools in this server? No
+
+✓ Created .khabot/mcp-servers/send_slack/
+  ├── package.json
+  ├── index.js       ← implement your logic here
+  └── README.md
+
+✓ Added to .khabot/settings.json mcpServers:
+  "send_slack": {
+    "transport": "stdio",
+    "command": "node",
+    "args": [".khabot/mcp-servers/send_slack/index.js"]
+  }
+```
+
+### 2.2 The Generated Code is Minimal
+
+The scaffolded `index.js` uses a thin MCP SDK wrapper. The user only writes the business logic:
+
+```javascript
+import { createServer, tool } from "@khabot/mcp-lite";
+
+const server = createServer({ name: "send_slack" });
+
+server.register(
+  tool({
+    name: "send_slack_message",
+    description: "Sends a message to a Slack channel",
+    input: {
+      channel: { type: "string", description: "Channel name", required: true },
+      message: { type: "string", description: "Message content", required: true },
+    },
+    async handler({ channel, message }) {
+      // ← User writes ONLY this function body
+      const resp = await fetch(`https://slack.com/api/chat.postMessage`, {
+        method: "POST",
+        headers: { Authorization: `Bearer ${process.env.SLACK_TOKEN}` },
+        body: JSON.stringify({ channel, text: message }),
+      });
+      return { ok: resp.ok };
+    },
+  })
+);
+
+server.start();
+```
+
+**Key point:** This is ~20 lines. The user writes exactly one function — the handler. The MCP protocol, JSON-RPC, stdio transport, and capability negotiation are all handled by `@khabot/mcp-lite` (a thin wrapper around the official `@modelcontextprotocol/sdk`).
+
+### 2.3 `@khabot/mcp-lite` — The Simplicity Bridge
+
+Ship a small npm package (or bundled module) that reduces MCP server creation to:
+
+| What the user does | What the SDK handles |
+|---|---|
+| Define tool name, description, input schema | JSON-RPC server setup |
+| Write an async handler function | stdio/HTTP transport |
+| `server.start()` | Capability negotiation handshake |
+| | Error serialization |
+| | Input validation against schema |
+| | Process lifecycle (graceful shutdown) |
+
+This gives users the **simplicity of custom scripts** with the **architecture of MCP**. No protocol knowledge required.
+
+### 2.4 Templates for Common Patterns
+
+Ship built-in templates:
+
+```bash
+$ khabot tool create --template http-api    # REST API wrapper
+$ khabot tool create --template database    # SQL query tool
+$ khabot tool create --template file-ops    # File manipulation
+$ khabot tool create --template webhook     # Outbound webhook
+$ khabot tool create --template multi       # Multi-tool server scaffold
+```
+
+Each template provides a working example with the handler stubbed out.
+
+---
+
+## 3. Security Considerations
+
+### MCP Servers (Recommended)
+
+| Aspect | Detail |
+|---|---|
+| **Process isolation** | Each MCP server runs as a separate process. Crash/exploit in one server doesn't affect the runtime or other servers. |
+| **Controlled I/O** | Communication only via JSON-RPC over stdio or HTTP. No ambient authority — the server can't access runtime internals. |
+| **Env-scoped secrets** | Secrets are passed via `env` in the MCP config, not embedded in code. The runtime can redact these from logs. |
+| **Capability declaration** | The server declares exactly what it can do at handshake. The runtime can enforce allowlists/denylists on tool names. |
+| **Permission integration** | MCP tools integrate with KhaBotCli's existing permission system (deny/ask/allow from settings.json §13). An MCP tool named `dangerous_delete` can be denied just like a built-in tool. |
+| **Transport security** | Streamable HTTP transport supports authorization headers (bearer tokens, OAuth). stdio is local-only by nature. |
+| **Resource limits** | The runtime can enforce timeouts on MCP tool calls and kill unresponsive servers. |
+
+### Custom Scripts (If they were kept — reasons to avoid)
+
+| Risk | Detail |
+|---|---|
+| **Arbitrary code execution** | `index.js` runs with the same permissions as the KhaBotCli process. It can read auth.json, delete files, exfiltrate data. |
+| **No sandboxing** | Node.js has no built-in sandbox. `vm` module is explicitly not a security mechanism. |
+| **Dependency chain attacks** | Custom scripts with `node_modules` introduce supply chain risk. MCP servers have the same risk but at least run in a separate process. |
+| **No capability negotiation** | The runtime has no way to know what a script will do until it executes. |
+
+### Mitigation for MCP (defense in depth)
+
+1. **Allowlist MCP tool names** in `settings.json` permissions — agents can only call tools explicitly permitted.
+2. **Timeout enforcement** — kill MCP tool calls that exceed `timeout_seconds`.
+3. **Log all tool calls** — every MCP invocation logged to task log for audit.
+4. **Process resource limits** — set memory/CPU limits on MCP server processes (via ulimit or cgroups).
+5. **Transport restriction** — default to `stdio` only. `streamable-http` requires explicit opt-in with authorization config.
+
+---
+
+## 4. Ecosystem & Community Momentum
+
+### MCP Adoption (as of early 2026)
+
+- **Anthropic** — Created MCP, deeply integrated into Claude Code, Claude Desktop, and the Claude API.
+- **OpenAI** — Added MCP support to ChatGPT and the Assistants API.
+- **Google** — MCP support in Gemini API and Android Studio.
+- **Cursor** — MCP is the primary extension mechanism for custom tools.
+- **Windsurf** — Full MCP support.
+- **VS Code** — GitHub Copilot supports MCP servers.
+- **Zed, Emacs, Neovim plugins** — Community-driven MCP integrations.
+
+### Pre-built MCP Server Ecosystem
+
+Hundreds of production-ready MCP servers exist:
+
+| Category | Examples |
+|---|---|
+| **Code & Git** | GitHub, GitLab, Bitbucket |
+| **Databases** | PostgreSQL, MySQL, SQLite, MongoDB, Redis |
+| **Cloud** | AWS, GCP, Azure, Cloudflare |
+| **Communication** | Slack, Discord, Email, Twilio |
+| **Search** | Brave Search, Google Search, Exa |
+| **Files** | Filesystem, Google Drive, S3 |
+| **Monitoring** | Sentry, Datadog, PagerDuty |
+| **Knowledge** | Notion, Linear, Jira, Confluence |
+
+**KhaBotCli users get all of these for free** with a single config entry. No custom code needed.
+
+### The Direction is Clear
+
+Every major AI tool vendor has converged on MCP. Investing in a proprietary custom script system means:
+- Maintaining it forever (backwards compatibility)
+- Competing with a growing ecosystem of free MCP servers
+- Explaining to users why they should write custom scripts instead of using existing MCP servers
+
+---
+
+## 5. Migration Path from Current Plan
+
+The current PLAN.md (§6) has custom tools at resolution priority 2-3 and MCP at priority 5. The `step-1.md` audit (§2.4) already recommends deprecating custom tools. Here's the concrete migration:
+
+### What to do
+
+1. **Remove** `.khabot/tools/<toolName>/` (tool.json + index.js) from the folder structure spec.
+2. **Remove** `~/.khabot/tools/` from the global structure.
+3. **Remove** tool resolution steps 2-4 from §6 (custom tool lookup + KhaBot.com tool registry).
+4. **Promote MCP** to resolution priority 2 (right after built-in tools).
+5. **Add** `.khabot/mcp-servers/` as a conventional location for user-created MCP servers.
+6. **Ship** `@khabot/mcp-lite` SDK and `khabot tool create` scaffolder.
+7. **Document** a "Your first tool in 5 minutes" guide using the scaffolder.
+
+### Updated Tool Resolution Order
+
+```
+1. Built-in KhaBotCli tools       ← always available (bash, read_file, etc.)
+2. MCP servers (settings.json)     ← stdio and streamable-http
+3. NOT FOUND                       ← agent receives tool error
+```
+
+Two steps. No ambiguity.
+
+---
+
+## 6. Summary
+
+| Decision | Choice |
+|---|---|
+| **Approach** | B — MCP-only |
+| **Simple tool story** | `khabot tool create` scaffolder + `@khabot/mcp-lite` SDK |
+| **Security model** | Process isolation + permission system integration + transport restrictions |
+| **Ecosystem bet** | Aligned with industry (Anthropic, OpenAI, Google, Cursor, etc.) |
+| **Maintenance cost** | One extension system, one discovery path, one execution engine |
+| **Migration** | Remove custom tool spec from PLAN.md, promote MCP to primary |
+
+The simplicity objection is real but solvable. A good scaffolder + thin SDK gives users the same developer experience as "write a function, done" while building on a standard protocol that connects them to a growing ecosystem.

package/.OLD/STEP-1/SCOPED-QUESTIONS-ANSWERS/scoped-session-storage.md

@@ -0,0 +1,455 @@
+# Answer: Best Storage Strategy for 50k+ AI Agent Sessions
+
+---
+
+## 1. Recommendation: Option B — SQLite (with a dev-phase escape hatch)
+
+**Use SQLite as the sole storage engine.** This is not a close call at scale.
+
+### Why SQLite wins
+
+| Concern | JSON Files | SQLite |
+|---------|-----------|--------|
+| 50k sessions + 50k tasks | 100k–200k files, directory listing takes seconds | Single file, instant indexed queries |
+| Query by agent/status/date | Full scan of every `meta.json` — O(n) | `SELECT ... WHERE agent = ? AND status = ? ORDER BY created_at DESC` — O(log n) |
+| Atomic writes | No. Crash mid-write = corrupted JSON | Built-in WAL mode, transactions, crash-safe |
+| Concurrent access | Last-write-wins, data loss risk | Multiple readers + one writer, safe by default |
+| Backup | `tar` an entire directory tree of 200k files | `cp data.db data.db.bak` — one file |
+| Cleanup/archival | Walk + stat + delete thousands of files | `DELETE FROM sessions WHERE created_at < ?` — milliseconds |
+| Disk usage | JSON overhead + filesystem block waste (4KB per file minimum) | ~40-60% smaller due to no filesystem overhead |
+| Dependencies | None | `better-sqlite3` (Node) or `go-sqlite3` (Go) — both zero-config, embedded, no external process |
+
+### Why not Hybrid (Option C)?
+
+The hybrid approach sounds appealing ("best of both") but in practice:
+- **Two sources of truth** = consistency bugs. What happens when SQLite says session exists but the message file is missing?
+- **Double the I/O code** — every read/write path has two implementations.
+- **Debugging gets harder**, not easier. You'll `sqlite3 data.db` to inspect metadata, then `cat` a JSON file for messages. That's worse than just using one tool.
+- The "human-readable" argument is weak. You'll build an API and a `khabot sessions inspect <id>` CLI command anyway. Nobody manually reads `messages.json` files at scale.
+
+### Why not JSONL (Option D)?
+
+- No random access. Finding session `sess_abc123` requires scanning the entire file.
+- Updating a record (e.g., changing session status from `active` to `closed`) requires rewriting the entire file.
+- At 50k sessions × 20 messages × avg 2KB per message = ~2GB in `messages.jsonl`. Every query scans the whole thing.
+- Append-only is great for logs, terrible for queryable state.
+
+---
+
+## 2. Lazy Loading Strategy
+
+### Principle: Load metadata eagerly, load content lazily
+
+```
+Startup:
+  ├── Open SQLite connection (WAL mode)
+  ├── Prepare statements (connection pool)
+  └── Done. No data loaded into memory.
+
+On GET /sessions (list):
+  └── SELECT id, agent_name, status, message_count, created_at, updated_at
+      FROM sessions
+      ORDER BY updated_at DESC
+      LIMIT 20 OFFSET 0
+      → Returns metadata only. Messages NOT loaded.
+
+On GET /sessions/:id (detail):
+  └── SELECT * FROM messages
+      WHERE session_id = ?
+      ORDER BY created_at ASC
+      → Messages loaded on-demand, only for this session.
+
+On agent loop (processing):
+  └── Load messages for active session into memory.
+      After each LLM response → INSERT new message row.
+      After compaction → DELETE old messages, INSERT summary.
+      On session close → release from memory.
+```
+
+### In-Memory Session Cache (LRU)
+
+```
+Settings:
+  maxActiveSessions: 50  (in-memory session objects with loaded messages)
+
+Behavior:
+  - When a session is accessed, promote it to top of LRU.
+  - When cache is full, evict least-recently-used session from memory.
+  - Eviction = drop from memory only. Data stays in SQLite.
+  - Active sessions (currently processing) are pinned — never evicted.
+```
+
+### SQLite Schema for Lazy Loading
+
+```sql
+-- Sessions table: lightweight, always queryable
+CREATE TABLE sessions (
+  id            TEXT PRIMARY KEY,
+  agent_name    TEXT NOT NULL,
+  agent_type    TEXT NOT NULL,
+  instance_folder TEXT,
+  status        TEXT NOT NULL DEFAULT 'active',
+  model         TEXT,
+  message_count INTEGER DEFAULT 0,
+  token_usage_input   INTEGER DEFAULT 0,
+  token_usage_output  INTEGER DEFAULT 0,
+  estimated_cost      REAL DEFAULT 0,
+  created_at    TEXT NOT NULL,
+  updated_at    TEXT NOT NULL
+);
+
+CREATE INDEX idx_sessions_agent ON sessions(agent_name);
+CREATE INDEX idx_sessions_status ON sessions(status);
+CREATE INDEX idx_sessions_created ON sessions(created_at);
+CREATE INDEX idx_sessions_updated ON sessions(updated_at);
+
+-- Messages table: loaded on-demand per session
+CREATE TABLE messages (
+  id          INTEGER PRIMARY KEY AUTOINCREMENT,
+  session_id  TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
+  role        TEXT NOT NULL,  -- system | user | assistant | tool
+  content     TEXT NOT NULL,  -- can be JSON string for multipart content
+  tool_calls  TEXT,           -- JSON array of tool calls (nullable)
+  tool_call_id TEXT,          -- for tool response messages
+  token_count INTEGER DEFAULT 0,
+  created_at  TEXT NOT NULL
+);
+
+CREATE INDEX idx_messages_session ON messages(session_id);
+CREATE INDEX idx_messages_session_order ON messages(session_id, created_at);
+
+-- Tasks table
+CREATE TABLE tasks (
+  id              TEXT PRIMARY KEY,
+  session_id      TEXT REFERENCES sessions(id),
+  agent_name      TEXT NOT NULL,
+  status          TEXT NOT NULL DEFAULT 'pending',
+  priority        TEXT DEFAULT 'normal',
+  parent_task_id  TEXT REFERENCES tasks(id),
+  input           TEXT NOT NULL,  -- JSON
+  output          TEXT,           -- JSON (nullable)
+  tags            TEXT,           -- JSON array
+  iterations      INTEGER DEFAULT 0,
+  max_iterations  INTEGER DEFAULT 50,
+  timeout_seconds INTEGER DEFAULT 300,
+  token_usage_input   INTEGER DEFAULT 0,
+  token_usage_output  INTEGER DEFAULT 0,
+  estimated_cost      REAL DEFAULT 0,
+  error           TEXT,
+  created_at      TEXT NOT NULL,
+  started_at      TEXT,
+  finished_at     TEXT,
+  updated_at      TEXT NOT NULL
+);
+
+CREATE INDEX idx_tasks_agent ON tasks(agent_name);
+CREATE INDEX idx_tasks_status ON tasks(status);
+CREATE INDEX idx_tasks_created ON tasks(created_at);
+CREATE INDEX idx_tasks_session ON tasks(session_id);
+
+-- Task logs: append-only, loaded on-demand
+CREATE TABLE task_logs (
+  id        INTEGER PRIMARY KEY AUTOINCREMENT,
+  task_id   TEXT NOT NULL REFERENCES tasks(id) ON DELETE CASCADE,
+  level     TEXT DEFAULT 'info',  -- info | warn | error | iteration
+  content   TEXT NOT NULL,
+  created_at TEXT NOT NULL
+);
+
+CREATE INDEX idx_task_logs_task ON task_logs(task_id);
+
+-- Schema version tracking
+CREATE TABLE schema_version (
+  version     INTEGER PRIMARY KEY,
+  applied_at  TEXT NOT NULL,
+  description TEXT
+);
+```
+
+### Key Design Decisions
+
+- **`TEXT` for timestamps** — ISO 8601 strings. SQLite compares them correctly with `<` / `>`. Avoids timezone bugs.
+- **`TEXT` for JSON fields** — `input`, `output`, `tool_calls`, `tags`. Use `json_extract()` in SQLite for rare queries into JSON fields.
+- **`ON DELETE CASCADE`** — Deleting a session auto-deletes its messages. Clean.
+- **Separate `task_logs` table** — Not inline in `tasks`. Logs can be large, loaded on-demand via `GET /tasks/:id/log`.
+- **`message_count` on sessions** — Denormalized for fast list queries without COUNT(*) joins.
+
+---
+
+## 3. Cleanup & Archival Strategy
+
+### Retention Policy (configurable)
+
+```json
+{
+  "storage": {
+    "retention": {
+      "sessions": {
+        "maxAgeDays": 90,
+        "maxCount": 10000,
+        "archiveBeforeDelete": true
+      },
+      "tasks": {
+        "maxAgeDays": 30,
+        "maxCount": 5000,
+        "archiveBeforeDelete": true
+      }
+    }
+  }
+}
+```
+
+### Cleanup Process
+
+```
+Runs: On startup + daily (or configurable interval)
+
+Step 1: Archive (if archiveBeforeDelete = true)
+  - SELECT sessions older than maxAgeDays OR exceeding maxCount
+  - Export to ~/.khabot/archive/sessions-2026-02.jsonl (JSONL per month)
+  - Each line = { session metadata + all messages } as JSON
+  - Compressed: ~/.khabot/archive/sessions-2026-02.jsonl.gz
+
+Step 2: Delete
+  - DELETE FROM sessions WHERE id IN (archived IDs)
+  - CASCADE deletes messages automatically
+
+Step 3: Vacuum (periodic, not every run)
+  - VACUUM reclaims disk space after large deletes
+  - Only run if >20% of DB space is reclaimable
+  - Run at most once per week (VACUUM is expensive)
+```
+
+### Why JSONL for archives?
+
+Archives are write-once, read-rarely. JSONL is:
+- Human-readable if you ever need to inspect
+- Gzip-compressible (10:1 ratio on text)
+- Importable back into SQLite if needed
+- No dependency required to read
+
+### Cleanup SQL
+
+```sql
+-- Find sessions to clean up (age-based)
+SELECT id FROM sessions
+WHERE created_at < datetime('now', '-90 days')
+  AND status = 'closed'
+ORDER BY created_at ASC;
+
+-- Find sessions to clean up (count-based, keep newest 10k)
+SELECT id FROM sessions
+WHERE id NOT IN (
+  SELECT id FROM sessions ORDER BY updated_at DESC LIMIT 10000
+);
+
+-- Actual delete (after archiving)
+DELETE FROM sessions WHERE id IN (?);
+-- Messages auto-deleted via CASCADE
+
+-- Reclaim space (periodic)
+PRAGMA page_count;     -- total pages
+PRAGMA freelist_count; -- reclaimable pages
+-- If freelist_count / page_count > 0.2:
+VACUUM;
+```
+
+---
+
+## 4. Scaling from Development to Production (50k+)
+
+### Phase 1: Development (0–100 sessions)
+
+- SQLite works out of the box. No configuration needed.
+- WAL mode enabled by default (concurrent reads during writes).
+- All queries are instant at this scale. No indexes even needed (but add them anyway).
+- Easy to inspect: `sqlite3 ~/.khabot/data.db "SELECT * FROM sessions"`
+- Easy to reset: `rm ~/.khabot/data.db` and restart.
+
+### Phase 2: Growth (100–5,000 sessions)
+
+- Indexes start mattering. The schema above has them from day one.
+- Enable auto-cleanup with default retention (90 days).
+- Monitor DB file size (should be ~50-200MB at this scale).
+- No changes needed — everything works as designed.
+
+### Phase 3: Production (5,000–50,000+ sessions)
+
+- DB size: ~500MB–2GB depending on message sizes.
+- SQLite handles this fine. SQLite is tested to terabyte scale.
+- Ensure `PRAGMA journal_mode=WAL` (set on first connection).
+- Ensure `PRAGMA synchronous=NORMAL` (faster writes, still safe with WAL).
+- Ensure `PRAGMA cache_size=-64000` (64MB cache, up from 2MB default).
+- Cleanup becomes important — archive old sessions monthly.
+- Consider `PRAGMA auto_vacuum=INCREMENTAL` to avoid full VACUUM.
+- If queries slow down, add composite indexes:
+  ```sql
+  CREATE INDEX idx_sessions_agent_status ON sessions(agent_name, status, updated_at DESC);
+  ```
+
+### Pragmas to set on every connection
+
+```sql
+PRAGMA journal_mode = WAL;          -- concurrent reads, crash-safe
+PRAGMA synchronous = NORMAL;        -- safe with WAL, 2x faster than FULL
+PRAGMA cache_size = -64000;         -- 64MB page cache
+PRAGMA foreign_keys = ON;           -- enforce CASCADE deletes
+PRAGMA busy_timeout = 5000;         -- wait 5s on lock instead of failing
+PRAGMA auto_vacuum = INCREMENTAL;   -- gradual space reclaim
+```
+
+---
+
+## 5. Backup & Migration Strategy
+
+### Backup
+
+**SQLite makes this trivial:**
+
+```bash
+# Hot backup (safe while DB is in use)
+sqlite3 ~/.khabot/data.db ".backup ~/.khabot/backups/data-$(date +%Y%m%d).db"
+
+# Or just copy (safe with WAL mode if done correctly)
+sqlite3 ~/.khabot/data.db "VACUUM INTO '~/.khabot/backups/data-snapshot.db'"
+```
+
+**Automated backup (built into KhaBotCli):**
+
+```json
+{
+  "storage": {
+    "backup": {
+      "enabled": true,
+      "interval": "daily",
+      "maxBackups": 7,
+      "path": "~/.khabot/backups/"
+    }
+  }
+}
+```
+
+On each backup interval:
+1. Run `VACUUM INTO` to create a snapshot
+2. Delete backups older than `maxBackups`
+
+### Migration Strategy
+
+**Schema migrations using a version table:**
+
+```
+~/.khabot/data.db contains:
+  schema_version table → tracks applied migrations
+
+On startup:
+  1. Open DB
+  2. Read current schema_version
+  3. Compare against built-in migration list
+  4. Apply any unapplied migrations in order
+  5. Each migration runs inside a transaction (atomic)
+```
+
+**Migration file structure (in KhaBotCli source code):**
+
+```
+src/storage/migrations/
+  001_initial.sql        ← sessions, messages, tasks, task_logs
+  002_add_token_usage.sql
+  003_add_task_tags.sql
+  ...
+```
+
+**Each migration:**
+```sql
+-- 002_add_token_usage.sql
+ALTER TABLE sessions ADD COLUMN token_usage_input INTEGER DEFAULT 0;
+ALTER TABLE sessions ADD COLUMN token_usage_output INTEGER DEFAULT 0;
+ALTER TABLE sessions ADD COLUMN estimated_cost REAL DEFAULT 0;
+
+INSERT INTO schema_version (version, applied_at, description)
+VALUES (2, datetime('now'), 'Add token usage tracking to sessions');
+```
+
+**Critical rule:** Migrations must be **additive only** in v1. Never rename or delete columns — only add columns, add tables, add indexes. This avoids complex ALTER TABLE gymnastics that SQLite doesn't fully support.
+
+### Export/Import for portability
+
+```bash
+# Export everything to portable format
+khabot export --format jsonl --output ./backup/
+
+# Import from portable format (or from another machine)
+khabot import --from ./backup/sessions.jsonl
+
+# Export specific agent's sessions
+khabot export --agent researcher --format jsonl
+```
+
+---
+
+## 6. How OpenCode Uses SQLite — Lessons Learned
+
+### OpenCode's Architecture
+
+OpenCode (Go-based AI coding agent) uses SQLite as its sole persistence layer:
+
+- **Single DB file:** `~/.local/share/opencode/opencode.db`
+- **ORM:** Drizzle ORM (in the TypeScript rewrite) for schema definitions and migrations
+- **Tables:** sessions, messages (at minimum) — stores full conversation history
+- **Data directory config:** `{ "data": { "directory": ".opencode" } }` — configurable per project
+- **Auto-compact:** When context reaches 95% of model's window, summarizes conversation into a new session
+
+### Lessons Learned from OpenCode
+
+**Lesson 1: Migration gates must be explicit, not implicit.**
+
+OpenCode had a critical bug ([#13654](https://github.com/anomalyco/opencode/issues/13654)): they migrated from JSON files to SQLite, but the migration check was "does `opencode.db` exist?" The DB could exist from earlier schema migrations without containing the session data. Result: **188 JSON session files permanently orphaned** for incremental upgraders.
+
+**KhaBotCli fix:** Use a `schema_version` table with explicit version numbers. Never use file existence as a migration gate. Each migration has a unique version number that's checked and recorded.
+
+**Lesson 2: Start with SQLite from day one.**
+
+OpenCode started with JSON files, then migrated to SQLite later. This created:
+- A one-time migration path that had bugs
+- Two code paths (old JSON reader + new SQLite reader) during transition
+- Users lost data during the migration
+
+**KhaBotCli fix:** Ship SQLite from the very first version. No migration needed. No legacy code path.
+
+**Lesson 3: Append-only message storage is correct.**
+
+Both OpenCode and the `coding-agent-session-search` project (which indexes sessions from 11+ AI agents) treat message storage as **append-only**:
+- Never update a message row after insertion
+- New messages are INSERT, never UPDATE
+- Compaction = INSERT summary + DELETE originals (in a transaction)
+- This prevents data corruption from partial updates
+
+**KhaBotCli fix:** Messages are append-only. The only mutation is DELETE (during compaction or cleanup). Never UPDATE a message row.
+
+**Lesson 4: Separate concerns — don't mix config DB with session DB.**
+
+OpenCode stores everything in one DB. If the DB corrupts, you lose both configuration state and session history.
+
+**KhaBotCli fix:** Config stays in JSON files (human-editable, version-controllable). Only runtime data (sessions, messages, tasks) goes in SQLite.
+
+**Lesson 5: WAL mode is non-negotiable.**
+
+OpenCode uses WAL mode for concurrent reads. Without it, any read blocks writes and vice versa — unacceptable for a server handling multiple sessions.
+
+**KhaBotCli fix:** `PRAGMA journal_mode=WAL` set on first connection open, every time.
+
+---
+
+## Summary Decision Matrix
+
+| Decision | Choice | Rationale |
+|----------|--------|-----------|
+| **Storage engine** | SQLite (Option B) | Indexing, atomic writes, single-file backup, scales to 50k+ trivially |
+| **Schema** | 4 tables: sessions, messages, tasks, task_logs | Normalized, with denormalized `message_count` on sessions for list perf |
+| **Lazy loading** | Metadata always queryable, messages loaded per-session on demand | LRU cache for active sessions (max 50 in memory) |
+| **Cleanup** | Configurable retention (90d sessions, 30d tasks), archive to gzipped JSONL | Runs on startup + daily. VACUUM only when >20% reclaimable. |
+| **Scaling** | Works unchanged from 1 to 50k+. Tune PRAGMAs at scale. | Add composite indexes only if specific queries slow down. |
+| **Backup** | `VACUUM INTO` daily snapshots, keep 7 | One command, safe during operation |
+| **Migration** | `schema_version` table, numbered SQL files, additive-only | Learned from OpenCode's migration bugs |
+| **Archives** | JSONL + gzip per month | Portable, human-readable, re-importable |
+| **Config vs data** | JSON files for config, SQLite for runtime data | Best of both: editable config, queryable data |