khabotcli 1.0.0

package/.OLD/STEP-2/RESEARCH-RESULTS/deep-research-ably-integration.md

@@ -0,0 +1,673 @@
+# Deep Research Results: Ably Integration Patterns for Agent Runtimes
+
+> **Research date:** Mar 1, 2026
+> **Sources:** Ably official documentation (ably.com/docs), Ably FAQs, Ably architecture docs
+> **Context:** KhaBotCLI — local Node.js agent runtime exposing REST API, needs remote access via Ably
+
+---
+
+## 1. Channel Architecture
+
+### Recommended: One Channel Per KhaBotCLI Instance
+
+Use **one bidirectional channel per KhaBotCLI instance**, not per agent or per session.
+
+**Why:**
+- Each KhaBotCLI instance is a single server process — one channel maps cleanly to one process
+- Per-agent or per-session channels would create excessive channel churn (Ably charges for peak concurrent channels and each channel consumes RAM on at least 2 Ably nodes)
+- A single channel keeps presence simple: one presence entry = one KhaBotCLI instance online
+- Multiple concurrent requests are handled via **correlation IDs** inside messages (see §2), not via separate channels
+
+**Channel naming convention:**
+```
+khabot:instance:<instanceId>
+```
+
+Where `instanceId` is a unique identifier assigned by KhaBot.com when the instance registers (e.g. `khabot:instance:inst_a1b2c3d4`).
+
+**Namespace:** `khabot` — allows applying Ably channel rules (persistence, TLS-only, identified clients) to all KhaBotCLI channels at once via the Ably dashboard.
+
+**Message name routing:** Ably messages have a `name` field separate from the `data` payload. Use the message name to distinguish message types on the single channel:
+- `api:request` — remote client → KhaBotCLI (via KhaBot.com)
+- `api:response` — KhaBotCLI → remote client (via KhaBot.com)
+- `system:ping` — KhaBot.com → KhaBotCLI health check
+- `system:pong` — KhaBotCLI → KhaBot.com health check response
+
+### Why NOT Per-Agent or Per-Session Channels
+
+| Approach | Channels | Pros | Cons |
+|----------|----------|------|------|
+| Per-instance | 1 | Simple, cheap, one presence entry | Must multiplex via correlation ID |
+| Per-agent | N (agents) | Clean separation | Channel churn, higher cost, complex presence |
+| Per-session | N (sessions) | Full isolation | Massive channel count, expensive, unnecessary |
+
+**Verdict:** Per-instance with multiplexed messages is the standard pattern for RPC-over-pub/sub.
+
+### Handling Multiple Concurrent Requests
+
+Multiple concurrent requests on the same channel are handled entirely via **correlation IDs** (see §2). KhaBotCLI processes them concurrently, sending responses tagged with the matching `requestId`. The remote client matches responses to requests by `requestId`.
+
+### Authorization
+
+Only authorized clients can publish to a channel because:
+1. KhaBot.com issues **scoped Ably tokens** with capabilities restricted to the specific instance channel
+2. The token's capability grants only `publish` + `subscribe` on `khabot:instance:<instanceId>`
+3. KhaBotCLI gets a token scoped to its own channel
+4. Remote clients never connect to Ably directly — KhaBot.com acts as the relay
+
+---
+
+## 2. Message Format (RPC over Pub/Sub)
+
+### Request/Response Pattern
+
+Ably is a pub/sub system, not an RPC system. There is no built-in request/response mechanism. The standard pattern is:
+
+1. **Publisher** sends a message with a unique `requestId`
+2. **Subscriber** processes the request, sends a response message with the same `requestId`
+3. **Publisher** matches the response by `requestId`
+
+### Recommended Message Format
+
+Mirror the REST API structure exactly so the Ably layer is a transparent proxy.
+
+**Request message** (name: `api:request`):
+```json
+{
+  "requestId": "req_a1b2c3d4e5f6",
+  "method": "POST",
+  "path": "/sessions/sess_123/messages",
+  "headers": {
+    "X-KhaBot-Secret": "mysecret"
+  },
+  "body": {
+    "content": "Hello, agent!"
+  },
+  "timestamp": 1740793200000,
+  "timeout": 30000
+}
+```
+
+**Response message** (name: `api:response`):
+```json
+{
+  "requestId": "req_a1b2c3d4e5f6",
+  "status": 200,
+  "headers": {
+    "Content-Type": "application/json"
+  },
+  "body": {
+    "id": "msg_xyz",
+    "role": "assistant",
+    "content": "Hello! How can I help?"
+  },
+  "timestamp": 1740793201500,
+  "duration": 1500
+}
+```
+
+**Error response:**
+```json
+{
+  "requestId": "req_a1b2c3d4e5f6",
+  "status": 500,
+  "headers": {},
+  "body": {
+    "error": {
+      "code": "INTERNAL_ERROR",
+      "message": "LLM provider returned 503"
+    }
+  },
+  "timestamp": 1740793202000,
+  "duration": 2000
+}
+```
+
+### Correlation ID Strategy
+
+- Use `requestId` as the correlation identifier (format: `req_<nanoid>`)
+- Generated by KhaBot.com (the remote-side publisher), NOT by the end client
+- KhaBotCLI includes the same `requestId` in every response message
+- KhaBot.com maintains a pending request map: `Map<requestId, { resolve, reject, timer }>`
+- On receiving `api:response`, lookup `requestId` → resolve the pending promise → clear timer
+
+### Timeout Handling
+
+KhaBot.com manages timeouts, not Ably:
+
+1. KhaBot.com publishes `api:request` and starts a timer (e.g. 30s)
+2. If no `api:response` arrives within the timeout:
+   - Resolve with a timeout error to the remote client
+   - Optionally publish an `api:cancel` message (KhaBotCLI can ignore stale requests)
+3. If KhaBotCLI is offline (not present on the channel):
+   - KhaBot.com checks presence **before** publishing
+   - If not present → return immediate error to remote client ("Instance offline")
+   - No message is published to Ably at all
+
+**Ably does NOT queue messages for offline subscribers.** Messages published while a subscriber is disconnected for >2 minutes are lost. This is critical — see §5 for details.
+
+### Maximum Message Size
+
+- **Default:** 64 KiB (65,536 bytes) per message
+- **Custom:** Can be increased on enterprise plans (contact Ably support)
+- **Encoding overhead:** JSON + base64 encoding increases payload ~33%, so effective data limit is ~48 KiB
+- **REST SDK enforces locally:** The 64 KiB limit is enforced client-side by the SDK
+
+**Handling large payloads:**
+- For most API requests/responses, 64 KiB is sufficient (agent messages, task status, session metadata)
+- For large outputs (file contents, long tool outputs): **truncate or paginate** at the API layer before relaying through Ably
+- For very large data: **do NOT send through Ably** — use a presigned URL / external storage and send only a reference through Ably
+- KhaBotCLI response handler should check `JSON.stringify(response).length` and truncate `body` if >60 KiB, replacing with a reference: `{ "truncated": true, "fetchUrl": "/api/v1/messages/msg_xyz/full" }`
+
+---
+
+## 3. Authentication Flow
+
+### Architecture
+
+```
+┌──────────────┐     ┌──────────────┐     ┌──────────────┐
+│ Remote Client │────▶│  KhaBot.com  │────▶│    Ably      │
+│ (web/mobile)  │◀────│  (auth srv)  │◀────│  (pub/sub)   │
+└──────────────┘     └──────────────┘     └──────────────┘
+                            │                      ▲
+                            │   Token provisioning │
+                            ▼                      │
+                     ┌──────────────┐              │
+                     │  KhaBotCLI   │──────────────┘
+                     │  (local)     │   Connects with scoped token
+                     └──────────────┘
+```
+
+### KhaBotCLI Authentication with Ably
+
+**Use `authCallback` with token authentication**, not API key auth.
+
+Flow:
+1. KhaBotCLI starts with Ably enabled (`settings.json: ably.enabled = true`)
+2. KhaBotCLI has a `khabot_token` (from `auth.json`) — this authenticates it with KhaBot.com
+3. KhaBotCLI creates an Ably Realtime client with an `authCallback`:
+   ```js
+   const ably = new Ably.Realtime({
+     authCallback: async (tokenParams, callback) => {
+       try {
+         // Call KhaBot.com to get an Ably TokenRequest
+         const res = await fetch('https://khabot.com/api/v1/ably/token', {
+           method: 'POST',
+           headers: {
+             'Authorization': `Bearer ${khabotToken}`,
+             'Content-Type': 'application/json'
+           },
+           body: JSON.stringify({ instanceId })
+         });
+         const tokenRequest = await res.json();
+         callback(null, tokenRequest);
+       } catch (err) {
+         callback(err, null);
+       }
+     },
+     clientId: instanceId  // e.g. "inst_a1b2c3d4"
+   });
+   ```
+4. KhaBot.com validates the `khabot_token`, looks up the instance, and calls `ably.auth.createTokenRequest()` with scoped capabilities
+5. Ably SDK handles token refresh automatically — calls `authCallback` again before token expiry
+
+### KhaBot.com Token Issuance (Scoped)
+
+KhaBot.com issues tokens with **minimal capabilities**:
+
+```js
+// On KhaBot.com server
+const tokenRequest = await ablyRest.auth.createTokenRequest({
+  clientId: instanceId,
+  capability: {
+    [`khabot:instance:${instanceId}`]: ['publish', 'subscribe', 'presence']
+  },
+  ttl: 3600 * 1000  // 1 hour (max 24h for access tokens)
+});
+```
+
+This ensures:
+- KhaBotCLI can only publish/subscribe to **its own channel**
+- Cannot read other instances' channels
+- Cannot create arbitrary channels
+- `clientId` is enforced by Ably — the instance cannot impersonate another
+
+### Token Refresh
+
+- **Automatic:** Ably SDK calls `authCallback` before token expiry — no manual refresh needed
+- **TTL:** Recommended 1 hour (max 24h for access tokens, max 1h if revocable tokens enabled)
+- **On failure:** If KhaBot.com is unreachable during refresh, connection enters `disconnected` → `suspended` → eventually `failed`
+- **KhaBotCLI should listen** for `failed` state and log a clear error: "Cannot refresh Ably token — KhaBot.com unreachable"
+
+### Remote Client Authentication
+
+Remote clients (web app, mobile) **never connect to Ably directly**:
+1. Remote client authenticates with KhaBot.com (JWT, session, etc.)
+2. Remote client calls KhaBot.com REST API: `POST /api/v1/instances/{id}/rpc`
+3. KhaBot.com publishes the message to Ably on behalf of the client
+4. KhaBot.com subscribes to responses and relays back to the client via HTTP response
+
+This keeps all Ably credentials server-side (on KhaBot.com) and gives KhaBot.com full control over rate limiting, validation, and access control.
+
+---
+
+## 4. Ably Node.js SDK
+
+### Package
+
+```
+npm install ably
+```
+
+The `ably` npm package includes both Realtime and REST clients. Use Realtime for KhaBotCLI (persistent WebSocket connection).
+
+### Connection and Subscription Pattern
+
+```js
+import * as Ably from 'ably';
+
+class AblyBridge {
+  constructor(instanceId, khabotToken, apiHandler) {
+    this.instanceId = instanceId;
+    this.khabotToken = khabotToken;
+    this.apiHandler = apiHandler;
+    this.channelName = `khabot:instance:${instanceId}`;
+    this.ably = null;
+    this.channel = null;
+  }
+
+  async connect() {
+    this.ably = new Ably.Realtime({
+      authCallback: async (tokenParams, callback) => {
+        try {
+          const tokenRequest = await this._fetchAblyToken();
+          callback(null, tokenRequest);
+        } catch (err) {
+          callback(err, null);
+        }
+      },
+      clientId: this.instanceId,
+      // Ably SDK handles reconnection automatically
+    });
+
+    // Monitor connection state
+    this.ably.connection.on((stateChange) => {
+      console.log(`Ably: ${stateChange.previous} → ${stateChange.current}`);
+      if (stateChange.current === 'failed') {
+        console.error('Ably connection failed:', stateChange.reason);
+      }
+    });
+
+    await this.ably.connection.once('connected');
+
+    // Get channel and subscribe
+    this.channel = this.ably.channels.get(this.channelName);
+
+    // Subscribe to API requests
+    await this.channel.subscribe('api:request', (message) => {
+      this._handleRequest(message);
+    });
+
+    // Subscribe to system messages
+    await this.channel.subscribe('system:ping', (message) => {
+      this.channel.publish('system:pong', {
+        requestId: message.data.requestId,
+        timestamp: Date.now()
+      });
+    });
+
+    // Enter presence to signal "online"
+    await this.channel.presence.enter({ version: '1.0.0', status: 'ready' });
+  }
+
+  async _handleRequest(message) {
+    const { requestId, method, path, headers, body, timeout } = message.data;
+    const startTime = Date.now();
+
+    try {
+      // Delegate to the local REST API handler
+      const result = await this.apiHandler({
+        method, path, headers, body
+      });
+
+      await this.channel.publish('api:response', {
+        requestId,
+        status: result.status,
+        headers: result.headers || {},
+        body: result.body,
+        timestamp: Date.now(),
+        duration: Date.now() - startTime
+      });
+    } catch (err) {
+      await this.channel.publish('api:response', {
+        requestId,
+        status: 500,
+        headers: {},
+        body: {
+          error: { code: 'INTERNAL_ERROR', message: err.message }
+        },
+        timestamp: Date.now(),
+        duration: Date.now() - startTime
+      });
+    }
+  }
+
+  async disconnect() {
+    if (this.channel) {
+      await this.channel.presence.leave();
+    }
+    if (this.ably) {
+      this.ably.close();
+    }
+  }
+
+  async _fetchAblyToken() {
+    const res = await fetch('https://khabot.com/api/v1/ably/token', {
+      method: 'POST',
+      headers: {
+        'Authorization': `Bearer ${this.khabotToken}`,
+        'Content-Type': 'application/json'
+      },
+      body: JSON.stringify({ instanceId: this.instanceId })
+    });
+    if (!res.ok) throw new Error(`Token fetch failed: ${res.status}`);
+    return res.json();
+  }
+}
+```
+
+### Connection State Management
+
+Ably SDK connection states:
+
+| State | Meaning | Action |
+|-------|---------|--------|
+| `initialized` | Client created, not yet connected | — |
+| `connecting` | Attempting to connect | — |
+| `connected` | Connected to Ably | Ready to send/receive |
+| `disconnected` | Temporarily lost connection | SDK auto-retries. Messages in transit may be lost. |
+| `suspended` | Disconnected >2 minutes | Connection state lost. SDK still retries but less frequently. |
+| `closing` | `close()` called | — |
+| `closed` | Connection closed | — |
+| `failed` | Fatal error (e.g. auth failure) | Must call `connect()` explicitly to retry |
+
+**Key behaviors:**
+- SDK auto-reconnects on `disconnected` → `connecting` → `connected`
+- After 2 minutes disconnected → `suspended` (channel continuity lost)
+- On reconnect from `suspended`, channels reattach but `resumed` flag = `false` (messages during gap are lost)
+- `failed` requires manual intervention (typically an auth problem)
+
+### Error Handling
+
+```js
+// Listen for all connection state changes
+this.ably.connection.on('failed', (stateChange) => {
+  logger.error('Ably connection failed', { reason: stateChange.reason });
+  // Could trigger alert or attempt re-auth
+});
+
+this.ably.connection.on('suspended', (stateChange) => {
+  logger.warn('Ably connection suspended — messages during gap will be lost');
+});
+
+// Listen for channel-level errors
+this.channel.on('failed', (stateChange) => {
+  logger.error('Channel failed', { reason: stateChange.reason });
+});
+
+// Handle publish errors
+try {
+  await this.channel.publish('api:response', responseData);
+} catch (err) {
+  logger.error('Failed to publish response', { err, requestId });
+  // Response is lost — remote client will timeout
+}
+```
+
+---
+
+## 5. Reconnection & Reliability
+
+### What Happens When KhaBotCLI Loses Internet
+
+| Duration | Ably State | What Happens |
+|----------|------------|--------------|
+| 0–2 min | `disconnected` | SDK auto-retries. Messages published to channel during this window are **stored in RAM** by Ably (no hard limit, but 2-minute window). On reconnect, missed messages are replayed automatically. |
+| 2+ min | `suspended` | Connection state lost. Channels detach. On reconnect, channels reattach but **messages during the gap are lost**. `resumed` flag = `false`. |
+| Extended | `suspended` → eventually `failed` | If auth token expires and can't refresh, moves to `failed`. Requires manual reconnection. |
+
+### Does Ably Queue Messages While Disconnected?
+
+**Yes, but only for ~2 minutes:**
+- Ably stores all messages for a channel in RAM for 2 minutes (default, no hard limit on count within this window)
+- If KhaBotCLI reconnects within 2 minutes, **all missed messages are replayed automatically** — the SDK handles this transparently
+- After 2 minutes, connection state is lost and messages are gone
+
+**With persistence enabled** (channel rule setting):
+- Messages can be stored on disk for up to 72 hours
+- But this uses the **history API** — you must explicitly fetch them, they are NOT auto-replayed
+- Enabling persistence adds cost
+
+### Should KhaBotCLI Process Queued Requests After Coming Back Online?
+
+**Within 2 minutes (auto-resumed):** Yes — they arrive automatically via the SDK. KhaBotCLI should check the `timestamp` field and skip requests older than the configured timeout (e.g. 30s). Stale requests should get a "request timed out" response.
+
+**After 2 minutes (channel reattached, not resumed):** Do NOT try to fetch and replay historical requests. The remote clients have already timed out. Instead:
+1. Detect `resumed === false` on channel reattach
+2. Log a warning
+3. Re-enter presence to signal "back online"
+4. Wait for new requests only
+
+**Stale request handling:**
+```js
+async _handleRequest(message) {
+  const { requestId, timestamp, timeout } = message.data;
+  const age = Date.now() - timestamp;
+  const maxAge = timeout || 30000;
+
+  if (age > maxAge) {
+    // Request already timed out on the remote side — skip it
+    logger.debug(`Skipping stale request ${requestId} (age: ${age}ms)`);
+    return;
+  }
+
+  // ... process normally
+}
+```
+
+---
+
+## 6. Performance & Limits
+
+### Ably Rate Limits
+
+| Limit | Default Value | Notes |
+|-------|---------------|-------|
+| **Publish rate per channel** | 50 msg/s | Hard limit. Exceeding returns error 42910. |
+| **Max message size** | 64 KiB (65,536 bytes) | Includes all fields (name, data, extras). Can be increased on enterprise. |
+| **Message history retention** | 2 minutes (RAM) | Extended to 24h/72h with persistence channel rules. |
+| **Concurrent channels** | Package-dependent | Free: 200, Pro: 10K, Enterprise: custom |
+| **Concurrent connections** | Package-dependent | Free: 200, Pro: 10K, Enterprise: custom |
+| **Max channels per connection** | Package-dependent | Typically hundreds |
+| **Token TTL** | Max 24 hours | Max 1 hour if revocable tokens enabled |
+| **Connection state preservation** | 2 minutes | After that, moves to `suspended` |
+
+### 50 msg/s Per Channel — Is It Enough?
+
+For KhaBotCLI's use case: **absolutely yes.**
+- One KhaBotCLI instance handles a handful of concurrent agents
+- Each agent interaction is typically 1 request + 1 response = 2 messages
+- Even with 10 concurrent sessions, that's ~20 msg/s peak
+- 50 msg/s per channel is more than sufficient
+- If somehow exceeded (bulk operations), implement client-side throttling with a queue
+
+### Latency Overhead
+
+- **Direct REST API (localhost):** ~1-5ms
+- **Ably relay:** ~50-150ms additional round-trip latency (client → Ably → KhaBotCLI + KhaBotCLI → Ably → client)
+- **Total for Ably-relayed request:** Original API time + ~100-300ms overhead
+- For agent interactions where LLM calls take 1-30 seconds, the Ably overhead is **negligible** (<1%)
+
+### Is Ably Suitable for Long-Running Task Status Polling?
+
+**Not ideal for polling, but not needed either.** Ably is a push system — use it for push notifications instead:
+
+1. Client creates a task via Ably
+2. KhaBotCLI processes the task (may take minutes)
+3. KhaBotCLI publishes progress updates as `task:progress` messages
+4. KhaBotCLI publishes final result as `api:response`
+
+This is **better than polling** — the remote client gets real-time updates without repeated requests.
+
+For tasks >2 minutes: the remote client should remain subscribed to the channel. As long as the client stays connected, it will receive the response whenever it arrives. If the client disconnects and reconnects, it can use the REST API (via KhaBot.com HTTP) to check task status.
+
+---
+
+## 7. Security Considerations
+
+### KhaBotCLI Should Validate Message Content
+
+Even though KhaBotCLI trusts KhaBot.com, **always validate messages defensively:**
+
+1. **Schema validation:** Validate that incoming `api:request` messages have all required fields (`requestId`, `method`, `path`)
+2. **Method whitelist:** Only allow `GET`, `POST`, `PUT`, `PATCH`, `DELETE`
+3. **Path validation:** Reject paths that don't match known API routes
+4. **Size limits:** Reject bodies larger than a reasonable limit (e.g. 1 MiB)
+5. **Rate limiting:** Track request rate per channel and reject if excessive
+
+**Why validate even though we trust KhaBot.com:**
+- Defense in depth — bugs on KhaBot.com could send malformed messages
+- If KhaBot.com is compromised, KhaBotCLI is the last line of defense
+- Prevents unexpected crashes from malformed input
+
+### Replay Attack Prevention
+
+Ably provides some inherent protection:
+- **TLS by default** — messages in transit cannot be intercepted or tampered with
+- **Token-based auth** — tokens are short-lived (1h recommended), limiting the window for stolen credentials
+- **`clientId` enforcement** — Ably enforces the `clientId` from the token, preventing impersonation
+
+Additional measures for KhaBotCLI:
+- **Timestamp validation:** Reject messages with `timestamp` older than 60 seconds (already handled by stale request logic)
+- **Request ID uniqueness:** Track recently processed `requestId`s in a Set (with TTL cleanup). Reject duplicates.
+- **Nonce (optional):** If paranoid, include a nonce in each request that KhaBot.com signs with a shared secret. KhaBotCLI verifies the signature.
+
+```js
+const recentRequestIds = new Map(); // requestId → timestamp
+
+function isDuplicate(requestId) {
+  if (recentRequestIds.has(requestId)) return true;
+  recentRequestIds.set(requestId, Date.now());
+  // Cleanup old entries every 5 minutes
+  return false;
+}
+```
+
+### If KhaBot.com Is Compromised
+
+If KhaBot.com is compromised, the attacker could:
+- Issue valid Ably tokens
+- Publish arbitrary `api:request` messages to any KhaBotCLI instance
+- Read all responses
+
+**Mitigations:**
+1. **KhaBotCLI permission system still applies** — all tool calls go through deny/ask/allow. Even via Ably, dangerous operations are blocked.
+2. **Local auth.json secret still required** — the `X-KhaBot-Secret` header in the relayed request must match the local secret
+3. **Optional: end-to-end encryption** — Ably supports AES-256-CBC encryption on channels. KhaBotCLI and the actual end-user share a key that KhaBot.com never sees. This prevents KhaBot.com from reading message payloads even if compromised. Trade-off: KhaBot.com can't validate/inspect messages.
+4. **Audit logging** — KhaBotCLI logs all Ably-relayed requests. Anomalies can be detected post-hoc.
+
+### Channel-Level Encryption (Optional, Future)
+
+Ably supports per-channel AES-256-CBC encryption:
+```js
+const channel = ably.channels.get('khabot:instance:inst_123', {
+  cipher: { key: sharedSecretKey }
+});
+```
+- Ably cannot decrypt the payload — true end-to-end encryption
+- Key distribution is your responsibility (not managed by Ably)
+- Only the `data` field is encrypted — `name` and `clientId` remain visible
+- Consider for high-security deployments where even KhaBot.com should not see message content
+
+---
+
+## 8. Recommended Implementation Plan for KhaBotCLI
+
+### Phase 1: Basic Integration
+1. Add `ably` npm dependency
+2. Create `AblyBridge` class (see §4 code pattern)
+3. Wire into server startup: if `settings.json` has `ably.enabled: true`, create and connect AblyBridge
+4. Map incoming Ably messages → local REST API handler → Ably response
+5. Handle graceful shutdown: leave presence, close connection
+
+### Phase 2: Reliability
+1. Implement stale request detection (skip requests older than timeout)
+2. Implement duplicate request detection (requestId dedup)
+3. Add connection state logging (all state transitions)
+4. Handle `suspended` state: log warning, re-enter presence on reconnect
+5. Implement large payload truncation with fetch-URL fallback
+
+### Phase 3: Production Hardening
+1. Message schema validation (reject malformed requests)
+2. Rate limiting on Ably requests (separate from REST API rate limits)
+3. Metrics: track Ably message count, latency, error rate
+4. Optional: channel encryption for high-security deployments
+5. Health check: respond to `system:ping` with instance status
+
+### Settings Schema Addition
+
+```json
+{
+  "ably": {
+    "enabled": false,
+    "instanceId": "",
+    "tokenEndpoint": "https://khabot.com/api/v1/ably/token",
+    "requestTimeout": 30000,
+    "maxRequestAge": 60000,
+    "encryption": false
+  }
+}
+```
+
+---
+
+## 9. Limitations & When Ably Is NOT Suitable
+
+### Ably IS suitable for:
+- Relaying REST API calls (request/response via correlation IDs)
+- Push-based task progress notifications
+- Online/offline presence detection
+- Low-to-moderate message volume (well under 50 msg/s per instance)
+
+### Ably is NOT suitable for:
+- **Streaming LLM output** — Ably's 64 KiB message limit and pub/sub model aren't designed for SSE-style token streaming. If streaming is ever needed, use a direct WebSocket or SSE connection, not Ably.
+- **Large file transfers** — 64 KiB limit. Use presigned URLs + external storage.
+- **Guaranteed delivery after >2 min offline** — messages are lost. If KhaBotCLI is frequently offline for extended periods, Ably is not a reliable transport. Consider a persistent message queue (Redis, SQS) as fallback.
+- **High-frequency telemetry** — if >50 messages/second are needed per instance, Ably's per-channel rate limit is a bottleneck. Shard across channels or use a different transport.
+
+### Fallback Strategy
+
+If Ably is unavailable or not suitable for a specific operation:
+1. **HTTP polling fallback:** Remote client polls KhaBot.com, which polls KhaBotCLI's REST API directly (requires port forwarding / tunnel — not ideal)
+2. **WebSocket tunnel:** KhaBotCLI opens a WebSocket to KhaBot.com directly (custom tunnel, bypasses Ably)
+3. **For v1:** Ably-only is fine. The fallback strategies are complexity that can be deferred.
+
+---
+
+## 10. Summary of Key Decisions
+
+| Decision | Recommendation |
+|----------|---------------|
+| Channel granularity | One channel per KhaBotCLI instance |
+| Channel naming | `khabot:instance:<instanceId>` |
+| Message routing | Message `name` field: `api:request`, `api:response`, etc. |
+| Correlation | `requestId` field in message data |
+| Auth method | `authCallback` with KhaBot.com-issued TokenRequests |
+| Token TTL | 1 hour, auto-refreshed by SDK |
+| Remote clients | Never connect to Ably directly — KhaBot.com relays |
+| Offline handling | <2 min: auto-resume. >2 min: messages lost, skip stale |
+| Large payloads | Truncate + reference URL for >60 KiB |
+| Validation | Always validate message schema, even trusting KhaBot.com |
+| Encryption | Optional AES-256-CBC per-channel for high-security |
+| Streaming | NOT via Ably — use direct connection if ever needed |