@shumkov/orchestra 0.2.0

package/README.md

@@ -0,0 +1,24 @@
+# @shumkov/orchestra
+
+The transport-agnostic **Claude Code CLI session engine**: spawn interactive `claude`
+sessions, inject messages via the MCP channels bridge, observe turns, and recover —
+independent of which chat network the messages came from.
+
+Extracted from [polygram](https://github.com/shumkov/polygram) after
+[water](https://github.com/shumkov/water) proved the copy works end-to-end against a
+real Claude. Shared by polygram (Telegram) and water (WhatsApp); each keeps its own
+transport, persistence, gate, and delivery.
+
+See [`docs/EXTRACTION.md`](docs/EXTRACTION.md) for the API and the extraction design.
+
+## Use
+
+```js
+const { createProcessFactory, createTmuxRunner, ProcessManager, claudeBin } = require('@shumkov/orchestra');
+// inject your transport delivery (toolDispatcher), surface hint (displayHint),
+// and file cap (maxOutboundFileBytes); the engine stays chat-network-agnostic.
+```
+
+## Licence
+
+MIT.

package/docs/EXTRACTION.md

@@ -0,0 +1,83 @@
+# orchestra — extraction design
+
+`@shumkov/orchestra` is the transport-agnostic **Claude-Code-CLI session engine**,
+extracted from polygram after water proved the copied code works
+(water `docs/SHARED-LIB.md`, proof `scripts/spikes/prove-session-engine.mjs`). Both
+polygram (Telegram) and water (WhatsApp) depend on it; the transport / persistence /
+gate layers stay in each consumer.
+
+## What moves in (the engine)
+
+From polygram@0.17.11 / water's proven copies:
+
+- `process/process.js` — abstract Process base
+- `process/process-manager.js` — weighted LRU pool, spawn/evict/pin, lazy respawn
+- `process/cli-process.js` — the tmux'd claude CLI driver (**2 app-couplings removed**, below)
+- `process/factory.js` — backend selection (cli; sdk is the consumer's escape hatch)
+- `process/channels-bridge.mjs` / `channels-bridge-server.js` / `channels-bridge-protocol.js`
+  — the MCP channels injection protocol
+- `process/hook-settings.js` / `hook-event-tail.js` / `hook-append.js` — turn observability
+- `process/attachment-base.js` — NEW: `DEFAULT_ATTACHMENT_BASE` + path validators (was in
+  the consumer's tool-dispatcher; the staging-dir concept is engine-level)
+- `tmux/*` — tmux lifecycle
+- `claude-bin.js` — pin + vendor the claude binary
+- `process-guard.js`, `async-lock.js`
+- `context-usage.js`, `compaction-warn.js`, `canonical-json.js`
+- `questions/store.js`, `approvals/store.js` — the ask/approval lifecycle stores
+
+## Consumer-provided (injected) — the app boundary
+
+The engine is parameterized by the consumer at construction, so it knows nothing about
+Telegram or WhatsApp:
+
+- `toolDispatcher(call) => {ok, error?, message_id?}` — delivers reply/edit/react on the
+  consumer's transport (already injected in polygram/water today).
+- `displayHint: string` — surface-rendering rules appended to the system prompt
+  (**new option**, replacing cli-process's hard `require('../delivery/display-hint')`).
+- `maxOutboundFileBytes: number` — outbound file cap (**new option**, replacing
+  `require('../attachments').resolveFileCaps`).
+- `claudeBin`, `botName`, `tmuxRunner`, `logger`, `db` (telemetry) — as today.
+
+## The 2 cli-process modifications (the only divergence from the verbatim copy)
+
+1. `WATER_DISPLAY_HINT` import → constructor option `displayHint` (default `''`).
+2. `resolveFileCaps()` import → constructor option `maxOutboundFileBytes`
+   (default 100 MB).
+
+Both are recorded in cli-process's provenance header as the extraction-time API design.
+Everything else is byte-identical to the proven copy.
+
+## Public API (`index.js`)
+
+```js
+const {
+  ProcessManager, CliProcess, createProcessFactory,   // the pool + driver
+  createTmuxRunner, orphanSweep,                        // tmux lifecycle
+  claudeBin,                                            // { ensureVendoredClaudeBin, ... }
+  ChannelsBridgeServer, bridgeProtocol,                 // the channels bridge
+  hookSettings, hookEventTail,                          // observability
+  DEFAULT_ATTACHMENT_BASE, validateAttachmentPath,      // staging-dir validation
+} = require('@shumkov/orchestra');
+```
+
+## Consumer wiring after extraction
+
+water/polygram replace their copied `lib/process/*`, `lib/tmux/*`, `lib/claude-bin.js`,
+`lib/process-guard.js`, `lib/async-lock.js`, `lib/context-usage.js`,
+`lib/compaction-warn.js`, `lib/canonical-json.js`, `lib/questions/store.js`,
+`lib/approvals/store.js` with `require('@shumkov/orchestra')`, and pass `displayHint` +
+`maxOutboundFileBytes` into the factory/CliProcess. Their transport, gate, dispatcher,
+delivery, and ops layers are unchanged.
+
+## Rollout
+
+1. **water first** (this repo → depend on orchestra), re-run unit tests + the
+   real-claude E2E to prove parity. water is not yet in production, so this is low-risk.
+2. **polygram** is a **production revenue system** — its migration to orchestra gets its
+   OWN spec + review + Ivan's merge (SHARED-LIB.md). Not done implicitly here.
+
+## Provenance
+
+Files retain their `// provenance: polygram@0.17.11 <path> (git 746bca6)` headers; the
+extraction only relocates them and applies the 2 documented cli-process options. Keep
+orchestra diff-clean against the pinned polygram so upstream fixes are cheap to pull.

package/index.js

@@ -0,0 +1,57 @@
+// @shumkov/orchestra — the transport-agnostic Claude-Code-CLI session engine.
+// Spawn, inject messages via the MCP channels bridge, observe turns, recover.
+// Extracted from polygram after water proved the copy (see docs/EXTRACTION.md).
+//
+// The engine is parameterized by the consumer (toolDispatcher, displayHint,
+// maxOutboundFileBytes, claudeBin, tmuxRunner) so it knows nothing about the chat
+// transport. Consumers keep their own transport / persistence / gate / delivery.
+
+'use strict';
+
+const { Process, UnsupportedOperationError } = require('./lib/process/process');
+const { ProcessManager, CALLBACK_TO_EVENT } = require('./lib/process/process-manager');
+const { CliProcess } = require('./lib/process/cli-process');
+const sdkProcess = require('./lib/process/sdk-process');
+const { SdkProcess, extractAssistantText, sumUsage, makeInputController } = sdkProcess;
+const { createProcessFactory, pickBackend } = require('./lib/process/factory');
+const { ChannelsBridgeServer } = require('./lib/process/channels-bridge-server');
+const bridgeProtocol = require('./lib/process/channels-bridge-protocol');
+const hookSettings = require('./lib/process/hook-settings');
+const hookEventTail = require('./lib/process/hook-event-tail');
+const startupGate = require('./lib/tmux/startup-gate');
+const attachmentBase = require('./lib/process/attachment-base');
+
+const { createTmuxRunner } = require('./lib/tmux/tmux-runner');
+const orphanSweep = require('./lib/tmux/orphan-sweep');
+const logTail = require('./lib/tmux/log-tail');
+const pollScheduler = require('./lib/tmux/poll-scheduler');
+
+const claudeBin = require('./lib/claude-bin');
+const processGuard = require('./lib/process-guard');
+const { createAsyncLock } = require('./lib/async-lock');
+const contextUsage = require('./lib/context-usage');
+const compactionWarn = require('./lib/compaction-warn');
+const canonicalJson = require('./lib/canonical-json');
+const questionsStore = require('./lib/questions/store');
+const approvalsStore = require('./lib/approvals/store');
+
+module.exports = {
+  // pool + driver
+  Process, UnsupportedOperationError, ProcessManager, CALLBACK_TO_EVENT, CliProcess, SdkProcess, extractAssistantText, sumUsage, makeInputController, createProcessFactory, pickBackend,
+  // channels bridge (MCP injection protocol)
+  ChannelsBridgeServer, bridgeProtocol,
+  // tmux lifecycle
+  createTmuxRunner, orphanSweep, pollScheduler, logTail, startupGate,
+  // claude binary pin+vendor
+  claudeBin,
+  // observability
+  hookSettings, hookEventTail,
+  // process safety
+  processGuard, createAsyncLock,
+  // claude context management
+  contextUsage, compactionWarn,
+  // ask / approval lifecycle stores
+  questionsStore, approvalsStore, canonicalJson,
+  // attachment staging-dir validation
+  attachmentBase,
+};

package/lib/approvals/store.js

@@ -0,0 +1,251 @@
+// provenance: polygram@0.17.11 lib/approvals/store.js (git 746bca6) — verbatim: env prefix WATER_, bridge name water-bridge, vendor path (SHARED-LIB.md).
+/**
+ * Approvals - inline keyboard gating for destructive tool calls.
+ *
+ * Claude Code fires a PreToolUse hook. The hook RPCs to the polygram daemon.
+ * The daemon inserts a pending row, posts [Approve]/[Deny] to the admin
+ * chat, and blocks on the operator's click (or a timeout).
+ *
+ * Persistence: `pending_approvals` row captures the whole lifecycle so we
+ * keep an audit trail even if polygram restarts. Rows never get deleted;
+ * 'pending' rows at boot are swept into 'timeout'.
+ */
+
+const crypto = require('crypto');
+const { canonicalizeToolInput } = require('../canonical-json');
+
+const DEFAULT_TIMEOUT_MS = 5 * 60 * 1000;
+// 16 random bytes → 22 base64url chars ≈ 128 bits of entropy. Prevents
+// brute-force guessing of approval callback tokens by anyone in the admin
+// chat (old 6-char value was ~36 bits, within chat-storm reach).
+const TOKEN_BYTES = 16;
+
+function digestInput(input) {
+  // Canonicalise object inputs so key-order doesn't change the digest.
+  // Pre-fix `JSON.stringify({a:1,b:2})` and `JSON.stringify({b:2,a:1})`
+  // produced different hashes — the dedup contract assumed logical
+  // equivalence but the impl was order-sensitive, so an SDK that
+  // re-serialised the input between turns would dedup-miss.
+  const json = typeof input === 'string'
+    ? input
+    : JSON.stringify(canonicalizeToolInput(input));
+  return crypto.createHash('sha256').update(json).digest('hex').slice(0, 16);
+}
+
+function newToken() {
+  return crypto.randomBytes(TOKEN_BYTES).toString('base64url');
+}
+
+/**
+ * Constant-time compare. Different lengths → false without timing leak
+ * (timingSafeEqual itself throws on length mismatch).
+ */
+function tokensEqual(a, b) {
+  if (typeof a !== 'string' || typeof b !== 'string') return false;
+  const ab = Buffer.from(a);
+  const bb = Buffer.from(b);
+  if (ab.length !== bb.length) return false;
+  return crypto.timingSafeEqual(ab, bb);
+}
+
+/**
+ * Translate a Claude-Code-style permission pattern into a RegExp.
+ * Supported forms:
+ *   "Bash"                       - any Bash tool call
+ *   "Bash(rm *)"                 - Bash whose first-argument string matches glob
+ *   "mcp__shopify__order_cancel" - exact MCP tool name
+ *   "mcp__*__invoice_create"     - glob on segment
+ *   "WebFetch"                   - any WebFetch
+ * `*` matches anything non-greedily within a segment, including spaces.
+ */
+function patternToRegex(pattern) {
+  const trimmed = String(pattern).trim();
+  const parenIdx = trimmed.indexOf('(');
+  const toolPart = parenIdx === -1 ? trimmed : trimmed.slice(0, parenIdx);
+  const argPart = parenIdx === -1
+    ? null
+    : trimmed.slice(parenIdx + 1, trimmed.lastIndexOf(')'));
+
+  const escape = (s) => s.replace(/[.+?^${}()|[\]\\]/g, '\\$&');
+  const globToRe = (s) => escape(s).replace(/\*/g, '.*');
+
+  const toolRe = new RegExp(`^${globToRe(toolPart)}$`);
+  const argRe = argPart == null ? null : new RegExp(`^${globToRe(argPart)}$`);
+  return { toolRe, argRe, raw: trimmed };
+}
+
+/**
+ * Check whether a tool call matches any of the configured patterns.
+ * `toolInput` is the original params object. We match on:
+ *   - Bash: first positional command (`command` param, space-split first token
+ *           or whole string for arg globs that include spaces)
+ *   - WebFetch: `url`
+ *   - others: the JSON stringification (coarse but safe default)
+ */
+function matchesAnyPattern(toolName, toolInput, patterns = []) {
+  const input = toolInput || {};
+  for (const raw of patterns) {
+    const p = patternToRegex(raw);
+    if (!p.toolRe.test(toolName)) continue;
+    if (!p.argRe) return { matched: true, pattern: p.raw };
+    let candidate = '';
+    if (toolName === 'Bash') {
+      candidate = input.command || '';
+    } else if (toolName === 'WebFetch') {
+      candidate = input.url || '';
+    } else {
+      candidate = typeof input === 'string' ? input : JSON.stringify(input);
+    }
+    if (p.argRe.test(candidate)) return { matched: true, pattern: p.raw };
+  }
+  return { matched: false };
+}
+
+function createStore(rawDb, now = () => Date.now()) {
+  const insertStmt = rawDb.prepare(`
+    INSERT INTO pending_approvals (
+      bot_name, turn_id, tool_use_id, requester_chat_id, approver_chat_id,
+      tool_name, tool_input_json, tool_input_digest,
+      callback_token, requested_ts, timeout_ts
+    ) VALUES (
+      @bot_name, @turn_id, @tool_use_id, @requester_chat_id, @approver_chat_id,
+      @tool_name, @tool_input_json, @tool_input_digest,
+      @callback_token, @requested_ts, @timeout_ts
+    )
+  `);
+  // 0.9.0-cleanup commit 10: stronger dedup via tool_use_id when the
+  // SDK provides one. tool_use_id is the SDK's stable per-call ID;
+  // unlike the legacy (turn_id, tool_input_digest) tuple, it survives
+  // JSON-key reordering between retries within a turn. Migration 010
+  // added the column + partial index `idx_pending_approvals_tool_use_id`
+  // which had been unused since rc.6 because no insert path populated
+  // the column. issue() now does.
+  const findDedupByToolUseIdStmt = rawDb.prepare(`
+    SELECT * FROM pending_approvals
+     WHERE bot_name = ? AND tool_use_id = ? AND status = 'pending'
+     LIMIT 1
+  `);
+  const findDedupStmt = rawDb.prepare(`
+    SELECT * FROM pending_approvals
+     WHERE bot_name = ? AND turn_id IS ? AND tool_input_digest = ?
+       AND status = 'pending'
+     LIMIT 1
+  `);
+  const setApproverMsgStmt = rawDb.prepare(`
+    UPDATE pending_approvals SET approver_msg_id = ? WHERE id = ?
+  `);
+  const resolveStmt = rawDb.prepare(`
+    UPDATE pending_approvals
+       SET status = @status,
+           decided_ts = @decided_ts,
+           decided_by_user_id = @decided_by_user_id,
+           decided_by_user = @decided_by_user,
+           reason = @reason
+     WHERE id = @id AND status = 'pending'
+  `);
+  const getByIdStmt = rawDb.prepare(`SELECT * FROM pending_approvals WHERE id = ?`);
+  const listTimedOutStmt = rawDb.prepare(`
+    SELECT * FROM pending_approvals
+     WHERE status = 'pending' AND timeout_ts < ?
+  `);
+  const listPendingStmt = rawDb.prepare(`
+    SELECT * FROM pending_approvals
+     WHERE bot_name = ? AND status = 'pending'
+     ORDER BY requested_ts DESC
+  `);
+
+  return {
+    issue({
+      bot_name, turn_id = null, tool_use_id = null,
+      requester_chat_id, approver_chat_id,
+      tool_name, tool_input, timeoutMs = DEFAULT_TIMEOUT_MS,
+    }) {
+      if (!bot_name) throw new Error('bot_name required');
+      if (!requester_chat_id) throw new Error('requester_chat_id required');
+      if (!approver_chat_id) throw new Error('approver_chat_id required');
+      if (!tool_name) throw new Error('tool_name required');
+
+      const tool_input_json = typeof tool_input === 'string'
+        ? tool_input
+        : JSON.stringify(tool_input || {});
+      const tool_input_digest = digestInput(tool_input_json);
+      const requested_ts = now();
+      const timeout_ts = requested_ts + timeoutMs;
+      const callback_token = newToken();
+
+      // Dedup: wrap find + insert in a single BEGIN IMMEDIATE transaction so
+      // two concurrent hook calls (same turn, same input) can't both miss
+      // the existing row and insert two. SQLite's UPSERT would also work if
+      // we added a UNIQUE partial index in a migration; keeping this in
+      // application code avoids a schema bump.
+      //
+      // 0.9.0: prefer dedup by SDK's stable tool_use_id when available.
+      // The legacy (turn_id, tool_input_digest) tuple survives only when
+      // the SDK doesn't provide a tool_use_id (cron-driven sends, IPC
+      // callers from older Claude versions). Both code paths route
+      // through the same INSERT below; the only thing that varies is
+      // which existing row counts as a "match."
+      let row, reused = false;
+      const tx = rawDb.transaction(() => {
+        const existing = tool_use_id
+          ? findDedupByToolUseIdStmt.get(bot_name, tool_use_id)
+          : findDedupStmt.get(bot_name, turn_id, tool_input_digest);
+        if (existing) { row = existing; reused = true; return; }
+        const res = insertStmt.run({
+          bot_name,
+          turn_id,
+          tool_use_id,
+          requester_chat_id: String(requester_chat_id),
+          approver_chat_id: String(approver_chat_id),
+          tool_name,
+          tool_input_json,
+          tool_input_digest,
+          callback_token,
+          requested_ts,
+          timeout_ts,
+        });
+        row = getByIdStmt.get(res.lastInsertRowid);
+      });
+      tx.immediate();  // BEGIN IMMEDIATE → write lock before the SELECT
+      if (reused) return { ...row, reused: true };
+      return row;
+    },
+
+    setApproverMsgId(id, msg_id) {
+      return setApproverMsgStmt.run(msg_id, id).changes;
+    },
+
+    resolve({ id, status, decided_by_user_id = null, decided_by_user = null, reason = null }) {
+      if (!['approved', 'denied', 'timeout', 'cancelled'].includes(status)) {
+        throw new Error(`bad status: ${status}`);
+      }
+      const res = resolveStmt.run({
+        id,
+        status,
+        decided_ts: now(),
+        decided_by_user_id,
+        decided_by_user,
+        reason,
+      });
+      return res.changes;
+    },
+
+    getById(id) { return getByIdStmt.get(id); },
+
+    listPending(bot_name) { return listPendingStmt.all(bot_name); },
+
+    sweepTimedOut() {
+      return listTimedOutStmt.all(now());
+    },
+  };
+}
+
+module.exports = {
+  createStore,
+  digestInput,
+  newToken,
+  tokensEqual,
+  patternToRegex,
+  matchesAnyPattern,
+  DEFAULT_TIMEOUT_MS,
+};

package/lib/async-lock.js

@@ -0,0 +1,50 @@
+// provenance: polygram@0.17.11 lib/async-lock.js (git 746bca6) — verbatim: env prefix WATER_, bridge name water-bridge, vendor path (SHARED-LIB.md).
+/**
+ * Per-key chain lock. Each acquire() returns a release function; the next
+ * acquire() awaits the previous one's release.
+ *
+ * Used by polygram to serialise stdin writes per session. Pre-work
+ * (attachment download, voice transcription, prompt formatting) runs
+ * concurrently; only the stdin write itself is serialised so Claude
+ * reads messages in arrival order and replies come out in the same
+ * order.
+ *
+ * Deliberately minimal — no timeouts, no cancellation, no fairness
+ * guarantees beyond FIFO. Callers are expected to ALWAYS call release,
+ * even on error paths, or the lock leaks (blocks all future acquires
+ * for that key forever).
+ */
+
+function createAsyncLock() {
+  const chains = new Map(); // key → Promise of last release
+
+  return {
+    async acquire(key) {
+      const prev = chains.get(key) || Promise.resolve();
+      let release;
+      const next = new Promise((resolve) => { release = resolve; });
+      // Save the chain-entry promise so the cleanup branch can compare
+      // against the SAME reference. Pre-fix this re-evaluated
+      // `prev.then(() => next)` (a fresh promise each call), so the
+      // === compare was always false and the Map leaked one entry per
+      // unique key.
+      const myEntry = prev.then(() => next);
+      chains.set(key, myEntry);
+      await prev;
+      // Return a wrapper that also clears the chain entry when this is
+      // the last holder — avoids the Map growing unbounded across the
+      // lifetime of the process. Idempotent: a double-release call is
+      // harmless (release() is a Promise resolver; calling resolve
+      // twice is a no-op).
+      return () => {
+        if (chains.get(key) === myEntry) {
+          chains.delete(key);
+        }
+        release();
+      };
+    },
+    get size() { return chains.size; },
+  };
+}
+
+module.exports = { createAsyncLock };

package/lib/canonical-json.js

@@ -0,0 +1,63 @@
+// provenance: polygram@0.17.11 lib/canonical-json.js (git 746bca6) — verbatim: env prefix WATER_, bridge name water-bridge, vendor path (SHARED-LIB.md).
+/**
+ * Canonical-JSON stringification for chat_tool_decisions dedup.
+ *
+ * Used by polygram.js's canUseTool flow (rc.6 Phase 2 step 6):
+ *   - lookup key for `chat_tool_decisions match_type='exact'`
+ *   - input_pattern stored on "Always allow / Always deny" clicks
+ *
+ * Why canonical: Claude can reorder JSON keys between retries of
+ * the same tool call (different SDK versions, different temperature
+ * sampling). Without canonicalisation, the dedup digest would
+ * differ for semantically-identical calls and the user would see
+ * the same approval card twice (v4 plan §6.6 ship-breaker M8
+ * mitigation).
+ *
+ * Properties:
+ *   - Keys sorted alphabetically at every nesting level
+ *   - Arrays preserve order (only object keys are sorted)
+ *   - No whitespace in output
+ *   - null / undefined / primitive inputs round-trip via JSON.stringify
+ *
+ * NOT a full JSON canonicalisation spec (RFC 8785 / I-D
+ * cyberphone-json-canonicalization-scheme); we don't normalise
+ * number representations (1.0 vs 1, exponents) or string escapes.
+ * Sufficient for SDK-shaped tool inputs which are well-formed JSON
+ * objects with string keys.
+ */
+
+'use strict';
+
+function canonicalizeToolInput(input) {
+  if (input == null || typeof input !== 'object') {
+    return JSON.stringify(input);
+  }
+  // Track in-flight (currently-on-stack) nodes to detect circular
+  // references. WeakSet membership marks "we are still inside this
+  // node"; we drop the entry after finishing recursion so DAG
+  // shapes (shared subtrees that aren't cycles) round-trip fine.
+  // Pre-fix sortRec recursed forever on `{a: 1, self: <self>}`
+  // and crashed the daemon — DoS path if any tool ever produces
+  // self-referencing input. Now throws a clean TypeError matching
+  // JSON.stringify's own "Converting circular structure to JSON".
+  const onStack = new WeakSet();
+  const sortRec = (v) => {
+    if (Array.isArray(v)) {
+      if (onStack.has(v)) throw new TypeError('Converting circular structure to JSON');
+      onStack.add(v);
+      const result = v.map(sortRec);
+      onStack.delete(v);
+      return result;
+    }
+    if (v == null || typeof v !== 'object') return v;
+    if (onStack.has(v)) throw new TypeError('Converting circular structure to JSON');
+    onStack.add(v);
+    const out = {};
+    for (const k of Object.keys(v).sort()) out[k] = sortRec(v[k]);
+    onStack.delete(v);
+    return out;
+  };
+  return JSON.stringify(sortRec(input));
+}
+
+module.exports = { canonicalizeToolInput };