shadow-claw 1.23.4 → 1.24.0

package/dist/public/docs/subsystems/control-plane.md

@@ -0,0 +1,158 @@
+# Control Plane & Client Bridge
+
+> Bidirectional communication gateway connecting CLI commands, external MCP clients, and connected browser/Electron tabs.
+
+**Source:** `src/server/control-plane.ts` · `src/server/client-registry.ts` · `src/core/control-plane-client.ts` · `bin/commands/`
+
+---
+
+## Overview
+
+ShadowClaw provides a real-time **Control Plane** and client bridge that enables external tools and CLI commands to interact directly with connected browser or Electron clients. Connected clients act as persistent execution surfaces with access to local browser features, WebMCP tools, and IndexedDB/OPFS storage.
+
+---
+
+## Architecture & Transports
+
+The Control Plane supports three interchangeable communication transports:
+
+1. **Server-Sent Events + HTTP POST (`sse`, Default & Recommended):**
+   - **Server $\rightarrow$ Client:** Persistent HTTP SSE stream (`GET /api/control/events?clientId=...`).
+   - **Client $\rightarrow$ Server:** Standard HTTP JSON requests (`POST /api/control/messages`).
+   - **Why Default:** Extremely resilient across corporate firewalls, reverse proxies (Nginx, Caddy, Traefik), and cloud edge/CDN setups that do not support or maintain WebSocket connection upgrades.
+2. **WebSocket (`websocket`, Opt-in):**
+   - Full-duplex WebSocket connection to `/ws/control?token=...`.
+   - Useful for low-latency direct connections or environments where WebSockets are preferred.
+3. **WebRTC DataChannel (`webrtc`, Peer-to-Peer):**
+   - Direct P2P browser-to-CLI communication via PeerJS DataChannels.
+   - CLI coordinates via `shadow-claw webrtc listen` and the local IPC socket bridge (`.cache/webrtc-ipc.sock`).
+   - Allows headless commands and automation without requiring an active control-plane HTTP/SSE session.
+
+```mermaid
+graph TB
+    subgraph CLI ["CLI & External Tools"]
+        CLI_BIN["shadow-claw CLI (clients, send, backup, tasks)"]
+        CLI_LISTEN["webrtc listen (IPC Socket: .cache/webrtc-ipc.sock)"]
+        EXT_MCP["External MCP Clients (Hermes, Claude, Cursor)"]
+    end
+
+    subgraph Server ["Express Server & Control Plane"]
+        SSE_ENDPOINT["/api/control/events (SSE Stream)"]
+        MSG_ENDPOINT["/api/control/messages (HTTP POST)"]
+        WS_ENDPOINT["/ws/control (WebSocket Server)"]
+        PEERJS_SERVER["/peerjs/* (PeerJS Signaling Server)"]
+        REST_ROUTES["/api/control/* (REST API)"]
+        REGISTRY["Client Registry (SQLite: clients.db)"]
+        TOKEN_STORE["Control Token Auth"]
+        PUSH_WAKE["Web Push Remote Command Wakeup"]
+    end
+
+    subgraph Browser ["Connected Browser Client"]
+        CP_CLIENT["src/core/control-plane-client.ts<br>(SSE Default / WS Configurable)"]
+        PEERJS_CHANNEL["src/subsystems/channels/peerjs.ts<br>(WebRTC DataChannel)"]
+        ORCH["Orchestrator"]
+        WEBMCP["WebMCP (document.modelContext)"]
+        STORAGE["OPFS Storage"]
+    end
+
+    CLI_BIN <-->|REST & WebSocket| WS_ENDPOINT
+    CLI_BIN <-->|Local IPC Socket| CLI_LISTEN
+    CLI_LISTEN <-->|WebRTC DataChannel| PEERJS_CHANNEL
+    EXT_MCP <-->|Standard MCP| REST_ROUTES
+    CP_CLIENT -->|SSE Stream (Commands)| SSE_ENDPOINT
+    CP_CLIENT -->|HTTP POST (Results)| MSG_ENDPOINT
+    CP_CLIENT -.->|Optional WebSocket| WS_ENDPOINT
+    PEERJS_CHANNEL <-->|Signaling| PEERJS_SERVER
+    WS_ENDPOINT --> REGISTRY
+    SSE_ENDPOINT --> REGISTRY
+    REGISTRY --> TOKEN_STORE
+    WS_ENDPOINT --> PUSH_WAKE
+
+    CP_CLIENT --> ORCH
+    CP_CLIENT --> WEBMCP
+    CP_CLIENT --> STORAGE
+    PEERJS_CHANNEL --> ORCH
+```
+
+---
+
+## Protocol Specification
+
+All messages are JSON objects matching the `ControlMessage` envelope:
+
+```ts
+interface ControlMessage {
+  id: string; // ULID for message/response correlation
+  type: string; // Message discriminator
+  replyTo?: string; // Request ID when responding
+  payload: unknown; // Type-specific payload
+}
+```
+
+### Message Types
+
+| Message Type        | Direction       | Description                                                                                               |
+| :------------------ | :-------------- | :-------------------------------------------------------------------------------------------------------- |
+| `client:register`   | Client → Server | Client registration with ID, device label, version, transport, and capabilities                           |
+| `server:registered` | Server → Client | Acknowledgement of successful registration                                                                |
+| `client:heartbeat`  | Client → Server | Periodic heartbeat to refresh active client timestamp                                                     |
+| `command:execute`   | Server → Client | Dispatches a command action (`send-message`, `list-tasks`, `invoke-tool`, `read-state`, `trigger-backup`) |
+| `command:result`    | Client → Server | Response containing execution status, result data, or error message                                       |
+
+### Command Payloads
+
+#### `command:execute` Payload
+
+```ts
+interface CommandExecutePayload {
+  commandId: string;
+  action:
+    | "send-message"
+    | "list-tasks"
+    | "trigger-backup"
+    | "invoke-tool"
+    | "read-state";
+  args: Record<string, unknown>;
+}
+```
+
+#### `command:result` Payload
+
+```ts
+interface CommandResultPayload {
+  commandId: string;
+  success: boolean;
+  data?: unknown;
+  error?: string;
+}
+```
+
+---
+
+## Authentication & Security
+
+1. **Control Token:** Connections must supply a secret control token via query parameter (`?token=...`), header (`x-control-token`), or Bearer authorization header (`Authorization: Bearer <token>`).
+2. **Token Generation:** On server startup, a token is read from `SHADOWCLAW_CONTROL_TOKEN` or generated and persisted in SQLite (`clients.db` metadata table) and `.cache/control-token.json`.
+3. **WebRTC CLI Peer Identity & Trust:** CLI clients connecting over WebRTC identify using a persistent peer ID in `.cache/cli-peer-id`. This ID can be retrieved or renewed using `npx shadow-claw peer-id [--renew|--set <id>]` and must be added in the browser under **Settings → WebRTC/PeerJS → Trusted Peer IDs** when peer restrictions are enabled.
+4. **Localhost Binding:** Binds to `127.0.0.1` by default.
+
+---
+
+## Browser Integration & Settings
+
+Browser tabs connect automatically to the Control Plane on startup using `createDefaultControlPlaneClient` when served from a server environment.
+
+### Transport Configuration
+
+Users can toggle between transports in **Settings $\rightarrow$ Integrations $\rightarrow$ Control Plane $\rightarrow$ Control Plane Transport**:
+
+- `Server-Sent Events (SSE + HTTP POST)` — Default
+- `WebSocket (Full Duplex)` — Opt-in
+
+### Registered Command Handlers
+
+- `send-message`: Submits message/prompt into Orchestrator queue (supports optional `groupId` e.g. `br:main` or `peer:<peerId>`).
+- `list-tasks`: Queries scheduled tasks from IndexedDB.
+- `read-state`: Returns current active conversation group and state.
+- `invoke-tool`: Invokes WebMCP (`document.modelContext`) tools directly.
+- `trigger-backup`: Initiates workspace upload via `BackupController`.

package/dist/public/docs/subsystems/notifications.md

@@ -3,7 +3,7 @@
 > Web Push notifications delivered via VAPID, a server-side SQLite task scheduler,
 > and a client-side cron evaluator — so tasks fire even when no browser tab is open.
 
-**Source:** `src/subsystems/notifications/` · `src/subsystems/tools/task-scheduler.ts` · `src/subsystems/tools/cron.ts` · `src/service-worker/push-handler.ts`
+**Source:** `src/subsystems/notifications/` · `src/subsystems/notifications/push-routes.ts` · `src/subsystems/tools/task-scheduler.ts` · `src/subsystems/tools/cron.ts` · `src/service-worker/push-handler.ts`
 
 ## System Overview
 
@@ -14,6 +14,7 @@ graph TD
     SW["Service Worker<br>push-handler.ts"]
     CS["ClientTaskScheduler<br>task-scheduler.ts"]
     PC["push-client.ts<br>Subscribe / Unsubscribe"]
+    OW["Open Windows<br>(postMessage)"]
   end
 
   subgraph Server ["Express / Electron Server"]
@@ -22,6 +23,7 @@ graph TD
     TSR["task-schedule-routes.ts<br>/schedule/tasks/* endpoints"]
     TSS["task-schedule-store.ts<br>SQLite scheduled tasks"]
     Sched["ServerTaskScheduler<br>60s cron ticks"]
+    CP["Control Plane<br>/push/command"]
   end
 
   subgraph Agent ["Agent"]
@@ -35,9 +37,11 @@ graph TD
   TT --> TSR
   TSR --> TSS
   TSS --> Sched
-  Sched -->|Web Push| SW
+  Sched -->|Web Push: scheduled-task| SW
   SW -->|relay to window| CS
   CS --> TT
+  CP -->|Web Push: remote-command| SW
+  SW -->|postMessage: remote-command-trigger| OW
 ```
 
 ## Push Notifications
@@ -75,6 +79,54 @@ On `push` event:
 
 The service worker uses a `BroadcastChannel("shadowclaw-push")` to relay push events to any open windows. The orchestrator listens on this channel and triggers the agent invocation for scheduled tasks.
 
+---
+
+## Remote Command Push Wakeup
+
+**Files:** `src/service-worker/push-handler.ts` · `src/subsystems/notifications/push-routes.ts`
+
+### Purpose
+
+The control plane may need to dispatch a command to a specific ShadowClaw client that is currently dormant — i.e. not connected to the WebSocket. Rather than dropping the command or waiting for the client to reconnect, the server sends a Web Push notification of type `remote-command` to wake the client up and relay the payload.
+
+### How it works
+
+```mermaid
+sequenceDiagram
+  participant CP as Control Plane
+  participant PR as push-routes.ts<br>POST /push/command
+  participant PS as push-store.ts<br>getAllSubscriptions()
+  participant WP as web-push
+  participant SW as Service Worker<br>push-handler.ts
+  participant OW as Open Windows
+
+  CP->>PR: POST /push/command { clientId, action, args, prompt }
+  PR->>PS: getAllSubscriptions() [ORDER BY … id DESC]
+  PS-->>PR: subscriptions[]
+  loop each subscription
+    PR->>WP: sendNotification(subscription, { type: "remote-command", clientId, action, args, prompt })
+  end
+  WP-->>SW: push event
+  SW->>OW: client.postMessage({ type: "remote-command-trigger", clientId, action, args, prompt })
+  SW->>SW: showNotification("ShadowClaw — Remote Command")
+```
+
+1. **Server:** The control plane POSTs to `/push/command` with `{ clientId, action, args, prompt }`.
+2. **`push-routes.ts`:** Calls `getAllSubscriptions()` (results are stably ordered by `id DESC`) and fans out a Web Push to every subscribed endpoint with payload `{ type: "remote-command", clientId, action, args, prompt }`.
+3. **Service worker (`push-handler.ts`):** On receiving a push whose `data.type === 'remote-command'`, the handler:
+   - Iterates all open `WindowClient`s and calls `client.postMessage({ type: 'remote-command-trigger', clientId, action, args, prompt })`.
+   - Shows an OS notification titled **`ShadowClaw — Remote Command`** so the user is informed even if no window catches the message.
+
+### Client-side handling
+
+Open windows listen for `message` events from the service worker. When a message of type `'remote-command-trigger'` arrives, the orchestrator can act on `action`, `args`, and `prompt` in the same way as a locally-initiated command. The `clientId` field lets the handler route the command only to the intended client and ignore it otherwise.
+
+### Push subscription ordering
+
+`push-store.ts`'s `getAllSubscriptions()` query now applies a secondary `ORDER BY … id DESC` clause, ensuring that subscriptions are returned in a stable, deterministic order. This prevents non-deterministic fan-out behaviour when multiple subscriptions exist for the same browser profile.
+
+---
+
 ## Client-Side Task Scheduler
 
 **File:** `src/subsystems/tools/task-scheduler.ts`

package/dist/public/docs/subsystems/reactive-ui.md

@@ -216,6 +216,7 @@ _profiles (State) ──────────┘
 | Page Header              | `<shadow-claw-page-header>`    | Reusable mobile-first header                                              |
 | Settings — Accounts      | `<shadow-claw-accounts>`       | General account connection and OAuth flows                                |
 | Settings — Channels      | `<shadow-claw-channel-config>` | Third-party messaging channels configuration                              |
+| Settings — Control Plane | `<shadow-claw-control-plane>`  | CLI and remote control plane client configuration                         |
 | Settings — Git           | `<shadow-claw-git>`            | Git token and proxy settings                                              |
 | Settings — Integrations  | `<shadow-claw-integrations>`   | Special API integrations and Web Search                                   |
 | Settings — LLM           | `<shadow-claw-llm>`            | Provider and model settings                                               |

package/dist/public/downloadGroupFile-By0GR6eV.js

@@ -0,0 +1 @@
+import{xn as e}from"./orchestrator-3h9xxHUm.js";import{t}from"./getGroupDir-P1h9wl6S.js";async function n(n,r,i){let a=await t(n,r),{dirs:o,filename:s}=e(i),c=a;for(let e of o)c=await c.getDirectoryHandle(e);let l=await(await c.getFileHandle(s)).getFile(),u=URL.createObjectURL(l),d=document.createElement(`a`);d.href=u,d.download=s,document.body.appendChild(d),d.click(),document.body.removeChild(d),URL.revokeObjectURL(u)}export{n as t};

package/dist/public/e2e/README.md

@@ -380,6 +380,7 @@ Settings panel controller.
 - `sidebarTasksHiddenToggle()` — Toggle visibility for Tasks navigation item in sidebar
 - `sidebarFilesHiddenToggle()` — Toggle visibility for Files navigation item in sidebar
 - `taskServerConfig()` — Task server configuration component (`<shadow-claw-task-server>`)
+- `controlPlaneConfig()` — Control plane configuration component (`<shadow-claw-control-plane>`)
 
 ## Shared Utilities (`shared/index.ts`)
 
@@ -454,11 +455,14 @@ The build, CLI, and prerender test suites live under `bin/` and run through the
 project Jest configuration. The build integration test covers both the normal
 `pages/` layout and a project with no `pages/` directory (which publishes the
 built-in `index.html` and `MEMORY.md` defaults), as well as the unified `shadow-claw`
-CLI tool commands (`build`, `dev`, `run`, `serve`, `init`, `--version`).
+CLI tool commands (`build`, `dev`, `run`, `serve`, `init`, `clients`, `send`, `backup`, `tasks`, `webrtc`, `peer-id`, `--version`).
 
 ```bash
 NODE_OPTIONS="--no-warnings --experimental-vm-modules" \
   npx jest --runInBand bin/cli.test.mjs \
+  bin/commands/peer-id.test.mjs \
+  bin/utils/control-client.test.mjs \
+  bin/utils/webrtc-control-client.test.mjs \
   bin/build/build.test.mjs \
   bin/prerender-dsd-shell/prerender-dsd-shell.test.mjs \
   bin/prerender-pretty-paths/prerender-pretty-paths.test.mjs

package/dist/public/{e2e-bridge-BQ4KzUt5.js → e2e-bridge-IGWsRl4i.js}

@@ -1 +1 @@
-import{W as e,it as t,tt as n,v as r}from"./orchestrator-DRfEudXk.js";function i(i,a){globalThis.__SHADOWCLAW_E2E__={isReady(){return i.ready},getDb(){return a.db},getActiveGroupId(){return i.activeGroupId},async createConversation(e){let t=a.db;if(!t)throw Error(`[E2E Bridge] DB not ready`);return i.createConversation(t,e)},async switchConversation(e){let t=a.db;if(!t)throw Error(`[E2E Bridge] DB not ready`);await i.switchConversation(t,e)},async loadTasks(){let e=a.db;if(!e)throw Error(`[E2E Bridge] DB not ready`);await i.loadTasks(e)},async configureProvider(o,s,c,l){let u=i.orchestrator,d=a.db;if(!u||!d)throw Error(`[E2E Bridge] Orchestrator or DB not ready`);await t(u,d,o,{loadApiKeyForProvider:u.loadApiKeyForProvider.bind(u),getApiKeyForHeaders:()=>e(u)}),await u.setApiKey(d,s),await n(u,d,c),await r(u,d,l)}}}function a(){return globalThis.__SHADOWCLAW_E2E_ENABLE__===!0}export{a as n,i as t};
+import{W as e,it as t,tt as n,v as r}from"./orchestrator-3h9xxHUm.js";function i(i,a){globalThis.__SHADOWCLAW_E2E__={isReady(){return i.ready},getDb(){return a.db},getActiveGroupId(){return i.activeGroupId},async createConversation(e){let t=a.db;if(!t)throw Error(`[E2E Bridge] DB not ready`);return i.createConversation(t,e)},async switchConversation(e){let t=a.db;if(!t)throw Error(`[E2E Bridge] DB not ready`);await i.switchConversation(t,e)},async loadTasks(){let e=a.db;if(!e)throw Error(`[E2E Bridge] DB not ready`);await i.loadTasks(e)},async configureProvider(o,s,c,l){let u=i.orchestrator,d=a.db;if(!u||!d)throw Error(`[E2E Bridge] Orchestrator or DB not ready`);await t(u,d,o,{loadApiKeyForProvider:u.loadApiKeyForProvider.bind(u),getApiKeyForHeaders:()=>e(u)}),await u.setApiKey(d,s),await n(u,d,c),await r(u,d,l)}}}function a(){return globalThis.__SHADOWCLAW_E2E_ENABLE__===!0}export{a as n,i as t};

package/dist/public/file-viewer-D0eflgt6.js

@@ -0,0 +1 @@
+import{o as e}from"./config-CrHchneq.js";import{n as t}from"./toast-60iDlgiH.js";import{_n as n,bn as r,xn as i}from"./orchestrator-3h9xxHUm.js";import{t as a}from"./getGroupDir-P1h9wl6S.js";async function o(e,t,n){let r=await a(e,t),{dirs:o,filename:s}=i(n),c=r;for(let e of o)c=await c.getDirectoryHandle(e);return await(await c.getFileHandle(s)).getFile()}const s=new class{_file;constructor(){this._file=new t.State(null)}closeFile(){this._file.set(null)}get file(){return this._file.get()}getFile(){return this.file}getPreviewBinaryMimeType(e){return{"3gp":`video/3gpp`,"7z":`application/x-7z-compressed`,aac:`audio/aac`,apng:`image/apng`,avi:`video/x-msvideo`,avif:`image/avif`,bin:`application/octet-stream`,bmp:`image/bmp`,db:`application/vnd.sqlite3`,dll:`application/octet-stream`,dmg:`application/octet-stream`,dylib:`application/octet-stream`,exe:`application/vnd.microsoft.portable-executable`,flac:`audio/flac`,flv:`video/x-flv`,gif:`image/gif`,gz:`application/gzip`,heic:`image/heic`,heif:`image/heif`,ico:`image/x-icon`,iso:`application/octet-stream`,jpeg:`image/jpeg`,jpg:`image/jpeg`,m4a:`audio/mp4`,m4v:`video/mp4`,mkv:`video/x-matroska`,mov:`video/mp4`,mp3:`audio/mpeg`,mp4:`video/mp4`,oga:`audio/ogg`,ogg:`audio/ogg`,ogv:`video/ogg`,png:`image/png`,rar:`application/vnd.rar`,so:`application/octet-stream`,sqlite:`application/vnd.sqlite3`,sqlite3:`application/vnd.sqlite3`,tar:`application/x-tar`,tif:`image/tiff`,tiff:`image/tiff`,wasm:`application/wasm`,wav:`audio/wav`,weba:`audio/webm`,webm:`video/webm`,webp:`image/webp`,wmv:`video/x-ms-wmv`,zip:`application/zip`}[e.toLowerCase().split(`.`).pop()||``]||``}async openFile(t,i,a=e){try{let e=i.split(`/`).pop()||i,s=/\.pdf$/i.test(e),c=this.getPreviewBinaryMimeType(e);if(s){let n=await r(t,a,i);this._file.set({name:e,path:i,groupId:a,content:``,kind:`pdf`,binaryContent:n,mimeType:`application/pdf`});return}if(c){let n=await o(t,a,i);this._file.set({name:e,path:i,groupId:a,content:``,kind:`binary`,binaryContent:null,nativeFile:n,mimeType:c});return}let l=await n(t,a,i);this._file.set({name:e,path:i,groupId:a,content:l,kind:`text`,binaryContent:null,mimeType:`text/plain`})}catch(e){throw console.error(`Failed to open file:`,i,e),e}}};export{s as t};

package/dist/public/getAllTasks-vpBlgGsc.js

@@ -0,0 +1 @@
+import{t as e}from"./txPromise-EBECky1b.js";function t(t){return e(t,`tasks`,`readonly`,e=>e.getAll()).then(e=>e.map(e=>({...e,enabled:!!e.enabled})))}export{t};

package/dist/public/getGroupDir-P1h9wl6S.js

@@ -0,0 +1 @@
+import{n as e}from"./rolldown-runtime-aKtaBQYM.js";import{r as t,x as n}from"./config-CrHchneq.js";import{i as r}from"./app-routes-Bs-riIvr.js";import{t as i}from"./txPromise-EBECky1b.js";import{t as a}from"./getConfig-D89uJgo5.js";import{t as o}from"./memoryStorage-C0KvLNUp.js";async function s(e,...t){let n=e;for(let e of t){let t=await n.getDirectoryHandle(e,{create:!0});if(typeof n.isSameEntry==`function`&&await n.isSameEntry(t))throw Error(`Directory navigation failed: "${e}" resolved to the same directory as its parent. The stored directory handle may be stale — try re-selecting the storage folder.`);n=t}return n}function c(e,t){return i(e,`config`,`readwrite`,e=>e.delete(t)).then(()=>void 0)}let l=null;async function u(e,t){try{return await e.entries().next(),`granted`}catch{return t}}function d(){let e=r();return e?`${n}-${e}`:n}async function f(){try{if(typeof navigator<`u`&&navigator.storage&&typeof navigator.storage.getDirectory==`function`){let e=await navigator.storage.getDirectory(),t=d();return await e.getDirectoryHandle(t,{create:!0})}}catch(e){console.warn(`OPFS is unavailable (e.g. Firefox Private Browsing). Using in-memory storage fallback:`,e)}return o()}function p(e){if(!e||typeof e!=`object`)return!1;let t=globalThis.FileSystemDirectoryHandle;return t!==void 0&&e instanceof t||typeof e.getDirectoryHandle==`function`&&typeof e.getFileHandle==`function`}async function m(e){if(l)return l;try{let n=await a(e,t.STORAGE_HANDLE);if(p(n)){let e=n;if(typeof e.queryPermission==`function`&&await e.queryPermission({mode:`readwrite`})!==`granted`&&typeof e.requestPermission==`function`)try{await e.requestPermission({mode:`readwrite`})}catch{}return e}}catch(e){console.warn(`Failed to retrieve local storage handle from DB:`,e)}return f()}function h(e){l=e}function g(){l=null}function _(e){return e instanceof DOMException&&e.name===`InvalidStateError`}async function v(e){await c(e,t.STORAGE_HANDLE),l=null}async function y(e){if(l)try{let e=await l.queryPermission({mode:`readwrite`});return{type:`local`,permission:e===`granted`?`granted`:await u(l,e),name:l.name}}catch{console.warn(`Getting storage status failed for explicit storage root.`)}try{let n=await a(e,t.STORAGE_HANDLE);if(p(n)){let e=n,t=await e.queryPermission({mode:`readwrite`});return{type:`local`,permission:t===`granted`?`granted`:await u(e,t),name:e.name}}}catch(e){console.warn(`Failed to check storage status:`,e)}return{type:`opfs`,permission:`granted`,name:`OPFS`}}var b=e({getGroupDir:()=>x});async function x(e,t){return s(await m(e),`groups`,t.replace(/:/g,`-`))}export{y as a,v as c,m as i,h as l,b as n,g as o,d as r,_ as s,x as t};