@activeagents/browser-sidecar 0.0.0-stage → 1.9.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Active Agents AI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,228 @@
-# Temporary Holding Version
+# @activeagents/browser-sidecar
 
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+One sandbox's browser for the Active Agent dashboard (the `actionagent` gem).
+It launches Chromium with a fresh profile, serves Playwright MCP to the
+agents run against the sandbox, and records every page with rrweb into the
+sandbox's session recording.
+
+The dashboard starts and stops it; you do not run it by hand. On the `:local`
+sandbox backend, install it once with:
+
+```bash
+bin/rails action_agent:browser:install   # this package and its Chromium, under the local sandbox root
+bin/rails action_agent:browser:doctor    # checks Node, this package's version and Chromium
+```
+
+Its version is always the engine's: the dashboard refuses a sidecar of any
+other version.
+
+## What it does
+
+- **Browser.** Chromium through Playwright, headless or headed, with a new
+  temporary profile directory that is removed when it stops. Playwright drives
+  Chromium over a pipe (`--remote-debugging-pipe`), so no debugging port is
+  ever opened, and Chromium's own sandbox stays on unless the configuration
+  turns it off.
+- **MCP.** Playwright MCP over HTTP at `POST /mcp`, one JSON-RPC message per
+  request and a JSON response, with sessions named by `Mcp-Session-Id`. The
+  tools are Playwright MCP's accessibility-snapshot tools, plus the optional
+  groups `testing`, `vision` and `pdf` when the start configuration asks for
+  them. `browser_run_code_unsafe`, which runs code in the sidecar's own
+  process, is never offered. A page snapshot an action writes to a file is
+  returned inline.
+- **Network.** Chromium makes every connection through a forward proxy in
+  the sidecar, loopback included, and WebRTC may not send UDP around it. The
+  proxy resolves each host itself, refuses a loopback, link-local or private
+  address, or a name that resolves to one, and connects to the address it
+  checked. Only the app's own host and port are exempt. Redirects, frames
+  and names whose answer changes between lookups all pass through it.
+  Service workers are blocked.
+- **Navigation.** A top-level navigation may only open the sandbox app's own
+  origin; a path such as `/login` is resolved against it. Chromium follows a
+  redirect without that check, so a tab a redirect takes to another origin
+  is closed as soon as it lands, and the tool call that led there returns an
+  error instead of its result.
+- **Uploads.** Playwright MCP reads files to upload from the process's
+  working directory, which is an empty directory made for the session, and
+  from the sidecar's output directory, where its own snapshots, screenshots
+  and console logs are written.
+- **Recording.** `@rrweb/record` runs in every page and frame, injected with
+  `addInitScript`, with input values and the text of `contenteditable`
+  elements masked before an event leaves the page.
+  Events reach the sidecar through a Playwright binding, so the app's CSP and
+  CORS do not apply, and are posted, gzipped, to the recording's ingest with
+  a token that can only write to that one recording. Console errors and
+  warnings, and markers for pages opening, navigating and closing, go with
+  them.
+- **Live view.** With `live` in its configuration, the sidecar streams the
+  page on screen to the dashboard over a WebSocket at `/live`, and relays the
+  input of the one person who has taken control (see [The live
+  view](#the-live-view)).
+
+## The HTTP surface
+
+| Request | Answer |
+|---------|--------|
+| `GET /health` | `{ "status": "ok", "version": "…" }` |
+| `POST /mcp` | one JSON-RPC message, answered as JSON |
+| `DELETE /mcp` | ends the session in `Mcp-Session-Id` |
+| `GET /live` | the live view's WebSocket, when `live` is configured |
+| `GET /storage-state` | `{ "storage_state": { "cookies": […], "origins": […] } }`: the browser's cookies and localStorage for the app, so a sign-in made in it can be kept |
+
+Every request, a WebSocket upgrade other than the live view's included, must:
+
+- carry `Authorization: Bearer <token>`;
+- name an accepted `Host`: the listening address and port, the loopback names
+  when listening on loopback or on every interface, and `allowed_hosts`;
+- carry no `Origin` header. A page in a browser always sends one, so a page
+  the browser visits cannot reach the sidecar, through a loopback request or
+  DNS rebinding.
+
+No CORS header is ever sent.
+
+## The live view
+
+The dashboard's page opens `ws://<host>:<port>/live` itself, so the upgrade
+carries the dashboard's `Origin` and no token. It is accepted only when it
+names an accepted `Host`, its `Origin` is one of `live.origins`, and its URL
+has no query. The first message must then be a ticket:
+
+```json
+{ "type": "auth", "ticket": "<payload>.<signature>" }
+```
+
+The dashboard issues a ticket per viewer (`POST
+/api/sandboxes/:id/browser/tickets`). Its payload is base64url JSON
+`{ v: 1, sid, sub, name, mode, iat, exp, jti }`, and its signature is the
+base64url HMAC-SHA256 of the payload under a key both sides derive from the
+browser token: HMAC-SHA256 of `activeagents/browser-live-ticket/v1` keyed
+with the token. The sidecar accepts a ticket once, for at most 60 seconds,
+for this sandbox (`sid` is `live.session_id`) and for `mode` `view` or
+`control`. A connection whose first message is anything else is closed
+(code 4401) before anything is sent to it, and one that sends nothing for 5
+seconds is closed with 4408.
+
+Once accepted, the viewer is sent `ready`, the last frame, and every frame
+after it: `{ type: "frame", data, width, height }`, a JPEG as base64 with the
+page's size in CSS pixels. The stream follows the page on screen: the newest
+tab, or the one the agent selected with `browser_tabs`. A still page sends no
+frames, so the last one is kept for whoever joins later. `control`, `agent`
+and `page` messages tell of who holds control, an agent call waiting for it,
+and the page on screen. A `control` message's `mine` says whether this
+connection holds control, and `yours` whether its user does on any
+connection, as after a reload while control waits out the grace period.
+
+Either mode of ticket lets a connection watch. To take control it sends
+`{ type: "take_control", ticket }` with a control ticket of its own, issued
+to the user it connected as, each time it asks. Withdrawing a person's
+permission therefore stops them taking control again, but does not take it
+from a connection that holds it. `{ type: "hand_back" }` gives control back.
+One person holds control at a time; their other connections may move it
+between them. While they hold it:
+
+- their `mouse`, `wheel`, `key` and `text` messages reach the page through
+  `Input.dispatchMouseEvent`, `Input.dispatchKeyEvent` and `Input.insertText`.
+  Pointer positions are fractions of the frame, scaled to the page. Nobody
+  else's input is relayed;
+- an agent's tool call that would change the page waits for them to hand
+  back, up to `live.agent_wait_ms`. If they have not, it returns an error
+  naming them, with `_meta["activeagents/takeover"]`:
+  `{ held_by, since, waited_ms }`. A call still waiting when the browser
+  stops returns an error saying it is stopping. Calls that only read the
+  page, such as `browser_snapshot`, go ahead;
+- if their connection drops, control is released after
+  `live.release_grace_ms`.
+
+Relayed input is never logged or recorded. Taking control and handing it
+back are recorded as `marker` events (`takeover_started`, `takeover_ended`,
+with `source: "human"` and who), and what the person changed shows in the
+masked rrweb stream.
+
+## The command
+
+```
+activeagents-browser-sidecar serve [--config-file PATH]
+activeagents-browser-sidecar check
+activeagents-browser-sidecar install-browser [playwright install options]
+activeagents-browser-sidecar --version
+```
+
+`serve` reads its configuration from stdin, or from `--config-file`, so the
+token never appears in a process listing or an environment. Once it listens
+it prints one line to stdout, `{"ready":true,"port":…,"version":"…","pid":…}`,
+and logs only to stderr. SIGTERM, SIGINT or SIGHUP stops it: it closes the
+browser, posts what it has recorded, and removes its directories.
+
+`check` prints `{ "version", "node", "chromium": { "installed", "executable" } }`.
+`install-browser` installs the Chromium this version drives, into
+`PLAYWRIGHT_BROWSERS_PATH` when that is set.
+
+## Configuration
+
+```json
+{
+  "token": "…",
+  "app_url": "http://127.0.0.1:3000",
+  "mode": "headless",
+  "capabilities": [],
+  "host": "127.0.0.1",
+  "port": 0,
+  "allowed_hosts": [],
+  "workdir": "/path/to/a/directory",
+  "stop_at": 1767225600000,
+  "chromium_sandbox": true,
+  "recording": {
+    "url": "http://127.0.0.1:3000/activeagents/api/session_recordings/7/events",
+    "token": "aarec_…",
+    "batch_events": 1000,
+    "batch_bytes": 1048576
+  },
+  "live": {
+    "session_id": "…",
+    "origins": ["http://localhost:3000"],
+    "agent_wait_ms": 20000,
+    "release_grace_ms": 10000
+  },
+  "storage_state": { "cookies": [], "origins": [] }
+}
+```
+
+| Key | Meaning |
+|-----|---------|
+| `token` | the bearer token every request must carry (at least 32 characters) |
+| `app_url` | the sandbox app; its origin is the only one a page may be opened on |
+| `mode` | `headless`, or `headed` for a visible window |
+| `capabilities` | optional Playwright MCP tool groups: `testing`, `vision`, `pdf` |
+| `host`, `port` | where to listen; port `0` picks a free one |
+| `allowed_hosts` | `Host` values to accept besides the listening address, such as a container name with its port |
+| `workdir` | where the profile, uploads and output directories are made; a temporary directory when unset |
+| `stop_at` | when to stop on its own, as epoch milliseconds or ISO 8601 |
+| `chromium_sandbox` | `false` where Chromium's own sandbox cannot run, as in a container without the privileges it needs |
+| `recording` | where to post recorded events, and the batch limits the ingest enforces; nothing is recorded when unset |
+| `live` | the live view: the sandbox session tickets must name, the dashboard origins that may open it, how long an agent's call waits for control (at most 45000 ms) and how long control outlives a dropped connection; no live view when unset |
+| `storage_state` | a Playwright storage state the browser starts with, such as a sign-in saved from `GET /storage-state`; only the cookies sent to the app's host and the localStorage of its origin are applied |
+
+## The image
+
+The `Dockerfile` here builds the same package with its Chromium, running as a
+non-root user. A host that runs each sandbox's browser as a container starts
+it with the configuration on stdin (`docker run -i`) or in a file:
+
+```bash
+docker run -i --rm -p 8931:8931 ghcr.io/activeagents/browser-sidecar:<version> < config.json
+```
+
+with `"host": "0.0.0.0"`, `"port": 8931`, and the name the dashboard reaches it
+by in `allowed_hosts`.
+
+## Developing
+
+```bash
+npm ci
+npm test                                    # the real-browser test is skipped without Chromium
+npx playwright install chromium && npm test # runs it too
+```
+
+To have the dashboard run a checkout of this directory instead of the
+installed package, set `ActionAgent.browser_sidecar_path` to it. The version
+check is skipped for a checkout.

package/bin/browser-sidecar.mjs

@@ -0,0 +1,85 @@
+#!/usr/bin/env node
+// activeagents-browser-sidecar [serve [--config-file PATH] | check | install-browser [ARGS...] | --version]
+//
+//   serve            reads the start configuration (lib/config.mjs) from stdin, or
+//                    from --config-file, starts the browser, and prints one JSON line
+//                    to stdout once it listens: { ready, port, version, pid }. Logs go
+//                    to stderr. SIGTERM or SIGINT stops it cleanly.
+//   check            prints { version, node, chromium: { installed, executable } }
+//   install-browser  installs the Chromium this sidecar drives (Playwright's installer)
+//   --version        prints the version
+import { spawnSync } from 'node:child_process';
+import { existsSync, readFileSync } from 'node:fs';
+import { createRequire } from 'node:module';
+import { dirname, join } from 'node:path';
+
+import { ConfigError, parseConfig } from '../lib/config.mjs';
+import { VERSION } from '../lib/version.mjs';
+
+const require = createRequire(import.meta.url);
+
+function readStdin() {
+  return new Promise((resolve, reject) => {
+    const chunks = [];
+    process.stdin.on('data', (chunk) => chunks.push(chunk));
+    process.stdin.on('end', () => resolve(Buffer.concat(chunks).toString('utf8')));
+    process.stdin.on('error', reject);
+  });
+}
+
+async function check() {
+  const { chromium } = await import('playwright');
+  const executable = chromium.executablePath();
+  const installed = Boolean(executable) && existsSync(executable);
+  process.stdout.write(`${JSON.stringify({ version: VERSION, node: process.version, chromium: { installed, executable } })}\n`);
+}
+
+function installBrowser(args) {
+  const cli = join(dirname(require.resolve('playwright/package.json')), 'cli.js');
+  const result = spawnSync(process.execPath, [cli, 'install', 'chromium', ...args], { stdio: 'inherit' });
+  process.exit(result.status ?? 1);
+}
+
+async function serve(args) {
+  const fileIndex = args.indexOf('--config-file');
+  const source = fileIndex === -1 ? await readStdin() : readFileSync(args[fileIndex + 1], 'utf8');
+  const config = parseConfig(source);
+
+  // Only the ready line goes to stdout; whoever started the sidecar may stop
+  // reading it after that.
+  console.log = console.error;
+  const { startSidecar } = await import('../lib/sidecar.mjs');
+  const sidecar = await startSidecar(config);
+  process.stdout.write(`${JSON.stringify({ ready: true, port: sidecar.port, version: VERSION, pid: process.pid })}\n`);
+
+  for (const signal of ['SIGTERM', 'SIGINT', 'SIGHUP']) {
+    process.once(signal, () => void sidecar.close(`received ${signal}`));
+  }
+  sidecar.done.then((status) => process.exit(status));
+}
+
+const [command = 'serve', ...rest] = process.argv.slice(2);
+
+try {
+  switch (command) {
+    case '--version':
+    case 'version':
+      process.stdout.write(`${VERSION}\n`);
+      break;
+    case 'check':
+      await check();
+      break;
+    case 'install-browser':
+      installBrowser(rest);
+      break;
+    case 'serve':
+      await serve(rest);
+      break;
+    default:
+      process.stderr.write('usage: activeagents-browser-sidecar [serve [--config-file PATH] | check | install-browser | --version]\n');
+      process.exit(2);
+  }
+} catch (error) {
+  process.stderr.write(`browser-sidecar: ${error instanceof ConfigError ? `invalid configuration: ${error.message}` : error.stack ?? error}\n`);
+  process.exit(1);
+}

package/lib/config.mjs

@@ -0,0 +1,191 @@
+// The start configuration, read as JSON from stdin (or a file the image is
+// given), so the bearer token never appears in argv or the environment:
+//
+//   {
+//     "token": "...",                       bearer token every request must carry
+//     "app_url": "http://127.0.0.1:3000",   the sandbox app; top-level navigation stays on its origin
+//     "mode": "headless" | "headed",
+//     "capabilities": ["testing"],          optional Playwright MCP tool groups
+//     "host": "127.0.0.1", "port": 0,       where the MCP endpoint listens (0 picks a free port)
+//     "allowed_hosts": ["browser:8931"],    Host header values accepted besides the listening address
+//     "workdir": "/path",                   where the profile, uploads and output directories go
+//     "stop_at": 1767225600000,             when to shut down on its own (epoch ms or ISO 8601)
+//     "chromium_sandbox": true,             false where Chromium's own sandbox cannot run (some containers)
+//     "recording": { "url": "...", "token": "...", "batch_events": 1000, "batch_bytes": 1048576 },
+//     "live": { "session_id": "...", "origins": ["http://localhost:3000"],
+//               "agent_wait_ms": 20000, "release_grace_ms": 10000 }
+//     "storage_state": { "cookies": [...], "origins": [...] }   a saved sign-in the browser starts with
+//   }
+
+import { appStorageState, StorageStateError } from './storage-state.mjs';
+
+export const MODES = ['headless', 'headed'];
+// Tool groups a browser may be started with. The others reach outside the
+// page: devtools writes traces and videos, storage reads and writes storage
+// state files, network rewrites responses, config describes the server.
+export const CAPABILITIES = ['testing', 'vision', 'pdf'];
+export const MIN_TOKEN_LENGTH = 32;
+const DEFAULT_BATCH_EVENTS = 1000;
+const DEFAULT_BATCH_BYTES = 1024 * 1024;
+// How long an agent's browser call waits for a person to hand control back
+// before it is refused. The dashboard gives up on a tool call after 60
+// seconds, so the wait ends well before that.
+const DEFAULT_AGENT_WAIT_MS = 20_000;
+export const MAX_AGENT_WAIT_MS = 45_000;
+const DEFAULT_RELEASE_GRACE_MS = 10_000;
+
+export class ConfigError extends Error {}
+
+function fail(message) {
+  throw new ConfigError(message);
+}
+
+function httpUrl(value, name) {
+  let url;
+  try {
+    url = new URL(value);
+  } catch {
+    fail(`${name} must be an absolute http(s) URL`);
+  }
+  if (url.protocol !== 'http:' && url.protocol !== 'https:') fail(`${name} must be an absolute http(s) URL`);
+  if (url.username || url.password) fail(`${name} must not carry credentials`);
+  return url;
+}
+
+function token(value, name) {
+  if (typeof value !== 'string' || value.length < MIN_TOKEN_LENGTH || /\s/.test(value)) {
+    fail(`${name} must be a string of at least ${MIN_TOKEN_LENGTH} characters without whitespace`);
+  }
+  return value;
+}
+
+function positiveInteger(value, name, fallback) {
+  if (value === undefined || value === null) return fallback;
+  if (!Number.isInteger(value) || value <= 0) fail(`${name} must be a positive integer`);
+  return value;
+}
+
+function stopAt(value) {
+  if (value === undefined || value === null) return null;
+  const time = typeof value === 'number' ? value : Date.parse(value);
+  if (!Number.isFinite(time)) fail('stop_at must be epoch milliseconds or an ISO 8601 time');
+  return time;
+}
+
+function recording(value) {
+  if (value === undefined || value === null) return null;
+  if (typeof value !== 'object' || Array.isArray(value)) fail('recording must be an object');
+
+  return {
+    url: httpUrl(value.url, 'recording.url').toString(),
+    token: token(value.token, 'recording.token'),
+    batchEvents: positiveInteger(value.batch_events, 'recording.batch_events', DEFAULT_BATCH_EVENTS),
+    batchBytes: positiveInteger(value.batch_bytes, 'recording.batch_bytes', DEFAULT_BATCH_BYTES),
+  };
+}
+
+function nonNegativeInteger(value, name, fallback, max) {
+  if (value === undefined || value === null) return fallback;
+  if (!Number.isInteger(value) || value < 0 || value > max) fail(`${name} must be an integer from 0 to ${max}`);
+  return value;
+}
+
+// Returns `value` as the Origin header a browser sends for it:
+// "HTTP://127.1:3000/" is "http://127.0.0.1:3000".
+function origin(value) {
+  let url;
+  try {
+    url = new URL(value);
+  } catch {
+    fail('live.origins must be a list of http(s) origins');
+  }
+  const bare = url.pathname === '/' && url.search === '' && url.hash === '' && url.username === '' && url.password === '';
+  if ((url.protocol !== 'http:' && url.protocol !== 'https:') || !bare) {
+    fail('live.origins must be a list of http(s) origins, such as http://localhost:3000');
+  }
+  return url.origin;
+}
+
+function live(value) {
+  if (value === undefined || value === null) return null;
+  if (typeof value !== 'object' || Array.isArray(value)) fail('live must be an object');
+  if (typeof value.session_id !== 'string' || value.session_id === '') fail('live.session_id must name the sandbox session');
+  if (!Array.isArray(value.origins) || value.origins.length === 0 || value.origins.some((entry) => typeof entry !== 'string')) {
+    fail('live.origins must be a list of http(s) origins');
+  }
+
+  return {
+    sessionId: value.session_id,
+    origins: [...new Set(value.origins.map(origin))],
+    agentWaitMs: nonNegativeInteger(value.agent_wait_ms, 'live.agent_wait_ms', DEFAULT_AGENT_WAIT_MS, MAX_AGENT_WAIT_MS),
+    releaseGraceMs: positiveInteger(value.release_grace_ms, 'live.release_grace_ms', DEFAULT_RELEASE_GRACE_MS),
+  };
+}
+
+/**
+ * Validates a start configuration and returns it with camelCase keys.
+ *
+ * @param {string|object} input the configuration, as JSON text or parsed
+ * @returns {object}
+ * @throws {ConfigError} naming the first invalid setting
+ */
+export function parseConfig(input) {
+  let raw = input;
+  if (typeof input === 'string') {
+    try {
+      raw = JSON.parse(input);
+    } catch {
+      fail('the configuration is not JSON');
+    }
+  }
+  if (!raw || typeof raw !== 'object' || Array.isArray(raw)) fail('the configuration must be a JSON object');
+
+  const mode = raw.mode ?? 'headless';
+  if (!MODES.includes(mode)) fail(`mode must be one of ${MODES.join(', ')}`);
+
+  const capabilities = raw.capabilities ?? [];
+  if (!Array.isArray(capabilities) || capabilities.some((name) => !CAPABILITIES.includes(name))) {
+    fail(`capabilities may only name ${CAPABILITIES.join(', ')}`);
+  }
+
+  const allowedHosts = raw.allowed_hosts ?? [];
+  if (!Array.isArray(allowedHosts) || allowedHosts.some((host) => typeof host !== 'string' || host === '')) {
+    fail('allowed_hosts must be a list of host names');
+  }
+
+  const port = raw.port ?? 0;
+  if (!Number.isInteger(port) || port < 0 || port > 65535) fail('port must be an integer from 0 to 65535');
+
+  if (raw.chromium_sandbox !== undefined && typeof raw.chromium_sandbox !== 'boolean') fail('chromium_sandbox must be true or false');
+
+  if (raw.workdir !== undefined && raw.workdir !== null && (typeof raw.workdir !== 'string' || raw.workdir === '')) {
+    fail('workdir must be a directory path');
+  }
+
+  const appOrigin = httpUrl(raw.app_url, 'app_url').origin;
+  let storageState = null;
+  if (raw.storage_state !== undefined && raw.storage_state !== null) {
+    try {
+      storageState = appStorageState(raw.storage_state, appOrigin);
+    } catch (error) {
+      if (error instanceof StorageStateError) fail(error.message);
+      throw error;
+    }
+  }
+
+  return {
+    token: token(raw.token, 'token'),
+    appOrigin,
+    mode,
+    capabilities: [...new Set(capabilities)],
+    host: typeof raw.host === 'string' && raw.host !== '' ? raw.host : '127.0.0.1',
+    port,
+    allowedHosts: allowedHosts.map((host) => host.toLowerCase()),
+    workdir: raw.workdir ?? null,
+    chromiumSandbox: raw.chromium_sandbox !== false,
+    stopAt: stopAt(raw.stop_at),
+    recording: recording(raw.recording),
+    live: live(raw.live),
+    storageState,
+  };
+}