@intent-systems/nexus 2026.1.5-3

package/docs/browser-linux-troubleshooting.md

@@ -0,0 +1,114 @@
+---
+summary: "Fix Chrome/Chromium CDP startup issues for Nexus browser control on Linux"
+read_when: "Browser control fails on Linux, especially with snap Chromium"
+---
+
+# Browser Troubleshooting (Linux)
+
+## Problem: "Failed to start Chrome CDP on port 18800"
+
+Nexus's browser control server fails to launch Chrome/Chromium with the error:
+```
+{"error":"Error: Failed to start Chrome CDP on port 18800 for profile \"nexus\"."}
+```
+
+### Root Cause
+
+On Ubuntu (and many Linux distros), the default Chromium installation is a **snap package**. Snap's AppArmor confinement interferes with how Nexus spawns and monitors the browser process.
+
+The `apt install chromium` command installs a stub package that redirects to snap:
+```
+Note, selecting 'chromium-browser' instead of 'chromium'
+chromium-browser is already the newest version (2:1snap1-0ubuntu2).
+```
+
+This is NOT a real browser — it's just a wrapper.
+
+### Solution 1: Install Google Chrome (Recommended)
+
+Install the official Google Chrome `.deb` package, which is not sandboxed by snap:
+
+```bash
+wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
+sudo dpkg -i google-chrome-stable_current_amd64.deb
+sudo apt --fix-broken install -y  # if there are dependency errors
+```
+
+Then update your Nexus config (`~/nexus/state/nexus.json`):
+
+```json
+{
+  "browser": {
+    "enabled": true,
+    "executablePath": "/usr/bin/google-chrome-stable",
+    "headless": true,
+    "noSandbox": true
+  }
+}
+```
+
+### Solution 2: Use Snap Chromium with Attach-Only Mode
+
+If you must use snap Chromium, configure Nexus to attach to a manually-started browser:
+
+1. Update config:
+```json
+{
+  "browser": {
+    "enabled": true,
+    "attachOnly": true,
+    "headless": true,
+    "noSandbox": true
+  }
+}
+```
+
+2. Start Chromium manually:
+```bash
+chromium-browser --headless --no-sandbox --disable-gpu \
+  --remote-debugging-port=18800 \
+  --user-data-dir=$HOME/.nexus/browser/nexus/user-data \
+  about:blank &
+```
+
+3. Optionally create a systemd user service to auto-start Chrome:
+```ini
+# ~/.config/systemd/user/nexus-browser.service
+[Unit]
+Description=Nexus Browser (Chrome CDP)
+After=network.target
+
+[Service]
+ExecStart=/snap/bin/chromium --headless --no-sandbox --disable-gpu --remote-debugging-port=18800 --user-data-dir=%h/.nexus/browser/nexus/user-data about:blank
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=default.target
+```
+
+Enable with: `systemctl --user enable --now nexus-browser.service`
+
+### Verifying the Browser Works
+
+Check status:
+```bash
+curl -s http://127.0.0.1:18791/ | jq '{running, pid, chosenBrowser}'
+```
+
+Test browsing:
+```bash
+curl -s -X POST http://127.0.0.1:18791/start
+curl -s http://127.0.0.1:18791/tabs
+```
+
+### Config Reference
+
+| Option | Description | Default |
+|--------|-------------|---------|
+| `browser.enabled` | Enable browser control | `true` |
+| `browser.executablePath` | Path to Chrome/Chromium binary | auto-detected |
+| `browser.headless` | Run without GUI | `false` |
+| `browser.noSandbox` | Add `--no-sandbox` flag (needed for some Linux setups) | `false` |
+| `browser.attachOnly` | Don't launch browser, only attach to existing | `false` |
+| `browser.cdpPort` | Chrome DevTools Protocol port | `18800` |

package/docs/browser.md

@@ -0,0 +1,293 @@
+---
+summary: "Spec: integrated browser control server + action commands"
+read_when:
+  - Adding agent-controlled browser automation
+  - Debugging why nexus is interfering with your own Chrome
+  - Implementing browser settings + lifecycle in the macOS app
+---
+
+# Browser (integrated) — nexus-managed Chrome
+
+Status: draft spec · Date: 2025-12-20
+
+Goal: give the **nexus** persona its own browser that is:
+- Visually distinct (lobster-orange, profile labeled "nexus").
+- Fully agent-manageable (start/stop, list tabs, focus/close tabs, open URLs, screenshot).
+- Non-interfering with the user's own browser (separate profile + dedicated ports).
+
+This doc covers the macOS app/gateway side. It intentionally does not mandate
+Playwright vs Puppeteer; the key is the **contract** and the **separation guarantees**.
+
+## User-facing settings
+
+Add a dedicated settings section (preferably under **Skills** or its own "Browser" tab):
+
+- **Enable nexus browser** (`default: on`)
+  - When off: no browser is launched, and browser tools return "disabled".
+- **Browser control URL** (`default: http://127.0.0.1:18791`)
+  - Interpreted as the base URL of the local/remote browser-control server.
+  - If the URL host is not loopback, Nexus must **not** attempt to launch a local
+    browser; it only connects.
+- **CDP URL** (`default: controlUrl + 1`)
+  - Base URL for Chrome DevTools Protocol (e.g. `http://127.0.0.1:18792`).
+  - Set this to a non-loopback host to attach the local control server to a remote
+    Chrome/Chromium CDP endpoint (SSH/Tailscale tunnel recommended).
+  - If the CDP URL host is non-loopback, nexus does **not** auto-launch a local browser.
+  - If you tunnel a remote CDP to `localhost`, set **Attach to existing only** to
+    avoid accidentally launching a local browser.
+- **Accent color** (`default: #FF4500`, "lobster-orange")
+  - Used to theme the nexus browser profile (best-effort) and to tint UI indicators
+    in Nexus.
+
+Optional (advanced, can be hidden behind Debug initially):
+- **Use headless browser** (`default: off`)
+- **Attach to existing only** (`default: off`) — if on, never launch; only connect if
+  already running.
+- **Browser executable path** (override, optional)
+- **No sandbox** (`default: off`) — adds `--no-sandbox` + `--disable-setuid-sandbox`
+
+### Port convention
+
+Nexus already uses:
+- Gateway WebSocket: `18789`
+- Bridge (voice/node): `18790`
+
+For the nexus browser-control server, use "family" ports:
+- Browser control HTTP API: `18791` (bridge + 1)
+- Browser CDP/debugging port: `18792` (control + 1)
+- Canvas host HTTP: `18793` by default, mounted at `/__nexus__/canvas/`
+
+The user usually only configures the **control URL** (port `18791`). CDP is an
+internal detail.
+
+## Browser isolation guarantees (non-negotiable)
+
+1) **Dedicated user data dir**
+   - Never attach to or reuse the user's default Chrome profile.
+   - Store nexus browser state under an app-owned directory, e.g.:
+     - `~/Library/Application Support/Nexus/browser/nexus/` (mac app)
+     - or `~/nexus/state/browser/nexus/` (gateway/CLI)
+
+2) **Dedicated ports**
+   - Never use `9222` (reserved for ad-hoc dev workflows; avoids colliding with
+     `agent-tools/browser-tools`).
+   - Default ports are `18791/18792` unless overridden.
+
+3) **Named tab/page management**
+   - The agent must be able to enumerate and target tabs deterministically (by
+     stable `targetId` or equivalent), not "last tab".
+
+## Browser selection (macOS + Linux)
+
+On startup (when enabled + local URL), Nexus chooses the browser executable
+in this order:
+1) **Google Chrome Canary** (if installed)
+2) **Chromium** (if installed)
+3) **Google Chrome** (fallback)
+
+Linux:
+- Looks for `google-chrome` / `chromium` in common system paths.
+- Use **Browser executable path** to force a specific binary.
+
+Implementation detail:
+- macOS: detection is by existence of the `.app` bundle under `/Applications`
+  (and optionally `~/Applications`), then using the resolved executable path.
+- Linux: common `/usr/bin`/`/snap/bin` paths.
+
+Rationale:
+- Canary/Chromium are easy to visually distinguish from the user's daily driver.
+- Chrome fallback ensures the feature works on a stock machine.
+
+## Visual differentiation ("lobster-orange")
+
+The nexus browser should be obviously different at a glance:
+- Profile name: **nexus**
+- Profile color: **#FF4500**
+
+Preferred behavior:
+- Seed/patch the profile's preferences on first launch so the color + name persist.
+
+Fallback behavior:
+- If preferences patching is not reliable, open with the dedicated profile and let
+  the user set the profile color/name once via Chrome UI; it must persist because
+  the `userDataDir` is persistent.
+
+## Control server contract (vNext)
+
+Expose a small local HTTP API (and/or gateway RPC surface) so the agent can manage
+state without touching the user's Chrome.
+
+Basics:
+- `GET /` status payload (enabled/running/pid/cdpPort/etc)
+- `POST /start` start browser
+- `POST /stop` stop browser
+- `GET /tabs` list tabs
+- `POST /tabs/open` open a new tab
+- `POST /tabs/focus` focus a tab by id/prefix
+- `DELETE /tabs/:targetId` close a tab by id/prefix
+
+Inspection:
+- `POST /screenshot` `{ targetId?, fullPage?, ref?, element?, type? }`
+- `GET /snapshot` `?format=aria|ai&targetId?&limit?`
+- `GET /console` `?level?&targetId?`
+- `POST /pdf` `{ targetId? }`
+
+Actions:
+- `POST /navigate`
+- `POST /act` `{ kind, targetId?, ... }` where `kind` is one of:
+  - `click`, `type`, `press`, `hover`, `drag`, `select`, `fill`, `wait`, `resize`, `close`, `evaluate`
+
+Hooks (arming):
+- `POST /hooks/file-chooser` `{ targetId?, paths, timeoutMs? }`
+- `POST /hooks/dialog` `{ targetId?, accept, promptText?, timeoutMs? }`
+
+### "Is it open or closed?"
+
+"Open" means:
+- the control server is reachable at the configured URL **and**
+- it reports a live browser connection.
+
+"Closed" means:
+- control server not reachable, or server reports no browser.
+
+Nexus should treat "open/closed" as a health check (fast path), not by scanning
+global Chrome processes (avoid false positives).
+
+## Multi-profile support
+
+Nexus supports multiple named browser profiles, each with:
+- Dedicated CDP port (auto-allocated from 18800-18899) **or** a per-profile CDP URL
+- Persistent user data directory (`~/nexus/state/browser/<name>/user-data/`)
+- Unique color for visual distinction
+
+### Configuration
+
+```json
+{
+  "browser": {
+    "enabled": true,
+    "defaultProfile": "nexus",
+    "profiles": {
+      "nexus": { "cdpPort": 18800, "color": "#FF4500" },
+      "work": { "cdpPort": 18801, "color": "#0066CC" },
+      "remote": { "cdpUrl": "http://10.0.0.42:9222", "color": "#00AA00" }
+    }
+  }
+}
+```
+
+### Profile actions
+
+- `GET /profiles` — list all profiles with status
+- `POST /profiles/create` `{ name, color?, cdpUrl? }` — create new profile (auto-allocates port if no `cdpUrl`)
+- `DELETE /profiles/:name` — delete profile (stops browser + removes user data for local profiles)
+- `POST /reset-profile?profile=<name>` — kill orphan process on profile's port (local profiles only)
+
+### Profile parameter
+
+All existing endpoints accept optional `?profile=<name>` query parameter:
+- `GET /?profile=work` — status for work profile
+- `POST /start?profile=work` — start work profile browser
+- `GET /tabs?profile=work` — list tabs for work profile
+- etc.
+
+When `profile` is omitted, uses `browser.defaultProfile` (defaults to "nexus").
+
+### Profile naming rules
+
+- Lowercase alphanumeric characters and hyphens only
+- Must start with a letter or number (not a hyphen)
+- Maximum 64 characters
+- Examples: `nexus`, `work`, `my-project-1`
+
+### Port allocation
+
+Ports are allocated from range 18800-18899 (~100 profiles max). This is far more
+than practical use — memory and CPU exhaustion occur well before port exhaustion.
+Ports are allocated once at profile creation and persisted permanently.
+Remote profiles are attach-only and do **not** use the local port range.
+## Interaction with the agent (nexus)
+
+The agent should use browser tools only when:
+- enabled in settings
+- control URL is configured
+
+If disabled, tools must fail fast with a friendly error ("Browser disabled in settings").
+
+The agent should not assume tabs are ephemeral. It should:
+- call `browser.tabs.list` to discover existing tabs first
+- reuse an existing tab when appropriate (e.g. a persistent "main" tab)
+- avoid opening duplicate tabs unless asked
+
+## CLI quick reference (one example each)
+
+All commands accept `--profile <name>` to target a specific profile (default: `nexus`).
+
+Profile management:
+- `nexus browser profiles`
+- `nexus browser create-profile --name work`
+- `nexus browser create-profile --name remote --cdp-url http://10.0.0.42:9222`
+- `nexus browser delete-profile --name work`
+Basics:
+- `nexus browser status`
+- `nexus browser start`
+- `nexus browser stop`
+- `nexus browser reset-profile`
+- `nexus browser tabs`
+- `nexus browser open https://example.com`
+- `nexus browser focus abcd1234`
+- `nexus browser close abcd1234`
+
+Inspection:
+- `nexus browser screenshot`
+- `nexus browser screenshot --full-page`
+- `nexus browser screenshot --ref 12`
+- `nexus browser snapshot`
+- `nexus browser snapshot --format aria --limit 200`
+
+Actions:
+- `nexus browser navigate https://example.com`
+- `nexus browser resize 1280 720`
+- `nexus browser click 12 --double`
+- `nexus browser type 23 "hello" --submit`
+- `nexus browser press Enter`
+- `nexus browser hover 44`
+- `nexus browser drag 10 11`
+- `nexus browser select 9 OptionA OptionB`
+- `nexus browser upload /tmp/file.pdf`
+- `nexus browser fill --fields '[{\"ref\":\"1\",\"value\":\"Ada\"}]'`
+- `nexus browser dialog --accept`
+- `nexus browser wait --text "Done"`
+- `nexus browser evaluate --fn '(el) => el.textContent' --ref 7`
+- `nexus browser evaluate --fn "document.querySelector('.my-class').click()"`
+- `nexus browser console --level error`
+- `nexus browser pdf`
+
+Notes:
+- `upload` and `dialog` are **arming** calls; run them before the click/press that triggers the chooser/dialog.
+- `upload` can take a `ref` to auto-click after arming (useful for single-step file uploads).
+- `upload` can also take `inputRef` (aria ref) or `element` (CSS selector) to set `<input type="file">` directly without waiting for a file chooser.
+- The arm default timeout is **2 minutes** (clamped to max 2 minutes); pass `timeoutMs` if you need shorter.
+- `snapshot` defaults to `ai`; `aria` returns an accessibility tree for debugging.
+- `click`/`type` require `ref` from `snapshot --format ai`; use `evaluate` for rare CSS selector one-offs.
+- Avoid `wait` by default; use it only in exceptional cases when there is no reliable UI state to wait on.
+
+## Security & privacy notes
+
+- The nexus browser profile is app-owned; it may contain logged-in sessions.
+  Treat it as sensitive data.
+- The control server must bind to loopback only by default (`127.0.0.1`) unless the
+  user explicitly configures a non-loopback URL.
+- Never reuse or copy the user's default Chrome profile.
+- Remote CDP endpoints should be tunneled or protected; CDP is highly privileged.
+
+## Non-goals (for the first cut)
+
+- Cross-device "sync" of tabs between Mac and Pi.
+- Sharing the user's logged-in Chrome sessions automatically.
+- General-purpose web scraping; this is primarily for "close-the-loop" verification
+  and interaction.
+
+## Troubleshooting
+
+For Linux-specific issues (especially Ubuntu with snap Chromium), see [browser-linux-troubleshooting](https://docs.nexus.bot/browser-linux-troubleshooting).

package/docs/bun.md

@@ -0,0 +1,56 @@
+# Bun (optional)
+
+Goal: allow running this repo with Bun without maintaining a Bun lockfile or losing pnpm patch behavior.
+
+## Status
+
+- pnpm remains the primary package manager/runtime for this repo.
+- Bun can be used for local installs/builds/tests, but Bun currently **cannot use** `pnpm-lock.yaml` and will ignore it.
+
+## Install (no Bun lockfile)
+
+Use Bun without writing `bun.lock`/`bun.lockb`:
+
+```sh
+bun install --no-save
+```
+
+This avoids maintaining two lockfiles. (`bun.lock`/`bun.lockb` are gitignored.)
+
+## Build / Test (Bun)
+
+```sh
+bun run build
+bun run vitest run
+```
+
+## pnpm patchedDependencies under Bun
+
+pnpm supports `package.json#pnpm.patchedDependencies` and records it in `pnpm-lock.yaml`.
+Bun does not support pnpm patches, so we apply them in `postinstall` when Bun is detected:
+
+- `scripts/postinstall.js` runs only for Bun installs and applies every entry from `package.json#pnpm.patchedDependencies` into `node_modules/...` using `git apply` (idempotent).
+
+To add a new patch that works in both pnpm + Bun:
+
+1. Add an entry to `package.json#pnpm.patchedDependencies`
+2. Add the patch file under `patches/`
+3. Run `pnpm install` (updates `pnpm-lock.yaml` patch hash)
+
+## Bun lifecycle scripts (blocked by default)
+
+Bun may block dependency lifecycle scripts unless explicitly trusted (`bun pm untrusted` / `bun pm trust`).
+For this repo, the commonly blocked scripts are not required:
+
+- `@whiskeysockets/baileys` `preinstall`: checks Node major >= 20 (we run Node 22+).
+- `protobufjs` `postinstall`: emits warnings about incompatible version schemes (no build artifacts).
+
+If you hit a real runtime issue that requires these scripts, trust them explicitly:
+
+```sh
+bun pm trust @whiskeysockets/baileys protobufjs
+```
+
+## Caveats
+
+- Some scripts still hardcode pnpm (e.g. `docs:build`, `ui:*`, `protocol:check`). Run those via pnpm for now.

package/docs/camera.md

@@ -0,0 +1,152 @@
+---
+summary: "Camera capture (iOS node + macOS app) for agent use: photos (jpg) and short video clips (mp4)"
+read_when:
+  - Adding or modifying camera capture on iOS nodes or macOS
+  - Extending agent-accessible MEDIA temp-file workflows
+---
+
+# Camera capture (agent)
+
+Nexus supports **camera capture** for agent workflows:
+
+- **iOS node** (paired via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
+- **Android node** (paired via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
+- **macOS app** (node via Gateway): capture a **photo** (`jpg`) or **short video clip** (`mp4`, with optional audio) via `node.invoke`.
+
+All camera access is gated behind **user-controlled settings**.
+
+## iOS node
+
+### User setting (default on)
+
+- iOS Settings tab → **Camera** → **Allow Camera** (`camera.enabled`)
+  - Default: **on** (missing key is treated as enabled).
+  - When off: `camera.*` commands return `CAMERA_DISABLED`.
+
+### Commands (via Gateway `node.invoke`)
+
+- `camera.list`
+  - Response payload:
+    - `devices`: array of `{ id, name, position, deviceType }`
+
+- `camera.snap`
+  - Params:
+    - `facing`: `front|back` (default: `front`)
+    - `maxWidth`: number (optional; default `1600` on the iOS node)
+    - `quality`: `0..1` (optional; default `0.9`)
+    - `format`: currently `jpg`
+    - `delayMs`: number (optional; default `0`)
+    - `deviceId`: string (optional; from `camera.list`)
+  - Response payload:
+    - `format: "jpg"`
+    - `base64: "<...>"`
+    - `width`, `height`
+  - Payload guard: photos are recompressed to keep the base64 payload under 5 MB.
+
+- `camera.clip`
+  - Params:
+    - `facing`: `front|back` (default: `front`)
+    - `durationMs`: number (default `3000`, clamped to a max of `60000`)
+    - `includeAudio`: boolean (default `true`)
+    - `format`: currently `mp4`
+    - `deviceId`: string (optional; from `camera.list`)
+  - Response payload:
+    - `format: "mp4"`
+    - `base64: "<...>"`
+    - `durationMs`
+    - `hasAudio`
+
+### Foreground requirement
+
+Like `canvas.*`, the iOS node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE`.
+
+### CLI helper (temp files + MEDIA)
+
+The easiest way to get attachments is via the CLI helper, which writes decoded media to a temp file and prints `MEDIA:<path>`.
+
+Examples:
+
+```bash
+nexus nodes camera snap --node <id>               # default: both front + back (2 MEDIA lines)
+nexus nodes camera snap --node <id> --facing front
+nexus nodes camera clip --node <id> --duration 3000
+nexus nodes camera clip --node <id> --no-audio
+```
+
+Notes:
+- `nodes camera snap` defaults to **both** facings to give the agent both views.
+- Output files are temporary (in the OS temp directory) unless you build your own wrapper.
+
+## Android node
+
+### User setting (default on)
+
+- Android Settings sheet → **Camera** → **Allow Camera** (`camera.enabled`)
+  - Default: **on** (missing key is treated as enabled).
+  - When off: `camera.*` commands return `CAMERA_DISABLED`.
+
+### Permissions
+
+- Android requires runtime permissions:
+  - `CAMERA` for both `camera.snap` and `camera.clip`.
+  - `RECORD_AUDIO` for `camera.clip` when `includeAudio=true`.
+
+If permissions are missing, the app will prompt when possible; if denied, `camera.*` requests fail with a
+`*_PERMISSION_REQUIRED` error.
+
+### Foreground requirement
+
+Like `canvas.*`, the Android node only allows `camera.*` commands in the **foreground**. Background invocations return `NODE_BACKGROUND_UNAVAILABLE`.
+
+### Payload guard
+
+Photos are recompressed to keep the base64 payload under 5 MB.
+
+## macOS app
+
+### User setting (default off)
+
+The macOS companion app exposes a checkbox:
+
+- **Settings → General → Allow Camera** (`nexus.cameraEnabled`)
+  - Default: **off**
+  - When off: camera requests return “Camera disabled by user”.
+
+### CLI helper (node invoke)
+
+Use the main `nexus` CLI to invoke camera commands on the macOS node.
+
+Examples:
+
+```bash
+nexus nodes camera list --node <id>            # list camera ids
+nexus nodes camera snap --node <id>            # prints MEDIA:<path>
+nexus nodes camera snap --node <id> --max-width 1280
+nexus nodes camera snap --node <id> --delay-ms 2000
+nexus nodes camera snap --node <id> --device-id <id>
+nexus nodes camera clip --node <id> --duration 10s          # prints MEDIA:<path>
+nexus nodes camera clip --node <id> --duration-ms 3000      # prints MEDIA:<path> (legacy flag)
+nexus nodes camera clip --node <id> --device-id <id>
+nexus nodes camera clip --node <id> --no-audio
+```
+
+Notes:
+- `nexus nodes camera snap` defaults to `maxWidth=1600` unless overridden.
+- On macOS, `camera.snap` waits `delayMs` (default 2000ms) after warm-up/exposure settle before capturing.
+- Photo payloads are recompressed to keep base64 under 5 MB.
+
+## Safety + practical limits
+
+- Camera and microphone access trigger the usual OS permission prompts (and require usage strings in Info.plist).
+- Video clips are capped (currently `<= 60s`) to avoid oversized bridge payloads (base64 overhead + message limits).
+
+## macOS screen video (OS-level)
+
+For *screen* video (not camera), use the macOS companion:
+
+```bash
+nexus nodes screen record --node <id> --duration 10s --fps 15   # prints MEDIA:<path>
+```
+
+Notes:
+- Requires macOS **Screen Recording** permission (TCC).