opendevbrowser 0.0.17 → 0.0.18

package/README.md

@@ -10,7 +10,10 @@
 
 > **Script-first browser automation for AI agents.** Snapshot → Refs → Actions.
 
-OpenDevBrowser is an agent-agnostic browser automation runtime. You can use it as an [OpenCode](https://opencode.ai) plugin, as a standalone CLI, or through the Chrome extension relay for logged-in browser sessions. It supports managed sessions, direct CDP attach, and extension-backed Ops sessions. Frontend website source and deployment are maintained in the private repo `opendevbrowser-website-deploy`.
+OpenDevBrowser is an agent-agnostic browser automation runtime for CLI workflows, [OpenCode](https://opencode.ai) tool calls, and Chrome extension relay sessions. It supports managed launches, direct CDP attach, and extension-backed Ops sessions.
+
+The current public surface includes [72 CLI commands and 65 `opendevbrowser_*` tools](docs/SURFACE_REFERENCE.md); see [docs/CLI.md](docs/CLI.md) for the operational command guide.
+Generated help is the canonical first-contact discovery surface: `npx opendevbrowser --help` and `npx opendevbrowser help` now lead with a `Find It Fast` block that uses the exact lookup terms `screencast / browser replay`, `desktop observation`, and `computer use / browser-scoped computer use`.
 
 <p align="center">
   <img src="assets/hero-image.png" alt="OpenDevBrowser hero image showing AI-assisted annotation and browser automation workflow" width="920" />
@@ -18,6 +21,30 @@ OpenDevBrowser is an agent-agnostic browser automation runtime. You can use it a
   <em>AI-assisted annotation and browser automation workflow</em>
 </p>
 
+## Table of Contents
+
+- [Use It Your Way](#use-it-your-way)
+- [Why OpenDevBrowser?](#why-opendevbrowser)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+- [Challenge Handling Boundary](#challenge-handling-boundary)
+- [Recent Features](#recent-features)
+- [Features](#features)
+- [Tool Reference](#tool-reference)
+- [Bundled Skills](#bundled-skills)
+- [Browser Modes](#browser-modes)
+- [Relay Channels](#relay-channels)
+- [Breaking Changes (latest)](#breaking-changes-latest)
+- [Chrome Extension (Optional)](#chrome-extension-optional)
+- [Configuration](#configuration)
+- [CLI Commands](#cli-commands)
+- [Security](#security)
+- [Updating](#updating)
+- [Architecture](#architecture)
+- [Development](#development)
+- [Privacy](#privacy)
+- [License](#license)
+
 ## Use It Your Way
 
 | Interface | OpenCode Required | Best For |
@@ -27,30 +54,16 @@ OpenDevBrowser is an agent-agnostic browser automation runtime. You can use it a
 | **OpenCode Plugin Tools** | Yes | Native tool-calling inside OpenCode (`opendevbrowser_*`) |
 | **Frontend Website (private repo)** | No | Product website and generated docs routes |
 
-All core automation flows are available through the CLI command surface and the plugin tool surface.
-Private website docs routes are generated from public repository source-of-truth docs and skill packs.
-
-Distribution split (current target state):
-- Public repo (`opendevbrowser`): runtime, CLI, extension, docs, npm, release artifacts.
-- Private repo (`opendevbrowser-website-deploy`): website deployment (`website-production` branch).
+The public repo owns the automation runtime and canonical docs; see [docs/SURFACE_REFERENCE.md](docs/SURFACE_REFERENCE.md) for the full surface inventory.
 
 ## Why OpenDevBrowser?
 
-| Feature | Benefit |
-|---------|---------|
-| **Script-first UX** | Snapshot → Refs → Actions workflow optimized for AI agents |
-| **Accessibility-tree snapshots** | Token-efficient page representation (not raw DOM) |
-| **Stable refs** | Elements identified by `backendNodeId`, not fragile selectors |
-| **Security by default** | CDP localhost-only, timing-safe auth, HTML sanitization |
-| **3 browser modes** | Managed, CDP connect, or extension relay for logged-in sessions |
-| **Ops + Canvas + CDP channels** | High-level multi-client `/ops`, dedicated `/canvas`, plus legacy `/cdp` compatibility |
-| **Relay Hub (FIFO leases)** | Single-owner CDP binding with a FIFO queue for multi-client safety |
-| **Flat-session routing** | Extension relay uses DebuggerSession sessionId routing (Chrome 125+) |
-| **Design canvas + code sync** | Persist repo-native design documents, attach same-session observers, round-trip bound TSX files, and drive preview tabs with `canvas_html` or opted-in `bound_app_runtime` reconciliation |
-| **Loop-closure diagnostics** | Console/network polling + unified debug trace snapshots for verification workflows |
-| **10 bundled skill directories** | Best practices plus login, forms, data extraction, research, shopping, and product asset workflows |
-| **49 tools** | Complete browser automation coverage, including the design-canvas command surface |
-| **97% test coverage** | Production-ready with strict TypeScript |
+- **Script-first automation model**: snapshot → refs → actions, built around accessibility-tree capture instead of brittle selector-first workflows.
+- **Stable interaction primitives**: refs resolve through `backendNodeId`, and low-level pointer commands remain available when normal DOM actions are not enough.
+- **Flexible session control**: run managed sessions, attach through direct CDP, or reuse logged-in tabs through the extension relay and `/ops`.
+- **Design and review workflows**: use the design canvas, shared annotation inbox, and repo-backed code-sync flows without leaving the runtime surface.
+- **Diagnostics and bounded challenge handling**: start with `session-inspector`, then drop to console or network polling and unified debug traces when you need channel-level detail.
+- **Production guardrails**: local-only CDP by default, timing-safe auth, sanitized exports, strict TypeScript, and branch coverage held at 97% or higher.
 
 ---
 
@@ -78,7 +91,7 @@ opendevbrowser --version
 
 ### Pre-release Local Package (No npm publish required)
 
-Use this path to validate first-run onboarding before public distribution:
+Use this flow to validate first-run local package onboarding before npm publish.
 
 ```bash
 cd <public-repo-root>
@@ -87,29 +100,16 @@ npm pack
 WORKDIR=$(mktemp -d /tmp/opendevbrowser-first-run-XXXXXX)
 cd "$WORKDIR"
 npm init -y
-npm install <public-repo-root>/opendevbrowser-0.0.17.tgz
+npm install <public-repo-root>/opendevbrowser-0.0.18.tgz
 npx --no-install opendevbrowser --help
 npx --no-install opendevbrowser help
 ```
 
-Full validated flow: [`docs/FIRST_RUN_ONBOARDING.md`](docs/FIRST_RUN_ONBOARDING.md).
-Dependency/runtime inventory: [`docs/DEPENDENCIES.md`](docs/DEPENDENCIES.md).
-Live CLI inventory/help surface: `npx opendevbrowser --help` or `npx opendevbrowser help`.
-
-Use OpenCode only if you want plugin tools. CLI and extension workflows work without OpenCode.
-
-On first successful install, the CLI attempts to install daemon auto-start on supported platforms so the relay is available on login.
-You can remove it later with `npx opendevbrowser daemon uninstall`.
+See [docs/FIRST_RUN_ONBOARDING.md](docs/FIRST_RUN_ONBOARDING.md) for the full onboarding checklist, [docs/DEPENDENCIES.md](docs/DEPENDENCIES.md) for runtime inventory, and [docs/SURFACE_REFERENCE.md](docs/SURFACE_REFERENCE.md) for the live CLI and tool surface.
 
-During install, bundled skills are synced for **OpenCode, Codex, ClaudeCode, and AmpCLI**.
-Default `--skills-global` targets:
-- `~/.config/opencode/skill` (OpenCode)
-- `$CODEX_HOME/skills` (fallback `~/.codex/skills`)
-- `$CLAUDECODE_HOME/skills` or `$CLAUDE_HOME/skills` (fallback `~/.claude/skills`)
-- `$AMPCLI_HOME/skills` or `$AMP_CLI_HOME/skills` or `$AMP_HOME/skills` (fallback `~/.amp/skills`)
+Successful installs reconcile daemon auto-start on supported platforms so the relay is available on login. If the current CLI entrypoint lives under a transient temp-root path such as a first-run `/tmp` or `/private/tmp` workspace, OpenDevBrowser refuses to persist that path as auto-start. Rerun `opendevbrowser daemon install`, or `npx --no-install opendevbrowser daemon install` from a persistent local package install, from a stable install location if you want login auto-start; remove it later with `opendevbrowser daemon uninstall`.
 
-Use `--skills-local` for project-local targets:
-- `./.opencode/skill`, `./.codex/skills`, `./.claude/skills`, `./.amp/skills`
+Bundled skills sync to **OpenCode, Codex, ClaudeCode, and AmpCLI** targets during install. Use `--skills-global` for user-wide installs or `--skills-local` for project-local installs; see [docs/CLI.md](docs/CLI.md) for exact target paths.
 
 ### CLI + Extension (No OpenCode)
 
@@ -142,6 +142,8 @@ Website build/data pipeline lives in the private repo:
 
 ### Agent Installation (OpenCode)
 
+Use OpenCode only when you want native `opendevbrowser_*` tool calls; the CLI and extension workflows work without it.
+
 Recommended (CLI, installs plugin + config + bundled skills + extension assets):
 
 ```bash
@@ -176,9 +178,9 @@ OpenDevBrowser uses the same automation model across plugin tools and CLI comman
 ```
 1. Launch a browser session
 2. Navigate to a URL
-3. Take a snapshot to get element refs
+3. Take a review to get target-aware actionables and refs
 4. Interact using refs (click, type, select)
-5. Re-snapshot after navigation
+5. Re-review or re-snapshot after navigation
 ```
 
 Shipping checklist for first-time users (local-package install, daemon, extension, first task, multi-tab auth/cookies):
@@ -192,9 +194,10 @@ Parallel execution is target-scoped (`ExecutionKey = (sessionId,targetId)`): sam
 |------|------|---------|
 | 1 | `opendevbrowser_launch` | Launch a session (extension relay first; managed fallback is explicit) |
 | 2 | `opendevbrowser_goto` | Navigate to URL |
-| 3 | `opendevbrowser_snapshot` | Get page structure with refs |
+| 3 | `opendevbrowser_review` | Inspect the active target and capture fresh actionables before acting |
 | 4 | `opendevbrowser_click` / `opendevbrowser_type` | Interact with elements |
-| 5 | `opendevbrowser_disconnect` | Clean up session |
+| 5 | `opendevbrowser_snapshot` | Re-capture refs after navigation or DOM changes |
+| 6 | `opendevbrowser_disconnect` | Clean up session |
 
 ---
 
@@ -207,7 +210,7 @@ Run a local daemon for persistent sessions, then drive automation via CLI comman
 npx opendevbrowser serve
 
 # Install auto-start (recommended for resilience)
-npx opendevbrowser daemon install
+opendevbrowser daemon install
 
 # Stop/kill the daemon before restarting
 npx opendevbrowser serve --stop
@@ -215,8 +218,8 @@ npx opendevbrowser serve --stop
 # Launch a session
 npx opendevbrowser launch --start-url https://example.com
 
-# Capture a snapshot
-npx opendevbrowser snapshot --session-id <session-id>
+# Review the active target and capture fresh refs
+npx opendevbrowser review --session-id <session-id>
 
 # Interact by ref
 npx opendevbrowser click --session-id <session-id> --ref r12
@@ -224,6 +227,8 @@ npx opendevbrowser click --session-id <session-id> --ref r12
 
 `opendevbrowser serve` includes stale-daemon preflight cleanup by default, so orphan daemon processes are terminated automatically
 before startup while preserving the active daemon on the requested port.
+If you are running from a temporary onboarding workspace, rerun `opendevbrowser daemon install` from a stable install location
+before expecting auto-start to survive login.
 
 For single-shot scripts:
 
@@ -233,20 +238,43 @@ npx opendevbrowser run --script ./script.json --output-format json
 
 Use `--output-format json|stream-json` for automation-friendly output.
 
+### Help-Led Discovery
+
+Start every surface check from generated help when you need the current public lanes. The terminal help now uses these exact phrases so agents can search by intent instead of guessing command names:
+
+- `screencast / browser replay`: `screencast-start`, `screencast-stop`
+- `desktop observation`: `desktop-status`, `desktop-windows`, `desktop-active-window`, `desktop-capture-desktop`, `desktop-capture-window`, `desktop-accessibility-snapshot`
+- `computer use / browser-scoped computer use`: `--challenge-automation-mode off|browser|browser_with_helper` on `research run`, `shopping run`, `product-video run`, and `macro-resolve --execute`; `review` and `session-inspector` remain the quickest proof surfaces, and the optional helper stays browser-scoped rather than becoming a desktop agent
+
+## Challenge Handling Boundary
+
+- `SessionStore` remains the blocker FSM source of truth. Managed and `/ops`-backed responses keep `meta.blocker`, `meta.blockerState`, and `meta.blockerResolution` stable and may append additive `meta.challenge` plus `meta.challengeOrchestration`.
+- Direct browser, `/ops`, and provider fallback paths now share one bounded challenge orchestration plane. It can try auth navigation, legitimate session or cookie reuse, non-secret field fill, and bounded interaction exploration before yielding to a human.
+- Workflow and manager callers can set `challengeAutomationMode` to `off`, `browser`, or `browser_with_helper`. Effective precedence is `run > session > config`, and hard gates still apply after resolution.
+- The optional helper bridge is browser-scoped, not a desktop agent. `browser` forces it to stand down, and `browser_with_helper` only evaluates it after the existing helper hard gates pass.
+- Shipped builds also expose a public read-only desktop observation plane under separate `desktop.*` config. It is enabled by default, does not widen `/ops` or `ChallengeRuntimeHandle`, and the internal composed path still routes desktop observation back through browser-owned review when challenge automation needs it.
+- Browser fallback returns explicit transport `disposition` values: `completed`, `challenge_preserved`, `deferred`, or `failed`. When orchestration runs during fallback, decision evidence is recorded under `details.challengeOrchestration`.
+- `ProviderRegistry` is the only durable anti-bot pressure authority. Shared runtime and policy own fallback ordering and resume policy; provider modules only contribute extraction logic and `recoveryHints()`.
+- In scope: preserved sessions, normal browser controls, bounded interaction experimentation, human yield packets for secret or human-authority boundaries, and owned-environment fixtures that use vendor test keys only.
+- Out of scope: hidden bypasses, CAPTCHA-solving services, token harvesting, or autonomous unsandboxed solving of third-party anti-bot systems.
+
 ---
 
 ## Recent Features
 
-### v0.0.17 (Latest)
+### v0.0.18 (Latest)
 
-- **Design canvas runtime is now shipped end-to-end** across core, CLI, tool, relay, and extension surfaces, including `canvas.html`, overlay control, preview feedback, and repo-backed code sync.
-- **Canvas surface governance and skill-pack coverage** now include current `/canvas` inventories, handshake/blocker templates, and feedback-evaluation artifacts.
-- **Release packaging/docs were refreshed for v0.0.17**, including current tarball examples, extension version sync, release evidence, and public/private cutover guidance.
+- **Canvas and workflow surfaces are materially stronger after `v0.0.17`**: adapter-plugin validation, starter or inventory lanes, framework-adapter code sync, review/session-inspector flows, and direct-run release probes all landed on `main`.
+- **Generated public-surface manifests and onboarding metadata now drive help, docs parity, release evidence, and mirrored website inputs** instead of leaving those surfaces hand-maintained.
+- **Challenge automation and browser-scoped computer-use controls are tighter** with explicit `challengeAutomationMode` precedence and a clearer browser-only helper boundary.
+- **Browser replay screencasts and public desktop observation now ship on the active public surface** with manager-owned replay artifacts, dedicated desktop permission or audit coverage, and no desktop agent claim.
+- **Public read-only desktop observation now ships as a sibling runtime** with dedicated permission and audit coverage while staying outside the public relay or `/ops` plane.
+- **Release and distribution operations were refreshed for `v0.0.18`** across GitHub release assets, npm packaging, Chrome Web Store prep, regenerated live store assets, and private website sync inputs.
 
 ### v0.0.16
 
 - **Release-gate hardening** with dedicated audit/compliance scripts (`audit-zombie-files`, `docs-drift-check`, `chrome-store-compliance-check`) and grouped release-gate tests.
-- **Live matrix robustness upgrades** across `live-regression-matrix` and `provider-live-matrix` (strict `--release-gate`, extension/CDP recovery paths, deterministic strict-mode semantics).
+- **Live direct-run release gates** across provider-by-provider and scenario-by-scenario scripts with explicit artifacts instead of broad matrix evidence.
 - **CLI/runtime reliability fixes** including launch RPC timeout derivation from wait hints, bounded macro execute timeouts, and stale extension `/cdp` attach retry handling.
 - **Version/distribution integrity checks** now enforce parity across `package.json`, `extension/manifest.json`, and `extension/package.json`.
 - **Dependency and docs refresh** for v0.0.16 release readiness, onboarding parity, and public/private distribution operations.
@@ -280,6 +308,7 @@ See [CHANGELOG.md](CHANGELOG.md) for complete version history.
 - **Click** - Click elements by ref
 - **Type** - Enter text into inputs
 - **Select** - Choose dropdown options
+- **Upload** - Send files to a file input or chooser by ref
 - **Scroll** - Scroll page or elements
 - **Wait** - Wait for selectors or navigation
 
@@ -287,12 +316,14 @@ See [CHANGELOG.md](CHANGELOG.md) for complete version history.
 - **Console Capture** - Monitor console.log, errors, warnings
 - **Network Tracking** - Request/response metadata (method, url, status)
 - **Debug Trace Snapshot** - Combined page/console/network/exception diagnostics with blocker metadata
-- **Screenshot** - Viewport PNG screenshot (file or base64)
+- **Screenshot** - Visible, ref-targeted, or full-page PNG capture (file or base64)
+- **Dialog** - Inspect or handle JavaScript dialogs per target
 - **Performance** - Page load metrics
 
 ### Session & Macro Utilities
 - **Cookie Import** - Validate and import cookies into active sessions
 - **Cookie List** - First-class cookie inspection with optional URL filters
+- **Session Inspector** - Session-first diagnostics with relay health, trace proof, and a suggested next action
 - **Macro Resolve/Execute** - Expand macro expressions into provider actions with optional execution
 
 ### Export & Clone
@@ -304,9 +335,11 @@ See [CHANGELOG.md](CHANGELOG.md) for complete version history.
 
 ## Tool Reference
 
-OpenDevBrowser provides **49 tools** organized by category:
+OpenDevBrowser provides **65 tools** organized by category:
 Most runtime actions also have CLI command equivalents (see [docs/CLI.md](docs/CLI.md)).
 Complete source-accurate inventory (tools + CLI + `/ops` + `/canvas` + `/cdp`): [docs/SURFACE_REFERENCE.md](docs/SURFACE_REFERENCE.md).
+Terminal help now mirrors the generated public-surface manifest rooted at `src/public-surface/source.ts` and refreshed by `scripts/generate-public-surface-manifest.mjs`. `npx opendevbrowser --help` and `npx opendevbrowser help` both show every command with its usage and primary flags, every grouped CLI flag, and every bundled `opendevbrowser_*` tool with its CLI equivalent or tool-only scope.
+See [docs/ASSET_INVENTORY.md](docs/ASSET_INVENTORY.md) for the brand and generated help/public-surface asset inventory used by packaging and website-sync flows.
 
 ### Session Management
 | Tool | Description |
@@ -317,6 +350,7 @@ Complete source-accurate inventory (tools + CLI + `/ops` + `/canvas` + `/cdp`):
 | `opendevbrowser_status` | Get session status and connection info (daemon status in hub mode) |
 | `opendevbrowser_cookie_import` | Import validated cookies into the current session |
 | `opendevbrowser_cookie_list` | List session cookies with optional URL filters |
+| `opendevbrowser_session_inspector` | Capture a session-first diagnostic bundle with relay health, trace proof, and a suggested next action |
 
 ### Tab/Target Management
 | Tool | Description |
@@ -339,6 +373,7 @@ Complete source-accurate inventory (tools + CLI + `/ops` + `/canvas` + `/cdp`):
 | `opendevbrowser_goto` | Navigate to URL |
 | `opendevbrowser_wait` | Wait for load state or element |
 | `opendevbrowser_snapshot` | Capture page accessibility tree with refs |
+| `opendevbrowser_review` | Capture target-aware actionables plus status context before acting |
 | `opendevbrowser_click` | Click element by ref |
 | `opendevbrowser_hover` | Hover element by ref |
 | `opendevbrowser_press` | Press a keyboard key (optionally focusing a ref) |
@@ -348,6 +383,11 @@ Complete source-accurate inventory (tools + CLI + `/ops` + `/canvas` + `/cdp`):
 | `opendevbrowser_select` | Select dropdown option by ref |
 | `opendevbrowser_scroll` | Scroll page or element |
 | `opendevbrowser_scroll_into_view` | Scroll element into view by ref |
+| `opendevbrowser_upload` | Upload files to a file input or chooser by ref |
+| `opendevbrowser_pointer_move` | Move the pointer to viewport coordinates |
+| `opendevbrowser_pointer_down` | Press a mouse button at viewport coordinates |
+| `opendevbrowser_pointer_up` | Release a mouse button at viewport coordinates |
+| `opendevbrowser_pointer_drag` | Drag between viewport coordinates |
 | `opendevbrowser_run` | Execute multiple actions in sequence |
 
 ### DOM Inspection
@@ -368,9 +408,24 @@ Complete source-accurate inventory (tools + CLI + `/ops` + `/canvas` + `/cdp`):
 | `opendevbrowser_network_poll` | Poll network requests since sequence |
 | `opendevbrowser_debug_trace_snapshot` | Capture a unified page + console + network + exception diagnostic bundle |
 | `opendevbrowser_screenshot` | Capture page screenshot |
+| `opendevbrowser_screencast_start` | Start a browser replay screencast capture |
+| `opendevbrowser_screencast_stop` | Stop a browser replay screencast capture and return artifact metadata |
+| `opendevbrowser_dialog` | Inspect or handle a JavaScript dialog |
 | `opendevbrowser_perf` | Get page performance metrics |
 | `opendevbrowser_prompting_guide` | Get best-practice prompting guidance |
 
+### Desktop Observation
+| Tool | Description |
+|------|-------------|
+| `opendevbrowser_desktop_status` | Inspect public read-only desktop observation availability |
+| `opendevbrowser_desktop_windows` | List windows exposed by the public read-only desktop observation plane |
+| `opendevbrowser_desktop_active_window` | Inspect the active window through the public read-only desktop observation plane |
+| `opendevbrowser_desktop_capture_desktop` | Capture the current desktop surface through the public read-only desktop observation plane |
+| `opendevbrowser_desktop_capture_window` | Capture a specific window through the public read-only desktop observation plane |
+| `opendevbrowser_desktop_accessibility_snapshot` | Capture desktop accessibility state through the public read-only desktop observation plane |
+
+Desktop observation currently ships as a public read-only macOS surface. Availability, window inventory, and accessibility snapshots rely on the local `swift` command, while screenshot capture uses the built-in `screencapture` utility. If `desktop-status` reports `desktop_unsupported` on macOS, install Xcode or a Swift toolchain and retry.
+
 ### Macro Workflows
 | Tool | Description |
 |------|-------------|
@@ -395,18 +450,19 @@ Complete source-accurate inventory (tools + CLI + `/ops` + `/canvas` + `/cdp`):
 ### Skills
 | Tool | Description |
 |------|-------------|
-| `opendevbrowser_skill_list` | List available skills |
-| `opendevbrowser_skill_load` | Load a skill by name (with optional topic filter) |
+| `opendevbrowser_skill_list` | List available skills before choosing a local workflow lane |
+| `opendevbrowser_skill_load` | Load a skill by name and topic, especially the bundled quick start |
 
 ---
 
 ## Bundled Skills
 
-OpenDevBrowser includes **8 OpenDevBrowser-specific skill packs** plus shared `research` and `shopping` compatibility directories that are synced by the installer:
+OpenDevBrowser includes **9 OpenDevBrowser-specific skill packs**. Install, update, and uninstall own the managed skill lifecycle across OpenCode, Codex, ClaudeCode, and AmpCLI targets:
 
 | Skill | Purpose |
 |-------|---------|
 | `opendevbrowser-best-practices` | Core prompting patterns and workflow guidance |
+| `opendevbrowser-design-agent` | Contract-first, research-backed frontend and `/canvas` design execution |
 | `opendevbrowser-continuity-ledger` | Long-running task state management |
 | `opendevbrowser-login-automation` | Authentication flow patterns |
 | `opendevbrowser-form-testing` | Form validation and submission workflows |
@@ -416,7 +472,9 @@ OpenDevBrowser includes **8 OpenDevBrowser-specific skill packs** plus shared `r
 | `opendevbrowser-product-presentation-asset` | Product screenshot/copy asset collection for presentation pipelines |
 
 Installer note:
-- `--skills-global` and `--skills-local` copy every directory under `skills/`, including the shared `research/` and `shopping/` packs.
+- `--skills-global` and `--skills-local` sync the 9 canonical `opendevbrowser-*` packs into managed global or project-local agent directories.
+- Reinstall and update refresh drifted managed copies and leave matching packs unchanged.
+- Uninstall removes managed canonical packs and only prunes legacy `research` or `shopping` leftovers when those directories are empty and clearly obsolete.
 
 Skills are discovered from (priority order):
 1. `.opencode/skill/` (project)
@@ -428,8 +486,10 @@ Skills are discovered from (priority order):
 7. `.amp/skills/` (AmpCLI project compatibility)
 8. `$AMPCLI_HOME/skills` or `$AMP_CLI_HOME/skills` or `$AMP_HOME/skills` (AmpCLI global compatibility; fallback `~/.amp/skills`)
 9. Custom paths via `skillPaths` config
+10. Bundled package fallback: packaged `skills/` directory after `skillPaths` when no installed copy matches
 
 Load a skill: `opendevbrowser_skill_load` with `name` and optional `topic` filter.
+`opendevbrowser_prompting_guide`, `opendevbrowser_skill_list`, and `opendevbrowser_skill_load` are local onboarding helpers, so they do not require a browser session, relay, or daemon bootstrap.
 
 ---
 
@@ -447,7 +507,7 @@ Extension relay relies on **flat CDP sessions (Chrome 125+)** and uses DebuggerS
 
 Relay ops endpoint: `ws://127.0.0.1:<relayPort>/ops`.
 The connect command also accepts base relay WS URLs (`ws://127.0.0.1:<relayPort>` or `ws://localhost:<relayPort>`) and normalizes them to `/ops`.
-Relay canvas endpoint: `ws://127.0.0.1:<relayPort>/canvas` for design-canvas editor, session attach/lease flow, code-sync, preview, feedback, and overlay flows.
+Relay canvas endpoint: `ws://127.0.0.1:<relayPort>/canvas` for design-canvas editor, session attach/lease flow, code-sync, preview, public feedback pull streams, and overlay flows.
 Legacy relay `/cdp` remains available with explicit opt-in (`--extension-legacy`).
 When pairing is enabled, `/ops`, `/canvas`, and `/cdp` require a relay token (`?token=<relayToken>`). Tools and the CLI auto-fetch relay config and tokens.
 
@@ -456,7 +516,7 @@ When pairing is enabled, `/ops`, `/canvas`, and `/cdp` require a relay token (`?
 | Channel | What It Does | When to Use It |
 |---------|---------------|----------------|
 | **`/ops` (default)** | High-level automation protocol with session ownership, event streaming, and multi-client handling | Preferred extension relay path for modern workflows |
-| **`/canvas`** | Typed design-canvas protocol for session handshakes/attach, live editor sync, TSX code sync, preview tabs, feedback streams, and overlay selection | Use with `opendevbrowser_canvas` or `opendevbrowser canvas` during design-canvas workflows |
+| **`/canvas`** | Typed design-canvas protocol for session handshakes/attach, Figma document import, reusable inventory list/insert, built-in starter list/apply flows, framework-adapter-backed code sync, preview tabs, public feedback pull streams, and overlay selection | Use with `opendevbrowser_canvas` or `opendevbrowser canvas` during design-canvas workflows |
 | **`/cdp` (legacy)** | Low-level CDP relay path with compatibility-focused behavior | Opt-in compatibility mode (`--extension-legacy`) |
 | **Direct CDP connect** | Attach to Chrome started with `--remote-debugging-port` | Existing debug/browser setups without extension relay |
 
@@ -513,7 +573,7 @@ If the relay is unavailable, the background worker retries `/config` + `/pair` w
 - If `relayPort` is `0`, the relay is off.
 - `relayToken: false` disables relay/hub behavior entirely.
 - `relayToken: ""` (empty string) keeps relay enabled but disables pairing requirements.
-- Install auto-start with `npx opendevbrowser daemon install` so the relay is available on login.
+- Install auto-start with `opendevbrowser daemon install` from a stable install location so the relay is available on login.
 - Clear extension local data and retry if the token/port seem stuck.
 - If another process owns the port, change `relayPort` or stop it; `opencode` listening is expected.
 
@@ -575,9 +635,25 @@ Optional config file: `~/.config/opencode/opendevbrowser.jsonc`
     "cookieSource": {
       "type": "file",
       "value": "~/.config/opencode/opendevbrowser.provider-cookies.json"
+    },
+    "challengeOrchestration": {
+      "mode": "browser_with_helper",
+      "optionalComputerUseBridge": {
+        "enabled": true
+      }
     }
   },
 
+  // Public read-only sibling desktop observation runtime (enabled by default; set "off" to opt out)
+  // On macOS, availability, window, and accessibility probes require the local swift command.
+  "desktop": {
+    "permissionLevel": "observe",
+    "commandTimeoutMs": 10000,
+    "auditArtifactsDir": ".opendevbrowser/desktop-runtime",
+    "accessibilityMaxDepth": 2,
+    "accessibilityMaxChildren": 25
+  },
+
   // Skills configuration
   "skills": {
     "nudge": {
@@ -618,10 +694,10 @@ All fields are optional. OpenDevBrowser works with sensible defaults.
 
 ## CLI Commands
 
-The CLI is agent-agnostic and supports the full automation surface (session, navigation, interaction, DOM, targets, pages, export, devtools, annotate, and canvas).
+The CLI is agent-agnostic and supports the full automation surface (session, navigation, interaction, DOM, browser capture and replay, desktop observation, targets, pages, export, devtools, annotate, and canvas).
 All commands listed in the CLI reference are implemented and available in the current codebase.
 See [docs/CLI.md](docs/CLI.md) for the full command and flag matrix.
-See [docs/SURFACE_REFERENCE.md](docs/SURFACE_REFERENCE.md) for the source-accurate inventory matrix (CLI commands, 49 tools, `/ops`, `/canvas`, and `/cdp` channel contracts).
+See [docs/SURFACE_REFERENCE.md](docs/SURFACE_REFERENCE.md) for the source-accurate inventory matrix (72 CLI commands, 65 tools, `/ops`, `/canvas`, and `/cdp` channel contracts).
 
 ### CLI Category Matrix (core command groups)
 
@@ -630,11 +706,13 @@ See [docs/SURFACE_REFERENCE.md](docs/SURFACE_REFERENCE.md) for the source-accura
 | Install/runtime | `install`, `update`, `uninstall`, `help`, `version`, `serve`, `daemon`, `native`, `run` |
 | Session/connection | `launch`, `connect`, `disconnect`, `status`, `cookie-import`, `cookie-list` |
 | Navigation | `goto`, `wait`, `snapshot` |
-| Interaction | `click`, `hover`, `press`, `check`, `uncheck`, `type`, `select`, `scroll`, `scroll-into-view` |
+| Interaction | `click`, `hover`, `press`, `check`, `uncheck`, `type`, `select`, `scroll`, `scroll-into-view`, `upload`, `pointer-move`, `pointer-down`, `pointer-up`, `pointer-drag` |
 | Targets/pages | `targets-list`, `target-use`, `target-new`, `target-close`, `page`, `pages`, `page-close` |
 | DOM | `dom-html`, `dom-text`, `dom-attr`, `dom-value`, `dom-visible`, `dom-enabled`, `dom-checked` |
+| Browser capture | `screenshot`, `screencast-start`, `screencast-stop` |
+| Desktop observation | `desktop-status`, `desktop-windows`, `desktop-active-window`, `desktop-capture-desktop`, `desktop-capture-window`, `desktop-accessibility-snapshot` |
 | Design canvas | `canvas` |
-| Export/diagnostics/macro/annotation/power | `clone-page`, `clone-component`, `perf`, `screenshot`, `console-poll`, `network-poll`, `debug-trace-snapshot`, `macro-resolve`, `annotate`, `rpc` |
+| Export/diagnostics/macro/annotation/power | `clone-page`, `clone-component`, `perf`, `dialog`, `console-poll`, `network-poll`, `debug-trace-snapshot`, `session-inspector`, `macro-resolve`, `annotate`, `rpc` |
 
 ### Install/Management
 
@@ -660,6 +738,7 @@ Start the daemon with `npx opendevbrowser serve`, then use:
 | `npx opendevbrowser connect` | Connect via relay or direct CDP endpoint |
 | `npx opendevbrowser disconnect` | Disconnect session |
 | `npx opendevbrowser status` | Show session status |
+| `npx opendevbrowser session-inspector --session-id <id>` | Capture a session-first diagnostic bundle with relay health, trace proof, and a suggested next action |
 | `npx opendevbrowser goto` | Navigate to URL |
 | `npx opendevbrowser wait` | Wait for load or element |
 | `npx opendevbrowser snapshot` | Capture snapshot with refs |
@@ -676,6 +755,14 @@ Workflow cookie controls (`research run`, `shopping run`, `product-video run`):
 - Per-run overrides: `--use-cookies`, `--cookie-policy-override` (alias `--cookie-policy`).
 - `auto` is non-blocking when cookies are unavailable; `required` fails fast with `reasonCode=auth_required`.
 
+Workflow challenge controls (`research run`, `shopping run`, `product-video run`):
+- Per-run override: `--challenge-automation-mode off|browser|browser_with_helper`, which maps to `challengeAutomationMode`.
+- Effective precedence is `run > session > config`.
+- `off` keeps detection and reporting active but stands down challenge actions.
+- `browser` allows browser-native lanes only and keeps the helper bridge disabled.
+- `browser_with_helper` keeps browser-native lanes first and evaluates the browser-scoped helper bridge second when hard gates pass.
+- The helper bridge remains browser-scoped and is not a desktop agent.
+
 ---
 
 ## Security
@@ -712,6 +799,7 @@ Release checklist: [docs/DISTRIBUTION_PLAN.md](docs/DISTRIBUTION_PLAN.md)
 Documentation index: [docs/README.md](docs/README.md)
 Frontend docs: [docs/FRONTEND.md](docs/FRONTEND.md)
 Dependency inventory: [docs/DEPENDENCIES.md](docs/DEPENDENCIES.md)
+Local-only generated artifacts such as `prompt-exports/`, root `artifacts/`, `coverage/`, `CONTINUITY*.md`, and `sub_continuity.md` stay uncommitted; `.gitignore` is authoritative.
 
 ---
 
@@ -729,7 +817,8 @@ Dependency inventory: [docs/DEPENDENCIES.md](docs/DEPENDENCIES.md)
          ▼                  ▼                  ▼                       ▼
 ┌─────────────────────────────────────────────────────────────────┐
 │                    Core Runtime (src/core/)                    │
-│  bootstrap.ts → wires managers, injects ToolDeps              │
+│  bootstrap.ts → wires managers, sibling desktop runtime,       │
+│                   automation coordinator, injects ToolDeps     │
 └────────┬────────────────────────────────────────────────────────┘
          │
     ┌────┴────┬─────────────┬──────────────┬──────────────┬──────────────┐
@@ -746,6 +835,8 @@ Dependency inventory: [docs/DEPENDENCIES.md](docs/DEPENDENCIES.md)
 └────────┘                                        └────────────┘
 ```
 
+The simplified map above omits the dedicated Challenge Coordinator, Desktop Runtime, and Automation Coordinator that now sit beside the browser managers; see [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full component map and ownership boundaries.
+
 ### Data Flow
 
 ```
@@ -768,22 +859,29 @@ Tool Call → Zod Validation → Manager/Runner → CDP/Playwright → Response
 ```
 .
 ├── src/              # Plugin implementation
-│   ├── browser/      # BrowserManager, TargetManager, CDP lifecycle
+│   ├── annotate/     # Annotation transports + output shaping
+│   ├── automation/    # Automation helpers and coordinator
+│   ├── browser/      # Browser sessions, target orchestration, canvas preview/code-sync
 │   ├── cache/        # Chrome executable resolution
+│   ├── canvas/       # Design-canvas document store, repo IO, code-sync, export helpers
+│   ├── challenges/   # Bounded challenge orchestration plane, evidence, recovery lanes
 │   ├── cli/          # CLI commands, daemon, installers
 │   ├── core/         # Bootstrap, runtime wiring, ToolDeps
-│   ├── canvas/       # Design-canvas document store, repo IO, export helpers
+│   ├── desktop/      # Read-only desktop observation runtime
 │   ├── devtools/     # Console/network trackers with redaction
 │   ├── export/       # DOM capture, React emitter, CSS extraction
+│   ├── integrations/ # External integration adapters (Figma import, etc.)
+│   ├── macros/       # Macro parsing, resolution, provider-action expansion
+│   ├── providers/    # Provider runtime, policy, workflows, browser fallback
+│   ├── public-surface/ # Generated manifest source, CLI/tool metadata
 │   ├── relay/        # Extension relay server, protocol types
 │   ├── skills/       # SkillLoader for skill pack discovery
 │   ├── snapshot/     # AX-tree snapshots, ref management
-│   ├── tools/        # 49 opendevbrowser_* tool definitions
-│   ├── annotate/     # Annotation transports + output shaping
+│   ├── tools/        # 65 opendevbrowser_* tool definitions
 │   └── utils/        # Shared utilities
 ├── extension/        # Chrome extension (relay client)
 ├── scripts/          # Operational scripts (build/sync/smoke)
-├── skills/           # Bundled skill directories (10 total; 8 canonical OpenDevBrowser packs + 2 shared compatibility packs)
+├── skills/           # Bundled skill directories (11 total; 9 canonical OpenDevBrowser packs + 2 shared compatibility packs)
 ├── tests/            # Vitest tests (97% coverage required)
 └── docs/             # Architecture, CLI, extension, distribution plans
 ```
@@ -820,8 +918,8 @@ Uniform versioning is required (source of truth: `package.json`):
    - `node scripts/chrome-store-compliance-check.mjs`
    - `./skills/opendevbrowser-best-practices/scripts/validate-skill-assets.sh`
 8. Run strict live release gates:
-   - `node scripts/provider-live-matrix.mjs --release-gate --out artifacts/release/vX.Y.Z/provider-live-matrix.json`
-   - `node scripts/live-regression-matrix.mjs --release-gate`
+   - `node scripts/provider-direct-runs.mjs --release-gate --out artifacts/release/vX.Y.Z/provider-direct-runs.json`
+   - `node scripts/live-regression-direct.mjs --release-gate --out artifacts/release/vX.Y.Z/live-regression-direct.json`
 9. Run first-time global install dry run checklist from `docs/FIRST_RUN_ONBOARDING.md`.
 10. Run: `npm run extension:pack` (outputs `./opendevbrowser-extension.zip`)
 11. Run: `npm pack`

package/dist/annotate/agent-inbox-store.d.ts

@@ -0,0 +1,58 @@
+import type { AgentInboxDeliveryState, AgentInboxReceipt, AnnotationDispatchSource, AnnotationPayload } from "../relay/protocol";
+export type AgentInboxAssetRef = {
+    id: string;
+    kind: "screenshot";
+    label?: string | null;
+    metadata: Record<string, unknown>;
+};
+export type AgentInboxScopeRegistration = {
+    chatScopeKey: string;
+    updatedAt: string;
+    messageId?: string | null;
+    agent?: string | null;
+    model?: {
+        providerID: string;
+        modelID: string;
+    } | null;
+    variant?: string | null;
+};
+export type AgentInboxEntry = {
+    id: string;
+    worktree: string;
+    chatScopeKey: string | null;
+    source: AnnotationDispatchSource;
+    label: string;
+    createdAt: string;
+    deliveryState: AgentInboxDeliveryState;
+    payloadSansScreenshots: AnnotationPayload;
+    assetRefs: AgentInboxAssetRef[];
+    payloadHash: string;
+    itemCount: number;
+    byteLength: number;
+    receipt: AgentInboxReceipt;
+};
+type EnqueueInput = {
+    payload: AnnotationPayload;
+    source: AnnotationDispatchSource;
+    label: string;
+    explicitChatScopeKey?: string | null;
+};
+export declare class AgentInboxStore {
+    private readonly worktree;
+    private readonly now;
+    constructor(worktree: string, now?: () => number);
+    registerScope(chatScopeKey: string, registration?: Omit<AgentInboxScopeRegistration, "chatScopeKey" | "updatedAt">): AgentInboxScopeRegistration;
+    listActiveScopes(): AgentInboxScopeRegistration[];
+    enqueue(input: EnqueueInput): AgentInboxEntry;
+    peekScope(chatScopeKey: string): AgentInboxEntry[];
+    latestEntry(): AgentInboxEntry | null;
+    consume(receiptIds: string[]): void;
+    private resolveScope;
+    private readEntries;
+    private writeEntries;
+    private readScopes;
+    private writeScopes;
+    private ensureInboxDir;
+}
+export {};
+//# sourceMappingURL=agent-inbox-store.d.ts.map

package/dist/annotate/agent-inbox-store.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"agent-inbox-store.d.ts","sourceRoot":"","sources":["../../src/annotate/agent-inbox-store.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EACV,uBAAuB,EACvB,iBAAiB,EACjB,wBAAwB,EACxB,iBAAiB,EAClB,MAAM,mBAAmB,CAAC;AAY3B,MAAM,MAAM,kBAAkB,GAAG;IAC/B,EAAE,EAAE,MAAM,CAAC;IACX,IAAI,EAAE,YAAY,CAAC;IACnB,KAAK,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACtB,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CACnC,CAAC;AAEF,MAAM,MAAM,2BAA2B,GAAG;IACxC,YAAY,EAAE,MAAM,CAAC;IACrB,SAAS,EAAE,MAAM,CAAC;IAClB,SAAS,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IAC1B,KAAK,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;IACtB,KAAK,CAAC,EAAE;QACN,UAAU,EAAE,MAAM,CAAC;QACnB,OAAO,EAAE,MAAM,CAAC;KACjB,GAAG,IAAI,CAAC;IACT,OAAO,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;CACzB,CAAC;AAEF,MAAM,MAAM,eAAe,GAAG;IAC5B,EAAE,EAAE,MAAM,CAAC;IACX,QAAQ,EAAE,MAAM,CAAC;IACjB,YAAY,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,MAAM,EAAE,wBAAwB,CAAC;IACjC,KAAK,EAAE,MAAM,CAAC;IACd,SAAS,EAAE,MAAM,CAAC;IAClB,aAAa,EAAE,uBAAuB,CAAC;IACvC,sBAAsB,EAAE,iBAAiB,CAAC;IAC1C,SAAS,EAAE,kBAAkB,EAAE,CAAC;IAChC,WAAW,EAAE,MAAM,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC;IAClB,UAAU,EAAE,MAAM,CAAC;IACnB,OAAO,EAAE,iBAAiB,CAAC;CAC5B,CAAC;AAMF,KAAK,YAAY,GAAG;IAClB,OAAO,EAAE,iBAAiB,CAAC;IAC3B,MAAM,EAAE,wBAAwB,CAAC;IACjC,KAAK,EAAE,MAAM,CAAC;IACd,oBAAoB,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;CACtC,CAAC;AAOF,qBAAa,eAAe;IAC1B,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAS;IAClC,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAe;gBAEvB,QAAQ,EAAE,MAAM,EAAE,GAAG,GAAE,MAAM,MAAyB;IAKlE,aAAa,CACX,YAAY,EAAE,MAAM,EACpB,YAAY,GAAE,IAAI,CAAC,2BAA2B,EAAE,cAAc,GAAG,WAAW,CAAM,GACjF,2BAA2B;IAmB9B,gBAAgB,IAAI,2BAA2B,EAAE;IAIjD,OAAO,CAAC,KAAK,EAAE,YAAY,GAAG,eAAe;IAuD7C,SAAS,CAAC,YAAY,EAAE,MAAM,GAAG,eAAe,EAAE;IAUlD,WAAW,IAAI,eAAe,GAAG,IAAI;IAKrC,OAAO,CAAC,UAAU,EAAE,MAAM,EAAE,GAAG,IAAI;IAuBnC,OAAO,CAAC,YAAY;IAepB,OAAO,CAAC,WAAW;IAqBnB,OAAO,CAAC,YAAY;IAOpB,OAAO,CAAC,UAAU;IAclB,OAAO,CAAC,WAAW;IAMnB,OAAO,CAAC,cAAc;CAOvB"}

package/dist/annotate/agent-inbox.d.ts

@@ -0,0 +1,25 @@
+import type { AgentInboxReceipt, AnnotationDispatchSource, AnnotationPayload } from "../relay/protocol";
+import type { AgentInboxEntry, AgentInboxScopeRegistration } from "./agent-inbox-store";
+export declare const AGENT_INBOX_SYSTEM_MARKER = "[opendevbrowser-agent-inbox]";
+export type AgentInboxSystemInjection = {
+    systemBlock: string;
+    receiptIds: string[];
+};
+export declare class AgentInbox {
+    private readonly store;
+    constructor(worktree: string, now?: () => number);
+    registerScope(chatScopeKey: string, registration?: Omit<AgentInboxScopeRegistration, "chatScopeKey" | "updatedAt">): AgentInboxScopeRegistration;
+    listActiveScopes(): AgentInboxScopeRegistration[];
+    enqueue(input: {
+        payload: AnnotationPayload;
+        source: AnnotationDispatchSource;
+        label: string;
+        explicitChatScopeKey?: string | null;
+    }): AgentInboxReceipt;
+    peekScope(chatScopeKey: string): AgentInboxEntry[];
+    consumeScope(chatScopeKey: string): AgentInboxEntry[];
+    acknowledge(receiptIds: string[]): void;
+    latestPayload(): AnnotationPayload | null;
+    buildSystemInjection(chatScopeKey: string): AgentInboxSystemInjection | null;
+}
+//# sourceMappingURL=agent-inbox.d.ts.map