@restregular/dsh-visual-browser 0.1.0 → 0.1.1

package/README.md

@@ -1,124 +1,129 @@
-# dsh-visual-browser
-
-A DeepSeek Harness plugin that gives the Agent a **real Chrome** and gives you a **window into it**.
-
-- The Agent never reads the DOM. It reads a numbered list of *visible interactive
-  elements* plus a short text excerpt — a few hundred tokens per page instead of a
-  full snapshot.
-- Every action the Agent takes (click, type, key, scroll, navigation) is mirrored
-  into a **面板 (panel)** with the page frame, an operation timeline you can replay,
-  and a picture you can click to take over the browser yourself.
-- Cookies persist across restarts (one dedicated Chrome profile), so a login the
-  Agent performs once keeps working.
-
-## Install
-
-```bash
-# The plugin is a normal DSH bundle; install it from the registry
-dsh plugin add @restregular/dsh-visual-browser
-```
-
-Or from the Harness UI: **Plugins → Add → `@restregular/dsh-visual-browser`**.
-
-Requirements: Google Chrome, Microsoft Edge, or Chromium on the machine
-(auto-detected), and Node.js 22 or newer (the plugin ships no runtime
-dependencies; it talks CDP over the built-in WebSocket).
-
-## Use
-
-1. Click the **浏览器 / Browser** icon in the sidebar to open the panel.
-2. Ask the Agent something that needs a website, for example
-   *"打开 example.com 看看首页写了什么"*. The Agent calls `browser_open`.
-3. Watch the frame update. Click the picture to click the page; wheel over it to
-   scroll; use the toolbar for back / forward / reload / live view / stop.
-
-## Tools the Agent gets
-
-| Tool | Purpose |
-| --- | --- |
-| `browser_open` | Open a URL (launches Chrome on first use) and return the numbered view. |
-| `browser_see` | Fresh numbered view of the current page. |
-| `browser_act` | `click` / `type` / `key` / `hover` / `scroll` / `select` / `back` / `forward` / `reload` / `wait` / `close`, then return the new view. |
-| `browser_text` | Read text of the page or of one element (a link also returns its `href`). |
-| `browser_screenshot` | Save a PNG/JPEG to disk and return the path (feed it to `read_image`). |
-
-### What the Agent sees
-
-```
-URL: https://example.com/login
-TITLE: 登录 - Example
-VIEW: 1280x800 scroll 0/2400 · 4/6 interactive
-[1] textbox "手机号" value=13800000000
-[2] textbox "密码" value=••••••
-[3] button "登录"
-[4] link "忘记密码" → /forgot
-HEADINGS: 用户登录 | 没有账号?
-TEXT: 用户登录 手机号 密码 登录 忘记密码
-ACTIVE: [1] textbox "手机号"
-```
-
-Numbers are reassigned on every view, so an action always uses the newest
-numbers; a stale one fails with a clear "take a fresh snapshot" message instead
-of clicking the wrong element.
-
-## Configuration
-
-Row config (in the profile's `cordis.patch.yml`, override by row id
-`dsh-visual-browser`):
-
-```yaml
-- id: dsh-visual-browser
-  config:
-    headless: true            # false = visible Chrome window
-    executablePath: ''        # explicit chrome.exe path
-    userDataDir: ''           # default: ~/.dsh/visual-browser/chrome-profile
-    viewport:
-      width: 1280
-      height: 800
-    maxElements: 100          # cap on the numbered list
-    maxTextChars: 320         # cap on the text excerpt
-    frameQuality: 62          # JPEG quality of panel frames
-    timelineFrames: 24        # replayable frames kept in memory
-    allowUserControl: true    # allow clicks from the panel
-    actionTimeoutMs: 12000
-    navigationTimeoutMs: 30000
-```
-
-## How it works
-
-- **Host** (`index.js`, `lib/`): launches Chrome with
-  `--remote-debugging-port=0 --user-data-dir=<profile>`, parses the DevTools
-  endpoint from stderr, attaches one flattened CDP session, and exposes:
-  - five tools through the `tools` service (registration is effect-scoped),
-  - a same-origin prefix route `/visual-browser/api` (`/state`, `/frame`,
-    `/live`, `/start`, `/stop`, `/capture`, `/click`, `/wheel`, `/key`, `/nav`)
-    that the panel polls and posts to.
-- **Snapshot** (`lib/page-scripts.js`): one serialized function runs in the page,
-  filters interactive elements to the viewport, assigns `data-dsh-vb-ref`
-  numbers, and returns structured JSON. Nothing else about the DOM leaves the
-  page.
-- **Frames** stay on the host: the panel displays the newest JPEG by sequence
-  number, so a live picture costs no session tokens. The timeline holds the last
-  N frames; clicking a row replays that moment.
-- **One code path**: panel clicks, wheel events and keys run the same
-  `BrowserManager` methods the tools use.
-
-Chrome is killed with its process tree when the plugin unloads.
-
-## Notes and limits
-
-- Cross-origin iframes are counted but not inspected; open the direct URL when a
-  flow lives inside a frame.
-- Blocking JS dialogs are auto-accepted and logged in the timeline.
-- The panel route is served by the Harness web server on its existing origin;
-  it performs no extra authentication, matching the other local panel routes.
-- Headless mode is the default. Some sites behave differently headless; set
-  `headless: false` when a site insists on a window.
-
-## Development
-
-```bash
-npm test    # syntax checks + snapshot/format/route/package smoke tests
-```
-
-MIT © RestRegular
+# dsh-visual-browser
+
+A DeepSeek Harness plugin that gives the Agent a **real Chrome** and gives you a **window into it**.
+
+- The Agent never reads the DOM. It reads a numbered list of *visible interactive
+  elements* plus a short text excerpt — a few hundred tokens per page instead of a
+  full snapshot.
+- Every action the Agent takes (click, type, key, scroll, navigation) is mirrored
+  into a **面板 (panel)** with the page frame, an operation timeline you can replay,
+  and a picture you can click to take over the browser yourself.
+- Cookies persist across restarts (one dedicated Chrome profile), so a login the
+  Agent performs once keeps working.
+
+## Install
+
+```bash
+# The plugin is a normal DSH bundle; install it from the registry
+dsh plugin add @restregular/dsh-visual-browser
+```
+
+Or from the Harness UI: **Plugins → Add → `@restregular/dsh-visual-browser`**.
+
+Requirements: Google Chrome, Microsoft Edge, or Chromium on the machine
+(auto-detected), and Node.js 22 or newer (the plugin ships no runtime
+dependencies; it talks CDP over the built-in WebSocket).
+
+## Use
+
+1. Click the **浏览器 / Browser** icon in the sidebar to open the panel.
+2. Ask the Agent something that needs a website, for example
+   *"打开 example.com 看看首页写了什么"*. The Agent calls `browser_open`.
+3. Watch the frame update. Click the picture to click the page; wheel over it to
+   scroll; use the toolbar for back / forward / reload / live view / stop.
+
+## Tools the Agent gets
+
+| Tool | Purpose |
+| --- | --- |
+| `browser_open` | Open a URL (launches Chrome on first use) and return the numbered view. |
+| `browser_see` | Fresh numbered view of the current page. |
+| `browser_act` | `click` / `type` / `key` / `hover` / `scroll` / `select` / `back` / `forward` / `reload` / `wait` / `close`, then return the new view. |
+| `browser_text` | Read text of the page or of one element (a link also returns its `href`). |
+| `browser_screenshot` | Save a PNG/JPEG to disk and return the path (feed it to `read_image`). |
+
+### What the Agent sees
+
+```
+URL: https://example.com/login
+TITLE: 登录 - Example
+VIEW: 1280x800 scroll 0/2400 · 4/6 interactive
+[1] textbox "手机号" value=13800000000
+[2] textbox "密码" value=••••••
+[3] button "登录"
+[4] link "忘记密码" → /forgot
+HEADINGS: 用户登录 | 没有账号?
+TEXT: 用户登录 手机号 密码 登录 忘记密码
+ACTIVE: [1] textbox "手机号"
+```
+
+Numbers are reassigned on every view, so an action always uses the newest
+numbers; a stale one fails with a clear "take a fresh snapshot" message instead
+of clicking the wrong element.
+
+## Configuration
+
+Row config (in the profile's `cordis.patch.yml`, override by row id
+`dsh-visual-browser`):
+
+```yaml
+- id: dsh-visual-browser
+  config:
+    headless: true            # false = visible Chrome window
+    executablePath: ''        # explicit chrome.exe path
+    userDataDir: ''           # default: ~/.dsh/visual-browser/chrome-profile
+    viewport:
+      width: 1280
+      height: 800
+    maxElements: 100          # cap on the numbered list
+    maxTextChars: 320         # cap on the text excerpt
+    frameQuality: 62          # JPEG quality of panel frames
+    timelineFrames: 24        # replayable frames kept in memory
+    allowUserControl: true    # allow clicks from the panel
+    reuseExisting: true       # adopt a Chrome that already owns the profile
+    actionTimeoutMs: 12000
+    navigationTimeoutMs: 30000
+```
+
+## How it works
+
+- **Host** (`index.js`, `lib/`): launches Chrome with
+  `--remote-debugging-port=0 --user-data-dir=<profile>`, parses the DevTools
+  endpoint from stderr, attaches one flattened CDP session, and exposes:
+  - five tools through the `tools` service (registration is effect-scoped),
+  - a same-origin prefix route `/visual-browser/api` (`/state`, `/frame`,
+    `/live`, `/start`, `/stop`, `/capture`, `/click`, `/wheel`, `/key`, `/nav`)
+    that the panel polls and posts to.
+- **Snapshot** (`lib/page-scripts.js`): one serialized function runs in the page,
+  filters interactive elements to the viewport, assigns `data-dsh-vb-ref`
+  numbers, and returns structured JSON. Nothing else about the DOM leaves the
+  page.
+- **Frames** stay on the host: the panel displays the newest JPEG by sequence
+  number, so a live picture costs no session tokens. The timeline holds the last
+  N frames; clicking a row replays that moment.
+- **One code path**: panel clicks, wheel events and keys run the same
+  `BrowserManager` methods the tools use.
+- **Survives a host restart**: Chrome writes `DevToolsActivePort` into its
+  profile, so a restarted host (or one whose previous host crashed with Chrome
+  still alive) attaches to that browser instead of failing on the profile lock.
+  An adopted browser is only detached — its process is not killed.
+
+Chrome is killed with its process tree when the plugin unloads.
+
+## Notes and limits
+
+- Cross-origin iframes are counted but not inspected; open the direct URL when a
+  flow lives inside a frame.
+- Blocking JS dialogs are auto-accepted and logged in the timeline.
+- The panel route is served by the Harness web server on its existing origin;
+  it performs no extra authentication, matching the other local panel routes.
+- Headless mode is the default. Some sites behave differently headless; set
+  `headless: false` when a site insists on a window.
+
+## Development
+
+```bash
+npm test    # syntax checks + snapshot/format/route/package smoke tests
+```
+
+MIT © RestRegular

package/lib/browser.js

@@ -8,7 +8,7 @@
 
 import { spawn } from 'node:child_process';
 import { existsSync, mkdirSync } from 'node:fs';
-import { writeFile } from 'node:fs/promises';
+import { readFile, writeFile } from 'node:fs/promises';
 import { homedir, tmpdir } from 'node:os';
 import path from 'node:path';
 
@@ -45,6 +45,7 @@ export const DEFAULT_CONFIG = {
   frameQuality: 62,
   timelineFrames: 24,
   allowUserControl: true,
+  reuseExisting: true,
   actionTimeoutMs: 12_000,
   navigationTimeoutMs: 30_000,
   startupTimeoutMs: 30_000,
@@ -73,6 +74,7 @@ export function normalizeConfig(input = {}) {
     frameQuality: clamp(source.frameQuality, 20, 95, DEFAULT_CONFIG.frameQuality),
     timelineFrames: clamp(source.timelineFrames, 4, 120, DEFAULT_CONFIG.timelineFrames),
     allowUserControl: source.allowUserControl === undefined ? DEFAULT_CONFIG.allowUserControl : source.allowUserControl !== false,
+    reuseExisting: source.reuseExisting === undefined ? DEFAULT_CONFIG.reuseExisting : source.reuseExisting !== false,
     actionTimeoutMs: clamp(source.actionTimeoutMs, 500, 60_000, DEFAULT_CONFIG.actionTimeoutMs),
     navigationTimeoutMs: clamp(source.navigationTimeoutMs, 1000, 120_000, DEFAULT_CONFIG.navigationTimeoutMs),
     startupTimeoutMs: clamp(source.startupTimeoutMs, 2000, 120_000, DEFAULT_CONFIG.startupTimeoutMs),
@@ -213,6 +215,7 @@ export class BrowserManager {
   #targetId = null;
   #starting = null;
   #running = false;
+  #owned = false;
   #executable = '';
   #url = 'about:blank';
   #title = '';
@@ -281,6 +284,22 @@ export class BrowserManager {
     } catch (error) {
       throw new BrowserError('PROFILE_DIR_FAILED', `Could not create the browser profile directory ${userDataDir}: ${error.message}`);
     }
+    if (this.#config.reuseExisting) {
+      const reused = await this.#reuseProfile(userDataDir).catch((error) => {
+        this.#logger?.info?.(`[visual-browser] reuse probe failed: ${error?.message ?? error}`);
+        return false;
+      });
+      if (reused) {
+        this.#owned = false;
+        await this.#attach();
+        this.#running = true;
+        this.#lastError = '';
+        await this.#capture({ force: true }).catch(() => null);
+        this.#record('start', userDataDir, 'attached to a running Chrome');
+        this.#logger?.info?.('[visual-browser] attached to the Chrome that already owns this profile');
+        return;
+      }
+    }
     const args = [
       '--remote-debugging-port=0',
       `--user-data-dir=${userDataDir}`,
@@ -326,7 +345,11 @@ export class BrowserManager {
         if (match) finish(null, match[1]);
       };
       const onExit = (code) => {
-        finish(new BrowserError('BROWSER_EXITED', `Chrome exited with code ${code} before it was ready. ${stderr.slice(-600)}`));
+        const hint =
+          stderr.trim() === '' && code === 0
+            ? 'Chrome handed the launch to a browser that already owns this profile, but no DevTools endpoint answered. Close the leftover browser (the panel Stop button, or Task Manager) or set a different `userDataDir`.'
+            : stderr.slice(-600);
+        finish(new BrowserError('BROWSER_EXITED', `Chrome exited with code ${code} before it was ready. ${hint}`));
       };
       const onError = (error) => {
         finish(new BrowserError('SPAWN_FAILED', `Could not start ${executable}: ${error.message}`));
@@ -345,6 +368,7 @@ export class BrowserManager {
     });
 
     const cdp = await CdpConnection.open(endpoint);
+    this.#owned = true;
     this.#cdp = cdp;
     child.once('exit', () => {
       if (this.#proc === child) this.#proc = null;
@@ -356,8 +380,49 @@ export class BrowserManager {
     await this.#attach();
     this.#running = true;
     this.#lastError = '';
-    this.#record('start', '', `${this.#config.headless ? 'headless' : 'windowed'} Chrome started`);
     await this.#capture({ force: true }).catch(() => null);
+    this.#record('start', '', `${this.#config.headless ? 'headless' : 'windowed'} Chrome started`);
+  }
+
+  /**
+   * Chrome writes `<userDataDir>/DevToolsActivePort` when it starts with remote
+   * debugging. A host restart, or a crashed host whose Chrome survived, can
+   * therefore adopt the running browser instead of failing on the profile lock.
+   */
+  async #reuseProfile(userDataDir) {
+    const portFile = path.join(userDataDir, 'DevToolsActivePort');
+    let text = '';
+    try {
+      text = await readFile(portFile, 'utf8');
+    } catch {
+      return false;
+    }
+    const lines = text.split(/\r?\n/).map((line) => line.trim());
+    const port = Number(lines[0]);
+    const wsPath = lines[1] ?? '';
+    if (!Number.isInteger(port) || port <= 0 || port > 65535) return false;
+    if (!wsPath.startsWith('/devtools/browser/')) return false;
+    let cdp = null;
+    try {
+      cdp = await CdpConnection.open(`ws://127.0.0.1:${port}${wsPath}`, { timeoutMs: 3000 });
+      await cdp.send('Browser.getVersion');
+    } catch {
+      try {
+        cdp?.close();
+      } catch {
+        /* already closed */
+      }
+      return false;
+    }
+    this.#cdp = cdp;
+    cdp.onClose(() => {
+      if (this.#cdp === cdp) {
+        this.#running = false;
+        this.#sessionId = null;
+        this.#targetId = null;
+      }
+    });
+    return true;
   }
 
   async #attach() {
@@ -398,17 +463,32 @@ export class BrowserManager {
     });
   }
 
-  /** Shut Chrome down; safe to call when it never started. */
+  /** Shut Chrome down (or just detach when this host did not spawn it). */
   async close() {
     const proc = this.#proc;
+    const owned = this.#owned;
+    const cdp = this.#cdp;
+    const targetId = this.#targetId;
     this.#proc = null;
+    this.#owned = false;
     this.#running = false;
     this.#sessionId = null;
     this.#targetId = null;
-    const cdp = this.#cdp;
     this.#cdp = null;
-    cdp?.close();
-    if (proc !== null) {
+    if (owned) {
+      cdp?.close();
+    } else if (cdp !== null && targetId !== null) {
+      // Only our own tab is ours to close: another host may still use the browser.
+      try {
+        await Promise.race([cdp.send('Target.closeTarget', { targetId }), sleep(1000)]);
+      } catch {
+        /* the browser may already be gone */
+      }
+      cdp.close();
+    } else {
+      cdp?.close();
+    }
+    if (owned && proc !== null) {
       try {
         if (process.platform === 'win32') {
           spawn('taskkill', ['/pid', String(proc.pid), '/T', '/F'], { stdio: 'ignore', windowsHide: true });
@@ -505,8 +585,8 @@ export class BrowserManager {
     return this.#serial(async () => {
       await this.#navigate(target);
       const snapshot = await this.snapshot();
-      this.#record('open', target, snapshot.title || '');
       await this.#capture().catch(() => null);
+      this.#record('open', target, snapshot.title || '');
       return snapshot;
     });
   }
@@ -791,8 +871,8 @@ export class BrowserManager {
       }
       await this.#settleAfterAction();
       const snapshot = await this.snapshot();
-      this.#record(action, target.ref !== undefined ? `[${target.ref}]` : (target.selector ?? ''), summary.slice(0, 160));
       await this.#capture().catch(() => null);
+      this.#record(action, target.ref !== undefined ? `[${target.ref}]` : (target.selector ?? ''), summary.slice(0, 160));
       return { summary, snapshot };
     });
   }
@@ -810,8 +890,8 @@ export class BrowserManager {
       await this.#clickResolved({ x: px, y: py, name: '' });
       await this.#settleAfterAction();
       const snapshot = await this.snapshot();
-      this.#record('user-click', `${Math.round(x * 100)}%,${Math.round(y * 100)}%`, `you clicked ${Math.round(x * 100)}%, ${Math.round(y * 100)}%`);
       await this.#capture().catch(() => null);
+      this.#record('user-click', `${Math.round(x * 100)}%,${Math.round(y * 100)}%`, `you clicked ${Math.round(x * 100)}%, ${Math.round(y * 100)}%`);
       return { summary: 'clicked in the panel', snapshot };
     });
   }
@@ -830,8 +910,8 @@ export class BrowserManager {
         pointerType: 'mouse',
       });
       await sleep(180);
-      this.#record('user-scroll', `${deltaY || deltaX}px`, `you scrolled ${Math.abs(deltaY || deltaX)}px`);
       await this.#capture().catch(() => null);
+      this.#record('user-scroll', `${deltaY || deltaX}px`, `you scrolled ${Math.abs(deltaY || deltaX)}px`);
       return { summary: 'scrolled in the panel' };
     });
   }
@@ -843,8 +923,8 @@ export class BrowserManager {
     return this.#serial(async () => {
       const spec = await this.#press(key);
       await this.#settleAfterAction();
-      this.#record('user-key', spec.key, `you pressed ${spec.key}`);
       await this.#capture().catch(() => null);
+      this.#record('user-key', spec.key, `you pressed ${spec.key}`);
       return { summary: `pressed ${spec.key}` };
     });
   }
@@ -871,6 +951,7 @@ export class BrowserManager {
       const filePath = requested !== '' ? path.resolve(requested) : path.join(tmpdir(), 'dsh-visual-browser', `page-${timestamp}.${format}`);
       mkdirSync(path.dirname(filePath), { recursive: true });
       await writeFile(filePath, bytes);
+      await this.#capture().catch(() => null);
       this.#record('screenshot', path.basename(filePath), `saved ${format} screenshot`);
       return { path: filePath, bytes: bytes.length, format };
     });
@@ -937,6 +1018,7 @@ export class BrowserManager {
       starting: this.#starting !== null,
       executable: this.#executable,
       headless: this.#config.headless,
+      managed: this.#owned,
       url: this.#url,
       title: this.#title,
       viewport: { width: this.#config.viewportWidth, height: this.#config.viewportHeight },