@restregular/dsh-visual-browser 0.1.1 → 0.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +165 -129
- package/client.js +1266 -434
- package/lib/browser.js +1255 -1045
- package/lib/routes.js +101 -47
- package/locale/en.json +1 -1
- package/locale/zh.json +1 -1
- package/package.json +3 -4
- package/test/package.test.mjs +70 -56
- package/test/routes.test.mjs +86 -45
package/README.md
CHANGED
|
@@ -1,129 +1,165 @@
|
|
|
1
|
-
# dsh-visual-browser
|
|
2
|
-
|
|
3
|
-
A DeepSeek Harness plugin that gives the Agent a **real Chrome** and gives you a **window into it
|
|
4
|
-
|
|
5
|
-
- The Agent never reads the DOM. It reads a numbered list of *visible interactive
|
|
6
|
-
elements* plus a short text excerpt — a few hundred tokens per page instead of a
|
|
7
|
-
full snapshot.
|
|
8
|
-
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
1
|
+
# dsh-visual-browser
|
|
2
|
+
|
|
3
|
+
A DeepSeek Harness plugin that gives the Agent a **real Chrome** and gives you a **live window into it** — as a tab in the **right Sidebar**, right under the built-in *Browser* entry.
|
|
4
|
+
|
|
5
|
+
- The Agent never reads the DOM. It reads a numbered list of *visible interactive
|
|
6
|
+
elements* plus a short text excerpt — a few hundred tokens per page instead of a
|
|
7
|
+
full DOM snapshot.
|
|
8
|
+
- The tab streams Chrome's own screencast (CDP `Page.startScreencast`) as a
|
|
9
|
+
length-prefixed JPEG stream, decoded off the main thread onto a canvas. No
|
|
10
|
+
screenshot polling, no per-frame request, no re-render of the panel per frame.
|
|
11
|
+
- Your pointer, wheel and keyboard go straight to the DevTools input domain in
|
|
12
|
+
animation-frame batches: click, drag, scroll and type with normal latency
|
|
13
|
+
instead of "screenshot after the fact" latency.
|
|
14
|
+
- Every action — the Agent's and yours — lands in a collapsible, scrollable
|
|
15
|
+
**operation log** whose height you can drag. Records only: there is nothing to
|
|
16
|
+
replay.
|
|
17
|
+
- Cookies persist across restarts (one dedicated Chrome profile), so a login the
|
|
18
|
+
Agent performs once keeps working.
|
|
19
|
+
|
|
20
|
+
## Install
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
dsh plugin add @restregular/dsh-visual-browser
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Or from the Harness UI: **Plugins → Add → `@restregular/dsh-visual-browser`**.
|
|
27
|
+
|
|
28
|
+
Requirements: Google Chrome, Microsoft Edge, or Chromium on the machine
|
|
29
|
+
(auto-detected), and Node.js 22 or newer (the plugin ships no runtime
|
|
30
|
+
dependencies; it talks CDP over the built-in WebSocket).
|
|
31
|
+
|
|
32
|
+
## Use
|
|
33
|
+
|
|
34
|
+
1. Open the right Sidebar. Under the built-in **浏览器 / Browser** card sits the
|
|
35
|
+
Visual Browser row — click it, or click the **浏览器** card itself and then the
|
|
36
|
+
extra row beneath it. The tab opens and replaces the guide in that pane.
|
|
37
|
+
2. Ask the Agent something that needs a website, for example
|
|
38
|
+
*"打开 example.com 看看首页写了什么"*. The Agent calls `browser_open`.
|
|
39
|
+
3. Watch the picture. Click, drag, wheel and type straight into it; the toolbar
|
|
40
|
+
has back / forward / reload / stop, and the address bar opens any URL.
|
|
41
|
+
|
|
42
|
+
The operation log at the bottom remembers the Agent's actions and yours. Click
|
|
43
|
+
its header to fold it away, or scroll it when the list is longer than the pane.
|
|
44
|
+
Drag the handle above the log to choose how much room it takes (the picture keeps
|
|
45
|
+
a minimum reserve); double-click the handle to go back to the default, or focus
|
|
46
|
+
it and use ↑ / ↓ (Shift for bigger steps, Home / End for the extremes). The
|
|
47
|
+
chosen height is remembered per browser.
|
|
48
|
+
|
|
49
|
+
### Dark mode
|
|
50
|
+
|
|
51
|
+
The panel only uses `--dsw-alias-*` theme tokens for surfaces, text, borders,
|
|
52
|
+
hover and focus, so light and dark (and theme switches) need no extra rules. The
|
|
53
|
+
only literal colour in the bundle is the `#fff` fallback after
|
|
54
|
+
`--dsw-alias-label-primary-foreground`.
|
|
55
|
+
|
|
56
|
+
## Tools the Agent gets
|
|
57
|
+
|
|
58
|
+
| Tool | Purpose |
|
|
59
|
+
| --- | --- |
|
|
60
|
+
| `browser_open` | Open a URL (launches Chrome on first use) and return the numbered view. |
|
|
61
|
+
| `browser_see` | Fresh numbered view of the current page. |
|
|
62
|
+
| `browser_act` | `click` / `type` / `key` / `hover` / `scroll` / `select` / `back` / `forward` / `reload` / `wait` / `close`, then return the new view. |
|
|
63
|
+
| `browser_text` | Read text of the page or of one element (a link also returns its `href`). |
|
|
64
|
+
| `browser_screenshot` | Save a PNG/JPEG to disk and return the path (feed it to `read_image`). |
|
|
65
|
+
|
|
66
|
+
### What the Agent sees
|
|
67
|
+
|
|
68
|
+
```
|
|
69
|
+
URL: https://example.com/login
|
|
70
|
+
TITLE: 登录 - Example
|
|
71
|
+
VIEW: 1280x800 scroll 0/2400 · 4/6 interactive
|
|
72
|
+
[1] textbox "手机号" value=13800000000
|
|
73
|
+
[2] textbox "密码" value=••••••
|
|
74
|
+
[3] button "登录"
|
|
75
|
+
[4] link "忘记密码" → /forgot
|
|
76
|
+
HEADINGS: 用户登录 | 没有账号?
|
|
77
|
+
TEXT: 用户登录 手机号 密码 登录 忘记密码
|
|
78
|
+
ACTIVE: [1] textbox "手机号"
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Numbers are reassigned on every view, so an action always uses the newest
|
|
82
|
+
numbers; a stale one fails with a clear "take a fresh snapshot" message instead
|
|
83
|
+
of clicking the wrong element.
|
|
84
|
+
|
|
85
|
+
## Configuration
|
|
86
|
+
|
|
87
|
+
Row config (in the profile's `cordis.patch.yml`, override by row id
|
|
88
|
+
`dsh-visual-browser`):
|
|
89
|
+
|
|
90
|
+
```yaml
|
|
91
|
+
- id: dsh-visual-browser
|
|
92
|
+
config:
|
|
93
|
+
headless: true # false = visible Chrome window
|
|
94
|
+
executablePath: '' # explicit chrome.exe path
|
|
95
|
+
userDataDir: '' # default: ~/.dsh/visual-browser/chrome-profile
|
|
96
|
+
viewport:
|
|
97
|
+
width: 1280
|
|
98
|
+
height: 800
|
|
99
|
+
maxElements: 100 # cap on the numbered list
|
|
100
|
+
maxTextChars: 320 # cap on the text excerpt
|
|
101
|
+
streamQuality: 60 # JPEG quality of the live screencast
|
|
102
|
+
allowUserControl: true # allow clicks/typing from the panel
|
|
103
|
+
reuseExisting: true # adopt a Chrome that already owns the profile
|
|
104
|
+
actionTimeoutMs: 12000
|
|
105
|
+
navigationTimeoutMs: 30000
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
## How it works
|
|
109
|
+
|
|
110
|
+
- **Host** (`index.js`, `lib/`): launches Chrome with
|
|
111
|
+
`--remote-debugging-port=0 --user-data-dir=<profile>`, parses the DevTools
|
|
112
|
+
endpoint from stderr, attaches one flattened CDP session, and exposes:
|
|
113
|
+
- five tools through the `tools` service (registration is effect-scoped),
|
|
114
|
+
- a same-origin prefix route `/visual-browser/api`:
|
|
115
|
+
| Route | Purpose |
|
|
116
|
+
| --- | --- |
|
|
117
|
+
| `GET /state` | Status, viewport and the operation log (tiny JSON poll). |
|
|
118
|
+
| `GET /stream` | One long response of `[len][seq][jpeg]` frames. |
|
|
119
|
+
| `POST /input` | A batch of pointer / wheel / key / text events. |
|
|
120
|
+
| `POST /start`, `POST /stop`, `POST /nav` | Open a URL, close Chrome, back/forward/reload. |
|
|
121
|
+
- **Live picture** (`lib/browser.js`): `Page.startScreencast` pushes a JPEG per
|
|
122
|
+
painted frame; the manager acks each frame and fans the same buffer out to every
|
|
123
|
+
panel. The first subscriber restarts the screencast so a still page appears at
|
|
124
|
+
once; the last one stops it, so a closed panel costs nothing. No screenshot is
|
|
125
|
+
taken per click, and nothing about the picture touches the model context.
|
|
126
|
+
- **Input** (`lib/browser.js`): panel events become `Input.dispatchMouseEvent` /
|
|
127
|
+
`Input.dispatchKeyEvent` / `Input.insertText` commands. Nothing waits for a
|
|
128
|
+
snapshot or a capture, so input latency is one round trip; the client batches
|
|
129
|
+
per animation frame so a drag or flick is one request, not thirty.
|
|
130
|
+
- **Snapshot** (`lib/page-scripts.js`): one serialized function runs in the page,
|
|
131
|
+
filters interactive elements to the viewport, assigns `data-dsh-vb-ref`
|
|
132
|
+
numbers, and returns structured JSON. Nothing else about the DOM leaves the
|
|
133
|
+
page.
|
|
134
|
+
- **Survives a host restart**: Chrome writes `DevToolsActivePort` into its
|
|
135
|
+
profile, so a restarted host (or one whose previous host crashed with Chrome
|
|
136
|
+
still alive) attaches to that browser instead of failing on the profile lock.
|
|
137
|
+
An adopted browser is only detached — its process is not killed.
|
|
138
|
+
|
|
139
|
+
Chrome is killed with its process tree when the plugin unloads.
|
|
140
|
+
|
|
141
|
+
## Notes and limits
|
|
142
|
+
|
|
143
|
+
- Cross-origin iframes are counted but not inspected; open the direct URL when a
|
|
144
|
+
flow lives inside a frame.
|
|
145
|
+
- Blocking JS dialogs are auto-accepted and logged in the operation log.
|
|
146
|
+
- The panel route is served by the Harness web server on its existing origin;
|
|
147
|
+
it performs no extra authentication, matching the other local panel routes.
|
|
148
|
+
- The stream is a raw binary response on purpose: `multipart/x-mixed-replace`
|
|
149
|
+
parsing differs between browser builds, so the panel parses frames itself.
|
|
150
|
+
- Headless mode is the default. Some sites behave differently headless; set
|
|
151
|
+
`headless: false` when a site insists on a window.
|
|
152
|
+
|
|
153
|
+
## Development
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
npm test # syntax checks + snapshot/format/route/package smoke tests
|
|
157
|
+
```
|
|
158
|
+
|
|
159
|
+
`.dsh-probe/` (git-ignored) holds the manual probes used while developing:
|
|
160
|
+
`live-probe.mjs` drives the real browser end to end, `ui-probe.mjs` renders the
|
|
161
|
+
real client bundle in Chrome against the real host API and drives the panel with
|
|
162
|
+
real DOM events, and `build-css.mjs` copies the panel CSS and theme tokens for
|
|
163
|
+
that harness.
|
|
164
|
+
|
|
165
|
+
MIT © RestRegular
|