dsh-inner-browser-use 0.27.0 → 0.34.28
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/client.js +103 -8
- package/docs/ARTIFACTS.md +19 -0
- package/docs/BENCH.md +376 -0
- package/docs/ROADMAP.md +88 -0
- package/index.js +320 -29
- package/package.json +5 -2
- package/read.js +111 -10
- package/review.js +213 -0
- package/sidecar.js +717 -40
- package/tools.js +635 -83
package/docs/ROADMAP.md
CHANGED
|
@@ -232,3 +232,91 @@ A picture is only an answer if it loaded. Next: report a large picture element
|
|
|
232
232
|
whose source failed (`naturalWidth === 0`) as a broken picture rather than
|
|
233
233
|
staying silent, since that is the commonest way a generated image "disappears".
|
|
234
234
|
|
|
235
|
+
## Phase 19 — the same picture again, 0.28.0
|
|
236
|
+
|
|
237
|
+
Reads are stateless: the same page costs the same tokens twice. Next: remember
|
|
238
|
+
the last read of a page by address and say only what changed, with a hard reset
|
|
239
|
+
when the address or the document changes.
|
|
240
|
+
|
|
241
|
+
## Phase 20 — two browsers, one conversation, 0.29.0
|
|
242
|
+
|
|
243
|
+
The column is a real browser, so the model inside it can be a reviewer, not just
|
|
244
|
+
a tool: hand it the repository address, ask for a review, read the answer back
|
|
245
|
+
through the same mirror, and take the next round from it. Nothing opens on the
|
|
246
|
+
desktop while that happens: a site that refuses a headless build gets a window
|
|
247
|
+
at -3000,-3000, and the column shows the only picture.
|
|
248
|
+
|
|
249
|
+
## Phase 21 — a review round that closes the loop, 0.30.0
|
|
250
|
+
|
|
251
|
+
`sidecar_ask` sends and returns an answer. Next: a round trip that carries the
|
|
252
|
+
repository address there and the review back, and records in the round log which
|
|
253
|
+
commit the review was about, so a later round can tell whether the advice was
|
|
254
|
+
already taken.
|
|
255
|
+
|
|
256
|
+
## Phase 22 — the reviewer sees the code, not a description of it, 0.31.0
|
|
257
|
+
|
|
258
|
+
The round log now records what was asked and what came back. Next: paste the
|
|
259
|
+
diff the review is about into the same message (bounded, so the column does not
|
|
260
|
+
fill with code), and let a later round say which points were already acted on,
|
|
261
|
+
so the loop stops repeating advice.
|
|
262
|
+
|
|
263
|
+
## Phase 23 — the drawing half, 0.31.0
|
|
264
|
+
|
|
265
|
+
A page that never shipped an image still has a canvas or a video frame; take
|
|
266
|
+
that surface as the picture when the page has no file to grab.
|
|
267
|
+
|
|
268
|
+
## Phases 24–39, condensed
|
|
269
|
+
|
|
270
|
+
The phases above were written one at a time as the work happened; the run from
|
|
271
|
+
0.32 to 0.34.18 is dense enough that the per-phase prose lives in
|
|
272
|
+
`docs/BENCH.md` and the changelog. What each one changed, in one line:
|
|
273
|
+
|
|
274
|
+
| Version | Phase | What it moved |
|
|
275
|
+
| --- | --- | --- |
|
|
276
|
+
| 0.32.0 | 22 | the diff a review is about travels with the request |
|
|
277
|
+
| 0.33.0 | 22b | the mirror outlives the visit: claim, takeover, remembered address |
|
|
278
|
+
| 0.33.2 | 22c | a duplicate registration cannot take the pane down |
|
|
279
|
+
| 0.34.0 | 23 | one tab per chat, a UA that does not say "headless", typing that looks like hands |
|
|
280
|
+
| 0.34.1 | 24 | a ref is checked against what the control says before it is used |
|
|
281
|
+
| 0.34.2 | 25 | one chat cannot take another chat's page; a password never comes back in a result |
|
|
282
|
+
| 0.34.3 | 26 | a picture reaches the model as a picture; read state is per chat |
|
|
283
|
+
| 0.34.4 | 27 | an empty mirror says what to do; remembering an address is written, not crashed |
|
|
284
|
+
| 0.34.5 | 28 | one ownership gate for every acting route |
|
|
285
|
+
| 0.34.6 | 29 | one tab per chat even when two calls race |
|
|
286
|
+
| 0.34.7 | 30 | a live cookie is never overwritten by a stale copy |
|
|
287
|
+
| 0.34.8 | 31 | a cold first tab in 0.6 s instead of 21 s |
|
|
288
|
+
| 0.34.9 | 32 | a plugin reload no longer closes the browser |
|
|
289
|
+
| 0.34.10 | 33 | a call log never shows another chat's calls |
|
|
290
|
+
| 0.34.11 | 34 | the column offers "restore the last page" in one click |
|
|
291
|
+
| 0.34.12 | 35 | every write needs the chat that asked for it |
|
|
292
|
+
| 0.34.13 | 36 | a review round is verifiable: each point marked fixed / verified / invalid / open |
|
|
293
|
+
| 0.34.14 | 36b | a reload must not cost the browser: the watchdog follows the application pid |
|
|
294
|
+
| 0.34.15 | 37 | an observation admits its blind spot: pictures, canvases, videos, frames; an error page says so |
|
|
295
|
+
| 0.34.16 | 38 | a reload does not blank the mirror: the owning chat's poll adopts the running browser |
|
|
296
|
+
| 0.34.17 | 39 | the blind-spot frame count follows the screen, not the DOM |
|
|
297
|
+
| 0.34.18 | — | the adoption rule is a named function with a test |
|
|
298
|
+
|
|
299
|
+
## 页游这一档(2026-10-28 起)
|
|
300
|
+
|
|
301
|
+
输入这一层已经到位:按住键、组合键、按住鼠标,一项 300 ms 的按住整次调用 305 ms,事件由页面自己证实。实时画面沿用同一条 CDP 帧路(60 fps、镜像延迟中位数 8 ms)。
|
|
302
|
+
|
|
303
|
+
还差的一层是「看」:4399 / 7k7k / 奥拉星这类页游把画面画在 canvas 上,文本树里没有可读的东西,判断走位、Boss 血条、按钮位置都要靠像素。本会话的模型没有视觉能力,所以这里的正确做法不是假装能玩,而是把可判读的信号做出来:canvas 局部像素采样、颜色/区域变化率、画面是否在动(`blind` 行已经在报 canvas 数量)。有视觉能力的模型接着用同一套帧路就能玩。
|
|
304
|
+
|
|
305
|
+
## Where the next round should go
|
|
306
|
+
|
|
307
|
+
Ranked by what a real task still loses:
|
|
308
|
+
|
|
309
|
+
1. **End-to-end proof of the places unit tests cannot reach.** The restore-in-one-click
|
|
310
|
+
path (0.34.11) and the mirror coming back after a reload (0.34.16) were checked by
|
|
311
|
+
HTTP and by pid, not by a person clicking. A browser-level test that opens the
|
|
312
|
+
column, kills the driver, and clicks restore is the missing evidence.
|
|
313
|
+
2. **One-click "open here" when this chat has no seat.** `state.chats` can lack this
|
|
314
|
+
chat, and the column then only tells the model to call `sidecar_open`. The button
|
|
315
|
+
belongs in the column.
|
|
316
|
+
3. **Competitor behaviours still unmatched.** Playwright MCP's accessibility refs are
|
|
317
|
+
ours, and the blind line covers the vision-less read; Stagehand v4's
|
|
318
|
+
act/observe/extract split with cached selectors is only partly ours, and its
|
|
319
|
+
self-healing of a selector between rounds has no equivalent here.
|
|
320
|
+
4. **Token numbers for a real session, not a synthetic page.** Reads are measured on
|
|
321
|
+
fixed pages; an actual 20-step task has never had its cumulative token cost
|
|
322
|
+
recorded.
|
package/index.js
CHANGED
|
@@ -20,8 +20,36 @@
|
|
|
20
20
|
* learns no port and no secret. // ui-copy-ignore
|
|
21
21
|
*/
|
|
22
22
|
|
|
23
|
+
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'
|
|
24
|
+
import { homedir } from 'node:os'
|
|
25
|
+
import { join } from 'node:path'
|
|
23
26
|
import { Sidecar } from './sidecar.js?v=1'
|
|
24
27
|
|
|
28
|
+
/** Where the mirror's owner is remembered across a reload: the column should not go dark because the code reloaded. */
|
|
29
|
+
/** Where the mirror's note lives; a test points this elsewhere so it never reads a real chat. */
|
|
30
|
+
const STATE_DIR = process.env.DSH_INNER_BROWSER_STATE_DIR || join(homedir(), '.dsh', 'inner-browser-use')
|
|
31
|
+
const MIRROR_FILE = join(STATE_DIR, 'mirror.json')
|
|
32
|
+
|
|
33
|
+
/** @returns {{owner: string, session: string, at: number}} the last owner a chat asked to keep. */
|
|
34
|
+
function readMirror() {
|
|
35
|
+
try {
|
|
36
|
+
const saved = JSON.parse(readFileSync(MIRROR_FILE, 'utf8'))
|
|
37
|
+
return { owner: String(saved?.owner ?? ''), session: String(saved?.session ?? ''), at: Number(saved?.at ?? 0) }
|
|
38
|
+
} catch {
|
|
39
|
+
return { owner: '', session: '', at: 0 }
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
/** @param {{owner: string, session: string, at: number}} value - what to keep. */
|
|
44
|
+
function writeMirror(value) {
|
|
45
|
+
try {
|
|
46
|
+
mkdirSync(STATE_DIR, { recursive: true })
|
|
47
|
+
writeFileSync(MIRROR_FILE, JSON.stringify(value))
|
|
48
|
+
} catch {
|
|
49
|
+
/* an unwritable note must not take the mirror down */
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
|
|
25
53
|
/** Route prefix owned by this plugin. Never under `/api`, which is the Host's own channel. */ // ui-copy-ignore
|
|
26
54
|
export const ROUTE = '/dsh-inner-browser-use'
|
|
27
55
|
|
|
@@ -57,12 +85,25 @@ function readBody(req) {
|
|
|
57
85
|
* Nothing else lives here; the browser itself is the Host's. // ui-copy-ignore
|
|
58
86
|
*/
|
|
59
87
|
export class Bridge {
|
|
60
|
-
|
|
88
|
+
/** @param {{persist?: boolean}} [options] - persist the owner across reloads. */
|
|
89
|
+
constructor(options = {}) {
|
|
61
90
|
/** @type {Map<string, import('node:http').ServerResponse>} */
|
|
62
91
|
this.seats = new Map()
|
|
63
92
|
/* The one chat that owns the mirror. The column is global, so the picture
|
|
64
93
|
may only ever be drawn for this chat; every other chat shows a note. */
|
|
65
94
|
this.owner = ''
|
|
95
|
+
/* The tab asked for, kept until the chat comes back into view. A chat that
|
|
96
|
+
is not on screen has no stream, so a push at ask time is lost; remembering
|
|
97
|
+
the ask is what makes the mirror appear when the person returns. */
|
|
98
|
+
this.wanted = { session: '', at: 0 }
|
|
99
|
+
/* A running plugin keeps the note; a bare Bridge in a test does not. */
|
|
100
|
+
this.persist = options.persist === true
|
|
101
|
+
const kept = this.persist ? readMirror() : { owner: '', session: '', at: 0 }
|
|
102
|
+
/* Only a real chat id is restored: a stray note must not hand the mirror to
|
|
103
|
+
a chat that no longer exists. */
|
|
104
|
+
if (/^session-/.test(kept.owner) !== true) kept.owner = ''
|
|
105
|
+
this.owner = kept.owner
|
|
106
|
+
this.wanted = { session: kept.session, at: kept.at }
|
|
66
107
|
}
|
|
67
108
|
|
|
68
109
|
/**
|
|
@@ -70,7 +111,23 @@ export class Bridge {
|
|
|
70
111
|
* @param {string} sessionId - chat that asked for it.
|
|
71
112
|
*/
|
|
72
113
|
claim(sessionId) {
|
|
73
|
-
if (sessionId
|
|
114
|
+
if (sessionId === '') return
|
|
115
|
+
this.owner = sessionId
|
|
116
|
+
if (this.persist === true) writeMirror({ owner: this.owner, session: this.wanted.session, at: this.wanted.at })
|
|
117
|
+
}
|
|
118
|
+
|
|
119
|
+
/**
|
|
120
|
+
* Remember that a chat wants the mirror, and tell it now if it is listening.
|
|
121
|
+
* @param {string} sessionId - chat that asked.
|
|
122
|
+
* @param {number} [at] - when it asked; older asks are not replayed.
|
|
123
|
+
* @returns {boolean} whether a live tab took the frame.
|
|
124
|
+
*/
|
|
125
|
+
want(sessionId, at = Date.now()) {
|
|
126
|
+
if (sessionId === '') return false
|
|
127
|
+
this.claim(sessionId)
|
|
128
|
+
this.wanted = { session: sessionId, at }
|
|
129
|
+
if (this.persist === true) writeMirror({ owner: this.owner, session: sessionId, at })
|
|
130
|
+
return this.push(sessionId, { type: 'sidecar-wanted', session: sessionId, at })
|
|
74
131
|
}
|
|
75
132
|
|
|
76
133
|
/**
|
|
@@ -81,6 +138,13 @@ export class Bridge {
|
|
|
81
138
|
attach(sessionId, res) {
|
|
82
139
|
this.seats.get(sessionId)?.end()
|
|
83
140
|
this.seats.set(sessionId, res)
|
|
141
|
+
/* A tab that connects after the ask still gets the mirror: this is the case
|
|
142
|
+
where the person was looking at another chat, or reloaded the page. An
|
|
143
|
+
ask older than two minutes is spent and is not replayed. */
|
|
144
|
+
const ask = this.wanted
|
|
145
|
+
if (ask.session === sessionId && Date.now() - ask.at <= 120_000) {
|
|
146
|
+
this.push(sessionId, { type: 'sidecar-wanted', session: sessionId, at: ask.at })
|
|
147
|
+
}
|
|
84
148
|
}
|
|
85
149
|
|
|
86
150
|
/**
|
|
@@ -110,6 +174,17 @@ export class Bridge {
|
|
|
110
174
|
return [...this.seats.keys()]
|
|
111
175
|
}
|
|
112
176
|
|
|
177
|
+
/**
|
|
178
|
+
* Forget a mirror nobody is waiting for any more.
|
|
179
|
+
* @param {string} sessionId - chat that closed its tab.
|
|
180
|
+
* @returns {boolean} whether the ask was the remembered one.
|
|
181
|
+
*/
|
|
182
|
+
forget(sessionId) {
|
|
183
|
+
if (sessionId === '' || this.wanted.session !== sessionId) return false
|
|
184
|
+
this.wanted = { session: '', at: 0 }
|
|
185
|
+
return true
|
|
186
|
+
}
|
|
187
|
+
|
|
113
188
|
/** Close every stream. */
|
|
114
189
|
dispose() {
|
|
115
190
|
for (const res of this.seats.values()) res.end()
|
|
@@ -120,7 +195,62 @@ export class Bridge {
|
|
|
120
195
|
/** Page read: address, title, controls, then text, as JSON the Host parses. */ // ui-copy-ignore
|
|
121
196
|
/* The read model and the tool surface live in their own files: this module is
|
|
122
197
|
the Host half (routes, bridge, lifetimes), not the model's vocabulary. */ // ui-copy-ignore
|
|
123
|
-
import { sidecarTools } from './tools.js?v=1'
|
|
198
|
+
import { KEYS, sidecarTools, sessionScope, workScope } from './tools.js?v=1'
|
|
199
|
+
|
|
200
|
+
/**
|
|
201
|
+
* A state poll may pick up a browser that outlived a reload, and only for the
|
|
202
|
+
* chat that owns it: any chat polling would otherwise start a browser nobody
|
|
203
|
+
* asked for, and a chat with no column would look like it had one.
|
|
204
|
+
* @param {{alive?: boolean}} said - what the browser half answered.
|
|
205
|
+
* @param {string} chat - the chat asking.
|
|
206
|
+
* @param {string} owner - the chat that owns the mirror.
|
|
207
|
+
* @returns {boolean} whether this poll should attach to a running browser.
|
|
208
|
+
*/
|
|
209
|
+
/**
|
|
210
|
+
* What the state poll hands the browser half when it picks a surviving browser
|
|
211
|
+
* up: the chat's own tab, at the address that chat last had. Without the chat
|
|
212
|
+
* the browser comes back with no tab for anybody, and the column stays empty.
|
|
213
|
+
* @param {string} chat - the chat that is polling.
|
|
214
|
+
* @param {string} remembered - where that chat left off, or ''.
|
|
215
|
+
* @returns {{session: string, url?: string}} options for `ensure`.
|
|
216
|
+
*/
|
|
217
|
+
/**
|
|
218
|
+
* Whether this chat's poll should put its tab back: the browser is running, the
|
|
219
|
+
* chat holds the mirror, and its tab is not there.
|
|
220
|
+
* @param {any} chatState - what `chatState(chat)` said.
|
|
221
|
+
* @param {string} chat - the chat that is polling.
|
|
222
|
+
* @param {string} owner - the chat that holds the mirror.
|
|
223
|
+
* @returns {boolean} true when a tab has to be opened for this chat.
|
|
224
|
+
*/
|
|
225
|
+
/**
|
|
226
|
+
* The folder the calling session works in, wherever it is kept.
|
|
227
|
+
* @param {any} execution - tool execution context.
|
|
228
|
+
* @returns {string} absolute path, or empty when nothing was said.
|
|
229
|
+
*/
|
|
230
|
+
export function cwdOf(execution) {
|
|
231
|
+
try {
|
|
232
|
+
const agent = execution?.agent
|
|
233
|
+
const value = agent?.cwd ?? agent?.session?.header?.cwd ?? agent?.header?.cwd
|
|
234
|
+
if (typeof value === 'string' && value !== '') return value
|
|
235
|
+
} catch {
|
|
236
|
+
/* a context that cannot be read is a context without a folder */
|
|
237
|
+
}
|
|
238
|
+
return ''
|
|
239
|
+
}
|
|
240
|
+
|
|
241
|
+
export function needsTab(chatState, chat, owner) {
|
|
242
|
+
if (String(chat ?? '') === '' || owner !== chat) return false
|
|
243
|
+
return chatState?.alive !== true
|
|
244
|
+
}
|
|
245
|
+
|
|
246
|
+
export function adoptOptions(chat, remembered) {
|
|
247
|
+
const back = String(remembered ?? '')
|
|
248
|
+
return back === '' ? { session: chat } : { session: chat, url: back }
|
|
249
|
+
}
|
|
250
|
+
|
|
251
|
+
export function shouldAdoptOnState(said, chat, owner) {
|
|
252
|
+
return said?.alive === false && chat !== '' && owner === chat
|
|
253
|
+
}
|
|
124
254
|
|
|
125
255
|
export const name = 'dsh-inner-browser-use'
|
|
126
256
|
export const inject = ['webServer']
|
|
@@ -130,7 +260,7 @@ export const inject = ['webServer']
|
|
|
130
260
|
* @param {any} ctx - plugin context.
|
|
131
261
|
*/
|
|
132
262
|
export function apply(ctx) {
|
|
133
|
-
const bridge = new Bridge()
|
|
263
|
+
const bridge = new Bridge({ persist: true })
|
|
134
264
|
/* The sidecar is a browser the Host owns outright: a real Chrome with its own // ui-copy-ignore
|
|
135
265
|
profile, shown in the column as a screencast. It outlives a plugin reload,
|
|
136
266
|
and it does not need the chat to be on screen to do work. */
|
|
@@ -148,15 +278,21 @@ export function apply(ctx) {
|
|
|
148
278
|
timer.unref?.()
|
|
149
279
|
return () => {
|
|
150
280
|
clearInterval(timer)
|
|
151
|
-
void sidecar.
|
|
281
|
+
void sidecar.handoff().catch(() => {})
|
|
152
282
|
}
|
|
153
283
|
}, 'dsh-inner-browser-use: sidecar lifetime')
|
|
154
284
|
|
|
155
285
|
/** @param {import('node:http').IncomingMessage} req - request. @param {import('node:http').ServerResponse} res - response. */
|
|
286
|
+
/** @type {{at: number, path: string, session: string}[]} the last few calls the column made. */
|
|
287
|
+
const recent = []
|
|
156
288
|
const handler = async (req, res) => {
|
|
157
289
|
const url = new URL(req.url ?? ROUTE, 'http://dsh.invalid')
|
|
158
290
|
const path = url.pathname.slice(ROUTE.length) || '/'
|
|
159
291
|
const sessionId = url.searchParams.get('session') ?? ''
|
|
292
|
+
/* Who came in and when: an empty picture has two very different causes, and
|
|
293
|
+
this is what tells a column that never connected from a page that is gone. */
|
|
294
|
+
recent.push({ at: Date.now(), path, session: sessionId.slice(-8) })
|
|
295
|
+
if (recent.length > 40) recent.splice(0, recent.length - 40)
|
|
160
296
|
|
|
161
297
|
/** @param {number} code - status. @param {unknown} value - body. */
|
|
162
298
|
const json = (code, value) => {
|
|
@@ -164,14 +300,40 @@ export function apply(ctx) {
|
|
|
164
300
|
res.end(JSON.stringify(value))
|
|
165
301
|
}
|
|
166
302
|
|
|
303
|
+
/* One answer to one question: may this chat act on the mirror? A browser
|
|
304
|
+
nobody holds is anybody's to take; once a chat holds it, another chat is
|
|
305
|
+
refused here instead of being trusted to check itself. */
|
|
306
|
+
/** @param {string} session - the chat a request speaks for. */
|
|
307
|
+
const held = (session) => session !== '' && bridge.owner !== '' && session !== bridge.owner
|
|
308
|
+
/** @returns {boolean} true when the request was refused. */
|
|
309
|
+
const refuse = () => {
|
|
310
|
+
json(403, { ok: false, text: 'this chat does not own the browser', owner: bridge.owner }) // ui-copy-ignore
|
|
311
|
+
return true
|
|
312
|
+
}
|
|
313
|
+
/* Every route that changes the page goes through one check: it must say
|
|
314
|
+
which chat it is for, and that chat must be the one holding the mirror.
|
|
315
|
+
Two rules in one place, so a new route cannot quietly skip one. */
|
|
316
|
+
/**
|
|
317
|
+
* @param {any} body - parsed request body, if any.
|
|
318
|
+
* @returns {string} the chat that may act, or '' when it was refused.
|
|
319
|
+
*/
|
|
320
|
+
const acting = (body) => {
|
|
321
|
+
const who = String(body?.session ?? '') || sessionId
|
|
322
|
+
if (who === '') {
|
|
323
|
+
json(400, { ok: false, text: 'which chat asked for it' }) // ui-copy-ignore
|
|
324
|
+
return ''
|
|
325
|
+
}
|
|
326
|
+
if (held(who) && refuse()) return ''
|
|
327
|
+
return who
|
|
328
|
+
}
|
|
329
|
+
|
|
167
330
|
if (path === '/sidecar/frames' && req.method === 'GET') {
|
|
168
331
|
/* The column is global and only one chat may hold the mirror: a chat that
|
|
169
332
|
does not own it is refused here, so a stale tab cannot stream a picture
|
|
170
333
|
of somebody else's browser. */
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
}
|
|
334
|
+
const asked = String(url.searchParams.get('session') ?? '')
|
|
335
|
+
const watching = asked !== '' ? asked : sessionId
|
|
336
|
+
if (held(watching) && refuse()) return
|
|
175
337
|
res.writeHead(200, {
|
|
176
338
|
'content-type': 'text/event-stream; charset=utf-8',
|
|
177
339
|
'cache-control': 'no-cache, no-transform',
|
|
@@ -191,9 +353,13 @@ export function apply(ctx) {
|
|
|
191
353
|
page wait: the newest picture is the only one worth sending. */
|
|
192
354
|
if (res.writableLength > 3 * 1024 * 1024) return
|
|
193
355
|
res.write(`data: ${JSON.stringify(frame)}\n\n`)
|
|
194
|
-
})
|
|
356
|
+
}, watching)
|
|
195
357
|
const info = await sidecar.state()
|
|
196
|
-
|
|
358
|
+
/* The pane needs one address to offer back when this chat's own page
|
|
359
|
+
went away; the state route answers the same question for the tab
|
|
360
|
+
header, so both carry it. */
|
|
361
|
+
const chat = watching === '' ? {} : { chat: await sidecar.chatState(watching) }
|
|
362
|
+
res.write(`data: ${JSON.stringify({ type: 'hello', ...info, ...chat })}\n\n`)
|
|
197
363
|
} catch (error) {
|
|
198
364
|
res.write(`data: ${JSON.stringify({ type: 'failed', text: String(error?.message ?? error) })}\n\n`)
|
|
199
365
|
}
|
|
@@ -203,8 +369,64 @@ export function apply(ctx) {
|
|
|
203
369
|
return
|
|
204
370
|
}
|
|
205
371
|
|
|
372
|
+
if (path === '/sidecar/claim' && req.method === 'POST') {
|
|
373
|
+
/* The pane itself asks back what it was showing: after a reload the
|
|
374
|
+
forgot who owned the mirror, and the chat that still has the tab open is
|
|
375
|
+
the only one allowed to say so. A different owner is refused. */
|
|
376
|
+
let body
|
|
377
|
+
try {
|
|
378
|
+
body = JSON.parse(await readBody(req))
|
|
379
|
+
} catch {
|
|
380
|
+
json(400, { ok: false, text: 'bad frame' })
|
|
381
|
+
return
|
|
382
|
+
}
|
|
383
|
+
const id = String(body.session ?? '')
|
|
384
|
+
if (id === '') {
|
|
385
|
+
json(400, { ok: false, text: 'no chat' })
|
|
386
|
+
return
|
|
387
|
+
}
|
|
388
|
+
if (bridge.owner !== '' && bridge.owner !== id) {
|
|
389
|
+
json(409, { ok: false, owner: bridge.owner })
|
|
390
|
+
return
|
|
391
|
+
}
|
|
392
|
+
bridge.claim(id)
|
|
393
|
+
json(200, { ok: true, owner: bridge.owner })
|
|
394
|
+
return
|
|
395
|
+
}
|
|
396
|
+
|
|
206
397
|
if (path === '/sidecar/state' && req.method === 'GET') {
|
|
207
|
-
|
|
398
|
+
/* Which chats have a column open right now: 0 means nobody is watching,
|
|
399
|
+
so an empty picture is the column's own problem, not the browser's. */
|
|
400
|
+
const chat = String(url.searchParams.get('session') ?? '')
|
|
401
|
+
/* Diagnostics belong to the chat that made them: a caller sees its own
|
|
402
|
+
calls, and one chat's browsing never explains another's. */
|
|
403
|
+
const mine = chat === '' ? recent : recent.filter((entry) => entry.session === chat.slice(-8))
|
|
404
|
+
let said = await sidecar.state()
|
|
405
|
+
/* The browser can be running and still have no tab for this chat: a reload
|
|
406
|
+
replaces the half that remembered which tab belonged to whom. The poll
|
|
407
|
+
is this chat asking, so it answers by putting that chat's tab back
|
|
408
|
+
before the column draws an empty picture — waiting for a claim first
|
|
409
|
+
left a blank frame in front of the person. */
|
|
410
|
+
let mineChat = chat === '' ? undefined : await sidecar.chatState(chat)
|
|
411
|
+
if (needsTab(mineChat, chat, bridge.owner)) {
|
|
412
|
+
try {
|
|
413
|
+
await sidecar.ensure(adoptOptions(chat, sidecar.rememberedTab?.(chat)))
|
|
414
|
+
said = await sidecar.state()
|
|
415
|
+
mineChat = await sidecar.chatState(chat)
|
|
416
|
+
} catch {}
|
|
417
|
+
}
|
|
418
|
+
/* A plugin reload leaves the browser running and this half brand new, so
|
|
419
|
+
the column's first poll would answer "nothing open" until somebody
|
|
420
|
+
called open again: the mirror would come back empty in a chat that
|
|
421
|
+
never closed it. The chat that owned the browser asks for its state, so
|
|
422
|
+
answering honestly means picking the browser up first. */
|
|
423
|
+
if (shouldAdoptOnState(said, chat, bridge.owner)) {
|
|
424
|
+
try {
|
|
425
|
+
await sidecar.ensure(adoptOptions(chat, sidecar.rememberedTab?.(chat)))
|
|
426
|
+
said = await sidecar.state()
|
|
427
|
+
} catch {}
|
|
428
|
+
}
|
|
429
|
+
json(200, { ...said, ...(chat === '' ? {} : { chat: mineChat ?? (await sidecar.chatState(chat)) }), owner: bridge.owner, chats: [...bridge.seats.keys()], recent: mine })
|
|
208
430
|
return
|
|
209
431
|
}
|
|
210
432
|
|
|
@@ -219,19 +441,21 @@ export function apply(ctx) {
|
|
|
219
441
|
const address = String(body.url ?? '')
|
|
220
442
|
/* The ask must name the chat it belongs to: a request that cannot say
|
|
221
443
|
who it is for never gets the mirror. */
|
|
222
|
-
|
|
444
|
+
/* An open names its chat and then takes the mirror over: that is what
|
|
445
|
+
asking for it means, so another owner is not a refusal here. */
|
|
446
|
+
const who = String(body.session ?? '') || sessionId
|
|
223
447
|
if (who === '') {
|
|
224
|
-
json(400, { ok: false, text: 'which chat asked for it' })
|
|
448
|
+
json(400, { ok: false, text: 'which chat asked for it' }) // ui-copy-ignore
|
|
225
449
|
return
|
|
226
450
|
}
|
|
227
451
|
await sidecar.ensure({
|
|
228
452
|
url: address === '' ? undefined : address,
|
|
229
453
|
headless: body.headless === undefined ? undefined : body.headless === true,
|
|
454
|
+
session: who,
|
|
230
455
|
})
|
|
231
456
|
/* The mirror belongs to the chat that asked for it; any earlier owner
|
|
232
457
|
loses it and its tab. */
|
|
233
|
-
bridge.
|
|
234
|
-
bridge.push(who, { type: 'sidecar-wanted', session: who })
|
|
458
|
+
bridge.want(who)
|
|
235
459
|
json(200, { ok: true, ...(await sidecar.state()), owner: bridge.owner })
|
|
236
460
|
} catch (error) {
|
|
237
461
|
json(500, { ok: false, text: String(error?.message ?? error) })
|
|
@@ -249,12 +473,10 @@ export function apply(ctx) {
|
|
|
249
473
|
}
|
|
250
474
|
/* Input follows the owner: a chat that lost the mirror cannot act on the
|
|
251
475
|
page from a stale tab either. */
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
return
|
|
255
|
-
}
|
|
476
|
+
const actor = acting(body)
|
|
477
|
+
if (actor === '') return
|
|
256
478
|
try {
|
|
257
|
-
const page = await sidecar.ensure()
|
|
479
|
+
const page = await sidecar.ensure({ session: actor })
|
|
258
480
|
const kind = String(body.kind ?? '')
|
|
259
481
|
const x = Number(body.x ?? 0)
|
|
260
482
|
const y = Number(body.y ?? 0)
|
|
@@ -304,12 +526,10 @@ export function apply(ctx) {
|
|
|
304
526
|
return
|
|
305
527
|
}
|
|
306
528
|
/* Navigation follows the owner too: a lost tab cannot steer the page. */
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
return
|
|
310
|
-
}
|
|
529
|
+
const actor = acting(body)
|
|
530
|
+
if (actor === '') return
|
|
311
531
|
try {
|
|
312
|
-
const page = await sidecar.ensure()
|
|
532
|
+
const page = await sidecar.ensure({ session: actor })
|
|
313
533
|
const op = String(body.op ?? 'goto')
|
|
314
534
|
if (op === 'reload') json(200, { ok: true, ...(await page.reload()) })
|
|
315
535
|
else if (op === 'back') json(200, { ok: true, ...(await page.history(-1)) })
|
|
@@ -325,7 +545,22 @@ export function apply(ctx) {
|
|
|
325
545
|
return
|
|
326
546
|
}
|
|
327
547
|
|
|
548
|
+
if (path === '/sidecar/forget' && req.method === 'POST') {
|
|
549
|
+
/* Closing the tab is an answer: coming back should not reopen it. Only the
|
|
550
|
+
chat that holds the mirror may withdraw its own ask. */
|
|
551
|
+
const body = await readBody(req).then((raw) => JSON.parse(raw || '{}')).catch(() => ({}))
|
|
552
|
+
const who = acting(body)
|
|
553
|
+
if (who === '') return
|
|
554
|
+
json(200, { ok: true, forgot: bridge.forget(who) })
|
|
555
|
+
return
|
|
556
|
+
}
|
|
557
|
+
|
|
328
558
|
if (path === '/sidecar/stop' && req.method === 'POST') {
|
|
559
|
+
/* Closing the browser is a write as well: a chat that does not hold the
|
|
560
|
+
mirror cannot close another chat's page. */
|
|
561
|
+
const body = await readBody(req).then((raw) => JSON.parse(raw || '{}')).catch(() => ({}))
|
|
562
|
+
const who = String(body.session ?? '') || sessionId
|
|
563
|
+
if (who !== '' && held(who) && refuse()) return
|
|
329
564
|
await sidecar.shutdown()
|
|
330
565
|
json(200, { ok: true })
|
|
331
566
|
return
|
|
@@ -359,13 +594,69 @@ export function apply(ctx) {
|
|
|
359
594
|
|
|
360
595
|
ctx.inject(['tools'], (scope) => {
|
|
361
596
|
const disposers = sidecarTools(sidecar, sidecarRefs, {
|
|
597
|
+
/* A screenshot the model can look at is the difference between planning
|
|
598
|
+
from the picture and guessing from a path. The store and the route that
|
|
599
|
+
can carry it are Harness services, so the tool asks here for it; when
|
|
600
|
+
either is missing the picture stays a file and the call still works. */
|
|
601
|
+
/**
|
|
602
|
+
* @param {string} data - base64 PNG.
|
|
603
|
+
* @param {string} name - file name for the attachment.
|
|
604
|
+
* @param {any} exec - the tool call's context.
|
|
605
|
+
* @returns {Promise<any>} the attachment value, or undefined to stay text-only.
|
|
606
|
+
*/
|
|
607
|
+
async showShot(data, name, exec) {
|
|
608
|
+
try {
|
|
609
|
+
const attachments = ctx.get('attachments')
|
|
610
|
+
const llm = ctx.get('llm')
|
|
611
|
+
if (attachments === undefined || llm === undefined) return undefined
|
|
612
|
+
const route = exec?.agent?.session?.requestHeader?.()?.config
|
|
613
|
+
const provider = route?.provider ?? exec?.agent?.options?.provider
|
|
614
|
+
const model = route?.model ?? exec?.agent?.options?.model
|
|
615
|
+
if (provider === undefined || model === undefined) return undefined
|
|
616
|
+
const info = await llm.resolveModelInfo(provider, model, exec?.signal)
|
|
617
|
+
if (!Array.isArray(info?.inputModalities) || !info.inputModalities.includes('image')) return undefined
|
|
618
|
+
const bytes = Buffer.from(data, 'base64')
|
|
619
|
+
const cap = Math.min(attachments.imageLimits.maxImageBytes, attachments.imageLimits.maxMessageImageBytes)
|
|
620
|
+
if (!(bytes.length > 0) || bytes.length > cap) return undefined
|
|
621
|
+
const ref = await attachments.saveImage({ data: bytes, mediaType: 'image/png', name })
|
|
622
|
+
return {
|
|
623
|
+
attachmentId: ref.attachmentId, // ui-copy-ignore
|
|
624
|
+
mediaType: ref.mediaType, // ui-copy-ignore
|
|
625
|
+
bytes: ref.bytes, // ui-copy-ignore
|
|
626
|
+
width: ref.width, // ui-copy-ignore
|
|
627
|
+
height: ref.height, // ui-copy-ignore
|
|
628
|
+
...(ref.name === undefined ? {} : { name: ref.name }),
|
|
629
|
+
}
|
|
630
|
+
} catch {
|
|
631
|
+
return undefined
|
|
632
|
+
}
|
|
633
|
+
},
|
|
362
634
|
/* The mirror changes hands here: the calling chat owns it from now on,
|
|
363
635
|
and every earlier owner loses it. */
|
|
364
636
|
want: (sessionId) => {
|
|
365
|
-
bridge.
|
|
366
|
-
bridge.push(sessionId, { type: 'sidecar-wanted', session: sessionId })
|
|
637
|
+
bridge.want(sessionId)
|
|
367
638
|
},
|
|
368
|
-
|
|
639
|
+
/* Who holds the mirror, and whether a chat is listening right now. */
|
|
640
|
+
owner: () => bridge.owner, // ui-copy-ignore
|
|
641
|
+
seats: () => [...bridge.seats.keys()].map((k) => String(k).slice(-8)).join(',') || 'none', // ui-copy-ignore
|
|
642
|
+
}).map((definition) => {
|
|
643
|
+
/* A reload can mount this surface twice before the old one is torn down.
|
|
644
|
+
A name already in the registry is kept, not fought over: a duplicate
|
|
645
|
+
registration is a warning, never a crash. */
|
|
646
|
+
try {
|
|
647
|
+
const run = definition.execute
|
|
648
|
+
const scoped = {
|
|
649
|
+
...definition,
|
|
650
|
+
/* Every tool call runs inside the chat that made it, so the browser
|
|
651
|
+
hands that chat its own tab and no other. */
|
|
652
|
+
execute: async (args, exec) => sessionScope.run(String(exec?.agent?.id ?? ''), () => workScope.run(cwdOf(exec), () => run(args, exec))),
|
|
653
|
+
}
|
|
654
|
+
return scope.tools.register(scoped)
|
|
655
|
+
} catch (error) {
|
|
656
|
+
ctx.logger?.warn?.(`browser tool ${definition.name} already registered: ${String(error?.message ?? error)}`)
|
|
657
|
+
return () => {}
|
|
658
|
+
}
|
|
659
|
+
})
|
|
369
660
|
scope.effect(() => () => {
|
|
370
661
|
for (const dispose of disposers) dispose()
|
|
371
662
|
}, 'dsh-inner-browser-use: tool surface')
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-inner-browser-use",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.34.28",
|
|
4
4
|
"description": "An agent-owned Chromium sidecar mirrored into the DeepSeek Harness right column: the agent drives it, the person watches and can take the mouse, and logins survive restarts.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "index.js",
|
|
@@ -16,6 +16,7 @@
|
|
|
16
16
|
"sidecar.js",
|
|
17
17
|
"tools.js",
|
|
18
18
|
"read.js",
|
|
19
|
+
"review.js",
|
|
19
20
|
"client.js",
|
|
20
21
|
"locale/*.json",
|
|
21
22
|
"cordis.patch.yml",
|
|
@@ -45,7 +46,9 @@
|
|
|
45
46
|
"test": "node --test test/*.test.mjs",
|
|
46
47
|
"doctor": "node scripts/doctor.mjs",
|
|
47
48
|
"breadth": "node scripts/breadth.mjs",
|
|
48
|
-
"live": "node scripts/live.mjs"
|
|
49
|
+
"live": "node scripts/live.mjs",
|
|
50
|
+
"adopt": "node scripts/check-adopt.mjs",
|
|
51
|
+
"game": "node scripts/check-game.mjs"
|
|
49
52
|
},
|
|
50
53
|
"license": "MIT",
|
|
51
54
|
"author": {
|