deepseek-computer-use 0.0.0-stage → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 STARDUSTLC666
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,303 @@
1
- # Temporary Holding Version
1
+ # deepseek-computer-use
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ > 社区插件,**非 DeepSeek 官方出品**。官方包在 `@deepseek-ai/*` scope 下。
4
+
5
+ 给 DeepSeek Harness 的 **Windows 桌面控制(Computer Use)** 插件。
6
+
7
+ 工具面对齐 Codex 的 window2 API(`list_apps` / `list_windows` / `get_window` /
8
+ `launch_app` / `get_window_state` / `click` / `press_key` / `type_text` / `scroll` /
9
+ `set_value` / `drag` / `perform_secondary_action` / `activate_window` /
10
+ `start_audio_recording` / `stop_audio_recording` / `computer_use_health`),底层是
11
+ Windows 原生三件套:**UI Automation 观测 + SendInput 注入 + Windows.Graphics.Capture 抓屏**。
12
+
13
+ 零 npm 原生依赖、零 .NET 运行时依赖:helper 和覆盖层用系统自带的 `csc.exe` 现编,
14
+ 抓屏(WGC)和录音(WASAPI)是 net48 程序,Windows 10/11 自带运行时。
15
+
16
+ ## 安装
17
+
18
+ 要求:**Windows 10/11**、**Node ≥ 18.17**。其余都是系统组件(.NET Framework 4.8、
19
+ UI Automation、WGC)。
20
+
21
+ ```bash
22
+ # 发布到 npm 之后
23
+ npm install --save deepseek-computer-use
24
+
25
+ # 现在还只能用 GitHub 源
26
+ npm install --save github:STARDUSTLC666/deepseek-computer-use
27
+ ```
28
+
29
+ 然后在本机 profile 的 `cordis.patch.yml` 里启用(详见本文件末尾的配置示例):
30
+
31
+ ```yaml
32
+ - insert:
33
+ - id: deepseek-computer-use
34
+ name: 'deepseek-computer-use'
35
+ ```
36
+
37
+ 首次调用任意 Computer Use 工具时会自动编译 helper 与覆盖层(约 1 秒)。
38
+ 想要抓屏/录音这两项可选能力,再跑一次:
39
+
40
+ ```bash
41
+ npm run build # helper + overlay(csc),再尝试 dotnet 编 capture/audio
42
+ ```
43
+
44
+ 没有 .NET SDK 也能用:抓屏自动退回 PrintWindow,录音会明确报 `audio_missing`。
45
+
46
+ ## 自检
47
+
48
+ ```bash
49
+ npm test # 13 项断言:注册、观测、点击、审批、硬拒绝、Esc 阻断
50
+ npm run test:all # 追加 helper 协议/音频 与插件级一致性测试
51
+ ```
52
+
53
+ ## 它解决什么问题
54
+
55
+ DSH 本体没有鼠标键盘工具,只能跑命令、读写文件。需要"操作图形界面"时(点按钮、
56
+ 填表单、操作没有 API 的桌面软件),这个插件补上这一层,并且把每一步都变成**可观测、
57
+ 可审批、可验证**的动作。
58
+
59
+ ## 架构
60
+
61
+ ```text
62
+ DSH 宿主
63
+ └─ lib/index.js apply(ctx) → 注册 16 个工具;图片走 attachments;审批走 approval
64
+ ├─ lib/tools.js 工具定义(Codex window2 语义)
65
+ ├─ lib/safety.js 白名单/黑名单 + 危险动作审批门
66
+ └─ lib/bridge.js 常驻进程 + 串行 JSON-lines RPC + 超时/取消
67
+ └─ native/bin/dsh-computer-use-helper.exe (用系统 csc.exe 现编)
68
+ ├─ native/Driver.cs IPlatformDriver 契约 + WindowsDriver(UIA / SendInput / PrintWindow)
69
+ ├─ native/Json.cs 零依赖 JSON 解析与序列化
70
+ └─ native/Program.cs stdio 分发
71
+ ```
72
+
73
+ 驱动层是接口(`IPlatformDriver`),今天是 `windows` 一个实现;macOS(AX + CGEvent)
74
+ 和 Linux(AT-SPI)可以照着补,插件层不用改。
75
+
76
+ helper 常驻的原因:元素编号必须和"那一次观测"绑在同一个进程里,否则
77
+ `click(element_index)` 无从解析。因此插件串行发请求、按 id 配对响应。
78
+
79
+ ## 工具
80
+
81
+ | 工具 | 作用 |
82
+ |---|---|
83
+ | `list_apps` | 列出当前有可操作窗口的应用及其窗口 |
84
+ | `list_windows` | 列出可见窗口:id / 标题 / 所属应用 / z 序 / 边框 |
85
+ | `get_window` | 按 id 重新读取窗口(换绑、刷新边框) |
86
+ | `launch_app` | 启动应用(可带参数),默认需要审批 |
87
+ | `get_window_state` | 观测:带编号的无障碍树 +(可选)窗口截图 |
88
+ | `click` | 按元素编号(优先走 Invoke,抗遮挡)或窗口相对坐标点击 |
89
+ | `press_key` | 单键或组合键,X keysym 风格:`Control_L+s`、`Return` |
90
+ | `type_text` | 注入 Unicode 文本(与键盘布局无关,中文安全) |
91
+ | `scroll` | 在坐标或元素处滚动 |
92
+ | `drag` | 窗口内拖拽 |
93
+ | `set_value` | 用 Value 模式直接改值,比"点击+全选+输入"更稳更省 |
94
+ | `perform_secondary_action` | Invoke / Expand / Collapse / Toggle / Select / Scroll Up… |
95
+ | `activate_window` | 把窗口提到前台 |
96
+ | `start_audio_recording` | 开始录制系统音频(WASAPI 环回),需 `audio: true` |
97
+ | `stop_audio_recording` | 停止录音并返回 WAV 路径与字节数 |
98
+ | `computer_use_health` | 自检:helper、编译器、策略、窗口数 |
99
+
100
+ ## 使用中指示器(对齐 Codex 的两个 overlay)
101
+
102
+ Codex 用两层覆盖层告诉用户"AI 正在动你的电脑",这里做的是同一套:
103
+
104
+ **Display Overlay — 呼吸细边**
105
+ - 一圈 **1.4px 的强调色细线** + 3 层递减光环,圆角 18px;只占屏幕最外圈,不压内容
106
+ - 亮度按约 3.2 秒周期呼吸(内线 alpha 170→240、光环 26→46)
107
+ - 4 条边 + 4 个圆角 = 8 个细长分层窗口,每帧位图几 KB
108
+ - 底部居中一枚自动宽度玻璃胶囊:脉冲圆点 + `DSH 正在使用您的电脑` +
109
+ `Esc` 键帽 + `取消`,停在任务栏上方留白
110
+
111
+ **Cursor Overlay — 虚拟光标 + 系统光标抑制**
112
+ - 覆盖层存在期间常驻一个带描边和投影的白色指针,启动时落在真实指针位置,
113
+ 之后 0.35 缓动滑向每个动作点;背后一层极淡的强调色光晕
114
+ - 默认抑制真实系统光标,退出/Esc/超时用 `SPI_SETCURSORS` 恢复;
115
+ helper 强杀覆盖层时也会兜底恢复一次
116
+
117
+ **DPI 关键点**:覆盖层进程必须 Per-Monitor-V2 感知,否则在 150% 缩放屏上
118
+ 系统会把 1707×1067 的逻辑画面放大到 2560×1600,描边变糊、位置偏移 1.5 倍。
119
+
120
+ **Esc 是按 turn 生效的**:首次触发报 `cancelled_by_user`,同一 turn 内后续调用直接
121
+ `turn_stopped`,新 turn 才恢复。
122
+
123
+ ## 安全模型
124
+
125
+ 1. **可达性**:`allowApps` / `denyApps` 决定这个插件能不能碰某个应用;空数组 = 不限制。
126
+ 2. **确认**:`approval` 三档
127
+ - `never`:从不询问(自担风险)
128
+ - `sensitive`(默认):目标元素标签命中危险词(delete/删除/卸载/支付/提交/安装…)才问;
129
+ `launch_app` 总是问
130
+ - `always`:每次动作都问
131
+ 审批走 DSH 自己的 approval seam;**没有审批服务时直接拒绝**,绝不静默降级。
132
+ 3. **前台证明**:打字、按键、坐标点击、滚动必须先证明目标窗口真的拿到了前台,
133
+ 否则抛 `focus_failed` 并告诉你当前是谁占着前台。SendInput 会打到有焦点的窗口,
134
+ 这一步省不得。
135
+ 4. **快照 TTL**:元素编号默认 120 秒过期,过期必须重新观测,避免"拿旧编号点新界面"。
136
+ 5. **`dryRun`**:只回放动作不注入,用于集成演练。
137
+
138
+ ## 安装
139
+
140
+ ```powershell
141
+ # 1. 放进 DSH 的 profile
142
+ cd "E:\path\to\dsh-computer-use"
143
+ # 方式 A:本地目录挂载(把本目录链进 profile 的 node_modules)
144
+ # 方式 B:npm pack 后用宿主插件管理器安装
145
+ ```
146
+
147
+ 插件通过 `package.json` 的 `dsh.bundle.patch` 声明自己:
148
+
149
+ ```json
150
+ { "dsh": { "bundle": { "patch": "./cordis.patch.yml" } } }
151
+ ```
152
+
153
+ `cordis.patch.yml` 负责往宿主配置里插插件行。装好后重启宿主,让模型调
154
+ `computer_use_health` 自检;第一次调用会自动编译 helper(约 1-2 秒)。
155
+
156
+ ## 配置
157
+
158
+ ```yaml
159
+ - id: deepseek-computer-use
160
+ config:
161
+ approval: sensitive # never | sensitive | always
162
+ allowApps: ["notepad", "chrome"] # 空 = 不限制
163
+ denyApps: []
164
+ defaultIncludeScreenshot: false # 默认不抓图,省 token
165
+ maxElements: 260
166
+ snapshotTtlMs: 120000
167
+ dryRun: false
168
+ toolPrefix: '' # 填 'cu_' 可避免与其他插件的 click 重名
169
+ timeoutMs: 20000
170
+ ```
171
+
172
+ ## 契约对齐(对着 Codex 逐条核过)
173
+
174
+ 工具名与参数按 `@oai/sky` 的 window2 契约实现,不是"看着像":
175
+
176
+ | 契约项 | Codex | 本插件 |
177
+ |---|---|---|
178
+ | `get_window_state` 默认 | 截图 ON、accessibility `null` | 一致(`include_text:true` 才出树) |
179
+ | 返回形态 | `{ window, screenshots[], accessibility }` | 一致;`screenshots` 是数组,含窗口 + 关联瞬态 UI(菜单/tooltip),每项带 `id`/`zIndex`/`originX`/`originY`/尺寸 |
180
+ | 坐标寻址 | `click`/`scroll`/`drag` 收 `screenshotId` | 一致;坐标系绑定那张截图,过期报 `stale_screenshot` |
181
+ | `scroll` | `x`,`y` 必填,无 `element_index` | 一致(缺 x/y 直接拒) |
182
+ | accessibility 结构化字段 | `tree`/`focused_element`/`selected_text`/`selected_elements`/`document_text` | 全部实现 |
183
+ | 观测节奏 | 动作后自动等约 1s 再抓 | 一致(`settleMs`,默认 1000) |
184
+ | 元素失效 | 编号只对产生它的观测有效 | 一致(TTL + helper 进程内缓存) |
185
+ | 最小化窗口 | 不该给出无意义画面 | 本插件直接报 `window_minimized`(实测旧版会吐出 334×48 的假图) |
186
+ | 键名 | X keysym + `Numpad_*` 别名 | 一致(`KP_0`/`Numpad_Add`/…) |
187
+ | `list_apps` | `lastUsedDate`、`useCount` | `lastUsedDate` 有;`useCount` 无数据源,不编造 |
188
+ | `launch_app` | app id / `.exe` 路径(含 UWP) | exe 路径、AUMID(`shell:AppsFolder\\…`)、shell URI |
189
+ | Win/Meta 键 | 硬拒绝 | 硬拒绝(插件层 + helper 层双保险) |
190
+ | 终端/Run/密码管理器/安全软件/认证对话框/宿主自身 | 硬拒绝 | 硬拒绝(不可被审批覆盖) |
191
+ | 按应用授权 | "Computer Use was not approved to use <app>" | 每会话每应用一次审批(`appConsent: session`),批准后下发 `approve_app`,**由 helper 强制**:未批准的应用直接 `app_not_approved` |
192
+ | 回合边界 | 新 turn 时向 helper 发 `end_turn`,清元素快照 | 一致:token 变化即 `end_turn`(清快照/截图区/Esc 阻断,保留应用授权) |
193
+ | 请求预算 | 普通 10s、`launch_app` 15s | 一致(`timeoutMs` / `launchTimeoutMs`) |
194
+ | 回合结束停止输入 | `turn_ended` 钩子 | 读会话日志判断 open turn,`turn_ended` 拒绝注入 |
195
+
196
+ ## 明确没做的(不属于 Windows window2 面)
197
+
198
+ - **`paste` / `select_text`**:属于 macOS 的 window 客户端(`target: "mac"`),Windows 上不存在。
199
+ - **播客/音频录制**(`start_audio_recording`):契约里是可选方法,未实现。
200
+ - **full-desktop 面**(`clipboard_read/write`、`key_down/up`、`move`、`drag_handle`、`get_screenshot`):那是 `target: "linux"` 的 API。
201
+ - **区域裁剪截图**:Codex 是"bounded screenshot",我们截整窗(长边封顶 2560)。token 成本略高,语义一致。
202
+ - **Windows.Graphics.Capture**:Codex 用 WGC;我们用 `PrintWindow(PW_RENDERFULLCONTENT)`,遮挡可用、最小化拒绝,不需要 D3D 互操作。
203
+
204
+ ## 抓屏链路(对齐 Codex 的 WGC)
205
+
206
+ 按顺序尝试,前一级成功就不走下一级:
207
+
208
+ 1. **Windows.Graphics.Capture**(`native/bin/capture/dsh-cu-capture.exe`,**.NET Framework 4.8** + SharpDX)
209
+ —— 抓窗口的合成层,遮挡可用、GPU/直合成界面正常、无 DPI 黑边。
210
+ **不依赖 .NET 8 运行时**(Framework 4.8 是 Windows 自带组件),载荷仅 0.68MB
211
+ 2. **PrintWindow(PW_RENDERFULLCONTENT)** —— 无 .NET 8 运行时的机器上的兜底
212
+ 3. **屏幕 BitBlt** —— 前两者失败或返回黑图时
213
+
214
+ WGC 走 .NET Framework 4.8(Windows 10/11 自带),**用户不需要安装任何运行时**。
215
+ 万一抓取失败仍会自动降级到第 2 级。
216
+ 覆盖层是独立窗口,WGC 按窗口抓取时天然不会被拍进去——和 Codex 的
217
+ "exclude display overlay from captures" 等效。
218
+
219
+ ## 已知限制
220
+
221
+ - **只有 Windows 驱动**。macOS / Linux 只有接口(`IPlatformDriver`),没有实现。
222
+ - **前台焦点是硬约束**。前台被置顶工具(如 Process Lasso)或提权窗口占住时,
223
+ `type_text`/`press_key`/坐标点击/`scroll` 会被拒绝并报出占用者。
224
+ 语义动作(`set_value`、Invoke 点击、Expand)不需要前台,照常工作。
225
+ - **Electron/Chromium 无障碍树很薄**,网页内容要走截图 + 坐标。
226
+ - **不绕过 UIPI**:目标窗口权限更高时注入会被系统拦。
227
+ - **DPI 虚拟化已处理**:老式/无 DPI 感知的程序按 96 DPI 渲染,窗口物理尺寸被 DWM 放大,
228
+ `PrintWindow` 返回的是未缩放画面(此前会带黑边)。现在抓屏会裁掉黑边,并记录
229
+ 「物理像素 / 截图像素」比例(实测某 WinForms 程序为 1.46×),坐标动作按该比例换算,
230
+ 所以**截图坐标点哪就点哪**。
231
+ - **`useCount`、应用使用历史**没有可靠本机数据源,宁缺不编。
232
+
233
+ ## 测试
234
+
235
+ ```powershell
236
+ node test/integration.mjs
237
+ ```
238
+
239
+ 固定装置是一个独立的 WinForms 可执行文件(`test/bin/dsh-cu-fixture.exe`,一个文本框 +
240
+ 一个名为 "Delete all" 的按钮;特意不用 shell 托管——shell 窗口本身被硬拒绝)。覆盖:
241
+ 14 个工具注册、截图数组与 origin/zIndex、`accessibility:null` 语义、`set_value`、
242
+ `type_text`、危险按钮触发审批、`dryRun` 不落地、`denyApps` 拒绝、Win 键硬拒绝、
243
+ `scroll` 契约。
244
+
245
+ ## 许可
246
+
247
+ MIT。第三方商用前请自行做安全评审:这个插件能让模型操作你的桌面。
248
+
249
+
250
+ ## 原生载荷(native/bin)
251
+
252
+ | 文件 | 用途 | 大小 |
253
+ |---|---|---|
254
+ | `dsh-computer-use-helper.exe` | UIA / SendInput / 抓屏调度 / 音频调度 | ~50KB |
255
+ | `dsh-cu-overlay.exe` | 鲸鱼娘 + 虚拟光标覆盖层 | ~30KB |
256
+ | `capture/` | WGC 抓屏(.NET Framework 4.8 + SharpDX) | 0.68MB |
257
+ | `audio/` | WASAPI 环回录音(NAudio) | 0.60MB |
258
+ | **合计** | | **~1.4MB** |
259
+
260
+ 全部基于 Windows 自带的 .NET Framework 4.8,**用户零安装**。
261
+
262
+ ## 与 Codex window2 的细节对齐(第二轮)
263
+
264
+ | 能力 | 实现 |
265
+ |---|---|
266
+ | 协议握手 | helper 提供 `hello` → `{ protocolVersion: 1, capabilities: [...] }`;bridge 首次连接校验,不匹配直接报 `protocol_mismatch` |
267
+ | 每请求预算 | 客户端把预算随请求下发(`__budgetMs`,对应 Codex 的 `x-oai-cua-request-budget-ms`),helper 用它收缩抓屏等待 |
268
+ | 坐标越界 | `point_outside_screenshot`,文案同 Codex("is outside screenshot bounds … Take a fresh screenshot before retrying") |
269
+ | 截图失效 | `stale_screenshot` 带同一句 "Take a fresh screenshot before retrying" |
270
+ | 优雅关闭 | `close` RPC → helper 落盘后退出 0;插件 dispose 时先发 close 再兜底 kill |
271
+ | Esc 语义 | 覆盖层退出码 **130** → helper 写入 `%LOCALAPPDATA%\dsh-computer-use\interrupts\<会话>-<turn>.flag` 并自身退出 130;同 turn 后续调用 `turn_stopped`,新 turn 放行 |
272
+ | 音频授权 | 独立于应用授权的一次性确认(Codex: `computer-audio` / "Allow Computer Use to record computer audio?") |
273
+ | 音频返回 | `{ filepath, bytes: base64 WAV, data_url }`,统一 **24 kHz 立体声 16-bit** |
274
+ | 音频上限 | 单次录音 ≤ 300 000 ms(Codex 同值),不足 500 ms 自动抬到 500 |
275
+ | App 字段 | `isRunning` + `lastUsedDate`(进程启动时间);`useCount` 无数据来源,按 Codex 的可选字段处理,不伪造 |
276
+
277
+ marker 文件按 **会话 + turn** 命名,避免旧会话的 Esc 标记误伤新会话。
278
+
279
+
280
+ ## 配置示例
281
+
282
+ ```yaml
283
+ - insert:
284
+ - id: deepseek-computer-use
285
+ name: 'deepseek-computer-use'
286
+ config:
287
+ approval: sensitive # never | sensitive | always
288
+ appConsent: session # 每个应用问一次;never 表示不问
289
+ audio: false # 开启 WASAPI 环回录音(start/stop_audio_recording)
290
+ redactApps: [] # 这些应用的截图永不出机器,例如 ["1password", "银行"]
291
+ defaultIncludeScreenshot: true
292
+ toolPrefix: '' # 例如 cu_ 可避免和其他插件的 click 重名
293
+ ```
294
+
295
+ ## 已知边界
296
+
297
+ - 只支持 Windows;驱动层是接口(`IPlatformDriver`),macOS / Linux 可照着补。
298
+ - Codex window2 没有的接口(`paste` / `clipboard_*` / `key_down` / `move`)这里也没有。
299
+ - `App.useCount` 没有数据来源,按可选字段留空,不伪造。
300
+
301
+ ## License
302
+
303
+ MIT
@@ -0,0 +1,28 @@
1
+ # deepseek-computer-use —— Windows 桌面控制(Computer Use)插件。
2
+ #
3
+ # 安装即向宿主注册 16 个工具:list_apps / list_windows / get_window / launch_app /
4
+ # get_window_state / click / press_key / type_text / scroll / set_value / drag /
5
+ # perform_secondary_action / activate_window / start_audio_recording /
6
+ # stop_audio_recording / computer_use_health。
7
+ #
8
+ # 首次加载会在 native/bin 下用系统自带 csc.exe 编译 helper(无需 .NET SDK、
9
+ # 无 npm 原生依赖)。全部配置项都有安全默认值,空配置即可启动:
10
+ #
11
+ # - id: deepseek-computer-use
12
+ # config:
13
+ # approval: sensitive # never | sensitive | always
14
+ # allowApps: [] # 空 = 不限制;可用 exe 路径子串或进程名
15
+ # denyApps: []
16
+ # defaultIncludeScreenshot: true # Codex parity: capture unless told otherwise
17
+ # settleMs: 1000 # wait for the UI to settle before the next capture
18
+ # appConsent: session # ask once per app (never = skip)
19
+ # maxElements: 260
20
+ # snapshotTtlMs: 120000
21
+ # dryRun: false
22
+ # toolPrefix: '' # 例如 cu_ 可避免与其他插件的 click 重名
23
+ # audio: false # 开启 WASAPI 环回录音(start/stop_audio_recording)
24
+ # redactApps: [] # 这些应用的截图永不出机器,例如 ["1password"]
25
+ - insert:
26
+ - id: deepseek-computer-use
27
+ name: 'deepseek-computer-use'
28
+ config: {}
package/lib/bridge.js ADDED
@@ -0,0 +1,224 @@
1
+ /**
2
+ * Lifecycle and framing for the native helper process.
3
+ *
4
+ * One long-lived helper owns the UI Automation element cache, so element
5
+ * indexes returned by get_window_state stay resolvable by click/set_value.
6
+ * Requests are serialized: UI Automation calls are not reentrant and a
7
+ * desynchronized element cache is worse than waiting a few milliseconds.
8
+ *
9
+ * @module deepseek-computer-use/bridge
10
+ */
11
+ import { spawn } from 'node:child_process';
12
+
13
+ /** Protocol version this plugin speaks; the helper reports its own in `hello`. */
14
+ const MIN_PROTOCOL = 1;
15
+
16
+ /** Codex's wording, reused verbatim so the model reacts the same way. */
17
+ const ESC_MESSAGE = 'Computer Use was stopped by the user with the physical Escape key. Stop your work, do not call further Computer Use tools in this turn, and send a final message noting that the user stopped Computer Use.';
18
+
19
+ /** An error raised by the helper, carrying its machine-readable code. */
20
+ export class HelperError extends Error {
21
+ constructor(code, message, details) {
22
+ super(message);
23
+ this.name = 'HelperError';
24
+ this.code = code;
25
+ this.details = details;
26
+ }
27
+ }
28
+
29
+ export class HelperBridge {
30
+ #child = null;
31
+ #buffer = '';
32
+ #pending = new Map();
33
+ #seq = 0;
34
+ #queue = Promise.resolve();
35
+ #exe;
36
+ #timeoutMs;
37
+ #protocol = null;
38
+ #interruptedTurn = null;
39
+ #lastTurn = null;
40
+ #disposed = false;
41
+ #log;
42
+
43
+ constructor({ exe, timeoutMs = 20000, log } = {}) {
44
+ if (typeof exe !== 'string' || exe.length === 0) throw new Error('HelperBridge requires an executable path');
45
+ this.#exe = exe;
46
+ this.#timeoutMs = timeoutMs;
47
+ this.#log = log;
48
+ }
49
+
50
+ get running() {
51
+ return this.#child !== null && this.#child.exitCode === null && !this.#child.killed;
52
+ }
53
+
54
+ #ensureChild() {
55
+ if (this.#disposed) throw new HelperError('disposed', 'the Computer Use bridge was disposed');
56
+ if (this.running) return this.#child;
57
+ const child = spawn(this.#exe, [], { stdio: ['pipe', 'pipe', 'pipe'], windowsHide: true });
58
+ child.stdout.setEncoding('utf8');
59
+ child.stderr.setEncoding('utf8');
60
+ child.stdout.on('data', chunk => this.#onData(chunk));
61
+ child.stderr.on('data', chunk => {
62
+ const text = String(chunk).trim();
63
+ if (text.length > 0) this.#log?.('[dsh-computer-use] helper stderr: ' + text);
64
+ });
65
+ child.on('exit', (code, signal) => {
66
+ this.#child = null;
67
+ this.#protocol = null;
68
+ // Codex's helper exits 130 when the user presses Esc; the transport turns that
69
+ // into the interrupt error so the model stops instead of retrying.
70
+ const interrupted = code === 130;
71
+ if (interrupted && this.#lastTurn !== null) this.#interruptedTurn = this.#lastTurn;
72
+ const reason = interrupted
73
+ ? ESC_MESSAGE
74
+ : 'the Computer Use helper exited (code ' + code + ', signal ' + signal + ')';
75
+ for (const [, entry] of this.#pending) {
76
+ clearTimeout(entry.timer);
77
+ entry.reject(new HelperError(interrupted ? 'cancelled_by_user' : 'helper_exited', reason));
78
+ }
79
+ this.#pending.clear();
80
+ });
81
+ child.on('error', error => {
82
+ this.#log?.('[dsh-computer-use] helper spawn error: ' + error.message);
83
+ });
84
+ this.#child = child;
85
+ return child;
86
+ }
87
+
88
+ #onData(chunk) {
89
+ this.#buffer += chunk;
90
+ let index = this.#buffer.indexOf('\n');
91
+ while (index >= 0) {
92
+ const line = this.#buffer.slice(0, index).trim();
93
+ this.#buffer = this.#buffer.slice(index + 1);
94
+ if (line.length > 0) this.#settle(line);
95
+ index = this.#buffer.indexOf('\n');
96
+ }
97
+ }
98
+
99
+ #settle(line) {
100
+ let payload;
101
+ try {
102
+ payload = JSON.parse(line);
103
+ } catch {
104
+ this.#log?.('[dsh-computer-use] helper produced non-JSON output: ' + line.slice(0, 200));
105
+ return;
106
+ }
107
+ const entry = this.#pending.get(payload.id);
108
+ if (entry === undefined) return;
109
+ this.#pending.delete(payload.id);
110
+ clearTimeout(entry.timer);
111
+ if (payload.ok === true) entry.resolve(payload.result);
112
+ else entry.reject(new HelperError(payload.error?.code ?? 'helper_error', payload.error?.message ?? 'the helper reported an unknown failure', payload.error));
113
+ }
114
+
115
+ /**
116
+ * Send one request. Calls are serialized through the queue; a caller that
117
+ * never gets an answer fails on the timeout rather than hanging the turn.
118
+ */
119
+ request(method, params = {}, { signal, timeoutMs, turnToken } = {}) {
120
+ const wire = turnToken === undefined ? params : { ...params, turnToken };
121
+ if (turnToken !== undefined) this.#lastTurn = turnToken;
122
+ const run = async () => {
123
+ // Codex stops issuing input for the rest of a turn once the user hits Esc.
124
+ // The helper persists the same fact, but failing here keeps the error
125
+ // immediate even while a fresh helper process is spinning up.
126
+ if (turnToken !== undefined && this.#interruptedTurn === turnToken) {
127
+ throw new HelperError('turn_stopped', ESC_MESSAGE);
128
+ }
129
+ await this.#ensureProtocol(signal);
130
+ try {
131
+ return await this.#call(method, wire, { signal, timeoutMs });
132
+ } catch (error) {
133
+ if ((error?.code === 'cancelled_by_user' || error?.code === 'turn_stopped') && turnToken !== undefined) {
134
+ this.#interruptedTurn = turnToken;
135
+ }
136
+ throw error;
137
+ }
138
+ };
139
+ const next = this.#queue.then(run, run);
140
+ this.#queue = next.then(() => undefined, () => undefined);
141
+ return next;
142
+ }
143
+
144
+ /**
145
+ * The helper reports `protocolVersion` from `hello`. Codex exposes the same fact as
146
+ * `window_protocol_version`; checking it up front turns a silent mismatch into one
147
+ * clear error instead of a stream of malformed-stream failures.
148
+ */
149
+ async #ensureProtocol(signal) {
150
+ if (this.#protocol !== null) return;
151
+ const hello = await this.#call('hello', {}, { signal, timeoutMs: 5000 });
152
+ const version = Number(hello?.protocolVersion);
153
+ if (!Number.isFinite(version) || version < MIN_PROTOCOL) {
154
+ throw new HelperError('protocol_mismatch',
155
+ 'the Computer Use helper speaks protocol ' + (hello?.protocolVersion ?? 'unknown') +
156
+ ', this plugin needs ' + MIN_PROTOCOL + ' or newer. Reinstall deepseek-computer-use so the helper and the plugin match.');
157
+ }
158
+ this.#protocol = version;
159
+ }
160
+
161
+ #call(method, params, { signal, timeoutMs }) {
162
+ if (signal?.aborted) return Promise.reject(new HelperError('cancelled', 'the request was cancelled before it started'));
163
+ const child = this.#ensureChild();
164
+ const id = ++this.#seq;
165
+ const budgetMs = Number.isFinite(timeoutMs) ? timeoutMs : this.#timeoutMs;
166
+ // Codex ships the budget with every call as x-oai-cua-request-budget-ms so the
167
+ // helper can bound its own waits; the same deadline travels here.
168
+ const payload = JSON.stringify({ id, method, params: { ...params, __budgetMs: budgetMs } });
169
+
170
+ const raw = new Promise((resolve, reject) => {
171
+ const budget = Number.isFinite(timeoutMs) ? timeoutMs : this.#timeoutMs;
172
+ const timer = setTimeout(() => {
173
+ this.#pending.delete(id);
174
+ reject(new HelperError('timeout', 'the Computer Use helper did not answer ' + method + ' within ' + budget + 'ms'));
175
+ }, budget);
176
+ timer.unref?.();
177
+
178
+ const onAbort = () => {
179
+ this.#pending.delete(id);
180
+ clearTimeout(timer);
181
+ reject(new HelperError('cancelled', 'the request was cancelled'));
182
+ };
183
+ signal?.addEventListener('abort', onAbort, { once: true });
184
+
185
+ this.#pending.set(id, {
186
+ timer,
187
+ resolve: value => {
188
+ signal?.removeEventListener('abort', onAbort);
189
+ resolve(value);
190
+ },
191
+ reject: error => {
192
+ signal?.removeEventListener('abort', onAbort);
193
+ reject(error);
194
+ },
195
+ });
196
+
197
+ try {
198
+ child.stdin.write(payload + '\n');
199
+ } catch (error) {
200
+ this.#pending.delete(id);
201
+ clearTimeout(timer);
202
+ reject(new HelperError('write_failed', 'cannot talk to the Computer Use helper: ' + error.message));
203
+ }
204
+ });
205
+ // The caller always handles this promise; the extra branch only stops Node from
206
+ // treating a momentarily unobserved rejection as a fatal unhandled one.
207
+ raw.catch(() => { /* observed by the caller; this branch only silences Node */ });
208
+ return raw;
209
+ }
210
+
211
+ dispose() {
212
+ this.#disposed = true;
213
+ const child = this.#child;
214
+ this.#child = null;
215
+ if (child !== null) {
216
+ // Codex closes the helper with a close RPC before tearing the process down.
217
+ try {
218
+ child.stdin.write(JSON.stringify({ id: ++this.#seq, method: 'close', params: {} }) + '\n');
219
+ } catch { /* already gone */ }
220
+ try { child.stdin.end(); } catch { /* already gone */ }
221
+ setTimeout(() => { try { child.kill(); } catch { /* already gone */ } }, 150).unref?.();
222
+ }
223
+ }
224
+ }