dsh-plugin-computer-use 2.4.1 → 2.5.0

package/LICENSE

@@ -1,21 +1,21 @@
-MIT License
-
-Copyright (c) 2026 dsh-plugin contributors
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE.
+MIT License
+
+Copyright (c) 2026 dsh-plugin contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -2,6 +2,63 @@
 
 让模型**看见窗口和桌面并操作鼠标键盘**——v2 起整个后端换装 [Cua Driver](https://cua.ai) 的**进程内 SDK**（`@trycua/cua-driver`，Rust 原生运行时随 npm 包按平台分发，直接加载进 dsh 宿主进程）：**无守护进程、无 MCP、无 PowerShell**，装上即用，跨 Windows / macOS / Linux。
 
+## v2.5：Ubuntu 26.04 / GNOME Wayland
+
+旧版没有开启驱动的原生 Wayland 开关，即使 `DISPLAY=:0` 存在，它也可能只是 XWayland，无法代表整个桌面。v2.5 在加载原生库前检测 `XDG_SESSION_TYPE` / `WAYLAND_DISPLAY`，自动设置 `CUA_DRIVER_RS_ENABLE_WAYLAND=1`；驱动依赖最低升到 npm 已发布的 0.28.3。环境中显式设置的 `0` / `false` 会保留，行配置 `wayland: on` / `off` 可覆盖。
+
+GNOME/Mutter 还需要 **WinRects Shell 扩展**提供窗口几何、截图和精确激活；输入通过 Remote Desktop portal/libei 获取系统授权。npm SDK 没带 CLI 发行包中的扩展，因此本插件随包附带上游 WinRects v8（支持 GNOME 45–50；含来源与 MIT 许可证），并提供安装和检查入口。无需另装 Cua 守护进程。
+
+**从 npm 安装/更新**（在 Ubuntu 的图形桌面终端运行，替换自己的 profile）：
+
+```sh
+dsh plugin --profile web add dsh-plugin-computer-use@2.5.0
+npx --yes --package=dsh-plugin-computer-use@2.5.0 dsh-computer-use-wayland install
+# 注销并重新登录一次后：
+npx --yes --package=dsh-plugin-computer-use@2.5.0 dsh-computer-use-wayland status
+```
+
+如果已经使用 `tofu-box`，通过 `dsh plugin --profile web update tofu-box` 更新依赖，再执行上面的扩展安装/检查命令。伞包现有 `^2.4.1` 范围覆盖 2.5.0；无需另外添加单独的 computer-use 插件行。更新与安装扩展后重启 dsh。
+
+**从本仓库试用：在 Ubuntu 的图形桌面终端运行**（把路径和 profile 换成自己的）：
+
+```sh
+cd /path/to/dsh-plugins
+node dsh-plugin-computer-use/scripts/wayland.mjs install
+```
+
+安装只写当前用户的扩展目录，保留已启用的其他扩展。之后需要**注销并重新登录一次**，让 GNOME 加载扩展；Wayland 下不能用 Alt+F2 → r 重启 Shell。
+
+```sh
+cd /path/to/dsh-plugins
+node dsh-plugin-computer-use/scripts/wayland.mjs status
+dsh plugin --profile web add "link:$(pwd)/dsh-plugin-computer-use"
+```
+
+重启 dsh，然后先试 `computer_windows` 和 `computer_screenshot`。第一次需要前台输入时，允许系统弹出的远程桌面/键鼠控制授权。此授权属于驱动运行时，重启后可能再次询问。插件安装不会自动授予它。
+
+**用本地 tgz 安装**：下载/复制 `dsh-plugin-computer-use-2.5.0.tgz` 到 Ubuntu 后运行：
+
+```sh
+dsh plugin --profile web add ./dsh-plugin-computer-use-2.5.0.tgz
+npx --yes --package=./dsh-plugin-computer-use-2.5.0.tgz dsh-computer-use-wayland install
+# 注销并重新登录后：
+npx --yes --package=./dsh-plugin-computer-use-2.5.0.tgz dsh-computer-use-wayland status
+```
+
+如果通过 `tofu-box` 安装，应更新伞包中的依赖；不要叠装单独的 computer-use 行。也可直接对该 profile 下已安装包的 `scripts/wayland.mjs` 运行 `install` / `status`。
+
+`status` 检查实际加载的扩展版本、Remote Desktop portal 的键鼠能力、无障碍总线与会话变量；缺项时返回非零退出码。Ubuntu 缺少组件时可以安装：
+
+```sh
+sudo apt install libglib2.0-bin at-spi2-core xdg-desktop-portal xdg-desktop-portal-gnome
+```
+
+请从同一个图形用户会话启动 dsh，并保留 `WAYLAND_DISPLAY`、`XDG_RUNTIME_DIR`、`DBUS_SESSION_BUS_ADDRESS`。不要用 root 或另一条不带桌面环境的 SSH 会话启动。若 GNOME 的 Extensions 总开关关闭，请先在扩展管理器开启，再运行检查。
+
+Wayland 的后台能力取决于应用的 AT-SPI 支持，像素/键盘输入可能需要临时前置窗口；不保证与 Windows 一样能操作完全遮挡的窗口。缺少扩展、portal 或权限时，驱动错误和空窗口表会附带安装提示。Sway/KDE 等沿用驱动各自的原生支持范围，WinRects 安装器只用于 GNOME。参见[上游平台支持说明](https://cua.ai/docs/cua-driver/concepts/platform-support)。
+
+当前开发机为 Windows；本版已做会话检测、安装器、诊断与插件接入回归，**尚未在 Ubuntu 26.04 实机完成截图/输入验收**。
+
 ## 打开应用的行为（v2.3 起）
 
 `computer_open` 只做一件事：**能复用就复用，确认没在跑才启动**。它按下面的顺序判断，顺序本身就是设计——"有没有进程"比"有没有窗口"可靠得多：
@@ -78,6 +135,7 @@
     askPolicy: once-per-agent  # 或 always / never
     telemetry: false           # Cua Driver 运行时的无内容产品遥测，默认关
     dpiAware: true             # Windows：驱动启动前把宿主进程切到 Per-Monitor-V2 DPI 感知
+    wayland: auto              # Linux：自动启用原生 Wayland；on/off 显式覆盖环境开关
     restore: true              # 允许把已运行应用的最小化窗口唤醒到前台（false = 永不抢焦点）
     activateRunning: false     # 应用在跑但没窗口（托盘态）时，是否让它的启动入口去激活（默认否，只报告）
 ```
@@ -90,7 +148,7 @@
 
 ## 平台与限制
 
-- **Windows 11 实测**（自测脚本见下）；macOS / Linux 走同一代码路径但**未实测**——macOS 需给运行 dsh 的宿主进程授予"辅助功能/屏幕录制"（TCC）权限，且驱动的 macOS 前台叠加层在进程内形态不可用。
+- **Windows 11 实测**（自测脚本见下）；macOS / Linux **未实机验证**。Linux Wayland 从 v2.5 起自动开启原生后端，GNOME 的安装与诊断见上；macOS 需给运行 dsh 的宿主进程授予"辅助功能/屏幕录制"（TCC）权限，且驱动的 macOS 前台叠加层在进程内形态不可用。
 - Windows 高 DPI：启动日志会打印 `host process DPI awareness = …`；正常为 `per-monitor-v2`。若显示 `UNAWARE`（宿主 manifest 或他方抢先固定了感知级别），桌面截图/坐标在缩放屏上不可靠，插件会在每张桌面截图摘要里带显式警告——优先改用 `app=` 窗口操作。
 - 桌面截图仅主显示器（窗口级截图不受限）；被最小化的窗口**读树可以、截图不行**，插件会先唤醒它再截——这是唯一会短暂占用前台的动作，`restore: false` 可关掉。
 - **窗口枚举可能失灵**：长生命周期的宿主里可能出现 `list_windows` 恒返回 0 窗口（此时桌面截图、光标读取、指定窗口的截图都还正常）。这种情况下按进程判断的部分仍然可靠（不会重复启动应用），但没法定位/点击那个窗口——重启 dsh 可重建枚举。
@@ -99,6 +157,12 @@
 
 ## 自测
 
+无需图形桌面的 Wayland 回归测试（检测、扩展安装与诊断）：
+
+```sh
+node --experimental-test-module-mocks --test tests/*.test.mjs
+```
+
 ```sh
 node selftest/run.mjs
 ```

package/index.js

@@ -4,8 +4,9 @@
  * v2 rewires the whole backend onto the Cua Driver SDK (@trycua/cua-driver):
  * a Rust in-process computer-use runtime loaded directly into the dsh host.
  * No daemon, no MCP server, no PowerShell scripts — `pnpm` pulls one native
- * package per OS/arch and the tools just work (Windows tested; the same code
- * path drives the macOS / Linux natives).
+ * package per OS/arch (Windows tested; macOS / Linux require their desktop
+ * permissions). v2.5 enables native Wayland before runtime initialization and
+ * bundles the GNOME WinRects helper with an explicit user-session installer.
  *
  * v2.2 fixes the two things that made v2 unusable on a real Windows desktop:
  *
@@ -83,6 +84,7 @@
 import zlib from 'node:zlib'
 import z from '@deepseek-ai/schemastery'
 import { defineTool } from '@deepseek-ai/dsh-tools'
+import { configureLinuxDesktop, probeGnomeHelper, linuxDesktopHint } from './linux.js'
 
 export const name = 'computer-use'
 export const inject = ['tools', 'attachments']
@@ -100,6 +102,8 @@ export const Config = z.object({
    * driver starts (strongly recommended; set false only if some other component
    * already owns process DPI awareness and you must not touch it). */
   dpiAware: z.boolean(),
+  /** Native Wayland: auto-detect by default; on/off override the driver env flag. */
+  wayland: z.union([z.const('auto'), z.const('on'), z.const('off')]),
 })
 
 // ─────────────────────────────────────────────────────────────────────────────
@@ -291,6 +295,11 @@ export function apply(ctx, config) {
   // that app exposes no window (tray state)? Off by default — see computer_open.
   const activateRunningCfg = !!(config && config.activateRunning === true)
   if (!telemetry) process.env.CUA_DRIVER_RS_TELEMETRY_ENABLED = '0'
+  // Must happen BEFORE importing/creating the native runtime. DISPLAY may be
+  // XWayland and is deliberately not used to select the desktop backend.
+  const linuxDesktop = configureLinuxDesktop({ wayland: config?.wayland ?? 'auto' })
+  const desktopHint = linuxDesktopHint(linuxDesktop)
+  const driverError = (tool, r, limit = 400) => new Error(tool + ': ' + String(r.text ?? 'failed').slice(0, limit) + desktopHint)
 
   // ── DPI bootstrap (Windows) ────────────────────────────────────────────────
   // The driver assumes a Per-Monitor-V2-aware host (its daemon exe ships that
@@ -372,8 +381,16 @@ export function apply(ctx, config) {
     if (driver) return Promise.resolve(driver)
     if (driverDead) throw new Error('Cua Driver runtime was shut down; restart dsh to reactivate computer-use')
     if (!driverPromise) {
-      driverPromise = ensureProcessDpiAware().then(() => {
+      driverPromise = ensureProcessDpiAware().then(async () => {
         ctx.logger.info('computer-use: host process DPI awareness = ' + dpiState.text)
+        if (linuxDesktop.native) {
+          if (!process.env.WAYLAND_DISPLAY) throw new Error('Native Wayland needs WAYLAND_DISPLAY from the graphical user session.' + desktopHint)
+          ctx.logger.info('computer-use: native Wayland backend enabled (XWayland DISPLAY does not override it)')
+          if (linuxDesktop.gnome) {
+            const helper = await probeGnomeHelper()
+            if (!helper.active) ctx.logger.warn('computer-use: GNOME WinRects helper missing or outdated.' + desktopHint)
+          }
+        }
         return import('@trycua/cua-driver')
       }).then((m) => {
         const created = m.CuaDriver.create()
@@ -438,7 +455,7 @@ export function apply(ctx, config) {
   /** Same, but throws on isError with the driver's own message. */
   const call = async (tool, args) => {
     const r = await raw(tool, args)
-    if (r.isError) throw new Error(tool + ': ' + String(r.text ?? 'failed').slice(0, 400))
+    if (r.isError) throw driverError(tool, r)
     return r
   }
   const structured = (r) => {
@@ -454,7 +471,7 @@ export function apply(ctx, config) {
    */
   const listWindows = async (args) => {
     const r = await raw('list_windows', args)
-    if (r.isError) throw new Error('list_windows: ' + String(r.text ?? 'failed').slice(0, 300))
+    if (r.isError) throw driverError('list_windows', r, 300)
     return structured(r)
   }
 
@@ -486,7 +503,7 @@ export function apply(ctx, config) {
       if (pid === undefined) {
         const needle = String(a.app).toLowerCase()
         const cands = wins.filter((x) => String(x.app_name ?? '').toLowerCase().includes(needle) || String(x.title ?? '').toLowerCase().includes(needle))
-        if (cands.length === 0) throw new Error('no window matches app "' + a.app + '" — check computer_windows')
+        if (cands.length === 0) throw new Error('no window matches app "' + a.app + '" — check computer_windows' + desktopHint)
         pid = cands[0].pid
       }
       let mine = wins.filter((x) => x.pid === pid)
@@ -925,11 +942,11 @@ export function apply(ctx, config) {
     const r = await raw(tool, base)
     if (r.isError && isEscalationHint(r)) {
       const r2 = await raw(tool, { ...base, delivery_mode: 'foreground' })
-      if (r2.isError) throw new Error(tool + ': ' + String(r2.text ?? 'failed').slice(0, 400))
+      if (r2.isError) throw driverError(tool, r2)
       r2.escalated = true
       return r2
     }
-    if (r.isError) throw new Error(tool + ': ' + String(r.text ?? 'failed').slice(0, 400))
+    if (r.isError) throw driverError(tool, r)
     return r
   }
   const actionNote = (r) => (r.escalated ? ' [background input was refused; re-delivered with a brief foreground swap]' : /"effect":"suspected_noop"/.test(String(r.rawJson ?? '')) ? ' [driver suspects the target ignored the input — verify on the screenshot]' : '')
@@ -954,7 +971,7 @@ export function apply(ctx, config) {
       await sleep(350)
       r2 = await raw('press_key', { ...targetArg, key, modifiers: mods, delivery_mode: 'foreground' })
     }
-    if (r2.isError) throw new Error('hotkey ' + mods.join('+') + '+' + key + ': ' + String(r.text ?? 'failed').slice(0, 260) + ' (foreground fallback too: ' + String(r2.text ?? 'failed').slice(0, 160) + ')')
+    if (r2.isError) throw new Error('hotkey ' + mods.join('+') + '+' + key + ': ' + String(r.text ?? 'failed').slice(0, 260) + ' (foreground fallback too: ' + String(r2.text ?? 'failed').slice(0, 160) + ')' + desktopHint)
     r2.escalated = true
     return r2
   }
@@ -992,7 +1009,8 @@ export function apply(ctx, config) {
 
   const COORD_DESC = 'Coordinate frames: with a window target, x/y are pixels of the LAST computer_screenshot OF THAT WINDOW; desktop scope uses pixels of the last desktop screenshot (physical-desktop px are auto-mapped into the frame).'
   const KEYB_FIRST = ' KEYBOARD-FIRST: prefer computer_open / computer_key / computer_sequence over pixel clicks whenever a keyboard path exists. ELEMENT-FIRST: if the last window screenshot listed an element for the target, click by its element= token — it works on backgrounded/minimized windows and never touches the user\'s cursor.'
-  const register = (definition) => ctx.effect(() => ctx.tools.register(defineTool(definition)), 'computer-use: tool ' + definition.name)
+  const platformNote = linuxDesktop.native ? ' On native Wayland, semantic AT-SPI actions may run in the background; raw input may require a foreground swap and system Remote Desktop consent. GNOME needs the WinRects helper; a missing route returns a setup hint.' : ''
+  const register = (definition) => ctx.effect(() => ctx.tools.register(defineTool({ ...definition, description: definition.description + platformNote })), 'computer-use: tool ' + definition.name)
 
   /** Shared action wrapper: grant, run, settle, re-capture the acted-on view. */
   const actAndShot = async (exec, grant, summary, run) => {
@@ -1089,7 +1107,7 @@ export function apply(ctx, config) {
         const needle = String(a.app).toLowerCase()
         wins = wins.filter((x) => String(x.app_name ?? '').toLowerCase().includes(needle) || String(x.title ?? '').toLowerCase().includes(needle))
       }
-      if (wins.length === 0) return { table: 'no windows matched' }
+      if (wins.length === 0) return { table: 'no windows matched' + ((sc.windows ?? []).length === 0 ? desktopHint : '') }
       const rows = wins.slice(0, 40).map((w) => 'pid=' + w.pid + ' win=' + (w.window_id ?? w.windowId) + (w.is_on_screen === false ? ' [off-screen]' : '') + (w.minimized ? ' [minimized]' : '') + ' z=' + (w.z_index ?? '?') + ' ' + JSON.stringify(w.app_name ?? '') + ' ' + JSON.stringify(String(w.title ?? '').slice(0, 60)) + ' ' + JSON.stringify(w.bounds ?? {}))
       return { table: rows.join('\n') + (wins.length > 40 ? '\n… ' + (wins.length - 40) + ' more (filter with app=)' : '') }
     },
@@ -1583,7 +1601,7 @@ export function apply(ctx, config) {
     return { kind: 'ask', reason: '允许 computer-use 操作应用？一次批准覆盖同一会话的后续动作（askPolicy: always 可改为每次一询）；默认后台注入——不移动你的鼠标、不抢焦点，仅个别拒收后台输入的应用会短暂前置。首个动作: ' + exec.name + '。' }
   })
 
-  ctx.logger.info('computer-use v2.4: cua-driver backend (session "' + SESSION_LABEL + '", all calls labelled); ' + MUTATING.size + ' mutating tools gated; askPolicy=' + askPolicy + '; telemetry=' + telemetry + '; restoreExisting=' + restoreCfg)
+  ctx.logger.info('computer-use v2.5: cua-driver backend (session "' + SESSION_LABEL + '", all calls labelled); ' + MUTATING.size + ' mutating tools gated; askPolicy=' + askPolicy + '; telemetry=' + telemetry + '; restoreExisting=' + restoreCfg + '; nativeWayland=' + linuxDesktop.native)
 
   // Best-effort: push the model-frame cap into the driver for window captures
   // (the driver resizes window screenshots AND keeps x,y/zoom coordinates

package/linux.js

@@ -0,0 +1,41 @@
+import { execFile } from 'node:child_process'
+
+export const WAYLAND_ENV = 'CUA_DRIVER_RS_ENABLE_WAYLAND'
+export const HELPER_UUID = 'winrects@cua'
+export const HELPER_VERSION = 8
+
+/** DISPLAY is also set by XWayland; it must not hide a native Wayland session. */
+export function configureLinuxDesktop({ platform = process.platform, env = process.env, wayland = 'auto' } = {}) {
+  const state = { linux: platform === 'linux', wayland: false, native: false, gnome: false }
+  if (!state.linux) return state
+  const session = String(env.XDG_SESSION_TYPE ?? '').toLowerCase()
+  state.wayland = session === 'wayland' || (session !== 'x11' && !!env.WAYLAND_DISPLAY)
+  state.gnome = /gnome|ubuntu/i.test(String(env.XDG_CURRENT_DESKTOP ?? '') + ':' + String(env.DESKTOP_SESSION ?? ''))
+  if (wayland === 'on') env[WAYLAND_ENV] = '1'
+  else if (wayland === 'off') env[WAYLAND_ENV] = '0'
+  else if (state.wayland && env[WAYLAND_ENV] === undefined) env[WAYLAND_ENV] = '1'
+  const flag = String(env[WAYLAND_ENV] ?? '').trim()
+  state.native = state.wayland && !!flag && !/^(0|false)$/i.test(flag)
+  return state
+}
+
+/** Bounded, shell-free desktop probes; never request screen/input permission here. */
+export function runDesktopCommand(command, args) {
+  return new Promise((resolve) => {
+    execFile(command, args, { timeout: 4000, maxBuffer: 256 * 1024, windowsHide: true }, (error, stdout, stderr) => {
+      resolve({ ok: !error, stdout: String(stdout ?? '').trim(), error: error ? String(stderr || error.message).trim() : '' })
+    })
+  })
+}
+
+export async function probeGnomeHelper(run = runDesktopCommand) {
+  const result = await run('gdbus', ['call', '--session', '--dest', 'org.cua.WinRects', '--object-path', '/org/cua/WinRects', '--method', 'org.cua.WinRects.GetVersion'])
+  const version = result.ok ? Number(/(?:uint32\s+)?(\d+)/.exec(result.stdout)?.[1] ?? 0) : 0
+  return { active: result.ok && version >= HELPER_VERSION, version, error: result.error }
+}
+
+export function linuxDesktopHint(state) {
+  if (!state.native) return ''
+  if (!state.gnome) return ' Native Wayland is enabled; available capture and input routes depend on the compositor. Run dsh in the graphical user session with WAYLAND_DISPLAY, XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS available.'
+  return ' GNOME/Wayland setup: run dsh-computer-use-wayland install (or node <plugin-dir>/scripts/wayland.mjs install), then log out and back in once; run dsh-computer-use-wayland status in the same desktop session. Allow the system Remote Desktop prompt on the first input action. Keep WAYLAND_DISPLAY, XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS available to the dsh process.'
+}

package/package.json

@@ -1,17 +1,19 @@
 {
   "name": "dsh-plugin-computer-use",
-  "version": "2.4.1",
+  "version": "2.5.0",
   "license": "MIT",
   "keywords": ["dsh-plugin", "deepseek-harness", "dsh", "computer-use", "desktop", "automation", "screenshot", "input"],
   "publishConfig": { "registry": "https://registry.npmjs.org" },
   "repository": { "type": "git", "url": "git+https://github.com/tofu-zyc/dsh-plugin-tofu-box.git", "directory": "dsh-plugin-computer-use" },
   "bugs": { "url": "https://github.com/tofu-zyc/dsh-plugin-tofu-box/issues" },
   "homepage": "https://github.com/tofu-zyc/dsh-plugin-tofu-box/tree/main/dsh-plugin-computer-use",
-  "description": "computer-use v2.4：基于 Cua Driver 进程内 SDK 的跨平台桌面操作。13 个工具（截图/窗口列表/缩放/光标/移动/点击/拖拽/滚动/输入/按键/打开应用/连续动作/等待）。v2.4.1 适配 dsh 0.2.0：审批探测改走公有 overrideOf/session 覆盖 + 部署默认策略，任何探测形状失效时保守放行，杜绝 danger-full-access 会话因私有 API 变动而自锁全部工具。v2.4 修「宿主闲置后窗口/应用枚举静默变空」：全部驱动调用统一挂命名 session（查询也不例外），session 卡死按 end→start→重试→重建运行时分级自愈，list_windows 报错不再吞成『无窗口』；computer_type 桌面输入遇非 ASCII 自动改走剪贴板粘贴（绕开 IME 对 CJK/emoji 的损坏，事后恢复用户剪贴板）；桌面坐标给成物理分辨率时自动映射进截图坐标系；computer_screenshot 新增 save= 同时存盘+进剪贴板。v2.3 系列能力保留：open 先查进程表绝不启动第二实例、识别启动即退出的假成功、托盘态应用默认只报告不代启。Windows 宿主进程自动切 Per-Monitor-V2 DPI 感知，默认后台注入不抢焦点，元素树+element_token 精确操作。无需守护进程/MCP/PowerShell。写操作走原生 tools/pre-execute 审批。",
+  "description": "computer-use v2.5：Ubuntu/GNOME Wayland 自动检测并启用原生后端；内置 WinRects 扩展安装与诊断命令，驱动最低 0.28.3。基于 Cua Driver 进程内 SDK 的跨平台桌面操作。13 个工具（截图/窗口列表/缩放/光标/移动/点击/拖拽/滚动/输入/按键/打开应用/连续动作/等待）。v2.4.1 适配 dsh 0.2.0：审批探测改走公有 overrideOf/session 覆盖 + 部署默认策略，任何探测形状失效时保守放行，杜绝 danger-full-access 会话因私有 API 变动而自锁全部工具。v2.4 修「宿主闲置后窗口/应用枚举静默变空」：全部驱动调用统一挂命名 session（查询也不例外），session 卡死按 end→start→重试→重建运行时分级自愈，list_windows 报错不再吞成『无窗口』；computer_type 桌面输入遇非 ASCII 自动改走剪贴板粘贴（绕开 IME 对 CJK/emoji 的损坏，事后恢复用户剪贴板）；桌面坐标给成物理分辨率时自动映射进截图坐标系；computer_screenshot 新增 save= 同时存盘+进剪贴板。v2.3 系列能力保留：open 先查进程表绝不启动第二实例、识别启动即退出的假成功、托盘态应用默认只报告不代启。Windows 宿主进程自动切 Per-Monitor-V2 DPI 感知，默认后台注入不抢焦点，元素树+element_token 精确操作。无需守护进程/MCP/PowerShell。写操作走原生 tools/pre-execute 审批。",
   "type": "module",
   "main": "index.js",
+  "bin": { "dsh-computer-use-wayland": "./scripts/wayland.mjs" },
+  "scripts": { "test": "node --experimental-test-module-mocks --test tests/*.test.mjs" },
   "dependencies": {
-    "@trycua/cua-driver": "^0.28.1",
+    "@trycua/cua-driver": "^0.28.3",
     "koffi": "^3.2.1"
   },
   "peerDependencies": {
@@ -33,6 +35,9 @@
   },
   "files": [
     "index.js",
+    "linux.js",
+    "scripts/wayland.mjs",
+    "wayland-helper/",
     "cordis.patch.yml",
     "README.md"
   ]

package/scripts/wayland.mjs

@@ -0,0 +1,98 @@
+#!/usr/bin/env node
+import { readFile, mkdir, copyFile, realpath } from 'node:fs/promises'
+import { homedir } from 'node:os'
+import { dirname, join, resolve } from 'node:path'
+import { fileURLToPath, pathToFileURL } from 'node:url'
+import { configureLinuxDesktop, probeGnomeHelper, runDesktopCommand, HELPER_UUID } from '../linux.js'
+
+const pluginDir = dirname(dirname(fileURLToPath(import.meta.url)))
+const source = join(pluginDir, 'wayland-helper', HELPER_UUID)
+
+// gsettings prints GVariant, not JSON. Extension UUIDs cannot contain quotes
+// or escapes, so accept only its narrow string-array representation.
+export function parseEnabledExtensions(value) {
+  const text = value.trim().replace(/^@as\s+/, '')
+  if (!/^\[\s*(?:'[^'\\]*'\s*(?:,\s*'[^'\\]*'\s*)*)?\]$/.test(text)) {
+    throw new Error('Cannot parse GNOME enabled-extensions; no settings were changed: ' + value)
+  }
+  return [...text.matchAll(/'([^']*)'/g)].map(match => match[1])
+}
+
+export async function installHelper({ env = process.env, home = homedir(), run = runDesktopCommand, sourceDir = source } = {}) {
+  const version = await run('gnome-shell', ['--version'])
+  if (!version.ok) throw new Error('gnome-shell is unavailable; run this command in your GNOME desktop session.')
+  const major = /\b(\d+)(?:\.\d+)?\b/.exec(version.stdout)?.[1]
+  const metadata = JSON.parse(await readFile(join(sourceDir, 'metadata.json'), 'utf8'))
+  if (!metadata['shell-version'].includes(major)) throw new Error('This bundled helper does not support GNOME ' + major + '; supported versions: ' + metadata['shell-version'].join(', '))
+  // Read and validate settings BEFORE copying files or changing anything.
+  const current = await run('gsettings', ['get', 'org.gnome.shell', 'enabled-extensions'])
+  if (!current.ok) throw new Error('Cannot read GNOME settings: ' + current.error)
+  const enabled = parseEnabledExtensions(current.stdout)
+  const blocked = await run('gsettings', ['get', 'org.gnome.shell', 'disabled-extensions'])
+  if (!blocked.ok) throw new Error('Cannot read GNOME disabled-extensions: ' + blocked.error)
+  const disabled = parseEnabledExtensions(blocked.stdout)
+  const destination = join(env.XDG_DATA_HOME || join(home, '.local', 'share'), 'gnome-shell', 'extensions', HELPER_UUID)
+  await mkdir(destination, { recursive: true })
+  for (const file of ['extension.js', 'metadata.json']) await copyFile(join(sourceDir, file), join(destination, file))
+  if (!enabled.includes(HELPER_UUID)) {
+    enabled.push(HELPER_UUID)
+    const updated = await run('gsettings', ['set', 'org.gnome.shell', 'enabled-extensions', JSON.stringify(enabled)])
+    if (!updated.ok) throw new Error('Helper files were installed, but enabling failed: ' + updated.error)
+  }
+  // GNOME's forced-disabled list takes precedence over enabled-extensions.
+  if (disabled.includes(HELPER_UUID)) {
+    const updated = await run('gsettings', ['set', 'org.gnome.shell', 'disabled-extensions', JSON.stringify(disabled.filter(uuid => uuid !== HELPER_UUID))])
+    if (!updated.ok) throw new Error('Helper files were installed, but clearing its disabled state failed: ' + updated.error)
+  }
+  return destination
+}
+
+export async function desktopStatus({ env = process.env, platform = process.platform, run = runDesktopCommand } = {}) {
+  // Inspect a copy: a status command must not change the caller's environment.
+  const state = configureLinuxDesktop({ platform, env: { ...env } })
+  if (!state.linux || !state.wayland || !state.gnome) return { ready: false, lines: ['This setup command targets a GNOME/Wayland user session.', 'Detected: ' + platform + ' / ' + (env.XDG_SESSION_TYPE || '?') + ' / ' + (env.XDG_CURRENT_DESKTOP || '?')] }
+  const [helper, portal, accessibility, disabled] = await Promise.all([
+    probeGnomeHelper(run),
+    run('gdbus', ['call', '--session', '--dest', 'org.freedesktop.portal.Desktop', '--object-path', '/org/freedesktop/portal/desktop', '--method', 'org.freedesktop.DBus.Properties.Get', 'org.freedesktop.portal.RemoteDesktop', 'AvailableDeviceTypes']),
+    run('gdbus', ['call', '--session', '--dest', 'org.a11y.Bus', '--object-path', '/org/a11y/bus', '--method', 'org.a11y.Bus.GetAddress']),
+    run('gsettings', ['get', 'org.gnome.shell', 'disable-user-extensions']),
+  ])
+  const devices = portal.ok ? Number(/uint32\s+(\d+)/.exec(portal.stdout)?.[1] ?? 0) : 0
+  const input = (devices & 3) === 3
+  const session = !!env.WAYLAND_DISPLAY && !!env.XDG_RUNTIME_DIR && !!env.DBUS_SESSION_BUS_ADDRESS
+  return {
+    ready: state.native && session && helper.active && input && accessibility.ok && disabled.ok && disabled.stdout === 'false',
+    lines: [
+      'Wayland environment: ' + (session ? 'OK' : 'MISSING (WAYLAND_DISPLAY / XDG_RUNTIME_DIR / DBUS_SESSION_BUS_ADDRESS)'),
+      'Native Wayland backend: ' + (state.native ? 'automatic' : 'disabled by CUA_DRIVER_RS_ENABLE_WAYLAND; unset it or use config wayland: on'),
+      'WinRects helper: ' + (helper.active ? 'ACTIVE (v' + helper.version + ')' : 'MISSING/OUTDATED; run install, then log out and back in once'),
+      'GNOME user extensions: ' + (disabled.ok && disabled.stdout === 'false' ? 'enabled' : 'disabled or unreadable; enable Extensions in GNOME first'),
+      'Remote Desktop portal (keyboard + pointer): ' + (input ? 'available (consent is requested on first use)' : 'MISSING; install xdg-desktop-portal and xdg-desktop-portal-gnome'),
+      'Accessibility bus: ' + (accessibility.ok ? 'available' : 'MISSING; install at-spi2-core'),
+      'Run dsh from this same graphical user session; restart it after setup.',
+    ],
+  }
+}
+
+export async function main(args = process.argv.slice(2)) {
+  const command = args[0] ?? 'status'
+  if (!['status', 'install', '--help', '-h'].includes(command) || args.length > 1) throw new Error('Usage: dsh-computer-use-wayland [status|install]')
+  if (command === '--help' || command === '-h') { console.log('Usage: dsh-computer-use-wayland [status|install]'); return }
+  if (process.platform !== 'linux') throw new Error('Run this command on the Ubuntu/GNOME machine where dsh controls the desktop.')
+  if (command === 'install') {
+    const state = configureLinuxDesktop({ env: { ...process.env } })
+    if (!state.wayland || !state.gnome) throw new Error('Run install from your GNOME/Wayland graphical session.')
+    const destination = await installHelper()
+    console.log('Installed ' + HELPER_UUID + ' to ' + destination + '.\nLog out and back in once, restart dsh, then run dsh-computer-use-wayland status.\nOn Wayland, restarting GNOME Shell with Alt+F2 r is unavailable.')
+  } else {
+    const status = await desktopStatus()
+    console.log(status.lines.join('\n'))
+    if (!status.ready) process.exitCode = 1
+  }
+}
+
+// npm exposes bins as symlinks on Linux; compare resolved files so the command
+// also executes when launched through node_modules/.bin or npx.
+if (process.argv[1] && pathToFileURL(await realpath(resolve(process.argv[1]))).href === import.meta.url) {
+  main().catch(error => { console.error(error.message); process.exitCode = 1 })
+}

package/wayland-helper/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Cua AI, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/wayland-helper/NOTICE.md

@@ -0,0 +1,17 @@
+The unmodified `winrects@cua/extension.js` and `metadata.json` are from
+[trycua/cua](https://github.com/trycua/cua/tree/1c6932f4eb62a06a881edd1128adf5f33b09ce92/libs/cua-driver/wayland-helper),
+tag `cua-driver-rs-v0.28.4`, commit `1c6932f4eb62a06a881edd1128adf5f33b09ce92`.
+Copyright (c) 2025 Cua AI, Inc.; MIT license, reproduced in LICENSE.md.
+
+Helper API v8 supports GNOME Shell 45–50. This helper version includes the
+GNOME stage screenshot fix and is compatible with the 0.28.3 npm runtime.
+The helper is bundled because the npm SDK distributes native bindings but
+does not include the GNOME Shell extension shipped in the upstream CLI.
+
+Install it explicitly with `dsh-computer-use-wayland install` (no root needed),
+then log out and back in once. It supplies compositor geometry, capture and
+activation. Input uses the driver's Remote Desktop portal and its system
+consent dialog. Installation does not grant portal input permission.
+
+To remove it, run `gnome-extensions disable winrects@cua`, delete only the
+`winrects@cua` directory reported by the installer, and log out and back in.