dsh-desktop-control 0.0.0-stage → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 qingying012
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,97 @@
1
- # Temporary Holding Version
1
+ # dsh-desktop-control
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ A [DeepSeek Harness](https://github.com/deepseek-ai) plugin that gives the model a native
4
+ `desktop` tool: **screenshot the Windows screen, move/click the mouse, type text and key combos**.
5
+
6
+ It wraps the battle-tested `gui.ps1` script (screenshot + Win32 input) as a DSH plugin, so the
7
+ model calls a typed tool instead of assembling PowerShell commands every time.
8
+
9
+ > Windows only. The package ships its own `assets/gui.ps1` (UTF-8 with BOM); it does not depend on
10
+ > a workspace script.
11
+
12
+ ## Actions
13
+
14
+ | action | What it does | Main args |
15
+ | :-- | :-- | :-- |
16
+ | `screenshot` | Save a PNG and return its absolute path | `window` / `region` / `print` |
17
+ | `windows` | List visible windows (HWND, PID, rect, title) | — |
18
+ | `click` | Move and click | `x` `y` `button` `count` |
19
+ | `move` | Move the cursor | `x` `y` |
20
+ | `drag` | Press, move, release | `x` `y` `x2` `y2` |
21
+ | `scroll` | Wheel | `x` `y` `amount` |
22
+ | `type` | Paste text via the clipboard | `text` |
23
+ | `key` | Key combo, e.g. `ctrl+s`, `alt+f4` | `key` |
24
+ | `activate` | Bring a window to the front | `window` |
25
+ | `pos` | Cursor position | — |
26
+ | `wait` | Sleep | `waitMs` |
27
+
28
+ **All coordinates are physical pixels**, matching the saved screenshot one-to-one (the script
29
+ enables per-monitor DPI awareness v2).
30
+
31
+ Typical loop:
32
+
33
+ ```text
34
+ screenshot -> read_image -> click / type -> screenshot again
35
+ ```
36
+
37
+ ## Install
38
+
39
+ Client (Electron): **Plugins → + Add plugin**, paste `dsh-desktop-control`, install, then
40
+ **Enable now**.
41
+
42
+ CLI profile:
43
+
44
+ ```powershell
45
+ dsh plugin --profile web add dsh-desktop-control
46
+ ```
47
+
48
+ ## Configure
49
+
50
+ ```yaml
51
+ - id: dsh-desktop-control
52
+ name: dsh-desktop-control
53
+ config:
54
+ enabled: true
55
+ powershellPath: powershell.exe
56
+ timeoutMs: 30000
57
+ screenshotDir: '' # empty = $DSH_HOME/desktop-control
58
+ ```
59
+
60
+ | Option | Default | Meaning |
61
+ | :-- | :-- | :-- |
62
+ | `enabled` | `true` | `false` registers no tool |
63
+ | `powershellPath` | `powershell.exe` | PowerShell executable |
64
+ | `timeoutMs` | `30000` | Per-call timeout |
65
+ | `screenshotDir` | `$DSH_HOME/desktop-control` | Where screenshots are saved |
66
+
67
+ ## Why a plugin instead of the pwsh tool
68
+
69
+ The DSH pwsh tool runs children at Low integrity, so `SetCursorPos` / `SendInput` are often
70
+ refused and the old script needed `danger-full-access` per approval. This plugin spawns
71
+ PowerShell from the host process, outside that sandbox, so input usually works without switching
72
+ the permission preset.
73
+
74
+ ## Safety
75
+
76
+ - Full desktop control: it can click anything and type anything, and screenshots enter the
77
+ model context once you open them with `read_image`.
78
+ - Do not enable it on an unattended machine.
79
+ - UAC secure desktop, the lock screen, and some games/anti-cheat/terminal apps cannot be driven.
80
+
81
+ ## Self-test
82
+
83
+ ```powershell
84
+ npm run selftest
85
+ ```
86
+
87
+ Pure argument-construction tests: no screenshot, no mouse movement.
88
+
89
+ ## Limitations
90
+
91
+ - Windows only.
92
+ - One action per call; each call spawns PowerShell (~300–600 ms).
93
+ - Static screenshots; the cursor marker is not exposed by the tool.
94
+
95
+ ## License
96
+
97
+ MIT
package/README.zh.md ADDED
@@ -0,0 +1,103 @@
1
+ # dsh-desktop-control
2
+
3
+ 一个 [DeepSeek Harness](https://github.com/deepseek-ai) 插件:给模型一个原生的 `desktop` 工具,**截屏看屏幕、移动/点击鼠标、输入文字和组合键**。
4
+
5
+ 它把之前那套 `gui.ps1` 脚本(截图 + 鼠标键盘控制)包进 DSH 插件,模型直接调用工具,不用每次手拼 PowerShell 命令、也不用担心执行策略。
6
+
7
+ > Windows only。插件包内自带 `assets/gui.ps1`(UTF-8 BOM),不依赖工作区里的脚本。
8
+
9
+ ## 工具动作
10
+
11
+ | action | 作用 | 主要参数 |
12
+ | :-- | :-- | :-- |
13
+ | `screenshot` | 截图并返回 PNG 绝对路径 | `window` / `region` / `print` |
14
+ | `windows` | 列出可见窗口(HWND、PID、矩形、标题) | — |
15
+ | `click` | 移动并点击 | `x` `y` `button` `count` |
16
+ | `move` | 移动光标 | `x` `y` |
17
+ | `drag` | 按住拖拽 | `x` `y` `x2` `y2` |
18
+ | `scroll` | 滚轮 | `x` `y` `amount` |
19
+ | `type` | 粘贴文本(走剪贴板,Unicode 安全) | `text` |
20
+ | `key` | 组合键,如 `ctrl+s`、`alt+f4` | `key` |
21
+ | `activate` | 把窗口提到前台 | `window` |
22
+ | `pos` | 光标当前位置 | — |
23
+ | `wait` | 等待 | `waitMs` |
24
+
25
+ **坐标全部是物理像素**,与 `screenshot` 保存的图片像素一一对应。脚本内部已开启 per-monitor DPI aware v2,图上量到多少就传多少。
26
+
27
+ 典型循环:
28
+
29
+ ```text
30
+ screenshot → read_image 看图 → click / type → 再 screenshot 确认
31
+ ```
32
+
33
+ ## 安装
34
+
35
+ 客户端(Electron):**插件 → + 添加插件**,填入 `dsh-desktop-control`,安装后 **立即启用**。
36
+
37
+ CLI profile:
38
+
39
+ ```powershell
40
+ dsh plugin --profile web add dsh-desktop-control
41
+ ```
42
+
43
+ ## 配置
44
+
45
+ 在 profile 的 `cordis.patch.yml` 里:
46
+
47
+ ```yaml
48
+ - id: dsh-desktop-control
49
+ name: dsh-desktop-control
50
+ config:
51
+ enabled: true
52
+ powershellPath: powershell.exe # 不在 PATH 时再改
53
+ timeoutMs: 30000
54
+ screenshotDir: '' # 空 = $DSH_HOME/desktop-control
55
+ ```
56
+
57
+ | 选项 | 默认值 | 说明 |
58
+ | :-- | :-- | :-- |
59
+ | `enabled` | `true` | 设为 `false` 时不注册工具 |
60
+ | `powershellPath` | `powershell.exe` | PowerShell 可执行文件 |
61
+ | `timeoutMs` | `30000` | 单次调用超时 |
62
+ | `screenshotDir` | `$DSH_HOME/desktop-control` | 截图落盘目录 |
63
+
64
+ ## 和直接跑 `gui.ps1` 的区别
65
+
66
+ | | `gui.ps1` + pwsh 工具 | 本插件 |
67
+ | :-- | :-- | :-- |
68
+ | 调用方式 | 每次拼命令行 | 原生工具调用,参数结构化 |
69
+ | 执行策略 | 受 Restricted 影响,要绕 | 插件内部固定 `-ExecutionPolicy Bypass` |
70
+ | 沙箱降权 | DSH 的 pwsh 子进程会被降为 Low 完整性,鼠标键盘常被拒 | 插件在宿主进程内 spawn,不经过 pwsh 沙箱 |
71
+ | 截图路径 | 自己管理 | 自动落盘到 DSH home 并返回绝对路径 |
72
+
73
+ > 也就是说:**用插件通常不需要切 `danger-full-access`**,这是它比脚本顺手的地方。
74
+
75
+ ## 安全提醒
76
+
77
+ - 这是完整的桌面控制:能点任何按钮、能敲任何键、能截下屏幕上的任何内容。
78
+ - 截图会进入模型上下文(你随后用 `read_image` 读它),涉密画面先关掉再截。
79
+ - 工具调用受宿主工具策略约束;但不要在无人值守的机器上启用。
80
+ - UAC 安全桌面、登录界面、锁屏无法操作;部分游戏/反作弊/终端会屏蔽模拟输入。
81
+
82
+ ## 自测
83
+
84
+ ```powershell
85
+ npm run selftest
86
+ ```
87
+
88
+ 只测参数构建和校验(不截图、不动鼠标):
89
+
90
+ - `screenshot` 生成绝对 `.png` 路径;
91
+ - `type` 对以 `-` 开头的文本使用 `--` 保护;
92
+ - `click` 缺坐标、未知 action 会报错;
93
+ - `scroll` 两种形式、`move`/`drag`/`key`/`activate`/`pos`/`wait` 的 argv 正确。
94
+
95
+ ## 已知限制
96
+
97
+ - 仅 Windows(依赖 PowerShell + user32/gdi32)。
98
+ - 每次调用起一个 PowerShell 进程,约 300–600ms 开销;高频操作建议一次调用串多个动作(当前工具一次只做一个动作)。
99
+ - 截图是整屏/窗口/区域的静态图,不包含光标(除非脚本带 `-cursor`,本工具暂未暴露)。
100
+
101
+ ## License
102
+
103
+ MIT
package/assets/gui.ps1 ADDED
@@ -0,0 +1,573 @@
1
+ <#
2
+ gui.ps1 — 屏幕读取与鼠标键盘控制(Windows / PowerShell 5.1+)
3
+
4
+ 用法示例:
5
+ .\gui.ps1 windows 列出可见顶层窗口
6
+ .\gui.ps1 shot out.png 截全屏(虚拟桌面)
7
+ .\gui.ps1 shot out.png -window "记事本" 截指定窗口
8
+ .\gui.ps1 shot out.png -region 0,0,800,600
9
+ .\gui.ps1 shot out.png -cursor 截图并标出光标位置
10
+ .\gui.ps1 pos 打印光标当前坐标
11
+ .\gui.ps1 move 800 500 移动光标
12
+ .\gui.ps1 click 800 500 左键单击
13
+ .\gui.ps1 click 800 500 -button right -count 2
14
+ .\gui.ps1 drag 100 100 400 300 左键拖拽
15
+ .\gui.ps1 scroll 800 500 3 滚轮(正=上,负=下,1=120单位)
16
+ .\gui.ps1 type "hello 中文" 逐字符 Unicode 输入
17
+ .\gui.ps1 key "ctrl+s" 组合键
18
+ .\gui.ps1 key "enter"
19
+ .\gui.ps1 activate "DeepSeek" 激活窗口到前台
20
+
21
+ 约定:所有坐标都是物理像素(脚本内部开启 per-monitor DPI v2),
22
+ 与 `shot` 输出的图片像素一一对应,可放心把图上的像素坐标直接当参数用。
23
+ #>
24
+
25
+ [CmdletBinding()]
26
+ param(
27
+ [Parameter(ValueFromRemainingArguments = $true)]
28
+ [object[]] $Argv
29
+ )
30
+
31
+ [Console]::OutputEncoding = [System.Text.Encoding]::UTF8
32
+ $ErrorActionPreference = 'Stop'
33
+
34
+ if (-not ('GuiNative' -as [type])) {
35
+ Add-Type -AssemblyName System.Windows.Forms, System.Drawing
36
+ Add-Type -TypeDefinition @'
37
+ using System;
38
+ using System.Collections.Generic;
39
+ using System.Diagnostics;
40
+ using System.Runtime.InteropServices;
41
+ using System.Text;
42
+
43
+ public class GuiNative
44
+ {
45
+ [StructLayout(LayoutKind.Sequential)]
46
+ public struct POINT { public int X; public int Y; }
47
+
48
+ [StructLayout(LayoutKind.Sequential)]
49
+ public struct RECT { public int Left; public int Top; public int Right; public int Bottom; }
50
+
51
+ [StructLayout(LayoutKind.Sequential)]
52
+ public struct MOUSEINPUT { public int dx; public int dy; public uint mouseData; public uint dwFlags; public uint time; public IntPtr dwExtraInfo; }
53
+
54
+ [StructLayout(LayoutKind.Sequential)]
55
+ public struct KEYBDINPUT { public ushort wVk; public ushort wScan; public uint dwFlags; public uint time; public IntPtr dwExtraInfo; }
56
+
57
+ [StructLayout(LayoutKind.Explicit)]
58
+ public struct INPUT
59
+ {
60
+ [FieldOffset(0)] public uint type;
61
+ [FieldOffset(4)] public MOUSEINPUT mi;
62
+ [FieldOffset(4)] public KEYBDINPUT ki;
63
+ }
64
+
65
+ public delegate bool EnumProc(IntPtr hWnd, IntPtr lParam);
66
+
67
+ [DllImport("user32.dll")] public static extern bool SetCursorPos(int X, int Y);
68
+ [DllImport("user32.dll")] public static extern bool GetCursorPos(out POINT p);
69
+ [DllImport("user32.dll")] public static extern int GetSystemMetrics(int nIndex);
70
+ [DllImport("user32.dll")] public static extern IntPtr GetForegroundWindow();
71
+ [DllImport("user32.dll", CharSet = CharSet.Unicode)] public static extern int GetWindowText(IntPtr h, StringBuilder s, int n);
72
+ [DllImport("user32.dll")] public static extern bool IsWindowVisible(IntPtr h);
73
+ [DllImport("user32.dll")] public static extern bool GetWindowRect(IntPtr h, out RECT r);
74
+ [DllImport("user32.dll")] public static extern bool EnumWindows(EnumProc cb, IntPtr p);
75
+ [DllImport("user32.dll")] public static extern uint GetWindowThreadProcessId(IntPtr h, out uint pid);
76
+ [DllImport("user32.dll")] public static extern bool SetForegroundWindow(IntPtr h);
77
+ [DllImport("user32.dll")] public static extern bool BringWindowToTop(IntPtr h);
78
+ [DllImport("user32.dll")] public static extern bool ShowWindow(IntPtr h, int cmd);
79
+ [DllImport("user32.dll")] public static extern bool IsIconic(IntPtr h);
80
+ [DllImport("user32.dll")] public static extern bool AttachThreadInput(uint a, uint b, bool attach);
81
+ [DllImport("kernel32.dll")] public static extern uint GetCurrentThreadId();
82
+ [DllImport("user32.dll")] public static extern bool PrintWindow(IntPtr h, IntPtr hdc, uint flags);
83
+ [DllImport("user32.dll")] public static extern bool SetProcessDpiAwarenessContext(IntPtr v);
84
+
85
+ const uint MOUSEEVENTF_MOVE = 0x0001;
86
+ const uint MOUSEEVENTF_LEFTDOWN = 0x0002;
87
+ const uint MOUSEEVENTF_LEFTUP = 0x0004;
88
+ const uint MOUSEEVENTF_RIGHTDOWN = 0x0008;
89
+ const uint MOUSEEVENTF_RIGHTUP = 0x0010;
90
+ const uint MOUSEEVENTF_MIDDLEDOWN = 0x0020;
91
+ const uint MOUSEEVENTF_MIDDLEUP = 0x0040;
92
+ const uint MOUSEEVENTF_WHEEL = 0x0800;
93
+ const uint MOUSEEVENTF_ABSOLUTE = 0x8000;
94
+ const uint MOUSEEVENTF_VIRTUALDESK = 0x4000;
95
+ const uint KEYEVENTF_KEYUP = 0x0002;
96
+ const uint KEYEVENTF_UNICODE = 0x0004;
97
+
98
+ public static void EnableDpiAwareness()
99
+ {
100
+ try { SetProcessDpiAwarenessContext(new IntPtr(-4)); } catch { }
101
+ }
102
+
103
+ static INPUT MouseInput(uint flags, int dx, int dy, uint data)
104
+ {
105
+ INPUT i = new INPUT();
106
+ i.type = 0;
107
+ i.mi.dx = dx;
108
+ i.mi.dy = dy;
109
+ i.mi.mouseData = data;
110
+ i.mi.dwFlags = flags;
111
+ return i;
112
+ }
113
+
114
+ static INPUT KeyInput(ushort vk, ushort scan, uint flags)
115
+ {
116
+ INPUT i = new INPUT();
117
+ i.type = 1;
118
+ i.ki.wVk = vk;
119
+ i.ki.wScan = scan;
120
+ i.ki.dwFlags = flags;
121
+ return i;
122
+ }
123
+
124
+ [DllImport("user32.dll", SetLastError = true, EntryPoint = "SendInput")]
125
+ static extern uint SendRaw(uint nInputs, IntPtr pInputs, int cbSize);
126
+ [DllImport("user32.dll")]
127
+ static extern void mouse_event(uint dwFlags, int dx, int dy, uint dwData, UIntPtr dwExtraInfo);
128
+ [DllImport("user32.dll")]
129
+ static extern void keybd_event(byte bVk, byte bScan, uint dwFlags, UIntPtr dwExtraInfo);
130
+
131
+ static int Cb() { return Marshal.SizeOf(typeof(INPUT)); }
132
+
133
+ static uint SendAll(INPUT[] arr)
134
+ {
135
+ int size = Cb();
136
+ IntPtr buf = Marshal.AllocHGlobal(size * arr.Length);
137
+ try
138
+ {
139
+ for (int i = 0; i < arr.Length; i++)
140
+ Marshal.StructureToPtr(arr[i], new IntPtr(buf.ToInt64() + (long)i * size), false);
141
+ return SendRaw((uint)arr.Length, buf, size);
142
+ }
143
+ finally { Marshal.FreeHGlobal(buf); }
144
+ }
145
+
146
+ public static bool MoveTo(int x, int y)
147
+ {
148
+ if (SetCursorPos(x, y))
149
+ {
150
+ POINT p;
151
+ if (GetCursorPos(out p) && Math.Abs(p.X - x) <= 2 && Math.Abs(p.Y - y) <= 2) return true;
152
+ }
153
+ int vx = GetSystemMetrics(76), vy = GetSystemMetrics(77);
154
+ int vw = GetSystemMetrics(78), vh = GetSystemMetrics(79);
155
+ if (vw <= 1) vw = 2;
156
+ if (vh <= 1) vh = 2;
157
+ int ax = (int)Math.Round((double)(x - vx) * 65535.0 / (vw - 1));
158
+ int ay = (int)Math.Round((double)(y - vy) * 65535.0 / (vh - 1));
159
+ mouse_event(MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE, ax, ay, 0, UIntPtr.Zero);
160
+ POINT q;
161
+ return GetCursorPos(out q) && Math.Abs(q.X - x) <= 2 && Math.Abs(q.Y - y) <= 2;
162
+ }
163
+
164
+ public static void MouseButton(string button, bool down)
165
+ {
166
+ string b = (button ?? "left").ToLowerInvariant();
167
+ uint mf;
168
+ if (b.StartsWith("r")) mf = down ? 0x0008u : 0x0010u;
169
+ else if (b.StartsWith("m")) mf = down ? 0x0020u : 0x0040u;
170
+ else mf = down ? 0x0002u : 0x0004u;
171
+ mouse_event(mf, 0, 0, 0, UIntPtr.Zero);
172
+ }
173
+
174
+ public static void Scroll(int notches)
175
+ {
176
+ uint data = (uint)(int)(notches * 120);
177
+ mouse_event(MOUSEEVENTF_WHEEL, 0, 0, data, UIntPtr.Zero);
178
+ }
179
+
180
+ public static void KeyDown(ushort vk) { keybd_event((byte)vk, 0, 0, UIntPtr.Zero); }
181
+ public static void KeyUp(ushort vk) { keybd_event((byte)vk, 0, KEYEVENTF_KEYUP, UIntPtr.Zero); }
182
+
183
+ public static void TypeText(string s)
184
+ {
185
+ if (string.IsNullOrEmpty(s)) return;
186
+ List<INPUT> list = new List<INPUT>();
187
+ foreach (char c in s)
188
+ {
189
+ ushort scan = (ushort)c;
190
+ list.Add(KeyInput(0, scan, KEYEVENTF_UNICODE));
191
+ list.Add(KeyInput(0, scan, KEYEVENTF_UNICODE | KEYEVENTF_KEYUP));
192
+ }
193
+ SendAll(list.ToArray());
194
+ }
195
+
196
+ public static POINT Cursor()
197
+ {
198
+ POINT p;
199
+ GetCursorPos(out p);
200
+ return p;
201
+ }
202
+
203
+ public static string[] ListWindows()
204
+ {
205
+ List<string> rows = new List<string>();
206
+ EnumWindows(delegate(IntPtr h, IntPtr l)
207
+ {
208
+ if (!IsWindowVisible(h)) return true;
209
+ StringBuilder sb = new StringBuilder(1024);
210
+ GetWindowText(h, sb, sb.Capacity);
211
+ string title = sb.ToString();
212
+ if (title.Trim().Length == 0) return true;
213
+ RECT r;
214
+ if (!GetWindowRect(h, out r)) return true;
215
+ int w = r.Right - r.Left, ht = r.Bottom - r.Top;
216
+ if (w < 40 || ht < 40) return true;
217
+ uint pid;
218
+ GetWindowThreadProcessId(h, out pid);
219
+ string proc = "";
220
+ try { proc = Process.GetProcessById((int)pid).ProcessName; } catch { }
221
+ rows.Add(string.Format("{0}\t{1}\t{2}\t{3}\t{4}\t{5}\t{6}\t{7}",
222
+ h.ToInt64(), title, r.Left, r.Top, w, ht, pid, proc));
223
+ return true;
224
+ }, IntPtr.Zero);
225
+ return rows.ToArray();
226
+ }
227
+
228
+ public static IntPtr FindWindow(string needle)
229
+ {
230
+ if (string.IsNullOrEmpty(needle)) return IntPtr.Zero;
231
+ string n = needle.ToLowerInvariant();
232
+ IntPtr best = IntPtr.Zero;
233
+ int bestScore = 0;
234
+ foreach (string row in ListWindows())
235
+ {
236
+ string[] p = row.Split('\t');
237
+ if (p.Length < 8) continue;
238
+ string title = p[1].ToLowerInvariant();
239
+ string proc = p[7].ToLowerInvariant();
240
+ int score = 0;
241
+ if (title == n) score = 100;
242
+ else if (title.StartsWith(n)) score = 80;
243
+ else if (title.Contains(n)) score = 60;
244
+ else if (proc == n) score = 50;
245
+ else if (proc.Contains(n)) score = 40;
246
+ if (score > bestScore)
247
+ {
248
+ bestScore = score;
249
+ best = new IntPtr(long.Parse(p[0]));
250
+ }
251
+ }
252
+ return best;
253
+ }
254
+
255
+ public static bool ForceForeground(IntPtr h)
256
+ {
257
+ if (h == IntPtr.Zero) return false;
258
+ if (IsIconic(h)) ShowWindow(h, 9);
259
+ uint pid;
260
+ uint tid = GetWindowThreadProcessId(h, out pid);
261
+ uint cur = GetCurrentThreadId();
262
+ bool ok;
263
+ if (tid != cur)
264
+ {
265
+ AttachThreadInput(cur, tid, true);
266
+ BringWindowToTop(h);
267
+ ok = SetForegroundWindow(h);
268
+ AttachThreadInput(cur, tid, false);
269
+ }
270
+ else
271
+ {
272
+ BringWindowToTop(h);
273
+ ok = SetForegroundWindow(h);
274
+ }
275
+ if (!ok) { ShowWindow(h, 9); ShowWindow(h, 5); ok = SetForegroundWindow(h); }
276
+ return ok;
277
+ }
278
+
279
+ public static bool GetRect(IntPtr h, out int left, out int top, out int width, out int height)
280
+ {
281
+ RECT r;
282
+ left = top = width = height = 0;
283
+ if (h == IntPtr.Zero || !GetWindowRect(h, out r)) return false;
284
+ left = r.Left; top = r.Top;
285
+ width = r.Right - r.Left; height = r.Bottom - r.Top;
286
+ return true;
287
+ }
288
+ }
289
+ '@
290
+ }
291
+
292
+ [GuiNative]::EnableDpiAwareness()
293
+
294
+ # ---- 参数解析:位置参数 + -name value ----
295
+ $pos = New-Object System.Collections.ArrayList
296
+ $opt = @{}
297
+ $literal = $false
298
+ for ($i = 0; $i -lt $Argv.Count; $i++) {
299
+ $a = $Argv[$i]
300
+ if (-not $literal -and $a -is [string] -and $a -eq '--') { $literal = $true; continue }
301
+ if (-not $literal -and $a -is [string] -and $a.Length -gt 1 -and $a.StartsWith('-') -and $a -notmatch '^-?\d') {
302
+ $name = $a.TrimStart('-').ToLowerInvariant()
303
+ $val = $null
304
+ if ($i + 1 -lt $Argv.Count) {
305
+ $nxt = $Argv[$i + 1]
306
+ if (-not ($nxt -is [string] -and $nxt.Length -gt 1 -and $nxt.StartsWith('-') -and $nxt -notmatch '^-?\d')) {
307
+ $val = $nxt; $i++
308
+ }
309
+ }
310
+ if ($null -eq $val) { $opt[$name] = $true } else { $opt[$name] = $val }
311
+ }
312
+ else { [void]$pos.Add($a) }
313
+ }
314
+
315
+ if ($pos.Count -eq 0) { Get-Help $PSCommandPath -ErrorAction SilentlyContinue; Write-Host "`n没有参数。用 'shot out.png' 或 'windows' 开始。" ; return }
316
+
317
+ $cmd = ([string]$pos[0]).ToLowerInvariant()
318
+ $rest = @()
319
+ if ($pos.Count -gt 1) { $rest = $pos[1..($pos.Count - 1)] }
320
+
321
+ function Get-RectFromString([string]$s) {
322
+ $p = $s -split ','
323
+ if ($p.Count -ne 4) { throw "region 需要 x,y,w,h 四段,例如 0,0,800,600" }
324
+ @{ X = [int]$p[0]; Y = [int]$p[1]; W = [int]$p[2]; H = [int]$p[3] }
325
+ }
326
+
327
+ switch ($cmd) {
328
+
329
+ 'windows' {
330
+ $rows = [GuiNative]::ListWindows()
331
+ "{0,-14} {1,-10} {2,-26} {3}" -f 'HWND', 'PID', 'RECT', 'TITLE'
332
+ foreach ($row in $rows) {
333
+ $p = $row -split "`t"
334
+ $hw = [long]$p[0]
335
+ "{0,-14} {1,-10} {2,-26} {3}" -f ('0x' + $hw.ToString('X')), $p[6], ("{0},{1} {2}x{3}" -f $p[2], $p[3], $p[4], $p[5]), ($p[1] + " [" + $p[7] + "]")
336
+ }
337
+ }
338
+
339
+ 'pos' {
340
+ $pt = [GuiNative]::Cursor()
341
+ "{0},{1}" -f $pt.X, $pt.Y
342
+ }
343
+
344
+ 'fg' {
345
+ $h = [GuiNative]::GetForegroundWindow()
346
+ $sb = New-Object System.Text.StringBuilder 512
347
+ [void][GuiNative]::GetWindowText($h, $sb, 512)
348
+ [pscustomobject]@{ hwnd = ('0x' + $h.ToInt64().ToString('X')); title = $sb.ToString() } | ConvertTo-Json -Compress
349
+ }
350
+
351
+ 'shot' {
352
+ $out = if ($rest.Count -ge 1) { [string]$rest[0] } else { 'shot.png' }
353
+ if (-not [System.IO.Path]::IsPathRooted($out)) { $out = Join-Path (Get-Location).Path $out }
354
+ $dir = [System.IO.Path]::GetDirectoryName($out)
355
+ if ($dir -and -not (Test-Path $dir)) { New-Item -ItemType Directory -Force -Path $dir | Out-Null }
356
+
357
+ $hwnd = [IntPtr]::Zero
358
+ if ($opt.ContainsKey('window')) { $hwnd = [GuiNative]::FindWindow([string]$opt['window']) }
359
+ elseif ($opt.ContainsKey('hwnd')) { $hwnd = [IntPtr]([long]$opt['hwnd']) }
360
+
361
+ if ($hwnd -ne [IntPtr]::Zero) {
362
+ $l = 0; $t = 0; $w = 0; $h = 0
363
+ if (-not [GuiNative]::GetRect($hwnd, [ref]$l, [ref]$t, [ref]$w, [ref]$h)) { throw "拿不到窗口矩形" }
364
+ $x = $l; $y = $t; $width = $w; $height = $h
365
+ }
366
+ elseif ($opt.ContainsKey('region')) {
367
+ $r = Get-RectFromString ([string]$opt['region'])
368
+ $x = $r.X; $y = $r.Y; $width = $r.W; $height = $r.H
369
+ }
370
+ else {
371
+ $x = [GuiNative]::GetSystemMetrics(76)
372
+ $y = [GuiNative]::GetSystemMetrics(77)
373
+ $width = [GuiNative]::GetSystemMetrics(78)
374
+ $height = [GuiNative]::GetSystemMetrics(79)
375
+ }
376
+ if ($width -le 0 -or $height -le 0) { throw "截图区域无效: ${width}x${height}" }
377
+
378
+ $bmp = New-Object System.Drawing.Bitmap $width, $height
379
+ try {
380
+ $g = [System.Drawing.Graphics]::FromImage($bmp)
381
+ try {
382
+ if ($hwnd -ne [IntPtr]::Zero -and $opt.ContainsKey('print')) {
383
+ $hdc = $g.GetHdc()
384
+ try { [void][GuiNative]::PrintWindow($hwnd, $hdc, 2) }
385
+ finally { $g.ReleaseHdc($hdc) }
386
+ }
387
+ else {
388
+ $g.CopyFromScreen($x, $y, 0, 0, (New-Object System.Drawing.Size $width, $height))
389
+ }
390
+ }
391
+ finally { $g.Dispose() }
392
+
393
+ if ($opt.ContainsKey('cursor')) {
394
+ $pt = [GuiNative]::Cursor()
395
+ $cx = $pt.X - $x; $cy = $pt.Y - $y
396
+ if ($cx -ge 0 -and $cy -ge 0 -and $cx -lt $width -and $cy -lt $height) {
397
+ $g2 = [System.Drawing.Graphics]::FromImage($bmp)
398
+ try {
399
+ $pen = New-Object System.Drawing.Pen ([System.Drawing.Color]::Red), 3
400
+ $g2.DrawEllipse($pen, $cx - 14, $cy - 14, 28, 28)
401
+ $g2.DrawLine($pen, $cx - 22, $cy, $cx + 22, $cy)
402
+ $g2.DrawLine($pen, $cx, $cy - 22, $cx, $cy + 22)
403
+ $pen.Dispose()
404
+ }
405
+ finally { $g2.Dispose() }
406
+ }
407
+ }
408
+
409
+ $bmp.Save($out, [System.Drawing.Imaging.ImageFormat]::Png)
410
+ }
411
+ finally { $bmp.Dispose() }
412
+
413
+ [pscustomobject]@{
414
+ path = $out
415
+ x = $x
416
+ y = $y
417
+ width = $width
418
+ height = $height
419
+ bytes = (Get-Item $out).Length
420
+ } | ConvertTo-Json -Compress
421
+ }
422
+
423
+ 'activate' {
424
+ if ($rest.Count -lt 1) { throw "activate 需要窗口标题关键字" }
425
+ $h = [GuiNative]::FindWindow([string]$rest[0])
426
+ if ($h -eq [IntPtr]::Zero) { throw "找不到窗口: $($rest[0])" }
427
+ $ok = [GuiNative]::ForceForeground($h)
428
+ [pscustomobject]@{ ok = $ok; hwnd = ('0x' + $h.ToInt64().ToString('X')) } | ConvertTo-Json -Compress
429
+ }
430
+
431
+ 'move' {
432
+ if ($rest.Count -lt 2) { throw "move 需要 x y" }
433
+ $x = [int]$rest[0]; $y = [int]$rest[1]
434
+ [GuiNative]::MoveTo($x, $y)
435
+ Start-Sleep -Milliseconds 40
436
+ $pt = [GuiNative]::Cursor()
437
+ "{0},{1}" -f $pt.X, $pt.Y
438
+ }
439
+
440
+ 'click' {
441
+ if ($rest.Count -ge 2) {
442
+ [GuiNative]::MoveTo([int]$rest[0], [int]$rest[1])
443
+ Start-Sleep -Milliseconds 60
444
+ }
445
+ $button = if ($opt.ContainsKey('button')) { [string]$opt['button'] } else { 'left' }
446
+ $count = if ($opt.ContainsKey('count')) { [int]$opt['count'] } else { 1 }
447
+ $gap = if ($opt.ContainsKey('gap')) { [int]$opt['gap'] } else { 90 }
448
+ for ($k = 0; $k -lt $count; $k++) {
449
+ [GuiNative]::MouseButton($button, $true)
450
+ Start-Sleep -Milliseconds 40
451
+ [GuiNative]::MouseButton($button, $false)
452
+ if ($k -lt $count - 1) { Start-Sleep -Milliseconds $gap }
453
+ }
454
+ $pt = [GuiNative]::Cursor()
455
+ "click {0} x{1} @ {2},{3}" -f $button, $count, $pt.X, $pt.Y
456
+ }
457
+
458
+ 'drag' {
459
+ if ($rest.Count -lt 4) { throw "drag 需要 x1 y1 x2 y2" }
460
+ $button = if ($opt.ContainsKey('button')) { [string]$opt['button'] } else { 'left' }
461
+ [GuiNative]::MoveTo([int]$rest[0], [int]$rest[1])
462
+ Start-Sleep -Milliseconds 60
463
+ [GuiNative]::MouseButton($button, $true)
464
+ Start-Sleep -Milliseconds 80
465
+ $x1 = [int]$rest[0]; $y1 = [int]$rest[1]; $x2 = [int]$rest[2]; $y2 = [int]$rest[3]
466
+ for ($s = 1; $s -le 10; $s++) {
467
+ $mx = [int]($x1 + ($x2 - $x1) * $s / 10.0)
468
+ $my = [int]($y1 + ($y2 - $y1) * $s / 10.0)
469
+ [GuiNative]::MoveTo($mx, $my)
470
+ Start-Sleep -Milliseconds 15
471
+ }
472
+ Start-Sleep -Milliseconds 80
473
+ [GuiNative]::MouseButton($button, $false)
474
+ "drag {0} ({1},{2}) -> ({3},{4})" -f $button, $x1, $y1, $x2, $y2
475
+ }
476
+
477
+ 'scroll' {
478
+ if ($rest.Count -ge 3) { [GuiNative]::MoveTo([int]$rest[0], [int]$rest[1]); Start-Sleep -Milliseconds 40 }
479
+ $d = if ($rest.Count -ge 3) { [int]$rest[2] } else { [int]$rest[0] }
480
+ [GuiNative]::Scroll($d)
481
+ "scroll $d"
482
+ }
483
+
484
+ 'type' {
485
+ $text = [string]::Join(' ', ($rest | ForEach-Object { [string]$_ }))
486
+ if ($text -eq '') { throw "type 需要文本" }
487
+ $text = $text -replace '\\n', "`n" -replace '\\t', "`t"
488
+
489
+ if ($opt.ContainsKey('raw')) {
490
+ [GuiNative]::TypeText($text)
491
+ "typed {0} chars (raw)" -f $text.Length
492
+ break
493
+ }
494
+
495
+ # 默认走剪贴板粘贴:对中文/任意 Unicode 都可靠,且结束后还原原剪贴板
496
+ $old = $null
497
+ try { $old = Get-Clipboard -Raw -ErrorAction Stop } catch { }
498
+ try { Set-Clipboard -Value $text -ErrorAction Stop }
499
+ catch { [System.Windows.Forms.Clipboard]::SetText($text) }
500
+ Start-Sleep -Milliseconds 120
501
+
502
+ [GuiNative]::KeyDown(0x11)
503
+ Start-Sleep -Milliseconds 30
504
+ [GuiNative]::KeyDown(0x56)
505
+ Start-Sleep -Milliseconds 30
506
+ [GuiNative]::KeyUp(0x56)
507
+ Start-Sleep -Milliseconds 30
508
+ [GuiNative]::KeyUp(0x11)
509
+ Start-Sleep -Milliseconds 180
510
+
511
+ if ($null -ne $old -and $old -ne '') {
512
+ try { Set-Clipboard -Value $old -ErrorAction Stop }
513
+ catch { try { [System.Windows.Forms.Clipboard]::SetText($old) } catch { } }
514
+ }
515
+ "pasted {0} chars" -f $text.Length
516
+ }
517
+
518
+ 'key' {
519
+ if ($rest.Count -lt 1) { throw "key 需要组合键,例如 ctrl+s" }
520
+ $combo = [string]::Join('+', ($rest | ForEach-Object { [string]$_ }))
521
+
522
+ $map = @{
523
+ 'ctrl' = 0x11; 'control' = 0x11; 'shift' = 0x10; 'alt' = 0x12; 'win' = 0x5B; 'lwin' = 0x5B; 'rwin' = 0x5C
524
+ 'enter' = 0x0D; 'return' = 0x0D; 'esc' = 0x1B; 'escape' = 0x1B; 'space' = 0x20; 'tab' = 0x09
525
+ 'backspace' = 0x08; 'bs' = 0x08; 'delete' = 0x2E; 'del' = 0x2E; 'insert' = 0x2D; 'ins' = 0x2D
526
+ 'home' = 0x24; 'end' = 0x23; 'pageup' = 0x21; 'pgup' = 0x21; 'pagedown' = 0x22; 'pgdn' = 0x22
527
+ 'up' = 0x26; 'down' = 0x28; 'left' = 0x25; 'right' = 0x27
528
+ 'capslock' = 0x14; 'printscreen' = 0x2C; 'pause' = 0x13; 'apps' = 0x5D
529
+ }
530
+ for ($n = 1; $n -le 24; $n++) { $map["f$n"] = 0x6F + $n }
531
+
532
+ function Resolve-Vk([string]$t) {
533
+ $t = $t.Trim().ToLowerInvariant()
534
+ if ($map.ContainsKey($t)) { return [int]$map[$t] }
535
+ if ($t -match '^[a-z]$') { return [int][char]([char]::ToUpperInvariant($t)) }
536
+ if ($t -match '^\d$') { return 0x30 + [int]$t }
537
+ $aliases = @{
538
+ '-' = 0xBD; '=' = 0xBB; '[' = 0xDB; ']' = 0xDD; '\' = 0xDC; ';' = 0xBA; "'" = 0xDE
539
+ ',' = 0xBC; '.' = 0xBE; '/' = 0xBF; '`' = 0xC0
540
+ }
541
+ if ($aliases.ContainsKey($t)) { return [int]$aliases[$t] }
542
+ throw "无法识别的按键: $t"
543
+ }
544
+
545
+ $parts = $combo -split '\+'
546
+ $vks = @()
547
+ foreach ($p in $parts) { if ($p.Trim() -ne '') { $vks += (Resolve-Vk $p) } }
548
+ if ($vks.Count -eq 0) { throw "组合键为空" }
549
+
550
+ $mods = @()
551
+ foreach ($v in $vks[0..($vks.Count - 2)]) { $mods += [uint16]$v }
552
+ $last = [uint16]$vks[$vks.Count - 1]
553
+
554
+ foreach ($m in $mods) { [GuiNative]::KeyDown($m); Start-Sleep -Milliseconds 25 }
555
+ [GuiNative]::KeyDown($last)
556
+ Start-Sleep -Milliseconds 35
557
+ [GuiNative]::KeyUp($last)
558
+ for ($k = $mods.Count - 1; $k -ge 0; $k--) { [GuiNative]::KeyUp($mods[$k]); Start-Sleep -Milliseconds 25 }
559
+ "key $combo"
560
+ }
561
+
562
+ 'wait' {
563
+ $ms = if ($rest.Count -ge 1) { [int]$rest[0] } else { 500 }
564
+ Start-Sleep -Milliseconds $ms
565
+ "waited $ms ms"
566
+ }
567
+
568
+ default {
569
+ Write-Host "未知命令: $cmd"
570
+ Write-Host "可用: windows | pos | shot | activate | move | click | drag | scroll | type | key | wait"
571
+ exit 1
572
+ }
573
+ }
@@ -0,0 +1,5 @@
1
+ # dsh bundle layer: mount the desktop-control tool.
2
+ # Loaded by DeepSeek Harness from package.json's dsh.bundle manifest.
3
+ - insert:
4
+ - id: dsh-desktop-control
5
+ name: dsh-desktop-control
package/dsh/index.js ADDED
@@ -0,0 +1,289 @@
1
+ // dsh-desktop-control — DeepSeek Harness plugin
2
+ //
3
+ // Registers one `desktop` tool that lets the model see the screen and drive
4
+ // the mouse/keyboard: screenshot, window list, click, move, drag, scroll,
5
+ // type, key, activate, cursor position and wait.
6
+ //
7
+ // Implementation: every call spawns the bundled `assets/gui.ps1` (Windows
8
+ // PowerShell 5.1) and parses its output. Keeping the Win32 work in PowerShell
9
+ // avoids a native addon, keeps the package tiny, and reuses the script that
10
+ // was already battle-tested on this machine.
11
+ //
12
+ // Coordinates are physical pixels and match the saved screenshot one-to-one:
13
+ // the script enables per-monitor DPI awareness v2 internally.
14
+ //
15
+ // Safety: this is full desktop control. It can click anything and type
16
+ // anything. Every call is a normal tool call, so the host's tool policy and
17
+ // the user's judgement are the only gates. Do not enable it on an unattended
18
+ // machine.
19
+ import { spawn } from 'node:child_process'
20
+ import { existsSync, mkdirSync } from 'node:fs'
21
+ import { homedir } from 'node:os'
22
+ import { join } from 'node:path'
23
+ import { fileURLToPath } from 'node:url'
24
+
25
+ export const name = 'dsh-desktop-control'
26
+
27
+ // `tools` is the registry this plugin contributes the `desktop` tool to.
28
+ export const inject = ['tools']
29
+
30
+ const SCRIPT = fileURLToPath(new URL('../assets/gui.ps1', import.meta.url))
31
+ const DSH_HOME = process.env.DSH_HOME || join(homedir(), '.dsh')
32
+ const SHOT_DIR = join(DSH_HOME, 'desktop-control')
33
+
34
+ const DEFAULTS = {
35
+ enabled: true,
36
+ // Override only if powershell.exe is not on PATH.
37
+ powershellPath: 'powershell.exe',
38
+ timeoutMs: 30_000,
39
+ // Keep the most recent screenshots in the DSH home so read_image can open them.
40
+ screenshotDir: SHOT_DIR,
41
+ }
42
+
43
+ const ACTIONS = [
44
+ 'screenshot',
45
+ 'windows',
46
+ 'click',
47
+ 'move',
48
+ 'drag',
49
+ 'scroll',
50
+ 'type',
51
+ 'key',
52
+ 'activate',
53
+ 'pos',
54
+ 'wait',
55
+ ]
56
+
57
+ const ACTION_SET = new Set(ACTIONS)
58
+
59
+ const PARAMETERS = {
60
+ type: 'object',
61
+ additionalProperties: false,
62
+ properties: {
63
+ action: {
64
+ type: 'string',
65
+ enum: ACTIONS,
66
+ description:
67
+ 'screenshot (save a PNG and return its path), windows (list visible windows), click, move, drag, scroll, type (paste text), key (press a combo such as ctrl+s), activate (bring a window to front), pos (cursor position), wait.',
68
+ },
69
+ x: { type: 'number', description: 'Target X in physical screen pixels.' },
70
+ y: { type: 'number', description: 'Target Y in physical screen pixels.' },
71
+ x2: { type: 'number', description: 'Drag end X.' },
72
+ y2: { type: 'number', description: 'Drag end Y.' },
73
+ text: { type: 'string', description: 'Text for action=type. Unicode is fine; it is pasted via the clipboard.' },
74
+ key: { type: 'string', description: 'Combo for action=key, e.g. "ctrl+s", "alt+f4", "enter".' },
75
+ window: {
76
+ type: 'string',
77
+ description: 'Window title or process name substring, for screenshot -window or activate.',
78
+ },
79
+ region: {
80
+ type: 'string',
81
+ description: 'Screenshot region "x,y,width,height" in physical pixels.',
82
+ },
83
+ button: { type: 'string', enum: ['left', 'right', 'middle'], description: 'Mouse button for click/drag.' },
84
+ count: { type: 'integer', description: 'Click count, 1-3.' },
85
+ amount: { type: 'integer', description: 'Scroll notches; positive is up. 1 = one wheel notch.' },
86
+ waitMs: { type: 'integer', description: 'Milliseconds for action=wait.' },
87
+ print: {
88
+ type: 'boolean',
89
+ description: 'For screenshot with a window: capture it with PrintWindow even when occluded.',
90
+ },
91
+ },
92
+ required: ['action'],
93
+ }
94
+
95
+ const OUTPUT = {
96
+ type: 'object',
97
+ additionalProperties: false,
98
+ properties: {
99
+ ok: { type: 'boolean' },
100
+ action: { type: 'string' },
101
+ output: { type: 'string' },
102
+ imagePath: { type: 'string' },
103
+ width: { type: 'integer' },
104
+ height: { type: 'integer' },
105
+ },
106
+ required: ['ok', 'action'],
107
+ }
108
+
109
+ function requireNumber(args, key) {
110
+ const value = args[key]
111
+ if (typeof value !== 'number' || !Number.isFinite(value)) {
112
+ throw new Error(`desktop action "${args.action}" requires a numeric "${key}"`)
113
+ }
114
+ return value
115
+ }
116
+
117
+ function requireString(args, key) {
118
+ const value = args[key]
119
+ if (typeof value !== 'string' || value.length === 0) {
120
+ throw new Error(`desktop action "${args.action}" requires a non-empty "${key}"`)
121
+ }
122
+ return value
123
+ }
124
+
125
+ function shotPath(config) {
126
+ const dir = config.screenshotDir
127
+ mkdirSync(dir, { recursive: true })
128
+ const d = new Date()
129
+ const pad = (n, w = 2) => String(n).padStart(w, '0')
130
+ const stamp =
131
+ `${d.getFullYear()}${pad(d.getMonth() + 1)}${pad(d.getDate())}` +
132
+ `-${pad(d.getHours())}${pad(d.getMinutes())}${pad(d.getSeconds())}-${pad(d.getMilliseconds(), 3)}`
133
+ return join(dir, `shot-${stamp}.png`)
134
+ }
135
+
136
+ /**
137
+ * Translate one tool call into gui.ps1 argv.
138
+ * Exported for the self-test; not part of the public plugin API.
139
+ */
140
+ export function buildArgs(args, config = DEFAULTS) {
141
+ const action = args.action
142
+ if (!ACTION_SET.has(action)) throw new Error(`desktop: unknown action "${action}"`)
143
+ switch (action) {
144
+ case 'screenshot': {
145
+ const path = shotPath(config)
146
+ const argv = ['shot', path]
147
+ if (typeof args.window === 'string' && args.window.length > 0) argv.push('-window', args.window)
148
+ if (typeof args.region === 'string' && args.region.length > 0) argv.push('-region', args.region)
149
+ if (args.print === true) argv.push('-print')
150
+ return { argv, json: true }
151
+ }
152
+ case 'windows':
153
+ return { argv: ['windows'], json: false }
154
+ case 'click': {
155
+ const argv = ['click', requireNumber(args, 'x'), requireNumber(args, 'y')]
156
+ if (args.button !== undefined) argv.push('-button', String(args.button))
157
+ if (args.count !== undefined) argv.push('-count', String(args.count))
158
+ return { argv, json: false }
159
+ }
160
+ case 'move':
161
+ return { argv: ['move', requireNumber(args, 'x'), requireNumber(args, 'y')], json: false }
162
+ case 'drag':
163
+ return {
164
+ argv: [
165
+ 'drag',
166
+ requireNumber(args, 'x'),
167
+ requireNumber(args, 'y'),
168
+ requireNumber(args, 'x2'),
169
+ requireNumber(args, 'y2'),
170
+ ],
171
+ json: false,
172
+ }
173
+ case 'scroll': {
174
+ const amount = typeof args.amount === 'number' ? args.amount : 3
175
+ if (typeof args.x === 'number' && typeof args.y === 'number') {
176
+ return { argv: ['scroll', args.x, args.y, amount], json: false }
177
+ }
178
+ return { argv: ['scroll', amount], json: false }
179
+ }
180
+ case 'type':
181
+ // `--` stops option parsing, so text beginning with "-" is safe.
182
+ return { argv: ['type', '--', requireString(args, 'text')], json: false }
183
+ case 'key':
184
+ return { argv: ['key', requireString(args, 'key')], json: false }
185
+ case 'activate':
186
+ return { argv: ['activate', requireString(args, 'window')], json: false }
187
+ case 'pos':
188
+ return { argv: ['pos'], json: false }
189
+ case 'wait': {
190
+ const ms = typeof args.waitMs === 'number' ? args.waitMs : 500
191
+ return { argv: ['wait', ms], json: false }
192
+ }
193
+ default:
194
+ throw new Error(`desktop: unknown action "${action}"`)
195
+ }
196
+ }
197
+
198
+ function runPowerShell(argv, config, signal) {
199
+ return new Promise((resolve, reject) => {
200
+ const child = spawn(
201
+ config.powershellPath,
202
+ ['-NoProfile', '-NonInteractive', '-ExecutionPolicy', 'Bypass', '-File', SCRIPT, ...argv.map(String)],
203
+ { windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'] },
204
+ )
205
+ let stdout = ''
206
+ let stderr = ''
207
+ let settled = false
208
+ const finish = (error, value) => {
209
+ if (settled) return
210
+ settled = true
211
+ clearTimeout(timer)
212
+ signal?.removeEventListener('abort', onAbort)
213
+ if (error) reject(error)
214
+ else resolve(value)
215
+ }
216
+ const onAbort = () => {
217
+ child.kill()
218
+ finish(new Error('desktop: cancelled'))
219
+ }
220
+ const timer = setTimeout(() => {
221
+ child.kill()
222
+ finish(new Error(`desktop: timed out after ${config.timeoutMs} ms`))
223
+ }, config.timeoutMs)
224
+ signal?.addEventListener('abort', onAbort, { once: true })
225
+
226
+ child.stdout.setEncoding('utf8')
227
+ child.stderr.setEncoding('utf8')
228
+ child.stdout.on('data', (chunk) => {
229
+ stdout += chunk
230
+ })
231
+ child.stderr.on('data', (chunk) => {
232
+ stderr += chunk
233
+ })
234
+ child.on('error', (error) => finish(error))
235
+ child.on('close', (code) => {
236
+ if (code !== 0) {
237
+ finish(new Error(`desktop: gui.ps1 exited ${code}: ${(stderr || stdout).trim()}`))
238
+ return
239
+ }
240
+ finish(undefined, { stdout: stdout.trim(), stderr: stderr.trim() })
241
+ })
242
+ })
243
+ }
244
+
245
+ export function apply(ctx, userConfig = {}) {
246
+ const config = { ...DEFAULTS, ...(userConfig || {}) }
247
+ if (!config.enabled) return
248
+ if (!existsSync(SCRIPT)) throw new Error(`desktop: bundled script missing at ${SCRIPT}`)
249
+
250
+ ctx.tools.register({
251
+ name: 'desktop',
252
+ description:
253
+ 'See and control the Windows desktop. action=screenshot saves a PNG and returns its absolute path; open it with read_image to see the screen. Coordinates are physical pixels matching that image. Typical loop: screenshot -> read_image -> click/type -> screenshot again. Supports windows, click, move, drag, scroll, type, key, activate, pos, wait. Use it only when a task truly has no CLI or API path.',
254
+ parameters: PARAMETERS,
255
+ output: {
256
+ schema: OUTPUT,
257
+ render(_args, value) {
258
+ const parts = []
259
+ if (value.imagePath) parts.push(`Screenshot saved: ${value.imagePath}`)
260
+ if (typeof value.width === 'number') parts.push(`Size: ${value.width}x${value.height}`)
261
+ if (value.output) parts.push(value.output)
262
+ return [{ type: 'text', text: parts.join('\n') || 'ok' }]
263
+ },
264
+ },
265
+ isConcurrencySafe: () => false,
266
+ timeoutMs: config.timeoutMs + 5_000,
267
+ async execute(args, exec) {
268
+ const { argv, json } = buildArgs(args, config)
269
+ const { stdout } = await runPowerShell(argv, config, exec?.signal)
270
+ if (json) {
271
+ let parsed
272
+ try {
273
+ parsed = JSON.parse(stdout)
274
+ } catch {
275
+ throw new Error(`desktop: gui.ps1 returned non-JSON for ${args.action}: ${stdout.slice(0, 400)}`)
276
+ }
277
+ return {
278
+ ok: true,
279
+ action: args.action,
280
+ imagePath: typeof parsed.path === 'string' ? parsed.path : undefined,
281
+ width: typeof parsed.width === 'number' ? parsed.width : undefined,
282
+ height: typeof parsed.height === 'number' ? parsed.height : undefined,
283
+ output: stdout,
284
+ }
285
+ }
286
+ return { ok: true, action: args.action, output: stdout }
287
+ },
288
+ })
289
+ }
package/package.json CHANGED
@@ -1,6 +1,53 @@
1
1
  {
2
2
  "name": "dsh-desktop-control",
3
- "version": "0.0.0-stage",
4
- "stub": true,
5
- "description": "Temporary package placeholder for staged publishing"
6
- }
3
+ "version": "0.1.0",
4
+ "description": "DeepSeek Harness plugin: a desktop tool that screenshots the Windows screen and drives the mouse and keyboard.",
5
+ "author": "qingying012",
6
+ "repository": {
7
+ "type": "git",
8
+ "url": "git+https://github.com/qingying012/dsh-desktop-control.git"
9
+ },
10
+ "homepage": "https://github.com/qingying012/dsh-desktop-control#readme",
11
+ "bugs": {
12
+ "url": "https://github.com/qingying012/dsh-desktop-control/issues"
13
+ },
14
+ "keywords": [
15
+ "dsh",
16
+ "dsh-plugin",
17
+ "deepseek-harness",
18
+ "desktop",
19
+ "screenshot",
20
+ "mouse",
21
+ "keyboard",
22
+ "automation",
23
+ "windows"
24
+ ],
25
+ "license": "MIT",
26
+ "type": "module",
27
+ "exports": {
28
+ ".": "./dsh/index.js",
29
+ "./package.json": "./package.json"
30
+ },
31
+ "files": [
32
+ "dsh",
33
+ "assets",
34
+ "scripts",
35
+ "cordis.patch.yml",
36
+ "README.md",
37
+ "README.zh.md"
38
+ ],
39
+ "scripts": {
40
+ "selftest": "node scripts/selftest.mjs"
41
+ },
42
+ "os": [
43
+ "win32"
44
+ ],
45
+ "engines": {
46
+ "node": ">=22"
47
+ },
48
+ "dsh": {
49
+ "bundle": {
50
+ "patch": "./cordis.patch.yml"
51
+ }
52
+ }
53
+ }
@@ -0,0 +1,55 @@
1
+ // Pure unit tests for argv construction and argument validation. Nothing here
2
+ // touches the real desktop: no screenshots, no mouse, no keyboard.
3
+ import assert from 'node:assert/strict'
4
+ import { buildArgs } from '../dsh/index.js'
5
+
6
+ // screenshot builds a shot command with an absolute .png path.
7
+ {
8
+ const { argv, json } = buildArgs({ action: 'screenshot' }, { screenshotDir: 'C:\\shots' })
9
+ assert.equal(argv[0], 'shot')
10
+ assert.match(argv[1], /C:\\shots\\shot-\d{8}-\d{6}-\d{3}\.png$/)
11
+ assert.equal(json, true)
12
+ }
13
+
14
+ // screenshot with window / region / print adds the right flags.
15
+ {
16
+ const { argv } = buildArgs(
17
+ { action: 'screenshot', window: '记事本', region: '0,0,800,600', print: true },
18
+ { screenshotDir: 'C:\\shots' },
19
+ )
20
+ assert.deepEqual(argv.slice(2), ['-window', '记事本', '-region', '0,0,800,600', '-print'])
21
+ }
22
+
23
+ // type uses `--` so leading dashes are literal text.
24
+ {
25
+ const { argv } = buildArgs({ action: 'type', text: '-not-a-flag 中文' })
26
+ assert.deepEqual(argv, ['type', '--', '-not-a-flag 中文'])
27
+ }
28
+
29
+ // click validates coordinates and forwards button/count.
30
+ {
31
+ const { argv } = buildArgs({ action: 'click', x: 10, y: 20, button: 'right', count: 2 })
32
+ assert.deepEqual(argv, ['click', 10, 20, '-button', 'right', '-count', '2'])
33
+ assert.throws(() => buildArgs({ action: 'click', x: 10 }), /requires a numeric "y"/)
34
+ }
35
+
36
+ // scroll has both positional and point forms.
37
+ {
38
+ assert.deepEqual(buildArgs({ action: 'scroll', amount: -2 }).argv, ['scroll', -2])
39
+ assert.deepEqual(buildArgs({ action: 'scroll', x: 5, y: 6, amount: 3 }).argv, ['scroll', 5, 6, 3])
40
+ }
41
+
42
+ // move / drag / key / activate / pos / wait.
43
+ {
44
+ assert.deepEqual(buildArgs({ action: 'move', x: 1, y: 2 }).argv, ['move', 1, 2])
45
+ assert.deepEqual(buildArgs({ action: 'drag', x: 1, y: 2, x2: 3, y2: 4 }).argv, ['drag', 1, 2, 3, 4])
46
+ assert.deepEqual(buildArgs({ action: 'key', key: 'ctrl+s' }).argv, ['key', 'ctrl+s'])
47
+ assert.deepEqual(buildArgs({ action: 'activate', window: 'DeepSeek' }).argv, ['activate', 'DeepSeek'])
48
+ assert.deepEqual(buildArgs({ action: 'pos' }).argv, ['pos'])
49
+ assert.deepEqual(buildArgs({ action: 'wait' }).argv, ['wait', 500])
50
+ }
51
+
52
+ // unknown action is rejected.
53
+ assert.throws(() => buildArgs({ action: 'teleport' }), /unknown action/)
54
+
55
+ console.log('selftest ok: argv construction and validation behave as expected')