dsh-gui-control 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +117 -95
- package/lib/scripts/gui-daemon.log +4 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -1,97 +1,84 @@
|
|
|
1
|
-
|
|
2
|
-
|
|
3
|
-
**给 DeepSeek Harness 的 Windows 桌面操控：不是"帮我点一下这个按钮"，而是"以后这个活都交给你"。**
|
|
1
|
+
<div align="center">
|
|
4
2
|
|
|
5
|
-
|
|
3
|
+
<img src="https://raw.githubusercontent.com/seetown/dsh-gui-control/main/assets/icon-180.png" width="88" alt="dsh-gui-control">
|
|
6
4
|
|
|
7
|
-
|
|
5
|
+
# dsh-gui-control
|
|
8
6
|
|
|
9
|
-
|
|
7
|
+
**给 DeepSeek Harness 的 Windows 桌面操控：不是「帮我点一下」,而是「以后这个活都交给你」。**
|
|
10
8
|
|
|
11
|
-
|
|
9
|
+
<p>
|
|
10
|
+
<a href="https://www.npmjs.com/package/dsh-gui-control"><img alt="npm" src="https://img.shields.io/npm/v/dsh-gui-control.svg?color=cb3837"></a>
|
|
11
|
+
<a href="https://www.npmjs.com/package/dsh-gui-control"><img alt="downloads" src="https://img.shields.io/npm/dm/dsh-gui-control.svg"></a>
|
|
12
|
+
<img alt="platform" src="https://img.shields.io/badge/platform-Windows-0078d4.svg">
|
|
13
|
+
<img alt="license" src="https://img.shields.io/badge/license-MIT-blue.svg">
|
|
14
|
+
<a href="https://github.com/topics/dsh-plugin"><img alt="dsh-plugin" src="https://img.shields.io/badge/topic-dsh--plugin-6f42c1.svg"></a>
|
|
15
|
+
</p>
|
|
12
16
|
|
|
13
|
-
|
|
17
|
+
[中文说明](#中文说明) · [English](#english)
|
|
14
18
|
|
|
15
|
-
|
|
19
|
+
</div>
|
|
16
20
|
|
|
17
|
-
|
|
21
|
+
---
|
|
18
22
|
|
|
19
|
-
|
|
23
|
+
## 中文说明
|
|
20
24
|
|
|
21
|
-
|
|
25
|
+
### 它和别的桌面 Agent 有什么不同
|
|
22
26
|
|
|
23
|
-
|
|
27
|
+
大多数桌面自动化解决的是**一次性操作**：看一眼屏幕、点一下、再看一眼。这个插件多做一层——**把一套动作固化成任务文件**，跑的时候模型完全不参与。
|
|
24
28
|
|
|
25
|
-
|
|
29
|
+
| | 常见做法 | dsh-gui-control |
|
|
30
|
+
|---|---|---|
|
|
31
|
+
| 感知 | 截图 → 模型猜坐标 | **无障碍树优先**，按控件名定位；树不可用时才截图 |
|
|
32
|
+
| 一次任务的往返 | 每步一次模型调用 | 一次调用跑完整个任务文件 |
|
|
33
|
+
| 失败之后 | 不知道点没点上 | **投递判定**：没发出去 / 可能发出去了 / 已验证 |
|
|
34
|
+
| 重复执行 | 每次重跑一遍 | 任务文件可复用，带条件等待与重试 |
|
|
26
35
|
|
|
27
|
-
|
|
28
|
-
|---|---|
|
|
29
|
-
| `gui_snapshot` | 读窗口的无障碍树（文本，精确、便宜、与位置无关），或带屏幕坐标刻度尺的截图 |
|
|
30
|
-
| `gui_act` | 做一个动作，带**身份校验**和**投递判定** |
|
|
31
|
-
| `gui_task` | 跑一个声明式任务文件：一步调用完成多步，带条件等待、重试、风险闸门 |
|
|
36
|
+
### 演示
|
|
32
37
|
|
|
33
|
-
|
|
38
|
+
**① 打开 B 站 → 搜番剧 → 播到指定集数**
|
|
34
39
|
|
|
35
|
-
|
|
40
|
+
|
|
36
41
|
|
|
37
|
-
|
|
42
|
+
**② 语义点击：鼠标位移 0 像素**（直接调用控件的 Invoke，鼠标不动、窗口在后台也能点）
|
|
38
43
|
|
|
39
|
-
|
|
40
|
-
|---|---|---|
|
|
41
|
-
| `not_delivered` | 什么都没发出去（派发前就被拒） | ✅ 安全重试 |
|
|
42
|
-
| `delivered_unverified` | 发出去了，但效果没确认 | ❌ **绝对不能盲目重放** |
|
|
43
|
-
| `delivered_verified` | 发出去了，且后置条件验证通过 | ❌ |
|
|
44
|
+
|
|
44
45
|
|
|
45
|
-
|
|
46
|
+
**③ 填一列数据**（17 步 / 0.9 秒 / 一次调用）
|
|
46
47
|
|
|
47
|
-
|
|
48
|
+
|
|
48
49
|
|
|
49
|
-
|
|
50
|
+
### 安装
|
|
50
51
|
|
|
52
|
+
```bash
|
|
53
|
+
dsh plugin add dsh-gui-control
|
|
51
54
|
```
|
|
52
|
-
identity mismatch at index 0: expected id "NoSuchAutomationId", found "CloseButton"
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
### 3. 三通道并存，不赌单一来源
|
|
56
55
|
|
|
57
|
-
|
|
58
|
-
- **OCR**：文字密集但无树的界面（本地 Windows.Media.Ocr，不外传图片）
|
|
59
|
-
- **坐标 + 刻度尺截图**：画布、游戏、自绘列表
|
|
60
|
-
|
|
61
|
-
WPS 表格这类「有窗口但没有单元格节点」的应用，只有后两条路能走。
|
|
62
|
-
|
|
63
|
-
## 安装
|
|
56
|
+
或从源码：
|
|
64
57
|
|
|
65
58
|
```bash
|
|
66
|
-
|
|
67
|
-
dsh plugin add /absolute/path/to/dsh-gui-control
|
|
68
|
-
|
|
69
|
-
# 从 npm
|
|
70
|
-
dsh plugin add dsh-gui-control
|
|
59
|
+
dsh plugin add seetown/dsh-gui-control
|
|
71
60
|
```
|
|
72
61
|
|
|
73
|
-
|
|
62
|
+
要求：Windows 10 1809+、PowerShell 5.1、DeepSeek Harness 0.1.5-rc.2 ~ 0.2.x。装完不需要重启。
|
|
74
63
|
|
|
75
|
-
|
|
64
|
+
### 三个工具
|
|
76
65
|
|
|
77
|
-
|
|
66
|
+
| 工具 | 作用 |
|
|
67
|
+
|---|---|
|
|
68
|
+
| `gui_snapshot` | 读窗口的无障碍树（文本，精确、便宜、与位置无关），或带屏幕坐标刻度尺的截图 |
|
|
69
|
+
| `gui_act` | 执行一个动作，带身份校验和投递判定 |
|
|
70
|
+
| `gui_task` | 跑声明式任务文件：一次调用完成多步，带条件、重试、风险闸门 |
|
|
78
71
|
|
|
79
72
|
```
|
|
80
73
|
gui_snapshot { window: "Notepad", mode: "tree" }
|
|
81
|
-
→ MenuI 文件
|
|
82
|
-
Butto 关闭标签页
|
|
83
|
-
Docum 文本编辑器
|
|
84
|
-
...
|
|
74
|
+
→ Butto 关闭标签页 / Docum 文本编辑器 / MenuI 文件 ...
|
|
85
75
|
|
|
86
76
|
gui_act { action: "click_text", window: "Notepad", text: "最大化" }
|
|
87
77
|
→ click ok via InvokePattern | verified: text appeared
|
|
88
|
-
|
|
89
|
-
gui_task { file: "C:\\tasks\\fill-sheet.json", dryRun: true }
|
|
90
|
-
→ task fill-sheet ok in 0 ms
|
|
91
|
-
1. ok click 412 ms not_delivered
|
|
92
|
-
2. ok type 380 ms delivered_unverified
|
|
93
78
|
```
|
|
94
79
|
|
|
80
|
+
**优先用 `click_text` 而不是坐标**：它按控件名定位，鼠标不动，窗口布局变了也不会点错。
|
|
81
|
+
|
|
95
82
|
### 任务文件
|
|
96
83
|
|
|
97
84
|
```json
|
|
@@ -99,7 +86,6 @@ gui_task { file: "C:\\tasks\\fill-sheet.json", dryRun: true }
|
|
|
99
86
|
"name": "fill-sheet",
|
|
100
87
|
"steps": [
|
|
101
88
|
{ "do": "activate", "win": "工作簿1" },
|
|
102
|
-
{ "do": "click", "x": 129, "y": 449 },
|
|
103
89
|
{ "do": "type", "text": "项目" },
|
|
104
90
|
{ "do": "key", "key": "tab" },
|
|
105
91
|
{ "do": "type", "text": "金额" },
|
|
@@ -109,57 +95,85 @@ gui_task { file: "C:\\tasks\\fill-sheet.json", dryRun: true }
|
|
|
109
95
|
}
|
|
110
96
|
```
|
|
111
97
|
|
|
112
|
-
|
|
98
|
+
每步字段：`do` + 该动作参数，外加 `until`（轮询到条件成立才算过）、`untilTimeout`、`retry`、`risk`。
|
|
113
99
|
|
|
114
|
-
`risk: "high"` 的步骤默认**拒绝执行**，必须显式 `confirm: true
|
|
100
|
+
- `risk: "high"` 的步骤默认**拒绝执行**，必须显式 `confirm: true`
|
|
101
|
+
- `dryRun: true` 只返回计划，不碰桌面
|
|
115
102
|
|
|
116
|
-
|
|
103
|
+
### 投递语义
|
|
117
104
|
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
|
121
|
-
|
|
122
|
-
|
|
|
123
|
-
|
|
|
124
|
-
|
|
|
105
|
+
每个动作返回三选一：
|
|
106
|
+
|
|
107
|
+
| 判定 | 含义 | 能否重试 |
|
|
108
|
+
|---|---|---|
|
|
109
|
+
| `not_delivered` | 什么都没发出去（派发前就被拒） | ✅ 安全 |
|
|
110
|
+
| `delivered_unverified` | 发出去了，效果未确认 | ❌ 不能盲目重放 |
|
|
111
|
+
| `delivered_verified` | 发出去了，且后置条件验证通过 | ❌ |
|
|
125
112
|
|
|
126
|
-
|
|
113
|
+
这一条决定了「发消息 / 删文件 / 提交表单」这类不可逆操作是否安全。
|
|
127
114
|
|
|
128
|
-
|
|
115
|
+
### 安全
|
|
129
116
|
|
|
130
117
|
- 只监听 `127.0.0.1:38471`，不对外暴露
|
|
131
|
-
-
|
|
118
|
+
- 密码字段一律输出 `<redacted>`
|
|
132
119
|
- 高风险步骤需要显式确认
|
|
133
|
-
-
|
|
120
|
+
- 结果不明时**绝不自动重试**
|
|
134
121
|
|
|
135
|
-
|
|
122
|
+
> 这个插件能操控整台机器的鼠标键盘。装之前请知情，只在可信环境使用。
|
|
136
123
|
|
|
137
|
-
|
|
124
|
+
### 实测数据
|
|
125
|
+
|
|
126
|
+
| 操作 | 耗时 |
|
|
127
|
+
|---|---|
|
|
128
|
+
| 单步动作（常驻进程） | 10.8 ms |
|
|
129
|
+
| 语义点击（UIA 树） | 26–34 ms |
|
|
130
|
+
| 紧凑树读取 | 86 ms，865 字符（比完整 dump 省 88%） |
|
|
131
|
+
| 7 步任务（含启动计算器） | 680 ms，一次调用 |
|
|
132
|
+
| 桌面建 3 个文件夹 | 3.45 s，3/3 成功 |
|
|
133
|
+
| WPS 填 2×3 表格 | 0.9 s，17 步 |
|
|
134
|
+
|
|
135
|
+
首次调用会预热常驻进程（约 1 秒），之后每个动作是一次 loopback 往返。
|
|
136
|
+
|
|
137
|
+
### 已知限制
|
|
138
138
|
|
|
139
139
|
- **仅 Windows**（Win32 + UI Automation）
|
|
140
|
-
- 需要 PowerShell 5.1
|
|
141
|
-
-
|
|
142
|
-
-
|
|
140
|
+
- 需要 PowerShell 5.1
|
|
141
|
+
- 画布类应用（WPS 表格、游戏、自绘列表）没有无障碍树，只能截图 + OCR，准确率下降
|
|
142
|
+
- 网页任务目前基于截图；CDP 通道已实现但尚未接入工具
|
|
143
143
|
|
|
144
|
-
|
|
144
|
+
### Roadmap
|
|
145
145
|
|
|
146
146
|
- [ ] 截图自动编号（Set-of-Marks），把坐标估计降级成编号分类
|
|
147
|
+
- [ ] 接入 CDP：网页任务按选择器操作，而不是按坐标
|
|
147
148
|
- [ ] 任务库与意图路由：一句话命中现成任务
|
|
148
|
-
- [ ] macOS / Linux
|
|
149
|
-
|
|
150
|
-
## License
|
|
151
|
-
|
|
152
|
-
MIT
|
|
149
|
+
- [ ] macOS / Linux 适配
|
|
153
150
|
|
|
154
151
|
---
|
|
155
152
|
|
|
156
|
-
|
|
153
|
+
## English
|
|
157
154
|
|
|
158
155
|
**Windows desktop control for DeepSeek Harness.** Not "click that button for me" — "this job is yours from now on."
|
|
159
156
|
|
|
160
|
-
Most desktop agents do one-off operations: look, click, look again. This
|
|
157
|
+
Most desktop agents do one-off operations: look, click, look again. This one adds a layer underneath: **it turns a sequence into a task file** that runs with no model in the loop.
|
|
158
|
+
|
|
159
|
+
### What makes it different
|
|
160
|
+
|
|
161
|
+
| | Typical approach | dsh-gui-control |
|
|
162
|
+
|---|---|---|
|
|
163
|
+
| Perception | Screenshot, then the model guesses coordinates | **Accessibility tree first**, addressed by control name; screenshots only when no tree exists |
|
|
164
|
+
| Round trips per task | One model call per step | One call runs the whole task file |
|
|
165
|
+
| After a failure | Unknown whether it landed | **Delivery verdict**: not delivered / may have landed / verified |
|
|
166
|
+
| Repeating a flow | Replay it by hand | Task files are reusable, with conditions and retries |
|
|
167
|
+
|
|
168
|
+
### Install
|
|
169
|
+
|
|
170
|
+
```bash
|
|
171
|
+
dsh plugin add dsh-gui-control
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
Windows 10 1809+, PowerShell 5.1, DeepSeek Harness 0.1.5-rc.2 – 0.2.x. No restart needed.
|
|
161
175
|
|
|
162
|
-
|
|
176
|
+
### Three tools
|
|
163
177
|
|
|
164
178
|
| Tool | Purpose |
|
|
165
179
|
|---|---|
|
|
@@ -167,22 +181,30 @@ Most desktop agents do one-off operations: look, click, look again. This plugin
|
|
|
167
181
|
| `gui_act` | One action, with an identity check and a delivery verdict |
|
|
168
182
|
| `gui_task` | Run a declarative task file: many steps in one call, with conditions, retries and a risk gate |
|
|
169
183
|
|
|
170
|
-
|
|
184
|
+
### Delivery semantics
|
|
171
185
|
|
|
172
186
|
| Verdict | Meaning | Retry |
|
|
173
187
|
|---|---|---|
|
|
174
188
|
| `not_delivered` | Nothing was sent | ✅ safe |
|
|
175
189
|
| `delivered_unverified` | Input may have landed, effect unconfirmed | ❌ never blind-retry |
|
|
176
|
-
| `delivered_verified` | Landed and the post-condition held | ❌ |
|
|
190
|
+
| `delivered_verified` | Landed, and the post-condition held | ❌ |
|
|
177
191
|
|
|
178
192
|
That distinction is what makes "send a message / delete a file / submit a form" safe to automate.
|
|
179
193
|
|
|
180
|
-
|
|
194
|
+
### Security
|
|
181
195
|
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
196
|
+
Loopback-only (`127.0.0.1:38471`), password fields are redacted, high-risk steps require explicit confirmation, and an uncertain outcome is never retried automatically.
|
|
197
|
+
|
|
198
|
+
> This plugin drives the real mouse and keyboard of the whole machine. Install it knowingly, in a trusted environment.
|
|
199
|
+
|
|
200
|
+
### Limitations
|
|
201
|
+
|
|
202
|
+
Windows only; PowerShell 5.1 required; canvas-like apps (spreadsheets, games, self-drawn lists) expose no accessibility tree and fall back to screenshot + OCR with lower accuracy; web tasks currently use screenshots (the CDP path exists but is not wired into the tools yet).
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
|
|
206
|
+
<div align="center">
|
|
185
207
|
|
|
186
|
-
|
|
208
|
+
MIT © [seetown](https://github.com/seetown)
|
|
187
209
|
|
|
188
|
-
|
|
210
|
+
</div>
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-gui-control",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.1",
|
|
4
4
|
"description": "Windows desktop control for DeepSeek Harness: read a window's accessibility tree, act with identity checks and delivery semantics, and run repeatable task files.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|