dsh-lens-computer-control 0.0.0-stage → 0.8.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +144 -2
- package/bin/lens.exe +0 -0
- package/bin/lenshud.exe +0 -0
- package/client.js +362 -0
- package/cordis.patch.yml +6 -0
- package/examples/ppt-textbox.txt +16 -0
- package/examples/table-fill.txt +21 -0
- package/examples/wechat-send.txt +37 -0
- package/icon.svg +13 -0
- package/index.js +943 -0
- package/locale/en.json +6 -0
- package/locale/zh.json +6 -0
- package/market.json +20 -0
- package/native/Lens/Capture.cs +99 -0
- package/native/Lens/Det/Ocr.cs +641 -0
- package/native/Lens/Det/Solid.cs +125 -0
- package/native/Lens/Det/Table.cs +756 -0
- package/native/Lens/Det/Template.cs +685 -0
- package/native/Lens/Det/TextGrid.cs +117 -0
- package/native/Lens/Det/Uia.cs +166 -0
- package/native/Lens/Flow.cs +577 -0
- package/native/Lens/Grid.cs +252 -0
- package/native/Lens/Input.cs +183 -0
- package/native/Lens/Model.cs +247 -0
- package/native/Lens/Program.cs +1296 -0
- package/native/Lens/Win32.cs +545 -0
- package/package.json +60 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Lens Computer Control contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,145 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Lens 电脑控制
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
零第三方依赖的 Windows 电脑控制插件。引擎是随包分发的 `bin/lens.exe`（C# 编写，用系统自带 `csc.exe` 编译，源码在 `native/`），只调用 Windows 自带的 Win32 / GDI+ / UIAutomation / WinRT OCR。
|
|
4
|
+
|
|
5
|
+
## 核心思路
|
|
6
|
+
|
|
7
|
+
**模型永远不输出绝对坐标。** 引擎把窗口扫成一张「候选表」并给每个目标编号，模型只负责说「点 #17」；坐标由引擎从候选框精确算出。
|
|
8
|
+
|
|
9
|
+
这样做的原因：让多模态模型直接报像素坐标，误差通常在几十像素；而让它从编号里挑一个，等于把定位问题降级成分类问题，误差归零。
|
|
10
|
+
|
|
11
|
+
## 六个感知通道
|
|
12
|
+
|
|
13
|
+
| 通道 | 覆盖场景 |
|
|
14
|
+
|---|---|
|
|
15
|
+
| `uia` | 常规 Win32 / WPF / Electron 应用（白送像素级精度） |
|
|
16
|
+
| `ocr` | Windows 自带 OCR，原图 / 反色 / 2× 最近邻上采样三路 |
|
|
17
|
+
| `solid` | 量化颜色连通域，专抓实心 / 圆角按钮 |
|
|
18
|
+
| `table` | 行列投影，认行情软件的表格网格 |
|
|
19
|
+
| `textgrid` | 靠文字位置聚类行列，兜住暗色低对比、只画内线的自绘表 |
|
|
20
|
+
| `template` | 零均值 NCC 锚点，教一次永久精确 |
|
|
21
|
+
|
|
22
|
+
自绘窗口（微信 / QQ / 飞书 / 券商行情 / Java Swing）UIA 几乎是空的，靠后五个通道兜住。
|
|
23
|
+
|
|
24
|
+
## 工具
|
|
25
|
+
|
|
26
|
+
| 工具 | 作用 |
|
|
27
|
+
|---|---|
|
|
28
|
+
| `computer_windows` | 列窗口（标题、DPI、进程、真实可见边框、最小化状态） |
|
|
29
|
+
| `computer_scan` | 多通道扫描 → 候选表 + 带编号的标注图 |
|
|
30
|
+
| `computer_act` | 按候选 id / 表格行列 / 坐标点击，发键，输入中文，拖动 |
|
|
31
|
+
| `computer_verify` | 点完做像素差分，判断到底有没有生效 |
|
|
32
|
+
| `computer_grid` | 坐标网格：边缘刻度 + 二级放大，没有候选时也不猜像素 |
|
|
33
|
+
| `computer_vision` | **纯视觉递归定位**：截图切 N 格、每格标注屏幕绝对坐标，选一格后只切那一格，逐步收窄直到够准 |
|
|
34
|
+
| `computer_scroll` | 滚轮：列表/聊天记录/网页/表格；真实滚轮事件，滚动前先置前并做遮挡检查 |
|
|
35
|
+
| `computer_run` | **脚本自动化**：一次调用执行多步（点击/输入/滚动/拖动/等待/核验），返回逐步 trace |
|
|
36
|
+
|
|
37
|
+
### 脚本自动化（computer_run）
|
|
38
|
+
|
|
39
|
+
把一串动作写成一个脚本，一次调用跑完，省掉 N 次往返：
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
window 微信
|
|
43
|
+
focus
|
|
44
|
+
scroll delta=-3 times=2
|
|
45
|
+
click find=text:张三
|
|
46
|
+
type text=在吗
|
|
47
|
+
key combo=enter
|
|
48
|
+
wait for=changed timeout=5000
|
|
49
|
+
verify expect=changed
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
支持的动词：`window` `focus` `click` `type` `key` `scroll` `drag` `wait` `verify` `sleep`。
|
|
53
|
+
默认遇错即停并返回 trace；`continueOnError` 可改成失败也继续。坐标可写 `w` 前缀表示相对窗口客户区（如 `from=w100,w100`）。
|
|
54
|
+
|
|
55
|
+
### 滚动（computer_scroll）
|
|
56
|
+
|
|
57
|
+
滚轮事件发给「光标当前所在」的窗口，所以引擎会先把光标移到目标位置（缓动），再逐格发送。
|
|
58
|
+
|
|
59
|
+
```
|
|
60
|
+
computer_scroll { window: "微信", delta: -3, times: 2 }
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
`delta` 为正 = 向上/向左，为负 = 向下/向右。给了 `window` 会先置前、并检查滚动点有没有被别的窗口遮挡——滚错窗口等于误操作，所以默认拒绝。
|
|
64
|
+
|
|
65
|
+
### 纯视觉递归定位（computer_vision）
|
|
66
|
+
|
|
67
|
+
当所有识别通道都失效时用这个——游戏、画布、远程桌面、自绘且无文字的面板。
|
|
68
|
+
|
|
69
|
+
1. `computer_vision {}` → 整屏切成 10 格（5×2），每格印着 `#编号`、`x,y WxH`、`@中心点`
|
|
70
|
+
2. 看图，报你要第几格 → `computer_vision {path:"7"}`
|
|
71
|
+
3. 只把那一格再切 10 份，继续选 → `{path:"7.4"}`
|
|
72
|
+
4. 够准了就 `computer_act {x: cx, y: cy}`
|
|
73
|
+
|
|
74
|
+
每一级的坐标都是**屏幕物理像素绝对值**，放大只影响看的清晰度，不影响坐标。默认按区域长宽比自动选接近正方形的行列数，也可以用 `cols`/`rows` 强制。
|
|
75
|
+
|
|
76
|
+
### 视觉反馈层（HUD）
|
|
77
|
+
|
|
78
|
+
AI 操作电脑时，屏幕上会有一圈渐变蓝边，顶部浮一个胶囊：
|
|
79
|
+
|
|
80
|
+
```
|
|
81
|
+
● Lens AI 正在操作你的电脑 | 正在点击
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
胶囊右边那句话由引擎自动更新（每个命令映射一句人话），最后一条动作结束约 3.5 秒后淡出。
|
|
85
|
+
|
|
86
|
+
- 第一次调用任何 `computer_*` 工具时自动拉起 `bin/lenshud.exe`，不用手动开
|
|
87
|
+
- 命令行：`lens hud --action start|stop|status`，或 `lens hud --action say --text "正在导出"`
|
|
88
|
+
- **点击穿透**（`WS_EX_TRANSPARENT`），不会挡住你操作
|
|
89
|
+
- **对截屏不可见**（`WDA_EXCLUDEFROMCAPTURE`），所以不会污染 `computer_scan` 的候选表、也不会出现在 `computer_vision` 的分格图里
|
|
90
|
+
- 不占任务栏、不进 Alt+Tab
|
|
91
|
+
|
|
92
|
+
### 微信 / QQ 这类 Qt 自绘窗口
|
|
93
|
+
|
|
94
|
+
它们没有可用的 UIA（节点通常 <10），候选几乎全靠 OCR，所以有几个必须知道的坑：
|
|
95
|
+
|
|
96
|
+
- **OCR 会认错字**：老师→老丝、取消→取肖、语音通话→浯音通话。`find=text` 要用**短关键词**，别拿整句去精确匹配。
|
|
97
|
+
- **找联系人别翻列表**：用 `Ctrl+F` 打开搜索框（微信 4.x 实测有效），输入名字回车，比在会话列表里 OCR 匹配稳得多。见 `examples/wechat-send.txt`。
|
|
98
|
+
- **登录窗和主窗是两个不同 hwnd**：登录成功后登录窗销毁、主窗新建。脚本里用 `window 微信` 按标题定位，别缓存 hwnd。
|
|
99
|
+
- **点了登录按钮窗口消失 = 成功**：引擎返回 `warning:"window_gone"`，不是失败。
|
|
100
|
+
- **最小化到托盘 ≠ 已退出**：窗口变成隐藏状态，`computer_scan` 会找不到。先调 `computer_focus`（会自动还原并返回 `restoredFromTray:true`），**千万不要重新启动应用**——那会开第二个实例、弹登录页。
|
|
101
|
+
|
|
102
|
+
## 命令行
|
|
103
|
+
|
|
104
|
+
引擎也能单独用（`bin/lens.exe --help` 看全部）：
|
|
105
|
+
|
|
106
|
+
```
|
|
107
|
+
lens info
|
|
108
|
+
lens scan --window "微信" --overlay out.png
|
|
109
|
+
lens click --window "微信" --id 6
|
|
110
|
+
lens table --window "Excel书" --json
|
|
111
|
+
lens fill --window "XX" --plan plan.txt
|
|
112
|
+
lens grid --window "XX" --cols 32 --rows 18
|
|
113
|
+
lens run --flow examples/ppt-textbox.txt
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
## 多步流程
|
|
117
|
+
|
|
118
|
+
流程是行式脚本，可进 git、可 review、可重放：
|
|
119
|
+
|
|
120
|
+
```
|
|
121
|
+
window Workbench
|
|
122
|
+
focus
|
|
123
|
+
drag from=w220,514 to=w540,594 duration=700
|
|
124
|
+
type text=季度报告
|
|
125
|
+
assert text=季度报告
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
坐标支持 `w` 前缀表示「相对窗口客户区原点」，窗口挪位置不用改流程。
|
|
129
|
+
`examples/` 里有两个可直接跑的流程。
|
|
130
|
+
|
|
131
|
+
## 安全
|
|
132
|
+
|
|
133
|
+
- **急停**：`POST http://127.0.0.1:45871/stop` 会立刻在工具层拒绝所有 `computer_*` 调用。急停不依赖 UI 渲染。
|
|
134
|
+
- **遮挡拒绝**：目标点被别的窗口盖住时返回 `TARGET_COVERED` 并拒绝点击，不会误操作你正在用的窗口。
|
|
135
|
+
|
|
136
|
+
## 已知限制
|
|
137
|
+
|
|
138
|
+
- 反截屏窗口（`WDA_EXCLUDEFROMCAPTURE`）下截图返回全黑，只能做引导不能代点。
|
|
139
|
+
- AWT/Swing 窗口不响应后台 `PostMessage` 点击，必须前台真点击。
|
|
140
|
+
- 填表的自动回读依赖 OCR，部分短词读不出，此时以「写入成功」为准（可用应用自身状态或差分复核）。
|
|
141
|
+
- 目前只支持 Windows。
|
|
142
|
+
|
|
143
|
+
## 许可
|
|
144
|
+
|
|
145
|
+
MIT
|
package/bin/lens.exe
ADDED
|
Binary file
|
package/bin/lenshud.exe
ADDED
|
Binary file
|
package/client.js
ADDED
|
@@ -0,0 +1,362 @@
|
|
|
1
|
+
window.__ModuleLoader__.load({
|
|
2
|
+
id: 'dsh-lens-computer-control',
|
|
3
|
+
factory(require) {
|
|
4
|
+
const React = require('react')
|
|
5
|
+
const h = React.createElement
|
|
6
|
+
const PORT = 45871
|
|
7
|
+
const BASE = 'http://127.0.0.1:' + PORT
|
|
8
|
+
|
|
9
|
+
// 设计目标：像一块"设备状态卡"，而不是一个调试面板。
|
|
10
|
+
// 三等分信息层次：品牌+状态 → 画面/动作 → 唯一的操作（急停）。
|
|
11
|
+
const CSS = `
|
|
12
|
+
.lensHud{position:fixed;right:20px;bottom:20px;width:max-content;z-index:2147483000;
|
|
13
|
+
font-family:-apple-system,BlinkMacSystemFont,"Segoe UI","Microsoft YaHei UI",system-ui,sans-serif;
|
|
14
|
+
color:#e6eefc;-webkit-font-smoothing:antialiased;user-select:none}
|
|
15
|
+
.lensHud *{box-sizing:border-box}
|
|
16
|
+
|
|
17
|
+
.lensCard{position:relative;width:320px;max-width:calc(100vw - 24px);border-radius:16px;overflow:hidden;
|
|
18
|
+
background:linear-gradient(180deg,rgba(17,22,32,.94),rgba(9,12,18,.97));
|
|
19
|
+
box-shadow:0 24px 60px -12px rgba(0,0,0,.7),0 0 0 1px rgba(255,255,255,.03),inset 0 1px 0 rgba(255,255,255,.05);
|
|
20
|
+
backdrop-filter:blur(20px) saturate(160%);-webkit-backdrop-filter:blur(20px) saturate(160%);
|
|
21
|
+
transition:width .18s cubic-bezier(.4,0,.2,1),box-shadow .18s ease}
|
|
22
|
+
/* 渐变描边：比纯色边框更有层次 */
|
|
23
|
+
.lensCard::before{content:'';position:absolute;inset:0;border-radius:inherit;padding:1px;pointer-events:none;
|
|
24
|
+
background:linear-gradient(145deg,rgba(56,189,248,.5),rgba(56,189,248,.05) 38%,rgba(255,255,255,.05) 68%,rgba(129,140,248,.42));
|
|
25
|
+
-webkit-mask:linear-gradient(#000 0 0) content-box,linear-gradient(#000 0 0);
|
|
26
|
+
-webkit-mask-composite:xor;mask-composite:exclude}
|
|
27
|
+
|
|
28
|
+
.lensHud.isStopped .lensCard::before{background:linear-gradient(145deg,rgba(248,113,113,.6),rgba(248,113,113,.08) 45%,rgba(248,113,113,.4))}
|
|
29
|
+
.lensHud.isOffline .lensCard::before{background:linear-gradient(145deg,rgba(148,163,184,.25),rgba(148,163,184,.05))}
|
|
30
|
+
.lensHud.isOffline{opacity:.92}
|
|
31
|
+
|
|
32
|
+
/* ── 头部 ── */
|
|
33
|
+
.lensHead{display:flex;align-items:center;gap:10px;padding:11px 14px;cursor:grab;touch-action:none}
|
|
34
|
+
.lensHead:active{cursor:grabbing}
|
|
35
|
+
.lensBrand{display:flex;align-items:center;gap:8px}
|
|
36
|
+
.lensDot{width:7px;height:7px;border-radius:50%;background:#34d399;position:relative;flex:0 0 auto}
|
|
37
|
+
.lensDot::after{content:'';position:absolute;inset:-4px;border-radius:50%;background:rgba(52,211,153,.35);animation:lensPing 2.1s cubic-bezier(0,0,.2,1) infinite}
|
|
38
|
+
@keyframes lensPing{0%{transform:scale(.6);opacity:.85}80%,100%{transform:scale(2.1);opacity:0}}
|
|
39
|
+
.lensDot.off{background:#f87171}
|
|
40
|
+
.lensDot.off::after{background:rgba(248,113,113,.4);animation:none}
|
|
41
|
+
.lensDot.idle{background:#64748b}
|
|
42
|
+
.lensDot.idle::after{display:none}
|
|
43
|
+
|
|
44
|
+
.lensName{font-size:12.5px;font-weight:650;letter-spacing:.16em;
|
|
45
|
+
background:linear-gradient(90deg,#e8f4ff,#7dd3fc);-webkit-background-clip:text;background-clip:text;color:transparent}
|
|
46
|
+
.lensState{margin-left:auto;font-size:10px;font-weight:600;letter-spacing:.1em;
|
|
47
|
+
padding:3px 9px;border-radius:999px;color:#7dd3fc;background:rgba(56,189,248,.12);
|
|
48
|
+
border:1px solid rgba(56,189,248,.22);font-variant-numeric:tabular-nums}
|
|
49
|
+
.lensHud.isStopped .lensState{color:#fca5a5;background:rgba(248,113,113,.12);border-color:rgba(248,113,113,.28)}
|
|
50
|
+
.lensHud.isOffline .lensState{color:#94a3b8;background:rgba(148,163,184,.1);border-color:rgba(148,163,184,.2)}
|
|
51
|
+
.lensChev{color:#64748b;display:flex;transition:transform .18s ease}
|
|
52
|
+
.lensHud.isCollapsed .lensChev{transform:rotate(-90deg)}
|
|
53
|
+
|
|
54
|
+
/* ── 画面 ── */
|
|
55
|
+
.lensBody{border-top:1px solid rgba(255,255,255,.06);cursor:grab}
|
|
56
|
+
.lensShot{position:relative;background:#080b11;overflow:hidden}
|
|
57
|
+
.lensShot img{display:block;width:100%;height:132px;object-fit:cover;object-position:top center;opacity:.96}
|
|
58
|
+
.lensShotBar{position:absolute;left:0;right:0;bottom:0;display:flex;align-items:center;gap:8px;
|
|
59
|
+
padding:18px 12px 9px;font-size:11px;
|
|
60
|
+
background:linear-gradient(180deg,rgba(6,9,14,0),rgba(6,9,14,.92))}
|
|
61
|
+
.lensWin{color:#dbeafe;font-weight:600;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;max-width:180px}
|
|
62
|
+
.lensChip{font-size:10px;padding:2px 7px;border-radius:6px;color:#7dd3fc;
|
|
63
|
+
background:rgba(56,189,248,.13);border:1px solid rgba(56,189,248,.18);font-variant-numeric:tabular-nums}
|
|
64
|
+
.lensChip.muted{color:#94a3b8;background:rgba(148,163,184,.1);border-color:rgba(148,163,184,.16)}
|
|
65
|
+
|
|
66
|
+
/* ── 空状态 ── */
|
|
67
|
+
.lensVoid{display:flex;flex-direction:column;align-items:center;gap:9px;padding:20px 16px 18px;text-align:center}
|
|
68
|
+
.lensRing{width:34px;height:34px;border-radius:50%;border:1px dashed rgba(125,211,252,.32);
|
|
69
|
+
display:flex;align-items:center;justify-content:center;position:relative}
|
|
70
|
+
.lensRing::before{content:'';width:9px;height:9px;border-radius:50%;background:rgba(125,211,252,.42);
|
|
71
|
+
box-shadow:0 0 12px rgba(56,189,248,.55);animation:lensBreath 2.6s ease-in-out infinite}
|
|
72
|
+
@keyframes lensBreath{0%,100%{opacity:.45;transform:scale(.85)}50%{opacity:1;transform:scale(1.1)}}
|
|
73
|
+
.lensVoidText{font-size:11.5px;color:#8b98ab;line-height:1.5}
|
|
74
|
+
.lensVoidText b{color:#cbd5e1;font-weight:600}
|
|
75
|
+
|
|
76
|
+
/* ── 动作流 ── */
|
|
77
|
+
.lensFeedHead{display:flex;align-items:center;justify-content:space-between;
|
|
78
|
+
padding:9px 14px 5px;font-size:10px;letter-spacing:.12em;color:#5f6b7d;font-weight:600}
|
|
79
|
+
.lensRows{cursor:default;padding:0 6px 6px;max-height:126px;overflow:auto;scrollbar-width:thin;scrollbar-color:rgba(148,163,184,.25) transparent}
|
|
80
|
+
.lensRows::-webkit-scrollbar{width:6px}
|
|
81
|
+
.lensRows::-webkit-scrollbar-thumb{background:rgba(148,163,184,.2);border-radius:3px}
|
|
82
|
+
.lensRow{display:grid;grid-template-columns:46px 1fr 7px;gap:9px;align-items:center;
|
|
83
|
+
padding:5px 8px;border-radius:8px;transition:background .13s ease}
|
|
84
|
+
.lensRow:hover{background:rgba(125,211,252,.06)}
|
|
85
|
+
.lensTime{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;font-size:9.5px;
|
|
86
|
+
color:#4b5768;font-variant-numeric:tabular-nums}
|
|
87
|
+
.lensAct{display:flex;align-items:baseline;gap:6px;min-width:0}
|
|
88
|
+
.lensVerb{font-size:10px;font-weight:700;letter-spacing:.04em;color:#7dd3fc;flex:0 0 auto}
|
|
89
|
+
.lensDet{font-size:11px;color:#94a3b8;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;min-width:0}
|
|
90
|
+
.lensOk{width:6px;height:6px;border-radius:50%;background:#34d399;box-shadow:0 0 8px rgba(52,211,153,.6)}
|
|
91
|
+
.lensOk.bad{background:#fbbf24;box-shadow:0 0 8px rgba(251,191,36,.6)}
|
|
92
|
+
|
|
93
|
+
/* ── 底部 ── */
|
|
94
|
+
.lensFoot{display:flex;align-items:center;gap:10px;padding:10px 12px;
|
|
95
|
+
border-top:1px solid rgba(255,255,255,.06);background:rgba(255,255,255,.012)}
|
|
96
|
+
.lensHint{font-size:10.5px;color:#5f6b7d;letter-spacing:.02em;flex:1 1 auto;
|
|
97
|
+
overflow:hidden;text-overflow:ellipsis;white-space:nowrap}
|
|
98
|
+
.lensStop{border:1px solid rgba(248,113,113,.34);background:rgba(248,113,113,.09);
|
|
99
|
+
color:#fca5a5;border-radius:9px;padding:7px 15px;font:600 11.5px/1 inherit;cursor:pointer;
|
|
100
|
+
letter-spacing:.05em;transition:all .14s ease;flex:0 0 auto}
|
|
101
|
+
.lensStop:hover:not(:disabled){background:rgba(248,113,113,.18);border-color:rgba(248,113,113,.6);
|
|
102
|
+
color:#fecaca;box-shadow:0 0 18px -4px rgba(248,113,113,.5)}
|
|
103
|
+
.lensStop:active:not(:disabled){transform:translateY(1px)}
|
|
104
|
+
.lensStop:disabled{opacity:.45;cursor:default}
|
|
105
|
+
.lensStop.armed{border-color:rgba(52,211,153,.36);background:rgba(52,211,153,.1);color:#6ee7b7}
|
|
106
|
+
.lensStop.armed:hover:not(:disabled){background:rgba(52,211,153,.18);border-color:rgba(52,211,153,.6);
|
|
107
|
+
color:#a7f3d0;box-shadow:0 0 18px -4px rgba(52,211,153,.5)}
|
|
108
|
+
|
|
109
|
+
/* 折叠态：宽度交给内容决定，不要留固定宽 —— 否则右边会多出一块既挡页面又拖不动的透明区 */
|
|
110
|
+
.lensHud.isCollapsed .lensCard{width:auto}
|
|
111
|
+
`
|
|
112
|
+
|
|
113
|
+
function useStatus() {
|
|
114
|
+
const [state, setState] = React.useState({ loading: true })
|
|
115
|
+
React.useEffect(() => {
|
|
116
|
+
let alive = true
|
|
117
|
+
const tick = () => {
|
|
118
|
+
fetch(BASE + '/status')
|
|
119
|
+
.then(function (r) { return r.json() })
|
|
120
|
+
.then(function (j) { if (alive) setState(j) })
|
|
121
|
+
.catch(function () { if (alive) setState({ offline: true }) })
|
|
122
|
+
}
|
|
123
|
+
tick()
|
|
124
|
+
const timer = setInterval(tick, 1200)
|
|
125
|
+
return function () {
|
|
126
|
+
alive = false
|
|
127
|
+
clearInterval(timer)
|
|
128
|
+
}
|
|
129
|
+
}, [])
|
|
130
|
+
return [state, setState]
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
// ── 位置记忆与拖动 ──
|
|
134
|
+
const POS_KEY = 'lens-hud-pos'
|
|
135
|
+
|
|
136
|
+
function loadPos() {
|
|
137
|
+
try {
|
|
138
|
+
const raw = window.localStorage.getItem(POS_KEY)
|
|
139
|
+
if (!raw) return null
|
|
140
|
+
const p = JSON.parse(raw)
|
|
141
|
+
if (typeof p.x === 'number' && typeof p.y === 'number') return p
|
|
142
|
+
} catch (e) {}
|
|
143
|
+
return null
|
|
144
|
+
}
|
|
145
|
+
|
|
146
|
+
function savePos(p) {
|
|
147
|
+
try {
|
|
148
|
+
if (p) window.localStorage.setItem(POS_KEY, JSON.stringify(p))
|
|
149
|
+
else window.localStorage.removeItem(POS_KEY)
|
|
150
|
+
} catch (e) {}
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
// 夹在视口内，别让面板被拖到看不见的地方
|
|
154
|
+
function clampPos(p, w, h) {
|
|
155
|
+
const pad = 6
|
|
156
|
+
const maxX = Math.max(pad, window.innerWidth - w - pad)
|
|
157
|
+
const maxY = Math.max(pad, window.innerHeight - h - pad)
|
|
158
|
+
return { x: Math.min(Math.max(pad, p.x), maxX), y: Math.min(Math.max(pad, p.y), maxY) }
|
|
159
|
+
}
|
|
160
|
+
function clock(ms) {
|
|
161
|
+
const d = new Date(ms)
|
|
162
|
+
const p = function (n) { return String(n).padStart(2, '0') }
|
|
163
|
+
return p(d.getHours()) + ':' + p(d.getMinutes()) + ':' + p(d.getSeconds())
|
|
164
|
+
}
|
|
165
|
+
|
|
166
|
+
// 细箭头，纯 CSS 画，避免引外部字体
|
|
167
|
+
function Chevron() {
|
|
168
|
+
return h('svg', { className: 'lensChev', width: 14, height: 14, viewBox: '0 0 24 24', fill: 'none' },
|
|
169
|
+
h('path', { d: 'M6 9l6 6 6-6', stroke: 'currentColor', strokeWidth: 2, strokeLinecap: 'round', strokeLinejoin: 'round' }))
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
function Hud() {
|
|
173
|
+
const [state, setState] = useStatus()
|
|
174
|
+
const [collapsed, setCollapsed] = React.useState(false)
|
|
175
|
+
const [busy, setBusy] = React.useState(false)
|
|
176
|
+
const [pos, setPos] = React.useState(loadPos)
|
|
177
|
+
|
|
178
|
+
const rootRef = React.useRef(null)
|
|
179
|
+
const dragRef = React.useRef(null)
|
|
180
|
+
const movedRef = React.useRef(false)
|
|
181
|
+
|
|
182
|
+
// 位置持久化
|
|
183
|
+
React.useEffect(function () { savePos(pos) }, [pos])
|
|
184
|
+
|
|
185
|
+
// 每次渲染后把面板拉回可见范围。
|
|
186
|
+
// 只靠拖动/resize 夹边界不够：初始加载（不同窗口尺寸的旧位置）、
|
|
187
|
+
// 折叠/展开（宽度变了）、动作流加长（高度变了）都可能让它跑出窗口。
|
|
188
|
+
React.useEffect(function () {
|
|
189
|
+
if (!pos) return
|
|
190
|
+
const el = rootRef.current
|
|
191
|
+
if (!el) return
|
|
192
|
+
const r = el.getBoundingClientRect()
|
|
193
|
+
const c = clampPos(pos, r.width, r.height)
|
|
194
|
+
if (Math.abs(c.x - pos.x) > 0.5 || Math.abs(c.y - pos.y) > 0.5) setPos(c)
|
|
195
|
+
})
|
|
196
|
+
|
|
197
|
+
React.useEffect(function () {
|
|
198
|
+
function onResize() {
|
|
199
|
+
const el = rootRef.current
|
|
200
|
+
if (!el || !pos) return
|
|
201
|
+
const r = el.getBoundingClientRect()
|
|
202
|
+
const c = clampPos(pos, r.width, r.height)
|
|
203
|
+
if (c.x !== pos.x || c.y !== pos.y) setPos(c)
|
|
204
|
+
}
|
|
205
|
+
window.addEventListener('resize', onResize)
|
|
206
|
+
return function () { window.removeEventListener('resize', onResize) }
|
|
207
|
+
}, [pos])
|
|
208
|
+
|
|
209
|
+
// 拖动：mousedown 记起点，window 上跟手，松手结束
|
|
210
|
+
React.useEffect(function () {
|
|
211
|
+
function onMove(e) {
|
|
212
|
+
const d = dragRef.current
|
|
213
|
+
if (!d) return
|
|
214
|
+
const dx = e.clientX - d.sx
|
|
215
|
+
const dy = e.clientY - d.sy
|
|
216
|
+
if (!d.moved && Math.abs(dx) + Math.abs(dy) < 4) return // 4px 死区：区分点击与拖动
|
|
217
|
+
d.moved = true
|
|
218
|
+
movedRef.current = true
|
|
219
|
+
// 每次移动都重新量尺寸：折叠/展开会改高度，用按下瞬间的旧值会把边界算错
|
|
220
|
+
const el = rootRef.current
|
|
221
|
+
const r = el ? el.getBoundingClientRect() : { width: d.w, height: d.h }
|
|
222
|
+
setPos(clampPos({ x: d.ox + dx, y: d.oy + dy }, r.width, r.height))
|
|
223
|
+
}
|
|
224
|
+
function onUp() {
|
|
225
|
+
if (dragRef.current && dragRef.current.moved) {
|
|
226
|
+
// 拖动结束后短暂抑制 click，避免误触折叠
|
|
227
|
+
window.setTimeout(function () { movedRef.current = false }, 0)
|
|
228
|
+
}
|
|
229
|
+
dragRef.current = null
|
|
230
|
+
}
|
|
231
|
+
window.addEventListener('mousemove', onMove)
|
|
232
|
+
window.addEventListener('mouseup', onUp)
|
|
233
|
+
return function () {
|
|
234
|
+
window.removeEventListener('mousemove', onMove)
|
|
235
|
+
window.removeEventListener('mouseup', onUp)
|
|
236
|
+
}
|
|
237
|
+
}, [])
|
|
238
|
+
|
|
239
|
+
const startDrag = function (e) {
|
|
240
|
+
if (e.button !== 0) return
|
|
241
|
+
const el = rootRef.current
|
|
242
|
+
if (!el) return
|
|
243
|
+
const r = el.getBoundingClientRect()
|
|
244
|
+
dragRef.current = { sx: e.clientX, sy: e.clientY, ox: r.left, oy: r.top, w: r.width, h: r.height, moved: false }
|
|
245
|
+
e.preventDefault()
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
const toggleCollapse = function () {
|
|
249
|
+
if (movedRef.current) return // 刚才是在拖，不是点
|
|
250
|
+
setCollapsed(!collapsed)
|
|
251
|
+
}
|
|
252
|
+
|
|
253
|
+
// 双击头部复位到默认位置（右下角）
|
|
254
|
+
const resetPos = function (e) {
|
|
255
|
+
e.stopPropagation()
|
|
256
|
+
setPos(null)
|
|
257
|
+
}
|
|
258
|
+
|
|
259
|
+
|
|
260
|
+
const offline = state.offline === true
|
|
261
|
+
const stopped = state.stopped === true
|
|
262
|
+
const rows = state.timeline || []
|
|
263
|
+
const last = state.lastScan
|
|
264
|
+
|
|
265
|
+
const hit = function (path) {
|
|
266
|
+
setBusy(true)
|
|
267
|
+
fetch(BASE + path, { method: 'POST' })
|
|
268
|
+
.then(function (r) { return r.json() })
|
|
269
|
+
.then(function (j) { setState(function (s) { return Object.assign({}, s, { stopped: j.stopped }) }) })
|
|
270
|
+
.catch(function () {})
|
|
271
|
+
.finally(function () { setBusy(false) })
|
|
272
|
+
}
|
|
273
|
+
|
|
274
|
+
const dotClass = 'lensDot' + (offline ? ' idle' : stopped ? ' off' : '')
|
|
275
|
+
const stateText = offline ? 'OFFLINE' : stopped ? 'STOPPED' : (last && last.level ? last.level : 'READY')
|
|
276
|
+
|
|
277
|
+
const rootClass = 'lensHud' + (collapsed ? ' isCollapsed' : '') +
|
|
278
|
+
(stopped ? ' isStopped' : '') + (offline ? ' isOffline' : '')
|
|
279
|
+
|
|
280
|
+
// 拖动后的位置用 left/top 覆盖默认的 right/bottom
|
|
281
|
+
const posStyle = pos ? { left: pos.x + 'px', top: pos.y + 'px', right: 'auto', bottom: 'auto' } : null
|
|
282
|
+
|
|
283
|
+
const head = h('div', {
|
|
284
|
+
className: 'lensHead',
|
|
285
|
+
onMouseDown: startDrag,
|
|
286
|
+
onClick: toggleCollapse,
|
|
287
|
+
onDoubleClick: resetPos,
|
|
288
|
+
title: '拖动可移动 · 单击折叠 · 双击回到右下角',
|
|
289
|
+
},
|
|
290
|
+
h('span', { className: 'lensBrand' },
|
|
291
|
+
h('span', { className: dotClass }),
|
|
292
|
+
h('span', { className: 'lensName' }, 'LENS')),
|
|
293
|
+
h('span', { className: 'lensState' }, stateText),
|
|
294
|
+
h(Chevron))
|
|
295
|
+
|
|
296
|
+
if (collapsed) {
|
|
297
|
+
return h('div', { className: rootClass, ref: rootRef, style: posStyle },
|
|
298
|
+
h('style', null, CSS), h('div', { className: 'lensCard' }, head))
|
|
299
|
+
}
|
|
300
|
+
|
|
301
|
+
// ── 画面区：有截图显示截图，否则是一个安静的待命态 ──
|
|
302
|
+
const shot = last && last.overlay
|
|
303
|
+
? h('div', { className: 'lensShot' },
|
|
304
|
+
h('img', { src: BASE + '/thumb?t=' + last.at, alt: last.window || 'scan' }),
|
|
305
|
+
h('div', { className: 'lensShotBar' },
|
|
306
|
+
h('span', { className: 'lensWin', title: last.window }, last.window || '未命名窗口'),
|
|
307
|
+
last.candidates != null
|
|
308
|
+
? h('span', { className: 'lensChip' }, last.candidates + ' 候选')
|
|
309
|
+
: null))
|
|
310
|
+
: h('div', { className: 'lensVoid' },
|
|
311
|
+
h('span', { className: 'lensRing' }),
|
|
312
|
+
h('div', { className: 'lensVoidText' },
|
|
313
|
+
offline
|
|
314
|
+
? ['Lens 引擎未连上', h('br'), '检查插件 Host 半是否启动']
|
|
315
|
+
: [h('b', null, '待命中'), ' · 还没有扫描过窗口']))
|
|
316
|
+
|
|
317
|
+
const feed = rows.length
|
|
318
|
+
? h('div', null,
|
|
319
|
+
h('div', { className: 'lensFeedHead' },
|
|
320
|
+
h('span', null, '最近动作'),
|
|
321
|
+
h('span', null, rows.length + '')),
|
|
322
|
+
h('div', { className: 'lensRows' },
|
|
323
|
+
rows.slice(0, 20).map(function (r, i) {
|
|
324
|
+
return h('div', { className: 'lensRow', key: i },
|
|
325
|
+
h('span', { className: 'lensTime' }, clock(r.at)),
|
|
326
|
+
h('span', { className: 'lensAct' },
|
|
327
|
+
h('span', { className: 'lensVerb' }, String(r.kind || '').toUpperCase()),
|
|
328
|
+
h('span', { className: 'lensDet', title: r.detail }, r.detail || '')),
|
|
329
|
+
h('span', { className: 'lensOk' + (r.ok ? '' : ' bad') }))
|
|
330
|
+
})))
|
|
331
|
+
: null
|
|
332
|
+
|
|
333
|
+
const foot = h('div', { className: 'lensFoot' },
|
|
334
|
+
h('span', { className: 'lensHint' },
|
|
335
|
+
offline ? '引擎离线' : stopped ? '已拦截所有 computer_* 调用' : 'AI 随时可以接管桌面'),
|
|
336
|
+
h('button', {
|
|
337
|
+
className: 'lensStop' + (stopped ? ' armed' : ''),
|
|
338
|
+
disabled: busy || offline,
|
|
339
|
+
onClick: function (e) { e.stopPropagation(); hit(stopped ? '/resume' : '/stop') },
|
|
340
|
+
}, stopped ? '解除' : '急停'))
|
|
341
|
+
|
|
342
|
+
return h('div', { className: rootClass, ref: rootRef, style: posStyle },
|
|
343
|
+
h('style', null, CSS),
|
|
344
|
+
h('div', { className: 'lensCard' },
|
|
345
|
+
head,
|
|
346
|
+
h('div', { className: 'lensBody', onMouseDown: startDrag }, shot, feed),
|
|
347
|
+
foot))
|
|
348
|
+
}
|
|
349
|
+
|
|
350
|
+
return {
|
|
351
|
+
inject: ['slots'],
|
|
352
|
+
apply(ctx) {
|
|
353
|
+
ctx.slots.inject('shell.overlay', function () {
|
|
354
|
+
return ctx.slots.register(
|
|
355
|
+
{ name: 'shell.overlay', id: 'lens-hud', order: 90, label: 'Lens HUD' },
|
|
356
|
+
Hud,
|
|
357
|
+
)
|
|
358
|
+
})
|
|
359
|
+
},
|
|
360
|
+
}
|
|
361
|
+
},
|
|
362
|
+
})
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# PPT 手搓式流程：新建文本框 → 输入文字 → 拖动调整位置
|
|
2
|
+
# 坐标用 w 前缀表示「相对窗口客户区原点」，窗口挪位置也不用改流程
|
|
3
|
+
window Workbench
|
|
4
|
+
focus
|
|
5
|
+
|
|
6
|
+
# 1) 在画布上拖出一个 320x80 的文本框
|
|
7
|
+
drag from=w220,514 to=w540,594 duration=700 settle=300
|
|
8
|
+
|
|
9
|
+
# 2) 输入文字（对象已选中，键盘直接落到它身上）
|
|
10
|
+
type text=季度报告 settle=300
|
|
11
|
+
|
|
12
|
+
# 3) 把它往右下挪一点
|
|
13
|
+
drag from=w260,544 to=w380,594 duration=600 settle=400
|
|
14
|
+
|
|
15
|
+
# 4) 确认画面确实变了
|
|
16
|
+
verify expect=changed
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# 复杂表格填写：按单元格定位逐格写入，最后断言某格内容真的出现了
|
|
2
|
+
window Workbench
|
|
3
|
+
focus
|
|
4
|
+
|
|
5
|
+
click cell=2,1 settle=200
|
|
6
|
+
type text=张三
|
|
7
|
+
|
|
8
|
+
click cell=2,2 settle=200
|
|
9
|
+
type text=研发部
|
|
10
|
+
|
|
11
|
+
click cell=2,3 settle=200
|
|
12
|
+
type text=E10086
|
|
13
|
+
|
|
14
|
+
click cell=3,1 settle=200
|
|
15
|
+
type text=李四
|
|
16
|
+
|
|
17
|
+
# 取消选中，让格子恢复普通配色（选中格是白字压蓝底，OCR 读不出）
|
|
18
|
+
key combo=escape settle=300
|
|
19
|
+
|
|
20
|
+
# 断言语义：界面里现在应该能找到「研发部」
|
|
21
|
+
assert text=研发部
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
# 微信发消息：走搜索框，比在会话列表里翻找可靠得多
|
|
2
|
+
# 用法：lens run --flow flows/wechat-send.txt
|
|
3
|
+
# 或 computer_run { steps: [...] } 里逐行传入
|
|
4
|
+
#
|
|
5
|
+
# 为什么用 Ctrl+F 而不是点会话列表：
|
|
6
|
+
# 微信是 Qt 自绘窗口，UIA 基本为空，会话名只能靠 OCR 认，
|
|
7
|
+
# 而 Windows OCR 会认错（老师→老丝、取消→取肖），拿错词去匹配就会点错会话。
|
|
8
|
+
# 搜索框是"输入什么就筛什么"，不依赖 OCR，稳得多。
|
|
9
|
+
|
|
10
|
+
window 微信
|
|
11
|
+
focus
|
|
12
|
+
sleep ms=300
|
|
13
|
+
|
|
14
|
+
# 打开搜索面板（微信 4.x 实测 Ctrl+F 有效）
|
|
15
|
+
key combo=ctrl+f
|
|
16
|
+
sleep ms=400
|
|
17
|
+
|
|
18
|
+
# 输入联系人名（改成你要找的人）
|
|
19
|
+
type text=文件传输助手
|
|
20
|
+
sleep ms=600
|
|
21
|
+
|
|
22
|
+
# 回车进入第一个搜索结果（会话）
|
|
23
|
+
key combo=enter
|
|
24
|
+
sleep ms=500
|
|
25
|
+
|
|
26
|
+
# 点一下输入框，确保焦点在输入区
|
|
27
|
+
click x=900 y=900
|
|
28
|
+
sleep ms=200
|
|
29
|
+
|
|
30
|
+
# 发消息
|
|
31
|
+
type text=这是一条由 lens 自动发送的测试消息
|
|
32
|
+
sleep ms=300
|
|
33
|
+
key combo=enter
|
|
34
|
+
sleep ms=300
|
|
35
|
+
|
|
36
|
+
# 核验界面有变化（消息发出去了）
|
|
37
|
+
verify expect=changed
|
package/icon.svg
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" width="64" height="64">
|
|
2
|
+
<defs>
|
|
3
|
+
<linearGradient id="g" x1="0" y1="0" x2="1" y2="1">
|
|
4
|
+
<stop offset="0" stop-color="#00C88A"/>
|
|
5
|
+
<stop offset="1" stop-color="#0A84FF"/>
|
|
6
|
+
</linearGradient>
|
|
7
|
+
</defs>
|
|
8
|
+
<rect x="4" y="4" width="56" height="56" rx="14" fill="#0E1117"/>
|
|
9
|
+
<rect x="4.75" y="4.75" width="54.5" height="54.5" rx="13.25" fill="none" stroke="url(#g)" stroke-width="1.5" opacity="0.9"/>
|
|
10
|
+
<circle cx="32" cy="32" r="14" fill="none" stroke="url(#g)" stroke-width="2.5"/>
|
|
11
|
+
<path d="M32 12v10M32 42v10M12 32h10M42 32h10" stroke="url(#g)" stroke-width="2.5" stroke-linecap="round"/>
|
|
12
|
+
<circle cx="32" cy="32" r="3.2" fill="url(#g)"/>
|
|
13
|
+
</svg>
|