dsh-win-cua 0.0.0-stage → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +35 -0
- package/NOTICE.md +42 -0
- package/README.md +106 -2
- package/cordis.patch.yml +10 -0
- package/icon.svg +6 -0
- package/index.js +1095 -0
- package/locale/en.json +6 -0
- package/locale/zh.json +6 -0
- package/package.json +44 -3
package/LICENSE
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 WJPGKZ
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
Third-party notices
|
|
26
|
+
|
|
27
|
+
koffi (https://www.npmjs.com/package/koffi) is loaded at runtime from the host
|
|
28
|
+
DSH profile; it is not bundled with this package. koffi is distributed under
|
|
29
|
+
the MIT license by its own authors.
|
|
30
|
+
|
|
31
|
+
The Win32 and GDI function signatures and the per-monitor DPI awareness setup
|
|
32
|
+
follow the publicly documented Windows API. The approach of driving those APIs
|
|
33
|
+
through koffi was learned from Coopanion's Cortico desktop-pet extension
|
|
34
|
+
(https://github.com/Pal-AI-Lab/Coopanion), which is not copied here; this file
|
|
35
|
+
is an independent implementation of the same public API surface.
|
package/NOTICE.md
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
# Third-party notices
|
|
2
|
+
|
|
3
|
+
`dsh-win32-cua` loads and drives Windows system libraries. It ships no
|
|
4
|
+
third-party code of its own beyond what is listed here.
|
|
5
|
+
|
|
6
|
+
## koffi
|
|
7
|
+
|
|
8
|
+
- Package: [`koffi`](https://www.npmjs.com/package/koffi)
|
|
9
|
+
- License: MIT
|
|
10
|
+
- Use: resolved at runtime from the DSH profile's `node_modules`, never bundled.
|
|
11
|
+
- Why: koffi is the FFI layer that calls `user32.dll` and `gdi32.dll` from Node.
|
|
12
|
+
|
|
13
|
+
## Windows API
|
|
14
|
+
|
|
15
|
+
`SendInput`, `SetCursorPos`, `GetCursorPos`, `GetSystemMetrics`, `EnumWindows`,
|
|
16
|
+
`GetWindowTextW`, `GetWindowRect`, `GetWindowThreadProcessId`,
|
|
17
|
+
`GetForegroundWindow`, `SetForegroundWindow`, `ShowWindow`, `BringWindowToTop`,
|
|
18
|
+
`GetDC`, `ReleaseDC`, `SetProcessDpiAwarenessContext` (user32) and `BitBlt`,
|
|
19
|
+
`CreateCompatibleDC`, `CreateCompatibleBitmap`, `SelectObject`, `DeleteObject`,
|
|
20
|
+
`DeleteDC`, `GetDIBits` (gdi32) are documented public Windows APIs. The
|
|
21
|
+
signatures in `index.js` follow the Microsoft documentation; no proprietary
|
|
22
|
+
header or binary is redistributed.
|
|
23
|
+
|
|
24
|
+
## Coopanion (inspiration, not code)
|
|
25
|
+
|
|
26
|
+
- Project: [Pal-AI-Lab/Coopanion](https://github.com/Pal-AI-Lab/Coopanion)
|
|
27
|
+
- Relationship: the **approach** of driving Win32 through koffi, capturing the
|
|
28
|
+
screen with `BitBlt | CAPTUREBLT`, and declaring per-monitor DPI awareness was
|
|
29
|
+
learned by reading Coopanion's Cortico `desktop-pet` extension
|
|
30
|
+
(`src/cua/engine/win32.ts`).
|
|
31
|
+
- What was **not** taken: no source file, no type declaration, no string, no
|
|
32
|
+
struct layout and no code path is copied from that project. `index.js` here is
|
|
33
|
+
an independent implementation written against the public Win32 documentation,
|
|
34
|
+
with its own PNG encoder, its own key table and its own tool surface.
|
|
35
|
+
- Coopanion is used as a reference for what is possible, not as a dependency.
|
|
36
|
+
|
|
37
|
+
## DSH
|
|
38
|
+
|
|
39
|
+
- Project: [DeepSeek Harness](https://github.com/deepseek-ai/deepseek-harness)
|
|
40
|
+
- The `dsh.bundle` / `cordis.patch.yml` bundle format and the `ctx.tools`
|
|
41
|
+
registration contract follow the DSH plugin documentation shipped with the
|
|
42
|
+
DSH agent preset skills.
|
package/README.md
CHANGED
|
@@ -1,3 +1,107 @@
|
|
|
1
|
-
#
|
|
1
|
+
# dsh-win-cua
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Windows computer-use for DeepSeek Harness: **real** mouse and keyboard input plus screen
|
|
4
|
+
capture, with no external driver.
|
|
5
|
+
|
|
6
|
+
## How it works
|
|
7
|
+
|
|
8
|
+
| Capability | Win32 API |
|
|
9
|
+
|---|---|
|
|
10
|
+
| Mouse move, click, wheel | `user32!SendInput` (INPUT_MOUSE) |
|
|
11
|
+
| Keyboard taps and shortcuts | `user32!SendInput` (INPUT_KEYBOARD) |
|
|
12
|
+
| Type literal text (incl. CJK) | `SendInput` with `KEYEVENTF_UNICODE` |
|
|
13
|
+
| Screen capture | `gdi32!BitBlt` + `CAPTUREBLT` (layered windows included) |
|
|
14
|
+
| Window list / focus | `user32!EnumWindows`, `ShowWindow`, `SetForegroundWindow` |
|
|
15
|
+
| DPI | `SetProcessDpiAwarenessContext(-4)` — per-monitor v2 |
|
|
16
|
+
|
|
17
|
+
FFI is [koffi](https://www.npmjs.com/package/koffi). Coordinates are physical pixels of
|
|
18
|
+
the virtual screen; the primary display's top-left is `(0, 0)`.
|
|
19
|
+
|
|
20
|
+
## Tools
|
|
21
|
+
|
|
22
|
+
**Look**
|
|
23
|
+
- `cua_screenshot` — capture the whole virtual screen to a PNG
|
|
24
|
+
- `cua_screenshot_window` — capture one window by handle (`PrintWindow`, screen-blit fallback)
|
|
25
|
+
- `cua_zoom` — capture a region and upscale it x1–x4 for reading small UI text
|
|
26
|
+
- `cua_windows` — visible top-level windows with bounds and flags
|
|
27
|
+
- `cua_app_list` — same list enriched with the process executable name
|
|
28
|
+
- `cua_cursor` — pointer position and virtual-screen geometry
|
|
29
|
+
|
|
30
|
+
**Act**
|
|
31
|
+
- `cua_mouse_move` — move the pointer along a human path
|
|
32
|
+
- `cua_mouse_click` — click / double / right / middle, optionally at x,y
|
|
33
|
+
- `cua_drag` — press, travel and release with the button held
|
|
34
|
+
- `cua_scroll` — wheel, vertical or horizontal
|
|
35
|
+
- `cua_key` — one key with optional modifiers
|
|
36
|
+
- `cua_type` — type text (unicode, incl. surrogate pairs)
|
|
37
|
+
|
|
38
|
+
**Window & app**
|
|
39
|
+
- `cua_focus_window` — restore and foreground a window
|
|
40
|
+
- `cua_wait` — sleep and/or poll until a window title appears
|
|
41
|
+
- `cua_app_launch` — launch an executable without a shell
|
|
42
|
+
|
|
43
|
+
Every tool declares `output { schema, render }` — the DSH tool registry rejects raw
|
|
44
|
+
definitions without it, so keep that shape when adding tools. Read-only tools are marked
|
|
45
|
+
`readOnly` and register as `isConcurrencySafe`.
|
|
46
|
+
|
|
47
|
+
## Config
|
|
48
|
+
|
|
49
|
+
Set in this bundle's `cordis.patch.yml` under `config`:
|
|
50
|
+
|
|
51
|
+
| Key | Default | Meaning |
|
|
52
|
+
|---|---|---|
|
|
53
|
+
| `humanize` | `true` | Pointer travels a smoothstep path, typing and clicking keep a human cadence |
|
|
54
|
+
| `keepScreenshots` | `30` | Newest captures kept per directory |
|
|
55
|
+
| `moveSteps` | `16` | Max pointer steps for a humanized move |
|
|
56
|
+
| `keyDelayMs` | `[12, 34]` | Per-character delay range for humanized typing |
|
|
57
|
+
| `clickSettleMs` | `[60, 140]` | Pause between the pointer arriving and the button press |
|
|
58
|
+
| `zoomFactor` | `2` | Default `cua_zoom` upscale factor (hard cap 4) |
|
|
59
|
+
| `zoomMaxPixels` | `24000000` | Largest upscaled image allowed |
|
|
60
|
+
| `maxWaitMs` | `10000` | Upper bound for `cua_wait` |
|
|
61
|
+
| `allowLaunch` | `true` | Set `false` to disable `cua_app_launch` |
|
|
62
|
+
|
|
63
|
+
Pointer targets are clamped into the virtual screen, and astral code points (emoji,
|
|
64
|
+
CJK extension B) are sent as surrogate pairs so `KEYEVENTF_UNICODE` types them correctly.
|
|
65
|
+
|
|
66
|
+
## No external driver
|
|
67
|
+
|
|
68
|
+
Unlike `dsh-computer-use` (which shells out to the `cua-driver` binary) this plugin calls
|
|
69
|
+
Win32 through koffi **inside the host process**: nothing to install, no daemon, no
|
|
70
|
+
`CUA_DRIVER_BIN`. The trade-off is honest: without the driver there is no UIA/accessibility
|
|
71
|
+
tree, so targeting is pixel-based — use `cua_zoom` / `cua_screenshot_window` to read a
|
|
72
|
+
window precisely before clicking.
|
|
73
|
+
|
|
74
|
+
## Install
|
|
75
|
+
|
|
76
|
+
From npm:
|
|
77
|
+
|
|
78
|
+
```powershell
|
|
79
|
+
dsh plugin --profile <profile> add dsh-win-cua
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Or from a checkout / local directory:
|
|
83
|
+
|
|
84
|
+
```powershell
|
|
85
|
+
dsh plugin --profile <profile> add <this directory>
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Or in the Desktop app: Plugin Manager → install bundle → pick this directory.
|
|
89
|
+
|
|
90
|
+
Windows only. `SendInput` injects into the session whose input queue is active — a window
|
|
91
|
+
on a locked or disconnected desktop receives nothing.
|
|
92
|
+
|
|
93
|
+
## Changelog
|
|
94
|
+
|
|
95
|
+
### 0.2.0
|
|
96
|
+
|
|
97
|
+
- New look tools: `cua_screenshot_window` (PrintWindow with screen-blit fallback),
|
|
98
|
+
`cua_zoom` (region capture upscaled x1–x4), `cua_app_list` (windows with process names).
|
|
99
|
+
- New act tools: `cua_drag`, `cua_wait`, `cua_app_launch` (disabled with `allowLaunch: false`).
|
|
100
|
+
- `humanize` is now a real config knob: smoothstep pointer path, per-character typing
|
|
101
|
+
delay and click settle range, with `moveSteps` / `keyDelayMs` / `clickSettleMs` tuning.
|
|
102
|
+
- PNG encoder, screenshot retention (`keepScreenshots`) and coordinate clamping tightened.
|
|
103
|
+
|
|
104
|
+
### 0.1.0
|
|
105
|
+
|
|
106
|
+
First public release: screenshot, cursor, window list/focus, mouse move/click/scroll,
|
|
107
|
+
key press and unicode typing.
|
package/cordis.patch.yml
ADDED
package/icon.svg
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64" width="64" height="64">
|
|
2
|
+
<rect x="4" y="8" width="56" height="40" rx="5" fill="#1f2a3a" stroke="#4da3ff" stroke-width="2.5"/>
|
|
3
|
+
<path d="M10 18h14M10 26h22M10 34h10" stroke="#4da3ff" stroke-width="2.5" stroke-linecap="round"/>
|
|
4
|
+
<path d="M38 22l12 7-5 1.5 3.2 6.2-3.4 1.6-3-6.1-3.6 3.4z" fill="#7ee0a1" stroke="#e8f4ff" stroke-width="1.4" stroke-linejoin="round"/>
|
|
5
|
+
<rect x="22" y="50" width="20" height="5" rx="2.5" fill="#4da3ff"/>
|
|
6
|
+
</svg>
|
package/index.js
ADDED
|
@@ -0,0 +1,1095 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* dsh-win-cua — Windows computer-use for DeepSeek Harness.
|
|
3
|
+
*
|
|
4
|
+
* Real mouse and keyboard input through user32 `SendInput`, screen capture through
|
|
5
|
+
* GDI `BitBlt` (CAPTUREBLT so layered windows are included), and a top-level window
|
|
6
|
+
* list through `EnumWindows`. The FFI layer is koffi, the same approach Coopanion's
|
|
7
|
+
* desktop-pet uses; no external driver process is involved.
|
|
8
|
+
*
|
|
9
|
+
* Coordinates are physical pixels of the virtual screen (primary top-left is 0,0),
|
|
10
|
+
* because the process declares itself per-monitor DPI aware before anything else.
|
|
11
|
+
*/
|
|
12
|
+
import { deflateSync } from 'node:zlib';
|
|
13
|
+
import { createRequire } from 'node:module';
|
|
14
|
+
import { mkdirSync, writeFileSync, readdirSync, unlinkSync, existsSync } from 'node:fs';
|
|
15
|
+
import { spawn } from 'node:child_process';
|
|
16
|
+
import { join } from 'node:path';
|
|
17
|
+
import { tmpdir } from 'node:os';
|
|
18
|
+
|
|
19
|
+
const SCREEN_DIR = join(tmpdir(), 'dsh-cua-screens');
|
|
20
|
+
|
|
21
|
+
/**
|
|
22
|
+
* Resolve koffi from wherever this profile keeps its packages. A bundle installed by
|
|
23
|
+
* link lives outside the profile, so a plain `import 'koffi'` cannot see it; the host
|
|
24
|
+
* exposes DSH_PROFILE_DIR for exactly that case. The fallbacks cover running from the
|
|
25
|
+
* profile itself and a checkout next to it.
|
|
26
|
+
*/
|
|
27
|
+
function loadKoffi() {
|
|
28
|
+
const candidates = [
|
|
29
|
+
process.env.DSH_PROFILE_DIR && join(process.env.DSH_PROFILE_DIR, 'node_modules', 'koffi'),
|
|
30
|
+
join(process.cwd(), 'node_modules', 'koffi'),
|
|
31
|
+
join(import.meta.dirname, 'node_modules', 'koffi'),
|
|
32
|
+
].filter(Boolean);
|
|
33
|
+
const errors = [];
|
|
34
|
+
for (const candidate of candidates) {
|
|
35
|
+
try {
|
|
36
|
+
return createRequire(join(candidate, 'noop.cjs'))(candidate);
|
|
37
|
+
} catch (error) {
|
|
38
|
+
errors.push(`${candidate}: ${error.message}`);
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
throw new Error(`cannot load koffi (tried ${candidates.length} locations): ${errors.join(' | ')}`);
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
let native = null;
|
|
45
|
+
|
|
46
|
+
/** Load Win32 through koffi once. */
|
|
47
|
+
function loadNative() {
|
|
48
|
+
if (native) return native;
|
|
49
|
+
if (process.platform !== 'win32') throw new Error('dsh-win32-cua only works on Windows');
|
|
50
|
+
const koffi = loadKoffi();
|
|
51
|
+
|
|
52
|
+
const user32 = koffi.load('user32.dll');
|
|
53
|
+
const gdi32 = koffi.load('gdi32.dll');
|
|
54
|
+
const kernel32 = koffi.load('kernel32.dll');
|
|
55
|
+
|
|
56
|
+
koffi.alias('HANDLE', 'void *');
|
|
57
|
+
koffi.alias('HWND', 'HANDLE');
|
|
58
|
+
|
|
59
|
+
const POINT = koffi.struct('POINT', { x: 'long', y: 'long' });
|
|
60
|
+
const RECT = koffi.struct('RECT', { left: 'long', top: 'long', right: 'long', bottom: 'long' });
|
|
61
|
+
const MOUSEINPUT = koffi.struct('MOUSEINPUT', {
|
|
62
|
+
dx: 'long', dy: 'long', mouseData: 'uint32_t', dwFlags: 'uint32_t',
|
|
63
|
+
time: 'uint32_t', dwExtraInfo: 'uintptr_t',
|
|
64
|
+
});
|
|
65
|
+
const KEYBDINPUT = koffi.struct('KEYBDINPUT', {
|
|
66
|
+
wVk: 'uint16_t', wScan: 'uint16_t', dwFlags: 'uint32_t',
|
|
67
|
+
time: 'uint32_t', dwExtraInfo: 'uintptr_t',
|
|
68
|
+
});
|
|
69
|
+
const HARDWAREINPUT = koffi.struct('HARDWAREINPUT', {
|
|
70
|
+
uMsg: 'uint32_t', wParamL: 'uint16_t', wParamH: 'uint16_t',
|
|
71
|
+
});
|
|
72
|
+
const INPUT = koffi.struct('INPUT', {
|
|
73
|
+
type: 'uint32_t',
|
|
74
|
+
u: koffi.union({ mi: MOUSEINPUT, ki: KEYBDINPUT, hi: HARDWAREINPUT }),
|
|
75
|
+
});
|
|
76
|
+
const BITMAPINFOHEADER = koffi.struct('BITMAPINFOHEADER', {
|
|
77
|
+
biSize: 'uint32_t', biWidth: 'int32_t', biHeight: 'int32_t',
|
|
78
|
+
biPlanes: 'uint16_t', biBitCount: 'uint16_t', biCompression: 'uint32_t',
|
|
79
|
+
biSizeImage: 'uint32_t', biXPelsPerMeter: 'int32_t', biYPelsPerMeter: 'int32_t',
|
|
80
|
+
biClrUsed: 'uint32_t', biClrImportant: 'uint32_t',
|
|
81
|
+
});
|
|
82
|
+
const EnumWindowsProc = koffi.proto('bool __stdcall EnumWindowsProc(HWND hwnd, intptr_t lParam)');
|
|
83
|
+
|
|
84
|
+
const api = {
|
|
85
|
+
koffi,
|
|
86
|
+
SetProcessDpiAwarenessContext: user32.func('bool __stdcall SetProcessDpiAwarenessContext(intptr_t value)'),
|
|
87
|
+
GetSystemMetrics: user32.func('int __stdcall GetSystemMetrics(int index)'),
|
|
88
|
+
SetCursorPos: user32.func('bool __stdcall SetCursorPos(int x, int y)'),
|
|
89
|
+
GetCursorPos: user32.func('bool __stdcall GetCursorPos(_Out_ POINT *p)'),
|
|
90
|
+
SendInput: user32.func('uint32_t __stdcall SendInput(uint32_t count, INPUT *inputs, int size)'),
|
|
91
|
+
GetDC: user32.func('HANDLE __stdcall GetDC(HWND hwnd)'),
|
|
92
|
+
ReleaseDC: user32.func('int __stdcall ReleaseDC(HWND hwnd, HANDLE dc)'),
|
|
93
|
+
EnumWindows: user32.func('bool __stdcall EnumWindows(EnumWindowsProc *cb, intptr_t lParam)'),
|
|
94
|
+
IsWindowVisible: user32.func('bool __stdcall IsWindowVisible(HWND hwnd)'),
|
|
95
|
+
IsIconic: user32.func('bool __stdcall IsIconic(HWND hwnd)'),
|
|
96
|
+
GetWindowTextW: user32.func('int __stdcall GetWindowTextW(HWND hwnd, _Out_ uint16_t *buf, int max)'),
|
|
97
|
+
GetWindowRect: user32.func('bool __stdcall GetWindowRect(HWND hwnd, _Out_ RECT *rect)'),
|
|
98
|
+
GetWindowThreadProcessId: user32.func('uint32_t __stdcall GetWindowThreadProcessId(HWND hwnd, _Out_ uint32_t *pid)'),
|
|
99
|
+
GetForegroundWindow: user32.func('HWND __stdcall GetForegroundWindow()'),
|
|
100
|
+
SetForegroundWindow: user32.func('bool __stdcall SetForegroundWindow(HWND hwnd)'),
|
|
101
|
+
ShowWindow: user32.func('bool __stdcall ShowWindow(HWND hwnd, int cmd)'),
|
|
102
|
+
BringWindowToTop: user32.func('bool __stdcall BringWindowToTop(HWND hwnd)'),
|
|
103
|
+
CreateCompatibleDC: gdi32.func('HANDLE __stdcall CreateCompatibleDC(HANDLE dc)'),
|
|
104
|
+
CreateCompatibleBitmap: gdi32.func('HANDLE __stdcall CreateCompatibleBitmap(HANDLE dc, int w, int h)'),
|
|
105
|
+
SelectObject: gdi32.func('HANDLE __stdcall SelectObject(HANDLE dc, HANDLE obj)'),
|
|
106
|
+
DeleteObject: gdi32.func('bool __stdcall DeleteObject(HANDLE obj)'),
|
|
107
|
+
DeleteDC: gdi32.func('bool __stdcall DeleteDC(HANDLE dc)'),
|
|
108
|
+
BitBlt: gdi32.func('bool __stdcall BitBlt(HANDLE dst, int x, int y, int w, int h, HANDLE src, int sx, int sy, uint32_t rop)'),
|
|
109
|
+
GetDIBits: gdi32.func('int __stdcall GetDIBits(HANDLE dc, HANDLE bmp, uint32_t start, uint32_t lines, void *bits, _Inout_ BITMAPINFOHEADER *bmi, uint32_t usage)'),
|
|
110
|
+
PrintWindow: user32.func('bool __stdcall PrintWindow(HWND hwnd, HANDLE dc, uint32_t flags)'),
|
|
111
|
+
GetWindowDC: user32.func('HANDLE __stdcall GetWindowDC(HWND hwnd)'),
|
|
112
|
+
OpenProcess: kernel32.func('HANDLE __stdcall OpenProcess(uint32_t access, bool inherit, uint32_t pid)'),
|
|
113
|
+
QueryFullProcessImageNameW: kernel32.func('bool __stdcall QueryFullProcessImageNameW(HANDLE proc, uint32_t flags, _Out_ uint16_t *buf, _Inout_ uint32_t *size)'),
|
|
114
|
+
CloseHandle: kernel32.func('bool __stdcall CloseHandle(HANDLE handle)'),
|
|
115
|
+
structs: { POINT, RECT, INPUT, BITMAPINFOHEADER },
|
|
116
|
+
};
|
|
117
|
+
try {
|
|
118
|
+
// -4 = DPI_AWARENESS_CONTEXT_PER_MONITOR_AWARE_V2
|
|
119
|
+
api.SetProcessDpiAwarenessContext(-4);
|
|
120
|
+
} catch {
|
|
121
|
+
/* older Windows: best effort */
|
|
122
|
+
}
|
|
123
|
+
native = api;
|
|
124
|
+
return api;
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
const SRCCOPY = 0x00cc0020;
|
|
128
|
+
const CAPTUREBLT = 0x40000000;
|
|
129
|
+
const INPUT_MOUSE = 0;
|
|
130
|
+
const INPUT_KEYBOARD = 1;
|
|
131
|
+
const MOUSEEVENTF_MOVE = 0x0001;
|
|
132
|
+
const MOUSEEVENTF_LEFTDOWN = 0x0002;
|
|
133
|
+
const MOUSEEVENTF_LEFTUP = 0x0004;
|
|
134
|
+
const MOUSEEVENTF_RIGHTDOWN = 0x0008;
|
|
135
|
+
const MOUSEEVENTF_RIGHTUP = 0x0010;
|
|
136
|
+
const MOUSEEVENTF_MIDDLEDOWN = 0x0020;
|
|
137
|
+
const MOUSEEVENTF_MIDDLEUP = 0x0040;
|
|
138
|
+
const MOUSEEVENTF_WHEEL = 0x0800;
|
|
139
|
+
const MOUSEEVENTF_ABSOLUTE = 0x8000;
|
|
140
|
+
const KEYEVENTF_KEYUP = 0x0002;
|
|
141
|
+
const KEYEVENTF_UNICODE = 0x0004;
|
|
142
|
+
const MAX_COORD = 65535;
|
|
143
|
+
const SW_RESTORE = 9;
|
|
144
|
+
const MOUSEEVENTF_HWHEEL = 0x01000;
|
|
145
|
+
const PROCESS_QUERY_LIMITED_INFORMATION = 0x1000;
|
|
146
|
+
const PW_RENDERFULLCONTENT = 0x00000002;
|
|
147
|
+
const MAX_CAPTURE_PIXELS = 40_000_000;
|
|
148
|
+
|
|
149
|
+
/** Behaviour defaults; the host may override them through cordis.patch.yml config. */
|
|
150
|
+
const DEFAULTS = {
|
|
151
|
+
humanize: true,
|
|
152
|
+
keepScreenshots: 30,
|
|
153
|
+
moveSteps: 16,
|
|
154
|
+
keyDelayMs: [12, 34],
|
|
155
|
+
clickSettleMs: [60, 140],
|
|
156
|
+
zoomFactor: 2,
|
|
157
|
+
zoomMaxPixels: 24_000_000,
|
|
158
|
+
maxWaitMs: 10_000,
|
|
159
|
+
allowLaunch: true,
|
|
160
|
+
};
|
|
161
|
+
const CFG = { ...DEFAULTS };
|
|
162
|
+
|
|
163
|
+
const sleep = (ms) => new Promise((resolve) => { setTimeout(resolve, ms); });
|
|
164
|
+
const between = ([min, max]) => min + Math.random() * Math.max(0, max - min);
|
|
165
|
+
|
|
166
|
+
/**
|
|
167
|
+
* Raw `tools.register()` does not validate arguments the way `defineTool` does,
|
|
168
|
+
* so every coordinate has to be checked here — otherwise a missing number turns
|
|
169
|
+
* into a NaN deep inside Buffer.alloc instead of a message the model can act on.
|
|
170
|
+
*/
|
|
171
|
+
function requireNumber(value, name) {
|
|
172
|
+
const n = Number(value);
|
|
173
|
+
if (!Number.isFinite(n)) throw new Error(`${name} must be a finite number (received ${JSON.stringify(value)})`);
|
|
174
|
+
return n;
|
|
175
|
+
}
|
|
176
|
+
|
|
177
|
+
/** Clamp a pointer target into the virtual screen so a bad coordinate cannot fly off. */
|
|
178
|
+
function clampPoint(A, x, y) {
|
|
179
|
+
const screen = virtualScreen(A);
|
|
180
|
+
const px = Math.round(requireNumber(x, 'x'));
|
|
181
|
+
const py = Math.round(requireNumber(y, 'y'));
|
|
182
|
+
return {
|
|
183
|
+
x: Math.min(Math.max(px, screen.x), screen.x + Math.max(0, screen.width - 1)),
|
|
184
|
+
y: Math.min(Math.max(py, screen.y), screen.y + Math.max(0, screen.height - 1)),
|
|
185
|
+
};
|
|
186
|
+
}
|
|
187
|
+
|
|
188
|
+
/**
|
|
189
|
+
* UTF-16 code units one KEYEVENTF_UNICODE scan needs. Astral code points
|
|
190
|
+
* (emoji, CJK extensions) travel as a surrogate pair; sending the raw code
|
|
191
|
+
* point would truncate to 16 bits and type the wrong glyph.
|
|
192
|
+
*/
|
|
193
|
+
function unicodeUnits(code) {
|
|
194
|
+
if (code <= 0xffff) return [code];
|
|
195
|
+
const rest = code - 0x10000;
|
|
196
|
+
return [0xd800 + ((rest >> 10) & 0x3ff), 0xdc00 + (rest & 0x3ff)];
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
/** Build a mouse INPUT record. */
|
|
200
|
+
function mouseInput({ dx = 0, dy = 0, data = 0, flags = 0 }) {
|
|
201
|
+
return {
|
|
202
|
+
type: INPUT_MOUSE,
|
|
203
|
+
u: { mi: { dx, dy, mouseData: data, dwFlags: flags, time: 0, dwExtraInfo: 0 } },
|
|
204
|
+
};
|
|
205
|
+
}
|
|
206
|
+
|
|
207
|
+
/** Build a keyboard INPUT record. */
|
|
208
|
+
function keyInput(vk, flags, scan = 0) {
|
|
209
|
+
return {
|
|
210
|
+
type: INPUT_KEYBOARD,
|
|
211
|
+
u: { ki: { wVk: vk, wScan: scan, dwFlags: flags, time: 0, dwExtraInfo: 0 } },
|
|
212
|
+
};
|
|
213
|
+
}
|
|
214
|
+
|
|
215
|
+
/** Send INPUT records; koffi converts a plain array of structs in place. */
|
|
216
|
+
function sendInputs(A, records) {
|
|
217
|
+
const sent = A.SendInput(records.length, records, A.koffi.sizeof(A.structs.INPUT));
|
|
218
|
+
if (sent !== records.length) throw new Error(`SendInput accepted ${sent}/${records.length} records`);
|
|
219
|
+
}
|
|
220
|
+
|
|
221
|
+
function sendInput(A, record) {
|
|
222
|
+
sendInputs(A, [record]);
|
|
223
|
+
}
|
|
224
|
+
|
|
225
|
+
/** Virtual-screen origin and size in physical pixels. */
|
|
226
|
+
function virtualScreen(A) {
|
|
227
|
+
return {
|
|
228
|
+
x: A.GetSystemMetrics(76),
|
|
229
|
+
y: A.GetSystemMetrics(77),
|
|
230
|
+
width: A.GetSystemMetrics(78),
|
|
231
|
+
height: A.GetSystemMetrics(79),
|
|
232
|
+
};
|
|
233
|
+
}
|
|
234
|
+
|
|
235
|
+
/** Screen pixels to the 0..65535 absolute space. */
|
|
236
|
+
function toAbsolute(A, x, y) {
|
|
237
|
+
const screen = virtualScreen(A);
|
|
238
|
+
return {
|
|
239
|
+
ax: Math.round(((x - screen.x) * MAX_COORD) / Math.max(1, screen.width - 1)),
|
|
240
|
+
ay: Math.round(((y - screen.y) * MAX_COORD) / Math.max(1, screen.height - 1)),
|
|
241
|
+
};
|
|
242
|
+
}
|
|
243
|
+
|
|
244
|
+
/** Pull BGRA rows out of a selected bitmap. */
|
|
245
|
+
function readBits(A, dc, bmp, width, height) {
|
|
246
|
+
if (width * height > MAX_CAPTURE_PIXELS) throw new Error(`capture too large: ${width}x${height}`);
|
|
247
|
+
const bmi = {
|
|
248
|
+
biSize: A.koffi.sizeof(A.structs.BITMAPINFOHEADER),
|
|
249
|
+
biWidth: width, biHeight: -height, biPlanes: 1, biBitCount: 32,
|
|
250
|
+
biCompression: 0, biSizeImage: 0, biXPelsPerMeter: 0, biYPelsPerMeter: 0,
|
|
251
|
+
biClrUsed: 0, biClrImportant: 0,
|
|
252
|
+
};
|
|
253
|
+
const buffer = Buffer.alloc(width * height * 4);
|
|
254
|
+
const lines = A.GetDIBits(dc, bmp, 0, height, buffer, bmi, 0);
|
|
255
|
+
if (lines !== height) throw new Error(`GetDIBits returned ${lines}/${height} lines`);
|
|
256
|
+
return { width, height, data: buffer };
|
|
257
|
+
}
|
|
258
|
+
|
|
259
|
+
/** BitBlt a screen rectangle (virtual-screen coordinates) into a fresh memory bitmap. */
|
|
260
|
+
function captureRect(A, sx, sy, width, height) {
|
|
261
|
+
const left = Math.round(requireNumber(sx, 'x'));
|
|
262
|
+
const top = Math.round(requireNumber(sy, 'y'));
|
|
263
|
+
const w = Math.max(1, Math.round(requireNumber(width, 'width')));
|
|
264
|
+
const h = Math.max(1, Math.round(requireNumber(height, 'height')));
|
|
265
|
+
const srcDc = A.GetDC(null);
|
|
266
|
+
if (!srcDc) throw new Error('GetDC returned null');
|
|
267
|
+
const memDc = A.CreateCompatibleDC(srcDc);
|
|
268
|
+
const bmp = A.CreateCompatibleBitmap(srcDc, w, h);
|
|
269
|
+
const old = A.SelectObject(memDc, bmp);
|
|
270
|
+
try {
|
|
271
|
+
if (!A.BitBlt(memDc, 0, 0, w, h, srcDc, left, top, SRCCOPY | CAPTUREBLT)) {
|
|
272
|
+
throw new Error('BitBlt failed');
|
|
273
|
+
}
|
|
274
|
+
return readBits(A, memDc, bmp, w, h);
|
|
275
|
+
} finally {
|
|
276
|
+
A.SelectObject(memDc, old);
|
|
277
|
+
A.DeleteObject(bmp);
|
|
278
|
+
A.DeleteDC(memDc);
|
|
279
|
+
A.ReleaseDC(null, srcDc);
|
|
280
|
+
}
|
|
281
|
+
}
|
|
282
|
+
|
|
283
|
+
/** Grab the whole virtual screen as raw BGRA rows. */
|
|
284
|
+
function captureScreen(A) {
|
|
285
|
+
const screen = virtualScreen(A);
|
|
286
|
+
return captureRect(A, screen.x, screen.y, screen.width, screen.height);
|
|
287
|
+
}
|
|
288
|
+
|
|
289
|
+
/**
|
|
290
|
+
* Capture one window. PrintWindow asks the window to render itself into our DC
|
|
291
|
+
* (so occluded windows still come out right); when the window refuses, fall back
|
|
292
|
+
* to blitting its on-screen rectangle.
|
|
293
|
+
*/
|
|
294
|
+
function captureWindow(A, hwnd) {
|
|
295
|
+
const rect = { left: 0, top: 0, right: 0, bottom: 0 };
|
|
296
|
+
if (!A.GetWindowRect(hwnd, rect)) throw new Error(`GetWindowRect failed for handle ${hwnd}`);
|
|
297
|
+
const width = Math.max(1, rect.right - rect.left);
|
|
298
|
+
const height = Math.max(1, rect.bottom - rect.top);
|
|
299
|
+
const bounds = { x: rect.left, y: rect.top, width, height };
|
|
300
|
+
if (width * height > MAX_CAPTURE_PIXELS) throw new Error(`window too large to capture: ${width}x${height}`);
|
|
301
|
+
const winDc = A.GetWindowDC(hwnd);
|
|
302
|
+
if (!winDc) throw new Error(`GetWindowDC returned null for handle ${hwnd}`);
|
|
303
|
+
const memDc = A.CreateCompatibleDC(winDc);
|
|
304
|
+
const bmp = A.CreateCompatibleBitmap(winDc, width, height);
|
|
305
|
+
const old = A.SelectObject(memDc, bmp);
|
|
306
|
+
try {
|
|
307
|
+
const printed = A.PrintWindow(hwnd, memDc, PW_RENDERFULLCONTENT);
|
|
308
|
+
if (!printed) {
|
|
309
|
+
const screenDc = A.GetDC(null);
|
|
310
|
+
if (!screenDc) throw new Error('GetDC returned null');
|
|
311
|
+
try {
|
|
312
|
+
if (!A.BitBlt(memDc, 0, 0, width, height, screenDc, bounds.x, bounds.y, SRCCOPY | CAPTUREBLT)) {
|
|
313
|
+
throw new Error('window capture failed (PrintWindow and BitBlt both refused)');
|
|
314
|
+
}
|
|
315
|
+
} finally {
|
|
316
|
+
A.ReleaseDC(null, screenDc);
|
|
317
|
+
}
|
|
318
|
+
}
|
|
319
|
+
return { ...readBits(A, memDc, bmp, width, height), bounds, printed: Boolean(printed) };
|
|
320
|
+
} finally {
|
|
321
|
+
A.SelectObject(memDc, old);
|
|
322
|
+
A.DeleteObject(bmp);
|
|
323
|
+
A.DeleteDC(memDc);
|
|
324
|
+
A.ReleaseDC(hwnd, winDc);
|
|
325
|
+
}
|
|
326
|
+
}
|
|
327
|
+
|
|
328
|
+
/** Nearest-neighbour upscale: crisp pixels, which is what reading small UI text needs. */
|
|
329
|
+
function scaleBgra({ width, height, data }, factor) {
|
|
330
|
+
const scale = Math.max(1, Math.floor(factor));
|
|
331
|
+
if (scale === 1) return { width, height, data };
|
|
332
|
+
const ow = width * scale;
|
|
333
|
+
const oh = height * scale;
|
|
334
|
+
if (ow * oh > MAX_CAPTURE_PIXELS) throw new Error(`zoomed image too large: ${ow}x${oh}`);
|
|
335
|
+
const out = Buffer.alloc(ow * oh * 4);
|
|
336
|
+
for (let y = 0; y < oh; y += 1) {
|
|
337
|
+
const rowBase = Math.min(height - 1, Math.floor(y / scale)) * width;
|
|
338
|
+
const target = y * ow;
|
|
339
|
+
for (let x = 0; x < ow; x += 1) {
|
|
340
|
+
const s = (rowBase + Math.min(width - 1, Math.floor(x / scale))) * 4;
|
|
341
|
+
const t = (target + x) * 4;
|
|
342
|
+
out[t] = data[s];
|
|
343
|
+
out[t + 1] = data[s + 1];
|
|
344
|
+
out[t + 2] = data[s + 2];
|
|
345
|
+
out[t + 3] = data[s + 3];
|
|
346
|
+
}
|
|
347
|
+
}
|
|
348
|
+
return { width: ow, height: oh, data: out };
|
|
349
|
+
}
|
|
350
|
+
|
|
351
|
+
const CRC_TABLE = (() => {
|
|
352
|
+
const table = new Uint32Array(256);
|
|
353
|
+
for (let n = 0; n < 256; n += 1) {
|
|
354
|
+
let c = n;
|
|
355
|
+
for (let k = 0; k < 8; k += 1) c = c & 1 ? 0xedb88320 ^ (c >>> 1) : c >>> 1;
|
|
356
|
+
table[n] = c >>> 0;
|
|
357
|
+
}
|
|
358
|
+
return table;
|
|
359
|
+
})();
|
|
360
|
+
|
|
361
|
+
function crc32(buffer) {
|
|
362
|
+
let c = 0xffffffff;
|
|
363
|
+
for (let i = 0; i < buffer.length; i += 1) c = CRC_TABLE[(c ^ buffer[i]) & 0xff] ^ (c >>> 8);
|
|
364
|
+
return (c ^ 0xffffffff) >>> 0;
|
|
365
|
+
}
|
|
366
|
+
|
|
367
|
+
function pngChunk(type, payload) {
|
|
368
|
+
const length = Buffer.alloc(4);
|
|
369
|
+
length.writeUInt32BE(payload.length, 0);
|
|
370
|
+
const body = Buffer.concat([Buffer.from(type, 'ascii'), payload]);
|
|
371
|
+
const crc = Buffer.alloc(4);
|
|
372
|
+
crc.writeUInt32BE(crc32(body), 0);
|
|
373
|
+
return Buffer.concat([length, body, crc]);
|
|
374
|
+
}
|
|
375
|
+
|
|
376
|
+
/** Encode BGRA rows as an RGBA PNG without an image library. */
|
|
377
|
+
function encodePng({ width, height, data }) {
|
|
378
|
+
const stride = width * 4;
|
|
379
|
+
const raw = Buffer.alloc((stride + 1) * height);
|
|
380
|
+
for (let y = 0; y < height; y += 1) {
|
|
381
|
+
const target = y * (stride + 1);
|
|
382
|
+
raw[target] = 0;
|
|
383
|
+
const source = y * stride;
|
|
384
|
+
for (let x = 0; x < width; x += 1) {
|
|
385
|
+
const s = source + x * 4;
|
|
386
|
+
const t = target + 1 + x * 4;
|
|
387
|
+
raw[t] = data[s + 2];
|
|
388
|
+
raw[t + 1] = data[s + 1];
|
|
389
|
+
raw[t + 2] = data[s];
|
|
390
|
+
raw[t + 3] = 255;
|
|
391
|
+
}
|
|
392
|
+
}
|
|
393
|
+
const header = Buffer.alloc(13);
|
|
394
|
+
header.writeUInt32BE(width, 0);
|
|
395
|
+
header.writeUInt32BE(height, 4);
|
|
396
|
+
header[8] = 8;
|
|
397
|
+
header[9] = 6;
|
|
398
|
+
return Buffer.concat([
|
|
399
|
+
Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
|
|
400
|
+
pngChunk('IHDR', header),
|
|
401
|
+
pngChunk('IDAT', deflateSync(raw, { level: 6 })),
|
|
402
|
+
pngChunk('IEND', Buffer.alloc(0)),
|
|
403
|
+
]);
|
|
404
|
+
}
|
|
405
|
+
|
|
406
|
+
/** Read a wide window title. */
|
|
407
|
+
function windowTitle(A, hwnd) {
|
|
408
|
+
const buffer = Buffer.alloc(512 * 2);
|
|
409
|
+
const length = A.GetWindowTextW(hwnd, buffer, 512);
|
|
410
|
+
return length > 0 ? buffer.toString('utf16le', 0, length * 2) : '';
|
|
411
|
+
}
|
|
412
|
+
|
|
413
|
+
/** Every visible top-level window with a title. */
|
|
414
|
+
function listWindows(A) {
|
|
415
|
+
const windows = [];
|
|
416
|
+
const foreground = Number(A.koffi.address(A.GetForegroundWindow()));
|
|
417
|
+
const proc = A.koffi.register((hwnd) => {
|
|
418
|
+
if (!A.IsWindowVisible(hwnd)) return true;
|
|
419
|
+
const title = windowTitle(A, hwnd);
|
|
420
|
+
if (!title) return true;
|
|
421
|
+
const rect = { left: 0, top: 0, right: 0, bottom: 0 };
|
|
422
|
+
A.GetWindowRect(hwnd, rect);
|
|
423
|
+
const pid = [0];
|
|
424
|
+
A.GetWindowThreadProcessId(hwnd, pid);
|
|
425
|
+
const handle = Number(A.koffi.address(hwnd));
|
|
426
|
+
windows.push({
|
|
427
|
+
handle,
|
|
428
|
+
title,
|
|
429
|
+
pid: pid[0],
|
|
430
|
+
minimized: Boolean(A.IsIconic(hwnd)),
|
|
431
|
+
active: handle === foreground,
|
|
432
|
+
bounds: {
|
|
433
|
+
x: rect.left,
|
|
434
|
+
y: rect.top,
|
|
435
|
+
width: rect.right - rect.left,
|
|
436
|
+
height: rect.bottom - rect.top,
|
|
437
|
+
},
|
|
438
|
+
});
|
|
439
|
+
return true;
|
|
440
|
+
}, A.koffi.pointer('EnumWindowsProc'));
|
|
441
|
+
try {
|
|
442
|
+
A.EnumWindows(proc, 0);
|
|
443
|
+
} finally {
|
|
444
|
+
A.koffi.unregister(proc);
|
|
445
|
+
}
|
|
446
|
+
return windows;
|
|
447
|
+
}
|
|
448
|
+
|
|
449
|
+
/** Virtual-key names a model is likely to use. */
|
|
450
|
+
const VIRTUAL_KEYS = {
|
|
451
|
+
enter: 0x0d, return: 0x0d, esc: 0x1b, escape: 0x1b, tab: 0x09, space: 0x20,
|
|
452
|
+
backspace: 0x08, delete: 0x2e, del: 0x2e, insert: 0x2d, home: 0x24, end: 0x23,
|
|
453
|
+
pageup: 0x21, pagedown: 0x22, up: 0x26, down: 0x28, left: 0x25, right: 0x27,
|
|
454
|
+
shift: 0x10, ctrl: 0x11, control: 0x11, alt: 0x12, win: 0x5b, meta: 0x5b,
|
|
455
|
+
capslock: 0x14, numlock: 0x90, scrolllock: 0x91, printscreen: 0x2c, pause: 0x13,
|
|
456
|
+
f1: 0x70, f2: 0x71, f3: 0x72, f4: 0x73, f5: 0x74, f6: 0x75,
|
|
457
|
+
f7: 0x76, f8: 0x77, f9: 0x78, f10: 0x79, f11: 0x7a, f12: 0x7b,
|
|
458
|
+
};
|
|
459
|
+
for (let i = 0; i < 26; i += 1) VIRTUAL_KEYS[String.fromCharCode(97 + i)] = 0x41 + i;
|
|
460
|
+
for (let i = 0; i <= 9; i += 1) VIRTUAL_KEYS[String(i)] = 0x30 + i;
|
|
461
|
+
|
|
462
|
+
function resolveKey(name) {
|
|
463
|
+
const key = String(name || '').trim().toLowerCase();
|
|
464
|
+
if (!key) throw new Error('empty key name');
|
|
465
|
+
const code = VIRTUAL_KEYS[key];
|
|
466
|
+
if (code !== undefined) return code;
|
|
467
|
+
throw new Error(`unknown key: ${name}`);
|
|
468
|
+
}
|
|
469
|
+
|
|
470
|
+
/** Tap one key, optionally with modifiers held. */
|
|
471
|
+
function tapKey(A, key, modifiers = []) {
|
|
472
|
+
const mods = modifiers.map(resolveKey);
|
|
473
|
+
for (const code of mods) sendInput(A, keyInput(code, 0));
|
|
474
|
+
const code = resolveKey(key);
|
|
475
|
+
sendInput(A, keyInput(code, 0));
|
|
476
|
+
sendInput(A, keyInput(code, KEYEVENTF_KEYUP));
|
|
477
|
+
for (const mod of mods.reverse()) sendInput(A, keyInput(mod, KEYEVENTF_KEYUP));
|
|
478
|
+
}
|
|
479
|
+
|
|
480
|
+
/** Type literal text through unicode keyboard events at a human cadence. */
|
|
481
|
+
async function typeText(A, text, humanize) {
|
|
482
|
+
for (const char of String(text)) {
|
|
483
|
+
const code = char.codePointAt(0);
|
|
484
|
+
if (code === undefined) continue;
|
|
485
|
+
if (code === 10 || code === 13) {
|
|
486
|
+
sendInput(A, keyInput(0x0d, 0));
|
|
487
|
+
sendInput(A, keyInput(0x0d, KEYEVENTF_KEYUP));
|
|
488
|
+
if (humanize) await sleep(between(CFG.keyDelayMs));
|
|
489
|
+
continue;
|
|
490
|
+
}
|
|
491
|
+
if (code < 0x20) continue;
|
|
492
|
+
for (const unit of unicodeUnits(code)) {
|
|
493
|
+
sendInput(A, keyInput(0, KEYEVENTF_UNICODE, unit));
|
|
494
|
+
sendInput(A, keyInput(0, KEYEVENTF_UNICODE | KEYEVENTF_KEYUP, unit));
|
|
495
|
+
}
|
|
496
|
+
if (humanize) await sleep(between(CFG.keyDelayMs));
|
|
497
|
+
}
|
|
498
|
+
}
|
|
499
|
+
|
|
500
|
+
/** Move the pointer to a screen pixel. */
|
|
501
|
+
function moveMouse(A, x, y) {
|
|
502
|
+
const { ax, ay } = toAbsolute(A, Math.round(x), Math.round(y));
|
|
503
|
+
sendInput(A, mouseInput({ dx: ax, dy: ay, flags: MOUSEEVENTF_MOVE | MOUSEEVENTF_ABSOLUTE }));
|
|
504
|
+
}
|
|
505
|
+
|
|
506
|
+
/**
|
|
507
|
+
* Move the pointer like a hand does: a smoothstep path in a few steps instead of
|
|
508
|
+
* one teleport, so hover states, tooltips and drag targets behave realistically.
|
|
509
|
+
* @returns the clamped destination actually used.
|
|
510
|
+
*/
|
|
511
|
+
async function moveMouseHuman(A, x, y, humanize) {
|
|
512
|
+
const target = clampPoint(A, x, y);
|
|
513
|
+
if (!humanize) {
|
|
514
|
+
moveMouse(A, target.x, target.y);
|
|
515
|
+
return target;
|
|
516
|
+
}
|
|
517
|
+
const from = cursorPosition(A);
|
|
518
|
+
const distance = Math.hypot(target.x - from.x, target.y - from.y);
|
|
519
|
+
const steps = Math.max(4, Math.min(CFG.moveSteps, Math.round(distance / 55)));
|
|
520
|
+
for (let i = 1; i <= steps; i += 1) {
|
|
521
|
+
const t = i / steps;
|
|
522
|
+
const eased = t * t * (3 - 2 * t);
|
|
523
|
+
moveMouse(A, from.x + (target.x - from.x) * eased, from.y + (target.y - from.y) * eased);
|
|
524
|
+
await sleep(6 + Math.random() * 9);
|
|
525
|
+
}
|
|
526
|
+
moveMouse(A, target.x, target.y);
|
|
527
|
+
return target;
|
|
528
|
+
}
|
|
529
|
+
|
|
530
|
+
/** Press and release a mouse button. */
|
|
531
|
+
function clickMouse(A, button = 'left', double = false) {
|
|
532
|
+
const table = {
|
|
533
|
+
left: [MOUSEEVENTF_LEFTDOWN, MOUSEEVENTF_LEFTUP],
|
|
534
|
+
right: [MOUSEEVENTF_RIGHTDOWN, MOUSEEVENTF_RIGHTUP],
|
|
535
|
+
middle: [MOUSEEVENTF_MIDDLEDOWN, MOUSEEVENTF_MIDDLEUP],
|
|
536
|
+
};
|
|
537
|
+
const pair = table[String(button).toLowerCase()];
|
|
538
|
+
if (!pair) throw new Error(`unknown mouse button: ${button}`);
|
|
539
|
+
for (let i = 0; i < (double ? 2 : 1); i += 1) {
|
|
540
|
+
sendInput(A, mouseInput({ flags: pair[0] }));
|
|
541
|
+
sendInput(A, mouseInput({ flags: pair[1] }));
|
|
542
|
+
}
|
|
543
|
+
}
|
|
544
|
+
|
|
545
|
+
/** Scroll the wheel; positive is up (or right on the horizontal axis). */
|
|
546
|
+
function scrollWheel(A, amount, axis = 'vertical') {
|
|
547
|
+
const delta = Math.round(requireNumber(amount, 'amount'));
|
|
548
|
+
if (!delta) return;
|
|
549
|
+
const flag = String(axis).toLowerCase() === 'horizontal' ? MOUSEEVENTF_HWHEEL : MOUSEEVENTF_WHEEL;
|
|
550
|
+
sendInput(A, mouseInput({ data: delta, flags: flag }));
|
|
551
|
+
}
|
|
552
|
+
|
|
553
|
+
/** Encode one frame to PNG on disk, then prune old captures. */
|
|
554
|
+
function savePng(frame, directory, prefix = 'screen') {
|
|
555
|
+
const target = directory || SCREEN_DIR;
|
|
556
|
+
mkdirSync(target, { recursive: true });
|
|
557
|
+
const png = encodePng(frame);
|
|
558
|
+
const path = join(target, `${prefix}-${Date.now()}.png`);
|
|
559
|
+
writeFileSync(path, png);
|
|
560
|
+
pruneScreens(target);
|
|
561
|
+
return { path, width: frame.width, height: frame.height, bytes: png.length };
|
|
562
|
+
}
|
|
563
|
+
|
|
564
|
+
/** Screen capture to a PNG on disk, keeping only the newest screenshots. */
|
|
565
|
+
function screenshot(A, directory) {
|
|
566
|
+
return savePng(captureScreen(A), directory);
|
|
567
|
+
}
|
|
568
|
+
|
|
569
|
+
/** Retention is best effort: a locked or unreadable file must not fail the capture. */
|
|
570
|
+
function pruneScreens(directory) {
|
|
571
|
+
const keep = Number(CFG.keepScreenshots);
|
|
572
|
+
if (!Number.isFinite(keep) || keep <= 0) return;
|
|
573
|
+
try {
|
|
574
|
+
const names = readdirSync(directory)
|
|
575
|
+
.filter((name) => name.startsWith('screen-') && name.endsWith('.png'))
|
|
576
|
+
.sort();
|
|
577
|
+
for (const name of names.slice(0, Math.max(0, names.length - keep))) {
|
|
578
|
+
try {
|
|
579
|
+
unlinkSync(join(directory, name));
|
|
580
|
+
} catch {
|
|
581
|
+
/* keep going */
|
|
582
|
+
}
|
|
583
|
+
}
|
|
584
|
+
} catch {
|
|
585
|
+
/* retention is best effort */
|
|
586
|
+
}
|
|
587
|
+
}
|
|
588
|
+
|
|
589
|
+
/** Read the pointer position. */
|
|
590
|
+
function cursorPosition(A) {
|
|
591
|
+
const point = { x: 0, y: 0 };
|
|
592
|
+
A.GetCursorPos(point);
|
|
593
|
+
return { x: point.x, y: point.y };
|
|
594
|
+
}
|
|
595
|
+
|
|
596
|
+
/** Executable file name behind a process id, or '' when it cannot be read. */
|
|
597
|
+
function exeName(A, pid) {
|
|
598
|
+
if (!pid) return '';
|
|
599
|
+
const handle = A.OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, false, pid);
|
|
600
|
+
if (!handle) return '';
|
|
601
|
+
try {
|
|
602
|
+
const size = [512];
|
|
603
|
+
const buffer = Buffer.alloc(512 * 2);
|
|
604
|
+
if (!A.QueryFullProcessImageNameW(handle, 0, buffer, size)) return '';
|
|
605
|
+
const full = buffer.toString('utf16le', 0, Math.max(0, size[0]) * 2);
|
|
606
|
+
return full.split(/[\\/]/).pop() || full;
|
|
607
|
+
} finally {
|
|
608
|
+
A.CloseHandle(handle);
|
|
609
|
+
}
|
|
610
|
+
}
|
|
611
|
+
|
|
612
|
+
/**
|
|
613
|
+
* Press, travel and release like a hand: the button stays down for the whole
|
|
614
|
+
* path so drag targets that watch hover state still see it.
|
|
615
|
+
*/
|
|
616
|
+
async function dragMouse(A, fromX, fromY, toX, toY, humanize) {
|
|
617
|
+
const start = clampPoint(A, fromX, fromY);
|
|
618
|
+
const end = clampPoint(A, toX, toY);
|
|
619
|
+
await moveMouseHuman(A, start.x, start.y, humanize);
|
|
620
|
+
if (humanize) await sleep(between([40, 90]));
|
|
621
|
+
sendInput(A, mouseInput({ flags: MOUSEEVENTF_LEFTDOWN }));
|
|
622
|
+
try {
|
|
623
|
+
if (humanize) {
|
|
624
|
+
const steps = Math.max(6, Math.min(CFG.moveSteps * 2, Math.round(Math.hypot(end.x - start.x, end.y - start.y) / 35)));
|
|
625
|
+
for (let i = 1; i <= steps; i += 1) {
|
|
626
|
+
const t = i / steps;
|
|
627
|
+
const eased = t * t * (3 - 2 * t);
|
|
628
|
+
moveMouse(A, start.x + (end.x - start.x) * eased, start.y + (end.y - start.y) * eased);
|
|
629
|
+
await sleep(8 + Math.random() * 10);
|
|
630
|
+
}
|
|
631
|
+
}
|
|
632
|
+
moveMouse(A, end.x, end.y);
|
|
633
|
+
if (humanize) await sleep(between([50, 110]));
|
|
634
|
+
} finally {
|
|
635
|
+
sendInput(A, mouseInput({ flags: MOUSEEVENTF_LEFTUP }));
|
|
636
|
+
}
|
|
637
|
+
return { from: start, to: end };
|
|
638
|
+
}
|
|
639
|
+
|
|
640
|
+
/** Sleep, and optionally poll until a window whose title contains `title` exists. */
|
|
641
|
+
async function waitFor(A, { ms, title, timeoutMs }) {
|
|
642
|
+
const budget = Math.max(0, Math.min(timeoutMs === undefined ? CFG.maxWaitMs : requireNumber(timeoutMs, 'timeoutMs'), CFG.maxWaitMs));
|
|
643
|
+
const started = Date.now();
|
|
644
|
+
const sleepBudget = Math.min(ms === undefined ? 0 : Math.max(0, requireNumber(ms, 'ms')), budget);
|
|
645
|
+
if (sleepBudget > 0) await sleep(sleepBudget);
|
|
646
|
+
if (!title) return { waitedMs: Date.now() - started, matched: false, timedOut: false };
|
|
647
|
+
const needle = String(title).toLowerCase();
|
|
648
|
+
while (Date.now() - started < budget) {
|
|
649
|
+
if (listWindows(A).some((w) => w.title.toLowerCase().includes(needle))) {
|
|
650
|
+
return { waitedMs: Date.now() - started, matched: true, timedOut: false };
|
|
651
|
+
}
|
|
652
|
+
await sleep(150);
|
|
653
|
+
}
|
|
654
|
+
return { waitedMs: Date.now() - started, matched: false, timedOut: true };
|
|
655
|
+
}
|
|
656
|
+
|
|
657
|
+
/** Resolve an executable name or path without ever going through a shell. */
|
|
658
|
+
function resolveExe(name) {
|
|
659
|
+
const raw = String(name || '').trim();
|
|
660
|
+
if (!raw) throw new Error('empty executable name');
|
|
661
|
+
if (/[\\/]/.test(raw)) {
|
|
662
|
+
if (!existsSync(raw)) throw new Error(`executable not found: ${raw}`);
|
|
663
|
+
return raw;
|
|
664
|
+
}
|
|
665
|
+
const exe = raw.toLowerCase().endsWith('.exe') ? raw : `${raw}.exe`;
|
|
666
|
+
const dirs = (process.env.PATH || '').split(';').filter(Boolean);
|
|
667
|
+
for (const dir of dirs) {
|
|
668
|
+
const candidate = join(dir, exe);
|
|
669
|
+
if (existsSync(candidate)) return candidate;
|
|
670
|
+
}
|
|
671
|
+
throw new Error(`executable not found on PATH: ${raw} (pass an absolute path instead)`);
|
|
672
|
+
}
|
|
673
|
+
|
|
674
|
+
/**
|
|
675
|
+
* JSON-Schema helpers for raw `tools.register()` definitions. The host registry
|
|
676
|
+
* requires every tool to carry `output { schema, render }` and a lossless-JSON
|
|
677
|
+
* object-root `parameters` schema; annotations (`description`, `title`) are the
|
|
678
|
+
* only extra keywords allowed on a node.
|
|
679
|
+
*/
|
|
680
|
+
const obj = (properties = {}, required = []) => ({
|
|
681
|
+
type: 'object',
|
|
682
|
+
properties,
|
|
683
|
+
...(required.length ? { required } : {}),
|
|
684
|
+
});
|
|
685
|
+
const str = (description) => ({ type: 'string', description });
|
|
686
|
+
const num = (description) => ({ type: 'number', description });
|
|
687
|
+
const bool = (description) => ({ type: 'boolean', description });
|
|
688
|
+
const text = (value) => [{ type: 'text', text: String(value) }];
|
|
689
|
+
|
|
690
|
+
/** Reusable output-schema fragments so every object node declares its fields. */
|
|
691
|
+
const point = (description) => ({
|
|
692
|
+
type: 'object',
|
|
693
|
+
description,
|
|
694
|
+
properties: { x: num('X in virtual-screen pixels.'), y: num('Y in virtual-screen pixels.') },
|
|
695
|
+
});
|
|
696
|
+
const rect = (description) => ({
|
|
697
|
+
type: 'object',
|
|
698
|
+
description,
|
|
699
|
+
properties: {
|
|
700
|
+
x: num('Left in virtual-screen pixels.'),
|
|
701
|
+
y: num('Top in virtual-screen pixels.'),
|
|
702
|
+
width: num('Width in pixels.'),
|
|
703
|
+
height: num('Height in pixels.'),
|
|
704
|
+
},
|
|
705
|
+
});
|
|
706
|
+
const windowItem = (description) => ({
|
|
707
|
+
type: 'object',
|
|
708
|
+
description,
|
|
709
|
+
properties: {
|
|
710
|
+
handle: { type: 'integer', description: 'Window handle; pass it to cua_focus_window.' },
|
|
711
|
+
title: str('Window title.'),
|
|
712
|
+
pid: { type: 'integer', description: 'Owner process id.' },
|
|
713
|
+
active: bool('Is the foreground window.'),
|
|
714
|
+
minimized: bool('Is minimized.'),
|
|
715
|
+
bounds: rect('Window rect on the virtual screen.'),
|
|
716
|
+
},
|
|
717
|
+
});
|
|
718
|
+
|
|
719
|
+
const TOOL_DECLS = [
|
|
720
|
+
{
|
|
721
|
+
name: 'cua_screenshot',
|
|
722
|
+
description:
|
|
723
|
+
'Capture the whole virtual screen to a PNG file and return its path and pixel size. Look before clicking so you act on what is really on screen.',
|
|
724
|
+
parameters: obj({
|
|
725
|
+
directory: str('Optional absolute directory for the PNG; defaults to a temp folder.'),
|
|
726
|
+
}),
|
|
727
|
+
run: (A, args) => screenshot(A, args.directory),
|
|
728
|
+
out: {
|
|
729
|
+
schema: obj({
|
|
730
|
+
path: str('Absolute path of the written PNG.'),
|
|
731
|
+
width: { type: 'integer', description: 'Capture width in pixels.' },
|
|
732
|
+
height: { type: 'integer', description: 'Capture height in pixels.' },
|
|
733
|
+
bytes: { type: 'integer', description: 'PNG size in bytes.' },
|
|
734
|
+
}),
|
|
735
|
+
render: (_args, v) => text(`Screenshot saved: ${v.path} (${v.width}x${v.height}, ${v.bytes} bytes). Read this file to view it.`),
|
|
736
|
+
},
|
|
737
|
+
readOnly: true,
|
|
738
|
+
},
|
|
739
|
+
{
|
|
740
|
+
name: 'cua_mouse_move',
|
|
741
|
+
description: 'Move the real mouse pointer to a screen pixel (virtual-screen coordinates).',
|
|
742
|
+
parameters: obj({
|
|
743
|
+
x: num('Screen X in pixels.'),
|
|
744
|
+
y: num('Screen Y in pixels.'),
|
|
745
|
+
}, ['x', 'y']),
|
|
746
|
+
run: async (A, args) => {
|
|
747
|
+
const point = await moveMouseHuman(A, args.x, args.y, CFG.humanize);
|
|
748
|
+
return { ok: true, ...point };
|
|
749
|
+
},
|
|
750
|
+
out: {
|
|
751
|
+
schema: obj({ ok: bool('Always true.'), x: num('Pointer X.'), y: num('Pointer Y.') }),
|
|
752
|
+
render: (_args, v) => text(`Pointer moved to (${v.x}, ${v.y}).`),
|
|
753
|
+
},
|
|
754
|
+
},
|
|
755
|
+
{
|
|
756
|
+
name: 'cua_mouse_click',
|
|
757
|
+
description:
|
|
758
|
+
'Click a real mouse button, optionally moving to x/y first. Double clicks repeat the press-release pair.',
|
|
759
|
+
parameters: obj({
|
|
760
|
+
x: num('Optional screen X; omit to click where the pointer is.'),
|
|
761
|
+
y: num('Optional screen Y; omit to click where the pointer is.'),
|
|
762
|
+
button: { type: 'string', enum: ['left', 'right', 'middle'], description: 'Mouse button; defaults to left.' },
|
|
763
|
+
double: bool('Double click when true.'),
|
|
764
|
+
}),
|
|
765
|
+
run: async (A, args) => {
|
|
766
|
+
if (typeof args.x === 'number' && typeof args.y === 'number') await moveMouseHuman(A, args.x, args.y, CFG.humanize);
|
|
767
|
+
if (CFG.humanize) await sleep(between(CFG.clickSettleMs));
|
|
768
|
+
const double = Boolean(args.double);
|
|
769
|
+
clickMouse(A, args.button, false);
|
|
770
|
+
if (double) {
|
|
771
|
+
// A real double click needs a visible gap; two immediate pairs collapse into one.
|
|
772
|
+
await sleep(between([50, 90]));
|
|
773
|
+
clickMouse(A, args.button, false);
|
|
774
|
+
}
|
|
775
|
+
return { ok: true, ...cursorPosition(A), button: args.button || 'left', double };
|
|
776
|
+
},
|
|
777
|
+
out: {
|
|
778
|
+
schema: obj({
|
|
779
|
+
ok: bool('Always true.'),
|
|
780
|
+
button: str('Button used.'),
|
|
781
|
+
x: num('Pointer X.'),
|
|
782
|
+
y: num('Pointer Y.'),
|
|
783
|
+
double: bool('Whether the click was doubled.'),
|
|
784
|
+
}),
|
|
785
|
+
render: (_args, v) => text(`${v.double ? 'Double-clicked' : 'Clicked'} ${v.button} button at (${v.x}, ${v.y}).`),
|
|
786
|
+
},
|
|
787
|
+
},
|
|
788
|
+
{
|
|
789
|
+
name: 'cua_scroll',
|
|
790
|
+
description: 'Turn the real mouse wheel at the pointer position; positive scrolls up (or right on the horizontal axis).',
|
|
791
|
+
parameters: obj({
|
|
792
|
+
amount: num('Wheel delta (120 per notch).'),
|
|
793
|
+
axis: { type: 'string', enum: ['vertical', 'horizontal'], description: 'Scroll axis; defaults to vertical.' },
|
|
794
|
+
}, ['amount']),
|
|
795
|
+
run: (A, args) => {
|
|
796
|
+
scrollWheel(A, args.amount, args.axis);
|
|
797
|
+
return { ok: true };
|
|
798
|
+
},
|
|
799
|
+
out: {
|
|
800
|
+
schema: obj({ ok: bool('Always true.') }),
|
|
801
|
+
render: (_args, v) => text(`Wheel scrolled (${v.ok ? 'sent' : 'failed'}).`),
|
|
802
|
+
},
|
|
803
|
+
},
|
|
804
|
+
{
|
|
805
|
+
name: 'cua_key',
|
|
806
|
+
description:
|
|
807
|
+
'Press a real key with optional modifiers. Key names: enter, esc, tab, space, backspace, delete, arrows, f1-f12, letters, digits. Modifiers: ctrl, shift, alt, win.',
|
|
808
|
+
parameters: obj({
|
|
809
|
+
key: str('Key to press, e.g. "enter", "a", "f5".'),
|
|
810
|
+
modifiers: { type: 'array', items: { type: 'string' }, description: 'Modifiers to hold, e.g. ["ctrl"].' },
|
|
811
|
+
}, ['key']),
|
|
812
|
+
run: (A, args) => {
|
|
813
|
+
tapKey(A, args.key, args.modifiers || []);
|
|
814
|
+
return { ok: true };
|
|
815
|
+
},
|
|
816
|
+
out: {
|
|
817
|
+
schema: obj({ ok: bool('Always true.') }),
|
|
818
|
+
render: (_args, v) => text(`Key press sent (${v.ok ? 'ok' : 'failed'}).`),
|
|
819
|
+
},
|
|
820
|
+
},
|
|
821
|
+
{
|
|
822
|
+
name: 'cua_type',
|
|
823
|
+
description: 'Type literal text into the focused window through real unicode key events.',
|
|
824
|
+
parameters: obj({
|
|
825
|
+
text: str('Text to type.'),
|
|
826
|
+
}, ['text']),
|
|
827
|
+
run: async (A, args) => {
|
|
828
|
+
await typeText(A, args.text, CFG.humanize);
|
|
829
|
+
return { ok: true, chars: [...String(args.text)].length };
|
|
830
|
+
},
|
|
831
|
+
out: {
|
|
832
|
+
schema: obj({ ok: bool('Always true.'), chars: { type: 'integer', description: 'Characters typed.' } }),
|
|
833
|
+
render: (_args, v) => text(`Typed ${v.chars} characters.`),
|
|
834
|
+
},
|
|
835
|
+
},
|
|
836
|
+
{
|
|
837
|
+
name: 'cua_windows',
|
|
838
|
+
description: 'List visible top-level windows: handle, title, process id, bounds, active and minimized flags.',
|
|
839
|
+
parameters: obj({}),
|
|
840
|
+
run: (A) => ({ windows: listWindows(A) }),
|
|
841
|
+
out: {
|
|
842
|
+
schema: obj({
|
|
843
|
+
windows: { type: 'array', items: windowItem('One visible window.'), description: 'Visible top-level windows.' },
|
|
844
|
+
}),
|
|
845
|
+
render: (_args, v) => text(
|
|
846
|
+
`Found ${v.windows.length} window(s):\n` + v.windows.map((w) =>
|
|
847
|
+
`#${w.handle}${w.active ? ' [active]' : ''}${w.minimized ? ' [minimized]' : ''} "${w.title}" pid=${w.pid} ${w.bounds.width}x${w.bounds.height} @ (${w.bounds.x},${w.bounds.y})`
|
|
848
|
+
).join('\n')),
|
|
849
|
+
},
|
|
850
|
+
readOnly: true,
|
|
851
|
+
},
|
|
852
|
+
{
|
|
853
|
+
name: 'cua_focus_window',
|
|
854
|
+
description: 'Restore a minimized window and bring it to the foreground by its handle from cua_windows.',
|
|
855
|
+
parameters: obj({
|
|
856
|
+
handle: num('Window handle from cua_windows.'),
|
|
857
|
+
}, ['handle']),
|
|
858
|
+
run: (A, args) => {
|
|
859
|
+
const hwnd = A.koffi.address(requireNumber(args.handle, 'handle'));
|
|
860
|
+
A.ShowWindow(hwnd, SW_RESTORE);
|
|
861
|
+
A.SetForegroundWindow(hwnd);
|
|
862
|
+
A.BringWindowToTop(hwnd);
|
|
863
|
+
return { ok: true, handle: args.handle };
|
|
864
|
+
},
|
|
865
|
+
out: {
|
|
866
|
+
schema: obj({ ok: bool('Always true.'), handle: num('Focused window handle.') }),
|
|
867
|
+
render: (_args, v) => text(`Window ${v.handle} brought to the foreground.`),
|
|
868
|
+
},
|
|
869
|
+
},
|
|
870
|
+
{
|
|
871
|
+
name: 'cua_cursor',
|
|
872
|
+
description: 'Report the current pointer position and the virtual-screen geometry.',
|
|
873
|
+
parameters: obj({}),
|
|
874
|
+
run: (A) => ({ ...cursorPosition(A), screen: virtualScreen(A) }),
|
|
875
|
+
out: {
|
|
876
|
+
schema: obj({
|
|
877
|
+
x: num('Pointer X.'),
|
|
878
|
+
y: num('Pointer Y.'),
|
|
879
|
+
screen: rect('Virtual-screen geometry.'),
|
|
880
|
+
}),
|
|
881
|
+
render: (_args, v) => text(`Cursor at (${v.x}, ${v.y}); virtual screen: ${JSON.stringify(v.screen)}.`),
|
|
882
|
+
},
|
|
883
|
+
readOnly: true,
|
|
884
|
+
},
|
|
885
|
+
{
|
|
886
|
+
name: 'cua_zoom',
|
|
887
|
+
description:
|
|
888
|
+
'Capture a screen region and upscale it so small text, icons and menu items become readable. Coordinates are virtual-screen pixels.',
|
|
889
|
+
parameters: obj({
|
|
890
|
+
x: num('Region left in virtual-screen pixels.'),
|
|
891
|
+
y: num('Region top in virtual-screen pixels.'),
|
|
892
|
+
width: num('Region width in pixels.'),
|
|
893
|
+
height: num('Region height in pixels.'),
|
|
894
|
+
factor: num('Integer upscale factor (default 2, max 4).'),
|
|
895
|
+
directory: str('Optional output directory for the PNG.'),
|
|
896
|
+
}, ['x', 'y', 'width', 'height']),
|
|
897
|
+
run: (A, args) => {
|
|
898
|
+
const screen = virtualScreen(A);
|
|
899
|
+
const x = Math.max(screen.x, Math.round(args.x));
|
|
900
|
+
const y = Math.max(screen.y, Math.round(args.y));
|
|
901
|
+
const right = Math.min(screen.x + screen.width, Math.round(args.x + args.width));
|
|
902
|
+
const bottom = Math.min(screen.y + screen.height, Math.round(args.y + args.height));
|
|
903
|
+
const w = right - x;
|
|
904
|
+
const h = bottom - y;
|
|
905
|
+
if (w <= 0 || h <= 0) throw new Error('zoom region falls outside the virtual screen');
|
|
906
|
+
let factor = Math.max(1, Math.min(4, Math.floor(Number(args.factor) || CFG.zoomFactor)));
|
|
907
|
+
while (factor > 1 && w * factor * h * factor > CFG.zoomMaxPixels) factor -= 1;
|
|
908
|
+
const scaled = scaleBgra(captureRect(A, x, y, w, h), factor);
|
|
909
|
+
const saved = savePng(scaled, args.directory, 'zoom');
|
|
910
|
+
return { ...saved, factor, source: { x, y, width: w, height: h } };
|
|
911
|
+
},
|
|
912
|
+
out: {
|
|
913
|
+
schema: obj({
|
|
914
|
+
path: str('Absolute path of the written PNG.'),
|
|
915
|
+
width: { type: 'integer', description: 'Saved width in pixels.' },
|
|
916
|
+
height: { type: 'integer', description: 'Saved height in pixels.' },
|
|
917
|
+
bytes: { type: 'integer', description: 'PNG size in bytes.' },
|
|
918
|
+
factor: { type: 'integer', description: 'Upscale factor actually used.' },
|
|
919
|
+
source: rect('Captured region in virtual-screen pixels.'),
|
|
920
|
+
}),
|
|
921
|
+
render: (_args, v) => text(
|
|
922
|
+
`Zoom saved: ${v.path} — ${v.source.width}x${v.source.height} at (${v.source.x},${v.source.y}) scaled x${v.factor} to ${v.width}x${v.height}.`),
|
|
923
|
+
},
|
|
924
|
+
readOnly: true,
|
|
925
|
+
},
|
|
926
|
+
{
|
|
927
|
+
name: 'cua_screenshot_window',
|
|
928
|
+
description:
|
|
929
|
+
'Capture one window by handle. Uses PrintWindow so occluded windows still render correctly, with a screen blit fallback.',
|
|
930
|
+
parameters: obj({
|
|
931
|
+
handle: num('Window handle from cua_windows.'),
|
|
932
|
+
directory: str('Optional output directory for the PNG.'),
|
|
933
|
+
}, ['handle']),
|
|
934
|
+
run: (A, args) => {
|
|
935
|
+
const hwnd = A.koffi.address(requireNumber(args.handle, 'handle'));
|
|
936
|
+
const frame = captureWindow(A, hwnd);
|
|
937
|
+
const saved = savePng(frame, args.directory, 'window');
|
|
938
|
+
return { ...saved, bounds: frame.bounds, printed: frame.printed };
|
|
939
|
+
},
|
|
940
|
+
out: {
|
|
941
|
+
schema: obj({
|
|
942
|
+
path: str('Absolute path of the written PNG.'),
|
|
943
|
+
width: { type: 'integer', description: 'Image width in pixels.' },
|
|
944
|
+
height: { type: 'integer', description: 'Image height in pixels.' },
|
|
945
|
+
bytes: { type: 'integer', description: 'PNG size in bytes.' },
|
|
946
|
+
printed: bool('True when PrintWindow rendered it; false means screen fallback.'),
|
|
947
|
+
bounds: rect('Window rect on the virtual screen.'),
|
|
948
|
+
}),
|
|
949
|
+
render: (_args, v) => text(
|
|
950
|
+
`Window captured (${v.printed ? 'PrintWindow' : 'screen fallback'}): ${v.path} — ${v.width}x${v.height} at (${v.bounds.x},${v.bounds.y}).`),
|
|
951
|
+
},
|
|
952
|
+
readOnly: true,
|
|
953
|
+
},
|
|
954
|
+
{
|
|
955
|
+
name: 'cua_drag',
|
|
956
|
+
description:
|
|
957
|
+
'Drag with the real left button held down: press at fromX/fromY, travel, release at toX/toY. Use it for sliders, selections, window moves and drag-and-drop.',
|
|
958
|
+
parameters: obj({
|
|
959
|
+
fromX: num('Press X in virtual-screen pixels.'),
|
|
960
|
+
fromY: num('Press Y in virtual-screen pixels.'),
|
|
961
|
+
toX: num('Release X in virtual-screen pixels.'),
|
|
962
|
+
toY: num('Release Y in virtual-screen pixels.'),
|
|
963
|
+
}, ['fromX', 'fromY', 'toX', 'toY']),
|
|
964
|
+
run: async (A, args) => {
|
|
965
|
+
const moved = await dragMouse(A, args.fromX, args.fromY, args.toX, args.toY, CFG.humanize);
|
|
966
|
+
return { ok: true, ...moved };
|
|
967
|
+
},
|
|
968
|
+
out: {
|
|
969
|
+
schema: obj({
|
|
970
|
+
ok: bool('Always true.'),
|
|
971
|
+
from: point('Actual press point after clamping.'),
|
|
972
|
+
to: point('Actual release point after clamping.'),
|
|
973
|
+
}),
|
|
974
|
+
render: (_args, v) => text(`Dragged (${v.from.x}, ${v.from.y}) -> (${v.to.x}, ${v.to.y}).`),
|
|
975
|
+
},
|
|
976
|
+
},
|
|
977
|
+
{
|
|
978
|
+
name: 'cua_wait',
|
|
979
|
+
description:
|
|
980
|
+
'Wait for the desktop to settle: sleep for a number of milliseconds and/or poll until a window whose title contains the given text exists.',
|
|
981
|
+
parameters: obj({
|
|
982
|
+
ms: num('Milliseconds to sleep first (default 0).'),
|
|
983
|
+
title: str('Optional window-title substring to wait for, case-insensitive.'),
|
|
984
|
+
timeoutMs: num('Upper bound for the wait; capped by the plugin config.'),
|
|
985
|
+
}),
|
|
986
|
+
run: async (A, args) => waitFor(A, args),
|
|
987
|
+
out: {
|
|
988
|
+
schema: obj({
|
|
989
|
+
waitedMs: { type: 'integer', description: 'Actual time spent waiting.' },
|
|
990
|
+
matched: bool('True when the title appeared.'),
|
|
991
|
+
timedOut: bool('True when the title never appeared in time.'),
|
|
992
|
+
}),
|
|
993
|
+
render: (_args, v) => text(
|
|
994
|
+
v.matched ? `Window appeared after ${v.waitedMs} ms.`
|
|
995
|
+
: v.timedOut ? `Timed out after ${v.waitedMs} ms waiting for the window.`
|
|
996
|
+
: `Waited ${v.waitedMs} ms.`),
|
|
997
|
+
},
|
|
998
|
+
readOnly: true,
|
|
999
|
+
timeoutMs: 15_000,
|
|
1000
|
+
},
|
|
1001
|
+
{
|
|
1002
|
+
name: 'cua_app_list',
|
|
1003
|
+
description: 'List visible top-level windows with their process executable name, so you can tell apps apart and pick a launch target.',
|
|
1004
|
+
parameters: obj({}),
|
|
1005
|
+
run: (A) => ({ apps: listWindows(A).map((w) => ({ ...w, process: exeName(A, w.pid) })) }),
|
|
1006
|
+
out: {
|
|
1007
|
+
schema: obj({
|
|
1008
|
+
apps: {
|
|
1009
|
+
type: 'array',
|
|
1010
|
+
items: {
|
|
1011
|
+
type: 'object',
|
|
1012
|
+
description: 'One window with its process executable.',
|
|
1013
|
+
properties: {
|
|
1014
|
+
handle: { type: 'integer', description: 'Window handle.' },
|
|
1015
|
+
title: str('Window title.'),
|
|
1016
|
+
pid: { type: 'integer', description: 'Owner process id.' },
|
|
1017
|
+
process: str('Process executable name, e.g. notepad.exe.'),
|
|
1018
|
+
active: bool('Is the foreground window.'),
|
|
1019
|
+
minimized: bool('Is minimized.'),
|
|
1020
|
+
bounds: rect('Window rect on the virtual screen.'),
|
|
1021
|
+
},
|
|
1022
|
+
},
|
|
1023
|
+
description: 'Windows with process names.',
|
|
1024
|
+
},
|
|
1025
|
+
}),
|
|
1026
|
+
render: (_args, v) => text(
|
|
1027
|
+
`${v.apps.length} app window(s):\n` + v.apps.map((a) =>
|
|
1028
|
+
`#${a.handle} ${a.process || '?'} (pid ${a.pid})${a.active ? ' [active]' : ''}${a.minimized ? ' [minimized]' : ''} "${a.title}"`).join('\n')),
|
|
1029
|
+
},
|
|
1030
|
+
readOnly: true,
|
|
1031
|
+
},
|
|
1032
|
+
{
|
|
1033
|
+
name: 'cua_app_launch',
|
|
1034
|
+
description:
|
|
1035
|
+
'Launch an application by executable name (resolved on PATH) or absolute path. Spawns directly without a shell, so no quoting games.',
|
|
1036
|
+
parameters: obj({
|
|
1037
|
+
exe: str('Executable name such as "notepad" / "calc", or an absolute .exe path.'),
|
|
1038
|
+
args: { type: 'array', items: { type: 'string' }, description: 'Command-line arguments.' },
|
|
1039
|
+
}, ['exe']),
|
|
1040
|
+
run: (A, args) => {
|
|
1041
|
+
if (CFG.allowLaunch !== true) throw new Error('cua_app_launch is disabled by config (allowLaunch: false)');
|
|
1042
|
+
const exe = resolveExe(args.exe);
|
|
1043
|
+
const argv = Array.isArray(args.args) ? args.args.map(String) : [];
|
|
1044
|
+
const child = spawn(exe, argv, { detached: true, stdio: 'ignore', windowsHide: false });
|
|
1045
|
+
child.unref();
|
|
1046
|
+
return { ok: true, exe, pid: child.pid ?? 0 };
|
|
1047
|
+
},
|
|
1048
|
+
out: {
|
|
1049
|
+
schema: obj({
|
|
1050
|
+
ok: bool('Always true.'),
|
|
1051
|
+
exe: str('Resolved executable path.'),
|
|
1052
|
+
pid: { type: 'integer', description: 'Process id of the launched app.' },
|
|
1053
|
+
}),
|
|
1054
|
+
render: (_args, v) => text(`Launched ${v.exe} (pid ${v.pid}).`),
|
|
1055
|
+
},
|
|
1056
|
+
},
|
|
1057
|
+
];
|
|
1058
|
+
|
|
1059
|
+
export const name = 'dsh-win-cua';
|
|
1060
|
+
|
|
1061
|
+
export const inject = ['tools'];
|
|
1062
|
+
|
|
1063
|
+
/**
|
|
1064
|
+
* Register the CUA tools on the host.
|
|
1065
|
+
*
|
|
1066
|
+
* Config knobs (all optional, from this bundle's cordis.patch.yml):
|
|
1067
|
+
* - `humanize` (default true): move the pointer along a smoothstep path and
|
|
1068
|
+
* type/click at a human cadence instead of teleporting instantly.
|
|
1069
|
+
* - `keepScreenshots` (default 30): newest captures kept per directory.
|
|
1070
|
+
* - `moveSteps` (default 16), `keyDelayMs` ([12, 34]), `clickSettleMs` ([60, 140]):
|
|
1071
|
+
* fine-tuning for the humanized cadence.
|
|
1072
|
+
* - `zoomFactor` (default 2), `zoomMaxPixels` (default 24e6): `cua_zoom` limits.
|
|
1073
|
+
* - `maxWaitMs` (default 10000): ceiling for `cua_wait`.
|
|
1074
|
+
* - `allowLaunch` (default true): set false to disable `cua_app_launch`.
|
|
1075
|
+
*/
|
|
1076
|
+
export function apply(ctx, config = {}) {
|
|
1077
|
+
for (const key of Object.keys(DEFAULTS)) {
|
|
1078
|
+
if (config[key] !== undefined) CFG[key] = config[key];
|
|
1079
|
+
}
|
|
1080
|
+
for (const decl of TOOL_DECLS) {
|
|
1081
|
+
const tool = {
|
|
1082
|
+
name: decl.name,
|
|
1083
|
+
description: decl.description,
|
|
1084
|
+
parameters: decl.parameters,
|
|
1085
|
+
output: decl.out,
|
|
1086
|
+
presentCall: (args) => ({ card: 'generic', title: decl.name, kind: 'other', rawInput: args ?? {} }),
|
|
1087
|
+
...decl.readOnly === true ? { isConcurrencySafe: () => true } : {},
|
|
1088
|
+
...decl.timeoutMs !== undefined ? { timeoutMs: decl.timeoutMs } : {},
|
|
1089
|
+
async execute(args = {}) {
|
|
1090
|
+
return decl.run(loadNative(), args ?? {});
|
|
1091
|
+
},
|
|
1092
|
+
};
|
|
1093
|
+
ctx.effect(() => ctx.tools.register(tool));
|
|
1094
|
+
}
|
|
1095
|
+
}
|
package/locale/en.json
ADDED
package/locale/zh.json
ADDED
package/package.json
CHANGED
|
@@ -1,6 +1,47 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-win-cua",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
3
|
+
"version": "0.2.0",
|
|
4
|
+
"description": "Windows computer-use for DSH: real mouse and keyboard input through Win32 SendInput, screen capture through GDI BitBlt, window control. No external driver.",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"main": "index.js",
|
|
7
|
+
"license": "MIT",
|
|
8
|
+
"author": "WJPGKZ",
|
|
9
|
+
"homepage": "https://github.com/WJPGKZ/dsh-win-cua#readme",
|
|
10
|
+
"repository": {
|
|
11
|
+
"type": "git",
|
|
12
|
+
"url": "git+https://github.com/WJPGKZ/dsh-win-cua.git"
|
|
13
|
+
},
|
|
14
|
+
"bugs": {
|
|
15
|
+
"url": "https://github.com/WJPGKZ/dsh-win-cua/issues"
|
|
16
|
+
},
|
|
17
|
+
"engines": {
|
|
18
|
+
"node": ">=20"
|
|
19
|
+
},
|
|
20
|
+
"keywords": [
|
|
21
|
+
"dsh",
|
|
22
|
+
"dsh-plugin",
|
|
23
|
+
"computer-use",
|
|
24
|
+
"win32",
|
|
25
|
+
"sendinput",
|
|
26
|
+
"bitblt",
|
|
27
|
+
"automation",
|
|
28
|
+
"windows"
|
|
29
|
+
],
|
|
30
|
+
"files": [
|
|
31
|
+
"index.js",
|
|
32
|
+
"cordis.patch.yml",
|
|
33
|
+
"locale/*.json",
|
|
34
|
+
"icon.svg",
|
|
35
|
+
"LICENSE",
|
|
36
|
+
"NOTICE.md",
|
|
37
|
+
"README.md"
|
|
38
|
+
],
|
|
39
|
+
"dsh": {
|
|
40
|
+
"bundle": {
|
|
41
|
+
"patch": "cordis.patch.yml"
|
|
42
|
+
},
|
|
43
|
+
"compatibility": {
|
|
44
|
+
"node": ">=20"
|
|
45
|
+
}
|
|
46
|
+
}
|
|
6
47
|
}
|