@denisixnpm/agent-rdp 0.7.9 → 0.7.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/README.md +287 -552
- package/dist/generated/AutomateRequest.d.ts +48 -0
- package/dist/generated/AutomateRequest.d.ts.map +1 -1
- package/dist/generated/ErrorCode.d.ts +1 -1
- package/dist/generated/ErrorCode.d.ts.map +1 -1
- package/dist/generated/FilePullRequest.d.ts +14 -0
- package/dist/generated/FilePullRequest.d.ts.map +1 -0
- package/dist/generated/FilePullRequest.js +3 -0
- package/dist/generated/FilePullRequest.js.map +1 -0
- package/dist/generated/FilePushRequest.d.ts +20 -0
- package/dist/generated/FilePushRequest.d.ts.map +1 -0
- package/dist/generated/FilePushRequest.js +3 -0
- package/dist/generated/FilePushRequest.js.map +1 -0
- package/dist/generated/FileTransferResult.d.ts +22 -0
- package/dist/generated/FileTransferResult.d.ts.map +1 -0
- package/dist/generated/FileTransferResult.js +3 -0
- package/dist/generated/FileTransferResult.js.map +1 -0
- package/dist/generated/Request.d.ts +6 -0
- package/dist/generated/Request.d.ts.map +1 -1
- package/dist/generated/ResponseData.d.ts +4 -1
- package/dist/generated/ResponseData.d.ts.map +1 -1
- package/dist/generated/index.d.ts +3 -0
- package/dist/generated/index.d.ts.map +1 -1
- package/dist/index.d.ts +19 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +35 -0
- package/dist/index.js.map +1 -1
- package/dist/types.d.ts +1 -1
- package/dist/types.d.ts.map +1 -1
- package/dist/types.js.map +1 -1
- package/package.json +7 -7
package/README.md
CHANGED
|
@@ -8,121 +8,76 @@
|
|
|
8
8
|
|
|
9
9
|
A CLI tool for AI agents to control Windows Remote Desktop sessions, built on [IronRDP](https://github.com/Devolutions/IronRDP).
|
|
10
10
|
|
|
11
|
+
Screenshots, mouse/keyboard input, clipboard sync, drive mapping, verified file
|
|
12
|
+
transfer, Windows UI Automation, OCR text location, and JSON output for every
|
|
13
|
+
command — across named sessions with automatic daemon lifecycle.
|
|
14
|
+
|
|
11
15
|
## Demo
|
|
12
16
|
|
|
13
17
|
Claude Code automating SQLite database and table creation via RDP:
|
|
14
18
|
|
|
15
19
|
https://github.com/user-attachments/assets/91892b39-4edb-412b-b265-55ccd75d7421
|
|
16
20
|
|
|
17
|
-
## Features
|
|
18
|
-
|
|
19
|
-
- **Connect to RDP servers** - Full RDP protocol support with TLS and CredSSP authentication
|
|
20
|
-
- **Take screenshots** - Capture the remote desktop as PNG or JPEG
|
|
21
|
-
- **Mouse control** - Click, double-click, right-click, drag, scroll
|
|
22
|
-
- **Keyboard input** - Type text, press key combinations (Ctrl+C, Alt+Tab, etc.)
|
|
23
|
-
- **Clipboard sync** - Copy/paste text between local machine and remote Windows
|
|
24
|
-
- **Drive mapping** - Map local directories as network drives on the remote machine
|
|
25
|
-
- **UI Automation** - Interact with Windows applications via accessibility API (click, select, toggle, expand)
|
|
26
|
-
- **OCR text location** - Find text on screen using OCR when UI Automation isn't available
|
|
27
|
-
- **JSON output** - Structured output for AI agent consumption
|
|
28
|
-
- **Session management** - Multiple named sessions with automatic daemon lifecycle
|
|
29
|
-
|
|
30
21
|
## Installation
|
|
31
22
|
|
|
32
|
-
### From npm
|
|
33
|
-
|
|
34
|
-
```bash
|
|
35
|
-
npm install -g @denisixnpm/agent-rdp
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
### As a Claude Code skill
|
|
39
|
-
|
|
40
23
|
```bash
|
|
41
|
-
|
|
24
|
+
npm install -g @denisixnpm/agent-rdp # CLI
|
|
25
|
+
npx add-skill https://github.com/denisix/agent-rdp # as a Claude Code skill
|
|
42
26
|
```
|
|
43
27
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
Each release attaches a standalone binary per platform, plus the OCR models as
|
|
47
|
-
`agent-rdp-models.tar.gz` / `agent-rdp-models.zip`. The models are
|
|
48
|
-
architecture-independent, so they ship once rather than inside every archive.
|
|
49
|
-
|
|
50
|
-
The binary alone covers everything except `locate`; that command needs the
|
|
51
|
-
models, so extract them too and point `AGENT_RDP_MODELS_DIR` at them.
|
|
28
|
+
<details>
|
|
29
|
+
<summary>From a GitHub release (standalone binary)</summary>
|
|
52
30
|
|
|
53
|
-
|
|
31
|
+
Each release attaches a binary per platform plus the OCR models as
|
|
32
|
+
`agent-rdp-models.tar.gz` / `.zip`. The models are architecture-independent, so
|
|
33
|
+
they ship once. The binary alone covers everything except `locate`, which needs
|
|
34
|
+
the models — extract them and point `AGENT_RDP_MODELS_DIR` at them.
|
|
54
35
|
|
|
55
36
|
```bash
|
|
56
37
|
tar -xzf agent-rdp-linux-x64.tar.gz -C ~/.local/bin # or agent-rdp-darwin-arm64.tar.gz
|
|
57
38
|
chmod +x ~/.local/bin/agent-rdp
|
|
58
39
|
mkdir -p ~/.local/share/agent-rdp/models
|
|
59
40
|
tar -xzf agent-rdp-models.tar.gz -C ~/.local/share/agent-rdp/models
|
|
60
|
-
export AGENT_RDP_MODELS_DIR="$HOME/.local/share/agent-rdp/models" # add to your shell rc
|
|
41
|
+
export AGENT_RDP_MODELS_DIR="$HOME/.local/share/agent-rdp/models" # add to your shell rc
|
|
61
42
|
```
|
|
62
43
|
|
|
63
|
-
On macOS the release binaries are unsigned, so Gatekeeper quarantines anything
|
|
64
|
-
downloaded from a browser. If you get *"cannot be opened because the developer
|
|
65
|
-
cannot be verified"*, clear the flag:
|
|
66
|
-
|
|
67
|
-
```bash
|
|
68
|
-
xattr -d com.apple.quarantine ~/.local/bin/agent-rdp
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
**Windows (PowerShell)**
|
|
72
|
-
|
|
73
44
|
```powershell
|
|
74
45
|
Expand-Archive agent-rdp-win32-x64.zip -DestinationPath "$env:LOCALAPPDATA\agent-rdp"
|
|
75
46
|
Expand-Archive agent-rdp-models.zip -DestinationPath "$env:LOCALAPPDATA\agent-rdp\models"
|
|
76
|
-
|
|
77
|
-
setx AGENT_RDP_MODELS_DIR "$env:LOCALAPPDATA\agent-rdp\models"
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
Installing from npm needs none of this — the models ship inside
|
|
81
|
-
`@denisixnpm/agent-rdp` and are located automatically.
|
|
82
|
-
|
|
83
|
-
### From source
|
|
84
|
-
|
|
85
|
-
```bash
|
|
86
|
-
git clone https://github.com/denisix/agent-rdp
|
|
87
|
-
cd agent-rdp
|
|
88
|
-
bun install
|
|
89
|
-
bun run build # Build native binary
|
|
90
|
-
bun run build:ts # Build TypeScript
|
|
47
|
+
setx AGENT_RDP_MODELS_DIR "$env:LOCALAPPDATA\agent-rdp\models" # restart the terminal after
|
|
91
48
|
```
|
|
92
49
|
|
|
93
|
-
|
|
50
|
+
macOS release binaries are unsigned. If Gatekeeper blocks it with *"cannot be
|
|
51
|
+
opened because the developer cannot be verified"*:
|
|
52
|
+
`xattr -d com.apple.quarantine ~/.local/bin/agent-rdp`
|
|
94
53
|
|
|
95
|
-
|
|
54
|
+
Installing from npm needs none of this — the models ship in the package.
|
|
55
|
+
</details>
|
|
96
56
|
|
|
97
|
-
|
|
57
|
+
<details>
|
|
58
|
+
<summary>From source</summary>
|
|
98
59
|
|
|
99
60
|
```bash
|
|
100
|
-
|
|
61
|
+
git clone https://github.com/denisix/agent-rdp && cd agent-rdp
|
|
62
|
+
bun install
|
|
63
|
+
bun run build # native binary
|
|
64
|
+
bun run build:ts # TypeScript
|
|
101
65
|
```
|
|
66
|
+
</details>
|
|
102
67
|
|
|
103
|
-
|
|
104
|
-
`@denisixnpm/agent-rdp` on first use if it isn't already on PATH.
|
|
105
|
-
|
|
106
|
-
Or install manually:
|
|
107
|
-
|
|
108
|
-
```bash
|
|
109
|
-
mkdir -p .claude/skills/agent-rdp
|
|
110
|
-
curl -o .claude/skills/agent-rdp/SKILL.md \
|
|
111
|
-
https://raw.githubusercontent.com/denisix/agent-rdp/main/skills/agent-rdp/SKILL.md
|
|
112
|
-
```
|
|
68
|
+
## Using with AI coding agents
|
|
113
69
|
|
|
114
|
-
|
|
70
|
+
**Claude Code** — `npx add-skill https://github.com/denisix/agent-rdp` installs
|
|
71
|
+
the [SKILL.md](skills/agent-rdp/SKILL.md) workflow so Claude knows the commands,
|
|
72
|
+
flags and gotchas without you explaining them. It installs the CLI on first use
|
|
73
|
+
if it isn't on PATH. Then ask in plain language:
|
|
115
74
|
|
|
116
75
|
```
|
|
117
76
|
Connect to 192.168.1.100 as Administrator (password: secret), open Notepad,
|
|
118
77
|
type "hello from Claude", and take a screenshot.
|
|
119
78
|
```
|
|
120
79
|
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
### Codex
|
|
124
|
-
|
|
125
|
-
Codex doesn't have a skill-install mechanism, but it reads `AGENTS.md` for project instructions. Point it at this tool by adding a section to your `AGENTS.md`:
|
|
80
|
+
**Codex** reads `AGENTS.md` instead. Add a section pointing at the tool:
|
|
126
81
|
|
|
127
82
|
```bash
|
|
128
83
|
cat >> AGENTS.md <<'EOF'
|
|
@@ -135,379 +90,235 @@ https://github.com/denisix/agent-rdp for the full command reference.
|
|
|
135
90
|
EOF
|
|
136
91
|
```
|
|
137
92
|
|
|
138
|
-
Then prompt Codex the same way:
|
|
139
|
-
|
|
140
|
-
```
|
|
141
|
-
codex "Connect to the Windows VM at 192.168.1.100 (user Administrator, password
|
|
142
|
-
secret) using agent-rdp, open the Run dialog, launch calc.exe, and confirm it's
|
|
143
|
-
open with a screenshot."
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
Codex will call `agent-rdp` as a regular shell command, same as any other CLI tool.
|
|
147
|
-
|
|
148
93
|
## Usage
|
|
149
94
|
|
|
150
|
-
### Connect
|
|
95
|
+
### Connect
|
|
151
96
|
|
|
152
97
|
```bash
|
|
153
|
-
# Using command line (password visible in process list - not recommended)
|
|
154
98
|
agent-rdp connect --host 192.168.1.100 --username Administrator --password 'secret'
|
|
155
99
|
|
|
156
|
-
#
|
|
157
|
-
export AGENT_RDP_USERNAME=Administrator
|
|
158
|
-
export AGENT_RDP_PASSWORD=secret
|
|
100
|
+
# Environment variables (recommended - keeps the password out of the process list)
|
|
101
|
+
export AGENT_RDP_USERNAME=Administrator AGENT_RDP_PASSWORD=secret
|
|
159
102
|
agent-rdp connect --host 192.168.1.100
|
|
160
103
|
|
|
161
|
-
#
|
|
162
|
-
echo 'secret' | agent-rdp connect --host 192.168.1.100
|
|
104
|
+
# stdin (most secure)
|
|
105
|
+
echo 'secret' | agent-rdp connect --host 192.168.1.100 -u Administrator --password-stdin
|
|
106
|
+
|
|
107
|
+
agent-rdp disconnect
|
|
163
108
|
```
|
|
164
109
|
|
|
165
|
-
###
|
|
110
|
+
### Screenshot
|
|
166
111
|
|
|
167
112
|
```bash
|
|
168
|
-
|
|
169
|
-
agent-rdp screenshot --output desktop.png
|
|
113
|
+
agent-rdp screenshot --output desktop.png # default ./screenshot.png
|
|
114
|
+
agent-rdp --json screenshot --output desktop.png # metadata; image always goes to disk
|
|
115
|
+
agent-rdp screenshot --region 100,380,600,30 -o row.png # crop; reports the offset back
|
|
116
|
+
```
|
|
170
117
|
|
|
171
|
-
|
|
172
|
-
|
|
118
|
+
A screenshot is the last frame the server painted, not a live poll. Each one
|
|
119
|
+
reports `frame_age_ms` (time since the server last sent anything), and `--json`
|
|
120
|
+
adds `frame_seq` and `frame_hash` — two screenshots with the same value are
|
|
121
|
+
guaranteed pixel-identical, which is how you confirm a frame actually changed
|
|
122
|
+
after an action rather than hashing the saved file yourself. A large
|
|
123
|
+
`frame_age_ms` usually just means an idle desktop, since RDP servers send
|
|
124
|
+
nothing when nothing changes; a genuinely dead connection is detected within
|
|
125
|
+
seconds by TCP keepalive and reported as disconnected.
|
|
173
126
|
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
```
|
|
127
|
+
> The CLI has no `screenshot --base64`. In Node, `rdp.screenshot({ path })`
|
|
128
|
+
> writes to disk and returns `{ path, width, height }` — prefer it over the
|
|
129
|
+
> base64-returning form when the caller doesn't need the bytes, since echoing a
|
|
130
|
+
> base64 image into an LLM context is expensive.
|
|
179
131
|
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
Every screenshot also reports `frame_age_ms` (`--json`) / a printed warning
|
|
183
|
-
(human mode, when large): milliseconds since the RDP server last sent any
|
|
184
|
-
data. An idle-but-alive desktop can legitimately sit at a large age — RDP
|
|
185
|
-
servers send nothing when nothing changes — but combined with `agent-rdp
|
|
186
|
-
session info`'s `last_frame_age_ms`, a value that never resets on an
|
|
187
|
-
otherwise-idle session is the signal to check the connection rather than
|
|
188
|
-
trust the frame. The daemon also enables TCP keepalive on the RDP socket, so
|
|
189
|
-
a genuinely dead connection (cable pull, black-holed network path) is now
|
|
190
|
-
detected and disconnected within seconds instead of the OS's own multi-minute
|
|
191
|
-
retransmission timeout — previously `screenshot` could keep returning a
|
|
192
|
-
stale, byte-identical frame under a fresh filename for the entire time the
|
|
193
|
-
transport was silently dead.
|
|
194
|
-
|
|
195
|
-
`--json` output (and the Node SDK's `frameSeq`/`frameHash`) also carries a
|
|
196
|
-
`frame_seq` (framebuffer generation counter) and `frame_hash` (FNV-1a hash of
|
|
197
|
-
the captured pixels). Two screenshots with the same seq or hash are
|
|
198
|
-
guaranteed pixel-identical — proving that directly, instead of hashing the
|
|
199
|
-
saved file yourself or reading a clock rendered on screen, is the reliable
|
|
200
|
-
way to confirm a frame actually changed after an action that should have
|
|
201
|
-
repainted it.
|
|
202
|
-
|
|
203
|
-
### Getting Coordinates Right
|
|
132
|
+
### Getting coordinates right
|
|
204
133
|
|
|
205
134
|
**Never estimate a coordinate by looking at a screenshot.** Screenshot pixels,
|
|
206
|
-
OCR boxes and click coordinates
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
lands on the wrong row, the wrong cell, or a search box.
|
|
135
|
+
OCR boxes and click coordinates share one space, so a coordinate the tool
|
|
136
|
+
returns is exact — one guessed from an image is not. Images get downscaled on
|
|
137
|
+
their way into a vision model, and a click 30px off lands on the wrong row.
|
|
210
138
|
|
|
211
139
|
Ask for the target by name and let the tool find it:
|
|
212
140
|
|
|
213
141
|
```bash
|
|
214
|
-
|
|
215
|
-
agent-rdp locate "Добавить" --click
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
agent-rdp locate "Добавить" --click --index 1
|
|
219
|
-
|
|
220
|
-
# Click a button by accessibility selector (preferred when UIA can see it)
|
|
221
|
-
agent-rdp automate click "#SaveButton"
|
|
222
|
-
```
|
|
223
|
-
|
|
224
|
-
To verify a value in place, read the region rather than cropping an image
|
|
225
|
-
yourself — coordinates still come back in full-screen space:
|
|
226
|
-
|
|
227
|
-
```bash
|
|
228
|
-
# Check one table row
|
|
229
|
-
agent-rdp locate --all --region 100,380,600,30
|
|
230
|
-
agent-rdp screenshot --region 100,380,600,30 -o row.png
|
|
231
|
-
|
|
232
|
-
# Confirm which field the keyboard is actually in before typing
|
|
233
|
-
agent-rdp automate focused
|
|
142
|
+
agent-rdp locate "Добавить" --click # coordinate never passes through your hands
|
|
143
|
+
agent-rdp locate "Добавить" --click --index 1 # several matches: choosing is explicit
|
|
144
|
+
agent-rdp automate click "#SaveButton" # preferred when UI Automation can see it
|
|
145
|
+
agent-rdp automate focused # confirm which field has focus before typing
|
|
234
146
|
```
|
|
235
147
|
|
|
236
148
|
`agent-rdp mouse click X Y` remains available for coordinates you got from
|
|
237
149
|
`locate`, `automate get`, or a deliberate calculation.
|
|
238
150
|
|
|
239
|
-
### Mouse
|
|
151
|
+
### Mouse and keyboard
|
|
240
152
|
|
|
241
153
|
```bash
|
|
242
|
-
# Click at position
|
|
243
154
|
agent-rdp mouse click 500 300
|
|
244
|
-
|
|
245
|
-
# Right-click
|
|
246
155
|
agent-rdp mouse right-click 500 300
|
|
247
|
-
|
|
248
|
-
# Double-click
|
|
249
156
|
agent-rdp mouse double-click 500 300
|
|
250
|
-
|
|
251
|
-
# Move cursor
|
|
252
157
|
agent-rdp mouse move 100 200
|
|
253
|
-
|
|
254
|
-
# Drag from (100,100) to (500,500)
|
|
255
158
|
agent-rdp mouse drag 100 100 500 500
|
|
256
|
-
```
|
|
257
|
-
|
|
258
|
-
### Keyboard Operations
|
|
259
159
|
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
agent-rdp keyboard
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
# command. More reliable than `type` - it cannot lose individual keystrokes,
|
|
266
|
-
# and there's no gap between setting the clipboard and pasting for focus to
|
|
267
|
-
# move in.
|
|
268
|
-
agent-rdp keyboard paste "Привет, мир! A very long string types can drop pieces of."
|
|
269
|
-
|
|
270
|
-
# Press key combinations
|
|
271
|
-
agent-rdp keyboard press "ctrl+c"
|
|
272
|
-
agent-rdp keyboard press "alt+tab"
|
|
273
|
-
agent-rdp keyboard press "ctrl+shift+esc"
|
|
274
|
-
|
|
275
|
-
# Press single keys (use press command)
|
|
276
|
-
agent-rdp keyboard press enter
|
|
277
|
-
agent-rdp keyboard press escape
|
|
278
|
-
agent-rdp keyboard press f5
|
|
279
|
-
|
|
280
|
-
# Hold a key across other commands (shift-click, hold-and-drag, ...)
|
|
281
|
-
agent-rdp keyboard down shift
|
|
282
|
-
agent-rdp mouse click 200 300
|
|
160
|
+
agent-rdp keyboard type "Hello, World!" # Unicode, batched into one round-trip
|
|
161
|
+
agent-rdp keyboard type "text" --delay 20 # pace it if the remote app drops fast input
|
|
162
|
+
agent-rdp keyboard paste "Привет, мир!" # clipboard + Ctrl+V as one command
|
|
163
|
+
agent-rdp keyboard press "ctrl+c" # also alt+tab, enter, escape, f5, win+r
|
|
164
|
+
agent-rdp keyboard down shift # hold across other commands
|
|
283
165
|
agent-rdp keyboard up shift
|
|
284
166
|
```
|
|
285
167
|
|
|
286
|
-
|
|
168
|
+
Prefer `keyboard paste` for long or non-Latin text: it cannot lose individual
|
|
169
|
+
keystrokes, and leaves no gap for focus to move between setting the clipboard
|
|
170
|
+
and pasting.
|
|
287
171
|
|
|
288
|
-
|
|
289
|
-
in — pass `--at` to target a specific window or control.
|
|
172
|
+
### Scroll
|
|
290
173
|
|
|
291
|
-
Amount is
|
|
174
|
+
Amount is positional (not `--amount`), and the default point is the **screen
|
|
175
|
+
center** rather than whatever pane you're working in — use `--at` to target one.
|
|
292
176
|
|
|
293
177
|
```bash
|
|
294
178
|
agent-rdp scroll up 3
|
|
295
|
-
agent-rdp scroll down 5
|
|
296
|
-
agent-rdp scroll left
|
|
297
|
-
agent-rdp scroll right
|
|
298
179
|
agent-rdp scroll down 5 --at 600 400
|
|
180
|
+
agent-rdp scroll left # also: right
|
|
299
181
|
```
|
|
300
182
|
|
|
301
|
-
###
|
|
302
|
-
|
|
303
|
-
Find text on screen using OCR (powered by [ocrs](https://github.com/robertknight/ocrs)). Useful when UI Automation can't access certain elements (WebView content, some dialogs).
|
|
183
|
+
### Clipboard and drive mapping
|
|
304
184
|
|
|
305
185
|
```bash
|
|
306
|
-
|
|
307
|
-
agent-rdp
|
|
308
|
-
|
|
309
|
-
# Pattern matching (glob-style)
|
|
310
|
-
agent-rdp locate "Save*" --pattern
|
|
311
|
-
|
|
312
|
-
# Exact whole-line matching. Default mode is substring containment, so
|
|
313
|
-
# "Провести" also matches a button reading "Провести и закрыть" - two matches,
|
|
314
|
-
# and --click refuses to guess between them. --exact matches only the line
|
|
315
|
-
# that reads exactly "Провести".
|
|
316
|
-
agent-rdp locate "Провести" --exact --click
|
|
317
|
-
|
|
318
|
-
# Get all text on screen
|
|
319
|
-
agent-rdp locate --all
|
|
320
|
-
|
|
321
|
-
# JSON output
|
|
322
|
-
agent-rdp locate "OK" --json
|
|
323
|
-
|
|
324
|
-
# Click the match directly (see "Getting Coordinates Right" above)
|
|
325
|
-
agent-rdp locate "Cancel" --click
|
|
326
|
-
agent-rdp locate "Cancel" --double-click
|
|
327
|
-
agent-rdp locate "Cancel" --right-click
|
|
328
|
-
|
|
329
|
-
# Search only part of the screen. Results stay in full-screen coordinates,
|
|
330
|
-
# so --region composes with --click. A tight region also reads small text
|
|
331
|
-
# more reliably than a full-screen pass.
|
|
332
|
-
agent-rdp locate --all --region 100,380,600,30
|
|
333
|
-
|
|
334
|
-
# Block until text appears (e.g. a dialog finishing its animation), instead
|
|
335
|
-
# of polling `locate` in a loop from the outside. Composes with --click.
|
|
336
|
-
agent-rdp locate "OK" --wait 10000 --click
|
|
337
|
-
|
|
338
|
-
# Constrain matches to those near a distinctive anchor label - useful when
|
|
339
|
-
# the same text repeats on screen (a column header in every row, a label
|
|
340
|
-
# that also appears in a tooltip). Anchor is matched by substring; if it
|
|
341
|
-
# isn't found at all, this returns zero matches rather than an error.
|
|
342
|
-
agent-rdp locate "Отменить" --near "Заказ №001" --click
|
|
343
|
-
```
|
|
344
|
-
|
|
345
|
-
Returns text lines with coordinates for clicking:
|
|
346
|
-
```
|
|
347
|
-
Found 1 line(s) containing 'Cancel':
|
|
348
|
-
'Cancel Button' at (650, 420) size 80x14 - center: (690, 427)
|
|
186
|
+
agent-rdp clipboard set "Hello from CLI"
|
|
187
|
+
agent-rdp clipboard set --file ./script.ps1 # or `--file -` to read stdin
|
|
188
|
+
agent-rdp clipboard get
|
|
349
189
|
|
|
350
|
-
|
|
190
|
+
# Drives must be mapped at connect time; multiple --drive flags are allowed
|
|
191
|
+
agent-rdp connect --host 192.168.1.100 -u Administrator -p secret \
|
|
192
|
+
--drive /home/user/documents:Documents --drive /tmp/shared:Shared
|
|
193
|
+
agent-rdp drive list
|
|
351
194
|
```
|
|
352
195
|
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
196
|
+
Mapped drives appear on the remote as network locations. Clipboard text is
|
|
197
|
+
sent with CRLF line endings, so multi-line content survives
|
|
198
|
+
`Get-Clipboard | Set-Content`; `clipboard get` returns Windows text with its
|
|
199
|
+
CRLF intact.
|
|
357
200
|
|
|
358
|
-
**
|
|
359
|
-
|
|
360
|
-
the
|
|
361
|
-
|
|
362
|
-
field) or a second independent read before acting on the amount.
|
|
201
|
+
> **Do not access `\\TSCLIENT\...` from `automate run`.** Drive redirection is
|
|
202
|
+
> serviced by the same task that carries the automation channel, so reading the
|
|
203
|
+
> share from inside the agent deadlocks — the command never returns and the
|
|
204
|
+
> session stops responding until you reconnect. Use `file push`/`file pull`.
|
|
363
205
|
|
|
364
|
-
###
|
|
206
|
+
### File transfer
|
|
365
207
|
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
isn't ambiguously close to more than one detected text region:
|
|
208
|
+
Requires `--enable-win-automation`. Copies in chunks over the automation
|
|
209
|
+
channel, with a SHA-256 computed independently on each end, so a truncated or
|
|
210
|
+
corrupted transfer fails loudly instead of leaving a plausible-looking file.
|
|
370
211
|
|
|
371
212
|
```bash
|
|
372
|
-
|
|
373
|
-
agent-rdp
|
|
374
|
-
|
|
375
|
-
# Tune the detection window and gap
|
|
376
|
-
agent-rdp click-at 665 209 --window 400x160 --min-gap 20
|
|
377
|
-
|
|
378
|
-
# Variants
|
|
379
|
-
agent-rdp click-at 665 209 --double-click
|
|
380
|
-
agent-rdp click-at 665 209 --right-click
|
|
381
|
-
|
|
382
|
-
# Cross-check: a second independent measurement for the same target (e.g. a
|
|
383
|
-
# vision model queried twice). Clicks the midpoint if the two agree within
|
|
384
|
-
# --max-divergence (default 40px); refuses if they don't. Formalizes the
|
|
385
|
-
# "click the intersection of two measurements" technique that catches a
|
|
386
|
-
# single call's misreads.
|
|
387
|
-
agent-rdp click-at 665 209 --confirm 670,212
|
|
388
|
-
agent-rdp click-at 665 209 --confirm 670,212 --max-divergence 20
|
|
213
|
+
agent-rdp file push ./report.xlsx "C:\\Users\\Admin\\report.xlsx"
|
|
214
|
+
agent-rdp file pull "C:\\Users\\Admin\\export.csv" ./export.csv
|
|
389
215
|
```
|
|
390
216
|
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
217
|
+
Transfers are byte-exact, which also makes this the reliable way to place a
|
|
218
|
+
script with non-ASCII content on the remote — writing one through the clipboard
|
|
219
|
+
or `Add-Content` re-encodes it and mangles anything outside ASCII. Limit: 128MB.
|
|
220
|
+
|
|
221
|
+
### Locate (OCR)
|
|
395
222
|
|
|
396
|
-
|
|
223
|
+
Finds text on screen with [ocrs](https://github.com/robertknight/ocrs). Use it
|
|
224
|
+
when UI Automation can't reach an element (WebView content, some dialogs).
|
|
397
225
|
|
|
398
226
|
```bash
|
|
399
|
-
|
|
400
|
-
agent-rdp
|
|
227
|
+
agent-rdp locate "Cancel" # substring match, returns coordinates
|
|
228
|
+
agent-rdp locate "Save*" --pattern # glob match
|
|
229
|
+
agent-rdp locate "Провести" --exact # whole-line match
|
|
230
|
+
agent-rdp locate --all # all text on screen
|
|
231
|
+
agent-rdp locate "Cancel" --click # also --double-click, --right-click
|
|
232
|
+
agent-rdp locate "OK" --wait 10000 --click # block until it appears, then click
|
|
233
|
+
agent-rdp locate --all --region 100,380,600,30 # coords stay full-screen
|
|
234
|
+
agent-rdp locate "Отменить" --near "Заказ №001" --click # anchor to a nearby label
|
|
235
|
+
```
|
|
401
236
|
|
|
402
|
-
|
|
403
|
-
agent-rdp clipboard get
|
|
237
|
+
Output carries coordinates ready to click:
|
|
404
238
|
|
|
405
|
-
|
|
406
|
-
|
|
239
|
+
```
|
|
240
|
+
Found 1 line(s) containing 'Cancel':
|
|
241
|
+
'Cancel Button' at (650, 420) size 80x14 - center: (690, 427)
|
|
407
242
|
```
|
|
408
243
|
|
|
409
|
-
|
|
244
|
+
Clicking is deliberately strict: no match, or several matches without
|
|
245
|
+
`--index`, is an error rather than a guess. Default matching is substring
|
|
246
|
+
containment, so `"Провести"` also matches `"Провести и закрыть"` — use
|
|
247
|
+
`--exact` to avoid the ambiguity, `--near` when the same text genuinely repeats
|
|
248
|
+
across rows, or narrow with `--region`.
|
|
249
|
+
|
|
250
|
+
**Numbers with thousands separators can lose their leading digit group** — OCR
|
|
251
|
+
has been observed reading `1 250,00` as `2250,00`. For monetary values, crop
|
|
252
|
+
tight with `--region` and verify with `automate get` or a second independent
|
|
253
|
+
read before acting on the amount.
|
|
410
254
|
|
|
411
|
-
|
|
255
|
+
### Click-at (safe clicking of externally-computed coordinates)
|
|
256
|
+
|
|
257
|
+
When the click point comes from outside agent-rdp — a vision model reading a
|
|
258
|
+
screenshot, a manual crop — `locate --click` can't help and a raw `mouse click`
|
|
259
|
+
has no safety net. `click-at` clicks only if the point isn't ambiguously close
|
|
260
|
+
to more than one detected text region:
|
|
412
261
|
|
|
413
262
|
```bash
|
|
414
|
-
|
|
415
|
-
agent-rdp
|
|
416
|
-
|
|
417
|
-
--drive /tmp/shared:Shared
|
|
263
|
+
agent-rdp click-at 665 209
|
|
264
|
+
agent-rdp click-at 665 209 --window 400x160 --min-gap 20 # tune detection window/gap
|
|
265
|
+
agent-rdp click-at 665 209 --double-click # also --right-click
|
|
418
266
|
|
|
419
|
-
#
|
|
420
|
-
|
|
267
|
+
# Cross-check two independent measurements: clicks their midpoint if they agree
|
|
268
|
+
# within --max-divergence (default 40px), refuses if they don't.
|
|
269
|
+
agent-rdp click-at 665 209 --confirm 670,212 --max-divergence 20
|
|
421
270
|
```
|
|
422
271
|
|
|
423
|
-
|
|
272
|
+
The check uses OCR *detection* only (bounding boxes, script-agnostic), not
|
|
273
|
+
recognition, so it works even for text OCR can't read — custom-rendered
|
|
274
|
+
Cyrillic UIs where both UI Automation and `locate` fail. On refusal it lists
|
|
275
|
+
the nearby regions and exits non-zero; nothing is clicked.
|
|
424
276
|
|
|
425
277
|
### UI Automation
|
|
426
278
|
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
279
|
+
Drives Windows applications through the UI Automation API using native patterns
|
|
280
|
+
(Invoke, SelectionItem, Toggle, ExpandCollapse). A PowerShell agent is injected
|
|
281
|
+
into the remote session and speaks to the daemon over a Dynamic Virtual Channel.
|
|
282
|
+
Full protocol details in [AUTOMATION.md](docs/AUTOMATION.md).
|
|
430
283
|
|
|
431
284
|
```bash
|
|
432
|
-
# Connect with automation enabled
|
|
433
285
|
agent-rdp connect --host 192.168.1.100 -u Admin -p secret --enable-win-automation
|
|
434
286
|
|
|
435
|
-
|
|
436
|
-
agent-rdp automate snapshot
|
|
437
|
-
|
|
438
|
-
#
|
|
439
|
-
|
|
440
|
-
agent-rdp automate
|
|
441
|
-
agent-rdp automate
|
|
442
|
-
agent-rdp automate
|
|
443
|
-
agent-rdp automate
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
# Use this after Tab/Enter to confirm where your keystrokes will go, instead
|
|
447
|
-
# of typing into a cell you assume is focused.
|
|
448
|
-
agent-rdp automate focused
|
|
449
|
-
# edit 'Количество' = "5,000" at (110, 391) 80x18 [focusable]
|
|
450
|
-
|
|
451
|
-
# Pattern-based element operations (refs use @eN format)
|
|
452
|
-
agent-rdp automate click "#SaveButton" # Click button
|
|
453
|
-
agent-rdp automate click "@e5" # Click by ref number from snapshot
|
|
454
|
-
agent-rdp automate click "@e5" -d # Double-click (for file list items)
|
|
455
|
-
agent-rdp automate select "@e10" # Select item (SelectionItemPattern)
|
|
456
|
-
agent-rdp automate toggle "@e7" # Toggle checkbox (TogglePattern)
|
|
457
|
-
agent-rdp automate expand "@e3" # Expand menu (ExpandCollapsePattern)
|
|
458
|
-
agent-rdp automate context-menu "@e5" # Open context menu (Shift+F10)
|
|
459
|
-
|
|
460
|
-
# Fill text fields
|
|
287
|
+
agent-rdp automate snapshot # full tree (refs always included)
|
|
288
|
+
agent-rdp automate snapshot -i -c -d 3 # interactive only, compact, depth 3
|
|
289
|
+
agent-rdp automate snapshot -s "~*Notepad*" # scope to a window
|
|
290
|
+
agent-rdp automate focused # what has keyboard focus right now
|
|
291
|
+
|
|
292
|
+
agent-rdp automate click "@e5" # also: "#SaveButton", ".Edit", "~*wild*"
|
|
293
|
+
agent-rdp automate click "@e5" -d # double-click
|
|
294
|
+
agent-rdp automate select "@e10" --item "Option 1"
|
|
295
|
+
agent-rdp automate toggle "@e7" --state on
|
|
296
|
+
agent-rdp automate expand "@e3" # also: collapse, context-menu, focus, clear
|
|
297
|
+
agent-rdp automate get "@e2" # includes Value: for text/multiline edits
|
|
461
298
|
agent-rdp automate fill ".Edit" "Hello World"
|
|
299
|
+
agent-rdp automate scroll "@e4" --direction down --amount 3
|
|
300
|
+
agent-rdp automate wait-for "#SaveButton" --timeout 5000 --state visible
|
|
462
301
|
|
|
463
|
-
# Window operations
|
|
464
302
|
agent-rdp automate window list
|
|
465
303
|
agent-rdp automate window focus "~*Notepad*"
|
|
304
|
+
agent-rdp automate window maximize|minimize|restore|close
|
|
466
305
|
|
|
467
|
-
|
|
468
|
-
agent-rdp automate run "
|
|
469
|
-
agent-rdp automate run "
|
|
470
|
-
agent-rdp automate run
|
|
306
|
+
agent-rdp automate run "Get-Process" --wait --process-timeout 5000
|
|
307
|
+
agent-rdp automate run "$PSVersionTable" --wait --shell pwsh.exe
|
|
308
|
+
agent-rdp automate run "ping -t 127.0.0.1" --stream # returns a pid immediately
|
|
309
|
+
agent-rdp automate run-poll <pid> # drain output; repeat until exited
|
|
471
310
|
|
|
472
|
-
|
|
473
|
-
agent-rdp automate
|
|
474
|
-
|
|
311
|
+
agent-rdp automate status # uptime, last DVC round-trip, failure count
|
|
312
|
+
agent-rdp automate restart # relaunch the agent, keeping the RDP session
|
|
313
|
+
```
|
|
475
314
|
|
|
476
|
-
|
|
477
|
-
|
|
315
|
+
**Selectors:** `@e5` (snapshot ref), `#SaveButton` (automation ID), `.Edit`
|
|
316
|
+
(Win32 class), `~*pattern*` (wildcard name), `File` (exact name).
|
|
478
317
|
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
agent-rdp automate restart
|
|
483
|
-
```
|
|
318
|
+
**Snapshot format**, with `disabled` shown on interactive elements — a free
|
|
319
|
+
pre-action state check (a disabled "Отменить проведение" tells you the document
|
|
320
|
+
isn't posted before you click anything):
|
|
484
321
|
|
|
485
|
-
An `automation indeterminate` error means the DVC reply was lost, not that
|
|
486
|
-
the action necessarily failed or succeeded - check state before retrying, or
|
|
487
|
-
a retried click/fill can apply twice. For read-only commands (`snapshot`,
|
|
488
|
-
`get`, `status`, `wait-for`, `window list`) the error text now says so
|
|
489
|
-
explicitly ("This command is read-only - retrying is safe"), since those can
|
|
490
|
-
always be retried safely.
|
|
491
|
-
|
|
492
|
-
Snapshots include `disabled` on interactive elements — a free pre-action
|
|
493
|
-
state check. A disabled "Отменить проведение" menu item, for example, tells
|
|
494
|
-
you the document isn't posted before you ever click anything.
|
|
495
|
-
|
|
496
|
-
**Keyboard navigation inside a panel (arrow keys + Enter) can land on the
|
|
497
|
-
wrong item.** Observed in 1C side panels ("Функции" etc.): Up/Down selection
|
|
498
|
-
followed by Enter is not reliably deterministic. Prefer `automate` refs when
|
|
499
|
-
the panel is exposed to UI Automation; when it isn't, use two independent
|
|
500
|
-
coordinate measurements and [`click-at --confirm`](#click-at-safe-clicking-of-externally-computed-coordinates)
|
|
501
|
-
rather than arrow-key navigation.
|
|
502
|
-
|
|
503
|
-
**Selector Types:**
|
|
504
|
-
- `@e5` or `@5` - Reference number from snapshot (e prefix recommended)
|
|
505
|
-
- `#SaveButton` - Automation ID
|
|
506
|
-
- `.Edit` - Win32 class name
|
|
507
|
-
- `~*pattern*` - Wildcard name match
|
|
508
|
-
- `File` - Element name (exact match)
|
|
509
|
-
|
|
510
|
-
**Snapshot Output Format:**
|
|
511
322
|
```
|
|
512
323
|
- Window "Notepad" [ref=e1, id=Notepad]
|
|
513
324
|
- MenuBar "Application" [ref=e2]
|
|
@@ -515,81 +326,64 @@ rather than arrow-key navigation.
|
|
|
515
326
|
- Edit "Text Editor" [ref=e5, value="Hello"]
|
|
516
327
|
```
|
|
517
328
|
|
|
518
|
-
|
|
329
|
+
**`automation indeterminate`** means the reply was lost, not that the action
|
|
330
|
+
failed. The agent journals recent results, so the daemon asks what actually
|
|
331
|
+
happened and usually returns the real outcome — or reports that the request
|
|
332
|
+
never ran and is safe to retry. A surviving `indeterminate` means the agent is
|
|
333
|
+
still busy: check state before retrying, or a click/fill can apply twice.
|
|
334
|
+
Read-only commands (`snapshot`, `get`, `status`, `wait-for`, `window list`) say
|
|
335
|
+
so explicitly, since those are always safe to retry.
|
|
336
|
+
|
|
337
|
+
**Long-running commands** get the time they ask for: the transport deadline,
|
|
338
|
+
the CLI socket timeout and the watchdog all extend to cover
|
|
339
|
+
`--process-timeout`/`--timeout`. But the agent handles one command at a time,
|
|
340
|
+
so a long `run --wait` blocks every other `automate` call — past about a
|
|
341
|
+
minute, prefer `run --stream` plus `run-poll`. Streamed output is captured to
|
|
342
|
+
files on the remote side, so nothing is lost if the process exits between
|
|
343
|
+
polls; a finished process stays pollable for 10 minutes and repeat polls return
|
|
344
|
+
`exited: true` with empty chunks. (If you redirect inside the command instead,
|
|
345
|
+
note that Windows PowerShell 5.1's `>` writes UTF-16LE.)
|
|
346
|
+
|
|
347
|
+
**`daemon_not_running` vs `daemon_unresponsive`.** The first means no daemon
|
|
348
|
+
process exists — reconnect, and `<session>/daemon.log` says why it exited. The
|
|
349
|
+
second means the process is alive but did not answer a health check within
|
|
350
|
+
10s: it is busy (a long `run --wait`, a file transfer). Wait and retry; do not
|
|
351
|
+
reconnect, that discards a working session. The message includes the tail of
|
|
352
|
+
daemon.log. `connect` replaces a daemon that stays unresponsive, killing the
|
|
353
|
+
stuck one first.
|
|
354
|
+
|
|
355
|
+
**Arrow-key navigation inside a panel can land on the wrong item.** Observed in
|
|
356
|
+
1C side panels: Up/Down then Enter is not reliably deterministic. Prefer
|
|
357
|
+
`automate` refs; when the panel isn't exposed to UI Automation, use two
|
|
358
|
+
independent measurements with [`click-at --confirm`](#click-at-safe-clicking-of-externally-computed-coordinates).
|
|
359
|
+
|
|
360
|
+
### Sessions and the web viewer
|
|
519
361
|
|
|
520
362
|
```bash
|
|
521
|
-
# List active sessions
|
|
522
363
|
agent-rdp session list
|
|
523
|
-
|
|
524
|
-
# Get current session info
|
|
525
364
|
agent-rdp session info
|
|
526
|
-
|
|
527
|
-
# Close a session
|
|
528
|
-
agent-rdp session close
|
|
529
|
-
|
|
530
|
-
# Use a named session
|
|
531
|
-
agent-rdp --session work connect --host work-pc.local ...
|
|
365
|
+
agent-rdp --session work connect --host work-pc.local ... # named session
|
|
532
366
|
agent-rdp --session work screenshot
|
|
533
|
-
```
|
|
534
|
-
|
|
535
|
-
### Disconnect
|
|
536
|
-
|
|
537
|
-
```bash
|
|
538
|
-
agent-rdp disconnect
|
|
539
|
-
```
|
|
540
367
|
|
|
541
|
-
|
|
368
|
+
agent-rdp wait 3000 # pause; useful right after connect, before the first input
|
|
542
369
|
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
```bash
|
|
546
|
-
# Open viewer (connects to default streaming port 9224)
|
|
547
|
-
agent-rdp view
|
|
548
|
-
|
|
549
|
-
# Specify a different port
|
|
550
|
-
agent-rdp view --port 9224
|
|
551
|
-
```
|
|
552
|
-
|
|
553
|
-
The viewer requires WebSocket streaming to be enabled. Start a session with streaming:
|
|
554
|
-
|
|
555
|
-
```bash
|
|
370
|
+
# Web viewer (needs streaming enabled at connect)
|
|
556
371
|
agent-rdp --stream-port 9224 connect --host 192.168.1.100 -u Admin -p secret
|
|
557
|
-
agent-rdp view
|
|
372
|
+
agent-rdp view --port 9224
|
|
558
373
|
```
|
|
559
374
|
|
|
560
|
-
|
|
375
|
+
There is no `session close` — use `disconnect`.
|
|
561
376
|
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
```bash
|
|
565
|
-
agent-rdp --json screenshot --output desktop.png
|
|
566
|
-
```
|
|
377
|
+
## JSON output
|
|
567
378
|
|
|
568
|
-
|
|
569
|
-
```json
|
|
570
|
-
{
|
|
571
|
-
"success": true,
|
|
572
|
-
"data": {
|
|
573
|
-
"type": "screenshot",
|
|
574
|
-
"path": "desktop.png",
|
|
575
|
-
"width": 1920,
|
|
576
|
-
"height": 1080
|
|
577
|
-
}
|
|
578
|
-
}
|
|
579
|
-
```
|
|
379
|
+
Add `--json` to any command:
|
|
580
380
|
|
|
581
|
-
**Error response:**
|
|
582
381
|
```json
|
|
583
|
-
{
|
|
584
|
-
|
|
585
|
-
"error": {
|
|
586
|
-
"code": "not_connected",
|
|
587
|
-
"message": "Not connected to an RDP server"
|
|
588
|
-
}
|
|
589
|
-
}
|
|
382
|
+
{ "success": true, "data": { "type": "screenshot", "path": "desktop.png", "width": 1920, "height": 1080 } }
|
|
383
|
+
{ "success": false, "error": { "code": "not_connected", "message": "Not connected to an RDP server" } }
|
|
590
384
|
```
|
|
591
385
|
|
|
592
|
-
## Environment
|
|
386
|
+
## Environment variables
|
|
593
387
|
|
|
594
388
|
| Variable | Description |
|
|
595
389
|
|----------|-------------|
|
|
@@ -599,173 +393,108 @@ agent-rdp --json screenshot --output desktop.png
|
|
|
599
393
|
| `AGENT_RDP_PASSWORD` | RDP password |
|
|
600
394
|
| `AGENT_RDP_SESSION` | Session name (default: "default") |
|
|
601
395
|
| `AGENT_RDP_STREAM_PORT` | WebSocket streaming port (0 = disabled) |
|
|
602
|
-
| `AGENT_RDP_MODELS_DIR` |
|
|
396
|
+
| `AGENT_RDP_MODELS_DIR` | OCR models directory (set automatically by the npm wrapper; needed for standalone binary installs) |
|
|
603
397
|
|
|
604
398
|
## Node.js API
|
|
605
399
|
|
|
606
|
-
Use agent-rdp programmatically from Node.js/TypeScript:
|
|
607
|
-
|
|
608
400
|
```typescript
|
|
609
401
|
import { RdpSession } from 'agent-rdp';
|
|
610
402
|
|
|
611
403
|
const rdp = new RdpSession({ session: 'default' });
|
|
612
|
-
|
|
613
404
|
await rdp.connect({
|
|
614
|
-
host: '192.168.1.100',
|
|
615
|
-
|
|
616
|
-
password: 'secret',
|
|
617
|
-
width: 1280,
|
|
618
|
-
height: 800,
|
|
405
|
+
host: '192.168.1.100', username: 'Administrator', password: 'secret',
|
|
406
|
+
width: 1280, height: 800,
|
|
619
407
|
drives: [{ path: '/tmp/share', name: 'Share' }],
|
|
620
|
-
enableWinAutomation: true,
|
|
408
|
+
enableWinAutomation: true,
|
|
621
409
|
});
|
|
622
410
|
|
|
623
|
-
// Screenshot - prefer `path` so a large base64 string never
|
|
624
|
-
|
|
625
|
-
const {
|
|
411
|
+
// Screenshot - prefer `path` so a large base64 string is never held in memory
|
|
412
|
+
const { path, width, height } = await rdp.screenshot({ format: 'png', path: 'shot.png' });
|
|
413
|
+
const { base64 } = await rdp.screenshot({ format: 'png' }); // or raw, for in-process use
|
|
626
414
|
|
|
627
|
-
|
|
628
|
-
const { base64 } = await rdp.screenshot({ format: 'png' });
|
|
629
|
-
|
|
630
|
-
// Mouse
|
|
631
|
-
await rdp.mouse.click({ x: 100, y: 200 });
|
|
632
|
-
await rdp.mouse.rightClick({ x: 100, y: 200 });
|
|
633
|
-
await rdp.mouse.doubleClick({ x: 100, y: 200 });
|
|
634
|
-
await rdp.mouse.move({ x: 150, y: 250 });
|
|
415
|
+
await rdp.mouse.click({ x: 100, y: 200 }); // also rightClick, doubleClick, move
|
|
635
416
|
await rdp.mouse.drag({ from: { x: 100, y: 100 }, to: { x: 500, y: 500 } });
|
|
636
417
|
|
|
637
|
-
// Keyboard
|
|
638
418
|
await rdp.keyboard.type({ text: 'Hello World' });
|
|
639
|
-
await rdp.keyboard.paste('Привет, мир!');
|
|
419
|
+
await rdp.keyboard.paste('Привет, мир!'); // reliable for long/non-Latin text
|
|
640
420
|
await rdp.keyboard.press({ keys: 'ctrl+c' });
|
|
641
|
-
await rdp.keyboard.
|
|
642
|
-
await rdp.keyboard.down('shift'); // Hold across other commands
|
|
643
|
-
await rdp.keyboard.up('shift');
|
|
644
|
-
|
|
645
|
-
// Scroll
|
|
646
|
-
await rdp.scroll.up(); // Default amount: 3
|
|
647
|
-
await rdp.scroll.down({ amount: 5 }); // Custom amount
|
|
648
|
-
await rdp.scroll.up({ x: 500, y: 300 }); // Scroll at position
|
|
421
|
+
await rdp.keyboard.down('shift'); await rdp.keyboard.up('shift');
|
|
649
422
|
|
|
650
|
-
//
|
|
423
|
+
await rdp.scroll.down({ amount: 5 }); // default 3; { x, y } to target a point
|
|
651
424
|
await rdp.clipboard.set({ text: 'text to copy' });
|
|
652
425
|
const text = await rdp.clipboard.get();
|
|
653
426
|
|
|
654
|
-
//
|
|
655
|
-
const matches = await rdp.locate({ text: 'Cancel' });
|
|
656
|
-
if (matches.length > 0) {
|
|
657
|
-
await rdp.mouse.click({ x: matches[0].center_x, y: matches[0].center_y });
|
|
658
|
-
}
|
|
659
|
-
|
|
660
|
-
// Get all text on screen
|
|
661
|
-
const allText = await rdp.locate({ all: true });
|
|
662
|
-
|
|
663
|
-
// Click a match directly - the coordinate never leaves the process
|
|
427
|
+
// OCR - click directly so the coordinate never leaves the process
|
|
664
428
|
await rdp.locate({ text: 'Cancel', click: 'left' });
|
|
665
|
-
|
|
666
|
-
// Exact whole-line match - won't also hit "Провести и закрыть"
|
|
667
429
|
await rdp.locate({ text: 'Провести', exact: true, click: 'left' });
|
|
430
|
+
await rdp.locate({ text: 'OK', waitMs: 10000, click: 'left' }); // block until it appears
|
|
431
|
+
const allText = await rdp.locate({ all: true });
|
|
668
432
|
|
|
669
|
-
//
|
|
670
|
-
await rdp.locate({ text: 'OK', waitMs: 10000, click: 'left' });
|
|
671
|
-
|
|
672
|
-
// Safe click of an externally-computed point (e.g. a vision-model bbox):
|
|
673
|
-
// refuses if it's ambiguously close to more than one detected label
|
|
433
|
+
// Safe click of an externally-computed point (e.g. a vision-model bbox)
|
|
674
434
|
const result = await rdp.clickAt(665, 209);
|
|
675
435
|
if (!result.clicked) console.log('Ambiguous:', result.nearby);
|
|
676
436
|
|
|
677
|
-
//
|
|
437
|
+
// File transfer (needs enableWinAutomation)
|
|
438
|
+
await rdp.files.push('./report.xlsx', 'C:\\Users\\Admin\\report.xlsx');
|
|
439
|
+
await rdp.files.pull('C:\\Users\\Admin\\export.csv', './export.csv');
|
|
440
|
+
|
|
441
|
+
// UI Automation
|
|
678
442
|
const snapshot = await rdp.automation.snapshot({ interactive: true });
|
|
679
|
-
const focused = await rdp.automation.focused();
|
|
680
|
-
await rdp.automation.click('@e5'
|
|
681
|
-
await rdp.automation.
|
|
682
|
-
await rdp.automation.
|
|
683
|
-
await rdp.automation.toggle('@e7'); // Toggle checkbox
|
|
684
|
-
await rdp.automation.expand('@e3'); // Expand menu
|
|
685
|
-
await rdp.automation.contextMenu('@e5'); // Open context menu
|
|
686
|
-
await rdp.automation.fill('#input', 'text'); // Fill text field
|
|
687
|
-
await rdp.automation.run('notepad.exe'); // Run command
|
|
443
|
+
const focused = await rdp.automation.focused();
|
|
444
|
+
await rdp.automation.click('@e5', { doubleClick: true });
|
|
445
|
+
await rdp.automation.fill('#input', 'text');
|
|
446
|
+
await rdp.automation.run('notepad.exe');
|
|
688
447
|
await rdp.automation.waitFor('#SaveButton', { timeout: 5000 });
|
|
689
|
-
|
|
690
|
-
// Window management
|
|
691
|
-
const windows = await rdp.automation.listWindows();
|
|
692
448
|
await rdp.automation.focusWindow('~*Notepad*');
|
|
693
|
-
await rdp.automation.maximizeWindow();
|
|
694
449
|
|
|
695
|
-
// Drives
|
|
696
450
|
const drives = await rdp.drives.list();
|
|
697
|
-
|
|
698
|
-
// Session info
|
|
699
451
|
const info = await rdp.getInfo();
|
|
700
|
-
|
|
701
|
-
// Disconnect
|
|
702
452
|
await rdp.disconnect();
|
|
703
453
|
```
|
|
704
454
|
|
|
705
|
-
|
|
706
|
-
|
|
707
|
-
Enable WebSocket streaming for real-time screen capture and bidirectional clipboard support:
|
|
708
|
-
|
|
709
|
-
```typescript
|
|
710
|
-
const rdp = new RdpSession({
|
|
711
|
-
session: 'viewer',
|
|
712
|
-
streamPort: 9224, // Enable streaming
|
|
713
|
-
});
|
|
714
|
-
|
|
715
|
-
await rdp.connect({...});
|
|
716
|
-
|
|
717
|
-
// Connect your WebSocket client to receive JPEG frames
|
|
718
|
-
const streamUrl = rdp.getStreamUrl(); // "ws://localhost:9224"
|
|
719
|
-
```
|
|
455
|
+
The process can stay alive after `disconnect()` unless the caller cleans up —
|
|
456
|
+
call `rdp.close()` or exit explicitly.
|
|
720
457
|
|
|
721
|
-
|
|
458
|
+
**WebSocket streaming** for real-time capture: construct with
|
|
459
|
+
`{ streamPort: 9224 }`, then `rdp.getStreamUrl()` returns `ws://localhost:9224`.
|
|
460
|
+
Message types, clipboard flow and input handling are specified in
|
|
461
|
+
[WEBSOCKET.md](docs/WEBSOCKET.md).
|
|
722
462
|
|
|
723
463
|
## Architecture
|
|
724
464
|
|
|
725
|
-
|
|
726
|
-
|
|
727
|
-
|
|
728
|
-
|
|
729
|
-
3. **IPC** - Unix sockets (macOS/Linux) or TCP (Windows)
|
|
730
|
-
|
|
731
|
-
The daemon is automatically started on the first command and persists until explicitly closed or the session times out.
|
|
465
|
+
A daemon per session: the **CLI** parses commands and talks to a **daemon** over
|
|
466
|
+
Unix sockets (macOS/Linux) or TCP (Windows); the daemon owns the RDP connection
|
|
467
|
+
and processes commands. It starts on the first command and persists until
|
|
468
|
+
closed. See [CLAUDE.md](CLAUDE.md) for the internals.
|
|
732
469
|
|
|
733
470
|
## Limitations
|
|
734
471
|
|
|
735
|
-
|
|
736
|
-
|
|
737
|
-
|
|
738
|
-
- **UAC
|
|
739
|
-
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
```
|
|
748
|
-
|
|
749
|
-
This is not highly reliable (OCR can misread characters, miss text, or return imprecise coordinates), but may work for simple cases like dialog buttons.
|
|
750
|
-
|
|
751
|
-
### Screenshot Coordinate Detection
|
|
752
|
-
|
|
753
|
-
**Claude models** (in non-computer-use mode, such as Claude Code) are poor at estimating pixel coordinates from screenshots. Do not ask Claude to look at a screenshot and guess where to click - it will likely be inaccurate.
|
|
754
|
-
|
|
755
|
-
**Gemini models** are generally good at pixel coordinate estimation from images.
|
|
756
|
-
|
|
757
|
-
If you need vision-based coordinate detection with Claude, implement your own harness using Claude's [Computer Use Tool](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use) which is specifically designed for this purpose.
|
|
472
|
+
- **WebViews** — UI Automation cannot see WebView content (Start menu search,
|
|
473
|
+
Edge, Electron apps). Launch programs with `automate run` or Win+R instead of
|
|
474
|
+
clicking through menus.
|
|
475
|
+
- **UAC dialogs** run on a secure desktop and are invisible to UI Automation.
|
|
476
|
+
There is no good workaround short of the remote user acting manually.
|
|
477
|
+
- **OCR** can misread characters, miss text, or return imprecise coordinates.
|
|
478
|
+
Use it only when UI Automation can't reach an element, and verify before
|
|
479
|
+
clicking anything destructive.
|
|
480
|
+
- **Claude models in non-computer-use mode** (Claude Code included) are poor at
|
|
481
|
+
estimating pixel coordinates from screenshots — don't ask for a guess from an
|
|
482
|
+
image. Gemini models are generally good at it; for Claude, use the
|
|
483
|
+
[Computer Use Tool](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use).
|
|
758
484
|
|
|
759
485
|
## Requirements
|
|
760
486
|
|
|
761
|
-
|
|
762
|
-
|
|
763
|
-
|
|
487
|
+
Rust 1.75+. The target must offer **TLS** for RDP — agent-rdp uses `rustls`,
|
|
488
|
+
which doesn't implement TLS 1.0/1.1, so legacy hosts (e.g. Windows Server
|
|
489
|
+
2008 R2) fail with a handshake error. **Stock Windows defaults work as
|
|
490
|
+
shipped.**
|
|
764
491
|
|
|
765
|
-
|
|
492
|
+
<details>
|
|
493
|
+
<summary>RDP security layer settings</summary>
|
|
766
494
|
|
|
767
|
-
All six combinations measured against Windows Server 2022, varying only these
|
|
768
|
-
values under
|
|
495
|
+
All six combinations measured against Windows Server 2022, varying only these
|
|
496
|
+
two values under
|
|
497
|
+
`HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp`:
|
|
769
498
|
|
|
770
499
|
| `SecurityLayer` | `UserAuthentication` | Meaning | Result |
|
|
771
500
|
|---|---|---|---|
|
|
@@ -777,12 +506,12 @@ values under `HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations
|
|
|
777
506
|
| `0` (RDP) | `0` | legacy RC4 only | fails |
|
|
778
507
|
|
|
779
508
|
The rule: **agent-rdp works wherever the host offers TLS.** With NLA on that is
|
|
780
|
-
always the case,
|
|
509
|
+
always the case, since NLA forces CredSSP/TLS regardless of `SecurityLayer`.
|
|
781
510
|
Both failing rows are NLA-off hosts that refuse TLS outright.
|
|
782
511
|
|
|
783
512
|
If `connect` reports *"server only supports Standard RDP Security"*, the host is
|
|
784
|
-
in one of those
|
|
785
|
-
|
|
513
|
+
in one of those rows. Enable NLA (preferred — it also gives pre-authentication)
|
|
514
|
+
or force TLS:
|
|
786
515
|
|
|
787
516
|
```powershell
|
|
788
517
|
$k = 'HKLM:\System\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp'
|
|
@@ -791,18 +520,24 @@ Set-ItemProperty -Path $k -Name SecurityLayer -Value 2 # force TLS
|
|
|
791
520
|
```
|
|
792
521
|
|
|
793
522
|
New connections normally pick this up immediately; restart `TermService` only if
|
|
794
|
-
they don't (
|
|
523
|
+
they don't (that drops existing sessions).
|
|
795
524
|
|
|
796
525
|
Neither failing case is fixable client-side. With `SecurityLayer=1` and NLA off
|
|
797
526
|
the server returns `SSL_NOT_ALLOWED_BY_SERVER` even though the client advertises
|
|
798
|
-
`PROTOCOL_SSL` — verified by testing a build that requested SSL *only*,
|
|
799
|
-
|
|
800
|
-
|
|
801
|
-
|
|
527
|
+
`PROTOCOL_SSL` — verified by testing a build that requested SSL *only*, rejected
|
|
528
|
+
identically. Standard RDP Security (`SecurityLayer=0`, NLA off) is refused by
|
|
529
|
+
IronRDP itself, which implements no RC4 transport; it uses a well-known key
|
|
530
|
+
derivation, so credentials sent over it are recoverable in transit.
|
|
531
|
+
</details>
|
|
802
532
|
|
|
803
533
|
## Credits
|
|
804
534
|
|
|
805
|
-
Originally created by [Nick Yu](https://github.com/thisnick)
|
|
535
|
+
Originally created by [Nick Yu](https://github.com/thisnick)
|
|
536
|
+
([thisnick/agent-rdp](https://github.com/thisnick/agent-rdp)). This fork
|
|
537
|
+
([denisix/agent-rdp](https://github.com/denisix/agent-rdp), published as
|
|
538
|
+
[`@denisixnpm/agent-rdp`](https://www.npmjs.com/package/@denisixnpm/agent-rdp))
|
|
539
|
+
is maintained independently; see
|
|
540
|
+
[CHANGELOG.md](packages/agent-rdp/CHANGELOG.md) for what's changed.
|
|
806
541
|
|
|
807
542
|
## License
|
|
808
543
|
|