@denisixnpm/agent-rdp 0.7.10 → 0.7.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/README.md +330 -580
- package/dist/daemon.d.ts +20 -0
- package/dist/daemon.d.ts.map +1 -1
- package/dist/daemon.js +94 -6
- package/dist/daemon.js.map +1 -1
- package/dist/generated/AutomateRequest.d.ts +9 -0
- package/dist/generated/AutomateRequest.d.ts.map +1 -1
- package/dist/generated/AutomationStatus.d.ts +5 -0
- package/dist/generated/AutomationStatus.d.ts.map +1 -1
- package/dist/generated/ErrorCode.d.ts +1 -1
- package/dist/generated/ErrorCode.d.ts.map +1 -1
- package/dist/generated/ResponseData.d.ts +13 -0
- package/dist/generated/ResponseData.d.ts.map +1 -1
- package/dist/generated/RunResult.d.ts +6 -0
- package/dist/generated/RunResult.d.ts.map +1 -1
- package/dist/generated/SessionInfo.d.ts +5 -0
- package/dist/generated/SessionInfo.d.ts.map +1 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/package.json +7 -7
package/README.md
CHANGED
|
@@ -8,121 +8,76 @@
|
|
|
8
8
|
|
|
9
9
|
A CLI tool for AI agents to control Windows Remote Desktop sessions, built on [IronRDP](https://github.com/Devolutions/IronRDP).
|
|
10
10
|
|
|
11
|
+
Screenshots, mouse/keyboard input, clipboard sync, drive mapping, verified file
|
|
12
|
+
transfer, Windows UI Automation, OCR text location, and JSON output for every
|
|
13
|
+
command — across named sessions with automatic daemon lifecycle.
|
|
14
|
+
|
|
11
15
|
## Demo
|
|
12
16
|
|
|
13
17
|
Claude Code automating SQLite database and table creation via RDP:
|
|
14
18
|
|
|
15
19
|
https://github.com/user-attachments/assets/91892b39-4edb-412b-b265-55ccd75d7421
|
|
16
20
|
|
|
17
|
-
## Features
|
|
18
|
-
|
|
19
|
-
- **Connect to RDP servers** - Full RDP protocol support with TLS and CredSSP authentication
|
|
20
|
-
- **Take screenshots** - Capture the remote desktop as PNG or JPEG
|
|
21
|
-
- **Mouse control** - Click, double-click, right-click, drag, scroll
|
|
22
|
-
- **Keyboard input** - Type text, press key combinations (Ctrl+C, Alt+Tab, etc.)
|
|
23
|
-
- **Clipboard sync** - Copy/paste text between local machine and remote Windows
|
|
24
|
-
- **Drive mapping** - Map local directories as network drives on the remote machine
|
|
25
|
-
- **UI Automation** - Interact with Windows applications via accessibility API (click, select, toggle, expand)
|
|
26
|
-
- **OCR text location** - Find text on screen using OCR when UI Automation isn't available
|
|
27
|
-
- **JSON output** - Structured output for AI agent consumption
|
|
28
|
-
- **Session management** - Multiple named sessions with automatic daemon lifecycle
|
|
29
|
-
|
|
30
21
|
## Installation
|
|
31
22
|
|
|
32
|
-
### From npm
|
|
33
|
-
|
|
34
|
-
```bash
|
|
35
|
-
npm install -g @denisixnpm/agent-rdp
|
|
36
|
-
```
|
|
37
|
-
|
|
38
|
-
### As a Claude Code skill
|
|
39
|
-
|
|
40
23
|
```bash
|
|
41
|
-
|
|
24
|
+
npm install -g @denisixnpm/agent-rdp # CLI
|
|
25
|
+
npx add-skill https://github.com/denisix/agent-rdp # as a Claude Code skill
|
|
42
26
|
```
|
|
43
27
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
Each release attaches a standalone binary per platform, plus the OCR models as
|
|
47
|
-
`agent-rdp-models.tar.gz` / `agent-rdp-models.zip`. The models are
|
|
48
|
-
architecture-independent, so they ship once rather than inside every archive.
|
|
49
|
-
|
|
50
|
-
The binary alone covers everything except `locate`; that command needs the
|
|
51
|
-
models, so extract them too and point `AGENT_RDP_MODELS_DIR` at them.
|
|
28
|
+
<details>
|
|
29
|
+
<summary>From a GitHub release (standalone binary)</summary>
|
|
52
30
|
|
|
53
|
-
|
|
31
|
+
Each release attaches a binary per platform plus the OCR models as
|
|
32
|
+
`agent-rdp-models.tar.gz` / `.zip`. The models are architecture-independent, so
|
|
33
|
+
they ship once. The binary alone covers everything except `locate`, which needs
|
|
34
|
+
the models — extract them and point `AGENT_RDP_MODELS_DIR` at them.
|
|
54
35
|
|
|
55
36
|
```bash
|
|
56
37
|
tar -xzf agent-rdp-linux-x64.tar.gz -C ~/.local/bin # or agent-rdp-darwin-arm64.tar.gz
|
|
57
38
|
chmod +x ~/.local/bin/agent-rdp
|
|
58
39
|
mkdir -p ~/.local/share/agent-rdp/models
|
|
59
40
|
tar -xzf agent-rdp-models.tar.gz -C ~/.local/share/agent-rdp/models
|
|
60
|
-
export AGENT_RDP_MODELS_DIR="$HOME/.local/share/agent-rdp/models" # add to your shell rc
|
|
41
|
+
export AGENT_RDP_MODELS_DIR="$HOME/.local/share/agent-rdp/models" # add to your shell rc
|
|
61
42
|
```
|
|
62
43
|
|
|
63
|
-
On macOS the release binaries are unsigned, so Gatekeeper quarantines anything
|
|
64
|
-
downloaded from a browser. If you get *"cannot be opened because the developer
|
|
65
|
-
cannot be verified"*, clear the flag:
|
|
66
|
-
|
|
67
|
-
```bash
|
|
68
|
-
xattr -d com.apple.quarantine ~/.local/bin/agent-rdp
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
**Windows (PowerShell)**
|
|
72
|
-
|
|
73
44
|
```powershell
|
|
74
45
|
Expand-Archive agent-rdp-win32-x64.zip -DestinationPath "$env:LOCALAPPDATA\agent-rdp"
|
|
75
46
|
Expand-Archive agent-rdp-models.zip -DestinationPath "$env:LOCALAPPDATA\agent-rdp\models"
|
|
76
|
-
|
|
77
|
-
setx AGENT_RDP_MODELS_DIR "$env:LOCALAPPDATA\agent-rdp\models"
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
Installing from npm needs none of this — the models ship inside
|
|
81
|
-
`@denisixnpm/agent-rdp` and are located automatically.
|
|
82
|
-
|
|
83
|
-
### From source
|
|
84
|
-
|
|
85
|
-
```bash
|
|
86
|
-
git clone https://github.com/denisix/agent-rdp
|
|
87
|
-
cd agent-rdp
|
|
88
|
-
bun install
|
|
89
|
-
bun run build # Build native binary
|
|
90
|
-
bun run build:ts # Build TypeScript
|
|
47
|
+
setx AGENT_RDP_MODELS_DIR "$env:LOCALAPPDATA\agent-rdp\models" # restart the terminal after
|
|
91
48
|
```
|
|
92
49
|
|
|
93
|
-
|
|
50
|
+
macOS release binaries are unsigned. If Gatekeeper blocks it with *"cannot be
|
|
51
|
+
opened because the developer cannot be verified"*:
|
|
52
|
+
`xattr -d com.apple.quarantine ~/.local/bin/agent-rdp`
|
|
94
53
|
|
|
95
|
-
|
|
54
|
+
Installing from npm needs none of this — the models ship in the package.
|
|
55
|
+
</details>
|
|
96
56
|
|
|
97
|
-
|
|
57
|
+
<details>
|
|
58
|
+
<summary>From source</summary>
|
|
98
59
|
|
|
99
60
|
```bash
|
|
100
|
-
|
|
61
|
+
git clone https://github.com/denisix/agent-rdp && cd agent-rdp
|
|
62
|
+
bun install
|
|
63
|
+
bun run build # native binary
|
|
64
|
+
bun run build:ts # TypeScript
|
|
101
65
|
```
|
|
66
|
+
</details>
|
|
102
67
|
|
|
103
|
-
|
|
104
|
-
`@denisixnpm/agent-rdp` on first use if it isn't already on PATH.
|
|
105
|
-
|
|
106
|
-
Or install manually:
|
|
107
|
-
|
|
108
|
-
```bash
|
|
109
|
-
mkdir -p .claude/skills/agent-rdp
|
|
110
|
-
curl -o .claude/skills/agent-rdp/SKILL.md \
|
|
111
|
-
https://raw.githubusercontent.com/denisix/agent-rdp/main/skills/agent-rdp/SKILL.md
|
|
112
|
-
```
|
|
68
|
+
## Using with AI coding agents
|
|
113
69
|
|
|
114
|
-
|
|
70
|
+
**Claude Code** — `npx add-skill https://github.com/denisix/agent-rdp` installs
|
|
71
|
+
the [SKILL.md](skills/agent-rdp/SKILL.md) workflow so Claude knows the commands,
|
|
72
|
+
flags and gotchas without you explaining them. It installs the CLI on first use
|
|
73
|
+
if it isn't on PATH. Then ask in plain language:
|
|
115
74
|
|
|
116
75
|
```
|
|
117
76
|
Connect to 192.168.1.100 as Administrator (password: secret), open Notepad,
|
|
118
77
|
type "hello from Claude", and take a screenshot.
|
|
119
78
|
```
|
|
120
79
|
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
### Codex
|
|
124
|
-
|
|
125
|
-
Codex doesn't have a skill-install mechanism, but it reads `AGENTS.md` for project instructions. Point it at this tool by adding a section to your `AGENTS.md`:
|
|
80
|
+
**Codex** reads `AGENTS.md` instead. Add a section pointing at the tool:
|
|
126
81
|
|
|
127
82
|
```bash
|
|
128
83
|
cat >> AGENTS.md <<'EOF'
|
|
@@ -135,414 +90,237 @@ https://github.com/denisix/agent-rdp for the full command reference.
|
|
|
135
90
|
EOF
|
|
136
91
|
```
|
|
137
92
|
|
|
138
|
-
Then prompt Codex the same way:
|
|
139
|
-
|
|
140
|
-
```
|
|
141
|
-
codex "Connect to the Windows VM at 192.168.1.100 (user Administrator, password
|
|
142
|
-
secret) using agent-rdp, open the Run dialog, launch calc.exe, and confirm it's
|
|
143
|
-
open with a screenshot."
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
Codex will call `agent-rdp` as a regular shell command, same as any other CLI tool.
|
|
147
|
-
|
|
148
93
|
## Usage
|
|
149
94
|
|
|
150
|
-
### Connect
|
|
95
|
+
### Connect
|
|
151
96
|
|
|
152
97
|
```bash
|
|
153
|
-
# Using command line (password visible in process list - not recommended)
|
|
154
98
|
agent-rdp connect --host 192.168.1.100 --username Administrator --password 'secret'
|
|
155
99
|
|
|
156
|
-
#
|
|
157
|
-
export AGENT_RDP_USERNAME=Administrator
|
|
158
|
-
export AGENT_RDP_PASSWORD=secret
|
|
100
|
+
# Environment variables (recommended - keeps the password out of the process list)
|
|
101
|
+
export AGENT_RDP_USERNAME=Administrator AGENT_RDP_PASSWORD=secret
|
|
159
102
|
agent-rdp connect --host 192.168.1.100
|
|
160
103
|
|
|
161
|
-
#
|
|
162
|
-
echo 'secret' | agent-rdp connect --host 192.168.1.100
|
|
104
|
+
# stdin (most secure)
|
|
105
|
+
echo 'secret' | agent-rdp connect --host 192.168.1.100 -u Administrator --password-stdin
|
|
106
|
+
|
|
107
|
+
agent-rdp disconnect
|
|
163
108
|
```
|
|
164
109
|
|
|
165
|
-
###
|
|
110
|
+
### Screenshot
|
|
166
111
|
|
|
167
112
|
```bash
|
|
168
|
-
|
|
169
|
-
agent-rdp screenshot --output desktop.png
|
|
113
|
+
agent-rdp screenshot --output desktop.png # default ./screenshot.png
|
|
114
|
+
agent-rdp --json screenshot --output desktop.png # metadata; image always goes to disk
|
|
115
|
+
agent-rdp screenshot --region 100,380,600,30 -o row.png # crop; reports the offset back
|
|
116
|
+
```
|
|
170
117
|
|
|
171
|
-
|
|
172
|
-
|
|
118
|
+
A screenshot is the last frame the server painted, not a live poll. Each one
|
|
119
|
+
reports `frame_age_ms` (time since the server last sent anything), and `--json`
|
|
120
|
+
adds `frame_seq` and `frame_hash` — two screenshots with the same value are
|
|
121
|
+
guaranteed pixel-identical, which is how you confirm a frame actually changed
|
|
122
|
+
after an action rather than hashing the saved file yourself. A large
|
|
123
|
+
`frame_age_ms` usually just means an idle desktop, since RDP servers send
|
|
124
|
+
nothing when nothing changes; a genuinely dead connection is detected within
|
|
125
|
+
seconds by TCP keepalive and reported as disconnected.
|
|
173
126
|
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
```
|
|
127
|
+
> The CLI has no `screenshot --base64`. In Node, `rdp.screenshot({ path })`
|
|
128
|
+
> writes to disk and returns `{ path, width, height }` — prefer it over the
|
|
129
|
+
> base64-returning form when the caller doesn't need the bytes, since echoing a
|
|
130
|
+
> base64 image into an LLM context is expensive.
|
|
179
131
|
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
Every screenshot also reports `frame_age_ms` (`--json`) / a printed warning
|
|
183
|
-
(human mode, when large): milliseconds since the RDP server last sent any
|
|
184
|
-
data. An idle-but-alive desktop can legitimately sit at a large age — RDP
|
|
185
|
-
servers send nothing when nothing changes — but combined with `agent-rdp
|
|
186
|
-
session info`'s `last_frame_age_ms`, a value that never resets on an
|
|
187
|
-
otherwise-idle session is the signal to check the connection rather than
|
|
188
|
-
trust the frame. The daemon also enables TCP keepalive on the RDP socket, so
|
|
189
|
-
a genuinely dead connection (cable pull, black-holed network path) is now
|
|
190
|
-
detected and disconnected within seconds instead of the OS's own multi-minute
|
|
191
|
-
retransmission timeout — previously `screenshot` could keep returning a
|
|
192
|
-
stale, byte-identical frame under a fresh filename for the entire time the
|
|
193
|
-
transport was silently dead.
|
|
194
|
-
|
|
195
|
-
`--json` output (and the Node SDK's `frameSeq`/`frameHash`) also carries a
|
|
196
|
-
`frame_seq` (framebuffer generation counter) and `frame_hash` (FNV-1a hash of
|
|
197
|
-
the captured pixels). Two screenshots with the same seq or hash are
|
|
198
|
-
guaranteed pixel-identical — proving that directly, instead of hashing the
|
|
199
|
-
saved file yourself or reading a clock rendered on screen, is the reliable
|
|
200
|
-
way to confirm a frame actually changed after an action that should have
|
|
201
|
-
repainted it.
|
|
202
|
-
|
|
203
|
-
### Getting Coordinates Right
|
|
132
|
+
### Getting coordinates right
|
|
204
133
|
|
|
205
134
|
**Never estimate a coordinate by looking at a screenshot.** Screenshot pixels,
|
|
206
|
-
OCR boxes and click coordinates
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
lands on the wrong row, the wrong cell, or a search box.
|
|
135
|
+
OCR boxes and click coordinates share one space, so a coordinate the tool
|
|
136
|
+
returns is exact — one guessed from an image is not. Images get downscaled on
|
|
137
|
+
their way into a vision model, and a click 30px off lands on the wrong row.
|
|
210
138
|
|
|
211
139
|
Ask for the target by name and let the tool find it:
|
|
212
140
|
|
|
213
141
|
```bash
|
|
214
|
-
|
|
215
|
-
agent-rdp locate "Добавить" --click
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
agent-rdp locate "Добавить" --click --index 1
|
|
219
|
-
|
|
220
|
-
# Click a button by accessibility selector (preferred when UIA can see it)
|
|
221
|
-
agent-rdp automate click "#SaveButton"
|
|
222
|
-
```
|
|
223
|
-
|
|
224
|
-
To verify a value in place, read the region rather than cropping an image
|
|
225
|
-
yourself — coordinates still come back in full-screen space:
|
|
226
|
-
|
|
227
|
-
```bash
|
|
228
|
-
# Check one table row
|
|
229
|
-
agent-rdp locate --all --region 100,380,600,30
|
|
230
|
-
agent-rdp screenshot --region 100,380,600,30 -o row.png
|
|
231
|
-
|
|
232
|
-
# Confirm which field the keyboard is actually in before typing
|
|
233
|
-
agent-rdp automate focused
|
|
142
|
+
agent-rdp locate "Добавить" --click # coordinate never passes through your hands
|
|
143
|
+
agent-rdp locate "Добавить" --click --index 1 # several matches: choosing is explicit
|
|
144
|
+
agent-rdp automate click "#SaveButton" # preferred when UI Automation can see it
|
|
145
|
+
agent-rdp automate focused # confirm which field has focus before typing
|
|
234
146
|
```
|
|
235
147
|
|
|
236
148
|
`agent-rdp mouse click X Y` remains available for coordinates you got from
|
|
237
149
|
`locate`, `automate get`, or a deliberate calculation.
|
|
238
150
|
|
|
239
|
-
### Mouse
|
|
151
|
+
### Mouse and keyboard
|
|
240
152
|
|
|
241
153
|
```bash
|
|
242
|
-
# Click at position
|
|
243
154
|
agent-rdp mouse click 500 300
|
|
244
|
-
|
|
245
|
-
# Right-click
|
|
246
155
|
agent-rdp mouse right-click 500 300
|
|
247
|
-
|
|
248
|
-
# Double-click
|
|
249
156
|
agent-rdp mouse double-click 500 300
|
|
250
|
-
|
|
251
|
-
# Move cursor
|
|
252
157
|
agent-rdp mouse move 100 200
|
|
253
|
-
|
|
254
|
-
# Drag from (100,100) to (500,500)
|
|
255
158
|
agent-rdp mouse drag 100 100 500 500
|
|
256
|
-
```
|
|
257
|
-
|
|
258
|
-
### Keyboard Operations
|
|
259
159
|
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
agent-rdp keyboard
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
# command. More reliable than `type` - it cannot lose individual keystrokes,
|
|
266
|
-
# and there's no gap between setting the clipboard and pasting for focus to
|
|
267
|
-
# move in.
|
|
268
|
-
agent-rdp keyboard paste "Привет, мир! A very long string types can drop pieces of."
|
|
269
|
-
|
|
270
|
-
# Press key combinations
|
|
271
|
-
agent-rdp keyboard press "ctrl+c"
|
|
272
|
-
agent-rdp keyboard press "alt+tab"
|
|
273
|
-
agent-rdp keyboard press "ctrl+shift+esc"
|
|
274
|
-
|
|
275
|
-
# Press single keys (use press command)
|
|
276
|
-
agent-rdp keyboard press enter
|
|
277
|
-
agent-rdp keyboard press escape
|
|
278
|
-
agent-rdp keyboard press f5
|
|
279
|
-
|
|
280
|
-
# Hold a key across other commands (shift-click, hold-and-drag, ...)
|
|
281
|
-
agent-rdp keyboard down shift
|
|
282
|
-
agent-rdp mouse click 200 300
|
|
160
|
+
agent-rdp keyboard type "Hello, World!" # Unicode, batched into one round-trip
|
|
161
|
+
agent-rdp keyboard type "text" --delay 20 # pace it if the remote app drops fast input
|
|
162
|
+
agent-rdp keyboard paste "Привет, мир!" # clipboard + Ctrl+V as one command
|
|
163
|
+
agent-rdp keyboard press "ctrl+c" # also alt+tab, enter, escape, f5, win+r
|
|
164
|
+
agent-rdp keyboard down shift # hold across other commands
|
|
283
165
|
agent-rdp keyboard up shift
|
|
284
166
|
```
|
|
285
167
|
|
|
286
|
-
|
|
168
|
+
Prefer `keyboard paste` for long or non-Latin text: it cannot lose individual
|
|
169
|
+
keystrokes, and leaves no gap for focus to move between setting the clipboard
|
|
170
|
+
and pasting.
|
|
287
171
|
|
|
288
|
-
|
|
289
|
-
in — pass `--at` to target a specific window or control.
|
|
172
|
+
### Scroll
|
|
290
173
|
|
|
291
|
-
Amount is
|
|
174
|
+
Amount is positional (not `--amount`), and the default point is the **screen
|
|
175
|
+
center** rather than whatever pane you're working in — use `--at` to target one.
|
|
292
176
|
|
|
293
177
|
```bash
|
|
294
178
|
agent-rdp scroll up 3
|
|
295
|
-
agent-rdp scroll down 5
|
|
296
|
-
agent-rdp scroll left
|
|
297
|
-
agent-rdp scroll right
|
|
298
179
|
agent-rdp scroll down 5 --at 600 400
|
|
180
|
+
agent-rdp scroll left # also: right
|
|
299
181
|
```
|
|
300
182
|
|
|
301
|
-
###
|
|
302
|
-
|
|
303
|
-
Find text on screen using OCR (powered by [ocrs](https://github.com/robertknight/ocrs)). Useful when UI Automation can't access certain elements (WebView content, some dialogs).
|
|
183
|
+
### Clipboard and drive mapping
|
|
304
184
|
|
|
305
185
|
```bash
|
|
306
|
-
|
|
307
|
-
agent-rdp
|
|
308
|
-
|
|
309
|
-
# Pattern matching (glob-style)
|
|
310
|
-
agent-rdp locate "Save*" --pattern
|
|
311
|
-
|
|
312
|
-
# Exact whole-line matching. Default mode is substring containment, so
|
|
313
|
-
# "Провести" also matches a button reading "Провести и закрыть" - two matches,
|
|
314
|
-
# and --click refuses to guess between them. --exact matches only the line
|
|
315
|
-
# that reads exactly "Провести".
|
|
316
|
-
agent-rdp locate "Провести" --exact --click
|
|
317
|
-
|
|
318
|
-
# Get all text on screen
|
|
319
|
-
agent-rdp locate --all
|
|
320
|
-
|
|
321
|
-
# JSON output
|
|
322
|
-
agent-rdp locate "OK" --json
|
|
323
|
-
|
|
324
|
-
# Click the match directly (see "Getting Coordinates Right" above)
|
|
325
|
-
agent-rdp locate "Cancel" --click
|
|
326
|
-
agent-rdp locate "Cancel" --double-click
|
|
327
|
-
agent-rdp locate "Cancel" --right-click
|
|
328
|
-
|
|
329
|
-
# Search only part of the screen. Results stay in full-screen coordinates,
|
|
330
|
-
# so --region composes with --click. A tight region also reads small text
|
|
331
|
-
# more reliably than a full-screen pass.
|
|
332
|
-
agent-rdp locate --all --region 100,380,600,30
|
|
333
|
-
|
|
334
|
-
# Block until text appears (e.g. a dialog finishing its animation), instead
|
|
335
|
-
# of polling `locate` in a loop from the outside. Composes with --click.
|
|
336
|
-
agent-rdp locate "OK" --wait 10000 --click
|
|
337
|
-
|
|
338
|
-
# Constrain matches to those near a distinctive anchor label - useful when
|
|
339
|
-
# the same text repeats on screen (a column header in every row, a label
|
|
340
|
-
# that also appears in a tooltip). Anchor is matched by substring; if it
|
|
341
|
-
# isn't found at all, this returns zero matches rather than an error.
|
|
342
|
-
agent-rdp locate "Отменить" --near "Заказ №001" --click
|
|
343
|
-
```
|
|
344
|
-
|
|
345
|
-
Returns text lines with coordinates for clicking:
|
|
346
|
-
```
|
|
347
|
-
Found 1 line(s) containing 'Cancel':
|
|
348
|
-
'Cancel Button' at (650, 420) size 80x14 - center: (690, 427)
|
|
186
|
+
agent-rdp clipboard set "Hello from CLI"
|
|
187
|
+
agent-rdp clipboard set --file ./script.ps1 # or `--file -` to read stdin
|
|
188
|
+
agent-rdp clipboard get
|
|
349
189
|
|
|
350
|
-
|
|
190
|
+
# Drives must be mapped at connect time; multiple --drive flags are allowed
|
|
191
|
+
agent-rdp connect --host 192.168.1.100 -u Administrator -p secret \
|
|
192
|
+
--drive /home/user/documents:Documents --drive /tmp/shared:Shared
|
|
193
|
+
agent-rdp drive list
|
|
351
194
|
```
|
|
352
195
|
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
196
|
+
Mapped drives appear on the remote as network locations. Clipboard text is
|
|
197
|
+
sent with CRLF line endings, so multi-line content survives
|
|
198
|
+
`Get-Clipboard | Set-Content`; `clipboard get` returns Windows text with its
|
|
199
|
+
CRLF intact.
|
|
357
200
|
|
|
358
|
-
**
|
|
359
|
-
|
|
360
|
-
the
|
|
361
|
-
|
|
362
|
-
field) or a second independent read before acting on the amount.
|
|
201
|
+
> **Do not access `\\TSCLIENT\...` from `automate run`.** Drive redirection is
|
|
202
|
+
> serviced by the same task that carries the automation channel, so reading the
|
|
203
|
+
> share from inside the agent deadlocks — the command never returns and the
|
|
204
|
+
> session stops responding until you reconnect. Use `file push`/`file pull`.
|
|
363
205
|
|
|
364
|
-
###
|
|
206
|
+
### File transfer
|
|
365
207
|
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
isn't ambiguously close to more than one detected text region:
|
|
208
|
+
Requires `--enable-win-automation`. Copies in chunks over the automation
|
|
209
|
+
channel, with a SHA-256 computed independently on each end, so a truncated or
|
|
210
|
+
corrupted transfer fails loudly instead of leaving a plausible-looking file.
|
|
370
211
|
|
|
371
212
|
```bash
|
|
372
|
-
|
|
373
|
-
agent-rdp
|
|
374
|
-
|
|
375
|
-
# Tune the detection window and gap
|
|
376
|
-
agent-rdp click-at 665 209 --window 400x160 --min-gap 20
|
|
377
|
-
|
|
378
|
-
# Variants
|
|
379
|
-
agent-rdp click-at 665 209 --double-click
|
|
380
|
-
agent-rdp click-at 665 209 --right-click
|
|
381
|
-
|
|
382
|
-
# Cross-check: a second independent measurement for the same target (e.g. a
|
|
383
|
-
# vision model queried twice). Clicks the midpoint if the two agree within
|
|
384
|
-
# --max-divergence (default 40px); refuses if they don't. Formalizes the
|
|
385
|
-
# "click the intersection of two measurements" technique that catches a
|
|
386
|
-
# single call's misreads.
|
|
387
|
-
agent-rdp click-at 665 209 --confirm 670,212
|
|
388
|
-
agent-rdp click-at 665 209 --confirm 670,212 --max-divergence 20
|
|
213
|
+
agent-rdp file push ./report.xlsx "C:\\Users\\Admin\\report.xlsx"
|
|
214
|
+
agent-rdp file pull "C:\\Users\\Admin\\export.csv" ./export.csv
|
|
389
215
|
```
|
|
390
216
|
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
refusal it lists the nearby regions and exits non-zero; nothing is clicked.
|
|
395
|
-
|
|
396
|
-
### Clipboard
|
|
217
|
+
Transfers are byte-exact, which also makes this the reliable way to place a
|
|
218
|
+
script with non-ASCII content on the remote — writing one through the clipboard
|
|
219
|
+
or `Add-Content` re-encodes it and mangles anything outside ASCII. Limit: 128MB.
|
|
397
220
|
|
|
398
|
-
|
|
399
|
-
# Set clipboard text (available when you paste on Windows)
|
|
400
|
-
agent-rdp clipboard set "Hello from CLI"
|
|
221
|
+
### Locate (OCR)
|
|
401
222
|
|
|
402
|
-
|
|
403
|
-
|
|
223
|
+
Finds text on screen with [ocrs](https://github.com/robertknight/ocrs). Use it
|
|
224
|
+
when UI Automation can't reach an element (WebView content, some dialogs).
|
|
404
225
|
|
|
405
|
-
|
|
406
|
-
agent-rdp
|
|
226
|
+
```bash
|
|
227
|
+
agent-rdp locate "Cancel" # substring match, returns coordinates
|
|
228
|
+
agent-rdp locate "Save*" --pattern # glob match
|
|
229
|
+
agent-rdp locate "Провести" --exact # whole-line match
|
|
230
|
+
agent-rdp locate --all # all text on screen
|
|
231
|
+
agent-rdp locate "Cancel" --click # also --double-click, --right-click
|
|
232
|
+
agent-rdp locate "OK" --wait 10000 --click # block until it appears, then click
|
|
233
|
+
agent-rdp locate --all --region 100,380,600,30 # coords stay full-screen
|
|
234
|
+
agent-rdp locate "Отменить" --near "Заказ №001" --click # anchor to a nearby label
|
|
407
235
|
```
|
|
408
236
|
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
Map local directories as network drives on the remote Windows machine. Drives must be mapped at connect time. Multiple drives can be specified.
|
|
412
|
-
|
|
413
|
-
```bash
|
|
414
|
-
# Map local directories during connection
|
|
415
|
-
agent-rdp connect --host 192.168.1.100 -u Administrator -p secret \
|
|
416
|
-
--drive /home/user/documents:Documents \
|
|
417
|
-
--drive /tmp/shared:Shared
|
|
237
|
+
Output carries coordinates ready to click:
|
|
418
238
|
|
|
419
|
-
|
|
420
|
-
|
|
239
|
+
```
|
|
240
|
+
Found 1 line(s) containing 'Cancel':
|
|
241
|
+
'Cancel Button' at (650, 420) size 80x14 - center: (690, 427)
|
|
421
242
|
```
|
|
422
243
|
|
|
423
|
-
|
|
244
|
+
Clicking is deliberately strict: no match, or several matches without
|
|
245
|
+
`--index`, is an error rather than a guess. Default matching is substring
|
|
246
|
+
containment, so `"Провести"` also matches `"Провести и закрыть"` — use
|
|
247
|
+
`--exact` to avoid the ambiguity, `--near` when the same text genuinely repeats
|
|
248
|
+
across rows, or narrow with `--region`.
|
|
424
249
|
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
> below to move files instead.
|
|
250
|
+
**Numbers with thousands separators can lose their leading digit group** — OCR
|
|
251
|
+
has been observed reading `1 250,00` as `2250,00`. For monetary values, crop
|
|
252
|
+
tight with `--region` and verify with `automate get` or a second independent
|
|
253
|
+
read before acting on the amount.
|
|
430
254
|
|
|
431
|
-
###
|
|
255
|
+
### Click-at (safe clicking of externally-computed coordinates)
|
|
432
256
|
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
257
|
+
When the click point comes from outside agent-rdp — a vision model reading a
|
|
258
|
+
screenshot, a manual crop — `locate --click` can't help and a raw `mouse click`
|
|
259
|
+
has no safety net. `click-at` clicks only if the point isn't ambiguously close
|
|
260
|
+
to more than one detected text region:
|
|
437
261
|
|
|
438
262
|
```bash
|
|
439
|
-
agent-rdp
|
|
440
|
-
agent-rdp
|
|
263
|
+
agent-rdp click-at 665 209
|
|
264
|
+
agent-rdp click-at 665 209 --window 400x160 --min-gap 20 # tune detection window/gap
|
|
265
|
+
agent-rdp click-at 665 209 --double-click # also --right-click
|
|
266
|
+
|
|
267
|
+
# Cross-check two independent measurements: clicks their midpoint if they agree
|
|
268
|
+
# within --max-divergence (default 40px), refuses if they don't.
|
|
269
|
+
agent-rdp click-at 665 209 --confirm 670,212 --max-divergence 20
|
|
441
270
|
```
|
|
442
271
|
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
272
|
+
The check uses OCR *detection* only (bounding boxes, script-agnostic), not
|
|
273
|
+
recognition, so it works even for text OCR can't read — custom-rendered
|
|
274
|
+
Cyrillic UIs where both UI Automation and `locate` fail. On refusal it lists
|
|
275
|
+
the nearby regions and exits non-zero; nothing is clicked.
|
|
447
276
|
|
|
448
277
|
### UI Automation
|
|
449
278
|
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
279
|
+
Drives Windows applications through the UI Automation API using native patterns
|
|
280
|
+
(Invoke, SelectionItem, Toggle, ExpandCollapse). A PowerShell agent is injected
|
|
281
|
+
into the remote session and speaks to the daemon over a Dynamic Virtual Channel.
|
|
282
|
+
Full protocol details in [AUTOMATION.md](docs/AUTOMATION.md).
|
|
453
283
|
|
|
454
284
|
```bash
|
|
455
|
-
# Connect with automation enabled
|
|
456
285
|
agent-rdp connect --host 192.168.1.100 -u Admin -p secret --enable-win-automation
|
|
457
286
|
|
|
458
|
-
|
|
459
|
-
agent-rdp automate snapshot
|
|
460
|
-
|
|
461
|
-
#
|
|
462
|
-
|
|
463
|
-
agent-rdp automate
|
|
464
|
-
agent-rdp automate
|
|
465
|
-
agent-rdp automate
|
|
466
|
-
agent-rdp automate
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
# Use this after Tab/Enter to confirm where your keystrokes will go, instead
|
|
470
|
-
# of typing into a cell you assume is focused.
|
|
471
|
-
agent-rdp automate focused
|
|
472
|
-
# edit 'Количество' = "5,000" at (110, 391) 80x18 [focusable]
|
|
473
|
-
|
|
474
|
-
# Pattern-based element operations (refs use @eN format)
|
|
475
|
-
agent-rdp automate click "#SaveButton" # Click button
|
|
476
|
-
agent-rdp automate click "@e5" # Click by ref number from snapshot
|
|
477
|
-
agent-rdp automate click "@e5" -d # Double-click (for file list items)
|
|
478
|
-
agent-rdp automate select "@e10" # Select item (SelectionItemPattern)
|
|
479
|
-
agent-rdp automate toggle "@e7" # Toggle checkbox (TogglePattern)
|
|
480
|
-
agent-rdp automate expand "@e3" # Expand menu (ExpandCollapsePattern)
|
|
481
|
-
agent-rdp automate context-menu "@e5" # Open context menu (Shift+F10)
|
|
482
|
-
|
|
483
|
-
# Fill text fields
|
|
287
|
+
agent-rdp automate snapshot # full tree (refs always included)
|
|
288
|
+
agent-rdp automate snapshot -i -c -d 3 # interactive only, compact, depth 3
|
|
289
|
+
agent-rdp automate snapshot -s "~*Notepad*" # scope to a window
|
|
290
|
+
agent-rdp automate focused # what has keyboard focus right now
|
|
291
|
+
|
|
292
|
+
agent-rdp automate click "@e5" # also: "#SaveButton", ".Edit", "~*wild*"
|
|
293
|
+
agent-rdp automate click "@e5" -d # double-click
|
|
294
|
+
agent-rdp automate select "@e10" --item "Option 1"
|
|
295
|
+
agent-rdp automate toggle "@e7" --state on
|
|
296
|
+
agent-rdp automate expand "@e3" # also: collapse, context-menu, focus, clear
|
|
297
|
+
agent-rdp automate get "@e2" # includes Value: for text/multiline edits
|
|
484
298
|
agent-rdp automate fill ".Edit" "Hello World"
|
|
299
|
+
agent-rdp automate scroll "@e4" --direction down --amount 3
|
|
300
|
+
agent-rdp automate wait-for "#SaveButton" --timeout 5000 --state visible
|
|
485
301
|
|
|
486
|
-
# Window operations
|
|
487
302
|
agent-rdp automate window list
|
|
488
303
|
agent-rdp automate window focus "~*Notepad*"
|
|
304
|
+
agent-rdp automate window maximize|minimize|restore|close
|
|
489
305
|
|
|
490
|
-
|
|
491
|
-
agent-rdp automate run "
|
|
492
|
-
agent-rdp automate run "
|
|
493
|
-
agent-rdp automate run
|
|
306
|
+
agent-rdp automate run "Get-Process" --wait --process-timeout 5000
|
|
307
|
+
agent-rdp automate run "$PSVersionTable" --wait --shell pwsh.exe
|
|
308
|
+
agent-rdp automate run "ping -t 127.0.0.1" --stream # returns a pid immediately
|
|
309
|
+
agent-rdp automate run-poll <pid> # drain output; repeat until exited
|
|
310
|
+
agent-rdp automate run "Add-Content C:\log.txt x" --wait --idempotency-key step-07
|
|
311
|
+
# a retry with the same key replays, never re-runs
|
|
494
312
|
|
|
495
|
-
|
|
496
|
-
agent-rdp automate
|
|
497
|
-
|
|
313
|
+
agent-rdp automate status # uptime, last DVC round-trip, failure count
|
|
314
|
+
agent-rdp automate restart # relaunch the agent, keeping the RDP session
|
|
315
|
+
```
|
|
498
316
|
|
|
499
|
-
|
|
500
|
-
|
|
317
|
+
**Selectors:** `@e5` (snapshot ref), `#SaveButton` (automation ID), `.Edit`
|
|
318
|
+
(Win32 class), `~*pattern*` (wildcard name), `File` (exact name).
|
|
501
319
|
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
agent-rdp automate restart
|
|
506
|
-
```
|
|
320
|
+
**Snapshot format**, with `disabled` shown on interactive elements — a free
|
|
321
|
+
pre-action state check (a disabled "Отменить проведение" tells you the document
|
|
322
|
+
isn't posted before you click anything):
|
|
507
323
|
|
|
508
|
-
An `automation indeterminate` error means the DVC reply was lost, not that
|
|
509
|
-
the action necessarily failed or succeeded. The agent now keeps the results of
|
|
510
|
-
its recent requests, so when a reply goes missing the daemon asks it what
|
|
511
|
-
actually happened and usually returns the real outcome instead - or, if the
|
|
512
|
-
agent has no record of the request, says outright that it never ran and
|
|
513
|
-
retrying is safe. Only when the agent is still busy does a genuine
|
|
514
|
-
`indeterminate` remain; check state before retrying in that case, or a
|
|
515
|
-
retried click/fill can apply twice. Read-only commands (`snapshot`, `get`,
|
|
516
|
-
`status`, `wait-for`, `window list`) say so explicitly, since those can always
|
|
517
|
-
be retried safely.
|
|
518
|
-
|
|
519
|
-
**Long-running commands.** `run --wait` and `wait-for` are given the time
|
|
520
|
-
they ask for: the transport deadline, the CLI's socket timeout and the
|
|
521
|
-
watchdog all extend to cover `--process-timeout`/`--timeout`. Note the agent
|
|
522
|
-
processes one command at a time, so a long `run --wait` blocks every other
|
|
523
|
-
`automate` command until it finishes — for anything beyond about a minute,
|
|
524
|
-
prefer `run --stream` plus `run-poll`, which returns immediately and lets you
|
|
525
|
-
drain output while the session stays responsive.
|
|
526
|
-
|
|
527
|
-
Snapshots include `disabled` on interactive elements — a free pre-action
|
|
528
|
-
state check. A disabled "Отменить проведение" menu item, for example, tells
|
|
529
|
-
you the document isn't posted before you ever click anything.
|
|
530
|
-
|
|
531
|
-
**Keyboard navigation inside a panel (arrow keys + Enter) can land on the
|
|
532
|
-
wrong item.** Observed in 1C side panels ("Функции" etc.): Up/Down selection
|
|
533
|
-
followed by Enter is not reliably deterministic. Prefer `automate` refs when
|
|
534
|
-
the panel is exposed to UI Automation; when it isn't, use two independent
|
|
535
|
-
coordinate measurements and [`click-at --confirm`](#click-at-safe-clicking-of-externally-computed-coordinates)
|
|
536
|
-
rather than arrow-key navigation.
|
|
537
|
-
|
|
538
|
-
**Selector Types:**
|
|
539
|
-
- `@e5` or `@5` - Reference number from snapshot (e prefix recommended)
|
|
540
|
-
- `#SaveButton` - Automation ID
|
|
541
|
-
- `.Edit` - Win32 class name
|
|
542
|
-
- `~*pattern*` - Wildcard name match
|
|
543
|
-
- `File` - Element name (exact match)
|
|
544
|
-
|
|
545
|
-
**Snapshot Output Format:**
|
|
546
324
|
```
|
|
547
325
|
- Window "Notepad" [ref=e1, id=Notepad]
|
|
548
326
|
- MenuBar "Application" [ref=e2]
|
|
@@ -550,81 +328,111 @@ rather than arrow-key navigation.
|
|
|
550
328
|
- Edit "Text Editor" [ref=e5, value="Hello"]
|
|
551
329
|
```
|
|
552
330
|
|
|
553
|
-
|
|
331
|
+
**`automation indeterminate`** means the reply was lost, not that the action
|
|
332
|
+
failed. The agent journals recent results, so the daemon asks what actually
|
|
333
|
+
happened and usually returns the real outcome — or reports that the request
|
|
334
|
+
never ran and is safe to retry. A surviving `indeterminate` means the agent is
|
|
335
|
+
still busy: check state before retrying, or a click/fill can apply twice.
|
|
336
|
+
Read-only commands (`snapshot`, `get`, `status`, `wait-for`, `window list`) say
|
|
337
|
+
so explicitly, since those are always safe to retry.
|
|
338
|
+
|
|
339
|
+
**Retrying `run` safely.** Give a mutating `run` an `--idempotency-key`. A
|
|
340
|
+
retry that reuses the key — after `indeterminate`, an IPC timeout or
|
|
341
|
+
`daemon_unresponsive` — gets the recorded result of the first execution back
|
|
342
|
+
(`replayed: true`) instead of running the command again; reusing a key for a
|
|
343
|
+
*different* command is refused (`idempotency_key_reused`). The journal is
|
|
344
|
+
per agent process (last 64 results) and empty after a reconnect, so after
|
|
345
|
+
`connect` verify the side effect instead of retrying blindly.
|
|
346
|
+
|
|
347
|
+
**`run` exit codes mean something.** The child runs with
|
|
348
|
+
`$ErrorActionPreference='Stop'`, so a cmdlet that fails non-terminatingly —
|
|
349
|
+
`Add-Content` to a locked file, `Set-Content` to a bad path — exits 1 with the
|
|
350
|
+
error on stderr instead of reporting success for a write that never happened.
|
|
351
|
+
A script that wants continue-on-error sets `$ErrorActionPreference='Continue'`
|
|
352
|
+
on its first line. Do not redirect `*>` into a file the script itself writes:
|
|
353
|
+
the child holds it open and the script's own writes fail with "being used by
|
|
354
|
+
another process" — `run --wait` captures output without a file, and
|
|
355
|
+
`run --stream` + `run-poll` covers long jobs.
|
|
356
|
+
|
|
357
|
+
**Long-running commands** get the time they ask for: the transport deadline,
|
|
358
|
+
the CLI socket timeout and the watchdog all extend to cover
|
|
359
|
+
`--process-timeout`/`--timeout`. But the agent handles one command at a time,
|
|
360
|
+
so a long `run --wait` blocks every other `automate` call — past about a
|
|
361
|
+
minute, prefer `run --stream` plus `run-poll`. Streamed output is captured to
|
|
362
|
+
files on the remote side, so nothing is lost if the process exits between
|
|
363
|
+
polls; a finished process stays pollable for 10 minutes and repeat polls return
|
|
364
|
+
`exited: true` with empty chunks. (If you redirect inside the command instead,
|
|
365
|
+
note that Windows PowerShell 5.1's `>` writes UTF-16LE.)
|
|
366
|
+
|
|
367
|
+
**`daemon_not_running` vs `daemon_unresponsive`.** The first means no daemon
|
|
368
|
+
process exists — reconnect, and `<session>/daemon.log` says why it exited. The
|
|
369
|
+
second means the process is alive but did not answer a health check within
|
|
370
|
+
10s: it is busy (a long `run --wait`, a file transfer). Wait and retry; do not
|
|
371
|
+
reconnect, that discards a working session. The message includes the tail of
|
|
372
|
+
daemon.log. `connect` replaces a daemon that stays unresponsive, killing the
|
|
373
|
+
stuck one first.
|
|
374
|
+
|
|
375
|
+
**`daemon_version_mismatch`** means the daemon was started by a different
|
|
376
|
+
agent-rdp version than the CLI — it kept running across an upgrade, and is
|
|
377
|
+
still serving the old code, including the automation agent it embeds. Run
|
|
378
|
+
`agent-rdp connect ...` again: it replaces the daemon (and the SDK does the
|
|
379
|
+
same on its own). `session info` shows both versions. This is worth knowing
|
|
380
|
+
about because it is how "upgraded, but the old bug still reproduces" happens.
|
|
381
|
+
|
|
382
|
+
**Arrow-key navigation inside a panel can land on the wrong item.** Observed in
|
|
383
|
+
1C side panels: Up/Down then Enter is not reliably deterministic. Prefer
|
|
384
|
+
`automate` refs; when the panel isn't exposed to UI Automation, use two
|
|
385
|
+
independent measurements with [`click-at --confirm`](#click-at-safe-clicking-of-externally-computed-coordinates).
|
|
386
|
+
|
|
387
|
+
### Sessions and the web viewer
|
|
554
388
|
|
|
555
389
|
```bash
|
|
556
|
-
# List active sessions
|
|
557
390
|
agent-rdp session list
|
|
558
|
-
|
|
559
|
-
# Get current session info
|
|
560
391
|
agent-rdp session info
|
|
561
|
-
|
|
562
|
-
# Close a session
|
|
563
|
-
agent-rdp session close
|
|
564
|
-
|
|
565
|
-
# Use a named session
|
|
566
|
-
agent-rdp --session work connect --host work-pc.local ...
|
|
392
|
+
agent-rdp --session work connect --host work-pc.local ... # named session
|
|
567
393
|
agent-rdp --session work screenshot
|
|
568
|
-
```
|
|
569
394
|
|
|
570
|
-
|
|
395
|
+
agent-rdp wait 3000 # pause; useful right after connect, before the first input
|
|
571
396
|
|
|
572
|
-
|
|
573
|
-
agent-rdp
|
|
397
|
+
# Web viewer (needs streaming enabled at connect)
|
|
398
|
+
agent-rdp --stream-port 9224 connect --host 192.168.1.100 -u Admin -p secret
|
|
399
|
+
agent-rdp view --port 9224
|
|
574
400
|
```
|
|
575
401
|
|
|
576
|
-
|
|
402
|
+
There is no `session close` — use `disconnect`.
|
|
577
403
|
|
|
578
|
-
|
|
404
|
+
### Diagnostics and bug reports
|
|
579
405
|
|
|
580
406
|
```bash
|
|
581
|
-
#
|
|
582
|
-
agent-rdp
|
|
583
|
-
|
|
584
|
-
# Specify a different port
|
|
585
|
-
agent-rdp view --port 9224
|
|
407
|
+
agent-rdp diagnose # -> ./agent-rdp-diagnostics-<session>-<ts>.zip
|
|
408
|
+
agent-rdp diagnose --output report.zip
|
|
586
409
|
```
|
|
587
410
|
|
|
588
|
-
The
|
|
411
|
+
The zip holds `daemon.log` (+ `.prev`), `transcript.jsonl` (one redacted line
|
|
412
|
+
per request: what was asked, outcome, duration), `diagnostics/` (failure
|
|
413
|
+
captures), a current screenshot, the remote automation agent's own log, and
|
|
414
|
+
`info.json` (versions, OS, daemon state, which `AGENT_RDP_*` variables are
|
|
415
|
+
set). It is built from disk first and the daemon second, so it works when the
|
|
416
|
+
daemon is dead or unresponsive — those parts are listed as skipped. The RDP
|
|
417
|
+
password is never logged, and is blanked if it appears anyway.
|
|
589
418
|
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
419
|
+
Failure captures are automatic: when `locate` finds nothing, `click-at` refuses,
|
|
420
|
+
an `automate` command errors, or a waited `run` exits non-zero, the daemon saves
|
|
421
|
+
`diagnostics/<ts>-<kind>-<code>.png` plus a `.json` with the request, the error
|
|
422
|
+
and — for OCR misses — every line OCR *did* read. At most one capture per 5s,
|
|
423
|
+
the newest 20 kept. Set `AGENT_RDP_DIAGNOSTICS=0` to turn the transcript and
|
|
424
|
+
captures off.
|
|
594
425
|
|
|
595
|
-
## JSON
|
|
426
|
+
## JSON output
|
|
596
427
|
|
|
597
|
-
|
|
428
|
+
Add `--json` to any command:
|
|
598
429
|
|
|
599
|
-
```bash
|
|
600
|
-
agent-rdp --json screenshot --output desktop.png
|
|
601
|
-
```
|
|
602
|
-
|
|
603
|
-
**Success response:**
|
|
604
430
|
```json
|
|
605
|
-
{
|
|
606
|
-
|
|
607
|
-
"data": {
|
|
608
|
-
"type": "screenshot",
|
|
609
|
-
"path": "desktop.png",
|
|
610
|
-
"width": 1920,
|
|
611
|
-
"height": 1080
|
|
612
|
-
}
|
|
613
|
-
}
|
|
431
|
+
{ "success": true, "data": { "type": "screenshot", "path": "desktop.png", "width": 1920, "height": 1080 } }
|
|
432
|
+
{ "success": false, "error": { "code": "not_connected", "message": "Not connected to an RDP server" } }
|
|
614
433
|
```
|
|
615
434
|
|
|
616
|
-
|
|
617
|
-
```json
|
|
618
|
-
{
|
|
619
|
-
"success": false,
|
|
620
|
-
"error": {
|
|
621
|
-
"code": "not_connected",
|
|
622
|
-
"message": "Not connected to an RDP server"
|
|
623
|
-
}
|
|
624
|
-
}
|
|
625
|
-
```
|
|
626
|
-
|
|
627
|
-
## Environment Variables
|
|
435
|
+
## Environment variables
|
|
628
436
|
|
|
629
437
|
| Variable | Description |
|
|
630
438
|
|----------|-------------|
|
|
@@ -634,173 +442,109 @@ agent-rdp --json screenshot --output desktop.png
|
|
|
634
442
|
| `AGENT_RDP_PASSWORD` | RDP password |
|
|
635
443
|
| `AGENT_RDP_SESSION` | Session name (default: "default") |
|
|
636
444
|
| `AGENT_RDP_STREAM_PORT` | WebSocket streaming port (0 = disabled) |
|
|
637
|
-
| `AGENT_RDP_MODELS_DIR` |
|
|
445
|
+
| `AGENT_RDP_MODELS_DIR` | OCR models directory (set automatically by the npm wrapper; needed for standalone binary installs) |
|
|
446
|
+
| `AGENT_RDP_DIAGNOSTICS` | Set to `0` to disable the request transcript and failure captures |
|
|
638
447
|
|
|
639
448
|
## Node.js API
|
|
640
449
|
|
|
641
|
-
Use agent-rdp programmatically from Node.js/TypeScript:
|
|
642
|
-
|
|
643
450
|
```typescript
|
|
644
451
|
import { RdpSession } from 'agent-rdp';
|
|
645
452
|
|
|
646
453
|
const rdp = new RdpSession({ session: 'default' });
|
|
647
|
-
|
|
648
454
|
await rdp.connect({
|
|
649
|
-
host: '192.168.1.100',
|
|
650
|
-
|
|
651
|
-
password: 'secret',
|
|
652
|
-
width: 1280,
|
|
653
|
-
height: 800,
|
|
455
|
+
host: '192.168.1.100', username: 'Administrator', password: 'secret',
|
|
456
|
+
width: 1280, height: 800,
|
|
654
457
|
drives: [{ path: '/tmp/share', name: 'Share' }],
|
|
655
|
-
enableWinAutomation: true,
|
|
458
|
+
enableWinAutomation: true,
|
|
656
459
|
});
|
|
657
460
|
|
|
658
|
-
// Screenshot - prefer `path` so a large base64 string never
|
|
659
|
-
|
|
660
|
-
const {
|
|
661
|
-
|
|
662
|
-
// Or get raw base64 directly (e.g. for further in-process processing)
|
|
663
|
-
const { base64 } = await rdp.screenshot({ format: 'png' });
|
|
461
|
+
// Screenshot - prefer `path` so a large base64 string is never held in memory
|
|
462
|
+
const { path, width, height } = await rdp.screenshot({ format: 'png', path: 'shot.png' });
|
|
463
|
+
const { base64 } = await rdp.screenshot({ format: 'png' }); // or raw, for in-process use
|
|
664
464
|
|
|
665
|
-
//
|
|
666
|
-
await rdp.mouse.click({ x: 100, y: 200 });
|
|
667
|
-
await rdp.mouse.rightClick({ x: 100, y: 200 });
|
|
668
|
-
await rdp.mouse.doubleClick({ x: 100, y: 200 });
|
|
669
|
-
await rdp.mouse.move({ x: 150, y: 250 });
|
|
465
|
+
await rdp.mouse.click({ x: 100, y: 200 }); // also rightClick, doubleClick, move
|
|
670
466
|
await rdp.mouse.drag({ from: { x: 100, y: 100 }, to: { x: 500, y: 500 } });
|
|
671
467
|
|
|
672
|
-
// Keyboard
|
|
673
468
|
await rdp.keyboard.type({ text: 'Hello World' });
|
|
674
|
-
await rdp.keyboard.paste('Привет, мир!');
|
|
469
|
+
await rdp.keyboard.paste('Привет, мир!'); // reliable for long/non-Latin text
|
|
675
470
|
await rdp.keyboard.press({ keys: 'ctrl+c' });
|
|
676
|
-
await rdp.keyboard.
|
|
677
|
-
await rdp.keyboard.down('shift'); // Hold across other commands
|
|
678
|
-
await rdp.keyboard.up('shift');
|
|
679
|
-
|
|
680
|
-
// Scroll
|
|
681
|
-
await rdp.scroll.up(); // Default amount: 3
|
|
682
|
-
await rdp.scroll.down({ amount: 5 }); // Custom amount
|
|
683
|
-
await rdp.scroll.up({ x: 500, y: 300 }); // Scroll at position
|
|
471
|
+
await rdp.keyboard.down('shift'); await rdp.keyboard.up('shift');
|
|
684
472
|
|
|
685
|
-
//
|
|
473
|
+
await rdp.scroll.down({ amount: 5 }); // default 3; { x, y } to target a point
|
|
686
474
|
await rdp.clipboard.set({ text: 'text to copy' });
|
|
687
475
|
const text = await rdp.clipboard.get();
|
|
688
476
|
|
|
689
|
-
//
|
|
690
|
-
const matches = await rdp.locate({ text: 'Cancel' });
|
|
691
|
-
if (matches.length > 0) {
|
|
692
|
-
await rdp.mouse.click({ x: matches[0].center_x, y: matches[0].center_y });
|
|
693
|
-
}
|
|
694
|
-
|
|
695
|
-
// Get all text on screen
|
|
696
|
-
const allText = await rdp.locate({ all: true });
|
|
697
|
-
|
|
698
|
-
// Click a match directly - the coordinate never leaves the process
|
|
477
|
+
// OCR - click directly so the coordinate never leaves the process
|
|
699
478
|
await rdp.locate({ text: 'Cancel', click: 'left' });
|
|
700
|
-
|
|
701
|
-
// Exact whole-line match - won't also hit "Провести и закрыть"
|
|
702
479
|
await rdp.locate({ text: 'Провести', exact: true, click: 'left' });
|
|
480
|
+
await rdp.locate({ text: 'OK', waitMs: 10000, click: 'left' }); // block until it appears
|
|
481
|
+
const allText = await rdp.locate({ all: true });
|
|
703
482
|
|
|
704
|
-
//
|
|
705
|
-
await rdp.locate({ text: 'OK', waitMs: 10000, click: 'left' });
|
|
706
|
-
|
|
707
|
-
// Safe click of an externally-computed point (e.g. a vision-model bbox):
|
|
708
|
-
// refuses if it's ambiguously close to more than one detected label
|
|
483
|
+
// Safe click of an externally-computed point (e.g. a vision-model bbox)
|
|
709
484
|
const result = await rdp.clickAt(665, 209);
|
|
710
485
|
if (!result.clicked) console.log('Ambiguous:', result.nearby);
|
|
711
486
|
|
|
712
|
-
//
|
|
487
|
+
// File transfer (needs enableWinAutomation)
|
|
488
|
+
await rdp.files.push('./report.xlsx', 'C:\\Users\\Admin\\report.xlsx');
|
|
489
|
+
await rdp.files.pull('C:\\Users\\Admin\\export.csv', './export.csv');
|
|
490
|
+
|
|
491
|
+
// UI Automation
|
|
713
492
|
const snapshot = await rdp.automation.snapshot({ interactive: true });
|
|
714
|
-
const focused = await rdp.automation.focused();
|
|
715
|
-
await rdp.automation.click('@e5'
|
|
716
|
-
await rdp.automation.
|
|
717
|
-
await rdp.automation.
|
|
718
|
-
await rdp.automation.toggle('@e7'); // Toggle checkbox
|
|
719
|
-
await rdp.automation.expand('@e3'); // Expand menu
|
|
720
|
-
await rdp.automation.contextMenu('@e5'); // Open context menu
|
|
721
|
-
await rdp.automation.fill('#input', 'text'); // Fill text field
|
|
722
|
-
await rdp.automation.run('notepad.exe'); // Run command
|
|
493
|
+
const focused = await rdp.automation.focused();
|
|
494
|
+
await rdp.automation.click('@e5', { doubleClick: true });
|
|
495
|
+
await rdp.automation.fill('#input', 'text');
|
|
496
|
+
await rdp.automation.run('notepad.exe');
|
|
723
497
|
await rdp.automation.waitFor('#SaveButton', { timeout: 5000 });
|
|
724
|
-
|
|
725
|
-
// Window management
|
|
726
|
-
const windows = await rdp.automation.listWindows();
|
|
727
498
|
await rdp.automation.focusWindow('~*Notepad*');
|
|
728
|
-
await rdp.automation.maximizeWindow();
|
|
729
499
|
|
|
730
|
-
// Drives
|
|
731
500
|
const drives = await rdp.drives.list();
|
|
732
|
-
|
|
733
|
-
// Session info
|
|
734
501
|
const info = await rdp.getInfo();
|
|
735
|
-
|
|
736
|
-
// Disconnect
|
|
737
502
|
await rdp.disconnect();
|
|
738
503
|
```
|
|
739
504
|
|
|
740
|
-
|
|
505
|
+
The process can stay alive after `disconnect()` unless the caller cleans up —
|
|
506
|
+
call `rdp.close()` or exit explicitly.
|
|
741
507
|
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
session: 'viewer',
|
|
747
|
-
streamPort: 9224, // Enable streaming
|
|
748
|
-
});
|
|
749
|
-
|
|
750
|
-
await rdp.connect({...});
|
|
751
|
-
|
|
752
|
-
// Connect your WebSocket client to receive JPEG frames
|
|
753
|
-
const streamUrl = rdp.getStreamUrl(); // "ws://localhost:9224"
|
|
754
|
-
```
|
|
755
|
-
|
|
756
|
-
For the complete WebSocket protocol specification (message types, clipboard flow, input handling), see [WEBSOCKET.md](https://github.com/denisix/agent-rdp/blob/main/docs/WEBSOCKET.md).
|
|
508
|
+
**WebSocket streaming** for real-time capture: construct with
|
|
509
|
+
`{ streamPort: 9224 }`, then `rdp.getStreamUrl()` returns `ws://localhost:9224`.
|
|
510
|
+
Message types, clipboard flow and input handling are specified in
|
|
511
|
+
[WEBSOCKET.md](docs/WEBSOCKET.md).
|
|
757
512
|
|
|
758
513
|
## Architecture
|
|
759
514
|
|
|
760
|
-
|
|
761
|
-
|
|
762
|
-
|
|
763
|
-
|
|
764
|
-
3. **IPC** - Unix sockets (macOS/Linux) or TCP (Windows)
|
|
765
|
-
|
|
766
|
-
The daemon is automatically started on the first command and persists until explicitly closed or the session times out.
|
|
515
|
+
A daemon per session: the **CLI** parses commands and talks to a **daemon** over
|
|
516
|
+
Unix sockets (macOS/Linux) or TCP (Windows); the daemon owns the RDP connection
|
|
517
|
+
and processes commands. It starts on the first command and persists until
|
|
518
|
+
closed. See [CLAUDE.md](CLAUDE.md) for the internals.
|
|
767
519
|
|
|
768
520
|
## Limitations
|
|
769
521
|
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
773
|
-
- **UAC
|
|
774
|
-
|
|
775
|
-
|
|
776
|
-
|
|
777
|
-
|
|
778
|
-
|
|
779
|
-
|
|
780
|
-
|
|
781
|
-
|
|
782
|
-
```
|
|
783
|
-
|
|
784
|
-
This is not highly reliable (OCR can misread characters, miss text, or return imprecise coordinates), but may work for simple cases like dialog buttons.
|
|
785
|
-
|
|
786
|
-
### Screenshot Coordinate Detection
|
|
787
|
-
|
|
788
|
-
**Claude models** (in non-computer-use mode, such as Claude Code) are poor at estimating pixel coordinates from screenshots. Do not ask Claude to look at a screenshot and guess where to click - it will likely be inaccurate.
|
|
789
|
-
|
|
790
|
-
**Gemini models** are generally good at pixel coordinate estimation from images.
|
|
791
|
-
|
|
792
|
-
If you need vision-based coordinate detection with Claude, implement your own harness using Claude's [Computer Use Tool](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use) which is specifically designed for this purpose.
|
|
522
|
+
- **WebViews** — UI Automation cannot see WebView content (Start menu search,
|
|
523
|
+
Edge, Electron apps). Launch programs with `automate run` or Win+R instead of
|
|
524
|
+
clicking through menus.
|
|
525
|
+
- **UAC dialogs** run on a secure desktop and are invisible to UI Automation.
|
|
526
|
+
There is no good workaround short of the remote user acting manually.
|
|
527
|
+
- **OCR** can misread characters, miss text, or return imprecise coordinates.
|
|
528
|
+
Use it only when UI Automation can't reach an element, and verify before
|
|
529
|
+
clicking anything destructive.
|
|
530
|
+
- **Claude models in non-computer-use mode** (Claude Code included) are poor at
|
|
531
|
+
estimating pixel coordinates from screenshots — don't ask for a guess from an
|
|
532
|
+
image. Gemini models are generally good at it; for Claude, use the
|
|
533
|
+
[Computer Use Tool](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use).
|
|
793
534
|
|
|
794
535
|
## Requirements
|
|
795
536
|
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
537
|
+
Rust 1.75+. The target must offer **TLS** for RDP — agent-rdp uses `rustls`,
|
|
538
|
+
which doesn't implement TLS 1.0/1.1, so legacy hosts (e.g. Windows Server
|
|
539
|
+
2008 R2) fail with a handshake error. **Stock Windows defaults work as
|
|
540
|
+
shipped.**
|
|
799
541
|
|
|
800
|
-
|
|
542
|
+
<details>
|
|
543
|
+
<summary>RDP security layer settings</summary>
|
|
801
544
|
|
|
802
|
-
All six combinations measured against Windows Server 2022, varying only these
|
|
803
|
-
values under
|
|
545
|
+
All six combinations measured against Windows Server 2022, varying only these
|
|
546
|
+
two values under
|
|
547
|
+
`HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp`:
|
|
804
548
|
|
|
805
549
|
| `SecurityLayer` | `UserAuthentication` | Meaning | Result |
|
|
806
550
|
|---|---|---|---|
|
|
@@ -812,12 +556,12 @@ values under `HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations
|
|
|
812
556
|
| `0` (RDP) | `0` | legacy RC4 only | fails |
|
|
813
557
|
|
|
814
558
|
The rule: **agent-rdp works wherever the host offers TLS.** With NLA on that is
|
|
815
|
-
always the case,
|
|
559
|
+
always the case, since NLA forces CredSSP/TLS regardless of `SecurityLayer`.
|
|
816
560
|
Both failing rows are NLA-off hosts that refuse TLS outright.
|
|
817
561
|
|
|
818
562
|
If `connect` reports *"server only supports Standard RDP Security"*, the host is
|
|
819
|
-
in one of those
|
|
820
|
-
|
|
563
|
+
in one of those rows. Enable NLA (preferred — it also gives pre-authentication)
|
|
564
|
+
or force TLS:
|
|
821
565
|
|
|
822
566
|
```powershell
|
|
823
567
|
$k = 'HKLM:\System\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp'
|
|
@@ -826,18 +570,24 @@ Set-ItemProperty -Path $k -Name SecurityLayer -Value 2 # force TLS
|
|
|
826
570
|
```
|
|
827
571
|
|
|
828
572
|
New connections normally pick this up immediately; restart `TermService` only if
|
|
829
|
-
they don't (
|
|
573
|
+
they don't (that drops existing sessions).
|
|
830
574
|
|
|
831
575
|
Neither failing case is fixable client-side. With `SecurityLayer=1` and NLA off
|
|
832
576
|
the server returns `SSL_NOT_ALLOWED_BY_SERVER` even though the client advertises
|
|
833
|
-
`PROTOCOL_SSL` — verified by testing a build that requested SSL *only*,
|
|
834
|
-
|
|
835
|
-
|
|
836
|
-
|
|
577
|
+
`PROTOCOL_SSL` — verified by testing a build that requested SSL *only*, rejected
|
|
578
|
+
identically. Standard RDP Security (`SecurityLayer=0`, NLA off) is refused by
|
|
579
|
+
IronRDP itself, which implements no RC4 transport; it uses a well-known key
|
|
580
|
+
derivation, so credentials sent over it are recoverable in transit.
|
|
581
|
+
</details>
|
|
837
582
|
|
|
838
583
|
## Credits
|
|
839
584
|
|
|
840
|
-
Originally created by [Nick Yu](https://github.com/thisnick)
|
|
585
|
+
Originally created by [Nick Yu](https://github.com/thisnick)
|
|
586
|
+
([thisnick/agent-rdp](https://github.com/thisnick/agent-rdp)). This fork
|
|
587
|
+
([denisix/agent-rdp](https://github.com/denisix/agent-rdp), published as
|
|
588
|
+
[`@denisixnpm/agent-rdp`](https://www.npmjs.com/package/@denisixnpm/agent-rdp))
|
|
589
|
+
is maintained independently; see
|
|
590
|
+
[CHANGELOG.md](packages/agent-rdp/CHANGELOG.md) for what's changed.
|
|
841
591
|
|
|
842
592
|
## License
|
|
843
593
|
|