@denisixnpm/agent-rdp 0.7.10 → 0.7.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/README.md +285 -585
- package/dist/generated/ErrorCode.d.ts +1 -1
- package/dist/generated/ErrorCode.d.ts.map +1 -1
- package/package.json +7 -7
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,12 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.7.11](https://github.com/denisix/agent-rdp/compare/agent-rdp-v0.7.10...agent-rdp-v0.7.11) (2026-09-02)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Features
|
|
7
|
+
|
|
8
|
+
* fix file push framing, run prelude, stream polling, clipboard CRLF, and daemon health verdicts ([1e470fd](https://github.com/denisix/agent-rdp/commit/1e470fdff55839237e5f8684b594520201039d8a))
|
|
9
|
+
|
|
3
10
|
## [0.7.10](https://github.com/denisix/agent-rdp/compare/agent-rdp-v0.7.9...agent-rdp-v0.7.10) (2026-09-01)
|
|
4
11
|
|
|
5
12
|
|
package/README.md
CHANGED
|
@@ -8,121 +8,76 @@
|
|
|
8
8
|
|
|
9
9
|
A CLI tool for AI agents to control Windows Remote Desktop sessions, built on [IronRDP](https://github.com/Devolutions/IronRDP).
|
|
10
10
|
|
|
11
|
+
Screenshots, mouse/keyboard input, clipboard sync, drive mapping, verified file
|
|
12
|
+
transfer, Windows UI Automation, OCR text location, and JSON output for every
|
|
13
|
+
command — across named sessions with automatic daemon lifecycle.
|
|
14
|
+
|
|
11
15
|
## Demo
|
|
12
16
|
|
|
13
17
|
Claude Code automating SQLite database and table creation via RDP:
|
|
14
18
|
|
|
15
19
|
https://github.com/user-attachments/assets/91892b39-4edb-412b-b265-55ccd75d7421
|
|
16
20
|
|
|
17
|
-
## Features
|
|
18
|
-
|
|
19
|
-
- **Connect to RDP servers** - Full RDP protocol support with TLS and CredSSP authentication
|
|
20
|
-
- **Take screenshots** - Capture the remote desktop as PNG or JPEG
|
|
21
|
-
- **Mouse control** - Click, double-click, right-click, drag, scroll
|
|
22
|
-
- **Keyboard input** - Type text, press key combinations (Ctrl+C, Alt+Tab, etc.)
|
|
23
|
-
- **Clipboard sync** - Copy/paste text between local machine and remote Windows
|
|
24
|
-
- **Drive mapping** - Map local directories as network drives on the remote machine
|
|
25
|
-
- **UI Automation** - Interact with Windows applications via accessibility API (click, select, toggle, expand)
|
|
26
|
-
- **OCR text location** - Find text on screen using OCR when UI Automation isn't available
|
|
27
|
-
- **JSON output** - Structured output for AI agent consumption
|
|
28
|
-
- **Session management** - Multiple named sessions with automatic daemon lifecycle
|
|
29
|
-
|
|
30
21
|
## Installation
|
|
31
22
|
|
|
32
|
-
### From npm
|
|
33
|
-
|
|
34
23
|
```bash
|
|
35
|
-
npm install -g @denisixnpm/agent-rdp
|
|
24
|
+
npm install -g @denisixnpm/agent-rdp # CLI
|
|
25
|
+
npx add-skill https://github.com/denisix/agent-rdp # as a Claude Code skill
|
|
36
26
|
```
|
|
37
27
|
|
|
38
|
-
|
|
28
|
+
<details>
|
|
29
|
+
<summary>From a GitHub release (standalone binary)</summary>
|
|
39
30
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
### From a GitHub release
|
|
45
|
-
|
|
46
|
-
Each release attaches a standalone binary per platform, plus the OCR models as
|
|
47
|
-
`agent-rdp-models.tar.gz` / `agent-rdp-models.zip`. The models are
|
|
48
|
-
architecture-independent, so they ship once rather than inside every archive.
|
|
49
|
-
|
|
50
|
-
The binary alone covers everything except `locate`; that command needs the
|
|
51
|
-
models, so extract them too and point `AGENT_RDP_MODELS_DIR` at them.
|
|
52
|
-
|
|
53
|
-
**macOS / Linux**
|
|
31
|
+
Each release attaches a binary per platform plus the OCR models as
|
|
32
|
+
`agent-rdp-models.tar.gz` / `.zip`. The models are architecture-independent, so
|
|
33
|
+
they ship once. The binary alone covers everything except `locate`, which needs
|
|
34
|
+
the models — extract them and point `AGENT_RDP_MODELS_DIR` at them.
|
|
54
35
|
|
|
55
36
|
```bash
|
|
56
37
|
tar -xzf agent-rdp-linux-x64.tar.gz -C ~/.local/bin # or agent-rdp-darwin-arm64.tar.gz
|
|
57
38
|
chmod +x ~/.local/bin/agent-rdp
|
|
58
39
|
mkdir -p ~/.local/share/agent-rdp/models
|
|
59
40
|
tar -xzf agent-rdp-models.tar.gz -C ~/.local/share/agent-rdp/models
|
|
60
|
-
export AGENT_RDP_MODELS_DIR="$HOME/.local/share/agent-rdp/models" # add to your shell rc
|
|
41
|
+
export AGENT_RDP_MODELS_DIR="$HOME/.local/share/agent-rdp/models" # add to your shell rc
|
|
61
42
|
```
|
|
62
43
|
|
|
63
|
-
On macOS the release binaries are unsigned, so Gatekeeper quarantines anything
|
|
64
|
-
downloaded from a browser. If you get *"cannot be opened because the developer
|
|
65
|
-
cannot be verified"*, clear the flag:
|
|
66
|
-
|
|
67
|
-
```bash
|
|
68
|
-
xattr -d com.apple.quarantine ~/.local/bin/agent-rdp
|
|
69
|
-
```
|
|
70
|
-
|
|
71
|
-
**Windows (PowerShell)**
|
|
72
|
-
|
|
73
44
|
```powershell
|
|
74
45
|
Expand-Archive agent-rdp-win32-x64.zip -DestinationPath "$env:LOCALAPPDATA\agent-rdp"
|
|
75
46
|
Expand-Archive agent-rdp-models.zip -DestinationPath "$env:LOCALAPPDATA\agent-rdp\models"
|
|
76
|
-
|
|
77
|
-
setx AGENT_RDP_MODELS_DIR "$env:LOCALAPPDATA\agent-rdp\models"
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
Installing from npm needs none of this — the models ship inside
|
|
81
|
-
`@denisixnpm/agent-rdp` and are located automatically.
|
|
82
|
-
|
|
83
|
-
### From source
|
|
84
|
-
|
|
85
|
-
```bash
|
|
86
|
-
git clone https://github.com/denisix/agent-rdp
|
|
87
|
-
cd agent-rdp
|
|
88
|
-
bun install
|
|
89
|
-
bun run build # Build native binary
|
|
90
|
-
bun run build:ts # Build TypeScript
|
|
47
|
+
setx AGENT_RDP_MODELS_DIR "$env:LOCALAPPDATA\agent-rdp\models" # restart the terminal after
|
|
91
48
|
```
|
|
92
49
|
|
|
93
|
-
|
|
50
|
+
macOS release binaries are unsigned. If Gatekeeper blocks it with *"cannot be
|
|
51
|
+
opened because the developer cannot be verified"*:
|
|
52
|
+
`xattr -d com.apple.quarantine ~/.local/bin/agent-rdp`
|
|
94
53
|
|
|
95
|
-
|
|
54
|
+
Installing from npm needs none of this — the models ship in the package.
|
|
55
|
+
</details>
|
|
96
56
|
|
|
97
|
-
|
|
57
|
+
<details>
|
|
58
|
+
<summary>From source</summary>
|
|
98
59
|
|
|
99
60
|
```bash
|
|
100
|
-
|
|
61
|
+
git clone https://github.com/denisix/agent-rdp && cd agent-rdp
|
|
62
|
+
bun install
|
|
63
|
+
bun run build # native binary
|
|
64
|
+
bun run build:ts # TypeScript
|
|
101
65
|
```
|
|
66
|
+
</details>
|
|
102
67
|
|
|
103
|
-
|
|
104
|
-
`@denisixnpm/agent-rdp` on first use if it isn't already on PATH.
|
|
105
|
-
|
|
106
|
-
Or install manually:
|
|
107
|
-
|
|
108
|
-
```bash
|
|
109
|
-
mkdir -p .claude/skills/agent-rdp
|
|
110
|
-
curl -o .claude/skills/agent-rdp/SKILL.md \
|
|
111
|
-
https://raw.githubusercontent.com/denisix/agent-rdp/main/skills/agent-rdp/SKILL.md
|
|
112
|
-
```
|
|
68
|
+
## Using with AI coding agents
|
|
113
69
|
|
|
114
|
-
|
|
70
|
+
**Claude Code** — `npx add-skill https://github.com/denisix/agent-rdp` installs
|
|
71
|
+
the [SKILL.md](skills/agent-rdp/SKILL.md) workflow so Claude knows the commands,
|
|
72
|
+
flags and gotchas without you explaining them. It installs the CLI on first use
|
|
73
|
+
if it isn't on PATH. Then ask in plain language:
|
|
115
74
|
|
|
116
75
|
```
|
|
117
76
|
Connect to 192.168.1.100 as Administrator (password: secret), open Notepad,
|
|
118
77
|
type "hello from Claude", and take a screenshot.
|
|
119
78
|
```
|
|
120
79
|
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
### Codex
|
|
124
|
-
|
|
125
|
-
Codex doesn't have a skill-install mechanism, but it reads `AGENTS.md` for project instructions. Point it at this tool by adding a section to your `AGENTS.md`:
|
|
80
|
+
**Codex** reads `AGENTS.md` instead. Add a section pointing at the tool:
|
|
126
81
|
|
|
127
82
|
```bash
|
|
128
83
|
cat >> AGENTS.md <<'EOF'
|
|
@@ -135,414 +90,235 @@ https://github.com/denisix/agent-rdp for the full command reference.
|
|
|
135
90
|
EOF
|
|
136
91
|
```
|
|
137
92
|
|
|
138
|
-
Then prompt Codex the same way:
|
|
139
|
-
|
|
140
|
-
```
|
|
141
|
-
codex "Connect to the Windows VM at 192.168.1.100 (user Administrator, password
|
|
142
|
-
secret) using agent-rdp, open the Run dialog, launch calc.exe, and confirm it's
|
|
143
|
-
open with a screenshot."
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
Codex will call `agent-rdp` as a regular shell command, same as any other CLI tool.
|
|
147
|
-
|
|
148
93
|
## Usage
|
|
149
94
|
|
|
150
|
-
### Connect
|
|
95
|
+
### Connect
|
|
151
96
|
|
|
152
97
|
```bash
|
|
153
|
-
# Using command line (password visible in process list - not recommended)
|
|
154
98
|
agent-rdp connect --host 192.168.1.100 --username Administrator --password 'secret'
|
|
155
99
|
|
|
156
|
-
#
|
|
157
|
-
export AGENT_RDP_USERNAME=Administrator
|
|
158
|
-
export AGENT_RDP_PASSWORD=secret
|
|
100
|
+
# Environment variables (recommended - keeps the password out of the process list)
|
|
101
|
+
export AGENT_RDP_USERNAME=Administrator AGENT_RDP_PASSWORD=secret
|
|
159
102
|
agent-rdp connect --host 192.168.1.100
|
|
160
103
|
|
|
161
|
-
#
|
|
162
|
-
echo 'secret' | agent-rdp connect --host 192.168.1.100
|
|
104
|
+
# stdin (most secure)
|
|
105
|
+
echo 'secret' | agent-rdp connect --host 192.168.1.100 -u Administrator --password-stdin
|
|
106
|
+
|
|
107
|
+
agent-rdp disconnect
|
|
163
108
|
```
|
|
164
109
|
|
|
165
|
-
###
|
|
110
|
+
### Screenshot
|
|
166
111
|
|
|
167
112
|
```bash
|
|
168
|
-
|
|
169
|
-
agent-rdp screenshot --output desktop.png
|
|
113
|
+
agent-rdp screenshot --output desktop.png # default ./screenshot.png
|
|
114
|
+
agent-rdp --json screenshot --output desktop.png # metadata; image always goes to disk
|
|
115
|
+
agent-rdp screenshot --region 100,380,600,30 -o row.png # crop; reports the offset back
|
|
116
|
+
```
|
|
170
117
|
|
|
171
|
-
|
|
172
|
-
|
|
118
|
+
A screenshot is the last frame the server painted, not a live poll. Each one
|
|
119
|
+
reports `frame_age_ms` (time since the server last sent anything), and `--json`
|
|
120
|
+
adds `frame_seq` and `frame_hash` — two screenshots with the same value are
|
|
121
|
+
guaranteed pixel-identical, which is how you confirm a frame actually changed
|
|
122
|
+
after an action rather than hashing the saved file yourself. A large
|
|
123
|
+
`frame_age_ms` usually just means an idle desktop, since RDP servers send
|
|
124
|
+
nothing when nothing changes; a genuinely dead connection is detected within
|
|
125
|
+
seconds by TCP keepalive and reported as disconnected.
|
|
173
126
|
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
```
|
|
127
|
+
> The CLI has no `screenshot --base64`. In Node, `rdp.screenshot({ path })`
|
|
128
|
+
> writes to disk and returns `{ path, width, height }` — prefer it over the
|
|
129
|
+
> base64-returning form when the caller doesn't need the bytes, since echoing a
|
|
130
|
+
> base64 image into an LLM context is expensive.
|
|
179
131
|
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
Every screenshot also reports `frame_age_ms` (`--json`) / a printed warning
|
|
183
|
-
(human mode, when large): milliseconds since the RDP server last sent any
|
|
184
|
-
data. An idle-but-alive desktop can legitimately sit at a large age — RDP
|
|
185
|
-
servers send nothing when nothing changes — but combined with `agent-rdp
|
|
186
|
-
session info`'s `last_frame_age_ms`, a value that never resets on an
|
|
187
|
-
otherwise-idle session is the signal to check the connection rather than
|
|
188
|
-
trust the frame. The daemon also enables TCP keepalive on the RDP socket, so
|
|
189
|
-
a genuinely dead connection (cable pull, black-holed network path) is now
|
|
190
|
-
detected and disconnected within seconds instead of the OS's own multi-minute
|
|
191
|
-
retransmission timeout — previously `screenshot` could keep returning a
|
|
192
|
-
stale, byte-identical frame under a fresh filename for the entire time the
|
|
193
|
-
transport was silently dead.
|
|
194
|
-
|
|
195
|
-
`--json` output (and the Node SDK's `frameSeq`/`frameHash`) also carries a
|
|
196
|
-
`frame_seq` (framebuffer generation counter) and `frame_hash` (FNV-1a hash of
|
|
197
|
-
the captured pixels). Two screenshots with the same seq or hash are
|
|
198
|
-
guaranteed pixel-identical — proving that directly, instead of hashing the
|
|
199
|
-
saved file yourself or reading a clock rendered on screen, is the reliable
|
|
200
|
-
way to confirm a frame actually changed after an action that should have
|
|
201
|
-
repainted it.
|
|
202
|
-
|
|
203
|
-
### Getting Coordinates Right
|
|
132
|
+
### Getting coordinates right
|
|
204
133
|
|
|
205
134
|
**Never estimate a coordinate by looking at a screenshot.** Screenshot pixels,
|
|
206
|
-
OCR boxes and click coordinates
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
lands on the wrong row, the wrong cell, or a search box.
|
|
135
|
+
OCR boxes and click coordinates share one space, so a coordinate the tool
|
|
136
|
+
returns is exact — one guessed from an image is not. Images get downscaled on
|
|
137
|
+
their way into a vision model, and a click 30px off lands on the wrong row.
|
|
210
138
|
|
|
211
139
|
Ask for the target by name and let the tool find it:
|
|
212
140
|
|
|
213
141
|
```bash
|
|
214
|
-
|
|
215
|
-
agent-rdp locate "Добавить" --click
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
agent-rdp locate "Добавить" --click --index 1
|
|
219
|
-
|
|
220
|
-
# Click a button by accessibility selector (preferred when UIA can see it)
|
|
221
|
-
agent-rdp automate click "#SaveButton"
|
|
222
|
-
```
|
|
223
|
-
|
|
224
|
-
To verify a value in place, read the region rather than cropping an image
|
|
225
|
-
yourself — coordinates still come back in full-screen space:
|
|
226
|
-
|
|
227
|
-
```bash
|
|
228
|
-
# Check one table row
|
|
229
|
-
agent-rdp locate --all --region 100,380,600,30
|
|
230
|
-
agent-rdp screenshot --region 100,380,600,30 -o row.png
|
|
231
|
-
|
|
232
|
-
# Confirm which field the keyboard is actually in before typing
|
|
233
|
-
agent-rdp automate focused
|
|
142
|
+
agent-rdp locate "Добавить" --click # coordinate never passes through your hands
|
|
143
|
+
agent-rdp locate "Добавить" --click --index 1 # several matches: choosing is explicit
|
|
144
|
+
agent-rdp automate click "#SaveButton" # preferred when UI Automation can see it
|
|
145
|
+
agent-rdp automate focused # confirm which field has focus before typing
|
|
234
146
|
```
|
|
235
147
|
|
|
236
148
|
`agent-rdp mouse click X Y` remains available for coordinates you got from
|
|
237
149
|
`locate`, `automate get`, or a deliberate calculation.
|
|
238
150
|
|
|
239
|
-
### Mouse
|
|
151
|
+
### Mouse and keyboard
|
|
240
152
|
|
|
241
153
|
```bash
|
|
242
|
-
# Click at position
|
|
243
154
|
agent-rdp mouse click 500 300
|
|
244
|
-
|
|
245
|
-
# Right-click
|
|
246
155
|
agent-rdp mouse right-click 500 300
|
|
247
|
-
|
|
248
|
-
# Double-click
|
|
249
156
|
agent-rdp mouse double-click 500 300
|
|
250
|
-
|
|
251
|
-
# Move cursor
|
|
252
157
|
agent-rdp mouse move 100 200
|
|
253
|
-
|
|
254
|
-
# Drag from (100,100) to (500,500)
|
|
255
158
|
agent-rdp mouse drag 100 100 500 500
|
|
256
|
-
```
|
|
257
|
-
|
|
258
|
-
### Keyboard Operations
|
|
259
159
|
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
agent-rdp keyboard
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
# command. More reliable than `type` - it cannot lose individual keystrokes,
|
|
266
|
-
# and there's no gap between setting the clipboard and pasting for focus to
|
|
267
|
-
# move in.
|
|
268
|
-
agent-rdp keyboard paste "Привет, мир! A very long string types can drop pieces of."
|
|
269
|
-
|
|
270
|
-
# Press key combinations
|
|
271
|
-
agent-rdp keyboard press "ctrl+c"
|
|
272
|
-
agent-rdp keyboard press "alt+tab"
|
|
273
|
-
agent-rdp keyboard press "ctrl+shift+esc"
|
|
274
|
-
|
|
275
|
-
# Press single keys (use press command)
|
|
276
|
-
agent-rdp keyboard press enter
|
|
277
|
-
agent-rdp keyboard press escape
|
|
278
|
-
agent-rdp keyboard press f5
|
|
279
|
-
|
|
280
|
-
# Hold a key across other commands (shift-click, hold-and-drag, ...)
|
|
281
|
-
agent-rdp keyboard down shift
|
|
282
|
-
agent-rdp mouse click 200 300
|
|
160
|
+
agent-rdp keyboard type "Hello, World!" # Unicode, batched into one round-trip
|
|
161
|
+
agent-rdp keyboard type "text" --delay 20 # pace it if the remote app drops fast input
|
|
162
|
+
agent-rdp keyboard paste "Привет, мир!" # clipboard + Ctrl+V as one command
|
|
163
|
+
agent-rdp keyboard press "ctrl+c" # also alt+tab, enter, escape, f5, win+r
|
|
164
|
+
agent-rdp keyboard down shift # hold across other commands
|
|
283
165
|
agent-rdp keyboard up shift
|
|
284
166
|
```
|
|
285
167
|
|
|
286
|
-
|
|
168
|
+
Prefer `keyboard paste` for long or non-Latin text: it cannot lose individual
|
|
169
|
+
keystrokes, and leaves no gap for focus to move between setting the clipboard
|
|
170
|
+
and pasting.
|
|
287
171
|
|
|
288
|
-
|
|
289
|
-
in — pass `--at` to target a specific window or control.
|
|
172
|
+
### Scroll
|
|
290
173
|
|
|
291
|
-
Amount is
|
|
174
|
+
Amount is positional (not `--amount`), and the default point is the **screen
|
|
175
|
+
center** rather than whatever pane you're working in — use `--at` to target one.
|
|
292
176
|
|
|
293
177
|
```bash
|
|
294
178
|
agent-rdp scroll up 3
|
|
295
|
-
agent-rdp scroll down 5
|
|
296
|
-
agent-rdp scroll left
|
|
297
|
-
agent-rdp scroll right
|
|
298
179
|
agent-rdp scroll down 5 --at 600 400
|
|
180
|
+
agent-rdp scroll left # also: right
|
|
299
181
|
```
|
|
300
182
|
|
|
301
|
-
###
|
|
302
|
-
|
|
303
|
-
Find text on screen using OCR (powered by [ocrs](https://github.com/robertknight/ocrs)). Useful when UI Automation can't access certain elements (WebView content, some dialogs).
|
|
183
|
+
### Clipboard and drive mapping
|
|
304
184
|
|
|
305
185
|
```bash
|
|
306
|
-
|
|
307
|
-
agent-rdp
|
|
308
|
-
|
|
309
|
-
# Pattern matching (glob-style)
|
|
310
|
-
agent-rdp locate "Save*" --pattern
|
|
311
|
-
|
|
312
|
-
# Exact whole-line matching. Default mode is substring containment, so
|
|
313
|
-
# "Провести" also matches a button reading "Провести и закрыть" - two matches,
|
|
314
|
-
# and --click refuses to guess between them. --exact matches only the line
|
|
315
|
-
# that reads exactly "Провести".
|
|
316
|
-
agent-rdp locate "Провести" --exact --click
|
|
317
|
-
|
|
318
|
-
# Get all text on screen
|
|
319
|
-
agent-rdp locate --all
|
|
320
|
-
|
|
321
|
-
# JSON output
|
|
322
|
-
agent-rdp locate "OK" --json
|
|
323
|
-
|
|
324
|
-
# Click the match directly (see "Getting Coordinates Right" above)
|
|
325
|
-
agent-rdp locate "Cancel" --click
|
|
326
|
-
agent-rdp locate "Cancel" --double-click
|
|
327
|
-
agent-rdp locate "Cancel" --right-click
|
|
328
|
-
|
|
329
|
-
# Search only part of the screen. Results stay in full-screen coordinates,
|
|
330
|
-
# so --region composes with --click. A tight region also reads small text
|
|
331
|
-
# more reliably than a full-screen pass.
|
|
332
|
-
agent-rdp locate --all --region 100,380,600,30
|
|
333
|
-
|
|
334
|
-
# Block until text appears (e.g. a dialog finishing its animation), instead
|
|
335
|
-
# of polling `locate` in a loop from the outside. Composes with --click.
|
|
336
|
-
agent-rdp locate "OK" --wait 10000 --click
|
|
337
|
-
|
|
338
|
-
# Constrain matches to those near a distinctive anchor label - useful when
|
|
339
|
-
# the same text repeats on screen (a column header in every row, a label
|
|
340
|
-
# that also appears in a tooltip). Anchor is matched by substring; if it
|
|
341
|
-
# isn't found at all, this returns zero matches rather than an error.
|
|
342
|
-
agent-rdp locate "Отменить" --near "Заказ №001" --click
|
|
343
|
-
```
|
|
344
|
-
|
|
345
|
-
Returns text lines with coordinates for clicking:
|
|
346
|
-
```
|
|
347
|
-
Found 1 line(s) containing 'Cancel':
|
|
348
|
-
'Cancel Button' at (650, 420) size 80x14 - center: (690, 427)
|
|
186
|
+
agent-rdp clipboard set "Hello from CLI"
|
|
187
|
+
agent-rdp clipboard set --file ./script.ps1 # or `--file -` to read stdin
|
|
188
|
+
agent-rdp clipboard get
|
|
349
189
|
|
|
350
|
-
|
|
190
|
+
# Drives must be mapped at connect time; multiple --drive flags are allowed
|
|
191
|
+
agent-rdp connect --host 192.168.1.100 -u Administrator -p secret \
|
|
192
|
+
--drive /home/user/documents:Documents --drive /tmp/shared:Shared
|
|
193
|
+
agent-rdp drive list
|
|
351
194
|
```
|
|
352
195
|
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
196
|
+
Mapped drives appear on the remote as network locations. Clipboard text is
|
|
197
|
+
sent with CRLF line endings, so multi-line content survives
|
|
198
|
+
`Get-Clipboard | Set-Content`; `clipboard get` returns Windows text with its
|
|
199
|
+
CRLF intact.
|
|
357
200
|
|
|
358
|
-
**
|
|
359
|
-
|
|
360
|
-
the
|
|
361
|
-
|
|
362
|
-
field) or a second independent read before acting on the amount.
|
|
201
|
+
> **Do not access `\\TSCLIENT\...` from `automate run`.** Drive redirection is
|
|
202
|
+
> serviced by the same task that carries the automation channel, so reading the
|
|
203
|
+
> share from inside the agent deadlocks — the command never returns and the
|
|
204
|
+
> session stops responding until you reconnect. Use `file push`/`file pull`.
|
|
363
205
|
|
|
364
|
-
###
|
|
206
|
+
### File transfer
|
|
365
207
|
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
isn't ambiguously close to more than one detected text region:
|
|
208
|
+
Requires `--enable-win-automation`. Copies in chunks over the automation
|
|
209
|
+
channel, with a SHA-256 computed independently on each end, so a truncated or
|
|
210
|
+
corrupted transfer fails loudly instead of leaving a plausible-looking file.
|
|
370
211
|
|
|
371
212
|
```bash
|
|
372
|
-
|
|
373
|
-
agent-rdp
|
|
374
|
-
|
|
375
|
-
# Tune the detection window and gap
|
|
376
|
-
agent-rdp click-at 665 209 --window 400x160 --min-gap 20
|
|
377
|
-
|
|
378
|
-
# Variants
|
|
379
|
-
agent-rdp click-at 665 209 --double-click
|
|
380
|
-
agent-rdp click-at 665 209 --right-click
|
|
381
|
-
|
|
382
|
-
# Cross-check: a second independent measurement for the same target (e.g. a
|
|
383
|
-
# vision model queried twice). Clicks the midpoint if the two agree within
|
|
384
|
-
# --max-divergence (default 40px); refuses if they don't. Formalizes the
|
|
385
|
-
# "click the intersection of two measurements" technique that catches a
|
|
386
|
-
# single call's misreads.
|
|
387
|
-
agent-rdp click-at 665 209 --confirm 670,212
|
|
388
|
-
agent-rdp click-at 665 209 --confirm 670,212 --max-divergence 20
|
|
213
|
+
agent-rdp file push ./report.xlsx "C:\\Users\\Admin\\report.xlsx"
|
|
214
|
+
agent-rdp file pull "C:\\Users\\Admin\\export.csv" ./export.csv
|
|
389
215
|
```
|
|
390
216
|
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
refusal it lists the nearby regions and exits non-zero; nothing is clicked.
|
|
395
|
-
|
|
396
|
-
### Clipboard
|
|
217
|
+
Transfers are byte-exact, which also makes this the reliable way to place a
|
|
218
|
+
script with non-ASCII content on the remote — writing one through the clipboard
|
|
219
|
+
or `Add-Content` re-encodes it and mangles anything outside ASCII. Limit: 128MB.
|
|
397
220
|
|
|
398
|
-
|
|
399
|
-
# Set clipboard text (available when you paste on Windows)
|
|
400
|
-
agent-rdp clipboard set "Hello from CLI"
|
|
221
|
+
### Locate (OCR)
|
|
401
222
|
|
|
402
|
-
|
|
403
|
-
|
|
223
|
+
Finds text on screen with [ocrs](https://github.com/robertknight/ocrs). Use it
|
|
224
|
+
when UI Automation can't reach an element (WebView content, some dialogs).
|
|
404
225
|
|
|
405
|
-
|
|
406
|
-
agent-rdp
|
|
226
|
+
```bash
|
|
227
|
+
agent-rdp locate "Cancel" # substring match, returns coordinates
|
|
228
|
+
agent-rdp locate "Save*" --pattern # glob match
|
|
229
|
+
agent-rdp locate "Провести" --exact # whole-line match
|
|
230
|
+
agent-rdp locate --all # all text on screen
|
|
231
|
+
agent-rdp locate "Cancel" --click # also --double-click, --right-click
|
|
232
|
+
agent-rdp locate "OK" --wait 10000 --click # block until it appears, then click
|
|
233
|
+
agent-rdp locate --all --region 100,380,600,30 # coords stay full-screen
|
|
234
|
+
agent-rdp locate "Отменить" --near "Заказ №001" --click # anchor to a nearby label
|
|
407
235
|
```
|
|
408
236
|
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
Map local directories as network drives on the remote Windows machine. Drives must be mapped at connect time. Multiple drives can be specified.
|
|
412
|
-
|
|
413
|
-
```bash
|
|
414
|
-
# Map local directories during connection
|
|
415
|
-
agent-rdp connect --host 192.168.1.100 -u Administrator -p secret \
|
|
416
|
-
--drive /home/user/documents:Documents \
|
|
417
|
-
--drive /tmp/shared:Shared
|
|
237
|
+
Output carries coordinates ready to click:
|
|
418
238
|
|
|
419
|
-
|
|
420
|
-
|
|
239
|
+
```
|
|
240
|
+
Found 1 line(s) containing 'Cancel':
|
|
241
|
+
'Cancel Button' at (650, 420) size 80x14 - center: (690, 427)
|
|
421
242
|
```
|
|
422
243
|
|
|
423
|
-
|
|
244
|
+
Clicking is deliberately strict: no match, or several matches without
|
|
245
|
+
`--index`, is an error rather than a guess. Default matching is substring
|
|
246
|
+
containment, so `"Провести"` also matches `"Провести и закрыть"` — use
|
|
247
|
+
`--exact` to avoid the ambiguity, `--near` when the same text genuinely repeats
|
|
248
|
+
across rows, or narrow with `--region`.
|
|
424
249
|
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
> below to move files instead.
|
|
250
|
+
**Numbers with thousands separators can lose their leading digit group** — OCR
|
|
251
|
+
has been observed reading `1 250,00` as `2250,00`. For monetary values, crop
|
|
252
|
+
tight with `--region` and verify with `automate get` or a second independent
|
|
253
|
+
read before acting on the amount.
|
|
430
254
|
|
|
431
|
-
###
|
|
255
|
+
### Click-at (safe clicking of externally-computed coordinates)
|
|
432
256
|
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
257
|
+
When the click point comes from outside agent-rdp — a vision model reading a
|
|
258
|
+
screenshot, a manual crop — `locate --click` can't help and a raw `mouse click`
|
|
259
|
+
has no safety net. `click-at` clicks only if the point isn't ambiguously close
|
|
260
|
+
to more than one detected text region:
|
|
437
261
|
|
|
438
262
|
```bash
|
|
439
|
-
agent-rdp
|
|
440
|
-
agent-rdp
|
|
263
|
+
agent-rdp click-at 665 209
|
|
264
|
+
agent-rdp click-at 665 209 --window 400x160 --min-gap 20 # tune detection window/gap
|
|
265
|
+
agent-rdp click-at 665 209 --double-click # also --right-click
|
|
266
|
+
|
|
267
|
+
# Cross-check two independent measurements: clicks their midpoint if they agree
|
|
268
|
+
# within --max-divergence (default 40px), refuses if they don't.
|
|
269
|
+
agent-rdp click-at 665 209 --confirm 670,212 --max-divergence 20
|
|
441
270
|
```
|
|
442
271
|
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
272
|
+
The check uses OCR *detection* only (bounding boxes, script-agnostic), not
|
|
273
|
+
recognition, so it works even for text OCR can't read — custom-rendered
|
|
274
|
+
Cyrillic UIs where both UI Automation and `locate` fail. On refusal it lists
|
|
275
|
+
the nearby regions and exits non-zero; nothing is clicked.
|
|
447
276
|
|
|
448
277
|
### UI Automation
|
|
449
278
|
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
279
|
+
Drives Windows applications through the UI Automation API using native patterns
|
|
280
|
+
(Invoke, SelectionItem, Toggle, ExpandCollapse). A PowerShell agent is injected
|
|
281
|
+
into the remote session and speaks to the daemon over a Dynamic Virtual Channel.
|
|
282
|
+
Full protocol details in [AUTOMATION.md](docs/AUTOMATION.md).
|
|
453
283
|
|
|
454
284
|
```bash
|
|
455
|
-
# Connect with automation enabled
|
|
456
285
|
agent-rdp connect --host 192.168.1.100 -u Admin -p secret --enable-win-automation
|
|
457
286
|
|
|
458
|
-
|
|
459
|
-
agent-rdp automate snapshot
|
|
460
|
-
|
|
461
|
-
#
|
|
462
|
-
|
|
463
|
-
agent-rdp automate
|
|
464
|
-
agent-rdp automate
|
|
465
|
-
agent-rdp automate
|
|
466
|
-
agent-rdp automate
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
# Use this after Tab/Enter to confirm where your keystrokes will go, instead
|
|
470
|
-
# of typing into a cell you assume is focused.
|
|
471
|
-
agent-rdp automate focused
|
|
472
|
-
# edit 'Количество' = "5,000" at (110, 391) 80x18 [focusable]
|
|
473
|
-
|
|
474
|
-
# Pattern-based element operations (refs use @eN format)
|
|
475
|
-
agent-rdp automate click "#SaveButton" # Click button
|
|
476
|
-
agent-rdp automate click "@e5" # Click by ref number from snapshot
|
|
477
|
-
agent-rdp automate click "@e5" -d # Double-click (for file list items)
|
|
478
|
-
agent-rdp automate select "@e10" # Select item (SelectionItemPattern)
|
|
479
|
-
agent-rdp automate toggle "@e7" # Toggle checkbox (TogglePattern)
|
|
480
|
-
agent-rdp automate expand "@e3" # Expand menu (ExpandCollapsePattern)
|
|
481
|
-
agent-rdp automate context-menu "@e5" # Open context menu (Shift+F10)
|
|
482
|
-
|
|
483
|
-
# Fill text fields
|
|
287
|
+
agent-rdp automate snapshot # full tree (refs always included)
|
|
288
|
+
agent-rdp automate snapshot -i -c -d 3 # interactive only, compact, depth 3
|
|
289
|
+
agent-rdp automate snapshot -s "~*Notepad*" # scope to a window
|
|
290
|
+
agent-rdp automate focused # what has keyboard focus right now
|
|
291
|
+
|
|
292
|
+
agent-rdp automate click "@e5" # also: "#SaveButton", ".Edit", "~*wild*"
|
|
293
|
+
agent-rdp automate click "@e5" -d # double-click
|
|
294
|
+
agent-rdp automate select "@e10" --item "Option 1"
|
|
295
|
+
agent-rdp automate toggle "@e7" --state on
|
|
296
|
+
agent-rdp automate expand "@e3" # also: collapse, context-menu, focus, clear
|
|
297
|
+
agent-rdp automate get "@e2" # includes Value: for text/multiline edits
|
|
484
298
|
agent-rdp automate fill ".Edit" "Hello World"
|
|
299
|
+
agent-rdp automate scroll "@e4" --direction down --amount 3
|
|
300
|
+
agent-rdp automate wait-for "#SaveButton" --timeout 5000 --state visible
|
|
485
301
|
|
|
486
|
-
# Window operations
|
|
487
302
|
agent-rdp automate window list
|
|
488
303
|
agent-rdp automate window focus "~*Notepad*"
|
|
304
|
+
agent-rdp automate window maximize|minimize|restore|close
|
|
489
305
|
|
|
490
|
-
|
|
491
|
-
agent-rdp automate run "
|
|
492
|
-
agent-rdp automate run "
|
|
493
|
-
agent-rdp automate run
|
|
306
|
+
agent-rdp automate run "Get-Process" --wait --process-timeout 5000
|
|
307
|
+
agent-rdp automate run "$PSVersionTable" --wait --shell pwsh.exe
|
|
308
|
+
agent-rdp automate run "ping -t 127.0.0.1" --stream # returns a pid immediately
|
|
309
|
+
agent-rdp automate run-poll <pid> # drain output; repeat until exited
|
|
494
310
|
|
|
495
|
-
|
|
496
|
-
agent-rdp automate
|
|
497
|
-
|
|
311
|
+
agent-rdp automate status # uptime, last DVC round-trip, failure count
|
|
312
|
+
agent-rdp automate restart # relaunch the agent, keeping the RDP session
|
|
313
|
+
```
|
|
498
314
|
|
|
499
|
-
|
|
500
|
-
|
|
315
|
+
**Selectors:** `@e5` (snapshot ref), `#SaveButton` (automation ID), `.Edit`
|
|
316
|
+
(Win32 class), `~*pattern*` (wildcard name), `File` (exact name).
|
|
501
317
|
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
agent-rdp automate restart
|
|
506
|
-
```
|
|
318
|
+
**Snapshot format**, with `disabled` shown on interactive elements — a free
|
|
319
|
+
pre-action state check (a disabled "Отменить проведение" tells you the document
|
|
320
|
+
isn't posted before you click anything):
|
|
507
321
|
|
|
508
|
-
An `automation indeterminate` error means the DVC reply was lost, not that
|
|
509
|
-
the action necessarily failed or succeeded. The agent now keeps the results of
|
|
510
|
-
its recent requests, so when a reply goes missing the daemon asks it what
|
|
511
|
-
actually happened and usually returns the real outcome instead - or, if the
|
|
512
|
-
agent has no record of the request, says outright that it never ran and
|
|
513
|
-
retrying is safe. Only when the agent is still busy does a genuine
|
|
514
|
-
`indeterminate` remain; check state before retrying in that case, or a
|
|
515
|
-
retried click/fill can apply twice. Read-only commands (`snapshot`, `get`,
|
|
516
|
-
`status`, `wait-for`, `window list`) say so explicitly, since those can always
|
|
517
|
-
be retried safely.
|
|
518
|
-
|
|
519
|
-
**Long-running commands.** `run --wait` and `wait-for` are given the time
|
|
520
|
-
they ask for: the transport deadline, the CLI's socket timeout and the
|
|
521
|
-
watchdog all extend to cover `--process-timeout`/`--timeout`. Note the agent
|
|
522
|
-
processes one command at a time, so a long `run --wait` blocks every other
|
|
523
|
-
`automate` command until it finishes — for anything beyond about a minute,
|
|
524
|
-
prefer `run --stream` plus `run-poll`, which returns immediately and lets you
|
|
525
|
-
drain output while the session stays responsive.
|
|
526
|
-
|
|
527
|
-
Snapshots include `disabled` on interactive elements — a free pre-action
|
|
528
|
-
state check. A disabled "Отменить проведение" menu item, for example, tells
|
|
529
|
-
you the document isn't posted before you ever click anything.
|
|
530
|
-
|
|
531
|
-
**Keyboard navigation inside a panel (arrow keys + Enter) can land on the
|
|
532
|
-
wrong item.** Observed in 1C side panels ("Функции" etc.): Up/Down selection
|
|
533
|
-
followed by Enter is not reliably deterministic. Prefer `automate` refs when
|
|
534
|
-
the panel is exposed to UI Automation; when it isn't, use two independent
|
|
535
|
-
coordinate measurements and [`click-at --confirm`](#click-at-safe-clicking-of-externally-computed-coordinates)
|
|
536
|
-
rather than arrow-key navigation.
|
|
537
|
-
|
|
538
|
-
**Selector Types:**
|
|
539
|
-
- `@e5` or `@5` - Reference number from snapshot (e prefix recommended)
|
|
540
|
-
- `#SaveButton` - Automation ID
|
|
541
|
-
- `.Edit` - Win32 class name
|
|
542
|
-
- `~*pattern*` - Wildcard name match
|
|
543
|
-
- `File` - Element name (exact match)
|
|
544
|
-
|
|
545
|
-
**Snapshot Output Format:**
|
|
546
322
|
```
|
|
547
323
|
- Window "Notepad" [ref=e1, id=Notepad]
|
|
548
324
|
- MenuBar "Application" [ref=e2]
|
|
@@ -550,81 +326,64 @@ rather than arrow-key navigation.
|
|
|
550
326
|
- Edit "Text Editor" [ref=e5, value="Hello"]
|
|
551
327
|
```
|
|
552
328
|
|
|
553
|
-
|
|
329
|
+
**`automation indeterminate`** means the reply was lost, not that the action
|
|
330
|
+
failed. The agent journals recent results, so the daemon asks what actually
|
|
331
|
+
happened and usually returns the real outcome — or reports that the request
|
|
332
|
+
never ran and is safe to retry. A surviving `indeterminate` means the agent is
|
|
333
|
+
still busy: check state before retrying, or a click/fill can apply twice.
|
|
334
|
+
Read-only commands (`snapshot`, `get`, `status`, `wait-for`, `window list`) say
|
|
335
|
+
so explicitly, since those are always safe to retry.
|
|
336
|
+
|
|
337
|
+
**Long-running commands** get the time they ask for: the transport deadline,
|
|
338
|
+
the CLI socket timeout and the watchdog all extend to cover
|
|
339
|
+
`--process-timeout`/`--timeout`. But the agent handles one command at a time,
|
|
340
|
+
so a long `run --wait` blocks every other `automate` call — past about a
|
|
341
|
+
minute, prefer `run --stream` plus `run-poll`. Streamed output is captured to
|
|
342
|
+
files on the remote side, so nothing is lost if the process exits between
|
|
343
|
+
polls; a finished process stays pollable for 10 minutes and repeat polls return
|
|
344
|
+
`exited: true` with empty chunks. (If you redirect inside the command instead,
|
|
345
|
+
note that Windows PowerShell 5.1's `>` writes UTF-16LE.)
|
|
346
|
+
|
|
347
|
+
**`daemon_not_running` vs `daemon_unresponsive`.** The first means no daemon
|
|
348
|
+
process exists — reconnect, and `<session>/daemon.log` says why it exited. The
|
|
349
|
+
second means the process is alive but did not answer a health check within
|
|
350
|
+
10s: it is busy (a long `run --wait`, a file transfer). Wait and retry; do not
|
|
351
|
+
reconnect, that discards a working session. The message includes the tail of
|
|
352
|
+
daemon.log. `connect` replaces a daemon that stays unresponsive, killing the
|
|
353
|
+
stuck one first.
|
|
354
|
+
|
|
355
|
+
**Arrow-key navigation inside a panel can land on the wrong item.** Observed in
|
|
356
|
+
1C side panels: Up/Down then Enter is not reliably deterministic. Prefer
|
|
357
|
+
`automate` refs; when the panel isn't exposed to UI Automation, use two
|
|
358
|
+
independent measurements with [`click-at --confirm`](#click-at-safe-clicking-of-externally-computed-coordinates).
|
|
359
|
+
|
|
360
|
+
### Sessions and the web viewer
|
|
554
361
|
|
|
555
362
|
```bash
|
|
556
|
-
# List active sessions
|
|
557
363
|
agent-rdp session list
|
|
558
|
-
|
|
559
|
-
# Get current session info
|
|
560
364
|
agent-rdp session info
|
|
561
|
-
|
|
562
|
-
# Close a session
|
|
563
|
-
agent-rdp session close
|
|
564
|
-
|
|
565
|
-
# Use a named session
|
|
566
|
-
agent-rdp --session work connect --host work-pc.local ...
|
|
365
|
+
agent-rdp --session work connect --host work-pc.local ... # named session
|
|
567
366
|
agent-rdp --session work screenshot
|
|
568
|
-
```
|
|
569
|
-
|
|
570
|
-
### Disconnect
|
|
571
|
-
|
|
572
|
-
```bash
|
|
573
|
-
agent-rdp disconnect
|
|
574
|
-
```
|
|
575
|
-
|
|
576
|
-
### Web Viewer
|
|
577
367
|
|
|
578
|
-
|
|
368
|
+
agent-rdp wait 3000 # pause; useful right after connect, before the first input
|
|
579
369
|
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
agent-rdp view
|
|
583
|
-
|
|
584
|
-
# Specify a different port
|
|
370
|
+
# Web viewer (needs streaming enabled at connect)
|
|
371
|
+
agent-rdp --stream-port 9224 connect --host 192.168.1.100 -u Admin -p secret
|
|
585
372
|
agent-rdp view --port 9224
|
|
586
373
|
```
|
|
587
374
|
|
|
588
|
-
|
|
375
|
+
There is no `session close` — use `disconnect`.
|
|
589
376
|
|
|
590
|
-
|
|
591
|
-
agent-rdp --stream-port 9224 connect --host 192.168.1.100 -u Admin -p secret
|
|
592
|
-
agent-rdp view
|
|
593
|
-
```
|
|
377
|
+
## JSON output
|
|
594
378
|
|
|
595
|
-
|
|
379
|
+
Add `--json` to any command:
|
|
596
380
|
|
|
597
|
-
All commands support `--json` for structured output:
|
|
598
|
-
|
|
599
|
-
```bash
|
|
600
|
-
agent-rdp --json screenshot --output desktop.png
|
|
601
|
-
```
|
|
602
|
-
|
|
603
|
-
**Success response:**
|
|
604
|
-
```json
|
|
605
|
-
{
|
|
606
|
-
"success": true,
|
|
607
|
-
"data": {
|
|
608
|
-
"type": "screenshot",
|
|
609
|
-
"path": "desktop.png",
|
|
610
|
-
"width": 1920,
|
|
611
|
-
"height": 1080
|
|
612
|
-
}
|
|
613
|
-
}
|
|
614
|
-
```
|
|
615
|
-
|
|
616
|
-
**Error response:**
|
|
617
381
|
```json
|
|
618
|
-
{
|
|
619
|
-
|
|
620
|
-
"error": {
|
|
621
|
-
"code": "not_connected",
|
|
622
|
-
"message": "Not connected to an RDP server"
|
|
623
|
-
}
|
|
624
|
-
}
|
|
382
|
+
{ "success": true, "data": { "type": "screenshot", "path": "desktop.png", "width": 1920, "height": 1080 } }
|
|
383
|
+
{ "success": false, "error": { "code": "not_connected", "message": "Not connected to an RDP server" } }
|
|
625
384
|
```
|
|
626
385
|
|
|
627
|
-
## Environment
|
|
386
|
+
## Environment variables
|
|
628
387
|
|
|
629
388
|
| Variable | Description |
|
|
630
389
|
|----------|-------------|
|
|
@@ -634,173 +393,108 @@ agent-rdp --json screenshot --output desktop.png
|
|
|
634
393
|
| `AGENT_RDP_PASSWORD` | RDP password |
|
|
635
394
|
| `AGENT_RDP_SESSION` | Session name (default: "default") |
|
|
636
395
|
| `AGENT_RDP_STREAM_PORT` | WebSocket streaming port (0 = disabled) |
|
|
637
|
-
| `AGENT_RDP_MODELS_DIR` |
|
|
396
|
+
| `AGENT_RDP_MODELS_DIR` | OCR models directory (set automatically by the npm wrapper; needed for standalone binary installs) |
|
|
638
397
|
|
|
639
398
|
## Node.js API
|
|
640
399
|
|
|
641
|
-
Use agent-rdp programmatically from Node.js/TypeScript:
|
|
642
|
-
|
|
643
400
|
```typescript
|
|
644
401
|
import { RdpSession } from 'agent-rdp';
|
|
645
402
|
|
|
646
403
|
const rdp = new RdpSession({ session: 'default' });
|
|
647
|
-
|
|
648
404
|
await rdp.connect({
|
|
649
|
-
host: '192.168.1.100',
|
|
650
|
-
|
|
651
|
-
password: 'secret',
|
|
652
|
-
width: 1280,
|
|
653
|
-
height: 800,
|
|
405
|
+
host: '192.168.1.100', username: 'Administrator', password: 'secret',
|
|
406
|
+
width: 1280, height: 800,
|
|
654
407
|
drives: [{ path: '/tmp/share', name: 'Share' }],
|
|
655
|
-
enableWinAutomation: true,
|
|
408
|
+
enableWinAutomation: true,
|
|
656
409
|
});
|
|
657
410
|
|
|
658
|
-
// Screenshot - prefer `path` so a large base64 string never
|
|
659
|
-
|
|
660
|
-
const {
|
|
661
|
-
|
|
662
|
-
// Or get raw base64 directly (e.g. for further in-process processing)
|
|
663
|
-
const { base64 } = await rdp.screenshot({ format: 'png' });
|
|
411
|
+
// Screenshot - prefer `path` so a large base64 string is never held in memory
|
|
412
|
+
const { path, width, height } = await rdp.screenshot({ format: 'png', path: 'shot.png' });
|
|
413
|
+
const { base64 } = await rdp.screenshot({ format: 'png' }); // or raw, for in-process use
|
|
664
414
|
|
|
665
|
-
//
|
|
666
|
-
await rdp.mouse.click({ x: 100, y: 200 });
|
|
667
|
-
await rdp.mouse.rightClick({ x: 100, y: 200 });
|
|
668
|
-
await rdp.mouse.doubleClick({ x: 100, y: 200 });
|
|
669
|
-
await rdp.mouse.move({ x: 150, y: 250 });
|
|
415
|
+
await rdp.mouse.click({ x: 100, y: 200 }); // also rightClick, doubleClick, move
|
|
670
416
|
await rdp.mouse.drag({ from: { x: 100, y: 100 }, to: { x: 500, y: 500 } });
|
|
671
417
|
|
|
672
|
-
// Keyboard
|
|
673
418
|
await rdp.keyboard.type({ text: 'Hello World' });
|
|
674
|
-
await rdp.keyboard.paste('Привет, мир!');
|
|
419
|
+
await rdp.keyboard.paste('Привет, мир!'); // reliable for long/non-Latin text
|
|
675
420
|
await rdp.keyboard.press({ keys: 'ctrl+c' });
|
|
676
|
-
await rdp.keyboard.
|
|
677
|
-
await rdp.keyboard.down('shift'); // Hold across other commands
|
|
678
|
-
await rdp.keyboard.up('shift');
|
|
421
|
+
await rdp.keyboard.down('shift'); await rdp.keyboard.up('shift');
|
|
679
422
|
|
|
680
|
-
//
|
|
681
|
-
await rdp.scroll.up(); // Default amount: 3
|
|
682
|
-
await rdp.scroll.down({ amount: 5 }); // Custom amount
|
|
683
|
-
await rdp.scroll.up({ x: 500, y: 300 }); // Scroll at position
|
|
684
|
-
|
|
685
|
-
// Clipboard
|
|
423
|
+
await rdp.scroll.down({ amount: 5 }); // default 3; { x, y } to target a point
|
|
686
424
|
await rdp.clipboard.set({ text: 'text to copy' });
|
|
687
425
|
const text = await rdp.clipboard.get();
|
|
688
426
|
|
|
689
|
-
//
|
|
690
|
-
const matches = await rdp.locate({ text: 'Cancel' });
|
|
691
|
-
if (matches.length > 0) {
|
|
692
|
-
await rdp.mouse.click({ x: matches[0].center_x, y: matches[0].center_y });
|
|
693
|
-
}
|
|
694
|
-
|
|
695
|
-
// Get all text on screen
|
|
696
|
-
const allText = await rdp.locate({ all: true });
|
|
697
|
-
|
|
698
|
-
// Click a match directly - the coordinate never leaves the process
|
|
427
|
+
// OCR - click directly so the coordinate never leaves the process
|
|
699
428
|
await rdp.locate({ text: 'Cancel', click: 'left' });
|
|
700
|
-
|
|
701
|
-
// Exact whole-line match - won't also hit "Провести и закрыть"
|
|
702
429
|
await rdp.locate({ text: 'Провести', exact: true, click: 'left' });
|
|
430
|
+
await rdp.locate({ text: 'OK', waitMs: 10000, click: 'left' }); // block until it appears
|
|
431
|
+
const allText = await rdp.locate({ all: true });
|
|
703
432
|
|
|
704
|
-
//
|
|
705
|
-
await rdp.locate({ text: 'OK', waitMs: 10000, click: 'left' });
|
|
706
|
-
|
|
707
|
-
// Safe click of an externally-computed point (e.g. a vision-model bbox):
|
|
708
|
-
// refuses if it's ambiguously close to more than one detected label
|
|
433
|
+
// Safe click of an externally-computed point (e.g. a vision-model bbox)
|
|
709
434
|
const result = await rdp.clickAt(665, 209);
|
|
710
435
|
if (!result.clicked) console.log('Ambiguous:', result.nearby);
|
|
711
436
|
|
|
712
|
-
//
|
|
437
|
+
// File transfer (needs enableWinAutomation)
|
|
438
|
+
await rdp.files.push('./report.xlsx', 'C:\\Users\\Admin\\report.xlsx');
|
|
439
|
+
await rdp.files.pull('C:\\Users\\Admin\\export.csv', './export.csv');
|
|
440
|
+
|
|
441
|
+
// UI Automation
|
|
713
442
|
const snapshot = await rdp.automation.snapshot({ interactive: true });
|
|
714
|
-
const focused = await rdp.automation.focused();
|
|
715
|
-
await rdp.automation.click('@e5'
|
|
716
|
-
await rdp.automation.
|
|
717
|
-
await rdp.automation.
|
|
718
|
-
await rdp.automation.toggle('@e7'); // Toggle checkbox
|
|
719
|
-
await rdp.automation.expand('@e3'); // Expand menu
|
|
720
|
-
await rdp.automation.contextMenu('@e5'); // Open context menu
|
|
721
|
-
await rdp.automation.fill('#input', 'text'); // Fill text field
|
|
722
|
-
await rdp.automation.run('notepad.exe'); // Run command
|
|
443
|
+
const focused = await rdp.automation.focused();
|
|
444
|
+
await rdp.automation.click('@e5', { doubleClick: true });
|
|
445
|
+
await rdp.automation.fill('#input', 'text');
|
|
446
|
+
await rdp.automation.run('notepad.exe');
|
|
723
447
|
await rdp.automation.waitFor('#SaveButton', { timeout: 5000 });
|
|
724
|
-
|
|
725
|
-
// Window management
|
|
726
|
-
const windows = await rdp.automation.listWindows();
|
|
727
448
|
await rdp.automation.focusWindow('~*Notepad*');
|
|
728
|
-
await rdp.automation.maximizeWindow();
|
|
729
449
|
|
|
730
|
-
// Drives
|
|
731
450
|
const drives = await rdp.drives.list();
|
|
732
|
-
|
|
733
|
-
// Session info
|
|
734
451
|
const info = await rdp.getInfo();
|
|
735
|
-
|
|
736
|
-
// Disconnect
|
|
737
452
|
await rdp.disconnect();
|
|
738
453
|
```
|
|
739
454
|
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
Enable WebSocket streaming for real-time screen capture and bidirectional clipboard support:
|
|
455
|
+
The process can stay alive after `disconnect()` unless the caller cleans up —
|
|
456
|
+
call `rdp.close()` or exit explicitly.
|
|
743
457
|
|
|
744
|
-
|
|
745
|
-
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
});
|
|
749
|
-
|
|
750
|
-
await rdp.connect({...});
|
|
751
|
-
|
|
752
|
-
// Connect your WebSocket client to receive JPEG frames
|
|
753
|
-
const streamUrl = rdp.getStreamUrl(); // "ws://localhost:9224"
|
|
754
|
-
```
|
|
755
|
-
|
|
756
|
-
For the complete WebSocket protocol specification (message types, clipboard flow, input handling), see [WEBSOCKET.md](https://github.com/denisix/agent-rdp/blob/main/docs/WEBSOCKET.md).
|
|
458
|
+
**WebSocket streaming** for real-time capture: construct with
|
|
459
|
+
`{ streamPort: 9224 }`, then `rdp.getStreamUrl()` returns `ws://localhost:9224`.
|
|
460
|
+
Message types, clipboard flow and input handling are specified in
|
|
461
|
+
[WEBSOCKET.md](docs/WEBSOCKET.md).
|
|
757
462
|
|
|
758
463
|
## Architecture
|
|
759
464
|
|
|
760
|
-
|
|
761
|
-
|
|
762
|
-
|
|
763
|
-
|
|
764
|
-
3. **IPC** - Unix sockets (macOS/Linux) or TCP (Windows)
|
|
765
|
-
|
|
766
|
-
The daemon is automatically started on the first command and persists until explicitly closed or the session times out.
|
|
465
|
+
A daemon per session: the **CLI** parses commands and talks to a **daemon** over
|
|
466
|
+
Unix sockets (macOS/Linux) or TCP (Windows); the daemon owns the RDP connection
|
|
467
|
+
and processes commands. It starts on the first command and persists until
|
|
468
|
+
closed. See [CLAUDE.md](CLAUDE.md) for the internals.
|
|
767
469
|
|
|
768
470
|
## Limitations
|
|
769
471
|
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
773
|
-
- **UAC
|
|
774
|
-
|
|
775
|
-
|
|
776
|
-
|
|
777
|
-
|
|
778
|
-
|
|
779
|
-
|
|
780
|
-
|
|
781
|
-
|
|
782
|
-
```
|
|
783
|
-
|
|
784
|
-
This is not highly reliable (OCR can misread characters, miss text, or return imprecise coordinates), but may work for simple cases like dialog buttons.
|
|
785
|
-
|
|
786
|
-
### Screenshot Coordinate Detection
|
|
787
|
-
|
|
788
|
-
**Claude models** (in non-computer-use mode, such as Claude Code) are poor at estimating pixel coordinates from screenshots. Do not ask Claude to look at a screenshot and guess where to click - it will likely be inaccurate.
|
|
789
|
-
|
|
790
|
-
**Gemini models** are generally good at pixel coordinate estimation from images.
|
|
791
|
-
|
|
792
|
-
If you need vision-based coordinate detection with Claude, implement your own harness using Claude's [Computer Use Tool](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use) which is specifically designed for this purpose.
|
|
472
|
+
- **WebViews** — UI Automation cannot see WebView content (Start menu search,
|
|
473
|
+
Edge, Electron apps). Launch programs with `automate run` or Win+R instead of
|
|
474
|
+
clicking through menus.
|
|
475
|
+
- **UAC dialogs** run on a secure desktop and are invisible to UI Automation.
|
|
476
|
+
There is no good workaround short of the remote user acting manually.
|
|
477
|
+
- **OCR** can misread characters, miss text, or return imprecise coordinates.
|
|
478
|
+
Use it only when UI Automation can't reach an element, and verify before
|
|
479
|
+
clicking anything destructive.
|
|
480
|
+
- **Claude models in non-computer-use mode** (Claude Code included) are poor at
|
|
481
|
+
estimating pixel coordinates from screenshots — don't ask for a guess from an
|
|
482
|
+
image. Gemini models are generally good at it; for Claude, use the
|
|
483
|
+
[Computer Use Tool](https://docs.anthropic.com/en/docs/agents-and-tools/computer-use).
|
|
793
484
|
|
|
794
485
|
## Requirements
|
|
795
486
|
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
487
|
+
Rust 1.75+. The target must offer **TLS** for RDP — agent-rdp uses `rustls`,
|
|
488
|
+
which doesn't implement TLS 1.0/1.1, so legacy hosts (e.g. Windows Server
|
|
489
|
+
2008 R2) fail with a handshake error. **Stock Windows defaults work as
|
|
490
|
+
shipped.**
|
|
799
491
|
|
|
800
|
-
|
|
492
|
+
<details>
|
|
493
|
+
<summary>RDP security layer settings</summary>
|
|
801
494
|
|
|
802
|
-
All six combinations measured against Windows Server 2022, varying only these
|
|
803
|
-
values under
|
|
495
|
+
All six combinations measured against Windows Server 2022, varying only these
|
|
496
|
+
two values under
|
|
497
|
+
`HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp`:
|
|
804
498
|
|
|
805
499
|
| `SecurityLayer` | `UserAuthentication` | Meaning | Result |
|
|
806
500
|
|---|---|---|---|
|
|
@@ -812,12 +506,12 @@ values under `HKLM:\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations
|
|
|
812
506
|
| `0` (RDP) | `0` | legacy RC4 only | fails |
|
|
813
507
|
|
|
814
508
|
The rule: **agent-rdp works wherever the host offers TLS.** With NLA on that is
|
|
815
|
-
always the case,
|
|
509
|
+
always the case, since NLA forces CredSSP/TLS regardless of `SecurityLayer`.
|
|
816
510
|
Both failing rows are NLA-off hosts that refuse TLS outright.
|
|
817
511
|
|
|
818
512
|
If `connect` reports *"server only supports Standard RDP Security"*, the host is
|
|
819
|
-
in one of those
|
|
820
|
-
|
|
513
|
+
in one of those rows. Enable NLA (preferred — it also gives pre-authentication)
|
|
514
|
+
or force TLS:
|
|
821
515
|
|
|
822
516
|
```powershell
|
|
823
517
|
$k = 'HKLM:\System\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp'
|
|
@@ -826,18 +520,24 @@ Set-ItemProperty -Path $k -Name SecurityLayer -Value 2 # force TLS
|
|
|
826
520
|
```
|
|
827
521
|
|
|
828
522
|
New connections normally pick this up immediately; restart `TermService` only if
|
|
829
|
-
they don't (
|
|
523
|
+
they don't (that drops existing sessions).
|
|
830
524
|
|
|
831
525
|
Neither failing case is fixable client-side. With `SecurityLayer=1` and NLA off
|
|
832
526
|
the server returns `SSL_NOT_ALLOWED_BY_SERVER` even though the client advertises
|
|
833
|
-
`PROTOCOL_SSL` — verified by testing a build that requested SSL *only*,
|
|
834
|
-
|
|
835
|
-
|
|
836
|
-
|
|
527
|
+
`PROTOCOL_SSL` — verified by testing a build that requested SSL *only*, rejected
|
|
528
|
+
identically. Standard RDP Security (`SecurityLayer=0`, NLA off) is refused by
|
|
529
|
+
IronRDP itself, which implements no RC4 transport; it uses a well-known key
|
|
530
|
+
derivation, so credentials sent over it are recoverable in transit.
|
|
531
|
+
</details>
|
|
837
532
|
|
|
838
533
|
## Credits
|
|
839
534
|
|
|
840
|
-
Originally created by [Nick Yu](https://github.com/thisnick)
|
|
535
|
+
Originally created by [Nick Yu](https://github.com/thisnick)
|
|
536
|
+
([thisnick/agent-rdp](https://github.com/thisnick/agent-rdp)). This fork
|
|
537
|
+
([denisix/agent-rdp](https://github.com/denisix/agent-rdp), published as
|
|
538
|
+
[`@denisixnpm/agent-rdp`](https://www.npmjs.com/package/@denisixnpm/agent-rdp))
|
|
539
|
+
is maintained independently; see
|
|
540
|
+
[CHANGELOG.md](packages/agent-rdp/CHANGELOG.md) for what's changed.
|
|
841
541
|
|
|
842
542
|
## License
|
|
843
543
|
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* Error codes for structured error handling.
|
|
3
3
|
*/
|
|
4
|
-
export type ErrorCode = "not_connected" | "already_connected" | "connection_failed" | "authentication_failed" | "timeout" | "invalid_request" | "not_supported" | "internal_error" | "session_not_found" | "ipc_error" | "daemon_not_running" | "clipboard_error" | "drive_error" | "automation_not_enabled" | "automation_error" | "automation_indeterminate" | "element_not_found" | "stale_ref" | "command_failed";
|
|
4
|
+
export type ErrorCode = "not_connected" | "already_connected" | "connection_failed" | "authentication_failed" | "timeout" | "invalid_request" | "not_supported" | "internal_error" | "session_not_found" | "ipc_error" | "daemon_not_running" | "daemon_unresponsive" | "clipboard_error" | "drive_error" | "automation_not_enabled" | "automation_error" | "automation_indeterminate" | "element_not_found" | "stale_ref" | "command_failed";
|
|
5
5
|
//# sourceMappingURL=ErrorCode.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"ErrorCode.d.ts","sourceRoot":"","sources":["../../src/generated/ErrorCode.ts"],"names":[],"mappings":"AAEA;;GAEG;AACH,MAAM,MAAM,SAAS,GAAG,eAAe,GAAG,mBAAmB,GAAG,mBAAmB,GAAG,uBAAuB,GAAG,SAAS,GAAG,iBAAiB,GAAG,eAAe,GAAG,gBAAgB,GAAG,mBAAmB,GAAG,WAAW,GAAG,oBAAoB,GAAG,iBAAiB,GAAG,aAAa,GAAG,wBAAwB,GAAG,kBAAkB,GAAG,0BAA0B,GAAG,mBAAmB,GAAG,WAAW,GAAG,gBAAgB,CAAC"}
|
|
1
|
+
{"version":3,"file":"ErrorCode.d.ts","sourceRoot":"","sources":["../../src/generated/ErrorCode.ts"],"names":[],"mappings":"AAEA;;GAEG;AACH,MAAM,MAAM,SAAS,GAAG,eAAe,GAAG,mBAAmB,GAAG,mBAAmB,GAAG,uBAAuB,GAAG,SAAS,GAAG,iBAAiB,GAAG,eAAe,GAAG,gBAAgB,GAAG,mBAAmB,GAAG,WAAW,GAAG,oBAAoB,GAAG,qBAAqB,GAAG,iBAAiB,GAAG,aAAa,GAAG,wBAAwB,GAAG,kBAAkB,GAAG,0BAA0B,GAAG,mBAAmB,GAAG,WAAW,GAAG,gBAAgB,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@denisixnpm/agent-rdp",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.11",
|
|
4
4
|
"description": "CLI tool for AI agents to control Windows Remote Desktop sessions",
|
|
5
5
|
"repository": {
|
|
6
6
|
"type": "git",
|
|
@@ -48,11 +48,11 @@
|
|
|
48
48
|
},
|
|
49
49
|
"homepage": "https://github.com/denisix/agent-rdp#readme",
|
|
50
50
|
"optionalDependencies": {
|
|
51
|
-
"@denisixnpm/agent-rdp-darwin-arm64": "^0.7.
|
|
52
|
-
"@denisixnpm/agent-rdp-darwin-x64": "^0.7.
|
|
53
|
-
"@denisixnpm/agent-rdp-linux-x64": "^0.7.
|
|
54
|
-
"@denisixnpm/agent-rdp-linux-arm64": "^0.7.
|
|
55
|
-
"@denisixnpm/agent-rdp-win32-x64": "^0.7.
|
|
56
|
-
"@denisixnpm/agent-rdp-win32-arm64": "^0.7.
|
|
51
|
+
"@denisixnpm/agent-rdp-darwin-arm64": "^0.7.11",
|
|
52
|
+
"@denisixnpm/agent-rdp-darwin-x64": "^0.7.11",
|
|
53
|
+
"@denisixnpm/agent-rdp-linux-x64": "^0.7.11",
|
|
54
|
+
"@denisixnpm/agent-rdp-linux-arm64": "^0.7.11",
|
|
55
|
+
"@denisixnpm/agent-rdp-win32-x64": "^0.7.11",
|
|
56
|
+
"@denisixnpm/agent-rdp-win32-arm64": "^0.7.11"
|
|
57
57
|
}
|
|
58
58
|
}
|